# Integrated RL Trainer: BCE + DQN + PPO + Aux on a Shared Encoder

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task.

**Goal:** Replace the pure-supervised BCE alpha-perception trainer with a single integrated trainer where the Mamba2+CfC encoder receives gradient from FIVE complementary loss heads — BCE direction, DQN distributional Q, PPO policy + value, and auxiliary prof/size — so the encoder learns features useful for **realized trading PnL**, not just price-direction prediction.

**Architecture:** One shared encoder produces `h_t ∈ R^128`. Five heads sit on top: (1) BCE direction (warm-start signal), (2) C51 distributional Q-head (off-policy critic), (3) categorical π policy head (on-policy actor), (4) scalar V value head, (5) prof+size aux head (per-trade calibration). Total loss is a weighted sum with adaptive λ via the existing loss-balance controller. Action space is shared 9-element discrete grid. Reward is per-trade realized PnL from the GPU LobSim. Replay buffer (PER) feeds Q-head off-policy; rollout buffer feeds π+V on-policy; both buffers fill from the SAME training-loop transitions.

**Tech Stack:** Rust 1.85+, Tokio 1.40, CUDA 12.4 L40S/H100, cudarc 0.19. Existing components reused: Mamba2 SSM + CfC encoder (ml-alpha/src/cfc/), aux trunk + heads (ml-alpha/src/aux_heads.rs, aux_trunk.rs), LobSim (ml-backtesting/src/sim/), perception trainer skeleton (ml-alpha/src/trainer/perception.rs). New components: DQN Q-head + replay, PPO π/V heads + rollout, action-grid kernels, reward-shaping kernels, loss-balance controller wiring for 5 heads.

---

## Why this architecture

The user's question motivating this plan: "the model should run inference during training to learn from its mistakes — does it?" Answer: NO, the current PerceptionTrainer optimizes only BCE on offline `sign(p[t+K] - p[t])` labels. It never sees its own trade outcomes, fees, slippage, or PnL.

Empirical evidence: alpha-perception-jvv7d trained to auc_h1000=0.576 (decent directional signal). Backtest `lob-backtest-sweep-jpdhg` on that checkpoint gave:
- profit_factor = 0.24
- sharpe_ann = -9.72
- total_pnl_usd = -$1.29M
- win-rate correlates with conviction (16% → 27%) BUT mean_pnl gets WORSE with conviction

The decoupling is structural: the encoder optimizes directional AUC but never sees the cost-aware PnL gradient. The fix is to give the encoder a PnL-aware gradient via RL, while keeping the BCE signal for stable warm-start.

### Why both DQN AND PPO (not OR)

DQN-only weaknesses: ε-greedy exploration wastes 10-30% of compute on random actions; struggles when action distinguishability is subtle.

PPO-only weaknesses: high variance from MC returns; on-policy = throws away old data; needs GAE-λ tricks.

Combined (SAC-style actor-critic): Q-head provides the LOW-VARIANCE advantage signal to π — no need for GAE-λ. π-head provides STOCHASTIC exploration informed by current policy beliefs — no ε-greedy. The two cross-validate (Q-argmax should agree with π-mode at convergence). One encoder gets BOTH gradient streams plus BCE + aux supervision.

---

## Architecture diagram

```
[MBP-10 raw features, 32 ticks]
              │
              ▼
   [snap_feature_assemble kernel]
              │  feature vector [B, 32, 40]
              ▼
   [Mamba2 SSM, state_dim=16]
              │
              ▼
   [CfC continuous-time recurrence, hidden=128]
              │  h_t [B, 128]
              ▼
   ┌──────────┼──────────┬──────────┬──────────┐
   │          │          │          │          │
   ▼          ▼          ▼          ▼          ▼
[BCE dir] [C51 Q]    [π policy] [V value] [Aux prof+size]
[B, 3]    [B, 9, 21] [B, 9]     [B, 1]    [B, 2, 3, 2]
 │         │          │          │           │
sigmoid   atoms over  softmax    raw         sigmoid+raw
 │         [Q_min,    │          │           │
 │         Q_max]    │          │           │
 │         │          │          │           │
L_BCE     L_Q (Bellman)  L_π    L_V (MSE)   L_aux
direction TD on s,a  (clipped   to Q-expect (BCE_prof
labels    target net surr,A_Q) ation        + Huber_size)

           ─────── weighted sum ───────
                   L_total
                      │
                      ▼ backward
              encoder ∇ = ∇ × λ_total
```

### Action space (9 discrete actions, shared by Q and π)

| idx | Action | Effect |
|-----|--------|--------|
| 0 | Short-large | Open short, size = 2× base |
| 1 | Short-small | Open short, size = 1× base |
| 2 | Hold | No change to position |
| 3 | Flat-from-long | Close long position if any |
| 4 | Flat-from-short | Close short position if any |
| 5 | Long-small | Open long, size = 1× base |
| 6 | Long-large | Open long, size = 2× base |
| 7 | Trail-tighten | Reduce trailing stop distance by 25% |
| 8 | Trail-loosen | Increase trailing stop distance by 25% |

Note: matches existing DQN action grid in `crates/ml/src/cuda_pipeline/sp5_isv_slots.rs` for parity. Trail-tighten/loosen are stop-management actions; the other 7 are position-management.

### Reward (per-trade realized PnL on close)

`reward = realized_pnl_usd - fees_usd - exit_path_cost`

Reward fires at trade CLOSE (when position goes to 0). Between closes, reward is 0. Per-trade signal matches what we actually optimize (PF, sharpe). Sparser than per-tick but unambiguous credit assignment.

For TD(λ) bootstrap of intermediate values, the V-head learns the discounted-future-reward estimate; Q-head learns per-action discounted return. λ_discount = 0.99 per HFT convention (matches old DQN gamma).

### Loss formulation

```
L_total = λ_bce × L_BCE(dir_logits, dir_labels)
        + λ_q   × L_Q(Q(s_t, a_t), r_t + γ × max_a' Q_target(s_{t+1}, a'))
        + λ_π   × L_π(π(s_t), A_Q(s_t, a_t) = Q(s_t, a_t) - V(s_t))
        + λ_v   × L_V(V(s_t), Σ_a π(a|s_t) × Q(s_t, a))
        + λ_aux × (L_aux_prof + L_aux_size)
```

Where:
- L_Q is Huber loss on the distributional TD error (per pearl_per_branch_c51_atom_span)
- L_π is the PPO clipped surrogate: `min(r_t × A, clip(r_t, 1-ε, 1+ε) × A)`, r_t = π_new/π_old
- L_V is MSE between V prediction and the expected value under the policy
- L_aux uses the existing aux_loss.cu kernels (BCE prof + masked Huber size)
- λ_k are learned per-head via the existing loss-balance controller (`pearl_loss_balance_controller`)

---

## State / observation

State at step t = encoder output `h_t ∈ R^128`. The encoder reads:
- Last 32 raw MBP-10 ticks (default `single_scale_32` from the just-reverted multi-resolution config)
- Regime context (6 EMAs cascade) per existing loader
- Δt Fourier features per existing snap_feature kernel
- (Optional) position state vector: `[current_position_lots, current_avg_entry_px, hold_duration_ns]` concatenated to h_t before head dispatch. This lets the policy see its own state.

---

## Training loop

```rust
loop {
    let batch = loader.next_sequence();           // existing
    let h_t = encoder.forward(batch.features);    // existing

    // Action selection during rollout: Thompson over Q distribution
    // (pearl_thompson_for_distributional_action_selection)
    let action = sample_thompson(q_head.forward(h_t));

    // Sim step: submit action, advance one event, get reward
    let (next_state, reward, done) = lobsim.step(action);

    // Store transition in BOTH buffers
    let transition = Transition {
        h_t,
        action,
        reward,
        next_h_t: next_state.encoded(),
        done,
        // Log probability for PPO importance ratio
        log_pi: pi_head.log_prob(action),
    };
    replay_buffer.push(transition.clone());  // off-policy for Q
    rollout_buffer.push(transition);          // on-policy for π+V

    // Off-policy DQN update (every step, from PER replay)
    if replay_buffer.len() > min_warmup {
        let dqn_batch = replay_buffer.sample_per(batch_size);
        let l_q = dqn_loss(dqn_batch, q_head, q_target_head);
    }

    // On-policy PPO update (at end of rollout)
    if rollout_buffer.full() {
        for _ in 0..n_ppo_epochs {
            let l_pi = ppo_clipped_surrogate(rollout_buffer, pi_head, q_head);
            let l_v  = value_mse(rollout_buffer, v_head, q_head, pi_head);
        }
        rollout_buffer.clear();
    }

    // BCE + aux losses (every step)
    let l_bce = bce_dir(bce_head.forward(h_t), batch.dir_labels);
    let l_aux = aux_loss(aux_head.forward(h_t), batch.aux_labels);

    // Loss-balance controller updates λ_k
    let lambdas = loss_balance_controller.update_and_get();

    let l_total = lambdas.bce * l_bce
                + lambdas.q   * l_q
                + lambdas.pi  * l_pi
                + lambdas.v   * l_v
                + lambdas.aux * l_aux;
    l_total.backward();

    // Periodic target-network soft update
    if step % target_update_freq == 0 {
        q_target_head.soft_update(q_head, tau = 0.005);
    }
}
```

Key design points:
1. **Both buffers fill from the SAME transitions** — no separate rollouts. PPO uses fresh data from the most-recent rollout window; DQN uses ANY data from the larger replay buffer.
2. **One forward pass per step** computes all 5 heads from `h_t`.
3. **One backward pass** combines all gradients into the encoder.
4. **Target network** is for Q only (PPO doesn't need one). Soft updates per `pearl_per_branch_c51_atom_span` infrastructure (already exists).
5. **Thompson action selection** at rollout time (pearl-mandated); argmax-Q only at eval time.

---

## ISV-driven adaptive control (mandatory per project memory)

Per `feedback_isv_for_adaptive_bounds` and `pearl_controller_anchors_isv_driven`,
**every** hyperparameter / threshold / cap in this trainer MUST be sourced from an
ISV slot updated by a per-step controller kernel. No `const f32 = 0.99` for γ, no
`const f32 = 0.2` for PPO clip ε, no `const f32 = 0.005` for target-net τ.

The existing `crates/ml/src/cuda_pipeline/sp5_isv_slots.rs` ISV bus has 56+ slots
(`ISV_TOTAL_DIM = 383`). The integrated RL trainer adds a new block of slots and
reuses the existing controller infrastructure:

### New ISV slots (allocate at top of ml-alpha-specific block, e.g. 400-450)

| Slot | Name | Role | Controller (producer) | Bootstrap |
|------|------|------|----------------------|-----------|
| 400 | `RL_GAMMA_INDEX` | Discount factor (Q + V + PPO advantage) | Trade-horizon EMA controller (input: mean trade duration / 60s anchor) | Pearl-A first observation; sentinel 0 → 0.99 init |
| 401 | `RL_TARGET_TAU_INDEX` | Target-net soft-update fraction | Q-divergence EMA controller (input: ‖Q_current - Q_target‖ EMA) | Pearl-A; sentinel 0 → 0.005 init |
| 402 | `RL_PPO_CLIP_INDEX` | Clipped surrogate ε | Policy-divergence controller (input: KL(π_new‖π_old) EMA, target = 0.01) | Pearl-A; sentinel 0 → 0.2 init |
| 403 | `RL_ENTROPY_COEF_INDEX` | Entropy bonus weight | Action-entropy deficit controller (target = 70% × ln(9)) per `pearl_per_branch_noisy_sigma` pattern | Pearl-A; sentinel 0 → 0.01 init |
| 404 | `RL_N_ROLLOUT_STEPS_INDEX` | Rollout buffer flush size | Advantage-variance controller (input: var(A_Q) / |A_Q| ratio, target = 0.1) | Pearl-A; sentinel 0 → 2048 init |
| 405 | `RL_PER_ALPHA_INDEX` | PER priority exponent | TD-error distribution controller (input: TD-error kurtosis EMA) | Pearl-A; sentinel 0 → 0.6 init |
| 406 | `RL_REWARD_SCALE_INDEX` | Per-trade reward standardization | PnL magnitude EMA (input: mean |realized_pnl_usd| over last 100 closes) | Pearl-A; sentinel 0 → 1.0 init |
| 407 | `RL_Q_ARG_VS_PI_AGREE_INDEX` | Agreement rate Q-argmax vs π-mode | Diagnostic-only (read by alarm; not controlled) | EMA-tracked, no init magic |
| 408-411 | `RL_LOSS_LAMBDA_BCE/Q/PI/V_INDEX` (4 slots) | Per-head loss weight (BCE, Q, π, V; aux uses existing slot) | Reuse `pearl_loss_balance_controller` Wiener-α + signal-modulated target controller | Pearl-A from `pearl_first_observation_bootstrap`; sentinel 0 → equal-weight 1/5 init |

The loss-balance controller (existing, `pearl_loss_balance_controller`) ALREADY
handles per-loss weight balancing for the perception trainer's 3 horizons. We
EXTEND it to 5 heads (BCE, Q, π, V, aux) by allocating 4 new λ slots (aux λ
already exists per current trainer). The controller's two-layer design
(signal-modulated target × Wiener-α adaptation) covers our need; no new
algorithm.

### Reused existing controllers (no new infrastructure)

| ISV slot | Reused for |
|----------|-----------|
| `EVAL_THOMPSON_TEMP_INDEX` (existing) | Thompson temperature at rollout action selection |
| `KELLY_F_SMOOTH_INDEX = 280` (existing) | Kelly fraction smoothing — fed to action sizing in `action.rs` |
| `CONVICTION_SMOOTH_INDEX = 281` (existing) | Conviction EMA for outcome bucketing diagnostic |
| `Q_VAR_PER_BRANCH_BASE = 222` (existing) | Per-branch Q-variance EMA — input to several RL controllers |
| `FLATNESS_BASE = 206` (existing) | Per-branch flatness signal — input to budget controllers |
| `NOISY_SIGMA_BASE = 210` (existing) | NoisyNet σ — REUSED for stochastic exploration on π head (instead of new entropy bonus mechanism) |
| `BUDGET_C51_BASE = 190` (existing) | C51 loss budget per branch — reused for our Q-head |
| `LB_C51_ACTIVE_BASE = 317` (existing) | Loss-balance C51 activation flag — extend to gate Q-head warmup |
| `PNL_TOTAL_INDEX = 286`, `PNL_MEAN_INDEX = 287`, `PNL_VAR_INDEX = 288`, `PNL_MAX_DD_INDEX = 289` (existing) | PnL telemetry; INPUT to reward-scale controller (RL_REWARD_SCALE_INDEX = 406) |
| `TRAINING_SHARPE_EMA_INDEX = 294`, `MAX_DD_EMA_INDEX = 295` (existing) | Training Sharpe/DD EMAs — INPUT to PnL magnitude controller |
| `REWARD_*_WEIGHT_INDEX` (340-348 existing, SP11 reward subsystem) | DIRECTLY reused as reward shaping component weights; no new reward kernel |
| `CURIOSITY_PRESSURE_INDEX = 346` (existing) | Direct reuse for exploration bonus on top of policy entropy |

### Controller kernels to add (new)

| Kernel | Reads (ISV inputs) | Writes (ISV outputs) | Notes |
|--------|-------------------|---------------------|-------|
| `rl_gamma_controller_kernel.cu` | `PNL_MEAN_INDEX`, `KELLY_F_SMOOTH_INDEX`, mean_trade_duration_ns | `RL_GAMMA_INDEX` (400) | γ derives from "what fraction of trades close within decision horizon" |
| `rl_target_tau_controller_kernel.cu` | Q-divergence EMA (computed in-kernel from current vs target weights) | `RL_TARGET_TAU_INDEX` (401) | τ adapts: faster updates when divergence high |
| `rl_ppo_clip_controller_kernel.cu` | KL(π_new‖π_old) EMA | `RL_PPO_CLIP_INDEX` (402) | ε adapts: tighter clip when KL drifts high |
| `rl_entropy_coef_controller_kernel.cu` | Action-entropy deficit (target = 0.7×ln(9)) | `RL_ENTROPY_COEF_INDEX` (403) | Wiener-α tracking per `pearl_wiener_alpha_floor_for_nonstationary` (α-floor 0.4) |
| `rl_rollout_steps_controller_kernel.cu` | var(A_Q) / \|A_Q\| ratio EMA | `RL_N_ROLLOUT_STEPS_INDEX` (404) | Longer rollout when advantage estimates are noisy |
| `rl_per_alpha_controller_kernel.cu` | TD-error kurtosis EMA | `RL_PER_ALPHA_INDEX` (405) | Higher α when TD-errors heavy-tailed |
| `rl_reward_scale_controller_kernel.cu` | Mean \|realized_pnl_usd\| EMA over last 100 closes (`PNL_MEAN_INDEX`) | `RL_REWARD_SCALE_INDEX` (406) | Standardize rewards so V-head's expected range matches C51 atom support |
| `rl_q_pi_agreement_kernel.cu` | argmax(Q) vs mode(π) per step | `RL_Q_ARG_VS_PI_AGREE_INDEX` (407) | Diagnostic; not consumed by loss; alarms when < 0.5 |

All 8 new controllers follow the existing pattern:
1. Wiener-α-with-floor (per `pearl_wiener_alpha_floor_for_nonstationary`, α-floor = 0.4)
2. First-observation bootstrap from sentinel 0 (per `pearl_first_observation_bootstrap`)
3. State-reset registry entry + matching `reset_named_state` arm (per `feedback_registry_entries_need_dispatch_arms`)
4. Pearl-D recovery cascade: if controller output drifts to invalid range, fold-boundary reset re-bootstraps

### Constants permitted (true structural constants only)

These remain as `const f32` because they're STRUCTURAL parameters of the algorithm,
not adaptive bounds:

- `N_ACTIONS = 9` (action grid cardinality — wired into kernel `#define`s)
- `Q_N_ATOMS = 21` (C51 atom count — kernel layout)
- `HIDDEN_DIM = 128` (encoder output dim — already a workspace const)
- `PPO_N_EPOCHS = 4` per rollout (algorithm hyperparameter; no signal to drive it adaptively)

Everything else is ISV.

### Plan task changes for ISV compliance

- Phase A.2: add the 12 new ISV slot constants (400-411) to a new `crates/ml-alpha/src/rl/isv_slots.rs` file.
- Phase B: each head reads its hyperparameters from ISV, not constants.
- Phase C.2 (Bellman loss): reads `RL_GAMMA_INDEX`, `RL_TARGET_TAU_INDEX` from device ISV.
- Phase D.2 (clipped surrogate): reads `RL_PPO_CLIP_INDEX`, `RL_ENTROPY_COEF_INDEX`.
- Phase E: training loop reads `RL_N_ROLLOUT_STEPS_INDEX`, `RL_PER_ALPHA_INDEX`.
- Phase F: reward scale derives from `RL_REWARD_SCALE_INDEX`, not from a hardcoded `/ 10.0` divisor.
- New Phase C.6 / D.6 sub-task per controller: implement + test the controller kernel itself.

The bootstrap values shown in the "Bootstrap" column above are what the
first-observation-after-sentinel-zero controller emits, not hardcoded — they're
written by the controller on first emit, then adapted by subsequent emits.

## Falsifiable gates

| Gate | Metric | Pass | Phase |
|------|--------|------|-------|
| G1 | Workspace builds | `cargo check --workspace` clean | A |
| G2 | Encoder gradient flows from each head independently | Synthetic forward-backward test for each head | B |
| G3 | DQN converges on toy bandit | Q(state, action_correct) > Q(state, action_wrong) after 10k steps | C |
| G4 | PPO converges on toy bandit | π(action_correct | state) > 0.9 after 10k steps | D |
| G5 | Joint trainer runs end-to-end on synthetic LobSim | 1 epoch completes without NaN/Inf in any loss | E |
| G6 | Reward shaping calibrated | Mean reward per trade closes positive on bull synthetic data | F |
| G7 | Argo smoke runs to completion | `argo get` shows Succeeded | G |
| **G8** | **Backtest gate** | **profit_factor > 1.0 on 2M-event sweep** | **H** |

G8 is the trading-readiness gate that the prior architecture failed. Pass = the integrated trainer produces a profitable model; fail = even RL can't extract enough signal from this data and we re-scope.

---

## Files touched

| File | Action | Responsibility |
|------|--------|----------------|
| `crates/ml-alpha/src/rl/mod.rs` | Create | Module entry point |
| `crates/ml-alpha/src/rl/common.rs` | Create | `Transition` struct, shared types |
| `crates/ml-alpha/src/rl/replay.rs` | Create | PER replay buffer (off-policy, DQN) |
| `crates/ml-alpha/src/rl/rollout.rs` | Create | On-policy rollout buffer (PPO) |
| `crates/ml-alpha/src/rl/dqn.rs` | Create | C51 Q-head + Bellman loss + target net |
| `crates/ml-alpha/src/rl/ppo.rs` | Create | π policy head + V value head + clipped surrogate |
| `crates/ml-alpha/src/rl/action.rs` | Create | 9-action grid definitions + Thompson sampler |
| `crates/ml-alpha/src/rl/reward.rs` | Create | Per-trade realized PnL extraction from LobSim |
| `crates/ml-alpha/cuda/dqn_distributional_q.cu` | Create | C51 forward+backward kernel |
| `crates/ml-alpha/cuda/ppo_clipped_surrogate.cu` | Create | Clipped surrogate forward+backward |
| `crates/ml-alpha/src/trainer/integrated.rs` | Create | The new integrated trainer (replaces `perception.rs::step`) |
| `crates/ml-alpha/src/trainer/perception.rs` | Modify | Expose `forward_encoder() -> h_t` for the integrated trainer |
| `crates/ml-alpha/src/lib.rs` | Modify | Register `rl` module |
| `crates/ml-alpha/Cargo.toml` | Modify | Add ml-backtesting dep (for LobSim integration) |
| `crates/ml-alpha/examples/alpha_rl_train.rs` | Create | New CLI binary, replaces `alpha_train` as canonical entry point |
| `crates/ml-alpha/tests/integrated_trainer.rs` | Create | End-to-end smoke test (synthetic data) |
| `infra/k8s/argo/alpha-rl-template.yaml` | Create | Argo workflow template |
| `scripts/argo-alpha-rl.sh` | Create | Dispatcher script |

The existing `alpha_train` binary stays (for diagnostic / parity runs). The new `alpha_rl_train` becomes the canonical production trainer.

---

## Phase A — Module skeleton (1 day)

### A.1 Create the rl module structure

**Files:**
- Create: `crates/ml-alpha/src/rl/mod.rs`
- Create: `crates/ml-alpha/src/rl/common.rs`
- Modify: `crates/ml-alpha/src/lib.rs` (add `pub mod rl;`)

### A.2 Define Transition struct

Append to `crates/ml-alpha/src/rl/common.rs`:

```rust
//! Shared types for the integrated RL trainer.

use cudarc::driver::CudaSlice;

/// Single (state, action, reward, next_state) transition.
/// Stored in both the off-policy replay buffer (DQN) and the on-policy
/// rollout buffer (PPO). The encoder representation `h_t` is stored
/// directly to avoid re-running the encoder during loss computation.
#[derive(Clone, Debug)]
pub struct Transition {
    pub h_t: CudaSlice<f32>,           // [B, HIDDEN_DIM=128]
    pub action: u32,                    // 0..9 (see action.rs)
    pub reward: f32,                    // Per-trade realized PnL on close, 0 between closes
    pub next_h_t: CudaSlice<f32>,       // [B, HIDDEN_DIM]
    pub done: bool,                     // True at trade close or end-of-day
    pub log_pi_old: f32,                // PPO importance ratio denominator
    pub q_value_old: [f32; 9],          // DQN bootstrap value at sample time
}
```

### A.3 Cargo dep update

Modify `crates/ml-alpha/Cargo.toml`: add `ml-backtesting = { path = "../ml-backtesting" }` to `[dependencies]`.

(Currently the dependency is reversed; flipping it requires breaking the cycle. If circular, the integrated trainer goes in ml-backtesting instead — flag at implementation time.)

### A.4 Validate

```bash
SQLX_OFFLINE=true cargo check -p ml-alpha 2>&1 | tail -5
```

Expected: clean build.

### A.5 Commit

```bash
git add crates/ml-alpha/src/lib.rs crates/ml-alpha/src/rl/ crates/ml-alpha/Cargo.toml
git commit -m "feat(rl): module skeleton for integrated RL trainer (Phase A)"
```

---

## Phase B — Encoder gradient hookup (2 days)

### B.1 Expose forward_encoder

Modify `crates/ml-alpha/src/trainer/perception.rs`: add public method `pub fn forward_encoder(&mut self, features: &[Mbp10RawInput]) -> CudaSlice<f32>` that runs the encoder forward and returns `h_t [B, 128]` WITHOUT computing the BCE head. The integrated trainer calls this, then dispatches its 5 heads on `h_t`.

### B.2 Test gradient flow

Write a synthetic forward-backward test in `crates/ml-alpha/tests/encoder_gradient.rs`:

```rust
#[test]
fn encoder_gradient_flows_from_synthetic_head() {
    let dev = ...;
    let mut trainer = PerceptionTrainer::new(...);
    let features = synth_features();
    let h_t = trainer.forward_encoder(&features);
    let loss = synth_loss(&h_t);  // any scalar loss on h_t
    loss.backward();
    // Assert: encoder weights moved (compare before/after)
    let w_before = trainer.encoder.w_in.snapshot();
    trainer.opt_step();
    let w_after = trainer.encoder.w_in.snapshot();
    assert!(diff(w_before, w_after) > 1e-6, "encoder weights must update");
}
```

### B.3 Commit
```
git commit -m "feat(rl): expose forward_encoder; gradient-flow test (Phase B)"
```

---

## Phase C — DQN head + replay + Bellman (5 days)

This is the largest phase. Breaks into 5 sub-tasks.

### C.1 C51 distributional Q-head structure

Create `crates/ml-alpha/src/rl/dqn.rs`:
- Struct `DqnHead` with `w_q: [9, AUX_HIDDEN=128]`, `b_q: [9]`, `support: [21]`, `target_w_q`, `target_b_q` (target network)
- `pub fn forward(&self, h_t: &CudaSlice<f32>) -> CudaSlice<f32>` returns `[B, 9, 21]` (logits per atom per action)
- Xavier init scaled by 0.1 per `pearl_scoped_init_seed_for_reproducibility`

### C.2 Bellman loss + target network

Create `crates/ml-alpha/cuda/dqn_distributional_q.cu`:
- Forward: sigmoid → categorical distribution per action
- Backward: TD loss = -log p(target_atom) where target_atom is derived from r + γ × max_a' Q_target(s', a')
- Target net soft update: `target = (1-τ) × target + τ × current`

### C.3 PER replay buffer

Create `crates/ml-alpha/src/rl/replay.rs`:
- `ReplayBuffer { capacity, transitions: Vec<Transition>, priorities: Vec<f32>, sum_tree: SumTree }`
- `pub fn push(&mut self, t: Transition, td_error: f32)`
- `pub fn sample(&mut self, batch_size: usize) -> Vec<(Transition, f32 /* IS weight */, usize /* idx */)>`
- `pub fn update_priorities(&mut self, idxs: &[usize], td_errors: &[f32])`

### C.4 Toy bandit test

Test on a toy bandit problem in `tests/dqn_toy.rs`: synthetic state where action 5 always gets reward +1, other actions get -1. After 10k training steps, assert `Q(s, 5) > Q(s, *)` for any state s. Validates the entire DQN pipeline before integrating.

### C.5 Commit
```
git commit -m "feat(rl): C51 distributional Q-head + PER replay + Bellman (Phase C)"
```

---

## Phase D — PPO π/V heads + clipped surrogate (5 days)

Mirrors Phase C structure for the actor side.

### D.1 Policy + value heads

Create `crates/ml-alpha/src/rl/ppo.rs`:
- `PolicyHead { w_pi: [9, 128], b_pi: [9] }` → softmax over 9 actions
- `ValueHead { w_v: [1, 128], b_v: [1] }` → scalar V(s)

### D.2 Clipped surrogate loss

Create `crates/ml-alpha/cuda/ppo_clipped_surrogate.cu`:
- Forward: `surr = min(r × A, clip(r, 1-ε, 1+ε) × A)`, ε = 0.2 per PPO default
- Backward: gradient of surrogate w.r.t. logits
- Value loss: MSE between V_pred and V_target (Q-expectation under current π)
- Entropy bonus: -coef × H(π) to encourage exploration; coef anneals from 0.01 → 0.001 over training

### D.3 Rollout buffer

Create `crates/ml-alpha/src/rl/rollout.rs`:
- `RolloutBuffer { capacity, transitions: Vec<Transition>, advantages: Vec<f32>, returns: Vec<f32> }`
- `pub fn push(&mut self, t: Transition)` — appends
- `pub fn compute_advantages(&mut self, v_head: &ValueHead, q_head: &DqnHead, gamma: f32, lambda: f32)` — uses Q-head's expected value as the baseline (replaces GAE-λ)
- `pub fn finalize_returns(&mut self)` — done-aware cumulative return

### D.4 Toy bandit test

Same toy bandit as C.4: after 10k steps, π(action=5|s) > 0.9.

### D.5 Commit
```
git commit -m "feat(rl): PPO π/V heads + clipped surrogate + rollout (Phase D)"
```

---

## Phase E — LobSim integration + training loop (3 days)

### E.1 LobSim adapter

Create `crates/ml-alpha/src/rl/reward.rs`:
- Wraps `ml_backtesting::sim::LobSimCuda`
- `pub fn step(&mut self, action: u32, h_t: &CudaSlice<f32>) -> (next_h_t, reward, done)`
- Reward extraction: read `realized_pnl_usd` delta on trade close

### E.2 Integrated trainer

Create `crates/ml-alpha/src/trainer/integrated.rs`:
- `pub struct IntegratedTrainer { encoder, bce_head, q_head, pi_head, v_head, aux_head, replay, rollout, lobsim, loss_balance_controller }`
- `pub fn step(&mut self) -> Result<StepStats>` runs the training loop above (one event per call)
- All 5 heads' losses combined, single backward into encoder

### E.3 End-to-end synthetic smoke

Test in `crates/ml-alpha/tests/integrated_trainer.rs`:
- 1000 events of synthetic LobSim data (deterministic price walk)
- Run 1 epoch through integrated trainer
- Assert: no NaN/Inf in any loss, all 5 heads' gradients flowed (compare before/after weights), replay buffer non-empty

### E.4 Commit
```
git commit -m "feat(rl): LobSim integration + integrated trainer end-to-end (Phase E)"
```

---

## Phase F — Reward shaping calibration (2 days)

### F.1 Reward components

Implement in `reward.rs`:
- Base: `realized_pnl_usd` on trade close
- Penalty: `fees_usd` (already subtracted in summary)
- Penalty: `exit_path_cost` (slippage at close, computed from LobSim)
- Bonus (optional): `exposure_bonus = -|position_lots| × 0.0001 × dt_ns` (small penalty for sitting in inventory)

### F.2 Reward scale calibration

Calibrate the reward magnitude to match the V-head's expected range. Typical PnL per trade in USD ≈ 0.1-100. Scale: `reward_scaled = reward / 10.0` so V can fit in [-10, +10] which matches the C51 atom support `[-1, +1]` after standardization.

### F.3 Sign-check test

On a synthetic bull-market stream where price monotonically rises, train for 1k events. Assert: mean reward per closed trade > 0 when the policy converges to "long all the way."

### F.4 Commit

```
git commit -m "feat(rl): reward shaping + calibration (Phase F)"
```

---

## Phase G — Argo workflow + dispatcher (1 day)

### G.1 Template

Create `infra/k8s/argo/alpha-rl-template.yaml`:
- Same structure as `alpha-perception-template.yaml`
- New params: `gamma` (0.99), `n-rollout-steps` (2048), `replay-size` (100000), `n-ppo-epochs` (4), `target-update-tau` (0.005), `target-update-freq` (100)
- Same `instrument-mode` and `multi-resolution` defaults as the perception template (front-month, 1:32)

### G.2 Dispatcher script

Create `scripts/argo-alpha-rl.sh`: mirrors `argo-alpha-perception.sh` with the new params.

### G.3 Commit

```
git commit -m "feat(rl): Argo template + dispatcher script (Phase G)"
```

---

## Phase H — Backtest gate (1 day + 1.5h smoke)

### H.1 Push + apply template + dispatch

```bash
git push origin ml-alpha-phase-a
kubectl apply -n foxhunt -f infra/k8s/argo/alpha-rl-template.yaml
./scripts/argo-alpha-rl.sh --instrument-mode front-month
```

### H.2 Pull results + backtest

After the workflow succeeds, pull the trained checkpoint:
- `/feature-cache/alpha-rl-runs/$SHA/integrated_best_h1000.bin`

Update `config/ml/sweep_smoke_perhoriz_cfc.yaml` to point at the new checkpoint, dispatch `argo-lob-sweep.sh`.

### H.3 Evaluate the gate

| Metric | Failure threshold | Pass |
|--------|--------------------|------|
| profit_factor | < 0.5 → re-architect needed | > 1.0 |
| sharpe_ann | < -3 → re-architect needed | > 0.5 |
| outcome_by_entry_conv | mono-anti-cal pattern persists | Higher conviction = higher mean_pnl |

### H.4 Document outcome

- Pass: write `pearl_integrated_rl_works.md`, plan production deployment
- Fail: document specific failure mode, decide whether to re-scope (different action space, different reward, different action grid)

---

## Self-review checklist

**Spec coverage:**
- [x] Encoder gradient from all 5 heads (Phase B)
- [x] DQN distributional Q + PER replay + Bellman (Phase C)
- [x] PPO π/V + clipped surrogate + rollout (Phase D)
- [x] LobSim integration with per-trade reward (Phase E)
- [x] Reward shaping (Phase F)
- [x] Argo workflows (Phase G)
- [x] Backtest gate G8 (Phase H)

**Risk analysis:**
- **Highest risk**: encoder gradient instability when 5 heads compete for shape. Mitigation: loss-balance controller (already in memory pearls) adaptively scales λ_k.
- **Second highest**: replay buffer GPU memory (100k transitions × 128 floats h_t × 4 bytes × 2 (h_t and next_h_t) ≈ 100 MB). Fits on L40S. PER sum-tree is CPU. Tradeoff.
- **Third**: PPO + DQN can diverge if both heads disagree strongly. Mitigation: monitor `argmax_Q vs argmax_π` agreement rate; if < 50%, raise alarm and pause training.

**Greenfield commitment:** No backward-compat with old DQN in `crates/ml/src/trainers/dqn/`. That code stays for the old policy (different action space). The new integrated trainer is ml-alpha-specific.

**Project policy compliance:**
- `feedback_no_partial_refactor`: each phase is atomic; subagents land complete files per phase.
- `feedback_no_feature_flags`: no `if use_rl` toggle — `alpha_rl_train` is the new canonical entry.
- `feedback_no_htod_htoh_only_mapped_pinned`: all CPU↔GPU via mapped-pinned (per pre-existing infra).
- `feedback_no_stubs`: no return-zero Q-heads or placeholder PPO loss.
- `pearl_thompson_for_distributional_action_selection`: Thompson at rollout, argmax only at eval.
- `pearl_c51_thompson_closed_phase_e3_gap`: C51 + Thompson is the proven combo.
- `pearl_loss_balance_controller`: existing controller handles λ_k for 5 heads.
- `pearl_cooperative_staging_eliminates_redundant_reads`: encoder forward once, all 5 heads share `h_t`.

**Falsifiability:** Each phase has a gate. G3/G4 prove DQN/PPO work in isolation. G5/G6 prove integration. G8 is the actual trading-readiness gate that the prior architecture failed.

**Calendar estimate:** ~3 weeks (16 dev days + 1.5h Argo smoke). With subagent parallelism in Phases C and D (independent algorithms), can compress to ~2.5 weeks.
