docs(plans): integrated RL trainer (DQN + PPO + BCE + aux) plan

Single integrated trainer where 5 loss heads (BCE direction, C51 distributional Q, categorical π, scalar V, aux prof+size) sit on a shared Mamba2+CfC encoder. Joint training with adaptive loss-balance λ weights via the existing pearl_loss_balance_controller infrastructure. Discrete 9-action grid shared between Q and π heads; reward = per-trade realized PnL from LobSimCuda. Designed in response to lob-backtest-sweep-jpdhg result (PF=0.24, sharpe=-9.72, mono-anti-cal in conviction→PnL): the encoder learns directional AUC but never sees PnL-aware gradient because stop_grad_aux_to_encoder blocks the aux head's gradient. RL training fixes this by making the encoder optimize per-trade realized PnL via Q-head Bellman + π-head clipped surrogate. Every hyperparameter (γ, τ, PPO clip ε, entropy coef, rollout steps, PER α, reward scale, loss-balance λs) is ISV-driven per pearl_controller_anchors_isv_driven and feedback_isv_for_adaptive_bounds. 12 new ISV slots (400-411) + 12 reused existing slots + 8 new controller kernels following the Wiener-α + first-observation-bootstrap pattern. 10 phases (A-H, ~3 weeks), falsifiable gate G8 = profit_factor > 1.0 on 2M-event backtest sweep.
2026-05-22 22:07:45 +02:00
parent dab4794fb0
commit 57de1a8b4e
1 changed files with 638 additions and 0 deletions
--- a/docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md
+++ b/docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md
@@ -0,0 +1,638 @@
+# Integrated RL Trainer: BCE + DQN + PPO + Aux on a Shared Encoder
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task.
+
+**Goal:** Replace the pure-supervised BCE alpha-perception trainer with a single integrated trainer where the Mamba2+CfC encoder receives gradient from FIVE complementary loss heads — BCE direction, DQN distributional Q, PPO policy + value, and auxiliary prof/size — so the encoder learns features useful for **realized trading PnL**, not just price-direction prediction.
+
+**Architecture:** One shared encoder produces `h_t ∈ R^128`. Five heads sit on top: (1) BCE direction (warm-start signal), (2) C51 distributional Q-head (off-policy critic), (3) categorical π policy head (on-policy actor), (4) scalar V value head, (5) prof+size aux head (per-trade calibration). Total loss is a weighted sum with adaptive λ via the existing loss-balance controller. Action space is shared 9-element discrete grid. Reward is per-trade realized PnL from the GPU LobSim. Replay buffer (PER) feeds Q-head off-policy; rollout buffer feeds π+V on-policy; both buffers fill from the SAME training-loop transitions.
+
+**Tech Stack:** Rust 1.85+, Tokio 1.40, CUDA 12.4 L40S/H100, cudarc 0.19. Existing components reused: Mamba2 SSM + CfC encoder (ml-alpha/src/cfc/), aux trunk + heads (ml-alpha/src/aux_heads.rs, aux_trunk.rs), LobSim (ml-backtesting/src/sim/), perception trainer skeleton (ml-alpha/src/trainer/perception.rs). New components: DQN Q-head + replay, PPO π/V heads + rollout, action-grid kernels, reward-shaping kernels, loss-balance controller wiring for 5 heads.
+
+---
+
+## Why this architecture
+
+The user's question motivating this plan: "the model should run inference during training to learn from its mistakes — does it?" Answer: NO, the current PerceptionTrainer optimizes only BCE on offline `sign(p[t+K] - p[t])` labels. It never sees its own trade outcomes, fees, slippage, or PnL.
+
+Empirical evidence: alpha-perception-jvv7d trained to auc_h1000=0.576 (decent directional signal). Backtest `lob-backtest-sweep-jpdhg` on that checkpoint gave:
+- profit_factor = 0.24
+- sharpe_ann = -9.72
+- total_pnl_usd = -$1.29M
+- win-rate correlates with conviction (16% → 27%) BUT mean_pnl gets WORSE with conviction
+
+The decoupling is structural: the encoder optimizes directional AUC but never sees the cost-aware PnL gradient. The fix is to give the encoder a PnL-aware gradient via RL, while keeping the BCE signal for stable warm-start.
+
+### Why both DQN AND PPO (not OR)
+
+DQN-only weaknesses: ε-greedy exploration wastes 10-30% of compute on random actions; struggles when action distinguishability is subtle.
+
+PPO-only weaknesses: high variance from MC returns; on-policy = throws away old data; needs GAE-λ tricks.
+
+Combined (SAC-style actor-critic): Q-head provides the LOW-VARIANCE advantage signal to π — no need for GAE-λ. π-head provides STOCHASTIC exploration informed by current policy beliefs — no ε-greedy. The two cross-validate (Q-argmax should agree with π-mode at convergence). One encoder gets BOTH gradient streams plus BCE + aux supervision.
+
+---
+
+## Architecture diagram
+
+```
+[MBP-10 raw features, 32 ticks]
+              │
+              ▼
+   [snap_feature_assemble kernel]
+              │  feature vector [B, 32, 40]
+              ▼
+   [Mamba2 SSM, state_dim=16]
+              │
+              ▼
+   [CfC continuous-time recurrence, hidden=128]
+              │  h_t [B, 128]
+              ▼
+   ┌──────────┼──────────┬──────────┬──────────┐
+   │          │          │          │          │
+   ▼          ▼          ▼          ▼          ▼
+[BCE dir] [C51 Q]    [π policy] [V value] [Aux prof+size]
+[B, 3]    [B, 9, 21] [B, 9]     [B, 1]    [B, 2, 3, 2]
+ │         │          │          │           │
+sigmoid   atoms over  softmax    raw         sigmoid+raw
+ │         [Q_min,    │          │           │
+ │         Q_max]    │          │           │
+ │         │          │          │           │
+L_BCE     L_Q (Bellman)  L_π    L_V (MSE)   L_aux
+direction TD on s,a  (clipped   to Q-expect (BCE_prof
+labels    target net surr,A_Q) ation        + Huber_size)
+
+           ─────── weighted sum ───────
+                   L_total
+                      │
+                      ▼ backward
+              encoder ∇ = ∇ × λ_total
+```
+
+### Action space (9 discrete actions, shared by Q and π)
+
+| idx | Action | Effect |
+|-----|--------|--------|
+| 0 | Short-large | Open short, size = 2× base |
+| 1 | Short-small | Open short, size = 1× base |
+| 2 | Hold | No change to position |
+| 3 | Flat-from-long | Close long position if any |
+| 4 | Flat-from-short | Close short position if any |
+| 5 | Long-small | Open long, size = 1× base |
+| 6 | Long-large | Open long, size = 2× base |
+| 7 | Trail-tighten | Reduce trailing stop distance by 25% |
+| 8 | Trail-loosen | Increase trailing stop distance by 25% |
+
+Note: matches existing DQN action grid in `crates/ml/src/cuda_pipeline/sp5_isv_slots.rs` for parity. Trail-tighten/loosen are stop-management actions; the other 7 are position-management.
+
+### Reward (per-trade realized PnL on close)
+
+`reward = realized_pnl_usd - fees_usd - exit_path_cost`
+
+Reward fires at trade CLOSE (when position goes to 0). Between closes, reward is 0. Per-trade signal matches what we actually optimize (PF, sharpe). Sparser than per-tick but unambiguous credit assignment.
+
+For TD(λ) bootstrap of intermediate values, the V-head learns the discounted-future-reward estimate; Q-head learns per-action discounted return. λ_discount = 0.99 per HFT convention (matches old DQN gamma).
+
+### Loss formulation
+
+```
+L_total = λ_bce × L_BCE(dir_logits, dir_labels)
+        + λ_q   × L_Q(Q(s_t, a_t), r_t + γ × max_a' Q_target(s_{t+1}, a'))
+        + λ_π   × L_π(π(s_t), A_Q(s_t, a_t) = Q(s_t, a_t) - V(s_t))
+        + λ_v   × L_V(V(s_t), Σ_a π(a|s_t) × Q(s_t, a))
+        + λ_aux × (L_aux_prof + L_aux_size)
+```
+
+Where:
+- L_Q is Huber loss on the distributional TD error (per pearl_per_branch_c51_atom_span)
+- L_π is the PPO clipped surrogate: `min(r_t × A, clip(r_t, 1-ε, 1+ε) × A)`, r_t = π_new/π_old
+- L_V is MSE between V prediction and the expected value under the policy
+- L_aux uses the existing aux_loss.cu kernels (BCE prof + masked Huber size)
+- λ_k are learned per-head via the existing loss-balance controller (`pearl_loss_balance_controller`)
+
+---
+
+## State / observation
+
+State at step t = encoder output `h_t ∈ R^128`. The encoder reads:
+- Last 32 raw MBP-10 ticks (default `single_scale_32` from the just-reverted multi-resolution config)
+- Regime context (6 EMAs cascade) per existing loader
+- Δt Fourier features per existing snap_feature kernel
+- (Optional) position state vector: `[current_position_lots, current_avg_entry_px, hold_duration_ns]` concatenated to h_t before head dispatch. This lets the policy see its own state.
+
+---
+
+## Training loop
+
+```rust
+loop {
+    let batch = loader.next_sequence();           // existing
+    let h_t = encoder.forward(batch.features);    // existing
+
+    // Action selection during rollout: Thompson over Q distribution
+    // (pearl_thompson_for_distributional_action_selection)
+    let action = sample_thompson(q_head.forward(h_t));
+
+    // Sim step: submit action, advance one event, get reward
+    let (next_state, reward, done) = lobsim.step(action);
+
+    // Store transition in BOTH buffers
+    let transition = Transition {
+        h_t,
+        action,
+        reward,
+        next_h_t: next_state.encoded(),
+        done,
+        // Log probability for PPO importance ratio
+        log_pi: pi_head.log_prob(action),
+    };
+    replay_buffer.push(transition.clone());  // off-policy for Q
+    rollout_buffer.push(transition);          // on-policy for π+V
+
+    // Off-policy DQN update (every step, from PER replay)
+    if replay_buffer.len() > min_warmup {
+        let dqn_batch = replay_buffer.sample_per(batch_size);
+        let l_q = dqn_loss(dqn_batch, q_head, q_target_head);
+    }
+
+    // On-policy PPO update (at end of rollout)
+    if rollout_buffer.full() {
+        for _ in 0..n_ppo_epochs {
+            let l_pi = ppo_clipped_surrogate(rollout_buffer, pi_head, q_head);
+            let l_v  = value_mse(rollout_buffer, v_head, q_head, pi_head);
+        }
+        rollout_buffer.clear();
+    }
+
+    // BCE + aux losses (every step)
+    let l_bce = bce_dir(bce_head.forward(h_t), batch.dir_labels);
+    let l_aux = aux_loss(aux_head.forward(h_t), batch.aux_labels);
+
+    // Loss-balance controller updates λ_k
+    let lambdas = loss_balance_controller.update_and_get();
+
+    let l_total = lambdas.bce * l_bce
+                + lambdas.q   * l_q
+                + lambdas.pi  * l_pi
+                + lambdas.v   * l_v
+                + lambdas.aux * l_aux;
+    l_total.backward();
+
+    // Periodic target-network soft update
+    if step % target_update_freq == 0 {
+        q_target_head.soft_update(q_head, tau = 0.005);
+    }
+}
+```
+
+Key design points:
+1. **Both buffers fill from the SAME transitions** — no separate rollouts. PPO uses fresh data from the most-recent rollout window; DQN uses ANY data from the larger replay buffer.
+2. **One forward pass per step** computes all 5 heads from `h_t`.
+3. **One backward pass** combines all gradients into the encoder.
+4. **Target network** is for Q only (PPO doesn't need one). Soft updates per `pearl_per_branch_c51_atom_span` infrastructure (already exists).
+5. **Thompson action selection** at rollout time (pearl-mandated); argmax-Q only at eval time.
+
+---
+
+## ISV-driven adaptive control (mandatory per project memory)
+
+Per `feedback_isv_for_adaptive_bounds` and `pearl_controller_anchors_isv_driven`,
+**every** hyperparameter / threshold / cap in this trainer MUST be sourced from an
+ISV slot updated by a per-step controller kernel. No `const f32 = 0.99` for γ, no
+`const f32 = 0.2` for PPO clip ε, no `const f32 = 0.005` for target-net τ.
+
+The existing `crates/ml/src/cuda_pipeline/sp5_isv_slots.rs` ISV bus has 56+ slots
+(`ISV_TOTAL_DIM = 383`). The integrated RL trainer adds a new block of slots and
+reuses the existing controller infrastructure:
+
+### New ISV slots (allocate at top of ml-alpha-specific block, e.g. 400-450)
+
+| Slot | Name | Role | Controller (producer) | Bootstrap |
+|------|------|------|----------------------|-----------|
+| 400 | `RL_GAMMA_INDEX` | Discount factor (Q + V + PPO advantage) | Trade-horizon EMA controller (input: mean trade duration / 60s anchor) | Pearl-A first observation; sentinel 0 → 0.99 init |
+| 401 | `RL_TARGET_TAU_INDEX` | Target-net soft-update fraction | Q-divergence EMA controller (input: ‖Q_current - Q_target‖ EMA) | Pearl-A; sentinel 0 → 0.005 init |
+| 402 | `RL_PPO_CLIP_INDEX` | Clipped surrogate ε | Policy-divergence controller (input: KL(π_new‖π_old) EMA, target = 0.01) | Pearl-A; sentinel 0 → 0.2 init |
+| 403 | `RL_ENTROPY_COEF_INDEX` | Entropy bonus weight | Action-entropy deficit controller (target = 70% × ln(9)) per `pearl_per_branch_noisy_sigma` pattern | Pearl-A; sentinel 0 → 0.01 init |
+| 404 | `RL_N_ROLLOUT_STEPS_INDEX` | Rollout buffer flush size | Advantage-variance controller (input: var(A_Q) / |A_Q| ratio, target = 0.1) | Pearl-A; sentinel 0 → 2048 init |
+| 405 | `RL_PER_ALPHA_INDEX` | PER priority exponent | TD-error distribution controller (input: TD-error kurtosis EMA) | Pearl-A; sentinel 0 → 0.6 init |
+| 406 | `RL_REWARD_SCALE_INDEX` | Per-trade reward standardization | PnL magnitude EMA (input: mean |realized_pnl_usd| over last 100 closes) | Pearl-A; sentinel 0 → 1.0 init |
+| 407 | `RL_Q_ARG_VS_PI_AGREE_INDEX` | Agreement rate Q-argmax vs π-mode | Diagnostic-only (read by alarm; not controlled) | EMA-tracked, no init magic |
+| 408-411 | `RL_LOSS_LAMBDA_BCE/Q/PI/V_INDEX` (4 slots) | Per-head loss weight (BCE, Q, π, V; aux uses existing slot) | Reuse `pearl_loss_balance_controller` Wiener-α + signal-modulated target controller | Pearl-A from `pearl_first_observation_bootstrap`; sentinel 0 → equal-weight 1/5 init |
+
+The loss-balance controller (existing, `pearl_loss_balance_controller`) ALREADY
+handles per-loss weight balancing for the perception trainer's 3 horizons. We
+EXTEND it to 5 heads (BCE, Q, π, V, aux) by allocating 4 new λ slots (aux λ
+already exists per current trainer). The controller's two-layer design
+(signal-modulated target × Wiener-α adaptation) covers our need; no new
+algorithm.
+
+### Reused existing controllers (no new infrastructure)
+
+| ISV slot | Reused for |
+|----------|-----------|
+| `EVAL_THOMPSON_TEMP_INDEX` (existing) | Thompson temperature at rollout action selection |
+| `KELLY_F_SMOOTH_INDEX = 280` (existing) | Kelly fraction smoothing — fed to action sizing in `action.rs` |
+| `CONVICTION_SMOOTH_INDEX = 281` (existing) | Conviction EMA for outcome bucketing diagnostic |
+| `Q_VAR_PER_BRANCH_BASE = 222` (existing) | Per-branch Q-variance EMA — input to several RL controllers |
+| `FLATNESS_BASE = 206` (existing) | Per-branch flatness signal — input to budget controllers |
+| `NOISY_SIGMA_BASE = 210` (existing) | NoisyNet σ — REUSED for stochastic exploration on π head (instead of new entropy bonus mechanism) |
+| `BUDGET_C51_BASE = 190` (existing) | C51 loss budget per branch — reused for our Q-head |
+| `LB_C51_ACTIVE_BASE = 317` (existing) | Loss-balance C51 activation flag — extend to gate Q-head warmup |
+| `PNL_TOTAL_INDEX = 286`, `PNL_MEAN_INDEX = 287`, `PNL_VAR_INDEX = 288`, `PNL_MAX_DD_INDEX = 289` (existing) | PnL telemetry; INPUT to reward-scale controller (RL_REWARD_SCALE_INDEX = 406) |
+| `TRAINING_SHARPE_EMA_INDEX = 294`, `MAX_DD_EMA_INDEX = 295` (existing) | Training Sharpe/DD EMAs — INPUT to PnL magnitude controller |
+| `REWARD_*_WEIGHT_INDEX` (340-348 existing, SP11 reward subsystem) | DIRECTLY reused as reward shaping component weights; no new reward kernel |
+| `CURIOSITY_PRESSURE_INDEX = 346` (existing) | Direct reuse for exploration bonus on top of policy entropy |
+
+### Controller kernels to add (new)
+
+| Kernel | Reads (ISV inputs) | Writes (ISV outputs) | Notes |
+|--------|-------------------|---------------------|-------|
+| `rl_gamma_controller_kernel.cu` | `PNL_MEAN_INDEX`, `KELLY_F_SMOOTH_INDEX`, mean_trade_duration_ns | `RL_GAMMA_INDEX` (400) | γ derives from "what fraction of trades close within decision horizon" |
+| `rl_target_tau_controller_kernel.cu` | Q-divergence EMA (computed in-kernel from current vs target weights) | `RL_TARGET_TAU_INDEX` (401) | τ adapts: faster updates when divergence high |
+| `rl_ppo_clip_controller_kernel.cu` | KL(π_new‖π_old) EMA | `RL_PPO_CLIP_INDEX` (402) | ε adapts: tighter clip when KL drifts high |
+| `rl_entropy_coef_controller_kernel.cu` | Action-entropy deficit (target = 0.7×ln(9)) | `RL_ENTROPY_COEF_INDEX` (403) | Wiener-α tracking per `pearl_wiener_alpha_floor_for_nonstationary` (α-floor 0.4) |
+| `rl_rollout_steps_controller_kernel.cu` | var(A_Q) / \|A_Q\| ratio EMA | `RL_N_ROLLOUT_STEPS_INDEX` (404) | Longer rollout when advantage estimates are noisy |
+| `rl_per_alpha_controller_kernel.cu` | TD-error kurtosis EMA | `RL_PER_ALPHA_INDEX` (405) | Higher α when TD-errors heavy-tailed |
+| `rl_reward_scale_controller_kernel.cu` | Mean \|realized_pnl_usd\| EMA over last 100 closes (`PNL_MEAN_INDEX`) | `RL_REWARD_SCALE_INDEX` (406) | Standardize rewards so V-head's expected range matches C51 atom support |
+| `rl_q_pi_agreement_kernel.cu` | argmax(Q) vs mode(π) per step | `RL_Q_ARG_VS_PI_AGREE_INDEX` (407) | Diagnostic; not consumed by loss; alarms when < 0.5 |
+
+All 8 new controllers follow the existing pattern:
+1. Wiener-α-with-floor (per `pearl_wiener_alpha_floor_for_nonstationary`, α-floor = 0.4)
+2. First-observation bootstrap from sentinel 0 (per `pearl_first_observation_bootstrap`)
+3. State-reset registry entry + matching `reset_named_state` arm (per `feedback_registry_entries_need_dispatch_arms`)
+4. Pearl-D recovery cascade: if controller output drifts to invalid range, fold-boundary reset re-bootstraps
+
+### Constants permitted (true structural constants only)
+
+These remain as `const f32` because they're STRUCTURAL parameters of the algorithm,
+not adaptive bounds:
+
+- `N_ACTIONS = 9` (action grid cardinality — wired into kernel `#define`s)
+- `Q_N_ATOMS = 21` (C51 atom count — kernel layout)
+- `HIDDEN_DIM = 128` (encoder output dim — already a workspace const)
+- `PPO_N_EPOCHS = 4` per rollout (algorithm hyperparameter; no signal to drive it adaptively)
+
+Everything else is ISV.
+
+### Plan task changes for ISV compliance
+
+- Phase A.2: add the 12 new ISV slot constants (400-411) to a new `crates/ml-alpha/src/rl/isv_slots.rs` file.
+- Phase B: each head reads its hyperparameters from ISV, not constants.
+- Phase C.2 (Bellman loss): reads `RL_GAMMA_INDEX`, `RL_TARGET_TAU_INDEX` from device ISV.
+- Phase D.2 (clipped surrogate): reads `RL_PPO_CLIP_INDEX`, `RL_ENTROPY_COEF_INDEX`.
+- Phase E: training loop reads `RL_N_ROLLOUT_STEPS_INDEX`, `RL_PER_ALPHA_INDEX`.
+- Phase F: reward scale derives from `RL_REWARD_SCALE_INDEX`, not from a hardcoded `/ 10.0` divisor.
+- New Phase C.6 / D.6 sub-task per controller: implement + test the controller kernel itself.
+
+The bootstrap values shown in the "Bootstrap" column above are what the
+first-observation-after-sentinel-zero controller emits, not hardcoded — they're
+written by the controller on first emit, then adapted by subsequent emits.
+
+## Falsifiable gates
+
+| Gate | Metric | Pass | Phase |
+|------|--------|------|-------|
+| G1 | Workspace builds | `cargo check --workspace` clean | A |
+| G2 | Encoder gradient flows from each head independently | Synthetic forward-backward test for each head | B |
+| G3 | DQN converges on toy bandit | Q(state, action_correct) > Q(state, action_wrong) after 10k steps | C |
+| G4 | PPO converges on toy bandit | π(action_correct | state) > 0.9 after 10k steps | D |
+| G5 | Joint trainer runs end-to-end on synthetic LobSim | 1 epoch completes without NaN/Inf in any loss | E |
+| G6 | Reward shaping calibrated | Mean reward per trade closes positive on bull synthetic data | F |
+| G7 | Argo smoke runs to completion | `argo get` shows Succeeded | G |
+| **G8** | **Backtest gate** | **profit_factor > 1.0 on 2M-event sweep** | **H** |
+
+G8 is the trading-readiness gate that the prior architecture failed. Pass = the integrated trainer produces a profitable model; fail = even RL can't extract enough signal from this data and we re-scope.
+
+---
+
+## Files touched
+
+| File | Action | Responsibility |
+|------|--------|----------------|
+| `crates/ml-alpha/src/rl/mod.rs` | Create | Module entry point |
+| `crates/ml-alpha/src/rl/common.rs` | Create | `Transition` struct, shared types |
+| `crates/ml-alpha/src/rl/replay.rs` | Create | PER replay buffer (off-policy, DQN) |
+| `crates/ml-alpha/src/rl/rollout.rs` | Create | On-policy rollout buffer (PPO) |
+| `crates/ml-alpha/src/rl/dqn.rs` | Create | C51 Q-head + Bellman loss + target net |
+| `crates/ml-alpha/src/rl/ppo.rs` | Create | π policy head + V value head + clipped surrogate |
+| `crates/ml-alpha/src/rl/action.rs` | Create | 9-action grid definitions + Thompson sampler |
+| `crates/ml-alpha/src/rl/reward.rs` | Create | Per-trade realized PnL extraction from LobSim |
+| `crates/ml-alpha/cuda/dqn_distributional_q.cu` | Create | C51 forward+backward kernel |
+| `crates/ml-alpha/cuda/ppo_clipped_surrogate.cu` | Create | Clipped surrogate forward+backward |
+| `crates/ml-alpha/src/trainer/integrated.rs` | Create | The new integrated trainer (replaces `perception.rs::step`) |
+| `crates/ml-alpha/src/trainer/perception.rs` | Modify | Expose `forward_encoder() -> h_t` for the integrated trainer |
+| `crates/ml-alpha/src/lib.rs` | Modify | Register `rl` module |
+| `crates/ml-alpha/Cargo.toml` | Modify | Add ml-backtesting dep (for LobSim integration) |
+| `crates/ml-alpha/examples/alpha_rl_train.rs` | Create | New CLI binary, replaces `alpha_train` as canonical entry point |
+| `crates/ml-alpha/tests/integrated_trainer.rs` | Create | End-to-end smoke test (synthetic data) |
+| `infra/k8s/argo/alpha-rl-template.yaml` | Create | Argo workflow template |
+| `scripts/argo-alpha-rl.sh` | Create | Dispatcher script |
+
+The existing `alpha_train` binary stays (for diagnostic / parity runs). The new `alpha_rl_train` becomes the canonical production trainer.
+
+---
+
+## Phase A — Module skeleton (1 day)
+
+### A.1 Create the rl module structure
+
+**Files:**
+- Create: `crates/ml-alpha/src/rl/mod.rs`
+- Create: `crates/ml-alpha/src/rl/common.rs`
+- Modify: `crates/ml-alpha/src/lib.rs` (add `pub mod rl;`)
+
+### A.2 Define Transition struct
+
+Append to `crates/ml-alpha/src/rl/common.rs`:
+
+```rust
+//! Shared types for the integrated RL trainer.
+
+use cudarc::driver::CudaSlice;
+
+/// Single (state, action, reward, next_state) transition.
+/// Stored in both the off-policy replay buffer (DQN) and the on-policy
+/// rollout buffer (PPO). The encoder representation `h_t` is stored
+/// directly to avoid re-running the encoder during loss computation.
+#[derive(Clone, Debug)]
+pub struct Transition {
+    pub h_t: CudaSlice<f32>,           // [B, HIDDEN_DIM=128]
+    pub action: u32,                    // 0..9 (see action.rs)
+    pub reward: f32,                    // Per-trade realized PnL on close, 0 between closes
+    pub next_h_t: CudaSlice<f32>,       // [B, HIDDEN_DIM]
+    pub done: bool,                     // True at trade close or end-of-day
+    pub log_pi_old: f32,                // PPO importance ratio denominator
+    pub q_value_old: [f32; 9],          // DQN bootstrap value at sample time
+}
+```
+
+### A.3 Cargo dep update
+
+Modify `crates/ml-alpha/Cargo.toml`: add `ml-backtesting = { path = "../ml-backtesting" }` to `[dependencies]`.
+
+(Currently the dependency is reversed; flipping it requires breaking the cycle. If circular, the integrated trainer goes in ml-backtesting instead — flag at implementation time.)
+
+### A.4 Validate
+
+```bash
+SQLX_OFFLINE=true cargo check -p ml-alpha 2>&1 | tail -5
+```
+
+Expected: clean build.
+
+### A.5 Commit
+
+```bash
+git add crates/ml-alpha/src/lib.rs crates/ml-alpha/src/rl/ crates/ml-alpha/Cargo.toml
+git commit -m "feat(rl): module skeleton for integrated RL trainer (Phase A)"
+```
+
+---
+
+## Phase B — Encoder gradient hookup (2 days)
+
+### B.1 Expose forward_encoder
+
+Modify `crates/ml-alpha/src/trainer/perception.rs`: add public method `pub fn forward_encoder(&mut self, features: &[Mbp10RawInput]) -> CudaSlice<f32>` that runs the encoder forward and returns `h_t [B, 128]` WITHOUT computing the BCE head. The integrated trainer calls this, then dispatches its 5 heads on `h_t`.
+
+### B.2 Test gradient flow
+
+Write a synthetic forward-backward test in `crates/ml-alpha/tests/encoder_gradient.rs`:
+
+```rust
+#[test]
+fn encoder_gradient_flows_from_synthetic_head() {
+    let dev = ...;
+    let mut trainer = PerceptionTrainer::new(...);
+    let features = synth_features();
+    let h_t = trainer.forward_encoder(&features);
+    let loss = synth_loss(&h_t);  // any scalar loss on h_t
+    loss.backward();
+    // Assert: encoder weights moved (compare before/after)
+    let w_before = trainer.encoder.w_in.snapshot();
+    trainer.opt_step();
+    let w_after = trainer.encoder.w_in.snapshot();
+    assert!(diff(w_before, w_after) > 1e-6, "encoder weights must update");
+}
+```
+
+### B.3 Commit
+```
+git commit -m "feat(rl): expose forward_encoder; gradient-flow test (Phase B)"
+```
+
+---
+
+## Phase C — DQN head + replay + Bellman (5 days)
+
+This is the largest phase. Breaks into 5 sub-tasks.
+
+### C.1 C51 distributional Q-head structure
+
+Create `crates/ml-alpha/src/rl/dqn.rs`:
+- Struct `DqnHead` with `w_q: [9, AUX_HIDDEN=128]`, `b_q: [9]`, `support: [21]`, `target_w_q`, `target_b_q` (target network)
+- `pub fn forward(&self, h_t: &CudaSlice<f32>) -> CudaSlice<f32>` returns `[B, 9, 21]` (logits per atom per action)
+- Xavier init scaled by 0.1 per `pearl_scoped_init_seed_for_reproducibility`
+
+### C.2 Bellman loss + target network
+
+Create `crates/ml-alpha/cuda/dqn_distributional_q.cu`:
+- Forward: sigmoid → categorical distribution per action
+- Backward: TD loss = -log p(target_atom) where target_atom is derived from r + γ × max_a' Q_target(s', a')
+- Target net soft update: `target = (1-τ) × target + τ × current`
+
+### C.3 PER replay buffer
+
+Create `crates/ml-alpha/src/rl/replay.rs`:
+- `ReplayBuffer { capacity, transitions: Vec<Transition>, priorities: Vec<f32>, sum_tree: SumTree }`
+- `pub fn push(&mut self, t: Transition, td_error: f32)`
+- `pub fn sample(&mut self, batch_size: usize) -> Vec<(Transition, f32 /* IS weight */, usize /* idx */)>`
+- `pub fn update_priorities(&mut self, idxs: &[usize], td_errors: &[f32])`
+
+### C.4 Toy bandit test
+
+Test on a toy bandit problem in `tests/dqn_toy.rs`: synthetic state where action 5 always gets reward +1, other actions get -1. After 10k training steps, assert `Q(s, 5) > Q(s, *)` for any state s. Validates the entire DQN pipeline before integrating.
+
+### C.5 Commit
+```
+git commit -m "feat(rl): C51 distributional Q-head + PER replay + Bellman (Phase C)"
+```
+
+---
+
+## Phase D — PPO π/V heads + clipped surrogate (5 days)
+
+Mirrors Phase C structure for the actor side.
+
+### D.1 Policy + value heads
+
+Create `crates/ml-alpha/src/rl/ppo.rs`:
+- `PolicyHead { w_pi: [9, 128], b_pi: [9] }` → softmax over 9 actions
+- `ValueHead { w_v: [1, 128], b_v: [1] }` → scalar V(s)
+
+### D.2 Clipped surrogate loss
+
+Create `crates/ml-alpha/cuda/ppo_clipped_surrogate.cu`:
+- Forward: `surr = min(r × A, clip(r, 1-ε, 1+ε) × A)`, ε = 0.2 per PPO default
+- Backward: gradient of surrogate w.r.t. logits
+- Value loss: MSE between V_pred and V_target (Q-expectation under current π)
+- Entropy bonus: -coef × H(π) to encourage exploration; coef anneals from 0.01 → 0.001 over training
+
+### D.3 Rollout buffer
+
+Create `crates/ml-alpha/src/rl/rollout.rs`:
+- `RolloutBuffer { capacity, transitions: Vec<Transition>, advantages: Vec<f32>, returns: Vec<f32> }`
+- `pub fn push(&mut self, t: Transition)` — appends
+- `pub fn compute_advantages(&mut self, v_head: &ValueHead, q_head: &DqnHead, gamma: f32, lambda: f32)` — uses Q-head's expected value as the baseline (replaces GAE-λ)
+- `pub fn finalize_returns(&mut self)` — done-aware cumulative return
+
+### D.4 Toy bandit test
+
+Same toy bandit as C.4: after 10k steps, π(action=5|s) > 0.9.
+
+### D.5 Commit
+```
+git commit -m "feat(rl): PPO π/V heads + clipped surrogate + rollout (Phase D)"
+```
+
+---
+
+## Phase E — LobSim integration + training loop (3 days)
+
+### E.1 LobSim adapter
+
+Create `crates/ml-alpha/src/rl/reward.rs`:
+- Wraps `ml_backtesting::sim::LobSimCuda`
+- `pub fn step(&mut self, action: u32, h_t: &CudaSlice<f32>) -> (next_h_t, reward, done)`
+- Reward extraction: read `realized_pnl_usd` delta on trade close
+
+### E.2 Integrated trainer
+
+Create `crates/ml-alpha/src/trainer/integrated.rs`:
+- `pub struct IntegratedTrainer { encoder, bce_head, q_head, pi_head, v_head, aux_head, replay, rollout, lobsim, loss_balance_controller }`
+- `pub fn step(&mut self) -> Result<StepStats>` runs the training loop above (one event per call)
+- All 5 heads' losses combined, single backward into encoder
+
+### E.3 End-to-end synthetic smoke
+
+Test in `crates/ml-alpha/tests/integrated_trainer.rs`:
+- 1000 events of synthetic LobSim data (deterministic price walk)
+- Run 1 epoch through integrated trainer
+- Assert: no NaN/Inf in any loss, all 5 heads' gradients flowed (compare before/after weights), replay buffer non-empty
+
+### E.4 Commit
+```
+git commit -m "feat(rl): LobSim integration + integrated trainer end-to-end (Phase E)"
+```
+
+---
+
+## Phase F — Reward shaping calibration (2 days)
+
+### F.1 Reward components
+
+Implement in `reward.rs`:
+- Base: `realized_pnl_usd` on trade close
+- Penalty: `fees_usd` (already subtracted in summary)
+- Penalty: `exit_path_cost` (slippage at close, computed from LobSim)
+- Bonus (optional): `exposure_bonus = -|position_lots| × 0.0001 × dt_ns` (small penalty for sitting in inventory)
+
+### F.2 Reward scale calibration
+
+Calibrate the reward magnitude to match the V-head's expected range. Typical PnL per trade in USD ≈ 0.1-100. Scale: `reward_scaled = reward / 10.0` so V can fit in [-10, +10] which matches the C51 atom support `[-1, +1]` after standardization.
+
+### F.3 Sign-check test
+
+On a synthetic bull-market stream where price monotonically rises, train for 1k events. Assert: mean reward per closed trade > 0 when the policy converges to "long all the way."
+
+### F.4 Commit
+
+```
+git commit -m "feat(rl): reward shaping + calibration (Phase F)"
+```
+
+---
+
+## Phase G — Argo workflow + dispatcher (1 day)
+
+### G.1 Template
+
+Create `infra/k8s/argo/alpha-rl-template.yaml`:
+- Same structure as `alpha-perception-template.yaml`
+- New params: `gamma` (0.99), `n-rollout-steps` (2048), `replay-size` (100000), `n-ppo-epochs` (4), `target-update-tau` (0.005), `target-update-freq` (100)
+- Same `instrument-mode` and `multi-resolution` defaults as the perception template (front-month, 1:32)
+
+### G.2 Dispatcher script
+
+Create `scripts/argo-alpha-rl.sh`: mirrors `argo-alpha-perception.sh` with the new params.
+
+### G.3 Commit
+
+```
+git commit -m "feat(rl): Argo template + dispatcher script (Phase G)"
+```
+
+---
+
+## Phase H — Backtest gate (1 day + 1.5h smoke)
+
+### H.1 Push + apply template + dispatch
+
+```bash
+git push origin ml-alpha-phase-a
+kubectl apply -n foxhunt -f infra/k8s/argo/alpha-rl-template.yaml
+./scripts/argo-alpha-rl.sh --instrument-mode front-month
+```
+
+### H.2 Pull results + backtest
+
+After the workflow succeeds, pull the trained checkpoint:
+- `/feature-cache/alpha-rl-runs/$SHA/integrated_best_h1000.bin`
+
+Update `config/ml/sweep_smoke_perhoriz_cfc.yaml` to point at the new checkpoint, dispatch `argo-lob-sweep.sh`.
+
+### H.3 Evaluate the gate
+
+| Metric | Failure threshold | Pass |
+|--------|--------------------|------|
+| profit_factor | < 0.5 → re-architect needed | > 1.0 |
+| sharpe_ann | < -3 → re-architect needed | > 0.5 |
+| outcome_by_entry_conv | mono-anti-cal pattern persists | Higher conviction = higher mean_pnl |
+
+### H.4 Document outcome
+
+- Pass: write `pearl_integrated_rl_works.md`, plan production deployment
+- Fail: document specific failure mode, decide whether to re-scope (different action space, different reward, different action grid)
+
+---
+
+## Self-review checklist
+
+**Spec coverage:**
+- [x] Encoder gradient from all 5 heads (Phase B)
+- [x] DQN distributional Q + PER replay + Bellman (Phase C)
+- [x] PPO π/V + clipped surrogate + rollout (Phase D)
+- [x] LobSim integration with per-trade reward (Phase E)
+- [x] Reward shaping (Phase F)
+- [x] Argo workflows (Phase G)
+- [x] Backtest gate G8 (Phase H)
+
+**Risk analysis:**
+- **Highest risk**: encoder gradient instability when 5 heads compete for shape. Mitigation: loss-balance controller (already in memory pearls) adaptively scales λ_k.
+- **Second highest**: replay buffer GPU memory (100k transitions × 128 floats h_t × 4 bytes × 2 (h_t and next_h_t) ≈ 100 MB). Fits on L40S. PER sum-tree is CPU. Tradeoff.
+- **Third**: PPO + DQN can diverge if both heads disagree strongly. Mitigation: monitor `argmax_Q vs argmax_π` agreement rate; if < 50%, raise alarm and pause training.
+
+**Greenfield commitment:** No backward-compat with old DQN in `crates/ml/src/trainers/dqn/`. That code stays for the old policy (different action space). The new integrated trainer is ml-alpha-specific.
+
+**Project policy compliance:**
+- `feedback_no_partial_refactor`: each phase is atomic; subagents land complete files per phase.
+- `feedback_no_feature_flags`: no `if use_rl` toggle — `alpha_rl_train` is the new canonical entry.
+- `feedback_no_htod_htoh_only_mapped_pinned`: all CPU↔GPU via mapped-pinned (per pre-existing infra).
+- `feedback_no_stubs`: no return-zero Q-heads or placeholder PPO loss.
+- `pearl_thompson_for_distributional_action_selection`: Thompson at rollout, argmax only at eval.
+- `pearl_c51_thompson_closed_phase_e3_gap`: C51 + Thompson is the proven combo.
+- `pearl_loss_balance_controller`: existing controller handles λ_k for 5 heads.
+- `pearl_cooperative_staging_eliminates_redundant_reads`: encoder forward once, all 5 heads share `h_t`.
+
+**Falsifiability:** Each phase has a gate. G3/G4 prove DQN/PPO work in isolation. G5/G6 prove integration. G8 is the actual trading-readiness gate that the prior architecture failed.
+
+**Calendar estimate:** ~3 weeks (16 dev days + 1.5h Argo smoke). With subagent parallelism in Phases C and D (independent algorithms), can compress to ~2.5 weeks.