From 57de1a8b4ed03cd5abf9d6ba0830d9b5150b7919 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 22 May 2026 22:07:45 +0200 Subject: [PATCH] docs(plans): integrated RL trainer (DQN + PPO + BCE + aux) plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single integrated trainer where 5 loss heads (BCE direction, C51 distributional Q, categorical π, scalar V, aux prof+size) sit on a shared Mamba2+CfC encoder. Joint training with adaptive loss-balance λ weights via the existing pearl_loss_balance_controller infrastructure. Discrete 9-action grid shared between Q and π heads; reward = per-trade realized PnL from LobSimCuda. Designed in response to lob-backtest-sweep-jpdhg result (PF=0.24, sharpe=-9.72, mono-anti-cal in conviction→PnL): the encoder learns directional AUC but never sees PnL-aware gradient because stop_grad_aux_to_encoder blocks the aux head's gradient. RL training fixes this by making the encoder optimize per-trade realized PnL via Q-head Bellman + π-head clipped surrogate. Every hyperparameter (γ, τ, PPO clip ε, entropy coef, rollout steps, PER α, reward scale, loss-balance λs) is ISV-driven per pearl_controller_anchors_isv_driven and feedback_isv_for_adaptive_bounds. 12 new ISV slots (400-411) + 12 reused existing slots + 8 new controller kernels following the Wiener-α + first-observation-bootstrap pattern. 10 phases (A-H, ~3 weeks), falsifiable gate G8 = profit_factor > 1.0 on 2M-event backtest sweep. --- .../plans/2026-05-22-integrated-rl-trainer.md | 638 ++++++++++++++++++ 1 file changed, 638 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md diff --git a/docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md b/docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md new file mode 100644 index 000000000..e528e4546 --- /dev/null +++ b/docs/superpowers/plans/2026-05-22-integrated-rl-trainer.md @@ -0,0 +1,638 @@ +# Integrated RL Trainer: BCE + DQN + PPO + Aux on a Shared Encoder + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. + +**Goal:** Replace the pure-supervised BCE alpha-perception trainer with a single integrated trainer where the Mamba2+CfC encoder receives gradient from FIVE complementary loss heads — BCE direction, DQN distributional Q, PPO policy + value, and auxiliary prof/size — so the encoder learns features useful for **realized trading PnL**, not just price-direction prediction. + +**Architecture:** One shared encoder produces `h_t ∈ R^128`. Five heads sit on top: (1) BCE direction (warm-start signal), (2) C51 distributional Q-head (off-policy critic), (3) categorical π policy head (on-policy actor), (4) scalar V value head, (5) prof+size aux head (per-trade calibration). Total loss is a weighted sum with adaptive λ via the existing loss-balance controller. Action space is shared 9-element discrete grid. Reward is per-trade realized PnL from the GPU LobSim. Replay buffer (PER) feeds Q-head off-policy; rollout buffer feeds π+V on-policy; both buffers fill from the SAME training-loop transitions. + +**Tech Stack:** Rust 1.85+, Tokio 1.40, CUDA 12.4 L40S/H100, cudarc 0.19. Existing components reused: Mamba2 SSM + CfC encoder (ml-alpha/src/cfc/), aux trunk + heads (ml-alpha/src/aux_heads.rs, aux_trunk.rs), LobSim (ml-backtesting/src/sim/), perception trainer skeleton (ml-alpha/src/trainer/perception.rs). New components: DQN Q-head + replay, PPO π/V heads + rollout, action-grid kernels, reward-shaping kernels, loss-balance controller wiring for 5 heads. + +--- + +## Why this architecture + +The user's question motivating this plan: "the model should run inference during training to learn from its mistakes — does it?" Answer: NO, the current PerceptionTrainer optimizes only BCE on offline `sign(p[t+K] - p[t])` labels. It never sees its own trade outcomes, fees, slippage, or PnL. + +Empirical evidence: alpha-perception-jvv7d trained to auc_h1000=0.576 (decent directional signal). Backtest `lob-backtest-sweep-jpdhg` on that checkpoint gave: +- profit_factor = 0.24 +- sharpe_ann = -9.72 +- total_pnl_usd = -$1.29M +- win-rate correlates with conviction (16% → 27%) BUT mean_pnl gets WORSE with conviction + +The decoupling is structural: the encoder optimizes directional AUC but never sees the cost-aware PnL gradient. The fix is to give the encoder a PnL-aware gradient via RL, while keeping the BCE signal for stable warm-start. + +### Why both DQN AND PPO (not OR) + +DQN-only weaknesses: ε-greedy exploration wastes 10-30% of compute on random actions; struggles when action distinguishability is subtle. + +PPO-only weaknesses: high variance from MC returns; on-policy = throws away old data; needs GAE-λ tricks. + +Combined (SAC-style actor-critic): Q-head provides the LOW-VARIANCE advantage signal to π — no need for GAE-λ. π-head provides STOCHASTIC exploration informed by current policy beliefs — no ε-greedy. The two cross-validate (Q-argmax should agree with π-mode at convergence). One encoder gets BOTH gradient streams plus BCE + aux supervision. + +--- + +## Architecture diagram + +``` +[MBP-10 raw features, 32 ticks] + │ + ▼ + [snap_feature_assemble kernel] + │ feature vector [B, 32, 40] + ▼ + [Mamba2 SSM, state_dim=16] + │ + ▼ + [CfC continuous-time recurrence, hidden=128] + │ h_t [B, 128] + ▼ + ┌──────────┼──────────┬──────────┬──────────┐ + │ │ │ │ │ + ▼ ▼ ▼ ▼ ▼ +[BCE dir] [C51 Q] [π policy] [V value] [Aux prof+size] +[B, 3] [B, 9, 21] [B, 9] [B, 1] [B, 2, 3, 2] + │ │ │ │ │ +sigmoid atoms over softmax raw sigmoid+raw + │ [Q_min, │ │ │ + │ Q_max] │ │ │ + │ │ │ │ │ +L_BCE L_Q (Bellman) L_π L_V (MSE) L_aux +direction TD on s,a (clipped to Q-expect (BCE_prof +labels target net surr,A_Q) ation + Huber_size) + + ─────── weighted sum ─────── + L_total + │ + ▼ backward + encoder ∇ = ∇ × λ_total +``` + +### Action space (9 discrete actions, shared by Q and π) + +| idx | Action | Effect | +|-----|--------|--------| +| 0 | Short-large | Open short, size = 2× base | +| 1 | Short-small | Open short, size = 1× base | +| 2 | Hold | No change to position | +| 3 | Flat-from-long | Close long position if any | +| 4 | Flat-from-short | Close short position if any | +| 5 | Long-small | Open long, size = 1× base | +| 6 | Long-large | Open long, size = 2× base | +| 7 | Trail-tighten | Reduce trailing stop distance by 25% | +| 8 | Trail-loosen | Increase trailing stop distance by 25% | + +Note: matches existing DQN action grid in `crates/ml/src/cuda_pipeline/sp5_isv_slots.rs` for parity. Trail-tighten/loosen are stop-management actions; the other 7 are position-management. + +### Reward (per-trade realized PnL on close) + +`reward = realized_pnl_usd - fees_usd - exit_path_cost` + +Reward fires at trade CLOSE (when position goes to 0). Between closes, reward is 0. Per-trade signal matches what we actually optimize (PF, sharpe). Sparser than per-tick but unambiguous credit assignment. + +For TD(λ) bootstrap of intermediate values, the V-head learns the discounted-future-reward estimate; Q-head learns per-action discounted return. λ_discount = 0.99 per HFT convention (matches old DQN gamma). + +### Loss formulation + +``` +L_total = λ_bce × L_BCE(dir_logits, dir_labels) + + λ_q × L_Q(Q(s_t, a_t), r_t + γ × max_a' Q_target(s_{t+1}, a')) + + λ_π × L_π(π(s_t), A_Q(s_t, a_t) = Q(s_t, a_t) - V(s_t)) + + λ_v × L_V(V(s_t), Σ_a π(a|s_t) × Q(s_t, a)) + + λ_aux × (L_aux_prof + L_aux_size) +``` + +Where: +- L_Q is Huber loss on the distributional TD error (per pearl_per_branch_c51_atom_span) +- L_π is the PPO clipped surrogate: `min(r_t × A, clip(r_t, 1-ε, 1+ε) × A)`, r_t = π_new/π_old +- L_V is MSE between V prediction and the expected value under the policy +- L_aux uses the existing aux_loss.cu kernels (BCE prof + masked Huber size) +- λ_k are learned per-head via the existing loss-balance controller (`pearl_loss_balance_controller`) + +--- + +## State / observation + +State at step t = encoder output `h_t ∈ R^128`. The encoder reads: +- Last 32 raw MBP-10 ticks (default `single_scale_32` from the just-reverted multi-resolution config) +- Regime context (6 EMAs cascade) per existing loader +- Δt Fourier features per existing snap_feature kernel +- (Optional) position state vector: `[current_position_lots, current_avg_entry_px, hold_duration_ns]` concatenated to h_t before head dispatch. This lets the policy see its own state. + +--- + +## Training loop + +```rust +loop { + let batch = loader.next_sequence(); // existing + let h_t = encoder.forward(batch.features); // existing + + // Action selection during rollout: Thompson over Q distribution + // (pearl_thompson_for_distributional_action_selection) + let action = sample_thompson(q_head.forward(h_t)); + + // Sim step: submit action, advance one event, get reward + let (next_state, reward, done) = lobsim.step(action); + + // Store transition in BOTH buffers + let transition = Transition { + h_t, + action, + reward, + next_h_t: next_state.encoded(), + done, + // Log probability for PPO importance ratio + log_pi: pi_head.log_prob(action), + }; + replay_buffer.push(transition.clone()); // off-policy for Q + rollout_buffer.push(transition); // on-policy for π+V + + // Off-policy DQN update (every step, from PER replay) + if replay_buffer.len() > min_warmup { + let dqn_batch = replay_buffer.sample_per(batch_size); + let l_q = dqn_loss(dqn_batch, q_head, q_target_head); + } + + // On-policy PPO update (at end of rollout) + if rollout_buffer.full() { + for _ in 0..n_ppo_epochs { + let l_pi = ppo_clipped_surrogate(rollout_buffer, pi_head, q_head); + let l_v = value_mse(rollout_buffer, v_head, q_head, pi_head); + } + rollout_buffer.clear(); + } + + // BCE + aux losses (every step) + let l_bce = bce_dir(bce_head.forward(h_t), batch.dir_labels); + let l_aux = aux_loss(aux_head.forward(h_t), batch.aux_labels); + + // Loss-balance controller updates λ_k + let lambdas = loss_balance_controller.update_and_get(); + + let l_total = lambdas.bce * l_bce + + lambdas.q * l_q + + lambdas.pi * l_pi + + lambdas.v * l_v + + lambdas.aux * l_aux; + l_total.backward(); + + // Periodic target-network soft update + if step % target_update_freq == 0 { + q_target_head.soft_update(q_head, tau = 0.005); + } +} +``` + +Key design points: +1. **Both buffers fill from the SAME transitions** — no separate rollouts. PPO uses fresh data from the most-recent rollout window; DQN uses ANY data from the larger replay buffer. +2. **One forward pass per step** computes all 5 heads from `h_t`. +3. **One backward pass** combines all gradients into the encoder. +4. **Target network** is for Q only (PPO doesn't need one). Soft updates per `pearl_per_branch_c51_atom_span` infrastructure (already exists). +5. **Thompson action selection** at rollout time (pearl-mandated); argmax-Q only at eval time. + +--- + +## ISV-driven adaptive control (mandatory per project memory) + +Per `feedback_isv_for_adaptive_bounds` and `pearl_controller_anchors_isv_driven`, +**every** hyperparameter / threshold / cap in this trainer MUST be sourced from an +ISV slot updated by a per-step controller kernel. No `const f32 = 0.99` for γ, no +`const f32 = 0.2` for PPO clip ε, no `const f32 = 0.005` for target-net τ. + +The existing `crates/ml/src/cuda_pipeline/sp5_isv_slots.rs` ISV bus has 56+ slots +(`ISV_TOTAL_DIM = 383`). The integrated RL trainer adds a new block of slots and +reuses the existing controller infrastructure: + +### New ISV slots (allocate at top of ml-alpha-specific block, e.g. 400-450) + +| Slot | Name | Role | Controller (producer) | Bootstrap | +|------|------|------|----------------------|-----------| +| 400 | `RL_GAMMA_INDEX` | Discount factor (Q + V + PPO advantage) | Trade-horizon EMA controller (input: mean trade duration / 60s anchor) | Pearl-A first observation; sentinel 0 → 0.99 init | +| 401 | `RL_TARGET_TAU_INDEX` | Target-net soft-update fraction | Q-divergence EMA controller (input: ‖Q_current - Q_target‖ EMA) | Pearl-A; sentinel 0 → 0.005 init | +| 402 | `RL_PPO_CLIP_INDEX` | Clipped surrogate ε | Policy-divergence controller (input: KL(π_new‖π_old) EMA, target = 0.01) | Pearl-A; sentinel 0 → 0.2 init | +| 403 | `RL_ENTROPY_COEF_INDEX` | Entropy bonus weight | Action-entropy deficit controller (target = 70% × ln(9)) per `pearl_per_branch_noisy_sigma` pattern | Pearl-A; sentinel 0 → 0.01 init | +| 404 | `RL_N_ROLLOUT_STEPS_INDEX` | Rollout buffer flush size | Advantage-variance controller (input: var(A_Q) / |A_Q| ratio, target = 0.1) | Pearl-A; sentinel 0 → 2048 init | +| 405 | `RL_PER_ALPHA_INDEX` | PER priority exponent | TD-error distribution controller (input: TD-error kurtosis EMA) | Pearl-A; sentinel 0 → 0.6 init | +| 406 | `RL_REWARD_SCALE_INDEX` | Per-trade reward standardization | PnL magnitude EMA (input: mean |realized_pnl_usd| over last 100 closes) | Pearl-A; sentinel 0 → 1.0 init | +| 407 | `RL_Q_ARG_VS_PI_AGREE_INDEX` | Agreement rate Q-argmax vs π-mode | Diagnostic-only (read by alarm; not controlled) | EMA-tracked, no init magic | +| 408-411 | `RL_LOSS_LAMBDA_BCE/Q/PI/V_INDEX` (4 slots) | Per-head loss weight (BCE, Q, π, V; aux uses existing slot) | Reuse `pearl_loss_balance_controller` Wiener-α + signal-modulated target controller | Pearl-A from `pearl_first_observation_bootstrap`; sentinel 0 → equal-weight 1/5 init | + +The loss-balance controller (existing, `pearl_loss_balance_controller`) ALREADY +handles per-loss weight balancing for the perception trainer's 3 horizons. We +EXTEND it to 5 heads (BCE, Q, π, V, aux) by allocating 4 new λ slots (aux λ +already exists per current trainer). The controller's two-layer design +(signal-modulated target × Wiener-α adaptation) covers our need; no new +algorithm. + +### Reused existing controllers (no new infrastructure) + +| ISV slot | Reused for | +|----------|-----------| +| `EVAL_THOMPSON_TEMP_INDEX` (existing) | Thompson temperature at rollout action selection | +| `KELLY_F_SMOOTH_INDEX = 280` (existing) | Kelly fraction smoothing — fed to action sizing in `action.rs` | +| `CONVICTION_SMOOTH_INDEX = 281` (existing) | Conviction EMA for outcome bucketing diagnostic | +| `Q_VAR_PER_BRANCH_BASE = 222` (existing) | Per-branch Q-variance EMA — input to several RL controllers | +| `FLATNESS_BASE = 206` (existing) | Per-branch flatness signal — input to budget controllers | +| `NOISY_SIGMA_BASE = 210` (existing) | NoisyNet σ — REUSED for stochastic exploration on π head (instead of new entropy bonus mechanism) | +| `BUDGET_C51_BASE = 190` (existing) | C51 loss budget per branch — reused for our Q-head | +| `LB_C51_ACTIVE_BASE = 317` (existing) | Loss-balance C51 activation flag — extend to gate Q-head warmup | +| `PNL_TOTAL_INDEX = 286`, `PNL_MEAN_INDEX = 287`, `PNL_VAR_INDEX = 288`, `PNL_MAX_DD_INDEX = 289` (existing) | PnL telemetry; INPUT to reward-scale controller (RL_REWARD_SCALE_INDEX = 406) | +| `TRAINING_SHARPE_EMA_INDEX = 294`, `MAX_DD_EMA_INDEX = 295` (existing) | Training Sharpe/DD EMAs — INPUT to PnL magnitude controller | +| `REWARD_*_WEIGHT_INDEX` (340-348 existing, SP11 reward subsystem) | DIRECTLY reused as reward shaping component weights; no new reward kernel | +| `CURIOSITY_PRESSURE_INDEX = 346` (existing) | Direct reuse for exploration bonus on top of policy entropy | + +### Controller kernels to add (new) + +| Kernel | Reads (ISV inputs) | Writes (ISV outputs) | Notes | +|--------|-------------------|---------------------|-------| +| `rl_gamma_controller_kernel.cu` | `PNL_MEAN_INDEX`, `KELLY_F_SMOOTH_INDEX`, mean_trade_duration_ns | `RL_GAMMA_INDEX` (400) | γ derives from "what fraction of trades close within decision horizon" | +| `rl_target_tau_controller_kernel.cu` | Q-divergence EMA (computed in-kernel from current vs target weights) | `RL_TARGET_TAU_INDEX` (401) | τ adapts: faster updates when divergence high | +| `rl_ppo_clip_controller_kernel.cu` | KL(π_new‖π_old) EMA | `RL_PPO_CLIP_INDEX` (402) | ε adapts: tighter clip when KL drifts high | +| `rl_entropy_coef_controller_kernel.cu` | Action-entropy deficit (target = 0.7×ln(9)) | `RL_ENTROPY_COEF_INDEX` (403) | Wiener-α tracking per `pearl_wiener_alpha_floor_for_nonstationary` (α-floor 0.4) | +| `rl_rollout_steps_controller_kernel.cu` | var(A_Q) / \|A_Q\| ratio EMA | `RL_N_ROLLOUT_STEPS_INDEX` (404) | Longer rollout when advantage estimates are noisy | +| `rl_per_alpha_controller_kernel.cu` | TD-error kurtosis EMA | `RL_PER_ALPHA_INDEX` (405) | Higher α when TD-errors heavy-tailed | +| `rl_reward_scale_controller_kernel.cu` | Mean \|realized_pnl_usd\| EMA over last 100 closes (`PNL_MEAN_INDEX`) | `RL_REWARD_SCALE_INDEX` (406) | Standardize rewards so V-head's expected range matches C51 atom support | +| `rl_q_pi_agreement_kernel.cu` | argmax(Q) vs mode(π) per step | `RL_Q_ARG_VS_PI_AGREE_INDEX` (407) | Diagnostic; not consumed by loss; alarms when < 0.5 | + +All 8 new controllers follow the existing pattern: +1. Wiener-α-with-floor (per `pearl_wiener_alpha_floor_for_nonstationary`, α-floor = 0.4) +2. First-observation bootstrap from sentinel 0 (per `pearl_first_observation_bootstrap`) +3. State-reset registry entry + matching `reset_named_state` arm (per `feedback_registry_entries_need_dispatch_arms`) +4. Pearl-D recovery cascade: if controller output drifts to invalid range, fold-boundary reset re-bootstraps + +### Constants permitted (true structural constants only) + +These remain as `const f32` because they're STRUCTURAL parameters of the algorithm, +not adaptive bounds: + +- `N_ACTIONS = 9` (action grid cardinality — wired into kernel `#define`s) +- `Q_N_ATOMS = 21` (C51 atom count — kernel layout) +- `HIDDEN_DIM = 128` (encoder output dim — already a workspace const) +- `PPO_N_EPOCHS = 4` per rollout (algorithm hyperparameter; no signal to drive it adaptively) + +Everything else is ISV. + +### Plan task changes for ISV compliance + +- Phase A.2: add the 12 new ISV slot constants (400-411) to a new `crates/ml-alpha/src/rl/isv_slots.rs` file. +- Phase B: each head reads its hyperparameters from ISV, not constants. +- Phase C.2 (Bellman loss): reads `RL_GAMMA_INDEX`, `RL_TARGET_TAU_INDEX` from device ISV. +- Phase D.2 (clipped surrogate): reads `RL_PPO_CLIP_INDEX`, `RL_ENTROPY_COEF_INDEX`. +- Phase E: training loop reads `RL_N_ROLLOUT_STEPS_INDEX`, `RL_PER_ALPHA_INDEX`. +- Phase F: reward scale derives from `RL_REWARD_SCALE_INDEX`, not from a hardcoded `/ 10.0` divisor. +- New Phase C.6 / D.6 sub-task per controller: implement + test the controller kernel itself. + +The bootstrap values shown in the "Bootstrap" column above are what the +first-observation-after-sentinel-zero controller emits, not hardcoded — they're +written by the controller on first emit, then adapted by subsequent emits. + +## Falsifiable gates + +| Gate | Metric | Pass | Phase | +|------|--------|------|-------| +| G1 | Workspace builds | `cargo check --workspace` clean | A | +| G2 | Encoder gradient flows from each head independently | Synthetic forward-backward test for each head | B | +| G3 | DQN converges on toy bandit | Q(state, action_correct) > Q(state, action_wrong) after 10k steps | C | +| G4 | PPO converges on toy bandit | π(action_correct | state) > 0.9 after 10k steps | D | +| G5 | Joint trainer runs end-to-end on synthetic LobSim | 1 epoch completes without NaN/Inf in any loss | E | +| G6 | Reward shaping calibrated | Mean reward per trade closes positive on bull synthetic data | F | +| G7 | Argo smoke runs to completion | `argo get` shows Succeeded | G | +| **G8** | **Backtest gate** | **profit_factor > 1.0 on 2M-event sweep** | **H** | + +G8 is the trading-readiness gate that the prior architecture failed. Pass = the integrated trainer produces a profitable model; fail = even RL can't extract enough signal from this data and we re-scope. + +--- + +## Files touched + +| File | Action | Responsibility | +|------|--------|----------------| +| `crates/ml-alpha/src/rl/mod.rs` | Create | Module entry point | +| `crates/ml-alpha/src/rl/common.rs` | Create | `Transition` struct, shared types | +| `crates/ml-alpha/src/rl/replay.rs` | Create | PER replay buffer (off-policy, DQN) | +| `crates/ml-alpha/src/rl/rollout.rs` | Create | On-policy rollout buffer (PPO) | +| `crates/ml-alpha/src/rl/dqn.rs` | Create | C51 Q-head + Bellman loss + target net | +| `crates/ml-alpha/src/rl/ppo.rs` | Create | π policy head + V value head + clipped surrogate | +| `crates/ml-alpha/src/rl/action.rs` | Create | 9-action grid definitions + Thompson sampler | +| `crates/ml-alpha/src/rl/reward.rs` | Create | Per-trade realized PnL extraction from LobSim | +| `crates/ml-alpha/cuda/dqn_distributional_q.cu` | Create | C51 forward+backward kernel | +| `crates/ml-alpha/cuda/ppo_clipped_surrogate.cu` | Create | Clipped surrogate forward+backward | +| `crates/ml-alpha/src/trainer/integrated.rs` | Create | The new integrated trainer (replaces `perception.rs::step`) | +| `crates/ml-alpha/src/trainer/perception.rs` | Modify | Expose `forward_encoder() -> h_t` for the integrated trainer | +| `crates/ml-alpha/src/lib.rs` | Modify | Register `rl` module | +| `crates/ml-alpha/Cargo.toml` | Modify | Add ml-backtesting dep (for LobSim integration) | +| `crates/ml-alpha/examples/alpha_rl_train.rs` | Create | New CLI binary, replaces `alpha_train` as canonical entry point | +| `crates/ml-alpha/tests/integrated_trainer.rs` | Create | End-to-end smoke test (synthetic data) | +| `infra/k8s/argo/alpha-rl-template.yaml` | Create | Argo workflow template | +| `scripts/argo-alpha-rl.sh` | Create | Dispatcher script | + +The existing `alpha_train` binary stays (for diagnostic / parity runs). The new `alpha_rl_train` becomes the canonical production trainer. + +--- + +## Phase A — Module skeleton (1 day) + +### A.1 Create the rl module structure + +**Files:** +- Create: `crates/ml-alpha/src/rl/mod.rs` +- Create: `crates/ml-alpha/src/rl/common.rs` +- Modify: `crates/ml-alpha/src/lib.rs` (add `pub mod rl;`) + +### A.2 Define Transition struct + +Append to `crates/ml-alpha/src/rl/common.rs`: + +```rust +//! Shared types for the integrated RL trainer. + +use cudarc::driver::CudaSlice; + +/// Single (state, action, reward, next_state) transition. +/// Stored in both the off-policy replay buffer (DQN) and the on-policy +/// rollout buffer (PPO). The encoder representation `h_t` is stored +/// directly to avoid re-running the encoder during loss computation. +#[derive(Clone, Debug)] +pub struct Transition { + pub h_t: CudaSlice, // [B, HIDDEN_DIM=128] + pub action: u32, // 0..9 (see action.rs) + pub reward: f32, // Per-trade realized PnL on close, 0 between closes + pub next_h_t: CudaSlice, // [B, HIDDEN_DIM] + pub done: bool, // True at trade close or end-of-day + pub log_pi_old: f32, // PPO importance ratio denominator + pub q_value_old: [f32; 9], // DQN bootstrap value at sample time +} +``` + +### A.3 Cargo dep update + +Modify `crates/ml-alpha/Cargo.toml`: add `ml-backtesting = { path = "../ml-backtesting" }` to `[dependencies]`. + +(Currently the dependency is reversed; flipping it requires breaking the cycle. If circular, the integrated trainer goes in ml-backtesting instead — flag at implementation time.) + +### A.4 Validate + +```bash +SQLX_OFFLINE=true cargo check -p ml-alpha 2>&1 | tail -5 +``` + +Expected: clean build. + +### A.5 Commit + +```bash +git add crates/ml-alpha/src/lib.rs crates/ml-alpha/src/rl/ crates/ml-alpha/Cargo.toml +git commit -m "feat(rl): module skeleton for integrated RL trainer (Phase A)" +``` + +--- + +## Phase B — Encoder gradient hookup (2 days) + +### B.1 Expose forward_encoder + +Modify `crates/ml-alpha/src/trainer/perception.rs`: add public method `pub fn forward_encoder(&mut self, features: &[Mbp10RawInput]) -> CudaSlice` that runs the encoder forward and returns `h_t [B, 128]` WITHOUT computing the BCE head. The integrated trainer calls this, then dispatches its 5 heads on `h_t`. + +### B.2 Test gradient flow + +Write a synthetic forward-backward test in `crates/ml-alpha/tests/encoder_gradient.rs`: + +```rust +#[test] +fn encoder_gradient_flows_from_synthetic_head() { + let dev = ...; + let mut trainer = PerceptionTrainer::new(...); + let features = synth_features(); + let h_t = trainer.forward_encoder(&features); + let loss = synth_loss(&h_t); // any scalar loss on h_t + loss.backward(); + // Assert: encoder weights moved (compare before/after) + let w_before = trainer.encoder.w_in.snapshot(); + trainer.opt_step(); + let w_after = trainer.encoder.w_in.snapshot(); + assert!(diff(w_before, w_after) > 1e-6, "encoder weights must update"); +} +``` + +### B.3 Commit +``` +git commit -m "feat(rl): expose forward_encoder; gradient-flow test (Phase B)" +``` + +--- + +## Phase C — DQN head + replay + Bellman (5 days) + +This is the largest phase. Breaks into 5 sub-tasks. + +### C.1 C51 distributional Q-head structure + +Create `crates/ml-alpha/src/rl/dqn.rs`: +- Struct `DqnHead` with `w_q: [9, AUX_HIDDEN=128]`, `b_q: [9]`, `support: [21]`, `target_w_q`, `target_b_q` (target network) +- `pub fn forward(&self, h_t: &CudaSlice) -> CudaSlice` returns `[B, 9, 21]` (logits per atom per action) +- Xavier init scaled by 0.1 per `pearl_scoped_init_seed_for_reproducibility` + +### C.2 Bellman loss + target network + +Create `crates/ml-alpha/cuda/dqn_distributional_q.cu`: +- Forward: sigmoid → categorical distribution per action +- Backward: TD loss = -log p(target_atom) where target_atom is derived from r + γ × max_a' Q_target(s', a') +- Target net soft update: `target = (1-τ) × target + τ × current` + +### C.3 PER replay buffer + +Create `crates/ml-alpha/src/rl/replay.rs`: +- `ReplayBuffer { capacity, transitions: Vec, priorities: Vec, sum_tree: SumTree }` +- `pub fn push(&mut self, t: Transition, td_error: f32)` +- `pub fn sample(&mut self, batch_size: usize) -> Vec<(Transition, f32 /* IS weight */, usize /* idx */)>` +- `pub fn update_priorities(&mut self, idxs: &[usize], td_errors: &[f32])` + +### C.4 Toy bandit test + +Test on a toy bandit problem in `tests/dqn_toy.rs`: synthetic state where action 5 always gets reward +1, other actions get -1. After 10k training steps, assert `Q(s, 5) > Q(s, *)` for any state s. Validates the entire DQN pipeline before integrating. + +### C.5 Commit +``` +git commit -m "feat(rl): C51 distributional Q-head + PER replay + Bellman (Phase C)" +``` + +--- + +## Phase D — PPO π/V heads + clipped surrogate (5 days) + +Mirrors Phase C structure for the actor side. + +### D.1 Policy + value heads + +Create `crates/ml-alpha/src/rl/ppo.rs`: +- `PolicyHead { w_pi: [9, 128], b_pi: [9] }` → softmax over 9 actions +- `ValueHead { w_v: [1, 128], b_v: [1] }` → scalar V(s) + +### D.2 Clipped surrogate loss + +Create `crates/ml-alpha/cuda/ppo_clipped_surrogate.cu`: +- Forward: `surr = min(r × A, clip(r, 1-ε, 1+ε) × A)`, ε = 0.2 per PPO default +- Backward: gradient of surrogate w.r.t. logits +- Value loss: MSE between V_pred and V_target (Q-expectation under current π) +- Entropy bonus: -coef × H(π) to encourage exploration; coef anneals from 0.01 → 0.001 over training + +### D.3 Rollout buffer + +Create `crates/ml-alpha/src/rl/rollout.rs`: +- `RolloutBuffer { capacity, transitions: Vec, advantages: Vec, returns: Vec }` +- `pub fn push(&mut self, t: Transition)` — appends +- `pub fn compute_advantages(&mut self, v_head: &ValueHead, q_head: &DqnHead, gamma: f32, lambda: f32)` — uses Q-head's expected value as the baseline (replaces GAE-λ) +- `pub fn finalize_returns(&mut self)` — done-aware cumulative return + +### D.4 Toy bandit test + +Same toy bandit as C.4: after 10k steps, π(action=5|s) > 0.9. + +### D.5 Commit +``` +git commit -m "feat(rl): PPO π/V heads + clipped surrogate + rollout (Phase D)" +``` + +--- + +## Phase E — LobSim integration + training loop (3 days) + +### E.1 LobSim adapter + +Create `crates/ml-alpha/src/rl/reward.rs`: +- Wraps `ml_backtesting::sim::LobSimCuda` +- `pub fn step(&mut self, action: u32, h_t: &CudaSlice) -> (next_h_t, reward, done)` +- Reward extraction: read `realized_pnl_usd` delta on trade close + +### E.2 Integrated trainer + +Create `crates/ml-alpha/src/trainer/integrated.rs`: +- `pub struct IntegratedTrainer { encoder, bce_head, q_head, pi_head, v_head, aux_head, replay, rollout, lobsim, loss_balance_controller }` +- `pub fn step(&mut self) -> Result` runs the training loop above (one event per call) +- All 5 heads' losses combined, single backward into encoder + +### E.3 End-to-end synthetic smoke + +Test in `crates/ml-alpha/tests/integrated_trainer.rs`: +- 1000 events of synthetic LobSim data (deterministic price walk) +- Run 1 epoch through integrated trainer +- Assert: no NaN/Inf in any loss, all 5 heads' gradients flowed (compare before/after weights), replay buffer non-empty + +### E.4 Commit +``` +git commit -m "feat(rl): LobSim integration + integrated trainer end-to-end (Phase E)" +``` + +--- + +## Phase F — Reward shaping calibration (2 days) + +### F.1 Reward components + +Implement in `reward.rs`: +- Base: `realized_pnl_usd` on trade close +- Penalty: `fees_usd` (already subtracted in summary) +- Penalty: `exit_path_cost` (slippage at close, computed from LobSim) +- Bonus (optional): `exposure_bonus = -|position_lots| × 0.0001 × dt_ns` (small penalty for sitting in inventory) + +### F.2 Reward scale calibration + +Calibrate the reward magnitude to match the V-head's expected range. Typical PnL per trade in USD ≈ 0.1-100. Scale: `reward_scaled = reward / 10.0` so V can fit in [-10, +10] which matches the C51 atom support `[-1, +1]` after standardization. + +### F.3 Sign-check test + +On a synthetic bull-market stream where price monotonically rises, train for 1k events. Assert: mean reward per closed trade > 0 when the policy converges to "long all the way." + +### F.4 Commit + +``` +git commit -m "feat(rl): reward shaping + calibration (Phase F)" +``` + +--- + +## Phase G — Argo workflow + dispatcher (1 day) + +### G.1 Template + +Create `infra/k8s/argo/alpha-rl-template.yaml`: +- Same structure as `alpha-perception-template.yaml` +- New params: `gamma` (0.99), `n-rollout-steps` (2048), `replay-size` (100000), `n-ppo-epochs` (4), `target-update-tau` (0.005), `target-update-freq` (100) +- Same `instrument-mode` and `multi-resolution` defaults as the perception template (front-month, 1:32) + +### G.2 Dispatcher script + +Create `scripts/argo-alpha-rl.sh`: mirrors `argo-alpha-perception.sh` with the new params. + +### G.3 Commit + +``` +git commit -m "feat(rl): Argo template + dispatcher script (Phase G)" +``` + +--- + +## Phase H — Backtest gate (1 day + 1.5h smoke) + +### H.1 Push + apply template + dispatch + +```bash +git push origin ml-alpha-phase-a +kubectl apply -n foxhunt -f infra/k8s/argo/alpha-rl-template.yaml +./scripts/argo-alpha-rl.sh --instrument-mode front-month +``` + +### H.2 Pull results + backtest + +After the workflow succeeds, pull the trained checkpoint: +- `/feature-cache/alpha-rl-runs/$SHA/integrated_best_h1000.bin` + +Update `config/ml/sweep_smoke_perhoriz_cfc.yaml` to point at the new checkpoint, dispatch `argo-lob-sweep.sh`. + +### H.3 Evaluate the gate + +| Metric | Failure threshold | Pass | +|--------|--------------------|------| +| profit_factor | < 0.5 → re-architect needed | > 1.0 | +| sharpe_ann | < -3 → re-architect needed | > 0.5 | +| outcome_by_entry_conv | mono-anti-cal pattern persists | Higher conviction = higher mean_pnl | + +### H.4 Document outcome + +- Pass: write `pearl_integrated_rl_works.md`, plan production deployment +- Fail: document specific failure mode, decide whether to re-scope (different action space, different reward, different action grid) + +--- + +## Self-review checklist + +**Spec coverage:** +- [x] Encoder gradient from all 5 heads (Phase B) +- [x] DQN distributional Q + PER replay + Bellman (Phase C) +- [x] PPO π/V + clipped surrogate + rollout (Phase D) +- [x] LobSim integration with per-trade reward (Phase E) +- [x] Reward shaping (Phase F) +- [x] Argo workflows (Phase G) +- [x] Backtest gate G8 (Phase H) + +**Risk analysis:** +- **Highest risk**: encoder gradient instability when 5 heads compete for shape. Mitigation: loss-balance controller (already in memory pearls) adaptively scales λ_k. +- **Second highest**: replay buffer GPU memory (100k transitions × 128 floats h_t × 4 bytes × 2 (h_t and next_h_t) ≈ 100 MB). Fits on L40S. PER sum-tree is CPU. Tradeoff. +- **Third**: PPO + DQN can diverge if both heads disagree strongly. Mitigation: monitor `argmax_Q vs argmax_π` agreement rate; if < 50%, raise alarm and pause training. + +**Greenfield commitment:** No backward-compat with old DQN in `crates/ml/src/trainers/dqn/`. That code stays for the old policy (different action space). The new integrated trainer is ml-alpha-specific. + +**Project policy compliance:** +- `feedback_no_partial_refactor`: each phase is atomic; subagents land complete files per phase. +- `feedback_no_feature_flags`: no `if use_rl` toggle — `alpha_rl_train` is the new canonical entry. +- `feedback_no_htod_htoh_only_mapped_pinned`: all CPU↔GPU via mapped-pinned (per pre-existing infra). +- `feedback_no_stubs`: no return-zero Q-heads or placeholder PPO loss. +- `pearl_thompson_for_distributional_action_selection`: Thompson at rollout, argmax only at eval. +- `pearl_c51_thompson_closed_phase_e3_gap`: C51 + Thompson is the proven combo. +- `pearl_loss_balance_controller`: existing controller handles λ_k for 5 heads. +- `pearl_cooperative_staging_eliminates_redundant_reads`: encoder forward once, all 5 heads share `h_t`. + +**Falsifiability:** Each phase has a gate. G3/G4 prove DQN/PPO work in isolation. G5/G6 prove integration. G8 is the actual trading-readiness gate that the prior architecture failed. + +**Calendar estimate:** ~3 weeks (16 dev days + 1.5h Argo smoke). With subagent parallelism in Phases C and D (independent algorithms), can compress to ~2.5 weeks.