# DQN Concepts Adoption Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Port PopArt normalization, spectral regularization, trade-outcome aux head, Q-bias correction, per-branch LR, curriculum weights, and adversarial regime injection from the old DQN trainer.

**Architecture:** Each feature is a self-contained module (struct + kernel + ISV slots + trainer wiring). Features are independent and can be enabled/disabled via ISV.

**Tech Stack:** Rust 1.85, cudarc 0.19, CUDA 12.4, pre-compiled cubins

---

## Priority 1 Tasks

### Task 1: PopArt Reward Normalization

**Files:**
- Create: `crates/ml-alpha/cuda/rl_popart_normalize.cu`
- Create: `crates/ml-alpha/src/rl/popart.rs`
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
- Modify: `crates/ml-alpha/build.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`

- [ ] **Step 1:** Add ISV slots: POPART_MEAN (553), POPART_VAR (554), POPART_COUNT (555), POPART_SIGMA_FLOOR (556)
- [ ] **Step 2:** Write `rl_popart_normalize.cu` — per-batch Welford-EMA update of (μ, σ²), then normalize: `r_out = (r - μ) / max(σ, floor)`
- [ ] **Step 3:** Write `popart.rs` — PopArt struct with V-head output correction method
- [ ] **Step 4:** Wire into trainer: replace `apply_reward_scale` with PopArt normalize, add V-correction after V-head forward
- [ ] **Step 5:** ISV bootstrap: mean=0, var=1, count=0, floor=1e-3
- [ ] **Step 6:** Smoke test + commit

### Task 2: Spectral Normalization + Decoupling

**Files:**
- Create: `crates/ml-alpha/cuda/rl_spectral_norm.cu`
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
- Modify: `crates/ml-alpha/build.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`

- [ ] **Step 1:** Add ISV slots: SPECTRAL_NORM_MAX (557), SPECTRAL_DECOUPLE_LAMBDA (558)
- [ ] **Step 2:** Write `rl_spectral_norm.cu` — power iteration for σ_max estimate + rescale W if σ_max > threshold. Single-block kernel per weight matrix.
- [ ] **Step 3:** Wire: launch after each Adam step on DQN w_d, IQN w_out_d, policy w_d
- [ ] **Step 4:** Add spectral decoupling loss term: `λ × mean(logits²)` added to Q-loss and π-loss in the existing loss kernels
- [ ] **Step 5:** ISV bootstrap: norm_max=3.0, lambda=0.01
- [ ] **Step 6:** Smoke test + commit

### Task 3: Trade-Outcome Auxiliary Head (K=3)

**Files:**
- Create: `crates/ml-alpha/src/rl/outcome_head.rs`
- Create: `crates/ml-alpha/cuda/rl_outcome_ce.cu`
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
- Modify: `crates/ml-alpha/build.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
- Modify: `crates/ml-alpha/examples/alpha_rl_train.rs`

- [ ] **Step 1:** Add ISV slots: OUTCOME_AUX_LAMBDA (559), OUTCOME_LABEL_INDEX (560)
- [ ] **Step 2:** Write `outcome_head.rs` — linear [HIDDEN_DIM → 3] with Xavier init, forward + backward
- [ ] **Step 3:** Write `rl_outcome_ce.cu` — per-batch softmax CE loss + grad for 3-class outcome. Label = {0=Loss, 1=Timeout, 2=Profit}. Sentinel -1 = no label yet.
- [ ] **Step 4:** Add outcome label tracking: on trade close (done=1), set label = 2 if PnL > 0, 0 if PnL < 0, 1 if timed out (steps_since_done >= max_hold)
- [ ] **Step 5:** Wire into step_synthetic: outcome CE loss + backward, gradient to encoder
- [ ] **Step 6:** Add to diag JSONL: outcome_ce_loss, outcome_accuracy per class
- [ ] **Step 7:** ISV bootstrap: lambda=0.1
- [ ] **Step 8:** Smoke test + commit

---

## Priority 2 Tasks

### Task 4: Q-Value Bias Correction (E1)

**Files:**
- Create: `crates/ml-alpha/cuda/rl_q_bias.cu`
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`

- [ ] **Step 1:** Add ISV slots: Q_BIAS_CORRECTION (561), Q_BIAS_EMA (562)
- [ ] **Step 2:** Write kernel: accumulate (Q_predicted - actual_return) EMA → emit correction to ISV
- [ ] **Step 3:** Wire: apply correction in bellman_target_projection, clip ±10.0
- [ ] **Step 4:** Commit

### Task 5: Per-Branch LR Scaling (E4)

**Files:**
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`

- [ ] **Step 1:** Add ISV slots: LR_SCALE_Q (563), LR_SCALE_PI (564), LR_SCALE_V (565), LR_SCALE_IQN (566)
- [ ] **Step 2:** After each step: compute per-head loss improvement rate = (loss_ema_prev - loss_ema_now) / loss_ema_prev
- [ ] **Step 3:** Emit LR scale: heads improving fast → scale down (0.5), stalled → scale up (2.0)
- [ ] **Step 4:** Apply in existing LR controller reads
- [ ] **Step 5:** Commit

### Task 6: Curriculum Weights (E8)

**Files:**
- Create: `crates/ml-alpha/cuda/rl_curriculum_weights.cu`
- Modify: `crates/ml-alpha/src/rl/gpu_replay.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`

- [ ] **Step 1:** Add per-transition segment_id to replay scalars (extend SCALARS_PER_TRANSITION to 8)
- [ ] **Step 2:** Write kernel: per-segment Sharpe → difficulty weight → PER priority multiplier
- [ ] **Step 3:** Integrate with rl_per_sample: multiply priority by segment weight
- [ ] **Step 4:** Commit

### Task 7: Adversarial Regime Injection

**Files:**
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`

- [ ] **Step 1:** Add ISV slots: ADVERSARIAL_DD_THRESHOLD (567), ADVERSARIAL_BOOST (568), MAX_DD_EMA (569)
- [ ] **Step 2:** Track max drawdown EMA from cumulative PnL per epoch
- [ ] **Step 3:** When DD < threshold: boost PER priority for recent losing transitions by ADVERSARIAL_BOOST
- [ ] **Step 4:** Commit

---

## Kill Criteria

- PopArt: reward variance stable across regime changes (var_ema doesn't spike > 10× on file transitions)
- Spectral norm: σ_max stays below threshold throughout training (no Q-explosion)
- Outcome head: CE loss drops below random baseline (ln(3) = 1.1) within 10k steps
- Q-bias: |bias_correction| stays < 5.0 (not fighting a structural problem)
- Per-branch LR: no head's loss diverges while others converge
- Curriculum: hard segments get 2-3× sampling weight, easy segments <0.5×
- Adversarial: max_dd after 100k steps is smaller than non-adversarial baseline
