From e0ea71e90ec989c3f53b97dedae09f936af37a02 Mon Sep 17 00:00:00 2001
From: jgrusewski <jgrusewski@localhost>
Date: Tue, 26 May 2026 08:59:51 +0200
Subject: [PATCH] =?UTF-8?q?spec+plan:=20DQN=20concepts=20adoption=20?=
 =?UTF-8?q?=E2=80=94=20PopArt,=20spectral=20norm,=20outcome=20head,=20enri?=
 =?UTF-8?q?chment=20E1-E8?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

7 tasks: PopArt normalization, spectral norm+decoupling, K=3 outcome
aux head, Q-bias correction, per-branch LR, curriculum weights,
adversarial regime injection.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---
 .../plans/2026-05-26-dqn-concepts-adoption.md | 126 ++++++++++++++++++
 .../specs/2026-05-26-dqn-concepts-adoption.md | 108 +++++++++++++++
 2 files changed, 234 insertions(+)
 create mode 100644 docs/superpowers/plans/2026-05-26-dqn-concepts-adoption.md
 create mode 100644 docs/superpowers/specs/2026-05-26-dqn-concepts-adoption.md

diff --git a/docs/superpowers/plans/2026-05-26-dqn-concepts-adoption.md b/docs/superpowers/plans/2026-05-26-dqn-concepts-adoption.md
new file mode 100644
index 000000000..935159930
--- /dev/null
+++ b/docs/superpowers/plans/2026-05-26-dqn-concepts-adoption.md
@@ -0,0 +1,126 @@
+# DQN Concepts Adoption Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Port PopArt normalization, spectral regularization, trade-outcome aux head, Q-bias correction, per-branch LR, curriculum weights, and adversarial regime injection from the old DQN trainer.
+
+**Architecture:** Each feature is a self-contained module (struct + kernel + ISV slots + trainer wiring). Features are independent and can be enabled/disabled via ISV.
+
+**Tech Stack:** Rust 1.85, cudarc 0.19, CUDA 12.4, pre-compiled cubins
+
+---
+
+## Priority 1 Tasks
+
+### Task 1: PopArt Reward Normalization
+
+**Files:**
+- Create: `crates/ml-alpha/cuda/rl_popart_normalize.cu`
+- Create: `crates/ml-alpha/src/rl/popart.rs`
+- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
+- Modify: `crates/ml-alpha/build.rs`
+- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
+
+- [ ] **Step 1:** Add ISV slots: POPART_MEAN (553), POPART_VAR (554), POPART_COUNT (555), POPART_SIGMA_FLOOR (556)
+- [ ] **Step 2:** Write `rl_popart_normalize.cu` — per-batch Welford-EMA update of (μ, σ²), then normalize: `r_out = (r - μ) / max(σ, floor)`
+- [ ] **Step 3:** Write `popart.rs` — PopArt struct with V-head output correction method
+- [ ] **Step 4:** Wire into trainer: replace `apply_reward_scale` with PopArt normalize, add V-correction after V-head forward
+- [ ] **Step 5:** ISV bootstrap: mean=0, var=1, count=0, floor=1e-3
+- [ ] **Step 6:** Smoke test + commit
+
+### Task 2: Spectral Normalization + Decoupling
+
+**Files:**
+- Create: `crates/ml-alpha/cuda/rl_spectral_norm.cu`
+- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
+- Modify: `crates/ml-alpha/build.rs`
+- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
+
+- [ ] **Step 1:** Add ISV slots: SPECTRAL_NORM_MAX (557), SPECTRAL_DECOUPLE_LAMBDA (558)
+- [ ] **Step 2:** Write `rl_spectral_norm.cu` — power iteration for σ_max estimate + rescale W if σ_max > threshold. Single-block kernel per weight matrix.
+- [ ] **Step 3:** Wire: launch after each Adam step on DQN w_d, IQN w_out_d, policy w_d
+- [ ] **Step 4:** Add spectral decoupling loss term: `λ × mean(logits²)` added to Q-loss and π-loss in the existing loss kernels
+- [ ] **Step 5:** ISV bootstrap: norm_max=3.0, lambda=0.01
+- [ ] **Step 6:** Smoke test + commit
+
+### Task 3: Trade-Outcome Auxiliary Head (K=3)
+
+**Files:**
+- Create: `crates/ml-alpha/src/rl/outcome_head.rs`
+- Create: `crates/ml-alpha/cuda/rl_outcome_ce.cu`
+- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
+- Modify: `crates/ml-alpha/build.rs`
+- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
+- Modify: `crates/ml-alpha/examples/alpha_rl_train.rs`
+
+- [ ] **Step 1:** Add ISV slots: OUTCOME_AUX_LAMBDA (559), OUTCOME_LABEL_INDEX (560)
+- [ ] **Step 2:** Write `outcome_head.rs` — linear [HIDDEN_DIM → 3] with Xavier init, forward + backward
+- [ ] **Step 3:** Write `rl_outcome_ce.cu` — per-batch softmax CE loss + grad for 3-class outcome. Label = {0=Loss, 1=Timeout, 2=Profit}. Sentinel -1 = no label yet.
+- [ ] **Step 4:** Add outcome label tracking: on trade close (done=1), set label = 2 if PnL > 0, 0 if PnL < 0, 1 if timed out (steps_since_done >= max_hold)
+- [ ] **Step 5:** Wire into step_synthetic: outcome CE loss + backward, gradient to encoder
+- [ ] **Step 6:** Add to diag JSONL: outcome_ce_loss, outcome_accuracy per class
+- [ ] **Step 7:** ISV bootstrap: lambda=0.1
+- [ ] **Step 8:** Smoke test + commit
+
+---
+
+## Priority 2 Tasks
+
+### Task 4: Q-Value Bias Correction (E1)
+
+**Files:**
+- Create: `crates/ml-alpha/cuda/rl_q_bias.cu`
+- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
+- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
+
+- [ ] **Step 1:** Add ISV slots: Q_BIAS_CORRECTION (561), Q_BIAS_EMA (562)
+- [ ] **Step 2:** Write kernel: accumulate (Q_predicted - actual_return) EMA → emit correction to ISV
+- [ ] **Step 3:** Wire: apply correction in bellman_target_projection, clip ±10.0
+- [ ] **Step 4:** Commit
+
+### Task 5: Per-Branch LR Scaling (E4)
+
+**Files:**
+- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
+- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
+
+- [ ] **Step 1:** Add ISV slots: LR_SCALE_Q (563), LR_SCALE_PI (564), LR_SCALE_V (565), LR_SCALE_IQN (566)
+- [ ] **Step 2:** After each step: compute per-head loss improvement rate = (loss_ema_prev - loss_ema_now) / loss_ema_prev
+- [ ] **Step 3:** Emit LR scale: heads improving fast → scale down (0.5), stalled → scale up (2.0)
+- [ ] **Step 4:** Apply in existing LR controller reads
+- [ ] **Step 5:** Commit
+
+### Task 6: Curriculum Weights (E8)
+
+**Files:**
+- Create: `crates/ml-alpha/cuda/rl_curriculum_weights.cu`
+- Modify: `crates/ml-alpha/src/rl/gpu_replay.rs`
+- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
+
+- [ ] **Step 1:** Add per-transition segment_id to replay scalars (extend SCALARS_PER_TRANSITION to 8)
+- [ ] **Step 2:** Write kernel: per-segment Sharpe → difficulty weight → PER priority multiplier
+- [ ] **Step 3:** Integrate with rl_per_sample: multiply priority by segment weight
+- [ ] **Step 4:** Commit
+
+### Task 7: Adversarial Regime Injection
+
+**Files:**
+- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
+- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
+
+- [ ] **Step 1:** Add ISV slots: ADVERSARIAL_DD_THRESHOLD (567), ADVERSARIAL_BOOST (568), MAX_DD_EMA (569)
+- [ ] **Step 2:** Track max drawdown EMA from cumulative PnL per epoch
+- [ ] **Step 3:** When DD < threshold: boost PER priority for recent losing transitions by ADVERSARIAL_BOOST
+- [ ] **Step 4:** Commit
+
+---
+
+## Kill Criteria
+
+- PopArt: reward variance stable across regime changes (var_ema doesn't spike > 10× on file transitions)
+- Spectral norm: σ_max stays below threshold throughout training (no Q-explosion)
+- Outcome head: CE loss drops below random baseline (ln(3) = 1.1) within 10k steps
+- Q-bias: |bias_correction| stays < 5.0 (not fighting a structural problem)
+- Per-branch LR: no head's loss diverges while others converge
+- Curriculum: hard segments get 2-3× sampling weight, easy segments <0.5×
+- Adversarial: max_dd after 100k steps is smaller than non-adversarial baseline
diff --git a/docs/superpowers/specs/2026-05-26-dqn-concepts-adoption.md b/docs/superpowers/specs/2026-05-26-dqn-concepts-adoption.md
new file mode 100644
index 000000000..593e93065
--- /dev/null
+++ b/docs/superpowers/specs/2026-05-26-dqn-concepts-adoption.md
@@ -0,0 +1,108 @@
+# DQN Concepts Adoption for ml-alpha
+
+**Goal:** Port high-value techniques from the old DQN trainer (`crates/ml/`) into the ml-alpha integrated RL trainer, improving training stability, OOS robustness, and convergence speed.
+
+---
+
+## Priority 1: Adopt Immediately
+
+### P1.1: PopArt Reward Normalization
+
+**Problem:** ml-alpha uses simple `reward_scale = 1/mean_abs_pnl` which drifts with market regime changes and produces bimodal/heavy-tailed scaled rewards.
+
+**Fix:** Adaptive normalization via running Welford mean + robust IQR (interquartile range). Normalize rewards in-place before TD targets. Auto-corrects V-head outputs when statistics shift (the "Art" in PopArt — Preserving Outputs Precisely, while Adaptively Rescaling Targets).
+
+**Implementation:**
+- New kernel: `rl_popart_normalize.cu` — per-batch `r_norm = (r - μ) / max(σ, ε)` with Welford-EMA μ/σ² in ISV slots
+- Replaces `apply_reward_scale` kernel
+- V-head output correction: `V_new = σ_old/σ_new × V_old + (μ_old - μ_new)/σ_new`
+- ISV slots: POPART_MEAN, POPART_VAR, POPART_COUNT
+
+### P1.2: Spectral Normalization + Decoupling
+
+**Problem:** Q-head weights can grow unbounded → Q-value explosion → unstable training. PPO policy logits can become overconfident → collapsed entropy.
+
+**Fix:**
+- **Spectral norm:** Constrain weight matrices so σ_max(W) ≤ threshold. Apply to DQN Q-head + IQN projection weights.
+- **Spectral decoupling:** L2 penalty on logit magnitudes (not weights). λ × ‖logits‖² added to Q-loss and π-loss.
+
+**Implementation:**
+- New kernel: `rl_spectral_norm.cu` — power iteration to estimate σ_max, then rescale W
+- Launch once per Adam step (after weight update, before next forward)
+- ISV-driven: SPECTRAL_NORM_MAX (default 3.0), SPECTRAL_DECOUPLE_LAMBDA (default 0.01)
+
+### P1.3: Trade-Outcome Auxiliary Head (K=3 Classifier)
+
+**Problem:** The Q-head learns value but not explicit trade outcome prediction. An auxiliary "will this trade be Profit/Timeout/Loss?" classifier provides direct supervision signal that's orthogonal to the TD learning.
+
+**Fix:**
+- 3-class softmax head on h_t: P(Profit), P(Timeout), P(Loss)
+- Label: derived from actual trade outcome at close (available retrospectively)
+- CE loss weighted by ISV lambda, gradient flows to encoder
+- Pairs with HER: the backward hindsight peak tells us the "Profit" label even for losing trades
+
+**Implementation:**
+- New module: `crates/ml-alpha/src/rl/outcome_head.rs` — linear [HIDDEN_DIM → 3] + softmax
+- New kernel: `rl_outcome_ce.cu` — fwd + bwd
+- Labels stored per-batch: updated on trade close (done=1), persistent until next close
+- ISV: OUTCOME_AUX_LAMBDA (default 0.1)
+
+---
+
+## Priority 2: Adopt Next Sprint
+
+### P2.1: Enrichment E1 — Q-Value Bias Correction
+
+**Problem:** C51/IQN Q-values systematically overestimate (optimism bias). The gap between predicted Q and actual realized PnL drifts over training.
+
+**Fix:** Per-epoch measure `bias = mean(predicted_Q - actual_return)`. Apply additive correction clipped to ±10.0 to Bellman targets. Shrinks the bias signal the Q-head chases.
+
+**Implementation:**
+- Accumulate (Q_predicted, actual_return) pairs on device
+- New kernel: `rl_q_bias_correction.cu` — reduce mean, emit correction to ISV
+- Applied in `bellman_target_projection` as `target -= bias_correction`
+
+### P2.2: Enrichment E4 — Per-Branch LR Scaling
+
+**Problem:** Different heads (Q, π, V, IQN) have different error rates. A single LR controller can't optimize all simultaneously — the loudest gradient wins.
+
+**Fix:** Per-head error autopsy: measure per-head loss-improvement rate. Scale LR by `[0.5, 2.0]` relative to base. Heads that are learning fast get LR reduced; stalled heads get boosted.
+
+**Implementation:**
+- Already have per-head grad norms in ISV (l2_norm of ss_q_grad_w_d etc.)
+- New controller: compare per-head loss EMA improvement vs target rate
+- Emit per-head LR multiplier to ISV (existing LR controller already reads these)
+
+### P2.3: Enrichment E8 — Curriculum Weights
+
+**Problem:** All training data is weighted equally. Some market segments (high volatility, news events) are harder to learn than others. Equal weighting wastes capacity on already-learned easy segments.
+
+**Fix:** Per-segment difficulty score = inverse Sharpe on that segment. z-score normalize → softmax → sampling weights. Harder segments sampled more often.
+
+**Implementation:**
+- Requires walk-forward evaluation (already exists)
+- Per-fold: compute per-segment Sharpe from trade records
+- Emit curriculum_weights to ISV or a device buffer
+- GPU PER sample kernel reads weights and adjusts priority by segment
+
+### P2.4: Adversarial Regime Injection
+
+**Problem:** Training on favorable data makes the agent brittle to drawdowns. When real markets turn hostile, the agent's learned policy collapses.
+
+**Fix:** On epochs where max drawdown is below threshold (agent is "too comfortable"), inject adversarial market conditions: synthetic spread widening, slippage increase, or replay worst-case historical segments with boosted priority.
+
+**Implementation:**
+- Monitor max_dd per epoch via diag
+- When max_dd < threshold: boost PER priority for high-volatility segments
+- ISV-driven: ADVERSARIAL_DD_THRESHOLD, ADVERSARIAL_BOOST
+
+---
+
+## Constraints
+
+All implementations must follow:
+- `feedback_no_atomicadd`: block tree-reduce only
+- `feedback_cpu_is_read_only`: GPU kernels for all compute
+- `feedback_no_htod_htoh_only_mapped_pinned`: mapped-pinned staging
+- `feedback_isv_for_adaptive_bounds`: all thresholds/bounds ISV-driven
+- `feedback_no_stubs`: wire in same commit