# DQN Concepts Adoption for ml-alpha

**Goal:** Port high-value techniques from the old DQN trainer (`crates/ml/`) into the ml-alpha integrated RL trainer, improving training stability, OOS robustness, and convergence speed.

---

## Priority 1: Adopt Immediately

### P1.1: PopArt Reward Normalization

**Problem:** ml-alpha uses simple `reward_scale = 1/mean_abs_pnl` which drifts with market regime changes and produces bimodal/heavy-tailed scaled rewards.

**Fix:** Adaptive normalization via running Welford mean + robust IQR (interquartile range). Normalize rewards in-place before TD targets. Auto-corrects V-head outputs when statistics shift (the "Art" in PopArt — Preserving Outputs Precisely, while Adaptively Rescaling Targets).

**Implementation:**
- New kernel: `rl_popart_normalize.cu` — per-batch `r_norm = (r - μ) / max(σ, ε)` with Welford-EMA μ/σ² in ISV slots
- Replaces `apply_reward_scale` kernel
- V-head output correction: `V_new = σ_old/σ_new × V_old + (μ_old - μ_new)/σ_new`
- ISV slots: POPART_MEAN, POPART_VAR, POPART_COUNT

### P1.2: Spectral Normalization + Decoupling

**Problem:** Q-head weights can grow unbounded → Q-value explosion → unstable training. PPO policy logits can become overconfident → collapsed entropy.

**Fix:**
- **Spectral norm:** Constrain weight matrices so σ_max(W) ≤ threshold. Apply to DQN Q-head + IQN projection weights.
- **Spectral decoupling:** L2 penalty on logit magnitudes (not weights). λ × ‖logits‖² added to Q-loss and π-loss.

**Implementation:**
- New kernel: `rl_spectral_norm.cu` — power iteration to estimate σ_max, then rescale W
- Launch once per Adam step (after weight update, before next forward)
- ISV-driven: SPECTRAL_NORM_MAX (default 3.0), SPECTRAL_DECOUPLE_LAMBDA (default 0.01)

### P1.3: Trade-Outcome Auxiliary Head (K=3 Classifier)

**Problem:** The Q-head learns value but not explicit trade outcome prediction. An auxiliary "will this trade be Profit/Timeout/Loss?" classifier provides direct supervision signal that's orthogonal to the TD learning.

**Fix:**
- 3-class softmax head on h_t: P(Profit), P(Timeout), P(Loss)
- Label: derived from actual trade outcome at close (available retrospectively)
- CE loss weighted by ISV lambda, gradient flows to encoder
- Pairs with HER: the backward hindsight peak tells us the "Profit" label even for losing trades

**Implementation:**
- New module: `crates/ml-alpha/src/rl/outcome_head.rs` — linear [HIDDEN_DIM → 3] + softmax
- New kernel: `rl_outcome_ce.cu` — fwd + bwd
- Labels stored per-batch: updated on trade close (done=1), persistent until next close
- ISV: OUTCOME_AUX_LAMBDA (default 0.1)

---

## Priority 2: Adopt Next Sprint

### P2.1: Enrichment E1 — Q-Value Bias Correction

**Problem:** C51/IQN Q-values systematically overestimate (optimism bias). The gap between predicted Q and actual realized PnL drifts over training.

**Fix:** Per-epoch measure `bias = mean(predicted_Q - actual_return)`. Apply additive correction clipped to ±10.0 to Bellman targets. Shrinks the bias signal the Q-head chases.

**Implementation:**
- Accumulate (Q_predicted, actual_return) pairs on device
- New kernel: `rl_q_bias_correction.cu` — reduce mean, emit correction to ISV
- Applied in `bellman_target_projection` as `target -= bias_correction`

### P2.2: Enrichment E4 — Per-Branch LR Scaling

**Problem:** Different heads (Q, π, V, IQN) have different error rates. A single LR controller can't optimize all simultaneously — the loudest gradient wins.

**Fix:** Per-head error autopsy: measure per-head loss-improvement rate. Scale LR by `[0.5, 2.0]` relative to base. Heads that are learning fast get LR reduced; stalled heads get boosted.

**Implementation:**
- Already have per-head grad norms in ISV (l2_norm of ss_q_grad_w_d etc.)
- New controller: compare per-head loss EMA improvement vs target rate
- Emit per-head LR multiplier to ISV (existing LR controller already reads these)

### P2.3: Enrichment E8 — Curriculum Weights

**Problem:** All training data is weighted equally. Some market segments (high volatility, news events) are harder to learn than others. Equal weighting wastes capacity on already-learned easy segments.

**Fix:** Per-segment difficulty score = inverse Sharpe on that segment. z-score normalize → softmax → sampling weights. Harder segments sampled more often.

**Implementation:**
- Requires walk-forward evaluation (already exists)
- Per-fold: compute per-segment Sharpe from trade records
- Emit curriculum_weights to ISV or a device buffer
- GPU PER sample kernel reads weights and adjusts priority by segment

### P2.4: Adversarial Regime Injection

**Problem:** Training on favorable data makes the agent brittle to drawdowns. When real markets turn hostile, the agent's learned policy collapses.

**Fix:** On epochs where max drawdown is below threshold (agent is "too comfortable"), inject adversarial market conditions: synthetic spread widening, slippage increase, or replay worst-case historical segments with boosted priority.

**Implementation:**
- Monitor max_dd per epoch via diag
- When max_dd < threshold: boost PER priority for high-volatility segments
- ISV-driven: ADVERSARIAL_DD_THRESHOLD, ADVERSARIAL_BOOST

---

## Constraints

All implementations must follow:
- `feedback_no_atomicadd`: block tree-reduce only
- `feedback_cpu_is_read_only`: GPU kernels for all compute
- `feedback_no_htod_htoh_only_mapped_pinned`: mapped-pinned staging
- `feedback_isv_for_adaptive_bounds`: all thresholds/bounds ISV-driven
- `feedback_no_stubs`: wire in same commit
