spec+plan: DQN concepts adoption — PopArt, spectral norm, outcome head, enrichment E1-E8
7 tasks: PopArt normalization, spectral norm+decoupling, K=3 outcome aux head, Q-bias correction, per-branch LR, curriculum weights, adversarial regime injection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
126
docs/superpowers/plans/2026-05-26-dqn-concepts-adoption.md
Normal file
126
docs/superpowers/plans/2026-05-26-dqn-concepts-adoption.md
Normal file
@@ -0,0 +1,126 @@
|
||||
# DQN Concepts Adoption Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Port PopArt normalization, spectral regularization, trade-outcome aux head, Q-bias correction, per-branch LR, curriculum weights, and adversarial regime injection from the old DQN trainer.
|
||||
|
||||
**Architecture:** Each feature is a self-contained module (struct + kernel + ISV slots + trainer wiring). Features are independent and can be enabled/disabled via ISV.
|
||||
|
||||
**Tech Stack:** Rust 1.85, cudarc 0.19, CUDA 12.4, pre-compiled cubins
|
||||
|
||||
---
|
||||
|
||||
## Priority 1 Tasks
|
||||
|
||||
### Task 1: PopArt Reward Normalization
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-alpha/cuda/rl_popart_normalize.cu`
|
||||
- Create: `crates/ml-alpha/src/rl/popart.rs`
|
||||
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
|
||||
- Modify: `crates/ml-alpha/build.rs`
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
|
||||
|
||||
- [ ] **Step 1:** Add ISV slots: POPART_MEAN (553), POPART_VAR (554), POPART_COUNT (555), POPART_SIGMA_FLOOR (556)
|
||||
- [ ] **Step 2:** Write `rl_popart_normalize.cu` — per-batch Welford-EMA update of (μ, σ²), then normalize: `r_out = (r - μ) / max(σ, floor)`
|
||||
- [ ] **Step 3:** Write `popart.rs` — PopArt struct with V-head output correction method
|
||||
- [ ] **Step 4:** Wire into trainer: replace `apply_reward_scale` with PopArt normalize, add V-correction after V-head forward
|
||||
- [ ] **Step 5:** ISV bootstrap: mean=0, var=1, count=0, floor=1e-3
|
||||
- [ ] **Step 6:** Smoke test + commit
|
||||
|
||||
### Task 2: Spectral Normalization + Decoupling
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-alpha/cuda/rl_spectral_norm.cu`
|
||||
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
|
||||
- Modify: `crates/ml-alpha/build.rs`
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
|
||||
|
||||
- [ ] **Step 1:** Add ISV slots: SPECTRAL_NORM_MAX (557), SPECTRAL_DECOUPLE_LAMBDA (558)
|
||||
- [ ] **Step 2:** Write `rl_spectral_norm.cu` — power iteration for σ_max estimate + rescale W if σ_max > threshold. Single-block kernel per weight matrix.
|
||||
- [ ] **Step 3:** Wire: launch after each Adam step on DQN w_d, IQN w_out_d, policy w_d
|
||||
- [ ] **Step 4:** Add spectral decoupling loss term: `λ × mean(logits²)` added to Q-loss and π-loss in the existing loss kernels
|
||||
- [ ] **Step 5:** ISV bootstrap: norm_max=3.0, lambda=0.01
|
||||
- [ ] **Step 6:** Smoke test + commit
|
||||
|
||||
### Task 3: Trade-Outcome Auxiliary Head (K=3)
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-alpha/src/rl/outcome_head.rs`
|
||||
- Create: `crates/ml-alpha/cuda/rl_outcome_ce.cu`
|
||||
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
|
||||
- Modify: `crates/ml-alpha/build.rs`
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
|
||||
- Modify: `crates/ml-alpha/examples/alpha_rl_train.rs`
|
||||
|
||||
- [ ] **Step 1:** Add ISV slots: OUTCOME_AUX_LAMBDA (559), OUTCOME_LABEL_INDEX (560)
|
||||
- [ ] **Step 2:** Write `outcome_head.rs` — linear [HIDDEN_DIM → 3] with Xavier init, forward + backward
|
||||
- [ ] **Step 3:** Write `rl_outcome_ce.cu` — per-batch softmax CE loss + grad for 3-class outcome. Label = {0=Loss, 1=Timeout, 2=Profit}. Sentinel -1 = no label yet.
|
||||
- [ ] **Step 4:** Add outcome label tracking: on trade close (done=1), set label = 2 if PnL > 0, 0 if PnL < 0, 1 if timed out (steps_since_done >= max_hold)
|
||||
- [ ] **Step 5:** Wire into step_synthetic: outcome CE loss + backward, gradient to encoder
|
||||
- [ ] **Step 6:** Add to diag JSONL: outcome_ce_loss, outcome_accuracy per class
|
||||
- [ ] **Step 7:** ISV bootstrap: lambda=0.1
|
||||
- [ ] **Step 8:** Smoke test + commit
|
||||
|
||||
---
|
||||
|
||||
## Priority 2 Tasks
|
||||
|
||||
### Task 4: Q-Value Bias Correction (E1)
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-alpha/cuda/rl_q_bias.cu`
|
||||
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
|
||||
|
||||
- [ ] **Step 1:** Add ISV slots: Q_BIAS_CORRECTION (561), Q_BIAS_EMA (562)
|
||||
- [ ] **Step 2:** Write kernel: accumulate (Q_predicted - actual_return) EMA → emit correction to ISV
|
||||
- [ ] **Step 3:** Wire: apply correction in bellman_target_projection, clip ±10.0
|
||||
- [ ] **Step 4:** Commit
|
||||
|
||||
### Task 5: Per-Branch LR Scaling (E4)
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
|
||||
|
||||
- [ ] **Step 1:** Add ISV slots: LR_SCALE_Q (563), LR_SCALE_PI (564), LR_SCALE_V (565), LR_SCALE_IQN (566)
|
||||
- [ ] **Step 2:** After each step: compute per-head loss improvement rate = (loss_ema_prev - loss_ema_now) / loss_ema_prev
|
||||
- [ ] **Step 3:** Emit LR scale: heads improving fast → scale down (0.5), stalled → scale up (2.0)
|
||||
- [ ] **Step 4:** Apply in existing LR controller reads
|
||||
- [ ] **Step 5:** Commit
|
||||
|
||||
### Task 6: Curriculum Weights (E8)
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-alpha/cuda/rl_curriculum_weights.cu`
|
||||
- Modify: `crates/ml-alpha/src/rl/gpu_replay.rs`
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
|
||||
|
||||
- [ ] **Step 1:** Add per-transition segment_id to replay scalars (extend SCALARS_PER_TRANSITION to 8)
|
||||
- [ ] **Step 2:** Write kernel: per-segment Sharpe → difficulty weight → PER priority multiplier
|
||||
- [ ] **Step 3:** Integrate with rl_per_sample: multiply priority by segment weight
|
||||
- [ ] **Step 4:** Commit
|
||||
|
||||
### Task 7: Adversarial Regime Injection
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml-alpha/src/rl/isv_slots.rs`
|
||||
- Modify: `crates/ml-alpha/src/trainer/integrated.rs`
|
||||
|
||||
- [ ] **Step 1:** Add ISV slots: ADVERSARIAL_DD_THRESHOLD (567), ADVERSARIAL_BOOST (568), MAX_DD_EMA (569)
|
||||
- [ ] **Step 2:** Track max drawdown EMA from cumulative PnL per epoch
|
||||
- [ ] **Step 3:** When DD < threshold: boost PER priority for recent losing transitions by ADVERSARIAL_BOOST
|
||||
- [ ] **Step 4:** Commit
|
||||
|
||||
---
|
||||
|
||||
## Kill Criteria
|
||||
|
||||
- PopArt: reward variance stable across regime changes (var_ema doesn't spike > 10× on file transitions)
|
||||
- Spectral norm: σ_max stays below threshold throughout training (no Q-explosion)
|
||||
- Outcome head: CE loss drops below random baseline (ln(3) = 1.1) within 10k steps
|
||||
- Q-bias: |bias_correction| stays < 5.0 (not fighting a structural problem)
|
||||
- Per-branch LR: no head's loss diverges while others converge
|
||||
- Curriculum: hard segments get 2-3× sampling weight, easy segments <0.5×
|
||||
- Adversarial: max_dd after 100k steps is smaller than non-adversarial baseline
|
||||
108
docs/superpowers/specs/2026-05-26-dqn-concepts-adoption.md
Normal file
108
docs/superpowers/specs/2026-05-26-dqn-concepts-adoption.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# DQN Concepts Adoption for ml-alpha
|
||||
|
||||
**Goal:** Port high-value techniques from the old DQN trainer (`crates/ml/`) into the ml-alpha integrated RL trainer, improving training stability, OOS robustness, and convergence speed.
|
||||
|
||||
---
|
||||
|
||||
## Priority 1: Adopt Immediately
|
||||
|
||||
### P1.1: PopArt Reward Normalization
|
||||
|
||||
**Problem:** ml-alpha uses simple `reward_scale = 1/mean_abs_pnl` which drifts with market regime changes and produces bimodal/heavy-tailed scaled rewards.
|
||||
|
||||
**Fix:** Adaptive normalization via running Welford mean + robust IQR (interquartile range). Normalize rewards in-place before TD targets. Auto-corrects V-head outputs when statistics shift (the "Art" in PopArt — Preserving Outputs Precisely, while Adaptively Rescaling Targets).
|
||||
|
||||
**Implementation:**
|
||||
- New kernel: `rl_popart_normalize.cu` — per-batch `r_norm = (r - μ) / max(σ, ε)` with Welford-EMA μ/σ² in ISV slots
|
||||
- Replaces `apply_reward_scale` kernel
|
||||
- V-head output correction: `V_new = σ_old/σ_new × V_old + (μ_old - μ_new)/σ_new`
|
||||
- ISV slots: POPART_MEAN, POPART_VAR, POPART_COUNT
|
||||
|
||||
### P1.2: Spectral Normalization + Decoupling
|
||||
|
||||
**Problem:** Q-head weights can grow unbounded → Q-value explosion → unstable training. PPO policy logits can become overconfident → collapsed entropy.
|
||||
|
||||
**Fix:**
|
||||
- **Spectral norm:** Constrain weight matrices so σ_max(W) ≤ threshold. Apply to DQN Q-head + IQN projection weights.
|
||||
- **Spectral decoupling:** L2 penalty on logit magnitudes (not weights). λ × ‖logits‖² added to Q-loss and π-loss.
|
||||
|
||||
**Implementation:**
|
||||
- New kernel: `rl_spectral_norm.cu` — power iteration to estimate σ_max, then rescale W
|
||||
- Launch once per Adam step (after weight update, before next forward)
|
||||
- ISV-driven: SPECTRAL_NORM_MAX (default 3.0), SPECTRAL_DECOUPLE_LAMBDA (default 0.01)
|
||||
|
||||
### P1.3: Trade-Outcome Auxiliary Head (K=3 Classifier)
|
||||
|
||||
**Problem:** The Q-head learns value but not explicit trade outcome prediction. An auxiliary "will this trade be Profit/Timeout/Loss?" classifier provides direct supervision signal that's orthogonal to the TD learning.
|
||||
|
||||
**Fix:**
|
||||
- 3-class softmax head on h_t: P(Profit), P(Timeout), P(Loss)
|
||||
- Label: derived from actual trade outcome at close (available retrospectively)
|
||||
- CE loss weighted by ISV lambda, gradient flows to encoder
|
||||
- Pairs with HER: the backward hindsight peak tells us the "Profit" label even for losing trades
|
||||
|
||||
**Implementation:**
|
||||
- New module: `crates/ml-alpha/src/rl/outcome_head.rs` — linear [HIDDEN_DIM → 3] + softmax
|
||||
- New kernel: `rl_outcome_ce.cu` — fwd + bwd
|
||||
- Labels stored per-batch: updated on trade close (done=1), persistent until next close
|
||||
- ISV: OUTCOME_AUX_LAMBDA (default 0.1)
|
||||
|
||||
---
|
||||
|
||||
## Priority 2: Adopt Next Sprint
|
||||
|
||||
### P2.1: Enrichment E1 — Q-Value Bias Correction
|
||||
|
||||
**Problem:** C51/IQN Q-values systematically overestimate (optimism bias). The gap between predicted Q and actual realized PnL drifts over training.
|
||||
|
||||
**Fix:** Per-epoch measure `bias = mean(predicted_Q - actual_return)`. Apply additive correction clipped to ±10.0 to Bellman targets. Shrinks the bias signal the Q-head chases.
|
||||
|
||||
**Implementation:**
|
||||
- Accumulate (Q_predicted, actual_return) pairs on device
|
||||
- New kernel: `rl_q_bias_correction.cu` — reduce mean, emit correction to ISV
|
||||
- Applied in `bellman_target_projection` as `target -= bias_correction`
|
||||
|
||||
### P2.2: Enrichment E4 — Per-Branch LR Scaling
|
||||
|
||||
**Problem:** Different heads (Q, π, V, IQN) have different error rates. A single LR controller can't optimize all simultaneously — the loudest gradient wins.
|
||||
|
||||
**Fix:** Per-head error autopsy: measure per-head loss-improvement rate. Scale LR by `[0.5, 2.0]` relative to base. Heads that are learning fast get LR reduced; stalled heads get boosted.
|
||||
|
||||
**Implementation:**
|
||||
- Already have per-head grad norms in ISV (l2_norm of ss_q_grad_w_d etc.)
|
||||
- New controller: compare per-head loss EMA improvement vs target rate
|
||||
- Emit per-head LR multiplier to ISV (existing LR controller already reads these)
|
||||
|
||||
### P2.3: Enrichment E8 — Curriculum Weights
|
||||
|
||||
**Problem:** All training data is weighted equally. Some market segments (high volatility, news events) are harder to learn than others. Equal weighting wastes capacity on already-learned easy segments.
|
||||
|
||||
**Fix:** Per-segment difficulty score = inverse Sharpe on that segment. z-score normalize → softmax → sampling weights. Harder segments sampled more often.
|
||||
|
||||
**Implementation:**
|
||||
- Requires walk-forward evaluation (already exists)
|
||||
- Per-fold: compute per-segment Sharpe from trade records
|
||||
- Emit curriculum_weights to ISV or a device buffer
|
||||
- GPU PER sample kernel reads weights and adjusts priority by segment
|
||||
|
||||
### P2.4: Adversarial Regime Injection
|
||||
|
||||
**Problem:** Training on favorable data makes the agent brittle to drawdowns. When real markets turn hostile, the agent's learned policy collapses.
|
||||
|
||||
**Fix:** On epochs where max drawdown is below threshold (agent is "too comfortable"), inject adversarial market conditions: synthetic spread widening, slippage increase, or replay worst-case historical segments with boosted priority.
|
||||
|
||||
**Implementation:**
|
||||
- Monitor max_dd per epoch via diag
|
||||
- When max_dd < threshold: boost PER priority for high-volatility segments
|
||||
- ISV-driven: ADVERSARIAL_DD_THRESHOLD, ADVERSARIAL_BOOST
|
||||
|
||||
---
|
||||
|
||||
## Constraints
|
||||
|
||||
All implementations must follow:
|
||||
- `feedback_no_atomicadd`: block tree-reduce only
|
||||
- `feedback_cpu_is_read_only`: GPU kernels for all compute
|
||||
- `feedback_no_htod_htoh_only_mapped_pinned`: mapped-pinned staging
|
||||
- `feedback_isv_for_adaptive_bounds`: all thresholds/bounds ISV-driven
|
||||
- `feedback_no_stubs`: wire in same commit
|
||||
Reference in New Issue
Block a user