diff --git a/docs/superpowers/specs/2026-05-01-sp5-magnitude-differentiation-and-eval-collapse-design.md b/docs/superpowers/specs/2026-05-01-sp5-magnitude-differentiation-and-eval-collapse-design.md new file mode 100644 index 000000000..c25c64b59 --- /dev/null +++ b/docs/superpowers/specs/2026-05-01-sp5-magnitude-differentiation-and-eval-collapse-design.md @@ -0,0 +1,222 @@ +# SP5: Comprehensive Per-Branch + Per-Group Adaptation Layer — Close All Known Hardcoded-Multiplier Deferrals + +**Date:** 2026-05-01 +**Status:** Design — awaiting user review before plan +**Branch:** to be created from main (HEAD `a60e7b092`) +**Triggering data:** `train-multi-seed-cv2mw` 50-epoch L40S baseline, terminated F0 ep10 after diagnostic-quality data captured. Plus all known deferrals from SP4 close-out, sweep audit, and project-history memory pearls. + +## Goal + +Close every known hardcoded-multiplier deferral in the DQN training loop in a single coherent project. Where SP4 made magnitude bounds adaptive, SP5 makes the entire learning machinery — atom resolution, loss-component budgets, exploration noise, optimizer hyperparameters, distributional quantile schedule, position-sizing safety floors, trail-stop thresholds, Thompson temperature — all per-branch / per-group ISV-driven via Pearls A+D. + +After SP5: zero hardcoded multipliers in any adaptive code path; every adaptive value lives on ISV with sentinel-bootstrap and Wiener-optimal smoothing. + +## Triggering findings + +From the 50-epoch baseline (commit `a60e7b092`, captured F0 epochs 4-9): + +| Symptom | Root cause | +|---|---| +| Magnitude head Q-values flat at O(1e-3), variances at O(1e-9 to 1e-8) | C51 atom span [-15, 15] with width 0.6 is 600× larger than mag Q-diff signal | +| `dist_q/h/f` freezes at exact 0.225/0.280/0.495 across 6+ epochs | NoisyNet σ=0.0226 dominates Q-diff=0.005 by 4-5× → action selection noise-dominated | +| `intent_dist` freezes at exact 0.250/0.500/0.250 (Bin(2,0.5)) | Suspicious binary decomposition in action_select kernel — needs investigation | +| Eval-mode argmax: trade_count=1 over 214,654 val bars | Q-variance below noise floor → argmax deterministically picks one action | +| `Sharpe=67` from one lucky trade | No kill-criterion catches `active_frac=1.0 with trade_count=1` degeneracy | + +Plus all SP4 close-out and sweep deferrals (see [Scope](#scope) below). + +## Scope + +| # | Component | Status | New ISV slots | +|---|---|---|---| +| 1-3 | Per-branch atom span / loss budget / NoisyNet σ | core SP5 | 52 | +| 4 | Per-param-group Adam β1/β2/ε ISV-driven | from SP4 close-out memory | 24 (3 × 8 param groups) | +| 5 | Per-branch IQN τ schedule | from SP5 out-of-scope | 20 (5 × 4 branches) | +| 6 | Kelly cap signal-driven floors | from project_magnitude_eval_collapse_kelly_capped.md | 6 | +| 7 | `dist_q/h/f Bin(2,0.5)` root-cause + action_select fix | new investigation | 0-8 | +| 8 | Trail stop signal-driven thresholds | new (Kelly-adjacent) | 6-8 | +| 9 | Thompson direction-branch temperature ISV-driven | from SP5 out-of-scope | 4 | +| 1-ext | Per-branch C51 num_atoms | extension of Pearl 1 | 4 | +| Layer A close-out | 5 host-EMA host→GPU migrations | from sweep audit | 5 sites (no slots) | +| Validation | 3-seed × 50-epoch harness as exit gate | from SP5 out-of-scope | (no code) | + +**Total: 120-128 new ISV slots, ~5000-7500 LOC, 11-13 producer kernels, ~12 consumer migrations.** + +After SP5: zero deferrals on hardcoded multipliers in DQN training loop. Every adaptive value ISV-driven. + +## Pearl design summaries + +### Pearl 1: `pearl_per_branch_c51_atom_span` + +Per-branch C51 atom positions (`v_center`, `v_half`) adapt to per-branch Q-magnitude. Magnitude head atoms shrink to ~[-0.025, 0.025] when Q-scale is 1e-3; direction head keeps wider span. Headroom factor itself ISV-driven via per-branch atom-clip rate (Pearl C-style 1% target rate). + +**Slots (16):** ATOM_V_CENTER[4], ATOM_V_HALF[4], ATOM_HEADROOM[4], ATOM_CLIP_RATE[4] + +### Pearl 1-ext: `pearl_per_branch_c51_num_atoms` + +Per-branch atom COUNT adapts to per-branch Q-resolution requirement. Direction branch can use fewer atoms with wider span; magnitude branch may need more atoms at narrow span. + +**Slots (4):** ATOM_NUM_ATOMS[4] +**Constraint:** total atom-tile shared memory bounded; sum across branches ≤ HW limit. + +### Pearl 2: `pearl_per_branch_loss_budget` + +Per-branch C51/IQN/CQL/Ens budgets driven by per-branch flatness signal `var(Q[b]) / σ[b]²`. IQN dominates when flat, C51 yields proportionally. + +**Slots (20):** BUDGET_{C51,IQN,CQL,ENS}[4], FLATNESS[4] + +### Pearl 3: `pearl_per_branch_noisy_sigma` + +Per-branch NoisyNet σ scales to per-branch Q-magnitude so noise/signal ratio stays constant. Sigma fraction itself ISV-driven via per-branch action-entropy gap (target 70% of max entropy as Invariant 1 anchor — only "structural" constant remaining at this layer). + +**Slots (12):** NOISY_SIGMA[4], SIGMA_FRACTION[4], BRANCH_ENTROPY[4] + +### Pearl 4: `pearl_per_group_adam_hyperparams` + +Per-param-group Adam β1, β2, ε ISV-driven (8 SP4 param groups: DqnTrunk/Value/Branches/Iqn/IqlHigh/IqlLow/Attn/Curiosity). Each group's Adam dynamics adapt to per-group gradient signal-to-noise. + +**Producer signal:** per-group gradient direction-stability EMA (cosine similarity of consecutive gradients). + +**Slots (24):** ADAM_BETA1[8], ADAM_BETA2[8], ADAM_EPS[8] + +### Pearl 5: `pearl_per_branch_iqn_tau_schedule` + +Per-branch IQN τ quantiles `{τ_05, τ_25, τ_50, τ_75, τ_95}` adapt to per-branch tail-thickness signal. Symmetric distributions can use uniform τ; skewed distributions concentrate τ in dense regions. + +**Producer signal:** per-branch Q-distribution skew + kurtosis EMA. + +**Slots (20):** IQN_TAU[4 branches × 5 quantiles] + +### Pearl 6: `pearl_kelly_cap_signal_driven_floors` + +Kelly cap removes hardcoded `0.5` warmup_floor and `0.5 + 0.5 × h` health_safety formula. Replaces with Pearls A+D smoothed kelly_f + conviction + trade-variance signals. Sentinel bootstrap handles cold-start naturally. + +**Slots (6):** KELLY_F_SMOOTH, CONVICTION_SMOOTH, TRADE_VAR_SMOOTH, KELLY_SAMPLE_COUNT, WIN_RATE_SMOOTH, LOSS_RATE_SMOOTH + +### Pearl 7: `pearl_action_select_decomposition_audit_or_fix` + +**Phase 7.1 (investigation):** root-cause why intent_dist freezes at exact `Bin(2, 0.5)` ratio (0.25/0.50/0.25). Hypothesis: action_select_kernel may have hidden binary decomposition (e.g., 2 independent {0,1} bits sampled then summed for magnitude). Audit the action selection logic. + +**Phase 7.2 (fix if found):** if hidden decomposition exists, replace with proper categorical sampling over 3 magnitude buckets, ISV-driven temperature. + +**Slots:** 0-8 depending on Phase 7.1 outcome. + +### Pearl 8: `pearl_trail_stop_signal_driven` + +Replace hardcoded `0.005f` ATR multiplier and other tuning constants in `check_trailing_stop` with per-trade signal-driven thresholds. ATR EMA, drawdown EMA, holding-time EMA all ISV-driven. Trail stop level adapts to per-trade volatility regime. + +**Slots (6-8):** ATR_EMA, DRAWDOWN_EMA, HOLD_TIME_EMA, plus per-direction adaptive trail multipliers. + +### Pearl 9: `pearl_thompson_temperature_signal_driven` + +Direction-branch Thompson sampling currently uses fixed temperature in `experience_action_select`. Make per-branch temperature ISV-driven from per-branch Q-distribution sharpness. Confident distributions → low temperature → near-argmax. Uncertain → high temperature → exploration. + +**Slots (4):** THOMPSON_TEMP[4 branches] + +### Layer A close-out: host-EMA → GPU sweep migration + +Migrate 5 remaining host-aggregated EMA sites to GPU per `feedback_no_cpu_compute_strict`: +- `training_sharpe_ema` — requires per-step PnL aggregation on GPU +- `max_dd_ema` — same +- `gamma_blend` — host-aggregated eval result, port to GPU +- `LearningHealth` composition pipeline — multi-step host arithmetic, port to GPU +- `HealthEmaTrackers` — q_gap_ema + q_var_ema + grad_norm_ema host-side EMAs + +Substantial: requires per-step trade-PnL aggregation kernel + multi-step composition kernel. Estimated 600-1000 LOC for this layer alone. + +## Architecture (mirror SP4 layer structure) + +### Layer A — additive infrastructure + +Multiple commits, no behavior change. Each commit adds: +1. ISV slot constants (per pearl) +2. Producer kernel(s) +3. Pearls A+D wire-up via existing `apply_pearls_ad_kernel` +4. StateResetRegistry entries (sentinel contract: ISV slot AND Wiener triple together) +5. Buffer growth + LAYOUT_FINGERPRINT_SEED bump (per commit accumulating) +6. GPU-gated unit tests + +Estimated 15 commits in Layer A. Final state: ~120-128 ISV slots wired, all producers running, no consumers migrated yet (zero behavior change). + +### Layer B — atomic consumer migration + +Single coordinated commit per `feedback_no_partial_refactor`. Migrates ~12 consumers: +1. `atoms_update_kernel.cu` — per-branch atom positions + counts from ISV +2. 4 `apply_*_saxpy` kernels — per-branch budget sub-launches (4 each = 16 sub-launches per step) +3. NoisyLinear forward kernel — per-branch σ scaling +4. 5 Adam launchers — per-group β1/β2/ε from ISV +5. IQL/IQN τ-quantile selection — per-branch τ from ISV +6. `apply_kelly_cap` — Pearl 6 ISV reads, hardcoded floors removed +7. `check_trailing_stop` — Pearl 8 ISV reads +8. `experience_action_select` Thompson direction branch — Pearl 9 ISV reads +9. (if Phase 7.2) `experience_action_select` magnitude branch — Pearl 7 fix +10. PnL aggregation kernel — Layer A close-out (host→GPU) + +Single large commit, ~2000-3000 LOC delta. Validates with smoke + L40S regression test post-commit. + +### Layer C — validation + cleanup + +1. Local 16+ GPU unit test pass on RTX 3050 Ti +2. L40S `multi_fold_convergence` smoke (5-epoch, single fold) — basic stability +3. L40S `train-multi-seed-baseline` (3-seed × 50-epoch × 3-fold) — full acceptance gate +4. Audit doc append (single comprehensive Layer A close-out + Layer B atomic + Layer C validation entry) +5. 9 new memory pearls + indices +6. Retire any superseded global slots / config fields + +## Acceptance criteria + +### Layer A acceptance (per-commit verification) + +- `cargo check -p ml --offline` clean +- Layer A unit tests pass on local RTX 3050 Ti +- Each commit independently buildable + reverible + +### Layer B acceptance (single commit) + +After Layer B atomic commit: +- Multi_fold_convergence smoke succeeds (5 epochs F0) +- Per-branch HEALTH_DIAG shows expected adaptation patterns: + - `var(Q[branch=Mag])` > `σ_mag² × 4` + - `v_half_mag < 1.0` + - Per-fold trade_count > 100 +- Zero graph capture failures +- All SP4-era smoke kill-criteria pass +- New SP5 kill-criterion: `Q-flatness-kill` if `var(Q[b]) < 0.5 × σ[b]²` by F0 ep10 + +### Layer C exit gate (full validation) + +3-seed × 50-epoch × 3-fold L40S run completes: +- All 9 fold-runs (3 seeds × 3 folds) reach epoch 50 +- Mean Best Sharpe across seeds positive +- intent_dist drifts epoch-to-epoch (not frozen) +- per-branch HEALTH_DIAG metrics show coherent adaptation +- Trade counts realistic (>100 per fold per seed) +- Memory pearls + audit doc landed + +## Architectural rules carryover from SP4 + +All `feedback_*` rules apply unchanged. Strict zero-tolerance: +- No CPU compute (`feedback_no_cpu_compute_strict`) — every producer is GPU; Pearls A+D applied via `apply_pearls_ad_kernel` +- No atomicAdd (`feedback_no_atomicadd`) — block tree-reduce only +- Mapped-pinned for HtoD/HtoH only +- No partial refactor — Layer B atomic commit +- No hardcoded multipliers (`feedback_isv_for_adaptive_bounds`) — only Invariant-1 numerical-stability anchors remain (EPS_CLAMP_FLOOR=1.0, ε_div=1e-8, base_iqn=0.11 anchor, entropy 70%-of-max anchor) +- Wire-up audit (`feedback_wire_everything_up`) — all 120-128 slots wired in Layer B; no orphan code +- No CPU forwards / test fallbacks — GPU oracle tests only +- No TODO/FIXME — declarative doc-comments only + +## Out of scope + +After SP5, no known adaptive-parameter deferrals remain in the DQN training loop. + +If anything surfaces post-SP5 (e.g., reward shaping issues exposed by improved magnitude differentiation), separate spec. + +## Open question for user review + +- 9 pearls + 1 layer-A close-out + 1 validation milestone in a single spec is large. Should the implementation plan use phased layers (A1, A2, A3 sub-layers per pearl, like SP4 had A1..A18 for individual tasks)? +- Pearl 7 Phase 7.1 may yield 0 slots if `Bin(2,0.5)` is downstream of Pearls 1-3 fixes (i.e., resolved without action_select changes). Should we prepare contingency: investigate first, then design Phase 7.2 IF needed? +- Pearl 4 (per-group Adam β1/β2/ε) introduces per-group hyperparameter dynamics. The Adam algorithm assumes constant β1/β2 over training; per-group adaptation breaks this assumption. Worth a separate pearl-design discussion? + +--- + +End of design. Implementation plan to be written next via `superpowers:writing-plans`. User review pending before plan dispatch.