docs(sp5): comprehensive per-branch + per-group adaptation spec — close all hardcoded-multiplier deferrals
SP5 design covers every known adaptive-parameter deferral in the DQN training loop in a single coherent project. After SP5: zero hardcoded multipliers, every adaptive value ISV-driven via Pearls A+D. 9 pearls + 1 sweep close-out + 1 validation milestone: 1-3. Per-branch atom span / loss budget / NoisyNet σ (52 slots) 4. Per-group Adam β1/β2/ε ISV-driven (24 slots) 5. Per-branch IQN τ schedule (20 slots) 6. Kelly cap signal-driven floors (6 slots) 7. dist_q/h/f Bin(2,0.5) audit + action_select fix (0-8 slots) 8. Trail stop signal-driven thresholds (6-8 slots) 9. Thompson direction-branch temperature (4 slots) 1-ext. Per-branch C51 num_atoms (4 slots) Layer A close-out: 5 host-EMA host→GPU migrations Validation: 3-seed × 50-epoch acceptance gate Total: 120-128 new ISV slots, ~5000-7500 LOC, 11-13 producer kernels, ~12 consumer migrations. Layer A (additive infrastructure, ~15 commits) → Layer B (atomic consumer migration, single coordinated commit) → Layer C (validation + cleanup). Mirrors SP4's layer pattern. Triggering data: train-multi-seed-cv2mw 50-epoch L40S baseline (terminated F0 ep10) revealed magnitude head Q-flatness, eval collapse, and frozen action distributions. Plus all SP4 close-out + sweep deferrals folded in per user direction "no deferrals — make a single plan based on ALL findings". Spec at: docs/superpowers/specs/2026-05-01-sp5-magnitude-differentiation-and-eval-collapse-design.md User review pending before invoking writing-plans skill.
This commit is contained in:
jgrusewski
@@ -0,0 +1,222 @@
|
||||
# SP5: Comprehensive Per-Branch + Per-Group Adaptation Layer — Close All Known Hardcoded-Multiplier Deferrals
|
||||
|
||||
**Date:** 2026-05-01
|
||||
**Status:** Design — awaiting user review before plan
|
||||
**Branch:** to be created from main (HEAD `a60e7b092`)
|
||||
**Triggering data:** `train-multi-seed-cv2mw` 50-epoch L40S baseline, terminated F0 ep10 after diagnostic-quality data captured. Plus all known deferrals from SP4 close-out, sweep audit, and project-history memory pearls.
|
||||
|
||||
## Goal
|
||||
|
||||
Close every known hardcoded-multiplier deferral in the DQN training loop in a single coherent project. Where SP4 made magnitude bounds adaptive, SP5 makes the entire learning machinery — atom resolution, loss-component budgets, exploration noise, optimizer hyperparameters, distributional quantile schedule, position-sizing safety floors, trail-stop thresholds, Thompson temperature — all per-branch / per-group ISV-driven via Pearls A+D.
|
||||
|
||||
After SP5: zero hardcoded multipliers in any adaptive code path; every adaptive value lives on ISV with sentinel-bootstrap and Wiener-optimal smoothing.
|
||||
|
||||
## Triggering findings
|
||||
|
||||
From the 50-epoch baseline (commit `a60e7b092`, captured F0 epochs 4-9):
|
||||
|
||||
| Symptom | Root cause |
|
||||
|---|---|
|
||||
| Magnitude head Q-values flat at O(1e-3), variances at O(1e-9 to 1e-8) | C51 atom span [-15, 15] with width 0.6 is 600× larger than mag Q-diff signal |
|
||||
| `dist_q/h/f` freezes at exact 0.225/0.280/0.495 across 6+ epochs | NoisyNet σ=0.0226 dominates Q-diff=0.005 by 4-5× → action selection noise-dominated |
|
||||
| `intent_dist` freezes at exact 0.250/0.500/0.250 (Bin(2,0.5)) | Suspicious binary decomposition in action_select kernel — needs investigation |
|
||||
| Eval-mode argmax: trade_count=1 over 214,654 val bars | Q-variance below noise floor → argmax deterministically picks one action |
|
||||
| `Sharpe=67` from one lucky trade | No kill-criterion catches `active_frac=1.0 with trade_count=1` degeneracy |
|
||||
|
||||
Plus all SP4 close-out and sweep deferrals (see [Scope](#scope) below).
|
||||
|
||||
## Scope
|
||||
|
||||
| # | Component | Status | New ISV slots |
|
||||
|---|---|---|---|
|
||||
| 1-3 | Per-branch atom span / loss budget / NoisyNet σ | core SP5 | 52 |
|
||||
| 4 | Per-param-group Adam β1/β2/ε ISV-driven | from SP4 close-out memory | 24 (3 × 8 param groups) |
|
||||
| 5 | Per-branch IQN τ schedule | from SP5 out-of-scope | 20 (5 × 4 branches) |
|
||||
| 6 | Kelly cap signal-driven floors | from project_magnitude_eval_collapse_kelly_capped.md | 6 |
|
||||
| 7 | `dist_q/h/f Bin(2,0.5)` root-cause + action_select fix | new investigation | 0-8 |
|
||||
| 8 | Trail stop signal-driven thresholds | new (Kelly-adjacent) | 6-8 |
|
||||
| 9 | Thompson direction-branch temperature ISV-driven | from SP5 out-of-scope | 4 |
|
||||
| 1-ext | Per-branch C51 num_atoms | extension of Pearl 1 | 4 |
|
||||
| Layer A close-out | 5 host-EMA host→GPU migrations | from sweep audit | 5 sites (no slots) |
|
||||
| Validation | 3-seed × 50-epoch harness as exit gate | from SP5 out-of-scope | (no code) |
|
||||
|
||||
**Total: 120-128 new ISV slots, ~5000-7500 LOC, 11-13 producer kernels, ~12 consumer migrations.**
|
||||
|
||||
After SP5: zero deferrals on hardcoded multipliers in DQN training loop. Every adaptive value ISV-driven.
|
||||
|
||||
## Pearl design summaries
|
||||
|
||||
### Pearl 1: `pearl_per_branch_c51_atom_span`
|
||||
|
||||
Per-branch C51 atom positions (`v_center`, `v_half`) adapt to per-branch Q-magnitude. Magnitude head atoms shrink to ~[-0.025, 0.025] when Q-scale is 1e-3; direction head keeps wider span. Headroom factor itself ISV-driven via per-branch atom-clip rate (Pearl C-style 1% target rate).
|
||||
|
||||
**Slots (16):** ATOM_V_CENTER[4], ATOM_V_HALF[4], ATOM_HEADROOM[4], ATOM_CLIP_RATE[4]
|
||||
|
||||
### Pearl 1-ext: `pearl_per_branch_c51_num_atoms`
|
||||
|
||||
Per-branch atom COUNT adapts to per-branch Q-resolution requirement. Direction branch can use fewer atoms with wider span; magnitude branch may need more atoms at narrow span.
|
||||
|
||||
**Slots (4):** ATOM_NUM_ATOMS[4]
|
||||
**Constraint:** total atom-tile shared memory bounded; sum across branches ≤ HW limit.
|
||||
|
||||
### Pearl 2: `pearl_per_branch_loss_budget`
|
||||
|
||||
Per-branch C51/IQN/CQL/Ens budgets driven by per-branch flatness signal `var(Q[b]) / σ[b]²`. IQN dominates when flat, C51 yields proportionally.
|
||||
|
||||
**Slots (20):** BUDGET_{C51,IQN,CQL,ENS}[4], FLATNESS[4]
|
||||
|
||||
### Pearl 3: `pearl_per_branch_noisy_sigma`
|
||||
|
||||
Per-branch NoisyNet σ scales to per-branch Q-magnitude so noise/signal ratio stays constant. Sigma fraction itself ISV-driven via per-branch action-entropy gap (target 70% of max entropy as Invariant 1 anchor — only "structural" constant remaining at this layer).
|
||||
|
||||
**Slots (12):** NOISY_SIGMA[4], SIGMA_FRACTION[4], BRANCH_ENTROPY[4]
|
||||
|
||||
### Pearl 4: `pearl_per_group_adam_hyperparams`
|
||||
|
||||
Per-param-group Adam β1, β2, ε ISV-driven (8 SP4 param groups: DqnTrunk/Value/Branches/Iqn/IqlHigh/IqlLow/Attn/Curiosity). Each group's Adam dynamics adapt to per-group gradient signal-to-noise.
|
||||
|
||||
**Producer signal:** per-group gradient direction-stability EMA (cosine similarity of consecutive gradients).
|
||||
|
||||
**Slots (24):** ADAM_BETA1[8], ADAM_BETA2[8], ADAM_EPS[8]
|
||||
|
||||
### Pearl 5: `pearl_per_branch_iqn_tau_schedule`
|
||||
|
||||
Per-branch IQN τ quantiles `{τ_05, τ_25, τ_50, τ_75, τ_95}` adapt to per-branch tail-thickness signal. Symmetric distributions can use uniform τ; skewed distributions concentrate τ in dense regions.
|
||||
|
||||
**Producer signal:** per-branch Q-distribution skew + kurtosis EMA.
|
||||
|
||||
**Slots (20):** IQN_TAU[4 branches × 5 quantiles]
|
||||
|
||||
### Pearl 6: `pearl_kelly_cap_signal_driven_floors`
|
||||
|
||||
Kelly cap removes hardcoded `0.5` warmup_floor and `0.5 + 0.5 × h` health_safety formula. Replaces with Pearls A+D smoothed kelly_f + conviction + trade-variance signals. Sentinel bootstrap handles cold-start naturally.
|
||||
|
||||
**Slots (6):** KELLY_F_SMOOTH, CONVICTION_SMOOTH, TRADE_VAR_SMOOTH, KELLY_SAMPLE_COUNT, WIN_RATE_SMOOTH, LOSS_RATE_SMOOTH
|
||||
|
||||
### Pearl 7: `pearl_action_select_decomposition_audit_or_fix`
|
||||
|
||||
**Phase 7.1 (investigation):** root-cause why intent_dist freezes at exact `Bin(2, 0.5)` ratio (0.25/0.50/0.25). Hypothesis: action_select_kernel may have hidden binary decomposition (e.g., 2 independent {0,1} bits sampled then summed for magnitude). Audit the action selection logic.
|
||||
|
||||
**Phase 7.2 (fix if found):** if hidden decomposition exists, replace with proper categorical sampling over 3 magnitude buckets, ISV-driven temperature.
|
||||
|
||||
**Slots:** 0-8 depending on Phase 7.1 outcome.
|
||||
|
||||
### Pearl 8: `pearl_trail_stop_signal_driven`
|
||||
|
||||
Replace hardcoded `0.005f` ATR multiplier and other tuning constants in `check_trailing_stop` with per-trade signal-driven thresholds. ATR EMA, drawdown EMA, holding-time EMA all ISV-driven. Trail stop level adapts to per-trade volatility regime.
|
||||
|
||||
**Slots (6-8):** ATR_EMA, DRAWDOWN_EMA, HOLD_TIME_EMA, plus per-direction adaptive trail multipliers.
|
||||
|
||||
### Pearl 9: `pearl_thompson_temperature_signal_driven`
|
||||
|
||||
Direction-branch Thompson sampling currently uses fixed temperature in `experience_action_select`. Make per-branch temperature ISV-driven from per-branch Q-distribution sharpness. Confident distributions → low temperature → near-argmax. Uncertain → high temperature → exploration.
|
||||
|
||||
**Slots (4):** THOMPSON_TEMP[4 branches]
|
||||
|
||||
### Layer A close-out: host-EMA → GPU sweep migration
|
||||
|
||||
Migrate 5 remaining host-aggregated EMA sites to GPU per `feedback_no_cpu_compute_strict`:
|
||||
- `training_sharpe_ema` — requires per-step PnL aggregation on GPU
|
||||
- `max_dd_ema` — same
|
||||
- `gamma_blend` — host-aggregated eval result, port to GPU
|
||||
- `LearningHealth` composition pipeline — multi-step host arithmetic, port to GPU
|
||||
- `HealthEmaTrackers` — q_gap_ema + q_var_ema + grad_norm_ema host-side EMAs
|
||||
|
||||
Substantial: requires per-step trade-PnL aggregation kernel + multi-step composition kernel. Estimated 600-1000 LOC for this layer alone.
|
||||
|
||||
## Architecture (mirror SP4 layer structure)
|
||||
|
||||
### Layer A — additive infrastructure
|
||||
|
||||
Multiple commits, no behavior change. Each commit adds:
|
||||
1. ISV slot constants (per pearl)
|
||||
2. Producer kernel(s)
|
||||
3. Pearls A+D wire-up via existing `apply_pearls_ad_kernel`
|
||||
4. StateResetRegistry entries (sentinel contract: ISV slot AND Wiener triple together)
|
||||
5. Buffer growth + LAYOUT_FINGERPRINT_SEED bump (per commit accumulating)
|
||||
6. GPU-gated unit tests
|
||||
|
||||
Estimated 15 commits in Layer A. Final state: ~120-128 ISV slots wired, all producers running, no consumers migrated yet (zero behavior change).
|
||||
|
||||
### Layer B — atomic consumer migration
|
||||
|
||||
Single coordinated commit per `feedback_no_partial_refactor`. Migrates ~12 consumers:
|
||||
1. `atoms_update_kernel.cu` — per-branch atom positions + counts from ISV
|
||||
2. 4 `apply_*_saxpy` kernels — per-branch budget sub-launches (4 each = 16 sub-launches per step)
|
||||
3. NoisyLinear forward kernel — per-branch σ scaling
|
||||
4. 5 Adam launchers — per-group β1/β2/ε from ISV
|
||||
5. IQL/IQN τ-quantile selection — per-branch τ from ISV
|
||||
6. `apply_kelly_cap` — Pearl 6 ISV reads, hardcoded floors removed
|
||||
7. `check_trailing_stop` — Pearl 8 ISV reads
|
||||
8. `experience_action_select` Thompson direction branch — Pearl 9 ISV reads
|
||||
9. (if Phase 7.2) `experience_action_select` magnitude branch — Pearl 7 fix
|
||||
10. PnL aggregation kernel — Layer A close-out (host→GPU)
|
||||
|
||||
Single large commit, ~2000-3000 LOC delta. Validates with smoke + L40S regression test post-commit.
|
||||
|
||||
### Layer C — validation + cleanup
|
||||
|
||||
1. Local 16+ GPU unit test pass on RTX 3050 Ti
|
||||
2. L40S `multi_fold_convergence` smoke (5-epoch, single fold) — basic stability
|
||||
3. L40S `train-multi-seed-baseline` (3-seed × 50-epoch × 3-fold) — full acceptance gate
|
||||
4. Audit doc append (single comprehensive Layer A close-out + Layer B atomic + Layer C validation entry)
|
||||
5. 9 new memory pearls + indices
|
||||
6. Retire any superseded global slots / config fields
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
### Layer A acceptance (per-commit verification)
|
||||
|
||||
- `cargo check -p ml --offline` clean
|
||||
- Layer A unit tests pass on local RTX 3050 Ti
|
||||
- Each commit independently buildable + reverible
|
||||
|
||||
### Layer B acceptance (single commit)
|
||||
|
||||
After Layer B atomic commit:
|
||||
- Multi_fold_convergence smoke succeeds (5 epochs F0)
|
||||
- Per-branch HEALTH_DIAG shows expected adaptation patterns:
|
||||
- `var(Q[branch=Mag])` > `σ_mag² × 4`
|
||||
- `v_half_mag < 1.0`
|
||||
- Per-fold trade_count > 100
|
||||
- Zero graph capture failures
|
||||
- All SP4-era smoke kill-criteria pass
|
||||
- New SP5 kill-criterion: `Q-flatness-kill` if `var(Q[b]) < 0.5 × σ[b]²` by F0 ep10
|
||||
|
||||
### Layer C exit gate (full validation)
|
||||
|
||||
3-seed × 50-epoch × 3-fold L40S run completes:
|
||||
- All 9 fold-runs (3 seeds × 3 folds) reach epoch 50
|
||||
- Mean Best Sharpe across seeds positive
|
||||
- intent_dist drifts epoch-to-epoch (not frozen)
|
||||
- per-branch HEALTH_DIAG metrics show coherent adaptation
|
||||
- Trade counts realistic (>100 per fold per seed)
|
||||
- Memory pearls + audit doc landed
|
||||
|
||||
## Architectural rules carryover from SP4
|
||||
|
||||
All `feedback_*` rules apply unchanged. Strict zero-tolerance:
|
||||
- No CPU compute (`feedback_no_cpu_compute_strict`) — every producer is GPU; Pearls A+D applied via `apply_pearls_ad_kernel`
|
||||
- No atomicAdd (`feedback_no_atomicadd`) — block tree-reduce only
|
||||
- Mapped-pinned for HtoD/HtoH only
|
||||
- No partial refactor — Layer B atomic commit
|
||||
- No hardcoded multipliers (`feedback_isv_for_adaptive_bounds`) — only Invariant-1 numerical-stability anchors remain (EPS_CLAMP_FLOOR=1.0, ε_div=1e-8, base_iqn=0.11 anchor, entropy 70%-of-max anchor)
|
||||
- Wire-up audit (`feedback_wire_everything_up`) — all 120-128 slots wired in Layer B; no orphan code
|
||||
- No CPU forwards / test fallbacks — GPU oracle tests only
|
||||
- No TODO/FIXME — declarative doc-comments only
|
||||
|
||||
## Out of scope
|
||||
|
||||
After SP5, no known adaptive-parameter deferrals remain in the DQN training loop.
|
||||
|
||||
If anything surfaces post-SP5 (e.g., reward shaping issues exposed by improved magnitude differentiation), separate spec.
|
||||
|
||||
## Open question for user review
|
||||
|
||||
- 9 pearls + 1 layer-A close-out + 1 validation milestone in a single spec is large. Should the implementation plan use phased layers (A1, A2, A3 sub-layers per pearl, like SP4 had A1..A18 for individual tasks)?
|
||||
- Pearl 7 Phase 7.1 may yield 0 slots if `Bin(2,0.5)` is downstream of Pearls 1-3 fixes (i.e., resolved without action_select changes). Should we prepare contingency: investigate first, then design Phase 7.2 IF needed?
|
||||
- Pearl 4 (per-group Adam β1/β2/ε) introduces per-group hyperparameter dynamics. The Adam algorithm assumes constant β1/β2 over training; per-group adaptation breaks this assumption. Worth a separate pearl-design discussion?
|
||||
|
||||
---
|
||||
|
||||
End of design. Implementation plan to be written next via `superpowers:writing-plans`. User review pending before plan dispatch.
|
||||
Reference in New Issue
Block a user