diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 01a8a1007..93bce538f 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -5600,3 +5600,117 @@ This is the policy-input mirror of the reward-side bug fixed in `35db31089`. Both share the same architectural lesson — see `pearl_symmetric_clamp_audit` (added in the SP11 close-out commit) and `pearl_bounded_modifier_outputs_require_structural_activation`. + +## SP11 B1b sharpe-degradation close-out (2026-05-04) + +The within-fold sharpe degradation flagged in the B1b smoke-recovery +"Open follow-up" above is now resolved. Root cause + fix chain +documented for future audit. + +### Root cause: asymmetric reward cap + +`experience_kernels.cu:2788` (pre-fix): + +```cuda +float capped_pnl = fminf(base_reward, 10.0f); +``` + +`base_reward` was P&L-derived and could be negative on losing +trades; the upper-only cap left the loss tail unbounded. Diagnostic +instrumentation (commit `774d7552a`, since removed) found the +post-cap reward reaching **-210336** by F0 ep3 — five orders of +magnitude past the spec bound on the loss side. + +The unbounded loss tail propagated through: + +1. **PopArt running magnitude EMA (slot 63)** inflated to track + extreme magnitudes, breaking C51/IQN normalisation across + epochs. HEALTH_DIAG `popart` jumping from O(10) to O(10⁵) within + one epoch was the smoking gun. +2. **C51 atom-span (per-branch v_range)** widened to cover the + pollution, then never recovered, holding subsequent epochs at + miscalibrated atom resolution. +3. **Loss-balance controller (SP7)** read inflated reward + magnitudes, mis-allocating per-loss-term budgets. +4. **Within-fold sharpe degraded** from ~6 (clean) to ~4.26 across + the fold — the residual flagged in the B1b close-out. + +### Fix: symmetric reward cap + +Commit `35db31089` at `experience_kernels.cu:2788`: + +```cuda +float capped_pnl = fmaxf(-10.0f, fminf(base_reward, 10.0f)); +``` + +Bilateral clamp at the canonical reward-emission site. Spec bound +`reward_per_bar ∈ [-10, 10]` is now enforced structurally on both +sides; downstream PopArt/C51/IQN normalisers receive a guaranteed +bounded signal. + +### Validation: smoke-test-trk72 + +L40S smoke at commit `35db31089` (5-epoch × 3-fold): + +| Metric | Pre-fix (degraded) | Post-fix (smoke-test-trk72) | Delta | +|---|---|---|---| +| F0 sharpe | 4.26 | 6.13 | **+44%** | +| `popart` magnitude | unbounded (peaks 10⁵+) | bounded ±10 throughout | clean | +| `curiosity_b` | unstable, drift-prone | stable 0.18-0.31 across folds | clean | +| Cross-fold convergence | F0 sharpe degraded mid-fold | All 3 folds converge cleanly | PASSED | + +The smoke validated the fix end-to-end: PopArt slot 63 bounded, +C51/IQN normalisation stable, sharpe recovered. + +### Mirror fix: plan_isv 4 sites (commit `348f6078b`) + +The implementer of `35db31089` flagged 4 additional asymmetric-clamp +sites in `plan_isv` policy-input features (table above). These are +the feature-side mirror of the reward-side bug — same shape, same +fix recipe, same architectural lesson. + +### Diagnostic cleanup (commit `b92dcc3df`) + +Diagnostic instrumentation from `774d7552a` served its purpose +(empirically identified the asymmetric cap as the inflater) and was +removed: + +- `reward_chain_diag_reduce_kernel.cu` (file deletion) +- 12 per-sample diagnostic buffers in `gpu_experience_collector` +- Kernel parameter threading in `experience_kernels.cu` +- Launcher + reader + mapped-pinned output in `gpu_dqn_trainer` +- Wire-up site + HEALTH_DIAG emit in `training_loop` + +The worktree returned to its pre-instrumentation state on the SP11 +reward chain. + +### Cross-references — both pearls + +The SP11 sharpe-degradation investigation produced two +complementary memory pearls: + +| Pearl | Domain | Fix recipe | +|-------|--------|-----------| +| `pearl_bounded_modifier_outputs_require_structural_activation` | Model OUTPUTS — learned scalars feeding multiplicative modifiers | Structural activation (sigmoid/tanh/softplus) on the architecture | +| `pearl_symmetric_clamp_audit` | Kernel-derived intermediate scalars (rewards, ISV features) | Bilateral runtime clamp `fmaxf(lo, fminf(x, hi))` | + +Both share the same architectural principle: **bounded contracts at +architectural boundaries must be enforced structurally, not by +hopes about input distribution**. They are siblings in the same +class — only the implementation site differs (model architecture +vs. kernel arithmetic). + +### Commit chain summary + +| Commit | Description | +|--------|-------------| +| `774d7552a` | diag: reward-chain instrumentation (transient) | +| `35db31089` | fix: symmetric reward cap (THE bug) | +| `348f6078b` | fix: plan_isv 4-site symmetric clamp mirrors | +| `b92dcc3df` | chore: remove reward-chain instrumentation | +| (this commit) | docs: pearl + audit close-out | + +The B1b sharpe-degradation residual is **resolved**. The B1b +controller (z-score normalised mag-ratio canary) and the SP11 +reward cap fix together close the within-fold degradation root +cause.