From b8a7ac6f70cd4c5b9fb5973a2bb47dba3aaeafb1 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Thu, 30 Apr 2026 12:10:46 +0200 Subject: [PATCH] =?UTF-8?q?fix(dqn):=20SP3=20Mech=208=20=E2=80=94=20slow?= =?UTF-8?q?=5Fema=20fold-boundary=20reset=20(real=20root=20cause)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User-identified root cause: grad_norm_slow_ema (α=0.001, half-life ~693 steps) persists across fold boundaries while every other distribution-tracking signal (Adam m/v, target nets, atom positions) gets reset. Mech 6's upper_bound formula (100 × slow_ema × isv) was anchored to the WRONG scale during F1 ramp-up, driving the non-monotonic multiplier-tuning dance across smokes: - smoke-test-fxvkk (mult=100×): F0=44, F1 NaN @ 3720 - smoke-test-ftdjz (mult=100×, +Mech7): F0=38, F1 collapse @ 2040 - smoke-test-d25vq (mult=5×): F0=21, F1 collapse @ 2820 The multiplier was searching the wrong dimension — the anchor itself was stale. Three coordinated changes (per feedback_no_partial_refactor): 1. RESTORE Mech 6 multiplier 5× → 100×. The original 100× was correct for steady-state; the F1 saturation was driven by anchor staleness, not multiplier looseness. Tightening it harmed F0 (over-clip during ramp-up when slow_ema lagged grad_norm_ema). 2. ADD reset_grad_norm_slow_ema method on GpuDqnTrainer. Zeroes the mapped-pinned scalar. First step of new fold builds up slow_ema fresh, with MIN_CLIP=1.0 floor active during the brief transient. 3. WIRE Mech 8 call in fused_training.rs::reset_for_fold, alongside Mech 4's existing Adam resets. Mech 6's anchor now aligns with the new fold's grad scale from step 1 — same philosophy as Mech 3 (target net hard-sync) and Mech 4 (Adam EMA reset). Net SP3 design: Mech 6 stays at 100× multiplier (broad, principled headroom), Mech 8 keeps the anchor honest. The pair is more robust than either change alone: - Without Mech 8: anchor is stale, multiplier tuning has no winning setting (5× hurts F0 ramp, 100× lets F1 saturate at slot 36). - With Mech 8: anchor is fresh per fold; 100× multiplier provides legitimate per-step headroom over CURRENT fold's grad norm. Mech 7 stays reverted (per-element clip was misdiagnosis — over-clipped legitimate gradient outliers without addressing the saturation root cause). F0 risk: low — F0 starts with slow_ema=0 anyway (cold start), so Mech 8 is a no-op on F0. Only changes F1+F2 fold-boundary behavior. F1+F2 expectation: Mech 6's upper_bound now scales with the CURRENT fold's grad norm, providing legitimate ~10× headroom per step without allowing Adam EMA saturation. Slot 36-42 should stay quiet. --- .../ml/src/cuda_pipeline/gpu_dqn_trainer.rs | 98 ++++++++++++------- crates/ml/src/trainers/dqn/fused_training.rs | 10 ++ docs/dqn-wire-up-audit.md | 2 + 3 files changed, 73 insertions(+), 37 deletions(-) diff --git a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs index 71ca8fcc3..bf5ca1e03 100644 --- a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +++ b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs @@ -3855,6 +3855,25 @@ impl GpuDqnTrainer { Ok(()) } + /// SP3 Mech 8: hard-reset grad_norm_slow_ema at fold boundary. Without + /// this reset, the α=0.001 EMA persists across folds with the prior + /// fold's value (half-life ~693 steps). Mech 6's upper_bound formula + /// (`100 × slow_ema × isv`) is then anchored to the wrong scale during + /// fold-N+1 ramp-up — either over-clipping (if slow_ema is too small + /// relative to current grads) or under-clipping (after slow_ema catches + /// up to the larger F1 grad scale, allowing Adam EMA saturation). + /// + /// Reset philosophy matches Mech 4 (Adam state reset) and Mech 3 (hard + /// target sync): fold boundaries are distribution-shift events; ALL + /// running EMAs of training dynamics should reinitialize, not just + /// some of them. Without Mech 8, Mech 6's anchor was the lone outlier + /// — every other distribution-tracking signal already resets per + /// existing infrastructure (`reset_for_fold` in fused_training.rs). + pub fn reset_grad_norm_slow_ema(&mut self) -> Result<(), MLError> { + unsafe { *self.grad_norm_slow_ema_pinned = 0.0_f32; } + Ok(()) + } + /// Reset Adam momentum (m/v) for branch weights only (indices 8-49). /// Trunk momentum (indices 0-7) preserved — carries valuable convergence history. /// Lighter than full rewind — tries to break Adam fixed point without changing weights. @@ -20317,57 +20336,62 @@ impl GpuDqnTrainer { } let new_clip = (self.grad_norm_ema * CLIP_MULTIPLIER).max(MIN_CLIP); - // SP3 Mech 6 v2 (2026-04-29): anchored upper bound on `new_clip`. - // The winsorizer above caps a SINGLE sample at K × prev_clip but - // does NOT prevent CONSECUTIVE elevated samples from compounding - // the EMA upward without bound. Over hundreds of steps the clip - // threshold ratchets to thousands while actual grad_norm tracks - // it from below → clipping becomes a no-op against in-distribution - // drift → Adam m/v EMAs poisoned → saturate. This is the F1 NaN - // root cause from smoke-test-5rqzs (commit b9edccfc1) at step 3060 - // (Mech 5 diag slots 36–38, 40–42 firing). + // SP3 Mech 6 (anchored) + Mech 8 (anchor reset, 2026-04-29): + // anchored upper bound on `new_clip`. The winsorizer above caps a + // SINGLE sample at K × prev_clip but does NOT prevent CONSECUTIVE + // elevated samples from compounding the EMA upward without bound. + // Over hundreds of steps the clip threshold ratchets to thousands + // while actual grad_norm tracks it from below → clipping becomes + // a no-op against in-distribution drift → Adam m/v EMAs poisoned + // → saturate. This is the F1 NaN root cause from smoke-test-5rqzs + // (commit b9edccfc1) at step 3060 (Mech 5 diag slots 36–38, 40–42 + // firing). // - // Bound: 5 × grad_norm_slow_ema × ISV[Q_ABS_REF=16].max(1.0) + // Bound: 100 × grad_norm_slow_ema × ISV[Q_ABS_REF=16].max(1.0) // - `grad_norm_slow_ema` (existing α=0.001 slow EMA, updated below // in this same function) anchors against the legitimate // steady-state grad norm; here we read the PREVIOUS step's slow // EMA from pinned memory (it gets overwritten further down). - // - 5× multiplier (revised from 100× — see "v2" rationale below): - // matches standard DL practice (5–10× steady-state grad norm). - // At 5×, per-element gradient max ≤ adaptive_clip = 5 × slow_ema - // × isv ≈ 10 (typical slow_ema ≈ 2, isv-floor = 1), well below - // slot 36 firing threshold of 100 × isv = 100. Adam m_X EMA - // steady-state with worst-case constant g = 10 reaches m_X = 10 - // — slots 36–42 should not fire. + // Mech 8 (`reset_grad_norm_slow_ema`) hard-resets this scalar at + // every fold boundary so the anchor tracks the CURRENT fold's + // grad-norm regime from step 1 instead of lagging the prior + // fold's value via the α=0.001 EMA half-life (~693 steps). The + // pair Mech 6 + Mech 8 is what makes the 100× multiplier safe; + // Mech 6 alone with a stale anchor was the F1-saturation + // pathology from smoke-test-fxvkk / -ftdjz. + // - 100× multiplier: matches the principled per-step headroom + // over CURRENT-fold steady-state grad norm. Earlier v2 of this + // bound used 5× to compensate for the stale anchor — but that + // over-clipped F0 ramp-up gradients (where slow_ema lags + // grad_norm_ema during cold-start) and still let F1 saturate + // once the EMA caught up. With Mech 8 keeping the anchor fresh, + // 100× provides legitimate 10× headroom over the slot-36 + // diagnostic threshold of 100 × isv without enabling the Adam-EMA + // saturation pathology. // - ISV[Q_ABS_REF=16] multiplier: scales the cap with the // Q-magnitude regime per `feedback_isv_for_adaptive_bounds.md`. // Same ISV slot used by SP3 Mechs 1+2+5 — no new slot. // - `.max(1.0_f32)` ε on the multiplier per the SP1 ε-floor pearl: // cold-start ISV[16] ≈ 0 must not collapse the bound. // - `.max(MIN_CLIP)` ε-floor on the bound itself: cold-start - // `grad_norm_slow_ema` ≈ 0 must not pin upper_bound at 0 (which - // would force `new_clip` down to MIN_CLIP every step until the - // slow EMA warms up). + // `grad_norm_slow_ema` ≈ 0 (true on F0 step 1 AND on every + // post-Mech-8 fold-boundary first step) must not pin upper_bound + // at 0; MIN_CLIP=1.0 floor stays active for the brief transient + // while the slow EMA warms back up over the new fold. // - // v2 rationale (multiplier 100 → 5): - // - Smoke smoke-test-fxvkk (Mech 6 v1, 100×) F1-NaN'd at step 3720 - // with slots 36-42 still firing. Per-element gradient max under - // v1 reached ≤ 100 × slow_ema × isv ≈ 200, and Adam m_X EMA - // steady-state matched, exceeding slot 36 threshold of 100. The - // 100× bound was wider than the diagnostic threshold — Mech 6 - // was active but the cap was the wrong width. - // - Why 5× and not tighter (e.g., 2×): per-step gradient norms can - // legitimately spike to ~5× slow_ema in normal training (e.g., - // gradient resumption after warmup, occasional curvature - // transients); tighter bounds would over-clip and harm fit. - // - Why not wider (e.g., 10× or 20×): >10× restores the slot 36 - // threshold-overshoot pathology with diminishing margin. 5× - // leaves robust 10× headroom against the 100× threshold. - // - F0 risk: low — F0 typical adaptive_clip stays under 5× - // slow_ema in steady training; the cap should be a no-op for F0. + // F1+ history that motivated Mech 8 (anchor reset): + // - smoke-test-fxvkk (Mech 6 only, 100×): F0=44, F1 NaN @ 3720 + // - smoke-test-ftdjz (Mech 6 + Mech 7, 100×): F0=38, F1 collapse + // @ 2040 + // - smoke-test-d25vq (Mech 6 v2, 5×): F0=21, F1 collapse @ 2820 + // The multiplier was searching the wrong dimension — the anchor + // itself was stale across fold boundaries, while every other + // distribution-tracking signal (Adam m/v, target nets, atom + // positions) reset per existing infrastructure. Mech 8 closes the + // partial-refactor gap; Mech 6 stays at the principled 100×. let abs_mult = self.read_isv_signal_at(Q_ABS_REF_INDEX).max(1.0_f32); let prev_slow_ema = unsafe { *self.grad_norm_slow_ema_pinned }; - let upper_bound = (prev_slow_ema * 5.0_f32 * abs_mult).max(MIN_CLIP); + let upper_bound = (prev_slow_ema * 100.0_f32 * abs_mult).max(MIN_CLIP); let new_clip = new_clip.min(upper_bound); // Write directly to pinned memory — GPU sees it on next kernel read diff --git a/crates/ml/src/trainers/dqn/fused_training.rs b/crates/ml/src/trainers/dqn/fused_training.rs index 7d4854fde..285048501 100644 --- a/crates/ml/src/trainers/dqn/fused_training.rs +++ b/crates/ml/src/trainers/dqn/fused_training.rs @@ -1064,6 +1064,16 @@ impl FusedTrainingCtx { } } + // SP3 Mech 8: reset grad_norm_slow_ema at fold boundary. Aligns + // Mech 6's anchor with the new fold's grad scale from step 1 + // (instead of lagging the prior fold's value via α=0.001 EMA). + // Without this, Mech 6's upper_bound was either over-clipping + // (slow_ema too small) or under-clipping (slow_ema caught up but + // F1 grad scale already too large for slot 36 threshold). + // Resetting here lets Mech 6 re-anchor to the new fold's scale + // properly while preserving the 100× multiplier. + self.trainer.reset_grad_norm_slow_ema()?; + // FoldReset state (eval_q_mean_ema, eval_q_std_ema, isv_v_range_slots, // isv_learning_health, isv_sharpe_ema, isv_q_means) is now handled by // StateResetRegistry-driven dispatch in DQNTrainer::reset_for_fold diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index c33cad6f8..d7070d002 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,8 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +SP3 Mech 8 — `grad_norm_slow_ema` fold-boundary reset (2026-04-29): closed the partial-refactor gap where the α=0.001 grad-norm slow EMA (driving Mech 6's anchored upper bound on `adaptive_clip`) persisted across fold boundaries while every other distribution-tracking signal — Adam m/v (`reset_adam_state`), main DQN target params (`sync_target_from_online`), IQN target + Adam (`iqn.sync_target_from_online` + `iqn.reset_adam_state`), TLOB / IQL / IQL-low / 4-head attention Adam (`reset_adam_state` on each), C51 atom-position EMAs, replay buffer — already reset per existing `reset_for_fold` infrastructure. The α=0.001 half-life (~693 steps) meant `grad_norm_slow_ema` lagged the new fold's grad-norm regime by hundreds of steps; Mech 6's `100 × slow_ema × isv` upper bound was anchored to the WRONG fold's scale during F1 ramp-up, producing the non-monotonic multiplier-tuning dance observed across smokes (smoke-test-fxvkk 100× F1-NaN @ 3720 / smoke-test-ftdjz 100×+Mech7 F1-collapse @ 2040 / smoke-test-d25vq 5× F1-collapse @ 2820 — the multiplier was searching the wrong dimension). Three coordinated changes per `feedback_no_partial_refactor.md`: (1) Mech 6 multiplier restored 5× → 100× in `gpu_dqn_trainer.rs::update_adaptive_clip` (the original principled setting; v2's 5× was over-clipping legitimate F0-ramp gradients while still allowing F1-anchor saturation), (2) new `GpuDqnTrainer::reset_grad_norm_slow_ema` method zeroes the existing mapped-pinned scalar (no new buffer, no new ISV slot — reuses Plan C Phase 2 follow-up K's grad_norm_slow_ema_pinned at `gpu_dqn_trainer.rs:3337`), (3) wired into `fused_training.rs::reset_for_fold` immediately after the SP3 Mech 4 attention Adam reset block (line ~1066) — same fold-boundary contract consumer chain as Mech 3+4. F0 unaffected (cold-starts at slow_ema=0 already, so Mech 8 is a no-op on F0); F1+F2 first step transient sees `upper_bound = 100 × 0 × isv → MIN_CLIP=1.0` floor for ~10-50 steps as the EMA warms, then converges to legitimate 100× headroom over CURRENT fold's grad norm. Mech 7 stays reverted (per-element clip was misdiagnosis). Touched: `cuda_pipeline/gpu_dqn_trainer.rs` (Mech 6 comment + multiplier 5.0→100.0 line ~20370; new `reset_grad_norm_slow_ema` method line ~3858), `trainers/dqn/fused_training.rs` (call wired line ~1067). cargo check clean. No fingerprint change — touches an already-allocated mapped-pinned scalar, not param-tensor layout. + HEALTH_DIAG GPU port — Phase 1 (2026-04-28, follow-up to commit `333ea7184`'s Phase 0 inventory): added `HealthDiagSnapshot` `#[repr(C)]` POD struct and `MappedHealthDiagSnapshot` mapped-pinned wrapper in new `crates/ml/src/cuda_pipeline/health_diag.rs`. The struct holds 147 fields (every numeric value emitted across the 7 HEALTH_DIAG log sites) as `f32` or `u32` so CPU and GPU agree byte-for-byte; controller fire booleans (`fire_lr`/`fire_tau`/etc.) promoted from `u8` → `u32` per the design review to avoid `#[repr(C)]` padding mismatch when followed by `f32` fields. `MappedHealthDiagSnapshot::new()` wraps `cuMemHostAlloc(DEVICEMAP|PORTABLE)` of `sizeof(HealthDiagSnapshot)` + `cuMemHostGetDevicePointer_v2` — the same pattern as `MappedF32Buffer` but typed for the snapshot. Re-exported from `cuda_pipeline/mod.rs` for downstream consumers. **No producer kernel and no consumer wiring in this commit** — Phase 2 lands the kernel family (`health_diag_per_sample_reduce`, `health_diag_q_mag_reduce`, `health_diag_eval_histogram`, `health_diag_isv_mirror`, `health_diag_finalise`) and the `gpu_health_diag.rs` orchestrator; Phase 3 wires the launch into the captured graph alongside `launch_h_s2_rms_ema`; Phase 4 rewrites the CPU emit path and deletes the 12 CPU-bound feeder methods identified in the Phase 0 inventory. Three unit tests (`snapshot_size_is_stable`, `default_is_zeroed`, `alignment_is_4_bytes`) pin the layout so any future field add/reorder requires an explicit test update — guards the kernel-side offset table from silent drift. Touched: `cuda_pipeline/health_diag.rs` (+339 LOC, new), `cuda_pipeline/mod.rs` (+8 LOC re-exports). cargo check clean at 12 warnings (workspace baseline). No fingerprint change — separate mapped-pinned alloc, not part of param-tensor layout the fingerprint guards. multi_fold_convergence smoke MBP-10 wiring (2026-04-28): the test now forwards `--mbp10-data-dir` and `--trades-data-dir` to its spawned `train_baseline_rl` subprocess, reading paths from `FOXHUNT_MBP10_DATA` / `FOXHUNT_TRADES_DATA` env vars (set by the L40S sanitizer-test + nsys-test Argo templates to `/data/test-data/{mbp10,trades}`) with fallback to `/../futures-baseline-{mbp10,trades}` for local repo runs. The test hard-fails fast if either path is missing, citing `feedback_mbp10_mandatory.md`. Without this, OFI features (state slots [18..26)) silently fell back to tick-rule proxy and the smoke validated a degraded model variant — a `feedback_no_partial_refactor.md` violation now closed.