fix(dqn): SP3 Mech 8 — slow_ema fold-boundary reset (real root cause)

User-identified root cause: grad_norm_slow_ema (α=0.001, half-life
~693 steps) persists across fold boundaries while every other
distribution-tracking signal (Adam m/v, target nets, atom positions)
gets reset. Mech 6's upper_bound formula (100 × slow_ema × isv) was
anchored to the WRONG scale during F1 ramp-up, driving the
non-monotonic multiplier-tuning dance across smokes:
  - smoke-test-fxvkk (mult=100×): F0=44, F1 NaN @ 3720
  - smoke-test-ftdjz (mult=100×, +Mech7): F0=38, F1 collapse @ 2040
  - smoke-test-d25vq (mult=5×): F0=21, F1 collapse @ 2820
The multiplier was searching the wrong dimension — the anchor itself
was stale.

Three coordinated changes (per feedback_no_partial_refactor):

1. RESTORE Mech 6 multiplier 5× → 100×. The original 100× was correct
   for steady-state; the F1 saturation was driven by anchor staleness,
   not multiplier looseness. Tightening it harmed F0 (over-clip during
   ramp-up when slow_ema lagged grad_norm_ema).

2. ADD reset_grad_norm_slow_ema method on GpuDqnTrainer. Zeroes the
   mapped-pinned scalar. First step of new fold builds up slow_ema
   fresh, with MIN_CLIP=1.0 floor active during the brief transient.

3. WIRE Mech 8 call in fused_training.rs::reset_for_fold, alongside
   Mech 4's existing Adam resets. Mech 6's anchor now aligns with
   the new fold's grad scale from step 1 — same philosophy as Mech 3
   (target net hard-sync) and Mech 4 (Adam EMA reset).

Net SP3 design: Mech 6 stays at 100× multiplier (broad, principled
headroom), Mech 8 keeps the anchor honest. The pair is more robust
than either change alone:
  - Without Mech 8: anchor is stale, multiplier tuning has no winning
    setting (5× hurts F0 ramp, 100× lets F1 saturate at slot 36).
  - With Mech 8: anchor is fresh per fold; 100× multiplier provides
    legitimate per-step headroom over CURRENT fold's grad norm.

Mech 7 stays reverted (per-element clip was misdiagnosis — over-clipped
legitimate gradient outliers without addressing the saturation root
cause).

F0 risk: low — F0 starts with slow_ema=0 anyway (cold start), so Mech 8
is a no-op on F0. Only changes F1+F2 fold-boundary behavior.

F1+F2 expectation: Mech 6's upper_bound now scales with the CURRENT
fold's grad norm, providing legitimate ~10× headroom per step without
allowing Adam EMA saturation. Slot 36-42 should stay quiet.

crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs

@@ -3855,6 +3855,25 @@ impl GpuDqnTrainer {
         Ok(())
     }
 
+    /// SP3 Mech 8: hard-reset grad_norm_slow_ema at fold boundary. Without
+    /// this reset, the α=0.001 EMA persists across folds with the prior
+    /// fold's value (half-life ~693 steps). Mech 6's upper_bound formula
+    /// (`100 × slow_ema × isv`) is then anchored to the wrong scale during
+    /// fold-N+1 ramp-up — either over-clipping (if slow_ema is too small
+    /// relative to current grads) or under-clipping (after slow_ema catches
+    /// up to the larger F1 grad scale, allowing Adam EMA saturation).
+    ///
+    /// Reset philosophy matches Mech 4 (Adam state reset) and Mech 3 (hard
+    /// target sync): fold boundaries are distribution-shift events; ALL
+    /// running EMAs of training dynamics should reinitialize, not just
+    /// some of them. Without Mech 8, Mech 6's anchor was the lone outlier
+    /// — every other distribution-tracking signal already resets per
+    /// existing infrastructure (`reset_for_fold` in fused_training.rs).
+    pub fn reset_grad_norm_slow_ema(&mut self) -> Result<(), MLError> {
+        unsafe { *self.grad_norm_slow_ema_pinned = 0.0_f32; }
+        Ok(())
+    }
+
     /// Reset Adam momentum (m/v) for branch weights only (indices 8-49).
     /// Trunk momentum (indices 0-7) preserved — carries valuable convergence history.
     /// Lighter than full rewind — tries to break Adam fixed point without changing weights.
@@ -20317,57 +20336,62 @@ impl GpuDqnTrainer {
         }
         let new_clip = (self.grad_norm_ema * CLIP_MULTIPLIER).max(MIN_CLIP);
 
-        // SP3 Mech 6 v2 (2026-04-29): anchored upper bound on `new_clip`.
-        // The winsorizer above caps a SINGLE sample at K × prev_clip but
-        // does NOT prevent CONSECUTIVE elevated samples from compounding
-        // the EMA upward without bound. Over hundreds of steps the clip
-        // threshold ratchets to thousands while actual grad_norm tracks
-        // it from below → clipping becomes a no-op against in-distribution
-        // drift → Adam m/v EMAs poisoned → saturate. This is the F1 NaN
-        // root cause from smoke-test-5rqzs (commit b9edccfc1) at step 3060
-        // (Mech 5 diag slots 36–38, 40–42 firing).
+        // SP3 Mech 6 (anchored) + Mech 8 (anchor reset, 2026-04-29):
+        // anchored upper bound on `new_clip`. The winsorizer above caps a
+        // SINGLE sample at K × prev_clip but does NOT prevent CONSECUTIVE
+        // elevated samples from compounding the EMA upward without bound.
+        // Over hundreds of steps the clip threshold ratchets to thousands
+        // while actual grad_norm tracks it from below → clipping becomes
+        // a no-op against in-distribution drift → Adam m/v EMAs poisoned
+        // → saturate. This is the F1 NaN root cause from smoke-test-5rqzs
+        // (commit b9edccfc1) at step 3060 (Mech 5 diag slots 36–38, 40–42
+        // firing).
         //
-        // Bound: 5 × grad_norm_slow_ema × ISV[Q_ABS_REF=16].max(1.0)
+        // Bound: 100 × grad_norm_slow_ema × ISV[Q_ABS_REF=16].max(1.0)
         // - `grad_norm_slow_ema` (existing α=0.001 slow EMA, updated below
         //   in this same function) anchors against the legitimate
         //   steady-state grad norm; here we read the PREVIOUS step's slow
         //   EMA from pinned memory (it gets overwritten further down).
-        // - 5× multiplier (revised from 100× — see "v2" rationale below):
-        //   matches standard DL practice (5–10× steady-state grad norm).
-        //   At 5×, per-element gradient max ≤ adaptive_clip = 5 × slow_ema
-        //   × isv ≈ 10 (typical slow_ema ≈ 2, isv-floor = 1), well below
-        //   slot 36 firing threshold of 100 × isv = 100. Adam m_X EMA
-        //   steady-state with worst-case constant g = 10 reaches m_X = 10
-        //   — slots 36–42 should not fire.
+        //   Mech 8 (`reset_grad_norm_slow_ema`) hard-resets this scalar at
+        //   every fold boundary so the anchor tracks the CURRENT fold's
+        //   grad-norm regime from step 1 instead of lagging the prior
+        //   fold's value via the α=0.001 EMA half-life (~693 steps). The
+        //   pair Mech 6 + Mech 8 is what makes the 100× multiplier safe;
+        //   Mech 6 alone with a stale anchor was the F1-saturation
+        //   pathology from smoke-test-fxvkk / -ftdjz.
+        // - 100× multiplier: matches the principled per-step headroom
+        //   over CURRENT-fold steady-state grad norm. Earlier v2 of this
+        //   bound used 5× to compensate for the stale anchor — but that
+        //   over-clipped F0 ramp-up gradients (where slow_ema lags
+        //   grad_norm_ema during cold-start) and still let F1 saturate
+        //   once the EMA caught up. With Mech 8 keeping the anchor fresh,
+        //   100× provides legitimate 10× headroom over the slot-36
+        //   diagnostic threshold of 100 × isv without enabling the Adam-EMA
+        //   saturation pathology.
         // - ISV[Q_ABS_REF=16] multiplier: scales the cap with the
         //   Q-magnitude regime per `feedback_isv_for_adaptive_bounds.md`.
         //   Same ISV slot used by SP3 Mechs 1+2+5 — no new slot.
         // - `.max(1.0_f32)` ε on the multiplier per the SP1 ε-floor pearl:
         //   cold-start ISV[16] ≈ 0 must not collapse the bound.
         // - `.max(MIN_CLIP)` ε-floor on the bound itself: cold-start
-        //   `grad_norm_slow_ema` ≈ 0 must not pin upper_bound at 0 (which
-        //   would force `new_clip` down to MIN_CLIP every step until the
-        //   slow EMA warms up).
+        //   `grad_norm_slow_ema` ≈ 0 (true on F0 step 1 AND on every
+        //   post-Mech-8 fold-boundary first step) must not pin upper_bound
+        //   at 0; MIN_CLIP=1.0 floor stays active for the brief transient
+        //   while the slow EMA warms back up over the new fold.
         //
-        // v2 rationale (multiplier 100 → 5):
-        // - Smoke smoke-test-fxvkk (Mech 6 v1, 100×) F1-NaN'd at step 3720
-        //   with slots 36-42 still firing. Per-element gradient max under
-        //   v1 reached ≤ 100 × slow_ema × isv ≈ 200, and Adam m_X EMA
-        //   steady-state matched, exceeding slot 36 threshold of 100. The
-        //   100× bound was wider than the diagnostic threshold — Mech 6
-        //   was active but the cap was the wrong width.
-        // - Why 5× and not tighter (e.g., 2×): per-step gradient norms can
-        //   legitimately spike to ~5× slow_ema in normal training (e.g.,
-        //   gradient resumption after warmup, occasional curvature
-        //   transients); tighter bounds would over-clip and harm fit.
-        // - Why not wider (e.g., 10× or 20×): >10× restores the slot 36
-        //   threshold-overshoot pathology with diminishing margin. 5×
-        //   leaves robust 10× headroom against the 100× threshold.
-        // - F0 risk: low — F0 typical adaptive_clip stays under 5×
-        //   slow_ema in steady training; the cap should be a no-op for F0.
+        // F1+ history that motivated Mech 8 (anchor reset):
+        // - smoke-test-fxvkk (Mech 6 only, 100×): F0=44, F1 NaN @ 3720
+        // - smoke-test-ftdjz (Mech 6 + Mech 7, 100×): F0=38, F1 collapse
+        //   @ 2040
+        // - smoke-test-d25vq (Mech 6 v2, 5×): F0=21, F1 collapse @ 2820
+        // The multiplier was searching the wrong dimension — the anchor
+        // itself was stale across fold boundaries, while every other
+        // distribution-tracking signal (Adam m/v, target nets, atom
+        // positions) reset per existing infrastructure. Mech 8 closes the
+        // partial-refactor gap; Mech 6 stays at the principled 100×.
         let abs_mult = self.read_isv_signal_at(Q_ABS_REF_INDEX).max(1.0_f32);
         let prev_slow_ema = unsafe { *self.grad_norm_slow_ema_pinned };
-        let upper_bound = (prev_slow_ema * 5.0_f32 * abs_mult).max(MIN_CLIP);
+        let upper_bound = (prev_slow_ema * 100.0_f32 * abs_mult).max(MIN_CLIP);
         let new_clip = new_clip.min(upper_bound);
 
         // Write directly to pinned memory — GPU sees it on next kernel read

crates/ml/src/trainers/dqn/fused_training.rs

@@ -1064,6 +1064,16 @@ impl FusedTrainingCtx {
             }
         }
 
+        // SP3 Mech 8: reset grad_norm_slow_ema at fold boundary. Aligns
+        // Mech 6's anchor with the new fold's grad scale from step 1
+        // (instead of lagging the prior fold's value via α=0.001 EMA).
+        // Without this, Mech 6's upper_bound was either over-clipping
+        // (slow_ema too small) or under-clipping (slow_ema caught up but
+        // F1 grad scale already too large for slot 36 threshold).
+        // Resetting here lets Mech 6 re-anchor to the new fold's scale
+        // properly while preserving the 100× multiplier.
+        self.trainer.reset_grad_norm_slow_ema()?;
+
         // FoldReset state (eval_q_mean_ema, eval_q_std_ema, isv_v_range_slots,
         // isv_learning_health, isv_sharpe_ema, isv_q_means) is now handled by
         // StateResetRegistry-driven dispatch in DQNTrainer::reset_for_fold

docs/dqn-wire-up-audit.md

@@ -2,6 +2,8 @@
 
 **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
 
+SP3 Mech 8 — `grad_norm_slow_ema` fold-boundary reset (2026-04-29): closed the partial-refactor gap where the α=0.001 grad-norm slow EMA (driving Mech 6's anchored upper bound on `adaptive_clip`) persisted across fold boundaries while every other distribution-tracking signal — Adam m/v (`reset_adam_state`), main DQN target params (`sync_target_from_online`), IQN target + Adam (`iqn.sync_target_from_online` + `iqn.reset_adam_state`), TLOB / IQL / IQL-low / 4-head attention Adam (`reset_adam_state` on each), C51 atom-position EMAs, replay buffer — already reset per existing `reset_for_fold` infrastructure. The α=0.001 half-life (~693 steps) meant `grad_norm_slow_ema` lagged the new fold's grad-norm regime by hundreds of steps; Mech 6's `100 × slow_ema × isv` upper bound was anchored to the WRONG fold's scale during F1 ramp-up, producing the non-monotonic multiplier-tuning dance observed across smokes (smoke-test-fxvkk 100× F1-NaN @ 3720 / smoke-test-ftdjz 100×+Mech7 F1-collapse @ 2040 / smoke-test-d25vq 5× F1-collapse @ 2820 — the multiplier was searching the wrong dimension). Three coordinated changes per `feedback_no_partial_refactor.md`: (1) Mech 6 multiplier restored 5× → 100× in `gpu_dqn_trainer.rs::update_adaptive_clip` (the original principled setting; v2's 5× was over-clipping legitimate F0-ramp gradients while still allowing F1-anchor saturation), (2) new `GpuDqnTrainer::reset_grad_norm_slow_ema` method zeroes the existing mapped-pinned scalar (no new buffer, no new ISV slot — reuses Plan C Phase 2 follow-up K's grad_norm_slow_ema_pinned at `gpu_dqn_trainer.rs:3337`), (3) wired into `fused_training.rs::reset_for_fold` immediately after the SP3 Mech 4 attention Adam reset block (line ~1066) — same fold-boundary contract consumer chain as Mech 3+4. F0 unaffected (cold-starts at slow_ema=0 already, so Mech 8 is a no-op on F0); F1+F2 first step transient sees `upper_bound = 100 × 0 × isv → MIN_CLIP=1.0` floor for ~10-50 steps as the EMA warms, then converges to legitimate 100× headroom over CURRENT fold's grad norm. Mech 7 stays reverted (per-element clip was misdiagnosis). Touched: `cuda_pipeline/gpu_dqn_trainer.rs` (Mech 6 comment + multiplier 5.0→100.0 line ~20370; new `reset_grad_norm_slow_ema` method line ~3858), `trainers/dqn/fused_training.rs` (call wired line ~1067). cargo check clean. No fingerprint change — touches an already-allocated mapped-pinned scalar, not param-tensor layout.
+
 HEALTH_DIAG GPU port — Phase 1 (2026-04-28, follow-up to commit `333ea7184`'s Phase 0 inventory): added `HealthDiagSnapshot` `#[repr(C)]` POD struct and `MappedHealthDiagSnapshot` mapped-pinned wrapper in new `crates/ml/src/cuda_pipeline/health_diag.rs`. The struct holds 147 fields (every numeric value emitted across the 7 HEALTH_DIAG log sites) as `f32` or `u32` so CPU and GPU agree byte-for-byte; controller fire booleans (`fire_lr`/`fire_tau`/etc.) promoted from `u8` → `u32` per the design review to avoid `#[repr(C)]` padding mismatch when followed by `f32` fields. `MappedHealthDiagSnapshot::new()` wraps `cuMemHostAlloc(DEVICEMAP|PORTABLE)` of `sizeof(HealthDiagSnapshot)` + `cuMemHostGetDevicePointer_v2` — the same pattern as `MappedF32Buffer` but typed for the snapshot. Re-exported from `cuda_pipeline/mod.rs` for downstream consumers. **No producer kernel and no consumer wiring in this commit** — Phase 2 lands the kernel family (`health_diag_per_sample_reduce`, `health_diag_q_mag_reduce`, `health_diag_eval_histogram`, `health_diag_isv_mirror`, `health_diag_finalise`) and the `gpu_health_diag.rs` orchestrator; Phase 3 wires the launch into the captured graph alongside `launch_h_s2_rms_ema`; Phase 4 rewrites the CPU emit path and deletes the 12 CPU-bound feeder methods identified in the Phase 0 inventory. Three unit tests (`snapshot_size_is_stable`, `default_is_zeroed`, `alignment_is_4_bytes`) pin the layout so any future field add/reorder requires an explicit test update — guards the kernel-side offset table from silent drift. Touched: `cuda_pipeline/health_diag.rs` (+339 LOC, new), `cuda_pipeline/mod.rs` (+8 LOC re-exports). cargo check clean at 12 warnings (workspace baseline). No fingerprint change — separate mapped-pinned alloc, not part of param-tensor layout the fingerprint guards.
 
 multi_fold_convergence smoke MBP-10 wiring (2026-04-28): the test now forwards `--mbp10-data-dir` and `--trades-data-dir` to its spawned `train_baseline_rl` subprocess, reading paths from `FOXHUNT_MBP10_DATA` / `FOXHUNT_TRADES_DATA` env vars (set by the L40S sanitizer-test + nsys-test Argo templates to `/data/test-data/{mbp10,trades}`) with fallback to `<data_dir>/../futures-baseline-{mbp10,trades}` for local repo runs. The test hard-fails fast if either path is missing, citing `feedback_mbp10_mandatory.md`. Without this, OFI features (state slots [18..26)) silently fell back to tick-rule proxy and the smoke validated a degraded model variant — a `feedback_no_partial_refactor.md` violation now closed.