fix(dqn): SP1 Phase C — ε floor fix-up #2 (cold-start clamp pathology)

Smoke smoke-test-dr2bn (commit 19b008e1c) F1-NaN'd at step 240 — earlier
than pre-fix smoke smoke-test-xvzgk (step 890). The fix made things
worse, indicating the ε floor `(1e6 × isv).max(1e3)` is actively
destabilizing F1 startup.

Diagnosis: at fold boundary, ISV[H_S2_RMS_EMA_INDEX=96] and
ISV[Q_DIR_ABS_REF_INDEX=21] reset to 0 (per StateResetRegistry).
Formula `(1e6 × 0).max(1e3) = 1e3` makes max_abs aggressively narrow.
F1 startup gradients can have natural magnitudes > 1e3 (post-fold
Bellman-target shift); clamping them to ±1e3 destabilizes Adam EMAs,
which then drive cuBLAS GEMM accumulators into pathological inputs
that overflow → slot 26 + 32 NaN.

New formula: `1e6 × isv.max(1.0)` guarantees max_abs ≥ 1e6 regardless
of ISV state:
  - ISV = 0   → max_abs = 1e6 × max(0, 1.0) = 1e6
  - ISV = 0.5 → max_abs = 1e6 × max(0.5, 1.0) = 1e6
  - ISV = 2.0 → max_abs = 1e6 × max(2.0, 1.0) = 2e6
  - ISV = 100 → max_abs = 1e8

F0 no-op intent preserved (F0 inputs ≪ 1e6 in all states; ISV[96]≈1.0
for converged F0 → max_abs = 1e6, well above F0-typical |iqn_d_h_s2|
≤ ~10²). F1 startup gradients ≤ 1e6 are un-clipped.

The 1.0 ε floor is on the ISV multiplier (Invariant 1 carve-out per
`feedback_isv_for_adaptive_bounds`), not on the bound itself — bound
is still ISV-driven when ISV is meaningful (≥ 1.0).

Two edit sites:
- gpu_dqn_trainer.rs:6967 (apply_iqn_trunk_gradient, slot 26 path)
- gpu_dqn_trainer.rs:18631 (launch_cublas_backward_to, slot 32 path)

F0 regression to 35.24 still unsolved — separate investigation thread
within SP1 (no deferral; Phase B instrumentation timing impact suspected).

crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs

@@ -6963,8 +6963,20 @@ impl GpuDqnTrainer {
         // (≥ 1e6) trigger clamp. The slot 35 NaN check at line 6949 fires
         // BEFORE this clamp, so any NaN entry is captured in the regression
         // sentinel before sanitisation — see Issue 1 fix in commit fix-up.
+        // SP1 Phase C fix-up #2 (cold-start ε floor): smoke-test-dr2bn
+        // (commit 19b008e1c) FAILED at F1 step 240 with the ORIGINAL formula
+        // `(1e6 × isv).max(1e3)` because fold-boundary ISV reset puts ISV[96]
+        // near 0, giving max_abs = 1e3 — too tight; clipped legitimate F1
+        // startup gradients to ±1e3, destabilizing Adam → cuBLAS overflow.
+        // New formula `1e6 × isv.max(1.0)` guarantees max_abs ≥ 1e6 in all
+        // ISV states (cold-start = 1e6, warm = 1e6 × isv). F0 no-op intent
+        // preserved (F0 inputs ≪ 1e6); F1 startup gradients ≤ 1e6 are
+        // un-clipped. Per `feedback_isv_for_adaptive_bounds`, the 1.0 ε
+        // floor is a numerical-stability bound (Invariant 1 carve-out) on
+        // the ISV multiplier, not a hardcoded tuning constant on the bound
+        // itself.
         let h_s2_rms_ema = self.read_isv_signal_at(H_S2_RMS_EMA_INDEX);
-        let max_abs_input = (1e6_f32 * h_s2_rms_ema).max(1e3_f32);
+        let max_abs_input = 1e6_f32 * h_s2_rms_ema.max(1.0_f32);
         self.launch_clamp_finite_f32(self.ptrs.bw_d_h_s2, b * sh2, max_abs_input)?;
 
         // ── 3. GRN trunk backward (encoder_backward_chain) ──
@@ -18627,8 +18639,15 @@ impl GpuDqnTrainer {
         // these pre-clamp checks the slot 24/25 flags would observe
         // sanitised zeros and never fire. Mirrors the slot 35 check at
         // line 6949 pattern.
+        // SP1 Phase C fix-up #2 (cold-start ε floor): same pathology as
+        // line 6967 — fold-boundary ISV reset puts Q_DIR_ABS_REF near 0
+        // under original `(1e6 × isv).max(1e3)`, clipping legitimate F1
+        // c51_grad outputs to ±1e3 and destabilizing Bottleneck Linear bwd.
+        // Formula `1e6 × isv.max(1.0)` guarantees max_abs ≥ 1e6 (cold-start
+        // = 1e6, warm = 1e6 × isv). F0 no-op intent preserved; F1 startup
+        // gradients ≤ 1e6 are un-clipped.
         let q_dir_abs_ref = self.read_isv_signal_at(Q_DIR_ABS_REF_INDEX);
-        let max_abs_q = (1e6_f32 * q_dir_abs_ref).max(1e3_f32);
+        let max_abs_q = 1e6_f32 * q_dir_abs_ref.max(1.0_f32);
         let n_val_clamp = self.d_value_logits_buf.len();
         let n_adv_clamp = self.d_adv_logits_buf.len();
         self.check_nan_f32(d_value_logits_ptr, n_val_clamp, 24)?;

docs/dqn-wire-up-audit.md

@@ -2260,3 +2260,5 @@ F0 risk: low — guards activate at 1e6× ISV magnitude (several orders of magni
 Permanent diagnostic infrastructure (Phase B slots 24-35) stays as regression sentinel — will detect any future regression of the cuBLAS-overflow NaN class. Combined RELATED commit per `feedback_no_partial_refactor` — both kernels patched in one commit since they share the same unsafe-pattern (cuBLAS GEMM extreme-intermediate-product overflow) + the same fix template.
 
 SP1 Phase C quality fix-up (2026-04-29): commit-quality follow-up to `97f1d25f5`. (1) The post-clamps in `apply_iqn_trunk_gradient` (slot 26 site) and `launch_cublas_backward_to` (slots 24/25/32 sites) ran BEFORE `run_nan_checks_post_backward` (fused_training.rs:1540), silently zeroing the buffers the diagnostic was supposed to observe — the regression sentinel for these 4 slots was effectively dead. Inline `check_nan_f32` calls now fire IMMEDIATELY BEFORE each `launch_clamp_finite_f32`, mirroring the existing slot 35 check at line 6949: the diagnostic captures NaN entry, the clamp then sanitises so propagation stops, and the flag remains for the regression sentinel readback path. (2) Corrected the F0 paper-review reference: the original commit cited `5.0 norm-clip at line 16747` which is actually a `cuMemsetD32Async` zero-init; the real L2 norm-clip lives at lines 16827-16868 (`max_d_logit_norm = 5.0_f32` + `clip_grad_kernel`). Reasoning tightened: L2 norm ≤ 5.0 worst-case bounds per-element |x| ≤ 5.0 (single-element edge), still several orders of magnitude below the 1e6 max_abs guard. (3) Inline comment at `gpu_dqn_trainer.rs:6951` mislabeled the clamp target as `slot 27 input` — the kernel actually clamps `bw_d_h_s2` post-DtoD, which is the slot 35 buffer; slot 27 is the source-side `iqn_d_h_s2_ptr`. Comment fixed. (4) Pre-clamp rationale articulated: the pre-clamps (slots 24/25/35) target buffers Phase B smoke confirmed CLEAN at F1 first-fire, so they are NOT the smoking gun. They are defense-in-depth regression protection — sanitise input buffers to forestall any FUTURE pathology that creates extreme-but-finite cuBLAS inputs (different from the F1 ep2 NaN where outputs overflow on finite-but-large inputs). They cost 4 additional graph nodes per step (~µs) and are no-ops on F0/F1 typical inputs (≪ 1e6 max_abs guard). The pattern covers both failure modes in one fix per `feedback_no_partial_refactor`. (5) Renamed `clamp_finite_f32_kernel` → `dqn_clamp_finite_f32_kernel` for consistency with sibling utility kernels (`dqn_nan_check_f32`, `dqn_zero_kernel`, `dqn_grad_norm_kernel`); the Rust wrapper retains its `launch_clamp_finite_f32` name (matches `launch_check_nan_f32` precedent — wrapper names don't carry the `dqn_*` prefix).
+
+SP1 Phase C fix-up #2 (2026-04-29): tightened ε floor formula at the two ISV-driven max_abs sites in `apply_iqn_trunk_gradient` (line ~6967) and `launch_cublas_backward_to` (line ~18631). Original `(1e6 × isv).max(1e3)` clipped legitimate F1 startup gradients to ±1e3 when fold-boundary ISV reset put `H_S2_RMS_EMA_INDEX` or `Q_DIR_ABS_REF_INDEX` near 0 — verified by smoke `smoke-test-dr2bn` (commit 19b008e1c) which F1-NaN'd at step 240 (vs step 890 in pre-fix smoke `smoke-test-xvzgk`). New formula `1e6 × isv.max(1.0)` guarantees `max_abs ≥ 1e6` in all ISV states (cold-start = 1e6, warm = 1e6 × isv), removing cold-start clamp pathology. F0 no-op intent preserved (F0 inputs ≪ 1e6 in all states). The 1.0 ε floor is on the ISV multiplier (Invariant 1 carve-out per `feedback_isv_for_adaptive_bounds`), not on the bound itself — so the bound is still ISV-driven when ISV is meaningful.