diff --git a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs index ccfcdcfdd..efc854573 100644 --- a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +++ b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs @@ -6963,8 +6963,20 @@ impl GpuDqnTrainer { // (≥ 1e6) trigger clamp. The slot 35 NaN check at line 6949 fires // BEFORE this clamp, so any NaN entry is captured in the regression // sentinel before sanitisation — see Issue 1 fix in commit fix-up. + // SP1 Phase C fix-up #2 (cold-start ε floor): smoke-test-dr2bn + // (commit 19b008e1c) FAILED at F1 step 240 with the ORIGINAL formula + // `(1e6 × isv).max(1e3)` because fold-boundary ISV reset puts ISV[96] + // near 0, giving max_abs = 1e3 — too tight; clipped legitimate F1 + // startup gradients to ±1e3, destabilizing Adam → cuBLAS overflow. + // New formula `1e6 × isv.max(1.0)` guarantees max_abs ≥ 1e6 in all + // ISV states (cold-start = 1e6, warm = 1e6 × isv). F0 no-op intent + // preserved (F0 inputs ≪ 1e6); F1 startup gradients ≤ 1e6 are + // un-clipped. Per `feedback_isv_for_adaptive_bounds`, the 1.0 ε + // floor is a numerical-stability bound (Invariant 1 carve-out) on + // the ISV multiplier, not a hardcoded tuning constant on the bound + // itself. let h_s2_rms_ema = self.read_isv_signal_at(H_S2_RMS_EMA_INDEX); - let max_abs_input = (1e6_f32 * h_s2_rms_ema).max(1e3_f32); + let max_abs_input = 1e6_f32 * h_s2_rms_ema.max(1.0_f32); self.launch_clamp_finite_f32(self.ptrs.bw_d_h_s2, b * sh2, max_abs_input)?; // ── 3. GRN trunk backward (encoder_backward_chain) ── @@ -18627,8 +18639,15 @@ impl GpuDqnTrainer { // these pre-clamp checks the slot 24/25 flags would observe // sanitised zeros and never fire. Mirrors the slot 35 check at // line 6949 pattern. + // SP1 Phase C fix-up #2 (cold-start ε floor): same pathology as + // line 6967 — fold-boundary ISV reset puts Q_DIR_ABS_REF near 0 + // under original `(1e6 × isv).max(1e3)`, clipping legitimate F1 + // c51_grad outputs to ±1e3 and destabilizing Bottleneck Linear bwd. + // Formula `1e6 × isv.max(1.0)` guarantees max_abs ≥ 1e6 (cold-start + // = 1e6, warm = 1e6 × isv). F0 no-op intent preserved; F1 startup + // gradients ≤ 1e6 are un-clipped. let q_dir_abs_ref = self.read_isv_signal_at(Q_DIR_ABS_REF_INDEX); - let max_abs_q = (1e6_f32 * q_dir_abs_ref).max(1e3_f32); + let max_abs_q = 1e6_f32 * q_dir_abs_ref.max(1.0_f32); let n_val_clamp = self.d_value_logits_buf.len(); let n_adv_clamp = self.d_adv_logits_buf.len(); self.check_nan_f32(d_value_logits_ptr, n_val_clamp, 24)?; diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index ab1e9e53b..3ef339fc6 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2260,3 +2260,5 @@ F0 risk: low — guards activate at 1e6× ISV magnitude (several orders of magni Permanent diagnostic infrastructure (Phase B slots 24-35) stays as regression sentinel — will detect any future regression of the cuBLAS-overflow NaN class. Combined RELATED commit per `feedback_no_partial_refactor` — both kernels patched in one commit since they share the same unsafe-pattern (cuBLAS GEMM extreme-intermediate-product overflow) + the same fix template. SP1 Phase C quality fix-up (2026-04-29): commit-quality follow-up to `97f1d25f5`. (1) The post-clamps in `apply_iqn_trunk_gradient` (slot 26 site) and `launch_cublas_backward_to` (slots 24/25/32 sites) ran BEFORE `run_nan_checks_post_backward` (fused_training.rs:1540), silently zeroing the buffers the diagnostic was supposed to observe — the regression sentinel for these 4 slots was effectively dead. Inline `check_nan_f32` calls now fire IMMEDIATELY BEFORE each `launch_clamp_finite_f32`, mirroring the existing slot 35 check at line 6949: the diagnostic captures NaN entry, the clamp then sanitises so propagation stops, and the flag remains for the regression sentinel readback path. (2) Corrected the F0 paper-review reference: the original commit cited `5.0 norm-clip at line 16747` which is actually a `cuMemsetD32Async` zero-init; the real L2 norm-clip lives at lines 16827-16868 (`max_d_logit_norm = 5.0_f32` + `clip_grad_kernel`). Reasoning tightened: L2 norm ≤ 5.0 worst-case bounds per-element |x| ≤ 5.0 (single-element edge), still several orders of magnitude below the 1e6 max_abs guard. (3) Inline comment at `gpu_dqn_trainer.rs:6951` mislabeled the clamp target as `slot 27 input` — the kernel actually clamps `bw_d_h_s2` post-DtoD, which is the slot 35 buffer; slot 27 is the source-side `iqn_d_h_s2_ptr`. Comment fixed. (4) Pre-clamp rationale articulated: the pre-clamps (slots 24/25/35) target buffers Phase B smoke confirmed CLEAN at F1 first-fire, so they are NOT the smoking gun. They are defense-in-depth regression protection — sanitise input buffers to forestall any FUTURE pathology that creates extreme-but-finite cuBLAS inputs (different from the F1 ep2 NaN where outputs overflow on finite-but-large inputs). They cost 4 additional graph nodes per step (~µs) and are no-ops on F0/F1 typical inputs (≪ 1e6 max_abs guard). The pattern covers both failure modes in one fix per `feedback_no_partial_refactor`. (5) Renamed `clamp_finite_f32_kernel` → `dqn_clamp_finite_f32_kernel` for consistency with sibling utility kernels (`dqn_nan_check_f32`, `dqn_zero_kernel`, `dqn_grad_norm_kernel`); the Rust wrapper retains its `launch_clamp_finite_f32` name (matches `launch_check_nan_f32` precedent — wrapper names don't carry the `dqn_*` prefix). + +SP1 Phase C fix-up #2 (2026-04-29): tightened ε floor formula at the two ISV-driven max_abs sites in `apply_iqn_trunk_gradient` (line ~6967) and `launch_cublas_backward_to` (line ~18631). Original `(1e6 × isv).max(1e3)` clipped legitimate F1 startup gradients to ±1e3 when fold-boundary ISV reset put `H_S2_RMS_EMA_INDEX` or `Q_DIR_ABS_REF_INDEX` near 0 — verified by smoke `smoke-test-dr2bn` (commit 19b008e1c) which F1-NaN'd at step 240 (vs step 890 in pre-fix smoke `smoke-test-xvzgk`). New formula `1e6 × isv.max(1.0)` guarantees `max_abs ≥ 1e6` in all ISV states (cold-start = 1e6, warm = 1e6 × isv), removing cold-start clamp pathology. F0 no-op intent preserved (F0 inputs ≪ 1e6 in all states). The 1.0 ε floor is on the ISV multiplier (Invariant 1 carve-out per `feedback_isv_for_adaptive_bounds`), not on the bound itself — so the bound is still ISV-driven when ISV is meaningful.