From a1945de031bc5637966d88e3e114d20fc4d372ea Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Wed, 13 May 2026 10:46:17 +0200 Subject: [PATCH] bisect(sp22): disable Step 8/11 launches (c51_aux_dw + adam_w_aux) User-requested static-analysis-first bisect (Option 2): comment out both launch_c51_aux_dw and launch_adam_w_aux in submit_adam_ops to isolate forward vs backward as NaN source. With these disabled: - Forward atom-shift kernels still run (no-op with W=0) - W stays at [0,0,0,0] (no Adam update) - dW kernel doesn't run Hypothesis test: - Clean smoke -> backward is the bug - NaN persists -> forward is the bug Audit doc updated. Cargo check clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs | 15 ++++++++------- docs/dqn-wire-up-audit.md | 15 +++++++++++++++ 2 files changed, 23 insertions(+), 7 deletions(-) diff --git a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs index 15ffd2abf..57527fdd7 100644 --- a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +++ b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs @@ -30323,13 +30323,14 @@ impl GpuDqnTrainer { self.launch_adam_update()?; // ── 6a. SP22 H6 Phase 3 α Step 8 + Step 11 (2026-05-13) ── - // Compute dW and Adam-update w_aux_to_q_dir [4] alongside the - // main Adam step. Both kernels are graph-capture-safe (all - // varying inputs via device-mapped pointers; constants via - // value args). The dW kernel reads scratch buffers populated - // by the captured c51_loss_kernel forward earlier in this step. - self.launch_c51_aux_dw()?; - self.launch_adam_w_aux()?; + // BISECT DIAGNOSTIC (2026-05-13): Step 8/11 launches DISABLED to + // isolate forward vs backward as NaN source. If smoke at this + // commit is clean (no dir/mag NaN), backward is the culprit. If + // NaN persists, forward (compute_expected_q / mag_concat_qdir / + // c51_loss SP code) is the culprit. + // Restore both launches once the bisect identifies the cause. + // self.launch_c51_aux_dw()?; + // self.launch_adam_w_aux()?; // ── 6.5. Snapshot grad_buf → prev_grad_buf for next step's vaccine comparison ── // Graph-safe: submit_adam_ops is captured in adam_update child graph. diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index e5f8de83b..3ff03923b 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -17390,3 +17390,18 @@ This prevents NaN propagation. State_121 with NaN → treated as 0 → atom_shif The defensive guard unblocks smoke verification of the rest of the Phase 3 mechanism. The state_121 NaN source investigation comes after the smoke confirms the guard works. **Verification**: cargo check -p ml --lib clean. + +#### Bisect cut #1 — disable Step 8/11 launches (2026-05-13) + +User-requested Option 2 (surgical bisect). Phase 2 baseline `f9192f70a` confirmed CLEAN — `q_var_per_branch [0, 0, 0, 0]` at HEALTH_DIAG[0]. So my Phase 3 changes definitively introduce the NaN. + +Static analysis ruled out: +- atom_positions_buf — Phase 3 doesn't touch it +- 0 × NaN propagation via state_121 — defensive guard added, NaN persists +- Kernel arg count mismatch — 52 args match between kernel and launcher + +First bisect cut: comment out `launch_c51_aux_dw()` and `launch_adam_w_aux()` calls in `submit_adam_ops`. Forward atom-shift kernels still run (no-op with W=0). + +Hypothesis test: +- Clean smoke → backward (Step 8/11) is the bug +- NaN persists → forward (compute_expected_q / mag_concat / c51_loss SP code) is the bug