bisect(sp22): disable Step 8/11 launches (c51_aux_dw + adam_w_aux)

User-requested static-analysis-first bisect (Option 2): comment out
both launch_c51_aux_dw and launch_adam_w_aux in submit_adam_ops to
isolate forward vs backward as NaN source.

With these disabled:
- Forward atom-shift kernels still run (no-op with W=0)
- W stays at [0,0,0,0] (no Adam update)
- dW kernel doesn't run

Hypothesis test:
- Clean smoke -> backward is the bug
- NaN persists -> forward is the bug

Audit doc updated. Cargo check clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-13 10:46:17 +02:00
parent a683c4bc52
commit a1945de031
2 changed files with 23 additions and 7 deletions

View File

@@ -30323,13 +30323,14 @@ impl GpuDqnTrainer {
self.launch_adam_update()?;
// ── 6a. SP22 H6 Phase 3 α Step 8 + Step 11 (2026-05-13) ──
// Compute dW and Adam-update w_aux_to_q_dir [4] alongside the
// main Adam step. Both kernels are graph-capture-safe (all
// varying inputs via device-mapped pointers; constants via
// value args). The dW kernel reads scratch buffers populated
// by the captured c51_loss_kernel forward earlier in this step.
self.launch_c51_aux_dw()?;
self.launch_adam_w_aux()?;
// BISECT DIAGNOSTIC (2026-05-13): Step 8/11 launches DISABLED to
// isolate forward vs backward as NaN source. If smoke at this
// commit is clean (no dir/mag NaN), backward is the culprit. If
// NaN persists, forward (compute_expected_q / mag_concat_qdir /
// c51_loss SP code) is the culprit.
// Restore both launches once the bisect identifies the cause.
// self.launch_c51_aux_dw()?;
// self.launch_adam_w_aux()?;
// ── 6.5. Snapshot grad_buf → prev_grad_buf for next step's vaccine comparison ──
// Graph-safe: submit_adam_ops is captured in adam_update child graph.

View File

@@ -17390,3 +17390,18 @@ This prevents NaN propagation. State_121 with NaN → treated as 0 → atom_shif
The defensive guard unblocks smoke verification of the rest of the Phase 3 mechanism. The state_121 NaN source investigation comes after the smoke confirms the guard works.
**Verification**: cargo check -p ml --lib clean.
#### Bisect cut #1 — disable Step 8/11 launches (2026-05-13)
User-requested Option 2 (surgical bisect). Phase 2 baseline `f9192f70a` confirmed CLEAN — `q_var_per_branch [0, 0, 0, 0]` at HEALTH_DIAG[0]. So my Phase 3 changes definitively introduce the NaN.
Static analysis ruled out:
- atom_positions_buf — Phase 3 doesn't touch it
- 0 × NaN propagation via state_121 — defensive guard added, NaN persists
- Kernel arg count mismatch — 52 args match between kernel and launcher
First bisect cut: comment out `launch_c51_aux_dw()` and `launch_adam_w_aux()` calls in `submit_adam_ops`. Forward atom-shift kernels still run (no-op with W=0).
Hypothesis test:
- Clean smoke → backward (Step 8/11) is the bug
- NaN persists → forward (compute_expected_q / mag_concat / c51_loss SP code) is the bug