bisect(sp22): disable Step 8/11 launches (c51_aux_dw + adam_w_aux)
User-requested static-analysis-first bisect (Option 2): comment out both launch_c51_aux_dw and launch_adam_w_aux in submit_adam_ops to isolate forward vs backward as NaN source. With these disabled: - Forward atom-shift kernels still run (no-op with W=0) - W stays at [0,0,0,0] (no Adam update) - dW kernel doesn't run Hypothesis test: - Clean smoke -> backward is the bug - NaN persists -> forward is the bug Audit doc updated. Cargo check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -30323,13 +30323,14 @@ impl GpuDqnTrainer {
|
||||
self.launch_adam_update()?;
|
||||
|
||||
// ── 6a. SP22 H6 Phase 3 α Step 8 + Step 11 (2026-05-13) ──
|
||||
// Compute dW and Adam-update w_aux_to_q_dir [4] alongside the
|
||||
// main Adam step. Both kernels are graph-capture-safe (all
|
||||
// varying inputs via device-mapped pointers; constants via
|
||||
// value args). The dW kernel reads scratch buffers populated
|
||||
// by the captured c51_loss_kernel forward earlier in this step.
|
||||
self.launch_c51_aux_dw()?;
|
||||
self.launch_adam_w_aux()?;
|
||||
// BISECT DIAGNOSTIC (2026-05-13): Step 8/11 launches DISABLED to
|
||||
// isolate forward vs backward as NaN source. If smoke at this
|
||||
// commit is clean (no dir/mag NaN), backward is the culprit. If
|
||||
// NaN persists, forward (compute_expected_q / mag_concat_qdir /
|
||||
// c51_loss SP code) is the culprit.
|
||||
// Restore both launches once the bisect identifies the cause.
|
||||
// self.launch_c51_aux_dw()?;
|
||||
// self.launch_adam_w_aux()?;
|
||||
|
||||
// ── 6.5. Snapshot grad_buf → prev_grad_buf for next step's vaccine comparison ──
|
||||
// Graph-safe: submit_adam_ops is captured in adam_update child graph.
|
||||
|
||||
@@ -17390,3 +17390,18 @@ This prevents NaN propagation. State_121 with NaN → treated as 0 → atom_shif
|
||||
The defensive guard unblocks smoke verification of the rest of the Phase 3 mechanism. The state_121 NaN source investigation comes after the smoke confirms the guard works.
|
||||
|
||||
**Verification**: cargo check -p ml --lib clean.
|
||||
|
||||
#### Bisect cut #1 — disable Step 8/11 launches (2026-05-13)
|
||||
|
||||
User-requested Option 2 (surgical bisect). Phase 2 baseline `f9192f70a` confirmed CLEAN — `q_var_per_branch [0, 0, 0, 0]` at HEALTH_DIAG[0]. So my Phase 3 changes definitively introduce the NaN.
|
||||
|
||||
Static analysis ruled out:
|
||||
- atom_positions_buf — Phase 3 doesn't touch it
|
||||
- 0 × NaN propagation via state_121 — defensive guard added, NaN persists
|
||||
- Kernel arg count mismatch — 52 args match between kernel and launcher
|
||||
|
||||
First bisect cut: comment out `launch_c51_aux_dw()` and `launch_adam_w_aux()` calls in `submit_adam_ops`. Forward atom-shift kernels still run (no-op with W=0).
|
||||
|
||||
Hypothesis test:
|
||||
- Clean smoke → backward (Step 8/11) is the bug
|
||||
- NaN persists → forward (compute_expected_q / mag_concat / c51_loss SP code) is the bug
|
||||
|
||||
Reference in New Issue
Block a user