diff --git a/docs/dqn-backward-nan-audit.md b/docs/dqn-backward-nan-audit.md index 5bd86e440..ea76e49a5 100644 --- a/docs/dqn-backward-nan-audit.md +++ b/docs/dqn-backward-nan-audit.md @@ -728,3 +728,49 @@ Closing SP1 with the following declared deliverables: The 7 SP1 pass criteria as originally specified are not all achievable within SP1's surgical-fix scope. Criteria 1, 2, 4, 5, 6, 7 require SP3 structural fix. Criterion 3 (F0 ≥ 53.08) requires SP2 instrumentation optimization. **Net SP1 outcome:** the investigation produced the diagnostic infrastructure + pinpointed source kernels + fixed the cold-start clamp pathology that this scope CAN address. The remaining items are correctly classified for SP2/SP3 scopes per the spec's 3-sub-project decomposition. + +--- + +## Phase E — SP2 fused NaN check kernel (instrumentation overhead fix) + +**Status:** SP2 implementation complete (Tasks A1-A4 on branch `plan-c-phase-2-thompson`). Gate 1 smoke pending (Task A6). + +### Change summary + +Replaces 8 per-step `dqn_nan_check_f32` launches in `run_nan_checks_post_backward` with a single launch of `dqn_nan_check_fused_f32_kernel` (1 kernel × 12 blocks). Slots 24-30 + 32 covered by the fused kernel; slot 31 deferred via null entry; slots 33-35 inline checks at backward orchestration phases stay separate (multi-point localization preserved). + +### Commits + +| Task | SHA | Scope | +|------|-----|-------| +| A1 | `0250722a0` | Append `dqn_nan_check_fused_f32_kernel` to `dqn_utility_kernels.cu` | +| A2 | `82b6bd369` | Add `nan_check_buf_ptrs`/`nan_check_buf_lens` device buffer fields | +| A3 | `fbf48df9d` | Refactor fields to `MappedU64Buffer`/`MappedI32Buffer` (eliminates HtoD); add `populate_nan_check_meta` + `launch_nan_check_fused_f32`; register kernel; call populate from constructor | +| A4 | `b5a064f6c` | Replace 8 individual launches in `run_nan_checks_post_backward` with single fused launch; simplify signature to no-args | + +Net delta: -109 lines (code shrinks via call-site collapse). + +### Mapped-pinned design + +The metadata buffers (`nan_check_buf_ptrs: MappedU64Buffer`, `nan_check_buf_lens: MappedI32Buffer`) use `cuMemHostAlloc(DEVICEMAP|PORTABLE) + cuMemHostGetDevicePointer_v2` — host-side write at construction, kernel reads via device-mapped pointer. Zero HtoD copies introduced. Per `feedback_no_htod_htoh_only_mapped_pinned`. + +### Sticky-flag semantics preserved + +Kernel writes `1` on NaN, never clears. Slot 24-35 firing topology should be IDENTICAL to pre-SP2 (proven by F1 step-3540 regression test that the SP1 audit characterized). + +### Slots 33-35 inline checks (intentionally separate) + +The fused kernel handles slots 24-30 + 32 (8 slots with concrete buffers) + slot 31 (null/deferred) + slots 33-35 (null entries — the inline checks fire at distinct backward orchestration phases for localization). Inline call sites: +- Slot 33 (`bw_d_h_s2_post_main`): `gpu_dqn_trainer.rs:18580` (post `backward_full` + branch concat) +- Slot 34 (`bw_d_h_s2_post_aux`): `gpu_dqn_trainer.rs:18601` (post `aux_heads_backward` SAXPY) +- Slot 35 (`bw_d_h_s2_post_iqn`): `gpu_dqn_trainer.rs:6939` (post IQN DtoD overwrite) + +Consolidating slot 33-35 into the fused kernel would lose the multi-point snapshot — these stay as-is. + +### Expected validation outcome (Gate 1 smoke) + +- F0 Best Sharpe ≥ 53.08 (return to ~55 baseline expected, vs ~35 in 3 prior smokes) +- Slots 24-35 firing topology unchanged at F1 step ~3540 (the existing regression test) +- No new errors / warnings + +If Gate 1 fails, halt before SP3 starts and re-investigate F0 cause (alignment effects? stochastic variance?).