Added dqn_nan_check_f32/bf16 kernels for zero-overhead GPU-side NaN
detection between graph stages. Key findings:
- NaN appears in f32 params at step 0 (before first training forward)
- States (input data) are always clean
- The graph capture warmup launches corrupt params via:
C51 forward → NaN gradients → Adam warmup → NaN weights
- Removed Adam warmup launch, but NaN persists — the forward
graph warmup itself may corrupt buffers through bf16 overflow
in cuBLAS backward path
- Step 14/22 NaN in later runs is from accumulated corruption
Next: investigate whether cuBLAS backward with random Xavier weights
produces extreme gradients that overflow bf16 d_logits staging buffers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
57844e1cc5