Single-block warp-shuffle scanner that printfs the first non-finite
index/value in a labeled [N] float buffer. Loaded unconditionally so
the module handle is always live; per-launch gating uses the new
`nan_scan_enabled` field, set from `FOXHUNT_NAN_SCAN=1` at trainer
construction. Production training pays zero cost when the env var
is unset.
label_id table (kernel + host helper kept in lockstep):
0 h_t, 1 v_pred, 2 q_logits, 3 pi_logits,
4 advantages, 5 ss_pi_l_pi_per_batch.
Diagnostic context: residual ~33% NaN at step 4 (seed 16962) after
the atomicAdd/V-envelope/0×Inf fixes (10d4614fb, b4aadff75, a6acc25ec).
Adding printf to v_head_fwd_bwd or running under nsys cured it, so the
remaining cause is microarchitectural (timing/memory-ordering) or
localized to a kernel not yet on the trail. nan_scan pinpoints the
first stage whose buffer goes bad when the NaN fires; instrumentation
of the per-step pipeline lands in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
60e96bf55c