Fold 1 of train-multi-seed-72fl6 hit NaN at step 5 with `flagged=[]` —
diagnostic told us nothing because:
1. The 8 NaN-check kernels in `run_nan_checks_*_forward` were never
invoked from production training. Allocated-zero flags read back
as zeros; halt-on-NaN reported "no buffer flagged" while loss was
demonstrably NaN via host-side pinned readback.
2. The error message labels (training_loop.rs:1853) referenced
`bf16_params` — dead code from before the bf16→TF32 switch — and
the format string did not match the actual kernel index→buffer
mapping.
Wired the existing NaN check kernels into `submit_post_aux_ops` so
they run every step inside the captured parent graph (post_aux child).
Flags persist across steps within a fold so the FIRST buffer to go
NaN remains visible at host-readback time; reset is host-side at fold
boundary in `reset_for_fold`.
Extended coverage beyond the original 8 buffers (states, on_v_logits,
on_b_logits, mse_loss, params, grad_buf, save_current_lp) with 4
loss-component buffers most likely to break under post-S&P + adversarial
regime stress:
Flag 8 save_projected C51 target distribution
Flag 9 moe_gate_softmax MoE gate over 8 experts
Flag 10 aux_nb_loss_scalar aux next-bar MSE
Flag 11 aux_rg_loss_scalar aux regime CE
Flag buffer grown 8 → 16; slots 12-15 reserved for future expansion
(CQL / IQN / per-branch backward).
Updated training_loop.rs error message: 16 named slots matching actual
kernel layout, per-flag indexed name in flagged list, dropped stale
`bf16_params` / `_bf16` suffixes, explicit hint when `flagged=[]`.
Cost per step: ~12 single-block reductions × ~5µs = ~60µs. <0.1%
overhead at the 258ms/step measured on L40S — pinpoints source
buffer on next NaN halt.
Per `feedback_no_legacy_aliases.md`, `feedback_no_stubs.md`,
`feedback_trust_code_not_docs.md`.
756b1ef317