Real fix for the F1 step-3540 NaN. Commit 8956c2fb7 wired Mech 9
(post-Adam |p_val| clamp) into ONE Adam kernel (dqn_adam_update_kernel)
out of FIVE in the codebase. The slot-26 GEMM (apply_iqn_trunk_gradient
output) computes `grad_iqn @ W_iqn^T`; W_iqn lives in IQN's separate
param buffer, updated by iqn_adam_kernel — never reached by Mech 9.
Slots 44-45 only cover trunk + heads, leaving IQN weights without a
diagnostic. The chain: IQN weights drift via iqn_adam_kernel → finite
gradient × non-finite weight → slot-26 NaN. Same partial-refactor class
as Mech 4's missing GpuAttention reset (caught by B5 audit), corrected
the same way: every consumer of the contract migrates together.
Extended Mech 9 to all four remaining Adam kernels:
- iqn_adam_kernel (iqn_dual_head_kernel.cu)
- iql_adam_kernel (iql_value_kernel.cu)
- attn_adam_kernel (attention_backward_kernel.cu — used by GpuAttention + GpuTlob)
- curiosity_adam_step (curiosity_training_kernel.cu)
Same `if (weight_clamp_max_abs > 0.0f) p = fminf(fmaxf(p, -bound), bound)`
pattern. Same `100 × Q_ABS_REF.max(1.0)` ISV-driven bound. Each launch
site computes the bound host-side and passes it as the trailing kernel
arg, mirroring the existing dqn_adam_update_kernel pattern. DT launch
keeps the 0.0 disable (offline-RL, outside SP3 scope). Curiosity reads
ISV from FusedTrainingCtx in training_loop and threads through the
collector wrapper (collector doesn't own ISV).
Added IQN-weight diagnostic slot 48:
- nan_flags_buf 48 → 49
- Fused-kernel block count 24 → 25; new branch for absolute slot 48
(relative slot 24): threshold = 1e3 × q_abs_ref_eff (matches slot 44-45)
- Metadata buffers (nan_check_buf_ptrs/_lens) populate slot 48 with
IQN online_params pointer + length (new public accessors on GpuIqnHead)
- name table (halt_grad_collapse): 49 entries with "iqn_weight_max"
- Audit doc: slot 48 row in Mech 5 table; Mech 9 row updated to span
all 5 Adam kernels
No new ISV slots. No graph-topology change beyond the +1 block in the
already-fused NaN check. cargo check -p ml --lib clean (12 pre-existing
warnings, none new).
Validation: deferred to one L40S smoke at this HEAD. Expected: F1
trains past step 3720 (Mech-6-only ceiling) and slots 36-43 + new
slot 48 stay quiet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1bcc703920