L40S production training crashed in launch_ofi_embed_backward with
CUDA_ERROR_ILLEGAL_ADDRESS at the first denoise_bias_grad_p1 launch in
the adam_grad child graph (workflow train-multi-seed-tdjtr fold 0).
Root cause: denoise_bias_grad_p1/p2 CUfunction handles were shared
across two independently-captured child graphs (post_aux and adam_grad).
On Ada/Hopper, when partials buffers land in different VRAM arenas
(L40S B=16384 post-MoE memory layout), the driver's per-node handle
validation trips. Pre-existing latent bug — masked on smaller GPUs and
pre-MoE allocator states where buffers happened to be address-adjacent.
Fix: load a dedicated CUmodule for the OFI bias-grad reduction kernels,
separate from the denoise variant. Same kernel bytecode (EXPECTED_Q_CUBIN
contains both); each child graph gets its own CUfunction reference.
Mirrors the graph_safe_copy_kernel precedent at line ~11523:
"Hopper CUfunction isolation: each child graph needs its own CUmodule".
Constructor adds ~2ms of cubin-load time; zero runtime cost. Smoke
validates no regression at B=64.
Production validation deferred to follow-up L40S re-deploy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aff69261e1