The f32→bf16 cast in cast_d_logits_to_bf16() reads
b*(b0+b1+b2+b3)*na + 32*4 elements, but d_adv_logits_buf and
d_adv_logits_bf16 were allocated with only +32*3 padding (96 vs 128).
The 32 extra f32 reads went past the allocation into adjacent GPU
memory. On H100 with 80GB and different memory layout, this grabbed
NaN/garbage values. The NaN propagated through the cuBLAS backward
pass into grad_buf, killing training at step ~720 (non-deterministic
depending on what lives in adjacent memory).
Fix: allocate +32*4 padding in d_adv_logits_buf, d_adv_logits_bf16,
and d_adv_logits_mse to match the cast kernel's access pattern.
Also reverts c51_alpha_max default to 1.0 — the gradient collapse
was from this buffer bug, not C51 premature convergence. The
c51_alpha_max parameter is kept for hyperopt exploration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fce606a421