compute-sanitizer --tool racecheck found 4000+ hazards per kernel invocation
in both c51_loss_batched and mse_loss_batched. The race is between consecutive
block_reduce calls sharing the same shmem_reduce buffer:
1. reduce_max returns shmem_reduce[0] → next reduce_sum writes shmem_reduce[warp]
2. reduce_sum returns shmem_reduce[0] → next reduce_max writes shmem_reduce[warp]
Fixed by adding __syncthreads() after reading the return value in both
block_reduce_max_f and block_reduce_sum_f (read-then-sync pattern). Also
added explicit barriers between spectral decoupling reduces and subsequent
softmax/log_softmax calls.
This race caused non-deterministic NaN in loss values. The effect was masked
by GPU printf statements (which acted as implicit memory barriers) and by
CUDA Graph replay timing differences.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
137480c887