Files
foxhunt/crates
jgrusewski 3d0824eb9b perf: TF32 tensor cores for backward GEMMs (2-3x throughput on H100)
Changed 2 backward cuBLAS GEMMs from CUBLAS_COMPUTE_32F to
CUBLAS_COMPUTE_32F_FAST_TF32. Both activation gradient (BF16→BF16)
and weight gradient (BF16→F32) paths now use H100 tensor cores.

TF32's 10-bit mantissa is sufficient for gradients — Adam's momentum
exponential moving average smooths any additional noise. Forward
GEMMs already use TF32 (unchanged).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 22:49:13 +02:00
..