jgrusewski 58ff8db406 fix: aux GEMM gradient explosion — scale 0.5→0.01 + clip after backward ...

The exposure aux backward GEMM amplifies gradients through the
hidden activation matrix (d_logits × h_b0), producing grad_norm=5000+.
With max_grad_norm=10, this throttles ALL gradients by 500x (lr/500).

Fix: reduce aux_weight 0.5→0.01 (50x) + clip_grad_buf_inplace after
aux GEMM to cap combined gradient at max_grad_norm.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-08 10:03:54 +02:00

..

plans

fix: aux GEMM gradient explosion — scale 0.5→0.01 + clip after backward

2026-04-08 10:03:54 +02:00

specs

docs: H100 graph-level optimization spec — 4 deferred fixes (~13s/epoch)

2026-04-08 08:48:36 +02:00