Files
foxhunt/crates
jgrusewski cfc44be13c fix: capture graph_mega at step 0 + remove MSE graph switch
Two root causes of cross-process non-determinism fixed:

1. MSE→C51 graph switch: graph_forward_mse and graph_forward were
   captured in different sessions with different cublasLtMatmul configs.
   Fix: always use blended graph_forward (c51_alpha controls blend).
   Deleted LossMode enum, graph_forward_mse, and all switching code.

2. Multiple capture sessions at steps 0-2: graph_forward, graph_ddqn,
   and graph_adam were captured individually before graph_mega at step 2.
   Each capture contaminated cuBLAS internal state.
   Fix: capture graph_mega at step 0 (single capture session).

Result: runs with same initial cublasLtMatmul capture are fully
bit-identical across all 30 epochs. Remaining cross-process variation
is from the single initial cublasLtMatmul call during graph capture.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 12:24:20 +02:00
..