The algo cache serialized heuristic-selected algo bytes to disk with
GPU name + CUDA version validation. This created a divergent code path
between dev (RTX 3050) and prod (H100) — different machines would get
different cached algos, producing different but "frozen" results.
Determinism should come from the algorithm itself being deterministic,
not from caching one machine's non-deterministic output. The proper
fix is routing evaluation through the trainer's CUDA-Graphed forward
pass (which IS deterministic by design).
Kept: COMPUTE_32F_PEDANTIC on all cublasLt matmul descriptors (disables
TF32, universal across GPUs). Kept: IQN + attention backward determinism,
single-stream eval, evaluator reuse.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminate training non-determinism from three major sources:
1. IQN backward: split iqn_backward_kernel into iqn_backward_per_sample
(saves dL_dq[B,N], d_h_s2 via register accumulation) +
iqn_weight_grad_reduce (deterministic per-parameter reduction from
dL_dq + saved activations). Two-phase grad norm, deterministic loss
reduce. Zero atomicAdd in IQN training path.
2. Attention backward: per-sample d_params buffer replaces cross-batch
atomicAdd on weight gradients. attn_weight_grad_reduce sums across
samples deterministically. Two-phase grad norm.
3. cublasLt: COMPUTE_32F_FAST_TF32 → COMPUTE_32F_PEDANTIC (disables
TF32 19-bit rounding). New cublas_algo_cache module serializes
heuristic-selected algo structs to disk (config/cublas_algo_cache.json)
with CUDA version + GPU name validation. Eliminates algo selection
variability between process invocations (91/91 cache hits verified).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.
Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
