foxhunt/docs/superpowers/specs/2026-06-02-determinism-foundation.md

Determinism Foundation — Stop Chasing Ghosts

Date: 2026-06-02 Status: Spec draft — MANDATORY PREREQUISITE for any further controller / architecture work Priority: BLOCKER for multi-head policy (2026-06-02-multi-head-policy-with-r-multiple.md), R-multiple reward, and the entire regime-invariance roadmap Empirical motivation: alpha-rl-determinism-check.sh on ml-alpha-regime-observer @ 63fc16f17 showed same-seed runs diverging at step 2, eval pnl Δ=$364,925, run A produced 0 trades while run B produced 347 trades. Bit-equivalence requirement: same seed → same diag rows to fp32 rel-tol 1e-5.

§0. Motivation — the going-in-circles diagnosis was incomplete

The session of 2026-06-01/02 cycled through 4 controller iterations (v4 → v5.2) plus today's grwwh eval. Each iteration produced a different result. We attributed the variance to:

1. Bad controller designs (the v5/v5.1 bugs)
2. Architectural mismatch (single policy can't generalize)
3. Reward signal contamination (Phase 5 hold-bonus)

What we did NOT consider: part of the variance was non-determinism.

The 2026-06-02 mid-smoke determinism check revealed:

• Same seed, same code, same hardware → divergence at step 2
• By step 199: 654 diag leaves diverging
• Eval pnl Δ=$365k between identical configurations
• Behavioral Δ: run A=0 trades vs run B=347 trades (different agent entirely)

Implications:

1. Every controller iteration today might have been measuring seed-noise, not controller effect
2. The "scaffold v5 stuck at w=1.0" might be partially seed-dependent (would different seeds also stick?)
3. The "grwwh eval +$74M then collapse to -$160M" might partially be intra-run variance
4. Until determinism is fixed, every future iteration is also seed-dependent and unfalsifiable

Per feedback_going_in_circles_pattern: "audit the foundation". The foundation has a bug. Fix the foundation before adding more layers.

§1. Investigation strategy — localize divergence to ONE component

Per feedback_systematic_debugging Phase 1 "Gather evidence in multi-component systems": instrument every boundary, find FIRST divergence.

§1.1 Deterministic checksum kernel (NEW infrastructure)

Add crates/ml-alpha/cuda/rl_deterministic_checksum.cu:

• Computes sum-of-squares of any device tensor with provably deterministic accumulation
• Single-block, single-thread accumulation if tensor < 1024 elements (trivially deterministic)
• For larger tensors: block-tree-reduce with fixed accumulation order (block 0 always reads block 1's result, never vice versa; __threadfence_system + counter cudaDeviceSynchronize if needed; NO atomic ops, NO "whichever-block-finishes-last" patterns)
• Returns one f64 per tensor (sum-of-squares, full precision)
• Compile-time guaranteed determinism via construction, not configuration

§1.2 Per-step component checksums in diag

Instrument every major component output in build_diag_value:

"checksums": {
    "encoder_output": checksum(encoder_out_d, b_size × seq_len × hidden_dim),
    "q_logits": checksum(q_logits_d, b_size × n_actions × n_atoms),
    "pi_logits": checksum(pi_logits_d, b_size × n_actions),
    "v_value": checksum(v_d, b_size),
    "replay_sample_indices": checksum_int(per_sampled_indices_d, b_size),
    "rewards_after_shape": checksum(rewards_d, b_size),
    "advantages": checksum(advantages_d, b_size),
    "q_grad": checksum(q_grad_d, all_params),
    "pi_grad": checksum(pi_grad_d, all_params),
    "v_grad": checksum(v_grad_d, all_params),
    "encoder_grad": checksum(encoder_grad_d, all_params),
    "adam_m_sum": checksum(adam_first_moments_d, all_params),
    "adam_v_sum": checksum(adam_second_moments_d, all_params),
    "isv_state": checksum(isv_dev_ptr, RL_SLOTS_END),
}

Cost: ~15 small kernel launches per step, each ~10us → 150us/step overhead. Acceptable in dev mode; toggle off in production.

§1.3 Run determinism-check + localization

scripts/determinism-check.sh now compares checksums leaf-by-leaf. Output: first leaf to diverge + step number + Δ value.

Expected outcomes (one of these wins):

• checksums.encoder_output diverges first → mamba2 culprit (most likely)
• checksums.q_logits diverges first (encoder same) → Q head GEMM split-K culprit
• checksums.pi_logits first → π head GEMM culprit
• checksums.replay_sample_indices first → PER sampling RNG culprit
• checksums.q_grad first (forward same) → backward kernel reduction culprit
• checksums.adam_m_sum first (grads same) → optimizer iteration-order culprit

§2. Standard fixes by component (apply based on §1 localization)

§2.A Mamba2 selective scan (most-likely culprit)

If encoder checksum diverges first:

• Chunk-size determinism: mamba2 selective scan uses chunk-parallel scan. Setting CHUNK_SIZE=1 reduces to sequential scan (trivially deterministic but ~10× slower). Use as ground-truth oracle.
• Reduction-order fix: within each chunk, the scan reduction must be fixed-order. Audit mamba2_selective_scan_fwd.cu for any atomicAdd, any "block 0 OR block 1 writes final" patterns, any non-deterministic chunk ordering.
• Reference: pearl pearl_mamba2_selective_scan_nan_source (Phase 2 v2 work) — same code path was extensively debugged for NaN issues. May have introduced non-det then.

§2.B cuBLAS GEMM split-K

If Q/π/V head checksums diverge first (encoder same):

• Set cublasSetMathMode(cublas_handle, CUBLAS_PEDANTIC_MATH) — disables auto-selection of non-deterministic algorithms
• Disable TF32 (cublasSetMathMode(CUBLAS_DEFAULT_MATH)) — TF32 split-K is non-deterministic per NVIDIA docs
• This REVERSES tasks 30/31 (TF32 enable). Document the perf cost trade-off (~10-15% on H100 GEMMs).
• Alternative: use FP32 explicitly via cublasSgemmStridedBatched with explicit algorithm selection

§2.C cuDNN deterministic mode

If conv/RNN kernels involved:

• cudnnSetMathMode(CUDNN_DEFAULT_MATH) (no TF32)
• Force CUDNN_DETERMINISTIC via algorithm selection
• Foxhunt may not use cuDNN heavily (most ops are custom CUDA kernels) — verify before applying

§2.D Custom kernel block-tree-reduce audit

Per feedback_no_atomicadd, all reductions use block-tree-reduce. Audit each kernel for:

• ANY use of "last block to finish writes final" patterns — must become "block 0 always writes final" with proper synchronization
• ANY use of cudaDeviceSynchronize inside kernels (rare) — implies non-graph-capturable patterns
• The block_reduce_to_global_with_counter pattern (if used) — non-deterministic by construction

§2.E Adam optimizer iteration order

If adam_m_sum checksum diverges (grads same):

• Audit AdamW step in crates/ml-alpha/src/networks/* for iteration order
• HashMap iteration is non-deterministic in Rust — use BTreeMap or pre-sorted Vec<(key, value)> for parameter groups
• Per-tensor iteration MUST be a fixed order

§2.F PER (Prioritized Experience Replay) sampling

If replay_sample_indices checksum diverges:

• The PER index sampling kernel must use a deterministic RNG (device-side xorshift32 seeded from host per pearl_rl_sample_tau_xorshift32)
• Verify the seed is correctly propagated and not re-randomized between runs
• Check that priority updates and reads are properly synchronized via events (not just stream ordering)

§3. Test infrastructure — make determinism a CONTRACT

§3.1 Local determinism gate

scripts/determinism-check.sh (already created by fast-dev-cycle items 1-5) is the gate. Add:

• Exit 0 if all checksums match to rel-tol 1e-5
• Exit 1 with first-divergent-leaf report otherwise
• Verbose mode shows checksum trajectory side-by-side

§3.2 GPU-oracle determinism invariant test

Add crates/ml-alpha/tests/determinism_invariants.rs:

• Test 1: 200-step mid-smoke twice with seed=42 → all checksums match
• Test 2: 200-step mid-smoke with seed=42 vs seed=43 → checksums DIFFER (sanity: seed actually does something)
• Test 3: 200-step mid-smoke with seed=42 on RTX 3050 → checksum at step 199 matches a frozen golden value (catches regression to non-det)
• All 3 must pass before any cluster smoke can submit (CI gate)

§3.3 Pre-commit hook

.git/hooks/pre-commit (or .claude/hooks/):

• If any .cu or .rs file in crates/ml-alpha/ changed → run 50-step micro-determinism-check
• If divergence → BLOCK commit
• Override with git commit --no-verify (logged + requires justification)

§3.4 Cluster determinism regression test

Once local determinism is fixed:

• One re-run of any cluster smoke with same SHA should produce identical eval_summary.json
• Add to scripts/argo-alpha-rl.sh: optional --regression-pair flag that submits TWO identical workflows and verifies they match

§4. Speed cost — document the dev-vs-prod trade-off

Determinism almost always costs throughput:

• TF32 → FP32: ~10-15% on H100 GEMMs (per NVIDIA docs)
• Mamba2 chunk-1: ~10× slower (if needed; should not be needed if reduction-order fix works)
• cuDNN deterministic: ~5-20% varies
• Checksum kernels: ~150us/step overhead

Acceptable in dev (user explicitly said "we are in dev, determism is most important").

Production toggle via env var FOXHUNT_DETERMINISTIC (default 1 for dev, 0 for production):

let pedantic = std::env::var("FOXHUNT_DETERMINISTIC").unwrap_or_else(|_| "1".to_string()) == "1";
if pedantic {
    cublasSetMathMode(handle, CUBLAS_PEDANTIC_MATH);
} else {
    cublasSetMathMode(handle, CUBLAS_TF32_TENSOR_OP_MATH);
}

Production runs (live trading, walk-forward production) MAY use non-deterministic mode for the 10-15% throughput gain. ALL dev / cluster smokes MUST use deterministic mode.

§5. CI guard

Add to GitLab CI (or whatever CI foxhunt uses — see feedback_argo_workflows_primary):

• Job: determinism-gate
  • Triggers: PR to main, or any branch with ml-alpha-* prefix
  • Runs: scripts/determinism-check.sh on the PR commit
  • Fails the PR if any divergence > rel-tol 1e-5
• Argo workflow template: add pre-train-determinism-check step to alpha-rl-template.yaml that runs determinism-check on the cluster GPU before submitting the actual training run

§6. Falsification criteria

PASS criteria (ALL must hold):

• determinism-check.sh passes (exit 0) for 5 different seeds (42, 43, 44, 45, 46) on RTX 3050
• One b=1024 cluster smoke re-run produces eval_summary.json total_pnl_usd matching to ±$1 (true bit-exact at scalar level)
• determinism_invariants.rs test 1 passes (twin run match)
• determinism_invariants.rs test 3 passes (golden checksum match) — proves no regression

FAIL criteria:

• Any test above fails after applying §2 fixes
• If failure persists after auditing all 6 candidate causes (mamba2, GEMM, cuDNN, custom kernels, Adam, PER) → escalate to NVIDIA support OR accept multi-seed averaging as the workaround (5× cluster cost per verdict, but trustworthy)

§7. Sequencing

Phase 1 ships first (the diagnostic):

1. §1.1 Write rl_deterministic_checksum.cu + register in build.rs (~2 hours)
2. §1.2 Instrument 15 component checksums into build_diag_value (~3 hours)
3. §1.3 Modify determinism-check.sh to compare checksums leaf-by-leaf, report first-divergent (~1 hour)
4. §3.1 Local determinism gate output (~1 hour)
5. Run the diagnostic: identify first-divergent component (~30 min)

Phase 1 total: ~8 hours / 1 day.

Phase 2 fixes depend on Phase 1 results (apply the relevant §2.X). Cost: ~1-3 days depending on which component is the culprit.

Phase 3 validation (§3.2, §3.3, §3.4) + Phase 4 documentation (§4): ~1 day.

Total: ~3-5 days to ship the foundation. This is the prerequisite for multi-head policy, R-multiple reward, and all subsequent regime-invariance work.

§8. Out of scope (defer)

• Multi-GPU determinism (foxhunt is single-GPU per process; defer until production)
• Cross-hardware determinism (RTX 3050 vs L40S vs H100 may produce different exact bits even with all fixes; rel-tol 1e-5 should hold but bit-exact across hardware is not required)
• Cross-CUDA-version determinism (pin CUDA 12.4 per current setup; if upgraded, re-audit)
• Floating-point order across compiler versions (pin rustc + cudart versions in build env)

§9. Why this comes BEFORE multi-head policy

Multi-head policy adds 3× the policy head parameters + a gating network + new aux loss term. ANY of these new components could introduce its own non-determinism (more GEMM calls, more block-tree-reduces, possibly more PER samples).

If we ship multi-head on a non-deterministic foundation:

• Multi-head eval shows +$3M → we cheer, but it's a lucky seed
• Multi-head eval shows -$5M → we revert, but it was an unlucky seed
• Multiple seeds → 5× cluster cost
• We never know if the architecture worked

If we ship determinism FIRST:

• Multi-head eval shows +$3M → reproducible, trust it, build on it
• Multi-head eval shows -$5M → reproducible, abandon it, learn from it
• Single seed per iteration → fast cluster cadence
• Every iteration produces real signal

The 3-5 day investment in determinism pays back on the FIRST controller iteration. Over the next 10 iterations, it saves 50% of cluster compute (no need for multi-seed averaging) and 100% of the "is this real or noise?" confusion that drove today's going-in-circles.

§10. Memory pearls to create after Phase 1 lands

• pearl_determinism_root_cause — documents which component(s) were the actual culprit + fix applied
• pearl_determinism_speed_cost — documents the actual measured throughput cost on H100 + L40S + RTX 3050
• pearl_determinism_contract — the rule going forward: every commit passes determinism gate, no exceptions