foxhunt/docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md

Determinism Foundation Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans to implement this plan task-by-task. Phase 1 is currently DISPATCHED separately (coder agent af76b910). Phases 2-6 execute after Phase 1 lands. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make foxhunt ml-alpha training bit-deterministic (rel-tol 1e-5) across same-seed runs, on a per-step basis, with a permanent CI guard preventing regression. Establish a contract: every dev cluster smoke and every local mid-smoke is reproducible.

Architecture: Localize → Fix → Test → Toggle → Guard. Each phase has explicit acceptance criteria and produces an artifact that the next phase consumes.

Tech stack: CUDA kernels (sm_86/sm_89/sm_90), cudarc + cuBLAS + cuDNN, Rust 1.85, Argo Workflows, foxhunt invariant test harness.

Linked spec: docs/superpowers/specs/2026-06-02-determinism-foundation.md

---

§0 Phase status at plan creation

• Phase 1 (diagnose): DISPATCHED as background coder agent af76b910 — instrumenting 15 checksums + running diagnostic. Output: docs/superpowers/notes/2026-06-02-determinism-phase1-diagnosis.md identifying ONE first-divergent component.
• Phase 2 (fix): this plan covers all 6 candidate causes; ship only the relevant subsection based on Phase 1 output.
• Phase 3 (test infra): independent of Phase 1; can ship in parallel.
• Phase 4 (dev/prod toggle): independent; ships after Phase 2 fix lands.
• Phase 5 (CI guard): independent; ships after Phase 3 test infra.
• Phase 6 (validation): gates merge to main; runs entire suite.

Phase 1 — already dispatched

See coder agent af76b910 output. Once it completes, read docs/superpowers/notes/2026-06-02-determinism-phase1-diagnosis.md for the CULPRIT identification, then jump to the matching §2.X subsection below.

---

Phase 2 — Apply targeted fix based on Phase 1 culprit

§2.A — IF Phase 1 finds: encoder_output diverged first (Mamba2 culprit)

Files:

• Modify: crates/ml-alpha/cuda/mamba2_selective_scan_fwd.cu
• Modify: crates/ml-alpha/cuda/mamba2_selective_scan_bwd.cu (likely same fix applies)
• Possibly: crates/ml-alpha/cuda/mamba2_block.cu
• Test: crates/ml-alpha/tests/mamba2_determinism.rs (NEW)

§2.A.1 — Reproduce mamba2 non-determinism in isolation

• Step 1: Write tests/mamba2_determinism.rs that calls mamba2 forward kernel twice with same input + same weights + same seed → checksum compare
• Step 2: Run it. Verify it reproduces the non-determinism (not just a downstream effect)
• Step 3: If it does NOT reproduce, the bug is downstream of mamba2 — re-read Phase 1 output, the culprit is elsewhere

§2.A.2 — Audit chunk-parallel scan for non-deterministic ordering

• Step 4: Read mamba2_selective_scan_fwd.cu paying attention to:
  • Chunk size constant (CHUNK_SIZE)
  • Block scheduling order
  • Any atomicAdd (should be zero per feedback_no_atomicadd)
  • Any "block 0 OR block 1 writes final" patterns
  • The chunk reduction operator (associativity errors compound)
• Step 5: Document audit findings as comments in the file. If clean → bug is in the scan associativity, not block order.

§2.A.3 — Ground-truth reference with CHUNK_SIZE=1

• Step 6: Add a MAMBA2_DETERMINISTIC compile-time flag that forces CHUNK_SIZE=1 (sequential scan)
• Step 7: With flag ON, run determinism check — should pass (sequential is trivially deterministic)
• Step 8: Measure speed cost of CHUNK_SIZE=1: expect ~10× slower

§2.A.4 — Fix: deterministic chunk-reduction with fixed accumulation order

• Step 9: In the chunk reduction operator, ensure accumulation order is FIXED (not parallel-reduce with non-deterministic synchronization):

  // BAD (non-det if multiple threads accumulate):
  __shared__ float acc;
  if (threadIdx.x == 0) acc = 0.0f;
  __syncthreads();
  atomicAdd(&acc, my_partial);  // ← non-det order
  
  // GOOD (sequential within block, fixed across blocks):
  __shared__ float partials[WARPS_PER_BLOCK];
  partials[warp_id] = warp_reduce(my_partial);
  __syncthreads();
  if (warp_id == 0) {
    float total = 0.0f;
    for (int i = 0; i < WARPS_PER_BLOCK; ++i) total += partials[i];
    // block 0 always writes final
  }
  

• Step 10: Apply the fix
• Step 11: Re-run determinism check — should pass with full chunk parallelism
• Step 12: Measure speed cost: expect ~1-3% (negligible)

§2.A.5 — Validation

• Step 13: cargo test mamba2_determinism --release --ignored passes
• Step 14: Full scripts/determinism-check.sh passes (no checksum divergence)
• Step 15: Commit: fix(mamba2): deterministic chunk-reduction accumulation order

---

§2.B — IF Phase 1 finds: q_logits / pi_logits / v_value diverged first (cuBLAS split-K culprit)

Files:

• Modify: crates/ml-alpha/src/networks/q_head.rs (cuBLAS handle init)
• Modify: crates/ml-alpha/src/networks/policy_head.rs
• Modify: crates/ml-alpha/src/networks/v_head.rs
• Modify: crates/ml-alpha/src/networks/dqn.rs (if cuBLAS handle is shared)
• Modify: crates/ml-alpha/src/networks/mamba2_block.rs (likely same)
• Test: crates/ml-alpha/tests/cublas_determinism.rs (NEW)

§2.B.1 — Confirm TF32 enabled is the cause

• Step 1: Tasks 30/31 (completed) enabled TF32 on cuBLAS handles. TF32 uses split-K with non-deterministic K-split heuristic.
• Step 2: Audit each cuBLAS handle init site (grep for cublasSetMathMode)
• Step 3: Document current math modes:

  q_head: CUBLAS_TF32_TENSOR_OP_MATH ← tasks 30/31
  pi_head: CUBLAS_TF32_TENSOR_OP_MATH ← tasks 30/31
  mamba2_block: CUBLAS_TF32_TENSOR_OP_MATH ← tasks 30/31
  

§2.B.2 — Add deterministic mode toggle

• Step 4: Add helper in crates/ml-alpha/src/cublas_init.rs (NEW or in existing setup file):

  pub fn cublas_set_math_mode_deterministic(handle: cublasHandle_t) -> Result<()> {
      let pedantic = std::env::var("FOXHUNT_DETERMINISTIC")
          .unwrap_or_else(|_| "1".to_string()) == "1";
      let mode = if pedantic {
          CUBLAS_PEDANTIC_MATH  // disables TF32 split-K, FP32 strict
      } else {
          CUBLAS_TF32_TENSOR_OP_MATH  // current default after tasks 30/31
      };
      unsafe { cublasSetMathMode(handle, mode); }
      Ok(())
  }
  

• Step 5: Replace every cublasSetMathMode(handle, CUBLAS_TF32_*) call with cublas_set_math_mode_deterministic(handle)
• Step 6: Verify with grep: no remaining CUBLAS_TF32_TENSOR_OP_MATH direct calls

§2.B.3 — Validation

• Step 7: Build release. Set FOXHUNT_DETERMINISTIC=1 and run determinism-check → should pass
• Step 8: Set FOXHUNT_DETERMINISTIC=0 and re-run → should fail (sanity: the toggle works)
• Step 9: Measure throughput cost on RTX 3050 mid-smoke: time scripts/local-mid-smoke.sh with toggle ON vs OFF. Expect 5-15% slower in deterministic mode.
• Step 10: Document the cost in docs/superpowers/notes/2026-06-02-determinism-speed-cost.md

§2.B.4 — Commit

• Step 11: fix(cublas): FOXHUNT_DETERMINISTIC env var toggles PEDANTIC vs TF32 math mode

---

§2.C — IF Phase 1 finds: cuDNN involved (less likely but possible)

Files:

• Same network files as §2.B (if they call cuDNN)
• Modify: crates/ml-alpha/src/cudnn_init.rs (if exists; or relevant module)

§2.C.1 — Check whether ml-alpha uses cuDNN at all

• Step 1: grep -rn "cudnn\|CUDNN" crates/ml-alpha/src/ crates/ml-alpha/cuda/ to confirm cuDNN usage
• Step 2: If zero hits → cuDNN not in path → skip §2.C entirely
• Step 3: If cuDNN is used: apply CUDNN_DEFAULT_MATH + CUDNN_DETERMINISTIC flags via similar toggle to §2.B.2

---

§2.D — IF Phase 1 finds: custom kernel grad/forward output diverged (block-tree audit)

Files:

• Audit: all .cu files in crates/ml-alpha/cuda/
• Most likely candidates: rl_*_backward.cu kernels, *_reduce.cu kernels

§2.D.1 — Identify the specific kernel(s)

• Step 1: Phase 1 narrows to e.g. q_grad_checksum diverges first. The Q gradient is computed by dqn_distributional_q_backward.cu (or similar). Identify the exact kernel.
• Step 2: Read the kernel for non-deterministic patterns:
  • Any atomicAdd (should be 0 per feedback_no_atomicadd)
  • Any "if (block_id == 0)" final-write pattern that depends on block scheduling
  • Any volatile memory access without proper fence
  • Any cudaDeviceSynchronize inside kernel (very rare)

§2.D.2 — Common fix patterns

• Step 3: If the kernel uses "first block to finish writes final" pattern, replace with:
  • Two-pass: first pass each block writes its partial to global memory, second pass single-block reads all partials in fixed order
  • OR: cooperative groups grid-wide sync (requires sm_70+, available on all foxhunt GPUs)
• Step 4: If the kernel reads stale values due to write/read race: add __threadfence_system() + counter pattern with explicit ordering
• Step 5: Apply the fix
• Step 6: Re-run determinism-check

§2.D.3 — Validation

• Step 7: GPU oracle test for the specific kernel passes bit-equivalence
• Step 8: Determinism-check passes
• Step 9: Commit: fix(rl): deterministic block-reduction in <kernel-name>

---

§2.E — IF Phase 1 finds: adam_m_sum / adam_v_sum diverged (optimizer iteration order)

Files:

• Modify: crates/ml-alpha/src/optimizers/adamw.rs (or wherever Adam state is iterated)

§2.E.1 — Audit parameter iteration order

• Step 1: Find the AdamW step function. Look for parameter iteration via HashMap, Vec, or BTreeMap.
• Step 2: If HashMap is used (e.g., HashMap<String, CudaSlice<f32>>), iteration order is non-deterministic in Rust by design.
• Step 3: Replace HashMap with BTreeMap OR pre-sorted Vec<(String, CudaSlice<f32>)> at construction time. Sort by parameter name.

§2.E.2 — Validation

• Step 4: Add unit test in crates/ml-alpha/tests/adam_determinism.rs: build trainer, do 10 steps, dump Adam m + v, repeat, compare — should be bit-equal
• Step 5: Determinism-check passes
• Step 6: Commit: fix(adam): deterministic parameter iteration order via BTreeMap

---

§2.F — IF Phase 1 finds: replay_sample_indices diverged (PER sampling)

Files:

• Modify: crates/ml-alpha/cuda/rl_per_sample.cu (or wherever PER sampling happens)
• Modify: crates/ml-alpha/src/replay/per.rs

§2.F.1 — Check PER RNG seeding

• Step 1: Locate the PER sample kernel. Should use device-side xorshift32 seeded from host (per pearl_rl_sample_tau_xorshift32)
• Step 2: Verify the seed is passed via ISV or kernel arg, deterministically advanced per step
• Step 3: If the seed source is std::time::SystemTime::now() or thread_rng() anywhere → that's the bug

§2.F.2 — Fix

• Step 4: Ensure PER seed derives from a deterministic counter (e.g., cli.seed + step_counter)
• Step 5: Verify priority update / read ordering uses CudaEvent, not stream-implicit ordering
• Step 6: Re-run determinism-check

§2.F.3 — Commit

• Step 7: fix(per): deterministic sample-RNG seeded from cli.seed + step

---

Phase 3 — Test infrastructure (parallel to Phase 2)

§3.1 — tests/determinism_invariants.rs

Files:

• Create: crates/ml-alpha/tests/determinism_invariants.rs

• Task 1: Write the test file with 3 tests:

//! GPU-oracle determinism invariants.
//!
//! Test 1: same seed → identical checksums (rel-tol 1e-5)
//! Test 2: different seed → different checksums (sanity check)
//! Test 3: same seed + golden checksum file → match (catches regression)

use anyhow::{Context, Result};
use serde_json::Value;
use std::path::PathBuf;
use std::process::Command;

fn binary_path() -> PathBuf { /* ... */ }
fn data_dir() -> PathBuf { /* ... */ }
fn read_jsonl(path: &Path) -> Result<Vec<Value>> { /* ... */ }
fn extract_checksums(row: &Value) -> Result<Vec<(String, f64)>> { /* ... */ }

#[test]
#[ignore = "requires CUDA + pre-built release binary + MBP-10 test data"]
fn determinism_twin_runs_match() -> Result<()> {
    let out_a = run_mid_smoke(seed=42)?;
    let out_b = run_mid_smoke(seed=42)?;
    let rows_a = read_jsonl(&out_a.join("diag.jsonl"))?;
    let rows_b = read_jsonl(&out_b.join("diag.jsonl"))?;
    for (i, (a, b)) in rows_a.iter().zip(rows_b.iter()).enumerate() {
        let ca = extract_checksums(a)?;
        let cb = extract_checksums(b)?;
        for ((ka, va), (kb, vb)) in ca.iter().zip(cb.iter()) {
            assert_eq!(ka, kb, "checksum key mismatch at row {i}: {ka} vs {kb}");
            let rel = ((va - vb) / va.abs().max(1e-12)).abs();
            anyhow::ensure!(rel < 1e-5,
                "step {i} {ka}: a={va:.6} b={vb:.6} rel_err={rel:.2e}");
        }
    }
    Ok(())
}

#[test]
#[ignore = "requires CUDA + pre-built release binary + MBP-10 test data"]
fn determinism_seed_actually_matters() -> Result<()> {
    let out_a = run_mid_smoke(seed=42)?;
    let out_b = run_mid_smoke(seed=43)?;
    let rows_a = read_jsonl(&out_a.join("diag.jsonl"))?;
    let rows_b = read_jsonl(&out_b.join("diag.jsonl"))?;
    // After 100 steps, at least one checksum should differ
    let row_100_a = &rows_a[99];
    let row_100_b = &rows_b[99];
    let ca = extract_checksums(row_100_a)?;
    let cb = extract_checksums(row_100_b)?;
    let any_differs = ca.iter().zip(cb.iter()).any(|((_, va), (_, vb))| {
        ((va - vb) / va.abs().max(1e-12)).abs() > 1e-3
    });
    anyhow::ensure!(any_differs, "different seeds produced identical results — seed not used");
    Ok(())
}

#[test]
#[ignore = "requires CUDA + pre-built release binary + MBP-10 test data + golden checksum file"]
fn determinism_golden_checksum_match() -> Result<()> {
    let out = run_mid_smoke(seed=42)?;
    let rows = read_jsonl(&out.join("diag.jsonl"))?;
    let last = &rows[rows.len() - 1];
    let observed = extract_checksums(last)?;
    let golden = read_golden("crates/ml-alpha/tests/fixtures/determinism-golden-seed42.json")?;
    for ((k, v_obs), (k_g, v_g)) in observed.iter().zip(golden.iter()) {
        assert_eq!(k, k_g);
        let rel = ((v_obs - v_g) / v_g.abs().max(1e-12)).abs();
        anyhow::ensure!(rel < 1e-5,
            "checksum {k} drift from golden: obs={v_obs:.6} gold={v_g:.6} rel_err={rel:.2e}");
    }
    Ok(())
}

fn run_mid_smoke(seed: u32) -> Result<PathBuf> { /* invoke local-mid-smoke.sh with seed */ }
fn read_golden(path: &str) -> Result<Vec<(String, f64)>> { /* parse JSON */ }

• Task 2: Generate golden checksum file crates/ml-alpha/tests/fixtures/determinism-golden-seed42.json:

  • After Phase 2 fix lands and determinism check passes
  • Run mid-smoke with seed=42, extract final checksums, save as golden
  • This freezes the post-fix behavior — any future regression detected

• Task 3: Verify tests pass on a clean checkout

---

§3.2 — Pre-commit hook

Files:

• Create: .claude/hooks/determinism-check.sh OR .git/hooks/pre-commit (foxhunt's convention)

• Task 4: Write a fast 50-step micro-determinism-check that runs on pre-commit:

  • Detects .cu or crates/ml-alpha/src/**/*.rs changes via git diff --cached --name-only
  • If no relevant changes → exit 0 immediately (no overhead)
  • Otherwise: run 50-step mid-smoke twice, compare checksums
  • Exit 1 + clear error message if determinism broke

• Task 5: Document the hook in CLAUDE.md

• Task 6: Add git commit --no-verify escape hatch documentation (must justify in commit message)

---

§3.3 — Cluster regression-pair workflow

Files:

• Modify: scripts/argo-alpha-rl.sh

• Modify: infra/k8s/argo/alpha-rl-template.yaml

• Task 7: Add --regression-pair flag to scripts/argo-alpha-rl.sh:

  • When set, submits TWO identical Argo workflows (same SHA, same seed, same params)
  • After both complete, compares eval_summary.json from both
  • Reports total_pnl_usd Δ in absolute and relative terms
  • Logs warning if Δ > 0.1% (true determinism = bit-exact)

• Task 8: Manual test: run argo-alpha-rl.sh --regression-pair --sha <post-fix-sha> → both runs should produce identical eval_summary

---

Phase 4 — Dev/prod toggle

Files:

• Modify: crates/ml-alpha/src/lib.rs or crates/ml-alpha/src/config.rs

• Modify: CLAUDE.md

• Task 1: Add global config struct:

pub struct DeterminismConfig {
    pub enabled: bool,
}
impl DeterminismConfig {
    pub fn from_env() -> Self {
        let enabled = std::env::var("FOXHUNT_DETERMINISTIC")
            .unwrap_or_else(|_| "1".to_string())  // default ON in dev
            == "1";
        Self { enabled }
    }
}

• Task 2: Plumb through to: cuBLAS init (§2.B.2), checksum kernel (skip if disabled to save 150us/step), CHUNK_SIZE selection if applicable (§2.A.6)

• Task 3: Production runs (live trading, walk-forward production) set FOXHUNT_DETERMINISTIC=0 for speed. Document in CLAUDE.md.

• Task 4: Add startup log: print determinism mode + expected speed cost so it's obvious in run logs which mode was active

---

Phase 5 — CI guard

Files:

• Modify: .gitlab-ci.yml (per CLAUDE.md, legacy CI exists but disabled; if enabled now use it; otherwise add Argo CI)

• Possibly create: infra/k8s/argo/determinism-gate-template.yaml

• Task 1: Add CI job determinism-gate:

  • Triggers: any push to a branch with ml-alpha-* prefix, or any PR to main
  • Runs: full mid-smoke (b=128, 2000+500) twice with FOXHUNT_DETERMINISTIC=1, then determinism-check.sh
  • Fails: if any checksum divergence > rel-tol 1e-5
  • Reports: which leaf diverged, at what step

• Task 2: Add pre-train determinism check to infra/k8s/argo/alpha-rl-template.yaml:

  • New step before main training: run 50-step micro-determinism-check on cluster
  • If divergence detected → fail the workflow with clear message
  • Cost: ~30 seconds per cluster smoke; saves N×70min if it catches a regression

---

Phase 6 — Full validation

• Task 1: All 3 tests in determinism_invariants.rs pass
• Task 2: scripts/determinism-check.sh passes for 5 different seeds: 42, 43, 44, 45, 46
• Task 3: scripts/argo-alpha-rl.sh --regression-pair produces identical eval_summary.json (Δ < $1) on one full b=1024 20k+5k cluster smoke
• Task 4: Speed cost documented: measured time of local-mid-smoke.sh with FOXHUNT_DETERMINISTIC=1 vs =0 on RTX 3050; same comparison on cluster L40S
• Task 5: Create memory pearls:
  • pearl_determinism_root_cause: the actual culprit + fix
  • pearl_determinism_speed_cost: measured throughput cost
  • pearl_determinism_contract: the rule for future commits
• Task 6: Update MEMORY.md index with the 3 new pearls
• Task 7: Atomic commit: feat(rl): deterministic training foundation + CI gate

---

Phase 7 — Multi-head policy unblock

Once Phase 6 lands, the multi-head policy spec (2026-06-02-multi-head-policy-with-r-multiple.md) is unblocked. Implementation can proceed with confidence that:

• Same-seed runs produce identical results
• Eval pnl differences between iterations are SIGNAL not noise
• Single-seed verdicts are trustworthy (no need for 5-seed averaging)

---

Failure modes and replanning

Failure mode 1: Phase 1 finds multiple components diverge simultaneously

Cause: A single upstream bug (e.g., mamba2) cascades to all downstream components. Response: Fix the upstream (likely mamba2 per §2.A). The downstream should fix itself once upstream is deterministic.

Failure mode 2: Phase 2 fix doesn't restore determinism

Cause: We fixed one bug but there are multiple sources. Response: Re-run Phase 1 with the partial fix applied. Identify the NEXT first-divergent component. Apply the matching §2.X. Iterate.

Failure mode 3: Determinism in dev mode but not in prod (FOXHUNT_DETERMINISTIC=0)

Cause: Expected — TF32 split-K is non-deterministic by design. Response: This is acceptable. Dev (where verdict matters) is deterministic; prod (where speed matters) is not. Document this.

Failure mode 4: Speed cost is too high (>20% slowdown)

Cause: CHUNK_SIZE=1 mamba2 path OR aggressive cuDNN deterministic mode. Response: Audit the specific bottleneck. Often a deterministic-by-construction reduction can be added without falling back to fully sequential. If unavoidable, accept the cost — verdict trust > 20% throughput.

Failure mode 5: Phase 1 finds bug is in third-party code (cudarc / cuBLAS internals)

Cause: NVIDIA / cudarc library determinism issue. Response: Pin the cudarc version + CUDA toolkit version. File a bug upstream. Use multi-seed averaging as workaround until library fix lands.

---

Discipline reminders (HARD)

• feedback_no_atomicadd — fixes must use block-tree-reduce only, never atomicAdd
• feedback_no_nvrtc — all kernels pre-compiled via build.rs
• feedback_cpu_is_read_only — checksums computed on GPU, result read on CPU only
• feedback_no_stubs — every checksum, every fix, every test must run end-to-end
• feedback_local_smoke_before_cluster — Phase 6 task 3 (cluster regression-pair) ONLY after Phase 6 task 1+2 pass locally
• feedback_no_partial_refactor — when modifying a cuBLAS handle init, modify ALL of them in the same commit (no half-toggle state)
• feedback_systematic_debugging — find root cause before fixing; don't patch symptoms
• feedback_going_in_circles_pattern — if Phase 2 fixes don't converge after 3 attempts → re-audit Phase 1 diagnostic

Acceptance criteria for the whole plan

The plan is COMPLETE when:

1. scripts/determinism-check.sh exit 0 on 5 different seeds locally
2. Cluster --regression-pair produces eval_summary Δ < $1
3. CI determinism-gate is wired and passing
4. 3 memory pearls saved + MEMORY.md indexed
5. Speed cost documented
6. Atomic commit landed on ml-alpha-regime-observer or main
7. Multi-head policy spec confirmed unblocked

Estimated total time across all phases: 3-5 working days (heavily depending on which §2.X applies — mamba2 fix is largest, cuBLAS toggle is smallest).