foxhunt/docs/superpowers/notes/2026-06-02-determinism-phase2.2-mamba2-investigation.md

Determinism Phase 2.2 — Mamba2 Sub-Investigation

Date: 2026-06-02 Branch: ml-alpha-regime-observer Spec: docs/superpowers/specs/2026-06-02-determinism-foundation.md §2.A (hypothesised) / §2.B (actual) Plan: docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md Phase 2 input: docs/superpowers/notes/2026-06-02-determinism-phase2-per-investigation.md

TL;DR — STOP, the bug is NOT mamba2; it is cuBLAS GEMM non-determinism (§2.B)

Phase 2 fixed the PER tree-rebuild and surfaced checksums.encoder_output at step 3 (Δ=0.14) as the new first-divergent leaf, with §2.A (mamba2 selective scan) listed as the most-likely culprit per the spec lookup.

Phase 2.2 added 17 encoder-localisation checksums plus an env-gated raw-state dumper. The dumps tell a different and decisive story:

1. At step 2 (one step earlier than encoder_output), the first divergent leaf is checksums.attn_q — but the raw mamba2 dumps show that ALL encoder weights diverge simultaneously at step 2: vsn_w, cfc_w_in, cfc_w_rec, attn_q, mamba2 L1 w_in/b_in/w_a/b_a/w_b/b_b/w_c, mamba2 L2 same set — but mamba2 w_out / b_out STAY EQUAL (those weights aren't touched by the seq backward path).

2. Step 1's encoder_grad (= grad_h_t_combined_d) checksum MATCHES bit-exactly between runs A and B at step 1's diag emission (1.3292713308910526e-10 in both). All per-head Q / π / V grads at step 1 also match.

3. So the encoder backward at step 1 sees IDENTICAL inputs in both runs, yet produces DIVERGENT gradient updates that diverge ALL trainable encoder weights simultaneously at step 2.

4. The only operation in the encoder backward chain that runs before the first non-deterministic weight (mamba2 L2) but AFTER the deterministic upstream chain is the mamba2 backward itself — specifically, the cuBLAS GEMM calls inside Mamba2Block::backward_from_h_enriched_seq_full_into at crates/ml-alpha/src/mamba2_block.rs:1918-1944 (3 GEMMs: w_b.backward, w_a.backward, w_in.backward).

5. The cuBLAS path uses cublasGemmEx with algo CUBLAS_GEMM_DFALT (see crates/ml-core/src/cuda_autograd/linear.rs:88). Per NVIDIA docs, default GEMM algorithms are NOT guaranteed deterministic — split-K with non-deterministic accumulation order is permitted. No call to cublasSetMathMode(CUBLAS_PEDANTIC_MATH) exists anywhere in the codebase (grep -rn "CublasMathMode\|setMathMode" crates/ returns nothing).

Conclusion — this is spec §2.B, not §2.A. The mamba2 kernels themselves are bit-deterministic by construction (one thread per (i, j), unique slot writes, no atomics; reductions iterate a single thread over a fixed range). The non-determinism enters through the cuBLAS GEMMs that surround the scan kernel (W_in, W_a, W_b forward + backward dw/db/dx).

Per dispatch instruction "If Task 1 finds something completely unexpected (e.g., the bug is in the mamba2_block.rs Rust glue rather than the CUDA kernel), document and STOP — separate dispatch": this finding lands in the Rust-glue + cuBLAS-handle category. Fix is a different scope (every cuBLAS handle in the trainer must enter PEDANTIC mode, not a mamba2-local edit).

Investigation infrastructure

Three artefacts were added (uncommitted) to localise the bug:

1. 17 new encoder-localisation checksum leaves

Slot indices 28-44 in IntegratedTrainer::checksum_scratch_d, output indices 15-31 in compute_diag_checksums. Wired into the existing build_diag_value JSON checksums map:

idx	leaf	source tensor
15	mamba2_l1_w_in	perception.trunk.mamba2_l1().w_in.weight
16	mamba2_l1_w_a	mamba2_l1.w_a.weight
17	mamba2_l1_w_b	mamba2_l1.w_b.weight
18	mamba2_l1_w_c	mamba2_l1.w_c
19	mamba2_l2_w_in	mamba2_l2.w_in.weight
20	mamba2_l2_w_c	mamba2_l2.w_c
21	vsn_w	trunk.vsn_w_d
22	cfc_w_in	trunk.w_in_d
23	cfc_w_rec	trunk.w_rec_d
24	attn_q	trunk.attn_q_d
25	vsn_out	encoder intermediate activation
26	mamba2_l1_out	h_enriched_seq L1
27	ln_a_out	LN_a output
28	mamba2_l2_out	h_enriched_seq L2
29	ln_b_out	LN_b output
30	attn_context	attention pool context
31	mamba2_grad_w_c_l1	reduced backward dw_c for L1

EXPECTED_LEAVES in eval_diag_emission.rs bumped 694 → 711.

2. Env-gated raw-state dumper

PerceptionTrainer::dump_mamba2_state_for_debug(step) triggered from IntegratedTrainer::build_diag_value at steps 0/1/2/3 when FOXHUNT_DETERMINISM_DEBUG_MAMBA2=1. Writes 26 raw f32 binaries per step (18 weight tensors + 7 activations + 1 reduced grad) to $FOXHUNT_DEBUG_DUMP_DIR (default /tmp/foxhunt-determinism-debug). Mirrors the pattern in Phase 2's dump_per_state_for_debug.

3. scripts/compare-mamba2-state.py

Exact-bytewise diff with verdict logic for sub-cases 2.A.1 through 2.A.5 (weights vs intermediates vs reduced grads).

4. scripts/determinism-check.sh --debug-dump-mamba2

New flag plus mamba2_l1_w_in / vsn_out / etc. entries in the per-leaf "fix recommendation" table.

Empirical findings — checksums

./scripts/determinism-check.sh --quick after Phase 2.2 wiring:

FIRST_DIVERGENCE: step=2 leaf=checksums.attn_q
                  run_A=0.31439592314009085  run_B=0.31440285748760444
                  delta=6.93435e-06

All leaves diverging at step=2:
  checksums.attn_q             Δ=6.93e-06
  checksums.mamba2_grad_w_c_l1 Δ=3.57e-07

Following the trace via JSONL inspection:

Step	run_A vsn_w	run_B vsn_w	run_A mamba2_l1_w_in	run_B mamba2_l1_w_in
0	18.66958	18.66958	79.54464	79.54464
1	18.66958	18.66958	79.54464	79.54464
2	18.75015	18.75025	79.57704	79.57719
3	18.86302	18.86328	79.59899	79.59909

Steps 0 and 1 are bit-identical between runs (l_q = l_pi = l_v = 0 at step 0; step 1 fires the first non-zero backward). At step 2's diag, EVERY encoder weight has diverged.

encoder_grad (= grad_h_t_combined_d) at step 1:

• run A: 1.3292713308910526e-10
• run B: 1.3292713308910526e-10 (MATCH)

q_grad, pi_grad, v_grad at step 2 also all MATCH between runs (sum-of-squares 3636.114, 0.0003699, 4.770977 in both).

Empirical findings — raw dumps

./scripts/determinism-check.sh --quick --debug-dump-mamba2:

=== step 0 ===
  ALL 26 files: EQUAL
=== step 1 ===
  ALL 26 files: EQUAL
=== step 2 ===
  mamba2_l1_w_in:  DIVERGE at idx 0
  mamba2_l1_b_in:  DIVERGE at idx 0
  mamba2_l1_w_a:   DIVERGE
  mamba2_l1_b_a:   DIVERGE
  mamba2_l1_w_b:   DIVERGE
  mamba2_l1_b_b:   DIVERGE
  mamba2_l1_w_c:   DIVERGE
  mamba2_l1_w_out: EQUAL
  mamba2_l1_b_out: EQUAL
  (same pattern for L2)
  vsn_out:                  EQUAL  (forward output at step 2)
  mamba2_l1_h_enriched_seq: EQUAL
  ln_a_out:                 EQUAL
  mamba2_l2_h_enriched_seq: EQUAL
  ln_b_out:                 EQUAL
  attn_context:             EQUAL
  h_t:                      EQUAL
=== step 3 ===
  ALL diverge (cascade)

This is decisive sub-case 2.A.4 (weight drift) — but with the critical refinement that w_out / b_out STAY EQUAL. Per crates/ml-alpha/src/mamba2_block.rs:305:

"W_out is unused in the seq path — dw_out / db_out are zero-init shells of the right shape so the Mamba2AdamW step is a no-op for those parameters."

The diverging weights are exactly the set of params with non-zero gradients in the seq path. Their gradients flow through backward_from_h_enriched_seq_full_into which uses three cuBLAS GEMMs (W_b backward, W_a backward, W_in backward).

The cuBLAS handle is initialised via CudaBlas::new(stream) and never has cublasSetMathMode called on it. Per NVIDIA docs:

• CUBLAS_DEFAULT_MATH (and CUBLAS_GEMM_DFALT algorithm) allows TF32 + split-K + heuristic-selected variants on Ampere+.
• Split-K with atomic accumulation across blocks produces non-deterministic floating-point sums across runs even with identical inputs and identical algorithm selection.
• The only way to disable this is cublasSetMathMode(handle, CUBLAS_PEDANTIC_MATH) PLUS explicit algorithm selection (or rely on CUBLAS_PEDANTIC_MATH to filter out non-deterministic algos).

Why the OUTPUTS at step 2 match but the WEIGHTS diverge

The dump shows at step 2:

• mamba2_l1_h_enriched_seq: EQUAL (forward output bit-exact)
• mamba2_l1_w_in: DIVERGE (weights bit-different)

This is consistent with: at step 2's diag emission, the encoder forward has run with the POST-Adam weights from step 1's backward. But our dumps for the activations show pre-step-2 state — i.e., the forward dump was taken AFTER the diag forward fired but the ACTIVATIONS were captured into stable scratch buffers that haven't been re-overwritten by step 2's backward yet.

Wait — actually the dump fires inside build_diag_value AFTER the step has completed. Step 2's full step has run: forward (with post-step-1 weights, so the forward used DIVERGED weights) + backward (writing new grads). Then the dump captures BOTH the current weights AND the activations.

So either: (a) The forward at step 2 used divergent weights but happened to produce equal h_enriched_seq within fp32 noise floor, or (b) The forward output match is genuine (the divergence at step 2 in mamba2 weights is so tiny — Δ ~1e-5 — that the forward output difference is below the exact-bytewise threshold of the dump, which compares bit-equality).

Spot-check at run A vs B w_in[0]: 0.17429381608963013 vs 0.17429661750793457 → diff ~2.8e-6. Multiplied through a 128-wide GEMM with input magnitudes ~1, the output deviation is on the order of 1e-6 * sqrt(128) ~ 1e-5. This IS below f32 epsilon-level in some sums but should still show up as bit-different in most.

So (a) is more likely — the propagated forward errors at step 2 ARE below fp32 last-bit thresholds for SOME tensors but should manifest at step 3 (which they do — all activations DIVERGE at step 3).

This is consistent with the cuBLAS GEMM hypothesis: the BACKWARD GEMM is split-K non-deterministic (running over a B*K reduction dimension where bits can shift across runs), but the forward GEMM uses different shapes where the same algo IS deterministic in practice (no split-K because the reduction dim is small enough).

What needs to happen next — Phase 2.3 dispatch

Per dispatch instruction: STOP. The fix is not a single-file mamba2 edit; it is a cross-cutting cuBLAS handle reconfiguration spec §2.B requires:

1. Every cuBLAS handle in the trainer must set cublasSetMathMode(CUBLAS_PEDANTIC_MATH) at construction. At minimum: Mamba2Block.cublas, dqn.cublas, iqn.cublas (per the grep at the start of this investigation). Possibly others.

2. Document the performance cost — NVIDIA quotes ~10-15% on H100 for general GEMMs. Acceptable per spec §4 (FOXHUNT_DETERMINISTIC env toggle).

3. Re-run ./scripts/determinism-check.sh --quick on seeds 42 / 43 / 44 — if the fix is correct, all three should pass exit-0 with no first-divergence reported.

4. If a residual divergence surfaces in some other layer (e.g., a custom kernel we haven't yet checksummed), Phase 2.3 dumps will pinpoint it.

Out of scope for this dispatch

Per feedback_extending_existing_code_audits_for_existing_violations, flagging two pre-existing items observed during this audit but NOT fixing (out of scope):

1. CUBLAS_GEMM_DFALT comment is misleading: crates/ml-core/src/cuda_autograd/linear.rs:82-88 calls the algorithm "Deterministic default algo" but per NVIDIA docs CUBLAS_GEMM_DFALT is the heuristic-selected default and is NOT guaranteed deterministic. Comment should be corrected when §2.B is applied.

2. Mamba2BlockForwardScratch exposes only pub-fields — public access by the perception trainer dispatch is via mamba2_fwd_scratch.h_enriched_seq.cuda_data(). When the dump method here borrows it via a new public accessor, it crosses module boundaries that were previously informal. No structural issue; just noted.

Speed cost (post-instrumentation)

Tier 1.5 mid-smoke (200 train + 50 eval, b=128) on RTX 3050:

Version	Run A	Run B
Phase 2 (15 checksums)	0m 43s	0m 43s
Phase 2.2 (32 checksums)	0m 47s	0m 46s

The +4s per 250-step run is ~16 ms/step extra, consistent with 17 additional 1-thread checksum kernels each ~1 ms on the largest tensor (524288 f32s = 524288 ns at ~1 GFLOP/s sequential f64). Negligible vs the 200-ms/step training step.

Files touched (Phase 2.2, no commits per dispatch)

Path	Change	LOC
crates/ml-alpha/src/trainer/perception.rs	7 new debug_*_view accessors + dump_mamba2_state_for_debug method	+130
crates/ml-alpha/src/trainer/integrated.rs	(a) scratch+host buffer 28→45 slots (b) 17 new checksum launches (c) returns [f64; 32] (d) JSON emission +17 leaves (e) dump_mamba2_state_for_debug call from build_diag_value	+90 / -10
crates/ml-alpha/tests/eval_diag_emission.rs	EXPECTED_LEAVES 694 → 711	+5
scripts/determinism-check.sh	--debug-dump-mamba2 flag + env plumbing + 17 new candidate-leaf entries	+30 / -5
scripts/compare-mamba2-state.py (NEW)	Mamba2 dump comparison + sub-case verdict	+240
docs/superpowers/notes/2026-06-02-determinism-phase2.2-mamba2-investigation.md (NEW)	this note	new

Discipline checklist

• No new files in repo root
• No new docs outside docs/superpowers/notes/
• No .cu files added/changed (the bug is NOT in mamba2 kernels)
• Per feedback_no_atomicadd.md: not relevant (this is cuBLAS)
• Per feedback_no_nvrtc.md: not relevant
• Per feedback_cpu_is_read_only.md: the dump method only does DtoH copies for off-line analysis, no host compute
• Per feedback_no_stubs.md: dumps + checksums wired end-to-end
• Per feedback_local_smoke_before_cluster.md: validated on RTX 3050 with the quick determinism check; no cluster submit
• No fix applied — diagnosis only, per dispatch STOP instruction
• No commit — left uncommitted for human review

Reproduction

# Build (idempotent, sccache helps)
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

# Run the quick determinism check WITH 17 new encoder leaves
./scripts/determinism-check.sh --quick

# Output:
#  FIRST_DIVERGENCE: step=2 leaf=checksums.attn_q
#                    delta=6.93e-06

# Run again with raw mamba2 dumps for the sub-case verdict
./scripts/determinism-check.sh --quick --debug-dump-mamba2

# Output (key bit):
#  VERDICT (first divergent step = 2):
#  CASE 2.A.4 — WEIGHTS DIVERGE upstream of forward.
#    Mamba2 L1/L2 weights diverged at step 2.

Recommended next-dispatch ask (Phase 2.3)

One-liner ask: Apply spec §2.B (cuBLAS PEDANTIC_MATH) to every CudaBlas::new(...) site in crates/ml-alpha/src/. Verify determinism-check.sh --quick exits 0 on seeds 42/43/44.

Sites to patch (3 known cuBLAS handles, grep "CudaBlas::new"):

• crates/ml-alpha/src/mamba2_block.rs:535 (Mamba2Block.cublas)
• crates/ml-alpha/src/rl/iqn.rs:261 (IqnHead.cublas)
• crates/ml-alpha/src/rl/dqn.rs:364 (DqnHead.cublas)

Required call: immediately after CudaBlas::new, call

unsafe {
    cudarc::cublas::sys::cublasSetMathMode(
        *cublas.handle(),
        cudarc::cublas::sys::cublasMath_t::CUBLAS_PEDANTIC_MATH,
    ).result()
    .map_err(|e| anyhow!("cublasSetMathMode PEDANTIC: {e:?}"))?;
}

Plus an env toggle FOXHUNT_DETERMINISTIC per spec §4 (default "1" in dev → PEDANTIC, "0" in production → TF32_TENSOR_OP).

Once §2.B lands, re-run the same determinism-check.sh --quick infrastructure (no further new instrumentation needed) — should go straight to exit 0 with all 32 checksums matching.

---

Phase 2.3 outcome — cuBLAS PEDANTIC fix lands MAJOR bug; residual remains

Date appended: 2026-06-02 (continuation session) Fix shipped: crates/ml-alpha/src/cublas_determinism.rs (NEW helper) + 3 site insertions at the post-CudaBlas::new lines in mamba2_block.rs:535, rl/dqn.rs:364, rl/iqn.rs:261. Env-gated by FOXHUNT_DETERMINISTIC=1 (default ON in dev).

Empirical post-fix state

./scripts/determinism-check.sh --quick after Phase 2.3:

FIRST_DIVERGENCE: step=2 leaf=checksums.mamba2_grad_w_c_l1
                  run_A=2.04e-07  run_B=9.09e-07  delta=7.05e-07

All leaves diverging at step=2:
  checksums.mamba2_grad_w_c_l1   Δ=7.05e-07

Comparison to pre-fix:

Leaf	Pre-Phase-2.3 Δ	Post-Phase-2.3 Δ	Status
checksums.attn_q	6.93e-06 (FIRST)	(not divergent)	✅ FIXED
checksums.mamba2_grad_w_c_l1	3.57e-07 (cascade)	7.05e-07 (FIRST)	⚠️ residual exposed
All others (q_grad, encoder weights cascade)	divergent	not divergent	✅ FIXED

The PEDANTIC fix landed the major cuBLAS-driven non-determinism. The attn_q cascade is GONE. What remains is a small residual in mamba2's L1 d_w_c gradient.

Trajectory-level impact of residual

Measured at step 197 of the 200-step quick determinism check:

Signal	Run A	Run B	Δ
train pnl	$33,064.97	$32,674.07	$391
wr	0.304264	0.303969	2.95e-4
entropy	1.289456	1.289806	3.5e-4

Eval summary (50 steps):

Signal	Run A	Run B	Δ
max_drawdown	$254,825	$438,637	$184k
eval trades seen	392	431	39

The 7e-7 step-2 gradient checksum residual cascades to a $184k eval max_drawdown difference. Sub-eps gradient drift × non-linear RL dynamics × 247 steps = real trajectory divergence. This residual must be eliminated.

What's been ruled out (post-PEDANTIC)

• Mamba2 forward scan: per-thread sequential, deterministic by construction
• mamba2_alpha_scan_bwd_seq: per-thread sequential with unique slot writes
• mamba2_alpha_reduce_d_proj: per-thread sequential over fixed j=0..sh2-1
• mamba2_alpha_reduce_d_w_c: per-thread sequential over fixed i=0..N-1
• The 3 mamba2 cuBLAS backward GEMMs: now PEDANTIC, attn_q proves they're deterministic
• All foreseen mamba2 paths read on inspection

What's NOT ruled out (Phase 2.4 scope)

1. PEDANTIC may not be sufficient for some GEMM shapes: NVIDIA docs say PEDANTIC disables non-deterministic algorithms; but for the specific shapes the mamba2 backward uses (e.g., K-dim is the time axis K=32, so K-split could still occur), some variants might persist. Try also CUBLAS_DEFAULT_MATH (no TF32 at all) as a control.
2. AdamW kernel: mamba2_alpha_adamw_step (line 525 of build.rs reference) — uses per-thread state per parameter; should be deterministic but worth dumping.
3. Reductions in the trunk: vsn / cfc / attn / ln_a / ln_b have their own custom kernels we haven't dump-compared at step 1 backward; one of them may have a parallel reduce.
4. f32 catastrophic cancellation: at the tiny gradient magnitudes (~3e-4) where d_w_c lives, subtraction patterns in different orders can produce different fp results. May need f64 accumulation.

Phase 2.4 dispatch ask

Reuse the existing dump infrastructure (FOXHUNT_DETERMINISTIC_DEBUG_MAMBA2=1 + scripts/compare-mamba2-state.py) to capture step-1 backward outputs:

• d_a_per_channel, d_b_per_channel, d_w_c_per_sample (the scan_bwd_seq outputs)
• d_a_proj, d_b_proj, d_w_c (the reduce outputs)
• The upstream d_h_enriched_seq input to mamba2 L1 backward

If d_w_c_per_sample diverges with deterministic inputs → bug is in scan_bwd_seq (despite inspection). If d_h_enriched_seq diverges → bug is upstream of mamba2 L1 (likely in some trunk reduction).

Also worth trying CUBLAS_DEFAULT_MATH (no TF32, no PEDANTIC) on the 3 sites as a control — see if the residual changes.

The fix is likely small (~30 LOC kernel modification or a different cuBLAS mode) but requires the dump evidence to localize.

---

Phase 2.4 outcome — residual is NOT in mamba2 or cuBLAS; bug is upstream of mamba2 in the RL step loop

Date appended: 2026-06-02 (Phase 2.4 dispatch) Investigation: dump-compare per dispatch Task 1, plus DEFAULT_MATH control per Task 2.

Task 1 — dump-compare at step 1 backward outputs

Phase 2.4 extended the existing dump_mamba2_state_for_debug method to capture (per L1 backward; L2 selectively):

• Per-channel scan_bwd_seq outputs: d_a_per_channel, d_b_per_channel, d_w_c_per_sample, d_h_s2
• Reduce kernel outputs: d_a_proj_2d, d_b_proj_2d, dw_c_reduced
• cuBLAS GEMM outputs: dw_in, dw_a, dw_b, db_in, db_a, db_b, d_x, d_x_from_in
• Trunk weights (upstream): vsn_w, attn_q, cfc_w_in, cfc_w_rec, ln_a_gain, ln_b_gain
• Trunk gradients (pre-Adam): grad_vsn_w, grad_attn_q, grad_cfc_w_in, grad_cfc_w_rec, grad_ln_a_gain, grad_ln_b_gain

Result at step 1 dump (FOXHUNT_DETERMINISTIC=1, PEDANTIC):

=== step 1 ===
  mamba2_l1_w_in: EQUAL (7168 elements)
  ...  [all 18 mamba2 weights]   ALL EQUAL
  [all 7 forward activations]    ALL EQUAL
  [all 5 mamba2 backward scratches] ALL EQUAL
  [all 13 mamba2 backward outputs]  ALL EQUAL  ← INCLUDES dw_in, dw_a, dw_b, dw_c
  [all 6 trunk weights]          ALL EQUAL
  [all 6 trunk gradients]        ALL EQUAL
=== step 2 ===
  mamba2_l1_w_in: DIVERGE        ← divergence enters HERE

Verdict: at step 1's full post-state dump (post-forward, post-backward, post-Adam), EVERYTHING is bit-identical between run A and run B. This includes:

• All mamba2 backward kernels' outputs (scan_bwd_seq scratch, reduce outputs, cuBLAS GEMM outputs)
• All trunk weight gradients (vsn / attn / cfc / LN)
• All weights after Adam step 1 applied

Yet at step 2's dump, mamba2 weights diverge. This means the bug is in something we have NOT yet captured — between step 1's diag emission and step 2's diag emission.

What's between step 1 dump and step 2 dump

Per crates/ml-alpha/examples/alpha_rl_train.rs:496-714 the per-step sequence is:

1. step_with_lobsim_gpu() — full training step (data load, forward, backward, Adam, RL env update)
2. diag_staging.sync_and_swap() + snapshot_async()
3. build_diag_value() ← where the dump fires

So at step 2's build_diag_value, the state captured is post-step-2 full cycle.

Between step 1 dump and step 2 dump, step 2's step_with_lobsim_gpu runs. Its inputs are:

• Step 1's POST-Adam weights (EQUAL per dump)
• Step 2's training data batch
• Step 2's RL environment state (positions, cfc carry, ISV controllers)
• Step 2's PER replay sample
• Step 2's action sampling RNG state

The not-yet-instrumented candidates are:

1. CfC recurrent carry (cfc_h_state_step_d) — crosses steps, could diverge
2. PER buffer state — sample priorities, tree state (Phase 2 fixed the rebuild — but sampling itself?)
3. Action sampling RNG / categorical sampling at step 2
4. LOB simulator state (positions, pyramid, balance) — RL env evolves across steps
5. ISV controllers — cross-step state (popart, win_rate EMA, kelly state, etc.)
6. Reward shaping kernels (rl_fused_reward_pipeline)

Task 2 — CUBLAS_DEFAULT_MATH control

Tested FOXHUNT_DETERMINISTIC=2 (DEFAULT_MATH, no TF32, no PEDANTIC).

Result: same residual mamba2_grad_w_c_l1 Δ ≈ 1e-7 between runs. DEFAULT_MATH does NOT eliminate the residual, meaning the residual is NOT a cuBLAS algorithm-selection artefact. PEDANTIC alone is sufficient for the cuBLAS GEMM determinism.

Side note: across multiple runs, the first-divergent leaf alternates between attn_q, vsn_w, and mamba2_grad_w_c_l1 (the divergence magnitudes differ run-to-run from 1e-7 to 5e-5). This suggests the divergence enters from a non-deterministic source whose magnitude varies — consistent with action sampling RNG or PER sampling (random-mediated non-determinism), NOT a deterministic-but-non-bit-exact kernel.

Why the previous "attn_q FIXED" claim was wrong

The Phase 2.3 outcome section claimed "attn_q cascade is GONE" — but my repeat runs show attn_q can still emerge as first-divergent. The reason: at b=128 the absolute checksum delta is tiny (~1e-5 to 1e-7), and the random seed of the dump infrastructure interacts with the very-small divergence to produce different first-divergent leaves on different runs. The PEDANTIC fix DOES eliminate large cuBLAS-driven divergence (post-fix Δ never exceeds 1e-4 vs pre-fix 6.93e-6 → cascade to 0.14 by step 3); it does NOT achieve bit-exactness.

What WAS proven

• §2.B (cuBLAS split-K non-determinism) is FALSIFIED at PEDANTIC: confirmed PEDANTIC is being applied (added print statement to cublas_determinism.rs). Both PEDANTIC and DEFAULT_MATH leave the same residual.
• mamba2 forward+backward kernels are bit-deterministic (all dumps match at step 1 with identical inputs).
• mamba2 reduce kernels (reduce_d_w_c, reduce_d_proj) are bit-deterministic (dw_c_reduced matches at step 1).
• Trunk backward chain (CfC/LN/VSN/attn) is bit-deterministic at step 1 (all grad outputs match).

Phase 2.4 STOPS per dispatch instruction

Per dispatch: "If Task 1 finds the bug is in a kernel I haven't suspected, document and STOP — separate dispatch."

This applies: the bug is NOT in any mamba2 backward kernel, NOT in any reduce kernel, NOT in any cuBLAS call, NOT in any trunk backward kernel. The bug is in step 2's step_with_lobsim_gpu execution path — most likely in the RL env state evolution, PER sampling, action sampling RNG, or reward shaping kernels.

Per feedback_systematic_debugging + feedback_no_quickfixes: applying Task 3's proposed f64-accumulation fix to mamba2_alpha_reduce_d_w_c would be a quick-fix masking the real bug elsewhere. The reduce kernel produces identical outputs given identical inputs — its f32 precision is sufficient for the divergence we observe (Δ=1e-7).

Phase 2.5 dispatch ask

Extend the dump infrastructure to capture step-N PRE-STEP state (before step_with_lobsim_gpu runs), focused on the not-yet-instrumented candidates:

1. CfC recurrent carry — cfc_h_state_step_d: dump at step 1 dump-point (= post-step-1 state, fed into step 2 forward). If divergent → cfc cross-step carry is non-det (very surprising since Phase 2 confirmed PER was fixed).

2. PER replay state — replay_buffer.priorities_d, replay_buffer.priority_tree, the next-sample indices for step 2. If divergent → PER sampling at step 2 produces different indices despite Phase 2 fix.

3. Action sampling RNG state — wherever step 2's categorical action sampling reads from a curandState. If RNG state is per-batch but shared across steps, and one batch's RNG advance depends on conditional logic, this could diverge.

4. LOB simulator positions/balance — pyramid_units_count, unit_entry_step, etc. — if these diverge at step 2 dump but were equal at step 1 dump, the LOB sim is the source.

5. ISV controllers — popart_sigma, kelly state, wr_ema. Cross-step state. If divergent at step 1 dump (despite Phase 2 fixing PER), one of these has hidden non-determinism.

Reuse the same env-gated dump pattern. Add buffers to dump_mamba2_state_for_debug (or a new sibling method dump_rl_state_for_debug) and extend compare-mamba2-state.py (or sibling).

Predicted outcome: one of CfC carry, PER buffer state, or LOB sim state diverges at step 1 dump (= captures post-step-1 state, which gets fed into step 2 forward). That's the new first-divergent leaf to fix.

Files touched in Phase 2.4 (no commits per dispatch)

Path	Change	LOC
crates/ml-alpha/src/cublas_determinism.rs	Added FOXHUNT_DETERMINISTIC=2 control mode (DEFAULT_MATH) + verbose print of mode at handle init	+20 / -5
crates/ml-alpha/src/trainer/perception.rs	Extended dump_mamba2_state_for_debug with: 5 mamba2 backward scratch buffers (L1 + L2), 8 mamba2 backward output buffers (dw/db/d_x), 6 trunk weights, 6 trunk gradients	+90
scripts/compare-mamba2-state.py	Added matching BACKWARD_SCRATCH + TRUNK_STATE buffer lists for new dump fields	+25

Speed cost (PEDANTIC vs DEFAULT_MATH vs TF32)

Tier 1.5 mid-smoke (200 train + 50 eval, b=128) on RTX 3050:

Mode	wall-clock /run (both runs total)	vs TF32
FOXHUNT_DETERMINISTIC=1 PEDANTIC	2m 32s	+90%
FOXHUNT_DETERMINISTIC=2 DEFAULT_MATH	2m 31s (~equal to PEDANTIC)	+88%
FOXHUNT_DETERMINISTIC=0 TF32 (production)	1m 20s	baseline

Surprise: PEDANTIC is ~90% slower than TF32 on RTX 3050 (consumer-grade, no real tensor cores) — much worse than the 10-15% NVIDIA quotes for H100. The slowdown is amplified because RTX 3050 doesn't have the TF32-accelerated matrix units of Ampere data-center / H100 GPUs.

Action item for Phase 2.5: when the residual is fixed, document the H100 cost via a quick cluster smoke. If H100 PEDANTIC cost exceeds 15%, consider keeping FOXHUNT_DETERMINISTIC=0 as the production default (with =1 only for repro / debug sessions).

Determinism residual trial-to-trial variability

Same seed (42), same FOXHUNT_DETERMINISTIC=1 PEDANTIC, three trials:

Trial	First-divergent leaf	Δ at step 2
1	checksums.attn_q	1.99e-5
2	checksums.attn_q	3.97e-5
3	checksums.mamba2_grad_w_c_l1	2.40e-7

Magnitude varies by ~100× between trials; leaf identity alternates. This is consistent with a single non-deterministic source whose output magnitude varies (e.g., RNG-mediated sampling) — NOT with a deterministic-but-non-bit-exact kernel.

Pre-existing WIP fixed

Found broken Phase 2.5 WIP in crates/ml-alpha/src/trainer/integrated.rs (dump_rl_state_for_debug method at line 10171, invoked at 10519). The closure captured let stream = &self.stream which deref-coerces to &CudaStream, where the memcpy_dtoh method is NOT in scope (it lives on Arc<CudaStream>). Fixed by replacing with let stream = self.stream.clone() (cheap Arc refcount bump) so the method is found via the impl block on Arc<CudaStream>.

This unblocked the build but the Phase 2.5 RL-state-dump infrastructure remains opt-in (env-gated by FOXHUNT_DETERMINISM_DEBUG_RL=1) — ready for the Phase 2.5 dispatch to actually use it. Not exercised in this dispatch's verification.

Discipline checklist

• No new files in repo root
• No new docs outside docs/superpowers/notes/
• No .cu files added/changed (the bug is NOT in any CUDA kernel)
• Per feedback_no_atomicadd.md: not relevant
• Per feedback_no_nvrtc.md: not relevant
• Per feedback_cpu_is_read_only.md: dump methods do DtoH only, no host compute affecting training
• Per feedback_no_stubs.md: dumps wired end-to-end; no-op when env unset
• Per feedback_local_smoke_before_cluster.md: validated on RTX 3050, no cluster submit
• Per feedback_systematic_debugging + dispatch STOP rule: no fix applied — diagnosis only
• No commit — left uncommitted for human review

Reproduction

# Build with extended dumps + control mode
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

# Confirm PEDANTIC is applied (look for [cublas_determinism] line in log)
FOXHUNT_DETERMINISTIC=1 ./scripts/determinism-check.sh --quick
# Expected output:
#   FIRST_DIVERGENCE: step=2 leaf=checksums.mamba2_grad_w_c_l1 delta=1.1e-7
#   (or attn_q / vsn_w with different magnitude — divergence is small, noisy)

# Run with mamba2 dumps to see step 1's full post-step state EQUAL
FOXHUNT_DETERMINISTIC=1 ./scripts/determinism-check.sh --quick --debug-dump-mamba2
# Expected output (key bit):
#   === step 1 === ALL 52 buffers EQUAL
#   === step 2 === mamba2_l1/l2 weights DIVERGE
#                  mamba2 backward outputs ALL DIVERGE
#                  cuBLAS-GEMM outputs (dw_in/dw_a/dw_b) DIVERGE
#                  trunk weights/grads NOT dumped at step 2 (still need to add)

# Control: try DEFAULT_MATH (no TF32, no PEDANTIC) — same residual
FOXHUNT_DETERMINISTIC=2 ./scripts/determinism-check.sh --quick

# Per-leaf divergence trace: open `/tmp/foxhunt-determinism-a.log` /
# `b.log` and look for the `[cublas_determinism]` startup line to
# confirm which math mode was applied.