# Determinism Phase 2.2 — Mamba2 Sub-Investigation

**Date**: 2026-06-02
**Branch**: ml-alpha-regime-observer
**Spec**: `docs/superpowers/specs/2026-06-02-determinism-foundation.md` §2.A (hypothesised) / §2.B (actual)
**Plan**: `docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md`
**Phase 2 input**: `docs/superpowers/notes/2026-06-02-determinism-phase2-per-investigation.md`

## TL;DR — STOP, the bug is NOT mamba2; it is cuBLAS GEMM non-determinism (§2.B)

Phase 2 fixed the PER tree-rebuild and surfaced
`checksums.encoder_output` at step 3 (Δ=0.14) as the new
first-divergent leaf, with §2.A (mamba2 selective scan) listed
as the most-likely culprit per the spec lookup.

Phase 2.2 added 17 encoder-localisation checksums plus an
env-gated raw-state dumper.  The dumps tell a different and
**decisive** story:

1. At **step 2** (one step earlier than `encoder_output`), the
   first divergent leaf is **`checksums.attn_q`** — but the raw
   mamba2 dumps show that ALL encoder weights diverge
   simultaneously at step 2: vsn_w, cfc_w_in, cfc_w_rec, attn_q,
   mamba2 L1 w_in/b_in/w_a/b_a/w_b/b_b/w_c, mamba2 L2 same
   set — but **mamba2 w_out / b_out STAY EQUAL** (those weights
   aren't touched by the seq backward path).

2. Step 1's `encoder_grad` (= `grad_h_t_combined_d`) checksum
   **MATCHES bit-exactly** between runs A and B at step 1's diag
   emission (1.3292713308910526e-10 in both).  All per-head Q / π /
   V grads at step 1 also match.

3. So the encoder backward at step 1 sees IDENTICAL inputs in
   both runs, yet produces DIVERGENT gradient updates that diverge
   ALL trainable encoder weights simultaneously at step 2.

4. The only operation in the encoder backward chain that runs
   **before** the first non-deterministic weight (mamba2 L2) but
   AFTER the deterministic upstream chain is the mamba2 backward
   itself — specifically, the **cuBLAS GEMM calls inside
   `Mamba2Block::backward_from_h_enriched_seq_full_into`** at
   `crates/ml-alpha/src/mamba2_block.rs:1918-1944` (3 GEMMs:
   w_b.backward, w_a.backward, w_in.backward).

5. The cuBLAS path uses `cublasGemmEx` with algo
   `CUBLAS_GEMM_DFALT` (see
   `crates/ml-core/src/cuda_autograd/linear.rs:88`).  Per NVIDIA
   docs, default GEMM algorithms are NOT guaranteed deterministic
   — split-K with non-deterministic accumulation order is
   permitted.  No call to `cublasSetMathMode(CUBLAS_PEDANTIC_MATH)`
   exists anywhere in the codebase
   (`grep -rn "CublasMathMode\|setMathMode" crates/` returns
   nothing).

**Conclusion — this is spec §2.B, not §2.A.**  The mamba2 kernels
themselves are bit-deterministic by construction (one thread per
(i, j), unique slot writes, no atomics; reductions iterate a
single thread over a fixed range).  The non-determinism enters
through the **cuBLAS GEMMs** that surround the scan kernel
(W_in, W_a, W_b forward + backward dw/db/dx).

Per dispatch instruction "If Task 1 finds something completely
unexpected (e.g., the bug is in the mamba2_block.rs Rust glue
rather than the CUDA kernel), document and STOP — separate
dispatch": this finding lands in the Rust-glue + cuBLAS-handle
category.  Fix is a different scope (every cuBLAS handle in the
trainer must enter PEDANTIC mode, not a mamba2-local edit).

## Investigation infrastructure

Three artefacts were added (uncommitted) to localise the bug:

### 1. 17 new encoder-localisation checksum leaves

Slot indices 28-44 in `IntegratedTrainer::checksum_scratch_d`, output
indices 15-31 in `compute_diag_checksums`.  Wired into the existing
`build_diag_value` JSON `checksums` map:

| idx | leaf | source tensor |
|-----|------|---------------|
| 15 | `mamba2_l1_w_in` | `perception.trunk.mamba2_l1().w_in.weight` |
| 16 | `mamba2_l1_w_a` | `mamba2_l1.w_a.weight` |
| 17 | `mamba2_l1_w_b` | `mamba2_l1.w_b.weight` |
| 18 | `mamba2_l1_w_c` | `mamba2_l1.w_c` |
| 19 | `mamba2_l2_w_in` | `mamba2_l2.w_in.weight` |
| 20 | `mamba2_l2_w_c` | `mamba2_l2.w_c` |
| 21 | `vsn_w` | `trunk.vsn_w_d` |
| 22 | `cfc_w_in` | `trunk.w_in_d` |
| 23 | `cfc_w_rec` | `trunk.w_rec_d` |
| 24 | `attn_q` | `trunk.attn_q_d` |
| 25 | `vsn_out` | encoder intermediate activation |
| 26 | `mamba2_l1_out` | `h_enriched_seq` L1 |
| 27 | `ln_a_out` | LN_a output |
| 28 | `mamba2_l2_out` | `h_enriched_seq` L2 |
| 29 | `ln_b_out` | LN_b output |
| 30 | `attn_context` | attention pool context |
| 31 | `mamba2_grad_w_c_l1` | reduced backward `dw_c` for L1 |

`EXPECTED_LEAVES` in `eval_diag_emission.rs` bumped 694 → 711.

### 2. Env-gated raw-state dumper

`PerceptionTrainer::dump_mamba2_state_for_debug(step)` triggered
from `IntegratedTrainer::build_diag_value` at steps 0/1/2/3 when
`FOXHUNT_DETERMINISM_DEBUG_MAMBA2=1`.  Writes 26 raw f32 binaries
per step (18 weight tensors + 7 activations + 1 reduced grad) to
`$FOXHUNT_DEBUG_DUMP_DIR` (default `/tmp/foxhunt-determinism-debug`).
Mirrors the pattern in Phase 2's `dump_per_state_for_debug`.

### 3. `scripts/compare-mamba2-state.py`

Exact-bytewise diff with verdict logic for sub-cases 2.A.1
through 2.A.5 (weights vs intermediates vs reduced grads).

### 4. `scripts/determinism-check.sh --debug-dump-mamba2`

New flag plus `mamba2_l1_w_in` / `vsn_out` / etc. entries in the
per-leaf "fix recommendation" table.

## Empirical findings — checksums

`./scripts/determinism-check.sh --quick` after Phase 2.2 wiring:

```
FIRST_DIVERGENCE: step=2 leaf=checksums.attn_q
                  run_A=0.31439592314009085  run_B=0.31440285748760444
                  delta=6.93435e-06

All leaves diverging at step=2:
  checksums.attn_q             Δ=6.93e-06
  checksums.mamba2_grad_w_c_l1 Δ=3.57e-07
```

Following the trace via JSONL inspection:

| Step | run_A vsn_w | run_B vsn_w | run_A mamba2_l1_w_in | run_B mamba2_l1_w_in |
|------|-------------|-------------|----------------------|----------------------|
| 0 | 18.66958 | 18.66958 | 79.54464 | 79.54464 |
| 1 | 18.66958 | 18.66958 | 79.54464 | 79.54464 |
| 2 | **18.75015** | **18.75025** | **79.57704** | **79.57719** |
| 3 | 18.86302 | 18.86328 | 79.59899 | 79.59909 |

Steps 0 and 1 are bit-identical between runs (l_q = l_pi = l_v = 0
at step 0; step 1 fires the first non-zero backward).  At step 2's
diag, EVERY encoder weight has diverged.

`encoder_grad` (= `grad_h_t_combined_d`) at step 1:
- run A: 1.3292713308910526e-10
- run B: 1.3292713308910526e-10  (MATCH)

`q_grad`, `pi_grad`, `v_grad` at step 2 also all MATCH between
runs (sum-of-squares 3636.114, 0.0003699, 4.770977 in both).

## Empirical findings — raw dumps

`./scripts/determinism-check.sh --quick --debug-dump-mamba2`:

```
=== step 0 ===
  ALL 26 files: EQUAL
=== step 1 ===
  ALL 26 files: EQUAL
=== step 2 ===
  mamba2_l1_w_in:  DIVERGE at idx 0
  mamba2_l1_b_in:  DIVERGE at idx 0
  mamba2_l1_w_a:   DIVERGE
  mamba2_l1_b_a:   DIVERGE
  mamba2_l1_w_b:   DIVERGE
  mamba2_l1_b_b:   DIVERGE
  mamba2_l1_w_c:   DIVERGE
  mamba2_l1_w_out: EQUAL
  mamba2_l1_b_out: EQUAL
  (same pattern for L2)
  vsn_out:                  EQUAL  (forward output at step 2)
  mamba2_l1_h_enriched_seq: EQUAL
  ln_a_out:                 EQUAL
  mamba2_l2_h_enriched_seq: EQUAL
  ln_b_out:                 EQUAL
  attn_context:             EQUAL
  h_t:                      EQUAL
=== step 3 ===
  ALL diverge (cascade)
```

**This is decisive sub-case 2.A.4 (weight drift) — but with the
critical refinement that `w_out` / `b_out` STAY EQUAL.**  Per
`crates/ml-alpha/src/mamba2_block.rs:305`:

> "W_out is unused in the seq path — dw_out / db_out are zero-init
>  shells of the right shape so the Mamba2AdamW step is a no-op for
>  those parameters."

The diverging weights are exactly the set of params with non-zero
gradients in the seq path.  Their gradients flow through
`backward_from_h_enriched_seq_full_into` which uses three
cuBLAS GEMMs (W_b backward, W_a backward, W_in backward).

The cuBLAS handle is initialised via `CudaBlas::new(stream)` and
never has `cublasSetMathMode` called on it.  Per NVIDIA docs:

- `CUBLAS_DEFAULT_MATH` (and `CUBLAS_GEMM_DFALT` algorithm) allows
  TF32 + split-K + heuristic-selected variants on Ampere+.
- Split-K with atomic accumulation across blocks produces
  **non-deterministic** floating-point sums across runs even with
  identical inputs and identical algorithm selection.
- The only way to disable this is `cublasSetMathMode(handle,
  CUBLAS_PEDANTIC_MATH)` PLUS explicit algorithm selection (or
  rely on `CUBLAS_PEDANTIC_MATH` to filter out non-deterministic
  algos).

## Why the OUTPUTS at step 2 match but the WEIGHTS diverge

The dump shows at step 2:
- `mamba2_l1_h_enriched_seq`: **EQUAL** (forward output bit-exact)
- `mamba2_l1_w_in`: **DIVERGE** (weights bit-different)

This is consistent with: at step 2's diag emission, the encoder
forward has run with the POST-Adam weights from step 1's backward.
But our dumps for the activations show pre-step-2 state — i.e.,
the forward dump was taken AFTER the diag forward fired but the
ACTIVATIONS were captured into stable scratch buffers that haven't
been re-overwritten by step 2's backward yet.

Wait — actually the dump fires inside `build_diag_value` AFTER the
step has completed.  Step 2's full step has run: forward (with
post-step-1 weights, so the forward used DIVERGED weights) +
backward (writing new grads).  Then the dump captures BOTH the
current weights AND the activations.

So either:
(a) The forward at step 2 used divergent weights but happened to
    produce equal h_enriched_seq within fp32 noise floor, or
(b) The forward output match is genuine (the divergence at step 2
    in mamba2 weights is so tiny — Δ ~1e-5 — that the forward
    output difference is below the exact-bytewise threshold of
    the dump, which compares bit-equality).

Spot-check at run A vs B w_in[0]: 0.17429381608963013 vs
0.17429661750793457 → diff ~2.8e-6.  Multiplied through a 128-wide
GEMM with input magnitudes ~1, the output deviation is on the
order of 1e-6 * sqrt(128) ~ 1e-5.  This IS below f32 epsilon-level
in some sums but should still show up as bit-different in most.

So (a) is more likely — the propagated forward errors at step 2
ARE below fp32 last-bit thresholds for SOME tensors but should
manifest at step 3 (which they do — all activations DIVERGE at
step 3).

This is consistent with the cuBLAS GEMM hypothesis: the BACKWARD
GEMM is split-K non-deterministic (running over a B*K reduction
dimension where bits can shift across runs), but the forward GEMM
uses different shapes where the same algo IS deterministic in
practice (no split-K because the reduction dim is small enough).

## What needs to happen next — Phase 2.3 dispatch

**Per dispatch instruction: STOP.**  The fix is not a single-file
mamba2 edit; it is a cross-cutting cuBLAS handle reconfiguration
spec §2.B requires:

1. Every cuBLAS handle in the trainer must set
   `cublasSetMathMode(CUBLAS_PEDANTIC_MATH)` at construction.
   At minimum: `Mamba2Block.cublas`,
   `dqn.cublas`, `iqn.cublas` (per the grep at the start of this
   investigation).  Possibly others.

2. Document the performance cost — NVIDIA quotes ~10-15% on H100
   for general GEMMs.  Acceptable per spec §4 (`FOXHUNT_DETERMINISTIC`
   env toggle).

3. Re-run `./scripts/determinism-check.sh --quick` on seeds
   42 / 43 / 44 — if the fix is correct, all three should pass
   exit-0 with no first-divergence reported.

4. If a residual divergence surfaces in some other layer (e.g.,
   a custom kernel we haven't yet checksummed), Phase 2.3 dumps
   will pinpoint it.

## Out of scope for this dispatch

Per `feedback_extending_existing_code_audits_for_existing_violations`,
flagging two pre-existing items observed during this audit but
NOT fixing (out of scope):

1. **`CUBLAS_GEMM_DFALT` comment is misleading**:
   `crates/ml-core/src/cuda_autograd/linear.rs:82-88` calls
   the algorithm "Deterministic default algo" but per NVIDIA
   docs `CUBLAS_GEMM_DFALT` is the heuristic-selected default
   and is NOT guaranteed deterministic.  Comment should be
   corrected when §2.B is applied.

2. **`Mamba2BlockForwardScratch` exposes only `pub`-fields** —
   public access by the perception trainer dispatch is via
   `mamba2_fwd_scratch.h_enriched_seq.cuda_data()`.  When the
   dump method here borrows it via a new public accessor, it
   crosses module boundaries that were previously informal.  No
   structural issue; just noted.

## Speed cost (post-instrumentation)

Tier 1.5 mid-smoke (200 train + 50 eval, b=128) on RTX 3050:

| Version | Run A | Run B |
|---------|-------|-------|
| Phase 2 (15 checksums)        | 0m 43s | 0m 43s |
| Phase 2.2 (32 checksums)      | 0m 47s | 0m 46s |

The +4s per 250-step run is ~16 ms/step extra, consistent with
17 additional 1-thread checksum kernels each ~1 ms on the largest
tensor (524288 f32s = 524288 ns at ~1 GFLOP/s sequential f64).
Negligible vs the 200-ms/step training step.

## Files touched (Phase 2.2, no commits per dispatch)

| Path | Change | LOC |
|------|--------|-----|
| `crates/ml-alpha/src/trainer/perception.rs` | 7 new `debug_*_view` accessors + `dump_mamba2_state_for_debug` method | +130 |
| `crates/ml-alpha/src/trainer/integrated.rs` | (a) scratch+host buffer 28→45 slots (b) 17 new checksum launches (c) returns `[f64; 32]` (d) JSON emission +17 leaves (e) `dump_mamba2_state_for_debug` call from `build_diag_value` | +90 / -10 |
| `crates/ml-alpha/tests/eval_diag_emission.rs` | `EXPECTED_LEAVES` 694 → 711 | +5 |
| `scripts/determinism-check.sh` | `--debug-dump-mamba2` flag + env plumbing + 17 new candidate-leaf entries | +30 / -5 |
| `scripts/compare-mamba2-state.py` (NEW) | Mamba2 dump comparison + sub-case verdict | +240 |
| `docs/superpowers/notes/2026-06-02-determinism-phase2.2-mamba2-investigation.md` (NEW) | this note | new |

## Discipline checklist

- [x] No new files in repo root
- [x] No new docs outside `docs/superpowers/notes/`
- [x] No `.cu` files added/changed (the bug is NOT in mamba2 kernels)
- [x] Per `feedback_no_atomicadd.md`: not relevant (this is cuBLAS)
- [x] Per `feedback_no_nvrtc.md`: not relevant
- [x] Per `feedback_cpu_is_read_only.md`: the dump method only does
      DtoH copies for off-line analysis, no host compute
- [x] Per `feedback_no_stubs.md`: dumps + checksums wired end-to-end
- [x] Per `feedback_local_smoke_before_cluster.md`: validated on
      RTX 3050 with the quick determinism check; no cluster submit
- [x] No fix applied — diagnosis only, per dispatch STOP instruction
- [x] No commit — left uncommitted for human review

## Reproduction

```bash
# Build (idempotent, sccache helps)
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

# Run the quick determinism check WITH 17 new encoder leaves
./scripts/determinism-check.sh --quick

# Output:
#  FIRST_DIVERGENCE: step=2 leaf=checksums.attn_q
#                    delta=6.93e-06

# Run again with raw mamba2 dumps for the sub-case verdict
./scripts/determinism-check.sh --quick --debug-dump-mamba2

# Output (key bit):
#  VERDICT (first divergent step = 2):
#  CASE 2.A.4 — WEIGHTS DIVERGE upstream of forward.
#    Mamba2 L1/L2 weights diverged at step 2.
```

## Recommended next-dispatch ask (Phase 2.3)

**One-liner ask**: Apply spec §2.B (cuBLAS PEDANTIC_MATH) to every
`CudaBlas::new(...)` site in `crates/ml-alpha/src/`.  Verify
`determinism-check.sh --quick` exits 0 on seeds 42/43/44.

**Sites to patch** (3 known cuBLAS handles, grep
`"CudaBlas::new"`):

- `crates/ml-alpha/src/mamba2_block.rs:535` (Mamba2Block.cublas)
- `crates/ml-alpha/src/rl/iqn.rs:261` (IqnHead.cublas)
- `crates/ml-alpha/src/rl/dqn.rs:364` (DqnHead.cublas)

Required call: immediately after `CudaBlas::new`, call
```rust
unsafe {
    cudarc::cublas::sys::cublasSetMathMode(
        *cublas.handle(),
        cudarc::cublas::sys::cublasMath_t::CUBLAS_PEDANTIC_MATH,
    ).result()
    .map_err(|e| anyhow!("cublasSetMathMode PEDANTIC: {e:?}"))?;
}
```

Plus an env toggle `FOXHUNT_DETERMINISTIC` per spec §4 (default
"1" in dev → PEDANTIC, "0" in production → TF32_TENSOR_OP).

Once §2.B lands, re-run the same `determinism-check.sh --quick`
infrastructure (no further new instrumentation needed) — should
go straight to exit 0 with all 32 checksums matching.

---

## Phase 2.3 outcome — cuBLAS PEDANTIC fix lands MAJOR bug; residual remains

**Date appended**: 2026-06-02 (continuation session)
**Fix shipped**: `crates/ml-alpha/src/cublas_determinism.rs` (NEW helper) + 3 site insertions at the post-`CudaBlas::new` lines in `mamba2_block.rs:535`, `rl/dqn.rs:364`, `rl/iqn.rs:261`. Env-gated by `FOXHUNT_DETERMINISTIC=1` (default ON in dev).

### Empirical post-fix state

`./scripts/determinism-check.sh --quick` after Phase 2.3:
```
FIRST_DIVERGENCE: step=2 leaf=checksums.mamba2_grad_w_c_l1
                  run_A=2.04e-07  run_B=9.09e-07  delta=7.05e-07

All leaves diverging at step=2:
  checksums.mamba2_grad_w_c_l1   Δ=7.05e-07
```

**Comparison to pre-fix**:
| Leaf | Pre-Phase-2.3 Δ | Post-Phase-2.3 Δ | Status |
|------|-----------------|------------------|--------|
| `checksums.attn_q` | 6.93e-06 (FIRST) | (not divergent) | ✅ FIXED |
| `checksums.mamba2_grad_w_c_l1` | 3.57e-07 (cascade) | 7.05e-07 (FIRST) | ⚠️ residual exposed |
| All others (q_grad, encoder weights cascade) | divergent | not divergent | ✅ FIXED |

The PEDANTIC fix landed the major cuBLAS-driven non-determinism. The attn_q cascade is GONE. What remains is a small residual in mamba2's L1 d_w_c gradient.

### Trajectory-level impact of residual

Measured at step 197 of the 200-step quick determinism check:

| Signal | Run A | Run B | Δ |
|--------|-------|-------|---|
| train pnl | $33,064.97 | $32,674.07 | $391 |
| wr | 0.304264 | 0.303969 | 2.95e-4 |
| entropy | 1.289456 | 1.289806 | 3.5e-4 |

Eval summary (50 steps):
| Signal | Run A | Run B | Δ |
|--------|-------|-------|---|
| max_drawdown | $254,825 | $438,637 | **$184k** |
| eval trades seen | 392 | 431 | 39 |

The 7e-7 step-2 gradient checksum residual cascades to a **$184k eval max_drawdown** difference. Sub-eps gradient drift × non-linear RL dynamics × 247 steps = real trajectory divergence. **This residual must be eliminated.**

### What's been ruled out (post-PEDANTIC)

- **Mamba2 forward scan**: per-thread sequential, deterministic by construction
- **`mamba2_alpha_scan_bwd_seq`**: per-thread sequential with unique slot writes
- **`mamba2_alpha_reduce_d_proj`**: per-thread sequential over fixed j=0..sh2-1
- **`mamba2_alpha_reduce_d_w_c`**: per-thread sequential over fixed i=0..N-1
- **The 3 mamba2 cuBLAS backward GEMMs**: now PEDANTIC, attn_q proves they're deterministic
- **All foreseen mamba2 paths read on inspection**

### What's NOT ruled out (Phase 2.4 scope)

1. **PEDANTIC may not be sufficient for some GEMM shapes**: NVIDIA docs say PEDANTIC disables non-deterministic algorithms; but for the specific shapes the mamba2 backward uses (e.g., K-dim is the time axis K=32, so K-split could still occur), some variants might persist. Try also `CUBLAS_DEFAULT_MATH` (no TF32 at all) as a control.
2. **AdamW kernel**: `mamba2_alpha_adamw_step` (line 525 of build.rs reference) — uses per-thread state per parameter; should be deterministic but worth dumping.
3. **Reductions in the trunk**: vsn / cfc / attn / ln_a / ln_b have their own custom kernels we haven't dump-compared at step 1 backward; one of them may have a parallel reduce.
4. **f32 catastrophic cancellation**: at the tiny gradient magnitudes (~3e-4) where d_w_c lives, subtraction patterns in different orders can produce different fp results. May need f64 accumulation.

### Phase 2.4 dispatch ask

Reuse the existing dump infrastructure (`FOXHUNT_DETERMINISTIC_DEBUG_MAMBA2=1` + `scripts/compare-mamba2-state.py`) to capture step-1 backward outputs:
- `d_a_per_channel`, `d_b_per_channel`, `d_w_c_per_sample` (the scan_bwd_seq outputs)
- `d_a_proj`, `d_b_proj`, `d_w_c` (the reduce outputs)
- The upstream `d_h_enriched_seq` input to mamba2 L1 backward

If `d_w_c_per_sample` diverges with deterministic inputs → bug is in scan_bwd_seq (despite inspection).
If `d_h_enriched_seq` diverges → bug is upstream of mamba2 L1 (likely in some trunk reduction).

Also worth trying `CUBLAS_DEFAULT_MATH` (no TF32, no PEDANTIC) on the 3 sites as a control — see if the residual changes.

The fix is likely small (~30 LOC kernel modification or a different cuBLAS mode) but requires the dump evidence to localize.

---

## Phase 2.4 outcome — residual is NOT in mamba2 or cuBLAS; bug is upstream of mamba2 in the RL step loop

**Date appended**: 2026-06-02 (Phase 2.4 dispatch)
**Investigation**: dump-compare per dispatch Task 1, plus DEFAULT_MATH control per Task 2.

### Task 1 — dump-compare at step 1 backward outputs

Phase 2.4 extended the existing `dump_mamba2_state_for_debug` method to capture (per L1 backward; L2 selectively):
- Per-channel scan_bwd_seq outputs: `d_a_per_channel`, `d_b_per_channel`, `d_w_c_per_sample`, `d_h_s2`
- Reduce kernel outputs: `d_a_proj_2d`, `d_b_proj_2d`, `dw_c_reduced`
- cuBLAS GEMM outputs: `dw_in`, `dw_a`, `dw_b`, `db_in`, `db_a`, `db_b`, `d_x`, `d_x_from_in`
- Trunk weights (upstream): `vsn_w`, `attn_q`, `cfc_w_in`, `cfc_w_rec`, `ln_a_gain`, `ln_b_gain`
- Trunk gradients (pre-Adam): `grad_vsn_w`, `grad_attn_q`, `grad_cfc_w_in`, `grad_cfc_w_rec`, `grad_ln_a_gain`, `grad_ln_b_gain`

**Result at step 1 dump (`FOXHUNT_DETERMINISTIC=1`, PEDANTIC)**:
```
=== step 1 ===
  mamba2_l1_w_in: EQUAL (7168 elements)
  ...  [all 18 mamba2 weights]   ALL EQUAL
  [all 7 forward activations]    ALL EQUAL
  [all 5 mamba2 backward scratches] ALL EQUAL
  [all 13 mamba2 backward outputs]  ALL EQUAL  ← INCLUDES dw_in, dw_a, dw_b, dw_c
  [all 6 trunk weights]          ALL EQUAL
  [all 6 trunk gradients]        ALL EQUAL
=== step 2 ===
  mamba2_l1_w_in: DIVERGE        ← divergence enters HERE
```

**Verdict**: at step 1's full post-state dump (post-forward, post-backward, post-Adam), **EVERYTHING is bit-identical** between run A and run B. This includes:
- All mamba2 backward kernels' outputs (scan_bwd_seq scratch, reduce outputs, cuBLAS GEMM outputs)
- All trunk weight gradients (vsn / attn / cfc / LN)
- All weights after Adam step 1 applied

Yet at step 2's dump, mamba2 weights diverge. **This means the bug is in something we have NOT yet captured** — between step 1's diag emission and step 2's diag emission.

### What's between step 1 dump and step 2 dump

Per `crates/ml-alpha/examples/alpha_rl_train.rs:496-714` the per-step sequence is:
1. `step_with_lobsim_gpu()` — full training step (data load, forward, backward, Adam, RL env update)
2. `diag_staging.sync_and_swap()` + `snapshot_async()`
3. `build_diag_value()` ← **where the dump fires**

So at step 2's `build_diag_value`, the state captured is **post-step-2 full cycle**.

Between step 1 dump and step 2 dump, step 2's `step_with_lobsim_gpu` runs. Its inputs are:
- Step 1's POST-Adam weights (EQUAL per dump)
- Step 2's training data batch
- Step 2's RL environment state (positions, cfc carry, ISV controllers)
- Step 2's PER replay sample
- Step 2's action sampling RNG state

The not-yet-instrumented candidates are:
1. **CfC recurrent carry** (`cfc_h_state_step_d`) — crosses steps, could diverge
2. **PER buffer state** — sample priorities, tree state (Phase 2 fixed the rebuild — but sampling itself?)
3. **Action sampling RNG / categorical sampling** at step 2
4. **LOB simulator state** (positions, pyramid, balance) — RL env evolves across steps
5. **ISV controllers** — cross-step state (popart, win_rate EMA, kelly state, etc.)
6. **Reward shaping kernels** (rl_fused_reward_pipeline)

### Task 2 — `CUBLAS_DEFAULT_MATH` control

Tested `FOXHUNT_DETERMINISTIC=2` (DEFAULT_MATH, no TF32, no PEDANTIC).

**Result**: same residual `mamba2_grad_w_c_l1` Δ ≈ 1e-7 between runs. **DEFAULT_MATH does NOT eliminate the residual**, meaning the residual is **NOT a cuBLAS algorithm-selection artefact**. PEDANTIC alone is sufficient for the cuBLAS GEMM determinism.

Side note: across multiple runs, the first-divergent leaf alternates between `attn_q`, `vsn_w`, and `mamba2_grad_w_c_l1` (the divergence magnitudes differ run-to-run from 1e-7 to 5e-5). This suggests the divergence enters from a non-deterministic source whose magnitude varies — consistent with **action sampling RNG** or **PER sampling** (random-mediated non-determinism), NOT a deterministic-but-non-bit-exact kernel.

### Why the previous "attn_q FIXED" claim was wrong

The Phase 2.3 outcome section claimed "attn_q cascade is GONE" — but my repeat runs show attn_q can still emerge as first-divergent. The reason: at b=128 the absolute checksum delta is tiny (~1e-5 to 1e-7), and the random seed of the dump infrastructure interacts with the very-small divergence to produce different first-divergent leaves on different runs. The PEDANTIC fix DOES eliminate large cuBLAS-driven divergence (post-fix Δ never exceeds 1e-4 vs pre-fix 6.93e-6 → cascade to 0.14 by step 3); it does NOT achieve bit-exactness.

### What WAS proven

- **§2.B (cuBLAS split-K non-determinism) is FALSIFIED at PEDANTIC**: confirmed PEDANTIC is being applied (added print statement to `cublas_determinism.rs`). Both PEDANTIC and DEFAULT_MATH leave the same residual.
- **mamba2 forward+backward kernels are bit-deterministic** (all dumps match at step 1 with identical inputs).
- **mamba2 reduce kernels (`reduce_d_w_c`, `reduce_d_proj`) are bit-deterministic** (dw_c_reduced matches at step 1).
- **Trunk backward chain (CfC/LN/VSN/attn) is bit-deterministic at step 1** (all grad outputs match).

### Phase 2.4 STOPS per dispatch instruction

Per dispatch: *"If Task 1 finds the bug is in a kernel I haven't suspected, document and STOP — separate dispatch."*

This applies: the bug is **NOT in any mamba2 backward kernel, NOT in any reduce kernel, NOT in any cuBLAS call, NOT in any trunk backward kernel**. The bug is in step 2's `step_with_lobsim_gpu` execution path — most likely in the RL env state evolution, PER sampling, action sampling RNG, or reward shaping kernels.

Per `feedback_systematic_debugging` + `feedback_no_quickfixes`: applying Task 3's proposed f64-accumulation fix to `mamba2_alpha_reduce_d_w_c` would be a quick-fix masking the real bug elsewhere. The reduce kernel produces identical outputs given identical inputs — its f32 precision is sufficient for the divergence we observe (Δ=1e-7).

### Phase 2.5 dispatch ask

Extend the dump infrastructure to capture step-N PRE-STEP state (before `step_with_lobsim_gpu` runs), focused on the not-yet-instrumented candidates:

1. **CfC recurrent carry** — `cfc_h_state_step_d`: dump at step 1 dump-point (= post-step-1 state, fed into step 2 forward). If divergent → cfc cross-step carry is non-det (very surprising since Phase 2 confirmed PER was fixed).

2. **PER replay state** — `replay_buffer.priorities_d`, `replay_buffer.priority_tree`, the next-sample indices for step 2. If divergent → PER sampling at step 2 produces different indices despite Phase 2 fix.

3. **Action sampling RNG state** — wherever step 2's categorical action sampling reads from a curandState. If RNG state is per-batch but shared across steps, and one batch's RNG advance depends on conditional logic, this could diverge.

4. **LOB simulator positions/balance** — `pyramid_units_count`, `unit_entry_step`, etc. — if these diverge at step 2 dump but were equal at step 1 dump, the LOB sim is the source.

5. **ISV controllers** — popart_sigma, kelly state, wr_ema. Cross-step state. If divergent at step 1 dump (despite Phase 2 fixing PER), one of these has hidden non-determinism.

Reuse the same env-gated dump pattern. Add buffers to `dump_mamba2_state_for_debug` (or a new sibling method `dump_rl_state_for_debug`) and extend `compare-mamba2-state.py` (or sibling).

**Predicted outcome**: one of CfC carry, PER buffer state, or LOB sim state diverges at step 1 dump (= captures post-step-1 state, which gets fed into step 2 forward). That's the new first-divergent leaf to fix.

### Files touched in Phase 2.4 (no commits per dispatch)

| Path | Change | LOC |
|------|--------|-----|
| `crates/ml-alpha/src/cublas_determinism.rs` | Added `FOXHUNT_DETERMINISTIC=2` control mode (DEFAULT_MATH) + verbose print of mode at handle init | +20 / -5 |
| `crates/ml-alpha/src/trainer/perception.rs` | Extended `dump_mamba2_state_for_debug` with: 5 mamba2 backward scratch buffers (L1 + L2), 8 mamba2 backward output buffers (dw/db/d_x), 6 trunk weights, 6 trunk gradients | +90 |
| `scripts/compare-mamba2-state.py` | Added matching `BACKWARD_SCRATCH` + `TRUNK_STATE` buffer lists for new dump fields | +25 |

### Speed cost (PEDANTIC vs DEFAULT_MATH vs TF32)

Tier 1.5 mid-smoke (200 train + 50 eval, b=128) on RTX 3050:

| Mode | wall-clock /run (both runs total) | vs TF32 |
|------|------|---------|
| `FOXHUNT_DETERMINISTIC=1` PEDANTIC | 2m 32s | +90% |
| `FOXHUNT_DETERMINISTIC=2` DEFAULT_MATH | 2m 31s (~equal to PEDANTIC) | +88% |
| `FOXHUNT_DETERMINISTIC=0` TF32 (production) | 1m 20s | baseline |

**Surprise**: PEDANTIC is ~90% slower than TF32 on RTX 3050 (consumer-grade, no real tensor cores) — much worse than the 10-15% NVIDIA quotes for H100. The slowdown is amplified because RTX 3050 doesn't have the TF32-accelerated matrix units of Ampere data-center / H100 GPUs.

**Action item for Phase 2.5**: when the residual is fixed, document the H100 cost via a quick cluster smoke. If H100 PEDANTIC cost exceeds 15%, consider keeping `FOXHUNT_DETERMINISTIC=0` as the production default (with `=1` only for repro / debug sessions).

### Determinism residual trial-to-trial variability

Same seed (42), same FOXHUNT_DETERMINISTIC=1 PEDANTIC, three trials:

| Trial | First-divergent leaf | Δ at step 2 |
|------:|----------------------|-------------|
| 1 | `checksums.attn_q` | 1.99e-5 |
| 2 | `checksums.attn_q` | 3.97e-5 |
| 3 | `checksums.mamba2_grad_w_c_l1` | 2.40e-7 |

Magnitude varies by ~100× between trials; leaf identity alternates. This is consistent with a single non-deterministic source whose output magnitude varies (e.g., RNG-mediated sampling) — NOT with a deterministic-but-non-bit-exact kernel.

### Pre-existing WIP fixed

Found broken Phase 2.5 WIP in `crates/ml-alpha/src/trainer/integrated.rs` (`dump_rl_state_for_debug` method at line 10171, invoked at 10519). The closure captured `let stream = &self.stream` which deref-coerces to `&CudaStream`, where the `memcpy_dtoh` method is NOT in scope (it lives on `Arc<CudaStream>`). Fixed by replacing with `let stream = self.stream.clone()` (cheap Arc refcount bump) so the method is found via the impl block on `Arc<CudaStream>`.

This unblocked the build but the Phase 2.5 RL-state-dump infrastructure remains opt-in (env-gated by `FOXHUNT_DETERMINISM_DEBUG_RL=1`) — ready for the Phase 2.5 dispatch to actually use it. Not exercised in this dispatch's verification.

### Discipline checklist

- [x] No new files in repo root
- [x] No new docs outside `docs/superpowers/notes/`
- [x] No `.cu` files added/changed (the bug is NOT in any CUDA kernel)
- [x] Per `feedback_no_atomicadd.md`: not relevant
- [x] Per `feedback_no_nvrtc.md`: not relevant
- [x] Per `feedback_cpu_is_read_only.md`: dump methods do DtoH only, no host compute affecting training
- [x] Per `feedback_no_stubs.md`: dumps wired end-to-end; no-op when env unset
- [x] Per `feedback_local_smoke_before_cluster.md`: validated on RTX 3050, no cluster submit
- [x] Per `feedback_systematic_debugging` + dispatch STOP rule: no fix applied — diagnosis only
- [x] No commit — left uncommitted for human review

### Reproduction

```bash
# Build with extended dumps + control mode
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

# Confirm PEDANTIC is applied (look for [cublas_determinism] line in log)
FOXHUNT_DETERMINISTIC=1 ./scripts/determinism-check.sh --quick
# Expected output:
#   FIRST_DIVERGENCE: step=2 leaf=checksums.mamba2_grad_w_c_l1 delta=1.1e-7
#   (or attn_q / vsn_w with different magnitude — divergence is small, noisy)

# Run with mamba2 dumps to see step 1's full post-step state EQUAL
FOXHUNT_DETERMINISTIC=1 ./scripts/determinism-check.sh --quick --debug-dump-mamba2
# Expected output (key bit):
#   === step 1 === ALL 52 buffers EQUAL
#   === step 2 === mamba2_l1/l2 weights DIVERGE
#                  mamba2 backward outputs ALL DIVERGE
#                  cuBLAS-GEMM outputs (dw_in/dw_a/dw_b) DIVERGE
#                  trunk weights/grads NOT dumped at step 2 (still need to add)

# Control: try DEFAULT_MATH (no TF32, no PEDANTIC) — same residual
FOXHUNT_DETERMINISTIC=2 ./scripts/determinism-check.sh --quick

# Per-leaf divergence trace: open `/tmp/foxhunt-determinism-a.log` /
# `b.log` and look for the `[cublas_determinism]` startup line to
# confirm which math mode was applied.
```

