# Determinism Phase 2 — PER Sub-investigation + Tree-Rebuild Fix

**Date**: 2026-06-02
**Branch**: ml-alpha-regime-observer
**Spec**: `docs/superpowers/specs/2026-06-02-determinism-foundation.md` §2.F
**Plan**: `docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md` §2.F
**Phase 1 input**: `docs/superpowers/notes/2026-06-02-determinism-phase1-diagnosis.md`

## TL;DR

Phase 1 identified `checksums.replay_sample_indices` as the first
divergent leaf at step 2 with Δ=2.14e6 on the sum-of-squares.  Because
the checksum is permutation-invariant, that result alone could not
distinguish between three candidates: (A) PRNG seeding drift, (B)
tree-walk fp comparison flip, or (C) tree-rebuild non-determinism.

This sub-investigation added env-gated raw-state dumps for
`sample_prng_d`, `priority_tree_d`, and `sample_indices_d` at steps
0/1/2, then ran the determinism check twice with the dumps enabled
and exact-bytewise-compared the resulting binaries.

**Finding — HYPOTHESIS C wins**: at step 2, `prng` matches
bit-exactly, `sample_indices` matches bit-exactly (steps 0/1/2 all
match), but `priority_tree` **DIVERGES at idx 1 (the root)**:

```
step 2:
  prng: EQUAL (128 elements)
  indices: EQUAL (128 elements)
  tree: DIVERGE at idx 1
        A=592.7081298828125  B=695.7353515625
```

This is the root of the priority sum-tree.  A divergence of ~100 at
the root cannot come from sub-eps fp noise: it is a genuine
order-of-addition difference in the tree rebuild.

## Cause — `__threadfence()` is not a grid-wide barrier

`crates/ml-alpha/cuda/rl_per_tree_rebuild.cu` previously launched as
`Grid=(128), Block=(256)` and synchronised between levels with
`__threadfence()`.  `__threadfence()` is a memory-ordering primitive
(it orders THIS thread's prior writes globally), NOT a grid-wide
barrier (it does NOT wait for OTHER blocks' writes).  Once any block
finished its level-0 writes and proceeded to level 1, it could read
INTER-MEDIATE values of level-0 nodes that other blocks were still
writing.  The resulting addition order varied across runs even at the
same seed, producing different `priority_tree[1]` (root) values.

The kernel did NOT use `atomicAdd`, but determinism requires more
than that — it requires a **fixed addition order** across runs.  The
parallel grid-stride loop with `__threadfence()` did not provide it.

## Fix — Option C: single-block deterministic rebuild

Files changed (uncommitted):

- `crates/ml-alpha/cuda/rl_per_tree_rebuild.cu` (~50 LOC, full rewrite
  of the kernel body): launch geometry collapsed to `Grid=(1)`,
  `Block=(1024)`; inter-level barrier switched from `__threadfence()`
  to `__syncthreads()` (a proper intra-block barrier).  Each thread
  processes nodes via a block-stride loop with a FIXED thread→node
  mapping (`i = tid + k*blockDim.x`), so every same-seed run uses
  identical addition orderings.  Header comment documents the
  determinism rationale, the speed analysis, and the upgrade path to
  `cooperative_groups::grid().sync()` if capacity ever grows past 2^21.

- `crates/ml-alpha/src/trainer/integrated.rs` (~8 LOC in the launch
  site at the K-loop body): updated the `raw_launch` config from
  `(128,1,1)/(256,1,1)` to `(1,1,1)/(1024,1,1)` with an inline
  comment pointing at the kernel header for the determinism rationale.

Diagnostic infrastructure (new files):

- `crates/ml-alpha/src/trainer/integrated.rs`
  `IntegratedTrainer::dump_per_state_for_debug` — env-gated method
  triggered from `build_diag_value` at steps 0/1/2.  Reads
  `FOXHUNT_DETERMINISM_DEBUG=1` to enable, dumps to
  `$FOXHUNT_DEBUG_DUMP_DIR` (default `/tmp/foxhunt-determinism-debug`).
  No-op in normal runs.

- `scripts/determinism-check.sh` `--debug-dump` flag: runs both A/B
  with the env vars set, then invokes `compare-per-state.py`.

- `scripts/compare-per-state.py` (NEW, 200 LOC): reads the raw dumps,
  exact-byte-compares prng / tree / indices, and prints a verdict
  identifying which hypothesis (A / B / C) is in play.

## Validation

### Task 3 — `scripts/determinism-check.sh --quick`

Post-fix output (full 200-step quick run):

```
FIRST_DIVERGENCE: step=3 leaf=checksums.encoder_output
                  run_A=6896.334218005304 run_B=6896.194631539084
                  delta=0.139586

All leaves diverging at step=3:
  checksums.encoder_output  A=6896.334...  B=6896.194...  Δ=0.139586
  checksums.v_value         A=0.23311...   B=0.23318...   Δ=6.59e-05
  checksums.v_grad          A=0.09886...   B=0.09888...   Δ=1.74e-05
```

PER is **no longer the first-divergent component**.  The next bug
surfaces at step 3 in `encoder_output` (mamba2 selective scan, per
spec §2.A).  Per dispatch instructions: documenting and stopping —
this is a separate Phase 2.2 dispatch.

The debug-dump run also confirms PER state is bit-equal at steps
0/1/2:

```
step 2:
  prng: EQUAL (128 elements)
  indices: EQUAL (128 elements)
  tree: EQUAL (65536 elements)
VERDICT: all three buffers match across all dumped steps.
```

### Speed cost

Quick mid-smoke timing on RTX 3050 (200 train + 50 eval, b=128):

| Phase | Run A | Run B |
|-------|-------|-------|
| Pre-fix (Grid=128, Block=256, __threadfence) | 0m 45s | 0m 44s |
| Post-fix (Grid=1, Block=1024, __syncthreads) | 0m 43s | 0m 43s |

**No measurable slowdown.**  The single-block rebuild touches ~65k
floats (15 tree levels), each thread doing ~64 ops, completing in a
few hundred μs.  The kernel launch overhead (one launch per K-loop
iter) is unchanged.  If anything, the single-block variant slightly
reduces launch-config overhead.

If the post-fix run is faster, the difference is within run-to-run
variance.  Confirmed safe to land.

### Acceptance criteria from §2.F

- [x] Phase 2.F.1 — PER RNG seeding check: confirmed deterministic
      (prng-equal dumps).
- [x] Phase 2.F.2 — Fix applied (tree-rebuild kernel + launch site).
- [x] Phase 2.F.3 — `determinism-check.sh` no longer reports
      `replay_sample_indices` as the first divergent leaf.  The next
      first-divergent leaf is `encoder_output` (separate Phase 2.A
      issue, mamba2 selective scan).

## Files touched (uncommitted)

| Path | Change | LOC |
|------|--------|-----|
| `crates/ml-alpha/cuda/rl_per_tree_rebuild.cu` | Single-block deterministic rebuild | +60 / -34 |
| `crates/ml-alpha/src/trainer/integrated.rs` | (a) launch geometry change (b) `dump_per_state_for_debug` method (c) call from `build_diag_value` | +120 / -2 |
| `scripts/determinism-check.sh` | `--debug-dump` flag + env-var plumbing | +35 / -5 |
| `scripts/compare-per-state.py` (NEW) | Diff PRNG / tree / indices, hypothesis verdict | +200 |
| `docs/superpowers/notes/2026-06-02-determinism-phase2-per-investigation.md` (NEW) | this note | new |

## Discipline checklist

- [x] No new files in repo root
- [x] No new docs outside `docs/superpowers/notes/`
- [x] Per `feedback_no_atomicadd`: kernel uses no atomics (block-wide
      barrier + fixed-order writes).
- [x] Per `feedback_no_nvrtc`: kernel registered in `build.rs` (no
      change required — already AOT compiled, just the source body
      changed).
- [x] Per `feedback_cpu_is_read_only`: the debug-dump method does a
      DtoH copy after a stream sync; this is the standard host-read
      pattern, no host-side compute.
- [x] Per `feedback_no_stubs`: dump is wired end-to-end, runs only
      when env-gate is set, otherwise zero overhead.
- [x] Per `feedback_local_smoke_before_cluster`: validated on RTX
      3050 with the quick determinism check.  No cluster submission.
- [x] Per `feedback_no_partial_refactor`: there is only ONE launch
      site for `rl_per_tree_rebuild` (verified via grep); updated in
      lockstep with the kernel.
- [x] Per `feedback_no_quickfixes`: chose the principled fix
      (deterministic-by-construction single-block) over a cheap epsilon
      shift; this is the same pattern the spec §1.1 already
      recommends for the deterministic checksum kernel.
- [x] No commit — left uncommitted for human review per dispatch.

## Next step — Phase 2.2 dispatch for `encoder_output`

The `encoder_output` divergence at step 3 (Δ=0.14 on a sum-of-squares
of ~6896) corresponds to mamba2 selective scan, per spec §2.A.  This
is a separate sub-investigation; the Phase 1 infrastructure (15-leaf
checksums + scripts/determinism-check.sh) already supports localising
it.  Recommend dispatching a Phase 2.2 worker on §2.A.

## Reproduction

```bash
# Build (idempotent, sccache helps)
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

# Validate the fix (no debug dumps)
./scripts/determinism-check.sh --quick

# Re-run the sub-investigation (env-gated dumps at steps 0/1/2)
./scripts/determinism-check.sh --quick --debug-dump
```
