# Determinism Phase 1 — Diagnosis

**Date**: 2026-06-02
**Branch**: ml-alpha-adaptive-controller-floors
**Spec**: `docs/superpowers/specs/2026-06-02-determinism-foundation.md` §1
**Status**: Phase 1 SHIPPED — diagnosis complete, fix is Phase 2 (separate dispatch).

## TL;DR

`scripts/determinism-check.sh --quick` (200 train + 50 eval at b=128, seed=42) on
`ml-alpha-adaptive-controller-floors` produces:

```
FIRST_DIVERGENCE: step=2 leaf=checksums.replay_sample_indices
                  run_A=709847.0 run_B=2851599.0 delta=2.14175e+06
```

**Culprit**: PER (Prioritized Experience Replay) sampling. Spec §2.F is the fix.

Steps 0-1 are bit-equivalent across runs. Divergence appears at step 2 the
moment the replay buffer transitions from "trivially deterministic" (small,
all entries equally probable) into "PER-sampled with priority weights".

## Phase 1 deliverables

1. **Kernel** — `crates/ml-alpha/cuda/rl_deterministic_checksum.cu`
   - Two entry points: `rl_deterministic_checksum_f32` + `rl_deterministic_checksum_i32`
   - Both are `(1,1,1)/(1,1,1)` launches — a single thread sums `data[i]²`
     into a double-precision accumulator. NO atomics, NO multi-block reduce,
     NO non-deterministic patterns by construction. Per `feedback_no_atomicadd`.
   - Cost per launch: ~5-20 μs launch overhead + sequential f64 accumulation
     (~1ns per element on RTX 3050). Largest tensor we checksum is the
     Q-head weight `ss_q_grad_w_d` at ~24k floats — 24 μs sequential.
   - Registered in `crates/ml-alpha/build.rs` (AOT cubin compile per
     `feedback_no_nvrtc`).

2. **Diag emission** — `crates/ml-alpha/src/trainer/integrated.rs`
   - New method `IntegratedTrainer::compute_diag_checksums(&mut self) → Result<[f64; 15]>`
   - Allocates a 28-f64 scratch buffer (`checksum_scratch_d`) once at
     trainer construction; 15 final-leaf slots + 12 Adam staging slots
     (the optimizer's m and v live in six separate sub-buffers for the
     DQN/π/V w+b parameter groups; host-folded into adam_m_sum / adam_v_sum
     after the dtoh).
   - `build_diag_value` is now `&mut self`; it calls `compute_diag_checksums`
     up-front and inserts the 15 leaves under a new top-level `checksums`
     JSON object.
   - 15 leaves emitted per spec §1.2: `encoder_output`, `q_logits`,
     `pi_logits`, `v_value`, `replay_sample_indices`, `rewards_after_shape`,
     `advantages`, `q_grad`, `pi_grad`, `v_grad`, `encoder_grad`,
     `adam_m_sum`, `adam_v_sum`, `isv_state`, `popart_sigma_state`.
   - `tests/eval_diag_emission.rs` `EXPECTED_LEAVES` bumped 679 → 694.
   - Call-site fix in `crates/ml-alpha/examples/alpha_rl_train.rs`: the
     train-loop `let isv = trainer.isv_host_slice()` borrow conflicted with
     the new `&mut self` on `build_diag_value` because the slice was
     referenced past the call in an `eprintln!`. Fix is local — pre-extract
     the 4 ISV values into `f32` locals before the diag call. Eval loop
     was unaffected (NLL drops the borrow before the diag call there).

3. **Diagnostic** — `scripts/determinism-check.sh` rewritten
   - Iterates over EVERY row of `diag.jsonl` (was: last 5 rows)
   - Compares only the `checksums.*` subtree leaf-by-leaf (other diag
     fields are derived quantities; localising to component checksums
     gives a cleaner signal)
   - Reports first `(step, leaf)` divergence at rel-tol=1e-5 + abs-tol=1e-7
   - Also lists ALL leaves diverging at the first divergent step (multiple
     components frequently co-diverge — the script ranks them by Δ
     magnitude so the top entry is the most-likely root cause)
   - Maps each known leaf to its spec §2 fix recommendation
   - Exit 0 = deterministic / 1 = drift detected / 2 = inputs missing

## Diagnostic findings — step-by-step

Output of `./scripts/determinism-check.sh --quick`:

| Step | Diverging leaves | Top Δ |
|------|------------------|-------|
| 0 | 0 (bit-equivalent) | — |
| 1 | 0 (bit-equivalent) | — |
| 2 | 5 simultaneously: **`replay_sample_indices`**, `q_grad`, `adam_m_sum`, `q_logits`, `adam_v_sum` | 2.14e6 on `replay_sample_indices` |
| 3 | 11 (drift cascades through forward + grads + ISV) | — |
| 4-199 | 10-15 leaves drift on every step | — |

Total divergent rows: 198 / 200.

## Interpretation

The cascade ordering is structurally clean:

1. **Step 0-1**: PER buffer has not accumulated enough transitions for
   priority-weighted sampling to actually choose different indices (or
   the priorities are still degenerate). Both runs produce identical
   indices, identical sampled `h_t`/`h_tp1`, identical Q forward, identical
   gradients, identical Adam state. **Bit-equivalent**.

2. **Step 2 — divergence enters via `replay_sample_indices`**. The two
   runs choose different replay transitions to sample. From this moment
   the Q gradient (which depends on the sampled batch) diverges, which
   immediately diverges Q-head Adam m+v, which on the same step diverges
   `q_logits` (via the immediately-following Q forward in the next step's
   prefill). The 5-simultaneous divergence at step 2 is consistent with
   a single root cause (PER sampling) propagating to all downstream
   buffers in the same training step.

3. **Step 3+**: now divergent gradients have been Adam-applied to
   parameters → encoder forward diverges (`encoder_output`), π and V
   forward diverge, advantages diverge, controllers' EMA inputs diverge,
   ISV state diverges. Cascade complete.

The fact that **`replay_sample_indices` is the FIRST and LARGEST-magnitude
divergence at step 2**, while every other field is bit-equivalent up
through step 1, is decisive: the bug is in the PER sampling kernel chain.

## Candidate root causes (Phase 2 investigation scope)

Spec §2.F lists the entry points to audit:

1. **`rl_per_sample` kernel** — stratified proportional sampling via
   sum-tree walk + gather. The walk uses a device-side xorshift32 PRNG
   (`gpu_replay.sample_prng_d`). Audit whether:
   - The PRNG state is seeded deterministically and not reset between
     runs (must use `cli.seed.wrapping_add(...)` per the existing CPU-side
     audit in items 1-5 impl log).
   - The kernel's launch configuration (Grid=B) reads state in a
     deterministic order (each block owns one batch slot → fixed).
   - The sum-tree walk converges deterministically — IEEE-754 sum-tree
     comparisons can produce different paths if the tree values were
     accumulated in a non-deterministic order (PER push_flush uses
     prefix-sum allocation, which is non-deterministic on parallel
     warps if the input order varies).

2. **`rl_per_push_flush` and `rl_per_tree_rebuild`** — coordinated
   coalesced replay writes + bottom-up parallel sum-tree rebuild. Both
   "have no atomics" per their build.rs comments, but the rebuild's
   bottom-up parallelism may produce different floating-point sums
   if the tree leaf order differs across runs. Audit: does the tree
   rebuild deterministically reduce children before parents?

3. **`rl_per_update_priority`** — writes `|TD|^α` to tree leaves + block-
   wide max reduction. The block-wide max should be deterministic, but
   any precision-sensitive aggregation could be the culprit. Less likely.

4. **Priority update sequencing** — `pearl_rl_sample_tau_xorshift32` notes
   the PRNG seeding pattern. Verify the seed reaches `sample_prng_d` AND
   is not perturbed by any unrelated state at construction time.

## Recommendation for Phase 2

Apply **spec §2.F** (PER sampling determinism audit). Concrete steps:

1. Add per-batch dumps of `sample_prng_d`, `priority_tree_d`, and
   `sample_indices_d` at step 0, 1, 2 of two same-seed runs. Compare
   PRNG state at step 1's exit / step 2's entry — if PRNG already
   diverges → fix the seeding; if PRNG matches but indices differ →
   the priority-tree walk is the culprit.
2. If priority tree is suspect, audit `rl_per_tree_rebuild` for any
   parallel reduction whose addition order depends on warp scheduling.
   The "no atomics" claim doesn't preclude order-of-addition variance
   in a parallel scan.
3. Apply the deterministic-by-construction fix (single-block reduction
   for small trees, or fixed-warp ordering for large trees).
4. Re-run `scripts/determinism-check.sh --quick` after the fix. Success
   = "DETERMINISTIC: all `checksums.*` leaves match across all 200 rows".

Once PER is fixed, the next divergence (if any) localises automatically.
Phase 1 infrastructure is reusable — Phase 2 just runs the same script.

## Files touched (Phase 1, no commits per dispatch instruction)

| Path | Change |
|------|--------|
| `crates/ml-alpha/cuda/rl_deterministic_checksum.cu` (NEW) | Sum-of-squares kernel, two entry points (f32 + i32) |
| `crates/ml-alpha/build.rs` | Registered the new kernel in `KERNELS` |
| `crates/ml-alpha/src/trainer/optim.rs` | Added immutable `m()` / `v()` accessors on `AdamW` |
| `crates/ml-alpha/src/trainer/integrated.rs` | New cubin const + struct fields, module loading in `new()`, `compute_diag_checksums()` method, `build_diag_value` switched to `&mut self` + emits `checksums` JSON object |
| `crates/ml-alpha/examples/alpha_rl_train.rs` | Pre-extracted ISV values into locals to release the `isv_host_slice()` borrow before calling `build_diag_value` (now `&mut self`) |
| `crates/ml-alpha/tests/eval_diag_emission.rs` | `EXPECTED_LEAVES` 679 → 694 |
| `scripts/determinism-check.sh` | Rewritten to compare all rows × `checksums.*` leaves; reports FIRST_DIVERGENCE + spec §2 fix recommendation |
| `docs/superpowers/notes/2026-06-02-determinism-phase1-diagnosis.md` (NEW, this file) | Diagnosis log |

## Discipline checklist

- [x] No new files in repo root
- [x] No new docs outside `docs/superpowers/notes/`
- [x] CUDA kernel registered in `build.rs` per `feedback_no_nvrtc`
- [x] Kernel uses NO atomics per `feedback_no_atomicadd`
- [x] Result read from CPU only; computation entirely on device per `feedback_cpu_is_read_only`
- [x] All 15 checksums end-to-end wired — no stubs per `feedback_no_stubs`
- [x] Local RTX 3050 smoke validation only — NO cluster submission
- [x] Read-before-Edit on all touched existing files
- [x] No fix applied — diagnosis only, per dispatch instruction (Phase 2 is separate)
- [x] No commit — left uncommitted for human review

## Reproduction

```bash
# Build
SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha

# Run
./scripts/determinism-check.sh --quick

# Expected output (current branch ml-alpha-adaptive-controller-floors):
# FIRST_DIVERGENCE: step=2 leaf=checksums.replay_sample_indices
#                   run_A=709847.0 run_B=2851599.0 delta=2.14175e+06
```
