# Determinism Foundation Implementation Plan

> **For agentic workers**: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans to implement this plan task-by-task. Phase 1 is currently DISPATCHED separately (coder agent af76b910). Phases 2-6 execute after Phase 1 lands. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal**: Make foxhunt ml-alpha training bit-deterministic (rel-tol 1e-5) across same-seed runs, on a per-step basis, with a permanent CI guard preventing regression. Establish a contract: every dev cluster smoke and every local mid-smoke is reproducible.

**Architecture**: Localize → Fix → Test → Toggle → Guard. Each phase has explicit acceptance criteria and produces an artifact that the next phase consumes.

**Tech stack**: CUDA kernels (sm_86/sm_89/sm_90), cudarc + cuBLAS + cuDNN, Rust 1.85, Argo Workflows, foxhunt invariant test harness.

**Linked spec**: `docs/superpowers/specs/2026-06-02-determinism-foundation.md`

---

## §0 Phase status at plan creation

- **Phase 1 (diagnose)**: DISPATCHED as background coder agent af76b910 — instrumenting 15 checksums + running diagnostic. Output: `docs/superpowers/notes/2026-06-02-determinism-phase1-diagnosis.md` identifying ONE first-divergent component.
- **Phase 2 (fix)**: this plan covers all 6 candidate causes; ship only the relevant subsection based on Phase 1 output.
- **Phase 3 (test infra)**: independent of Phase 1; can ship in parallel.
- **Phase 4 (dev/prod toggle)**: independent; ships after Phase 2 fix lands.
- **Phase 5 (CI guard)**: independent; ships after Phase 3 test infra.
- **Phase 6 (validation)**: gates merge to main; runs entire suite.

## Phase 1 — already dispatched

See coder agent af76b910 output. Once it completes, read `docs/superpowers/notes/2026-06-02-determinism-phase1-diagnosis.md` for the CULPRIT identification, then jump to the matching §2.X subsection below.

---

## Phase 2 — Apply targeted fix based on Phase 1 culprit

### §2.A — IF Phase 1 finds: encoder_output diverged first (Mamba2 culprit)

**Files**:
- Modify: `crates/ml-alpha/cuda/mamba2_selective_scan_fwd.cu`
- Modify: `crates/ml-alpha/cuda/mamba2_selective_scan_bwd.cu` (likely same fix applies)
- Possibly: `crates/ml-alpha/cuda/mamba2_block.cu`
- Test: `crates/ml-alpha/tests/mamba2_determinism.rs` (NEW)

#### §2.A.1 — Reproduce mamba2 non-determinism in isolation
- [ ] **Step 1**: Write `tests/mamba2_determinism.rs` that calls mamba2 forward kernel twice with same input + same weights + same seed → checksum compare
- [ ] **Step 2**: Run it. Verify it reproduces the non-determinism (not just a downstream effect)
- [ ] **Step 3**: If it does NOT reproduce, the bug is downstream of mamba2 — re-read Phase 1 output, the culprit is elsewhere

#### §2.A.2 — Audit chunk-parallel scan for non-deterministic ordering
- [ ] **Step 4**: Read `mamba2_selective_scan_fwd.cu` paying attention to:
  - Chunk size constant (CHUNK_SIZE)
  - Block scheduling order
  - Any `atomicAdd` (should be zero per `feedback_no_atomicadd`)
  - Any "block 0 OR block 1 writes final" patterns
  - The chunk reduction operator (associativity errors compound)
- [ ] **Step 5**: Document audit findings as comments in the file. If clean → bug is in the scan associativity, not block order.

#### §2.A.3 — Ground-truth reference with CHUNK_SIZE=1
- [ ] **Step 6**: Add a `MAMBA2_DETERMINISTIC` compile-time flag that forces CHUNK_SIZE=1 (sequential scan)
- [ ] **Step 7**: With flag ON, run determinism check — should pass (sequential is trivially deterministic)
- [ ] **Step 8**: Measure speed cost of CHUNK_SIZE=1: expect ~10× slower

#### §2.A.4 — Fix: deterministic chunk-reduction with fixed accumulation order
- [ ] **Step 9**: In the chunk reduction operator, ensure accumulation order is FIXED (not parallel-reduce with non-deterministic synchronization):
  ```cuda
  // BAD (non-det if multiple threads accumulate):
  __shared__ float acc;
  if (threadIdx.x == 0) acc = 0.0f;
  __syncthreads();
  atomicAdd(&acc, my_partial);  // ← non-det order
  
  // GOOD (sequential within block, fixed across blocks):
  __shared__ float partials[WARPS_PER_BLOCK];
  partials[warp_id] = warp_reduce(my_partial);
  __syncthreads();
  if (warp_id == 0) {
    float total = 0.0f;
    for (int i = 0; i < WARPS_PER_BLOCK; ++i) total += partials[i];
    // block 0 always writes final
  }
  ```
- [ ] **Step 10**: Apply the fix
- [ ] **Step 11**: Re-run determinism check — should pass with full chunk parallelism
- [ ] **Step 12**: Measure speed cost: expect ~1-3% (negligible)

#### §2.A.5 — Validation
- [ ] **Step 13**: `cargo test mamba2_determinism --release --ignored` passes
- [ ] **Step 14**: Full `scripts/determinism-check.sh` passes (no checksum divergence)
- [ ] **Step 15**: Commit: `fix(mamba2): deterministic chunk-reduction accumulation order`

---

### §2.B — IF Phase 1 finds: q_logits / pi_logits / v_value diverged first (cuBLAS split-K culprit)

**Files**:
- Modify: `crates/ml-alpha/src/networks/q_head.rs` (cuBLAS handle init)
- Modify: `crates/ml-alpha/src/networks/policy_head.rs`
- Modify: `crates/ml-alpha/src/networks/v_head.rs`
- Modify: `crates/ml-alpha/src/networks/dqn.rs` (if cuBLAS handle is shared)
- Modify: `crates/ml-alpha/src/networks/mamba2_block.rs` (likely same)
- Test: `crates/ml-alpha/tests/cublas_determinism.rs` (NEW)

#### §2.B.1 — Confirm TF32 enabled is the cause
- [ ] **Step 1**: Tasks 30/31 (completed) enabled TF32 on cuBLAS handles. TF32 uses split-K with non-deterministic K-split heuristic.
- [ ] **Step 2**: Audit each cuBLAS handle init site (grep for `cublasSetMathMode`)
- [ ] **Step 3**: Document current math modes:
  ```
  q_head: CUBLAS_TF32_TENSOR_OP_MATH ← tasks 30/31
  pi_head: CUBLAS_TF32_TENSOR_OP_MATH ← tasks 30/31
  mamba2_block: CUBLAS_TF32_TENSOR_OP_MATH ← tasks 30/31
  ```

#### §2.B.2 — Add deterministic mode toggle
- [ ] **Step 4**: Add helper in `crates/ml-alpha/src/cublas_init.rs` (NEW or in existing setup file):
  ```rust
  pub fn cublas_set_math_mode_deterministic(handle: cublasHandle_t) -> Result<()> {
      let pedantic = std::env::var("FOXHUNT_DETERMINISTIC")
          .unwrap_or_else(|_| "1".to_string()) == "1";
      let mode = if pedantic {
          CUBLAS_PEDANTIC_MATH  // disables TF32 split-K, FP32 strict
      } else {
          CUBLAS_TF32_TENSOR_OP_MATH  // current default after tasks 30/31
      };
      unsafe { cublasSetMathMode(handle, mode); }
      Ok(())
  }
  ```
- [ ] **Step 5**: Replace every `cublasSetMathMode(handle, CUBLAS_TF32_*)` call with `cublas_set_math_mode_deterministic(handle)`
- [ ] **Step 6**: Verify with grep: no remaining `CUBLAS_TF32_TENSOR_OP_MATH` direct calls

#### §2.B.3 — Validation
- [ ] **Step 7**: Build release. Set `FOXHUNT_DETERMINISTIC=1` and run determinism-check → should pass
- [ ] **Step 8**: Set `FOXHUNT_DETERMINISTIC=0` and re-run → should fail (sanity: the toggle works)
- [ ] **Step 9**: Measure throughput cost on RTX 3050 mid-smoke: `time scripts/local-mid-smoke.sh` with toggle ON vs OFF. Expect 5-15% slower in deterministic mode.
- [ ] **Step 10**: Document the cost in `docs/superpowers/notes/2026-06-02-determinism-speed-cost.md`

#### §2.B.4 — Commit
- [ ] **Step 11**: `fix(cublas): FOXHUNT_DETERMINISTIC env var toggles PEDANTIC vs TF32 math mode`

---

### §2.C — IF Phase 1 finds: cuDNN involved (less likely but possible)

**Files**:
- Same network files as §2.B (if they call cuDNN)
- Modify: `crates/ml-alpha/src/cudnn_init.rs` (if exists; or relevant module)

#### §2.C.1 — Check whether ml-alpha uses cuDNN at all
- [ ] **Step 1**: `grep -rn "cudnn\|CUDNN" crates/ml-alpha/src/ crates/ml-alpha/cuda/` to confirm cuDNN usage
- [ ] **Step 2**: If zero hits → cuDNN not in path → skip §2.C entirely
- [ ] **Step 3**: If cuDNN is used: apply `CUDNN_DEFAULT_MATH` + `CUDNN_DETERMINISTIC` flags via similar toggle to §2.B.2

---

### §2.D — IF Phase 1 finds: custom kernel grad/forward output diverged (block-tree audit)

**Files**:
- Audit: all `.cu` files in `crates/ml-alpha/cuda/`
- Most likely candidates: `rl_*_backward.cu` kernels, `*_reduce.cu` kernels

#### §2.D.1 — Identify the specific kernel(s)
- [ ] **Step 1**: Phase 1 narrows to e.g. `q_grad_checksum diverges first`. The Q gradient is computed by `dqn_distributional_q_backward.cu` (or similar). Identify the exact kernel.
- [ ] **Step 2**: Read the kernel for non-deterministic patterns:
  - Any atomicAdd (should be 0 per `feedback_no_atomicadd`)
  - Any "if (block_id == 0)" final-write pattern that depends on block scheduling
  - Any `volatile` memory access without proper fence
  - Any `cudaDeviceSynchronize` inside kernel (very rare)

#### §2.D.2 — Common fix patterns
- [ ] **Step 3**: If the kernel uses "first block to finish writes final" pattern, replace with:
  - Two-pass: first pass each block writes its partial to global memory, second pass single-block reads all partials in fixed order
  - OR: cooperative groups grid-wide sync (requires sm_70+, available on all foxhunt GPUs)
- [ ] **Step 4**: If the kernel reads stale values due to write/read race: add `__threadfence_system()` + counter pattern with explicit ordering
- [ ] **Step 5**: Apply the fix
- [ ] **Step 6**: Re-run determinism-check

#### §2.D.3 — Validation
- [ ] **Step 7**: GPU oracle test for the specific kernel passes bit-equivalence
- [ ] **Step 8**: Determinism-check passes
- [ ] **Step 9**: Commit: `fix(rl): deterministic block-reduction in <kernel-name>`

---

### §2.E — IF Phase 1 finds: adam_m_sum / adam_v_sum diverged (optimizer iteration order)

**Files**:
- Modify: `crates/ml-alpha/src/optimizers/adamw.rs` (or wherever Adam state is iterated)

#### §2.E.1 — Audit parameter iteration order
- [ ] **Step 1**: Find the AdamW step function. Look for parameter iteration via HashMap, Vec, or BTreeMap.
- [ ] **Step 2**: If HashMap is used (e.g., `HashMap<String, CudaSlice<f32>>`), iteration order is non-deterministic in Rust by design.
- [ ] **Step 3**: Replace HashMap with BTreeMap OR pre-sorted `Vec<(String, CudaSlice<f32>)>` at construction time. Sort by parameter name.

#### §2.E.2 — Validation
- [ ] **Step 4**: Add unit test in `crates/ml-alpha/tests/adam_determinism.rs`: build trainer, do 10 steps, dump Adam m + v, repeat, compare — should be bit-equal
- [ ] **Step 5**: Determinism-check passes
- [ ] **Step 6**: Commit: `fix(adam): deterministic parameter iteration order via BTreeMap`

---

### §2.F — IF Phase 1 finds: replay_sample_indices diverged (PER sampling)

**Files**:
- Modify: `crates/ml-alpha/cuda/rl_per_sample.cu` (or wherever PER sampling happens)
- Modify: `crates/ml-alpha/src/replay/per.rs`

#### §2.F.1 — Check PER RNG seeding
- [ ] **Step 1**: Locate the PER sample kernel. Should use device-side xorshift32 seeded from host (per `pearl_rl_sample_tau_xorshift32`)
- [ ] **Step 2**: Verify the seed is passed via ISV or kernel arg, deterministically advanced per step
- [ ] **Step 3**: If the seed source is `std::time::SystemTime::now()` or `thread_rng()` anywhere → that's the bug

#### §2.F.2 — Fix
- [ ] **Step 4**: Ensure PER seed derives from a deterministic counter (e.g., `cli.seed + step_counter`)
- [ ] **Step 5**: Verify priority update / read ordering uses CudaEvent, not stream-implicit ordering
- [ ] **Step 6**: Re-run determinism-check

#### §2.F.3 — Commit
- [ ] **Step 7**: `fix(per): deterministic sample-RNG seeded from cli.seed + step`

---

## Phase 3 — Test infrastructure (parallel to Phase 2)

### §3.1 — `tests/determinism_invariants.rs`

**Files**:
- Create: `crates/ml-alpha/tests/determinism_invariants.rs`

- [ ] **Task 1**: Write the test file with 3 tests:

```rust
//! GPU-oracle determinism invariants.
//!
//! Test 1: same seed → identical checksums (rel-tol 1e-5)
//! Test 2: different seed → different checksums (sanity check)
//! Test 3: same seed + golden checksum file → match (catches regression)

use anyhow::{Context, Result};
use serde_json::Value;
use std::path::PathBuf;
use std::process::Command;

fn binary_path() -> PathBuf { /* ... */ }
fn data_dir() -> PathBuf { /* ... */ }
fn read_jsonl(path: &Path) -> Result<Vec<Value>> { /* ... */ }
fn extract_checksums(row: &Value) -> Result<Vec<(String, f64)>> { /* ... */ }

#[test]
#[ignore = "requires CUDA + pre-built release binary + MBP-10 test data"]
fn determinism_twin_runs_match() -> Result<()> {
    let out_a = run_mid_smoke(seed=42)?;
    let out_b = run_mid_smoke(seed=42)?;
    let rows_a = read_jsonl(&out_a.join("diag.jsonl"))?;
    let rows_b = read_jsonl(&out_b.join("diag.jsonl"))?;
    for (i, (a, b)) in rows_a.iter().zip(rows_b.iter()).enumerate() {
        let ca = extract_checksums(a)?;
        let cb = extract_checksums(b)?;
        for ((ka, va), (kb, vb)) in ca.iter().zip(cb.iter()) {
            assert_eq!(ka, kb, "checksum key mismatch at row {i}: {ka} vs {kb}");
            let rel = ((va - vb) / va.abs().max(1e-12)).abs();
            anyhow::ensure!(rel < 1e-5,
                "step {i} {ka}: a={va:.6} b={vb:.6} rel_err={rel:.2e}");
        }
    }
    Ok(())
}

#[test]
#[ignore = "requires CUDA + pre-built release binary + MBP-10 test data"]
fn determinism_seed_actually_matters() -> Result<()> {
    let out_a = run_mid_smoke(seed=42)?;
    let out_b = run_mid_smoke(seed=43)?;
    let rows_a = read_jsonl(&out_a.join("diag.jsonl"))?;
    let rows_b = read_jsonl(&out_b.join("diag.jsonl"))?;
    // After 100 steps, at least one checksum should differ
    let row_100_a = &rows_a[99];
    let row_100_b = &rows_b[99];
    let ca = extract_checksums(row_100_a)?;
    let cb = extract_checksums(row_100_b)?;
    let any_differs = ca.iter().zip(cb.iter()).any(|((_, va), (_, vb))| {
        ((va - vb) / va.abs().max(1e-12)).abs() > 1e-3
    });
    anyhow::ensure!(any_differs, "different seeds produced identical results — seed not used");
    Ok(())
}

#[test]
#[ignore = "requires CUDA + pre-built release binary + MBP-10 test data + golden checksum file"]
fn determinism_golden_checksum_match() -> Result<()> {
    let out = run_mid_smoke(seed=42)?;
    let rows = read_jsonl(&out.join("diag.jsonl"))?;
    let last = &rows[rows.len() - 1];
    let observed = extract_checksums(last)?;
    let golden = read_golden("crates/ml-alpha/tests/fixtures/determinism-golden-seed42.json")?;
    for ((k, v_obs), (k_g, v_g)) in observed.iter().zip(golden.iter()) {
        assert_eq!(k, k_g);
        let rel = ((v_obs - v_g) / v_g.abs().max(1e-12)).abs();
        anyhow::ensure!(rel < 1e-5,
            "checksum {k} drift from golden: obs={v_obs:.6} gold={v_g:.6} rel_err={rel:.2e}");
    }
    Ok(())
}

fn run_mid_smoke(seed: u32) -> Result<PathBuf> { /* invoke local-mid-smoke.sh with seed */ }
fn read_golden(path: &str) -> Result<Vec<(String, f64)>> { /* parse JSON */ }
```

- [ ] **Task 2**: Generate golden checksum file `crates/ml-alpha/tests/fixtures/determinism-golden-seed42.json`:
  - After Phase 2 fix lands and determinism check passes
  - Run mid-smoke with seed=42, extract final checksums, save as golden
  - This freezes the post-fix behavior — any future regression detected

- [ ] **Task 3**: Verify tests pass on a clean checkout

---

### §3.2 — Pre-commit hook

**Files**:
- Create: `.claude/hooks/determinism-check.sh` OR `.git/hooks/pre-commit` (foxhunt's convention)

- [ ] **Task 4**: Write a fast 50-step micro-determinism-check that runs on pre-commit:
  - Detects `.cu` or `crates/ml-alpha/src/**/*.rs` changes via `git diff --cached --name-only`
  - If no relevant changes → exit 0 immediately (no overhead)
  - Otherwise: run 50-step mid-smoke twice, compare checksums
  - Exit 1 + clear error message if determinism broke

- [ ] **Task 5**: Document the hook in CLAUDE.md
- [ ] **Task 6**: Add `git commit --no-verify` escape hatch documentation (must justify in commit message)

---

### §3.3 — Cluster regression-pair workflow

**Files**:
- Modify: `scripts/argo-alpha-rl.sh`
- Modify: `infra/k8s/argo/alpha-rl-template.yaml`

- [ ] **Task 7**: Add `--regression-pair` flag to `scripts/argo-alpha-rl.sh`:
  - When set, submits TWO identical Argo workflows (same SHA, same seed, same params)
  - After both complete, compares `eval_summary.json` from both
  - Reports `total_pnl_usd` Δ in absolute and relative terms
  - Logs warning if Δ > 0.1% (true determinism = bit-exact)

- [ ] **Task 8**: Manual test: run `argo-alpha-rl.sh --regression-pair --sha <post-fix-sha>` → both runs should produce identical eval_summary

---

## Phase 4 — Dev/prod toggle

**Files**:
- Modify: `crates/ml-alpha/src/lib.rs` or `crates/ml-alpha/src/config.rs`
- Modify: `CLAUDE.md`

- [ ] **Task 1**: Add global config struct:
```rust
pub struct DeterminismConfig {
    pub enabled: bool,
}
impl DeterminismConfig {
    pub fn from_env() -> Self {
        let enabled = std::env::var("FOXHUNT_DETERMINISTIC")
            .unwrap_or_else(|_| "1".to_string())  // default ON in dev
            == "1";
        Self { enabled }
    }
}
```

- [ ] **Task 2**: Plumb through to: cuBLAS init (§2.B.2), checksum kernel (skip if disabled to save 150us/step), CHUNK_SIZE selection if applicable (§2.A.6)

- [ ] **Task 3**: Production runs (live trading, walk-forward production) set `FOXHUNT_DETERMINISTIC=0` for speed. Document in CLAUDE.md.

- [ ] **Task 4**: Add startup log: print determinism mode + expected speed cost so it's obvious in run logs which mode was active

---

## Phase 5 — CI guard

**Files**:
- Modify: `.gitlab-ci.yml` (per CLAUDE.md, legacy CI exists but disabled; if enabled now use it; otherwise add Argo CI)
- Possibly create: `infra/k8s/argo/determinism-gate-template.yaml`

- [ ] **Task 1**: Add CI job `determinism-gate`:
  - Triggers: any push to a branch with `ml-alpha-*` prefix, or any PR to main
  - Runs: full mid-smoke (b=128, 2000+500) twice with `FOXHUNT_DETERMINISTIC=1`, then determinism-check.sh
  - Fails: if any checksum divergence > rel-tol 1e-5
  - Reports: which leaf diverged, at what step

- [ ] **Task 2**: Add pre-train determinism check to `infra/k8s/argo/alpha-rl-template.yaml`:
  - New step before main training: run 50-step micro-determinism-check on cluster
  - If divergence detected → fail the workflow with clear message
  - Cost: ~30 seconds per cluster smoke; saves N×70min if it catches a regression

---

## Phase 6 — Full validation

- [ ] **Task 1**: All 3 tests in `determinism_invariants.rs` pass
- [ ] **Task 2**: `scripts/determinism-check.sh` passes for 5 different seeds: 42, 43, 44, 45, 46
- [ ] **Task 3**: `scripts/argo-alpha-rl.sh --regression-pair` produces identical eval_summary.json (Δ < $1) on one full b=1024 20k+5k cluster smoke
- [ ] **Task 4**: Speed cost documented: measured time of `local-mid-smoke.sh` with `FOXHUNT_DETERMINISTIC=1` vs `=0` on RTX 3050; same comparison on cluster L40S
- [ ] **Task 5**: Create memory pearls:
  - `pearl_determinism_root_cause`: the actual culprit + fix
  - `pearl_determinism_speed_cost`: measured throughput cost
  - `pearl_determinism_contract`: the rule for future commits
- [ ] **Task 6**: Update `MEMORY.md` index with the 3 new pearls
- [ ] **Task 7**: Atomic commit: `feat(rl): deterministic training foundation + CI gate`

---

## Phase 7 — Multi-head policy unblock

Once Phase 6 lands, the multi-head policy spec (`2026-06-02-multi-head-policy-with-r-multiple.md`) is unblocked. Implementation can proceed with confidence that:
- Same-seed runs produce identical results
- Eval pnl differences between iterations are SIGNAL not noise
- Single-seed verdicts are trustworthy (no need for 5-seed averaging)

---

## Failure modes and replanning

### Failure mode 1: Phase 1 finds multiple components diverge simultaneously
**Cause**: A single upstream bug (e.g., mamba2) cascades to all downstream components.
**Response**: Fix the upstream (likely mamba2 per §2.A). The downstream should fix itself once upstream is deterministic.

### Failure mode 2: Phase 2 fix doesn't restore determinism
**Cause**: We fixed one bug but there are multiple sources.
**Response**: Re-run Phase 1 with the partial fix applied. Identify the NEXT first-divergent component. Apply the matching §2.X. Iterate.

### Failure mode 3: Determinism in dev mode but not in prod (FOXHUNT_DETERMINISTIC=0)
**Cause**: Expected — TF32 split-K is non-deterministic by design.
**Response**: This is acceptable. Dev (where verdict matters) is deterministic; prod (where speed matters) is not. Document this.

### Failure mode 4: Speed cost is too high (>20% slowdown)
**Cause**: CHUNK_SIZE=1 mamba2 path OR aggressive cuDNN deterministic mode.
**Response**: Audit the specific bottleneck. Often a deterministic-by-construction reduction can be added without falling back to fully sequential. If unavoidable, accept the cost — verdict trust > 20% throughput.

### Failure mode 5: Phase 1 finds bug is in third-party code (cudarc / cuBLAS internals)
**Cause**: NVIDIA / cudarc library determinism issue.
**Response**: Pin the cudarc version + CUDA toolkit version. File a bug upstream. Use multi-seed averaging as workaround until library fix lands.

---

## Discipline reminders (HARD)

- `feedback_no_atomicadd` — fixes must use block-tree-reduce only, never atomicAdd
- `feedback_no_nvrtc` — all kernels pre-compiled via build.rs
- `feedback_cpu_is_read_only` — checksums computed on GPU, result read on CPU only
- `feedback_no_stubs` — every checksum, every fix, every test must run end-to-end
- `feedback_local_smoke_before_cluster` — Phase 6 task 3 (cluster regression-pair) ONLY after Phase 6 task 1+2 pass locally
- `feedback_no_partial_refactor` — when modifying a cuBLAS handle init, modify ALL of them in the same commit (no half-toggle state)
- `feedback_systematic_debugging` — find root cause before fixing; don't patch symptoms
- `feedback_going_in_circles_pattern` — if Phase 2 fixes don't converge after 3 attempts → re-audit Phase 1 diagnostic

## Acceptance criteria for the whole plan

The plan is COMPLETE when:
1. `scripts/determinism-check.sh` exit 0 on 5 different seeds locally
2. Cluster `--regression-pair` produces eval_summary Δ < $1
3. CI determinism-gate is wired and passing
4. 3 memory pearls saved + MEMORY.md indexed
5. Speed cost documented
6. Atomic commit landed on `ml-alpha-regime-observer` or main
7. Multi-head policy spec confirmed unblocked

Estimated total time across all phases: **3-5 working days** (heavily depending on which §2.X applies — mamba2 fix is largest, cuBLAS toggle is smallest).
