# Fast Dev Cycle — Items 1-5 Implementation Log

**Date**: 2026-06-02
**Branch**: ml-alpha-adaptive-controller-floors
**Spec**: `docs/superpowers/specs/2026-06-02-fast-dev-cycle.md` §3.1 - §3.6 (items 1-5 only)
**Out of scope**: §3.4 async double-buffer checkpoint (item 6, separate effort)
**Status**: Items 1-5 SHIPPED. Item 3 surfaced a critical determinism break — see §3 below.

## Summary

Built a working Tier 1.5 mid-smoke funnel that gates 80%+ of bad hypotheses before they reach cluster:

- `test_data/futures-baseline-mid/ES.FUT/` — symlinked train+eval files (Q1 + Q2 ES MBP-10, 19 GB resolved, 4 KB on-disk symlinks)
- `scripts/local-mid-smoke.sh` — full mid-smoke runner (5m 51s wall-clock on RTX 3050 Ti at b=128 / 2000 train + 500 eval)
- `scripts/tier1_5_verdict.py` — behavioral kill verdict (7 signals + per-step/eval_summary consistency check, stdlib-only)
- `scripts/determinism-check.sh` — reproducibility self-test (runs mid-smoke twice with same seed, diffs final 5 rows)
- `CLAUDE.md` — documented under "Build & Test" with three-tier funnel table
- `.gitignore` — added `test_data/futures-baseline-mid/`

**Verdict tooling works end-to-end on real data** — first run flagged `KILL_popart.sigma_CV+eval_pnl` AND surfaced a $4.5M per-step/eval_summary discrepancy ($517k vs -$4M), matching the canonical `pearl_grwwh_eval_catastrophic_collapse` failure mode.

**CRITICAL FINDING (item 3)**: trainer is NOT deterministic. Two same-seed runs diverge by step 2 in `grad_norm_ema.q` (Δ=4.5), and by end of 200 steps:
- Train pnl diverges $33,632 vs $32,225 (Δ=$1,406)
- Eval pnl diverges $0 vs $364,925 (Δ=$364,925)
- Eval trade count diverges 0 vs 347

Most CPU-side seeding looks correct on audit. Root cause is GPU-side non-determinism (likely cuDNN heuristics or non-deterministic CUDA reductions). Fix is out of scope for this spec.

## Item 1 — §3.1 PVC data sync

### Plan deviation

Spec called for "rsync 2-3 highest-value MBP-10 files to local `test_data/futures-baseline-mid/` (~1.5 GB)". Actual reality:

- PVC path is `/training-data/futures-baseline-mbp10/ES.FUT` via `training-data-pvc` (NOT `/feature-cache/futures-baseline-mbp10` per spec — feature-cache-pvc only has predecoded sidecars)
- Smallest MBP-10 file is 2024-Q1 at 4.7 GB. The "~1.5 GB total" spec estimate was wrong by 10×
- User already had `test_data/futures-baseline-mbp10/ES.FUT/{Q1, Q2}.dbn.zst` locally (verified SHA256 match against PVC)

### What shipped

1. Created `test_data/futures-baseline-mid/ES.FUT/` containing two symlinks (zero disk cost, 19 GB resolved):
   - `ES.FUT_2024-Q1.dbn.zst` → `../../futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q1.dbn.zst` (4.8 GB train)
   - `ES.FUT_2024-Q2.dbn.zst` → `../../futures-baseline-mbp10/ES.FUT/ES.FUT_2024-Q2.dbn.zst` (14 GB eval)
2. SHA256 verified against PVC: `9ba3bcc468ccf148d8341cd71cec008ae2532676b0f4bb8c238b0e61c5aca625` (Q1, matches PVC)
3. `.gitignore`: added `test_data/futures-baseline-mid/`
4. `CLAUDE.md`: updated "Build & Test" section with tiered-validation table and the three new scripts

### Walk-forward fold scheme

With 2 files and `--n-folds 2 --fold-idx 0`, the trainer splits:
- block_size = 1 (files / folds)
- train = files[0..1] = Q1
- eval = files[1..2] = Q2

This gives real walk-forward semantics (no train/eval overlap), matches `--mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT` invocation pattern from `infra/k8s/argo/alpha-rl-template.yaml:181`.

## Item 2 — §3.2 Local mid-smoke script

### What shipped

`scripts/local-mid-smoke.sh` (executable, 152 lines):
- Argparse for `--seed`, `--out`, `--n-backtests`, `--n-steps`, `--n-eval-steps`, `--n-folds`, `--fold-idx`
- Defaults: seed=42, b=128, 2000 train + 500 eval, fold 0 of 2
- Builds via `SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha`
- Invokes binary with `--mbp10-data-dir test_data/futures-baseline-mid/ES.FUT --instrument-mode front-month --log-every 200 --gpu-idx 0`
- Pre-flight checks: directory exists, ≥ 2 MBP-10 files, follows symlinks via `find -L`
- Wall-clock reporting + next-step hint pointing to verdict script

### Measured wall-clock on RTX 3050 Ti

| Configuration | Wall-clock |
|---|---|
| Default (b=128, 2000+500) | **5m 51s** |
| Sanity check (b=16, 200+50) | 4m 45s (dominated by 210s eval-file load) |

GPU memory used: 1.2 GB / 4 GB (b=128) — well within RTX 3050 Ti capacity per spec §2.

### Path correction during shake-down

Initial draft passed `--mbp10-data-dir` at the parent of `ES.FUT/`. The `discover_mbp10_files_sorted` (`crates/ml-alpha/src/data/loader.rs:294`) is non-recursive — it expects the path to be the instrument's `.dbn.zst` directory directly. Fixed to match argo template (`--mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT`).

## Item 3 — §3.5 Determinism audit

### CPU-side seeding audit (all clean)

Every randomness entry point in `crates/ml-alpha/src/` derives from `cli.seed`:

| Source | Seeded by | Location |
|---|---|---|
| Loader random-anchor RNG (`MultiHorizonLoader.rng`) | `ChaCha8Rng::seed_from_u64(cfg.seed)` | `crates/ml-alpha/src/data/loader.rs:422` |
| `upload_to_gpu` seed | `cli.seed` arg | `crates/ml-alpha/examples/alpha_rl_train.rs:391` |
| DQN trainer seed | `cli.seed.wrapping_add(0xDA)` | `crates/ml-alpha/examples/alpha_rl_train.rs:410` |
| PPO trainer seed | `cli.seed.wrapping_add(0xBE)` | `crates/ml-alpha/examples/alpha_rl_train.rs:411` |
| PER buffer seed | `cli.seed.wrapping_add(0x9E37_79B9)` | `crates/ml-alpha/examples/alpha_rl_train.rs:413` |
| Eval-phase loader | `cli.seed.wrapping_add(0xE7AE)` | `crates/ml-alpha/examples/alpha_rl_train.rs:786` |
| Device-side Thompson xorshift32 | `ChaCha8Rng::seed_from_u64(cfg.dqn_seed.wrapping_add(0xC0_C0_C0_C0))` | `crates/ml-alpha/src/trainer/integrated.rs:2142` |
| Loader explicitly does NOT shuffle | `// (no shuffle — file order is the caller's…)` | `crates/ml-alpha/src/data/loader.rs:412` |

No Python data loader exists — entirely Rust. No `thread_rng()` / `random()` / `SystemTime::now()` calls in `ml-alpha` hot path. `ml-ppo` and `ml-core` use `thread_rng()` but those are not on the RL training path.

### Self-test result: DETERMINISM IS BROKEN

`scripts/determinism-check.sh --quick` (200 train + 50 eval, seed=42):

```
FIRST NON-TIME DIVERGENCE at row 2 step=2: 4 drifts
  grad_norm_ema.q                          A=64.80356  B=60.30021  Δ=4.50335
  risk_stack.atom_calibration.target_bot_saturation_rate
                                           A=0.14137   B=0.15253   Δ=0.01116
  isv_ema_in.td_kurtosis                   A=0.00092   B=0.00093   Δ=9.59e-06
  controller_branch.per_alpha_input        A=0.00092   B=0.00093   Δ=9.59e-06
```

By step 199:
- 654 leaves diverging
- `trading.realized_pnl_cum_usd`: $33,631 vs $32,225 (Δ=$1,406)
- `confidence_gate.gated_count_total`: 4826 vs 495 (~10× difference)
- `units.trail_distance[*]` differs across most of the 128 batch slots
- `risk_stack.regime.tail.session_pnl_variance_ema`: 261 vs 1008 (~4× difference)

Eval (50 steps) shows much larger split:
- Run A: 0 trades, $0 pnl, 0 win_rate
- Run B: 347 trades, $364,925 pnl, 0.343 win_rate

### Inferred root cause (not fixed per spec)

CPU-side seeding is comprehensive. The drift originates GPU-side, candidates:

1. **Non-deterministic CUDA reductions** — block-tree reductions on different launch geometries can produce fp32 differences in addition order even with identical inputs. Compounding through ~200 Adam steps yields macroscopic divergence.
2. **cuDNN heuristic auto-tune** — mamba2 forward / attention kernels may run different algorithm selections on first call vs warm-cache.
3. **Stream ordering** — multiple CUDA streams (`prefill_graph`, `postfill_graph`, `reward_graph`, `training_graph`) interleave in arbitrary order at sync boundaries.

The `grad_norm_ema.q` divergence at step 2 (Δ=4.5 on a value of ~64) is much larger than fp32 noise (1e-5), so this is structural, not roundoff. Inspection scope is a separate spec — flagged here for the next session.

### Determinism-check script behavior

`scripts/determinism-check.sh`:
- Default 2000+500 (matches mid-smoke). `--quick` → 200+50 for faster diagnostic.
- Diffs last 5 rows of `diag.jsonl` leaf-by-leaf with rel-tol=1e-5, abs-tol=1e-7
- Skips `elapsed_s` / `sps` / wall-clock signals (inherently non-deterministic)
- Reports top 10 drifts per row sorted by absolute magnitude
- Exit 0 = deterministic, 1 = drift, 2 = run failure

## Item 4 — §3.3 Behavioral kill verdict script

### What shipped

`scripts/tier1_5_verdict.py` (executable, ~430 lines, Python 3 stdlib only — no pandas/numpy):

7 behavioral signals + 1 consistency check, each with explicit KILL/WARN/OK thresholds:

| Signal | Source field | KILL | Target | Falsifies |
|---|---|---|---|---|
| `action_entropy` | last train row | <0.6 or >2.3 | [1.20, 2.04] | policy collapse / random |
| `q_pi_agree_ema` | last train row | <0.3 | ≥0.6 | Q/π decoupled |
| `pearson(rewards.sum, Δpnl)` | full train trace | <0.3 | ≥0.5 | `pearl_reward_signal_anti_aligned_with_pnl` |
| `avg_hold_steps trend` | last 1500 train rows (linreg slope) | slope<-0.01 | slope>0 | surfer pattern absent |
| `popart.sigma CV` | full train trace | CV>0.5 | CV<0.5 | controller wildly oscillating |
| `wr_train` (Kelly EMA) | last train row | <0.15 | ≥0.25 | random or worse |
| `wr_eval` (Kelly EMA) | last eval row | <0.15 | ≥0.20 | random or worse |
| `eval_pnl` | eval_summary.total_pnl_usd | <-$3M | >-$1M | catastrophic |

Output modes: human-readable (default), `--quiet` (one-line verdict), `--json` (machine-readable).

Exit codes: 0=OK, 1=KILL, 2=input missing/insufficient.

### Sample output on real mid-smoke

Ran on `/tmp/foxhunt-mid-smoke` (full b=128, 2000+500, seed=42):

```
========================================================================
Tier 1.5 verdict: KILL_popart.sigma_CV+eval_pnl_(eval_summary.total_pnl_usd)
========================================================================
  train rows:        2000
  eval rows:         500
  eval_summary:      present

STATUS  SIGNAL                                      EXPLANATION
------  ------------------------------------------  ----------------
OK      action_entropy                              1.755 within target [1.20, 2.04]
OK      q_pi_agree_ema                              0.963 ≥ 0.6
WARN    pearson(rewards.sum, Δpnl)                  0.318 below target ≥ 0.5 (n=1999)
OK      avg_hold_steps trend                        slope +0.00264/step over last 1500 rows
KILL    popart.sigma CV                             0.810 > 0.5 — controller wildly oscillating
OK      wr_train (trading.win_rate)                 0.341 ≥ 0.25
OK      wr_eval (trading.win_rate)                  0.336 ≥ 0.2
KILL    eval_pnl (eval_summary.total_pnl_usd)       $-3,996,663 < $-3,000,000 — catastrophic
WARN    per-step vs eval_summary                    per_step=$517,914 | eval_summary=$-3,996,663 | gap=$4,514,576 (112.96%) > 5% — accounting axis discrepancy

[KILL] KILL_popart.sigma_CV+eval_pnl_(eval_summary.total_pnl_usd) — DO NOT submit cluster smoke. Fix locally first.
```

The verdict caught **two structural problems** that would have wasted a 70-min cluster submit AND surfaced the known per-step/eval_summary disagreement — full payoff matrix.

### Notes on signal sourcing

- `wr_*` prefers `isv_config.kelly.win_rate_ema` (cumulative EMA across all trades) but falls back to `trading.win_rate` (point measure of last batch) when EMA path is absent. Source is reported in the signal name.
- `pearson(rewards.sum, Δpnl)` pairs `rewards.sum` (the gradient signal) with `Δ(trading.realized_pnl_cum_usd)` (the actual pnl direction). Per `pearl_reward_signal_anti_aligned_with_pnl`, this is the load-bearing signal — a sign-flipped or noisy correlation explains negative-eval hysteresis.
- `avg_hold_steps` is computed by linear regression on `(step, trading.avg_hold_steps)` over the trailing 1500-step window. Robust to NaN — drops rows where the field is non-finite.

## Item 5 — §3.6 Per-step / eval_summary consistency

### What shipped

Folded into `tier1_5_verdict.py` as `signal_consistency()`:
- Reads last `eval_diag.jsonl` row → `trading.realized_pnl_cum_usd`
- Reads `eval_summary.json` → `total_pnl_usd`
- Computes `|per_step − eval_summary| / max(|per_step|, |eval_summary|, $1)`
- WARN if fraction > 5% (NOT KILL — gives signal without blocking, per spec)

### Sample finding on the real mid-smoke

```
WARN    per-step vs eval_summary  per_step=$517,914 | eval_summary=$-3,996,663 | gap=$4,514,576 (112.96%) > 5% — accounting axis discrepancy
```

This is the same accounting-axis bug pattern as `pearl_grwwh_eval_catastrophic_collapse` ($234M gap at grwwh step 4863). The per-step diag's `trading.realized_pnl_cum_usd` and the eval_summary's `total_pnl_usd` are computed from **different** sources (per-step running sum of `raw_rewards` × b_size vs eval_summary's aggregation of LobSim closed-trade records). A 112% gap means the per-step diag is materially misleading for eval-period pnl observability.

Per spec §3.6 long-term action: "fix the accounting axis discrepancy in the diag/summary code" — out of scope for items 1-5, flagged for follow-up spec.

## Files added / changed

| Path | Change |
|---|---|
| `test_data/futures-baseline-mid/ES.FUT/` (NEW) | 2 symlinks (Q1, Q2) into existing test_data |
| `scripts/local-mid-smoke.sh` (NEW, 152 lines, +x) | Tier 1.5 runner |
| `scripts/tier1_5_verdict.py` (NEW, ~430 lines, +x) | Behavioral verdict + consistency check |
| `scripts/determinism-check.sh` (NEW, ~125 lines, +x) | Reproducibility self-test |
| `.gitignore` | Added `test_data/futures-baseline-mid/` |
| `CLAUDE.md` | Added "Tiered local validation" subsection under Build & Test |

No CUDA / Rust / training-math changes. Pure dev infrastructure. No commits — left uncommitted for human review.

## Discipline checklist

- [x] No files saved to repo root
- [x] No new *.md files except this requested impl log
- [x] Read before Edit on all touched existing files (`.gitignore`, `CLAUDE.md`)
- [x] No stubs, no TODO/FIXME/XXX markers
- [x] Symlinks chosen over duplicate-data download (per `feedback_no_quickfixes` — leverage existing canonical files)
- [x] PVC SHA256 verification before deciding local files are usable
- [x] Verdict signals all tied to existing memory entries (pearl_grwwh, pearl_reward_signal_anti_aligned, pearl_edge_lives_at_wave_timescale)
- [x] Per-step/eval_summary check emits WARN not KILL (per spec §3.6: "WARNING — gives signal without blocking")

## Recommended next steps (for next session)

1. **Determinism fix spec** — `grad_norm_ema.q` diverges at step 2. Inspect block-tree reduction ordering and stream synchronization in the encoder grad-norm path. The fact that eval pnl can differ by $365k across same-seed runs makes ALL controller-iteration verdicts noisy.
2. **Per-step/eval_summary reconciliation spec** — the $4.5M gap on this small smoke is the same shape as `pearl_grwwh_eval_catastrophic_collapse` $234M. Audit the LobSim trade accounting vs the per-step `raw_rewards` axis.
3. **Item 6 — async checkpoint** — separate ~4-day effort per spec §3.4. Unblocks resume-from-baseline + saves ~58 min per cluster smoke.
4. **Tighten Pearson threshold** — current run shows 0.318 (WARN, below 0.5 target). Confirm whether this is the established baseline or whether the mid-smoke window is too short for the signal to settle.

## Reproduction commands

```bash
# Setup (one-time; symlinks already in place from this session)
ls test_data/futures-baseline-mid/ES.FUT/    # should show 2 symlinks

# Run mid-smoke (5-6 min on RTX 3050 Ti)
./scripts/local-mid-smoke.sh

# Verdict
python3 scripts/tier1_5_verdict.py /tmp/foxhunt-mid-smoke

# Determinism self-test (~10 min, --quick for ~6 min)
./scripts/determinism-check.sh --quick
```
