# Fast Dev Cycle — Closing the 700× Gap

**Date**: 2026-06-02
**Motivation**: today's session burned 4 cluster smokes × ~70min each on controller iterations that could have been killed in ~10min locally. Per `feedback_going_in_circles_pattern`: "3+ consecutive cluster runs each surfacing a new failure introduced by the prior fix" — the going-in-circles pattern is partially a CONSEQUENCE of slow dev cycle, not a separate root cause.
**Linked**: `feedback_local_smoke_before_cluster`, `feedback_kill_runs_on_anomaly_quickly`, `pearl_grwwh_eval_catastrophic_collapse`

## §0. The current gap

| Tier | Setup | Time | What it validates |
|---|---|---|---|
| 1 (local smoke) | RTX 3050, b=16, 200 steps, 1 data file | **~6 sec** | Kernel correctness, ISV propagation, no NaN |
| **GAP — no intermediate validation** | — | — | — |
| 2 (cluster smoke) | L40S, b=1024, 20k+5k, 9 PVC files | **~70 min** | Behavioral + eval generalization |

**Cost of the gap**: every hypothesis costs 70min to falsify. 4 hypotheses today = 4.5 hours of cluster time + 4× context switching while waiting. The Perplexity 2026-06-02 RL-engineering review confirms the canonical fix is a **funnel** with intermediate behavioral validation.

## §1. Goal — add Tier 1.5 local mid-smoke

| Tier | Setup | Time | Verdict signals |
|---|---|---|---|
| 1 (correctness) | RTX 3050, b=16, 200, 1 file | 6 sec | shapes, NaN, ISV slots |
| **1.5 (NEW: behavior)** | **RTX 3050, b=128, 2000+500, 2 PVC files (~1.5GB local)** | **~10 min** | entropy trajectory, hold growth, wr crossover, controller stability, scaffold/aux signal sanity, early Pearson |
| 2 (eval verdict) | L40S, b=1024, 20k+5k, 9 PVC files | 70 min | eval pnl, eval wr, regime stratification |

**Tier 1.5 kills 80%+ of bad hypotheses before they reach cluster**. The 10-min cost is recovered ~7× per saved cluster smoke.

## §2. RTX 3050 capacity check (4GB VRAM)

| Component | b=16 (now) | b=128 (proposed Tier 1.5) | b=1024 (cluster) |
|---|---|---|---|
| Encoder activations (mamba2, hidden=256, seq=32, 6 layers) | 3 MB | **24 MB** | 192 MB |
| Q/V/π heads forward+backward | 1 MB | **8 MB** | 64 MB |
| Replay buffer (capacity 4096) | ~100 MB | 100 MB | 100 MB |
| Optimizer state (AdamW × ~1M params × 16B) | 16 MB | 16 MB | 16 MB |
| LobSim per-account state (1024 max units, ~4KB/account) | ~64 KB | ~512 KB | 4 MB |
| Misc (CUDA workspace, kernel launch overhead) | 200 MB | 200 MB | 200 MB |
| **TOTAL** | ~320 MB | **~350 MB** | ~580 MB |

**Verdict**: RTX 3050 (4GB) has comfortable headroom for b=128. b=256 also fits. b=512 is the edge.

## §3. Implementation tasks

### §3.1 PVC data sync (one-time setup)
1. Identify the 2-3 highest-value MBP-10 files from `/feature-cache/futures-baseline-mbp10/` (recommend: most-recent fold-1 train file + most-recent fold-1 eval file)
2. `rsync` (via kubectl exec or a bridge pod) to local `test_data/futures-baseline-mid/` (~1.5 GB local disk)
3. Add `test_data/futures-baseline-mid/` to `.gitignore` (don't commit data)
4. Document in `CLAUDE.md` under "Build & Test" the new `FOXHUNT_MID_TEST_DATA` env var

### §3.2 Local mid-smoke script
1. Create `scripts/local-mid-smoke.sh` (NEW file in `scripts/`):
   ```bash
   #!/usr/bin/env bash
   set -euo pipefail
   SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha
   target/release/examples/alpha_rl_train \
     --n-steps 2000 --n-eval-steps 500 \
     --fold-idx 1 --n-folds 3 \
     --mbp10-data-dir test_data/futures-baseline-mid \
     --predecoded-dir test_data/futures-baseline-mid \
     --out /tmp/foxhunt-mid-smoke \
     --eval-diag-jsonl /tmp/foxhunt-mid-smoke/eval_diag.jsonl \
     --instrument-mode front-month --n-backtests 128 \
     --log-every 500 --seed 42
   ```
2. Per-run elapsed time target: ≤ 10 min on RTX 3050
3. Add to `Makefile` or `justfile` as `mid-smoke` target

### §3.3 Behavioral kill criteria (Tier 1.5 verdict)
After mid-smoke completes, run a verdict script that reads diag.jsonl + eval_diag.jsonl and produces PASS/FAIL based on behavioral signals (cheaper than waiting for eval pnl):

| Signal | Train target | Eval target | Kill if |
|---|---|---|---|
| `action_entropy` final | ≥ log(11) × 0.5 (1.20) AND ≤ log(11) × 0.85 (2.04) | similar | < 0.6 or > 2.3 — policy collapsed or random |
| `q_pi_agree_ema` final | ≥ 0.6 | n/a | < 0.3 — Q and π decoupled |
| Pearson(rewards.sum, Δrealized_pnl) | ≥ 0.5 | ≥ 0.4 | < 0.3 — gradient anti-aligned |
| `avg_hold_steps` | growing trend over 1500 steps | n/a | flat or decreasing — surfer pattern absent |
| `popart.sigma` | stable (CV < 50%) | n/a | wild oscillations — controller broken |
| `wr_ema` final | ≥ 0.25 | ≥ 0.20 | < 0.15 — random or worse |
| `eval pnl` | n/a | > -$1M (allow margin given small batch) | < -$3M — catastrophic |

Implement as `scripts/tier1_5_verdict.py` that reads diag.jsonl and emits OK / KILL_<reason> + optional explanation.

Per `feedback_kill_runs_on_anomaly_quickly`: any KILL_* response means DO NOT submit cluster smoke. Fix locally first.

### §3.4 Checkpoint-resume — async double-buffer for graph compatibility

**Architectural constraint (verified 2026-06-02)**: ml-alpha already runs in mega-graph mode with 4 active CUDA graphs (`prefill_graph`, `postfill_graph`, `reward_graph`, `training_graph` per `crates/ml-alpha/src/trainer/integrated.rs:715-719`). The team has actively eliminated host-blocking sync points (per `integrated.rs:707` comment "instead of `train_stream.synchronize()` — event wait is GPU-side" and `integrated.rs:8459` "saves ~2ms/step"). **Any synchronous DtoH for checkpoint would regress this perf work and break graph capture compatibility.**

Therefore checkpoint-resume MUST use **async double-buffer** from the start — sync is not a viable interim.

#### §3.4.1 Prerequisite — audit existing checkpoint code for sync points

Before any new code:
1. Read `crates/ml-alpha/src/trainer/integrated.rs` around the checkpoint save/load functions (referenced near line 9621 per `2026-05-31-checkpoints-and-eval-diag-design.md`)
2. Identify every `stream.synchronize()`, `device.synchronize()`, or implicit-sync call in the checkpoint path
3. Each one is a graph-mode breaker — either remove or replace with `CudaEvent::record` + later `event.synchronize_host()` deferred until off the training stream
4. Document the audit findings as a note: "existing checkpoint code has N sync points at lines ... and uses ... pattern"

If the existing checkpoint code is already async (which is possible given the team's discipline), §3.4.2-§3.4.5 below describe DOCUMENTING it and wiring the `--resume-from` flag. If it's sync, the design below replaces it.

#### §3.4.2 Design — async double-buffer

**Two pinned host buffers** (each sized to total checkpoint payload, ~500MB):
```rust
struct CheckpointBuffers {
    buf_a: CudaSlice<u8>,    // pinned host, 500MB
    buf_b: CudaSlice<u8>,    // pinned host, 500MB
    active: AtomicUsize,     // 0 → buf_a, 1 → buf_b
    a_complete: CudaEvent,   // signaled when DtoH into buf_a finishes
    b_complete: CudaEvent,   // signaled when DtoH into buf_b finishes
}
```

**Save path** (called from training loop at checkpoint cadence):
1. Read `active` atomically → decide target buffer (say `buf_a`)
2. Verify the OTHER buffer's serialization is complete (i.e., previous checkpoint's I/O thread done)
3. Enqueue DtoH copies for all checkpoint payload onto the training stream targeting `buf_a`
4. Enqueue `a_complete.record(stream)` on the training stream
5. Notify the serialization thread that `buf_a` is being filled
6. Flip `active` to `buf_b` so next checkpoint won't collide
7. **Training stream continues without sync** — graph capture not broken

**Serialization thread** (separate `std::thread::spawn`):
1. Wait on `a_complete.synchronize_host()` (blocks ONLY this thread, not training)
2. Serialize the pinned buffer contents to disk via `std::io::Write`
3. Mark "serialization complete for buf_a" so next checkpoint into buf_a can proceed

**Load path** (called once at run start, before training begins — no graph conflict):
1. Read checkpoint file from disk to pinned host buffer
2. HtoD copy to GPU (sync OK here since training hasn't started)
3. Re-enter graph capture mode

#### §3.4.3 Checkpoint payload

Per `pearl_F4.1` checkpoint infra, the payload includes:
- Encoder weights + Adam state
- Q-head weights + Adam state
- V-head weights + Adam state
- π-head weights + Adam state
- Bootstrap ISV (760+ floats — small, but include for reproducibility)
- popart Welford state (mean + variance + count)
- Kelly EMAs (avg_win, avg_loss, win_rate)
- Edge-decay PH state per batch
- regime_observer state per batch

Audit in §3.4.1 must verify ALL of these are captured. Per `pearl_adaptive_carryover_discipline`: missing any one means resume produces a different agent than save.

#### §3.4.4 Hypothesis-test workflow

**Pre-train baseline checkpoint** (one-time per SHA):
1. Full 20k-step cluster run with `--checkpoint-every 5000 --checkpoint-dir /feature-cache/baselines/<sha>/`
2. Produces 4 checkpoints + final = `/feature-cache/baselines/<sha>/{ck-5k, ck-10k, ck-15k, ck-20k}.bin`
3. Cluster smoke = 70 min (baseline cost)

**Hypothesis test runs** (per controller iteration):
1. New SHA with controller change
2. Resume from `/feature-cache/baselines/<previous-sha>/ck-15k.bin` (skip first 15k steps)
3. Train 5k additional steps with the new code → continues from learned state
4. Eval 5k
5. Verdict in ~12min instead of 70min

**Caveat**: code-incompatible changes (struct layout, slot allocation) require fresh baseline. Versioned checkpoint format with magic number + schema hash.

#### §3.4.5 Implementation tasks

1. Audit existing checkpoint code (§3.4.1) — ~1 hour
2. Implement async double-buffer save (§3.4.2) — ~1.5 days
3. Add serialization thread + completion tracking — ~0.5 day
4. Implement load path (~half day)
5. Wire `--resume-from <path>` + `--checkpoint-every N` flags to `alpha_rl_train.rs` (~half day)
6. Add checkpoint schema versioning (~half day)
7. Local invariant test: save → resume → continue produces bit-equivalent diag for next 100 steps (~half day)

**Total**: ~4 days. Saves ~58min per cluster smoke (~80% reduction) once shipped.

#### §3.4.6 Sync-vs-async trade-off summary

| Approach | Implementation | Graph compat | Maintenance |
|---|---|---|---|
| Sync DtoH | 1 day | ❌ Breaks ml-alpha's 4 active CUDA graphs | Forces rewrite when graph dependency hardens |
| **Async double-buffer (this spec)** | **4 days** | ✅ Native graph compat via CudaEvent | Stable; aligned with existing perf discipline |

The 3 extra days are recovered after the first cluster smoke (sync would need rework before merging back into graph mode anyway).

### §3.5 Determinism / seed pinning (reproducibility audit)
Reproducibility is required for any tiered validation to be trustworthy. Verify:
1. Python (data loading) seeds: pin via env var (already done)
2. NumPy seeds: pin (already done)
3. cudarc / curand kernel seeds: verify pinned (per `pearl_rl_sample_tau_xorshift32`: device-side xorshift seeded from host)
4. Data order: NO random shuffle in walk-forward (verify in trainer)
5. Replay buffer: PER is randomized BUT seeded — verify replay seed propagation

Local mid-smoke MUST be deterministic: same seed → same final loss within fp32 precision. Add a self-test: run mid-smoke twice with seed=42, diff the final 5 rows of diag.jsonl — should match to 1e-5.

### §3.6 Per-step diag / eval_summary consistency check
Per `pearl_grwwh_eval_catastrophic_collapse`: at grwwh step 4863 per-step showed +$74M, final eval_summary showed -$160M. $234M gap is intolerable — the per-step diag is misleading.

1. Pull the LAST eval row of `eval_diag.jsonl` AND `eval_summary.json` from cluster runs
2. If `|per_step_realized_pnl_cum - eval_summary_total_pnl| > 5% × max(abs)`, log a discrepancy warning
3. Add this to the verdict script (§3.3 above) so Tier 1.5 also catches the issue
4. Long-term: fix the accounting axis discrepancy in the diag/summary code

## §4. Estimated impact on dev cycle

**Today (4 controller iterations on the surfer-scaffold)**:
- Tier 1 (5s × 4 = 20s) + Tier 2 (70min × 4 = 280min) = **4.7 hours of cluster time**
- Plus context-switching waiting for cluster results

**With Tier 1.5 + checkpoint-resume**:
- Tier 1 (5s × 4 = 20s) + Tier 1.5 (10min × 4 = 40min) + Tier 2 only on Tier 1.5 PASS (assume 2 out of 4 PASS) = **40min Tier 1.5 + 24min Tier 2 = ~1 hour total**
- **~4× faster total wall time**
- **2× fewer cluster smokes** (only PASS the gate get to Tier 2)
- Plus the savings from checkpoint-resume on the Tier 2 runs

## §5. Sequencing

Ship in this order (each step independently useful, but §3.4 is now the longest chunk):

1. **§3.1 PVC data sync** (~30 min setup) — gives local access to real fold-1 data
2. **§3.2 Local mid-smoke script** (~30 min) — make `cargo run --release ... --n-backtests 128 --n-steps 2000` work
3. **§3.5 Determinism audit** (~1 hour) — verify seed pinning, add self-test
4. **§3.3 Behavioral kill verdict script** (~1 hour) — `scripts/tier1_5_verdict.py`
5. **§3.6 Per-step / eval_summary consistency check** (~30 min) — add to verdict script
6. **§3.4 Async double-buffer checkpoint-resume** (~4 days) — see §3.4.5 task list; sync DtoH is NOT viable due to existing mega-graph mode (see §3.4 architectural constraint)

**Total spec implementation**: ~5 hours (items 1-5) + ~4 days (item 6) = ~5 working days.

Items 1-5 ship in ~1 day and unlock most of the dev cycle gain (Tier 1.5 mid-smoke gates 80% of bad hypotheses without needing checkpoint-resume). Item 6 (async checkpoint) is the bigger investment but is on the critical path for graph-mode compatibility AND saves ~58min per cluster smoke once shipped.

**Parallel path option**: items 1-5 can be done by one developer; §3.4 in parallel by a second developer or a coder agent. The fast wins land in 1 day; the deep checkpoint refactor lands in 4-5 days.

## §6. Out of scope (for now)

- **torch.compile / kernel autotuning**: foxhunt already uses pre-compiled CUDA kernels with handcrafted block-tree-reduce per `feedback_nvidia_grade_perf_for_kernels`. Further compile-time tuning is high-cost low-reward.
- **Surrogate models / amortized inference**: building a simulator that mimics foxhunt's training dynamics would be its own multi-week project. Defer until the standard funnel proves insufficient.
- **Multi-seed reproducibility runs**: requires the deterministic foundation from §3.5 first. Add after Tier 1.5 is operational.
- **Production walk-forward distinct from cluster smoke**: out of scope; the existing 3-fold walk-forward is the production gate. Tier 1.5 is a development accelerator, not a production replacement.

## §7. Open questions

1. **rsync vs sshfs**: rsync (one-time copy) preferred for steady-state read performance. ~1.5GB local disk cost is negligible.
2. **b=128 vs b=256 for mid-smoke**: b=128 fits comfortably on RTX 3050 with margin. b=256 may stretch. Start with b=128, raise to b=256 if behavioral signals are noisy.
3. **Step count tuning**: 2000 train + 500 eval might be too few for some signals (popart σ takes 1000+ steps to stabilize). Could extend to 3000+750 if needed; still < 15 min total.
4. **Checkpoint format compatibility**: existing checkpoint infra from F1.3 saves model + Adam + ISV. Does it save popart Welford state? Kelly EMAs? `regime_observer` state? Audit and fix gaps before §3.4 ships.

## §8. Discipline reminders

- `feedback_local_smoke_before_cluster` extends naturally: now Tier 1 → Tier 1.5 → Tier 2 in strict order
- `feedback_kill_runs_on_anomaly_quickly` applies at Tier 1.5 just as much as Tier 2 — KILL on behavioral red flags before cluster submit
- `feedback_going_in_circles_pattern` — if the same Tier 1.5 KILL_* triggers 3 times in a row, STOP and re-audit the foundation, not the controller
