foxhunt/docs/superpowers/specs/2026-06-02-fast-dev-cycle.md

Fast Dev Cycle — Closing the 700× Gap

Date: 2026-06-02 Motivation: today's session burned 4 cluster smokes × ~70min each on controller iterations that could have been killed in ~10min locally. Per feedback_going_in_circles_pattern: "3+ consecutive cluster runs each surfacing a new failure introduced by the prior fix" — the going-in-circles pattern is partially a CONSEQUENCE of slow dev cycle, not a separate root cause. Linked: feedback_local_smoke_before_cluster, feedback_kill_runs_on_anomaly_quickly, pearl_grwwh_eval_catastrophic_collapse

§0. The current gap

Tier	Setup	Time	What it validates
1 (local smoke)	RTX 3050, b=16, 200 steps, 1 data file	~6 sec	Kernel correctness, ISV propagation, no NaN
GAP — no intermediate validation	—	—	—
2 (cluster smoke)	L40S, b=1024, 20k+5k, 9 PVC files	~70 min	Behavioral + eval generalization

Cost of the gap: every hypothesis costs 70min to falsify. 4 hypotheses today = 4.5 hours of cluster time + 4× context switching while waiting. The Perplexity 2026-06-02 RL-engineering review confirms the canonical fix is a funnel with intermediate behavioral validation.

§1. Goal — add Tier 1.5 local mid-smoke

Tier	Setup	Time	Verdict signals
1 (correctness)	RTX 3050, b=16, 200, 1 file	6 sec	shapes, NaN, ISV slots
1.5 (NEW: behavior)	RTX 3050, b=128, 2000+500, 2 PVC files (~1.5GB local)	~10 min	entropy trajectory, hold growth, wr crossover, controller stability, scaffold/aux signal sanity, early Pearson
2 (eval verdict)	L40S, b=1024, 20k+5k, 9 PVC files	70 min	eval pnl, eval wr, regime stratification

Tier 1.5 kills 80%+ of bad hypotheses before they reach cluster. The 10-min cost is recovered ~7× per saved cluster smoke.

§2. RTX 3050 capacity check (4GB VRAM)

Component	b=16 (now)	b=128 (proposed Tier 1.5)	b=1024 (cluster)
Encoder activations (mamba2, hidden=256, seq=32, 6 layers)	3 MB	24 MB	192 MB
Q/V/π heads forward+backward	1 MB	8 MB	64 MB
Replay buffer (capacity 4096)	~100 MB	100 MB	100 MB
Optimizer state (AdamW × ~1M params × 16B)	16 MB	16 MB	16 MB
LobSim per-account state (1024 max units, ~4KB/account)	~64 KB	~512 KB	4 MB
Misc (CUDA workspace, kernel launch overhead)	200 MB	200 MB	200 MB
TOTAL	~320 MB	~350 MB	~580 MB

Verdict: RTX 3050 (4GB) has comfortable headroom for b=128. b=256 also fits. b=512 is the edge.

§3. Implementation tasks

§3.1 PVC data sync (one-time setup)

1. Identify the 2-3 highest-value MBP-10 files from /feature-cache/futures-baseline-mbp10/ (recommend: most-recent fold-1 train file + most-recent fold-1 eval file)
2. rsync (via kubectl exec or a bridge pod) to local test_data/futures-baseline-mid/ (~1.5 GB local disk)
3. Add test_data/futures-baseline-mid/ to .gitignore (don't commit data)
4. Document in CLAUDE.md under "Build & Test" the new FOXHUNT_MID_TEST_DATA env var

§3.2 Local mid-smoke script

1. Create scripts/local-mid-smoke.sh (NEW file in scripts/):

   #!/usr/bin/env bash
   set -euo pipefail
   SQLX_OFFLINE=true cargo build --release --example alpha_rl_train -p ml-alpha
   target/release/examples/alpha_rl_train \
     --n-steps 2000 --n-eval-steps 500 \
     --fold-idx 1 --n-folds 3 \
     --mbp10-data-dir test_data/futures-baseline-mid \
     --predecoded-dir test_data/futures-baseline-mid \
     --out /tmp/foxhunt-mid-smoke \
     --eval-diag-jsonl /tmp/foxhunt-mid-smoke/eval_diag.jsonl \
     --instrument-mode front-month --n-backtests 128 \
     --log-every 500 --seed 42
   

2. Per-run elapsed time target: ≤ 10 min on RTX 3050
3. Add to Makefile or justfile as mid-smoke target

§3.3 Behavioral kill criteria (Tier 1.5 verdict)

After mid-smoke completes, run a verdict script that reads diag.jsonl + eval_diag.jsonl and produces PASS/FAIL based on behavioral signals (cheaper than waiting for eval pnl):

Signal	Train target	Eval target	Kill if
action_entropy final	≥ log(11) × 0.5 (1.20) AND ≤ log(11) × 0.85 (2.04)	similar	< 0.6 or > 2.3 — policy collapsed or random
q_pi_agree_ema final	≥ 0.6	n/a	< 0.3 — Q and π decoupled
Pearson(rewards.sum, Δrealized_pnl)	≥ 0.5	≥ 0.4	< 0.3 — gradient anti-aligned
avg_hold_steps	growing trend over 1500 steps	n/a	flat or decreasing — surfer pattern absent
popart.sigma	stable (CV < 50%)	n/a	wild oscillations — controller broken
wr_ema final	≥ 0.25	≥ 0.20	< 0.15 — random or worse
eval pnl	n/a	> -$1M (allow margin given small batch)	< -$3M — catastrophic

Implement as scripts/tier1_5_verdict.py that reads diag.jsonl and emits OK / KILL_ + optional explanation.

Per feedback_kill_runs_on_anomaly_quickly: any KILL_* response means DO NOT submit cluster smoke. Fix locally first.

§3.4 Checkpoint-resume — async double-buffer for graph compatibility

Architectural constraint (verified 2026-06-02): ml-alpha already runs in mega-graph mode with 4 active CUDA graphs (prefill_graph, postfill_graph, reward_graph, training_graph per crates/ml-alpha/src/trainer/integrated.rs:715-719). The team has actively eliminated host-blocking sync points (per integrated.rs:707 comment "instead of train_stream.synchronize() — event wait is GPU-side" and integrated.rs:8459 "saves ~2ms/step"). Any synchronous DtoH for checkpoint would regress this perf work and break graph capture compatibility.

Therefore checkpoint-resume MUST use async double-buffer from the start — sync is not a viable interim.

§3.4.1 Prerequisite — audit existing checkpoint code for sync points

Before any new code:

1. Read crates/ml-alpha/src/trainer/integrated.rs around the checkpoint save/load functions (referenced near line 9621 per 2026-05-31-checkpoints-and-eval-diag-design.md)
2. Identify every stream.synchronize(), device.synchronize(), or implicit-sync call in the checkpoint path
3. Each one is a graph-mode breaker — either remove or replace with CudaEvent::record + later event.synchronize_host() deferred until off the training stream
4. Document the audit findings as a note: "existing checkpoint code has N sync points at lines ... and uses ... pattern"

If the existing checkpoint code is already async (which is possible given the team's discipline), §3.4.2-§3.4.5 below describe DOCUMENTING it and wiring the --resume-from flag. If it's sync, the design below replaces it.

§3.4.2 Design — async double-buffer

Two pinned host buffers (each sized to total checkpoint payload, ~500MB):

struct CheckpointBuffers {
    buf_a: CudaSlice<u8>,    // pinned host, 500MB
    buf_b: CudaSlice<u8>,    // pinned host, 500MB
    active: AtomicUsize,     // 0 → buf_a, 1 → buf_b
    a_complete: CudaEvent,   // signaled when DtoH into buf_a finishes
    b_complete: CudaEvent,   // signaled when DtoH into buf_b finishes
}

Save path (called from training loop at checkpoint cadence):

1. Read active atomically → decide target buffer (say buf_a)
2. Verify the OTHER buffer's serialization is complete (i.e., previous checkpoint's I/O thread done)
3. Enqueue DtoH copies for all checkpoint payload onto the training stream targeting buf_a
4. Enqueue a_complete.record(stream) on the training stream
5. Notify the serialization thread that buf_a is being filled
6. Flip active to buf_b so next checkpoint won't collide
7. Training stream continues without sync — graph capture not broken

Serialization thread (separate std::thread::spawn):

1. Wait on a_complete.synchronize_host() (blocks ONLY this thread, not training)
2. Serialize the pinned buffer contents to disk via std::io::Write
3. Mark "serialization complete for buf_a" so next checkpoint into buf_a can proceed

Load path (called once at run start, before training begins — no graph conflict):

1. Read checkpoint file from disk to pinned host buffer
2. HtoD copy to GPU (sync OK here since training hasn't started)
3. Re-enter graph capture mode

§3.4.3 Checkpoint payload

Per pearl_F4.1 checkpoint infra, the payload includes:

• Encoder weights + Adam state
• Q-head weights + Adam state
• V-head weights + Adam state
• π-head weights + Adam state
• Bootstrap ISV (760+ floats — small, but include for reproducibility)
• popart Welford state (mean + variance + count)
• Kelly EMAs (avg_win, avg_loss, win_rate)
• Edge-decay PH state per batch
• regime_observer state per batch

Audit in §3.4.1 must verify ALL of these are captured. Per pearl_adaptive_carryover_discipline: missing any one means resume produces a different agent than save.

§3.4.4 Hypothesis-test workflow

Pre-train baseline checkpoint (one-time per SHA):

1. Full 20k-step cluster run with --checkpoint-every 5000 --checkpoint-dir /feature-cache/baselines/<sha>/
2. Produces 4 checkpoints + final = /feature-cache/baselines/<sha>/{ck-5k, ck-10k, ck-15k, ck-20k}.bin
3. Cluster smoke = 70 min (baseline cost)

Hypothesis test runs (per controller iteration):

1. New SHA with controller change
2. Resume from /feature-cache/baselines/<previous-sha>/ck-15k.bin (skip first 15k steps)
3. Train 5k additional steps with the new code → continues from learned state
4. Eval 5k
5. Verdict in ~12min instead of 70min

Caveat: code-incompatible changes (struct layout, slot allocation) require fresh baseline. Versioned checkpoint format with magic number + schema hash.

§3.4.5 Implementation tasks

1. Audit existing checkpoint code (§3.4.1) — ~1 hour
2. Implement async double-buffer save (§3.4.2) — ~1.5 days
3. Add serialization thread + completion tracking — ~0.5 day
4. Implement load path (~half day)
5. Wire --resume-from <path> + --checkpoint-every N flags to alpha_rl_train.rs (~half day)
6. Add checkpoint schema versioning (~half day)
7. Local invariant test: save → resume → continue produces bit-equivalent diag for next 100 steps (~half day)

Total: ~4 days. Saves ~58min per cluster smoke (~80% reduction) once shipped.

§3.4.6 Sync-vs-async trade-off summary

Approach	Implementation	Graph compat	Maintenance
Sync DtoH	1 day	❌ Breaks ml-alpha's 4 active CUDA graphs	Forces rewrite when graph dependency hardens
Async double-buffer (this spec)	4 days	✅ Native graph compat via CudaEvent	Stable; aligned with existing perf discipline

The 3 extra days are recovered after the first cluster smoke (sync would need rework before merging back into graph mode anyway).

§3.5 Determinism / seed pinning (reproducibility audit)

Reproducibility is required for any tiered validation to be trustworthy. Verify:

1. Python (data loading) seeds: pin via env var (already done)
2. NumPy seeds: pin (already done)
3. cudarc / curand kernel seeds: verify pinned (per pearl_rl_sample_tau_xorshift32: device-side xorshift seeded from host)
4. Data order: NO random shuffle in walk-forward (verify in trainer)
5. Replay buffer: PER is randomized BUT seeded — verify replay seed propagation

Local mid-smoke MUST be deterministic: same seed → same final loss within fp32 precision. Add a self-test: run mid-smoke twice with seed=42, diff the final 5 rows of diag.jsonl — should match to 1e-5.

§3.6 Per-step diag / eval_summary consistency check

Per pearl_grwwh_eval_catastrophic_collapse: at grwwh step 4863 per-step showed +$74M, final eval_summary showed -$160M. $234M gap is intolerable — the per-step diag is misleading.

1. Pull the LAST eval row of eval_diag.jsonl AND eval_summary.json from cluster runs
2. If |per_step_realized_pnl_cum - eval_summary_total_pnl| > 5% × max(abs), log a discrepancy warning
3. Add this to the verdict script (§3.3 above) so Tier 1.5 also catches the issue
4. Long-term: fix the accounting axis discrepancy in the diag/summary code

§4. Estimated impact on dev cycle

Today (4 controller iterations on the surfer-scaffold):

• Tier 1 (5s × 4 = 20s) + Tier 2 (70min × 4 = 280min) = 4.7 hours of cluster time
• Plus context-switching waiting for cluster results

With Tier 1.5 + checkpoint-resume:

• Tier 1 (5s × 4 = 20s) + Tier 1.5 (10min × 4 = 40min) + Tier 2 only on Tier 1.5 PASS (assume 2 out of 4 PASS) = 40min Tier 1.5 + 24min Tier 2 = ~1 hour total
• ~4× faster total wall time
• 2× fewer cluster smokes (only PASS the gate get to Tier 2)
• Plus the savings from checkpoint-resume on the Tier 2 runs

§5. Sequencing

Ship in this order (each step independently useful, but §3.4 is now the longest chunk):

1. §3.1 PVC data sync (~30 min setup) — gives local access to real fold-1 data
2. §3.2 Local mid-smoke script (~30 min) — make cargo run --release ... --n-backtests 128 --n-steps 2000 work
3. §3.5 Determinism audit (~1 hour) — verify seed pinning, add self-test
4. §3.3 Behavioral kill verdict script (~1 hour) — scripts/tier1_5_verdict.py
5. §3.6 Per-step / eval_summary consistency check (~30 min) — add to verdict script
6. §3.4 Async double-buffer checkpoint-resume (~4 days) — see §3.4.5 task list; sync DtoH is NOT viable due to existing mega-graph mode (see §3.4 architectural constraint)

Total spec implementation: ~5 hours (items 1-5) + ~4 days (item 6) = ~5 working days.

Items 1-5 ship in ~1 day and unlock most of the dev cycle gain (Tier 1.5 mid-smoke gates 80% of bad hypotheses without needing checkpoint-resume). Item 6 (async checkpoint) is the bigger investment but is on the critical path for graph-mode compatibility AND saves ~58min per cluster smoke once shipped.

Parallel path option: items 1-5 can be done by one developer; §3.4 in parallel by a second developer or a coder agent. The fast wins land in 1 day; the deep checkpoint refactor lands in 4-5 days.

§6. Out of scope (for now)

• torch.compile / kernel autotuning: foxhunt already uses pre-compiled CUDA kernels with handcrafted block-tree-reduce per feedback_nvidia_grade_perf_for_kernels. Further compile-time tuning is high-cost low-reward.
• Surrogate models / amortized inference: building a simulator that mimics foxhunt's training dynamics would be its own multi-week project. Defer until the standard funnel proves insufficient.
• Multi-seed reproducibility runs: requires the deterministic foundation from §3.5 first. Add after Tier 1.5 is operational.
• Production walk-forward distinct from cluster smoke: out of scope; the existing 3-fold walk-forward is the production gate. Tier 1.5 is a development accelerator, not a production replacement.

§7. Open questions

1. rsync vs sshfs: rsync (one-time copy) preferred for steady-state read performance. ~1.5GB local disk cost is negligible.
2. b=128 vs b=256 for mid-smoke: b=128 fits comfortably on RTX 3050 with margin. b=256 may stretch. Start with b=128, raise to b=256 if behavioral signals are noisy.
3. Step count tuning: 2000 train + 500 eval might be too few for some signals (popart σ takes 1000+ steps to stabilize). Could extend to 3000+750 if needed; still < 15 min total.
4. Checkpoint format compatibility: existing checkpoint infra from F1.3 saves model + Adam + ISV. Does it save popart Welford state? Kelly EMAs? regime_observer state? Audit and fix gaps before §3.4 ships.

§8. Discipline reminders

• feedback_local_smoke_before_cluster extends naturally: now Tier 1 → Tier 1.5 → Tier 2 in strict order
• feedback_kill_runs_on_anomaly_quickly applies at Tier 1.5 just as much as Tier 2 — KILL on behavioral red flags before cluster submit
• feedback_going_in_circles_pattern — if the same Tier 1.5 KILL_* triggers 3 times in a row, STOP and re-audit the foundation, not the controller