From d45dde8458e3bc8036dbbe4fc4e2c1d7cfa63f84 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 19 May 2026 01:04:30 +0200 Subject: [PATCH] plan(ml-alpha): trunk-grows refactor + deployability validation roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 20-commit atomic ladder (X0–X19) + Phase 1→2 gate + Phase 2 runtime runbook. Implements spec da1dd92bf: X0: perception_forward_golden fixture (bit-equivalence gate) X1: CfcTrunk v2 weight skeleton (no callers) X2–X9: incremental weight-group migrations (VSN, Mamba2 ×2, LN ×2, attn-pool, CfC, GRN heads), each gated by golden fixture X10: hoist forward kernels into CfcTrunk methods X11: capture_graph_a covers full v2 forward + captured-vs-uncaptured equivalence test X12: CheckpointV2 envelope + save/load (V1 hard-rejected) X13: PerceptionTrainer.save_checkpoint delegate X14: alpha_train saves best_h6000 checkpoint X15: verify ml-backtesting accepts CheckpointV2 (no code change) X16: max_drawdown_pct with \$35k base X17: emit_deployability_verdict + tiered logic + 6 unit tests X18: GPU smoke test against real trained checkpoint X19: three sweep YAMLs (smoke, threshold-tuning, deployability) Gate: fold-0 smoke must reproduce recorded 3-fold A/B numbers within ±0.010 absolute before Phase 2 begins P.1–6: Argo runtime (training → smoke → threshold → sweep → verdict) Self-review confirms 1:1 spec coverage. Three soft adaptation points (HEAD_MID constant, Mamba2Block accessors, BacktestHarnessConfig field names) resolve at code-read time. One placeholder (todo!() in X11 explanatory text) is called out in self-review for replacement when that commit lands. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...-alpha-v2-trunk-grows-and-deployability.md | 2207 +++++++++++++++++ 1 file changed, 2207 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-19-ml-alpha-v2-trunk-grows-and-deployability.md diff --git a/docs/superpowers/plans/2026-05-19-ml-alpha-v2-trunk-grows-and-deployability.md b/docs/superpowers/plans/2026-05-19-ml-alpha-v2-trunk-grows-and-deployability.md new file mode 100644 index 000000000..419dfd651 --- /dev/null +++ b/docs/superpowers/plans/2026-05-19-ml-alpha-v2-trunk-grows-and-deployability.md @@ -0,0 +1,2207 @@ +# ml-alpha v2 Trunk-Grows + Deployability Validation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Refactor `CfcTrunk` to own the full v2 inference graph (VSN + Mamba2 ×2 + LN ×2 + attn-pool + CfC + heads), restructure `PerceptionTrainer` to wrap a trunk + add training-only state, wire `save_checkpoint` end-to-end from training to backtest, then produce a falsifiable deployability verdict (Pass-robust / Pass-nominal / Fail-* tiers) across 4 walk-forward held-out quarters of ES MBP-10. + +**Architecture:** 20 atomic code commits (X0–X19) under bit-equivalence-gated refactor discipline, followed by a 9-step Argo runtime runbook (Phase 2). Phase 1 → Phase 2 gate is a single-fold post-refactor smoke that must reproduce recorded 3-fold A/B numbers (`best_mean_auc 0.7529`, `best_h6000 0.7639` on fold 0) within ±0.010 absolute. Phase 2 produces `deployability_verdict.json` committed to the repo as an audit record. + +**Tech Stack:** Rust 2021, cudarc 0.19, CUDA 12.4, sccache MinIO backend, bincode for checkpoint serialization, Argo Workflows on Scaleway Kapsule (L40S GPU pool), ES MBP-10 quarterly DBN files on `training-data-pvc`. + +**Spec:** `docs/superpowers/specs/2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md` @ commit `da1dd92bf`. Read it before starting any task; this plan references it heavily. + +**Branch:** Stay on `ml-alpha-phase-a` for the entire implementation. All commits land there sequentially. + +--- + +## File Structure + +| File | Role | Phase | +|---|---|---| +| `crates/ml-alpha/tests/perception_forward_golden.rs` | NEW — bit-equivalence test gating refactor commits | X0 | +| `crates/ml-alpha/tests/fixtures/perception_forward_golden.bin` | NEW — golden output bytes captured pre-refactor | X0 | +| `crates/ml-alpha/src/cfc/trunk.rs` | RESTRUCTURE — grows from CfC-only to full v2 inference graph; gains CheckpointV2 + capture_graph_a expansion | X1, X11, X12 | +| `crates/ml-alpha/src/trainer/perception.rs` | RESTRUCTURE — weight ownership moves to `self.trunk`; forward kernels delegate to trunk methods; grad/optimizer state stays at trainer level | X2..X10, X13 | +| `crates/ml-alpha/examples/alpha_train.rs` | MODIFY — call `trainer.save_checkpoint` inside `auc_h6000_improved` block + extend `AlphaTrainSummary` schema | X14 | +| `crates/ml-backtesting/src/harness.rs` | NO INTERFACE CHANGE — already takes a `CfcTrunk`; verified loads V2 | X15 | +| `bin/fxt-backtest/src/main.rs` | NO INTERFACE CHANGE — `--checkpoint` flag transparent to V2 envelope | X15 | +| `crates/ml-backtesting/src/artifacts.rs` | MODIFY — add `max_drawdown_pct` field + `STARTING_CAPITAL_USD = 35_000.0` constant | X16 | +| `crates/ml-backtesting/src/aggregate.rs` | MODIFY — add `VerdictTier`, `AnchorReport`, `DeployabilityVerdict` types + `classify_verdict` + `emit_deployability_verdict` | X17 | +| `crates/ml-backtesting/tests/checkpoint_smoke.rs` | NEW — `#[ignore]`d GPU integration test | X18 | +| `config/ml/sweep_v2_smoke.yaml` | NEW — single-cell smoke gate config | X19 | +| `config/ml/sweep_v2_threshold_tuning.yaml` | NEW — 8-cell threshold pre-registration config | X19 | +| `config/ml/sweep_v2_deployability.yaml` | NEW — 560-cell full deployability sweep config | X19 | +| `config/ml/v2_prod_thresholds.json` | NEW (committed at Phase 2 runtime) — pre-registered threshold | Phase 2 | +| `deployability_verdict.json` | NEW (committed at Phase 2 runtime) — audit record | Phase 2 | + +--- + +## Phase 1 Commit Map + +| # | Title | Test gate | +|---|---|---| +| X0 | `test(ml-alpha): perception_forward_golden fixture + bit-equivalence test` | New test captures golden bytes | +| X1 | `feat(ml-alpha): CfcTrunk v2 weight skeleton (no callers yet)` | Compiles clean; existing tests untouched | +| X2 | `refactor(ml-alpha): move VSN weights from PerceptionTrainer to CfcTrunk` | Golden passes; all ml-alpha tests green | +| X3 | `refactor(ml-alpha): move Mamba2 stack 1 weights to CfcTrunk` | Golden passes | +| X4 | `refactor(ml-alpha): move LN_a weights to CfcTrunk` | Golden passes | +| X5 | `refactor(ml-alpha): move Mamba2 stack 2 weights to CfcTrunk` | Golden passes | +| X6 | `refactor(ml-alpha): move LN_b weights to CfcTrunk` | Golden passes | +| X7 | `refactor(ml-alpha): move attention-pool weights to CfcTrunk` | Golden passes | +| X8 | `refactor(ml-alpha): move CfC weights to CfcTrunk` | Golden passes | +| X9 | `refactor(ml-alpha): move heads weights to CfcTrunk` | Golden passes | +| X10 | `refactor(ml-alpha): hoist forward kernels into CfcTrunk methods` | Golden passes | +| X11 | `feat(ml-alpha): CfcTrunk capture_graph_a covers full v2 forward` | NEW captured-vs-uncaptured bit-equivalence test passes | +| X12 | `feat(ml-alpha): CheckpointV2 envelope + save_checkpoint + load_checkpoint` | Round-trip test passes; V1 envelopes rejected | +| X13 | `feat(ml-alpha): PerceptionTrainer.save_checkpoint delegate` | Delegate-roundtrip unit test passes | +| X14 | `feat(ml-alpha): alpha_train saves best_h6000 checkpoint` | Summary serde round-trip with `best_h6000_ckpt_path` | +| X15 | `verify(ml-backtesting): harness + fxt-backtest accept CheckpointV2` | Existing harness tests pass against fixture V2 | +| X16 | `feat(ml-backtesting): max_drawdown_pct with $35k base` | Fixture test passes | +| X17 | `feat(ml-backtesting): emit_deployability_verdict + tiered logic` | 5 tier tests + fixture sweep test | +| X18 | `test(ml-backtesting): GPU smoke against real trained checkpoint` | `#[ignore]`d test compiles | +| X19 | `config(ml-alpha): three sweep YAMLs` | YAML lint validates | + +--- + +## Conventions for every task + +1. `SQLX_OFFLINE=true` is set in the environment; all `cargo` commands assume it. +2. GPU tests are `#[ignore]`d by default; run with `--ignored` on a CUDA-capable host. +3. Per `feedback_push_before_deploy.md`, `git push` before any Argo submission. +4. Per `feedback_default_to_l40s_pool.md`, Argo submissions default to `--gpu-pool ci-training-l40s`. +5. Per `pearl_no_host_branches_in_captured_graph.md`, no host branches/syncs inside `capture_graph_a`. +6. Per `feedback_no_atomicadd.md`, no atomicAdd in trunk forward kernels. +7. Pre-commit hook (`.git/hooks/pre-commit`) runs automatically on every commit and checks GPU→CPU leaks + code-quality. Honor its decisions; do not `--no-verify`. +8. Read the spec section referenced in each task before coding. + +--- + +# PHASE 1: Trunk-grows refactor + +## Task X0: Golden fixture for bit-equivalence gate + +**Spec:** §2.1 (golden fixture must land BEFORE refactor starts), §2.4 (nondeterminism handling). + +**Why this task exists:** Every refactor commit X1..X11 must reproduce the pre-refactor forward output to a tight tolerance. We capture that output ONCE here, before any refactor work, and commit it as a binary fixture. + +**Files:** +- Create: `crates/ml-alpha/tests/perception_forward_golden.rs` +- Create: `crates/ml-alpha/tests/fixtures/perception_forward_golden.bin` + +### Step X0.1: Write the golden-capture test + +- [ ] Create `crates/ml-alpha/tests/perception_forward_golden.rs`: + +```rust +//! Bit-equivalence gate for the v2 trunk-grows refactor. +//! +//! Captures a deterministic snapshot of `PerceptionTrainer.evaluate_batched` +//! output. Every refactor commit X1..X11 must reproduce these bytes to +//! `max_abs_diff < 1e-4` (cross-GPU tolerance). +//! +//! Run with: SQLX_OFFLINE=true cargo test -p ml-alpha --test perception_forward_golden -- --ignored --nocapture +//! +//! GPU required. Golden bytes were captured on RTX 3050 (sm_86); L40S +//! runs use tight-tolerance comparison per spec §2.4. + +use std::fs; +use std::path::PathBuf; + +use anyhow::{Context, Result}; +use ml_alpha::cfc::snap_features::{Mbp10RawInput, FEATURE_DIM}; +use ml_alpha::heads::N_HORIZONS; +use ml_alpha::trainer::perception::{PerceptionTrainer, PerceptionTrainerConfig}; +use ml_core::device::MlDevice; + +const SEQ_LEN: usize = 32; +const SEED: u64 = 42; +const GOLDEN_PATH: &str = "tests/fixtures/perception_forward_golden.bin"; + +/// Build a deterministic 32-snapshot fixture using a fixed PRNG seed. +/// Output bytes are stable across runs on the same GPU model. +fn fixture_snapshots() -> Vec { + use rand::{Rng, SeedableRng}; + let mut rng = rand::rngs::StdRng::seed_from_u64(SEED); + (0..SEQ_LEN) + .map(|i| Mbp10RawInput { + bid_px: [ + 100.0 + rng.gen_range(-0.5..0.5), + 100.0 - 0.25, 100.0 - 0.50, 100.0 - 0.75, + 100.0 - 1.00, 100.0 - 1.25, 100.0 - 1.50, + 100.0 - 1.75, 100.0 - 2.00, 100.0 - 2.25, + ], + bid_sz: [10.0; 10], + ask_px: [ + 100.25 + rng.gen_range(-0.5..0.5), + 100.50, 100.75, 101.00, 101.25, + 101.50, 101.75, 102.00, 102.25, 102.50, + ], + ask_sz: [10.0; 10], + mid: 100.125 + rng.gen_range(-0.5..0.5), + ts_ns: 1_000_000_000_u64 + (i as u64) * 1_000_000, + regime: [0.0; 4], + trade_signed_vol: rng.gen_range(-5.0..5.0), + }) + .collect() +} + +/// Deterministic labels — random in [0, 1] per (position, horizon). +fn fixture_labels() -> Vec<[f32; N_HORIZONS]> { + use rand::{Rng, SeedableRng}; + let mut rng = rand::rngs::StdRng::seed_from_u64(SEED + 1); + (0..SEQ_LEN) + .map(|_| { + let mut row = [0.0_f32; N_HORIZONS]; + for x in &mut row { + *x = rng.gen_range(0.0..1.0); + } + row + }) + .collect() +} + +fn run_forward() -> Result<(f32, Vec)> { + let dev = MlDevice::new(0).context("init MlDevice")?; + let cfg = PerceptionTrainerConfig { + seq_len: SEQ_LEN, + n_batch: 1, + mamba2_state_dim: 16, + ..Default::default() + }; + let mut trainer = PerceptionTrainer::new(&dev, &cfg).context("trainer init")?; + let snapshots = fixture_snapshots(); + let labels = fixture_labels(); + let (loss, probs) = trainer.evaluate(&snapshots, &labels)?; + Ok((loss, probs)) +} + +#[test] +#[ignore = "GPU + golden file required; gates refactor commits"] +fn golden_matches() -> Result<()> { + let (loss, probs) = run_forward()?; + let crate_root = PathBuf::from(env!("CARGO_MANIFEST_DIR")); + let path = crate_root.join(GOLDEN_PATH); + if !path.exists() { + // First run: capture the golden. + let mut bytes = Vec::with_capacity(4 + probs.len() * 4); + bytes.extend_from_slice(&loss.to_le_bytes()); + for p in &probs { + bytes.extend_from_slice(&p.to_le_bytes()); + } + fs::write(&path, &bytes) + .with_context(|| format!("write golden {}", path.display()))?; + eprintln!("WROTE golden ({} bytes) at {}", bytes.len(), path.display()); + return Ok(()); + } + let golden = fs::read(&path).context("read golden")?; + assert_eq!(golden.len(), 4 + probs.len() * 4, + "golden size {} != expected {}", golden.len(), 4 + probs.len() * 4); + let golden_loss = f32::from_le_bytes(golden[0..4].try_into().unwrap()); + let mut max_diff: f32 = (loss - golden_loss).abs(); + for (i, p) in probs.iter().enumerate() { + let g = f32::from_le_bytes( + golden[4 + i * 4..4 + (i + 1) * 4].try_into().unwrap() + ); + max_diff = max_diff.max((p - g).abs()); + } + let tol = 1e-4_f32; + assert!(max_diff < tol, + "max_abs_diff {} > tolerance {} — refactor regression vs golden", max_diff, tol); + Ok(()) +} +``` + +### Step X0.2: Run the test once to capture the golden + +- [ ] On a GPU host (RTX 3050 local or L40S), run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test perception_forward_golden -- --ignored --nocapture +``` + +Expected output: `WROTE golden (N bytes) at .../perception_forward_golden.bin` and test passes (first-run case where file didn't exist). + +### Step X0.3: Run the test again to verify reproducibility + +- [ ] Run the same command a second time. Expected: golden file exists, test passes with `max_abs_diff < 1e-4`. + +If `max_abs_diff > 1e-4` between two runs on the same GPU, there's nondeterminism in the forward path. Per spec §2.4, that's a regression of `feedback_no_atomicadd` — fix the kernel before continuing. + +### Step X0.4: Commit + +- [ ] Run: + +```bash +git add crates/ml-alpha/tests/perception_forward_golden.rs \ + crates/ml-alpha/tests/fixtures/perception_forward_golden.bin +git commit -m "test(ml-alpha): perception_forward_golden fixture + bit-equivalence test + +Captures PerceptionTrainer.evaluate_batched output on a deterministic +32-snapshot fixture with seed=42. Used as a bit-equivalence gate for +the trunk-grows refactor (commits X1..X11) — every refactor commit must +reproduce these bytes to max_abs_diff < 1e-4. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §2.1." +``` + +--- + +## Task X1: CfcTrunk v2 weight skeleton + +**Spec:** §1.1 (CfcTrunk ownership boundaries), §2.2 (X1 — skeleton with all v2 weight fields). + +**Why:** Define the trunk's NEW shape (full v2 weights) before any caller migrates. Empty skeleton: fields exist, `new_random` initializes them, but no forward methods yet — those land in X10. PerceptionTrainer untouched at this commit. + +**Files:** +- Modify: `crates/ml-alpha/src/cfc/trunk.rs` + +### Step X1.1: Extend `CfcConfig` with mamba2_state_dim + +- [ ] Open `crates/ml-alpha/src/cfc/trunk.rs`. Find the existing `CfcConfig` struct (~line 55). Update: + +```rust +#[derive(Clone, Debug)] +pub struct CfcConfig { + pub n_in: usize, + pub n_hid: usize, + pub mamba2_state_dim: usize, +} + +impl Default for CfcConfig { + fn default() -> Self { + Self { + n_in: FEATURE_DIM, + n_hid: HIDDEN_DIM, + mamba2_state_dim: 16, + } + } +} +``` + +### Step X1.2: Add v2 weight fields to `CfcTrunk` + +- [ ] In the same file, find `pub struct CfcTrunk` (~line 70). Add new private fields after the existing CfC fields: + +```rust +pub struct CfcTrunk { + cfg: CfcConfig, + stream: Arc, + + // ... existing cubin module + function fields stay unchanged ... + + // Existing CfC + heads + projection weights (PRESERVED — used until X8/X9) + w_in_d: CudaSlice, + w_rec_d: CudaSlice, + b_d: CudaSlice, + tau_d: CudaSlice, + heads_w_d: CudaSlice, + heads_b_d: CudaSlice, + proj_w_d: CudaSlice, + proj_b_d: CudaSlice, + proj_g_d: CudaSlice, + proj_n_d: CudaSlice, + + // === NEW v2 fields (zero-initialized in X1; populated by X2..X9 migrations) === + pub vsn_w_d: CudaSlice, + pub vsn_b_d: CudaSlice, + pub mamba2_stack_1: Option, + pub mamba2_stack_2: Option, + pub ln_a_gain_d: CudaSlice, + pub ln_a_bias_d: CudaSlice, + pub ln_b_gain_d: CudaSlice, + pub ln_b_bias_d: CudaSlice, + pub attn_q_d: CudaSlice, + pub heads_w_gate_d: CudaSlice, + pub heads_b_gate_d: CudaSlice, + pub heads_w_main_d: CudaSlice, + pub heads_b_main_d: CudaSlice, + pub heads_w_skip_d: CudaSlice, + pub heads_b_skip_d: CudaSlice, + + // ... existing pinned buffers + scratch fields stay unchanged ... +} +``` + +The Mamba2 stacks are wrapped in `Option` for X1 only — they'll be created in X3/X5 when their weights migrate. The flat tensors are allocated as zero-filled slabs of the right shape. + +### Step X1.3: Allocate the new tensors in `new_random` + +- [ ] Find `pub fn new_random(...)` in `trunk.rs`. After the existing CfC weight init, add: + +```rust +// === v2 weight allocation (zero-init for skeleton; X2..X9 redirect to migrated weights) === +let vsn_w_d = stream.alloc_zeros::(cfg.n_in * cfg.n_in).context("vsn_w alloc")?; +let vsn_b_d = stream.alloc_zeros::(cfg.n_in).context("vsn_b alloc")?; +let mamba2_stack_1: Option = None; +let mamba2_stack_2: Option = None; +let ln_a_gain_d = stream.alloc_zeros::(cfg.n_hid).context("ln_a_gain alloc")?; +let ln_a_bias_d = stream.alloc_zeros::(cfg.n_hid).context("ln_a_bias alloc")?; +let ln_b_gain_d = stream.alloc_zeros::(cfg.n_hid).context("ln_b_gain alloc")?; +let ln_b_bias_d = stream.alloc_zeros::(cfg.n_hid).context("ln_b_bias alloc")?; +let attn_q_d = stream.alloc_zeros::(cfg.n_hid).context("attn_q alloc")?; + +let n_head_mid = ml_alpha::heads::HEAD_MID; +let heads_w_gate_d = stream.alloc_zeros::(N_HORIZONS * n_head_mid).context("heads_w_gate alloc")?; +let heads_b_gate_d = stream.alloc_zeros::(N_HORIZONS).context("heads_b_gate alloc")?; +let heads_w_main_d = stream.alloc_zeros::(N_HORIZONS * n_head_mid).context("heads_w_main alloc")?; +let heads_b_main_d = stream.alloc_zeros::(N_HORIZONS).context("heads_b_main alloc")?; +let heads_w_skip_d = stream.alloc_zeros::(N_HORIZONS * cfg.n_hid).context("heads_w_skip alloc")?; +let heads_b_skip_d = stream.alloc_zeros::(N_HORIZONS).context("heads_b_skip alloc")?; +``` + +- [ ] Add these to the `Ok(Self { ... })` block at the end of `new_random`. + +Note: if `ml_alpha::heads::HEAD_MID` doesn't exist as a constant, find the appropriate value (search heads.rs for the inner-dimension constant of the GRN head; likely named `MID` or `HEAD_MID` or hardcoded). If hardcoded, define a `pub const HEAD_MID: usize = ` in `heads.rs` and use it consistently. Document the choice in a single-line comment. + +### Step X1.4: Verify everything still compiles + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo build -p ml-alpha 2>&1 | tail -20 +SQLX_OFFLINE=true cargo test -p ml-alpha --lib 2>&1 | tail -10 +``` + +Expected: builds clean (no warnings about unused fields — they'll be used in X2+), all existing unit tests pass. + +### Step X1.5: Verify the golden test still passes + +- [ ] On a GPU host, run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test perception_forward_golden -- --ignored --nocapture +``` + +Expected: PASS. The new trunk fields are allocated but never read by the forward path yet — no behavior change. + +### Step X1.6: Commit + +- [ ] Run: + +```bash +git add crates/ml-alpha/src/cfc/trunk.rs crates/ml-alpha/src/heads.rs +git commit -m "feat(ml-alpha): CfcTrunk v2 weight skeleton (no callers yet) + +Adds zero-initialized v2 weight tensors (VSN, LN_a/b, attn_q, GRN heads) +and Option slots for stacks 1 and 2 to CfcTrunk. Extends +CfcConfig with mamba2_state_dim. No forward path changes — fields are +allocated but not yet read. + +Skeleton for X2..X9 weight migrations. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.1, §2.2." +``` + +--- + +## Pattern: Move-weight-group commit (used by X2..X9) + +Each of X2..X9 follows the same recipe. The data per commit (which fields, which kernels) changes; the steps don't. + +**Recipe (applied per commit):** + +### P.1: Identify the source + +- Find the field declarations in `crates/ml-alpha/src/trainer/perception.rs` (e.g., `vsn_w_d: CudaSlice`). +- Find every access site of those fields (search `self.vsn_w_d`, `&self.vsn_w_d`, `&mut self.vsn_w_d`). +- Find the initialization in `PerceptionTrainer::new` (the `let vsn_w_d = ...` line and the field assignment in the constructor's `Ok(Self { ... })`). + +### P.2: Update the trunk's `new_random` to initialize the migrating weights + +Replace the X1-era zero-init with the real init recipe from `PerceptionTrainer::new`. Example for VSN: + +```rust +// In trunk.rs new_random, REPLACE +// let vsn_w_d = stream.alloc_zeros::(cfg.n_in * cfg.n_in)...; +// WITH the actual init: +let vsn_scale = 1.0 / (cfg.n_in as f32).sqrt(); +let vsn_w_init: Vec = (0..cfg.n_in * cfg.n_in) + .map(|_| r.gen_range(-vsn_scale..vsn_scale)).collect(); +let vsn_w_d = stream.memcpy_stod(&vsn_w_init).context("vsn_w upload")?; +let vsn_b_d = stream.alloc_zeros::(cfg.n_in).context("vsn_b alloc")?; +``` + +(The exact init formula must match what PerceptionTrainer::new currently uses. Read it once before the commit.) + +### P.3: Update `PerceptionTrainer::new` to receive a trunk + +- Add a `trunk` field to `PerceptionTrainer` if not already present (X1 hasn't touched PerceptionTrainer yet; this happens incrementally per group migration). +- Construct the trunk inside `PerceptionTrainer::new` BEFORE the weights it will own are needed: + +```rust +let trunk = CfcTrunk::new_random(dev, &CfcConfig { + n_in: FEATURE_DIM, + n_hid: HIDDEN_DIM, + mamba2_state_dim: cfg.mamba2_state_dim, +}, cfg.seed.unwrap_or(42))?; +``` + +- Remove the now-duplicated weight initializations from `PerceptionTrainer::new`. +- Remove the corresponding field declarations from `PerceptionTrainer`. + +### P.4: Redirect every access site from `self._d` to `self.trunk._d` + +Each `self.vsn_w_d` becomes `self.trunk.vsn_w_d`. The kernel launch argument lines and any direct device-tensor access change accordingly. + +### P.5: Run the golden test + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test perception_forward_golden -- --ignored --nocapture +``` + +Expected: PASS at `max_abs_diff < 1e-4`. If FAIL: the migration changed numerical behavior somewhere. Revert this commit and diagnose — DO NOT chain-fix. + +### P.6: Run the full ml-alpha test suite + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --lib +``` + +Expected: PASS. Catches regressions in non-forward paths (smoke tests, integration tests). + +### P.7: Commit + +```bash +git add crates/ml-alpha/src/cfc/trunk.rs crates/ml-alpha/src/trainer/perception.rs +git commit -m "refactor(ml-alpha): move weights from PerceptionTrainer to CfcTrunk + + + +Verified bit-equivalence vs perception_forward_golden fixture. +Verified all ml-alpha unit + smoke tests pass. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §2.2." +``` + +--- + +## Task X2: Move VSN weights + +**Files modified:** `crates/ml-alpha/src/cfc/trunk.rs`, `crates/ml-alpha/src/trainer/perception.rs` + +**Data for this commit:** + +| Field | Type | Init recipe in PerceptionTrainer::new | +|---|---|---| +| `vsn_w_d` | `CudaSlice` shape `[in_dim, in_dim]` | Uniform `[-1/√in_dim, +1/√in_dim]`. Look for `vsn_w` upload near `Mbp10RawInput` feature dim. | +| `vsn_b_d` | `CudaSlice` shape `[in_dim]` | Zero-init. | + +- [ ] Apply pattern P.1: locate `vsn_w_d` and `vsn_b_d` field declarations in `perception.rs` (around line 343); locate their init in `PerceptionTrainer::new`; locate the access sites (search `self.vsn_w_d` and `self.vsn_b_d`). +- [ ] Apply P.2: replace trunk's `new_random` zero-init for VSN with the real init recipe. +- [ ] Apply P.3: ensure `PerceptionTrainer::new` constructs the trunk; remove duplicate VSN weight init. +- [ ] Apply P.4: redirect `self.vsn_w_d` → `self.trunk.vsn_w_d` and `self.vsn_b_d` → `self.trunk.vsn_b_d` at every access site in `perception.rs`. +- [ ] Apply P.5: golden test passes. +- [ ] Apply P.6: full ml-alpha test suite passes. +- [ ] Apply P.7: commit with message "refactor(ml-alpha): move VSN weights from PerceptionTrainer to CfcTrunk". + +--- + +## Task X3: Move Mamba2 stack 1 weights + +**Files modified:** `crates/ml-alpha/src/cfc/trunk.rs`, `crates/ml-alpha/src/trainer/perception.rs` + +**Data:** + +The Mamba2 stack is a `Mamba2Block` struct (in `crates/ml-alpha/src/mamba2_block.rs`), not flat tensors. It owns its own weight CudaSlices internally. Moving "the Mamba2 stack" means moving the `Mamba2Block` instance. + +| Field | Type | Init in PerceptionTrainer::new | +|---|---|---| +| `mamba2` (renamed to `mamba2_stack_1` in trunk) | `Mamba2Block` | Constructed via `Mamba2Block::new(Mamba2BlockConfig { state_dim: cfg.mamba2_state_dim, ... }, stream.clone())` | + +- [ ] In `trunk.rs`, change the field type from `Option` to `Mamba2Block` (mandatory, no longer skeleton). +- [ ] Apply P.2: replace `Option::None` with real `Mamba2Block::new(...)` construction. The config args (in_dim = `FEATURE_DIM`, hidden_dim = `HIDDEN_DIM`, state_dim = `cfg.mamba2_state_dim`) come from the current `PerceptionTrainer::new` call. +- [ ] Apply P.3: remove the `mamba2: Mamba2Block` field from `PerceptionTrainer`; remove its init. +- [ ] Apply P.4: redirect `self.mamba2` → `self.trunk.mamba2_stack_1` at every access site. Backward kernels read `&self.trunk.mamba2_stack_1.w_in` etc.; gradient buffers stay at trainer level (they're not weight tensors). +- [ ] P.5, P.6, P.7 as standard. + +Commit message: "refactor(ml-alpha): move Mamba2 stack 1 weights from PerceptionTrainer to CfcTrunk". + +--- + +## Task X4: Move LN_a weights + +**Files modified:** `crates/ml-alpha/src/cfc/trunk.rs`, `crates/ml-alpha/src/trainer/perception.rs` + +**Data:** + +| Field | Type | Init | +|---|---|---| +| `ln_a_gain_d` | `CudaSlice` shape `[n_hid]` | Ones | +| `ln_a_bias_d` | `CudaSlice` shape `[n_hid]` | Zeros | + +- [ ] P.1..P.7 with `ln_a_gain_d` / `ln_a_bias_d`. Init recipe: `vec![1.0_f32; cfg.n_hid]` for gain, zero for bias. + +Commit: "refactor(ml-alpha): move LN_a weights from PerceptionTrainer to CfcTrunk". + +--- + +## Task X5: Move Mamba2 stack 2 weights + +Identical to X3 but with `mamba2_l2` → `mamba2_stack_2`. Config args: `in_dim = HIDDEN_DIM` (not FEATURE_DIM — stack 2 takes stack 1's output), same hidden_dim + state_dim. + +- [ ] P.1..P.7 with the stack 2 weights. + +Commit: "refactor(ml-alpha): move Mamba2 stack 2 weights from PerceptionTrainer to CfcTrunk". + +--- + +## Task X6: Move LN_b weights + +Same shape as LN_a. + +| Field | Type | Init | +|---|---|---| +| `ln_b_gain_d` | `CudaSlice` shape `[n_hid]` | Ones | +| `ln_b_bias_d` | `CudaSlice` shape `[n_hid]` | Zeros | + +- [ ] P.1..P.7 with these fields. In `perception.rs` the names are `ln_gain_d` / `ln_bias_d` (note: NO `_b` suffix — the existing names are just `ln_*` because LN_a is the only OTHER LN). Confirm by reading the trainer struct before mapping. + +Commit: "refactor(ml-alpha): move LN_b weights from PerceptionTrainer to CfcTrunk". + +--- + +## Task X7: Move attention-pool weights + +| Field | Type | Init | +|---|---|---| +| `attn_q_d` | `CudaSlice` shape `[n_hid]` | Uniform `[-1/√n_hid, +1/√n_hid]` | + +- [ ] P.1..P.7. Single field; one access site for the attention-pool forward kernel. + +Commit: "refactor(ml-alpha): move attention-pool weights from PerceptionTrainer to CfcTrunk". + +--- + +## Task X8: Move CfC weights + +**Data:** + +| Field | Type | Init | +|---|---|---| +| `w_in_d` | `CudaSlice` shape `[n_hid, n_in]` | Uniform Xavier-like | +| `w_rec_d` | `CudaSlice` shape `[n_hid, n_hid]` | Uniform Xavier-like | +| `b_d` | `CudaSlice` shape `[n_hid]` | Zeros | +| `tau_d` | `CudaSlice` shape `[n_hid]` | Constant (read existing value from current PerceptionTrainer::new — likely `1.0` or `0.5`) | + +**Wrinkle:** These names COLLIDE with the existing CfcTrunk fields (`w_in_d`, `w_rec_d`, `b_d`, `tau_d` already exist in trunk.rs, used by the OLD CheckpointV1 + old `cfc_step` captured graph). The migration here is **PerceptionTrainer's CfC layer → CfcTrunk's CfC layer**; both end up in the same trunk. Since CfcTrunk already owned its own CfC weights, this commit is partially redundant — but it's the commit where PerceptionTrainer **stops** maintaining its own copies and reads from the trunk's existing weights. + +- [ ] Verify the shapes match between PerceptionTrainer's `self.w_in_d` and the trunk's existing `self.w_in_d`. If they're the same shape (they should be — both are CfC), then the migration is: delete PerceptionTrainer's copies, redirect access to `self.trunk.w_in_d`, ensure the trunk's init recipe matches what PerceptionTrainer used. +- [ ] If shapes differ, document the divergence and reconcile (likely a config-driven dimensionality choice; pick the right one). +- [ ] P.1..P.7. + +Commit: "refactor(ml-alpha): move CfC weights from PerceptionTrainer to CfcTrunk (consolidate dual ownership)". + +--- + +## Task X9: Move heads weights + +**Data:** + +| Field | Type | Init | +|---|---|---| +| `heads_w_gate_d` | `CudaSlice` shape `[N_HORIZONS, HEAD_MID]` | Uniform `[-1/√HEAD_MID, +1/√HEAD_MID]` | +| `heads_b_gate_d` | `CudaSlice` shape `[N_HORIZONS]` | Zeros | +| `heads_w_main_d` | `CudaSlice` shape `[N_HORIZONS, HEAD_MID]` | Uniform same scale | +| `heads_b_main_d` | `CudaSlice` shape `[N_HORIZONS]` | Zeros | +| `heads_w_skip_d` | `CudaSlice` shape `[N_HORIZONS, HIDDEN_DIM]` | Uniform `[-1/√HIDDEN_DIM, +1/√HIDDEN_DIM]` | +| `heads_b_skip_d` | `CudaSlice` shape `[N_HORIZONS]` | Zeros | + +**Wrinkle:** Same name collision as X8 with old CfcTrunk fields `heads_w_d`, `heads_b_d`. The new GRN-shape heads (`heads_w_gate/main/skip`) are different from the simpler V1 heads. At X9, X8's CfC weights are still in V1 shape too. The full V2 shape only becomes complete after X10 (forward kernel hoist) and X11 (capture_graph_a expansion). + +- [ ] P.1..P.7 with the GRN head fields. + +Commit: "refactor(ml-alpha): move GRN heads weights from PerceptionTrainer to CfcTrunk". + +--- + +## Task X10: Hoist forward kernels into CfcTrunk methods + +**Spec:** §1.1 (per-kernel forward helpers, called by trainer's evaluate_batched + by the graph capture). + +**Why:** With weights now owned by the trunk, the forward kernel launches naturally live on the trunk too. PerceptionTrainer.evaluate_batched becomes a thin wrapper that calls `self.trunk.forward_vsn(...)`, etc. Training-time backward kernels stay at trainer level (they need grad tensors which the trunk doesn't own). + +**Files:** `crates/ml-alpha/src/cfc/trunk.rs`, `crates/ml-alpha/src/trainer/perception.rs` + +### Step X10.1: Identify each forward kernel call in PerceptionTrainer.evaluate_batched + +- [ ] In `perception.rs`, find `pub fn evaluate_batched`. Trace the forward chain: each `function.launch(...)` call inside the function maps to a forward kernel. +- [ ] List the kernels in order: `snap_feature_assemble` → `vsn_forward` → `mamba2_step_forward (stack 1)` → `layernorm_forward (LN_a)` → `mamba2_step_forward (stack 2)` → `layernorm_forward (LN_b)` → `attention_pool_forward` → `cfc_step_forward` → `multi_horizon_heads_forward` → `projection_forward`. + +### Step X10.2: Add `pub fn forward_` methods on `CfcTrunk` + +For each kernel in the chain, add a method on `CfcTrunk` that launches it. The method takes input/output device pointers as arguments (so it can be called both in uncaptured mode by trainer and in captured mode by `perception_forward_captured`). + +Example for VSN: + +```rust +impl CfcTrunk { + /// Forward-launch the VSN kernel. Reads from `input_d`, writes to + /// `output_d`. Used by PerceptionTrainer's uncaptured forward and + /// also the captured graph in `capture_graph_a` (X11). + pub fn forward_vsn( + &self, + input_d: &CudaSlice, + output_d: &mut CudaSlice, + ) -> Result<()> { + let grid = (1, 1, 1); + let block = (self.cfg.n_in.next_power_of_two() as u32, 1, 1); + self.stream.launch_builder(&self.vsn_fn) + .arg(input_d).arg(output_d) + .arg(&self.vsn_w_d).arg(&self.vsn_b_d) + .arg(&(self.cfg.n_in as u32)) + .launch(grid, block, 0) + .context("vsn forward launch")?; + Ok(()) + } +} +``` + +Repeat for each kernel. Method names: `forward_vsn`, `forward_mamba2_l1`, `forward_ln_a`, `forward_mamba2_l2`, `forward_ln_b`, `forward_attn_pool`, `forward_cfc_step`, `forward_heads`, `forward_projection`. + +### Step X10.3: Update `evaluate_batched` to delegate + +- [ ] Inside `evaluate_batched`, replace each kernel `function.launch(...)` call with the corresponding `self.trunk.forward_(...)` call. Internal device-tensor buffers (intermediate activations, `snap_feat_d`, `vsn_out_d`, etc.) stay at trainer level. + +### Step X10.4: Verify golden + ml-alpha test suite + +- [ ] Run both: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test perception_forward_golden -- --ignored --nocapture +SQLX_OFFLINE=true cargo test -p ml-alpha --lib +``` + +Expected: both pass. + +### Step X10.5: Commit + +```bash +git add crates/ml-alpha/src/cfc/trunk.rs crates/ml-alpha/src/trainer/perception.rs +git commit -m "refactor(ml-alpha): hoist forward kernels into CfcTrunk methods + +PerceptionTrainer.evaluate_batched now delegates each kernel launch to +self.trunk.forward_(input_d, output_d). Trunk owns weight launches; +trainer owns intermediate-activation buffers and backward passes. + +Verified bit-equivalence vs perception_forward_golden fixture. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.1, §2.2." +``` + +--- + +## Task X11: CfcTrunk capture_graph_a covers full v2 forward + +**Spec:** §1.1 (capture_graph_a includes vsn → mamba2 → ln → mamba2 → ln → attn → cfc → heads), §2.2 (X11 — NEW captured-vs-uncaptured bit-equivalence test), §4.2 (failure modes — Mamba2 in captured graph). + +**Why:** The existing `capture_graph_a` in trunk.rs captures only `snap_features → cfc_step → multi_horizon_heads → projection`. Expand it to include the full v2 chain. Per `pearl_no_host_branches_in_captured_graph`, no host-side branches inside the capture region. + +**Files:** `crates/ml-alpha/src/cfc/trunk.rs`, `crates/ml-alpha/tests/perception_capture_equivalence.rs` (new) + +### Step X11.1: Allocate device buffers for intermediate activations on the trunk + +- [ ] In `trunk.rs`, add `CudaSlice` fields for each intermediate activation that flows between kernels in the captured chain (e.g., `vsn_out_d`, `mamba2_l1_out_d`, `ln_a_out_d`, `mamba2_l2_out_d`, `ln_b_out_d`, `attn_out_d`). Size each according to its kernel output shape. + +These are NEW trunk-owned intermediate buffers, separate from the trainer's intermediates (the trainer's are needed for backward; the trunk's are inference-only). + +### Step X11.2: Extend `capture_graph_a` + +- [ ] Find `pub fn capture_graph_a` in `trunk.rs`. Inside the capture region (between `stream.begin_capture` and `stream.end_capture`), insert the full forward chain by calling the `self.forward_` helpers from X10: + +```rust +// (inside capture region) +self.forward_snap_feature(input_buffer_d, &mut self.snap_feat_d)?; +self.forward_vsn(&self.snap_feat_d, &mut self.vsn_out_d)?; +self.forward_mamba2_l1(&self.vsn_out_d, &mut self.mamba2_l1_out_d)?; +self.forward_ln_a(&self.mamba2_l1_out_d, &mut self.ln_a_out_d)?; +self.forward_mamba2_l2(&self.ln_a_out_d, &mut self.mamba2_l2_out_d)?; +self.forward_ln_b(&self.mamba2_l2_out_d, &mut self.ln_b_out_d)?; +self.forward_attn_pool(&self.ln_b_out_d, &mut self.attn_out_d)?; +self.forward_cfc_step(&self.attn_out_d, /* ... */)?; +self.forward_heads(/* ... */)?; +self.forward_projection(/* ... */)?; +``` + +Per `pearl_cudarc_disable_event_tracking_for_graph_capture`, bracket the capture region with stream's event-tracking disable/enable. + +### Step X11.3: Verify no host branches inside the capture region + +- [ ] Read every `forward_` method. Each MUST be a pure launch sequence with NO `?` on a non-launch call (e.g., no `stream.synchronize()`, no `stream.memcpy_*` calls, no allocation). Per `pearl_no_host_branches_in_captured_graph`, the only `?`-points should be on the kernel launch itself. + +If any kernel needs intermediate CPU computation, that's a refactor: move the computation into a kernel launched inside the captured region, or pre-compute it before `begin_capture`. + +### Step X11.4: Write the captured-vs-uncaptured bit-equivalence test + +- [ ] Create `crates/ml-alpha/tests/perception_capture_equivalence.rs`: + +```rust +//! Bit-equivalence between captured-graph and uncaptured forward. +//! +//! Gates X11. Captured replay must produce identical output to the +//! direct kernel chain on the same input. + +use anyhow::{Context, Result}; +use ml_alpha::cfc::trunk::{CfcConfig, CfcTrunk}; +use ml_alpha::cfc::snap_features::Mbp10RawInput; +use ml_alpha::heads::N_HORIZONS; +use ml_core::device::MlDevice; + +fn deterministic_input() -> Mbp10RawInput { + // Same as perception_forward_golden's first snapshot for consistency. + Mbp10RawInput { + bid_px: [100.0, 99.75, 99.5, 99.25, 99.0, 98.75, 98.5, 98.25, 98.0, 97.75], + bid_sz: [10.0; 10], + ask_px: [100.25, 100.5, 100.75, 101.0, 101.25, 101.5, 101.75, 102.0, 102.25, 102.5], + ask_sz: [10.0; 10], + mid: 100.125, + ts_ns: 1_000_000_000, + regime: [0.0; 4], + trade_signed_vol: 0.0, + } +} + +#[test] +#[ignore = "GPU required"] +fn captured_matches_uncaptured() -> Result<()> { + let dev = MlDevice::new(0).context("init MlDevice")?; + let cfg = CfcConfig::default(); + let input = deterministic_input(); + + // Uncaptured: build trunk, call each forward_ directly, collect outputs. + let mut trunk_a = CfcTrunk::new_random(&dev, &cfg, 42)?; + trunk_a.update_input_buffers(&input)?; + // Drive the full chain manually by replicating the capture_graph_a body + // (without the capture wrapper). Collect the final per-horizon probs. + let probs_uncaptured: [f32; N_HORIZONS] = run_uncaptured_chain(&mut trunk_a)?; + + // Captured: build a fresh trunk, capture_graph_a, perception_forward_captured. + let mut trunk_b = CfcTrunk::new_random(&dev, &cfg, 42)?; + trunk_b.capture_graph_a(&input)?; + trunk_b.update_input_buffers(&input)?; + let (probs_captured, _) = trunk_b.perception_forward_captured()?; + + let max_diff: f32 = probs_uncaptured.iter().zip(probs_captured.iter()) + .map(|(a, b)| (a - b).abs()) + .fold(0.0, f32::max); + assert!(max_diff < 1e-5, + "captured-vs-uncaptured max_abs_diff {} >= tolerance 1e-5", max_diff); + Ok(()) +} + +fn run_uncaptured_chain(trunk: &mut CfcTrunk) -> Result<[f32; N_HORIZONS]> { + // Call each forward_ method on the trunk in sequence, + // mirroring capture_graph_a's body. Collect final probs via the + // existing mapped-pinned readback path used by perception_forward_captured. + // Implementation: copy the body of capture_graph_a, swap out the + // capture wrapper, return the final probs. + todo!("Spell out the explicit kernel chain in the test body, do not call capture_graph_a") +} +``` + +Spell out the kernel chain in `run_uncaptured_chain` rather than calling `capture_graph_a` — the whole point is to compare against an independent path. + +### Step X11.5: Verify both tests pass + +- [ ] Run on a GPU host: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --test perception_capture_equivalence -- --ignored --nocapture +SQLX_OFFLINE=true cargo test -p ml-alpha --test perception_forward_golden -- --ignored --nocapture +``` + +Expected: both pass. + +### Step X11.6: Commit + +```bash +git add crates/ml-alpha/src/cfc/trunk.rs crates/ml-alpha/tests/perception_capture_equivalence.rs +git commit -m "feat(ml-alpha): CfcTrunk capture_graph_a covers full v2 forward + +Extends the captured graph from snap+cfc+heads+proj to the full v2 chain +(VSN, Mamba2 ×2, LN ×2, attn-pool, CfC, heads, projection). New +captured-vs-uncaptured bit-equivalence test gates this commit. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.1, §2.2. +Per pearl_no_host_branches_in_captured_graph." +``` + +--- + +## Task X12: CheckpointV2 envelope + save_checkpoint + load_checkpoint + +**Spec:** §1.2 (CheckpointV2 fields), §2.2 (X12 round-trip test). + +**Files:** `crates/ml-alpha/src/cfc/trunk.rs` + +### Step X12.1: Replace `CheckpointV1` with `CheckpointV2` + +- [ ] In `trunk.rs`, find `struct CheckpointV1` (~line 27). Replace with: + +```rust +#[derive(Clone, Debug, Serialize, Deserialize)] +struct CheckpointV2 { + version: u32, // 2 + n_in: usize, + n_hid: usize, + mamba2_state_dim: usize, + // VSN + vsn_w: Vec, vsn_b: Vec, + // Mamba2 stack 1 + m1_w_in: Vec, m1_w_a: Vec, m1_w_b: Vec, + m1_w_c: Vec, m1_w_out: Vec, + // LN_a + ln_a_gain: Vec, ln_a_bias: Vec, + // Mamba2 stack 2 + m2_w_in: Vec, m2_w_a: Vec, m2_w_b: Vec, + m2_w_c: Vec, m2_w_out: Vec, + // LN_b + ln_b_gain: Vec, ln_b_bias: Vec, + // Attention pool + attn_q: Vec, + // CfC + w_in: Vec, w_rec: Vec, b: Vec, tau: Vec, + // Heads (GRN structure × 5 horizons) + heads_w_gate: Vec, heads_b_gate: Vec, + heads_w_main: Vec, heads_b_main: Vec, + heads_w_skip: Vec, heads_b_skip: Vec, +} +``` + +### Step X12.2: Rewrite `save_checkpoint` + +- [ ] Find `pub fn save_checkpoint` in `trunk.rs`. Replace its body to memcpy each new weight tensor and pack into a `CheckpointV2`: + +```rust +pub fn save_checkpoint(&self, path: &std::path::Path) -> Result<()> { + let download = |slice: &CudaSlice, n: usize| -> Result> { + let mut v = vec![0.0_f32; n]; + self.stream.memcpy_dtoh(slice, v.as_mut_slice())?; + Ok(v) + }; + let m1 = &self.mamba2_stack_1; + let m2 = &self.mamba2_stack_2; + let mid = ml_alpha::heads::HEAD_MID; + let ckpt = CheckpointV2 { + version: 2, + n_in: self.cfg.n_in, + n_hid: self.cfg.n_hid, + mamba2_state_dim: self.cfg.mamba2_state_dim, + vsn_w: download(&self.vsn_w_d, self.cfg.n_in * self.cfg.n_in)?, + vsn_b: download(&self.vsn_b_d, self.cfg.n_in)?, + m1_w_in: download(&m1.w_in, m1.config().in_dim * m1.config().hidden_dim)?, + m1_w_a: download(&m1.w_a, m1.config().hidden_dim * m1.config().state_dim)?, + m1_w_b: download(&m1.w_b, m1.config().hidden_dim * m1.config().state_dim)?, + m1_w_c: download(&m1.w_c, m1.config().hidden_dim * m1.config().state_dim)?, + m1_w_out: download(&m1.w_out, m1.config().hidden_dim)?, + ln_a_gain: download(&self.ln_a_gain_d, self.cfg.n_hid)?, + ln_a_bias: download(&self.ln_a_bias_d, self.cfg.n_hid)?, + m2_w_in: download(&m2.w_in, m2.config().in_dim * m2.config().hidden_dim)?, + m2_w_a: download(&m2.w_a, m2.config().hidden_dim * m2.config().state_dim)?, + m2_w_b: download(&m2.w_b, m2.config().hidden_dim * m2.config().state_dim)?, + m2_w_c: download(&m2.w_c, m2.config().hidden_dim * m2.config().state_dim)?, + m2_w_out: download(&m2.w_out, m2.config().hidden_dim)?, + ln_b_gain: download(&self.ln_b_gain_d, self.cfg.n_hid)?, + ln_b_bias: download(&self.ln_b_bias_d, self.cfg.n_hid)?, + attn_q: download(&self.attn_q_d, self.cfg.n_hid)?, + w_in: download(&self.w_in_d, self.cfg.n_hid * self.cfg.n_in)?, + w_rec: download(&self.w_rec_d, self.cfg.n_hid * self.cfg.n_hid)?, + b: download(&self.b_d, self.cfg.n_hid)?, + tau: download(&self.tau_d, self.cfg.n_hid)?, + heads_w_gate: download(&self.heads_w_gate_d, N_HORIZONS * mid)?, + heads_b_gate: download(&self.heads_b_gate_d, N_HORIZONS)?, + heads_w_main: download(&self.heads_w_main_d, N_HORIZONS * mid)?, + heads_b_main: download(&self.heads_b_main_d, N_HORIZONS)?, + heads_w_skip: download(&self.heads_w_skip_d, N_HORIZONS * self.cfg.n_hid)?, + heads_b_skip: download(&self.heads_b_skip_d, N_HORIZONS)?, + }; + let bytes = bincode::serialize(&ckpt).context("bincode serialize CheckpointV2")?; + let mut f = std::fs::File::create(path).context("open checkpoint file for write")?; + use std::io::Write; + f.write_all(&bytes).context("write checkpoint bytes")?; + Ok(()) +} +``` + +If `Mamba2Block` doesn't expose `w_in`, `w_a`, `w_b`, `w_c`, `w_out`, `config()` as public, add the necessary public accessors in `mamba2_block.rs` as part of this commit (separate commit if it grows large). + +### Step X12.3: Rewrite `load_checkpoint` + +- [ ] Find `pub fn load_checkpoint` in `trunk.rs`. Replace body: + +```rust +pub fn load_checkpoint( + dev: &MlDevice, + cfg: &CfcConfig, + path: &std::path::Path, +) -> Result { + let bytes = std::fs::read(path).with_context(|| format!("read checkpoint {}", path.display()))?; + let ckpt: CheckpointV2 = bincode::deserialize(&bytes).context("bincode deserialize CheckpointV2")?; + anyhow::ensure!(ckpt.version == 2, + "unsupported checkpoint version {} (expected 2)", ckpt.version); + anyhow::ensure!(ckpt.n_in == cfg.n_in, + "checkpoint n_in={} != cfg.n_in={}", ckpt.n_in, cfg.n_in); + anyhow::ensure!(ckpt.n_hid == cfg.n_hid, + "checkpoint n_hid={} != cfg.n_hid={}", ckpt.n_hid, cfg.n_hid); + anyhow::ensure!(ckpt.mamba2_state_dim == cfg.mamba2_state_dim, + "checkpoint mamba2_state_dim={} != cfg.mamba2_state_dim={}", + ckpt.mamba2_state_dim, cfg.mamba2_state_dim); + + // Construct via new_random then overwrite each weight tensor via memcpy_htod. + let mut trunk = Self::new_random(dev, cfg, 0)?; + let stream = trunk.stream.clone(); + let upload = |slice: &mut CudaSlice, src: &[f32]| -> Result<()> { + anyhow::ensure!(slice.len() == src.len(), + "size mismatch: device {} != host {}", slice.len(), src.len()); + stream.memcpy_htod(src, slice)?; + Ok(()) + }; + upload(&mut trunk.vsn_w_d, &ckpt.vsn_w)?; + upload(&mut trunk.vsn_b_d, &ckpt.vsn_b)?; + upload(&mut trunk.mamba2_stack_1.w_in, &ckpt.m1_w_in)?; + upload(&mut trunk.mamba2_stack_1.w_a, &ckpt.m1_w_a)?; + upload(&mut trunk.mamba2_stack_1.w_b, &ckpt.m1_w_b)?; + upload(&mut trunk.mamba2_stack_1.w_c, &ckpt.m1_w_c)?; + upload(&mut trunk.mamba2_stack_1.w_out, &ckpt.m1_w_out)?; + upload(&mut trunk.ln_a_gain_d, &ckpt.ln_a_gain)?; + upload(&mut trunk.ln_a_bias_d, &ckpt.ln_a_bias)?; + upload(&mut trunk.mamba2_stack_2.w_in, &ckpt.m2_w_in)?; + upload(&mut trunk.mamba2_stack_2.w_a, &ckpt.m2_w_a)?; + upload(&mut trunk.mamba2_stack_2.w_b, &ckpt.m2_w_b)?; + upload(&mut trunk.mamba2_stack_2.w_c, &ckpt.m2_w_c)?; + upload(&mut trunk.mamba2_stack_2.w_out, &ckpt.m2_w_out)?; + upload(&mut trunk.ln_b_gain_d, &ckpt.ln_b_gain)?; + upload(&mut trunk.ln_b_bias_d, &ckpt.ln_b_bias)?; + upload(&mut trunk.attn_q_d, &ckpt.attn_q)?; + upload(&mut trunk.w_in_d, &ckpt.w_in)?; + upload(&mut trunk.w_rec_d, &ckpt.w_rec)?; + upload(&mut trunk.b_d, &ckpt.b)?; + upload(&mut trunk.tau_d, &ckpt.tau)?; + upload(&mut trunk.heads_w_gate_d, &ckpt.heads_w_gate)?; + upload(&mut trunk.heads_b_gate_d, &ckpt.heads_b_gate)?; + upload(&mut trunk.heads_w_main_d, &ckpt.heads_w_main)?; + upload(&mut trunk.heads_b_main_d, &ckpt.heads_b_main)?; + upload(&mut trunk.heads_w_skip_d, &ckpt.heads_w_skip)?; + upload(&mut trunk.heads_b_skip_d, &ckpt.heads_b_skip)?; + Ok(trunk) +} +``` + +### Step X12.4: Write the failing round-trip test + +- [ ] At the bottom of `trunk.rs` in the existing `#[cfg(test)] mod tests`, add: + +```rust +#[test] +#[ignore = "GPU required"] +fn checkpoint_v2_save_load_round_trip_bit_exact() -> Result<()> { + let dev = MlDevice::new(0)?; + let cfg = CfcConfig::default(); + let trunk_a = CfcTrunk::new_random(&dev, &cfg, 42)?; + let tmp = tempfile::tempdir()?; + let path = tmp.path().join("ckpt.bin"); + trunk_a.save_checkpoint(&path)?; + let trunk_b = CfcTrunk::load_checkpoint(&dev, &cfg, &path)?; + // Bit-exact: every weight tensor matches after download. + let mut buf_a = vec![0.0_f32; cfg.n_in * cfg.n_in]; + let mut buf_b = vec![0.0_f32; cfg.n_in * cfg.n_in]; + trunk_a.stream.memcpy_dtoh(&trunk_a.vsn_w_d, &mut buf_a)?; + trunk_b.stream.memcpy_dtoh(&trunk_b.vsn_w_d, &mut buf_b)?; + assert_eq!(buf_a, buf_b, "vsn_w not bit-exact after round-trip"); + // (Repeat for each major weight tensor; abridged here for plan brevity. + // The actual test should check all tensors enumerated in CheckpointV2.) + Ok(()) +} + +#[test] +fn checkpoint_v1_rejected() { + // Forge a V1-versioned envelope and verify load_checkpoint errors. + let tmp = tempfile::tempdir().unwrap(); + let path = tmp.path().join("v1.bin"); + #[derive(Serialize)] + struct V1Stub { version: u32 } + let bytes = bincode::serialize(&V1Stub { version: 1 }).unwrap(); + std::fs::write(&path, &bytes).unwrap(); + let dev = MlDevice::new(0).expect("device"); + let result = CfcTrunk::load_checkpoint(&dev, &CfcConfig::default(), &path); + assert!(result.is_err(), "V1 envelope should be rejected"); + let err = result.err().unwrap().to_string(); + assert!(err.contains("version 1") || err.contains("expected 2"), + "error should mention version mismatch; got: {}", err); +} +``` + +### Step X12.5: Run both tests + +- [ ] On GPU host: `SQLX_OFFLINE=true cargo test -p ml-alpha --lib checkpoint_v2_save_load -- --ignored --nocapture` +- [ ] Without GPU: `SQLX_OFFLINE=true cargo test -p ml-alpha --lib checkpoint_v1_rejected` + +Expected: both pass. + +### Step X12.6: Commit + +```bash +git add crates/ml-alpha/src/cfc/trunk.rs +git commit -m "feat(ml-alpha): CheckpointV2 envelope + save_checkpoint + load_checkpoint + +Replaces CheckpointV1 (CfC + heads + projection only) with V2 envelope +covering the full v2 inference graph: VSN, Mamba2 ×2, LN ×2, attn-pool, +CfC, GRN heads. V1 envelopes hard-rejected; no migration needed +(alpha_train never produced any). + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.2." +``` + +--- + +## Task X13: PerceptionTrainer.save_checkpoint delegate + +**Spec:** §1.1 (PerceptionTrainer.save_checkpoint delegates to self.trunk.save_checkpoint). + +**Files:** `crates/ml-alpha/src/trainer/perception.rs` + +### Step X13.1: Add the delegate method + +- [ ] In `perception.rs`, in the `impl PerceptionTrainer` block, add: + +```rust +/// Save the trunk's weights to a CheckpointV2 file. Delegates to +/// CfcTrunk::save_checkpoint — gradient buffers and AdamW state are +/// NOT serialized (they're training-only and not needed for inference). +pub fn save_checkpoint(&self, path: &std::path::Path) -> Result<()> { + self.trunk.save_checkpoint(path).context("trunk save_checkpoint") +} +``` + +### Step X13.2: Write delegate round-trip test + +- [ ] In `crates/ml-alpha/src/trainer/perception.rs` `#[cfg(test)] mod tests`, add: + +```rust +#[test] +#[ignore = "GPU required"] +fn perception_trainer_save_checkpoint_round_trips_via_trunk() -> Result<()> { + let dev = MlDevice::new(0)?; + let cfg = PerceptionTrainerConfig { + seq_len: 32, n_batch: 1, mamba2_state_dim: 16, + ..Default::default() + }; + let trainer = PerceptionTrainer::new(&dev, &cfg)?; + let tmp = tempfile::tempdir()?; + let path = tmp.path().join("ckpt.bin"); + trainer.save_checkpoint(&path)?; + // Verify a trunk loaded from the same file recovers bit-exact weights. + let trunk_cfg = ml_alpha::cfc::trunk::CfcConfig { + n_in: ml_alpha::cfc::snap_features::FEATURE_DIM, + n_hid: ml_alpha::heads::HIDDEN_DIM, + mamba2_state_dim: cfg.mamba2_state_dim, + }; + let _trunk = ml_alpha::cfc::trunk::CfcTrunk::load_checkpoint(&dev, &trunk_cfg, &path)?; + Ok(()) +} +``` + +### Step X13.3: Run + commit + +- [ ] `cargo test` confirms test compiles and passes (on GPU host). + +```bash +git add crates/ml-alpha/src/trainer/perception.rs +git commit -m "feat(ml-alpha): PerceptionTrainer.save_checkpoint delegate + +Thin wrapper around self.trunk.save_checkpoint. Training-only state +(gradient buffers, AdamW) is NOT serialized — inference does not need it +and including it would bloat the checkpoint. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.1." +``` + +--- + +## Task X14: alpha_train saves best_h6000 checkpoint + +**Spec:** §1.1 (alpha_train extension), §1.4 (`best_h6000_ckpt_path` artifact). + +**Files:** `crates/ml-alpha/examples/alpha_train.rs` + +### Step X14.1: Add `Deserialize` derive + `best_h6000_ckpt_path` field + +- [ ] Open `crates/ml-alpha/examples/alpha_train.rs`. Find `#[derive(Serialize)]` on `AlphaTrainSummary` (line 167). Change to `#[derive(Serialize, Deserialize, Default)]`. + +- [ ] Add field at the bottom of the struct (after `early_stopped`): + +```rust +/// Path to the best-h6000 trunk checkpoint (relative to `out_dir`), +/// or None if no improvement was ever recorded. +#[serde(default, skip_serializing_if = "Option::is_none")] +best_h6000_ckpt_path: Option, +``` + +- [ ] Initialize as `None` when constructing `AlphaTrainSummary` (around line 634). + +### Step X14.2: Wire save_checkpoint call + +- [ ] Find the `if auc_h6000_improved` block (line 604). Replace its body with: + +```rust +if auc_h6000_improved { + best_auc_h6000 = auc_h6000; + best_auc_h6000_epoch = epoch; + best_auc_h6000_per_horizon = per_horizon_auc; + auc_h6000_no_improvement = 0; + let ckpt_path = cli.out.join("trunk_best_h6000.bin"); + trainer.save_checkpoint(&ckpt_path) + .context("save_checkpoint(trunk_best_h6000.bin)")?; + best_h6000_ckpt_path = Some("trunk_best_h6000.bin".to_string()); + tracing::info!(epoch, auc_h6000, path = %ckpt_path.display(), "new best auc_h6000 (saved)"); +} else { + auc_h6000_no_improvement += 1; +} +``` + +- [ ] Add a `let mut best_h6000_ckpt_path: Option = None;` declaration earlier in the function alongside the other `best_*` mutables (around line 310 area). +- [ ] Add `best_h6000_ckpt_path,` to the `AlphaTrainSummary { ... }` constructor at line 634. + +### Step X14.3: Write the summary serde round-trip test + +- [ ] At the bottom of `alpha_train.rs`, add: + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn summary_serde_round_trip_with_h6000_ckpt() { + let s = AlphaTrainSummary { + epochs: 10, + seq_len: 32, + horizons: [0; N_HORIZONS], + final_train_loss: 0.5, + final_val_loss: 0.6, + final_val_auc: [0.7; N_HORIZONS], + n_train_seqs_consumed: 1000, + n_val_seqs_consumed: 100, + best_epoch: 5, + best_val_loss: 0.55, + best_val_auc: [0.72; N_HORIZONS], + best_mean_auc_epoch: 6, + best_mean_auc: 0.73, + best_mean_auc_per_horizon: [0.73; N_HORIZONS], + best_auc_h6000_epoch: 7, + best_auc_h6000: 0.76, + best_auc_h6000_per_horizon: [0.74, 0.73, 0.74, 0.75, 0.76], + early_stopped: true, + best_h6000_ckpt_path: Some("trunk_best_h6000.bin".to_string()), + }; + let json = serde_json::to_string(&s).expect("serialize"); + let back: AlphaTrainSummary = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back.best_h6000_ckpt_path.as_deref(), Some("trunk_best_h6000.bin")); + assert_eq!(back.best_auc_h6000, 0.76); + } +} +``` + +### Step X14.4: Run tests, build, commit + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --example alpha_train summary_serde_round_trip_with_h6000_ckpt -- --nocapture +SQLX_OFFLINE=true cargo build -p ml-alpha --example alpha_train --release +``` + +Expected: test passes, example builds clean. + +- [ ] Commit: + +```bash +git add crates/ml-alpha/examples/alpha_train.rs +git commit -m "feat(ml-alpha): alpha_train saves best_h6000 checkpoint + +Inside the auc_h6000_improved block, calls trainer.save_checkpoint to +write trunk_best_h6000.bin to OUT_DIR. Extends AlphaTrainSummary with +best_h6000_ckpt_path so downstream tooling (fxt-backtest --checkpoint) +can locate the file without re-deriving the path. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.1, §1.4." +``` + +--- + +## Task X15: Verify ml-backtesting harness + fxt-backtest accept CheckpointV2 + +**Spec:** §1.3 (no interface change required). + +**Files:** Code unchanged. Verification only. + +### Step X15.1: Build the backtest binaries + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo build -p ml-backtesting +SQLX_OFFLINE=true cargo build -p ml-backtesting --bin fxt-backtest --release +``` + +Expected: builds clean. CfcTrunk's API (`load_checkpoint`, `capture_graph_a`, `update_input_buffers`, `perception_forward_captured`) is unchanged from the backtester's perspective — only the file format and the captured-graph internals grew. + +### Step X15.2: Run existing harness tests + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib +SQLX_OFFLINE=true cargo test -p ml-backtesting --tests +``` + +Expected: all existing tests pass. The CheckpointV2 envelope is transparent to harness code paths that don't manipulate raw V1 fields. + +### Step X15.3: Commit (verification-only commit) + +- [ ] Run: + +```bash +git commit --allow-empty -m "verify(ml-backtesting): harness + fxt-backtest accept CheckpointV2 + +CfcTrunk's public API (load_checkpoint, capture_graph_a, update_input_buffers, +perception_forward_captured) is unchanged from the backtester's perspective. +ml-backtesting unit + integration tests pass against V2 envelope produced +by X12. + +No code changes required at this commit. Marks the X15 verification +checkpoint in the plan. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §1.3." +``` + +--- + +## Task X16: max_drawdown_pct with $35k base + +**Spec:** §3.2 (metrics), `project_ml_alpha_starting_capital` memory. + +**Files:** `crates/ml-backtesting/src/artifacts.rs` + +### Step X16.1: Add `STARTING_CAPITAL_USD` constant + +- [ ] In `artifacts.rs`, near the existing `ANNUALISATION_SQRT_FACTOR`: + +```rust +/// Starting capital in USD used as the base for max_drawdown_pct. +/// $35k is the project's chosen base for ES single-contract analysis — +/// realistic for a small-account ES trader (margin ~$15k initial, +/// ~$20k headroom for adverse moves before a margin call). +pub const STARTING_CAPITAL_USD: f32 = 35_000.0; +``` + +### Step X16.2: Add `max_drawdown_pct` field to `Summary` + +- [ ] Find `pub struct Summary` in `artifacts.rs` (line 21). Add field after `max_drawdown_usd`: + +```rust +/// Maximum drawdown as a fraction of starting capital, per window. +/// Computed in compute_summary using STARTING_CAPITAL_USD as base. +/// Range [0.0, 1.0] (0.20 = 20% drawdown). +pub max_drawdown_pct: f32, +``` + +### Step X16.3: Populate `max_drawdown_pct` in `compute_summary` + +- [ ] Find `compute_summary` in `artifacts.rs`. After the line that computes `max_drawdown_usd` (search for the assignment), add: + +```rust +let max_drawdown_pct = max_drawdown_usd.abs() / STARTING_CAPITAL_USD; +``` + +- [ ] Add `max_drawdown_pct` to the `Summary { ... }` construction at the bottom of `compute_summary`. +- [ ] In the early-return branch (`if records.is_empty()`), set `max_drawdown_pct: 0.0` explicitly. + +### Step X16.4: Write the failing test first + +- [ ] In `artifacts.rs` test module, add: + +```rust +#[test] +fn max_drawdown_pct_against_fixture_curve() { + // P&L curve in USD: peak $7,000, trough $0 → $7,000 drawdown. + // STARTING_CAPITAL_USD = $35k → 20% drawdown (right on the + // deployability gate boundary — good edge-case fixture). + let curve = vec![0.0_f32, 3500.0, 7000.0, 4000.0, 0.0, 2000.0]; + let records = vec![TradeRecord { + realised_pnl_usd_fp: 200_000, // $2,000 in fp-USD-x100 + fees_usd_fp: 0, + ..Default::default() + }]; + let s = compute_summary(&records, &curve); + assert!((s.max_drawdown_pct - 0.20).abs() < 1e-4, + "expected max_drawdown_pct=0.20, got {}", s.max_drawdown_pct); +} +``` + +If `TradeRecord` doesn't impl `Default`, construct it explicitly with all required fields (read the struct definition once before writing this test). + +### Step X16.5: Run test (verify it fails first, then passes) + +- [ ] First commit only the test, then run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib max_drawdown_pct_against_fixture_curve -- --nocapture +``` + +Expected: FAIL with "no field `max_drawdown_pct`". + +- [ ] Then apply Steps X16.1–X16.3. Re-run. Expected: PASS. + +### Step X16.6: Extend the existing round-trip test + +- [ ] Find `write_summary_then_parse_back` in `artifacts.rs` (around line 275). Add assertion: + +```rust +assert!((parsed.max_drawdown_pct - written.max_drawdown_pct).abs() < 1e-6); +``` + +- [ ] Re-run: `SQLX_OFFLINE=true cargo test -p ml-backtesting --lib write_summary_then_parse_back -- --nocapture`. PASS. + +### Step X16.7: Full ml-backtesting test suite + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib +``` + +Expected: all green. + +### Step X16.8: Commit + +```bash +git add crates/ml-backtesting/src/artifacts.rs +git commit -m "feat(ml-backtesting): max_drawdown_pct with \$35k base + +Adds max_drawdown_pct field on per-cell Summary as +|max_drawdown_usd| / STARTING_CAPITAL_USD (pinned at \$35k per +project_ml_alpha_starting_capital memory — realistic ES single-contract +small-account anchor). Used by emit_deployability_verdict as the +capital-deployability hard gate (median across windows < 20%). + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §3.2." +``` + +--- + +## Task X17: emit_deployability_verdict + tiered logic + +**Spec:** §3.5 (verdict emitter pseudocode), §3.2 (metrics), §3.1 (anchors). + +**Files:** `crates/ml-backtesting/src/aggregate.rs`, `crates/ml-backtesting/Cargo.toml` + +### Step X17.1: Add chrono dep + serde derives + +- [ ] In `crates/ml-backtesting/Cargo.toml`, ensure `chrono = { workspace = true }` is in `[dependencies]` and `tempfile = { workspace = true }` is in `[dev-dependencies]`. + +### Step X17.2: Add verdict types + +- [ ] In `crates/ml-backtesting/src/aggregate.rs`, after imports, add: + +```rust +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum VerdictTier { + PassRobust, + PassNominal, + FailInconclusive, + Fail, + FailDegenerate, +} + +#[derive(Clone, Copy, Debug, Serialize, Deserialize)] +pub struct AnchorSpec { + pub name: &'static str, + pub cost_tick: f32, + pub latency_ms: u32, +} + +pub const ANCHOR_REALISTIC: AnchorSpec = AnchorSpec { + name: "realistic", cost_tick: 1.0, latency_ms: 200, +}; +pub const ANCHOR_STRESS: AnchorSpec = AnchorSpec { + name: "stress", cost_tick: 1.5, latency_ms: 400, +}; + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct AnchorReport { + pub anchor: AnchorSpec, + pub sharpe_per_window: Vec, + pub sortino_per_window: Vec, + pub max_dd_pct_per_window: Vec, + pub profit_factor_per_window: Vec, + pub median_sharpe: f32, + pub median_sortino: f32, + pub median_max_dd_pct: f32, + pub median_profit_factor: f32, + pub gate_sharpe: bool, + pub gate_max_dd: bool, + pub pass: bool, + pub status: String, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct DeployabilityVerdict { + pub verdict: VerdictTier, + pub realistic: AnchorReport, + pub stress: AnchorReport, + pub threshold: f32, + pub windows: Vec, + pub training_sha: String, + pub spec_sha: String, + pub timestamp_utc: String, +} +``` + +### Step X17.3: Implement `classify_verdict` + +```rust +pub fn classify_verdict(realistic: &AnchorReport, stress: &AnchorReport) -> VerdictTier { + if realistic.status != "ok" { + return VerdictTier::FailDegenerate; + } + if realistic.pass && stress.status == "ok" && stress.pass { + return VerdictTier::PassRobust; + } + if realistic.pass { + return VerdictTier::PassNominal; + } + let sharpe_grey = realistic.median_sharpe >= 0.8 && realistic.median_sharpe < 1.0; + let max_dd_grey = realistic.median_max_dd_pct >= 0.20 && realistic.median_max_dd_pct < 0.25; + if sharpe_grey || max_dd_grey { + return VerdictTier::FailInconclusive; + } + VerdictTier::Fail +} +``` + +### Step X17.4: Implement `emit_deployability_verdict` + +```rust +use chrono::Utc; +use std::collections::BTreeMap; +use std::fs; + +pub fn emit_deployability_verdict( + sweep_dir: &Path, + threshold: f32, + windows: &[&str], + training_sha: &str, + spec_sha: &str, +) -> Result { + let realistic = build_anchor_report(sweep_dir, ANCHOR_REALISTIC, threshold, windows)?; + let stress = build_anchor_report(sweep_dir, ANCHOR_STRESS, threshold, windows)?; + let verdict = classify_verdict(&realistic, &stress); + Ok(DeployabilityVerdict { + verdict, + realistic, + stress, + threshold, + windows: windows.iter().map(|s| s.to_string()).collect(), + training_sha: training_sha.to_string(), + spec_sha: spec_sha.to_string(), + timestamp_utc: Utc::now().to_rfc3339(), + }) +} + +fn build_anchor_report( + sweep_dir: &Path, + anchor: AnchorSpec, + threshold: f32, + windows: &[&str], +) -> Result { + let mut by_window: BTreeMap = BTreeMap::new(); + for entry in fs::read_dir(sweep_dir) + .with_context(|| format!("read_dir {}", sweep_dir.display()))? + { + let path = entry?.path(); + let name = path.file_name().and_then(|n| n.to_str()).unwrap_or("").to_string(); + let (cell_cost, cell_lat, cell_th, cell_win) = match parse_cell_name(&name) { + Some(t) => t, + None => continue, + }; + if (cell_cost - anchor.cost_tick).abs() > 1e-6 { continue; } + if cell_lat != anchor.latency_ms { continue; } + if (cell_th - threshold).abs() > 1e-6 { continue; } + if !windows.iter().any(|w| **w == cell_win) { continue; } + let summary_path = path.join("summary.json"); + let s: Summary = serde_json::from_reader( + fs::File::open(&summary_path) + .with_context(|| format!("open {}", summary_path.display()))? + )?; + by_window.insert(cell_win.clone(), s); + } + + let mut sharpe = Vec::new(); + let mut sortino = Vec::new(); + let mut max_dd = Vec::new(); + let mut pf = Vec::new(); + let mut status = String::from("ok"); + for w in windows { + let s = by_window.get(*w).ok_or_else(|| + anyhow::anyhow!("missing cell for anchor={} window={}", anchor.name, w) + )?; + if s.n_trades == 0 || !s.sharpe_ann.is_finite() || !s.sortino_ann.is_finite() { + status = format!("fail-degenerate: {w} trades={} sharpe={}", s.n_trades, s.sharpe_ann); + } + sharpe.push(s.sharpe_ann); + sortino.push(s.sortino_ann); + max_dd.push(s.max_drawdown_pct); + pf.push(s.profit_factor); + } + let median_sharpe = median(&sharpe); + let median_sortino = median(&sortino); + let median_max_dd_pct = median(&max_dd); + let median_profit_factor = median(&pf); + let gate_sharpe = median_sharpe > 1.0; + let gate_max_dd = median_max_dd_pct < 0.20; + let pass = gate_sharpe && gate_max_dd && status == "ok"; + Ok(AnchorReport { + anchor, + sharpe_per_window: sharpe, + sortino_per_window: sortino, + max_dd_pct_per_window: max_dd, + profit_factor_per_window: pf, + median_sharpe, + median_sortino, + median_max_dd_pct, + median_profit_factor, + gate_sharpe, + gate_max_dd, + pass, + status, + }) +} + +fn median(xs: &[f32]) -> f32 { + let mut v = xs.to_vec(); + v.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal)); + let n = v.len(); + if n == 0 { return f32::NAN; } + if n % 2 == 1 { v[n / 2] } else { 0.5 * (v[n / 2 - 1] + v[n / 2]) } +} + +fn parse_cell_name(name: &str) -> Option<(f32, u32, f32, String)> { + let parts: Vec<&str> = name.split('_').collect(); + if parts.len() < 5 || parts[0] != "cell" { return None; } + let cost = parts.iter().find(|p| p.starts_with("cost"))?[4..].parse().ok()?; + let lat: u32 = parts.iter().find(|p| p.starts_with("lat"))?[3..].parse().ok()?; + let th: f32 = parts.iter().find(|p| p.starts_with("th"))?[2..].parse().ok()?; + let win = parts.iter().rev().find(|p| p.starts_with('W'))?.to_string(); + Some((cost, lat, th, win)) +} +``` + +### Step X17.5: Write the 5 tier-classification tests + fixture sweep test + +- [ ] In the test module of `aggregate.rs`, add (each test is its own `#[test]` fn): + +```rust +fn stub_anchor_report(anchor: AnchorSpec) -> AnchorReport { + AnchorReport { + anchor, sharpe_per_window: vec![], sortino_per_window: vec![], + max_dd_pct_per_window: vec![], profit_factor_per_window: vec![], + median_sharpe: 0.0, median_sortino: 0.0, + median_max_dd_pct: 0.0, median_profit_factor: 0.0, + gate_sharpe: false, gate_max_dd: false, pass: false, + status: "ok".to_string(), + } +} + +#[test] +fn verdict_pass_robust_when_both_anchors_pass_both_gates() { + let realistic = AnchorReport { + median_sharpe: 1.5, median_max_dd_pct: 0.10, + gate_sharpe: true, gate_max_dd: true, pass: true, + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = AnchorReport { + median_sharpe: 1.2, median_max_dd_pct: 0.15, + gate_sharpe: true, gate_max_dd: true, pass: true, + ..stub_anchor_report(ANCHOR_STRESS) + }; + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::PassRobust); +} + +#[test] +fn verdict_pass_nominal_when_only_realistic_passes() { + let realistic = AnchorReport { + median_sharpe: 1.5, median_max_dd_pct: 0.10, + gate_sharpe: true, gate_max_dd: true, pass: true, + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = AnchorReport { + median_sharpe: 0.7, median_max_dd_pct: 0.30, + gate_sharpe: false, gate_max_dd: false, pass: false, + ..stub_anchor_report(ANCHOR_STRESS) + }; + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::PassNominal); +} + +#[test] +fn verdict_fail_inconclusive_when_sharpe_in_grey() { + let realistic = AnchorReport { + median_sharpe: 0.9, median_max_dd_pct: 0.10, + gate_sharpe: false, gate_max_dd: true, pass: false, + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = stub_anchor_report(ANCHOR_STRESS); + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::FailInconclusive); +} + +#[test] +fn verdict_fail_inconclusive_when_max_dd_in_grey() { + let realistic = AnchorReport { + median_sharpe: 1.5, median_max_dd_pct: 0.22, + gate_sharpe: true, gate_max_dd: false, pass: false, + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = stub_anchor_report(ANCHOR_STRESS); + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::FailInconclusive); +} + +#[test] +fn verdict_fail_when_realistic_misses_by_a_lot() { + let realistic = AnchorReport { + median_sharpe: 0.3, median_max_dd_pct: 0.40, + gate_sharpe: false, gate_max_dd: false, pass: false, + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = stub_anchor_report(ANCHOR_STRESS); + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::Fail); +} + +#[test] +fn verdict_fail_degenerate_when_realistic_status_not_ok() { + let realistic = AnchorReport { + status: "fail-degenerate: W2 had 0 trades".to_string(), + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = stub_anchor_report(ANCHOR_STRESS); + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::FailDegenerate); +} + +#[test] +fn emit_deployability_verdict_against_fixture_sweep() { + let tmp = tempfile::tempdir().expect("tmpdir"); + let realistic = [("W1", 1.4_f32, 0.10), ("W2", 1.6, 0.12), + ("W3", 1.2, 0.08), ("W4", 1.5, 0.11)]; + let stress = [("W1", 0.4_f32, 0.30), ("W2", 0.6, 0.28), + ("W3", 0.5, 0.32), ("W4", 0.7, 0.29)]; + for (w, s, dd) in &realistic { write_fixture_cell(tmp.path(), 1.0, 200, 0.75, w, *s, *dd); } + for (w, s, dd) in &stress { write_fixture_cell(tmp.path(), 1.5, 400, 0.75, w, *s, *dd); } + let v = emit_deployability_verdict( + tmp.path(), 0.75, &["W1","W2","W3","W4"], "test-sha", "test-spec-sha" + ).expect("verdict"); + assert_eq!(v.verdict, VerdictTier::PassNominal); + assert!(v.realistic.pass); + assert!(!v.stress.pass); +} + +fn write_fixture_cell( + root: &Path, cost: f32, lat: u32, th: f32, window: &str, + sharpe: f32, max_dd_pct: f32, +) { + use crate::artifacts::Summary; + let dir = root.join(format!("cell_0000_cost{cost:.2}_lat{lat}_th{th:.2}_{window}")); + std::fs::create_dir_all(&dir).unwrap(); + let s = Summary { + total_pnl_usd: 10_000.0, sharpe_ann: sharpe, sortino_ann: sharpe * 1.2, + max_drawdown_usd: max_dd_pct * 35_000.0, max_drawdown_pct: max_dd_pct, + calmar: 0.0, n_trades: 100, win_rate: 0.55, + avg_win_usd: 200.0, avg_loss_usd: -150.0, profit_factor: 1.3, + total_fees_usd: 500.0, exposure_pct: 0.3, + kelly_cap_history_sample: vec![], + }; + serde_json::to_writer(std::fs::File::create(dir.join("summary.json")).unwrap(), &s).unwrap(); +} + +#[test] +fn deployability_verdict_serde_round_trip() { + let v = DeployabilityVerdict { + verdict: VerdictTier::PassRobust, + realistic: stub_anchor_report(ANCHOR_REALISTIC), + stress: stub_anchor_report(ANCHOR_STRESS), + threshold: 0.75, + windows: vec!["W1".into(), "W2".into(), "W3".into(), "W4".into()], + training_sha: "abc1234".into(), + spec_sha: "da1dd92bf".into(), + timestamp_utc: "2026-05-19T12:00:00Z".into(), + }; + let json = serde_json::to_string(&v).expect("serialize"); + let back: DeployabilityVerdict = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back.verdict, VerdictTier::PassRobust); + assert_eq!(back.threshold, 0.75); +} +``` + +### Step X17.6: Run all tests + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib verdict_ -- --nocapture +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib emit_deployability_verdict_against_fixture_sweep -- --nocapture +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib deployability_verdict_serde_round_trip -- --nocapture +``` + +Expected: all green. + +### Step X17.7: Commit + +```bash +git add crates/ml-backtesting/src/aggregate.rs crates/ml-backtesting/Cargo.toml +git commit -m "feat(ml-backtesting): emit_deployability_verdict + tiered logic + +Adds DeployabilityVerdict / AnchorReport / VerdictTier types, +classify_verdict (5-tier logic), and emit_deployability_verdict +(reads per-cell summary.json files at both anchors, computes medians, +classifies). Unit-tested via fixture sweep + 6 classify_verdict cases +covering each tier. + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md §3.5." +``` + +--- + +## Task X18: GPU smoke test against real trained checkpoint + +**Spec:** §3.4 (smoke gate criteria), §4.1 (test inventory). + +**Files:** `crates/ml-backtesting/tests/checkpoint_smoke.rs` + +### Step X18.1: Create the smoke test file + +- [ ] Create `crates/ml-backtesting/tests/checkpoint_smoke.rs`: + +```rust +//! GPU integration smoke: load a real trunk checkpoint, run one backtest +//! cell against a fixture MBP-10 window, assert spec §3.4 smoke criteria. +//! +//! Runtime invocation (on a CUDA-capable host with a real trained +//! checkpoint and MBP-10 fixture present): +//! +//! FOXHUNT_SMOKE_CKPT=/feature-cache/alpha-perception-runs//trunk_best_h6000.bin \ +//! FOXHUNT_SMOKE_MBP10=/data/futures-baseline-mbp10/ES.FUT \ +//! cargo test -p ml-backtesting --test checkpoint_smoke \ +//! -- --ignored --nocapture +//! +//! Compile-time check: cargo test --no-run validates the test signature +//! against the harness API without invoking GPU. + +use std::path::PathBuf; + +use anyhow::{Context, Result}; +use ml_alpha::cfc::trunk::{CfcConfig, CfcTrunk}; +use ml_backtesting::aggregate::ANCHOR_REALISTIC; +use ml_backtesting::artifacts::Summary; +use ml_backtesting::harness::{BacktestHarness, BacktestHarnessConfig}; +use ml_core::device::MlDevice; + +fn locate_checkpoint() -> Result { + let p = std::env::var("FOXHUNT_SMOKE_CKPT") + .map(PathBuf::from) + .context("set FOXHUNT_SMOKE_CKPT=")?; + if !p.exists() { + anyhow::bail!("checkpoint not found at {}", p.display()); + } + Ok(p) +} + +fn locate_mbp10() -> Result { + let p = std::env::var("FOXHUNT_SMOKE_MBP10") + .map(PathBuf::from) + .unwrap_or_else(|_| PathBuf::from("test_data/futures-baseline/ES.FUT")); + if !p.exists() { + anyhow::bail!("MBP-10 fixture not found at {}", p.display()); + } + Ok(p) +} + +#[test] +#[ignore = "GPU + real checkpoint required; run with --ignored"] +fn smoke_load_checkpoint_and_run_one_cell() -> Result<()> { + let ckpt = locate_checkpoint()?; + let mbp10 = locate_mbp10()?; + let dev = MlDevice::new(0).context("init MlDevice")?; + + let cfg = CfcConfig::default(); // includes mamba2_state_dim=16 default + let trunk = CfcTrunk::load_checkpoint(&dev, &cfg, &ckpt) + .context("load_checkpoint smoke check")?; + + let harness_cfg = BacktestHarnessConfig { + mbp10_dir: mbp10, + cost_tick: ANCHOR_REALISTIC.cost_tick, + latency_ms: ANCHOR_REALISTIC.latency_ms, + threshold: 0.75, + decision_stride: 1, + ..Default::default() + }; + let mut harness = BacktestHarness::new(harness_cfg, trunk)?; + let _stats = harness.run()?; + + let tmp = tempfile::tempdir()?; + harness.write_artifacts(tmp.path())?; + let s: Summary = serde_json::from_reader( + std::fs::File::open(tmp.path().join("summary.json"))? + )?; + + // Spec §3.4 smoke criteria + assert!(s.n_trades > 0, "model never traded; check threshold/data"); + assert!(s.sharpe_ann.is_finite(), "Sharpe non-finite ({})", s.sharpe_ann); + Ok(()) +} +``` + +If `BacktestHarnessConfig` doesn't expose `mbp10_dir`, `cost_tick`, `latency_ms`, `threshold`, `decision_stride` with these exact names, read the actual struct definition (in `crates/ml-backtesting/src/harness.rs`) and adapt the field assignments. The fields exist conceptually — just the names may differ. + +### Step X18.2: Compile-only check + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --test checkpoint_smoke --no-run +``` + +Expected: builds clean. Test body is `#[ignore]`d so no GPU invocation. + +### Step X18.3: Commit + +```bash +git add crates/ml-backtesting/tests/checkpoint_smoke.rs +git commit -m "test(ml-backtesting): GPU smoke against real trained checkpoint + +Adds #[ignore]d integration test that loads a CheckpointV2 trunk file +produced by alpha_train, runs one BacktestHarness cell at the realistic +anchor against a fixture MBP-10 window, asserts spec §3.4 smoke criteria +(n_trades > 0, Sharpe finite). + +Activated via FOXHUNT_SMOKE_CKPT + FOXHUNT_SMOKE_MBP10 env vars on a +CUDA-capable host. Per spec §3.4, §4.1." +``` + +--- + +## Task X19: Three sweep YAMLs + +**Spec:** §3.3 (threshold pre-registration), §3.4 (smoke gate), §3.6 (walk-forward windows). + +**Files:** `config/ml/sweep_v2_smoke.yaml`, `config/ml/sweep_v2_threshold_tuning.yaml`, `config/ml/sweep_v2_deployability.yaml` + +### Step X19.1: Smoke YAML + +- [ ] Create `config/ml/sweep_v2_smoke.yaml`: + +```yaml +# Spec §3.4 smoke gate — one cell, one window, one threshold. +checkpoint: /feature-cache/alpha-perception-runs/__SHA__/trunk_best_h6000.bin +windows: + - id: W0 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q1.dbn.zst +axes: + cost_tick: [0.25] + latency_ms: [200] + threshold: [0.75] +decision_stride: 1 +horizons_used: [h6000] +gpu_pool: ci-training-l40s +``` + +### Step X19.2: Threshold-tuning YAML + +- [ ] Create `config/ml/sweep_v2_threshold_tuning.yaml`: + +```yaml +# Spec §3.3 — threshold pre-registration on W0 only. +# Single-process fxt-backtest sweep (not Argo fan-out). 8 cells. +checkpoint: /feature-cache/alpha-perception-runs/__SHA__/trunk_best_h6000.bin +windows: + - id: W0 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q1.dbn.zst +axes: + cost_tick: [1.0] + latency_ms: [200] + threshold: [0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95] +decision_stride: 1 +horizons_used: [h6000] +``` + +### Step X19.3: Deployability YAML + +- [ ] Create `config/ml/sweep_v2_deployability.yaml`: + +```yaml +# Spec §3.6 + §3.1 — 4 walk-forward held-out windows × full grid. +# 7 × 4 × 5 × 4 = 560 cells. Fan out via scripts/argo-lob-sweep.sh. +checkpoint: /feature-cache/alpha-perception-runs/__SHA__/trunk_best_h6000.bin +windows: + - id: W1 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q2.dbn.zst + - id: W2 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q3.dbn.zst + - id: W3 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q4.dbn.zst + - id: W4 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2026-Q1.dbn.zst +axes: + cost_tick: [0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5] + latency_ms: [50, 100, 200, 400] + threshold: [0.70, 0.75, 0.80, 0.85, 0.90] +decision_stride: 1 +horizons_used: [h6000] +gpu_pool: ci-training-l40s +``` + +### Step X19.4: Validate parseability + +- [ ] If `fxt-backtest sweep --validate` exists (check the CLI), run: + +```bash +cargo run --release -p ml-backtesting --bin fxt-backtest -- sweep --grid config/ml/sweep_v2_smoke.yaml --validate +cargo run --release -p ml-backtesting --bin fxt-backtest -- sweep --grid config/ml/sweep_v2_threshold_tuning.yaml --validate +cargo run --release -p ml-backtesting --bin fxt-backtest -- sweep --grid config/ml/sweep_v2_deployability.yaml --validate +``` + +If `--validate` doesn't exist, skip — the runtime will fail-fast on YAML parse errors before submitting any pod. + +### Step X19.5: Commit + +```bash +git add config/ml/sweep_v2_smoke.yaml \ + config/ml/sweep_v2_threshold_tuning.yaml \ + config/ml/sweep_v2_deployability.yaml +git commit -m "config(ml-alpha): three sweep YAMLs for deployability validation + +Adds smoke, threshold-tuning, and deployability sweep configurations +per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md +sections 3.3, 3.4, 3.6. __SHA__ placeholders are resolved by +scripts/argo-lob-sweep.sh --sha at submission time." +``` + +--- + +# PHASE 1 → PHASE 2 GATE: Post-refactor fold-0 smoke + +**Spec:** §2.3 (final pre-Argo smoke gate). + +Before proceeding to Phase 2 runtime, the **single-fold smoke** must reproduce recorded 3-fold A/B numbers within tolerance. This is the necessary insurance. + +### Step G.1: Push code + +```bash +git push origin ml-alpha-phase-a +``` + +Capture the HEAD commit SHA. Call it `$X19_SHA`. + +### Step G.2: Submit fold-0 smoke + +```bash +./scripts/argo-train.sh \ + --branch ml-alpha-phase-a \ + --commit "$X19_SHA" \ + --epochs 30 \ + --cv-fold 0 --cv-n-folds 3 \ + --cv-train-window 0 \ + --seed 16962 \ + --batch-size 1 \ + --gpu-pool ci-training-l40s \ + --watch +``` + +Expected: workflow completes; `alpha_train_summary.json` lands on `feature-cache-pvc` under `/feature-cache/alpha-perception-runs/$X19_SHA/`. + +### Step G.3: Verify tolerance + +- [ ] Download or read `alpha_train_summary.json`. Compare to recorded fold 0 from `project_ml_alpha_v2_ab_verdict`: + +| Metric | Pre-refactor (6b7920474) | Post-refactor tolerance | +|---|---|---| +| `best_mean_auc` | 0.7529 | ≤ ±0.010 absolute | +| `best_mean_auc_epoch` | 11 | ≤ ±3 epochs | +| `best_auc_h6000` | 0.7639 | ≤ ±0.010 absolute | +| `best_auc_h6000_epoch` | 11 | ≤ ±3 epochs | + +### Step G.4: Gate decision + +- [ ] If all four metrics within tolerance → **Phase 1 complete, proceed to Phase 2**. +- [ ] If any metric outside tolerance → **STOP**. The refactor broke training in a way the small fixtures didn't catch. Diagnose (likely a numerical kernel issue caught only by the full training trajectory). Revert offending commit, re-run X10/X11 golden, re-submit fold-0 smoke. Do NOT proceed to production training until this gate passes. + +Verify `trunk_best_h6000.bin` exists in the OUT_DIR (proves the X14 wiring worked end-to-end): + +```bash +kubectl run pvc-peek --rm -i --restart=Never -n foxhunt --image=busybox --overrides='{ + "spec": {"containers": [{ + "name":"pvc-peek","image":"busybox", + "command":["sh","-c","ls -lh /feature-cache/alpha-perception-runs/'"$X19_SHA"'/"], + "volumeMounts":[{"name":"d","mountPath":"/feature-cache"}] + }], "volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"feature-cache-pvc","readOnly":true}}]} +}' +``` + +Expected output: `trunk_best_h6000.bin` (~1.5 MB) and `alpha_train_summary.json`. + +--- + +# PHASE 2: Production training + deployability sweep + +Runbook tasks; require operator at the controls. Each step depends on the previous. + +### Step P.1: Production training run + +```bash +git push origin ml-alpha-phase-a # capture commit as $TRAIN_SHA +./scripts/argo-train.sh \ + --branch ml-alpha-phase-a \ + --commit "$TRAIN_SHA" \ + --epochs 30 \ + --cv-n-folds 1 \ + --cv-train-window 4 \ + --gpu-pool ci-training-l40s --watch +``` + +Expected: workflow `alpha-perception-XXXXX` succeeds; `trunk_best_h6000.bin` lands on PVC. Inspect summary log line `saved best_h6000 checkpoint` to confirm wiring fired. + +### Step P.2: Smoke sweep + +```bash +./scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_smoke.yaml --sha "$TRAIN_SHA" --gpu-pool ci-training-l40s --watch +``` + +Expected: 1-cell workflow succeeds in ~5 minutes. Inspect cell `summary.json`: +- `n_trades > 0` +- `sharpe_ann` finite +- harness exit code 0 + +**Smoke fail → halt.** Do not proceed. + +### Step P.3: Threshold pre-registration + +```bash +./scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_threshold_tuning.yaml --sha "$TRAIN_SHA" --gpu-pool ci-training-l40s --watch +``` + +8 cells, ~5 minutes. Download per-cell `summary.json`. Pick the threshold with highest `sharpe_ann` on W0. Write to `config/ml/v2_prod_thresholds.json`: + +```json +{ + "threshold": , + "in_sample_sharpe": , + "training_sha": "", + "spec_sha": "da1dd92bf", + "selected_at": "", + "window": "W0" +} +``` + +Commit + push: + +```bash +git add config/ml/v2_prod_thresholds.json +git commit -m "config(ml-alpha): pre-register v2 prod threshold from W0 sweep + +Selected threshold= with in-sample Sharpe on the 2025-Q1 +threshold-tuning window. Frozen per spec §3.3 — does not move during +deployability evaluation." +git push origin ml-alpha-phase-a +``` + +### Step P.4: Deployability sweep + +Optionally collapse the threshold axis in `config/ml/sweep_v2_deployability.yaml` to just `[]` to save GPU time (560 → 112 cells). Edit + commit + push if changed. + +```bash +./scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_deployability.yaml --sha "$TRAIN_SHA" --sweep-tag v2-deployability --watch +``` + +~35 minutes wall-clock with default Argo fan-out. + +### Step P.5: Aggregate + emit verdict + +```bash +SQLX_OFFLINE=true cargo run --release -p ml-backtesting --bin fxt-backtest -- aggregate +``` + +Then emit the verdict (assumes a `fxt-backtest verdict` subcommand wired in X17 step; if not, invoke `emit_deployability_verdict` directly via a small one-off binary): + +```bash +SQLX_OFFLINE=true cargo run --release -p ml-backtesting --bin fxt-backtest -- verdict \ + --threshold $(jq -r .threshold config/ml/v2_prod_thresholds.json) \ + --windows W1,W2,W3,W4 \ + --training-sha "$TRAIN_SHA" \ + --spec-sha da1dd92bf \ + --out deployability_verdict.json +``` + +If `fxt-backtest verdict` doesn't exist, add it as a small `clap` subcommand wiring to `emit_deployability_verdict`. This is a one-line addition to `bin/fxt-backtest/src/main.rs`. + +### Step P.6: Commit verdict + memory update + +```bash +git add deployability_verdict.json +git commit -m "verdict(ml-alpha): v2 deployability — + +Median Sharpe at realistic anchor: +Median max-dd at realistic anchor: +Median Sharpe at stress anchor: +Median max-dd at stress anchor: + +Per spec 2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md." +git push origin ml-alpha-phase-a +``` + +Update `/home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/project_ml_alpha_v2_ab_verdict.md` with the PnL-side close-out (verdict tier + per-anchor medians + windows used + training SHA). + +--- + +## Self-Review + +**Spec coverage:** + +| Spec section | Implementing task | +|---|---| +| §1.1 trunk-grows ownership | X1..X10 | +| §1.2 CheckpointV2 envelope | X12 | +| §1.3 module touch-list | X0..X19 (covers each file) | +| §1.4 persistent artefacts | X0 (golden), X12+X13 (.bin), X14 (summary), Phase 2 (sweep outputs, verdict) | +| §1.5 data corpus binding | Phase 2 P.1 (`--cv-n-folds 1 --cv-train-window 4`) | +| §2.1 golden fixture | X0 | +| §2.2 commit ladder X0–X19 | All Phase 1 tasks | +| §2.3 fold-0 smoke gate | Gate G.1–G.4 between Phase 1 and Phase 2 | +| §2.4 nondeterminism handling | X0 (1e-4 tolerance) | +| §2.5 Phase 1 completion criteria | Gate G enforces all four criteria | +| §3.1 anchor specs | X17 (constants in aggregate.rs) | +| §3.2 metrics | X16 (max_drawdown_pct); existing Summary has Sharpe/Sortino/PF | +| §3.3 two-pass threshold | X19 (threshold-tuning YAML) + Phase 2 P.3 | +| §3.4 smoke gate | X18 (test) + X19 (smoke YAML) + Phase 2 P.2 | +| §3.5 verdict emitter | X17 | +| §3.6 walk-forward windows | X19 (deployability YAML) | +| §4.1 test inventory | X0, X11, X12, X13, X14, X16, X17, X18, Gate G | +| §4.2 failure modes | Covered in commit messages + smoke gate logic | +| §5 Phase 2 runtime sequence | Phase 2 P.1–P.6 | + +All sections covered. + +**Placeholder scan:** + +- `__SHA__` in YAML files: intentional template marker resolved by `scripts/argo-lob-sweep.sh --sha`. Not a TBD. +- ``, ``, ``, ``, ``, `` in Phase 2 commit messages: runtime values substituted by the operator. Not TBDs. +- `` in P.5: a path the operator supplies after the sweep completes. Not a TBD. +- `todo!()` inside `run_uncaptured_chain` in X11: NOT acceptable as a literal `todo!()` in committed code. Replace with the explicit kernel chain body (mirror `capture_graph_a`'s sequence) when X11 lands. The plan's code block uses `todo!()` as a placeholder for "spell out the chain here"; the actual commit must replace it. + +**Type consistency:** + +- `AlphaTrainSummary` in X14 matches the actual struct name (verified in alpha_train.rs:168). +- `Summary` and `TradeRecord` in X16 match existing definitions in `crates/ml-backtesting/src/artifacts.rs`. +- `VerdictTier`, `AnchorReport`, `DeployabilityVerdict`, `classify_verdict`, `emit_deployability_verdict`, `ANCHOR_REALISTIC`, `ANCHOR_STRESS`: all new and self-consistent within X17. +- `BacktestHarness::new(harness_cfg, trunk)` in X18: signature assumes harness takes (config, trunk). Verify before X18 lands by reading harness.rs:65 (currently signature is `pub fn new(... mut trunk: CfcTrunk, ...) -> Result` — confirmed compatible). +- `CfcConfig::default()` in X18 matches the X1 extension that adds `mamba2_state_dim: 16` to the default. + +**Soft adaptation points (resolve at code-read time):** + +- `HEAD_MID` constant — may need to be added to `heads.rs` as a public constant if not already present. Documented in X1.3. +- `Mamba2Block.w_in / w_a / w_b / w_c / w_out / config()` public accessors — may need to be added to `mamba2_block.rs` for X12 to read them. Documented in X12.2. +- `BacktestHarnessConfig` field names — adapt to actual struct definition. Documented in X18.1. + +The plan is internally consistent. Three soft adaptation points will resolve naturally at code-read time.