From 4938ac2ec5a86c54dfd0c6050c31a07cb930a18c Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 19 May 2026 00:40:15 +0200 Subject: [PATCH] =?UTF-8?q?plan(ml-alpha):=20v2=20deployability=20validati?= =?UTF-8?q?on=20=E2=80=94=20atomic-commit=20roadmap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five-commit implementation plan + two-step runtime runbook for the deployability spec committed at 07d5de504. Bite-sized TDD tasks: C1: wire save_checkpoint(best_h6000) into alpha_train.rs (~10 LOC) C2: max_drawdown_pct field on Summary, $35k starting-capital base C3: emit_deployability_verdict + tiered logic + 6 unit tests C4: GPU integration smoke test (#[ignore], env-var-gated) C5: three sweep YAMLs (smoke, threshold-tuning, deployability) C6: runtime — Argo prod training → smoke → threshold pre-reg → full sweep C7: commit verdict, update memory Self-review confirms 1:1 spec section ↔ task coverage. Two soft adaptation points (AlphaTrainSummary struct name, BacktestHarnessConfig defaults) marked as code-read-and-adapt; no hard TBDs. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...19-ml-alpha-v2-deployability-validation.md | 1363 +++++++++++++++++ 1 file changed, 1363 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-19-ml-alpha-v2-deployability-validation.md diff --git a/docs/superpowers/plans/2026-05-19-ml-alpha-v2-deployability-validation.md b/docs/superpowers/plans/2026-05-19-ml-alpha-v2-deployability-validation.md new file mode 100644 index 000000000..9aa7d32b9 --- /dev/null +++ b/docs/superpowers/plans/2026-05-19-ml-alpha-v2-deployability-validation.md @@ -0,0 +1,1363 @@ +# ml-alpha v2 Deployability Validation — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Close the wiring gap between the v2 ml-alpha trainer and the existing real-LOB backtest system, produce a falsifiable deployability verdict (median Sharpe > 1.0 AND median max-drawdown < 20% across 4 walk-forward held-out quarters at the realistic Scaleway→IBKR anchor). + +**Architecture:** Minimum-wiring deployability gate on top of the C1–C19 real-LOB system already on this branch. (1) Wire `CfcTrunk::save_checkpoint` into `alpha_train.rs`'s `auc_h6000_improved` block. (2) Add `max_drawdown_pct` to per-cell Summary (the other three metrics already exist). (3) Add three sweep YAMLs (smoke, threshold-tuning, deployability) + a two-anchor tiered verdict emitter. (4) Smoke-gated Argo workflow: production training → smoke sweep → threshold pre-registration → deployability sweep → verdict. + +**Tech Stack:** Rust 2021, cudarc 0.19, CUDA 12.4, Argo Workflows, MinIO PVC `feature-cache-pvc`, ES MBP-10 quarterly DBN files on `training-data-pvc`. + +**Spec:** `docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md` @ commit `07d5de504`. Read it before starting any task; this plan references it heavily. + +**Branch:** Stay on `ml-alpha-phase-a` for the entire implementation. + +--- + +## File Structure + +Code changes are concentrated in five files; runtime artefacts in three YAML configs and one Argo workflow. + +| File | Role | Status | +|---|---|---| +| `crates/ml-alpha/examples/alpha_train.rs` | Training entrypoint. Add `save_checkpoint` call + extend `AlphaTrainSummary` schema. | Modify | +| `crates/ml-backtesting/src/artifacts.rs` | Per-cell `Summary` struct + `compute_summary()`. Add `max_drawdown_pct` field. | Modify | +| `crates/ml-backtesting/src/aggregate.rs` | Add `emit_deployability_verdict()` function + tiered verdict logic. | Modify | +| `crates/ml-backtesting/tests/checkpoint_smoke.rs` | GPU integration test loading a real trained checkpoint, running one cell. | Create | +| `config/ml/sweep_v2_smoke.yaml` | Smoke-gate single-cell sweep config. | Create | +| `config/ml/sweep_v2_threshold_tuning.yaml` | Threshold pre-registration sweep config (W0 only). | Create | +| `config/ml/sweep_v2_deployability.yaml` | Full deployability sweep config (W1–W4 × 7 × 4 × 5 = 560 cells). | Create | + +Runtime artefacts produced by the workflow (committed to repo as audit records): + +| Artefact | When written | +|---|---| +| `config/ml/v2_prod_thresholds.json` | After threshold pre-registration sweep | +| `deployability_verdict.json` | After deployability sweep + verdict emission | + +--- + +## Commit Map (atomic commits) + +| # | Title | Scope | +|---|---|---| +| C1 | `feat(ml-alpha): alpha_train saves best_h6000 checkpoint` | Wire `save_checkpoint` + extend summary JSON. | +| C2 | `feat(ml-backtesting): max_drawdown_pct on per-cell Summary` | New field + compute_summary update + tests. | +| C3 | `feat(ml-backtesting): emit_deployability_verdict + tiered logic` | Two-anchor verdict emitter + unit tests for tiered logic. | +| C4 | `test(ml-backtesting): GPU smoke against real trained checkpoint` | Integration test (`#[ignore]`, GPU-required). | +| C5 | `config(ml-alpha): three sweep YAMLs for deployability validation` | smoke, threshold-tuning, deployability. | +| C6 | `runtime: production checkpoint + smoke + verdict` | Operator runbook (no code commit). | +| C7 | `verdict(ml-alpha): commit deployability_verdict.json + memory update` | Audit record + memory close-out. | + +--- + +## Conventions for every task + +1. `SQLX_OFFLINE=true` is set in the environment; all `cargo` commands assume it. +2. GPU tests are `#[ignore]`d by default; run with `--ignored` on a CUDA-capable host. +3. Per `feedback_push_before_deploy.md`, `git push` before any Argo submission. +4. Per `feedback_default_to_l40s_pool.md`, Argo submissions default to `--gpu-pool ci-training-l40s`. +5. Read the spec section referenced in each task before coding. + +--- + +## Task C1: Wire save_checkpoint into alpha_train.rs + +**Spec:** §1.1 (touch-list), §1.2 (`trunk_best_h6000.bin` artefact), §1.3 (corpus split). + +**Files:** +- Modify: `crates/ml-alpha/examples/alpha_train.rs` + +### Step 1.1: Locate the existing `auc_h6000_improved` block + +- [ ] Read `crates/ml-alpha/examples/alpha_train.rs` around line 603, confirming the block: + +```rust +let auc_h6000_improved = auc_h6000 > best_auc_h6000; +if auc_h6000_improved { + best_auc_h6000 = auc_h6000; + best_auc_h6000_epoch = epoch; + best_auc_h6000_per_horizon = per_horizon_auc; + auc_h6000_no_improvement = 0; + tracing::info!(epoch, auc_h6000, "new best auc_h6000"); +} else { + auc_h6000_no_improvement += 1; +} +``` + +### Step 1.2: Add `best_h6000_ckpt_path` to the summary struct + +- [ ] Find the summary struct definition in `crates/ml-alpha/examples/alpha_train.rs` (search `struct.*Summary` or similar — it's the struct serialized to `alpha_train_summary.json`). + +- [ ] Add field: + +```rust +/// Path to the best-h6000 trunk checkpoint (relative to `out_dir`), +/// or None if no improvement was ever recorded (e.g., epoch 0 only). +#[serde(default, skip_serializing_if = "Option::is_none")] +best_h6000_ckpt_path: Option, +``` + +- [ ] Initialize as `None` at construction. + +### Step 1.3: Wire the save_checkpoint call + +- [ ] Inside the `if auc_h6000_improved` block, add (immediately after `auc_h6000_no_improvement = 0;`): + +```rust +let ckpt_path = cli.out.join("trunk_best_h6000.bin"); +trunk_a.save_checkpoint(&ckpt_path) + .context("save_checkpoint(trunk_best_h6000.bin)")?; +summary.best_h6000_ckpt_path = Some("trunk_best_h6000.bin".to_string()); +tracing::info!(epoch, path = %ckpt_path.display(), "saved best_h6000 checkpoint"); +``` + +Variable names (`summary`, `cli.out`, `trunk_a`) must match what's already in the binary. If they differ, adjust to the existing naming — do NOT introduce new bindings. + +### Step 1.4: Add a unit test for the summary schema + +- [ ] At the bottom of `crates/ml-alpha/examples/alpha_train.rs` (or in a `#[cfg(test)] mod tests` block), add: + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn summary_serde_round_trip_with_h6000_ckpt() { + let mut s = AlphaTrainSummary::default(); // adjust to actual type name + s.best_h6000_ckpt_path = Some("trunk_best_h6000.bin".to_string()); + let json = serde_json::to_string(&s).expect("serialize"); + let back: AlphaTrainSummary = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back.best_h6000_ckpt_path.as_deref(), Some("trunk_best_h6000.bin")); + } +} +``` + +If `AlphaTrainSummary` doesn't have a `Default` impl, construct it explicitly with required fields. + +### Step 1.5: Run tests + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-alpha --example alpha_train summary_serde_round_trip_with_h6000_ckpt -- --nocapture +``` + +Expected: 1 test passes. + +### Step 1.6: Build the example binary + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo build -p ml-alpha --example alpha_train --release +``` + +Expected: builds clean, no warnings about unused `ckpt_path`. + +### Step 1.7: Commit + +- [ ] Run: + +```bash +git add crates/ml-alpha/examples/alpha_train.rs +git commit -m "feat(ml-alpha): alpha_train saves best_h6000 checkpoint + +Inside the auc_h6000_improved block, call CfcTrunk::save_checkpoint to +write trunk_best_h6000.bin to OUT_DIR. Extend AlphaTrainSummary with a +best_h6000_ckpt_path field so downstream tooling can locate the file +without re-deriving the path. + +Closes wire-up gap noted in feedback_wire_everything_up — load_checkpoint ++ fxt-backtest --checkpoint were ready (C14), but no producer ever +emitted a checkpoint." +``` + +--- + +## Task C2: max_drawdown_pct on per-cell Summary + +**Spec:** §2.2 (metrics), §3.2 (testing — \"max-drawdown math against a known fixture P&L curve\"). + +**Files:** +- Modify: `crates/ml-backtesting/src/artifacts.rs` + +### Step 2.1: Add field to the Summary struct + +- [ ] In `crates/ml-backtesting/src/artifacts.rs` (struct starts around line 21), add field after `max_drawdown_usd`: + +```rust +/// Maximum drawdown as a fraction of starting capital, per window. +/// Computed in compute_summary using STARTING_CAPITAL_USD as base. +/// Range [0.0, 1.0] (0.20 = 20% drawdown). +pub max_drawdown_pct: f32, +``` + +### Step 2.2: Pin a starting-capital constant + +- [ ] In the same file, near the existing `ANNUALISATION_SQRT_FACTOR` constant, add: + +```rust +/// Starting capital in USD used as the base for max_drawdown_pct. +/// $35k is the project's chosen base for ES single-contract analysis — +/// realistic for a small-account ES trader (margin ~$15k initial, +/// ~$20k headroom for adverse moves before a margin call). +pub const STARTING_CAPITAL_USD: f32 = 35_000.0; +``` + +### Step 2.3: Populate max_drawdown_pct in compute_summary + +- [ ] Find the section of `compute_summary` that computes `max_drawdown_usd` (search for `max_drawdown` inside the function body). After that line, add: + +```rust +let max_drawdown_pct = max_drawdown_usd.abs() / STARTING_CAPITAL_USD; +``` + +- [ ] Add `max_drawdown_pct` to the Summary construction at the end of the function. + +- [ ] In the early-return branch (`if records.is_empty()`), explicitly set `max_drawdown_pct: 0.0`. + +### Step 2.4: Write failing test for max-dd-pct math + +- [ ] In the existing `#[cfg(test)] mod tests` at the bottom of `crates/ml-backtesting/src/artifacts.rs`, add: + +```rust +#[test] +fn max_drawdown_pct_against_fixture_curve() { + // P&L curve in USD: peak $7,000, trough $0 → $7,000 drawdown. + // STARTING_CAPITAL_USD = $35k → 20% drawdown (right on the + // deployability gate boundary — good edge-case fixture). + let curve = vec![0.0, 3500.0, 7000.0, 4000.0, 0.0, 2000.0]; + // One trade record needed so compute_summary doesn't take the + // empty-records early return. + let records = vec![TradeRecord { + realised_pnl_usd_fp: 200_000, // $2,000 in fp-USD-x100 + fees_usd_fp: 0, + ..Default::default() + }]; + let s = compute_summary(&records, &curve); + assert!((s.max_drawdown_pct - 0.20).abs() < 1e-4, + "expected max_drawdown_pct=0.20, got {}", s.max_drawdown_pct); +} +``` + +If `TradeRecord` doesn't impl `Default`, construct it explicitly with all required fields (read the struct definition once before writing this test). + +### Step 2.5: Run test, verify it fails first, then passes + +- [ ] First commit only the test (Step 2.4), then run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib max_drawdown_pct_against_fixture_curve -- --nocapture +``` + +Expected: FAIL with "no field `max_drawdown_pct`" — confirms the test would have caught absence of the field. + +- [ ] Then apply Steps 2.1–2.3, rerun: + +Expected: PASS. + +### Step 2.6: Extend existing round-trip test + +- [ ] Find `write_summary_then_parse_back` in `crates/ml-backtesting/src/artifacts.rs` (around line 275). Add an assertion for the new field: + +```rust +assert!((parsed.max_drawdown_pct - written.max_drawdown_pct).abs() < 1e-6); +``` + +- [ ] Re-run the round-trip test: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib write_summary_then_parse_back -- --nocapture +``` + +Expected: PASS. + +### Step 2.7: Run the full ml-backtesting test suite to catch regressions + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib +``` + +Expected: all existing tests still pass (the new field is `f32`, defaults to `0.0` on the early-empty branch — no behavior change for existing fixtures). + +### Step 2.8: Commit + +- [ ] Run: + +```bash +git add crates/ml-backtesting/src/artifacts.rs +git commit -m "feat(ml-backtesting): max_drawdown_pct on per-cell Summary + +Adds max_drawdown_pct field as |max_drawdown_usd| / STARTING_CAPITAL_USD +(pinned at \$35k per project_ml_alpha_starting_capital memory — realistic +ES single-contract small-account anchor). Used by emit_deployability_verdict +as the capital- +deployability hard gate (median across windows < 20%). + +Per spec 2026-05-19-ml-alpha-v2-deployability-validation-design.md §2.2." +``` + +--- + +## Task C3: emit_deployability_verdict + tiered logic + +**Spec:** §2.6 (verdict emitter pseudocode), §0 (verdict tiers), §3.3 (failure modes). + +**Files:** +- Modify: `crates/ml-backtesting/src/aggregate.rs` + +### Step 3.1: Add the verdict types + +- [ ] At the top of `crates/ml-backtesting/src/aggregate.rs` (after the existing imports), add: + +```rust +use serde::{Deserialize, Serialize}; + +/// One of the five verdict tiers from spec §0. +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum VerdictTier { + PassRobust, + PassNominal, + FailInconclusive, + Fail, + FailDegenerate, +} + +/// Anchor specification (cost + latency in physical units). +#[derive(Clone, Copy, Debug, Serialize, Deserialize)] +pub struct AnchorSpec { + pub name: &'static str, + pub cost_tick: f32, + pub latency_ms: u32, +} + +pub const ANCHOR_REALISTIC: AnchorSpec = AnchorSpec { + name: "realistic", + cost_tick: 1.0, + latency_ms: 200, +}; + +pub const ANCHOR_STRESS: AnchorSpec = AnchorSpec { + name: "stress", + cost_tick: 1.5, + latency_ms: 400, +}; + +/// Per-anchor aggregated stats across walk-forward windows. +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct AnchorReport { + pub anchor: AnchorSpec, + pub sharpe_per_window: Vec, + pub sortino_per_window: Vec, + pub max_dd_pct_per_window: Vec, + pub profit_factor_per_window: Vec, + pub median_sharpe: f32, + pub median_sortino: f32, + pub median_max_dd_pct: f32, + pub median_profit_factor: f32, + pub gate_sharpe: bool, + pub gate_max_dd: bool, + pub pass: bool, + /// "ok" if all windows had finite Sharpe/Sortino + trades_count > 0. + /// Otherwise "fail-degenerate" with a per-window breakdown. + pub status: String, +} + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct DeployabilityVerdict { + pub verdict: VerdictTier, + pub realistic: AnchorReport, + pub stress: AnchorReport, + pub threshold: f32, + pub windows: Vec, + pub training_sha: String, + pub spec_sha: String, + pub timestamp_utc: String, +} +``` + +### Step 3.2: Write failing tests for tiered logic + +- [ ] At the bottom of `crates/ml-backtesting/src/aggregate.rs`, in the existing test module, add: + +```rust +#[test] +fn verdict_pass_robust_when_both_anchors_pass_both_gates() { + let realistic = AnchorReport { + median_sharpe: 1.5, median_max_dd_pct: 0.10, + gate_sharpe: true, gate_max_dd: true, pass: true, + status: "ok".to_string(), + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = AnchorReport { + median_sharpe: 1.2, median_max_dd_pct: 0.15, + gate_sharpe: true, gate_max_dd: true, pass: true, + status: "ok".to_string(), + ..stub_anchor_report(ANCHOR_STRESS) + }; + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::PassRobust); +} + +#[test] +fn verdict_pass_nominal_when_only_realistic_passes() { + let realistic = AnchorReport { + median_sharpe: 1.5, median_max_dd_pct: 0.10, + gate_sharpe: true, gate_max_dd: true, pass: true, + status: "ok".to_string(), + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = AnchorReport { + median_sharpe: 0.7, median_max_dd_pct: 0.30, + gate_sharpe: false, gate_max_dd: false, pass: false, + status: "ok".to_string(), + ..stub_anchor_report(ANCHOR_STRESS) + }; + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::PassNominal); +} + +#[test] +fn verdict_fail_inconclusive_in_grey_zone() { + let realistic = AnchorReport { + median_sharpe: 0.9, median_max_dd_pct: 0.10, + gate_sharpe: false, gate_max_dd: true, pass: false, + status: "ok".to_string(), + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = stub_anchor_report(ANCHOR_STRESS); + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::FailInconclusive); + + let realistic2 = AnchorReport { + median_sharpe: 1.5, median_max_dd_pct: 0.22, + gate_sharpe: true, gate_max_dd: false, pass: false, + status: "ok".to_string(), + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + assert_eq!(classify_verdict(&realistic2, &stress), VerdictTier::FailInconclusive); +} + +#[test] +fn verdict_fail_when_realistic_misses_by_a_lot() { + let realistic = AnchorReport { + median_sharpe: 0.3, median_max_dd_pct: 0.40, + gate_sharpe: false, gate_max_dd: false, pass: false, + status: "ok".to_string(), + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = stub_anchor_report(ANCHOR_STRESS); + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::Fail); +} + +#[test] +fn verdict_fail_degenerate_when_realistic_has_nan_or_zero_trades() { + let realistic = AnchorReport { + status: "fail-degenerate: W2 had 0 trades".to_string(), + ..stub_anchor_report(ANCHOR_REALISTIC) + }; + let stress = stub_anchor_report(ANCHOR_STRESS); + assert_eq!(classify_verdict(&realistic, &stress), VerdictTier::FailDegenerate); +} + +fn stub_anchor_report(anchor: AnchorSpec) -> AnchorReport { + AnchorReport { + anchor, + sharpe_per_window: vec![], + sortino_per_window: vec![], + max_dd_pct_per_window: vec![], + profit_factor_per_window: vec![], + median_sharpe: 0.0, + median_sortino: 0.0, + median_max_dd_pct: 0.0, + median_profit_factor: 0.0, + gate_sharpe: false, + gate_max_dd: false, + pass: false, + status: "ok".to_string(), + } +} + +#[test] +fn deployability_verdict_serde_round_trip() { + let v = DeployabilityVerdict { + verdict: VerdictTier::PassRobust, + realistic: stub_anchor_report(ANCHOR_REALISTIC), + stress: stub_anchor_report(ANCHOR_STRESS), + threshold: 0.75, + windows: vec!["W1".into(), "W2".into(), "W3".into(), "W4".into()], + training_sha: "abc1234".into(), + spec_sha: "07d5de504".into(), + timestamp_utc: "2026-05-19T12:00:00Z".into(), + }; + let json = serde_json::to_string(&v).expect("serialize"); + let back: DeployabilityVerdict = + serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back.verdict, VerdictTier::PassRobust); + assert_eq!(back.threshold, 0.75); + assert_eq!(back.windows.len(), 4); +} +``` + +### Step 3.3: Run tests, verify all fail (no implementation yet) + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib verdict_ -- --nocapture +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib deployability_verdict_serde_round_trip -- --nocapture +``` + +Expected: all fail with "cannot find function `classify_verdict`" or similar — confirms tests would have caught absence of the function. + +### Step 3.4: Implement classify_verdict + +- [ ] In `crates/ml-backtesting/src/aggregate.rs`, add: + +```rust +/// Tiered verdict from two anchor reports per spec §2.6. +pub fn classify_verdict( + realistic: &AnchorReport, + stress: &AnchorReport, +) -> VerdictTier { + if realistic.status != "ok" { + return VerdictTier::FailDegenerate; + } + if realistic.pass && stress.status == "ok" && stress.pass { + return VerdictTier::PassRobust; + } + if realistic.pass { + return VerdictTier::PassNominal; + } + let sharpe_grey = realistic.median_sharpe >= 0.8 + && realistic.median_sharpe < 1.0; + let max_dd_grey = realistic.median_max_dd_pct >= 0.20 + && realistic.median_max_dd_pct < 0.25; + if sharpe_grey || max_dd_grey { + return VerdictTier::FailInconclusive; + } + VerdictTier::Fail +} +``` + +### Step 3.5: Re-run tests, verify all pass + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib verdict_ -- --nocapture +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib deployability_verdict_serde_round_trip -- --nocapture +``` + +Expected: all 6 tests pass. + +### Step 3.6: Implement emit_deployability_verdict + +- [ ] Add the main function: + +```rust +use chrono::Utc; +use std::collections::BTreeMap; +use std::fs; + +/// Build a DeployabilityVerdict by reading per-cell summary.json files +/// from `sweep_dir`, matching the two anchor cells against the +/// pre-registered threshold, and applying classify_verdict. +/// +/// Spec §2.6. Window matching is by exact (cost_tick, latency_ms, +/// threshold) tuple; cell directories must encode these in their names +/// per fxt-backtest sweep convention (cell__cost_lat_th_). +pub fn emit_deployability_verdict( + sweep_dir: &Path, + threshold: f32, + windows: &[&str], + training_sha: &str, + spec_sha: &str, +) -> Result { + let realistic = build_anchor_report( + sweep_dir, ANCHOR_REALISTIC, threshold, windows + )?; + let stress = build_anchor_report( + sweep_dir, ANCHOR_STRESS, threshold, windows + )?; + let verdict = classify_verdict(&realistic, &stress); + Ok(DeployabilityVerdict { + verdict, + realistic, + stress, + threshold, + windows: windows.iter().map(|s| s.to_string()).collect(), + training_sha: training_sha.to_string(), + spec_sha: spec_sha.to_string(), + timestamp_utc: Utc::now().to_rfc3339(), + }) +} + +fn build_anchor_report( + sweep_dir: &Path, + anchor: AnchorSpec, + threshold: f32, + windows: &[&str], +) -> Result { + let mut by_window: BTreeMap = BTreeMap::new(); + for entry in fs::read_dir(sweep_dir) + .with_context(|| format!("read_dir {}", sweep_dir.display()))? + { + let path = entry?.path(); + let name = path.file_name() + .and_then(|n| n.to_str()) + .unwrap_or("") + .to_string(); + let (cell_cost, cell_lat, cell_th, cell_win) = + match parse_cell_name(&name) { + Some(t) => t, + None => continue, + }; + if (cell_cost - anchor.cost_tick).abs() > 1e-6 { continue; } + if cell_lat != anchor.latency_ms { continue; } + if (cell_th - threshold).abs() > 1e-6 { continue; } + if !windows.iter().any(|w| **w == cell_win) { continue; } + let summary_path = path.join("summary.json"); + let s: Summary = serde_json::from_reader( + fs::File::open(&summary_path) + .with_context(|| format!("open {}", summary_path.display()))? + )?; + by_window.insert(cell_win.clone(), s); + } + + let mut sharpe = Vec::new(); + let mut sortino = Vec::new(); + let mut max_dd = Vec::new(); + let mut pf = Vec::new(); + let mut status = String::from("ok"); + for w in windows { + let s = by_window.get(*w).ok_or_else(|| + anyhow::anyhow!("missing cell for anchor={} window={}", anchor.name, w) + )?; + if s.n_trades == 0 || !s.sharpe_ann.is_finite() || !s.sortino_ann.is_finite() { + status = format!("fail-degenerate: {w} trades={} sharpe={}", s.n_trades, s.sharpe_ann); + } + sharpe.push(s.sharpe_ann); + sortino.push(s.sortino_ann); + max_dd.push(s.max_drawdown_pct); + pf.push(s.profit_factor); + } + let median_sharpe = median(&sharpe); + let median_sortino = median(&sortino); + let median_max_dd_pct = median(&max_dd); + let median_profit_factor = median(&pf); + let gate_sharpe = median_sharpe > 1.0; + let gate_max_dd = median_max_dd_pct < 0.20; + let pass = gate_sharpe && gate_max_dd && status == "ok"; + Ok(AnchorReport { + anchor, + sharpe_per_window: sharpe, + sortino_per_window: sortino, + max_dd_pct_per_window: max_dd, + profit_factor_per_window: pf, + median_sharpe, + median_sortino, + median_max_dd_pct, + median_profit_factor, + gate_sharpe, + gate_max_dd, + pass, + status, + }) +} + +fn median(xs: &[f32]) -> f32 { + let mut v = xs.to_vec(); + v.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal)); + let n = v.len(); + if n == 0 { return f32::NAN; } + if n % 2 == 1 { v[n / 2] } else { 0.5 * (v[n / 2 - 1] + v[n / 2]) } +} + +/// Parse a cell directory name like +/// `cell_0042_cost1.00_lat200_th0.75_W1` into its dimensions. +fn parse_cell_name(name: &str) -> Option<(f32, u32, f32, String)> { + // Defensive parsing: returns None if the name doesn't match the + // expected layout. Callers skip non-matching entries. + let parts: Vec<&str> = name.split('_').collect(); + if parts.len() < 5 || parts[0] != "cell" { return None; } + let cost = parts.iter().find(|p| p.starts_with("cost"))?[4..].parse().ok()?; + let lat = parts.iter().find(|p| p.starts_with("lat"))?[3..].parse().ok()?; + let th = parts.iter().find(|p| p.starts_with("th"))?[2..].parse().ok()?; + let win = parts.iter().rev().find(|p| p.starts_with('W'))?.to_string(); + Some((cost, lat, th, win)) +} +``` + +### Step 3.7: Verify aggregate.rs still compiles + +- [ ] Add `chrono = { workspace = true }` to `crates/ml-backtesting/Cargo.toml` `[dependencies]` if not already present. Run: + +```bash +SQLX_OFFLINE=true cargo build -p ml-backtesting +``` + +Expected: builds clean. + +### Step 3.8: Add a fixture-based test for emit_deployability_verdict + +- [ ] In the test module of `aggregate.rs`, add: + +```rust +#[test] +fn emit_deployability_verdict_against_fixture_sweep() { + let tmp = tempfile::tempdir().expect("tmpdir"); + // Build a tiny fixture: 8 cells = 2 anchors × 4 windows, all at th=0.75. + // Realistic anchor passes; stress anchor fails on Sharpe. + let realistic_summaries = [ + ("W1", 1.4, 0.10), + ("W2", 1.6, 0.12), + ("W3", 1.2, 0.08), + ("W4", 1.5, 0.11), + ]; + let stress_summaries = [ + ("W1", 0.4, 0.30), + ("W2", 0.6, 0.28), + ("W3", 0.5, 0.32), + ("W4", 0.7, 0.29), + ]; + for (win, sharpe, dd) in &realistic_summaries { + write_fixture_cell(tmp.path(), 1.0, 200, 0.75, win, *sharpe, *dd); + } + for (win, sharpe, dd) in &stress_summaries { + write_fixture_cell(tmp.path(), 1.5, 400, 0.75, win, *sharpe, *dd); + } + let v = emit_deployability_verdict( + tmp.path(), 0.75, &["W1","W2","W3","W4"], + "test-sha", "test-spec-sha" + ).expect("verdict"); + assert_eq!(v.verdict, VerdictTier::PassNominal); + assert!(v.realistic.pass); + assert!(!v.stress.pass); + assert!((v.realistic.median_sharpe - 1.45).abs() < 0.05, + "median sharpe {}", v.realistic.median_sharpe); +} + +fn write_fixture_cell( + root: &Path, cost: f32, lat: u32, th: f32, window: &str, + sharpe: f32, max_dd_pct: f32, +) { + let dir = root.join(format!( + "cell_0000_cost{cost:.2}_lat{lat}_th{th:.2}_{window}" + )); + fs::create_dir_all(&dir).unwrap(); + let s = Summary { + total_pnl_usd: 10_000.0, + sharpe_ann: sharpe, + sortino_ann: sharpe * 1.2, + max_drawdown_usd: max_dd_pct * 35_000.0, + max_drawdown_pct: max_dd_pct, + calmar: 0.0, + n_trades: 100, + win_rate: 0.55, + avg_win_usd: 200.0, + avg_loss_usd: -150.0, + profit_factor: 1.3, + total_fees_usd: 500.0, + exposure_pct: 0.3, + kelly_cap_history_sample: vec![], + }; + serde_json::to_writer( + fs::File::create(dir.join("summary.json")).unwrap(), + &s + ).unwrap(); +} +``` + +- [ ] Add `tempfile` to `[dev-dependencies]` of `crates/ml-backtesting/Cargo.toml` if not already present. + +### Step 3.9: Run the fixture test + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --lib emit_deployability_verdict_against_fixture_sweep -- --nocapture +``` + +Expected: PASS — fixture asserts the PassNominal verdict (realistic passes, stress fails). + +### Step 3.10: Commit + +- [ ] Run: + +```bash +git add crates/ml-backtesting/src/aggregate.rs crates/ml-backtesting/Cargo.toml +git commit -m "feat(ml-backtesting): emit_deployability_verdict + tiered logic + +Adds DeployabilityVerdict + AnchorReport types and emit_deployability_verdict +that reads per-cell summary.json files at the realistic (1 tick, 200 ms) and +stress (1.5 tick, 400 ms) anchors, computes median Sharpe / Sortino / +max_dd_pct / profit_factor across 4 walk-forward windows, applies the +tiered pass logic (Pass-robust / Pass-nominal / Fail-inconclusive / Fail / +Fail-degenerate), and writes the audit JSON. + +Per spec 2026-05-19-ml-alpha-v2-deployability-validation-design.md §2.6. +Unit-tested via fixture sweep + 5 classify_verdict cases covering each tier." +``` + +--- + +## Task C4: GPU smoke against real trained checkpoint + +**Spec:** §3.1 (smoke gate criteria), §3.2 (testing — \"end-to-end with real checkpoint\"). + +**Files:** +- Create: `crates/ml-backtesting/tests/checkpoint_smoke.rs` + +### Step 4.1: Create the test file + +- [ ] Create `crates/ml-backtesting/tests/checkpoint_smoke.rs` with: + +```rust +//! GPU integration smoke: load a real trunk checkpoint, run one backtest +//! cell against a fixture MBP-10 window, assert spec §3.1 smoke criteria. +//! +//! Marked #[ignore]; run with `cargo test -p ml-backtesting --test +//! checkpoint_smoke -- --ignored --nocapture` on a CUDA-capable host. + +use std::path::PathBuf; + +use anyhow::{Context, Result}; +use ml_alpha::cfc::trunk::{CfcConfig, CfcTrunk}; +use ml_backtesting::aggregate::{ANCHOR_REALISTIC}; +use ml_backtesting::artifacts::Summary; +use ml_backtesting::harness::{BacktestHarness, BacktestHarnessConfig}; +use ml_core::device::MlDevice; + +/// Path to a CheckpointV1 file produced by alpha_train. The test +/// expects the file to exist at this path; if not, the test errors with +/// a clear "set FOXHUNT_SMOKE_CKPT" message. +fn locate_checkpoint() -> Result { + let path = std::env::var("FOXHUNT_SMOKE_CKPT").map(PathBuf::from) + .context("set FOXHUNT_SMOKE_CKPT to the path of a trunk_best_h6000.bin file")?; + if !path.exists() { + anyhow::bail!("checkpoint not found at {}", path.display()); + } + Ok(path) +} + +fn locate_mbp10_fixture() -> Result { + let p = PathBuf::from(std::env::var("FOXHUNT_SMOKE_MBP10") + .unwrap_or_else(|_| "test_data/futures-baseline/ES.FUT".into())); + if !p.exists() { + anyhow::bail!("MBP-10 fixture dir not found at {}", p.display()); + } + Ok(p) +} + +#[test] +#[ignore = "GPU + real checkpoint required; run with --ignored"] +fn smoke_load_checkpoint_and_run_one_cell() -> Result<()> { + let ckpt = locate_checkpoint()?; + let mbp10 = locate_mbp10_fixture()?; + let dev = MlDevice::new(0).context("init MlDevice")?; + let cfg = CfcConfig::default_v2(); // adjust to whatever the v2 prod cfg constructor is + let _trunk = CfcTrunk::load_checkpoint(&dev, &cfg, &ckpt) + .context("load_checkpoint smoke check")?; + + // Drive one backtest cell against the fixture window. + let mut harness = BacktestHarness::new( + BacktestHarnessConfig { + checkpoint: ckpt, + mbp10_dir: mbp10, + cost_tick: ANCHOR_REALISTIC.cost_tick, + latency_ms: ANCHOR_REALISTIC.latency_ms, + threshold: 0.75, + decision_stride: 1, + ..Default::default() // fill in remaining fields per actual struct + }, + &dev, + )?; + let stats = harness.run()?; + let tmp = tempfile::tempdir()?; + harness.write_artifacts(tmp.path())?; + let s: Summary = serde_json::from_reader( + std::fs::File::open(tmp.path().join("summary.json"))? + )?; + + // Spec §3.1 smoke criteria + assert!(s.n_trades > 0, "model never traded; check threshold/data"); + assert!(s.sharpe_ann.is_finite(), "Sharpe non-finite ({})", s.sharpe_ann); + assert!(stats.event_count > 0, "harness saw no events"); + // mean_decision_latency check would go here once the harness exposes + // it; the spec calls for <100us on L40S. If not exposed yet, mark TODO + // and revisit when the field lands. + Ok(()) +} +``` + +Adjust `CfcConfig::default_v2()` and the `BacktestHarnessConfig` field list to whatever the actual constructors expect (read the modules once before finalizing the test). + +### Step 4.2: Add tempfile as dev-dependency + +- [ ] If not already present in `crates/ml-backtesting/Cargo.toml`: + +```toml +[dev-dependencies] +tempfile = { workspace = true } +``` + +### Step 4.3: Compile the test (no GPU required) + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo test -p ml-backtesting --test checkpoint_smoke --no-run +``` + +Expected: compiles clean. The test body is `#[ignore]`d so no GPU is invoked at this point. + +### Step 4.4: Document the run-time invocation in a comment + +- [ ] At the top of the test file (after the existing module doc-comment), add a note: + +```rust +//! Runtime invocation (on a CUDA-capable host with a real trained +//! checkpoint and MBP-10 fixture present): +//! +//! FOXHUNT_SMOKE_CKPT=/feature-cache/alpha-perception-runs//trunk_best_h6000.bin \ +//! FOXHUNT_SMOKE_MBP10=/data/futures-baseline-mbp10/ES.FUT \ +//! cargo test -p ml-backtesting --test checkpoint_smoke \ +//! -- --ignored --nocapture +``` + +### Step 4.5: Commit + +- [ ] Run: + +```bash +git add crates/ml-backtesting/tests/checkpoint_smoke.rs crates/ml-backtesting/Cargo.toml +git commit -m "test(ml-backtesting): GPU smoke against real trained checkpoint + +Adds #[ignore]d integration test that loads a CheckpointV1 file produced +by alpha_train, runs one BacktestHarness cell at the realistic anchor +against a fixture MBP-10 window, asserts spec §3.1 smoke criteria +(n_trades > 0, Sharpe finite, harness saw events). + +Activated via FOXHUNT_SMOKE_CKPT + FOXHUNT_SMOKE_MBP10 env vars; meant +to run on the same Argo pod that produces the training checkpoint, before +unlocking the full deployability sweep." +``` + +--- + +## Task C5: Three sweep YAMLs + +**Spec:** §2.3 (threshold-tuning), §2.5 (deployability grid), §3.1 (smoke). + +**Files:** +- Create: `config/ml/sweep_v2_smoke.yaml` +- Create: `config/ml/sweep_v2_threshold_tuning.yaml` +- Create: `config/ml/sweep_v2_deployability.yaml` + +### Step 5.1: Write the smoke YAML + +- [ ] Create `config/ml/sweep_v2_smoke.yaml`: + +```yaml +# Spec §3.1 smoke gate — one cell, one window, one threshold. +# Purpose: verify checkpoint loads + harness drives end-to-end + non-degenerate output +# before unlocking the full deployability sweep. +checkpoint: /feature-cache/alpha-perception-runs/__SHA__/trunk_best_h6000.bin +windows: + - id: W0 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q1.dbn.zst +axes: + cost_tick: [0.25] + latency_ms: [200] + threshold: [0.75] +decision_stride: 1 +horizons_used: [h6000] +gpu_pool: ci-training-l40s +``` + +`__SHA__` is replaced at submission time by `scripts/argo-lob-sweep.sh --sha `. (Verify this is the right placeholder convention by reading `scripts/argo-lob-sweep.sh` once.) + +### Step 5.2: Write the threshold-tuning YAML + +- [ ] Create `config/ml/sweep_v2_threshold_tuning.yaml`: + +```yaml +# Spec §2.3 — threshold pre-registration on W0 only. +# Single-process fxt-backtest sweep (not Argo fan-out). 8 cells. +checkpoint: /feature-cache/alpha-perception-runs/__SHA__/trunk_best_h6000.bin +windows: + - id: W0 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q1.dbn.zst +axes: + cost_tick: [1.0] + latency_ms: [200] + threshold: [0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95] +decision_stride: 1 +horizons_used: [h6000] +# Per-horizon-quantile interpretation: each threshold is the empirical +# quantile applied to p_h6000(t) on W0. The cell with highest in-sample +# Sharpe is written to config/ml/v2_prod_thresholds.json. +``` + +### Step 5.3: Write the deployability YAML + +- [ ] Create `config/ml/sweep_v2_deployability.yaml`: + +```yaml +# Spec §2.5 — full grid against 4 held-out walk-forward windows. +# 7 × 4 × 5 × 4 = 560 cells. Fan out via scripts/argo-lob-sweep.sh. +checkpoint: /feature-cache/alpha-perception-runs/__SHA__/trunk_best_h6000.bin +windows: + - id: W1 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q2.dbn.zst + - id: W2 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q3.dbn.zst + - id: W3 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q4.dbn.zst + - id: W4 + dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2026-Q1.dbn.zst +axes: + cost_tick: [0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5] + latency_ms: [50, 100, 200, 400] + threshold: [0.70, 0.75, 0.80, 0.85, 0.90] +decision_stride: 1 +horizons_used: [h6000] +gpu_pool: ci-training-l40s +``` + +### Step 5.4: Validate the YAMLs against the existing parser + +- [ ] Run: + +```bash +SQLX_OFFLINE=true cargo run --release -p ml-backtesting --bin fxt-backtest -- \ + sweep --grid config/ml/sweep_v2_smoke.yaml --validate +SQLX_OFFLINE=true cargo run --release -p ml-backtesting --bin fxt-backtest -- \ + sweep --grid config/ml/sweep_v2_threshold_tuning.yaml --validate +SQLX_OFFLINE=true cargo run --release -p ml-backtesting --bin fxt-backtest -- \ + sweep --grid config/ml/sweep_v2_deployability.yaml --validate +``` + +Expected: each prints `valid` or equivalent and exits 0. If `--validate` flag doesn't exist yet on the sweep subcommand, skip this step and rely on the Argo workflow's first-pass parse to surface any YAML errors. + +### Step 5.5: Commit + +- [ ] Run: + +```bash +git add config/ml/sweep_v2_smoke.yaml \ + config/ml/sweep_v2_threshold_tuning.yaml \ + config/ml/sweep_v2_deployability.yaml +git commit -m "config(ml-alpha): three sweep YAMLs for deployability validation + +Adds the smoke, threshold-tuning, and deployability sweep configurations +per spec 2026-05-19-ml-alpha-v2-deployability-validation-design.md +sections 2.3, 2.5, 3.1. + +Smoke: 1 cell on W0 to gate the full sweep. +Threshold tuning: 8 thresholds on W0 to pre-register operating point. +Deployability: 560 cells on W1-W4 for surface + verdict at anchors. + +__SHA__ placeholders are resolved by scripts/argo-lob-sweep.sh --sha at +submission time." +``` + +--- + +## Task C6: Runtime — production training + smoke + thresholds + sweep + +This task is a runbook, not a code commit. The operator runs the commands in +sequence; each gate must pass before proceeding. Per `feedback_push_before_deploy.md`, +`git push` before each Argo submission. + +### Step 6.1: Push the code commits + +- [ ] Run: + +```bash +git push origin ml-alpha-phase-a +``` + +Expected: push succeeds, remote tip matches local HEAD (which should be C5). + +Capture the commit SHA — call it `$TRAIN_SHA`. Subsequent steps reference it. + +### Step 6.2: Submit the production training run + +- [ ] Run: + +```bash +./scripts/argo-train.sh \ + --branch ml-alpha-phase-a \ + --commit "$TRAIN_SHA" \ + --epochs 30 \ + --cv-n-folds 1 \ + --cv-train-window 4 \ + --gpu-pool ci-training-l40s \ + --watch +``` + +Expected: alpha-perception workflow named `alpha-perception-XXXXX` submitted, training pod starts within ~3 minutes. Watch streams logs. + +If `argo-train.sh` doesn't expose `--cv-train-window`, submit via `argo submit` directly with the alpha-perception template and the parameter inline. Refer to `infra/k8s/argo/alpha-perception-template.yaml` parameter list. + +Watch for: +- `new best auc_h6000` log lines + accompanying `saved best_h6000 checkpoint` (the new wiring) +- Training completes (Succeeded), or early-stops on `mean_auc` patience=5 + +### Step 6.3: Verify checkpoint landed on PVC + +- [ ] Inspect the OUT_DIR on `feature-cache-pvc`. Use a one-off pod: + +```bash +kubectl run pvc-peek --rm -i --restart=Never -n foxhunt --image=busybox --overrides='{ + "spec": { + "containers": [{ + "name": "pvc-peek", + "image": "busybox", + "command": ["sh","-c","ls -lh /feature-cache/alpha-perception-runs/'"$TRAIN_SHA"'/"], + "volumeMounts": [{"name":"d","mountPath":"/feature-cache"}] + }], + "volumes": [{"name":"d","persistentVolumeClaim":{"claimName":"feature-cache-pvc","readOnly":true}}] + } +}' +``` + +Expected output includes `trunk_best_h6000.bin` (~90 KB) and `alpha_train_summary.json`. + +If `trunk_best_h6000.bin` is absent, the training never improved on h6000 — open a new investigation; do NOT continue to the sweep. + +### Step 6.4: Submit the smoke sweep + +- [ ] Update `config/ml/sweep_v2_smoke.yaml` `__SHA__` placeholder using your existing tooling, OR pass `--sha` to the lob-sweep script (read `scripts/argo-lob-sweep.sh` for the convention). + +- [ ] Run: + +```bash +./scripts/argo-lob-sweep.sh \ + --grid config/ml/sweep_v2_smoke.yaml \ + --sha "$TRAIN_SHA" \ + --gpu-pool ci-training-l40s \ + --watch +``` + +Expected: workflow `alpha-lob-sweep-XXXXX` succeeds in ~5 minutes. Inspect the resulting `summary.json` for the one cell against: + +- `n_trades > 0` +- `sharpe_ann` finite +- harness exit code 0 + +**Smoke fail → halt.** Open a new investigation; do not run threshold tuning or the full sweep. Likely causes: format drift, captured-graph misalignment, or model converged to a "never trade" policy. + +### Step 6.5: Threshold pre-registration sweep + +- [ ] Run: + +```bash +./scripts/argo-lob-sweep.sh \ + --grid config/ml/sweep_v2_threshold_tuning.yaml \ + --sha "$TRAIN_SHA" \ + --gpu-pool ci-training-l40s \ + --watch +``` + +Expected: 8 cells (single Argo workflow, single L40S pod runs all 8 since the grid is small enough to keep in one process). Wall-clock ~5 minutes. + +- [ ] Download the per-cell `summary.json` files. For each cell, record `threshold` and `sharpe_ann`. Pick the threshold with the **highest in-sample Sharpe**. + +- [ ] Write the result to `config/ml/v2_prod_thresholds.json`: + +```json +{ + "threshold": 0.80, + "in_sample_sharpe": 1.42, + "training_sha": "abc1234", + "spec_sha": "07d5de504", + "selected_at": "2026-05-19T13:00:00Z", + "window": "W0" +} +``` + +(`0.80` and `1.42` are placeholders — substitute actual values from your sweep.) + +- [ ] Commit: + +```bash +git add config/ml/v2_prod_thresholds.json +git commit -m "config(ml-alpha): pre-register v2 prod threshold from W0 sweep + +Selected threshold= with in-sample Sharpe on the 2025-Q1 +threshold-tuning window. Frozen per spec §2.3 — does not move during +deployability evaluation. + +Per spec 2026-05-19-ml-alpha-v2-deployability-validation-design.md." +git push origin ml-alpha-phase-a +``` + +### Step 6.6: Update the deployability YAML threshold axis (optional optimization) + +If you want to keep the descriptive surface broad: leave `threshold: [0.70, 0.75, 0.80, 0.85, 0.90]` as-is (5 cells × everything else = 560 cells). + +If you want to save GPU time: collapse the threshold axis to just `[]` since only that value feeds the verdict (the descriptive surface around it is informational). This drops 560 → 112 cells. + +- [ ] (Optional) If collapsing, edit `config/ml/sweep_v2_deployability.yaml`: + +```yaml + threshold: [0.80] # pre-registered per v2_prod_thresholds.json +``` + +- [ ] Commit + push if changed. + +### Step 6.7: Submit the deployability sweep + +- [ ] Run: + +```bash +./scripts/argo-lob-sweep.sh \ + --grid config/ml/sweep_v2_deployability.yaml \ + --sha "$TRAIN_SHA" \ + --gpu-pool ci-training-l40s \ + --sweep-tag v2-deployability \ + --watch +``` + +Expected: 560-cell (or 112-cell) workflow, ~35 minutes wall-clock with default Argo fan-out. Each cell writes `summary.json` to its per-cell directory under the sweep tag root. + +### Step 6.8: Aggregate + emit verdict + +- [ ] Download the per-cell results, OR run aggregation in-cluster: + +```bash +SQLX_OFFLINE=true cargo run --release -p ml-backtesting --bin fxt-backtest -- \ + aggregate +``` + +This produces `aggregate.parquet` + `pareto_frontier.json` per existing C9 behavior. + +- [ ] Then explicitly emit the verdict (new code from C3): + +```bash +SQLX_OFFLINE=true cargo run --release -p ml-backtesting --bin fxt-backtest -- \ + verdict \ + --threshold $(jq -r .threshold config/ml/v2_prod_thresholds.json) \ + --windows W1,W2,W3,W4 \ + --training-sha "$TRAIN_SHA" \ + --spec-sha 07d5de504 \ + --out deployability_verdict.json +``` + +(If `fxt-backtest verdict` subcommand doesn't exist yet, add it in C3 — this is a one-line `clap` subcommand wiring `emit_deployability_verdict` to the CLI. Mention this back-reference if it's the cleaner place to wire.) + +Expected: `deployability_verdict.json` written with the tier + all per-anchor + per-window numbers. + +--- + +## Task C7: Commit verdict + memory update + +### Step 7.1: Inspect the verdict + +- [ ] Read `deployability_verdict.json`. The top-level `verdict` field will be one of: + - `Pass-robust` → strongest signal, deployable in adverse + nominal regimes + - `Pass-nominal` → deployable in nominal regime, operational caveat on adverse + - `Fail-inconclusive` → grey zone, counts as Fail + - `Fail` → shelf decision + - `Fail-degenerate` → wiring problem; investigate before drawing conclusions + +### Step 7.2: Commit the verdict to repo + +- [ ] Run: + +```bash +git add deployability_verdict.json +git commit -m "verdict(ml-alpha): v2 deployability — + +Median Sharpe at realistic anchor: +Median max-dd at realistic anchor: +Median Sharpe at stress anchor: +Median max-dd at stress anchor: + +Per spec 2026-05-19-ml-alpha-v2-deployability-validation-design.md. +Sweep run at commit \$TRAIN_SHA." +git push origin ml-alpha-phase-a +``` + +### Step 7.3: Update project memory + +- [ ] Append a closing section to +`/home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/project_ml_alpha_v2_ab_verdict.md`: + +```markdown +--- + +## Deployability close-out @ commit `$TRAIN_SHA` (date) + +**Verdict:** + +Realistic anchor (200ms, 1 tick): median Sharpe , median max-dd % +Stress anchor (400ms, 1.5 tick): median Sharpe , median max-dd % + +Walk-forward windows: 2025-Q2..2026-Q1 (4 quarters × ~63 trading days). +Pre-registered threshold: , in-sample Sharpe on W0 (2025-Q1): . + +[[pearl_phase1d4_backtest_cost_edge_frontier]] — comparable Sharpe-anchor +reading for v2 ml-alpha vs the Phase 1d.4 stacker baseline. +``` + +Update the `description` frontmatter to mention the new close-out tier. + +### Step 7.4: (Conditional on Pass-*) Open paper-trade plan brainstorm + +If the verdict was Pass-robust or Pass-nominal, the natural next step is a +paper-trade plan (separate spec, separate brainstorming flow). Note this in +the memory close-out — do NOT auto-spawn the paper-trade work; the user gates +it. + +If the verdict was any Fail tier, document one sentence on the most likely +architectural diagnosis based on which gate failed (low Sharpe → variance +gaming or wrong horizon; high max-dd → trade sizing too aggressive; degenerate +→ wiring) and leave the next-step decision to the user. + +--- + +## Self-Review + +**Spec coverage:** + +| Spec section | Implementing task | +|---|---| +| §1.1 alpha_train wiring | C1 | +| §1.1 verdict emitter | C3 | +| §1.1 smoke integration test | C4 | +| §1.1 three sweep YAMLs | C5 | +| §1.2 trunk_best_h6000.bin | C1 | +| §1.2 v2_prod_thresholds.json | C6.5 | +| §1.2 deployability_verdict.json | C6.8 + C7.2 | +| §1.3 data corpus binding | C6.2 (`--cv-n-folds 1 --cv-train-window 4`) | +| §2.1 anchor specs | C3 (constants) | +| §2.2 four metrics | C2 (max_drawdown_pct adds the missing one; rest already in Summary) | +| §2.3 threshold pre-registration | C5 + C6.5 | +| §2.4 walk-forward windows | C5 (deployability YAML) | +| §2.5 deployability grid | C5 (deployability YAML) | +| §2.6 verdict emitter pseudocode | C3 | +| §3.1 smoke gate | C5 (smoke YAML) + C6.4 | +| §3.2 testing (new) | C2 + C3 + C4 | +| §3.3 failure modes | C4 + C7.4 | + +All covered. + +**Placeholder scan:** +- `__SHA__` in YAML files — intentional template marker resolved by `scripts/argo-lob-sweep.sh --sha`. Not a TBD. +- ``, ``, `` etc. in C6 / C7 commit messages — runtime values, substituted by operator. Not a TBD. +- C4 step 4.1 has a comment "mean_decision_latency check ... once the harness exposes it; if not exposed yet, mark TODO and revisit when the field lands." This is a real soft TBD — it depends on whether the harness already records this. To remove: read `crates/ml-backtesting/src/harness.rs` once before C4, decide whether the assertion is enabled or skipped, and lock that decision in the test. + +**Type consistency:** +- `AlphaTrainSummary` in C1.4 test must match the actual struct name in `examples/alpha_train.rs`; the test is templated to adapt. +- `Summary`, `TradeRecord` in C2 / C3 must match the existing definitions in `crates/ml-backtesting/src/artifacts.rs`. Both are used in existing tests so the names are stable. +- `VerdictTier`, `AnchorReport`, `DeployabilityVerdict`, `classify_verdict`, `emit_deployability_verdict` are new and self-consistent within C3. +- `BacktestHarness::new`, `BacktestHarnessConfig`, `harness.run`, `harness.write_artifacts` in C4 must match the existing C8-landed harness API; field defaults need verification against the actual struct. +- `CfcConfig::default_v2()` in C4 is a guess — replace with whatever the actual v2 config constructor is (likely `CfcConfig::new_v2(...)` or similar; read `crates/ml-alpha/src/cfc/trunk.rs` configs to confirm). + +The plan is internally consistent. Two soft adaptation points (test struct names, harness config defaults) will resolve themselves at code-read time.