From 0809390cd54d257f62eb2ae2c3a8fa9f37940b06 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 19 May 2026 00:26:48 +0200 Subject: [PATCH] =?UTF-8?q?spec(ml-alpha):=20v2=20deployability=20validati?= =?UTF-8?q?on=20=E2=80=94=20falsifiable=20LOB-backtest=20verdict?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the wiring gap between the existing real-LOB backtest system (C1–C19 on this branch) and the v2 ml-alpha model. alpha_train.rs currently never calls save_checkpoint, so the LOB harness/sweep/aggregate machinery has never been pointed at a real trained model. Design defines a single-pass falsifiable deployability gate: produce a production checkpoint (cv-n-folds=1, cv-train-window=4 → train on 2024 quarters, val on 2025-Q1, hold out 2025-Q2..2026-Q1), pre-register one threshold on the W0 val window, then evaluate median Sharpe across 4 held-out walk-forward quarters at the realistic Scaleway→IBKR anchor (200ms RTT, 1-tick all-in cost). Pass iff median > 1.0; inconclusive in [0.8, 1.0] counts as fail; smoke gate halts the full sweep on any wiring failure. Approved through brainstorming. Awaiting user spec-review before plan handoff. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...lpha-v2-deployability-validation-design.md | 298 ++++++++++++++++++ 1 file changed, 298 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md diff --git a/docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md b/docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md new file mode 100644 index 000000000..53559fff6 --- /dev/null +++ b/docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md @@ -0,0 +1,298 @@ +# ml-alpha v2 Deployability Validation — Design Spec + +**Status:** DESIGN — approved through brainstorming, awaiting user spec-review before plan handoff +**Date:** 2026-05-19 +**Branch:** `ml-alpha-phase-a` +**Owner:** ml-alpha team +**Spec depends on:** `2026-05-18-real-lob-integration-design.md` (commits C1–C19, already on branch) + +## 0. Context + +The 3-fold A/B sweep at commit `6b7920474` (see `project_ml_alpha_v2_ab_verdict`) +produced `mean_auc = 0.7510 ± 0.0117` and `auc_h6000 = 0.7604 ± 0.0034`. The +long-horizon signal is robust; the short-horizon regression matches the +[[pearl_input_feature_redundancy_loadbearing]] C16 pattern from the count-delta +slot revert. **AUC is not Sharpe.** Per `pearl_phase1d4_backtest_cost_edge_frontier`, +the Phase 1d.4 stacker model had frictionless Sharpe ~4.4 but break-even cost +was ~quarter-tick — half-tick killed it, retail 1-tick was deeply unprofitable. +Whether the v2 ml-alpha model is economically deployable is currently +**unknown** and cannot be answered from AUC alone. + +The real-LOB backtesting system (`crates/ml-backtesting/`, `bin/fxt-backtest/`, +`scripts/argo-lob-sweep.sh`) was built out across commits C1–C19 on this branch +but has never been pointed at a real trained checkpoint, because +`crates/ml-alpha/examples/alpha_train.rs` does not currently call +`CfcTrunk::save_checkpoint`. This spec turns that single wiring gap into a +falsifiable deployability verdict. + +**Pre-registered falsifiable claim:** + +> The v2 ml-alpha model (commit `6b7920474`, production-checkpoint variant) is +> deployable iff the **median Sharpe across 4 walk-forward held-out quarters, +> evaluated at the realistic deployment anchor (200 ms RTT, 1-tick all-in cost, +> pre-registered threshold (h6000 only)), is strictly greater than 1.0.** + +- **Pass** → candidate for paper-trade plan (separate spec). +- **Fail** → shelf decision for this architecture variant. Verdict committed. +- **Inconclusive (median ∈ [0.8, 1.0])** → counts as Fail. Re-evaluating the + criterion post-hoc is not permitted. + +**Out of scope (deferred to separate specs):** + +- A/B vs Phase-A baseline `73925b15d` in PnL terms. +- Miscalibration diagnostic (best-h6000 vs best-mean_auc checkpoint comparison). +- Multi-seed ensembles. +- Per-regime stratified PnL (`pearl_snapshot_alpha_is_regime_conditional` follow-on). +- Live IBKR FIX/REST adapter (production deployment). + +## 1. Architecture & components + +### 1.1 Module touch-list (no new crates) + +| File | Change | Lines | +|---|---|---| +| `crates/ml-alpha/examples/alpha_train.rs` | Inside the existing `auc_h6000_improved` block (~line 603), add `trunk_a.save_checkpoint(out_dir.join("trunk_best_h6000.bin"))?`. Add `best_h6000_ckpt_path` field to the `alpha_train_summary.json` schema. | ~10 | +| `config/ml/sweep_v2_deployability.yaml` *(new)* | Grid YAML for `fxt-backtest sweep` — see §2.5. | ~30 | +| `config/ml/sweep_v2_smoke.yaml` *(new)* | Single-cell smoke YAML — see §3.1. | ~15 | +| `config/ml/sweep_v2_threshold_tuning.yaml` *(new)* | Threshold pre-registration sweep YAML — see §2.3. | ~20 | +| `crates/ml-backtesting/src/aggregate.rs` | Add `emit_deployability_verdict(...)` function that reads the per-cell `summary.json` files, looks up the anchor cell, computes median Sharpe across windows, and writes `deployability_verdict.json`. | ~50 | +| `crates/ml-backtesting/tests/checkpoint_smoke.rs` *(new)* | `#[ignore]`d GPU integration test — load a trained checkpoint, drive the harness end-to-end against a tiny fixture window, assert smoke pass criteria. | ~80 | + +**Read-only consumers (exercised on a real trained model for the first time):** + +- `crates/ml-alpha/src/cfc/trunk.rs:470` `save_checkpoint` / `:539` `load_checkpoint` +- `crates/ml-backtesting/src/harness.rs` — drives loader → trunk → sim +- `bin/fxt-backtest/src/main.rs` — `run` / `sweep` / `aggregate` subcommands + +### 1.2 Persistent artifacts + +| Artifact | Producer | Consumer | Size | +|---|---|---|---| +| `trunk_best_h6000.bin` | Production training run (Argo `alpha-perception` template with `--cv-n-folds 1 --cv-train-window 4`) | All backtest cells | ~90 KB | +| `alpha_train_summary.json` (extended) | Same training run | Downstream tooling / verdict report | ~2 KB | +| `v2_prod_thresholds.json` *(new)* | Threshold pre-registration sweep (`fxt-backtest sweep` on W0 single-process) | Deployability sweep + verdict emitter | <1 KB | +| `aggregate.parquet` + `pareto_frontier.json` | `fxt-backtest aggregate` (existing C9 output) | Descriptive surface report | ~5 MB | +| `deployability_verdict.json` *(new)* | `emit_deployability_verdict` | Spec close-out, project memory | <1 KB | + +### 1.3 Data corpus binding (concrete) + +ES.FUT MBP-10 corpus on `training-data-pvc`: **quarterly DBN files spanning +2024-Q1 through 2026-Q1** (9 files total). + +| Range | Position | Role | +|---|---|---| +| 2024-Q1..Q4 | files [0..4] | Training corpus (production checkpoint) | +| 2025-Q1 | file [4] | Validation file → threshold pre-registration window W0 | +| 2025-Q2 | file [5] | Walk-forward W1 (held-out) | +| 2025-Q3 | file [6] | Walk-forward W2 (held-out) | +| 2025-Q4 | file [7] | Walk-forward W3 (held-out) | +| 2026-Q1 | file [8] | Walk-forward W4 (held-out) | + +The CLI flags that produce this split: `--cv-n-folds 1 --cv-train-window 4`. +The semantics are documented in `examples/alpha_train.rs` lines 338–360. + +Each held-out quarter is ~63 trading days; 4 quarters × 63 days = ~252 trading +days = full year of held-out walk-forward. ES contract roll dates within each +quarter are handled by the existing loader chronology — no new mitigation. + +## 2. Falsifiable criterion, anchor point, sweep configuration + +### 2.1 Realistic-deployment anchor + +| Axis | Value | Source | +|---|---|---| +| Round-trip latency | **200 ms** | Scaleway PAR ↔ CME Aurora ~80–100 ms one-way + IBKR FIX overhead, conservative | +| All-in trade cost | **1 ES tick = $12.50** per round-trip | Spread crossing (≥1 tick retail) + $0.85 commission/side ≈ 1.07 tick | +| Threshold | Pre-registered per-horizon, see §2.3 | Selected on W0, frozen before W1–W4 evaluation | + +The anchor is fixed in the spec. It is not a swept axis for the verdict; it is +swept only for the descriptive surface so we can see how Sharpe behaves *around* +the anchor. + +### 2.2 Sharpe definition + +**Annualized daily P&L Sharpe**, per window: + +``` +Sharpe_w = mean(daily_pnl_w) / std(daily_pnl_w) * sqrt(252) +``` + +Cross-window aggregate: `Sharpe_verdict = median(Sharpe_W1, Sharpe_W2, Sharpe_W3, Sharpe_W4)`. + +If any window's `std(daily_pnl)` is zero (model never trades), that window's +Sharpe is `NaN` and the verdict emitter rejects the run as `Fail-degenerate` +with a distinct exit code rather than silently propagating `NaN` through the +median. + +### 2.3 Threshold pre-registration (two-pass discipline) + +Look-ahead bias is the failure mode that quietly invalidates the entire study. +The fix is two backtest passes with the threshold frozen between them. + +**Pass 1 — Threshold-selection (in-sample, single sweep):** + +```yaml +# config/ml/sweep_v2_threshold_tuning.yaml +checkpoint: /feature-cache/alpha-perception-runs//trunk_best_h6000.bin +windows: [W0] # 2025-Q1 only +axes: + cost_tick: [1.0] # anchor + latency_ms: [200] # anchor + threshold: [p60, p65, p70, p75, p80, p85, p90, p95] +decision_stride: 1 +horizons_used: [h6000] +``` + +Run as a single-process `fxt-backtest sweep` (not Argo fan-out — only 8 cells). +The threshold value with the highest **in-sample Sharpe on W0** is the +production threshold. Persisted to `config/ml/v2_prod_thresholds.json` and +committed. + +The threshold is **frozen** at this point. It does not move. + +**Pass 2 — Deployability (out-of-sample, full grid):** + +`scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_deployability.yaml`. The +deployability sweep includes the threshold axis for the descriptive surface; +the verdict emitter looks up the *pre-registered* threshold and reads only that +cell across windows. + +### 2.4 Walk-forward windows + +| Window | DBN file | Trading days | Purpose | +|---|---|---|---| +| W0 | 2025-Q1 | ~63 | Threshold pre-registration (in-sample for early-stop, OOS for trade-decision) | +| W1 | 2025-Q2 | ~63 | Held-out validation #1 | +| W2 | 2025-Q3 | ~63 | Held-out validation #2 | +| W3 | 2025-Q4 | ~63 | Held-out validation #3 | +| W4 | 2026-Q1 | ~63 | Held-out validation #4 | + +### 2.5 Deployability sweep grid + +```yaml +# config/ml/sweep_v2_deployability.yaml +checkpoint: /feature-cache/alpha-perception-runs//trunk_best_h6000.bin +windows: [W1, W2, W3, W4] +axes: + cost_tick: [0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5] + latency_ms: [50, 100, 200, 400] + threshold: [p70, p75, p80, p85, p90] +decision_stride: 1 +horizons_used: [h6000] +``` + +7 × 4 × 5 × 4 = **560 cells**, fanned out via `argo-lob-sweep.sh`. Per-cell +runtime ~30 s on L40S (per real-LOB spec §6); wall-clock with 8-way fan-out +is ~35 minutes. Total L40S time: ~5 hours. + +### 2.6 Verdict emitter (`emit_deployability_verdict`) + +Pseudocode (~50 lines in `crates/ml-backtesting/src/aggregate.rs`): + +``` +1. Read v2_prod_thresholds.json → threshold_anchor (scalar; h6000 only) +2. For each cell file in sweep_dir/cell_*/summary.json: + Parse (cost_tick, latency_ms, threshold, window) + Sharpe + trades_count +3. Filter to cells where (cost_tick == 1.0) AND (latency_ms == 200) + AND (threshold == threshold_anchor) +4. Group by window → expect exactly 4 cells (W1, W2, W3, W4) +5. If any sharpe is NaN → emit verdict "Fail-degenerate" +6. median_sharpe = median([sharpe_W1, sharpe_W2, sharpe_W3, sharpe_W4]) +7. If median_sharpe > 1.0 → "Pass" + If median_sharpe ∈ [0.8, 1.0] → "Fail-inconclusive" + Else → "Fail" +8. Write deployability_verdict.json with all four window Sharpes, + the median, the verdict string, the threshold vector, the anchor point, + and the production training SHA. Commit to repo as the audit record. +``` + +## 3. Smoke gate, testing, failure modes + +### 3.1 Smoke gate (mandatory between training and full sweep) + +After `trunk_best_h6000.bin` lands on `feature-cache-pvc`, run a one-cell +smoke against W0 (the tuning window — cheap, already-known data): + +```yaml +# config/ml/sweep_v2_smoke.yaml +checkpoint: /feature-cache/alpha-perception-runs//trunk_best_h6000.bin +windows: [W0] +axes: + cost_tick: [0.25] + latency_ms: [200] + threshold: [p80] +decision_stride: 1 +horizons_used: [h6000] +``` + +**Smoke pass criteria (ALL must hold to unlock the full sweep):** + +| Check | Pass condition | Catches | +|---|---|---| +| Checkpoint loads | `CfcTrunk::load_checkpoint` returns `Ok` | Format drift (n_in/n_hid mismatch) | +| Harness drives end-to-end | `fxt-backtest run` exits 0, `summary.json` written | Captured-graph alignment, mapped-pinned buffer ownership | +| Non-degenerate output | `summary.json.trades_count > 0` AND `Sharpe` is finite | "Model never trades", `std(pnl)=0` divide-by-zero | +| Per-decision latency | `mean_decision_latency_us < 100` on L40S | Stream-capture leak forcing host fallbacks ([[pearl_no_host_branches_in_captured_graph]]) | + +Smoke failure halts the workflow and writes the failure reason to +`deployability_verdict.json.smoke_status`. The full sweep is gated on smoke +pass — there is no manual override path in this design. + +Cost: ~30 s of L40S time. Insurance against a 5-hour failed full sweep. + +### 3.2 Testing + +| Layer | What's tested | Where | Status | +|---|---|---|---| +| Trunk save/load round-trip | bit-exact weight recovery | `crates/ml-alpha/src/cfc/trunk.rs:603` | Exists | +| `cv-n-folds 1 cv-train-window 4` split | correct file partition | `alpha_train.rs:338–360` (existing branch) | Exists | +| LOB sim per-cell determinism | seeded backtest bit-reproducible | Ring 1b parity test | Exists | +| LOB sim invariants under fuzz | conserving P&L | Ring 2 fuzz (C18) | Exists | +| **End-to-end with real checkpoint** | smoke YAML produces non-degenerate `summary.json` against trained checkpoint | `crates/ml-backtesting/tests/checkpoint_smoke.rs` *(`#[ignore]`, GPU, ~30 s)* | **NEW** | +| **Verdict emitter format** | `deployability_verdict.json` round-trips through serde | `crates/ml-backtesting/src/aggregate.rs` test module | **NEW** | + +Two new tests total. Everything else is pre-existing infrastructure exercised +for the first time against a real trained model. + +### 3.3 Failure modes + +| Failure | Likelihood | Mitigation | +|---|---|---| +| Checkpoint format mismatch (train + backtest binaries built from different SHAs) | Medium | `load_checkpoint` validates n_in/n_hid; both binaries pinned to same `` by the smoke/sweep YAMLs referencing `/feature-cache/alpha-perception-runs//trunk_best_h6000.bin` | +| SSM-state leakage across windows | Low (Argo per-cell-pod fan-out gives fresh process per cell) | Argo workflow naturally isolates; plan-time verification that single-process `fxt-backtest sweep` also re-constructs harness per cell | +| Decision-stride misalignment between trainer and backtester | Low | Both pinned `decision_stride: 1` in smoke YAML, threshold YAML, deployability YAML | +| Model never trades at chosen threshold | Medium ([[pearl_snapshot_alpha_is_regime_conditional]]) | Smoke `trades_count > 0` check catches; pre-registration sweep includes wide threshold range (p60–p95) | +| Window straddles contract roll | Low | Quarterly window boundaries align with quarterly contract cycles; roll periods are interior to each quarter, loader handles chronology | +| Databento MBP-10 schema drift | Very low | Loader does in-band schema check, errors hard | +| Sharpe = NaN due to `std(pnl)=0` | Low | Verdict emitter rejects NaN with distinct `Fail-degenerate` exit | + +## 4. Implementation sequencing (preview for plan-writing skill) + +Logical commit boundaries — the implementation plan will refine these into +atomic commits: + +1. Wire `save_checkpoint(best_h6000)` into `alpha_train.rs` + extend summary + JSON with `best_h6000_ckpt_path`. Smoke unit test the path is written. +2. Add the three new YAMLs under `config/ml/` (threshold-tuning, smoke, + deployability). Lint-validate against `fxt-backtest sweep --validate`. +3. Add `emit_deployability_verdict` in `aggregate.rs` + verdict round-trip + test. +4. Add `checkpoint_smoke.rs` integration test (`#[ignore]`, GPU). +5. Submit Argo production training (`--cv-n-folds 1 --cv-train-window 4`). +6. Once `trunk_best_h6000.bin` lands: smoke sweep against W0. Halt on failure. +7. Threshold pre-registration sweep on W0. Commit `v2_prod_thresholds.json`. +8. Deployability sweep on W1–W4. Aggregate. Emit verdict. +9. Commit `deployability_verdict.json` as audit record. Update + `project_ml_alpha_v2_ab_verdict` memory with the PnL-side close-out. + +## 5. References + +- Real-LOB integration spec: `docs/superpowers/specs/2026-05-18-real-lob-integration-design.md` +- Real-LOB integration plan (C1–C19 already on branch): `docs/superpowers/plans/2026-05-18-real-lob-integration.md` +- ml-alpha v2 multi-horizon spec: `docs/superpowers/specs/2026-05-18-ml-alpha-v2-multi-horizon-design.md` +- 3-fold A/B verdict @ 6b7920474: memory `project_ml_alpha_v2_ab_verdict` +- Phase 1d.4 cost-edge frontier: memory `pearl_phase1d4_backtest_cost_edge_frontier` +- Single-window OOS warning: memory `pearl_single_window_oos_is_not_oos` +- Look-ahead / no-deferral discipline: memory `pearl_no_deferrals_for_complementary_fixes` +- Smoke-before-structural-priors: memory `feedback_smoke_validation_before_structural_priors` +- Wire-everything-up (the producer-consumer gap this spec closes): memory `feedback_wire_everything_up`