spec(ml-alpha): v2 deployability validation — falsifiable LOB-backtest verdict
Closes the wiring gap between the existing real-LOB backtest system (C1–C19 on this branch) and the v2 ml-alpha model. alpha_train.rs currently never calls save_checkpoint, so the LOB harness/sweep/aggregate machinery has never been pointed at a real trained model. Design defines a single-pass falsifiable deployability gate: produce a production checkpoint (cv-n-folds=1, cv-train-window=4 → train on 2024 quarters, val on 2025-Q1, hold out 2025-Q2..2026-Q1), pre-register one threshold on the W0 val window, then evaluate median Sharpe across 4 held-out walk-forward quarters at the realistic Scaleway→IBKR anchor (200ms RTT, 1-tick all-in cost). Pass iff median > 1.0; inconclusive in [0.8, 1.0] counts as fail; smoke gate halts the full sweep on any wiring failure. Approved through brainstorming. Awaiting user spec-review before plan handoff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,298 @@
|
||||
# ml-alpha v2 Deployability Validation — Design Spec
|
||||
|
||||
**Status:** DESIGN — approved through brainstorming, awaiting user spec-review before plan handoff
|
||||
**Date:** 2026-05-19
|
||||
**Branch:** `ml-alpha-phase-a`
|
||||
**Owner:** ml-alpha team
|
||||
**Spec depends on:** `2026-05-18-real-lob-integration-design.md` (commits C1–C19, already on branch)
|
||||
|
||||
## 0. Context
|
||||
|
||||
The 3-fold A/B sweep at commit `6b7920474` (see `project_ml_alpha_v2_ab_verdict`)
|
||||
produced `mean_auc = 0.7510 ± 0.0117` and `auc_h6000 = 0.7604 ± 0.0034`. The
|
||||
long-horizon signal is robust; the short-horizon regression matches the
|
||||
[[pearl_input_feature_redundancy_loadbearing]] C16 pattern from the count-delta
|
||||
slot revert. **AUC is not Sharpe.** Per `pearl_phase1d4_backtest_cost_edge_frontier`,
|
||||
the Phase 1d.4 stacker model had frictionless Sharpe ~4.4 but break-even cost
|
||||
was ~quarter-tick — half-tick killed it, retail 1-tick was deeply unprofitable.
|
||||
Whether the v2 ml-alpha model is economically deployable is currently
|
||||
**unknown** and cannot be answered from AUC alone.
|
||||
|
||||
The real-LOB backtesting system (`crates/ml-backtesting/`, `bin/fxt-backtest/`,
|
||||
`scripts/argo-lob-sweep.sh`) was built out across commits C1–C19 on this branch
|
||||
but has never been pointed at a real trained checkpoint, because
|
||||
`crates/ml-alpha/examples/alpha_train.rs` does not currently call
|
||||
`CfcTrunk::save_checkpoint`. This spec turns that single wiring gap into a
|
||||
falsifiable deployability verdict.
|
||||
|
||||
**Pre-registered falsifiable claim:**
|
||||
|
||||
> The v2 ml-alpha model (commit `6b7920474`, production-checkpoint variant) is
|
||||
> deployable iff the **median Sharpe across 4 walk-forward held-out quarters,
|
||||
> evaluated at the realistic deployment anchor (200 ms RTT, 1-tick all-in cost,
|
||||
> pre-registered threshold (h6000 only)), is strictly greater than 1.0.**
|
||||
|
||||
- **Pass** → candidate for paper-trade plan (separate spec).
|
||||
- **Fail** → shelf decision for this architecture variant. Verdict committed.
|
||||
- **Inconclusive (median ∈ [0.8, 1.0])** → counts as Fail. Re-evaluating the
|
||||
criterion post-hoc is not permitted.
|
||||
|
||||
**Out of scope (deferred to separate specs):**
|
||||
|
||||
- A/B vs Phase-A baseline `73925b15d` in PnL terms.
|
||||
- Miscalibration diagnostic (best-h6000 vs best-mean_auc checkpoint comparison).
|
||||
- Multi-seed ensembles.
|
||||
- Per-regime stratified PnL (`pearl_snapshot_alpha_is_regime_conditional` follow-on).
|
||||
- Live IBKR FIX/REST adapter (production deployment).
|
||||
|
||||
## 1. Architecture & components
|
||||
|
||||
### 1.1 Module touch-list (no new crates)
|
||||
|
||||
| File | Change | Lines |
|
||||
|---|---|---|
|
||||
| `crates/ml-alpha/examples/alpha_train.rs` | Inside the existing `auc_h6000_improved` block (~line 603), add `trunk_a.save_checkpoint(out_dir.join("trunk_best_h6000.bin"))?`. Add `best_h6000_ckpt_path` field to the `alpha_train_summary.json` schema. | ~10 |
|
||||
| `config/ml/sweep_v2_deployability.yaml` *(new)* | Grid YAML for `fxt-backtest sweep` — see §2.5. | ~30 |
|
||||
| `config/ml/sweep_v2_smoke.yaml` *(new)* | Single-cell smoke YAML — see §3.1. | ~15 |
|
||||
| `config/ml/sweep_v2_threshold_tuning.yaml` *(new)* | Threshold pre-registration sweep YAML — see §2.3. | ~20 |
|
||||
| `crates/ml-backtesting/src/aggregate.rs` | Add `emit_deployability_verdict(...)` function that reads the per-cell `summary.json` files, looks up the anchor cell, computes median Sharpe across windows, and writes `deployability_verdict.json`. | ~50 |
|
||||
| `crates/ml-backtesting/tests/checkpoint_smoke.rs` *(new)* | `#[ignore]`d GPU integration test — load a trained checkpoint, drive the harness end-to-end against a tiny fixture window, assert smoke pass criteria. | ~80 |
|
||||
|
||||
**Read-only consumers (exercised on a real trained model for the first time):**
|
||||
|
||||
- `crates/ml-alpha/src/cfc/trunk.rs:470` `save_checkpoint` / `:539` `load_checkpoint`
|
||||
- `crates/ml-backtesting/src/harness.rs` — drives loader → trunk → sim
|
||||
- `bin/fxt-backtest/src/main.rs` — `run` / `sweep` / `aggregate` subcommands
|
||||
|
||||
### 1.2 Persistent artifacts
|
||||
|
||||
| Artifact | Producer | Consumer | Size |
|
||||
|---|---|---|---|
|
||||
| `trunk_best_h6000.bin` | Production training run (Argo `alpha-perception` template with `--cv-n-folds 1 --cv-train-window 4`) | All backtest cells | ~90 KB |
|
||||
| `alpha_train_summary.json` (extended) | Same training run | Downstream tooling / verdict report | ~2 KB |
|
||||
| `v2_prod_thresholds.json` *(new)* | Threshold pre-registration sweep (`fxt-backtest sweep` on W0 single-process) | Deployability sweep + verdict emitter | <1 KB |
|
||||
| `aggregate.parquet` + `pareto_frontier.json` | `fxt-backtest aggregate` (existing C9 output) | Descriptive surface report | ~5 MB |
|
||||
| `deployability_verdict.json` *(new)* | `emit_deployability_verdict` | Spec close-out, project memory | <1 KB |
|
||||
|
||||
### 1.3 Data corpus binding (concrete)
|
||||
|
||||
ES.FUT MBP-10 corpus on `training-data-pvc`: **quarterly DBN files spanning
|
||||
2024-Q1 through 2026-Q1** (9 files total).
|
||||
|
||||
| Range | Position | Role |
|
||||
|---|---|---|
|
||||
| 2024-Q1..Q4 | files [0..4] | Training corpus (production checkpoint) |
|
||||
| 2025-Q1 | file [4] | Validation file → threshold pre-registration window W0 |
|
||||
| 2025-Q2 | file [5] | Walk-forward W1 (held-out) |
|
||||
| 2025-Q3 | file [6] | Walk-forward W2 (held-out) |
|
||||
| 2025-Q4 | file [7] | Walk-forward W3 (held-out) |
|
||||
| 2026-Q1 | file [8] | Walk-forward W4 (held-out) |
|
||||
|
||||
The CLI flags that produce this split: `--cv-n-folds 1 --cv-train-window 4`.
|
||||
The semantics are documented in `examples/alpha_train.rs` lines 338–360.
|
||||
|
||||
Each held-out quarter is ~63 trading days; 4 quarters × 63 days = ~252 trading
|
||||
days = full year of held-out walk-forward. ES contract roll dates within each
|
||||
quarter are handled by the existing loader chronology — no new mitigation.
|
||||
|
||||
## 2. Falsifiable criterion, anchor point, sweep configuration
|
||||
|
||||
### 2.1 Realistic-deployment anchor
|
||||
|
||||
| Axis | Value | Source |
|
||||
|---|---|---|
|
||||
| Round-trip latency | **200 ms** | Scaleway PAR ↔ CME Aurora ~80–100 ms one-way + IBKR FIX overhead, conservative |
|
||||
| All-in trade cost | **1 ES tick = $12.50** per round-trip | Spread crossing (≥1 tick retail) + $0.85 commission/side ≈ 1.07 tick |
|
||||
| Threshold | Pre-registered per-horizon, see §2.3 | Selected on W0, frozen before W1–W4 evaluation |
|
||||
|
||||
The anchor is fixed in the spec. It is not a swept axis for the verdict; it is
|
||||
swept only for the descriptive surface so we can see how Sharpe behaves *around*
|
||||
the anchor.
|
||||
|
||||
### 2.2 Sharpe definition
|
||||
|
||||
**Annualized daily P&L Sharpe**, per window:
|
||||
|
||||
```
|
||||
Sharpe_w = mean(daily_pnl_w) / std(daily_pnl_w) * sqrt(252)
|
||||
```
|
||||
|
||||
Cross-window aggregate: `Sharpe_verdict = median(Sharpe_W1, Sharpe_W2, Sharpe_W3, Sharpe_W4)`.
|
||||
|
||||
If any window's `std(daily_pnl)` is zero (model never trades), that window's
|
||||
Sharpe is `NaN` and the verdict emitter rejects the run as `Fail-degenerate`
|
||||
with a distinct exit code rather than silently propagating `NaN` through the
|
||||
median.
|
||||
|
||||
### 2.3 Threshold pre-registration (two-pass discipline)
|
||||
|
||||
Look-ahead bias is the failure mode that quietly invalidates the entire study.
|
||||
The fix is two backtest passes with the threshold frozen between them.
|
||||
|
||||
**Pass 1 — Threshold-selection (in-sample, single sweep):**
|
||||
|
||||
```yaml
|
||||
# config/ml/sweep_v2_threshold_tuning.yaml
|
||||
checkpoint: /feature-cache/alpha-perception-runs/<sha>/trunk_best_h6000.bin
|
||||
windows: [W0] # 2025-Q1 only
|
||||
axes:
|
||||
cost_tick: [1.0] # anchor
|
||||
latency_ms: [200] # anchor
|
||||
threshold: [p60, p65, p70, p75, p80, p85, p90, p95]
|
||||
decision_stride: 1
|
||||
horizons_used: [h6000]
|
||||
```
|
||||
|
||||
Run as a single-process `fxt-backtest sweep` (not Argo fan-out — only 8 cells).
|
||||
The threshold value with the highest **in-sample Sharpe on W0** is the
|
||||
production threshold. Persisted to `config/ml/v2_prod_thresholds.json` and
|
||||
committed.
|
||||
|
||||
The threshold is **frozen** at this point. It does not move.
|
||||
|
||||
**Pass 2 — Deployability (out-of-sample, full grid):**
|
||||
|
||||
`scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_deployability.yaml`. The
|
||||
deployability sweep includes the threshold axis for the descriptive surface;
|
||||
the verdict emitter looks up the *pre-registered* threshold and reads only that
|
||||
cell across windows.
|
||||
|
||||
### 2.4 Walk-forward windows
|
||||
|
||||
| Window | DBN file | Trading days | Purpose |
|
||||
|---|---|---|---|
|
||||
| W0 | 2025-Q1 | ~63 | Threshold pre-registration (in-sample for early-stop, OOS for trade-decision) |
|
||||
| W1 | 2025-Q2 | ~63 | Held-out validation #1 |
|
||||
| W2 | 2025-Q3 | ~63 | Held-out validation #2 |
|
||||
| W3 | 2025-Q4 | ~63 | Held-out validation #3 |
|
||||
| W4 | 2026-Q1 | ~63 | Held-out validation #4 |
|
||||
|
||||
### 2.5 Deployability sweep grid
|
||||
|
||||
```yaml
|
||||
# config/ml/sweep_v2_deployability.yaml
|
||||
checkpoint: /feature-cache/alpha-perception-runs/<sha>/trunk_best_h6000.bin
|
||||
windows: [W1, W2, W3, W4]
|
||||
axes:
|
||||
cost_tick: [0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5]
|
||||
latency_ms: [50, 100, 200, 400]
|
||||
threshold: [p70, p75, p80, p85, p90]
|
||||
decision_stride: 1
|
||||
horizons_used: [h6000]
|
||||
```
|
||||
|
||||
7 × 4 × 5 × 4 = **560 cells**, fanned out via `argo-lob-sweep.sh`. Per-cell
|
||||
runtime ~30 s on L40S (per real-LOB spec §6); wall-clock with 8-way fan-out
|
||||
is ~35 minutes. Total L40S time: ~5 hours.
|
||||
|
||||
### 2.6 Verdict emitter (`emit_deployability_verdict`)
|
||||
|
||||
Pseudocode (~50 lines in `crates/ml-backtesting/src/aggregate.rs`):
|
||||
|
||||
```
|
||||
1. Read v2_prod_thresholds.json → threshold_anchor (scalar; h6000 only)
|
||||
2. For each cell file in sweep_dir/cell_*/summary.json:
|
||||
Parse (cost_tick, latency_ms, threshold, window) + Sharpe + trades_count
|
||||
3. Filter to cells where (cost_tick == 1.0) AND (latency_ms == 200)
|
||||
AND (threshold == threshold_anchor)
|
||||
4. Group by window → expect exactly 4 cells (W1, W2, W3, W4)
|
||||
5. If any sharpe is NaN → emit verdict "Fail-degenerate"
|
||||
6. median_sharpe = median([sharpe_W1, sharpe_W2, sharpe_W3, sharpe_W4])
|
||||
7. If median_sharpe > 1.0 → "Pass"
|
||||
If median_sharpe ∈ [0.8, 1.0] → "Fail-inconclusive"
|
||||
Else → "Fail"
|
||||
8. Write deployability_verdict.json with all four window Sharpes,
|
||||
the median, the verdict string, the threshold vector, the anchor point,
|
||||
and the production training SHA. Commit to repo as the audit record.
|
||||
```
|
||||
|
||||
## 3. Smoke gate, testing, failure modes
|
||||
|
||||
### 3.1 Smoke gate (mandatory between training and full sweep)
|
||||
|
||||
After `trunk_best_h6000.bin` lands on `feature-cache-pvc`, run a one-cell
|
||||
smoke against W0 (the tuning window — cheap, already-known data):
|
||||
|
||||
```yaml
|
||||
# config/ml/sweep_v2_smoke.yaml
|
||||
checkpoint: /feature-cache/alpha-perception-runs/<sha>/trunk_best_h6000.bin
|
||||
windows: [W0]
|
||||
axes:
|
||||
cost_tick: [0.25]
|
||||
latency_ms: [200]
|
||||
threshold: [p80]
|
||||
decision_stride: 1
|
||||
horizons_used: [h6000]
|
||||
```
|
||||
|
||||
**Smoke pass criteria (ALL must hold to unlock the full sweep):**
|
||||
|
||||
| Check | Pass condition | Catches |
|
||||
|---|---|---|
|
||||
| Checkpoint loads | `CfcTrunk::load_checkpoint` returns `Ok` | Format drift (n_in/n_hid mismatch) |
|
||||
| Harness drives end-to-end | `fxt-backtest run` exits 0, `summary.json` written | Captured-graph alignment, mapped-pinned buffer ownership |
|
||||
| Non-degenerate output | `summary.json.trades_count > 0` AND `Sharpe` is finite | "Model never trades", `std(pnl)=0` divide-by-zero |
|
||||
| Per-decision latency | `mean_decision_latency_us < 100` on L40S | Stream-capture leak forcing host fallbacks ([[pearl_no_host_branches_in_captured_graph]]) |
|
||||
|
||||
Smoke failure halts the workflow and writes the failure reason to
|
||||
`deployability_verdict.json.smoke_status`. The full sweep is gated on smoke
|
||||
pass — there is no manual override path in this design.
|
||||
|
||||
Cost: ~30 s of L40S time. Insurance against a 5-hour failed full sweep.
|
||||
|
||||
### 3.2 Testing
|
||||
|
||||
| Layer | What's tested | Where | Status |
|
||||
|---|---|---|---|
|
||||
| Trunk save/load round-trip | bit-exact weight recovery | `crates/ml-alpha/src/cfc/trunk.rs:603` | Exists |
|
||||
| `cv-n-folds 1 cv-train-window 4` split | correct file partition | `alpha_train.rs:338–360` (existing branch) | Exists |
|
||||
| LOB sim per-cell determinism | seeded backtest bit-reproducible | Ring 1b parity test | Exists |
|
||||
| LOB sim invariants under fuzz | conserving P&L | Ring 2 fuzz (C18) | Exists |
|
||||
| **End-to-end with real checkpoint** | smoke YAML produces non-degenerate `summary.json` against trained checkpoint | `crates/ml-backtesting/tests/checkpoint_smoke.rs` *(`#[ignore]`, GPU, ~30 s)* | **NEW** |
|
||||
| **Verdict emitter format** | `deployability_verdict.json` round-trips through serde | `crates/ml-backtesting/src/aggregate.rs` test module | **NEW** |
|
||||
|
||||
Two new tests total. Everything else is pre-existing infrastructure exercised
|
||||
for the first time against a real trained model.
|
||||
|
||||
### 3.3 Failure modes
|
||||
|
||||
| Failure | Likelihood | Mitigation |
|
||||
|---|---|---|
|
||||
| Checkpoint format mismatch (train + backtest binaries built from different SHAs) | Medium | `load_checkpoint` validates n_in/n_hid; both binaries pinned to same `<sha>` by the smoke/sweep YAMLs referencing `/feature-cache/alpha-perception-runs/<sha>/trunk_best_h6000.bin` |
|
||||
| SSM-state leakage across windows | Low (Argo per-cell-pod fan-out gives fresh process per cell) | Argo workflow naturally isolates; plan-time verification that single-process `fxt-backtest sweep` also re-constructs harness per cell |
|
||||
| Decision-stride misalignment between trainer and backtester | Low | Both pinned `decision_stride: 1` in smoke YAML, threshold YAML, deployability YAML |
|
||||
| Model never trades at chosen threshold | Medium ([[pearl_snapshot_alpha_is_regime_conditional]]) | Smoke `trades_count > 0` check catches; pre-registration sweep includes wide threshold range (p60–p95) |
|
||||
| Window straddles contract roll | Low | Quarterly window boundaries align with quarterly contract cycles; roll periods are interior to each quarter, loader handles chronology |
|
||||
| Databento MBP-10 schema drift | Very low | Loader does in-band schema check, errors hard |
|
||||
| Sharpe = NaN due to `std(pnl)=0` | Low | Verdict emitter rejects NaN with distinct `Fail-degenerate` exit |
|
||||
|
||||
## 4. Implementation sequencing (preview for plan-writing skill)
|
||||
|
||||
Logical commit boundaries — the implementation plan will refine these into
|
||||
atomic commits:
|
||||
|
||||
1. Wire `save_checkpoint(best_h6000)` into `alpha_train.rs` + extend summary
|
||||
JSON with `best_h6000_ckpt_path`. Smoke unit test the path is written.
|
||||
2. Add the three new YAMLs under `config/ml/` (threshold-tuning, smoke,
|
||||
deployability). Lint-validate against `fxt-backtest sweep --validate`.
|
||||
3. Add `emit_deployability_verdict` in `aggregate.rs` + verdict round-trip
|
||||
test.
|
||||
4. Add `checkpoint_smoke.rs` integration test (`#[ignore]`, GPU).
|
||||
5. Submit Argo production training (`--cv-n-folds 1 --cv-train-window 4`).
|
||||
6. Once `trunk_best_h6000.bin` lands: smoke sweep against W0. Halt on failure.
|
||||
7. Threshold pre-registration sweep on W0. Commit `v2_prod_thresholds.json`.
|
||||
8. Deployability sweep on W1–W4. Aggregate. Emit verdict.
|
||||
9. Commit `deployability_verdict.json` as audit record. Update
|
||||
`project_ml_alpha_v2_ab_verdict` memory with the PnL-side close-out.
|
||||
|
||||
## 5. References
|
||||
|
||||
- Real-LOB integration spec: `docs/superpowers/specs/2026-05-18-real-lob-integration-design.md`
|
||||
- Real-LOB integration plan (C1–C19 already on branch): `docs/superpowers/plans/2026-05-18-real-lob-integration.md`
|
||||
- ml-alpha v2 multi-horizon spec: `docs/superpowers/specs/2026-05-18-ml-alpha-v2-multi-horizon-design.md`
|
||||
- 3-fold A/B verdict @ 6b7920474: memory `project_ml_alpha_v2_ab_verdict`
|
||||
- Phase 1d.4 cost-edge frontier: memory `pearl_phase1d4_backtest_cost_edge_frontier`
|
||||
- Single-window OOS warning: memory `pearl_single_window_oos_is_not_oos`
|
||||
- Look-ahead / no-deferral discipline: memory `pearl_no_deferrals_for_complementary_fixes`
|
||||
- Smoke-before-structural-priors: memory `feedback_smoke_validation_before_structural_priors`
|
||||
- Wire-everything-up (the producer-consumer gap this spec closes): memory `feedback_wire_everything_up`
|
||||
Reference in New Issue
Block a user