# ml-alpha v2 Deployability Validation — Design Spec

**Status:** SUPERSEDED on 2026-05-19 by `2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md`

**Reason:** This spec's §1.1 assumed `CfcTrunk::save_checkpoint` was the producer-side wiring point. Discovered at execution time that `alpha_train` trains via `PerceptionTrainer` (full v2 model with VSN + Mamba2 ×2 + LN ×2 + attn-pool + CfC + heads), not a `CfcTrunk` (CfC + heads + projection only). Closing this architectural gap requires a trunk-grows refactor that this spec did not include. The superseding spec covers both the refactor and the deployability validation in one document.

Kept in history as the audit trail of the design evolution. Implementation tracks the superseding spec only.

---

**Date:** 2026-05-19
**Branch:** `ml-alpha-phase-a`
**Owner:** ml-alpha team
**Spec depends on:** `2026-05-18-real-lob-integration-design.md` (commits C1–C19, already on branch)

## 0. Context

The 3-fold A/B sweep at commit `6b7920474` (see `project_ml_alpha_v2_ab_verdict`)
produced `mean_auc = 0.7510 ± 0.0117` and `auc_h6000 = 0.7604 ± 0.0034`. The
long-horizon signal is robust; the short-horizon regression matches the
[[pearl_input_feature_redundancy_loadbearing]] C16 pattern from the count-delta
slot revert. **AUC is not Sharpe.** Per `pearl_phase1d4_backtest_cost_edge_frontier`,
the Phase 1d.4 stacker model had frictionless Sharpe ~4.4 but break-even cost
was ~quarter-tick — half-tick killed it, retail 1-tick was deeply unprofitable.
Whether the v2 ml-alpha model is economically deployable is currently
**unknown** and cannot be answered from AUC alone.

The real-LOB backtesting system (`crates/ml-backtesting/`, `bin/fxt-backtest/`,
`scripts/argo-lob-sweep.sh`) was built out across commits C1–C19 on this branch
but has never been pointed at a real trained checkpoint, because
`crates/ml-alpha/examples/alpha_train.rs` does not currently call
`CfcTrunk::save_checkpoint`. This spec turns that single wiring gap into a
falsifiable deployability verdict.

**Pre-registered falsifiable claim:**

> The v2 ml-alpha model (commit `6b7920474`, production-checkpoint variant) is
> deployable iff, across 4 walk-forward held-out quarters at the realistic
> deployment anchor (200 ms RTT, 1-tick all-in cost, pre-registered threshold
> (h6000 only)), **median Sharpe > 1.0 AND median max-drawdown < 20%**.
> An additional stress anchor (400 ms RTT, 1.5-tick cost) on the same windows
> grades the Pass into **Pass-robust** (both anchors satisfy both gates) vs
> **Pass-nominal** (only realistic anchor satisfies both gates).

- **Pass-robust** → candidate for paper-trade plan (separate spec). Signal has
  real edge to spare even in adverse routing/spread regimes.
- **Pass-nominal** → candidate for paper-trade plan with explicit operational
  caveat that adverse routing/spread regimes break it. Recorded in audit.
- **Fail-inconclusive** (Sharpe ∈ [0.8, 1.0] OR max-dd ∈ [20%, 25%] at the
  realistic anchor) → counts as Fail. Re-evaluating post-hoc is not permitted.
- **Fail** otherwise → shelf decision for this architecture variant. Verdict
  committed.

**Out of scope (deferred to separate specs):**

- A/B vs Phase-A baseline `73925b15d` in PnL terms.
- Miscalibration diagnostic (best-h6000 vs best-mean_auc checkpoint comparison).
- Multi-seed ensembles.
- Per-regime stratified PnL (`pearl_snapshot_alpha_is_regime_conditional` follow-on).
- Live IBKR FIX/REST adapter (production deployment).

## 1. Architecture & components

### 1.1 Module touch-list (no new crates)

| File | Change | Lines |
|---|---|---|
| `crates/ml-alpha/examples/alpha_train.rs` | Inside the existing `auc_h6000_improved` block (~line 603), add `trunk_a.save_checkpoint(out_dir.join("trunk_best_h6000.bin"))?`. Add `best_h6000_ckpt_path` field to the `alpha_train_summary.json` schema. | ~10 |
| `config/ml/sweep_v2_deployability.yaml` *(new)* | Grid YAML for `fxt-backtest sweep` — see §2.5. | ~30 |
| `config/ml/sweep_v2_smoke.yaml` *(new)* | Single-cell smoke YAML — see §3.1. | ~15 |
| `config/ml/sweep_v2_threshold_tuning.yaml` *(new)* | Threshold pre-registration sweep YAML — see §2.3. | ~20 |
| `crates/ml-backtesting/src/aggregate.rs` | Add `emit_deployability_verdict(...)` function that reads the per-cell `summary.json` files, looks up both anchor cells (realistic + stress), computes median across windows for Sharpe / max-drawdown / Sortino / profit-factor, applies the tiered pass logic, and writes `deployability_verdict.json`. Also extend per-cell `summary.json` writer to emit Sortino, max_dd, profit_factor alongside the existing Sharpe field. | ~80 |
| `crates/ml-backtesting/tests/checkpoint_smoke.rs` *(new)* | `#[ignore]`d GPU integration test — load a trained checkpoint, drive the harness end-to-end against a tiny fixture window, assert smoke pass criteria. | ~80 |

**Read-only consumers (exercised on a real trained model for the first time):**

- `crates/ml-alpha/src/cfc/trunk.rs:470` `save_checkpoint` / `:539` `load_checkpoint`
- `crates/ml-backtesting/src/harness.rs` — drives loader → trunk → sim
- `bin/fxt-backtest/src/main.rs` — `run` / `sweep` / `aggregate` subcommands

### 1.2 Persistent artifacts

| Artifact | Producer | Consumer | Size |
|---|---|---|---|
| `trunk_best_h6000.bin` | Production training run (Argo `alpha-perception` template with `--cv-n-folds 1 --cv-train-window 4`) | All backtest cells | ~90 KB |
| `alpha_train_summary.json` (extended) | Same training run | Downstream tooling / verdict report | ~2 KB |
| `v2_prod_thresholds.json` *(new)* | Threshold pre-registration sweep (`fxt-backtest sweep` on W0 single-process) | Deployability sweep + verdict emitter | <1 KB |
| `aggregate.parquet` + `pareto_frontier.json` | `fxt-backtest aggregate` (existing C9 output) | Descriptive surface report | ~5 MB |
| `deployability_verdict.json` *(new)* | `emit_deployability_verdict` | Spec close-out, project memory | <1 KB |

### 1.3 Data corpus binding (concrete)

ES.FUT MBP-10 corpus on `training-data-pvc`: **quarterly DBN files spanning
2024-Q1 through 2026-Q1** (9 files total).

| Range | Position | Role |
|---|---|---|
| 2024-Q1..Q4 | files [0..4] | Training corpus (production checkpoint) |
| 2025-Q1 | file [4] | Validation file → threshold pre-registration window W0 |
| 2025-Q2 | file [5] | Walk-forward W1 (held-out) |
| 2025-Q3 | file [6] | Walk-forward W2 (held-out) |
| 2025-Q4 | file [7] | Walk-forward W3 (held-out) |
| 2026-Q1 | file [8] | Walk-forward W4 (held-out) |

The CLI flags that produce this split: `--cv-n-folds 1 --cv-train-window 4`.
The semantics are documented in `examples/alpha_train.rs` lines 338–360.

Each held-out quarter is ~63 trading days; 4 quarters × 63 days = ~252 trading
days = full year of held-out walk-forward. ES contract roll dates within each
quarter are handled by the existing loader chronology — no new mitigation.

## 2. Falsifiable criterion, anchor point, sweep configuration

### 2.1 Deployment anchors (realistic + stress)

Two anchor cells on the swept surface drive the verdict. The grid (§2.5)
already contains both as ordinary cells — "anchor" is a labelling, not a
separate run.

| Anchor | Latency | Cost (round-trip) | Threshold | Role |
|---|---|---|---|---|
| **Realistic** | 200 ms | 1 tick ($12.50) | Pre-registered (§2.3) | **Hard gate.** Pass requires both Sharpe and max-dd gates satisfied here. |
| **Stress** | 400 ms | 1.5 tick ($18.75) | Pre-registered (§2.3) | **Robustness grade.** Passing both gates here promotes Pass → Pass-robust. Failing here while passing realistic = Pass-nominal. |

Sources:

- 200 ms RTT: Scaleway PAR ↔ CME Aurora ~80–100 ms one-way + IBKR FIX overhead.
- 400 ms RTT: doubled-latency excursion plausible during routing degradation,
  network event, or wider-area failover.
- 1 tick: spread crossing (≥1 tick at IBKR retail) + $0.85 commission/side
  ≈ 1.07 tick.
- 1.5 tick: wider spread regime (low-liquidity sessions, around news, contract
  roll proximity).

Anchors are fixed in the spec. They are swept only for the descriptive surface
so we can see how the metrics behave *around* the anchors.

### 2.2 Metrics per window

Each (cell, window) tuple produces four risk-adjusted metrics, computed from
the per-window daily P&L series and trade ledger. Two are hard gates, two are
diagnostics.

| Metric | Definition | Role |
|---|---|---|
| **Annualized daily Sharpe** | `mean(daily_pnl) / std(daily_pnl) × √252` | **Hard gate.** Pass requires `median_across_windows > 1.0` at the relevant anchor. |
| **Max drawdown** | Peak-to-trough equity decline as a fraction of starting capital, per window | **Hard gate.** Pass requires `median_across_windows < 20%` at the relevant anchor. Captures capital-deployability — high-Sharpe strategies with deep drawdowns are unusable because the operator can't hold through them. |
| **Sortino ratio** | `mean(daily_pnl) / downside_deviation × √252`, where `downside_deviation = std(min(daily_pnl, 0))` | **Diagnostic.** Sharpe-vs-Sortino disagreement diagnoses asymmetric P&L. Sortino ≫ Sharpe ⇒ right-tail-heavy (good); Sortino ≪ Sharpe ⇒ left-tail-heavy (bad). Recorded, not gated. |
| **Profit factor** | `Σ winning_trades / |Σ losing_trades|` over the window's trade ledger | **Diagnostic.** PF > 1.2 in the median window indicates a real edge. PF ≈ 1.0 alongside positive Sharpe is the variance-gaming signature. Recorded, not gated. |

Cross-window aggregates use **median across the 4 walk-forward windows** for
all four metrics. Median is preferred over mean to keep the verdict robust to
single-quarter regime shocks.

If any window's `std(daily_pnl)` is zero (model never trades), that window's
Sharpe and Sortino are `NaN`. The verdict emitter rejects the run as
`Fail-degenerate` with a distinct exit code rather than propagating `NaN`
through the medians.

### 2.3 Threshold pre-registration (two-pass discipline)

Look-ahead bias is the failure mode that quietly invalidates the entire study.
The fix is two backtest passes with the threshold frozen between them.

**Pass 1 — Threshold-selection (in-sample, single sweep):**

```yaml
# config/ml/sweep_v2_threshold_tuning.yaml
checkpoint: /feature-cache/alpha-perception-runs/<sha>/trunk_best_h6000.bin
windows: [W0]                    # 2025-Q1 only
axes:
  cost_tick:  [1.0]              # anchor
  latency_ms: [200]              # anchor
  threshold:  [p60, p65, p70, p75, p80, p85, p90, p95]
decision_stride: 1
horizons_used: [h6000]
```

Run as a single-process `fxt-backtest sweep` (not Argo fan-out — only 8 cells).
The threshold value with the highest **in-sample Sharpe on W0** is the
production threshold. Persisted to `config/ml/v2_prod_thresholds.json` and
committed.

The threshold is **frozen** at this point. It does not move.

**Pass 2 — Deployability (out-of-sample, full grid):**

`scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_deployability.yaml`. The
deployability sweep includes the threshold axis for the descriptive surface;
the verdict emitter looks up the *pre-registered* threshold and reads only that
cell across windows.

### 2.4 Walk-forward windows

| Window | DBN file | Trading days | Purpose |
|---|---|---|---|
| W0 | 2025-Q1 | ~63 | Threshold pre-registration (in-sample for early-stop, OOS for trade-decision) |
| W1 | 2025-Q2 | ~63 | Held-out validation #1 |
| W2 | 2025-Q3 | ~63 | Held-out validation #2 |
| W3 | 2025-Q4 | ~63 | Held-out validation #3 |
| W4 | 2026-Q1 | ~63 | Held-out validation #4 |

### 2.5 Deployability sweep grid

```yaml
# config/ml/sweep_v2_deployability.yaml
checkpoint: /feature-cache/alpha-perception-runs/<sha>/trunk_best_h6000.bin
windows: [W1, W2, W3, W4]
axes:
  cost_tick:  [0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5]
  latency_ms: [50, 100, 200, 400]
  threshold:  [p70, p75, p80, p85, p90]
decision_stride: 1
horizons_used: [h6000]
```

7 × 4 × 5 × 4 = **560 cells**, fanned out via `argo-lob-sweep.sh`. Per-cell
runtime ~30 s on L40S (per real-LOB spec §6); wall-clock with 8-way fan-out
is ~35 minutes. Total L40S time: ~5 hours.

### 2.6 Verdict emitter (`emit_deployability_verdict`)

Pseudocode (~80 lines in `crates/ml-backtesting/src/aggregate.rs`):

```
1. Read v2_prod_thresholds.json → threshold_anchor (scalar; h6000 only)
2. For each cell file in sweep_dir/cell_*/summary.json:
     Parse (cost_tick, latency_ms, threshold, window)
     + Sharpe + Sortino + max_dd + profit_factor + trades_count
3. ANCHORS = [
     ("realistic", cost_tick=1.0,  latency_ms=200),
     ("stress",    cost_tick=1.5,  latency_ms=400),
   ]
4. For each anchor in ANCHORS:
     Filter to cells matching anchor + (threshold == threshold_anchor)
     Group by window → expect exactly 4 cells (W1, W2, W3, W4)
     If any Sharpe or Sortino is NaN OR trades_count == 0 in any window
        → record anchor.status = "Fail-degenerate"
        → continue
     anchor.median_sharpe = median(sharpe_per_window)
     anchor.median_max_dd = median(max_dd_per_window)
     anchor.median_sortino = median(sortino_per_window)
     anchor.median_pf = median(pf_per_window)
     anchor.gate_sharpe = anchor.median_sharpe > 1.0
     anchor.gate_max_dd = anchor.median_max_dd < 0.20
     anchor.pass = anchor.gate_sharpe AND anchor.gate_max_dd
5. realistic = ANCHORS["realistic"]
   stress = ANCHORS["stress"]
   If realistic.status == "Fail-degenerate" → verdict = "Fail-degenerate"
   Else if realistic.pass AND stress.pass → verdict = "Pass-robust"
   Else if realistic.pass → verdict = "Pass-nominal"
   Else if realistic.median_sharpe ∈ [0.8, 1.0]
        OR realistic.median_max_dd ∈ [0.20, 0.25]
        → verdict = "Fail-inconclusive"
   Else → verdict = "Fail"
6. Write deployability_verdict.json containing:
   - verdict string (one of: Pass-robust, Pass-nominal, Fail-inconclusive,
     Fail, Fail-degenerate)
   - per-anchor: median + per-window arrays for all four metrics
                 (Sharpe, Sortino, max_dd, profit_factor)
   - per-anchor: gate_sharpe, gate_max_dd booleans
   - threshold value, anchor specs, production training SHA, spec SHA,
     timestamp
7. Commit deployability_verdict.json to repo as audit record.
```

## 3. Smoke gate, testing, failure modes

### 3.1 Smoke gate (mandatory between training and full sweep)

After `trunk_best_h6000.bin` lands on `feature-cache-pvc`, run a one-cell
smoke against W0 (the tuning window — cheap, already-known data):

```yaml
# config/ml/sweep_v2_smoke.yaml
checkpoint: /feature-cache/alpha-perception-runs/<sha>/trunk_best_h6000.bin
windows: [W0]
axes:
  cost_tick:  [0.25]
  latency_ms: [200]
  threshold:  [p80]
decision_stride: 1
horizons_used: [h6000]
```

**Smoke pass criteria (ALL must hold to unlock the full sweep):**

| Check | Pass condition | Catches |
|---|---|---|
| Checkpoint loads | `CfcTrunk::load_checkpoint` returns `Ok` | Format drift (n_in/n_hid mismatch) |
| Harness drives end-to-end | `fxt-backtest run` exits 0, `summary.json` written | Captured-graph alignment, mapped-pinned buffer ownership |
| Non-degenerate output | `summary.json.trades_count > 0` AND `Sharpe` is finite | "Model never trades", `std(pnl)=0` divide-by-zero |
| Per-decision latency | `mean_decision_latency_us < 100` on L40S | Stream-capture leak forcing host fallbacks ([[pearl_no_host_branches_in_captured_graph]]) |

Smoke failure halts the workflow and writes the failure reason to
`deployability_verdict.json.smoke_status`. The full sweep is gated on smoke
pass — there is no manual override path in this design.

Cost: ~30 s of L40S time. Insurance against a 5-hour failed full sweep.

### 3.2 Testing

| Layer | What's tested | Where | Status |
|---|---|---|---|
| Trunk save/load round-trip | bit-exact weight recovery | `crates/ml-alpha/src/cfc/trunk.rs:603` | Exists |
| `cv-n-folds 1 cv-train-window 4` split | correct file partition | `alpha_train.rs:338–360` (existing branch) | Exists |
| LOB sim per-cell determinism | seeded backtest bit-reproducible | Ring 1b parity test | Exists |
| LOB sim invariants under fuzz | conserving P&L | Ring 2 fuzz (C18) | Exists |
| **End-to-end with real checkpoint** | smoke YAML produces non-degenerate `summary.json` against trained checkpoint | `crates/ml-backtesting/tests/checkpoint_smoke.rs` *(`#[ignore]`, GPU, ~30 s)* | **NEW** |
| **Verdict emitter format** | `deployability_verdict.json` round-trips through serde | `crates/ml-backtesting/src/aggregate.rs` test module | **NEW** |

Two new tests total. Everything else is pre-existing infrastructure exercised
for the first time against a real trained model.

### 3.3 Failure modes

| Failure | Likelihood | Mitigation |
|---|---|---|
| Checkpoint format mismatch (train + backtest binaries built from different SHAs) | Medium | `load_checkpoint` validates n_in/n_hid; both binaries pinned to same `<sha>` by the smoke/sweep YAMLs referencing `/feature-cache/alpha-perception-runs/<sha>/trunk_best_h6000.bin` |
| SSM-state leakage across windows | Low (Argo per-cell-pod fan-out gives fresh process per cell) | Argo workflow naturally isolates; plan-time verification that single-process `fxt-backtest sweep` also re-constructs harness per cell |
| Decision-stride misalignment between trainer and backtester | Low | Both pinned `decision_stride: 1` in smoke YAML, threshold YAML, deployability YAML |
| Model never trades at chosen threshold | Medium ([[pearl_snapshot_alpha_is_regime_conditional]]) | Smoke `trades_count > 0` check catches; pre-registration sweep includes wide threshold range (p60–p95) |
| Window straddles contract roll | Low | Quarterly window boundaries align with quarterly contract cycles; roll periods are interior to each quarter, loader handles chronology |
| Databento MBP-10 schema drift | Very low | Loader does in-band schema check, errors hard |
| Sharpe = NaN due to `std(pnl)=0` | Low | Verdict emitter rejects NaN with distinct `Fail-degenerate` exit |
| Max-drawdown computation buggy (peak tracking off-by-one, intra-day vs end-of-day basis) | Medium — new code path | Verdict round-trip test (§3.2) covers the field's presence + round-trip; a separate unit test in `aggregate.rs` covers max-dd math against a known fixture P&L curve |
| Sortino-Sharpe agree on a degenerate window (both flat, both ~0) but the model is producing real trades elsewhere | Low | Verdict emitter requires `trades_count > 0` per window before counting that window in the median; "Fail-degenerate" exit on zero-trade windows |

## 4. Implementation sequencing (preview for plan-writing skill)

Logical commit boundaries — the implementation plan will refine these into
atomic commits:

1. Wire `save_checkpoint(best_h6000)` into `alpha_train.rs` + extend summary
   JSON with `best_h6000_ckpt_path`. Smoke unit test the path is written.
2. Add the three new YAMLs under `config/ml/` (threshold-tuning, smoke,
   deployability). Lint-validate against `fxt-backtest sweep --validate`.
3. Add `emit_deployability_verdict` in `aggregate.rs` + verdict round-trip
   test.
4. Add `checkpoint_smoke.rs` integration test (`#[ignore]`, GPU).
5. Submit Argo production training (`--cv-n-folds 1 --cv-train-window 4`).
6. Once `trunk_best_h6000.bin` lands: smoke sweep against W0. Halt on failure.
7. Threshold pre-registration sweep on W0. Commit `v2_prod_thresholds.json`.
8. Deployability sweep on W1–W4. Aggregate. Emit verdict.
9. Commit `deployability_verdict.json` as audit record. Update
   `project_ml_alpha_v2_ab_verdict` memory with the PnL-side close-out.

## 5. References

- Real-LOB integration spec: `docs/superpowers/specs/2026-05-18-real-lob-integration-design.md`
- Real-LOB integration plan (C1–C19 already on branch): `docs/superpowers/plans/2026-05-18-real-lob-integration.md`
- ml-alpha v2 multi-horizon spec: `docs/superpowers/specs/2026-05-18-ml-alpha-v2-multi-horizon-design.md`
- 3-fold A/B verdict @ 6b7920474: memory `project_ml_alpha_v2_ab_verdict`
- Phase 1d.4 cost-edge frontier: memory `pearl_phase1d4_backtest_cost_edge_frontier`
- Single-window OOS warning: memory `pearl_single_window_oos_is_not_oos`
- Look-ahead / no-deferral discipline: memory `pearl_no_deferrals_for_complementary_fixes`
- Smoke-before-structural-priors: memory `feedback_smoke_validation_before_structural_priors`
- Wire-everything-up (the producer-consumer gap this spec closes): memory `feedback_wire_everything_up`
