gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spec(ml-alpha): trunk-grows refactor + deployability validation (supersedes prior)

Browse Source 
Supersedes the 2026-05-19 deployability spec (commit 07d5de504). The
prior spec assumed CfcTrunk::save_checkpoint was the producer-side
wiring point — discovered at execution time that alpha_train trains via
PerceptionTrainer (full v2: VSN + Mamba2 ×2 + LN ×2 + attn-pool + CfC +
heads), not the simpler CfcTrunk. The existing LOB backtester loads
CheckpointV1 envelopes that only know about CfC weights, so there is no
producer for a checkpoint containing the full v2 model.

New scope: one bigger spec covering refactor + deployability end-to-end.

Phase 1 (X0–X19, code commits): grow CfcTrunk to own the full v2
inference graph; restructure PerceptionTrainer to wrap a trunk + add
training-only state (grads, AdamW). Discipline: bit-equivalence golden
fixture (X0) gates every refactor commit (X1–X11). CheckpointV2
envelope (X12) replaces V1. Verdict emitter (X17) reuses the tiered
classification (Pass-robust / Pass-nominal / Fail-inconclusive / Fail /
Fail-degenerate) from the superseded spec.

Phase 2 (Argo runtime): production training → smoke gate → threshold
pre-registration → 560-cell deployability sweep → verdict + memory
update.

Hard gate before Phase 2: post-refactor fold-0 smoke must reproduce
recorded 3-fold A/B numbers (best_mean_auc 0.7529, best_h6000 0.7639,
both within ±0.010 absolute) from project_ml_alpha_v2_ab_verdict
memory. Prior spec marked SUPERSEDED in its header, kept in history as
audit trail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-19 00:54:45 +02:00
parent 4938ac2ec5
commit da1dd92bf8
2 changed files with 423 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md
View File

@@ -1,6 +1,13 @@
# ml-alpha v2 Deployability Validation — Design Spec # ml-alpha v2 Deployability Validation — Design Spec
**Status:** DESIGN — approved through brainstorming, awaiting user spec-review before plan handoff **Status:** SUPERSEDED on 2026-05-19 by `2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md`
**Reason:** This spec's §1.1 assumed `CfcTrunk::save_checkpoint` was the producer-side wiring point. Discovered at execution time that `alpha_train` trains via `PerceptionTrainer` (full v2 model with VSN + Mamba2 ×2 + LN ×2 + attn-pool + CfC + heads), not a `CfcTrunk` (CfC + heads + projection only). Closing this architectural gap requires a trunk-grows refactor that this spec did not include. The superseding spec covers both the refactor and the deployability validation in one document.
Kept in history as the audit trail of the design evolution. Implementation tracks the superseding spec only.
---
**Date:** 2026-05-19 **Date:** 2026-05-19
**Branch:** `ml-alpha-phase-a` **Branch:** `ml-alpha-phase-a`
**Owner:** ml-alpha team **Owner:** ml-alpha team

415
docs/superpowers/specs/2026-05-19-ml-alpha-v2-trunk-grows-and-deployability-design.md Normal file
View File

@@ -0,0 +1,415 @@
# ml-alpha v2 Trunk-Grows Refactor + Deployability Validation
**Status:** DESIGN — approved through brainstorming, awaiting user spec-review before plan handoff
**Date:** 2026-05-19
**Branch:** `ml-alpha-phase-a`
**Owner:** ml-alpha team
**Supersedes:** `docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md` (commit `07d5de504`) — the prior spec assumed `CfcTrunk::save_checkpoint` was the wiring point. That assumption was architecturally wrong: alpha_train trains via `PerceptionTrainer` (VSN + Mamba2 ×2 + LN ×2 + attn-pool + CfC + heads), not the simpler CfcTrunk. This spec subsumes the deployability validation work and prepends the architectural refactor required to make it possible.
## 0. Context
Two distinct gaps prevent answering the v2 ml-alpha deployability question:
**Gap 1 (architectural):** The training graph (`PerceptionTrainer` — full v2 model) and the inference graph used by the LOB backtester (`CfcTrunk` — CfC + heads + projection only) are different shapes. The backtester loads `CheckpointV1` envelopes that only know about CfC weights. There is no producer for a checkpoint that contains the full v2 model. Closing this gap requires growing `CfcTrunk` into the full v2 inference graph and restructuring `PerceptionTrainer` to wrap a trunk + add training-only state (gradients, AdamW).
**Gap 2 (deployment metrics):** Once Gap 1 is closed, the question "is the v2 model economically deployable at the realistic Scaleway→IBKR anchor" is still unanswered. The infrastructure to answer it (LOB harness, sweep tool, Argo workflow) exists at commits C1–C19 on this branch. Pre-registered thresholds + tiered verdict (Pass-robust / Pass-nominal / Fail-inconclusive / Fail / Fail-degenerate) at realistic + stress anchors with 4 metrics ($35k-base Sharpe / max-dd / Sortino / profit-factor) define a falsifiable test.
This spec covers both gaps as one project: Phase 1 (trunk-grows refactor, 19 atomic commits) then Phase 2 (production training + smoke + sweeps + verdict, ~3 runtime steps).
**Pre-registered falsifiable claim** (unchanged from superseded spec):
> The v2 ml-alpha model (production-checkpoint variant trained after the trunk-grows refactor lands) is deployable iff, across 4 walk-forward held-out quarters at the realistic deployment anchor (200 ms RTT, 1-tick all-in cost, pre-registered threshold), **median Sharpe > 1.0 AND median max-drawdown < 20% of $35k starting capital**. An additional stress anchor (400 ms RTT, 1.5-tick cost) grades the Pass into Pass-robust vs Pass-nominal.
**Out of scope:**
- A/B comparison of v2 vs Phase-A baseline (`73925b15d`) in PnL terms. Per the 3-fold AUC logs already in [[project_ml_alpha_v2_ab_verdict]], the AUC-side A/B already settled; PnL-side A/B is a separate question.
- Miscalibration diagnostic (best-h6000 vs best-mean_auc checkpoint comparison).
- Multi-seed ensembles.
- Per-regime stratified PnL.
- Live IBKR FIX/REST adapter.
- Re-architecting `PerceptionTrainer.step_batched` or any training-side logic beyond what the trunk-grows refactor requires.
## 1. Architecture & components
### 1.1 Trunk-grows refactor — ownership boundaries
```
CfcTrunk (inference graph — captured-graph capable forward)
├── all weight device tensors (moved from PerceptionTrainer):
│ - vsn_w_d, vsn_b_d (VSN)
│ - mamba2_stack_1 (full Mamba2Block weights) (Mamba2 L1)
│ - ln_a_gain_d, ln_a_bias_d (LN_a)
│ - mamba2_stack_2 (full Mamba2Block weights) (Mamba2 L2)
│ - ln_b_gain_d, ln_b_bias_d (LN_b)
│ - attn_q_d (attention pool)
│ - w_in_d, w_rec_d, b_d, tau_d (CfC)
│ - heads_w_gate/main/skip_d × {gate,main,skip} + biases (heads)
├── captured graph A: snap_features → vsn → mamba2_l1 → ln_a → mamba2_l2 → ln_b → attn_pool → cfc_step → multi_horizon_heads → projection
├── pub fn new_random(&MlDevice, &CfcConfig, seed: u64) -> Result<Self>
├── pub fn capture_graph_a(&mut self, &Mbp10RawInput) -> Result<()>
├── pub fn update_input_buffers(&mut self, &Mbp10RawInput) -> Result<()>
├── pub fn perception_forward_captured(&mut self) -> Result<([f32; N_HORIZONS], [f32; PROJ_DIM])>
├── pub fn save_checkpoint(&self, &Path) -> Result<()> // writes CheckpointV2
├── pub fn load_checkpoint(&MlDevice, &CfcConfig, &Path) -> Result<Self>
└── (per-kernel forward helpers, called by trainer's evaluate_batched + by the graph capture)
PerceptionTrainer (training wrapper)
├── trunk: CfcTrunk // owns the inference graph
├── all grad_*_d device tensors // training-only
├── AdamW optimizers (opt_*) // training-only
├── backward kernels (cfc_step_bwd, mamba2_*_bwd, etc.) // training-only
├── ISV controllers (lambda, log_sigma_h) // training-only
├── pub fn step / step_batched (unchanged behavior)
├── pub fn evaluate / evaluate_batched (now delegates forward to self.trunk)
└── pub fn save_checkpoint(&self, path: &Path) -> Result<()> // delegates to self.trunk.save_checkpoint
```
**Naming:** Keep `CfcTrunk`. The `Cfc` prefix becomes a historical artifact; the trunk metaphor still fits as "the whole inference graph from input to per-horizon probs." Renaming would cascade through every caller for cosmetic benefit only.
### 1.2 CheckpointV2 envelope
```rust
#[derive(Serialize, Deserialize)]
struct CheckpointV2 {
version: u32, // 2
n_in: usize,
n_hid: usize,
mamba2_state_dim: usize,
// VSN
vsn_w: Vec<f32>, vsn_b: Vec<f32>,
// Mamba2 stack 1
m1_w_in: Vec<f32>, m1_w_a: Vec<f32>, m1_w_b: Vec<f32>,
m1_w_c: Vec<f32>, m1_w_out: Vec<f32>,
// LN_a
ln_a_gain: Vec<f32>, ln_a_bias: Vec<f32>,
// Mamba2 stack 2 (same fields as stack 1)
m2_w_in: Vec<f32>, m2_w_a: Vec<f32>, m2_w_b: Vec<f32>,
m2_w_c: Vec<f32>, m2_w_out: Vec<f32>,
// LN_b
ln_b_gain: Vec<f32>, ln_b_bias: Vec<f32>,
// Attention pool
attn_q: Vec<f32>,
// CfC
w_in: Vec<f32>, w_rec: Vec<f32>, b: Vec<f32>, tau: Vec<f32>,
// Heads (GRN structure × 5 horizons)
heads_w_gate: Vec<f32>, heads_b_gate: Vec<f32>,
heads_w_main: Vec<f32>, heads_b_main: Vec<f32>,
heads_w_skip: Vec<f32>, heads_b_skip: Vec<f32>,
}
```
Serialized via `bincode` (same approach as `CheckpointV1`). Rough size: ~1.5–2 MB.
**V1 → V2 migration:** Hard break. `load_checkpoint` returns an error on `version != 2`. No existing V1 files in the wild — alpha_train never produced them, so nothing to migrate. The fxt-backtest CLI's existing `--checkpoint` flag uses V2 going forward.
### 1.3 Module touch-list
| File | Change | Phase |
|---|---|---|
| `crates/ml-alpha/tests/fixtures/perception_forward_golden.bin` | NEW — golden output bytes captured pre-refactor | X0 |
| `crates/ml-alpha/tests/perception_forward_golden.rs` | NEW — bit-equivalence test against golden | X0, asserted X1..X11 |
| `crates/ml-alpha/src/cfc/trunk.rs` | EXTEND — add all v2 weight fields, V2 envelope, save/load, capture_graph_a expanded to full forward | X1, X11, X12 |
| `crates/ml-alpha/src/trainer/perception.rs` | RESTRUCTURE — move weight ownership to `self.trunk`; forward kernels delegate to trunk methods; grad/optimizer state stays | X2..X10 (incremental) |
| `crates/ml-alpha/examples/alpha_train.rs` | EXTEND — call `trainer.save_checkpoint(out_dir.join("trunk_best_h6000.bin"))` inside auc_h6000_improved block; add `best_h6000_ckpt_path` field on `AlphaTrainSummary` | X14 |
| `crates/ml-backtesting/src/harness.rs` | NO INTERFACE CHANGE — already takes a `CfcTrunk`; just exercises the V2 loader path | X15 (verification only) |
| `bin/fxt-backtest/src/main.rs` | NO INTERFACE CHANGE — loads `CfcTrunk` via `--checkpoint` flag; V2 envelope is transparent to the CLI | X15 (verification only) |
| `crates/ml-backtesting/src/artifacts.rs` | EXTEND — add `max_drawdown_pct: f32` field on `Summary`; pin `STARTING_CAPITAL_USD = 35_000.0`; update `compute_summary` | X16 |
| `crates/ml-backtesting/src/aggregate.rs` | EXTEND — add `VerdictTier`, `AnchorReport`, `DeployabilityVerdict` types; `classify_verdict` + `emit_deployability_verdict` functions | X17 |
| `crates/ml-backtesting/tests/checkpoint_smoke.rs` | NEW — `#[ignore]`d GPU integration test loading real trained checkpoint, running one cell | X18 |
| `config/ml/sweep_v2_smoke.yaml` | NEW — single-cell smoke gate config | X19 |
| `config/ml/sweep_v2_threshold_tuning.yaml` | NEW — 8-cell threshold pre-registration config | X19 |
| `config/ml/sweep_v2_deployability.yaml` | NEW — 560-cell full sweep config | X19 |
### 1.4 Persistent artefacts produced by the workflow
| Artefact | Producer | Consumer | Size |
|---|---|---|---|
| `crates/ml-alpha/tests/fixtures/perception_forward_golden.bin` | X0 test (committed to repo) | X1..X11 bit-equivalence gates | ~5 KB |
| `trunk_best_h6000.bin` (CheckpointV2) | Production training run | All backtest cells | ~1.5 MB |
| `alpha_train_summary.json` (extended with `best_h6000_ckpt_path`) | Production training run | Audit + sweep config resolution | ~3 KB |
| `config/ml/v2_prod_thresholds.json` | Threshold pre-registration sweep | Deployability sweep + verdict emitter | <1 KB |
| `aggregate.parquet` + `pareto_frontier.json` | Deployability sweep aggregate | Descriptive surface report | ~5 MB |
| `deployability_verdict.json` | Verdict emitter (X17) | Audit record committed to repo | <1 KB |
### 1.5 Data corpus binding (unchanged from superseded spec)
ES.FUT MBP-10 corpus on `training-data-pvc`: **quarterly DBN files spanning
2024-Q1 through 2026-Q1** (9 files total).
| Range | Position | Role |
|---|---|---|
| 2024-Q1..Q4 | files [0..4] | Training corpus (production checkpoint, `--cv-n-folds 1 --cv-train-window 4`) |
| 2025-Q1 | file [4] | Validation file → threshold pre-registration window W0 |
| 2025-Q2 | file [5] | Walk-forward W1 (held-out) |
| 2025-Q3 | file [6] | Walk-forward W2 (held-out) |
| 2025-Q4 | file [7] | Walk-forward W3 (held-out) |
| 2026-Q1 | file [8] | Walk-forward W4 (held-out) |
Each held-out quarter is ~63 trading days; 4 quarters × 63 = ~252 trading days of held-out walk-forward.
## 2. Refactor sequencing + bit-equivalence test discipline
### 2.1 Golden fixture (X0 — MUST land before any refactor commit)
A deterministic fixture sequence + label sequence drives a single
`PerceptionTrainer.evaluate_batched` call at seed=42 on the pre-refactor
branch tip. The resulting `(per-horizon probs, BCE loss)` is committed
as `crates/ml-alpha/tests/fixtures/perception_forward_golden.bin`. Every
subsequent commit must reproduce these bytes to a tight tolerance.
```rust
// crates/ml-alpha/tests/perception_forward_golden.rs
//
// Sets up a deterministic 32-snapshot fixture, constructs
// PerceptionTrainer with seed=42, runs evaluate_batched, compares
// output to the golden file at f32 max-abs-diff < 1e-5.
//
// Run with: cargo test -p ml-alpha --test perception_forward_golden
// (no GPU host: --ignored; with GPU: default; bit-exactness depends on
// GPU model — see §2.4 nondeterminism handling)
```
The golden file is recorded with **one fixed GPU** (3050 local). L40S verification is confirmatory, not bit-exact — see §2.4.
### 2.2 Commit ladder (X0–X18 code commits + X19 YAMLs)
| # | Title | Test gate |
|---|---|---|
| X0 | `test(ml-alpha): perception_forward_golden fixture + bit-equivalence test` | New test passes against current evaluate_batched output |
| X1 | `feat(ml-alpha): CfcTrunk v2 weight skeleton (no callers yet)` | Compiles; existing tests untouched |
| X2 | `refactor(ml-alpha): move VSN weights from PerceptionTrainer to CfcTrunk` | Golden passes |
| X3 | `refactor(ml-alpha): move Mamba2 stack 1 weights to CfcTrunk` | Golden passes |
| X4 | `refactor(ml-alpha): move LN_a weights to CfcTrunk` | Golden passes |
| X5 | `refactor(ml-alpha): move Mamba2 stack 2 weights to CfcTrunk` | Golden passes |
| X6 | `refactor(ml-alpha): move LN_b weights to CfcTrunk` | Golden passes |
| X7 | `refactor(ml-alpha): move attention-pool weights to CfcTrunk` | Golden passes |
| X8 | `refactor(ml-alpha): move CfC weights to CfcTrunk` | Golden passes |
| X9 | `refactor(ml-alpha): move heads weights to CfcTrunk` | Golden passes |
| X10 | `refactor(ml-alpha): hoist forward kernels into CfcTrunk methods` | Golden passes |
| X11 | `feat(ml-alpha): CfcTrunk capture_graph_a covers full v2 forward` | NEW captured-vs-uncaptured bit-equivalence test passes |
| X12 | `feat(ml-alpha): CheckpointV2 envelope + save_checkpoint + load_checkpoint` | Round-trip test: save → load → bit-exact weights; load errors on `version != 2` |
| X13 | `feat(ml-alpha): PerceptionTrainer.save_checkpoint delegates to trunk` | Unit test: trainer.save_checkpoint produces a file trunk.load_checkpoint accepts |
| X14 | `feat(ml-alpha): alpha_train saves best_h6000 checkpoint` | Summary serde round-trip test for `best_h6000_ckpt_path` |
| X15 | `verify(ml-backtesting): harness + fxt-backtest load CheckpointV2` | Existing harness tests pass against a fixture CheckpointV2 |
| X16 | `feat(ml-backtesting): max_drawdown_pct on per-cell Summary` | Fixture test: $7k drawdown → 20% at $35k base |
| X17 | `feat(ml-backtesting): emit_deployability_verdict + tiered logic` | 6 unit tests covering each verdict tier + fixture sweep round-trip |
| X18 | `test(ml-backtesting): GPU smoke against real trained checkpoint` | `#[ignore]`d integration test compiles + passes when env vars set |
| X19 | `config(ml-alpha): three sweep YAMLs for deployability validation` | YAML lint validation |
Each commit is atomic and rollback-safe. If any X1..X11 commit fails the golden gate, **revert that commit and diagnose** — do not chain-fix in a follow-up commit. The golden gate is the safety net for the refactor; preserving it preserves the option to roll back without re-running 3-fold A/B.
### 2.3 Final pre-Argo smoke gate (between X18 and Phase 2 runtime)
Before paying for the production training run + full deployability sweep,
run a **single fold** of the 3-fold A/B at the original settings to
verify the refactor preserved training behavior end-to-end:
```bash
git push origin ml-alpha-phase-a
./scripts/argo-train.sh \
--branch ml-alpha-phase-a \
--commit <X19-tip> \
--epochs 30 \
--cv-fold 0 --cv-n-folds 3 \
--cv-train-window 0 \
--gpu-pool ci-training-l40s --watch
```
Compare to the **recorded fold 0 result** from
[[project_ml_alpha_v2_ab_verdict]]:
| Metric | Pre-refactor (commit 6b7920474) | Post-refactor tolerance |
|---|---|---|
| `best_mean_auc` (epoch) | 0.7529 (e11) | ≤ ±0.010 absolute, ±3 epochs |
| `best_auc_h6000` (epoch) | 0.7639 (e11) | ≤ ±0.010 absolute, ±3 epochs |
If post-refactor fold-0 numbers fall outside tolerance, the refactor broke
training in a way the small fixtures didn't catch. **Stop**, diagnose
the divergence (likely a numerical kernel issue caught only by the full
training trajectory), fix, re-run fold-0 smoke. Do NOT proceed to
production training until this gate passes.
### 2.4 Nondeterminism handling
| Source | Mitigation |
|---|---|
| CUDA non-deterministic reductions (atomicAdd) | Forbidden in trunk kernels per [[feedback_no_atomicadd]]. If golden bytes turn out non-deterministic on the same GPU model, that's a regression of the rule — fix the kernel. |
| Cross-GPU drift (3050 vs L40S small fp divergences) | Golden pinned to 3050 local; L40S runs use **tight-tolerance comparison** (`max_abs_diff < 1e-4`) rather than bit-exact. Document the tolerance choice in `perception_forward_golden.rs` with the rationale. |
### 2.5 Phase 1 completion criteria
Phase 1 (X0–X19) is complete when:
1. All 20 commits land cleanly with passing pre-commit hooks
2. `cargo test -p ml-alpha` and `cargo test -p ml-backtesting` are green
3. The post-refactor fold-0 smoke (§2.3) lands within tolerance
4. The golden test (`perception_forward_golden`) still passes at the X19 tip
Phase 2 (Argo runtime) does not begin until all four criteria hold.
## 3. Deployability validation (Phase 2)
The verdict architecture is unchanged from the superseded spec — anchors,
metrics, thresholds, walk-forward windows, sweep grid, verdict tiers,
and smoke gate all carry over byte-for-byte. This section restates the
contract briefly; refer to the superseded spec (`07d5de504`) for the full
rationale.
### 3.1 Deployment anchors
| Anchor | Latency | Cost (round-trip) | Threshold | Role |
|---|---|---|---|---|
| **Realistic** | 200 ms | 1 tick ($12.50) | Pre-registered on W0 | **Hard gate.** Pass requires both Sharpe and max-dd gates satisfied here. |
| **Stress** | 400 ms | 1.5 tick ($18.75) | Pre-registered on W0 | **Robustness grade.** Passing both gates here promotes Pass → Pass-robust. |
### 3.2 Metrics (per window)
| Metric | Definition | Role |
|---|---|---|
| **Annualized daily Sharpe** | `mean(daily_pnl) / std(daily_pnl) × √252` | **Hard gate** — median across windows > 1.0 at the relevant anchor |
| **Max drawdown %** | `|max_drawdown_usd| / STARTING_CAPITAL_USD` with `STARTING_CAPITAL_USD = 35_000.0` (see [[project_ml_alpha_starting_capital]]) | **Hard gate** — median across windows < 20% at the relevant anchor |
| **Sortino ratio** | `mean(daily_pnl) / downside_deviation × √252` | **Diagnostic** — recorded for asymmetry analysis |
| **Profit factor** | `Σ winners / |Σ losers|` per window | **Diagnostic** — recorded for edge-vs-variance-gaming detection |
Cross-window aggregate uses **median across the 4 walk-forward windows**.
### 3.3 Two-pass threshold pre-registration
**Pass 1 (in-sample, single sweep):** `fxt-backtest sweep` over 8
thresholds (p60–p95 in 5pt steps) at the realistic anchor cost+latency
against W0 (2025-Q1). Select threshold with highest in-sample Sharpe.
Persist to `config/ml/v2_prod_thresholds.json`. Commit.
**Pass 2 (out-of-sample, full grid):** `argo-lob-sweep.sh` runs
7 (cost) × 4 (latency) × 5 (threshold) × 4 (window) = **560 cells**.
Threshold axis is for the descriptive surface; the verdict reads the
pre-registered threshold value.
### 3.4 Smoke gate (X18 test + runtime smoke sweep)
Before unlocking the full deployability sweep, run a one-cell smoke
against W0:
```yaml
# config/ml/sweep_v2_smoke.yaml
checkpoint: /feature-cache/alpha-perception-runs/__SHA__/trunk_best_h6000.bin
windows: [{id: W0, dbn_file: /data/futures-baseline-mbp10/ES.FUT/ES.FUT_2025-Q1.dbn.zst}]
axes: {cost_tick: [0.25], latency_ms: [200], threshold: [0.75]}
decision_stride: 1
horizons_used: [h6000]
```
Smoke pass criteria:
- Checkpoint loads (CheckpointV2 parse + GPU upload)
- Harness drives end-to-end (`fxt-backtest run` exits 0, `summary.json` written)
- `n_trades > 0` AND `sharpe_ann` finite
- `mean_decision_latency_us < 100` on L40S
Smoke fail halts the workflow. Reasons recorded in
`deployability_verdict.json.smoke_status`.
### 3.5 Verdict emitter (`emit_deployability_verdict`)
Implementation in `crates/ml-backtesting/src/aggregate.rs`. Reads
per-cell `summary.json` files for the two anchor cells × 4 windows × the
pre-registered threshold, computes median Sharpe / max-dd / Sortino /
profit-factor across windows, applies tiered classification:
```
realistic.pass = (median_sharpe > 1.0) AND (median_max_dd_pct < 0.20)
AND (no NaN, no zero-trade windows)
stress.pass = same gates applied to stress anchor
if realistic.status != "ok": verdict = "Fail-degenerate"
elif realistic.pass && stress.pass: verdict = "Pass-robust"
elif realistic.pass: verdict = "Pass-nominal"
elif median_sharpe ∈ [0.8, 1.0]
OR median_max_dd_pct ∈ [0.20, 0.25]: verdict = "Fail-inconclusive"
else: verdict = "Fail"
```
Output `deployability_verdict.json` committed to repo as audit record.
### 3.6 Walk-forward windows
| Window | DBN file | Trading days | Purpose |
|---|---|---|---|
| W0 | 2025-Q1 | ~63 | Threshold pre-registration |
| W1 | 2025-Q2 | ~63 | Held-out validation |
| W2 | 2025-Q3 | ~63 | Held-out validation |
| W3 | 2025-Q4 | ~63 | Held-out validation |
| W4 | 2026-Q1 | ~63 | Held-out validation |
## 4. Testing & failure modes
### 4.1 Test inventory
| Test | Layer | Status | Gate for |
|---|---|---|---|
| `perception_forward_golden` (bit-equivalence vs golden bytes) | Trunk forward correctness | NEW (X0) | X1..X11 |
| Captured-vs-uncaptured forward bit-equivalence | Trunk captured graph | NEW (X11) | X11 |
| CheckpointV2 save/load round-trip | Serialization | NEW (X12) | X12 |
| `PerceptionTrainer.save_checkpoint` delegate test | Trainer wrapper | NEW (X13) | X13 |
| `AlphaTrainSummary` serde round-trip with `best_h6000_ckpt_path` | Summary schema | NEW (X14) | X14 |
| Existing ml-backtesting harness tests against fixture CheckpointV2 | Backtester loader | EXISTING (verifies on V2) | X15 |
| `max_drawdown_pct_against_fixture_curve` | Per-cell Summary math | NEW (X16) | X16 |
| 5 `classify_verdict` tests (one per tier) | Verdict logic | NEW (X17) | X17 |
| `emit_deployability_verdict_against_fixture_sweep` | Verdict end-to-end | NEW (X17) | X17 |
| `checkpoint_smoke` (GPU integration) | End-to-end inference | NEW (X18) | X18, runtime smoke gate |
| Fold-0 post-refactor smoke (recorded fold 0 numbers within tolerance) | Training trajectory preservation | RECORDED LOGS (the 3-fold A/B already ran; logs in memory) | Phase 1 → Phase 2 gate |
### 4.2 Failure modes
| Failure | Likelihood | Mitigation |
|---|---|---|
| Refactor breaks training silently (golden passes, real training diverges) | Medium — non-trivial kernels with subtle precision differences | Post-refactor fold-0 smoke (§2.3) catches before paying for production run |
| CheckpointV2 format drift between train/backtest binaries | Low | Both binaries pinned to same `<sha>` via Argo workflow + sweep YAML resolution |
| SSM-state leakage across windows | Low (Argo per-cell-pod fan-out gives fresh process per cell) | Naturally isolated by infrastructure |
| Decision-stride mismatch | Low | Both pinned `decision_stride: 1` in all YAMLs |
| Model never trades at chosen threshold | Medium | Smoke `n_trades > 0` check; pre-registration sweep covers wide threshold range |
| Window straddles contract roll | Low | Quarterly window boundaries align with quarterly contract cycles |
| Sharpe = NaN due to `std(pnl)=0` | Low | Verdict emitter rejects NaN with `Fail-degenerate` exit |
| Max-dd math bug | Medium (new code) | Fixture test with known curve (X16) |
| Captured-graph break by adding Mamba2 kernels | Medium — Mamba2 kernels were never inside a captured graph before | X11 commit specifically tests captured-vs-uncaptured equivalence; if it fails, isolate the offending kernel and fix host-branch leak per [[pearl_no_host_branches_in_captured_graph]] |
## 5. Phase 2 runtime sequence (operator runbook)
After Phase 1 lands (X0..X19 + fold-0 smoke gate passed):
```
1. git push origin ml-alpha-phase-a # capture commit as $TRAIN_SHA
2. ./scripts/argo-train.sh --commit $TRAIN_SHA --epochs 30 \
--cv-n-folds 1 --cv-train-window 4 --gpu-pool ci-training-l40s --watch
→ trunk_best_h6000.bin lands on feature-cache-pvc
3. ./scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_smoke.yaml \
--sha $TRAIN_SHA --watch
→ halt on failure; do not proceed
4. ./scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_threshold_tuning.yaml \
--sha $TRAIN_SHA --watch
→ write config/ml/v2_prod_thresholds.json; commit; push
5. ./scripts/argo-lob-sweep.sh --grid config/ml/sweep_v2_deployability.yaml \
--sha $TRAIN_SHA --sweep-tag v2-deployability --watch
6. fxt-backtest aggregate <sweep_root>
7. fxt-backtest verdict <sweep_root> --threshold <pre-registered> \
--windows W1,W2,W3,W4 --training-sha $TRAIN_SHA \
--spec-sha <this spec's commit> --out deployability_verdict.json
8. git add deployability_verdict.json; git commit; git push
9. Update project_ml_alpha_v2_ab_verdict memory with verdict close-out
```
## 6. References
- Superseded spec: `docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md` @ `07d5de504`
- Superseded plan: `docs/superpowers/plans/2026-05-19-ml-alpha-v2-deployability-validation.md` @ `4938ac2ec`
- Real-LOB integration spec: `docs/superpowers/specs/2026-05-18-real-lob-integration-design.md`
- Real-LOB integration plan (C1–C19 already on branch): `docs/superpowers/plans/2026-05-18-real-lob-integration.md`
- ml-alpha v2 multi-horizon spec: `docs/superpowers/specs/2026-05-18-ml-alpha-v2-multi-horizon-design.md`
- 3-fold A/B verdict @ 6b7920474: memory `project_ml_alpha_v2_ab_verdict`
- $35k starting capital decision: memory `project_ml_alpha_starting_capital`
- Phase 1d.4 cost-edge frontier: memory `pearl_phase1d4_backtest_cost_edge_frontier`
- Single-window OOS warning: memory `pearl_single_window_oos_is_not_oos`
- Smoke-before-structural-priors: memory `feedback_smoke_validation_before_structural_priors`
- Wire-everything-up: memory `feedback_wire_everything_up`
- No host branches in captured graph: memory `pearl_no_host_branches_in_captured_graph`
- No atomicAdd in trunk kernels: memory `feedback_no_atomicadd`
- Temporal encoder must train: memory `pearl_temporal_encoder_must_train`