From 07d5de50486f190ddf90b24c61dd9854b7366499 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 19 May 2026 00:32:30 +0200 Subject: [PATCH] =?UTF-8?q?spec(ml-alpha):=20v2=20deployability=20?= =?UTF-8?q?=E2=80=94=20stress=20anchor=20+=20max-dd=20gate=20+=20diagnosti?= =?UTF-8?q?cs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User revisions to design spec from this session: 1. §2.1 — split anchor into realistic (200 ms, 1 tick) and stress (400 ms, 1.5 tick). Realistic remains the hard verdict gate; stress grades the Pass into Pass-robust vs Pass-nominal. 2. §2.2 — expand metrics from Sharpe-only to four: annualized daily Sharpe and max-drawdown are hard gates (median across windows > 1.0 and < 20% respectively); Sortino and profit factor are diagnostics. Per-window summary.json schema extended. 3. §2.6 — verdict emitter rewrites to two-anchor logic, tiered output: Pass-robust / Pass-nominal / Fail-inconclusive / Fail / Fail-degenerate. 4. §3.3 — added max-dd computation and zero-trade-window failure modes. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...lpha-v2-deployability-validation-design.md | 137 ++++++++++++------ 1 file changed, 95 insertions(+), 42 deletions(-) diff --git a/docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md b/docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md index 53559fff6..9d5ac18d4 100644 --- a/docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md +++ b/docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md @@ -28,14 +28,21 @@ falsifiable deployability verdict. **Pre-registered falsifiable claim:** > The v2 ml-alpha model (commit `6b7920474`, production-checkpoint variant) is -> deployable iff the **median Sharpe across 4 walk-forward held-out quarters, -> evaluated at the realistic deployment anchor (200 ms RTT, 1-tick all-in cost, -> pre-registered threshold (h6000 only)), is strictly greater than 1.0.** +> deployable iff, across 4 walk-forward held-out quarters at the realistic +> deployment anchor (200 ms RTT, 1-tick all-in cost, pre-registered threshold +> (h6000 only)), **median Sharpe > 1.0 AND median max-drawdown < 20%**. +> An additional stress anchor (400 ms RTT, 1.5-tick cost) on the same windows +> grades the Pass into **Pass-robust** (both anchors satisfy both gates) vs +> **Pass-nominal** (only realistic anchor satisfies both gates). -- **Pass** → candidate for paper-trade plan (separate spec). -- **Fail** → shelf decision for this architecture variant. Verdict committed. -- **Inconclusive (median ∈ [0.8, 1.0])** → counts as Fail. Re-evaluating the - criterion post-hoc is not permitted. +- **Pass-robust** → candidate for paper-trade plan (separate spec). Signal has + real edge to spare even in adverse routing/spread regimes. +- **Pass-nominal** → candidate for paper-trade plan with explicit operational + caveat that adverse routing/spread regimes break it. Recorded in audit. +- **Fail-inconclusive** (Sharpe ∈ [0.8, 1.0] OR max-dd ∈ [20%, 25%] at the + realistic anchor) → counts as Fail. Re-evaluating post-hoc is not permitted. +- **Fail** otherwise → shelf decision for this architecture variant. Verdict + committed. **Out of scope (deferred to separate specs):** @@ -55,7 +62,7 @@ falsifiable deployability verdict. | `config/ml/sweep_v2_deployability.yaml` *(new)* | Grid YAML for `fxt-backtest sweep` — see §2.5. | ~30 | | `config/ml/sweep_v2_smoke.yaml` *(new)* | Single-cell smoke YAML — see §3.1. | ~15 | | `config/ml/sweep_v2_threshold_tuning.yaml` *(new)* | Threshold pre-registration sweep YAML — see §2.3. | ~20 | -| `crates/ml-backtesting/src/aggregate.rs` | Add `emit_deployability_verdict(...)` function that reads the per-cell `summary.json` files, looks up the anchor cell, computes median Sharpe across windows, and writes `deployability_verdict.json`. | ~50 | +| `crates/ml-backtesting/src/aggregate.rs` | Add `emit_deployability_verdict(...)` function that reads the per-cell `summary.json` files, looks up both anchor cells (realistic + stress), computes median across windows for Sharpe / max-drawdown / Sortino / profit-factor, applies the tiered pass logic, and writes `deployability_verdict.json`. Also extend per-cell `summary.json` writer to emit Sortino, max_dd, profit_factor alongside the existing Sharpe field. | ~80 | | `crates/ml-backtesting/tests/checkpoint_smoke.rs` *(new)* | `#[ignore]`d GPU integration test — load a trained checkpoint, drive the harness end-to-end against a tiny fixture window, assert smoke pass criteria. | ~80 | **Read-only consumers (exercised on a real trained model for the first time):** @@ -97,32 +104,51 @@ quarter are handled by the existing loader chronology — no new mitigation. ## 2. Falsifiable criterion, anchor point, sweep configuration -### 2.1 Realistic-deployment anchor +### 2.1 Deployment anchors (realistic + stress) -| Axis | Value | Source | +Two anchor cells on the swept surface drive the verdict. The grid (§2.5) +already contains both as ordinary cells — "anchor" is a labelling, not a +separate run. + +| Anchor | Latency | Cost (round-trip) | Threshold | Role | +|---|---|---|---|---| +| **Realistic** | 200 ms | 1 tick ($12.50) | Pre-registered (§2.3) | **Hard gate.** Pass requires both Sharpe and max-dd gates satisfied here. | +| **Stress** | 400 ms | 1.5 tick ($18.75) | Pre-registered (§2.3) | **Robustness grade.** Passing both gates here promotes Pass → Pass-robust. Failing here while passing realistic = Pass-nominal. | + +Sources: + +- 200 ms RTT: Scaleway PAR ↔ CME Aurora ~80–100 ms one-way + IBKR FIX overhead. +- 400 ms RTT: doubled-latency excursion plausible during routing degradation, + network event, or wider-area failover. +- 1 tick: spread crossing (≥1 tick at IBKR retail) + $0.85 commission/side + ≈ 1.07 tick. +- 1.5 tick: wider spread regime (low-liquidity sessions, around news, contract + roll proximity). + +Anchors are fixed in the spec. They are swept only for the descriptive surface +so we can see how the metrics behave *around* the anchors. + +### 2.2 Metrics per window + +Each (cell, window) tuple produces four risk-adjusted metrics, computed from +the per-window daily P&L series and trade ledger. Two are hard gates, two are +diagnostics. + +| Metric | Definition | Role | |---|---|---| -| Round-trip latency | **200 ms** | Scaleway PAR ↔ CME Aurora ~80–100 ms one-way + IBKR FIX overhead, conservative | -| All-in trade cost | **1 ES tick = $12.50** per round-trip | Spread crossing (≥1 tick retail) + $0.85 commission/side ≈ 1.07 tick | -| Threshold | Pre-registered per-horizon, see §2.3 | Selected on W0, frozen before W1–W4 evaluation | +| **Annualized daily Sharpe** | `mean(daily_pnl) / std(daily_pnl) × √252` | **Hard gate.** Pass requires `median_across_windows > 1.0` at the relevant anchor. | +| **Max drawdown** | Peak-to-trough equity decline as a fraction of starting capital, per window | **Hard gate.** Pass requires `median_across_windows < 20%` at the relevant anchor. Captures capital-deployability — high-Sharpe strategies with deep drawdowns are unusable because the operator can't hold through them. | +| **Sortino ratio** | `mean(daily_pnl) / downside_deviation × √252`, where `downside_deviation = std(min(daily_pnl, 0))` | **Diagnostic.** Sharpe-vs-Sortino disagreement diagnoses asymmetric P&L. Sortino ≫ Sharpe ⇒ right-tail-heavy (good); Sortino ≪ Sharpe ⇒ left-tail-heavy (bad). Recorded, not gated. | +| **Profit factor** | `Σ winning_trades / |Σ losing_trades|` over the window's trade ledger | **Diagnostic.** PF > 1.2 in the median window indicates a real edge. PF ≈ 1.0 alongside positive Sharpe is the variance-gaming signature. Recorded, not gated. | -The anchor is fixed in the spec. It is not a swept axis for the verdict; it is -swept only for the descriptive surface so we can see how Sharpe behaves *around* -the anchor. - -### 2.2 Sharpe definition - -**Annualized daily P&L Sharpe**, per window: - -``` -Sharpe_w = mean(daily_pnl_w) / std(daily_pnl_w) * sqrt(252) -``` - -Cross-window aggregate: `Sharpe_verdict = median(Sharpe_W1, Sharpe_W2, Sharpe_W3, Sharpe_W4)`. +Cross-window aggregates use **median across the 4 walk-forward windows** for +all four metrics. Median is preferred over mean to keep the verdict robust to +single-quarter regime shocks. If any window's `std(daily_pnl)` is zero (model never trades), that window's -Sharpe is `NaN` and the verdict emitter rejects the run as `Fail-degenerate` -with a distinct exit code rather than silently propagating `NaN` through the -median. +Sharpe and Sortino are `NaN`. The verdict emitter rejects the run as +`Fail-degenerate` with a distinct exit code rather than propagating `NaN` +through the medians. ### 2.3 Threshold pre-registration (two-pass discipline) @@ -187,23 +213,48 @@ is ~35 minutes. Total L40S time: ~5 hours. ### 2.6 Verdict emitter (`emit_deployability_verdict`) -Pseudocode (~50 lines in `crates/ml-backtesting/src/aggregate.rs`): +Pseudocode (~80 lines in `crates/ml-backtesting/src/aggregate.rs`): ``` 1. Read v2_prod_thresholds.json → threshold_anchor (scalar; h6000 only) 2. For each cell file in sweep_dir/cell_*/summary.json: - Parse (cost_tick, latency_ms, threshold, window) + Sharpe + trades_count -3. Filter to cells where (cost_tick == 1.0) AND (latency_ms == 200) - AND (threshold == threshold_anchor) -4. Group by window → expect exactly 4 cells (W1, W2, W3, W4) -5. If any sharpe is NaN → emit verdict "Fail-degenerate" -6. median_sharpe = median([sharpe_W1, sharpe_W2, sharpe_W3, sharpe_W4]) -7. If median_sharpe > 1.0 → "Pass" - If median_sharpe ∈ [0.8, 1.0] → "Fail-inconclusive" - Else → "Fail" -8. Write deployability_verdict.json with all four window Sharpes, - the median, the verdict string, the threshold vector, the anchor point, - and the production training SHA. Commit to repo as the audit record. + Parse (cost_tick, latency_ms, threshold, window) + + Sharpe + Sortino + max_dd + profit_factor + trades_count +3. ANCHORS = [ + ("realistic", cost_tick=1.0, latency_ms=200), + ("stress", cost_tick=1.5, latency_ms=400), + ] +4. For each anchor in ANCHORS: + Filter to cells matching anchor + (threshold == threshold_anchor) + Group by window → expect exactly 4 cells (W1, W2, W3, W4) + If any Sharpe or Sortino is NaN OR trades_count == 0 in any window + → record anchor.status = "Fail-degenerate" + → continue + anchor.median_sharpe = median(sharpe_per_window) + anchor.median_max_dd = median(max_dd_per_window) + anchor.median_sortino = median(sortino_per_window) + anchor.median_pf = median(pf_per_window) + anchor.gate_sharpe = anchor.median_sharpe > 1.0 + anchor.gate_max_dd = anchor.median_max_dd < 0.20 + anchor.pass = anchor.gate_sharpe AND anchor.gate_max_dd +5. realistic = ANCHORS["realistic"] + stress = ANCHORS["stress"] + If realistic.status == "Fail-degenerate" → verdict = "Fail-degenerate" + Else if realistic.pass AND stress.pass → verdict = "Pass-robust" + Else if realistic.pass → verdict = "Pass-nominal" + Else if realistic.median_sharpe ∈ [0.8, 1.0] + OR realistic.median_max_dd ∈ [0.20, 0.25] + → verdict = "Fail-inconclusive" + Else → verdict = "Fail" +6. Write deployability_verdict.json containing: + - verdict string (one of: Pass-robust, Pass-nominal, Fail-inconclusive, + Fail, Fail-degenerate) + - per-anchor: median + per-window arrays for all four metrics + (Sharpe, Sortino, max_dd, profit_factor) + - per-anchor: gate_sharpe, gate_max_dd booleans + - threshold value, anchor specs, production training SHA, spec SHA, + timestamp +7. Commit deployability_verdict.json to repo as audit record. ``` ## 3. Smoke gate, testing, failure modes @@ -265,6 +316,8 @@ for the first time against a real trained model. | Window straddles contract roll | Low | Quarterly window boundaries align with quarterly contract cycles; roll periods are interior to each quarter, loader handles chronology | | Databento MBP-10 schema drift | Very low | Loader does in-band schema check, errors hard | | Sharpe = NaN due to `std(pnl)=0` | Low | Verdict emitter rejects NaN with distinct `Fail-degenerate` exit | +| Max-drawdown computation buggy (peak tracking off-by-one, intra-day vs end-of-day basis) | Medium — new code path | Verdict round-trip test (§3.2) covers the field's presence + round-trip; a separate unit test in `aggregate.rs` covers max-dd math against a known fixture P&L curve | +| Sortino-Sharpe agree on a degenerate window (both flat, both ~0) but the model is producing real trades elsewhere | Low | Verdict emitter requires `trades_count > 0` per window before counting that window in the median; "Fail-degenerate" exit on zero-trade windows | ## 4. Implementation sequencing (preview for plan-writing skill)