gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spec(ml-alpha): v2 deployability — stress anchor + max-dd gate + diagnostics

Browse Source 
User revisions to design spec from this session:

1. §2.1 — split anchor into realistic (200 ms, 1 tick) and stress (400 ms,
   1.5 tick). Realistic remains the hard verdict gate; stress grades the Pass
   into Pass-robust vs Pass-nominal.

2. §2.2 — expand metrics from Sharpe-only to four: annualized daily Sharpe
   and max-drawdown are hard gates (median across windows > 1.0 and < 20%
   respectively); Sortino and profit factor are diagnostics. Per-window
   summary.json schema extended.

3. §2.6 — verdict emitter rewrites to two-anchor logic, tiered output:
   Pass-robust / Pass-nominal / Fail-inconclusive / Fail / Fail-degenerate.

4. §3.3 — added max-dd computation and zero-trade-window failure modes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-19 00:32:30 +02:00
parent 0809390cd5
commit 07d5de5048
1 changed files with 95 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

137
docs/superpowers/specs/2026-05-19-ml-alpha-v2-deployability-validation-design.md
View File

@@ -28,14 +28,21 @@ falsifiable deployability verdict.
**Pre-registered falsifiable claim:**
> The v2 ml-alpha model (commit `6b7920474`, production-checkpoint variant) is
> deployable iff the **median Sharpe across 4 walk-forward held-out quarters,
> evaluated at the realistic deployment anchor (200 ms RTT, 1-tick all-in cost,
> pre-registered threshold (h6000 only)), is strictly greater than 1.0.**
> deployable iff, across 4 walk-forward held-out quarters at the realistic
> deployment anchor (200 ms RTT, 1-tick all-in cost, pre-registered threshold
> (h6000 only)), **median Sharpe > 1.0 AND median max-drawdown < 20%**.
> An additional stress anchor (400 ms RTT, 1.5-tick cost) on the same windows
> grades the Pass into **Pass-robust** (both anchors satisfy both gates) vs
> **Pass-nominal** (only realistic anchor satisfies both gates).
- **Pass** → candidate for paper-trade plan (separate spec).
- **Fail** → shelf decision for this architecture variant. Verdict committed.
- **Inconclusive (median ∈ [0.8, 1.0])** → counts as Fail. Re-evaluating the
criterion post-hoc is not permitted.
- **Pass-robust** → candidate for paper-trade plan (separate spec). Signal has
real edge to spare even in adverse routing/spread regimes.
- **Pass-nominal** → candidate for paper-trade plan with explicit operational
caveat that adverse routing/spread regimes break it. Recorded in audit.
- **Fail-inconclusive** (Sharpe ∈ [0.8, 1.0] OR max-dd ∈ [20%, 25%] at the
realistic anchor) → counts as Fail. Re-evaluating post-hoc is not permitted.
- **Fail** otherwise → shelf decision for this architecture variant. Verdict
committed.
**Out of scope (deferred to separate specs):**
@@ -55,7 +62,7 @@ falsifiable deployability verdict.
| `config/ml/sweep_v2_deployability.yaml` *(new)* | Grid YAML for `fxt-backtest sweep` — see §2.5. | ~30 |
| `config/ml/sweep_v2_smoke.yaml` *(new)* | Single-cell smoke YAML — see §3.1. | ~15 |
| `config/ml/sweep_v2_threshold_tuning.yaml` *(new)* | Threshold pre-registration sweep YAML — see §2.3. | ~20 |
| `crates/ml-backtesting/src/aggregate.rs` | Add `emit_deployability_verdict(...)` function that reads the per-cell `summary.json` files, looks up the anchor cell, computes median Sharpe across windows, and writes `deployability_verdict.json`. | ~50 |
| `crates/ml-backtesting/src/aggregate.rs` | Add `emit_deployability_verdict(...)` function that reads the per-cell `summary.json` files, looks up both anchor cells (realistic + stress), computes median across windows for Sharpe / max-drawdown / Sortino / profit-factor, applies the tiered pass logic, and writes `deployability_verdict.json`. Also extend per-cell `summary.json` writer to emit Sortino, max_dd, profit_factor alongside the existing Sharpe field. | ~80 |
| `crates/ml-backtesting/tests/checkpoint_smoke.rs` *(new)* | `#[ignore]`d GPU integration test — load a trained checkpoint, drive the harness end-to-end against a tiny fixture window, assert smoke pass criteria. | ~80 |
**Read-only consumers (exercised on a real trained model for the first time):**
@@ -97,32 +104,51 @@ quarter are handled by the existing loader chronology — no new mitigation.
## 2. Falsifiable criterion, anchor point, sweep configuration
### 2.1 Realistic-deployment anchor
### 2.1 Deployment anchors (realistic + stress)
| Axis | Value | Source |
Two anchor cells on the swept surface drive the verdict. The grid (§2.5)
already contains both as ordinary cells — "anchor" is a labelling, not a
separate run.
| Anchor | Latency | Cost (round-trip) | Threshold | Role |
|---|---|---|---|---|
| **Realistic** | 200 ms | 1 tick ($12.50) | Pre-registered (§2.3) | **Hard gate.** Pass requires both Sharpe and max-dd gates satisfied here. |
| **Stress** | 400 ms | 1.5 tick ($18.75) | Pre-registered (§2.3) | **Robustness grade.** Passing both gates here promotes Pass → Pass-robust. Failing here while passing realistic = Pass-nominal. |
Sources:
- 200 ms RTT: Scaleway PAR ↔ CME Aurora ~80–100 ms one-way + IBKR FIX overhead.
- 400 ms RTT: doubled-latency excursion plausible during routing degradation,
network event, or wider-area failover.
- 1 tick: spread crossing (≥1 tick at IBKR retail) + $0.85 commission/side
≈ 1.07 tick.
- 1.5 tick: wider spread regime (low-liquidity sessions, around news, contract
roll proximity).
Anchors are fixed in the spec. They are swept only for the descriptive surface
so we can see how the metrics behave *around* the anchors.
### 2.2 Metrics per window
Each (cell, window) tuple produces four risk-adjusted metrics, computed from
the per-window daily P&L series and trade ledger. Two are hard gates, two are
diagnostics.
| Metric | Definition | Role |
|---|---|---|
| Round-trip latency | **200 ms** | Scaleway PAR ↔ CME Aurora ~80–100 ms one-way + IBKR FIX overhead, conservative |
| All-in trade cost | **1 ES tick = $12.50** per round-trip | Spread crossing (≥1 tick retail) + $0.85 commission/side ≈ 1.07 tick |
| Threshold | Pre-registered per-horizon, see §2.3 | Selected on W0, frozen before W1–W4 evaluation |
| **Annualized daily Sharpe** | `mean(daily_pnl) / std(daily_pnl) × √252` | **Hard gate.** Pass requires `median_across_windows > 1.0` at the relevant anchor. |
| **Max drawdown** | Peak-to-trough equity decline as a fraction of starting capital, per window | **Hard gate.** Pass requires `median_across_windows < 20%` at the relevant anchor. Captures capital-deployability — high-Sharpe strategies with deep drawdowns are unusable because the operator can't hold through them. |
| **Sortino ratio** | `mean(daily_pnl) / downside_deviation × √252`, where `downside_deviation = std(min(daily_pnl, 0))` | **Diagnostic.** Sharpe-vs-Sortino disagreement diagnoses asymmetric P&L. Sortino ≫ Sharpe ⇒ right-tail-heavy (good); Sortino ≪ Sharpe ⇒ left-tail-heavy (bad). Recorded, not gated. |
| **Profit factor** | `Σ winning_trades / |Σ losing_trades|` over the window's trade ledger | **Diagnostic.** PF > 1.2 in the median window indicates a real edge. PF ≈ 1.0 alongside positive Sharpe is the variance-gaming signature. Recorded, not gated. |
The anchor is fixed in the spec. It is not a swept axis for the verdict; it is
swept only for the descriptive surface so we can see how Sharpe behaves *around*
the anchor.
### 2.2 Sharpe definition
**Annualized daily P&L Sharpe**, per window:
```
Sharpe_w = mean(daily_pnl_w) / std(daily_pnl_w) * sqrt(252)
```
Cross-window aggregate: `Sharpe_verdict = median(Sharpe_W1, Sharpe_W2, Sharpe_W3, Sharpe_W4)`.
Cross-window aggregates use **median across the 4 walk-forward windows** for
all four metrics. Median is preferred over mean to keep the verdict robust to
single-quarter regime shocks.
If any window's `std(daily_pnl)` is zero (model never trades), that window's
Sharpe is `NaN` and the verdict emitter rejects the run as `Fail-degenerate`
with a distinct exit code rather than silently propagating `NaN` through the
median.
Sharpe and Sortino are `NaN`. The verdict emitter rejects the run as
`Fail-degenerate` with a distinct exit code rather than propagating `NaN`
through the medians.
### 2.3 Threshold pre-registration (two-pass discipline)
@@ -187,23 +213,48 @@ is ~35 minutes. Total L40S time: ~5 hours.
### 2.6 Verdict emitter (`emit_deployability_verdict`)
Pseudocode (~50 lines in `crates/ml-backtesting/src/aggregate.rs`):
Pseudocode (~80 lines in `crates/ml-backtesting/src/aggregate.rs`):
```
1. Read v2_prod_thresholds.json → threshold_anchor (scalar; h6000 only)
2. For each cell file in sweep_dir/cell_*/summary.json:
Parse (cost_tick, latency_ms, threshold, window) + Sharpe + trades_count
3. Filter to cells where (cost_tick == 1.0) AND (latency_ms == 200)
AND (threshold == threshold_anchor)
4. Group by window → expect exactly 4 cells (W1, W2, W3, W4)
5. If any sharpe is NaN → emit verdict "Fail-degenerate"
6. median_sharpe = median([sharpe_W1, sharpe_W2, sharpe_W3, sharpe_W4])
7. If median_sharpe > 1.0 → "Pass"
If median_sharpe ∈ [0.8, 1.0] → "Fail-inconclusive"
Else → "Fail"
8. Write deployability_verdict.json with all four window Sharpes,
the median, the verdict string, the threshold vector, the anchor point,
and the production training SHA. Commit to repo as the audit record.
Parse (cost_tick, latency_ms, threshold, window)
+ Sharpe + Sortino + max_dd + profit_factor + trades_count
3. ANCHORS = [
("realistic", cost_tick=1.0, latency_ms=200),
("stress", cost_tick=1.5, latency_ms=400),
]
4. For each anchor in ANCHORS:
Filter to cells matching anchor + (threshold == threshold_anchor)
Group by window → expect exactly 4 cells (W1, W2, W3, W4)
If any Sharpe or Sortino is NaN OR trades_count == 0 in any window
→ record anchor.status = "Fail-degenerate"
→ continue
anchor.median_sharpe = median(sharpe_per_window)
anchor.median_max_dd = median(max_dd_per_window)
anchor.median_sortino = median(sortino_per_window)
anchor.median_pf = median(pf_per_window)
anchor.gate_sharpe = anchor.median_sharpe > 1.0
anchor.gate_max_dd = anchor.median_max_dd < 0.20
anchor.pass = anchor.gate_sharpe AND anchor.gate_max_dd
5. realistic = ANCHORS["realistic"]
stress = ANCHORS["stress"]
If realistic.status == "Fail-degenerate" → verdict = "Fail-degenerate"
Else if realistic.pass AND stress.pass → verdict = "Pass-robust"
Else if realistic.pass → verdict = "Pass-nominal"
Else if realistic.median_sharpe ∈ [0.8, 1.0]
OR realistic.median_max_dd ∈ [0.20, 0.25]
→ verdict = "Fail-inconclusive"
Else → verdict = "Fail"
6. Write deployability_verdict.json containing:
- verdict string (one of: Pass-robust, Pass-nominal, Fail-inconclusive,
Fail, Fail-degenerate)
- per-anchor: median + per-window arrays for all four metrics
(Sharpe, Sortino, max_dd, profit_factor)
- per-anchor: gate_sharpe, gate_max_dd booleans
- threshold value, anchor specs, production training SHA, spec SHA,
timestamp
7. Commit deployability_verdict.json to repo as audit record.
```
## 3. Smoke gate, testing, failure modes
@@ -265,6 +316,8 @@ for the first time against a real trained model.
| Window straddles contract roll | Low | Quarterly window boundaries align with quarterly contract cycles; roll periods are interior to each quarter, loader handles chronology |
| Databento MBP-10 schema drift | Very low | Loader does in-band schema check, errors hard |
| Sharpe = NaN due to `std(pnl)=0` | Low | Verdict emitter rejects NaN with distinct `Fail-degenerate` exit |
| Max-drawdown computation buggy (peak tracking off-by-one, intra-day vs end-of-day basis) | Medium — new code path | Verdict round-trip test (§3.2) covers the field's presence + round-trip; a separate unit test in `aggregate.rs` covers max-dd math against a known fixture P&L curve |
| Sortino-Sharpe agree on a degenerate window (both flat, both ~0) but the model is producing real trades elsewhere | Low | Verdict emitter requires `trades_count > 0` per window before counting that window in the median; "Fail-degenerate" exit on zero-trade windows |
## 4. Implementation sequencing (preview for plan-writing skill)