9d4fda36ab
Sweep YAML now supports the batched flow per spec §3.3 + Task 6:
- SweepBase.sim_variants: Vec<SimVariant> — list of (cost, latency,
threshold, ...) variants. When non-empty, each cell runs ONE harness
at n_parallel=variants.len() with BatchedSimConfig::from_grid instead
of the legacy one-harness-per-cell fan-out.
- SweepBase.data_template: Option<String> — when set with `{window}`
placeholder, each cell's `window` field interpolates the per-cell
data path. Replaces single scalar `data` for the windowed flow.
- SweepCell.window: Option<String> — window identifier (e.g., "2025-Q2").
- SimVariant: threshold + cost_per_lot_per_side required (the spec's
primary axes); other fields optional overrides on top of SweepBase
scalars.
New runner pieces:
- BatchedSimConfig::from_grid(&[ResolvedSimVariant]) in
crates/ml-backtesting/src/sim/batched_config.rs.
- ResolvedSimVariant — per-variant fully-resolved sim params.
- resolve_sim_variants(&SweepBase) in main.rs — layers per-variant
overrides over base scalars.
- run_batched_cell() in main.rs — builds the harness with
sim_config_override + variant_names plumbed through. Writes per-
backtest artifacts to sim_<variant_name>/ subdirs (spec §3.3).
Harness side:
- BacktestHarnessConfig gains variant_names + sim_config_override
Option fields. When sim_config_override is Some, harness uses that
directly instead of building from_uniform off scalar cfg. When
variant_names is Some, write_artifacts uses sim_<name>/ instead of
cell_NNNN/ subdirs. Both None preserve legacy single-cell behaviour
(smoke, fixtures unchanged).
YAML configs:
- config/ml/sweep_threshold_tuning.yaml: 1 cell (W0) × 8 sim_variants
(p60-p95 in 5pt steps) with cost=0.125 (1-tick anchor). Threshold
pre-registration pass.
- config/ml/sweep_deployability.yaml: 4 cells (W1-W4) × 140 variants
each (7 costs × 4 latencies × 5 thresholds). Generated by
scripts/generate_sweep_variants.py — placeholder threshold values
(p60-p95) until threshold-tuning publishes calibrated absolutes to
config/ml/v2_prod_thresholds.json.
Deferred to P7 (operational glue):
- argo-lob-sweep.sh adaptation for the batched flow (cells = windows,
not sim-variants; one Argo task per window invokes `fxt-backtest sweep`
end-to-end inside the pod rather than `fxt-backtest run`).
Regression: all 7 existing CUDA tests pass through the new harness
construction path (sim_config_override = None → from_uniform fallback).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
76 lines
3.4 KiB
Python
Executable File
76 lines
3.4 KiB
Python
Executable File
#!/usr/bin/env python3
|
||
"""P6 helper: emit the 140 sim_variants for the deployability sweep.
|
||
|
||
Cross-product: 7 costs × 4 latencies × 5 thresholds = 140 variants.
|
||
|
||
Threshold values are placeholders — the actual values come from
|
||
config/ml/v2_prod_thresholds.json after the threshold-tuning step.
|
||
Re-run this script post-tuning to refresh sweep_deployability.yaml with
|
||
calibrated thresholds.
|
||
|
||
Usage:
|
||
scripts/generate_sweep_variants.py > config/ml/sweep_deployability.yaml
|
||
"""
|
||
from __future__ import annotations
|
||
import json, sys
|
||
from pathlib import Path
|
||
|
||
# 7 costs (price-units / lot / side). 0.125 = 1 tick (ES), 0.0625 = 0.5 tick (best-case),
|
||
# stepping through realistic retail (0.25, 0.375 = 2-3 ticks), and worst-case (1.0 = 8 ticks).
|
||
COSTS = [0.0625, 0.125, 0.1875, 0.25, 0.375, 0.5, 1.0]
|
||
|
||
# 4 latencies (ns). 100M = 100ms (best), 200M = realistic anchor, 300M = degraded,
|
||
# 400M = stress anchor.
|
||
LATENCIES_NS = [100_000_000, 200_000_000, 300_000_000, 400_000_000]
|
||
|
||
# 5 thresholds — REPLACE THESE WITH THE OUTPUT OF THE THRESHOLD-TUNING PASS.
|
||
# Read from config/ml/v2_prod_thresholds.json if it exists; otherwise use
|
||
# the canonical p60-p95 placeholder (5 steps spanning the range).
|
||
THRESH_JSON = Path(__file__).resolve().parent.parent / "config" / "ml" / "v2_prod_thresholds.json"
|
||
if THRESH_JSON.exists():
|
||
blob = json.loads(THRESH_JSON.read_text())
|
||
THRESHOLDS = blob["thresholds_absolute"] # e.g. {"p60": 0.42, "p70": 0.55, ...}
|
||
THRESHOLD_NAMES = list(THRESHOLDS.keys())
|
||
THRESHOLD_VALUES = list(THRESHOLDS.values())
|
||
assert len(THRESHOLD_VALUES) == 5, f"expected 5 thresholds; got {len(THRESHOLD_VALUES)}"
|
||
else:
|
||
print("# WARNING: v2_prod_thresholds.json missing — using p60-p95 placeholder thresholds",
|
||
file=sys.stderr)
|
||
THRESHOLD_NAMES = ["p60", "p70", "p80", "p90", "p95"]
|
||
THRESHOLD_VALUES = [0.40, 0.55, 0.70, 0.85, 0.95]
|
||
|
||
assert len(COSTS) == 7
|
||
assert len(LATENCIES_NS) == 4
|
||
|
||
print("# P6 deployability sweep — 4 windows × 140 sim_variants (7 cost × 4 latency × 5 threshold).")
|
||
print("# Generated by scripts/generate_sweep_variants.py — do NOT hand-edit.")
|
||
print("# Re-run after threshold-tuning to refresh threshold absolute values.")
|
||
print()
|
||
print("base:")
|
||
print(" data_template: /mnt/training-data/futures-baseline-mbp10/ES.FUT/ES.FUT_{window}.dbn.zst")
|
||
print(" predecoded_dir: /feature-cache/predecoded")
|
||
print(" n_parallel: 1 # batched flow overrides with variants.len()=140")
|
||
print(" decision_stride: 4")
|
||
print(" latency_ns: 200000000 # default (overridden per variant)")
|
||
print(" target_annual_vol_units: 50.0")
|
||
print(" annualisation_factor: 825.0")
|
||
print(" max_lots: 5")
|
||
print(" max_events: 0")
|
||
print(" seed: 12648430")
|
||
print(" # __SHA__ replaced by submitter with post-trunk-grows training short-SHA.")
|
||
print(" checkpoint: /feature-cache/alpha-perception-runs/__SHA__/trunk_best_h6000.bin")
|
||
print(" sim_variants:")
|
||
|
||
for ci, cost in enumerate(COSTS):
|
||
for li, latency in enumerate(LATENCIES_NS):
|
||
for ti, (tname, tval) in enumerate(zip(THRESHOLD_NAMES, THRESHOLD_VALUES)):
|
||
name = f"c{ci}_l{li}_{tname}"
|
||
print(f" - {{ name: {name}, threshold: {tval}, cost_per_lot_per_side: {cost}, latency_ns: {latency} }}")
|
||
|
||
print()
|
||
print("cells:")
|
||
print(" - { name: W1, window: 2025-Q2 }")
|
||
print(" - { name: W2, window: 2025-Q3 }")
|
||
print(" - { name: W3, window: 2025-Q4 }")
|
||
print(" - { name: W4, window: 2026-Q1 }")
|