0d373da490
Creates scripts/validation/ with per-tier exit checks consuming the
aggregate JSON from scripts/aggregate-multi-seed-metrics.py (P5T1B):
check_tier1.py — convergence (std/mean ≤ 0.15 on val_sharpe /
avg_q_value / train_loss; avg_q_value max ≤ 500 fold-1 explosion
guard; placeholders for Q-saturation + hot-path-DtoH per spec).
check_tier2.py — behavioural (val_trades_per_bar ≥ 0.005,
val_active_frac > 0.2, dir argmax entropy > 0.8·log4 with
val_dir_entropy primary + val_dir_dist_* fallback).
check_tier3.py — profitability (val_sharpe_annualised > 1.0 with
val_sharpe per-bar fallback, val_win_rate ≥ 0.52 gated on
>500 trades, val_profit_factor mean ≥ 1.1 AND cross-seed std < 0.3).
check_all_tiers.py — subprocess wrapper, exits 0 only if all pass.
Stdlib-only (statistics / argparse / json / subprocess) — no new deps.
Defensive missing-metric handling: each check FAILs with an explanatory
message when its required aggregate key is absent rather than silently
passing, so missing HEALTH_DIAG metrics are surfaced loudly.
Test harness scripts/validation/tests/test_tier_checks.sh exercises
good + bad fixtures across all four scripts and against the wrapper.
Audit row added to docs/dqn-wire-up-audit.md documenting the suite +
the deferred metrics list (val_trades_per_bar, val_active_frac,
val_dir_entropy/_dist_*, val_sharpe_annualised, val_win_rate,
val_profit_factor, val_trade_count) that HEALTH_DIAG must emit before
tiers 2/3 can ever PASS on real data — tracked for Plan 5 Task 5
pre-flight wire-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
73 lines
1.9 KiB
Python
Executable File
73 lines
1.9 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""Top-level wrapper: run check_tier{1,2,3}.py in sequence (Plan 5 Task 4).
|
|
|
|
Subprocess-each strategy chosen over in-process import so that:
|
|
- per-tier checks remain runnable standalone (single source of truth);
|
|
- per-tier exit codes propagate naturally;
|
|
- tier scripts can evolve independently without touching this wrapper.
|
|
|
|
Exits 0 only if all three tier checks PASS. Each tier's own PASS/FAIL
|
|
lines stream through to stdout so the top-level summary is auditable.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
|
|
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
TIER_SCRIPTS = (
|
|
"check_tier1.py",
|
|
"check_tier2.py",
|
|
"check_tier3.py",
|
|
)
|
|
|
|
|
|
def main():
|
|
ap = argparse.ArgumentParser(
|
|
description="Run all three tiered exit checks (Plan 5 Task 4).",
|
|
)
|
|
ap.add_argument(
|
|
"metrics_json",
|
|
help="Aggregated metrics JSON (output of aggregate-multi-seed-metrics.py).",
|
|
)
|
|
ap.add_argument(
|
|
"--warmup-end",
|
|
type=int,
|
|
default=15,
|
|
help="Epoch at which warmup ends; forwarded to each tier check.",
|
|
)
|
|
args = ap.parse_args()
|
|
|
|
overall_ok = True
|
|
for script in TIER_SCRIPTS:
|
|
path = os.path.join(HERE, script)
|
|
print(f"=== {script} ===")
|
|
result = subprocess.run(
|
|
[
|
|
sys.executable,
|
|
path,
|
|
args.metrics_json,
|
|
"--warmup-end",
|
|
str(args.warmup_end),
|
|
],
|
|
check=False,
|
|
)
|
|
if result.returncode != 0:
|
|
overall_ok = False
|
|
print(f"--- {script}: FAIL (exit {result.returncode}) ---")
|
|
else:
|
|
print(f"--- {script}: PASS ---")
|
|
|
|
if overall_ok:
|
|
print("ALL TIERS PASS")
|
|
sys.exit(0)
|
|
print("ONE OR MORE TIERS FAILED")
|
|
sys.exit(1)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|