0d373da490
Creates scripts/validation/ with per-tier exit checks consuming the
aggregate JSON from scripts/aggregate-multi-seed-metrics.py (P5T1B):
check_tier1.py — convergence (std/mean ≤ 0.15 on val_sharpe /
avg_q_value / train_loss; avg_q_value max ≤ 500 fold-1 explosion
guard; placeholders for Q-saturation + hot-path-DtoH per spec).
check_tier2.py — behavioural (val_trades_per_bar ≥ 0.005,
val_active_frac > 0.2, dir argmax entropy > 0.8·log4 with
val_dir_entropy primary + val_dir_dist_* fallback).
check_tier3.py — profitability (val_sharpe_annualised > 1.0 with
val_sharpe per-bar fallback, val_win_rate ≥ 0.52 gated on
>500 trades, val_profit_factor mean ≥ 1.1 AND cross-seed std < 0.3).
check_all_tiers.py — subprocess wrapper, exits 0 only if all pass.
Stdlib-only (statistics / argparse / json / subprocess) — no new deps.
Defensive missing-metric handling: each check FAILs with an explanatory
message when its required aggregate key is absent rather than silently
passing, so missing HEALTH_DIAG metrics are surfaced loudly.
Test harness scripts/validation/tests/test_tier_checks.sh exercises
good + bad fixtures across all four scripts and against the wrapper.
Audit row added to docs/dqn-wire-up-audit.md documenting the suite +
the deferred metrics list (val_trades_per_bar, val_active_frac,
val_dir_entropy/_dist_*, val_sharpe_annualised, val_win_rate,
val_profit_factor, val_trade_count) that HEALTH_DIAG must emit before
tiers 2/3 can ever PASS on real data — tracked for Plan 5 Task 5
pre-flight wire-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
29 lines
1.6 KiB
JSON
29 lines
1.6 KiB
JSON
{
|
|
"tag": "synthetic-bad-tier1",
|
|
"multi_seed": 3,
|
|
"folds": 1,
|
|
"warmup_end_epoch": 15,
|
|
"streams_seen": 3,
|
|
"aggregates": {
|
|
"val_sharpe": [
|
|
{"epoch": 0, "mean": 0.10, "std": 0.50, "median": 0.10, "min": -0.5, "max": 0.7, "n_samples": 3},
|
|
{"epoch": 15, "mean": 1.00, "std": 10.0, "median": 1.00, "min": -9.0, "max": 11.0, "n_samples": 3},
|
|
{"epoch": 20, "mean": 1.05, "std": 9.5, "median": 1.05, "min": -8.5, "max": 10.5, "n_samples": 3},
|
|
{"epoch": 25, "mean": 0.95, "std": 9.7, "median": 0.95, "min": -8.7, "max": 10.7, "n_samples": 3},
|
|
{"epoch": 29, "mean": 1.00, "std": 10.2, "median": 1.00, "min": -9.0, "max": 11.0, "n_samples": 3}
|
|
],
|
|
"avg_q_value": [
|
|
{"epoch": 0, "mean": 1.0, "std": 5.0, "median": 1.0, "min": -4.0, "max": 6.0, "n_samples": 3},
|
|
{"epoch": 5, "mean": 50.0, "std": 30.0, "median": 50.0, "min": 20.0, "max": 80.0, "n_samples": 3},
|
|
{"epoch": 10, "mean": 500.0, "std": 300.0, "median": 500.0, "min": 200.0, "max": 800.0, "n_samples": 3},
|
|
{"epoch": 15, "mean": 5000.0, "std": 3000.0,"median": 5000.0,"min": 2000.0, "max": 10000.0, "n_samples": 3}
|
|
],
|
|
"train_loss": [
|
|
{"epoch": 0, "mean": 5.00, "std": 2.00, "median": 5.00, "min": 3.00, "max": 7.00, "n_samples": 3},
|
|
{"epoch": 15, "mean": 50.0, "std": 100.0, "median": 50.0, "min": -50.0, "max": 150.0, "n_samples": 3},
|
|
{"epoch": 20, "mean": 60.0, "std": 110.0, "median": 60.0, "min": -50.0, "max": 170.0, "n_samples": 3},
|
|
{"epoch": 29, "mean": 55.0, "std": 105.0, "median": 55.0, "min": -50.0, "max": 160.0, "n_samples": 3}
|
|
]
|
|
}
|
|
}
|