0d373da490
Creates scripts/validation/ with per-tier exit checks consuming the
aggregate JSON from scripts/aggregate-multi-seed-metrics.py (P5T1B):
check_tier1.py — convergence (std/mean ≤ 0.15 on val_sharpe /
avg_q_value / train_loss; avg_q_value max ≤ 500 fold-1 explosion
guard; placeholders for Q-saturation + hot-path-DtoH per spec).
check_tier2.py — behavioural (val_trades_per_bar ≥ 0.005,
val_active_frac > 0.2, dir argmax entropy > 0.8·log4 with
val_dir_entropy primary + val_dir_dist_* fallback).
check_tier3.py — profitability (val_sharpe_annualised > 1.0 with
val_sharpe per-bar fallback, val_win_rate ≥ 0.52 gated on
>500 trades, val_profit_factor mean ≥ 1.1 AND cross-seed std < 0.3).
check_all_tiers.py — subprocess wrapper, exits 0 only if all pass.
Stdlib-only (statistics / argparse / json / subprocess) — no new deps.
Defensive missing-metric handling: each check FAILs with an explanatory
message when its required aggregate key is absent rather than silently
passing, so missing HEALTH_DIAG metrics are surfaced loudly.
Test harness scripts/validation/tests/test_tier_checks.sh exercises
good + bad fixtures across all four scripts and against the wrapper.
Audit row added to docs/dqn-wire-up-audit.md documenting the suite +
the deferred metrics list (val_trades_per_bar, val_active_frac,
val_dir_entropy/_dist_*, val_sharpe_annualised, val_win_rate,
val_profit_factor, val_trade_count) that HEALTH_DIAG must emit before
tiers 2/3 can ever PASS on real data — tracked for Plan 5 Task 5
pre-flight wire-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
47 lines
2.5 KiB
Bash
Executable File
47 lines
2.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Test: tier validation scripts accept good fixtures and reject bad ones.
|
|
#
|
|
# This is the validation surface for the Plan 5 Task 4 tier exit suite —
|
|
# verifies the shape of pass/fail logic on synthetic aggregate JSONs
|
|
# without needing a real training run.
|
|
set -euo pipefail
|
|
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
SCRIPT_DIR="$(cd "${HERE}/.." && pwd)"
|
|
FIXTURES="${HERE}/fixtures"
|
|
|
|
# ── Tier 1 ─────────────────────────────────────────────────────────────────
|
|
# Bad metrics JSON should fail (exit code 1, masked by `if`).
|
|
if python3 "${SCRIPT_DIR}/check_tier1.py" "${FIXTURES}/bad_tier1.json" --warmup-end 15; then
|
|
echo "FAIL: tier1 check accepted bad metrics"
|
|
exit 1
|
|
fi
|
|
|
|
# Known-good metrics JSON should pass (exit code 0).
|
|
python3 "${SCRIPT_DIR}/check_tier1.py" "${FIXTURES}/good_tier1.json" --warmup-end 15
|
|
|
|
# ── Tier 2 ─────────────────────────────────────────────────────────────────
|
|
# Bad fixture lacks val_trades_per_bar / val_active_frac / dir-entropy entirely
|
|
# → tier2 must FAIL with the missing-metric explanation.
|
|
if python3 "${SCRIPT_DIR}/check_tier2.py" "${FIXTURES}/bad_tier1.json" --warmup-end 15; then
|
|
echo "FAIL: tier2 check accepted aggregate without behavioural metrics"
|
|
exit 1
|
|
fi
|
|
|
|
# ── Tier 3 ─────────────────────────────────────────────────────────────────
|
|
# Bad fixture lacks val_sharpe_annualised / val_win_rate / val_profit_factor
|
|
# entirely → tier3 must FAIL with the missing-metric explanation.
|
|
if python3 "${SCRIPT_DIR}/check_tier3.py" "${FIXTURES}/bad_tier1.json" --warmup-end 15; then
|
|
echo "FAIL: tier3 check accepted aggregate without profitability metrics"
|
|
exit 1
|
|
fi
|
|
|
|
# ── all-tiers wrapper ──────────────────────────────────────────────────────
|
|
# Wrapper must reject any aggregate that fails any tier (uses bad fixture).
|
|
if python3 "${SCRIPT_DIR}/check_all_tiers.py" "${FIXTURES}/bad_tier1.json" --warmup-end 15; then
|
|
echo "FAIL: check_all_tiers.py accepted aggregate that fails tier1+2+3"
|
|
exit 1
|
|
fi
|
|
|
|
echo "PASS"
|