gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

plan5(task4): tiered-exit validation script suite (tier1/2/3 checks)

Browse Source 
Creates scripts/validation/ with per-tier exit checks consuming the
aggregate JSON from scripts/aggregate-multi-seed-metrics.py (P5T1B):

  check_tier1.py — convergence (std/mean ≤ 0.15 on val_sharpe /
    avg_q_value / train_loss; avg_q_value max ≤ 500 fold-1 explosion
    guard; placeholders for Q-saturation + hot-path-DtoH per spec).
  check_tier2.py — behavioural (val_trades_per_bar ≥ 0.005,
    val_active_frac > 0.2, dir argmax entropy > 0.8·log4 with
    val_dir_entropy primary + val_dir_dist_* fallback).
  check_tier3.py — profitability (val_sharpe_annualised > 1.0 with
    val_sharpe per-bar fallback, val_win_rate ≥ 0.52 gated on
    >500 trades, val_profit_factor mean ≥ 1.1 AND cross-seed std < 0.3).
  check_all_tiers.py — subprocess wrapper, exits 0 only if all pass.

Stdlib-only (statistics / argparse / json / subprocess) — no new deps.
Defensive missing-metric handling: each check FAILs with an explanatory
message when its required aggregate key is absent rather than silently
passing, so missing HEALTH_DIAG metrics are surfaced loudly.

Test harness scripts/validation/tests/test_tier_checks.sh exercises
good + bad fixtures across all four scripts and against the wrapper.

Audit row added to docs/dqn-wire-up-audit.md documenting the suite +
the deferred metrics list (val_trades_per_bar, val_active_frac,
val_dir_entropy/_dist_*, val_sharpe_annualised, val_win_rate,
val_profit_factor, val_trade_count) that HEALTH_DIAG must emit before
tiers 2/3 can ever PASS on real data — tracked for Plan 5 Task 5
pre-flight wire-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-26 12:33:28 +02:00
parent 2606506cd8
commit 0d373da490
8 changed files with 679 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/dqn-wire-up-audit.md
View File

@@ -48,6 +48,7 @@
| `config/metric-bands.toml` | `MetricBandsRegistry::load_from_toml` (auto-loaded by `DQNTrainer::new_internal`) | Wired | Plan 5 Task 2 — per-metric warn/error bands (mean ± 5σ / mean ± 10σ); populated by `scripts/populate-metric-bands-from-runs.py` from Plan 5 Task 1B aggregate JSONs | — |
| `scripts/populate-metric-bands-from-runs.py` | reads aggregate JSONs from `scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2); writes `config/metric-bands.toml` `[bands]` section | Wired | Plan 5 Task 2 helper — N>=3 sample path; N=1 multiplicative fallback | — |
| `scripts/argo-train.sh --profile` + `infra/k8s/argo/train-multi-seed-template.yaml` (`profile` parameter, conditional `nsys profile` wrapper, `mc cp` upload to `foxhunt-training-artifacts/profiles/<sha>/`) + `infra/docker/Dockerfile.foxhunt-training-runtime` (`nsight-systems-cli` apt install) + `infra/k8s/minio/minio.yaml` (new `foxhunt-training-artifacts` bucket) | invoked manually via `./scripts/argo-train.sh dqn --profile [...]`; consumed by `scripts/compare-nsys-profiles.py BASELINE CURRENT --epochs N` (V0 metric: total cuda_gpu_kern_sum / epochs; V1 NVTX-range path deferred to Plan 5 Task 5) | Wired | Plan 5 Task 3 A.4.1 — nsys profile harness with regression-comparison script. `--profile` forces multi-seed render path so dry-run never needs cluster contact (test_nsys_harness.sh); MinIO creds optional on `train-single` (warn-skips upload if absent). **Baseline capture deferred to Plan 5 Task 5** (T5 runs the harness on L40S as part of the multi-seed acceptance pass — avoids burning local GPU time in T3) | — |
| `scripts/validation/check_tier1.py` + `check_tier2.py` + `check_tier3.py` + `check_all_tiers.py` (+ `tests/test_tier_checks.sh` + `tests/fixtures/{good,bad}_tier1.json`) | reads aggregate JSONs from `scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2); invoked manually as `python3 scripts/validation/check_all_tiers.py <metrics.json> [--warmup-end N]` and from Plan 5 Task 5's acceptance pass | Wired | Plan 5 Task 4 — per-tier exit checks for the spec §2 tiered acceptance criteria. Tier 1 (convergence: std/mean ≤ 0.15 on `val_sharpe`/`avg_q_value`/`train_loss` over stable epochs + no avg_q_value > 500 fold-1 explosion + Q-saturation/hot-path-DtoH placeholders); Tier 2 (behavioural: `val_trades_per_bar ≥ 0.005`, `val_active_frac > 0.2`, dir entropy > 0.8·log4); Tier 3 (profitability: `val_sharpe_annualised > 1.0` with per-bar fallback, `val_win_rate ≥ 0.52` gated on >500 trades, `val_profit_factor` mean ≥ 1.1 AND cross-seed std < 0.3). Stdlib only (no numpy/scipy). Defensive missing-metric handling — each missing aggregate key fails the relevant check with an explanatory message rather than silently passing. **Deferred metrics not yet emitted by HEALTH_DIAG** (current aggregator key set is `<block>__<key>` per `aggregate-multi-seed-metrics.py:115`): `val_trades_per_bar`, `val_active_frac`, `val_dir_entropy` / `val_dir_dist_*`, `val_sharpe_annualised`, `val_win_rate`, `val_profit_factor`, `val_trade_count`. Wiring those into `training_loop.rs::log_epoch_metrics_and_financials` (and either flat-emitting or tracking through the band-check harvest path) is the prerequisite for Tiers 2/3 to ever PASS on real data — tracked under Plan 5 Task 5's pre-flight. | — |
| `trainers/dqn/adaptive_monitor.rs` | Read-only observer trait + harness (FireRateStats, DiagSnapshot, IsvBus<'a>); consumers added in Plan 1 Tasks 9-17 (atoms/gamma/kelly_cap/tau/epsilon/grad_balancer monitors) | Wired (consumers added in same plan) | C.6 GPU-drives-CPU-reads | — |
| `trainers/dqn/monitors/grad_balancer_monitor.rs` | Read-only observer for grad_balance_isv_update kernel output (ISV slots 31..35); consumers: HEALTH_DIAG + controller_activity smoke | Wired | Plan 1 Task 17 | — |
| `trainers/dqn/monitors/tau_monitor.rs` | Read-only observer for tau_update kernel output (ISV slot 42); consumers: HEALTH_DIAG + controller_activity smoke | Wired | Plan 1 Task 13 | — |

72
scripts/validation/check_all_tiers.py Executable file
View File

@@ -0,0 +1,72 @@
#!/usr/bin/env python3
"""Top-level wrapper: run check_tier{1,2,3}.py in sequence (Plan 5 Task 4).
Subprocess-each strategy chosen over in-process import so that:
- per-tier checks remain runnable standalone (single source of truth);
- per-tier exit codes propagate naturally;
- tier scripts can evolve independently without touching this wrapper.
Exits 0 only if all three tier checks PASS. Each tier's own PASS/FAIL
lines stream through to stdout so the top-level summary is auditable.
"""
from __future__ import annotations
import argparse
import os
import subprocess
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
TIER_SCRIPTS = (
"check_tier1.py",
"check_tier2.py",
"check_tier3.py",
)
def main():
ap = argparse.ArgumentParser(
description="Run all three tiered exit checks (Plan 5 Task 4).",
)
ap.add_argument(
"metrics_json",
help="Aggregated metrics JSON (output of aggregate-multi-seed-metrics.py).",
)
ap.add_argument(
"--warmup-end",
type=int,
default=15,
help="Epoch at which warmup ends; forwarded to each tier check.",
)
args = ap.parse_args()
overall_ok = True
for script in TIER_SCRIPTS:
path = os.path.join(HERE, script)
print(f"=== {script} ===")
result = subprocess.run(
[
sys.executable,
path,
args.metrics_json,
"--warmup-end",
str(args.warmup_end),
],
check=False,
)
if result.returncode != 0:
overall_ok = False
print(f"--- {script}: FAIL (exit {result.returncode}) ---")
else:
print(f"--- {script}: PASS ---")
if overall_ok:
print("ALL TIERS PASS")
sys.exit(0)
print("ONE OR MORE TIERS FAILED")
sys.exit(1)
if __name__ == "__main__":
main()

133
scripts/validation/check_tier1.py Executable file
View File

@@ -0,0 +1,133 @@
#!/usr/bin/env python3
"""Tier 1 convergence exit check (Plan 5 Task 4).
Consumes the aggregated metrics JSON produced by
`scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2).
Pass criteria (per spec §2 Tier 1 — convergence):
1. std/mean ≤ 0.15 on stable epochs (>= warmup_end) for `val_sharpe`,
`avg_q_value`, `train_loss`.
2. avg_q_value.max ≤ 500 across all epochs/streams (no fold-1 explosion;
the fold-1 explosion bug from task #31 produced max_q ~ 1e4+).
3. Q-saturation placeholder (see A.6 audit — full check landed elsewhere).
4. Hot-path DtoH placeholder (enforced by pre-commit hook).
Exits 0 on PASS, 1 on FAIL. Each check prints a `PASS:` / `FAIL:` line.
"""
from __future__ import annotations
import argparse
import json
import sys
from statistics import mean
def _stable(series, warmup_end):
"""Return entries with epoch >= warmup_end."""
return [x for x in series if x["epoch"] >= warmup_end]
def check_std_over_mean(agg, warmup_end):
"""std/mean ≤ 0.15 on stable epochs for the three Tier 1 metrics.
Defensive: if a metric is missing from the aggregate, return False with
a clear message so the run can be flagged for re-aggregation rather
than silently passing.
"""
for metric in ("val_sharpe", "avg_q_value", "train_loss"):
series = agg.get(metric)
if not series:
return False, f"{metric} missing from aggregate JSON"
stable = _stable(series, warmup_end)
if not stable:
return False, (
f"No stable epochs for {metric} past warmup_end={warmup_end}"
)
m = mean(x["mean"] for x in stable)
s = mean(x["std"] for x in stable)
ratio = abs(s / (m + 1e-9)) if abs(m) > 1e-6 else float("inf")
if ratio > 0.15:
return False, (
f"{metric} std/mean = {ratio:.3f} > 0.15 on stable epochs"
)
return True, "std/mean <= 0.15 on all Tier 1 metrics"
def check_no_fold1_explosion(agg):
"""avg_q_value.max ≤ 500 — catches the fold-1 loss explosion (task #31)."""
series = agg.get("avg_q_value", [])
if not series:
# Defensive: missing series cannot prove there was no explosion;
# failing is the safe direction.
return False, "avg_q_value missing from aggregate JSON"
max_q = max((x["max"] for x in series), default=0)
if abs(max_q) > 500:
return False, f"avg_q_value max={max_q}, possible fold-1 explosion"
return True, "no fold-1 explosion detected"
def check_no_q_saturation(_agg):
"""Q-saturation placeholder.
Full per-bin saturation check (q_full / q_half / q_quarter within 1%
of ±abs_half) is tracked under the A.6 audit rather than embedded in
the tier1 script — the relevant signal lives in HEALTH_DIAG `mag`
block and is best validated end-to-end on a real run.
"""
return True, "no Q-saturation check (placeholder — see A.6 audit)"
def check_no_hot_path_dtoh(_agg):
"""Hot-path DtoH enforcement is a pre-commit responsibility.
`scripts/gpu-hotpath-guard.sh` rejects any new `DeviceCopy`-style
DtoH inside the per-step training path. Reaching this script implies
that guard already passed, so we surface the invariant here as PASS
rather than re-deriving it from runtime metrics.
"""
return True, "hot-path audit enforced by pre-commit"
def main():
ap = argparse.ArgumentParser(
description="Tier 1 convergence exit check (Plan 5 Task 4).",
)
ap.add_argument(
"metrics_json",
help="Aggregated metrics JSON (output of aggregate-multi-seed-metrics.py).",
)
ap.add_argument(
"--warmup-end",
type=int,
default=15,
help="Epoch at which warmup ends (stable epochs are > this); default 15.",
)
args = ap.parse_args()
try:
with open(args.metrics_json) as fh:
data = json.load(fh)
except (OSError, json.JSONDecodeError) as exc:
print(f"FAIL: cannot read {args.metrics_json}: {exc}")
sys.exit(1)
agg = data.get("aggregates")
if not isinstance(agg, dict):
print("FAIL: missing or malformed `aggregates` block in metrics JSON")
sys.exit(1)
checks = [
check_std_over_mean(agg, args.warmup_end),
check_no_fold1_explosion(agg),
check_no_q_saturation(agg),
check_no_hot_path_dtoh(agg),
]
all_pass = all(ok for ok, _ in checks)
for ok, msg in checks:
print(("PASS" if ok else "FAIL") + ": " + msg)
sys.exit(0 if all_pass else 1)
if __name__ == "__main__":
main()

178
scripts/validation/check_tier2.py Executable file
View File

@@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""Tier 2 behavioural exit check (Plan 5 Task 4).
Consumes the aggregated metrics JSON produced by
`scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2).
Pass criteria (per spec §2 Tier 2 — behavioural):
1. `val_trades_per_bar` mean ≥ 0.005 on stable epochs (model trades).
2. `val_active_frac` mean > 0.2 on stable epochs (model is not always Hold).
3. Direction-argmax entropy > 0.8 × log(4) ≈ 1.109 on stable epochs.
Preferred source: a `val_dir_entropy` aggregate metric. Fallback:
derive from `val_dir_dist_*` (per-direction probability) if present.
If neither is in the aggregate, FAIL with an explanatory message so
the missing metric gets added to HEALTH_DIAG in a follow-up run.
Exits 0 on PASS, 1 on FAIL. Each check prints a `PASS:` / `FAIL:` line.
"""
from __future__ import annotations
import argparse
import json
import math
import sys
from statistics import mean
# 0.8 * log(4) ≈ 1.10903549
DIR_ENTROPY_FLOOR = 0.8 * math.log(4.0)
def _stable(series, warmup_end):
return [x for x in series if x["epoch"] >= warmup_end]
def check_trades_per_bar(agg, warmup_end):
series = agg.get("val_trades_per_bar")
if not series:
return False, (
"val_trades_per_bar missing from aggregate — required for Tier 2"
)
stable = _stable(series, warmup_end)
if not stable:
return False, (
f"No stable epochs for val_trades_per_bar past warmup_end={warmup_end}"
)
m = mean(x["mean"] for x in stable)
if m < 0.005:
return False, f"val_trades_per_bar mean = {m:.5f} < 0.005"
return True, f"val_trades_per_bar mean = {m:.5f} >= 0.005"
def check_active_frac(agg, warmup_end):
series = agg.get("val_active_frac")
if not series:
return False, (
"val_active_frac missing from aggregate — required for Tier 2"
)
stable = _stable(series, warmup_end)
if not stable:
return False, (
f"No stable epochs for val_active_frac past warmup_end={warmup_end}"
)
m = mean(x["mean"] for x in stable)
if m <= 0.2:
return False, f"val_active_frac mean = {m:.4f} <= 0.2"
return True, f"val_active_frac mean = {m:.4f} > 0.2"
def _entropy_from_dist(probs):
"""Shannon entropy in nats; ignores non-positive probabilities."""
return -sum(p * math.log(p) for p in probs if p > 0)
def check_dir_entropy(agg, warmup_end):
"""Direction argmax entropy > 0.8 × log(4).
Prefers a pre-computed `val_dir_entropy` aggregate. Falls back to
deriving entropy from per-direction probability mass (`val_dir_dist_short`
/ `val_dir_dist_hold` / `val_dir_dist_long` / `val_dir_dist_flat` —
naming kept loose to match whatever HEALTH_DIAG ends up emitting).
"""
direct = agg.get("val_dir_entropy")
if direct:
stable = _stable(direct, warmup_end)
if not stable:
return False, (
f"No stable epochs for val_dir_entropy past warmup_end={warmup_end}"
)
m = mean(x["mean"] for x in stable)
if m <= DIR_ENTROPY_FLOOR:
return False, (
f"val_dir_entropy mean = {m:.4f} <= {DIR_ENTROPY_FLOOR:.4f} "
f"(0.8 × log 4)"
)
return True, (
f"val_dir_entropy mean = {m:.4f} > {DIR_ENTROPY_FLOOR:.4f} "
f"(0.8 × log 4)"
)
# Fallback: derive from per-direction distribution.
dist_keys = [
"val_dir_dist_short",
"val_dir_dist_hold",
"val_dir_dist_long",
"val_dir_dist_flat",
]
dist_series = [agg.get(k) for k in dist_keys]
if all(s for s in dist_series):
# Align by epoch on stable epochs.
stables = [_stable(s, warmup_end) for s in dist_series]
if any(not s for s in stables):
return False, (
f"No stable epochs for val_dir_dist_* past warmup_end={warmup_end}"
)
# Average per-direction mean across stable epochs, then compute entropy.
means = [mean(x["mean"] for x in s) for s in stables]
total = sum(means)
if total <= 0:
return False, "val_dir_dist_* sums to <= 0; cannot compute entropy"
probs = [m / total for m in means]
ent = _entropy_from_dist(probs)
if ent <= DIR_ENTROPY_FLOOR:
return False, (
f"derived dir entropy = {ent:.4f} <= {DIR_ENTROPY_FLOOR:.4f}"
)
return True, (
f"derived dir entropy = {ent:.4f} > {DIR_ENTROPY_FLOOR:.4f}"
)
return False, (
"neither val_dir_entropy nor val_dir_dist_* present in aggregate — "
"add a direction-distribution metric to HEALTH_DIAG to enable "
"Tier 2 entropy check"
)
def main():
ap = argparse.ArgumentParser(
description="Tier 2 behavioural exit check (Plan 5 Task 4).",
)
ap.add_argument(
"metrics_json",
help="Aggregated metrics JSON (output of aggregate-multi-seed-metrics.py).",
)
ap.add_argument(
"--warmup-end",
type=int,
default=15,
help="Epoch at which warmup ends (stable epochs are > this); default 15.",
)
args = ap.parse_args()
try:
with open(args.metrics_json) as fh:
data = json.load(fh)
except (OSError, json.JSONDecodeError) as exc:
print(f"FAIL: cannot read {args.metrics_json}: {exc}")
sys.exit(1)
agg = data.get("aggregates")
if not isinstance(agg, dict):
print("FAIL: missing or malformed `aggregates` block in metrics JSON")
sys.exit(1)
checks = [
check_trades_per_bar(agg, args.warmup_end),
check_active_frac(agg, args.warmup_end),
check_dir_entropy(agg, args.warmup_end),
]
all_pass = all(ok for ok, _ in checks)
for ok, msg in checks:
print(("PASS" if ok else "FAIL") + ": " + msg)
sys.exit(0 if all_pass else 1)
if __name__ == "__main__":
main()

185
scripts/validation/check_tier3.py Executable file
View File

@@ -0,0 +1,185 @@
#!/usr/bin/env python3
"""Tier 3 profitability exit check (Plan 5 Task 4).
Consumes the aggregated metrics JSON produced by
`scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2).
Pass criteria (per spec §2 Tier 3 — profitability):
1. `val_sharpe_annualised` mean > 1.0 on stable epochs.
If only `val_sharpe` (per-bar) is in the aggregate, fall back to it
as a proxy and document the difference in the PASS/FAIL message.
2. `val_win_rate` mean ≥ 0.52 — only enforced when the corresponding
trade-count series indicates > 500 trades/bar over stable epochs.
Below 500-trade gate, win-rate is too noisy to assess.
3. `val_profit_factor` mean ≥ 1.1 AND across-seed std < 0.3 on stable
epochs (the std is the across-seed dispersion baked into each
epoch entry by the aggregator).
Exits 0 on PASS, 1 on FAIL. Each check prints a `PASS:` / `FAIL:` line.
"""
from __future__ import annotations
import argparse
import json
import sys
from statistics import mean
def _stable(series, warmup_end):
return [x for x in series if x["epoch"] >= warmup_end]
def check_sharpe(agg, warmup_end):
annualised = agg.get("val_sharpe_annualised")
if annualised:
stable = _stable(annualised, warmup_end)
if not stable:
return False, (
f"No stable epochs for val_sharpe_annualised past "
f"warmup_end={warmup_end}"
)
m = mean(x["mean"] for x in stable)
if m <= 1.0:
return False, f"val_sharpe_annualised mean = {m:.4f} <= 1.0"
return True, f"val_sharpe_annualised mean = {m:.4f} > 1.0"
# Fallback: per-bar val_sharpe — note proxy in message.
proxy = agg.get("val_sharpe")
if proxy:
stable = _stable(proxy, warmup_end)
if not stable:
return False, (
f"No stable epochs for val_sharpe past warmup_end={warmup_end}"
)
m = mean(x["mean"] for x in stable)
# Per-bar sharpe is much smaller than annualised; use the same > 1.0
# threshold but flag the proxy substitution loud.
if m <= 1.0:
return False, (
f"val_sharpe (per-bar PROXY for annualised) mean = {m:.4f} "
f"<= 1.0 — annualised metric missing from aggregate"
)
return True, (
f"val_sharpe (per-bar PROXY for annualised) mean = {m:.4f} > 1.0"
)
return False, (
"neither val_sharpe_annualised nor val_sharpe present in aggregate "
"— required for Tier 3 profitability check"
)
def check_win_rate(agg, warmup_end):
series = agg.get("val_win_rate")
if not series:
return False, (
"val_win_rate missing from aggregate — required for Tier 3"
)
stable = _stable(series, warmup_end)
if not stable:
return False, (
f"No stable epochs for val_win_rate past warmup_end={warmup_end}"
)
# Trade-count gate. Use val_trades_per_bar mean × an assumed bar count
# of 500; if explicit val_trade_count is present, prefer that.
trade_count_series = agg.get("val_trade_count")
if trade_count_series:
tc_stable = _stable(trade_count_series, warmup_end)
if tc_stable:
tc = mean(x["mean"] for x in tc_stable)
if tc <= 500:
return True, (
f"val_win_rate skipped — trade count {tc:.0f} <= 500 "
f"(too noisy)"
)
else:
# Approximate from per-bar trade rate × stable epoch span; if the
# rate is below ~0.001 over even a long horizon the count is below
# 500. Conservative — only skip if we can prove count <= 500.
tpb_series = agg.get("val_trades_per_bar")
if tpb_series:
tpb_stable = _stable(tpb_series, warmup_end)
if tpb_stable:
tpb = mean(x["mean"] for x in tpb_stable)
# Without knowing the eval window length, assume a
# conservative 5e5 bars (a single day at 1Hz). If even
# that yields <= 500 trades, skip.
if tpb * 5e5 <= 500:
return True, (
f"val_win_rate skipped — estimated trade count "
f"{tpb * 5e5:.0f} <= 500 (too noisy)"
)
m = mean(x["mean"] for x in stable)
if m < 0.52:
return False, f"val_win_rate mean = {m:.4f} < 0.52"
return True, f"val_win_rate mean = {m:.4f} >= 0.52"
def check_profit_factor(agg, warmup_end):
series = agg.get("val_profit_factor")
if not series:
return False, (
"val_profit_factor missing from aggregate — required for Tier 3"
)
stable = _stable(series, warmup_end)
if not stable:
return False, (
f"No stable epochs for val_profit_factor past warmup_end={warmup_end}"
)
m = mean(x["mean"] for x in stable)
s = mean(x["std"] for x in stable)
if m < 1.1:
return False, f"val_profit_factor mean = {m:.4f} < 1.1"
if s >= 0.3:
return False, (
f"val_profit_factor cross-seed std = {s:.4f} >= 0.3 (unstable)"
)
return True, (
f"val_profit_factor mean = {m:.4f} >= 1.1, std = {s:.4f} < 0.3"
)
def main():
ap = argparse.ArgumentParser(
description="Tier 3 profitability exit check (Plan 5 Task 4).",
)
ap.add_argument(
"metrics_json",
help="Aggregated metrics JSON (output of aggregate-multi-seed-metrics.py).",
)
ap.add_argument(
"--warmup-end",
type=int,
default=15,
help="Epoch at which warmup ends (stable epochs are > this); default 15.",
)
args = ap.parse_args()
try:
with open(args.metrics_json) as fh:
data = json.load(fh)
except (OSError, json.JSONDecodeError) as exc:
print(f"FAIL: cannot read {args.metrics_json}: {exc}")
sys.exit(1)
agg = data.get("aggregates")
if not isinstance(agg, dict):
print("FAIL: missing or malformed `aggregates` block in metrics JSON")
sys.exit(1)
checks = [
check_sharpe(agg, args.warmup_end),
check_win_rate(agg, args.warmup_end),
check_profit_factor(agg, args.warmup_end),
]
all_pass = all(ok for ok, _ in checks)
for ok, msg in checks:
print(("PASS" if ok else "FAIL") + ": " + msg)
sys.exit(0 if all_pass else 1)
if __name__ == "__main__":
main()

28
scripts/validation/tests/fixtures/bad_tier1.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"tag": "synthetic-bad-tier1",
"multi_seed": 3,
"folds": 1,
"warmup_end_epoch": 15,
"streams_seen": 3,
"aggregates": {
"val_sharpe": [
{"epoch": 0, "mean": 0.10, "std": 0.50, "median": 0.10, "min": -0.5, "max": 0.7, "n_samples": 3},
{"epoch": 15, "mean": 1.00, "std": 10.0, "median": 1.00, "min": -9.0, "max": 11.0, "n_samples": 3},
{"epoch": 20, "mean": 1.05, "std": 9.5, "median": 1.05, "min": -8.5, "max": 10.5, "n_samples": 3},
{"epoch": 25, "mean": 0.95, "std": 9.7, "median": 0.95, "min": -8.7, "max": 10.7, "n_samples": 3},
{"epoch": 29, "mean": 1.00, "std": 10.2, "median": 1.00, "min": -9.0, "max": 11.0, "n_samples": 3}
],
"avg_q_value": [
{"epoch": 0, "mean": 1.0, "std": 5.0, "median": 1.0, "min": -4.0, "max": 6.0, "n_samples": 3},
{"epoch": 5, "mean": 50.0, "std": 30.0, "median": 50.0, "min": 20.0, "max": 80.0, "n_samples": 3},
{"epoch": 10, "mean": 500.0, "std": 300.0, "median": 500.0, "min": 200.0, "max": 800.0, "n_samples": 3},
{"epoch": 15, "mean": 5000.0, "std": 3000.0,"median": 5000.0,"min": 2000.0, "max": 10000.0, "n_samples": 3}
],
"train_loss": [
{"epoch": 0, "mean": 5.00, "std": 2.00, "median": 5.00, "min": 3.00, "max": 7.00, "n_samples": 3},
{"epoch": 15, "mean": 50.0, "std": 100.0, "median": 50.0, "min": -50.0, "max": 150.0, "n_samples": 3},
{"epoch": 20, "mean": 60.0, "std": 110.0, "median": 60.0, "min": -50.0, "max": 170.0, "n_samples": 3},
{"epoch": 29, "mean": 55.0, "std": 105.0, "median": 55.0, "min": -50.0, "max": 160.0, "n_samples": 3}
]
}
}

36
scripts/validation/tests/fixtures/good_tier1.json vendored Normal file
View File

@@ -0,0 +1,36 @@
{
"tag": "synthetic-good-tier1",
"multi_seed": 3,
"folds": 1,
"warmup_end_epoch": 15,
"streams_seen": 3,
"aggregates": {
"val_sharpe": [
{"epoch": 0, "mean": 0.10, "std": 0.50, "median": 0.10, "min": -0.5, "max": 0.7, "n_samples": 3},
{"epoch": 5, "mean": 0.80, "std": 0.30, "median": 0.80, "min": 0.5, "max": 1.1, "n_samples": 3},
{"epoch": 10, "mean": 1.50, "std": 0.20, "median": 1.50, "min": 1.3, "max": 1.7, "n_samples": 3},
{"epoch": 15, "mean": 2.00, "std": 0.05, "median": 2.00, "min": 1.95, "max": 2.05, "n_samples": 3},
{"epoch": 20, "mean": 2.05, "std": 0.06, "median": 2.05, "min": 1.99, "max": 2.11, "n_samples": 3},
{"epoch": 25, "mean": 2.02, "std": 0.04, "median": 2.02, "min": 1.98, "max": 2.06, "n_samples": 3},
{"epoch": 29, "mean": 2.04, "std": 0.05, "median": 2.04, "min": 1.99, "max": 2.09, "n_samples": 3}
],
"avg_q_value": [
{"epoch": 0, "mean": 1.0, "std": 5.0, "median": 1.0, "min": -4.0, "max": 6.0, "n_samples": 3},
{"epoch": 5, "mean": 5.0, "std": 2.0, "median": 5.0, "min": 3.0, "max": 7.0, "n_samples": 3},
{"epoch": 10, "mean": 8.0, "std": 1.0, "median": 8.0, "min": 7.0, "max": 9.0, "n_samples": 3},
{"epoch": 15, "mean": 10.0, "std": 1.0, "median": 10.0, "min": 9.0, "max": 11.0, "n_samples": 3},
{"epoch": 20, "mean": 10.2, "std": 0.8, "median": 10.2, "min": 9.4, "max": 11.0, "n_samples": 3},
{"epoch": 25, "mean": 10.1, "std": 0.9, "median": 10.1, "min": 9.2, "max": 11.0, "n_samples": 3},
{"epoch": 29, "mean": 10.3, "std": 1.0, "median": 10.3, "min": 9.3, "max": 11.3, "n_samples": 3}
],
"train_loss": [
{"epoch": 0, "mean": 5.00, "std": 2.00, "median": 5.00, "min": 3.00, "max": 7.00, "n_samples": 3},
{"epoch": 5, "mean": 2.00, "std": 0.50, "median": 2.00, "min": 1.50, "max": 2.50, "n_samples": 3},
{"epoch": 10, "mean": 1.00, "std": 0.20, "median": 1.00, "min": 0.80, "max": 1.20, "n_samples": 3},
{"epoch": 15, "mean": 0.50, "std": 0.05, "median": 0.50, "min": 0.45, "max": 0.55, "n_samples": 3},
{"epoch": 20, "mean": 0.48, "std": 0.06, "median": 0.48, "min": 0.42, "max": 0.54, "n_samples": 3},
{"epoch": 25, "mean": 0.49, "std": 0.04, "median": 0.49, "min": 0.45, "max": 0.53, "n_samples": 3},
{"epoch": 29, "mean": 0.50, "std": 0.05, "median": 0.50, "min": 0.45, "max": 0.55, "n_samples": 3}
]
}
}

46
scripts/validation/tests/test_tier_checks.sh Executable file
View File

@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# Test: tier validation scripts accept good fixtures and reject bad ones.
#
# This is the validation surface for the Plan 5 Task 4 tier exit suite —
# verifies the shape of pass/fail logic on synthetic aggregate JSONs
# without needing a real training run.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT_DIR="$(cd "${HERE}/.." && pwd)"
FIXTURES="${HERE}/fixtures"
# ── Tier 1 ─────────────────────────────────────────────────────────────────
# Bad metrics JSON should fail (exit code 1, masked by `if`).
if python3 "${SCRIPT_DIR}/check_tier1.py" "${FIXTURES}/bad_tier1.json" --warmup-end 15; then
echo "FAIL: tier1 check accepted bad metrics"
exit 1
fi
# Known-good metrics JSON should pass (exit code 0).
python3 "${SCRIPT_DIR}/check_tier1.py" "${FIXTURES}/good_tier1.json" --warmup-end 15
# ── Tier 2 ─────────────────────────────────────────────────────────────────
# Bad fixture lacks val_trades_per_bar / val_active_frac / dir-entropy entirely
# → tier2 must FAIL with the missing-metric explanation.
if python3 "${SCRIPT_DIR}/check_tier2.py" "${FIXTURES}/bad_tier1.json" --warmup-end 15; then
echo "FAIL: tier2 check accepted aggregate without behavioural metrics"
exit 1
fi
# ── Tier 3 ─────────────────────────────────────────────────────────────────
# Bad fixture lacks val_sharpe_annualised / val_win_rate / val_profit_factor
# entirely → tier3 must FAIL with the missing-metric explanation.
if python3 "${SCRIPT_DIR}/check_tier3.py" "${FIXTURES}/bad_tier1.json" --warmup-end 15; then
echo "FAIL: tier3 check accepted aggregate without profitability metrics"
exit 1
fi
# ── all-tiers wrapper ──────────────────────────────────────────────────────
# Wrapper must reject any aggregate that fails any tier (uses bad fixture).
if python3 "${SCRIPT_DIR}/check_all_tiers.py" "${FIXTURES}/bad_tier1.json" --warmup-end 15; then
echo "FAIL: check_all_tiers.py accepted aggregate that fails tier1+2+3"
exit 1
fi
echo "PASS"