Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:
Phase 2 PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
in rl_per_tree_rebuild.cu.
Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
(0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).
Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
(1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
— all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
+ new sibling rl_iqn_advance_prng_state launched on same stream
(kernel-launch ordering = grid-wide barrier).
(2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
falling back to time+thread-id RNG. Stayed dormant until first done
event activated non-sentinel labels and divergent weights flowed via
grad_h_t_outcome into encoder gradient. Fix: add seed param + install
scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.
Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
- All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
- eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
- diag.jsonl byte-equal modulo elapsed_s
- Eval pnl identical run-A vs run-B at seed 42
Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.
Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).
Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.
Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
- scripts/local-mid-smoke.sh b=128, 2000+500, ~10min on RTX 3050
- scripts/determinism-check.sh runs mid-smoke twice, diffs checksums
- scripts/tier1_5_verdict.py behavioral kill verdict
- AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
- IntegratedTrainer checkpoint save/load (resume from checkpoint)
- 15 Phase 1 checksum leaves in build_diag_value
- Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
for future divergence-chasing — never run in production
Documentation:
- docs/superpowers/specs/2026-06-02-determinism-foundation.md
- docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
- docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
- docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
- Adjacent specs/plans/notes from the analytical chain that surfaced determinism
as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)
Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
537 lines
24 KiB
Python
Executable File
537 lines
24 KiB
Python
Executable File
#!/usr/bin/env python3
|
||
"""Tier 1.5 mid-smoke behavioral verdict.
|
||
|
||
Reads `diag.jsonl` + `eval_diag.jsonl` + `eval_summary.json` from the
|
||
mid-smoke output directory and computes 7 behavioral signals + 1
|
||
per-step/eval_summary consistency check. Emits OK / KILL_<reason> /
|
||
WARN_<reason>.
|
||
|
||
Spec: docs/superpowers/specs/2026-06-02-fast-dev-cycle.md §3.3, §3.6
|
||
Linked: pearl_grwwh_eval_catastrophic_collapse (per-step vs eval_summary gap)
|
||
|
||
Per `feedback_kill_runs_on_anomaly_quickly`: any KILL_* response means
|
||
DO NOT submit cluster smoke. Fix locally first.
|
||
|
||
Usage:
|
||
python3 scripts/tier1_5_verdict.py /tmp/foxhunt-mid-smoke
|
||
python3 scripts/tier1_5_verdict.py /tmp/foxhunt-mid-smoke --quiet
|
||
python3 scripts/tier1_5_verdict.py /tmp/foxhunt-mid-smoke --json
|
||
|
||
Exit codes:
|
||
0 OK (all signals within tolerance)
|
||
1 KILL (one or more behavioral red flags)
|
||
2 Inputs missing / malformed (cannot verdict)
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import argparse
|
||
import json
|
||
import math
|
||
import statistics
|
||
import sys
|
||
from pathlib import Path
|
||
from typing import Any
|
||
|
||
|
||
# ── Verdict thresholds (per spec §3.3 table) ──────────────────────────
|
||
# Hard floors/ceilings are KILL bounds (catastrophic).
|
||
# Soft targets are reported but not gating (informational).
|
||
THRESHOLDS = {
|
||
# action_entropy final: log(11) = 2.397; target [0.5×log(11), 0.85×log(11)]
|
||
# KILL if < 0.6 (policy collapsed) or > 2.3 (essentially random).
|
||
"action_entropy_kill_low": 0.6,
|
||
"action_entropy_kill_high": 2.3,
|
||
"action_entropy_target_low": 1.20, # 0.5 × log(11)
|
||
"action_entropy_target_high": 2.04, # 0.85 × log(11)
|
||
# q_pi_agree_ema final: ≥ 0.6 target, KILL if < 0.3 (Q/π decoupled)
|
||
"q_pi_agree_kill": 0.3,
|
||
"q_pi_agree_target": 0.6,
|
||
# Pearson(rewards.sum, Δrealized_pnl) ≥ 0.5 target, KILL < 0.3
|
||
"pearson_kill": 0.3,
|
||
"pearson_target": 0.5,
|
||
# popart.sigma CV (stddev/mean): KILL > 0.5 (wild oscillation)
|
||
"popart_sigma_cv_kill": 0.5,
|
||
# win_rate (train) ≥ 0.25 target, KILL < 0.15
|
||
"wr_train_kill": 0.15,
|
||
"wr_train_target": 0.25,
|
||
# win_rate (eval) ≥ 0.20 target, KILL < 0.15
|
||
"wr_eval_kill": 0.15,
|
||
"wr_eval_target": 0.20,
|
||
# Eval pnl > -$1M target, KILL < -$3M (catastrophic)
|
||
"eval_pnl_kill": -3_000_000.0,
|
||
"eval_pnl_target": -1_000_000.0,
|
||
# Per-step vs eval_summary discrepancy: > 5% gap → WARN (not KILL)
|
||
"consistency_warn_frac": 0.05,
|
||
# Min steps required for a meaningful verdict.
|
||
"min_train_rows": 500,
|
||
"min_eval_rows": 100,
|
||
# Hold-step trend: linear regression slope over last N rows must be > 0.
|
||
# KILL if slope is significantly negative.
|
||
"hold_trend_window": 1500,
|
||
"hold_trend_slope_kill": -0.01, # tolerate small decline as noise
|
||
}
|
||
|
||
|
||
def parse_args() -> argparse.Namespace:
|
||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||
p.add_argument("out_dir", type=Path, help="Mid-smoke output directory (--out from local-mid-smoke.sh)")
|
||
p.add_argument("--quiet", action="store_true", help="Suppress per-signal explanations")
|
||
p.add_argument("--json", action="store_true", help="Emit machine-readable JSON verdict")
|
||
return p.parse_args()
|
||
|
||
|
||
def load_jsonl(path: Path, *, max_rows: int | None = None) -> list[dict[str, Any]]:
|
||
"""Stream-parse a JSONL file; return list of decoded records."""
|
||
rows: list[dict[str, Any]] = []
|
||
with path.open("r") as f:
|
||
for line in f:
|
||
line = line.strip()
|
||
if not line:
|
||
continue
|
||
try:
|
||
rows.append(json.loads(line))
|
||
except json.JSONDecodeError as exc:
|
||
# Stop at first malformed line — typically means truncated run.
|
||
print(f"WARN: malformed JSON at line {len(rows) + 1} of {path}: {exc}", file=sys.stderr)
|
||
break
|
||
if max_rows is not None and len(rows) >= max_rows:
|
||
break
|
||
return rows
|
||
|
||
|
||
def get_nested(obj: Any, dotted: str, default: Any = None) -> Any:
|
||
"""Dotted-path lookup: get_nested({'a':{'b':1}}, 'a.b') → 1."""
|
||
cur = obj
|
||
for key in dotted.split("."):
|
||
if not isinstance(cur, dict) or key not in cur:
|
||
return default
|
||
cur = cur[key]
|
||
return cur
|
||
|
||
|
||
def pearson(xs: list[float], ys: list[float]) -> float | None:
|
||
"""Return Pearson correlation, or None if input is degenerate."""
|
||
n = len(xs)
|
||
if n != len(ys) or n < 3:
|
||
return None
|
||
mx = statistics.fmean(xs)
|
||
my = statistics.fmean(ys)
|
||
sx2 = sum((x - mx) ** 2 for x in xs)
|
||
sy2 = sum((y - my) ** 2 for y in ys)
|
||
if sx2 == 0.0 or sy2 == 0.0:
|
||
return None
|
||
sxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
|
||
return sxy / math.sqrt(sx2 * sy2)
|
||
|
||
|
||
def linreg_slope(xs: list[float], ys: list[float]) -> float | None:
|
||
"""OLS slope of y over x; None if degenerate."""
|
||
n = len(xs)
|
||
if n != len(ys) or n < 3:
|
||
return None
|
||
mx = statistics.fmean(xs)
|
||
my = statistics.fmean(ys)
|
||
num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
|
||
den = sum((x - mx) ** 2 for x in xs)
|
||
if den == 0.0:
|
||
return None
|
||
return num / den
|
||
|
||
|
||
def coefficient_of_variation(xs: list[float]) -> float | None:
|
||
"""stdev / |mean|; None if mean ≈ 0 or fewer than 2 points."""
|
||
if len(xs) < 2:
|
||
return None
|
||
m = statistics.fmean(xs)
|
||
if abs(m) < 1e-12:
|
||
return None
|
||
sd = statistics.stdev(xs)
|
||
return sd / abs(m)
|
||
|
||
|
||
# ── Signal computations ───────────────────────────────────────────────
|
||
|
||
|
||
class SignalResult:
|
||
"""A single verdict line — name, value, status, explanation."""
|
||
|
||
__slots__ = ("name", "value", "status", "explain")
|
||
|
||
STATUS_OK = "OK"
|
||
STATUS_KILL = "KILL"
|
||
STATUS_WARN = "WARN"
|
||
STATUS_SKIP = "SKIP"
|
||
|
||
def __init__(self, name: str, value: Any, status: str, explain: str) -> None:
|
||
self.name = name
|
||
self.value = value
|
||
self.status = status
|
||
self.explain = explain
|
||
|
||
def to_dict(self) -> dict[str, Any]:
|
||
return {
|
||
"name": self.name,
|
||
"value": self.value,
|
||
"status": self.status,
|
||
"explain": self.explain,
|
||
}
|
||
|
||
|
||
def signal_action_entropy(train_rows: list[dict[str, Any]]) -> SignalResult:
|
||
"""Final action_entropy: KILL if collapsed (<0.6) or random (>2.3)."""
|
||
val = get_nested(train_rows[-1], "action_entropy")
|
||
if val is None or not isinstance(val, (int, float)):
|
||
return SignalResult("action_entropy", None, SignalResult.STATUS_SKIP, "field missing from final train row")
|
||
if val < THRESHOLDS["action_entropy_kill_low"]:
|
||
return SignalResult("action_entropy", val, SignalResult.STATUS_KILL,
|
||
f"{val:.3f} < {THRESHOLDS['action_entropy_kill_low']} — policy collapsed to δ-function")
|
||
if val > THRESHOLDS["action_entropy_kill_high"]:
|
||
return SignalResult("action_entropy", val, SignalResult.STATUS_KILL,
|
||
f"{val:.3f} > {THRESHOLDS['action_entropy_kill_high']} — policy essentially uniform/random")
|
||
if val < THRESHOLDS["action_entropy_target_low"]:
|
||
return SignalResult("action_entropy", val, SignalResult.STATUS_WARN,
|
||
f"{val:.3f} below target [{THRESHOLDS['action_entropy_target_low']:.2f}, {THRESHOLDS['action_entropy_target_high']:.2f}]")
|
||
if val > THRESHOLDS["action_entropy_target_high"]:
|
||
return SignalResult("action_entropy", val, SignalResult.STATUS_WARN,
|
||
f"{val:.3f} above target [{THRESHOLDS['action_entropy_target_low']:.2f}, {THRESHOLDS['action_entropy_target_high']:.2f}]")
|
||
return SignalResult("action_entropy", val, SignalResult.STATUS_OK,
|
||
f"{val:.3f} within target [{THRESHOLDS['action_entropy_target_low']:.2f}, {THRESHOLDS['action_entropy_target_high']:.2f}]")
|
||
|
||
|
||
def signal_q_pi_agree(train_rows: list[dict[str, Any]]) -> SignalResult:
|
||
"""Final q_pi_agree_ema: KILL if decoupled (<0.3)."""
|
||
val = get_nested(train_rows[-1], "q_pi_agree_ema")
|
||
if val is None or not isinstance(val, (int, float)):
|
||
return SignalResult("q_pi_agree_ema", None, SignalResult.STATUS_SKIP, "field missing from final train row")
|
||
if val < THRESHOLDS["q_pi_agree_kill"]:
|
||
return SignalResult("q_pi_agree_ema", val, SignalResult.STATUS_KILL,
|
||
f"{val:.3f} < {THRESHOLDS['q_pi_agree_kill']} — Q and π decoupled (π is dead weight)")
|
||
if val < THRESHOLDS["q_pi_agree_target"]:
|
||
return SignalResult("q_pi_agree_ema", val, SignalResult.STATUS_WARN,
|
||
f"{val:.3f} below target ≥ {THRESHOLDS['q_pi_agree_target']}")
|
||
return SignalResult("q_pi_agree_ema", val, SignalResult.STATUS_OK,
|
||
f"{val:.3f} ≥ {THRESHOLDS['q_pi_agree_target']}")
|
||
|
||
|
||
def signal_reward_pnl_pearson(train_rows: list[dict[str, Any]]) -> SignalResult:
|
||
"""Pearson(rewards.sum, Δrealized_pnl): KILL if anti-aligned (<0.3).
|
||
|
||
Per `pearl_reward_signal_anti_aligned_with_pnl` (2026-06-01): the
|
||
decisive bug behind 64 negative-eval commits — reward gradient
|
||
systematically anti-aligned with pnl direction.
|
||
"""
|
||
rewards: list[float] = []
|
||
dpnls: list[float] = []
|
||
prev_pnl: float | None = None
|
||
for row in train_rows:
|
||
rs = get_nested(row, "rewards.sum")
|
||
cur_pnl = get_nested(row, "trading.realized_pnl_cum_usd")
|
||
if rs is None or cur_pnl is None:
|
||
continue
|
||
if not isinstance(rs, (int, float)) or not isinstance(cur_pnl, (int, float)):
|
||
continue
|
||
if prev_pnl is not None:
|
||
rewards.append(float(rs))
|
||
dpnls.append(float(cur_pnl) - float(prev_pnl))
|
||
prev_pnl = float(cur_pnl)
|
||
p = pearson(rewards, dpnls)
|
||
if p is None:
|
||
return SignalResult("pearson(rewards.sum, Δpnl)", None, SignalResult.STATUS_SKIP,
|
||
f"degenerate input (n={len(rewards)})")
|
||
if p < THRESHOLDS["pearson_kill"]:
|
||
return SignalResult("pearson(rewards.sum, Δpnl)", p, SignalResult.STATUS_KILL,
|
||
f"{p:.3f} < {THRESHOLDS['pearson_kill']} — reward gradient anti-aligned with pnl (n={len(rewards)})")
|
||
if p < THRESHOLDS["pearson_target"]:
|
||
return SignalResult("pearson(rewards.sum, Δpnl)", p, SignalResult.STATUS_WARN,
|
||
f"{p:.3f} below target ≥ {THRESHOLDS['pearson_target']} (n={len(rewards)})")
|
||
return SignalResult("pearson(rewards.sum, Δpnl)", p, SignalResult.STATUS_OK,
|
||
f"{p:.3f} ≥ {THRESHOLDS['pearson_target']} (n={len(rewards)})")
|
||
|
||
|
||
def signal_hold_trend(train_rows: list[dict[str, Any]]) -> SignalResult:
|
||
"""avg_hold_steps trend over last N rows: KILL if significantly decreasing.
|
||
|
||
The surfer pattern (wave-timescale edge per `pearl_edge_lives_at_wave_timescale_not_tick`)
|
||
requires hold-steps growing through training.
|
||
"""
|
||
window = THRESHOLDS["hold_trend_window"]
|
||
tail = train_rows[-window:] if len(train_rows) > window else train_rows
|
||
xs: list[float] = []
|
||
ys: list[float] = []
|
||
for row in tail:
|
||
step = get_nested(row, "step")
|
||
hold = get_nested(row, "trading.avg_hold_steps")
|
||
if step is None or hold is None:
|
||
continue
|
||
if not isinstance(step, (int, float)) or not isinstance(hold, (int, float)):
|
||
continue
|
||
if math.isnan(float(hold)) or math.isinf(float(hold)):
|
||
continue
|
||
xs.append(float(step))
|
||
ys.append(float(hold))
|
||
if len(xs) < 50:
|
||
return SignalResult("avg_hold_steps trend", None, SignalResult.STATUS_SKIP,
|
||
f"only {len(xs)} usable points in trend window")
|
||
slope = linreg_slope(xs, ys)
|
||
if slope is None:
|
||
return SignalResult("avg_hold_steps trend", None, SignalResult.STATUS_SKIP, "degenerate regression")
|
||
if slope < THRESHOLDS["hold_trend_slope_kill"]:
|
||
return SignalResult("avg_hold_steps trend", slope, SignalResult.STATUS_KILL,
|
||
f"slope {slope:.5f}/step < {THRESHOLDS['hold_trend_slope_kill']} — hold-steps decreasing, surfer pattern absent")
|
||
if slope <= 0.0:
|
||
return SignalResult("avg_hold_steps trend", slope, SignalResult.STATUS_WARN,
|
||
f"slope {slope:.5f}/step — hold-steps not growing")
|
||
return SignalResult("avg_hold_steps trend", slope, SignalResult.STATUS_OK,
|
||
f"slope +{slope:.5f}/step over last {len(xs)} rows")
|
||
|
||
|
||
def signal_popart_sigma_cv(train_rows: list[dict[str, Any]]) -> SignalResult:
|
||
"""popart.sigma coefficient of variation: KILL if wild oscillation (CV > 0.5)."""
|
||
sigmas: list[float] = []
|
||
for row in train_rows:
|
||
s = get_nested(row, "popart.sigma")
|
||
if s is None or not isinstance(s, (int, float)):
|
||
continue
|
||
if math.isnan(float(s)) or math.isinf(float(s)):
|
||
continue
|
||
sigmas.append(float(s))
|
||
if len(sigmas) < 100:
|
||
return SignalResult("popart.sigma CV", None, SignalResult.STATUS_SKIP,
|
||
f"only {len(sigmas)} usable samples")
|
||
cv = coefficient_of_variation(sigmas)
|
||
if cv is None:
|
||
return SignalResult("popart.sigma CV", None, SignalResult.STATUS_SKIP,
|
||
"mean ≈ 0 — undefined CV")
|
||
if cv > THRESHOLDS["popart_sigma_cv_kill"]:
|
||
return SignalResult("popart.sigma CV", cv, SignalResult.STATUS_KILL,
|
||
f"{cv:.3f} > {THRESHOLDS['popart_sigma_cv_kill']} — controller wildly oscillating")
|
||
return SignalResult("popart.sigma CV", cv, SignalResult.STATUS_OK,
|
||
f"{cv:.3f} ≤ {THRESHOLDS['popart_sigma_cv_kill']}")
|
||
|
||
|
||
def signal_wr_train(train_rows: list[dict[str, Any]]) -> SignalResult:
|
||
"""Final win_rate_ema (train): KILL if random/worse (<0.15)."""
|
||
# Prefer kelly.win_rate_ema (EMA across all trades) over trading.win_rate
|
||
# which is a point measure of last batch.
|
||
val = get_nested(train_rows[-1], "isv_config.kelly.win_rate_ema")
|
||
if val is None:
|
||
val = get_nested(train_rows[-1], "trading.win_rate")
|
||
source = "trading.win_rate"
|
||
else:
|
||
source = "isv_config.kelly.win_rate_ema"
|
||
if val is None or not isinstance(val, (int, float)):
|
||
return SignalResult(f"wr_train ({source})", None, SignalResult.STATUS_SKIP,
|
||
"field missing from final train row")
|
||
if val < THRESHOLDS["wr_train_kill"]:
|
||
return SignalResult(f"wr_train ({source})", val, SignalResult.STATUS_KILL,
|
||
f"{val:.3f} < {THRESHOLDS['wr_train_kill']} — random or worse")
|
||
if val < THRESHOLDS["wr_train_target"]:
|
||
return SignalResult(f"wr_train ({source})", val, SignalResult.STATUS_WARN,
|
||
f"{val:.3f} below target ≥ {THRESHOLDS['wr_train_target']}")
|
||
return SignalResult(f"wr_train ({source})", val, SignalResult.STATUS_OK,
|
||
f"{val:.3f} ≥ {THRESHOLDS['wr_train_target']}")
|
||
|
||
|
||
def signal_wr_eval(eval_rows: list[dict[str, Any]]) -> SignalResult:
|
||
"""Final win_rate_ema (eval): KILL if < 0.15."""
|
||
if not eval_rows:
|
||
return SignalResult("wr_eval", None, SignalResult.STATUS_SKIP, "no eval rows")
|
||
val = get_nested(eval_rows[-1], "isv_config.kelly.win_rate_ema")
|
||
if val is None:
|
||
val = get_nested(eval_rows[-1], "trading.win_rate")
|
||
source = "trading.win_rate"
|
||
else:
|
||
source = "isv_config.kelly.win_rate_ema"
|
||
if val is None or not isinstance(val, (int, float)):
|
||
return SignalResult(f"wr_eval ({source})", None, SignalResult.STATUS_SKIP,
|
||
"field missing from final eval row")
|
||
if val < THRESHOLDS["wr_eval_kill"]:
|
||
return SignalResult(f"wr_eval ({source})", val, SignalResult.STATUS_KILL,
|
||
f"{val:.3f} < {THRESHOLDS['wr_eval_kill']} — random or worse")
|
||
if val < THRESHOLDS["wr_eval_target"]:
|
||
return SignalResult(f"wr_eval ({source})", val, SignalResult.STATUS_WARN,
|
||
f"{val:.3f} below target ≥ {THRESHOLDS['wr_eval_target']}")
|
||
return SignalResult(f"wr_eval ({source})", val, SignalResult.STATUS_OK,
|
||
f"{val:.3f} ≥ {THRESHOLDS['wr_eval_target']}")
|
||
|
||
|
||
def signal_eval_pnl(eval_summary: dict[str, Any] | None,
|
||
eval_rows: list[dict[str, Any]]) -> SignalResult:
|
||
"""Final eval pnl: KILL if catastrophic (< -$3M).
|
||
|
||
Prefer eval_summary.total_pnl_usd (authoritative) over per-step
|
||
realized_pnl_cum_usd (per `pearl_grwwh_eval_catastrophic_collapse`
|
||
these can disagree by hundreds of millions).
|
||
"""
|
||
val: float | None = None
|
||
source = "none"
|
||
if eval_summary is not None and "total_pnl_usd" in eval_summary:
|
||
val = float(eval_summary["total_pnl_usd"])
|
||
source = "eval_summary.total_pnl_usd"
|
||
elif eval_rows:
|
||
last = get_nested(eval_rows[-1], "trading.realized_pnl_cum_usd")
|
||
if isinstance(last, (int, float)):
|
||
val = float(last)
|
||
source = "eval_diag last realized_pnl_cum_usd"
|
||
if val is None:
|
||
return SignalResult("eval_pnl", None, SignalResult.STATUS_SKIP,
|
||
"neither eval_summary.json nor eval_diag.jsonl has pnl")
|
||
if val < THRESHOLDS["eval_pnl_kill"]:
|
||
return SignalResult(f"eval_pnl ({source})", val, SignalResult.STATUS_KILL,
|
||
f"${val:,.0f} < ${THRESHOLDS['eval_pnl_kill']:,.0f} — catastrophic")
|
||
if val < THRESHOLDS["eval_pnl_target"]:
|
||
return SignalResult(f"eval_pnl ({source})", val, SignalResult.STATUS_WARN,
|
||
f"${val:,.0f} below target > ${THRESHOLDS['eval_pnl_target']:,.0f}")
|
||
return SignalResult(f"eval_pnl ({source})", val, SignalResult.STATUS_OK,
|
||
f"${val:,.0f} > ${THRESHOLDS['eval_pnl_target']:,.0f}")
|
||
|
||
|
||
def signal_consistency(eval_rows: list[dict[str, Any]],
|
||
eval_summary: dict[str, Any] | None) -> SignalResult:
|
||
"""Per-step vs eval_summary consistency check (§3.6).
|
||
|
||
WARN (not KILL): the per-step diag's realized_pnl_cum_usd and
|
||
eval_summary.json's total_pnl_usd should match within 5%. A divergence
|
||
indicates the per-step diag is misleading (per `pearl_grwwh_eval_catastrophic_collapse`).
|
||
"""
|
||
if eval_summary is None:
|
||
return SignalResult("per-step vs eval_summary", None, SignalResult.STATUS_SKIP,
|
||
"eval_summary.json missing")
|
||
if "total_pnl_usd" not in eval_summary:
|
||
return SignalResult("per-step vs eval_summary", None, SignalResult.STATUS_SKIP,
|
||
"eval_summary.json has no total_pnl_usd")
|
||
if not eval_rows:
|
||
return SignalResult("per-step vs eval_summary", None, SignalResult.STATUS_SKIP,
|
||
"no eval rows")
|
||
per_step = get_nested(eval_rows[-1], "trading.realized_pnl_cum_usd")
|
||
if per_step is None or not isinstance(per_step, (int, float)):
|
||
return SignalResult("per-step vs eval_summary", None, SignalResult.STATUS_SKIP,
|
||
"last eval row missing realized_pnl_cum_usd")
|
||
summary = float(eval_summary["total_pnl_usd"])
|
||
per_step = float(per_step)
|
||
diff = abs(per_step - summary)
|
||
denom = max(abs(per_step), abs(summary), 1.0)
|
||
frac = diff / denom
|
||
detail = f"per_step=${per_step:,.0f} | eval_summary=${summary:,.0f} | gap=${diff:,.0f} ({frac*100:.2f}%)"
|
||
if frac > THRESHOLDS["consistency_warn_frac"]:
|
||
return SignalResult("per-step vs eval_summary", frac, SignalResult.STATUS_WARN,
|
||
f"{detail} > {THRESHOLDS['consistency_warn_frac']*100:.0f}% — accounting axis discrepancy")
|
||
return SignalResult("per-step vs eval_summary", frac, SignalResult.STATUS_OK, detail)
|
||
|
||
|
||
# ── Driver ────────────────────────────────────────────────────────────
|
||
|
||
|
||
def run_verdict(out_dir: Path) -> tuple[str, list[SignalResult], dict[str, Any]]:
|
||
"""Compute all signals; return overall verdict + per-signal list + meta."""
|
||
diag_path = out_dir / "diag.jsonl"
|
||
eval_diag_path = out_dir / "eval_diag.jsonl"
|
||
eval_summary_path = out_dir / "eval_summary.json"
|
||
|
||
meta: dict[str, Any] = {
|
||
"out_dir": str(out_dir),
|
||
"diag_path": str(diag_path),
|
||
"eval_diag_path": str(eval_diag_path),
|
||
"eval_summary_path": str(eval_summary_path),
|
||
}
|
||
|
||
if not diag_path.exists():
|
||
print(f"ERROR: {diag_path} not found", file=sys.stderr)
|
||
return "INPUT_MISSING", [], meta
|
||
|
||
train_rows = load_jsonl(diag_path)
|
||
meta["n_train_rows"] = len(train_rows)
|
||
if len(train_rows) < THRESHOLDS["min_train_rows"]:
|
||
print(f"ERROR: only {len(train_rows)} train rows in {diag_path} (need ≥ {THRESHOLDS['min_train_rows']})", file=sys.stderr)
|
||
return "INPUT_INSUFFICIENT", [], meta
|
||
|
||
eval_rows = load_jsonl(eval_diag_path) if eval_diag_path.exists() else []
|
||
meta["n_eval_rows"] = len(eval_rows)
|
||
|
||
eval_summary: dict[str, Any] | None = None
|
||
if eval_summary_path.exists():
|
||
try:
|
||
with eval_summary_path.open("r") as f:
|
||
eval_summary = json.load(f)
|
||
except (OSError, json.JSONDecodeError) as exc:
|
||
print(f"WARN: failed to read {eval_summary_path}: {exc}", file=sys.stderr)
|
||
eval_summary = None
|
||
meta["eval_summary_present"] = eval_summary is not None
|
||
|
||
signals: list[SignalResult] = [
|
||
signal_action_entropy(train_rows),
|
||
signal_q_pi_agree(train_rows),
|
||
signal_reward_pnl_pearson(train_rows),
|
||
signal_hold_trend(train_rows),
|
||
signal_popart_sigma_cv(train_rows),
|
||
signal_wr_train(train_rows),
|
||
signal_wr_eval(eval_rows),
|
||
signal_eval_pnl(eval_summary, eval_rows),
|
||
signal_consistency(eval_rows, eval_summary),
|
||
]
|
||
|
||
kill_reasons = [s.name for s in signals if s.status == SignalResult.STATUS_KILL]
|
||
if kill_reasons:
|
||
overall = "KILL_" + "+".join(s.replace(" ", "_") for s in kill_reasons)
|
||
else:
|
||
overall = "OK"
|
||
return overall, signals, meta
|
||
|
||
|
||
def format_text(overall: str, signals: list[SignalResult], meta: dict[str, Any]) -> str:
|
||
lines: list[str] = []
|
||
lines.append("=" * 72)
|
||
lines.append(f"Tier 1.5 verdict: {overall}")
|
||
lines.append("=" * 72)
|
||
lines.append(f" out_dir: {meta['out_dir']}")
|
||
lines.append(f" train rows: {meta.get('n_train_rows', 0)}")
|
||
lines.append(f" eval rows: {meta.get('n_eval_rows', 0)}")
|
||
lines.append(f" eval_summary: {'present' if meta.get('eval_summary_present') else 'missing'}")
|
||
lines.append("")
|
||
lines.append(f"{'STATUS':<6} {'SIGNAL':<42} EXPLANATION")
|
||
lines.append(f"{'-'*6} {'-'*42} {'-'*64}")
|
||
for s in signals:
|
||
marker = {
|
||
SignalResult.STATUS_OK: "OK",
|
||
SignalResult.STATUS_WARN: "WARN",
|
||
SignalResult.STATUS_KILL: "KILL",
|
||
SignalResult.STATUS_SKIP: "SKIP",
|
||
}[s.status]
|
||
lines.append(f"{marker:<6} {s.name:<42} {s.explain}")
|
||
lines.append("")
|
||
if overall == "OK":
|
||
lines.append("[OK] No behavioral red flags. Cluster submit permitted.")
|
||
elif overall.startswith("KILL_"):
|
||
lines.append(f"[KILL] {overall} — DO NOT submit cluster smoke. Fix locally first.")
|
||
else:
|
||
lines.append(f"[{overall}] Inputs incomplete — re-run mid-smoke to generate full diag.")
|
||
return "\n".join(lines)
|
||
|
||
|
||
def main() -> int:
|
||
args = parse_args()
|
||
overall, signals, meta = run_verdict(args.out_dir)
|
||
|
||
if args.json:
|
||
payload = {
|
||
"overall": overall,
|
||
"signals": [s.to_dict() for s in signals],
|
||
"meta": meta,
|
||
}
|
||
print(json.dumps(payload, indent=2))
|
||
elif args.quiet:
|
||
print(overall)
|
||
else:
|
||
print(format_text(overall, signals, meta))
|
||
|
||
if overall == "OK":
|
||
return 0
|
||
if overall.startswith("KILL_"):
|
||
return 1
|
||
return 2
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|