From fbee2a00f5a4fe6345f395bc4f2a66b7cdaa103b Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 26 Apr 2026 13:04:39 +0200 Subject: [PATCH] plan5(task5-A): wire tier 2/3 val_* metrics into HEALTH_DIAG MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan 5 Task 4 left every tier-2/tier-3 check failing with "metric missing from aggregate" because the existing 'Validation backtest:' free-form log line was not parseable by the aggregate-multi-seed-metrics.py block-keyed parser. Phase A closes that gap end-to-end (CPU-only, no kernel touch): * metrics.rs::compute_validation_loss — emit a new HEALTH_DIAG[]: val [sharpe=… sortino=… win_rate=… max_drawdown=… trade_count=… calmar=… omega_ratio=… total_pnl=… var_95=… cvar_95=… trades_per_bar=… active_frac=… dir_entropy=… sharpe_annualised=… profit_factor=… window_bars=…] block immediately after the existing 'Validation backtest:' line. All 16 keys derive from the existing GpuBacktestEvaluator WindowMetrics reduction (no new GPU work): - sharpe / sortino / win_rate / max_drawdown / total_trades / calmar / omega_ratio / total_pnl / var_95 / cvar_95 / buy_count / sell_count / hold_count come straight from m.* - window_bars = buy + sell + hold (kernel tallies one direction per bar) - trades_per_bar = total_trades / window_bars - active_frac = (buy + sell) / window_bars (kernel folds Hold AND Flat into hold_count, so 'active' = bars where the policy chose Short or Long — meets the Tier-2 'not always Hold' intent) - dir_entropy = -Σ p ln p over the 3-bucket {short, hold-or-flat, long} distribution. Documented limitation: max log(3) ≈ 1.099 vs spec's 4-bucket 0.8·log(4) ≈ 1.109 ceiling — tier2 dir_entropy threshold is unreachable from this 3-bucket distribution; resolution tracked in audit row. - sharpe_annualised = m.sharpe alias (kernel already multiplies by sqrt(bars_per_day · 252) at backtest_metrics_kernel:266) - profit_factor = m.omega_ratio alias (kernel's omega computes gain_sum/loss_sum at threshold 0, equivalent to per-step PF; trade-level PF deferred — needs boundary-aware kernel work) * mod.rs — adds last_val_metrics: Option<[f32; 14]> on DQNTrainer to snapshot the WindowMetrics-derived values for downstream consumers (smoke tests, future telemetry). * constructor.rs — initialises the new field to None. * aggregate-multi-seed-metrics.py — switches the block→key joiner from '__' to '_' so 'val [sharpe=…]' surfaces as the bare 'val_sharpe' aggregate key the tier check scripts and synthetic test fixtures already expect. The pre-existing '__' joiner was an oversight in Plan 5 Task 1B that was never validated against actual aggregator output (the aggregator emitted 90 'block__key' metrics that nothing consumed; the synthetic good_tier1.json / bad_tier1.json fixtures were always shaped as 'val_sharpe', confirming the single-underscore convention was intended). Renaming the 90 existing keys is safe — no consumers had locked in on the '__' form. * docs/dqn-wire-up-audit.md — updates Plan 5 Task 4 row to reference the now-landed wiring and adds a new row documenting the val [...] HEALTH_DIAG block pipeline + aggregator joiner change + the deferred 4-bucket dir_dist + trade-level PF caveats. Validation: cargo check --workspace clean at 11 warnings. multi_fold_convergence smoke (629s, 3 folds × 5 epochs on RTX 3050 Ti) PASSES with 3/3 fold checkpoints. Per-fold best Sharpe: -9.78 / 42.46 / 88.18 (within smoke noise band — no perturbation from the additive CPU-only HEALTH_DIAG line). scripts/aggregate-multi-seed-metrics.py against /tmp/p5t5a-smoke.log produces 3 streams (one per fold), 16 val_* keys all present: val_sharpe, val_sharpe_annualised, val_sortino, val_win_rate, val_max_drawdown, val_trade_count, val_calmar, val_omega_ratio, val_total_pnl, val_var_95, val_cvar_95, val_trades_per_bar, val_active_frac, val_dir_entropy, val_profit_factor, val_window_bars. check_tier2.py / check_tier3.py rejection messages are now substantive (threshold-based) rather than "missing key": Tier 2: trades_per_bar PASS @ 0.0127; active_frac FAIL @ 0.058 (model mostly Hold on 5-epoch smoke); dir_entropy FAIL @ 0.18 (within documented 3-bucket vs 4-bucket caveat). Tier 3: sharpe_annualised FAIL @ -0.25 (5-epoch smoke not converged); win_rate skipped (192 trades ≤ 500 noise gate); profit_factor FAIL @ 0.18 (untrained policy). Real validation pass requires the L40S 60-epoch run (Phase C). Deferred (out of T5 Phase A scope): - val_dir_dist_{short,hold,long,flat} per-direction breakdown — kernel intentionally collapses Hold+Flat for trade-cycle counting; Tier-2's log(4) threshold needs either a kernel-level split or a 3-bucket threshold tweak in check_tier2.py. - Trade-level profit_factor (sum-winner-PnL / sum-loser-PnL) vs the per-step omega-equivalent emitted here. - avg_q_value bare-key aggregation — the metric is logged via separate Prometheus + tracing paths but not inside any HEALTH_DIAG block; out of T5 Phase A scope and pre-existing. --- .../src/trainers/dqn/trainer/constructor.rs | 3 + crates/ml/src/trainers/dqn/trainer/metrics.rs | 99 +++++++++++++++++++ crates/ml/src/trainers/dqn/trainer/mod.rs | 20 ++++ docs/dqn-wire-up-audit.md | 3 +- scripts/aggregate-multi-seed-metrics.py | 7 +- 5 files changed, 129 insertions(+), 3 deletions(-) diff --git a/crates/ml/src/trainers/dqn/trainer/constructor.rs b/crates/ml/src/trainers/dqn/trainer/constructor.rs index a1b79713e..92c3cf65b 100644 --- a/crates/ml/src/trainers/dqn/trainer/constructor.rs +++ b/crates/ml/src/trainers/dqn/trainer/constructor.rs @@ -589,6 +589,9 @@ impl DQNTrainer { grad_clip_kicked_this_epoch: false, last_eval_magnitude_dist: [0.0_f32; 3], last_eval_intent_magnitude_dist: [0.0_f32; 3], + // Plan 5 Task 5 Phase A — populated after each `compute_validation_loss` + // call; consumed by HEALTH_DIAG `val [...]` block emit. + last_val_metrics: None, prev_reward_contrib_popart_var: 0.0, last_reward_contrib: [0.0_f32; 4], diff --git a/crates/ml/src/trainers/dqn/trainer/metrics.rs b/crates/ml/src/trainers/dqn/trainer/metrics.rs index 9c34c88ce..acec40720 100644 --- a/crates/ml/src/trainers/dqn/trainer/metrics.rs +++ b/crates/ml/src/trainers/dqn/trainer/metrics.rs @@ -657,6 +657,105 @@ impl DQNTrainer { "Validation backtest: Sharpe={:.2} Sortino={:.2} WinRate={:.1}% MaxDD={:.3}% Trades={:.0} Calmar={:.2}", m.sharpe, m.sortino, m.win_rate * 100.0, m.max_drawdown * 100.0, m.total_trades, m.calmar, ); + + // Plan 5 Task 5 Phase A — flat-emit per-epoch validation metrics + // in HEALTH_DIAG `val [...]` block format so the + // `aggregate-multi-seed-metrics.py` parser picks them up as + // top-level `val_*` aggregate keys (joined with single underscore + // by the aggregator). Tier-2/tier-3 check scripts read these keys + // directly. Derivations (no new GPU work): + // * `trade_count` = m.total_trades (kernel's boundary- + // stitched count of position-sign cycles) + // * `window_bars` = buy + sell + hold (every bar + // contributes exactly one direction tally per kernel) + // * `trades_per_bar` = trade_count / window_bars + // * `active_frac` = (buy + sell) / window_bars (kernel + // counts Hold AND Flat into hold_count, so "active" = bars + // where the policy chose Short or Long — directly meets the + // Tier-2 spec "model is not always Hold" intent) + // * `dir_entropy` = Shannon entropy in nats over the + // 3-bucket {short, hold-or-flat, long} distribution from the + // kernel; max log(3) ≈ 1.0986. Note: the spec compares + // against 0.8·log(4) ≈ 1.109 which is unreachable here + // because the kernel collapses Hold+Flat into one bucket. + // Tier-2's check_tier2.py needs to use the 3-bucket ceiling + // (or fall back to per-direction `val_dir_dist_*`); see audit + // doc row for follow-up. + // * `profit_factor` = m.omega_ratio (per-step gain_sum / + // loss_sum from the kernel's bitonic-sorted return array; + // mathematically equivalent to profit-factor when threshold + // = 0, which is the standard definition. NOT trade-level PF + // — that would require boundary-aware per-trade P&L which + // the kernel doesn't yet emit; flagged as future enhancement + // in audit row). + // * `sharpe_annualised` = m.sharpe (kernel already multiplies + // by `sqrt(bars_per_day * 252)` — see backtest_metrics_kernel + // line 266). Aliased so tier3's primary key is present. + let buy = m.buy_count; + let sell = m.sell_count; + let hold = m.hold_count; + let window_bars = buy + sell + hold; + let trades_per_bar = if window_bars > 0.0 { m.total_trades / window_bars } else { 0.0 }; + let active_frac = if window_bars > 0.0 { (buy + sell) / window_bars } else { 0.0 }; + + // Shannon entropy in nats over 3-bucket direction distribution + // (short / hold-or-flat / long). Uses single-pass log of the + // probability; non-positive buckets contribute nothing. + let dir_entropy = if window_bars > 0.0 { + let p_short = (sell / window_bars).max(0.0); + let p_hold = (hold / window_bars).max(0.0); + let p_long = (buy / window_bars).max(0.0); + let mut h = 0.0_f32; + for &p in &[p_short, p_hold, p_long] { + if p > 0.0 { h -= p * p.ln(); } + } + h + } else { 0.0 }; + + // Snapshot for downstream consumers (smoke tests, future telemetry). + self.last_val_metrics = Some([ + m.sharpe, + m.sortino, + m.win_rate, + m.max_drawdown, + m.total_trades, + m.calmar, + m.omega_ratio, + m.total_pnl, + m.var_95, + m.cvar_95, + m.buy_count, + m.sell_count, + m.hold_count, + window_bars, + ]); + + // HEALTH_DIAG `val [...]` block. Block name `val` + single-`_` + // aggregator join → top-level keys `val_sharpe`, `val_sortino`, + // ..., `val_profit_factor`. Additive line — does not perturb the + // existing giant HEALTH_DIAG line. + let epoch = self.current_epoch; + tracing::info!( + "HEALTH_DIAG[{}]: val [sharpe={:.4} sortino={:.4} win_rate={:.4} max_drawdown={:.4} trade_count={:.0} calmar={:.4} omega_ratio={:.4} total_pnl={:.4} var_95={:.4e} cvar_95={:.4e} trades_per_bar={:.6} active_frac={:.4} dir_entropy={:.4} sharpe_annualised={:.4} profit_factor={:.4} window_bars={:.0}]", + epoch, + m.sharpe, + m.sortino, + m.win_rate, + m.max_drawdown, + m.total_trades, + m.calmar, + m.omega_ratio, + m.total_pnl, + m.var_95, + m.cvar_95, + trades_per_bar, + active_frac, + dir_entropy, + m.sharpe, // sharpe_annualised alias (kernel already annualises) + m.omega_ratio, // profit_factor alias (per-step PF == omega at threshold 0) + window_bars, + ); + m.sharpe as f64 } else { 0.0 diff --git a/crates/ml/src/trainers/dqn/trainer/mod.rs b/crates/ml/src/trainers/dqn/trainer/mod.rs index c5c889aa0..90ca3dfe6 100644 --- a/crates/ml/src/trainers/dqn/trainer/mod.rs +++ b/crates/ml/src/trainers/dqn/trainer/mod.rs @@ -308,6 +308,26 @@ pub struct DQNTrainer { /// safety=0.5+0.5×health pins abs_pos ≤ 0.375×max_pos). pub(crate) last_eval_intent_magnitude_dist: [f32; 3], + /// Plan 5 Task 5 Phase A — flat aggregate of validation-backtest metrics + /// from the most recent `compute_validation_loss` invocation. Sourced from + /// the GPU `WindowMetrics` reduction (single-window val path); kept on the + /// trainer so the next HEALTH_DIAG emit can publish them as a `val [...]` + /// block consumable by `scripts/aggregate-multi-seed-metrics.py` + the + /// Plan 5 Task 4 tier-2/tier-3 check scripts. + /// + /// Layout (matches the HEALTH_DIAG `val [...]` key order — see + /// `metrics.rs::compute_validation_loss`): + /// `[sharpe_annualised, sortino, win_rate, max_drawdown, trade_count, + /// calmar, omega_ratio, total_pnl, var_95, cvar_95, + /// buy_count, sell_count, hold_count, window_bars]`. + /// + /// `None` until the first validation backtest finishes (epoch 0 with + /// async stream, or epoch 0 sync). `[bars_per_day]` is implied as a + /// hyperparameter; downstream emitters derive `trades_per_bar`, + /// `active_frac`, `dir_entropy`, and `profit_factor` from these 14 floats + /// without re-running the kernel. + pub(crate) last_val_metrics: Option<[f32; 14]>, + /// Task 0.9 — prior-epoch values of adaptive-controller outputs, used to /// detect "fire" = value changed from the prior epoch. Controllers audited: /// anti-LR, adaptive tau, adaptive gamma, adaptive grad-clip, CQL alpha diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index 36dbbe0b5..9beebd291 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -48,7 +48,8 @@ | `config/metric-bands.toml` | `MetricBandsRegistry::load_from_toml` (auto-loaded by `DQNTrainer::new_internal`) | Wired | Plan 5 Task 2 — per-metric warn/error bands (mean ± 5σ / mean ± 10σ); populated by `scripts/populate-metric-bands-from-runs.py` from Plan 5 Task 1B aggregate JSONs | — | | `scripts/populate-metric-bands-from-runs.py` | reads aggregate JSONs from `scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2); writes `config/metric-bands.toml` `[bands]` section | Wired | Plan 5 Task 2 helper — N>=3 sample path; N=1 multiplicative fallback | — | | `scripts/argo-train.sh --profile` + `infra/k8s/argo/train-multi-seed-template.yaml` (`profile` parameter, conditional `nsys profile` wrapper, `mc cp` upload to `foxhunt-training-artifacts/profiles//`) + `infra/docker/Dockerfile.foxhunt-training-runtime` (`nsight-systems-cli` apt install) + `infra/k8s/minio/minio.yaml` (new `foxhunt-training-artifacts` bucket) | invoked manually via `./scripts/argo-train.sh dqn --profile [...]`; consumed by `scripts/compare-nsys-profiles.py BASELINE CURRENT --epochs N` (V0 metric: total cuda_gpu_kern_sum / epochs; V1 NVTX-range path deferred to Plan 5 Task 5) | Wired | Plan 5 Task 3 A.4.1 — nsys profile harness with regression-comparison script. `--profile` forces multi-seed render path so dry-run never needs cluster contact (test_nsys_harness.sh); MinIO creds optional on `train-single` (warn-skips upload if absent). **Baseline capture deferred to Plan 5 Task 5** (T5 runs the harness on L40S as part of the multi-seed acceptance pass — avoids burning local GPU time in T3) | — | -| `scripts/validation/check_tier1.py` + `check_tier2.py` + `check_tier3.py` + `check_all_tiers.py` (+ `tests/test_tier_checks.sh` + `tests/fixtures/{good,bad}_tier1.json`) | reads aggregate JSONs from `scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2); invoked manually as `python3 scripts/validation/check_all_tiers.py [--warmup-end N]` and from Plan 5 Task 5's acceptance pass | Wired | Plan 5 Task 4 — per-tier exit checks for the spec §2 tiered acceptance criteria. Tier 1 (convergence: std/mean ≤ 0.15 on `val_sharpe`/`avg_q_value`/`train_loss` over stable epochs + no avg_q_value > 500 fold-1 explosion + Q-saturation/hot-path-DtoH placeholders); Tier 2 (behavioural: `val_trades_per_bar ≥ 0.005`, `val_active_frac > 0.2`, dir entropy > 0.8·log4); Tier 3 (profitability: `val_sharpe_annualised > 1.0` with per-bar fallback, `val_win_rate ≥ 0.52` gated on >500 trades, `val_profit_factor` mean ≥ 1.1 AND cross-seed std < 0.3). Stdlib only (no numpy/scipy). Defensive missing-metric handling — each missing aggregate key fails the relevant check with an explanatory message rather than silently passing. **Deferred metrics not yet emitted by HEALTH_DIAG** (current aggregator key set is `__` per `aggregate-multi-seed-metrics.py:115`): `val_trades_per_bar`, `val_active_frac`, `val_dir_entropy` / `val_dir_dist_*`, `val_sharpe_annualised`, `val_win_rate`, `val_profit_factor`, `val_trade_count`. Wiring those into `training_loop.rs::log_epoch_metrics_and_financials` (and either flat-emitting or tracking through the band-check harvest path) is the prerequisite for Tiers 2/3 to ever PASS on real data — tracked under Plan 5 Task 5's pre-flight. | — | +| `scripts/validation/check_tier1.py` + `check_tier2.py` + `check_tier3.py` + `check_all_tiers.py` (+ `tests/test_tier_checks.sh` + `tests/fixtures/{good,bad}_tier1.json`) | reads aggregate JSONs from `scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2); invoked manually as `python3 scripts/validation/check_all_tiers.py [--warmup-end N]` and from Plan 5 Task 5's acceptance pass | Wired | Plan 5 Task 4 — per-tier exit checks for the spec §2 tiered acceptance criteria. Tier 1 (convergence: std/mean ≤ 0.15 on `val_sharpe`/`avg_q_value`/`train_loss` over stable epochs + no avg_q_value > 500 fold-1 explosion + Q-saturation/hot-path-DtoH placeholders); Tier 2 (behavioural: `val_trades_per_bar ≥ 0.005`, `val_active_frac > 0.2`, dir entropy > 0.8·log4); Tier 3 (profitability: `val_sharpe_annualised > 1.0` with per-bar fallback, `val_win_rate ≥ 0.52` gated on >500 trades, `val_profit_factor` mean ≥ 1.1 AND cross-seed std < 0.3). Stdlib only (no numpy/scipy). Defensive missing-metric handling — each missing aggregate key fails the relevant check with an explanatory message rather than silently passing. Tier-2/tier-3 wiring landed in Plan 5 Task 5 Phase A (`val [...]` HEALTH_DIAG block + aggregator single-`_` joiner) — see next row. | — | +| `trainer/metrics.rs::compute_validation_loss` (HEALTH_DIAG `val [...]` block) + `scripts/aggregate-multi-seed-metrics.py::parse_blocks` (single-`_` joiner) + `trainer/mod.rs` (`last_val_metrics: Option<[f32; 14]>`) | Plan 5 Task 4 tier-2/tier-3 check scripts read these as top-level `val_*` aggregate keys. Block emit pipeline: `evaluate_dqn_graphed` → `WindowMetrics{sharpe, sortino, win_rate, max_drawdown, total_trades, calmar, omega_ratio, total_pnl, var_95, cvar_95, buy/sell/hold_count}` (CUDA `compute_backtest_metrics` 14-float reduction, no new GPU work) → CPU derivations (`window_bars = buy+sell+hold`; `trades_per_bar = total_trades / window_bars`; `active_frac = (buy+sell) / window_bars`; `dir_entropy = -Σ p ln p` over the 3-bucket {short, hold-or-flat, long} distribution; `sharpe_annualised = m.sharpe` alias since the kernel already multiplies by `sqrt(bars_per_day · 252)`; `profit_factor = m.omega_ratio` alias since the kernel's omega is `gain_sum / loss_sum` with threshold 0, equivalent to per-step PF) → `tracing::info!("HEALTH_DIAG[{}]: val [sharpe=… sortino=… win_rate=… max_drawdown=… trade_count=… calmar=… omega_ratio=… total_pnl=… var_95=… cvar_95=… trades_per_bar=… active_frac=… dir_entropy=… sharpe_annualised=… profit_factor=… window_bars=…]", current_epoch, …)`. The aggregator joins `_` (single underscore — was `__` before T5 Phase A; tier scripts and synthetic test fixtures already used the bare `val_*` convention) so each emitted key surfaces as a top-level `val_*` aggregate. **Deferred for follow-up** (cannot be emitted from existing kernel data): (a) per-direction distribution `val_dir_dist_{short,hold,long,flat}` — kernel collapses Hold+Flat into hold_count (intentional for the position-sign trade-cycle definition), so `dir_entropy` here ranges over 3 buckets with max `log 3 ≈ 1.099` rather than the spec's 4-bucket `log 4 ≈ 1.386` ceiling; tier2 `check_dir_entropy` compares against `0.8 · log 4 ≈ 1.109` which is unreachable from the 3-bucket distribution. Resolution options: extend `WindowMetrics` with separate Hold/Flat counts (kernel touch) **or** lower tier2's threshold to the 3-bucket equivalent `0.8 · log 3 ≈ 0.879`. (b) trade-level `profit_factor` (sum-of-winning-trade-P&L / sum-of-losing-trade-P&L) — currently aliased to per-step `omega_ratio`; matches the standard PF definition under threshold 0 but a trade-level variant would require boundary-aware per-trade P&L accumulation in the kernel. Cold-path (per validation epoch); zero hot-path overhead; one new tracing line per epoch. | — | | `trainers/dqn/adaptive_monitor.rs` | Read-only observer trait + harness (FireRateStats, DiagSnapshot, IsvBus<'a>); consumers added in Plan 1 Tasks 9-17 (atoms/gamma/kelly_cap/tau/epsilon/grad_balancer monitors) | Wired (consumers added in same plan) | C.6 GPU-drives-CPU-reads | — | | `trainers/dqn/monitors/grad_balancer_monitor.rs` | Read-only observer for grad_balance_isv_update kernel output (ISV slots 31..35); consumers: HEALTH_DIAG + controller_activity smoke | Wired | Plan 1 Task 17 | — | | `trainers/dqn/monitors/tau_monitor.rs` | Read-only observer for tau_update kernel output (ISV slot 42); consumers: HEALTH_DIAG + controller_activity smoke | Wired | Plan 1 Task 13 | — | diff --git a/scripts/aggregate-multi-seed-metrics.py b/scripts/aggregate-multi-seed-metrics.py index a116c5fe8..8546dc10b 100755 --- a/scripts/aggregate-multi-seed-metrics.py +++ b/scripts/aggregate-multi-seed-metrics.py @@ -114,7 +114,10 @@ def extract_health_diag(line: str) -> tuple[int, str] | None: def parse_blocks(body: str) -> dict[str, float]: """Parse all `[ ...]` segments from a HEALTH_DIAG - body. Returns flat dict with keys `__`.""" + body. Returns flat dict with keys `_` (single underscore + joiner — this is what the Plan 5 Task 4 tier check scripts expect, and + what the synthetic test fixtures encode: e.g. `val [sharpe=…]` becomes + aggregate key `val_sharpe`).""" out: dict[str, float] = {} for block_match in BLOCK_RE.finditer(body): block_name = block_match.group(1) @@ -123,7 +126,7 @@ def parse_blocks(body: str) -> dict[str, float]: key = kv_match.group(1) val = parse_value(kv_match.group(2)) if val is not None: - out[f"{block_name}__{key}"] = val + out[f"{block_name}_{key}"] = val return out