gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

plan5(task5-A): wire tier 2/3 val_* metrics into HEALTH_DIAG

Browse Source 
Plan 5 Task 4 left every tier-2/tier-3 check failing with "metric missing
from aggregate" because the existing 'Validation backtest:' free-form log
line was not parseable by the aggregate-multi-seed-metrics.py block-keyed
parser. Phase A closes that gap end-to-end (CPU-only, no kernel touch):

* metrics.rs::compute_validation_loss — emit a new
    HEALTH_DIAG[<epoch>]: val [sharpe=… sortino=… win_rate=…
                               max_drawdown=… trade_count=… calmar=…
                               omega_ratio=… total_pnl=… var_95=… cvar_95=…
                               trades_per_bar=… active_frac=… dir_entropy=…
                               sharpe_annualised=… profit_factor=…
                               window_bars=…]
  block immediately after the existing 'Validation backtest:' line. All
  16 keys derive from the existing GpuBacktestEvaluator WindowMetrics
  reduction (no new GPU work):
    - sharpe / sortino / win_rate / max_drawdown / total_trades /
      calmar / omega_ratio / total_pnl / var_95 / cvar_95 / buy_count /
      sell_count / hold_count come straight from m.*
    - window_bars = buy + sell + hold (kernel tallies one direction
      per bar)
    - trades_per_bar = total_trades / window_bars
    - active_frac = (buy + sell) / window_bars (kernel folds Hold AND
      Flat into hold_count, so 'active' = bars where the policy chose
      Short or Long — meets the Tier-2 'not always Hold' intent)
    - dir_entropy = -Σ p ln p over the 3-bucket {short, hold-or-flat,
      long} distribution. Documented limitation: max log(3) ≈ 1.099
      vs spec's 4-bucket 0.8·log(4) ≈ 1.109 ceiling — tier2 dir_entropy
      threshold is unreachable from this 3-bucket distribution; resolution
      tracked in audit row.
    - sharpe_annualised = m.sharpe alias (kernel already multiplies by
      sqrt(bars_per_day · 252) at backtest_metrics_kernel:266)
    - profit_factor = m.omega_ratio alias (kernel's omega computes
      gain_sum/loss_sum at threshold 0, equivalent to per-step PF;
      trade-level PF deferred — needs boundary-aware kernel work)

* mod.rs — adds last_val_metrics: Option<[f32; 14]> on DQNTrainer to
  snapshot the WindowMetrics-derived values for downstream consumers
  (smoke tests, future telemetry).

* constructor.rs — initialises the new field to None.

* aggregate-multi-seed-metrics.py — switches the block→key joiner from
  '__' to '_' so 'val [sharpe=…]' surfaces as the bare 'val_sharpe'
  aggregate key the tier check scripts and synthetic test fixtures
  already expect. The pre-existing '__' joiner was an oversight in
  Plan 5 Task 1B that was never validated against actual aggregator
  output (the aggregator emitted 90 'block__key' metrics that nothing
  consumed; the synthetic good_tier1.json / bad_tier1.json fixtures
  were always shaped as 'val_sharpe', confirming the single-underscore
  convention was intended). Renaming the 90 existing keys is safe — no
  consumers had locked in on the '__' form.

* docs/dqn-wire-up-audit.md — updates Plan 5 Task 4 row to reference
  the now-landed wiring and adds a new row documenting the val [...]
  HEALTH_DIAG block pipeline + aggregator joiner change + the deferred
  4-bucket dir_dist + trade-level PF caveats.

Validation:
  cargo check --workspace clean at 11 warnings.

  multi_fold_convergence smoke (629s, 3 folds × 5 epochs on RTX 3050 Ti)
  PASSES with 3/3 fold checkpoints. Per-fold best Sharpe: -9.78 / 42.46 /
  88.18 (within smoke noise band — no perturbation from the additive
  CPU-only HEALTH_DIAG line).

  scripts/aggregate-multi-seed-metrics.py against /tmp/p5t5a-smoke.log
  produces 3 streams (one per fold), 16 val_* keys all present:
  val_sharpe, val_sharpe_annualised, val_sortino, val_win_rate,
  val_max_drawdown, val_trade_count, val_calmar, val_omega_ratio,
  val_total_pnl, val_var_95, val_cvar_95, val_trades_per_bar,
  val_active_frac, val_dir_entropy, val_profit_factor, val_window_bars.

  check_tier2.py / check_tier3.py rejection messages are now substantive
  (threshold-based) rather than "missing key":
    Tier 2: trades_per_bar PASS @ 0.0127; active_frac FAIL @ 0.058 (model
            mostly Hold on 5-epoch smoke); dir_entropy FAIL @ 0.18 (within
            documented 3-bucket vs 4-bucket caveat).
    Tier 3: sharpe_annualised FAIL @ -0.25 (5-epoch smoke not converged);
            win_rate skipped (192 trades ≤ 500 noise gate); profit_factor
            FAIL @ 0.18 (untrained policy).
  Real validation pass requires the L40S 60-epoch run (Phase C).

Deferred (out of T5 Phase A scope):
  - val_dir_dist_{short,hold,long,flat} per-direction breakdown — kernel
    intentionally collapses Hold+Flat for trade-cycle counting; Tier-2's
    log(4) threshold needs either a kernel-level split or a 3-bucket
    threshold tweak in check_tier2.py.
  - Trade-level profit_factor (sum-winner-PnL / sum-loser-PnL) vs the
    per-step omega-equivalent emitted here.
  - avg_q_value bare-key aggregation — the metric is logged via separate
    Prometheus + tracing paths but not inside any HEALTH_DIAG block; out
    of T5 Phase A scope and pre-existing.
This commit is contained in:
jgrusewski
2026-04-26 13:04:39 +02:00
parent 0d373da490
commit fbee2a00f5
5 changed files with 129 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/dqn-wire-up-audit.md
View File

@@ -48,7 +48,8 @@
| `config/metric-bands.toml` | `MetricBandsRegistry::load_from_toml` (auto-loaded by `DQNTrainer::new_internal`) | Wired | Plan 5 Task 2 — per-metric warn/error bands (mean ± 5σ / mean ± 10σ); populated by `scripts/populate-metric-bands-from-runs.py` from Plan 5 Task 1B aggregate JSONs | — |
| `scripts/populate-metric-bands-from-runs.py` | reads aggregate JSONs from `scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2); writes `config/metric-bands.toml` `[bands]` section | Wired | Plan 5 Task 2 helper — N>=3 sample path; N=1 multiplicative fallback | — |
| `scripts/argo-train.sh --profile` + `infra/k8s/argo/train-multi-seed-template.yaml` (`profile` parameter, conditional `nsys profile` wrapper, `mc cp` upload to `foxhunt-training-artifacts/profiles/<sha>/`) + `infra/docker/Dockerfile.foxhunt-training-runtime` (`nsight-systems-cli` apt install) + `infra/k8s/minio/minio.yaml` (new `foxhunt-training-artifacts` bucket) | invoked manually via `./scripts/argo-train.sh dqn --profile [...]`; consumed by `scripts/compare-nsys-profiles.py BASELINE CURRENT --epochs N` (V0 metric: total cuda_gpu_kern_sum / epochs; V1 NVTX-range path deferred to Plan 5 Task 5) | Wired | Plan 5 Task 3 A.4.1 — nsys profile harness with regression-comparison script. `--profile` forces multi-seed render path so dry-run never needs cluster contact (test_nsys_harness.sh); MinIO creds optional on `train-single` (warn-skips upload if absent). **Baseline capture deferred to Plan 5 Task 5** (T5 runs the harness on L40S as part of the multi-seed acceptance pass — avoids burning local GPU time in T3) | — |
| `scripts/validation/check_tier1.py` + `check_tier2.py` + `check_tier3.py` + `check_all_tiers.py` (+ `tests/test_tier_checks.sh` + `tests/fixtures/{good,bad}_tier1.json`) | reads aggregate JSONs from `scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2); invoked manually as `python3 scripts/validation/check_all_tiers.py <metrics.json> [--warmup-end N]` and from Plan 5 Task 5's acceptance pass | Wired | Plan 5 Task 4 — per-tier exit checks for the spec §2 tiered acceptance criteria. Tier 1 (convergence: std/mean ≤ 0.15 on `val_sharpe`/`avg_q_value`/`train_loss` over stable epochs + no avg_q_value > 500 fold-1 explosion + Q-saturation/hot-path-DtoH placeholders); Tier 2 (behavioural: `val_trades_per_bar ≥ 0.005`, `val_active_frac > 0.2`, dir entropy > 0.8·log4); Tier 3 (profitability: `val_sharpe_annualised > 1.0` with per-bar fallback, `val_win_rate ≥ 0.52` gated on >500 trades, `val_profit_factor` mean ≥ 1.1 AND cross-seed std < 0.3). Stdlib only (no numpy/scipy). Defensive missing-metric handling — each missing aggregate key fails the relevant check with an explanatory message rather than silently passing. **Deferred metrics not yet emitted by HEALTH_DIAG** (current aggregator key set is `<block>__<key>` per `aggregate-multi-seed-metrics.py:115`): `val_trades_per_bar`, `val_active_frac`, `val_dir_entropy` / `val_dir_dist_*`, `val_sharpe_annualised`, `val_win_rate`, `val_profit_factor`, `val_trade_count`. Wiring those into `training_loop.rs::log_epoch_metrics_and_financials` (and either flat-emitting or tracking through the band-check harvest path) is the prerequisite for Tiers 2/3 to ever PASS on real data — tracked under Plan 5 Task 5's pre-flight. | — |
| `scripts/validation/check_tier1.py` + `check_tier2.py` + `check_tier3.py` + `check_all_tiers.py` (+ `tests/test_tier_checks.sh` + `tests/fixtures/{good,bad}_tier1.json`) | reads aggregate JSONs from `scripts/aggregate-multi-seed-metrics.py` (Plan 5 Task 1B.2); invoked manually as `python3 scripts/validation/check_all_tiers.py <metrics.json> [--warmup-end N]` and from Plan 5 Task 5's acceptance pass | Wired | Plan 5 Task 4 — per-tier exit checks for the spec §2 tiered acceptance criteria. Tier 1 (convergence: std/mean ≤ 0.15 on `val_sharpe`/`avg_q_value`/`train_loss` over stable epochs + no avg_q_value > 500 fold-1 explosion + Q-saturation/hot-path-DtoH placeholders); Tier 2 (behavioural: `val_trades_per_bar ≥ 0.005`, `val_active_frac > 0.2`, dir entropy > 0.8·log4); Tier 3 (profitability: `val_sharpe_annualised > 1.0` with per-bar fallback, `val_win_rate ≥ 0.52` gated on >500 trades, `val_profit_factor` mean ≥ 1.1 AND cross-seed std < 0.3). Stdlib only (no numpy/scipy). Defensive missing-metric handling — each missing aggregate key fails the relevant check with an explanatory message rather than silently passing. Tier-2/tier-3 wiring landed in Plan 5 Task 5 Phase A (`val [...]` HEALTH_DIAG block + aggregator single-`_` joiner) — see next row. | — |
| `trainer/metrics.rs::compute_validation_loss` (HEALTH_DIAG `val [...]` block) + `scripts/aggregate-multi-seed-metrics.py::parse_blocks` (single-`_` joiner) + `trainer/mod.rs` (`last_val_metrics: Option<[f32; 14]>`) | Plan 5 Task 4 tier-2/tier-3 check scripts read these as top-level `val_*` aggregate keys. Block emit pipeline: `evaluate_dqn_graphed` → `WindowMetrics{sharpe, sortino, win_rate, max_drawdown, total_trades, calmar, omega_ratio, total_pnl, var_95, cvar_95, buy/sell/hold_count}` (CUDA `compute_backtest_metrics` 14-float reduction, no new GPU work) → CPU derivations (`window_bars = buy+sell+hold`; `trades_per_bar = total_trades / window_bars`; `active_frac = (buy+sell) / window_bars`; `dir_entropy = -Σ p ln p` over the 3-bucket {short, hold-or-flat, long} distribution; `sharpe_annualised = m.sharpe` alias since the kernel already multiplies by `sqrt(bars_per_day · 252)`; `profit_factor = m.omega_ratio` alias since the kernel's omega is `gain_sum / loss_sum` with threshold 0, equivalent to per-step PF) → `tracing::info!("HEALTH_DIAG[{}]: val [sharpe=… sortino=… win_rate=… max_drawdown=… trade_count=… calmar=… omega_ratio=… total_pnl=… var_95=… cvar_95=… trades_per_bar=… active_frac=… dir_entropy=… sharpe_annualised=… profit_factor=… window_bars=…]", current_epoch, …)`. The aggregator joins `<block>_<key>` (single underscore — was `__` before T5 Phase A; tier scripts and synthetic test fixtures already used the bare `val_*` convention) so each emitted key surfaces as a top-level `val_*` aggregate. **Deferred for follow-up** (cannot be emitted from existing kernel data): (a) per-direction distribution `val_dir_dist_{short,hold,long,flat}` — kernel collapses Hold+Flat into hold_count (intentional for the position-sign trade-cycle definition), so `dir_entropy` here ranges over 3 buckets with max `log 3 ≈ 1.099` rather than the spec's 4-bucket `log 4 ≈ 1.386` ceiling; tier2 `check_dir_entropy` compares against `0.8 · log 4 ≈ 1.109` which is unreachable from the 3-bucket distribution. Resolution options: extend `WindowMetrics` with separate Hold/Flat counts (kernel touch) **or** lower tier2's threshold to the 3-bucket equivalent `0.8 · log 3 ≈ 0.879`. (b) trade-level `profit_factor` (sum-of-winning-trade-P&L / sum-of-losing-trade-P&L) — currently aliased to per-step `omega_ratio`; matches the standard PF definition under threshold 0 but a trade-level variant would require boundary-aware per-trade P&L accumulation in the kernel. Cold-path (per validation epoch); zero hot-path overhead; one new tracing line per epoch. | — |
| `trainers/dqn/adaptive_monitor.rs` | Read-only observer trait + harness (FireRateStats, DiagSnapshot, IsvBus<'a>); consumers added in Plan 1 Tasks 9-17 (atoms/gamma/kelly_cap/tau/epsilon/grad_balancer monitors) | Wired (consumers added in same plan) | C.6 GPU-drives-CPU-reads | — |
| `trainers/dqn/monitors/grad_balancer_monitor.rs` | Read-only observer for grad_balance_isv_update kernel output (ISV slots 31..35); consumers: HEALTH_DIAG + controller_activity smoke | Wired | Plan 1 Task 17 | — |
| `trainers/dqn/monitors/tau_monitor.rs` | Read-only observer for tau_update kernel output (ISV slot 42); consumers: HEALTH_DIAG + controller_activity smoke | Wired | Plan 1 Task 13 | — |