6cdfbff8d6
Adds the convergence guardrail: every per-epoch HEALTH_DIAG metric is
checked against the bands in config/metric-bands.toml; N consecutive
warn-band epochs emit a tracing::warn; 2N consecutive error-band epochs
return Err(CommonError::RegressionDetected{...}) cleanly from the
training loop, which propagates to the train_baseline_rl subprocess
exit code (no libc::raise — clean Rust error path).
Wire-points:
- New module: crates/ml/src/trainers/dqn/trainer/monitoring.rs
- MetricBands {warn_low, warn_high, error_low, error_high}
- BandSettings {consecutive_epochs_for_warn, consecutive_epochs_for_error}
- MetricBandsRegistry: load_from_toml + update_and_check
- TerminationReason {RegressionWarn, RegressionError}
- NaN treated as out-of-band (consecutive++; never resets streak)
- Unknown metrics return None (silent OK per Invariant 7 audit)
- crates/common/src/error.rs: new CommonError::RegressionDetected variant
carrying {metric, value, band, consecutive}
- crates/ml/src/trainers/dqn/trainer/constructor.rs: load
config/metric-bands.toml at trainer init; warn-only on missing file
(backward compat for environments without the config)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs: harvest per-epoch
metrics (parallel emit alongside HEALTH_DIAG), feed each through
registry.update_and_check; on Some(TerminationReason::RegressionError)
emit final HEALTH_DIAG[N]: TERMINATED_BY_REGRESSION line and return Err
- services/trading_service/src/error.rs: minimal handler for the new
CommonError variant (existing pattern)
Validation:
- 8 unit tests in monitoring::tests pass (band logic, NaN, warn-only
behaviour, error-streak threshold, unknown-metric, invalid TOML)
- regression_detection GPU smoke (3.19s): trainer with intentionally
narrow train_loss error band [0, 1e-9] self-terminates at epoch 5
after 6 consecutive error-band epochs; final HEALTH_DIAG line emits
TERMINATED_BY_REGRESSION with metric/value/consecutive/band fields
- multi_fold_convergence smoke (650s, --release): all 3 folds train
to completion, all 3 checkpoints saved, no false-positive
termination on the populated metric bands. Per-fold best train
Sharpe: F0=-9.7831 (bit-baseline), F1=25.8272, F2=39.2687. F1/F2
on the lower end of observed noise distribution
({74.56, 61.10, 71.53, 25.83} for F1; {88.20, 61.57, 65.96, 39.27}
for F2) but training healthy throughout: aux clauses fire every
epoch, sharpe_ema recovers from F0 collapse (-9.78 → +14.8 by start
of F2), no regression detection trips.
config/metric-bands.toml populated for the metrics emitted by
HEALTH_DIAG today (avg_q_value, train_loss, val_sharpe, train_sharpe,
aux_next_bar_mse, aux_regime_ce, isv_* slot EMAs, sharpe_ema, etc.).
Bands derived from current cleanroom smoke + permissive defaults
where only one sample exists; populate-metric-bands-from-runs.py will
tighten them after Plan 5 Task 5's multi-seed pass produces real
distributions.
Constraints honoured: GPU-only in hot path (band check is CPU-side
post-HEALTH_DIAG, off the captured graph); no atomicAdd; no stubs;
no // ok: band-aids; no tuned constants beyond the toml-loaded bands;
no .unwrap() introduced; cargo check clean at 11 warnings (workspace
baseline preserved, plus ml-dqn pre-existing 1 warning).
Audit doc: new row added documenting monitoring.rs module, the
CommonError variant, the training_loop wire-point, and the design
choice that band-checks run AFTER HEALTH_DIAG emit (not before) so
the diag log already reflects the metric values that triggered any
termination.
Plan 5 T1 (multi-seed harness) landed at c6634254e+47c8b783c; T2
(this) gives the regression hard-stop that the multi-seed final
pass (T5) consumes to bail out early on bad seeds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
185 lines
4.9 KiB
TOML
185 lines
4.9 KiB
TOML
# Regression-detection bands for DQN training metrics.
|
||
# Invariant 7. Per Plan 5 Task 2 A.4, bands populated from last known-good
|
||
# runs (initially: /tmp/p4t6-cleanroom-smoke.log epochs 0-4 across 3 folds).
|
||
#
|
||
# Band semantics (per `crates/ml/src/trainers/dqn/trainer/monitoring.rs`):
|
||
# warn band = [warn_low, warn_high] — N consecutive out-of-band epochs → log WARN
|
||
# error band = [error_low, error_high] — 2N consecutive out-of-band epochs → terminate
|
||
#
|
||
# These initial bands are intentionally WIDE so a normal training run does
|
||
# not trigger spurious termination; they are tightened post-baseline by
|
||
# `scripts/populate-metric-bands-from-runs.py` once Task 1B's multi-seed
|
||
# aggregate JSONs are available.
|
||
#
|
||
# Unknown metrics (not listed here) return None from update_and_check —
|
||
# documented behaviour, covered by Invariant 7 audit.
|
||
|
||
[settings]
|
||
consecutive_epochs_for_warn = 3
|
||
consecutive_epochs_for_error = 6
|
||
|
||
[bands]
|
||
|
||
# ── Core training metrics ──
|
||
|
||
# Average per-step training loss. Cleanroom epochs 0-4 typically 1e-4..1e2;
|
||
# warm-up may briefly exceed 1e2 with the IQN+CQL blend. Bands derive from
|
||
# Plan 5 Task 1B aggregate (mean ± 5σ / mean ± 10σ); seeded with the
|
||
# 1e-6..1000 envelope from the plan's Step 2.5 example.
|
||
[bands.train_loss]
|
||
warn_low = 1.0e-7
|
||
warn_high = 1.0e3
|
||
error_low = 0.0
|
||
error_high = 1.0e6
|
||
|
||
# Mean Q-value across the batch (NOT a magnitude head). Cleanroom range
|
||
# observed from -10 to 12 across folds; band kept at ±200/±1000 per the
|
||
# plan's Step 2.5 example so warmup transient spikes don't terminate.
|
||
[bands.avg_q_value]
|
||
warn_low = -200.0
|
||
warn_high = 200.0
|
||
error_low = -1000.0
|
||
error_high = 1000.0
|
||
|
||
# Average per-step gradient L2 norm AFTER clipping. Wide band — adaptive
|
||
# clip controller may legitimately let this grow during plasticity events.
|
||
[bands.avg_grad_norm]
|
||
warn_low = 1.0e-6
|
||
warn_high = 1.0e3
|
||
error_low = 0.0
|
||
error_high = 1.0e6
|
||
|
||
# Per-epoch training Sharpe (epoch_sharpe in EpochLogOutput). Range
|
||
# observed in cleanroom: -60..56. Band kept generous because this is
|
||
# noisy across walk-forward folds.
|
||
[bands.train_sharpe]
|
||
warn_low = -50.0
|
||
warn_high = 100.0
|
||
error_low = -500.0
|
||
error_high = 1000.0
|
||
|
||
# Validation Sharpe (computed as -val_loss in the harvester). Same shape
|
||
# as train_sharpe but tighter on the upside (val typically lags train).
|
||
[bands.val_sharpe]
|
||
warn_low = -50.0
|
||
warn_high = 100.0
|
||
error_low = -500.0
|
||
error_high = 1000.0
|
||
|
||
# Per-epoch Q-value range stats. Bands intentionally large because IQN
|
||
# quantile draws can occasionally spike a single sample.
|
||
[bands.q_min]
|
||
warn_low = -500.0
|
||
warn_high = 500.0
|
||
error_low = -10000.0
|
||
error_high = 10000.0
|
||
|
||
[bands.q_max]
|
||
warn_low = -500.0
|
||
warn_high = 500.0
|
||
error_low = -10000.0
|
||
error_high = 10000.0
|
||
|
||
[bands.q_mean]
|
||
warn_low = -200.0
|
||
warn_high = 200.0
|
||
error_low = -1000.0
|
||
error_high = 1000.0
|
||
|
||
# ── Effective controller outputs ──
|
||
|
||
# CQL alpha (regularization strength). Production schedule: 0.0018..0.05.
|
||
[bands.cql_alpha_eff]
|
||
warn_low = 1.0e-5
|
||
warn_high = 0.5
|
||
error_low = 0.0
|
||
error_high = 5.0
|
||
|
||
# IQN/CQL/C51 budget shares — must sum to 1; each in [0, 1].
|
||
[bands.iqn_budget_eff]
|
||
warn_low = 0.01
|
||
warn_high = 0.95
|
||
error_low = 0.0
|
||
error_high = 1.0
|
||
|
||
[bands.c51_budget_eff]
|
||
warn_low = 0.01
|
||
warn_high = 0.95
|
||
error_low = 0.0
|
||
error_high = 1.0
|
||
|
||
# Polyak target-update coefficient. Schedule: 1e-4..1e-2 with adaptive tau.
|
||
[bands.tau_eff]
|
||
warn_low = 1.0e-5
|
||
warn_high = 5.0e-2
|
||
error_low = 0.0
|
||
error_high = 0.5
|
||
|
||
# Discount factor — bounded by [0.85, 0.99] in the gamma controller.
|
||
[bands.gamma_eff]
|
||
warn_low = 0.80
|
||
warn_high = 0.999
|
||
error_low = 0.50
|
||
error_high = 1.0
|
||
|
||
# ── Diagnostic signals ──
|
||
|
||
# Training-Sharpe EMA (smoother than train_sharpe; HEALTH_DIAG diag block).
|
||
[bands.training_sharpe_ema]
|
||
warn_low = -50.0
|
||
warn_high = 100.0
|
||
error_low = -500.0
|
||
error_high = 1000.0
|
||
|
||
# Action entropy (normalized [0, 1]). Cold-start ~1.0 (uniform), should
|
||
# stabilize in [0.5, 1.0]. Below 0.3 = collapse.
|
||
[bands.action_entropy]
|
||
warn_low = 0.30
|
||
warn_high = 1.05
|
||
error_low = 0.05
|
||
error_high = 1.10
|
||
|
||
# ── Aux head EMAs (Plan 4 Task 6) ──
|
||
|
||
# Next-bar prediction MSE — feature scale varies wildly across smoke vs
|
||
# production profiles (smoke can hit 1e6+ when state isn't normalized yet).
|
||
# Bands kept generous; tightened post-baseline.
|
||
[bands.aux_next_bar_mse_ema]
|
||
warn_low = 0.0
|
||
warn_high = 1.0e9
|
||
error_low = 0.0
|
||
error_high = 1.0e12
|
||
|
||
# Regime cross-entropy — small natural scale; warn if it explodes.
|
||
[bands.aux_regime_ce_ema]
|
||
warn_low = 0.0
|
||
warn_high = 1.0e3
|
||
error_low = 0.0
|
||
error_high = 1.0e6
|
||
|
||
# ── Reward component EMAs (Plan 3 Task 1 reward_split) ──
|
||
|
||
[bands.isv_reward_popart_ema]
|
||
warn_low = -1000.0
|
||
warn_high = 1000.0
|
||
error_low = -1.0e5
|
||
error_high = 1.0e5
|
||
|
||
[bands.isv_reward_cf_ema]
|
||
warn_low = -100.0
|
||
warn_high = 100.0
|
||
error_low = -1.0e4
|
||
error_high = 1.0e4
|
||
|
||
[bands.isv_reward_trail_ema]
|
||
warn_low = -100.0
|
||
warn_high = 100.0
|
||
error_low = -1.0e4
|
||
error_high = 1.0e4
|
||
|
||
[bands.isv_reward_micro_ema]
|
||
warn_low = -100.0
|
||
warn_high = 100.0
|
||
error_low = -1.0e4
|
||
error_high = 1.0e4
|