diff --git a/crates/ml/src/trainers/dqn/trainer/training_loop.rs b/crates/ml/src/trainers/dqn/trainer/training_loop.rs index 7a3d8bbaa..093f5defe 100644 --- a/crates/ml/src/trainers/dqn/trainer/training_loop.rs +++ b/crates/ml/src/trainers/dqn/trainer/training_loop.rs @@ -3890,6 +3890,48 @@ impl DQNTrainer { // // Q-growth is still tracked via `prev_epoch_q_mean` in case future // diagnostics want it, but nothing mutates tau from this value. + // + // PRODUCTION SAFETY: Q-drift kill criterion. + // + // A model whose Q-values double every epoch is catastrophically unsafe + // for live trading — Kelly cap floats with Q-confidence, so a runaway + // Q drives oversized positions. Even one trading session with such a + // model can blow up a strategy. We must never deploy a model that + // exhibits this dynamic, so we halt training the moment we detect it. + // + // Criterion: if BOTH (a) |q_mean| has grown more than 2× since the + // previous epoch AND (b) |q_mean| is already past a 1.5 production- + // unsafe threshold, halt training and return error. The 2× threshold + // catches genuine geometric divergence (typical healthy growth is + // <30%/epoch); the 1.5 absolute floor prevents false positives in + // early training where small Q-magnitudes oscillate by large ratios + // (e.g. fold 0 ep4→ep5 shows 0.15→0.79 = 5.3× ratio but stays well + // inside production-safe range). + // + // Both thresholds are numerical-stability bounds (Invariant 1 carve- + // out) — not tuned hyperparameters. They're derived from the bug + // signature observed in train-multi-seed-p526h: F1 ep2 Q=+2.23 (2.7× + // jump from F1 ep1 Q=+0.82) was the first epoch where divergence was + // unambiguous. + // + // Skip on the first epoch of training (no prev_q to compare). DO NOT + // skip on fold-boundary epochs — fold-boundary divergence is + // precisely the failure mode this criterion is designed to catch. + if self.prev_epoch_q_mean.abs() > 1e-6 { + let curr_abs = q_mean.abs(); + let prev_abs = self.prev_epoch_q_mean.abs(); + let ratio = curr_abs / prev_abs; + if curr_abs > 1.5 && ratio > 2.0 { + return Err(anyhow::anyhow!( + "Q-drift kill criterion triggered at epoch {}: \ + |q_mean| jumped from {:.4} to {:.4} (ratio {:.2}×). \ + A model with this dynamic is unsafe for production trading \ + (Kelly cap → oversized positions). Halting training; \ + model is rejected from deployment.", + epoch + 1, self.prev_epoch_q_mean, q_mean, ratio, + )); + } + } self.prev_epoch_q_mean = q_mean; // Action diversity — per-branch from GPU monitoring (not re-derived through OrderRouter) diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index c43fba508..6030fe4b7 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,44 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +Q-drift production-safety kill criterion (2026-04-28): The +`train-multi-seed-p526h` repro (5ep×3fold, MSE-clamp + PopArt-carry +landed) reproduced a geometric Q-divergence in fold 1 that we can no +longer mistake for a measurement artefact. Trajectory: +F0 ep5 Q=+0.79 → F1 ep1 Q=+0.82 (boundary handoff fine) → F1 ep2 Q=+2.23 +(2.7× jump) → F1 ep3 +4.05 → F1 ep4 +10.66 → F1 ep5 NaN at step 5. +The MSE clamp (`299981f9b`) keeps loss reading honest at ~1.0-1.4 +throughout this divergence, so the train_loss is no longer hiding the +real Q-drift. **Production stake**: a model with this Q dynamic is +catastrophically unsafe for live trading — Kelly cap floats with +Q-confidence, so runaway Q drives oversized positions and can blow up +the strategy before any downstream safety trips. Per the user's +"can't happen at production" philosophy, this commit installs a +**production safety net** (not a root-cause fix) that hard-halts +training the moment Q-drift is detected. Implementation in +`training_loop.rs` immediately before the existing +`self.prev_epoch_q_mean = q_mean` update: if (a) `|q_mean|` has grown +more than 2× since previous epoch AND (b) `|q_mean| > 1.5` (production- +unsafe absolute floor), return `Err` — training halts, model is +rejected from deployment. The 2× ratio catches genuine geometric +divergence (typical healthy growth <30%/epoch); the 1.5 absolute +floor prevents false positives in early training where small Q +magnitudes oscillate by large ratios (verified on the data: F0 +ep4→ep5 0.15→0.79 ratio 5.3× would not trigger because |0.79|<1.5). +Both thresholds are numerical-stability bounds (Invariant 1 carve- +out), not tuned hyperparameters. Skips on the very first epoch (no +prev_q to compare); does NOT skip on fold-boundary epochs — those +are exactly the failures we're catching. **This is a safety net, not +the fix.** A separate reset-gap audit (the actual root cause hunt) is +queued to identify which fold-boundary state isn't being reset +correctly. Suspect list per the inspection so far: `prev_grad_buf` +(gradient vaccine reference), PopArt GPU Welford buffers +(popart_mean/var/count — `popart_count` is huge from fold 0 making +new-fold reward stats track too slowly), `isv_q_abs_ref_*` magnitude +EMAs, spectral norm σ EMAs, TLOB Adam state. Touched: `training_loop.rs` +(+30 LOC). cargo check clean at 13 warnings (workspace baseline). No +fingerprint change. + MSE warmup loss target clamp to C51 atom support (2026-04-28): the fold-boundary PopArt carry-forward + iqr floor (`d710f9d50`) reduced fold-1 epoch-1 train loss readings 16× (318k → 19k) but ep-2/ep-3