gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(safety): Q-drift kill criterion — halt training before model goes live

Browse Source 
The train-multi-seed-p526h repro (with MSE-clamp + PopArt-carry already
landed) reproduced a geometric Q-divergence in fold 1 that train_loss
can no longer hide:

  F0 ep5 Q=+0.79 (healthy carryover)
  F1 ep1 Q=+0.82 (boundary handoff fine)
  F1 ep2 Q=+2.23 (2.7× jump — drift starts)
  F1 ep3 Q=+4.05
  F1 ep4 Q=+10.66
  F1 ep5 NaN at step 5

A model with this Q dynamic is catastrophically unsafe for live trading:
Kelly cap floats with Q-confidence, so runaway Q drives oversized
positions and can blow up the strategy before any downstream safety
trips. This commit installs a production safety net — NOT a root-cause
fix — that hard-halts training the moment Q-drift is detected:

  if |q_mean| > 2× prev AND |q_mean| > 1.5 production-unsafe floor
      → return Err, training halts, model rejected from deployment

Inserted in training_loop.rs immediately before the existing
`self.prev_epoch_q_mean = q_mean` update so the comparison uses the
same source-of-truth values that already feed downstream diagnostics.

The 2× ratio catches genuine geometric divergence (typical healthy
growth <30%/epoch); the 1.5 absolute floor prevents false positives
in early training where small Q magnitudes oscillate by large ratios
(verified: F0 ep4→ep5 ratio 5.3× would NOT trigger because |0.79|<1.5).
Both thresholds are numerical-stability bounds (Invariant 1 carve-
out), not tuned hyperparameters. Skips first epoch (no prev_q). Does
NOT skip fold-boundary epochs — those are exactly the failures
we're catching.

This is a safety net. The actual root cause is a fold-boundary reset
gap (which reset isn't firing correctly?). A systematic audit is
queued to identify it. Suspects: prev_grad_buf, PopArt GPU Welford
buffers (popart_count huge from fold 0 → new-fold stats track too
slowly), isv_q_abs_ref_* magnitude EMAs, spectral norm σ EMAs, TLOB
Adam state.

Per user's "can't happen at production" philosophy: production
deployments must have this safety net regardless of whether the
root-cause fix lands first.
This commit is contained in:
jgrusewski
2026-04-28 18:53:36 +02:00
parent 299981f9b0
commit 1c917e3cb0
2 changed files with 80 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
crates/ml/src/trainers/dqn/trainer/training_loop.rs
View File

@@ -3890,6 +3890,48 @@ impl DQNTrainer {
//
// Q-growth is still tracked via `prev_epoch_q_mean` in case future
// diagnostics want it, but nothing mutates tau from this value.
//
// PRODUCTION SAFETY: Q-drift kill criterion.
//
// A model whose Q-values double every epoch is catastrophically unsafe
// for live trading — Kelly cap floats with Q-confidence, so a runaway
// Q drives oversized positions. Even one trading session with such a
// model can blow up a strategy. We must never deploy a model that
// exhibits this dynamic, so we halt training the moment we detect it.
//
// Criterion: if BOTH (a) |q_mean| has grown more than 2× since the
// previous epoch AND (b) |q_mean| is already past a 1.5 production-
// unsafe threshold, halt training and return error. The 2× threshold
// catches genuine geometric divergence (typical healthy growth is
// <30%/epoch); the 1.5 absolute floor prevents false positives in
// early training where small Q-magnitudes oscillate by large ratios
// (e.g. fold 0 ep4→ep5 shows 0.15→0.79 = 5.3× ratio but stays well
// inside production-safe range).
//
// Both thresholds are numerical-stability bounds (Invariant 1 carve-
// out) — not tuned hyperparameters. They're derived from the bug
// signature observed in train-multi-seed-p526h: F1 ep2 Q=+2.23 (2.7×
// jump from F1 ep1 Q=+0.82) was the first epoch where divergence was
// unambiguous.
//
// Skip on the first epoch of training (no prev_q to compare). DO NOT
// skip on fold-boundary epochs — fold-boundary divergence is
// precisely the failure mode this criterion is designed to catch.
if self.prev_epoch_q_mean.abs() > 1e-6 {
let curr_abs = q_mean.abs();
let prev_abs = self.prev_epoch_q_mean.abs();
let ratio = curr_abs / prev_abs;
if curr_abs > 1.5 && ratio > 2.0 {
return Err(anyhow::anyhow!(
"Q-drift kill criterion triggered at epoch {}: \
|q_mean| jumped from {:.4} to {:.4} (ratio {:.2}×). \
A model with this dynamic is unsafe for production trading \
(Kelly cap → oversized positions). Halting training; \
model is rejected from deployment.",
epoch + 1, self.prev_epoch_q_mean, q_mean, ratio,
));
}
}
self.prev_epoch_q_mean = q_mean;
// Action diversity — per-branch from GPU monitoring (not re-derived through OrderRouter)

38
docs/dqn-wire-up-audit.md
View File

@@ -2,6 +2,44 @@
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
Q-drift production-safety kill criterion (2026-04-28): The
`train-multi-seed-p526h` repro (5ep×3fold, MSE-clamp + PopArt-carry
landed) reproduced a geometric Q-divergence in fold 1 that we can no
longer mistake for a measurement artefact. Trajectory:
F0 ep5 Q=+0.79 → F1 ep1 Q=+0.82 (boundary handoff fine) → F1 ep2 Q=+2.23
(2.7× jump) → F1 ep3 +4.05 → F1 ep4 +10.66 → F1 ep5 NaN at step 5.
The MSE clamp (`299981f9b`) keeps loss reading honest at ~1.0-1.4
throughout this divergence, so the train_loss is no longer hiding the
real Q-drift. **Production stake**: a model with this Q dynamic is
catastrophically unsafe for live trading — Kelly cap floats with
Q-confidence, so runaway Q drives oversized positions and can blow up
the strategy before any downstream safety trips. Per the user's
"can't happen at production" philosophy, this commit installs a
**production safety net** (not a root-cause fix) that hard-halts
training the moment Q-drift is detected. Implementation in
`training_loop.rs` immediately before the existing
`self.prev_epoch_q_mean = q_mean` update: if (a) `|q_mean|` has grown
more than 2× since previous epoch AND (b) `|q_mean| > 1.5` (production-
unsafe absolute floor), return `Err` — training halts, model is
rejected from deployment. The 2× ratio catches genuine geometric
divergence (typical healthy growth <30%/epoch); the 1.5 absolute
floor prevents false positives in early training where small Q
magnitudes oscillate by large ratios (verified on the data: F0
ep4→ep5 0.15→0.79 ratio 5.3× would not trigger because |0.79|<1.5).
Both thresholds are numerical-stability bounds (Invariant 1 carve-
out), not tuned hyperparameters. Skips on the very first epoch (no
prev_q to compare); does NOT skip on fold-boundary epochs — those
are exactly the failures we're catching. **This is a safety net, not
the fix.** A separate reset-gap audit (the actual root cause hunt) is
queued to identify which fold-boundary state isn't being reset
correctly. Suspect list per the inspection so far: `prev_grad_buf`
(gradient vaccine reference), PopArt GPU Welford buffers
(popart_mean/var/count — `popart_count` is huge from fold 0 making
new-fold reward stats track too slowly), `isv_q_abs_ref_*` magnitude
EMAs, spectral norm σ EMAs, TLOB Adam state. Touched: `training_loop.rs`
(+30 LOC). cargo check clean at 13 warnings (workspace baseline). No
fingerprint change.
MSE warmup loss target clamp to C51 atom support (2026-04-28): the
fold-boundary PopArt carry-forward + iqr floor (`d710f9d50`) reduced
fold-1 epoch-1 train loss readings 16× (318k → 19k) but ep-2/ep-3