gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(safety): adaptive Q-drift kill threshold from ISV |Q| reference (no tuned constant)

Browse Source 
The previous absolute floor (`|q_mean| > 1.5`) in the Q-drift kill
criterion was a tuned constant in violation of
`feedback_adaptive_not_tuned.md` and
`feedback_isv_for_adaptive_bounds.md`. Replace with an ISV-driven
adaptive threshold:

    kill_floor = max(0.5, 3.0 × max(ISV[Q_ABS_REF_INDEX=16],
                                    ISV[Q_DIR_ABS_REF_INDEX=21]))

Both ISV slots are per-branch EMAs of `max(|Q_mean|)` already
maintained on-GPU by `q_stats_kernel.cu` and consumed by
`c51_loss_kernel`/`c51_grad_kernel`. The kill floor now anchors on
the same recently-observed healthy Q scale that the loss kernels
already use to normalise their collapse-fraction signals.

The 3.0× multiplier is architectural ("RL Q-divergence shows up at
2-4× healthy scale"); the 0.5 cold-start floor is an Invariant-1
numerical-stability bound active only while both ISV slots are still
≤ 1e-6 in fold-0 epoch-1, then dominated by the runtime formula. The
2× ratio gate is unchanged — already an architectural rate-of-change
bound.

ISV reads use the existing pinned/device-mapped path
(`fused.trainer().read_isv_signal_at`) — no HtoD/DtoH per
`feedback_no_htod_htoh_only_mapped_pinned.md`.

The bug-signature trajectory that motivated the original 1.5 floor
(F1 ep2 Q=+2.23 from F1 ep1 Q=+0.82) still trips: ISV[16] would have
been ~0.6 with α=0.05 EMA tracking, giving kill_floor ≈ 1.8, and
Q=+2.23 > 1.8 with ratio 2.7× trips both gates.

Touched: `training_loop.rs` (+70 LOC, two new use-list imports),
`docs/dqn-wire-up-audit.md` (Invariant 7 audit entry).

cargo check clean at 13 warnings (workspace baseline). No
fingerprint change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-28 20:12:29 +02:00
parent 3c0d262927
commit fe2b77388d
2 changed files with 86 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

83
crates/ml/src/trainers/dqn/trainer/training_loop.rs
View File

@@ -22,7 +22,8 @@ use common::metrics::{questdb_sink, training_metrics};
use tracing::{debug, info, warn};
use crate::cuda_pipeline::gpu_dqn_trainer::{
EPOCH_IDX_INDEX, EPSILON_EFF_INDEX, GAMMA_DIR_EFF_INDEX, TLOB_REGIME_FOCUS_EMA_INDEX,
EPOCH_IDX_INDEX, EPSILON_EFF_INDEX, GAMMA_DIR_EFF_INDEX, Q_ABS_REF_INDEX,
Q_DIR_ABS_REF_INDEX, TLOB_REGIME_FOCUS_EMA_INDEX,
};
use crate::dqn::logging::{log_epoch_start, log_epoch_end, log_training_progress};
use crate::dqn::target_update::convergence_half_life;
@@ -3925,20 +3926,51 @@ impl DQNTrainer {
// model can blow up a strategy. We must never deploy a model that
// exhibits this dynamic, so we halt training the moment we detect it.
//
// Criterion: if BOTH (a) |q_mean| has grown more than 2× since the
// previous epoch AND (b) |q_mean| is already past a 1.5 production-
// unsafe threshold, halt training and return error. The 2× threshold
// catches genuine geometric divergence (typical healthy growth is
// <30%/epoch); the 1.5 absolute floor prevents false positives in
// early training where small Q-magnitudes oscillate by large ratios
// (e.g. fold 0 ep4→ep5 shows 0.15→0.79 = 5.3× ratio but stays well
// inside production-safe range).
// Criterion: if BOTH (a) `|q_mean|` has grown more than 2× since the
// previous epoch AND (b) `|q_mean|` is already past an ISV-derived
// production-unsafe floor, halt training and return error.
//
// Both thresholds are numerical-stability bounds (Invariant 1 carve-
// out) — not tuned hyperparameters. They're derived from the bug
// signature observed in train-multi-seed-p526h: F1 ep2 Q=+2.23 (2.7×
// jump from F1 ep1 Q=+0.82) was the first epoch where divergence was
// unambiguous.
// * 2× ratio — architectural rate-of-change bound. Healthy training
// grows |q_mean| by <30%/epoch; a per-epoch doubling is geometric
// divergence regardless of absolute scale. Not a tuned constant.
//
// * Adaptive floor `kill_floor = 3.0 × max(ISV[Q_ABS_REF_INDEX],
// ISV[Q_DIR_ABS_REF_INDEX])` — both ISV slots track per-branch
// EMAs of `max(|Q_mean|)` over the magnitude (slot 16) and
// direction (slot 21) heads. The same EMAs already drive C51
// bin-weighting in `c51_loss_kernel.cu` and conviction shaping in
// `experience_kernels.cu` (search `q_abs_ref` / `q_dir_abs_ref`),
// so they reflect the model's *currently observed* healthy Q
// scale rather than a fixed deployment threshold. We take the
// `max()` across the two EMAs because either branch diverging
// past 3× its own normal scale is sufficient evidence of runaway
// — using min() would let one branch hide the other's drift.
//
// The `3.0×` multiplier is architectural: empirically RL Q-
// divergence shows up at roughly 2-4× normal scale on the same
// trajectory, with healthy oscillations staying inside ~1.5×.
// Per `feedback_isv_for_adaptive_bounds.md`, the *absolute level*
// comes from runtime ISV; the multiplier is the architectural
// "drift starts here" factor, not a tuned magnitude.
//
// * Bootstrap fallback: when neither ISV slot has graduated above
// `1e-6` (cold start, before the first stats-cadence write), the
// adaptive floor degenerates to 0. We then floor the bound at
// `0.5` so the kill criterion still has *some* lower bound during
// the first few epochs of fold 0. The `0.5` is a numerical-
// stability bound (Invariant 1 carve-out) chosen to sit safely
// below the ISV cold-start observed in healthy training (~0.05-
// 0.5 by epoch 3) — NOT a tuned trigger. Once ISV slots populate
// the formula uses runtime data alone.
//
// Both the 3.0× multiplier and the 0.5 cold-start floor are
// numerical-stability bounds (Invariant 1 carve-out), not tuned
// hyperparameters. The bug-signature trajectory that motivated the
// original 1.5 floor (train-multi-seed-p526h: F1 ep2 Q=+2.23, a 2.7×
// jump from F1 ep1 Q=+0.82) still trips this gate — the magnitude
// EMA ISV[16] would have been ~0.6 at that point (F1 first epoch
// |Q|≈0.82 with the standard α=0.05 EMA tracking), giving
// `kill_floor ≈ 3 × 0.6 = 1.8` and Q=+2.23 > 1.8 + ratio 2.7 trips.
//
// Skip on the first epoch of training (no prev_q to compare). DO NOT
// skip on fold-boundary epochs — fold-boundary divergence is
@@ -3947,14 +3979,33 @@ impl DQNTrainer {
let curr_abs = q_mean.abs();
let prev_abs = self.prev_epoch_q_mean.abs();
let ratio = curr_abs / prev_abs;
if curr_abs > 1.5 && ratio > 2.0 {
// Read both per-branch |Q| reference EMAs from the pinned/
// device-mapped ISV bus (zero-copy, mapped-pinned page — see
// `feedback_no_htod_htoh_only_mapped_pinned.md`). Take max() so
// either branch's healthy scale anchors the floor; using min()
// would let an under-active branch suppress drift on the other.
let (q_abs_ref_mag, q_abs_ref_dir) = if let Some(ref fused) = self.fused_ctx {
let mag = fused.trainer().read_isv_signal_at(Q_ABS_REF_INDEX).max(0.0);
let dir = fused.trainer().read_isv_signal_at(Q_DIR_ABS_REF_INDEX).max(0.0);
(mag, dir)
} else {
(0.0_f32, 0.0_f32)
};
let q_abs_ref_max = q_abs_ref_mag.max(q_abs_ref_dir);
// Cold-start floor (Invariant 1 carve-out): keep some bound
// before ISV EMAs graduate above the producer kernels' 1e-6
// epsilon. 0.5 sits safely below early-fold-0 healthy |Q|.
let kill_floor = (3.0_f64 * q_abs_ref_max as f64).max(0.5);
if curr_abs > kill_floor && ratio > 2.0 {
return Err(anyhow::anyhow!(
"Q-drift kill criterion triggered at epoch {}: \
|q_mean| jumped from {:.4} to {:.4} (ratio {:.2}×). \
|q_mean| jumped from {:.4} to {:.4} (ratio {:.2}×, \
adaptive floor {:.4} = max(0.5, 3× max(ISV[16]={:.4}, ISV[21]={:.4}))). \
A model with this dynamic is unsafe for production trading \
(Kelly cap → oversized positions). Halting training; \
model is rejected from deployment.",
epoch + 1, self.prev_epoch_q_mean, q_mean, ratio,
kill_floor, q_abs_ref_mag, q_abs_ref_dir,
));
}
}

19
docs/dqn-wire-up-audit.md
View File

@@ -40,6 +40,25 @@ EMAs, spectral norm σ EMAs, TLOB Adam state. Touched: `training_loop.rs`
(+30 LOC). cargo check clean at 13 warnings (workspace baseline). No
fingerprint change.
Q-drift kill threshold made adaptive (2026-04-28, follow-up to commit
`1c917e3cb`): the absolute floor in the kill criterion (`curr_abs > 1.5`)
was a tuned constant in violation of `feedback_adaptive_not_tuned.md` and
`feedback_isv_for_adaptive_bounds.md`. Replaced with `kill_floor = max(0.5,
3.0 × max(ISV[Q_ABS_REF_INDEX=16], ISV[Q_DIR_ABS_REF_INDEX=21]))`. Both ISV
slots are per-branch EMAs of `max(|Q_mean|)` already maintained on-GPU by
`q_stats_kernel.cu` and consumed by `c51_loss_kernel`/`c51_grad_kernel`
for collapse-fraction normalisation; the kill criterion now anchors on
the same recently-observed healthy Q scale. The 3.0× multiplier is
architectural ("drift starts at ~2-4× healthy"), the 0.5 cold-start floor
is an Invariant-1 numerical-stability bound (active only while both ISV
slots are still ≤ 1e-6 in fold-0 epoch-1, then dominated by the ISV
formula). The 2× ratio gate is unchanged — it's already an architectural
rate-of-change bound. ISV reads use the existing pinned/device-mapped
path via `fused.trainer().read_isv_signal_at(...)` — no HtoD/DtoH per
`feedback_no_htod_htoh_only_mapped_pinned.md`. Touched: `training_loop.rs`
(+~70 LOC, two new use-list imports). cargo check clean at 13 warnings.
No fingerprint change.
Fold-boundary reset gap — IQN readiness gauge (2026-04-28, Fix 3 of
3): the IQN readiness gauge on `GpuDqnTrainer` is a streaming
improvement fraction `iqn_readiness = (iqn_loss_initial -