gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(dqn): unstick IQN trunk gradient — drop iqn_readiness multiplier from SAXPY scale

Browse Source 
`apply_iqn_trunk_gradient` and the parallel VSN-range SAXPY both scaled their
contribution by `iqn_lambda × iqn_readiness × iqn_budget`. The readiness
scalar initialises to 0.0 and only ramps up when `iqn_loss_ema` drops below
`iqn_loss_initial` — but that improvement requires the trunk to learn IQN's
gradient, which the readiness gate just blocked. Bootstrap deadlock:
trunk_iqn=0.0000 across every observed L40S epoch, downstream strangling
direction-Q discrimination → eval strict-argmax glues to one direction →
22-34 trades per 858k-bar window vs healthy 1257-trade burst at the one
epoch where the gate momentarily lifted.

iqn_budget already throttles the IQN contribution via the per-component
budget controller (60% IQN, ISV-driven), so readiness was an additive
band-aid that became load-bearing. New scale: `iqn_lambda × iqn_budget`.

The `iqn_readiness` field stays on `self` because the C51 loss kernel
launch site reuses `iqn_readiness_dev_ptr` as a CVaR-alpha pointer
(gpu_dqn_trainer.rs:~16227) — that semantic overload is broken in a
different way (CVaR α=0 is degenerate) and is tracked for follow-up.

Verified on cluster trace `train-multi-seed-vg2r9` (epochs 0–13):
trunk_iqn=0.0000 every epoch, q_gap_comp=0.00 every epoch, val
trade_count locked at 22–34 except epoch 2 (1257 trades) where the
gate accidentally cleared.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-26 18:04:36 +02:00
parent 8c20ad7958
commit f86353840e
2 changed files with 33 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs
View File

@@ -6451,14 +6451,26 @@ impl GpuDqnTrainer {
}
// ── 4. Plain SAXPY: grad_buf[trunk] += iqn_lambda * iqn_budget * scratch ──
// Adaptive lambda: scales with IQN loss readiness (0 when uncertain,
// 1 when converged). B4/G5: further scaled by iqn_budget.
// Scale = config.iqn_lambda × ISV-adaptive iqn_budget. The previous
// formula multiplied by `iqn_readiness` as a "loss-improvement gate",
// but readiness initialises to 0.0 and only ramps up when iqn_loss_ema
// drops below iqn_loss_initial — which cannot happen until IQN's
// gradient reaches the trunk. Bootstrap deadlock: trunk_iqn=0.0000
// for every observed epoch in the L40S validation runs, downstream
// strangling direction-Q discrimination → val argmax glues to one
// direction → 22-34 trades per 858k-bar window. iqn_budget already
// throttles via the per-component budget controller (60% IQN /
// ISV-driven), so readiness was an additive band-aid that became
// load-bearing. Removed; readiness is retained on `self` solely
// because the C51 loss kernel reuses iqn_readiness_dev_ptr as a CVaR
// alpha pointer (see launch site near line 16227) — that is a
// separate semantic overload tracked for follow-up.
// Uses aux_child-specific handle — saxpy_f32_kernel is captured in
// forward_child and a separate handle is needed here in aux_child.
{
let scratch_ptr = self.ptrs.iqn_trunk_m;
let grad_ptr = self.ptrs.grad_buf;
let scale = self.config.iqn_lambda * self.iqn_readiness * iqn_budget;
let scale = self.config.iqn_lambda * iqn_budget;
let n_i32 = trunk_grad_total as i32;
let blocks = ((trunk_grad_total + 255) / 256) as u32;
unsafe {
@@ -6478,18 +6490,18 @@ impl GpuDqnTrainer {
}
// ── 5. Plan 4 Task 1B-iv-ext: VSN-range SAXPY ────────────────────
// grad_buf[VSN range] += iqn_lambda * iqn_readiness * iqn_budget *
// vsn_dw_iqn_aux_scratch
// Same scaling factor as the trunk SAXPY above — IQN's adaptive
// lambda gates both contributions in lockstep so the per-component
// budget remains 60% IQN regardless of how the gradient distributes
// across trunk vs VSN tensors.
// grad_buf[VSN range] += iqn_lambda * iqn_budget * vsn_dw_iqn_aux_scratch
// Same scaling factor as the trunk SAXPY above (step 4) — both must
// stay in lockstep so the per-component IQN budget distributes
// identically across trunk and VSN tensors. iqn_readiness removed
// for the same bootstrap-deadlock reason documented at the trunk
// SAXPY above.
if self.config.bottleneck_dim > 0 {
let vsn_begin_off = padded_byte_offset(&param_sizes, 95);
let vsn_count = self.vsn_dw_iqn_aux_scratch.len();
let scratch_ptr = self.vsn_dw_iqn_aux_scratch.raw_ptr();
let grad_vsn_ptr = self.ptrs.grad_buf + vsn_begin_off;
let scale = self.config.iqn_lambda * self.iqn_readiness * iqn_budget;
let scale = self.config.iqn_lambda * iqn_budget;
let n_i32 = vsn_count as i32;
let blocks = ((vsn_count + 255) / 256) as u32;
unsafe {

11
docs/dqn-wire-up-audit.md
View File

@@ -2,6 +2,17 @@
**Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7.
IQN trunk-gradient readiness gate removed (2026-04-26): `gpu_dqn_trainer.rs:6461,6492`
SAXPY scale was `iqn_lambda × iqn_readiness × iqn_budget`. `iqn_readiness` initialises
to 0.0 and only ramps when `iqn_loss_ema` drops below `iqn_loss_initial`, but that
improvement requires the trunk to receive IQN gradient — which the gate blocked.
Bootstrap deadlock confirmed in cluster run `train-multi-seed-vg2r9` epochs 0-13:
`trunk_iqn=0.0000` every epoch, `q_gap_comp=0.00` every epoch, val `trade_count`
locked at 22-34 per 858k-bar window. New scale: `iqn_lambda × iqn_budget`. The
`iqn_readiness` field stays on `self` because the C51 loss kernel reuses
`iqn_readiness_dev_ptr` as a CVaR-α pointer (separate semantic overload, tracked
for follow-up — CVaR α=0 is degenerate).
P5T5 Phase G (2026-04-26): vol_normalizer numerical robustness + diagnostic logging. Phase D's aux label-scale EMA treated symptoms only; root cause was `epoch_vol_normalizer` in `training_loop.rs:495` producing wildly different values across machines (local raw=0.541, cluster raw≈1e-7), inflating return features [0..3] up to 50,000× their intended unit-variance scale. Fix: Welford's online variance (numerically stable, single-pass), sanity bounds [1e-5, 1e-1], explicit per-epoch `tracing::info!` log. Out-of-band raw values trigger `tracing::warn!` + fall back to 5e-4 default (typical equity-index 1-min vol). multi_fold_convergence smoke (3 folds × 5 epochs, 682.85s): F0=2.3551, F1=80.8206, F2=92.1063 — all positive, F2 strongest result yet. The "targets[0] not raw_close on this dataset" warning surfaces a deeper data-pipeline question (what's actually at index 0 of the 6-tuple) for future investigation; Phase G makes training scale-correct regardless. cargo check clean at 11 warnings.
P5T5 Phase E (2026-04-26): per-fold reset for `MetricBandsRegistry` regression-detection streaks. Bug found during Phase D smoke — P5T2's registry never cleared `consecutive_warn` / `consecutive_error` HashMaps at fold boundaries, so walk-forward folds accumulated streaks across boundaries. F2 epoch in error band tripped termination at "6 consecutive" even though no fold individually crossed 2N=6. Fix: `reset_streaks()` method clears both HashMaps; called from `DQNTrainer::reset_for_fold()`. New unit test `reset_streaks_clears_consecutive_counters` verifies the per-fold isolation. 9/9 monitoring unit tests pass.