From f86353840e6d4b246630b25db20550f2847a7b7c Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 26 Apr 2026 18:04:36 +0200 Subject: [PATCH] =?UTF-8?q?fix(dqn):=20unstick=20IQN=20trunk=20gradient=20?= =?UTF-8?q?=E2=80=94=20drop=20iqn=5Freadiness=20multiplier=20from=20SAXPY?= =?UTF-8?q?=20scale?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `apply_iqn_trunk_gradient` and the parallel VSN-range SAXPY both scaled their contribution by `iqn_lambda × iqn_readiness × iqn_budget`. The readiness scalar initialises to 0.0 and only ramps up when `iqn_loss_ema` drops below `iqn_loss_initial` — but that improvement requires the trunk to learn IQN's gradient, which the readiness gate just blocked. Bootstrap deadlock: trunk_iqn=0.0000 across every observed L40S epoch, downstream strangling direction-Q discrimination → eval strict-argmax glues to one direction → 22-34 trades per 858k-bar window vs healthy 1257-trade burst at the one epoch where the gate momentarily lifted. iqn_budget already throttles the IQN contribution via the per-component budget controller (60% IQN, ISV-driven), so readiness was an additive band-aid that became load-bearing. New scale: `iqn_lambda × iqn_budget`. The `iqn_readiness` field stays on `self` because the C51 loss kernel launch site reuses `iqn_readiness_dev_ptr` as a CVaR-alpha pointer (gpu_dqn_trainer.rs:~16227) — that semantic overload is broken in a different way (CVaR α=0 is degenerate) and is tracked for follow-up. Verified on cluster trace `train-multi-seed-vg2r9` (epochs 0–13): trunk_iqn=0.0000 every epoch, q_gap_comp=0.00 every epoch, val trade_count locked at 22–34 except epoch 2 (1257 trades) where the gate accidentally cleared. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../ml/src/cuda_pipeline/gpu_dqn_trainer.rs | 32 +++++++++++++------ docs/dqn-wire-up-audit.md | 11 +++++++ 2 files changed, 33 insertions(+), 10 deletions(-) diff --git a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs index 561167028..82109cf29 100644 --- a/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs +++ b/crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs @@ -6451,14 +6451,26 @@ impl GpuDqnTrainer { } // ── 4. Plain SAXPY: grad_buf[trunk] += iqn_lambda * iqn_budget * scratch ── - // Adaptive lambda: scales with IQN loss readiness (0 when uncertain, - // 1 when converged). B4/G5: further scaled by iqn_budget. + // Scale = config.iqn_lambda × ISV-adaptive iqn_budget. The previous + // formula multiplied by `iqn_readiness` as a "loss-improvement gate", + // but readiness initialises to 0.0 and only ramps up when iqn_loss_ema + // drops below iqn_loss_initial — which cannot happen until IQN's + // gradient reaches the trunk. Bootstrap deadlock: trunk_iqn=0.0000 + // for every observed epoch in the L40S validation runs, downstream + // strangling direction-Q discrimination → val argmax glues to one + // direction → 22-34 trades per 858k-bar window. iqn_budget already + // throttles via the per-component budget controller (60% IQN / + // ISV-driven), so readiness was an additive band-aid that became + // load-bearing. Removed; readiness is retained on `self` solely + // because the C51 loss kernel reuses iqn_readiness_dev_ptr as a CVaR + // alpha pointer (see launch site near line 16227) — that is a + // separate semantic overload tracked for follow-up. // Uses aux_child-specific handle — saxpy_f32_kernel is captured in // forward_child and a separate handle is needed here in aux_child. { let scratch_ptr = self.ptrs.iqn_trunk_m; let grad_ptr = self.ptrs.grad_buf; - let scale = self.config.iqn_lambda * self.iqn_readiness * iqn_budget; + let scale = self.config.iqn_lambda * iqn_budget; let n_i32 = trunk_grad_total as i32; let blocks = ((trunk_grad_total + 255) / 256) as u32; unsafe { @@ -6478,18 +6490,18 @@ impl GpuDqnTrainer { } // ── 5. Plan 4 Task 1B-iv-ext: VSN-range SAXPY ──────────────────── - // grad_buf[VSN range] += iqn_lambda * iqn_readiness * iqn_budget * - // vsn_dw_iqn_aux_scratch - // Same scaling factor as the trunk SAXPY above — IQN's adaptive - // lambda gates both contributions in lockstep so the per-component - // budget remains 60% IQN regardless of how the gradient distributes - // across trunk vs VSN tensors. + // grad_buf[VSN range] += iqn_lambda * iqn_budget * vsn_dw_iqn_aux_scratch + // Same scaling factor as the trunk SAXPY above (step 4) — both must + // stay in lockstep so the per-component IQN budget distributes + // identically across trunk and VSN tensors. iqn_readiness removed + // for the same bootstrap-deadlock reason documented at the trunk + // SAXPY above. if self.config.bottleneck_dim > 0 { let vsn_begin_off = padded_byte_offset(¶m_sizes, 95); let vsn_count = self.vsn_dw_iqn_aux_scratch.len(); let scratch_ptr = self.vsn_dw_iqn_aux_scratch.raw_ptr(); let grad_vsn_ptr = self.ptrs.grad_buf + vsn_begin_off; - let scale = self.config.iqn_lambda * self.iqn_readiness * iqn_budget; + let scale = self.config.iqn_lambda * iqn_budget; let n_i32 = vsn_count as i32; let blocks = ((vsn_count + 255) / 256) as u32; unsafe { diff --git a/docs/dqn-wire-up-audit.md b/docs/dqn-wire-up-audit.md index be3c6f696..a22005c52 100644 --- a/docs/dqn-wire-up-audit.md +++ b/docs/dqn-wire-up-audit.md @@ -2,6 +2,17 @@ **Status:** Populated during Plan 1 Task 6 (A.5 orphan audit). Updated on every commit per Invariant 7. +IQN trunk-gradient readiness gate removed (2026-04-26): `gpu_dqn_trainer.rs:6461,6492` +SAXPY scale was `iqn_lambda × iqn_readiness × iqn_budget`. `iqn_readiness` initialises +to 0.0 and only ramps when `iqn_loss_ema` drops below `iqn_loss_initial`, but that +improvement requires the trunk to receive IQN gradient — which the gate blocked. +Bootstrap deadlock confirmed in cluster run `train-multi-seed-vg2r9` epochs 0-13: +`trunk_iqn=0.0000` every epoch, `q_gap_comp=0.00` every epoch, val `trade_count` +locked at 22-34 per 858k-bar window. New scale: `iqn_lambda × iqn_budget`. The +`iqn_readiness` field stays on `self` because the C51 loss kernel reuses +`iqn_readiness_dev_ptr` as a CVaR-α pointer (separate semantic overload, tracked +for follow-up — CVaR α=0 is degenerate). + P5T5 Phase G (2026-04-26): vol_normalizer numerical robustness + diagnostic logging. Phase D's aux label-scale EMA treated symptoms only; root cause was `epoch_vol_normalizer` in `training_loop.rs:495` producing wildly different values across machines (local raw=0.541, cluster raw≈1e-7), inflating return features [0..3] up to 50,000× their intended unit-variance scale. Fix: Welford's online variance (numerically stable, single-pass), sanity bounds [1e-5, 1e-1], explicit per-epoch `tracing::info!` log. Out-of-band raw values trigger `tracing::warn!` + fall back to 5e-4 default (typical equity-index 1-min vol). multi_fold_convergence smoke (3 folds × 5 epochs, 682.85s): F0=2.3551, F1=80.8206, F2=92.1063 — all positive, F2 strongest result yet. The "targets[0] not raw_close on this dataset" warning surfaces a deeper data-pipeline question (what's actually at index 0 of the 6-tuple) for future investigation; Phase G makes training scale-correct regardless. cargo check clean at 11 warnings. P5T5 Phase E (2026-04-26): per-fold reset for `MetricBandsRegistry` regression-detection streaks. Bug found during Phase D smoke — P5T2's registry never cleared `consecutive_warn` / `consecutive_error` HashMaps at fold boundaries, so walk-forward folds accumulated streaks across boundaries. F2 epoch in error band tripped termination at "6 consecutive" even though no fold individually crossed 2N=6. Fix: `reset_streaks()` method clears both HashMaps; called from `DQNTrainer::reset_for_fold()`. New unit test `reset_streaks_clears_consecutive_counters` verifies the per-fold isolation. 9/9 monitoring unit tests pass.