From 29e54a5bb934b236a531debc5dad642aa64bdfb7 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 24 Apr 2026 00:26:39 +0200 Subject: [PATCH] fix(dqn): invert adaptive gamma direction on low atom utilization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Val-Flat-collapse fix #4 (task #94, 2026-04-24). Research-agent audit of train-bv2n5 identified a positive-feedback loop driving atom utilisation to 1%: util < 0.4 → γ -= 0.01 (toward 0.90 floor) γ × delta_z shrinks → TD targets concentrate on fewer bins → more collapse → util lower → γ lower ... The original controller had the right structure but the wrong direction on the low-util branch. Flipping the sign (`-=` → `+=`) breaks the loop: when atoms collapse, raising γ widens the effective TD-target support `γ × atom_support`, pushing targets across more bins of the C51 grid and giving the network more distributional signal to learn from. Evidence from train-bv2n5 (pre-fix): atom_util stuck at 0.01 for 16 epochs; γ slowly drifted toward 0.90. No mechanism recovered. Change localised to a single `else if util < 0.4` branch at `training_loop.rs:660-675`. Same step size (0.005) as the healthy branch avoids overshoot on the recovery path. Clamps retained [0.90, 0.95] at this layer; the fused trainer re-clamps to [0.90, 0.995] after regime adjustment downstream. Other pending atom-util fixes from the audit (deferred pending validation): - Atom-entropy regularisation term in C51 loss (medium risk) - Absolute v_half floor in update_eval_v_range (low risk) - TD-target dithering before Bellman projection (low risk) This single-line fix addresses the dominant mechanism — research agent labelled it "lowest risk, highest leverage". If util remains low after this, the others stack additively. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../src/trainers/dqn/trainer/training_loop.rs | 28 +++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/crates/ml/src/trainers/dqn/trainer/training_loop.rs b/crates/ml/src/trainers/dqn/trainer/training_loop.rs index 7327f415c..069145746 100644 --- a/crates/ml/src/trainers/dqn/trainer/training_loop.rs +++ b/crates/ml/src/trainers/dqn/trainer/training_loop.rs @@ -654,7 +654,29 @@ impl DQNTrainer { } } - // G4: Gamma annealing — adaptive horizon based on atom utilization + // G4: Gamma annealing — adaptive horizon based on atom utilization. + // + // Prior direction (pre-2026-04-24 val-Flat-collapse fix #4): + // util > 0.6 → γ += 0.005 (extend horizon when healthy) + // util < 0.4 → γ -= 0.010 (shorten horizon when collapsed) + // + // That second arm creates a positive-feedback loop: atom + // collapse (low util) → lower γ → `γ × delta_z` shrinks → + // TD targets concentrate on fewer bins → deeper collapse. + // Observed on train-bv2n5: atom util stuck at 1% for 16 + // epochs while this controller quietly nudged γ toward its + // 0.90 floor, reinforcing the symptom it was meant to + // mitigate. Per research-agent audit (2026-04-24). + // + // New direction: when atoms are collapsed, RAISE γ so + // `γ × atom_support` spans a larger fraction of the grid, + // spreading TD targets across more bins and breaking the + // loop. Slower step (0.005) than the healthy-branch to + // avoid overshoot. + // + // Clamps retained: [0.90, 0.95] at this layer; the fused + // trainer re-clamps to [0.90, 0.995] after its regime + // adjustment. { let util = self.fused_ctx.as_ref() .map(|f| f.utilization_ema()) @@ -662,7 +684,9 @@ impl DQNTrainer { if util > 0.6 { self.adaptive_gamma = (self.adaptive_gamma + 0.005).min(0.95); } else if util < 0.4 { - self.adaptive_gamma = (self.adaptive_gamma - 0.01).max(0.90); + // Direction flipped (atom-collapse fix): RAISE γ to + // widen the TD-target spread across the atom grid. + self.adaptive_gamma = (self.adaptive_gamma + 0.005).min(0.95); } // C4/P4: Wire adaptive gamma into GPU trainer with temporal-coupled adjustment. // apply_adaptive_gamma reads health+regime from ISV, adjusts, clamps [0.9, 0.995],