fix(dqn): invert adaptive gamma direction on low atom utilization
Val-Flat-collapse fix #4 (task #94, 2026-04-24). Research-agent audit of train-bv2n5 identified a positive-feedback loop driving atom utilisation to 1%: util < 0.4 → γ -= 0.01 (toward 0.90 floor) γ × delta_z shrinks → TD targets concentrate on fewer bins → more collapse → util lower → γ lower ... The original controller had the right structure but the wrong direction on the low-util branch. Flipping the sign (`-=` → `+=`) breaks the loop: when atoms collapse, raising γ widens the effective TD-target support `γ × atom_support`, pushing targets across more bins of the C51 grid and giving the network more distributional signal to learn from. Evidence from train-bv2n5 (pre-fix): atom_util stuck at 0.01 for 16 epochs; γ slowly drifted toward 0.90. No mechanism recovered. Change localised to a single `else if util < 0.4` branch at `training_loop.rs:660-675`. Same step size (0.005) as the healthy branch avoids overshoot on the recovery path. Clamps retained [0.90, 0.95] at this layer; the fused trainer re-clamps to [0.90, 0.995] after regime adjustment downstream. Other pending atom-util fixes from the audit (deferred pending validation): - Atom-entropy regularisation term in C51 loss (medium risk) - Absolute v_half floor in update_eval_v_range (low risk) - TD-target dithering before Bellman projection (low risk) This single-line fix addresses the dominant mechanism — research agent labelled it "lowest risk, highest leverage". If util remains low after this, the others stack additively. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -654,7 +654,29 @@ impl DQNTrainer {
|
||||
}
|
||||
}
|
||||
|
||||
// G4: Gamma annealing — adaptive horizon based on atom utilization
|
||||
// G4: Gamma annealing — adaptive horizon based on atom utilization.
|
||||
//
|
||||
// Prior direction (pre-2026-04-24 val-Flat-collapse fix #4):
|
||||
// util > 0.6 → γ += 0.005 (extend horizon when healthy)
|
||||
// util < 0.4 → γ -= 0.010 (shorten horizon when collapsed)
|
||||
//
|
||||
// That second arm creates a positive-feedback loop: atom
|
||||
// collapse (low util) → lower γ → `γ × delta_z` shrinks →
|
||||
// TD targets concentrate on fewer bins → deeper collapse.
|
||||
// Observed on train-bv2n5: atom util stuck at 1% for 16
|
||||
// epochs while this controller quietly nudged γ toward its
|
||||
// 0.90 floor, reinforcing the symptom it was meant to
|
||||
// mitigate. Per research-agent audit (2026-04-24).
|
||||
//
|
||||
// New direction: when atoms are collapsed, RAISE γ so
|
||||
// `γ × atom_support` spans a larger fraction of the grid,
|
||||
// spreading TD targets across more bins and breaking the
|
||||
// loop. Slower step (0.005) than the healthy-branch to
|
||||
// avoid overshoot.
|
||||
//
|
||||
// Clamps retained: [0.90, 0.95] at this layer; the fused
|
||||
// trainer re-clamps to [0.90, 0.995] after its regime
|
||||
// adjustment.
|
||||
{
|
||||
let util = self.fused_ctx.as_ref()
|
||||
.map(|f| f.utilization_ema())
|
||||
@@ -662,7 +684,9 @@ impl DQNTrainer {
|
||||
if util > 0.6 {
|
||||
self.adaptive_gamma = (self.adaptive_gamma + 0.005).min(0.95);
|
||||
} else if util < 0.4 {
|
||||
self.adaptive_gamma = (self.adaptive_gamma - 0.01).max(0.90);
|
||||
// Direction flipped (atom-collapse fix): RAISE γ to
|
||||
// widen the TD-target spread across the atom grid.
|
||||
self.adaptive_gamma = (self.adaptive_gamma + 0.005).min(0.95);
|
||||
}
|
||||
// C4/P4: Wire adaptive gamma into GPU trainer with temporal-coupled adjustment.
|
||||
// apply_adaptive_gamma reads health+regime from ISV, adjusts, clamps [0.9, 0.995],
|
||||
|
||||
Reference in New Issue
Block a user