fix(dqn): invert adaptive gamma direction on low atom utilization

Val-Flat-collapse fix #4 (task #94, 2026-04-24). Research-agent audit
of train-bv2n5 identified a positive-feedback loop driving atom
utilisation to 1%:

    util < 0.4 → γ -= 0.01 (toward 0.90 floor)
    γ × delta_z shrinks → TD targets concentrate on fewer bins
    → more collapse → util lower → γ lower ...

The original controller had the right structure but the wrong
direction on the low-util branch. Flipping the sign (`-=` → `+=`)
breaks the loop: when atoms collapse, raising γ widens the effective
TD-target support `γ × atom_support`, pushing targets across more
bins of the C51 grid and giving the network more distributional
signal to learn from.

Evidence from train-bv2n5 (pre-fix): atom_util stuck at 0.01 for 16
epochs; γ slowly drifted toward 0.90. No mechanism recovered.

Change localised to a single `else if util < 0.4` branch at
`training_loop.rs:660-675`. Same step size (0.005) as the healthy
branch avoids overshoot on the recovery path. Clamps retained
[0.90, 0.95] at this layer; the fused trainer re-clamps to
[0.90, 0.995] after regime adjustment downstream.

Other pending atom-util fixes from the audit (deferred pending
validation):
- Atom-entropy regularisation term in C51 loss (medium risk)
- Absolute v_half floor in update_eval_v_range (low risk)
- TD-target dithering before Bellman projection (low risk)
This single-line fix addresses the dominant mechanism — research
agent labelled it "lowest risk, highest leverage". If util remains
low after this, the others stack additively.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-24 00:26:39 +02:00
parent 543e3c11b9
commit 29e54a5bb9

View File

@@ -654,7 +654,29 @@ impl DQNTrainer {
}
}
// G4: Gamma annealing — adaptive horizon based on atom utilization
// G4: Gamma annealing — adaptive horizon based on atom utilization.
//
// Prior direction (pre-2026-04-24 val-Flat-collapse fix #4):
// util > 0.6 → γ += 0.005 (extend horizon when healthy)
// util < 0.4 → γ -= 0.010 (shorten horizon when collapsed)
//
// That second arm creates a positive-feedback loop: atom
// collapse (low util) → lower γ → `γ × delta_z` shrinks →
// TD targets concentrate on fewer bins → deeper collapse.
// Observed on train-bv2n5: atom util stuck at 1% for 16
// epochs while this controller quietly nudged γ toward its
// 0.90 floor, reinforcing the symptom it was meant to
// mitigate. Per research-agent audit (2026-04-24).
//
// New direction: when atoms are collapsed, RAISE γ so
// `γ × atom_support` spans a larger fraction of the grid,
// spreading TD targets across more bins and breaking the
// loop. Slower step (0.005) than the healthy-branch to
// avoid overshoot.
//
// Clamps retained: [0.90, 0.95] at this layer; the fused
// trainer re-clamps to [0.90, 0.995] after its regime
// adjustment.
{
let util = self.fused_ctx.as_ref()
.map(|f| f.utilization_ema())
@@ -662,7 +684,9 @@ impl DQNTrainer {
if util > 0.6 {
self.adaptive_gamma = (self.adaptive_gamma + 0.005).min(0.95);
} else if util < 0.4 {
self.adaptive_gamma = (self.adaptive_gamma - 0.01).max(0.90);
// Direction flipped (atom-collapse fix): RAISE γ to
// widen the TD-target spread across the atom grid.
self.adaptive_gamma = (self.adaptive_gamma + 0.005).min(0.95);
}
// C4/P4: Wire adaptive gamma into GPU trainer with temporal-coupled adjustment.
// apply_adaptive_gamma reads health+regime from ISV, adjusts, clamps [0.9, 0.995],