fix(dqn/c51): clamp adaptive v_range to config bounds — Fold 1 explosion

Root cause of the L40S 10-epoch smoke Fold 1 loss explosion
(train-7r9zf, final_loss=1.6e16, grad_norm=62B): update_eval_v_range
recomputed [v_min, v_max] every epoch from eval_q_mean_ema ± gap_width
with NO upper bound, while the bounded theoretical range in
config.v_min/v_max (derived from reward_scale/(1-gamma)*1.2, clamp [20, 300])
was only honored at buffer initialization.

The runaway path (visible in the epoch-by-epoch trace):

  Epoch 3: Q-range=[-10, 10] — pegged at config bounds ✓
  Epoch 4: Q-range=[-10, 14.16] — Q escapes the configured window
  Epoch 5: Q-range=[-10, 22.30], loss=43,660 (up from 30)
  Epoch 10: loss=1.6e16, grad_norm=62B, Fold 1 converged=false

Positive feedback loop:
  Q overestimate → eval_q_mean_ema drifts up → v_max follows unbounded
  → C51 atom support widens → TD targets grow → Q targets grow
  → network chases → gradient explodes → repeat.

Fold 0 stayed in the safe region only because its reward dynamics never
pushed Q far enough to escape ±10; Fold 1's larger window (3.7M bars vs
2.9M) contained the trigger event.

Fix: gpu_dqn_trainer::update_eval_v_range now clamps BOTH the half-width
to (config.v_max - config.v_min) / 2 AND the centre to the range
[config.v_min + half, config.v_max - half], guaranteeing the adaptive
window always fits inside the theoretical bounds. Single source of truth
— the config-level clamp is now actually authoritative.

Verified: multi-trial smoke 5/5 pass, median_q_gap=2.07 (beats 2.00
baseline from commit cb2015ab2), mean_sharpe_ema=7.82. No regression on
Fold 0's normal-range behaviour.

Closes task #31.
This commit is contained in:
jgrusewski
2026-04-21 17:53:38 +02:00
parent 932ac2bda8
commit 679881fa53

View File

@@ -2148,10 +2148,26 @@ impl GpuDqnTrainer {
// Floor = max(10 * q_gap, 3 * q_std) — enough atoms to resolve actions.
// With 51 atoms, a width of 10*q_gap gives q_gap/delta_z ≈ 5 atoms of
// resolution between best and worst action.
//
// Clamp the adaptive [v_min, v_max] to the theoretical bounds supplied
// at construction (config.v_{min,max}, derived from reward_scale/gamma).
// Without this clamp the window can drift arbitrarily far from 0: once
// the policy overestimates, eval_q_mean_ema tracks it up → v_max follows
// → C51 atoms span a wider range → even larger TD targets → loss
// explodes geometrically (Fold 1 smoke: final_loss=1.6e16, grad=62B).
//
// Two clamps are needed — one on half-width so the band can't exceed
// the full theoretical range, and one on the centre so the band always
// fits inside [config.v_min, config.v_max]:
let gap_width = (10.0 * q_gap).max(3.0 * self.eval_q_std_ema).max(0.1);
let half = gap_width;
let v_min = self.eval_q_mean_ema - half;
let v_max = self.eval_q_mean_ema + half;
let abs_half = (self.config.v_max - self.config.v_min) * 0.5;
let half = gap_width.min(abs_half);
let center = self.eval_q_mean_ema.clamp(
self.config.v_min + half,
self.config.v_max - half,
);
let v_min = center - half;
let v_max = center + half;
// Pinned device-mapped: CPU writes directly, GPU reads via dev_ptr. No HtoD copy.
unsafe {
*self.eval_v_range_pinned = v_min;