fix(cuda): branch-gate done in compute_advantage_return (resolves step-4 NaN)

Replaces multiplication-gating `advantages[b] = done * (ret - vt)` with
branch-gating `is_done ? (ret - vt) : 0.0f`. The multiplication form
fails on IEEE-754 `0 * Inf = NaN`: when any batch's encoder produces a
non-finite `v_tp1` early in training (random-init outliers across 128
parallel backtests), the prior expression propagated that non-finiteness
to ALL non-done batches' advantages via `0 * (Inf - vt) = NaN`, which
then broke `compute_advantage_rms` (sum of A² → NaN), PPO surrogate
(A/RMS → NaN → ratio×A → NaN → l_pi=NaN), and the aux head through
shared encoder gradient. Validated step-4 NaN bisect 2026-05-29.

The branch-gate is the canonical fix per IEEE-754 done-gating discipline:
- `is_done ? r : (r + γ × vtp1)` — non-finite vtp1 never enters a done
  batch's ret, and non-done batches' ret can still be non-finite (which
  is fine because the V regression envelope clamp salvages it back to
  bounds via fmaxf/fminf NaN-passthrough semantics)
- `is_done ? (ret - vt) : 0.0f` — non-done batches get 0 unconditionally,
  regardless of ret's finiteness

Smoke at HEAD (b4aadff75 = atomicAdd-fixed + V envelope ±10) with the
prior multiplication-gate: deterministic NaN at step 4 (l_pi/l_aux NaN,
l_v finite at 6.329).

Smoke at HEAD + this fix: clean trajectory through 110 steps:
- step 100: l_q=1.80, l_pi=0.49, l_v=0.37, dones=10, pnl=-$8.9k, wr=0.36
- step 109: l_q=1.81, l_pi=0.75, l_v=0.33, dones=13, pnl=-$9.9k, wr=0.37

l_v dropped from 6.33 to 0.33 because the envelope's ±10 bound is now
actually constraining V regression targets (previously the masking
behavior of ±200 envelope made high l_v "look healthy" while hiding the
underlying NaN propagation).

This finally unblocks the Phase 2.1 smoke gate and the Phase 2.2/2.3
implementations downstream.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-29 01:37:49 +02:00
parent b4aadff756
commit a6acc25ec8

View File

@@ -19,13 +19,26 @@ extern "C" __global__ void compute_advantage_return(
const int b = blockIdx.x * blockDim.x + threadIdx.x;
if (b >= b_size) return;
const float gamma = isv[RL_GAMMA_INDEX];
const float r = rewards[b];
const float done = dones[b];
const float vt = v_t[b];
const float vtp1 = v_tp1[b];
const float gamma = isv[RL_GAMMA_INDEX];
const float r = rewards[b];
const float is_done = (dones[b] > 0.5f);
const float vt = v_t[b];
const float vtp1 = v_tp1[b];
const float ret = r + gamma * (1.0f - done) * vtp1;
returns[b] = ret;
advantages[b] = done * (ret - vt);
// 2026-05-29: branch-gate (not multiplication-gate) the V_tp1 term
// and the (ret - vt) advantage. Multiplication-gating fails on IEEE
// `0 * Inf = NaN`: if any batch's encoder produces a non-finite V
// prediction early in training (random init outliers), the prior
// `done * (ret - vt)` multiplication propagated NaN to ALL non-done
// batches' advantages, which then broke compute_advantage_rms (sum
// of A² → NaN) and PPO (A/RMS → NaN, ratio×A → NaN, l_pi → NaN).
// Branch-gating ensures non-finite vtp1/vt are NEVER mixed into a
// non-done batch's advantage. Validated by the smoke bisect at
// 10d4614fb (deterministic NaN at step 4 with l_v=6.329 stable —
// proving V REGRESSION is fine while V FORWARD has at least one
// non-finite output that the 0*Inf trick was promoting to NaN
// everywhere). Per `pearl_atomicadd_masks_v_instability`.
const float ret = is_done ? r : (r + gamma * vtp1);
returns[b] = ret;
advantages[b] = is_done ? (ret - vt) : 0.0f;
}