fix(cuda): branch-gate done in compute_advantage_return (resolves step-4 NaN)
Replaces multiplication-gating `advantages[b] = done * (ret - vt)` with
branch-gating `is_done ? (ret - vt) : 0.0f`. The multiplication form
fails on IEEE-754 `0 * Inf = NaN`: when any batch's encoder produces a
non-finite `v_tp1` early in training (random-init outliers across 128
parallel backtests), the prior expression propagated that non-finiteness
to ALL non-done batches' advantages via `0 * (Inf - vt) = NaN`, which
then broke `compute_advantage_rms` (sum of A² → NaN), PPO surrogate
(A/RMS → NaN → ratio×A → NaN → l_pi=NaN), and the aux head through
shared encoder gradient. Validated step-4 NaN bisect 2026-05-29.
The branch-gate is the canonical fix per IEEE-754 done-gating discipline:
- `is_done ? r : (r + γ × vtp1)` — non-finite vtp1 never enters a done
batch's ret, and non-done batches' ret can still be non-finite (which
is fine because the V regression envelope clamp salvages it back to
bounds via fmaxf/fminf NaN-passthrough semantics)
- `is_done ? (ret - vt) : 0.0f` — non-done batches get 0 unconditionally,
regardless of ret's finiteness
Smoke at HEAD (b4aadff75 = atomicAdd-fixed + V envelope ±10) with the
prior multiplication-gate: deterministic NaN at step 4 (l_pi/l_aux NaN,
l_v finite at 6.329).
Smoke at HEAD + this fix: clean trajectory through 110 steps:
- step 100: l_q=1.80, l_pi=0.49, l_v=0.37, dones=10, pnl=-$8.9k, wr=0.36
- step 109: l_q=1.81, l_pi=0.75, l_v=0.33, dones=13, pnl=-$9.9k, wr=0.37
l_v dropped from 6.33 to 0.33 because the envelope's ±10 bound is now
actually constraining V regression targets (previously the masking
behavior of ±200 envelope made high l_v "look healthy" while hiding the
underlying NaN propagation).
This finally unblocks the Phase 2.1 smoke gate and the Phase 2.2/2.3
implementations downstream.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -19,13 +19,26 @@ extern "C" __global__ void compute_advantage_return(
|
||||
const int b = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (b >= b_size) return;
|
||||
|
||||
const float gamma = isv[RL_GAMMA_INDEX];
|
||||
const float r = rewards[b];
|
||||
const float done = dones[b];
|
||||
const float vt = v_t[b];
|
||||
const float vtp1 = v_tp1[b];
|
||||
const float gamma = isv[RL_GAMMA_INDEX];
|
||||
const float r = rewards[b];
|
||||
const float is_done = (dones[b] > 0.5f);
|
||||
const float vt = v_t[b];
|
||||
const float vtp1 = v_tp1[b];
|
||||
|
||||
const float ret = r + gamma * (1.0f - done) * vtp1;
|
||||
returns[b] = ret;
|
||||
advantages[b] = done * (ret - vt);
|
||||
// 2026-05-29: branch-gate (not multiplication-gate) the V_tp1 term
|
||||
// and the (ret - vt) advantage. Multiplication-gating fails on IEEE
|
||||
// `0 * Inf = NaN`: if any batch's encoder produces a non-finite V
|
||||
// prediction early in training (random init outliers), the prior
|
||||
// `done * (ret - vt)` multiplication propagated NaN to ALL non-done
|
||||
// batches' advantages, which then broke compute_advantage_rms (sum
|
||||
// of A² → NaN) and PPO (A/RMS → NaN, ratio×A → NaN, l_pi → NaN).
|
||||
// Branch-gating ensures non-finite vtp1/vt are NEVER mixed into a
|
||||
// non-done batch's advantage. Validated by the smoke bisect at
|
||||
// 10d4614fb (deterministic NaN at step 4 with l_v=6.329 stable —
|
||||
// proving V REGRESSION is fine while V FORWARD has at least one
|
||||
// non-finite output that the 0*Inf trick was promoting to NaN
|
||||
// everywhere). Per `pearl_atomicadd_masks_v_instability`.
|
||||
const float ret = is_done ? r : (r + gamma * vtp1);
|
||||
returns[b] = ret;
|
||||
advantages[b] = is_done ? (ret - vt) : 0.0f;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user