From a6acc25ec8b5ae44cc9ad7366a1d85473ae2fd83 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 29 May 2026 01:37:49 +0200 Subject: [PATCH] fix(cuda): branch-gate done in compute_advantage_return (resolves step-4 NaN) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces multiplication-gating `advantages[b] = done * (ret - vt)` with branch-gating `is_done ? (ret - vt) : 0.0f`. The multiplication form fails on IEEE-754 `0 * Inf = NaN`: when any batch's encoder produces a non-finite `v_tp1` early in training (random-init outliers across 128 parallel backtests), the prior expression propagated that non-finiteness to ALL non-done batches' advantages via `0 * (Inf - vt) = NaN`, which then broke `compute_advantage_rms` (sum of A² → NaN), PPO surrogate (A/RMS → NaN → ratio×A → NaN → l_pi=NaN), and the aux head through shared encoder gradient. Validated step-4 NaN bisect 2026-05-29. The branch-gate is the canonical fix per IEEE-754 done-gating discipline: - `is_done ? r : (r + γ × vtp1)` — non-finite vtp1 never enters a done batch's ret, and non-done batches' ret can still be non-finite (which is fine because the V regression envelope clamp salvages it back to bounds via fmaxf/fminf NaN-passthrough semantics) - `is_done ? (ret - vt) : 0.0f` — non-done batches get 0 unconditionally, regardless of ret's finiteness Smoke at HEAD (b4aadff75 = atomicAdd-fixed + V envelope ±10) with the prior multiplication-gate: deterministic NaN at step 4 (l_pi/l_aux NaN, l_v finite at 6.329). Smoke at HEAD + this fix: clean trajectory through 110 steps: - step 100: l_q=1.80, l_pi=0.49, l_v=0.37, dones=10, pnl=-$8.9k, wr=0.36 - step 109: l_q=1.81, l_pi=0.75, l_v=0.33, dones=13, pnl=-$9.9k, wr=0.37 l_v dropped from 6.33 to 0.33 because the envelope's ±10 bound is now actually constraining V regression targets (previously the masking behavior of ±200 envelope made high l_v "look healthy" while hiding the underlying NaN propagation). This finally unblocks the Phase 2.1 smoke gate and the Phase 2.2/2.3 implementations downstream. Co-Authored-By: Claude Opus 4.7 --- .../ml-alpha/cuda/compute_advantage_return.cu | 29 ++++++++++++++----- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/crates/ml-alpha/cuda/compute_advantage_return.cu b/crates/ml-alpha/cuda/compute_advantage_return.cu index 45d016064..e850923ed 100644 --- a/crates/ml-alpha/cuda/compute_advantage_return.cu +++ b/crates/ml-alpha/cuda/compute_advantage_return.cu @@ -19,13 +19,26 @@ extern "C" __global__ void compute_advantage_return( const int b = blockIdx.x * blockDim.x + threadIdx.x; if (b >= b_size) return; - const float gamma = isv[RL_GAMMA_INDEX]; - const float r = rewards[b]; - const float done = dones[b]; - const float vt = v_t[b]; - const float vtp1 = v_tp1[b]; + const float gamma = isv[RL_GAMMA_INDEX]; + const float r = rewards[b]; + const float is_done = (dones[b] > 0.5f); + const float vt = v_t[b]; + const float vtp1 = v_tp1[b]; - const float ret = r + gamma * (1.0f - done) * vtp1; - returns[b] = ret; - advantages[b] = done * (ret - vt); + // 2026-05-29: branch-gate (not multiplication-gate) the V_tp1 term + // and the (ret - vt) advantage. Multiplication-gating fails on IEEE + // `0 * Inf = NaN`: if any batch's encoder produces a non-finite V + // prediction early in training (random init outliers), the prior + // `done * (ret - vt)` multiplication propagated NaN to ALL non-done + // batches' advantages, which then broke compute_advantage_rms (sum + // of A² → NaN) and PPO (A/RMS → NaN, ratio×A → NaN, l_pi → NaN). + // Branch-gating ensures non-finite vtp1/vt are NEVER mixed into a + // non-done batch's advantage. Validated by the smoke bisect at + // 10d4614fb (deterministic NaN at step 4 with l_v=6.329 stable — + // proving V REGRESSION is fine while V FORWARD has at least one + // non-finite output that the 0*Inf trick was promoting to NaN + // everywhere). Per `pearl_atomicadd_masks_v_instability`. + const float ret = is_done ? r : (r + gamma * vtp1); + returns[b] = ret; + advantages[b] = is_done ? (ret - vt) : 0.0f; }