Standard PPO practice missing in this codebase. Without it, lifting
the WIN clamp from 1.0 to 167× (yesterday's fix) expanded |A| by
167× → gradients 167× → KL 167× → ε saturated MAX → unbounded
policy drift → overfit (alpha-rl-wf-f0 step 2000: l_pi=105k, dones=0).
Implementation:
- New kernel compute_advantage_rms.cu: single-block warp-shuffle
reduce computes sqrt(mean(A²)) over the batch each step
- New ISV slot 589 RL_ADVANTAGE_BATCH_RMS_INDEX
- ppo_clipped_surrogate fwd/bwd divide A by RMS (with floor bootstrap)
- Dispatched after compute_advantage_return in step_with_lobsim
Scale-only normalization (RMS, not z-score) preserves A=0 on
non-done steps → no spurious PPO gradient on Hold actions where
no trade closed.
This is architectural correctness, not a quickfix. Fights the
ROOT CAUSE (reward range expansion breaking LR calibration), not
the SYMPTOM (ε saturation, l_pi explosion).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6d33f18b75