Standard PPO practice (Schulman et al. 2017): normalize advantages
per-batch BEFORE PPO surrogate computation:
advantage[b] ← (advantage[b] − mean_b) / sqrt(var_b + ε²)
Self-adaptive — uses observed per-batch statistics, no tuned
hyperparameters (fits feedback_adaptive_not_tuned).
Why now: Phase 4.3 cluster (alpha-rl-qkdm2) ended with pnl=+$16.28M
(2x Plan A v2's +$8.5M) but l_pi=1.6e9 vs Plan A v2's 3.6e5 — a
4500× increase in PPO surrogate magnitude. Adam internally normalizes
the gradient direction, but per-step magnitude variance is high
→ pnl trajectory chops (saw ±$2-4M swings between milestones).
Advantage normalization fixes this regardless of V baseline source:
batches with high-magnitude advantages get scaled down to ~unit
variance, ensuring consistent PPO update strength across batches.
Standard in Stable-Baselines3, RLlib, OpenAI Baselines.
New kernel: rl_advantage_normalize.cu (~80 LOC).
Grid=(1,1,1), block=(min(B,1024),1,1).
Single block parallel reduction: pass 1 = mean, pass 2 = variance,
pass 3 = normalize in-place. ε² floor on variance prevents
div-by-zero when all advantages identical.
Trainer wiring (~30 LOC):
Load kernel + handle field + struct init.
Single launch immediately AFTER compute_advantage_return,
BEFORE PPO surrogate consumes advantages_d. In-place.
Stacks with Phase 4.4 adaptive V blend on same branch
(ml-alpha-phase4-dueling-head). Two complementary stabilization
mechanisms:
- Phase 4.4: adapts WHICH V baseline (scalar vs dq) to use based
on observed tracking error → reduces variance source
- Phase 4.5: normalizes advantages regardless of variance source
→ reduces variance impact
Validated:
- cargo build --release clean
- integrated_trainer_smoke 1 step passes
- alpha_rl_train --steps 3 --b 128 under compute-sanitizer: 0 errors
Cluster validation pending — submit Phase 4.4+4.5 combined to test
whether dampened-variance preserves Phase 4.3's +$16M pnl gain.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6695785666