Three critical stability fixes for PPO training:
1. Gradient clipping: Added clip_grads() that scales gradients by
max_norm/norm when L2 norm exceeds max_grad_norm (0.5). Previously
the code only logged a warning — gradient norms of 27-171x above
the threshold were applied raw to weights, causing divergence.
Fixed in all 8 locations across PpoTrainer (6) and ContinuousPPO (2).
2. Return normalization: Value loss now normalizes returns to N(0,1)
before computing MSE. Raw cumulative returns (±1000s) caused enormous
value gradients that destabilized the critic network.
3. Hyperopt search space: Tightened value LR upper bound from 1e-3 to
1e-4 (1e-3 is documented unstable), policy LR from 1e-3 to 3e-4.
Also fixed LSTM path where optimizer.step() ran BEFORE gradient norm
check — now clip → step (not step → warn).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
48aac3c601