jgrusewski 62a7613a73 feat(rl): advantage normalization — stabilizes Q-advantage PPO ...

Add normalize_advantages kernel: zero mean, unit variance across batch
via block tree-reduce. Launched after compute_advantage_return at both
call sites (reward pipeline + method).

Without normalization, Q-advantage (E_Q - V) produced all-positive or
all-negative advantages that made PPO reinforce/suppress all actions
uniformly. l_pi exploded to 454k, qpa crashed to -0.95.

With normalization: l_pi=1170 (stable), qpa rises from -0.005 to +0.584
in 500 steps. PPO now discriminates between actions based on relative
Q-value, not absolute magnitude.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-05-26 23:10:00 +02:00

..

cuda

feat(rl): advantage normalization — stabilizes Q-advantage PPO

2026-05-26 23:10:00 +02:00

examples

fix(rl): wire BCE + aux losses into GPU loader path + event-based sync

2026-05-26 22:18:30 +02:00

src

feat(rl): advantage normalization — stabilizes Q-advantage PPO

2026-05-26 23:10:00 +02:00

tests

perf(rl): fuse adamw_increment into adamw_step + tau into iqn_forward

2026-05-26 16:46:10 +02:00

build.rs

perf(rl): GPU-resident data loader — zero CPU per step

2026-05-26 20:51:06 +02:00

Cargo.toml

feat(rl): R6 — replace LobEnv with kernel-driven GPU-pure env step

2026-05-23 10:39:25 +02:00