jgrusewski e5ced809aa feat(rl): split KL into static reward + dynamic advantage ...

Static: β×log(π_ref(a)) added to reward, stored in PER. Q sees
consistent Hold preference across replays.

Dynamic: -β×log(π_θ(a)) added to advantage only (not stored in PER).
Provides entropy-regularized signal (SAC-like) that adapts to
current policy. Low π_θ → positive advantage (explore). High π_θ →
near-zero (don't over-concentrate).

Result: entropy STABLE 0.94-1.29 through 5000 steps (no collapse).
Hold 62-100%. wr=0.36 (best local). Removed KL gradient kernel
from training step (KL is now entirely in the reward/advantage).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-05-27 11:46:13 +02:00

..

cuda

feat(rl): split KL into static reward + dynamic advantage

2026-05-27 11:46:13 +02:00

examples

fix(rl): wire BCE + aux losses into GPU loader path + event-based sync

2026-05-26 22:18:30 +02:00

src

feat(rl): KL penalty as REWARD not gradient — fixes structural imbalance

2026-05-27 11:38:09 +02:00

tests

perf(rl): fuse adamw_increment into adamw_step + tau into iqn_forward

2026-05-26 16:46:10 +02:00

build.rs

feat(rl): KL reference policy — proven RLHF pattern for Hold preservation

2026-05-27 01:58:01 +02:00

Cargo.toml

feat(rl): R6 — replace LobEnv with kernel-driven GPU-pure env step

2026-05-23 10:39:25 +02:00