Replace the separate KL gradient kernel with an RLHF-style KL reward:
r_modified = r_env - β × (log π_θ(a|s) - log π_ref(a))
This feeds through PPO's standard advantage pipeline — no structural
imbalance between KL (100% of steps) and PPO (6% done-steps). The KL
signal has the same per-step coverage as the advantage because it IS
the advantage on non-done steps.
Remove done-gating: all steps now get advantages (KL reward provides
directional signal on non-done steps). Remove distillation done-gating
too (distill on all steps now that PPO has signal everywhere).
Result: entropy STABLE 0.81-1.31 (was collapsing to 0.0 with gradient
KL). Hold oscillates 62-100% at b=16 (expected — sparse PnL). At
b=1024 should settle ~50% with 60× denser PnL signal.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
844412f1df