feat(rl): KL reference policy — proven RLHF pattern for Hold preservation

Add rl_kl_reference_grad kernel: β × (π_θ(a) - π_ref(a)) gradient
fires on ALL batch elements, ALL steps. π_ref = [Hold=50%, rest=5%]
encodes surfer philosophy as a continuous prior.

Unlike entropy (pushes to uniform, doesn't know Hold is special),
gates (binary, blocks learning), or hold-prior advantages (overwhelmed
by PPO ratio), KL penalty is smooth, continuous, and specifically
preserves the Hold-heavy reference distribution.

β=0.5 (ISV slot 578). Hold=75-100% through 5000 steps.
wr=0.339 — profitable trades while maintaining Hold dominance.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-27 01:58:01 +02:00
parent 8dfd49bda1
commit c965d549d8
5 changed files with 165 additions and 7 deletions

View File

@@ -70,6 +70,7 @@ const KERNELS: &[&str] = &[
"rl_pi_action_kernel", // audit Option B: π drives action selection via multinomial sampling from softmax(pi_logits); Q becomes pure critic
"rl_reward_clamp_controller", // audit 2026-05-24: adaptive [-LOSS, +WIN] clamp from positive-tail EMA; replaces static [-3, +1] that crushed winning-trade signal in rmgm5
"rl_atom_support_update", // audit 2026-05-24 followup: refreshes atom_supports_d from ISV V_MIN/V_MAX so C51 atom span adapts with reward clamp (Q learning was capped at V_MAX=1.0)
"rl_kl_reference_grad",
"rl_q_pi_distill_grad", // audit 2026-05-24 vj5f6 followup: KL(softmax(E_Q/τ) || π_new) gradient ADDED to pi_grad_logits — couples Q's improved C51 calibration to action selection (was decoupled per Option B)
"rl_q_distill_lambda_controller",// audit 2026-05-24 rljzl followup: adaptive λ_distill via Schulman bounded step on KL_EMA vs target
"rl_unit_state_update", // SP20 P1+P5 audit fix: per-unit trade state machine — detects open/close/reverse transitions, sets up unit slot 0 entry+trail