feat(rl): KL reference policy — proven RLHF pattern for Hold preservation
Add rl_kl_reference_grad kernel: β × (π_θ(a) - π_ref(a)) gradient fires on ALL batch elements, ALL steps. π_ref = [Hold=50%, rest=5%] encodes surfer philosophy as a continuous prior. Unlike entropy (pushes to uniform, doesn't know Hold is special), gates (binary, blocks learning), or hold-prior advantages (overwhelmed by PPO ratio), KL penalty is smooth, continuous, and specifically preserves the Hold-heavy reference distribution. β=0.5 (ISV slot 578). Hold=75-100% through 5000 steps. wr=0.339 — profitable trades while maintaining Hold dominance. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -70,6 +70,7 @@ const KERNELS: &[&str] = &[
|
||||
"rl_pi_action_kernel", // audit Option B: π drives action selection via multinomial sampling from softmax(pi_logits); Q becomes pure critic
|
||||
"rl_reward_clamp_controller", // audit 2026-05-24: adaptive [-LOSS, +WIN] clamp from positive-tail EMA; replaces static [-3, +1] that crushed winning-trade signal in rmgm5
|
||||
"rl_atom_support_update", // audit 2026-05-24 followup: refreshes atom_supports_d from ISV V_MIN/V_MAX so C51 atom span adapts with reward clamp (Q learning was capped at V_MAX=1.0)
|
||||
"rl_kl_reference_grad",
|
||||
"rl_q_pi_distill_grad", // audit 2026-05-24 vj5f6 followup: KL(softmax(E_Q/τ) || π_new) gradient ADDED to pi_grad_logits — couples Q's improved C51 calibration to action selection (was decoupled per Option B)
|
||||
"rl_q_distill_lambda_controller",// audit 2026-05-24 rljzl followup: adaptive λ_distill via Schulman bounded step on KL_EMA vs target
|
||||
"rl_unit_state_update", // SP20 P1+P5 audit fix: per-unit trade state machine — detects open/close/reverse transitions, sets up unit slot 0 entry+trail
|
||||
|
||||
Reference in New Issue
Block a user