gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

tune(rl): lower KL β=0.0005 + reward_kl=0.001 for b=1024

Browse Source 
β=0.003 pinned Hold=100% at b=1024 despite 60 dones/step.
The KL gradient overwhelms even the dense PPO signal.
Try 6× lower β to find the balance point.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-27 11:14:58 +02:00
parent 6b89dbfcb8
commit 9e2c036c5e
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
crates/ml-alpha/src/trainer/integrated.rs
View File

@@ -2768,8 +2768,8 @@ impl IntegratedTrainer {
(crate::rl::isv_slots::RL_TARGET_TAU_MAX_INDEX, 0.005),
(crate::rl::isv_slots::RL_HOLD_PRIOR_INDEX, 0.1),
(crate::rl::isv_slots::RL_HOLD_TARGET_FRAC_INDEX, 0.5),
(crate::rl::isv_slots::RL_KL_REF_BETA_INDEX, 0.003),
(crate::rl::isv_slots::RL_REWARD_KL_BETA_INDEX, 0.005),
(crate::rl::isv_slots::RL_KL_REF_BETA_INDEX, 0.0005),
(crate::rl::isv_slots::RL_REWARD_KL_BETA_INDEX, 0.001),
(crate::rl::isv_slots::RL_KL_REF_TARGET_INDEX, 0.5),
(crate::rl::isv_slots::RL_EPS_BOOTSTRAP_INDEX, 0.2),
(crate::rl::isv_slots::RL_ROLLOUT_BOOTSTRAP_INDEX, 2048.0),