gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(rl): preserve normalization EMAs across eval boundary

Browse Source 
reset_session_state was resetting four EMAs that calibrate signal
scale rather than predict behavior:
  - RL_POS_SCALED_REWARD_MAX_EMA_INDEX (clamp scale anchor)
  - RL_NEG_SCALED_REWARD_MAX_EMA_INDEX (clamp scale anchor)
  - RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX (clamp feedback)
  - RL_POPART_MAX_ABS_REWARD_EMA_INDEX (F4 envelope)

At the train→eval fold boundary this disabled the reward clamp
(cap = pos_max_ema × ratio = 0) and let eval's first ~10 trade
outliers blow up the popart envelope σ 1500× (57 → 4471) over
~14 steps. PPO surrogate (A_unnorm = σ × A_norm) ran with
catastrophically mis-scaled gradients for the next ~1000 eval
steps, accounting for the bulk of the -$185M eval loss at
alpha-rl-6kghr fold 1.

Refines pearl_adaptive_carryover_discipline: RESET PREDICTIVE
EMAs (Kelly, dd, recency) but PRESERVE NORMALIZATION EMAs.

Spec: docs/superpowers/specs/2026-05-31-eval-boundary-normalization-preservation-design.md
Pearl: pearl_popart_reset_at_eval_boundary_shocks_normalization.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-31 21:10:59 +02:00
parent 22e6ddbcac
commit 72684ed3ef
1 changed files with 16 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
crates/ml-alpha/src/trainer/integrated.rs
View File

@@ -3208,10 +3208,20 @@ impl IntegratedTrainer {
// Layer 3 (Inventory) EMAs — NEW in v9.
(crate::rl::isv_slots::RL_INVENTORY_PENALTY_BETA_INDEX, 0.0_f32), // sentinel
(crate::rl::isv_slots::RL_INVENTORY_VARIANCE_EMA_INDEX, 0.0_f32), // sentinel
// Reward-clamp EMAs — NEW in v9.
(crate::rl::isv_slots::RL_POS_SCALED_REWARD_MAX_EMA_INDEX, 0.0_f32),
(crate::rl::isv_slots::RL_NEG_SCALED_REWARD_MAX_EMA_INDEX, 0.0_f32),
(crate::rl::isv_slots::RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX, 0.0_f32),
// NORMALIZATION EMAs intentionally NOT reset (refines
// [[pearl_adaptive_carryover_discipline]] via
// [[pearl_popart_reset_at_eval_boundary_shocks_normalization]]):
// RL_POS_SCALED_REWARD_MAX_EMA_INDEX
// RL_NEG_SCALED_REWARD_MAX_EMA_INDEX
// RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX
// RL_POPART_MAX_ABS_REWARD_EMA_INDEX (F4 envelope)
// These calibrate signal scale, not predict behavior. The model
// was trained against rewards at this scale; preserving the
// scale keeps the reward clamp armed (cap = pos_max_ema × ratio)
// and popart's envelope at train-final magnitude. Resetting
// them disabled the clamp (cap = 0 × ratio = 0) and let outliers
// explode the envelope σ 1500× over ~14 steps → -$185M eval
// loss (alpha-rl-6kghr fold 1, 2026-05-31).
// v9 — arm the defensive warmup window.
(crate::rl::isv_slots::RL_REGIME_TRANSITION_REMAINING_INDEX, warmup_steps),
// Regime observer boundary policy (F1.7 — spec 2026-05-31 v3,
@@ -3222,7 +3232,8 @@ impl IntegratedTrainer {
// - KELLY_EPS_RECOVERY_LIVE → 0.50 (= ε_max default)
// - PREV_WORST_PNL → 0 (CRITICAL — must align with the just-reset
// session_pnl_worst above to prevent spurious 3σ tail event)
// - POPART_MAX_ABS_REWARD_EMA → 0 (F4 envelope reset)
// - POPART_MAX_ABS_REWARD_EMA intentionally NOT reset — see
// normalization-EMA preservation block above.
// - Welford state (M2/MEAN/COUNT) intentionally NOT reset — the
// variance of session_pnl_change is regime-invariant.
(crate::rl::isv_slots::RL_REGIME_DEAD_ZONE_FLAG_INDEX, 0.0_f32),
@@ -3232,7 +3243,6 @@ impl IntegratedTrainer {
(crate::rl::isv_slots::RL_REGIME_RECOVERY_FACTOR_INDEX, 1.0_f32),
(crate::rl::isv_slots::RL_KELLY_EPS_RECOVERY_LIVE_INDEX, 0.50_f32),
(crate::rl::isv_slots::RL_REGIME_PREV_WORST_PNL_INDEX, 0.0_f32),
(crate::rl::isv_slots::RL_POPART_MAX_ABS_REWARD_EMA_INDEX, 0.0_f32),
] {
let mut args = RawArgs::new();
args.push_ptr(self.isv_dev_ptr);