foxhunt/docs/superpowers/specs/2026-05-31-eval-boundary-normalization-preservation-design.md

Eval-Boundary Normalization-EMA Preservation — Design

Date: 2026-05-31 Status: Draft (pre-implementation) Branch target: ml-alpha-eval-boundary-norm-fix (off ml-alpha-regime-observer) Related work: Phase A (eval_diag), E.1-E.5 (eval-summary aggregation), v9 defensive eval-boundary calibration Pearls referenced:

• pearl_popart_reset_at_eval_boundary_shocks_normalization — full diagnosis
• pearl_adaptive_carryover_discipline — being refined here
• F4 envelope design
• feedback_isv_for_adaptive_bounds, feedback_adaptive_not_tuned

---

1. Motivation

Fold-1 eval cluster runs (alpha-rl-8ll7j, alpha-rl-jz48s, alpha-rl-6kghr) all show the same pattern: total eval pnl = −$185M, dominated by a catastrophic shock window in the first ~1000 eval steps where popart σ explodes from train-final ~57 to >4400 within 14 steps.

Phase A's eval_diag.jsonl enabled the per-step diagnosis. Mechanism (verified empirically):

1. reset_session_state at the train→eval fold boundary resets reward-clamp scale EMAs to 0:
   • RL_POS_SCALED_REWARD_MAX_EMA_INDEX → 0
   • RL_NEG_SCALED_REWARD_MAX_EMA_INDEX → 0
   • RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX → 0
   • RL_POPART_MAX_ABS_REWARD_EMA_INDEX → 0 (F4 envelope)
2. The reward clamp's cap is cap = pos_max_ema × margin_ratio. With pos_max_ema = 0, cap = 0 — the clamp is functionally disabled. All raw rewards pass through unclamped.
3. Eval's first ~10 trades produce realized pnl swings 10-300× larger than train's typical (Kelly resets to bootstrap=1.0 → larger positions; eval data may also have different volatility).
4. Unclamped large rewards feed rl_popart_normalize. F4's envelope ("fast-up, slow-decay" by design — see F4 spec) captures the outliers wholesale. σ_effective = max(σ_welford, envelope) becomes envelope-driven.
5. σ explodes 57 → 4471 (78× over train calibration) by eval step 14.
6. PPO surrogate's A_unnorm = σ × A_norm scales 78×. Policy gradient magnitudes are catastrophically wrong. Next ~1000 eval steps thrash with mis-scaled updates.
7. Per-trade losses accumulate to dominate the -$185M aggregate.

This is NOT model overfitting in the classical sense. The model's policy survives the shock (qpa recovers to 0.99 by eval step 1500). It IS a state-management bug at the regime boundary.

2. Goals

G1 — Preserve normalization-state EMAs across regime boundaries

At reset_session_state, do not reset:

• RL_POS_SCALED_REWARD_MAX_EMA_INDEX
• RL_NEG_SCALED_REWARD_MAX_EMA_INDEX
• RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX
• RL_POPART_MAX_ABS_REWARD_EMA_INDEX

These are NORMALIZATION EMAs (calibrate signal scale). The model was trained against rewards at this scale; preserving the scale lets the model interpret eval rewards with the same calibration.

G2 — Keep resetting predictive EMAs

Continue resetting Kelly EMAs (win_rate, avg_win/loss, cumulative_dones), dd_trip, cooldown, recency, ε_recovery, session_pnl. These predict future behavior from past, and past behavior in train doesn't predict eval regime.

G3 — Update the pearl rule

Distill the refined rule into pearl_adaptive_carryover_discipline: "RESET PREDICTIVE EMAs (Kelly, dd_trip, cooldown). PRESERVE NORMALIZATION EMAs (clamp pos/neg_max, popart envelope, clamp clip-rate). Don't reset blindly."

G4 — Non-goals

• Don't fix policy-side overfitting yet. If the model's policy is genuinely bad on eval data, that's a separate (Layer 3) problem. Fix the boundary-normalization shock first; the resulting eval pnl is the true measure of model quality.
• Don't introduce a pure-eval mode (frozen weights). That's a separate, larger refactor — defer until we know if normalization preservation alone is sufficient.
• Don't widen TRADE_LOG_CAP further. Already at 4096 (E.3), sufficient.

3. Design

3.1 The change

crates/ml-alpha/src/trainer/integrated.rs:3231-3239 — remove four lines from reset_session_state's reset list:

// BEFORE:
// Reward-clamp EMAs — NEW in v9.
(crate::rl::isv_slots::RL_POS_SCALED_REWARD_MAX_EMA_INDEX,   0.0_f32),
(crate::rl::isv_slots::RL_NEG_SCALED_REWARD_MAX_EMA_INDEX,   0.0_f32),
(crate::rl::isv_slots::RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX,  0.0_f32),
// ...
// Regime observer boundary policy:
//   ...
//   - POPART_MAX_ABS_REWARD_EMA → 0 (F4 envelope reset)
(crate::rl::isv_slots::RL_POPART_MAX_ABS_REWARD_EMA_INDEX,       0.0_f32),

// AFTER (4 lines removed; comments around the regime-observer block updated):
// Normalization EMAs intentionally NOT reset — these calibrate signal scale.
// The model was trained against rewards at this scale; preserving the scale
// keeps the reward clamp armed (cap = pos_max_ema × ratio) and lets popart
// continue with the same normalization. RESETTING them disabled the clamp
// (cap=0×ratio=0) and let outliers blow up the envelope → σ shock window
// → -$185M eval loss (validated alpha-rl-6kghr 2026-05-31).

3.2 What stays the same

Everything else in reset_session_state is unchanged:

• Layer 1 CMDP summary slots (session_pnl, worst_pnl, dd_triggered, consec_loss, cooldown) — PREDICTIVE, correctly reset
• Layer 4 (Kelly) EMAs (win_rate, avg_win/loss, cumulative_dones, kelly_fraction) — PREDICTIVE, correctly reset
• Layer 3 (Inventory) EMAs (inventory_penalty_β, inventory_variance) — PREDICTIVE, correctly reset
• Defensive warmup arm (RL_REGIME_TRANSITION_REMAINING_INDEX = 500 steps)
• Regime observer state (dead_zone flag/duration/timeout, tail recency, recovery_factor, ε_recovery_live, prev_worst_pnl) — PREDICTIVE/transient, correctly reset
• Per-batch buffer memsets (session_pnl_per_batch_d, etc.)

3.3 Why not preserve popart.σ / .var / .mean themselves?

popart's Welford state (σ/var/mean) is ALREADY preserved across the boundary — it's not in the reset list. The bug was that env.max (which dominates σ_effective via max(σ_welford, env.max)) was being reset to 0, exposing the bare σ_welford = 1. The 4-line fix preserves env.max so σ_effective stays at train-calibrated level.

3.4 Why preserve clip_rate_ema?

RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX is the rate at which rewards were being clipped during train. The clamp's margin controller uses this to adjust the margin ratio adaptively (target clip rate ~5%). Resetting it to 0 makes the controller think it's never clipping → grows margin → defeats clamp purpose further. Preserve.

3.5 Will preserving NORMALIZATION cause its own problems?

Scenarios to consider:

Scenario	Preserved env.max	New regime ACTUAL scale	Behavior
Eval ≈ train scale	57	57	clamp works as in train, σ stable ✓
Eval larger scale	57	200	clamp clips some real signal, but pos_max_ema adapts up over ~100 steps; σ grows smoothly
Eval smaller scale	57	10	clamp lets everything through (most rewards ≤ cap), pos_max_ema slowly adapts down; σ stays at 57 until rewards consistently smaller

None of these are catastrophic. The current "reset to 0" scenario is uniquely bad because it produces an unbounded amplification rather than mild miscalibration.

4. Validation

V1 — Local smoke

Run alpha_rl_train with fold-1 walk-forward at b=16, 1k train + 250 eval. Verify:

• Pre-eval pos_max_ema is non-zero (preserved)
• First eval step's clamp cap > 0
• popart σ stays within 5× of train-final value across all 250 eval steps
• eval_summary n_eval_trades_dropped = 0

V2 — Cluster smoke

Same as v11/Phase A config (fold 1, b=1024, 20k+5k eval, seed 16962). Compare:

• eval_diag.jsonl first 50 steps: popart.σ should stay near train-final (~57), NOT spike to >4000
• eval_summary.json total_pnl_usd: should be dramatically less negative than -$185M (predicted: -$5M to -$30M range, but exact value unknown — that's the residual model overfit)

V3 — Pearl update

Patch pearl_adaptive_carryover_discipline with the predictive-vs-normalization distinction.

5. Implementation phases

Phase 1 — Code change (5 min)

Remove the 4 reset lines from reset_session_state. Update the comment to explain the new policy.

Phase 2 — Local smoke (15 min)

Same fold-1 b=16 smoke as the eval-summary fix. Inspect eval_diag.jsonl popart sigma trajectory.

Phase 3 — Cluster validation (~95 min)

Push, submit, wait, inspect.

Phase 4 — Pearl update (5 min)

Refine pearl_adaptive_carryover_discipline with the predictive-vs-normalization distinction.

6. Open decisions

#	Question	Recommendation
1	Preserve ALL 4 EMAs, or just the 3 reward-clamp ones (skipping popart envelope)?	Preserve all 4. Without env.max, the clamp may suppress the worst overshoots but the envelope itself still re-bootstraps from 0 and can spike on a single large reward. Both layers of defense matter.
2	Add a controller-level invariant test?	Yes, V1 local smoke — assert popart.σ after eval step 50 is within 5× of train-final. Cheap to verify.
3	Update pearl_adaptive_carryover_discipline NOW or wait for cluster confirmation?	After cluster. Update the pearl WITH the empirical evidence of the fix working.
4	Risk that preserving env.max masks true regime shift in eval data?	Low. EMAs adapt over time; preserving the train-final value just means eval starts calibrated rather than at 0. If eval has truly larger scale, env.max grows there too, just from 57 instead of 0.

7. Done means

• Lines removed from reset_session_state, commit made
• Local b=16 fold-1 smoke shows popart.σ bounded across eval
• Cluster b=1024 fold-1 run completes, eval_diag.jsonl shows σ never exceeds 5× train-final
• eval_summary.json total_pnl significantly less negative than -$185M (number TBD; if still very negative, that's true Layer-3 overfit work)
• Pearl pearl_adaptive_carryover_discipline updated with the refined rule

8. Risks

• Risk A (low): If train-final pos_max_ema was somehow off (e.g. from train tail event), preserving it carries the mis-calibration into eval. Mitigation: EMA decays over eval steps anyway.
• Risk B (medium): If eval shock losses come from MULTIPLE compounding factors beyond just clamp-disable (e.g. Kelly reset compounded with mis-calibrated normalization), preserving only normalization may only partially fix. Mitigation: cluster validation will measure exact remaining gap.
• Risk C (low): Pearl rule refinement might be over-broad if there are other PREDICTIVE-vs-NORMALIZATION EMAs we haven't catalogued. Mitigation: only apply the rule to the 4 EMAs we're changing today; document the principle but require future cases to verify per-EMA.