# Eval-Boundary Normalization-EMA Preservation — Design

**Date:** 2026-05-31
**Status:** Draft (pre-implementation)
**Branch target:** `ml-alpha-eval-boundary-norm-fix` (off `ml-alpha-regime-observer`)
**Related work:** Phase A (eval_diag), E.1-E.5 (eval-summary aggregation), v9 defensive eval-boundary calibration
**Pearls referenced:**
- [`pearl_popart_reset_at_eval_boundary_shocks_normalization`](MEMORY.md) — full diagnosis
- [`pearl_adaptive_carryover_discipline`](MEMORY.md) — being refined here
- F4 envelope design
- `feedback_isv_for_adaptive_bounds`, `feedback_adaptive_not_tuned`

---

## 1. Motivation

Fold-1 eval cluster runs (alpha-rl-8ll7j, alpha-rl-jz48s, alpha-rl-6kghr) all show the same pattern: total eval pnl = −$185M, dominated by a catastrophic shock window in the first ~1000 eval steps where popart σ explodes from train-final ~57 to >4400 within 14 steps.

Phase A's `eval_diag.jsonl` enabled the per-step diagnosis. Mechanism (verified empirically):

1. `reset_session_state` at the train→eval fold boundary resets reward-clamp scale EMAs to 0:
   - `RL_POS_SCALED_REWARD_MAX_EMA_INDEX → 0`
   - `RL_NEG_SCALED_REWARD_MAX_EMA_INDEX → 0`
   - `RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX → 0`
   - `RL_POPART_MAX_ABS_REWARD_EMA_INDEX → 0` (F4 envelope)
2. The reward clamp's cap is `cap = pos_max_ema × margin_ratio`. With `pos_max_ema = 0`, cap = 0 — the clamp is functionally disabled. All raw rewards pass through unclamped.
3. Eval's first ~10 trades produce realized pnl swings 10-300× larger than train's typical (Kelly resets to bootstrap=1.0 → larger positions; eval data may also have different volatility).
4. Unclamped large rewards feed `rl_popart_normalize`. F4's envelope ("fast-up, slow-decay" by design — see F4 spec) captures the outliers wholesale. σ_effective = `max(σ_welford, envelope)` becomes envelope-driven.
5. σ explodes 57 → 4471 (78× over train calibration) by eval step 14.
6. PPO surrogate's `A_unnorm = σ × A_norm` scales 78×. Policy gradient magnitudes are catastrophically wrong. Next ~1000 eval steps thrash with mis-scaled updates.
7. Per-trade losses accumulate to dominate the -$185M aggregate.

This is **NOT** model overfitting in the classical sense. The model's policy survives the shock (qpa recovers to 0.99 by eval step 1500). It IS a state-management bug at the regime boundary.

## 2. Goals

### G1 — Preserve normalization-state EMAs across regime boundaries
At `reset_session_state`, do not reset:
- `RL_POS_SCALED_REWARD_MAX_EMA_INDEX`
- `RL_NEG_SCALED_REWARD_MAX_EMA_INDEX`
- `RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX`
- `RL_POPART_MAX_ABS_REWARD_EMA_INDEX`

These are NORMALIZATION EMAs (calibrate signal scale). The model was trained against rewards at this scale; preserving the scale lets the model interpret eval rewards with the same calibration.

### G2 — Keep resetting predictive EMAs
Continue resetting Kelly EMAs (win_rate, avg_win/loss, cumulative_dones), dd_trip, cooldown, recency, ε_recovery, session_pnl. These predict future behavior from past, and past behavior in train doesn't predict eval regime.

### G3 — Update the pearl rule
Distill the refined rule into `pearl_adaptive_carryover_discipline`: "RESET PREDICTIVE EMAs (Kelly, dd_trip, cooldown). PRESERVE NORMALIZATION EMAs (clamp pos/neg_max, popart envelope, clamp clip-rate). Don't reset blindly."

### G4 — Non-goals
- **Don't fix policy-side overfitting yet.** If the model's policy is genuinely bad on eval data, that's a separate (Layer 3) problem. Fix the boundary-normalization shock first; the resulting eval pnl is the true measure of model quality.
- **Don't introduce a pure-eval mode (frozen weights).** That's a separate, larger refactor — defer until we know if normalization preservation alone is sufficient.
- **Don't widen `TRADE_LOG_CAP` further.** Already at 4096 (E.3), sufficient.

## 3. Design

### 3.1 The change

`crates/ml-alpha/src/trainer/integrated.rs:3231-3239` — remove four lines from `reset_session_state`'s reset list:

```rust
// BEFORE:
// Reward-clamp EMAs — NEW in v9.
(crate::rl::isv_slots::RL_POS_SCALED_REWARD_MAX_EMA_INDEX,   0.0_f32),
(crate::rl::isv_slots::RL_NEG_SCALED_REWARD_MAX_EMA_INDEX,   0.0_f32),
(crate::rl::isv_slots::RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX,  0.0_f32),
// ...
// Regime observer boundary policy:
//   ...
//   - POPART_MAX_ABS_REWARD_EMA → 0 (F4 envelope reset)
(crate::rl::isv_slots::RL_POPART_MAX_ABS_REWARD_EMA_INDEX,       0.0_f32),

// AFTER (4 lines removed; comments around the regime-observer block updated):
// Normalization EMAs intentionally NOT reset — these calibrate signal scale.
// The model was trained against rewards at this scale; preserving the scale
// keeps the reward clamp armed (cap = pos_max_ema × ratio) and lets popart
// continue with the same normalization. RESETTING them disabled the clamp
// (cap=0×ratio=0) and let outliers blow up the envelope → σ shock window
// → -$185M eval loss (validated alpha-rl-6kghr 2026-05-31).
```

### 3.2 What stays the same

Everything else in `reset_session_state` is unchanged:
- Layer 1 CMDP summary slots (session_pnl, worst_pnl, dd_triggered, consec_loss, cooldown) — PREDICTIVE, correctly reset
- Layer 4 (Kelly) EMAs (win_rate, avg_win/loss, cumulative_dones, kelly_fraction) — PREDICTIVE, correctly reset
- Layer 3 (Inventory) EMAs (inventory_penalty_β, inventory_variance) — PREDICTIVE, correctly reset
- Defensive warmup arm (`RL_REGIME_TRANSITION_REMAINING_INDEX` = 500 steps)
- Regime observer state (dead_zone flag/duration/timeout, tail recency, recovery_factor, ε_recovery_live, prev_worst_pnl) — PREDICTIVE/transient, correctly reset
- Per-batch buffer memsets (session_pnl_per_batch_d, etc.)

### 3.3 Why not preserve popart.σ / .var / .mean themselves?

popart's Welford state (σ/var/mean) is ALREADY preserved across the boundary — it's not in the reset list. The bug was that env.max (which dominates σ_effective via `max(σ_welford, env.max)`) was being reset to 0, exposing the bare σ_welford = 1. The 4-line fix preserves env.max so σ_effective stays at train-calibrated level.

### 3.4 Why preserve clip_rate_ema?

`RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX` is the rate at which rewards were being clipped during train. The clamp's margin controller uses this to adjust the margin ratio adaptively (target clip rate ~5%). Resetting it to 0 makes the controller think it's never clipping → grows margin → defeats clamp purpose further. Preserve.

### 3.5 Will preserving NORMALIZATION cause its own problems?

Scenarios to consider:

| Scenario | Preserved env.max | New regime ACTUAL scale | Behavior |
|----------|------------------:|------------------------:|----------|
| Eval ≈ train scale | 57 | 57 | clamp works as in train, σ stable ✓ |
| Eval larger scale | 57 | 200 | clamp clips some real signal, but pos_max_ema adapts up over ~100 steps; σ grows smoothly |
| Eval smaller scale | 57 | 10 | clamp lets everything through (most rewards ≤ cap), pos_max_ema slowly adapts down; σ stays at 57 until rewards consistently smaller |

None of these are catastrophic. The current "reset to 0" scenario is uniquely bad because it produces an unbounded amplification rather than mild miscalibration.

## 4. Validation

### V1 — Local smoke
Run alpha_rl_train with fold-1 walk-forward at b=16, 1k train + 250 eval. Verify:
- Pre-eval pos_max_ema is non-zero (preserved)
- First eval step's clamp cap > 0
- popart σ stays within 5× of train-final value across all 250 eval steps
- eval_summary n_eval_trades_dropped = 0

### V2 — Cluster smoke
Same as v11/Phase A config (fold 1, b=1024, 20k+5k eval, seed 16962). Compare:
- eval_diag.jsonl first 50 steps: popart.σ should stay near train-final (~57), NOT spike to >4000
- eval_summary.json total_pnl_usd: should be dramatically less negative than -$185M (predicted: -$5M to -$30M range, but exact value unknown — that's the residual model overfit)

### V3 — Pearl update
Patch `pearl_adaptive_carryover_discipline` with the predictive-vs-normalization distinction.

## 5. Implementation phases

### Phase 1 — Code change (5 min)
Remove the 4 reset lines from `reset_session_state`. Update the comment to explain the new policy.

### Phase 2 — Local smoke (15 min)
Same fold-1 b=16 smoke as the eval-summary fix. Inspect eval_diag.jsonl popart sigma trajectory.

### Phase 3 — Cluster validation (~95 min)
Push, submit, wait, inspect.

### Phase 4 — Pearl update (5 min)
Refine `pearl_adaptive_carryover_discipline` with the predictive-vs-normalization distinction.

## 6. Open decisions

| # | Question | Recommendation |
|---|----------|---------------|
| 1 | Preserve ALL 4 EMAs, or just the 3 reward-clamp ones (skipping popart envelope)? | **Preserve all 4.** Without env.max, the clamp may suppress the worst overshoots but the envelope itself still re-bootstraps from 0 and can spike on a single large reward. Both layers of defense matter. |
| 2 | Add a controller-level invariant test? | **Yes, V1 local smoke** — assert `popart.σ` after eval step 50 is within 5× of train-final. Cheap to verify. |
| 3 | Update pearl_adaptive_carryover_discipline NOW or wait for cluster confirmation? | **After cluster.** Update the pearl WITH the empirical evidence of the fix working. |
| 4 | Risk that preserving env.max masks true regime shift in eval data? | **Low.** EMAs adapt over time; preserving the train-final value just means eval starts calibrated rather than at 0. If eval has truly larger scale, env.max grows there too, just from 57 instead of 0. |

## 7. Done means

- Lines removed from `reset_session_state`, commit made
- Local b=16 fold-1 smoke shows popart.σ bounded across eval
- Cluster b=1024 fold-1 run completes, eval_diag.jsonl shows σ never exceeds 5× train-final
- eval_summary.json total_pnl significantly less negative than -$185M (number TBD; if still very negative, that's true Layer-3 overfit work)
- Pearl `pearl_adaptive_carryover_discipline` updated with the refined rule

## 8. Risks

- **Risk A (low):** If train-final pos_max_ema was somehow off (e.g. from train tail event), preserving it carries the mis-calibration into eval. Mitigation: EMA decays over eval steps anyway.
- **Risk B (medium):** If eval shock losses come from MULTIPLE compounding factors beyond just clamp-disable (e.g. Kelly reset compounded with mis-calibrated normalization), preserving only normalization may only partially fix. Mitigation: cluster validation will measure exact remaining gap.
- **Risk C (low):** Pearl rule refinement might be over-broad if there are other PREDICTIVE-vs-NORMALIZATION EMAs we haven't catalogued. Mitigation: only apply the rule to the 4 EMAs we're changing today; document the principle but require future cases to verify per-EMA.
