# Eval-Boundary Addendum — reward_scale boot_floor + train-phase fat-tail clamp

**Date:** 2026-05-31
**Status:** Draft (post-cluster-validation diagnosis)
**Parent spec:** `2026-05-31-eval-boundary-normalization-preservation-design.md`
**Validation evidence:** alpha-rl-jnct8 (SHA `72684ed3e`, 10k train + 2500 eval, fold-1, b=1024)
**Pearls referenced:**
- `pearl_popart_reset_at_eval_boundary_shocks_normalization` (parent — needs revision)
- `pearl_adaptive_reward_clamp_from_positive_tail` (related)
- `pearl_welford_trade_count_is_step_not_trade` (related)

---

## 1. What the parent fix did and didn't do

Parent fix (commit `72684ed3e`): removed reset of `RL_POS_SCALED_REWARD_MAX_EMA_INDEX`, `RL_NEG_SCALED_REWARD_MAX_EMA_INDEX`, `RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX`, `RL_POPART_MAX_ABS_REWARD_EMA_INDEX` from `reset_session_state`.

**Worked as designed**: at alpha-rl-jnct8 eval[1] (vs baseline alpha-rl-6kghr eval[1]):

| Slot | 6kghr (no fix) | jnct8 (with fix) |
|---|---:|---:|
| popart.σ | 2.85 (reset) | **71.88** (preserved ✓) |
| env.max | 2.85 (reset) | **71.88** (preserved ✓) |
| pos_max_ema | 0.0 (reset) | **34.53** (preserved ✓) |

**Did NOT prevent σ explosion** — by eval[5], jnct8 σ=4451 vs baseline σ=103 (baseline σ peaked at 3233 by step 10). The catastrophe shifted but didn't disappear.

## 2. Issue A — reward_scale snaps to 0.10 (the actual eval-boundary catastrophe)

### 2.1 Mechanism (verified from jnct8 diag)

| Slot | TRAIN-END (9999) | EVAL[1] |
|---|---:|---:|
| `RL_REWARD_SCALE_INDEX` (406) | 0.0046 | **0.10** |
| `RL_CUMULATIVE_DONES_INDEX` (660) | 738,821 | **100** |

The ACTIVE controller path is `rl_fused_controllers.cu:586-625` (NOT the standalone `rl_reward_scale_controller.cu` which is legacy and reads the wrong slot). The fused controller's bootstrap-floor gate:

```c
const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX];      // slot 660
const float min_trades  = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX]; // slot 661
const float boot        = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX]; // = 1.0
const float boot_floor  = (trade_count < min_trades)
    ? boot * BOOTSTRAP_FRACTION_FLOOR  // 1.0 * 0.1 = 0.10
    : scale_min;                       // ~1e-4
// later:
out = fmaxf(boot_floor, fminf(out, REWARD_SCALE_MAX));
```

The parent fix's `reset_session_state` resets `RL_CUMULATIVE_DONES_INDEX` (slot 660) to 0 ("re-enter warmup"). At eval[1], cumulative_dones is 100 (1 step × b=1024 × ~10% done rate), below the `RL_MIN_TRADES_FOR_RELEASE_INDEX` threshold (default ~700). boot_floor evaluates to 0.10. reward_scale's preserved-and-otherwise-valid value 0.0046 gets clamped **upward** to 0.10. Result: eval rewards are 22× larger in magnitude than train rewards.

### 2.2 Why this dominates σ behavior

- Train: typical scaled reward magnitude ~0.5 (rewards × 0.0046 × clamp)
- Eval: typical scaled reward magnitude ~11 (rewards × 0.10 × clamp)
- env.max fast-up captures the new magnitude wholesale → 22× scaled rewards beat the preserved env.max=72 within steps
- σ_effective = max(σ_welford, env.max) tracks env.max → explodes to 4451 by eval[5]

The parent fix preserved env.max=72, but this was overwhelmed by reward_scale's 22× upward jump. Both `reset to 0` (baseline) and `preserve at 72` (jnct8) reach σ ~ 4000+ within 10 steps.

### 2.3 Why the dual-purpose counter creates a conflict

`RL_CUMULATIVE_DONES_INDEX` (slot 660) is consumed by TWO controllers with opposing reset semantics:

| Consumer | Uses counter for | Reset semantics at boundary |
|---|---|---|
| reward_scale boot_floor | NORMALIZATION-startup safeguard (skip floor after enough trades) | should **preserve** — counter has earned its way past floor |
| Kelly resurrection gate (slot 681 threshold) | PREDICTIVE-safety gate (don't size aggressively without eval history) | should **reset** — train history doesn't predict eval regime |

Per `pearl_adaptive_carryover_discipline` framework refined in parent spec: the SAME slot has both NORMALIZATION and PREDICTIVE consumers. Single preserve-or-reset can't satisfy both.

### 2.4 Fix options

| Option | Description | Tradeoff |
|---|---|---|
| **A1** | Preserve slot 660 at boundary | reward_scale floor stays inactive ✓; Kelly's gate opens immediately at eval[1] with neutral EMAs → may size aggressively without eval data |
| **A2** | Add new ISV slot `RL_RUN_CUMULATIVE_DONES_INDEX` (never resets); reward_scale reads it. Keep slot 660 for Kelly (gets reset). | Clean separation; requires new slot, new accumulator, kernel update, bootstrap entry |
| **A3** | Add new ISV slot `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (set once at run-startup after first 100 dones, never resets). reward_scale's boot_floor reads this flag instead of comparing counter. | Cleanest semantic — "is this initial cold-start?" is the actual question; minimal logic change |
| **A4** | Preserve `RL_REWARD_SCALE_INDEX` (slot 406) and add early-return in fused controller when `prev != 0 && prev != boot && trade_count < min_trades` (skip floor for already-calibrated scales) | Code-level guard; doesn't require new slot; but adds branch complexity |

### 2.5 Recommendation: A3 (warmed-flag) is cleanest

A3 directly encodes the semantic the boot_floor protection was designed for: "is this run's reward_scale controller still in initial cold-start, or has it already converged once?". After that initial convergence, the floor protection serves no purpose — the controller has demonstrated it can adapt to real magnitudes without crashing. Whether we're at a regime boundary or mid-stream doesn't matter.

Implementation:
1. Allocate `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (next free slot)
2. Bootstrap to 0 in `with_controllers_bootstrapped`
3. In `rl_fused_controllers.cu`'s reward_scale block, set flag to 1 once `trade_count >= min_trades` AND keep it at 1 thereafter
4. `boot_floor = (warmed_flag == 0) ? boot * BOOTSTRAP_FRACTION_FLOOR : scale_min`
5. **Critical**: do NOT add this flag to `reset_session_state`'s reset list — it must survive boundaries

This solves Issue A without touching the Kelly counter semantics and without preserving slot 660.

### 2.6 Why A1 (preserve slot 660) is risky

Kelly's safety threshold lives at slot 681 and reads from the same counter (slot 660). Preserving slot 660 means at eval[1] Kelly sees `cumulative_dones=738,821 ≫ kelly_min_trades`. The gate opens immediately — Kelly applies its formula with `win_rate_ema=0.5` (reset, neutral), `avg_win/loss=0` (reset). Likely the Kelly controller has a guard for avg=0 (dead-signal hold), but the policy mass already there has trade size based on Kelly's bootstrap (1.0 = full size). If guard works → Kelly stays at 1.0 anyway until eval data accumulates, same as if gate were closed. So A1 might be SAFE — but requires verifying the Kelly controller's behavior with mixed (cumulative_dones high, predictive EMAs reset) state.

### 2.7 Why A2 (separate counter) is overkill

Two counters track the same underlying event (closed trades). Two accumulators means two GPU writes per step, two bootstrap entries, two diag emissions. The semantic difference (NORMALIZATION-startup vs PREDICTIVE-safety) doesn't justify the duplication. A3 captures the same semantic via a 1-bit flag.

## 3. Issue B — train-phase fat-tail spikes (pre-existing, lower priority)

### 3.1 Mechanism (verified from jnct8 diag at step 3895)

| step | pre_clamp_max | pos_max_ema (post) | clamp_win cap (est.) | env.max | popart.σ |
|---:|---:|---:|---:|---:|---:|
| 3880 | 9.4 | 9.65 | ~14 | 87.5 | 87.5 |
| 3890 | 109.1 | 112.6 | ~14 | 121.7 | 121.7 |
| **3895** | **724.3** | **834.5** | ~170 (still less than 724) | **1306.2** | **1306.2** |
| 3950 | 19.3 | 18.79 | ~1250 (over-grown) | 765 | 765 |

Sequence:
1. Step 3890: small fat-tail event (109 magnitude scaled). pos_max_ema bootstraps 9 → 112 via Wiener-α first-observation logic.
2. Step 3895: huge fat-tail event (724 magnitude scaled). pos_max_ema blends 112 + α×724 → 834.
3. clamp_win = MARGIN × prior pos_max_ema. At step 3895's apply moment, prior was ~112 → clamp_win ≈ 170. The 724 magnitude reward exceeds the cap but the controller had already adapted away from the bound.
4. env.max fast-up captures the post-clamp magnitude (around 1306, suggesting post-clamp ≈ 1306 across some account in batch).
5. Slow-decay returns env.max to ~50 over 600 steps (α≈0.005, math: 1306 × 0.9947^600 ≈ 50).

### 3.2 Why this is a design tension

Documented in `pearl_adaptive_reward_clamp_from_positive_tail`: the static cap `[-3, +1]` over-clipped 85% of steps but bounded extremes. The adaptive Wiener-α growth (α=0.4 floor) of pos_max_ema lets clamp_win FOLLOW the magnitude up — losing the bound exactly when it's needed.

The system has rate limits on reward_scale (2% per step) but NOT on pos_max_ema. The Wiener-α floor=0.4 admits 40% of any single observation directly into the EMA.

### 3.3 Possible fixes (NOT addressed in this addendum)

| Option | Approach | Tradeoff |
|---|---|---|
| B-1 | Add per-step rate cap on pos_max_ema growth (e.g. ≤ 1.5× prior, same Schulman pattern as reward_scale) | Slower adaptation to genuine regime shifts |
| B-2 | Use median/percentile instead of max for envelope | More robust but loses "worst-case" signal |
| B-3 | Hard cap on env.max growth per step | Simple, but adds another magic constant |

### 3.4 Recommendation: defer Issue B

The train-phase spikes recover via slow-decay before they propagate to eval (provided no spike fires within ~600 steps of eval boundary). Train-phase mid-stream policy disruption is real but documented as design tension, not as a blocking bug. PPO's per-batch advantage normalization (Phase 4.5) partially compensates.

**Action: log the issue in the addendum but don't fix in this PR.** Track as a follow-up.

## 4. Validation plan for Fix A3

### V1 — Implement
- Allocate `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (next free slot)
- Bootstrap to 0 in `with_controllers_bootstrapped`
- Modify `rl_fused_controllers.cu`'s reward_scale block: replace `(trade_count < min_trades)` test with `(warmed_flag == 0)`; set flag once trade_count crosses threshold
- Do NOT add to `reset_session_state` reset list
- Update diag to emit the flag

### V2 — Local smoke
- 800 train + 200 eval, b=16, fold-1 n_folds=3 (same as parent fix smoke)
- Verify: at eval[1], `reward_scale` ≈ train-end value (±2%) — NOT 0.10
- Verify: warmed_flag = 1 at train-end AND eval[1]

### V3 — Cluster validation
- Submit fold-1 walk-forward 10k+2500 (same scale as jnct8)
- Verify: eval[0..50] σ stays within 5× of train-end σ (no 60× explosion)
- Compare eval_summary total_pnl_usd vs jnct8 and 6kghr

## 5. Pearl revision needed

`pearl_popart_reset_at_eval_boundary_shocks_normalization`:
- Add: "The eval-boundary catastrophe has TWO contributing mechanisms — (1) env.max/clamp EMA reset (fixed by 72684ed3e), and (2) reward_scale snapping to bootstrap_floor=0.10 because cumulative_dones counter (slot 660) is reset by reset_session_state, triggering rl_fused_controllers' bootstrap-fraction-floor protection. Mechanism 2 dominates: scale jumps 22× upward, scaled rewards grow 22×, env.max captures them and σ explodes regardless of starting point."
- Note the dual-purpose slot 660 conflict (NORMALIZATION-startup vs PREDICTIVE-safety)
- Cross-reference `pearl_welford_trade_count_is_step_not_trade`

## 6. Done means

- Warmed-flag ISV slot allocated; bootstrap + emit + read paths wired
- `rl_fused_controllers.cu` reward_scale block uses warmed_flag, not trade_count comparison
- Local smoke passes V2
- Cluster validation V3: eval[0..50] σ within 5× of train-end σ, total eval pnl substantially less negative than alpha-rl-jnct8
- Pearl revised
- Issue B (train-phase fat-tail spikes) logged as a separate follow-up

## 7. Risks

- **Risk α (medium)**: The warmed_flag could fire spuriously in early TRAIN if `min_trades` is set very low. Mitigation: keep min_trades high enough that flag only sets after meaningful warmup (current default ~700 dones is fine).
- **Risk β (low)**: Other controllers may also have boot_floor-style protections gated on cumulative_dones. If so, this fix only solves reward_scale; other controllers still snap at boundary. Mitigation: grep for `CUMULATIVE_DONES_INDEX` reads in all `*.cu` and audit each consumer's reset semantics.
- **Risk γ (low)**: σ may still explode at eval boundary from OTHER mechanisms not yet diagnosed (e.g. PPO ratio clamp interaction, action mask change). Mitigation: V3 cluster validation will reveal residual gaps.
