# pos_max_ema Cold-Start Cascade Fix — ISV-Driven Adaptive Redesign

**Date:** 2026-05-31
**Status:** Draft (post-cluster-validation diagnosis)
**Parent docs:**
- `2026-05-31-eval-boundary-normalization-preservation-design.md`
- `2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md`
**Evidence:** alpha-rl-4xmxm step 37 cascade (env.max=11375, pos_max_ema=7074)
**Pearls referenced:**
- `pearl_controller_anchors_isv_driven` — every controller anchor must be ISV-driven
- `pearl_adaptive_carryover_discipline` — refined twice already
- `feedback_isv_for_adaptive_bounds` — adaptive bounds live in ISV
- `feedback_adaptive_not_tuned` — fixes are signal-driven, not tuned constants
- `pearl_first_observation_bootstrap` — needs revision (see §7)
- `pearl_multiplicative_controllers_need_bounded_step_and_noise_floor`

---

## 1. Motivation

The current `rl_reward_clamp_controller` has a structural cascade vulnerability:

```c
// Current (BUG):
if (ema_prev == 0.0f) {
    ema_new = pos_max;       // ← first-observation bootstrap, UNBOUNDED
} else {
    ema_new = (1-α) × prev + α × pos_max;  // Wiener-α blend
}
```

The first positive reward of training BECOMES `pos_max_ema` directly. Because reward magnitudes are scaled-by-controller (initial scale ≈ 1.0 bootstrap), a single fat-tail trade close at step ~5 can seed `pos_max_ema = 947` (alpha-rl-4xmxm). From there:

1. `clamp_win = MARGIN × pos_max_ema = 1.5 × 947 = 1421` for next step
2. Subsequent rewards (could be 1000+ in magnitude) pass through largely unclamped
3. popart envelope `env.max` captures the post-clamp magnitudes (now bounded only by the inflated clamp_win)
4. σ explodes 11375× at step 37 because env.max = max-abs across batch = 11375

Recovery via slow-decay over ~3000 steps (verified: env.max 11375 → 49 over 3069 steps), but during the spike PPO gradients are mis-scaled, V regression targets escape C51 support, and the policy trains on a distorted signal.

The cascade is **structural** — any controller that bootstraps to the first observation has this property. The fix attacks the root: eliminate the special-case bootstrap branch by initializing the EMA to a conservative non-zero value, so the controller has ONE coherent update rule from cold-start onward.

## 2. Goals

### G1 — Eliminate the cold-start cascade
At training step 0, `pos_max_ema` and `neg_max_ema` initialized to conservative neutral values, NOT 0. First-observation special-case branch removed. Single Wiener-α + rate-cap update path applies uniformly.

### G2 — All adaptive bounds ISV-driven
Per `feedback_isv_for_adaptive_bounds`:
- Cold-start bootstrap value → ISV slot
- Per-step growth rate cap → ISV slot
- (Optional) Adaptive growth cap derived from signal Welford statistics → ISV-fed controller

### G3 — Apply the pattern to other affected controllers
Audit all `*_ema_update*.cu` and `rl_*_controller.cu` files for the same `if (ema_prev == 0.0f) ema_new = observation;` anti-pattern. Fix uniformly.

### G4 — Compatible with parent eval-boundary fix
Preservation of `RL_POS_SCALED_REWARD_MAX_EMA_INDEX` across `reset_session_state` (parent fix) remains in place. Cold-start ema initialization happens ONCE per run via `with_controllers_bootstrapped`, not per fold.

### G5 — Non-goals
- Don't refactor twice-per-step controller invocation (separate concern; orthogonal)
- Don't change MARGIN adaptation (already ISV-driven, doesn't need fix)
- Don't preserve `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` semantics (already correct via addendum A3)

## 3. Design

### 3.1 ISV slot allocation

Three new slots:

| Slot | Name | Initial value | Description |
|---|---|---:|---|
| 717 | `RL_POS_MAX_EMA_COLD_START_INDEX` | 1.0 | Cold-start initialization for `pos_max_ema` (and `neg_max_ema` since their semantics are mirror-image). Conservative magnitude assumption: "1.0 = unit scaled reward". `pos_max_ema = cold_start` means clamp_win starts at `MARGIN × cold_start = 1.5` (heavy clipping until adaptation). |
| 718 | `RL_POS_MAX_EMA_GROWTH_CAP_BASE_INDEX` | 1.225 | Base per-invocation growth cap (√1.5 to compensate for twice-per-step invocation, netting ~1.5×/step). |
| 719 | `RL_POS_MAX_EMA_GROWTH_CAP_CV_GAIN_INDEX` | 0.0 (=disabled) | Multiplicative gain applied to coefficient-of-variation of reward magnitude. When non-zero, growth_cap becomes adaptive: `base + cv_gain × min(CV, 1.0)`. Set to 0 to disable adaptive layer; controller falls back to fixed base. |

Note: slot 716 (`RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX`) is already in use from the addendum. The new slots start at 717.

### 3.2 Bootstrap entries (`with_controllers_bootstrapped`)

```rust
// Existing — keep:
(crate::rl::isv_slots::RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX, 0.0_f32),

// NEW — replace zero-bootstrap with cold-start ISV slot value:
(crate::rl::isv_slots::RL_POS_MAX_EMA_COLD_START_INDEX,            1.0_f32),
(crate::rl::isv_slots::RL_POS_MAX_EMA_GROWTH_CAP_BASE_INDEX,       1.225_f32),
(crate::rl::isv_slots::RL_POS_MAX_EMA_GROWTH_CAP_CV_GAIN_INDEX,    0.0_f32),

// And use the ISV value to bootstrap the EMA slots themselves (read at init):
// Need a small init kernel that reads RL_POS_MAX_EMA_COLD_START_INDEX and writes
// pos/neg slots, OR set the bootstrap entries directly:
(crate::rl::isv_slots::RL_POS_SCALED_REWARD_MAX_EMA_INDEX,  1.0_f32),  // was 0.0
(crate::rl::isv_slots::RL_NEG_SCALED_REWARD_MAX_EMA_INDEX,  1.0_f32),  // was 0.0
```

The bootstrap value duplication (cold_start ISV + direct EMA bootstrap) is intentional: the ISV slot exposes the constant for runtime tuning + diagnostic emission, while the EMA bootstrap entry ensures the slot is populated at training start. Future enhancement: a `rl_ema_cold_start_init` kernel runs once at init to copy cold_start → all EMA slots, eliminating duplication.

### 3.3 Controller logic (`rl_reward_clamp_controller.cu`)

```c
// Read ISV-driven adaptive bounds.
const float growth_base    = isv[RL_POS_MAX_EMA_GROWTH_CAP_BASE_INDEX];
const float growth_cv_gain = isv[RL_POS_MAX_EMA_GROWTH_CAP_CV_GAIN_INDEX];

// Adaptive growth cap from Welford coefficient of variation (Goal G2).
// When cv_gain = 0, growth_cap = base (non-adaptive fallback).
// When cv_gain > 0 and signal is volatile (high CV), growth_cap relaxes,
// allowing faster adaptation to genuine regime shifts.
const float wf_count = isv[RL_REWARD_MAG_VAR_COUNT_INDEX];
float growth_cap;
if (wf_count > 1.0f && growth_cv_gain > 0.0f) {
    const float wf_m2   = isv[RL_REWARD_MAG_VAR_M2_INDEX];
    const float wf_mean = isv[RL_REWARD_MAG_VAR_MEAN_INDEX];
    const float wf_var  = wf_m2 / (wf_count - 1.0f);
    const float wf_cv   = (wf_mean > 1e-6f) ? sqrtf(wf_var) / wf_mean : 0.0f;
    growth_cap = growth_base + growth_cv_gain * fminf(wf_cv, 1.0f);
} else {
    growth_cap = growth_base;
}

float ema_new = ema_prev;
if (pos_max > 0.0f) {
    // SINGLE update rule — no first-observation special case (G1).
    // ema_prev was initialized to cold_start (≥ 1.0) at run init, so
    // (1-α) × ema_prev + α × pos_max is always well-defined, and rate-cap
    // applies uniformly.
    const float a = fmaxf(alpha, wiener_floor);
    float ema_raw = (1.0f - a) * ema_prev + a * pos_max;
    const float max_grow = ema_prev * growth_cap;
    ema_new = fminf(ema_raw, max_grow);
    isv[RL_POS_SCALED_REWARD_MAX_EMA_INDEX] = ema_new;
}
// Dry-spell skip preserved: when pos_max=0, ema stays at prev value.
```

Same treatment for `neg_max_ema` block.

### 3.4 Math properties

**Cold-start uniform bound** (no observation can blow up the EMA):
- Worst case: pos_max = ∞. Wiener-α at floor 0.4: ema_raw = 0.6 × 1.0 + 0.4 × ∞ = ∞. Rate-cap kicks in: max_grow = 1.0 × 1.225 = 1.225. ema_new = min(∞, 1.225) = 1.225.
- After N invocations of max-out observations: ema ≤ 1.0 × 1.225^N.
- Twice-per-step invocation: per-step max growth ≤ 1.225² = 1.50.
- After 22 training steps: ema ≤ 1.5^22 ≈ 11,500 (sufficient for realistic magnitudes).
- After 30 training steps: ema ≤ 1.5^30 ≈ 191,750 (saturates ANY realistic magnitude).

**Steady-state convergence** (no change from existing behavior):
- pos_max stable around magnitude M: ema converges to M via standard Wiener-α blend in ~5-10 invocations (rate-cap doesn't engage when growth is already small).

**Decay path** (no change):
- pos_max smaller than prior ema: ema_raw < ema_prev. max_grow > ema_prev > ema_raw. Rate-cap doesn't engage. ema decreases via Wiener-α as before.

**Adaptive layer property** (when cv_gain > 0):
- Stable regime (CV ≈ 0): growth_cap = base = 1.225, tight bound.
- Volatile regime (CV ≈ 1): growth_cap = base + cv_gain × 1.0 = up to 1.225 + 2.0 = 3.225, relaxed bound.
- Smooth interpolation: no discontinuous regime switching.

### 3.5 Cluster scenario validation

| Scenario | Before fix | After fix | Verdict |
|---|---|---|---|
| Cold-start tail at step 5 (alpha-rl-4xmxm: pos=947) | pos_ema bootstraps to 947 → cascade | pos_ema = min(0.6+378.8, 1.225) = 1.225 → bounded | ✓ cascade eliminated |
| Mid-train spike (alpha-rl-jnct8 step 3895: pos=724 from ema=112) | pos_ema 112→834 in 5 steps (uncapped Wiener) | pos_ema 112 × 1.5^N → bounded growth | ✓ rate-cap active |
| Eval boundary preservation | ema reset to 0 | ema preserved via parent fix | ✓ unchanged |
| Reward magnitude legitimately huge ($50k+ trades) | model trains on huge unbounded rewards | model adapts in ~30 steps via geometric growth | ✓ acceptable warmup tradeoff |
| Reward magnitude legitimately tiny ($0.10 ticks) | EMA decays to typical magnitude via Wiener | EMA decays from 1.0 via Wiener; rate-cap doesn't restrict | ✓ unchanged |
| Volatile regime (high CV) with cv_gain enabled | rate-cap stuck at base | rate-cap relaxes adaptively | ✓ new capability |

### 3.6 Generalization — the "conservative-bootstrap" pattern

This fix exposes a general principle for ALL adaptive EMAs that gate signal magnitudes:

> **NORMALIZATION EMAs should bootstrap to a CONSERVATIVE neutral value (one that produces SAFE downstream behavior), never to a sentinel like 0 that requires special-case bootstrap-to-observation handling.**

For `pos_max_ema` specifically:
- Bootstrap value = `cold_start = 1.0` (= MIN_WIN, the static cold-start cap floor)
- Conservative: `clamp_win = MARGIN × 1.0 = 1.5` → rewards heavily clipped until data accumulates
- Safe: σ stays bounded, V regression targets in C51 support [-3, +1], PPO gradients well-scaled

For other EMAs needing the same treatment:
- `neg_max_ema` (slot 489) — same pattern, mirror-image
- `inventory_variance_ema` (slot 675) — bootstrap to small positive sentinel (controller has dead-signal hold but cold-start could still cascade)
- `avg_win_usd_ema` (slot 678), `avg_loss_usd_ema` (slot 679) — bootstrap to neutral (e.g. 1.0 USD) for Kelly safety
- `mean_abs_pnl_ema` — feeds reward_scale, similar cascade risk

### 3.7 Why this design over alternatives

| Alternative | Why not chosen |
|---|---|
| Cap per-invocation only (B-1, current) | Doesn't address cold-start; rate-cap path skipped at bootstrap |
| Cap with baseline = max(ema_prev, MIN_WIN) | Works but adds complex semantics; doesn't generalize |
| Add separate "step-end-snapshot" ISV slot | Robust to invocation count drift but adds slot, new copy kernel, more state |
| Refactor twice-per-step to once-per-step | Bigger change, requires invocation audit; orthogonal concern |
| Median/percentile envelope | Computationally expensive; max IS the right statistic for tail bounding |

B-2 (this spec) is the minimal change that eliminates the structural problem: ONE update rule, ISV-driven, math-verifiable, compatible with all existing fixes.

## 4. Implementation phases

### Phase 1 — ISV slot allocation + bootstrap (5 min)
- Add slots 717, 718, 719 to `isv_slots.rs`
- Bump `RL_SLOTS_END = 720`
- Add bootstrap entries (5 new lines in `with_controllers_bootstrapped`)
- Change `RL_POS_SCALED_REWARD_MAX_EMA_INDEX` bootstrap value 0.0 → 1.0
- Change `RL_NEG_SCALED_REWARD_MAX_EMA_INDEX` bootstrap value 0.0 → 1.0
- Bump array length annotation
- cargo check

### Phase 2 — Controller kernel update (15 min)
- `rl_reward_clamp_controller.cu`: remove first-observation bootstrap branch
- Add Welford CV computation for adaptive growth cap
- Replace `POS_MAX_EMA_MAX_GROWTH_PER_STEP` define with ISV reads
- Update comments to reference spec + pearl
- Same treatment for `neg_max_ema` block
- cargo build (recompiles cubins)

### Phase 3 — Diag emission (5 min)
- Emit `pos_max_ema_growth_cap` (resolved value after CV-adaptive computation) to diag
- Emit `pos_max_ema_cold_start` (the configured ISV value) to diag
- Update `eval_diag_emission` test EXPECTED_LEAVES count (644 → 646 for two new leaves)

### Phase 4 — Audit + apply pattern (30 min)
- `grep -rn "if (ema_prev == 0.0f)" crates/ml-alpha/cuda/*.cu`
- For each match: bootstrap the slot to MIN value, remove the special-case branch
- Test each change with cargo check + local smoke

### Phase 5 — Local smoke validation (15 min)
- Run 800-train + 200-eval b=16 smoke (same as parent-fix smoke)
- Verify: `pos_max_ema` at step 1 = `cold_start = 1.0`, NOT first observation
- Verify: `pos_max_ema` growth per step ≤ 1.5 (or growth_cap if adaptive enabled)
- Verify: `env.max` does NOT spike above some sanity threshold early in training
- Verify: eval-boundary preservation continues to work

### Phase 6 — Cluster validation (~50 min)
- Submit 10k+2500 fold-1 b=1024 L40S at the new SHA
- Compare diag against alpha-rl-4xmxm:
  - Step 5: should NOT show pos_ema = 879; should show pos_ema ≈ 1.2 (cold-start adaptation just starting)
  - Step 37: should NOT show env.max = 11375; should show env.max ≤ some bounded value (1.5^N where N = step count × 2 invocations)
  - Eval[1]: σ preserved, warmed_flag = 1
- eval_summary total_pnl should be substantially less negative than alpha-rl-jnct8 -$106M

## 5. Validation gates

### V1 — Cold-start cascade eliminated
At training step 10, `pos_max_ema` ≤ 1.5^20 ≈ 3325 (theoretical max under twice-per-step worst case). In practice expect << 3325 because actual observations don't max-out the rate-cap every step.

### V2 — env.max bounded by clamp
At every training step t: `env.max[t] ≤ MARGIN × pos_max_ema[t-1] + small slack`. If `env.max` ever exceeds this, the clamp is NOT being applied (regression — investigate).

### V3 — Eval-boundary preservation
At eval[0]: `pos_max_ema[eval 0]` == `pos_max_ema[train end]` (parent fix). `env.max[eval 0]` == `env.max[train end]`. `reward_scale[eval 0]` ≈ `reward_scale[train end]` (within ±2% controller clamp; warmed_flag survives).

### V4 — σ behavior
σ trajectory across eval phase stays within 5× of train-final σ. NO 60× explosion seen in alpha-rl-jnct8/4xmxm.

### V5 — eval pnl improvement
Cluster total_pnl_usd ≥ -50% of alpha-rl-jnct8's -$106M (target: less negative than -$53M). Stretch: positive eval pnl.

## 6. Risks

- **Risk α (low)**: Heavy early-clipping for ~20 steps. Model trains on bounded rewards during warmup. Mitigation: this is standard RL early-training behavior. Bound rewards are LESS harmful than spiked rewards.
- **Risk β (low)**: Existing tests assuming `pos_max_ema` starts at 0. Mitigation: update tests in Phase 5. Run full test suite.
- **Risk γ (medium)**: CV-adaptive growth cap (Phase 2 optional layer) could interact unexpectedly with MARGIN adaptation (both partially driven by clip-rate proxies). Mitigation: ship Phase 2 with cv_gain=0 initially; enable via ISV tuning after validation.
- **Risk δ (low)**: Other controllers with the same bootstrap-cascade anti-pattern (audit Phase 4). Mitigation: explicit grep + audit + same-pattern fix.
- **Risk ε (medium)**: The hardcoded cold_start = 1.0 assumes scaled-reward unit magnitude is meaningful. If reward_scale at run-start produces typical magnitudes far from 1.0, the bootstrap is mis-calibrated. Mitigation: ISV slot allows tuning; can be matched to reward_scale's bootstrap value (currently 1.0, consistent).

## 7. Pearl update

Revise `pearl_first_observation_bootstrap` (the broader pattern in the codebase) to add:

> **Adaptive EMAs whose values gate SIGNAL MAGNITUDES** (clamp anchors, normalization scales, envelope detectors) **must NOT bootstrap to the first observation**. They must bootstrap to a CONSERVATIVE neutral value (one that produces a TIGHT bound during cold-start). A single fat-tail observation at run-start otherwise seeds the EMA at a magnitude that takes O(thousands of decay steps) to recover from, and lets subsequent rewards escape the bound during the cascade window.
>
> **EMAs whose values are PREDICTIVE INPUTS** (Kelly's win_rate, avg_pnl, etc.) MAY bootstrap to first observation because they don't gate any downstream magnitude bound — they only inform decision logic that has its own safety mechanisms (e.g., Kelly fraction's MIN_TRADES_FOR_RELEASE gate).
>
> Cross-reference:
> - `pearl_popart_reset_at_eval_boundary_shocks_normalization` (boundary version)
> - `2026-05-31-pos-max-ema-cold-start-redesign.md` (cold-start version)

## 8. Done means

- 3 ISV slots allocated, bootstrapped, audited
- Controller kernel updated (single update rule, ISV-driven rate-cap)
- 2 EMA slot bootstrap values changed 0 → 1.0
- Audit complete (Phase 4) — all `if (ema_prev == 0.0f)` anti-patterns identified and fixed
- Local smoke shows env.max ≤ rate-cap bound at every step
- Cluster validation V1-V5 pass
- Pearl revised
- Spec committed with empirical evidence from cluster run
