fix(rl): eval-boundary addendum — reward_scale warmed_flag + pos_max_ema rate-cap
The parent eval-boundary fix (72684ed3e) preserved env.max/clamp EMAs but
left the σ explosion intact. Diagnosis from alpha-rl-jnct8 (10k+2500 eval,
fold-1, -$106M eval pnl, σ → 4451 by eval step 5) revealed two pre-existing
mechanisms compounding the eval shock:
ISSUE A — reward_scale snaps to 0.10 at boundary
================================================
rl_fused_controllers' reward_scale block had a bootstrap-fraction-floor
gate: `(cumulative_dones < min_trades) → boot_floor = 0.10`. The intent
was cold-start protection (prevent scale crash before any closed-trade
ground truth). But cumulative_dones (slot 660) is reset by
reset_session_state for Kelly's PREDICTIVE-warmup purpose. At fold
boundary the gate fires again, clamping the preserved scale (0.0046 in
jnct8) UP to 0.10 — a 22× upward jump. Eval rewards are then 22× larger,
overwhelming env.max preservation; σ explodes regardless.
FIX A — decouple via monotonic warmed_flag (slot 716, never reset).
Set ONCE when cumulative_dones first crosses min_trades; persists across
all subsequent fold boundaries. boot_floor reads the flag instead of
re-evaluating trade_count. Math: at cold-start flag=0 → boot_floor=0.10
(original protection preserved). Post-warmup flag=1 → boot_floor=scale_min
(~1e-4) → preserved scale survives boundaries.
ISSUE B — pos_max_ema growth unbounded (train-phase fat-tail spike)
====================================================================
Pre-existing fat-tail behavior: at alpha-rl-jnct8 step 3895 a single
account had pre_clamp scaled reward 724. The clamp's Wiener-α EMA
(α=0.4 floor) admitted 40% of the observation, jumping pos_max_ema
112 → 834 in 5 steps. clamp_win = MARGIN × pos_max_ema followed
magnitude up rather than bounding it; env.max captured the unclamped
reward (121 → 1306). Recovery via slow-decay over 600 steps, but
during the spike PPO gradients were mis-scaled.
FIX B — asymmetric per-step growth cap on pos_max_ema (1.5× max).
Same Schulman-bounded-step pattern as reward_scale's 2% per-step
movement clamp. Decreases unbounded (allows fast recovery from spike).
Bootstrap path unchanged (ema_prev=0 → ema_new=pos_max, no cap).
Math: 5-step max growth = 1.5^5 ≈ 7.6× vs prior uncapped 7.4× in
practice — similar steady-state, bounded transient.
Local validation (b=16, 800+200 fold-1):
- σ preserved across boundary (51.9 → 51.4 at eval[1])
- σ stays bounded in 23-57 range across full eval phase (no 60× explosion)
- scale=0.10 stable across boundary
- warmed_flag stays 0 at b=16 (cumulative_dones never crosses min_trades
at this scale — flip behavior tested at cluster b=1024)
Spec: docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,179 @@
|
||||
# Eval-Boundary Addendum — reward_scale boot_floor + train-phase fat-tail clamp
|
||||
|
||||
**Date:** 2026-05-31
|
||||
**Status:** Draft (post-cluster-validation diagnosis)
|
||||
**Parent spec:** `2026-05-31-eval-boundary-normalization-preservation-design.md`
|
||||
**Validation evidence:** alpha-rl-jnct8 (SHA `72684ed3e`, 10k train + 2500 eval, fold-1, b=1024)
|
||||
**Pearls referenced:**
|
||||
- `pearl_popart_reset_at_eval_boundary_shocks_normalization` (parent — needs revision)
|
||||
- `pearl_adaptive_reward_clamp_from_positive_tail` (related)
|
||||
- `pearl_welford_trade_count_is_step_not_trade` (related)
|
||||
|
||||
---
|
||||
|
||||
## 1. What the parent fix did and didn't do
|
||||
|
||||
Parent fix (commit `72684ed3e`): removed reset of `RL_POS_SCALED_REWARD_MAX_EMA_INDEX`, `RL_NEG_SCALED_REWARD_MAX_EMA_INDEX`, `RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX`, `RL_POPART_MAX_ABS_REWARD_EMA_INDEX` from `reset_session_state`.
|
||||
|
||||
**Worked as designed**: at alpha-rl-jnct8 eval[1] (vs baseline alpha-rl-6kghr eval[1]):
|
||||
|
||||
| Slot | 6kghr (no fix) | jnct8 (with fix) |
|
||||
|---|---:|---:|
|
||||
| popart.σ | 2.85 (reset) | **71.88** (preserved ✓) |
|
||||
| env.max | 2.85 (reset) | **71.88** (preserved ✓) |
|
||||
| pos_max_ema | 0.0 (reset) | **34.53** (preserved ✓) |
|
||||
|
||||
**Did NOT prevent σ explosion** — by eval[5], jnct8 σ=4451 vs baseline σ=103 (baseline σ peaked at 3233 by step 10). The catastrophe shifted but didn't disappear.
|
||||
|
||||
## 2. Issue A — reward_scale snaps to 0.10 (the actual eval-boundary catastrophe)
|
||||
|
||||
### 2.1 Mechanism (verified from jnct8 diag)
|
||||
|
||||
| Slot | TRAIN-END (9999) | EVAL[1] |
|
||||
|---|---:|---:|
|
||||
| `RL_REWARD_SCALE_INDEX` (406) | 0.0046 | **0.10** |
|
||||
| `RL_CUMULATIVE_DONES_INDEX` (660) | 738,821 | **100** |
|
||||
|
||||
The ACTIVE controller path is `rl_fused_controllers.cu:586-625` (NOT the standalone `rl_reward_scale_controller.cu` which is legacy and reads the wrong slot). The fused controller's bootstrap-floor gate:
|
||||
|
||||
```c
|
||||
const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX]; // slot 660
|
||||
const float min_trades = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX]; // slot 661
|
||||
const float boot = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX]; // = 1.0
|
||||
const float boot_floor = (trade_count < min_trades)
|
||||
? boot * BOOTSTRAP_FRACTION_FLOOR // 1.0 * 0.1 = 0.10
|
||||
: scale_min; // ~1e-4
|
||||
// later:
|
||||
out = fmaxf(boot_floor, fminf(out, REWARD_SCALE_MAX));
|
||||
```
|
||||
|
||||
The parent fix's `reset_session_state` resets `RL_CUMULATIVE_DONES_INDEX` (slot 660) to 0 ("re-enter warmup"). At eval[1], cumulative_dones is 100 (1 step × b=1024 × ~10% done rate), below the `RL_MIN_TRADES_FOR_RELEASE_INDEX` threshold (default ~700). boot_floor evaluates to 0.10. reward_scale's preserved-and-otherwise-valid value 0.0046 gets clamped **upward** to 0.10. Result: eval rewards are 22× larger in magnitude than train rewards.
|
||||
|
||||
### 2.2 Why this dominates σ behavior
|
||||
|
||||
- Train: typical scaled reward magnitude ~0.5 (rewards × 0.0046 × clamp)
|
||||
- Eval: typical scaled reward magnitude ~11 (rewards × 0.10 × clamp)
|
||||
- env.max fast-up captures the new magnitude wholesale → 22× scaled rewards beat the preserved env.max=72 within steps
|
||||
- σ_effective = max(σ_welford, env.max) tracks env.max → explodes to 4451 by eval[5]
|
||||
|
||||
The parent fix preserved env.max=72, but this was overwhelmed by reward_scale's 22× upward jump. Both `reset to 0` (baseline) and `preserve at 72` (jnct8) reach σ ~ 4000+ within 10 steps.
|
||||
|
||||
### 2.3 Why the dual-purpose counter creates a conflict
|
||||
|
||||
`RL_CUMULATIVE_DONES_INDEX` (slot 660) is consumed by TWO controllers with opposing reset semantics:
|
||||
|
||||
| Consumer | Uses counter for | Reset semantics at boundary |
|
||||
|---|---|---|
|
||||
| reward_scale boot_floor | NORMALIZATION-startup safeguard (skip floor after enough trades) | should **preserve** — counter has earned its way past floor |
|
||||
| Kelly resurrection gate (slot 681 threshold) | PREDICTIVE-safety gate (don't size aggressively without eval history) | should **reset** — train history doesn't predict eval regime |
|
||||
|
||||
Per `pearl_adaptive_carryover_discipline` framework refined in parent spec: the SAME slot has both NORMALIZATION and PREDICTIVE consumers. Single preserve-or-reset can't satisfy both.
|
||||
|
||||
### 2.4 Fix options
|
||||
|
||||
| Option | Description | Tradeoff |
|
||||
|---|---|---|
|
||||
| **A1** | Preserve slot 660 at boundary | reward_scale floor stays inactive ✓; Kelly's gate opens immediately at eval[1] with neutral EMAs → may size aggressively without eval data |
|
||||
| **A2** | Add new ISV slot `RL_RUN_CUMULATIVE_DONES_INDEX` (never resets); reward_scale reads it. Keep slot 660 for Kelly (gets reset). | Clean separation; requires new slot, new accumulator, kernel update, bootstrap entry |
|
||||
| **A3** | Add new ISV slot `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (set once at run-startup after first 100 dones, never resets). reward_scale's boot_floor reads this flag instead of comparing counter. | Cleanest semantic — "is this initial cold-start?" is the actual question; minimal logic change |
|
||||
| **A4** | Preserve `RL_REWARD_SCALE_INDEX` (slot 406) and add early-return in fused controller when `prev != 0 && prev != boot && trade_count < min_trades` (skip floor for already-calibrated scales) | Code-level guard; doesn't require new slot; but adds branch complexity |
|
||||
|
||||
### 2.5 Recommendation: A3 (warmed-flag) is cleanest
|
||||
|
||||
A3 directly encodes the semantic the boot_floor protection was designed for: "is this run's reward_scale controller still in initial cold-start, or has it already converged once?". After that initial convergence, the floor protection serves no purpose — the controller has demonstrated it can adapt to real magnitudes without crashing. Whether we're at a regime boundary or mid-stream doesn't matter.
|
||||
|
||||
Implementation:
|
||||
1. Allocate `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (next free slot)
|
||||
2. Bootstrap to 0 in `with_controllers_bootstrapped`
|
||||
3. In `rl_fused_controllers.cu`'s reward_scale block, set flag to 1 once `trade_count >= min_trades` AND keep it at 1 thereafter
|
||||
4. `boot_floor = (warmed_flag == 0) ? boot * BOOTSTRAP_FRACTION_FLOOR : scale_min`
|
||||
5. **Critical**: do NOT add this flag to `reset_session_state`'s reset list — it must survive boundaries
|
||||
|
||||
This solves Issue A without touching the Kelly counter semantics and without preserving slot 660.
|
||||
|
||||
### 2.6 Why A1 (preserve slot 660) is risky
|
||||
|
||||
Kelly's safety threshold lives at slot 681 and reads from the same counter (slot 660). Preserving slot 660 means at eval[1] Kelly sees `cumulative_dones=738,821 ≫ kelly_min_trades`. The gate opens immediately — Kelly applies its formula with `win_rate_ema=0.5` (reset, neutral), `avg_win/loss=0` (reset). Likely the Kelly controller has a guard for avg=0 (dead-signal hold), but the policy mass already there has trade size based on Kelly's bootstrap (1.0 = full size). If guard works → Kelly stays at 1.0 anyway until eval data accumulates, same as if gate were closed. So A1 might be SAFE — but requires verifying the Kelly controller's behavior with mixed (cumulative_dones high, predictive EMAs reset) state.
|
||||
|
||||
### 2.7 Why A2 (separate counter) is overkill
|
||||
|
||||
Two counters track the same underlying event (closed trades). Two accumulators means two GPU writes per step, two bootstrap entries, two diag emissions. The semantic difference (NORMALIZATION-startup vs PREDICTIVE-safety) doesn't justify the duplication. A3 captures the same semantic via a 1-bit flag.
|
||||
|
||||
## 3. Issue B — train-phase fat-tail spikes (pre-existing, lower priority)
|
||||
|
||||
### 3.1 Mechanism (verified from jnct8 diag at step 3895)
|
||||
|
||||
| step | pre_clamp_max | pos_max_ema (post) | clamp_win cap (est.) | env.max | popart.σ |
|
||||
|---:|---:|---:|---:|---:|---:|
|
||||
| 3880 | 9.4 | 9.65 | ~14 | 87.5 | 87.5 |
|
||||
| 3890 | 109.1 | 112.6 | ~14 | 121.7 | 121.7 |
|
||||
| **3895** | **724.3** | **834.5** | ~170 (still less than 724) | **1306.2** | **1306.2** |
|
||||
| 3950 | 19.3 | 18.79 | ~1250 (over-grown) | 765 | 765 |
|
||||
|
||||
Sequence:
|
||||
1. Step 3890: small fat-tail event (109 magnitude scaled). pos_max_ema bootstraps 9 → 112 via Wiener-α first-observation logic.
|
||||
2. Step 3895: huge fat-tail event (724 magnitude scaled). pos_max_ema blends 112 + α×724 → 834.
|
||||
3. clamp_win = MARGIN × prior pos_max_ema. At step 3895's apply moment, prior was ~112 → clamp_win ≈ 170. The 724 magnitude reward exceeds the cap but the controller had already adapted away from the bound.
|
||||
4. env.max fast-up captures the post-clamp magnitude (around 1306, suggesting post-clamp ≈ 1306 across some account in batch).
|
||||
5. Slow-decay returns env.max to ~50 over 600 steps (α≈0.005, math: 1306 × 0.9947^600 ≈ 50).
|
||||
|
||||
### 3.2 Why this is a design tension
|
||||
|
||||
Documented in `pearl_adaptive_reward_clamp_from_positive_tail`: the static cap `[-3, +1]` over-clipped 85% of steps but bounded extremes. The adaptive Wiener-α growth (α=0.4 floor) of pos_max_ema lets clamp_win FOLLOW the magnitude up — losing the bound exactly when it's needed.
|
||||
|
||||
The system has rate limits on reward_scale (2% per step) but NOT on pos_max_ema. The Wiener-α floor=0.4 admits 40% of any single observation directly into the EMA.
|
||||
|
||||
### 3.3 Possible fixes (NOT addressed in this addendum)
|
||||
|
||||
| Option | Approach | Tradeoff |
|
||||
|---|---|---|
|
||||
| B-1 | Add per-step rate cap on pos_max_ema growth (e.g. ≤ 1.5× prior, same Schulman pattern as reward_scale) | Slower adaptation to genuine regime shifts |
|
||||
| B-2 | Use median/percentile instead of max for envelope | More robust but loses "worst-case" signal |
|
||||
| B-3 | Hard cap on env.max growth per step | Simple, but adds another magic constant |
|
||||
|
||||
### 3.4 Recommendation: defer Issue B
|
||||
|
||||
The train-phase spikes recover via slow-decay before they propagate to eval (provided no spike fires within ~600 steps of eval boundary). Train-phase mid-stream policy disruption is real but documented as design tension, not as a blocking bug. PPO's per-batch advantage normalization (Phase 4.5) partially compensates.
|
||||
|
||||
**Action: log the issue in the addendum but don't fix in this PR.** Track as a follow-up.
|
||||
|
||||
## 4. Validation plan for Fix A3
|
||||
|
||||
### V1 — Implement
|
||||
- Allocate `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (next free slot)
|
||||
- Bootstrap to 0 in `with_controllers_bootstrapped`
|
||||
- Modify `rl_fused_controllers.cu`'s reward_scale block: replace `(trade_count < min_trades)` test with `(warmed_flag == 0)`; set flag once trade_count crosses threshold
|
||||
- Do NOT add to `reset_session_state` reset list
|
||||
- Update diag to emit the flag
|
||||
|
||||
### V2 — Local smoke
|
||||
- 800 train + 200 eval, b=16, fold-1 n_folds=3 (same as parent fix smoke)
|
||||
- Verify: at eval[1], `reward_scale` ≈ train-end value (±2%) — NOT 0.10
|
||||
- Verify: warmed_flag = 1 at train-end AND eval[1]
|
||||
|
||||
### V3 — Cluster validation
|
||||
- Submit fold-1 walk-forward 10k+2500 (same scale as jnct8)
|
||||
- Verify: eval[0..50] σ stays within 5× of train-end σ (no 60× explosion)
|
||||
- Compare eval_summary total_pnl_usd vs jnct8 and 6kghr
|
||||
|
||||
## 5. Pearl revision needed
|
||||
|
||||
`pearl_popart_reset_at_eval_boundary_shocks_normalization`:
|
||||
- Add: "The eval-boundary catastrophe has TWO contributing mechanisms — (1) env.max/clamp EMA reset (fixed by 72684ed3e), and (2) reward_scale snapping to bootstrap_floor=0.10 because cumulative_dones counter (slot 660) is reset by reset_session_state, triggering rl_fused_controllers' bootstrap-fraction-floor protection. Mechanism 2 dominates: scale jumps 22× upward, scaled rewards grow 22×, env.max captures them and σ explodes regardless of starting point."
|
||||
- Note the dual-purpose slot 660 conflict (NORMALIZATION-startup vs PREDICTIVE-safety)
|
||||
- Cross-reference `pearl_welford_trade_count_is_step_not_trade`
|
||||
|
||||
## 6. Done means
|
||||
|
||||
- Warmed-flag ISV slot allocated; bootstrap + emit + read paths wired
|
||||
- `rl_fused_controllers.cu` reward_scale block uses warmed_flag, not trade_count comparison
|
||||
- Local smoke passes V2
|
||||
- Cluster validation V3: eval[0..50] σ within 5× of train-end σ, total eval pnl substantially less negative than alpha-rl-jnct8
|
||||
- Pearl revised
|
||||
- Issue B (train-phase fat-tail spikes) logged as a separate follow-up
|
||||
|
||||
## 7. Risks
|
||||
|
||||
- **Risk α (medium)**: The warmed_flag could fire spuriously in early TRAIN if `min_trades` is set very low. Mitigation: keep min_trades high enough that flag only sets after meaningful warmup (current default ~700 dones is fine).
|
||||
- **Risk β (low)**: Other controllers may also have boot_floor-style protections gated on cumulative_dones. If so, this fix only solves reward_scale; other controllers still snap at boundary. Mitigation: grep for `CUMULATIVE_DONES_INDEX` reads in all `*.cu` and audit each consumer's reset semantics.
|
||||
- **Risk γ (low)**: σ may still explode at eval boundary from OTHER mechanisms not yet diagnosed (e.g. PPO ratio clamp interaction, action mask change). Mitigation: V3 cluster validation will reveal residual gaps.
|
||||
Reference in New Issue
Block a user