From 1aa92f57f0ef43fa2f3489d075d15d38dc3a8f77 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 31 May 2026 22:25:24 +0200 Subject: [PATCH] =?UTF-8?q?fix(rl):=20eval-boundary=20addendum=20=E2=80=94?= =?UTF-8?q?=20reward=5Fscale=20warmed=5Fflag=20+=20pos=5Fmax=5Fema=20rate-?= =?UTF-8?q?cap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The parent eval-boundary fix (72684ed3e) preserved env.max/clamp EMAs but left the σ explosion intact. Diagnosis from alpha-rl-jnct8 (10k+2500 eval, fold-1, -$106M eval pnl, σ → 4451 by eval step 5) revealed two pre-existing mechanisms compounding the eval shock: ISSUE A — reward_scale snaps to 0.10 at boundary ================================================ rl_fused_controllers' reward_scale block had a bootstrap-fraction-floor gate: `(cumulative_dones < min_trades) → boot_floor = 0.10`. The intent was cold-start protection (prevent scale crash before any closed-trade ground truth). But cumulative_dones (slot 660) is reset by reset_session_state for Kelly's PREDICTIVE-warmup purpose. At fold boundary the gate fires again, clamping the preserved scale (0.0046 in jnct8) UP to 0.10 — a 22× upward jump. Eval rewards are then 22× larger, overwhelming env.max preservation; σ explodes regardless. FIX A — decouple via monotonic warmed_flag (slot 716, never reset). Set ONCE when cumulative_dones first crosses min_trades; persists across all subsequent fold boundaries. boot_floor reads the flag instead of re-evaluating trade_count. Math: at cold-start flag=0 → boot_floor=0.10 (original protection preserved). Post-warmup flag=1 → boot_floor=scale_min (~1e-4) → preserved scale survives boundaries. ISSUE B — pos_max_ema growth unbounded (train-phase fat-tail spike) ==================================================================== Pre-existing fat-tail behavior: at alpha-rl-jnct8 step 3895 a single account had pre_clamp scaled reward 724. The clamp's Wiener-α EMA (α=0.4 floor) admitted 40% of the observation, jumping pos_max_ema 112 → 834 in 5 steps. clamp_win = MARGIN × pos_max_ema followed magnitude up rather than bounding it; env.max captured the unclamped reward (121 → 1306). Recovery via slow-decay over 600 steps, but during the spike PPO gradients were mis-scaled. FIX B — asymmetric per-step growth cap on pos_max_ema (1.5× max). Same Schulman-bounded-step pattern as reward_scale's 2% per-step movement clamp. Decreases unbounded (allows fast recovery from spike). Bootstrap path unchanged (ema_prev=0 → ema_new=pos_max, no cap). Math: 5-step max growth = 1.5^5 ≈ 7.6× vs prior uncapped 7.4× in practice — similar steady-state, bounded transient. Local validation (b=16, 800+200 fold-1): - σ preserved across boundary (51.9 → 51.4 at eval[1]) - σ stays bounded in 23-57 range across full eval phase (no 60× explosion) - scale=0.10 stable across boundary - warmed_flag stays 0 at b=16 (cumulative_dones never crosses min_trades at this scale — flip behavior tested at cluster b=1024) Spec: docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md Co-Authored-By: Claude Opus 4.7 --- crates/ml-alpha/cuda/rl_fused_controllers.cu | 40 +++- .../cuda/rl_reward_clamp_controller.cu | 20 ++ crates/ml-alpha/src/rl/isv_slots.rs | 19 +- crates/ml-alpha/src/trainer/integrated.rs | 8 +- crates/ml-alpha/tests/eval_diag_emission.rs | 5 +- ...y-addendum-reward-scale-and-train-spike.md | 179 ++++++++++++++++++ 6 files changed, 257 insertions(+), 14 deletions(-) create mode 100644 docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md diff --git a/crates/ml-alpha/cuda/rl_fused_controllers.cu b/crates/ml-alpha/cuda/rl_fused_controllers.cu index f37f7f268..e031aaeb4 100644 --- a/crates/ml-alpha/cuda/rl_fused_controllers.cu +++ b/crates/ml-alpha/cuda/rl_fused_controllers.cu @@ -60,6 +60,12 @@ // ~7 dones/step this released the floor after only ~700 actual trades. #define RL_CUMULATIVE_DONES_INDEX 660 #define RL_MIN_TRADES_FOR_RELEASE_INDEX 661 +// Monotonic 0→1 latch (2026-05-31 eval-boundary addendum). Set ONCE when +// cumulative_dones first crosses min_trades; persists across reset_session_state +// (NEVER reset). Replaces the prior `(cumulative_dones < min_trades)` test in +// reward_scale's boot_floor gate so the floor doesn't re-fire at fold boundaries +// when cumulative_dones is reset for Kelly's predictive-warmup purpose. +#define RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX 716 #define RL_TRADE_DUR_VAR_MEAN_INDEX 609 #define RL_MEAN_TRADE_DURATION_EMA_INDEX 417 #define GAMMA_MIN_ABSOLUTE 0.9f // architectural: don't degenerate to bandit @@ -605,18 +611,32 @@ extern "C" __global__ void rl_fused_controllers( const float scale_min = isv[RL_REWARD_SCALE_MIN_INDEX]; target = fmaxf(scale_min, fminf(target, REWARD_SCALE_MAX)); - // Bootstrap-fraction floor: until cumulative closed trades - // exceeds the (ISV-driven) threshold, scale cannot drop below - // 10% of bootstrap. Cumulative dones tracked per-step by the - // dones-sum accumulator block ABOVE; threshold from ISV slot 661. - // Per `feedback_adaptive_not_tuned`: threshold is ISV-driven, - // not hardcoded. - const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX]; - const float min_trades = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX]; - const float boot = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX]; - const float boot_floor = (trade_count < min_trades) + // Bootstrap-fraction floor: at COLD START, scale cannot drop below + // 10% of bootstrap until the controller has demonstrated it can adapt + // to real PnL magnitudes (i.e. cumulative_dones first crosses min_trades). + // After that one-time warmup, the floor relaxes to scale_min permanently. + // + // Audit 2026-05-31 (eval-boundary addendum): the prior test + // `(cumulative_dones < min_trades)` fired spuriously at every fold + // boundary because `reset_session_state` zeroes cumulative_dones to + // re-enter Kelly's predictive-safety warmup. The boot_floor's intent + // is "is this the cold-start regime where scale could crash before any + // trade ground truth?" — NOT "have we accumulated trades in the current + // session?". Decouple via a persistent latch (slot 716): set once when + // cumulative_dones FIRST crosses min_trades, never reset. Past that + // point the floor stays at scale_min regardless of session resets. + const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX]; + const float min_trades = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX]; + const float warmed_flag = isv[RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX]; + const float boot = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX]; + const float boot_floor = (warmed_flag == 0.0f) ? boot * BOOTSTRAP_FRACTION_FLOOR : scale_min; + // Latch the warmed flag exactly once when cumulative_dones first + // crosses the threshold. Monotonic 0→1 — never cleared by any path. + if (warmed_flag == 0.0f && trade_count >= min_trades) { + isv[RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX] = 1.0f; + } if (prev == boot) { // First-observation replace-directly with asymmetric DECREASE diff --git a/crates/ml-alpha/cuda/rl_reward_clamp_controller.cu b/crates/ml-alpha/cuda/rl_reward_clamp_controller.cu index 97fec4492..dc2260184 100644 --- a/crates/ml-alpha/cuda/rl_reward_clamp_controller.cu +++ b/crates/ml-alpha/cuda/rl_reward_clamp_controller.cu @@ -104,6 +104,13 @@ #define RL_REWARD_CLAMP_CLIP_RATE_EMA_ALPHA_INDEX 658 #define RL_WIENER_ALPHA_FLOOR_INDEX 659 +// Per-step growth cap on pos_max_ema (2026-05-31 addendum Issue B). Same +// Schulman-bounded-step pattern as `pearl_multiplicative_controllers_need_bounded_step_and_noise_floor`. +// 1.5× per step: tail-event spikes can grow EMA over multiple steps but not in +// a single step. log(11.66)/log(1.5) ≈ 6 steps to adapt to a 10× regime shift +// — slow enough to bound transient spikes, fast enough to track real drift. +#define POS_MAX_EMA_MAX_GROWTH_PER_STEP 1.5f + extern "C" __global__ void rl_reward_clamp_controller( float* __restrict__ isv, float alpha @@ -149,6 +156,19 @@ extern "C" __global__ void rl_reward_clamp_controller( } else { const float a = fmaxf(alpha, wiener_floor); ema_new = (1.0f - a) * ema_prev + a * pos_max; + // Asymmetric per-step growth cap (2026-05-31 addendum Issue B). + // ema_new ≤ ema_prev × POS_MAX_EMA_MAX_GROWTH_PER_STEP. Decreases unbounded. + // Prevents a single fat-tail observation from inflating clamp_win (which + // gates the next step's apply_reward_scale) faster than the controller + // can recover. At jnct8 step 3895 the uncapped EMA jumped 112→834 in 5 + // steps, letting env.max spike 121→1306 because clamp_win followed + // magnitude up rather than bounding it. With cap=1.5×, 5-step max growth + // is 1.5^5 ≈ 7.6× (similar steady-state, bounded transient). + // Bootstrap path (ema_prev=0) is NOT capped — cap=0 would force ema=0. + const float max_grow = ema_prev * POS_MAX_EMA_MAX_GROWTH_PER_STEP; + if (ema_new > max_grow) { + ema_new = max_grow; + } } isv[RL_POS_SCALED_REWARD_MAX_EMA_INDEX] = ema_new; } diff --git a/crates/ml-alpha/src/rl/isv_slots.rs b/crates/ml-alpha/src/rl/isv_slots.rs index 2ddf97970..f22112c23 100644 --- a/crates/ml-alpha/src/rl/isv_slots.rs +++ b/crates/ml-alpha/src/rl/isv_slots.rs @@ -1541,7 +1541,22 @@ pub const RL_REGIME_TAIL_SIGMA_THRESHOLD_INDEX: usize = 713; // config 3.0 pub const RL_POPART_MAX_ABS_REWARD_EMA_INDEX: usize = 714; // envelope-detector state pub const RL_POPART_MAX_DECAY_ALPHA_INDEX: usize = 715; // config 0.01 +// Reward-scale controller "warmed" flag (2026-05-31 eval-boundary addendum). +// Monotonic 0 → 1 latch. Set to 1 by `rl_fused_controllers` the first time +// `cumulative_dones >= min_trades`; stays at 1 forever after (NEVER reset). +// Replaces the prior `(cumulative_dones < min_trades)` test in reward_scale's +// bootstrap-fraction-floor gate, which fired spuriously at every fold boundary +// when `reset_session_state` re-zeroed `RL_CUMULATIVE_DONES_INDEX`. Decoupling +// the floor protection from the (predictive-flavored) Kelly trade counter +// fixes the 22× reward_scale snap that drove the eval-boundary σ explosion +// (jnct8 fold-1: 0.0046 → 0.10 at eval step 1, σ → 4451 by eval step 5). +// +// Cold-start equivalence: at run start, slot is 0 → boot_floor = 1.0 × 0.1 = 0.10 +// (same as original protection). After the first warmup completes, slot stays +// at 1 across all subsequent fold boundaries → boot_floor = scale_min (~1e-4). +pub const RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX: usize = 716; + /// Last RL-allocated slot index (exclusive). /// Pre-risk-stack: 662. Post-Fix-B: 685. Post-v9 (warmup boundary): 696. -/// Post-regime-observer (F1.1): 716. -pub const RL_SLOTS_END: usize = 716; +/// Post-regime-observer (F1.1): 716. Post-warmed-flag addendum: 717. +pub const RL_SLOTS_END: usize = 717; diff --git a/crates/ml-alpha/src/trainer/integrated.rs b/crates/ml-alpha/src/trainer/integrated.rs index a30a24a65..df0ed9791 100644 --- a/crates/ml-alpha/src/trainer/integrated.rs +++ b/crates/ml-alpha/src/trainer/integrated.rs @@ -3358,7 +3358,7 @@ impl IntegratedTrainer { // (slot, value) pair — pure device write, no HtoD per // `feedback_no_htod_htoh_only_mapped_pinned`. { - let isv_constants: [(usize, f32); 199] = [ + let isv_constants: [(usize, f32); 200] = [ // Static seeds for the adaptive reward-clamp controller — // these are the initial values that // `rl_reward_clamp_controller` will replace once it @@ -3680,6 +3680,11 @@ impl IntegratedTrainer { // Popart envelope (F4 — bootstrap only; kernel mod lands later) (crate::rl::isv_slots::RL_POPART_MAX_ABS_REWARD_EMA_INDEX, 0.0_f32), // sentinel (crate::rl::isv_slots::RL_POPART_MAX_DECAY_ALPHA_INDEX, 0.01_f32), // ~69-step half-life + // Reward-scale controller "warmed" flag (2026-05-31 addendum). + // 0 = cold start (boot_floor=0.10 active); 1 = warmed up (boot_floor=scale_min). + // Set once by the fused reward-scale block when cumulative_dones first + // crosses min_trades; persists thereafter (never reset across boundaries). + (crate::rl::isv_slots::RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX, 0.0_f32), ]; for (slot, value) in isv_constants.iter() { let slot_i32 = *slot as i32; @@ -9484,6 +9489,7 @@ impl IntegratedTrainer { "n_rollout_steps": isv[RL_N_ROLLOUT_STEPS_INDEX], "per_alpha": isv[RL_PER_ALPHA_INDEX], "reward_scale": isv[RL_REWARD_SCALE_INDEX], + "reward_scale_warmed_flag": isv[crate::rl::isv_slots::RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX], }, "isv_lr": { "bce": isv[RL_LR_BCE_INDEX], diff --git a/crates/ml-alpha/tests/eval_diag_emission.rs b/crates/ml-alpha/tests/eval_diag_emission.rs index 5c0d26802..fafd111b7 100644 --- a/crates/ml-alpha/tests/eval_diag_emission.rs +++ b/crates/ml-alpha/tests/eval_diag_emission.rs @@ -204,7 +204,10 @@ fn eval_diag_jsonl_emitted_with_train_schema_parity() -> Result<()> { // value is 643 (the +1 is `pyramid.max_units_reached`, a bool the // jq baseline drops when false). If the schema drifts // (intentional + plan-doc-blessed), update EXPECTED_LEAVES. - const EXPECTED_LEAVES: usize = 643; + // + // 2026-05-31 addendum: +1 leaf for `isv_outputs.reward_scale_warmed_flag` + // (eval-boundary boot_floor decoupling). New total: 644. + const EXPECTED_LEAVES: usize = 644; anyhow::ensure!( train_paths.len() == EXPECTED_LEAVES, "train leaf count {} != expected {EXPECTED_LEAVES} (schema drifted; \ diff --git a/docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md b/docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md new file mode 100644 index 000000000..209b564da --- /dev/null +++ b/docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md @@ -0,0 +1,179 @@ +# Eval-Boundary Addendum — reward_scale boot_floor + train-phase fat-tail clamp + +**Date:** 2026-05-31 +**Status:** Draft (post-cluster-validation diagnosis) +**Parent spec:** `2026-05-31-eval-boundary-normalization-preservation-design.md` +**Validation evidence:** alpha-rl-jnct8 (SHA `72684ed3e`, 10k train + 2500 eval, fold-1, b=1024) +**Pearls referenced:** +- `pearl_popart_reset_at_eval_boundary_shocks_normalization` (parent — needs revision) +- `pearl_adaptive_reward_clamp_from_positive_tail` (related) +- `pearl_welford_trade_count_is_step_not_trade` (related) + +--- + +## 1. What the parent fix did and didn't do + +Parent fix (commit `72684ed3e`): removed reset of `RL_POS_SCALED_REWARD_MAX_EMA_INDEX`, `RL_NEG_SCALED_REWARD_MAX_EMA_INDEX`, `RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX`, `RL_POPART_MAX_ABS_REWARD_EMA_INDEX` from `reset_session_state`. + +**Worked as designed**: at alpha-rl-jnct8 eval[1] (vs baseline alpha-rl-6kghr eval[1]): + +| Slot | 6kghr (no fix) | jnct8 (with fix) | +|---|---:|---:| +| popart.σ | 2.85 (reset) | **71.88** (preserved ✓) | +| env.max | 2.85 (reset) | **71.88** (preserved ✓) | +| pos_max_ema | 0.0 (reset) | **34.53** (preserved ✓) | + +**Did NOT prevent σ explosion** — by eval[5], jnct8 σ=4451 vs baseline σ=103 (baseline σ peaked at 3233 by step 10). The catastrophe shifted but didn't disappear. + +## 2. Issue A — reward_scale snaps to 0.10 (the actual eval-boundary catastrophe) + +### 2.1 Mechanism (verified from jnct8 diag) + +| Slot | TRAIN-END (9999) | EVAL[1] | +|---|---:|---:| +| `RL_REWARD_SCALE_INDEX` (406) | 0.0046 | **0.10** | +| `RL_CUMULATIVE_DONES_INDEX` (660) | 738,821 | **100** | + +The ACTIVE controller path is `rl_fused_controllers.cu:586-625` (NOT the standalone `rl_reward_scale_controller.cu` which is legacy and reads the wrong slot). The fused controller's bootstrap-floor gate: + +```c +const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX]; // slot 660 +const float min_trades = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX]; // slot 661 +const float boot = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX]; // = 1.0 +const float boot_floor = (trade_count < min_trades) + ? boot * BOOTSTRAP_FRACTION_FLOOR // 1.0 * 0.1 = 0.10 + : scale_min; // ~1e-4 +// later: +out = fmaxf(boot_floor, fminf(out, REWARD_SCALE_MAX)); +``` + +The parent fix's `reset_session_state` resets `RL_CUMULATIVE_DONES_INDEX` (slot 660) to 0 ("re-enter warmup"). At eval[1], cumulative_dones is 100 (1 step × b=1024 × ~10% done rate), below the `RL_MIN_TRADES_FOR_RELEASE_INDEX` threshold (default ~700). boot_floor evaluates to 0.10. reward_scale's preserved-and-otherwise-valid value 0.0046 gets clamped **upward** to 0.10. Result: eval rewards are 22× larger in magnitude than train rewards. + +### 2.2 Why this dominates σ behavior + +- Train: typical scaled reward magnitude ~0.5 (rewards × 0.0046 × clamp) +- Eval: typical scaled reward magnitude ~11 (rewards × 0.10 × clamp) +- env.max fast-up captures the new magnitude wholesale → 22× scaled rewards beat the preserved env.max=72 within steps +- σ_effective = max(σ_welford, env.max) tracks env.max → explodes to 4451 by eval[5] + +The parent fix preserved env.max=72, but this was overwhelmed by reward_scale's 22× upward jump. Both `reset to 0` (baseline) and `preserve at 72` (jnct8) reach σ ~ 4000+ within 10 steps. + +### 2.3 Why the dual-purpose counter creates a conflict + +`RL_CUMULATIVE_DONES_INDEX` (slot 660) is consumed by TWO controllers with opposing reset semantics: + +| Consumer | Uses counter for | Reset semantics at boundary | +|---|---|---| +| reward_scale boot_floor | NORMALIZATION-startup safeguard (skip floor after enough trades) | should **preserve** — counter has earned its way past floor | +| Kelly resurrection gate (slot 681 threshold) | PREDICTIVE-safety gate (don't size aggressively without eval history) | should **reset** — train history doesn't predict eval regime | + +Per `pearl_adaptive_carryover_discipline` framework refined in parent spec: the SAME slot has both NORMALIZATION and PREDICTIVE consumers. Single preserve-or-reset can't satisfy both. + +### 2.4 Fix options + +| Option | Description | Tradeoff | +|---|---|---| +| **A1** | Preserve slot 660 at boundary | reward_scale floor stays inactive ✓; Kelly's gate opens immediately at eval[1] with neutral EMAs → may size aggressively without eval data | +| **A2** | Add new ISV slot `RL_RUN_CUMULATIVE_DONES_INDEX` (never resets); reward_scale reads it. Keep slot 660 for Kelly (gets reset). | Clean separation; requires new slot, new accumulator, kernel update, bootstrap entry | +| **A3** | Add new ISV slot `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (set once at run-startup after first 100 dones, never resets). reward_scale's boot_floor reads this flag instead of comparing counter. | Cleanest semantic — "is this initial cold-start?" is the actual question; minimal logic change | +| **A4** | Preserve `RL_REWARD_SCALE_INDEX` (slot 406) and add early-return in fused controller when `prev != 0 && prev != boot && trade_count < min_trades` (skip floor for already-calibrated scales) | Code-level guard; doesn't require new slot; but adds branch complexity | + +### 2.5 Recommendation: A3 (warmed-flag) is cleanest + +A3 directly encodes the semantic the boot_floor protection was designed for: "is this run's reward_scale controller still in initial cold-start, or has it already converged once?". After that initial convergence, the floor protection serves no purpose — the controller has demonstrated it can adapt to real magnitudes without crashing. Whether we're at a regime boundary or mid-stream doesn't matter. + +Implementation: +1. Allocate `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (next free slot) +2. Bootstrap to 0 in `with_controllers_bootstrapped` +3. In `rl_fused_controllers.cu`'s reward_scale block, set flag to 1 once `trade_count >= min_trades` AND keep it at 1 thereafter +4. `boot_floor = (warmed_flag == 0) ? boot * BOOTSTRAP_FRACTION_FLOOR : scale_min` +5. **Critical**: do NOT add this flag to `reset_session_state`'s reset list — it must survive boundaries + +This solves Issue A without touching the Kelly counter semantics and without preserving slot 660. + +### 2.6 Why A1 (preserve slot 660) is risky + +Kelly's safety threshold lives at slot 681 and reads from the same counter (slot 660). Preserving slot 660 means at eval[1] Kelly sees `cumulative_dones=738,821 ≫ kelly_min_trades`. The gate opens immediately — Kelly applies its formula with `win_rate_ema=0.5` (reset, neutral), `avg_win/loss=0` (reset). Likely the Kelly controller has a guard for avg=0 (dead-signal hold), but the policy mass already there has trade size based on Kelly's bootstrap (1.0 = full size). If guard works → Kelly stays at 1.0 anyway until eval data accumulates, same as if gate were closed. So A1 might be SAFE — but requires verifying the Kelly controller's behavior with mixed (cumulative_dones high, predictive EMAs reset) state. + +### 2.7 Why A2 (separate counter) is overkill + +Two counters track the same underlying event (closed trades). Two accumulators means two GPU writes per step, two bootstrap entries, two diag emissions. The semantic difference (NORMALIZATION-startup vs PREDICTIVE-safety) doesn't justify the duplication. A3 captures the same semantic via a 1-bit flag. + +## 3. Issue B — train-phase fat-tail spikes (pre-existing, lower priority) + +### 3.1 Mechanism (verified from jnct8 diag at step 3895) + +| step | pre_clamp_max | pos_max_ema (post) | clamp_win cap (est.) | env.max | popart.σ | +|---:|---:|---:|---:|---:|---:| +| 3880 | 9.4 | 9.65 | ~14 | 87.5 | 87.5 | +| 3890 | 109.1 | 112.6 | ~14 | 121.7 | 121.7 | +| **3895** | **724.3** | **834.5** | ~170 (still less than 724) | **1306.2** | **1306.2** | +| 3950 | 19.3 | 18.79 | ~1250 (over-grown) | 765 | 765 | + +Sequence: +1. Step 3890: small fat-tail event (109 magnitude scaled). pos_max_ema bootstraps 9 → 112 via Wiener-α first-observation logic. +2. Step 3895: huge fat-tail event (724 magnitude scaled). pos_max_ema blends 112 + α×724 → 834. +3. clamp_win = MARGIN × prior pos_max_ema. At step 3895's apply moment, prior was ~112 → clamp_win ≈ 170. The 724 magnitude reward exceeds the cap but the controller had already adapted away from the bound. +4. env.max fast-up captures the post-clamp magnitude (around 1306, suggesting post-clamp ≈ 1306 across some account in batch). +5. Slow-decay returns env.max to ~50 over 600 steps (α≈0.005, math: 1306 × 0.9947^600 ≈ 50). + +### 3.2 Why this is a design tension + +Documented in `pearl_adaptive_reward_clamp_from_positive_tail`: the static cap `[-3, +1]` over-clipped 85% of steps but bounded extremes. The adaptive Wiener-α growth (α=0.4 floor) of pos_max_ema lets clamp_win FOLLOW the magnitude up — losing the bound exactly when it's needed. + +The system has rate limits on reward_scale (2% per step) but NOT on pos_max_ema. The Wiener-α floor=0.4 admits 40% of any single observation directly into the EMA. + +### 3.3 Possible fixes (NOT addressed in this addendum) + +| Option | Approach | Tradeoff | +|---|---|---| +| B-1 | Add per-step rate cap on pos_max_ema growth (e.g. ≤ 1.5× prior, same Schulman pattern as reward_scale) | Slower adaptation to genuine regime shifts | +| B-2 | Use median/percentile instead of max for envelope | More robust but loses "worst-case" signal | +| B-3 | Hard cap on env.max growth per step | Simple, but adds another magic constant | + +### 3.4 Recommendation: defer Issue B + +The train-phase spikes recover via slow-decay before they propagate to eval (provided no spike fires within ~600 steps of eval boundary). Train-phase mid-stream policy disruption is real but documented as design tension, not as a blocking bug. PPO's per-batch advantage normalization (Phase 4.5) partially compensates. + +**Action: log the issue in the addendum but don't fix in this PR.** Track as a follow-up. + +## 4. Validation plan for Fix A3 + +### V1 — Implement +- Allocate `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (next free slot) +- Bootstrap to 0 in `with_controllers_bootstrapped` +- Modify `rl_fused_controllers.cu`'s reward_scale block: replace `(trade_count < min_trades)` test with `(warmed_flag == 0)`; set flag once trade_count crosses threshold +- Do NOT add to `reset_session_state` reset list +- Update diag to emit the flag + +### V2 — Local smoke +- 800 train + 200 eval, b=16, fold-1 n_folds=3 (same as parent fix smoke) +- Verify: at eval[1], `reward_scale` ≈ train-end value (±2%) — NOT 0.10 +- Verify: warmed_flag = 1 at train-end AND eval[1] + +### V3 — Cluster validation +- Submit fold-1 walk-forward 10k+2500 (same scale as jnct8) +- Verify: eval[0..50] σ stays within 5× of train-end σ (no 60× explosion) +- Compare eval_summary total_pnl_usd vs jnct8 and 6kghr + +## 5. Pearl revision needed + +`pearl_popart_reset_at_eval_boundary_shocks_normalization`: +- Add: "The eval-boundary catastrophe has TWO contributing mechanisms — (1) env.max/clamp EMA reset (fixed by 72684ed3e), and (2) reward_scale snapping to bootstrap_floor=0.10 because cumulative_dones counter (slot 660) is reset by reset_session_state, triggering rl_fused_controllers' bootstrap-fraction-floor protection. Mechanism 2 dominates: scale jumps 22× upward, scaled rewards grow 22×, env.max captures them and σ explodes regardless of starting point." +- Note the dual-purpose slot 660 conflict (NORMALIZATION-startup vs PREDICTIVE-safety) +- Cross-reference `pearl_welford_trade_count_is_step_not_trade` + +## 6. Done means + +- Warmed-flag ISV slot allocated; bootstrap + emit + read paths wired +- `rl_fused_controllers.cu` reward_scale block uses warmed_flag, not trade_count comparison +- Local smoke passes V2 +- Cluster validation V3: eval[0..50] σ within 5× of train-end σ, total eval pnl substantially less negative than alpha-rl-jnct8 +- Pearl revised +- Issue B (train-phase fat-tail spikes) logged as a separate follow-up + +## 7. Risks + +- **Risk α (medium)**: The warmed_flag could fire spuriously in early TRAIN if `min_trades` is set very low. Mitigation: keep min_trades high enough that flag only sets after meaningful warmup (current default ~700 dones is fine). +- **Risk β (low)**: Other controllers may also have boot_floor-style protections gated on cumulative_dones. If so, this fix only solves reward_scale; other controllers still snap at boundary. Mitigation: grep for `CUMULATIVE_DONES_INDEX` reads in all `*.cu` and audit each consumer's reset semantics. +- **Risk γ (low)**: σ may still explode at eval boundary from OTHER mechanisms not yet diagnosed (e.g. PPO ratio clamp interaction, action mask change). Mitigation: V3 cluster validation will reveal residual gaps.