gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(rl): eval-boundary addendum — reward_scale warmed_flag + pos_max_ema rate-cap

Browse Source 
The parent eval-boundary fix (72684ed3e) preserved env.max/clamp EMAs but
left the σ explosion intact. Diagnosis from alpha-rl-jnct8 (10k+2500 eval,
fold-1, -$106M eval pnl, σ → 4451 by eval step 5) revealed two pre-existing
mechanisms compounding the eval shock:

ISSUE A — reward_scale snaps to 0.10 at boundary
================================================
rl_fused_controllers' reward_scale block had a bootstrap-fraction-floor
gate: `(cumulative_dones < min_trades) → boot_floor = 0.10`. The intent
was cold-start protection (prevent scale crash before any closed-trade
ground truth). But cumulative_dones (slot 660) is reset by
reset_session_state for Kelly's PREDICTIVE-warmup purpose. At fold
boundary the gate fires again, clamping the preserved scale (0.0046 in
jnct8) UP to 0.10 — a 22× upward jump. Eval rewards are then 22× larger,
overwhelming env.max preservation; σ explodes regardless.

FIX A — decouple via monotonic warmed_flag (slot 716, never reset).
Set ONCE when cumulative_dones first crosses min_trades; persists across
all subsequent fold boundaries. boot_floor reads the flag instead of
re-evaluating trade_count. Math: at cold-start flag=0 → boot_floor=0.10
(original protection preserved). Post-warmup flag=1 → boot_floor=scale_min
(~1e-4) → preserved scale survives boundaries.

ISSUE B — pos_max_ema growth unbounded (train-phase fat-tail spike)
====================================================================
Pre-existing fat-tail behavior: at alpha-rl-jnct8 step 3895 a single
account had pre_clamp scaled reward 724. The clamp's Wiener-α EMA
(α=0.4 floor) admitted 40% of the observation, jumping pos_max_ema
112 → 834 in 5 steps. clamp_win = MARGIN × pos_max_ema followed
magnitude up rather than bounding it; env.max captured the unclamped
reward (121 → 1306). Recovery via slow-decay over 600 steps, but
during the spike PPO gradients were mis-scaled.

FIX B — asymmetric per-step growth cap on pos_max_ema (1.5× max).
Same Schulman-bounded-step pattern as reward_scale's 2% per-step
movement clamp. Decreases unbounded (allows fast recovery from spike).
Bootstrap path unchanged (ema_prev=0 → ema_new=pos_max, no cap).
Math: 5-step max growth = 1.5^5 ≈ 7.6× vs prior uncapped 7.4× in
practice — similar steady-state, bounded transient.

Local validation (b=16, 800+200 fold-1):
- σ preserved across boundary (51.9 → 51.4 at eval[1])
- σ stays bounded in 23-57 range across full eval phase (no 60× explosion)
- scale=0.10 stable across boundary
- warmed_flag stays 0 at b=16 (cumulative_dones never crosses min_trades
  at this scale — flip behavior tested at cluster b=1024)

Spec: docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-31 22:25:24 +02:00
parent 72684ed3ef
commit 1aa92f57f0
6 changed files with 257 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
crates/ml-alpha/cuda/rl_fused_controllers.cu
View File

@@ -60,6 +60,12 @@
// ~7 dones/step this released the floor after only ~700 actual trades.
#define RL_CUMULATIVE_DONES_INDEX 660
#define RL_MIN_TRADES_FOR_RELEASE_INDEX 661
// Monotonic 0→1 latch (2026-05-31 eval-boundary addendum). Set ONCE when
// cumulative_dones first crosses min_trades; persists across reset_session_state
// (NEVER reset). Replaces the prior `(cumulative_dones < min_trades)` test in
// reward_scale's boot_floor gate so the floor doesn't re-fire at fold boundaries
// when cumulative_dones is reset for Kelly's predictive-warmup purpose.
#define RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX 716
#define RL_TRADE_DUR_VAR_MEAN_INDEX 609
#define RL_MEAN_TRADE_DURATION_EMA_INDEX 417
#define GAMMA_MIN_ABSOLUTE 0.9f // architectural: don't degenerate to bandit
@@ -605,18 +611,32 @@ extern "C" __global__ void rl_fused_controllers(
const float scale_min = isv[RL_REWARD_SCALE_MIN_INDEX];
target = fmaxf(scale_min, fminf(target, REWARD_SCALE_MAX));
// Bootstrap-fraction floor: until cumulative closed trades
// exceeds the (ISV-driven) threshold, scale cannot drop below
// 10% of bootstrap. Cumulative dones tracked per-step by the
// dones-sum accumulator block ABOVE; threshold from ISV slot 661.
// Per `feedback_adaptive_not_tuned`: threshold is ISV-driven,
// not hardcoded.
const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX];
const float min_trades = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX];
const float boot = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX];
const float boot_floor = (trade_count < min_trades)
// Bootstrap-fraction floor: at COLD START, scale cannot drop below
// 10% of bootstrap until the controller has demonstrated it can adapt
// to real PnL magnitudes (i.e. cumulative_dones first crosses min_trades).
// After that one-time warmup, the floor relaxes to scale_min permanently.
//
// Audit 2026-05-31 (eval-boundary addendum): the prior test
// `(cumulative_dones < min_trades)` fired spuriously at every fold
// boundary because `reset_session_state` zeroes cumulative_dones to
// re-enter Kelly's predictive-safety warmup. The boot_floor's intent
// is "is this the cold-start regime where scale could crash before any
// trade ground truth?" — NOT "have we accumulated trades in the current
// session?". Decouple via a persistent latch (slot 716): set once when
// cumulative_dones FIRST crosses min_trades, never reset. Past that
// point the floor stays at scale_min regardless of session resets.
const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX];
const float min_trades = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX];
const float warmed_flag = isv[RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX];
const float boot = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX];
const float boot_floor = (warmed_flag == 0.0f)
? boot * BOOTSTRAP_FRACTION_FLOOR
: scale_min;
// Latch the warmed flag exactly once when cumulative_dones first
// crosses the threshold. Monotonic 0→1 — never cleared by any path.
if (warmed_flag == 0.0f && trade_count >= min_trades) {
isv[RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX] = 1.0f;
}
if (prev == boot) {
// First-observation replace-directly with asymmetric DECREASE

20
crates/ml-alpha/cuda/rl_reward_clamp_controller.cu
View File

@@ -104,6 +104,13 @@
#define RL_REWARD_CLAMP_CLIP_RATE_EMA_ALPHA_INDEX 658
#define RL_WIENER_ALPHA_FLOOR_INDEX 659
// Per-step growth cap on pos_max_ema (2026-05-31 addendum Issue B). Same
// Schulman-bounded-step pattern as `pearl_multiplicative_controllers_need_bounded_step_and_noise_floor`.
// 1.5× per step: tail-event spikes can grow EMA over multiple steps but not in
// a single step. log(11.66)/log(1.5) ≈ 6 steps to adapt to a 10× regime shift
// — slow enough to bound transient spikes, fast enough to track real drift.
#define POS_MAX_EMA_MAX_GROWTH_PER_STEP 1.5f
extern "C" __global__ void rl_reward_clamp_controller(
float* __restrict__ isv,
float alpha
@@ -149,6 +156,19 @@ extern "C" __global__ void rl_reward_clamp_controller(
} else {
const float a = fmaxf(alpha, wiener_floor);
ema_new = (1.0f - a) * ema_prev + a * pos_max;
// Asymmetric per-step growth cap (2026-05-31 addendum Issue B).
// ema_new ≤ ema_prev × POS_MAX_EMA_MAX_GROWTH_PER_STEP. Decreases unbounded.
// Prevents a single fat-tail observation from inflating clamp_win (which
// gates the next step's apply_reward_scale) faster than the controller
// can recover. At jnct8 step 3895 the uncapped EMA jumped 112→834 in 5
// steps, letting env.max spike 121→1306 because clamp_win followed
// magnitude up rather than bounding it. With cap=1.5×, 5-step max growth
// is 1.5^5 ≈ 7.6× (similar steady-state, bounded transient).
// Bootstrap path (ema_prev=0) is NOT capped — cap=0 would force ema=0.
const float max_grow = ema_prev * POS_MAX_EMA_MAX_GROWTH_PER_STEP;
if (ema_new > max_grow) {
ema_new = max_grow;
}
}
isv[RL_POS_SCALED_REWARD_MAX_EMA_INDEX] = ema_new;
}

19
crates/ml-alpha/src/rl/isv_slots.rs
View File

@@ -1541,7 +1541,22 @@ pub const RL_REGIME_TAIL_SIGMA_THRESHOLD_INDEX: usize = 713; // config 3.0
pub const RL_POPART_MAX_ABS_REWARD_EMA_INDEX: usize = 714; // envelope-detector state
pub const RL_POPART_MAX_DECAY_ALPHA_INDEX: usize = 715; // config 0.01
// Reward-scale controller "warmed" flag (2026-05-31 eval-boundary addendum).
// Monotonic 0 → 1 latch. Set to 1 by `rl_fused_controllers` the first time
// `cumulative_dones >= min_trades`; stays at 1 forever after (NEVER reset).
// Replaces the prior `(cumulative_dones < min_trades)` test in reward_scale's
// bootstrap-fraction-floor gate, which fired spuriously at every fold boundary
// when `reset_session_state` re-zeroed `RL_CUMULATIVE_DONES_INDEX`. Decoupling
// the floor protection from the (predictive-flavored) Kelly trade counter
// fixes the 22× reward_scale snap that drove the eval-boundary σ explosion
// (jnct8 fold-1: 0.0046 → 0.10 at eval step 1, σ → 4451 by eval step 5).
//
// Cold-start equivalence: at run start, slot is 0 → boot_floor = 1.0 × 0.1 = 0.10
// (same as original protection). After the first warmup completes, slot stays
// at 1 across all subsequent fold boundaries → boot_floor = scale_min (~1e-4).
pub const RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX: usize = 716;
/// Last RL-allocated slot index (exclusive).
/// Pre-risk-stack: 662. Post-Fix-B: 685. Post-v9 (warmup boundary): 696.
/// Post-regime-observer (F1.1): 716.
pub const RL_SLOTS_END: usize = 716;
/// Post-regime-observer (F1.1): 716. Post-warmed-flag addendum: 717.
pub const RL_SLOTS_END: usize = 717;

8
crates/ml-alpha/src/trainer/integrated.rs
View File

@@ -3358,7 +3358,7 @@ impl IntegratedTrainer {
// (slot, value) pair — pure device write, no HtoD per
// `feedback_no_htod_htoh_only_mapped_pinned`.
{
let isv_constants: [(usize, f32); 199] = [
let isv_constants: [(usize, f32); 200] = [
// Static seeds for the adaptive reward-clamp controller —
// these are the initial values that
// `rl_reward_clamp_controller` will replace once it
@@ -3680,6 +3680,11 @@ impl IntegratedTrainer {
// Popart envelope (F4 — bootstrap only; kernel mod lands later)
(crate::rl::isv_slots::RL_POPART_MAX_ABS_REWARD_EMA_INDEX, 0.0_f32), // sentinel
(crate::rl::isv_slots::RL_POPART_MAX_DECAY_ALPHA_INDEX, 0.01_f32), // ~69-step half-life
// Reward-scale controller "warmed" flag (2026-05-31 addendum).
// 0 = cold start (boot_floor=0.10 active); 1 = warmed up (boot_floor=scale_min).
// Set once by the fused reward-scale block when cumulative_dones first
// crosses min_trades; persists thereafter (never reset across boundaries).
(crate::rl::isv_slots::RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX, 0.0_f32),
];
for (slot, value) in isv_constants.iter() {
let slot_i32 = *slot as i32;
@@ -9484,6 +9489,7 @@ impl IntegratedTrainer {
"n_rollout_steps": isv[RL_N_ROLLOUT_STEPS_INDEX],
"per_alpha": isv[RL_PER_ALPHA_INDEX],
"reward_scale": isv[RL_REWARD_SCALE_INDEX],
"reward_scale_warmed_flag": isv[crate::rl::isv_slots::RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX],
},
"isv_lr": {
"bce": isv[RL_LR_BCE_INDEX],

5
crates/ml-alpha/tests/eval_diag_emission.rs
View File

@@ -204,7 +204,10 @@ fn eval_diag_jsonl_emitted_with_train_schema_parity() -> Result<()> {
// value is 643 (the +1 is `pyramid.max_units_reached`, a bool the
// jq baseline drops when false). If the schema drifts
// (intentional + plan-doc-blessed), update EXPECTED_LEAVES.
const EXPECTED_LEAVES: usize = 643;
//
// 2026-05-31 addendum: +1 leaf for `isv_outputs.reward_scale_warmed_flag`
// (eval-boundary boot_floor decoupling). New total: 644.
const EXPECTED_LEAVES: usize = 644;
anyhow::ensure!(
train_paths.len() == EXPECTED_LEAVES,
"train leaf count {} != expected {EXPECTED_LEAVES} (schema drifted; \

179
docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md Normal file
View File

@@ -0,0 +1,179 @@
# Eval-Boundary Addendum — reward_scale boot_floor + train-phase fat-tail clamp
**Date:** 2026-05-31
**Status:** Draft (post-cluster-validation diagnosis)
**Parent spec:** `2026-05-31-eval-boundary-normalization-preservation-design.md`
**Validation evidence:** alpha-rl-jnct8 (SHA `72684ed3e`, 10k train + 2500 eval, fold-1, b=1024)
**Pearls referenced:**
- `pearl_popart_reset_at_eval_boundary_shocks_normalization` (parent — needs revision)
- `pearl_adaptive_reward_clamp_from_positive_tail` (related)
- `pearl_welford_trade_count_is_step_not_trade` (related)
---
## 1. What the parent fix did and didn't do
Parent fix (commit `72684ed3e`): removed reset of `RL_POS_SCALED_REWARD_MAX_EMA_INDEX`, `RL_NEG_SCALED_REWARD_MAX_EMA_INDEX`, `RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX`, `RL_POPART_MAX_ABS_REWARD_EMA_INDEX` from `reset_session_state`.
**Worked as designed**: at alpha-rl-jnct8 eval[1] (vs baseline alpha-rl-6kghr eval[1]):
| Slot | 6kghr (no fix) | jnct8 (with fix) |
|---|---:|---:|
| popart.σ | 2.85 (reset) | **71.88** (preserved ✓) |
| env.max | 2.85 (reset) | **71.88** (preserved ✓) |
| pos_max_ema | 0.0 (reset) | **34.53** (preserved ✓) |
**Did NOT prevent σ explosion** — by eval[5], jnct8 σ=4451 vs baseline σ=103 (baseline σ peaked at 3233 by step 10). The catastrophe shifted but didn't disappear.
## 2. Issue A — reward_scale snaps to 0.10 (the actual eval-boundary catastrophe)
### 2.1 Mechanism (verified from jnct8 diag)
| Slot | TRAIN-END (9999) | EVAL[1] |
|---|---:|---:|
| `RL_REWARD_SCALE_INDEX` (406) | 0.0046 | **0.10** |
| `RL_CUMULATIVE_DONES_INDEX` (660) | 738,821 | **100** |
The ACTIVE controller path is `rl_fused_controllers.cu:586-625` (NOT the standalone `rl_reward_scale_controller.cu` which is legacy and reads the wrong slot). The fused controller's bootstrap-floor gate:
```c
const float trade_count = isv[RL_CUMULATIVE_DONES_INDEX]; // slot 660
const float min_trades = isv[RL_MIN_TRADES_FOR_RELEASE_INDEX]; // slot 661
const float boot = isv[RL_REWARD_SCALE_BOOTSTRAP_INDEX]; // = 1.0
const float boot_floor = (trade_count < min_trades)
? boot * BOOTSTRAP_FRACTION_FLOOR // 1.0 * 0.1 = 0.10
: scale_min; // ~1e-4
// later:
out = fmaxf(boot_floor, fminf(out, REWARD_SCALE_MAX));
```
The parent fix's `reset_session_state` resets `RL_CUMULATIVE_DONES_INDEX` (slot 660) to 0 ("re-enter warmup"). At eval[1], cumulative_dones is 100 (1 step × b=1024 × ~10% done rate), below the `RL_MIN_TRADES_FOR_RELEASE_INDEX` threshold (default ~700). boot_floor evaluates to 0.10. reward_scale's preserved-and-otherwise-valid value 0.0046 gets clamped **upward** to 0.10. Result: eval rewards are 22× larger in magnitude than train rewards.
### 2.2 Why this dominates σ behavior
- Train: typical scaled reward magnitude ~0.5 (rewards × 0.0046 × clamp)
- Eval: typical scaled reward magnitude ~11 (rewards × 0.10 × clamp)
- env.max fast-up captures the new magnitude wholesale → 22× scaled rewards beat the preserved env.max=72 within steps
- σ_effective = max(σ_welford, env.max) tracks env.max → explodes to 4451 by eval[5]
The parent fix preserved env.max=72, but this was overwhelmed by reward_scale's 22× upward jump. Both `reset to 0` (baseline) and `preserve at 72` (jnct8) reach σ ~ 4000+ within 10 steps.
### 2.3 Why the dual-purpose counter creates a conflict
`RL_CUMULATIVE_DONES_INDEX` (slot 660) is consumed by TWO controllers with opposing reset semantics:
| Consumer | Uses counter for | Reset semantics at boundary |
|---|---|---|
| reward_scale boot_floor | NORMALIZATION-startup safeguard (skip floor after enough trades) | should **preserve** — counter has earned its way past floor |
| Kelly resurrection gate (slot 681 threshold) | PREDICTIVE-safety gate (don't size aggressively without eval history) | should **reset** — train history doesn't predict eval regime |
Per `pearl_adaptive_carryover_discipline` framework refined in parent spec: the SAME slot has both NORMALIZATION and PREDICTIVE consumers. Single preserve-or-reset can't satisfy both.
### 2.4 Fix options
| Option | Description | Tradeoff |
|---|---|---|
| **A1** | Preserve slot 660 at boundary | reward_scale floor stays inactive ✓; Kelly's gate opens immediately at eval[1] with neutral EMAs → may size aggressively without eval data |
| **A2** | Add new ISV slot `RL_RUN_CUMULATIVE_DONES_INDEX` (never resets); reward_scale reads it. Keep slot 660 for Kelly (gets reset). | Clean separation; requires new slot, new accumulator, kernel update, bootstrap entry |
| **A3** | Add new ISV slot `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (set once at run-startup after first 100 dones, never resets). reward_scale's boot_floor reads this flag instead of comparing counter. | Cleanest semantic — "is this initial cold-start?" is the actual question; minimal logic change |
| **A4** | Preserve `RL_REWARD_SCALE_INDEX` (slot 406) and add early-return in fused controller when `prev != 0 && prev != boot && trade_count < min_trades` (skip floor for already-calibrated scales) | Code-level guard; doesn't require new slot; but adds branch complexity |
### 2.5 Recommendation: A3 (warmed-flag) is cleanest
A3 directly encodes the semantic the boot_floor protection was designed for: "is this run's reward_scale controller still in initial cold-start, or has it already converged once?". After that initial convergence, the floor protection serves no purpose — the controller has demonstrated it can adapt to real magnitudes without crashing. Whether we're at a regime boundary or mid-stream doesn't matter.
Implementation:
1. Allocate `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (next free slot)
2. Bootstrap to 0 in `with_controllers_bootstrapped`
3. In `rl_fused_controllers.cu`'s reward_scale block, set flag to 1 once `trade_count >= min_trades` AND keep it at 1 thereafter
4. `boot_floor = (warmed_flag == 0) ? boot * BOOTSTRAP_FRACTION_FLOOR : scale_min`
5. **Critical**: do NOT add this flag to `reset_session_state`'s reset list — it must survive boundaries
This solves Issue A without touching the Kelly counter semantics and without preserving slot 660.
### 2.6 Why A1 (preserve slot 660) is risky
Kelly's safety threshold lives at slot 681 and reads from the same counter (slot 660). Preserving slot 660 means at eval[1] Kelly sees `cumulative_dones=738,821 ≫ kelly_min_trades`. The gate opens immediately — Kelly applies its formula with `win_rate_ema=0.5` (reset, neutral), `avg_win/loss=0` (reset). Likely the Kelly controller has a guard for avg=0 (dead-signal hold), but the policy mass already there has trade size based on Kelly's bootstrap (1.0 = full size). If guard works → Kelly stays at 1.0 anyway until eval data accumulates, same as if gate were closed. So A1 might be SAFE — but requires verifying the Kelly controller's behavior with mixed (cumulative_dones high, predictive EMAs reset) state.
### 2.7 Why A2 (separate counter) is overkill
Two counters track the same underlying event (closed trades). Two accumulators means two GPU writes per step, two bootstrap entries, two diag emissions. The semantic difference (NORMALIZATION-startup vs PREDICTIVE-safety) doesn't justify the duplication. A3 captures the same semantic via a 1-bit flag.
## 3. Issue B — train-phase fat-tail spikes (pre-existing, lower priority)
### 3.1 Mechanism (verified from jnct8 diag at step 3895)
| step | pre_clamp_max | pos_max_ema (post) | clamp_win cap (est.) | env.max | popart.σ |
|---:|---:|---:|---:|---:|---:|
| 3880 | 9.4 | 9.65 | ~14 | 87.5 | 87.5 |
| 3890 | 109.1 | 112.6 | ~14 | 121.7 | 121.7 |
| **3895** | **724.3** | **834.5** | ~170 (still less than 724) | **1306.2** | **1306.2** |
| 3950 | 19.3 | 18.79 | ~1250 (over-grown) | 765 | 765 |
Sequence:
1. Step 3890: small fat-tail event (109 magnitude scaled). pos_max_ema bootstraps 9 → 112 via Wiener-α first-observation logic.
2. Step 3895: huge fat-tail event (724 magnitude scaled). pos_max_ema blends 112 + α×724 → 834.
3. clamp_win = MARGIN × prior pos_max_ema. At step 3895's apply moment, prior was ~112 → clamp_win ≈ 170. The 724 magnitude reward exceeds the cap but the controller had already adapted away from the bound.
4. env.max fast-up captures the post-clamp magnitude (around 1306, suggesting post-clamp ≈ 1306 across some account in batch).
5. Slow-decay returns env.max to ~50 over 600 steps (α≈0.005, math: 1306 × 0.9947^600 ≈ 50).
### 3.2 Why this is a design tension
Documented in `pearl_adaptive_reward_clamp_from_positive_tail`: the static cap `[-3, +1]` over-clipped 85% of steps but bounded extremes. The adaptive Wiener-α growth (α=0.4 floor) of pos_max_ema lets clamp_win FOLLOW the magnitude up — losing the bound exactly when it's needed.
The system has rate limits on reward_scale (2% per step) but NOT on pos_max_ema. The Wiener-α floor=0.4 admits 40% of any single observation directly into the EMA.
### 3.3 Possible fixes (NOT addressed in this addendum)
| Option | Approach | Tradeoff |
|---|---|---|
| B-1 | Add per-step rate cap on pos_max_ema growth (e.g. ≤ 1.5× prior, same Schulman pattern as reward_scale) | Slower adaptation to genuine regime shifts |
| B-2 | Use median/percentile instead of max for envelope | More robust but loses "worst-case" signal |
| B-3 | Hard cap on env.max growth per step | Simple, but adds another magic constant |
### 3.4 Recommendation: defer Issue B
The train-phase spikes recover via slow-decay before they propagate to eval (provided no spike fires within ~600 steps of eval boundary). Train-phase mid-stream policy disruption is real but documented as design tension, not as a blocking bug. PPO's per-batch advantage normalization (Phase 4.5) partially compensates.
**Action: log the issue in the addendum but don't fix in this PR.** Track as a follow-up.
## 4. Validation plan for Fix A3
### V1 — Implement
- Allocate `RL_REWARD_SCALE_CONTROLLER_WARMED_FLAG_INDEX` (next free slot)
- Bootstrap to 0 in `with_controllers_bootstrapped`
- Modify `rl_fused_controllers.cu`'s reward_scale block: replace `(trade_count < min_trades)` test with `(warmed_flag == 0)`; set flag once trade_count crosses threshold
- Do NOT add to `reset_session_state` reset list
- Update diag to emit the flag
### V2 — Local smoke
- 800 train + 200 eval, b=16, fold-1 n_folds=3 (same as parent fix smoke)
- Verify: at eval[1], `reward_scale` ≈ train-end value (±2%) — NOT 0.10
- Verify: warmed_flag = 1 at train-end AND eval[1]
### V3 — Cluster validation
- Submit fold-1 walk-forward 10k+2500 (same scale as jnct8)
- Verify: eval[0..50] σ stays within 5× of train-end σ (no 60× explosion)
- Compare eval_summary total_pnl_usd vs jnct8 and 6kghr
## 5. Pearl revision needed
`pearl_popart_reset_at_eval_boundary_shocks_normalization`:
- Add: "The eval-boundary catastrophe has TWO contributing mechanisms — (1) env.max/clamp EMA reset (fixed by 72684ed3e), and (2) reward_scale snapping to bootstrap_floor=0.10 because cumulative_dones counter (slot 660) is reset by reset_session_state, triggering rl_fused_controllers' bootstrap-fraction-floor protection. Mechanism 2 dominates: scale jumps 22× upward, scaled rewards grow 22×, env.max captures them and σ explodes regardless of starting point."
- Note the dual-purpose slot 660 conflict (NORMALIZATION-startup vs PREDICTIVE-safety)
- Cross-reference `pearl_welford_trade_count_is_step_not_trade`
## 6. Done means
- Warmed-flag ISV slot allocated; bootstrap + emit + read paths wired
- `rl_fused_controllers.cu` reward_scale block uses warmed_flag, not trade_count comparison
- Local smoke passes V2
- Cluster validation V3: eval[0..50] σ within 5× of train-end σ, total eval pnl substantially less negative than alpha-rl-jnct8
- Pearl revised
- Issue B (train-phase fat-tail spikes) logged as a separate follow-up
## 7. Risks
- **Risk α (medium)**: The warmed_flag could fire spuriously in early TRAIN if `min_trades` is set very low. Mitigation: keep min_trades high enough that flag only sets after meaningful warmup (current default ~700 dones is fine).
- **Risk β (low)**: Other controllers may also have boot_floor-style protections gated on cumulative_dones. If so, this fix only solves reward_scale; other controllers still snap at boundary. Mitigation: grep for `CUMULATIVE_DONES_INDEX` reads in all `*.cu` and audit each consumer's reset semantics.
- **Risk γ (low)**: σ may still explode at eval boundary from OTHER mechanisms not yet diagnosed (e.g. PPO ratio clamp interaction, action mask change). Mitigation: V3 cluster validation will reveal residual gaps.