foxhunt/docs/superpowers/specs/2026-06-02-regime-invariance-four-interventions.md

Regime-Invariance — Four Interventions for the Train→Eval Gap

Date: 2026-06-02 Status: Spec draft Branch: ml-alpha-regime-observer (current) Reference SHA: 63fc16f17 (v5.2 adaptive scaffold) Linked pearls: pearl_surfer_baseline_was_train_only_never_eval, pearl_reward_signal_anti_aligned_with_pnl, project_adaptive_scaffold_session_2026_06_02

§0. Why this spec exists

The session of 2026-06-01/02 burned 4 cluster smokes iterating on the surfer-scaffold controller (v4 → v5 → v5.1 → v5.2). Each iteration revealed a controller bug introduced by the previous fix. None reached eval verdict before being killed; only v5.2 (alpha-rl-grwwh) was let to run to completion. Pattern matches feedback_going_in_circles_pattern.

Diagnostic conclusion: the controller iterations are tuning the wrong knob. The fundamental failure mode is that the agent learns a trend-specialized policy from fold-1 train data that does not generalize to OOS regime shifts. Phase 5 hold-bonus teaches "ride winners, cut losers" — works in trends, fails in chop. No fade timing fixes this — the inductive bias itself is regime-dependent.

Perplexity consultation 2026-06-02 confirmed this is a known class of problem in financial RL with a documented playbook:

• Domain randomization over market regimes
• Regime-conditional architectures
• Distributionally robust objectives
• Microstructure features tied to invariants, not trend structure

This spec captures the 4 actionable interventions ranked by leverage/risk ratio. Each intervention is self-contained — they can ship in series or in parallel.

§1. Intervention #1 — R-multiple reward (HIGHEST LEVERAGE)

Problem: Phase 5 hold-bonus rewards hold_bonus × √hold_time for profitable holds. This is dollar-magnitude reward in market-dependent units. In trending markets, holding profitable positions accumulates more dollars per step. In choppy markets, holding profitable positions reverses → reward drops → policy un-learns the hold pattern. Reward magnitude scales with the trend regime.

Fix: replace Phase 5 hold-bonus + long-ride-bonus with R-multiple reward normalized by entry-time ATR:

// At close event (done > 0.5):
const float atr_at_entry = unit_atr_at_entry[b];  // captured at flat→open transition
const float r_multiple   = realized_pnl_delta / fmaxf(atr_at_entry, EPS_ATR);
r = r_multiple;  // dimensionless, regime-invariant

Why this generalizes: a $1000 win in high-vol ATR=$2000 → R=0.5. A $300 win in low-vol ATR=$600 → R=0.5. Same reward signal. Policy learns "capture 0.5 ATR worth of move" not "make $X regardless of vol."

§1.1 Implementation

• New CUDA buffer: unit_atr_at_entry_d (per-account f32, captured at flat→open transition in extract_position)
• Read ATR from existing risk-stack signals (already computed for Kelly position sizing)
• Replace Phase 5 steps 2-4 entirely with R-multiple computation at close events
• Drop the surfer-scaffold controller — no longer needed if reward is regime-invariant
• Keep Phase 5 step 1 (entry cost) — fees are regime-invariant in absolute terms; alternatively also divide by ATR

§1.2 Why surfer-scaffold controller can retire

The 753 ISV slot (RL_SURFER_SCAFFOLD_WEIGHT_INDEX) and its 3 config slots (754/755/756) become obsolete. R-multiple reward is its own invariant — no scaffold needed because the gradient is already regime-correct. Slots delete back to RL_SLOTS_END=753. Per feedback_no_feature_flags + feedback_single_source_of_truth_no_duplicates the scaffold infrastructure (controller kernel + bootstrap entries + Phase 5 gating logic) gets deleted in the same commit.

§1.3 Falsification

Cluster smoke fold-1 b=1024 20k+5k. Verdict criteria:

• PASS: train wr ≥ 0.30 by step 5k AND eval wr ≥ 0.28 (within 10% of train) AND eval pnl > 0
• FAIL: eval wr < 0.25 OR eval pnl < −$5M → R-multiple normalization insufficient; proceed to Intervention #2 (regime features) on top

Scope: ~30 LOC kernel + ~50 LOC trainer (ATR capture) + delete ~150 LOC scaffold infra. Net code REDUCTION.

§2. Intervention #2 — Wire regime_observer features into policy network input

Problem: F1.3 already computes regime descriptors (Welford σ_pnl, dead_zone, recovery_factor, eps_live) but they're emitted to diag.jsonl only. The policy network has no input that tells it "we're in a chop regime, behave differently."

Fix: extend the encoder input vector with 4-6 regime features. Policy explicitly conditions on regime instead of baking trend-assumption into its core behavior.

§2.1 Existing signals to wire

Read from ISV:

• RL_REGIME_FLAT_COUNT_INDEX — fraction of accounts that are flat (regime activity proxy)
• RL_REGIME_WELFORD_SIGMA_INDEX — pnl variance EMA (vol regime)
• RL_REGIME_DEAD_ZONE_INDEX — fraction of time in dead-zone (chop indicator)
• RL_REGIME_RECOVERY_FACTOR_INDEX — current/max DD ratio (drawdown regime)
• RL_EDGE_PH_MEAN_INDEX — Page-Hinkley edge-decay signal (per-trade EV regime)

§2.2 Implementation

• Extend encoder input projection from current N dims to N+5 dims
• Bootstrap the new weight columns to small random or zero — agent learns to use them
• 5 new tokens broadcast to every account in the batch via existing ISV-read kernel

Scope: ~80 LOC (encoder projection weight matrix + 5 ISV reads in encoder forward) + checkpoint format bump

§2.3 Caveat

This works ONLY when training data exposes multiple regimes (which #1 alone doesn't solve since we still train on fold 1). Therefore best deployed AFTER or WITH Intervention #3.

§3. Intervention #3 — Cross-fold training (domain randomization)

Problem: Current walk-forward setup gives fixed fold 1 train + fold 1 eval. If fold 1 has a different regime mix than the deployment distribution, the agent over-fits to fold-1's regime. Per pearl_fold_split_is_quarterly_regime_break (SUPERSEDED for current PVC data but the principle stands), folds can differ substantially.

Fix: at episode start, sample a random fold from {0, 1, 2} for training; evaluate on a held-out fold (e.g., the latest quarter). Forces the policy to be regime-mix robust.

§3.1 Implementation

• Modify walk-forward sampler in alpha_rl_train.rs (or trainer init) to randomly sample fold per batch element
• Need 3× the training data resident on PVC — already there (24 months ES MBP-10 per pearl_mbp10_data_is_structurally_broken correction)
• Eval stays on a held-out window (e.g., last 3 months never seen in train)

Scope: ~30 LOC trainer + verification that all 3 folds load cleanly

§3.2 Risk

Trade rate may drop initially (more diverse regimes → more uncertainty → policy more conservative). Worth ~5k extra steps of training to compensate.

§4. Intervention #4 — Mixture-of-experts gated by regime

Problem: A single policy network must serve all regimes. The optimal trading strategy is genuinely different across regimes (trend vs mean-revert). One policy averages them → mediocre everywhere.

Fix: 2-3 specialist policies + gating network. Gating reads regime features (from #2), outputs softmax weights over experts. Each expert specializes on its dominant regime.

Now spec'd separately: see docs/superpowers/specs/2026-06-02-multi-head-policy-with-r-multiple.md for the architecturally clean version (multi-head policy on shared encoder, no reward-shaping diversity, no invented ISV signals). That spec was written after sp-critical-reviewer BLOCKED the prior in-line v6 attempt for inventing a trend_strength signal that does not exist and re-introducing the anti-Pearson reward shaping pattern.

§4.1 Architecture

• Expert A: trend-follower (current Phase 5 baseline behavior)
• Expert B: mean-reversion (different reward shaping — capture short oscillations)
• Optional Expert C: stay-flat (no positions when neither A nor B is confident)
• Gating net: 2-layer MLP on regime features → softmax(K) over experts
• Forward: action = Σ_k gate_k × expert_k(state)
• Backward: each expert's gradient weighted by gate_k; gate net learns from policy gradient

§4.2 Implementation

• 3× the policy head parameters
• Each expert trained on the same data but the reward shaping differs (use a per-expert reward signal — trend-expert uses R-multiple of hold-duration; reversion-expert uses R-multiple of inverse-hold-duration)
• Most complex of the 4 interventions

Scope: ~250 LOC over 2 days. Defer until #1-#3 are validated.

§5. Sequencing & decision points

Order of operations

1. Ship #1 (R-multiple reward) — single commit, kills the surfer-scaffold infra in the same commit. Cluster smoke fold-1 b=1024 20k+5k. Plan at docs/superpowers/plans/2026-06-02-regime-invariance-implementation.md.
2. Wait for #1 verdict. If eval pnl > 0: #1 was sufficient.
3. If #1 eval fails: ship multi-head policy (#4 — separate spec at docs/superpowers/specs/2026-06-02-multi-head-policy-with-r-multiple.md). Architectural fix that does not violate Pearson alignment.
4. If #1+#4 eval fails: ship #3 (cross-fold training) and #2 (regime features) in sequence as remaining levers.

Falsification criteria (each step)

Each intervention has the same eval-gate:

• Eval pnl > 0 (any positive amount) → PASS, ship
• Eval pnl in [−$2M, 0] → MARGINAL, continue to next intervention
• Eval pnl < −$2M → FAIL, deeper rethink (maybe the architecture / data pipeline itself)

Termination clause

If ALL 4 interventions ship and eval still fails, the conclusion is that this neural architecture (mamba2 + DQN-IQN + PPO) cannot extract alpha from MBP-10 ES futures at b=1024 20k steps. We then either:

• Change data scale (much longer training, multi-instrument)
• Change architecture (attention-based, longer context window)
• Change approach (behavior cloning from a profitable rule-based bot, then fine-tune)

§6. Reward-shaping discipline (cross-cutting)

All 4 interventions assume the reward signal must be:

1. Regime-invariant in units (R-multiple, not dollars)
2. Aligned with realized pnl direction (Pearson ≥ 0.7 per pearl_reward_signal_anti_aligned_with_pnl)
3. Sparse at close-events only (not per-step inflation)
4. Bounded (no exponential growth with hold time)

Phase 5 hold-bonus violates (1), (3), (4). The v5.2 adaptive scaffold partially restored (2) but not (1). R-multiple satisfies all four cleanly.

§7. Open questions to resolve before implementation

1. ATR source for R-multiple denominator: use existing Kelly ATR (already computed) or compute fresh? Check Kelly controller for double-update risk.
2. Bootstrap value for ATR at first-trade: ATR EMA needs warmup. Use max(ATR_ema, ATR_floor) with floor ≈ 1 ES tick × tick-multiplier.
3. Eval-boundary discipline: R-multiple uses ATR EMA — must reset at eval boundary per pearl_adaptive_carryover_discipline (or NOT, depending on whether we want eval to start with train-end calibration).
4. Fold sampling weight: equal-weight all 3 folds, or weight by regime-novelty? Active domain randomization (ADR) literature suggests the latter; defer until #1 works.