# Multi-Head Policy with R-Multiple Reward (Intervention #4-clean)

**Date**: 2026-06-02 (ADDENDUM 2026-06-03 — Q-distill + empirical regime-attenuation findings)
**Status**: Spec draft (post-sp-critical-reviewer BLOCK on prior v6 attempt)
**Prerequisite**: ~~Intervention #1 (R-multiple reward) has shipped and the eval verdict is in~~ **SUPERSEDED — see ADDENDUM § R below.** Empirical regime-stratification of Phase 0 baseline shows the multi-head architectural intervention is directly motivated and doesn't require R-multiple as prerequisite.
**Linked specs**: `2026-06-02-regime-invariance-four-interventions.md` (parent)
**Linked pearls**: [[pearl_surfer_baseline_was_train_only_never_eval]], [[pearl_two_alpha_modes_surfer_vs_trend]], [[pearl_reward_signal_anti_aligned_with_pnl]], [[pearl_pi_actor_collapses_without_entropy_floor]]

## §0. Empirical motivation — single-policy regime collapse is EMPIRICAL, not theoretical

The previous v6 attempt tried to motivate multi-head policy via a theoretical "average optimal action" argument. The sp-critical-reviewer correctly BLOCKED that as hand-waving. This spec grounds the motivation in cluster-run evidence:

| Run SHA | Train wr | Eval wr | wr collapse | Per-pearl |
|---|---|---|---|---|
| dd049d9a4 | 0.56 (hold=30) | **0.219** | **2.6×** drop | `pearl_surfer_baseline_was_train_only_never_eval` |
| 87a8259c6 (8gtk2) | wr_max 0.418 / final 0.343 | 0.340 | minor train→eval but train wr peaked then declined within training itself | `pearl_reward_signal_anti_aligned_with_pnl` |
| 63fc16f17 (grwwh, current) | 0.312 final | TBD ~25min | TBD | [[project_adaptive_scaffold_session_2026_06_02]] |

The dd049d9a4 evidence is decisive: a single policy network trained on Phase-5 reward shape achieved profitable train trade-quality and **collapsed to barely-above-random** on eval data. Per [[pearl_two_alpha_modes_surfer_vs_trend]], this isn't a bug — it's "philosophy not bug" — but the philosophy was BAKED INTO weights by training. Without architectural support for multiple philosophies, the single policy can only embed one philosophy at a time.

**This spec hypothesizes** (NOT proves) that giving the policy K heads with different initial biases + a learned gating produces a policy that retains MULTIPLE philosophies in weights, allowing it to switch at inference time based on encoder context. The verification is empirical: cluster smoke + eval-wr stratified by regime markers.

## §1. Design

### §1.1 Architecture (verified against current codebase)

Current architecture (read from `crates/ml-alpha/src/networks/policy_head.rs` and `crates/ml-alpha/src/trainer/integrated.rs`):
- Encoder: mamba2 trunk → encoder_output of size HIDDEN_DIM
- Policy head: Linear(HIDDEN_DIM → N_ACTIONS=11), outputs `pi_logits[b, a]`
- Q head: dueling distributional → `q_logits[b, a, z]`
- V head: scalar critic for PPO

Proposed change (multi-head policy ONLY — Q and V heads unchanged):
- Encoder: UNCHANGED
- K policy heads (K=3 proposed): K × Linear(HIDDEN_DIM → N_ACTIONS), outputs `pi_logits_k[b, a]` for k ∈ {0, 1, 2}
- Gating head: small Linear(HIDDEN_DIM → K), outputs `gate_logits[b, k]`
- Final pi_logits: `pi_logits[b, a] = log( Σ_k softmax(gate_logits[b])[k] × softmax(pi_logits_k[b])[a] )`
- Equivalently in logit space, the mixture is taken on probabilities (not logits) then log'd back. This is the **product-of-experts ↔ mixture-of-experts** distinction; we use mixture (additive over probabilities).

### §1.2 Reward signal — UNCHANGED from Intervention #1

This spec does NOT modify the reward kernel. R-multiple reward from Intervention #1 stays exactly as-is. The reward path is identical for all K heads — they all receive the SAME per-event reward signal.

**This addresses BLOCKER 4 (Pearson alignment)**: since the reward path is unchanged, `Pearson(reward, Δpnl)` is identical to whatever #1 achieves. Multi-head policy adds POLICY expressiveness without touching the reward. No anti-aligned multiplicative shaping is introduced.

### §1.3 Specialization mechanism (no new ISV signals)

**This addresses BLOCKER 1**: heads specialize WITHOUT requiring any new regime-observation ISV slot. Specialization arises from FOUR sources internal to the network:

1. **Different head initialization**: each head's output projection is initialized with a different action bias:
   - Head 0 (trend): bias toward HOLD_LONG / HOLD_SHORT actions (+0.5 logit on HOLD actions, 0 elsewhere)
   - Head 1 (reversion): bias toward FLAT_LONG / FLAT_SHORT actions (+0.5 logit on FLAT actions, 0 elsewhere)
   - Head 2 (flat): bias toward FLAT_FLAT action (+0.5 logit on FLAT only)
2. **Per-head entropy floor**: each head has an independent entropy lower bound preventing collapse to deterministic. Floor value: `log(11) × 0.4` (~0.96).
3. **Gating entropy floor**: gating distribution forced to retain at least `log(K) × 0.5` (~0.55 for K=3) entropy. Prevents gate-collapse-to-one-head mode (per [[pearl_pi_actor_collapses_without_entropy_floor]]).
4. **Per-head auxiliary action-prior loss (§1.4 below)** — continuous regularizer that prevents heads from converging to identical distributions.

The init bias alone (1) is FRAGILE: heads start different but see identical PPO gradient, so they would converge over training. Item 4 provides the CONTINUOUSLY-APPLIED regularizer that maintains specialization across training.

### §1.4 Per-head auxiliary action-prior loss (CLEAN per-expert specialization)

For each head k, an auxiliary KL term pulls its action distribution toward a fixed prior `p_prior_k`:

```
aux_loss_k = β × KL( p_k(a|s) ∥ p_prior_k(a) )
```

Combined loss per head:
```
total_loss_k = ppo_loss_k + aux_loss_k
```

The mixture policy's total loss is `Σ_k g_k × total_loss_k` where `g_k` is the gating weight.

**Priors** (defined per head, FIXED — no learned drift):
- `p_prior_0` (trend): 0.10 uniform over non-HOLD, 0.30 on each of the 3 HOLD_LONG / HOLD_SHORT / HOLD_FLAT actions (sums to 1.0)
- `p_prior_1` (reversion): 0.10 uniform over non-FLAT-of-position, 0.40 on FLAT_LONG and FLAT_SHORT
- `p_prior_2` (flat): 0.40 on FLAT_FLAT, 0.06 uniform over others

(The exact prior weights are tunable but should sum to 1.0 and concentrate ~30-40% of mass on the head's specialty action group.)

**β coefficient**: ISV-driven via new slot `RL_POLICY_AUX_PRIOR_BETA_INDEX`. Bootstrap 0.05 (5% of PPO loss magnitude). When β=0, the spec reduces to §1.3 init-bias-only — useful as ablation.

**Why this addresses Pearson alignment (re-confirms BLOCKER 4 stays addressed)**: the aux_loss is added to the POLICY LOSS, not to the REWARD signal. `Pearson(reward, Δpnl)` is unchanged because the reward kernel is unmodified. The aux loss is a REGULARIZER on policy output; it does not alter the gradient direction of the PPO surrogate's REWARD term — only adds an orthogonal pull toward each head's prior.

**Sign-preservation**: KL divergence is always ≥ 0; aux_loss is always positive; subtracting it from the maximization objective always reduces the policy's deviation from prior. Cannot induce sign-flips in the reward direction.

**Reference**: standard mixture-of-experts auxiliary loss pattern. Shazeer et al. 2017 (sparsely-gated MoE), Lepikhin et al. 2020 (GShard) — well-studied technique, low novelty risk.

## §2. ISV slot additions (4 slots, verified against current `RL_SLOTS_END`)

Current `RL_SLOTS_END = 757` (post-grwwh / v5.2, will be 757 again after #1 R-multiple ships per the #1 plan). New slots:

```
RL_POLICY_NUM_HEADS_INDEX            usize = 757   // K (current 3 — could allow runtime override)
RL_POLICY_GATING_ENTROPY_FLOOR_INDEX usize = 758   // ~0.55 (log(K) × 0.5)
RL_POLICY_HEAD_ENTROPY_FLOOR_INDEX   usize = 759   // ~0.96 (log(11) × 0.4)
RL_POLICY_AUX_PRIOR_BETA_INDEX       usize = 760   // 0.05 bootstrap; β for per-head action-prior KL
```

`RL_SLOTS_END = 761` after this spec ships.

**No new producer kernels needed**. The entropy floors are read by the EXISTING entropy controller infrastructure (per `crates/ml-alpha/cuda/rl_entropy_coef_controller.cu` and similar).

## §3. Implementation tasks (concrete, ordered)

### §3.1 Network changes (Rust)
1. Modify `PolicyHead` struct in `crates/ml-alpha/src/networks/policy_head.rs`:
   - Add `Vec<Linear>` for K head projections instead of single `Linear`
   - Add `Linear` for gating projection
   - Adam optimizer states for each (use existing `Adam` per-tensor pattern)
2. Modify forward pass:
   - Compute K head logits in parallel (matmul against shared encoder_output)
   - Compute gate logits
   - Mixture via `log Σ softmax(gate) × softmax(head)`
3. Backward pass: gradient flows through gating × head expression. Standard PPO surrogate on the mixture pi_logits.

### §3.2 Kernel changes (CUDA)
Multi-head policy is mostly Rust-side because the heads are linear projections. New kernels needed:
- `rl_policy_mixture_forward.cu`: takes K head logits + gate logits → produces mixture logits (LogSumExp over heads with gate weighting). Block-tree-reduce, no atomicAdd. ~80 LOC.
- `rl_policy_mixture_backward.cu`: backward through the mixture. ~120 LOC.
- `rl_policy_aux_prior_loss.cu`: computes Σ_k g_k × β × KL(p_k ∥ p_prior_k) and gradient ∂L_aux/∂head_k_logits. Reads `RL_POLICY_AUX_PRIOR_BETA_INDEX` for β. ~60 LOC.

Per `feedback_no_nvrtc`: pre-compiled cubins via `build.rs`.

### §3.3 Trainer integration
- Allocate K head weights + gating weights in trainer constructor
- Initialize each head with the action-bias pattern from §1.3
- Wire entropy floors via existing controller infrastructure
- Per-head entropy diagnostic emitted to `diag.jsonl` (3 new leaves: `policy.head_0_entropy`, etc.) + gate entropy diagnostic (1 leaf)

### §3.4 Diag emission
Add to `build_diag_value`:
- `policy.head_count = K`
- `policy.head_k_entropy[k]` for k in 0..K
- `policy.gate_entropy`
- `policy.gate_probs[k]` for k in 0..K (mean across batch)

EXPECTED_LEAVES: 679 → 679 + (K + 1 + K) = 686 (K=3).

## §4. Falsification — strict kill gates (addresses BLOCKER 5)

### §4.1 Per-step kill criteria (per `feedback_kill_runs_on_anomaly_quickly`)

Kill the cluster smoke if ANY of:
- `Pearson(reward, Δpnl) < 0.5` at step 2000 (sparse close events should align by then)
- `gate_probs[k] < 0.05` for any k at step 5000 (mode collapse — gate ignored a head)
- `gate_entropy < log(K) × 0.4` at step 5000 (gate determinism — should be > 0.45)
- `action_entropy < log(11) × 0.5` at step 5000 (overall policy collapse)
- `realized_pnl_cum < -$50M` at step 10000 (catastrophic loss, abandon)

### §4.2 Verdict gates (PASS / FAIL)

PASS criteria (ALL must hold):
- Train Pearson(reward, Δpnl) ≥ 0.6 at steps 1500-2000
- Train wr ≥ 0.35 by step 10k (above any reasonable random baseline; threshold above current grwwh peak of 0.31)
- Train action_entropy ∈ [log(11) × 0.5, log(11) × 0.85] (concentrated but not collapsed)
- Train gate_entropy ≥ log(K) × 0.4 (heads actively used)
- **Eval pnl > 0** (the actual generalization test)
- **Eval wr ≥ 0.30** (above the random-baseline floor for K=11 actions)
- Per-regime eval wr shows specialization: split eval rows by `session_pnl_variance_ema` quartiles; the spread `max(wr_q) − min(wr_q)` should exceed `0.05` (heads pulling agent in different directions based on regime).

FAIL criteria:
- Any per-step kill gate fires
- Eval pnl < -$2M (defines the fail boundary; in [-$2M, 0] is marginal → next intervention)
- Per-regime spread < 0.02 (heads didn't specialize → architectural gain near zero)

### §4.3 Per-regime stratification methodology

The falsification needs regime markers that ALREADY EXIST in diag.jsonl. Verified existing signals:
- `risk_stack.regime.session_pnl_variance_ema` (slot 700) — proxy for vol regime
- `risk_stack.regime.dead_zone_flag` (slot 696) — proxy for chop regime
- `risk_stack.regime.recovery_factor` (slot 699) — proxy for drawdown regime

Stratify eval steps by quartiles of `session_pnl_variance_ema` (low-vol vs high-vol regimes); compute wr separately on each quartile. The hypothesis is that multi-head policy will produce different wr in different quartiles (heads specializing), while single-policy grwwh would produce uniform wr across quartiles.

## §5. Risks and mitigations

### §5.1 Mode collapse on gating
Risk: gating distribution collapses to single head (`gate_probs ≈ [1, 0, 0]`); other heads become dead weight.

Mitigation: `RL_POLICY_GATING_ENTROPY_FLOOR_INDEX` enforces hard floor on gate entropy. Per-step kill gate at step 5000 catches this.

Reference: [[pearl_pi_actor_collapses_without_entropy_floor]] — entropy floor canonically applied to action distribution; here applied to gating distribution at the same conceptual layer.

### §5.2 Heads converge to identical behavior — MITIGATED by §1.4 aux prior loss
Risk: with shared encoder + shared reward, the heads might converge to identical action distributions, making multi-head equivalent to a single head with K× parameters wasted.

**Primary mitigation**: §1.4 per-head auxiliary action-prior KL loss. Continuously pulls each head toward its distinct prior, preventing convergence. Standard MoE specialization technique (Shazeer 2017).

**Secondary mitigations**: per-head init bias (§1.3 item 1) + per-head entropy floor (§1.3 item 2). The aux prior provides the lasting pressure; init + entropy provide the boundary conditions.

Diagnostic: emit pairwise KL between heads to diag (`policy.head_pairwise_kl[i,j]`). If KL drops below 0.1 across all pairs, heads have converged → β too low or aux prior loss disconnected.

**β-tuning escape hatch**: `RL_POLICY_AUX_PRIOR_BETA_INDEX` is ISV-driven. If empirically heads converge despite β=0.05 bootstrap, the controller can adapt β upward via an ISV controller (similar pattern to existing entropy-coef controller). Add this as a follow-up controller only if needed.

### §5.3 Training instability from mixture
Risk: PPO surrogate on a mixture has higher variance than on a single Gaussian-ish policy. Could destabilize training.

Mitigation: existing PPO ratio clamp from B-7 / B-9 work handles this. Per-batch advantage normalization (Phase 4.5) helps.

---

# ADDENDUM 2026-06-03 — Q-distill compatibility + Empirical regime-attenuation findings

## §R.0 — Why this addendum exists

The original spec was written assuming PPO trains π. Per [[pearl_foxhunt_pi_trained_by_q_distillation_not_ppo]] (Phase 1B-C STOP discovery): **foxhunt's π is trained by Q→π distillation, NOT by PPO surrogate.** `crates/ml-alpha/src/trainer/integrated.rs:5577` zeros `ss_pi_grad_logits_d` BEFORE the Q-distill kernel writes its KL gradient. The PPO surrogate runs as a diagnostic-only loss.

This addendum revises the spec to:
1. Document Q-distill gradient flow for multi-head (it still works — just via a different channel)
2. Incorporate empirical findings on VSN regime-attenuation
3. Update falsification gates with concrete numerical anchors from baseline data
4. Make multi-head a STANDALONE intervention (not contingent on R-multiple)

## §R.1 — Empirical motivation refined (Phase 0 baseline data)

Phase 0 baseline (`cfaa420bd`, b=128 mid-smoke, eval_pnl = −$4.46M) action distribution stratified by regime quartiles:

| Regime marker | TVD(Q1, Q4 action dist) | Pnl Q1 vs Q4 | Interpretation |
|---|---|---|---|
| `popart_envelope_max` (agent-driven via reward signal) | **0.169** | −$1.99M vs −$32k | Policy IS partially regime-aware via reward-signal channel |
| `session_pnl_variance_ema` (market-driven vol) | **0.015** | −$1.40M vs −$2.05M | Policy is NOT regime-aware on market vol — despite vol context in encoder input at `regime[3]` (snap_feature out[23]) |

**The asymmetry is diagnostic.** The policy can condition on popart_envelope (which modifies reward magnitude directly) but cannot condition on market vol (which exists in encoder input as `regime[3] = log1p(ema_rv_med × 1e4)`).

### The mechanism

`crates/ml-alpha/cuda/variable_selection.cu` implements TFT-style softmax-gated feature selection over the first 40 features of the 56-dim encoder input. **Softmax gates sum to 1.0 across 40 features — average gate ≈ 0.025.** For `regime[3]` to influence Mamba2 meaningfully, it would need an above-average gate, but it competes with 39 other features (microstructure z-scores, OFI deltas, trade activity, spread, Δt Fourier features) for that scarce attention budget. The training signal (BCE labels + V regression + Q-distill targets) doesn't reward attending to `regime[3]` specifically, so VSN learns to allocate gates to whatever helps immediate prediction targets — not regime-conditional policy.

**Multi-head policy with regime-gated routing bypasses VSN's softmax bottleneck for regime context.** The gating head reads regime features directly (a parallel channel that doesn't compete via VSN's 40-way softmax). K heads receive a fully-dimensioned regime signal via the gating output (K-way softmax over a small K, not 40-way) and can specialize per regime.

### Policy collapse signature (additional)

Phase 0 baseline action_hist shows **action 2 (probably FLAT_SHORT-class) dominating 40-54% of action mass.** This is effectively a 1-action-dominated policy with noise — the encoder's representation has collapsed to "mostly emit action 2." Multi-head + per-head aux KL priors directly attacks this by REQUIRING per-head action distributions to retain mass across distinct action groups.

## §R.2 — Q-distill compatibility (the architectural correction)

Original §1.1 assumed PPO surrogate writes the policy gradient. Corrected flow:

```
Current single-head:
  encoder → h_t → policy_head Linear(128→11) → π_logits
                                 ↓
                            ss_pi_grad_logits_d ← (zeroed line 5577) ← Q-distill KL gradient ← Q-Thompson sample from PER
                                 ↓
                            policy_head backward + Adam

Proposed multi-head:
  encoder → h_t (HIDDEN_DIM=128)
                ↓
            K policy heads × Linear(128→11) → π_logits_k for k ∈ {0,1,2}
                ↓
            Gating head Linear(GATE_INPUT_DIM → K) → gate_logits → softmax → gate[k]
                ↓
            π_logits = log Σ_k gate[k] · softmax(π_logits_k)   ← mixture distribution
                ↓
            ss_pi_grad_logits_d ← (zeroed) ← Q-distill KL gradient writes here
                ↓
            π_logits.backward → K head backward + gating head backward
                ↓
            Adam updates on K head weights + gating head weights ONLY
```

**Critical: Q-distill kernel doesn't care about multi-head structure.** It writes a gradient on `π_logits` regardless of how `π_logits` was computed. The backward pass through the mixture computes per-head + gate gradients automatically via chain rule.

The PPO surrogate gradient (still computed but still zeroed by line 5577) remains diagnostic-only. **No PPO behavior change required.** The aux KL prior loss (§1.4 in original spec) is added DIRECTLY to the policy backward pass, NOT zeroed by line 5577 — it lives in a separate gradient buffer that's added to `ss_pi_grad_logits_d` AFTER the Q-distill write. See §R.6 below for the gradient-merge sketch.

## §R.3 — Gating head input (the load-bearing design choice)

This is the single most important design choice in the addendum. Three options ordered by empirical motivation:

### Option A — Gate from encoder output `h_t` only (original spec §1.1 default)
```
gate_logits = Linear_HIDDEN_DIM_to_K(h_t)
```
**Problem**: h_t is the same representation that the empirical evidence shows under-encodes regime context. VSN's softmax bottleneck has already attenuated regime[3] before h_t is produced. Gating from h_t propagates the same bottleneck into the gating head.

### Option B — Gate from regime features DIRECTLY (recommended)
```
// Read the 6 raw regime features per batch element (computed by loader,
// already in trainer's step_regime_d device buffer).
regime_h = step_regime_d[b][0..6]   // 6-dim raw regime context
gate_logits = Linear_6_to_K(regime_h)
```

**Why this works**: bypasses VSN entirely. The gating head's softmax is K-way (K=3) so each regime feature gets ≈ 1/3 attention budget, not 1/40. The gating head can fully condition on `regime[3]` (vol) without VSN's gating constraint.

**Where to source**: `step_regime_d` is allocated in `crates/ml-alpha/src/trainer/perception.rs:1795` and populated per-batch from the loader. It's already on device, already accessible per (b, k). For multi-head gating we want one regime vector per batch element (use t=K-1 of the BPTT window for "current regime") or aggregate across K (mean).

### Option C — Gate from concat(h_t, regime_features) (compromise)
```
gate_input = concat(h_t, regime_h)   // dim 128 + 6 = 134
gate_logits = Linear_134_to_K(gate_input)
```
Gives gate access to both the encoder's processed state AND raw regime. More parameters, more risk of overfit but more expressiveness.

**Decision**: Default to **Option B**. The empirical evidence (TVD=0.015 on session_pnl_var) shows h_t doesn't encode market vol; routing the gate through h_t inherits that limitation. Option B is the cleanest bypass of the diagnosed bottleneck.

## §R.4 — Per-head aux KL priors (refined)

Original §1.4 prior assignments:
- Head 0 (trend): biased toward HOLD actions
- Head 1 (reversion): biased toward FLAT actions
- Head 2 (flat): biased toward FLAT_FLAT

**Refined per empirical Phase 0 data**:

Looking at the action distribution in popart Q1 (worst regime, −$1.99M loss):
- Action 0+1 (FLAT-ish): 0.10
- Action 2 (currently dominant): 0.40
- Action 5+6 (HOLDs): 0.22
- Action 7+8: 0.24

The Q4 distribution (best, −$32k) is heavily skewed toward action 2 (0.54) and away from HOLDs (0.11). The empirically-correct Q1 policy would likely DECREASE the HOLDs and increase decisive transitions. So the priors should target:

- **Head 0 (defensive — for high vol / drawdown recovery regime)**: bias toward FLAT_FLAT + decisive transition actions. Prior weights ~0.40 on action 0, 0.10 each on 1-3, 0.05 each on others.
- **Head 1 (active — for low-vol stable regime)**: bias toward HOLDs and transitions. Prior weights ~0.18 each on actions 5-8, 0.05 on others.
- **Head 2 (transition — for regime change events)**: bias toward action 2 (the currently-dominant one). Prior weights ~0.35 on action 2, 0.10 each on 1+3, 0.05 elsewhere.

These priors are SCAFFOLDING — `β=0.05` so the priors don't dominate training. Heads can diverge from priors if Q-distill signal pushes them elsewhere.

## §R.5 — Falsification gates (concrete numerical anchors from baseline)

Original §4.2 gates revised with measurable anchors:

### G1 — Regime conditioning EMERGES via the multi-head architecture
- TVD(Q1 action dist, Q4 action dist) on **session_pnl_variance_ema axis** ≥ **0.10** by step 1500
- (Baseline Phase 0: 0.015 — multi-head should lift this 6× if regime routing works)
- This is the **direct test that multi-head bypasses VSN's regime bottleneck**.

### G2 — Per-head specialization holds
- Pairwise KL between any two heads ≥ 0.20 by step 1500
- Gate entropy ≥ log(K) × 0.45 = 0.49 (heads actually used, not collapsed to one)
- No `gate_probs[k] < 0.10` for any k at step 5000 (mode-collapse kill)

### G3 — Action diversity recovers (policy collapse signature)
- Argmax action mass ≤ **0.30** (down from baseline 0.40-0.54 dominated by action 2)
- This tests whether the per-head priors + Q-distill differentiation prevent collapse

### G4 — Eval pnl improvement on POPART Q1 (lower variance than aggregate)
- Sum of Δpnl in popart_envelope Q1 quartile improves by **≥ $1M** vs Phase 0 baseline (Q1: −$1.99M)
- Per-quartile measurement has lower variance than aggregate (since quartile selection itself smooths some noise)
- **This is the load-bearing empirical gate** — if multi-head moves the worst regime quartile, the chain works

### G5 — Aggregate eval pnl (informational, given noise floor)
- `eval_summary.total_pnl_usd` better than seeds 42-44 mean by 2σ (σ ≈ $1.9M empirical)
- Required improvement: ~$3.8M (e.g., from −$5.6M mean to better than −$1.8M)
- This is a HIGH BAR. G4 is the primary gate; G5 is corroboration.

### G_kill — Per-step kill criteria
- Action entropy < log(11) × 0.30 = 0.72 at step 5000 (policy collapse)
- Eval realized_pnl_cum_usd < −$10M at step 10000 (catastrophic abandon)
- Cross-source consistency drift > $50 (Phase 0 contract violation)
- Determinism check fails at any seed

### Stratified verdict methodology
Per the noise floor finding (σ ≈ $1.9M at b=128 single-seed), use multi-seed AND per-quartile measurement:
- Run smokes at seeds 42, 43, 44 (3-seed minimum)
- For each, compute per-quartile Δpnl on popart_envelope
- Compare mean per-quartile improvement vs the 3-seed Phase 0 baseline distribution

## §R.6 — Gradient flow sketch (implementation guidance)

```
// Forward (multi-head):
for k in 0..K:
    pi_logits_k[b, a] = w_k[a, :] @ h_t[b, :] + b_k[a]    // K linear heads
gate_logits[b, k] = w_gate[k, :] @ regime_h[b, :] + b_gate[k]   // gating head reads regime_h DIRECTLY
gate_probs[b, k] = softmax(gate_logits[b])
pi_probs_k[b, a] = softmax(pi_logits_k[b])
pi_probs[b, a] = Σ_k gate_probs[b, k] * pi_probs_k[b, a]   // mixture in probability space
pi_logits[b, a] = log(pi_probs[b, a] + 1e-12)              // back to logit space for downstream

// Backward (Q-distill + aux):
ss_pi_grad_logits_d[b, a] := 0   // existing zero at line 5577
rl_q_pi_distill_grad writes into ss_pi_grad_logits_d  // existing path, unchanged
ss_pi_aux_grad_logits_d[b, a] := Σ_k β · gate_probs[b, k] · ∂KL(pi_probs_k || pi_prior_k) / ∂pi_logits_k[b, a]   // NEW
ss_pi_grad_logits_d[b, a] += ss_pi_aux_grad_logits_d[b, a]   // merge

// Then policy_head.backward (existing path) propagates ss_pi_grad_logits_d
// back through the mixture (∂pi_logits/∂pi_logits_k via mixture jacobian)
// and through the gating head (∂pi_logits/∂gate_logits via mixture jacobian)
// Adam updates: w_k, b_k for each head + w_gate, b_gate
```

The mixture's backward jacobian is standard (Shazeer 2017 MoE; standard implementation in production frameworks). ~100 LOC for two backward kernels (mixture_backward + aux_prior_backward).

## §R.7 — Scope estimate (revised)

| Component | Original spec | Revised |
|---|---|---|
| Multi-head policy struct + Adam states | ~80 LOC | ~80 LOC |
| Mixture forward kernel | ~80 LOC | ~80 LOC |
| Mixture backward kernel | ~120 LOC | ~140 LOC (handles K=3 with gate jacobian) |
| Aux prior loss + backward | ~60 LOC | ~80 LOC |
| Gating head: regime-direct input wiring | not in original | ~60 LOC (read step_regime_d, project to K) |
| Bootstrap entries for new ISV slots | ~20 LOC | ~20 LOC |
| Diag emission (per-head + gate + TVD diagnostics) | ~40 LOC | ~50 LOC (more diagnostics for regime conditioning) |
| Tests (invariants for mixture, prior pull, gate entropy) | ~150 LOC | ~200 LOC |
| **Total** | **~550 LOC** | **~710 LOC** |

Multi-week intervention. Single coder commit (per-component tasks would split it).

## §R.8 — Why this is now the right intervention (over R-multiple or further reward work)

Per ADDENDUM 2026-06-02e of `pearl_reward_signal_anti_aligned_with_pnl`: reward dimension has been tested twice (Phase 1A pure-pnl, Path A GAE-V). Both produced null/noise-level changes. Continuing in the reward dimension would be "going under the streetlight" (per `feedback_investigation_first_falsification_methodology`).

Multi-head policy targets a different mechanism:
- **Architecturally addresses the diagnosed VSN bottleneck** (regime[3] attenuation)
- **Empirically motivated**: TVD(0.015) on vol axis vs TVD(0.17) on popart axis proves the policy CAN condition on regime when the signal reaches it through the right channel
- **Q-distill compatible**: gradient flows through mixture's standard backward jacobian; the existing Q-distill kernel is unchanged
- **Targets the popart Q1 regime (worst $1.99M of $4.46M loss)** with measurable per-quartile gate
- **Single ~700 LOC intervention** (multi-week but bounded)

If multi-head fails G1 (TVD on vol axis doesn't lift): the eval-collapse is genuinely not regime-mediated, and the architecture/data are the bottleneck. We'd then have empirical justification for the deeper interventions (regime-stratified training, encoder architecture changes, etc.).

### §5.4 Eval doesn't show specialization (heads pulled to average)
Risk: even with all mitigations, eval might produce uniform wr across regime strata → no specialization actually achieved.

Mitigation: this IS the falsification gate. If specialization spread < 0.02, the architectural change failed. The conclusion is then NOT to ship #4.B; reroute to #3 (cross-fold training) or #4.C (full MoE).

## §6. Open questions (to resolve before implementation, NOT during)

1. **Initial K**: 3 (trend/reversion/flat) vs 5 (add trend-reversal, vol-breakout)? Larger K is more expressive but harder to keep specialized. Recommend K=3 to start.
2. **Loss decomposition**: PPO loss on the mixture (single loss) vs per-head PPO loss weighted by gate (decomposed). Empirically the mixture-loss is simpler and standard in MoE literature. Recommend mixture-loss.
3. **Eval boundary discipline**: does the gating's running state need reset at eval boundary? Per [[pearl_adaptive_carryover_discipline]], any stateful adaptive signal needs either reset or re-bootstrap. The gating is INSTANT (computed from encoder state per step) so no carry-over — but the per-head entropy controller IS stateful. Use neutral sentinel reset at eval boundary.
4. **Distillation interaction**: existing B-11-β-style Q→π distill could conflict with mixture policy. Verify the distill grad kernel handles K-head mixture output correctly before enabling distill.

## §7. Out of scope

- **Per-expert REWARD shaping** (which the BLOCKED v6 attempt proposed via multiplicative `r_trend × hold_bonus_factor`) — explicitly rejected because it violates Pearson alignment per §6.2 of the parent spec. NOTE: per-expert POLICY-LOSS shaping IS in scope (§1.4 aux prior loss) — the distinction is reward-vs-loss; reward stays clean (Pearson-preserving), loss has per-head regularizer.
- Per-expert IQN τ (different risk preference per head) — clean alternative to §1.4 but interacts with the existing IQN risk-stack. Defer until §1.4 is validated. Add as §1.5 follow-up if needed.
- Per-expert encoder (full MoE / §4.C of parent) — defer to follow-up spec if multi-head with shared encoder is insufficient.
- Cross-fold training (#3) — orthogonal intervention; can be combined with multi-head as a follow-up.

## §8. Spec/plan separation discipline (addresses BLOCKER 2)

This spec is INTENTIONALLY separate from `2026-06-02-regime-invariance-four-interventions.md` and from any implementation plan. The implementation plan for this spec will be drafted in a separate document AFTER:
1. Intervention #1 (R-multiple) ships and eval verdict is collected
2. If eval verdict is negative, sp-critical-reviewer reviews this spec
3. If reviewer APPROVES, the implementation plan is drafted

Per `feedback_no_partial_refactor`: spec and plan must align atomically. The plan does not exist yet.

## §9. Verified facts (audit trail for sp-critical-reviewer)

The following claims in this spec have been verified by direct code inspection:

| Claim | Verification |
|---|---|
| `RL_SLOTS_END = 757` post-grwwh | `crates/ml-alpha/src/rl/isv_slots.rs:1771` |
| `regime_observer` writes slots 696-701 (no trend_strength) | `crates/ml-alpha/cuda/rl_regime_observer.cu:36-100` and `crates/ml-alpha/src/rl/isv_slots.rs:1522-1534` |
| Reward kernel is `rl_fused_reward_pipeline.cu` (current Phase 5 logic) | `crates/ml-alpha/cuda/rl_fused_reward_pipeline.cu` |
| Per `feedback_pi_actor_collapses_without_entropy_floor`: entropy floor canonical pattern | pearl exists in MEMORY.md |
| Per `feedback_kill_runs_on_anomaly_quickly`: aggressive kill gates required | pearl exists in MEMORY.md |
| Per `feedback_no_partial_refactor`: spec/plan must align | pearl exists in MEMORY.md |

Claims NOT verified that the reviewer should check:
- Exact `PolicyHead` struct layout in `policy_head.rs` (referenced but not opened)
- Exact `EXPECTED_LEAVES` value pre-this-spec (referenced as 679 from prior R-multiple invariant test)
- Whether mixture-of-policies has any existing precedent in foxhunt (search for prior MoE attempts)
