gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revert "feat(rl): B-11-β Q-distill informativeness gate"

Browse Source 
This reverts commit b93971726d.
This commit is contained in:
jgrusewski
2026-06-01 14:57:41 +02:00
parent b93971726d
commit f428be794b
6 changed files with 6 additions and 632 deletions
Download Patch File Download Diff File Expand all files Collapse all files

326
docs/superpowers/specs/2026-06-01-b11-beta-q-distill-informativeness-gate.md
View File

@@ -1,326 +0,0 @@
# B-11-β — Q-Distill Informativeness Gate
**Date:** 2026-06-01
**Status:** Draft (pre-implementation)
**Branch target:** `ml-alpha-regime-observer`
**Predecessors:** B-7 (clamp toggle) → B-8 (σ_welford) → B-9 (atom saturation) → B-10 (cascade diagnostic)
**Cluster evidence:** `alpha-rl-88f5c` at SHA `c1dc84a34` (B-10 smoke, 2k+500 b=1024 fold-1 H100)
**Pearls referenced:**
- [[pearl_q_to_pi_distillation_breaks_v_vanishing_trap]] — original distill activation rationale (wwcsz 2026-05-24)
- [[pearl_q_thompson_actor_makes_pi_dead_weight]] — Q-π decoupling pattern
- [[pearl_pi_actor_collapses_without_entropy_floor]] — entropy regularization is load-bearing — DO NOT remove distill blindly
- [[feedback_isv_for_adaptive_bounds]], [[feedback_no_quickfixes]], [[feedback_adaptive_not_tuned]]
**Research references** (perplexity 2026-06-01):
- SAC adapts entropy temperature α via target-entropy constraint (Soft Actor-Critic); analogous to our τ.
- Count-based Soft Q-Learning: `1/β(s)` (effective τ) decreases as state-visits grow → distill informativeness rises with Q calibration.
- **Key finding**: when Q is uniform across actions, `softmax(Q/τ)` is uniform for ANY τ — so the fix has to either gate the distill loss OR scale τ to Q's actual range, not just lower a fixed τ.
---
## 0. Discipline boundary — first behavior change since B-7
The B-7 → B-8 → B-9 → B-10 chain (committed `55d049ecf` → `1739d9c17` → `29b5acad5` → `8c7ce02da` + `c1dc84a34`) was **observability-only** apart from B-7's reward-clamp ISV toggle. B-11-β is the **first spec since B-7 that mutates a per-step gradient value** (`grad_distill` in `rl_q_pi_distill_grad.cu` is scaled by `λ_eff` instead of `λ_base`). Implications:
1. **Determinism explicitly broken** vs prior cluster runs (`alpha-rl-8gtk2`, `alpha-rl-88f5c`). V4 (§4) admits this. Cluster comparison is on aggregate signals (eval pnl, mean l_pi trajectory), NOT byte-identical rewards/dones/actions.
2. **A/B testing pattern** uses ISV-driven boolean toggle (slot 743) for cluster-side regression — direct precedent: B-7 used the same pattern via `RL_REWARD_CLAMP_ENABLED_INDEX = 724`. See §6 row 1 + §3.1 comment for the precedent citation.
3. **Falsification criteria gate the next iteration**: §2.G5 buckets predetermine whether B-11-γ or B-11-α need to follow based on cluster-observed eval pnl.
---
## 1. Motivation
`alpha-rl-88f5c` (B-10 cluster smoke, 2k+500 b=1024 fold-1 H100 at SHA `c1dc84a34`) **confirmed hypothesis β** from the B-10 spec's falsification matrix:
| B-10 signal @ step 1706 | Value | Threshold | Verdict |
|---|---|---|---|
| `q_distill_target_entropy` | **2.3976** / ln(11)=2.398 max | > 0.95 × ln(11) = 2.28 = β | **CONFIRMED (99.98% of max)** |
| `q_dist_entropy_mean` | 2.31 / ln(21)=3.045 (76%) | > 2.89 = α | not yet met (Q is informative) |
| `q_value_range_mean` | 13.79 | — | Q has substantial discrimination across actions |
| `q_distill_temperature` | 20.18 | — | Current ISV value (controlled by SAC-α auto-tune) |
| `q_distill_lambda` | 0.224 | — | Current ISV value (controlled by KL-target Schulman step) |
| `q_pi_agree_ema` | 0.97 | — | Q-π aligned at smoke scale |
| `ppo_ratio_clip_rate` | 0.225 | > ε/2 = 0.025 = γ | also confirmed but secondary |
**The mechanism**:
- Q's expected-value range is 13.79 across actions (Q HAS preferences).
- Distill target = `softmax(E_Q / τ=20.18)`. With Q_range=13.79 and τ=20.18, `softmax(13.79/20.18)` ≈ `softmax(0.68)` → distribution is barely concentrated, near-uniform.
- Target entropy 2.3976 vs max ln(11)=2.398 → distill target is INFORMATION-FREE.
- `q_distill_lambda = 0.224` × KL(uniform_target ‖ π_new) → distill loss = pure max-entropy regularization on π, independent of Q content.
- π is being pushed to maximum entropy by distill while PPO simultaneously pushes it toward action-weighted advantages → policy thrashes between conflicting gradient directions → `l_pi` diverges 92.5 (best) → 80,349 (step 20k in `alpha-rl-8gtk2`).
- Q-π agreement starts at 0.97 (aligned) but the distill-driven entropy regularization compounds over 20k steps → policy converges to near-uniform regardless of Q's preferences → eval pnl −$218M at full run.
**B-11-β fixes this by gating λ_distill smoothly on target informativeness**: when `softmax(E_Q/τ)` is near-uniform (high entropy), the effective `λ` attenuates toward 0 because the distill term provides no informational signal — only entropy regularization that fights PPO. When the target IS informative (low entropy), `λ` operates at full strength so Q→π distillation actually transfers Q's preferences.
This is **not** the same as removing distill entirely (`pearl_pi_actor_collapses_without_entropy_floor` — entropy regularization is load-bearing). It is making distill self-aware: the term contributes when it has signal, attenuates when it doesn't. The SAC-style entropy controller (existing path) continues to provide an entropy floor via the dedicated `RL_SAC_ALPHA_INDEX` term, which is structurally independent of the distill informativeness gate.
## 2. Goals
### G1 — Smooth informativeness-gated λ
Effective lambda for the distill gradient:
```
gate = max(0, 1 − target_entropy / ln(N_ACTIONS))^p
λ_eff = λ_base × gate
```
where `λ_base = isv[RL_Q_DISTILL_LAMBDA_INDEX]` (existing controller's output), and `p` is an ISV-driven sensitivity exponent (default 2.0). The gate is smooth across all target entropy values:
| target_entropy / ln(N_ACTIONS) | gate (p=2) | interpretation |
|---|---|---|
| 0.0 (target = one-hot) | 1.000 | full distill: Q's argmax dominates |
| 0.5 (mild concentration) | 0.250 | partial distill |
| 0.9 (mostly uniform) | 0.010 | mostly off |
| 0.95 (near uniform) | 0.0025 | effectively off |
| 0.99 (uniform) | 1e-4 | off |
### G2 — Preserve SAC-α entropy floor (NON-GOAL DELETION)
The existing entropy-regularization path in `rl_q_pi_distill_grad.cu:121` (the `grad_entropy = -α × pi_a × (log_pi_a + 1 + H)` term, gated by `RL_SAC_ALPHA_INDEX = 581`) is **untouched**. B-11-β attenuates only `grad_distill = λ × (π_θ(a) - π_target(a))`. The SAC-α auto-tune continues to maintain target entropy; distill simply stops contributing pseudo-entropy regularization when it has no signal to add.
This separation matters because `pearl_pi_actor_collapses_without_entropy_floor` documents the catastrophic mode (entropy → 0.7, action collapse to Hold @ alpha-rl-9k9x6) that the SAC-α path was designed to prevent. B-11-β does NOT remove that protection.
### G3 — ISV-toggleable + adaptive sensitivity
Per `feedback_isv_for_adaptive_bounds`:
- `RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX` (default 1.0 = ON). Set 0.0 for A/B regression on next cluster run.
- `RL_Q_DISTILL_INFO_GATE_SENSITIVITY_INDEX` (default 2.0, the exponent `p`). Tune via re-seed if cluster shows under/over-attenuation.
### G4 — Diag observability of the gate
Emit `policy_diagnostic.q_distill_lambda_effective = λ_eff` to diag. Together with the existing `isv_config.q_distill_lambda` (the base λ from the KL-target controller) the diff makes the gate's per-step action observable.
### G5 — Falsification criteria (the cluster run after B-11-β)
- If `eval pnl > −$50M` at 20k+5k fold-1 b=1024 (improvement of $50M+ over baselines −$100M to −$218M) → **β was the dominant cause, B-11-β is the fix**.
- If `eval pnl ≈ −$50M to −$100M` → β contributed materially but not exclusively; move to B-11-γ (heavy-tailed advantage standardization).
- If `eval pnl < −$100M` (no improvement) → β was not dominant in isolation; combined fix needed (B-11-α atom-span freeze + B-11-γ advantage standardization).
- If `l_pi` no longer diverges over 20k steps → mechanism confirmed regardless of pnl direction.
### G6 — Non-goals
- **No τ adaptation change.** The existing SAC-style τ auto-tune in `rl_q_pi_distill_grad.cu:167-182` stays. We don't want to fight two controllers.
- **No λ_base modification.** The existing KL-target controller for λ_base also stays.
- **No atom-span change.** Per the reverted Fix F lesson, atom-span adaptation needs its own evidence (B-11-α territory).
- **No PPO change.** The heavy-tailed advantage signal that B-10 also surfaced is real but separate (B-11-γ territory).
- **No SAC-α change.** The entropy floor mechanism is the design backstop.
## 3. Design
### 3.1 ISV slots (3 new)
```rust
// crates/ml-alpha/src/rl/isv_slots.rs
// Contiguous after B-10's slot 742.
// B-11-β (2026-06-01): Q-distill informativeness gate. Attenuates the
// effective distill lambda when softmax(Q/τ) target entropy is high
// (target near-uniform → distill = pure entropy regularization, no
// informational signal). Verified cluster cause @ alpha-rl-88f5c:
// target_entropy = 99.98% of ln(N_ACTIONS) with Q_range=13.79, τ=20.18.
// See spec docs/superpowers/specs/2026-06-01-b11-beta-q-distill-informativeness-gate.md.
//
// Slot 743 (ENABLED) is an ISV boolean toggle following the precedent
// established by B-7's `RL_REWARD_CLAMP_ENABLED_INDEX = 724` — A/B
// cluster regression for behavior-change specs. `feedback_no_feature_flags.md`
// applies to code-level booleans; ISV-driven kernel toggles for
// behavior-validation are the canonical foxhunt pattern.
pub const RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX: usize = 743;
pub const RL_Q_DISTILL_INFO_GATE_SENSITIVITY_INDEX: usize = 744;
// Slot 745 is the published λ_effective for diag observability — separate
// from the controller's λ_base (slot 486, owned by the existing
// KL-target Schulman controller, which B-11-β does NOT modify).
pub const RL_Q_DISTILL_LAMBDA_EFFECTIVE_INDEX: usize = 745;
/// Last RL-allocated slot index (exclusive).
/// ... (existing chain) ...
/// Post-B-10 policy-quality cascade diagnostic: 743.
/// Post-B-11-β Q-distill informativeness gate: 746.
pub const RL_SLOTS_END: usize = 746;
```
Bootstrap entries (3 new):
```rust
// B-11-β defaults:
// ENABLED = 1.0 (ON). Set to 0.0 via re-seed for A/B cluster regression
// vs alpha-rl-88f5c (same-SHA disabled run as control).
// SENSITIVITY (the exponent p) = 2.0. Tunable via re-seed if cluster
// shows under/over-attenuation. Gate factor at p=2:
// target_entropy 0% of ln(N) → gate 1.000 (full distill)
// target_entropy 50% of ln(N) → gate 0.250
// target_entropy 80% of ln(N) → gate 0.040
// target_entropy 95% of ln(N) → gate 0.0025
// target_entropy 99.9% of ln(N) → gate 1e-6 (effectively off)
// LAMBDA_EFFECTIVE = 0.0 sentinel; overwritten by the kernel every step.
(RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX, 1.0_f32),
(RL_Q_DISTILL_INFO_GATE_SENSITIVITY_INDEX, 2.0_f32),
(RL_Q_DISTILL_LAMBDA_EFFECTIVE_INDEX, 0.0_f32),
```
Bootstrap array `[(usize, f32); 230]` at `integrated.rs:3394` advances 230 → 233 (B-10 baseline 230 + 3 new). Same hard-error-on-mismatch protocol as B-8/B-9/B-10.
### 3.2 Kernel change
`crates/ml-alpha/cuda/rl_q_pi_distill_grad.cu`. Three structural changes; the existing `s_pi_target` computation (currently inside the `if (a == 0)` thread-0 block, lines 80-91) is leveraged but NOT modified.
**Design choice**: per-block λ_eff (not block-0-broadcast). Every block computes its own target entropy from its own `s_pi_target[]` array, gates its own `λ_eff`, and uses it in the per-thread `grad_distill` term. Per-batch gating is more honest to the per-step semantics; the reduction overhead is one N_ACTIONS=11-element sequential sum by thread-0 per block. Negligible.
**Code changes** (in order they appear in the kernel):
1. **New shared variable `s_lambda_eff`** declared at top of kernel alongside existing `s_pi_target` / `s_pi_new`:
```c
__shared__ float s_lambda_eff;
```
2. **Per-block λ_eff compute** added INSIDE the existing thread-0 block that computes `s_pi_target` (after line 91, before the `__syncthreads()` that exits the softmax block). Reads `lambda` from the function-scope const at line 54 (every thread loaded it, so it's available to thread 0):
```c
if (a == 0) {
// ... existing s_pi_target softmax code at lines 71-82 ...
// B-11-β: compute per-block target entropy from s_pi_target[], gate
// lambda, store to __shared__ for broadcast to all threads.
float h_target_local = 0.0f;
#pragma unroll
for (int i = 0; i < N_ACTIONS; ++i) {
const float pt = s_pi_target[i];
if (pt > 1e-12f) h_target_local -= pt * logf(pt);
}
const float gate_enabled = isv[RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX];
const float gate_p = isv[RL_Q_DISTILL_INFO_GATE_SENSITIVITY_INDEX];
const float ln_nactions = logf((float)N_ACTIONS);
const float gate_arg = fmaxf(0.0f, 1.0f - h_target_local / ln_nactions);
const float gate_factor = (gate_enabled > 0.5f) ? powf(gate_arg, gate_p) : 1.0f;
s_lambda_eff = lambda * gate_factor;
}
__syncthreads(); // ALL threads now see s_lambda_eff
```
3. **`grad_distill` line modified** at the per-thread gradient compute (current line 117):
```c
// BEFORE:
const float grad_distill = lambda * (pi_a - s_pi_target[a]);
// AFTER:
const float grad_distill = s_lambda_eff * (pi_a - s_pi_target[a]);
```
4. **Block-0 diag emit** for `RL_Q_DISTILL_LAMBDA_EFFECTIVE_INDEX = 745` — added inside the existing `if (b == 0 && a == 0)` diagnostic block at line 128, alongside the B-10 G2 emits already there. Only block 0 writes to slot 745 (no race):
```c
if (b == 0 && a == 0) {
// ... existing B-10 G2 / KL / SAC-α emits ...
isv[RL_Q_DISTILL_LAMBDA_EFFECTIVE_INDEX] = s_lambda_eff;
}
```
**What stays unchanged** (verified read-only against current kernel):
- `lambda` const at line 54 — read by all threads, unchanged
- `s_pi_target` softmax at lines 71-82 — unchanged
- `s_pi_new` + entropy at lines 86-110 — unchanged
- `grad_entropy = -alpha * pi_a * (log_pi_a + 1.0f + s_entropy)` at line 122 — **unchanged** (SAC-α path is the entropy floor per `pearl_pi_actor_collapses_without_entropy_floor`)
- KL EMA / qpa cosine emit at lines 137-151 — unchanged
- SAC-α auto-tune at lines 167-182 — unchanged
- B-10 G2 emit at lines 130-141 (target entropy) — unchanged
**Disabled-mode invariance**: when `gate_enabled = 0`, `gate_factor = 1.0` exactly, so `s_lambda_eff = lambda * 1.0 = lambda` and the kernel is bitwise identical to pre-B-11-β. Essential for the V4 A/B cluster regression.
### 3.3 Diag emission
`policy_diagnostic` block in `integrated.rs::build_diag_value`:
```rust
"policy_diagnostic": {
// ... existing B-10 leaves ...
// B-11-β: effective λ after informativeness gating. Compare against
// isv_config.q_distill_lambda (the controller's base λ) to see the
// gate's per-step action.
"q_distill_lambda_effective": isv[RL_Q_DISTILL_LAMBDA_EFFECTIVE_INDEX],
},
```
`EXPECTED_LEAVES`: 671 → **672** (1 new leaf).
### 3.5 Invariant test
`crates/ml-alpha/tests/q_distill_info_gate_invariants.rs`:
```rust
//! B-11-β invariants for Q-distill informativeness gating.
//!
//! 1. Gate identity: λ_effective = λ_base × max(0, 1 − target_entropy/ln(N_ACTIONS))^p
//! within float ε across all diag rows (when gate enabled).
//! 2. Gate bounds: 0 ≤ λ_effective ≤ λ_base for all rows.
//! 3. Saturation: when target_entropy ≥ 0.999 × ln(N_ACTIONS), λ_effective < λ_base × 1e-5.
//! 4. Disabled behavior: if RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX = 0,
//! then λ_effective ≡ λ_base bitwise.
```
Local 200+50 b=16 fold-1 smoke with B-7 clamp default OFF.
## 4. Validation
### V1 — Local smoke
Build release; verify all 4 invariants. Inspect mid-run diag: `q_distill_lambda_effective` should be ~99% smaller than `q_distill_lambda` (the controller's λ_base) when target_entropy is near-max.
### V2 — Cluster smoke (2k+500 b=1024 fold-1 H100)
Compare against `alpha-rl-88f5c` baseline (B-10 instrumentation, no gate):
- Train: l_pi divergence direction (should be slower or absent if gate engages)
- q_pi_agree_ema trajectory (should stay near 1.0 longer)
- π entropy trajectory (should still satisfy SAC-α floor)
- Eval pnl direction
### V3 — Cluster full (20k+5k b=1024 fold-1 H100)
Apples-to-apples vs `alpha-rl-8gtk2` (the B-7+B-8+B-9 baseline that produced −$218M).
Falsification per G5.
### V4 — Determinism
`rewards`/`dones`/`actions` differ vs pre-B-11-β run (gate changes the policy gradient → action distribution → trade outcomes). NOT byte-identical — this is a behavior change, not pure observability.
## 5. Implementation phases
| Phase | Step | LOC |
|------|------|-----|
| 1 | Add 3 ISV slots; bump RL_SLOTS_END | 8 |
| 2 | Add 3 bootstrap entries; bump fixed-size array 230 → 233 in `integrated.rs:3394` | 6 |
| 3 | Modify `rl_q_pi_distill_grad.cu`: per-block target entropy + gate compute + λ_eff in grad line; block-0 ISV emit for λ_eff | ~40 |
| 4 | Add `q_distill_lambda_effective` leaf in `build_diag_value` | 4 |
| 5 | Bump `EXPECTED_LEAVES` to 672 | 1 |
| 6 | Write `q_distill_info_gate_invariants.rs` test | ~140 |
| 7 | Build release; run new test + 200+50 smoke + schema parity | — |
| 8 | Single commit on `ml-alpha-regime-observer` | — |
Total source delta: ~200 LOC across 4 files. Substantially smaller than B-10 because it builds on the same kernel's existing diagnostic block.
## 6. Open decisions
| # | Question | Recommendation |
|---|----------|---------------|
| 1 | Should the gate also apply to grad_entropy (the SAC-α path)? | **No.** That term is the entropy floor protecting against `pearl_pi_actor_collapses_without_entropy_floor`. Touching it conflates concerns. The gate is purely on `grad_distill`; SAC-α stays the dedicated entropy regulator. Note: the `RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX` boolean toggle follows the **B-7 precedent** (`RL_REWARD_CLAMP_ENABLED_INDEX = 724`) — `feedback_no_feature_flags.md` applies to code-level `enable_*` booleans, not to ISV-driven kernel toggles used for cluster A/B regression. Both B-7 and B-11-β use the same pattern: default 1.0 (ON), set to 0.0 via re-seed for same-SHA control runs. |
| 2 | Should `p=2` default be more aggressive (p=4)? | **Stay at 2.** Smooth transition; if cluster shows under-attenuation, re-seed via ISV — that's why p is a slot. |
| 3 | Should the gate kick in only above some entropy threshold instead of smooth? | **Smooth.** Hard threshold creates non-differentiable cliffs in λ_eff(target_entropy) and adds a hyperparameter. The polynomial gate is differentiable and has p as its sole knob. |
| 4 | Should we ALSO modify τ (e.g., cap τ ≤ Q_range / 3)? | **No** in B-11-β. Track via diag; if data shows the gate alone is insufficient, write B-11-β.1 separately. |
| 5 | Is per-block λ_eff computation worth the extra reduction work over block-0-broadcasts-to-all? | **Yes.** Per-batch entropy varies in principle; honest per-block is cleaner. Reduction is one N_ACTIONS=11 loop, negligible. |
## 7. Done means
- 3 ISV slots allocated (743-745); `RL_SLOTS_END = 746`
- `rl_q_pi_distill_grad.cu` computes per-block λ_eff and emits it via block-0
- 1 new diag leaf `policy_diagnostic.q_distill_lambda_effective`
- `EXPECTED_LEAVES = 672`
- 4 invariant tests pass on RTX 3050 Ti
- 200+50 b=16 fold-1 local smoke shows λ_eff ≈ 0 when target_entropy is near max
- Single commit on `ml-alpha-regime-observer`
## 8. Risks
- **Risk A (medium):** If the gate over-attenuates λ_eff (sets it to ~0 at target_entropy 80% of max, which the smoke showed), distill effectively turns off. The SAC-α path is the safety net but may not fully replace distill's role of injecting Q's preferences into π. Mitigation: V3 cluster validation checks `q_pi_agree_ema` trajectory — if it diverges (Q and π drift apart), the gate is too aggressive; re-tune `p` via ISV.
- **Risk B (low):** Per-block target entropy computation adds 1 reduction per block (N_ACTIONS=11 sequential sum). At b=1024 that's 1024 blocks doing 11 ops each = trivial.
- **Risk C (low):** First behavior change since B-7's clamp toggle. Per V4 the cluster smoke will diverge from prior runs by RNG path. Mitigation: rely on aggregate statistics (eval pnl, mean l_pi trajectory) not exact reproducibility.
- **Risk D (low / surfaced by spec):** If `λ_base` (the existing KL-target controller's output at slot 486) drifts higher to compensate for the gate's attenuation, we may get oscillation.
- **Mechanism**: the KL-target controller reads `RL_Q_DISTILL_KL_EMA_INDEX = 488`. After B-11-β the gated grad_distill changes π → so the next step's KL(target ‖ π_new) reflects the gated dynamics. The controller sees the gated KL as input, not the raw KL. If `λ_base` ramps to recover KL toward its target, the gate's attenuation is partially offset.
- **Observable threshold for B-11-β.1**: if `isv_config.q_distill_lambda` (slot 486, λ_base) exceeds **2× its pre-B-11-β baseline EMA** sustained for 5,000 steps in the cluster smoke, the controller is in compensatory-ramp territory. Write B-11-β.1 to gate the controller's input on the gated KL (i.e., feed the controller with the post-gate effective KL, not the policy-observed KL).
- **Cluster diag**: V3 inspection compares `isv_config.q_distill_lambda` (the controller base λ) vs `policy_diagnostic.q_distill_lambda_effective` (post-gate). The ratio of base/effective is the gate factor, and base's drift over time vs `alpha-rl-88f5c` (the pre-B-11-β baseline) is the diagnostic.
## 9. What B-11-β does NOT promise
- Does not fix the eval-pnl problem in isolation if γ (heavy-tailed advantage normalization) is also load-bearing. The cluster outcome will tell us.
- Does not change Q's learning trajectory directly — Q continues to be trained by the Bellman target as before; only its DOWNSTREAM influence on π is modulated.
- Does not introduce a new controller. Uses the existing KL-target controller's λ output multiplicatively.
## 10. After B-11-β: candidate B-11-γ and B-11-α paths (NOT in this spec)
Per the B-10 spec §10 framework, the falsification criteria (§2.G5) tell us which path runs after V3 cluster:
- **B-11-γ** (heavy-tailed advantage standardization): if eval pnl improves but only partially (-$50M to -$100M range), the 6.5σ A_norm outliers and 17-22% ratio clip rate from B-10 data motivate winsorization or trimmed-mean standardization in `rl_advantage_normalize.cu`.
- **B-11-α** (atom-span informativeness): if eval pnl doesn't improve AND `q_dist_entropy_mean` crosses 0.95 × ln(Q_N_ATOMS) in long runs, freeze V_MAX_eff to surfer zone (≤5) per `pearl_c51_v_max_freeze_required_for_surfer.md`.