First behavior change since B-7 (preceding B-8/B-9/B-10 were
observability-only). Attenuates the distill gradient when softmax(Q/τ)
is near-uniform — when target_entropy → ln(N_ACTIONS), the distill term
is pure max-entropy regularization with no informational signal, so it
fights PPO instead of carrying Q's preferences into π.
Verified cluster cause @ alpha-rl-88f5c (B-10 smoke, c1dc84a34):
target_entropy = 2.3976 / ln(11) = 2.398 max → 99.98% of max with Q_range
13.79 and τ=20.18. Distill at λ=0.224 was applying ~uniform-target KL
regularization across 20k steps → eval pnl -$218M at full run
(alpha-rl-8gtk2).
Gate (per-block, smooth):
λ_eff = λ_base × max(0, 1 − h_target / ln(N_ACTIONS))^p
where λ_base is the existing KL-target Schulman controller's output
(slot 486, untouched) and p defaults to 2.0 (slot 744).
Bitwise disabled-mode: when RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX = 0.0
the ternary forces gate_factor = 1.0 verbatim and s_lambda_eff = lambda
exactly. Same ISV-toggle pattern as B-7's reward-clamp control (slot 724).
Source delta:
- 3 new ISV slots: RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX = 743,
RL_Q_DISTILL_INFO_GATE_SENSITIVITY_INDEX = 744,
RL_Q_DISTILL_LAMBDA_EFFECTIVE_INDEX = 745; RL_SLOTS_END 743 → 746.
- Bootstrap array 230 → 233 (defaults: ENABLED=1.0, SENSITIVITY=2.0,
LAMBDA_EFFECTIVE=0.0 sentinel; kernel block-0 overwrites every step).
- rl_q_pi_distill_grad.cu: shared s_lambda_eff, per-block target-entropy
reduction in the existing thread-0 s_pi_target block, gate compute,
__syncthreads broadcast; grad_distill uses s_lambda_eff. Block-0 emits
slot 745 alongside existing B-10 G2 emits.
- Diag: policy_diagnostic.q_distill_lambda_effective (672 leaves total).
- New invariant smoke: tests/q_distill_info_gate_invariants.rs (4 gates).
Local 200+50 b=16 fold-1 smoke validates all 4 invariants across 249 rows:
peak h_target/ln(N) = 1.000 (β cause confirmed at smoke scale too); gate
factor range [0, 5.99e-6]; 132 saturated rows (h_frac ≥ 0.999). l_q_b
healthy throughout.
NOT byte-identical to pre-B-11-β runs by design: gated grad_distill
changes π → action distribution → trade outcomes. V4 (spec §4) calls
this out explicitly. Cluster comparison is on aggregate signals (eval
pnl, l_pi trajectory), not exact reproducibility.
Falsification criteria (spec §2.G5) gate the next iteration:
- eval pnl > -$50M → β was dominant, B-11-β is the fix
- -$50M to -$100M → β contributed, write B-11-γ (advantage standardization)
- < -$100M → β not dominant alone; B-11-α + B-11-γ combined
Risk D explicit threshold (spec §8): if λ_base (slot 486) exceeds 2× its
pre-B-11-β baseline EMA sustained 5k steps, the KL-target controller is
in compensatory-ramp territory and B-11-β.1 gates the controller's input
on the gated KL.
Spec: docs/superpowers/specs/2026-06-01-b11-beta-q-distill-informativeness-gate.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
b93971726d