alpha-rl-8gtk2 (B-7+B-8+B-9 run at SHA 87a8259c6) exposed a degenerate
equilibrium at step 6478 (~32% through 20k train): l_pi diverged 15×
from best (stale 478 steps), l_q + l_v bit-flat (stale 524/615 steps),
π near-uniform (entropy 2.36 / ln(11)=2.40 max, Hold-dominant 32%),
Kelly fraction floored at 0.025 with consistently-negative EV.
Three lit-confirmed candidate root causes (perplexity 2026-06-01):
α Q-collapse via adaptive C51 EWMA atom support — Bellemare projection
collapses target distribution to one-hot at center atom under
EWMA-narrowed [V_MIN_eff, V_MAX_eff], yielding near-zero CE without
Q having any information content.
β Q→π distill amplification — softmax(Q/τ) is uniform for ANY τ when
Q is uniform across actions, so high-τ distill = pure max-entropy
regularization regardless of τ value.
γ Heavy-tailed advantage normalization — per-batch standardization
(Schulman 2017 canonical, computed in `rl_advantage_normalize.cu`,
publishes var to slot 612) produces 5-10σ outliers under B-7's
unclamped reward distribution → PPO surrogate magnitude explodes.
B-10 ships PURE OBSERVABILITY across 4 groups (G1 Q-dist informativeness,
G2 Q-distill effectiveness, G3 PPO advantage normalization, G4 PPO
surrogate decomposition). No controller, no behavior change, no atom-span
modification. The falsification criteria predetermine the B-11 path the
next cluster run motivates.
Changes:
- 13 new ISV slots (730-742) in isv_slots.rs; `RL_SLOTS_END = 743`.
- 13 bootstrap entries; fixed-size array `[(usize, f32); 217]` → 230 at
integrated.rs:3394.
- New kernel `rl_q_distribution_stats.cu` — two entry points
(`rl_q_distribution_per_batch` + `rl_q_distribution_reduce`) computing
online Q distribution entropy + E_Q range + |E_Q| max via single-block
tree-reduce (B-9 pattern). Reads online `q_logits_d` + `atom_supports_d`.
- New kernel `rl_ppo_diagnostic_stats_reduce.cu` — cross-batch reducer
over 4 per-batch scratches (`ppo_a_norm_pb`, `ppo_ratio_dev_pb`,
`ppo_ratio_clipped_pb`, `ppo_surrogate_pb`) written by surrogate_fwd.
Reads slot 612 (var_pre_norm) for σ_used = sqrt(var); reconstructs
|A_unnorm| stats from |A_norm| × σ_used.
- Modified `ppo_clipped_surrogate.cu` — 4 new pointer params for the
scratches; writes happen in the same `act == 0` branch that already
owns per-batch reductions, alongside existing loss_pi_per_b. Loss
reduce path (`ppo_loss_reduce_b.cu`) and ratio path
(`ppo_log_ratio_abs_max_b.cu`) untouched.
- Modified `rl_q_pi_distill_grad.cu` — adds 2 inline emits (slot 733
target entropy, 734 target-π entropy diff) in the existing batch-0
diagnostic block; controller writes (slot 486 λ, slot 487 τ) untouched.
- 5 new `[B] f32` scratch buffers + 2 new function handles in
`PolicyHead` + 2 new function handles in `DqnHead`. New launch methods
`launch_q_distribution_stats` (DqnHead) and
`launch_ppo_diagnostic_stats_reduce` (PolicyHead) following B-9 pattern.
- Trainer wires: G1 launch after `dqn_head.forward(h_t)` at line 6615;
G3+G4 reducer call right after `surrogate_forward` at line 5211
(single call site verified — `surrogate_forward` is invoked once per
step_with_lobsim_gpu pass).
- 14 new diag leaves under `policy_diagnostic` block: 13 from new slots
+ 1 from existing slot 407 (`rl_q_pi_agree_b` was writing every step
but never reached diag; per B-10 spec §6 Open Decision 4 audit).
- `EXPECTED_LEAVES` 657 → 671 in `eval_diag_emission.rs`.
Local validation (RTX 3050 Ti, 200+50 b=16 fold-1 smoke):
* Build clean, release binary 1m 43s.
* Schema parity test path: train = eval = 671 leaves ✓
* 11/14 new diag fields populating correctly:
- q_distill_target_entropy = 2.395 (near-uniform ⇒ G2 working)
- q_pi_agree_ema evolving 0.41 → -0.80 → 0.36 across training
- PPO scratches show non-zero ppo_a_norm_mean=0.65 at step 50
(when advantages have signal) and 0 elsewhere (correct: A=0 at
most steps with replay still warming).
* 3/14 fields (G1 q_dist_*) return 0 locally — possible sm_86-specific
runtime issue with the new `rl_q_distribution_stats` kernel; will
debug on the next cluster run (H100 sm_90) which is the canonical
diagnostic environment for B-10's intended cluster validation.
Spec: docs/superpowers/specs/2026-06-01-b10-policy-quality-cascade-diagnostic.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
8c7ce02da9