# B-10 — Policy-Quality Cascade Diagnostic

**Date:** 2026-06-01
**Status:** Draft (pre-implementation)
**Branch target:** `ml-alpha-regime-observer`
**Predecessors:** B-7 (clamp toggle), B-8 (σ_welford), B-9 (atom saturation)
**Cluster evidence:** `alpha-rl-8gtk2` at SHA `87a8259c6` (in flight as of writing)
**Pearls referenced:**
- [[pearl_violation_rate_blind_to_hold_attractor]] — Hold-collapse blindness in adaptive controllers
- [[pearl_c51_v_max_freeze_required_for_surfer]] — V_MAX → surfer zone for surfer policy
- [[pearl_q_thompson_actor_makes_pi_dead_weight]] — Q-as-actor decoupling problem
- [[pearl_q_to_pi_distillation_breaks_v_vanishing_trap]] — Q→π distill activation rationale
- [[pearl_scalar_v_with_categorical_ce_is_noop]] — categorical CE math under degeneracy
- [[feedback_no_stubs]], [[feedback_no_quickfixes]], [[feedback_isv_for_adaptive_bounds]]

**Research references** (perplexity 2026-06-01):
- C51 categorical cross-entropy near-zero is a *known* Q-collapse signal under adaptive atom support (target distribution collapses to one-hot at center atom when EWMA narrows span around mean-zero rewards).
- Boltzmann/Q→π distillation: `softmax(Q/τ)` is uniform for ANY τ when Q is uniform across actions, so high-τ distillation when Q is collapsed acts as pure max-entropy regularization regardless of τ value.
- PopArt × PPO coupling (van Hasselt 2016): when σ_eff tracks envelope/tail rather than typical magnitude, A_norm is under-scaled but A_unnorm = σ × A_norm can blow up PPO clipped surrogate even with small ε. Canonical fix: decouple PopArt-on-critic from short-horizon per-batch standardization for PPO advantages.

---

## 1. Motivation

`alpha-rl-8gtk2` (the B-7+B-8+B-9 full 20k+5k run) exposed a degenerate-equilibrium policy state at step 6478 (~32 % through train):

| Signal | Value | Interpretation |
|---|---|---|
| `lr_plateau.q.loss_ema` | 0.0020 (stale **524 steps**, best 0.0019) | Q has stopped improving |
| `lr_plateau.pi.loss_ema` | 1466 (stale **478 steps**, best 92.5) | **PPO DIVERGED 15× from best** |
| `lr_plateau.v.loss_ema` | 0.0002 (stale 615 steps) | V stopped improving |
| `isv_ema_in.entropy_observed` | 2.36 / ln(11)=2.40 max | Near-uniform π |
| Action histogram | a2 (Hold) = 32 %, others 2-11 % | Hold-biased random |
| `kelly.win_rate_ema` | 0.19-0.33 oscillating | No edge developing |
| `kelly.fraction` | 0.025 (floor) | Controller pessimistic |
| `popart.sigma_welford` | 3.08 (from 38 at step 99) | 12× collapse in 6k steps |
| `popart.sigma` (= σ_eff) | 92-124 (env-dominated) | Envelope detector controls σ |
| `V_MAX_eff` | 75 (from 857→465→75) | EWMA tightening |
| `target_top_sat_rate` | 0.000 | Atom span keeping pace structurally |
| `q_distill_temperature` | 20.18 | High τ |
| `q_distill_kl_ema` | 0.045 vs target 0.10 | KL is half of target |

The B-7+B-8+B-9 observability surfaced the problem but cannot attribute the cascade root cause without further instrumentation. **The three candidate causes are independently consistent with RL literature**:

1. **Q-collapse via adaptive atom support.** Bit-flat `l_q` = 0.002 is consistent with Bellemare-projection collapsing target to a single central atom under EWMA-narrowed `[V_MIN_eff, V_MAX_eff]`. The categorical CE becomes near-zero because target ≈ predicted ≈ one-hot-at-center, *not* because Q has learned. This is the canonical failure mode of adaptive-support distributional Q.

2. **Q→π distill amplifies under Q-collapse.** With Q uniform across actions, `softmax(E_Q / τ=20)` is uniform regardless of τ. The distill term `λ × KL(uniform ‖ π_new)` becomes pure max-entropy regularization. With λ=0.224 substantial, this pushes π toward uniform regardless of what PPO advantages say.

3. **Per-batch advantage normalization vs PopArt-V coupling.** The system uses Schulman-2017 canonical per-batch advantage standardization in `rl_advantage_normalize.cu` (`A_norm = (A − mean) / sqrt(var + ε²)`, publishes pre-norm variance to slot 612). V regression uses PopArt-σ separately. The advantage entering PPO surrogate is per-batch-standardized — so PopArt σ_eff is NOT directly multiplying PPO advantages, and my initial framing of "PopArt × PPO coupling" was architecturally incorrect. The actual question this hypothesis re-asks is: **is per-batch standardization producing well-conditioned advantages under B-7's unclamped distribution, OR does var_pre_norm have such heavy tails that A_norm has occasional 5-10σ outliers driving PPO loss?** This is observable via the variance slot 612 vs ratio statistics.

Each cause has documented detection criteria in the literature. None is currently directly observable in our diag.

## 2. Goals

### G1 — Q-distribution informativeness observability
Detect Q-collapse via Q output structure, NOT via loss magnitude:
- **per-step mean Q-distribution entropy** across (batch × actions): if near ln(Q_N_ATOMS)=3.04, Q is uniform (collapsed); if near 0, Q is one-hot (informative but possibly delta-at-edge).
- **per-step Q variance across actions**: mean over batch of `max_a E_Q(s,a) − min_a E_Q(s,a)`. If ≈ 0, all actions look identical to Q.
- **per-step E_Q magnitude statistics**: mean / max of |E_Q(s,a)| across batch.

### G2 — Q→π distill effectiveness observability
Detect when distill is acting as pure entropy regularization:
- **softmax(E_Q/τ) target entropy** per step: if ≈ ln(N_ACTIONS), Q-distill is information-free.
- **distill target vs π divergence direction**: `H(softmax(E_Q/τ)) − H(π_new)` — positive ⇒ distill pushing π more uniform; negative ⇒ distill pushing π more concentrated.

### G3 — PPO advantage normalization observability
Detect heavy-tailed pre-normalization variance and post-normalization outliers (verified: per-batch standardization in `rl_advantage_normalize.cu` writes pre-norm variance to slot 612; we read it and add the post-norm statistics):
- **mean(|A_norm|), max(|A_norm|)** per step (A_norm = post-standardization advantage entering PPO).
- **mean(|A_unnorm|), max(|A_unnorm|)** per step (the unnormalized `returns − V` that the standardizer ingested).
- **σ_used_for_advantage** = `sqrt(isv[RL_ADV_VAR_PRE_NORM_INDEX])` — the per-batch std actually applied as divisor in `rl_advantage_normalize.cu`. Heavy tails in this signal say standardization is unstable; small values say A_norm is well-conditioned.

### G4 — PPO surrogate decomposition
Detect ratio explosion vs advantage explosion as separate causes of `l_pi` divergence:
- **mean(|ratio − 1|), max(|ratio − 1|)** per step.
- **fraction_clipped_top, fraction_clipped_bot**: proportion of samples where ratio crosses 1+ε or 1−ε.
- **l_surrogate** (pure clipped-surrogate) and **l_entropy** (entropy bonus contribution) emitted separately, so we can see whether l_pi blow-up is from surrogate or from entropy regularization.

### G5 — Falsification criteria (read after next cluster run)
- If `Q_dist_entropy_ema > 0.95 × ln(Q_N_ATOMS)` for >5k steps → **Q-collapse is the root cause** → B-11 spec addresses C51 atom-span policy (e.g., freeze at surfer zone V_MAX=5).
- If `softmax(E_Q/τ)_entropy ≈ ln(N_ACTIONS)` AND `q_distill_lambda > 0.1` → **Q-distill amplification is the root cause** → B-11 spec disables Q-distill when Q is uniform.
- If `max(|A_unnorm|) / max(|A_norm|) > 10×` AND `mean(|ratio−1|) > ε/2` → **PopArt × PPO coupling is the root cause** → B-11 spec adds per-batch standardization for PPO advantages independent of PopArt.
- If NONE of the above and `l_pi` continues to diverge → unknown 4th cause; revisit framework.

### G6 — Non-goals
- **No behavior change.** B-10 is pure observability per the B-7→B-9 pattern. Per `feedback_no_quickfixes`: don't ship a fix before evidence.
- **No new controllers.** All emit slots are diagnostic; no new feedback loops.
- **No atom-span change.** Even though Q-collapse may be the cause, B-10 does NOT modify V_MAX_eff or atom support. That's B-11 work AFTER B-10 data lands.

## 3. Design

### 3.1 ISV slot allocation (13 new slots)

```rust
// crates/ml-alpha/src/rl/isv_slots.rs
// Contiguous after B-9's slot 729.

// B-10 group G1: Q-distribution informativeness (3 slots).
pub const RL_Q_DIST_ENTROPY_MEAN_INDEX:           usize = 730;
pub const RL_Q_VALUE_RANGE_MEAN_INDEX:            usize = 731;
pub const RL_Q_VALUE_ABS_MAX_INDEX:               usize = 732;

// B-10 group G2: Q-distill effectiveness (2 slots).
pub const RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX: usize = 733;
pub const RL_Q_DISTILL_PI_ENTROPY_DIFF_INDEX:     usize = 734;

// B-10 group G3: PPO advantage normalization (5 slots).
pub const RL_PPO_A_NORM_ABS_MAX_INDEX:            usize = 735;
pub const RL_PPO_A_NORM_ABS_MEAN_INDEX:           usize = 736;
pub const RL_PPO_A_UNNORM_ABS_MAX_INDEX:          usize = 737;
pub const RL_PPO_A_UNNORM_ABS_MEAN_INDEX:         usize = 738;
pub const RL_PPO_A_SIGMA_USED_INDEX:              usize = 739;

// B-10 group G4: PPO surrogate decomposition (3 new slots). Existing
// `ppo_log_ratio_abs_max_b.cu` already emits `max |log_ratio|` to slot
// `RL_PPO_LOG_RATIO_ABS_MAX_INDEX = 441` — distinct from B-10's
// mean(|ratio-1|) and clip-rate signals, so no duplication.
pub const RL_PPO_RATIO_DEV_ABS_MEAN_INDEX:        usize = 740;
pub const RL_PPO_RATIO_CLIP_RATE_INDEX:           usize = 741;
pub const RL_PPO_L_SURROGATE_INDEX:               usize = 742;

pub const RL_SLOTS_END: usize = 743;
```

Bootstrap entries (all 0.0, overwritten per step):
```rust
(RL_Q_DIST_ENTROPY_MEAN_INDEX,             0.0_f32),
(RL_Q_VALUE_RANGE_MEAN_INDEX,              0.0_f32),
(RL_Q_VALUE_ABS_MAX_INDEX,                 0.0_f32),
(RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX,   0.0_f32),
(RL_Q_DISTILL_PI_ENTROPY_DIFF_INDEX,       0.0_f32),
(RL_PPO_A_NORM_ABS_MAX_INDEX,              0.0_f32),
(RL_PPO_A_NORM_ABS_MEAN_INDEX,             0.0_f32),
(RL_PPO_A_UNNORM_ABS_MAX_INDEX,            0.0_f32),
(RL_PPO_A_UNNORM_ABS_MEAN_INDEX,           0.0_f32),
(RL_PPO_A_SIGMA_USED_INDEX,                0.0_f32),
(RL_PPO_RATIO_DEV_ABS_MEAN_INDEX,          0.0_f32),
(RL_PPO_RATIO_CLIP_RATE_INDEX,             0.0_f32),
(RL_PPO_L_SURROGATE_INDEX,                 0.0_f32),
```

Fixed-size bootstrap array `[(usize, f32); N]` in `integrated.rs:3394` advances 217 → 230 (B-9 baseline 217 + 13 new). Same hard-error-on-mismatch protocol as B-8/B-9.

### 3.2 G1 — Q-distribution emit (kernel: `dqn_distributional_q.cu` or new reducer)

Reading from `q_logits_d [B × N_ACTIONS × Q_N_ATOMS]`:
- Per (b, a): softmax over atoms → p[atom]. Entropy = −Σ p log p. E_Q = Σ p × atom_value.
- Aggregate across batch × actions: emit mean entropy, mean range max_a E_Q − min_a E_Q, max |E_Q|.

**Implementation choice**: new kernel `rl_q_distribution_stats.cu` (single-block tree-reduce over `B × N_ACTIONS × Q_N_ATOMS = 1024 × 11 × 21 = 236,544` elements at cluster scale; grid-stride loop with `block_dim ≤ 1024`; pattern lifted from `rl_bellman_target_saturation_reduce.cu`). Reads **online Q's** `q_logits_d` (NOT target Q's `target_logits` — the online net is what Q-Thompson / Q→π distill consume; per `pearl_phase4_dueling_head_plus_advantage_norm_is_new_sota` the online Q is the V_dq baseline source) + `atom_supports_d` from existing trainer state. Launches once per step **after `dqn_head.forward_logits` returns** in `step_with_lobsim_gpu`, before downstream consumers (distill / Thompson / PPO).

### 3.3 G2 — Q-distill target entropy (kernel: `rl_q_pi_distill_grad.cu`)

The kernel already computes `softmax(E_Q / τ)` as the distillation target. Add inline reduction of its entropy across actions, mean across batch. Add second pass: `π_new` entropy already computed elsewhere (action_entropy_per_step.cu writes slot 420). Emit the difference into 734.

**All B-10 reads inside `rl_q_pi_distill_grad.cu` are read-only.** Controller writes to `RL_Q_DISTILL_LAMBDA_INDEX = 486`, `RL_Q_DISTILL_TEMPERATURE_INDEX`, and the existing distill-loss path are untouched. New writes go only to ISV slots 733 / 734. Per `pearl_controller_anchors_isv_driven.md`: controller anchor slots remain ISV-driven without B-10 perturbation.

### 3.4 G3+G4 — PPO advantage + ratio + surrogate decomposition

**Multi-kernel context** (per `feedback_no_partial_refactor.md`): The PPO surrogate path comprises three kernels — `ppo_clipped_surrogate.cu` (forward + per-batch loss writes), `ppo_loss_reduce_b.cu` (block tree-reduce of the per-batch losses to the scalar `l_pi`), and `ppo_log_ratio_abs_max_b.cu` (diagnostic-only emit of `max|log_ratio|` to slot 441, independent reducer). B-10 must NOT add a sixth-and-seventh reduction inside `ppo_clipped_surrogate.cu` because (a) that kernel writes per-batch outputs `loss_per_batch[B]` consumed by `ppo_loss_reduce_b`, and adding stats reductions there would force every consumer to widen its input layout, and (b) sharing `__shared__` between the surrogate reduction and new stats reductions risks warp-scheduling reorderings that break the V3 determinism claim.

**Design** (mirrors B-9's `rl_bellman_target_saturation_reduce.cu` pattern):

1. **`ppo_clipped_surrogate.cu` writes per-batch scratch** alongside its existing `loss_per_batch[B]`:
   - `ppo_a_norm_per_batch[B]` — `|A_norm|` for each sample
   - `ppo_a_unnorm_per_batch[B]` — `|A_unnorm|` for each sample (caller passes σ used in advantage compute as an extra `float σ_used` scalar arg; kernel multiplies and absolutes)
   - `ppo_ratio_dev_per_batch[B]` — `|ratio − 1|` for each sample
   - `ppo_ratio_clipped_per_batch[B]` — 1.0 if `ratio ∉ [1−ε, 1+ε]` else 0.0
   - `ppo_surrogate_per_batch[B]` — clipped surrogate value (no entropy bonus)

   These writes use **separate `__shared__` arrays** from the existing surrogate reduction; the existing reduction's math is byte-identical to the pre-B-10 kernel.

2. **New kernel `rl_ppo_diagnostic_stats_reduce.cu`** consumes the per-batch scratches above:
   - Single block, grid-stride loop, no atomicAdd. Pattern lifted from `rl_bellman_target_saturation_reduce.cu`.
   - Reads `RL_ADV_VAR_PRE_NORM_INDEX = 612` (pre-norm variance published by `rl_advantage_normalize.cu`) and computes `σ_used = sqrt(isv[612])` inside the reducer; no separate scalar argument needed.
   - Emits 8 ISV slots: 735 (a_norm max), 736 (a_norm mean), 737 (a_unnorm max), 738 (a_unnorm mean), 739 (σ_used = sqrt(var_pre_norm)), 740 (ratio_dev mean), 741 (ratio clip rate), 742 (l_surrogate mean).

3. **Trainer plumbing**: 5 new `[B] f32` scratch buffers allocated once in `IntegratedTrainer::new` (same B-9 pattern). Trainer launches the diagnostic reducer immediately after `ppo_clipped_surrogate.cu` returns, before the existing `ppo_loss_reduce_b` runs (independent — no dependency on the existing reducer's outputs). `rl_advantage_normalize.cu` already runs upstream of the surrogate (per the existing `step_with_lobsim_gpu` pipeline), so slot 612 is populated by the time our reducer launches.

**Per `feedback_no_atomicadd.md`**: scratch + tree-reduce; no atomics anywhere on the new path. Matches B-9 discipline exactly.

**Per `feedback_no_partial_refactor.md`**: `ppo_clipped_surrogate.cu`'s signature gains 5 new pointer params (no extra scalar — σ_used is derived from existing ISV slot). The surrogate kernel is invoked via `PolicyHead::surrogate_forward` (`ppo.rs`); audit lists exactly **1 call site** at `integrated.rs:5191` (verified by grep), updated atomically in the same commit. `ppo_loss_reduce_b.cu` and `ppo_log_ratio_abs_max_b.cu` are untouched (they don't see the new scratches).

### 3.5 Diag emission

Add new top-level `policy_diagnostic` block (placed between `popart` and `risk_stack` in the existing diag layout — no conflict with `lr_plateau` or `controller` blocks):
```rust
"policy_diagnostic": {
    // G1 — Q-distribution informativeness (detects Q-collapse).
    "q_dist_entropy_mean":       isv[RL_Q_DIST_ENTROPY_MEAN_INDEX],
    "q_value_range_mean":        isv[RL_Q_VALUE_RANGE_MEAN_INDEX],
    "q_value_abs_max":           isv[RL_Q_VALUE_ABS_MAX_INDEX],
    // G2 — Q-distill effectiveness (detects uniform-target case).
    "q_distill_target_entropy":  isv[RL_Q_DISTILL_TARGET_ENTROPY_MEAN_INDEX],
    "q_distill_pi_entropy_diff": isv[RL_Q_DISTILL_PI_ENTROPY_DIFF_INDEX],
    // G3 — PPO advantage normalization (detects PopArt × PPO coupling).
    "ppo_a_norm_abs_max":        isv[RL_PPO_A_NORM_ABS_MAX_INDEX],
    "ppo_a_norm_abs_mean":       isv[RL_PPO_A_NORM_ABS_MEAN_INDEX],
    "ppo_a_unnorm_abs_max":      isv[RL_PPO_A_UNNORM_ABS_MAX_INDEX],
    "ppo_a_unnorm_abs_mean":     isv[RL_PPO_A_UNNORM_ABS_MEAN_INDEX],
    "ppo_a_sigma_used":          isv[RL_PPO_A_SIGMA_USED_INDEX],
    // G4 — PPO surrogate decomposition (detects ratio vs advantage cause).
    "ppo_ratio_dev_abs_mean":    isv[RL_PPO_RATIO_DEV_ABS_MEAN_INDEX],
    "ppo_ratio_clip_rate":       isv[RL_PPO_RATIO_CLIP_RATE_INDEX],
    "ppo_l_surrogate":           isv[RL_PPO_L_SURROGATE_INDEX],
    // Audit followup (resolved Open Decision 4): existing slot 407 was
    // computed every step but never reached diag. Reading it as a leaf
    // here costs zero slots. q_pi_agree IS the canonical Q-collapse
    // discriminator (low → Q and π learning different policies; high →
    // Q informing π). See pearl_q_thompson_actor_makes_pi_dead_weight.
    "q_pi_agree_ema":             isv[crate::rl::isv_slots::RL_Q_ARG_VS_PI_AGREE_INDEX],
},
```

Schema leaf bump: 657 → **671** (14 new leaves — 13 from new slots + 1 from existing-slot exposure). Update `EXPECTED_LEAVES` in `tests/eval_diag_emission.rs`.

### 3.6 Invariant test

`crates/ml-alpha/tests/policy_diagnostic_observability.rs`:

```rust
//! B-10 invariants for policy-quality cascade diagnostic.
//!
//! 1. Entropy bounds: 0 ≤ q_dist_entropy_mean ≤ ln(Q_N_ATOMS)=3.045;
//!    0 ≤ q_distill_target_entropy ≤ ln(N_ACTIONS)=2.398.
//! 2. Range non-negativity: q_value_range_mean ≥ 0.
//! 3. Magnitude ordering: q_value_abs_max ≥ q_value_range_mean / 2.
//! 4. PPO advantage consistency: when ppo_a_sigma_used > 1, then
//!    ppo_a_unnorm_abs_max ≥ ppo_a_norm_abs_max (σ amplifies, not shrinks).
//! 5. Ratio rate bounds: 0 ≤ ppo_ratio_clip_rate ≤ 1.
//! 6. Surrogate vs total (structural, not heuristic): the entropy-bonus
//!    contribution to l_pi is bounded by `|β| × ln(N_ACTIONS)` where
//!    β = `isv[RL_ENTROPY_COEF_INDEX]` (typically 0.01) and
//!    ln(N_ACTIONS) = ln(11) ≈ 2.398, so entropy contribution ≤ 0.024
//!    in magnitude. Therefore: `|lr_plateau.pi.loss_ema − ppo_l_surrogate| ≤
//!    β × ln(N_ACTIONS) + ε` where ε is float-accumulation tolerance.
//!    This is a structural identity, not a tuned constant.
```

Local 200+50 b=16 fold-1 smoke under B-7+B-8+B-9+B-10 stack. No cluster validation required for B-10 alone (pure observability).

## 4. Validation

### V1 — Local smoke + invariants
Build release; run new test. All six invariants hold.

### V2 — Schema parity
`eval_diag_emission` passes at `EXPECTED_LEAVES = 671`.

### V3 — Determinism check
Two-tier:
1. **Byte-identical**: `rewards`, `dones`, `actions` host buffers compared against a pre-B-10 baseline run with seed 42, 200+50 b=16 fold-1. The new reductions use **separate `__shared__` arrays** from existing reductions (§3.4) so compute order on the gradient-producing path is unchanged.
2. **Float-tolerant on loss values**: `lr_plateau.{q,pi,v}.loss_ema` allowed up to `1e-5 × value` relative tolerance against the baseline. The added per-batch scratch writes in `ppo_clipped_surrogate.cu` don't interact with the existing surrogate reduction's compute (separate shared-mem), but kernel-launch-ordering changes from the additional launches can produce float-associativity differences below 1e-5.

If invariant (1) fails, B-10 has introduced a real determinism bug. If only (2) tolerance is consumed, that's expected float-non-associativity and acceptable for observability changes.

### V4 — Cluster validation (B-7+B-8+B-9+B-10 joint, after B-10 lands)
Submit fold-1 walk-forward 20k+5k b=1024 on H100. Read the 13 new diag leaves to apply the falsification criteria from G5 and decide B-11 scope.

## 5. Implementation phases

| Phase | Step | LOC |
|------|------|-----|
| 1 | Add 13 ISV slots to `isv_slots.rs`; bump `RL_SLOTS_END` to 743 | 25 |
| 2 | Add 13 bootstrap entries; bump fixed-size array 217 → 230 in `integrated.rs:3394` | 14 |
| 3 | Write `rl_q_distribution_stats.cu` (new kernel: Q-fwd output → 3 slots, single-block tree-reduce, no atomicAdd) | ~120 |
| 4 | Register new kernel in `build.rs` + load in trainer's DqnHead-or-equivalent | 10 |
| 5 | Add `launch_q_distribution_stats` call after each Q forward in `step_with_lobsim_gpu` | ~20 |
| 6 | Modify `rl_q_pi_distill_grad.cu`: emit target entropy + π entropy diff (2 slots, inline in existing reduce) | ~30 |
| 7 | Modify `ppo_clipped_surrogate.cu`: write 5 per-batch scratch arrays alongside existing `loss_per_batch[B]`; 5 new pointer params on the kernel + matching update at the single call site `integrated.rs:5191` (`PolicyHead::surrogate_forward`) | ~80 |
| 7b | Write new kernel `rl_ppo_diagnostic_stats_reduce.cu` (single-block tree-reduce; pattern from `rl_bellman_target_saturation_reduce.cu`); reads slot 612 for σ_used = sqrt(var_pre_norm) | ~110 |
| 7c | Register `rl_ppo_diagnostic_stats_reduce` in `build.rs`; load handle + add `launch_ppo_diagnostic_stats_reduce` method in `PolicyHead` (`ppo.rs`); call from trainer immediately after surrogate_forward returns | ~30 |
| 7d | Allocate 5 new `[B] f32` scratch buffers in `IntegratedTrainer::new` (`ppo_a_norm_pb_d`, `ppo_a_unnorm_pb_d`, `ppo_ratio_dev_pb_d`, `ppo_ratio_clipped_pb_d`, `ppo_surrogate_pb_d`) and add struct fields | ~25 |
| 8 | Add 14 diag leaves under new `policy_diagnostic` block in `build_diag_value` (13 new slots + `q_pi_agree_ema` from existing slot 407) | 20 |
| 9 | Bump `EXPECTED_LEAVES` to **671** in `eval_diag_emission.rs` | 1 |
| 10 | Write `policy_diagnostic_observability.rs` invariant test — 6 assertions × ~15 LOC each + harness boilerplate (binary path, JSONL reader, dot_get helper, smoke command — patterns lifted from `popart_disaggregation_invariants.rs` ~225 LOC and `c51_atom_saturation_diagnostic.rs` ~200 LOC) | ~120 |
| 11 | Build release; run new test + schema test + 200+50 smoke | — |
| 12 | Single commit `feat(rl): B-10 policy-quality cascade diagnostic` | — |

Total source delta: ~470 LOC across 8 files. Largest of the B-series so far due to three kernel edits + new kernel.

## 6. Open decisions

| # | Question | Recommendation |
|---|----------|---------------|
| 1 | Should `q_dist_entropy_mean` average across all (b, a) cells or only over the *selected action* per batch element? | All (b, a) cells. The all-action entropy is what's input to Q→π distill; selected-action only would mask the cross-action degeneracy that drives the cascade. |
| 2 | Should `ppo_a_sigma_used` read the live PopArt σ_eff or compute the empirical std of `A_unnorm / A_norm`? | Empirical. PopArt σ_eff is one input but the actual divisor in `compute_advantage_return` may differ (per-batch std in some implementations). Empirical reveals truth. |
| 3 | Does `ppo_l_surrogate` include the value loss term `c1 × L_VF`? | No. Surrogate-only (the clipped policy term). Value loss is separately published as `lr_plateau.v.loss_ema`. |
| 4 | Should B-10 also emit `q_pi_agree_ema` (slot 407)? | **RESOLVED: yes, add as 14th diag leaf WITHOUT new slot.** Audit (grep `q_pi_agree` in `integrated.rs` diag emission paths) returns zero hits — the `rl_q_pi_agree_b` kernel is loaded (`integrated.rs:985-986`, 2035-2040) and runs per-step writing slot 407, but the value never reaches diag. B-10 adds one diag leaf `policy_diagnostic.q_pi_agree_ema = isv[RL_Q_ARG_VS_PI_AGREE_INDEX]` reading the existing slot. Net effect: 13 new slots, **14 new diag leaves**, EXPECTED_LEAVES 657 → **671**. Q-π agreement is THE canonical signal for the Q-collapse vs Q-learning question that B-10 is built to diagnose — strictly worse to ship without it. |
| 5 | Risk that 14 new diag leaves + 2 new reducer kernels (`rl_q_distribution_stats`, `rl_ppo_diagnostic_stats_reduce`) + 5 scratches in `ppo_clipped_surrogate.cu` slow training meaningfully on b=1024? | Likely negligible — each new launch is single-block, no cross-step state. Concrete gate: V1 smoke compares sps pre/post; **fail if >5% sps regression measured**. Per `pearl_cudarc_sync_training_event_was_87pct_wall_time.md` historic profiling shows reducer-launch overhead is dwarfed by host-side sync on this codebase, so 2 new launches add ~50-100µs combined. |

## 7. Done means

- 13 new ISV slots allocated (730-742); `RL_SLOTS_END = 743`
- 2 new kernels (`rl_q_distribution_stats.cu` + `rl_ppo_diagnostic_stats_reduce.cu`) + 2 modified kernels (`rl_q_pi_distill_grad.cu`, `ppo_clipped_surrogate.cu` — 6 new pointer params + 1 scalar; 2 call sites updated atomically)
- 14 new diag leaves under `policy_diagnostic` (13 from new slots + 1 from existing slot 407 exposure)
- `EXPECTED_LEAVES = 671` schema test passes
- Invariant test passes locally
- 200+50 b=16 fold-1 smoke clean
- Determinism check passes (rewards/dones byte-identical)
- Single commit on `ml-alpha-regime-observer`

## 8. Risks

- **Risk A (medium):** 13 new slots + 3 kernel touches is the largest B-series instrumentation. Mitigation: schema-parity test catches drift; invariant test catches semantic errors.
- **Risk B (low):** `ppo_clipped_surrogate.cu` is on the hot training path. Adding 5 reductions increases its work. Mitigation: existing kernel already does loss reduction; add stats reductions in parallel without serializing.
- **Risk C (low):** `rl_q_pi_distill_grad.cu` modifications must remain mathematically equivalent. Mitigation: V3's byte-identical check on `rewards/dones/actions` (rewards drive gradients on the close step; if gradients perturb, next-step rewards diverge). Float-tolerant tier of V3 covers the `lr_plateau.pi.loss_ema` drift expected from extra kernel launches.
- **Risk D (low):** New kernel adds one more sync point per step. Mitigation: launches on same stream as Q forward; <5% step-time impact expected.

## 9. What B-10 does NOT promise

- Does not fix the eval-pnl problem. It IS NOT a policy-quality fix.
- Does not motivate B-11 in any particular direction — the falsification criteria are designed to let the data choose between Q-collapse, Q-distill-amplification, and PopArt-PPO-coupling hypotheses.
- Does not change training dynamics, RNG, gradients, or any kernel output. Pure observability.

## 10. After B-10: candidate B-11 paths (NOT in this spec)

Based on literature + the falsification criteria, candidate downstream specs:

- **B-11-α (if Q-collapse dominates)**: Cap V_MAX_eff to surfer zone (~5) per `pearl_c51_v_max_freeze_required_for_surfer`, OR replace adaptive EWMA with a magnitude-aware controller that holds atom span when Q discrimination is high.
- **B-11-β (if Q-distill amplification dominates)**: Gate `q_distill_lambda` to 0 when `q_dist_entropy_mean > 0.9 × ln(Q_N_ATOMS)` (don't distill from a collapsed Q); OR adopt SAC-style adaptive temperature scheme.
- **B-11-γ (if PopArt × PPO coupling dominates)**: Decouple advantage normalization (use per-batch standardization for PPO surrogate, keep PopArt for V regression target only — Schulman 2017 canonical).

Each B-11 path will have its own spec written *after* B-10 data confirms which root cause to address.
