# B-9 — C51 Bellman-Target Saturation Diagnostic

**Date:** 2026-06-01
**Status:** Draft (pre-implementation)
**Branch target:** `ml-alpha-adaptive-controller-floors`
**Predecessors:** B-7 (clamp-disabled default) → B-8 (popart σ disaggregation)
**Pearls referenced:**
- [[pearl_c51_atom_span_must_track_clamp_range]] — atom span / clamp range coupling history
- [[pearl_c51_v_max_freeze_required_for_surfer]] — `c51_v_max=1.0` frozen for surfer policy (DO NOT widen blindly)
- [[pearl_clamp_v_target_at_atom_span]] — V-regression target clamped to atom span
- [[feedback_no_atomicadd]] — no atomicAdd; scratch + tree-reduce only
- [[feedback_use_consts_not_literals_for_structural_dims]] — `Q_N_ATOMS`, not `N_ATOMS`
- [[feedback_no_partial_refactor]] — both bellman entry points instrumented atomically

---

## 1. Motivation

Under B-7 (clamp default-disabled), the per-transition reward in `bellman_target_projection` and its fused twin `bellman_fused_select_project` is no longer bounded to `[-3, +1]`. Inside the kernel (file `crates/ml-alpha/cuda/bellman_target_projection.cu`):

```c
// line 188-190 (both entry points have identical math):
const float atom_value = V_MIN_eff + (float)atom * DELTA_Z;
const float t_z        = r + gamma_eff * atom_value;
const float t_z_clamp  = fmaxf(V_MIN_eff, fminf(V_MAX_eff, t_z));   // <-- saturation here
```

`V_MIN_eff` / `V_MAX_eff` come from `isv[RL_C51_V_MIN_INDEX=485]` / `isv[RL_C51_V_MAX_INDEX=484]`. Per `pearl_c51_v_max_freeze_required_for_surfer`, V_MAX is held frozen at 1.0 to preserve the surfer policy (widening past 100-200 → trend-follower, past 1000 → degraded). A reverted Fix F (commit `0fe825a8c` → revert `6d4a962e5`) already attempted atom-span decoupling and degraded the policy.

B-7's removed clamp now lets `r` exceed `V_MAX_eff` routinely. Each `t_z` overshoot is silently clamped before being mapped onto the discrete atom support. We do not know:
- How often `t_z_clamp != t_z` per training step (saturation rate)
- How large the overshoots are in absolute magnitude (`max(t_z_unclamped)`)

This information is necessary to evaluate whether the frozen atom span has become a bottleneck under B-7. Per the prior Fix F failure, **B-9 is pure observability**. Any adaptation decision is a separate spec downstream of B-9 data.

## 2. Goals

### G1 — Per-step saturation rate
For each training step, compute the fraction of `(batch, source_atom_z)` pairs where the pre-clamp `t_z` exceeds `V_MAX_eff` or falls below `V_MIN_eff`. Denominator: `B × Q_N_ATOMS` (each thread in each block evaluates one `t_z` per pearl_no_atomicadd-safe pattern). Publish as two ISV slots — top and bottom rates kept separate (asymmetric tails matter under loss-aversion).

### G2 — Per-step extreme overshoot magnitude
Track `max(t_z)` and `min(t_z)` BEFORE clamping across `B × Q_N_ATOMS`. Publishes to two more ISV slots. Lets diag see "how far past V_MAX did we go" — a peak of 5.0 against V_MAX=1.0 means 5× overshoot; tells us whether widening atom span by 2× would help or whether tails escape by orders of magnitude.

### G3 — Instrument both kernel entry points
`bellman_target_projection` and `bellman_fused_select_project` are mathematically identical for the t_z computation. Per `feedback_no_partial_refactor`, both must be instrumented in the same commit; otherwise reads of the four ISV slots become trainer-path-dependent.

### G4 — No behavior change
Projection still clamps; Q-learning targets unchanged. Identical RNG / gradient trajectory vs B-7+B-8 baseline.

### G5 — Non-goals
- **No atom-span adaptation.** Prior Fix F revert is the cautionary tale. B-9 ships pure observability; adaptation (B-10 or later) needs explicit cluster evidence.
- **No N_ATOMS change.** Resolution stays at 21.
- **No reward-side change.** B-7 already owns the clamp; B-9 measures downstream effect.

## 3. Design

### 3.1 ISV slots (4 new)

```rust
// crates/ml-alpha/src/rl/isv_slots.rs
// Contiguous after B-8's slot 725.

/// B-9 (2026-06-01): fraction (0..1) of (b, atom_z) pairs per step
/// where pre-clamp t_z > V_MAX_eff (top atom saturation). Computed in
/// rl_bellman_target_saturation_reduce.cu over scratch arrays written
/// by both bellman_target_projection entry points.
pub const RL_Q_TARGET_TOP_SATURATION_RATE_INDEX: usize = 726;

/// B-9: same for t_z < V_MIN_eff (bottom atom saturation). Asymmetric
/// tails matter under HFT loss-aversion — keep separate from top.
pub const RL_Q_TARGET_BOT_SATURATION_RATE_INDEX: usize = 727;

/// B-9: max(t_z) across (b, atom_z) BEFORE clamp. Diagnoses
/// overshoot magnitude — peak of 5.0 against V_MAX=1.0 means 5× over.
pub const RL_Q_TARGET_MAX_PRE_PROJ_INDEX: usize = 728;

/// B-9: min(t_z) across (b, atom_z) BEFORE clamp. Symmetric to MAX
/// for negative tail.
pub const RL_Q_TARGET_MIN_PRE_PROJ_INDEX: usize = 729;

pub const RL_SLOTS_END: usize = 730;
```

Bootstrap entries (all 0.0, overwritten per step by the reducer):
```rust
(RL_Q_TARGET_TOP_SATURATION_RATE_INDEX, 0.0_f32),
(RL_Q_TARGET_BOT_SATURATION_RATE_INDEX, 0.0_f32),
(RL_Q_TARGET_MAX_PRE_PROJ_INDEX,        0.0_f32),
(RL_Q_TARGET_MIN_PRE_PROJ_INDEX,        0.0_f32),
```

The bootstrap is a **fixed-size const array** at `integrated.rs:3372`. Assuming B-8 lands first (baseline `[(usize, f32); 213]`), B-9 must in the same commit per `feedback_wire_everything_up.md`:

1. `isv_slots.rs`: bump `RL_SLOTS_END` to 730.
2. `integrated.rs:3372`: change array type annotation `[(usize, f32); 213]` → `[(usize, f32); 217]` AND append the four bootstrap tuples.

If B-8 has NOT landed when B-9 is implemented, baseline is `212` and target is `216` — verify against current `RL_SLOTS_END` before editing.

### 3.2 Cross-batch reduction strategy (no atomicAdd)

Bellman runs grid `(B, 1, 1)` × block `(Q_N_ATOMS, 1, 1)` — one block per batch element. Cross-batch aggregation requires either atomicAdd (forbidden) or scratch + separate reduce. Use **scratch + reduce**, the standard codebase pattern.

**New device scratch arrays (allocated once at trainer init):**
- `target_sat_top_per_batch_d: [B] f32` — per-block sum of top-saturation counts
- `target_sat_bot_per_batch_d: [B] f32` — per-block sum of bottom-saturation counts
- `target_max_pre_per_batch_d: [B] f32` — per-block max of t_z pre-clamp
- `target_min_pre_per_batch_d: [B] f32` — per-block min of t_z pre-clamp

**New reducer kernel** `crates/ml-alpha/cuda/rl_bellman_target_saturation_reduce.cu`: single block, block_dim threads, grid-stride loop over B; tree-reduce in shared mem; thread 0 writes 4 ISV slots. Pattern lifted from `apply_reward_scale.cu`'s three-parallel-reduction kernel.

### 3.3 Kernel changes

#### A. `bellman_target_projection.cu` — both entry points

Each block has `Q_N_ATOMS = 21` threads (an ODD count, see §7 Risk A). A symmetric tree-reduce over 21 threads has a partner-skip bug: at stride=11 thread 10 has no partner; at stride=5 thread 10's value is unreachable. Instead, **gather per-thread `t_z` into shared memory, then have thread 0 sequentially reduce** — this matches the existing softmax max/sum pattern in the same kernel (lines 157-163 and 171-176, and 234-240 / 248-253 in the non-fused variant).

Sketch (applied identically to both `bellman_target_projection` and `bellman_fused_select_project`, in the section after `t_z` is computed at line 189 / 290):

```c
// B-9 (2026-06-01): per-(b, atom_z) saturation tally over pre-clamp t_z.
// Sequential thread-0 reduction over Q_N_ATOMS=21 — odd count rules out
// a symmetric tree-reduce (partner-skip bug at stride 11 + stride 5).
// Pattern lifted from this kernel's softmax max-subtract / sumexp
// (lines 157 / 171). Per pearl_no_atomicadd: NO atomicAdd; per-block
// shared-mem write + thread-0 fan-in.
__shared__ float s_t_z[Q_N_ATOMS];
s_t_z[atom] = t_z;          // each thread writes its own pre-clamp t_z
__syncthreads();

if (atom == 0) {
    float sum_top = 0.0f, sum_bot = 0.0f;
    float max_tz  = s_t_z[0], min_tz = s_t_z[0];
    #pragma unroll
    for (int z = 0; z < Q_N_ATOMS; ++z) {
        const float tz_z = s_t_z[z];
        sum_top += (tz_z > V_MAX_eff) ? 1.0f : 0.0f;
        sum_bot += (tz_z < V_MIN_eff) ? 1.0f : 0.0f;
        max_tz   = fmaxf(max_tz, tz_z);
        min_tz   = fminf(min_tz, tz_z);
    }
    target_sat_top_per_batch[batch] = sum_top;
    target_sat_bot_per_batch[batch] = sum_bot;
    target_max_pre_per_batch[batch] = max_tz;
    target_min_pre_per_batch[batch] = min_tz;
}
__syncthreads();   // ensure all threads cleared barrier before kernel returns
```

Shared-mem cost: `Q_N_ATOMS × sizeof(float) = 84 bytes` per block (a single `s_t_z` array; the four aggregates are thread-0 registers, not shared mem). One additional `__syncthreads()` before the thread-0 block, one after. Compared to the existing softmax pattern (which has 4 syncthreads already), this is a +2 syncthreads delta. Per kernel comment "Q_N_ATOMS = 21 inner-loop syncs is negligible (single block per batch, small block size)" the addition is in-budget.

Both kernel entry points gain **4 new pointer parameters** (the 4 scratch arrays). Trainer launch site updated for both. Per `feedback_no_partial_refactor.md`: in the same commit.

**Per-step bellman variant invariant.** The trainer runs **exactly one** of the two bellman entry points per step (the fused variant if action selection is needed, the non-fused variant if `target_logits` is already gathered upstream). Both write to the SAME scratch buffers; the reducer reads them after whichever variant ran. Per-step `denom = B × Q_N_ATOMS` is correct because exactly one variant filled the per-batch scratch this step. Per `feedback_wire_everything_up.md`: when adding a third bellman variant in the future, instrument it identically in the same commit.

#### B. New kernel: `rl_bellman_target_saturation_reduce.cu`

```c
// rl_bellman_target_saturation_reduce.cu — cross-batch tree-reduce of
// per-batch saturation tallies from bellman_target_projection. Writes
// per-step rates + extremes to ISV. Single block, no atomicAdd
// (feedback_no_atomicadd).

#define RL_Q_TARGET_TOP_SATURATION_RATE_INDEX  726
#define RL_Q_TARGET_BOT_SATURATION_RATE_INDEX  727
#define RL_Q_TARGET_MAX_PRE_PROJ_INDEX         728
#define RL_Q_TARGET_MIN_PRE_PROJ_INDEX         729
#define Q_N_ATOMS                              21

extern "C" __global__ void rl_bellman_target_saturation_reduce(
    float*       __restrict__ isv,
    const float* __restrict__ sat_top_per_batch,    // [B]
    const float* __restrict__ sat_bot_per_batch,    // [B]
    const float* __restrict__ max_pre_per_batch,    // [B]
    const float* __restrict__ min_pre_per_batch,    // [B]
    int                       B
) {
    extern __shared__ float smem[];
    float* s_top = smem;
    float* s_bot = smem +     blockDim.x;
    float* s_max = smem + 2 * blockDim.x;
    float* s_min = smem + 3 * blockDim.x;
    const int tid = threadIdx.x;

    // Grid-stride per-thread accumulate (sum-sum-max-min).
    float l_top = 0.0f, l_bot = 0.0f;
    float l_max = -INFINITY, l_min = INFINITY;
    for (int b = tid; b < B; b += blockDim.x) {
        l_top += sat_top_per_batch[b];
        l_bot += sat_bot_per_batch[b];
        l_max  = fmaxf(l_max, max_pre_per_batch[b]);
        l_min  = fminf(l_min, min_pre_per_batch[b]);
    }
    s_top[tid] = l_top;
    s_bot[tid] = l_bot;
    s_max[tid] = l_max;
    s_min[tid] = l_min;
    __syncthreads();

    for (int stride = blockDim.x / 2; stride > 0; stride >>= 1) {
        if (tid < stride) {
            s_top[tid] += s_top[tid + stride];
            s_bot[tid] += s_bot[tid + stride];
            s_max[tid]  = fmaxf(s_max[tid], s_max[tid + stride]);
            s_min[tid]  = fminf(s_min[tid], s_min[tid + stride]);
        }
        __syncthreads();
    }

    if (tid == 0) {
        const float denom = (float)(B * Q_N_ATOMS);
        isv[RL_Q_TARGET_TOP_SATURATION_RATE_INDEX] = s_top[0] / denom;
        isv[RL_Q_TARGET_BOT_SATURATION_RATE_INDEX] = s_bot[0] / denom;
        isv[RL_Q_TARGET_MAX_PRE_PROJ_INDEX]        = s_max[0];
        isv[RL_Q_TARGET_MIN_PRE_PROJ_INDEX]        = s_min[0];
    }
}
```

Caller: `block_dim = min(B, 256)`, `grid = (1,1,1)`, `shared = 4 * block_dim * sizeof(float)`.

#### C. Trainer plumbing (`integrated.rs`)

- Allocate 4 new `[B] f32` device scratch arrays in trainer construction (CudaSlice).
- Update launch sites for `bellman_target_projection` and `bellman_fused_select_project` to pass the 4 scratch pointers. Both call sites need the change atomically.
- Add `rl_bellman_target_saturation_reduce` launch immediately AFTER whichever bellman variant ran for that step.
- Register kernel in `build.rs`.

### 3.4 Diag emission

`build_diag_value`, under existing `"q"` block:

```rust
"q": {
    // ... existing leaves ...
    // B-9 (2026-06-01): C51 Bellman target saturation observability.
    // top/bot rates count (b, atom_z) pairs where pre-clamp t_z escapes
    // [V_MIN_eff, V_MAX_eff]; max/min_pre_proj show overshoot magnitude.
    // NOTE — distinct from popart.max_abs_reward_ema: the popart slot
    // is a Welford/F4 envelope of per-step reward magnitudes (one value
    // per step per account). q.target_max_pre_proj is the per-step max
    // of the Bellman target `r + γ·atom_value` across (B × Q_N_ATOMS),
    // which can exceed the reward magnitude by γ·V_MAX_eff and is
    // independent of popart's normalization layer. See pearl_popart_blind_to_session_signals.
    "target_top_saturation_rate": isv[RL_Q_TARGET_TOP_SATURATION_RATE_INDEX],
    "target_bot_saturation_rate": isv[RL_Q_TARGET_BOT_SATURATION_RATE_INDEX],
    "target_max_pre_proj":        isv[RL_Q_TARGET_MAX_PRE_PROJ_INDEX],
    "target_min_pre_proj":        isv[RL_Q_TARGET_MIN_PRE_PROJ_INDEX],
},
```

Schema leaf bump: 653 → 657 (4 new leaves; assumes B-8 already shipped, so baseline is 653). Update `EXPECTED_LEAVES` in `eval_diag_emission.rs`:

```rust
// 2026-06-01 B-9: +4 leaves for q.target_{top,bot}_saturation_rate +
// q.target_{max,min}_pre_proj. New total: 657.
const EXPECTED_LEAVES: usize = 657;
```

### 3.5 Invariant test

`crates/ml-alpha/tests/c51_atom_saturation_diagnostic.rs`:

```rust
//! B-9 invariants for C51 Bellman-target saturation observability.
//!
//! Validates four invariants over a 200+100 b=16 fold-1 smoke run:
//!
//! 1. Rates lie in [0.0, 1.0]   — proportion invariant per row of diag.jsonl
//! 2. max_pre_proj >= min_pre_proj    — sanity
//! 3. Consistency: if top_rate > 0, max_pre_proj > V_MAX_eff;
//!    if bot_rate > 0, min_pre_proj < V_MIN_eff.
//!    (V_MAX_eff/V_MIN_eff read per-row from the same diag — they are
//!    ISV-driven and may move during the run.)
//! 4. Entry-point parity (separate harness — see §3.5.1 below).
```

#### 3.5.1 — Entry-point parity test

The two bellman entry points have DIFFERENT input signatures:
- `bellman_target_projection(target_logits[B×Q_N_ATOMS], rewards, dones, n_step_gammas, isv, B, target_dist)` — consumes pre-selected target logits.
- `bellman_fused_select_project(full_logits[B×N_ACTIONS×Q_N_ATOMS], actions[B], rewards, dones, n_step_gammas, isv, B, target_dist)` — selects action row internally.

A naive "same inputs" parity test cannot run. Instead, write a dedicated GPU-oracle harness (NOT integrated with the training binary) that:

1. Allocates a deterministic `full_logits[B×N_ACTIONS×Q_N_ATOMS]` (seeded RNG) and a deterministic `actions[B]` vector.
2. Calls `dqn_select_action_atoms` on `(full_logits, actions)` → produces `target_logits[B×Q_N_ATOMS]` device-side.
3. Allocates two separate sets of scratch buffers — `scratch_A` for the non-fused call, `scratch_B` for the fused call.
4. Calls `bellman_target_projection(target_logits, ...)` with `scratch_A`.
5. Calls `bellman_fused_select_project(full_logits, actions, ...)` with `scratch_B`.
6. Asserts every (b, atom) entry of `scratch_A.{sat_top, sat_bot, max_pre, min_pre}` is bit-identical to `scratch_B.{...}`.

Per `feedback_no_cpu_test_fallbacks.md`: NO host-side reference implementation; both kernels are the oracle for each other. The test fails if either kernel diverges, even by float ε.

### 3.6 Falsification criteria (what kills downstream B-10)

These outcomes from the next cluster run (B-7 alone OR B-7+B-8+B-9, whichever ships) tell us atom-span adaptation is unnecessary:

- Train-phase `target_top_saturation_rate + target_bot_saturation_rate` stays < 1% across 20k steps
- Eval-phase combined saturation rate stays < 5% across all 5k eval steps
- `target_max_pre_proj` rarely exceeds 2× V_MAX_eff

If saturation is sustained > 5% in train or > 20% in eval, atom-span adaptation becomes the next priority and gets its own spec with empirical data (NOT a precautionary fix per `feedback_no_quickfixes`).

## 4. Validation

### V1 — Local smoke (15 min)
Build release. Run 200+100 b=16 fold-1 with B-7 clamp disabled. Verify:
- All 4 new leaves emitted in train AND eval diag
- Rates in [0,1], max ≥ min, consistency invariants hold
- Saturation rates likely non-zero (the point of the diagnostic)

### V2 — Schema parity
`eval_diag_emission` test passes with `EXPECTED_LEAVES = 657`.

### V3 — Test parity between bellman entry points
Run the dedicated parity harness described in §3.5.1. Both `bellman_target_projection` and `bellman_fused_select_project` must produce bit-identical scratch values when fed equivalent inputs (constructed by running `dqn_select_action_atoms` externally to bridge the signature gap).

### V4 — Determinism
With seed fixed, B-9 reads/writes don't perturb training. Verify by diffing `rewards_host` and `dones_host` against a pre-B-9 baseline run.

## 5. Implementation phases

| Phase | Step | LOC |
|------|------|-----|
| 1 | Add 4 ISV slots to `isv_slots.rs`; bump `RL_SLOTS_END` to 730 | 10 |
| 2 | Add 4 bootstrap entries in `with_controllers_bootstrapped` | 4 |
| 3 | Allocate 4 `[B] f32` scratch buffers in trainer | 8 |
| 4 | Modify `bellman_target_projection.cu` — both entry points get per-block tally + scratch write; add 4 new pointer params | ~80 |
| 5 | Write new `rl_bellman_target_saturation_reduce.cu` | ~60 |
| 6 | Register reduce kernel in `build.rs` | 1 |
| 7 | Update launch sites in trainer for both bellman variants + new reducer | ~30 |
| 8 | Add 4 diag leaves under `q.*` in `build_diag_value` | 6 |
| 9 | Bump `EXPECTED_LEAVES` to 657 | 1 |
| 10 | Write `c51_atom_saturation_diagnostic.rs` test | ~120 |
| 11 | Build release; run new test + schema-parity test + 200+100 smoke | — |
| 12 | Single commit on `ml-alpha-adaptive-controller-floors` | — |

Total source-file delta: ~320 LOC across 7 files. Larger than B-8 because cross-batch reduction needs scratch buffers + a new reducer kernel.

## 6. Done means

- 4 ISV slots allocated (726-729); `RL_SLOTS_END = 730`
- 4 scratch buffers allocated and wired
- Both bellman entry points instrumented identically (parity test passes)
- New reducer kernel registered and called per step
- 4 new diag leaves under `q.*`
- `EXPECTED_LEAVES = 657`
- Invariant test passes locally
- Determinism preserved (rewards/dones byte-identical vs baseline)
- Commit message: `feat(rl): B-9 C51 Bellman-target saturation observability`

## 7. Risks

- **Risk A (low / mitigated):** Q_N_ATOMS = 21 is ODD; a naive symmetric tree-reduce loses index 10. Mitigated by §3.3's sequential thread-0 reduction over `s_t_z[21]` (matches existing softmax pattern in the same kernel). Shared-mem cost: `Q_N_ATOMS × 4 = 84` bytes per block (single array); +2 `__syncthreads()` calls. Validate with `compute-sanitizer racecheck` in V1.
- **Risk B (low):** Both bellman entry points need parity. Mitigation: V3 test asserts bit-identical saturation values.
- **Risk C (low):** Adding 4 new pointer params to existing kernels — call sites need atomic update. Mitigation: search `bellman_target_projection`/`bellman_fused_select_project` call sites; both should be in `integrated.rs`.
- **Risk D (low):** Scratch buffer lifetime vs `reset_session_state`. The 4 per-batch `[B] f32` scratch arrays are overwritten by the bellman kernel at every training step (one block per batch element, thread-0 writes its row of `sat_top / sat_bot / max_pre / min_pre`). The reducer's grid-stride loop reads exactly `B` elements per step. No state survives between steps that could corrupt the next step's saturation rate. No `reset_session_state` change required; no preservation needed at the train→eval boundary. If batch size `B` ever becomes variable per-step (currently fixed), the reducer's `int B` parameter already takes the per-call value — no stale-tail issue.
- **Risk E (low / grandfathered):** The new `rl_bellman_target_saturation_reduce.cu` kernel `#define`s the 4 slot indices (726-729) locally, mirroring the Rust `isv_slots.rs` consts. Identical pattern to B-8's Risk D and to the existing `bellman_target_projection.cu` (which `#define`s `RL_C51_V_MAX_INDEX`, `RL_C51_V_MIN_INDEX`, `RL_GAMMA_INDEX`). Per `feedback_use_consts_not_literals_for_structural_dims.md` (spirit): the Rust const is canonical; the kernel `#define` is a manual mirror enforced by same-commit discipline. Drift would surface either as a wrong diag value (caught by §3.5 invariant test) or as a compile-time slot-index conflict in `isv_slots.rs`.
- **Risk E (low):** Schema parity assumes B-8 already shipped (baseline 653). If B-9 lands without B-8, baseline is 652 and `EXPECTED_LEAVES` should be 656. Spec assumes B-8 → B-9 ordering.

## 8. What B-9 does NOT promise

- It does not change the policy, the kernel outputs, or training dynamics. Pure observability.
- It does not pre-stage an atom-span adaptation. Per the reverted Fix F lesson, adaptation requires cluster evidence and its own spec.
- It does not promise the next cluster run improves eval pnl. It promises *attribution*: after the run, we can read the saturation rates and decide whether atom span is the next bottleneck.
