# SP17 — Dueling Q-Network: Identifiable V/A Decomposition

> **For agentic workers:** REQUIRED SUB-SKILL: `superpowers:subagent-driven-development`. Steps use checkbox `- [ ]` syntax. Each task is bite-sized TDD: failing test → run-fail → implement → run-pass → commit.

> **Status: DRAFT FOR HUMAN REVIEW.** Several genuine design choices are flagged with **DESIGN DECISION:** below. Please redirect or confirm before any subagent dispatch.

---

## Goal

Add the **identifiability constraint** to the existing distributional dueling Q-architecture so action selection responds to relative advantage `A(s,a) − mean_a A(s,a)`, not absolute state value `V(s)`. This is the structural fix for the multi-sink Q-bias that train-multi-seed-b5gmp (sha 641aa0dfd) exposed.

**Concretely:**

```
PRE-SP17:  Q[a, z] = V_logit[z] + A_logit[a, z]                       (additive, not identifiable)
POST-SP17: Q[a, z] = V_logit[z] + (A_logit[a, z] − mean_a A_logit[a, z])  (mean-zero centered)
```

---

## Critical pre-discovery (load-bearing — read before approving)

The codebase already has structurally separate value and advantage heads:

| Buffer | Shape | Source | Purpose |
|---|---|---|---|
| `on_v_logits_buf` | `[B, NA]` | shared value head (cuBLAS pass) | per-atom value distribution `Z_V[z]` |
| `on_b_logits_buf` | `[B, (B0+B1+B2+B3)*NA]` | per-action advantage head (cuBLAS pass) | per-action atom logits `Z_A[a,z]` |

`compute_expected_q` (`crates/ml/src/cuda_pipeline/experience_kernels.cu:4499`) explicitly comments **"combines value + advantage logits via dueling aggregation"**, but the aggregation is naive addition — no `A − mean A` subtraction anywhere. Same naive addition in:

- `compute_expected_q` (line 4583): `float logit = v_row[z] + adv_a[z];`
- `mag_concat_qdir` (line 5072): `float logit = dir_logits_v[b * NA + z] + dir_logits_b[…]`
- `c51_loss_kernel.cu` (block_bellman_project), `c51_grad_kernel.cu` — same pattern

**This is the SP17 surgical insertion point.** We are not building dueling; we are completing the identifiability term that was either never written or got dropped. The required surgery is small, isolated, and atomic across all consumers (per `feedback_no_partial_refactor`).

This dramatically simplifies SP17 vs. the original sketch: we are **not** adding new Q-head parameters, **not** splitting into separate V/A heads, **not** allocating new param tensors or Adam state. We are inserting a per-sample per-branch reduction into existing kernels.

---

## Background — what train-multi-seed-b5gmp proved

Fold 1 final on commit 641aa0dfd (post full SP16 chain):

| Metric | Fold 0 final | Fold 1 final | pfh9n baseline (Fold 1) |
|---|---|---|---|
| val Sharpe | 81.42 | 55.55 | 64.24 |
| Hold% | 38.1% | 52.6% | 45.5% |
| dir_entropy | 0.81 | 0.70 | 0.715 |
| Q(Flat) − Q(Long) gap | 0.060 | **0.148** | n/a (not measured) |
| Q-mean Fold 1 epoch trajectory | n/a | **0.258 → 0.412 (+60%)** | n/a |
| Q(Flat) Fold 1 epoch trajectory | n/a | **0.358 → 0.484 (+35%)** | n/a |

Three structural failures, in priority order:

1. **(parallel fix in flight)** Wiener-α stationarity violation
2. **Multi-sink Q-bias.** Hold% pressure migrated the bias from Q(Hold) to Q(Flat). The model selects whichever zero-variance "do nothing" sink dominates. Per-bar Hold cost (SP16) attacks one sink; the bias jumps to the other. Without identifiability, the *next* fix would just push it to the third sink.
3. **Bellman target inflation.** Q-mean climbed +60% on Fold 1 with flat train_loss → optimistic bootstrap on non-action states (Hold/Flat). Same root cause: V(s) leaks into the action-selection signal.

The literature mechanism (Wang et al. 2016 Dueling DQN) is exact: when `Q = V + A` lacks identifiability, gradient updates on `Q(s,a)` are split arbitrarily between `V` and `A`, and the network drifts toward "explain everything via V." Mean-zero `A` forces the action-relevant signal into `A` by construction.

---

## Out of scope

- New separate V/A *parameter* heads (the heads already exist — we modify aggregation only)
- Replacing C51 with non-distributional Q (orthogonal; we keep C51-distributional dueling)
- IQN dueling (separate change — flagged in **DESIGN DECISION 5** below)
- Reward shaping (SP11/SP16 territory — handled elsewhere)

---

## Design decisions (REQUIRES HUMAN REVIEW)

### DESIGN DECISION 1: Mean-zero vs. max-zero identifiability

Wang et al. 2016 propose two forms:

```
mean-zero:  Q(s,a) = V(s) + (A(s,a) − (1/|A|) · Σ_a A(s,a))
max-zero:   Q(s,a) = V(s) + (A(s,a) − max_a A(s,a))
```

**Recommendation:** **mean-zero per branch.** Wang's Fig. 1 ablation shows mean-zero is more stable; max-zero ties one A logit to the V head and creates non-smooth gradient. Mean-zero is a smooth differentiable reduction over |branch_size| values (4, 3, 3, 3) — trivial to implement in the existing softmax kernels.

> _Human reviewer: confirm mean-zero, or pick max-zero._

### DESIGN DECISION 2: C51 + dueling — atom-level vs. expectation-level mean

Two possible loci of the `A − mean A` subtraction:

```
(A) atom-level:        Z_A[a, z] − mean_a Z_A[a, z]      (subtract per atom z, before softmax)
(B) expectation-level: Z_A[a, z] − mean_a E[Z_A[a, z]]   (subtract a scalar, equal across atoms)
```

**Recommendation:** **(A) atom-level.** Subtracting per atom preserves the distributional shape per action (the atom-level identifiability constraint Wang & Rajeswar 2018 derive for distributional dueling). Option (B) would couple the atom shape to the action-mean expectation, which collapses distributional information.

Operationally both are cheap. Atom-level: `Z_A[a, z] -= (1/|A|) Σ_a Z_A[a, z]` per (sample, atom) — one reduction per atom. Expectation-level: per-action E[Z_A] then collapse — adds one full softmax-expectation pass.

> _Human reviewer: confirm atom-level mean-zero._

### DESIGN DECISION 3: Scope — direction-only or all four branches

Branches: dir(4), mag(3), ord(3), urg(3). Each branch already has separately-allocated advantage logits in `b_logits` BRANCH-MAJOR layout (see `experience_kernels.cu:4527-4531`).

**Recommendation:** **all four branches.** The Q-bias evidence is direction-specific (Q(Flat) vs Q(Long)), but applying identifiability to one branch and not the others creates the partial-refactor pathology (`feedback_no_partial_refactor`): three consumers reading the old contract, one reading the new, and they disagree more than they did before. Per-branch mean is independent (each branch's mean over its own |branch| actions), so the implementation cost scales linearly.

The shared V head feeds all four branches anyway — once V is identifiable for direction, the magnitude/order/urgency logits should be centered for the same architectural reason or risk re-introducing the same pathology in those subspaces.

> _Human reviewer: confirm all-4-branch scope._

### DESIGN DECISION 4: Target network — also identifiable

When the Bellman target is computed, the target net's `Q_target = V_target + A_target` must use the **same** identifiability projection as the online net, or training/target asymmetry will create chronic Bellman-residual bias.

**Recommendation:** **target net uses same projection.** No alternative is structurally sound — they are the same architecture, target is a delayed copy. Single helper function consumed by both.

> _Human reviewer: confirm._

### DESIGN DECISION 5: IQN dueling — defer to follow-up

The trainer also runs IQN (`iqn_dual_head_kernel.cu`, `iqn_cvar_kernel.cu`, `iqn_quantile_ema_kernel.cu`). IQN does not currently use the V/A decomposition (it's a single quantile head per action).

**Recommendation:** **defer IQN dueling.** Add as SP18 follow-up. The C51 Q-distribution is what `compute_expected_q` produces and what feeds Thompson selection; IQN's contribution to `q_eff` is via a separate τ-blend pathway (`pearl_per_branch_iqn_tau_schedule`). If C51 dueling fixes the multi-sink bias the IQN treatment becomes a Phase 5+ optimization, not a structural fix. Mark as "out of scope, may revisit."

> _Human reviewer: confirm defer, or fold IQN dueling into SP17._

### DESIGN DECISION 6: V-head shared across branches vs. per-branch

The current architecture has **one** value head `v_logits ∈ [B, NA]` shared across all four action branches. Wang's original paper has one V-head for one action space; here we have a multi-branch action space with one V-head shared.

This is sound: V(s) is a state property and the four branches all act on the same state. Per-branch V would be saying "the value of this state for the dir-branch is different from the value for the mag-branch" — which violates the Markov state-value definition.

**Recommendation:** **keep shared V-head. No change.**

> _Human reviewer: confirm._

### DESIGN DECISION 7: ISV slots for diagnostics

To detect whether identifiability is doing work post-deployment, we want HEALTH_DIAG observables:

- `A_var_ema` per branch (variance of mean-centered A across actions) — should be > 0 if A is doing real work
- `V_share` per branch (fraction of Q-magnitude attributable to V vs. A) — diagnostic for "V is dominating" pathology
- `advantage_clip_bound` — adaptive clamp on `|A_centered|` to bound runaway A-tails per `feedback_isv_for_adaptive_bounds`

**Recommendation:** **9 new ISV slots** at `[474..483)` (A_var_ema × 4 branches + V_share × 4 branches + advantage_clip_bound × 1). Per `feedback_isv_for_adaptive_bounds`, the bound is signal-driven (tracks p99 of `|A_centered|` × safety_factor, not a hardcoded scalar).

Alternative: **0 new slots, diagnostic-only via existing per-branch Q-mean infrastructure.** Mean-centering is mathematically rigid (no adaptation knob); A_var/V_share can be computed at HEALTH_DIAG cadence on existing buffers without a slot.

> _Human reviewer: pick (a) 9 slots with adaptive clip + diagnostics, or (b) 0 slots, diagnostics-only computed at HEALTH_DIAG. The slot allocation is the architecturally cleaner choice but adds wire-up complexity. Recommend (a)._

---

## Phase ordering (deviates from sketch)

The original sketch had Phase 0 (diagnostic) → Phase 1 (split heads) → Phase 2 (behavioral tests) → Phase 3 (integrate) → Phase 4 (smoke) → Phase 5 (full validation).

**Revised ordering** (heads already exist, integration is small):

| Phase | Purpose | Roughly bite-sized tasks |
|---|---|---|
| Pre-Phase | Branch + worktree + ISV slots + V/A diagnostic emit | 3 |
| Phase 0 | V/A decomposition diagnostic — confirm Q(Flat) over-attribution is V-driven (KILL CRITERION) | 3 |
| Phase 1 | Mean-zero identifiability in `compute_expected_q` (online + target) — atomic across consumers | 5 |
| Phase 2 | Mean-zero identifiability in `c51_loss_kernel` + `c51_grad_kernel` (training path) | 4 |
| Phase 3 | Mean-zero identifiability in `mag_concat_qdir` + Thompson direction-selection consumers | 3 |
| Phase 4 | Behavioral test suite (synthetic markets — V/A separation oracle tests) | 4 |
| Phase 5 | HEALTH_DIAG diagnostic emit + adaptive `advantage_clip_bound` producer | 3 |
| Phase 6 | 5-epoch L40S smoke validation | 1 |
| Phase 7 | 30-epoch full validation post-smoke-pass | 1 |

Total: **27 tasks**. Each is a single TDD cycle.

The sketch's "Phase 1 split current Q-head into V-head + A-head" doesn't apply — heads are already split. The actual work is the identifiability projection plus its atomic consumer migration.

---

## File Structure

```
docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md   ← this file
crates/ml/src/cuda_pipeline/sp14_isv_slots.rs                  ← +9 SP17 ISV slots OR no change (DD7)
crates/ml/src/cuda_pipeline/state_layout.cuh                   ← C-side mirror (only if DD7=a)
crates/ml/src/cuda_pipeline/experience_kernels.cu              ← `compute_expected_q`, `mag_concat_qdir`, action select
crates/ml/src/cuda_pipeline/c51_loss_kernel.cu                 ← block_bellman_project + softmax over (V+A)
crates/ml/src/cuda_pipeline/c51_grad_kernel.cu                 ← grad path through identifiability
crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs                 ← optional new diagnostic readers, ISV registration
crates/ml/src/trainers/dqn/state_reset_registry.rs             ← SP17 ISV slots fold-reset entries (if DD7=a)
crates/ml/src/trainers/dqn/trainer/training_loop.rs            ← HEALTH_DIAG emit for V/A diagnostics
crates/ml/tests/sp17_dueling_oracle_tests.rs                   ← NEW test file, mirrors sp14_oracle_tests.rs pattern
crates/ml/build.rs                                             ← register new test cubins (only if new kernels)
bin/fxt/src/main.rs                                            ← no change (no new CLI flags)
```

---

## Hard rules (apply to every task in this plan)

- **No CPU compute.** All V/A reductions GPU-side. Tests use `MappedF32Buffer` — no `unsafe { … memcpy }` outside mapped-pinned (`feedback_no_htod_htoh_only_mapped_pinned`).
- **No atomicAdd.** Per-sample per-branch mean uses block tree-reduce or per-thread sequential reduction (each thread owns one sample, sums |branch| values from registers — no cross-thread reduction needed).
- **No partial refactor.** When `compute_expected_q` is migrated to centered-A, the same commit migrates `c51_loss_kernel`, `c51_grad_kernel`, and `mag_concat_qdir`. Phases 1–3 are scoped per-consumer but **must land within a single feature branch's commit graph before any L40S dispatch.** A partially-migrated state where some kernels read centered-A and others read raw-A is strictly worse than no change at all — disagreement at the multi-consumer boundary.
- **Wire-up audit per task.** No orphan changes (`feedback_wire_everything_up`). If a task adds a buffer/kernel/slot, the same commit wires it to its consumer.
- **ISV-driven adaptive bounds.** `advantage_clip_bound` (if **DD7=a**) is signal-driven (tracks `p99(|A_centered|) × safety_factor`), never hardcoded (`feedback_isv_for_adaptive_bounds`).
- **Use `sp4_histogram_p99.cuh`** for percentile computations (per `pearl_fused_per_group_statistics_oracle`).
- **Pearl candidate per phase.** Each phase ends with a memory-pearl draft for the human reviewer to land or reject.

---

## Pre-Phase: Branch + worktree + ISV slot allocation

**Why first:** Set up isolation per `feedback_no_concurrent_agents_shared_tree`; allocate the ISV slots so subsequent phases can reference final indices.

### Task PP.1: Create worktree (do not create — included for reviewer)

```bash
# Run from main repo root; do not run as part of this plan dispatch.
cd /home/jgrusewski/Work/foxhunt
git worktree add .worktrees/sp17-dueling-q-network -b feat/sp17-dueling main
cd .worktrees/sp17-dueling-q-network
```

- [ ] **Step 1: Reviewer runs the above before subagent dispatch.**

No tests, no commits — the worktree command is an environmental setup the reviewer performs.

### Task PP.2: Allocate 9 SP17 ISV slots [474..483) — IF DD7=a

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/sp14_isv_slots.rs` — new section at end
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — bump `ISV_TOTAL_DIM` 474 → 483, update layout fingerprint seed
- Modify: `crates/ml/src/cuda_pipeline/state_layout.cuh` — mirror constant if applicable
- Test: `crates/ml/src/cuda_pipeline/sp14_isv_slots.rs` — new `#[test]` `sp17_slot_layout_locked`

- [ ] **Step 1: Write the failing test** (append to existing `mod tests` in sp14_isv_slots.rs)

```rust
/// Lock SP17 (2026-05-08) dueling-Q identifiability diagnostic slot layout.
/// 9 contiguous slots [474..483) — A_var_ema per branch (4) + V_share
/// per branch (4) + adaptive advantage_clip_bound (1).
#[test]
fn sp17_dueling_slot_layout_locked() {
    assert_eq!(SP17_DUELING_SLOT_BASE, 474);
    assert_eq!(SP17_DUELING_SLOT_END,  483);
    assert_eq!(A_VAR_EMA_DIR_INDEX,    474);
    assert_eq!(A_VAR_EMA_MAG_INDEX,    475);
    assert_eq!(A_VAR_EMA_ORD_INDEX,    476);
    assert_eq!(A_VAR_EMA_URG_INDEX,    477);
    assert_eq!(V_SHARE_DIR_INDEX,      478);
    assert_eq!(V_SHARE_MAG_INDEX,      479);
    assert_eq!(V_SHARE_ORD_INDEX,      480);
    assert_eq!(V_SHARE_URG_INDEX,      481);
    assert_eq!(ADVANTAGE_CLIP_BOUND_INDEX, 482);
    // Pearl-A bootstrap sentinels.
    assert_eq!(SENTINEL_A_VAR_EMA,  0.0);
    assert_eq!(SENTINEL_V_SHARE,    0.5);  // initial 50/50 split
    assert_eq!(SENTINEL_ADVANTAGE_CLIP_BOUND, 1.0);  // 1.0 = "no effective clipping" cold-start
    // Category-1 dimensional safety bounds per `feedback_isv_for_adaptive_bounds`.
    assert_eq!(ADVANTAGE_CLIP_BOUND_MIN, 0.1);
    assert_eq!(ADVANTAGE_CLIP_BOUND_MAX, 100.0);
}

#[test]
fn all_sp17_slots_fit_within_isv_total_dim() {
    use crate::cuda_pipeline::gpu_dqn_trainer::ISV_TOTAL_DIM;
    assert!(
        SP17_DUELING_SLOT_END <= ISV_TOTAL_DIM,
        "SP17_DUELING_SLOT_END={} exceeds ISV_TOTAL_DIM={}; \
         bump ISV_TOTAL_DIM in gpu_dqn_trainer.rs (and update layout_fingerprint_seed()).",
        SP17_DUELING_SLOT_END, ISV_TOTAL_DIM,
    );
}
```

Run: `SQLX_OFFLINE=true cargo test -p ml --lib sp17_dueling_slot_layout_locked all_sp17_slots_fit_within_isv_total_dim`
Expected: FAIL — constants undefined.

- [ ] **Step 2: Add SP17 ISV section to sp14_isv_slots.rs** (append after SP16 T3 block at line 614)

```rust
// ── SP17 (2026-05-08): dueling Q-network identifiability diagnostics ───
// Adds 9 ISV slots driving the V/A diagnostic chain that observes
// whether the post-SP17 mean-centered advantage is doing real work and
// detects regression to the pre-SP17 V-dominated regime. Per
// `feedback_isv_for_adaptive_bounds`, all signal-driven; the
// `advantage_clip_bound` is producer-tracked from p99(|A_centered|),
// never hardcoded.
//
// Mathematical rationale: with mean-zero identifiability
// `A_centered[a, z] = A[a, z] − (1/|branch|) Σ_a A[a, z]`, the per-branch
// variance of `A_centered` over actions measures how much the policy
// discriminates between actions. Var(A) → 0 = "all actions equivalent,
// V dominates" = pre-SP17 pathology re-emerging. Var(A) > 0 with V_share
// stable around 0.5 = healthy dueling decomposition.
//
// Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md
pub const A_VAR_EMA_DIR_INDEX:        usize = 474;  // Var(A_centered) over dir actions, EMA
pub const A_VAR_EMA_MAG_INDEX:        usize = 475;
pub const A_VAR_EMA_ORD_INDEX:        usize = 476;
pub const A_VAR_EMA_URG_INDEX:        usize = 477;
pub const V_SHARE_DIR_INDEX:          usize = 478;  // |V_contribution| / (|V| + |A_centered|)
pub const V_SHARE_MAG_INDEX:          usize = 479;
pub const V_SHARE_ORD_INDEX:          usize = 480;
pub const V_SHARE_URG_INDEX:          usize = 481;
pub const ADVANTAGE_CLIP_BOUND_INDEX: usize = 482;  // adaptive |A_centered| upper bound

// Sentinels — Pearl-A first-observation bootstrap.
pub const SENTINEL_A_VAR_EMA:           f32 = 0.0;
pub const SENTINEL_V_SHARE:             f32 = 0.5;   // healthy 50/50 split as cold-start
pub const SENTINEL_ADVANTAGE_CLIP_BOUND: f32 = 1.0;  // 1.0 → no effective clipping pre-first-obs

// Category-1 dimensional safety bounds per `feedback_isv_for_adaptive_bounds`.
// Floor 0.1: below this any centered advantage is essentially noise; clip
// would erase the action signal. Ceiling 100.0: above this a single
// adversarial outlier can dominate per-batch EMAs.
pub const ADVANTAGE_CLIP_BOUND_MIN: f32 = 0.1;
pub const ADVANTAGE_CLIP_BOUND_MAX: f32 = 100.0;

// Producer constants. Safety factor 1.5× p99 mirrors the SP14 P0-A
// REWARD_POS_CAP producer pattern. EMA α slow per-fold cadence (the bound
// shouldn't track per-batch noise).
pub const ADVANTAGE_CLIP_SAFETY_FACTOR: f32 = 1.5;
pub const ADVANTAGE_CLIP_EMA_ALPHA:     f32 = 0.01;

pub const SP17_DUELING_SLOT_BASE: usize = 474;
pub const SP17_DUELING_SLOT_END:  usize = 483;
```

- [ ] **Step 3: Bump `ISV_TOTAL_DIM` in `gpu_dqn_trainer.rs`** at line 2523

Change `pub(crate) const ISV_TOTAL_DIM: usize = 474;` to `483`. Append the SP17 explanation at the end of the existing docstring chain (mirror the SP16 T3 pattern at the top of the comment block — short paragraph naming the slots and the producer kernel).

- [ ] **Step 4: Update layout fingerprint seed** at line 3605

Change `ISV_TOTAL_DIM=474;` to `ISV_TOTAL_DIM=483;` and append the new slot names to the seed string in the same `b""` literal:

```
SLOT_474_A_VAR_EMA_DIR=474;\
SLOT_475_A_VAR_EMA_MAG=475;\
SLOT_476_A_VAR_EMA_ORD=476;\
SLOT_477_A_VAR_EMA_URG=477;\
SLOT_478_V_SHARE_DIR=478;\
SLOT_479_V_SHARE_MAG=479;\
SLOT_480_V_SHARE_ORD=480;\
SLOT_481_V_SHARE_URG=481;\
SLOT_482_ADVANTAGE_CLIP_BOUND=482;\
ISV_TOTAL_DIM=483;\
```

- [ ] **Step 5: Add fold-reset registry entries** in `state_reset_registry.rs`

For each of the 9 slots, add a `RegistryEntry { name: "sp17_<slot_name>", category: ResetCategory::FoldReset, description: "..." }` entry matching the existing SP14 Layer C pattern at the end of the registry's `new()` method.

- [ ] **Step 6: Run test** — Expected: PASS.

```bash
SQLX_OFFLINE=true cargo test -p ml --lib sp17_dueling_slot_layout_locked all_sp17_slots_fit_within_isv_total_dim
```

- [ ] **Step 7: Commit**

```bash
git add -A
git commit -m "$(cat <<'EOF'
plan(sp17): allocate ISV slots [474..483) for dueling-Q diagnostics

9 slots: A_var_ema × 4 branches (per-branch advantage variance EMA)
+ V_share × 4 branches (V/(V+A) magnitude share) + adaptive
advantage_clip_bound. All Pearl-A first-observation bootstrap with
fold-reset registry entries. Bump ISV_TOTAL_DIM 474 → 483 + layout
fingerprint seed.

Per `feedback_isv_for_adaptive_bounds`: advantage_clip_bound is
producer-tracked from p99(|A_centered|) × 1.5 safety factor, never
hardcoded. Bounds [0.1, 100.0] are Category-1 dimensional safety floors.

No consumer change in this commit (additive infrastructure). Phase 1
mean-centering producer (atomic across compute_expected_q +
c51_loss + c51_grad + mag_concat_qdir) lands in the next 4 commits.

Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```

> _If reviewer picks **DD7=b**, this entire task is skipped and the diagnostic emit in Phase 5 is computed at HEALTH_DIAG cadence on existing buffers without new slots._

### Task PP.3: HEALTH_DIAG emit for raw V/A means (PRE-CENTERING — sets the baseline)

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — find existing per-epoch HEALTH_DIAG block, add `v_a_means` line
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — add `read_v_a_means_pre_centering()` reading from existing `on_v_logits_buf` + `on_b_logits_buf`
- Test: `crates/ml/tests/sp17_dueling_oracle_tests.rs` — NEW file with `v_a_means_pre_centering_emits_expected_layout`

- [ ] **Step 1: Write failing test** (creates new file `crates/ml/tests/sp17_dueling_oracle_tests.rs`)

```rust
//! SP17 dueling-Q oracle tests.
//!
//! Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md
//!
//! Layer A (CPU oracle) tests verify the math of mean-zero identifiability
//! against synthetic logit tensors without GPU dependencies.
//!
//! Layer B (GPU oracle) tests verify the kernel-level implementations
//! match the CPU oracle bit-for-bit using `MappedF32Buffer` per
//! `feedback_no_htod_htoh_only_mapped_pinned`. All B-tests are
//! `#[ignore = "requires GPU"]`.

#[cfg(feature = "cuda")]
#[allow(unsafe_code)] // CUDA kernel launch + mapped-pinned memory.
mod gpu {
    // ── A.1: V/A means readout layout sanity check ─────────────────
    /// Verifies the diagnostic readout buffer for HEALTH_DIAG has the
    /// expected layout: [v_mean_pre, a_mean_dir_pre, a_mean_mag_pre,
    ///                   a_mean_ord_pre, a_mean_urg_pre]. Pre-centering
    /// (Phase 0 — no centering yet applied).
    #[test]
    #[ignore = "requires GPU"]
    fn v_a_means_pre_centering_emits_expected_layout() {
        // Force a synthetic distribution where v_logits = constant 0.3
        // and per-branch A means are known [0.1, 0.05, -0.02, -0.05].
        // Launch read_v_a_means_pre_centering. Assert the readback matches.
        unimplemented!(
            "Phase PP.3 Step 3 lands the implementation; this test gates the wire-up"
        );
    }
}
```

Run: `cargo test -p ml --test sp17_dueling_oracle_tests --features cuda v_a_means_pre_centering_emits_expected_layout -- --ignored --nocapture`
Expected: FAIL with "not yet implemented".

- [ ] **Step 2: Register new test in `crates/ml/Cargo.toml`** (mirror `sp14_oracle_tests` `[[test]]` entry)

- [ ] **Step 3: Implement `read_v_a_means_pre_centering()` in `gpu_dqn_trainer.rs`**

Add a small block-tree-reduce kernel `v_a_means_diag_kernel` that emits 5 floats per launch:
```
[E[V] over batch and atoms,
 E[A_dir] over batch and (actions × atoms),
 E[A_mag] over batch and (actions × atoms),
 E[A_ord] over batch and (actions × atoms),
 E[A_urg] over batch and (actions × atoms)]
```

Reduction is per-block warp-shuffle then block-shared-mem then a finalize kernel that sequentially sums block partials (no atomicAdd, per `feedback_no_atomicadd`).

Allocate a `MappedF32Buffer<5>` `v_a_means_diag_buf` on `GpuDqnTrainer` for cold-path readback.

- [ ] **Step 4: Wire `read_v_a_means_pre_centering()` into HEALTH_DIAG emit**

In `training_loop.rs` per-epoch HEALTH_DIAG block (search for `q_by_action` or `mag` line), append:
```rust
let v_a = trainer.read_v_a_means_pre_centering();
info!(
    "HEALTH_DIAG[{epoch}]: v_a_means [v={:.4} a_dir={:.4} a_mag={:.4} a_ord={:.4} a_urg={:.4}]",
    v_a[0], v_a[1], v_a[2], v_a[3], v_a[4]
);
```

- [ ] **Step 5: Implement test body**

Allocate input buffers via `MappedF32Buffer::new`. Set `v_logits = 0.3` everywhere, `b_logits` with per-branch known means. Launch reader. Assert readback matches expected to ε=1e-5.

- [ ] **Step 6: Run test** — Expected: PASS.

- [ ] **Step 7: Commit**

```bash
git add -A
git commit -m "$(cat <<'EOF'
plan(sp17): HEALTH_DIAG v_a_means baseline (PRE-CENTERING)

Adds per-epoch readout of the V/A breakdown so we can observe the
pre-SP17 baseline distribution before the identifiability projection
lands. Also instruments the kill criterion for Phase 0:

  HEALTH_DIAG[N]: v_a_means [v=X a_dir=Y a_mag=... a_ord=... a_urg=...]

If train-multi-seed-b5gmp's Q(Flat) over-attribution is V-driven, V
should be elevated (~0.4+) while a_dir is small. If V is balanced and
A is doing the work, dueling cannot help and we abort SP17.

Block-tree-reduce kernel `v_a_means_diag_kernel` (no atomicAdd per
`feedback_no_atomicadd`); MappedF32Buffer per
`feedback_no_htod_htoh_only_mapped_pinned`.

Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```

---

## Phase 0 — Diagnostic verification + KILL CRITERION

**Why first:** Confirm the V-dominance hypothesis before architectural surgery. If V is already small and A is dominant, the identifiability projection is a no-op and SP17 is dead.

### Task 0.1: 1-epoch Fold-1 diagnostic dispatch

- [ ] **Step 1: Push branch + dispatch 1-epoch L40S smoke**

The reviewer (NOT the subagent) runs:

```bash
git push -u origin feat/sp17-dueling
./scripts/argo-train.sh --model dqn --epochs 1 --label sp17-phase0-vapre
```

- [ ] **Step 2: Read HEALTH_DIAG output**

Tail the `v_a_means` lines across the full Fold 1 epoch. Compute mean and trajectory of `[v, a_dir, a_mag, a_ord, a_urg]`.

- [ ] **Step 3: KILL CRITERION**

```
ABORT SP17 if   |v| < 0.5 × max(|a_dir|, |a_mag|, |a_ord|, |a_urg|)   (V is small, A dominates)
PROCEED  if   |v| > max(|a_dir|, |a_mag|, |a_ord|, |a_urg|)            (V dominates → identifiability will help)
INCONCLUSIVE if neither — convene human review
```

- [ ] **Step 4: Document result**

Append a "Phase 0 evidence" subsection to this plan with raw numbers and decision.

No commit unless evidence is appended.

---

## Phase 1 — Mean-zero identifiability in `compute_expected_q`

**Why this kernel first:** It is the action-selection reader (Thompson, eval). The behavior change here is observable end-to-end.

### Task 1.1: CPU oracle test for mean-centered Q computation

**Files:**
- Modify: `crates/ml/tests/sp17_dueling_oracle_tests.rs` — add CPU-side `compute_expected_q_centered_cpu_oracle` test

- [ ] **Step 1: Write failing test**

```rust
/// CPU oracle for mean-zero identifiability.
///
/// Given v_logits = [0.5, 0.5] (NA=2 atoms) and per-branch advantage logits
/// b_logits_dir = [[0.1, 0.0], [0.3, 0.2], [0.0, 0.0], [-0.2, -0.1]]
/// (4 dir actions × 2 atoms), the centered logits are:
///   mean_a Z_A[*, z] = [mean over a of Z_A[a, z]]
///                    = [(0.1+0.3+0.0-0.2)/4, (0.0+0.2+0.0-0.1)/4]
///                    = [0.05, 0.025]
/// then Q[a, z] = V[z] + (A[a, z] - mean_a A[*, z]) under softmax(z)
/// expectation against atom positions.
///
/// This test pins the math contract independent of GPU implementation.
#[test]
fn compute_expected_q_centered_cpu_oracle() {
    let v: [f32; 2] = [0.5, 0.5];
    let a_raw: [[f32; 2]; 4] = [
        [0.1,  0.0],
        [0.3,  0.2],
        [0.0,  0.0],
        [-0.2, -0.1],
    ];
    // mean over a, per atom z
    let mean_a_z = [
        (a_raw[0][0] + a_raw[1][0] + a_raw[2][0] + a_raw[3][0]) / 4.0,
        (a_raw[0][1] + a_raw[1][1] + a_raw[2][1] + a_raw[3][1]) / 4.0,
    ];
    assert!((mean_a_z[0] - 0.05).abs() < 1e-6);
    assert!((mean_a_z[1] - 0.025).abs() < 1e-6);

    // Centered A
    let mut a_centered = [[0.0f32; 2]; 4];
    for a in 0..4 {
        for z in 0..2 {
            a_centered[a][z] = a_raw[a][z] - mean_a_z[z];
        }
    }

    // Per-z sum of centered A across actions == 0 (identifiability)
    for z in 0..2 {
        let s: f32 = (0..4).map(|a| a_centered[a][z]).sum();
        assert!(s.abs() < 1e-6, "centered A must sum to 0 per atom, got {} at z={}", s, z);
    }

    // Combined Q logits
    let mut q_logits = [[0.0f32; 2]; 4];
    for a in 0..4 {
        for z in 0..2 {
            q_logits[a][z] = v[z] + a_centered[a][z];
        }
    }

    // E[Q] under softmax(z) with atom positions [0.0, 1.0]
    let atoms = [0.0f32, 1.0];
    for a in 0..4 {
        let m = q_logits[a][0].max(q_logits[a][1]);
        let e0 = (q_logits[a][0] - m).exp();
        let e1 = (q_logits[a][1] - m).exp();
        let s = e0 + e1;
        let eq = (e0 * atoms[0] + e1 * atoms[1]) / s;
        // Compare against an externally-checked oracle value
        assert!(eq.is_finite());
    }
}
```

Run: `cargo test -p ml --test sp17_dueling_oracle_tests compute_expected_q_centered_cpu_oracle`
Expected: PASS immediately (pure math).

- [ ] **Step 2: Commit**

```bash
git add -A
git commit -m "$(cat <<'EOF'
test(sp17): CPU oracle for mean-zero centered Q computation

Locks the mean-zero identifiability math contract independent of GPU
implementation. Subsequent Phase 1 GPU oracle tests compare kernel
output against this contract bit-for-bit.

Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```

### Task 1.2: GPU oracle test for `compute_expected_q` with centered A

- [ ] **Step 1: Write failing GPU test**

```rust
/// B.1: compute_expected_q GPU implementation matches the CPU oracle
/// when DUELING_CENTERED is engaged (always engaged post-SP17).
#[test]
#[ignore = "requires GPU"]
fn compute_expected_q_centered_matches_cpu_oracle() {
    // Build the same v / a_raw / atoms inputs as the CPU oracle test on
    // mapped-pinned device buffers.
    // Launch compute_expected_q with the post-SP17 kernel.
    // Compare per-action E[Q] against the CPU oracle to ε=1e-5.
    unimplemented!("Phase 1 Step 3 lands the kernel modification");
}
```

Expected on first run: FAIL.

- [ ] **Step 2: Modify `compute_expected_q` kernel** (`experience_kernels.cu` line 4499)

Insert the per-atom mean-A reduction inside the per-branch outer loop, BEFORE the action-loop logit accumulation:

```cuda
/* SP17 mean-zero identifiability: subtract per-atom mean of A across this branch's
 * actions. The reduction is sample-local (one thread = one sample) and per-atom z,
 * computed from the n_d action logits already in the per-thread inner loop's
 * scope. No cross-thread reduction needed; no atomicAdd; no shared memory.
 *
 * Per Wang et al. 2016 Dueling Networks (mean-zero variant), this form is
 * preferred over max-zero because:
 *   - smooth differentiable reduction (no argmax tie-breaking)
 *   - all actions contribute proportionally to V/A split (max-zero pins one
 *     action to V which creates the per-action gradient asymmetry the paper
 *     identifies as "less stable" in Fig. 1)
 *
 * Per-branch independent: dir uses dir actions only, mag uses mag actions only,
 * etc. The shared V head is invariant across branches by construction (one V
 * per state, all branches share).
 *
 * Atom-level (not expectation-level): subtract per atom z, before softmax.
 * Preserves distributional shape per action (Wang & Rajeswar 2018 distributional
 * dueling identifiability result).
 */
float a_mean_per_atom[NUM_ATOMS_MAX]; /* register array sized to compile-time max NA */
float inv_n_d = 1.0f / (float)n_d;
for (int z = 0; z < num_atoms; z++) {
    float s = 0.0f;
    for (int a = 0; a < n_d; a++) {
        s += branch_base[(long long)a * num_atoms + z];
    }
    a_mean_per_atom[z] = s * inv_n_d;
}

/* ── now the existing online softmax loop reads `adv_a[z] - a_mean_per_atom[z]`
 *     instead of plain `adv_a[z]` ─────────────────────────────────────────── */
for (int a = 0; a < n_d; a++) {
    const float* adv_a = branch_base + (long long)a * num_atoms;
    float M = -1e30f;
    float S = 0.0f;
    /* …same accumulators as before… */
    for (int z = 0; z < num_atoms; z++) {
        float a_centered = adv_a[z] - a_mean_per_atom[z];   /* SP17 */
        float logit = v_row[z] + a_centered;                  /* was: v_row[z] + adv_a[z] */
        /* …rest of online softmax unchanged… */
    }
    /* …rest of action loop unchanged… */
}
```

`NUM_ATOMS_MAX`: align with the existing `THOMPSON_MAX_ATOMS = 128` constant. Add `#ifndef NUM_ATOMS_MAX #define NUM_ATOMS_MAX 128 #endif` near the kernel top, with an `assert(num_atoms <= NUM_ATOMS_MAX)` device-side `__trap()` if violated (cold-fail, never silent truncation).

- [ ] **Step 3: Implement test body** + run.

Expected: PASS.

- [ ] **Step 4: Commit (atomic — only `compute_expected_q`, NO consumer migration yet)**

```bash
git add -A
git commit -m "$(cat <<'EOF'
feat(sp17): mean-zero identifiability in compute_expected_q

Inserts per-atom mean-A reduction inside the per-branch outer loop:
Q[a, z] = V[z] + (A[a, z] - mean_a A[*, z])
The post-centering atom-level identifiability holds: Σ_a A_centered[a, z] = 0
for every (sample, branch, atom).

Per Wang et al. 2016 Dueling Networks: mean-zero is smoother and more
stable than max-zero. Atom-level subtraction (not expectation-level)
preserves distributional shape per action.

Sample-local register reduction; no atomicAdd, no shared memory, no
cross-thread sync. NUM_ATOMS_MAX=128 mirrors existing THOMPSON_MAX_ATOMS;
device __trap() on overflow.

⚠ INTERIM STATE: c51_loss_kernel + c51_grad_kernel + mag_concat_qdir
still read RAW (un-centered) advantage logits. Phase 1 Tasks 1.3-1.5
migrate them in lockstep within this branch BEFORE any L40S dispatch.
A partially-migrated state is forbidden per `feedback_no_partial_refactor`.

Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```

### Task 1.3: Pre-migration grep for all consumers of `b_logits` raw advantage

**Why:** Per `feedback_no_partial_refactor`, every consumer of the changed contract migrates in the same branch.

- [ ] **Step 1: Audit script**

```bash
grep -rn "v_logits\|adv_a\[\|b_logits\[\|adv_logit" \
    crates/ml/src/cuda_pipeline/ | \
    grep -vE "(test|//|\\*)" > /tmp/sp17_consumers.txt
cat /tmp/sp17_consumers.txt
```

Catalogue every line. Expected consumers (grouped):
- `compute_expected_q` (Task 1.2 done)
- `c51_loss_kernel.cu`: `block_bellman_project_f`, `c51_loss_batched`
- `c51_grad_kernel.cu`: gradient through softmax(V+A)
- `mag_concat_qdir`: direction Q for magnitude-conditioning concatenation
- `add_advantage_noise`: writes Gaussian noise into Q-values *post-aggregation* — NO change required (operates on already-centered Q)
- `experience_kernels.cu` Thompson direction-selection block (line ~1264-1500): uses `b_logits_dir` directly for Thompson sampling

- [ ] **Step 2: Append the consumer list to this plan** as Phase 1 evidence — locked checklist for Tasks 1.4–1.5 to mark off.

No commit (audit-only).

### Task 1.4: Migrate `c51_loss_kernel` + `c51_grad_kernel` to centered A (atomic)

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/c51_loss_kernel.cu` — insert mean-A reduction in `block_bellman_project_f` and `c51_loss_batched`
- Modify: `crates/ml/src/cuda_pipeline/c51_grad_kernel.cu` — propagate gradient through the centering (the centering is a linear projection, so `dL/dA[a, z] -= mean_a (dL/dA[*, z])`)

- [ ] **Step 1: Write failing GPU oracle test**

```rust
/// B.2: c51_loss kernel uses centered A and reproduces the CPU oracle's
/// E[Q] under the centered formulation when computing the projection
/// target.
#[test]
#[ignore = "requires GPU"]
fn c51_loss_uses_centered_a_in_projection() {
    // Synthetic v_logits + b_logits + reward + next_v_logits + next_b_logits,
    // run c51_loss with the post-SP17 kernel, recover the projection target,
    // confirm it matches the centered-A CPU oracle prediction to ε=1e-5.
    unimplemented!("Phase 1 Step 3 lands the kernel modification");
}
```

- [ ] **Step 2: Modify both kernels** with the same per-atom mean-A subtraction pattern as Task 1.2.

For the gradient kernel: per chain rule, the linear projection `A_centered = A - (1/n_d) Σ_a A` has Jacobian `J[a, a'] = δ(a, a') - 1/n_d`. Therefore `dL/dA[a, z] = dL/dA_centered[a, z] - (1/n_d) Σ_a' dL/dA_centered[a', z]`. Applied per atom z, this is one extra reduction per branch per atom — same pattern as forward. Document the math inline in a comment block above the gradient kernel modification.

- [ ] **Step 3: Run test, verify PASS**

- [ ] **Step 4: Wire-up audit**

```bash
grep -rn "b_logits\[\|adv_a\[" crates/ml/src/cuda_pipeline/c51_loss_kernel.cu crates/ml/src/cuda_pipeline/c51_grad_kernel.cu | grep -v "centered\|a_mean_per_atom\|//"
```
Expected output: empty (every advantage read goes through centering).

- [ ] **Step 5: Commit**

```bash
git add -A
git commit -m "$(cat <<'EOF'
feat(sp17): centered A in c51_loss + c51_grad (atomic with compute_expected_q)

Migrates the Bellman target projection (block_bellman_project_f) and
its gradient kernel to the post-SP17 mean-zero identifiability. The
projection's Jacobian for the linear centering A_c = A - (1/n) Σ A is
J[a, a'] = δ(a, a') - 1/n; therefore the gradient backprop applies the
SAME per-atom mean subtraction (idempotent for symmetric reductions).

Inline math comment block in c51_grad_kernel above the modified
reduction documents the chain rule.

Wire-up audit: grep confirms zero un-centered b_logits reads in the
two kernels.

Per `feedback_no_partial_refactor`: this commit closes the contract
gap that compute_expected_q opened. After this commit, all Bellman-
target paths consume centered A.

Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```

### Task 1.5: Migrate `mag_concat_qdir` + Thompson direction-selection block (atomic with c51)

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — `mag_concat_qdir` (line 5031), Thompson direction block (line ~1264)

`mag_concat_qdir` reads `dir_logits_v + dir_logits_b` (line 5072) per direction action. Insert the per-atom mean-A reduction over the 4 direction actions (b0_size=4) before the inner softmax. Identical pattern to Tasks 1.2/1.4.

Thompson block: `softmax_c51_inline(logits_i + d * n_atoms, n_atoms, probs_d)` reads raw direction advantage. The Thompson sampler operates on *direction advantage softmax*; it must read centered A or it disagrees with `compute_expected_q`'s E[Q] (the conviction quantity). Insert mean-A reduction before the per-direction softmax loop (line ~1276).

- [ ] **Step 1: Write failing GPU oracle test for mag_concat_qdir centered**

```rust
#[test]
#[ignore = "requires GPU"]
fn mag_concat_qdir_uses_centered_dir_advantage() {
    // Build v + b_logits_dir with known a_mean_per_atom = [m_z0, m_z1, ...]
    // Launch mag_concat_qdir.
    // Confirm Q_dir written into the concat tail equals the centered E[Q],
    // matching the CPU oracle for the dir branch.
    unimplemented!();
}
```

- [ ] **Step 2: Write failing GPU oracle test for Thompson centered**

```rust
#[test]
#[ignore = "requires GPU"]
fn thompson_dir_selection_consumes_centered_advantage() {
    // Synthetic dir advantage where raw argmax = Hold but centered argmax = Long
    // (achievable by setting the V head high so V dominates the raw sum).
    // Launch the action-selection kernel; assert the picked direction is Long
    // with high probability (>0.8 across N=10000 Philox draws).
    unimplemented!();
}
```

- [ ] **Step 3: Migrate both call sites** as in Tasks 1.2/1.4. Test PASS.

- [ ] **Step 4: Wire-up audit**

```bash
grep -rn "dir_logits_b\|b_logits_dir\|adv_a\[" crates/ml/src/cuda_pipeline/experience_kernels.cu | grep -v "centered\|a_mean_per_atom\|//\|\\*"
```

Expected: every match is inside a `// pre-centering removed` comment or inside the now-centered code blocks.

- [ ] **Step 5: Commit**

```bash
git add -A
git commit -m "$(cat <<'EOF'
feat(sp17): centered A in mag_concat_qdir + Thompson dir-select

Closes the consumer migration that compute_expected_q + c51_loss +
c51_grad opened. After this commit, every advantage-logit read in the
DQN pipeline goes through the mean-zero identifiability projection.

mag_concat_qdir provides the direction Q tail to the magnitude branch
input; if magnitude saw raw E[Q_dir] while c51_loss saw centered, the
trunk would learn a mixed contract. Atomic migration prevents that.

Thompson direction-selection: the conviction quantity that drives the
picked-direction probability now matches the Bellman target's V/A
decomposition. Without this migration, action selection would respond
to V (the very pathology SP17 is fixing).

Per feedback_wire_everything_up: the migration is complete; no kernel
in the cuda_pipeline reads raw advantage logits.

Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```

---

## Phase 2 — Behavioral test suite (V/A separation)

These are oracle tests that probe the **architectural contract**, not just the math. They run on synthetic markets without dispatching to L40S.

### Task 2.1: V-invariance test

> _"Identical state, varying available actions → V should be invariant."_ But our action space is fixed; the analogue here is: **identical state across two consecutive bars, varying advantage signal → V trajectory should reflect state-value evolution, A trajectory should reflect action-relevant change.**

**Files:**
- Modify: `crates/ml/tests/sp17_dueling_oracle_tests.rs` — add `v_invariance_under_action_perturbation`

- [ ] **Step 1: Write failing test**

```rust
#[test]
#[ignore = "requires GPU"]
fn v_invariance_under_action_perturbation() {
    // Setup: synthetic state s with v_logits = [v_z for each atom].
    // Build TWO advantage tensors A1, A2 such that (A1 - mean A1) ≠ (A2 - mean A2)
    //   but (mean A1 = mean A2) → V should give identical E[V] for both.
    // Verify: E[V] (read from the v-only diagnostic readout) is identical to ε=1e-5
    //   across both A1 and A2 forward passes through compute_expected_q.
    unimplemented!();
}
```

- [ ] **Step 2-4: Implement, run, commit.**

```bash
git commit -m "$(cat <<'EOF'
test(sp17): V-invariance behavioral test

Verifies the architectural contract: V depends only on state, not on
the specific advantage tensor. Two A tensors with same mean produce
the same V; non-equivalent A tensors are distinguishable only via
A_centered.

This is the structural property that makes dueling work — without it,
the model conflates "state value" with "action value" and gradient
updates corrupt V via raw-A noise.

Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF
)"
```

### Task 2.2: A-invariance test

> _"Same action, varying market state → A should be invariant."_ The analogue: A_centered must respond only to the action-relevant relative ordering, not to absolute V scale.

- [ ] **Step 1: Write failing test**

```rust
#[test]
#[ignore = "requires GPU"]
fn a_centered_invariance_under_v_shift() {
    // Two states s1, s2 differ only in V scale: v_logits_s2 = v_logits_s1 + 10.0
    // Same advantage tensor A applied to both.
    // Verify: A_centered_s1 == A_centered_s2 to ε=1e-5 (the centering is
    //   computed from A only — the V shift cannot leak in).
    unimplemented!();
}
```

- [ ] **Step 2-4: Implement, run, commit.**

### Task 2.3: Symmetric A → uniform action distribution

> _"If A is identically zero across actions, action selection should be uniform-random under Thompson."_

- [ ] **Step 1: Write failing test**

```rust
#[test]
#[ignore = "requires GPU"]
fn uniform_a_yields_uniform_thompson_dir_selection() {
    // A = 0 everywhere across (action, atom). V arbitrary.
    // Run Thompson direction selection N=10000 times under different
    // Philox seeds. Assert: per-direction frequency ∈ [0.20, 0.30] for
    // each of the 4 directions (uniform = 0.25 ± 5pp Wilson interval).
    unimplemented!();
}
```

- [ ] **Step 2-4: Implement, run, commit.**

### Task 2.4: Asymmetric A → deterministic argmax under low Thompson temp

- [ ] **Step 1: Write failing test**

```rust
#[test]
#[ignore = "requires GPU"]
fn asymmetric_a_yields_deterministic_argmax() {
    // A_centered for direction = [0.0, 0.0, 1.0, 0.0]  (Long dominates)
    // V arbitrary. Thompson temp = 0.0 → pure argmax E[Q].
    // Assert: pick = DIR_LONG in 100% of N=1000 runs.
    unimplemented!();
}
```

- [ ] **Step 2-4: Implement, run, commit.**

---

## Phase 3 — HEALTH_DIAG diagnostic emit + adaptive `advantage_clip_bound` producer

(Only if **DD7=a**. If **DD7=b**, this phase is folded into Phase 5 as a HEALTH_DIAG-only readout with no producer.)

### Task 3.1: A_var_ema producer kernel + ISV writes

**Files:**
- New: `crates/ml/src/cuda_pipeline/sp17_va_diagnostic_kernel.cu`
- Modify: `crates/ml/build.rs` to register cubin
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — load + launch site
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — HEALTH_DIAG readout

- [ ] **Step 1: Write failing test**

```rust
#[test]
#[ignore = "requires GPU"]
fn a_var_ema_per_branch_tracks_centered_variance() {
    // Synthetic A_centered with known per-branch variances.
    // Launch sp17_va_diagnostic_kernel.
    // Assert ISV[A_VAR_EMA_*] readback matches expected variance per branch
    //   to ε=1e-5 after Pearl-A first-observation bootstrap.
    unimplemented!();
}
```

- [ ] **Step 2: Implement kernel** — single-thread cold-path (mirrors `min_hold_temperature_update_kernel` design — one launch per epoch, 4 branches × per-sample-per-atom variance Welford with Pearl-A first-observation bootstrap on `A_VAR_EMA_*` slots)

- [ ] **Step 3: Wire launcher into per-epoch HEALTH_DIAG emit path**

- [ ] **Step 4: Run, commit.**

### Task 3.2: V_share producer + adaptive clip bound

`V_share[branch] = |E[V]| / (|E[V]| + |E[A_centered, picked_action]|)` per branch. Read at HEALTH_DIAG.

`advantage_clip_bound`: `p99(|A_centered|) × 1.5` produced by `sp4_histogram_p99` over the per-batch flattened `|A_centered|` tensor (no atomicAdd per `pearl_fused_per_group_statistics_oracle`).

- [ ] **Steps 1-4: Failing test → implement → run → commit.**

### Task 3.3: HEALTH_DIAG full V/A trajectory emit

After the producers land:

```
HEALTH_DIAG[N]: dueling [v_share=(d=X m=Y o=Z u=W)] [a_var=(d=A m=B o=C u=D)] [clip=K]
```

Reader emits 4 + 4 + 1 = 9 slots per epoch. The reviewer can detect post-deploy whether identifiability is doing real work (high a_var, balanced v_share) or regressing (a_var → 0, v_share → 1.0).

- [ ] **Steps 1-4: Failing test → implement → run → commit.**

---

## Phase 4 — 5-epoch L40S smoke validation

### Task 4.1: Push + dispatch + monitor

**This task is reviewer-executed (not subagent).**

- [ ] **Step 1: Push final SP17 branch**

```bash
git push origin feat/sp17-dueling
```

- [ ] **Step 2: Dispatch L40S 5-epoch smoke**

```bash
./scripts/argo-train.sh --model dqn --epochs 5 --label sp17-smoke-v1
```

- [ ] **Step 3: Smoke gate criteria**

| Gate | Threshold | Source |
|---|---|---|
| Q(Flat) − Q(Long) gap trajectory | NOT widening monotonically Fold 0→Fold 1 | new dueling HEALTH_DIAG + existing q_by_action |
| dir_entropy Fold 1 final | ≥ 0.75 (b5gmp baseline 0.70) | existing |
| Hold% Fold 1 final | ≤ 50% (b5gmp baseline 52.6%) | existing |
| a_var per branch | > 0.01 throughout (a_var → 0 = "A is doing nothing" failure mode) | new |
| v_share per branch | ∈ [0.3, 0.7] throughout (drift outside = imbalance) | new |
| val Sharpe Fold 1 final | ≥ 60 (b5gmp 55.55, pfh9n 64.24) | existing |
| advantage_clip_bound trajectory | settled within [0.5, 5.0] by epoch 3 | new |
| no NaN / no train hang | always | existing |

- [ ] **Step 4: Document outcome**

Append "Phase 4 smoke evidence" subsection to this plan.

---

## Phase 5 — 30-epoch full validation (post-smoke pass)

### Task 5.1: Full dispatch + analysis

**Reviewer-executed.**

- [ ] **Step 1: 30-epoch dispatch**

```bash
./scripts/argo-train.sh --model dqn --epochs 30 --label sp17-full-v1
```

- [ ] **Step 2: Full gate criteria**

| Gate | Threshold |
|---|---|
| val Sharpe across 5 folds | mean ≥ 65, no fold < 50 |
| Hold% trajectory across folds | stable (not climbing fold-over-fold) |
| Q(Flat) − Q(Long) gap | bounded throughout (no monotonic widening) |
| dir_entropy stability | stays ∈ [0.7, 1.1] across folds |
| a_var per branch | bounded > 0 throughout |

- [ ] **Step 3: Pass/fail decision + memory pearl land**

If pass: land `pearl_dueling_identifiability_for_multi_sink_q_bias.md` to memory. If fail: open SP18 follow-up plan.

---

## Self-review checklist

- [ ] Every task is bite-sized TDD: failing test → implement → pass → commit.
- [ ] No task references `todo!()` outside of an explicit "this lands later" marker.
- [ ] No `// ...` ellipses in code blocks. Where I omit context I cite the file:line that contains the surrounding code.
- [ ] Phase 1 atomic-consumer-migration is explicit: 4 commits, all in same branch, NO L40S dispatch between them.
- [ ] All adaptive bounds are ISV-driven (advantage_clip_bound producer-tracked, not hardcoded).
- [ ] All reductions use block tree-reduce or sample-local register reduction (no atomicAdd).
- [ ] All CPU↔GPU transfers use `MappedF32Buffer` (tests not exempt).
- [ ] Pearl candidates flagged at end of each phase for human review/landing.
- [ ] Kill criterion in Phase 0 is concrete and disqualifying (not aspirational).
- [ ] Design decisions are explicit and reviewer-actionable (DD1–DD7 above).

---

## Memory pearl candidates

(Drafts for the human reviewer to land or reject after each phase.)

### Pearl 1 (post-Phase 1): `pearl_dueling_identifiability_atomic_consumer_migration.md`

> When the V/A aggregation contract changes, every consumer (forward Bellman, gradient backprop, action selection, conditional inputs to other branches) migrates atomically. Partial migration creates a strict-worse state where action selection sees one V/A contract and Bellman target sees another — disagreement at the multi-consumer boundary that didn't exist before the change.

### Pearl 2 (post-Phase 4 — IF smoke passes): `pearl_dueling_identifiability_for_multi_sink_q_bias.md`

> Multi-sink Q-bias (Hold% pressure migrates Q-bias from Q(Hold) to Q(Flat) — the SP15→SP16→b5gmp pattern) is solved by mean-zero identifiability `Q(s,a) = V(s) + (A(s,a) − mean_a A(s,a))`. Without identifiability, gradient updates split arbitrarily between V and A; per-action cost shaping pushes one sink down but the model migrates V upward and a different action becomes the new sink. Mean-zero forces the action-selection signal entirely into A by construction; no migration possible.

### Pearl 3 (post-Phase 4 — IF smoke fails): `pearl_dueling_alone_does_not_break_q_bias.md`

> Mean-zero identifiability is necessary but not sufficient when Q-bias has *non-architectural* origin (Adam optimizer asymmetry, reward asymmetry, bootstrap target inflation). If Phase 4 smoke shows a_var > 0 and v_share balanced but Q(Flat) bias persists, the structural fix didn't address root cause; investigate Q-target inflation (Phase 4 SP18 candidate) or per-action Adam state separation.

---

## Execution handoff

Total estimated commits: **27 tasks × 1 commit each = 27 commits across feat/sp17-dueling**.

Estimated wall time:
- Pre-Phase + Phase 0 (diagnostic + KILL CRITERION): ~1 day with 1-epoch L40S dispatch
- Phases 1-3 (consumer migration + tests): ~2-3 days
- Phase 4 smoke: ~6 hours dispatch + analysis
- Phase 5 full: ~24 hours dispatch + analysis

Reviewer must:
1. Resolve **DD1–DD7** before subagent dispatch.
2. Run Pre-Phase Task PP.1 (worktree creation) manually.
3. Run all `argo-train.sh` dispatches manually (Phases 0, 4, 5).
4. Land/reject memory pearls at end of each phase.

Subagent must:
1. Hold to atomic-consumer-migration discipline in Phase 1 (Tasks 1.2–1.5 land in same branch before any L40S dispatch).
2. Stop and ask if any kernel under modification has a graph-capture interaction not visible in the consumer audit (per `pearl_no_host_branches_in_captured_graph` — the per-atom mean reduction is pure-device, but new kernels in Phase 3 must be audited).
3. Stop and ask if Phase 0 KILL CRITERION returns INCONCLUSIVE.
