From 509035dea653c54c53e1660f8ebfe63de36744f3 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 8 May 2026 20:57:28 +0200 Subject: [PATCH] docs(sp17): copy plan file to feat/sp17-dueling branch Plan was authored on sister worktree (sp15-phase1-honest-numbers) by the SP17 planner agent; copying it to feat/sp17-dueling so future task references in commit messages and audit-doc entries resolve from this branch. No content change vs sister-worktree copy. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../2026-05-08-sp17-dueling-q-network.md | 1191 +++++++++++++++++ 1 file changed, 1191 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md diff --git a/docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md b/docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md new file mode 100644 index 000000000..c2140ecff --- /dev/null +++ b/docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md @@ -0,0 +1,1191 @@ +# SP17 — Dueling Q-Network: Identifiable V/A Decomposition + +> **For agentic workers:** REQUIRED SUB-SKILL: `superpowers:subagent-driven-development`. Steps use checkbox `- [ ]` syntax. Each task is bite-sized TDD: failing test → run-fail → implement → run-pass → commit. + +> **Status: DRAFT FOR HUMAN REVIEW.** Several genuine design choices are flagged with **DESIGN DECISION:** below. Please redirect or confirm before any subagent dispatch. + +--- + +## Goal + +Add the **identifiability constraint** to the existing distributional dueling Q-architecture so action selection responds to relative advantage `A(s,a) − mean_a A(s,a)`, not absolute state value `V(s)`. This is the structural fix for the multi-sink Q-bias that train-multi-seed-b5gmp (sha 641aa0dfd) exposed. + +**Concretely:** + +``` +PRE-SP17: Q[a, z] = V_logit[z] + A_logit[a, z] (additive, not identifiable) +POST-SP17: Q[a, z] = V_logit[z] + (A_logit[a, z] − mean_a A_logit[a, z]) (mean-zero centered) +``` + +--- + +## Critical pre-discovery (load-bearing — read before approving) + +The codebase already has structurally separate value and advantage heads: + +| Buffer | Shape | Source | Purpose | +|---|---|---|---| +| `on_v_logits_buf` | `[B, NA]` | shared value head (cuBLAS pass) | per-atom value distribution `Z_V[z]` | +| `on_b_logits_buf` | `[B, (B0+B1+B2+B3)*NA]` | per-action advantage head (cuBLAS pass) | per-action atom logits `Z_A[a,z]` | + +`compute_expected_q` (`crates/ml/src/cuda_pipeline/experience_kernels.cu:4499`) explicitly comments **"combines value + advantage logits via dueling aggregation"**, but the aggregation is naive addition — no `A − mean A` subtraction anywhere. Same naive addition in: + +- `compute_expected_q` (line 4583): `float logit = v_row[z] + adv_a[z];` +- `mag_concat_qdir` (line 5072): `float logit = dir_logits_v[b * NA + z] + dir_logits_b[…]` +- `c51_loss_kernel.cu` (block_bellman_project), `c51_grad_kernel.cu` — same pattern + +**This is the SP17 surgical insertion point.** We are not building dueling; we are completing the identifiability term that was either never written or got dropped. The required surgery is small, isolated, and atomic across all consumers (per `feedback_no_partial_refactor`). + +This dramatically simplifies SP17 vs. the original sketch: we are **not** adding new Q-head parameters, **not** splitting into separate V/A heads, **not** allocating new param tensors or Adam state. We are inserting a per-sample per-branch reduction into existing kernels. + +--- + +## Background — what train-multi-seed-b5gmp proved + +Fold 1 final on commit 641aa0dfd (post full SP16 chain): + +| Metric | Fold 0 final | Fold 1 final | pfh9n baseline (Fold 1) | +|---|---|---|---| +| val Sharpe | 81.42 | 55.55 | 64.24 | +| Hold% | 38.1% | 52.6% | 45.5% | +| dir_entropy | 0.81 | 0.70 | 0.715 | +| Q(Flat) − Q(Long) gap | 0.060 | **0.148** | n/a (not measured) | +| Q-mean Fold 1 epoch trajectory | n/a | **0.258 → 0.412 (+60%)** | n/a | +| Q(Flat) Fold 1 epoch trajectory | n/a | **0.358 → 0.484 (+35%)** | n/a | + +Three structural failures, in priority order: + +1. **(parallel fix in flight)** Wiener-α stationarity violation +2. **Multi-sink Q-bias.** Hold% pressure migrated the bias from Q(Hold) to Q(Flat). The model selects whichever zero-variance "do nothing" sink dominates. Per-bar Hold cost (SP16) attacks one sink; the bias jumps to the other. Without identifiability, the *next* fix would just push it to the third sink. +3. **Bellman target inflation.** Q-mean climbed +60% on Fold 1 with flat train_loss → optimistic bootstrap on non-action states (Hold/Flat). Same root cause: V(s) leaks into the action-selection signal. + +The literature mechanism (Wang et al. 2016 Dueling DQN) is exact: when `Q = V + A` lacks identifiability, gradient updates on `Q(s,a)` are split arbitrarily between `V` and `A`, and the network drifts toward "explain everything via V." Mean-zero `A` forces the action-relevant signal into `A` by construction. + +--- + +## Out of scope + +- New separate V/A *parameter* heads (the heads already exist — we modify aggregation only) +- Replacing C51 with non-distributional Q (orthogonal; we keep C51-distributional dueling) +- IQN dueling (separate change — flagged in **DESIGN DECISION 5** below) +- Reward shaping (SP11/SP16 territory — handled elsewhere) + +--- + +## Design decisions (REQUIRES HUMAN REVIEW) + +### DESIGN DECISION 1: Mean-zero vs. max-zero identifiability + +Wang et al. 2016 propose two forms: + +``` +mean-zero: Q(s,a) = V(s) + (A(s,a) − (1/|A|) · Σ_a A(s,a)) +max-zero: Q(s,a) = V(s) + (A(s,a) − max_a A(s,a)) +``` + +**Recommendation:** **mean-zero per branch.** Wang's Fig. 1 ablation shows mean-zero is more stable; max-zero ties one A logit to the V head and creates non-smooth gradient. Mean-zero is a smooth differentiable reduction over |branch_size| values (4, 3, 3, 3) — trivial to implement in the existing softmax kernels. + +> _Human reviewer: confirm mean-zero, or pick max-zero._ + +### DESIGN DECISION 2: C51 + dueling — atom-level vs. expectation-level mean + +Two possible loci of the `A − mean A` subtraction: + +``` +(A) atom-level: Z_A[a, z] − mean_a Z_A[a, z] (subtract per atom z, before softmax) +(B) expectation-level: Z_A[a, z] − mean_a E[Z_A[a, z]] (subtract a scalar, equal across atoms) +``` + +**Recommendation:** **(A) atom-level.** Subtracting per atom preserves the distributional shape per action (the atom-level identifiability constraint Wang & Rajeswar 2018 derive for distributional dueling). Option (B) would couple the atom shape to the action-mean expectation, which collapses distributional information. + +Operationally both are cheap. Atom-level: `Z_A[a, z] -= (1/|A|) Σ_a Z_A[a, z]` per (sample, atom) — one reduction per atom. Expectation-level: per-action E[Z_A] then collapse — adds one full softmax-expectation pass. + +> _Human reviewer: confirm atom-level mean-zero._ + +### DESIGN DECISION 3: Scope — direction-only or all four branches + +Branches: dir(4), mag(3), ord(3), urg(3). Each branch already has separately-allocated advantage logits in `b_logits` BRANCH-MAJOR layout (see `experience_kernels.cu:4527-4531`). + +**Recommendation:** **all four branches.** The Q-bias evidence is direction-specific (Q(Flat) vs Q(Long)), but applying identifiability to one branch and not the others creates the partial-refactor pathology (`feedback_no_partial_refactor`): three consumers reading the old contract, one reading the new, and they disagree more than they did before. Per-branch mean is independent (each branch's mean over its own |branch| actions), so the implementation cost scales linearly. + +The shared V head feeds all four branches anyway — once V is identifiable for direction, the magnitude/order/urgency logits should be centered for the same architectural reason or risk re-introducing the same pathology in those subspaces. + +> _Human reviewer: confirm all-4-branch scope._ + +### DESIGN DECISION 4: Target network — also identifiable + +When the Bellman target is computed, the target net's `Q_target = V_target + A_target` must use the **same** identifiability projection as the online net, or training/target asymmetry will create chronic Bellman-residual bias. + +**Recommendation:** **target net uses same projection.** No alternative is structurally sound — they are the same architecture, target is a delayed copy. Single helper function consumed by both. + +> _Human reviewer: confirm._ + +### DESIGN DECISION 5: IQN dueling — defer to follow-up + +The trainer also runs IQN (`iqn_dual_head_kernel.cu`, `iqn_cvar_kernel.cu`, `iqn_quantile_ema_kernel.cu`). IQN does not currently use the V/A decomposition (it's a single quantile head per action). + +**Recommendation:** **defer IQN dueling.** Add as SP18 follow-up. The C51 Q-distribution is what `compute_expected_q` produces and what feeds Thompson selection; IQN's contribution to `q_eff` is via a separate τ-blend pathway (`pearl_per_branch_iqn_tau_schedule`). If C51 dueling fixes the multi-sink bias the IQN treatment becomes a Phase 5+ optimization, not a structural fix. Mark as "out of scope, may revisit." + +> _Human reviewer: confirm defer, or fold IQN dueling into SP17._ + +### DESIGN DECISION 6: V-head shared across branches vs. per-branch + +The current architecture has **one** value head `v_logits ∈ [B, NA]` shared across all four action branches. Wang's original paper has one V-head for one action space; here we have a multi-branch action space with one V-head shared. + +This is sound: V(s) is a state property and the four branches all act on the same state. Per-branch V would be saying "the value of this state for the dir-branch is different from the value for the mag-branch" — which violates the Markov state-value definition. + +**Recommendation:** **keep shared V-head. No change.** + +> _Human reviewer: confirm._ + +### DESIGN DECISION 7: ISV slots for diagnostics + +To detect whether identifiability is doing work post-deployment, we want HEALTH_DIAG observables: + +- `A_var_ema` per branch (variance of mean-centered A across actions) — should be > 0 if A is doing real work +- `V_share` per branch (fraction of Q-magnitude attributable to V vs. A) — diagnostic for "V is dominating" pathology +- `advantage_clip_bound` — adaptive clamp on `|A_centered|` to bound runaway A-tails per `feedback_isv_for_adaptive_bounds` + +**Recommendation:** **9 new ISV slots** at `[474..483)` (A_var_ema × 4 branches + V_share × 4 branches + advantage_clip_bound × 1). Per `feedback_isv_for_adaptive_bounds`, the bound is signal-driven (tracks p99 of `|A_centered|` × safety_factor, not a hardcoded scalar). + +Alternative: **0 new slots, diagnostic-only via existing per-branch Q-mean infrastructure.** Mean-centering is mathematically rigid (no adaptation knob); A_var/V_share can be computed at HEALTH_DIAG cadence on existing buffers without a slot. + +> _Human reviewer: pick (a) 9 slots with adaptive clip + diagnostics, or (b) 0 slots, diagnostics-only computed at HEALTH_DIAG. The slot allocation is the architecturally cleaner choice but adds wire-up complexity. Recommend (a)._ + +--- + +## Phase ordering (deviates from sketch) + +The original sketch had Phase 0 (diagnostic) → Phase 1 (split heads) → Phase 2 (behavioral tests) → Phase 3 (integrate) → Phase 4 (smoke) → Phase 5 (full validation). + +**Revised ordering** (heads already exist, integration is small): + +| Phase | Purpose | Roughly bite-sized tasks | +|---|---|---| +| Pre-Phase | Branch + worktree + ISV slots + V/A diagnostic emit | 3 | +| Phase 0 | V/A decomposition diagnostic — confirm Q(Flat) over-attribution is V-driven (KILL CRITERION) | 3 | +| Phase 1 | Mean-zero identifiability in `compute_expected_q` (online + target) — atomic across consumers | 5 | +| Phase 2 | Mean-zero identifiability in `c51_loss_kernel` + `c51_grad_kernel` (training path) | 4 | +| Phase 3 | Mean-zero identifiability in `mag_concat_qdir` + Thompson direction-selection consumers | 3 | +| Phase 4 | Behavioral test suite (synthetic markets — V/A separation oracle tests) | 4 | +| Phase 5 | HEALTH_DIAG diagnostic emit + adaptive `advantage_clip_bound` producer | 3 | +| Phase 6 | 5-epoch L40S smoke validation | 1 | +| Phase 7 | 30-epoch full validation post-smoke-pass | 1 | + +Total: **27 tasks**. Each is a single TDD cycle. + +The sketch's "Phase 1 split current Q-head into V-head + A-head" doesn't apply — heads are already split. The actual work is the identifiability projection plus its atomic consumer migration. + +--- + +## File Structure + +``` +docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md ← this file +crates/ml/src/cuda_pipeline/sp14_isv_slots.rs ← +9 SP17 ISV slots OR no change (DD7) +crates/ml/src/cuda_pipeline/state_layout.cuh ← C-side mirror (only if DD7=a) +crates/ml/src/cuda_pipeline/experience_kernels.cu ← `compute_expected_q`, `mag_concat_qdir`, action select +crates/ml/src/cuda_pipeline/c51_loss_kernel.cu ← block_bellman_project + softmax over (V+A) +crates/ml/src/cuda_pipeline/c51_grad_kernel.cu ← grad path through identifiability +crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs ← optional new diagnostic readers, ISV registration +crates/ml/src/trainers/dqn/state_reset_registry.rs ← SP17 ISV slots fold-reset entries (if DD7=a) +crates/ml/src/trainers/dqn/trainer/training_loop.rs ← HEALTH_DIAG emit for V/A diagnostics +crates/ml/tests/sp17_dueling_oracle_tests.rs ← NEW test file, mirrors sp14_oracle_tests.rs pattern +crates/ml/build.rs ← register new test cubins (only if new kernels) +bin/fxt/src/main.rs ← no change (no new CLI flags) +``` + +--- + +## Hard rules (apply to every task in this plan) + +- **No CPU compute.** All V/A reductions GPU-side. Tests use `MappedF32Buffer` — no `unsafe { … memcpy }` outside mapped-pinned (`feedback_no_htod_htoh_only_mapped_pinned`). +- **No atomicAdd.** Per-sample per-branch mean uses block tree-reduce or per-thread sequential reduction (each thread owns one sample, sums |branch| values from registers — no cross-thread reduction needed). +- **No partial refactor.** When `compute_expected_q` is migrated to centered-A, the same commit migrates `c51_loss_kernel`, `c51_grad_kernel`, and `mag_concat_qdir`. Phases 1–3 are scoped per-consumer but **must land within a single feature branch's commit graph before any L40S dispatch.** A partially-migrated state where some kernels read centered-A and others read raw-A is strictly worse than no change at all — disagreement at the multi-consumer boundary. +- **Wire-up audit per task.** No orphan changes (`feedback_wire_everything_up`). If a task adds a buffer/kernel/slot, the same commit wires it to its consumer. +- **ISV-driven adaptive bounds.** `advantage_clip_bound` (if **DD7=a**) is signal-driven (tracks `p99(|A_centered|) × safety_factor`), never hardcoded (`feedback_isv_for_adaptive_bounds`). +- **Use `sp4_histogram_p99.cuh`** for percentile computations (per `pearl_fused_per_group_statistics_oracle`). +- **Pearl candidate per phase.** Each phase ends with a memory-pearl draft for the human reviewer to land or reject. + +--- + +## Pre-Phase: Branch + worktree + ISV slot allocation + +**Why first:** Set up isolation per `feedback_no_concurrent_agents_shared_tree`; allocate the ISV slots so subsequent phases can reference final indices. + +### Task PP.1: Create worktree (do not create — included for reviewer) + +```bash +# Run from main repo root; do not run as part of this plan dispatch. +cd /home/jgrusewski/Work/foxhunt +git worktree add .worktrees/sp17-dueling-q-network -b feat/sp17-dueling main +cd .worktrees/sp17-dueling-q-network +``` + +- [ ] **Step 1: Reviewer runs the above before subagent dispatch.** + +No tests, no commits — the worktree command is an environmental setup the reviewer performs. + +### Task PP.2: Allocate 9 SP17 ISV slots [474..483) — IF DD7=a + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/sp14_isv_slots.rs` — new section at end +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — bump `ISV_TOTAL_DIM` 474 → 483, update layout fingerprint seed +- Modify: `crates/ml/src/cuda_pipeline/state_layout.cuh` — mirror constant if applicable +- Test: `crates/ml/src/cuda_pipeline/sp14_isv_slots.rs` — new `#[test]` `sp17_slot_layout_locked` + +- [ ] **Step 1: Write the failing test** (append to existing `mod tests` in sp14_isv_slots.rs) + +```rust +/// Lock SP17 (2026-05-08) dueling-Q identifiability diagnostic slot layout. +/// 9 contiguous slots [474..483) — A_var_ema per branch (4) + V_share +/// per branch (4) + adaptive advantage_clip_bound (1). +#[test] +fn sp17_dueling_slot_layout_locked() { + assert_eq!(SP17_DUELING_SLOT_BASE, 474); + assert_eq!(SP17_DUELING_SLOT_END, 483); + assert_eq!(A_VAR_EMA_DIR_INDEX, 474); + assert_eq!(A_VAR_EMA_MAG_INDEX, 475); + assert_eq!(A_VAR_EMA_ORD_INDEX, 476); + assert_eq!(A_VAR_EMA_URG_INDEX, 477); + assert_eq!(V_SHARE_DIR_INDEX, 478); + assert_eq!(V_SHARE_MAG_INDEX, 479); + assert_eq!(V_SHARE_ORD_INDEX, 480); + assert_eq!(V_SHARE_URG_INDEX, 481); + assert_eq!(ADVANTAGE_CLIP_BOUND_INDEX, 482); + // Pearl-A bootstrap sentinels. + assert_eq!(SENTINEL_A_VAR_EMA, 0.0); + assert_eq!(SENTINEL_V_SHARE, 0.5); // initial 50/50 split + assert_eq!(SENTINEL_ADVANTAGE_CLIP_BOUND, 1.0); // 1.0 = "no effective clipping" cold-start + // Category-1 dimensional safety bounds per `feedback_isv_for_adaptive_bounds`. + assert_eq!(ADVANTAGE_CLIP_BOUND_MIN, 0.1); + assert_eq!(ADVANTAGE_CLIP_BOUND_MAX, 100.0); +} + +#[test] +fn all_sp17_slots_fit_within_isv_total_dim() { + use crate::cuda_pipeline::gpu_dqn_trainer::ISV_TOTAL_DIM; + assert!( + SP17_DUELING_SLOT_END <= ISV_TOTAL_DIM, + "SP17_DUELING_SLOT_END={} exceeds ISV_TOTAL_DIM={}; \ + bump ISV_TOTAL_DIM in gpu_dqn_trainer.rs (and update layout_fingerprint_seed()).", + SP17_DUELING_SLOT_END, ISV_TOTAL_DIM, + ); +} +``` + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib sp17_dueling_slot_layout_locked all_sp17_slots_fit_within_isv_total_dim` +Expected: FAIL — constants undefined. + +- [ ] **Step 2: Add SP17 ISV section to sp14_isv_slots.rs** (append after SP16 T3 block at line 614) + +```rust +// ── SP17 (2026-05-08): dueling Q-network identifiability diagnostics ─── +// Adds 9 ISV slots driving the V/A diagnostic chain that observes +// whether the post-SP17 mean-centered advantage is doing real work and +// detects regression to the pre-SP17 V-dominated regime. Per +// `feedback_isv_for_adaptive_bounds`, all signal-driven; the +// `advantage_clip_bound` is producer-tracked from p99(|A_centered|), +// never hardcoded. +// +// Mathematical rationale: with mean-zero identifiability +// `A_centered[a, z] = A[a, z] − (1/|branch|) Σ_a A[a, z]`, the per-branch +// variance of `A_centered` over actions measures how much the policy +// discriminates between actions. Var(A) → 0 = "all actions equivalent, +// V dominates" = pre-SP17 pathology re-emerging. Var(A) > 0 with V_share +// stable around 0.5 = healthy dueling decomposition. +// +// Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md +pub const A_VAR_EMA_DIR_INDEX: usize = 474; // Var(A_centered) over dir actions, EMA +pub const A_VAR_EMA_MAG_INDEX: usize = 475; +pub const A_VAR_EMA_ORD_INDEX: usize = 476; +pub const A_VAR_EMA_URG_INDEX: usize = 477; +pub const V_SHARE_DIR_INDEX: usize = 478; // |V_contribution| / (|V| + |A_centered|) +pub const V_SHARE_MAG_INDEX: usize = 479; +pub const V_SHARE_ORD_INDEX: usize = 480; +pub const V_SHARE_URG_INDEX: usize = 481; +pub const ADVANTAGE_CLIP_BOUND_INDEX: usize = 482; // adaptive |A_centered| upper bound + +// Sentinels — Pearl-A first-observation bootstrap. +pub const SENTINEL_A_VAR_EMA: f32 = 0.0; +pub const SENTINEL_V_SHARE: f32 = 0.5; // healthy 50/50 split as cold-start +pub const SENTINEL_ADVANTAGE_CLIP_BOUND: f32 = 1.0; // 1.0 → no effective clipping pre-first-obs + +// Category-1 dimensional safety bounds per `feedback_isv_for_adaptive_bounds`. +// Floor 0.1: below this any centered advantage is essentially noise; clip +// would erase the action signal. Ceiling 100.0: above this a single +// adversarial outlier can dominate per-batch EMAs. +pub const ADVANTAGE_CLIP_BOUND_MIN: f32 = 0.1; +pub const ADVANTAGE_CLIP_BOUND_MAX: f32 = 100.0; + +// Producer constants. Safety factor 1.5× p99 mirrors the SP14 P0-A +// REWARD_POS_CAP producer pattern. EMA α slow per-fold cadence (the bound +// shouldn't track per-batch noise). +pub const ADVANTAGE_CLIP_SAFETY_FACTOR: f32 = 1.5; +pub const ADVANTAGE_CLIP_EMA_ALPHA: f32 = 0.01; + +pub const SP17_DUELING_SLOT_BASE: usize = 474; +pub const SP17_DUELING_SLOT_END: usize = 483; +``` + +- [ ] **Step 3: Bump `ISV_TOTAL_DIM` in `gpu_dqn_trainer.rs`** at line 2523 + +Change `pub(crate) const ISV_TOTAL_DIM: usize = 474;` to `483`. Append the SP17 explanation at the end of the existing docstring chain (mirror the SP16 T3 pattern at the top of the comment block — short paragraph naming the slots and the producer kernel). + +- [ ] **Step 4: Update layout fingerprint seed** at line 3605 + +Change `ISV_TOTAL_DIM=474;` to `ISV_TOTAL_DIM=483;` and append the new slot names to the seed string in the same `b""` literal: + +``` +SLOT_474_A_VAR_EMA_DIR=474;\ +SLOT_475_A_VAR_EMA_MAG=475;\ +SLOT_476_A_VAR_EMA_ORD=476;\ +SLOT_477_A_VAR_EMA_URG=477;\ +SLOT_478_V_SHARE_DIR=478;\ +SLOT_479_V_SHARE_MAG=479;\ +SLOT_480_V_SHARE_ORD=480;\ +SLOT_481_V_SHARE_URG=481;\ +SLOT_482_ADVANTAGE_CLIP_BOUND=482;\ +ISV_TOTAL_DIM=483;\ +``` + +- [ ] **Step 5: Add fold-reset registry entries** in `state_reset_registry.rs` + +For each of the 9 slots, add a `RegistryEntry { name: "sp17_", category: ResetCategory::FoldReset, description: "..." }` entry matching the existing SP14 Layer C pattern at the end of the registry's `new()` method. + +- [ ] **Step 6: Run test** — Expected: PASS. + +```bash +SQLX_OFFLINE=true cargo test -p ml --lib sp17_dueling_slot_layout_locked all_sp17_slots_fit_within_isv_total_dim +``` + +- [ ] **Step 7: Commit** + +```bash +git add -A +git commit -m "$(cat <<'EOF' +plan(sp17): allocate ISV slots [474..483) for dueling-Q diagnostics + +9 slots: A_var_ema × 4 branches (per-branch advantage variance EMA) ++ V_share × 4 branches (V/(V+A) magnitude share) + adaptive +advantage_clip_bound. All Pearl-A first-observation bootstrap with +fold-reset registry entries. Bump ISV_TOTAL_DIM 474 → 483 + layout +fingerprint seed. + +Per `feedback_isv_for_adaptive_bounds`: advantage_clip_bound is +producer-tracked from p99(|A_centered|) × 1.5 safety factor, never +hardcoded. Bounds [0.1, 100.0] are Category-1 dimensional safety floors. + +No consumer change in this commit (additive infrastructure). Phase 1 +mean-centering producer (atomic across compute_expected_q + +c51_loss + c51_grad + mag_concat_qdir) lands in the next 4 commits. + +Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +> _If reviewer picks **DD7=b**, this entire task is skipped and the diagnostic emit in Phase 5 is computed at HEALTH_DIAG cadence on existing buffers without new slots._ + +### Task PP.3: HEALTH_DIAG emit for raw V/A means (PRE-CENTERING — sets the baseline) + +**Files:** +- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — find existing per-epoch HEALTH_DIAG block, add `v_a_means` line +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — add `read_v_a_means_pre_centering()` reading from existing `on_v_logits_buf` + `on_b_logits_buf` +- Test: `crates/ml/tests/sp17_dueling_oracle_tests.rs` — NEW file with `v_a_means_pre_centering_emits_expected_layout` + +- [ ] **Step 1: Write failing test** (creates new file `crates/ml/tests/sp17_dueling_oracle_tests.rs`) + +```rust +//! SP17 dueling-Q oracle tests. +//! +//! Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md +//! +//! Layer A (CPU oracle) tests verify the math of mean-zero identifiability +//! against synthetic logit tensors without GPU dependencies. +//! +//! Layer B (GPU oracle) tests verify the kernel-level implementations +//! match the CPU oracle bit-for-bit using `MappedF32Buffer` per +//! `feedback_no_htod_htoh_only_mapped_pinned`. All B-tests are +//! `#[ignore = "requires GPU"]`. + +#[cfg(feature = "cuda")] +#[allow(unsafe_code)] // CUDA kernel launch + mapped-pinned memory. +mod gpu { + // ── A.1: V/A means readout layout sanity check ───────────────── + /// Verifies the diagnostic readout buffer for HEALTH_DIAG has the + /// expected layout: [v_mean_pre, a_mean_dir_pre, a_mean_mag_pre, + /// a_mean_ord_pre, a_mean_urg_pre]. Pre-centering + /// (Phase 0 — no centering yet applied). + #[test] + #[ignore = "requires GPU"] + fn v_a_means_pre_centering_emits_expected_layout() { + // Force a synthetic distribution where v_logits = constant 0.3 + // and per-branch A means are known [0.1, 0.05, -0.02, -0.05]. + // Launch read_v_a_means_pre_centering. Assert the readback matches. + unimplemented!( + "Phase PP.3 Step 3 lands the implementation; this test gates the wire-up" + ); + } +} +``` + +Run: `cargo test -p ml --test sp17_dueling_oracle_tests --features cuda v_a_means_pre_centering_emits_expected_layout -- --ignored --nocapture` +Expected: FAIL with "not yet implemented". + +- [ ] **Step 2: Register new test in `crates/ml/Cargo.toml`** (mirror `sp14_oracle_tests` `[[test]]` entry) + +- [ ] **Step 3: Implement `read_v_a_means_pre_centering()` in `gpu_dqn_trainer.rs`** + +Add a small block-tree-reduce kernel `v_a_means_diag_kernel` that emits 5 floats per launch: +``` +[E[V] over batch and atoms, + E[A_dir] over batch and (actions × atoms), + E[A_mag] over batch and (actions × atoms), + E[A_ord] over batch and (actions × atoms), + E[A_urg] over batch and (actions × atoms)] +``` + +Reduction is per-block warp-shuffle then block-shared-mem then a finalize kernel that sequentially sums block partials (no atomicAdd, per `feedback_no_atomicadd`). + +Allocate a `MappedF32Buffer<5>` `v_a_means_diag_buf` on `GpuDqnTrainer` for cold-path readback. + +- [ ] **Step 4: Wire `read_v_a_means_pre_centering()` into HEALTH_DIAG emit** + +In `training_loop.rs` per-epoch HEALTH_DIAG block (search for `q_by_action` or `mag` line), append: +```rust +let v_a = trainer.read_v_a_means_pre_centering(); +info!( + "HEALTH_DIAG[{epoch}]: v_a_means [v={:.4} a_dir={:.4} a_mag={:.4} a_ord={:.4} a_urg={:.4}]", + v_a[0], v_a[1], v_a[2], v_a[3], v_a[4] +); +``` + +- [ ] **Step 5: Implement test body** + +Allocate input buffers via `MappedF32Buffer::new`. Set `v_logits = 0.3` everywhere, `b_logits` with per-branch known means. Launch reader. Assert readback matches expected to ε=1e-5. + +- [ ] **Step 6: Run test** — Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add -A +git commit -m "$(cat <<'EOF' +plan(sp17): HEALTH_DIAG v_a_means baseline (PRE-CENTERING) + +Adds per-epoch readout of the V/A breakdown so we can observe the +pre-SP17 baseline distribution before the identifiability projection +lands. Also instruments the kill criterion for Phase 0: + + HEALTH_DIAG[N]: v_a_means [v=X a_dir=Y a_mag=... a_ord=... a_urg=...] + +If train-multi-seed-b5gmp's Q(Flat) over-attribution is V-driven, V +should be elevated (~0.4+) while a_dir is small. If V is balanced and +A is doing the work, dueling cannot help and we abort SP17. + +Block-tree-reduce kernel `v_a_means_diag_kernel` (no atomicAdd per +`feedback_no_atomicadd`); MappedF32Buffer per +`feedback_no_htod_htoh_only_mapped_pinned`. + +Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Phase 0 — Diagnostic verification + KILL CRITERION + +**Why first:** Confirm the V-dominance hypothesis before architectural surgery. If V is already small and A is dominant, the identifiability projection is a no-op and SP17 is dead. + +### Task 0.1: 1-epoch Fold-1 diagnostic dispatch + +- [ ] **Step 1: Push branch + dispatch 1-epoch L40S smoke** + +The reviewer (NOT the subagent) runs: + +```bash +git push -u origin feat/sp17-dueling +./scripts/argo-train.sh --model dqn --epochs 1 --label sp17-phase0-vapre +``` + +- [ ] **Step 2: Read HEALTH_DIAG output** + +Tail the `v_a_means` lines across the full Fold 1 epoch. Compute mean and trajectory of `[v, a_dir, a_mag, a_ord, a_urg]`. + +- [ ] **Step 3: KILL CRITERION** + +``` +ABORT SP17 if |v| < 0.5 × max(|a_dir|, |a_mag|, |a_ord|, |a_urg|) (V is small, A dominates) +PROCEED if |v| > max(|a_dir|, |a_mag|, |a_ord|, |a_urg|) (V dominates → identifiability will help) +INCONCLUSIVE if neither — convene human review +``` + +- [ ] **Step 4: Document result** + +Append a "Phase 0 evidence" subsection to this plan with raw numbers and decision. + +No commit unless evidence is appended. + +--- + +## Phase 1 — Mean-zero identifiability in `compute_expected_q` + +**Why this kernel first:** It is the action-selection reader (Thompson, eval). The behavior change here is observable end-to-end. + +### Task 1.1: CPU oracle test for mean-centered Q computation + +**Files:** +- Modify: `crates/ml/tests/sp17_dueling_oracle_tests.rs` — add CPU-side `compute_expected_q_centered_cpu_oracle` test + +- [ ] **Step 1: Write failing test** + +```rust +/// CPU oracle for mean-zero identifiability. +/// +/// Given v_logits = [0.5, 0.5] (NA=2 atoms) and per-branch advantage logits +/// b_logits_dir = [[0.1, 0.0], [0.3, 0.2], [0.0, 0.0], [-0.2, -0.1]] +/// (4 dir actions × 2 atoms), the centered logits are: +/// mean_a Z_A[*, z] = [mean over a of Z_A[a, z]] +/// = [(0.1+0.3+0.0-0.2)/4, (0.0+0.2+0.0-0.1)/4] +/// = [0.05, 0.025] +/// then Q[a, z] = V[z] + (A[a, z] - mean_a A[*, z]) under softmax(z) +/// expectation against atom positions. +/// +/// This test pins the math contract independent of GPU implementation. +#[test] +fn compute_expected_q_centered_cpu_oracle() { + let v: [f32; 2] = [0.5, 0.5]; + let a_raw: [[f32; 2]; 4] = [ + [0.1, 0.0], + [0.3, 0.2], + [0.0, 0.0], + [-0.2, -0.1], + ]; + // mean over a, per atom z + let mean_a_z = [ + (a_raw[0][0] + a_raw[1][0] + a_raw[2][0] + a_raw[3][0]) / 4.0, + (a_raw[0][1] + a_raw[1][1] + a_raw[2][1] + a_raw[3][1]) / 4.0, + ]; + assert!((mean_a_z[0] - 0.05).abs() < 1e-6); + assert!((mean_a_z[1] - 0.025).abs() < 1e-6); + + // Centered A + let mut a_centered = [[0.0f32; 2]; 4]; + for a in 0..4 { + for z in 0..2 { + a_centered[a][z] = a_raw[a][z] - mean_a_z[z]; + } + } + + // Per-z sum of centered A across actions == 0 (identifiability) + for z in 0..2 { + let s: f32 = (0..4).map(|a| a_centered[a][z]).sum(); + assert!(s.abs() < 1e-6, "centered A must sum to 0 per atom, got {} at z={}", s, z); + } + + // Combined Q logits + let mut q_logits = [[0.0f32; 2]; 4]; + for a in 0..4 { + for z in 0..2 { + q_logits[a][z] = v[z] + a_centered[a][z]; + } + } + + // E[Q] under softmax(z) with atom positions [0.0, 1.0] + let atoms = [0.0f32, 1.0]; + for a in 0..4 { + let m = q_logits[a][0].max(q_logits[a][1]); + let e0 = (q_logits[a][0] - m).exp(); + let e1 = (q_logits[a][1] - m).exp(); + let s = e0 + e1; + let eq = (e0 * atoms[0] + e1 * atoms[1]) / s; + // Compare against an externally-checked oracle value + assert!(eq.is_finite()); + } +} +``` + +Run: `cargo test -p ml --test sp17_dueling_oracle_tests compute_expected_q_centered_cpu_oracle` +Expected: PASS immediately (pure math). + +- [ ] **Step 2: Commit** + +```bash +git add -A +git commit -m "$(cat <<'EOF' +test(sp17): CPU oracle for mean-zero centered Q computation + +Locks the mean-zero identifiability math contract independent of GPU +implementation. Subsequent Phase 1 GPU oracle tests compare kernel +output against this contract bit-for-bit. + +Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +### Task 1.2: GPU oracle test for `compute_expected_q` with centered A + +- [ ] **Step 1: Write failing GPU test** + +```rust +/// B.1: compute_expected_q GPU implementation matches the CPU oracle +/// when DUELING_CENTERED is engaged (always engaged post-SP17). +#[test] +#[ignore = "requires GPU"] +fn compute_expected_q_centered_matches_cpu_oracle() { + // Build the same v / a_raw / atoms inputs as the CPU oracle test on + // mapped-pinned device buffers. + // Launch compute_expected_q with the post-SP17 kernel. + // Compare per-action E[Q] against the CPU oracle to ε=1e-5. + unimplemented!("Phase 1 Step 3 lands the kernel modification"); +} +``` + +Expected on first run: FAIL. + +- [ ] **Step 2: Modify `compute_expected_q` kernel** (`experience_kernels.cu` line 4499) + +Insert the per-atom mean-A reduction inside the per-branch outer loop, BEFORE the action-loop logit accumulation: + +```cuda +/* SP17 mean-zero identifiability: subtract per-atom mean of A across this branch's + * actions. The reduction is sample-local (one thread = one sample) and per-atom z, + * computed from the n_d action logits already in the per-thread inner loop's + * scope. No cross-thread reduction needed; no atomicAdd; no shared memory. + * + * Per Wang et al. 2016 Dueling Networks (mean-zero variant), this form is + * preferred over max-zero because: + * - smooth differentiable reduction (no argmax tie-breaking) + * - all actions contribute proportionally to V/A split (max-zero pins one + * action to V which creates the per-action gradient asymmetry the paper + * identifies as "less stable" in Fig. 1) + * + * Per-branch independent: dir uses dir actions only, mag uses mag actions only, + * etc. The shared V head is invariant across branches by construction (one V + * per state, all branches share). + * + * Atom-level (not expectation-level): subtract per atom z, before softmax. + * Preserves distributional shape per action (Wang & Rajeswar 2018 distributional + * dueling identifiability result). + */ +float a_mean_per_atom[NUM_ATOMS_MAX]; /* register array sized to compile-time max NA */ +float inv_n_d = 1.0f / (float)n_d; +for (int z = 0; z < num_atoms; z++) { + float s = 0.0f; + for (int a = 0; a < n_d; a++) { + s += branch_base[(long long)a * num_atoms + z]; + } + a_mean_per_atom[z] = s * inv_n_d; +} + +/* ── now the existing online softmax loop reads `adv_a[z] - a_mean_per_atom[z]` + * instead of plain `adv_a[z]` ─────────────────────────────────────────── */ +for (int a = 0; a < n_d; a++) { + const float* adv_a = branch_base + (long long)a * num_atoms; + float M = -1e30f; + float S = 0.0f; + /* …same accumulators as before… */ + for (int z = 0; z < num_atoms; z++) { + float a_centered = adv_a[z] - a_mean_per_atom[z]; /* SP17 */ + float logit = v_row[z] + a_centered; /* was: v_row[z] + adv_a[z] */ + /* …rest of online softmax unchanged… */ + } + /* …rest of action loop unchanged… */ +} +``` + +`NUM_ATOMS_MAX`: align with the existing `THOMPSON_MAX_ATOMS = 128` constant. Add `#ifndef NUM_ATOMS_MAX #define NUM_ATOMS_MAX 128 #endif` near the kernel top, with an `assert(num_atoms <= NUM_ATOMS_MAX)` device-side `__trap()` if violated (cold-fail, never silent truncation). + +- [ ] **Step 3: Implement test body** + run. + +Expected: PASS. + +- [ ] **Step 4: Commit (atomic — only `compute_expected_q`, NO consumer migration yet)** + +```bash +git add -A +git commit -m "$(cat <<'EOF' +feat(sp17): mean-zero identifiability in compute_expected_q + +Inserts per-atom mean-A reduction inside the per-branch outer loop: +Q[a, z] = V[z] + (A[a, z] - mean_a A[*, z]) +The post-centering atom-level identifiability holds: Σ_a A_centered[a, z] = 0 +for every (sample, branch, atom). + +Per Wang et al. 2016 Dueling Networks: mean-zero is smoother and more +stable than max-zero. Atom-level subtraction (not expectation-level) +preserves distributional shape per action. + +Sample-local register reduction; no atomicAdd, no shared memory, no +cross-thread sync. NUM_ATOMS_MAX=128 mirrors existing THOMPSON_MAX_ATOMS; +device __trap() on overflow. + +⚠ INTERIM STATE: c51_loss_kernel + c51_grad_kernel + mag_concat_qdir +still read RAW (un-centered) advantage logits. Phase 1 Tasks 1.3-1.5 +migrate them in lockstep within this branch BEFORE any L40S dispatch. +A partially-migrated state is forbidden per `feedback_no_partial_refactor`. + +Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +### Task 1.3: Pre-migration grep for all consumers of `b_logits` raw advantage + +**Why:** Per `feedback_no_partial_refactor`, every consumer of the changed contract migrates in the same branch. + +- [ ] **Step 1: Audit script** + +```bash +grep -rn "v_logits\|adv_a\[\|b_logits\[\|adv_logit" \ + crates/ml/src/cuda_pipeline/ | \ + grep -vE "(test|//|\\*)" > /tmp/sp17_consumers.txt +cat /tmp/sp17_consumers.txt +``` + +Catalogue every line. Expected consumers (grouped): +- `compute_expected_q` (Task 1.2 done) +- `c51_loss_kernel.cu`: `block_bellman_project_f`, `c51_loss_batched` +- `c51_grad_kernel.cu`: gradient through softmax(V+A) +- `mag_concat_qdir`: direction Q for magnitude-conditioning concatenation +- `add_advantage_noise`: writes Gaussian noise into Q-values *post-aggregation* — NO change required (operates on already-centered Q) +- `experience_kernels.cu` Thompson direction-selection block (line ~1264-1500): uses `b_logits_dir` directly for Thompson sampling + +- [ ] **Step 2: Append the consumer list to this plan** as Phase 1 evidence — locked checklist for Tasks 1.4–1.5 to mark off. + +No commit (audit-only). + +### Task 1.4: Migrate `c51_loss_kernel` + `c51_grad_kernel` to centered A (atomic) + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/c51_loss_kernel.cu` — insert mean-A reduction in `block_bellman_project_f` and `c51_loss_batched` +- Modify: `crates/ml/src/cuda_pipeline/c51_grad_kernel.cu` — propagate gradient through the centering (the centering is a linear projection, so `dL/dA[a, z] -= mean_a (dL/dA[*, z])`) + +- [ ] **Step 1: Write failing GPU oracle test** + +```rust +/// B.2: c51_loss kernel uses centered A and reproduces the CPU oracle's +/// E[Q] under the centered formulation when computing the projection +/// target. +#[test] +#[ignore = "requires GPU"] +fn c51_loss_uses_centered_a_in_projection() { + // Synthetic v_logits + b_logits + reward + next_v_logits + next_b_logits, + // run c51_loss with the post-SP17 kernel, recover the projection target, + // confirm it matches the centered-A CPU oracle prediction to ε=1e-5. + unimplemented!("Phase 1 Step 3 lands the kernel modification"); +} +``` + +- [ ] **Step 2: Modify both kernels** with the same per-atom mean-A subtraction pattern as Task 1.2. + +For the gradient kernel: per chain rule, the linear projection `A_centered = A - (1/n_d) Σ_a A` has Jacobian `J[a, a'] = δ(a, a') - 1/n_d`. Therefore `dL/dA[a, z] = dL/dA_centered[a, z] - (1/n_d) Σ_a' dL/dA_centered[a', z]`. Applied per atom z, this is one extra reduction per branch per atom — same pattern as forward. Document the math inline in a comment block above the gradient kernel modification. + +- [ ] **Step 3: Run test, verify PASS** + +- [ ] **Step 4: Wire-up audit** + +```bash +grep -rn "b_logits\[\|adv_a\[" crates/ml/src/cuda_pipeline/c51_loss_kernel.cu crates/ml/src/cuda_pipeline/c51_grad_kernel.cu | grep -v "centered\|a_mean_per_atom\|//" +``` +Expected output: empty (every advantage read goes through centering). + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "$(cat <<'EOF' +feat(sp17): centered A in c51_loss + c51_grad (atomic with compute_expected_q) + +Migrates the Bellman target projection (block_bellman_project_f) and +its gradient kernel to the post-SP17 mean-zero identifiability. The +projection's Jacobian for the linear centering A_c = A - (1/n) Σ A is +J[a, a'] = δ(a, a') - 1/n; therefore the gradient backprop applies the +SAME per-atom mean subtraction (idempotent for symmetric reductions). + +Inline math comment block in c51_grad_kernel above the modified +reduction documents the chain rule. + +Wire-up audit: grep confirms zero un-centered b_logits reads in the +two kernels. + +Per `feedback_no_partial_refactor`: this commit closes the contract +gap that compute_expected_q opened. After this commit, all Bellman- +target paths consume centered A. + +Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +### Task 1.5: Migrate `mag_concat_qdir` + Thompson direction-selection block (atomic with c51) + +**Files:** +- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` — `mag_concat_qdir` (line 5031), Thompson direction block (line ~1264) + +`mag_concat_qdir` reads `dir_logits_v + dir_logits_b` (line 5072) per direction action. Insert the per-atom mean-A reduction over the 4 direction actions (b0_size=4) before the inner softmax. Identical pattern to Tasks 1.2/1.4. + +Thompson block: `softmax_c51_inline(logits_i + d * n_atoms, n_atoms, probs_d)` reads raw direction advantage. The Thompson sampler operates on *direction advantage softmax*; it must read centered A or it disagrees with `compute_expected_q`'s E[Q] (the conviction quantity). Insert mean-A reduction before the per-direction softmax loop (line ~1276). + +- [ ] **Step 1: Write failing GPU oracle test for mag_concat_qdir centered** + +```rust +#[test] +#[ignore = "requires GPU"] +fn mag_concat_qdir_uses_centered_dir_advantage() { + // Build v + b_logits_dir with known a_mean_per_atom = [m_z0, m_z1, ...] + // Launch mag_concat_qdir. + // Confirm Q_dir written into the concat tail equals the centered E[Q], + // matching the CPU oracle for the dir branch. + unimplemented!(); +} +``` + +- [ ] **Step 2: Write failing GPU oracle test for Thompson centered** + +```rust +#[test] +#[ignore = "requires GPU"] +fn thompson_dir_selection_consumes_centered_advantage() { + // Synthetic dir advantage where raw argmax = Hold but centered argmax = Long + // (achievable by setting the V head high so V dominates the raw sum). + // Launch the action-selection kernel; assert the picked direction is Long + // with high probability (>0.8 across N=10000 Philox draws). + unimplemented!(); +} +``` + +- [ ] **Step 3: Migrate both call sites** as in Tasks 1.2/1.4. Test PASS. + +- [ ] **Step 4: Wire-up audit** + +```bash +grep -rn "dir_logits_b\|b_logits_dir\|adv_a\[" crates/ml/src/cuda_pipeline/experience_kernels.cu | grep -v "centered\|a_mean_per_atom\|//\|\\*" +``` + +Expected: every match is inside a `// pre-centering removed` comment or inside the now-centered code blocks. + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "$(cat <<'EOF' +feat(sp17): centered A in mag_concat_qdir + Thompson dir-select + +Closes the consumer migration that compute_expected_q + c51_loss + +c51_grad opened. After this commit, every advantage-logit read in the +DQN pipeline goes through the mean-zero identifiability projection. + +mag_concat_qdir provides the direction Q tail to the magnitude branch +input; if magnitude saw raw E[Q_dir] while c51_loss saw centered, the +trunk would learn a mixed contract. Atomic migration prevents that. + +Thompson direction-selection: the conviction quantity that drives the +picked-direction probability now matches the Bellman target's V/A +decomposition. Without this migration, action selection would respond +to V (the very pathology SP17 is fixing). + +Per feedback_wire_everything_up: the migration is complete; no kernel +in the cuda_pipeline reads raw advantage logits. + +Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +--- + +## Phase 2 — Behavioral test suite (V/A separation) + +These are oracle tests that probe the **architectural contract**, not just the math. They run on synthetic markets without dispatching to L40S. + +### Task 2.1: V-invariance test + +> _"Identical state, varying available actions → V should be invariant."_ But our action space is fixed; the analogue here is: **identical state across two consecutive bars, varying advantage signal → V trajectory should reflect state-value evolution, A trajectory should reflect action-relevant change.** + +**Files:** +- Modify: `crates/ml/tests/sp17_dueling_oracle_tests.rs` — add `v_invariance_under_action_perturbation` + +- [ ] **Step 1: Write failing test** + +```rust +#[test] +#[ignore = "requires GPU"] +fn v_invariance_under_action_perturbation() { + // Setup: synthetic state s with v_logits = [v_z for each atom]. + // Build TWO advantage tensors A1, A2 such that (A1 - mean A1) ≠ (A2 - mean A2) + // but (mean A1 = mean A2) → V should give identical E[V] for both. + // Verify: E[V] (read from the v-only diagnostic readout) is identical to ε=1e-5 + // across both A1 and A2 forward passes through compute_expected_q. + unimplemented!(); +} +``` + +- [ ] **Step 2-4: Implement, run, commit.** + +```bash +git commit -m "$(cat <<'EOF' +test(sp17): V-invariance behavioral test + +Verifies the architectural contract: V depends only on state, not on +the specific advantage tensor. Two A tensors with same mean produce +the same V; non-equivalent A tensors are distinguishable only via +A_centered. + +This is the structural property that makes dueling work — without it, +the model conflates "state value" with "action value" and gradient +updates corrupt V via raw-A noise. + +Plan: docs/superpowers/plans/2026-05-08-sp17-dueling-q-network.md + +Co-Authored-By: Claude Opus 4.7 (1M context) +EOF +)" +``` + +### Task 2.2: A-invariance test + +> _"Same action, varying market state → A should be invariant."_ The analogue: A_centered must respond only to the action-relevant relative ordering, not to absolute V scale. + +- [ ] **Step 1: Write failing test** + +```rust +#[test] +#[ignore = "requires GPU"] +fn a_centered_invariance_under_v_shift() { + // Two states s1, s2 differ only in V scale: v_logits_s2 = v_logits_s1 + 10.0 + // Same advantage tensor A applied to both. + // Verify: A_centered_s1 == A_centered_s2 to ε=1e-5 (the centering is + // computed from A only — the V shift cannot leak in). + unimplemented!(); +} +``` + +- [ ] **Step 2-4: Implement, run, commit.** + +### Task 2.3: Symmetric A → uniform action distribution + +> _"If A is identically zero across actions, action selection should be uniform-random under Thompson."_ + +- [ ] **Step 1: Write failing test** + +```rust +#[test] +#[ignore = "requires GPU"] +fn uniform_a_yields_uniform_thompson_dir_selection() { + // A = 0 everywhere across (action, atom). V arbitrary. + // Run Thompson direction selection N=10000 times under different + // Philox seeds. Assert: per-direction frequency ∈ [0.20, 0.30] for + // each of the 4 directions (uniform = 0.25 ± 5pp Wilson interval). + unimplemented!(); +} +``` + +- [ ] **Step 2-4: Implement, run, commit.** + +### Task 2.4: Asymmetric A → deterministic argmax under low Thompson temp + +- [ ] **Step 1: Write failing test** + +```rust +#[test] +#[ignore = "requires GPU"] +fn asymmetric_a_yields_deterministic_argmax() { + // A_centered for direction = [0.0, 0.0, 1.0, 0.0] (Long dominates) + // V arbitrary. Thompson temp = 0.0 → pure argmax E[Q]. + // Assert: pick = DIR_LONG in 100% of N=1000 runs. + unimplemented!(); +} +``` + +- [ ] **Step 2-4: Implement, run, commit.** + +--- + +## Phase 3 — HEALTH_DIAG diagnostic emit + adaptive `advantage_clip_bound` producer + +(Only if **DD7=a**. If **DD7=b**, this phase is folded into Phase 5 as a HEALTH_DIAG-only readout with no producer.) + +### Task 3.1: A_var_ema producer kernel + ISV writes + +**Files:** +- New: `crates/ml/src/cuda_pipeline/sp17_va_diagnostic_kernel.cu` +- Modify: `crates/ml/build.rs` to register cubin +- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` — load + launch site +- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — HEALTH_DIAG readout + +- [ ] **Step 1: Write failing test** + +```rust +#[test] +#[ignore = "requires GPU"] +fn a_var_ema_per_branch_tracks_centered_variance() { + // Synthetic A_centered with known per-branch variances. + // Launch sp17_va_diagnostic_kernel. + // Assert ISV[A_VAR_EMA_*] readback matches expected variance per branch + // to ε=1e-5 after Pearl-A first-observation bootstrap. + unimplemented!(); +} +``` + +- [ ] **Step 2: Implement kernel** — single-thread cold-path (mirrors `min_hold_temperature_update_kernel` design — one launch per epoch, 4 branches × per-sample-per-atom variance Welford with Pearl-A first-observation bootstrap on `A_VAR_EMA_*` slots) + +- [ ] **Step 3: Wire launcher into per-epoch HEALTH_DIAG emit path** + +- [ ] **Step 4: Run, commit.** + +### Task 3.2: V_share producer + adaptive clip bound + +`V_share[branch] = |E[V]| / (|E[V]| + |E[A_centered, picked_action]|)` per branch. Read at HEALTH_DIAG. + +`advantage_clip_bound`: `p99(|A_centered|) × 1.5` produced by `sp4_histogram_p99` over the per-batch flattened `|A_centered|` tensor (no atomicAdd per `pearl_fused_per_group_statistics_oracle`). + +- [ ] **Steps 1-4: Failing test → implement → run → commit.** + +### Task 3.3: HEALTH_DIAG full V/A trajectory emit + +After the producers land: + +``` +HEALTH_DIAG[N]: dueling [v_share=(d=X m=Y o=Z u=W)] [a_var=(d=A m=B o=C u=D)] [clip=K] +``` + +Reader emits 4 + 4 + 1 = 9 slots per epoch. The reviewer can detect post-deploy whether identifiability is doing real work (high a_var, balanced v_share) or regressing (a_var → 0, v_share → 1.0). + +- [ ] **Steps 1-4: Failing test → implement → run → commit.** + +--- + +## Phase 4 — 5-epoch L40S smoke validation + +### Task 4.1: Push + dispatch + monitor + +**This task is reviewer-executed (not subagent).** + +- [ ] **Step 1: Push final SP17 branch** + +```bash +git push origin feat/sp17-dueling +``` + +- [ ] **Step 2: Dispatch L40S 5-epoch smoke** + +```bash +./scripts/argo-train.sh --model dqn --epochs 5 --label sp17-smoke-v1 +``` + +- [ ] **Step 3: Smoke gate criteria** + +| Gate | Threshold | Source | +|---|---|---| +| Q(Flat) − Q(Long) gap trajectory | NOT widening monotonically Fold 0→Fold 1 | new dueling HEALTH_DIAG + existing q_by_action | +| dir_entropy Fold 1 final | ≥ 0.75 (b5gmp baseline 0.70) | existing | +| Hold% Fold 1 final | ≤ 50% (b5gmp baseline 52.6%) | existing | +| a_var per branch | > 0.01 throughout (a_var → 0 = "A is doing nothing" failure mode) | new | +| v_share per branch | ∈ [0.3, 0.7] throughout (drift outside = imbalance) | new | +| val Sharpe Fold 1 final | ≥ 60 (b5gmp 55.55, pfh9n 64.24) | existing | +| advantage_clip_bound trajectory | settled within [0.5, 5.0] by epoch 3 | new | +| no NaN / no train hang | always | existing | + +- [ ] **Step 4: Document outcome** + +Append "Phase 4 smoke evidence" subsection to this plan. + +--- + +## Phase 5 — 30-epoch full validation (post-smoke pass) + +### Task 5.1: Full dispatch + analysis + +**Reviewer-executed.** + +- [ ] **Step 1: 30-epoch dispatch** + +```bash +./scripts/argo-train.sh --model dqn --epochs 30 --label sp17-full-v1 +``` + +- [ ] **Step 2: Full gate criteria** + +| Gate | Threshold | +|---|---| +| val Sharpe across 5 folds | mean ≥ 65, no fold < 50 | +| Hold% trajectory across folds | stable (not climbing fold-over-fold) | +| Q(Flat) − Q(Long) gap | bounded throughout (no monotonic widening) | +| dir_entropy stability | stays ∈ [0.7, 1.1] across folds | +| a_var per branch | bounded > 0 throughout | + +- [ ] **Step 3: Pass/fail decision + memory pearl land** + +If pass: land `pearl_dueling_identifiability_for_multi_sink_q_bias.md` to memory. If fail: open SP18 follow-up plan. + +--- + +## Self-review checklist + +- [ ] Every task is bite-sized TDD: failing test → implement → pass → commit. +- [ ] No task references `todo!()` outside of an explicit "this lands later" marker. +- [ ] No `// ...` ellipses in code blocks. Where I omit context I cite the file:line that contains the surrounding code. +- [ ] Phase 1 atomic-consumer-migration is explicit: 4 commits, all in same branch, NO L40S dispatch between them. +- [ ] All adaptive bounds are ISV-driven (advantage_clip_bound producer-tracked, not hardcoded). +- [ ] All reductions use block tree-reduce or sample-local register reduction (no atomicAdd). +- [ ] All CPU↔GPU transfers use `MappedF32Buffer` (tests not exempt). +- [ ] Pearl candidates flagged at end of each phase for human review/landing. +- [ ] Kill criterion in Phase 0 is concrete and disqualifying (not aspirational). +- [ ] Design decisions are explicit and reviewer-actionable (DD1–DD7 above). + +--- + +## Memory pearl candidates + +(Drafts for the human reviewer to land or reject after each phase.) + +### Pearl 1 (post-Phase 1): `pearl_dueling_identifiability_atomic_consumer_migration.md` + +> When the V/A aggregation contract changes, every consumer (forward Bellman, gradient backprop, action selection, conditional inputs to other branches) migrates atomically. Partial migration creates a strict-worse state where action selection sees one V/A contract and Bellman target sees another — disagreement at the multi-consumer boundary that didn't exist before the change. + +### Pearl 2 (post-Phase 4 — IF smoke passes): `pearl_dueling_identifiability_for_multi_sink_q_bias.md` + +> Multi-sink Q-bias (Hold% pressure migrates Q-bias from Q(Hold) to Q(Flat) — the SP15→SP16→b5gmp pattern) is solved by mean-zero identifiability `Q(s,a) = V(s) + (A(s,a) − mean_a A(s,a))`. Without identifiability, gradient updates split arbitrarily between V and A; per-action cost shaping pushes one sink down but the model migrates V upward and a different action becomes the new sink. Mean-zero forces the action-selection signal entirely into A by construction; no migration possible. + +### Pearl 3 (post-Phase 4 — IF smoke fails): `pearl_dueling_alone_does_not_break_q_bias.md` + +> Mean-zero identifiability is necessary but not sufficient when Q-bias has *non-architectural* origin (Adam optimizer asymmetry, reward asymmetry, bootstrap target inflation). If Phase 4 smoke shows a_var > 0 and v_share balanced but Q(Flat) bias persists, the structural fix didn't address root cause; investigate Q-target inflation (Phase 4 SP18 candidate) or per-action Adam state separation. + +--- + +## Execution handoff + +Total estimated commits: **27 tasks × 1 commit each = 27 commits across feat/sp17-dueling**. + +Estimated wall time: +- Pre-Phase + Phase 0 (diagnostic + KILL CRITERION): ~1 day with 1-epoch L40S dispatch +- Phases 1-3 (consumer migration + tests): ~2-3 days +- Phase 4 smoke: ~6 hours dispatch + analysis +- Phase 5 full: ~24 hours dispatch + analysis + +Reviewer must: +1. Resolve **DD1–DD7** before subagent dispatch. +2. Run Pre-Phase Task PP.1 (worktree creation) manually. +3. Run all `argo-train.sh` dispatches manually (Phases 0, 4, 5). +4. Land/reject memory pearls at end of each phase. + +Subagent must: +1. Hold to atomic-consumer-migration discipline in Phase 1 (Tasks 1.2–1.5 land in same branch before any L40S dispatch). +2. Stop and ask if any kernel under modification has a graph-capture interaction not visible in the consumer audit (per `pearl_no_host_branches_in_captured_graph` — the per-atom mean reduction is pure-device, but new kernels in Phase 3 must be audited). +3. Stop and ask if Phase 0 KILL CRITERION returns INCONCLUSIVE.