# Session follow-up — Phase 3 IQN-Complement post-mortem + Phase 4 foundation

**Date:** 2026-05-29 → 2026-05-30
**Branch lineage:** Plan A v2 `fd3174262` → 5 failed experiments → Phase 4 design + foundation
**Production baseline preserved:** Plan A v2 (+$9.3M peak, 20k clean, no NaN)

---

## 1. Session arc

This session went deep on adding **proper dueling Q-learning** to the
foxhunt RL trainer. Started from Plan A v2 baseline (vanilla C51 + IQN
auxiliary, no dueling). After 5 cluster experiments and 4 architectural
pearls, landed on the correct design (Phase 4: independent dueling head)
and shipped its foundation.

### Timeline

| Phase | Architecture | Commit | Outcome |
|---|---|---|---|
| Plan A v2 | C51 + IQN aux, scalar V baseline | `fd3174262` | **+$9.3M peak, 20k clean** (production baseline) |
| Phase 2 v2 | C51 + scalar dueling V | `eac382d20` | ❌ Math no-op (Phase 2 from prior session) |
| Phase 3.1 | IQN V/A split, composed Q output | `6b7c9521d` | ❌ Cluster step 745: qpa=-0.93 |
| Phase 3.2 | 3.1 + V_IQN replaces V_scalar PPO | `0fdce0b44` | ❌ Cluster step 556: qpa=-0.96 |
| Phase 3.2c | 3.2 + α-warmup blend | `8616cf9cb` | ❌ Cluster step 762: qpa=-0.94 |
| Phase 3.1-fix | 3.1 + dual output (raw A for ensemble) | `87540e748` | ❌ Cluster step 746: qpa=-0.92 |
| **Phase 4 spec** | Independent dueling head | `5160ba781` | ✅ Design captured |
| **Phase 4.0** | Forward kernel + struct | `af35bc778` | ✅ Compiles, smoke passes |
| **Phase 4.1** | Loss + Bellman target + decompose backward | `13bf277cd` | ✅ Compiles, smoke passes |

---

## 2. The architectural journey — what each failure taught us

### Phase 2 v2 (prior session, recapped)

**Attempt:** Add scalar V mediation to categorical C51 via
`composed_logits[b, a, z] = V[b] + A_logits[b, a, z] − mean_a A_logits[b, a, z]`.

**Failure:** Mathematical no-op.
`softmax_z(V[b] + X[a, z]) = softmax_z(X[a, z])` because softmax is
translation-invariant in z and V is constant across z. V drops out
entirely. CE gradient also collapses: `Σ_z (softmax − target) = 0`.

**Pearl:** `pearl_scalar_v_with_categorical_ce_is_noop`

### Phase 3 IQN-Complement approach

**Insight:** IQN's per-τ Q values are scalars (not categorical
distributions over atoms). Scalar dueling works mathematically at fixed
τ. So add IQN-with-dueling as a complement to C51 (which keeps Plan A
v2 behavior for Q→π distill), and use V_IQN derived from IQN's composed
Q for the PPO advantage baseline.

**Pearl:** `pearl_complement_dont_replace_with_dual_architecture`

### Phase 3.2 — first attempt (direct V swap)

**Attempt:** Phase 3.1 (IQN V/A split) + V_IQN replaces V_scalar in PPO
advantage. `V_IQN(s) = (1/N) × Σ_a E_τ[Q(s, a, τ)]` (exact recovery
from dueling identifiability).

**Failure:** Cluster step 556 qpa=-0.96. The math is exact but the
PRACTICAL issue: `A = E_Q_C51 − V_IQN` mixes outputs from two
networks at different output scales. Early in training (when neither
network's V/Q values are well-calibrated), the difference
systematically biases the advantage sign → π pushed AWAY from Q's
preferences → anti-correlation.

**Pearl:** `pearl_complement_architectures_must_be_calibrated_when_mixed`

### Phase 3.2c — α-blend warmup

**Attempt:** Blend `V_used = α × V_scalar + (1-α) × V_IQN` with α
ramping from 1.0 (Plan A v2 behavior) to 0.0 over steps 2000-10000.
At α=1.0, V_used = V_scalar exactly — should match Plan A v2.

**Failure:** Cluster step 762 qpa=-0.94. Even with α=1.0 (V_blended =
V_scalar mathematically), qpa collapses identically. This was the
diagnostic smoking gun — the V baseline integration **wasn't the
problem**. Something earlier in Phase 3.1 was.

### Phase 3.1 isolation

**Attempt:** Just Phase 3.1 (IQN dueling internal), Plan A v2's
V_scalar path UNCHANGED. Isolation test to localize the breakage.

**Failure:** Cluster step 745 qpa=-0.93. Confirmed: **Phase 3.1
alone breaks downstream consumers, independent of V baseline**.

**Pearl (initial form):** `pearl_complement_internal_loss_vs_external_consumers`

**Root cause:** Phase 3.1 changed `iqn_expected_q` to compute mean_τ
over COMPOSED Q (`V + A − mean_a A`). The values become CENTERED
around V(s). The ensemble Q kernel blends C51's UNCENTERED expected_q
with IQN's CENTERED expected_q. The blend's softmax distribution
behaves differently from Plan A v2's, perturbing the q_pi_agree
diagnostic.

### Phase 3.1-fix — dual output

**Attempt:** Add a new kernel `rl_iqn_extract_raw_a_with_bias` that
extracts raw A logits (uncentered) alongside the composed Q output.
expected_q now operates on raw A (Plan A v2 distribution shape) for
ensemble use; composed Q still feeds the Huber loss.

**Failure:** Cluster step 746 qpa=-0.92. **Dual output extraction is
necessary but not sufficient.** Even though raw A has matching
distribution shape (translation-equivalent to Plan A v2's uncentered
Q), the WEIGHTS train differently in Phase 3.1's dueling regime:
- A's gradient has mean-zero structure across actions:
  `grad_A[a] = grad_q[a] − (1/N) Σ_a' grad_q[a']`
- sum_a grad_A[a] = 0 by construction
- A weights and bias train to capture per-state action DEVIATIONS,
  not raw Q values

The "raw A + bias" extracted in Phase 3.1-fix represents
deviation-shaped values, not Plan A v2's uncentered Q values. When
fed to ensemble + softmax, the absolute magnitudes are smaller →
ensemble dominated by C51's larger magnitudes → ranking dynamics
shift → qpa stays decoupled.

**Pearl (updated):** `pearl_complement_internal_loss_vs_external_consumers`
addendum documents this gradient-structure finding.

---

## 3. The four architectural pearls

These pearls form a complete design pattern for adding complementary
ML architectures. They generalize beyond this project — any future
work that adds a complement to a working primary architecture should
respect all four:

### Pearl 1 — Don't replace what works
`pearl_complement_dont_replace_with_dual_architecture`

When primary architecture (C51) has demonstrated edge but can't
mathematically support a desirable property (dueling), add a
complementary architecture (IQN) that natively has it. Use each for
what it's mathematically suited to.

### Pearl 2 — Calibrate cross-architecture signals
`pearl_complement_architectures_must_be_calibrated_when_mixed`

When the complement's output is mixed with the primary's in a single
loss term, the two outputs MUST be on similar scales — otherwise even
mathematically correct values produce systematic sign bias.

### Pearl 3 — Match output distribution shape AND gradient structure
`pearl_complement_internal_loss_vs_external_consumers` (+ addendum)

When refactoring a complement's loss function, ALL downstream
consumers of its outputs must be checked. Distribution shape matching
is NECESSARY but NOT SUFFICIENT — the gradient structure that trains
weights must also match the primary's, or downstream consumers see
different absolute magnitudes over training.

### Pearl 4 — Forward math ≠ training-correct
(implicit from Phase 3.1-fix failure)

Mathematical analysis of forward outputs is insufficient for adding
complementary architectures. Backward gradient flow shapes weight
training over thousands of steps; downstream consumers depend on
trained weight magnitudes for temperature scaling, ranking, etc.
Architectural changes that pass forward-math sanity can still break
downstream dynamics if gradient structure changes.

---

## 4. Phase 4 design — what makes it different

**The synthesis of all four pearls:** to add dueling without breaking
anything, **completely encapsulate** the dueling architecture. Don't
modify C51's or IQN's weights, outputs, or gradient paths in any way.

### Architecture

```
Encoder (shared) → h_t
  ├→ C51 head           (Plan A v2 — UNCHANGED)
  ├→ IQN head           (Plan A v2 — UNCHANGED)
  ├→ Policy head π      (Plan A v2 — UNCHANGED)
  ├→ value_head V_scalar (still trains for diag comparison)
  └→ NEW: DuelingQHead  (parallel, fully encapsulated)
       • Has its OWN weights w_v, b_v, w_a, b_a
       • Trained by its OWN Bellman loss
       • V output → PPO advantage baseline (Phase 4.3 swap)
       • composed_Q NEVER feeds ensemble/distill/selection
```

### Why this is structurally safe (cross-checking against each pearl)

| Pearl | Phase 4 mitigation |
|---|---|
| 1: Don't replace | C51, IQN, π, value_head all unchanged |
| 2: Calibrate when mixed | V_dq trained on SAME Bellman reward signal as C51 — scales align by construction |
| 3: Match distribution + gradient | composed_Q_dq never feeds ANY downstream consumer — gradient structure changes affect only DuelingQHead's own weights |
| 4: Training-correct ≠ math-correct | DuelingQHead's weights are isolated; perturbations only affect V_dq output which has a single, well-defined consumer (PPO advantage) |

Every prior failure mode is **structurally impossible** in Phase 4.

---

## 5. Phase 4 implementation state

### Shipped (this session)

**Phase 4.0** (`af35bc778`):
- `rl_dueling_q_forward.cu` kernel — V[B] + A[B,N] + composed_Q[B,N]
  in one fused pass. Grid (B,1,1), block (HIDDEN_DIM,1,1).
- `DuelingQHead` Rust struct in `crates/ml-alpha/src/rl/dueling_q.rs`:
  - Config + new() with Xavier init
  - Online weights: w_v_d, b_v_d, w_a_d, b_a_d
  - Target weights: mirrors (soft-update wiring in 4.2)
  - forward() + forward_target() + forward_inner()

**Phase 4.1** (`13bf277cd`):
- `rl_dueling_q_bellman_target.cu` — argmax + scalar Bellman target
- `rl_dueling_q_loss_and_grad.cu` — Huber loss + grad_composed
- `rl_dueling_q_decompose_and_bwd.cu` — Jacobian + outer-product weight grads
- 3 new DuelingQHead methods:
  - build_bellman_target()
  - compute_loss_and_grad()
  - decompose_and_backward_to_weights()

### Remaining (next session)

**Phase 4.2 — trainer integration (diagnostic mode, ~150 LOC):**

Add to `IntegratedTrainer`:
- `dueling_q_head: DuelingQHead` field + 4 Adam states (w_v, b_v, w_a, b_a)
- 10+ buffer fields:
  - V_dq[B] / V_dq_tp1[B] (for PPO advantage when wired)
  - A_dq[B,N] (diag)
  - composed_Q_dq[B,N] (for loss)
  - target_composed_Q_dq[B,N] (for Bellman target build)
  - target_value_dq[B] (Bellman target output)
  - loss_pb_dq[B]
  - grad_composed_dq[B,N]
  - per-batch weight grads (4 buffers)
  - reduced weight grads (4 buffers)
- Wiring in `step_with_lobsim_gpu_body`:
  1. forward(h_t_borrow) → V_dq, A_dq, composed_Q_dq
  2. forward(h_tp1_d) → V_dq_tp1 (for PPO)
  3. forward(sampled_h_t_d) → online composed_Q for loss
  4. forward_target(sampled_h_tp1_d) → target composed_Q
  5. build_bellman_target → target_value_dq
  6. compute_loss_and_grad → grad_composed_dq + loss_pb_dq
  7. decompose_and_backward_to_weights → per-batch grads
  8. reduce_axis0 → final weight grads
  9. Adam step on each weight
  10. Soft-update target net (reuse `dqn_target_soft_update_fn`)
- **DO NOT yet swap v_pred_d in compute_advantage_return** —
  diagnostic-only mode for Phase 4.2

**Phase 4.3 — V_dq → PPO swap (~10 LOC):**

After 4.2 cluster-validates V_dq trains stably without affecting
Plan A v2 metrics:
- Replace `v_pred_d` → `dueling_v_d` in compute_advantage_return call
- Replace `v_pred_tp1_d` → `dueling_v_tp1_d`
- value_head still trains via MSE on returns (for diagnostic
  comparison V_dq vs V_scalar)

### Validation strategy (per spec §8)

1. Local smoke + compute-sanitizer at b=128 — should produce 0 errors
   with healthy l_dueling > 0
2. Cluster smoke b=1024 SHORT (5k steps) at Phase 4.2 — verify:
   - qpa_ema near 0 (Plan A v2 unchanged because V_dq doesn't feed PPO)
   - V_dq trains stably (loss decreasing, V values reasonable)
   - dueling_v_at_taken_ema rises over training
3. Cluster full b=1024 20k at Phase 4.3 — verify:
   - qpa healthy throughout
   - PnL matches or exceeds Plan A v2 +$9.3M peak

### Kill criteria

Any one triggers abort:
- qpa_ema < -0.3 at any checkpoint
- nan_abort_step > 0
- l_dueling explodes (> 100× Plan A v2's l_q baseline)
- pnl_cum_usd < Plan A v2's at same step by > $5M

---

## 6. Operational lessons (separate from architectural pearls)

### Branch ref trap — hit twice

Argo workflow's `git checkout origin/$BRANCH` always picks the
branch TIP, not the `commit-sha` param (which is used only for
output paths). Lessons:

1. **For isolation tests**, create a dedicated branch with the
   specific commit as tip:
   ```
   git branch -f ml-alpha-phaseN-only <commit-sha>
   git push origin ml-alpha-phaseN-only --force
   ```

2. **Always verify cluster-side**: check the diag JSONL path output
   (`/feature-cache/alpha-rl-runs/$SHA/`) matches the expected
   commit-sha before assuming the cluster is running what you
   intended.

3. **Avoid detached HEAD commits** — they create branch ref vs HEAD
   divergence. After each commit on a worktree, verify
   `git rev-parse $BRANCH == git rev-parse HEAD`.

### JSONL monitoring beats stdout

Argo workflow template's `log-every=5000` only emits 4 visible step
markers across a 20k run. The per-step JSONL (`diag.jsonl` on
feature-cache PVC) is the source of truth. Monitor pattern:

```bash
kubectl exec -n foxhunt $WF -c main -- bash -c \
  "tail -1 /feature-cache/alpha-rl-runs/$SHA/fold0/diag.jsonl"
```

Polls every 90s, emits per 500-step milestone. Catches qpa collapse
within ~3 minutes of training start.

### Kill early on qpa collapse

The `q_pi_agree_ema` diagnostic is THE most sensitive cross-arch
health signal. If qpa drops below -0.3 by step 1000, KILL — the
architecture is broken and waiting longer just confirms what's
already known. Saved hours of cluster cycles by killing at step
556 (Phase 3.2), 762 (Phase 3.2c), 745 (3.1 isolation), 746
(3.1-fix).

---

## 7. References

### Pearls created this session

- `pearl_complement_dont_replace_with_dual_architecture` (2026-05-29)
- `pearl_complement_architectures_must_be_calibrated_when_mixed` (2026-05-30)
- `pearl_complement_internal_loss_vs_external_consumers` (2026-05-30 + addendum)
- `pearl_scalar_v_with_categorical_ce_is_noop` (carried from prior session)

### Documents written this session

- `docs/superpowers/plans/2026-05-29-phase2-post-mortem-and-redesign.md`
- `docs/superpowers/specs/2026-05-30-phase4-independent-dueling-head-design.md`
- This file: `docs/superpowers/notes/2026-05-30-phase3-iqn-complement-postmortem-and-phase4-foundation.md`

### Cluster runs (chronological)

| Workflow | Commit | Result | Notes |
|---|---|---|---|
| alpha-rl-8ksnj | fd3174262 | ✅ Succeeded | Plan A v2 production baseline (+$9.3M peak) |
| alpha-rl-dxlnm | 7790c6815 | Failed | First branch ref trap |
| alpha-rl-hcmk7 | a7b0d6d1a | Failed | HIDDEN_DIM=512 OOB (Phase 3.2-pre) |
| alpha-rl-cs-pq2jg | a7b0d6d1a | Diagnosed | compute-sanitizer found the OOB |
| alpha-rl-2qbmz | eac382d20 | Killed | Phase 2 v2 step 4267 qpa=-0.79 |
| alpha-rl-gvqk5 | 0fdce0b44 | Killed | Phase 3.2 step 556 qpa=-0.96 |
| alpha-rl-nl5c9 | 8616cf9cb | Killed | Phase 3.2c step 762 qpa=-0.94 |
| alpha-rl-582bn | (wrong branch) | Killed | Second branch ref trap |
| alpha-rl-xrbgr | 6b7c9521d | Killed | Phase 3.1 isolation step 745 qpa=-0.93 |
| alpha-rl-9q2pd | 87540e748 | Killed | Phase 3.1-fix step 746 qpa=-0.92 |

10 cluster cycles. Plan A v2 baseline preserved untouched throughout.

### Branches on remote (for archaeology)

- `ml-alpha-phase3-iqn-complement` @ `8616cf9cb` (Phase 3.1 + 3.2 + 3.2c)
- `ml-alpha-phase3.1-only` @ `6b7c9521d` (Phase 3.1 isolation test)
- `ml-alpha-phase3.1-fix` @ `87540e748` (Phase 3.1 + dual output fix)
- `ml-alpha-phase4-dueling-head` @ `13bf277cd` **← Phase 4 foundation, current work**
- `main` (Plan A v2 baseline at `fd3174262` via merge)

---

## 8. Next session opening checklist

When resuming:

1. ☐ Pull `ml-alpha-phase4-dueling-head` @ `13bf277cd`
2. ☐ Read this note + spec `docs/superpowers/specs/2026-05-30-phase4-independent-dueling-head-design.md` §6
3. ☐ Create worktree (or reuse `/tmp/foxhunt-de-tests/phase4-dueling-head`)
4. ☐ Start Phase 4.2: add DuelingQHead instance + buffers + Adam states to IntegratedTrainer
5. ☐ Wire 10-step launch sequence in step_with_lobsim_gpu_body
6. ☐ DO NOT swap v_pred_d yet (4.3 is separate commit)
7. ☐ Local smoke + sanitizer at b=128 — expect 0 errors, l_dueling > 0
8. ☐ Cluster smoke b=1024 5k — verify qpa unchanged from Plan A v2
9. ☐ If ✅, Phase 4.3: swap V_dq into compute_advantage_return
10. ☐ Cluster b=1024 20k full validation
