foxhunt/docs/superpowers/notes/2026-05-30-phase3-iqn-complement-postmortem-and-phase4-foundation.md

Session follow-up — Phase 3 IQN-Complement post-mortem + Phase 4 foundation

Date: 2026-05-29 → 2026-05-30 Branch lineage: Plan A v2 fd3174262 → 5 failed experiments → Phase 4 design + foundation Production baseline preserved: Plan A v2 (+$9.3M peak, 20k clean, no NaN)

---

1. Session arc

This session went deep on adding proper dueling Q-learning to the foxhunt RL trainer. Started from Plan A v2 baseline (vanilla C51 + IQN auxiliary, no dueling). After 5 cluster experiments and 4 architectural pearls, landed on the correct design (Phase 4: independent dueling head) and shipped its foundation.

Timeline

Phase	Architecture	Commit	Outcome
Plan A v2	C51 + IQN aux, scalar V baseline	fd3174262	+$9.3M peak, 20k clean (production baseline)
Phase 2 v2	C51 + scalar dueling V	eac382d20	❌ Math no-op (Phase 2 from prior session)
Phase 3.1	IQN V/A split, composed Q output	6b7c9521d	❌ Cluster step 745: qpa=-0.93
Phase 3.2	3.1 + V_IQN replaces V_scalar PPO	0fdce0b44	❌ Cluster step 556: qpa=-0.96
Phase 3.2c	3.2 + α-warmup blend	8616cf9cb	❌ Cluster step 762: qpa=-0.94
Phase 3.1-fix	3.1 + dual output (raw A for ensemble)	87540e748	❌ Cluster step 746: qpa=-0.92
Phase 4 spec	Independent dueling head	5160ba781	✅ Design captured
Phase 4.0	Forward kernel + struct	af35bc778	✅ Compiles, smoke passes
Phase 4.1	Loss + Bellman target + decompose backward	13bf277cd	✅ Compiles, smoke passes

---

2. The architectural journey — what each failure taught us

Phase 2 v2 (prior session, recapped)

Attempt: Add scalar V mediation to categorical C51 via composed_logits[b, a, z] = V[b] + A_logits[b, a, z] − mean_a A_logits[b, a, z].

Failure: Mathematical no-op. softmax_z(V[b] + X[a, z]) = softmax_z(X[a, z]) because softmax is translation-invariant in z and V is constant across z. V drops out entirely. CE gradient also collapses: Σ_z (softmax − target) = 0.

Pearl: pearl_scalar_v_with_categorical_ce_is_noop

Phase 3 IQN-Complement approach

Insight: IQN's per-τ Q values are scalars (not categorical distributions over atoms). Scalar dueling works mathematically at fixed τ. So add IQN-with-dueling as a complement to C51 (which keeps Plan A v2 behavior for Q→π distill), and use V_IQN derived from IQN's composed Q for the PPO advantage baseline.

Pearl: pearl_complement_dont_replace_with_dual_architecture

Phase 3.2 — first attempt (direct V swap)

Attempt: Phase 3.1 (IQN V/A split) + V_IQN replaces V_scalar in PPO advantage. V_IQN(s) = (1/N) × Σ_a E_τ[Q(s, a, τ)] (exact recovery from dueling identifiability).

Failure: Cluster step 556 qpa=-0.96. The math is exact but the PRACTICAL issue: A = E_Q_C51 − V_IQN mixes outputs from two networks at different output scales. Early in training (when neither network's V/Q values are well-calibrated), the difference systematically biases the advantage sign → π pushed AWAY from Q's preferences → anti-correlation.

Pearl: pearl_complement_architectures_must_be_calibrated_when_mixed

Phase 3.2c — α-blend warmup

Attempt: Blend V_used = α × V_scalar + (1-α) × V_IQN with α ramping from 1.0 (Plan A v2 behavior) to 0.0 over steps 2000-10000. At α=1.0, V_used = V_scalar exactly — should match Plan A v2.

Failure: Cluster step 762 qpa=-0.94. Even with α=1.0 (V_blended = V_scalar mathematically), qpa collapses identically. This was the diagnostic smoking gun — the V baseline integration wasn't the problem. Something earlier in Phase 3.1 was.

Phase 3.1 isolation

Attempt: Just Phase 3.1 (IQN dueling internal), Plan A v2's V_scalar path UNCHANGED. Isolation test to localize the breakage.

Failure: Cluster step 745 qpa=-0.93. Confirmed: Phase 3.1 alone breaks downstream consumers, independent of V baseline.

Pearl (initial form): pearl_complement_internal_loss_vs_external_consumers

Root cause: Phase 3.1 changed iqn_expected_q to compute mean_τ over COMPOSED Q (V + A − mean_a A). The values become CENTERED around V(s). The ensemble Q kernel blends C51's UNCENTERED expected_q with IQN's CENTERED expected_q. The blend's softmax distribution behaves differently from Plan A v2's, perturbing the q_pi_agree diagnostic.

Phase 3.1-fix — dual output

Attempt: Add a new kernel rl_iqn_extract_raw_a_with_bias that extracts raw A logits (uncentered) alongside the composed Q output. expected_q now operates on raw A (Plan A v2 distribution shape) for ensemble use; composed Q still feeds the Huber loss.

Failure: Cluster step 746 qpa=-0.92. Dual output extraction is necessary but not sufficient. Even though raw A has matching distribution shape (translation-equivalent to Plan A v2's uncentered Q), the WEIGHTS train differently in Phase 3.1's dueling regime:

• A's gradient has mean-zero structure across actions: grad_A[a] = grad_q[a] − (1/N) Σ_a' grad_q[a']
• sum_a grad_A[a] = 0 by construction
• A weights and bias train to capture per-state action DEVIATIONS, not raw Q values

The "raw A + bias" extracted in Phase 3.1-fix represents deviation-shaped values, not Plan A v2's uncentered Q values. When fed to ensemble + softmax, the absolute magnitudes are smaller → ensemble dominated by C51's larger magnitudes → ranking dynamics shift → qpa stays decoupled.

Pearl (updated): pearl_complement_internal_loss_vs_external_consumers addendum documents this gradient-structure finding.

---

3. The four architectural pearls

These pearls form a complete design pattern for adding complementary ML architectures. They generalize beyond this project — any future work that adds a complement to a working primary architecture should respect all four:

Pearl 1 — Don't replace what works

pearl_complement_dont_replace_with_dual_architecture

When primary architecture (C51) has demonstrated edge but can't mathematically support a desirable property (dueling), add a complementary architecture (IQN) that natively has it. Use each for what it's mathematically suited to.

Pearl 2 — Calibrate cross-architecture signals

pearl_complement_architectures_must_be_calibrated_when_mixed

When the complement's output is mixed with the primary's in a single loss term, the two outputs MUST be on similar scales — otherwise even mathematically correct values produce systematic sign bias.

Pearl 3 — Match output distribution shape AND gradient structure

pearl_complement_internal_loss_vs_external_consumers (+ addendum)

When refactoring a complement's loss function, ALL downstream consumers of its outputs must be checked. Distribution shape matching is NECESSARY but NOT SUFFICIENT — the gradient structure that trains weights must also match the primary's, or downstream consumers see different absolute magnitudes over training.

Pearl 4 — Forward math ≠ training-correct

(implicit from Phase 3.1-fix failure)

Mathematical analysis of forward outputs is insufficient for adding complementary architectures. Backward gradient flow shapes weight training over thousands of steps; downstream consumers depend on trained weight magnitudes for temperature scaling, ranking, etc. Architectural changes that pass forward-math sanity can still break downstream dynamics if gradient structure changes.

---

4. Phase 4 design — what makes it different

The synthesis of all four pearls: to add dueling without breaking anything, completely encapsulate the dueling architecture. Don't modify C51's or IQN's weights, outputs, or gradient paths in any way.

Architecture

Encoder (shared) → h_t
  ├→ C51 head           (Plan A v2 — UNCHANGED)
  ├→ IQN head           (Plan A v2 — UNCHANGED)
  ├→ Policy head π      (Plan A v2 — UNCHANGED)
  ├→ value_head V_scalar (still trains for diag comparison)
  └→ NEW: DuelingQHead  (parallel, fully encapsulated)
       • Has its OWN weights w_v, b_v, w_a, b_a
       • Trained by its OWN Bellman loss
       • V output → PPO advantage baseline (Phase 4.3 swap)
       • composed_Q NEVER feeds ensemble/distill/selection

Why this is structurally safe (cross-checking against each pearl)

Pearl	Phase 4 mitigation
1: Don't replace	C51, IQN, π, value_head all unchanged
2: Calibrate when mixed	V_dq trained on SAME Bellman reward signal as C51 — scales align by construction
3: Match distribution + gradient	composed_Q_dq never feeds ANY downstream consumer — gradient structure changes affect only DuelingQHead's own weights
4: Training-correct ≠ math-correct	DuelingQHead's weights are isolated; perturbations only affect V_dq output which has a single, well-defined consumer (PPO advantage)

Every prior failure mode is structurally impossible in Phase 4.

---

5. Phase 4 implementation state

Shipped (this session)

Phase 4.0 (af35bc778):

• rl_dueling_q_forward.cu kernel — V[B] + A[B,N] + composed_Q[B,N] in one fused pass. Grid (B,1,1), block (HIDDEN_DIM,1,1).
• DuelingQHead Rust struct in crates/ml-alpha/src/rl/dueling_q.rs:
  • Config + new() with Xavier init
  • Online weights: w_v_d, b_v_d, w_a_d, b_a_d
  • Target weights: mirrors (soft-update wiring in 4.2)
  • forward() + forward_target() + forward_inner()

Phase 4.1 (13bf277cd):

• rl_dueling_q_bellman_target.cu — argmax + scalar Bellman target
• rl_dueling_q_loss_and_grad.cu — Huber loss + grad_composed
• rl_dueling_q_decompose_and_bwd.cu — Jacobian + outer-product weight grads
• 3 new DuelingQHead methods:
  • build_bellman_target()
  • compute_loss_and_grad()
  • decompose_and_backward_to_weights()

Remaining (next session)

Phase 4.2 — trainer integration (diagnostic mode, ~150 LOC):

Add to IntegratedTrainer:

• dueling_q_head: DuelingQHead field + 4 Adam states (w_v, b_v, w_a, b_a)
• 10+ buffer fields:
  • V_dq[B] / V_dq_tp1[B] (for PPO advantage when wired)
  • A_dq[B,N] (diag)
  • composed_Q_dq[B,N] (for loss)
  • target_composed_Q_dq[B,N] (for Bellman target build)
  • target_value_dq[B] (Bellman target output)
  • loss_pb_dq[B]
  • grad_composed_dq[B,N]
  • per-batch weight grads (4 buffers)
  • reduced weight grads (4 buffers)
• Wiring in step_with_lobsim_gpu_body:
  1. forward(h_t_borrow) → V_dq, A_dq, composed_Q_dq
  2. forward(h_tp1_d) → V_dq_tp1 (for PPO)
  3. forward(sampled_h_t_d) → online composed_Q for loss
  4. forward_target(sampled_h_tp1_d) → target composed_Q
  5. build_bellman_target → target_value_dq
  6. compute_loss_and_grad → grad_composed_dq + loss_pb_dq
  7. decompose_and_backward_to_weights → per-batch grads
  8. reduce_axis0 → final weight grads
  9. Adam step on each weight
  10. Soft-update target net (reuse dqn_target_soft_update_fn)
• DO NOT yet swap v_pred_d in compute_advantage_return — diagnostic-only mode for Phase 4.2

Phase 4.3 — V_dq → PPO swap (~10 LOC):

After 4.2 cluster-validates V_dq trains stably without affecting Plan A v2 metrics:

• Replace v_pred_d → dueling_v_d in compute_advantage_return call
• Replace v_pred_tp1_d → dueling_v_tp1_d
• value_head still trains via MSE on returns (for diagnostic comparison V_dq vs V_scalar)

Validation strategy (per spec §8)

1. Local smoke + compute-sanitizer at b=128 — should produce 0 errors with healthy l_dueling > 0
2. Cluster smoke b=1024 SHORT (5k steps) at Phase 4.2 — verify:
   • qpa_ema near 0 (Plan A v2 unchanged because V_dq doesn't feed PPO)
   • V_dq trains stably (loss decreasing, V values reasonable)
   • dueling_v_at_taken_ema rises over training
3. Cluster full b=1024 20k at Phase 4.3 — verify:
   • qpa healthy throughout
   • PnL matches or exceeds Plan A v2 +$9.3M peak

Kill criteria

Any one triggers abort:

• qpa_ema < -0.3 at any checkpoint
• nan_abort_step > 0
• l_dueling explodes (> 100× Plan A v2's l_q baseline)
• pnl_cum_usd < Plan A v2's at same step by > $5M

---

6. Operational lessons (separate from architectural pearls)

Branch ref trap — hit twice

Argo workflow's git checkout origin/$BRANCH always picks the branch TIP, not the commit-sha param (which is used only for output paths). Lessons:

1. For isolation tests, create a dedicated branch with the specific commit as tip:

   git branch -f ml-alpha-phaseN-only <commit-sha>
   git push origin ml-alpha-phaseN-only --force
   

2. Always verify cluster-side: check the diag JSONL path output (/feature-cache/alpha-rl-runs/$SHA/) matches the expected commit-sha before assuming the cluster is running what you intended.

3. Avoid detached HEAD commits — they create branch ref vs HEAD divergence. After each commit on a worktree, verify git rev-parse $BRANCH == git rev-parse HEAD.

JSONL monitoring beats stdout

Argo workflow template's log-every=5000 only emits 4 visible step markers across a 20k run. The per-step JSONL (diag.jsonl on feature-cache PVC) is the source of truth. Monitor pattern:

kubectl exec -n foxhunt $WF -c main -- bash -c \
  "tail -1 /feature-cache/alpha-rl-runs/$SHA/fold0/diag.jsonl"

Polls every 90s, emits per 500-step milestone. Catches qpa collapse within ~3 minutes of training start.

Kill early on qpa collapse

The q_pi_agree_ema diagnostic is THE most sensitive cross-arch health signal. If qpa drops below -0.3 by step 1000, KILL — the architecture is broken and waiting longer just confirms what's already known. Saved hours of cluster cycles by killing at step 556 (Phase 3.2), 762 (Phase 3.2c), 745 (3.1 isolation), 746 (3.1-fix).

---

7. References

Pearls created this session

• pearl_complement_dont_replace_with_dual_architecture (2026-05-29)
• pearl_complement_architectures_must_be_calibrated_when_mixed (2026-05-30)
• pearl_complement_internal_loss_vs_external_consumers (2026-05-30 + addendum)
• pearl_scalar_v_with_categorical_ce_is_noop (carried from prior session)

Documents written this session

• docs/superpowers/plans/2026-05-29-phase2-post-mortem-and-redesign.md
• docs/superpowers/specs/2026-05-30-phase4-independent-dueling-head-design.md
• This file: docs/superpowers/notes/2026-05-30-phase3-iqn-complement-postmortem-and-phase4-foundation.md

Cluster runs (chronological)

Workflow	Commit	Result	Notes
alpha-rl-8ksnj	fd3174262	✅ Succeeded	Plan A v2 production baseline (+$9.3M peak)
alpha-rl-dxlnm	7790c6815	Failed	First branch ref trap
alpha-rl-hcmk7	a7b0d6d1a	Failed	HIDDEN_DIM=512 OOB (Phase 3.2-pre)
alpha-rl-cs-pq2jg	a7b0d6d1a	Diagnosed	compute-sanitizer found the OOB
alpha-rl-2qbmz	eac382d20	Killed	Phase 2 v2 step 4267 qpa=-0.79
alpha-rl-gvqk5	0fdce0b44	Killed	Phase 3.2 step 556 qpa=-0.96
alpha-rl-nl5c9	8616cf9cb	Killed	Phase 3.2c step 762 qpa=-0.94
alpha-rl-582bn	(wrong branch)	Killed	Second branch ref trap
alpha-rl-xrbgr	6b7c9521d	Killed	Phase 3.1 isolation step 745 qpa=-0.93
alpha-rl-9q2pd	87540e748	Killed	Phase 3.1-fix step 746 qpa=-0.92

10 cluster cycles. Plan A v2 baseline preserved untouched throughout.

Branches on remote (for archaeology)

• ml-alpha-phase3-iqn-complement @ 8616cf9cb (Phase 3.1 + 3.2 + 3.2c)
• ml-alpha-phase3.1-only @ 6b7c9521d (Phase 3.1 isolation test)
• ml-alpha-phase3.1-fix @ 87540e748 (Phase 3.1 + dual output fix)
• ml-alpha-phase4-dueling-head @ 13bf277cd ← Phase 4 foundation, current work
• main (Plan A v2 baseline at fd3174262 via merge)

---

8. Next session opening checklist

When resuming:

1. ☐ Pull ml-alpha-phase4-dueling-head @ 13bf277cd
2. ☐ Read this note + spec docs/superpowers/specs/2026-05-30-phase4-independent-dueling-head-design.md §6
3. ☐ Create worktree (or reuse /tmp/foxhunt-de-tests/phase4-dueling-head)
4. ☐ Start Phase 4.2: add DuelingQHead instance + buffers + Adam states to IntegratedTrainer
5. ☐ Wire 10-step launch sequence in step_with_lobsim_gpu_body
6. ☐ DO NOT swap v_pred_d yet (4.3 is separate commit)
7. ☐ Local smoke + sanitizer at b=128 — expect 0 errors, l_dueling > 0
8. ☐ Cluster smoke b=1024 5k — verify qpa unchanged from Plan A v2
9. ☐ If ✅, Phase 4.3: swap V_dq into compute_advantage_return
10. ☐ Cluster b=1024 20k full validation