foxhunt/docs/superpowers/plans/2026-05-29-phase2-post-mortem-and-redesign.md

Phase 2 v2 Post-Mortem & Redesign Plan

Date: 2026-05-29 Status: Phase 2 v2 KILLED at step 4267 (alpha-rl-2qbmz) — architectural failure confirmed Baseline accepted: Plan A v2 fd3174262 (+$9.3M peak, 20k clean, no NaN) Next decision: Track A (improve baseline) vs Track B (proper dueling redesign) vs Track C (IQN switch)

---

1. Post-Mortem: Why Phase 2 v2 Failed

Three discoveries, layered

Layer 1 — Surface bug (FIXED): v_head_bwd_from_grad.cu had #define HIDDEN_DIM 512 while the rest of the codebase uses 128. Caused 1378 OOB writes at b=1024, mamba2 Adam cascade-failed. Fixed in eac382d20. (Per pearl_use_consts_not_literals_for_structural_dims.)

Layer 2 — Mathematical no-op (ROOT CAUSE): The composition kernel does

composed_logits[b, a, z] = V[b] + A_logits[b, a, z] − mean_a A_logits[b, a, z]

where V[b] is a scalar per batch. The categorical CE loss operates on softmax_z(composed_logits). Two failures:

• Forward null: softmax_z(V[b] + A_centered[a, z]) = softmax_z(A_centered[a, z]) because softmax is translation-invariant in z. V[b] is constant across z for a fixed (b, a), so it shifts all atom logits equally and drops out of the softmax. V has zero effect on the Q distribution.

• Backward null: grad_V[b] = Σ_z grad_composed_logits[b, a_t, z] = Σ_z (softmax[z] − target[z]) = 0 because both softmax and target distributions sum to 1. V receives zero gradient from CE.

The per-z mean centering (− mean_a A_logits[b, a, z]) DOES change the distribution (per-z values vary), but it provides only the identifiability constraint, not a V signal. Empirically, removing the shared atom-mass pattern across actions destroys action-differentiation information rather than improving it.

Layer 3 — Cluster confirmation (DIAGNOSTIC): ISV diagnostic at step 4267 of alpha-rl-2qbmz:

Metric	Value	Verdict
gamma	0.995	✓ correct
reward_scale	0.00139	✓ adapted
entropy	1.95	✓ no collapse
Hold%	38%	✓ no Hold-attractor
wr	0.567	✓ winning more than losing
q_pi_agree_ema	−0.79	🚨 SEVERE Q↔π misalignment
pnl_cum	-$4.5M	trapped, oscillating

q_pi_agree_ema = −0.79 confirms: Q's true ranking has decoupled from π's distillation target. The broken Q distribution (from per-z mean centering distortion) feeds wrong rankings into the distill loss, π drifts AWAY from Q's preferences over training. Result: high wr (Q finds correct direction)

• negative pnl (π's misaligned action selection turns Q's signal into losing trades).

Key learning saved as pearl

pearl_scalar_v_with_categorical_ce_is_noop.md — added to MEMORY.md index.

---

2. Three Strategic Tracks

Track A: Accept Plan A v2 + Improve Reward Shaping

Hypothesis: Plan A v2 (vanilla C51) already achieves +$9.3M. Most of the remaining upside is in reward shaping (per pearl_reward_shape_drives_quality_over_quantity), not architecture.

Targets:

1. Surfer reward terms (entry cost + min-hold + hold-time × win bonus + quadratic carry)
2. Adaptive C51 atom support (widen V_MAX for fat-tail trades)
3. Entropy regularization tuning (avoid late-stage greedy collapse from +$9.3M peak to +$8.5M final)
4. Longer training (50k or 100k steps with proper checkpointing)
5. Walk-forward cross-fold validation for out-of-sample confidence

Effort: ~1-2 days per target, incremental. Low risk, each step measurable in isolation.

Expected upside: 10-30% PnL improvement per change, potential to reach +$15-20M peak through compounding gains. Limited by Q architecture's implicit state-value handling.

Risk: Low. Each change is small, easy to revert. Plan A v2 baseline always available.

When to choose this: Default. Lock in wins, iterate on what's working.

---

Track B: Proper Distributional Dueling C51 (V as distribution over atoms)

Hypothesis: The dueling decomposition WILL help once V is a true distribution. Better long-term architecture, sets up future improvements.

Architecture:

V_logits[b, z]    — distribution over Z atoms for V(s)
A_logits[b, a, z] — per-action advantage distributions

composed_logits[b, a, z] = V_logits[b, z] + A_logits[b, a, z] − mean_a A_logits[b, a, z]
softmax_z(composed_logits[b, a, :]) → Q distribution

Why this works (vs Phase 2 v2):

• V_logits[b, z] varies across z → genuinely shapes each atom's logit independently
• Different z atoms get different V contributions → softmax distribution IS affected by V
• Decompose backward: grad_V_logits[b, z] = grad_composed_logits[b, a_t, z] (no Σ_z reduction — V learns per-atom)
• Identifiability via mean-subtraction preserved → action-specific deviations cleanly separated

Implementation phases:

B.1: V head distributional rewrite (~150 LOC)

• Modify v_head_fwd_bwd.cu: input h_t [B, HIDDEN_DIM] → output v_logits [B, Z]
• Forward: v_logits[b, z] = Σ_c w[z, c] × h_t[b, c] + b[z] (z = 0..Q_N_ATOMS-1)
• Backward: gradient of CE on softmax_z(v_logits) vs target_v_dist
• Weight shape: [Z × HIDDEN_DIM] instead of [1 × HIDDEN_DIM]
• Adam state for new params

B.2: V target net (~50 LOC)

• Mirror of online V head with soft-update τ
• Used by Bellman target build at sampled_h_tp1

B.3: Modify composition/decomposition kernels (~30 LOC)

• composed_q_compose_fwd.cu: read v_logits[b, z] (was v[b])
• composed_q_decompose_bwd.cu: grad_v_logits[b, z] = grad_composed_logits[b, a_t, z] (no atom-sum reduction)
• Same Jacobian for A as before (mean-subtraction preserved)

B.4: Integrated trainer wiring update (~150 LOC)

• Buffer shapes: v_at_sampled_h_t_d [B, Z], v_at_sampled_h_tp1_d [B, Z]
• V target forward at sampled_h_tp1 (separate forward call)
• V backward at sampled_h_t now returns grad_v_logits [B, Z] instead of scalar
• V grad routing: dueling_v_grad_w_per_batch_d [B, Z, HIDDEN_DIM], reduce, sum

B.5: PPO V loss adaptation (~50 LOC)

• v_pred = E[V] = Σ_z p(z|v_logits) × atom_value[z] for advantage computation
• Existing MSE(v_pred, returns) becomes MSE(E[V], returns) — minimal change
• V_dist target for the dueling backward already trains V_logits via composed Bellman CE

B.6: Smoke + cluster validation

• Local b=128 smoke first (compute-sanitizer clean gate)
• Cluster b=1024 20k validation (compare vs Plan A v2 baseline)
• Kill criteria: q_pi_agree_ema ≪ 0, pnl trapped negative, entropy collapse

Total effort: ~430 LOC, 2-3 days focused work.

Risk: Medium. Dueling may not help meaningfully for trading even when architecturally correct. Plan A v2's Q-Thompson + Q→π distill already handles state value implicitly via softmax(E_Q/τ).

Expected upside: Uncertain. Best case +50% (architecture genuinely unlocks better state-value reasoning). Worst case 0% (Plan A v2 already capture all available alpha given current encoder + reward shaping).

When to choose this: After Track A hits a clear ceiling, OR if we want to prepare the architecture for harder problems (more actions, more nuanced state).

---

Track C: IQN-based Q Learning (switch from C51)

Hypothesis: Drop categorical (atom-based) for quantile-based Q learning. Scalar dueling works naturally for IQN: Q(s, a, τ) = V(s, τ) + A(s, a, τ) − mean_a A(s, a, τ) with scalar values at each sampled τ.

Why consider:

• IQN head already exists in codebase (iqn_head, iqn_q_values_d)
• More expressive than C51 (continuous quantile vs discrete atoms)
• Scalar dueling is well-defined at each τ

Effort: Major refactor — change primary Q learner from C51 to IQN. All C51-specific controllers, kernels, loss code, Bellman projection, action selection logic need updating. Estimated 1-2 weeks focused work.

Risk: High. Many controllers were tuned for C51 dynamics (atom support, projection, etc). IQN has different convergence properties. Major regression risk.

When to choose this: Long-term architectural reset, NOT a near-term fix.

---

3. Recommendation

Immediate (this week):

• ✅ Done: Kill alpha-rl-2qbmz, pearl saved, plan documented
• Accept Plan A v2 (fd3174262) as production baseline
• Tag baseline branch / merge to main if appropriate

Next 1-2 weeks: Track A

• Reward shaping iteration (surfer terms, fat-tail atom widening)
• Entropy/exploration tuning to prevent peak-vs-final pnl decay
• Longer training (50k, 100k) with checkpoint+resume
• Walk-forward validation

Optional parallel: Track B spec preparation

• Write detailed kernel + Rust API specs for distributional V
• Don't IMPLEMENT yet — wait for Track A to plateau
• This lets us pivot quickly if Track A hits a ceiling

Long-term: Track C reconsidered if both A and B plateau

---

4. Open Questions / Risks

1. Does Plan A v2 generalize? Single-seed +$9.3M peak. Need walk-forward across folds to confirm it's alpha, not lucky seed.

2. Track A target ranking: Which reward shaping term to try first? Suggest order by confidence: (1) atom widening (low risk, validates fat-tail handling), (2) entropy tuning, (3) hold-time bonus, (4) quadratic carry.

3. Track B V target net cost: Adds soft-update step to existing target net infrastructure (Q target net already exists). Should be additive complexity, not multiplicative.

4. Track B PPO V loss interpretation: Currently V is a critic for PPO advantage. Switching to distributional V means E[V] is the critic — same semantics, just computed differently. Risk: if V_dist is poorly calibrated early, PPO advantages get noisy → policy explores randomly.

5. Should we even keep Q→π distill? Plan A v2 uses it. q_pi_agree_ema is the right diagnostic either way. For Track B, the distillation loss operates on the COMPOSED Q which now genuinely has V signal, so q_pi alignment should be much better.

---

5. References

Code:

• Plan A v2 baseline: commit fd3174262, branch ml-alpha-plan-a-v2
• Phase 2 v2 failed: commit eac382d20, branch ml-alpha-phase2-v2
• Workflows: alpha-rl-8ksnj (A v2 ✅), alpha-rl-2qbmz (v2 ❌, terminated step 4267)
• Spec being superseded: docs/superpowers/specs/2026-05-29-phase2-v2-proper-dueling-c51-design.md

Pearls referenced:

• pearl_scalar_v_with_categorical_ce_is_noop (NEW, this session)
• pearl_use_consts_not_literals_for_structural_dims
• pearl_q_thompson_actor_makes_pi_dead_weight
• pearl_reward_shape_drives_quality_over_quantity
• pearl_dd049d9a4_surfer_baseline_verified
• pearl_two_alpha_modes_surfer_vs_trend

Literature:

• Wang et al. 2016 — Dueling Network Architectures (scalar dueling DQN)
• Bellemare et al. 2017 — A Distributional Perspective on Reinforcement Learning (C51)
• Dabney et al. 2018 — Implicit Quantile Networks (IQN; alt to C51)