# Phase 2 v2 Post-Mortem & Redesign Plan

**Date:** 2026-05-29
**Status:** Phase 2 v2 KILLED at step 4267 (alpha-rl-2qbmz) — architectural failure confirmed
**Baseline accepted:** Plan A v2 `fd3174262` (+$9.3M peak, 20k clean, no NaN)
**Next decision:** Track A (improve baseline) vs Track B (proper dueling redesign) vs Track C (IQN switch)

---

## 1. Post-Mortem: Why Phase 2 v2 Failed

### Three discoveries, layered

**Layer 1 — Surface bug (FIXED):** `v_head_bwd_from_grad.cu` had `#define HIDDEN_DIM 512` while
the rest of the codebase uses 128. Caused 1378 OOB writes at b=1024, mamba2 Adam cascade-failed.
Fixed in `eac382d20`. (Per `pearl_use_consts_not_literals_for_structural_dims`.)

**Layer 2 — Mathematical no-op (ROOT CAUSE):** The composition kernel does
```
composed_logits[b, a, z] = V[b] + A_logits[b, a, z] − mean_a A_logits[b, a, z]
```
where `V[b]` is a **scalar per batch**. The categorical CE loss operates on `softmax_z(composed_logits)`.
Two failures:

- **Forward null:** `softmax_z(V[b] + A_centered[a, z]) = softmax_z(A_centered[a, z])` because
  softmax is translation-invariant in z. V[b] is constant across z for a fixed (b, a), so it shifts
  all atom logits equally and drops out of the softmax. **V has zero effect on the Q distribution.**

- **Backward null:** `grad_V[b] = Σ_z grad_composed_logits[b, a_t, z] = Σ_z (softmax[z] − target[z]) = 0`
  because both softmax and target distributions sum to 1. **V receives zero gradient from CE.**

The per-z mean centering (`− mean_a A_logits[b, a, z]`) DOES change the distribution (per-z values
vary), but it provides only the identifiability constraint, not a V signal. Empirically, removing
the shared atom-mass pattern across actions **destroys** action-differentiation information rather
than improving it.

**Layer 3 — Cluster confirmation (DIAGNOSTIC):** ISV diagnostic at step 4267 of alpha-rl-2qbmz:

| Metric | Value | Verdict |
|---|---|---|
| gamma | 0.995 | ✓ correct |
| reward_scale | 0.00139 | ✓ adapted |
| entropy | 1.95 | ✓ no collapse |
| Hold% | 38% | ✓ no Hold-attractor |
| wr | 0.567 | ✓ winning more than losing |
| **q_pi_agree_ema** | **−0.79** | 🚨 SEVERE Q↔π misalignment |
| pnl_cum | -$4.5M | trapped, oscillating |

q_pi_agree_ema = −0.79 confirms: Q's true ranking has decoupled from π's distillation target. The
broken Q distribution (from per-z mean centering distortion) feeds wrong rankings into the distill
loss, π drifts AWAY from Q's preferences over training. Result: high wr (Q finds correct direction)
+ negative pnl (π's misaligned action selection turns Q's signal into losing trades).

### Key learning saved as pearl

`pearl_scalar_v_with_categorical_ce_is_noop.md` — added to MEMORY.md index.

---

## 2. Three Strategic Tracks

### Track A: Accept Plan A v2 + Improve Reward Shaping

**Hypothesis:** Plan A v2 (vanilla C51) already achieves +$9.3M. Most of the remaining upside is in
**reward shaping** (per `pearl_reward_shape_drives_quality_over_quantity`), not architecture.

**Targets:**
1. Surfer reward terms (entry cost + min-hold + hold-time × win bonus + quadratic carry)
2. Adaptive C51 atom support (widen V_MAX for fat-tail trades)
3. Entropy regularization tuning (avoid late-stage greedy collapse from +$9.3M peak to +$8.5M final)
4. Longer training (50k or 100k steps with proper checkpointing)
5. Walk-forward cross-fold validation for out-of-sample confidence

**Effort:** ~1-2 days per target, incremental. Low risk, each step measurable in isolation.

**Expected upside:** 10-30% PnL improvement per change, potential to reach +$15-20M peak through
compounding gains. Limited by Q architecture's implicit state-value handling.

**Risk:** Low. Each change is small, easy to revert. Plan A v2 baseline always available.

**When to choose this:** Default. Lock in wins, iterate on what's working.

---

### Track B: Proper Distributional Dueling C51 (V as distribution over atoms)

**Hypothesis:** The dueling decomposition WILL help once V is a true distribution. Better long-term
architecture, sets up future improvements.

**Architecture:**
```
V_logits[b, z]    — distribution over Z atoms for V(s)
A_logits[b, a, z] — per-action advantage distributions

composed_logits[b, a, z] = V_logits[b, z] + A_logits[b, a, z] − mean_a A_logits[b, a, z]
softmax_z(composed_logits[b, a, :]) → Q distribution
```

**Why this works (vs Phase 2 v2):**
- V_logits[b, z] varies across z → genuinely shapes each atom's logit independently
- Different z atoms get different V contributions → softmax distribution IS affected by V
- Decompose backward: `grad_V_logits[b, z] = grad_composed_logits[b, a_t, z]` (no Σ_z reduction
  — V learns per-atom)
- Identifiability via mean-subtraction preserved → action-specific deviations cleanly separated

**Implementation phases:**

**B.1: V head distributional rewrite (~150 LOC)**
- Modify `v_head_fwd_bwd.cu`: input h_t [B, HIDDEN_DIM] → output v_logits [B, Z]
- Forward: `v_logits[b, z] = Σ_c w[z, c] × h_t[b, c] + b[z]` (z = 0..Q_N_ATOMS-1)
- Backward: gradient of CE on softmax_z(v_logits) vs target_v_dist
- Weight shape: `[Z × HIDDEN_DIM]` instead of `[1 × HIDDEN_DIM]`
- Adam state for new params

**B.2: V target net (~50 LOC)**
- Mirror of online V head with soft-update τ
- Used by Bellman target build at sampled_h_tp1

**B.3: Modify composition/decomposition kernels (~30 LOC)**
- `composed_q_compose_fwd.cu`: read `v_logits[b, z]` (was `v[b]`)
- `composed_q_decompose_bwd.cu`: `grad_v_logits[b, z] = grad_composed_logits[b, a_t, z]`
  (no atom-sum reduction)
- Same Jacobian for A as before (mean-subtraction preserved)

**B.4: Integrated trainer wiring update (~150 LOC)**
- Buffer shapes: `v_at_sampled_h_t_d [B, Z]`, `v_at_sampled_h_tp1_d [B, Z]`
- V target forward at sampled_h_tp1 (separate forward call)
- V backward at sampled_h_t now returns `grad_v_logits [B, Z]` instead of scalar
- V grad routing: `dueling_v_grad_w_per_batch_d [B, Z, HIDDEN_DIM]`, reduce, sum

**B.5: PPO V loss adaptation (~50 LOC)**
- `v_pred = E[V] = Σ_z p(z|v_logits) × atom_value[z]` for advantage computation
- Existing MSE(v_pred, returns) becomes MSE(E[V], returns) — minimal change
- V_dist target for the dueling backward already trains V_logits via composed Bellman CE

**B.6: Smoke + cluster validation**
- Local b=128 smoke first (compute-sanitizer clean gate)
- Cluster b=1024 20k validation (compare vs Plan A v2 baseline)
- Kill criteria: q_pi_agree_ema ≪ 0, pnl trapped negative, entropy collapse

**Total effort:** ~430 LOC, 2-3 days focused work.

**Risk:** Medium. Dueling may not help meaningfully for trading even when architecturally correct.
Plan A v2's Q-Thompson + Q→π distill already handles state value implicitly via softmax(E_Q/τ).

**Expected upside:** Uncertain. Best case +50% (architecture genuinely unlocks better
state-value reasoning). Worst case 0% (Plan A v2 already capture all available alpha given
current encoder + reward shaping).

**When to choose this:** After Track A hits a clear ceiling, OR if we want to prepare the
architecture for harder problems (more actions, more nuanced state).

---

### Track C: IQN-based Q Learning (switch from C51)

**Hypothesis:** Drop categorical (atom-based) for quantile-based Q learning. Scalar dueling works
naturally for IQN: `Q(s, a, τ) = V(s, τ) + A(s, a, τ) − mean_a A(s, a, τ)` with scalar values at
each sampled τ.

**Why consider:**
- IQN head already exists in codebase (`iqn_head`, `iqn_q_values_d`)
- More expressive than C51 (continuous quantile vs discrete atoms)
- Scalar dueling is well-defined at each τ

**Effort:** Major refactor — change primary Q learner from C51 to IQN. All C51-specific
controllers, kernels, loss code, Bellman projection, action selection logic need updating.
Estimated 1-2 weeks focused work.

**Risk:** High. Many controllers were tuned for C51 dynamics (atom support, projection, etc).
IQN has different convergence properties. Major regression risk.

**When to choose this:** Long-term architectural reset, NOT a near-term fix.

---

## 3. Recommendation

**Immediate (this week):**
- ✅ Done: Kill alpha-rl-2qbmz, pearl saved, plan documented
- Accept Plan A v2 (`fd3174262`) as production baseline
- Tag baseline branch / merge to main if appropriate

**Next 1-2 weeks: Track A**
- Reward shaping iteration (surfer terms, fat-tail atom widening)
- Entropy/exploration tuning to prevent peak-vs-final pnl decay
- Longer training (50k, 100k) with checkpoint+resume
- Walk-forward validation

**Optional parallel: Track B spec preparation**
- Write detailed kernel + Rust API specs for distributional V
- Don't IMPLEMENT yet — wait for Track A to plateau
- This lets us pivot quickly if Track A hits a ceiling

**Long-term: Track C reconsidered if both A and B plateau**

---

## 4. Open Questions / Risks

1. **Does Plan A v2 generalize?** Single-seed +$9.3M peak. Need walk-forward across folds to
   confirm it's alpha, not lucky seed.

2. **Track A target ranking:** Which reward shaping term to try first? Suggest order by
   confidence: (1) atom widening (low risk, validates fat-tail handling), (2) entropy tuning,
   (3) hold-time bonus, (4) quadratic carry.

3. **Track B V target net cost:** Adds soft-update step to existing target net infrastructure
   (Q target net already exists). Should be additive complexity, not multiplicative.

4. **Track B PPO V loss interpretation:** Currently V is a critic for PPO advantage. Switching to
   distributional V means E[V] is the critic — same semantics, just computed differently. Risk: if
   V_dist is poorly calibrated early, PPO advantages get noisy → policy explores randomly.

5. **Should we even keep Q→π distill?** Plan A v2 uses it. q_pi_agree_ema is the right diagnostic
   either way. For Track B, the distillation loss operates on the COMPOSED Q which now genuinely
   has V signal, so q_pi alignment should be much better.

---

## 5. References

**Code:**
- Plan A v2 baseline: commit `fd3174262`, branch `ml-alpha-plan-a-v2`
- Phase 2 v2 failed: commit `eac382d20`, branch `ml-alpha-phase2-v2`
- Workflows: alpha-rl-8ksnj (A v2 ✅), alpha-rl-2qbmz (v2 ❌, terminated step 4267)
- Spec being superseded: `docs/superpowers/specs/2026-05-29-phase2-v2-proper-dueling-c51-design.md`

**Pearls referenced:**
- `pearl_scalar_v_with_categorical_ce_is_noop` (NEW, this session)
- `pearl_use_consts_not_literals_for_structural_dims`
- `pearl_q_thompson_actor_makes_pi_dead_weight`
- `pearl_reward_shape_drives_quality_over_quantity`
- `pearl_dd049d9a4_surfer_baseline_verified`
- `pearl_two_alpha_modes_surfer_vs_trend`

**Literature:**
- Wang et al. 2016 — Dueling Network Architectures (scalar dueling DQN)
- Bellemare et al. 2017 — A Distributional Perspective on Reinforcement Learning (C51)
- Dabney et al. 2018 — Implicit Quantile Networks (IQN; alt to C51)
