# Distributional RL Action Selection: Thompson Sampling on C51 + IQN

**Date**: 2026-04-27 (last revised; original 2026-04-26)
**Status**: Design approved, awaiting implementation plans (4 sub-plans)
**Origin**: Task #178 — Design proper C51 expected-Q Hold/Flat bias mitigation
**Predecessor fixes from val-Flat-collapse investigation (all KEPT — real fixes for real bugs)**:
- Kelly cap warm-branch (`0c9d1ee39`) — post-decision physics layer
- Train Return display + Sharpe annualization (non-tau parts of `7a3d88646`) — display/metric layer
- Direction-branch Boltzmann tau-floor (tau part of `7a3d88646`) and adaptive eps_dir floor (`d54b49efc`) — gates inside direction-branch action selection. Phase 2 replaces direction-branch action selection wholesale (Thompson replaces eps-greedy + Boltzmann), so these gates become dead code FOR THE DIRECTION BRANCH ONLY. Magnitude/order/urgency keep their existing eps-greedy + Boltzmann + tau-floor + EPS_FLOOR paths intact. Phase 3 cleanup deletes only the direction-branch unreachable code, preserving all logic for other branches.

---

## The Pearl (project-wide principle)

> **Whenever you reduce a probability distribution or ensemble to a scalar for action selection, you must preserve the second moment (uncertainty) — or use a sampling-based selector that doesn't reduce.**
>
> Aggregating to `E[Q]` collapses `(mean=0, σ=0)` and `(mean=−ε, σ=large)` to neighbouring scalars, where the deterministic action wins regardless of latent option value. Boltzmann/argmax on `E[Q]` alone produces structural bias toward low-variance actions. Thompson sampling — drawing one value from the full distribution per action and picking argmax of those samples — never aggregates and is parameter-free.

This pearl applies to ALL distributional or ensemble-based value methods (C51, IQN, Bayesian RL, dropout uncertainty, ensemble Q-heads). It is the upstream principle for the C51 Hold/Flat bias documented here, and a recurring pattern across distributional value methods generally. Other historic bugs in the DQN system (Kelly bootstrap deadlock, val-Flat-collapse, magnitude saturation) had different mechanisms — formula collapses, clamp bugs, kernel bugs — though they share the broader theme of "information lost between value computation and decision."

---

## The Problem

### Statement

C51 represents per-direction Q as a discrete distribution over atoms `[v_min, v_max]`. IQN represents the same Q as quantile values at fixed `τ ∈ {0.05, 0.25, 0.5, 0.75, 0.95}`.

Action selection currently uses scalar `E[Q_d]` via Boltzmann:
```
P(d) ∝ exp(E[Q_d] / τ_boltzmann)
```

Where the existing `compute_expected_q` kernel produces `E[Q_d]` per direction. The current implementation reads from one or both distributional sources blended by training schedule (the C51-vs-MSE warmup blend, see project memory `dqn-session-*`). Phase 2 replaces this single kernel call with **two operations** (one for E[Q] used at eval and conviction; one for samples used at training Thompson) that read from C51 atom probabilities `p_d_i` AND IQN quantile values `Q_τ(d)` independently. Implementation detail: the eval-time E[Q] computation is `E[Q_d] = 0.5·(Σᵢ p_d_i · v_i) + 0.5·mean({Q_τ(d)})` — equal weighting, no tuned blend. Phase 0 Test 0.F verifies this against the existing kernel's E[Q] output.

### The asymmetry the model rationally learns

- **Flat**: trajectory return ≡ 0 (no position → no PnL change). After training, `C51_flat → δ(v=0)`, so `E[Q_flat] = 0` AND `σ(Q_flat) ≈ 0`.
- **Long/Short**: stochastic trajectory return minus tx_cost. Without discovered edge, `E[Q_directional] ≈ −tx_cost < 0`, AND `σ(Q_directional) > 0` (real return variance).

Aggregation to scalar `E[Q]` discards σ. Boltzmann sees only `(0, −ε, −ε, 0)` for `(short, hold, long, flat)` and rationally prefers Flat. There is no signal in `E[Q]` alone that Long has option value the deterministic Flat does not.

### The bias might be CORRECT BEHAVIOUR (Gap 4 verification)

Steel-manning the model: "I see Long loses tx_cost without compensating edge → Q[Long] < Q[Flat] → I correctly choose Flat." This is statistically right under the data the model has seen.

If the underlying data **has no edge**, no amount of exploration helps — Thompson sampling still leads to losing trades; Q[Long] stays below Q[Flat]; eval correctly avoids trading. The fix only matters if the data DOES have edge but the model's exploration is too weak to discover it.

Phase 0 includes a synthetic-edge test: construct a state with KNOWN positive Long expected value, verify Thompson sampling discovers it within N epochs. If even synthetic edge is undiscoverable, the structural fix is insufficient and reward shaping must change.

### Empirical confirmation of the symptom

Three runs (`train-dhmp6`, `train-bscl2` post-Kelly-fix) show consistent post-training behaviour:
```
Epoch 1 (random init):   val_dir_dist [S=.236 H=.181 L=.411 F=.173]   active=64.6%
Epoch 2 (one epoch):     val_dir_dist [S=.135 H=.358 L=.142 F=.364]   active=27.7%
```

The 35→72% Hold+Flat shift in one epoch is the C51 attractor in action. Two attempts at the symptom layer (ISV-adaptive Boltzmann tau floor, adaptive eps_dir floor) produced no measurable effect — confirming the issue is in the aggregation step, not in sampling sharpness or exploration rate.

---

## The Fix Design

### Architecture overview

```
┌─────────────────────────────────────────────────────────────────┐
│  Per-direction Q-distribution sources                            │
│  ┌──────────────┐         ┌──────────────┐         ┌──────────┐ │
│  │ C51 atoms    │         │ IQN quantiles│         │ Ensemble │ │
│  │ p_d_i, v_i   │         │ Q_τ(d)       │         │ Q_m(d)   │ │
│  └──────┬───────┘         └──────┬───────┘         └────┬─────┘ │
│         │                        │                       │       │
│         ▼                        ▼                       ▼       │
│   sample_C51(d)             sample_IQN(d)         sample_ENS(d) │
│         │                        │                       │       │
│         └────────┬───────────────┘                       │       │
│                  ▼                                       │       │
│         q_sample(d) = mean(C51, IQN)              (future)       │
│                  │                                               │
│                  ▼                                               │
│       TRAINING action selection (modified):                       │
│       dir_idx = argmax(q_sample)        // Thompson              │
│                                                                   │
│       EVAL action selection:                                      │
│       dir_idx = argmax((E[Q_C51] + E[Q_IQN]) / 2)                │
│       // raw expected value, no exploration noise                │
│       // (NOTE: behavior change from current Boltzmann-with-tau) │
└─────────────────────────────────────────────────────────────────┘
```

### Thompson sampling math

**From C51 atoms**: sample atom index by inverse-CDF over `p_d_i`, return `atom_values[sampled_idx]`.

```
u = uniform(0,1)
cum = 0
for a in 0..n_atoms:
    cum += p_d[a]
    if u < cum: return atom_values[a]
return atom_values[n_atoms-1]   // numerical fallback
```

**From IQN quantiles**: sample `τ ∈ [0,1]` uniformly, interpolate from the 5 fixed-quantile Q values.

```
u = uniform(0,1)
// Linear interpolation between adjacent fixed quantiles
// Quantile boundaries: 0.05, 0.25, 0.50, 0.75, 0.95
sample = interpolate_quantile(u, Q_τ)
return sample
```

**Combined**: equal-weight average of C51 sample and IQN sample (no tuned blend weight; both distributions are equally valid uncertainty representations of Q).

```
q_sample(d) = 0.5 × (sample_C51(d) + sample_IQN(d))
```

**Direction selection**: `dir_idx = argmax(q_sample[0..b0_size])`.

### Train vs eval: distinct selectors

**TRAINING** (`!eval_mode`):
- Direction branch: Thompson sampling per above
- Magnitude/order/urgency: existing Boltzmann (no change)

**EVAL** (`eval_mode`):
- Direction branch: `dir_idx = argmax(E_C51 + E_IQN)` — pure argmax of summed expected value (the `/2` factor is monotonic and irrelevant for argmax). No sampling, no exploration.
- Magnitude/order/urgency: existing Boltzmann with adaptive tau. Eval mode already sets eps=0 in the existing kernel (`eval_mode = (eps_start < 1e-6 && eps_end < 1e-6)`), so the random-sample path is naturally skipped; no ε-greedy override semantic changes. Boltzmann remains near-deterministic for tight Q-spreads.

**Why distinct selectors are essential**: Eval is the policy's HONEST report of what action it recommends. Thompson sampling at eval would inject exploration noise into reported decisions, conflating "model recommends Long" with "model is uncertain about Long." For trading, where reported recommendations drive real capital, eval must report the policy's mode without exploration overlay.

**This is a behavior change from current eval semantics**: today, `eval_mode=true` sets eps=0 but the direction Boltzmann softmax with adaptive tau still applies — eval picks are "near-deterministic" (sharp Boltzmann) but not strictly argmax. Post-Phase-2, eval direction picks become **strictly argmax of mean E[Q_C51]+E[Q_IQN]**. Implication: post-fix `val_dir_dist` will likely be more concentrated (close to 100% on whichever direction has highest E[Q]) than current Boltzmann-soft eval. This is HONEST — it's exactly what the model recommends — but explicit acknowledgment so readers don't expect Boltzmann-style soft eval distributions.

**Implication for verification**: val_dir_dist may persist in showing Hold/Flat bias UNTIL Thompson exploration during training generates enough Long/Short experience for `E[Q_directional]` to genuinely exceed `E[Q_flat]` on profitable opportunities. This is a longer feedback loop than 5 epochs. See verification protocol below.

### Direction-branch only — DO NOT extend

The bias is structurally direction-specific because Flat is the only action with deterministic 0-PnL distribution. Magnitude (Quarter/Half/Full) all generate information regardless of which is chosen.

**Critical caveat**: the magnitude branch likely has its own σ-bias dynamics (Full position has larger return variance than Quarter — wider Thompson sample range), but in the OPPOSITE direction from direction branch: σ-aware sampling would systematically PREFER Full, potentially worsening the magnitude saturation pattern documented in project memory (`Q_full → +49`, `Q_half → -47`). The exact interaction needs empirical investigation before any extension. **NEVER extend Thompson to magnitude branch without separate design and verification.** Phase 0.F should also report per-magnitude E[Q] and σ to inform that future analysis.

### No κ to tune

Thompson sampling is parameter-free. There is no temperature, no exploration coefficient, no decay schedule. The exploration intensity is determined entirely by the SHAPE of the learned Q-distributions:
- High σ in Q[Long] → frequent samples above Q[Flat] → frequent Long picks
- Low σ in Q[Long] (model converged) → rare samples above Q[Flat] → rare Long picks (correct exploit)

This is more principled than UCB (which requires κ choice) and more principled than ε-greedy (which requires ε schedule).

### Conviction stays E[Q]-based

Conviction (`q_range / max(isv[21], q_range)`) used by Kelly cap warmup floor must reflect the model's actual learned Q-spread, NOT a Thompson sample (which is stochastic and would jitter the Kelly cap). Conviction computation continues to read raw `E[Q]`, not `q_sample`. (Implementation: compute E[Q] per direction once, reuse for conviction AND eval-mode argmax — no redundant computation.)

---

## Interaction Matrix with Existing Mechanisms

21 existing mechanisms in the DQN system were audited for interaction with Thompson. Three categories:

### Category 1 — Compose with Thompson (no change required, 13 mechanisms)

| Mechanism | Layer | Interaction |
|---|---|---|
| **Counterfactual reward augmentation** (cf_ratio) | Reward | Augments per-bar reward; propagates through Q-target. Compatible. Provides hindsight Q signal that updates Q values for actions NOT taken (especially relevant for the Hold-stuck regime). |
| **B.2 novelty bonus** (Flat→Positioned) | Reward | Adds bonus on directional entry. Reduces tx_cost penalty during Thompson exploration. Compatible. |
| **PopArt normalization** | Reward / target | Q-target normalization. Doesn't affect action selection. Compatible. |
| **Saboteur** (training reward perturbation) | Reward | Adds noise to training rewards. Compatible. |
| **Curiosity-driven reward** (intrinsic motivation) | Reward | Rewards unexpected states. **Orthogonal to Thompson** — Thompson explores in action space, curiosity in state space. Multiplicatively beneficial: Thompson explores actions whose Q is uncertain; curiosity explores states whose dynamics are uncertain. Both desirable. |
| **NoisyNets / VSN σ params** | Network forward | Each forward pass produces different Q outputs (parameter-space sampling). **NoisyNets is already a form of training-time Thompson at the parameter level.** Output-space Thompson on direction stacks on top — total exploration = parameter-space ⊗ output-space. Multiplicatively beneficial. |
| **Distillation** | Knowledge transfer | Trains via auxiliary target Q. Compatible. |
| **CQL (Conservative Q-Learning)** | Loss | Penalizes overestimation in offline RL. Compatible — CQL operates on the LOSS while Thompson operates on action selection. |
| **Target Polyak EMA** | Slow target network | Q-target uses target Q; Thompson uses online Q. Compatible. |
| **HER (Hindsight Experience Replay)** | Replay augmentation | Augments replay with hindsight goals. Compatible. |
| **Prioritized Experience Replay (PER)** | Replay sampling | Samples from replay by priority. Compatible. |
| **Replay buffer warm-start (scripted policy)** | Initial experience | Generates initial replay. Thompson takes over after warm-up. Compatible. |
| **Multi-fold validation harness** | Eval orchestration | Runs multiple val folds. Compatible. |

### Category 2 — Trivially adapt to Thompson (one-line changes, 4 mechanisms)

| Mechanism | Layer | Required adaptation |
|---|---|---|
| **D7/N7 contrarian sign flip** | Action select | Currently negates Q before Boltzmann (`q_sign = -1.0f`). With Thompson: negate the Q SAMPLE per direction. One-line change in the Thompson kernel: `q_sample_d *= q_sign`. |
| **Cosine epsilon schedule for ε-greedy** | Action select | Currently sets `eps_dir`, `eps_mag`, `eps_ord`, `eps_urg` from cosine schedule. Direction branch under Thompson no longer uses `eps_dir`. Schedule continues to apply to mag/ord/urg branches. No code change beyond Phase 3 cleanup of dead direction-eps code. |
| **Per-sample epsilon (IQL expectile gap)** | Action select | Same as above — no longer affects direction; still affects mag/ord/urg. |
| **Adaptive Boltzmann tau** | Action select | Direction branch no longer uses Boltzmann under Thompson. Tau still applies to mag/ord/urg. No change beyond Phase 3 cleanup of direction-tau code. |

### Category 3 — Take precedence over Thompson (hard constraints, 4 mechanisms)

| Mechanism | Layer | Precedence rule |
|---|---|---|
| **Plan-based action lock** | Action select | When plan is active (PS_PLAN_TARGET_BARS > plan_thr), force `dir_idx = current_position_sign` (existing logic). **Plan lock takes precedence over Thompson** — Thompson sample is computed but discarded if plan is active. Same precedence as currently with Boltzmann. |
| **Per-magnitude Kelly cap** | Post-decision physics | Operates AFTER action selection. Independent of how dir_idx was chosen. Already handled via existing fix. |
| **Trail stop** | Post-decision physics | Operates AFTER action selection. Independent. |
| **Capital floor breach** | Post-decision physics | Forces flat regardless of action. Takes precedence over everything. Already handled. |

---

## Outside-the-Box Improvements (v2 enhancements after v1 ships)

Vanilla Thompson is the v1 fix. The following are documented enhancements to consider AFTER Phase 2 stabilizes — each addresses a specific limitation of vanilla Thompson revealed by our system's quirks.

### v2.1 — Triple-source Thompson (C51 + IQN + Ensemble)

We have three uncertainty sources: C51 atoms, IQN quantiles, AND **ensemble Q-head** (currently only used for `ens_agree` metric). Vanilla v1 averages C51+IQN samples. v2 incorporates ensemble:

```
sample_d = (sample_C51(d) + sample_IQN(d) + sample_member_uniform(d)) / 3
```

Or simpler: round-robin per-bar between {C51, IQN, ensemble}. Three-way uncertainty is more robust to any single source being miscalibrated.

**Enables**: variance disagreement detection across sources (warning if they disagree).

### v2.2 — Persistent Thompson (anti-churn for HFT)

Vanilla Thompson at every bar may oscillate Long↔Short when Q's are similar, paying tx_cost each time. The Q-distributions may genuinely have similar means; vanilla Thompson treats each bar independently and IS allowed to flip.

**Persistent Thompson** biases sampling toward the current direction:

```
if current_direction == d:
    q_sample_d *= (1 + persistence_bonus)
```

Where `persistence_bonus` is ISV-driven (high conviction → high persistence; uncertain → 0). When the model has high conviction in a direction, persistence locks in. When uncertain, vanilla Thompson behavior.

This addresses a real cost driver: each direction switch is a tx_cost. Reducing oscillation could improve net Sharpe directly.

**Risk**: persistence might delay legitimate direction reversals. Need ISV signal that drops persistence when state changes (regime shift, ADX cross, etc.).

### v2.3 — CVaR-aware eval (risk-adjusted deployment)

Vanilla v1 eval = `argmax E[Q]`. For a trading system where compounded loss is existential, **risk-adjusted argmax** is safer:

```
Q_risk_adjusted(d) = E[Q_d] − λ · CVaR_α[Q_d]
dir_idx = argmax(Q_risk_adjusted)
```

CVaR_α = expected return conditional on the worst α% tail. Computable from C51 atoms + IQN quantiles directly (no new compute). λ is risk aversion parameter (default 0 = current vanilla; > 0 = increasingly pessimistic).

**Use case**: for production deployment with real capital, set λ > 0 to require positive CVaR before taking directional action. Avoids "expected positive but tail-risky" trades.

**Implementation**: λ is the only tuned param. Could be ISV-driven from realized drawdown.

### v2.4 — Information-Directed Sampling (IDS)

Russo & Van Roy (2014). Pick action minimizing `regret² / information_gain`. More efficient than vanilla Thompson when learning saturates — IDS exploits more aggressively when info gain is low.

**For our case**: IDS would correctly recognize when training has plateaued on Long/Short discovery and shift to exploit Hold/Flat (the rational no-trade decision when no further info is available).

**Cost**: requires computing posterior entropy reduction per action. Significant kernel work.

**Defer to v2/v3** unless v1 verification shows learning saturates suspiciously fast.

### v2.5 — Hierarchical Thompson on (trade vs no-trade) → (which direction)

Our 4-direction action space has natural structure: {Short, Hold, Long, Flat} = {trade, no-trade} × {sign, mode}.

**Hierarchical sampling**:
1. Sample "trade decision" from {best_trade_Q_sample vs best_no_trade_Q_sample}
2. If trade selected: sample Long vs Short
3. If no-trade selected: sample Hold vs Flat

The structural argument is NOT about action-space balance (Long/Short already = 2/4 = Hold/Flat). It's about DECOUPLING the decisions: with flat sampling, a slight σ asymmetry across the 4 individual options biases the choice. With hierarchical sampling, the "trade vs no-trade" decision is made between the BEST OF EACH cluster — so a single highly-uncertain Long with positive sample can pull "trade" up against the best of {Hold, Flat}, even if the *other* trade option (Short) has negative sample.

**Risk**: complicates implementation; v1 vanilla Thompson is simpler to verify.

### v2.6 — Composition with curiosity-driven exploration

Curiosity reward already exists. v2 enhancement: explicitly couple curiosity bonus to Thompson sample magnitude. When curiosity is high (state is unfamiliar), inflate Thompson sample variance. When curiosity is low (state is familiar), let Thompson reflect learned Q.

**Composes naturally** with existing curiosity reward path. Implementation is small if curiosity_inference_kernel already produces a per-state scalar.

### Decision: v1 ships vanilla Thompson; v2 enhancements documented and deferred

v1 is the design described in Phases 0-3 below. Each v2 enhancement is documented above for future implementation. None are required for v1 to be a structural fix; they're refinements for production maturity.

The decision to defer v2 keeps Phase 2 verification clean: we measure pure Thompson's effect, not Thompson + N other simultaneous changes. If v1 verification falls short, we add v2 elements one at a time (separate task per enhancement, separate spec).

---

## Phased Implementation

### Phase 0 — GPU-Direct Hypothesis Verification + Synthetic Edge Test

**Goal**: Prove (or disprove) the bias hypothesis by exercising the actual production kernels (or thin wrappers around them) with controlled inputs. No CPU mirror functions — tests target the real GPU code path so kernel-only bugs (CUDA index errors, race conditions, type cast mistakes) are caught at the unit-test level rather than waiting for production training.

**Deliverables**:

A test module `crates/ml/src/trainers/dqn/distributional_q_tests.rs` containing:
- A small kernel wrapper `launch_thompson_direction(c51_probs, atom_values, iqn_quantiles, rng_seed) -> [u8; b0_size]` that launches a tiny test-only CUDA kernel with the SAME math the Phase 2 kernel will use. This becomes the actual production code in Phase 2; Phase 0 tests it standalone first.
- A small kernel wrapper `launch_argmax_eq(c51_probs, atom_values, iqn_quantiles) -> u8` for eval-mode argmax of E[Q_C51]+E[Q_IQN].
- Helper `compute_sigma_from_c51_atoms(probs, atoms) -> f32` and `compute_sigma_from_iqn_quantiles(qs) -> f32` — these run on the GPU via thin kernel wrappers and exist as observation utilities for Test 0.F.

Each test launches the kernel(s), reads outputs back, asserts properties. All tests are `#[ignore]` (GPU-required); explicitly run via `cargo test -p ml --lib distributional_q_tests -- --ignored --nocapture`.

**Six tests** (5 GPU-direct kernel tests + 1 CPU algorithmic-property test):

- **0.A — Bias mechanism reproduces (GPU)**: build a synthetic `c51_probs[4 dirs × n_atoms]` and `iqn_quantiles[4 dirs × 5]` representing the failure mode (Flat = δ(0); Long/Short = wide distribution centered at −0.001). Launch `launch_argmax_eq` AND launch `launch_thompson_direction` 10000× with different seeds. Assert: argmax picks Flat 100% of trials; Thompson picks (Long+Short) ≥ 30% of trials.

- **0.B — Thompson C51 sample distribution matches input (GPU)**: launch `launch_thompson_direction` 100k× with single direction having known atom probabilities `p = [0.1, 0.7, 0.2]` over atoms `[−1, 0, +1]`. Histogram returned samples (binned to atom indices). Assert: each bin within 1% of input probability.

- **0.C — Thompson IQN sample distribution matches quantile inputs (GPU)**: similar to 0.B but with synthetic quantile values matching a known Gaussian. Empirical quantiles of 100k samples within 5% of analytic quantiles (Kolmogorov-Smirnov–style check).

- **0.D — Thompson reverses bias on synthetic failure mode (GPU)**: same setup as 0.A but assertion focused on directional preference. With `(E_flat=0, σ_flat=0, E_long=−0.001, σ_long=0.05)`: argmax picks Long with probability 0 (deterministic Flat); Thompson picks Long with probability ≥ 0.20 over 10k seeds. Quantitatively confirms σ-driven exploration.

- **0.E — Synthetic edge discovery (CPU, Q-learning simulation)**: a 100-iteration Q-learning loop on a 1-state 2-action bandit with KNOWN edge. Tests an ALGORITHMIC PROPERTY (does Thompson exploration converge to right answer when edge exists?), not a kernel correctness property. CPU is appropriate because no production kernel is being tested — only the math of "samples from a tightening distribution converge to the underlying Q if edge exists." Confirm: with argmax-only updates, Q[Long] stays below Q[Flat]; with Thompson updates, Q[Long] > Q[Flat] by iteration 50.

- **0.F — Real checkpoint extraction (GPU, integration)**: load a "converged" checkpoint. **"Converged" definition**: a checkpoint from a training run of ≥60 epochs (matching prior validation runs in project history per memory `dqn-session-2026-03-25.md`) where val_sharpe has stabilised (no >10% change over the last 10 epochs). Run the existing forward pass to extract C51 atoms + IQN quantiles per direction on a representative val state batch. Use `compute_sigma_from_c51_atoms` and `compute_sigma_from_iqn_quantiles` to check structural assertions:
  - σ_C51[FLAT] < 0.01 × σ_C51[LONG] (Flat dist is much tighter than Long; structural)
  - σ_IQN[FLAT] < 0.01 × σ_IQN[LONG] (same for IQN)
  - E[Q_FLAT] > E[Q_LONG] (the bias actually exists in this checkpoint)
  - `launch_argmax_eq` picks FLAT
  - `launch_thompson_direction` over 1000 trials produces P(LONG) + P(SHORT) ≥ 0.20

**Exit gate**: All 6 tests pass. Phase 0 first ENUMERATES existing checkpoints (project history has multiple long training runs from `train-7rgqd` 60-epoch validation per task #80, `train-bv2n5`, etc.); if any qualifies as "converged" per the above definition, use it. Only if none qualifies does Phase 0 train a new 60-epoch checkpoint (~3 hours on L40S, not "1 day").

**Time budget**: 2-3 days. GPU test setup (kernel wrappers, fixture loading) ~half day; tests 0.A-0.D ~2 hours each (8 hours); checkpoint enumeration ~1 hour; Test 0.F ~3 hours; Test 0.E (CPU simulation) ~2 hours; new checkpoint training only if needed (~3 hours runtime, can run while implementer works on other tests).

**Why GPU-direct rather than CPU mirror**: tests now exercise the actual production kernel math, catching CUDA-specific bugs (index errors, race conditions, type cast mistakes) that CPU mirror tests would miss. Local development uses RTX 3050 GPU per `memory/user_dev_environment.md`. CI runs `--ignored` flag to skip GPU tests on CPU-only runners; GPU runners (Argo training pods, local) run them as part of the L1 verification gate.

### Phase 1 — Audit & Repair Existing Reward Levers (TDD, conditional)

**Goal**: Verify B.2 novelty bonus, counterfactual reward, PopArt, and Q-target propagation work as designed. **If a bug is found, fix it before Phase 2.**

**Six unit tests** (unchanged from prior design):
- **1.A — B.2 novelty bonus producer formula matches kernel math**
- **1.B — B.2 consumer wires to correct (i,t) reward slot**
- **1.C — Counterfactual reward sign + symmetry**
- **1.D — Reward composition matches HEALTH_DIAG reward_split**
- **1.E — Q-target propagates realized rewards (atom mass shift toward observed return)**
- **1.F — PopArt preserves relative reward ordering between directions**

**Exit gate**: All 6 pass; OR a bug is found, fixed, and tests re-run green.

**Time budget**: 1 day if all 6 tests pass green; 2-5 days if any test fails and bug requires fixing (depends on scope of bug).

### Phase 2 — Thompson Sampling Integration

**Goal**: Land the structural fix.

#### Component 4.1: `experience_action_select` modification (no new kernel needed)

`crates/ml/src/cuda_pipeline/experience_kernels.cu`:

The existing kernel already receives C51 atoms and IQN quantiles (or has access to them through Q-value buffers). Modify the direction branch to do Thompson sampling at training, argmax of E[Q] at eval. Magnitude/order/urgency unchanged.

```c
// Direction branch (existing eps-greedy / Boltzmann replaced).
// dir_idx is declared earlier in the kernel scope (existing variable).
// `i`, `timestep`, `rng_ctr` are the existing Philox stream parameters
// (rng_ctr is post-incremented inside the existing kernel; usage here
// matches the existing pattern at experience_kernels.cu:858 and below).
if (eval_mode) {
    // EVAL: argmax of (E_C51 + E_IQN) — /2 is monotonic, dropped for argmax
    float best_q = -CUDART_INF_F;          // CUDA-style infinity macro
    int best_d = 0;
    for (int d = 0; d < b0_size; d++) {
        float e_c51 = compute_e_c51(c51_probs, atom_values, d);
        float e_iqn = compute_e_iqn(iqn_quantiles, d);
        float e_joint = e_c51 + e_iqn;     // argmax-equivalent to (e_c51 + e_iqn) / 2
        if (e_joint > best_q) { best_q = e_joint; best_d = d; }
    }
    dir_idx = best_d;
} else {
    // TRAINING: Thompson sampling — sample from C51 + IQN, argmax of samples
    float best_sample = -CUDART_INF_F;
    int best_d = 0;
    for (int d = 0; d < b0_size; d++) {
        float u_c51 = philox_uniform(i, timestep, rng_ctr++);
        float s_c51 = sample_c51(c51_probs + d * n_atoms, atom_values, n_atoms, u_c51);
        float u_iqn = philox_uniform(i, timestep, rng_ctr++);
        float s_iqn = sample_iqn(iqn_quantiles + d * N_IQN_QUANTILES, u_iqn);
        float q_sample = s_c51 + s_iqn;    // argmax-equivalent to 0.5*(s_c51+s_iqn)
        if (q_sample > best_sample) { best_sample = q_sample; best_d = d; }
    }
    dir_idx = best_d;
}
```

Where `N_IQN_QUANTILES` is the existing project constant (currently 5: τ ∈ {0.05, 0.25, 0.5, 0.75, 0.95} per task #147). Should be referenced by name, not hardcoded `5`, to track future quantile-set changes.

`CUDART_INF_F` is the standard CUDA float infinity macro (`<math_constants.h>`); use whichever existing convention the codebase uses.

`compute_e_c51`, `compute_e_iqn`, `sample_c51`, `sample_iqn`: device-inline functions, mirrors of CPU functions tested in Phase 0.

The existing direction-branch ε-greedy code path is REMOVED (Thompson IS the exploration mechanism). Direction-branch Boltzmann is REMOVED for both train and eval.

Magnitude/order/urgency Boltzmann + ε-greedy: unchanged.

#### Component 4.2: Buffer wiring

The existing `experience_action_select` kernel currently consumes `q_values` (a flat per-direction E[Q] buffer produced by `compute_expected_q`). It does NOT currently see C51 atom probabilities or IQN quantiles directly. Phase 2 needs to thread three additional buffers through:

- C51 atom probabilities buffer `c51_probs` — exists on GPU; produced by network forward; not currently passed to action_select
- C51 atom values buffer `atom_values` — exists on GPU as a constant per-epoch buffer
- IQN quantile values buffer `iqn_quantiles` — exists on GPU; produced by IQN forward path

Plus `n_atoms` (variable per training config; currently 51 atoms per project memory) and `N_IQN_QUANTILES` (constant; currently 5 per task #147).

Phase 2 implementation: add these as new kernel parameters in the signature; thread them through from `gpu_dqn_trainer.rs` and `gpu_backtest_evaluator.rs` call sites. No new GPU buffers needed (all already exist on GPU); just new parameter passing.

#### Component 4.3: Conviction stays E[Q]-based

The existing conviction computation (`experience_action_select` after action selection, see `experience_kernels.cu` around line 1091) reads `q_b0[a]` per direction and computes `q_range / max(isv[21], q_range)`. After Phase 2:

- `q_b0[a]` (the per-direction E[Q] used for conviction) MUST continue to read raw E[Q] from C51+IQN, NOT a Thompson sample. This requires computing E[Q] per direction in BOTH train and eval paths (eval uses it for argmax; train uses it for conviction).
- Implementation hint: compute `e_joint[d] = compute_e_c51(d) + compute_e_iqn(d)` once per direction, store in a stack-local array `[b0_size]`, reuse for both eval-mode argmax AND conviction. No redundant computation.

Even though Thompson selects the action at training, the policy's CONFIDENCE about the choice is still E[Q]-based — Thompson samples are stochastic and would jitter the Kelly cap warmup floor.

#### Component 4.4: Phase 2 unit tests (all GPU-direct against the production kernel)

- **2.A — End-to-end Thompson + argmax modes (GPU)**: launch the modified `experience_action_select` kernel with synthetic C51+IQN distributions for `eval_mode=false` (training) and `eval_mode=true` (eval). Assert: training mode produces stochastic picks matching distribution shapes (Long+Short ≥ 30% over many seeds when distribution favours them); eval mode produces deterministic argmax picks (same direction across all seeds for given inputs).
- **2.B — Phase 2 kernel matches Phase 0 standalone wrapper (GPU)**: run the Phase 2 production kernel and the Phase 0 `launch_thompson_direction` wrapper with identical inputs and seeded RNG. Confirm: outputs are bit-identical. This catches drift between standalone test wrapper and integrated production kernel.
- **2.C — Magnitude/order/urgency unaffected (GPU)**: launch the kernel with eval_mode=true and given inputs. Confirm: mag/ord/urg picks are bit-identical to a baseline run BEFORE Phase 2 (use a stored snapshot of pre-Phase-2 kernel output for the same inputs).
- **2.D — Real-batch e2e on converged checkpoint (GPU)**: load the converged checkpoint from Phase 0.F; run a full action-select pass over a representative val batch. Training mode: `active_frac` (Long+Short combined) > 40% over the batch. Eval mode: reproduces previously-observed argmax picks (picks the same direction as `launch_argmax_eq` on the same checkpoint).

#### Component 4.5: Documentation + audit

Add to `docs/dqn-wire-up-audit.md` Distributional Q-head Aggregation Contract table:

| Q-head | Produces E[Q]? | Sampleable? | Wired to training Thompson? | Wired to eval argmax? |
|---|---|---|---|---|
| C51 atom kernel | ✅ | ✅ via inverse-CDF | ✅ direction only (v1) | ✅ direction only (v1) |
| IQN quantile kernel | ✅ | ✅ via uniform-τ interpolation | ✅ direction only (v1) | ✅ direction only (v1) |
| Ensemble Q-head (existing — currently used only for `ens_agree` metric) | needs verification | ✅ pick member uniformly | v2.1 enhancement | v2.1 enhancement |
| Future Q-head additions | required | required | required | required |

**Note**: TFT (task #148) is a Variable Selection Network for trunk feature processing — **not a Q-head**. It does not produce per-direction Q values directly and does not need to participate in this contract. The contract applies specifically to modules that produce per-(state, action) Q estimates.

Pre-commit hook (existing Invariant 7) extended to require: any new Q-head MUST appear in this table with all four columns marked.

Pearl entry to memory: `pearl_thompson_for_distributional_action_selection.md`.

**Exit gate**: All Phase 0 + Phase 1 + Phase 2 tests pass; L3 smoke verification primary gates pass.

**Time budget**: 2-3 days.

### Phase 3 — Verification + Direction-Branch Dead-Code Cleanup

**Deliverables**:
- L4 multi-fold deploy (1 seed × 6 folds × 30 epoch) — long enough to see edge discovery via Thompson exploration
- L5 full validation matrix per `docs/superpowers/plans/2026-04-24-dqn-v2-plan-5-validation.md` Task 5: Tier 1/2/3 PASS
- **Direction-branch dead-code cleanup** (NOT band-aid removal — these are real fixes whose direction-branch code paths simply become unreachable after Phase 2):
  - **Kelly cap warm-branch fix (`0c9d1ee39`)**: KEEP. Post-decision physics layer; Thompson-independent. Real fix for a real bug; would break Long/Short trades again if reverted.
  - **Train Return display + Sharpe annualization fixes (non-tau parts of `7a3d88646`)**: KEEP. Display/metric layer; Thompson-independent. Real fixes for real bugs.
  - **Direction-branch Boltzmann tau-floor (tau-floor part of `7a3d88646`)**: code path becomes structurally unreachable for direction once Thompson replaces direction Boltzmann. The tau-floor remains active for magnitude/order/urgency branches (which keep Boltzmann). Cleanup pass: delete direction-only tau-floor code; keep mag/ord/urg paths intact. Not a band-aid removal — dead-code elimination per `feedback_no_legacy_aliases.md` and `feedback_no_partial_refactor.md`.
  - **Adaptive eps_dir floor (`d54b49efc`)**: same — gates the random-sample path of direction-branch eps-greedy. Direction branch no longer uses eps-greedy under Thompson; this code becomes structurally unreachable for direction. Cleanup pass code-level detail: the existing block at `experience_kernels.cu` lines 814-865 has `eps_dir = fmaxf(eps_dir, EPS_FLOOR)` followed by an adaptive boost from ISV[71/72]. Remove `eps_dir` from both the static `EPS_FLOOR` clamp and the adaptive boost block; the corresponding eps_mag/eps_ord/eps_urg lines stay. The variable `eps_dir` should also be removed from the kernel signature if no other branch uses it (verify by grep).
- nsys regression check (Phase H continues independently; Thompson overhead measured)
- Final architecture doc updates

**Why "cleanup" not "removal"**: every band-aid fix shipped during the val-Flat-collapse investigation addressed a real bug at its layer and should be preserved. Phase 2 replaces the direction-branch action-selection mechanism wholesale (eps-greedy + Boltzmann → Thompson). The eps_dir adaptive floor and the direction Boltzmann tau-floor are both inside the direction-branch action-selection path that Thompson replaces. Their direction-only code becomes dead — not because they were unnecessary, but because the larger surrounding mechanism changed shape. Dead code that runs but does nothing is a maintenance hazard (same anti-pattern as backward-compatibility shims for removed features). Phase 3 cleans it up.

**Time budget**: per Plan 5 existing budget + 0.5 day for dead-code cleanup.

---

## Verification Protocol (5-Layer Gate)

The single 5-epoch L40S smoke is INSUFFICIENT to verify Thompson — long-term edge discovery needs more epochs. Two empirical layers (L3 short, L4 long) replace what was previously a single layer.

| Layer | Scope | Pass criterion | Runtime |
|---|---|---|---|
| **L1**: GPU-direct unit tests | Phase 0 (5 GPU + 1 CPU) + Phase 1 (6) + Phase 2 (4 GPU) | All asserts pass | minutes (GPU launch overhead) |
| **L2**: GPU integration test | Tests 0.F + 2.B + 2.D against converged checkpoint | kernel output matches structural assertions | seconds |
| **L3**: Short smoke (3-fold × 5-epoch L40S) | Immediate Thompson effect | **training-time `train_active_frac` > 40%** across all epochs (NEW HEALTH_DIAG metric — see Phase 2 deliverable below) | ~12 min |
| **L4**: Long smoke (1-fold × 30-epoch L40S) | Long-term edge discovery | val_sharpe trends positive; val_active_frac increases as edge discovered | ~1 hour |
| **L5**: Full validation matrix (per Plan 5 Task 5 spec) | Tier 1/2/3 exit criteria | per Plan 5 spec | ~6 hours |

**NEW Phase 2 deliverable**: training-time active_frac instrumentation. Existing HEALTH_DIAG reports val_active_frac (post-eval) but NOT train_active_frac. L3 verification depends on training-time metric. Add a `train_active_frac` HEALTH_DIAG line (same calculation as val side: `(action_count[Long] + action_count[Short]) / total_actions` over the training rollout). This is small (~10 lines in metrics.rs) but must be in Phase 2 deliverable list, not Phase 0 or Phase 1.

### Success criteria (the gates that define "fixed")

**Primary (gate A — REQUIRED, immediate)**:
- TRAINING-time action distribution stays diverse (training active_frac > 40% across all epochs) — Thompson exploration working
- val_dir_dist == val_picked_dir_dist (Kelly fix property; already verified)

**Secondary (gate B — REQUIRED for full close-out, long-term)**:
- val_sharpe trend across 30+ epochs is non-negative (model improves with Thompson exploration)
- val_active_frac evolves with training: starts low if no edge yet, increases as edge is discovered
- If val_active_frac stays low after 30 epochs AND val_sharpe stays flat: data has no edge; this is correct model behaviour, NOT a bug to fix

**Important reframing from prior spec**: val_dir_dist may stay biased toward Hold/Flat in the short term and that's CORRECT. Forcing val to be active would be inflating reported edge. The win condition is "training discovers edge if it exists, reports it honestly at eval."

---

## Risk Register

### R1: Thompson sampling computation cost

**Scenario**: Thompson per-direction requires 2 RNG calls + 2 sampling operations + 1 argmax. ~10× cost of Boltzmann.

**Mitigation**: Parallelize across (sample, direction). Total cost still ~50µs per batch — negligible vs forward pass. Profile with nsys after landing.

### R2: Sampling RNG quality

**Scenario**: Philox PRNG correlations across (i, timestep, rng_ctr) might bias sampling. Two consecutive samples from same Philox stream might be near-identical.

**Mitigation**: Phase 0 Test 0.B verifies sample distribution matches input over 100k draws. If PRNG quality issues surface in test, swap to a different stream seed per (i, d) tuple.

### R3: Thompson at training, argmax at eval — wasted exploration on bad actions

**Scenario**: Thompson explores Long/Short during training. If the data has no edge, those experiences are losing trades. Replay buffer fills with bad-Long-Short data. Q-learning correctly learns Q[Long]<Q[Flat]. Eval argmax then picks Flat. Net effect: training time spent gathering experience that confirms what the model already knew.

**Assessment**: This is a BY-DESIGN tradeoff, not a bug. Q-learning learns the Q FUNCTION (not the policy), so the Q values converge to true expected returns regardless of how exploration happens. Argmax of learned Q is the optimal greedy policy regardless of exploration source. The "wasted" exploration is the COST of finding out whether edge exists — equivalent to a/b testing in production: you must run the test to know the result. If the test (with proper Thompson exploration) shows no edge, the negative result is itself valuable information.

**Mitigation if cost is unacceptable**: gate Thompson by Phase 0.E synthetic-edge confirmation that aggregation IS the bottleneck. If synthetic edge is undiscoverable even with Thompson, don't ship — pivot to reward shaping (per stop condition 1).

### R4: Synthetic edge test (0.E) fails

**Scenario**: Even with Thompson and KNOWN edge in the synthetic MDP, model doesn't discover it.

**Implication**: Aggregation isn't the only issue; reward shaping or Q-target propagation is also broken. Stop. Don't ship UCB/Thompson — pivot to reward design first.

### R5: σ collapses to 0 across all directions late in training

**Scenario**: Model trains long enough that ALL distributions tighten to near-deterministic. Thompson samples become deterministic too.

**Assessment**: Correct behaviour — model is confident. Thompson reduces to argmax in the limit. Not a risk; this is the desired exploit phase.

### R6: Real distributions are degenerate — both σ ≈ 0

**Scenario**: Phase 0.F shows σ_C51 ≈ σ_IQN ≈ 0 for ALL directions on a real converged checkpoint.

**Implication**: Distributional learning has collapsed (atoms all on one bin, quantiles all equal). Thompson is inactive. This is a deeper bug — the distributional representation isn't being trained. Stop. Investigate distributional loss kernels.

### R7: Performance regression from removing Boltzmann

**Scenario**: Removing Boltzmann on direction branch breaks something downstream that depends on stochastic picks (e.g., entropy-bonus computation).

**Mitigation**: Audit downstream consumers of direction-branch Boltzmann output. Confirm Thompson picks satisfy the same interface (one direction per sample). Magnitude/order/urgency keep Boltzmann so any "softmax entropy" computation still has 3-of-4 branches.

### R8: Longer verification needed than Plan 5 budget

**Scenario**: L4 (30 epochs) reveals edge discovery is slower than expected; need 60+ epochs to validate.

**Mitigation**: Build L4 budget around 30 epochs (~1 hour); if data is insufficient, escalate to multi-seed × multi-fold per Plan 5. Worst case: pivot to reward-shaping changes (separate task).

### Stop conditions (halt and reassess)

1. **Phase 0 Test 0.E fails** (synthetic edge undiscoverable) — issue isn't aggregation
2. **Phase 0 Test 0.F shows σ ≈ 0 everywhere** — distributional representation broken
3. **Phase 1 reveals reward shaping bug** — fix reward first
4. **L3 smoke shows training-time active_frac < 40%** — Thompson not generating exploration; PRNG or interpolation bug
5. **L4 long smoke shows val_sharpe degrading** — Thompson exploration is hurting convergence

---

## What v1 Deliberately Does NOT Touch

(See Interaction Matrix above for the FULL audit of all 20 existing mechanisms. Below is the short list of v1-untouched items.)

- Loss kernels (C51 KL-loss, IQN Huber loss): gradient flow unchanged
- Reward shaping (B.2 novelty bonus, counterfactual, PopArt, curiosity, etc.): unchanged unless Phase 1 finds a bug
- Magnitude/order/urgency action selection: unchanged (Boltzmann + ε-greedy preserved; tau-floor + EPS_FLOOR preserved for these branches)
- Q-target propagation (Bellman backup with C51 atom projection / IQN quantile regression): unchanged — operates on full distribution per the existing C51 KL-loss and IQN Huber loss kernels. Thompson does NOT modify Q-target computation; it only modifies action SELECTION at the policy level.
- Eval mode for non-direction branches: unchanged
- Kelly cap fix: KEPT (addresses post-decision physics, different layer)
- **All v2 enhancements (triple-source Thompson, persistent Thompson, CVaR-eval, IDS, hierarchical Thompson, curiosity coupling)**: documented in "Outside-the-Box Improvements" section above; deferred to follow-up specs to keep v1 verification clean

**ISV slots**: v1 adds no new ISV slots. v2.2 (persistent Thompson) and v2.3 (CVaR-eval) may add ISV slots in their respective follow-up specs (persistence multiplier, CVaR λ).

---

## Cumulative Effort

| Plan | Description | Time |
|---|---|---|
| Plan A | Phase 0: GPU-direct hypothesis + synthetic edge verification | 2-3 days |
| Plan B | Phase 1: existing-lever audit | 1 day (longer if bug) |
| Plan C | Phase 2: Thompson sampling integration | 2-3 days |
| Plan D | Phase 3: long verification + direction-branch dead-code cleanup | per Plan 5 + 0.5 day |

Each plan gets its own writing-plans cycle.

---

## References

- Predecessor design: `docs/superpowers/specs/2026-04-24-dqn-v2-unified-design.md`
- Plan 5 (validation): `docs/superpowers/plans/2026-04-24-dqn-v2-plan-5-validation.md`
- Pearl precedents:
  - `memory/pearl_blend_formulas_must_have_permanent_floor.md` (Kelly bootstrap deadlock)
  - `memory/pearl_one_unbounded_signal_per_reward.md` (reward-shaping discipline)
  - `memory/pearl_cold_path_no_exception_to_gpu_drives.md` (compute placement)
- Wire-up audit: `docs/dqn-wire-up-audit.md`
- Empirical evidence runs: `train-dhmp6` (pre-Kelly-fix), `train-pp9d8`, `train-4r6p8` (post-Kelly-fix), `train-bscl2` (post-eps-floor)
- Thompson sampling theory:
  - Thompson, W.R. (1933) "On the likelihood that one unknown probability exceeds another in view of the evidence of two samples" — the original Thompson sampling paper
  - Osband, Russo, Van Roy (2013) "(More) Efficient Reinforcement Learning via Posterior Sampling" — modern RL formulation
  - Russo & Van Roy (2014) "Learning to Optimize via Information-Directed Sampling" — IDS extension (cited for v2.4)
  - Bellemare et al. (2017) "A Distributional Perspective on Reinforcement Learning" — C51 foundation
  - Dabney et al. (2018) "Implicit Quantile Networks for Distributional Reinforcement Learning" — IQN foundation

---

## Aggregation Contract (project-wide pearl, enforced)

For any future distributional or ensemble-based value head added to the system:

1. The kernel/module MUST expose a SAMPLEABLE interface (inverse-CDF for atoms, quantile interpolation for IQN, member-pick for ensembles)
2. Training-time action selection MUST use Thompson sampling (or another sampling-based selector that doesn't reduce to scalar)
3. Eval-time action selection MUST use raw `E[Q]` (or another non-stochastic aggregator) — never Thompson; never UCB; never any exploration overlay
4. The `dqn-wire-up-audit.md` Distributional Q-head Aggregation Contract table MUST be updated to list the new head with all four columns marked
5. The pre-commit hook will refuse merges of distributional or ensemble heads missing the sampleable interface

This contract turns the principle into an engineering invariant. New methods can join the system without re-discovering the bias class.

---

## Status: PAUSED — urgency framing retracted; rollout still planned, evidence-driven sequencing

**Date:** 2026-04-27. **Commits clarifying the picture:** `a86fba2b1` (val_dir_dist + active_frac measurement fix) and `b8788511c` (mag_stats wr_h/wr_f attribution fix).

**The technical proposal stands. Thompson sampling on C51+IQN distributions is the canonical action selector for distributional RL** (Bellemare 2017, Dabney 2018). The math is sound:

- Argmax of `E[Q]` structurally underweights high-σ actions when E[Q]s are close; this is a property of any distributional Q-head, not specific to our model.
- Thompson restores the uncertainty signal at the action-selection layer.
- Phase 0 Tests 0.A-0.E (commits `2186f96e8` → `82d39a895`) empirically validate the math on real GPU kernels.
- The Aggregation Contract above is a sound engineering invariant for any future distributional or ensemble Q-head.

**What changed is the URGENCY framing, not the technical validity.** The val-Flat-collapse / Short-collapse / "65% → 28% active across one epoch" observations cited as motivating evidence were partly distorted by three measurement bugs in the diagnostic infrastructure:

1. `actions_history_buf` was zero-initialised; after `done_flags[w]=1` (capital floor breach), `backtest_env_step` early-returned without writing the buffer for `[done_step, max_len)`. Unwritten zeros decoded as `dir = 0/27 = 0 = Short Quarter Market Normal` → val_dir_dist read 83% Short and active_frac 87-91% as artefacts.
2. The `backtest_metrics_kernel` reader had no `< 0` guard, so unwritten -1 slots after the init fix still decoded as Short via C integer division (`-1 / 9 = 0`).
3. `mag_stats` was binning trade-close events by `actual_mag_core`, which is forced to 0 (Quarter) when `actual_dir` is Hold/Flat — every Half/Full close attributed to Quarter, structurally pinning `wr_h = wr_f = 0`.

**Post-fix smoke data** (3-fold × 5-epoch local, RTX 3050 Ti) shows the diagnostic infrastructure is now telling the truth: val_dir_dist matches val_picked_dir_dist within ~5pp (no severe collapse — diverse picks); active_frac ≈ 31-50%; dir_entropy ≈ 0.83-1.02; Hold+Flat ≈ 50% from real conservative early-training behaviour; eval Quarter dominance explained by Hold/Flat→Q accounting + var_scale shrinkage.

**This means the URGENT framing is gone (the diagnostic numbers reported were inflated artefacts), but Thompson remains the principled improvement.**

**Sequencing plan (paused, evidence-driven)**:

1. **Bug-hunt first.** A fresh L40S 30-epoch baseline run (`train-f8h6q`, 2026-04-27 12:20) on post-fix code is the truth-telling pass. If it surfaces NEW anomalies (NaN, regression-band breaches, residual measurement artefacts in other metrics, gradient pathologies, etc.), fix those before adding new mechanisms. `feedback_stop_on_anomaly.md` applies — kill the run on anomaly, diagnose, fix, re-run.

2. **Then ship Thompson Phase 2 as principled improvement.** Plans B / C / D are not cancelled — they are PAUSED pending the bug-hunt's exit. Once the model is shown to be healthy and stable on the post-fix baseline, Phase 2 lands as a measured improvement (with the Phase 0 tests as the gating harness). Empirical motivation comes from "Thompson preserves second moment, model has measurable σ asymmetry across directions even in healthy state" rather than "Hold/Flat collapse is destroying eval".

3. **A/B against the post-fix baseline.** With both fixes landed, the baseline metrics are now trustworthy. Thompson Phase 2 ships with a clean A/B comparison against `train-f8h6q`'s metrics, not against pre-fix artefact-distorted metrics.

**Phase 0 unit tests + Aggregation Contract: KEEP as latent infrastructure.** They were correct math validation; they neither motivated the rollout based on bad data nor depend on the val-Flat-collapse hypothesis being severe.

**Plans B / C / D: PAUSED.** Files remain in `docs/superpowers/plans/` for resumption after the bug-hunt cycle on `train-f8h6q`. Resumption gate: post-fix baseline run is bug-free or all surfaced bugs are fixed.

---

## Status update: ACTIVE — Plan C Phase 2 resumed 2026-04-29

**Triggering pathology (different from original Hold/Flat-collapse motivation):**

`ff00af68a` (`refactor(dqn): delete vestigial RegimeConditionalDQN — MoE replaces it`) introduced a `multi_fold_convergence` regression: F0 Q-drift kill at epoch 2 (q_mean -0.077 → 0.793, ratio 10.28×), cascading into F1+F2 grad-collapse early-stop. The commit's `crates/ml/src/trainers/dqn/trainer/action.rs:107-118` change removed an `Option::None` ghost gate, activating UCB count bonuses at the GPU action selector for the first time. Verified by paired DIAG smokes (`smoke-test-qlz7t` on ff00af68a + diag fails F0; `smoke-test-wmsht` on a52d99613 + diag passes all 3 folds).

The mechanism: UCB bonus is added to direction-branch Q-values inside `experience_action_select` before Boltzmann softmax, but the Bellman target Q computation uses raw argmax (no bonus). This selector/target asymmetry lets q_mean drift up unchecked → Q-drift kill triggers.

**Why Thompson supersedes the bug fix:**

UCB-as-additive-on-Q is the broken pattern. Three local fixes were considered:
- (A) Adaptive UCB coefficient via ISV decay — preserves selector/target asymmetry (just attenuates)
- (B) UCB-as-intrinsic-reward via Bellman — fixes asymmetry but adds reward-stream tuning surface
- (C) Replace eps-greedy + Boltzmann + UCB-bonus with Thompson — eliminates asymmetry by construction (Thompson samples from the learned distribution, no augment-then-argmax step)

Option C is now Plan C Phase 2. The C51+IQN substrate already produces the distributions Thompson needs; Phase 0 tests already validate the math; eliminating the additive-on-Q exploration mechanism is a net code-surface decrease, not an addition.

**Resumption gate satisfied:** all surfaced bugs in the original measurement-artefact hunt are fixed (commits `a86fba2b1`, `b8788511c`). The new regression introduced by `ff00af68a` is precisely the "Thompson preserves second moment, model has measurable σ asymmetry" empirical motivation referenced in §2 of the original sequencing plan — the σ asymmetry is now causing measurable training-time Q-drift, not just suboptimal selection.

**Phase 2 execution:** plan at `docs/superpowers/plans/2026-04-27-distributional-rl-thompson-plan-C-phase-2.md`. Plans B / D remain on hold pending Phase 2 outcome.
