# C51 Atom Resolution — Design Alternatives (post Fix F failure)

**Status**: Investigation + design-alternatives document. **No code change recommended at this time** — see Section 5.

**Why this exists**: cluster `alpha-rl-qrgjr` (v7, SHA `0fe825a8c`) tested Fix F (atom span decoupled from clamp ceiling via `mean_abs_pnl_ema × atom_margin = 10`). Local b=128 smoke looked great (qpa=+0.958, Δz=8.7 vs prior 184), but at cluster b=1024 **qpa crashed from +0.898 at step 371 to −0.976 at step 500** and stayed stuck through step 1259. Fix F was reverted in `6d4a962e5`.

This document captures the diagnosis and the architectural alternatives we considered, so a future iteration doesn't repeat the same mistake.

---

## 1. Why Fix F failed — the structural constraint

C51 distributional Q-learning has a self-consistency requirement between **atom span** and **Bellman target range**:

```
V_target(s, a) = scaled_reward + γ · V(s')
V_max         = clamp_max + γ · V_max     (fixed-point)
              = clamp_max / (1 − γ)
```

For typical training conditions:
- `clamp_max` (WIN) is the post-scale, post-clamp positive reward ceiling — adaptive via reward_clamp_controller (Fix C). Hovers at MARGIN × pos_max_ema.
- `γ` is the discount factor, controlled by `rl_gamma_controller` toward `1 − 1/(2d)` where `d` is mean trade duration. Cluster values: 0.95–0.99.

At γ=0.98 and WIN=44 (v7 step 500):
- Structural minimum `atom_max ≥ 44 / (1 − 0.98) = 2200`
- Fix F forced `atom_max` to `atom_margin × mean_abs_pnl ≈ 10 × 6 = 60`
- **V_target distribution: [-clamp_loss, +clamp_max + γ·V_max] = [-200, +260]** — sometimes wider as Q's V_dq head outputs scale up
- Atom span [-63, +63] truncated V_target → Bellman projection saturated at top/bottom atoms

**What that did to learning:**
- `Σ atom × prob` for Q stayed bounded by atom_max=63, but V regression (separate scalar head, popart-normalized) could predict higher
- PPO advantage `A = Q(s,a) − V(s)` became inconsistent: Q clamped, V not
- Q-π distillation `softmax(E_Q/τ)` became degenerate when E_Q is uniform-at-ceiling across actions
- π received conflicting gradient signals from PPO and from distillation → qpa decorrelated → −0.97 stuck

**Key insight**: the atom span isn't *free* to size for "typical reward magnitude" — it's *constrained* by Bellman self-consistency at the chosen γ. The pre-Fix-F design (atom span EWMA-tracking WIN) was implementing `atom_span ≈ WIN/(1−γ)` via dynamics rather than explicit formula. That's mathematically correct.

---

## 2. Why local smoke missed this

v7 local smoke (b=128, 1000 steps) showed qpa=+0.958 with Δz=8.7. Why didn't it crash?

- **Smaller batch → tighter V_target distribution**: at b=128 the per-step diversity of V(s') is reduced. The empirical V_target range stays within the (too-tight) atom span by chance.
- **Shorter horizon**: 1000 steps doesn't let V regression saturate its capacity. Q-V divergence builds over many backups, so the failure is downstream.
- **γ slower to grow**: at smoke scale γ stayed at 0.98 instead of climbing toward 0.99+, so the structural minimum atom_max was smaller.

**Lesson for future protocol**: local b=128 smoke validates *kernel correctness* and *immediate dynamics*, but not *Q-V self-consistency at scale*. Either run a smaller cluster b=1024 5k-step smoke before committing structural changes, or add an explicit invariant test that probes `atom_span × (1−γ) ≥ WIN`.

---

## 3. The Pareto frontier — there is no free lunch

The atom-resolution problem is bounded by a fundamental trade-off:

| Design | Δz (resolution) | Q-V consistency | Notes |
|---|---|---|---|
| **Wide atom span = WIN/(1−γ)** (current, v5/v8) | Coarse (~100-200) | ✅ Self-consistent | Pre-Fix-F default. Q can represent full V_target range. |
| **Tight atom span = k × typical reward** (Fix F) | Fine (~5-10) | ❌ V_target saturates | What we tried. qpa crashes when V_dq grows beyond atom span. |
| **Non-uniform atoms** (concentrated near 0, spread for tails) | Fine where it matters | ✅ With proper projection | Major architectural rework. Touches bellman_target_projection.cu + dueling_q_head.cu + atom_supports.cu + offline training. |
| **Lower γ** (e.g. 0.9 instead of 0.99) | Tighter (γ ↓ → atom span ↓) | ✅ | Changes policy time horizon — myopic agent. May lose surfer-style trades. |
| **Two-headed Q** (categorical for direction + regression for magnitude) | Hybrid | ✅ With careful loss | Major architectural rework. Q head doubles in size. |

**The lottery-ticket strategy the agent has discovered has R-multiple ~14-43.** With γ near 1, the structural minimum atom span IS coarse. No tuning of `atom_margin` alone resolves this.

---

## 4. Alternatives evaluated

### 4.A — Increase `atom_margin` (Fix F-v2)

Set `atom_margin` from 10 → 100. Atom span = 100 × mean_abs_pnl ≈ 600. Closer to v5's 2000 but tighter.

**Risk**: at γ=0.99, structural minimum is `WIN × 100 ≈ 4400`. Still under-sized. Possibly buys headroom but doesn't resolve the structural constraint.

**Verdict**: not a clean fix. Defer.

### 4.B — Anchor on `WIN / (1 − γ_max)` explicitly

Make the structural formula explicit:

```c
const float gamma_max = isv[RL_GAMMA_MAX_INDEX];          // 0.99 typical
const float headroom  = 1.0f / fmaxf(0.01f, 1.0f - gamma_max);  // = 100 at γ=0.99
const float v_max_target = fmaxf(v_bound_floor, headroom * win_bound);
```

Equivalent to v5/v8's behavior at steady-state, but immediate response instead of EWMA-lagged. Could improve transient handling.

**Verdict**: marginal improvement, low risk. Defer to a future cleanup pass.

### 4.C — Non-uniform atoms

Atom positions concentrated near zero (e.g., quantiles of a typical-magnitude distribution) and exponentially spread for tails.

Requires:
- New atom-support kernel emitting non-uniform `atom_supports_d[Q_N_ATOMS]`
- New `bellman_target_projection` math that handles non-uniform projection (cross-quantile interpolation)
- Coordination with `dueling_q_head` decomposition (V = mean of atoms × supports)

**Estimated cost**: 1-2 weeks. Touches every file that reads atom_supports_d. Risk: hot-path performance impact on per-step projection.

**Verdict**: real improvement. Defer to a major refactor cycle.

### 4.D — Adaptive γ tied to fold/training-phase

Start γ low (0.9 — narrow horizon) and grow γ over training as the agent stabilizes. Atom span follows γ via the existing controller. Early-training has fine resolution; late-training has coarse-but-correct atom span.

**Risk**: changes the policy's effective time horizon during training — could mask issues that only appear at high γ.

**Verdict**: interesting research direction. Not an immediate fix.

### 4.E — Two-headed Q (categorical direction + scalar magnitude)

Standard architectural pattern in RL — Q decomposed as `Q(s,a) = direction_class(s,a) × magnitude(s,a)`. Categorical loss on direction (signed atom), regression on magnitude.

**Estimated cost**: 1-2 weeks. Full retraining required. Risk: distillation logic needs rewriting.

**Verdict**: real improvement but expensive. Defer.

---

## 5. Recommendation

**Ship A+B+C+D+E only (v8 SHA `6d4a962e5`) and observe full-run results.**

The pre-Fix-F design implements the structurally-correct atom span via WIN-tracking EWMA. The atom resolution it produces (Δz ~100-200) is the *minimum coarseness* compatible with the agent's R-multiple-43 strategy at γ=0.99. This is not a bug — it's a property of the chosen reward distribution × discount factor combination.

If v8 trains cleanly through 20k + 5k eval with positive eval PnL, the resolution coarseness is acceptable and **no further atom-span change is needed**. The agent is finding alpha within the constraints.

If v8 trains but eval is mediocre, the next investigation is whether Q's coarse mid-range resolution is the bottleneck — that would justify the cost of Option 4.C (non-uniform atoms) or 4.E (two-headed Q).

If v8 fails (qpa crash, NaN, unbounded loss), then the issue is NOT atom resolution and we look elsewhere.

---

## 6. Memory pearl to save (after v8 result)

Pending — capture either:

> **pearl_atom_span_must_match_bellman_horizon**: atom span has a structural minimum of `WIN/(1−γ)` for Q-V Bellman self-consistency. Sizing atoms below this saturates V_target projection → Q clamps at top atom → distillation degenerates → qpa crashes. Local b=128 smoke under-exercises the failure mode (smaller V_target distribution range). Tested empirically in `alpha-rl-qrgjr` v7 (2026-05-30, Fix F atom_margin=10).

OR (depending on v8 outcome):

> **pearl_lottery_ticket_strategy_forces_coarse_atoms**: when the agent discovers a high-R-multiple lottery-ticket strategy, atom span must scale with `WIN/(1−γ)` which becomes coarse at γ=0.99. Resolution loss is structural, not a bug. Q learns direction correctly even at Δz=200; PPO advantage via V regression preserves magnitude. Confirmed in `alpha-rl-vxbpq` v8 (2026-05-31).
