docs(spec): atom resolution design alternatives (post Fix F failure)
Documents why Fix F crashed at cluster scale (qpa: +0.898 → -0.976
between step 371 and step 500 in alpha-rl-qrgjr v7), the structural
Bellman self-consistency constraint atom_max >= WIN/(1-γ) that makes
the resolution trade-off fundamental, and the design alternatives
considered (non-uniform atoms, adaptive γ, two-headed Q). Recommends
shipping A+B+C+D+E only — the pre-Fix-F design implements the
structurally-correct atom span via WIN-tracking EWMA.
No code change in this commit — design document only. Empirical
result from v8 (alpha-rl-vxbpq, SHA 6d4a962e5) will determine whether
deeper atom-resolution work (Options 4.C or 4.E) is justified.
This commit is contained in:
jgrusewski
@@ -0,0 +1,146 @@
|
||||
# C51 Atom Resolution — Design Alternatives (post Fix F failure)
|
||||
|
||||
**Status**: Investigation + design-alternatives document. **No code change recommended at this time** — see Section 5.
|
||||
|
||||
**Why this exists**: cluster `alpha-rl-qrgjr` (v7, SHA `0fe825a8c`) tested Fix F (atom span decoupled from clamp ceiling via `mean_abs_pnl_ema × atom_margin = 10`). Local b=128 smoke looked great (qpa=+0.958, Δz=8.7 vs prior 184), but at cluster b=1024 **qpa crashed from +0.898 at step 371 to −0.976 at step 500** and stayed stuck through step 1259. Fix F was reverted in `6d4a962e5`.
|
||||
|
||||
This document captures the diagnosis and the architectural alternatives we considered, so a future iteration doesn't repeat the same mistake.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why Fix F failed — the structural constraint
|
||||
|
||||
C51 distributional Q-learning has a self-consistency requirement between **atom span** and **Bellman target range**:
|
||||
|
||||
```
|
||||
V_target(s, a) = scaled_reward + γ · V(s')
|
||||
V_max = clamp_max + γ · V_max (fixed-point)
|
||||
= clamp_max / (1 − γ)
|
||||
```
|
||||
|
||||
For typical training conditions:
|
||||
- `clamp_max` (WIN) is the post-scale, post-clamp positive reward ceiling — adaptive via reward_clamp_controller (Fix C). Hovers at MARGIN × pos_max_ema.
|
||||
- `γ` is the discount factor, controlled by `rl_gamma_controller` toward `1 − 1/(2d)` where `d` is mean trade duration. Cluster values: 0.95–0.99.
|
||||
|
||||
At γ=0.98 and WIN=44 (v7 step 500):
|
||||
- Structural minimum `atom_max ≥ 44 / (1 − 0.98) = 2200`
|
||||
- Fix F forced `atom_max` to `atom_margin × mean_abs_pnl ≈ 10 × 6 = 60`
|
||||
- **V_target distribution: [-clamp_loss, +clamp_max + γ·V_max] = [-200, +260]** — sometimes wider as Q's V_dq head outputs scale up
|
||||
- Atom span [-63, +63] truncated V_target → Bellman projection saturated at top/bottom atoms
|
||||
|
||||
**What that did to learning:**
|
||||
- `Σ atom × prob` for Q stayed bounded by atom_max=63, but V regression (separate scalar head, popart-normalized) could predict higher
|
||||
- PPO advantage `A = Q(s,a) − V(s)` became inconsistent: Q clamped, V not
|
||||
- Q-π distillation `softmax(E_Q/τ)` became degenerate when E_Q is uniform-at-ceiling across actions
|
||||
- π received conflicting gradient signals from PPO and from distillation → qpa decorrelated → −0.97 stuck
|
||||
|
||||
**Key insight**: the atom span isn't *free* to size for "typical reward magnitude" — it's *constrained* by Bellman self-consistency at the chosen γ. The pre-Fix-F design (atom span EWMA-tracking WIN) was implementing `atom_span ≈ WIN/(1−γ)` via dynamics rather than explicit formula. That's mathematically correct.
|
||||
|
||||
---
|
||||
|
||||
## 2. Why local smoke missed this
|
||||
|
||||
v7 local smoke (b=128, 1000 steps) showed qpa=+0.958 with Δz=8.7. Why didn't it crash?
|
||||
|
||||
- **Smaller batch → tighter V_target distribution**: at b=128 the per-step diversity of V(s') is reduced. The empirical V_target range stays within the (too-tight) atom span by chance.
|
||||
- **Shorter horizon**: 1000 steps doesn't let V regression saturate its capacity. Q-V divergence builds over many backups, so the failure is downstream.
|
||||
- **γ slower to grow**: at smoke scale γ stayed at 0.98 instead of climbing toward 0.99+, so the structural minimum atom_max was smaller.
|
||||
|
||||
**Lesson for future protocol**: local b=128 smoke validates *kernel correctness* and *immediate dynamics*, but not *Q-V self-consistency at scale*. Either run a smaller cluster b=1024 5k-step smoke before committing structural changes, or add an explicit invariant test that probes `atom_span × (1−γ) ≥ WIN`.
|
||||
|
||||
---
|
||||
|
||||
## 3. The Pareto frontier — there is no free lunch
|
||||
|
||||
The atom-resolution problem is bounded by a fundamental trade-off:
|
||||
|
||||
| Design | Δz (resolution) | Q-V consistency | Notes |
|
||||
|---|---|---|---|
|
||||
| **Wide atom span = WIN/(1−γ)** (current, v5/v8) | Coarse (~100-200) | ✅ Self-consistent | Pre-Fix-F default. Q can represent full V_target range. |
|
||||
| **Tight atom span = k × typical reward** (Fix F) | Fine (~5-10) | ❌ V_target saturates | What we tried. qpa crashes when V_dq grows beyond atom span. |
|
||||
| **Non-uniform atoms** (concentrated near 0, spread for tails) | Fine where it matters | ✅ With proper projection | Major architectural rework. Touches bellman_target_projection.cu + dueling_q_head.cu + atom_supports.cu + offline training. |
|
||||
| **Lower γ** (e.g. 0.9 instead of 0.99) | Tighter (γ ↓ → atom span ↓) | ✅ | Changes policy time horizon — myopic agent. May lose surfer-style trades. |
|
||||
| **Two-headed Q** (categorical for direction + regression for magnitude) | Hybrid | ✅ With careful loss | Major architectural rework. Q head doubles in size. |
|
||||
|
||||
**The lottery-ticket strategy the agent has discovered has R-multiple ~14-43.** With γ near 1, the structural minimum atom span IS coarse. No tuning of `atom_margin` alone resolves this.
|
||||
|
||||
---
|
||||
|
||||
## 4. Alternatives evaluated
|
||||
|
||||
### 4.A — Increase `atom_margin` (Fix F-v2)
|
||||
|
||||
Set `atom_margin` from 10 → 100. Atom span = 100 × mean_abs_pnl ≈ 600. Closer to v5's 2000 but tighter.
|
||||
|
||||
**Risk**: at γ=0.99, structural minimum is `WIN × 100 ≈ 4400`. Still under-sized. Possibly buys headroom but doesn't resolve the structural constraint.
|
||||
|
||||
**Verdict**: not a clean fix. Defer.
|
||||
|
||||
### 4.B — Anchor on `WIN / (1 − γ_max)` explicitly
|
||||
|
||||
Make the structural formula explicit:
|
||||
|
||||
```c
|
||||
const float gamma_max = isv[RL_GAMMA_MAX_INDEX]; // 0.99 typical
|
||||
const float headroom = 1.0f / fmaxf(0.01f, 1.0f - gamma_max); // = 100 at γ=0.99
|
||||
const float v_max_target = fmaxf(v_bound_floor, headroom * win_bound);
|
||||
```
|
||||
|
||||
Equivalent to v5/v8's behavior at steady-state, but immediate response instead of EWMA-lagged. Could improve transient handling.
|
||||
|
||||
**Verdict**: marginal improvement, low risk. Defer to a future cleanup pass.
|
||||
|
||||
### 4.C — Non-uniform atoms
|
||||
|
||||
Atom positions concentrated near zero (e.g., quantiles of a typical-magnitude distribution) and exponentially spread for tails.
|
||||
|
||||
Requires:
|
||||
- New atom-support kernel emitting non-uniform `atom_supports_d[Q_N_ATOMS]`
|
||||
- New `bellman_target_projection` math that handles non-uniform projection (cross-quantile interpolation)
|
||||
- Coordination with `dueling_q_head` decomposition (V = mean of atoms × supports)
|
||||
|
||||
**Estimated cost**: 1-2 weeks. Touches every file that reads atom_supports_d. Risk: hot-path performance impact on per-step projection.
|
||||
|
||||
**Verdict**: real improvement. Defer to a major refactor cycle.
|
||||
|
||||
### 4.D — Adaptive γ tied to fold/training-phase
|
||||
|
||||
Start γ low (0.9 — narrow horizon) and grow γ over training as the agent stabilizes. Atom span follows γ via the existing controller. Early-training has fine resolution; late-training has coarse-but-correct atom span.
|
||||
|
||||
**Risk**: changes the policy's effective time horizon during training — could mask issues that only appear at high γ.
|
||||
|
||||
**Verdict**: interesting research direction. Not an immediate fix.
|
||||
|
||||
### 4.E — Two-headed Q (categorical direction + scalar magnitude)
|
||||
|
||||
Standard architectural pattern in RL — Q decomposed as `Q(s,a) = direction_class(s,a) × magnitude(s,a)`. Categorical loss on direction (signed atom), regression on magnitude.
|
||||
|
||||
**Estimated cost**: 1-2 weeks. Full retraining required. Risk: distillation logic needs rewriting.
|
||||
|
||||
**Verdict**: real improvement but expensive. Defer.
|
||||
|
||||
---
|
||||
|
||||
## 5. Recommendation
|
||||
|
||||
**Ship A+B+C+D+E only (v8 SHA `6d4a962e5`) and observe full-run results.**
|
||||
|
||||
The pre-Fix-F design implements the structurally-correct atom span via WIN-tracking EWMA. The atom resolution it produces (Δz ~100-200) is the *minimum coarseness* compatible with the agent's R-multiple-43 strategy at γ=0.99. This is not a bug — it's a property of the chosen reward distribution × discount factor combination.
|
||||
|
||||
If v8 trains cleanly through 20k + 5k eval with positive eval PnL, the resolution coarseness is acceptable and **no further atom-span change is needed**. The agent is finding alpha within the constraints.
|
||||
|
||||
If v8 trains but eval is mediocre, the next investigation is whether Q's coarse mid-range resolution is the bottleneck — that would justify the cost of Option 4.C (non-uniform atoms) or 4.E (two-headed Q).
|
||||
|
||||
If v8 fails (qpa crash, NaN, unbounded loss), then the issue is NOT atom resolution and we look elsewhere.
|
||||
|
||||
---
|
||||
|
||||
## 6. Memory pearl to save (after v8 result)
|
||||
|
||||
Pending — capture either:
|
||||
|
||||
> **pearl_atom_span_must_match_bellman_horizon**: atom span has a structural minimum of `WIN/(1−γ)` for Q-V Bellman self-consistency. Sizing atoms below this saturates V_target projection → Q clamps at top atom → distillation degenerates → qpa crashes. Local b=128 smoke under-exercises the failure mode (smaller V_target distribution range). Tested empirically in `alpha-rl-qrgjr` v7 (2026-05-30, Fix F atom_margin=10).
|
||||
|
||||
OR (depending on v8 outcome):
|
||||
|
||||
> **pearl_lottery_ticket_strategy_forces_coarse_atoms**: when the agent discovers a high-R-multiple lottery-ticket strategy, atom span must scale with `WIN/(1−γ)` which becomes coarse at γ=0.99. Resolution loss is structural, not a bug. Q learns direction correctly even at Δz=200; PPO advantage via V regression preserves magnitude. Confirmed in `alpha-rl-vxbpq` v8 (2026-05-31).
|
||||
Reference in New Issue
Block a user