From d57bee05423e4e302f5a4b7d5f4cfa8bd2063f15 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 31 May 2026 00:10:05 +0200 Subject: [PATCH] docs(spec): atom resolution design alternatives (post Fix F failure) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents why Fix F crashed at cluster scale (qpa: +0.898 → -0.976 between step 371 and step 500 in alpha-rl-qrgjr v7), the structural Bellman self-consistency constraint atom_max >= WIN/(1-γ) that makes the resolution trade-off fundamental, and the design alternatives considered (non-uniform atoms, adaptive γ, two-headed Q). Recommends shipping A+B+C+D+E only — the pre-Fix-F design implements the structurally-correct atom span via WIN-tracking EWMA. No code change in this commit — design document only. Empirical result from v8 (alpha-rl-vxbpq, SHA 6d4a962e5) will determine whether deeper atom-resolution work (Options 4.C or 4.E) is justified. --- ...c51-atom-resolution-design-alternatives.md | 146 ++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-31-c51-atom-resolution-design-alternatives.md diff --git a/docs/superpowers/specs/2026-05-31-c51-atom-resolution-design-alternatives.md b/docs/superpowers/specs/2026-05-31-c51-atom-resolution-design-alternatives.md new file mode 100644 index 000000000..d3e7cfc0e --- /dev/null +++ b/docs/superpowers/specs/2026-05-31-c51-atom-resolution-design-alternatives.md @@ -0,0 +1,146 @@ +# C51 Atom Resolution — Design Alternatives (post Fix F failure) + +**Status**: Investigation + design-alternatives document. **No code change recommended at this time** — see Section 5. + +**Why this exists**: cluster `alpha-rl-qrgjr` (v7, SHA `0fe825a8c`) tested Fix F (atom span decoupled from clamp ceiling via `mean_abs_pnl_ema × atom_margin = 10`). Local b=128 smoke looked great (qpa=+0.958, Δz=8.7 vs prior 184), but at cluster b=1024 **qpa crashed from +0.898 at step 371 to −0.976 at step 500** and stayed stuck through step 1259. Fix F was reverted in `6d4a962e5`. + +This document captures the diagnosis and the architectural alternatives we considered, so a future iteration doesn't repeat the same mistake. + +--- + +## 1. Why Fix F failed — the structural constraint + +C51 distributional Q-learning has a self-consistency requirement between **atom span** and **Bellman target range**: + +``` +V_target(s, a) = scaled_reward + γ · V(s') +V_max = clamp_max + γ · V_max (fixed-point) + = clamp_max / (1 − γ) +``` + +For typical training conditions: +- `clamp_max` (WIN) is the post-scale, post-clamp positive reward ceiling — adaptive via reward_clamp_controller (Fix C). Hovers at MARGIN × pos_max_ema. +- `γ` is the discount factor, controlled by `rl_gamma_controller` toward `1 − 1/(2d)` where `d` is mean trade duration. Cluster values: 0.95–0.99. + +At γ=0.98 and WIN=44 (v7 step 500): +- Structural minimum `atom_max ≥ 44 / (1 − 0.98) = 2200` +- Fix F forced `atom_max` to `atom_margin × mean_abs_pnl ≈ 10 × 6 = 60` +- **V_target distribution: [-clamp_loss, +clamp_max + γ·V_max] = [-200, +260]** — sometimes wider as Q's V_dq head outputs scale up +- Atom span [-63, +63] truncated V_target → Bellman projection saturated at top/bottom atoms + +**What that did to learning:** +- `Σ atom × prob` for Q stayed bounded by atom_max=63, but V regression (separate scalar head, popart-normalized) could predict higher +- PPO advantage `A = Q(s,a) − V(s)` became inconsistent: Q clamped, V not +- Q-π distillation `softmax(E_Q/τ)` became degenerate when E_Q is uniform-at-ceiling across actions +- π received conflicting gradient signals from PPO and from distillation → qpa decorrelated → −0.97 stuck + +**Key insight**: the atom span isn't *free* to size for "typical reward magnitude" — it's *constrained* by Bellman self-consistency at the chosen γ. The pre-Fix-F design (atom span EWMA-tracking WIN) was implementing `atom_span ≈ WIN/(1−γ)` via dynamics rather than explicit formula. That's mathematically correct. + +--- + +## 2. Why local smoke missed this + +v7 local smoke (b=128, 1000 steps) showed qpa=+0.958 with Δz=8.7. Why didn't it crash? + +- **Smaller batch → tighter V_target distribution**: at b=128 the per-step diversity of V(s') is reduced. The empirical V_target range stays within the (too-tight) atom span by chance. +- **Shorter horizon**: 1000 steps doesn't let V regression saturate its capacity. Q-V divergence builds over many backups, so the failure is downstream. +- **γ slower to grow**: at smoke scale γ stayed at 0.98 instead of climbing toward 0.99+, so the structural minimum atom_max was smaller. + +**Lesson for future protocol**: local b=128 smoke validates *kernel correctness* and *immediate dynamics*, but not *Q-V self-consistency at scale*. Either run a smaller cluster b=1024 5k-step smoke before committing structural changes, or add an explicit invariant test that probes `atom_span × (1−γ) ≥ WIN`. + +--- + +## 3. The Pareto frontier — there is no free lunch + +The atom-resolution problem is bounded by a fundamental trade-off: + +| Design | Δz (resolution) | Q-V consistency | Notes | +|---|---|---|---| +| **Wide atom span = WIN/(1−γ)** (current, v5/v8) | Coarse (~100-200) | ✅ Self-consistent | Pre-Fix-F default. Q can represent full V_target range. | +| **Tight atom span = k × typical reward** (Fix F) | Fine (~5-10) | ❌ V_target saturates | What we tried. qpa crashes when V_dq grows beyond atom span. | +| **Non-uniform atoms** (concentrated near 0, spread for tails) | Fine where it matters | ✅ With proper projection | Major architectural rework. Touches bellman_target_projection.cu + dueling_q_head.cu + atom_supports.cu + offline training. | +| **Lower γ** (e.g. 0.9 instead of 0.99) | Tighter (γ ↓ → atom span ↓) | ✅ | Changes policy time horizon — myopic agent. May lose surfer-style trades. | +| **Two-headed Q** (categorical for direction + regression for magnitude) | Hybrid | ✅ With careful loss | Major architectural rework. Q head doubles in size. | + +**The lottery-ticket strategy the agent has discovered has R-multiple ~14-43.** With γ near 1, the structural minimum atom span IS coarse. No tuning of `atom_margin` alone resolves this. + +--- + +## 4. Alternatives evaluated + +### 4.A — Increase `atom_margin` (Fix F-v2) + +Set `atom_margin` from 10 → 100. Atom span = 100 × mean_abs_pnl ≈ 600. Closer to v5's 2000 but tighter. + +**Risk**: at γ=0.99, structural minimum is `WIN × 100 ≈ 4400`. Still under-sized. Possibly buys headroom but doesn't resolve the structural constraint. + +**Verdict**: not a clean fix. Defer. + +### 4.B — Anchor on `WIN / (1 − γ_max)` explicitly + +Make the structural formula explicit: + +```c +const float gamma_max = isv[RL_GAMMA_MAX_INDEX]; // 0.99 typical +const float headroom = 1.0f / fmaxf(0.01f, 1.0f - gamma_max); // = 100 at γ=0.99 +const float v_max_target = fmaxf(v_bound_floor, headroom * win_bound); +``` + +Equivalent to v5/v8's behavior at steady-state, but immediate response instead of EWMA-lagged. Could improve transient handling. + +**Verdict**: marginal improvement, low risk. Defer to a future cleanup pass. + +### 4.C — Non-uniform atoms + +Atom positions concentrated near zero (e.g., quantiles of a typical-magnitude distribution) and exponentially spread for tails. + +Requires: +- New atom-support kernel emitting non-uniform `atom_supports_d[Q_N_ATOMS]` +- New `bellman_target_projection` math that handles non-uniform projection (cross-quantile interpolation) +- Coordination with `dueling_q_head` decomposition (V = mean of atoms × supports) + +**Estimated cost**: 1-2 weeks. Touches every file that reads atom_supports_d. Risk: hot-path performance impact on per-step projection. + +**Verdict**: real improvement. Defer to a major refactor cycle. + +### 4.D — Adaptive γ tied to fold/training-phase + +Start γ low (0.9 — narrow horizon) and grow γ over training as the agent stabilizes. Atom span follows γ via the existing controller. Early-training has fine resolution; late-training has coarse-but-correct atom span. + +**Risk**: changes the policy's effective time horizon during training — could mask issues that only appear at high γ. + +**Verdict**: interesting research direction. Not an immediate fix. + +### 4.E — Two-headed Q (categorical direction + scalar magnitude) + +Standard architectural pattern in RL — Q decomposed as `Q(s,a) = direction_class(s,a) × magnitude(s,a)`. Categorical loss on direction (signed atom), regression on magnitude. + +**Estimated cost**: 1-2 weeks. Full retraining required. Risk: distillation logic needs rewriting. + +**Verdict**: real improvement but expensive. Defer. + +--- + +## 5. Recommendation + +**Ship A+B+C+D+E only (v8 SHA `6d4a962e5`) and observe full-run results.** + +The pre-Fix-F design implements the structurally-correct atom span via WIN-tracking EWMA. The atom resolution it produces (Δz ~100-200) is the *minimum coarseness* compatible with the agent's R-multiple-43 strategy at γ=0.99. This is not a bug — it's a property of the chosen reward distribution × discount factor combination. + +If v8 trains cleanly through 20k + 5k eval with positive eval PnL, the resolution coarseness is acceptable and **no further atom-span change is needed**. The agent is finding alpha within the constraints. + +If v8 trains but eval is mediocre, the next investigation is whether Q's coarse mid-range resolution is the bottleneck — that would justify the cost of Option 4.C (non-uniform atoms) or 4.E (two-headed Q). + +If v8 fails (qpa crash, NaN, unbounded loss), then the issue is NOT atom resolution and we look elsewhere. + +--- + +## 6. Memory pearl to save (after v8 result) + +Pending — capture either: + +> **pearl_atom_span_must_match_bellman_horizon**: atom span has a structural minimum of `WIN/(1−γ)` for Q-V Bellman self-consistency. Sizing atoms below this saturates V_target projection → Q clamps at top atom → distillation degenerates → qpa crashes. Local b=128 smoke under-exercises the failure mode (smaller V_target distribution range). Tested empirically in `alpha-rl-qrgjr` v7 (2026-05-30, Fix F atom_margin=10). + +OR (depending on v8 outcome): + +> **pearl_lottery_ticket_strategy_forces_coarse_atoms**: when the agent discovers a high-R-multiple lottery-ticket strategy, atom span must scale with `WIN/(1−γ)` which becomes coarse at γ=0.99. Resolution loss is structural, not a bug. Q learns direction correctly even at Δz=200; PPO advantage via V regression preserves magnitude. Confirmed in `alpha-rl-vxbpq` v8 (2026-05-31).