# C51 Atom Span Decoupling from Reward Clamp Ceiling — Design
**Goal**: Decouple the C51 distributional-Q atom span from the reward-clamp ceiling so the atom grid stays sized for *typical* trade magnitudes while the clamp continues to admit *fat-tail* wins/losses without truncation. This preserves Q's distributional resolution at the magnitudes that matter for learning, restoring fine-grained value differentiation between small wins/losses while keeping the wide-clamp benefit shipped in `7064c9269` (Fix C).
**Why now**: cluster `alpha-rl-rjsjq` (v5, SHA `7064c9269` = Fix A+B+C+D) at step 371 showed `c51_v_max` ratcheting up monotonically — 1 → 24 → 53 → 859 → 990 → 1544 → **1938** — while the `WIN` clamp swung wildly (4910 → 568 → 7540 → 44). The Δz atom resolution settled at ~184 per atom by step 371, ~600 by long-running steady state. Q can no longer discriminate $500-magnitude actions from $5000-magnitude actions within the same atom.
**Out of scope**: changes to V regression (uses popart, already handles wide range), changes to the reward-scale controller, changes to PPO clip / IQN τ / Kelly. Fix E (force-close on DD trip) is shipped separately. The fat-tail single-step DD-trigger lag (positions losing $11k+ in one tick) needs an intra-step circuit breaker — that's a separate spec.
---
## Diagnosis — what the JSONL says
From v5 cluster `/feature-cache/alpha-rl-runs/7064c9269/fold1/diag.jsonl`:
1. WIN clamp width = MARGIN × pos_max_ema, with MARGIN saturated at MAX=5.0 once `clip_rate > 0.05`. Updates fast (α=0.4 = Wiener floor for pos_max_ema).
2. C51 atom v_max follows WIN via slow EWMA (`v_bound_ewma_alpha = 0.001`, half-life ~700 steps). Asymmetric in practice: WIN can drop fast (from 7540 to 44 over ~300 steps), but v_max only crawls down 0.001× per step.
3. v_max **time-averages** WIN. Because pos_max_ema is the MAX of `b_size = 1024` samples (extreme value statistic), the EMA always sits far above the median or mean reward magnitude. WIN's lower bound is roughly `MAX_MARGIN × pos_max_p99`, not `MAX_MARGIN × pos_max_typical`.
**Net effect**: atom span sized for the rare $42k fat-tail wins, not the $7k typical wins. Δz at v_max=1938 is `2×1938/21 ≈ 184 per atom`. A +$500 reward and a +$5000 reward both map to within 1 atom of zero post-scale.
**Why this matters for learning**:
- Bellman target projection: V_target lands at coarse atom — loses information about exact magnitude.
- Q-π distillation: `softmax(E_Q/τ)` — when all small-magnitude actions land in the same atom, distillation collapses to near-uniform.
- PPO advantage: `A = Q(s,a) - V(s)` — wide-atom Q is noisy on typical-magnitude actions, so PPO update is noisy.
The agent still trains (qpa=+0.87 at step 371) but is *leaving learning signal on the table*.
---
## Proposed design — anchor atom span on the typical-magnitude EMA
Replace the v_max EWMA target from `win_bound` (= MARGIN × pos_max_ema) to a slower, more central magnitude estimate. Two candidates:
`RL_MEAN_ABS_PNL_EMA_INDEX` (slot 477) tracks the EMA of `|reward|` across recent batches. It's a true central-tendency signal — not the extreme-value MAX. Already populated by the reward pipeline.
With `atom_margin = 10` and `mean_abs_pnl_ema ≈ 1.5` (post-scale typical), `v_max_target ≈ 15`. Δz = 30/21 ≈ 1.4 per atom — fine resolution for typical trades. Fat-tail wins of 100+ scaled magnitude still get the WIN clamp (244) → projected to top atom; **information about magnitude beyond +15 is lost in Q but preserved in V regression** (V is a scalar head, not categorical).
Use the same pos_max_ema input as WIN, but with `atom_margin = 1.5` instead of MARGIN = 5.0. Net effect: atom span = (1.5/5.0) × WIN = 30% of clamp width.
Simpler — reuses existing EMA, just different multiplier. Atom span scales with same dynamics as clamp but tighter.
### Recommendation: **Option A**
Mean-absolute-PnL is fundamentally a better signal for atom sizing because it represents the magnitude where most rewards land, not the extreme-value tail. Δz adapts to the actual distribution.
**Caveat for Option A**: `mean_abs_pnl_ema` is scaled by `reward_scale`. If reward_scale collapses (as it does — 0.004 at step 371), the EMA shrinks too. That's actually correct: smaller scaled rewards → tighter atom span needed for resolution. The reward_scale controller and atom span thus become coupled in a healthy way (both track the post-scale reward magnitude).
**Atom span LOSS (v_min) follows analogously**: `v_min_target = min(-v_bound_floor, -atom_margin × neg_max_ema)` OR `-RATIO × v_max_target` for symmetric scaling with the loss-aversion ratio.
---
## Math walkthrough — expected behavior at steady state
Take v5 cluster step 371 actual numbers:
-`mean_abs_pnl_ema` ≈ 1.5 (post-scale, estimated — slot would need to be added to diag)
- v_max trails toward 44 at α=0.001 → reaches 44 only after ~5000 steps
- But because WIN was historically much higher (7540 at step 50), the trail has been "remembering" big WIN values → v_max stuck at 1938
**Option A (v_max = 10 × mean_abs_pnl_ema)**:
- v_max_target = 10 × 1.5 = 15
- Δz = 30/21 = 1.43 per atom
- A +$3000 scaled reward (typical big win pre-scale × 0.004 scale) = +12 → atom 19 (mid-upper)
- A +$15k scaled reward = +60 → clamped to WIN=44, projected to atom 20
- A +$30 scaled reward (typical small win) = +0.12 → atom 11 (just above zero) — **finely resolved**
Resolution at typical-magnitude is **125× better** than current.
**Risk**: when pos_max spikes, V_target can land outside atom span and Q learns "this is just top-atom" — loses fat-tail magnitude information. Mitigation: V regression head (not categorical) retains the magnitude. PPO uses V for advantage. Only Q's distributional learning loses fat-tail discrimination, which is acceptable for a lottery-ticket strategy that cares more about "is this action a winner" than "exactly how much does it win".
Mitigation: V regression head (scalar) keeps full magnitude. PPO advantage uses V, so PG signal preserved. Q only loses *categorical* resolution of magnitudes beyond atom span — its argmax/expected ranking remains correct.
**Risk 2: `mean_abs_pnl_ema` could be too small early on, atom span collapses to floor.**
Mitigation: `v_bound_floor = 1.0` prevents collapse below baseline atom span. Atom span only widens beyond floor when EMA stabilizes — slow EWMA `α=0.001` ensures gentle ramp-up.
**Risk 3: Coupling with reward_scale controller.**
Both atom span and reward_scale react to reward magnitudes. Need to verify no feedback loop. Per `[[pearl_no_feedback_loop_atom_span_clamp]]`, the original G.2 freeze was over-conservative; this coupling is *desirable* — both controllers track the same underlying signal (post-scale typical reward magnitude) and stabilize together.
**Alternative considered: Option B (pos_max_ema × smaller margin)**.
Simpler but uses the extreme-value statistic. With b=1024, pos_max is heavily right-skewed, so even `1.5 × pos_max_ema` overshoots typical reward magnitude. Mean-abs-PnL is the right statistic.
**Alternative considered: Non-uniform atoms (concentrated near zero, spread for tails)**.
Architectural change to C51's atom grid. Major work — would touch `bellman_target_projection.cu`, `dueling_q_head.cu`, etc. Deferred. Option A delivers most of the benefit with surgical kernel edit.
Hysteresis to prevent atom span churn. But the underlying issue is the *target* signal (pos_max_ema biases high), not the EWMA mechanics. Fixing the target signal fixes both directions naturally.
6.**If positive eval**, ship as Fix F. If negative, revert and investigate Q gradient explosion separately.
---
## Related issues — future spec candidates
### Fat-tail single-step DD (out of scope here)
Positions can lose more than `dd_limit` in a single futures-market tick before the DD controller can react. Fix E force-closes on the FIRST cooldown step but the *triggering* loss is already realized. Real fix needs an intra-step circuit breaker — e.g., apply a hard stop at `−2 × dd_limit` directly in the reward pipeline before the CMDP check.
Estimated cost: kernel edit to `rl_fused_reward_pipeline.cu`, ~20 LOC. Defer until v6 cluster eval shows whether residual fat-tail loss is materially affecting PnL.
### MARGIN ratchet hysteresis (low priority)
MARGIN slams to MAX=5.0 within ~12 steps from cold start (`adjust_rate^N = MAX/MIN` → N = log(5/1)/log(1.1) ≈ 17). This causes WIN to overshoot before settling. Could add a warmup period during which MARGIN is capped lower.
Estimated cost: 3-line edit. Defer; current behavior absorbs into atom span EWMA and converges by step ~500.
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
jgrusewski