feat(rl): adaptive RATIO + EWMA V_MIN/V_MAX + λ_distill bump
wwcsz analysis identified atom-resolution starvation + asymmetric
RATIO mismatch as the empirical ceiling on win rate (38.56% vs
break-even 45.3%). Three coupled fixes shipped in one pass per
"no deferrals":
(a) Adaptive RATIO from observed |loss|/|win| EMAs:
- apply_reward_scale tracks max(-scaled, 0) per step (slot 489)
- rl_reward_clamp_controller maintains neg_max_ema (slot 490,
sparse-aware like pos_max_ema)
- RATIO = clamp(MIN=1.0, neg_ema/pos_ema, MAX=3.0); writes to
slot 481
- Removes built-in 3:1 loss-aversion bias when reality is
symmetric (wwcsz: actual avg|loss|/avg(win) = 0.83). Floor
1.0 prevents inverted asymmetry; ceiling 3.0 preserves
original loss-aversion as the worst case.
(b) C51 V_MAX/V_MIN: ratchet → slow EWMA (α=0.001, half-life ~700
steps):
- Static ratchet wasted atom resolution on rare tails — wwcsz
had V_MIN=-60, V_MAX=20 but realized rewards mostly in [-5, +5]
(Δz=4 vs typical reward magnitude 1-5)
- Slow EWMA lets atom span shrink toward active reward range,
gaining resolution where data lives. Floors at [-1, +1]
preserve original C51 baseline as the worst case.
- Slow α gives Q's atom mapping time to be valid across
encoder/head co-adaptation (vs aggressive EWMA which would
invalidate Q's learned distribution every step)
(c) Q→π distillation λ bumped 0.01 → 0.05:
- wwcsz showed KL dropped 2.10 → 0.30 with λ=0.01 — Q signal
landing but conservatively. Bump tests whether stronger Q
pull translates to better policy → better R/done.
Diag exposes neg_scaled_max + neg_scaled_max_ema so the RATIO
adaptation chain is observable.
apply_reward_scale shared_mem doubled from 2× to 3× block × f32
to fit the three parallel reductions (abs, pos, neg).
Companion to investigation (e) — n_rollout_steps controller was
suspected of misalignment (256-8192 vs trade_duration ≈ 6 steps)
but turned out to be a K-loop param, not used in Bellman target.
1-step Bellman with γ-bootstrap is the actual mechanism; closed
without code change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -68,15 +68,31 @@
|
||||
// MARGIN adaptation slots (audit 2026-05-24 follow-up).
|
||||
#define RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX 482
|
||||
#define RL_REWARD_CLAMP_CLIP_RATE_TARGET_INDEX 483
|
||||
// C51 atom span ratchet slots (audit 2026-05-24 second follow-up).
|
||||
// Coupling: V_MAX tracks max(V_MAX_prev, max(1.0, WIN_clamp));
|
||||
// V_MIN tracks min(V_MIN_prev, min(-1.0, -LOSS_clamp)). Ratchet
|
||||
// (monotone-grow) prevents destabilising Q's learned distribution
|
||||
// when WIN_clamp temporarily shrinks.
|
||||
// C51 atom span slots (audit 2026-05-24 second follow-up).
|
||||
// Initial design: ratchet (monotone-grow). wwcsz followup
|
||||
// 2026-05-24: replaced with slow EWMA (α=0.001, half-life ~700
|
||||
// steps) to focus atom resolution on the ACTIVE reward range.
|
||||
// Static ratchet wasted resolution on rare tails — with avg
|
||||
// rewards in [-5, +5] but atom span at [-60, +20], Δz=4 meant Q
|
||||
// couldn't differentiate "slightly winning" from "slightly losing"
|
||||
// actions. Slow EWMA lets the span shrink toward the current
|
||||
// active range while floors at [-1, +1] preserve the original C51
|
||||
// baseline. The slow α gives Q's atom mapping time to be valid
|
||||
// across encoder/head co-adaptation.
|
||||
#define RL_C51_V_MAX_INDEX 484
|
||||
#define RL_C51_V_MIN_INDEX 485
|
||||
|
||||
// Adaptive RATIO slots (wwcsz followup) — negative tail signal that
|
||||
// drives RATIO = LOSS/WIN bound asymmetry. The static RATIO=3.0
|
||||
// over-emphasised loss-aversion vs observed |loss|/|win| ≈ 0.83.
|
||||
#define RL_NEG_SCALED_REWARD_MAX_INDEX 489
|
||||
#define RL_NEG_SCALED_REWARD_MAX_EMA_INDEX 490
|
||||
|
||||
#define MIN_WIN 1.0f
|
||||
#define MIN_RATIO 1.0f // no inverted asymmetry (loss never < win)
|
||||
#define MAX_RATIO 3.0f // no worse than original loss-aversion
|
||||
#define V_BOUND_FLOOR 1.0f // |V_MIN| and V_MAX both ≥ 1.0
|
||||
#define V_BOUND_EWMA_ALPHA 0.001f // slow — half-life ~700 steps
|
||||
#define WIENER_ALPHA_FLOOR 0.4f
|
||||
|
||||
// MARGIN Schulman bounded-step bounds + adjust rate per
|
||||
@@ -178,43 +194,77 @@ extern "C" __global__ void rl_reward_clamp_controller(
|
||||
isv[RL_REWARD_CLAMP_MARGIN_INDEX] = margin;
|
||||
}
|
||||
|
||||
// Step 4: compute adaptive WIN bound: MARGIN × EMA, floored.
|
||||
// No upper cap per the header note — the MARGIN ∈ [1, 5] × EMA
|
||||
// composition is the structural bound; downstream C51 projection
|
||||
// saturates beyond V_MAX without needing this kernel to enforce.
|
||||
const float win_eff = fmaxf(MIN_WIN, margin * ema_new);
|
||||
// ── Adaptive RATIO from observed neg/pos EMAs (wwcsz followup). ──
|
||||
//
|
||||
// The static RATIO=3.0 baked 3:1 loss-aversion into Q's value
|
||||
// representation. wwcsz showed observed |loss|/|win| ≈ 0.83,
|
||||
// making Q over-cautious. Track per-step max(-scaled, 0) via a
|
||||
// sparse-aware EMA (same discipline as pos_max_ema) and compute
|
||||
// RATIO = clamp(MIN=1.0, neg_ema/pos_ema, MAX=3.0). Floor 1.0
|
||||
// prevents inverted asymmetry; ceiling 3.0 preserves original
|
||||
// loss-aversion as the worst case.
|
||||
const float neg_max = isv[RL_NEG_SCALED_REWARD_MAX_INDEX];
|
||||
const float neg_prev = isv[RL_NEG_SCALED_REWARD_MAX_EMA_INDEX];
|
||||
float neg_new = neg_prev;
|
||||
if (neg_max > 0.0f) {
|
||||
if (neg_prev == 0.0f) {
|
||||
neg_new = neg_max;
|
||||
} else {
|
||||
const float a = fmaxf(alpha, WIENER_ALPHA_FLOOR);
|
||||
neg_new = (1.0f - a) * neg_prev + a * neg_max;
|
||||
}
|
||||
isv[RL_NEG_SCALED_REWARD_MAX_EMA_INDEX] = neg_new;
|
||||
}
|
||||
// Adaptive RATIO. Defer if EMAs not warmed up yet — keep
|
||||
// statically-seeded RATIO=3.0 until both EMAs are non-zero.
|
||||
if (ema_new > 0.0f && neg_new > 0.0f) {
|
||||
const float ratio_target = neg_new / ema_new;
|
||||
const float ratio_clamped = fmaxf(MIN_RATIO,
|
||||
fminf(MAX_RATIO, ratio_target));
|
||||
isv[RL_REWARD_CLAMP_RATIO_INDEX] = ratio_clamped;
|
||||
}
|
||||
|
||||
// LOSS = RATIO × WIN preserves asymmetry. RATIO seeded to 3.0.
|
||||
// Step 4: compute adaptive WIN/LOSS bounds: MARGIN × EMA, floored.
|
||||
// No upper cap on WIN per the header note — the MARGIN ∈ [1, 5]
|
||||
// × EMA composition is the structural bound. LOSS = RATIO × WIN
|
||||
// with RATIO now adaptive.
|
||||
const float win_eff = fmaxf(MIN_WIN, margin * ema_new);
|
||||
const float ratio = isv[RL_REWARD_CLAMP_RATIO_INDEX];
|
||||
const float loss_eff = ratio * win_eff;
|
||||
|
||||
isv[RL_REWARD_CLAMP_WIN_INDEX] = win_eff;
|
||||
isv[RL_REWARD_CLAMP_LOSS_INDEX] = loss_eff;
|
||||
|
||||
// ── C51 atom span ratchet (audit follow-up). ──────────────────
|
||||
// ── C51 atom span — slow EWMA (was ratchet). ──────────────────
|
||||
//
|
||||
// V_MAX/V_MIN track the reward clamp with monotone-grow semantics
|
||||
// — once we've admitted a winning trade of magnitude X, the C51
|
||||
// atom mapping permanently includes X in its support. This lets
|
||||
// bellman_target_projection's clip pass through wins > 1.0 to
|
||||
// distinct atoms instead of saturating the top atom.
|
||||
//
|
||||
// Floors: V_MAX ≥ 1.0, V_MIN ≤ -1.0 preserve the original C51
|
||||
// [-1, +1] support as a hard minimum (Q's distribution can't
|
||||
// become coarser than the design baseline).
|
||||
// wwcsz followup: replace monotone-grow with slow EWMA so atom
|
||||
// resolution focuses on the active reward range. Static ratchet
|
||||
// wasted Δz on rare tails (V_MAX=20 with rewards mostly in
|
||||
// [-5, +5] → Δz=2 wasted on extremes). Slow α=0.001
|
||||
// (half-life ~700 steps) gives Q's atom mapping time to be valid
|
||||
// across encoder/head co-adaptation. Floors at [-1, +1]
|
||||
// preserve the original C51 baseline as the worst case.
|
||||
const float v_max_prev = isv[RL_C51_V_MAX_INDEX];
|
||||
const float v_min_prev = isv[RL_C51_V_MIN_INDEX];
|
||||
const float v_max_floor = 1.0f;
|
||||
const float v_min_floor = -1.0f;
|
||||
// Ratchet: target = max(floor, win); kept = max(prev, target).
|
||||
const float v_max_target = fmaxf(v_max_floor, win_eff);
|
||||
const float v_min_target = fminf(v_min_floor, -loss_eff);
|
||||
// Bootstrap on sentinel 0 — first emit replaces directly with
|
||||
// target (per pearl_first_observation_bootstrap).
|
||||
const float v_max_new = (v_max_prev == 0.0f) ? v_max_target
|
||||
: fmaxf(v_max_prev, v_max_target);
|
||||
const float v_min_new = (v_min_prev == 0.0f) ? v_min_target
|
||||
: fminf(v_min_prev, v_min_target);
|
||||
const float v_max_target = fmaxf(V_BOUND_FLOOR, win_eff);
|
||||
const float v_min_target = fminf(-V_BOUND_FLOOR, -loss_eff);
|
||||
// Bootstrap on sentinel 0 — first emit replaces directly (per
|
||||
// pearl_first_observation_bootstrap).
|
||||
float v_max_new, v_min_new;
|
||||
if (v_max_prev == 0.0f) {
|
||||
v_max_new = v_max_target;
|
||||
} else {
|
||||
v_max_new = (1.0f - V_BOUND_EWMA_ALPHA) * v_max_prev
|
||||
+ V_BOUND_EWMA_ALPHA * v_max_target;
|
||||
v_max_new = fmaxf(V_BOUND_FLOOR, v_max_new);
|
||||
}
|
||||
if (v_min_prev == 0.0f) {
|
||||
v_min_new = v_min_target;
|
||||
} else {
|
||||
v_min_new = (1.0f - V_BOUND_EWMA_ALPHA) * v_min_prev
|
||||
+ V_BOUND_EWMA_ALPHA * v_min_target;
|
||||
v_min_new = fminf(-V_BOUND_FLOOR, v_min_new);
|
||||
}
|
||||
isv[RL_C51_V_MAX_INDEX] = v_max_new;
|
||||
isv[RL_C51_V_MIN_INDEX] = v_min_new;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user