gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(rl): sparse-aware EMA + Q→π distillation breaks defensive trap

Browse Source 
Two coupled fixes addressing vj5f6 findings:

(1) WIN_clamp oscillation — sparse-aware EMA

  vj5f6 showed WIN_clamp oscillating 1.0 ↔ 67.0 across 40k steps.
  Root cause: the Wiener-α blend in rl_reward_clamp_controller
  treated pos_max=0 as "no win this step ≡ win magnitude is zero,"
  exponentially decaying the EMA toward 0 during dry-spell windows
  (no closed winning trades). With α=0.4, ten dry steps decayed EMA
  by 0.6^10 ≈ 0.006, collapsing WIN back to MIN_WIN=1.0 floor.

  Fix: only update pos_max_ema AND clip_rate_ema AND MARGIN when
  pos_max > 0. A dry step is "no signal," not "zero signal." The
  EMA retains its last winning-period estimate; the controller
  doesn't ratchet on stale data.

(2) Q→π distillation — couples Q's improved calibration to π

  vj5f6 showed l_q dropping 100× (2.37 → 0.02) but reward economics
  IDENTICAL to 8xwq8 (no C51 V_MAX lift). Per Option B, π drives
  action selection but is trained by PPO surrogate using advantage
  = returns - V. V regression doesn't benefit from C51 calibration,
  so Q's improved knowledge stays trapped in the critic head.

  Deep audit revealed a self-reinforcing defensive trap:
    Q learned "big positions lose money" → π_target favors small
    actions → π picks a3+a4 (tiny long / Hold) → position lots ≈ 0
    → rewards mostly 0 → V learns "everything is 0" → V_pred ≈ 0
    → advantage = returns - V_pred ≈ 0 → PPO gradient ≈ 0 → π
    frozen at defensive attractor → loop. Trade count dropped 3×
    (rdgzl 25k → 8xwq8/vj5f6 9k closes per 10k steps), win rate
    inversely correlated with l_q (50% early → 22% late) because
    only forced closes happen (stops = losses).

  Fix: new rl_q_pi_distill_grad.cu computes
    π_target = softmax(E_Q[s,*] / τ)
    ∂L/∂logits[a] = λ × (π_new(a) - π_target(a))
  and ADDS this gradient to pi_grad_logits AFTER the PPO surrogate
  backward. Couples Q's preferences directly into π's update without
  going through advantage. λ=0.01 (small, PPO dominant), τ=1.0
  (canonical Boltzmann). 3 new ISV slots (λ + τ + KL_ema diag).

Diag exposes c51_v_max/v_min, q_distill_lambda/temperature, and
q_distill_kl_ema so the adaptation + distillation loop is observable.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-24 13:47:21 +02:00
parent 2d498bec3a
commit 79756a2153
6 changed files with 329 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

97
crates/ml-alpha/cuda/rl_reward_clamp_controller.cu
View File

@@ -101,26 +101,34 @@ extern "C" __global__ void rl_reward_clamp_controller(
const float pos_max = isv[RL_POS_SCALED_REWARD_MAX_INDEX];
const float ema_prev = isv[RL_POS_SCALED_REWARD_MAX_EMA_INDEX];
// EMA maintenance — bootstrap on sentinel 0 with first non-zero
// raw observation; otherwise Wiener-α blend with floor.
float ema_new;
if (ema_prev == 0.0f) {
if (pos_max == 0.0f) {
// Both prev and obs sentinels — defer adaptation, leave the
// statically-seeded WIN(1.0) / LOSS(3.0) bounds untouched.
// Next call where the kernel observes a positive reward
// will bootstrap the EMA and start adapting.
return;
// EMA maintenance — sparse-sample-aware. Only update the EMA on
// steps that ACTUALLY observed a positive reward (pos_max > 0).
//
// Audit 2026-05-24 (vj5f6 follow-up): the prior implementation
// blended pos_max=0 into the EMA via the Wiener-α step, which
// exponentially decayed the EMA toward 0 during dry-spell windows
// (no closed winning trades). With α=0.4, ten consecutive
// dry steps decay EMA by 0.6^10 ≈ 0.006, collapsing the WIN
// bound back to the MIN_WIN floor. WIN_clamp was observed
// oscillating 1.0 ↔ 67.0 across 40k steps in vj5f6.
//
// Fix: pos_max=0 means "no signal this step," not "win magnitude
// is zero." Skip EMA updates during dry spells; the EMA retains
// the last winning-period estimate until the next observation.
float ema_new = ema_prev;
if (pos_max > 0.0f) {
if (ema_prev == 0.0f) {
ema_new = pos_max; // first-observation bootstrap
} else {
const float a = fmaxf(alpha, WIENER_ALPHA_FLOOR);
ema_new = (1.0f - a) * ema_prev + a * pos_max;
}
ema_new = pos_max;
} else {
const float a = fmaxf(alpha, WIENER_ALPHA_FLOOR);
// If pos_max==0 this step (no positive reward sample), still
// blend toward 0 — captures regimes where wins disappear. The
// EMA decays gracefully toward 0 over ~1/alpha steps.
ema_new = (1.0f - a) * ema_prev + a * pos_max;
isv[RL_POS_SCALED_REWARD_MAX_EMA_INDEX] = ema_new;
}
isv[RL_POS_SCALED_REWARD_MAX_EMA_INDEX] = ema_new;
// If ema_prev was 0 AND pos_max was 0 → ema_new still 0 here →
// WIN_eff below = max(MIN_WIN, MARGIN × 0) = MIN_WIN. That's
// the correct "no data yet" behavior (statically-seeded WIN=1.0
// remains effective).
// ── MARGIN adaptation from clip-rate EMA (audit follow-up). ───
//
@@ -136,36 +144,39 @@ extern "C" __global__ void rl_reward_clamp_controller(
clip_indicator = (pos_max > win_active) ? 1.0f : 0.0f;
}
// Step 2: blend clip indicator into EMA. Fixed α=0.05 here (not
// Wiener-driven — the indicator is bimodal 0/1 so variance-based α
// would saturate; constant α gives ~20-step half-life which
// matches the policy-update cadence).
// Step 2: blend clip indicator into EMA. Sparse-aware — only
// update when pos_max > 0 (matches the pos_max EMA discipline).
// A step with pos_max=0 isn't "0% clipped"; it's "no data." Mixing
// dry steps in would bias clip_rate_ema downward and the MARGIN
// controller would shrink WIN unnecessarily.
const float clip_rate_prev = isv[RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX];
float clip_rate_new;
if (clip_rate_prev == 0.0f) {
// Bootstrap on sentinel — set to current indicator (avoids
// long warmup where MARGIN stays static).
clip_rate_new = clip_indicator;
} else {
clip_rate_new = (1.0f - CLIP_RATE_EMA_ALPHA) * clip_rate_prev
+ CLIP_RATE_EMA_ALPHA * clip_indicator;
float clip_rate_new = clip_rate_prev;
if (pos_max > 0.0f) {
if (clip_rate_prev == 0.0f) {
clip_rate_new = clip_indicator; // bootstrap
} else {
clip_rate_new = (1.0f - CLIP_RATE_EMA_ALPHA) * clip_rate_prev
+ CLIP_RATE_EMA_ALPHA * clip_indicator;
}
isv[RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX] = clip_rate_new;
}
isv[RL_REWARD_CLAMP_CLIP_RATE_EMA_INDEX] = clip_rate_new;
// Step 3: Schulman bounded step on MARGIN — if clip rate > target
// × TOLERANCE, raise MARGIN (widen WIN, capture more tail); if
// clip rate < target / TOLERANCE, lower MARGIN (tighten WIN,
// sharper regression). Dead-zone in between prevents oscillation.
const float clip_target = isv[RL_REWARD_CLAMP_CLIP_RATE_TARGET_INDEX];
// Step 3: Schulman bounded step on MARGIN — only adjust when we
// actually have a fresh clip-rate observation. If pos_max=0 this
// step, MARGIN retains its prior value (consistent with the EMA
// behavior above; the controller doesn't ratchet on stale data).
float margin = isv[RL_REWARD_CLAMP_MARGIN_INDEX];
const float upper = clip_target * MARGIN_TOLERANCE;
const float lower = clip_target / MARGIN_TOLERANCE;
if (clip_rate_new > upper) {
margin = fminf(MAX_MARGIN, margin * MARGIN_ADJUST_RATE);
} else if (clip_rate_new < lower) {
margin = fmaxf(MIN_MARGIN, margin / MARGIN_ADJUST_RATE);
if (pos_max > 0.0f) {
const float clip_target = isv[RL_REWARD_CLAMP_CLIP_RATE_TARGET_INDEX];
const float upper = clip_target * MARGIN_TOLERANCE;
const float lower = clip_target / MARGIN_TOLERANCE;
if (clip_rate_new > upper) {
margin = fminf(MAX_MARGIN, margin * MARGIN_ADJUST_RATE);
} else if (clip_rate_new < lower) {
margin = fmaxf(MIN_MARGIN, margin / MARGIN_ADJUST_RATE);
}
isv[RL_REWARD_CLAMP_MARGIN_INDEX] = margin;
}
isv[RL_REWARD_CLAMP_MARGIN_INDEX] = margin;
// Step 4: compute adaptive WIN bound: MARGIN × EMA, floored.
// No upper cap per the header note — the MARGIN ∈ [1, 5] × EMA