fix(rl): WIN/LOSS + C51 atom span are STRUCTURAL, not adaptive (G.2)
Closes the F.5 diagnosed l_v=104 + l_pi=-31 spike pattern at the root.
Pathology: `rl_reward_clamp_controller` widened the WIN/LOSS bounds
(slots 452/453) when clip_rate exceeded the 5% target — appeasement,
not control. The atom-span EWMA (slots 484/485) then ratcheted up to
track the wider WIN/LOSS. F.5 200-step smoke trajectory:
WIN: 1.0 → 41.3 (41×)
LOSS: 3.0 → 41.3 (14×)
V_MAX: 1.0 → 2.66
V_MIN: -1.0 → -2.74
Scaled rewards up to 14.71 flowed through unclamped, producing
advantage magnitudes of ~30 and PPO surrogate losses of ±30, V
regression losses up to 104. Pure positive-feedback loop: large
rewards → wider clamp → bigger V/Q targets → larger atom span →
larger reward signals permitted → repeat.
Fix: STOP writing to slots 452/453/484/485. The trainer-seeded
values (WIN=1.0, LOSS=3.0, V_MAX=1.0, V_MIN=-1.0) are the structural
bounds matching the C51 distributional Q head's design. Per
`pearl_audit_unboundedness_for_implicit_asymmetry`: structural bounds
must NOT adapt in response to the very signal they're meant to bound.
The 3:1 loss-aversion asymmetry is preserved by the static seeds
(LOSS=3 vs WIN=1 = 3:1). The C51 distributional resolution stays
matched to the bound. Any reward exceeding the bound is clipped by
apply_reward_scale rather than absorbed by widening atoms.
Diagnostic-only state retained:
* pos_max_ema (slot 478) — observed positive-tail magnitude
* neg_max_ema (slot 489) — observed negative-tail magnitude
* clip_rate_ema (slot 482) — fraction of steps where clamp fired
* MARGIN (slot 480) — what the controller WOULD widen to
* RATIO (slot 481) — what observed LOSS/WIN ratio implies
These surface what an unbounded controller WOULD adapt to under the
observed reward distribution — useful for understanding drift even
though the LOAD-BEARING slots are now static.
F.5 vs G.2 smoke comparison (same seed=4242, 200 steps, b_size=4):
Pre G.2 Post G.2 Reduction
l_pi abs_max 31.15 8.09 4×
l_v max 103.69 3.60 29×
l_v mean 1.79 0.19 10×
l_pi mean -0.12 0.06 ~stable
l_frd mean 0.43 0.50 unchanged
WIN bound →41.3 1.0 static
LOSS bound →41.3 3.0 static
V_MAX →2.66 1.0 static
V_MIN →-2.74 -1.0 static
Spike steps 20+ 5 ≥4×
Remaining 5 spikes are early-training noise (steps 11-59) that fade
naturally as V/Q converge. After step 59 only one mild spike at
step 131 (l_pi=3.35, l_v=2.75).
Pairs with G.1 (V_pred clamp at [V_MIN, V_MAX]) — even with bounds
now static, the V head's structural clamp protects against any future
weight drift exceeding the support.
Verification:
* cargo check -p ml-alpha → clean
* lib tests 66/66 (default), 5/6 ignored (1 pre-existing
fxcache_local_smoke env failure, unrelated)
* GPU tests: integrated_trainer_smoke 1/1 + frd_head 10/10 +
trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
This commit is contained in:
jgrusewski
@@ -224,47 +224,51 @@ extern "C" __global__ void rl_reward_clamp_controller(
|
||||
isv[RL_REWARD_CLAMP_RATIO_INDEX] = ratio_clamped;
|
||||
}
|
||||
|
||||
// Step 4: compute adaptive WIN/LOSS bounds: MARGIN × EMA, floored.
|
||||
// No upper cap on WIN per the header note — the MARGIN ∈ [1, 5]
|
||||
// × EMA composition is the structural bound. LOSS = RATIO × WIN
|
||||
// with RATIO now adaptive.
|
||||
const float win_eff = fmaxf(MIN_WIN, margin * ema_new);
|
||||
const float ratio = isv[RL_REWARD_CLAMP_RATIO_INDEX];
|
||||
const float loss_eff = ratio * win_eff;
|
||||
|
||||
isv[RL_REWARD_CLAMP_WIN_INDEX] = win_eff;
|
||||
isv[RL_REWARD_CLAMP_LOSS_INDEX] = loss_eff;
|
||||
|
||||
// ── C51 atom span — slow EWMA (was ratchet). ──────────────────
|
||||
// Step 4 (G.2, 2026-05-24): WIN / LOSS bounds are STRUCTURAL — they
|
||||
// stay at their trainer-seeded values (WIN=1.0, LOSS=3.0) matching
|
||||
// the C51 atom span × 3:1 loss-aversion asymmetry. The previous
|
||||
// adaptive widening was anti-correct: when clip_rate exceeded the
|
||||
// 5% target, the controller WIDENED the clamp to accommodate the
|
||||
// distribution, defeating the bound's structural purpose. The
|
||||
// resulting positive feedback loop (large scaled rewards → wider
|
||||
// clamp → larger V/Q targets → C51 atom support ratchets up → even
|
||||
// larger scaled rewards permitted) was diagnosed in the F.5 200-step
|
||||
// local smoke: WIN drifted 1.0 → 41.3 (41×) in 150 steps, scaled
|
||||
// rewards of 14.71 flowed through, V regression spiked to l_v=104.
|
||||
//
|
||||
// wwcsz followup: replace monotone-grow with slow EWMA so atom
|
||||
// resolution focuses on the active reward range. Static ratchet
|
||||
// wasted Δz on rare tails (V_MAX=20 with rewards mostly in
|
||||
// [-5, +5] → Δz=2 wasted on extremes). Slow α=0.001
|
||||
// (half-life ~700 steps) gives Q's atom mapping time to be valid
|
||||
// across encoder/head co-adaptation. Floors at [-1, +1]
|
||||
// preserve the original C51 baseline as the worst case.
|
||||
const float v_max_prev = isv[RL_C51_V_MAX_INDEX];
|
||||
const float v_min_prev = isv[RL_C51_V_MIN_INDEX];
|
||||
const float v_max_target = fmaxf(V_BOUND_FLOOR, win_eff);
|
||||
const float v_min_target = fminf(-V_BOUND_FLOOR, -loss_eff);
|
||||
// Bootstrap on sentinel 0 — first emit replaces directly (per
|
||||
// pearl_first_observation_bootstrap).
|
||||
float v_max_new, v_min_new;
|
||||
if (v_max_prev == 0.0f) {
|
||||
v_max_new = v_max_target;
|
||||
} else {
|
||||
v_max_new = (1.0f - V_BOUND_EWMA_ALPHA) * v_max_prev
|
||||
+ V_BOUND_EWMA_ALPHA * v_max_target;
|
||||
v_max_new = fmaxf(V_BOUND_FLOOR, v_max_new);
|
||||
}
|
||||
if (v_min_prev == 0.0f) {
|
||||
v_min_new = v_min_target;
|
||||
} else {
|
||||
v_min_new = (1.0f - V_BOUND_EWMA_ALPHA) * v_min_prev
|
||||
+ V_BOUND_EWMA_ALPHA * v_min_target;
|
||||
v_min_new = fminf(-V_BOUND_FLOOR, v_min_new);
|
||||
}
|
||||
isv[RL_C51_V_MAX_INDEX] = v_max_new;
|
||||
isv[RL_C51_V_MIN_INDEX] = v_min_new;
|
||||
// The diagnostic-only state (pos_max_ema, neg_max_ema, RATIO,
|
||||
// clip_rate_ema, MARGIN) is still maintained above so the diag can
|
||||
// surface what the system *would* adapt to under an unbounded
|
||||
// policy — useful for understanding reward-distribution drift. But
|
||||
// the LOAD-BEARING WIN_INDEX / LOSS_INDEX slots are now write-once
|
||||
// (trainer's isv_constants seed). Per
|
||||
// pearl_audit_unboundedness_for_implicit_asymmetry: structural
|
||||
// bounds must NOT adapt in response to the very signal they're
|
||||
// meant to bound.
|
||||
//
|
||||
// Suppress unused-variable warnings on the diagnostic-only blocks.
|
||||
(void) margin;
|
||||
(void) ema_new;
|
||||
|
||||
// ── C51 atom span — STRUCTURAL (G.2, 2026-05-24). ─────────────
|
||||
//
|
||||
// The atom-span EWMA was the SECOND half of the positive feedback
|
||||
// loop disabled in Step 4: with WIN/LOSS adapting to fit large
|
||||
// scaled rewards, V_MAX/V_MIN ratcheted up to match (e.g., F.5
|
||||
// smoke saw V_MAX=2.66 by step 149 from a seed of 1.0). Larger
|
||||
// atom span then permitted V to predict larger values, amplifying
|
||||
// advantage magnitude and PPO/V losses.
|
||||
//
|
||||
// Fix: the C51 atom span stays at its trainer-seeded value
|
||||
// (V_MAX=1.0, V_MIN=-1.0). Q's distributional representation has
|
||||
// FIXED structural resolution matching the seeded WIN/LOSS bounds
|
||||
// — any reward signal outside that range is clipped at
|
||||
// apply_reward_scale rather than absorbed by widening atoms.
|
||||
// Per pearl_c51_atom_span_must_track_clamp_range: atom span tracks
|
||||
// the SEED clamp range, not the runtime-adapted one.
|
||||
//
|
||||
// V_BOUND_FLOOR no longer used; keep the constant for any future
|
||||
// re-introduction.
|
||||
(void) V_BOUND_FLOOR;
|
||||
(void) V_BOUND_EWMA_ALPHA;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user