Loss minimization (wr=0.561 but PnL negative — losses 32% > wins):
A1: Exempt FlatL/FlatS from confidence gate — exit actions never
blocked, model can always close losing positions.
A2+A3: New rl_drawdown_stop kernel — per-step drawdown penalty
(min(0, unrealized_r) × rate) creates continuous exit gradient.
Hard stop-loss force-closes when unrealized_r < -threshold.
Both ISV-driven (slots 586, 587).
A4: Adaptive LOSS clamp — tracks observed neg/pos EMA ratio instead
of static 3.0. LOSS = clamp(1.0, ratio×1.1, 3.0). Q sees
accurate loss magnitudes.
Performance:
B0: Remove gratuitous stream.synchronize() in apply_snapshot
(sim/mod.rs) — same-stream ordering makes it unnecessary.
Expected: -12-49ms/step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
G.2 disabled atom span adaptation to break a positive feedback loop.
With clamp bounds now FROZEN, re-enabling span adaptation is safe.
The span anchors on the CLAMP bounds (WIN=1.0, LOSS=3.0) — the
structural ceiling on post-clamp rewards that Q actually sees. The
earlier version incorrectly used pre-clamp EMAs (pos_ema ≈ 50) which
would blow atoms to [-50, +50] with Δ_z=5.
V_MAX EWMAs from 1.0 toward WIN=1.0 (stays put).
V_MIN EWMAs from -1.0 toward -LOSS=-3.0 (slowly widens to cover the
loss tail). Final asymmetric span [-3, +1] with Δ_z=0.2 gives Q
full resolution across the clamped reward range.
α=0.001 (half-life ~700 steps), floor ±1.0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the F.5 diagnosed l_v=104 + l_pi=-31 spike pattern at the root.
Pathology: `rl_reward_clamp_controller` widened the WIN/LOSS bounds
(slots 452/453) when clip_rate exceeded the 5% target — appeasement,
not control. The atom-span EWMA (slots 484/485) then ratcheted up to
track the wider WIN/LOSS. F.5 200-step smoke trajectory:
WIN: 1.0 → 41.3 (41×)
LOSS: 3.0 → 41.3 (14×)
V_MAX: 1.0 → 2.66
V_MIN: -1.0 → -2.74
Scaled rewards up to 14.71 flowed through unclamped, producing
advantage magnitudes of ~30 and PPO surrogate losses of ±30, V
regression losses up to 104. Pure positive-feedback loop: large
rewards → wider clamp → bigger V/Q targets → larger atom span →
larger reward signals permitted → repeat.
Fix: STOP writing to slots 452/453/484/485. The trainer-seeded
values (WIN=1.0, LOSS=3.0, V_MAX=1.0, V_MIN=-1.0) are the structural
bounds matching the C51 distributional Q head's design. Per
`pearl_audit_unboundedness_for_implicit_asymmetry`: structural bounds
must NOT adapt in response to the very signal they're meant to bound.
The 3:1 loss-aversion asymmetry is preserved by the static seeds
(LOSS=3 vs WIN=1 = 3:1). The C51 distributional resolution stays
matched to the bound. Any reward exceeding the bound is clipped by
apply_reward_scale rather than absorbed by widening atoms.
Diagnostic-only state retained:
* pos_max_ema (slot 478) — observed positive-tail magnitude
* neg_max_ema (slot 489) — observed negative-tail magnitude
* clip_rate_ema (slot 482) — fraction of steps where clamp fired
* MARGIN (slot 480) — what the controller WOULD widen to
* RATIO (slot 481) — what observed LOSS/WIN ratio implies
These surface what an unbounded controller WOULD adapt to under the
observed reward distribution — useful for understanding drift even
though the LOAD-BEARING slots are now static.
F.5 vs G.2 smoke comparison (same seed=4242, 200 steps, b_size=4):
Pre G.2 Post G.2 Reduction
l_pi abs_max 31.15 8.09 4×
l_v max 103.69 3.60 29×
l_v mean 1.79 0.19 10×
l_pi mean -0.12 0.06 ~stable
l_frd mean 0.43 0.50 unchanged
WIN bound →41.3 1.0 static
LOSS bound →41.3 3.0 static
V_MAX →2.66 1.0 static
V_MIN →-2.74 -1.0 static
Spike steps 20+ 5 ≥4×
Remaining 5 spikes are early-training noise (steps 11-59) that fade
naturally as V/Q converge. After step 59 only one mild spike at
step 131 (l_pi=3.35, l_v=2.75).
Pairs with G.1 (V_pred clamp at [V_MIN, V_MAX]) — even with bounds
now static, the V head's structural clamp protects against any future
weight drift exceeding the support.
Verification:
* cargo check -p ml-alpha → clean
* lib tests 66/66 (default), 5/6 ignored (1 pre-existing
fxcache_local_smoke env failure, unrelated)
* GPU tests: integrated_trainer_smoke 1/1 + frd_head 10/10 +
trade_management_kernels 5/5 → no regression
* audit-rust-consts → 0 flags
wwcsz analysis identified atom-resolution starvation + asymmetric
RATIO mismatch as the empirical ceiling on win rate (38.56% vs
break-even 45.3%). Three coupled fixes shipped in one pass per
"no deferrals":
(a) Adaptive RATIO from observed |loss|/|win| EMAs:
- apply_reward_scale tracks max(-scaled, 0) per step (slot 489)
- rl_reward_clamp_controller maintains neg_max_ema (slot 490,
sparse-aware like pos_max_ema)
- RATIO = clamp(MIN=1.0, neg_ema/pos_ema, MAX=3.0); writes to
slot 481
- Removes built-in 3:1 loss-aversion bias when reality is
symmetric (wwcsz: actual avg|loss|/avg(win) = 0.83). Floor
1.0 prevents inverted asymmetry; ceiling 3.0 preserves
original loss-aversion as the worst case.
(b) C51 V_MAX/V_MIN: ratchet → slow EWMA (α=0.001, half-life ~700
steps):
- Static ratchet wasted atom resolution on rare tails — wwcsz
had V_MIN=-60, V_MAX=20 but realized rewards mostly in [-5, +5]
(Δz=4 vs typical reward magnitude 1-5)
- Slow EWMA lets atom span shrink toward active reward range,
gaining resolution where data lives. Floors at [-1, +1]
preserve original C51 baseline as the worst case.
- Slow α gives Q's atom mapping time to be valid across
encoder/head co-adaptation (vs aggressive EWMA which would
invalidate Q's learned distribution every step)
(c) Q→π distillation λ bumped 0.01 → 0.05:
- wwcsz showed KL dropped 2.10 → 0.30 with λ=0.01 — Q signal
landing but conservatively. Bump tests whether stronger Q
pull translates to better policy → better R/done.
Diag exposes neg_scaled_max + neg_scaled_max_ema so the RATIO
adaptation chain is observable.
apply_reward_scale shared_mem doubled from 2× to 3× block × f32
to fit the three parallel reductions (abs, pos, neg).
Companion to investigation (e) — n_rollout_steps controller was
suspected of misalignment (256-8192 vs trade_duration ≈ 6 steps)
but turned out to be a K-loop param, not used in Bellman target.
1-step Bellman with γ-bootstrap is the actual mechanism; closed
without code change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two coupled fixes addressing vj5f6 findings:
(1) WIN_clamp oscillation — sparse-aware EMA
vj5f6 showed WIN_clamp oscillating 1.0 ↔ 67.0 across 40k steps.
Root cause: the Wiener-α blend in rl_reward_clamp_controller
treated pos_max=0 as "no win this step ≡ win magnitude is zero,"
exponentially decaying the EMA toward 0 during dry-spell windows
(no closed winning trades). With α=0.4, ten dry steps decayed EMA
by 0.6^10 ≈ 0.006, collapsing WIN back to MIN_WIN=1.0 floor.
Fix: only update pos_max_ema AND clip_rate_ema AND MARGIN when
pos_max > 0. A dry step is "no signal," not "zero signal." The
EMA retains its last winning-period estimate; the controller
doesn't ratchet on stale data.
(2) Q→π distillation — couples Q's improved calibration to π
vj5f6 showed l_q dropping 100× (2.37 → 0.02) but reward economics
IDENTICAL to 8xwq8 (no C51 V_MAX lift). Per Option B, π drives
action selection but is trained by PPO surrogate using advantage
= returns - V. V regression doesn't benefit from C51 calibration,
so Q's improved knowledge stays trapped in the critic head.
Deep audit revealed a self-reinforcing defensive trap:
Q learned "big positions lose money" → π_target favors small
actions → π picks a3+a4 (tiny long / Hold) → position lots ≈ 0
→ rewards mostly 0 → V learns "everything is 0" → V_pred ≈ 0
→ advantage = returns - V_pred ≈ 0 → PPO gradient ≈ 0 → π
frozen at defensive attractor → loop. Trade count dropped 3×
(rdgzl 25k → 8xwq8/vj5f6 9k closes per 10k steps), win rate
inversely correlated with l_q (50% early → 22% late) because
only forced closes happen (stops = losses).
Fix: new rl_q_pi_distill_grad.cu computes
π_target = softmax(E_Q[s,*] / τ)
∂L/∂logits[a] = λ × (π_new(a) - π_target(a))
and ADDS this gradient to pi_grad_logits AFTER the PPO surrogate
backward. Couples Q's preferences directly into π's update without
going through advantage. λ=0.01 (small, PPO dominant), τ=1.0
(canonical Boltzmann). 3 new ISV slots (λ + τ + KL_ema diag).
Diag exposes c51_v_max/v_min, q_distill_lambda/temperature, and
q_distill_kl_ema so the adaptation + distillation loop is observable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
rdgzl follow-up — chain hypothesis layer 2:
reward clamp lift unlocked V regression + PPO advantage (R/done
-$1.39 → -$0.48), but Q's distributional learning was structurally
capped at hardcoded V_MAX=1.0 in bellman_target_projection.cu —
any Bellman target > 1.0 categorically projected to atom 20 (top)
regardless of clamp. Even with WIN=3.8 clamp, Q never saw a +3.8
reward signal as distinct from a +1.0 reward signal.
This commit makes V_MIN/V_MAX ISV-driven with monotone-grow ratchet
coupled to the reward clamp. The C51 distribution support adapts
WITHOUT destabilising Q's learned values — atom 20 always represents
at least the widest WIN we've ever admitted (only grows, never shrinks).
Implementation:
- 2 new ISV slots (484 V_MAX, 485 V_MIN) with [-1, +1] floors
seeded by rl_isv_write
- rl_reward_clamp_controller.cu also ratchets these slots:
V_MAX_new = max(V_MAX_prev, max(1.0, WIN_clamp))
V_MIN_new = min(V_MIN_prev, min(-1.0, -LOSS_clamp))
- bellman_target_projection.cu reads V_MIN/V_MAX from ISV, derives
DELTA_Z inline (was #define)
- New rl_atom_support_update.cu (21-thread block) refreshes
atom_supports_d = linspace(V_MIN, V_MAX, 21) per step so
downstream C51 kernels (argmax_expected_q, rl_action_kernel,
dqn_distributional_q) see the current span
- Trainer launches atom-support updater after each reward-clamp
controller launch (both helper + step_with_lobsim inline paths)
- Diag exposes c51_v_max + c51_v_min for adaptation visibility
Floors at [-1, +1] preserve original C51 design as hard minimum —
the atom support can only become wider, never narrower than the
baseline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
rdgzl follow-up — chain hypothesis test:
clip rate stayed at 25-40% across windows (target ~5%)
win rate oscillated 27-47% with no clear trend
positive-tail distribution: p50=1.85 p90=10.1 p99=76.9 max=2230
MAX_WIN=20 hit ceiling in EVERY window (load-bearing cap)
static MARGIN=1.5 couldn't chase the tail
Two interventions in one commit:
(1) MARGIN is now adaptive in rl_reward_clamp_controller.cu via a
Schulman bounded-step on clip-rate EMA vs target:
clip_indicator = (pos_max > current_WIN && pos_max > 0) ? 1 : 0
clip_rate_ema = (1-α) * prev + α * indicator (α=0.05)
if clip_rate_ema > target × 1.5 → MARGIN *= 1.2 (up to MAX_MARGIN=5)
if clip_rate_ema < target / 1.5 → MARGIN /= 1.2 (down to MIN_MARGIN=1)
Target clip rate seeded at 0.05 — accept 5% tail outliers, capture
the rest. Two new ISV slots (482 clip-rate EMA, 483 target).
(2) MAX_WIN cap REMOVED — the hardcoded ceiling defeated the purpose
of adaptation. Safety reasoning: WIN = MARGIN × pos_max_ema with
MARGIN ∈ [1, 5] and pos_max_ema bounded by reward_scale × raw_PnL
(both finite). MIN_WIN=1.0 floor retained.
Diag exposes clip_rate_ema + reward_clamp_clip_rate_target so the
adaptation loop is observable in the JSONL.
KNOWN DOWNSTREAM CEILING: bellman_target_projection.cu hardcodes C51
atom span at V_MIN=-1.0, V_MAX=+1.0. Any Bellman target outside this
range is categorically clipped regardless of our reward clamp. So
lifting WIN > 1.0 helps V regression + PPO advantage (which see real
magnitude) but Q's distributional learning is structurally capped at
V_MAX=1.0. A separate intervention to lift C51 V_MAX would be needed
to unlock Q's atom-distribution learning beyond +1.0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha-rl-rmgm5 (commit a776fab31) deep-diag finding:
- static `[-3, +1]` clamp fired on 85% of steps
- pre-clamp max: p95=15.5 p99=45.2 max=2830 (in WIN-bound units)
- win distribution avg=+$2.06 max=+$11 squished to +1.0
- loss distribution avg=-$3.84 routinely exceeded -3.0
- per-trade EV = 0.357 * 2.06 + 0.643 * (-3.84) = -$1.74
The static clamp was crushing the gradient differential between
profitable and unprofitable trades, leaving Q with no signal to
distinguish good actions from bad. Adaptive bounds let the actual
winning-trade distribution reach the C51 atom support.
Implementation:
- apply_reward_scale.cu: dual reduction (max|scaled| + max(positive
scaled, 0)); positive-tail published to new ISV slot 478
- rl_reward_clamp_controller.cu: maintains EMA of slot 478 in slot
479 via Wiener-α blend (floor 0.4 per pearl_wiener_alpha_floor);
writes WIN_eff = clamp(MARGIN * EMA, [1.0, 20.0]) to slot 452
and LOSS_eff = RATIO * WIN to slot 453
- 4 new ISV slots (478-481): raw + EMA + margin + ratio
- Trainer per-step launch added at both apply_reward_scale sites
(helper method + step_with_lobsim inline path)
- Shared-mem bytes doubled at both apply_reward_scale launches
- Static-default seeds added to with_controllers_bootstrapped
(MARGIN=1.5, RATIO=3.0) — controller's bootstrap-on-sentinel
path takes over from these once first positive reward observed
- Diag JSONL exposes pos_scaled_max, pos_scaled_max_ema, and the
margin/ratio config
Preserves 3:1 loss-aversion asymmetry per
pearl_audit_unboundedness_for_implicit_asymmetry — RATIO is itself
ISV-tunable. WIN floor 1.0 / ceiling 20.0 are hardcoded per the
user-stated "floors and clamp bounds" exemption (2026-05-24).
Adds #![recursion_limit = "256"] to alpha_rl_train.rs — the diag
json! block crossed serde_json's default 128-arg expansion budget.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
