fix(rl): per-step ±2% clamp on reward_scale movement

The reward_scale oscillated 10,873× (min 0.000123, max 1.33) because
the Wiener-α=0.4 blend tracked sparse trade PnL spikes instantly.
Old transitions in PER had stale-scale rewards even with raw-reward
re-normalization (the current scale itself was unstable).

Per-step clamp: scale can only move ±2% from previous value.
Doubling takes ~35 steps, halving takes ~35 steps — fast enough to
track regime changes, stable enough for Q targets across the replay
window. Max oscillation over 1000 steps: ~7× (was 10,873×).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-25 13:50:25 +02:00
parent 3517830b1b
commit 408e0ef4ac

View File

@@ -130,6 +130,15 @@ extern "C" __global__ void rl_reward_scale_controller(
const float a = fmaxf(alpha_step, WIENER_ALPHA_FLOOR);
float out = (1.0f - a) * prev + a * target;
// Per-step movement clamp: scale moves at most ±2% from previous.
// Damps the 10,000× oscillation from sparse trade PnL spikes to
// a smooth ramp. At 2%/step, doubling takes ~35 steps, halving
// takes ~35 steps — fast enough to track regime changes, slow
// enough to give Q stable targets across the replay window.
const float max_move = prev * 1.02f;
const float min_move = prev * 0.98f;
out = fmaxf(min_move, fminf(out, max_move));
out = fmaxf(scale_min, fminf(out, REWARD_SCALE_MAX));
isv[RL_REWARD_SCALE_INDEX] = out;
}