fix(rl): per-step ±2% clamp on reward_scale movement
The reward_scale oscillated 10,873× (min 0.000123, max 1.33) because the Wiener-α=0.4 blend tracked sparse trade PnL spikes instantly. Old transitions in PER had stale-scale rewards even with raw-reward re-normalization (the current scale itself was unstable). Per-step clamp: scale can only move ±2% from previous value. Doubling takes ~35 steps, halving takes ~35 steps — fast enough to track regime changes, stable enough for Q targets across the replay window. Max oscillation over 1000 steps: ~7× (was 10,873×). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -130,6 +130,15 @@ extern "C" __global__ void rl_reward_scale_controller(
|
||||
const float a = fmaxf(alpha_step, WIENER_ALPHA_FLOOR);
|
||||
float out = (1.0f - a) * prev + a * target;
|
||||
|
||||
// Per-step movement clamp: scale moves at most ±2% from previous.
|
||||
// Damps the 10,000× oscillation from sparse trade PnL spikes to
|
||||
// a smooth ramp. At 2%/step, doubling takes ~35 steps, halving
|
||||
// takes ~35 steps — fast enough to track regime changes, slow
|
||||
// enough to give Q stable targets across the replay window.
|
||||
const float max_move = prev * 1.02f;
|
||||
const float min_move = prev * 0.98f;
|
||||
out = fmaxf(min_move, fminf(out, max_move));
|
||||
|
||||
out = fmaxf(scale_min, fminf(out, REWARD_SCALE_MAX));
|
||||
isv[RL_REWARD_SCALE_INDEX] = out;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user