Root cause of Q non-convergence: at reward_scale ≈ 0.003, a typical
±$5 trade maps to ±0.015 scaled reward. With 21 atoms spanning [-1,+1]
(step=0.1), the entire reward distribution falls within a SINGLE atom.
Q cannot distinguish wins from losses — confirmed by q_pi_agree ≈ 0.
Fix 1: Tighten atom span to [-0.5, +0.5]. Now atom_step = 0.05.
A ±$5 trade at scale=0.01 spans 2 atoms; at scale=0.05 spans 10.
The reward clamp controller ratchets the span at runtime if rewards
exceed the bounds.
Fix 2: PER capacity 4096 → 32768. At b=16, old capacity retained
only 256 unique steps — Q replayed each transition 1-2× before
eviction. New capacity retains 2048 steps, giving Q proper replay
depth for convergence.
Also aligns reward clamp bounds (WIN/LOSS) with the new span.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
79ac46c672