Root cause: C51 distributional softmax structurally favors zero/low-variance
actions (Flat for direction, Small for magnitude). This created irrecoverable
feedback loops through FOUR paths: gradient, Bellman target argmax, action
selection, and the Q-gap conviction filter.
Direction branch (new):
- Zero C51 gradient for direction (d==0) — same treatment as magnitude
- Boltzmann softmax replaces argmax for direction selection (tau=2×Q_range)
- 2× MSE gradient amplification for direction branch head
- Mean advantage (not C51 softmax) for Bellman target argmax at d==0
- Q-gap conviction filter REMOVED from training path (redundant with
Boltzmann, harmful after high-Sharpe epochs where Bellman max bootstrap
raises Q(Flat) towards Q(directional), shrinking the gap)
Magnitude branch (Bellman target fix):
- Mean advantage for Bellman target argmax at d==1 in both MSE + C51 kernels
- Proper softmax retained for online_eq and target_eq (learning objective)
Metric fixes:
- Diversity denominator: 9 → 7 (Flat forces mag=Half, max reachable is 7)
- Threshold: 0.5% of total → 1% of directional (Flat-dominant policies
mechanically killed diversity under the old metric)
Result: 7/7 dir×mag diversity sustained epochs 7-10, Flat stable at 7-9%
(was 60-87% growing). 19/19 smoke tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5d5a2c1f3d