The C2 fix from the previous session was too aggressive — it eliminated
ALL exploration mechanisms simultaneously:
1. epsilon forced to 0.0 when noisy nets active (select_action)
2. count bonus removed from Q-value computation (metrics only)
3. noisy_epsilon_floor config field declared but never read
This left noisy nets as the sole exploration mechanism, which produces
perturbations too small to overcome Q-value gaps (A4=0.12 vs others≈0.02).
Result: 20/20 hyperopt trials hit fallback objective with 1/5 diversity.
Fixes:
- select_action: use noisy_epsilon_floor (not 0.0) as effective_epsilon
when noisy nets active — guarantees minimum random action rate
- select_action: re-enable UCB count bonus on Q-values before argmax
(both IQN and standard paths) for directed exploration
- select_action_with_confidence: same fixes for consistency
- trainer: set epsilon to noisy_epsilon_floor (not 0.0) at init
- hyperopt: widen noisy_epsilon_floor range from [0.0, 0.05] to
[0.03, 0.15] with default 0.05
Exploration now has two complementary mechanisms:
- noisy_epsilon_floor: random actions feed diverse replay buffer
- count bonus: UCB term biases greedy selection toward under-visited actions
- noisy nets: weight perturbation adds stochasticity to Q-values
select_action_inference (production) is unchanged — pure exploitation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
e412f12e33