Phase E.1 Task 12. Stabilized H=600 DQN smoke ran end-to-end on full
500K-snapshot data. All three preconditions PASS but the rvr gate FAILS:
Q_SPREAD_EMA = 35.54 ≥ 0.05 PASS
ACTION_ENTROPY_EMA = 1.91 ≥ 1.099 PASS
RETURN_VS_RANDOM_EMA = -2.58 ≥ 0.0 FAIL ← policy WORSE than random
EARLY_Q_MOVEMENT_EMA = 0.096 ≥ 0.01 PASS
rvr trajectory across 1000 episodes:
ep 50 | rollout_R= -11049 | rvr = -1.18 (near random)
ep 200 | rollout_R= -9620 | rvr = -0.97 (briefly improving)
ep 600 | rollout_R= -18140 | rvr = -2.03 (degrading)
ep 1000 | rollout_R= -18272 | rvr = -2.58 (deterministic-bad)
Random baseline at H=600 = -5185 mean, std=4953. Trained policy loses
3.5× worse than random.
Diagnostics performed:
reward_scale=10000 → rvr=-2.37 (no help)
alpha_m=0 (vanilla DQN, no Munchausen) → rvr=-2.44 (no help)
Root cause: "first-best-action lock-in" of linear Q + ε-greedy. DQN's
TD update only modifies Q[a] for the TAKEN action; with ε-decay, the
argmax action self-reinforces while other actions' Q stays frozen at
random Xavier init. Random policy samples all 9 uniformly → 11% chance
of "lucky" close-position at any step → exits bad trades. Trained
policy converges deterministic on one bad action → never exits.
Per plan: pivot to NoisyNet (Task 19) — parameter-space noise breaks
the lock-in. Alternative: wire alpha_logit from Phase 1d.3 stacker
(currently hardcoded to 0.0 placeholder) so the policy has actual
directional signal to work with.
Kill-criteria gate worked as designed — correctly flagged that linear
Q + ε-greedy on this env is insufficient without further intervention.
Memory note: project_phase_e1_h600_smoke_verdict.md (full analysis +
hypothesis tree + recommended next steps).
8548d126fd