foxhunt/docs/plans/2026-03-06-dqn-hyperopt-overhaul-design.md

DQN Hyperopt Overhaul Design

Problem

Hyperopt produces configs with great objectives (e.g., -464) that fail catastrophically on out-of-sample evaluation (Sharpe=-284, MaxDD=100%, -100% return, 58K trades in 3 months).

Root cause analysis identified 13 issues; 6 were fixed in prior commits. This design addresses the remaining 7.

Fixes

B1: Eval Action Selection — Softmax, Not Greedy

• evaluate_baseline.rs: Replace batch_greedy_actions() with batch_hierarchical_softmax_actions() using eval_softmax_temp from hyperopt params.
• dqn.rs: Add set_eval_mode() — resets noisy layer noise, uses mean weights (no noise).
• Eval binary calls set_eval_mode(true) before inference.
• Load _best checkpoint variant, not latest epoch.

B2: NormStats — Save From Training, Load In Eval

• evaluate_baseline.rs: Load norm_stats_fold{N}.json from checkpoint directory instead of computing fresh stats from test data.
• Fall back to test-data computation with warning if file not found.

B3: Portfolio State — Sync Within Eval Chunks

• evaluate_baseline.rs: After each process_bar_factored(), update portfolio features (equity, exposure, drawdown) for the next bar's state vector within the same chunk.

C1: Hierarchical Reward (Curiosity Separation)

• Store only extrinsic (trading) reward in replay buffer.
• TD loss trains on pure trading P&L — Q-values reflect actual profitability.
• Curiosity bonus applied as exploration term during action selection only: Q(s,a) + β * curiosity(s,a), with β annealing from curiosity_weight → 0.
• Files: trainer.rs

C2: Single Exploration Mechanism (Noisy Nets Only)

• Remove epsilon floor from training (set to 0.0). Noisy nets provide exploration.
• Remove count bonus from Q-value computation during action selection.
• Keep epsilon floor (0.02) for hyperopt eval backtest only.
• Keep count bonus module for diversity metrics (not Q-value modification).
• Files: dqn.rs, trainer.rs, hyperopt/adapters/dqn.rs

C3: Neutral Hold Reward

• Hold reward = 0.0 (no penalty, no bonus).
• Remove hold_penalty_weight from hyperopt search space (31D → 30D).
• Diversity penalty in objective already catches "always hold" degenerate trials.
• Files: reward.rs, dqn.rs, hyperopt/adapters/dqn.rs

C4: Smart Early Stopping on Backtest Sharpe

• Re-enable early_stopping_enabled in hyperopt adapter.
• Use backtest Sharpe (not val-loss) as stopping metric.
• Adaptive plateau window: max(epochs / 2, 3).
• Respect min_epochs_before_stopping from search space.
• On early stop: return best-epoch checkpoint with best-epoch metrics.
• Files: trainer.rs, hyperopt/adapters/dqn.rs

File Changes

File	Changes
evaluate_baseline.rs	B1, B2, B3
dqn.rs	B1 (eval mode), C2 (remove count bonus from Q), C3 (hold=0)
trainer.rs	C1 (extrinsic-only replay), C2 (epsilon=0), C4 (Sharpe stopping)
reward.rs	C3 (remove hold penalty)
hyperopt/adapters/dqn.rs	C3 (remove hold_penalty_weight dim), C4 (re-enable stopping)
count_bonus.rs	C2 (metrics only, not Q-value)

Success Criteria

1. evaluate_baseline results within 20% of hyperopt backtest on same data
2. Trade count < 1000 per 3-month fold
3. No -100% drawdown
4. TPE shows monotonic convergence
5. Early stopping triggers on ~30-50% of trials