# DQN Hyperopt Overhaul Design

## Problem

Hyperopt produces configs with great objectives (e.g., -464) that fail catastrophically
on out-of-sample evaluation (Sharpe=-284, MaxDD=100%, -100% return, 58K trades in 3 months).

Root cause analysis identified 13 issues; 6 were fixed in prior commits.
This design addresses the remaining 7.

## Fixes

### B1: Eval Action Selection — Softmax, Not Greedy

- `evaluate_baseline.rs`: Replace `batch_greedy_actions()` with
  `batch_hierarchical_softmax_actions()` using `eval_softmax_temp` from hyperopt params.
- `dqn.rs`: Add `set_eval_mode()` — resets noisy layer noise, uses mean weights (no noise).
- Eval binary calls `set_eval_mode(true)` before inference.
- Load `_best` checkpoint variant, not latest epoch.

### B2: NormStats — Save From Training, Load In Eval

- `evaluate_baseline.rs`: Load `norm_stats_fold{N}.json` from checkpoint directory
  instead of computing fresh stats from test data.
- Fall back to test-data computation with warning if file not found.

### B3: Portfolio State — Sync Within Eval Chunks

- `evaluate_baseline.rs`: After each `process_bar_factored()`, update portfolio features
  (equity, exposure, drawdown) for the next bar's state vector within the same chunk.

### C1: Hierarchical Reward (Curiosity Separation)

- Store only extrinsic (trading) reward in replay buffer.
- TD loss trains on pure trading P&L — Q-values reflect actual profitability.
- Curiosity bonus applied as exploration term during action selection only:
  `Q(s,a) + β * curiosity(s,a)`, with β annealing from `curiosity_weight` → 0.
- Files: `trainer.rs`

### C2: Single Exploration Mechanism (Noisy Nets Only)

- Remove epsilon floor from training (set to 0.0). Noisy nets provide exploration.
- Remove count bonus from Q-value computation during action selection.
- Keep epsilon floor (0.02) for hyperopt eval backtest only.
- Keep count bonus module for diversity metrics (not Q-value modification).
- Files: `dqn.rs`, `trainer.rs`, `hyperopt/adapters/dqn.rs`

### C3: Neutral Hold Reward

- Hold reward = 0.0 (no penalty, no bonus).
- Remove `hold_penalty_weight` from hyperopt search space (31D → 30D).
- Diversity penalty in objective already catches "always hold" degenerate trials.
- Files: `reward.rs`, `dqn.rs`, `hyperopt/adapters/dqn.rs`

### C4: Smart Early Stopping on Backtest Sharpe

- Re-enable `early_stopping_enabled` in hyperopt adapter.
- Use backtest Sharpe (not val-loss) as stopping metric.
- Adaptive plateau window: `max(epochs / 2, 3)`.
- Respect `min_epochs_before_stopping` from search space.
- On early stop: return best-epoch checkpoint with best-epoch metrics.
- Files: `trainer.rs`, `hyperopt/adapters/dqn.rs`

## File Changes

| File | Changes |
|------|---------|
| `evaluate_baseline.rs` | B1, B2, B3 |
| `dqn.rs` | B1 (eval mode), C2 (remove count bonus from Q), C3 (hold=0) |
| `trainer.rs` | C1 (extrinsic-only replay), C2 (epsilon=0), C4 (Sharpe stopping) |
| `reward.rs` | C3 (remove hold penalty) |
| `hyperopt/adapters/dqn.rs` | C3 (remove hold_penalty_weight dim), C4 (re-enable stopping) |
| `count_bonus.rs` | C2 (metrics only, not Q-value) |

## Success Criteria

1. `evaluate_baseline` results within 20% of hyperopt backtest on same data
2. Trade count < 1000 per 3-month fold
3. No -100% drawdown
4. TPE shows monotonic convergence
5. Early stopping triggers on ~30-50% of trials
