From ce847fd0d48fcc66aa50f047733dc49312a60f74 Mon Sep 17 00:00:00 2001
From: jgrusewski <jgrusewski@localhost>
Date: Fri, 6 Mar 2026 09:59:54 +0100
Subject: [PATCH] =?UTF-8?q?docs:=20DQN=20hyperopt=20overhaul=20design=20?=
 =?UTF-8?q?=E2=80=94=207=20remaining=20root=20cause=20fixes?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Addresses eval/training mismatch (B1-B3), reward architecture (C1-C3),
and early stopping (C4). See design doc for full analysis.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 ...2026-03-06-dqn-hyperopt-overhaul-design.md | 81 +++++++++++++++++++
 1 file changed, 81 insertions(+)
 create mode 100644 docs/plans/2026-03-06-dqn-hyperopt-overhaul-design.md

diff --git a/docs/plans/2026-03-06-dqn-hyperopt-overhaul-design.md b/docs/plans/2026-03-06-dqn-hyperopt-overhaul-design.md
new file mode 100644
index 000000000..0a8535b0e
--- /dev/null
+++ b/docs/plans/2026-03-06-dqn-hyperopt-overhaul-design.md
@@ -0,0 +1,81 @@
+# DQN Hyperopt Overhaul Design
+
+## Problem
+
+Hyperopt produces configs with great objectives (e.g., -464) that fail catastrophically
+on out-of-sample evaluation (Sharpe=-284, MaxDD=100%, -100% return, 58K trades in 3 months).
+
+Root cause analysis identified 13 issues; 6 were fixed in prior commits.
+This design addresses the remaining 7.
+
+## Fixes
+
+### B1: Eval Action Selection — Softmax, Not Greedy
+
+- `evaluate_baseline.rs`: Replace `batch_greedy_actions()` with
+  `batch_hierarchical_softmax_actions()` using `eval_softmax_temp` from hyperopt params.
+- `dqn.rs`: Add `set_eval_mode()` — resets noisy layer noise, uses mean weights (no noise).
+- Eval binary calls `set_eval_mode(true)` before inference.
+- Load `_best` checkpoint variant, not latest epoch.
+
+### B2: NormStats — Save From Training, Load In Eval
+
+- `evaluate_baseline.rs`: Load `norm_stats_fold{N}.json` from checkpoint directory
+  instead of computing fresh stats from test data.
+- Fall back to test-data computation with warning if file not found.
+
+### B3: Portfolio State — Sync Within Eval Chunks
+
+- `evaluate_baseline.rs`: After each `process_bar_factored()`, update portfolio features
+  (equity, exposure, drawdown) for the next bar's state vector within the same chunk.
+
+### C1: Hierarchical Reward (Curiosity Separation)
+
+- Store only extrinsic (trading) reward in replay buffer.
+- TD loss trains on pure trading P&L — Q-values reflect actual profitability.
+- Curiosity bonus applied as exploration term during action selection only:
+  `Q(s,a) + β * curiosity(s,a)`, with β annealing from `curiosity_weight` → 0.
+- Files: `trainer.rs`
+
+### C2: Single Exploration Mechanism (Noisy Nets Only)
+
+- Remove epsilon floor from training (set to 0.0). Noisy nets provide exploration.
+- Remove count bonus from Q-value computation during action selection.
+- Keep epsilon floor (0.02) for hyperopt eval backtest only.
+- Keep count bonus module for diversity metrics (not Q-value modification).
+- Files: `dqn.rs`, `trainer.rs`, `hyperopt/adapters/dqn.rs`
+
+### C3: Neutral Hold Reward
+
+- Hold reward = 0.0 (no penalty, no bonus).
+- Remove `hold_penalty_weight` from hyperopt search space (31D → 30D).
+- Diversity penalty in objective already catches "always hold" degenerate trials.
+- Files: `reward.rs`, `dqn.rs`, `hyperopt/adapters/dqn.rs`
+
+### C4: Smart Early Stopping on Backtest Sharpe
+
+- Re-enable `early_stopping_enabled` in hyperopt adapter.
+- Use backtest Sharpe (not val-loss) as stopping metric.
+- Adaptive plateau window: `max(epochs / 2, 3)`.
+- Respect `min_epochs_before_stopping` from search space.
+- On early stop: return best-epoch checkpoint with best-epoch metrics.
+- Files: `trainer.rs`, `hyperopt/adapters/dqn.rs`
+
+## File Changes
+
+| File | Changes |
+|------|---------|
+| `evaluate_baseline.rs` | B1, B2, B3 |
+| `dqn.rs` | B1 (eval mode), C2 (remove count bonus from Q), C3 (hold=0) |
+| `trainer.rs` | C1 (extrinsic-only replay), C2 (epsilon=0), C4 (Sharpe stopping) |
+| `reward.rs` | C3 (remove hold penalty) |
+| `hyperopt/adapters/dqn.rs` | C3 (remove hold_penalty_weight dim), C4 (re-enable stopping) |
+| `count_bonus.rs` | C2 (metrics only, not Q-value) |
+
+## Success Criteria
+
+1. `evaluate_baseline` results within 20% of hyperopt backtest on same data
+2. Trade count < 1000 per 3-month fold
+3. No -100% drawdown
+4. TPE shows monotonic convergence
+5. Early stopping triggers on ~30-50% of trials