From 7f5dfb70c645b506c8d4d035e992c34b6a2dab63 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 3 Mar 2026 20:40:46 +0100 Subject: [PATCH] docs: add DQN production hardening design 4-phase plan from deep audit: critical bug fixes (weight_decay, IQN PER, q_value_std, 54/51 dim mismatch), train/eval parity (TradeExecutor, checkpoint validation), SOTA improvements (differential Sharpe, PQN LayerNorm, temporal PER, dormant neuron resets), and search space cleanup. Co-Authored-By: Claude Opus 4.6 --- ...6-03-03-dqn-production-hardening-design.md | 226 ++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 docs/plans/2026-03-03-dqn-production-hardening-design.md diff --git a/docs/plans/2026-03-03-dqn-production-hardening-design.md b/docs/plans/2026-03-03-dqn-production-hardening-design.md new file mode 100644 index 000000000..e215be39c --- /dev/null +++ b/docs/plans/2026-03-03-dqn-production-hardening-design.md @@ -0,0 +1,226 @@ +# DQN Production Hardening Design + +**Goal:** Fix critical bugs, add SOTA improvements, achieve train/eval parity, and clean up dead code to make DQN production-ready for live futures trading. + +**Architecture:** Modular fixes to existing DQN pipeline. Each phase is independent and testable separately. No new crates — all changes within `crates/ml/src/dqn/`, `crates/ml/src/hyperopt/adapters/dqn.rs`, `crates/ml/src/trainers/dqn/`, and `crates/ml/examples/evaluate_baseline.rs`. + +**Tech Stack:** Candle v0.9.1 (Rust), existing DQN modules, 45 factored actions, 51+3 features. + +--- + +## Phase 1: Critical Bug Fixes + +Every hyperopt result prior to these fixes is unreliable. Fix these first. + +### 1.1 Wire weight_decay to Adam optimizer + +**Problem:** Hyperopt tunes `weight_decay` across `[1e-5, 1e-3]` but `dqn.rs` and `trainer.rs` construct Adam with `weight_decay: None` hardcoded. L2 regularization never applied. + +**Fix:** Pass `config.weight_decay` to `ParamsAdamW` in both construction sites. + +**Files:** `crates/ml/src/dqn/dqn.rs` (~line 500), `crates/ml/src/trainers/dqn/trainer.rs` (~line 1500) + +### 1.2 Wire IQN PER importance-sampling weights + +**Problem:** `let _ = weights_tensor;` on line ~1832 of `dqn.rs` drops PER IS correction for quantile Huber loss. Training is biased toward high-priority transitions. + +**Fix:** Multiply quantile Huber loss element-wise by `weights_tensor` before mean reduction, same as the C51 and standard loss paths. + +**Files:** `crates/ml/src/dqn/dqn.rs` (IQN loss computation, ~lines 1752-1833) + +### 1.3 Compute and propagate q_value_std + +**Problem:** Stability penalty in objective function (15% weight) uses `q_value_std` which is always 0.0 (TODO comment). Hyperopt cannot detect exploding Q-values. + +**Fix:** In the DQN trainer, compute `q_values.std()` periodically (every 100 steps) and store in `additional_metrics`. Propagate to hyperopt adapter's `DQNMetrics`. + +**Files:** `crates/ml/src/trainers/dqn/trainer.rs`, `crates/ml/src/hyperopt/adapters/dqn.rs` (~line 2748) + +### 1.4 Fix train/eval feature dimension mismatch (54 vs 51) + +**Problem:** Trainer uses `input_dim=54` (51 market + 3 portfolio features). Evaluator constructs network with `state_dim=51` and never appends portfolio features. Checkpoint load will fail on shape mismatch. + +**Fix:** Make evaluator append 3 portfolio features (position_size, unrealized_pnl, drawdown_pct) using position tracking. Requires wiring TradeExecutor (see Phase 2). + +**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/src/trainers/dqn/features.rs` + +### 1.5 Restore validate_for_hft_trendfollowing() + +**Problem:** All constraint checks commented out. Invalid hyperparameter combos reach training uncaught. + +**Fix:** Uncomment and update validation logic: v_min < v_max, num_atoms > 1, batch_size > 0, etc. + +**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs` + +### 1.6 Fix stress tester circular dependency + +**Problem:** `stress_tester` is always `None` due to circular dependency workaround comment. + +**Fix:** Use two-phase initialization — create DQNTrainer first, then set stress tester via a setter method. + +**Files:** `crates/ml/src/trainers/dqn/trainer.rs` (~lines 552-558) + +--- + +## Phase 2: Train/Eval Parity + +### 2.1 Wire TradeExecutor into evaluation + +**Problem:** Eval binary uses simplified `exposure * pct_change - flat_cost`. No position tracking, no cost differentiation, agent can flip +100% to -100% in one bar. + +**Fix:** Import and use `TradeExecutor` from `dqn/trade_executor.rs` in the evaluation loop. Track position state, compute position deltas, apply costs only on position changes. + +**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/src/dqn/trade_executor.rs` + +### 2.2 Order type and urgency cost differentiation + +**Problem:** 30 of 45 actions produce identical returns at eval (order type and urgency ignored). Agent's order type selection doesn't matter. + +**Fix:** Use `OrderType::transaction_cost()` and `Urgency::weight()` in eval cost model. + +**Files:** `crates/ml/examples/evaluate_baseline.rs` + +### 2.3 NormStats lookahead — hard error on missing stats + +**Problem:** Missing `norm_stats_fold{N}.json` falls back to computing from test data (lookahead bias), silently producing inflated metrics. + +**Fix:** Return error instead of fallback. Log the expected path clearly. + +**Files:** `crates/ml/examples/evaluate_baseline.rs` + +### 2.4 Checkpoint architecture validation + +**Problem:** No verification that loaded checkpoint matches network config. Shape mismatch fails at runtime or loads truncated weights. + +**Fix:** Embed `PPOConfig`/`DQNConfig` hash in safetensors metadata on save. Validate on load. + +**Files:** `crates/ml/src/dqn/dqn.rs` (save/load methods) + +### 2.5 Atomic checkpoint writes + +**Problem:** Crash mid-save corrupts checkpoint. + +**Fix:** Write to `{path}.tmp`, then `fs::rename()` (atomic on POSIX). + +**Files:** `crates/ml/src/dqn/dqn.rs` (checkpoint save) + +### 2.6 DQN CUDA sync fix + +**Problem:** DQN hyperopt adapter uses `thread::sleep(100ms)` for GPU sync. Not a real sync. + +**Fix:** Use tensor readback pattern (same as PPO adapter fix from previous work). + +**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs` + +### 2.7 Per-symbol annualization constant + +**Problem:** Hardcoded `252 * 1380` bars/year is wrong for 6E (23h/day) and ZN (7h pit session). + +**Fix:** Add `bars_per_year` to evaluation config, defaulting per symbol: ES/NQ=347760, 6E=345000, ZN=105840. + +**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/examples/baseline_common/mod.rs` + +--- + +## Phase 3: SOTA Algorithmic Improvements + +### 3.1 Differential Sharpe Ratio reward + +**Problem:** Current reward normalizer divides by EMA variance but doesn't compute the actual differential Sharpe. + +**Fix:** Implement Moody-Saffell DSR: `DSR_t = (B_{t-1}*r_t - 0.5*A_{t-1}*r_t^2) / (B_{t-1} - A_{t-1}^2)^{3/2}`. Add Calmar ratio blend option. Add blend `alpha` to hyperopt search space. + +**Ref:** Moody & Saffell, NeurIPS 1998; Risk-Aware RL Reward (2025) + +**Files:** `crates/ml/src/dqn/reward.rs` (new `DifferentialSharpe` struct) + +### 3.2 LayerNorm everywhere + optional target network removal + +**Problem:** LayerNorm exists in `rmsnorm.rs` but not used consistently. Target network adds memory overhead and stale-target bias. + +**Fix:** Add LayerNorm between all hidden layers in the Q-network. Add `use_target_network: bool` to `DQNConfig` (default true for backward compat). When false, use online network directly for target computation (PQN style). + +**Ref:** PQN (ICML 2024, arXiv 2407.04811), BTR (ICML 2025, arXiv 2411.03820) + +**Files:** `crates/ml/src/dqn/network.rs`, `crates/ml/src/dqn/dqn.rs`, `crates/ml/src/dqn/rainbow_config.rs` + +### 3.3 Temporal stratification in PER + +**Problem:** During regime shifts, PER samples mostly old-regime data. + +**Fix:** Force 20% of each batch from the most recent N timesteps regardless of priority. Add `temporal_fraction: f32` config (default 0.2). + +**Ref:** "Addressing Non-Stationarity in FX Trading" (ACM ICAIF 2022) + +**Files:** `crates/ml/src/dqn/prioritized_replay.rs` + +### 3.4 Masked softmax (fix action masking gradient leak) + +**Problem:** Hard masking zeros Q-values for invalid actions. When agent returns to normal position, stale values corrupt selection. + +**Fix:** Add `-1e9` to masked logit values before softmax instead of zeroing. Gradients flow through unmasked actions. + +**Files:** `crates/ml/src/dqn/action_space.rs` + +### 3.5 Dormant neuron resets (BTR) + +**Problem:** Financial markets are non-stationary. Dead neurons accumulate, plasticity loss kills live Sharpe. + +**Fix:** Track per-neuron activation magnitude. Every K steps, reset neurons with <1% activation frequency to Xavier init. Add `dormant_reset_interval: usize` config. + +**Ref:** BTR (ICML 2025), "Loss of Plasticity in Continual Deep RL" (ICML 2023) + +**Files:** New `crates/ml/src/dqn/dormant_neurons.rs`, wire into training loop + +### 3.6 Randomized transaction costs + +**Problem:** Fixed 15bps costs don't reflect real slippage variability. Agent over-trades in volatile sessions. + +**Fix:** Sample costs from `Normal(base_cost, base_cost * 0.3)` during training. Curriculum Phase 1 uses low-variance costs, Phase 2 uses realistic distribution. + +**Files:** `crates/ml/src/trainers/dqn/trainer.rs` (reward computation) + +--- + +## Phase 4: Search Space Cleanup + Dead Code + +### 4.1 Search space rationalization + +Remove dimensions for disabled features (ensemble_size, beta_variance, beta_disagreement, beta_entropy). Add: DSR blend alpha, LayerNorm toggle, temporal PER fraction, dormant reset interval. Remove weight_decay from search if wired as fixed value. + +**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs` + +### 4.2 Delete RainbowDQNAgent shell + +`rainbow_integration.rs` is a 2KB placeholder not connected to any path. + +**Files:** `crates/ml/src/dqn/rainbow_integration.rs`, `crates/ml/src/dqn/mod.rs` + +### 4.3 Fix or delete commented-out re-exports in dqn/mod.rs + +`noisy_exploration`, `performance_tests`, `performance_validation` are suppressed. + +**Files:** `crates/ml/src/dqn/mod.rs` + +### 4.4 Fix RegimeConditionalDQN::forward() regime routing + +Forward pass defaults to Trending because it receives `&Tensor` without regime signal. Training loss uses wrong regime head. + +**Files:** `crates/ml/src/dqn/regime_conditional.rs` + +### 4.5 Kelly fraction from C51 distribution + +Derive P(win) and E[win]/E[loss] from distributional output. Compute Kelly fraction as position sizing multiplier. + +**Files:** `crates/ml/src/dqn/dqn.rs` (action selection), `crates/risk/src/kelly.rs` + +--- + +## Implementation Order + +Phases 1-2 are prerequisites (correctness). Phase 3 is algorithmic improvement. Phase 4 is cleanup. + +- Phase 1 items are independent (parallel) +- Phase 2.1 must complete before 1.4 (TradeExecutor needed for portfolio features in eval) +- Phase 3 items are independent (parallel) +- Phase 4 depends on Phase 3 (search space depends on new features)