docs: add DQN production hardening design
4-phase plan from deep audit: critical bug fixes (weight_decay, IQN PER, q_value_std, 54/51 dim mismatch), train/eval parity (TradeExecutor, checkpoint validation), SOTA improvements (differential Sharpe, PQN LayerNorm, temporal PER, dormant neuron resets), and search space cleanup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
226
docs/plans/2026-03-03-dqn-production-hardening-design.md
Normal file
226
docs/plans/2026-03-03-dqn-production-hardening-design.md
Normal file
@@ -0,0 +1,226 @@
|
||||
# DQN Production Hardening Design
|
||||
|
||||
**Goal:** Fix critical bugs, add SOTA improvements, achieve train/eval parity, and clean up dead code to make DQN production-ready for live futures trading.
|
||||
|
||||
**Architecture:** Modular fixes to existing DQN pipeline. Each phase is independent and testable separately. No new crates — all changes within `crates/ml/src/dqn/`, `crates/ml/src/hyperopt/adapters/dqn.rs`, `crates/ml/src/trainers/dqn/`, and `crates/ml/examples/evaluate_baseline.rs`.
|
||||
|
||||
**Tech Stack:** Candle v0.9.1 (Rust), existing DQN modules, 45 factored actions, 51+3 features.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Critical Bug Fixes
|
||||
|
||||
Every hyperopt result prior to these fixes is unreliable. Fix these first.
|
||||
|
||||
### 1.1 Wire weight_decay to Adam optimizer
|
||||
|
||||
**Problem:** Hyperopt tunes `weight_decay` across `[1e-5, 1e-3]` but `dqn.rs` and `trainer.rs` construct Adam with `weight_decay: None` hardcoded. L2 regularization never applied.
|
||||
|
||||
**Fix:** Pass `config.weight_decay` to `ParamsAdamW` in both construction sites.
|
||||
|
||||
**Files:** `crates/ml/src/dqn/dqn.rs` (~line 500), `crates/ml/src/trainers/dqn/trainer.rs` (~line 1500)
|
||||
|
||||
### 1.2 Wire IQN PER importance-sampling weights
|
||||
|
||||
**Problem:** `let _ = weights_tensor;` on line ~1832 of `dqn.rs` drops PER IS correction for quantile Huber loss. Training is biased toward high-priority transitions.
|
||||
|
||||
**Fix:** Multiply quantile Huber loss element-wise by `weights_tensor` before mean reduction, same as the C51 and standard loss paths.
|
||||
|
||||
**Files:** `crates/ml/src/dqn/dqn.rs` (IQN loss computation, ~lines 1752-1833)
|
||||
|
||||
### 1.3 Compute and propagate q_value_std
|
||||
|
||||
**Problem:** Stability penalty in objective function (15% weight) uses `q_value_std` which is always 0.0 (TODO comment). Hyperopt cannot detect exploding Q-values.
|
||||
|
||||
**Fix:** In the DQN trainer, compute `q_values.std()` periodically (every 100 steps) and store in `additional_metrics`. Propagate to hyperopt adapter's `DQNMetrics`.
|
||||
|
||||
**Files:** `crates/ml/src/trainers/dqn/trainer.rs`, `crates/ml/src/hyperopt/adapters/dqn.rs` (~line 2748)
|
||||
|
||||
### 1.4 Fix train/eval feature dimension mismatch (54 vs 51)
|
||||
|
||||
**Problem:** Trainer uses `input_dim=54` (51 market + 3 portfolio features). Evaluator constructs network with `state_dim=51` and never appends portfolio features. Checkpoint load will fail on shape mismatch.
|
||||
|
||||
**Fix:** Make evaluator append 3 portfolio features (position_size, unrealized_pnl, drawdown_pct) using position tracking. Requires wiring TradeExecutor (see Phase 2).
|
||||
|
||||
**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/src/trainers/dqn/features.rs`
|
||||
|
||||
### 1.5 Restore validate_for_hft_trendfollowing()
|
||||
|
||||
**Problem:** All constraint checks commented out. Invalid hyperparameter combos reach training uncaught.
|
||||
|
||||
**Fix:** Uncomment and update validation logic: v_min < v_max, num_atoms > 1, batch_size > 0, etc.
|
||||
|
||||
**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs`
|
||||
|
||||
### 1.6 Fix stress tester circular dependency
|
||||
|
||||
**Problem:** `stress_tester` is always `None` due to circular dependency workaround comment.
|
||||
|
||||
**Fix:** Use two-phase initialization — create DQNTrainer first, then set stress tester via a setter method.
|
||||
|
||||
**Files:** `crates/ml/src/trainers/dqn/trainer.rs` (~lines 552-558)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Train/Eval Parity
|
||||
|
||||
### 2.1 Wire TradeExecutor into evaluation
|
||||
|
||||
**Problem:** Eval binary uses simplified `exposure * pct_change - flat_cost`. No position tracking, no cost differentiation, agent can flip +100% to -100% in one bar.
|
||||
|
||||
**Fix:** Import and use `TradeExecutor` from `dqn/trade_executor.rs` in the evaluation loop. Track position state, compute position deltas, apply costs only on position changes.
|
||||
|
||||
**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/src/dqn/trade_executor.rs`
|
||||
|
||||
### 2.2 Order type and urgency cost differentiation
|
||||
|
||||
**Problem:** 30 of 45 actions produce identical returns at eval (order type and urgency ignored). Agent's order type selection doesn't matter.
|
||||
|
||||
**Fix:** Use `OrderType::transaction_cost()` and `Urgency::weight()` in eval cost model.
|
||||
|
||||
**Files:** `crates/ml/examples/evaluate_baseline.rs`
|
||||
|
||||
### 2.3 NormStats lookahead — hard error on missing stats
|
||||
|
||||
**Problem:** Missing `norm_stats_fold{N}.json` falls back to computing from test data (lookahead bias), silently producing inflated metrics.
|
||||
|
||||
**Fix:** Return error instead of fallback. Log the expected path clearly.
|
||||
|
||||
**Files:** `crates/ml/examples/evaluate_baseline.rs`
|
||||
|
||||
### 2.4 Checkpoint architecture validation
|
||||
|
||||
**Problem:** No verification that loaded checkpoint matches network config. Shape mismatch fails at runtime or loads truncated weights.
|
||||
|
||||
**Fix:** Embed `PPOConfig`/`DQNConfig` hash in safetensors metadata on save. Validate on load.
|
||||
|
||||
**Files:** `crates/ml/src/dqn/dqn.rs` (save/load methods)
|
||||
|
||||
### 2.5 Atomic checkpoint writes
|
||||
|
||||
**Problem:** Crash mid-save corrupts checkpoint.
|
||||
|
||||
**Fix:** Write to `{path}.tmp`, then `fs::rename()` (atomic on POSIX).
|
||||
|
||||
**Files:** `crates/ml/src/dqn/dqn.rs` (checkpoint save)
|
||||
|
||||
### 2.6 DQN CUDA sync fix
|
||||
|
||||
**Problem:** DQN hyperopt adapter uses `thread::sleep(100ms)` for GPU sync. Not a real sync.
|
||||
|
||||
**Fix:** Use tensor readback pattern (same as PPO adapter fix from previous work).
|
||||
|
||||
**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs`
|
||||
|
||||
### 2.7 Per-symbol annualization constant
|
||||
|
||||
**Problem:** Hardcoded `252 * 1380` bars/year is wrong for 6E (23h/day) and ZN (7h pit session).
|
||||
|
||||
**Fix:** Add `bars_per_year` to evaluation config, defaulting per symbol: ES/NQ=347760, 6E=345000, ZN=105840.
|
||||
|
||||
**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/examples/baseline_common/mod.rs`
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: SOTA Algorithmic Improvements
|
||||
|
||||
### 3.1 Differential Sharpe Ratio reward
|
||||
|
||||
**Problem:** Current reward normalizer divides by EMA variance but doesn't compute the actual differential Sharpe.
|
||||
|
||||
**Fix:** Implement Moody-Saffell DSR: `DSR_t = (B_{t-1}*r_t - 0.5*A_{t-1}*r_t^2) / (B_{t-1} - A_{t-1}^2)^{3/2}`. Add Calmar ratio blend option. Add blend `alpha` to hyperopt search space.
|
||||
|
||||
**Ref:** Moody & Saffell, NeurIPS 1998; Risk-Aware RL Reward (2025)
|
||||
|
||||
**Files:** `crates/ml/src/dqn/reward.rs` (new `DifferentialSharpe` struct)
|
||||
|
||||
### 3.2 LayerNorm everywhere + optional target network removal
|
||||
|
||||
**Problem:** LayerNorm exists in `rmsnorm.rs` but not used consistently. Target network adds memory overhead and stale-target bias.
|
||||
|
||||
**Fix:** Add LayerNorm between all hidden layers in the Q-network. Add `use_target_network: bool` to `DQNConfig` (default true for backward compat). When false, use online network directly for target computation (PQN style).
|
||||
|
||||
**Ref:** PQN (ICML 2024, arXiv 2407.04811), BTR (ICML 2025, arXiv 2411.03820)
|
||||
|
||||
**Files:** `crates/ml/src/dqn/network.rs`, `crates/ml/src/dqn/dqn.rs`, `crates/ml/src/dqn/rainbow_config.rs`
|
||||
|
||||
### 3.3 Temporal stratification in PER
|
||||
|
||||
**Problem:** During regime shifts, PER samples mostly old-regime data.
|
||||
|
||||
**Fix:** Force 20% of each batch from the most recent N timesteps regardless of priority. Add `temporal_fraction: f32` config (default 0.2).
|
||||
|
||||
**Ref:** "Addressing Non-Stationarity in FX Trading" (ACM ICAIF 2022)
|
||||
|
||||
**Files:** `crates/ml/src/dqn/prioritized_replay.rs`
|
||||
|
||||
### 3.4 Masked softmax (fix action masking gradient leak)
|
||||
|
||||
**Problem:** Hard masking zeros Q-values for invalid actions. When agent returns to normal position, stale values corrupt selection.
|
||||
|
||||
**Fix:** Add `-1e9` to masked logit values before softmax instead of zeroing. Gradients flow through unmasked actions.
|
||||
|
||||
**Files:** `crates/ml/src/dqn/action_space.rs`
|
||||
|
||||
### 3.5 Dormant neuron resets (BTR)
|
||||
|
||||
**Problem:** Financial markets are non-stationary. Dead neurons accumulate, plasticity loss kills live Sharpe.
|
||||
|
||||
**Fix:** Track per-neuron activation magnitude. Every K steps, reset neurons with <1% activation frequency to Xavier init. Add `dormant_reset_interval: usize` config.
|
||||
|
||||
**Ref:** BTR (ICML 2025), "Loss of Plasticity in Continual Deep RL" (ICML 2023)
|
||||
|
||||
**Files:** New `crates/ml/src/dqn/dormant_neurons.rs`, wire into training loop
|
||||
|
||||
### 3.6 Randomized transaction costs
|
||||
|
||||
**Problem:** Fixed 15bps costs don't reflect real slippage variability. Agent over-trades in volatile sessions.
|
||||
|
||||
**Fix:** Sample costs from `Normal(base_cost, base_cost * 0.3)` during training. Curriculum Phase 1 uses low-variance costs, Phase 2 uses realistic distribution.
|
||||
|
||||
**Files:** `crates/ml/src/trainers/dqn/trainer.rs` (reward computation)
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Search Space Cleanup + Dead Code
|
||||
|
||||
### 4.1 Search space rationalization
|
||||
|
||||
Remove dimensions for disabled features (ensemble_size, beta_variance, beta_disagreement, beta_entropy). Add: DSR blend alpha, LayerNorm toggle, temporal PER fraction, dormant reset interval. Remove weight_decay from search if wired as fixed value.
|
||||
|
||||
**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs`
|
||||
|
||||
### 4.2 Delete RainbowDQNAgent shell
|
||||
|
||||
`rainbow_integration.rs` is a 2KB placeholder not connected to any path.
|
||||
|
||||
**Files:** `crates/ml/src/dqn/rainbow_integration.rs`, `crates/ml/src/dqn/mod.rs`
|
||||
|
||||
### 4.3 Fix or delete commented-out re-exports in dqn/mod.rs
|
||||
|
||||
`noisy_exploration`, `performance_tests`, `performance_validation` are suppressed.
|
||||
|
||||
**Files:** `crates/ml/src/dqn/mod.rs`
|
||||
|
||||
### 4.4 Fix RegimeConditionalDQN::forward() regime routing
|
||||
|
||||
Forward pass defaults to Trending because it receives `&Tensor` without regime signal. Training loss uses wrong regime head.
|
||||
|
||||
**Files:** `crates/ml/src/dqn/regime_conditional.rs`
|
||||
|
||||
### 4.5 Kelly fraction from C51 distribution
|
||||
|
||||
Derive P(win) and E[win]/E[loss] from distributional output. Compute Kelly fraction as position sizing multiplier.
|
||||
|
||||
**Files:** `crates/ml/src/dqn/dqn.rs` (action selection), `crates/risk/src/kelly.rs`
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
Phases 1-2 are prerequisites (correctness). Phase 3 is algorithmic improvement. Phase 4 is cleanup.
|
||||
|
||||
- Phase 1 items are independent (parallel)
|
||||
- Phase 2.1 must complete before 1.4 (TradeExecutor needed for portfolio features in eval)
|
||||
- Phase 3 items are independent (parallel)
|
||||
- Phase 4 depends on Phase 3 (search space depends on new features)
|
||||
Reference in New Issue
Block a user