# DQN Production Hardening Design

**Goal:** Fix critical bugs, add SOTA improvements, achieve train/eval parity, and clean up dead code to make DQN production-ready for live futures trading.

**Architecture:** Modular fixes to existing DQN pipeline. Each phase is independent and testable separately. No new crates — all changes within `crates/ml/src/dqn/`, `crates/ml/src/hyperopt/adapters/dqn.rs`, `crates/ml/src/trainers/dqn/`, and `crates/ml/examples/evaluate_baseline.rs`.

**Tech Stack:** Candle v0.9.1 (Rust), existing DQN modules, 45 factored actions, 51+3 features.

---

## Phase 1: Critical Bug Fixes

Every hyperopt result prior to these fixes is unreliable. Fix these first.

### 1.1 Wire weight_decay to Adam optimizer

**Problem:** Hyperopt tunes `weight_decay` across `[1e-5, 1e-3]` but `dqn.rs` and `trainer.rs` construct Adam with `weight_decay: None` hardcoded. L2 regularization never applied.

**Fix:** Pass `config.weight_decay` to `ParamsAdamW` in both construction sites.

**Files:** `crates/ml/src/dqn/dqn.rs` (~line 500), `crates/ml/src/trainers/dqn/trainer.rs` (~line 1500)

### 1.2 Wire IQN PER importance-sampling weights

**Problem:** `let _ = weights_tensor;` on line ~1832 of `dqn.rs` drops PER IS correction for quantile Huber loss. Training is biased toward high-priority transitions.

**Fix:** Multiply quantile Huber loss element-wise by `weights_tensor` before mean reduction, same as the C51 and standard loss paths.

**Files:** `crates/ml/src/dqn/dqn.rs` (IQN loss computation, ~lines 1752-1833)

### 1.3 Compute and propagate q_value_std

**Problem:** Stability penalty in objective function (15% weight) uses `q_value_std` which is always 0.0 (TODO comment). Hyperopt cannot detect exploding Q-values.

**Fix:** In the DQN trainer, compute `q_values.std()` periodically (every 100 steps) and store in `additional_metrics`. Propagate to hyperopt adapter's `DQNMetrics`.

**Files:** `crates/ml/src/trainers/dqn/trainer.rs`, `crates/ml/src/hyperopt/adapters/dqn.rs` (~line 2748)

### 1.4 Fix train/eval feature dimension mismatch (54 vs 51)

**Problem:** Trainer uses `input_dim=54` (51 market + 3 portfolio features). Evaluator constructs network with `state_dim=51` and never appends portfolio features. Checkpoint load will fail on shape mismatch.

**Fix:** Make evaluator append 3 portfolio features (position_size, unrealized_pnl, drawdown_pct) using position tracking. Requires wiring TradeExecutor (see Phase 2).

**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/src/trainers/dqn/features.rs`

### 1.5 Restore validate_for_hft_trendfollowing()

**Problem:** All constraint checks commented out. Invalid hyperparameter combos reach training uncaught.

**Fix:** Uncomment and update validation logic: v_min < v_max, num_atoms > 1, batch_size > 0, etc.

**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs`

### 1.6 Fix stress tester circular dependency

**Problem:** `stress_tester` is always `None` due to circular dependency workaround comment.

**Fix:** Use two-phase initialization — create DQNTrainer first, then set stress tester via a setter method.

**Files:** `crates/ml/src/trainers/dqn/trainer.rs` (~lines 552-558)

---

## Phase 2: Train/Eval Parity

### 2.1 Wire TradeExecutor into evaluation

**Problem:** Eval binary uses simplified `exposure * pct_change - flat_cost`. No position tracking, no cost differentiation, agent can flip +100% to -100% in one bar.

**Fix:** Import and use `TradeExecutor` from `dqn/trade_executor.rs` in the evaluation loop. Track position state, compute position deltas, apply costs only on position changes.

**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/src/dqn/trade_executor.rs`

### 2.2 Order type and urgency cost differentiation

**Problem:** 30 of 45 actions produce identical returns at eval (order type and urgency ignored). Agent's order type selection doesn't matter.

**Fix:** Use `OrderType::transaction_cost()` and `Urgency::weight()` in eval cost model.

**Files:** `crates/ml/examples/evaluate_baseline.rs`

### 2.3 NormStats lookahead — hard error on missing stats

**Problem:** Missing `norm_stats_fold{N}.json` falls back to computing from test data (lookahead bias), silently producing inflated metrics.

**Fix:** Return error instead of fallback. Log the expected path clearly.

**Files:** `crates/ml/examples/evaluate_baseline.rs`

### 2.4 Checkpoint architecture validation

**Problem:** No verification that loaded checkpoint matches network config. Shape mismatch fails at runtime or loads truncated weights.

**Fix:** Embed `PPOConfig`/`DQNConfig` hash in safetensors metadata on save. Validate on load.

**Files:** `crates/ml/src/dqn/dqn.rs` (save/load methods)

### 2.5 Atomic checkpoint writes

**Problem:** Crash mid-save corrupts checkpoint.

**Fix:** Write to `{path}.tmp`, then `fs::rename()` (atomic on POSIX).

**Files:** `crates/ml/src/dqn/dqn.rs` (checkpoint save)

### 2.6 DQN CUDA sync fix

**Problem:** DQN hyperopt adapter uses `thread::sleep(100ms)` for GPU sync. Not a real sync.

**Fix:** Use tensor readback pattern (same as PPO adapter fix from previous work).

**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs`

### 2.7 Per-symbol annualization constant

**Problem:** Hardcoded `252 * 1380` bars/year is wrong for 6E (23h/day) and ZN (7h pit session).

**Fix:** Add `bars_per_year` to evaluation config, defaulting per symbol: ES/NQ=347760, 6E=345000, ZN=105840.

**Files:** `crates/ml/examples/evaluate_baseline.rs`, `crates/ml/examples/baseline_common/mod.rs`

---

## Phase 3: SOTA Algorithmic Improvements

### 3.1 Differential Sharpe Ratio reward

**Problem:** Current reward normalizer divides by EMA variance but doesn't compute the actual differential Sharpe.

**Fix:** Implement Moody-Saffell DSR: `DSR_t = (B_{t-1}*r_t - 0.5*A_{t-1}*r_t^2) / (B_{t-1} - A_{t-1}^2)^{3/2}`. Add Calmar ratio blend option. Add blend `alpha` to hyperopt search space.

**Ref:** Moody & Saffell, NeurIPS 1998; Risk-Aware RL Reward (2025)

**Files:** `crates/ml/src/dqn/reward.rs` (new `DifferentialSharpe` struct)

### 3.2 LayerNorm everywhere + optional target network removal

**Problem:** LayerNorm exists in `rmsnorm.rs` but not used consistently. Target network adds memory overhead and stale-target bias.

**Fix:** Add LayerNorm between all hidden layers in the Q-network. Add `use_target_network: bool` to `DQNConfig` (default true for backward compat). When false, use online network directly for target computation (PQN style).

**Ref:** PQN (ICML 2024, arXiv 2407.04811), BTR (ICML 2025, arXiv 2411.03820)

**Files:** `crates/ml/src/dqn/network.rs`, `crates/ml/src/dqn/dqn.rs`, `crates/ml/src/dqn/rainbow_config.rs`

### 3.3 Temporal stratification in PER

**Problem:** During regime shifts, PER samples mostly old-regime data.

**Fix:** Force 20% of each batch from the most recent N timesteps regardless of priority. Add `temporal_fraction: f32` config (default 0.2).

**Ref:** "Addressing Non-Stationarity in FX Trading" (ACM ICAIF 2022)

**Files:** `crates/ml/src/dqn/prioritized_replay.rs`

### 3.4 Masked softmax (fix action masking gradient leak)

**Problem:** Hard masking zeros Q-values for invalid actions. When agent returns to normal position, stale values corrupt selection.

**Fix:** Add `-1e9` to masked logit values before softmax instead of zeroing. Gradients flow through unmasked actions.

**Files:** `crates/ml/src/dqn/action_space.rs`

### 3.5 Dormant neuron resets (BTR)

**Problem:** Financial markets are non-stationary. Dead neurons accumulate, plasticity loss kills live Sharpe.

**Fix:** Track per-neuron activation magnitude. Every K steps, reset neurons with <1% activation frequency to Xavier init. Add `dormant_reset_interval: usize` config.

**Ref:** BTR (ICML 2025), "Loss of Plasticity in Continual Deep RL" (ICML 2023)

**Files:** New `crates/ml/src/dqn/dormant_neurons.rs`, wire into training loop

### 3.6 Randomized transaction costs

**Problem:** Fixed 15bps costs don't reflect real slippage variability. Agent over-trades in volatile sessions.

**Fix:** Sample costs from `Normal(base_cost, base_cost * 0.3)` during training. Curriculum Phase 1 uses low-variance costs, Phase 2 uses realistic distribution.

**Files:** `crates/ml/src/trainers/dqn/trainer.rs` (reward computation)

---

## Phase 4: Search Space Cleanup + Dead Code

### 4.1 Search space rationalization

Remove dimensions for disabled features (ensemble_size, beta_variance, beta_disagreement, beta_entropy). Add: DSR blend alpha, LayerNorm toggle, temporal PER fraction, dormant reset interval. Remove weight_decay from search if wired as fixed value.

**Files:** `crates/ml/src/hyperopt/adapters/dqn.rs`

### 4.2 Delete RainbowDQNAgent shell

`rainbow_integration.rs` is a 2KB placeholder not connected to any path.

**Files:** `crates/ml/src/dqn/rainbow_integration.rs`, `crates/ml/src/dqn/mod.rs`

### 4.3 Fix or delete commented-out re-exports in dqn/mod.rs

`noisy_exploration`, `performance_tests`, `performance_validation` are suppressed.

**Files:** `crates/ml/src/dqn/mod.rs`

### 4.4 Fix RegimeConditionalDQN::forward() regime routing

Forward pass defaults to Trending because it receives `&Tensor` without regime signal. Training loss uses wrong regime head.

**Files:** `crates/ml/src/dqn/regime_conditional.rs`

### 4.5 Kelly fraction from C51 distribution

Derive P(win) and E[win]/E[loss] from distributional output. Compute Kelly fraction as position sizing multiplier.

**Files:** `crates/ml/src/dqn/dqn.rs` (action selection), `crates/risk/src/kelly.rs`

---

## Implementation Order

Phases 1-2 are prerequisites (correctness). Phase 3 is algorithmic improvement. Phase 4 is cleanup.

- Phase 1 items are independent (parallel)
- Phase 2.1 must complete before 1.4 (TradeExecutor needed for portfolio features in eval)
- Phase 3 items are independent (parallel)
- Phase 4 depends on Phase 3 (search space depends on new features)
