# DQN Hyperopt Overhaul v2 — Full Root Cause Fix

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Fix all 19 root causes preventing DQN hyperopt from achieving positive Sharpe ratio.

**Architecture:** Replace double-normalized reward pipeline with Differential Sharpe Ratio (DSR), fix broken IQN training path, add RMSNorm to distributional-dueling network, remove dead features, tighten hyperopt search space from 26D to ~22D.

**Tech Stack:** Rust, Candle ML framework, PSO hyperopt, OHLCV futures data

---

## Phase 1: Reward Function (7 tasks)

### Task 1.1: Implement Differential Sharpe Ratio struct

**Files:**
- Modify: `crates/ml/src/dqn/reward.rs:170-199` (RewardConfig)
- Modify: `crates/ml/src/dqn/reward.rs:399-410` (RewardFunction struct)

**Step 1: Add DSR struct and config fields**

Add before `RewardConfig` (around line 165):

```rust
/// Differential Sharpe Ratio (Moody & Saffell, 2001)
///
/// Computes the incremental change in Sharpe ratio per step.
/// Replaces both EMA reward normalizer and risk-adjusted division.
/// Produces dense, well-scaled, risk-adjusted reward per step.
pub struct DifferentialSharpeRatio {
    ema_return: f64,      // A_t: EMA of returns
    ema_return_sq: f64,   // B_t: EMA of squared returns
    eta: f64,             // Decay rate (0.01 = ~100-step window)
    initialized: bool,
}

impl DifferentialSharpeRatio {
    pub fn new(eta: f64) -> Self {
        Self {
            ema_return: 0.0,
            ema_return_sq: 0.0,
            eta: eta.clamp(0.0001, 0.1),
            initialized: false,
        }
    }

    /// Compute differential Sharpe ratio for a new return observation.
    ///
    /// Formula (Moody & Saffell 2001):
    ///   DSR_t = (B_{t-1} * r_t - 0.5 * A_{t-1} * (r_t^2 - B_{t-1})) / (B_{t-1} - A_{t-1}^2)^{3/2}
    ///
    /// Returns 0.0 on first call (no prior statistics).
    pub fn step(&mut self, r_t: f64) -> f64 {
        if !self.initialized {
            self.ema_return = r_t;
            self.ema_return_sq = r_t * r_t + 1e-8; // Small epsilon for numerical stability
            self.initialized = true;
            return 0.0;
        }

        let a = self.ema_return;      // A_{t-1}
        let b = self.ema_return_sq;   // B_{t-1}
        let r_sq = r_t * r_t;

        // Variance of returns: Var = E[r^2] - E[r]^2
        let variance = b - a * a;

        let dsr = if variance > 1e-12 {
            let numer = b * r_t - 0.5 * a * (r_sq - b);
            let denom = variance.powf(1.5);
            (numer / denom).clamp(-5.0, 5.0) // Bound extreme DSR values
        } else {
            // Insufficient variance — return scaled raw return as fallback
            r_t * 100.0 // Scale raw PnL (~0.0002) to ~0.02 range
        };

        // Update EMAs AFTER computing DSR (use previous stats for current step)
        self.ema_return = self.eta * r_t + (1.0 - self.eta) * a;
        self.ema_return_sq = self.eta * r_sq + (1.0 - self.eta) * b;

        dsr
    }

    /// Reset state between episodes/epochs for stationarity.
    pub fn reset(&mut self) {
        self.ema_return = 0.0;
        self.ema_return_sq = 0.0;
        self.initialized = false;
    }
}
```

**Step 2: Add config fields to RewardConfig (lines 170-199)**

Add two fields after `sharpe_window` (line 198):

```rust
    /// Use Differential Sharpe Ratio instead of EMA normalizer + risk-adjusted division
    pub use_dsr: bool,
    /// DSR EMA decay rate (0.001=slow/1000-step, 0.05=fast/20-step). Tunable in hyperopt.
    pub dsr_eta: f64,
```

**Step 3: Update RewardConfig::default() (lines 201-220)**

Add to default impl after `sharpe_window: 20`:

```rust
            use_dsr: false,     // Backward compat: off by default
            dsr_eta: 0.01,      // ~100-step effective window
```

**Step 4: Add DSR field to RewardFunction struct (lines 399-410)**

Add field:

```rust
    dsr: Option<DifferentialSharpeRatio>,
```

**Step 5: Initialize DSR in RewardFunction constructor (line 433)**

After normalizer initialization:

```rust
        let dsr = config.use_dsr.then(|| DifferentialSharpeRatio::new(config.dsr_eta));
```

**Step 6: Add DSR to RewardConfigBuilder**

Add builder method (after `enable_normalization` builder at line 294):

```rust
    pub fn use_dsr(mut self, enable: bool) -> Self {
        self.use_dsr = Some(enable);
        self
    }

    pub fn dsr_eta(mut self, eta: f64) -> Self {
        self.dsr_eta = Some(eta);
        self
    }
```

And add fields to builder struct + build():

```rust
    use_dsr: Option<bool>,
    dsr_eta: Option<f64>,
```

In `build()`:

```rust
            use_dsr: self.use_dsr.unwrap_or(false),
            dsr_eta: self.dsr_eta.unwrap_or(0.01),
```

**Step 7: Add `reset_dsr()` method to RewardFunction**

```rust
    pub fn reset_dsr(&mut self) {
        if let Some(ref mut dsr) = self.dsr {
            dsr.reset();
        }
    }
```

**Step 8: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib -- reward 2>&1 | tail -5
```

Expected: All existing reward tests pass.

**Step 9: Add DSR unit tests**

Add at bottom of reward.rs tests module:

```rust
    #[test]
    fn test_dsr_basic() {
        let mut dsr = DifferentialSharpeRatio::new(0.01);
        // First step returns 0 (no prior stats)
        assert_eq!(dsr.step(0.001), 0.0);
        // Positive return after positive → positive DSR
        let d1 = dsr.step(0.002);
        assert!(d1 > 0.0, "Positive return should give positive DSR: {}", d1);
        // Negative return → negative DSR
        let d2 = dsr.step(-0.003);
        assert!(d2 < 0.0, "Negative return should give negative DSR: {}", d2);
    }

    #[test]
    fn test_dsr_bounded() {
        let mut dsr = DifferentialSharpeRatio::new(0.01);
        dsr.step(0.001);
        // Even extreme returns should be clamped
        let d = dsr.step(100.0);
        assert!(d <= 5.0 && d >= -5.0, "DSR should be clamped: {}", d);
    }

    #[test]
    fn test_dsr_reset() {
        let mut dsr = DifferentialSharpeRatio::new(0.01);
        dsr.step(0.001);
        dsr.step(0.002);
        dsr.reset();
        // After reset, first step returns 0 again
        assert_eq!(dsr.step(0.001), 0.0);
    }

    #[test]
    fn test_dsr_config_roundtrip() {
        let config = RewardConfig::builder()
            .use_dsr(true)
            .dsr_eta(0.02)
            .build();
        assert!(config.use_dsr);
        assert!((config.dsr_eta - 0.02).abs() < 1e-10);
    }
```

**Step 10: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib -- dsr 2>&1 | tail -10
```

Expected: 4 new tests pass.

**Step 11: Commit**

```bash
git add crates/ml/src/dqn/reward.rs
git commit -m "feat(ml): add Differential Sharpe Ratio (DSR) struct and config"
```

---

### Task 1.2: Wire DSR into calculate_reward()

**Files:**
- Modify: `crates/ml/src/dqn/reward.rs:485-615` (calculate_reward method)

**Step 1: Modify calculate_reward to use DSR when enabled**

Replace the section from line 524 (diversity bonus) through line 603 (normalizer) with:

```rust
        // Calculate diversity bonus (in-training regularization)
        let entropy = calculate_entropy(recent_actions);
        let entropy_threshold = Decimal::try_from(0.3).unwrap_or(Decimal::ZERO);
        let diversity_bonus = if entropy < entropy_threshold {
            // DSR mode: scale penalty proportional to trading signal (~0.0005)
            // Legacy mode: -0.1 (500x larger — known to dominate reward signal)
            if self.config.use_dsr {
                Decimal::try_from(-0.0005).unwrap_or(Decimal::ZERO)
            } else {
                self.config.diversity_weight
            }
        } else {
            Decimal::ZERO
        };

        let pnl_base_reward = base_reward + diversity_bonus;

        let final_reward_f64: f64 = if self.config.use_dsr {
            // DSR path: skip Sharpe blend and EMA normalizer
            // Pass raw PnL + diversity through DSR transform
            let raw_pnl: f64 = pnl_base_reward.try_into()
                .map_err(|e| MLError::InvalidInput(format!("Reward conversion failed: {}", e)))?;

            if let Some(ref mut dsr) = self.dsr {
                dsr.step(raw_pnl)
            } else {
                // Fallback: scale raw PnL to reasonable range
                raw_pnl * 100.0
            }
        } else {
            // Legacy path: Sharpe blend + optional EMA normalization (unchanged)
            let pnl_normalized_f64: f64 = ((next_state.portfolio_features.get(0).unwrap_or(&100000.0)
                - current_state.portfolio_features.get(0).unwrap_or(&100000.0))
                / current_state.portfolio_features.get(0).unwrap_or(&100000.0)) as f64;

            self.returns_buffer.push_back(pnl_normalized_f64);
            if self.returns_buffer.len() > self.config.sharpe_window {
                self.returns_buffer.pop_front();
            }

            let sharpe_component = if self.returns_buffer.len() > 1 {
                let mean: f64 = self.returns_buffer.iter().sum::<f64>() / self.returns_buffer.len() as f64;
                let variance: f64 = self.returns_buffer.iter()
                    .map(|r| (r - mean).powi(2))
                    .sum::<f64>() / self.returns_buffer.len() as f64;
                let std_dev = variance.sqrt();
                if std_dev > 1e-8 { mean / std_dev } else { 0.0 }
            } else {
                0.0
            };
            let sharpe_decimal = Decimal::try_from(sharpe_component).unwrap_or(Decimal::ZERO);

            let sharpe_weight = self.config.sharpe_weight;
            let pnl_weight_adjusted = Decimal::ONE - sharpe_weight;
            let final_reward = pnl_base_reward * pnl_weight_adjusted + sharpe_decimal * sharpe_weight;

            let final_reward_f64: f64 = final_reward.try_into()
                .map_err(|e| MLError::InvalidInput(format!("Reward conversion failed: {}", e)))?;

            // Apply EMA normalization if enabled
            if let Some(normalizer) = &mut self.normalizer {
                let norm = normalizer.normalize(final_reward_f64);
                normalizer.update(final_reward_f64);
                norm.clamp(-3.0, 3.0)
            } else {
                final_reward_f64
            }
        };
```

**Step 2: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib -- reward 2>&1 | tail -5
```

**Step 3: Commit**

```bash
git add crates/ml/src/dqn/reward.rs
git commit -m "feat(ml): wire DSR into calculate_reward, scale diversity penalty"
```

---

### Task 1.3: Disable risk-adjusted division when DSR active

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:2234` (risk-adjusted call)
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:508` (RewardConfig construction)

**Step 1: Update RewardConfig construction in trainer (line 508)**

Change the RewardConfig construction to enable DSR when hyperparams say so. Add `use_dsr` and `dsr_eta` fields to `DQNHyperparameters` first.

In `crates/ml/src/trainers/dqn/config.rs`, add to `DQNHyperparameters`:

```rust
    pub use_dsr: bool,
    pub dsr_eta: f64,
```

And in its default:

```rust
            use_dsr: false,
            dsr_eta: 0.01,
```

Then in trainer.rs around line 508, update the RewardConfig construction:

```rust
            use_dsr: self.hyperparams.use_dsr,
            dsr_eta: self.hyperparams.dsr_eta,
```

**Step 2: Skip risk-adjusted reward when DSR active (line 2234)**

Replace line 2234:

```rust
                    // When DSR is active, it already handles risk-adjustment.
                    // Legacy path: divide by PnL volatility (Sharpe-like normalization)
                    let risk_adjusted_reward = if self.hyperparams.use_dsr {
                        barrier_scaled_reward // DSR output is already risk-adjusted
                    } else {
                        self.calculate_risk_adjusted_reward(barrier_scaled_reward)
                    };
```

**Step 3: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5
```

**Step 4: Commit**

```bash
git add crates/ml/src/trainers/dqn/config.rs crates/ml/src/trainers/dqn/trainer.rs
git commit -m "feat(ml): skip risk-adjusted division when DSR active"
```

---

### Task 1.4: Increase experience reward precision

**Files:**
- Modify: `crates/ml/src/dqn/experience.rs:31,43`

**Step 1: Change scale factor**

At line 31 (or wherever `Experience::new` stores reward):

```rust
// Old: reward: (reward * 10000.0) as i32,
// New: 100x more precision — DSR values ~±2.0 map to ±2_000_000 (fits i32)
reward: (reward * 1_000_000.0) as i32,
```

At line 43 (or `reward_f32()` method):

```rust
// Old: self.reward as f32 / 10000.0
self.reward as f32 / 1_000_000.0
```

**Step 2: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib -- experience 2>&1 | tail -5
```

**Step 3: Commit**

```bash
git add crates/ml/src/dqn/experience.rs
git commit -m "fix(ml): increase experience reward precision from 10000 to 1000000"
```

---

### Task 1.5: Reset portfolio tracker between epochs

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:1511-1546`

**Step 1: Add portfolio reset at epoch boundary**

Around line 1503 (where `self.pnl_history.clear()` already exists), add:

```rust
            self.pnl_history.clear();

            // Reset portfolio tracker to initial capital for reward stationarity.
            // Without this, portfolio compounding creates non-stationary rewards
            // (same action produces 5x different reward depending on epoch),
            // corrupting replay buffer and preventing Q-value convergence.
            self.portfolio_tracker.reset();
```

Also add DSR reset at the same location:

```rust
            // Reset DSR EMA state for clean epoch
            self.reward_fn.reset_dsr();
```

And reset the returns_buffer in the reward function. Add a method:

In `reward.rs`, add to `RewardFunction`:

```rust
    /// Reset per-episode/epoch state (returns buffer + DSR)
    pub fn reset_epoch_state(&mut self) {
        self.returns_buffer.clear();
        self.reset_dsr();
        // Reset normalizer running stats to prevent drift
        if let Some(ref mut normalizer) = self.normalizer {
            *normalizer = RewardNormalizer::new();
        }
    }
```

Then in trainer.rs at epoch boundary, call `self.reward_fn.reset_epoch_state()` instead of just `reset_dsr()`.

**Step 2: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5
```

**Step 3: Commit**

```bash
git add crates/ml/src/trainers/dqn/trainer.rs crates/ml/src/dqn/reward.rs
git commit -m "fix(ml): reset portfolio, DSR, and reward stats between epochs"
```

---

### Task 1.6: Reset DSR at episode boundaries

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:2080,2342`

At barrier exit (line 2080) and time/data boundary (line 2342), add:

```rust
            self.reward_fn.reset_dsr();
```

This ensures DSR doesn't carry state across independent trading episodes.

**Step: Commit**

```bash
git add crates/ml/src/trainers/dqn/trainer.rs
git commit -m "fix(ml): reset DSR at episode boundaries (barrier exit, time boundary)"
```

---

### Task 1.7: Phase 1 integration test

**Step 1: Run full ml test suite**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10
```

Expected: 2742+ tests pass, 0 failures.

**Step 2: Verify compile with check**

```bash
SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5
```

Expected: 0 errors, 0 warnings.

---

## Phase 2: Training Steps & Exploration (5 tasks)

### Task 2.1: Wire noisy_sigma_init to NoisyLinear

**Files:**
- Modify: `crates/ml/src/dqn/noisy_layers.rs:59-80`
- Modify: `crates/ml/src/dqn/dqn.rs` (pass sigma_init to NoisyLinear)

**Step 1: Add sigma_init parameter to NoisyLinear::new()**

Change `NoisyLinear::new()` signature to accept `sigma_init: f64`:

At line 80, replace:

```rust
// Old:
let sigma_init = 0.5 / (in_features as f64).sqrt();
// New: use caller-provided sigma, fall back to Fortunato et al. default
```

Update the constructor to accept and use the parameter. Also update all call sites in `dqn.rs` where `NoisyLinear::new()` is called, passing `self.config.noisy_sigma_init`.

**Step 2: Add epsilon floor back for NoisyNets**

In `crates/ml/src/dqn/dqn.rs` around line 1460-1461, change:

```rust
// Old: self.config.noisy_epsilon_floor  (was 0.0)
// New: small epsilon floor ensures exploration in early training
// when NoisyNet sigma hasn't annealed enough
let effective_epsilon = if self.config.use_noisy_nets {
    self.config.noisy_epsilon_floor.max(0.02) // Minimum 2% random
} else {
    self.epsilon
};
```

**Step 3: Commit**

```bash
git add crates/ml/src/dqn/noisy_layers.rs crates/ml/src/dqn/dqn.rs
git commit -m "fix(ml): wire noisy_sigma_init to NoisyLinear, add 2% epsilon floor"
```

---

### Task 2.2: Increase epochs, cap batch_size, fix min_replay_size

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:463,504,2310`

**Step 1: Change hyperopt bounds**

Line 463 — batch_size:

```rust
// Old: (64.0, 4096.0)
(64.0, 512.0),                  // 1: batch_size (capped: 4096 → too few gradient steps)
```

Line 504 — min_epochs_before_stopping:

```rust
// Remove from search space entirely (early stopping disabled in hyperopt)
// This dimension will be removed in Task 2.3
```

**Step 2: Change epochs parameter**

Find where the hyperopt adapter constructs the trainer (look for `epochs` parameter passed to `DqnHyperoptAdapter::new()` or similar). Change from 8 to 30.

This is passed by the caller — likely in the hyperopt binary or workflow template. Find with:

```bash
grep -rn "DqnHyperoptAdapter::new\|epochs.*=.*8" crates/ml/src/hyperopt/
```

Update the caller to pass `epochs: 30`.

**Step 3: Fix min_replay_size**

Around line 2310 where `min_replay_size` is set:

```rust
// Old: min_replay_size: params.batch_size * 2,
min_replay_size: (params.batch_size * 50).min(params.buffer_size / 2),
```

**Step 4: Fix PER beta annealing**

Find where `per_beta_annealing_steps` or similar is set. Replace hardcoded `epochs * 1000` with:

```rust
// Old: per_beta_annealing_steps: epochs * 1000,
// New: match actual gradient steps
per_beta_annealing_steps: (training_data_len / params.batch_size) * epochs,
```

If `training_data_len` isn't available at this point, use a conservative estimate:

```rust
per_beta_annealing_steps: epochs * 2000, // ~2000 steps/epoch at batch_size=64 with 130k bars
```

**Step 5: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): cap batch_size=512, epochs=30, min_replay=50x, fix PER annealing"
```

---

### Task 2.3: Remove wasted hyperopt dimensions

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:460-506` (continuous_bounds)
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:509-654` (from_continuous)
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:655-720` (to_continuous, param_names)

**Step 1: Remove dimensions and add DSR eta**

Remove from search space:
- Index 24: `min_epochs_before_stopping` (early stopping disabled)
- `sharpe_weight` (DSR replaces — if it's separate from the bounds array, just fix to 0.0)

Add to search space:
- `dsr_eta`: log scale `(0.001_f64.ln(), 0.05_f64.ln())` — replaces removed dimension

Update `continuous_bounds()`, `from_continuous()`, `to_continuous()`, and `param_names()` to match. The index numbers for all subsequent parameters shift — be careful with the mapping.

**Step 2: Enable DSR in hyperopt**

In `from_continuous()` where hyperparameters are assembled (~line 2350):

```rust
            use_dsr: true,  // Always enabled for hyperopt
            dsr_eta: params.dsr_eta,
            enable_risk_adjusted_rewards: false,  // DSR handles risk-adjustment
```

And in the RewardConfig construction in the hyperopt adapter:

```rust
            enable_normalization: false,  // DSR replaces EMA normalizer
            use_dsr: true,
            dsr_eta: params.dsr_eta,
```

**Step 3: Update DQNParams struct**

Add `dsr_eta: f64` field to `DQNParams` struct and its default.

**Step 4: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib -- hyperopt 2>&1 | tail -10
```

Fix any test that hardcodes the 26D dimension count.

**Step 5: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "feat(ml): add DSR eta to hyperopt, remove wasted dimensions (26D→~24D)"
```

---

### Task 2.4: Widen gamma range

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:464`

```rust
// Old: (0.88, 0.96)
(0.88, 0.99),                  // 2: gamma (widened: allow longer-horizon strategies)
```

**Step: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): widen gamma hyperopt range to [0.88, 0.99]"
```

---

### Task 2.5: Phase 2 integration test

```bash
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10
SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5
```

---

## Phase 3: Fix IQN/QR-DQN & C51 (4 tasks)

### Task 3.1: Fix IQN training path

**Files:**
- Modify: `crates/ml/src/dqn/dqn.rs:2316+` (compute_loss_internal)
- Modify: `crates/ml/src/dqn/dqn.rs:258-262` (defaults)

The bug: `use_iqn=true` creates a separate `iqn_network` (QuantileNetwork). The training loss (`compute_loss_internal`) trains this network. But `batch_greedy_actions()` and the backtest evaluation path in the hyperopt adapter may use the `dist_dueling_network` instead.

**Step 1: Audit the inference path**

Read `dqn.rs` around lines 1481, 1634, 1775 to verify which network is used for action selection when `use_iqn=true`. Read `batch_greedy_actions` to confirm it also checks `use_iqn`.

**Step 2: Fix — ensure consistent network usage**

If `batch_greedy_actions` does NOT check `use_iqn` and always uses `dist_dueling_network`, add the IQN branch:

```rust
// In batch_greedy_actions():
if self.config.use_iqn && self.iqn_network.is_some() {
    // Use IQN network for Q-value estimation
    let iqn_net = self.iqn_network.as_ref().unwrap();
    // ... compute Q-values via IQN
} else {
    // Use dist_dueling_network (C51 path)
    // ... existing code
}
```

**Step 3: Enable IQN by default when num_atoms > 100**

Change the default at line 258-259:

```rust
// Old: use_iqn: false,
// IQN: Now fixed — same network used for training and inference
use_iqn: false, // Still default false; hyperopt enables via num_atoms > 100
```

The hyperopt already enables it at `trainer.rs:454`. Just verify the training path works.

**Step 4: Add/update IQN integration test**

Check existing test at `trainer.rs:3811` (`test_iqn_training_step`). Ensure it verifies that:
- IQN loss decreases over multiple steps
- Action selection uses the IQN network (not dist_dueling)
- Target network update works for IQN

**Step 5: Commit**

```bash
git add crates/ml/src/dqn/dqn.rs
git commit -m "fix(ml): fix IQN training path — consistent network for train+inference"
```

---

### Task 3.2: Fix C51 v_min/v_max for DSR rewards

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:474-475`
- Modify: `crates/ml/src/dqn/dqn.rs:238-239`

**Step 1: Update hyperopt bounds**

DSR rewards are typically ±2.0. With gamma=0.92, Q-values ≈ ±25.

```rust
// Old: (-2.0, -0.5),  // 10: v_min
//      (0.5, 2.0),    // 11: v_max
(-30.0, -5.0),         // 10: v_min (DSR Q-values ≈ ±25)
(5.0, 30.0),           // 11: v_max
```

**Step 2: Update DQNConfig defaults**

```rust
// Old: v_min: -10.0, v_max: 10.0
v_min: -25.0,
v_max: 25.0,
```

**Step 3: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs crates/ml/src/dqn/dqn.rs
git commit -m "fix(ml): align C51 v_min/v_max with DSR Q-value range (±25)"
```

---

### Task 3.3: Cap CQL alpha

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:500`

```rust
// Old: (0.0, 0.5)     // 22: cql_alpha
(0.0, 0.1),            // 22: cql_alpha (capped: 0.5 adds ~0.8 loss, crushes 5-action Q-diffs)
```

**Step: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): cap CQL alpha at 0.1 to prevent Q-value crushing"
```

---

### Task 3.4: Phase 3 integration test

```bash
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10
```

---

## Phase 4: Network Architecture (4 tasks)

### Task 4.1: Add RMSNorm to distributional-dueling network

**Files:**
- Modify: `crates/ml/src/dqn/distributional_dueling.rs:159-170` (shared layers)
- Read: `crates/ml/src/dqn/rmsnorm.rs` (existing RMSNorm impl)

**Step 1: Import RMSNorm**

Add import at top of `distributional_dueling.rs`:

```rust
use super::rmsnorm::RMSNorm;
```

**Step 2: Add RMSNorm after each hidden layer in shared backbone**

In the constructor where shared layers are built (around lines 159-170), add RMSNorm after each Linear+LeakyReLU:

```rust
// For each shared layer:
// linear → LeakyReLU → RMSNorm
let rmsnorm = RMSNorm::new(layer_dim, vb.pp(format!("shared_rmsnorm_{}", i)))?;
```

Store the RMSNorm layers in the struct and apply them in `forward()`.

**Step 3: Add RMSNorm to value/advantage stream hidden layers**

Same pattern for the value and advantage stream hidden layers.

**Step 4: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib -- distributional 2>&1 | tail -10
```

**Step 5: Commit**

```bash
git add crates/ml/src/dqn/distributional_dueling.rs
git commit -m "feat(ml): add RMSNorm to distributional-dueling shared backbone and streams"
```

---

### Task 4.2: Constant-width hidden layers

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer.rs:288-305`
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` (hidden_dim_base bounds)

**Step 1: Change hidden_dims construction**

Find lines 288-305 where `hidden_dims` is built as `[base, base/2, base/4]`:

```rust
// Old: vec![base, base / 2, base / 4]
// New: constant-width (no tapering, no silently-discarded 64-wide layer)
vec![base, base]
```

**Step 2: Narrow hidden_dim_base range**

In hyperopt bounds (index 21):

```rust
// Old: (256.0, 4096.0)
(256.0, 1024.0),               // 21: hidden_dim_base (narrowed: 4096 can't converge in 30k steps)
```

**Step 3: Commit**

```bash
git add crates/ml/src/trainers/dqn/trainer.rs crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): constant-width [base,base] hidden layers, narrow range to [256,1024]"
```

---

### Task 4.3: Default n_steps=3

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:478`

```rust
// Old: (1.0, 5.0)    // 14: n_steps
(3.0, 5.0),            // 14: n_steps (raised minimum: n=3 is Rainbow standard, 3rd most impactful)
```

**Step: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): raise n_steps minimum from 1 to 3 (Rainbow standard)"
```

---

### Task 4.4: Phase 4 integration test

```bash
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10
SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5
```

---

## Phase 5: State Representation (4 tasks)

### Task 5.1: Remove dead features

**Files:**
- Modify: `crates/ml/src/features/extraction.rs:545` (regime features)
- Modify: `crates/ml/src/features/extraction.rs:274` (OFI features)
- Modify: `crates/ml/src/dqn/dqn.rs:205` (state_dim default)
- Modify: all files referencing `state_dim: 54` or feature indices 40-50

**Step 1: Remove regime features (indices 40-42)**

In `extract_regime_features_v2()`, instead of writing 3 zeros, make it a no-op (don't write to output, reduce feature count by 3).

**Step 2: Remove OFI features (indices 43-50)**

In the OFI extraction, instead of zero-padding 8 features, skip them entirely when MBP-10 data is unavailable (which is always during hyperopt).

**Step 3: Update state_dim**

```rust
// Old: state_dim: 54
state_dim: 43, // 51 market - 3 regime - 8 OFI + 3 portfolio
```

Update all hardcoded references to 54.

**Step 4: Fix normalize_with_skip indices**

In `crates/ml/src/trainers/dqn/statistics.rs`, the skip indices [125, 126, 127] are remnants of the old 140-feature architecture. Update to match the new 43-feature layout or remove entirely (all features should be z-scored).

**Step 5: Run tests — expect some failures from dimension changes**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | grep -E "FAIL|test result"
```

Fix any tests that hardcode 54-dim or specific feature indices.

**Step 6: Commit**

```bash
git add crates/ml/src/features/extraction.rs crates/ml/src/dqn/dqn.rs crates/ml/src/trainers/dqn/statistics.rs
git commit -m "fix(ml): remove 11 dead features (regime+OFI), state_dim 54→43"
```

---

### Task 5.2: Fix volume normalization

**Files:**
- Modify: `crates/ml/src/features/extraction.rs:584` (volume)

```rust
// Old: safe_normalize(bar.volume, 0.0, 1_000_000.0)
// New: log-transform volume before normalizing (futures volume can be 10M+)
safe_normalize(bar.volume.max(1.0).ln(), 0.0, 18.0)  // ln(1)=0, ln(10M)≈16.1
```

**Step: Commit**

```bash
git add crates/ml/src/features/extraction.rs
git commit -m "fix(ml): log-transform volume normalization for futures scale"
```

---

### Task 5.3: Fix ATR normalization

**Files:**
- Modify: `crates/ml/src/features/extraction.rs:319` (ATR)

```rust
// Old: safe_normalize(indicators.last_atr, 0.0, 100.0)
// New: log-transform ATR (futures ATR ranges 0.5 to 5000+)
safe_normalize(indicators.last_atr.max(0.001).ln(), -7.0, 9.0)  // ln(0.001)≈-6.9, ln(5000)≈8.5
```

**Step: Commit**

```bash
git add crates/ml/src/features/extraction.rs
git commit -m "fix(ml): log-transform ATR normalization for futures scale"
```

---

### Task 5.4: Center portfolio value feature

**Files:**
- Modify: `crates/ml/src/dqn/portfolio_tracker.rs:162`

```rust
// Old: portfolio_value / self.initial_capital  (centered on 1.0)
// New: center on 0.0 for consistency with z-scored market features
(portfolio_value / self.initial_capital) - 1.0
```

**Step: Commit**

```bash
git add crates/ml/src/dqn/portfolio_tracker.rs
git commit -m "fix(ml): center portfolio value feature at 0.0 (was 1.0)"
```

---

## Phase 6: Hyperopt Objective (2 tasks)

### Task 6.1: Increase Sharpe weight in composite objective

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:3380-3383` (tanh composite)

Find the tanh composite score lines and change weights:

```rust
// Old: 0.4 * sortino + 0.3 * calmar + 0.2 * sharpe + 0.1 * omega
// New: Sharpe-primary (it's what we actually care about)
let composite_score = 0.40 * (adjusted_sharpe / 2.0).tanh()
    + 0.25 * (sortino / 3.0).tanh()
    + 0.20 * (calmar / 5.0).tanh()
    + 0.15 * (omega / 2.0).tanh();
```

This makes Sharpe effective weight = 0.6 × 0.4 × S/2 = 12% (up from 6%).

**Step: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): increase Sharpe weight in hyperopt objective (6%→12% effective)"
```

---

### Task 6.2: Final integration test and commit

**Step 1: Run full test suite**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10
```

Expected: All tests pass (adjusted for state_dim changes).

**Step 2: Run clippy**

```bash
SQLX_OFFLINE=true cargo clippy --workspace 2>&1 | tail -10
```

Expected: 0 errors, 0 warnings.

**Step 3: Run workspace check**

```bash
SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5
```

**Step 4: Final commit**

```bash
git add -A
git commit -m "feat(ml): DQN hyperopt overhaul v2 — complete 6-phase fix

Phase 1: DSR reward function (replaces double-normalized pipeline)
Phase 2: Training steps (30 epochs, batch≤512, wired noisy_sigma)
Phase 3: Fixed IQN path, C51 v_min/v_max, capped CQL alpha
Phase 4: RMSNorm in dist-dueling, constant-width layers, n_steps≥3
Phase 5: Removed 11 dead features, log-scaled volume/ATR
Phase 6: Sharpe-primary objective weighting

Fixes 19 root causes identified by 6-agent deep audit.
Reduces hyperopt search space from 26D to ~24D.
Expected: Sharpe >> 0 on next hyperopt run."
```

---

## Summary: All Changes by File

| File | Changes |
|------|---------|
| `dqn/reward.rs` | DSR struct, config fields, wired into calculate_reward, diversity scaled, reset methods |
| `dqn/experience.rs` | Precision 10000→1000000 |
| `dqn/noisy_layers.rs` | Wire sigma_init parameter |
| `dqn/dqn.rs` | Fix IQN path, v_min/v_max defaults, state_dim 54→43, epsilon floor 0.02 |
| `dqn/distributional_dueling.rs` | Add RMSNorm to shared backbone + streams |
| `dqn/portfolio_tracker.rs` | Center portfolio value at 0.0 |
| `trainers/dqn/trainer.rs` | Skip risk-adjusted when DSR, reset portfolio/DSR per epoch, constant-width hidden layers, use_dsr config |
| `trainers/dqn/config.rs` | Add use_dsr, dsr_eta to DQNHyperparameters |
| `hyperopt/adapters/dqn.rs` | DSR eta tunable, remove wasted dims, batch≤512, n_steps≥3, CQL≤0.1, v_min/v_max scaled, gamma≤0.99, hidden_dim≤1024, Sharpe-primary objective, enable DSR+disable normalizers |
| `features/extraction.rs` | Remove regime+OFI features, log-transform volume/ATR |
| `trainers/dqn/statistics.rs` | Fix stale skip indices |

## Post-Implementation

1. Push to main
2. Wait for Argo CI (compile-services + compile-training)
3. Submit hyperopt v4 workflow: 20 trials × 30 epochs
4. Monitor via Prometheus: `foxhunt_hyperopt_best_objective`, `foxhunt_training_epoch_sharpe`
