foxhunt/docs/plans/2026-03-07-dqn-hyperopt-overhaul-v2.md

DQN Hyperopt Overhaul v2 — Full Root Cause Fix

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Fix all 19 root causes preventing DQN hyperopt from achieving positive Sharpe ratio.

Architecture: Replace double-normalized reward pipeline with Differential Sharpe Ratio (DSR), fix broken IQN training path, add RMSNorm to distributional-dueling network, remove dead features, tighten hyperopt search space from 26D to ~22D.

Tech Stack: Rust, Candle ML framework, PSO hyperopt, OHLCV futures data

---

Phase 1: Reward Function (7 tasks)

Task 1.1: Implement Differential Sharpe Ratio struct

Files:

• Modify: crates/ml/src/dqn/reward.rs:170-199 (RewardConfig)
• Modify: crates/ml/src/dqn/reward.rs:399-410 (RewardFunction struct)

Step 1: Add DSR struct and config fields

Add before RewardConfig (around line 165):

/// Differential Sharpe Ratio (Moody & Saffell, 2001)
///
/// Computes the incremental change in Sharpe ratio per step.
/// Replaces both EMA reward normalizer and risk-adjusted division.
/// Produces dense, well-scaled, risk-adjusted reward per step.
pub struct DifferentialSharpeRatio {
    ema_return: f64,      // A_t: EMA of returns
    ema_return_sq: f64,   // B_t: EMA of squared returns
    eta: f64,             // Decay rate (0.01 = ~100-step window)
    initialized: bool,
}

impl DifferentialSharpeRatio {
    pub fn new(eta: f64) -> Self {
        Self {
            ema_return: 0.0,
            ema_return_sq: 0.0,
            eta: eta.clamp(0.0001, 0.1),
            initialized: false,
        }
    }

    /// Compute differential Sharpe ratio for a new return observation.
    ///
    /// Formula (Moody & Saffell 2001):
    ///   DSR_t = (B_{t-1} * r_t - 0.5 * A_{t-1} * (r_t^2 - B_{t-1})) / (B_{t-1} - A_{t-1}^2)^{3/2}
    ///
    /// Returns 0.0 on first call (no prior statistics).
    pub fn step(&mut self, r_t: f64) -> f64 {
        if !self.initialized {
            self.ema_return = r_t;
            self.ema_return_sq = r_t * r_t + 1e-8; // Small epsilon for numerical stability
            self.initialized = true;
            return 0.0;
        }

        let a = self.ema_return;      // A_{t-1}
        let b = self.ema_return_sq;   // B_{t-1}
        let r_sq = r_t * r_t;

        // Variance of returns: Var = E[r^2] - E[r]^2
        let variance = b - a * a;

        let dsr = if variance > 1e-12 {
            let numer = b * r_t - 0.5 * a * (r_sq - b);
            let denom = variance.powf(1.5);
            (numer / denom).clamp(-5.0, 5.0) // Bound extreme DSR values
        } else {
            // Insufficient variance — return scaled raw return as fallback
            r_t * 100.0 // Scale raw PnL (~0.0002) to ~0.02 range
        };

        // Update EMAs AFTER computing DSR (use previous stats for current step)
        self.ema_return = self.eta * r_t + (1.0 - self.eta) * a;
        self.ema_return_sq = self.eta * r_sq + (1.0 - self.eta) * b;

        dsr
    }

    /// Reset state between episodes/epochs for stationarity.
    pub fn reset(&mut self) {
        self.ema_return = 0.0;
        self.ema_return_sq = 0.0;
        self.initialized = false;
    }
}

Step 2: Add config fields to RewardConfig (lines 170-199)

Add two fields after sharpe_window (line 198):

    /// Use Differential Sharpe Ratio instead of EMA normalizer + risk-adjusted division
    pub use_dsr: bool,
    /// DSR EMA decay rate (0.001=slow/1000-step, 0.05=fast/20-step). Tunable in hyperopt.
    pub dsr_eta: f64,

Step 3: Update RewardConfig::default() (lines 201-220)

Add to default impl after sharpe_window: 20:

            use_dsr: false,     // Backward compat: off by default
            dsr_eta: 0.01,      // ~100-step effective window

Step 4: Add DSR field to RewardFunction struct (lines 399-410)

Add field:

    dsr: Option<DifferentialSharpeRatio>,

Step 5: Initialize DSR in RewardFunction constructor (line 433)

After normalizer initialization:

        let dsr = config.use_dsr.then(|| DifferentialSharpeRatio::new(config.dsr_eta));

Step 6: Add DSR to RewardConfigBuilder

Add builder method (after enable_normalization builder at line 294):

    pub fn use_dsr(mut self, enable: bool) -> Self {
        self.use_dsr = Some(enable);
        self
    }

    pub fn dsr_eta(mut self, eta: f64) -> Self {
        self.dsr_eta = Some(eta);
        self
    }

And add fields to builder struct + build():

    use_dsr: Option<bool>,
    dsr_eta: Option<f64>,

In build():

            use_dsr: self.use_dsr.unwrap_or(false),
            dsr_eta: self.dsr_eta.unwrap_or(0.01),

Step 7: Add reset_dsr() method to RewardFunction

    pub fn reset_dsr(&mut self) {
        if let Some(ref mut dsr) = self.dsr {
            dsr.reset();
        }
    }

Step 8: Run tests

SQLX_OFFLINE=true cargo test -p ml --lib -- reward 2>&1 | tail -5

Expected: All existing reward tests pass.

Step 9: Add DSR unit tests

Add at bottom of reward.rs tests module:

    #[test]
    fn test_dsr_basic() {
        let mut dsr = DifferentialSharpeRatio::new(0.01);
        // First step returns 0 (no prior stats)
        assert_eq!(dsr.step(0.001), 0.0);
        // Positive return after positive → positive DSR
        let d1 = dsr.step(0.002);
        assert!(d1 > 0.0, "Positive return should give positive DSR: {}", d1);
        // Negative return → negative DSR
        let d2 = dsr.step(-0.003);
        assert!(d2 < 0.0, "Negative return should give negative DSR: {}", d2);
    }

    #[test]
    fn test_dsr_bounded() {
        let mut dsr = DifferentialSharpeRatio::new(0.01);
        dsr.step(0.001);
        // Even extreme returns should be clamped
        let d = dsr.step(100.0);
        assert!(d <= 5.0 && d >= -5.0, "DSR should be clamped: {}", d);
    }

    #[test]
    fn test_dsr_reset() {
        let mut dsr = DifferentialSharpeRatio::new(0.01);
        dsr.step(0.001);
        dsr.step(0.002);
        dsr.reset();
        // After reset, first step returns 0 again
        assert_eq!(dsr.step(0.001), 0.0);
    }

    #[test]
    fn test_dsr_config_roundtrip() {
        let config = RewardConfig::builder()
            .use_dsr(true)
            .dsr_eta(0.02)
            .build();
        assert!(config.use_dsr);
        assert!((config.dsr_eta - 0.02).abs() < 1e-10);
    }

Step 10: Run tests

SQLX_OFFLINE=true cargo test -p ml --lib -- dsr 2>&1 | tail -10

Expected: 4 new tests pass.

Step 11: Commit

git add crates/ml/src/dqn/reward.rs
git commit -m "feat(ml): add Differential Sharpe Ratio (DSR) struct and config"

---

Task 1.2: Wire DSR into calculate_reward()

Files:

• Modify: crates/ml/src/dqn/reward.rs:485-615 (calculate_reward method)

Step 1: Modify calculate_reward to use DSR when enabled

Replace the section from line 524 (diversity bonus) through line 603 (normalizer) with:

        // Calculate diversity bonus (in-training regularization)
        let entropy = calculate_entropy(recent_actions);
        let entropy_threshold = Decimal::try_from(0.3).unwrap_or(Decimal::ZERO);
        let diversity_bonus = if entropy < entropy_threshold {
            // DSR mode: scale penalty proportional to trading signal (~0.0005)
            // Legacy mode: -0.1 (500x larger — known to dominate reward signal)
            if self.config.use_dsr {
                Decimal::try_from(-0.0005).unwrap_or(Decimal::ZERO)
            } else {
                self.config.diversity_weight
            }
        } else {
            Decimal::ZERO
        };

        let pnl_base_reward = base_reward + diversity_bonus;

        let final_reward_f64: f64 = if self.config.use_dsr {
            // DSR path: skip Sharpe blend and EMA normalizer
            // Pass raw PnL + diversity through DSR transform
            let raw_pnl: f64 = pnl_base_reward.try_into()
                .map_err(|e| MLError::InvalidInput(format!("Reward conversion failed: {}", e)))?;

            if let Some(ref mut dsr) = self.dsr {
                dsr.step(raw_pnl)
            } else {
                // Fallback: scale raw PnL to reasonable range
                raw_pnl * 100.0
            }
        } else {
            // Legacy path: Sharpe blend + optional EMA normalization (unchanged)
            let pnl_normalized_f64: f64 = ((next_state.portfolio_features.get(0).unwrap_or(&100000.0)
                - current_state.portfolio_features.get(0).unwrap_or(&100000.0))
                / current_state.portfolio_features.get(0).unwrap_or(&100000.0)) as f64;

            self.returns_buffer.push_back(pnl_normalized_f64);
            if self.returns_buffer.len() > self.config.sharpe_window {
                self.returns_buffer.pop_front();
            }

            let sharpe_component = if self.returns_buffer.len() > 1 {
                let mean: f64 = self.returns_buffer.iter().sum::<f64>() / self.returns_buffer.len() as f64;
                let variance: f64 = self.returns_buffer.iter()
                    .map(|r| (r - mean).powi(2))
                    .sum::<f64>() / self.returns_buffer.len() as f64;
                let std_dev = variance.sqrt();
                if std_dev > 1e-8 { mean / std_dev } else { 0.0 }
            } else {
                0.0
            };
            let sharpe_decimal = Decimal::try_from(sharpe_component).unwrap_or(Decimal::ZERO);

            let sharpe_weight = self.config.sharpe_weight;
            let pnl_weight_adjusted = Decimal::ONE - sharpe_weight;
            let final_reward = pnl_base_reward * pnl_weight_adjusted + sharpe_decimal * sharpe_weight;

            let final_reward_f64: f64 = final_reward.try_into()
                .map_err(|e| MLError::InvalidInput(format!("Reward conversion failed: {}", e)))?;

            // Apply EMA normalization if enabled
            if let Some(normalizer) = &mut self.normalizer {
                let norm = normalizer.normalize(final_reward_f64);
                normalizer.update(final_reward_f64);
                norm.clamp(-3.0, 3.0)
            } else {
                final_reward_f64
            }
        };

Step 2: Run tests

SQLX_OFFLINE=true cargo test -p ml --lib -- reward 2>&1 | tail -5

Step 3: Commit

git add crates/ml/src/dqn/reward.rs
git commit -m "feat(ml): wire DSR into calculate_reward, scale diversity penalty"

---

Task 1.3: Disable risk-adjusted division when DSR active

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs:2234 (risk-adjusted call)
• Modify: crates/ml/src/trainers/dqn/trainer.rs:508 (RewardConfig construction)

Step 1: Update RewardConfig construction in trainer (line 508)

Change the RewardConfig construction to enable DSR when hyperparams say so. Add use_dsr and dsr_eta fields to DQNHyperparameters first.

In crates/ml/src/trainers/dqn/config.rs, add to DQNHyperparameters:

    pub use_dsr: bool,
    pub dsr_eta: f64,

And in its default:

            use_dsr: false,
            dsr_eta: 0.01,

Then in trainer.rs around line 508, update the RewardConfig construction:

            use_dsr: self.hyperparams.use_dsr,
            dsr_eta: self.hyperparams.dsr_eta,

Step 2: Skip risk-adjusted reward when DSR active (line 2234)

Replace line 2234:

                    // When DSR is active, it already handles risk-adjustment.
                    // Legacy path: divide by PnL volatility (Sharpe-like normalization)
                    let risk_adjusted_reward = if self.hyperparams.use_dsr {
                        barrier_scaled_reward // DSR output is already risk-adjusted
                    } else {
                        self.calculate_risk_adjusted_reward(barrier_scaled_reward)
                    };

Step 3: Run tests

SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5

Step 4: Commit

git add crates/ml/src/trainers/dqn/config.rs crates/ml/src/trainers/dqn/trainer.rs
git commit -m "feat(ml): skip risk-adjusted division when DSR active"

---

Task 1.4: Increase experience reward precision

Files:

• Modify: crates/ml/src/dqn/experience.rs:31,43

Step 1: Change scale factor

At line 31 (or wherever Experience::new stores reward):

// Old: reward: (reward * 10000.0) as i32,
// New: 100x more precision — DSR values ~±2.0 map to ±2_000_000 (fits i32)
reward: (reward * 1_000_000.0) as i32,

At line 43 (or reward_f32() method):

// Old: self.reward as f32 / 10000.0
self.reward as f32 / 1_000_000.0

Step 2: Run tests

SQLX_OFFLINE=true cargo test -p ml --lib -- experience 2>&1 | tail -5

Step 3: Commit

git add crates/ml/src/dqn/experience.rs
git commit -m "fix(ml): increase experience reward precision from 10000 to 1000000"

---

Task 1.5: Reset portfolio tracker between epochs

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs:1511-1546

Step 1: Add portfolio reset at epoch boundary

Around line 1503 (where self.pnl_history.clear() already exists), add:

            self.pnl_history.clear();

            // Reset portfolio tracker to initial capital for reward stationarity.
            // Without this, portfolio compounding creates non-stationary rewards
            // (same action produces 5x different reward depending on epoch),
            // corrupting replay buffer and preventing Q-value convergence.
            self.portfolio_tracker.reset();

Also add DSR reset at the same location:

            // Reset DSR EMA state for clean epoch
            self.reward_fn.reset_dsr();

And reset the returns_buffer in the reward function. Add a method:

In reward.rs, add to RewardFunction:

    /// Reset per-episode/epoch state (returns buffer + DSR)
    pub fn reset_epoch_state(&mut self) {
        self.returns_buffer.clear();
        self.reset_dsr();
        // Reset normalizer running stats to prevent drift
        if let Some(ref mut normalizer) = self.normalizer {
            *normalizer = RewardNormalizer::new();
        }
    }

Then in trainer.rs at epoch boundary, call self.reward_fn.reset_epoch_state() instead of just reset_dsr().

Step 2: Run tests

SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5

Step 3: Commit

git add crates/ml/src/trainers/dqn/trainer.rs crates/ml/src/dqn/reward.rs
git commit -m "fix(ml): reset portfolio, DSR, and reward stats between epochs"

---

Task 1.6: Reset DSR at episode boundaries

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs:2080,2342

At barrier exit (line 2080) and time/data boundary (line 2342), add:

            self.reward_fn.reset_dsr();

This ensures DSR doesn't carry state across independent trading episodes.

Step: Commit

git add crates/ml/src/trainers/dqn/trainer.rs
git commit -m "fix(ml): reset DSR at episode boundaries (barrier exit, time boundary)"

---

Task 1.7: Phase 1 integration test

Step 1: Run full ml test suite

SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10

Expected: 2742+ tests pass, 0 failures.

Step 2: Verify compile with check

SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5

Expected: 0 errors, 0 warnings.

---

Phase 2: Training Steps & Exploration (5 tasks)

Task 2.1: Wire noisy_sigma_init to NoisyLinear

Files:

• Modify: crates/ml/src/dqn/noisy_layers.rs:59-80
• Modify: crates/ml/src/dqn/dqn.rs (pass sigma_init to NoisyLinear)

Step 1: Add sigma_init parameter to NoisyLinear::new()

Change NoisyLinear::new() signature to accept sigma_init: f64:

At line 80, replace:

// Old:
let sigma_init = 0.5 / (in_features as f64).sqrt();
// New: use caller-provided sigma, fall back to Fortunato et al. default

Update the constructor to accept and use the parameter. Also update all call sites in dqn.rs where NoisyLinear::new() is called, passing self.config.noisy_sigma_init.

Step 2: Add epsilon floor back for NoisyNets

In crates/ml/src/dqn/dqn.rs around line 1460-1461, change:

// Old: self.config.noisy_epsilon_floor  (was 0.0)
// New: small epsilon floor ensures exploration in early training
// when NoisyNet sigma hasn't annealed enough
let effective_epsilon = if self.config.use_noisy_nets {
    self.config.noisy_epsilon_floor.max(0.02) // Minimum 2% random
} else {
    self.epsilon
};

Step 3: Commit

git add crates/ml/src/dqn/noisy_layers.rs crates/ml/src/dqn/dqn.rs
git commit -m "fix(ml): wire noisy_sigma_init to NoisyLinear, add 2% epsilon floor"

---

Task 2.2: Increase epochs, cap batch_size, fix min_replay_size

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:463,504,2310

Step 1: Change hyperopt bounds

Line 463 — batch_size:

// Old: (64.0, 4096.0)
(64.0, 512.0),                  // 1: batch_size (capped: 4096 → too few gradient steps)

Line 504 — min_epochs_before_stopping:

// Remove from search space entirely (early stopping disabled in hyperopt)
// This dimension will be removed in Task 2.3

Step 2: Change epochs parameter

Find where the hyperopt adapter constructs the trainer (look for epochs parameter passed to DqnHyperoptAdapter::new() or similar). Change from 8 to 30.

This is passed by the caller — likely in the hyperopt binary or workflow template. Find with:

grep -rn "DqnHyperoptAdapter::new\|epochs.*=.*8" crates/ml/src/hyperopt/

Update the caller to pass epochs: 30.

Step 3: Fix min_replay_size

Around line 2310 where min_replay_size is set:

// Old: min_replay_size: params.batch_size * 2,
min_replay_size: (params.batch_size * 50).min(params.buffer_size / 2),

Step 4: Fix PER beta annealing

Find where per_beta_annealing_steps or similar is set. Replace hardcoded epochs * 1000 with:

// Old: per_beta_annealing_steps: epochs * 1000,
// New: match actual gradient steps
per_beta_annealing_steps: (training_data_len / params.batch_size) * epochs,

If training_data_len isn't available at this point, use a conservative estimate:

per_beta_annealing_steps: epochs * 2000, // ~2000 steps/epoch at batch_size=64 with 130k bars

Step 5: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): cap batch_size=512, epochs=30, min_replay=50x, fix PER annealing"

---

Task 2.3: Remove wasted hyperopt dimensions

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:460-506 (continuous_bounds)
• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:509-654 (from_continuous)
• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:655-720 (to_continuous, param_names)

Step 1: Remove dimensions and add DSR eta

Remove from search space:

• Index 24: min_epochs_before_stopping (early stopping disabled)
• sharpe_weight (DSR replaces — if it's separate from the bounds array, just fix to 0.0)

Add to search space:

• dsr_eta: log scale (0.001_f64.ln(), 0.05_f64.ln()) — replaces removed dimension

Update continuous_bounds(), from_continuous(), to_continuous(), and param_names() to match. The index numbers for all subsequent parameters shift — be careful with the mapping.

Step 2: Enable DSR in hyperopt

In from_continuous() where hyperparameters are assembled (~line 2350):

            use_dsr: true,  // Always enabled for hyperopt
            dsr_eta: params.dsr_eta,
            enable_risk_adjusted_rewards: false,  // DSR handles risk-adjustment

And in the RewardConfig construction in the hyperopt adapter:

            enable_normalization: false,  // DSR replaces EMA normalizer
            use_dsr: true,
            dsr_eta: params.dsr_eta,

Step 3: Update DQNParams struct

Add dsr_eta: f64 field to DQNParams struct and its default.

Step 4: Run tests

SQLX_OFFLINE=true cargo test -p ml --lib -- hyperopt 2>&1 | tail -10

Fix any test that hardcodes the 26D dimension count.

Step 5: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "feat(ml): add DSR eta to hyperopt, remove wasted dimensions (26D→~24D)"

---

Task 2.4: Widen gamma range

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:464

// Old: (0.88, 0.96)
(0.88, 0.99),                  // 2: gamma (widened: allow longer-horizon strategies)

Step: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): widen gamma hyperopt range to [0.88, 0.99]"

---

Task 2.5: Phase 2 integration test

SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10
SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5

---

Phase 3: Fix IQN/QR-DQN & C51 (4 tasks)

Task 3.1: Fix IQN training path

Files:

• Modify: crates/ml/src/dqn/dqn.rs:2316+ (compute_loss_internal)
• Modify: crates/ml/src/dqn/dqn.rs:258-262 (defaults)

The bug: use_iqn=true creates a separate iqn_network (QuantileNetwork). The training loss (compute_loss_internal) trains this network. But batch_greedy_actions() and the backtest evaluation path in the hyperopt adapter may use the dist_dueling_network instead.

Step 1: Audit the inference path

Read dqn.rs around lines 1481, 1634, 1775 to verify which network is used for action selection when use_iqn=true. Read batch_greedy_actions to confirm it also checks use_iqn.

Step 2: Fix — ensure consistent network usage

If batch_greedy_actions does NOT check use_iqn and always uses dist_dueling_network, add the IQN branch:

// In batch_greedy_actions():
if self.config.use_iqn && self.iqn_network.is_some() {
    // Use IQN network for Q-value estimation
    let iqn_net = self.iqn_network.as_ref().unwrap();
    // ... compute Q-values via IQN
} else {
    // Use dist_dueling_network (C51 path)
    // ... existing code
}

Step 3: Enable IQN by default when num_atoms > 100

Change the default at line 258-259:

// Old: use_iqn: false,
// IQN: Now fixed — same network used for training and inference
use_iqn: false, // Still default false; hyperopt enables via num_atoms > 100

The hyperopt already enables it at trainer.rs:454. Just verify the training path works.

Step 4: Add/update IQN integration test

Check existing test at trainer.rs:3811 (test_iqn_training_step). Ensure it verifies that:

• IQN loss decreases over multiple steps
• Action selection uses the IQN network (not dist_dueling)
• Target network update works for IQN

Step 5: Commit

git add crates/ml/src/dqn/dqn.rs
git commit -m "fix(ml): fix IQN training path — consistent network for train+inference"

---

Task 3.2: Fix C51 v_min/v_max for DSR rewards

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:474-475
• Modify: crates/ml/src/dqn/dqn.rs:238-239

Step 1: Update hyperopt bounds

DSR rewards are typically ±2.0. With gamma=0.92, Q-values ≈ ±25.

// Old: (-2.0, -0.5),  // 10: v_min
//      (0.5, 2.0),    // 11: v_max
(-30.0, -5.0),         // 10: v_min (DSR Q-values ≈ ±25)
(5.0, 30.0),           // 11: v_max

Step 2: Update DQNConfig defaults

// Old: v_min: -10.0, v_max: 10.0
v_min: -25.0,
v_max: 25.0,

Step 3: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs crates/ml/src/dqn/dqn.rs
git commit -m "fix(ml): align C51 v_min/v_max with DSR Q-value range (±25)"

---

Task 3.3: Cap CQL alpha

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:500

// Old: (0.0, 0.5)     // 22: cql_alpha
(0.0, 0.1),            // 22: cql_alpha (capped: 0.5 adds ~0.8 loss, crushes 5-action Q-diffs)

Step: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): cap CQL alpha at 0.1 to prevent Q-value crushing"

---

Task 3.4: Phase 3 integration test

SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10

---

Phase 4: Network Architecture (4 tasks)

Task 4.1: Add RMSNorm to distributional-dueling network

Files:

• Modify: crates/ml/src/dqn/distributional_dueling.rs:159-170 (shared layers)
• Read: crates/ml/src/dqn/rmsnorm.rs (existing RMSNorm impl)

Step 1: Import RMSNorm

Add import at top of distributional_dueling.rs:

use super::rmsnorm::RMSNorm;

Step 2: Add RMSNorm after each hidden layer in shared backbone

In the constructor where shared layers are built (around lines 159-170), add RMSNorm after each Linear+LeakyReLU:

// For each shared layer:
// linear → LeakyReLU → RMSNorm
let rmsnorm = RMSNorm::new(layer_dim, vb.pp(format!("shared_rmsnorm_{}", i)))?;

Store the RMSNorm layers in the struct and apply them in forward().

Step 3: Add RMSNorm to value/advantage stream hidden layers

Same pattern for the value and advantage stream hidden layers.

Step 4: Run tests

SQLX_OFFLINE=true cargo test -p ml --lib -- distributional 2>&1 | tail -10

Step 5: Commit

git add crates/ml/src/dqn/distributional_dueling.rs
git commit -m "feat(ml): add RMSNorm to distributional-dueling shared backbone and streams"

---

Task 4.2: Constant-width hidden layers

Files:

• Modify: crates/ml/src/trainers/dqn/trainer.rs:288-305
• Modify: crates/ml/src/hyperopt/adapters/dqn.rs (hidden_dim_base bounds)

Step 1: Change hidden_dims construction

Find lines 288-305 where hidden_dims is built as [base, base/2, base/4]:

// Old: vec![base, base / 2, base / 4]
// New: constant-width (no tapering, no silently-discarded 64-wide layer)
vec![base, base]

Step 2: Narrow hidden_dim_base range

In hyperopt bounds (index 21):

// Old: (256.0, 4096.0)
(256.0, 1024.0),               // 21: hidden_dim_base (narrowed: 4096 can't converge in 30k steps)

Step 3: Commit

git add crates/ml/src/trainers/dqn/trainer.rs crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): constant-width [base,base] hidden layers, narrow range to [256,1024]"

---

Task 4.3: Default n_steps=3

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:478

// Old: (1.0, 5.0)    // 14: n_steps
(3.0, 5.0),            // 14: n_steps (raised minimum: n=3 is Rainbow standard, 3rd most impactful)

Step: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): raise n_steps minimum from 1 to 3 (Rainbow standard)"

---

Task 4.4: Phase 4 integration test

SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10
SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5

---

Phase 5: State Representation (4 tasks)

Task 5.1: Remove dead features

Files:

• Modify: crates/ml/src/features/extraction.rs:545 (regime features)
• Modify: crates/ml/src/features/extraction.rs:274 (OFI features)
• Modify: crates/ml/src/dqn/dqn.rs:205 (state_dim default)
• Modify: all files referencing state_dim: 54 or feature indices 40-50

Step 1: Remove regime features (indices 40-42)

In extract_regime_features_v2(), instead of writing 3 zeros, make it a no-op (don't write to output, reduce feature count by 3).

Step 2: Remove OFI features (indices 43-50)

In the OFI extraction, instead of zero-padding 8 features, skip them entirely when MBP-10 data is unavailable (which is always during hyperopt).

Step 3: Update state_dim

// Old: state_dim: 54
state_dim: 43, // 51 market - 3 regime - 8 OFI + 3 portfolio

Update all hardcoded references to 54.

Step 4: Fix normalize_with_skip indices

In crates/ml/src/trainers/dqn/statistics.rs, the skip indices [125, 126, 127] are remnants of the old 140-feature architecture. Update to match the new 43-feature layout or remove entirely (all features should be z-scored).

Step 5: Run tests — expect some failures from dimension changes

SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | grep -E "FAIL|test result"

Fix any tests that hardcode 54-dim or specific feature indices.

Step 6: Commit

git add crates/ml/src/features/extraction.rs crates/ml/src/dqn/dqn.rs crates/ml/src/trainers/dqn/statistics.rs
git commit -m "fix(ml): remove 11 dead features (regime+OFI), state_dim 54→43"

---

Task 5.2: Fix volume normalization

Files:

• Modify: crates/ml/src/features/extraction.rs:584 (volume)

// Old: safe_normalize(bar.volume, 0.0, 1_000_000.0)
// New: log-transform volume before normalizing (futures volume can be 10M+)
safe_normalize(bar.volume.max(1.0).ln(), 0.0, 18.0)  // ln(1)=0, ln(10M)≈16.1

Step: Commit

git add crates/ml/src/features/extraction.rs
git commit -m "fix(ml): log-transform volume normalization for futures scale"

---

Task 5.3: Fix ATR normalization

Files:

• Modify: crates/ml/src/features/extraction.rs:319 (ATR)

// Old: safe_normalize(indicators.last_atr, 0.0, 100.0)
// New: log-transform ATR (futures ATR ranges 0.5 to 5000+)
safe_normalize(indicators.last_atr.max(0.001).ln(), -7.0, 9.0)  // ln(0.001)≈-6.9, ln(5000)≈8.5

Step: Commit

git add crates/ml/src/features/extraction.rs
git commit -m "fix(ml): log-transform ATR normalization for futures scale"

---

Task 5.4: Center portfolio value feature

Files:

• Modify: crates/ml/src/dqn/portfolio_tracker.rs:162

// Old: portfolio_value / self.initial_capital  (centered on 1.0)
// New: center on 0.0 for consistency with z-scored market features
(portfolio_value / self.initial_capital) - 1.0

Step: Commit

git add crates/ml/src/dqn/portfolio_tracker.rs
git commit -m "fix(ml): center portfolio value feature at 0.0 (was 1.0)"

---

Phase 6: Hyperopt Objective (2 tasks)

Task 6.1: Increase Sharpe weight in composite objective

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:3380-3383 (tanh composite)

Find the tanh composite score lines and change weights:

// Old: 0.4 * sortino + 0.3 * calmar + 0.2 * sharpe + 0.1 * omega
// New: Sharpe-primary (it's what we actually care about)
let composite_score = 0.40 * (adjusted_sharpe / 2.0).tanh()
    + 0.25 * (sortino / 3.0).tanh()
    + 0.20 * (calmar / 5.0).tanh()
    + 0.15 * (omega / 2.0).tanh();

This makes Sharpe effective weight = 0.6 × 0.4 × S/2 = 12% (up from 6%).

Step: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(ml): increase Sharpe weight in hyperopt objective (6%→12% effective)"

---

Task 6.2: Final integration test and commit

Step 1: Run full test suite

SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -10

Expected: All tests pass (adjusted for state_dim changes).

Step 2: Run clippy

SQLX_OFFLINE=true cargo clippy --workspace 2>&1 | tail -10

Expected: 0 errors, 0 warnings.

Step 3: Run workspace check

SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5

Step 4: Final commit

git add -A
git commit -m "feat(ml): DQN hyperopt overhaul v2 — complete 6-phase fix

Phase 1: DSR reward function (replaces double-normalized pipeline)
Phase 2: Training steps (30 epochs, batch≤512, wired noisy_sigma)
Phase 3: Fixed IQN path, C51 v_min/v_max, capped CQL alpha
Phase 4: RMSNorm in dist-dueling, constant-width layers, n_steps≥3
Phase 5: Removed 11 dead features, log-scaled volume/ATR
Phase 6: Sharpe-primary objective weighting

Fixes 19 root causes identified by 6-agent deep audit.
Reduces hyperopt search space from 26D to ~24D.
Expected: Sharpe >> 0 on next hyperopt run."

---

Summary: All Changes by File

File	Changes
dqn/reward.rs	DSR struct, config fields, wired into calculate_reward, diversity scaled, reset methods
dqn/experience.rs	Precision 10000→1000000
dqn/noisy_layers.rs	Wire sigma_init parameter
dqn/dqn.rs	Fix IQN path, v_min/v_max defaults, state_dim 54→43, epsilon floor 0.02
dqn/distributional_dueling.rs	Add RMSNorm to shared backbone + streams
dqn/portfolio_tracker.rs	Center portfolio value at 0.0
trainers/dqn/trainer.rs	Skip risk-adjusted when DSR, reset portfolio/DSR per epoch, constant-width hidden layers, use_dsr config
trainers/dqn/config.rs	Add use_dsr, dsr_eta to DQNHyperparameters
hyperopt/adapters/dqn.rs	DSR eta tunable, remove wasted dims, batch≤512, n_steps≥3, CQL≤0.1, v_min/v_max scaled, gamma≤0.99, hidden_dim≤1024, Sharpe-primary objective, enable DSR+disable normalizers
features/extraction.rs	Remove regime+OFI features, log-transform volume/ATR
trainers/dqn/statistics.rs	Fix stale skip indices

Post-Implementation

1. Push to main
2. Wait for Argo CI (compile-services + compile-training)
3. Submit hyperopt v4 workflow: 20 trials × 30 epochs
4. Monitor via Prometheus: foxhunt_hyperopt_best_objective, foxhunt_training_epoch_sharpe