foxhunt/docs/dqn-hyperopt-2025-quick-ref.md

DQN Hyperopt 2025 Quick Reference

Status: B+ (85/100) - Production-ready but incomplete Critical Issues: 3 | Recommendations: 12

---

🔴 Critical Issues (Fix Immediately)

1. Learning Rate Range TOO NARROW

// CURRENT (Wave 19)
(2e-5, 8e-5)  // 4x range - EXCLUDES production default 1e-4!

// FIX (Restore Wave 17)
(1e-5, 3e-4)  // 30x range - includes all proven configs

Impact: May miss global optimum File: ml/src/hyperopt/adapters/dqn.rs:321

---

2. Ensemble Uncertainty NOT Integrated

// MISSING from DQNHyperparameters (6D):
pub use_ensemble_uncertainty: bool,
pub ensemble_size: usize,              // 3-7 agents
pub ensemble_beta_variance: f64,       // 0.0-1.0
pub ensemble_beta_disagreement: f64,   // 0.0-1.0
pub ensemble_beta_entropy: f64,        // 0.0-0.5
pub ensemble_variance_cap: f64,        // 1.0-10.0

Impact: Missing +15-25% sample efficiency (SOTA exploration) Files:

• ml/src/trainers/dqn/config.rs (add fields)
• ml/src/hyperopt/adapters/dqn.rs (expand search space)

---

3. C51 Distributional RL DISABLED (BUG #36)

// WASTED search space (3D):
v_min: (-3.0, -1.0),      // UNUSED
v_max: (1.0, 3.0),        // UNUSED
num_atoms: (51, 201),     // UNUSED

Root Cause: Candle scatter_add breaks gradient flow Impact: 3D wasted, missing +15-25% expected boost Action: Remove from search OR fix Candle library

---

🟡 High Priority (Next Sprint)

4. Network Architecture Search (Missing 3D)

// ADD to search space:
pub hidden_dim_1: usize,  // 128-512 (step=64)
pub hidden_dim_2: usize,  // 64-256 (step=32)
pub hidden_dim_3: usize,  // 32-128 (step=32)

Current: Hardcoded [256, 128, 64] Impact: +5-10% via capacity tuning

---

5. Dueling Architecture Range TOO NARROW

// CURRENT
(128, 512)  // Only 4 discrete values

// FIX
(64, 1024)  // 16 discrete values (step=64)

Impact: +3-5% better architecture matching

---

Current Search Space (22D)

✅ Well-Tuned (16D)

Category	Params	Status
Base	batch_size, gamma, buffer_size, huber_delta, entropy, tx_cost	✅ Optimal
PER	per_alpha, per_beta_start	✅ Rainbow standard
Risk	kelly_fractional, kelly_max_fraction, kelly_min_trades, volatility_window	✅ Comprehensive
Misc	minimum_profit_factor	✅ Slippage protection

🟡 Suboptimal (3D)

Param	Range	Issue
learning_rate	2e-5 to 8e-5	🔴 TOO NARROW
hold_penalty_weight	1.0 to 2.0	🟡 Was [0.5, 5.0]
max_position_absolute	4.0 to 8.0	🟡 Was [1.0, 10.0]

🔴 Broken/Unused (3D)

Param	Status	Action
v_min, v_max, num_atoms	UNUSED (BUG #36)	Remove OR fix C51

---

Recommended 2025 Configuration

Immediate Changes (22D → 28D)

+ Expand learning_rate: [1e-5, 3e-4]
+ Add ensemble_uncertainty: 6D (use, size, beta_var, beta_dis, beta_ent, var_cap)
- Remove C51 params: -3D (v_min, v_max, num_atoms)
= Net: +3D (22D → 25D)

Next Sprint (25D → 31D)

+ Add network_architecture: 3D (hidden_dim_1, hidden_dim_2, hidden_dim_3)
+ Expand dueling_hidden_dim: [64, 1024] (from [128, 512])
+ Add warmup_ratio: 1D (0.0-0.1)
= Net: +4D (25D → 29D)

Full 2025 Target: 34D

• Base (11D): ✅ + expanded LR
• Rainbow (3D): ✅ - removed C51
• Risk (4D): ✅
• Misc (1D): ✅
• Ensemble (6D): NEW
• Architecture (3D): NEW
• Warmup (1D): NEW
• Portfolio (1D): NEW (risk aversion)
• Exploration (4D): OPTIONAL (Noisy preferred)

---

Performance Impact Estimates

Enhancement	Complexity	Expected Boost	Priority
Ensemble uncertainty (6D)	Medium	+15-25%	🔴 Critical
Fix LR range	Trivial	+3-5%	🔴 Critical
Architecture search (3D)	Low	+5-10%	🟡 High
Fix C51 (BUG #36)	High	+15-25%	🟡 High
Expand dueling range	Trivial	+3-5%	🟡 High
Warmup ratio tuning	Low	+2-5%	🟢 Medium

Total Potential: +40-75% improvement over current 22D config

---

Implementation Checklist

Week 1: Critical Fixes

• Expand LR range to [1e-5, 3e-4]
• Add 6D ensemble params to DQNHyperparameters
• Integrate ensemble into hyperopt search space
• Remove v_min/v_max/num_atoms (BUG #36)
• Update DQNParams::from_continuous() (22D → 25D)

Week 2: Architecture Search

• Add 3D network architecture params
• Expand dueling range to [64, 1024]
• Update DQN trainer to use tunable dims

Week 3: Validation

• Run 50-trial hyperopt (25D space)
• Compare Sharpe: old vs new
• Document best params in dqn_best_2025.toml

---

Files to Modify

1. Add Ensemble Fields

File: ml/src/trainers/dqn/config.rs Line: ~450 (after gradient_collapse_patience)

// WAVE XX: Ensemble Uncertainty Integration
pub use_ensemble_uncertainty: bool,
pub ensemble_size: usize,
pub ensemble_beta_variance: f64,
pub ensemble_beta_disagreement: f64,
pub ensemble_beta_entropy: f64,
pub ensemble_variance_cap: f64,

2. Expand Hyperopt Search Space

File: ml/src/hyperopt/adapters/dqn.rs Line: 319-353 (continuous_bounds())

vec![
    // Fix LR range
    (1e-5_f64.ln(), 3e-4_f64.ln()),  // Was: (2e-5, 8e-5)

    // ... existing 11D base params ...

    // Remove C51 (BUG #36)
    // (deleted v_min, v_max, num_atoms)

    // Add Ensemble (6D NEW)
    (0.0, 1.0),        // use_ensemble_uncertainty (bool)
    (3.0, 7.0),        // ensemble_size
    (0.0, 1.0),        // ensemble_beta_variance
    (0.0, 1.0),        // ensemble_beta_disagreement
    (0.0, 0.5),        // ensemble_beta_entropy
    (1.0, 10.0),       // ensemble_variance_cap
]

3. Update Parameter Conversion

File: ml/src/hyperopt/adapters/dqn.rs Line: 356-455 (from_continuous())

fn from_continuous(x: &[f64]) -> Result<Self, MLError> {
    if x.len() != 25 {  // Was: 22
        return Err(MLError::ConfigError {
            reason: format!("Expected 25 params (removed C51, added ensemble), got {}", x.len()),
        });
    }

    // ... existing base params ...

    // Ensemble params (NEW)
    let use_ensemble_uncertainty = x[19] > 0.5;  // Boolean threshold
    let ensemble_size = x[20].round().clamp(3.0, 7.0) as usize;
    let ensemble_beta_variance = x[21].clamp(0.0, 1.0);
    let ensemble_beta_disagreement = x[22].clamp(0.0, 1.0);
    let ensemble_beta_entropy = x[23].clamp(0.0, 0.5);
    let ensemble_variance_cap = x[24].clamp(1.0, 10.0);

    Ok(Self {
        // ... existing fields ...
        use_ensemble_uncertainty,
        ensemble_size,
        ensemble_beta_variance,
        ensemble_beta_disagreement,
        ensemble_beta_entropy,
        ensemble_variance_cap,
    })
}

---

Rainbow DQN Component Status

Component	Status	Hyperopt	2025 Grade
Double DQN	✅ Enabled	Hardcoded	✅ A
Dueling	✅ Enabled	dueling_dim (128-512)	🟡 B (expand to 64-1024)
PER	✅ Enabled	per_alpha, per_beta	✅ A
N-Step	✅ Enabled	n_steps (1-5)	✅ A
Noisy Nets	✅ Enabled	noisy_sigma (0.1-1.0)	✅ A
C51	🔴 DISABLED	UNUSED	🔴 F (BUG #36)

Overall Rainbow Grade: B+ (5/6 enabled, 1/6 blocked by bug)

---

Key Metrics

Current Performance

• Search Space: 22D continuous
• Training Success: 60% (40% fail Epoch 2 from C51 bug)
• Best Sharpe: 0.77-2.0 (standard DQN, no C51)
• Sample Efficiency: Baseline

2025 Target

• Search Space: 34D continuous (ensemble + architecture + C51 fix)
• Training Success: 95%+ (C51 fixed or removed)
• Best Sharpe: 1.5-3.0 (+50% target)
• Sample Efficiency: +40% (ensemble exploration)

---

Quick Commands

Check current hyperopt config

grep "continuous_bounds" ml/src/hyperopt/adapters/dqn.rs -A50

Verify ensemble module exists

ls -lh ml/src/dqn/ensemble_uncertainty.rs

Run hyperopt with new config (after fixes)

cargo run --release --bin ml_training_service -- \
  dqn-hyperopt \
  --data-dir test_data/real/databento/ml_training/ \
  --epochs 100 \
  --trials 50 \
  --run-name "2025-ensemble-integration"

---

Last Updated: 2025-11-27 Owner: ML Team Priority: 🔴 Critical (implement Week 1 fixes immediately)