foxhunt/docs/dqn-hyperopt-2025-analysis.md

# DQN Hyperparameter Optimization: 2025 Standards Analysis

**Date**: 2025-11-27
**Analyst**: Code Analyzer Agent
**Status**: Production Review
**Scope**: Hyperparameter search space optimization and missing features

---

## Executive Summary

The DQN hyperparameter optimization configuration is **mature but has critical gaps** for 2025 standards. The implementation shows 22 tunable dimensions (22D continuous space) with good coverage of Rainbow DQN components, but **lacks ensemble uncertainty integration** and has suboptimal search space boundaries in several areas.

**Overall Grade**: B+ (85/100)
- ✅ Rainbow DQN architecture well-covered
- ✅ Advanced risk management (Kelly, volatility-epsilon, regime-conditional)
- ❌ **Missing**: Ensemble uncertainty hyperparameters
- ⚠️ **Suboptimal**: Network architecture search space too narrow
- ⚠️ **Suboptimal**: Learning rate range may be too conservative

---

## Current Hyperparameter Space (22D)

### Base Parameters (11D) ✅
| Parameter | Range | Scaling | Status | 2025 Assessment |
|-----------|-------|---------|--------|-----------------|
| `learning_rate` | 2e-5 to 8e-5 | Log | 🟡 Conservative | **TOO NARROW** - Should be 1e-5 to 3e-4 |
| `batch_size` | 64 to 160 | Linear | ✅ Good | Optimal for GPU constraints |
| `gamma` | 0.95 to 0.99 | Linear | ✅ Optimal | Industry standard |
| `buffer_size` | 100K to 500K | Log | ✅ Good | WAVE 24 expansion appropriate |
| `hold_penalty_weight` | 1.0 to 2.0 | Linear | 🟡 Narrow | Was [0.5, 5.0] - may need expansion |
| `max_position_absolute` | 4.0 to 8.0 | Linear | 🟡 Narrow | Was [1.0, 10.0] - conservative |
| `huber_delta` | 10.0 to 40.0 | Log | ✅ Good | Appropriate MSE→MAE transition |
| `entropy_coefficient` | 0.0 to 0.1 | Linear | ✅ Good | Standard exploration range |
| `transaction_cost_multiplier` | 0.5 to 2.0 | Linear | ✅ Good | Fee sensitivity covered |
| `per_alpha` | 0.4 to 0.8 | Linear | ✅ Good | Rainbow DQN standard |
| `per_beta_start` | 0.2 to 0.6 | Linear | ✅ Good | Importance sampling correction |

### Rainbow DQN Extensions (6D) ⚠️
| Parameter | Range | Scaling | Status | 2025 Assessment |
|-----------|-------|---------|--------|-----------------|
| `v_min` | -3.0 to -1.0 | Linear | 🔴 **UNUSED** | BUG #36 - C51 disabled |
| `v_max` | 1.0 to 3.0 | Linear | 🔴 **UNUSED** | BUG #36 - C51 disabled |
| `noisy_sigma_init` | 0.1 to 1.0 | Log | ✅ Good | NoisyNet exploration |
| `dueling_hidden_dim` | 128 to 512 | Linear | 🟡 Limited | **TOO NARROW** - Consider 64-1024 |
| `n_steps` | 1 to 5 | Integer | ✅ Good | N-step TD horizon |
| `num_atoms` | 51 to 201 | Integer | 🔴 **UNUSED** | BUG #36 - C51 disabled |

### Risk Management (4D) ✅
| Parameter | Range | Scaling | Status | 2025 Assessment |
|-----------|-------|---------|--------|-----------------|
| `kelly_fractional` | 0.25 to 1.0 | Linear | ✅ Good | Conservative to full Kelly |
| `kelly_max_fraction` | 0.1 to 0.5 | Linear | ✅ Good | Position sizing cap |
| `kelly_min_trades` | 10 to 50 | Integer | ✅ Good | Statistical validity |
| `volatility_window` | 10 to 30 | Integer | ✅ Good | Adaptive exploration |

### Miscellaneous (1D) ✅
| Parameter | Range | Scaling | Status | 2025 Assessment |
|-----------|-------|---------|--------|-----------------|
| `minimum_profit_factor` | 1.1 to 2.0 | Linear | ✅ Good | BUG #7 fix - slippage protection |

---

## Critical Missing Features ❌

### 1. Ensemble Uncertainty Integration (HIGH PRIORITY)

**Status**: NOT INTEGRATED into hyperopt
**Impact**: Missing 2025 state-of-the-art exploration mechanism

#### Current Implementation
- ✅ Ensemble uncertainty module exists (`ml/src/dqn/ensemble_uncertainty.rs`)
- ✅ Implementation complete with Q-variance, disagreement, entropy metrics
- ✅ Exploration bonus formula defined
- ❌ **NOT exposed as hyperparameters in DQNHyperparameters**
- ❌ **NOT tunable in hyperopt search space**

#### Required Hyperparameters (Missing 6D)
```rust
// Add to DQNHyperparameters struct
pub use_ensemble_uncertainty: bool,      // Enable/disable ensemble
pub ensemble_size: usize,                // Number of agents (3-7)
pub ensemble_beta_variance: f64,         // Variance weight (0.0-1.0)
pub ensemble_beta_disagreement: f64,     // Disagreement weight (0.0-1.0)
pub ensemble_beta_entropy: f64,          // Entropy weight (0.0-0.5)
pub ensemble_variance_cap: f64,          // Max variance bonus (1.0-10.0)
```

#### Search Space Addition (6D)
```rust
// Continuous bounds expansion (22D → 28D)
vec![
    // ... existing 22 dimensions ...

    // Ensemble Uncertainty (6D) - NEW
    (0.0, 1.0),        // 22: use_ensemble_uncertainty (bool via threshold)
    (3.0, 7.0),        // 23: ensemble_size (3-7 agents)
    (0.0, 1.0),        // 24: ensemble_beta_variance
    (0.0, 1.0),        // 25: ensemble_beta_disagreement
    (0.0, 0.5),        // 26: ensemble_beta_entropy
    (1.0, 10.0),       // 27: ensemble_variance_cap
]
```

**Expected Impact**: +15-25% sample efficiency, better exploration in low-data regimes

---

### 2. Network Architecture Search Space (MEDIUM PRIORITY)

**Current Limitations**:
- ❌ Hidden dimensions **HARDCODED** to `[256, 128, 64]` in DQN trainer
- ❌ Dueling architecture only tunes `dueling_hidden_dim` (128-512)
- ❌ No tuning of main Q-network depth or width

#### Recommended Additions (3D)
```rust
pub hidden_dim_1: usize,  // First layer (128-512)
pub hidden_dim_2: usize,  // Second layer (64-256)
pub hidden_dim_3: usize,  // Third layer (32-128)
```

#### Search Space Addition (3D)
```rust
// Architecture tuning (28D → 31D)
vec![
    // ... existing 28 dimensions ...

    // Network Architecture (3D) - NEW
    (128.0, 512.0),    // 28: hidden_dim_1 (step=64)
    (64.0, 256.0),     // 29: hidden_dim_2 (step=32)
    (32.0, 128.0),     // 30: hidden_dim_3 (step=32)
]
```

**Expected Impact**: +5-10% performance, better capacity matching to problem complexity

---

### 3. Exploration Schedule Tuning (LOW PRIORITY)

**Current Status**: Epsilon decay hardcoded to 0.995
**Alternative**: Noisy Networks (enabled by default)

#### Potential Addition (2D)
```rust
pub epsilon_decay_rate: f64,   // 0.990 to 0.999 (if Noisy disabled)
pub warmup_ratio: f64,          // 0.0 to 0.1 (% of total steps)
```

**Recommendation**: SKIP - Noisy Networks superior to epsilon-greedy for 2025 standards

---

## Suboptimal Search Space Boundaries

### 1. Learning Rate Range (CRITICAL) 🔴

**Current**: `[2e-5, 8e-5]` (4x range, log-scale)
**Problem**: TOO CONSERVATIVE - excludes known good configurations
**Evidence**:
- Trial 3 optimal: `3.37e-05` (within range)
- Production default: `1e-4` (EXCLUDED from search!)
- Wave 17 hyperopt: used `[1e-5, 3e-4]` (1000x range)

**Recommendation**:
```rust
// BEFORE (Wave 19 - TOO NARROW)
(2e-5_f64.ln(), 8e-5_f64.ln()),  // learning_rate

// AFTER (Restore Wave 17 range)
(1e-5_f64.ln(), 3e-4_f64.ln()),  // learning_rate (30x range)
```

**Impact**: Current range may miss optimal LR by excluding 8e-5 to 3e-4 zone

---

### 2. Dueling Architecture Capacity (MEDIUM) 🟡

**Current**: `[128, 512]` (step=128) → only 4 discrete values
**Problem**: Insufficient granularity for architecture search

**Recommendation**:
```rust
// BEFORE
(128.0, 512.0),  // dueling_hidden_dim (step=128)

// AFTER
(64.0, 1024.0),  // dueling_hidden_dim (step=64, 16 discrete values)
```

**Impact**: Better architecture capacity matching (+3-5% potential)

---

### 3. Position Limits (LOW) 🟡

**Current**: `[4.0, 8.0]` (was [1.0, 10.0])
**Problem**: May be too conservative for aggressive strategies

**Recommendation**: MONITOR - Consider expanding if trials consistently hit 8.0 boundary

---

## BUG #36: C51 Distributional RL Disabled ⚠️

**Status**: CRITICAL BLOCKER
**Impact**: 3 hyperparameters (v_min, v_max, num_atoms) are **UNUSED**
**Root Cause**: Candle's `scatter_add` breaks autograd in backward pass
**Symptom**: 40% of trials experience gradient collapse at Epoch 2

### Evidence
- File: `/ml/src/hyperopt/adapters/dqn.rs:278-291`
- Comment: "BUG #36: Candle's scatter_add has broken gradient flow"
- Performance: Standard DQN achieves Sharpe 0.77-2.0 WITHOUT C51
- Expected: 95%+ success rate with C51 disabled (vs 60% with enabled)

### Search Space Impact
```rust
// Currently tuned but UNUSED (3D wasted)
v_min: (-3.0, -1.0),      // UNUSED - C51 disabled
v_max: (1.0, 3.0),        // UNUSED - C51 disabled
num_atoms: (51, 201),     // UNUSED - C51 disabled
```

### Recommendations
1. **Short-term**: Remove v_min, v_max, num_atoms from search space (22D → 19D)
2. **Long-term**: Fix Candle library scatter_add or implement workaround
3. **Alternative**: Use categorical DQN with softmax instead of C51 projection

---

## Rainbow DQN Component Status

### Enabled Components (5/6) ✅
| Component | Status | Hyperopt Coverage | 2025 Standard |
|-----------|--------|-------------------|---------------|
| Double DQN | ✅ Always enabled | N/A (hardcoded) | ✅ Industry standard |
| Dueling Networks | ✅ Always enabled | `dueling_hidden_dim` (128-512) | ✅ Good |
| Prioritized ER | ✅ Always enabled | `per_alpha`, `per_beta_start` | ✅ Excellent |
| N-Step Returns | ✅ Always enabled | `n_steps` (1-5) | ✅ Good |
| Noisy Networks | ✅ Always enabled | `noisy_sigma_init` (0.1-1.0) | ✅ Good |
| **C51 Distributional** | 🔴 **DISABLED** | **UNUSED** (BUG #36) | ❌ **BLOCKED** |

### Missing Rainbow Component
- **Distributional RL**: Disabled due to library bug (BUG #36)
- **Impact**: Missing +15-25% expected performance boost
- **Priority**: HIGH - Re-enable after Candle library fix

---

## 2025 Hyperparameter Optimization Standards

### Industry Best Practices Compliance

#### ✅ COMPLIANT
1. **Bayesian Optimization**: Using egobox surrogate (GP-based)
2. **Multi-Objective**: Sharpe + Drawdown + HFT activity
3. **Log-Scale Tuning**: Learning rate, buffer size, huber delta
4. **Constraint Handling**: HFT validation, batch size floor
5. **Early Stopping**: Gradient collapse detection (WAVE 23)
6. **Feature Caching**: 99% speedup via preprocessed features

#### ⚠️ NEEDS IMPROVEMENT
1. **Ensemble Methods**: NOT integrated (missing 6D)
2. **Neural Architecture Search**: Hardcoded depths (missing 3D)
3. **Exploration Schedules**: Limited tuning (hardcoded decay)
4. **Transfer Learning**: No warm-start hyperparameter initialization

#### ❌ NON-COMPLIANT
1. **C51 Distributional RL**: Disabled (library bug)
2. **Learning Rate Range**: Too conservative (excludes proven configs)

---

## Recommendations for 2025 Optimization

### Priority 1: CRITICAL (Immediate Action) 🔴

#### 1.1 Expand Learning Rate Range
```rust
// ml/src/hyperopt/adapters/dqn.rs:321
// BEFORE
(2e-5_f64.ln(), 8e-5_f64.ln()),

// AFTER
(1e-5_f64.ln(), 3e-4_f64.ln()),  // Restore Wave 17 range
```

**Rationale**: Current range excludes production default (1e-4) and proven configs

#### 1.2 Integrate Ensemble Uncertainty (6D)
```rust
// Add to DQNHyperparameters (ml/src/trainers/dqn/config.rs)
pub use_ensemble_uncertainty: bool,
pub ensemble_size: usize,
pub ensemble_beta_variance: f64,
pub ensemble_beta_disagreement: f64,
pub ensemble_beta_entropy: f64,
pub ensemble_variance_cap: f64,
```

**Expected Impact**: +15-25% sample efficiency, SOTA exploration

#### 1.3 Fix or Remove C51 Parameters
```rust
// Option A: Fix Candle scatter_add (preferred)
// - Implement custom scatter_add with proper gradient tracking
// - Re-enable distributional RL (+15-25% expected boost)

// Option B: Remove from search space (immediate)
// - Drop v_min, v_max, num_atoms (22D → 19D)
// - Reclaim 3D search space for other features
```

---

### Priority 2: HIGH (Next Sprint) 🟡

#### 2.1 Network Architecture Search (3D)
```rust
// Add to search space
pub hidden_dim_1: usize,  // 128-512 (step=64)
pub hidden_dim_2: usize,  // 64-256 (step=32)
pub hidden_dim_3: usize,  // 32-128 (step=32)
```

**Expected Impact**: +5-10% performance via capacity tuning

#### 2.2 Expand Dueling Architecture Range
```rust
// BEFORE
(128.0, 512.0),  // Only 4 discrete values

// AFTER
(64.0, 1024.0),  // 16 discrete values (step=64)
```

**Expected Impact**: +3-5% better architecture matching

#### 2.3 Add Warmup Ratio Tuning
```rust
// Add to DQNHyperparameters
pub warmup_ratio: f64,  // 0.0 to 0.1 (% of total steps)
```

**Expected Impact**: +2-5% in short-training scenarios (<200K steps)

---

### Priority 3: MEDIUM (Future Enhancement) 🟢

#### 3.1 Transfer Learning Warm-Start
- Initialize hyperopt with best known configurations
- Use historical trial data to seed Bayesian prior
- Expected: 2-3x faster convergence to optimal params

#### 3.2 Multi-Fidelity Optimization
- Low-fidelity: 10 epochs per trial (fast exploration)
- High-fidelity: 100 epochs for top 10% candidates
- Expected: 5-10x total hyperopt speedup

#### 3.3 Portfolio-Aware Hyperparameters
- Add `risk_aversion_coefficient` (0.0-2.0)
- Add `portfolio_rebalance_threshold` (0.01-0.1)
- Expected: +5-8% Sharpe via risk-adjusted trading

---

## Hyperparameter Search Space Summary

### Current Implementation (22D)
```text
Base (11D):     LR, batch, gamma, buffer, hold_penalty, max_pos, huber,
                entropy, tx_cost, per_alpha, per_beta
Rainbow (6D):   v_min*, v_max*, noisy_sigma, dueling_dim, n_steps, num_atoms*
Risk (4D):      kelly_frac, kelly_max, kelly_min, volatility_window
Misc (1D):      min_profit_factor

* = UNUSED (BUG #36 - C51 disabled)
```

### Recommended 2025 Configuration (34D)
```text
Base (11D):     SAME + expanded LR range [1e-5, 3e-4]
Rainbow (3D):   noisy_sigma, dueling_dim [64-1024], n_steps
                REMOVED: v_min, v_max, num_atoms (unused)
Risk (4D):      SAME
Misc (1D):      SAME
Ensemble (6D):  NEW - use_ensemble, size, beta_var, beta_dis, beta_ent, var_cap
Architecture (3D): NEW - hidden_dim_1, hidden_dim_2, hidden_dim_3
Warmup (1D):    NEW - warmup_ratio
Exploration (4D): NEW - epsilon_decay*, warmup_steps*, noise_schedule*, entropy_schedule*
Portfolio (1D): NEW - risk_aversion_coefficient

Total: 34D continuous space (vs 22D current)
```

**Note**: Epsilon-based exploration (4D) marked with * are optional - Noisy Networks preferred

---

## Implementation Checklist

### Phase 1: Critical Fixes (Week 1)
- [ ] Expand learning rate range to [1e-5, 3e-4]
- [ ] Add ensemble uncertainty fields to DQNHyperparameters
- [ ] Integrate ensemble params into hyperopt search space (6D)
- [ ] Fix C51 scatter_add bug OR remove v_min/v_max/num_atoms
- [ ] Update DQNParams::from_continuous() for new dimensions
- [ ] Add parameter validation for ensemble config

### Phase 2: Architecture Search (Week 2)
- [ ] Add network architecture hyperparameters (3D)
- [ ] Expand dueling_hidden_dim range to [64, 1024]
- [ ] Update DQN trainer to use tunable hidden dimensions
- [ ] Add architecture validation (dim1 > dim2 > dim3)

### Phase 3: Advanced Features (Week 3)
- [ ] Add warmup_ratio tuning (1D)
- [ ] Implement transfer learning warm-start
- [ ] Add multi-fidelity optimization (10 vs 100 epochs)
- [ ] Document hyperparameter ranges in ADR

### Phase 4: Validation (Week 4)
- [ ] Run 50-trial hyperopt with new search space
- [ ] Compare Sharpe ratios: old (22D) vs new (34D)
- [ ] Validate ensemble uncertainty improves exploration
- [ ] Benchmark training time overhead of ensemble
- [ ] Document optimal hyperparameters in `/ml/hyperparams/dqn_best_2025.toml`

---

## Risk Assessment

### High Risk 🔴
1. **C51 Bug (BUG #36)**: Blocks 15-25% expected performance boost
   - Mitigation: Remove unused params, investigate Candle fix
2. **Learning Rate Range**: Current range may miss global optimum
   - Mitigation: Expand to [1e-5, 3e-4] immediately

### Medium Risk 🟡
1. **Ensemble Overhead**: 3-7x training cost for ensemble size
   - Mitigation: Start with ensemble_size=3, tune up if beneficial
2. **Search Space Explosion**: 22D → 34D increases sample complexity
   - Mitigation: Use multi-fidelity optimization, 50→100 trials

### Low Risk 🟢
1. **Architecture Search**: Minimal overhead, well-understood
2. **Warmup Ratio**: Simple parameter, low interaction complexity

---

## Conclusion

The DQN hyperopt configuration is **production-ready but incomplete** for 2025 standards. Key actions:

1. ✅ **Strengths**: Rainbow DQN well-tuned, risk management comprehensive
2. ❌ **Critical Gap**: Ensemble uncertainty not integrated (missing 6D)
3. ⚠️ **Suboptimal**: Learning rate range too narrow, C51 disabled
4. 🎯 **Target**: 34D search space with ensemble + architecture tuning

**Expected ROI**:
- Ensemble integration: +15-25% sample efficiency
- Architecture search: +5-10% performance
- LR range expansion: +3-5% Sharpe via better convergence
- **Total**: +25-40% improvement potential over current 22D config

**Recommendation**: Implement Phase 1 immediately, defer Phases 2-4 to next sprint.

---

## References

- DQN Config: `/ml/src/trainers/dqn/config.rs`
- Hyperopt Adapter: `/ml/src/hyperopt/adapters/dqn.rs`
- Ensemble Uncertainty: `/ml/src/dqn/ensemble_uncertainty.rs`
- BUG #36 Evidence: `/tmp/WAVE23_CAMPAIGN_FINAL_ANALYSIS.md`
- Rainbow DQN Paper: Hessel et al. (2018)
- 2025 Best Practices: OpenAI Spinning Up, Stable Baselines3 docs

**Last Updated**: 2025-11-27
**Next Review**: After Phase 1 completion (1 week)