# DQN Hyperopt 2025 Quick Reference

**Status**: B+ (85/100) - Production-ready but incomplete
**Critical Issues**: 3 | Recommendations: 12

---

## 🔴 Critical Issues (Fix Immediately)

### 1. Learning Rate Range TOO NARROW
```rust
// CURRENT (Wave 19)
(2e-5, 8e-5)  // 4x range - EXCLUDES production default 1e-4!

// FIX (Restore Wave 17)
(1e-5, 3e-4)  // 30x range - includes all proven configs
```
**Impact**: May miss global optimum
**File**: `ml/src/hyperopt/adapters/dqn.rs:321`

---

### 2. Ensemble Uncertainty NOT Integrated
```rust
// MISSING from DQNHyperparameters (6D):
pub use_ensemble_uncertainty: bool,
pub ensemble_size: usize,              // 3-7 agents
pub ensemble_beta_variance: f64,       // 0.0-1.0
pub ensemble_beta_disagreement: f64,   // 0.0-1.0
pub ensemble_beta_entropy: f64,        // 0.0-0.5
pub ensemble_variance_cap: f64,        // 1.0-10.0
```
**Impact**: Missing +15-25% sample efficiency (SOTA exploration)
**Files**:
- `ml/src/trainers/dqn/config.rs` (add fields)
- `ml/src/hyperopt/adapters/dqn.rs` (expand search space)

---

### 3. C51 Distributional RL DISABLED (BUG #36)
```rust
// WASTED search space (3D):
v_min: (-3.0, -1.0),      // UNUSED
v_max: (1.0, 3.0),        // UNUSED
num_atoms: (51, 201),     // UNUSED
```
**Root Cause**: Candle `scatter_add` breaks gradient flow
**Impact**: 3D wasted, missing +15-25% expected boost
**Action**: Remove from search OR fix Candle library

---

## 🟡 High Priority (Next Sprint)

### 4. Network Architecture Search (Missing 3D)
```rust
// ADD to search space:
pub hidden_dim_1: usize,  // 128-512 (step=64)
pub hidden_dim_2: usize,  // 64-256 (step=32)
pub hidden_dim_3: usize,  // 32-128 (step=32)
```
**Current**: Hardcoded `[256, 128, 64]`
**Impact**: +5-10% via capacity tuning

---

### 5. Dueling Architecture Range TOO NARROW
```rust
// CURRENT
(128, 512)  // Only 4 discrete values

// FIX
(64, 1024)  // 16 discrete values (step=64)
```
**Impact**: +3-5% better architecture matching

---

## Current Search Space (22D)

### ✅ Well-Tuned (16D)
| Category | Params | Status |
|----------|--------|--------|
| **Base** | `batch_size`, `gamma`, `buffer_size`, `huber_delta`, `entropy`, `tx_cost` | ✅ Optimal |
| **PER** | `per_alpha`, `per_beta_start` | ✅ Rainbow standard |
| **Risk** | `kelly_fractional`, `kelly_max_fraction`, `kelly_min_trades`, `volatility_window` | ✅ Comprehensive |
| **Misc** | `minimum_profit_factor` | ✅ Slippage protection |

### 🟡 Suboptimal (3D)
| Param | Range | Issue |
|-------|-------|-------|
| `learning_rate` | 2e-5 to 8e-5 | 🔴 TOO NARROW |
| `hold_penalty_weight` | 1.0 to 2.0 | 🟡 Was [0.5, 5.0] |
| `max_position_absolute` | 4.0 to 8.0 | 🟡 Was [1.0, 10.0] |

### 🔴 Broken/Unused (3D)
| Param | Status | Action |
|-------|--------|--------|
| `v_min`, `v_max`, `num_atoms` | UNUSED (BUG #36) | Remove OR fix C51 |

---

## Recommended 2025 Configuration

### Immediate Changes (22D → 28D)
```diff
+ Expand learning_rate: [1e-5, 3e-4]
+ Add ensemble_uncertainty: 6D (use, size, beta_var, beta_dis, beta_ent, var_cap)
- Remove C51 params: -3D (v_min, v_max, num_atoms)
= Net: +3D (22D → 25D)
```

### Next Sprint (25D → 31D)
```diff
+ Add network_architecture: 3D (hidden_dim_1, hidden_dim_2, hidden_dim_3)
+ Expand dueling_hidden_dim: [64, 1024] (from [128, 512])
+ Add warmup_ratio: 1D (0.0-0.1)
= Net: +4D (25D → 29D)
```

### Full 2025 Target: 34D
- Base (11D): ✅ + expanded LR
- Rainbow (3D): ✅ - removed C51
- Risk (4D): ✅
- Misc (1D): ✅
- **Ensemble (6D)**: NEW
- **Architecture (3D)**: NEW
- **Warmup (1D)**: NEW
- **Portfolio (1D)**: NEW (risk aversion)
- **Exploration (4D)**: OPTIONAL (Noisy preferred)

---

## Performance Impact Estimates

| Enhancement | Complexity | Expected Boost | Priority |
|-------------|------------|----------------|----------|
| Ensemble uncertainty (6D) | Medium | +15-25% | 🔴 Critical |
| Fix LR range | Trivial | +3-5% | 🔴 Critical |
| Architecture search (3D) | Low | +5-10% | 🟡 High |
| Fix C51 (BUG #36) | High | +15-25% | 🟡 High |
| Expand dueling range | Trivial | +3-5% | 🟡 High |
| Warmup ratio tuning | Low | +2-5% | 🟢 Medium |

**Total Potential**: +40-75% improvement over current 22D config

---

## Implementation Checklist

### Week 1: Critical Fixes
- [ ] Expand LR range to [1e-5, 3e-4]
- [ ] Add 6D ensemble params to `DQNHyperparameters`
- [ ] Integrate ensemble into hyperopt search space
- [ ] Remove v_min/v_max/num_atoms (BUG #36)
- [ ] Update `DQNParams::from_continuous()` (22D → 25D)

### Week 2: Architecture Search
- [ ] Add 3D network architecture params
- [ ] Expand dueling range to [64, 1024]
- [ ] Update DQN trainer to use tunable dims

### Week 3: Validation
- [ ] Run 50-trial hyperopt (25D space)
- [ ] Compare Sharpe: old vs new
- [ ] Document best params in `dqn_best_2025.toml`

---

## Files to Modify

### 1. Add Ensemble Fields
**File**: `ml/src/trainers/dqn/config.rs`
**Line**: ~450 (after `gradient_collapse_patience`)
```rust
// WAVE XX: Ensemble Uncertainty Integration
pub use_ensemble_uncertainty: bool,
pub ensemble_size: usize,
pub ensemble_beta_variance: f64,
pub ensemble_beta_disagreement: f64,
pub ensemble_beta_entropy: f64,
pub ensemble_variance_cap: f64,
```

### 2. Expand Hyperopt Search Space
**File**: `ml/src/hyperopt/adapters/dqn.rs`
**Line**: 319-353 (`continuous_bounds()`)
```rust
vec![
    // Fix LR range
    (1e-5_f64.ln(), 3e-4_f64.ln()),  // Was: (2e-5, 8e-5)

    // ... existing 11D base params ...

    // Remove C51 (BUG #36)
    // (deleted v_min, v_max, num_atoms)

    // Add Ensemble (6D NEW)
    (0.0, 1.0),        // use_ensemble_uncertainty (bool)
    (3.0, 7.0),        // ensemble_size
    (0.0, 1.0),        // ensemble_beta_variance
    (0.0, 1.0),        // ensemble_beta_disagreement
    (0.0, 0.5),        // ensemble_beta_entropy
    (1.0, 10.0),       // ensemble_variance_cap
]
```

### 3. Update Parameter Conversion
**File**: `ml/src/hyperopt/adapters/dqn.rs`
**Line**: 356-455 (`from_continuous()`)
```rust
fn from_continuous(x: &[f64]) -> Result<Self, MLError> {
    if x.len() != 25 {  // Was: 22
        return Err(MLError::ConfigError {
            reason: format!("Expected 25 params (removed C51, added ensemble), got {}", x.len()),
        });
    }

    // ... existing base params ...

    // Ensemble params (NEW)
    let use_ensemble_uncertainty = x[19] > 0.5;  // Boolean threshold
    let ensemble_size = x[20].round().clamp(3.0, 7.0) as usize;
    let ensemble_beta_variance = x[21].clamp(0.0, 1.0);
    let ensemble_beta_disagreement = x[22].clamp(0.0, 1.0);
    let ensemble_beta_entropy = x[23].clamp(0.0, 0.5);
    let ensemble_variance_cap = x[24].clamp(1.0, 10.0);

    Ok(Self {
        // ... existing fields ...
        use_ensemble_uncertainty,
        ensemble_size,
        ensemble_beta_variance,
        ensemble_beta_disagreement,
        ensemble_beta_entropy,
        ensemble_variance_cap,
    })
}
```

---

## Rainbow DQN Component Status

| Component | Status | Hyperopt | 2025 Grade |
|-----------|--------|----------|------------|
| Double DQN | ✅ Enabled | Hardcoded | ✅ A |
| Dueling | ✅ Enabled | `dueling_dim` (128-512) | 🟡 B (expand to 64-1024) |
| PER | ✅ Enabled | `per_alpha`, `per_beta` | ✅ A |
| N-Step | ✅ Enabled | `n_steps` (1-5) | ✅ A |
| Noisy Nets | ✅ Enabled | `noisy_sigma` (0.1-1.0) | ✅ A |
| **C51** | 🔴 **DISABLED** | **UNUSED** | 🔴 F (BUG #36) |

**Overall Rainbow Grade**: B+ (5/6 enabled, 1/6 blocked by bug)

---

## Key Metrics

### Current Performance
- **Search Space**: 22D continuous
- **Training Success**: 60% (40% fail Epoch 2 from C51 bug)
- **Best Sharpe**: 0.77-2.0 (standard DQN, no C51)
- **Sample Efficiency**: Baseline

### 2025 Target
- **Search Space**: 34D continuous (ensemble + architecture + C51 fix)
- **Training Success**: 95%+ (C51 fixed or removed)
- **Best Sharpe**: 1.5-3.0 (+50% target)
- **Sample Efficiency**: +40% (ensemble exploration)

---

## Quick Commands

### Check current hyperopt config
```bash
grep "continuous_bounds" ml/src/hyperopt/adapters/dqn.rs -A50
```

### Verify ensemble module exists
```bash
ls -lh ml/src/dqn/ensemble_uncertainty.rs
```

### Run hyperopt with new config (after fixes)
```bash
cargo run --release --bin ml_training_service -- \
  dqn-hyperopt \
  --data-dir test_data/real/databento/ml_training/ \
  --epochs 100 \
  --trials 50 \
  --run-name "2025-ensemble-integration"
```

---

**Last Updated**: 2025-11-27
**Owner**: ML Team
**Priority**: 🔴 Critical (implement Week 1 fixes immediately)