foxhunt/docs/plans/2026-03-03-hyperopt-improvements.md

# DQN Hyperopt Improvements Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Fix "do nothing" convergence (57%), catastrophic leverage (29%), and memory leaks in DQN hyperopt by replacing cliff penalties with smooth gradients, enforcing position limits, cleaning CUDA memory, reducing search space from 45D→25D, and replacing PSO with TPE (Tree-Parzen Estimator).

**Architecture:** Surgical fixes to the objective function (Part A) + new TPE optimizer (Part B). Both are orthogonal — A fixes the landscape, B navigates it better. The existing PSO path is preserved behind a flag.

**Tech Stack:** Rust, Candle ML, statrs (KDE), serde_json (trial persistence)

---

## Phase 1: Smooth Penalties (Part A1)

### Task 1: Replace diversity cliff penalty with smooth quadratic

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:1840-1860`
- Test: existing tests in same file

**Step 1: Write the failing test**

Add to the `tests` module at the bottom of `dqn.rs`:

```rust
#[test]
fn test_diversity_penalty_smooth() {
    // Old behavior: cliff at 0.5 entropy
    // New behavior: smooth quadratic, no cliff

    // Zero entropy (100% single action) = maximum penalty
    let penalty_zero = calculate_diversity_penalty(&[1.0, 0.0, 0.0]);
    assert!(penalty_zero < -4.0, "Zero entropy should give strong penalty: {}", penalty_zero);

    // Low entropy (80% single action) = moderate penalty
    let penalty_low = calculate_diversity_penalty(&[0.8, 0.1, 0.1]);
    assert!(penalty_low < -1.0, "Low entropy should give moderate penalty: {}", penalty_low);
    assert!(penalty_low > penalty_zero, "Lower entropy = stronger penalty");

    // Medium entropy (60% single action) = mild penalty
    let penalty_mid = calculate_diversity_penalty(&[0.6, 0.2, 0.2]);
    assert!(penalty_mid > -1.0, "Medium entropy should give mild penalty: {}", penalty_mid);

    // High entropy (uniform) = no penalty
    let penalty_uniform = calculate_diversity_penalty(&[0.33, 0.33, 0.34]);
    assert!(penalty_uniform.abs() < 0.1, "Uniform should give ~0 penalty: {}", penalty_uniform);

    // Gradient continuity: penalties should be monotonically ordered
    assert!(penalty_uniform > penalty_mid);
    assert!(penalty_mid > penalty_low);
    assert!(penalty_low > penalty_zero);
}
```

**Step 2: Run test to verify it fails**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib test_diversity_penalty_smooth -- --nocapture`
Expected: FAIL (current cliff function doesn't produce smooth gradient)

**Step 3: Implement smooth diversity penalty**

Replace `calculate_diversity_penalty` (lines 1840-1860):

```rust
fn calculate_diversity_penalty(action_distribution: &[f64; 3]) -> f64 {
    let total_actions = 1000;
    let action_counts: Vec<usize> = action_distribution
        .iter()
        .map(|&pct| (pct * total_actions as f64).round() as usize)
        .collect();

    let entropy = calculate_action_entropy(&*action_counts);
    let max_entropy = (3.0_f64).log2(); // ~1.585

    // Smooth quadratic penalty: -5.0 * (1 - entropy/max_entropy)²
    // At entropy=0: -5.0, at entropy=max: 0.0
    // Smooth gradient everywhere — no cliff
    let normalized = (entropy / max_entropy).clamp(0.0, 1.0);
    -5.0 * (1.0 - normalized).powi(2)
}
```

**Step 4: Run test to verify it passes**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib test_diversity_penalty_smooth -- --nocapture`
Expected: PASS

**Step 5: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(hyperopt): replace diversity cliff penalty with smooth quadratic"
```

### Task 2: Replace completion cliff penalty with linear scale

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:2198-2215`

**Step 1: Write the failing test**

```rust
#[test]
fn test_completion_penalty_smooth() {
    // 0 epochs = max penalty (but not 1000)
    let p0 = calculate_completion_penalty(0, 10, true);
    assert!((p0 - 50.0).abs() < 0.1, "0 epochs should give 50.0 penalty: {}", p0);

    // Half epochs = half penalty
    let p5 = calculate_completion_penalty(5, 10, false);
    assert!((p5 - 25.0).abs() < 0.1, "5/10 epochs should give ~25.0: {}", p5);

    // Full epochs = no penalty
    let p10 = calculate_completion_penalty(10, 10, false);
    assert!(p10.abs() < 0.01, "Full epochs should give 0: {}", p10);

    // Over min = no penalty
    let p15 = calculate_completion_penalty(15, 10, false);
    assert!(p15.abs() < 0.01, "Over min should give 0: {}", p15);

    // Monotonic decrease
    assert!(p0 > p5);
    assert!(p5 > p10);
}
```

**Step 2: Run test, verify fails**

**Step 3: Implement linear completion penalty**

```rust
fn calculate_completion_penalty(
    epochs_completed: u32,
    min_epochs: u32,
    _early_stop_triggered: bool,
) -> f64 {
    if min_epochs == 0 {
        return 0.0;
    }
    // Linear scale: 50.0 * (1 - completed/min_epochs), clamped to [0, 50]
    let completion_ratio = (epochs_completed as f64 / min_epochs as f64).clamp(0.0, 1.0);
    50.0 * (1.0 - completion_ratio)
}
```

**Step 4: Run test, verify passes**

**Step 5: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(hyperopt): replace completion cliff penalty (1000) with linear scale (max 50)"
```

## Phase 2: Position Limits & Drawdown Guard (Part A2)

### Task 3: Narrow max_position_absolute search range to [1.0, 4.0]

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:476` (bounds)
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:560` (from_continuous clamp)
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:390` (default)
- Test: `test_dqn_params_bounds`

**Step 1: Update bounds**

Change line 476 from `(4.0, 8.0)` to `(1.0, 4.0)`.
Change line 560 clamp from `.clamp(4.0, 8.0)` to `.clamp(1.0, 4.0)`.
Change line 390 default from `2.0` to `2.0` (keep).
Update test assertion at line 3460 from `(4.0, 8.0)` to `(1.0, 4.0)`.

**Step 2: Run tests**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib test_dqn_params_bounds -- --nocapture`

**Step 3: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(hyperopt): narrow max_position_absolute from [4,8] to [1,4] to prevent catastrophic leverage"
```

### Task 4: Add drawdown circuit breaker to portfolio tracker

**Files:**
- Modify: `crates/ml/src/dqn/portfolio_tracker.rs`

**Step 1: Write failing test**

```rust
#[test]
fn test_drawdown_circuit_breaker() {
    let mut tracker = PortfolioTracker::new(100_000.0, 0.0, 0.0);
    // Simulate a 25% drawdown
    tracker.cash = 70_000.0;
    tracker.position_size = 0.0;
    tracker.peak_value = 100_000.0;

    let action = FactoredAction::new(ExposureLevel::Long100, OrderType::Market, Urgency::Normal);
    tracker.execute_action(action, 100.0, 4.0);

    // Should refuse to open position when drawdown > 20%
    assert!(tracker.current_position().abs() < f32::EPSILON,
        "Circuit breaker should prevent new positions at >20% drawdown");
}
```

**Step 2: Implement drawdown check in execute_action_internal**

At the top of `execute_action_internal`, before calculating target position:

```rust
// Drawdown circuit breaker: refuse new positions if drawdown > 20%
let current_value = self.get_portfolio_value(price);
if self.peak_value > 0.0 {
    let drawdown = 1.0 - (current_value / self.peak_value);
    if drawdown > 0.20 && self.position_size.abs() < f32::EPSILON {
        // Already flat, don't open new positions
        return;
    }
    if drawdown > 0.20 {
        // Force close: set target to flat
        // (fall through with target_exposure = 0)
    }
}
```

Add `peak_value: f32` field to PortfolioTracker, update it in `get_portfolio_value()`.

**Step 3: Run test, verify passes**

**Step 4: Commit**

```bash
git add crates/ml/src/dqn/portfolio_tracker.rs
git commit -m "fix(hyperopt): add 20% drawdown circuit breaker to portfolio tracker"
```

## Phase 3: CUDA Memory Cleanup (Part A3)

### Task 5: Add proper CUDA synchronization and cache clearing between trials

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:3076-3091`

**Step 1: Implement CUDA cleanup**

Replace the cleanup block (lines 3076-3091):

```rust
// CRITICAL: Explicit memory cleanup between trials
info!("Cleaning up trial {} resources...", current_trial);
drop(training_metrics);
drop(internal_trainer);

// Force CUDA synchronize + cache clear
#[cfg(feature = "cuda")]
{
    if let candle_core::Device::Cuda(cuda_dev) = &device {
        // Synchronize to ensure all GPU operations complete
        if let Err(e) = cuda_dev.synchronize() {
            tracing::warn!("CUDA synchronize failed: {}", e);
        }
    }
}

// Force Rust allocator to release memory
// VarMap tensors are reference-counted — ensure no lingering refs
std::mem::drop(std::hint::black_box(()));

info!("Resource cleanup complete for trial {}", current_trial);
```

**Step 2: Run build check**

Run: `SQLX_OFFLINE=true cargo check -p ml`

**Step 3: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(hyperopt): proper CUDA synchronization between trials to prevent memory leaks"
```

## Phase 4: Search Space Reduction 45D→25D (Part A4)

### Task 6: Fix 20 parameters to validated defaults

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` — `continuous_bounds()`, `from_continuous()`, `to_continuous()`, `param_names()`

**Step 1: Write the failing test**

```rust
#[test]
fn test_reduced_search_space_25d() {
    let params = DQNParams::default();
    let bounds = params.continuous_bounds();
    assert_eq!(bounds.len(), 25, "Search space should be 25D, got {}D", bounds.len());

    let names = params.param_names();
    assert_eq!(names.len(), 25, "Should have 25 param names");

    // Roundtrip
    let continuous = params.to_continuous();
    assert_eq!(continuous.len(), 25);
    let recovered = DQNParams::from_continuous(&continuous);
    assert!((recovered.learning_rate - params.learning_rate).abs() < 1e-6);
}
```

**Step 2: Implement 25D search space**

Remove 20 parameters from `continuous_bounds()`, `from_continuous()`, `to_continuous()`, `param_names()`. Set the removed parameters to fixed defaults in `from_continuous()`:

Fixed defaults:
- minimum_profit_factor = 1.5
- kelly_min_trades = 20
- ensemble_size = 5, beta_variance = 0.5, beta_disagreement = 0.5, beta_entropy = 0.2, variance_cap = 1.0
- warmup_ratio = 0.0
- td_error_clamp_max = 10.0, batch_diversity_cooldown = 50.0
- lr_decay_type = 0 (constant)
- sharpe_weight = 0.0
- gae_lambda = 0.95
- noisy_sigma_initial = 0.5, noisy_sigma_final = 0.3
- norm_type = 1 (RMSNorm), activation_type = 1 (LeakyReLU)
- num_quantiles = 64, qr_kappa = 1.0
- count_bonus_coefficient = 0.1

**Step 3: Run all hyperopt tests**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib hyperopt -- --nocapture`

**Step 4: Fix any broken tests that referenced 45D**

**Step 5: Commit**

```bash
git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "feat(hyperopt): reduce DQN search space from 45D to 25D (fix 20 params to validated defaults)"
```

## Phase 5: TPE Optimizer (Part B1)

### Task 7: Add statrs dependency for KDE

**Files:**
- Modify: `crates/ml/Cargo.toml`

**Step 1: Add statrs**

Add `statrs = "0.18"` to `[dependencies]` in `crates/ml/Cargo.toml`.

**Step 2: Verify build**

Run: `SQLX_OFFLINE=true cargo check -p ml`

**Step 3: Commit**

```bash
git add crates/ml/Cargo.toml
git commit -m "chore(ml): add statrs dependency for TPE kernel density estimation"
```

### Task 8: Implement TPE optimizer core

**Files:**
- Create: `crates/ml/src/hyperopt/tpe.rs` (~300 lines)

**Step 1: Write failing tests**

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_tpe_split_good_bad() {
        let mut tpe = TpeOptimizer::new(2, 5, 0.25);
        // Add 4 trials with known objectives
        tpe.add_trial(vec![0.5, 0.5], 10.0); // bad
        tpe.add_trial(vec![0.3, 0.7], 5.0);  // good (lower = better)
        tpe.add_trial(vec![0.8, 0.2], 20.0); // bad
        tpe.add_trial(vec![0.2, 0.8], 1.0);  // good (best)

        let (good, bad) = tpe.split_trials();
        assert_eq!(good.len(), 1); // top 25% of 4 = 1
        assert_eq!(bad.len(), 3);
        assert!((good[0].objective - 1.0).abs() < 1e-6); // best trial
    }

    #[test]
    fn test_tpe_suggest_within_bounds() {
        let bounds = vec![(0.0, 1.0), (0.0, 1.0)];
        let mut tpe = TpeOptimizer::new(2, 10, 0.25);

        // Add some initial trials
        tpe.add_trial(vec![0.5, 0.5], 5.0);
        tpe.add_trial(vec![0.3, 0.7], 3.0);
        tpe.add_trial(vec![0.7, 0.3], 8.0);

        let suggestion = tpe.suggest(&bounds);
        assert_eq!(suggestion.len(), 2);
        assert!(suggestion[0] >= 0.0 && suggestion[0] <= 1.0);
        assert!(suggestion[1] >= 0.0 && suggestion[1] <= 1.0);
    }

    #[test]
    fn test_tpe_lhs_initial() {
        let bounds = vec![(0.0, 10.0), (-1.0, 1.0)];
        let tpe = TpeOptimizer::new(2, 10, 0.25);
        let samples = tpe.latin_hypercube_sample(&bounds, 5);
        assert_eq!(samples.len(), 5);
        for s in &samples {
            assert_eq!(s.len(), 2);
            assert!(s[0] >= 0.0 && s[0] <= 10.0);
            assert!(s[1] >= -1.0 && s[1] <= 1.0);
        }
    }
}
```

**Step 2: Implement TpeOptimizer**

Core struct:
```rust
pub struct TpeOptimizer {
    n_dims: usize,
    max_trials: usize,
    gamma: f64,            // good/bad split quantile (0.25)
    n_candidates: usize,   // EI candidates per suggestion (100)
    trials: Vec<TrialRecord>,
    rng: StdRng,
}

struct TrialRecord {
    params: Vec<f64>,
    objective: f64,
}
```

Key methods:
- `suggest(&self, bounds: &[(f64, f64)]) -> Vec<f64>` — if < n_initial trials, return LHS sample; else, build KDEs and maximize EI
- `add_trial(&mut self, params: Vec<f64>, objective: f64)` — record result
- `split_trials(&self) -> (Vec<&TrialRecord>, Vec<&TrialRecord>)` — top gamma% = good
- `kde_log_pdf(samples: &[&[f64]], point: &[f64], bounds: &[(f64, f64)]) -> f64` — per-dimension KDE with Silverman bandwidth
- `latin_hypercube_sample(bounds, n) -> Vec<Vec<f64>>` — LHS for initial exploration

**Step 3: Run tests**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib tpe -- --nocapture`

**Step 4: Commit**

```bash
git add crates/ml/src/hyperopt/tpe.rs
git commit -m "feat(hyperopt): implement TPE (Tree-Parzen Estimator) optimizer core"
```

### Task 9: Wire TPE into optimizer module

**Files:**
- Modify: `crates/ml/src/hyperopt/mod.rs` (export tpe)
- Modify: `crates/ml/src/hyperopt/optimizer.rs` (add OptimizerType enum)

**Step 1: Add TPE to mod.rs exports**

```rust
pub mod tpe;
```

**Step 2: Add OptimizerType enum to optimizer.rs**

```rust
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum OptimizerType {
    Pso,
    Tpe,
}

impl Default for OptimizerType {
    fn default() -> Self {
        OptimizerType::Tpe // New default
    }
}
```

**Step 3: Commit**

```bash
git add crates/ml/src/hyperopt/mod.rs crates/ml/src/hyperopt/optimizer.rs
git commit -m "feat(hyperopt): wire TPE into optimizer module with OptimizerType enum"
```

### Task 10: Integrate TPE into campaign runner

**Files:**
- Modify: `crates/ml/src/hyperopt/campaign.rs`

**Step 1: Add TPE path to campaign's optimize loop**

The campaign currently calls `ArgminOptimizer::optimize()`. Add a branch:

```rust
match config.optimizer_type {
    OptimizerType::Pso => { /* existing PSO path */ },
    OptimizerType::Tpe => {
        let mut tpe = TpeOptimizer::new(n_dims, config.num_trials, 0.25);
        let bounds = trainer.continuous_bounds();

        // Initial LHS phase
        let initial_samples = tpe.latin_hypercube_sample(&bounds, config.n_initial);
        for sample in initial_samples {
            let params = Params::from_continuous(&sample);
            let metrics = trainer.train_with_params(params)?;
            let objective = Trainer::extract_objective(&metrics);
            tpe.add_trial(sample, objective);
        }

        // TPE-guided phase
        for trial in config.n_initial..config.num_trials {
            let suggestion = tpe.suggest(&bounds);
            let params = Params::from_continuous(&suggestion);
            let metrics = trainer.train_with_params(params)?;
            let objective = Trainer::extract_objective(&metrics);
            tpe.add_trial(suggestion, objective);
        }
    }
}
```

**Step 2: Build check**

Run: `SQLX_OFFLINE=true cargo check -p ml`

**Step 3: Commit**

```bash
git add crates/ml/src/hyperopt/campaign.rs
git commit -m "feat(hyperopt): integrate TPE optimizer into campaign runner"
```

### Task 11: Add --optimizer CLI flag

**Files:**
- Modify: relevant CLI hyperopt command in `bin/fxt/`

**Step 1: Find and modify the hyperopt CLI**

Add `--optimizer` flag with `pso` and `tpe` options (default: `tpe`).

**Step 2: Wire through to campaign config**

**Step 3: Build check**

Run: `SQLX_OFFLINE=true cargo check -p fxt`

**Step 4: Commit**

```bash
git add bin/fxt/
git commit -m "feat(fxt): add --optimizer flag to hyperopt command (pso|tpe, default: tpe)"
```

## Phase 6: Trial Persistence (Part B2)

### Task 12: Add trial history save/load to TPE

**Files:**
- Modify: `crates/ml/src/hyperopt/tpe.rs`

**Step 1: Add persistence methods**

```rust
impl TpeOptimizer {
    pub fn save_history(&self, path: &Path) -> Result<(), std::io::Error> { ... }
    pub fn load_history(&mut self, path: &Path) -> Result<usize, std::io::Error> { ... }
}
```

**Step 2: Wire into campaign to auto-save after each trial and load on startup**

**Step 3: Test roundtrip**

**Step 4: Commit**

```bash
git add crates/ml/src/hyperopt/tpe.rs crates/ml/src/hyperopt/campaign.rs
git commit -m "feat(hyperopt): add trial history persistence for TPE warm-starting"
```

## Phase 7: Validation

### Task 13: Run full test suite

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- --nocapture 2>&1 | tail -5`

All 2506+ tests must pass, 0 clippy warnings.

### Task 14: Run workspace build check

Run: `SQLX_OFFLINE=true cargo check --workspace`
Run: `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings`

### Task 15: Commit and push

```bash
git push -u origin feature/hyperopt-improvements
```