foxhunt/docs/plans/2026-03-03-hyperopt-improvements.md

DQN Hyperopt Improvements Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Fix "do nothing" convergence (57%), catastrophic leverage (29%), and memory leaks in DQN hyperopt by replacing cliff penalties with smooth gradients, enforcing position limits, cleaning CUDA memory, reducing search space from 45D→25D, and replacing PSO with TPE (Tree-Parzen Estimator).

Architecture: Surgical fixes to the objective function (Part A) + new TPE optimizer (Part B). Both are orthogonal — A fixes the landscape, B navigates it better. The existing PSO path is preserved behind a flag.

Tech Stack: Rust, Candle ML, statrs (KDE), serde_json (trial persistence)

---

Phase 1: Smooth Penalties (Part A1)

Task 1: Replace diversity cliff penalty with smooth quadratic

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:1840-1860
• Test: existing tests in same file

Step 1: Write the failing test

Add to the tests module at the bottom of dqn.rs:

#[test]
fn test_diversity_penalty_smooth() {
    // Old behavior: cliff at 0.5 entropy
    // New behavior: smooth quadratic, no cliff

    // Zero entropy (100% single action) = maximum penalty
    let penalty_zero = calculate_diversity_penalty(&[1.0, 0.0, 0.0]);
    assert!(penalty_zero < -4.0, "Zero entropy should give strong penalty: {}", penalty_zero);

    // Low entropy (80% single action) = moderate penalty
    let penalty_low = calculate_diversity_penalty(&[0.8, 0.1, 0.1]);
    assert!(penalty_low < -1.0, "Low entropy should give moderate penalty: {}", penalty_low);
    assert!(penalty_low > penalty_zero, "Lower entropy = stronger penalty");

    // Medium entropy (60% single action) = mild penalty
    let penalty_mid = calculate_diversity_penalty(&[0.6, 0.2, 0.2]);
    assert!(penalty_mid > -1.0, "Medium entropy should give mild penalty: {}", penalty_mid);

    // High entropy (uniform) = no penalty
    let penalty_uniform = calculate_diversity_penalty(&[0.33, 0.33, 0.34]);
    assert!(penalty_uniform.abs() < 0.1, "Uniform should give ~0 penalty: {}", penalty_uniform);

    // Gradient continuity: penalties should be monotonically ordered
    assert!(penalty_uniform > penalty_mid);
    assert!(penalty_mid > penalty_low);
    assert!(penalty_low > penalty_zero);
}

Step 2: Run test to verify it fails

Run: SQLX_OFFLINE=true cargo test -p ml --lib test_diversity_penalty_smooth -- --nocapture Expected: FAIL (current cliff function doesn't produce smooth gradient)

Step 3: Implement smooth diversity penalty

Replace calculate_diversity_penalty (lines 1840-1860):

fn calculate_diversity_penalty(action_distribution: &[f64; 3]) -> f64 {
    let total_actions = 1000;
    let action_counts: Vec<usize> = action_distribution
        .iter()
        .map(|&pct| (pct * total_actions as f64).round() as usize)
        .collect();

    let entropy = calculate_action_entropy(&*action_counts);
    let max_entropy = (3.0_f64).log2(); // ~1.585

    // Smooth quadratic penalty: -5.0 * (1 - entropy/max_entropy)²
    // At entropy=0: -5.0, at entropy=max: 0.0
    // Smooth gradient everywhere — no cliff
    let normalized = (entropy / max_entropy).clamp(0.0, 1.0);
    -5.0 * (1.0 - normalized).powi(2)
}

Step 4: Run test to verify it passes

Run: SQLX_OFFLINE=true cargo test -p ml --lib test_diversity_penalty_smooth -- --nocapture Expected: PASS

Step 5: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(hyperopt): replace diversity cliff penalty with smooth quadratic"

Task 2: Replace completion cliff penalty with linear scale

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:2198-2215

Step 1: Write the failing test

#[test]
fn test_completion_penalty_smooth() {
    // 0 epochs = max penalty (but not 1000)
    let p0 = calculate_completion_penalty(0, 10, true);
    assert!((p0 - 50.0).abs() < 0.1, "0 epochs should give 50.0 penalty: {}", p0);

    // Half epochs = half penalty
    let p5 = calculate_completion_penalty(5, 10, false);
    assert!((p5 - 25.0).abs() < 0.1, "5/10 epochs should give ~25.0: {}", p5);

    // Full epochs = no penalty
    let p10 = calculate_completion_penalty(10, 10, false);
    assert!(p10.abs() < 0.01, "Full epochs should give 0: {}", p10);

    // Over min = no penalty
    let p15 = calculate_completion_penalty(15, 10, false);
    assert!(p15.abs() < 0.01, "Over min should give 0: {}", p15);

    // Monotonic decrease
    assert!(p0 > p5);
    assert!(p5 > p10);
}

Step 2: Run test, verify fails

Step 3: Implement linear completion penalty

fn calculate_completion_penalty(
    epochs_completed: u32,
    min_epochs: u32,
    _early_stop_triggered: bool,
) -> f64 {
    if min_epochs == 0 {
        return 0.0;
    }
    // Linear scale: 50.0 * (1 - completed/min_epochs), clamped to [0, 50]
    let completion_ratio = (epochs_completed as f64 / min_epochs as f64).clamp(0.0, 1.0);
    50.0 * (1.0 - completion_ratio)
}

Step 4: Run test, verify passes

Step 5: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(hyperopt): replace completion cliff penalty (1000) with linear scale (max 50)"

Phase 2: Position Limits & Drawdown Guard (Part A2)

Task 3: Narrow max_position_absolute search range to [1.0, 4.0]

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:476 (bounds)
• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:560 (from_continuous clamp)
• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:390 (default)
• Test: test_dqn_params_bounds

Step 1: Update bounds

Change line 476 from (4.0, 8.0) to (1.0, 4.0). Change line 560 clamp from .clamp(4.0, 8.0) to .clamp(1.0, 4.0). Change line 390 default from 2.0 to 2.0 (keep). Update test assertion at line 3460 from (4.0, 8.0) to (1.0, 4.0).

Step 2: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib test_dqn_params_bounds -- --nocapture

Step 3: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(hyperopt): narrow max_position_absolute from [4,8] to [1,4] to prevent catastrophic leverage"

Task 4: Add drawdown circuit breaker to portfolio tracker

Files:

• Modify: crates/ml/src/dqn/portfolio_tracker.rs

Step 1: Write failing test

#[test]
fn test_drawdown_circuit_breaker() {
    let mut tracker = PortfolioTracker::new(100_000.0, 0.0, 0.0);
    // Simulate a 25% drawdown
    tracker.cash = 70_000.0;
    tracker.position_size = 0.0;
    tracker.peak_value = 100_000.0;

    let action = FactoredAction::new(ExposureLevel::Long100, OrderType::Market, Urgency::Normal);
    tracker.execute_action(action, 100.0, 4.0);

    // Should refuse to open position when drawdown > 20%
    assert!(tracker.current_position().abs() < f32::EPSILON,
        "Circuit breaker should prevent new positions at >20% drawdown");
}

Step 2: Implement drawdown check in execute_action_internal

At the top of execute_action_internal, before calculating target position:

// Drawdown circuit breaker: refuse new positions if drawdown > 20%
let current_value = self.get_portfolio_value(price);
if self.peak_value > 0.0 {
    let drawdown = 1.0 - (current_value / self.peak_value);
    if drawdown > 0.20 && self.position_size.abs() < f32::EPSILON {
        // Already flat, don't open new positions
        return;
    }
    if drawdown > 0.20 {
        // Force close: set target to flat
        // (fall through with target_exposure = 0)
    }
}

Add peak_value: f32 field to PortfolioTracker, update it in get_portfolio_value().

Step 3: Run test, verify passes

Step 4: Commit

git add crates/ml/src/dqn/portfolio_tracker.rs
git commit -m "fix(hyperopt): add 20% drawdown circuit breaker to portfolio tracker"

Phase 3: CUDA Memory Cleanup (Part A3)

Task 5: Add proper CUDA synchronization and cache clearing between trials

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs:3076-3091

Step 1: Implement CUDA cleanup

Replace the cleanup block (lines 3076-3091):

// CRITICAL: Explicit memory cleanup between trials
info!("Cleaning up trial {} resources...", current_trial);
drop(training_metrics);
drop(internal_trainer);

// Force CUDA synchronize + cache clear
#[cfg(feature = "cuda")]
{
    if let candle_core::Device::Cuda(cuda_dev) = &device {
        // Synchronize to ensure all GPU operations complete
        if let Err(e) = cuda_dev.synchronize() {
            tracing::warn!("CUDA synchronize failed: {}", e);
        }
    }
}

// Force Rust allocator to release memory
// VarMap tensors are reference-counted — ensure no lingering refs
std::mem::drop(std::hint::black_box(()));

info!("Resource cleanup complete for trial {}", current_trial);

Step 2: Run build check

Run: SQLX_OFFLINE=true cargo check -p ml

Step 3: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "fix(hyperopt): proper CUDA synchronization between trials to prevent memory leaks"

Phase 4: Search Space Reduction 45D→25D (Part A4)

Task 6: Fix 20 parameters to validated defaults

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs — continuous_bounds(), from_continuous(), to_continuous(), param_names()

Step 1: Write the failing test

#[test]
fn test_reduced_search_space_25d() {
    let params = DQNParams::default();
    let bounds = params.continuous_bounds();
    assert_eq!(bounds.len(), 25, "Search space should be 25D, got {}D", bounds.len());

    let names = params.param_names();
    assert_eq!(names.len(), 25, "Should have 25 param names");

    // Roundtrip
    let continuous = params.to_continuous();
    assert_eq!(continuous.len(), 25);
    let recovered = DQNParams::from_continuous(&continuous);
    assert!((recovered.learning_rate - params.learning_rate).abs() < 1e-6);
}

Step 2: Implement 25D search space

Remove 20 parameters from continuous_bounds(), from_continuous(), to_continuous(), param_names(). Set the removed parameters to fixed defaults in from_continuous():

Fixed defaults:

• minimum_profit_factor = 1.5
• kelly_min_trades = 20
• ensemble_size = 5, beta_variance = 0.5, beta_disagreement = 0.5, beta_entropy = 0.2, variance_cap = 1.0
• warmup_ratio = 0.0
• td_error_clamp_max = 10.0, batch_diversity_cooldown = 50.0
• lr_decay_type = 0 (constant)
• sharpe_weight = 0.0
• gae_lambda = 0.95
• noisy_sigma_initial = 0.5, noisy_sigma_final = 0.3
• norm_type = 1 (RMSNorm), activation_type = 1 (LeakyReLU)
• num_quantiles = 64, qr_kappa = 1.0
• count_bonus_coefficient = 0.1

Step 3: Run all hyperopt tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib hyperopt -- --nocapture

Step 4: Fix any broken tests that referenced 45D

Step 5: Commit

git add crates/ml/src/hyperopt/adapters/dqn.rs
git commit -m "feat(hyperopt): reduce DQN search space from 45D to 25D (fix 20 params to validated defaults)"

Phase 5: TPE Optimizer (Part B1)

Task 7: Add statrs dependency for KDE

Files:

• Modify: crates/ml/Cargo.toml

Step 1: Add statrs

Add statrs = "0.18" to [dependencies] in crates/ml/Cargo.toml.

Step 2: Verify build

Run: SQLX_OFFLINE=true cargo check -p ml

Step 3: Commit

git add crates/ml/Cargo.toml
git commit -m "chore(ml): add statrs dependency for TPE kernel density estimation"

Task 8: Implement TPE optimizer core

Files:

• Create: crates/ml/src/hyperopt/tpe.rs (~300 lines)

Step 1: Write failing tests

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_tpe_split_good_bad() {
        let mut tpe = TpeOptimizer::new(2, 5, 0.25);
        // Add 4 trials with known objectives
        tpe.add_trial(vec![0.5, 0.5], 10.0); // bad
        tpe.add_trial(vec![0.3, 0.7], 5.0);  // good (lower = better)
        tpe.add_trial(vec![0.8, 0.2], 20.0); // bad
        tpe.add_trial(vec![0.2, 0.8], 1.0);  // good (best)

        let (good, bad) = tpe.split_trials();
        assert_eq!(good.len(), 1); // top 25% of 4 = 1
        assert_eq!(bad.len(), 3);
        assert!((good[0].objective - 1.0).abs() < 1e-6); // best trial
    }

    #[test]
    fn test_tpe_suggest_within_bounds() {
        let bounds = vec![(0.0, 1.0), (0.0, 1.0)];
        let mut tpe = TpeOptimizer::new(2, 10, 0.25);

        // Add some initial trials
        tpe.add_trial(vec![0.5, 0.5], 5.0);
        tpe.add_trial(vec![0.3, 0.7], 3.0);
        tpe.add_trial(vec![0.7, 0.3], 8.0);

        let suggestion = tpe.suggest(&bounds);
        assert_eq!(suggestion.len(), 2);
        assert!(suggestion[0] >= 0.0 && suggestion[0] <= 1.0);
        assert!(suggestion[1] >= 0.0 && suggestion[1] <= 1.0);
    }

    #[test]
    fn test_tpe_lhs_initial() {
        let bounds = vec![(0.0, 10.0), (-1.0, 1.0)];
        let tpe = TpeOptimizer::new(2, 10, 0.25);
        let samples = tpe.latin_hypercube_sample(&bounds, 5);
        assert_eq!(samples.len(), 5);
        for s in &samples {
            assert_eq!(s.len(), 2);
            assert!(s[0] >= 0.0 && s[0] <= 10.0);
            assert!(s[1] >= -1.0 && s[1] <= 1.0);
        }
    }
}

Step 2: Implement TpeOptimizer

Core struct:

pub struct TpeOptimizer {
    n_dims: usize,
    max_trials: usize,
    gamma: f64,            // good/bad split quantile (0.25)
    n_candidates: usize,   // EI candidates per suggestion (100)
    trials: Vec<TrialRecord>,
    rng: StdRng,
}

struct TrialRecord {
    params: Vec<f64>,
    objective: f64,
}

Key methods:

• suggest(&self, bounds: &[(f64, f64)]) -> Vec<f64> — if < n_initial trials, return LHS sample; else, build KDEs and maximize EI
• add_trial(&mut self, params: Vec<f64>, objective: f64) — record result
• split_trials(&self) -> (Vec<&TrialRecord>, Vec<&TrialRecord>) — top gamma% = good
• kde_log_pdf(samples: &[&[f64]], point: &[f64], bounds: &[(f64, f64)]) -> f64 — per-dimension KDE with Silverman bandwidth
• latin_hypercube_sample(bounds, n) -> Vec<Vec<f64>> — LHS for initial exploration

Step 3: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib tpe -- --nocapture

Step 4: Commit

git add crates/ml/src/hyperopt/tpe.rs
git commit -m "feat(hyperopt): implement TPE (Tree-Parzen Estimator) optimizer core"

Task 9: Wire TPE into optimizer module

Files:

• Modify: crates/ml/src/hyperopt/mod.rs (export tpe)
• Modify: crates/ml/src/hyperopt/optimizer.rs (add OptimizerType enum)

Step 1: Add TPE to mod.rs exports

pub mod tpe;

Step 2: Add OptimizerType enum to optimizer.rs

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum OptimizerType {
    Pso,
    Tpe,
}

impl Default for OptimizerType {
    fn default() -> Self {
        OptimizerType::Tpe // New default
    }
}

Step 3: Commit

git add crates/ml/src/hyperopt/mod.rs crates/ml/src/hyperopt/optimizer.rs
git commit -m "feat(hyperopt): wire TPE into optimizer module with OptimizerType enum"

Task 10: Integrate TPE into campaign runner

Files:

• Modify: crates/ml/src/hyperopt/campaign.rs

Step 1: Add TPE path to campaign's optimize loop

The campaign currently calls ArgminOptimizer::optimize(). Add a branch:

match config.optimizer_type {
    OptimizerType::Pso => { /* existing PSO path */ },
    OptimizerType::Tpe => {
        let mut tpe = TpeOptimizer::new(n_dims, config.num_trials, 0.25);
        let bounds = trainer.continuous_bounds();

        // Initial LHS phase
        let initial_samples = tpe.latin_hypercube_sample(&bounds, config.n_initial);
        for sample in initial_samples {
            let params = Params::from_continuous(&sample);
            let metrics = trainer.train_with_params(params)?;
            let objective = Trainer::extract_objective(&metrics);
            tpe.add_trial(sample, objective);
        }

        // TPE-guided phase
        for trial in config.n_initial..config.num_trials {
            let suggestion = tpe.suggest(&bounds);
            let params = Params::from_continuous(&suggestion);
            let metrics = trainer.train_with_params(params)?;
            let objective = Trainer::extract_objective(&metrics);
            tpe.add_trial(suggestion, objective);
        }
    }
}

Step 2: Build check

Run: SQLX_OFFLINE=true cargo check -p ml

Step 3: Commit

git add crates/ml/src/hyperopt/campaign.rs
git commit -m "feat(hyperopt): integrate TPE optimizer into campaign runner"

Task 11: Add --optimizer CLI flag

Files:

• Modify: relevant CLI hyperopt command in bin/fxt/

Step 1: Find and modify the hyperopt CLI

Add --optimizer flag with pso and tpe options (default: tpe).

Step 2: Wire through to campaign config

Step 3: Build check

Run: SQLX_OFFLINE=true cargo check -p fxt

Step 4: Commit

git add bin/fxt/
git commit -m "feat(fxt): add --optimizer flag to hyperopt command (pso|tpe, default: tpe)"

Phase 6: Trial Persistence (Part B2)

Task 12: Add trial history save/load to TPE

Files:

• Modify: crates/ml/src/hyperopt/tpe.rs

Step 1: Add persistence methods

impl TpeOptimizer {
    pub fn save_history(&self, path: &Path) -> Result<(), std::io::Error> { ... }
    pub fn load_history(&mut self, path: &Path) -> Result<usize, std::io::Error> { ... }
}

Step 2: Wire into campaign to auto-save after each trial and load on startup

Step 3: Test roundtrip

Step 4: Commit

git add crates/ml/src/hyperopt/tpe.rs crates/ml/src/hyperopt/campaign.rs
git commit -m "feat(hyperopt): add trial history persistence for TPE warm-starting"

Phase 7: Validation

Task 13: Run full test suite

Run: SQLX_OFFLINE=true cargo test -p ml --lib -- --nocapture 2>&1 | tail -5

All 2506+ tests must pass, 0 clippy warnings.

Task 14: Run workspace build check

Run: SQLX_OFFLINE=true cargo check --workspace Run: SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings

Task 15: Commit and push

git push -u origin feature/hyperopt-improvements