foxhunt/docs/plans/2026-03-03-hyperopt-improvements-design.md

DQN Hyperopt Improvements Design

Problem Statement

The current DQN hyperparameter optimization has three critical failures:

1. "Do Nothing" convergence (57% of trials): 8/14 trials converge to objective=44.86 (penalty value), producing 0 trades. PSO with 20 particles in 45D space can't explore effectively. Cliff penalties (-10.0 entropy, 1000.0 completion) give zero gradient signal.

2. Catastrophic leverage (29% of trials): 4/14 trials produce >100% max drawdown. max_position_absolute range [4.0, 8.0] allows 8x leverage on $100K. No drawdown circuit breaker during backtest evaluation.

3. Memory growth (844-1435 MB/trial): No explicit CUDA cache clearing between trials. drop() + 100ms sleep is insufficient for Candle's CUDA allocator.

Approach: A+B (Surgical Fixes + Bayesian Optimizer)

Part A: Objective Function & Environment Fixes

A1. Smooth Penalties (replace cliffs)

Current: Binary cliff penalties destroy PSO gradient signal.

entropy < 0.5  →  -10.0  (cliff)
entropy >= 0.5 →    0.0  (cliff)
completion < min_epochs → 1000.0 (dominates everything)

Proposed: Smooth, gradient-friendly penalties.

diversity_penalty = -5.0 * max(0, 1.0 - entropy/1.0)²   # Quadratic, smooth at boundary
completion_penalty = 50.0 * max(0, 1.0 - epochs/min_epochs)  # Linear scale, not 1000x

Files: crates/ml/src/hyperopt/adapters/dqn.rs — calculate_diversity_penalty() (line 1840), calculate_completion_penalty() (line 2198)

A2. Position Limit Enforcement

Current: max_position_absolute range [4.0, 8.0] with no drawdown guard.

Proposed:

• Narrow search range to [1.0, 4.0] (max 4x leverage)
• Add drawdown circuit breaker in backtest: if drawdown > 20%, force-close all positions and halt trading for remainder of episode
• Wire into execute_action_internal in portfolio_tracker.rs

Files: crates/ml/src/hyperopt/adapters/dqn.rs (search bounds line 476), crates/ml/src/dqn/portfolio_tracker.rs (execute_action_internal line 232)

A3. CUDA Memory Cleanup

Current: drop() + 100ms sleep. No CUDA synchronize or cache clear.

Proposed: After dropping trainer/metrics, call CUDA synchronize via Candle's backend, then force a GC pass. Add memory budget check — if usage exceeds 80% of available, trigger aggressive cleanup.

Files: crates/ml/src/hyperopt/adapters/dqn.rs (cleanup block line 3076)

A4. Search Space Reduction (45D → ~25D)

Fix 20 parameters to validated defaults. Keep the parameters that genuinely affect trading performance in the search space.

Keep in search (25D):

• Base: learning_rate, batch_size, gamma, buffer_size, hold_penalty_weight, max_position_absolute, huber_delta, entropy_coefficient, transaction_cost_multiplier (9)
• PER: per_alpha, per_beta_start (2)
• Rainbow: v_min, v_max, noisy_sigma_init, dueling_hidden_dim, n_steps, num_atoms (6)
• Risk: kelly_fractional, kelly_max_fraction, volatility_window (3)
• Training: weight_decay, curiosity_weight, tau, hidden_dim_base (4)
• Exploration: noisy_epsilon_floor (1)

Fix to defaults (20D):

• minimum_profit_factor → 1.5 (midpoint)
• kelly_min_trades → 20 (validated)
• ensemble_size → 5, beta_variance → 0.5, beta_disagreement → 0.5, beta_entropy → 0.2, variance_cap → 1.0 (ensemble defaults)
• warmup_ratio → 0.0 (hyperopt trials too short)
• td_error_clamp_max → 10.0, batch_diversity_cooldown → 50.0 (validated)
• lr_decay_type → 0 (constant for short trials)
• sharpe_weight → 0.0 (already in objective)
• gae_lambda → 0.95 (standard)
• noisy_sigma_initial → 0.5, noisy_sigma_final → 0.3 (validated)
• norm_type → 1 (RMSNorm), activation_type → 1 (LeakyReLU)
• num_quantiles → 64, qr_kappa → 1.0 (QR-DQN defaults)
• count_bonus_coefficient → 0.1 (validated)

Files: crates/ml/src/hyperopt/adapters/dqn.rs — continuous_bounds() (line 469), from_continuous() (line 549)

Part B: Replace PSO with Bayesian Optimization (TPE)

B1. TPE Optimizer Implementation

Replace ArgminOptimizer (PSO) with a Tree-Parzen Estimator:

1. Surrogate model: Maintain two kernel density estimates (KDEs) — one for parameters from "good" trials (top γ=25%), one for "bad" trials
2. Acquisition function: Expected Improvement (EI) = l(x)/g(x) where l(x) is the good KDE and g(x) is the bad KDE
3. Sampling: Draw candidates from l(x), score by EI, pick best
4. Initial exploration: Keep LHS (Latin Hypercube Sampling) for first 5 trials

The egobox crate (already in the workspace) provides EGO (GP-based BO) as an alternative, but TPE scales better to 25D. We'll implement a lightweight TPE using the statrs crate for KDE.

Architecture:

ArgminOptimizer (PSO, 20 particles, 50 iters/restart)
       ↓ replace with
TpeOptimizer {
    n_initial: 5,          // LHS warmup
    max_trials: 30,        // Same budget
    gamma: 0.25,           // Top 25% = "good"
    n_candidates: 100,     // EI candidates per trial
    bandwidth: "silverman" // KDE bandwidth selection
}

Files:

• New: crates/ml/src/hyperopt/tpe.rs (~300 lines)
• Modify: crates/ml/src/hyperopt/optimizer.rs (add TPE variant)
• Modify: crates/ml/src/hyperopt/mod.rs (export)
• Modify: crates/ml/src/hyperopt/campaign.rs (wire TPE)
• Modify CLI: bin/fxt/src/commands/train/hyperopt.rs (--optimizer=tpe flag)

B2. Trial History & Warm-Starting

TPE benefits from trial history across runs. Add JSON-based trial persistence:

• Save all (params, objective) pairs to hyperopt_dir/trial_history.json
• On restart, load history to seed TPE's KDE
• This makes interrupted runs resume intelligently instead of starting from scratch

Files: crates/ml/src/hyperopt/tpe.rs (persistence methods)

Non-Goals

• Changing the reward function or backtest environment (separate concern)
• Multi-GPU support (single L40S is the target)
• Changing the DQN architecture itself
• Modifying the FactoredAction space

Expected Impact

Metric	Current (PSO 45D)	Target (TPE 25D + fixes)
"Do nothing" rate	57% (8/14)	<15%
Catastrophic DD rate	29% (4/14)	<5%
Memory per trial	844-1435 MB	<200 MB delta
Best Sharpe found	1.73 (1 trial)	>2.0 consistently
Trials to find signal	~7	~3-5

Testing Strategy

• Unit tests for TPE KDE, EI acquisition, smooth penalties
• Integration test: 5-trial micro-hyperopt on synthetic data
• Regression: ensure existing PSO path still works (--optimizer=pso flag)