docs: add hyperopt improvements design and implementation plan

A+B approach: smooth penalties + position limits + CUDA cleanup + search space reduction (45D→25D) + TPE optimizer replacing PSO. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 18:17:23 +01:00
parent 4f6e893d2b
commit a0a2a1d736
2 changed files with 753 additions and 0 deletions
--- a/docs/plans/2026-03-03-hyperopt-improvements-design.md
+++ b/docs/plans/2026-03-03-hyperopt-improvements-design.md
@@ -0,0 +1,144 @@
+# DQN Hyperopt Improvements Design
+
+## Problem Statement
+
+The current DQN hyperparameter optimization has three critical failures:
+
+1. **"Do Nothing" convergence (57% of trials)**: 8/14 trials converge to objective=44.86 (penalty value), producing 0 trades. PSO with 20 particles in 45D space can't explore effectively. Cliff penalties (-10.0 entropy, 1000.0 completion) give zero gradient signal.
+
+2. **Catastrophic leverage (29% of trials)**: 4/14 trials produce >100% max drawdown. `max_position_absolute` range [4.0, 8.0] allows 8x leverage on $100K. No drawdown circuit breaker during backtest evaluation.
+
+3. **Memory growth (844-1435 MB/trial)**: No explicit CUDA cache clearing between trials. `drop()` + 100ms sleep is insufficient for Candle's CUDA allocator.
+
+## Approach: A+B (Surgical Fixes + Bayesian Optimizer)
+
+### Part A: Objective Function & Environment Fixes
+
+#### A1. Smooth Penalties (replace cliffs)
+
+**Current**: Binary cliff penalties destroy PSO gradient signal.
+```
+entropy < 0.5  →  -10.0  (cliff)
+entropy >= 0.5 →    0.0  (cliff)
+completion < min_epochs → 1000.0 (dominates everything)
+```
+
+**Proposed**: Smooth, gradient-friendly penalties.
+```
+diversity_penalty = -5.0 * max(0, 1.0 - entropy/1.0)²   # Quadratic, smooth at boundary
+completion_penalty = 50.0 * max(0, 1.0 - epochs/min_epochs)  # Linear scale, not 1000x
+```
+
+**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` — `calculate_diversity_penalty()` (line 1840), `calculate_completion_penalty()` (line 2198)
+
+#### A2. Position Limit Enforcement
+
+**Current**: `max_position_absolute` range [4.0, 8.0] with no drawdown guard.
+
+**Proposed**:
+- Narrow search range to [1.0, 4.0] (max 4x leverage)
+- Add drawdown circuit breaker in backtest: if drawdown > 20%, force-close all positions and halt trading for remainder of episode
+- Wire into `execute_action_internal` in portfolio_tracker.rs
+
+**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` (search bounds line 476), `crates/ml/src/dqn/portfolio_tracker.rs` (execute_action_internal line 232)
+
+#### A3. CUDA Memory Cleanup
+
+**Current**: `drop()` + 100ms sleep. No CUDA synchronize or cache clear.
+
+**Proposed**: After dropping trainer/metrics, call CUDA synchronize via Candle's backend, then force a GC pass. Add memory budget check — if usage exceeds 80% of available, trigger aggressive cleanup.
+
+**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` (cleanup block line 3076)
+
+#### A4. Search Space Reduction (45D → ~25D)
+
+Fix 20 parameters to validated defaults. Keep the parameters that genuinely affect trading performance in the search space.
+
+**Keep in search (25D)**:
+- Base: learning_rate, batch_size, gamma, buffer_size, hold_penalty_weight, max_position_absolute, huber_delta, entropy_coefficient, transaction_cost_multiplier (9)
+- PER: per_alpha, per_beta_start (2)
+- Rainbow: v_min, v_max, noisy_sigma_init, dueling_hidden_dim, n_steps, num_atoms (6)
+- Risk: kelly_fractional, kelly_max_fraction, volatility_window (3)
+- Training: weight_decay, curiosity_weight, tau, hidden_dim_base (4)
+- Exploration: noisy_epsilon_floor (1)
+
+**Fix to defaults (20D)**:
+- minimum_profit_factor → 1.5 (midpoint)
+- kelly_min_trades → 20 (validated)
+- ensemble_size → 5, beta_variance → 0.5, beta_disagreement → 0.5, beta_entropy → 0.2, variance_cap → 1.0 (ensemble defaults)
+- warmup_ratio → 0.0 (hyperopt trials too short)
+- td_error_clamp_max → 10.0, batch_diversity_cooldown → 50.0 (validated)
+- lr_decay_type → 0 (constant for short trials)
+- sharpe_weight → 0.0 (already in objective)
+- gae_lambda → 0.95 (standard)
+- noisy_sigma_initial → 0.5, noisy_sigma_final → 0.3 (validated)
+- norm_type → 1 (RMSNorm), activation_type → 1 (LeakyReLU)
+- num_quantiles → 64, qr_kappa → 1.0 (QR-DQN defaults)
+- count_bonus_coefficient → 0.1 (validated)
+
+**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` — `continuous_bounds()` (line 469), `from_continuous()` (line 549)
+
+### Part B: Replace PSO with Bayesian Optimization (TPE)
+
+#### B1. TPE Optimizer Implementation
+
+Replace `ArgminOptimizer` (PSO) with a Tree-Parzen Estimator:
+
+1. **Surrogate model**: Maintain two kernel density estimates (KDEs) — one for parameters from "good" trials (top γ=25%), one for "bad" trials
+2. **Acquisition function**: Expected Improvement (EI) = l(x)/g(x) where l(x) is the good KDE and g(x) is the bad KDE
+3. **Sampling**: Draw candidates from l(x), score by EI, pick best
+4. **Initial exploration**: Keep LHS (Latin Hypercube Sampling) for first 5 trials
+
+The `egobox` crate (already in the workspace) provides EGO (GP-based BO) as an alternative, but TPE scales better to 25D. We'll implement a lightweight TPE using the `statrs` crate for KDE.
+
+**Architecture**:
+```
+ArgminOptimizer (PSO, 20 particles, 50 iters/restart)
+       ↓ replace with
+TpeOptimizer {
+    n_initial: 5,          // LHS warmup
+    max_trials: 30,        // Same budget
+    gamma: 0.25,           // Top 25% = "good"
+    n_candidates: 100,     // EI candidates per trial
+    bandwidth: "silverman" // KDE bandwidth selection
+}
+```
+
+**Files**:
+- New: `crates/ml/src/hyperopt/tpe.rs` (~300 lines)
+- Modify: `crates/ml/src/hyperopt/optimizer.rs` (add TPE variant)
+- Modify: `crates/ml/src/hyperopt/mod.rs` (export)
+- Modify: `crates/ml/src/hyperopt/campaign.rs` (wire TPE)
+- Modify CLI: `bin/fxt/src/commands/train/hyperopt.rs` (--optimizer=tpe flag)
+
+#### B2. Trial History & Warm-Starting
+
+TPE benefits from trial history across runs. Add JSON-based trial persistence:
+- Save all (params, objective) pairs to `hyperopt_dir/trial_history.json`
+- On restart, load history to seed TPE's KDE
+- This makes interrupted runs resume intelligently instead of starting from scratch
+
+**Files**: `crates/ml/src/hyperopt/tpe.rs` (persistence methods)
+
+## Non-Goals
+
+- Changing the reward function or backtest environment (separate concern)
+- Multi-GPU support (single L40S is the target)
+- Changing the DQN architecture itself
+- Modifying the FactoredAction space
+
+## Expected Impact
+
+| Metric | Current (PSO 45D) | Target (TPE 25D + fixes) |
+|--------|-------------------|--------------------------|
+| "Do nothing" rate | 57% (8/14) | <15% |
+| Catastrophic DD rate | 29% (4/14) | <5% |
+| Memory per trial | 844-1435 MB | <200 MB delta |
+| Best Sharpe found | 1.73 (1 trial) | >2.0 consistently |
+| Trials to find signal | ~7 | ~3-5 |
+
+## Testing Strategy
+
+- Unit tests for TPE KDE, EI acquisition, smooth penalties
+- Integration test: 5-trial micro-hyperopt on synthetic data
+- Regression: ensure existing PSO path still works (--optimizer=pso flag)
--- a/docs/plans/2026-03-03-hyperopt-improvements.md
+++ b/docs/plans/2026-03-03-hyperopt-improvements.md
@@ -0,0 +1,609 @@
+# DQN Hyperopt Improvements Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
+
+**Goal:** Fix "do nothing" convergence (57%), catastrophic leverage (29%), and memory leaks in DQN hyperopt by replacing cliff penalties with smooth gradients, enforcing position limits, cleaning CUDA memory, reducing search space from 45D→25D, and replacing PSO with TPE (Tree-Parzen Estimator).
+
+**Architecture:** Surgical fixes to the objective function (Part A) + new TPE optimizer (Part B). Both are orthogonal — A fixes the landscape, B navigates it better. The existing PSO path is preserved behind a flag.
+
+**Tech Stack:** Rust, Candle ML, statrs (KDE), serde_json (trial persistence)
+
+---
+
+## Phase 1: Smooth Penalties (Part A1)
+
+### Task 1: Replace diversity cliff penalty with smooth quadratic
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:1840-1860`
+- Test: existing tests in same file
+
+**Step 1: Write the failing test**
+
+Add to the `tests` module at the bottom of `dqn.rs`:
+
+```rust
+#[test]
+fn test_diversity_penalty_smooth() {
+    // Old behavior: cliff at 0.5 entropy
+    // New behavior: smooth quadratic, no cliff
+
+    // Zero entropy (100% single action) = maximum penalty
+    let penalty_zero = calculate_diversity_penalty(&[1.0, 0.0, 0.0]);
+    assert!(penalty_zero < -4.0, "Zero entropy should give strong penalty: {}", penalty_zero);
+
+    // Low entropy (80% single action) = moderate penalty
+    let penalty_low = calculate_diversity_penalty(&[0.8, 0.1, 0.1]);
+    assert!(penalty_low < -1.0, "Low entropy should give moderate penalty: {}", penalty_low);
+    assert!(penalty_low > penalty_zero, "Lower entropy = stronger penalty");
+
+    // Medium entropy (60% single action) = mild penalty
+    let penalty_mid = calculate_diversity_penalty(&[0.6, 0.2, 0.2]);
+    assert!(penalty_mid > -1.0, "Medium entropy should give mild penalty: {}", penalty_mid);
+
+    // High entropy (uniform) = no penalty
+    let penalty_uniform = calculate_diversity_penalty(&[0.33, 0.33, 0.34]);
+    assert!(penalty_uniform.abs() < 0.1, "Uniform should give ~0 penalty: {}", penalty_uniform);
+
+    // Gradient continuity: penalties should be monotonically ordered
+    assert!(penalty_uniform > penalty_mid);
+    assert!(penalty_mid > penalty_low);
+    assert!(penalty_low > penalty_zero);
+}
+```
+
+**Step 2: Run test to verify it fails**
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib test_diversity_penalty_smooth -- --nocapture`
+Expected: FAIL (current cliff function doesn't produce smooth gradient)
+
+**Step 3: Implement smooth diversity penalty**
+
+Replace `calculate_diversity_penalty` (lines 1840-1860):
+
+```rust
+fn calculate_diversity_penalty(action_distribution: &[f64; 3]) -> f64 {
+    let total_actions = 1000;
+    let action_counts: Vec<usize> = action_distribution
+        .iter()
+        .map(|&pct| (pct * total_actions as f64).round() as usize)
+        .collect();
+
+    let entropy = calculate_action_entropy(&*action_counts);
+    let max_entropy = (3.0_f64).log2(); // ~1.585
+
+    // Smooth quadratic penalty: -5.0 * (1 - entropy/max_entropy)²
+    // At entropy=0: -5.0, at entropy=max: 0.0
+    // Smooth gradient everywhere — no cliff
+    let normalized = (entropy / max_entropy).clamp(0.0, 1.0);
+    -5.0 * (1.0 - normalized).powi(2)
+}
+```
+
+**Step 4: Run test to verify it passes**
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib test_diversity_penalty_smooth -- --nocapture`
+Expected: PASS
+
+**Step 5: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/adapters/dqn.rs
+git commit -m "fix(hyperopt): replace diversity cliff penalty with smooth quadratic"
+```
+
+### Task 2: Replace completion cliff penalty with linear scale
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:2198-2215`
+
+**Step 1: Write the failing test**
+
+```rust
+#[test]
+fn test_completion_penalty_smooth() {
+    // 0 epochs = max penalty (but not 1000)
+    let p0 = calculate_completion_penalty(0, 10, true);
+    assert!((p0 - 50.0).abs() < 0.1, "0 epochs should give 50.0 penalty: {}", p0);
+
+    // Half epochs = half penalty
+    let p5 = calculate_completion_penalty(5, 10, false);
+    assert!((p5 - 25.0).abs() < 0.1, "5/10 epochs should give ~25.0: {}", p5);
+
+    // Full epochs = no penalty
+    let p10 = calculate_completion_penalty(10, 10, false);
+    assert!(p10.abs() < 0.01, "Full epochs should give 0: {}", p10);
+
+    // Over min = no penalty
+    let p15 = calculate_completion_penalty(15, 10, false);
+    assert!(p15.abs() < 0.01, "Over min should give 0: {}", p15);
+
+    // Monotonic decrease
+    assert!(p0 > p5);
+    assert!(p5 > p10);
+}
+```
+
+**Step 2: Run test, verify fails**
+
+**Step 3: Implement linear completion penalty**
+
+```rust
+fn calculate_completion_penalty(
+    epochs_completed: u32,
+    min_epochs: u32,
+    _early_stop_triggered: bool,
+) -> f64 {
+    if min_epochs == 0 {
+        return 0.0;
+    }
+    // Linear scale: 50.0 * (1 - completed/min_epochs), clamped to [0, 50]
+    let completion_ratio = (epochs_completed as f64 / min_epochs as f64).clamp(0.0, 1.0);
+    50.0 * (1.0 - completion_ratio)
+}
+```
+
+**Step 4: Run test, verify passes**
+
+**Step 5: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/adapters/dqn.rs
+git commit -m "fix(hyperopt): replace completion cliff penalty (1000) with linear scale (max 50)"
+```
+
+## Phase 2: Position Limits & Drawdown Guard (Part A2)
+
+### Task 3: Narrow max_position_absolute search range to [1.0, 4.0]
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:476` (bounds)
+- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:560` (from_continuous clamp)
+- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:390` (default)
+- Test: `test_dqn_params_bounds`
+
+**Step 1: Update bounds**
+
+Change line 476 from `(4.0, 8.0)` to `(1.0, 4.0)`.
+Change line 560 clamp from `.clamp(4.0, 8.0)` to `.clamp(1.0, 4.0)`.
+Change line 390 default from `2.0` to `2.0` (keep).
+Update test assertion at line 3460 from `(4.0, 8.0)` to `(1.0, 4.0)`.
+
+**Step 2: Run tests**
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib test_dqn_params_bounds -- --nocapture`
+
+**Step 3: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/adapters/dqn.rs
+git commit -m "fix(hyperopt): narrow max_position_absolute from [4,8] to [1,4] to prevent catastrophic leverage"
+```
+
+### Task 4: Add drawdown circuit breaker to portfolio tracker
+
+**Files:**
+- Modify: `crates/ml/src/dqn/portfolio_tracker.rs`
+
+**Step 1: Write failing test**
+
+```rust
+#[test]
+fn test_drawdown_circuit_breaker() {
+    let mut tracker = PortfolioTracker::new(100_000.0, 0.0, 0.0);
+    // Simulate a 25% drawdown
+    tracker.cash = 70_000.0;
+    tracker.position_size = 0.0;
+    tracker.peak_value = 100_000.0;
+
+    let action = FactoredAction::new(ExposureLevel::Long100, OrderType::Market, Urgency::Normal);
+    tracker.execute_action(action, 100.0, 4.0);
+
+    // Should refuse to open position when drawdown > 20%
+    assert!(tracker.current_position().abs() < f32::EPSILON,
+        "Circuit breaker should prevent new positions at >20% drawdown");
+}
+```
+
+**Step 2: Implement drawdown check in execute_action_internal**
+
+At the top of `execute_action_internal`, before calculating target position:
+
+```rust
+// Drawdown circuit breaker: refuse new positions if drawdown > 20%
+let current_value = self.get_portfolio_value(price);
+if self.peak_value > 0.0 {
+    let drawdown = 1.0 - (current_value / self.peak_value);
+    if drawdown > 0.20 && self.position_size.abs() < f32::EPSILON {
+        // Already flat, don't open new positions
+        return;
+    }
+    if drawdown > 0.20 {
+        // Force close: set target to flat
+        // (fall through with target_exposure = 0)
+    }
+}
+```
+
+Add `peak_value: f32` field to PortfolioTracker, update it in `get_portfolio_value()`.
+
+**Step 3: Run test, verify passes**
+
+**Step 4: Commit**
+
+```bash
+git add crates/ml/src/dqn/portfolio_tracker.rs
+git commit -m "fix(hyperopt): add 20% drawdown circuit breaker to portfolio tracker"
+```
+
+## Phase 3: CUDA Memory Cleanup (Part A3)
+
+### Task 5: Add proper CUDA synchronization and cache clearing between trials
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs:3076-3091`
+
+**Step 1: Implement CUDA cleanup**
+
+Replace the cleanup block (lines 3076-3091):
+
+```rust
+// CRITICAL: Explicit memory cleanup between trials
+info!("Cleaning up trial {} resources...", current_trial);
+drop(training_metrics);
+drop(internal_trainer);
+
+// Force CUDA synchronize + cache clear
+#[cfg(feature = "cuda")]
+{
+    if let candle_core::Device::Cuda(cuda_dev) = &device {
+        // Synchronize to ensure all GPU operations complete
+        if let Err(e) = cuda_dev.synchronize() {
+            tracing::warn!("CUDA synchronize failed: {}", e);
+        }
+    }
+}
+
+// Force Rust allocator to release memory
+// VarMap tensors are reference-counted — ensure no lingering refs
+std::mem::drop(std::hint::black_box(()));
+
+info!("Resource cleanup complete for trial {}", current_trial);
+```
+
+**Step 2: Run build check**
+
+Run: `SQLX_OFFLINE=true cargo check -p ml`
+
+**Step 3: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/adapters/dqn.rs
+git commit -m "fix(hyperopt): proper CUDA synchronization between trials to prevent memory leaks"
+```
+
+## Phase 4: Search Space Reduction 45D→25D (Part A4)
+
+### Task 6: Fix 20 parameters to validated defaults
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` — `continuous_bounds()`, `from_continuous()`, `to_continuous()`, `param_names()`
+
+**Step 1: Write the failing test**
+
+```rust
+#[test]
+fn test_reduced_search_space_25d() {
+    let params = DQNParams::default();
+    let bounds = params.continuous_bounds();
+    assert_eq!(bounds.len(), 25, "Search space should be 25D, got {}D", bounds.len());
+
+    let names = params.param_names();
+    assert_eq!(names.len(), 25, "Should have 25 param names");
+
+    // Roundtrip
+    let continuous = params.to_continuous();
+    assert_eq!(continuous.len(), 25);
+    let recovered = DQNParams::from_continuous(&continuous);
+    assert!((recovered.learning_rate - params.learning_rate).abs() < 1e-6);
+}
+```
+
+**Step 2: Implement 25D search space**
+
+Remove 20 parameters from `continuous_bounds()`, `from_continuous()`, `to_continuous()`, `param_names()`. Set the removed parameters to fixed defaults in `from_continuous()`:
+
+Fixed defaults:
+- minimum_profit_factor = 1.5
+- kelly_min_trades = 20
+- ensemble_size = 5, beta_variance = 0.5, beta_disagreement = 0.5, beta_entropy = 0.2, variance_cap = 1.0
+- warmup_ratio = 0.0
+- td_error_clamp_max = 10.0, batch_diversity_cooldown = 50.0
+- lr_decay_type = 0 (constant)
+- sharpe_weight = 0.0
+- gae_lambda = 0.95
+- noisy_sigma_initial = 0.5, noisy_sigma_final = 0.3
+- norm_type = 1 (RMSNorm), activation_type = 1 (LeakyReLU)
+- num_quantiles = 64, qr_kappa = 1.0
+- count_bonus_coefficient = 0.1
+
+**Step 3: Run all hyperopt tests**
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib hyperopt -- --nocapture`
+
+**Step 4: Fix any broken tests that referenced 45D**
+
+**Step 5: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/adapters/dqn.rs
+git commit -m "feat(hyperopt): reduce DQN search space from 45D to 25D (fix 20 params to validated defaults)"
+```
+
+## Phase 5: TPE Optimizer (Part B1)
+
+### Task 7: Add statrs dependency for KDE
+
+**Files:**
+- Modify: `crates/ml/Cargo.toml`
+
+**Step 1: Add statrs**
+
+Add `statrs = "0.18"` to `[dependencies]` in `crates/ml/Cargo.toml`.
+
+**Step 2: Verify build**
+
+Run: `SQLX_OFFLINE=true cargo check -p ml`
+
+**Step 3: Commit**
+
+```bash
+git add crates/ml/Cargo.toml
+git commit -m "chore(ml): add statrs dependency for TPE kernel density estimation"
+```
+
+### Task 8: Implement TPE optimizer core
+
+**Files:**
+- Create: `crates/ml/src/hyperopt/tpe.rs` (~300 lines)
+
+**Step 1: Write failing tests**
+
+```rust
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_tpe_split_good_bad() {
+        let mut tpe = TpeOptimizer::new(2, 5, 0.25);
+        // Add 4 trials with known objectives
+        tpe.add_trial(vec![0.5, 0.5], 10.0); // bad
+        tpe.add_trial(vec![0.3, 0.7], 5.0);  // good (lower = better)
+        tpe.add_trial(vec![0.8, 0.2], 20.0); // bad
+        tpe.add_trial(vec![0.2, 0.8], 1.0);  // good (best)
+
+        let (good, bad) = tpe.split_trials();
+        assert_eq!(good.len(), 1); // top 25% of 4 = 1
+        assert_eq!(bad.len(), 3);
+        assert!((good[0].objective - 1.0).abs() < 1e-6); // best trial
+    }
+
+    #[test]
+    fn test_tpe_suggest_within_bounds() {
+        let bounds = vec![(0.0, 1.0), (0.0, 1.0)];
+        let mut tpe = TpeOptimizer::new(2, 10, 0.25);
+
+        // Add some initial trials
+        tpe.add_trial(vec![0.5, 0.5], 5.0);
+        tpe.add_trial(vec![0.3, 0.7], 3.0);
+        tpe.add_trial(vec![0.7, 0.3], 8.0);
+
+        let suggestion = tpe.suggest(&bounds);
+        assert_eq!(suggestion.len(), 2);
+        assert!(suggestion[0] >= 0.0 && suggestion[0] <= 1.0);
+        assert!(suggestion[1] >= 0.0 && suggestion[1] <= 1.0);
+    }
+
+    #[test]
+    fn test_tpe_lhs_initial() {
+        let bounds = vec![(0.0, 10.0), (-1.0, 1.0)];
+        let tpe = TpeOptimizer::new(2, 10, 0.25);
+        let samples = tpe.latin_hypercube_sample(&bounds, 5);
+        assert_eq!(samples.len(), 5);
+        for s in &samples {
+            assert_eq!(s.len(), 2);
+            assert!(s[0] >= 0.0 && s[0] <= 10.0);
+            assert!(s[1] >= -1.0 && s[1] <= 1.0);
+        }
+    }
+}
+```
+
+**Step 2: Implement TpeOptimizer**
+
+Core struct:
+```rust
+pub struct TpeOptimizer {
+    n_dims: usize,
+    max_trials: usize,
+    gamma: f64,            // good/bad split quantile (0.25)
+    n_candidates: usize,   // EI candidates per suggestion (100)
+    trials: Vec<TrialRecord>,
+    rng: StdRng,
+}
+
+struct TrialRecord {
+    params: Vec<f64>,
+    objective: f64,
+}
+```
+
+Key methods:
+- `suggest(&self, bounds: &[(f64, f64)]) -> Vec<f64>` — if < n_initial trials, return LHS sample; else, build KDEs and maximize EI
+- `add_trial(&mut self, params: Vec<f64>, objective: f64)` — record result
+- `split_trials(&self) -> (Vec<&TrialRecord>, Vec<&TrialRecord>)` — top gamma% = good
+- `kde_log_pdf(samples: &[&[f64]], point: &[f64], bounds: &[(f64, f64)]) -> f64` — per-dimension KDE with Silverman bandwidth
+- `latin_hypercube_sample(bounds, n) -> Vec<Vec<f64>>` — LHS for initial exploration
+
+**Step 3: Run tests**
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib tpe -- --nocapture`
+
+**Step 4: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/tpe.rs
+git commit -m "feat(hyperopt): implement TPE (Tree-Parzen Estimator) optimizer core"
+```
+
+### Task 9: Wire TPE into optimizer module
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/mod.rs` (export tpe)
+- Modify: `crates/ml/src/hyperopt/optimizer.rs` (add OptimizerType enum)
+
+**Step 1: Add TPE to mod.rs exports**
+
+```rust
+pub mod tpe;
+```
+
+**Step 2: Add OptimizerType enum to optimizer.rs**
+
+```rust
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum OptimizerType {
+    Pso,
+    Tpe,
+}
+
+impl Default for OptimizerType {
+    fn default() -> Self {
+        OptimizerType::Tpe // New default
+    }
+}
+```
+
+**Step 3: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/mod.rs crates/ml/src/hyperopt/optimizer.rs
+git commit -m "feat(hyperopt): wire TPE into optimizer module with OptimizerType enum"
+```
+
+### Task 10: Integrate TPE into campaign runner
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/campaign.rs`
+
+**Step 1: Add TPE path to campaign's optimize loop**
+
+The campaign currently calls `ArgminOptimizer::optimize()`. Add a branch:
+
+```rust
+match config.optimizer_type {
+    OptimizerType::Pso => { /* existing PSO path */ },
+    OptimizerType::Tpe => {
+        let mut tpe = TpeOptimizer::new(n_dims, config.num_trials, 0.25);
+        let bounds = trainer.continuous_bounds();
+
+        // Initial LHS phase
+        let initial_samples = tpe.latin_hypercube_sample(&bounds, config.n_initial);
+        for sample in initial_samples {
+            let params = Params::from_continuous(&sample);
+            let metrics = trainer.train_with_params(params)?;
+            let objective = Trainer::extract_objective(&metrics);
+            tpe.add_trial(sample, objective);
+        }
+
+        // TPE-guided phase
+        for trial in config.n_initial..config.num_trials {
+            let suggestion = tpe.suggest(&bounds);
+            let params = Params::from_continuous(&suggestion);
+            let metrics = trainer.train_with_params(params)?;
+            let objective = Trainer::extract_objective(&metrics);
+            tpe.add_trial(suggestion, objective);
+        }
+    }
+}
+```
+
+**Step 2: Build check**
+
+Run: `SQLX_OFFLINE=true cargo check -p ml`
+
+**Step 3: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/campaign.rs
+git commit -m "feat(hyperopt): integrate TPE optimizer into campaign runner"
+```
+
+### Task 11: Add --optimizer CLI flag
+
+**Files:**
+- Modify: relevant CLI hyperopt command in `bin/fxt/`
+
+**Step 1: Find and modify the hyperopt CLI**
+
+Add `--optimizer` flag with `pso` and `tpe` options (default: `tpe`).
+
+**Step 2: Wire through to campaign config**
+
+**Step 3: Build check**
+
+Run: `SQLX_OFFLINE=true cargo check -p fxt`
+
+**Step 4: Commit**
+
+```bash
+git add bin/fxt/
+git commit -m "feat(fxt): add --optimizer flag to hyperopt command (pso|tpe, default: tpe)"
+```
+
+## Phase 6: Trial Persistence (Part B2)
+
+### Task 12: Add trial history save/load to TPE
+
+**Files:**
+- Modify: `crates/ml/src/hyperopt/tpe.rs`
+
+**Step 1: Add persistence methods**
+
+```rust
+impl TpeOptimizer {
+    pub fn save_history(&self, path: &Path) -> Result<(), std::io::Error> { ... }
+    pub fn load_history(&mut self, path: &Path) -> Result<usize, std::io::Error> { ... }
+}
+```
+
+**Step 2: Wire into campaign to auto-save after each trial and load on startup**
+
+**Step 3: Test roundtrip**
+
+**Step 4: Commit**
+
+```bash
+git add crates/ml/src/hyperopt/tpe.rs crates/ml/src/hyperopt/campaign.rs
+git commit -m "feat(hyperopt): add trial history persistence for TPE warm-starting"
+```
+
+## Phase 7: Validation
+
+### Task 13: Run full test suite
+
+Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- --nocapture 2>&1 | tail -5`
+
+All 2506+ tests must pass, 0 clippy warnings.
+
+### Task 14: Run workspace build check
+
+Run: `SQLX_OFFLINE=true cargo check --workspace`
+Run: `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings`
+
+### Task 15: Commit and push
+
+```bash
+git push -u origin feature/hyperopt-improvements
+```