# DQN Hyperopt Improvements Design

## Problem Statement

The current DQN hyperparameter optimization has three critical failures:

1. **"Do Nothing" convergence (57% of trials)**: 8/14 trials converge to objective=44.86 (penalty value), producing 0 trades. PSO with 20 particles in 45D space can't explore effectively. Cliff penalties (-10.0 entropy, 1000.0 completion) give zero gradient signal.

2. **Catastrophic leverage (29% of trials)**: 4/14 trials produce >100% max drawdown. `max_position_absolute` range [4.0, 8.0] allows 8x leverage on $100K. No drawdown circuit breaker during backtest evaluation.

3. **Memory growth (844-1435 MB/trial)**: No explicit CUDA cache clearing between trials. `drop()` + 100ms sleep is insufficient for Candle's CUDA allocator.

## Approach: A+B (Surgical Fixes + Bayesian Optimizer)

### Part A: Objective Function & Environment Fixes

#### A1. Smooth Penalties (replace cliffs)

**Current**: Binary cliff penalties destroy PSO gradient signal.
```
entropy < 0.5  →  -10.0  (cliff)
entropy >= 0.5 →    0.0  (cliff)
completion < min_epochs → 1000.0 (dominates everything)
```

**Proposed**: Smooth, gradient-friendly penalties.
```
diversity_penalty = -5.0 * max(0, 1.0 - entropy/1.0)²   # Quadratic, smooth at boundary
completion_penalty = 50.0 * max(0, 1.0 - epochs/min_epochs)  # Linear scale, not 1000x
```

**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` — `calculate_diversity_penalty()` (line 1840), `calculate_completion_penalty()` (line 2198)

#### A2. Position Limit Enforcement

**Current**: `max_position_absolute` range [4.0, 8.0] with no drawdown guard.

**Proposed**:
- Narrow search range to [1.0, 4.0] (max 4x leverage)
- Add drawdown circuit breaker in backtest: if drawdown > 20%, force-close all positions and halt trading for remainder of episode
- Wire into `execute_action_internal` in portfolio_tracker.rs

**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` (search bounds line 476), `crates/ml/src/dqn/portfolio_tracker.rs` (execute_action_internal line 232)

#### A3. CUDA Memory Cleanup

**Current**: `drop()` + 100ms sleep. No CUDA synchronize or cache clear.

**Proposed**: After dropping trainer/metrics, call CUDA synchronize via Candle's backend, then force a GC pass. Add memory budget check — if usage exceeds 80% of available, trigger aggressive cleanup.

**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` (cleanup block line 3076)

#### A4. Search Space Reduction (45D → ~25D)

Fix 20 parameters to validated defaults. Keep the parameters that genuinely affect trading performance in the search space.

**Keep in search (25D)**:
- Base: learning_rate, batch_size, gamma, buffer_size, hold_penalty_weight, max_position_absolute, huber_delta, entropy_coefficient, transaction_cost_multiplier (9)
- PER: per_alpha, per_beta_start (2)
- Rainbow: v_min, v_max, noisy_sigma_init, dueling_hidden_dim, n_steps, num_atoms (6)
- Risk: kelly_fractional, kelly_max_fraction, volatility_window (3)
- Training: weight_decay, curiosity_weight, tau, hidden_dim_base (4)
- Exploration: noisy_epsilon_floor (1)

**Fix to defaults (20D)**:
- minimum_profit_factor → 1.5 (midpoint)
- kelly_min_trades → 20 (validated)
- ensemble_size → 5, beta_variance → 0.5, beta_disagreement → 0.5, beta_entropy → 0.2, variance_cap → 1.0 (ensemble defaults)
- warmup_ratio → 0.0 (hyperopt trials too short)
- td_error_clamp_max → 10.0, batch_diversity_cooldown → 50.0 (validated)
- lr_decay_type → 0 (constant for short trials)
- sharpe_weight → 0.0 (already in objective)
- gae_lambda → 0.95 (standard)
- noisy_sigma_initial → 0.5, noisy_sigma_final → 0.3 (validated)
- norm_type → 1 (RMSNorm), activation_type → 1 (LeakyReLU)
- num_quantiles → 64, qr_kappa → 1.0 (QR-DQN defaults)
- count_bonus_coefficient → 0.1 (validated)

**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` — `continuous_bounds()` (line 469), `from_continuous()` (line 549)

### Part B: Replace PSO with Bayesian Optimization (TPE)

#### B1. TPE Optimizer Implementation

Replace `ArgminOptimizer` (PSO) with a Tree-Parzen Estimator:

1. **Surrogate model**: Maintain two kernel density estimates (KDEs) — one for parameters from "good" trials (top γ=25%), one for "bad" trials
2. **Acquisition function**: Expected Improvement (EI) = l(x)/g(x) where l(x) is the good KDE and g(x) is the bad KDE
3. **Sampling**: Draw candidates from l(x), score by EI, pick best
4. **Initial exploration**: Keep LHS (Latin Hypercube Sampling) for first 5 trials

The `egobox` crate (already in the workspace) provides EGO (GP-based BO) as an alternative, but TPE scales better to 25D. We'll implement a lightweight TPE using the `statrs` crate for KDE.

**Architecture**:
```
ArgminOptimizer (PSO, 20 particles, 50 iters/restart)
       ↓ replace with
TpeOptimizer {
    n_initial: 5,          // LHS warmup
    max_trials: 30,        // Same budget
    gamma: 0.25,           // Top 25% = "good"
    n_candidates: 100,     // EI candidates per trial
    bandwidth: "silverman" // KDE bandwidth selection
}
```

**Files**:
- New: `crates/ml/src/hyperopt/tpe.rs` (~300 lines)
- Modify: `crates/ml/src/hyperopt/optimizer.rs` (add TPE variant)
- Modify: `crates/ml/src/hyperopt/mod.rs` (export)
- Modify: `crates/ml/src/hyperopt/campaign.rs` (wire TPE)
- Modify CLI: `bin/fxt/src/commands/train/hyperopt.rs` (--optimizer=tpe flag)

#### B2. Trial History & Warm-Starting

TPE benefits from trial history across runs. Add JSON-based trial persistence:
- Save all (params, objective) pairs to `hyperopt_dir/trial_history.json`
- On restart, load history to seed TPE's KDE
- This makes interrupted runs resume intelligently instead of starting from scratch

**Files**: `crates/ml/src/hyperopt/tpe.rs` (persistence methods)

## Non-Goals

- Changing the reward function or backtest environment (separate concern)
- Multi-GPU support (single L40S is the target)
- Changing the DQN architecture itself
- Modifying the FactoredAction space

## Expected Impact

| Metric | Current (PSO 45D) | Target (TPE 25D + fixes) |
|--------|-------------------|--------------------------|
| "Do nothing" rate | 57% (8/14) | <15% |
| Catastrophic DD rate | 29% (4/14) | <5% |
| Memory per trial | 844-1435 MB | <200 MB delta |
| Best Sharpe found | 1.73 (1 trial) | >2.0 consistently |
| Trials to find signal | ~7 | ~3-5 |

## Testing Strategy

- Unit tests for TPE KDE, EI acquisition, smooth penalties
- Integration test: 5-trial micro-hyperopt on synthetic data
- Regression: ensure existing PSO path still works (--optimizer=pso flag)