a0a2a1d736
A+B approach: smooth penalties + position limits + CUDA cleanup + search space reduction (45D→25D) + TPE optimizer replacing PSO. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
145 lines
6.4 KiB
Markdown
145 lines
6.4 KiB
Markdown
# DQN Hyperopt Improvements Design
|
||
|
||
## Problem Statement
|
||
|
||
The current DQN hyperparameter optimization has three critical failures:
|
||
|
||
1. **"Do Nothing" convergence (57% of trials)**: 8/14 trials converge to objective=44.86 (penalty value), producing 0 trades. PSO with 20 particles in 45D space can't explore effectively. Cliff penalties (-10.0 entropy, 1000.0 completion) give zero gradient signal.
|
||
|
||
2. **Catastrophic leverage (29% of trials)**: 4/14 trials produce >100% max drawdown. `max_position_absolute` range [4.0, 8.0] allows 8x leverage on $100K. No drawdown circuit breaker during backtest evaluation.
|
||
|
||
3. **Memory growth (844-1435 MB/trial)**: No explicit CUDA cache clearing between trials. `drop()` + 100ms sleep is insufficient for Candle's CUDA allocator.
|
||
|
||
## Approach: A+B (Surgical Fixes + Bayesian Optimizer)
|
||
|
||
### Part A: Objective Function & Environment Fixes
|
||
|
||
#### A1. Smooth Penalties (replace cliffs)
|
||
|
||
**Current**: Binary cliff penalties destroy PSO gradient signal.
|
||
```
|
||
entropy < 0.5 → -10.0 (cliff)
|
||
entropy >= 0.5 → 0.0 (cliff)
|
||
completion < min_epochs → 1000.0 (dominates everything)
|
||
```
|
||
|
||
**Proposed**: Smooth, gradient-friendly penalties.
|
||
```
|
||
diversity_penalty = -5.0 * max(0, 1.0 - entropy/1.0)² # Quadratic, smooth at boundary
|
||
completion_penalty = 50.0 * max(0, 1.0 - epochs/min_epochs) # Linear scale, not 1000x
|
||
```
|
||
|
||
**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` — `calculate_diversity_penalty()` (line 1840), `calculate_completion_penalty()` (line 2198)
|
||
|
||
#### A2. Position Limit Enforcement
|
||
|
||
**Current**: `max_position_absolute` range [4.0, 8.0] with no drawdown guard.
|
||
|
||
**Proposed**:
|
||
- Narrow search range to [1.0, 4.0] (max 4x leverage)
|
||
- Add drawdown circuit breaker in backtest: if drawdown > 20%, force-close all positions and halt trading for remainder of episode
|
||
- Wire into `execute_action_internal` in portfolio_tracker.rs
|
||
|
||
**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` (search bounds line 476), `crates/ml/src/dqn/portfolio_tracker.rs` (execute_action_internal line 232)
|
||
|
||
#### A3. CUDA Memory Cleanup
|
||
|
||
**Current**: `drop()` + 100ms sleep. No CUDA synchronize or cache clear.
|
||
|
||
**Proposed**: After dropping trainer/metrics, call CUDA synchronize via Candle's backend, then force a GC pass. Add memory budget check — if usage exceeds 80% of available, trigger aggressive cleanup.
|
||
|
||
**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` (cleanup block line 3076)
|
||
|
||
#### A4. Search Space Reduction (45D → ~25D)
|
||
|
||
Fix 20 parameters to validated defaults. Keep the parameters that genuinely affect trading performance in the search space.
|
||
|
||
**Keep in search (25D)**:
|
||
- Base: learning_rate, batch_size, gamma, buffer_size, hold_penalty_weight, max_position_absolute, huber_delta, entropy_coefficient, transaction_cost_multiplier (9)
|
||
- PER: per_alpha, per_beta_start (2)
|
||
- Rainbow: v_min, v_max, noisy_sigma_init, dueling_hidden_dim, n_steps, num_atoms (6)
|
||
- Risk: kelly_fractional, kelly_max_fraction, volatility_window (3)
|
||
- Training: weight_decay, curiosity_weight, tau, hidden_dim_base (4)
|
||
- Exploration: noisy_epsilon_floor (1)
|
||
|
||
**Fix to defaults (20D)**:
|
||
- minimum_profit_factor → 1.5 (midpoint)
|
||
- kelly_min_trades → 20 (validated)
|
||
- ensemble_size → 5, beta_variance → 0.5, beta_disagreement → 0.5, beta_entropy → 0.2, variance_cap → 1.0 (ensemble defaults)
|
||
- warmup_ratio → 0.0 (hyperopt trials too short)
|
||
- td_error_clamp_max → 10.0, batch_diversity_cooldown → 50.0 (validated)
|
||
- lr_decay_type → 0 (constant for short trials)
|
||
- sharpe_weight → 0.0 (already in objective)
|
||
- gae_lambda → 0.95 (standard)
|
||
- noisy_sigma_initial → 0.5, noisy_sigma_final → 0.3 (validated)
|
||
- norm_type → 1 (RMSNorm), activation_type → 1 (LeakyReLU)
|
||
- num_quantiles → 64, qr_kappa → 1.0 (QR-DQN defaults)
|
||
- count_bonus_coefficient → 0.1 (validated)
|
||
|
||
**Files**: `crates/ml/src/hyperopt/adapters/dqn.rs` — `continuous_bounds()` (line 469), `from_continuous()` (line 549)
|
||
|
||
### Part B: Replace PSO with Bayesian Optimization (TPE)
|
||
|
||
#### B1. TPE Optimizer Implementation
|
||
|
||
Replace `ArgminOptimizer` (PSO) with a Tree-Parzen Estimator:
|
||
|
||
1. **Surrogate model**: Maintain two kernel density estimates (KDEs) — one for parameters from "good" trials (top γ=25%), one for "bad" trials
|
||
2. **Acquisition function**: Expected Improvement (EI) = l(x)/g(x) where l(x) is the good KDE and g(x) is the bad KDE
|
||
3. **Sampling**: Draw candidates from l(x), score by EI, pick best
|
||
4. **Initial exploration**: Keep LHS (Latin Hypercube Sampling) for first 5 trials
|
||
|
||
The `egobox` crate (already in the workspace) provides EGO (GP-based BO) as an alternative, but TPE scales better to 25D. We'll implement a lightweight TPE using the `statrs` crate for KDE.
|
||
|
||
**Architecture**:
|
||
```
|
||
ArgminOptimizer (PSO, 20 particles, 50 iters/restart)
|
||
↓ replace with
|
||
TpeOptimizer {
|
||
n_initial: 5, // LHS warmup
|
||
max_trials: 30, // Same budget
|
||
gamma: 0.25, // Top 25% = "good"
|
||
n_candidates: 100, // EI candidates per trial
|
||
bandwidth: "silverman" // KDE bandwidth selection
|
||
}
|
||
```
|
||
|
||
**Files**:
|
||
- New: `crates/ml/src/hyperopt/tpe.rs` (~300 lines)
|
||
- Modify: `crates/ml/src/hyperopt/optimizer.rs` (add TPE variant)
|
||
- Modify: `crates/ml/src/hyperopt/mod.rs` (export)
|
||
- Modify: `crates/ml/src/hyperopt/campaign.rs` (wire TPE)
|
||
- Modify CLI: `bin/fxt/src/commands/train/hyperopt.rs` (--optimizer=tpe flag)
|
||
|
||
#### B2. Trial History & Warm-Starting
|
||
|
||
TPE benefits from trial history across runs. Add JSON-based trial persistence:
|
||
- Save all (params, objective) pairs to `hyperopt_dir/trial_history.json`
|
||
- On restart, load history to seed TPE's KDE
|
||
- This makes interrupted runs resume intelligently instead of starting from scratch
|
||
|
||
**Files**: `crates/ml/src/hyperopt/tpe.rs` (persistence methods)
|
||
|
||
## Non-Goals
|
||
|
||
- Changing the reward function or backtest environment (separate concern)
|
||
- Multi-GPU support (single L40S is the target)
|
||
- Changing the DQN architecture itself
|
||
- Modifying the FactoredAction space
|
||
|
||
## Expected Impact
|
||
|
||
| Metric | Current (PSO 45D) | Target (TPE 25D + fixes) |
|
||
|--------|-------------------|--------------------------|
|
||
| "Do nothing" rate | 57% (8/14) | <15% |
|
||
| Catastrophic DD rate | 29% (4/14) | <5% |
|
||
| Memory per trial | 844-1435 MB | <200 MB delta |
|
||
| Best Sharpe found | 1.73 (1 trial) | >2.0 consistently |
|
||
| Trials to find signal | ~7 | ~3-5 |
|
||
|
||
## Testing Strategy
|
||
|
||
- Unit tests for TPE KDE, EI acquisition, smooth penalties
|
||
- Integration test: 5-trial micro-hyperopt on synthetic data
|
||
- Regression: ensure existing PSO path still works (--optimizer=pso flag)
|