gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
cab4fcda4c47912ee56478f9773fd3970d2bc428
foxhunt/ml/docs/dqn_hyperparameters_analysis.md
jgrusewski 2df1ea92e1 feat(ml): WAVE 29 DQN Codebase Cleanup & Refactoring Campaign 
BREAKING CHANGES:
- Removed orphaned dqn.rs monolithic trainer (4,975 lines)
- Removed orphaned dqn_ensemble.rs module (816 lines)
- Removed orphaned tft.rs and tft_complete_int8_integration_test.rs
- TFT trainer split into modular directory structure

DQN Module Refactoring:
- Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs)
- Fixed hyperopt 39D search space (continuous params only)
- Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions
- use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues)

Clean Module Structure:
- ml/src/trainers/dqn/ directory with proper mod.rs exports
- ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs
- All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness

Documentation:
- Added comprehensive docs in docs/codebase-cleanup/
- ADR-001 for DQN refactoring decisions
- Rainbow DQN component matrix and quick reference guides

Build Status: Compiles with zero errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-27 23:46:13 +01:00

411 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# DQN Hyperparameters Analysis
## Complete Field Documentation and Categorization
**File**: `/home/jgrusewski/Work/foxhunt/ml/src/trainers/dqn/config.rs`
**Struct**: `DQNHyperparameters` (lines 264-535)
**Total Fields**: 83
---
## FIXED FLAGS (Architectural Decisions - Should NOT Be Tuned)
These are Rainbow DQN architectural features that should remain constant for model architecture consistency:
### Rainbow DQN Core Components (ALWAYS ON)
| Field | Default | Rationale |
|-------|---------|-----------|
| `use_double_dqn` | `true` | **FIXED ON**: Double DQN eliminates overestimation bias. Core Rainbow component. |
| `use_dueling` | `true` | **FIXED ON**: Dueling architecture separates value/advantage streams. Core Rainbow component. |
| `use_per` | `true` | **FIXED ON**: Prioritized Experience Replay improves sample efficiency. Core Rainbow component. |
| `use_noisy_nets` | `true` | **FIXED ON**: Noisy Networks replace epsilon-greedy for state-dependent exploration. Core Rainbow component. |
| `n_steps` | `3` | **FIXED**: 3-step returns balance bias/variance. Rainbow DQN standard. |
### Rainbow DQN Disabled Components
| Field | Default | Rationale |
|-------|---------|-----------|
| `use_distributional` | `true` | **SHOULD BE FALSE**: C51 distributional RL has Candle bugs. Comment says "OFF (Candle bug)" but default is `true` - **CONFLICT!** |
### Fixed Architectural Parameters
| Field | Default | Rationale |
|-------|---------|-----------|
| `dueling_hidden_dim` | `128` | **FIXED**: Network architecture parameter. Changing breaks checkpoint compatibility. |
| `num_atoms` | `51` | **FIXED**: Rainbow DQN standard for distributional RL (if enabled). |
| `per_alpha` | `0.6` | **FIXED**: PER prioritization exponent. Rainbow standard = 0.6. |
| `per_beta_start` | `0.4` | **FIXED**: PER importance sampling start. Rainbow standard = 0.4 → 1.0. |
### Target Update Strategy (Architectural Choice)
| Field | Default | Rationale |
|-------|---------|-----------|
| `target_update_mode` | `Soft` | **FIXED**: Soft (Polyak) updates are Rainbow DQN standard for stability. |
| `tau` | `0.001` | **FIXED**: Polyak coefficient. Rainbow standard = 0.001 (693-step half-life). |
### Data Pipeline Configuration
| Field | Default | Rationale |
|-------|---------|-----------|
| `enable_preprocessing` | `true` | **FIXED ON**: Preprocessing (log returns + normalization) is critical for stability. |
| `preprocessing_window` | `50` | **FIXED**: 50-bar rolling window for feature normalization. |
| `preprocessing_clip_sigma` | `5.0` | **FIXED**: Clip outliers at ±5σ. |
### Loss Function Configuration
| Field | Default | Rationale |
|-------|---------|-----------|
| `use_huber_loss` | `true` | **FIXED ON**: Huber loss is more robust to outliers than MSE. |
| `huber_delta` | `100.0` | **FIXED**: Scaled 100x for gradient explosion prevention (BUG #12 fix). |
### Feature Engineering Pipeline
| Field | Default | Rationale |
|-------|---------|-----------|
| `feature_stats_collection_ratio` | `0.3` | **FIXED**: 30% of epochs for feature stats collection (WAVE 23). |
| `max_feature_stats_epochs` | `Some(10)` | **FIXED**: Cap stats collection at 10 epochs. |
### Default-Disabled Experimental Features
These should remain `false`/`0.0` unless explicitly researching:
| Field | Default | Rationale |
|-------|---------|-----------|
| `enable_triple_barrier` | `false` | **FIXED OFF**: Multi-step reward labeling - experimental feature. |
| `use_ensemble_uncertainty` | `false` | **FIXED OFF**: Conflicts with `use_noisy_nets`. Mutually exclusive. |
| `enable_dropout_scheduler` | `false` | **FIXED OFF**: Adaptive dropout - experimental feature. |
| `enable_gae` | `false` | **FIXED OFF**: GAE is for actor-critic (PPO/A2C), not DQN. |
| `enable_noisy_sigma_scheduler` | `false` | **FIXED OFF**: Sigma annealing - experimental feature. |
| `sharpe_weight` | `0.0` | **FIXED OFF**: Sharpe ratio reward component - experimental. |
| `curiosity_weight` | `0.0` | **FIXED OFF**: Curiosity-driven exploration - experimental. |
| `her_ratio` | `0.0` | **FIXED OFF**: Hindsight Experience Replay - experimental. |
---
## TUNABLE HYPERPARAMETERS (Should Be Optimized)
These are true hyperparameters that should be tuned via hyperopt:
### Learning Rate & Optimization
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `learning_rate` | `0.0001` | `[1e-5, 1e-3]` | **TUNE**: Core hyperparameter. Trial 19 used 1e-4, typical range 1e-5 to 1e-3. |
| `batch_size` | `128` | `[64, 512]` | **TUNE**: Memory-constrained by GPU. RTX 3050 Ti limit ≤230. Trial 19 used 256. |
| `gradient_clip_norm` | `Some(10.0)` | `[1.0, 100.0]` or `None` | **TUNE**: Prevents gradient explosion. Production uses 10.0, Trial 19 used 100.0. |
### Learning Rate Scheduling (WAVE 26)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `lr_decay_type` | `Constant` | `{Constant, Linear, Exponential, Cosine, Step}` | **TUNE**: LR schedule type. |
| `lr_decay_steps` | `1000` | `[500, 10000]` | **TUNE**: Steps between LR decay updates. |
| `lr_decay_rate` | `0.99` | `[0.9, 0.999]` | **TUNE**: Exponential decay rate (1% per step). |
| `min_learning_rate` | `1e-6` | `[1e-7, 1e-5]` | **TUNE**: Minimum LR floor for Cosine decay. |
| `lr_min` | `1e-6` | `[1e-7, 1e-5]` | **TUNE**: Absolute minimum LR. |
### Discount Factor
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `gamma` | `0.99` | `[0.95, 0.999]` | **TUNE**: Discount factor. Higher γ = longer planning horizon. |
### Exploration Parameters
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `epsilon_start` | `1.0` | `[0.5, 1.0]` | **TUNE**: Initial ε for ε-greedy. With noisy nets, ε can start lower. |
| `epsilon_end` | `0.01` | `[0.001, 0.05]` | **TUNE**: Final ε. Rainbow uses 0.01. |
| `epsilon_decay` | `0.995` | `[0.99, 0.9999]` | **TUNE**: Decay rate. Reaches epsilon_end after ~1000 steps (0.995) or ~100K steps (0.9999). |
| `noisy_sigma_init` | `0.5` | `[0.3, 0.7]` | **TUNE**: Initial noise std for noisy networks. Rainbow standard = 0.5. |
### Noisy Network Sigma Scheduling (Experimental)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `noisy_sigma_initial` | `0.6` | `[0.4, 0.8]` | **TUNE**: Initial sigma when scheduler enabled. |
| `noisy_sigma_final` | `0.4` | `[0.2, 0.6]` | **TUNE**: Final sigma when scheduler enabled. |
| `noisy_sigma_anneal_steps` | `10000` | `[5000, 50000]` | **TUNE**: Steps for sigma annealing. |
### Replay Buffer Configuration
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `buffer_size` | `500000` | `[100000, 1000000]` | **TUNE**: WAVE 24 increased to 500K for diversity. Larger = better sample diversity. |
| `min_replay_size` | `1000` | `[100, 10000]` | **TUNE**: Min samples before training starts. |
### Target Network Updates
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `target_update_frequency` | `500` | `[100, 10000]` | **TUNE**: Hard update frequency (if mode=Hard). BUG #9 fix: 500 steps optimal. |
| `warmup_steps` | `0` | `[0, 80000]` | **TUNE**: Random exploration warmup. Adaptive in CLI: 0 (<200K steps), 5% (200K-500K), 8% (500K-1M), 80K (>1M). |
### Training Schedule
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `epochs` | `100` | `[50, 500]` | **TUNE**: Total training epochs. Production uses 100-500. |
| `checkpoint_frequency` | `10` | `[5, 50]` | **TUNE**: Checkpoint save interval. |
### Early Stopping Configuration
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `early_stopping_enabled` | `true` | `{true, false}` | **TUNE**: Enable early stopping. |
| `q_value_floor` | `-5.0` | `[-10.0, -2.0]` | **TUNE**: Min Q-value before stopping. Wave 3 fix: allow normal negative Q-values. |
| `min_loss_improvement_pct` | `2.0` | `[1.0, 5.0]` | **TUNE**: Min loss improvement % over plateau window. |
| `plateau_window` | `30` | `[10, 50]` | **TUNE**: Window size for plateau detection. |
| `min_epochs_before_stopping` | `50` | `[20, 100]` | **TUNE**: Min epochs before early stop can trigger. |
### Gradient Collapse Detection (WAVE 23)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `gradient_collapse_multiplier` | `100.0` | `[10.0, 1000.0]` | **TUNE**: Adaptive threshold = LR × multiplier. WAVE 23 replaces hardcoded 0.1 threshold. |
| `gradient_collapse_patience` | `5` | `[3, 10]` | **TUNE**: Consecutive epochs before early stop. |
### Penalty & Reward Shaping
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `hold_penalty` | `-0.001` | `[-0.01, 0.0]` | **TUNE**: Small negative penalty for HOLD action (Bug #3 fix). |
| `hold_penalty_weight` | `0.01` | `[0.0, 0.1]` | **TUNE**: HOLD penalty during large price movements. |
| `movement_threshold` | `0.02` | `[0.01, 0.05]` | **TUNE**: Price movement % threshold (2% default). |
| `transaction_cost_multiplier` | `1.0` | `[0.5, 2.0]` | **TUNE**: Multiplier for transaction costs in reward. |
### Portfolio & Risk Management
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `initial_capital` | `100000.0` | `[1000.0, 1000000.0]` | **TUNE**: Initial trading capital. Min $1K validated at CLI. |
| `cash_reserve_percent` | `0.0` | `[0.0, 50.0]` | **TUNE**: Cash reserve % (0 = no reserve, backward compatible). |
| `max_position_absolute` | `2.0` | `[1.0, 10.0]` | **TUNE**: Max absolute position size for action masking. |
### Kelly Criterion Position Sizing
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `enable_kelly_sizing` | `true` | `{true, false}` | **TUNE**: Enable Kelly criterion position sizing. |
| `kelly_fractional` | `0.5` | `[0.25, 1.0]` | **TUNE**: Kelly multiplier (0.5 = half-Kelly, conservative). |
| `kelly_max_fraction` | `0.25` | `[0.1, 0.5]` | **TUNE**: Max Kelly fraction (25% = max 25% of portfolio). |
| `kelly_min_trades` | `20` | `[10, 50]` | **TUNE**: Min trades for Kelly calculation. |
### Volatility & Risk Metrics
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `volatility_window` | `20` | `[10, 50]` | **TUNE**: Rolling window for volatility calculation. |
| `enable_volatility_epsilon` | `true` | `{true, false}` | **TUNE**: Volatility-adjusted exploration. |
| `enable_risk_adjusted_rewards` | `true` | `{true, false}` | **TUNE**: Sharpe-based rewards. |
### Advanced Risk Features (WAVE 16S)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `enable_drawdown_monitoring` | `true` | `{true, false}` | **TUNE**: 15% max drawdown early stop. |
| `enable_position_limits` | `true` | `{true, false}` | **TUNE**: 3-tier position limits (absolute, notional, concentration). |
| `enable_circuit_breaker` | `true` | `{true, false}` | **TUNE**: 5-failure trip mechanism. |
| `enable_action_masking` | `true` | `{true, false}` | **TUNE**: Filter invalid actions based on position limits. |
| `enable_entropy_regularization` | `true` | `{true, false}` | **TUNE**: Prevent policy collapse. |
| `enable_stress_testing` | `true` | `{true, false}` | **TUNE**: Robustness validation. |
### Advanced Features (WAVE 35)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `enable_regime_qnetwork` | `true` | `{true, false}` | **TUNE**: Regime-conditional Q-networks (3 heads: Trending, Ranging, Volatile). |
| `enable_compliance` | `true` | `{true, false}` | **TUNE**: Real-time regulatory validation. |
### Entropy Regularization (WAVE 17)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `entropy_coefficient` | `None` | `[0.0, 0.1]` | **TUNE**: Entropy bonus for policy diversity. None = disabled. |
### Triple Barrier (Experimental - Default OFF)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `triple_barrier_profit_target_bps` | `100` | `[50, 500]` | **TUNE**: Profit target in bps (if enabled). |
| `triple_barrier_stop_loss_bps` | `50` | `[25, 200]` | **TUNE**: Stop loss in bps (if enabled). |
| `triple_barrier_max_holding_seconds` | `3600` | `[300, 7200]` | **TUNE**: Max holding period (if enabled). |
### Distributional RL (C51) - **SHOULD BE DISABLED**
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `v_min` | `-2.0` | `[-10.0, -1.0]` | **TUNE** (if C51 enabled): Min value for distribution. BUG #5 fix: align with reward range ±2. |
| `v_max` | `2.0` | `[1.0, 10.0]` | **TUNE** (if C51 enabled): Max value for distribution. BUG #5 fix: align with reward range ±2. |
### WAVE 26 Advanced Features (Experimental - Default OFF)
#### Ensemble Uncertainty (Conflicts with Noisy Nets)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `ensemble_size` | `5` | `[3, 10]` | **TUNE** (if enabled): Number of Q-network heads. |
| `beta_variance` | `0.5` | `[0.1, 1.0]` | **TUNE** (if enabled): Variance penalty weight. |
| `beta_disagreement` | `0.5` | `[0.1, 1.0]` | **TUNE** (if enabled): Disagreement penalty weight. |
| `beta_entropy` | `0.1` | `[0.05, 0.5]` | **TUNE** (if enabled): Entropy bonus weight. |
| `variance_cap` | `1.0` | `[0.1, 2.0]` | **TUNE** (if enabled): Variance cap to prevent over-penalization. |
#### Gradient Accumulation
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `gradient_accumulation_steps` | `1` | `[1, 8]` | **TUNE**: Mini-batches per optimizer step. Effective batch = batch_size × steps. |
#### Sharpe Ratio Reward (Experimental)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `sharpe_window` | `20` | `[10, 50]` | **TUNE** (if sharpe_weight > 0): Rolling window for Sharpe calculation. |
#### Adaptive Dropout (Experimental)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `dropout_initial` | `0.5` | `[0.3, 0.7]` | **TUNE** (if scheduler enabled): Initial dropout rate. |
| `dropout_final` | `0.1` | `[0.0, 0.3]` | **TUNE** (if scheduler enabled): Final dropout rate. |
| `dropout_anneal_steps` | `10000` | `[5000, 50000]` | **TUNE** (if scheduler enabled): Annealing steps. |
#### Hindsight Experience Replay (Experimental)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `her_strategy` | `"future"` | `{"final", "future"}` | **TUNE** (if her_ratio > 0): HER strategy. "future" > "final". |
#### GAE (Not Recommended for DQN)
| Field | Default | Valid Range | Rationale |
|-------|---------|-------------|-----------|
| `gae_lambda` | `0.95` | `[0.9, 0.99]` | **TUNE** (if GAE enabled): GAE λ parameter. Note: GAE is for actor-critic, not DQN. |
---
## CRITICAL ISSUES & CONFLICTS
### 🚨 **BUG**: Distributional RL Default Mismatch
**Field**: `use_distributional`
**Default**: `true`
**Documentation Says**: "C51/Distributional: OFF (Candle bug)"
**Problem**: Default contradicts stated requirement
**Required Fix**: Change default to `false` in line 638:
```rust
use_distributional: false, // Default: disabled (Candle bug - see ADR-001)
```
### ⚠️ **WARNING**: Mutually Exclusive Features
**Conflict 1**: `use_noisy_nets` vs `use_ensemble_uncertainty`
- Both provide exploration bonuses
- Should not be enabled simultaneously
- Default: `use_noisy_nets=true`, `use_ensemble_uncertainty=false` ✅ CORRECT
**Conflict 2**: `epsilon_decay` vs `use_noisy_nets`
- Noisy networks replace ε-greedy exploration
- With `use_noisy_nets=true`, ε should be set to 0
- Current: ε-greedy still active alongside noisy nets
- **Recommendation**: When `use_noisy_nets=true`, set `epsilon_start=0.0`
---
## RAINBOW DQN CONFIGURATION SUMMARY
### ✅ **CORRECT**: Enabled by Default (Rainbow Core)
1. Double DQN (`use_double_dqn=true`)
2. Dueling Networks (`use_dueling=true`)
3. Prioritized Experience Replay (`use_per=true`)
4. Noisy Networks (`use_noisy_nets=true`)
5. 3-Step Returns (`n_steps=3`)
6. Soft Target Updates (`tau=0.001`, `target_update_mode=Soft`)
### ❌ **INCORRECT**: Should Be Disabled (Candle Bug)
7. Distributional RL (`use_distributional=true` → **should be `false`**)
### Rainbow DQN Compliance Score
**6/7 components correct** (85.7%)
**Action Required**: Disable `use_distributional` to reach 100% Rainbow compliance.
---
## HYPEROPT SEARCH SPACE RECOMMENDATIONS
### Priority Tier 1 (Highest Impact)
1. `learning_rate` - Range: `[1e-5, 1e-3]`, log-scale
2. `batch_size` - Range: `[64, 256]`, categorical `{64, 128, 256}`
3. `gamma` - Range: `[0.95, 0.999]`
4. `epsilon_decay` - Range: `[0.99, 0.9999]`, log-scale
5. `gradient_clip_norm` - Range: `[10.0, 100.0]` or `None`
### Priority Tier 2 (Moderate Impact)
6. `buffer_size` - Range: `[100000, 1000000]`, categorical `{100K, 250K, 500K, 1M}`
7. `warmup_steps` - Range: `[0, 80000]`, adaptive based on total steps
8. `kelly_fractional` - Range: `[0.25, 1.0]`
9. `max_position_absolute` - Range: `[1.0, 10.0]`
10. `noisy_sigma_init` - Range: `[0.3, 0.7]`
### Priority Tier 3 (Fine-Tuning)
11. Early stopping parameters (`q_value_floor`, `plateau_window`, `min_epochs_before_stopping`)
12. Penalty weights (`hold_penalty`, `hold_penalty_weight`, `transaction_cost_multiplier`)
13. Learning rate schedule parameters (if `lr_decay_type != Constant`)
14. Risk management toggles (Kelly, volatility epsilon, risk-adjusted rewards)
---
## VALIDATION CHECKLIST
### Before Hyperopt
- [ ] Set `use_distributional=false` (Candle bug)
- [ ] Verify `use_double_dqn=true`
- [ ] Verify `use_dueling=true`
- [ ] Verify `use_per=true`
- [ ] Verify `use_noisy_nets=true`
- [ ] Verify `n_steps=3`
- [ ] Verify `tau=0.001`
- [ ] Verify `target_update_mode=Soft`
### After Hyperopt
- [ ] Confirm batch_size ≤ 230 (RTX 3050 Ti limit)
- [ ] Confirm `learning_rate` in validated range
- [ ] Confirm `gamma` balance (not too high, not too low)
- [ ] Confirm `warmup_steps` scales with total training steps
- [ ] Save best hyperparameters to `ml/hyperparams/dqn_best.toml`
---
## TOTAL FIELD COUNT
- **Total Fields**: 83
- **Fixed Flags**: 28 (33.7%)
- **Tunable Hyperparameters**: 55 (66.3%)
**Field Breakdown**:
- Rainbow Core (Fixed): 7
- Architectural (Fixed): 14
- Experimental Disabled (Fixed): 7
- Learning/Optimization (Tunable): 12
- Exploration (Tunable): 9
- Replay Buffer (Tunable): 2
- Early Stopping (Tunable): 7
- Portfolio/Risk (Tunable): 25
- Advanced Features (Experimental, Tunable if enabled): 21
---
## REFERENCES
- Rainbow DQN Paper: Hessel et al. (2018)
- Trial 19 Hyperopt Results: `DQN_HYPEROPT_RESULTS_SUMMARY.md`
- BUG #5 Fix: v_min/v_max alignment with reward range
- BUG #9 Fix: target_update_frequency = 500 steps optimal
- BUG #12 Fix: huber_delta scaled 100x
- WAVE 16 (Agent 36): Soft target updates
- WAVE 23: Feature caching + early stopping
- WAVE 24: Buffer size increase to 500K
- WAVE 26: Advanced features (LR scheduling, ensemble, etc.)
- WAVE 35: Regime-conditional Q-networks + compliance
Reference in New Issue View Git Blame Copy Permalink