04d8802c94
Remove use_double_dqn, use_dueling, use_per, use_branching,
use_distributional, use_noisy_nets, use_huber_loss, and use_cql
from DQNConfig, DQNHyperparameters, and DqnParams structs.
These features are always enabled (Rainbow DQN standard). The boolean
flags were dead code — every constructor set them to true, and the
only code paths that set them to false were in tests that disabled
features for simplicity. With the fields removed, the features are
unconditionally active, eliminating ~490 lines of dead configuration.
Key changes:
- Struct field declarations removed from 3 core config structs
- Conditional branches (if use_X { ... } else { ... }) simplified:
dueling/branching/PER network creation is now unconditional
- Checkpoint metadata hardcodes "true" for backward compatibility
- Hyperopt search space index 11 (use_branching) fixed at 1.0
- TOML/YAML config files cleaned of removed fields
- Tests that toggled these flags updated or rewritten
45 files changed, -487 net lines. Zero new test failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
68 lines
2.6 KiB
TOML
68 lines
2.6 KiB
TOML
# DQN Best Hyperparameters
|
|
# Source: Historical training + manual tuning
|
|
# Date: 2025-11-02
|
|
# Status: PENDING - Awaiting hyperopt completion for optimal values
|
|
# Dataset: ES_FUT_180d.parquet
|
|
|
|
# NOTE: These are conservative defaults based on successful training runs.
|
|
# After completing DQN hyperopt (with action-dependent rewards fix), update
|
|
# this file with optimal parameters from the best trial.
|
|
|
|
# Learning Parameters
|
|
learning_rate = 0.0001 # Conservative LR for stable convergence
|
|
batch_size = 128 # Optimal for RTX 3050 Ti 4GB (max 230)
|
|
gamma = 0.99 # Discount factor (standard RL)
|
|
|
|
# Exploration Schedule (Epsilon-Greedy)
|
|
epsilon_start = 1.0 # Start with full exploration
|
|
epsilon_end = 0.01 # Minimum exploration (1%)
|
|
epsilon_decay = 0.995 # Decay rate per episode
|
|
|
|
# Replay Buffer Configuration
|
|
buffer_size = 100000 # Experience replay capacity
|
|
min_replay_size = 1000 # Minimum experiences before training
|
|
|
|
# Branching DQN (Tavakoli et al., 2018)
|
|
branch_hidden_dim = 128
|
|
|
|
# Training Configuration
|
|
epochs = 100 # Training epochs (default)
|
|
checkpoint_frequency = 10 # Save every 10 epochs
|
|
|
|
# Early Stopping Configuration
|
|
early_stopping_enabled = true
|
|
q_value_floor = 0.5 # Minimum Q-value threshold
|
|
min_loss_improvement_pct = 2.0 # Minimum 2% improvement over window
|
|
plateau_window = 30 # Epochs to check for plateau
|
|
min_epochs_before_stopping = 50 # Minimum training duration
|
|
|
|
# GPU Configuration
|
|
# Max batch size for RTX 3050 Ti 4GB: 230
|
|
# Max batch size for RTX A4000 16GB: 512+
|
|
# Recommended: 128 (safe for all GPUs)
|
|
|
|
# Historical Performance Notes
|
|
# - Training time: ~15s for 50 epochs (RTX 3050 Ti)
|
|
# - GPU memory: ~6MB model size
|
|
# - Inference: ~200μs per action
|
|
# - Tests: 16/16 passing (100% coverage)
|
|
|
|
# Known Issues (FIXED as of 2025-11-02)
|
|
# ✅ Action-dependent rewards implemented (lines 722-743, trainers/dqn.rs)
|
|
# ✅ Hyperopt objective fixed to maximize episode rewards (not loss)
|
|
# ✅ 16 comprehensive tests added (dqn_action_dependent_reward_test.rs)
|
|
# ⏳ Awaiting hyperopt deployment to find optimal hyperparameters
|
|
|
|
# Deployment
|
|
# After hyperopt completes, update this file with:
|
|
# - Optimal batch_size (expected: 128-512)
|
|
# - Optimal learning_rate (expected: 1e-5 to 1e-3)
|
|
# - Optimal gamma (expected: 0.95-0.99)
|
|
# - Optimal epsilon_decay (expected: 0.99-0.999)
|
|
|
|
# Related Files
|
|
# - DQN_ACTION_DEPENDENT_REWARDS_FIX_SUMMARY.md: Fix documentation
|
|
# - DQN_HYPEROPT_CORRECTED_QUICKREF.md: Hyperopt guide
|
|
# - ml/src/trainers/dqn.rs: Trainer implementation
|
|
# - deploy_dqn_hyperopt_corrected.sh: Deployment script
|