9c3d741a08
Move 17 library crates into crates/, CLI binary into bin/fxt, consolidate 10 test crates into testing/, split config crate from deployment config files. Root directory reduced from 38+ to ~17 directories. All Cargo.toml paths and build.rs proto refs updated. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
139 lines
3.7 KiB
Markdown
139 lines
3.7 KiB
Markdown
# ML Hyperparameter Configurations
|
|
|
|
This directory contains canonical hyperparameter configurations for ML models based on hyperopt optimization results.
|
|
|
|
## Files
|
|
|
|
### `ppo_best.toml`
|
|
- **Source**: Hyperopt Trial #1 (Pod bpxgh10c5ocus5)
|
|
- **Date**: 2025-11-01
|
|
- **Objective**: 2.4023 (best of 63 trials)
|
|
- **Duration**: 14.3 minutes
|
|
- **Cost**: $0.06
|
|
|
|
**Key Parameters**:
|
|
- Policy LR: 1.0e-6 (ultra-conservative)
|
|
- Value LR: 0.001 (aggressive, 1000x higher)
|
|
- Clip Epsilon: 0.1126
|
|
- Entropy Coef: 0.006142
|
|
|
|
**Status**: ✅ Production-ready
|
|
|
|
### `dqn_best.toml`
|
|
- **Source**: Conservative defaults (awaiting hyperopt)
|
|
- **Date**: 2025-11-02
|
|
- **Status**: ⏳ Pending hyperopt deployment
|
|
|
|
**Current Parameters**:
|
|
- Learning Rate: 0.0001
|
|
- Batch Size: 128
|
|
- Gamma: 0.99
|
|
- Epsilon Decay: 0.995
|
|
|
|
**Note**: These are conservative defaults. Update after DQN hyperopt completes with action-dependent rewards.
|
|
|
|
## Usage
|
|
|
|
### In Code (Recommended)
|
|
|
|
Use the `conservative()` method for testing and development:
|
|
|
|
```rust
|
|
use ml::trainers::ppo::PpoHyperparameters;
|
|
use ml::trainers::dqn::DQNHyperparameters;
|
|
|
|
// PPO
|
|
let ppo_params = PpoHyperparameters::conservative();
|
|
|
|
// DQN
|
|
let dqn_params = DQNHyperparameters::conservative();
|
|
```
|
|
|
|
### Loading from TOML (Production)
|
|
|
|
For production deployments, load from TOML files:
|
|
|
|
```rust
|
|
use std::fs;
|
|
use toml;
|
|
|
|
// Load PPO hyperparameters
|
|
let ppo_config = fs::read_to_string("ml/hyperparams/ppo_best.toml")?;
|
|
let ppo_params: PpoHyperparameters = toml::from_str(&ppo_config)?;
|
|
|
|
// Load DQN hyperparameters
|
|
let dqn_config = fs::read_to_string("ml/hyperparams/dqn_best.toml")?;
|
|
let dqn_params: DQNHyperparameters = toml::from_str(&dqn_config)?;
|
|
```
|
|
|
|
## Why No Default Implementation?
|
|
|
|
The `Default` trait has been **removed** from `PpoHyperparameters` and `DQNHyperparameters` to prevent accidental use of suboptimal hyperparameters in production.
|
|
|
|
### Previous Issue (PPO)
|
|
|
|
Using `Default::default()` caused loss stagnation in production:
|
|
- Pod 0hczpx9nj1ub88: Loss stuck at 1.158-1.159 for 200+ epochs
|
|
- Root cause: Single learning rate (0.001) was 1000x too high for policy network
|
|
- Cost: ~$0.10 wasted compute
|
|
|
|
### Solution
|
|
|
|
1. **Development/Testing**: Use `::conservative()` method
|
|
2. **Production**: Load from TOML files (this directory)
|
|
3. **Optimization**: Run hyperopt to find optimal parameters
|
|
|
|
## Hyperopt History
|
|
|
|
### PPO Hyperopt
|
|
- **Date**: 2025-11-01
|
|
- **Pod**: bpxgh10c5ocus5 (EUR-IS-1, RTX A4000)
|
|
- **Trials**: 63 (target: 50)
|
|
- **Duration**: 14.3 minutes (99.8% faster than estimate)
|
|
- **Cost**: $0.06 (98.7% cheaper than estimate)
|
|
- **Best Trial**: #1 (objective: 2.4023)
|
|
|
|
### DQN Hyperopt
|
|
- **Status**: ⏳ Pending
|
|
- **Fix Applied**: Action-dependent rewards (2025-11-02)
|
|
- **Expected**: 50 trials, ~25 minutes, $0.12
|
|
- **Note**: Previous hyperopt produced identical objectives (bug fixed)
|
|
|
|
## Related Documentation
|
|
|
|
- **PPO**: `PPO_PARAMETERS_QUICK_REF.md` (root directory)
|
|
- **DQN**: `DQN_ACTION_DEPENDENT_REWARDS_FIX_SUMMARY.md` (root directory)
|
|
- **General**: `CLAUDE.md` (system architecture, hyperopt results)
|
|
|
|
## Updating Hyperparameters
|
|
|
|
After running hyperopt:
|
|
|
|
1. Identify best trial (highest objective for PPO, lowest for DQN)
|
|
2. Extract hyperparameters from trial results
|
|
3. Update corresponding `.toml` file
|
|
4. Document source (pod ID, trial number, objective score)
|
|
5. Update this README with new metadata
|
|
|
|
## Testing
|
|
|
|
All trainer tests use `::conservative()` method:
|
|
|
|
```bash
|
|
# Test PPO trainer
|
|
cargo test -p ml --lib trainers::ppo::tests
|
|
|
|
# Test DQN trainer
|
|
cargo test -p ml --lib trainers::dqn::tests
|
|
|
|
# All trainer tests
|
|
cargo test -p ml --lib trainers
|
|
```
|
|
|
|
**Status**: ✅ 42/42 trainer tests passing
|
|
|
|
---
|
|
|
|
**Last Updated**: 2025-11-02
|
|
**Maintainer**: ML Training Team
|