gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Bug #29 fix: Per-epoch epsilon decay for hyperopt stability

Browse Source 
Root Cause:
- Previous per-batch epsilon decay caused premature exploration collapse
- With batch_size=72, epsilon hit floor (0.05) after 2.1 epochs
- Resulted in 2.2% action diversity (1/45 actions used)

Fix Applied:
- Moved epsilon decay from per-batch to per-epoch
- After 15 epochs: epsilon = 0.3 × (0.995^15) = 0.2783 (27.8% exploration)
- Ensures consistent exploration across different batch sizes

Expected Impact:
- Action diversity: 2.2% → 50-100%
- Q-values: Negative (Bug #30) → Positive (secondary fix)
- Trial success rate: 25% → 75-100%

Files Modified:
- ml/src/trainers/dqn.rs (lines 1265-1267, 1330-1336)

Bug #30 Status:
- Closed as secondary to Bug #29
- Q-value instability was mathematical consequence of single-action learning
- Will automatically resolve when action diversity restored
This commit is contained in:
jgrusewski
2025-11-14 20:59:37 +01:00
parent 15496deb1d
commit ec2ff34aea
1 changed files with 11 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
ml/src/trainers/dqn.rs
View File

@@ -1262,13 +1262,9 @@ impl DQNTrainer {
// WAVE 9-11: Track Q-value range for production monitoring
monitor.track_q_value_range(q_value);
// WAVE 11 FIX: Update epsilon per training step (not per epoch)
// This ensures epsilon decays properly: epsilon_start * (epsilon_decay ** step)
// With epsilon_decay=0.995, epsilon reaches 0.05 at step ~2,977 (~2.1 epochs)
{
let mut agent = self.agent.write().await;
agent.update_epsilon();
}
// BUG #29 FIX: Epsilon decay moved to per-epoch (see line ~1340)
// Previous per-batch decay caused premature exploration collapse in short hyperopt trials
// With batch_size=72, epsilon hit floor (0.05) after 2.1 epochs, freezing at 2.2% diversity
},
Err(e) => {
// Log error but continue training (some batches may fail due to sampling issues)
@@ -1331,6 +1327,14 @@ impl DQNTrainer {
epoch_duration.as_secs_f64()
);
// BUG #29 FIX: Update epsilon once per epoch (not per batch)
// This ensures consistent exploration across different batch sizes
// With epsilon_decay=0.995, after 15 epochs: 0.3 × (0.995^15) = 0.2783 (27.8% exploration)
{
let mut agent = self.agent.write().await;
agent.update_epsilon();
}
// WAVE 9-11: Log Q-value range for production monitoring
if train_step_count > 0 {
info!(