CRITICAL FIX: Enable soft updates in hyperopt adapter

Root Cause Found:
- Hyperopt adapter hardcoded tau=1.0 (hard updates) at line 412
- This OVERRODE the default tau=0.001 we set in dqn.rs
- Result: 100% trial pruning rate (gradient explosion 10K-16K)

Fix Applied:
- ml/src/hyperopt/adapters/dqn.rs lines 411-415
- Changed: tau: 1.0 → 0.001
- Changed: TargetUpdateMode::Hard → Soft
- Changed: target_update_frequency: 10000 → 1

Expected Impact:
- Gradient norms: 10K-16K → 50-500
- Trial success rate: 0% → 90-100%
- Q-value stability: Prevents explosion feedback loop

User Insight:
User correctly identified we were 'going in circles' -
changing defaults but hyperopt ignored them. This fix
addresses the actual running code path.

Testing: 2-trial validation running now
This commit is contained in:
jgrusewski
2025-11-14 23:44:22 +01:00
parent 46807e373c
commit 46fea9a0e3

View File

@@ -408,10 +408,11 @@ impl DQNTrainer {
early_stopping_plateau_window: 5, // Default: 5 epochs (hyperopt optimized)
early_stopping_min_epochs: 1000, // Default: 1000 (effectively disabled - Wave 7 validation)
trial_counter: 0, // Start at trial 0
// WAVE 16 (Agent 38): Switched to Hard updates for stability
tau: 1.0, // Hard updates use full copy
target_update_mode: crate::trainers::TargetUpdateMode::Hard, // Hard updates (Stable Baselines3)
target_update_frequency: 10000, // Stable Baselines3 standard: 10K steps
// FIX: Enable soft updates (root cause of gradient explosion)
// Hard updates were causing 10K-16K gradient norms (Wave 16H baseline: 1,707)
tau: 0.001, // Soft updates: 0.1% target blend per step (Rainbow DQN standard)
target_update_mode: crate::trainers::TargetUpdateMode::Soft, // Soft updates prevent Q-value explosion
target_update_frequency: 1, // Update every step with soft blending
enable_preprocessing: true, // Preprocessing enabled by default (Wave 14)
preprocessing_window: 50, // Default: 50-bar rolling window
preprocessing_clip_sigma: 5.0, // Default: clip at ±5σ