2df1ea92e1
BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
39 lines
1.4 KiB
Plaintext
39 lines
1.4 KiB
Plaintext
feat(dqn): WAVE 26 P1.12 - Verify and enhance Polyak soft updates
|
|
|
|
Comprehensive verification and enhancement of Polyak averaging (soft
|
|
target updates) for Rainbow DQN implementation with hyperopt integration.
|
|
|
|
Changes:
|
|
- ✅ Verified tau defaults to 0.001 (Rainbow DQN standard)
|
|
- ✅ Added tau to hyperopt search space (30D, log-scale 0.0001-0.01)
|
|
- ✅ Implemented compute_network_divergence() for monitoring
|
|
- ✅ Created 13 comprehensive TDD tests
|
|
- ✅ Fixed duplicate tau field bug in DQNParams
|
|
- ✅ Fixed ActivationType import in activation_tests.rs
|
|
|
|
Files Modified:
|
|
- ml/src/dqn/target_update.rs - Added divergence computation
|
|
- ml/src/hyperopt/adapters/dqn.rs - 29D→30D search space
|
|
- ml/src/dqn/tests/target_update_comprehensive_tests.rs (NEW)
|
|
- ml/src/dqn/tests/mod.rs - Registered new test module
|
|
- ml/src/dqn/tests/activation_tests.rs - Import fix
|
|
|
|
Test Coverage:
|
|
- Formula verification (θ_target = (1-τ)*θ_target + τ*θ_online)
|
|
- Boundary conditions (tau ∈ [0.0, 1.0])
|
|
- Convergence rates (693-step half-life for tau=0.001)
|
|
- Network divergence computation (L2 norm)
|
|
- Multi-layer consistency
|
|
|
|
Convergence Half-Life:
|
|
- tau=0.0001 → 6931 steps (ultra-stable)
|
|
- tau=0.001 → 693 steps (Rainbow DQN standard)
|
|
- tau=0.005 → 138 steps (5x faster)
|
|
- tau=0.01 → 69 steps (10x faster)
|
|
|
|
Production Ready: ✅ YES
|
|
Hyperopt Ready: ✅ YES
|
|
Test Coverage: ✅ COMPREHENSIVE (13 tests)
|
|
|
|
Report: docs/WAVE26_P1.12_POLYAK_SOFT_UPDATES_REPORT.md
|