2df1ea92e1
BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
426 lines
45 KiB
Plaintext
426 lines
45 KiB
Plaintext
╔══════════════════════════════════════════════════════════════════════════════╗
|
|
║ DQN 2025 UPGRADE - VISUAL ROADMAP ║
|
|
║ ║
|
|
║ Full Docs: docs/codebase-cleanup/DQN_2025_UPGRADE_ROADMAP.md ║
|
|
║ Quick Ref: docs/codebase-cleanup/DQN_2025_UPGRADE_QUICK_REF.md ║
|
|
╚══════════════════════════════════════════════════════════════════════════════╝
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 📊 PRIORITY OVERVIEW │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Priority Items Timeline Expected Gain │
|
|
│ ──────────────────────────────────────────────────────────────────────── │
|
|
│ 🚨 P0 8 2-3 weeks +15-20% performance, Production stable │
|
|
│ ⚡ P1 12 2-3 weeks +10-15% performance │
|
|
│ ⭐ P2 9 1-2 weeks +5-10% performance │
|
|
│ │
|
|
│ 🎯 TOTAL POTENTIAL GAIN: +30-45% performance improvement │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🚨 P0: CRITICAL FIXES (Production Blockers) │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ WEEK 1: STABILITY FIXES (15 hours total) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ P0.1 ⚙️ L2 Weight Decay [2h] agent.rs:336-342 LOW │ │
|
|
│ │ ↳ Prevents overfitting, aligns with TFT/Mamba2 (1e-4) │ │
|
|
│ │ │ │
|
|
│ │ P0.2 🔒 TD-Error Clamping [4h] prioritized_replay.rs MED │ │
|
|
│ │ ↳ Prevents gradient explosion, Q-value stability +30% │ │
|
|
│ │ │ │
|
|
│ │ P0.3 📊 Batch Normalization [5h] network.rs, rainbow MED │ │
|
|
│ │ ↳ Reduces covariate shift, training stability +20% │ │
|
|
│ │ │ │
|
|
│ │ P0.4 ✂️ Gradient Clipping [4h] agent.rs:450-470 MED │ │
|
|
│ │ ↳ Prevents exploding gradients, enables higher LR │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ WEEK 2: ARCHITECTURE REFACTORING (29 hours total) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ P0.5 🏗️ Split dqn.rs Monolith [12h] dqn.rs (2396→500) HIGH │ │
|
|
│ │ ↳ Create core/ module: agent_base, training_loop, etc. │ │
|
|
│ │ ↳ Improves maintainability by 80%, enables parallel dev │ │
|
|
│ │ │ │
|
|
│ │ P0.6 🎲 Experience Diversity [5h] NEW diversity_tracker MED │ │
|
|
│ │ ↳ Prevents temporal correlation, sample efficiency +15% │ │
|
|
│ │ │ │
|
|
│ │ P0.7 ⏰ Stale Priority Detection [4h] prioritized_replay MED │ │
|
|
│ │ ↳ Ensures fresh priorities, reduces bias by 10-15% │ │
|
|
│ │ │ │
|
|
│ │ P0.8 💾 Memory Layout Optimization [8h] prioritized_replay HIGH │ │
|
|
│ │ ↳ Structure of Arrays (SoA), sampling speed +15-25% │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ✅ P0 SUCCESS METRICS: │
|
|
│ • Zero training divergences (gradient explosion eliminated) │
|
|
│ • Generalization gap < 5% (train vs validation) │
|
|
│ • Code quality: all files < 500 lines │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ ⚡ P1: IMPORTANT IMPROVEMENTS (SOTA Performance) │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ WEEK 3: ADVANCED REGULARIZATION (38 hours total) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ P1.1 🌊 Spectral Normalization [8h] NEW spectral_norm.rs HIGH │ │
|
|
│ │ ↳ Lipschitz constraint, prevents Q-value divergence │ │
|
|
│ │ │ │
|
|
│ │ P1.2 📉 Adaptive Dropout Scheduling [5h] NEW adaptive_dropout MED │ │
|
|
│ │ ↳ High dropout early → low late, generalization +5-10% │ │
|
|
│ │ │ │
|
|
│ │ P1.3 📏 Layer Normalization [4h] network.rs, rainbow MED │ │
|
|
│ │ ↳ Standardize across networks, convergence +5-10% faster │ │
|
|
│ │ │ │
|
|
│ │ P1.4 🎯 Hindsight Experience Replay[10h] NEW her.rs HIGH │ │
|
|
│ │ ↳ 🔥 5-10x data efficiency in sparse rewards! 🔥 │ │
|
|
│ │ │ │
|
|
│ │ P1.5 🔍 Curiosity Integration [6h] curiosity.rs (exists) MED │ │
|
|
│ │ ↳ Better exploration, sample efficiency +15-20% │ │
|
|
│ │ │ │
|
|
│ │ P1.6 📈 GAE Returns [5h] multi_step.rs MED │ │
|
|
│ │ ↳ Lower variance, faster convergence 10-20% │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ WEEK 4: FINAL OPTIMIZATIONS (33 hours total) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ P1.7 🎲 Rank-Based Sampling [4h] prioritized_replay MED │ │
|
|
│ │ ↳ More robust, generalization +5-8% │ │
|
|
│ │ │ │
|
|
│ │ P1.8 🔊 Noisy Network Tuning [3h] noisy_layers.rs LOW │ │
|
|
│ │ ↳ Learnable sigma, sample efficiency +8-12% │ │
|
|
│ │ │ │
|
|
│ │ P1.9 ⏱️ Delayed Updates [4h] dqn.rs MED │ │
|
|
│ │ ↳ Soft Polyak averaging, stability +20-30% │ │
|
|
│ │ │ │
|
|
│ │ P1.10 🎰 Ensemble Bootstrapping [4h] ensemble.rs (exists) MED │ │
|
|
│ │ ↳ Better uncertainty, ensemble +10-15% │ │
|
|
│ │ │ │
|
|
│ │ P1.11 📊 Quantile Regression [10h] NEW quantile_reg.rs HIGH │ │
|
|
│ │ ↳ Better risk modeling for trading, CVaR calculation │ │
|
|
│ │ │ │
|
|
│ │ P1.12 ⚡ AMP Training (FP16) [8h] agent.rs HIGH │ │
|
|
│ │ ↳ 🔥 2x training speedup, 50% memory reduction! 🔥 │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ✅ P1 SUCCESS METRICS: │
|
|
│ • Validation accuracy > 80% (+15% from baseline) │
|
|
│ • Training time < 4 hours (58s → 34s/epoch with AMP) │
|
|
│ • Sample efficiency < 500K experiences (from 1.2M) │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ ⭐ P2: NICE-TO-HAVE ENHANCEMENTS (Research Features) │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ P2.1 🌟 Dreamer Model-Based RL [25h] World model simulation │
|
|
│ P2.2 🧠 Meta-Learning (MAML) [20h] Fast adaptation to regimes │
|
|
│ P2.3 🔄 Successor Features [12h] Transfer learning across assets│
|
|
│ P2.4 📚 Curriculum Learning [8h] Automatic difficulty adjustment│
|
|
│ P2.5 🎭 Distributional SAC [30h] Risk-aware maximum entropy │
|
|
│ P2.6 💾 Offline RL (CQL) [15h] Learn from historical data │
|
|
│ P2.7 🔬 Causal Reasoning [25h] Causal regime detection │
|
|
│ P2.8 👁️ Attention Mechanisms [12h] Transformer-based encoder │
|
|
│ P2.9 🎓 Inverse RL (IRL) [30h] Learn from expert trades │
|
|
│ │
|
|
│ ⚠️ P2 items are future roadmap - implement based on business priority │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 📈 PERFORMANCE IMPACT SUMMARY │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ BASELINE (Current Implementation) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Validation Accuracy: 65-70% │ │
|
|
│ │ Training Time (500 ep): 8 hours │ │
|
|
│ │ Sample Efficiency: 1.2M experiences → 70% accuracy │ │
|
|
│ │ Convergence Rate: 80% runs converge │ │
|
|
│ │ GPU Memory Usage: 3.2GB (FP32) │ │
|
|
│ │ Largest File: dqn.rs (2,396 lines) ❌ │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ TARGET (After P0 + P1 Implementation) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Validation Accuracy: 80-85% ✅ (+15-20%) │ │
|
|
│ │ Training Time: 4 hours ✅ (-50%) │ │
|
|
│ │ Sample Efficiency: 500K exp → 80% accuracy ✅ (-58%) │ │
|
|
│ │ Convergence Rate: 95% runs converge ✅ (+15%) │ │
|
|
│ │ GPU Memory Usage: 1.8GB (FP16) ✅ (-44%) │ │
|
|
│ │ Largest File: 500 lines ✅ (-79%) │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ 🔥 CRITICAL WINS: │
|
|
│ • HER: 5-10x data efficiency boost (P1.4) │
|
|
│ • AMP: 2x training speedup (P1.12) │
|
|
│ • Code refactor: 80% maintainability improvement (P0.5) │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🗓️ TIMELINE & MILESTONES │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Week 1: Critical Stability Fixes │
|
|
│ ├─ Day 1-2: P0.1 Weight Decay + P0.4 Gradient Clipping │
|
|
│ ├─ Day 3-4: P0.2 TD-Error Clamping │
|
|
│ ├─ Day 5-7: P0.3 Batch Normalization + Testing │
|
|
│ └─ Milestone: Zero gradient explosions, stable training │
|
|
│ │
|
|
│ Week 2: Architecture Refactoring │
|
|
│ ├─ Day 1-3: P0.5 Split dqn.rs into core/ modules │
|
|
│ ├─ Day 4: P0.6 Experience Diversity Tracking │
|
|
│ ├─ Day 5: P0.7 Stale Priority Detection │
|
|
│ ├─ Day 6-7: P0.8 Memory Layout Optimization │
|
|
│ └─ Milestone: All files < 500 lines, 20% faster sampling │
|
|
│ │
|
|
│ Week 3: Advanced Regularization │
|
|
│ ├─ Day 1-2: P1.1 Spectral Norm + P1.3 Layer Norm │
|
|
│ ├─ Day 3: P1.2 Adaptive Dropout │
|
|
│ ├─ Day 4-5: P1.4 Hindsight Experience Replay ⭐ │
|
|
│ ├─ Day 6: P1.5 Curiosity Integration │
|
|
│ ├─ Day 7: P1.6 GAE Returns │
|
|
│ └─ Milestone: 5-10x sample efficiency with HER │
|
|
│ │
|
|
│ Week 4: Final Optimizations │
|
|
│ ├─ Day 1: P1.7-P1.9 (Rank sampling, Noisy tuning, Delayed updates) │
|
|
│ ├─ Day 2: P1.10 Ensemble Bootstrapping │
|
|
│ ├─ Day 3-4: P1.11 Quantile Regression │
|
|
│ ├─ Day 5-6: P1.12 AMP Training ⚡ │
|
|
│ ├─ Day 7: Integration testing + Deployment prep │
|
|
│ └─ Milestone: Production-ready, 2x speedup, 80%+ accuracy │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🧩 NEW MODULES & FILE STRUCTURE │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ BEFORE (Current): │
|
|
│ ml/src/dqn/ │
|
|
│ ├── dqn.rs (2,396 lines) ❌ TOO LARGE │
|
|
│ ├── agent.rs (1,354 lines) │
|
|
│ ├── network.rs (469 lines) │
|
|
│ ├── prioritized_replay.rs (670 lines) │
|
|
│ └── ... (37 other modules) │
|
|
│ │
|
|
│ AFTER (Target): │
|
|
│ ml/src/dqn/ │
|
|
│ ├── core/ ✅ NEW (from P0.5) │
|
|
│ │ ├── mod.rs (~50 lines) │
|
|
│ │ ├── agent_base.rs (~400 lines) │
|
|
│ │ ├── training_loop.rs (~500 lines) │
|
|
│ │ ├── experience_buffer.rs (~400 lines) │
|
|
│ │ ├── network_builder.rs (~300 lines) │
|
|
│ │ └── checkpoint.rs (~250 lines) │
|
|
│ ├── spectral_norm.rs ✅ NEW (P1.1) │
|
|
│ ├── adaptive_dropout.rs ✅ NEW (P1.2) │
|
|
│ ├── diversity_tracker.rs ✅ NEW (P0.6) │
|
|
│ ├── her.rs ✅ NEW (P1.4) │
|
|
│ ├── quantile_regression.rs ✅ NEW (P1.11) │
|
|
│ ├── agent.rs (1,354 lines → add AMP, grad clipping) │
|
|
│ ├── network.rs (469 lines → add BatchNorm, LayerNorm) │
|
|
│ ├── prioritized_replay.rs (670 lines → add diversity, stale detect) │
|
|
│ └── ... (existing modules) │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🔧 HYPERPARAMETER ADDITIONS │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ DQNHyperparameters struct additions (15 new fields): │
|
|
│ │
|
|
│ // P0 Critical │
|
|
│ weight_decay: 1e-4, // [1e-5, 1e-3] │
|
|
│ td_error_clip: 10.0, // [5.0, 20.0] │
|
|
│ max_grad_norm: 10.0, // [5.0, 50.0] │
|
|
│ use_batch_norm: true, │
|
|
│ batch_norm_momentum: 0.1, // [0.01, 0.2] │
|
|
│ │
|
|
│ // P1 Important │
|
|
│ use_spectral_norm: false, // Experimental │
|
|
│ dropout_schedule: Linear, // Linear, Cosine, Validation │
|
|
│ initial_dropout: 0.3, // [0.2, 0.5] │
|
|
│ final_dropout: 0.1, // [0.05, 0.2] │
|
|
│ use_gae: true, │
|
|
│ gae_lambda: 0.95, // [0.9, 0.99] │
|
|
│ use_her: false, // Enable for sparse rewards │
|
|
│ her_k: 4, // [2, 8] │
|
|
│ polyak_tau: 0.005, // [0.001, 0.01] │
|
|
│ use_amp: true, // Auto-detect CUDA │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🧪 TESTING REQUIREMENTS │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Per Feature Testing Strategy: │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ ✅ Unit Tests → cargo test --package ml <feature_name> │ │
|
|
│ │ ✅ Integration Tests → cargo test --package ml --test dqn_integration │ │
|
|
│ │ ✅ Benchmarks → cargo bench --package ml --bench training │ │
|
|
│ │ ✅ Ablation Study → Feature on/off comparison │ │
|
|
│ │ ✅ Regression Check → Performance vs baseline │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Coverage Requirements: │
|
|
│ • Unit test coverage: > 90% for all new code │
|
|
│ • Integration tests: Full training pipeline with all features │
|
|
│ • Performance tests: No regression vs baseline │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🔄 ROLLBACK & RISK MITIGATION │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Git Branching Strategy: │
|
|
│ main → Protected, production-ready │
|
|
│ ├── feature/p0-stability → P0.1-P0.4 stability fixes │
|
|
│ ├── feature/p0-refactor → P0.5 code refactoring │
|
|
│ ├── feature/p0-memory → P0.6-P0.8 memory optimizations │
|
|
│ ├── feature/p1-regularize → P1.1-P1.6 advanced regularization │
|
|
│ └── feature/p1-optimize → P1.7-P1.12 final optimizations │
|
|
│ │
|
|
│ Checkpoint Compatibility: │
|
|
│ ⚠️ Breaking changes: P0.3 BatchNorm, P1.1 SpectralNorm, P1.11 Quantile │
|
|
│ ✅ Solution: Checkpoint versioning with backward compatibility flags │
|
|
│ │
|
|
│ Emergency Rollback: │
|
|
│ 1. Git revert: git checkout <feature-branch> && git revert HEAD~3 │
|
|
│ 2. Feature flag: config.enable_<feature> = false; │
|
|
│ 3. Backup restore: mv <file>.rs.backup <file>.rs │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 📚 KEY REFERENCES & PAPERS │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Must-Read Papers (Top 5): │
|
|
│ 1. "Decoupled Weight Decay Regularization" (ICLR 2019) - P0.1 │
|
|
│ 2. "Prioritized Experience Replay" (ICLR 2016) - P0.2 │
|
|
│ 3. "Hindsight Experience Replay" (NeurIPS 2017) - P1.4 ⭐ │
|
|
│ 4. "High-Dimensional Continuous Control Using GAE" (ICLR 2016)- P1.6 │
|
|
│ 5. "Distributional RL with Quantile Regression" (AAAI 2018) - P1.11 │
|
|
│ │
|
|
│ Implementation References: │
|
|
│ • Spectral Norm: PyTorch torch.nn.utils.spectral_norm │
|
|
│ • AMP Training: NVIDIA Mixed Precision Guide │
|
|
│ • HER: OpenAI Baselines reference implementation │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🎯 SUCCESS METRICS & KPIs │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Phase 1: P0 Critical (Week 1-2) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ ✅ Zero training divergences (gradient explosions eliminated) │ │
|
|
│ │ ✅ Generalization gap < 5% (train vs validation) │ │
|
|
│ │ ✅ All files < 500 lines (code quality) │ │
|
|
│ │ ✅ Sampling speed +15-25% (memory optimization) │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Phase 2: P1 Important (Week 3-4) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ ✅ Validation accuracy > 80% (+15-20% from baseline) │ │
|
|
│ │ ✅ Training time < 4 hours (50% reduction) │ │
|
|
│ │ ✅ Sample efficiency < 500K experiences (58% reduction) │ │
|
|
│ │ ✅ Convergence rate > 95% (15% improvement) │ │
|
|
│ │ ✅ GPU memory usage < 2GB (44% reduction with AMP) │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Production Readiness: │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ ✅ Comprehensive test suite (95%+ coverage) │ │
|
|
│ │ ✅ Production deployment documentation │ │
|
|
│ │ ✅ Hyperparameter tuning results │ │
|
|
│ │ ✅ Rollback procedures documented │ │
|
|
│ │ ✅ Performance benchmarks validated │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 🚀 QUICK START COMMANDS │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ # Phase 1: Setup and baseline │
|
|
│ cargo clean │
|
|
│ cargo build --release --package ml │
|
|
│ cargo test --package ml --lib -- --test-threads=1 │
|
|
│ │
|
|
│ # Phase 2: Implement P0.1 (example) │
|
|
│ git checkout -b feature/p0-1-weight-decay │
|
|
│ # ... make changes to agent.rs ... │
|
|
│ cargo test --package ml weight_decay │
|
|
│ cargo bench --package ml --bench dqn_training_speed │
|
|
│ │
|
|
│ # Phase 3: Integration testing │
|
|
│ cargo test --package ml --test dqn_integration_test │
|
|
│ cargo test --package ml --test dqn_hyperopt_integration_test │
|
|
│ │
|
|
│ # Phase 4: Performance validation │
|
|
│ python ml/scripts/ablation_study.py --feature weight_decay │
|
|
│ python ml/scripts/compare_metrics.py --baseline v1.0 --current v2.0 │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
|
│ 👥 AGENT ASSIGNMENT RECOMMENDATIONS │
|
|
├──────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Week 1: Stability Fixes (4 agents in parallel) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Agent 1: P0.1 Weight Decay + P0.4 Gradient Clipping │ │
|
|
│ │ Agent 2: P0.2 TD-Error Clamping │ │
|
|
│ │ Agent 3: P0.3 Batch Normalization │ │
|
|
│ │ Agent 4: Testing & validation for P0.1-P0.4 │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Week 2: Architecture Refactoring (4 agents in parallel) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Agent 1: P0.5 Code refactoring (lead, complex task) │ │
|
|
│ │ Agent 2: P0.6 Experience Diversity Tracking │ │
|
|
│ │ Agent 3: P0.7 Stale Priority Detection │ │
|
|
│ │ Agent 4: P0.8 Memory Layout Optimization │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Week 3-4: Advanced Features (4 agents in parallel) │
|
|
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Agent 1: P1.1 Spectral Norm + P1.3 Layer Norm + smaller items │ │
|
|
│ │ Agent 2: P1.4 HER (dedicated, complex) │ │
|
|
│ │ Agent 3: P1.11 Quantile Regression (dedicated, complex) │ │
|
|
│ │ Agent 4: P1.12 AMP + remaining items + integration testing │ │
|
|
│ └────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────┘
|
|
|
|
╔══════════════════════════════════════════════════════════════════════════════╗
|
|
║ TOTAL ESTIMATED EFFORT ║
|
|
║ ║
|
|
║ P0 Critical: 44 hours (2-3 weeks) ║
|
|
║ P1 Important: 71 hours (2-3 weeks) ║
|
|
║ P2 Enhancement: 172 hours (1-2 weeks for selected items) ║
|
|
║ ║
|
|
║ Recommended Phase 1 (P0 + P1): 115 hours ≈ 4 weeks with 4-agent swarm ║
|
|
║ ║
|
|
║ 🎯 Expected Outcome: +30-45% performance, production-ready DQN 2025 ║
|
|
╚══════════════════════════════════════════════════════════════════════════════╝
|
|
|
|
For detailed implementation guides, see:
|
|
→ docs/codebase-cleanup/DQN_2025_UPGRADE_ROADMAP.md (Full details)
|
|
→ docs/codebase-cleanup/DQN_2025_UPGRADE_QUICK_REF.md (Quick reference)
|
|
|
|
Ready for swarm execution! 🚀
|