foxhunt/docs/codebase-cleanup/DQN_2025_UPGRADE_QUICK_REF.md

DQN 2025 Upgrade - Quick Reference Card

Last Updated: 2025-11-27 Full Roadmap: DQN_2025_UPGRADE_ROADMAP.md

---

🎯 Priority Overview

Priority	Count	Timeline	Expected Gain
P0 Critical	8 items	2-3 weeks	+15-20% performance, Production stability
P1 Important	12 items	2-3 weeks	+10-15% performance
P2 Enhancement	9 items	1-2 weeks	+5-10% performance

Total Potential Gain: +30-45% performance improvement

---

🚨 P0: Critical Fixes (Production Blockers)

Quick Action Items

# Week 1: Stability Fixes (15 hours)
1. Add L2 Weight Decay          (2h)  → Prevents overfitting
2. TD-Error Clamping            (4h)  → Stops gradient explosion
3. Batch Normalization          (5h)  → Training stability +20%
4. Gradient Clipping            (4h)  → Prevents exploding gradients

# Week 2: Architecture (29 hours)
5. Split dqn.rs (2396→500)     (12h)  → Maintainable codebase
6. Experience Diversity         (5h)  → Better sample quality
7. Stale Priority Detection     (4h)  → Fresh priorities
8. Memory Layout Optimization   (8h)  → 15-25% faster sampling

P0 Files to Modify

Item	File	Lines	Complexity
P0.1	agent.rs	336-342	LOW
P0.2	prioritized_replay.rs	145-155	MEDIUM
P0.3	network.rs, rainbow_network.rs	104-122	MEDIUM
P0.4	agent.rs	450-470	MEDIUM
P0.5	dqn.rs (2396 lines)	FULL FILE	HIGH
P0.6	prioritized_replay.rs + NEW	NEW MODULE	MEDIUM
P0.7	prioritized_replay.rs	NEW FIELDS	MEDIUM
P0.8	prioritized_replay.rs	20-50	HIGH

---

⚡ P1: Important Improvements (Performance)

Quick Action Items

# Week 3: Advanced Regularization (38 hours)
1. Spectral Normalization       (8h)  → Lipschitz constraint
2. Adaptive Dropout             (5h)  → Smart regularization
3. Layer Normalization          (4h)  → Reduce covariate shift
4. Hindsight Experience Replay (10h)  → 5-10x data efficiency
5. Curiosity Integration        (6h)  → Better exploration
6. GAE Returns                  (5h)  → Lower variance

# Week 4: Final Optimizations (33 hours)
7. Rank-Based Sampling          (4h)  → Better diversity
8. Noisy Network Tuning         (3h)  → Better exploration
9. Delayed Updates              (4h)  → Stable Q-values
10. Ensemble Bootstrapping      (4h)  → Better uncertainty
11. Quantile Regression        (10h)  → Better risk estimation
12. AMP Training                (8h)  → 2x speedup!

P1 New Modules Required

ml/src/dqn/
├── spectral_norm.rs       (P1.1)  - Spectral normalization
├── adaptive_dropout.rs    (P1.2)  - Dropout scheduling
├── her.rs                 (P1.4)  - Hindsight experience replay
├── quantile_regression.rs (P1.11) - Quantile regression DQN
└── core/                  (P0.5)  - Refactored dqn.rs modules

---

📊 Impact Matrix

Performance Improvements

Feature	Metric	Before	After	Gain
P0.1 Weight Decay	Generalization gap	15%	5%	67% reduction
P0.2 TD Clipping	Q-value oscillation	High	Low	-30-40%
P0.3 Batch Norm	Training stability	80%	95%	+15%
P0.8 Memory Layout	Sampling speed	100ms	75ms	+25% faster
P1.4 HER	Sample efficiency	1.2M	480K	5-10x
P1.12 AMP	Training time	58s/epoch	34s/epoch	41% faster

Code Quality Improvements

Metric	Before	After	Change
Largest file	2,396 lines	500 lines	-79%
Avg function length	45 lines	25 lines	-44%
Cyclomatic complexity	15 (high)	8 (low)	-47%
Test coverage	85%	95%	+10%

---

🔧 Implementation Checklist

Week 1: P0.1-P0.4 (Critical Stability)

• P0.1: Add L2 weight decay to agent.rs

  • Add weight_decay: f64 to DQNHyperparameters
  • Switch from ParamsAdam to ParamsAdamW
  • Add hyperparameter range: [1e-5, 1e-3]
  • Update tests
  • Verify convergence improvement

• P0.2: Implement TD-error clamping

  • Add td_error_clip: f32 to config (default: 10.0)
  • Update update_priorities() method signature
  • Add clipping logic
  • Update 3 call sites in trainer.rs
  • Test extreme TD errors

• P0.3: Add batch normalization

  • Add BatchNorm1d layers to networks
  • Implement train/eval mode switching
  • Update checkpoint save/load
  • Add batch_norm_momentum hyperparameter
  • Create tests

• P0.4: Implement gradient clipping

  • Add max_grad_norm: f32 to config
  • Implement compute_global_grad_norm()
  • Implement scale_gradients()
  • Add gradient norm logging
  • Test edge cases

Week 2: P0.5-P0.8 (Architecture)

• P0.5: Split dqn.rs into modules

  • Extract agent_base.rs (~400 lines)
  • Extract training_loop.rs (~500 lines)
  • Extract experience_buffer.rs (~400 lines)
  • Extract network_builder.rs (~300 lines)
  • Extract checkpoint.rs (~250 lines)
  • Create core/mod.rs with re-exports
  • Update dqn/mod.rs
  • Run full test suite

• P0.6: Experience diversity tracking

  • Create diversity_tracker.rs
  • Implement cooldown mechanism
  • Integrate into PrioritizedReplayBuffer
  • Add diversity metrics
  • Create tests

• P0.7: Stale priority detection

  • Add age tracking fields
  • Implement stale detection logic
  • Add periodic refresh mechanism
  • Add metrics
  • Create tests

• P0.8: Memory layout optimization

  • Refactor to Structure of Arrays (SoA)
  • Update add() method
  • Update sample() method
  • Add performance benchmarks
  • Keep rollback option

Week 3: P1.1-P1.6 (Advanced Features)

• P1.1: Spectral normalization
• P1.2: Adaptive dropout scheduling
• P1.3: Layer normalization
• P1.4: Hindsight Experience Replay
• P1.5: Curiosity-driven exploration
• P1.6: GAE returns

Week 4: P1.7-P1.12 (Final Optimizations)

• P1.7: Rank-based sampling
• P1.8: Noisy network tuning
• P1.9: Delayed updates
• P1.10: Ensemble bootstrapping
• P1.11: Quantile regression
• P1.12: AMP training

---

🧪 Testing Strategy

Per Feature Testing

# Unit tests
cargo test --package ml <feature_name>

# Integration tests
cargo test --package ml --test dqn_integration_test

# Performance benchmarks
cargo bench --package ml --bench dqn_training_speed

Verification Checklist (Per Feature)

• Unit tests pass
• Integration tests pass
• No performance regression
• Code coverage maintained (>90%)
• Documentation updated
• Hyperopt compatibility verified

---

📈 Success Metrics

Baseline (Current)

• Validation Accuracy: 65-70%
• Training Time: 8 hours (500 epochs)
• Sample Efficiency: 1M experiences → 70% accuracy
• Convergence Rate: 80% runs converge

Target (After All P0+P1)

• Validation Accuracy: 80-85% (+15-20%)
• Training Time: 4 hours (-50%)
• Sample Efficiency: 500K experiences → 80% (-50%)
• Convergence Rate: 95% runs (+15%)

---

🔄 Rollback Plans

Emergency Rollback (Any Feature)

# Option 1: Git revert
git checkout feature/<feature-name>
git revert HEAD~3

# Option 2: Feature flag
config.enable_<feature> = false;

# Option 3: Restore backup
mv ml/src/dqn/<file>.rs.backup ml/src/dqn/<file>.rs
cargo check --package ml

Checkpoint Compatibility

Features that break checkpoint compatibility:

• ❌ P0.3 Batch Normalization (add versioning)
• ❌ P1.1 Spectral Normalization (add versioning)
• ❌ P1.11 Quantile Regression (new architecture)

Solution: Implement checkpoint versioning

pub struct CheckpointMetadata {
    version: String,  // "v2.0-batch-norm"
    features_enabled: Vec<String>,
    compatible_with: Vec<String>,
}

---

🎓 New Hyperparameters

P0 Critical Parameters

weight_decay: 1e-4,              // [1e-5, 1e-3]
td_error_clip: 10.0,             // [5.0, 20.0]
use_batch_norm: true,
batch_norm_momentum: 0.1,        // [0.01, 0.2]
max_grad_norm: 10.0,             // [5.0, 50.0]

P1 Important Parameters

use_spectral_norm: false,        // experimental
spectral_norm_iters: 1,          // [1, 5]
dropout_schedule: Linear,
initial_dropout: 0.3,            // [0.2, 0.5]
final_dropout: 0.1,              // [0.05, 0.2]
use_gae: true,
gae_lambda: 0.95,                // [0.9, 0.99]
use_her: false,                  // sparse rewards only
her_strategy: Future,
her_k: 4,                        // [2, 8]
curiosity_weight: 0.5,           // [0.0, 1.0]
use_rank_based_per: true,
rank_alpha: 0.7,                 // [0.5, 1.0]
polyak_tau: 0.005,               // [0.001, 0.01]
target_update_freq: 2,           // [1, 10]
use_amp: true,                   // if CUDA
num_quantiles: 200,              // [50, 200]

---

📚 Key References

Must-Read Papers

1. Weight Decay: "Decoupled Weight Decay Regularization" (ICLR 2019)
2. PER: "Prioritized Experience Replay" (ICLR 2016)
3. HER: "Hindsight Experience Replay" (NeurIPS 2017)
4. GAE: "High-Dimensional Continuous Control Using GAE" (ICLR 2016)
5. Quantile DQN: "Distributional RL with Quantile Regression" (AAAI 2018)

Implementation Guides

• Spectral Norm: PyTorch implementation (torch.nn.utils.spectral_norm)
• AMP: NVIDIA Mixed Precision Guide
• HER: OpenAI Baselines reference implementation

---

🚀 Quick Start Commands

Build and Test

# Full rebuild with new features
cargo clean
cargo build --release --package ml

# Run all tests
cargo test --package ml --lib -- --test-threads=1

# Run specific test suite
cargo test --package ml --test dqn_rainbow_test

Performance Profiling

# Training speed benchmark
cargo bench --package ml --bench dqn_training_speed

# Memory usage profiling
valgrind --tool=massif target/release/dqn_train

# GPU utilization monitoring
nvidia-smi -l 1

Hyperparameter Tuning

# Launch hyperopt with new parameters
python ml/hyperopt/dqn_hyperopt.py \
    --trials 100 \
    --search-space-version 2.0 \
    --enable-new-features

---

📝 Documentation Updates Required

For each P0/P1 feature:

1. Code comments with paper references
2. ADR (Architecture Decision Record)
3. User guide update
4. Test documentation
5. Changelog entry

Templates: See docs/templates/ directory

---

🎯 Agent Assignment Suggestions

Parallel Execution (Week 1)

• Agent 1: P0.1 Weight Decay + P0.4 Gradient Clipping
• Agent 2: P0.2 TD-Error Clamping
• Agent 3: P0.3 Batch Normalization
• Agent 4: Write comprehensive tests for P0.1-P0.4

Parallel Execution (Week 2)

• Agent 1: P0.5 Code refactoring (lead)
• Agent 2: P0.6 Experience Diversity
• Agent 3: P0.7 Stale Priority Detection
• Agent 4: P0.8 Memory Layout Optimization

Parallel Execution (Week 3)

• Agent 1: P1.1 Spectral Norm + P1.3 Layer Norm
• Agent 2: P1.2 Adaptive Dropout + P1.8 Noisy Tuning
• Agent 3: P1.4 HER (complex, dedicated agent)
• Agent 4: P1.5 Curiosity + P1.6 GAE

Parallel Execution (Week 4)

• Agent 1: P1.11 Quantile Regression (complex)
• Agent 2: P1.12 AMP Training
• Agent 3: P1.7, P1.9, P1.10 (smaller features)
• Agent 4: Integration testing + deployment prep

---

🎉 Completion Criteria

P0 Critical (Production Ready)

• ✅ All 8 P0 items implemented
• ✅ Zero training divergences
• ✅ Generalization gap < 5%
• ✅ All files < 500 lines
• ✅ 95% test coverage

P1 Important (SOTA Performance)

• ✅ All 12 P1 items implemented
• ✅ Validation accuracy > 80%
• ✅ Training time < 4 hours
• ✅ Sample efficiency < 500K experiences
• ✅ 2x speedup with AMP

P2 Enhancements (Research-Ready)

• ✅ Selected P2 items based on business needs
• ✅ Transfer learning capabilities
• ✅ Offline RL support

---

📞 Contact & Support

Full Roadmap: See DQN_2025_UPGRADE_ROADMAP.md for:

• Detailed implementation guides
• Code examples
• Risk analysis
• Literature references
• Performance benchmarks

Agent Coordination: Use Claude Flow memory coordination for cross-agent communication

Questions: Reference ADR documents in docs/adr/

---

Last Updated: 2025-11-27 Next Review: After Week 1 completion Maintainer: Strategic Planning Agent