# DQN 2025 Upgrade - Quick Reference Card

**Last Updated**: 2025-11-27
**Full Roadmap**: [DQN_2025_UPGRADE_ROADMAP.md](./DQN_2025_UPGRADE_ROADMAP.md)

---

## 🎯 Priority Overview

| Priority | Count | Timeline | Expected Gain |
|----------|-------|----------|---------------|
| **P0 Critical** | 8 items | 2-3 weeks | +15-20% performance, Production stability |
| **P1 Important** | 12 items | 2-3 weeks | +10-15% performance |
| **P2 Enhancement** | 9 items | 1-2 weeks | +5-10% performance |

**Total Potential Gain**: +30-45% performance improvement

---

## 🚨 P0: Critical Fixes (Production Blockers)

### Quick Action Items

```bash
# Week 1: Stability Fixes (15 hours)
1. Add L2 Weight Decay          (2h)  → Prevents overfitting
2. TD-Error Clamping            (4h)  → Stops gradient explosion
3. Batch Normalization          (5h)  → Training stability +20%
4. Gradient Clipping            (4h)  → Prevents exploding gradients

# Week 2: Architecture (29 hours)
5. Split dqn.rs (2396→500)     (12h)  → Maintainable codebase
6. Experience Diversity         (5h)  → Better sample quality
7. Stale Priority Detection     (4h)  → Fresh priorities
8. Memory Layout Optimization   (8h)  → 15-25% faster sampling
```

### P0 Files to Modify

| Item | File | Lines | Complexity |
|------|------|-------|------------|
| P0.1 | `agent.rs` | 336-342 | LOW |
| P0.2 | `prioritized_replay.rs` | 145-155 | MEDIUM |
| P0.3 | `network.rs`, `rainbow_network.rs` | 104-122 | MEDIUM |
| P0.4 | `agent.rs` | 450-470 | MEDIUM |
| P0.5 | `dqn.rs` (2396 lines) | FULL FILE | HIGH |
| P0.6 | `prioritized_replay.rs` + NEW | NEW MODULE | MEDIUM |
| P0.7 | `prioritized_replay.rs` | NEW FIELDS | MEDIUM |
| P0.8 | `prioritized_replay.rs` | 20-50 | HIGH |

---

## ⚡ P1: Important Improvements (Performance)

### Quick Action Items

```bash
# Week 3: Advanced Regularization (38 hours)
1. Spectral Normalization       (8h)  → Lipschitz constraint
2. Adaptive Dropout             (5h)  → Smart regularization
3. Layer Normalization          (4h)  → Reduce covariate shift
4. Hindsight Experience Replay (10h)  → 5-10x data efficiency
5. Curiosity Integration        (6h)  → Better exploration
6. GAE Returns                  (5h)  → Lower variance

# Week 4: Final Optimizations (33 hours)
7. Rank-Based Sampling          (4h)  → Better diversity
8. Noisy Network Tuning         (3h)  → Better exploration
9. Delayed Updates              (4h)  → Stable Q-values
10. Ensemble Bootstrapping      (4h)  → Better uncertainty
11. Quantile Regression        (10h)  → Better risk estimation
12. AMP Training                (8h)  → 2x speedup!
```

### P1 New Modules Required

```
ml/src/dqn/
├── spectral_norm.rs       (P1.1)  - Spectral normalization
├── adaptive_dropout.rs    (P1.2)  - Dropout scheduling
├── her.rs                 (P1.4)  - Hindsight experience replay
├── quantile_regression.rs (P1.11) - Quantile regression DQN
└── core/                  (P0.5)  - Refactored dqn.rs modules
```

---

## 📊 Impact Matrix

### Performance Improvements

| Feature | Metric | Before | After | Gain |
|---------|--------|--------|-------|------|
| P0.1 Weight Decay | Generalization gap | 15% | 5% | 67% reduction |
| P0.2 TD Clipping | Q-value oscillation | High | Low | -30-40% |
| P0.3 Batch Norm | Training stability | 80% | 95% | +15% |
| P0.8 Memory Layout | Sampling speed | 100ms | 75ms | +25% faster |
| P1.4 HER | Sample efficiency | 1.2M | 480K | **5-10x** |
| P1.12 AMP | Training time | 58s/epoch | 34s/epoch | **41% faster** |

### Code Quality Improvements

| Metric | Before | After | Change |
|--------|--------|-------|--------|
| Largest file | 2,396 lines | 500 lines | -79% |
| Avg function length | 45 lines | 25 lines | -44% |
| Cyclomatic complexity | 15 (high) | 8 (low) | -47% |
| Test coverage | 85% | 95% | +10% |

---

## 🔧 Implementation Checklist

### Week 1: P0.1-P0.4 (Critical Stability)

- [ ] **P0.1**: Add L2 weight decay to `agent.rs`
  - [ ] Add `weight_decay: f64` to `DQNHyperparameters`
  - [ ] Switch from `ParamsAdam` to `ParamsAdamW`
  - [ ] Add hyperparameter range: [1e-5, 1e-3]
  - [ ] Update tests
  - [ ] Verify convergence improvement

- [ ] **P0.2**: Implement TD-error clamping
  - [ ] Add `td_error_clip: f32` to config (default: 10.0)
  - [ ] Update `update_priorities()` method signature
  - [ ] Add clipping logic
  - [ ] Update 3 call sites in `trainer.rs`
  - [ ] Test extreme TD errors

- [ ] **P0.3**: Add batch normalization
  - [ ] Add `BatchNorm1d` layers to networks
  - [ ] Implement train/eval mode switching
  - [ ] Update checkpoint save/load
  - [ ] Add `batch_norm_momentum` hyperparameter
  - [ ] Create tests

- [ ] **P0.4**: Implement gradient clipping
  - [ ] Add `max_grad_norm: f32` to config
  - [ ] Implement `compute_global_grad_norm()`
  - [ ] Implement `scale_gradients()`
  - [ ] Add gradient norm logging
  - [ ] Test edge cases

### Week 2: P0.5-P0.8 (Architecture)

- [ ] **P0.5**: Split dqn.rs into modules
  - [ ] Extract `agent_base.rs` (~400 lines)
  - [ ] Extract `training_loop.rs` (~500 lines)
  - [ ] Extract `experience_buffer.rs` (~400 lines)
  - [ ] Extract `network_builder.rs` (~300 lines)
  - [ ] Extract `checkpoint.rs` (~250 lines)
  - [ ] Create `core/mod.rs` with re-exports
  - [ ] Update `dqn/mod.rs`
  - [ ] Run full test suite

- [ ] **P0.6**: Experience diversity tracking
  - [ ] Create `diversity_tracker.rs`
  - [ ] Implement cooldown mechanism
  - [ ] Integrate into `PrioritizedReplayBuffer`
  - [ ] Add diversity metrics
  - [ ] Create tests

- [ ] **P0.7**: Stale priority detection
  - [ ] Add age tracking fields
  - [ ] Implement stale detection logic
  - [ ] Add periodic refresh mechanism
  - [ ] Add metrics
  - [ ] Create tests

- [ ] **P0.8**: Memory layout optimization
  - [ ] Refactor to Structure of Arrays (SoA)
  - [ ] Update `add()` method
  - [ ] Update `sample()` method
  - [ ] Add performance benchmarks
  - [ ] Keep rollback option

### Week 3: P1.1-P1.6 (Advanced Features)

- [ ] **P1.1**: Spectral normalization
- [ ] **P1.2**: Adaptive dropout scheduling
- [ ] **P1.3**: Layer normalization
- [ ] **P1.4**: Hindsight Experience Replay
- [ ] **P1.5**: Curiosity-driven exploration
- [ ] **P1.6**: GAE returns

### Week 4: P1.7-P1.12 (Final Optimizations)

- [ ] **P1.7**: Rank-based sampling
- [ ] **P1.8**: Noisy network tuning
- [ ] **P1.9**: Delayed updates
- [ ] **P1.10**: Ensemble bootstrapping
- [ ] **P1.11**: Quantile regression
- [ ] **P1.12**: AMP training

---

## 🧪 Testing Strategy

### Per Feature Testing
```bash
# Unit tests
cargo test --package ml <feature_name>

# Integration tests
cargo test --package ml --test dqn_integration_test

# Performance benchmarks
cargo bench --package ml --bench dqn_training_speed
```

### Verification Checklist (Per Feature)
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] No performance regression
- [ ] Code coverage maintained (>90%)
- [ ] Documentation updated
- [ ] Hyperopt compatibility verified

---

## 📈 Success Metrics

### Baseline (Current)
- **Validation Accuracy**: 65-70%
- **Training Time**: 8 hours (500 epochs)
- **Sample Efficiency**: 1M experiences → 70% accuracy
- **Convergence Rate**: 80% runs converge

### Target (After All P0+P1)
- **Validation Accuracy**: **80-85%** (+15-20%)
- **Training Time**: **4 hours** (-50%)
- **Sample Efficiency**: **500K experiences** → 80% (-50%)
- **Convergence Rate**: **95% runs** (+15%)

---

## 🔄 Rollback Plans

### Emergency Rollback (Any Feature)
```bash
# Option 1: Git revert
git checkout feature/<feature-name>
git revert HEAD~3

# Option 2: Feature flag
config.enable_<feature> = false;

# Option 3: Restore backup
mv ml/src/dqn/<file>.rs.backup ml/src/dqn/<file>.rs
cargo check --package ml
```

### Checkpoint Compatibility
Features that break checkpoint compatibility:
- ❌ P0.3 Batch Normalization (add versioning)
- ❌ P1.1 Spectral Normalization (add versioning)
- ❌ P1.11 Quantile Regression (new architecture)

**Solution**: Implement checkpoint versioning
```rust
pub struct CheckpointMetadata {
    version: String,  // "v2.0-batch-norm"
    features_enabled: Vec<String>,
    compatible_with: Vec<String>,
}
```

---

## 🎓 New Hyperparameters

### P0 Critical Parameters
```rust
weight_decay: 1e-4,              // [1e-5, 1e-3]
td_error_clip: 10.0,             // [5.0, 20.0]
use_batch_norm: true,
batch_norm_momentum: 0.1,        // [0.01, 0.2]
max_grad_norm: 10.0,             // [5.0, 50.0]
```

### P1 Important Parameters
```rust
use_spectral_norm: false,        // experimental
spectral_norm_iters: 1,          // [1, 5]
dropout_schedule: Linear,
initial_dropout: 0.3,            // [0.2, 0.5]
final_dropout: 0.1,              // [0.05, 0.2]
use_gae: true,
gae_lambda: 0.95,                // [0.9, 0.99]
use_her: false,                  // sparse rewards only
her_strategy: Future,
her_k: 4,                        // [2, 8]
curiosity_weight: 0.5,           // [0.0, 1.0]
use_rank_based_per: true,
rank_alpha: 0.7,                 // [0.5, 1.0]
polyak_tau: 0.005,               // [0.001, 0.01]
target_update_freq: 2,           // [1, 10]
use_amp: true,                   // if CUDA
num_quantiles: 200,              // [50, 200]
```

---

## 📚 Key References

### Must-Read Papers
1. **Weight Decay**: "Decoupled Weight Decay Regularization" (ICLR 2019)
2. **PER**: "Prioritized Experience Replay" (ICLR 2016)
3. **HER**: "Hindsight Experience Replay" (NeurIPS 2017)
4. **GAE**: "High-Dimensional Continuous Control Using GAE" (ICLR 2016)
5. **Quantile DQN**: "Distributional RL with Quantile Regression" (AAAI 2018)

### Implementation Guides
- **Spectral Norm**: PyTorch implementation (torch.nn.utils.spectral_norm)
- **AMP**: NVIDIA Mixed Precision Guide
- **HER**: OpenAI Baselines reference implementation

---

## 🚀 Quick Start Commands

### Build and Test
```bash
# Full rebuild with new features
cargo clean
cargo build --release --package ml

# Run all tests
cargo test --package ml --lib -- --test-threads=1

# Run specific test suite
cargo test --package ml --test dqn_rainbow_test
```

### Performance Profiling
```bash
# Training speed benchmark
cargo bench --package ml --bench dqn_training_speed

# Memory usage profiling
valgrind --tool=massif target/release/dqn_train

# GPU utilization monitoring
nvidia-smi -l 1
```

### Hyperparameter Tuning
```bash
# Launch hyperopt with new parameters
python ml/hyperopt/dqn_hyperopt.py \
    --trials 100 \
    --search-space-version 2.0 \
    --enable-new-features
```

---

## 📝 Documentation Updates Required

For each P0/P1 feature:
1. **Code comments** with paper references
2. **ADR** (Architecture Decision Record)
3. **User guide** update
4. **Test documentation**
5. **Changelog** entry

**Templates**: See `docs/templates/` directory

---

## 🎯 Agent Assignment Suggestions

### Parallel Execution (Week 1)
- **Agent 1**: P0.1 Weight Decay + P0.4 Gradient Clipping
- **Agent 2**: P0.2 TD-Error Clamping
- **Agent 3**: P0.3 Batch Normalization
- **Agent 4**: Write comprehensive tests for P0.1-P0.4

### Parallel Execution (Week 2)
- **Agent 1**: P0.5 Code refactoring (lead)
- **Agent 2**: P0.6 Experience Diversity
- **Agent 3**: P0.7 Stale Priority Detection
- **Agent 4**: P0.8 Memory Layout Optimization

### Parallel Execution (Week 3)
- **Agent 1**: P1.1 Spectral Norm + P1.3 Layer Norm
- **Agent 2**: P1.2 Adaptive Dropout + P1.8 Noisy Tuning
- **Agent 3**: P1.4 HER (complex, dedicated agent)
- **Agent 4**: P1.5 Curiosity + P1.6 GAE

### Parallel Execution (Week 4)
- **Agent 1**: P1.11 Quantile Regression (complex)
- **Agent 2**: P1.12 AMP Training
- **Agent 3**: P1.7, P1.9, P1.10 (smaller features)
- **Agent 4**: Integration testing + deployment prep

---

## 🎉 Completion Criteria

### P0 Critical (Production Ready)
- ✅ All 8 P0 items implemented
- ✅ Zero training divergences
- ✅ Generalization gap < 5%
- ✅ All files < 500 lines
- ✅ 95% test coverage

### P1 Important (SOTA Performance)
- ✅ All 12 P1 items implemented
- ✅ Validation accuracy > 80%
- ✅ Training time < 4 hours
- ✅ Sample efficiency < 500K experiences
- ✅ 2x speedup with AMP

### P2 Enhancements (Research-Ready)
- ✅ Selected P2 items based on business needs
- ✅ Transfer learning capabilities
- ✅ Offline RL support

---

## 📞 Contact & Support

**Full Roadmap**: See [DQN_2025_UPGRADE_ROADMAP.md](./DQN_2025_UPGRADE_ROADMAP.md) for:
- Detailed implementation guides
- Code examples
- Risk analysis
- Literature references
- Performance benchmarks

**Agent Coordination**: Use Claude Flow memory coordination for cross-agent communication

**Questions**: Reference ADR documents in `docs/adr/`

---

**Last Updated**: 2025-11-27
**Next Review**: After Week 1 completion
**Maintainer**: Strategic Planning Agent