feat(ml): WAVE 29 DQN Codebase Cleanup & Refactoring Campaign
BREAKING CHANGES: - Removed orphaned dqn.rs monolithic trainer (4,975 lines) - Removed orphaned dqn_ensemble.rs module (816 lines) - Removed orphaned tft.rs and tft_complete_int8_integration_test.rs - TFT trainer split into modular directory structure DQN Module Refactoring: - Split trainers/dqn.rs into modular structure (config.rs, statistics.rs, trainer.rs) - Fixed hyperopt 39D search space (continuous params only) - Boolean flags (use_dueling, use_double_dqn, use_per, use_noisy_nets) are now FIXED architectural decisions - use_distributional defaults to false (Candle BUG #36 - scatter_add gradient issues) Clean Module Structure: - ml/src/trainers/dqn/ directory with proper mod.rs exports - ml/src/trainers/tft/ directory with config.rs, types.rs, model.rs, trainer.rs, tests.rs - All P0 features validated: TD-error clamping, batch diversity, LR scheduler, priority staleness Documentation: - Added comprehensive docs in docs/codebase-cleanup/ - ADR-001 for DQN refactoring decisions - Rainbow DQN component matrix and quick reference guides Build Status: Compiles with zero errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jgrusewski
201
docs/tft-split-summary.txt
Normal file
201
docs/tft-split-summary.txt
Normal file
@@ -0,0 +1,201 @@
|
||||
═══════════════════════════════════════════════════════════════════════════════
|
||||
TFT TRAINER SPLIT PLAN SUMMARY
|
||||
═══════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
Current: ml/src/trainers/tft.rs (2,915 lines)
|
||||
Target: ml/src/trainers/tft/ (6 files)
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ PROPOSED FILE STRUCTURE │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
ml/src/trainers/tft/
|
||||
│
|
||||
├── mod.rs (~50 lines)
|
||||
│ └─ Module exports and documentation
|
||||
│
|
||||
├── config.rs (~350 lines)
|
||||
│ ├─ TFTTrainerConfig
|
||||
│ ├─ impl Default for TFTTrainerConfig
|
||||
│ └─ Config conversion methods (to_model_config, to_training_config)
|
||||
│
|
||||
├── types.rs (~250 lines)
|
||||
│ ├─ QAT Metrics (5 structs)
|
||||
│ │ ├─ QATMetrics
|
||||
│ │ ├─ ScaleStatistics
|
||||
│ │ ├─ ZeroPointStatistics
|
||||
│ │ ├─ ObserverRangeStatistics
|
||||
│ │ └─ LayerQuantizationMetrics
|
||||
│ ├─ Progress Types
|
||||
│ │ ├─ TrainingProgress
|
||||
│ │ └─ ResourceUsage
|
||||
│ ├─ Result Types
|
||||
│ │ └─ TrainingMetrics
|
||||
│ └─ Internal Types (pub(crate))
|
||||
│ ├─ TrainingState
|
||||
│ └─ ValidationMetrics
|
||||
│
|
||||
├── model.rs (~100 lines)
|
||||
│ ├─ pub trait TFTModel
|
||||
│ ├─ impl TFTModel for TemporalFusionTransformer
|
||||
│ └─ (Future: QAT implementation)
|
||||
│
|
||||
├── trainer.rs (~2,000 lines)
|
||||
│ ├─ pub struct TFTTrainer
|
||||
│ ├─ impl Debug for TFTTrainer
|
||||
│ └─ impl TFTTrainer (30+ methods)
|
||||
│ ├─ Public API
|
||||
│ │ ├─ new()
|
||||
│ │ ├─ train()
|
||||
│ │ └─ set_progress_callback()
|
||||
│ ├─ Training Methods
|
||||
│ │ ├─ train_epoch()
|
||||
│ │ ├─ validate_epoch()
|
||||
│ │ ├─ compute_quantile_loss()
|
||||
│ │ └─ compute_rmse()
|
||||
│ ├─ QAT Methods
|
||||
│ │ ├─ run_qat_calibration()
|
||||
│ │ ├─ export_qat_metrics()
|
||||
│ │ └─ apply_qat_lr_schedule()
|
||||
│ └─ Helper Methods
|
||||
│ ├─ initialize_optimizer()
|
||||
│ ├─ save_checkpoint()
|
||||
│ └─ send_progress_update()
|
||||
│
|
||||
└── tests.rs (~165 lines)
|
||||
├─ test_tft_trainer_creation()
|
||||
├─ test_training_config_conversion()
|
||||
├─ test_checkpoint_save_load()
|
||||
└─ OOM retry tests
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ DEPENDENCY GRAPH │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
mod.rs
|
||||
│
|
||||
┌───────────────────┼───────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
config.rs types.rs model.rs
|
||||
│ │ │
|
||||
└───────────────────┴───────────────────┘
|
||||
│
|
||||
▼
|
||||
trainer.rs
|
||||
│
|
||||
▼
|
||||
tests.rs
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ LINE DISTRIBUTION │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
File Lines Percentage Content
|
||||
────────────────────────────────────────────────────────────────────────────
|
||||
mod.rs 50 1.7% Module organization
|
||||
config.rs 350 12.0% Configuration structs
|
||||
types.rs 250 8.6% Metrics and helper types
|
||||
model.rs 100 3.4% TFTModel trait
|
||||
trainer.rs 2,000 68.6% Main implementation
|
||||
tests.rs 165 5.7% Test suite
|
||||
────────────────────────────────────────────────────────────────────────────
|
||||
TOTAL 2,915 100.0% (Same as original)
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ COMPARISON WITH DQN │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Aspect DQN TFT
|
||||
───────────────────────────────────────────────────────────────────────────
|
||||
Total Files 4 6
|
||||
config.rs 566 lines 350 lines
|
||||
statistics/types 134 lines 250 lines
|
||||
trainer.rs 4,007 lines 2,000 lines
|
||||
mod.rs 30 lines 50 lines
|
||||
Extra Files - model.rs (100), tests.rs (165)
|
||||
|
||||
Key Differences:
|
||||
✓ TFT has TFTModel trait abstraction (model.rs)
|
||||
✓ TFT has extensive QAT metrics (larger types.rs)
|
||||
✓ TFT has larger test suite (separate tests.rs)
|
||||
✓ DQN trainer is larger (4,007 vs 2,000 lines)
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ MIGRATION CHECKLIST │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Phase 1: Setup
|
||||
[ ] Create ml/src/trainers/tft/ directory
|
||||
[ ] Backup current tft.rs (git handles this)
|
||||
|
||||
Phase 2: Extract Files (in order)
|
||||
[ ] Create types.rs (QAT metrics, progress, state)
|
||||
[ ] Create model.rs (TFTModel trait)
|
||||
[ ] Create config.rs (TFTTrainerConfig)
|
||||
[ ] Create trainer.rs (TFTTrainer impl)
|
||||
[ ] Create tests.rs (test module)
|
||||
[ ] Create mod.rs (re-exports)
|
||||
|
||||
Phase 3: Update Imports
|
||||
[ ] Update trainer.rs imports (use super::*)
|
||||
[ ] Update tests.rs imports
|
||||
[ ] Verify mod.rs re-exports
|
||||
|
||||
Phase 4: Verification
|
||||
[ ] cargo check --package ml
|
||||
[ ] cargo test --package ml trainers::tft
|
||||
[ ] cargo clippy --package ml
|
||||
[ ] cargo fmt --package ml
|
||||
|
||||
Phase 5: Cleanup
|
||||
[ ] Remove old tft.rs
|
||||
[ ] Update ml/src/trainers/mod.rs if needed
|
||||
[ ] Verify external imports still work
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ RISK ASSESSMENT │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
🟢 LOW RISK
|
||||
• types.rs: Pure data structures
|
||||
• config.rs: Simple configuration
|
||||
• model.rs: Clean trait abstraction
|
||||
|
||||
🟡 MEDIUM RISK
|
||||
• trainer.rs: Large impl block (keep together)
|
||||
• tests.rs: May need import adjustments
|
||||
|
||||
Mitigation Strategies:
|
||||
✓ Keep all trainer methods in one file (no split impl blocks)
|
||||
✓ Use pub(crate) for internal types
|
||||
✓ Re-export everything from mod.rs (backward compatibility)
|
||||
✓ Test after each file creation
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ BENEFITS │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
✅ Modularity: Clear separation of concerns
|
||||
✅ Maintainability: Files under 2,500 lines each
|
||||
✅ Testability: Dedicated test file
|
||||
✅ Compilation: Smaller compilation units
|
||||
✅ Navigation: Easier to find components
|
||||
✅ Consistency: Matches DQN structure
|
||||
✅ QAT Support: Clean separation of QAT metrics
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ CONCLUSION │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Status: ✅ READY FOR IMPLEMENTATION
|
||||
|
||||
The split plan is:
|
||||
• Well-defined with clear boundaries
|
||||
• Low risk with proven pattern (DQN reference)
|
||||
• Backward compatible (re-exports preserve API)
|
||||
• Testable at each step
|
||||
|
||||
Recommendation: PROCEED with implementation following the checklist above.
|
||||
|
||||
═══════════════════════════════════════════════════════════════════════════════
|
||||
Reference in New Issue
Block a user