Option A Implementation: Real baseline measurements before full training
New Files Created (3 files, 865 lines):
1. download_ml_training_data.py (365 lines):
- Downloads 90 days × 4 symbols from Databento
- Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
- Estimated cost: ~$2.00 (~360 files, 180K bars)
- Features: Dry-run preview, progress tracking, cost estimation
- Skips existing files for resume capability
- Validates data quality with record counts
2. benchmark_training_time.py (330 lines):
- Measures ACTUAL training time on RTX 3050 Ti
- Tests all 4 models: MAMBA-2, DQN, PPO, TFT
- Runs small-scale experiments (5-10 epochs)
- Tracks GPU utilization, VRAM usage, epoch timing
- Extrapolates to full training timeline
- Compares actual vs projected performance
- Saves results to training_benchmarks.json
3. ML_DATA_DOWNLOAD_GUIDE.md (170 lines):
- Complete walkthrough for data download + benchmarks
- Prerequisites, step-by-step instructions
- Troubleshooting common issues
- Decision matrix: local GPU vs cloud GPU
- Expected outcomes and success criteria
- Timeline: 1-2 hours total (download + benchmarks)
User Workflow:
Step 1: Download Data (30-60 min, ~$2)
export DATABENTO_API_KEY='your-key-here'
source .venv_databento/bin/activate
python3 download_ml_training_data.py
Step 2: Benchmark Training (10-20 min)
python3 benchmark_training_time.py
# Measures actual RTX 3050 Ti performance
# Output: training_benchmarks.json
Step 3: Analyze & Decide
cat training_benchmarks.json | jq '.total_weeks'
# If < 2 weeks: Use local GPU ✅
# If > 2 weeks: Consider cloud GPU (A100)
Step 4: Start Full Training
cargo run -p ml_training_service -- train-all
Benefits:
- Real hardware performance data (not projections)
- Validated training timeline before committing weeks
- Cost-effective decision (local GPU vs cloud)
- Confidence in feasibility
Technical Approach:
- Python scripts for Databento API integration
- GPU monitoring with nvidia-smi
- Epoch timing extrapolation
- JSON results for analysis
- Resume-capable downloads (skip existing files)
Expected Results (Based on Projections):
- MAMBA-2: 100 epochs, ~1-2 hours (real data TBD)
- DQN: 50 epochs, ~30-60 min (real data TBD)
- PPO: 50 epochs, ~30-60 min (real data TBD)
- TFT: 80 epochs, ~1-2 hours (real data TBD)
- Total: ~3-6 hours sequential (RTX 3050 Ti estimate)
Note: Projections from ML_TRAINING_ROADMAP.md were 4-6 weeks
Benchmarks will reveal actual RTX 3050 Ti performance
Could be 10-100x faster or slower depending on model size
Duration: 45 minutes (script creation + documentation)
Impact: Smart approach - validate assumptions with real measurements
before investing weeks of GPU time
