jgrusewski 0c09b5ad06 Add ML data download and training benchmark infrastructure ...

Option A Implementation: Real baseline measurements before full training

New Files Created (3 files, 865 lines):

1. download_ml_training_data.py (365 lines):
   - Downloads 90 days × 4 symbols from Databento
   - Symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
   - Estimated cost: ~$2.00 (~360 files, 180K bars)
   - Features: Dry-run preview, progress tracking, cost estimation
   - Skips existing files for resume capability
   - Validates data quality with record counts

2. benchmark_training_time.py (330 lines):
   - Measures ACTUAL training time on RTX 3050 Ti
   - Tests all 4 models: MAMBA-2, DQN, PPO, TFT
   - Runs small-scale experiments (5-10 epochs)
   - Tracks GPU utilization, VRAM usage, epoch timing
   - Extrapolates to full training timeline
   - Compares actual vs projected performance
   - Saves results to training_benchmarks.json

3. ML_DATA_DOWNLOAD_GUIDE.md (170 lines):
   - Complete walkthrough for data download + benchmarks
   - Prerequisites, step-by-step instructions
   - Troubleshooting common issues
   - Decision matrix: local GPU vs cloud GPU
   - Expected outcomes and success criteria
   - Timeline: 1-2 hours total (download + benchmarks)

User Workflow:

Step 1: Download Data (30-60 min, ~$2)
  export DATABENTO_API_KEY='your-key-here'
  source .venv_databento/bin/activate
  python3 download_ml_training_data.py

Step 2: Benchmark Training (10-20 min)
  python3 benchmark_training_time.py
  # Measures actual RTX 3050 Ti performance
  # Output: training_benchmarks.json

Step 3: Analyze & Decide
  cat training_benchmarks.json | jq '.total_weeks'
  # If < 2 weeks: Use local GPU ✅
  # If > 2 weeks: Consider cloud GPU (A100)

Step 4: Start Full Training
  cargo run -p ml_training_service -- train-all

Benefits:
- Real hardware performance data (not projections)
- Validated training timeline before committing weeks
- Cost-effective decision (local GPU vs cloud)
- Confidence in feasibility

Technical Approach:
- Python scripts for Databento API integration
- GPU monitoring with nvidia-smi
- Epoch timing extrapolation
- JSON results for analysis
- Resume-capable downloads (skip existing files)

Expected Results (Based on Projections):
- MAMBA-2: 100 epochs, ~1-2 hours (real data TBD)
- DQN: 50 epochs, ~30-60 min (real data TBD)
- PPO: 50 epochs, ~30-60 min (real data TBD)
- TFT: 80 epochs, ~1-2 hours (real data TBD)
- Total: ~3-6 hours sequential (RTX 3050 Ti estimate)

Note: Projections from ML_TRAINING_ROADMAP.md were 4-6 weeks
      Benchmarks will reveal actual RTX 3050 Ti performance
      Could be 10-100x faster or slower depending on model size

Duration: 45 minutes (script creation + documentation)

Impact: Smart approach - validate assumptions with real measurements
        before investing weeks of GPU time

2025-10-13 12:33:27 +02:00

12 KiB

Executable File

Raw Blame History

View Raw