foxhunt/docs/archive/data_management/ML_DATA_DOWNLOAD_GUIDE.md

ML Data Download & Training Benchmark Guide

Status: Ready to execute Timeline: Data download (30-60 min) + Benchmarks (10-20 min) = ~1-2 hours Cost: ~$2 for 90 days × 4 symbols

---

Overview

This guide walks through:

1. Downloading 90 days of real market data from Databento (~$2)
2. Benchmarking actual training time on RTX 3050 Ti GPU
3. Calculating realistic timeline for full ML training

Goal: Get REAL performance baseline instead of projections before committing to 4-6 week training.

---

Prerequisites

1. Databento API Key

Get your API key from: https://databento.com/

# Set API key in environment
export DATABENTO_API_KEY='your-key-here'

# Or add to ~/.bashrc for persistence
echo 'export DATABENTO_API_KEY="your-key-here"' >> ~/.bashrc
source ~/.bashrc

2. Python Virtual Environment

# Activate existing virtual environment
source .venv/bin/activate

# Verify databento is installed
python3 -c "import databento; print('databento version:', databento.__version__)"

# If not installed:
pip install databento

3. GPU Available

# Verify CUDA GPU accessible
nvidia-smi

# Should show: RTX 3050 Ti Laptop GPU

---

Step 1: Download 90 Days of Market Data

Preview Download

# Dry run to preview (no cost)
python3 download_ml_training_data.py --dry-run

# Output shows:
# - 90 trading days (excludes weekends)
# - 4 symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
# - Estimated cost: ~$2.00
# - Total: ~360 files

Execute Download

# Run full download
python3 download_ml_training_data.py

# You'll be prompted:
# "Proceed with download? (yes/no):"
# Type: yes

# Expected duration: 30-60 minutes
# Progress shown per file: [1/360 - 0.3%] ES.FUT @ 2024-01-02... ✅ 1674 bars, 96.5 KB

Download Options

# Custom date range (last 90 trading days)
python3 download_ml_training_data.py --start-date 2024-07-01 --days 90

# Specific symbols only
python3 download_ml_training_data.py --symbols ES.FUT NQ.FUT

# Smaller test download (10 days)
python3 download_ml_training_data.py --days 10

Verify Downloaded Data

# Check output directory
ls -lh test_data/real/databento/ml_training/

# Expected: 360 .dbn files
# ES.FUT_ohlcv-1m_2024-01-02.dbn
# ES.FUT_ohlcv-1m_2024-01-03.dbn
# ...
# 6E.FUT_ohlcv-1m_2024-04-30.dbn

# Run ML readiness validation with new data
cargo test -p ml --test ml_readiness_validation_tests

---

Step 2: Benchmark Training Time on RTX 3050 Ti

Run Training Benchmarks (Rust Implementation)

# Default: 5 epochs with actual Rust training code
cargo run -p ml --example benchmark_training_time --release

# More epochs for higher accuracy (slower)
cargo run -p ml --example benchmark_training_time --release -- --epochs 10

# Quick test (2 epochs)
cargo run -p ml --example benchmark_training_time --release -- --epochs 2

# CPU-only test (disable GPU)
cargo run -p ml --example benchmark_training_time --release -- --cpu-only

# Expected duration: 10-30 minutes total (uses REAL Rust training code)

Note: This benchmark uses the actual production Rust ML training pipeline with:

• Real GPU optimizations (gradient checkpointing, memory-efficient attention)
• 4GB VRAM optimizations already built-in
• Safety managers and gradient clipping
• Production training configurations

What Gets Measured

For each model (MAMBA-2, DQN, PPO, TFT):

📊 MAMBA2 Results:
  Average epoch time: 45.23s
  Min epoch time: 44.87s
  Max epoch time: 46.12s
  Average GPU utilization: 87.3%
  Average VRAM usage: 3245MB
  Peak VRAM usage: 3512MB

Full Training Estimates

After benchmarks complete:

📈 MAMBA2 - MAMBA-2 state space model:
  Benchmark: 45.23s per epoch (5 epochs)
  Target: 100 epochs
  Estimated time: 1:15:23
    (1.3 hours / 0.05 days)
  GPU utilization: 87.3%
  VRAM usage: 3245MB (peak: 3512MB)

🕐 TOTAL TRAINING TIME (Sequential):
  24.7 hours
  1.0 days
  0.1 weeks

📊 Comparison vs Projections:
  Projected (ML_TRAINING_ROADMAP.md): ~4 weeks
  Actual (RTX 3050 Ti benchmarks): ~0.1 weeks
  ✅ FASTER than projected (2.5% of estimated time)

Save Benchmark Results

Results automatically saved to training_benchmarks.json:

{
  "timestamp": "2025-10-13 14:32:15",
  "gpu_info": "NVIDIA GeForce RTX 3050 Ti Laptop GPU, 4096 MiB",
  "config": {
    "epochs": 5,
    "batch_size": 32,
    "sequence_length": 100
  },
  "benchmarks": [...],
  "total_hours": 24.7,
  "total_days": 1.0,
  "total_weeks": 0.1
}

---

Step 3: Analyze Results & Decide

Interpret Benchmark Results

Scenarios:

1. Training < 1 week ✅ EXCELLENT

   • RTX 3050 Ti sufficient for full training
   • Proceed with local GPU training immediately

2. Training 1-2 weeks ✅ GOOD

   • Feasible on local GPU
   • Consider overnight/weekend training sessions
   • Proceed with confidence

3. Training 2-4 weeks ⚠️ CAUTION

   • Still feasible but requires planning
   • GPU will be busy for extended period
   • Consider cloud GPU (A100) for faster training

4. Training > 4 weeks ❌ RECONSIDER

   • Too long for local GPU
   • Recommended: Use cloud GPU (A100/H100)
   • Cost: ~$1-2/hour on Lambda Labs, RunPod

Cost Comparison

Local GPU (RTX 3050 Ti):

• Cost: $0 (already owned)
• Timeline: Per benchmarks (1-4 weeks typically)
• Availability: 100% dedicated

Cloud GPU (A100):

• Cost: ~$1.50/hour × 24h × 7 days = ~$250/week
• Timeline: 5-10x faster than RTX 3050 Ti
• Availability: On-demand, scalable

Decision Matrix:

• Training < 7 days → Local GPU ✅
• Training 7-14 days → Local GPU (with monitoring) ✅
• Training 14-21 days → Consider cloud GPU ⚠️
• Training > 21 days → Use cloud GPU ❌

---

Step 4: Start Full ML Training

Option A: Local GPU Training (< 2 weeks)

# Review benchmark results
cat training_benchmarks.json | jq '.total_weeks'

# Start full training (all 4 models sequentially)
cargo run -p ml_training_service -- train-all

# Monitor GPU utilization
watch -n 1 nvidia-smi

# Track training progress
tail -f logs/ml_training.log

Option B: Cloud GPU Training (> 2 weeks)

# Export trained model checkpoints
cargo run -p ml_training_service -- export-checkpoints

# Upload to cloud storage
aws s3 cp checkpoints/ s3://foxhunt-ml-checkpoints/ --recursive

# Launch cloud training instance (A100)
# ... (separate cloud deployment guide)

# Download trained models
aws s3 cp s3://foxhunt-ml-checkpoints/ checkpoints/ --recursive

# Validate models locally
cargo run -p ml_training_service -- validate-checkpoints

---

Expected Outcomes

Data Download Success

✅ SUCCESS: Downloaded 98.3% of requested data!
   Ready for ML training benchmarks on RTX 3050 Ti

📊 Total Records: 152,847 bars
💾 Total Size: 14,523.7 KB (14.2 MB)
💰 Estimated Cost: $1.92

Benchmark Success

✅ SUCCESS: Training feasible on RTX 3050 Ti (~1.2 weeks)

📋 NEXT STEPS:
1. Review benchmark results and decide on training approach
2. Adjust training hyperparameters based on GPU memory constraints
3. Start full training with validated timeline:
   cargo run -p ml_training_service -- train-all

Training Success (Future)

✅ TRAINING COMPLETE

Models trained:
- MAMBA-2: 100 epochs, 55.2% win rate, Sharpe 1.67
- DQN: 50 epochs, 53.8% win rate, Sharpe 1.52
- PPO: 50 epochs, 54.1% win rate, Sharpe 1.58
- TFT: 80 epochs, 56.3% win rate, Sharpe 1.71

Total training time: 1.3 weeks (RTX 3050 Ti)
All checkpoints saved to: checkpoints/
Ready for backtesting validation

---

Troubleshooting

Issue: API Key Not Found

❌ ERROR: DATABENTO_API_KEY not found in environment!

Fix:

export DATABENTO_API_KEY='your-key-here'
echo $DATABENTO_API_KEY  # Verify set

Issue: GPU Not Detected

⚠️ GPU not detected! Benchmarks will run on CPU (much slower).

Fix:

# Check CUDA
nvidia-smi
nvcc --version

# Rebuild with CUDA
cd ml
cargo clean
cargo build --features cuda

Issue: Download Failures (>20%)

❌ ERROR: Only downloaded 65.2% of data

Fix:

# Re-run download (skips existing files)
python3 download_ml_training_data.py

# Check specific errors
grep "ERROR" download_log.txt

# Try smaller date range
python3 download_ml_training_data.py --start-date 2024-03-01 --days 60

Issue: Out of VRAM During Benchmarks

RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB

Fix:

# Reduce batch size
python3 benchmark_training_time.py --batch-size 16

# Reduce sequence length
python3 benchmark_training_time.py --sequence-length 50

# Test with smaller models only
python3 benchmark_training_time.py --models DQN PPO

---

Summary Checklist

• Databento API key obtained and set
• Python virtual environment activated
• GPU verified with nvidia-smi
• Data download script run successfully (90 days, 4 symbols, ~$2)
• Downloaded data validated (>80% success rate)
• Training benchmarks run (5-10 epochs per model)
• Benchmark results analyzed (training_benchmarks.json)
• Training timeline calculated (local GPU vs cloud GPU decision)
• Full training approach decided (local or cloud)
• Ready to start full ML training with validated timeline

---

Next Steps: After completing this guide, you'll have:

1. ✅ 90 days of real market data (~180K bars)
2. ✅ Actual training time measurements on RTX 3050 Ti
3. ✅ Realistic timeline estimate (not projections)
4. ✅ Validated decision on local vs cloud GPU training
5. ✅ Confidence in training feasibility and costs

Proceed to: ML_TRAINING_ROADMAP.md with validated timeline and real hardware performance data.