# ML Data Download & Training Benchmark Guide

**Status**: Ready to execute
**Timeline**: Data download (30-60 min) + Benchmarks (10-20 min) = ~1-2 hours
**Cost**: ~$2 for 90 days × 4 symbols

---

## Overview

This guide walks through:
1. **Downloading 90 days of real market data** from Databento (~$2)
2. **Benchmarking actual training time** on RTX 3050 Ti GPU
3. **Calculating realistic timeline** for full ML training

**Goal**: Get REAL performance baseline instead of projections before committing to 4-6 week training.

---

## Prerequisites

### 1. Databento API Key

Get your API key from: https://databento.com/

```bash
# Set API key in environment
export DATABENTO_API_KEY='your-key-here'

# Or add to ~/.bashrc for persistence
echo 'export DATABENTO_API_KEY="your-key-here"' >> ~/.bashrc
source ~/.bashrc
```

### 2. Python Virtual Environment

```bash
# Activate existing virtual environment
source .venv/bin/activate

# Verify databento is installed
python3 -c "import databento; print('databento version:', databento.__version__)"

# If not installed:
pip install databento
```

### 3. GPU Available

```bash
# Verify CUDA GPU accessible
nvidia-smi

# Should show: RTX 3050 Ti Laptop GPU
```

---

## Step 1: Download 90 Days of Market Data

### Preview Download

```bash
# Dry run to preview (no cost)
python3 download_ml_training_data.py --dry-run

# Output shows:
# - 90 trading days (excludes weekends)
# - 4 symbols: ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT
# - Estimated cost: ~$2.00
# - Total: ~360 files
```

### Execute Download

```bash
# Run full download
python3 download_ml_training_data.py

# You'll be prompted:
# "Proceed with download? (yes/no):"
# Type: yes

# Expected duration: 30-60 minutes
# Progress shown per file: [1/360 - 0.3%] ES.FUT @ 2024-01-02... ✅ 1674 bars, 96.5 KB
```

### Download Options

```bash
# Custom date range (last 90 trading days)
python3 download_ml_training_data.py --start-date 2024-07-01 --days 90

# Specific symbols only
python3 download_ml_training_data.py --symbols ES.FUT NQ.FUT

# Smaller test download (10 days)
python3 download_ml_training_data.py --days 10
```

### Verify Downloaded Data

```bash
# Check output directory
ls -lh test_data/real/databento/ml_training/

# Expected: 360 .dbn files
# ES.FUT_ohlcv-1m_2024-01-02.dbn
# ES.FUT_ohlcv-1m_2024-01-03.dbn
# ...
# 6E.FUT_ohlcv-1m_2024-04-30.dbn

# Run ML readiness validation with new data
cargo test -p ml --test ml_readiness_validation_tests
```

---

## Step 2: Benchmark Training Time on RTX 3050 Ti

### Run Training Benchmarks (Rust Implementation)

```bash
# Default: 5 epochs with actual Rust training code
cargo run -p ml --example benchmark_training_time --release

# More epochs for higher accuracy (slower)
cargo run -p ml --example benchmark_training_time --release -- --epochs 10

# Quick test (2 epochs)
cargo run -p ml --example benchmark_training_time --release -- --epochs 2

# CPU-only test (disable GPU)
cargo run -p ml --example benchmark_training_time --release -- --cpu-only

# Expected duration: 10-30 minutes total (uses REAL Rust training code)
```

**Note**: This benchmark uses the **actual production Rust ML training pipeline** with:
- Real GPU optimizations (gradient checkpointing, memory-efficient attention)
- 4GB VRAM optimizations already built-in
- Safety managers and gradient clipping
- Production training configurations

### What Gets Measured

For each model (MAMBA-2, DQN, PPO, TFT):

```
📊 MAMBA2 Results:
  Average epoch time: 45.23s
  Min epoch time: 44.87s
  Max epoch time: 46.12s
  Average GPU utilization: 87.3%
  Average VRAM usage: 3245MB
  Peak VRAM usage: 3512MB
```

### Full Training Estimates

After benchmarks complete:

```
📈 MAMBA2 - MAMBA-2 state space model:
  Benchmark: 45.23s per epoch (5 epochs)
  Target: 100 epochs
  Estimated time: 1:15:23
    (1.3 hours / 0.05 days)
  GPU utilization: 87.3%
  VRAM usage: 3245MB (peak: 3512MB)

🕐 TOTAL TRAINING TIME (Sequential):
  24.7 hours
  1.0 days
  0.1 weeks

📊 Comparison vs Projections:
  Projected (ML_TRAINING_ROADMAP.md): ~4 weeks
  Actual (RTX 3050 Ti benchmarks): ~0.1 weeks
  ✅ FASTER than projected (2.5% of estimated time)
```

### Save Benchmark Results

Results automatically saved to `training_benchmarks.json`:

```json
{
  "timestamp": "2025-10-13 14:32:15",
  "gpu_info": "NVIDIA GeForce RTX 3050 Ti Laptop GPU, 4096 MiB",
  "config": {
    "epochs": 5,
    "batch_size": 32,
    "sequence_length": 100
  },
  "benchmarks": [...],
  "total_hours": 24.7,
  "total_days": 1.0,
  "total_weeks": 0.1
}
```

---

## Step 3: Analyze Results & Decide

### Interpret Benchmark Results

**Scenarios**:

1. **Training < 1 week** ✅ **EXCELLENT**
   - RTX 3050 Ti sufficient for full training
   - Proceed with local GPU training immediately

2. **Training 1-2 weeks** ✅ **GOOD**
   - Feasible on local GPU
   - Consider overnight/weekend training sessions
   - Proceed with confidence

3. **Training 2-4 weeks** ⚠️ **CAUTION**
   - Still feasible but requires planning
   - GPU will be busy for extended period
   - Consider cloud GPU (A100) for faster training

4. **Training > 4 weeks** ❌ **RECONSIDER**
   - Too long for local GPU
   - **Recommended**: Use cloud GPU (A100/H100)
   - Cost: ~$1-2/hour on Lambda Labs, RunPod

### Cost Comparison

**Local GPU (RTX 3050 Ti)**:
- Cost: $0 (already owned)
- Timeline: Per benchmarks (1-4 weeks typically)
- Availability: 100% dedicated

**Cloud GPU (A100)**:
- Cost: ~$1.50/hour × 24h × 7 days = ~$250/week
- Timeline: 5-10x faster than RTX 3050 Ti
- Availability: On-demand, scalable

**Decision Matrix**:
- Training < 7 days → Local GPU ✅
- Training 7-14 days → Local GPU (with monitoring) ✅
- Training 14-21 days → Consider cloud GPU ⚠️
- Training > 21 days → Use cloud GPU ❌

---

## Step 4: Start Full ML Training

### Option A: Local GPU Training (< 2 weeks)

```bash
# Review benchmark results
cat training_benchmarks.json | jq '.total_weeks'

# Start full training (all 4 models sequentially)
cargo run -p ml_training_service -- train-all

# Monitor GPU utilization
watch -n 1 nvidia-smi

# Track training progress
tail -f logs/ml_training.log
```

### Option B: Cloud GPU Training (> 2 weeks)

```bash
# Export trained model checkpoints
cargo run -p ml_training_service -- export-checkpoints

# Upload to cloud storage
aws s3 cp checkpoints/ s3://foxhunt-ml-checkpoints/ --recursive

# Launch cloud training instance (A100)
# ... (separate cloud deployment guide)

# Download trained models
aws s3 cp s3://foxhunt-ml-checkpoints/ checkpoints/ --recursive

# Validate models locally
cargo run -p ml_training_service -- validate-checkpoints
```

---

## Expected Outcomes

### Data Download Success

```
✅ SUCCESS: Downloaded 98.3% of requested data!
   Ready for ML training benchmarks on RTX 3050 Ti

📊 Total Records: 152,847 bars
💾 Total Size: 14,523.7 KB (14.2 MB)
💰 Estimated Cost: $1.92
```

### Benchmark Success

```
✅ SUCCESS: Training feasible on RTX 3050 Ti (~1.2 weeks)

📋 NEXT STEPS:
1. Review benchmark results and decide on training approach
2. Adjust training hyperparameters based on GPU memory constraints
3. Start full training with validated timeline:
   cargo run -p ml_training_service -- train-all
```

### Training Success (Future)

```
✅ TRAINING COMPLETE

Models trained:
- MAMBA-2: 100 epochs, 55.2% win rate, Sharpe 1.67
- DQN: 50 epochs, 53.8% win rate, Sharpe 1.52
- PPO: 50 epochs, 54.1% win rate, Sharpe 1.58
- TFT: 80 epochs, 56.3% win rate, Sharpe 1.71

Total training time: 1.3 weeks (RTX 3050 Ti)
All checkpoints saved to: checkpoints/
Ready for backtesting validation
```

---

## Troubleshooting

### Issue: API Key Not Found

```
❌ ERROR: DATABENTO_API_KEY not found in environment!
```

**Fix**:
```bash
export DATABENTO_API_KEY='your-key-here'
echo $DATABENTO_API_KEY  # Verify set
```

### Issue: GPU Not Detected

```
⚠️ GPU not detected! Benchmarks will run on CPU (much slower).
```

**Fix**:
```bash
# Check CUDA
nvidia-smi
nvcc --version

# Rebuild with CUDA
cd ml
cargo clean
cargo build --features cuda
```

### Issue: Download Failures (>20%)

```
❌ ERROR: Only downloaded 65.2% of data
```

**Fix**:
```bash
# Re-run download (skips existing files)
python3 download_ml_training_data.py

# Check specific errors
grep "ERROR" download_log.txt

# Try smaller date range
python3 download_ml_training_data.py --start-date 2024-03-01 --days 60
```

### Issue: Out of VRAM During Benchmarks

```
RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB
```

**Fix**:
```bash
# Reduce batch size
python3 benchmark_training_time.py --batch-size 16

# Reduce sequence length
python3 benchmark_training_time.py --sequence-length 50

# Test with smaller models only
python3 benchmark_training_time.py --models DQN PPO
```

---

## Summary Checklist

- [ ] Databento API key obtained and set
- [ ] Python virtual environment activated
- [ ] GPU verified with `nvidia-smi`
- [ ] Data download script run successfully (90 days, 4 symbols, ~$2)
- [ ] Downloaded data validated (>80% success rate)
- [ ] Training benchmarks run (5-10 epochs per model)
- [ ] Benchmark results analyzed (`training_benchmarks.json`)
- [ ] Training timeline calculated (local GPU vs cloud GPU decision)
- [ ] Full training approach decided (local or cloud)
- [ ] Ready to start full ML training with validated timeline

---

**Next Steps**: After completing this guide, you'll have:
1. ✅ 90 days of real market data (~180K bars)
2. ✅ Actual training time measurements on RTX 3050 Ti
3. ✅ Realistic timeline estimate (not projections)
4. ✅ Validated decision on local vs cloud GPU training
5. ✅ Confidence in training feasibility and costs

**Proceed to**: `ML_TRAINING_ROADMAP.md` with validated timeline and real hardware performance data.