# GPU Training Benchmark - Quick Start

## Prerequisites
- RTX 3050 Ti GPU (4GB VRAM) or equivalent
- CUDA 12.8+ installed
- 360 DBN files in `test_data/real/databento/ml_training/`

## 1. Run Benchmark (Default: 10 epochs)
```bash
cd /home/jgrusewski/Work/foxhunt
cargo run -p ml --example gpu_training_benchmark --release
```

**Expected Duration**: 30-60 minutes
**Output**: `ml/benchmark_results/gpu_training_benchmark_{timestamp}.json`

## 2. Interpret Results

The JSON report contains:
- `dqn_results.statistics.mean_seconds` - Average epoch time for DQN
- `ppo_results.statistics.mean_seconds` - Average epoch time for PPO
- `decision.recommendation` - "local_gpu" or "cloud_gpu"
- `decision.rationale` - Why this recommendation

## 3. Decision Framework

**Recommendation: "local_gpu"**
- Total training time < 24 hours
- Action: Proceed with local RTX 3050 Ti training
- Cost: $0 (local hardware)

**Recommendation: "cloud_gpu"**
- Total training time > 48 hours
- Action: Use cloud GPU (A100 recommended)
- Cost: ~$250/week

**Recommendation: "either"**
- Total training time 24-48 hours
- Action: User choice based on urgency/cost
- Compare `estimated_cost_local` vs `estimated_cost_cloud`

## 4. Advanced Options

### Custom epochs (more accurate)
```bash
cargo run -p ml --example gpu_training_benchmark --release -- --epochs 20
```

### CPU-only mode (testing)
```bash
cargo run -p ml --example gpu_training_benchmark --release -- --cpu-only
```

### Custom output path
```bash
cargo run -p ml --example gpu_training_benchmark --release -- --output my_results.json
```

## 5. Troubleshooting

**Error: "CUDA out of memory"**
- TFT model requires 1.5-2.5GB VRAM
- Solution: Batch size automatically limited to 4
- If still OOM: Close other GPU applications

**Warning: "Thermal throttling detected"**
- GPU temperature >85°C
- Solution: Improve cooling, reduce room temperature
- Training will continue but slower

**Error: "DBN files not found"**
- Missing training data
- Solution: Run download script first:
  ```bash
  cargo run -p data --example download_ml_training_data --release
  ```

## 6. Full Documentation

See `ml/docs/GPU_BENCHMARK_GUIDE.md` for comprehensive documentation.

## Expected Output Example

```json
{
  "timestamp": "2025-10-13T12:30:00Z",
  "decision": {
    "recommendation": "local_gpu",
    "rationale": "Training time 18.5 hours < 24h threshold. Local GPU viable.",
    "estimated_local_hours": 18.5,
    "estimated_cost_local": 0.42,
    "estimated_cost_cloud": 9.73
  },
  "dqn_results": {
    "statistics": {
      "mean_seconds": 0.045,
      "confidence_interval_95": [0.042, 0.048],
      "p95": 0.052,
      "p99": 0.058
    },
    "memory_peak_mb": 127.3,
    "stability": {"is_stable": true}
  },
  "ppo_results": { ... }
}
```

---

**Questions?** See full documentation or troubleshooting guide.