foxhunt/TFT_CUDA_QUICK_REFERENCE.md

# TFT CUDA Quick Reference Card

**Status**: ✅ READY - CUDA fully configured
**Expected Speedup**: 30-60x (10-12x measured)
**Training Time**: 17-25 min (100 epochs) vs 3-5 hours CPU

---

## ⚡ Quick Commands

### Verify CUDA Setup (3 seconds)
```bash
# Check GPU
nvidia-smi

# Test CUDA connectivity
cargo run -p ml --features cuda --release --example cuda_test
```

### Train TFT with GPU (17-25 min for 100 epochs)
```bash
# CRITICAL: Always use --features cuda flag!
cargo run -p ml --features cuda --release --example train_tft_dbn -- \
  --data-dir test_data/real/databento/ml_training \
  --epochs 100 \
  --batch-size 32 \
  --use-gpu true

# Monitor GPU in separate terminal
watch -n 1 nvidia-smi
```

### Run Benchmark (30-60 min)
```bash
cargo run -p ml --features cuda --release --example gpu_training_benchmark
```

---

## 🎯 Key Configuration

| Setting | Value | Why |
|---------|-------|-----|
| **Batch Size** | 32 | Optimal for 4GB VRAM |
| **Hidden Dim** | 128 | Reduced for memory |
| **Attention Heads** | 4 | Reduced for memory |
| **Gradient Accumulation** | 16 | Simulate larger batches |
| **Expected Epoch Time** | 10-15s | 10-12x faster than CPU |
| **Expected VRAM** | 1.5-2.5 GB | Safe on 4GB GPU |
| **Expected GPU Util** | 70-95% | Compute-bound |

---

## ⚠️ Common Mistakes

### ❌ Forgot `--features cuda` flag
```bash
# WRONG - Will use CPU (10-12x slower!)
cargo run -p ml --release --example train_tft_dbn
```

**Fix**:
```bash
# CORRECT - Uses GPU
cargo run -p ml --features cuda --release --example train_tft_dbn
```

### ❌ Batch size too large
```bash
# WRONG - Will cause OOM on 4GB GPU
--batch-size 128  # ❌ OOM!
```

**Fix**:
```bash
# CORRECT - Safe for 4GB VRAM
--batch-size 32   # ✅ Safe
```

---

## 📊 Expected Performance

| Metric | GPU (RTX 3050 Ti) | CPU (AMD Ryzen) | Speedup |
|--------|------------------|-----------------|---------|
| **Epoch Time** | 10-15s | 120-180s | **10-12x** |
| **100 Epochs** | 17-25 min | 3.3-5.0 hrs | **10-12x** |
| **Inference** | <1ms | 15-25ms | **15-20x** |

---

## 🔍 Monitoring

### Real-Time GPU Monitoring
```bash
# Continuous monitoring (1-second refresh)
watch -n 1 nvidia-smi

# Memory-focused
nvidia-smi dmon -s mu -c 100
```

### Expected Metrics During Training
- **VRAM Usage**: 1.5-2.5 GB (safe margin)
- **GPU Utilization**: 70-95% (healthy)
- **Temperature**: 65-75°C (normal)
- **Power Draw**: 30-40W (near max TDP)

---

## 🚨 Troubleshooting

### GPU Utilization <50%
- Increase batch size: 16 → 32
- Check data loading speed
- Verify `--features cuda` flag used

### Out of Memory (OOM)
- Reduce batch size: 32 → 16 → 8
- Enable mixed precision
- Reduce hidden_dim: 128 → 64

### Training on CPU (slow)
- Rebuild: `cargo build -p ml --features cuda --release`
- Verify: Look for "Using device: Cuda" in logs
- Check: `nvidia-smi` should show process during training

---

## 📝 Hardware Specs

**RTX 3050 Ti**:
- 4GB VRAM (GDDR6)
- 2,560 CUDA cores
- 80 Tensor cores (3rd gen)
- CUDA Capability 8.6
- TDP 40W (mobile)

**CUDA Installation**:
- Version: 13.0.88
- Driver: 580.65.06
- cuDNN: Enabled
- Environment: Configured in `~/.bashrc`

---

## ✅ Pre-Training Checklist

Before starting 100-epoch training:

- [ ] `nvidia-smi` shows RTX 3050 Ti
- [ ] `cargo run --example cuda_test --features cuda` passes
- [ ] Run 10-epoch test first (<3 min)
- [ ] Verify GPU utilization >50% during test
- [ ] Verify VRAM usage 1.5-2.5 GB during test
- [ ] Monitor temperature stays <80°C

---

**Last Updated**: 2025-10-14
**Status**: ✅ PRODUCTION READY
**Documentation**: See `TFT_CUDA_CONFIGURATION_REPORT.md` for details