foxhunt/docs/archive/ml_models/TFT_CUDA_QUICK_REFERENCE.md

TFT CUDA Quick Reference Card

Status: ✅ READY - CUDA fully configured Expected Speedup: 30-60x (10-12x measured) Training Time: 17-25 min (100 epochs) vs 3-5 hours CPU

---

⚡ Quick Commands

Verify CUDA Setup (3 seconds)

# Check GPU
nvidia-smi

# Test CUDA connectivity
cargo run -p ml --features cuda --release --example cuda_test

Train TFT with GPU (17-25 min for 100 epochs)

# CRITICAL: Always use --features cuda flag!
cargo run -p ml --features cuda --release --example train_tft_dbn -- \
  --data-dir test_data/real/databento/ml_training \
  --epochs 100 \
  --batch-size 32 \
  --use-gpu true

# Monitor GPU in separate terminal
watch -n 1 nvidia-smi

Run Benchmark (30-60 min)

cargo run -p ml --features cuda --release --example gpu_training_benchmark

---

🎯 Key Configuration

Setting	Value	Why
Batch Size	32	Optimal for 4GB VRAM
Hidden Dim	128	Reduced for memory
Attention Heads	4	Reduced for memory
Gradient Accumulation	16	Simulate larger batches
Expected Epoch Time	10-15s	10-12x faster than CPU
Expected VRAM	1.5-2.5 GB	Safe on 4GB GPU
Expected GPU Util	70-95%	Compute-bound

---

⚠️ Common Mistakes

❌ Forgot --features cuda flag

# WRONG - Will use CPU (10-12x slower!)
cargo run -p ml --release --example train_tft_dbn

Fix:

# CORRECT - Uses GPU
cargo run -p ml --features cuda --release --example train_tft_dbn

❌ Batch size too large

# WRONG - Will cause OOM on 4GB GPU
--batch-size 128  # ❌ OOM!

Fix:

# CORRECT - Safe for 4GB VRAM
--batch-size 32   # ✅ Safe

---

📊 Expected Performance

Metric	GPU (RTX 3050 Ti)	CPU (AMD Ryzen)	Speedup
Epoch Time	10-15s	120-180s	10-12x
100 Epochs	17-25 min	3.3-5.0 hrs	10-12x
Inference	<1ms	15-25ms	15-20x

---

🔍 Monitoring

Real-Time GPU Monitoring

# Continuous monitoring (1-second refresh)
watch -n 1 nvidia-smi

# Memory-focused
nvidia-smi dmon -s mu -c 100

Expected Metrics During Training

• VRAM Usage: 1.5-2.5 GB (safe margin)
• GPU Utilization: 70-95% (healthy)
• Temperature: 65-75°C (normal)
• Power Draw: 30-40W (near max TDP)

---

🚨 Troubleshooting

GPU Utilization <50%

• Increase batch size: 16 → 32
• Check data loading speed
• Verify --features cuda flag used

Out of Memory (OOM)

• Reduce batch size: 32 → 16 → 8
• Enable mixed precision
• Reduce hidden_dim: 128 → 64

Training on CPU (slow)

• Rebuild: cargo build -p ml --features cuda --release
• Verify: Look for "Using device: Cuda" in logs
• Check: nvidia-smi should show process during training

---

📝 Hardware Specs

RTX 3050 Ti:

• 4GB VRAM (GDDR6)
• 2,560 CUDA cores
• 80 Tensor cores (3rd gen)
• CUDA Capability 8.6
• TDP 40W (mobile)

CUDA Installation:

• Version: 13.0.88
• Driver: 580.65.06
• cuDNN: Enabled
• Environment: Configured in ~/.bashrc

---

✅ Pre-Training Checklist

Before starting 100-epoch training:

• nvidia-smi shows RTX 3050 Ti
• cargo run --example cuda_test --features cuda passes
• Run 10-epoch test first (<3 min)
• Verify GPU utilization >50% during test
• Verify VRAM usage 1.5-2.5 GB during test
• Monitor temperature stays <80°C

---

Last Updated: 2025-10-14 Status: ✅ PRODUCTION READY Documentation: See TFT_CUDA_CONFIGURATION_REPORT.md for details