foxhunt/RUNPOD_QUICK_START.md

RunPod Quick Start - 30 Minute Deployment

For: Impatient developers who want to get training ASAP Time: 30 minutes to first model training Cost: $0.34-$0.68 (1-2 hour RTX 4090 training)

---

Step 1: Provision RunPod Instance (5 minutes)

1. Go to https://www.runpod.io/console/gpu-cloud
2. Select RTX 4090 (24GB VRAM, $0.34/hr spot)
3. Template: RunPod PyTorch 2.1 (CUDA 12.1 pre-installed)
4. Storage: 50GB SSD
5. Click Deploy On-Demand or Deploy Spot

Wait: 2-3 minutes for instance to start

---

Step 2: SSH + Install Rust (10 minutes)

# 1. SSH into RunPod (get details from dashboard)
ssh root@<runpod-ip> -p <ssh-port>

# 2. Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env

# 3. Clone repo
cd /workspace
git clone https://github.com/<org>/foxhunt.git
cd foxhunt

# 4. Verify CUDA
nvidia-smi  # Should show RTX 4090 with 24GB VRAM

---

Step 3: Upload Training Data (15 minutes)

Option A: rsync (Recommended)

# From local machine
rsync -avz --progress test_data/*.parquet root@<runpod-ip>:/workspace/foxhunt/test_data/

Option B: RunPod File Manager

1. Open RunPod dashboard
2. Click File Manager
3. Upload Parquet files to /workspace/foxhunt/test_data/

---

Step 4: Build + Quick Test (10 minutes)

# 1. Build ML crate with CUDA
cd /workspace/foxhunt
cargo build --release --features cuda -p ml

# 2. Quick validation test (1 epoch, ~2 minutes)
time cargo run -p ml --example train_tft_parquet --release --features cuda -- \
  --parquet-file test_data/ES_FUT_small.parquet \
  --epochs 1

# Expected: ✅ Success in ~30-45 seconds

---

Step 5: Full Training (1-2 hours)

# TFT-225 with PTQ quantization (ES.FUT 180d, 50 epochs)
time cargo run -p ml --example train_tft_parquet --release --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --epochs 50 \
  --use-int8 \
  --auto-batch-size

# Expected:
# - Training time: 1-2 hours (vs 3.5 hours local)
# - Batch size: 64-128 (vs 16-32 local)
# - GPU utilization: 85-95%
# - Checkpoint: ml/trained_models/tft_225_epoch_50.safetensors

---

Step 6: Download Checkpoint (5 minutes)

# From local machine
scp -P <ssh-port> \
  root@<runpod-ip>:/workspace/foxhunt/ml/trained_models/tft_225_epoch_50.safetensors \
  ./local_models/

---

Step 7: Terminate Instance (1 minute)

IMPORTANT: Don't forget to terminate the RunPod instance to avoid charges!

1. Go to RunPod dashboard
2. Click Terminate on your pod
3. Confirm termination

Cost: ~$0.68-$1.36 for 2-4 hours

---

🔥 Common Issues

Issue 1: OOM Error

Fix: Reduce batch size

--batch-size 32  # Instead of auto-batch-size

Issue 2: CUDA Error

Fix: Verify CUDA

nvidia-smi  # Check GPU is available
nvcc --version  # Check CUDA toolkit

Issue 3: Compilation Error

Fix: Clean build

cargo clean
cargo build --release --features cuda -p ml

---

📊 Performance Comparison

Metric	RTX 3050 Ti (4GB)	RTX 4090 (24GB)	Speedup
Training Time	3.5 hours	1-2 hours	3.5x
Batch Size	16-32	64-128	4x
GPU Util	60-70%	85-95%	+25%

---

🎯 Next Steps

After successful deployment:

1. Train All 4 Models (2-3 hours):

   ./run_training.sh --sequential
   

2. Validate INT8 Accuracy (30 minutes):

   • Compare FP32 vs INT8 checkpoints
   • Test inference latency (<5ms target)

3. Deploy to Production (1 week):

   • See WAVE_D_DEPLOYMENT_GUIDE.md

---

Full Guide: See RUNPOD_DEPLOYMENT_READY.md for comprehensive documentation

Support: Discord #ml-training or #runpod-cloud

Total Cost: $0.68-$1.36 per training run (negligible vs developer time)