foxhunt/CUDA_VERSION_ENFORCEMENT_QUICK_START.md

CUDA Version Enforcement - Quick Start Guide

Date: 2025-10-27 Status: Ready for Implementation Time Required: 75 minutes (4 phases) Cost: $0.15 (testing only)

---

The Problem (In 30 Seconds)

• Local builds use CUDA 13.0 (default symlink)
• Runpod runtime uses CUDA 12.9.1 (driver 550 limit)
• Result: PTX version mismatch = binaries crash on Runpod
• Solution: Enforce CUDA 12.4-12.9 at build time (prevent, don't react)

---

Implementation Steps

Phase 1: Core Enforcement (30 min) - DO THIS NOW

Step 1.1: Update ml/build.rs (10 min)

Replace /home/jgrusewski/Work/foxhunt/ml/build.rs with the version in AGENT_4_CUDA_VERSION_ENFORCEMENT_PLAN.md (lines 86-168).

Step 1.2: Create validation script (10 min)

Create /home/jgrusewski/Work/foxhunt/scripts/validate_cuda_env.sh from the plan (lines 203-306).

Make executable:

chmod +x scripts/validate_cuda_env.sh

Step 1.3: Revert Dockerfile (2 min)

Edit /home/jgrusewski/Work/foxhunt/Dockerfile.runpod line 24:

-FROM nvidia/cuda:13.0.0-devel-ubuntu22.04
+FROM nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04

Step 1.4: Test (8 min)

# Test validation script
./scripts/validate_cuda_env.sh

# If CUDA 13.0 detected, switch to 12.9
sudo rm /etc/alternatives/cuda
sudo ln -s /usr/local/cuda-12.9 /etc/alternatives/cuda

# Verify
nvcc --version  # Should show CUDA 12.9

# Test build (should succeed)
cargo clean
cargo build -p ml --release --features cuda --example train_tft_parquet

# Verify linkage
ldd target/release/examples/train_tft_parquet | grep cublas
# Expected: libcublas.so.12 (not .so.13)

---

Phase 2: Deployment Integration (20 min)

Step 2.1: Enhance deployment script (10 min)

Add validation functions to scripts/runpod_deploy.py:

• Insert lines 17-96 from plan (binary validation functions)
• Insert line 356 from plan (call validate_all_binaries())

Step 2.2: Test deployment validation (5 min)

# Dry run (should validate binaries)
python3 scripts/runpod_deploy.py --dry-run

# Expected: ✅ All binaries validated (CUDA 12.x compatible)

Step 2.3: Verify Docker (5 min)

# Rebuild Docker image
docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .

# Verify CUDA 12.9.1
docker run --rm jgrusewski/foxhunt:latest bash -c "nvcc --version"
# Expected: release 12.9

---

Phase 3: Documentation (15 min)

Step 3.1: Update CLAUDE.md (5 min)

Add CUDA requirements section after line 360 (see plan lines 535-554).

Step 3.2: Update ML_TRAINING_PARQUET_GUIDE.md (5 min)

Add CUDA validation section (see plan lines 559-579).

Step 3.3: Optional - Create CI/CD workflow (5 min)

Create .github/workflows/build-binaries.yml from plan (lines 388-434).

---

Phase 4: Validation & Deployment (10 min)

Step 4.1: Rebuild all binaries (5 min)

# Ensure CUDA 12.9 active
./scripts/validate_cuda_env.sh

# Clean
cargo clean

# Build all 4 models
cargo build -p ml --release --features cuda --example train_tft_parquet
cargo build -p ml --release --features cuda --example train_mamba2_parquet
cargo build -p ml --release --features cuda --example train_dqn
cargo build -p ml --release --features cuda --example train_ppo

# Verify all have CUDA 12 linkage
for binary in target/release/examples/train_*; do
  echo "Checking $binary..."
  ldd "$binary" | grep cublas
done
# All should show libcublas.so.12

Step 4.2: Upload to Runpod (2 min)

# Upload binaries to Runpod volume
# (Use existing upload script or manual upload via S3)

Step 4.3: Deploy test pod (3 min)

# Deploy with validation
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"

# Monitor pod startup
# Expected: Training starts, NO PTX errors

---

Testing Checklist

After implementation, verify:

• ./scripts/validate_cuda_env.sh exits 0 with CUDA 12.9
• ./scripts/validate_cuda_env.sh exits 1 with CUDA 13.0
• Build with CUDA 13.0 fails with clear error message
• Build with CUDA 12.9 succeeds with "✅ CUDA 12.9 detected"
• ldd shows libcublas.so.12 (not .so.13)
• Deployment script validates binaries pre-upload
• Docker image has CUDA 12.9.1 (not 13.0)
• Runpod pod trains successfully (NO PTX errors)

---

Rollback (If Needed)

If implementation breaks builds:

# Revert changes
git checkout HEAD~1 ml/build.rs
git checkout HEAD~1 Dockerfile.runpod
git checkout HEAD~1 scripts/runpod_deploy.py
rm scripts/validate_cuda_env.sh

# Clean and rebuild
cargo clean
cargo build --release --features cuda

Timeline: 2 minutes

---

Quick Commands

Check CUDA Version

nvcc --version
ls -la /usr/local/cuda

Switch to CUDA 12.9

sudo rm /etc/alternatives/cuda
sudo ln -s /usr/local/cuda-12.9 /etc/alternatives/cuda
nvcc --version  # Verify

Validate Environment

./scripts/validate_cuda_env.sh

Build with Validation

cargo clean
cargo build -p ml --release --features cuda

Verify Binary

ldd target/release/examples/train_tft_parquet | grep cublas
# Expected: libcublas.so.12

Deploy to Runpod

python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"

---

Expected Error Messages

If CUDA 13.0 Detected at Build Time

╔═══════════════════════════════════════════════════════════════════╗
║ ❌ CUDA VERSION ERROR - BUILD ABORTED                            ║
╚═══════════════════════════════════════════════════════════════════╝

  Detected CUDA: 13.0 (TOO NEW)
  Required:      12.4 - 12.9
  Reason:        Runpod driver 550 does NOT support CUDA 13.0+

┌───────────────────────────────────────────────────────────────────┐
│ FIX: Switch to CUDA 12.9                                          │
└───────────────────────────────────────────────────────────────────┘

  sudo rm /etc/alternatives/cuda
  sudo ln -s /usr/local/cuda-12.9 /etc/alternatives/cuda
  nvcc --version  # Verify CUDA 12.9

  cargo clean
  cargo build --release --features cuda

Fix: Follow the instructions in the error message

---

If CUDA 13 Binary Detected at Deployment

❌ ERROR: target/release/examples/train_tft_parquet linked against CUDA 13 (incompatible with Runpod)
   Expected: libcublas.so.12, libcublasLt.so.12
   Found: CUDA 13 libraries

   FIX: Rebuild with CUDA 12.9:
   1. ./scripts/validate_cuda_env.sh
   2. cargo clean
   3. cargo build --release --features cuda

❌ DEPLOYMENT BLOCKED: Binaries compiled with incompatible CUDA version
   Runpod requires CUDA 12.x (driver 550 does not support CUDA 13.0+)

Fix: Rebuild binaries with CUDA 12.9

---

Success Message

✅ CUDA 12.9 is compatible with Runpod driver 550

  Docker Image:  nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04
  Binary PTX:    Will use CUDA 12.9 format
  Runtime:       Compatible (Runpod has CUDA 12.9.1)

---

Cost & Timeline Summary

Phase	Time	Cost	Blocker
1. Core Enforcement	30 min	$0	None
2. Deployment Integration	20 min	$0	Phase 1
3. Documentation	15 min	$0	Phase 2
4. Validation	10 min	$0.15	Phase 3
TOTAL	75 min	$0.15	-

---

Why This Matters

Before Implementation:

• ❌ Builds use whatever CUDA version system has (13.0 default)
• ❌ No validation until runtime (Runpod deployment fails)
• ❌ PTX errors are cryptic and hard to debug
• ❌ Wastes time and money ($0.25/hr Runpod while debugging)

After Implementation:

• ✅ Build fails immediately if wrong CUDA version (10 sec feedback)
• ✅ Clear error messages with exact fix instructions
• ✅ Multiple validation layers (build, pre-deploy, runtime)
• ✅ Zero PTX errors on Runpod (prevented at source)
• ✅ Saves debugging time and deployment cost

---

Next Steps

1. Read full plan: /home/jgrusewski/Work/foxhunt/AGENT_4_CUDA_VERSION_ENFORCEMENT_PLAN.md
2. Execute Phase 1: Core enforcement (30 min) - START HERE
3. Execute Phase 2: Deployment integration (20 min)
4. Execute Phase 3: Documentation (15 min)
5. Execute Phase 4: Validation & deployment (10 min)
6. Monitor: First Runpod deployment for any PTX errors

---

Support

Full Documentation: See AGENT_4_CUDA_VERSION_ENFORCEMENT_PLAN.md

Questions:

• How do I check my CUDA version? → nvcc --version
• How do I switch CUDA versions? → See "Switch to CUDA 12.9" section
• What if I don't have CUDA 12.9? → Install from NVIDIA CUDA Archive
• What if build still fails? → Check rollback section, revert changes

---

Status: ✅ Ready for Implementation Priority: P0 (blocks Runpod deployment) Confidence: 95% (thoroughly planned, low risk) Recommendation: Execute Phase 1 immediately (30 min)