foxhunt/CUDA_PTX_VERSION_FIX.md

CUDA PTX Version Mismatch Fix

Date: 2025-10-27 Status: DIAGNOSED - FIX READY Issue: CUDA_ERROR_UNSUPPORTED_PTX_VERSION Root Cause: Binary compiled with CUDA 12.9, driver expects CUDA 13.0 PTX

---

Diagnosis Summary

System Configuration

Component	Version	Status
GPU	NVIDIA GeForce RTX 3050 Ti	✅
GPU Compute Capability	8.6 (sm_86)	✅
Driver Version	580.65.06	✅
Driver CUDA Support	13.0	✅
Installed CUDA Toolkits	12.8, 12.9, 13.0	✅
Default CUDA Symlink	12.9	⚠️ MISMATCH
nvcc Version	12.9.86	⚠️ MISMATCH
Binary Compiled With	CUDA 12.9 PTX	⚠️ MISMATCH

Root Cause

The error occurs because:

1. Driver 580.65.06 supports CUDA 13.0 (and is optimized for it)
2. Binary was compiled with CUDA 12.9 PTX instructions
3. PTX forward compatibility only works within the same major version
4. CUDA 12.9 → 13.0 crossing major version boundary causes PTX rejection

Error Message:

CUDA error: CUDA_ERROR_UNSUPPORTED_PTX_VERSION: the provided PTX was compiled with an unsupported toolchain.

This is NOT a "driver too old" issue - it's a "binary too old for driver" issue.

---

Solution: Rebuild with CUDA 13.0

Option A: Without Changing System Default (RECOMMENDED)

Script: /tmp/cuda_fix_no_sudo.sh

#!/bin/bash
cd /home/jgrusewski/Work/foxhunt

# Clean previous builds
cargo clean

# Rebuild with explicit CUDA 13.0
export CUDA_COMPUTE_CAP="sm_86"  # RTX 3050 Ti
export CUDA_PATH="/usr/local/cuda-13.0"
export PATH="/usr/local/cuda-13.0/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-13.0/lib64:$LD_LIBRARY_PATH"

cargo build -p ml --example hyperopt_mamba2_demo --release --features cuda

# Test
./target/release/examples/hyperopt_mamba2_demo --help

Advantages:

• No system changes required
• No sudo needed
• Safe for other projects using CUDA 12.9

Run with:

/tmp/cuda_fix_no_sudo.sh

---

Option B: Change System Default (Requires sudo)

Script: /tmp/cuda_fix_commands.sh

#!/bin/bash
# Switch system CUDA to 13.0
sudo ln -sf /usr/local/cuda-13.0 /usr/local/cuda

cd /home/jgrusewski/Work/foxhunt
cargo clean
export CUDA_COMPUTE_CAP="sm_86"
cargo build -p ml --example hyperopt_mamba2_demo --release --features cuda

Advantages:

• Permanent fix for all future builds
• Matches driver version

Disadvantages:

• Requires sudo
• May affect other projects

Run with:

/tmp/cuda_fix_commands.sh

---

Verification Steps

After rebuilding, test with:

# Quick test (should not crash)
./target/release/examples/hyperopt_mamba2_demo --help

# Smoke test (1 trial, 1 epoch - expect OOM or success)
./target/release/examples/hyperopt_mamba2_demo \
  --parquet-file test_data/ES_FUT_small.parquet \
  --trials 1 \
  --epochs 1

# Full test (if smoke test passes)
./target/release/examples/hyperopt_mamba2_demo \
  --parquet-file test_data/ES_FUT_small.parquet \
  --trials 20 \
  --epochs 50

Expected Results:

• ✅ No CUDA_ERROR_UNSUPPORTED_PTX_VERSION
• ✅ Training starts (may hit OOM on 4GB GPU, which is expected)
• ✅ Binary runs without PTX errors

---

Alternative: Skip Local Validation, Use Runpod Only

Given that:

1. Local validation already confirmed 13 parameters work correctly
2. Runpod uses CUDA 12.9.1 (matches the current binary)
3. OOM on 4GB GPU is expected behavior for full optimization

Recommended Path:

1. ✅ Skip local execution entirely
2. ✅ Deploy directly to Runpod with existing CUDA 12.9 Docker image
3. ✅ Run hyperopt on RTX A4000 16GB (no CUDA mismatch, no OOM)

Runpod Deployment:

# Build Docker (CUDA 12.9.1 - compatible with Runpod driver 550)
docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
docker push jgrusewski/foxhunt:latest

# Deploy pod
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"

# Run hyperopt inside pod
docker exec -it <container> /runpod-volume/binaries/hyperopt_mamba2_demo \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --trials 20 \
  --epochs 50

---

Impact on Deployment

Local Development (RTX 3050 Ti)

• Before: CUDA 12.9 PTX → Driver 580 (CUDA 13.0) = ERROR
• After Fix: CUDA 13.0 PTX → Driver 580 (CUDA 13.0) = SUCCESS
• Binary Size: ~20MB (unchanged)
• Training Speed: Same (GPU-accelerated)

Runpod Deployment (CUDA 12.9.1)

• No changes needed - Runpod uses CUDA 12.9.1 Docker image
• Current binary (CUDA 12.9) already compatible with Runpod
• Driver 550 on Runpod supports CUDA 12.9 perfectly

Docker Image

• Dockerfile.runpod uses CUDA 12.9.1 base image
• No rebuild needed - image already correct for Runpod
• Local CUDA 13.0 fix only affects local development

---

Recommendation

CHOOSE ONE:

Path 1: Fix Local + Keep Runpod As-Is (RECOMMENDED)

1. Run /tmp/cuda_fix_no_sudo.sh (rebuild with CUDA 13.0 locally)
2. Test locally with --trials 1 --epochs 1
3. Deploy to Runpod with existing CUDA 12.9 Docker image
4. Run full optimization on Runpod (no CUDA mismatch, no OOM)

Advantages:

• Local dev environment fixed (no PTX errors)
• Runpod unchanged (already correct)
• Best of both worlds

Path 2: Skip Local, Use Runpod Only (FASTEST)

1. Skip local execution entirely
2. Deploy directly to Runpod with existing Docker image
3. Run hyperopt on RTX A4000 16GB (16x more memory than local)

Advantages:

• No local rebuild needed
• Faster time to results
• Avoids OOM on 4GB GPU

---

Files Created

• /tmp/cuda_fix_no_sudo.sh - Fix script without sudo (Option A)
• /tmp/cuda_fix_commands.sh - Fix script with sudo (Option B)
• /home/jgrusewski/Work/foxhunt/CUDA_PTX_VERSION_FIX.md - This document

---

Next Steps

1. CHOOSE: Path 1 (fix local) or Path 2 (skip local)
2. IF Path 1: Run /tmp/cuda_fix_no_sudo.sh
3. IF Path 2: Deploy to Runpod immediately
4. VERIFY: Test with smoke test (1 trial, 1 epoch)
5. RUN: Full optimization (20 trials, 50 epochs)

---

Success Criteria

PASS if ANY of:

• ✅ Binary runs locally without CUDA_ERROR_UNSUPPORTED_PTX_VERSION (Path 1)
• ✅ Hyperopt runs successfully on Runpod (Path 2)
• ✅ Training starts and completes at least 1 epoch

Expected Timeline:

• Path 1: 15 min (rebuild 5 min + test 10 min)
• Path 2: 10 min (deploy 5 min + start training 5 min)

---

Status

• ✅ Root cause identified: CUDA 12.9 PTX vs. CUDA 13.0 driver
• ✅ Solution designed: Rebuild with CUDA 13.0 or deploy to Runpod
• ⏳ Fix pending: User choice between Path 1 or Path 2
• ⏳ Verification pending: Smoke test after fix

---

RECOMMENDATION: Use Path 1 (fix local) - it only takes 15 minutes and ensures local dev environment is production-ready.