foxhunt/CUDA_VERSION_MISMATCH_ANALYSIS.md

CUDA Version Mismatch Analysis - Runpod Deployment Failure

Date: 2025-10-26 Issue: Runpod container fails with libcublas.so.13: cannot open shared object file Status: 🔴 BLOCKING DEPLOYMENT

---

Executive Summary

ROOT CAUSE: Binaries compiled against CUDA 13.0, Docker container has CUDA 12.9

• Local Development: CUDA 13.0 (default symlink)
• Compiled Binaries: Link against libcublas.so.13 + libcublasLt.so.13
• Docker Container: CUDA 12.9.1 with libcublas.so.12 + libcublasLt.so.12
• Result: Runtime library mismatch (.so.13 vs .so.12)

RECOMMENDED SOLUTION: Option A - Recompile binaries with CUDA 12.9

---

1. Binary CUDA Dependencies (ldd Output)

All 4 ML training binaries are linked against CUDA 13 libraries:

train_mamba2_parquet

libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1
libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10
libcublas.so.13 => /usr/local/cuda/lib64/libcublas.so.13          ⚠️ CUDA 13
libcudnn.so.9 => /lib/x86_64-linux-gnu/libcudnn.so.9
libcublasLt.so.13 => /usr/local/cuda/lib64/libcublasLt.so.13      ⚠️ CUDA 13

train_tft_parquet

libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1
libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10
libcublas.so.13 => /usr/local/cuda/lib64/libcublas.so.13          ⚠️ CUDA 13
libcudnn.so.9 => /lib/x86_64-linux-gnu/libcudnn.so.9
libcublasLt.so.13 => /usr/local/cuda/lib64/libcublasLt.so.13      ⚠️ CUDA 13

train_dqn

libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1
libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10
libcublas.so.13 => /usr/local/cuda/lib64/libcublas.so.13          ⚠️ CUDA 13
libcudnn.so.9 => /lib/x86_64-linux-gnu/libcudnn.so.9
libcublasLt.so.13 => /usr/local/cuda/lib64/libcublasLt.so.13      ⚠️ CUDA 13

train_ppo

libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1
libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10
libcublas.so.13 => /usr/local/cuda/lib64/libcublas.so.13          ⚠️ CUDA 13
libcudnn.so.9 => /lib/x86_64-linux-gnu/libcudnn.so.9
libcublasLt.so.13 => /usr/local/cuda/lib64/libcublasLt.so.13      ⚠️ CUDA 13

Key Finding: ALL binaries link against libcublas.so.13 and libcublasLt.so.13

---

2. Local CUDA Installation

CUDA Compiler Version

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Aug_20_01:58:59_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0

CUDA Installations

/usr/local/cuda-12.8/     # CUDA 12.8
/usr/local/cuda-12.9/     # CUDA 12.9
/usr/local/cuda-13.0/     # CUDA 13.0 (DEFAULT)

Symlink Resolution

/usr/local/cuda -> /etc/alternatives/cuda -> /usr/local/cuda-13.0

CRITICAL: /usr/local/cuda symlink points to CUDA 13.0, causing all builds to link against CUDA 13 libraries.

CUDA 13.0 Libraries

$ ls -la /usr/local/cuda/lib64/libcublas.so*
lrwxrwxrwx  libcublas.so -> libcublas.so.13
lrwxrwxrwx  libcublas.so.13 -> libcublas.so.13.0.2.14
-rw-r--r--  libcublas.so.13.0.2.14 (54 MB)

CUDA 12.9 Libraries (Available but NOT used)

$ ls -la /usr/local/cuda-12.9/lib64/libcublas.so*
lrwxrwxrwx  libcublas.so -> libcublas.so.12
lrwxrwxrwx  libcublas.so.12 -> libcublas.so.12.9.1.4
-rw-r--r--  libcublas.so.12.9.1.4 (105 MB)

lrwxrwxrwx  libcublasLt.so -> libcublasLt.so.12
lrwxrwxrwx  libcublasLt.so.12 -> libcublasLt.so.12.9.1.4
-rw-r--r--  libcublasLt.so.12.9.1.4 (749 MB)

---

3. Docker Container Configuration

Dockerfile.runpod (Line 24)

FROM nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04

CUDA Environment Variables (Lines 46-49)

ENV CUDA_HOME=/usr/local/cuda
ENV PATH="${CUDA_HOME}/bin:${PATH}"
ENV LD_LIBRARY_PATH="${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}"

Available Libraries in Container

• libcublas.so.12 (CUDA 12.9.1)
• libcublasLt.so.12 (CUDA 12.9.1)
• libcurand.so.10 (CUDA 12.9.1)
• libcudnn.so.9 (cuDNN 9)

MISMATCH: Container has .so.12, binaries expect .so.13

---

4. Candle CUDA Configuration

ml/Cargo.toml (Line 82)

# Using specific git rev (671de1db) for cudarc 0.17.3 CUDA 13.0 compatibility
# Rev 671de1db is v0.9.1 + cudarc 0.17.3 upgrade
candle-core = { git = "https://github.com/huggingface/candle", rev = "671de1db" }

Cargo.lock

[[package]]
name = "cudarc"
version = "0.17.3"
source = "registry+https://github.com/rust-lang/crates.io-index"

cudarc 0.17.3 Supported CUDA Versions

According to https://github.com/coreylowman/cudarc/blob/main/Cargo.toml:

✅ CUDA 12.9 IS SUPPORTED via cuda-12090 feature flag

Feature flags:

• cuda-11040 through cuda-11080 (CUDA 11.4-11.8)
• cuda-12000 through cuda-12090 (CUDA 12.0-12.9) ✅
• cuda-13000 (CUDA 13.0)
• cuda-version-from-build-system (auto-detect)

KEY INSIGHT: cudarc 0.17.3 supports BOTH CUDA 12.9 and 13.0

---

5. Root Cause Analysis

Why Binaries Link Against CUDA 13

1. Default CUDA Symlink: /usr/local/cuda -> /usr/local/cuda-13.0
2. Rust Build Process: Uses $CUDA_HOME or /usr/local/cuda
3. cudarc Behavior: Auto-detects CUDA version from system (likely cuda-version-from-build-system)
4. Dynamic Linking: Binaries link against detected CUDA libraries (.so.13)

Why Docker Container Has CUDA 12.9

1. Deliberate Choice: CLAUDE.md states CUDA 12.9 chosen for Runpod driver 550 compatibility
2. Driver Compatibility: CUDA 13.0 requires driver 580+ (not available on Runpod)
3. Dockerfile Base: nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04

The Conflict

┌─────────────────────────────────────────────┐
│ LOCAL DEVELOPMENT (CUDA 13.0)              │
│ Binaries: libcublas.so.13                  │
└─────────────────────────────────────────────┘
                    ↓
           ❌ INCOMPATIBLE ❌
                    ↓
┌─────────────────────────────────────────────┐
│ RUNPOD CONTAINER (CUDA 12.9.1)             │
│ Runtime: libcublas.so.12                   │
└─────────────────────────────────────────────┘

Result: Runtime error when trying to load libcublas.so.13 (not found)

---

6. Solution Options

Option A: Recompile Binaries with CUDA 12.9 ✅ RECOMMENDED

Approach: Point /usr/local/cuda to CUDA 12.9 before compilation

Steps:

# 1. Switch CUDA symlink to 12.9
sudo rm /etc/alternatives/cuda
sudo ln -s /usr/local/cuda-12.9 /etc/alternatives/cuda

# Verify
nvcc --version  # Should show CUDA 12.9
ls -la /usr/local/cuda/lib64/libcublas.so  # Should point to .so.12

# 2. Clean previous builds
cargo clean

# 3. Rebuild with CUDA 12.9
cargo build --release --features cuda --example train_tft_parquet
cargo build --release --features cuda --example train_mamba2_parquet
cargo build --release --features cuda --example train_dqn
cargo build --release --features cuda --example train_ppo

# 4. Verify linkage
ldd target/release/examples/train_tft_parquet | grep cublas
# Expected: libcublas.so.12 (not .so.13)

# 5. Upload new binaries to Runpod volume

Pros:

• ✅ Minimal changes (just symlink switch)
• ✅ Compatible with Runpod driver 550
• ✅ No Docker changes needed
• ✅ cudarc 0.17.3 already supports CUDA 12.9
• ✅ Tested Docker image (CUDA 12.9.1)

Cons:

• ⚠️ Local development uses CUDA 12.9 (downgrade from 13.0)
• ⚠️ ~10 min rebuild time (4 binaries)

Cost: $0 (local rebuild only) Time: ~10 minutes Risk: Low (CUDA 12.9 is stable, tested in Docker)

---

Option B: Upgrade Docker to CUDA 13.0 ❌ NOT RECOMMENDED

Approach: Change Dockerfile base image to CUDA 13.0

Steps:

# Dockerfile.runpod (Line 24)
FROM nvidia/cuda:13.0-cudnn-devel-ubuntu24.04

Pros:

• ✅ No rebuild needed (binaries already CUDA 13)
• ✅ Uses latest CUDA version

Cons:

• ❌ BREAKING: CUDA 13.0 requires driver 580+ (Runpod has driver 550)
• ❌ Incompatible with Runpod infrastructure
• ❌ Violates design decision in CLAUDE.md
• ❌ Docker image rebuild required
• ❌ Untested on Runpod hardware

Cost: N/A (won't work on Runpod) Time: N/A Risk: High (driver incompatibility)

Verdict: REJECTED - Runpod driver 550 cannot run CUDA 13.0

---

Option C: Static Linking / Bundle CUDA Libraries ⚠️ COMPLEX

Approach: Statically link CUDA libraries or bundle .so.13 files in Docker

Static Linking:

# Build with static CUDA libraries
export CUDA_STATIC=1
cargo build --release --features cuda

Bundle Libraries:

# Copy CUDA 13 libraries into Docker image
COPY /usr/local/cuda-13.0/lib64/libcublas.so.13* /usr/local/cuda/lib64/
COPY /usr/local/cuda-13.0/lib64/libcublasLt.so.13* /usr/local/cuda/lib64/

Pros:

• ✅ No rebuild needed
• ✅ Could work with mixed CUDA versions

Cons:

• ❌ Static linking may not be supported by cudarc
• ❌ Bundling increases Docker image size (+800MB)
• ❌ Library version conflicts (12.9 + 13.0 in same container)
• ❌ Potential ABI incompatibilities
• ❌ Complex, fragile solution

Cost: $0 (local work only) Time: 2-4 hours (experimentation) Risk: High (ABI conflicts, undefined behavior)

Verdict: NOT RECOMMENDED - Too complex, fragile, untested

---

7. Recommended Solution: Option A (Recompile with CUDA 12.9)

Implementation Plan

Phase 1: Verification (2 min)

# Check current CUDA symlink
ls -la /usr/local/cuda
# Output: /usr/local/cuda -> /usr/local/cuda-13.0

# Verify CUDA 12.9 installation
ls -la /usr/local/cuda-12.9/lib64/libcublas.so*
# Should show libcublas.so.12

Phase 2: Switch CUDA Version (1 min)

# Switch to CUDA 12.9
sudo rm /etc/alternatives/cuda
sudo ln -s /usr/local/cuda-12.9 /etc/alternatives/cuda

# Verify switch
nvcc --version
# Expected: release 12.9, V12.9.x

ls -la /usr/local/cuda/lib64/libcublas.so
# Expected: -> libcublas.so.12

Phase 3: Clean Build (5 min)

# Remove old CUDA 13 artifacts
cargo clean

# Verify clean
rm -rf target/release/examples/train_*

Phase 4: Rebuild Binaries (10 min)

# Build all 4 ML training binaries
cd /home/jgrusewski/Work/foxhunt

# TFT (~2 min)
cargo build --release --features cuda --example train_tft_parquet

# MAMBA-2 (~2 min)
cargo build --release --features cuda --example train_mamba2_parquet

# DQN (~3 min)
cargo build --release --features cuda --example train_dqn

# PPO (~3 min)
cargo build --release --features cuda --example train_ppo

Phase 5: Verify CUDA 12.9 Linkage (1 min)

# Check each binary
ldd target/release/examples/train_tft_parquet | grep cublas
# Expected: libcublas.so.12 (not .so.13)

ldd target/release/examples/train_mamba2_parquet | grep cublas
# Expected: libcublas.so.12

ldd target/release/examples/train_dqn | grep cublas
# Expected: libcublas.so.12

ldd target/release/examples/train_ppo | grep cublas
# Expected: libcublas.so.12

Phase 6: Test Locally (5 min)

# Quick smoke test (TFT - smallest dataset)
cargo run --release --features cuda --example train_tft_parquet -- \
  --parquet-file test_data/ES_FUT_180d.parquet --epochs 5

# Check for CUDA errors
# Expected: No library loading errors, training starts

Phase 7: Upload to Runpod Volume (2 min)

# Copy binaries to upload staging
mkdir -p /tmp/runpod-binaries
cp target/release/examples/train_tft_parquet /tmp/runpod-binaries/
cp target/release/examples/train_mamba2_parquet /tmp/runpod-binaries/
cp target/release/examples/train_dqn /tmp/runpod-binaries/
cp target/release/examples/train_ppo /tmp/runpod-binaries/

# Upload to Runpod S3 (using existing script)
# See scripts/upload_binaries_to_runpod.sh

Total Time: ~26 minutes Total Cost: $0 (local only)

---

8. Expected Outcomes

After Recompilation

Binary Dependencies (ldd output):

libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1
libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10
libcublas.so.12 => /usr/local/cuda-12.9/lib64/libcublas.so.12      ✅ CUDA 12.9
libcudnn.so.9 => /lib/x86_64-linux-gnu/libcudnn.so.9
libcublasLt.so.12 => /usr/local/cuda-12.9/lib64/libcublasLt.so.12  ✅ CUDA 12.9

Docker Container Runtime:

Container has: libcublas.so.12, libcublasLt.so.12
Binary needs:  libcublas.so.12, libcublasLt.so.12
Result: ✅ MATCH - Training starts successfully

Performance Impact

CUDA 12.9 vs 13.0:

• ✅ Minimal performance difference (<2% in most workloads)
• ✅ Same cuDNN 9 support
• ✅ Same Tensor Core operations
• ✅ Compatible with RTX 3050 Ti, V100, A4000

No Expected Regressions:

• Training speed: Same (both use cuBLAS + cuDNN)
• Memory usage: Same (library version doesn't affect model memory)
• Accuracy: Identical (same numerical precision)

---

9. Post-Fix Validation

Local Validation

# 1. Verify CUDA 12.9 linkage
ldd target/release/examples/train_tft_parquet | grep -E "cublas|curand"

# 2. Run full TFT training (2 min)
cargo run --release --features cuda --example train_tft_parquet -- \
  --parquet-file test_data/ES_FUT_180d.parquet --epochs 50

# 3. Check output model
ls -lh tft_model.safetensors
# Expected: ~50MB, no errors

Runpod Validation

# 1. Deploy pod with updated binaries
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"

# 2. Monitor startup logs
# Expected: "CUDA device found: ...", "Training started", NO library errors

# 3. Check training progress
# Expected: Epoch logs, loss decreasing, GPU utilization >80%

# 4. Verify output
aws s3 ls s3://se3zdnb5o4/models/ --profile runpod --recursive
# Expected: New model checkpoint uploaded

---

10. Rollback Plan

If CUDA 12.9 causes issues (unlikely):

# Revert to CUDA 13.0
sudo rm /etc/alternatives/cuda
sudo ln -s /usr/local/cuda-13.0 /etc/alternatives/cuda

# Rebuild with CUDA 13.0
cargo clean
cargo build --release --features cuda --example train_tft_parquet

# Restore old binaries
# (Keep backup before upload)

Backup Strategy: Keep CUDA 13.0 binaries in /tmp/cuda13-backup/ before upload

---

11. Long-Term Considerations

CUDA Version Management

Current State:

• Local dev: Multiple CUDA versions (12.8, 12.9, 13.0)
• Docker: CUDA 12.9.1
• Runpod: Driver 550 (CUDA 12.x max)

Recommendation: Standardize on CUDA 12.9

• ✅ Compatible with Runpod infrastructure
• ✅ Supported by cudarc 0.17.3
• ✅ Stable, production-ready
• ✅ Sufficient for current models

Future-Proofing:

• Monitor Runpod driver updates
• CUDA 13.0 upgrade when driver 580+ available
• Document CUDA version in CLAUDE.md

CI/CD Integration

Add to GitHub Actions:

- name: Verify CUDA linkage
  run: |
    ldd target/release/examples/train_tft_parquet | grep cublas
    # Fail if libcublas.so.13 detected

Pre-Upload Validation:

# scripts/validate_cuda_version.sh
#!/bin/bash
if ldd target/release/examples/train_tft_parquet | grep -q "libcublas.so.13"; then
  echo "ERROR: Binary linked against CUDA 13 (incompatible with Runpod)"
  exit 1
fi
echo "✅ CUDA 12.9 linkage verified"

---

12. Summary

The Problem

• Binaries: Compiled with CUDA 13.0 (libcublas.so.13)
• Docker: CUDA 12.9.1 (libcublas.so.12)
• Error: Runtime library mismatch

The Solution

• Switch /usr/local/cuda symlink to CUDA 12.9
• Rebuild 4 ML binaries (~10 min)
• Upload to Runpod volume
• Verify CUDA 12.9 linkage

Why This Works

• ✅ cudarc 0.17.3 supports CUDA 12.9 (cuda-12090 feature)
• ✅ CUDA 12.9 compatible with Runpod driver 550
• ✅ No Docker changes needed
• ✅ Minimal performance impact
• ✅ Tested, stable, low-risk

Timeline

• Immediate: Execute recompilation (26 min)
• Short-term: Validate on Runpod ($0.25/hr, 10 min)
• Long-term: Add CI/CD CUDA version checks

---

Appendix A: Command Reference

Check CUDA Version

nvcc --version                        # Compiler version
ls -la /usr/local/cuda               # Symlink target
ldd <binary> | grep cublas           # Binary linkage

Switch CUDA Version

# To CUDA 12.9
sudo rm /etc/alternatives/cuda
sudo ln -s /usr/local/cuda-12.9 /etc/alternatives/cuda

# To CUDA 13.0
sudo rm /etc/alternatives/cuda
sudo ln -s /usr/local/cuda-13.0 /etc/alternatives/cuda

Rebuild All Binaries

cargo clean
cargo build --release --features cuda --example train_tft_parquet
cargo build --release --features cuda --example train_mamba2_parquet
cargo build --release --features cuda --example train_dqn
cargo build --release --features cuda --example train_ppo

---

Appendix B: File Locations

Binaries

target/release/examples/train_tft_parquet       (21 MB)
target/release/examples/train_mamba2_parquet    (20 MB)
target/release/examples/train_dqn               (21 MB)
target/release/examples/train_ppo               (14 MB)

CUDA Libraries (Local)

/usr/local/cuda-12.9/lib64/libcublas.so.12      (105 MB)
/usr/local/cuda-12.9/lib64/libcublasLt.so.12    (749 MB)
/usr/local/cuda-13.0/lib64/libcublas.so.13      (54 MB)
/usr/local/cuda-13.0/lib64/libcublasLt.so.13    (N/A)

Docker Configuration

Dockerfile.runpod                               (Base: CUDA 12.9.1)
entrypoint-generic.sh                           (Training launcher)
entrypoint-self-terminate.sh                    (Auto-terminate wrapper)

---

Next Action: Execute Option A (Recompile with CUDA 12.9) - ETA 26 minutes