foxhunt/runpod_debug/DEPLOY_TESTS.md

RunPod Minimal Test Deployment Guide

Purpose: Step-by-step instructions to run minimal reproduction tests on RunPod.

---

Quick Start (5 minutes)

1. Create RunPod Pod

Go to RunPod dashboard and create a pod with:

• GPU: RTX 4090 or RTX 3060 (any CUDA-capable)
• Template: runpod/pytorch:2.1.0-py3.10-cuda12.1.0-devel-ubuntu22.04
• Network Volume: Attach se3zdnb5o4 (already contains test binaries)
• Disk: 20 GB (minimum)

2. Connect via SSH

Click "Connect" → "SSH over exposed TCP" and run:

ssh root@<pod-ip> -p <port> -i ~/.ssh/id_ed25519

3. Run Test 1 (Minimal Hello)

# Make executable
chmod +x /runpod-volume/debug_tests/test1_hello

# Run test
/runpod-volume/debug_tests/test1_hello

Expected Output:

=== TEST 1: HELLO FROM RUNPOD ===
This is the simplest possible Rust binary
Sleeping for 10 seconds to keep pod alive...
=== TEST 1 COMPLETE - EXITING CLEANLY ===

If it crashes: Pod/container environment issue (report glibc version, kernel)

If it succeeds: ✅ Pod environment is fine, proceed to Test 2

4. Run Test 2 (CUDA Check)

# Make executable
chmod +x /runpod-volume/debug_tests/test2_cuda_check

# Run test
/runpod-volume/debug_tests/test2_cuda_check

Expected Output:

Test 2: Checking CUDA availability
nvidia-smi stdout:
[CUDA device info]
CUDA_HOME = /usr/local/cuda
...
Test 2 complete

If it crashes: CUDA runtime issue (report CUDA version, driver version)

If it succeeds: ✅ CUDA is accessible, problem is in ML binary

5. Build Training Binary on RunPod (Critical Test)

If Tests 1-2 succeed, the problem is CUDA version mismatch. Build natively:

# Install Rust (if not present)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env

# Clone repo
cd /workspace
git clone https://github.com/user/foxhunt.git
cd foxhunt

# Start Docker services (if needed)
docker-compose up -d postgres redis

# Build training binary with RunPod's native CUDA
cd ml
cargo build --release --example train_tft_parquet --features cuda

# Check binary
ls -lh ../target/release/examples/train_tft_parquet

# Run test
../target/release/examples/train_tft_parquet \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --epochs 1 \
  --batch-size 32

If build succeeds and runs: ✅ CUDA 13.0 vs 12.x mismatch confirmed

If build fails: Check error messages for missing dependencies

If build succeeds but crashes: Different issue (memory, GPU, data file)

---

Detailed Troubleshooting

Test 1 Crashes

Symptoms: Segfault, illegal instruction, "not found" errors

Likely Causes:

1. glibc version too old (need ≥2.31)
2. CPU architecture mismatch (need x86_64)
3. Missing system libraries

Debug Commands:

# Check glibc
ldd --version

# Check CPU
uname -m

# Check missing libs
ldd /runpod-volume/debug_tests/test1_hello

# Run with verbose errors
RUST_BACKTRACE=full /runpod-volume/debug_tests/test1_hello

Test 2 Crashes

Symptoms: CUDA errors, driver issues, device not found

Likely Causes:

1. NVIDIA driver not loaded
2. CUDA runtime libraries missing
3. GPU not accessible

Debug Commands:

# Check NVIDIA driver
nvidia-smi

# Check CUDA libraries
ls -la /usr/local/cuda*/lib*/

# Check LD_LIBRARY_PATH
echo $LD_LIBRARY_PATH

# Test CUDA directly
nvidia-smi -L

Build on RunPod Fails

Symptoms: Cargo errors, linker errors, cudarc panics

Likely Causes:

1. CUDA development headers missing
2. Insufficient disk space
3. Cargo/Rust version too old

Debug Commands:

# Check CUDA dev tools
nvcc --version
which nvcc

# Check disk space
df -h

# Check Rust version
rustc --version
cargo --version

# Update Rust if needed
rustup update stable

---

Success Criteria

✅ Scenario 1: Tests 1-2 work, native build works

Conclusion: CUDA 13.0 vs 12.x mismatch confirmed Solution: Always build on RunPod or use CUDA 12.x locally Action: Update deployment docs, create Dockerfile with CUDA 12.x

✅ Scenario 2: Tests 1-2 work, native build fails

Conclusion: Missing dependencies or configuration Solution: Install missing packages, fix environment Action: Update Dockerfile with all dependencies

🔴 Scenario 3: Test 1 crashes

Conclusion: Fundamental runtime incompatibility Solution: Use different base image or build statically Action: Try alpine-based image or musl target

🔴 Scenario 4: Test 2 crashes but test 1 works

Conclusion: CUDA driver or runtime issue Solution: Use RunPod template with CUDA pre-installed Action: Switch to runpod/pytorch or nvidia/cuda base image

---

Reporting Results

After running tests, report back with:

1. Test 1 Result: SUCCESS / CRASH
2. Test 2 Result: SUCCESS / CRASH (if test 1 succeeded)
3. Native Build Result: SUCCESS / FAILED (if tests 1-2 succeeded)
4. Logs: Full output from all tests
5. Environment:
   • nvidia-smi output
   • nvcc --version output
   • ldd --version output
   • uname -a output

---

Files Location

• Test Binaries: /runpod-volume/debug_tests/test1_hello, test2_cuda_check
• Training Data: /runpod-volume/test_data/ES_FUT_180d.parquet (if uploaded)
• Repo Clone: /workspace/foxhunt/ (create during native build test)

---

Time Estimate

• Pod creation: 2 min
• SSH setup: 1 min
• Test 1: 30 sec
• Test 2: 30 sec
• Native build: 5-10 min (cargo compile)
• Total: 10-15 min

---

Next Steps After Testing

If CUDA mismatch confirmed:

1. Update RUNPOD_DEPLOYMENT_CHECKLIST.md with "build on RunPod" requirement
2. Create Dockerfile.runpod_build with CUDA 12.x
3. Document cross-compilation setup for local dev

If different issue found:

1. Create new agent to address specific problem
2. Update hypothesis in AGENT_05_MINIMAL_REPRODUCTION.md
3. Implement fix and retest

---

Contact

Questions? See:

• AGENT_05_MINIMAL_REPRODUCTION.md for technical details
• RUNPOD_DEPLOYMENT_CHECKLIST.md for deployment context
• QAT_BLOCKERS_ROOT_CAUSE_ANALYSIS.md for ML training issues