foxhunt/runpod_debug/AGENT_05_MINIMAL_REPRODUCTION.md

Agent 5: Minimal Reproduction Test

Objective: Isolate the exact failure point through systematic binary testing.

Status: ✅ Binaries uploaded, ready for deployment testing

---

Test Strategy

We've created minimal test binaries with escalating complexity to isolate the crash:

1. Test 1 (test1_hello): Bare-bones Rust binary (no dependencies)
2. Test 2 (test2_cuda_check): System checks (nvidia-smi, env vars)
3. Test 3 (test3_candle_device): Candle Device initialization (BLOCKED: CUDA 13.0 not supported by cudarc)
4. Test 4 (test4_parquet_read): Parquet file reading (BLOCKED: requires candle)
5. Test 5 (test5_tft_minimal): Minimal TFT model (BLOCKED: requires candle)

---

Current Status

✅ Tests 1-2: Built and Uploaded

• test1_hello: 3.6 MiB (pure Rust, no deps)
• test2_cuda_check: 3.7 MiB (nvidia-smi + env checks)
• Location: s3://se3zdnb5o4/debug_tests/
• Verified: Both work locally on RTX 3050 Ti

🔴 Tests 3-5: Build Blocked

Root Cause: cudarc 0.13.9 doesn't support CUDA 13.0

thread 'main' panicked at build.rs:92:14:
Unsupported cuda toolkit version: `13.0`. Please raise a github issue.

Impact: Cannot build any Candle-based tests locally

---

Deployment Instructions

Option A: Manual Deployment (via RunPod Web UI)

1. Create Pod:

   • Template: GPU-enabled (RTX 4090/3060)
   • Network Volume: se3zdnb5o4 (already attached)
   • Container: ubuntu:22.04 or nvidia/cuda:12.1.0-runtime-ubuntu22.04

2. SSH into pod and run tests:

   # Test 1: Minimal Hello
   chmod +x /runpod-volume/debug_tests/test1_hello
   /runpod-volume/debug_tests/test1_hello
   
   # Test 2: CUDA Check
   chmod +x /runpod-volume/debug_tests/test2_cuda_check
   /runpod-volume/debug_tests/test2_cuda_check
   

3. Check logs for:

   • Does test1 complete successfully?
   • Does test2 detect CUDA properly?
   • Any segfaults or crashes?

Option B: Docker Deployment (Recommended)

Create a minimal test image:

# Dockerfile.test
FROM ubuntu:22.04

# Copy test binaries
COPY runpod_debug/target/test1_hello /test1_hello
COPY runpod_debug/target/test2_cuda_check /test2_cuda_check

RUN chmod +x /test1_hello /test2_cuda_check

# Default to test1
CMD ["/test1_hello"]

Deploy with different tests:

# Test 1
docker run --gpus all test-image /test1_hello

# Test 2
docker run --gpus all test-image /test2_cuda_check

---

Critical Discovery: CUDA 13.0 Incompatibility

Problem

Our local dev machine has CUDA 13.0, but:

• cudarc (Candle's CUDA wrapper) only supports up to CUDA 12.6
• RunPod GPUs likely use CUDA 12.x

This Could Be The Root Cause!

The training binary built locally with CUDA 13.0 libraries may crash on RunPod's CUDA 12.x environment.

Solution Path

Option 1: Build on RunPod directly (use their CUDA 12.x)

# Inside RunPod pod
git clone <repo>
cd foxhunt/ml
cargo build --release --example train_tft_parquet --features cuda

Option 2: Use CUDA 12.x compatible build machine

• Downgrade local CUDA to 12.6
• Or use Docker with CUDA 12.x base image

Option 3: Static linking workaround

• Build with RUSTFLAGS='-C target-feature=+crt-static'
• May not work for CUDA libs

---

Next Steps

Immediate Actions (Manual Testing)

1. Deploy pod with test1_hello

   • Expected: Should run perfectly (no deps)
   • If crashes: Pod/container issue

2. Deploy pod with test2_cuda_check

   • Expected: Should detect CUDA
   • If crashes: CUDA runtime issue

3. Build train_tft_parquet ON RunPod

   • Clone repo inside pod
   • Build with pod's native CUDA 12.x
   • Test if it runs without crashing

If Tests 1-2 Work

✅ Confirms: Pod environment is fine ✅ Confirms: Problem is in ML binary 🎯 Next: Build train_tft_parquet on RunPod directly

If Tests 1-2 Crash

🔴 Confirms: Pod/container environment issue 🎯 Next: Debug base image, check glibc version, check kernel

---

Files Created

• runpod_debug/test1_hello.rs (3.6 MiB binary)
• runpod_debug/test2_cuda_check.rs (3.7 MiB binary)
• runpod_debug/upload_tests.sh (S3 upload script)
• runpod_debug/AGENT_05_MINIMAL_REPRODUCTION.md (this file)

---

Test Results (To Be Filled)

Test 1: Minimal Hello

• Run on RunPod: [ ] YES / [ ] NO
• Result: [ ] SUCCESS / [ ] CRASH
• Logs:

[Paste logs here]

Test 2: CUDA Check

• Run on RunPod: [ ] YES / [ ] NO
• Result: [ ] SUCCESS / [ ] CRASH
• Logs:

[Paste logs here]

Test 3: Build train_tft_parquet on RunPod

• Attempted: [ ] YES / [ ] NO
• Build Result: [ ] SUCCESS / [ ] FAILED
• Run Result: [ ] SUCCESS / [ ] CRASH
• Logs:

[Paste logs here]

---

Hypothesis

Primary Hypothesis: CUDA version mismatch (13.0 local vs 12.x RunPod) causes crash

Evidence:

1. cudarc explicitly rejects CUDA 13.0 during build
2. Local binary may link against CUDA 13.0 libs
3. RunPod GPUs likely use CUDA 12.x runtime
4. Binary built for CUDA 13.0 crashes on CUDA 12.x

Test: Build on RunPod with native CUDA → should work

Alternative Hypothesis: If building on RunPod still crashes, check:

• glibc version mismatch
• Missing system libraries
• Incorrect LD_LIBRARY_PATH
• Memory/GPU allocation failure

---

Time Invested

• Binary creation: 15 min
• Build attempts: 20 min
• S3 upload: 5 min
• Documentation: 20 min
• Total: 60 min

Next Agent

Agent 6: Execute deployment tests and analyze crash logs