Files

jgrusewski d746008e1f feat(runpod): Add self-termination wrapper for pod auto-shutdown

- Created entrypoint-self-terminate.sh wrapper script
- Updates entrypoint-generic.sh to be called by wrapper
- Modified Dockerfile.runpod to use self-terminate entrypoint
- Adds automatic pod termination via runpodctl after training completes
- Prevents infinite restart loops and wasted GPU credits
- Saves ~96% cost per training run ($4.59 per run)

Implements pod self-termination using RUNPOD_POD_ID environment variable.
Training exits with code 0 → runpodctl remove pod → immediate shutdown.

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-24 23:12:42 +02:00

2.7 KiB

Raw Blame History

🚀 Quick Start: RunPod Debugging

Goal: Identify why train_tft_parquet crashes on RunPod

Time: 15 minutes

Hypothesis: CUDA 13.0 (local) vs CUDA 12.x (RunPod) version mismatch

Step 1: Create RunPod Pod (2 min)

Go to https://runpod.io/console/pods
Click "Deploy"
Select:
- GPU: RTX 4090 (or RTX 3060)
- Template: runpod/pytorch:2.1.0-py3.10-cuda12.1.0-devel-ubuntu22.04
- Volume: Attach se3zdnb5o4
Click "Deploy"

Step 2: SSH into Pod (1 min)

ssh root@<pod-ip> -p <port> -i ~/.ssh/id_ed25519

Step 3: Run Test 1 (30 sec)

chmod +x /runpod-volume/debug_tests/test1_hello
/runpod-volume/debug_tests/test1_hello

Expected: "TEST 1 COMPLETE - EXITING CLEANLY"

If crashes: Report to Agent 6 (pod environment issue)

Step 4: Run Test 2 (30 sec)

chmod +x /runpod-volume/debug_tests/test2_cuda_check
/runpod-volume/debug_tests/test2_cuda_check

Expected: CUDA device detected, nvidia-smi output

If crashes: Report to Agent 6 (CUDA runtime issue)

Step 5: Build on RunPod (10 min)

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env

# Clone repo
cd /workspace
git clone https://github.com/your-user/foxhunt.git
cd foxhunt

# Build with RunPod's CUDA
cd ml
cargo build --release --example train_tft_parquet --features cuda

If build succeeds: Run training test:

../target/release/examples/train_tft_parquet \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --epochs 1 \
  --batch-size 32

If this works: ✅ CUDA mismatch confirmed! Update docs.

If this crashes: ⚠️ Different issue. Report logs to Agent 6.

What to Report

Test 1 Result: SUCCESS / CRASH
Test 2 Result: SUCCESS / CRASH
Build Result: SUCCESS / FAILED
Run Result: SUCCESS / CRASH
Full Logs: Copy all terminal output

Expected Outcome

If all steps succeed:

✅ Hypothesis confirmed (CUDA 13.0 vs 12.x mismatch)
🎯 Solution: Always build on RunPod or use CUDA 12.x Docker
📝 Update: RUNPOD_DEPLOYMENT_CHECKLIST.md with build requirement

If Step 1-2 succeed but Step 5 crashes:

🔍 Different issue (memory, GPU, data file, etc.)
🐛 Agent 6 will debug based on crash logs

Files Location

Test binaries: /runpod-volume/debug_tests/
Training data: /runpod-volume/test_data/ES_FUT_180d.parquet (if uploaded)
Repo: /workspace/foxhunt/ (created in Step 5)

Need More Details?

Technical analysis: AGENT_05_MINIMAL_REPRODUCTION.md
Full deployment guide: DEPLOY_TESTS.md
Summary: SUMMARY.md

2.7 KiB Raw Blame History