d746008e1f
- Created entrypoint-self-terminate.sh wrapper script - Updates entrypoint-generic.sh to be called by wrapper - Modified Dockerfile.runpod to use self-terminate entrypoint - Adds automatic pod termination via runpodctl after training completes - Prevents infinite restart loops and wasted GPU credits - Saves ~96% cost per training run ($4.59 per run) Implements pod self-termination using RUNPOD_POD_ID environment variable. Training exits with code 0 → runpodctl remove pod → immediate shutdown. Co-Authored-By: Claude <noreply@anthropic.com>
124 lines
2.7 KiB
Markdown
124 lines
2.7 KiB
Markdown
# 🚀 Quick Start: RunPod Debugging
|
|
|
|
**Goal**: Identify why `train_tft_parquet` crashes on RunPod
|
|
|
|
**Time**: 15 minutes
|
|
|
|
**Hypothesis**: CUDA 13.0 (local) vs CUDA 12.x (RunPod) version mismatch
|
|
|
|
---
|
|
|
|
## Step 1: Create RunPod Pod (2 min)
|
|
|
|
1. Go to https://runpod.io/console/pods
|
|
2. Click "Deploy"
|
|
3. Select:
|
|
- GPU: RTX 4090 (or RTX 3060)
|
|
- Template: `runpod/pytorch:2.1.0-py3.10-cuda12.1.0-devel-ubuntu22.04`
|
|
- Volume: Attach `se3zdnb5o4`
|
|
4. Click "Deploy"
|
|
|
|
---
|
|
|
|
## Step 2: SSH into Pod (1 min)
|
|
|
|
```bash
|
|
ssh root@<pod-ip> -p <port> -i ~/.ssh/id_ed25519
|
|
```
|
|
|
|
---
|
|
|
|
## Step 3: Run Test 1 (30 sec)
|
|
|
|
```bash
|
|
chmod +x /runpod-volume/debug_tests/test1_hello
|
|
/runpod-volume/debug_tests/test1_hello
|
|
```
|
|
|
|
**Expected**: "TEST 1 COMPLETE - EXITING CLEANLY"
|
|
|
|
**If crashes**: Report to Agent 6 (pod environment issue)
|
|
|
|
---
|
|
|
|
## Step 4: Run Test 2 (30 sec)
|
|
|
|
```bash
|
|
chmod +x /runpod-volume/debug_tests/test2_cuda_check
|
|
/runpod-volume/debug_tests/test2_cuda_check
|
|
```
|
|
|
|
**Expected**: CUDA device detected, nvidia-smi output
|
|
|
|
**If crashes**: Report to Agent 6 (CUDA runtime issue)
|
|
|
|
---
|
|
|
|
## Step 5: Build on RunPod (10 min)
|
|
|
|
```bash
|
|
# Install Rust
|
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
|
source $HOME/.cargo/env
|
|
|
|
# Clone repo
|
|
cd /workspace
|
|
git clone https://github.com/your-user/foxhunt.git
|
|
cd foxhunt
|
|
|
|
# Build with RunPod's CUDA
|
|
cd ml
|
|
cargo build --release --example train_tft_parquet --features cuda
|
|
```
|
|
|
|
**If build succeeds**: Run training test:
|
|
```bash
|
|
../target/release/examples/train_tft_parquet \
|
|
--parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
|
|
--epochs 1 \
|
|
--batch-size 32
|
|
```
|
|
|
|
**If this works**: ✅ CUDA mismatch confirmed! Update docs.
|
|
|
|
**If this crashes**: ⚠️ Different issue. Report logs to Agent 6.
|
|
|
|
---
|
|
|
|
## What to Report
|
|
|
|
1. **Test 1 Result**: SUCCESS / CRASH
|
|
2. **Test 2 Result**: SUCCESS / CRASH
|
|
3. **Build Result**: SUCCESS / FAILED
|
|
4. **Run Result**: SUCCESS / CRASH
|
|
5. **Full Logs**: Copy all terminal output
|
|
|
|
---
|
|
|
|
## Expected Outcome
|
|
|
|
**If all steps succeed**:
|
|
- ✅ Hypothesis confirmed (CUDA 13.0 vs 12.x mismatch)
|
|
- 🎯 Solution: Always build on RunPod or use CUDA 12.x Docker
|
|
- 📝 Update: `RUNPOD_DEPLOYMENT_CHECKLIST.md` with build requirement
|
|
|
|
**If Step 1-2 succeed but Step 5 crashes**:
|
|
- 🔍 Different issue (memory, GPU, data file, etc.)
|
|
- 🐛 Agent 6 will debug based on crash logs
|
|
|
|
---
|
|
|
|
## Files Location
|
|
|
|
- Test binaries: `/runpod-volume/debug_tests/`
|
|
- Training data: `/runpod-volume/test_data/ES_FUT_180d.parquet` (if uploaded)
|
|
- Repo: `/workspace/foxhunt/` (created in Step 5)
|
|
|
|
---
|
|
|
|
## Need More Details?
|
|
|
|
- Technical analysis: `AGENT_05_MINIMAL_REPRODUCTION.md`
|
|
- Full deployment guide: `DEPLOY_TESTS.md`
|
|
- Summary: `SUMMARY.md`
|