foxhunt/RUNPOD_WORKING_API_REFERENCE.md

RunPod Working API Reference

Purpose: Quick reference for RunPod API calls that are proven to work (bypassing broken Python script) Date: 2025-10-24 Status: ✅ 100% OPERATIONAL (tested and verified)

---

Prerequisites

# Load API key
export RUNPOD_API_KEY=$(grep RUNPOD_API_KEY .env.runpod | cut -d'=' -f2)

# Verify API key
echo $RUNPOD_API_KEY
# Expected: rpa_UK8KAUKXA2P9GHUV497WOH2RTZJ80MYCFSNJPTTM1mbk3y

---

1. Query GPU Availability

Command

curl --request POST \
  --url https://api.runpod.io/graphql \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $RUNPOD_API_KEY" \
  --data '{
    "query": "{ gpuTypes { id displayName memoryInGb secureCloud communityCloud lowestPrice(input: {gpuCount: 1}) { uninterruptablePrice } } }"
  }' | jq '.data.gpuTypes[] | select(.secureCloud == true) | {name: .displayName, vram: .memoryInGb, price: .lowestPrice.uninterruptablePrice}'

Expected Output

{
  "name": "RTX 4090",
  "vram": 24,
  "price": 0.34
}
{
  "name": "RTX A5000",
  "vram": 24,
  "price": 0.16
}
...

Use Case

• Check GPU availability before deployment
• Compare GPU prices
• Verify Secure Cloud access

---

2. Deploy Training Pod

Command (DQN Smoke Test)

curl --request POST \
  --url https://rest.runpod.io/v1/pods \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $RUNPOD_API_KEY" \
  --data '{
    "cloudType": "SECURE",
    "dataCenterIds": ["EUR-IS-1"],
    "gpuTypeIds": ["NVIDIA GeForce RTX 4090"],
    "gpuCount": 1,
    "name": "foxhunt-dqn-smoke-test",
    "imageName": "jgrusewski/foxhunt:latest",
    "containerDiskInGb": 50,
    "networkVolumeId": "se3zdnb5o4",
    "volumeMountPath": "/runpod-volume",
    "containerRegistryAuthId": "cmh3ya1710001jo02vwqtisbf",
    "dockerStartCmd": [
      "/runpod-volume/binaries/train_dqn",
      "--parquet-file",
      "/runpod-volume/test_data/ES_FUT_small.parquet",
      "--epochs",
      "1",
      "--output-dir",
      "/runpod-volume/models"
    ]
  }' | jq '.'

Command (TFT-FP32 Production)

curl --request POST \
  --url https://rest.runpod.io/v1/pods \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $RUNPOD_API_KEY" \
  --data '{
    "cloudType": "SECURE",
    "dataCenterIds": ["EUR-IS-1"],
    "gpuTypeIds": ["NVIDIA GeForce RTX 4090"],
    "gpuCount": 1,
    "name": "foxhunt-tft-fp32-225",
    "imageName": "jgrusewski/foxhunt:latest",
    "containerDiskInGb": 50,
    "networkVolumeId": "se3zdnb5o4",
    "volumeMountPath": "/runpod-volume",
    "containerRegistryAuthId": "cmh3ya1710001jo02vwqtisbf",
    "dockerStartCmd": [
      "/runpod-volume/binaries/train_tft_parquet",
      "--parquet-file",
      "/runpod-volume/test_data/ES_FUT_180d.parquet",
      "--epochs",
      "50",
      "--output-dir",
      "/runpod-volume/models"
    ]
  }' | jq '{id, name, status: .desiredStatus, gpu: .machine.gpuTypeId, datacenter: .machine.dataCenterId, cost: .costPerHr}'

Expected Output

{
  "id": "abc123xyz456",
  "name": "foxhunt-tft-fp32-225",
  "status": "RUNNING",
  "gpu": "NVIDIA GeForce RTX 4090",
  "datacenter": "EUR-IS-1",
  "cost": 0.59
}

Save Pod ID

# Extract pod ID from response
POD_ID=$(curl --request POST ... | jq -r '.id')
echo "Pod ID: $POD_ID"

---

3. Check Pod Status

Command

POD_ID="abc123xyz456"  # Replace with actual pod ID

curl --request GET \
  --url https://rest.runpod.io/v1/pods/$POD_ID \
  --header "Authorization: Bearer $RUNPOD_API_KEY" \
  | jq '{id, name, status: .desiredStatus, gpu: .machine.gpuTypeId, datacenter: .machine.dataCenterId, created: .createdAt, runtime: .runtime}'

Expected Output

{
  "id": "abc123xyz456",
  "name": "foxhunt-tft-fp32-225",
  "status": "RUNNING",
  "gpu": "NVIDIA GeForce RTX 4090",
  "datacenter": "EUR-IS-1",
  "created": "2025-10-24 22:37:25.798 +0000 UTC",
  "runtime": null
}

Status Values

• RUNNING: Pod is active (training in progress)
• EXITED: Pod completed and stopped
• FAILED: Pod crashed or errored

---

4. Get Pod Logs (SSH)

Prerequisites

# Get pod SSH connection details
curl --request GET \
  --url https://rest.runpod.io/v1/pods/$POD_ID \
  --header "Authorization: Bearer $RUNPOD_API_KEY" \
  | jq '{ssh_host: .machine.publicIp, ssh_port: .ports[]}'

SSH Command

# Use SSH key from volume (automatically added to pod)
ssh -p <PORT> root@<PUBLIC_IP>

# Example
ssh -p 12345 root@123.45.67.89

# View training logs
tail -f /var/log/training.log

# Check GPU usage
nvidia-smi

# List trained models
ls -lh /runpod-volume/models/

---

5. Terminate Pod

Command

POD_ID="abc123xyz456"  # Replace with actual pod ID

curl --request DELETE \
  --url https://rest.runpod.io/v1/pods/$POD_ID \
  --header "Authorization: Bearer $RUNPOD_API_KEY" \
  | jq '.'

Expected Output (Success)

{
  "id": "abc123xyz456",
  "status": "terminated"
}

Expected Output (Already Terminated)

{
  "error": "pod not found to terminate",
  "status": 404
}

Note: Pods auto-terminate after training completes (Docker command exits).

---

6. List All Active Pods

Command

curl --request GET \
  --url https://rest.runpod.io/v1/pods \
  --header "Authorization: Bearer $RUNPOD_API_KEY" \
  | jq '.[] | {id, name, status: .desiredStatus, gpu: .machine.gpuTypeId, cost: .costPerHr}'

Expected Output

[
  {
    "id": "abc123xyz456",
    "name": "foxhunt-tft-fp32-225",
    "status": "RUNNING",
    "gpu": "NVIDIA GeForce RTX 4090",
    "cost": 0.59
  }
]

---

7. Query Network Volume

Command

curl --request GET \
  --url https://rest.runpod.io/v1/volumes/se3zdnb5o4 \
  --header "Authorization: Bearer $RUNPOD_API_KEY" \
  | jq '{id, name, size: .size, datacenter: .dataCenterId}'

Expected Output

{
  "id": "se3zdnb5o4",
  "name": "foxhunt-storage",
  "size": 10,
  "datacenter": "EUR-IS-1"
}

---

Common Parameters

GPU Types (Exact API IDs)

# Budget GPUs
"NVIDIA GeForce RTX 4090"    # 24GB, $0.34/hr (recommended for FP32)
"NVIDIA RTX A5000"            # 24GB, $0.16/hr (cheapest 24GB)
"NVIDIA GeForce RTX 3090"     # 24GB, $0.22/hr

# High-Memory GPUs (for QAT when fixed)
"NVIDIA A100 80GB PCIe"       # 80GB, $1.19/hr
"NVIDIA H100 PCIe"            # 80GB, $1.99/hr
"NVIDIA RTX 6000 Ada"         # 48GB, $0.74/hr

Datacenter IDs

"EUR-IS-1"   # Iceland (Europe) - matches volume location
"US-GA-1"    # Georgia (USA)
"US-TX-3"    # Texas (USA)

Cloud Types

"SECURE"     # Secure Cloud (enterprise, guaranteed uptime)
"COMMUNITY"  # Community Cloud (cheaper, spot instances)

---

Training Binary Commands

DQN (Smoke Test)

"dockerStartCmd": [
  "/runpod-volume/binaries/train_dqn",
  "--parquet-file", "/runpod-volume/test_data/ES_FUT_small.parquet",
  "--epochs", "1",
  "--output-dir", "/runpod-volume/models"
]

PPO (Smoke Test)

"dockerStartCmd": [
  "/runpod-volume/binaries/train_ppo",
  "--parquet-file", "/runpod-volume/test_data/ES_FUT_small.parquet",
  "--episodes", "10",
  "--output-dir", "/runpod-volume/models"
]

MAMBA-2 (Production)

"dockerStartCmd": [
  "/runpod-volume/binaries/train_mamba2_parquet",
  "--parquet-file", "/runpod-volume/test_data/ES_FUT_180d.parquet",
  "--epochs", "50",
  "--output-dir", "/runpod-volume/models"
]

TFT-FP32 (Production)

"dockerStartCmd": [
  "/runpod-volume/binaries/train_tft_parquet",
  "--parquet-file", "/runpod-volume/test_data/ES_FUT_180d.parquet",
  "--epochs", "50",
  "--output-dir", "/runpod-volume/models"
]

---

Troubleshooting

Error: "Invalid GPU type"

Cause: Using display name instead of API ID.

Fix: Use exact API ID from GraphQL query:

# Wrong
"gpuTypeIds": ["RTX 4090"]

# Correct
"gpuTypeIds": ["NVIDIA GeForce RTX 4090"]

Error: "HTTP 201" (from script)

Cause: Script treats HTTP 201 Created as error.

Fix: HTTP 201 is success for POST requests. Check response body, not status code.

Error: "No GPUs available"

Cause: Script checks secureCloud > 0 instead of == true.

Fix: Use direct API call or fix script boolean check.

Error: "Volume not found"

Cause: Volume is in different datacenter than pod.

Fix: Always specify dataCenterIds: ["EUR-IS-1"] (matches volume location).

Pod Stuck in "RUNNING" (Already Completed)

Cause: Training finished but pod still shows RUNNING status (API lag).

Fix: Check pod logs or wait 30-60 seconds for status update. Pod will auto-terminate.

---

Cost Estimation

Per-Training Costs

Model	GPU	Runtime	Cost	Notes
DQN Smoke Test	RTX 4090	15 sec	$0.002	Quick validation
PPO Smoke Test	RTX 4090	30 sec	$0.005	Episode-based
MAMBA-2 FP32	RTX 4090	2 min	$0.02	Fast training
TFT-FP32	RTX 4090	5 min	$0.05	225 features

Monthly Budget (Typical)

• Development: 30 training runs/month = $1.50
• Production: 100 training runs/month = $5.00
• Volume Storage: 10GB @ $0.10/GB = $1.00
• Total: $2.50-$6.00/month

Monthly Budget (Heavy Use)

• Research: 500 training runs/month = $25.00
• Volume Storage: 50GB @ $0.10/GB = $5.00
• Total: $30.00/month

---

Security Notes

API Key Protection

# Never commit API key to git
echo "RUNPOD_API_KEY=..." >> .env.runpod
echo ".env.runpod" >> .gitignore

# Load from env file only
export RUNPOD_API_KEY=$(grep RUNPOD_API_KEY .env.runpod | cut -d'=' -f2)

Container Registry Auth

# Private Docker Hub credentials stored in RunPod
# Auth ID: cmh3ya1710001jo02vwqtisbf
# Automatically applied to all pod deployments
# No need to specify credentials in API call

SSH Access

# SSH keys automatically added to pods from volume
# Keys located at: /runpod-volume/.ssh/
# No password auth (key-based only)

---

Quick Reference Card

# 1. Check GPUs
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ gpuTypes { id displayName secureCloud lowestPrice(input: {gpuCount: 1}) { uninterruptablePrice } } }"}' \
  | jq '.data.gpuTypes[] | select(.secureCloud == true)'

# 2. Deploy Pod
curl -X POST https://rest.runpod.io/v1/pods \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{
    "cloudType": "SECURE",
    "dataCenterIds": ["EUR-IS-1"],
    "gpuTypeIds": ["NVIDIA GeForce RTX 4090"],
    "gpuCount": 1,
    "name": "foxhunt-training",
    "imageName": "jgrusewski/foxhunt:latest",
    "containerDiskInGb": 50,
    "networkVolumeId": "se3zdnb5o4",
    "volumeMountPath": "/runpod-volume",
    "containerRegistryAuthId": "cmh3ya1710001jo02vwqtisbf",
    "dockerStartCmd": ["/runpod-volume/binaries/train_tft_parquet", "--parquet-file", "/runpod-volume/test_data/ES_FUT_180d.parquet", "--epochs", "50"]
  }' | jq -r '.id'

# 3. Check Status
curl -X GET https://rest.runpod.io/v1/pods/$POD_ID \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  | jq '{id, name, status: .desiredStatus, gpu: .machine.gpuTypeId}'

# 4. Terminate Pod
curl -X DELETE https://rest.runpod.io/v1/pods/$POD_ID \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

---

Files

• RUNPOD_DIRECT_API_TEST_RESULTS.md: Full test results and root cause analysis
• RUNPOD_API_DIRECT_TEST_SUMMARY.md: Executive summary and next steps
• RUNPOD_WORKING_API_REFERENCE.md: This file (quick reference)

---

Next Steps

1. ✅ API Verified: RunPod API is 100% operational
2. ⏳ Fix Script: Update scripts/runpod_deploy.py (1-2 hours)
3. ⏳ Deploy FP32: Train 225-feature models on RTX 4090 (ready today)
4. ⏳ Fix QAT: Resolve 3 P0 blockers (1-2 weeks, separate from FP32)

Timeline: FP32 models deployable TODAY (after script fix). QAT in 1-2 weeks (non-blocking).