Foxhunt RunPod Integration

Python module for managing RunPod GPU pods with enhanced features:

Pod deployment with GPU selection
Real-time S3 log streaming
Auto-termination on completion
Health checks and status monitoring

Installation

cd ml/python
pip install -r requirements.txt

Usage

Command-line (via runpod_deploy.py)

# Basic deployment
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"

# With monitoring
python3 scripts/runpod_deploy.py --monitor --timeout 120m

# Full automation
python3 scripts/runpod_deploy.py --monitor --auto-stop --timeout 2h

Python API

from foxhunt_runpod import RunPodClient, S3LogMonitor

# Deploy a pod
client = RunPodClient(
    api_key="your_api_key",
    volume_id="your_volume_id"
)

gpus = client.get_available_gpus(min_vram=16)
pod_data = client.deploy_pod(
    gpu_id=gpus[0]['id'],
    image="jgrusewski/foxhunt:latest",
    command="--epochs 100"
)

# Monitor logs
monitor = S3LogMonitor(
    bucket_name="your_bucket",
    aws_access_key="your_key",
    aws_secret_key="your_secret"
)

monitor.stream_logs(
    pod_id=pod_data['id'],
    interval=10,
    timeout=7200
)

# Terminate when done
client.terminate_pod(pod_data['id'])

Environment Variables

Required in .env.runpod:

RUNPOD_API_KEY - RunPod API key
RUNPOD_VOLUME_ID - Network volume ID
RUNPOD_CONTAINER_REGISTRY_AUTH_ID - Docker registry auth (optional)

Optional for S3 monitoring:

RUNPOD_S3_BUCKET - S3 bucket name
RUNPOD_S3_ACCESS_KEY - AWS access key
RUNPOD_S3_SECRET_KEY - AWS secret key

Architecture

scripts/runpod_deploy.py (CLI)
    ↓
ml/python/foxhunt_runpod/
    ├── client.py       - RunPodClient (pod management)
    ├── s3_monitor.py   - S3LogMonitor (log streaming)
    └── __init__.py     - Module exports

Features

RunPodClient

Query available GPUs with pricing
Deploy pods with datacenter filtering
Stop/terminate pods
Get pod status

S3LogMonitor

Stream logs in real-time from S3
Detect training completion
Configurable polling interval
Timeout support

Backward Compatibility

The script automatically falls back to legacy implementation if the module cannot be imported. All existing command-line flags are preserved.