106 lines
2.3 KiB
Markdown
106 lines
2.3 KiB
Markdown
# Foxhunt RunPod Integration
|
|
|
|
Python module for managing RunPod GPU pods with enhanced features:
|
|
- Pod deployment with GPU selection
|
|
- Real-time S3 log streaming
|
|
- Auto-termination on completion
|
|
- Health checks and status monitoring
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
cd ml/python
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Command-line (via runpod_deploy.py)
|
|
|
|
```bash
|
|
# Basic deployment
|
|
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"
|
|
|
|
# With monitoring
|
|
python3 scripts/runpod_deploy.py --monitor --timeout 120m
|
|
|
|
# Full automation
|
|
python3 scripts/runpod_deploy.py --monitor --auto-stop --timeout 2h
|
|
```
|
|
|
|
### Python API
|
|
|
|
```python
|
|
from foxhunt_runpod import RunPodClient, S3LogMonitor
|
|
|
|
# Deploy a pod
|
|
client = RunPodClient(
|
|
api_key="your_api_key",
|
|
volume_id="your_volume_id"
|
|
)
|
|
|
|
gpus = client.get_available_gpus(min_vram=16)
|
|
pod_data = client.deploy_pod(
|
|
gpu_id=gpus[0]['id'],
|
|
image="jgrusewski/foxhunt:latest",
|
|
command="--epochs 100"
|
|
)
|
|
|
|
# Monitor logs
|
|
monitor = S3LogMonitor(
|
|
bucket_name="your_bucket",
|
|
aws_access_key="your_key",
|
|
aws_secret_key="your_secret"
|
|
)
|
|
|
|
monitor.stream_logs(
|
|
pod_id=pod_data['id'],
|
|
interval=10,
|
|
timeout=7200
|
|
)
|
|
|
|
# Terminate when done
|
|
client.terminate_pod(pod_data['id'])
|
|
```
|
|
|
|
## Environment Variables
|
|
|
|
Required in `.env.runpod`:
|
|
- `RUNPOD_API_KEY` - RunPod API key
|
|
- `RUNPOD_VOLUME_ID` - Network volume ID
|
|
- `RUNPOD_CONTAINER_REGISTRY_AUTH_ID` - Docker registry auth (optional)
|
|
|
|
Optional for S3 monitoring:
|
|
- `RUNPOD_S3_BUCKET` - S3 bucket name
|
|
- `RUNPOD_S3_ACCESS_KEY` - AWS access key
|
|
- `RUNPOD_S3_SECRET_KEY` - AWS secret key
|
|
|
|
## Architecture
|
|
|
|
```
|
|
scripts/runpod_deploy.py (CLI)
|
|
↓
|
|
ml/python/foxhunt_runpod/
|
|
├── client.py - RunPodClient (pod management)
|
|
├── s3_monitor.py - S3LogMonitor (log streaming)
|
|
└── __init__.py - Module exports
|
|
```
|
|
|
|
## Features
|
|
|
|
### RunPodClient
|
|
- Query available GPUs with pricing
|
|
- Deploy pods with datacenter filtering
|
|
- Stop/terminate pods
|
|
- Get pod status
|
|
|
|
### S3LogMonitor
|
|
- Stream logs in real-time from S3
|
|
- Detect training completion
|
|
- Configurable polling interval
|
|
- Timeout support
|
|
|
|
## Backward Compatibility
|
|
|
|
The script automatically falls back to legacy implementation if the module cannot be imported. All existing command-line flags are preserved.
|