182 lines
4.5 KiB
Markdown
182 lines
4.5 KiB
Markdown
# RunPod Deployment Quick Reference
|
|
|
|
## Prerequisites
|
|
|
|
```bash
|
|
# 1. Activate virtual environment
|
|
source .venv/bin/activate
|
|
|
|
# 2. Install dependencies
|
|
pip install -r ml/python/requirements.txt
|
|
|
|
# 3. Configure credentials in .env.runpod
|
|
# Required:
|
|
# RUNPOD_API_KEY=<your_api_key>
|
|
# RUNPOD_VOLUME_ID=<your_volume_id>
|
|
# Optional (for monitoring):
|
|
# RUNPOD_S3_BUCKET=<bucket_name>
|
|
# RUNPOD_S3_ACCESS_KEY=<access_key>
|
|
# RUNPOD_S3_SECRET_KEY=<secret_key>
|
|
```
|
|
|
|
## Common Commands
|
|
|
|
### Basic Deployment
|
|
```bash
|
|
# Auto-select cheapest GPU
|
|
python3 scripts/runpod_deploy.py
|
|
|
|
# Specific GPU
|
|
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"
|
|
|
|
# Dry run (no deployment)
|
|
python3 scripts/runpod_deploy.py --dry-run
|
|
```
|
|
|
|
### With Monitoring
|
|
```bash
|
|
# Monitor training logs
|
|
python3 scripts/runpod_deploy.py --monitor --timeout 2h
|
|
|
|
# Full automation (deploy + monitor + auto-stop)
|
|
python3 scripts/runpod_deploy.py \
|
|
--gpu-type "RTX A4000" \
|
|
--monitor \
|
|
--auto-stop \
|
|
--timeout 2h
|
|
```
|
|
|
|
### Custom Training
|
|
```bash
|
|
# Custom epochs
|
|
python3 scripts/runpod_deploy.py \
|
|
--command "--parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 100"
|
|
|
|
# Different model
|
|
python3 scripts/runpod_deploy.py \
|
|
--command "/runpod-volume/binaries/train_mamba2_parquet --epochs 50"
|
|
```
|
|
|
|
## Flag Reference
|
|
|
|
| Flag | Description | Default |
|
|
|------|-------------|---------|
|
|
| `--gpu-type` | Preferred GPU (e.g., "RTX 4090") | Auto-select cheapest |
|
|
| `--image` | Docker image | jgrusewski/foxhunt:latest |
|
|
| `--command` | Training command | TFT with ES_FUT_180d.parquet |
|
|
| `--container-disk` | Disk size in GB | 50 |
|
|
| `--dry-run` | Show plan without deploying | False |
|
|
| `--monitor` | Stream logs from S3 | False |
|
|
| `--auto-stop` | Auto-terminate on completion | False |
|
|
| `--timeout` | Max monitoring time (30m, 2h) | 120m |
|
|
| `--monitor-interval` | Log polling interval (seconds) | 10 |
|
|
|
|
## Cost Estimates
|
|
|
|
| GPU | VRAM | Price/hr | 100 epochs* |
|
|
|-----|------|----------|-------------|
|
|
| RTX A4000 | 16GB | $0.25 | ~$0.50 |
|
|
| RTX A5000 | 24GB | $0.35 | ~$0.70 |
|
|
| Tesla V100 | 16GB | $0.10 | ~$0.20 |
|
|
| A100 80GB | 80GB | $1.20 | ~$2.40 |
|
|
|
|
*Estimated for TFT training (~2h)
|
|
|
|
## Troubleshooting
|
|
|
|
### Module Import Error
|
|
```
|
|
WARNING: Could not import foxhunt_runpod module
|
|
```
|
|
**Fix**: Install dependencies
|
|
```bash
|
|
pip install -r ml/python/requirements.txt
|
|
```
|
|
|
|
### S3 Monitoring Disabled
|
|
```
|
|
WARNING: S3 credentials not configured
|
|
```
|
|
**Fix**: Add to .env.runpod
|
|
```
|
|
RUNPOD_S3_BUCKET=se3zdnb5o4
|
|
RUNPOD_S3_ACCESS_KEY=<key>
|
|
RUNPOD_S3_SECRET_KEY=<secret>
|
|
```
|
|
|
|
### GPU Not Available
|
|
```
|
|
GPU not available in EUR-IS datacenters
|
|
```
|
|
**Fix**: Try again (availability changes frequently) or select different GPU
|
|
|
|
## Manual Pod Management
|
|
|
|
```bash
|
|
# View pods
|
|
curl -H "Authorization: Bearer $RUNPOD_API_KEY" \
|
|
https://rest.runpod.io/v1/pods
|
|
|
|
# Stop pod
|
|
curl -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" \
|
|
https://rest.runpod.io/v1/pods/<pod_id>/stop
|
|
|
|
# Terminate pod
|
|
curl -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" \
|
|
https://rest.runpod.io/v1/pods/<pod_id>/terminate
|
|
```
|
|
|
|
## Python API Usage
|
|
|
|
```python
|
|
from foxhunt_runpod import RunPodClient, S3LogMonitor
|
|
|
|
# Deploy
|
|
client = RunPodClient(api_key="...", volume_id="...")
|
|
gpus = client.get_available_gpus(min_vram=16)
|
|
pod = client.deploy_pod(gpu_id=gpus[0]['id'], image="jgrusewski/foxhunt:latest")
|
|
|
|
# Monitor
|
|
monitor = S3LogMonitor(bucket_name="...", aws_access_key="...", aws_secret_key="...")
|
|
monitor.stream_logs(pod_id=pod['id'], timeout=7200)
|
|
|
|
# Cleanup
|
|
client.terminate_pod(pod['id'])
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
1. **Always use --dry-run first** to verify configuration
|
|
2. **Set --timeout** to prevent runaway costs
|
|
3. **Use --monitor --auto-stop** for unattended training
|
|
4. **Check GPU pricing** before deploying (shown in output)
|
|
5. **Terminate pods** when done to avoid charges
|
|
|
|
## Example Workflow
|
|
|
|
```bash
|
|
# 1. Dry run to check configuration
|
|
python3 scripts/runpod_deploy.py --dry-run
|
|
|
|
# 2. Deploy with monitoring
|
|
python3 scripts/runpod_deploy.py \
|
|
--gpu-type "RTX A4000" \
|
|
--monitor \
|
|
--auto-stop \
|
|
--timeout 2h \
|
|
--command "--parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 100"
|
|
|
|
# 3. Script will:
|
|
# - Find available RTX A4000 in EUR-IS-1
|
|
# - Deploy pod with volume mounted
|
|
# - Stream logs in real-time
|
|
# - Auto-terminate when complete
|
|
# - Show final cost estimate
|
|
```
|
|
|
|
## Support
|
|
|
|
- **Module docs**: `ml/python/README.md`
|
|
- **Integration guide**: `RUNPOD_MODULE_INTEGRATION.md`
|
|
- **Script help**: `python3 scripts/runpod_deploy.py --help`
|