d746008e1f
- Created entrypoint-self-terminate.sh wrapper script - Updates entrypoint-generic.sh to be called by wrapper - Modified Dockerfile.runpod to use self-terminate entrypoint - Adds automatic pod termination via runpodctl after training completes - Prevents infinite restart loops and wasted GPU credits - Saves ~96% cost per training run ($4.59 per run) Implements pod self-termination using RUNPOD_POD_ID environment variable. Training exits with code 0 → runpodctl remove pod → immediate shutdown. Co-Authored-By: Claude <noreply@anthropic.com>
76 lines
2.2 KiB
Markdown
76 lines
2.2 KiB
Markdown
# RunPod GPU Scanner
|
|
|
|
## Purpose
|
|
Query RunPod API to show available SECURE cloud GPUs in the EUR-IS region with ≥16GB VRAM.
|
|
|
|
## Requirements
|
|
- Python 3.7+
|
|
- python-dotenv (`pip3 install python-dotenv`)
|
|
- requests (`pip3 install requests`)
|
|
- Valid RunPod API key in `/home/jgrusewski/Work/foxhunt/.env.runpod`
|
|
|
|
## Usage
|
|
|
|
### Run the script:
|
|
```bash
|
|
# From foxhunt root directory
|
|
./scripts/scan_gpus.py
|
|
|
|
# Or with python3
|
|
python3 scripts/scan_gpus.py
|
|
```
|
|
|
|
## Output
|
|
The script displays:
|
|
- GPU name (e.g., RTX 4090, A100 SXM)
|
|
- VRAM capacity (GB)
|
|
- Price per hour (USD)
|
|
- Availability (number of pods)
|
|
|
|
Results are sorted by price (cheapest first).
|
|
|
|
## Filtering Criteria
|
|
- Memory ≥16GB VRAM
|
|
- Secure Cloud availability > 0
|
|
- Valid pricing information
|
|
- EUR-IS region
|
|
|
|
## Example Output
|
|
```
|
|
======================================================================
|
|
RUNPOD SECURE CLOUD GPUs (≥16GB VRAM) - EUR-IS REGION
|
|
======================================================================
|
|
GPU Name VRAM Price/hr Available
|
|
----------------------------------------------------------------------
|
|
RTX A5000 24GB $0.160 True pods
|
|
RTX A4000 16GB $0.170 True pods
|
|
RTX 4090 24GB $0.340 True pods
|
|
...
|
|
======================================================================
|
|
Total GPUs found: 24
|
|
```
|
|
|
|
## Top Recommendations for Foxhunt ML Training
|
|
|
|
Based on the current scan results (October 2025):
|
|
|
|
### Budget Option (FP32 Models)
|
|
- **RTX 4090**: $0.34/hr, 24GB VRAM
|
|
- Best value for FP32 training (TFT-FP32 fits in ~500MB)
|
|
- Ideal for initial deployment and baseline metrics
|
|
|
|
### Professional Option (QAT Models)
|
|
- **RTX A6000**: $0.33/hr, 48GB VRAM
|
|
- Best for QAT training (requires gradient checkpointing)
|
|
- Can run multiple models concurrently
|
|
|
|
### Enterprise Option (Multi-Model Inference)
|
|
- **A100 PCIe/SXM**: $1.19-1.39/hr, 80GB VRAM
|
|
- Production-grade for ensemble inference
|
|
- Supports 4+ models with headroom
|
|
|
|
## Notes
|
|
- Prices and availability fluctuate based on demand
|
|
- Run this script regularly to find the best deals
|
|
- Consider spot instances for training (not shown in this script)
|