chore: Pre-cleanup commit - save current state before major reorganization

2025-10-30 00:54:01 +01:00
parent e61e8f54da
commit d73316da3d
87 changed files with 28355 additions and 1010 deletions
--- a/ml/python/README.md
+++ b/ml/python/README.md
@@ -0,0 +1,105 @@
+# Foxhunt RunPod Integration
+
+Python module for managing RunPod GPU pods with enhanced features:
+- Pod deployment with GPU selection
+- Real-time S3 log streaming
+- Auto-termination on completion
+- Health checks and status monitoring
+
+## Installation
+
+```bash
+cd ml/python
+pip install -r requirements.txt
+```
+
+## Usage
+
+### Command-line (via runpod_deploy.py)
+
+```bash
+# Basic deployment
+python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"
+
+# With monitoring
+python3 scripts/runpod_deploy.py --monitor --timeout 120m
+
+# Full automation
+python3 scripts/runpod_deploy.py --monitor --auto-stop --timeout 2h
+```
+
+### Python API
+
+```python
+from foxhunt_runpod import RunPodClient, S3LogMonitor
+
+# Deploy a pod
+client = RunPodClient(
+    api_key="your_api_key",
+    volume_id="your_volume_id"
+)
+
+gpus = client.get_available_gpus(min_vram=16)
+pod_data = client.deploy_pod(
+    gpu_id=gpus[0]['id'],
+    image="jgrusewski/foxhunt:latest",
+    command="--epochs 100"
+)
+
+# Monitor logs
+monitor = S3LogMonitor(
+    bucket_name="your_bucket",
+    aws_access_key="your_key",
+    aws_secret_key="your_secret"
+)
+
+monitor.stream_logs(
+    pod_id=pod_data['id'],
+    interval=10,
+    timeout=7200
+)
+
+# Terminate when done
+client.terminate_pod(pod_data['id'])
+```
+
+## Environment Variables
+
+Required in `.env.runpod`:
+- `RUNPOD_API_KEY` - RunPod API key
+- `RUNPOD_VOLUME_ID` - Network volume ID
+- `RUNPOD_CONTAINER_REGISTRY_AUTH_ID` - Docker registry auth (optional)
+
+Optional for S3 monitoring:
+- `RUNPOD_S3_BUCKET` - S3 bucket name
+- `RUNPOD_S3_ACCESS_KEY` - AWS access key
+- `RUNPOD_S3_SECRET_KEY` - AWS secret key
+
+## Architecture
+
+```
+scripts/runpod_deploy.py (CLI)
+    ↓
+ml/python/foxhunt_runpod/
+    ├── client.py       - RunPodClient (pod management)
+    ├── s3_monitor.py   - S3LogMonitor (log streaming)
+    └── __init__.py     - Module exports
+```
+
+## Features
+
+### RunPodClient
+- Query available GPUs with pricing
+- Deploy pods with datacenter filtering
+- Stop/terminate pods
+- Get pod status
+
+### S3LogMonitor
+- Stream logs in real-time from S3
+- Detect training completion
+- Configurable polling interval
+- Timeout support
+
+## Backward Compatibility
+
+The script automatically falls back to legacy implementation if the module cannot be imported. All existing command-line flags are preserved.