# RunPod Module Integration - Complete

## Summary

Successfully integrated the `foxhunt_runpod` Python module into `scripts/runpod_deploy.py` with full backward compatibility and new features.

## Changes Made

### 1. Created foxhunt_runpod Module

**Location**: `/home/jgrusewski/Work/foxhunt/ml/python/foxhunt_runpod/`

**Files**:
- `__init__.py` - Module exports
- `client.py` - RunPodClient class (pod management)
- `s3_monitor.py` - S3LogMonitor class (log streaming)
- `config.py` - Configuration management (existing)
- `errors.py` - Error classes (existing)
- `s3_client.py` - Enhanced S3 operations (existing)

**Key Classes**:

#### RunPodClient
```python
from foxhunt_runpod import RunPodClient

client = RunPodClient(
    api_key="your_api_key",
    volume_id="your_volume_id",
    registry_auth_id="optional_auth_id"
)

# Query GPUs
gpus = client.get_available_gpus(min_vram=16)

# Deploy pod
pod_data = client.deploy_pod(
    gpu_id=gpus[0]['id'],
    image="jgrusewski/foxhunt:latest",
    command="--epochs 100",
    container_disk=50
)

# Manage pods
client.stop_pod(pod_id)
client.terminate_pod(pod_id)
status = client.get_pod_status(pod_id)
```

#### S3LogMonitor
```python
from foxhunt_runpod import S3LogMonitor

monitor = S3LogMonitor(
    bucket_name="your_bucket",
    aws_access_key="your_key",
    aws_secret_key="your_secret"
)

# Stream logs
monitor.stream_logs(
    pod_id="pod_id",
    interval=10,
    timeout=7200,
    completion_callback=monitor.check_completion
)
```

### 2. Refactored runpod_deploy.py

**New Features**:

1. **Module Integration**:
   - Automatic import of `foxhunt_runpod` module
   - Graceful fallback to legacy implementation if import fails
   - .venv activation check with warning

2. **S3 Log Monitoring** (`--monitor`):
   - Streams training logs from RunPod S3 in real-time
   - Configurable polling interval (default: 10s)
   - Configurable timeout (e.g., `30m`, `2h`)

3. **Auto-Termination** (`--auto-stop`):
   - Automatically terminates pod when training completes
   - Requires `--monitor` flag
   - Detects completion patterns in logs
   - Estimates final cost

4. **Enhanced Error Messages**:
   - Better guidance for missing dependencies
   - Clear warnings for misconfiguration

**New Command-Line Flags**:
```bash
--monitor              # Enable S3 log monitoring
--auto-stop            # Auto-terminate on completion
--timeout 120m         # Monitoring timeout (30m, 2h, etc.)
--monitor-interval 10  # Polling interval in seconds
```

**Example Usage**:
```bash
# Basic deployment (unchanged)
python3 scripts/runpod_deploy.py

# With monitoring
python3 scripts/runpod_deploy.py --monitor --timeout 120m

# Full automation
python3 scripts/runpod_deploy.py \
  --gpu-type "RTX A4000" \
  --monitor \
  --auto-stop \
  --timeout 2h
```

### 3. Backward Compatibility

**Preserved Functionality**:
- All existing flags work unchanged
- Dry-run mode preserved
- GPU fallback logic preserved
- Legacy implementation available as fallback

**Dual Implementation Strategy**:
- `get_available_gpu_types()` - Wrapper that uses new or legacy
- `deploy_pod_rest_api()` - Wrapper that uses new or legacy
- Functions suffixed with `_new()` use RunPodClient
- Functions suffixed with `_legacy()` use original requests code

### 4. Documentation

**Created**:
- `/home/jgrusewski/Work/foxhunt/ml/python/README.md` - Module documentation
- `/home/jgrusewski/Work/foxhunt/ml/python/requirements.txt` - Dependencies
- `/home/jgrusewski/Work/foxhunt/RUNPOD_MODULE_INTEGRATION.md` - This file

**Updated**:
- `scripts/runpod_deploy.py` docstring - New features
- Help text (--help) - New examples and requirements

## Dependencies

**Required** (ml/python/requirements.txt):
```
boto3>=1.26.0        # S3 log monitoring
python-dotenv>=0.19.0  # .env.runpod file loading
requests>=2.28.0     # RunPod REST API
```

**Optional** (for enhanced features):
- `pydantic>=2.0.0` - Config validation (existing module)
- `rich>=13.0.0` - Progress bars (existing module)

## Environment Variables

**Required** (.env.runpod):
- `RUNPOD_API_KEY` - RunPod API key
- `RUNPOD_VOLUME_ID` - Network volume ID

**Optional** (for monitoring):
- `RUNPOD_S3_BUCKET` - S3 bucket name
- `RUNPOD_S3_ACCESS_KEY` - AWS access key
- `RUNPOD_S3_SECRET_KEY` - AWS secret key
- `RUNPOD_CONTAINER_REGISTRY_AUTH_ID` - Docker registry auth

## Testing

**Verified**:
1. Script help output works (--help)
2. Module imports correctly
3. Fallback to legacy works when module unavailable
4. All flags accepted by argparse
5. .venv warning displays correctly

**Not Tested** (requires RunPod credentials):
- Actual pod deployment
- S3 log monitoring
- Auto-termination

## Architecture

```
scripts/runpod_deploy.py (CLI)
    ↓
    ├─ USE_NEW_MODULE = True
    │   ↓
    │   ml/python/foxhunt_runpod/
    │       ├── client.py (RunPodClient)
    │       └── s3_monitor.py (S3LogMonitor)
    │
    └─ USE_NEW_MODULE = False
        ↓
        Legacy implementation (requests + GraphQL)
```

## Next Steps

1. **Test with real deployment**:
   ```bash
   python3 scripts/runpod_deploy.py --dry-run
   ```

2. **Install dependencies** (if needed):
   ```bash
   pip install -r ml/python/requirements.txt
   ```

3. **Configure S3 credentials** (for monitoring):
   - Add RUNPOD_S3_* variables to .env.runpod

4. **Full automation test**:
   ```bash
   python3 scripts/runpod_deploy.py \
     --gpu-type "RTX A4000" \
     --monitor \
     --auto-stop \
     --timeout 30m
   ```

## Benefits

1. **Cleaner Code**: Moved complexity to reusable module
2. **Better Testing**: Can unit test RunPodClient separately
3. **Enhanced Features**: Log monitoring + auto-termination
4. **Backward Compatible**: Existing workflows unchanged
5. **Maintainability**: Single source of truth for RunPod API
6. **Extensibility**: Easy to add new features to client

## Files Modified

- `scripts/runpod_deploy.py` (refactored, ~640 lines)

## Files Created

- `ml/python/foxhunt_runpod/__init__.py` (16 lines)
- `ml/python/foxhunt_runpod/client.py` (315 lines)
- `ml/python/foxhunt_runpod/s3_monitor.py` (179 lines, reused existing)
- `ml/python/README.md` (documentation)
- `ml/python/requirements.txt` (dependencies)
- `RUNPOD_MODULE_INTEGRATION.md` (this file)

## Total Lines of Code

- Module: ~510 lines
- Script refactor: ~640 lines
- Documentation: ~200 lines
- **Total**: ~1,350 lines