foxhunt/scripts/RUNPOD_DEPLOYMENT_SCRIPTS.md

# Runpod Deployment Scripts

This directory contains scripts for deploying Foxhunt ML models to Runpod GPU infrastructure.

## Quick Start

```bash
# 1. Set environment variables
export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-1.runpod.io  # Your datacenter
export AWS_PROFILE=runpod
export DOCKER_USERNAME=jgrusewski

# 2. Configure AWS CLI for Runpod
aws configure --profile runpod
# AWS Access Key ID: <Your Runpod User ID>
# AWS Secret Access Key: <Your Runpod API Key>
# Default region name: us-east-1
# Default output format: json

# 3. Test prerequisites (dry run)
./scripts/runpod_deploy_test.sh

# 4. Deploy to Runpod
./scripts/runpod_deploy.sh
```

---

## Scripts Overview

### `runpod_deploy.sh` - Master Deployment Script

**Purpose**: Orchestrates the complete Runpod deployment workflow

**What it does**:
1. ✅ Validates prerequisites (cargo, docker, aws cli, credentials)
2. 🔨 Builds release binaries (5-6 minutes)
3. ☁️ Uploads binaries to Runpod S3 volume (~500 MB)
4. 📊 Uploads test data to Runpod S3 volume (~13 MB)
5. 🔐 Prompts for .env upload (optional, secure confirmation)
6. 🐳 Builds Docker image (~2-3 minutes)
7. 📤 Pushes to Docker Hub (3-5 minutes)
8. 📋 Prints deployment instructions

**Duration**: ~15-20 minutes total

**Idempotent**: Yes (safe to re-run, skips already-uploaded files)

**Usage**:
```bash
export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-1.runpod.io
export AWS_PROFILE=runpod
export DOCKER_USERNAME=jgrusewski
export S3_BUCKET=your-network-volume-id  # Optional (default: foxhunt-runpod)

./scripts/runpod_deploy.sh
```

**Output**:
- Uploaded binaries: `s3://<bucket>/binaries/` (4 files, ~500 MB)
- Uploaded data: `s3://<bucket>/test_data/` (9 files, ~13 MB)
- Docker image: `jgrusewski/foxhunt:latest` (~2 GB)
- Deployment instructions printed to console

---

### `runpod_deploy_test.sh` - Dry Run Test

**Purpose**: Validates prerequisites without deploying

**What it tests**:
1. ✅ Cargo (Rust toolchain)
2. ✅ Docker (daemon running)
3. ✅ AWS CLI (for S3 uploads)
4. ✅ Environment variables (RUNPOD_S3_ENDPOINT, AWS_PROFILE, DOCKER_USERNAME)
5. ✅ AWS profile configuration
6. ✅ Test data files (9 parquet files)
7. ✅ Dockerfile.runpod exists
8. ✅ entrypoint.sh exists and is executable
9. ✅ S3 connectivity (optional)

**Duration**: ~5 seconds

**Usage**:
```bash
./scripts/runpod_deploy_test.sh
```

**Exit codes**:
- `0`: All tests passed (ready for deployment)
- `>0`: Number of issues found (fix before deploying)

**Example output**:
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Runpod Deployment Test - Dry Run
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Test 1: Cargo
✓ Cargo installed: 1.83.0

Test 2: Docker
✓ Docker running: 27.3.1

Test 3: AWS CLI
✓ AWS CLI installed: 2.15.10

Test 4: Environment Variables
✓ RUNPOD_S3_ENDPOINT: https://s3api-us-ca-1.runpod.io
✓ AWS_PROFILE: runpod
✓ DOCKER_USERNAME: jgrusewski

Test 5: AWS Profile Configuration
✓ AWS profile 'runpod' configured
✓ Access Key: abc12345***

Test 6: Test Data Files
✓ Found 9 parquet files in /home/user/foxhunt/test_data
    - ES_FUT_180d.parquet (2.90 MB)
    - NQ_FUT_180d.parquet (4.34 MB)
    - 6E_FUT_180d.parquet (2.74 MB)

Test 7: Dockerfile
✓ Dockerfile.runpod exists

Test 8: Entrypoint Script
✓ entrypoint.sh exists and is executable

Test 9: S3 Connectivity (Optional)
✓ S3 connectivity works

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Test Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ All tests passed! Ready for deployment.

Run deployment with:
  ./scripts/runpod_deploy.sh
```

---

## Prerequisites

### 1. Rust Toolchain
```bash
# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Verify
cargo --version  # Should show 1.70+
```

### 2. Docker
```bash
# Install Docker (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install docker.io
sudo systemctl start docker
sudo systemctl enable docker

# Add user to docker group (optional, avoids sudo)
sudo usermod -aG docker $USER
newgrp docker

# Verify
docker --version  # Should show 20.10+
docker info       # Should show running
```

### 3. AWS CLI
```bash
# Install AWS CLI
pip install awscli

# Or via apt (Ubuntu/Debian)
sudo apt-get install awscli

# Verify
aws --version  # Should show 2.0+
```

### 4. Runpod Account & Credentials

#### Create Runpod Account
1. Go to: https://www.runpod.io/console
2. Sign up (requires credit card)
3. Add credits ($10-20 recommended for testing)

#### Get Runpod Credentials
1. **S3 Endpoint**: Based on datacenter
   - US-CA-1: `https://s3api-us-ca-1.runpod.io`
   - EU-RO-1: `https://s3api-eu-ro-1.runpod.io`
   - Find yours: https://docs.runpod.io/storage/s3-api#endpoint-urls

2. **User ID** (AWS Access Key):
   - Go to: https://www.runpod.io/console/settings
   - Copy your User ID (shown at top)

3. **API Key** (AWS Secret Key):
   - Go to: https://www.runpod.io/console/settings
   - Click "API Keys" tab
   - Create new API key with "Read/Write" permissions
   - Copy secret key (only shown once!)

#### Configure AWS Profile
```bash
# Configure runpod profile
aws configure --profile runpod

# Enter credentials:
# AWS Access Key ID: <Your Runpod User ID>
# AWS Secret Access Key: <Your Runpod API Key>
# Default region name: us-east-1
# Default output format: json

# Verify
aws configure list --profile runpod
```

#### Create Network Volume
1. Go to: https://www.runpod.io/console/storage
2. Click "Create Network Volume"
3. Name: `foxhunt-runpod`
4. Size: 50 GB (recommended)
5. Datacenter: Same as S3 endpoint (e.g., US-CA-1)
6. Copy Network Volume ID (acts as S3 bucket name)

### 5. Docker Hub Account
```bash
# Create account: https://hub.docker.com/signup

# Login
docker login
# Username: jgrusewski
# Password: <your-docker-hub-password>

# Verify
docker info | grep Username  # Should show your username
```

### 6. Environment Variables
```bash
# Add to ~/.bashrc or ~/.zshrc
export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-1.runpod.io  # Your datacenter
export AWS_PROFILE=runpod
export DOCKER_USERNAME=jgrusewski
export S3_BUCKET=your-network-volume-id  # Optional

# Reload shell
source ~/.bashrc
```

---

## Deployment Workflow

### Step 1: Test Prerequisites
```bash
./scripts/runpod_deploy_test.sh
```

**Expected output**: All tests passed ✅

**If tests fail**: Follow error messages to install missing tools or configure credentials

### Step 2: Run Deployment
```bash
./scripts/runpod_deploy.sh
```

**Duration**: ~15-20 minutes

**What happens**:
1. **Validates prerequisites** (~5 seconds)
   - Checks cargo, docker, aws cli, credentials
   - Fails fast if any prerequisite missing

2. **Builds release binaries** (~5-6 minutes)
   ```
   Compiling foxhunt workspace...
   ✓ train_tft_parquet (125.32 MB)
   ✓ train_mamba2_parquet (118.45 MB)
   ✓ train_dqn (89.67 MB)
   ✓ train_ppo (92.11 MB)
   Total binaries: 425.55 MB
   ```

3. **Uploads binaries to Runpod S3** (~2-3 minutes)
   ```
   ▶ Uploading train_tft_parquet (125.32 MB)...
   ✓ train_tft_parquet uploaded
   [... 3 more binaries ...]
   ✓ Uploaded 4 binaries to Runpod S3
   ```

4. **Uploads test data to Runpod S3** (~30 seconds)
   ```
   ▶ Uploading ES_FUT_180d.parquet (2.90 MB)...
   ✓ ES_FUT_180d.parquet uploaded
   [... 8 more files ...]
   ✓ Uploaded 9 data files (12.85 MB)
   ```

5. **Prompts for .env upload** (optional)
   ```
   ⚠ The .env file may contain sensitive credentials
   Upload .env? (y/N): n
   ✓ .env upload skipped (recommended)
   ```

6. **Builds Docker image** (~2-3 minutes)
   ```
   ▶ Building Docker image: jgrusewski/foxhunt:latest
   [... Docker build output ...]
   ✓ Docker image built in 2m 34s
   ✓ Image size: 1.98GB
   ```

7. **Pushes to Docker Hub** (~3-5 minutes)
   ```
   Is your Docker Hub repo PRIVATE? (y/N): y
   ▶ Pushing jgrusewski/foxhunt:latest to Docker Hub...
   [... Docker push output ...]
   ✓ Image pushed in 4m 12s
   ```

8. **Prints deployment instructions**
   ```
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   READY FOR RUNPOD DEPLOYMENT
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   📦 Uploaded Resources:
     - Binaries: s3://foxhunt-runpod/binaries/ (425.55 MB)
     - Test Data: s3://foxhunt-runpod/test_data/ (12.85 MB)
     - Docker Image: jgrusewski/foxhunt:latest (1.98GB)

   🚀 Deploy on Runpod Console:
     [... detailed instructions ...]
   ```

### Step 3: Deploy on Runpod Console

1. **Go to Runpod**: https://www.runpod.io/console/pods

2. **Click "Deploy"** and configure:

   **GPU Configuration**:
   - GPU Type: `Tesla V100 16GB` ($0.14-0.39/hr) or `RTX 4090 24GB` ($0.60/hr)
   - vCPU: 6-8 cores (recommended)
   - RAM: 30GB+ (recommended)
   - Container Disk: 20GB minimum

   **Docker Configuration**:
   - Docker Image: `jgrusewski/foxhunt:latest`
   - Docker Hub Credentials: Required (private repo)
     - Username: `jgrusewski`
     - Password: Your Docker Hub password

   **Volume Configuration**:
   - Volume Path: `/runpod-volume`
   - Network Volume: `foxhunt-runpod` (select from dropdown)
   - Access Mode: Read/Write

   **Environment Variables**:
   - `BINARY_NAME=train_tft_parquet`
   - `RUST_LOG=info`

   **Container Arguments** (override CMD):
   ```
   --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet
   --epochs 50
   --batch-size 32
   --lookback-window 60
   --forecast-horizon 10
   ```

3. **Click "Deploy"** and wait ~30 seconds for pod to start

4. **Monitor logs** in Runpod console:
   ```
   ==========================================
   Foxhunt HFT - Runpod Volume Mount Training
   ==========================================
   Configuration:
     Binary: train_tft_parquet
     Volume Path: /runpod-volume/
     Architecture: Direct volume mount (NO downloads)

   ==========================================
   Verifying Runpod Network Volume Mount
   ==========================================
   ✓ Volume mounted successfully at /runpod-volume/

   ==========================================
   Starting Training...
   ==========================================
   Epoch 1/50 [██████████] 100% | Loss: 0.0234
   Epoch 2/50 [██████████] 100% | Loss: 0.0189
   [... training continues ...]
   Epoch 50/50 [██████████] 100% | Loss: 0.0056
   ✓ Training complete! Model saved to /workspace/models/
   ```

5. **Download trained models**:
   ```bash
   # Via SSH
   scp root@<pod-ssh>:/workspace/models/*.pt ./models/

   # Via S3 (if models uploaded to volume)
   aws s3 sync s3://foxhunt-runpod/models/ ./models/ \
     --endpoint-url https://s3api-us-ca-1.runpod.io \
     --profile runpod
   ```

---

## Troubleshooting

### Issue: AWS CLI can't connect to Runpod S3
**Symptoms**:
```
An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation
```

**Fix**:
```bash
# Reconfigure AWS profile
aws configure --profile runpod

# Verify credentials
aws configure list --profile runpod

# Test connectivity
aws s3 ls --endpoint-url $RUNPOD_S3_ENDPOINT --profile runpod
```

---

### Issue: Docker build fails
**Symptoms**:
```
ERROR: failed to solve: process "/bin/sh -c cargo build --release..." did not complete successfully
```

**Fix**:
```bash
# Clean build artifacts
cd /home/jgrusewski/Work/foxhunt
cargo clean

# Rebuild
cargo build --release --workspace --features cuda

# Retry Docker build
docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
```

---

### Issue: Runpod volume not mounting
**Symptoms** (in pod logs):
```
ERROR: /runpod-volume directory does not exist
Runpod Network Volume is NOT mounted!
```

**Fix**:
1. Stop pod
2. Edit pod configuration
3. Add volume mount:
   - Path: `/runpod-volume`
   - Network Volume: `foxhunt-runpod` (select from dropdown)
4. Redeploy pod

---

### Issue: Training binary not found
**Symptoms** (in pod logs):
```
ERROR: Training binary not found: /runpod-volume/binaries/train_tft_parquet
```

**Fix**:
```bash
# Re-upload binaries
cd /home/jgrusewski/Work/foxhunt
cargo build --release --workspace --features cuda

# Upload to S3
aws s3 cp target/release/examples/train_tft_parquet \
  s3://foxhunt-runpod/binaries/train_tft_parquet \
  --endpoint-url $RUNPOD_S3_ENDPOINT \
  --profile runpod

# Verify upload
aws s3 ls s3://foxhunt-runpod/binaries/ \
  --endpoint-url $RUNPOD_S3_ENDPOINT \
  --profile runpod
```

---

### Issue: Out of memory during training
**Symptoms** (in pod logs):
```
CUDA error: out of memory
```

**Fix**:
1. Use smaller batch size:
   ```
   --batch-size 16  # Instead of 32
   ```

2. Or upgrade GPU:
   - V100 16GB → RTX 4090 24GB
   - RTX 4090 24GB → A100 40GB

---

## Cost Estimation

### GPU Pricing (Runpod Community Cloud)
| GPU | VRAM | Price/Hour | Full Training | 100 Runs |
|---|---|---|---|---|
| Tesla V100 16GB | 16GB | $0.14-0.39 | $0.04-0.10 | $4-10 |
| RTX 4090 24GB | 24GB | $0.60 | $0.15 | $15 |
| A100 40GB | 40GB | $1.50 | $0.38 | $38 |

### Storage Pricing (Runpod Network Volume)
| Resource | Size | Cost |
|---|---|---|
| Network Volume | 50GB | $5/month |
| Binaries | ~500 MB | Included |
| Test Data | ~13 MB | Included |

### Total Cost (V100 16GB)
- **Initial setup**: $0 (one-time deployment)
- **Single training run**: $0.04-0.10 (~15 minutes)
- **100 training runs**: $4-10 (hyperparameter tuning)
- **Monthly storage**: $5 (Network Volume)

**Total first month**: ~$15-25 (includes storage + 100 training runs)

---

## Security Best Practices

1. **Docker Hub Repository**:
   - ✅ Set to PRIVATE (jgrusewski/foxhunt)
   - ❌ Never set to PUBLIC (contains proprietary code)
   - Verify: https://hub.docker.com/repository/docker/jgrusewski/foxhunt/settings

2. **Runpod API Key**:
   - ✅ Store in AWS CLI profile (`~/.aws/credentials`)
   - ❌ Never commit to git
   - ❌ Never share publicly
   - Rotate every 3-6 months

3. **Environment Files**:
   - ✅ Skip .env upload (contains Vault tokens, DB passwords)
   - ❌ Only upload if absolutely required
   - ❌ Never commit .env to git

4. **Network Volume Access**:
   - ✅ Restrict to your Runpod account only
   - ❌ Never share volume ID publicly
   - ❌ Never expose binaries/data externally

---

## Performance Benchmarks

### Training Time (Tesla V100 16GB)
| Model | Epochs | Time | GPU Memory | Cost (@ $0.25/hr) |
|---|---|---|---|---|
| DQN | 20 | 15-20s | 6MB | $0.001 |
| PPO | 20 | 7-10s | 145MB | $0.001 |
| MAMBA-2 | 50 | 2-3min | 164MB | $0.01 |
| TFT-FP32 | 50 | 3-5min | 500MB | $0.02 |
| **Total** | - | **10-15min** | **815MB peak** | **$0.06** |

### Training Time (RTX 4090 24GB)
| Model | Epochs | Time | GPU Memory | Cost (@ $0.60/hr) |
|---|---|---|---|---|
| DQN | 20 | 10-15s | 6MB | $0.003 |
| PPO | 20 | 5-7s | 145MB | $0.002 |
| MAMBA-2 | 50 | 1-2min | 164MB | $0.02 |
| TFT-FP32 | 50 | 2-3min | 500MB | $0.03 |
| **Total** | - | **5-8min** | **815MB peak** | **$0.10** |

---

## Next Steps

After successful deployment:

1. **Validate models**: Download trained models and test locally
2. **Run backtests**: Use `backtesting_service` to validate performance
3. **Paper trading**: Deploy to staging environment for live testing
4. **Production**: Deploy to production after paper trading validation

See:
- `RUNPOD_DEPLOYMENT_CHECKLIST.md` - Full deployment checklist
- `WAVE_D_DEPLOYMENT_GUIDE.md` - Wave D production deployment guide
- `CLAUDE.md` - System status and next priorities

---

**Last Updated**: 2025-10-24
**Status**: ✅ Production Ready (FP32 models only, QAT blocked by 3 P0 issues)