foxhunt/scripts/RUNPOD_DEPLOYMENT_SCRIPTS.md

Runpod Deployment Scripts

This directory contains scripts for deploying Foxhunt ML models to Runpod GPU infrastructure.

Quick Start

# 1. Set environment variables
export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-1.runpod.io  # Your datacenter
export AWS_PROFILE=runpod
export DOCKER_USERNAME=jgrusewski

# 2. Configure AWS CLI for Runpod
aws configure --profile runpod
# AWS Access Key ID: <Your Runpod User ID>
# AWS Secret Access Key: <Your Runpod API Key>
# Default region name: us-east-1
# Default output format: json

# 3. Test prerequisites (dry run)
./scripts/runpod_deploy_test.sh

# 4. Deploy to Runpod
./scripts/runpod_deploy.sh

---

Scripts Overview

runpod_deploy.sh - Master Deployment Script

Purpose: Orchestrates the complete Runpod deployment workflow

What it does:

1. ✅ Validates prerequisites (cargo, docker, aws cli, credentials)
2. 🔨 Builds release binaries (5-6 minutes)
3. ☁️ Uploads binaries to Runpod S3 volume (~500 MB)
4. 📊 Uploads test data to Runpod S3 volume (~13 MB)
5. 🔐 Prompts for .env upload (optional, secure confirmation)
6. 🐳 Builds Docker image (~2-3 minutes)
7. 📤 Pushes to Docker Hub (3-5 minutes)
8. 📋 Prints deployment instructions

Duration: ~15-20 minutes total

Idempotent: Yes (safe to re-run, skips already-uploaded files)

Usage:

export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-1.runpod.io
export AWS_PROFILE=runpod
export DOCKER_USERNAME=jgrusewski
export S3_BUCKET=your-network-volume-id  # Optional (default: foxhunt-runpod)

./scripts/runpod_deploy.sh

Output:

• Uploaded binaries: s3://<bucket>/binaries/ (4 files, ~500 MB)
• Uploaded data: s3://<bucket>/test_data/ (9 files, ~13 MB)
• Docker image: jgrusewski/foxhunt:latest (~2 GB)
• Deployment instructions printed to console

---

runpod_deploy_test.sh - Dry Run Test

Purpose: Validates prerequisites without deploying

What it tests:

1. ✅ Cargo (Rust toolchain)
2. ✅ Docker (daemon running)
3. ✅ AWS CLI (for S3 uploads)
4. ✅ Environment variables (RUNPOD_S3_ENDPOINT, AWS_PROFILE, DOCKER_USERNAME)
5. ✅ AWS profile configuration
6. ✅ Test data files (9 parquet files)
7. ✅ Dockerfile.runpod exists
8. ✅ entrypoint.sh exists and is executable
9. ✅ S3 connectivity (optional)

Duration: ~5 seconds

Usage:

./scripts/runpod_deploy_test.sh

Exit codes:

• 0: All tests passed (ready for deployment)
• >0: Number of issues found (fix before deploying)

Example output:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Runpod Deployment Test - Dry Run
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Test 1: Cargo
✓ Cargo installed: 1.83.0

Test 2: Docker
✓ Docker running: 27.3.1

Test 3: AWS CLI
✓ AWS CLI installed: 2.15.10

Test 4: Environment Variables
✓ RUNPOD_S3_ENDPOINT: https://s3api-us-ca-1.runpod.io
✓ AWS_PROFILE: runpod
✓ DOCKER_USERNAME: jgrusewski

Test 5: AWS Profile Configuration
✓ AWS profile 'runpod' configured
✓ Access Key: abc12345***

Test 6: Test Data Files
✓ Found 9 parquet files in /home/user/foxhunt/test_data
    - ES_FUT_180d.parquet (2.90 MB)
    - NQ_FUT_180d.parquet (4.34 MB)
    - 6E_FUT_180d.parquet (2.74 MB)

Test 7: Dockerfile
✓ Dockerfile.runpod exists

Test 8: Entrypoint Script
✓ entrypoint.sh exists and is executable

Test 9: S3 Connectivity (Optional)
✓ S3 connectivity works

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Test Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ All tests passed! Ready for deployment.

Run deployment with:
  ./scripts/runpod_deploy.sh

---

Prerequisites

1. Rust Toolchain

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Verify
cargo --version  # Should show 1.70+

2. Docker

# Install Docker (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install docker.io
sudo systemctl start docker
sudo systemctl enable docker

# Add user to docker group (optional, avoids sudo)
sudo usermod -aG docker $USER
newgrp docker

# Verify
docker --version  # Should show 20.10+
docker info       # Should show running

3. AWS CLI

# Install AWS CLI
pip install awscli

# Or via apt (Ubuntu/Debian)
sudo apt-get install awscli

# Verify
aws --version  # Should show 2.0+

4. Runpod Account & Credentials

Create Runpod Account

1. Go to: https://www.runpod.io/console
2. Sign up (requires credit card)
3. Add credits ($10-20 recommended for testing)

Get Runpod Credentials

1. S3 Endpoint: Based on datacenter

   • US-CA-1: https://s3api-us-ca-1.runpod.io
   • EU-RO-1: https://s3api-eu-ro-1.runpod.io
   • Find yours: https://docs.runpod.io/storage/s3-api#endpoint-urls

2. User ID (AWS Access Key):

   • Go to: https://www.runpod.io/console/settings
   • Copy your User ID (shown at top)

3. API Key (AWS Secret Key):

   • Go to: https://www.runpod.io/console/settings
   • Click "API Keys" tab
   • Create new API key with "Read/Write" permissions
   • Copy secret key (only shown once!)

Configure AWS Profile

# Configure runpod profile
aws configure --profile runpod

# Enter credentials:
# AWS Access Key ID: <Your Runpod User ID>
# AWS Secret Access Key: <Your Runpod API Key>
# Default region name: us-east-1
# Default output format: json

# Verify
aws configure list --profile runpod

Create Network Volume

1. Go to: https://www.runpod.io/console/storage
2. Click "Create Network Volume"
3. Name: foxhunt-runpod
4. Size: 50 GB (recommended)
5. Datacenter: Same as S3 endpoint (e.g., US-CA-1)
6. Copy Network Volume ID (acts as S3 bucket name)

5. Docker Hub Account

# Create account: https://hub.docker.com/signup

# Login
docker login
# Username: jgrusewski
# Password: <your-docker-hub-password>

# Verify
docker info | grep Username  # Should show your username

6. Environment Variables

# Add to ~/.bashrc or ~/.zshrc
export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-1.runpod.io  # Your datacenter
export AWS_PROFILE=runpod
export DOCKER_USERNAME=jgrusewski
export S3_BUCKET=your-network-volume-id  # Optional

# Reload shell
source ~/.bashrc

---

Deployment Workflow

Step 1: Test Prerequisites

./scripts/runpod_deploy_test.sh

Expected output: All tests passed ✅

If tests fail: Follow error messages to install missing tools or configure credentials

Step 2: Run Deployment

./scripts/runpod_deploy.sh

Duration: ~15-20 minutes

What happens:

1. Validates prerequisites (~5 seconds)

   • Checks cargo, docker, aws cli, credentials
   • Fails fast if any prerequisite missing

2. Builds release binaries (~5-6 minutes)

   Compiling foxhunt workspace...
   ✓ train_tft_parquet (125.32 MB)
   ✓ train_mamba2_parquet (118.45 MB)
   ✓ train_dqn (89.67 MB)
   ✓ train_ppo (92.11 MB)
   Total binaries: 425.55 MB
   

3. Uploads binaries to Runpod S3 (~2-3 minutes)

   ▶ Uploading train_tft_parquet (125.32 MB)...
   ✓ train_tft_parquet uploaded
   [... 3 more binaries ...]
   ✓ Uploaded 4 binaries to Runpod S3
   

4. Uploads test data to Runpod S3 (~30 seconds)

   ▶ Uploading ES_FUT_180d.parquet (2.90 MB)...
   ✓ ES_FUT_180d.parquet uploaded
   [... 8 more files ...]
   ✓ Uploaded 9 data files (12.85 MB)
   

5. Prompts for .env upload (optional)

   ⚠ The .env file may contain sensitive credentials
   Upload .env? (y/N): n
   ✓ .env upload skipped (recommended)
   

6. Builds Docker image (~2-3 minutes)

   ▶ Building Docker image: jgrusewski/foxhunt:latest
   [... Docker build output ...]
   ✓ Docker image built in 2m 34s
   ✓ Image size: 1.98GB
   

7. Pushes to Docker Hub (~3-5 minutes)

   Is your Docker Hub repo PRIVATE? (y/N): y
   ▶ Pushing jgrusewski/foxhunt:latest to Docker Hub...
   [... Docker push output ...]
   ✓ Image pushed in 4m 12s
   

8. Prints deployment instructions

   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   READY FOR RUNPOD DEPLOYMENT
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   
   📦 Uploaded Resources:
     - Binaries: s3://foxhunt-runpod/binaries/ (425.55 MB)
     - Test Data: s3://foxhunt-runpod/test_data/ (12.85 MB)
     - Docker Image: jgrusewski/foxhunt:latest (1.98GB)
   
   🚀 Deploy on Runpod Console:
     [... detailed instructions ...]
   

Step 3: Deploy on Runpod Console

1. Go to Runpod: https://www.runpod.io/console/pods

2. Click "Deploy" and configure:

   GPU Configuration:

   • GPU Type: Tesla V100 16GB ($0.14-0.39/hr) or RTX 4090 24GB ($0.60/hr)
   • vCPU: 6-8 cores (recommended)
   • RAM: 30GB+ (recommended)
   • Container Disk: 20GB minimum

   Docker Configuration:

   • Docker Image: jgrusewski/foxhunt:latest
   • Docker Hub Credentials: Required (private repo)
     • Username: jgrusewski
     • Password: Your Docker Hub password

   Volume Configuration:

   • Volume Path: /runpod-volume
   • Network Volume: foxhunt-runpod (select from dropdown)
   • Access Mode: Read/Write

   Environment Variables:

   • BINARY_NAME=train_tft_parquet
   • RUST_LOG=info

   Container Arguments (override CMD):

   --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet
   --epochs 50
   --batch-size 32
   --lookback-window 60
   --forecast-horizon 10
   

3. Click "Deploy" and wait ~30 seconds for pod to start

4. Monitor logs in Runpod console:

   ==========================================
   Foxhunt HFT - Runpod Volume Mount Training
   ==========================================
   Configuration:
     Binary: train_tft_parquet
     Volume Path: /runpod-volume/
     Architecture: Direct volume mount (NO downloads)
   
   ==========================================
   Verifying Runpod Network Volume Mount
   ==========================================
   ✓ Volume mounted successfully at /runpod-volume/
   
   ==========================================
   Starting Training...
   ==========================================
   Epoch 1/50 [██████████] 100% | Loss: 0.0234
   Epoch 2/50 [██████████] 100% | Loss: 0.0189
   [... training continues ...]
   Epoch 50/50 [██████████] 100% | Loss: 0.0056
   ✓ Training complete! Model saved to /workspace/models/
   

5. Download trained models:

   # Via SSH
   scp root@<pod-ssh>:/workspace/models/*.pt ./models/
   
   # Via S3 (if models uploaded to volume)
   aws s3 sync s3://foxhunt-runpod/models/ ./models/ \
     --endpoint-url https://s3api-us-ca-1.runpod.io \
     --profile runpod
   

---

Troubleshooting

Issue: AWS CLI can't connect to Runpod S3

Symptoms:

An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation

Fix:

# Reconfigure AWS profile
aws configure --profile runpod

# Verify credentials
aws configure list --profile runpod

# Test connectivity
aws s3 ls --endpoint-url $RUNPOD_S3_ENDPOINT --profile runpod

---

Issue: Docker build fails

Symptoms:

ERROR: failed to solve: process "/bin/sh -c cargo build --release..." did not complete successfully

Fix:

# Clean build artifacts
cd /home/jgrusewski/Work/foxhunt
cargo clean

# Rebuild
cargo build --release --workspace --features cuda

# Retry Docker build
docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .

---

Issue: Runpod volume not mounting

Symptoms (in pod logs):

ERROR: /runpod-volume directory does not exist
Runpod Network Volume is NOT mounted!

Fix:

1. Stop pod
2. Edit pod configuration
3. Add volume mount:
   • Path: /runpod-volume
   • Network Volume: foxhunt-runpod (select from dropdown)
4. Redeploy pod

---

Issue: Training binary not found

Symptoms (in pod logs):

ERROR: Training binary not found: /runpod-volume/binaries/train_tft_parquet

Fix:

# Re-upload binaries
cd /home/jgrusewski/Work/foxhunt
cargo build --release --workspace --features cuda

# Upload to S3
aws s3 cp target/release/examples/train_tft_parquet \
  s3://foxhunt-runpod/binaries/train_tft_parquet \
  --endpoint-url $RUNPOD_S3_ENDPOINT \
  --profile runpod

# Verify upload
aws s3 ls s3://foxhunt-runpod/binaries/ \
  --endpoint-url $RUNPOD_S3_ENDPOINT \
  --profile runpod

---

Issue: Out of memory during training

Symptoms (in pod logs):

CUDA error: out of memory

Fix:

1. Use smaller batch size:

   --batch-size 16  # Instead of 32
   

2. Or upgrade GPU:

   • V100 16GB → RTX 4090 24GB
   • RTX 4090 24GB → A100 40GB

---

Cost Estimation

GPU Pricing (Runpod Community Cloud)

GPU	VRAM	Price/Hour	Full Training	100 Runs
Tesla V100 16GB	16GB	$0.14-0.39	$0.04-0.10	$4-10
RTX 4090 24GB	24GB	$0.60	$0.15	$15
A100 40GB	40GB	$1.50	$0.38	$38

Storage Pricing (Runpod Network Volume)

Resource	Size	Cost
Network Volume	50GB	$5/month
Binaries	~500 MB	Included
Test Data	~13 MB	Included

Total Cost (V100 16GB)

• Initial setup: $0 (one-time deployment)
• Single training run: $0.04-0.10 (~15 minutes)
• 100 training runs: $4-10 (hyperparameter tuning)
• Monthly storage: $5 (Network Volume)

Total first month: ~$15-25 (includes storage + 100 training runs)

---

Security Best Practices

1. Docker Hub Repository:

   • ✅ Set to PRIVATE (jgrusewski/foxhunt)
   • ❌ Never set to PUBLIC (contains proprietary code)
   • Verify: https://hub.docker.com/repository/docker/jgrusewski/foxhunt/settings

2. Runpod API Key:

   • ✅ Store in AWS CLI profile (~/.aws/credentials)
   • ❌ Never commit to git
   • ❌ Never share publicly
   • Rotate every 3-6 months

3. Environment Files:

   • ✅ Skip .env upload (contains Vault tokens, DB passwords)
   • ❌ Only upload if absolutely required
   • ❌ Never commit .env to git

4. Network Volume Access:

   • ✅ Restrict to your Runpod account only
   • ❌ Never share volume ID publicly
   • ❌ Never expose binaries/data externally

---

Performance Benchmarks

Training Time (Tesla V100 16GB)

Model	Epochs	Time	GPU Memory	Cost (@ $0.25/hr)
DQN	20	15-20s	6MB	$0.001
PPO	20	7-10s	145MB	$0.001
MAMBA-2	50	2-3min	164MB	$0.01
TFT-FP32	50	3-5min	500MB	$0.02
Total	-	10-15min	815MB peak	$0.06

Training Time (RTX 4090 24GB)

Model	Epochs	Time	GPU Memory	Cost (@ $0.60/hr)
DQN	20	10-15s	6MB	$0.003
PPO	20	5-7s	145MB	$0.002
MAMBA-2	50	1-2min	164MB	$0.02
TFT-FP32	50	2-3min	500MB	$0.03
Total	-	5-8min	815MB peak	$0.10

---

Next Steps

After successful deployment:

1. Validate models: Download trained models and test locally
2. Run backtests: Use backtesting_service to validate performance
3. Paper trading: Deploy to staging environment for live testing
4. Production: Deploy to production after paper trading validation

See:

• RUNPOD_DEPLOYMENT_CHECKLIST.md - Full deployment checklist
• WAVE_D_DEPLOYMENT_GUIDE.md - Wave D production deployment guide
• CLAUDE.md - System status and next priorities

---

Last Updated: 2025-10-24 Status: ✅ Production Ready (FP32 models only, QAT blocked by 3 P0 issues)