foxhunt/terraform/runpod

Foxhunt Runpod Deployment - OpenTofu/Terraform Configuration

Overview

This directory contains OpenTofu/Terraform configuration files for deploying the Foxhunt HFT Trading System on Runpod GPU infrastructure.

What This Deploys:

• 1x GPU Pod (Tesla V100 or RTX 4090)
• 1x Network Volume (persistent storage for market data, models, logs)
• Foxhunt Docker container with all 5 microservices
• PostgreSQL, Redis, Vault, Grafana, Prometheus
• SSH access and public IP (optional)

File Structure

terraform/runpod/
├── main.tf                    # Pod and volume resource definitions
├── variables.tf               # Input variables with validation
├── outputs.tf                 # Output values (IPs, endpoints, costs)
├── terraform.tfvars.example   # Template for credentials (COPY THIS)
├── README.md                  # This file
└── .terraform/                # Terraform state (auto-generated, gitignored)

Prerequisites

1. OpenTofu or Terraform Installed

   # Option 1: Install OpenTofu (recommended, open-source fork)
   curl --proto '=https' --tlsv1.2 -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh
   chmod +x install-opentofu.sh
   ./install-opentofu.sh --install-method standalone
   
   # Option 2: Install Terraform
   brew install terraform  # macOS
   # OR download from: https://www.terraform.io/downloads
   

2. Runpod Account & API Key

   • Sign up: https://www.runpod.io/
   • Get API key: https://www.runpod.io/console/user/settings
   • Fund account: $10 minimum (Tesla V100 = $0.44/hr)

3. Docker Hub Access

   • Ensure jgrusewski/foxhunt image is accessible (PRIVATE)
   • Add Docker Hub credentials in Runpod settings

4. Required Secrets (Stored in /runpod-volume/.env)

   • Databento API key (get from: https://databento.com/)
   • PostgreSQL password (generate strong random password)
   • JWT secret (generate with: openssl rand -base64 64)
   • Vault token (generate with: openssl rand -base64 32)

   IMPORTANT: Credentials are NO LONGER stored in Terraform state. They must be uploaded to the Runpod volume as a .env file BEFORE deploying pods.

Quick Start (5 Minutes)

Step 1: Copy Configuration Template

cd terraform/runpod
cp terraform.tfvars.example terraform.tfvars

Step 2: Create .env File for Credentials

Create a .env file with credentials (will be uploaded to Runpod volume):

# Create .env file with credentials
cat > .env.runpod <<EOF
POSTGRES_PASSWORD="$(openssl rand -base64 32)"
VAULT_TOKEN="$(openssl rand -base64 32)"
DATABENTO_API_KEY="YOUR_DATABENTO_API_KEY_HERE"
JWT_SECRET="$(openssl rand -base64 64)"
EOF

# Store securely (DO NOT commit to git)
chmod 600 .env.runpod

Step 3: Edit terraform.tfvars

Fill in required values:

# Runpod API key (from: https://www.runpod.io/console/user/settings)
runpod_api_key = "YOUR_RUNPOD_API_KEY_HERE"

# Optional: Change GPU type, data center, volume size
gpu_type       = "NVIDIA Tesla V100"  # or "NVIDIA GeForce RTX 4090"
data_center_id = "US-CA-1"            # or "US-TX-1", "EU-RO-1", "EU-SE-1"
volume_size_gb = 100                  # adjust based on data needs

# NOTE: Credentials are in .env.runpod file, NOT in Terraform state

Step 4: Initialize Terraform

# Initialize Terraform (downloads Runpod provider)
tofu init  # OR: terraform init

Step 5: Plan Deployment

# Preview changes (dry-run, no actual deployment)
tofu plan  # OR: terraform plan

# Expected output:
#   + runpod_network_volume.foxhunt_volume
#   + runpod_pod.foxhunt_trading_pod

Step 6: Deploy Volume (First Time Only)

# Deploy volume first (needed to upload .env file)
tofu apply -target=runpod_network_volume.foxhunt_volume

# Type 'yes' when prompted

Step 7: Upload .env File to Volume

# Option 1: Via Runpod Web Console
# 1. Go to: https://www.runpod.io/console/volumes
# 2. Click on 'foxhunt-data-volume'
# 3. Upload .env.runpod as '.env'

# Option 2: Via Runpod S3 API (requires volume ID from output)
VOLUME_ID=$(tofu output -raw volume_id)
# Upload via S3 API (see Runpod docs for S3 endpoint details)

Step 8: Deploy Pod

# Deploy pod (will load credentials from /runpod-volume/.env)
tofu apply

# Type 'yes' when prompted

Step 9: Verify Deployment

# Check deployment status
tofu output connection_summary

# Expected output:
#   ========================================
#   Foxhunt HFT Trading System - Runpod Deployment
#   ========================================
#   Pod ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
#   GPU: 1x NVIDIA Tesla V100
#   Public IP: XX.XX.XX.XX
#   SSH Command: ssh root@XX.XX.XX.XX -p 22
#   API Gateway: XX.XX.XX.XX:50051
#   Grafana: http://XX.XX.XX.XX:3000
#   ========================================

Step 10: Connect to Pod

# SSH into pod (get command from output)
ssh root@<PUBLIC_IP> -p 22

# Verify credentials loaded from .env file
echo $POSTGRES_PASSWORD  # Should print password
echo $VAULT_TOKEN        # Should print token

# Verify services are running
docker-compose ps

# Check logs
docker-compose logs -f api_gateway

Step 11: Access Grafana Dashboard

# Open in browser (from output)
open http://<PUBLIC_IP>:3000

# Login credentials:
#   Username: admin
#   Password: foxhunt123

Configuration Options

GPU Selection

GPU Type	VRAM	Est. Cost	Use Case
Tesla V100	16 GB	$0.44/hr	Production (recommended)
RTX 4090	24 GB	$0.69/hr	Large models / multi-model
RTX 3090	24 GB	$0.59/hr	Alternative high-memory
A100	40 GB	$1.89/hr	High-performance training

Foxhunt Requirements:

• FP32 Models: 815 MB VRAM (fits on all GPUs)
• INT8 Models: 440 MB VRAM (fits on all GPUs)
• Recommended: Tesla V100 (best cost/performance)

Cloud Type Selection

Cloud Type	Pricing	Availability	Use Case
SECURE	On-demand	Guaranteed	Production (live trading)
COMMUNITY	Spot (50-70% cheaper)	May be interrupted	Development, backtesting

Recommendation:

• Use SECURE for live trading (guaranteed uptime)
• Use COMMUNITY for development (save 50-70% on costs)

Data Center Selection

Region	ID	Latency	Use Case
US West	US-CA-1	Low to US exchanges	NYSE, NASDAQ trading
US Central	US-TX-1	Low to US exchanges	Alternative US region
EU East	EU-RO-1	Low to EU exchanges	Euronext, LSE trading
EU North	EU-SE-1	Low to EU exchanges	Alternative EU region

Recommendation:

• US-CA-1 for US markets (lowest latency to NYSE/NASDAQ)
• EU-RO-1 for European markets

Volume Size Guidelines

Size	Use Case
50 GB	Development / testing (minimal data)
100 GB	Production (90-180 days market data)
200 GB	Long-term storage (1+ year data)

Foxhunt Requirements:

• Docker images: ~5 GB
• PostgreSQL database: ~10 GB (90 days data)
• Market data (Parquet): ~500 MB per 90 days per symbol
• ML models: ~2 GB (all 5 models)
• Logs: ~1 GB/day (info level)

Recommendation: 100 GB for production

Outputs Reference

After deployment, Terraform provides these outputs:

# Get all outputs
tofu output

# Get specific output
tofu output pod_public_ip
tofu output api_gateway_endpoint
tofu output connection_summary

Key Outputs:

• pod_id: Runpod pod identifier
• pod_public_ip: Public IP for SSH and web access
• pod_ssh_command: Full SSH command with port
• api_gateway_endpoint: gRPC endpoint (XX.XX.XX.XX:50051)
• grafana_url: Monitoring dashboard URL
• volume_id: Persistent storage identifier
• estimated_hourly_cost_usd: Hourly cost estimate

Common Workflows

Check Pod Status

tofu refresh
tofu output pod_status

Update Configuration

# Edit terraform.tfvars (e.g., change GPU type)
vim terraform.tfvars

# Apply changes
tofu apply

Scale GPU Count

# Edit terraform.tfvars
gpu_count = 2  # increase from 1 to 2

# Apply changes
tofu apply

Increase Volume Size

# Edit terraform.tfvars
volume_size_gb = 200  # increase from 100 to 200

# Apply changes (volume will be resized, data preserved)
tofu apply

Destroy Deployment

# WARNING: This will DELETE the pod and volume (data loss!)
tofu destroy

# Type 'yes' when prompted

Preserve Volume, Destroy Pod Only

# Remove pod from state (keeps volume)
tofu state rm runpod_pod.foxhunt_trading_pod

# Destroy pod only
tofu destroy -target=runpod_pod.foxhunt_trading_pod

Cost Management

Estimated Costs

Tesla V100 (Recommended for Production):

• Hourly: $0.44
• Daily (24 hours): $10.56
• Monthly (30 days): $316.80

RTX 4090 (High-Performance):

• Hourly: $0.69
• Daily (24 hours): $16.56
• Monthly (30 days): $496.80

Network Volume:

• $0.10/GB/month (100 GB = $10/month)

Total Monthly Cost (Tesla V100):

• Pod: $316.80
• Volume: $10.00
• Total: $326.80/month (24/7 operation)

Cost Optimization Strategies

1. Use Spot Pricing (COMMUNITY) for Development

   cloud_type = "COMMUNITY"  # 50-70% cheaper than SECURE
   

2. Stop Pod When Not Trading

   tofu destroy  # stops billing immediately
   tofu apply    # restart when needed (volume data preserved)
   

3. Use Smaller Volume for Development

   volume_size_gb = 50  # minimum 10 GB
   

4. Share Volume Across Pods

   • Deploy pod for training → destroy
   • Deploy pod for backtesting → destroy
   • Deploy pod for live trading (keep running)
   • All pods share the same volume (data persists)

5. Monitor GPU Utilization

   • Check Grafana dashboard: http://<PUBLIC_IP>:3000
   • Aim for >70% GPU utilization
   • If <50%, consider downgrading GPU type

Troubleshooting

Issue: "Invalid API key"

Solution:

1. Verify API key at: https://www.runpod.io/console/user/settings
2. Ensure no extra spaces or newlines
3. Check API key has sufficient permissions

Issue: "GPU type not available"

Solution:

1. Check availability: https://www.runpod.io/console/gpu-cloud
2. Try different data_center_id (e.g., US-TX-1 instead of US-CA-1)
3. Try cloud_type = "COMMUNITY" (spot instances)
4. Try different gpu_type (e.g., RTX 3090 instead of V100)

Issue: "Volume size too small"

Solution:

volume_size_gb = 100  # increase from 50

Issue: "Docker pull failed"

Solution:

1. Verify Docker Hub credentials in Runpod settings
2. Ensure jgrusewski/foxhunt image is accessible
3. Check Docker Hub status: https://status.docker.com/

Issue: "Port already in use"

Solution:

1. Change pod_name to avoid conflicts
2. Destroy old pod: tofu destroy
3. Redeploy with new name: tofu apply

Issue: "SSH connection refused"

Solution:

1. Verify enable_public_ip = true in terraform.tfvars
2. Check firewall rules in Runpod console
3. Wait 2-3 minutes for pod initialization
4. Check SSH port: tofu output pod_ssh_command

Issue: "Services not starting"

Solution:

1. SSH into pod: ssh root@<PUBLIC_IP> -p 22
2. Check logs: docker-compose logs -f
3. Verify environment variables: docker-compose config
4. Restart services: docker-compose restart

Issue: "High costs / unexpected charges"

Solution:

1. Check running pods: https://www.runpod.io/console/pods
2. Destroy unused pods: tofu destroy
3. Use COMMUNITY cloud type for development
4. Set up billing alerts in Runpod console

Security Best Practices

Credential Management (NEW ARCHITECTURE)

Foxhunt uses a secure credential management architecture:

1. Credentials are NEVER stored in Terraform state

   • All secrets in /runpod-volume/.env file
   • Terraform only stores non-sensitive config (GPU type, ports, etc.)
   • No risk of credential exposure via Terraform state file leaks

2. How It Works:

   ┌─────────────────┐
   │  Terraform      │ ← Only contains: GPU type, ports, non-sensitive config
   │  State File     │
   └─────────────────┘
   
   ┌─────────────────┐
   │ Runpod Volume   │ ← Contains: POSTGRES_PASSWORD, VAULT_TOKEN,
   │ .env File       │              DATABENTO_API_KEY, JWT_SECRET
   └─────────────────┘
           │
           │ --env-file /runpod-volume/.env
           ▼
   ┌─────────────────┐
   │  Docker         │ ← Loads credentials at runtime
   │  Container      │
   └─────────────────┘
   

3. Benefits:

   • ✅ Credentials encrypted at rest on Runpod volume
   • ✅ Easy credential rotation without Terraform apply
   • ✅ No credentials in version control
   • ✅ No credentials in CI/CD logs
   • ✅ Reduced attack surface

Traditional Security Best Practices

1. Never Commit terraform.tfvars or .env.runpod

   • Both already in .gitignore
   • .env.runpod contains ALL credentials
   • terraform.tfvars contains Runpod API key

2. Use Strong Passwords

   # Generate strong password (32 characters)
   openssl rand -base64 32
   
   # Generate JWT secret (64 characters)
   openssl rand -base64 64
   

3. Rotate Credentials Regularly

   # Edit .env.runpod with new credentials
   vim .env.runpod
   
   # Upload to volume (via Runpod console or S3 API)
   # Restart pod to reload credentials
   tofu apply  # or: docker-compose restart (if already running)
   

   Rotation Schedule:

   • JWT secret: every 90 days
   • Postgres password: every 90 days
   • Vault token: every 180 days
   • Databento API key: per Databento policy

4. Enable Audit Logging

   • Vault audit logs in /runpod-volume/vault/audit
   • PostgreSQL logs in /runpod-volume/postgres/logs
   • Service logs in Grafana

5. Use SECURE Cloud Type for Production

   cloud_type = "SECURE"  # guaranteed uptime
   

6. Restrict SSH Access

   ssh_public_key = "ssh-rsa AAAAB3Nza..."  # only your key
   

7. Monitor for Suspicious Activity

   • Check Grafana dashboards daily
   • Set up Prometheus alerts
   • Review Vault audit logs weekly

8. Backup .env.runpod Securely

   # Store in 1Password, LastPass, or AWS Secrets Manager
   # DO NOT store in:
   #   - Git repository
   #   - Slack/Discord/email
   #   - Unencrypted cloud storage
   #   - Screenshots or documents
   

Advanced Features

Serverless Endpoint (Optional)

If you want to expose the API via HTTP endpoint:

enable_endpoint = true
max_workers     = 3

Custom Start Script

Override default startup behavior:

start_script = <<-EOT
  #!/bin/bash
  cd /app
  docker-compose up -d
  ./scripts/wait_for_services.sh
  cargo run -p ml --example train_tft_parquet --release --features cuda -- \
    --parquet-file /runpod-volume/data/ES_FUT_180d.parquet --epochs 50
EOT

Multiple Pods with Shared Volume

Deploy multiple pods sharing the same volume:

# File: main_multi.tf
resource "runpod_pod" "foxhunt_training_pod" {
  name              = "foxhunt-training"
  network_volume_id = runpod_network_volume.foxhunt_volume.id
  # ... other config
}

resource "runpod_pod" "foxhunt_trading_pod" {
  name              = "foxhunt-trading"
  network_volume_id = runpod_network_volume.foxhunt_volume.id
  # ... other config
}

Integration with CI/CD

GitHub Actions Example

name: Deploy to Runpod
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: hashicorp/setup-terraform@v2

      - name: Terraform Init
        working-directory: terraform/runpod
        run: terraform init

      - name: Terraform Apply
        working-directory: terraform/runpod
        env:
          TF_VAR_runpod_api_key: ${{ secrets.RUNPOD_API_KEY }}
          TF_VAR_postgres_password: ${{ secrets.POSTGRES_PASSWORD }}
        run: terraform apply -auto-approve

Support

• Foxhunt Documentation: /home/jgrusewski/Work/foxhunt/docs/
• Runpod Documentation: https://docs.runpod.io/
• Terraform Runpod Provider: https://registry.terraform.io/providers/runpod/runpod/latest/docs
• Foxhunt CLAUDE.md: /home/jgrusewski/Work/foxhunt/CLAUDE.md
• Runpod Deployment Guide: /home/jgrusewski/Work/foxhunt/RUNPOD_DEPLOYMENT_GUIDE.md

Next Steps

After successful deployment:

1. Verify Services

   ssh root@<PUBLIC_IP> -p 22
   docker-compose ps
   

2. Access Grafana

   • URL: http://<PUBLIC_IP>:3000
   • Credentials: admin / foxhunt123

3. Train Models

   cargo run -p ml --example train_tft_parquet --release --features cuda -- \
     --parquet-file /runpod-volume/data/ES_FUT_180d.parquet --epochs 50
   

4. Start Trading

   # Configure TLI client
   tli config set gateway-url <PUBLIC_IP>:50051
   
   # Submit trade
   tli trade ml submit --symbol ES.FUT --action BUY --quantity 10
   

5. Monitor Performance

   • Check Grafana dashboards
   • Review Prometheus metrics
   • Monitor GPU utilization
   • Track cost vs. performance

---

Deployment Status: ✅ Ready to deploy (FP32 models validated, 0 blockers)

Estimated Deployment Time: 5-10 minutes (manual setup) + 2-5 minutes (pod provisioning)

Estimated Monthly Cost: $326.80 (Tesla V100 + 100 GB volume, 24/7 operation)