foxhunt/foxhunt-deploy/DEPLOYMENT_QUICK_START.md

RunPod Deployment Quick Start Guide

Prerequisites

1. Initialize Configuration:

foxhunt-deploy init

2. Edit Configuration: Edit ~/.runpod/config.toml and add your RunPod API key:

[runpod]
api_key = "YOUR_RUNPOD_API_KEY_HERE"

Common Deployment Scenarios

1. PPO Training (Production)

Deploy PPO training with dual learning rates:

foxhunt-deploy deploy \
  --command "train_ppo_parquet --policy-lr 1e-6 --value-lr 0.001 --epochs 200" \
  --gpu-type "RTX A4000" \
  --name "ppo-production"

Expected Cost: $0.25/hr (~$0.50 for 2 hours)

2. DQN Hyperparameter Optimization

Run hyperopt for DQN with 50 trials:

foxhunt-deploy deploy \
  --command "hyperopt_dqn_demo --n-trials 50" \
  --gpu-type "RTX A4000" \
  --name "dqn-hyperopt"

Expected Cost: $0.25/hr (~$0.13 for 30 minutes)

3. MAMBA-2 Training (High Memory)

Train MAMBA-2 with RTX 4090 for larger batch sizes:

foxhunt-deploy deploy \
  --command "train_mamba2_dbn --batch-size 256" \
  --gpu-type "RTX 4090" \
  --name "mamba2-batch256"

Expected Cost: $0.59/hr (~$0.30 for 30 minutes)

4. TFT Training (Fast Cache)

Train TFT with cache optimization:

foxhunt-deploy deploy \
  --command "train_tft_parquet --parquet-file /runpod-volume/data/ES_FUT_180d.parquet --epochs 50" \
  --gpu-type "RTX A4000" \
  --name "tft-fp32"

Expected Cost: $0.25/hr (~$0.02 for 5 minutes)

5. Multi-Model Hyperopt (Long Running)

Run hyperopt for multiple models (TFT + PPO):

# TFT Hyperopt
foxhunt-deploy deploy \
  --command "hyperopt_tft_demo --n-trials 50" \
  --gpu-type "RTX A4000" \
  --name "tft-hyperopt"

# PPO Hyperopt
foxhunt-deploy deploy \
  --command "hyperopt_ppo_demo --n-trials 50" \
  --gpu-type "RTX A4000" \
  --name "ppo-hyperopt"

Expected Cost: $0.25/hr each (~$0.25 total for parallel execution)

6. Custom Environment Variables

Deploy with custom logging and debugging:

foxhunt-deploy deploy \
  --command "train_ppo_parquet --epochs 100" \
  --env "RUST_LOG=debug" \
  --env "RUST_BACKTRACE=1" \
  --env "CUDA_VISIBLE_DEVICES=0"

7. Dry Run (Validation)

Test deployment configuration without actually deploying:

foxhunt-deploy deploy \
  --command "train_ppo --epochs 100" \
  --gpu-type "RTX 4090" \
  --dry-run

Output:

✓ Selected GPU: RTX 4090 (24 GB VRAM, $0.59/hr)

ℹ [DRY RUN] Deployment Summary
────────────────────────────────────────────────────────────
  Pod Name:        foxhunt-20251102-091802
  GPU:             RTX 4090
  VRAM:            24 GB
  Cost:            $0.59/hr
  Datacenter:      EUR-IS-1
  Image:           jgrusewski/foxhunt:latest
  Command:         train_ppo --epochs 100
  Container Disk:  50 GB
────────────────────────────────────────────────────────────

✓ Dry run completed successfully. Request is valid.

GPU Selection Guide

RTX A4000 (Default)

• VRAM: 16 GB
• Cost: $0.25/hr
• Best For: Most training jobs (PPO, DQN, TFT)
• Selection: Auto-selected or --gpu-type "RTX A4000"

RTX 4090

• VRAM: 24 GB
• Cost: $0.59/hr
• Best For: Large batch sizes (MAMBA-2, TFT with large cache)
• Selection: --gpu-type "RTX 4090"

A40

• VRAM: 48 GB
• Cost: $0.45/hr
• Best For: Very large models or ensembles
• Selection: --gpu-type "A40"

Datacenter Selection

EUR-IS-1 (Default)

• Location: Europe (Iceland)
• Latency: ~50ms from EU
• Availability: High
• Selection: Auto-selected or --datacenter "EUR-IS-1"

US-TX-3

• Location: US (Texas)
• Latency: ~30ms from US East
• Availability: Medium
• Selection: --datacenter "US-TX-3"

Monitoring and Management

Monitor Pod Logs

After deployment, monitor training progress:

# Replace <pod_id> with actual pod ID from deployment output
foxhunt-deploy monitor <pod_id>

Terminate Pod

When training completes:

# Replace <pod_id> with actual pod ID
foxhunt-deploy run terminate --pod-id <pod_id>

Check Last Pod ID

The CLI saves the last deployed pod ID to .last_pod_id:

cat .last_pod_id

Cost Estimates

Training Job	GPU	Duration	Cost
PPO Production	RTX A4000	2 hours	$0.50
DQN Hyperopt	RTX A4000	30 min	$0.13
MAMBA-2 Training	RTX 4090	30 min	$0.30
TFT Training	RTX A4000	5 min	$0.02
PPO Hyperopt	RTX A4000	15 min	$0.06

Total for full ML suite: ~$1.00 (with hyperopt)

Troubleshooting

API Key Not Set

Error: RunPod API key not set

Solution:

# Edit config file
nano ~/.runpod/config.toml

# Add API key
[runpod]
api_key = "YOUR_RUNPOD_API_KEY_HERE"

GPU Not Available

Error: GPU type 'RTX 4090' not found

Solution:

1. Try auto-selection (remove --gpu-type flag)
2. Use RTX A4000 instead: --gpu-type "RTX A4000"
3. Check RunPod dashboard for availability

Command Failed

Error: Docker start command cannot be empty

Solution: Ensure --command flag is provided:

foxhunt-deploy deploy --command "train_ppo --epochs 100"

Invalid Environment Variable

Error: Invalid environment variable format: KEY

Solution: Use KEY=VALUE format:

foxhunt-deploy deploy --command "train_ppo" --env "RUST_LOG=debug"

Best Practices

1. Always use dry-run first for new deployments:

   foxhunt-deploy deploy --command "..." --dry-run
   

2. Use descriptive pod names for easy identification:

   --name "ppo-production-v2" instead of auto-generated
   

3. Monitor costs using dry-run cost estimates before deploying

4. Terminate pods immediately when training completes to avoid charges

5. Use RTX A4000 for most jobs (best value, sufficient for 99% of workloads)

6. Set environment variables for debugging during development:

   --env "RUST_LOG=debug" --env "RUST_BACKTRACE=1"
   

Next Steps

After successful deployment:

1. Monitor Logs: foxhunt-deploy monitor <pod_id>
2. Check Training Progress: Watch for epoch updates, loss metrics
3. Download Results: Results saved to /runpod-volume/ml_training/
4. Terminate Pod: foxhunt-deploy run terminate --pod-id <pod_id>
5. Retrieve Models: Access via RunPod S3 or volume download

Support

• RunPod Dashboard: https://www.runpod.io/console/pods
• Foxhunt Docs: See RUNPOD_DEPLOYMENT_IMPLEMENTATION.md
• CLI Help: foxhunt-deploy deploy --help