foxhunt/docs/archive/wave_d/reports/DEPLOYMENT_COMMANDS.md

Foxhunt FP32 Production Deployment - Quick Command Reference

Last Updated: 2025-10-25
Status: ✅ READY FOR PRODUCTION DEPLOYMENT
Target: Runpod GPU Training (EUR-IS-1)

---

📋 Table of Contents

1. Build Commands
2. Binary Upload
3. Runpod Deployment
4. Model Training
5. Monitoring & Logs
6. Model Download
7. Rollback Procedures
8. Troubleshooting

---

🔨 Build Commands

Full Production Build (All Models)

# Build all ML binaries with CUDA + mimalloc
./deploy_fp32_production.sh

# Manual build (if script not available)
cargo build --release -p ml --features "cuda,mimalloc-allocator" --examples

# Build time: ~6 minutes
# Output: target/release/examples/train_*

Individual Model Builds

# TFT-225 only
cargo build --release -p ml --features "cuda,mimalloc-allocator" --example train_tft_parquet

# MAMBA-2 only
cargo build --release -p ml --features "cuda,mimalloc-allocator" --example train_mamba2_parquet

# DQN only
cargo build --release -p ml --features "cuda,mimalloc-allocator" --example train_dqn

# PPO only
cargo build --release -p ml --features "cuda,mimalloc-allocator" --example train_ppo

Verify Binaries

# List built binaries
ls -lh target/release/examples/train_*

# Verify CUDA linkage
ldd target/release/examples/train_tft_parquet | grep -i cuda

# Expected output:
#   libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1
#   libcurand.so.10 => /lib/x86_64-linux-gnu/libcurand.so.10
#   libcublas.so.13 => /lib/x86_64-linux-gnu/libcublas.so.13

---

📤 Binary Upload

Upload to Runpod Volume via SSH

# 1. Get pod ID from Runpod console
export POD_ID="your-pod-id-here"

# 2. SSH into pod
ssh root@${POD_ID}.ssh.runpod.io

# 3. Create binaries directory
mkdir -p /runpod-volume/binaries
mkdir -p /runpod-volume/test_data
mkdir -p /runpod-volume/models

# 4. Exit SSH and upload from local machine
scp target/release/examples/train_* root@${POD_ID}.ssh.runpod.io:/runpod-volume/binaries/

# 5. Make binaries executable
ssh root@${POD_ID}.ssh.runpod.io 'chmod +x /runpod-volume/binaries/train_*'

# 6. Verify upload
ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/binaries/'

Upload Test Data (if not already present)

# Upload Parquet files to volume
scp test_data/*.parquet root@${POD_ID}.ssh.runpod.io:/runpod-volume/test_data/

# Verify data upload
ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/test_data/'

---

🚀 Runpod Deployment

Automated Deployment (Recommended)

# Dry run (show plan without deploying)
./scripts/runpod_deploy.py --datacenter EUR-IS-1 --dry-run

# Deploy DQN smoke test (1 epoch, ~2 minutes, ~$0.01)
./scripts/runpod_deploy.py --datacenter EUR-IS-1

# Deploy TFT-225 production training (50 epochs, ~100 minutes, ~$0.50)
./scripts/runpod_deploy.py --datacenter EUR-IS-1 \
  --command '/runpod-volume/binaries/train_tft_parquet --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 50 --use-gpu'

# Deploy MAMBA-2 training
./scripts/runpod_deploy.py --datacenter EUR-IS-1 \
  --command '/runpod-volume/binaries/train_mamba2_parquet --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 50 --use-gpu'

# Deploy with specific GPU preference
./scripts/runpod_deploy.py --datacenter EUR-IS-1 --gpu-type "RTX A4000" \
  --command '/runpod-volume/binaries/train_tft_parquet --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 50 --use-gpu'

Manual Deployment (Runpod Console)

Configuration:
  • GPU: Tesla V100-PCIE-16GB or RTX A4000 (16GB+ VRAM)
  • Cloud Type: SECURE (not community)
  • Datacenter: EUR-IS-1 (CRITICAL: matches volume location)
  • Docker Image: jgrusewski/foxhunt:latest
  • Container Disk: 50GB
  • Network Volume: Select your volume, mount at /runpod-volume
  • Ports: 22/tcp (SSH), 8888/http (Jupyter - optional)
  • Environment Variables:
      BINARY_NAME=train_tft_parquet
  • Docker Arguments (override CMD):
      --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 50 --use-gpu

---

🧠 Model Training

Training Commands (via dockerStartCmd)

# TFT-225 (50 epochs, ~100 minutes)
/runpod-volume/binaries/train_tft_parquet \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --epochs 50 \
  --use-gpu \
  --output-dir /runpod-volume/models

# TFT-225 with INT8 quantization (50 epochs, ~100 minutes, 75% memory savings)
/runpod-volume/binaries/train_tft_parquet \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --epochs 50 \
  --use-gpu \
  --use-int8 \
  --output-dir /runpod-volume/models

# MAMBA-2 (50 epochs, ~20 minutes)
/runpod-volume/binaries/train_mamba2_parquet \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --epochs 50 \
  --use-gpu \
  --output-dir /runpod-volume/models

# DQN (100 epochs, ~2 minutes)
/runpod-volume/binaries/train_dqn \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --epochs 100 \
  --output-dir /runpod-volume/models

# PPO (100 epochs, ~10 minutes)
/runpod-volume/binaries/train_ppo \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --episodes 100 \
  --output-dir /runpod-volume/models

# DQN smoke test (1 epoch, ~2 minutes, validation only)
/runpod-volume/binaries/train_dqn \
  --parquet-file /runpod-volume/test_data/ES_FUT_small.parquet \
  --epochs 1 \
  --output-dir /runpod-volume/models

Multi-Asset Training

# Train on multiple symbols (90-180 day datasets)
/runpod-volume/binaries/train_tft_parquet \
  --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet \
  --parquet-file /runpod-volume/test_data/NQ_FUT_180d.parquet \
  --parquet-file /runpod-volume/test_data/6E_FUT_180d.parquet \
  --parquet-file /runpod-volume/test_data/ZN_FUT_90d.parquet \
  --epochs 50 \
  --use-gpu \
  --output-dir /runpod-volume/models

---

📊 Monitoring & Logs

Real-Time Log Access

# Via Runpod console
# 1. Go to https://www.runpod.io/console/pods
# 2. Click on your pod
# 3. Click "Logs" tab
# 4. Logs auto-refresh every 5 seconds

# Via SSH (if pod still running)
ssh root@${POD_ID}.ssh.runpod.io 'tail -f /tmp/foxhunt-crash.log'

# View GPU utilization
ssh root@${POD_ID}.ssh.runpod.io 'watch -n 1 nvidia-smi'

Key Log Patterns

SUCCESS PATTERNS:
  • "✓ Volume mounted successfully"
  • "✓ Binary found: /runpod-volume/binaries/train_tft_parquet"
  • "GPU Information: Tesla V100-PCIE-16GB"
  • "Epoch 1/50 completed in 120.5s"
  • "[TIMESTAMP] Training completed successfully!"

ERROR PATTERNS:
  • "ERROR: /runpod-volume directory does not exist" → Volume not mounted
  • "ERROR: Training binary not found" → Binary not uploaded
  • "CUDA error: out of memory" → GPU OOM (reduce batch size or use INT8)
  • "FATAL ERROR: Binary disappeared" → Volume unmounted during training
  • "Training binary exited with code: 101" → Binary crash (check logs above)

---

💾 Model Download

Download Trained Models

# List trained models
ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/models/'

# Download all models
scp -r root@${POD_ID}.ssh.runpod.io:/runpod-volume/models/ ./trained_models_runpod/

# Download specific model
scp root@${POD_ID}.ssh.runpod.io:/runpod-volume/models/tft_225_epoch_49.safetensors ./ml/trained_models/

# Download with timestamp verification
ssh root@${POD_ID}.ssh.runpod.io 'ls -lh --time-style=long-iso /runpod-volume/models/' | grep "epoch_49"

Model Validation (Local)

# Copy to ml/trained_models/
cp trained_models_runpod/tft_225_epoch_49.safetensors ml/trained_models/

# Run validation test
cargo test -p ml --release -- --exact tft_model_inference

# Expected output:
#   test tft_model_inference ... ok
#   Inference time: ~2.9ms (FP32) or ~3.2ms (INT8)

---

🔄 Rollback Procedures

Rollback Trained Model

# 1. List all trained models with timestamps
ssh root@${POD_ID}.ssh.runpod.io 'ls -lt /runpod-volume/models/'

# 2. Download previous version
scp root@${POD_ID}.ssh.runpod.io:/runpod-volume/models/tft_225_epoch_49_backup.safetensors ./ml/trained_models/

# 3. Restore to production
cp ml/trained_models/tft_225_epoch_49_backup.safetensors ml/trained_models/tft_225_epoch_49.safetensors

# 4. Validate rollback
cargo test -p ml --release -- --exact tft_model_inference

Rollback Binary (Bad Build)

# 1. Download working binary from volume
scp root@${POD_ID}.ssh.runpod.io:/runpod-volume/binaries/train_tft_parquet ./target/release/examples/train_tft_parquet.working

# 2. Replace bad binary
cp ./target/release/examples/train_tft_parquet.working ./target/release/examples/train_tft_parquet

# 3. Re-upload to volume
scp ./target/release/examples/train_tft_parquet root@${POD_ID}.ssh.runpod.io:/runpod-volume/binaries/

# 4. Test deployment
./scripts/runpod_deploy.py --datacenter EUR-IS-1 --dry-run

Emergency Stop

# Stop training pod via Runpod API
export RUNPOD_API_KEY="your-api-key"
export POD_ID="your-pod-id"

curl -X POST https://rest.runpod.io/v1/pods/${POD_ID}/stop \
  -H "Authorization: Bearer ${RUNPOD_API_KEY}" \
  -H "Content-Type: application/json"

# Or via Runpod console:
# 1. Go to https://www.runpod.io/console/pods
# 2. Click on pod
# 3. Click "Stop" button

---

🔧 Troubleshooting

Volume Not Mounted

ERROR: /runpod-volume directory does not exist

SOLUTION:
  1. Stop pod
  2. Edit pod configuration
  3. Add volume mount: /runpod-volume
  4. Select Runpod Network Volume from dropdown
  5. Redeploy pod

Binary Not Found

ERROR: Training binary not found: /runpod-volume/binaries/train_tft_parquet

SOLUTION:
  1. Verify binaries uploaded:
     ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/binaries/'
  
  2. Re-upload if missing:
     scp target/release/examples/train_tft_parquet root@${POD_ID}.ssh.runpod.io:/runpod-volume/binaries/
  
  3. Make executable:
     ssh root@${POD_ID}.ssh.runpod.io 'chmod +x /runpod-volume/binaries/train_tft_parquet'

GPU Out of Memory

CUDA error: out of memory (error 2)

SOLUTION (choose one):
  1. Use INT8 quantization (75% memory savings):
     --use-int8
  
  2. Reduce batch size (add to command):
     --batch-size 16  # Default: 32
  
  3. Use larger GPU:
     ./scripts/runpod_deploy.py --gpu-type "RTX A6000"
  
  4. Enable gradient checkpointing (when implemented):
     --gradient-checkpointing

Pod Restart Loop

SYMPTOM: Pod restarts every 30-60 seconds

SOLUTION:
  1. Check crash logs in Runpod console
  
  2. Common causes:
     - Volume not mounted → Add volume mount
     - Binary crash → Check ldd compatibility
     - CUDA mismatch → Use CUDA 12.x+ GPU
     - OOM → Reduce batch size or use INT8
  
  3. Test locally first:
     cargo run -p ml --example train_tft_parquet --release --features cuda -- \
       --parquet-file test_data/ES_FUT_180d.parquet --epochs 1 --use-gpu

Wrong Datacenter

SYMPTOM: Volume not accessible, empty /runpod-volume/

SOLUTION:
  1. Check volume location:
     # Runpod console → Storage → Network Volumes → Your Volume → Datacenter
  
  2. CRITICAL: Pod MUST deploy to same datacenter as volume
     # Volume in EUR-IS-1 → Pod MUST deploy to EUR-IS-1
  
  3. Deploy with datacenter targeting:
     ./scripts/runpod_deploy.py --datacenter EUR-IS-1

---

📈 Performance Targets

FP32 Models (Current Production)

Model	Training Time	GPU Memory	Inference	Cost (50 epochs)
TFT-225	~100 min	~500MB	~2.9ms	~$0.50
MAMBA-2	~20 min	~164MB	~500μs	~$0.10
DQN	~2 min	~6MB	~200μs	~$0.01
PPO	~10 min	~145MB	~324μs	~$0.05

INT8 Quantized (PTQ - Memory Constrained)

Model	Training Time	GPU Memory	Inference	Cost (50 epochs)
TFT-INT8	~100 min	~125MB	~3.2ms	~$0.50

Costs based on Tesla V100 @ $0.29/hr

---

🎯 Quick Reference Card

# BUILD
./deploy_fp32_production.sh

# UPLOAD
scp target/release/examples/train_* root@${POD_ID}.ssh.runpod.io:/runpod-volume/binaries/

# DEPLOY (smoke test)
./scripts/runpod_deploy.py --datacenter EUR-IS-1

# DEPLOY (TFT-225 production)
./scripts/runpod_deploy.py --datacenter EUR-IS-1 \
  --command '/runpod-volume/binaries/train_tft_parquet --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --epochs 50 --use-gpu'

# MONITOR
# Go to: https://www.runpod.io/console/pods → Click pod → Logs tab

# DOWNLOAD
scp -r root@${POD_ID}.ssh.runpod.io:/runpod-volume/models/ ./trained_models_runpod/

# STOP
# Runpod console → Pods → Click pod → Stop

---

See Also:

• deploy_fp32_production.sh - Automated build script
• PRE_FLIGHT_CHECKLIST.md - Deployment validation
• SUCCESS_METRICS.md - Performance targets
• RUNPOD_REGION_FIX_COMPLETE.md - Datacenter configuration