foxhunt/PRE_FLIGHT_CHECKLIST.md

Foxhunt FP32 Production Deployment - Pre-Flight Checklist

Last Updated: 2025-10-25
Purpose: Validate all deployment prerequisites before production rollout
Estimated Time: 15-20 minutes

---

📋 Checklist Overview

This checklist ensures all components are ready for FP32 production deployment to Runpod GPU infrastructure. Complete all sections sequentially before deploying.

Legend:

• ✅ = Complete and verified
• 🔧 = In progress
• ❌ = Not started or failed
• ⚠️ = Warning (non-blocking)
• 🔥 = Critical blocker

---

1️⃣ Local Build Environment

CUDA Installation

• CUDA Toolkit installed (12.0+)

  nvcc --version | grep "release"
  # Expected: release 12.x or 13.x
  

• CUDA libraries accessible

  ls /usr/local/cuda/lib64/libcublas.so*
  # Expected: libcublas.so.12 or libcublas.so.13
  

• cuDNN installed (8.x or 9.x)

  ls /usr/lib/x86_64-linux-gnu/libcudnn.so*
  # Expected: libcudnn.so.8 or libcudnn.so.9
  

• Local GPU accessible (for testing)

  nvidia-smi
  # Expected: GPU details displayed
  

Rust Environment

• Cargo installed (1.70+)

  cargo --version
  # Expected: cargo 1.7x.0 or newer
  

• Workspace compiles cleanly

  cargo check --workspace
  # Expected: 0 errors
  

• Release mode builds successfully

  cargo build --release -p ml --features cuda --examples
  # Expected: Build completes in ~6 minutes, 0 errors
  

Test Pass Rate

• All ML tests passing (FP32 only)

  cargo test -p ml --release -- --skip qat
  # Expected: 597/608 tests passing (exclude 11 broken QAT tests)
  

• Trading Engine tests passing

  cargo test -p trading_engine --release
  # Expected: 314/314 tests passing (100%)
  

• Integration tests operational

  cargo test --workspace --release -- --skip qat
  # Expected: 2,062/2,074 passing (99.4%)
  

---

2️⃣ Binaries Preparation

Build All Models

• TFT-225 binary built

  ls -lh target/release/examples/train_tft_parquet
  # Expected: ~50MB, executable
  

• MAMBA-2 binary built

  ls -lh target/release/examples/train_mamba2_parquet
  # Expected: ~45MB, executable
  

• DQN binary built

  ls -lh target/release/examples/train_dqn
  # Expected: ~30MB, executable
  

• PPO binary built

  ls -lh target/release/examples/train_ppo
  # Expected: ~35MB, executable
  

CUDA Linkage Verification

• TFT linked to CUDA libraries

  ldd target/release/examples/train_tft_parquet | grep -i cuda
  # Expected: libcuda.so.1, libcurand.so.10, libcublas.so.13
  

• No missing dependencies

  ldd target/release/examples/train_tft_parquet | grep "not found"
  # Expected: No output (all libraries found)
  

Local Smoke Test

• DQN 1-epoch smoke test passes

  cargo run -p ml --example train_dqn --release --features cuda -- \
    --parquet-file test_data/ES_FUT_small.parquet --epochs 1
  # Expected: Completes in ~15 seconds, 0 errors
  

• GPU utilization confirmed

  # Run in separate terminal during smoke test:
  watch -n 1 nvidia-smi
  # Expected: GPU utilization 50-90%, GPU memory used ~100MB
  

---

3️⃣ Docker Infrastructure

Image Build

• Dockerfile.runpod exists and valid

  ls -lh Dockerfile.runpod
  # Expected: ~8KB file
  

• Docker image builds cleanly

  docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
  # Expected: Build completes in ~2 minutes, image ~8.4GB
  

• Image pushed to Docker Hub

  docker push jgrusewski/foxhunt:latest
  # Expected: Push completes, image available at jgrusewski/foxhunt:latest
  

• Docker Hub repository set to PRIVATE

  # Manual check:
  # 1. Go to https://hub.docker.com/r/jgrusewski/foxhunt
  # 2. Settings → Visibility → Private
  # Expected: Repository is PRIVATE
  

Entrypoint Scripts

• entrypoint.sh exists and executable

  ls -lh entrypoint.sh
  # Expected: ~14KB, executable (rwxr-xr-x)
  

• Crash logging logic present

  grep -i "CRASH LOG" entrypoint.sh
  # Expected: Multiple matches (crash log capture enabled)
  

---

4️⃣ Runpod Infrastructure

Account & Credentials

• Runpod account active

  # Manual check: https://www.runpod.io/console/user/settings
  # Expected: Account in good standing, billing enabled
  

• API key configured

  grep "RUNPOD_API_KEY" .env.runpod
  # Expected: RUNPOD_API_KEY=xxx... (72 characters)
  

• API key valid

  curl -H "Authorization: Bearer ${RUNPOD_API_KEY}" \
    https://api.runpod.io/graphql \
    -d '{"query": "{myself{id}}"}' | jq
  # Expected: {"data": {"myself": {"id": "xxx..."}}}
  

Network Volume

• Volume created (50GB minimum)

  # Manual check: https://www.runpod.io/console/user/storage
  # Expected: Network Volume exists, 50GB+, datacenter EUR-IS-1
  

• Volume ID configured

  grep "RUNPOD_VOLUME_ID" .env.runpod
  # Expected: RUNPOD_VOLUME_ID=xxx... (12 characters)
  

• Volume accessible (verify via temporary pod)

  # Manual check: Deploy temporary pod with volume mount
  # SSH into pod: ssh root@${POD_ID}.ssh.runpod.io
  # Run: ls -lh /runpod-volume/
  # Expected: Directory accessible
  

Binaries Uploaded

• Binaries directory exists on volume

  ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/binaries/'
  # Expected: 4 binaries (train_tft_parquet, train_mamba2_parquet, train_dqn, train_ppo)
  

• Binaries executable

  ssh root@${POD_ID}.ssh.runpod.io 'ls -l /runpod-volume/binaries/' | grep "rwxr"
  # Expected: All files show rwxr-xr-x permissions
  

• Total binary size ~160MB

  ssh root@${POD_ID}.ssh.runpod.io 'du -sh /runpod-volume/binaries/'
  # Expected: ~160M or 160MB
  

Test Data Uploaded

• Test data directory exists

  ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/test_data/'
  # Expected: 9 Parquet files
  

• ES.FUT 180-day data present

  ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/test_data/ES_FUT_180d.parquet'
  # Expected: ~2.9MB file
  

• Total test data size ~14MB

  ssh root@${POD_ID}.ssh.runpod.io 'du -sh /runpod-volume/test_data/'
  # Expected: ~14M or 14MB
  

Credentials File

• .env file uploaded to volume

  ssh root@${POD_ID}.ssh.runpod.io 'ls -l /runpod-volume/.env'
  # Expected: File exists, permissions 600 (rw-------)
  

• .env file has correct permissions

  ssh root@${POD_ID}.ssh.runpod.io 'stat -c%a /runpod-volume/.env'
  # Expected: 600 (owner read/write only)
  

---

5️⃣ Deployment Scripts

Script Availability

• deployment script exists

  ls -lh scripts/runpod_deploy.py
  # Expected: ~14KB, executable
  

• Script dependencies installed

  python3 -c "import requests; import dotenv"
  # Expected: No ImportError
  

Dry Run Validation

• Dry run executes without errors

  ./scripts/runpod_deploy.py --datacenter EUR-IS-1 --dry-run
  # Expected: Shows deployment plan, no errors
  

• GPU types listed

  ./scripts/runpod_deploy.py --datacenter EUR-IS-1 --dry-run | grep "GPU:"
  # Expected: Shows available GPUs (Tesla V100, RTX A4000, etc.)
  

• Datacenter targeting correct

  ./scripts/runpod_deploy.py --datacenter EUR-IS-1 --dry-run | grep "Datacenters:"
  # Expected: Datacenters: EUR-IS-1 (tries in order)
  

---

6️⃣ Database & Services (Optional for Training)

Local Services

• Docker Compose running (if needed)

  docker-compose ps
  # Expected: postgres, redis, vault UP (for local development only)
  

• PostgreSQL accessible (optional)

  psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT 1"
  # Expected: "1" output (or skip if not using DB during training)
  

• Redis accessible (optional)

  redis-cli -h localhost -p 6379 ping
  # Expected: PONG (or skip if not using Redis during training)
  

Database Migrations

• Migration 045 applied (regime detection)

  cargo sqlx migrate info
  # Expected: 045/045 migrations applied (or skip if training standalone)
  

• Regime tables operational

  psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
    -c "SELECT COUNT(*) FROM regime_states"
  # Expected: 0 or more (or skip if not using DB)
  

---

7️⃣ Monitoring & Observability

Runpod Console Access

• Can access Runpod console

  # Manual check: https://www.runpod.io/console/pods
  # Expected: Console loads, can view pods
  

• Can view pod logs

  # Manual check: Deploy test pod → Click pod → Logs tab
  # Expected: Logs visible, auto-refresh working
  

Local Monitoring (Optional)

• Grafana accessible (optional)

  curl -I http://localhost:3000
  # Expected: HTTP 200 OK (or skip if not using Grafana)
  

• Prometheus accessible (optional)

  curl -I http://localhost:9090
  # Expected: HTTP 200 OK (or skip if not using Prometheus)
  

---

8️⃣ Cost Budget Validation

Budget Limits Set

• Daily budget limit confirmed

  # Manual check: Runpod console → Settings → Billing → Spending Limits
  # Recommended: $10/day limit
  

• Monthly budget confirmed

  # Recommended: $100/month limit (covers ~340 hours V100)
  

Estimated Costs Reviewed

• Training cost estimates calculated

  TFT-225 (50 epochs):     ~$0.50 (~100 minutes @ $0.29/hr)
  MAMBA-2 (50 epochs):     ~$0.10 (~20 minutes)
  DQN (100 epochs):        ~$0.01 (~2 minutes)
  PPO (100 epochs):        ~$0.05 (~10 minutes)
  
  Daily (4 runs):          ~$0.66
  Monthly (120 runs):      ~$20
  Volume storage:          ~$5/month
  
  TOTAL: ~$25-30/month
  

• Cost alerts configured

  # Manual check: Runpod console → Settings → Notifications
  # Recommended: Alert at $5, $10, $20 thresholds
  

---

9️⃣ Security Validation

Credentials Protection

• No credentials in Docker image

  docker run --rm jgrusewski/foxhunt:latest env | grep -E "(API_KEY|PASSWORD|TOKEN)"
  # Expected: No output (credentials not baked in)
  

• .env files gitignored

  git status .env.runpod
  # Expected: "not staged for commit" or "untracked"
  

• Docker Hub repository private

  # Manual check: https://hub.docker.com/r/jgrusewski/foxhunt
  # Expected: "Private" badge visible
  

Volume Security

• Volume access restricted to account

  # Manual check: Runpod console → Storage → Network Volumes → Your Volume
  # Expected: Only accessible by your account
  

• SSH keys configured (not passwords)

  grep "PasswordAuthentication" Dockerfile.runpod
  # Expected: PasswordAuthentication no
  

---

🔟 Final Validation

Integration Test

• End-to-end smoke test successful

  # 1. Deploy DQN 1-epoch smoke test
  ./scripts/runpod_deploy.py --datacenter EUR-IS-1
  
  # 2. Wait 2-3 minutes for pod initialization
  
  # 3. Check logs in Runpod console
  # Expected: "Training completed successfully!"
  
  # 4. Verify model saved
  ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/models/'
  # Expected: dqn_epoch_0.safetensors present
  

Documentation Review

• DEPLOYMENT_COMMANDS.md read and understood
• SUCCESS_METRICS.md targets reviewed
• Rollback procedures documented
• Emergency contact info available

Team Readiness

• Deployment window scheduled
• Stakeholders notified
• Rollback plan communicated
• On-call rotation confirmed

---

✅ Final Sign-Off

Deployment Approval

Checklist Completion:

• All critical items (🔥) completed
• ≥90% of items checked (warnings okay)
• No blockers remaining

Approvals:

• Technical lead reviewed checklist
• Budget approved by finance
• Deployment window confirmed

Deployment Decision:

• GO - All checks passed, ready to deploy
• NO-GO - Blockers remain, see issues below

Issues/Blockers (if NO-GO):

(List any blockers here)

Deployment Timestamp: _______________
Deployed By: _______________
Pod ID: _______________
Model Trained: _______________
Success Metrics: See SUCCESS_METRICS.md

---

📞 Emergency Contacts

Runpod Support:

• Email: support@runpod.io
• Discord: https://discord.gg/runpod
• Docs: https://docs.runpod.io/

Internal:

• Deployment Lead: (Your contact info)
• On-Call Engineer: (Your contact info)
• Budget Owner: (Your contact info)

---

📚 Related Documentation

• deploy_fp32_production.sh - Automated build script
• DEPLOYMENT_COMMANDS.md - Command reference
• SUCCESS_METRICS.md - Performance targets
• RUNPOD_REGION_FIX_COMPLETE.md - Datacenter configuration
• CLAUDE.md - System architecture and status

---

Last Reviewed: 2025-10-25
Next Review: Before each major deployment
Checklist Version: 1.0 (FP32 Production)