foxhunt/PRE_FLIGHT_CHECKLIST.md

# Foxhunt FP32 Production Deployment - Pre-Flight Checklist

**Last Updated**: 2025-10-25  
**Purpose**: Validate all deployment prerequisites before production rollout  
**Estimated Time**: 15-20 minutes

---

## 📋 Checklist Overview

This checklist ensures all components are ready for FP32 production deployment to Runpod GPU infrastructure. Complete all sections sequentially before deploying.

**Legend**:
- ✅ = Complete and verified
- 🔧 = In progress
- ❌ = Not started or failed
- ⚠️ = Warning (non-blocking)
- 🔥 = Critical blocker

---

## 1️⃣ Local Build Environment

### CUDA Installation
- [ ] CUDA Toolkit installed (12.0+)
  ```bash
  nvcc --version | grep "release"
  # Expected: release 12.x or 13.x
  ```

- [ ] CUDA libraries accessible
  ```bash
  ls /usr/local/cuda/lib64/libcublas.so*
  # Expected: libcublas.so.12 or libcublas.so.13
  ```

- [ ] cuDNN installed (8.x or 9.x)
  ```bash
  ls /usr/lib/x86_64-linux-gnu/libcudnn.so*
  # Expected: libcudnn.so.8 or libcudnn.so.9
  ```

- [ ] Local GPU accessible (for testing)
  ```bash
  nvidia-smi
  # Expected: GPU details displayed
  ```

### Rust Environment
- [ ] Cargo installed (1.70+)
  ```bash
  cargo --version
  # Expected: cargo 1.7x.0 or newer
  ```

- [ ] Workspace compiles cleanly
  ```bash
  cargo check --workspace
  # Expected: 0 errors
  ```

- [ ] Release mode builds successfully
  ```bash
  cargo build --release -p ml --features cuda --examples
  # Expected: Build completes in ~6 minutes, 0 errors
  ```

### Test Pass Rate
- [ ] All ML tests passing (FP32 only)
  ```bash
  cargo test -p ml --release -- --skip qat
  # Expected: 597/608 tests passing (exclude 11 broken QAT tests)
  ```

- [ ] Trading Engine tests passing
  ```bash
  cargo test -p trading_engine --release
  # Expected: 314/314 tests passing (100%)
  ```

- [ ] Integration tests operational
  ```bash
  cargo test --workspace --release -- --skip qat
  # Expected: 2,062/2,074 passing (99.4%)
  ```

---

## 2️⃣ Binaries Preparation

### Build All Models
- [ ] TFT-225 binary built
  ```bash
  ls -lh target/release/examples/train_tft_parquet
  # Expected: ~50MB, executable
  ```

- [ ] MAMBA-2 binary built
  ```bash
  ls -lh target/release/examples/train_mamba2_parquet
  # Expected: ~45MB, executable
  ```

- [ ] DQN binary built
  ```bash
  ls -lh target/release/examples/train_dqn
  # Expected: ~30MB, executable
  ```

- [ ] PPO binary built
  ```bash
  ls -lh target/release/examples/train_ppo
  # Expected: ~35MB, executable
  ```

### CUDA Linkage Verification
- [ ] TFT linked to CUDA libraries
  ```bash
  ldd target/release/examples/train_tft_parquet | grep -i cuda
  # Expected: libcuda.so.1, libcurand.so.10, libcublas.so.13
  ```

- [ ] No missing dependencies
  ```bash
  ldd target/release/examples/train_tft_parquet | grep "not found"
  # Expected: No output (all libraries found)
  ```

### Local Smoke Test
- [ ] DQN 1-epoch smoke test passes
  ```bash
  cargo run -p ml --example train_dqn --release --features cuda -- \
    --parquet-file test_data/ES_FUT_small.parquet --epochs 1
  # Expected: Completes in ~15 seconds, 0 errors
  ```

- [ ] GPU utilization confirmed
  ```bash
  # Run in separate terminal during smoke test:
  watch -n 1 nvidia-smi
  # Expected: GPU utilization 50-90%, GPU memory used ~100MB
  ```

---

## 3️⃣ Docker Infrastructure

### Image Build
- [ ] Dockerfile.runpod exists and valid
  ```bash
  ls -lh Dockerfile.runpod
  # Expected: ~8KB file
  ```

- [ ] Docker image builds cleanly
  ```bash
  docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
  # Expected: Build completes in ~2 minutes, image ~8.4GB
  ```

- [ ] Image pushed to Docker Hub
  ```bash
  docker push jgrusewski/foxhunt:latest
  # Expected: Push completes, image available at jgrusewski/foxhunt:latest
  ```

- [ ] Docker Hub repository set to PRIVATE
  ```
  # Manual check:
  # 1. Go to https://hub.docker.com/r/jgrusewski/foxhunt
  # 2. Settings → Visibility → Private
  # Expected: Repository is PRIVATE
  ```

### Entrypoint Scripts
- [ ] entrypoint.sh exists and executable
  ```bash
  ls -lh entrypoint.sh
  # Expected: ~14KB, executable (rwxr-xr-x)
  ```

- [ ] Crash logging logic present
  ```bash
  grep -i "CRASH LOG" entrypoint.sh
  # Expected: Multiple matches (crash log capture enabled)
  ```

---

## 4️⃣ Runpod Infrastructure

### Account & Credentials
- [ ] Runpod account active
  ```
  # Manual check: https://www.runpod.io/console/user/settings
  # Expected: Account in good standing, billing enabled
  ```

- [ ] API key configured
  ```bash
  grep "RUNPOD_API_KEY" .env.runpod
  # Expected: RUNPOD_API_KEY=xxx... (72 characters)
  ```

- [ ] API key valid
  ```bash
  curl -H "Authorization: Bearer ${RUNPOD_API_KEY}" \
    https://api.runpod.io/graphql \
    -d '{"query": "{myself{id}}"}' | jq
  # Expected: {"data": {"myself": {"id": "xxx..."}}}
  ```

### Network Volume
- [ ] Volume created (50GB minimum)
  ```
  # Manual check: https://www.runpod.io/console/user/storage
  # Expected: Network Volume exists, 50GB+, datacenter EUR-IS-1
  ```

- [ ] Volume ID configured
  ```bash
  grep "RUNPOD_VOLUME_ID" .env.runpod
  # Expected: RUNPOD_VOLUME_ID=xxx... (12 characters)
  ```

- [ ] Volume accessible (verify via temporary pod)
  ```
  # Manual check: Deploy temporary pod with volume mount
  # SSH into pod: ssh root@${POD_ID}.ssh.runpod.io
  # Run: ls -lh /runpod-volume/
  # Expected: Directory accessible
  ```

### Binaries Uploaded
- [ ] Binaries directory exists on volume
  ```bash
  ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/binaries/'
  # Expected: 4 binaries (train_tft_parquet, train_mamba2_parquet, train_dqn, train_ppo)
  ```

- [ ] Binaries executable
  ```bash
  ssh root@${POD_ID}.ssh.runpod.io 'ls -l /runpod-volume/binaries/' | grep "rwxr"
  # Expected: All files show rwxr-xr-x permissions
  ```

- [ ] Total binary size ~160MB
  ```bash
  ssh root@${POD_ID}.ssh.runpod.io 'du -sh /runpod-volume/binaries/'
  # Expected: ~160M or 160MB
  ```

### Test Data Uploaded
- [ ] Test data directory exists
  ```bash
  ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/test_data/'
  # Expected: 9 Parquet files
  ```

- [ ] ES.FUT 180-day data present
  ```bash
  ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/test_data/ES_FUT_180d.parquet'
  # Expected: ~2.9MB file
  ```

- [ ] Total test data size ~14MB
  ```bash
  ssh root@${POD_ID}.ssh.runpod.io 'du -sh /runpod-volume/test_data/'
  # Expected: ~14M or 14MB
  ```

### Credentials File
- [ ] .env file uploaded to volume
  ```bash
  ssh root@${POD_ID}.ssh.runpod.io 'ls -l /runpod-volume/.env'
  # Expected: File exists, permissions 600 (rw-------)
  ```

- [ ] .env file has correct permissions
  ```bash
  ssh root@${POD_ID}.ssh.runpod.io 'stat -c%a /runpod-volume/.env'
  # Expected: 600 (owner read/write only)
  ```

---

## 5️⃣ Deployment Scripts

### Script Availability
- [ ] deployment script exists
  ```bash
  ls -lh scripts/runpod_deploy.py
  # Expected: ~14KB, executable
  ```

- [ ] Script dependencies installed
  ```bash
  python3 -c "import requests; import dotenv"
  # Expected: No ImportError
  ```

### Dry Run Validation
- [ ] Dry run executes without errors
  ```bash
  ./scripts/runpod_deploy.py --datacenter EUR-IS-1 --dry-run
  # Expected: Shows deployment plan, no errors
  ```

- [ ] GPU types listed
  ```bash
  ./scripts/runpod_deploy.py --datacenter EUR-IS-1 --dry-run | grep "GPU:"
  # Expected: Shows available GPUs (Tesla V100, RTX A4000, etc.)
  ```

- [ ] Datacenter targeting correct
  ```bash
  ./scripts/runpod_deploy.py --datacenter EUR-IS-1 --dry-run | grep "Datacenters:"
  # Expected: Datacenters: EUR-IS-1 (tries in order)
  ```

---

## 6️⃣ Database & Services (Optional for Training)

### Local Services
- [ ] Docker Compose running (if needed)
  ```bash
  docker-compose ps
  # Expected: postgres, redis, vault UP (for local development only)
  ```

- [ ] PostgreSQL accessible (optional)
  ```bash
  psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c "SELECT 1"
  # Expected: "1" output (or skip if not using DB during training)
  ```

- [ ] Redis accessible (optional)
  ```bash
  redis-cli -h localhost -p 6379 ping
  # Expected: PONG (or skip if not using Redis during training)
  ```

### Database Migrations
- [ ] Migration 045 applied (regime detection)
  ```bash
  cargo sqlx migrate info
  # Expected: 045/045 migrations applied (or skip if training standalone)
  ```

- [ ] Regime tables operational
  ```bash
  psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
    -c "SELECT COUNT(*) FROM regime_states"
  # Expected: 0 or more (or skip if not using DB)
  ```

---

## 7️⃣ Monitoring & Observability

### Runpod Console Access
- [ ] Can access Runpod console
  ```
  # Manual check: https://www.runpod.io/console/pods
  # Expected: Console loads, can view pods
  ```

- [ ] Can view pod logs
  ```
  # Manual check: Deploy test pod → Click pod → Logs tab
  # Expected: Logs visible, auto-refresh working
  ```

### Local Monitoring (Optional)
- [ ] Grafana accessible (optional)
  ```bash
  curl -I http://localhost:3000
  # Expected: HTTP 200 OK (or skip if not using Grafana)
  ```

- [ ] Prometheus accessible (optional)
  ```bash
  curl -I http://localhost:9090
  # Expected: HTTP 200 OK (or skip if not using Prometheus)
  ```

---

## 8️⃣ Cost Budget Validation

### Budget Limits Set
- [ ] Daily budget limit confirmed
  ```
  # Manual check: Runpod console → Settings → Billing → Spending Limits
  # Recommended: $10/day limit
  ```

- [ ] Monthly budget confirmed
  ```
  # Recommended: $100/month limit (covers ~340 hours V100)
  ```

### Estimated Costs Reviewed
- [ ] Training cost estimates calculated
  ```
  TFT-225 (50 epochs):     ~$0.50 (~100 minutes @ $0.29/hr)
  MAMBA-2 (50 epochs):     ~$0.10 (~20 minutes)
  DQN (100 epochs):        ~$0.01 (~2 minutes)
  PPO (100 epochs):        ~$0.05 (~10 minutes)
  
  Daily (4 runs):          ~$0.66
  Monthly (120 runs):      ~$20
  Volume storage:          ~$5/month
  
  TOTAL: ~$25-30/month
  ```

- [ ] Cost alerts configured
  ```
  # Manual check: Runpod console → Settings → Notifications
  # Recommended: Alert at $5, $10, $20 thresholds
  ```

---

## 9️⃣ Security Validation

### Credentials Protection
- [ ] No credentials in Docker image
  ```bash
  docker run --rm jgrusewski/foxhunt:latest env | grep -E "(API_KEY|PASSWORD|TOKEN)"
  # Expected: No output (credentials not baked in)
  ```

- [ ] .env files gitignored
  ```bash
  git status .env.runpod
  # Expected: "not staged for commit" or "untracked"
  ```

- [ ] Docker Hub repository private
  ```
  # Manual check: https://hub.docker.com/r/jgrusewski/foxhunt
  # Expected: "Private" badge visible
  ```

### Volume Security
- [ ] Volume access restricted to account
  ```
  # Manual check: Runpod console → Storage → Network Volumes → Your Volume
  # Expected: Only accessible by your account
  ```

- [ ] SSH keys configured (not passwords)
  ```bash
  grep "PasswordAuthentication" Dockerfile.runpod
  # Expected: PasswordAuthentication no
  ```

---

## 🔟 Final Validation

### Integration Test
- [ ] End-to-end smoke test successful
  ```bash
  # 1. Deploy DQN 1-epoch smoke test
  ./scripts/runpod_deploy.py --datacenter EUR-IS-1
  
  # 2. Wait 2-3 minutes for pod initialization
  
  # 3. Check logs in Runpod console
  # Expected: "Training completed successfully!"
  
  # 4. Verify model saved
  ssh root@${POD_ID}.ssh.runpod.io 'ls -lh /runpod-volume/models/'
  # Expected: dqn_epoch_0.safetensors present
  ```

### Documentation Review
- [ ] DEPLOYMENT_COMMANDS.md read and understood
- [ ] SUCCESS_METRICS.md targets reviewed
- [ ] Rollback procedures documented
- [ ] Emergency contact info available

### Team Readiness
- [ ] Deployment window scheduled
- [ ] Stakeholders notified
- [ ] Rollback plan communicated
- [ ] On-call rotation confirmed

---

## ✅ Final Sign-Off

### Deployment Approval

**Checklist Completion**:
- [ ] All critical items (🔥) completed
- [ ] ≥90% of items checked (warnings okay)
- [ ] No blockers remaining

**Approvals**:
- [ ] Technical lead reviewed checklist
- [ ] Budget approved by finance
- [ ] Deployment window confirmed

**Deployment Decision**:
- [ ] **GO** - All checks passed, ready to deploy
- [ ] **NO-GO** - Blockers remain, see issues below

**Issues/Blockers** (if NO-GO):
```
(List any blockers here)
```

**Deployment Timestamp**: _______________  
**Deployed By**: _______________  
**Pod ID**: _______________  
**Model Trained**: _______________  
**Success Metrics**: See SUCCESS_METRICS.md

---

## 📞 Emergency Contacts

**Runpod Support**:
- Email: support@runpod.io
- Discord: https://discord.gg/runpod
- Docs: https://docs.runpod.io/

**Internal**:
- Deployment Lead: (Your contact info)
- On-Call Engineer: (Your contact info)
- Budget Owner: (Your contact info)

---

## 📚 Related Documentation

- `deploy_fp32_production.sh` - Automated build script
- `DEPLOYMENT_COMMANDS.md` - Command reference
- `SUCCESS_METRICS.md` - Performance targets
- `RUNPOD_REGION_FIX_COMPLETE.md` - Datacenter configuration
- `CLAUDE.md` - System architecture and status

---

**Last Reviewed**: 2025-10-25  
**Next Review**: Before each major deployment  
**Checklist Version**: 1.0 (FP32 Production)