foxhunt/SAFE_DEPLOYMENT_CHECKLIST.md

Safe Deployment Checklist

CRITICAL: Follow this checklist EVERY TIME before deploying to Runpod.

Last Updated: 2025-10-29 Purpose: Prevent deploying wrong binaries to production (prevents wasted time and money)

---

Pre-Deployment Validation

1. Build Binary

# Build the binary with CUDA support
cargo build -p ml --example <binary_name> --release --features cuda

Example binaries:

• hyperopt_mamba2_demo
• hyperopt_dqn_demo
• hyperopt_ppo_demo
• hyperopt_tft_demo
• train_mamba2_parquet
• train_dqn
• train_ppo

2. Test Binary Locally

# Smoke test - verify binary runs
target/release/examples/<binary_name> --help

# Full test with actual data (1 trial, 1 epoch)
target/release/examples/<binary_name> \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --base-dir /tmp/test \
  --trials 1 --epochs 1

CRITICAL: Verify the binary has --base-dir argument:

target/release/examples/<binary_name> --help | grep "base-dir"

If --base-dir is missing, the binary was built BEFORE VarMap fix and must be rebuilt.

3. Validate Checksum (MANDATORY)

./scripts/validate_binary.sh <binary_name>

Expected Output:

========================================
Binary Validation: <binary_name>
========================================
✅ Local binary exists
✅ Local binary works and has correct arguments
🔍 Calculating local SHA256...
Local SHA256:  <64-char-hash>
🔍 Checking S3 binary...
S3 Size:       <bytes>
S3 Modified:   <timestamp>
⬇️  Downloading S3 binary for checksum...
🔍 Calculating S3 SHA256...
S3 SHA256:     <64-char-hash>

========================================
✅ VALIDATION PASSED
Local and S3 binaries match perfectly
========================================

If validation fails:

# 1. Delete outdated S3 binary
aws s3 rm s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --profile runpod

# 2. Upload correct local binary
aws s3 cp target/release/examples/<binary_name> \
  s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --profile runpod

# 3. Re-validate
./scripts/validate_binary.sh <binary_name>

4. Deploy Pod

# Deploy with automatic checksum validation
python3 scripts/runpod_deploy.py \
  --gpu-type "RTX A4000" \
  --command "/runpod-volume/binaries/<binary_name> --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --base-dir /runpod-volume/hyperopt --trials 10 --epochs 50"

The deployment script will automatically:

1. Check binary exists in S3
2. Validate local vs S3 checksum
3. Block deployment if checksums don't match
4. Only deploy if validation passes

---

Validation Guarantees

✅ Local binary exists and is executable ✅ Local binary has correct CLI arguments (--base-dir present) ✅ Local binary passes smoke test ✅ Local SHA256 matches S3 SHA256 ✅ Deployment blocked if validation fails

---

What This Prevents

❌ Deploying outdated binaries from S3 ❌ Deploying binaries without VarMap fixes ❌ Wasting time/money on broken pods (3 failed pods = $0.75+ wasted) ❌ Manual checksum verification errors

---

Common Issues and Solutions

Issue 1: Binary missing --base-dir argument

Error:

❌ FAIL: Local binary missing --base-dir argument
This binary was built BEFORE VarMap fix!

Solution:

# Rebuild binary
cargo clean -p ml
cargo build -p ml --example <binary_name> --release --features cuda

# Verify fix
target/release/examples/<binary_name> --help | grep "base-dir"

Issue 2: Checksum mismatch

Error:

❌ VALIDATION FAILED
Local and S3 binaries DO NOT MATCH

Local:  abc123...
S3:     def456...

Solution:

# Option A: Upload new local binary (if local is correct)
aws s3 rm s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io --profile runpod

aws s3 cp target/release/examples/<binary_name> \
  s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io --profile runpod

# Option B: Rebuild local binary (if S3 is correct)
cargo clean -p ml
cargo build -p ml --example <binary_name> --release --features cuda

Issue 3: S3 binary doesn't exist

Output:

⚠️  S3 binary does not exist - will be uploaded
VALIDATION: LOCAL_ONLY

Solution:

# Upload binary to S3
aws s3 cp target/release/examples/<binary_name> \
  s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --profile runpod

# Verify upload
aws s3 ls s3://se3zdnb5o4/binaries/current/ \
  --endpoint-url https://s3api-eur-is-1.runpod.io --profile runpod

---

Testing Instructions

Run Full Validation Test Suite

./scripts/test_binary_validation.sh

Expected Output:

========================================
Binary Validation System Test Suite
========================================

Test 1: Validate hyperopt_mamba2_demo binary
----------------------------------------
✅ Test 1 passed: Validation script works

Test 2: Smoke test binary functionality
----------------------------------------
✅ Test 2 passed: Binary has correct CLI arguments

Test 3: Checksum calculation
----------------------------------------
Local SHA256: <hash>
✅ Test 3 passed: Checksum calculated correctly (64 chars)

Test 4: Python script integration
----------------------------------------
✅ Test 4 passed: Python can call validation script

========================================
✅ All validation tests passed
========================================

Test Deployment (Dry Run)

# Test deployment without actually deploying
python3 scripts/runpod_deploy.py --dry-run \
  --command "/runpod-volume/binaries/hyperopt_mamba2_demo --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --base-dir /runpod-volume/hyperopt --trials 1 --epochs 1"

This will:

1. Run checksum validation
2. Show deployment plan
3. Skip actual deployment

---

Deployment Workflow Example

Scenario: Deploy MAMBA-2 hyperopt with 100 trials

# Step 1: Build binary
cargo build -p ml --example hyperopt_mamba2_demo --release --features cuda

# Step 2: Test locally (1 trial smoke test)
target/release/examples/hyperopt_mamba2_demo \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --base-dir /tmp/test \
  --trials 1 --epochs 1

# Step 3: Validate checksum
./scripts/validate_binary.sh hyperopt_mamba2_demo

# Step 4: Deploy
python3 scripts/runpod_deploy.py \
  --gpu-type "RTX A4000" \
  --command "/runpod-volume/binaries/hyperopt_mamba2_demo --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --base-dir /runpod-volume/hyperopt --trials 100 --epochs 50"

# Step 5: Monitor (from another terminal)
python3 scripts/monitor_pod.py

---

Cost Savings

Before validation system:

• 3 failed deployments = 3 × $0.25/hr × 1hr = $0.75 wasted
• Manual investigation time = 30-60 minutes

After validation system:

• Deployment blocked before pod creation = $0.00 cost
• Automatic detection = 2 minutes

Savings per incident: $0.75 + 30-60 minutes of engineering time

---

Related Documentation

• scripts/validate_binary.sh: Checksum validation script
• scripts/runpod_deploy.py: Deployment script with validation
• scripts/test_binary_validation.sh: Test suite
• RUNPOD_VOLUME_MOUNT_ARCHITECTURE.md: Deployment architecture
• ML_TRAINING_PARQUET_GUIDE.md: Training guide

---

Questions?

If validation fails and you're unsure why:

1. Check recent commits: git log --oneline -10
2. Verify VarMap fix is present: git log --grep="VarMap"
3. Check binary size: ls -lh target/release/examples/<binary_name>
4. Compare with known-good binary: ./scripts/validate_binary.sh hyperopt_mamba2_demo

Always validate before deploying. Always.