foxhunt/docs/checklists/SAFE_DEPLOYMENT_CHECKLIST.md

# Safe Deployment Checklist

**CRITICAL**: Follow this checklist EVERY TIME before deploying to Runpod.

**Last Updated**: 2025-10-29
**Purpose**: Prevent deploying wrong binaries to production (prevents wasted time and money)

---

## Pre-Deployment Validation

### 1. Build Binary

```bash
# Build the binary with CUDA support
cargo build -p ml --example <binary_name> --release --features cuda
```

**Example binaries:**
- `hyperopt_mamba2_demo`
- `hyperopt_dqn_demo`
- `hyperopt_ppo_demo`
- `hyperopt_tft_demo`
- `train_mamba2_parquet`
- `train_dqn`
- `train_ppo`

### 2. Test Binary Locally

```bash
# Smoke test - verify binary runs
target/release/examples/<binary_name> --help

# Full test with actual data (1 trial, 1 epoch)
target/release/examples/<binary_name> \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --base-dir /tmp/test \
  --trials 1 --epochs 1
```

**CRITICAL**: Verify the binary has `--base-dir` argument:
```bash
target/release/examples/<binary_name> --help | grep "base-dir"
```

If `--base-dir` is missing, the binary was built BEFORE VarMap fix and must be rebuilt.

### 3. Validate Checksum (MANDATORY)

```bash
./scripts/validate_binary.sh <binary_name>
```

**Expected Output:**
```
========================================
Binary Validation: <binary_name>
========================================
✅ Local binary exists
✅ Local binary works and has correct arguments
🔍 Calculating local SHA256...
Local SHA256:  <64-char-hash>
🔍 Checking S3 binary...
S3 Size:       <bytes>
S3 Modified:   <timestamp>
⬇️  Downloading S3 binary for checksum...
🔍 Calculating S3 SHA256...
S3 SHA256:     <64-char-hash>

========================================
✅ VALIDATION PASSED
Local and S3 binaries match perfectly
========================================
```

**If validation fails:**

```bash
# 1. Delete outdated S3 binary
aws s3 rm s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --profile runpod

# 2. Upload correct local binary
aws s3 cp target/release/examples/<binary_name> \
  s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --profile runpod

# 3. Re-validate
./scripts/validate_binary.sh <binary_name>
```

### 4. Deploy Pod

```bash
# Deploy with automatic checksum validation
python3 scripts/runpod_deploy.py \
  --gpu-type "RTX A4000" \
  --command "/runpod-volume/binaries/<binary_name> --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --base-dir /runpod-volume/hyperopt --trials 10 --epochs 50"
```

The deployment script will automatically:
1. Check binary exists in S3
2. Validate local vs S3 checksum
3. Block deployment if checksums don't match
4. Only deploy if validation passes

---

## Validation Guarantees

✅ **Local binary exists and is executable**
✅ **Local binary has correct CLI arguments (`--base-dir` present)**
✅ **Local binary passes smoke test**
✅ **Local SHA256 matches S3 SHA256**
✅ **Deployment blocked if validation fails**

---

## What This Prevents

❌ **Deploying outdated binaries from S3**
❌ **Deploying binaries without VarMap fixes**
❌ **Wasting time/money on broken pods** (3 failed pods = $0.75+ wasted)
❌ **Manual checksum verification errors**

---

## Common Issues and Solutions

### Issue 1: Binary missing `--base-dir` argument

**Error:**
```
❌ FAIL: Local binary missing --base-dir argument
This binary was built BEFORE VarMap fix!
```

**Solution:**
```bash
# Rebuild binary
cargo clean -p ml
cargo build -p ml --example <binary_name> --release --features cuda

# Verify fix
target/release/examples/<binary_name> --help | grep "base-dir"
```

### Issue 2: Checksum mismatch

**Error:**
```
❌ VALIDATION FAILED
Local and S3 binaries DO NOT MATCH

Local:  abc123...
S3:     def456...
```

**Solution:**
```bash
# Option A: Upload new local binary (if local is correct)
aws s3 rm s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io --profile runpod

aws s3 cp target/release/examples/<binary_name> \
  s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io --profile runpod

# Option B: Rebuild local binary (if S3 is correct)
cargo clean -p ml
cargo build -p ml --example <binary_name> --release --features cuda
```

### Issue 3: S3 binary doesn't exist

**Output:**
```
⚠️  S3 binary does not exist - will be uploaded
VALIDATION: LOCAL_ONLY
```

**Solution:**
```bash
# Upload binary to S3
aws s3 cp target/release/examples/<binary_name> \
  s3://se3zdnb5o4/binaries/current/<binary_name> \
  --endpoint-url https://s3api-eur-is-1.runpod.io \
  --profile runpod

# Verify upload
aws s3 ls s3://se3zdnb5o4/binaries/current/ \
  --endpoint-url https://s3api-eur-is-1.runpod.io --profile runpod
```

---

## Testing Instructions

### Run Full Validation Test Suite

```bash
./scripts/test_binary_validation.sh
```

**Expected Output:**
```
========================================
Binary Validation System Test Suite
========================================

Test 1: Validate hyperopt_mamba2_demo binary
----------------------------------------
✅ Test 1 passed: Validation script works

Test 2: Smoke test binary functionality
----------------------------------------
✅ Test 2 passed: Binary has correct CLI arguments

Test 3: Checksum calculation
----------------------------------------
Local SHA256: <hash>
✅ Test 3 passed: Checksum calculated correctly (64 chars)

Test 4: Python script integration
----------------------------------------
✅ Test 4 passed: Python can call validation script

========================================
✅ All validation tests passed
========================================
```

### Test Deployment (Dry Run)

```bash
# Test deployment without actually deploying
python3 scripts/runpod_deploy.py --dry-run \
  --command "/runpod-volume/binaries/hyperopt_mamba2_demo --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --base-dir /runpod-volume/hyperopt --trials 1 --epochs 1"
```

This will:
1. Run checksum validation
2. Show deployment plan
3. Skip actual deployment

---

## Deployment Workflow Example

**Scenario:** Deploy MAMBA-2 hyperopt with 100 trials

```bash
# Step 1: Build binary
cargo build -p ml --example hyperopt_mamba2_demo --release --features cuda

# Step 2: Test locally (1 trial smoke test)
target/release/examples/hyperopt_mamba2_demo \
  --parquet-file test_data/ES_FUT_180d.parquet \
  --base-dir /tmp/test \
  --trials 1 --epochs 1

# Step 3: Validate checksum
./scripts/validate_binary.sh hyperopt_mamba2_demo

# Step 4: Deploy
python3 scripts/runpod_deploy.py \
  --gpu-type "RTX A4000" \
  --command "/runpod-volume/binaries/hyperopt_mamba2_demo --parquet-file /runpod-volume/test_data/ES_FUT_180d.parquet --base-dir /runpod-volume/hyperopt --trials 100 --epochs 50"

# Step 5: Monitor (from another terminal)
python3 scripts/monitor_pod.py
```

---

## Cost Savings

**Before validation system:**
- 3 failed deployments = 3 × $0.25/hr × 1hr = **$0.75 wasted**
- Manual investigation time = **30-60 minutes**

**After validation system:**
- Deployment blocked before pod creation = **$0.00 cost**
- Automatic detection = **2 minutes**

**Savings per incident:** $0.75 + 30-60 minutes of engineering time

---

## Related Documentation

- **`scripts/validate_binary.sh`**: Checksum validation script
- **`scripts/runpod_deploy.py`**: Deployment script with validation
- **`scripts/test_binary_validation.sh`**: Test suite
- **`RUNPOD_VOLUME_MOUNT_ARCHITECTURE.md`**: Deployment architecture
- **`ML_TRAINING_PARQUET_GUIDE.md`**: Training guide

---

## Questions?

If validation fails and you're unsure why:

1. Check recent commits: `git log --oneline -10`
2. Verify VarMap fix is present: `git log --grep="VarMap"`
3. Check binary size: `ls -lh target/release/examples/<binary_name>`
4. Compare with known-good binary: `./scripts/validate_binary.sh hyperopt_mamba2_demo`

**Always validate before deploying. Always.**