# Database Migration Runbook

**Last Updated**: 2025-10-22
**Purpose**: Safe database schema changes
**RPO**: 1 hour (hourly backups)

---

## Pre-Migration Checklist

- [ ] Migration tested in development environment
- [ ] Migration is backward-compatible (or multi-step plan documented)
- [ ] Database backup completed within last hour
- [ ] Estimated migration time calculated
- [ ] Rollback migration tested
- [ ] Application code supports both old and new schema
- [ ] Low-traffic window scheduled (if long-running migration)

---

## Migration Procedure

### Step 1: Backup Database

```bash
# Kubernetes
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    pg_dump -U foxhunt -d foxhunt -Fc > /backup/foxhunt_pre_migration_$(date +%Y%m%d_%H%M%S).dump

# Verify backup
ls -lh /backup/foxhunt_pre_migration_*.dump
```

### Step 2: Test Migration (Dry Run)

```bash
# Create test database
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d postgres -c "CREATE DATABASE foxhunt_test TEMPLATE foxhunt;"

# Run migration on test database
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt_test < migrations/045_regime_detection.up.sql

# Verify schema
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt_test -c "\dt"

# Measure migration time
time kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt_test < migrations/045_regime_detection.up.sql

# Drop test database
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d postgres -c "DROP DATABASE foxhunt_test;"
```

### Step 3: Apply Migration

```bash
# Option A: Using sqlx (recommended)
kubectl exec -n foxhunt foxhunt-api-gateway-0 -- sqlx migrate run

# Option B: Manual SQL
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt < migrations/045_regime_detection.up.sql

# Verify migration applied
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT version, description, success FROM _sqlx_migrations ORDER BY version DESC LIMIT 5;"
```

### Step 4: Validate Migration

```bash
# Check table structure
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "\d regime_states"

# Test queries
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM regime_states;"

# Verify indexes created
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT indexname, indexdef FROM pg_indexes WHERE tablename='regime_states';"
```

### Step 5: Rollback (if needed)

```bash
# Option A: Restore from backup
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    pg_restore -U foxhunt -d foxhunt -c /backup/foxhunt_pre_migration_20251022_100000.dump

# Option B: Run down migration
kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt < migrations/045_regime_detection.down.sql
```

---

## Long-Running Migrations

For migrations expected to take >5 minutes:

### Use `pt-online-schema-change` (for large tables)

```bash
# Install percona-toolkit
kubectl exec -n foxhunt foxhunt-postgresql-0 -- apt-get update && apt-get install -y percona-toolkit

# Run online schema change (creates shadow table, migrates data, swaps tables)
pt-online-schema-change \
    --alter "ADD COLUMN new_field TEXT" \
    --execute \
    h=localhost,D=foxhunt,t=orders,u=foxhunt,p=${DB_PASSWORD}
```

### Background Index Creation

```sql
-- Instead of:
CREATE INDEX idx_orders_symbol ON orders(symbol);

-- Use:
CREATE INDEX CONCURRENTLY idx_orders_symbol ON orders(symbol);
-- Allows concurrent writes, takes longer but no downtime
```

---

**End of Database Migration Runbook**