foxhunt/docs/runbooks/disaster-recovery.md

Disaster Recovery Runbook

Last Updated: 2025-10-22 RTO (Recovery Time Objective): 4 hours RPO (Recovery Point Objective): 1 hour

---

Disaster Scenarios

Scenario	Probability	Impact	Recovery Strategy
Single node failure	High	Low	Auto-recovery via K8s
Availability zone outage	Medium	Medium	Multi-AZ failover
Region-wide outage	Low	High	Multi-region DR
Database corruption	Low	High	Point-in-time recovery
Ransomware attack	Very Low	Critical	Offline backups

---

Backup Strategy

Daily Backups

# PostgreSQL daily backup (automated via CronJob)
cat > postgres-backup-cronjob.yaml << 'PBEOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
  namespace: foxhunt
spec:
  schedule: "0 3 * * *"  # 3 AM daily
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres:15
            command:
            - /bin/sh
            - -c
            - |
              pg_dump -U foxhunt -h foxhunt-postgresql -d foxhunt -Fc > /backup/foxhunt_$(date +\%Y\%m\%d).dump
              aws s3 cp /backup/foxhunt_$(date +\%Y\%m\%d).dump s3://foxhunt-backups/daily/
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: foxhunt-secrets
                  key: database-password
            volumeMounts:
            - name: backup
              mountPath: /backup
          volumes:
          - name: backup
            emptyDir: {}
          restartPolicy: OnFailure
PBEOF

kubectl apply -f postgres-backup-cronjob.yaml

Backup Verification

# List backups
aws s3 ls s3://foxhunt-backups/daily/ | tail -10

# Test restore (monthly)
aws s3 cp s3://foxhunt-backups/daily/foxhunt_20251022.dump /tmp/
kubectl exec -i -n foxhunt foxhunt-postgresql-test-0 -- \
    pg_restore -U foxhunt -d foxhunt_test -c < /tmp/foxhunt_20251022.dump

# Verify record count matches

---

Multi-Region DR Setup

Primary Region: us-east-1

DR Region: us-west-2

# Set up replication (AWS RDS example)
aws rds create-db-instance-read-replica \
    --db-instance-identifier foxhunt-postgres-dr \
    --source-db-instance-identifier foxhunt-postgres \
    --db-instance-class db.r5.2xlarge \
    --region us-west-2

# Monitor replication lag
aws rds describe-db-instances \
    --db-instance-identifier foxhunt-postgres-dr \
    --region us-west-2 \
    --query 'DBInstances[0].StatusInfos'

Failover Procedure (Regional Outage)

# 1. Promote DR database to primary
aws rds promote-read-replica \
    --db-instance-identifier foxhunt-postgres-dr \
    --region us-west-2

# 2. Update DNS to point to DR region
aws route53 change-resource-record-sets \
    --hosted-zone-id Z1234567890ABC \
    --change-batch '{ 
        "Changes": [{
            "Action": "UPSERT",
            "ResourceRecordSet": {
                "Name": "api.foxhunt.example.com",
                "Type": "A",
                "AliasTarget": {
                    "HostedZoneId": "Z2WEST",
                    "DNSName": "foxhunt-dr-alb.us-west-2.elb.amazonaws.com",
                    "EvaluateTargetHealth": true
                }
            }
        }]
    }'

# 3. Deploy application to DR region
kubectl config use-context foxhunt-dr-us-west-2
helm install foxhunt ./helm/foxhunt \
    --namespace foxhunt \
    --values values-dr.yaml \
    --wait

# 4. Verify DR system operational
curl -sf https://api.foxhunt.example.com/health

---

End of Disaster Recovery Runbook