# Rollback Procedures **Last Updated**: 2025-10-22 **Target Environment**: All (Development, Staging, Production) **RTO (Recovery Time Objective)**: 15 minutes **RPO (Recovery Point Objective)**: 1 hour --- ## Table of Contents 1. [Overview](#overview) 2. [Rollback Decision Matrix](#rollback-decision-matrix) 3. [Level 1: Configuration Rollback](#level-1-configuration-rollback) 4. [Level 2: Application Rollback](#level-2-application-rollback) 5. [Level 3: Database Rollback](#level-3-database-rollback) 6. [Level 4: Full System Rollback](#level-4-full-system-rollback) 7. [Post-Rollback Validation](#post-rollback-validation) 8. [Incident Documentation](#incident-documentation) --- ## Overview This document defines standardized rollback procedures for the Foxhunt HFT trading system. Rollbacks are categorized into 4 levels based on severity and scope. ### Rollback Levels | Level | Scope | Downtime | Complexity | Use Case | |-------|-------|----------|------------|----------| | **L1** | Configuration only | 0-2 min | Low | Feature flags, env vars, ConfigMaps | | **L2** | Application code | 5-10 min | Medium | Code bugs, performance issues | | **L3** | Database schema | 10-20 min | High | Schema changes, migration failures | | **L4** | Full system | 15-30 min | Critical | Catastrophic failures | ### Rollback Decision Matrix ```mermaid graph TD START[Incident Detected] START --> ASSESS{Assess Severity} ASSESS -->|Config Issue| L1[Level 1:
Config Rollback] ASSESS -->|Code Bug| L2[Level 2:
App Rollback] ASSESS -->|DB Issue| L3[Level 3:
DB Rollback] ASSESS -->|System Failure| L4[Level 4:
Full Rollback] L1 --> VALIDATE1{Health OK?} L2 --> VALIDATE2{Health OK?} L3 --> VALIDATE3{Health OK?} L4 --> VALIDATE4{Health OK?} VALIDATE1 -->|Yes| DONE[Incident Resolved] VALIDATE1 -->|No| ESCALATE1[Escalate to L2] VALIDATE2 -->|Yes| DONE VALIDATE2 -->|No| ESCALATE2[Escalate to L3] VALIDATE3 -->|Yes| DONE VALIDATE3 -->|No| ESCALATE3[Escalate to L4] VALIDATE4 -->|Yes| DONE VALIDATE4 -->|No| POSTMORTEM[Postmortem Required] ``` --- ## Level 1: Configuration Rollback **Duration**: 0-2 minutes **Risk**: Low **Downtime**: None (rolling restart) ### Use Cases - Feature flag changes causing errors - Environment variable misconfigurations - ConfigMap/Secret updates breaking functionality - Rate limit adjustments causing service degradation ### Prerequisites ```bash # Ensure kubectl access kubectl cluster-info # Verify current ConfigMap version kubectl get configmap foxhunt-config -n foxhunt -o yaml | grep 'resourceVersion' ``` ### Procedure #### Step 1: Identify Configuration Version to Rollback ```bash # List ConfigMap revision history kubectl rollout history configmap foxhunt-config -n foxhunt # Show specific revision kubectl rollout history configmap foxhunt-config -n foxhunt --revision=2 # Example output: # REVISION CHANGE-CAUSE # 1 Initial deployment # 2 Enable feature X # 3 Increase rate limit (CURRENT - PROBLEMATIC) ``` #### Step 2: Rollback ConfigMap ```bash # Option 1: Rollback to previous revision kubectl rollout undo configmap foxhunt-config -n foxhunt # Option 2: Rollback to specific revision kubectl rollout undo configmap foxhunt-config -n foxhunt --to-revision=2 # Verify rollback kubectl get configmap foxhunt-config -n foxhunt -o yaml ``` #### Step 3: Trigger Rolling Restart ```bash # Restart affected deployments (Kubernetes) kubectl rollout restart deployment/foxhunt-api-gateway -n foxhunt kubectl rollout restart deployment/foxhunt-trading-service -n foxhunt # Wait for rollout to complete kubectl rollout status deployment/foxhunt-api-gateway -n foxhunt # Docker Compose docker-compose restart api_gateway trading_service ``` #### Step 4: Validate Service Health ```bash # Check pod status kubectl get pods -n foxhunt -l app.kubernetes.io/component=api-gateway # Verify health endpoints for i in {1..10}; do curl -f http://localhost:8080/health && echo " OK" || echo " FAILED" sleep 2 done ``` ### Helm Rollback (if deployed via Helm) ```bash # List Helm revisions helm history foxhunt -n foxhunt # REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION # 1 Mon Oct 21 10:00:00 2025 superseded foxhunt-0.1.0 0.1.0 Install complete # 2 Tue Oct 22 08:00:00 2025 deployed foxhunt-0.1.0 0.1.0 Upgrade complete # Rollback to previous revision helm rollback foxhunt -n foxhunt # Rollback to specific revision helm rollback foxhunt 1 -n foxhunt # Wait for rollback to complete helm status foxhunt -n foxhunt ``` --- ## Level 2: Application Rollback **Duration**: 5-10 minutes **Risk**: Medium **Downtime**: 0-5 minutes (depending on strategy) ### Use Cases - Code bugs introduced in recent deployment - Performance regressions - Memory leaks or resource exhaustion - Breaking changes in API ### Prerequisites ```bash # Identify current deployment version kubectl describe deployment foxhunt-api-gateway -n foxhunt | grep Image: # Image: ghcr.io/your-org/foxhunt/api-gateway:v0.1.5 # List available versions (from container registry) docker images | grep foxhunt/api-gateway # ghcr.io/your-org/foxhunt/api-gateway v0.1.5 abc123 2 hours ago 1.2GB # ghcr.io/your-org/foxhunt/api-gateway v0.1.4 def456 1 day ago 1.2GB # ghcr.io/your-org/foxhunt/api-gateway v0.1.3 ghi789 2 days ago 1.2GB ``` ### Procedure #### Step 1: Stop Incoming Traffic (Optional - Production Only) ```bash # Scale down API Gateway to prevent new connections kubectl scale deployment foxhunt-api-gateway -n foxhunt --replicas=0 # Wait for existing connections to drain (30-60 seconds) sleep 60 # Verify no traffic kubectl logs -n foxhunt -l app.kubernetes.io/component=api-gateway --tail=10 ``` #### Step 2: Rollback Deployment **Kubernetes Deployment Rollback**: ```bash # Rollback to previous version kubectl rollout undo deployment/foxhunt-api-gateway -n foxhunt # Rollback to specific revision kubectl rollout undo deployment/foxhunt-api-gateway -n foxhunt --to-revision=3 # Check rollout status kubectl rollout status deployment/foxhunt-api-gateway -n foxhunt # Waiting for deployment "foxhunt-api-gateway" rollout to finish: 1 out of 3 new replicas have been updated... # deployment "foxhunt-api-gateway" successfully rolled out # Verify new image version kubectl get deployment foxhunt-api-gateway -n foxhunt -o jsonpath='{.spec.template.spec.containers[0].image}' # ghcr.io/your-org/foxhunt/api-gateway:v0.1.4 ``` **Docker Compose Rollback**: ```bash # Stop services docker-compose down # Checkout previous git version git log --oneline -10 # Identify commit to rollback to git checkout # Rebuild images docker-compose build --parallel # Start services docker-compose up -d # Verify versions docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" ``` #### Step 3: Rollback All Affected Services ```bash # Rollback all application services (parallel execution) for service in api-gateway trading-service backtesting-service ml-training-service trading-agent; do echo "Rolling back foxhunt-$service..." kubectl rollout undo deployment/foxhunt-$service -n foxhunt & done # Wait for all rollbacks to complete wait # Verify all services are healthy kubectl get pods -n foxhunt -o wide ``` #### Step 4: Restore Traffic ```bash # Scale API Gateway back up kubectl scale deployment foxhunt-api-gateway -n foxhunt --replicas=3 # Monitor pod startup watch kubectl get pods -n foxhunt -l app.kubernetes.io/component=api-gateway ``` --- ## Level 3: Database Rollback **Duration**: 10-20 minutes **Risk**: High (potential data loss) **Downtime**: 5-15 minutes ### Use Cases - Failed database migration - Schema corruption - Data integrity issues - Incompatible schema changes ### Prerequisites ```bash # CRITICAL: Verify backup exists BEFORE rollback kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c "SELECT version, description, success FROM _sqlx_migrations ORDER BY version DESC LIMIT 5;" # Verify backup file exists ls -lh /backup/foxhunt_backup_*.sql # -rw-r--r-- 1 root root 2.5G Oct 22 08:00 /backup/foxhunt_backup_2025-10-22_08-00-00.sql ``` ### Procedure #### Step 1: Stop All Application Services ```bash # Kubernetes: Scale down all services (prevents writes during rollback) for service in api-gateway trading-service backtesting-service ml-training-service trading-agent; do kubectl scale deployment/foxhunt-$service -n foxhunt --replicas=0 done # Docker Compose: Stop application services docker-compose stop api_gateway trading_service backtesting_service ml_training_service trading_agent # Verify no active connections kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM pg_stat_activity WHERE datname='foxhunt';" # count # ----- # 1 (Only postgres itself) ``` #### Step 2: Create Emergency Backup (Before Rollback) ```bash # Kubernetes: Backup current state kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ pg_dump -U foxhunt -d foxhunt > /backup/foxhunt_emergency_backup_$(date +%Y%m%d_%H%M%S).sql # Docker Compose: Backup current state docker-compose exec postgres pg_dump -U foxhunt -d foxhunt > /backup/foxhunt_emergency_backup_$(date +%Y%m%d_%H%M%S).sql # Verify backup file size ls -lh /backup/foxhunt_emergency_backup_*.sql ``` #### Step 3: Restore from Backup **Option A: Full Database Restore** ```bash # Kubernetes: Drop and recreate database kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d postgres -c "DROP DATABASE foxhunt;" kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d postgres -c "CREATE DATABASE foxhunt;" # Restore from backup kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt < /backup/foxhunt_backup_2025-10-22_08-00-00.sql # Verify restoration kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c "\dt" ``` **Option B: Migration Rollback** (preferred for migration failures) ```bash # Identify failed migration kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c "SELECT version, description, success FROM _sqlx_migrations WHERE success=false;" # Revert specific migration using sqlx kubectl exec -n foxhunt foxhunt-api-gateway-0 -- sqlx migrate revert # Or manually rollback migration kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -f /app/migrations/down/.sql ``` #### Step 4: Verify Database Integrity ```bash # Check table counts kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c "SELECT schemaname, tablename, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;" # Verify critical tables exist kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c "SELECT tablename FROM pg_tables WHERE schemaname='public';" | grep -E 'orders|positions|regime_states' # Check for data corruption kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM orders WHERE created_at IS NULL;" # count # ----- # 0 (Good - no NULL timestamps) ``` #### Step 5: Restart Application Services ```bash # Kubernetes: Scale services back up for service in api-gateway trading-service backtesting-service ml-training-service trading-agent; do kubectl scale deployment/foxhunt-$service -n foxhunt --replicas=3 done # Docker Compose: Start services docker-compose up -d # Monitor startup kubectl get pods -n foxhunt -w ``` --- ## Level 4: Full System Rollback **Duration**: 15-30 minutes **Risk**: Critical **Downtime**: 15-30 minutes ### Use Cases - Catastrophic system failure - Multiple cascading failures - Data corruption across services - Infrastructure compromise ### Prerequisites ```bash # Verify ALL backups are available ls -lh /backup/ | grep $(date +%Y-%m-%d) # foxhunt_backup_2025-10-22_08-00-00.sql # redis_dump_2025-10-22_08-00-00.rdb # vault_backup_2025-10-22_08-00-00.tar.gz # Verify cluster health kubectl get nodes # All nodes should be Ready # Identify last known good version git log --oneline -20 git tag -l | tail -10 ``` ### Procedure #### Step 1: Declare System Emergency ```bash # Notify team via Slack/PagerDuty curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \ -H 'Content-Type: application/json' \ -d '{"text":"🚨 EMERGENCY: Full system rollback initiated for Foxhunt trading system. ETA: 30 minutes."}' # Set maintenance mode (if available) kubectl create configmap foxhunt-maintenance -n foxhunt \ --from-literal=enabled=true \ --from-literal=message="System maintenance in progress. Trading suspended." ``` #### Step 2: Stop All Services ```bash # Kubernetes: Delete all application deployments kubectl delete deployment -n foxhunt --all # Docker Compose: Stop all services docker-compose down # Verify all stopped kubectl get pods -n foxhunt # No resources found in foxhunt namespace. ``` #### Step 3: Restore Infrastructure Services **PostgreSQL**: ```bash # Restore PostgreSQL data kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d postgres -c "DROP DATABASE IF EXISTS foxhunt;" kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d postgres -c "CREATE DATABASE foxhunt;" kubectl exec -i -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt < /backup/foxhunt_backup_2025-10-22_08-00-00.sql ``` **Redis**: ```bash # Restore Redis data kubectl cp /backup/redis_dump_2025-10-22_08-00-00.rdb \ foxhunt/foxhunt-redis-master-0:/data/dump.rdb kubectl exec -n foxhunt foxhunt-redis-master-0 -- redis-cli SHUTDOWN SAVE kubectl delete pod foxhunt-redis-master-0 -n foxhunt # Wait for Redis to restart kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=redis -n foxhunt --timeout=120s ``` **Vault**: ```bash # Restore Vault data (if self-hosted) kubectl exec -n foxhunt foxhunt-vault-0 -- vault operator unseal kubectl exec -n foxhunt foxhunt-vault-0 -- vault operator unseal kubectl exec -n foxhunt foxhunt-vault-0 -- vault operator unseal # Restore secrets tar -xzf /backup/vault_backup_2025-10-22_08-00-00.tar.gz -C /tmp/vault_restore kubectl cp /tmp/vault_restore foxhunt/foxhunt-vault-0:/vault/restore kubectl exec -n foxhunt foxhunt-vault-0 -- vault kv put secret/foxhunt/database @/vault/restore/database.json kubectl exec -n foxhunt foxhunt-vault-0 -- vault kv put secret/foxhunt/redis @/vault/restore/redis.json kubectl exec -n foxhunt foxhunt-vault-0 -- vault kv put secret/foxhunt/jwt @/vault/restore/jwt.json ``` #### Step 4: Checkout Last Known Good Version ```bash # Identify last stable version git tag -l | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -5 # v0.1.1 # v0.1.2 # v0.1.3 # v0.1.4 (last known good) # v0.1.5 (current - problematic) # Checkout last stable version git checkout v0.1.4 # Or use specific commit git checkout # Verify version git describe --tags # v0.1.4 ``` #### Step 5: Rebuild and Redeploy **Kubernetes (Helm)**: ```bash # Update Helm chart to use previous version helm upgrade foxhunt ./helm/foxhunt \ --namespace foxhunt \ --set global.imageTag=v0.1.4 \ --force \ --wait \ --timeout 15m # Monitor rollout kubectl rollout status deployment -n foxhunt --watch ``` **Docker Compose**: ```bash # Rebuild images from checked-out version docker-compose build --parallel --no-cache # Start all services docker-compose up -d # Monitor startup docker-compose logs -f ``` #### Step 6: Comprehensive Health Validation ```bash # Run full validation script bash /home/jgrusewski/Work/foxhunt/scripts/validate-deployment.sh # Expected output: # ✅ PostgreSQL: Healthy # ✅ Redis: Healthy # ✅ Vault: Healthy (unsealed) # ✅ API Gateway: Healthy (3/3 replicas ready) # ✅ Trading Service: Healthy (5/5 replicas ready) # ✅ Backtesting Service: Healthy (2/2 replicas ready) # ✅ ML Training Service: Healthy (1/1 replicas ready) # ✅ Trading Agent: Healthy (3/3 replicas ready) # ✅ Database migrations: Up to date (version 044) # ✅ End-to-end test: PASSED ``` #### Step 7: Disable Maintenance Mode ```bash # Remove maintenance ConfigMap kubectl delete configmap foxhunt-maintenance -n foxhunt # Notify team curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \ -H 'Content-Type: application/json' \ -d '{"text":"✅ Full system rollback complete. Foxhunt trading system operational on version v0.1.4."}' ``` --- ## Post-Rollback Validation ### 1. Service Health Checks ```bash # Check all service health endpoints services=("api_gateway:8080" "trading_service:8081" "backtesting_service:8082" "ml_training_service:8095" "trading_agent:8083") for svc in "${services[@]}"; do IFS=':' read -r name port <<< "$svc" echo -n "Checking $name... " if curl -sf http://localhost:$port/health > /dev/null; then echo "✅ Healthy" else echo "❌ FAILED" fi done ``` ### 2. Database Integrity Validation ```bash # Run integrity checks kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c " SELECT 'orders' as table_name, count(*) as row_count, count(DISTINCT symbol) as unique_symbols FROM orders UNION ALL SELECT 'positions' as table_name, count(*) as row_count, count(DISTINCT symbol) as unique_symbols FROM positions; " ``` ### 3. End-to-End Smoke Test ```bash # Test TLI authentication tli auth login --username admin --password $ADMIN_PASSWORD # Submit test order (paper trading) tli trade order submit \ --symbol ES.FUT \ --action BUY \ --quantity 1 \ --order-type MARKET \ --paper-trade # Verify order appears in database kubectl exec -n foxhunt foxhunt-postgresql-0 -- \ psql -U foxhunt -d foxhunt -c "SELECT * FROM orders ORDER BY created_at DESC LIMIT 1;" ``` ### 4. Performance Baseline Validation ```bash # Measure API Gateway latency for i in {1..100}; do curl -w "%{time_total}\n" -o /dev/null -s http://localhost:8080/health done | awk '{sum+=$1; sumsq+=($1)^2} END { avg=sum/NR; stddev=sqrt(sumsq/NR - avg^2); print "Avg: " avg*1000 " ms"; print "P95: " (avg + 1.645*stddev)*1000 " ms"; print "P99: " (avg + 2.326*stddev)*1000 " ms" }' # Expected: # Avg: 0.5 ms # P95: 2.0 ms # P99: 5.0 ms ``` --- ## Incident Documentation After every rollback, complete an incident report: ### Incident Report Template ```markdown # Incident Report: [INCIDENT-ID] **Date**: 2025-10-22 **Time**: 10:00 UTC **Severity**: P1 (Critical) **Duration**: 25 minutes **Rollback Level**: Level 3 (Database Rollback) ## Summary Brief description of what happened. ## Impact - Users affected: 150 active traders - Trading halted: Yes (25 minutes) - Data loss: None - Financial impact: $0 (paper trading only) ## Timeline - 09:55 UTC: Deployment v0.1.5 initiated - 10:00 UTC: Database migration 046 applied - 10:02 UTC: Trading Service unable to connect to database - 10:05 UTC: Decision to rollback - 10:10 UTC: Database restored from backup - 10:15 UTC: Services restarted on v0.1.4 - 10:25 UTC: Full system operational ## Root Cause Migration 046 had incompatible schema changes. ## Resolution Rolled back to database backup from 08:00 UTC, restored v0.1.4. ## Action Items 1. [ ] Add database migration testing to CI/CD pipeline 2. [ ] Implement migration dry-run procedure 3. [ ] Update deployment checklist ## Lessons Learned - Backup frequency (1 hour RPO) was adequate - Database rollback procedure worked as documented - Need automated schema compatibility testing ``` --- **End of Rollback Procedures**