# Wave D Rollback Procedure **Date**: 2025-10-18 **Agent**: G24 (Final Production Certification) **Wave**: D Phase 6 - Production Deployment **Status**: ðŸŸĄ **PENDING TESTING** --- ## Executive Summary This document provides comprehensive rollback procedures for Wave D deployment. It defines three rollback levels (feature-only, database, full) with increasing scope and downtime. The recommended approach is Level 1 (feature toggle) which provides zero-downtime rollback with no data loss. **Rollback Philosophy**: "Plan for failure, hope for success" --- ## 1. Rollback Levels Overview | Level | Scope | Downtime | Data Loss | Recovery Time | Complexity | When to Use | |-------|-------|----------|-----------|---------------|------------|-------------| | **Level 1** | Feature toggle | <1 min | None | <1 min | Low | Flip-flopping, false positives, NaN/Inf | | **Level 2** | Database schema | ~5 min | Wave D data | ~5 min | Medium | Database corruption, migration issues | | **Level 3** | Full rollback | ~15 min | Wave D data | ~15 min | High | System unavailable, critical bugs | --- ## 2. Level 1: Feature Toggle Rollback (RECOMMENDED) ### 2.1 Overview **Scope**: Disable Wave D features (24 regime detection features, indices 201-224) without redeploying services. **Advantages**: - Zero downtime - No data loss - Instant rollback (<1 minute) - Easy to re-enable **Disadvantages**: - Wave D data remains in database (unused) - Services continue to run Wave D code (dormant) ### 2.2 Procedure **Step 1: Set Feature Toggle (1 minute)** ```bash # Option A: Environment Variable (requires service restart) export ENABLE_WAVE_D_FEATURES=false systemctl restart trading_service systemctl restart backtesting_service systemctl restart ml_training_service # Option B: Runtime Configuration (no restart required, if implemented) curl -X POST http://localhost:50051/api/v1/config/feature-flags \ -H "Authorization: Bearer $JWT_TOKEN" \ -d '{"enable_wave_d_features": false}' ``` **Step 2: Verify Rollback (30 seconds)** ```bash # Check that 225-feature extraction is disabled curl http://localhost:50052/health | jq '.feature_count' # Expected: 201 (Wave C only) # Check that regime detection endpoints return 404 grpcurl -plaintext localhost:50051 foxhunt.TradingService/GetRegimeState # Expected: "method not found" or "feature disabled" ``` **Step 3: Monitor System (5 minutes)** ```bash # Monitor Grafana dashboards # - Feature extraction latency (should drop to Wave C baseline) # - Memory usage (should drop to Wave C baseline) # - Error rate (should be zero) # Check logs for errors docker-compose logs -f trading_service | grep ERROR ``` **Step 4: Document Rollback** ```bash # Log rollback event in audit log psql $DATABASE_URL -c " INSERT INTO audit_log (timestamp, user_id, action, resource_type, resource_id, details) VALUES (NOW(), 'system', 'rollback_wave_d_level_1', 'feature_toggle', 'wave_d_features', '{\"reason\": \"\", \"level\": 1}'); " ``` ### 2.3 Rollback Triggers (Level 1) | Trigger | Threshold | Alert | Action | Timeframe | |---------|-----------|-------|--------|-----------| | **Flip-flopping** | >50 transitions/hour | P1 HIGH | Disable Wave D | <15 min | | **False positives** | >80% inaccurate | P1 HIGH | Disable Wave D | <15 min | | **NaN/Inf in features** | >0 instances | P0 CRITICAL | Disable Wave D | <5 min | | **Performance degradation** | Latency >2x baseline | P2 MEDIUM | Disable Wave D | <1 hour | ### 2.4 Re-Enable Procedure ```bash # Step 1: Set feature toggle export ENABLE_WAVE_D_FEATURES=true systemctl restart trading_service # Step 2: Verify re-enable curl http://localhost:50052/health | jq '.feature_count' # Expected: 225 (Wave C + Wave D) # Step 3: Monitor for 1 hour # - Check regime transitions are reasonable (5-10/hour) # - Verify no flip-flopping # - Confirm feature extraction is stable ``` --- ## 3. Level 2: Database Rollback ### 3.1 Overview **Scope**: Rollback migration 045 (Wave D schema) while keeping services running. **Advantages**: - Removes Wave D data (clean state) - Services can continue running (degraded) **Disadvantages**: - ~5 minutes downtime (database migration) - Wave D data lost (not recoverable) - Requires service restart ### 3.2 Pre-Rollback Backup **CRITICAL**: ALWAYS backup before rolling back database schema. ```bash # Step 1: Backup Wave D data (2 minutes) pg_dump -U foxhunt -h localhost -p 5432 foxhunt \ -t regime_states \ -t regime_transitions \ -t adaptive_strategy_metrics \ > /backup/wave_d_backup_$(date +%Y%m%d_%H%M%S).sql # Step 2: Verify backup (30 seconds) ls -lh /backup/wave_d_backup_*.sql # Expected: Non-zero file size (e.g., 1-10MB) # Step 3: Test restore (optional, 2 minutes) psql -U foxhunt -h localhost -p 5432 foxhunt_test < /backup/wave_d_backup_*.sql ``` ### 3.3 Rollback Procedure **Step 1: Stop Services (1 minute)** ```bash # Stop all services that write to Wave D tables systemctl stop trading_service systemctl stop backtesting_service systemctl stop ml_training_service systemctl stop trading_agent_service ``` **Step 2: Create Rollback Migration (3 minutes)** ```bash # Create migration file: migrations/046_rollback_wave_d.sql cat > migrations/046_rollback_wave_d.sql << 'EOF' -- Wave D Rollback Migration (046) -- Rolls back migration 045 (Wave D schema) BEGIN; -- Drop Wave D tables (in reverse dependency order) DROP TABLE IF EXISTS adaptive_strategy_metrics CASCADE; DROP TABLE IF EXISTS regime_transitions CASCADE; DROP TABLE IF EXISTS regime_states CASCADE; -- Drop Wave D indexes DROP INDEX IF EXISTS idx_regime_states_symbol_timestamp; DROP INDEX IF EXISTS idx_regime_transitions_symbol_timestamp; DROP INDEX IF EXISTS idx_adaptive_strategy_metrics_symbol_timestamp; -- Drop Wave D views (if any) DROP VIEW IF EXISTS regime_summary CASCADE; -- Update migration version DELETE FROM _sqlx_migrations WHERE version = 45; COMMIT; EOF ``` **Step 3: Apply Rollback Migration (1 minute)** ```bash # Apply rollback migration cargo sqlx migrate run # Verify tables are dropped psql $DATABASE_URL -c "\dt" | grep -E "regime|adaptive" # Expected: No output (tables dropped) ``` **Step 4: Restart Services (1 minute)** ```bash # Restart services with Wave D disabled export ENABLE_WAVE_D_FEATURES=false systemctl start trading_service systemctl start backtesting_service systemctl start ml_training_service systemctl start trading_agent_service ``` **Step 5: Verify Rollback (2 minutes)** ```bash # Check services are healthy curl http://localhost:50052/health curl http://localhost:50053/health curl http://localhost:50054/health # Verify feature count curl http://localhost:50052/health | jq '.feature_count' # Expected: 201 (Wave C only) # Check logs for errors docker-compose logs -f | grep ERROR ``` ### 3.4 Rollback Triggers (Level 2) | Trigger | Threshold | Alert | Action | Timeframe | |---------|-----------|-------|--------|-----------| | **Database corruption** | Wave D tables corrupted | P0 CRITICAL | Rollback DB | <10 min | | **Migration failure** | Migration 045 fails | P1 HIGH | Rollback DB | <10 min | | **Data inconsistency** | >10% data mismatch | P1 HIGH | Rollback DB | <30 min | --- ## 4. Level 3: Full Rollback to Wave C ### 4.1 Overview **Scope**: Complete rollback to Wave C (201 features, no regime detection). **Advantages**: - Known stable state (Wave C validated) - All Wave D code removed **Disadvantages**: - ~15 minutes downtime - Wave D data lost - Requires full redeployment - Most complex rollback ### 4.2 Pre-Rollback Checklist ```bash # 1. Backup database (5 minutes) pg_dump -U foxhunt -h localhost -p 5432 foxhunt > /backup/foxhunt_full_backup_$(date +%Y%m%d_%H%M%S).sql # 2. Backup configuration files (1 minute) cp /etc/foxhunt/*.toml /backup/config_backup_$(date +%Y%m%d_%H%M%S)/ # 3. Document rollback reason (1 minute) echo "Rollback Reason: " > /backup/rollback_reason_$(date +%Y%m%d_%H%M%S).txt ``` ### 4.3 Rollback Procedure **Step 1: Stop All Services (2 minutes)** ```bash # Stop all Foxhunt services systemctl stop api_gateway systemctl stop trading_service systemctl stop backtesting_service systemctl stop ml_training_service systemctl stop trading_agent_service # Verify services are stopped systemctl status api_gateway systemctl status trading_service # Expected: "inactive (dead)" ``` **Step 2: Rollback Git Repository (2 minutes)** ```bash # Find Wave C commit hash git log --oneline --grep="Wave C" | head -1 # Example: 1309eb7c Wave 17: Eliminate 98% of compilation warnings # Checkout Wave C commit git checkout 1309eb7c # Verify checkout git log -1 --oneline # Expected: 1309eb7c Wave 17: Eliminate 98% of compilation warnings ``` **Step 3: Rebuild Services (5 minutes)** ```bash # Clean build cargo clean cargo build --release --workspace # Verify build success ls -lh target/release/trading_service ls -lh target/release/api_gateway # Expected: Non-zero file sizes ``` **Step 4: Rollback Database (3 minutes)** ```bash # Apply Level 2 rollback migration (drop Wave D tables) cargo sqlx migrate run # Verify rollback psql $DATABASE_URL -c "SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;" # Expected: 44 (pre-Wave D) ``` **Step 5: Restart Services (2 minutes)** ```bash # Start services with Wave C configuration export ENABLE_WAVE_D_FEATURES=false systemctl start api_gateway systemctl start trading_service systemctl start backtesting_service systemctl start ml_training_service systemctl start trading_agent_service # Verify services are running systemctl status trading_service # Expected: "active (running)" ``` **Step 6: Verify Rollback (5 minutes)** ```bash # Check services are healthy curl http://localhost:50051/health curl http://localhost:50052/health curl http://localhost:50053/health curl http://localhost:50054/health # Verify feature count curl http://localhost:50052/health | jq '.feature_count' # Expected: 201 (Wave C only) # Test trading functionality tli trade submit --symbol ES.FUT --action BUY --quantity 1 --dry-run # Expected: Success # Check logs for errors docker-compose logs -f | grep ERROR # Expected: No errors ``` **Step 7: Monitor for 1 Hour** ```bash # Monitor Grafana dashboards # - Feature extraction latency (should be Wave C baseline) # - Memory usage (should be Wave C baseline) # - Error rate (should be zero) # - Order submission latency (should be normal) # Check Prometheus metrics curl http://localhost:9090/api/v1/query?query=feature_extraction_latency_p99 # Expected: ~40Ξs (Wave C baseline) ``` ### 4.4 Rollback Triggers (Level 3) | Trigger | Threshold | Alert | Action | Timeframe | |---------|-----------|-------|--------|-----------| | **System unavailable** | >5 min downtime | P0 CRITICAL | Full rollback | Immediate | | **Critical bugs** | System crashes | P0 CRITICAL | Full rollback | Immediate | | **Data corruption** | Database corrupted | P0 CRITICAL | Full rollback | Immediate | | **Level 1/2 failure** | Partial rollback fails | P1 HIGH | Full rollback | <30 min | --- ## 5. Rollback Testing & Validation ### 5.1 Pre-Deployment Testing **CRITICAL**: Test all 3 rollback levels BEFORE production deployment. **Level 1 Test (15 minutes)**: ```bash # 1. Deploy Wave D to staging # 2. Set ENABLE_WAVE_D_FEATURES=false # 3. Verify feature count drops to 201 # 4. Re-enable and verify feature count returns to 225 # 5. Document test results ``` **Level 2 Test (30 minutes)**: ```bash # 1. Deploy Wave D to staging # 2. Create test data in Wave D tables # 3. Backup Wave D data # 4. Apply rollback migration # 5. Verify tables are dropped # 6. Verify services restart successfully # 7. Restore from backup and verify data # 8. Document test results ``` **Level 3 Test (45 minutes)**: ```bash # 1. Deploy Wave D to staging # 2. Backup full database # 3. Checkout Wave C commit # 4. Rebuild services # 5. Apply rollback migration # 6. Restart all services # 7. Verify feature count is 201 # 8. Test trading functionality # 9. Document test results ``` ### 5.2 Rollback Validation Checklist After any rollback, verify: | Check | Command | Expected Result | |-------|---------|----------------| | ✅ Services running | `systemctl status trading_service` | "active (running)" | | ✅ Feature count | `curl localhost:50052/health \| jq '.feature_count'` | 201 (Wave C) | | ✅ Database tables | `psql $DATABASE_URL -c "\\dt"` | No Wave D tables | | ✅ Migration version | `psql $DATABASE_URL -c "SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;"` | 44 (pre-Wave D) | | ✅ No errors in logs | `docker-compose logs -f \| grep ERROR` | No errors | | ✅ Trading functional | `tli trade submit --dry-run` | Success | | ✅ Grafana dashboards | Check http://localhost:3000 | Normal metrics | | ✅ Prometheus alerts | Check http://localhost:9090/alerts | No firing alerts | --- ## 6. Rollback Communication Plan ### 6.1 Internal Communication **Immediate Notification** (within 5 minutes of rollback): ``` Subject: [URGENT] Wave D Rollback Initiated - Level To: Engineering Team, DevOps, Management From: On-Call Engineer Wave D rollback initiated at . Level: <1/2/3> Reason: Expected Downtime: