foxhunt/WAVE_D_ROLLBACK_PROCEDURE.md

Wave D Rollback Procedure

Date: 2025-10-18 Agent: G24 (Final Production Certification) Wave: D Phase 6 - Production Deployment Status: 🟡 PENDING TESTING

---

Executive Summary

This document provides comprehensive rollback procedures for Wave D deployment. It defines three rollback levels (feature-only, database, full) with increasing scope and downtime. The recommended approach is Level 1 (feature toggle) which provides zero-downtime rollback with no data loss.

Rollback Philosophy: "Plan for failure, hope for success"

---

1. Rollback Levels Overview

Level	Scope	Downtime	Data Loss	Recovery Time	Complexity	When to Use
Level 1	Feature toggle	<1 min	None	<1 min	Low	Flip-flopping, false positives, NaN/Inf
Level 2	Database schema	~5 min	Wave D data	~5 min	Medium	Database corruption, migration issues
Level 3	Full rollback	~15 min	Wave D data	~15 min	High	System unavailable, critical bugs

---

2. Level 1: Feature Toggle Rollback (RECOMMENDED)

2.1 Overview

Scope: Disable Wave D features (24 regime detection features, indices 201-224) without redeploying services.

Advantages:

• Zero downtime
• No data loss
• Instant rollback (<1 minute)
• Easy to re-enable

Disadvantages:

• Wave D data remains in database (unused)
• Services continue to run Wave D code (dormant)

2.2 Procedure

Step 1: Set Feature Toggle (1 minute)

# Option A: Environment Variable (requires service restart)
export ENABLE_WAVE_D_FEATURES=false
systemctl restart trading_service
systemctl restart backtesting_service
systemctl restart ml_training_service

# Option B: Runtime Configuration (no restart required, if implemented)
curl -X POST http://localhost:50051/api/v1/config/feature-flags \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -d '{"enable_wave_d_features": false}'

Step 2: Verify Rollback (30 seconds)

# Check that 225-feature extraction is disabled
curl http://localhost:50052/health | jq '.feature_count'
# Expected: 201 (Wave C only)

# Check that regime detection endpoints return 404
grpcurl -plaintext localhost:50051 foxhunt.TradingService/GetRegimeState
# Expected: "method not found" or "feature disabled"

Step 3: Monitor System (5 minutes)

# Monitor Grafana dashboards
# - Feature extraction latency (should drop to Wave C baseline)
# - Memory usage (should drop to Wave C baseline)
# - Error rate (should be zero)

# Check logs for errors
docker-compose logs -f trading_service | grep ERROR

Step 4: Document Rollback

# Log rollback event in audit log
psql $DATABASE_URL -c "
INSERT INTO audit_log (timestamp, user_id, action, resource_type, resource_id, details)
VALUES (NOW(), 'system', 'rollback_wave_d_level_1', 'feature_toggle', 'wave_d_features', '{\"reason\": \"<REASON>\", \"level\": 1}');
"

2.3 Rollback Triggers (Level 1)

Trigger	Threshold	Alert	Action	Timeframe
Flip-flopping	>50 transitions/hour	P1 HIGH	Disable Wave D	<15 min
False positives	>80% inaccurate	P1 HIGH	Disable Wave D	<15 min
NaN/Inf in features	>0 instances	P0 CRITICAL	Disable Wave D	<5 min
Performance degradation	Latency >2x baseline	P2 MEDIUM	Disable Wave D	<1 hour

2.4 Re-Enable Procedure

# Step 1: Set feature toggle
export ENABLE_WAVE_D_FEATURES=true
systemctl restart trading_service

# Step 2: Verify re-enable
curl http://localhost:50052/health | jq '.feature_count'
# Expected: 225 (Wave C + Wave D)

# Step 3: Monitor for 1 hour
# - Check regime transitions are reasonable (5-10/hour)
# - Verify no flip-flopping
# - Confirm feature extraction is stable

---

3. Level 2: Database Rollback

3.1 Overview

Scope: Rollback migration 045 (Wave D schema) while keeping services running.

Advantages:

• Removes Wave D data (clean state)
• Services can continue running (degraded)

Disadvantages:

• ~5 minutes downtime (database migration)
• Wave D data lost (not recoverable)
• Requires service restart

3.2 Pre-Rollback Backup

CRITICAL: ALWAYS backup before rolling back database schema.

# Step 1: Backup Wave D data (2 minutes)
pg_dump -U foxhunt -h localhost -p 5432 foxhunt \
  -t regime_states \
  -t regime_transitions \
  -t adaptive_strategy_metrics \
  > /backup/wave_d_backup_$(date +%Y%m%d_%H%M%S).sql

# Step 2: Verify backup (30 seconds)
ls -lh /backup/wave_d_backup_*.sql
# Expected: Non-zero file size (e.g., 1-10MB)

# Step 3: Test restore (optional, 2 minutes)
psql -U foxhunt -h localhost -p 5432 foxhunt_test < /backup/wave_d_backup_*.sql

3.3 Rollback Procedure

Step 1: Stop Services (1 minute)

# Stop all services that write to Wave D tables
systemctl stop trading_service
systemctl stop backtesting_service
systemctl stop ml_training_service
systemctl stop trading_agent_service

Step 2: Create Rollback Migration (3 minutes)

# Create migration file: migrations/046_rollback_wave_d.sql
cat > migrations/046_rollback_wave_d.sql << 'EOF'
-- Wave D Rollback Migration (046)
-- Rolls back migration 045 (Wave D schema)

BEGIN;

-- Drop Wave D tables (in reverse dependency order)
DROP TABLE IF EXISTS adaptive_strategy_metrics CASCADE;
DROP TABLE IF EXISTS regime_transitions CASCADE;
DROP TABLE IF EXISTS regime_states CASCADE;

-- Drop Wave D indexes
DROP INDEX IF EXISTS idx_regime_states_symbol_timestamp;
DROP INDEX IF EXISTS idx_regime_transitions_symbol_timestamp;
DROP INDEX IF EXISTS idx_adaptive_strategy_metrics_symbol_timestamp;

-- Drop Wave D views (if any)
DROP VIEW IF EXISTS regime_summary CASCADE;

-- Update migration version
DELETE FROM _sqlx_migrations WHERE version = 45;

COMMIT;
EOF

Step 3: Apply Rollback Migration (1 minute)

# Apply rollback migration
cargo sqlx migrate run

# Verify tables are dropped
psql $DATABASE_URL -c "\dt" | grep -E "regime|adaptive"
# Expected: No output (tables dropped)

Step 4: Restart Services (1 minute)

# Restart services with Wave D disabled
export ENABLE_WAVE_D_FEATURES=false
systemctl start trading_service
systemctl start backtesting_service
systemctl start ml_training_service
systemctl start trading_agent_service

Step 5: Verify Rollback (2 minutes)

# Check services are healthy
curl http://localhost:50052/health
curl http://localhost:50053/health
curl http://localhost:50054/health

# Verify feature count
curl http://localhost:50052/health | jq '.feature_count'
# Expected: 201 (Wave C only)

# Check logs for errors
docker-compose logs -f | grep ERROR

3.4 Rollback Triggers (Level 2)

Trigger	Threshold	Alert	Action	Timeframe
Database corruption	Wave D tables corrupted	P0 CRITICAL	Rollback DB	<10 min
Migration failure	Migration 045 fails	P1 HIGH	Rollback DB	<10 min
Data inconsistency	>10% data mismatch	P1 HIGH	Rollback DB	<30 min

---

4. Level 3: Full Rollback to Wave C

4.1 Overview

Scope: Complete rollback to Wave C (201 features, no regime detection).

Advantages:

• Known stable state (Wave C validated)
• All Wave D code removed

Disadvantages:

• ~15 minutes downtime
• Wave D data lost
• Requires full redeployment
• Most complex rollback

4.2 Pre-Rollback Checklist

# 1. Backup database (5 minutes)
pg_dump -U foxhunt -h localhost -p 5432 foxhunt > /backup/foxhunt_full_backup_$(date +%Y%m%d_%H%M%S).sql

# 2. Backup configuration files (1 minute)
cp /etc/foxhunt/*.toml /backup/config_backup_$(date +%Y%m%d_%H%M%S)/

# 3. Document rollback reason (1 minute)
echo "Rollback Reason: <REASON>" > /backup/rollback_reason_$(date +%Y%m%d_%H%M%S).txt

4.3 Rollback Procedure

Step 1: Stop All Services (2 minutes)

# Stop all Foxhunt services
systemctl stop api_gateway
systemctl stop trading_service
systemctl stop backtesting_service
systemctl stop ml_training_service
systemctl stop trading_agent_service

# Verify services are stopped
systemctl status api_gateway
systemctl status trading_service
# Expected: "inactive (dead)"

Step 2: Rollback Git Repository (2 minutes)

# Find Wave C commit hash
git log --oneline --grep="Wave C" | head -1
# Example: 1309eb7c Wave 17: Eliminate 98% of compilation warnings

# Checkout Wave C commit
git checkout 1309eb7c

# Verify checkout
git log -1 --oneline
# Expected: 1309eb7c Wave 17: Eliminate 98% of compilation warnings

Step 3: Rebuild Services (5 minutes)

# Clean build
cargo clean
cargo build --release --workspace

# Verify build success
ls -lh target/release/trading_service
ls -lh target/release/api_gateway
# Expected: Non-zero file sizes

Step 4: Rollback Database (3 minutes)

# Apply Level 2 rollback migration (drop Wave D tables)
cargo sqlx migrate run

# Verify rollback
psql $DATABASE_URL -c "SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;"
# Expected: 44 (pre-Wave D)

Step 5: Restart Services (2 minutes)

# Start services with Wave C configuration
export ENABLE_WAVE_D_FEATURES=false
systemctl start api_gateway
systemctl start trading_service
systemctl start backtesting_service
systemctl start ml_training_service
systemctl start trading_agent_service

# Verify services are running
systemctl status trading_service
# Expected: "active (running)"

Step 6: Verify Rollback (5 minutes)

# Check services are healthy
curl http://localhost:50051/health
curl http://localhost:50052/health
curl http://localhost:50053/health
curl http://localhost:50054/health

# Verify feature count
curl http://localhost:50052/health | jq '.feature_count'
# Expected: 201 (Wave C only)

# Test trading functionality
tli trade submit --symbol ES.FUT --action BUY --quantity 1 --dry-run
# Expected: Success

# Check logs for errors
docker-compose logs -f | grep ERROR
# Expected: No errors

Step 7: Monitor for 1 Hour

# Monitor Grafana dashboards
# - Feature extraction latency (should be Wave C baseline)
# - Memory usage (should be Wave C baseline)
# - Error rate (should be zero)
# - Order submission latency (should be normal)

# Check Prometheus metrics
curl http://localhost:9090/api/v1/query?query=feature_extraction_latency_p99
# Expected: ~40μs (Wave C baseline)

4.4 Rollback Triggers (Level 3)

Trigger	Threshold	Alert	Action	Timeframe
System unavailable	>5 min downtime	P0 CRITICAL	Full rollback	Immediate
Critical bugs	System crashes	P0 CRITICAL	Full rollback	Immediate
Data corruption	Database corrupted	P0 CRITICAL	Full rollback	Immediate
Level 1/2 failure	Partial rollback fails	P1 HIGH	Full rollback	<30 min

---

5. Rollback Testing & Validation

5.1 Pre-Deployment Testing

CRITICAL: Test all 3 rollback levels BEFORE production deployment.

Level 1 Test (15 minutes):

# 1. Deploy Wave D to staging
# 2. Set ENABLE_WAVE_D_FEATURES=false
# 3. Verify feature count drops to 201
# 4. Re-enable and verify feature count returns to 225
# 5. Document test results

Level 2 Test (30 minutes):

# 1. Deploy Wave D to staging
# 2. Create test data in Wave D tables
# 3. Backup Wave D data
# 4. Apply rollback migration
# 5. Verify tables are dropped
# 6. Verify services restart successfully
# 7. Restore from backup and verify data
# 8. Document test results

Level 3 Test (45 minutes):

# 1. Deploy Wave D to staging
# 2. Backup full database
# 3. Checkout Wave C commit
# 4. Rebuild services
# 5. Apply rollback migration
# 6. Restart all services
# 7. Verify feature count is 201
# 8. Test trading functionality
# 9. Document test results

5.2 Rollback Validation Checklist

After any rollback, verify:

Check	Command	Expected Result
✅ Services running	systemctl status trading_service	"active (running)"
✅ Feature count	curl localhost:50052/health | jq '.feature_count'	201 (Wave C)
✅ Database tables	psql $DATABASE_URL -c "\\dt"	No Wave D tables
✅ Migration version	psql $DATABASE_URL -c "SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;"	44 (pre-Wave D)
✅ No errors in logs	docker-compose logs -f | grep ERROR	No errors
✅ Trading functional	tli trade submit --dry-run	Success
✅ Grafana dashboards	Check http://localhost:3000	Normal metrics
✅ Prometheus alerts	Check http://localhost:9090/alerts	No firing alerts

---

6. Rollback Communication Plan

6.1 Internal Communication

Immediate Notification (within 5 minutes of rollback):

Subject: [URGENT] Wave D Rollback Initiated - Level <LEVEL>

To: Engineering Team, DevOps, Management
From: On-Call Engineer

Wave D rollback initiated at <TIMESTAMP>.

Level: <1/2/3>
Reason: <REASON>
Expected Downtime: <TIME>
Estimated Recovery: <TIME>

Current Status: <IN PROGRESS / COMPLETE>

Actions Taken:
- <ACTION 1>
- <ACTION 2>

Next Steps:
- <STEP 1>
- <STEP 2>

Monitoring: <GRAFANA_DASHBOARD_URL>

Post-Rollback Report (within 24 hours):

Subject: Wave D Rollback Post-Mortem - Level <LEVEL>

To: Engineering Team, DevOps, Management
From: On-Call Engineer

Rollback Summary:
- Level: <1/2/3>
- Reason: <DETAILED REASON>
- Duration: <START> to <END> (<DURATION>)
- Downtime: <DOWNTIME>
- Data Loss: <YES/NO> (<DETAILS>)
- Recovery: <SUCCESS/PARTIAL/FAILED>

Root Cause Analysis:
- <CAUSE 1>
- <CAUSE 2>

Lessons Learned:
- <LESSON 1>
- <LESSON 2>

Action Items:
- <ACTION ITEM 1> (Owner: <NAME>, Due: <DATE>)
- <ACTION ITEM 2> (Owner: <NAME>, Due: <DATE>)

Attachments:
- Rollback logs
- Grafana dashboard screenshots
- Database backup manifest

6.2 External Communication (if applicable)

Customer Notification (if trading impacted):

Subject: Service Disruption - Wave D Rollback

Dear Valued Customer,

We experienced a technical issue with our Wave D deployment that required
a rollback to our previous stable version (Wave C). Your trading service
was temporarily unavailable from <START> to <END> (<DURATION>).

Impact:
- Feature count: 225 → 201 (regime detection features temporarily disabled)
- Performance: No impact (Wave C baseline maintained)
- Data: No data loss (all positions and orders preserved)

Current Status: RESOLVED (as of <TIMESTAMP>)

Next Steps:
- We are investigating the root cause
- Wave D will be redeployed after fixes are validated
- No action required on your part

We apologize for any inconvenience. If you have questions, please contact
support@foxhunt.com or call 1-800-FOXHUNT.

Sincerely,
Foxhunt Engineering Team

---

7. Post-Rollback Actions

7.1 Immediate Actions (within 1 hour)

1. Document Rollback:

   • Reason for rollback
   • Level executed (1/2/3)
   • Downtime duration
   • Data loss (if any)
   • Recovery steps

2. Notify Stakeholders:

   • Engineering team
   • DevOps
   • Management
   • Customers (if impacted)

3. Preserve Evidence:

   • Save logs from all services
   • Export Grafana dashboards
   • Backup error state database
   • Screenshot Prometheus alerts

4. Verify System Stability:

   • Monitor for 1 hour
   • Check all health endpoints
   • Verify trading functionality
   • Confirm no residual errors

7.2 Root Cause Analysis (within 24 hours)

1. Analyze Logs:

   • Identify error patterns
   • Trace error sequences
   • Correlate with metrics

2. Reproduce Issue (in staging):

   • Deploy Wave D to staging
   • Trigger rollback condition
   • Verify rollback procedure

3. Document RCA:

   • Root cause
   • Contributing factors
   • Timeline of events
   • Impact assessment

7.3 Prevention & Remediation (within 1 week)

1. Fix Root Cause:

   • Implement fix in code
   • Add regression tests
   • Validate fix in staging

2. Update Rollback Procedures:

   • Document lessons learned
   • Improve rollback automation
   • Update runbooks

3. Deploy Fix:

   • Redeploy Wave D with fix
   • Monitor for 24 hours
   • Verify issue resolved

---

8. Rollback Automation (Future Enhancement)

8.1 Automated Rollback Triggers

Proposed: Implement automated rollback for P0 CRITICAL triggers.

# Pseudo-code for automated rollback
def monitor_wave_d_health():
    if flip_flop_rate > 50_per_hour:
        trigger_rollback(level=1, reason="flip_flopping")
    elif false_positive_rate > 0.8:
        trigger_rollback(level=1, reason="false_positives")
    elif nan_inf_count > 0:
        trigger_rollback(level=1, reason="nan_inf_features")
    elif downtime > 5_minutes:
        trigger_rollback(level=3, reason="system_unavailable")

8.2 Rollback Dashboard

Proposed: Create Grafana dashboard for rollback monitoring.

Metrics:

• Rollback trigger thresholds (flip-flop rate, false positive rate, NaN/Inf count)
• Rollback history (timestamp, level, reason, duration)
• System recovery time (time to stable state)
• Data loss (bytes lost per rollback)

---

9. Appendix

9.1 Rollback Commands Quick Reference

Level 1 (Feature Toggle):

export ENABLE_WAVE_D_FEATURES=false
systemctl restart trading_service
curl http://localhost:50052/health | jq '.feature_count'  # Expect: 201

Level 2 (Database Rollback):

pg_dump -t regime_states -t regime_transitions -t adaptive_strategy_metrics > backup.sql
cargo sqlx migrate run  # After creating 046_rollback_wave_d.sql
systemctl restart trading_service

Level 3 (Full Rollback):

git checkout 1309eb7c  # Wave C commit
cargo build --release --workspace
cargo sqlx migrate run  # After creating 046_rollback_wave_d.sql
systemctl restart trading_service api_gateway backtesting_service

9.2 Emergency Contacts

Role	Name	Phone	Email
On-Call Engineer	TBD	TBD	oncall@foxhunt.com
DevOps Lead	TBD	TBD	devops@foxhunt.com
Engineering Manager	TBD	TBD	engineering@foxhunt.com
CTO	TBD	TBD	cto@foxhunt.com

9.3 Rollback Logs Location

• Service Logs: /var/log/foxhunt/<service>.log
• Rollback Logs: /var/log/foxhunt/rollback_<timestamp>.log
• Database Backups: /backup/foxhunt_backup_<timestamp>.sql
• Configuration Backups: /backup/config_backup_<timestamp>/

---

Document Created By: Agent G24 Date: 2025-10-18 Status: 🟡 PENDING TESTING (rollback procedures not yet validated) Next Steps: Test all 3 rollback levels in staging environment before production deployment