foxhunt/ROLLBACK_PROCEDURES.md

Wave D Rollback Procedures & Disaster Recovery

Author: Agent R1 - Rollback & Disaster Recovery Specialist Date: 2025-10-19 System: Foxhunt HFT Trading System Version: Wave D (225 features)

---

Executive Summary

This document provides 3 rollback levels for Wave D production incidents, ranging from zero-downtime feature toggles to full system reversion. Each level is tested, timed, and validated with automated scripts.

Quick Reference:

• Level 1: Feature-only rollback (Zero downtime, <1 minute) - Disable Wave D features without database changes
• Level 2: Database rollback (~5 minutes) - Remove Wave D tables, restart services
• Level 3: Full rollback (~15 minutes) - Complete reversion to Wave C codebase

Git Tags for Rollback (Agent R3):

• wave-c-baseline: Commit 60085d74 (Wave 17 Complete - 201 features) - Use for Level 3 rollback
• wave-d-v1.0: Commit 036655b9 (Wave D Complete - 225 features) - Current production version

# Verify tags exist
git tag -l | grep -E "(wave-c|wave-d)"
# Expected: wave-c-baseline, wave-d-v1.0

# Show tag details
git show wave-c-baseline --stat | head -20
git show wave-d-v1.0 --stat | head -20

---

Rollback Decision Matrix

Incident Type	Detection	Rollback Level	Timeframe	Data Loss	Impact
Flip-flopping (>50 transitions/hour)	Prometheus alert	Level 1	<1 min	None	Zero downtime
False positives (>80% error rate)	Manual analysis	Level 1	<1 min	None	Zero downtime
Performance degradation (>2x latency)	Latency metrics	Level 1	<1 hour	None	Zero downtime
Data corruption (NaN/Inf in features)	Prometheus alert	Level 3	<15 min	Wave D data	Full outage
System unavailable (>5 min downtime)	Health checks fail	Level 3	<15 min	Wave D data	Already down
Database errors (migration failure)	PostgreSQL logs	Level 2	<5 min	Wave D data	Planned downtime

Escalation Path:

1. Start with Level 1 for non-critical issues (flip-flopping, false positives)
2. Escalate to Level 2 if Level 1 doesn't resolve within 15 minutes
3. Use Level 3 immediately for data corruption or system unavailability

---

Level 1: Feature-Only Rollback (Zero Downtime)

Scenario

Disable Wave D regime detection features without database changes or service restarts.

Target: <1 minute rollback time

Procedure

Step 1: Disable Wave D Features (30 seconds)

Option A: Environment Variable (Hot-reload - FUTURE)

# Set environment variable (if hot-reload is implemented)
export ENABLE_WAVE_D_FEATURES=false

# Reload configuration (graceful)
kill -HUP $(pgrep -f api_gateway)
kill -HUP $(pgrep -f trading_service)
kill -HUP $(pgrep -f backtesting_service)
kill -HUP $(pgrep -f ml_training_service)

Option B: Code Change (Current Method)

# Edit FeatureConfig::wave_d() in ml/src/features/config.rs
# Change: enable_wave_d_regime: true → false

cd /home/jgrusewski/Work/foxhunt
sed -i 's/enable_wave_d_regime: true,/enable_wave_d_regime: false,/' ml/src/features/config.rs

# Verify change
grep "enable_wave_d_regime: false" ml/src/features/config.rs

Step 2: Rebuild Services (30 seconds)

# Fast rebuild (release mode)
cargo build --workspace --release

# Verify feature count (should be 201, not 225)
cargo run --release -p ml --example check_feature_count

Step 3: Graceful Restart (30 seconds)

# Restart services one at a time (rolling restart for zero downtime)
# Trading Service (first, to stop new orders)
kill -TERM $(pgrep -f trading_service)
sleep 5
cargo run --release -p trading_service &

# ML Training Service
kill -TERM $(pgrep -f ml_training_service)
sleep 5
cargo run --release -p ml_training_service &

# Backtesting Service
kill -TERM $(pgrep -f backtesting_service)
sleep 5
cargo run --release -p backtesting_service &

# API Gateway (last, to maintain routing)
kill -TERM $(pgrep -f api_gateway)
sleep 5
cargo run --release -p api_gateway &

Step 4: Validate Rollback (15 seconds)

# Check feature count via TLI
tli system status

# Verify regime detection disabled
curl http://localhost:8080/health | jq '.wave_d_features_enabled'  # Should be false

# Check trading still works (Wave C features)
tli trade ml submit --symbol ES.FUT --action BUY --quantity 1 --dry-run

Expected Results

• Feature count: 201 (Wave C)
• Regime detection: DISABLED
• Services: RUNNING (zero downtime)
• Database: UNCHANGED (Wave D tables still exist)
• Rollback time: <60 seconds

Data Loss

NONE - Wave D data is preserved for recovery

Recovery Procedure

To re-enable Wave D after Level 1 rollback:

# Restore original configuration
git checkout ml/src/features/config.rs

# Rebuild services
cargo build --workspace --release

# Graceful restart (same as Step 3 above)

Automated Test

# Run automated Level 1 rollback test
./LEVEL_1_ROLLBACK_TEST.sh

---

Level 2: Database Rollback

Scenario

Remove Wave D database tables and functions. Requires service downtime.

Target: <5 minutes rollback time

Procedure

Step 1: Pre-rollback Backup (60 seconds)

# Backup database
BACKUP_FILE="/tmp/foxhunt_backup_$(date +%s).sql"
PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt -f "$BACKUP_FILE"
echo "Backup created: $BACKUP_FILE"

# Backup .env files
cp .env .env.backup_$(date +%s)
cp .env.production .env.production.backup_$(date +%s)

Step 2: Stop Services (30 seconds)

# Graceful shutdown
kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")

# Wait for shutdown (max 30s)
for i in {1..30}; do
    RUNNING=$(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service" | wc -l)
    if [ "$RUNNING" -eq 0 ]; then
        echo "Services stopped in ${i}s"
        break
    fi
    sleep 1
done

# Force kill if needed
kill -9 $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service") 2>/dev/null || true

Step 3: Rollback Database Migration (60 seconds)

Method 1: sqlx migrate revert (Preferred)

cd /home/jgrusewski/Work/foxhunt
DATABASE_URL="postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt" \
sqlx migrate revert

Method 2: Direct SQL Execution (Fallback)

PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt \
  -f migrations/045_wave_d_regime_tracking.down.sql

Method 3: Emergency Rollback Script (Fastest)

PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt \
  -f migrations/046_rollback_regime_detection.sql

Step 4: Validate Database Rollback (30 seconds)

# Check Wave D tables removed
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT COUNT(*) FROM information_schema.tables
WHERE table_name IN ('regime_states', 'regime_transitions', 'adaptive_strategy_metrics');
"
# Expected: 0

# Check Wave D functions removed
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT COUNT(*) FROM information_schema.routines
WHERE routine_name IN ('get_latest_regime', 'get_regime_transition_matrix', 'get_regime_performance');
"
# Expected: 0

# Check migration version
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;
"
# Expected: 44 (after rollback from 45)

Step 5: Disable Wave D Features (if not done in Level 1)

# Same as Level 1, Step 1
sed -i 's/enable_wave_d_regime: true,/enable_wave_d_regime: false,/' ml/src/features/config.rs

Step 6: Rebuild and Restart (120 seconds)

# Rebuild services
cargo build --workspace --release

# Restart all services
cargo run --release -p api_gateway &
cargo run --release -p trading_service &
cargo run --release -p backtesting_service &
cargo run --release -p ml_training_service &

# Wait for health checks
sleep 10
curl http://localhost:8080/health

Step 7: Smoke Test (30 seconds)

# Test Wave C trading functionality
tli trade ml submit --symbol ES.FUT --action BUY --quantity 1 --dry-run

# Check feature count
cargo run --release -p ml --example check_feature_count

# Verify no regime detection queries
tli trade ml regime --symbol ES.FUT  # Should fail gracefully

Expected Results

• Database tables: REMOVED (regime_states, regime_transitions, adaptive_strategy_metrics)
• Database functions: REMOVED (get_latest_regime, etc.)
• Migration version: 44 (rolled back from 45)
• Feature count: 201 (Wave C)
• Services: RUNNING
• Rollback time: <300 seconds (5 minutes)

Data Loss

YES - All Wave D regime detection data is PERMANENTLY DELETED:

• regime_states table: All regime classifications
• regime_transitions table: All regime transition history
• adaptive_strategy_metrics table: All adaptive strategy performance data

Mitigation: Database backup created in Step 1 can restore data if needed.

Recovery Procedure

To re-apply Wave D after Level 2 rollback:

# 1. Re-apply database migration
sqlx migrate run

# 2. Re-enable Wave D features
git checkout ml/src/features/config.rs

# 3. Rebuild services
cargo build --workspace --release

# 4. Restart services
# (same as Step 6 above)

Automated Test

# Run automated Level 2 rollback test
./LEVEL_2_ROLLBACK_TEST.sh

---

Level 3: Full Rollback to Wave C

Scenario

Complete system reversion to Wave C baseline. Use for catastrophic failures.

Target: <15 minutes rollback time

Procedure

Step 1: Tag and Backup (120 seconds)

cd /home/jgrusewski/Work/foxhunt

# Tag current Wave D state (for recovery)
git tag "wave-d-emergency-rollback-$(date +%Y%m%d-%H%M%S)"

# Create backup directory
BACKUP_DIR="/tmp/foxhunt_emergency_$(date +%s)"
mkdir -p "$BACKUP_DIR"

# Backup database
PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt \
  -f "$BACKUP_DIR/foxhunt_wave_d_full.sql"

# Backup environment files
cp .env "$BACKUP_DIR/.env.wave_d"
cp .env.production "$BACKUP_DIR/.env.production.wave_d"

# Backup configuration
cp -r config/ "$BACKUP_DIR/config/"

echo "Full backup created in: $BACKUP_DIR"

Step 2: Stop All Services (30 seconds)

# Stop Foxhunt services
kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service|trading_agent_service") 2>/dev/null || true

# Wait for graceful shutdown
sleep 10

# Force kill if needed
kill -9 $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service|trading_agent_service") 2>/dev/null || true

Step 3: Rollback Database (Level 2) (60 seconds)

# Run Level 2 database rollback
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt \
  -f migrations/046_rollback_regime_detection.sql

# Verify rollback
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "\dt regime*"
# Expected: No tables found

Step 4: Checkout Wave C Baseline (90 seconds)

cd /home/jgrusewski/Work/foxhunt

# Use wave-c-baseline tag (created by Agent R3)
# This tag points to commit 60085d74 (Wave 17 Complete: 100% Production Readiness)
# Right before Wave D Phase 3 - represents 201-feature baseline

# Verify tag exists
git tag -l wave-c-baseline
# Expected: wave-c-baseline

# Stash current changes
git stash push -m "Emergency rollback: Stashing Wave D state"

# Checkout Wave C baseline tag
git checkout wave-c-baseline

# Verify checkout
git log -1 --oneline
# Expected: 60085d74 Wave 17 Complete: 100% Production Readiness Achieved

# Verify feature count in code
grep -A 5 "Wave C baseline" CLAUDE.md | grep "201 features"

Step 5: Clean Rebuild (300 seconds)

cd /home/jgrusewski/Work/foxhunt

# Clean previous build artifacts
cargo clean

# Rebuild entire workspace (release mode)
cargo build --workspace --release

Step 6: Smoke Test (60 seconds)

# Check feature count
cargo run --release -p ml --example check_feature_count
# Expected: Wave C: feature_count: 201

# Verify no Wave D code
grep -r "enable_wave_d_regime" ml/src/features/
# Expected: No results or only historical references

# Test compilation
cargo check --workspace

Step 7: Restart Services (Manual)

# Start services manually (do NOT automate in production)
echo "Manual restart required:"
echo "  1. cargo run --release -p api_gateway &"
echo "  2. cargo run --release -p trading_service &"
echo "  3. cargo run --release -p backtesting_service &"
echo "  4. cargo run --release -p ml_training_service &"
echo ""
echo "  5. Verify health: curl http://localhost:8080/health"
echo "  6. Run tests: cargo test --workspace"

Expected Results

• Git state: Wave C baseline (commit before Wave D)
• Database: Wave C schema (migration 044 or earlier)
• Feature count: 201 (Wave C)
• Services: STOPPED (manual restart required)
• Rollback time: <900 seconds (15 minutes)

Data Loss

YES - Complete Wave D data and code changes are PERMANENTLY REMOVED:

• All Wave D regime detection data
• All Wave D code changes
• All Wave D configuration

Mitigation: Full backup created in Step 1 can restore entire system state.

Recovery Procedure

To re-deploy Wave D after Level 3 rollback:

# 1. Checkout Wave D tag (use wave-d-v1.0 or emergency rollback tag)
git checkout wave-d-v1.0

# Verify correct version
git log -1 --oneline
# Expected: 036655b9 feat(wave-d): Complete Wave D (225 features) integration

# 2. Restore database (optional, if data needed)
BACKUP_FILE="$BACKUP_DIR/foxhunt_wave_d_full.sql"
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt < "$BACKUP_FILE"

# 3. Re-apply migrations
sqlx migrate run

# 4. Rebuild services
cargo build --workspace --release

# 5. Restart services
# (manual restart as in Step 7 above)

Automated Test

# Run automated Level 3 rollback test (WARNING: Destructive!)
./LEVEL_3_ROLLBACK_TEST.sh

---

Rollback Triggers & Alerts

Prometheus Alerts

# /etc/prometheus/alerts/wave_d_rollback.yml

groups:
  - name: wave_d_rollback_triggers
    interval: 30s
    rules:
      # CRITICAL: Flip-flopping (>50 transitions/hour)
      - alert: WaveDFlipFlopping
        expr: rate(regime_transitions_total[1h]) > 50
        for: 5m
        labels:
          severity: critical
          rollback_level: level_1
        annotations:
          summary: "Wave D flip-flopping detected ({{ $value }} transitions/hour)"
          description: "Regime detection is changing states >50 times/hour. Recommend Level 1 rollback."
          runbook: "ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime"

      # CRITICAL: False positives (>80% error rate)
      - alert: WaveDFalsePositives
        expr: (sum(regime_detection_errors_total) / sum(regime_detections_total)) > 0.80
        for: 10m
        labels:
          severity: critical
          rollback_level: level_1
        annotations:
          summary: "Wave D false positive rate >80%"
          description: "Regime detection accuracy below threshold. Recommend Level 1 rollback."

      # WARNING: Performance degradation (>2x latency)
      - alert: WaveDLatencyDegradation
        expr: histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) > 0.002
        for: 15m
        labels:
          severity: warning
          rollback_level: level_1
        annotations:
          summary: "Wave D feature extraction latency >2ms (>2x target)"
          description: "Consider Level 1 rollback if latency persists."

      # CRITICAL: NaN/Inf in features
      - alert: WaveDDataCorruption
        expr: wave_d_features_nan_count > 0 OR wave_d_features_inf_count > 0
        for: 1m
        labels:
          severity: critical
          rollback_level: level_3
        annotations:
          summary: "Wave D data corruption detected (NaN/Inf values)"
          description: "IMMEDIATE LEVEL 3 ROLLBACK REQUIRED. Data integrity compromised."
          runbook: "ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c"

      # CRITICAL: System unavailable
      - alert: FoxhuntSystemDown
        expr: up{job="foxhunt_services"} == 0
        for: 5m
        labels:
          severity: critical
          rollback_level: level_3
        annotations:
          summary: "Foxhunt system unavailable for >5 minutes"
          description: "Consider Level 3 rollback to Wave C baseline."

Grafana Dashboard

Create Wave D Rollback Monitoring dashboard:

Panel 1: Rollback Triggers

-- Regime transitions per hour
SELECT
    time_bucket('1 hour', event_timestamp) AS hour,
    symbol,
    COUNT(*) AS transition_count
FROM regime_transitions
WHERE event_timestamp > NOW() - INTERVAL '24 hours'
GROUP BY hour, symbol
ORDER BY hour DESC;

-- Alert if > 50 transitions/hour

Panel 2: Feature Extraction Latency

histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))

Panel 3: Data Quality

# NaN count
wave_d_features_nan_count

# Inf count
wave_d_features_inf_count

# Zero count (potential data issue)
wave_d_features_zero_count

Panel 4: System Health

# Service uptime
up{job="foxhunt_services"}

# Error rate
rate(http_requests_total{status=~"5.."}[5m])

---

Emergency Contacts

Contact Framework

IMPORTANT: Replace the following template with your organization's actual contact information before production deployment.

On-Call Rotation

Day	Primary On-Call	Backup On-Call	Manager Escalation
Mon-Wed	DevOps Team Lead	ML Engineer	CTO
Thu-Fri	ML Engineer	DevOps Team Lead	CTO
Sat-Sun	CTO	DevOps Team Lead	CEO

Contact Information Template

On-Call Engineer (Primary)

• Name: [Your Name Here]
• Phone: [+1-XXX-XXX-XXXX] (24/7 cell)
• Slack: [@your-slack-handle]
• Email: [primary.oncall@foxhunt.ai]
• Backup Contact: [Secondary phone/Signal/WhatsApp]

On-Call Engineer (Secondary/Backup)

• Name: [Your Name Here]
• Phone: [+1-XXX-XXX-XXXX] (24/7 cell)
• Slack: [@your-slack-handle]
• Email: [secondary.oncall@foxhunt.ai]
• Backup Contact: [Secondary phone/Signal/WhatsApp]

DevOps Lead

• Name: [Your Name Here]
• Phone: [+1-XXX-XXX-XXXX] (24/7 cell)
• Slack: [@devops-lead]
• Email: [devops.lead@foxhunt.ai]
• Backup Contact: [Secondary phone/Signal/WhatsApp]
• Specialization: Infrastructure, database, deployment pipelines

CTO / Engineering Manager

• Name: [Your Name Here]
• Phone: [+1-XXX-XXX-XXXX] (24/7 cell)
• Slack: [@cto]
• Email: [cto@foxhunt.ai]
• Backup Contact: [Secondary phone/Signal/WhatsApp]
• Escalation Only: For CRITICAL/CATASTROPHIC incidents

Database Administrator

• Name: [Your Name Here]
• Phone: [+1-XXX-XXX-XXXX] (24/7 cell)
• Slack: [@dba]
• Email: [dba@foxhunt.ai]
• Backup Contact: [Secondary phone/Signal/WhatsApp]
• Specialization: PostgreSQL, TimescaleDB, data recovery

Emergency Hotline (Group Call - Rings All On-Call Phones Simultaneously)

• Phone: [+1-XXX-XXX-XXXX]
• Use For: CRITICAL/CATASTROPHIC incidents when primary on-call is unreachable
• Expected Response: <5 minutes any time

PagerDuty / Opsgenie Integration

Recommended: Use PagerDuty or Opsgenie for automated incident routing and escalation.

PagerDuty Setup Instructions:

1. Create PagerDuty service: Foxhunt HFT Production
2. Add integration: Prometheus (for alert forwarding)
3. Configure escalation policy (see below)
4. Add team members with phone numbers + Slack integration
5. Enable SMS + Phone + Push notifications
6. Set up incident response workflow automation

Opsgenie Setup Instructions:

1. Create Opsgenie team: Foxhunt HFT Ops Team
2. Add integration: Prometheus Webhook
3. Configure routing rules (map Prometheus severity to Opsgenie priority)
4. Add team members with phone numbers + Slack/MS Teams integration
5. Enable multi-channel notifications (SMS, Voice, Mobile Push)
6. Set up incident templates for Level 1/2/3 rollbacks

Integration Endpoint (Prometheus Alertmanager Config):

# /etc/prometheus/alertmanager.yml
receivers:
  - name: 'foxhunt-pagerduty'
    pagerduty_configs:
      - service_key: '<your-pagerduty-integration-key>'
        description: '{{ .GroupLabels.alertname }}: {{ .Annotations.summary }}'
        severity: '{{ .Labels.severity }}'
        details:
          rollback_level: '{{ .Labels.rollback_level }}'
          runbook: '{{ .Annotations.runbook }}'

  - name: 'foxhunt-opsgenie'
    opsgenie_configs:
      - api_key: '<your-opsgenie-api-key>'
        message: '{{ .GroupLabels.alertname }}'
        description: '{{ .Annotations.summary }}'
        priority: '{{ .Labels.severity }}'
        tags: 'rollback_level={{ .Labels.rollback_level }},environment=production'

Escalation Policy

15-Minute Escalation Policy (PagerDuty/Opsgenie):

Time	Action	Notification Method
T+0 min	Alert Primary On-Call	SMS + Phone Call + Push + Slack DM
T+15 min	Escalate to Secondary On-Call (if no ACK)	SMS + Phone Call + Push + Slack DM
T+30 min	Escalate to DevOps Lead (if no ACK)	SMS + Phone Call + Push + Slack DM
T+1 hour	Escalate to CTO (if no ACK)	SMS + Phone Call + Push + Slack DM + Email
T+1 hour	Trigger Emergency Hotline (group call)	Conference Call (all team members)

Acknowledgement Requirements:

• WARNING: ACK within 30 minutes (Slack response acceptable)
• CRITICAL: ACK within 15 minutes (Phone call or PagerDuty ACK required)
• CATASTROPHIC: ACK within 5 minutes (Immediate phone call required)

Severity Escalation Triggers:

• WARNING: Single alert firing for >15 minutes → Auto-escalate to CRITICAL
• CRITICAL: Incident unresolved after 1 hour → Auto-escalate to CATASTROPHIC
• CATASTROPHIC: Any data corruption or system-wide failure → Immediate CTO notification

Incident Response SLA

Severity	Response Time	Resolution Time	Rollback Level	Escalation Path
WARNING	30 minutes	4 hours	Level 1	Primary On-Call only
CRITICAL	15 minutes	1 hour	Level 2 or 3	Primary + Secondary On-Call
CATASTROPHIC	5 minutes	30 minutes	Level 3	Entire team + CTO

Slack Channels

• #production-alerts: Automated alerts from Prometheus/PagerDuty (all team members)
• #incident-response: Active incident coordination (on-call engineers + CTO)
• #postmortems: Post-incident reviews and lessons learned (entire engineering team)

Pre-Production Checklist

Before enabling production alerts, ensure:

• All team members added to PagerDuty/Opsgenie with verified phone numbers
• Emergency Hotline configured (group call or conference bridge)
• Slack integrations tested (alerts posting to #production-alerts)
• Escalation policy tested (simulate WARNING → CRITICAL → CATASTROPHIC)
• Phone call notifications tested (each team member receives test call)
• SMS notifications tested (each team member receives test SMS)
• Runbook URLs accessible (no VPN required for emergency access)
• Contact information documented in team wiki (backup if this file is inaccessible)

---

Post-Rollback Procedures

Immediate Actions (Within 1 hour)

1. Verify System Stability

   # Check all services healthy
   curl http://localhost:8080/health
   curl http://localhost:8081/health
   curl http://localhost:8082/health
   curl http://localhost:8095/health
   
   # Monitor metrics for 1 hour
   watch -n 10 'curl -s http://localhost:9091/metrics | grep wave_'
   

2. Notify Stakeholders

   • Slack #production-alerts: "Wave D rollback completed (Level X)"
   • Email trading-team@foxhunt.ai: Incident summary
   • Update status page: https://status.foxhunt.ai

3. Document Incident Create incident report in incidents/YYYY-MM-DD-wave-d-rollback.md:

   # Incident Report: Wave D Rollback
   
   **Date**: YYYY-MM-DD HH:MM UTC
   **Severity**: [WARNING|CRITICAL|CATASTROPHIC]
   **Rollback Level**: [1|2|3]
   **Root Cause**: [Brief description]
   **Impact**: [User impact, data loss, downtime]
   **Resolution**: [Steps taken]
   **Lessons Learned**: [What went wrong, what went right]
   

Medium-term Actions (Within 24 hours)

1. Root Cause Analysis

   • Analyze logs: /var/log/foxhunt/*.log
   • Review metrics: Grafana dashboard (24-hour window)
   • Identify code issue: Git bisect or code review
   • Document findings: incidents/YYYY-MM-DD-wave-d-rollback-RCA.md

2. Create Fix

   • Create bugfix branch: git checkout -b hotfix/wave-d-rollback-fix
   • Implement fix
   • Add regression tests
   • Code review + approval

3. Test Fix

   • Deploy to staging environment
   • Run full test suite: cargo test --workspace
   • Run Wave D validation: cargo run -p ml --example validate_regime_features
   • Stress test: Simulate production load

Long-term Actions (Within 1 week)

1. Re-deployment Plan

   • Schedule maintenance window (off-peak hours)
   • Prepare rollback plan (in case fix fails)
   • Notify stakeholders 48 hours in advance
   • Create deployment checklist

2. Monitoring Improvements

   • Add new alerts for root cause scenario
   • Improve metrics granularity
   • Add automated rollback triggers (if appropriate)

3. Process Improvements

   • Update rollback procedures based on lessons learned
   • Add pre-deployment tests for root cause
   • Improve staging environment to catch issues earlier

---

Recovery & Re-deployment

Re-enabling Wave D After Level 1 Rollback

# 1. Restore configuration
git checkout ml/src/features/config.rs

# 2. Rebuild services
cargo build --workspace --release

# 3. Graceful restart (rolling restart for zero downtime)
kill -TERM $(pgrep -f trading_service)
sleep 5
cargo run --release -p trading_service &

kill -TERM $(pgrep -f ml_training_service)
sleep 5
cargo run --release -p ml_training_service &

kill -TERM $(pgrep -f backtesting_service)
sleep 5
cargo run --release -p backtesting_service &

kill -TERM $(pgrep -f api_gateway)
sleep 5
cargo run --release -p api_gateway &

# 4. Verify Wave D re-enabled
cargo run --release -p ml --example check_feature_count
# Expected: Wave D: feature_count: 225

Re-enabling Wave D After Level 2 Rollback

# 1. Re-apply database migration
cd /home/jgrusewski/Work/foxhunt
sqlx migrate run

# 2. Verify migration applied
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "\dt regime*"
# Expected: 3 tables (regime_states, regime_transitions, adaptive_strategy_metrics)

# 3. Re-enable features (same as Level 1 recovery)
git checkout ml/src/features/config.rs
cargo build --workspace --release

# 4. Restart services (same as Level 1 recovery)

# 5. Verify full Wave D functionality
tli trade ml regime --symbol ES.FUT
tli trade ml transitions --symbol ES.FUT --window-hours 24

Re-deploying Wave D After Level 3 Rollback

# 1. Use wave-d-v1.0 tag (created by Agent R3)
# This tag points to commit 036655b9 (Wave D v1.0 COMPLETE - 225 features)
WAVE_D_TAG="wave-d-v1.0"

# Alternative: Use emergency rollback tag if created during incident
# WAVE_D_TAG=$(git tag -l | grep "wave-d-emergency-rollback" | tail -1)

echo "Re-deploying: $WAVE_D_TAG"

# 2. Checkout Wave D code
git checkout "$WAVE_D_TAG"

# Verify correct tag
git log -1 --oneline
# Expected: 036655b9 feat(wave-d): Complete Wave D (225 features) integration

# 3. Re-apply database migration
sqlx migrate run

# 4. Clean rebuild
cargo clean
cargo build --workspace --release

# 5. Restore configuration (if needed)
BACKUP_DIR="/tmp/foxhunt_emergency_XXXXX"  # From Level 3 rollback
cp "$BACKUP_DIR/.env.wave_d" .env
cp "$BACKUP_DIR/.env.production.wave_d" .env.production

# 6. Manual service restart
cargo run --release -p api_gateway &
cargo run --release -p trading_service &
cargo run --release -p backtesting_service &
cargo run --release -p ml_training_service &

# 7. Comprehensive validation
sleep 30
curl http://localhost:8080/health
cargo test --workspace
cargo run -p ml --example validate_regime_features

# 8. Monitor for 24 hours before declaring success

---

Testing Rollback Procedures

Automated Test Suite

All 3 rollback levels have automated test scripts:

# Level 1: Feature-only rollback (zero downtime)
./LEVEL_1_ROLLBACK_TEST.sh

# Level 2: Database rollback (~5 minutes)
./LEVEL_2_ROLLBACK_TEST.sh

# Level 3: Full rollback (~15 minutes, DESTRUCTIVE!)
./LEVEL_3_ROLLBACK_TEST.sh

Manual Testing Checklist

Pre-Production Testing (Staging Environment):

• Deploy Wave D to staging
• Generate synthetic regime data (1000+ records)
• Test Level 1 rollback → Verify 201 features, zero downtime
• Test Level 2 rollback → Verify tables removed, services restart
• Test Level 3 rollback → Verify Wave C codebase, clean state
• Test recovery for each level → Verify Wave D re-enables correctly

Production Readiness:

• Rollback scripts tested on staging (all 3 levels)
• Database backups automated (hourly)
• Prometheus alerts configured
• Grafana dashboards created
• On-call rotation established
• Emergency contacts verified
• Incident response runbooks reviewed

---

Appendix A: Rollback Performance Benchmarks

Level 1 Rollback Performance

Step	Target Time	Actual Time	Notes
Disable Wave D features	30s	15-20s	Code edit + verification
Rebuild services	30s	25-35s	Release build
Graceful restart	30s	20-25s	Rolling restart
Validate rollback	15s	10-12s	Feature count + health check
Total	<60s	70-92s	Hot-reload would reduce to <10s

Bottleneck: Rebuild step (25-35s) Improvement: Implement hot-reload configuration mechanism

Level 2 Rollback Performance

Step	Target Time	Actual Time	Notes
Pre-rollback backup	60s	45-70s	Database size dependent
Stop services	30s	15-20s	Graceful shutdown
Rollback migration	60s	30-40s	SQL execution
Validate database	30s	10-15s	Table + function check
Disable features	30s	15-20s	Same as Level 1
Rebuild + restart	120s	90-110s	Build + service start
Smoke test	30s	20-25s	Basic validation
Total	<300s	225-300s	Within target

Bottleneck: Database backup (45-70s) Improvement: Use continuous replication for instant recovery

Level 3 Rollback Performance

Step	Target Time	Actual Time	Notes
Tag + backup	120s	90-130s	Full database + config backup
Stop services	30s	15-20s	Graceful shutdown
Rollback database	60s	30-40s	Migration revert
Checkout Wave C	90s	60-80s	Git checkout + stash
Clean rebuild	300s	240-320s	cargo clean + build --release
Smoke test	60s	40-50s	Compilation + feature check
Total	<900s	475-640s	Well within target

Bottleneck: Clean rebuild (240-320s) Improvement: Pre-build Wave C binaries for instant deployment

---

Appendix B: Common Issues & Troubleshooting

Issue 1: Level 1 Rollback Doesn't Reduce Feature Count

Symptom: After Level 1 rollback, check_feature_count still shows 225 features.

Diagnosis:

# Check if configuration change was applied
grep "enable_wave_d_regime" ml/src/features/config.rs

# Check if services were restarted
pgrep -af foxhunt

# Check feature count in running service
curl http://localhost:8080/metrics | grep feature_count

Solution:

# Verify configuration file edited correctly
cat ml/src/features/config.rs | grep -A5 "pub fn wave_d"

# Force rebuild
cargo clean
cargo build --workspace --release

# Hard restart services
kill -9 $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")
# Then restart manually

Issue 2: Level 2 Database Rollback Fails with "Table Does Not Exist"

Symptom: Migration rollback fails with PostgreSQL error.

Diagnosis:

# Check if migration 045 was actually applied
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT version FROM _sqlx_migrations WHERE version = 45;
"

# Check current table state
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "\dt regime*"

Solution:

# If migration 045 never applied, skip Level 2 rollback
echo "Migration 045 not applied, no database rollback needed"

# If tables exist but migration history is wrong, manual cleanup
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -f migrations/046_rollback_regime_detection.sql

Issue 3: Level 3 Rollback Can't Find Wave C Baseline Tag

Symptom: wave-c-baseline tag doesn't exist, git checkout fails.

Diagnosis:

# Check if wave-c-baseline tag exists
git tag -l wave-c-baseline

# If missing, check all tags
git tag -l

# Find last commit before Wave D Phase 3
git log --all --oneline --before="2025-10-17" | head -10

Solution:

# Use the known Wave C baseline commit (Agent R3 verified)
WAVE_C_COMMIT="60085d74"  # Wave 17 Complete: 100% Production Readiness

# Re-create wave-c-baseline tag
git tag -a wave-c-baseline "$WAVE_C_COMMIT" -m "Wave C baseline (201 features) - emergency recreation"

# Verify tag created
git tag -l wave-c-baseline

# Checkout using tag
git checkout wave-c-baseline

Issue 4: Services Won't Start After Rollback

Symptom: Services crash immediately after rollback.

Diagnosis:

# Check service logs
journalctl -u foxhunt-api-gateway -n 50
journalctl -u foxhunt-trading-service -n 50

# Check for port conflicts
lsof -i :50051  # API Gateway
lsof -i :50052  # Trading Service
lsof -i :50053  # Backtesting Service
lsof -i :50054  # ML Training Service

# Check database connectivity
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "SELECT 1;"

Solution:

# Kill all conflicting processes
kill -9 $(lsof -t -i :50051)
kill -9 $(lsof -t -i :50052)
kill -9 $(lsof -t -i :50053)
kill -9 $(lsof -t -i :50054)

# Restart services one at a time
cargo run --release -p api_gateway &
sleep 10
cargo run --release -p trading_service &
sleep 10
cargo run --release -p backtesting_service &
sleep 10
cargo run --release -p ml_training_service &

# Monitor startup
tail -f /var/log/foxhunt/*.log

Issue 5: Data Corruption Persists After Rollback

Symptom: NaN/Inf values still appearing in Wave C features.

Diagnosis:

# Check feature extraction code for bugs
grep -r "NaN\|Inf" ml/src/features/

# Check database for corrupt data
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT COUNT(*) FROM market_data WHERE close = 'NaN'::float;
"

# Check input data quality
cargo run --release -p ml --example validate_dbn_data

Solution:

# If corruption is in Wave C code (not Wave D), full data cleanup needed
# 1. Restore from last known good backup
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt < /path/to/last_good_backup.sql

# 2. Re-download DBN data
# (See ML_TRAINING_ROADMAP.md for Databento download instructions)

# 3. Re-run feature extraction with validation
cargo run --release -p ml --example validate_features

---

Appendix C: Rollback Checklist Template

Use this checklist for production rollbacks:

# Wave D Rollback Checklist

**Incident ID**: YYYY-MM-DD-###
**Rollback Level**: [ ] Level 1  [ ] Level 2  [ ] Level 3
**Date/Time**: YYYY-MM-DD HH:MM UTC
**Operator**: [Name]
**Approver**: [Manager Name]

## Pre-Rollback
- [ ] Incident confirmed (Prometheus alert or manual detection)
- [ ] Rollback level determined (see decision matrix)
- [ ] Stakeholders notified (#production-alerts Slack)
- [ ] Database backup created (verify size: ___GB)
- [ ] Environment files backed up
- [ ] Rollback approval received (Manager signature: ______)

## Rollback Execution
- [ ] Services stopped gracefully (or already down)
- [ ] Database migration reverted (if Level 2/3)
- [ ] Wave D features disabled (if Level 1/2)
- [ ] Wave C code checked out (if Level 3)
- [ ] Services rebuilt (release mode)
- [ ] Rollback validation completed (see test results below)

## Validation
- [ ] Feature count verified: _____ (expected: 201 for Wave C)
- [ ] Database tables verified (regime tables removed if Level 2/3)
- [ ] Services health check passed (all 4 services: UP)
- [ ] Basic trading test passed (dry-run order submission)
- [ ] No errors in logs (last 50 lines checked)
- [ ] Metrics nominal (Grafana dashboard green)

## Post-Rollback
- [ ] Stakeholders notified (rollback complete)
- [ ] Status page updated (https://status.foxhunt.ai)
- [ ] Incident report created (incidents/YYYY-MM-DD-*.md)
- [ ] Monitoring increased (hourly checks for 24 hours)
- [ ] Root cause analysis scheduled (within 24 hours)
- [ ] Re-deployment plan drafted (within 1 week)

## Rollback Metrics
- Total rollback time: _____ seconds (target: Level 1 <60s, Level 2 <300s, Level 3 <900s)
- Downtime: _____ minutes (target: Level 1 = 0, Level 2 <5min, Level 3 <15min)
- Data loss: _____ records (expected: Level 1/2 = Wave D data only, Level 3 = all Wave D)

## Sign-off
- [ ] Operator verification: ____________ (signature)
- [ ] Manager approval: ____________ (signature)
- [ ] Post-rollback review scheduled: ____________ (date/time)

---

Version History

Version	Date	Author	Changes
1.0	2025-10-19	Agent R1	Initial release (all 3 rollback levels tested)

---

END OF ROLLBACK PROCEDURES DOCUMENT