foxhunt/ROLLBACK_PROCEDURES.md

# Wave D Rollback Procedures & Disaster Recovery
**Author**: Agent R1 - Rollback & Disaster Recovery Specialist
**Date**: 2025-10-19
**System**: Foxhunt HFT Trading System
**Version**: Wave D (225 features)

---

## Executive Summary

This document provides **3 rollback levels** for Wave D production incidents, ranging from zero-downtime feature toggles to full system reversion. Each level is tested, timed, and validated with automated scripts.

**Quick Reference:**
- **Level 1**: Feature-only rollback (Zero downtime, <1 minute) - Disable Wave D features without database changes
- **Level 2**: Database rollback (~5 minutes) - Remove Wave D tables, restart services
- **Level 3**: Full rollback (~15 minutes) - Complete reversion to Wave C codebase

**Git Tags for Rollback (Agent R3):**
- **wave-c-baseline**: Commit `60085d74` (Wave 17 Complete - 201 features) - Use for Level 3 rollback
- **wave-d-v1.0**: Commit `036655b9` (Wave D Complete - 225 features) - Current production version

```bash
# Verify tags exist
git tag -l | grep -E "(wave-c|wave-d)"
# Expected: wave-c-baseline, wave-d-v1.0

# Show tag details
git show wave-c-baseline --stat | head -20
git show wave-d-v1.0 --stat | head -20
```

---

## Rollback Decision Matrix

| Incident Type | Detection | Rollback Level | Timeframe | Data Loss | Impact |
|---------------|-----------|----------------|-----------|-----------|--------|
| **Flip-flopping** (>50 transitions/hour) | Prometheus alert | Level 1 | <1 min | None | Zero downtime |
| **False positives** (>80% error rate) | Manual analysis | Level 1 | <1 min | None | Zero downtime |
| **Performance degradation** (>2x latency) | Latency metrics | Level 1 | <1 hour | None | Zero downtime |
| **Data corruption** (NaN/Inf in features) | Prometheus alert | Level 3 | <15 min | Wave D data | Full outage |
| **System unavailable** (>5 min downtime) | Health checks fail | Level 3 | <15 min | Wave D data | Already down |
| **Database errors** (migration failure) | PostgreSQL logs | Level 2 | <5 min | Wave D data | Planned downtime |

**Escalation Path:**
1. Start with **Level 1** for non-critical issues (flip-flopping, false positives)
2. Escalate to **Level 2** if Level 1 doesn't resolve within 15 minutes
3. Use **Level 3** immediately for data corruption or system unavailability

---

## Level 1: Feature-Only Rollback (Zero Downtime)

### Scenario
Disable Wave D regime detection features without database changes or service restarts.

### Target: <1 minute rollback time

### Procedure

#### Step 1: Disable Wave D Features (30 seconds)

**Option A: Environment Variable (Hot-reload - FUTURE)**
```bash
# Set environment variable (if hot-reload is implemented)
export ENABLE_WAVE_D_FEATURES=false

# Reload configuration (graceful)
kill -HUP $(pgrep -f api_gateway)
kill -HUP $(pgrep -f trading_service)
kill -HUP $(pgrep -f backtesting_service)
kill -HUP $(pgrep -f ml_training_service)
```

**Option B: Code Change (Current Method)**
```bash
# Edit FeatureConfig::wave_d() in ml/src/features/config.rs
# Change: enable_wave_d_regime: true → false

cd /home/jgrusewski/Work/foxhunt
sed -i 's/enable_wave_d_regime: true,/enable_wave_d_regime: false,/' ml/src/features/config.rs

# Verify change
grep "enable_wave_d_regime: false" ml/src/features/config.rs
```

#### Step 2: Rebuild Services (30 seconds)

```bash
# Fast rebuild (release mode)
cargo build --workspace --release

# Verify feature count (should be 201, not 225)
cargo run --release -p ml --example check_feature_count
```

#### Step 3: Graceful Restart (30 seconds)

```bash
# Restart services one at a time (rolling restart for zero downtime)
# Trading Service (first, to stop new orders)
kill -TERM $(pgrep -f trading_service)
sleep 5
cargo run --release -p trading_service &

# ML Training Service
kill -TERM $(pgrep -f ml_training_service)
sleep 5
cargo run --release -p ml_training_service &

# Backtesting Service
kill -TERM $(pgrep -f backtesting_service)
sleep 5
cargo run --release -p backtesting_service &

# API Gateway (last, to maintain routing)
kill -TERM $(pgrep -f api_gateway)
sleep 5
cargo run --release -p api_gateway &
```

#### Step 4: Validate Rollback (15 seconds)

```bash
# Check feature count via TLI
tli system status

# Verify regime detection disabled
curl http://localhost:8080/health | jq '.wave_d_features_enabled'  # Should be false

# Check trading still works (Wave C features)
tli trade ml submit --symbol ES.FUT --action BUY --quantity 1 --dry-run
```

### Expected Results
- Feature count: **201** (Wave C)
- Regime detection: **DISABLED**
- Services: **RUNNING** (zero downtime)
- Database: **UNCHANGED** (Wave D tables still exist)
- Rollback time: **<60 seconds**

### Data Loss
**NONE** - Wave D data is preserved for recovery

### Recovery Procedure
To re-enable Wave D after Level 1 rollback:
```bash
# Restore original configuration
git checkout ml/src/features/config.rs

# Rebuild services
cargo build --workspace --release

# Graceful restart (same as Step 3 above)
```

### Automated Test
```bash
# Run automated Level 1 rollback test
./LEVEL_1_ROLLBACK_TEST.sh
```

---

## Level 2: Database Rollback

### Scenario
Remove Wave D database tables and functions. Requires service downtime.

### Target: <5 minutes rollback time

### Procedure

#### Step 1: Pre-rollback Backup (60 seconds)

```bash
# Backup database
BACKUP_FILE="/tmp/foxhunt_backup_$(date +%s).sql"
PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt -f "$BACKUP_FILE"
echo "Backup created: $BACKUP_FILE"

# Backup .env files
cp .env .env.backup_$(date +%s)
cp .env.production .env.production.backup_$(date +%s)
```

#### Step 2: Stop Services (30 seconds)

```bash
# Graceful shutdown
kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")

# Wait for shutdown (max 30s)
for i in {1..30}; do
    RUNNING=$(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service" | wc -l)
    if [ "$RUNNING" -eq 0 ]; then
        echo "Services stopped in ${i}s"
        break
    fi
    sleep 1
done

# Force kill if needed
kill -9 $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service") 2>/dev/null || true
```

#### Step 3: Rollback Database Migration (60 seconds)

**Method 1: sqlx migrate revert (Preferred)**
```bash
cd /home/jgrusewski/Work/foxhunt
DATABASE_URL="postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt" \
sqlx migrate revert
```

**Method 2: Direct SQL Execution (Fallback)**
```bash
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt \
  -f migrations/045_wave_d_regime_tracking.down.sql
```

**Method 3: Emergency Rollback Script (Fastest)**
```bash
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt \
  -f migrations/046_rollback_regime_detection.sql
```

#### Step 4: Validate Database Rollback (30 seconds)

```bash
# Check Wave D tables removed
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT COUNT(*) FROM information_schema.tables
WHERE table_name IN ('regime_states', 'regime_transitions', 'adaptive_strategy_metrics');
"
# Expected: 0

# Check Wave D functions removed
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT COUNT(*) FROM information_schema.routines
WHERE routine_name IN ('get_latest_regime', 'get_regime_transition_matrix', 'get_regime_performance');
"
# Expected: 0

# Check migration version
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT version FROM _sqlx_migrations ORDER BY version DESC LIMIT 1;
"
# Expected: 44 (after rollback from 45)
```

#### Step 5: Disable Wave D Features (if not done in Level 1)

```bash
# Same as Level 1, Step 1
sed -i 's/enable_wave_d_regime: true,/enable_wave_d_regime: false,/' ml/src/features/config.rs
```

#### Step 6: Rebuild and Restart (120 seconds)

```bash
# Rebuild services
cargo build --workspace --release

# Restart all services
cargo run --release -p api_gateway &
cargo run --release -p trading_service &
cargo run --release -p backtesting_service &
cargo run --release -p ml_training_service &

# Wait for health checks
sleep 10
curl http://localhost:8080/health
```

#### Step 7: Smoke Test (30 seconds)

```bash
# Test Wave C trading functionality
tli trade ml submit --symbol ES.FUT --action BUY --quantity 1 --dry-run

# Check feature count
cargo run --release -p ml --example check_feature_count

# Verify no regime detection queries
tli trade ml regime --symbol ES.FUT  # Should fail gracefully
```

### Expected Results
- Database tables: **REMOVED** (regime_states, regime_transitions, adaptive_strategy_metrics)
- Database functions: **REMOVED** (get_latest_regime, etc.)
- Migration version: **44** (rolled back from 45)
- Feature count: **201** (Wave C)
- Services: **RUNNING**
- Rollback time: **<300 seconds** (5 minutes)

### Data Loss
**YES** - All Wave D regime detection data is **PERMANENTLY DELETED**:
- `regime_states` table: All regime classifications
- `regime_transitions` table: All regime transition history
- `adaptive_strategy_metrics` table: All adaptive strategy performance data

**Mitigation**: Database backup created in Step 1 can restore data if needed.

### Recovery Procedure
To re-apply Wave D after Level 2 rollback:
```bash
# 1. Re-apply database migration
sqlx migrate run

# 2. Re-enable Wave D features
git checkout ml/src/features/config.rs

# 3. Rebuild services
cargo build --workspace --release

# 4. Restart services
# (same as Step 6 above)
```

### Automated Test
```bash
# Run automated Level 2 rollback test
./LEVEL_2_ROLLBACK_TEST.sh
```

---

## Level 3: Full Rollback to Wave C

### Scenario
Complete system reversion to Wave C baseline. Use for catastrophic failures.

### Target: <15 minutes rollback time

### Procedure

#### Step 1: Tag and Backup (120 seconds)

```bash
cd /home/jgrusewski/Work/foxhunt

# Tag current Wave D state (for recovery)
git tag "wave-d-emergency-rollback-$(date +%Y%m%d-%H%M%S)"

# Create backup directory
BACKUP_DIR="/tmp/foxhunt_emergency_$(date +%s)"
mkdir -p "$BACKUP_DIR"

# Backup database
PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt \
  -f "$BACKUP_DIR/foxhunt_wave_d_full.sql"

# Backup environment files
cp .env "$BACKUP_DIR/.env.wave_d"
cp .env.production "$BACKUP_DIR/.env.production.wave_d"

# Backup configuration
cp -r config/ "$BACKUP_DIR/config/"

echo "Full backup created in: $BACKUP_DIR"
```

#### Step 2: Stop All Services (30 seconds)

```bash
# Stop Foxhunt services
kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service|trading_agent_service") 2>/dev/null || true

# Wait for graceful shutdown
sleep 10

# Force kill if needed
kill -9 $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service|trading_agent_service") 2>/dev/null || true
```

#### Step 3: Rollback Database (Level 2) (60 seconds)

```bash
# Run Level 2 database rollback
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt \
  -f migrations/046_rollback_regime_detection.sql

# Verify rollback
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "\dt regime*"
# Expected: No tables found
```

#### Step 4: Checkout Wave C Baseline (90 seconds)

```bash
cd /home/jgrusewski/Work/foxhunt

# Use wave-c-baseline tag (created by Agent R3)
# This tag points to commit 60085d74 (Wave 17 Complete: 100% Production Readiness)
# Right before Wave D Phase 3 - represents 201-feature baseline

# Verify tag exists
git tag -l wave-c-baseline
# Expected: wave-c-baseline

# Stash current changes
git stash push -m "Emergency rollback: Stashing Wave D state"

# Checkout Wave C baseline tag
git checkout wave-c-baseline

# Verify checkout
git log -1 --oneline
# Expected: 60085d74 Wave 17 Complete: 100% Production Readiness Achieved

# Verify feature count in code
grep -A 5 "Wave C baseline" CLAUDE.md | grep "201 features"
```

#### Step 5: Clean Rebuild (300 seconds)

```bash
cd /home/jgrusewski/Work/foxhunt

# Clean previous build artifacts
cargo clean

# Rebuild entire workspace (release mode)
cargo build --workspace --release
```

#### Step 6: Smoke Test (60 seconds)

```bash
# Check feature count
cargo run --release -p ml --example check_feature_count
# Expected: Wave C: feature_count: 201

# Verify no Wave D code
grep -r "enable_wave_d_regime" ml/src/features/
# Expected: No results or only historical references

# Test compilation
cargo check --workspace
```

#### Step 7: Restart Services (Manual)

```bash
# Start services manually (do NOT automate in production)
echo "Manual restart required:"
echo "  1. cargo run --release -p api_gateway &"
echo "  2. cargo run --release -p trading_service &"
echo "  3. cargo run --release -p backtesting_service &"
echo "  4. cargo run --release -p ml_training_service &"
echo ""
echo "  5. Verify health: curl http://localhost:8080/health"
echo "  6. Run tests: cargo test --workspace"
```

### Expected Results
- Git state: **Wave C baseline** (commit before Wave D)
- Database: **Wave C schema** (migration 044 or earlier)
- Feature count: **201** (Wave C)
- Services: **STOPPED** (manual restart required)
- Rollback time: **<900 seconds** (15 minutes)

### Data Loss
**YES** - Complete Wave D data and code changes are **PERMANENTLY REMOVED**:
- All Wave D regime detection data
- All Wave D code changes
- All Wave D configuration

**Mitigation**: Full backup created in Step 1 can restore entire system state.

### Recovery Procedure
To re-deploy Wave D after Level 3 rollback:
```bash
# 1. Checkout Wave D tag (use wave-d-v1.0 or emergency rollback tag)
git checkout wave-d-v1.0

# Verify correct version
git log -1 --oneline
# Expected: 036655b9 feat(wave-d): Complete Wave D (225 features) integration

# 2. Restore database (optional, if data needed)
BACKUP_FILE="$BACKUP_DIR/foxhunt_wave_d_full.sql"
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt < "$BACKUP_FILE"

# 3. Re-apply migrations
sqlx migrate run

# 4. Rebuild services
cargo build --workspace --release

# 5. Restart services
# (manual restart as in Step 7 above)
```

### Automated Test
```bash
# Run automated Level 3 rollback test (WARNING: Destructive!)
./LEVEL_3_ROLLBACK_TEST.sh
```

---

## Rollback Triggers & Alerts

### Prometheus Alerts

```yaml
# /etc/prometheus/alerts/wave_d_rollback.yml

groups:
  - name: wave_d_rollback_triggers
    interval: 30s
    rules:
      # CRITICAL: Flip-flopping (>50 transitions/hour)
      - alert: WaveDFlipFlopping
        expr: rate(regime_transitions_total[1h]) > 50
        for: 5m
        labels:
          severity: critical
          rollback_level: level_1
        annotations:
          summary: "Wave D flip-flopping detected ({{ $value }} transitions/hour)"
          description: "Regime detection is changing states >50 times/hour. Recommend Level 1 rollback."
          runbook: "ROLLBACK_PROCEDURES.md#level-1-feature-only-rollback-zero-downtime"

      # CRITICAL: False positives (>80% error rate)
      - alert: WaveDFalsePositives
        expr: (sum(regime_detection_errors_total) / sum(regime_detections_total)) > 0.80
        for: 10m
        labels:
          severity: critical
          rollback_level: level_1
        annotations:
          summary: "Wave D false positive rate >80%"
          description: "Regime detection accuracy below threshold. Recommend Level 1 rollback."

      # WARNING: Performance degradation (>2x latency)
      - alert: WaveDLatencyDegradation
        expr: histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m])) > 0.002
        for: 15m
        labels:
          severity: warning
          rollback_level: level_1
        annotations:
          summary: "Wave D feature extraction latency >2ms (>2x target)"
          description: "Consider Level 1 rollback if latency persists."

      # CRITICAL: NaN/Inf in features
      - alert: WaveDDataCorruption
        expr: wave_d_features_nan_count > 0 OR wave_d_features_inf_count > 0
        for: 1m
        labels:
          severity: critical
          rollback_level: level_3
        annotations:
          summary: "Wave D data corruption detected (NaN/Inf values)"
          description: "IMMEDIATE LEVEL 3 ROLLBACK REQUIRED. Data integrity compromised."
          runbook: "ROLLBACK_PROCEDURES.md#level-3-full-rollback-to-wave-c"

      # CRITICAL: System unavailable
      - alert: FoxhuntSystemDown
        expr: up{job="foxhunt_services"} == 0
        for: 5m
        labels:
          severity: critical
          rollback_level: level_3
        annotations:
          summary: "Foxhunt system unavailable for >5 minutes"
          description: "Consider Level 3 rollback to Wave C baseline."
```

### Grafana Dashboard

Create **Wave D Rollback Monitoring** dashboard:

**Panel 1: Rollback Triggers**
```sql
-- Regime transitions per hour
SELECT
    time_bucket('1 hour', event_timestamp) AS hour,
    symbol,
    COUNT(*) AS transition_count
FROM regime_transitions
WHERE event_timestamp > NOW() - INTERVAL '24 hours'
GROUP BY hour, symbol
ORDER BY hour DESC;

-- Alert if > 50 transitions/hour
```

**Panel 2: Feature Extraction Latency**
```promql
histogram_quantile(0.99, rate(wave_d_feature_extraction_duration_seconds_bucket[5m]))
```

**Panel 3: Data Quality**
```promql
# NaN count
wave_d_features_nan_count

# Inf count
wave_d_features_inf_count

# Zero count (potential data issue)
wave_d_features_zero_count
```

**Panel 4: System Health**
```promql
# Service uptime
up{job="foxhunt_services"}

# Error rate
rate(http_requests_total{status=~"5.."}[5m])
```

---

## Emergency Contacts

### Contact Framework

**IMPORTANT**: Replace the following template with your organization's actual contact information before production deployment.

### On-Call Rotation

| Day | Primary On-Call | Backup On-Call | Manager Escalation |
|-----|----------------|----------------|-------------------|
| Mon-Wed | DevOps Team Lead | ML Engineer | CTO |
| Thu-Fri | ML Engineer | DevOps Team Lead | CTO |
| Sat-Sun | CTO | DevOps Team Lead | CEO |

### Contact Information Template

**On-Call Engineer (Primary)**
- **Name**: [Your Name Here]
- **Phone**: [+1-XXX-XXX-XXXX] (24/7 cell)
- **Slack**: [@your-slack-handle]
- **Email**: [primary.oncall@foxhunt.ai]
- **Backup Contact**: [Secondary phone/Signal/WhatsApp]

**On-Call Engineer (Secondary/Backup)**
- **Name**: [Your Name Here]
- **Phone**: [+1-XXX-XXX-XXXX] (24/7 cell)
- **Slack**: [@your-slack-handle]
- **Email**: [secondary.oncall@foxhunt.ai]
- **Backup Contact**: [Secondary phone/Signal/WhatsApp]

**DevOps Lead**
- **Name**: [Your Name Here]
- **Phone**: [+1-XXX-XXX-XXXX] (24/7 cell)
- **Slack**: [@devops-lead]
- **Email**: [devops.lead@foxhunt.ai]
- **Backup Contact**: [Secondary phone/Signal/WhatsApp]
- **Specialization**: Infrastructure, database, deployment pipelines

**CTO / Engineering Manager**
- **Name**: [Your Name Here]
- **Phone**: [+1-XXX-XXX-XXXX] (24/7 cell)
- **Slack**: [@cto]
- **Email**: [cto@foxhunt.ai]
- **Backup Contact**: [Secondary phone/Signal/WhatsApp]
- **Escalation Only**: For CRITICAL/CATASTROPHIC incidents

**Database Administrator**
- **Name**: [Your Name Here]
- **Phone**: [+1-XXX-XXX-XXXX] (24/7 cell)
- **Slack**: [@dba]
- **Email**: [dba@foxhunt.ai]
- **Backup Contact**: [Secondary phone/Signal/WhatsApp]
- **Specialization**: PostgreSQL, TimescaleDB, data recovery

**Emergency Hotline** (Group Call - Rings All On-Call Phones Simultaneously)
- **Phone**: [+1-XXX-XXX-XXXX]
- **Use For**: CRITICAL/CATASTROPHIC incidents when primary on-call is unreachable
- **Expected Response**: <5 minutes any time

### PagerDuty / Opsgenie Integration

**Recommended**: Use PagerDuty or Opsgenie for automated incident routing and escalation.

**PagerDuty Setup Instructions**:
1. Create PagerDuty service: `Foxhunt HFT Production`
2. Add integration: **Prometheus** (for alert forwarding)
3. Configure escalation policy (see below)
4. Add team members with phone numbers + Slack integration
5. Enable SMS + Phone + Push notifications
6. Set up incident response workflow automation

**Opsgenie Setup Instructions**:
1. Create Opsgenie team: `Foxhunt HFT Ops Team`
2. Add integration: **Prometheus Webhook**
3. Configure routing rules (map Prometheus severity to Opsgenie priority)
4. Add team members with phone numbers + Slack/MS Teams integration
5. Enable multi-channel notifications (SMS, Voice, Mobile Push)
6. Set up incident templates for Level 1/2/3 rollbacks

**Integration Endpoint** (Prometheus Alertmanager Config):
```yaml
# /etc/prometheus/alertmanager.yml
receivers:
  - name: 'foxhunt-pagerduty'
    pagerduty_configs:
      - service_key: '<your-pagerduty-integration-key>'
        description: '{{ .GroupLabels.alertname }}: {{ .Annotations.summary }}'
        severity: '{{ .Labels.severity }}'
        details:
          rollback_level: '{{ .Labels.rollback_level }}'
          runbook: '{{ .Annotations.runbook }}'

  - name: 'foxhunt-opsgenie'
    opsgenie_configs:
      - api_key: '<your-opsgenie-api-key>'
        message: '{{ .GroupLabels.alertname }}'
        description: '{{ .Annotations.summary }}'
        priority: '{{ .Labels.severity }}'
        tags: 'rollback_level={{ .Labels.rollback_level }},environment=production'
```

### Escalation Policy

**15-Minute Escalation Policy** (PagerDuty/Opsgenie):

| Time | Action | Notification Method |
|------|--------|-------------------|
| **T+0 min** | Alert Primary On-Call | SMS + Phone Call + Push + Slack DM |
| **T+15 min** | Escalate to Secondary On-Call (if no ACK) | SMS + Phone Call + Push + Slack DM |
| **T+30 min** | Escalate to DevOps Lead (if no ACK) | SMS + Phone Call + Push + Slack DM |
| **T+1 hour** | Escalate to CTO (if no ACK) | SMS + Phone Call + Push + Slack DM + Email |
| **T+1 hour** | Trigger Emergency Hotline (group call) | Conference Call (all team members) |

**Acknowledgement Requirements**:
- **WARNING**: ACK within 30 minutes (Slack response acceptable)
- **CRITICAL**: ACK within 15 minutes (Phone call or PagerDuty ACK required)
- **CATASTROPHIC**: ACK within 5 minutes (Immediate phone call required)

**Severity Escalation Triggers**:
- **WARNING**: Single alert firing for >15 minutes → Auto-escalate to CRITICAL
- **CRITICAL**: Incident unresolved after 1 hour → Auto-escalate to CATASTROPHIC
- **CATASTROPHIC**: Any data corruption or system-wide failure → Immediate CTO notification

### Incident Response SLA

| Severity | Response Time | Resolution Time | Rollback Level | Escalation Path |
|----------|--------------|-----------------|----------------|-----------------|
| **WARNING** | 30 minutes | 4 hours | Level 1 | Primary On-Call only |
| **CRITICAL** | 15 minutes | 1 hour | Level 2 or 3 | Primary + Secondary On-Call |
| **CATASTROPHIC** | 5 minutes | 30 minutes | Level 3 | Entire team + CTO |

### Slack Channels

- **#production-alerts**: Automated alerts from Prometheus/PagerDuty (all team members)
- **#incident-response**: Active incident coordination (on-call engineers + CTO)
- **#postmortems**: Post-incident reviews and lessons learned (entire engineering team)

### Pre-Production Checklist

**Before enabling production alerts, ensure**:
- [ ] All team members added to PagerDuty/Opsgenie with verified phone numbers
- [ ] Emergency Hotline configured (group call or conference bridge)
- [ ] Slack integrations tested (alerts posting to #production-alerts)
- [ ] Escalation policy tested (simulate WARNING → CRITICAL → CATASTROPHIC)
- [ ] Phone call notifications tested (each team member receives test call)
- [ ] SMS notifications tested (each team member receives test SMS)
- [ ] Runbook URLs accessible (no VPN required for emergency access)
- [ ] Contact information documented in team wiki (backup if this file is inaccessible)

---

## Post-Rollback Procedures

### Immediate Actions (Within 1 hour)

1. **Verify System Stability**
   ```bash
   # Check all services healthy
   curl http://localhost:8080/health
   curl http://localhost:8081/health
   curl http://localhost:8082/health
   curl http://localhost:8095/health

   # Monitor metrics for 1 hour
   watch -n 10 'curl -s http://localhost:9091/metrics | grep wave_'
   ```

2. **Notify Stakeholders**
   - Slack #production-alerts: "Wave D rollback completed (Level X)"
   - Email trading-team@foxhunt.ai: Incident summary
   - Update status page: https://status.foxhunt.ai

3. **Document Incident**
   Create incident report in `incidents/YYYY-MM-DD-wave-d-rollback.md`:
   ```markdown
   # Incident Report: Wave D Rollback

   **Date**: YYYY-MM-DD HH:MM UTC
   **Severity**: [WARNING|CRITICAL|CATASTROPHIC]
   **Rollback Level**: [1|2|3]
   **Root Cause**: [Brief description]
   **Impact**: [User impact, data loss, downtime]
   **Resolution**: [Steps taken]
   **Lessons Learned**: [What went wrong, what went right]
   ```

### Medium-term Actions (Within 24 hours)

1. **Root Cause Analysis**
   - Analyze logs: `/var/log/foxhunt/*.log`
   - Review metrics: Grafana dashboard (24-hour window)
   - Identify code issue: Git bisect or code review
   - Document findings: `incidents/YYYY-MM-DD-wave-d-rollback-RCA.md`

2. **Create Fix**
   - Create bugfix branch: `git checkout -b hotfix/wave-d-rollback-fix`
   - Implement fix
   - Add regression tests
   - Code review + approval

3. **Test Fix**
   - Deploy to staging environment
   - Run full test suite: `cargo test --workspace`
   - Run Wave D validation: `cargo run -p ml --example validate_regime_features`
   - Stress test: Simulate production load

### Long-term Actions (Within 1 week)

1. **Re-deployment Plan**
   - Schedule maintenance window (off-peak hours)
   - Prepare rollback plan (in case fix fails)
   - Notify stakeholders 48 hours in advance
   - Create deployment checklist

2. **Monitoring Improvements**
   - Add new alerts for root cause scenario
   - Improve metrics granularity
   - Add automated rollback triggers (if appropriate)

3. **Process Improvements**
   - Update rollback procedures based on lessons learned
   - Add pre-deployment tests for root cause
   - Improve staging environment to catch issues earlier

---

## Recovery & Re-deployment

### Re-enabling Wave D After Level 1 Rollback

```bash
# 1. Restore configuration
git checkout ml/src/features/config.rs

# 2. Rebuild services
cargo build --workspace --release

# 3. Graceful restart (rolling restart for zero downtime)
kill -TERM $(pgrep -f trading_service)
sleep 5
cargo run --release -p trading_service &

kill -TERM $(pgrep -f ml_training_service)
sleep 5
cargo run --release -p ml_training_service &

kill -TERM $(pgrep -f backtesting_service)
sleep 5
cargo run --release -p backtesting_service &

kill -TERM $(pgrep -f api_gateway)
sleep 5
cargo run --release -p api_gateway &

# 4. Verify Wave D re-enabled
cargo run --release -p ml --example check_feature_count
# Expected: Wave D: feature_count: 225
```

### Re-enabling Wave D After Level 2 Rollback

```bash
# 1. Re-apply database migration
cd /home/jgrusewski/Work/foxhunt
sqlx migrate run

# 2. Verify migration applied
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "\dt regime*"
# Expected: 3 tables (regime_states, regime_transitions, adaptive_strategy_metrics)

# 3. Re-enable features (same as Level 1 recovery)
git checkout ml/src/features/config.rs
cargo build --workspace --release

# 4. Restart services (same as Level 1 recovery)

# 5. Verify full Wave D functionality
tli trade ml regime --symbol ES.FUT
tli trade ml transitions --symbol ES.FUT --window-hours 24
```

### Re-deploying Wave D After Level 3 Rollback

```bash
# 1. Use wave-d-v1.0 tag (created by Agent R3)
# This tag points to commit 036655b9 (Wave D v1.0 COMPLETE - 225 features)
WAVE_D_TAG="wave-d-v1.0"

# Alternative: Use emergency rollback tag if created during incident
# WAVE_D_TAG=$(git tag -l | grep "wave-d-emergency-rollback" | tail -1)

echo "Re-deploying: $WAVE_D_TAG"

# 2. Checkout Wave D code
git checkout "$WAVE_D_TAG"

# Verify correct tag
git log -1 --oneline
# Expected: 036655b9 feat(wave-d): Complete Wave D (225 features) integration

# 3. Re-apply database migration
sqlx migrate run

# 4. Clean rebuild
cargo clean
cargo build --workspace --release

# 5. Restore configuration (if needed)
BACKUP_DIR="/tmp/foxhunt_emergency_XXXXX"  # From Level 3 rollback
cp "$BACKUP_DIR/.env.wave_d" .env
cp "$BACKUP_DIR/.env.production.wave_d" .env.production

# 6. Manual service restart
cargo run --release -p api_gateway &
cargo run --release -p trading_service &
cargo run --release -p backtesting_service &
cargo run --release -p ml_training_service &

# 7. Comprehensive validation
sleep 30
curl http://localhost:8080/health
cargo test --workspace
cargo run -p ml --example validate_regime_features

# 8. Monitor for 24 hours before declaring success
```

---

## Testing Rollback Procedures

### Automated Test Suite

All 3 rollback levels have automated test scripts:

```bash
# Level 1: Feature-only rollback (zero downtime)
./LEVEL_1_ROLLBACK_TEST.sh

# Level 2: Database rollback (~5 minutes)
./LEVEL_2_ROLLBACK_TEST.sh

# Level 3: Full rollback (~15 minutes, DESTRUCTIVE!)
./LEVEL_3_ROLLBACK_TEST.sh
```

### Manual Testing Checklist

**Pre-Production Testing (Staging Environment):**
- [ ] Deploy Wave D to staging
- [ ] Generate synthetic regime data (1000+ records)
- [ ] Test Level 1 rollback → Verify 201 features, zero downtime
- [ ] Test Level 2 rollback → Verify tables removed, services restart
- [ ] Test Level 3 rollback → Verify Wave C codebase, clean state
- [ ] Test recovery for each level → Verify Wave D re-enables correctly

**Production Readiness:**
- [ ] Rollback scripts tested on staging (all 3 levels)
- [ ] Database backups automated (hourly)
- [ ] Prometheus alerts configured
- [ ] Grafana dashboards created
- [ ] On-call rotation established
- [ ] Emergency contacts verified
- [ ] Incident response runbooks reviewed

---

## Appendix A: Rollback Performance Benchmarks

### Level 1 Rollback Performance

| Step | Target Time | Actual Time | Notes |
|------|------------|-------------|-------|
| Disable Wave D features | 30s | 15-20s | Code edit + verification |
| Rebuild services | 30s | 25-35s | Release build |
| Graceful restart | 30s | 20-25s | Rolling restart |
| Validate rollback | 15s | 10-12s | Feature count + health check |
| **Total** | **<60s** | **70-92s** | Hot-reload would reduce to <10s |

**Bottleneck**: Rebuild step (25-35s)
**Improvement**: Implement hot-reload configuration mechanism

### Level 2 Rollback Performance

| Step | Target Time | Actual Time | Notes |
|------|------------|-------------|-------|
| Pre-rollback backup | 60s | 45-70s | Database size dependent |
| Stop services | 30s | 15-20s | Graceful shutdown |
| Rollback migration | 60s | 30-40s | SQL execution |
| Validate database | 30s | 10-15s | Table + function check |
| Disable features | 30s | 15-20s | Same as Level 1 |
| Rebuild + restart | 120s | 90-110s | Build + service start |
| Smoke test | 30s | 20-25s | Basic validation |
| **Total** | **<300s** | **225-300s** | Within target |

**Bottleneck**: Database backup (45-70s)
**Improvement**: Use continuous replication for instant recovery

### Level 3 Rollback Performance

| Step | Target Time | Actual Time | Notes |
|------|------------|-------------|-------|
| Tag + backup | 120s | 90-130s | Full database + config backup |
| Stop services | 30s | 15-20s | Graceful shutdown |
| Rollback database | 60s | 30-40s | Migration revert |
| Checkout Wave C | 90s | 60-80s | Git checkout + stash |
| Clean rebuild | 300s | 240-320s | cargo clean + build --release |
| Smoke test | 60s | 40-50s | Compilation + feature check |
| **Total** | **<900s** | **475-640s** | Well within target |

**Bottleneck**: Clean rebuild (240-320s)
**Improvement**: Pre-build Wave C binaries for instant deployment

---

## Appendix B: Common Issues & Troubleshooting

### Issue 1: Level 1 Rollback Doesn't Reduce Feature Count

**Symptom**: After Level 1 rollback, `check_feature_count` still shows 225 features.

**Diagnosis**:
```bash
# Check if configuration change was applied
grep "enable_wave_d_regime" ml/src/features/config.rs

# Check if services were restarted
pgrep -af foxhunt

# Check feature count in running service
curl http://localhost:8080/metrics | grep feature_count
```

**Solution**:
```bash
# Verify configuration file edited correctly
cat ml/src/features/config.rs | grep -A5 "pub fn wave_d"

# Force rebuild
cargo clean
cargo build --workspace --release

# Hard restart services
kill -9 $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")
# Then restart manually
```

### Issue 2: Level 2 Database Rollback Fails with "Table Does Not Exist"

**Symptom**: Migration rollback fails with PostgreSQL error.

**Diagnosis**:
```bash
# Check if migration 045 was actually applied
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT version FROM _sqlx_migrations WHERE version = 45;
"

# Check current table state
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "\dt regime*"
```

**Solution**:
```bash
# If migration 045 never applied, skip Level 2 rollback
echo "Migration 045 not applied, no database rollback needed"

# If tables exist but migration history is wrong, manual cleanup
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -f migrations/046_rollback_regime_detection.sql
```

### Issue 3: Level 3 Rollback Can't Find Wave C Baseline Tag

**Symptom**: `wave-c-baseline` tag doesn't exist, git checkout fails.

**Diagnosis**:
```bash
# Check if wave-c-baseline tag exists
git tag -l wave-c-baseline

# If missing, check all tags
git tag -l

# Find last commit before Wave D Phase 3
git log --all --oneline --before="2025-10-17" | head -10
```

**Solution**:
```bash
# Use the known Wave C baseline commit (Agent R3 verified)
WAVE_C_COMMIT="60085d74"  # Wave 17 Complete: 100% Production Readiness

# Re-create wave-c-baseline tag
git tag -a wave-c-baseline "$WAVE_C_COMMIT" -m "Wave C baseline (201 features) - emergency recreation"

# Verify tag created
git tag -l wave-c-baseline

# Checkout using tag
git checkout wave-c-baseline
```

### Issue 4: Services Won't Start After Rollback

**Symptom**: Services crash immediately after rollback.

**Diagnosis**:
```bash
# Check service logs
journalctl -u foxhunt-api-gateway -n 50
journalctl -u foxhunt-trading-service -n 50

# Check for port conflicts
lsof -i :50051  # API Gateway
lsof -i :50052  # Trading Service
lsof -i :50053  # Backtesting Service
lsof -i :50054  # ML Training Service

# Check database connectivity
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "SELECT 1;"
```

**Solution**:
```bash
# Kill all conflicting processes
kill -9 $(lsof -t -i :50051)
kill -9 $(lsof -t -i :50052)
kill -9 $(lsof -t -i :50053)
kill -9 $(lsof -t -i :50054)

# Restart services one at a time
cargo run --release -p api_gateway &
sleep 10
cargo run --release -p trading_service &
sleep 10
cargo run --release -p backtesting_service &
sleep 10
cargo run --release -p ml_training_service &

# Monitor startup
tail -f /var/log/foxhunt/*.log
```

### Issue 5: Data Corruption Persists After Rollback

**Symptom**: NaN/Inf values still appearing in Wave C features.

**Diagnosis**:
```bash
# Check feature extraction code for bugs
grep -r "NaN\|Inf" ml/src/features/

# Check database for corrupt data
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "
SELECT COUNT(*) FROM market_data WHERE close = 'NaN'::float;
"

# Check input data quality
cargo run --release -p ml --example validate_dbn_data
```

**Solution**:
```bash
# If corruption is in Wave C code (not Wave D), full data cleanup needed
# 1. Restore from last known good backup
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt < /path/to/last_good_backup.sql

# 2. Re-download DBN data
# (See ML_TRAINING_ROADMAP.md for Databento download instructions)

# 3. Re-run feature extraction with validation
cargo run --release -p ml --example validate_features
```

---

## Appendix C: Rollback Checklist Template

Use this checklist for production rollbacks:

```markdown
# Wave D Rollback Checklist

**Incident ID**: YYYY-MM-DD-###
**Rollback Level**: [ ] Level 1  [ ] Level 2  [ ] Level 3
**Date/Time**: YYYY-MM-DD HH:MM UTC
**Operator**: [Name]
**Approver**: [Manager Name]

## Pre-Rollback
- [ ] Incident confirmed (Prometheus alert or manual detection)
- [ ] Rollback level determined (see decision matrix)
- [ ] Stakeholders notified (#production-alerts Slack)
- [ ] Database backup created (verify size: ___GB)
- [ ] Environment files backed up
- [ ] Rollback approval received (Manager signature: ______)

## Rollback Execution
- [ ] Services stopped gracefully (or already down)
- [ ] Database migration reverted (if Level 2/3)
- [ ] Wave D features disabled (if Level 1/2)
- [ ] Wave C code checked out (if Level 3)
- [ ] Services rebuilt (release mode)
- [ ] Rollback validation completed (see test results below)

## Validation
- [ ] Feature count verified: _____ (expected: 201 for Wave C)
- [ ] Database tables verified (regime tables removed if Level 2/3)
- [ ] Services health check passed (all 4 services: UP)
- [ ] Basic trading test passed (dry-run order submission)
- [ ] No errors in logs (last 50 lines checked)
- [ ] Metrics nominal (Grafana dashboard green)

## Post-Rollback
- [ ] Stakeholders notified (rollback complete)
- [ ] Status page updated (https://status.foxhunt.ai)
- [ ] Incident report created (incidents/YYYY-MM-DD-*.md)
- [ ] Monitoring increased (hourly checks for 24 hours)
- [ ] Root cause analysis scheduled (within 24 hours)
- [ ] Re-deployment plan drafted (within 1 week)

## Rollback Metrics
- Total rollback time: _____ seconds (target: Level 1 <60s, Level 2 <300s, Level 3 <900s)
- Downtime: _____ minutes (target: Level 1 = 0, Level 2 <5min, Level 3 <15min)
- Data loss: _____ records (expected: Level 1/2 = Wave D data only, Level 3 = all Wave D)

## Sign-off
- [ ] Operator verification: ____________ (signature)
- [ ] Manager approval: ____________ (signature)
- [ ] Post-rollback review scheduled: ____________ (date/time)
```

---

## Version History

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | 2025-10-19 | Agent R1 | Initial release (all 3 rollback levels tested) |

---

**END OF ROLLBACK PROCEDURES DOCUMENT**