foxhunt/ROLLBACK_QUICK_REFERENCE.md

# Wave D Rollback Quick Reference Card
**Emergency Hotline**: [+1-XXX-XXX-XXXX] (24/7) - REPLACE WITH ACTUAL NUMBER
**Full Documentation**: `ROLLBACK_PROCEDURES.md`

---

## Decision Matrix (10-Second Guide)

| Symptom | Action | Timeframe |
|---------|--------|-----------|
| **Flip-flopping** (>50 transitions/hour) | **Level 1** | <1 min |
| **False positives** (>80% error) | **Level 1** | <1 min |
| **Slow performance** (>2x latency) | **Level 1** | <1 hour |
| **NaN/Inf in features** | **Level 3** | <15 min |
| **System down** (>5 min) | **Level 3** | <15 min |

---

## Level 1: Feature-Only (Zero Downtime, <1 min)

```bash
# 1. Disable Wave D (30s)
sed -i 's/enable_wave_d_regime: true,/enable_wave_d_regime: false,/' ml/src/features/config.rs

# 2. Rebuild (30s)
cargo build --workspace --release

# 3. Restart (30s, rolling restart)
kill -TERM $(pgrep -f trading_service); sleep 5; cargo run --release -p trading_service &
kill -TERM $(pgrep -f ml_training_service); sleep 5; cargo run --release -p ml_training_service &
kill -TERM $(pgrep -f backtesting_service); sleep 5; cargo run --release -p backtesting_service &
kill -TERM $(pgrep -f api_gateway); sleep 5; cargo run --release -p api_gateway &

# 4. Verify
cargo run --release -p ml --example check_feature_count  # Should show 201
```

**Data Loss**: NONE
**Recovery**: `git checkout ml/src/features/config.rs` + rebuild + restart

---

## Level 2: Database Rollback (~5 min)

```bash
# 1. Backup (60s)
PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt -f /tmp/backup_$(date +%s).sql

# 2. Stop services (30s)
kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")

# 3. Rollback database (60s)
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -f migrations/046_rollback_regime_detection.sql

# 4. Disable features (30s, same as Level 1 step 1)
sed -i 's/enable_wave_d_regime: true,/enable_wave_d_regime: false,/' ml/src/features/config.rs

# 5. Rebuild + restart (120s)
cargo build --workspace --release
cargo run --release -p api_gateway &
cargo run --release -p trading_service &
cargo run --release -p backtesting_service &
cargo run --release -p ml_training_service &

# 6. Verify
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "\dt regime*"  # Should be empty
```

**Data Loss**: All Wave D regime data (regime_states, regime_transitions, adaptive_strategy_metrics)
**Recovery**: `sqlx migrate run` + re-enable features + rebuild + restart

---

## Level 3: Full Rollback (~15 min)

```bash
# 1. Tag + backup (120s)
git tag "wave-d-emergency-$(date +%Y%m%d-%H%M%S)"
PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt -f /tmp/full_backup_$(date +%s).sql

# 2. Stop services (30s)
kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")

# 3. Rollback database (60s, same as Level 2 step 3)
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -f migrations/046_rollback_regime_detection.sql

# 4. Checkout Wave C (90s)
WAVE_C_COMMIT=$(git log --all --oneline | grep -E "WAVE_C.*COMPLETE" | head -1 | awk '{print $1}')
git stash push -m "Emergency rollback"
git checkout "$WAVE_C_COMMIT"

# 5. Clean rebuild (300s)
cargo clean
cargo build --workspace --release

# 6. Verify
cargo run --release -p ml --example check_feature_count  # Should show 201

# 7. Manual restart (DO NOT AUTOMATE)
echo "Start services manually after verification"
```

**Data Loss**: All Wave D code + data
**Recovery**: `git checkout <wave-d-tag>` + restore database + migrate + rebuild + restart

---

## Automated Tests

```bash
# Run before production deployment (staging only!)
./LEVEL_1_ROLLBACK_TEST.sh  # Zero downtime test
./LEVEL_2_ROLLBACK_TEST.sh  # Database rollback test
./LEVEL_3_ROLLBACK_TEST.sh  # Full rollback test (DESTRUCTIVE!)
```

---

## Emergency Contacts

**REPLACE WITH YOUR TEAM'S CONTACT INFO BEFORE PRODUCTION**

**On-Call Engineer (Primary)**
- Phone: [+1-XXX-XXX-XXXX] | Slack: [@your-handle] | Email: [primary.oncall@foxhunt.ai]

**On-Call Engineer (Secondary/Backup)**
- Phone: [+1-XXX-XXX-XXXX] | Slack: [@your-handle] | Email: [secondary.oncall@foxhunt.ai]

**DevOps Lead**
- Phone: [+1-XXX-XXX-XXXX] | Slack: [@devops-lead] | Email: [devops.lead@foxhunt.ai]
- Specialization: Infrastructure, database, deployment

**CTO / Engineering Manager**
- Phone: [+1-XXX-XXX-XXXX] | Slack: [@cto] | Email: [cto@foxhunt.ai]
- Escalation Only: CRITICAL/CATASTROPHIC incidents

**Database Administrator**
- Phone: [+1-XXX-XXX-XXXX] | Slack: [@dba] | Email: [dba@foxhunt.ai]
- Specialization: PostgreSQL, TimescaleDB, data recovery

**Emergency Hotline** (Group Call - Rings All On-Call Phones)
- Phone: [+1-XXX-XXX-XXXX]
- Use For: CRITICAL/CATASTROPHIC when primary unreachable
- Expected Response: <5 minutes

**Escalation Timeline**:
- T+0 min: Primary On-Call (SMS + Phone + Slack)
- T+15 min: Secondary On-Call (if no ACK)
- T+30 min: DevOps Lead (if no ACK)
- T+1 hour: CTO + Emergency Hotline (if no ACK)

**PagerDuty/Opsgenie**: Recommended for automated routing
**Slack Channels**: #production-alerts, #incident-response, #postmortems

---

## Post-Rollback Checklist

- [ ] Verify feature count (201 for Wave C)
- [ ] Check all services UP (`curl http://localhost:8080/health`)
- [ ] Test basic trading (`tli trade ml submit --dry-run`)
- [ ] Notify #production-alerts on Slack
- [ ] Create incident report (`incidents/YYYY-MM-DD-*.md`)
- [ ] Schedule root cause analysis (within 24 hours)
- [ ] Monitor for 24 hours (hourly checks)

---

**When in doubt, call the Emergency Hotline. Don't try to be a hero.**