foxhunt/ROLLBACK_QUICK_REFERENCE.md

Wave D Rollback Quick Reference Card

Emergency Hotline: [+1-XXX-XXX-XXXX] (24/7) - REPLACE WITH ACTUAL NUMBER Full Documentation: ROLLBACK_PROCEDURES.md

---

Decision Matrix (10-Second Guide)

Symptom	Action	Timeframe
Flip-flopping (>50 transitions/hour)	Level 1	<1 min
False positives (>80% error)	Level 1	<1 min
Slow performance (>2x latency)	Level 1	<1 hour
NaN/Inf in features	Level 3	<15 min
System down (>5 min)	Level 3	<15 min

---

Level 1: Feature-Only (Zero Downtime, <1 min)

# 1. Disable Wave D (30s)
sed -i 's/enable_wave_d_regime: true,/enable_wave_d_regime: false,/' ml/src/features/config.rs

# 2. Rebuild (30s)
cargo build --workspace --release

# 3. Restart (30s, rolling restart)
kill -TERM $(pgrep -f trading_service); sleep 5; cargo run --release -p trading_service &
kill -TERM $(pgrep -f ml_training_service); sleep 5; cargo run --release -p ml_training_service &
kill -TERM $(pgrep -f backtesting_service); sleep 5; cargo run --release -p backtesting_service &
kill -TERM $(pgrep -f api_gateway); sleep 5; cargo run --release -p api_gateway &

# 4. Verify
cargo run --release -p ml --example check_feature_count  # Should show 201

Data Loss: NONE Recovery: git checkout ml/src/features/config.rs + rebuild + restart

---

Level 2: Database Rollback (~5 min)

# 1. Backup (60s)
PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt -f /tmp/backup_$(date +%s).sql

# 2. Stop services (30s)
kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")

# 3. Rollback database (60s)
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -f migrations/046_rollback_regime_detection.sql

# 4. Disable features (30s, same as Level 1 step 1)
sed -i 's/enable_wave_d_regime: true,/enable_wave_d_regime: false,/' ml/src/features/config.rs

# 5. Rebuild + restart (120s)
cargo build --workspace --release
cargo run --release -p api_gateway &
cargo run --release -p trading_service &
cargo run --release -p backtesting_service &
cargo run --release -p ml_training_service &

# 6. Verify
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -c "\dt regime*"  # Should be empty

Data Loss: All Wave D regime data (regime_states, regime_transitions, adaptive_strategy_metrics) Recovery: sqlx migrate run + re-enable features + rebuild + restart

---

Level 3: Full Rollback (~15 min)

# 1. Tag + backup (120s)
git tag "wave-d-emergency-$(date +%Y%m%d-%H%M%S)"
PGPASSWORD=foxhunt_dev_password pg_dump -h localhost -U foxhunt -d foxhunt -f /tmp/full_backup_$(date +%s).sql

# 2. Stop services (30s)
kill -TERM $(pgrep -f "api_gateway|trading_service|backtesting_service|ml_training_service")

# 3. Rollback database (60s, same as Level 2 step 3)
PGPASSWORD=foxhunt_dev_password psql -h localhost -U foxhunt -d foxhunt -f migrations/046_rollback_regime_detection.sql

# 4. Checkout Wave C (90s)
WAVE_C_COMMIT=$(git log --all --oneline | grep -E "WAVE_C.*COMPLETE" | head -1 | awk '{print $1}')
git stash push -m "Emergency rollback"
git checkout "$WAVE_C_COMMIT"

# 5. Clean rebuild (300s)
cargo clean
cargo build --workspace --release

# 6. Verify
cargo run --release -p ml --example check_feature_count  # Should show 201

# 7. Manual restart (DO NOT AUTOMATE)
echo "Start services manually after verification"

Data Loss: All Wave D code + data Recovery: git checkout <wave-d-tag> + restore database + migrate + rebuild + restart

---

Automated Tests

# Run before production deployment (staging only!)
./LEVEL_1_ROLLBACK_TEST.sh  # Zero downtime test
./LEVEL_2_ROLLBACK_TEST.sh  # Database rollback test
./LEVEL_3_ROLLBACK_TEST.sh  # Full rollback test (DESTRUCTIVE!)

---

Emergency Contacts

REPLACE WITH YOUR TEAM'S CONTACT INFO BEFORE PRODUCTION

On-Call Engineer (Primary)

• Phone: [+1-XXX-XXX-XXXX] | Slack: [@your-handle] | Email: [primary.oncall@foxhunt.ai]

On-Call Engineer (Secondary/Backup)

• Phone: [+1-XXX-XXX-XXXX] | Slack: [@your-handle] | Email: [secondary.oncall@foxhunt.ai]

DevOps Lead

• Phone: [+1-XXX-XXX-XXXX] | Slack: [@devops-lead] | Email: [devops.lead@foxhunt.ai]
• Specialization: Infrastructure, database, deployment

CTO / Engineering Manager

• Phone: [+1-XXX-XXX-XXXX] | Slack: [@cto] | Email: [cto@foxhunt.ai]
• Escalation Only: CRITICAL/CATASTROPHIC incidents

Database Administrator

• Phone: [+1-XXX-XXX-XXXX] | Slack: [@dba] | Email: [dba@foxhunt.ai]
• Specialization: PostgreSQL, TimescaleDB, data recovery

Emergency Hotline (Group Call - Rings All On-Call Phones)

• Phone: [+1-XXX-XXX-XXXX]
• Use For: CRITICAL/CATASTROPHIC when primary unreachable
• Expected Response: <5 minutes

Escalation Timeline:

• T+0 min: Primary On-Call (SMS + Phone + Slack)
• T+15 min: Secondary On-Call (if no ACK)
• T+30 min: DevOps Lead (if no ACK)
• T+1 hour: CTO + Emergency Hotline (if no ACK)

PagerDuty/Opsgenie: Recommended for automated routing Slack Channels: #production-alerts, #incident-response, #postmortems

---

Post-Rollback Checklist

• Verify feature count (201 for Wave C)
• Check all services UP (curl http://localhost:8080/health)
• Test basic trading (tli trade ml submit --dry-run)
• Notify #production-alerts on Slack
• Create incident report (incidents/YYYY-MM-DD-*.md)
• Schedule root cause analysis (within 24 hours)
• Monitor for 24 hours (hourly checks)

---

When in doubt, call the Emergency Hotline. Don't try to be a hero.