foxhunt/deployment/disaster-recovery-procedures.md

Disaster Recovery Testing Procedures - Foxhunt HFT System

🚨 CRITICAL PRODUCTION SAFEGUARDS

Purpose: Validate system resilience and recovery capabilities before production deployment
Frequency: Required before production deployment, quarterly thereafter
Duration: 4-6 hours full test suite

📋 DISASTER SCENARIOS TESTING MATRIX

Scenario 1: Database Failure

Impact: Complete data persistence loss
Recovery Target: < 5 minutes RTO, < 1 minute RPO

# Test procedure
./deployment/scripts/test-database-failover.sh

Test Steps:

1. Stop primary PostgreSQL instance
2. Verify automatic failover to read replica
3. Test write operations on new primary
4. Validate data consistency
5. Measure recovery time

Expected Results:

• ✅ Services maintain operation during failover
• ✅ No data loss during transition
• ✅ Recovery completes within 5 minutes
• ✅ All services reconnect automatically

Scenario 2: ML Training Service Crash

Impact: Model training interruption, inference degradation
Recovery Target: < 2 minutes RTO, graceful degradation

Test Steps:

1. Force-kill ML Training Service process
2. Verify trading service switches to cached models
3. Test model inference with fallback models
4. Restart ML service and verify recovery
5. Check training job resumption

Expected Results:

• ✅ Trading continues with fallback models
• ✅ No trading interruption during ML service restart
• ✅ Training jobs resume from last checkpoint
• ✅ Performance degradation alerts trigger

Scenario 3: Network Partition (Split Brain)

Impact: Service isolation, potential data inconsistency
Recovery Target: < 3 minutes detection, automatic partition handling

Test Steps:

1. Simulate network partition between services
2. Verify partition detection mechanisms
3. Test service behavior in isolation mode
4. Restore network connectivity
5. Validate data reconciliation

Expected Results:

• ✅ Services detect partition within 30 seconds
• ✅ Read-only mode activated for isolated services
• ✅ No conflicting writes during partition
• ✅ Automatic reconciliation after recovery

Scenario 4: Risk Engine Failure

Impact: Loss of risk monitoring, potential capital loss
Recovery Target: < 30 seconds RTO, immediate trading halt

Test Steps:

1. Force-stop risk management service
2. Verify immediate trading halt
3. Test emergency position liquidation
4. Restart risk service
5. Validate risk limit restoration

Expected Results:

• ✅ Trading halts within 5 seconds
• ✅ Emergency liquidation protocols activate
• ✅ No new positions opened during outage
• ✅ Risk limits enforced immediately after restart

Scenario 5: Market Data Feed Interruption

Impact: Blind trading, potential adverse selection
Recovery Target: < 10 seconds detection, automatic feed switching

Test Steps:

1. Disconnect primary market data feed
2. Verify automatic failover to backup feed
3. Test data quality validation
4. Restore primary feed
5. Validate feed switching logic

Expected Results:

• ✅ Backup feed activates within 10 seconds
• ✅ No stale data used for trading decisions
• ✅ Data quality alerts trigger appropriately
• ✅ Smooth transition back to primary feed

🔧 AUTOMATED DISASTER TESTING SCRIPT

#!/bin/bash
# Comprehensive Disaster Recovery Test Suite
# Location: deployment/scripts/disaster-recovery-test.sh

echo "Starting Foxhunt DR Testing Suite..."

# Pre-test validation
./deployment/scripts/production-validation.sh
if [[ $? -ne 0 ]]; then
    echo "❌ System not ready for DR testing - fix issues first"
    exit 1
fi

# Test 1: Database Failover
echo "🔄 Testing database failover..."
./tests/disaster-recovery/test-database-failover.sh

# Test 2: Service Crash Recovery
echo "🔄 Testing service crash recovery..."
./tests/disaster-recovery/test-service-crash.sh

# Test 3: Network Partition
echo "🔄 Testing network partition handling..."
./tests/disaster-recovery/test-network-partition.sh

# Test 4: Risk Engine Failure
echo "🔄 Testing risk engine failure..."
./tests/disaster-recovery/test-risk-engine-failure.sh

# Test 5: Market Data Interruption
echo "🔄 Testing market data interruption..."
./tests/disaster-recovery/test-market-data-failure.sh

echo "✅ All disaster recovery tests completed"

📊 RECOVERY TIME OBJECTIVES (RTO) & RECOVERY POINT OBJECTIVES (RPO)

Component	RTO Target	RPO Target	Current Status
Database	< 5 min	< 1 min	⚠️ Needs Implementation
ML Training Service	< 2 min	< 5 min	⚠️ Needs Implementation
Risk Engine	< 30 sec	0 (real-time)	⚠️ Needs Implementation
Trading Service	< 1 min	< 1 sec	⚠️ Needs Implementation
Market Data	< 10 sec	0 (real-time)	⚠️ Needs Implementation
TLI Dashboard	< 5 min	< 15 min	⚠️ Needs Implementation

🚨 EMERGENCY RESPONSE PROCEDURES

Step 1: Incident Detection

• Automated: Prometheus alerts trigger
• Manual: Operations team notification
• Escalation: Page-duty engineer within 2 minutes

Step 2: Initial Assessment

# Quick system health check
./deployment/scripts/health-check-validation.sh

# Check system resources
top -bn1 | head -20
df -h
free -m

# Review recent logs
journalctl -u foxhunt-* --since "5 minutes ago"

Step 3: Service Isolation

# Emergency trading halt
curl -X POST http://localhost:8080/emergency/halt

# Isolate failing services
systemctl stop foxhunt-ml-training
systemctl stop foxhunt-risk-management

# Enable read-only mode
curl -X POST http://localhost:8080/mode/readonly

Step 4: Recovery Execution

# Database recovery
./deployment/scripts/recover-database.sh

# Service restart with health validation
./deployment/scripts/restart-services.sh --validate

# Data consistency check
./deployment/scripts/validate-data-consistency.sh

Step 5: Post-Recovery Validation

# Full system validation
./deployment/scripts/production-validation.sh

# Performance baseline check
./tests/performance/latency-validation.sh

# Trading resumption
curl -X POST http://localhost:8080/trading/resume

📝 TESTING CHECKLIST

Pre-Testing Requirements

• All critical issues from expert analysis fixed
• Production validation script passes
• Backup systems verified operational
• Test environment mirrors production
• Emergency contacts available
• Rollback procedures prepared

During Testing

• Monitor system metrics continuously
• Record recovery times for each scenario
• Document any unexpected behaviors
• Validate data integrity after each test
• Test alert notifications
• Verify automated recovery mechanisms

Post-Testing Activities

• Update RTO/RPO targets based on results
• Document lessons learned
• Update emergency procedures
• Schedule follow-up improvements
• Brief operations team on results
• Update monitoring thresholds

🔍 MONITORING DURING DR TESTING

Key Metrics to Monitor

# System performance
watch -n 1 'echo "=== SYSTEM METRICS ===" && \
uptime && \
free -m && \
df -h / && \
echo "=== SERVICE STATUS ===" && \
systemctl status foxhunt-* | grep Active'

# Trading metrics
watch -n 1 'echo "=== TRADING METRICS ===" && \
curl -s http://localhost:8080/metrics | grep -E "(position_count|order_latency|risk_score)"'

# Database connections
watch -n 1 'echo "=== DATABASE ===" && \
psql -h localhost -U foxhunt -c "SELECT count(*) FROM pg_stat_activity;"'

Alert Thresholds During Testing

• Response time > 100ms
• Error rate > 1%
• Memory usage > 90%
• Disk usage > 95%
• Database connections > 80% of max

🚧 CURRENT IMPLEMENTATION STATUS

❌ Critical Gaps (Must Implement Before Production)

1. Database Failover: No automatic failover configured
2. Service Health Checks: Basic health endpoints missing
3. Emergency Trading Halt: Manual process only
4. Backup Feed Switching: No redundant data sources
5. Checkpoint Recovery: ML training state persistence missing

⚠️ High Priority Improvements

1. Automated Recovery: Manual intervention required
2. Data Consistency: No automated validation
3. Performance Monitoring: Limited metrics during failure
4. Alert Integration: Basic notifications only
5. Runbook Automation: Procedures not scripted

✅ Implemented Features

1. Service Compilation: All modules build successfully
2. Basic Monitoring: Prometheus metrics available
3. Configuration Management: Environment-based config
4. Security Documentation: Incident response procedures
5. Validation Scripts: Basic health check capabilities

📞 EMERGENCY CONTACTS

On-Call Engineer: [CONFIGURE]
Database Admin: [CONFIGURE]
Infrastructure Team: [CONFIGURE]
Business Stakeholders: [CONFIGURE]

🔄 TESTING SCHEDULE

• Initial: Before production deployment
• Regular: Quarterly during maintenance windows
• Ad-hoc: After major system changes
• Emergency: During actual incidents

---

Document Status: 🚧 Draft - Requires Implementation
Next Review: After critical DR infrastructure implementation
Owner: DevOps Team
Last Updated: 2025-01-21