foxhunt/docs/DISASTER_RECOVERY.md

# Foxhunt HFT Trading System - Disaster Recovery Procedures

## Table of Contents

1. [Overview](#overview)
2. [Recovery Objectives](#recovery-objectives)
3. [Disaster Scenarios](#disaster-scenarios)
4. [Recovery Strategies](#recovery-strategies)
5. [Backup Infrastructure](#backup-infrastructure)
6. [Recovery Procedures](#recovery-procedures)
7. [Failover Systems](#failover-systems)
8. [Testing & Validation](#testing--validation)
9. [Communication Plans](#communication-plans)
10. [Post-Recovery Actions](#post-recovery-actions)

## Overview

This document outlines comprehensive disaster recovery procedures for the Foxhunt HFT trading system. The plan ensures business continuity with minimal downtime and data loss in the event of various disaster scenarios.

### Scope
- **Primary Trading Systems**: Core trading infrastructure
- **Data Storage**: All databases and persistent storage
- **Network Infrastructure**: Connectivity and routing
- **Security Systems**: Authentication and authorization
- **Monitoring & Alerting**: Observability infrastructure

### Responsibilities
- **Incident Commander**: Overall response coordination
- **Technical Lead**: System recovery execution
- **Database Administrator**: Data recovery and integrity
- **Network Engineer**: Connectivity restoration
- **Security Officer**: Security validation and compliance
- **Communications Lead**: Stakeholder notifications

## Recovery Objectives

### Recovery Time Objective (RTO)
- **Critical Trading Systems**: 15 minutes
- **Database Systems**: 10 minutes
- **Monitoring Systems**: 30 minutes
- **Full System Restoration**: 60 minutes

### Recovery Point Objective (RPO)
- **Transaction Data**: 1 minute (maximum data loss)
- **Market Data**: 5 minutes
- **Configuration Data**: 15 minutes
- **Log Data**: 1 hour

### Service Level Objectives
- **System Availability**: 99.95% uptime
- **Data Integrity**: 100% (zero data corruption)
- **Performance**: <10% degradation post-recovery
- **Security**: Full security controls operational

## Disaster Scenarios

### Scenario 1: Hardware Failure

#### Single Server Failure
**Impact**: Reduced capacity, potential service degradation
**RTO**: 5 minutes (automatic failover)
**RPO**: 30 seconds

**Immediate Actions**:
```bash
# 1. Verify failover activation
./scripts/verify-failover-status.sh

# 2. Check load balancer status
curl -f http://load-balancer:8080/health

# 3. Monitor performance metrics
./scripts/monitor-failover-performance.sh

# 4. Notify operations team
./scripts/send-alert.sh "WARN: Server failover activated"
```

#### Multiple Server Failure
**Impact**: Significant service disruption
**RTO**: 15 minutes
**RPO**: 5 minutes

**Recovery Steps**:
```bash
# 1. Assess scope of failure
./scripts/assess-infrastructure-status.sh

# 2. Activate secondary data center
./scripts/activate-secondary-datacenter.sh

# 3. Update DNS records
./scripts/update-dns-failover.sh

# 4. Verify service restoration
./scripts/verify-full-system-health.sh
```

### Scenario 2: Database Corruption

#### PostgreSQL Corruption
**Impact**: Transaction data unavailable
**RTO**: 10 minutes
**RPO**: 1 minute

**Recovery Steps**:
```bash
# 1. Stop database connections
sudo systemctl stop foxhunt-*
sudo systemctl stop postgresql

# 2. Assess corruption extent
sudo -u postgres pg_checksums -D /var/lib/postgresql/14/main

# 3. Restore from backup
sudo -u postgres pg_restore \
  --clean --create --verbose \
  /backup/postgresql/latest.dump

# 4. Verify data integrity
sudo -u postgres psql -c "SELECT COUNT(*) FROM trades WHERE created_at > NOW() - INTERVAL '1 hour';"

# 5. Restart services
sudo systemctl start postgresql
sudo systemctl start foxhunt-*
```

#### InfluxDB Data Loss
**Impact**: Historical metrics unavailable
**RTO**: 5 minutes
**RPO**: 15 minutes

**Recovery Steps**:
```bash
# 1. Stop InfluxDB
sudo systemctl stop influxdb

# 2. Restore from backup
influx restore \
  --bucket foxhunt \
  --full /backup/influxdb/latest

# 3. Restart and verify
sudo systemctl start influxdb
influx query 'SHOW MEASUREMENTS'
```

### Scenario 3: Network Partition

#### Exchange Connectivity Loss
**Impact**: Unable to execute trades
**RTO**: 2 minutes (automatic failover)
**RPO**: 0 (no data loss)

**Recovery Steps**:
```bash
# 1. Verify primary connection status
./scripts/check-exchange-connectivity.sh

# 2. Activate backup connections
./scripts/activate-backup-exchange-routes.sh

# 3. Update broker configurations
./scripts/update-broker-routing.sh

# 4. Verify order execution capability
./scripts/test-order-execution.sh
```

#### Internet Connectivity Loss
**Impact**: Complete isolation from external services
**RTO**: 10 minutes
**RPO**: 5 minutes

**Recovery Steps**:
```bash
# 1. Switch to backup ISP
./scripts/activate-backup-isp.sh

# 2. Update routing tables
./scripts/update-network-routing.sh

# 3. Re-establish VPN connections
./scripts/reconnect-vpn.sh

# 4. Verify external connectivity
./scripts/verify-external-connectivity.sh
```

### Scenario 4: Security Breach

#### Unauthorized Access
**Impact**: Potential data compromise, trading halt required
**RTO**: 30 minutes
**RPO**: 0 (no data loss acceptable)

**Response Steps**:
```bash
# 1. Immediate containment
./scripts/security-lockdown.sh

# 2. Isolate compromised systems
sudo iptables -A INPUT -s <compromised_ip> -j DROP

# 3. Preserve forensic evidence
./scripts/preserve-evidence.sh

# 4. Reset all credentials
./scripts/reset-all-credentials.sh

# 5. Restore from clean backup
./scripts/restore-from-clean-backup.sh
```

### Scenario 5: Data Center Failure

#### Primary Data Center Loss
**Impact**: Complete system unavailability
**RTO**: 30 minutes
**RPO**: 5 minutes

**Recovery Steps**:
```bash
# 1. Activate disaster recovery site
./scripts/activate-dr-site.sh

# 2. Update DNS to point to DR site
./scripts/update-dns-to-dr.sh

# 3. Restore data from replicas
./scripts/restore-from-replicas.sh

# 4. Verify all services operational
./scripts/verify-dr-site-health.sh

# 5. Notify stakeholders
./scripts/notify-dr-activation.sh
```

## Recovery Strategies

### High Availability Architecture

```
Primary Data Center (DC1)          Secondary Data Center (DC2)
┌─────────────────────────────┐    ┌─────────────────────────────┐
│  Load Balancer (Active)     │    │  Load Balancer (Standby)    │
├─────────────────────────────┤    ├─────────────────────────────┤
│  Trading Servers (Active)   │    │  Trading Servers (Standby)  │
│  ├── Server-1 (Primary)     │    │  ├── Server-3 (Replica)     │
│  └── Server-2 (Replica)     │    │  └── Server-4 (Replica)     │
├─────────────────────────────┤    ├─────────────────────────────┤
│  Database Cluster           │    │  Database Cluster           │
│  ├── PostgreSQL Primary     │    │  ├── PostgreSQL Replica     │
│  ├── InfluxDB Primary       │    │  ├── InfluxDB Replica       │
│  └── Redis Primary          │    │  └── Redis Replica          │
└─────────────────────────────┘    └─────────────────────────────┘
              │                                    │
              └────────── Sync Replication ────────┘
```

### Replication Configuration

#### PostgreSQL Streaming Replication
```bash
# Primary server configuration
echo "wal_level = replica" >> /etc/postgresql/14/main/postgresql.conf
echo "max_wal_senders = 3" >> /etc/postgresql/14/main/postgresql.conf
echo "wal_keep_size = 1000" >> /etc/postgresql/14/main/postgresql.conf

# Replica server setup
pg_basebackup -h primary-server -D /var/lib/postgresql/14/replica -U replication -P -v -R
```

#### InfluxDB Replication
```bash
# Configure continuous queries for cross-datacenter replication
influx write --bucket foxhunt_replica \
  'FROM(bucket: "foxhunt") |> range(start: -1h) |> to(bucket: "foxhunt_replica", host: "dc2-influxdb")'
```

#### Redis Replication
```bash
# Configure Redis replica
echo "replicaof primary-redis 6379" >> /etc/redis/redis.conf
echo "replica-read-only yes" >> /etc/redis/redis.conf
```

## Backup Infrastructure

### Backup Strategy

#### Automated Backup Schedule
```bash
# /etc/cron.d/foxhunt-backups

# Full database backup (daily at 2 AM)
0 2 * * * foxhunt /usr/local/bin/full-backup.sh

# Incremental backup (every 4 hours)
0 */4 * * * foxhunt /usr/local/bin/incremental-backup.sh

# Configuration backup (daily at 3 AM)
0 3 * * * foxhunt /usr/local/bin/config-backup.sh

# Log backup (hourly)
0 * * * * foxhunt /usr/local/bin/log-backup.sh
```

#### Backup Storage Locations
1. **Local Storage**: Fast recovery, 7 days retention
2. **Network Storage**: Cross-datacenter, 30 days retention
3. **Cloud Storage**: Long-term archive, 1 year retention
4. **Offline Storage**: Compliance archive, 7 years retention

### Backup Verification

#### Automated Backup Testing
```bash
#!/bin/bash
# /usr/local/bin/verify-backups.sh

BACKUP_DATE=$(date +%Y%m%d)
TEST_DB="foxhunt_backup_test_$BACKUP_DATE"

# Test PostgreSQL backup
sudo -u postgres createdb $TEST_DB
sudo -u postgres pg_restore -d $TEST_DB /backup/postgresql/latest.dump

if [ $? -eq 0 ]; then
    echo "PostgreSQL backup verification: PASS"
    sudo -u postgres dropdb $TEST_DB
else
    echo "PostgreSQL backup verification: FAIL"
    exit 1
fi

# Test InfluxDB backup
influx restore --bucket test_bucket /backup/influxdb/latest
if [ $? -eq 0 ]; then
    echo "InfluxDB backup verification: PASS"
    influx delete --bucket test_bucket --start 1970-01-01T00:00:00Z --stop $(date -u +%Y-%m-%dT%H:%M:%SZ)
else
    echo "InfluxDB backup verification: FAIL"
    exit 1
fi

echo "All backup verifications passed"
```

## Recovery Procedures

### Automated Recovery Scripts

#### Database Recovery
```bash
#!/bin/bash
# /usr/local/bin/database-recovery.sh

BACKUP_TIMESTAMP=$1
RECOVERY_TYPE=${2:-full}  # full or point-in-time

case $RECOVERY_TYPE in
    "full")
        echo "Starting full database recovery..."
        
        # Stop services
        sudo systemctl stop foxhunt-*
        sudo systemctl stop postgresql influxdb redis-server
        
        # PostgreSQL recovery
        sudo -u postgres pg_restore \
            --clean --create --verbose \
            /backup/postgresql/$BACKUP_TIMESTAMP.dump
        
        # InfluxDB recovery
        influx restore --bucket foxhunt /backup/influxdb/$BACKUP_TIMESTAMP
        
        # Redis recovery
        sudo cp /backup/redis/$BACKUP_TIMESTAMP.rdb /var/lib/redis/dump.rdb
        sudo chown redis:redis /var/lib/redis/dump.rdb
        
        # Start services
        sudo systemctl start postgresql influxdb redis-server
        sudo systemctl start foxhunt-*
        ;;
        
    "point-in-time")
        echo "Starting point-in-time recovery..."
        # Implementation for PITR
        ;;
esac

# Verify recovery
./scripts/verify-database-integrity.sh
```

#### Application Recovery
```bash
#!/bin/bash
# /usr/local/bin/application-recovery.sh

DEPLOYMENT_TAG=${1:-latest}

echo "Starting application recovery with tag: $DEPLOYMENT_TAG"

# Stop current services
sudo systemctl stop foxhunt-*

# Backup current deployment
sudo cp -r /opt/foxhunt /opt/foxhunt.backup.$(date +%Y%m%d_%H%M%S)

# Deploy known good version
git checkout $DEPLOYMENT_TAG
cargo build --release

# Update systemd services
sudo cp deployment/systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload

# Start services in order
sudo systemctl start foxhunt-core
sudo systemctl start foxhunt-data
sudo systemctl start foxhunt-risk
sudo systemctl start foxhunt-ml
sudo systemctl start foxhunt-tli

# Verify deployment
./scripts/verify-application-health.sh
```

### Manual Recovery Procedures

#### Emergency Database Recovery
```sql
-- Check database connectivity
SELECT version();

-- Verify recent data
SELECT COUNT(*) FROM trades WHERE created_at > NOW() - INTERVAL '1 hour';

-- Check for corruption
SELECT COUNT(*) FROM pg_stat_database WHERE datname = 'foxhunt_production';

-- Rebuild indexes if needed
REINDEX DATABASE foxhunt_production;

-- Update statistics
ANALYZE;
```

#### Network Recovery
```bash
# Check network interfaces
ip addr show

# Test connectivity to exchanges
ping -c 5 exchanges.hostname.com

# Check routing table
ip route show

# Test DNS resolution
nslookup exchanges.hostname.com

# Verify firewall rules
sudo iptables -L -n

# Test application connectivity
curl -f http://localhost:8080/health
```

## Failover Systems

### Automatic Failover

#### Database Failover
```bash
#!/bin/bash
# Database failover script

PRIMARY_DB="primary-db"
REPLICA_DB="replica-db"

# Check primary database health
if ! pg_isready -h $PRIMARY_DB -p 5432; then
    echo "Primary database unreachable, initiating failover"
    
    # Promote replica to primary
    sudo -u postgres pg_promote -D /var/lib/postgresql/14/replica
    
    # Update application configuration
    sed -i "s/$PRIMARY_DB/$REPLICA_DB/g" .env.production
    
    # Restart applications
    sudo systemctl restart foxhunt-*
    
    # Update load balancer
    ./scripts/update-load-balancer-db.sh $REPLICA_DB
    
    echo "Database failover completed"
fi
```

#### Service Failover
```bash
#!/bin/bash
# Service failover monitoring

SERVICES=("foxhunt-core" "foxhunt-data" "foxhunt-risk" "foxhunt-ml" "foxhunt-tli")

for service in "${SERVICES[@]}"; do
    if ! systemctl is-active --quiet $service; then
        echo "Service $service is down, attempting restart"
        
        # Try restart first
        sudo systemctl restart $service
        sleep 10
        
        if systemctl is-active --quiet $service; then
            echo "Service $service restarted successfully"
        else
            echo "Service $service restart failed, escalating"
            ./scripts/escalate-service-failure.sh $service
        fi
    fi
done
```

### Load Balancer Configuration

#### HAProxy Configuration
```
# /etc/haproxy/haproxy.cfg
global
    maxconn 4096
    log stdout local0 debug

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend foxhunt_frontend
    bind *:80
    bind *:443 ssl crt /etc/ssl/certs/foxhunt.pem
    redirect scheme https if !{ ssl_fc }
    default_backend foxhunt_servers

backend foxhunt_servers
    balance roundrobin
    option httpchk GET /health
    server server1 10.0.1.10:8080 check
    server server2 10.0.1.11:8080 check backup
    server server3 10.0.2.10:8080 check backup
```

## Testing & Validation

### Disaster Recovery Testing Schedule

#### Monthly Tests
- **Backup Restoration**: Verify backup integrity and restoration time
- **Service Failover**: Test automatic failover mechanisms
- **Network Failover**: Validate network redundancy paths

#### Quarterly Tests
- **Full DR Drill**: Complete disaster recovery site activation
- **Security Incident Response**: Simulate security breach response
- **Cross-Datacenter Failover**: Test geographic failover

#### Annual Tests
- **Full Business Continuity**: End-to-end disaster simulation
- **Regulatory Compliance**: Audit trail and compliance verification
- **Performance Validation**: Ensure DR systems meet performance SLAs

### Testing Procedures

#### DR Site Activation Test
```bash
#!/bin/bash
# /usr/local/bin/dr-test.sh

echo "Starting DR site activation test"

# 1. Simulate primary site failure
./scripts/simulate-primary-failure.sh

# 2. Activate DR site
./scripts/activate-dr-site.sh

# 3. Test all services
./scripts/test-dr-services.sh

# 4. Validate data integrity
./scripts/validate-dr-data.sh

# 5. Performance testing
./scripts/performance-test-dr.sh

# 6. Failback to primary
./scripts/failback-to-primary.sh

echo "DR test completed"
```

#### Recovery Time Testing
```bash
#!/bin/bash
# Measure actual recovery times

START_TIME=$(date +%s)

# Simulate failure
./scripts/simulate-database-failure.sh

# Execute recovery
./scripts/database-recovery.sh latest

# Measure recovery time
END_TIME=$(date +%s)
RECOVERY_TIME=$((END_TIME - START_TIME))

echo "Database recovery time: ${RECOVERY_TIME} seconds"

# Log results for trending
echo "$(date),$RECOVERY_TIME,database_recovery" >> /var/log/foxhunt/recovery_metrics.csv
```

## Communication Plans

### Stakeholder Notification

#### Internal Notifications
```bash
#!/bin/bash
# /usr/local/bin/notify-stakeholders.sh

INCIDENT_LEVEL=$1  # critical, major, minor
MESSAGE=$2

case $INCIDENT_LEVEL in
    "critical")
        # Immediate notification to all stakeholders
        ./scripts/send-sms.sh "CRITICAL: $MESSAGE" "+1-555-0101,+1-555-0102,+1-555-0103"
        ./scripts/send-email.sh "CRITICAL: Foxhunt System Alert" "$MESSAGE" "ops-team@foxhunt.com"
        ./scripts/post-slack.sh "#critical-alerts" "🚨 CRITICAL: $MESSAGE"
        ;;
    "major")
        # Email and Slack notification
        ./scripts/send-email.sh "MAJOR: Foxhunt System Alert" "$MESSAGE" "ops-team@foxhunt.com"
        ./scripts/post-slack.sh "#alerts" "⚠️ MAJOR: $MESSAGE"
        ;;
    "minor")
        # Slack notification only
        ./scripts/post-slack.sh "#monitoring" "ℹ️ MINOR: $MESSAGE"
        ;;
esac
```

#### External Notifications
```bash
#!/bin/bash
# External stakeholder notification

OUTAGE_TYPE=$1
ESTIMATED_RESOLUTION=$2

# Notify brokers of potential impact
if [[ "$OUTAGE_TYPE" == "trading" ]]; then
    ./scripts/notify-brokers.sh "Trading system maintenance in progress. ETA: $ESTIMATED_RESOLUTION"
fi

# Notify regulatory bodies if required
if [[ "$OUTAGE_TYPE" == "critical" ]]; then
    ./scripts/notify-regulators.sh "System outage reported. Recovery in progress."
fi

# Update status page
./scripts/update-status-page.sh "$OUTAGE_TYPE" "$ESTIMATED_RESOLUTION"
```

### Communication Templates

#### Critical Incident Template
```
Subject: [CRITICAL] Foxhunt Trading System Incident

Incident Summary:
- Start Time: [TIMESTAMP]
- Impact: [DESCRIPTION]
- Affected Services: [LIST]
- Current Status: [STATUS]

Actions Taken:
1. [ACTION 1]
2. [ACTION 2]
3. [ACTION 3]

Next Steps:
- [NEXT ACTION]
- [ETA]

Recovery Status: [PERCENTAGE]%
Estimated Resolution: [TIMESTAMP]

Incident Commander: [NAME]
Contact: [PHONE/EMAIL]
```

## Post-Recovery Actions

### System Validation

#### Post-Recovery Checklist
```bash
#!/bin/bash
# /usr/local/bin/post-recovery-validation.sh

echo "Starting post-recovery validation"

# 1. System health check
./scripts/comprehensive-health-check.sh

# 2. Performance validation
./scripts/performance-baseline-test.sh

# 3. Data integrity check
./scripts/data-integrity-validation.sh

# 4. Security validation
./scripts/security-posture-check.sh

# 5. Functionality testing
./scripts/end-to-end-functional-test.sh

# 6. Generate recovery report
./scripts/generate-recovery-report.sh

echo "Post-recovery validation completed"
```

### Root Cause Analysis

#### Incident Documentation
```bash
#!/bin/bash
# Generate incident report

INCIDENT_ID=$1
INCIDENT_START=$2
INCIDENT_END=$3

cat > /var/log/foxhunt/incidents/incident_${INCIDENT_ID}.md << EOF
# Incident Report: $INCIDENT_ID

## Summary
- **Start Time**: $INCIDENT_START
- **End Time**: $INCIDENT_END
- **Duration**: $(date -d "$INCIDENT_END" +%s) - $(date -d "$INCIDENT_START" +%s) seconds
- **Impact**: [DESCRIPTION]

## Timeline
$(grep "$INCIDENT_START" /var/log/foxhunt/*.log | head -20)

## Root Cause
[ANALYSIS]

## Resolution
[STEPS TAKEN]

## Prevention Measures
[FUTURE IMPROVEMENTS]

## Lessons Learned
[KEY TAKEAWAYS]
EOF
```

### Performance Monitoring

#### Recovery Performance Metrics
```sql
-- Monitor system performance post-recovery
SELECT 
    date_trunc('minute', timestamp) as minute,
    avg(latency_ns) as avg_latency,
    max(latency_ns) as max_latency,
    count(*) as operation_count
FROM performance_metrics 
WHERE timestamp > NOW() - INTERVAL '1 hour'
GROUP BY minute
ORDER BY minute;
```

### Continuous Improvement

#### DR Plan Updates
```bash
#!/bin/bash
# Update DR procedures based on lessons learned

# 1. Review incident reports
./scripts/analyze-incident-trends.sh

# 2. Update RTO/RPO targets if needed
./scripts/update-recovery-objectives.sh

# 3. Enhance automation scripts
./scripts/improve-automation.sh

# 4. Update documentation
git add docs/DISASTER_RECOVERY.md
git commit -m "Update DR procedures based on incident $INCIDENT_ID"

# 5. Schedule additional training
./scripts/schedule-dr-training.sh
```

This disaster recovery plan provides comprehensive procedures for handling various failure scenarios while meeting strict RTO and RPO requirements for the Foxhunt HFT trading system. Regular testing and continuous improvement ensure the plan remains effective and current.