foxhunt/docs/DISASTER_RECOVERY.md

Foxhunt HFT Trading System - Disaster Recovery Procedures

Table of Contents

1. Overview
2. Recovery Objectives
3. Disaster Scenarios
4. Recovery Strategies
5. Backup Infrastructure
6. Recovery Procedures
7. Failover Systems
8. Testing & Validation
9. Communication Plans
10. Post-Recovery Actions

Overview

This document outlines comprehensive disaster recovery procedures for the Foxhunt HFT trading system. The plan ensures business continuity with minimal downtime and data loss in the event of various disaster scenarios.

Scope

• Primary Trading Systems: Core trading infrastructure
• Data Storage: All databases and persistent storage
• Network Infrastructure: Connectivity and routing
• Security Systems: Authentication and authorization
• Monitoring & Alerting: Observability infrastructure

Responsibilities

• Incident Commander: Overall response coordination
• Technical Lead: System recovery execution
• Database Administrator: Data recovery and integrity
• Network Engineer: Connectivity restoration
• Security Officer: Security validation and compliance
• Communications Lead: Stakeholder notifications

Recovery Objectives

Recovery Time Objective (RTO)

• Critical Trading Systems: 15 minutes
• Database Systems: 10 minutes
• Monitoring Systems: 30 minutes
• Full System Restoration: 60 minutes

Recovery Point Objective (RPO)

• Transaction Data: 1 minute (maximum data loss)
• Market Data: 5 minutes
• Configuration Data: 15 minutes
• Log Data: 1 hour

Service Level Objectives

• System Availability: 99.95% uptime
• Data Integrity: 100% (zero data corruption)
• Performance: <10% degradation post-recovery
• Security: Full security controls operational

Disaster Scenarios

Scenario 1: Hardware Failure

Single Server Failure

Impact: Reduced capacity, potential service degradation RTO: 5 minutes (automatic failover) RPO: 30 seconds

Immediate Actions:

# 1. Verify failover activation
./scripts/verify-failover-status.sh

# 2. Check load balancer status
curl -f http://load-balancer:8080/health

# 3. Monitor performance metrics
./scripts/monitor-failover-performance.sh

# 4. Notify operations team
./scripts/send-alert.sh "WARN: Server failover activated"

Multiple Server Failure

Impact: Significant service disruption RTO: 15 minutes RPO: 5 minutes

Recovery Steps:

# 1. Assess scope of failure
./scripts/assess-infrastructure-status.sh

# 2. Activate secondary data center
./scripts/activate-secondary-datacenter.sh

# 3. Update DNS records
./scripts/update-dns-failover.sh

# 4. Verify service restoration
./scripts/verify-full-system-health.sh

Scenario 2: Database Corruption

PostgreSQL Corruption

Impact: Transaction data unavailable RTO: 10 minutes RPO: 1 minute

Recovery Steps:

# 1. Stop database connections
sudo systemctl stop foxhunt-*
sudo systemctl stop postgresql

# 2. Assess corruption extent
sudo -u postgres pg_checksums -D /var/lib/postgresql/14/main

# 3. Restore from backup
sudo -u postgres pg_restore \
  --clean --create --verbose \
  /backup/postgresql/latest.dump

# 4. Verify data integrity
sudo -u postgres psql -c "SELECT COUNT(*) FROM trades WHERE created_at > NOW() - INTERVAL '1 hour';"

# 5. Restart services
sudo systemctl start postgresql
sudo systemctl start foxhunt-*

InfluxDB Data Loss

Impact: Historical metrics unavailable RTO: 5 minutes RPO: 15 minutes

Recovery Steps:

# 1. Stop InfluxDB
sudo systemctl stop influxdb

# 2. Restore from backup
influx restore \
  --bucket foxhunt \
  --full /backup/influxdb/latest

# 3. Restart and verify
sudo systemctl start influxdb
influx query 'SHOW MEASUREMENTS'

Scenario 3: Network Partition

Exchange Connectivity Loss

Impact: Unable to execute trades RTO: 2 minutes (automatic failover) RPO: 0 (no data loss)

Recovery Steps:

# 1. Verify primary connection status
./scripts/check-exchange-connectivity.sh

# 2. Activate backup connections
./scripts/activate-backup-exchange-routes.sh

# 3. Update broker configurations
./scripts/update-broker-routing.sh

# 4. Verify order execution capability
./scripts/test-order-execution.sh

Internet Connectivity Loss

Impact: Complete isolation from external services RTO: 10 minutes RPO: 5 minutes

Recovery Steps:

# 1. Switch to backup ISP
./scripts/activate-backup-isp.sh

# 2. Update routing tables
./scripts/update-network-routing.sh

# 3. Re-establish VPN connections
./scripts/reconnect-vpn.sh

# 4. Verify external connectivity
./scripts/verify-external-connectivity.sh

Scenario 4: Security Breach

Unauthorized Access

Impact: Potential data compromise, trading halt required RTO: 30 minutes RPO: 0 (no data loss acceptable)

Response Steps:

# 1. Immediate containment
./scripts/security-lockdown.sh

# 2. Isolate compromised systems
sudo iptables -A INPUT -s <compromised_ip> -j DROP

# 3. Preserve forensic evidence
./scripts/preserve-evidence.sh

# 4. Reset all credentials
./scripts/reset-all-credentials.sh

# 5. Restore from clean backup
./scripts/restore-from-clean-backup.sh

Scenario 5: Data Center Failure

Primary Data Center Loss

Impact: Complete system unavailability RTO: 30 minutes RPO: 5 minutes

Recovery Steps:

# 1. Activate disaster recovery site
./scripts/activate-dr-site.sh

# 2. Update DNS to point to DR site
./scripts/update-dns-to-dr.sh

# 3. Restore data from replicas
./scripts/restore-from-replicas.sh

# 4. Verify all services operational
./scripts/verify-dr-site-health.sh

# 5. Notify stakeholders
./scripts/notify-dr-activation.sh

Recovery Strategies

High Availability Architecture

Primary Data Center (DC1)          Secondary Data Center (DC2)
┌─────────────────────────────┐    ┌─────────────────────────────┐
│  Load Balancer (Active)     │    │  Load Balancer (Standby)    │
├─────────────────────────────┤    ├─────────────────────────────┤
│  Trading Servers (Active)   │    │  Trading Servers (Standby)  │
│  ├── Server-1 (Primary)     │    │  ├── Server-3 (Replica)     │
│  └── Server-2 (Replica)     │    │  └── Server-4 (Replica)     │
├─────────────────────────────┤    ├─────────────────────────────┤
│  Database Cluster           │    │  Database Cluster           │
│  ├── PostgreSQL Primary     │    │  ├── PostgreSQL Replica     │
│  ├── InfluxDB Primary       │    │  ├── InfluxDB Replica       │
│  └── Redis Primary          │    │  └── Redis Replica          │
└─────────────────────────────┘    └─────────────────────────────┘
              │                                    │
              └────────── Sync Replication ────────┘

Replication Configuration

PostgreSQL Streaming Replication

# Primary server configuration
echo "wal_level = replica" >> /etc/postgresql/14/main/postgresql.conf
echo "max_wal_senders = 3" >> /etc/postgresql/14/main/postgresql.conf
echo "wal_keep_size = 1000" >> /etc/postgresql/14/main/postgresql.conf

# Replica server setup
pg_basebackup -h primary-server -D /var/lib/postgresql/14/replica -U replication -P -v -R

InfluxDB Replication

# Configure continuous queries for cross-datacenter replication
influx write --bucket foxhunt_replica \
  'FROM(bucket: "foxhunt") |> range(start: -1h) |> to(bucket: "foxhunt_replica", host: "dc2-influxdb")'

Redis Replication

# Configure Redis replica
echo "replicaof primary-redis 6379" >> /etc/redis/redis.conf
echo "replica-read-only yes" >> /etc/redis/redis.conf

Backup Infrastructure

Backup Strategy

Automated Backup Schedule

# /etc/cron.d/foxhunt-backups

# Full database backup (daily at 2 AM)
0 2 * * * foxhunt /usr/local/bin/full-backup.sh

# Incremental backup (every 4 hours)
0 */4 * * * foxhunt /usr/local/bin/incremental-backup.sh

# Configuration backup (daily at 3 AM)
0 3 * * * foxhunt /usr/local/bin/config-backup.sh

# Log backup (hourly)
0 * * * * foxhunt /usr/local/bin/log-backup.sh

Backup Storage Locations

1. Local Storage: Fast recovery, 7 days retention
2. Network Storage: Cross-datacenter, 30 days retention
3. Cloud Storage: Long-term archive, 1 year retention
4. Offline Storage: Compliance archive, 7 years retention

Backup Verification

Automated Backup Testing

#!/bin/bash
# /usr/local/bin/verify-backups.sh

BACKUP_DATE=$(date +%Y%m%d)
TEST_DB="foxhunt_backup_test_$BACKUP_DATE"

# Test PostgreSQL backup
sudo -u postgres createdb $TEST_DB
sudo -u postgres pg_restore -d $TEST_DB /backup/postgresql/latest.dump

if [ $? -eq 0 ]; then
    echo "PostgreSQL backup verification: PASS"
    sudo -u postgres dropdb $TEST_DB
else
    echo "PostgreSQL backup verification: FAIL"
    exit 1
fi

# Test InfluxDB backup
influx restore --bucket test_bucket /backup/influxdb/latest
if [ $? -eq 0 ]; then
    echo "InfluxDB backup verification: PASS"
    influx delete --bucket test_bucket --start 1970-01-01T00:00:00Z --stop $(date -u +%Y-%m-%dT%H:%M:%SZ)
else
    echo "InfluxDB backup verification: FAIL"
    exit 1
fi

echo "All backup verifications passed"

Recovery Procedures

Automated Recovery Scripts

Database Recovery

#!/bin/bash
# /usr/local/bin/database-recovery.sh

BACKUP_TIMESTAMP=$1
RECOVERY_TYPE=${2:-full}  # full or point-in-time

case $RECOVERY_TYPE in
    "full")
        echo "Starting full database recovery..."
        
        # Stop services
        sudo systemctl stop foxhunt-*
        sudo systemctl stop postgresql influxdb redis-server
        
        # PostgreSQL recovery
        sudo -u postgres pg_restore \
            --clean --create --verbose \
            /backup/postgresql/$BACKUP_TIMESTAMP.dump
        
        # InfluxDB recovery
        influx restore --bucket foxhunt /backup/influxdb/$BACKUP_TIMESTAMP
        
        # Redis recovery
        sudo cp /backup/redis/$BACKUP_TIMESTAMP.rdb /var/lib/redis/dump.rdb
        sudo chown redis:redis /var/lib/redis/dump.rdb
        
        # Start services
        sudo systemctl start postgresql influxdb redis-server
        sudo systemctl start foxhunt-*
        ;;
        
    "point-in-time")
        echo "Starting point-in-time recovery..."
        # Implementation for PITR
        ;;
esac

# Verify recovery
./scripts/verify-database-integrity.sh

Application Recovery

#!/bin/bash
# /usr/local/bin/application-recovery.sh

DEPLOYMENT_TAG=${1:-latest}

echo "Starting application recovery with tag: $DEPLOYMENT_TAG"

# Stop current services
sudo systemctl stop foxhunt-*

# Backup current deployment
sudo cp -r /opt/foxhunt /opt/foxhunt.backup.$(date +%Y%m%d_%H%M%S)

# Deploy known good version
git checkout $DEPLOYMENT_TAG
cargo build --release

# Update systemd services
sudo cp deployment/systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload

# Start services in order
sudo systemctl start foxhunt-core
sudo systemctl start foxhunt-data
sudo systemctl start foxhunt-risk
sudo systemctl start foxhunt-ml
sudo systemctl start foxhunt-tli

# Verify deployment
./scripts/verify-application-health.sh

Manual Recovery Procedures

Emergency Database Recovery

-- Check database connectivity
SELECT version();

-- Verify recent data
SELECT COUNT(*) FROM trades WHERE created_at > NOW() - INTERVAL '1 hour';

-- Check for corruption
SELECT COUNT(*) FROM pg_stat_database WHERE datname = 'foxhunt_production';

-- Rebuild indexes if needed
REINDEX DATABASE foxhunt_production;

-- Update statistics
ANALYZE;

Network Recovery

# Check network interfaces
ip addr show

# Test connectivity to exchanges
ping -c 5 exchanges.hostname.com

# Check routing table
ip route show

# Test DNS resolution
nslookup exchanges.hostname.com

# Verify firewall rules
sudo iptables -L -n

# Test application connectivity
curl -f http://localhost:8080/health

Failover Systems

Automatic Failover

Database Failover

#!/bin/bash
# Database failover script

PRIMARY_DB="primary-db"
REPLICA_DB="replica-db"

# Check primary database health
if ! pg_isready -h $PRIMARY_DB -p 5432; then
    echo "Primary database unreachable, initiating failover"
    
    # Promote replica to primary
    sudo -u postgres pg_promote -D /var/lib/postgresql/14/replica
    
    # Update application configuration
    sed -i "s/$PRIMARY_DB/$REPLICA_DB/g" .env.production
    
    # Restart applications
    sudo systemctl restart foxhunt-*
    
    # Update load balancer
    ./scripts/update-load-balancer-db.sh $REPLICA_DB
    
    echo "Database failover completed"
fi

Service Failover

#!/bin/bash
# Service failover monitoring

SERVICES=("foxhunt-core" "foxhunt-data" "foxhunt-risk" "foxhunt-ml" "foxhunt-tli")

for service in "${SERVICES[@]}"; do
    if ! systemctl is-active --quiet $service; then
        echo "Service $service is down, attempting restart"
        
        # Try restart first
        sudo systemctl restart $service
        sleep 10
        
        if systemctl is-active --quiet $service; then
            echo "Service $service restarted successfully"
        else
            echo "Service $service restart failed, escalating"
            ./scripts/escalate-service-failure.sh $service
        fi
    fi
done

Load Balancer Configuration

HAProxy Configuration

# /etc/haproxy/haproxy.cfg
global
    maxconn 4096
    log stdout local0 debug

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend foxhunt_frontend
    bind *:80
    bind *:443 ssl crt /etc/ssl/certs/foxhunt.pem
    redirect scheme https if !{ ssl_fc }
    default_backend foxhunt_servers

backend foxhunt_servers
    balance roundrobin
    option httpchk GET /health
    server server1 10.0.1.10:8080 check
    server server2 10.0.1.11:8080 check backup
    server server3 10.0.2.10:8080 check backup

Testing & Validation

Disaster Recovery Testing Schedule

Monthly Tests

• Backup Restoration: Verify backup integrity and restoration time
• Service Failover: Test automatic failover mechanisms
• Network Failover: Validate network redundancy paths

Quarterly Tests

• Full DR Drill: Complete disaster recovery site activation
• Security Incident Response: Simulate security breach response
• Cross-Datacenter Failover: Test geographic failover

Annual Tests

• Full Business Continuity: End-to-end disaster simulation
• Regulatory Compliance: Audit trail and compliance verification
• Performance Validation: Ensure DR systems meet performance SLAs

Testing Procedures

DR Site Activation Test

#!/bin/bash
# /usr/local/bin/dr-test.sh

echo "Starting DR site activation test"

# 1. Simulate primary site failure
./scripts/simulate-primary-failure.sh

# 2. Activate DR site
./scripts/activate-dr-site.sh

# 3. Test all services
./scripts/test-dr-services.sh

# 4. Validate data integrity
./scripts/validate-dr-data.sh

# 5. Performance testing
./scripts/performance-test-dr.sh

# 6. Failback to primary
./scripts/failback-to-primary.sh

echo "DR test completed"

Recovery Time Testing

#!/bin/bash
# Measure actual recovery times

START_TIME=$(date +%s)

# Simulate failure
./scripts/simulate-database-failure.sh

# Execute recovery
./scripts/database-recovery.sh latest

# Measure recovery time
END_TIME=$(date +%s)
RECOVERY_TIME=$((END_TIME - START_TIME))

echo "Database recovery time: ${RECOVERY_TIME} seconds"

# Log results for trending
echo "$(date),$RECOVERY_TIME,database_recovery" >> /var/log/foxhunt/recovery_metrics.csv

Communication Plans

Stakeholder Notification

Internal Notifications

#!/bin/bash
# /usr/local/bin/notify-stakeholders.sh

INCIDENT_LEVEL=$1  # critical, major, minor
MESSAGE=$2

case $INCIDENT_LEVEL in
    "critical")
        # Immediate notification to all stakeholders
        ./scripts/send-sms.sh "CRITICAL: $MESSAGE" "+1-555-0101,+1-555-0102,+1-555-0103"
        ./scripts/send-email.sh "CRITICAL: Foxhunt System Alert" "$MESSAGE" "ops-team@foxhunt.com"
        ./scripts/post-slack.sh "#critical-alerts" "🚨 CRITICAL: $MESSAGE"
        ;;
    "major")
        # Email and Slack notification
        ./scripts/send-email.sh "MAJOR: Foxhunt System Alert" "$MESSAGE" "ops-team@foxhunt.com"
        ./scripts/post-slack.sh "#alerts" "⚠️ MAJOR: $MESSAGE"
        ;;
    "minor")
        # Slack notification only
        ./scripts/post-slack.sh "#monitoring" "ℹ️ MINOR: $MESSAGE"
        ;;
esac

External Notifications

#!/bin/bash
# External stakeholder notification

OUTAGE_TYPE=$1
ESTIMATED_RESOLUTION=$2

# Notify brokers of potential impact
if [[ "$OUTAGE_TYPE" == "trading" ]]; then
    ./scripts/notify-brokers.sh "Trading system maintenance in progress. ETA: $ESTIMATED_RESOLUTION"
fi

# Notify regulatory bodies if required
if [[ "$OUTAGE_TYPE" == "critical" ]]; then
    ./scripts/notify-regulators.sh "System outage reported. Recovery in progress."
fi

# Update status page
./scripts/update-status-page.sh "$OUTAGE_TYPE" "$ESTIMATED_RESOLUTION"

Communication Templates

Critical Incident Template

Subject: [CRITICAL] Foxhunt Trading System Incident

Incident Summary:
- Start Time: [TIMESTAMP]
- Impact: [DESCRIPTION]
- Affected Services: [LIST]
- Current Status: [STATUS]

Actions Taken:
1. [ACTION 1]
2. [ACTION 2]
3. [ACTION 3]

Next Steps:
- [NEXT ACTION]
- [ETA]

Recovery Status: [PERCENTAGE]%
Estimated Resolution: [TIMESTAMP]

Incident Commander: [NAME]
Contact: [PHONE/EMAIL]

Post-Recovery Actions

System Validation

Post-Recovery Checklist

#!/bin/bash
# /usr/local/bin/post-recovery-validation.sh

echo "Starting post-recovery validation"

# 1. System health check
./scripts/comprehensive-health-check.sh

# 2. Performance validation
./scripts/performance-baseline-test.sh

# 3. Data integrity check
./scripts/data-integrity-validation.sh

# 4. Security validation
./scripts/security-posture-check.sh

# 5. Functionality testing
./scripts/end-to-end-functional-test.sh

# 6. Generate recovery report
./scripts/generate-recovery-report.sh

echo "Post-recovery validation completed"

Root Cause Analysis

Incident Documentation

#!/bin/bash
# Generate incident report

INCIDENT_ID=$1
INCIDENT_START=$2
INCIDENT_END=$3

cat > /var/log/foxhunt/incidents/incident_${INCIDENT_ID}.md << EOF
# Incident Report: $INCIDENT_ID

## Summary
- **Start Time**: $INCIDENT_START
- **End Time**: $INCIDENT_END
- **Duration**: $(date -d "$INCIDENT_END" +%s) - $(date -d "$INCIDENT_START" +%s) seconds
- **Impact**: [DESCRIPTION]

## Timeline
$(grep "$INCIDENT_START" /var/log/foxhunt/*.log | head -20)

## Root Cause
[ANALYSIS]

## Resolution
[STEPS TAKEN]

## Prevention Measures
[FUTURE IMPROVEMENTS]

## Lessons Learned
[KEY TAKEAWAYS]
EOF

Performance Monitoring

Recovery Performance Metrics

-- Monitor system performance post-recovery
SELECT 
    date_trunc('minute', timestamp) as minute,
    avg(latency_ns) as avg_latency,
    max(latency_ns) as max_latency,
    count(*) as operation_count
FROM performance_metrics 
WHERE timestamp > NOW() - INTERVAL '1 hour'
GROUP BY minute
ORDER BY minute;

Continuous Improvement

DR Plan Updates

#!/bin/bash
# Update DR procedures based on lessons learned

# 1. Review incident reports
./scripts/analyze-incident-trends.sh

# 2. Update RTO/RPO targets if needed
./scripts/update-recovery-objectives.sh

# 3. Enhance automation scripts
./scripts/improve-automation.sh

# 4. Update documentation
git add docs/DISASTER_RECOVERY.md
git commit -m "Update DR procedures based on incident $INCIDENT_ID"

# 5. Schedule additional training
./scripts/schedule-dr-training.sh

This disaster recovery plan provides comprehensive procedures for handling various failure scenarios while meeting strict RTO and RPO requirements for the Foxhunt HFT trading system. Regular testing and continuous improvement ensure the plan remains effective and current.