foxhunt/docs/archive/historical/EMERGENCY_PROCEDURES.md

Emergency Procedures - Foxhunt HFT Trading System

Purpose: Critical incident response procedures for immediate action Version: 1.0.0 Last Updated: 2025-10-07

---

🚨 EMERGENCY CONTACTS

Fill in before production deployment:

Role	Name	Phone	Email	Escalation Time
On-Call Engineer	[CONFIGURE]	[CONFIGURE]	[CONFIGURE]	Immediate
Database Admin	[CONFIGURE]	[CONFIGURE]	[CONFIGURE]	5 minutes
Security Lead	[CONFIGURE]	[CONFIGURE]	[CONFIGURE]	10 minutes
CTO/Technical Lead	[CONFIGURE]	[CONFIGURE]	[CONFIGURE]	15 minutes
CEO	[CONFIGURE]	[CONFIGURE]	[CONFIGURE]	30 minutes

PagerDuty: [CONFIGURE] Slack Channel: #foxhunt-incidents

---

🎯 Severity Levels

SEV-1: CRITICAL (Immediate Response)

• Impact: Trading completely halted, data loss, security breach
• Response Time: <5 minutes
• Examples: System down, database corruption, unauthorized access
• Escalation: Immediate page to on-call + CTO

SEV-2: HIGH (Urgent Response)

• Impact: Significant degradation, trading impaired but operational
• Response Time: <15 minutes
• Examples: High latency (>500μs), service failure, data inconsistency
• Escalation: Page on-call, notify team in Slack

SEV-3: MEDIUM (Priority Response)

• Impact: Minor degradation, non-critical component failure
• Response Time: <1 hour
• Examples: Non-critical service down, elevated error rate
• Escalation: Slack notification, log ticket

SEV-4: LOW (Standard Response)

• Impact: No immediate impact, potential future issue
• Response Time: <4 hours (business hours)
• Examples: Warning alerts, resource usage trends
• Escalation: Standard ticket

---

🔴 CRITICAL SCENARIOS (SEV-1)

Scenario 1: System-Wide Trading Halt

Trigger: All trading stopped, no orders processing Response Time: <2 minutes to activate kill switch, <15 minutes to diagnose

Immediate Actions (First 60 seconds)

# 1. CONFIRM HALT - Check if trading actually stopped
curl http://localhost:9092/metrics | grep trading_active
# Expected: trading_active 0

# 2. ACTIVATE KILL SWITCH (if not already active)
export KILL_SWITCH_TOKEN="your-master-token"
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

# 3. VERIFY KILL SWITCH ACTIVE
curl http://localhost:8080/status
# Expected: {"status": "HALTED", "reason": "Emergency kill switch activated"}

# 4. NOTIFY STAKEHOLDERS
./deployment/scripts/send-emergency-alert.sh "Trading halted - investigating"

Diagnosis (Next 5 minutes)

# Check service status
docker-compose ps  # Docker
sudo systemctl status foxhunt-*  # Bare-metal

# Check recent errors
docker-compose logs --tail=100 trading_service | grep -i error
sudo journalctl -u foxhunt-trading --since "5 minutes ago" | grep -i error

# Check system resources
df -h  # Disk space
free -m  # Memory
uptime  # Load average

# Check database connectivity
psql $DATABASE_URL -c "SELECT 1;" || echo "❌ Database connection failed"

# Check network connectivity
ping -c 3 <exchange-server> || echo "❌ Network connectivity failed"

Decision Tree

If database is down:

# See: Scenario 2 - Database Failure

If service crashed:

# See: Scenario 3 - Service Crash

If network issue:

# See: Scenario 5 - Network Failure

If cause unknown:

# 1. Restart services in safe mode (read-only)
./deployment/scripts/restart-safe-mode.sh

# 2. Escalate to CTO
./deployment/scripts/escalate.sh --level=cto --message="Trading halt, cause unknown"

# 3. Begin detailed investigation
./deployment/scripts/capture-diagnostics.sh > /var/log/foxhunt/incident-$(date +%s).log

Recovery Procedure

# 1. Fix root cause (see specific scenarios)

# 2. Run validation tests
./deployment/scripts/pre-trading-validation.sh

# 3. Resume trading (only after validation passes)
curl -X POST http://localhost:8080/trading/resume \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Issue resolved - verified", "approved_by": "your-name"}'

# 4. Monitor closely for 30 minutes
watch -n 10 'curl -s http://localhost:9092/metrics | grep -E "(order_latency|error_rate)"'

# 5. Send all-clear notification
./deployment/scripts/send-emergency-alert.sh "Trading resumed - incident resolved"

---

Scenario 2: Database Failure

Trigger: Cannot connect to PostgreSQL Response Time: <5 minutes to failover

Immediate Actions

# 1. ACTIVATE KILL SWITCH
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

# 2. CHECK DATABASE STATUS
sudo systemctl status postgresql  # Bare-metal
docker-compose ps postgres  # Docker

# 3. CHECK REPLICATION STATUS (if configured)
psql -h replica-server -U foxhunt -c "SELECT pg_is_in_recovery();"
# Expected: t (true = replica)

Recovery Options

Option A: Database is Running but Unresponsive

# 1. Check connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# 2. Kill long-running queries
psql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '5 minutes';"

# 3. Restart PostgreSQL
sudo systemctl restart postgresql  # Bare-metal
docker-compose restart postgres  # Docker

# 4. Wait for startup
sleep 10

# 5. Test connection
psql $DATABASE_URL -c "SELECT 1;"

Option B: Database Crashed - Restore from Replica

# 1. Promote replica to primary
ssh replica-server "sudo -u postgres /usr/lib/postgresql/16/bin/pg_ctl promote -D /var/lib/postgresql/16/main"

# 2. Update connection strings
export NEW_DB_URL=postgresql://foxhunt:password@replica-server:5432/foxhunt

# 3. Update services
docker-compose down
sed -i "s|DATABASE_URL=.*|DATABASE_URL=$NEW_DB_URL|" .env
docker-compose up -d

# OR for bare-metal
sudo systemctl stop foxhunt-*
sudo systemctl set-environment DATABASE_URL="$NEW_DB_URL"
sudo systemctl start foxhunt-*

# 4. Verify connectivity
psql $NEW_DB_URL -c "SELECT count(*) FROM orders;"

Option C: Complete Restore from Backup

# 1. Download latest backup from S3
aws s3 cp s3://foxhunt-backups/postgres/latest.dump /tmp/restore.dump

# 2. Create new database
createdb foxhunt_restored

# 3. Restore
pg_restore -h localhost -U foxhunt -d foxhunt_restored -v /tmp/restore.dump

# 4. Swap databases
psql -U postgres <<EOF
ALTER DATABASE foxhunt RENAME TO foxhunt_old;
ALTER DATABASE foxhunt_restored RENAME TO foxhunt;
EOF

# 5. Restart services
sudo systemctl restart foxhunt-*

# Estimated downtime: 5-15 minutes (depends on backup size)

---

Scenario 3: Service Crash/Unresponsive

Trigger: Service health check failing, not responding Response Time: <1 minute to restart

Immediate Actions

# 1. IDENTIFY CRASHED SERVICE
docker-compose ps | grep -v "Up"  # Docker
sudo systemctl status foxhunt-* | grep failed  # Bare-metal

# 2. CHECK LOGS FOR CRASH REASON
docker-compose logs --tail=100 <service> | grep -E "(panic|error|fatal)"
sudo journalctl -u foxhunt-<service> -n 100 | grep -E "(panic|error|fatal)"

# 3. RESTART SERVICE
docker-compose restart <service>  # Docker
sudo systemctl restart foxhunt-<service>  # Bare-metal

# 4. VERIFY RESTART SUCCESSFUL
docker-compose ps <service>
sudo systemctl status foxhunt-<service>

# 5. CHECK FOR ERRORS AFTER RESTART
docker-compose logs --tail=50 <service>
sudo journalctl -u foxhunt-<service> -n 50

If Restart Fails

# 1. Check for port conflicts
sudo lsof -i :50052  # Trading service port
sudo kill -9 <PID>  # Kill conflicting process

# 2. Check disk space
df -h
# If >90% full, clean up logs
sudo find /var/log/foxhunt -name "*.log" -mtime +7 -delete

# 3. Check memory
free -m
# If memory exhausted, identify memory hog
ps aux --sort=-%mem | head -10

# 4. Try safe mode restart
docker-compose stop <service>
docker-compose up -d <service> --force-recreate

# OR bare-metal
sudo systemctl stop foxhunt-<service>
sudo rm -rf /tmp/foxhunt/*  # Clear temp files
sudo systemctl start foxhunt-<service>

# 5. If still failing, rollback
sudo cp /opt/foxhunt/bin/<service>.backup /opt/foxhunt/bin/<service>
sudo systemctl restart foxhunt-<service>

---

Scenario 4: Security Breach Detected

Trigger: Unauthorized access, data exfiltration, suspicious activity Response Time: IMMEDIATE - <30 seconds to isolate

Immediate Actions (CRITICAL - Do NOT delay)

# 1. ACTIVATE KILL SWITCH IMMEDIATELY
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

# 2. ISOLATE SYSTEM (disconnect from network)
# Bare-metal
sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
sudo iptables -P FORWARD DROP

# Docker
docker network disconnect foxhunt_foxhunt-network <container>

# 3. CAPTURE FORENSIC DATA
./deployment/scripts/capture-forensics.sh > /var/log/foxhunt/security-incident-$(date +%s).log

# 4. NOTIFY SECURITY TEAM IMMEDIATELY
./deployment/scripts/security-alert.sh --severity=critical --message="Security breach detected - system isolated"

# 5. PRESERVE EVIDENCE
sudo cp -r /var/log/foxhunt /var/log/foxhunt.incident-$(date +%s)
sudo tar -czf /tmp/incident-evidence-$(date +%s).tar.gz /var/log/foxhunt /opt/foxhunt/config

Investigation (Security Team)

# 1. Review audit logs
grep "$(date +%Y-%m-%d)" /var/log/foxhunt/audit.log | grep -E "(unauthorized|failed|suspicious)"

# 2. Check for unauthorized access
psql $DATABASE_URL -c "SELECT * FROM audit_logs WHERE action = 'login_failed' AND created_at > NOW() - INTERVAL '24 hours' ORDER BY created_at DESC;"

# 3. Review network connections
sudo netstat -anp | grep ESTABLISHED

# 4. Check for malware
sudo rkhunter --check
sudo chkrootkit

# 5. Check file integrity
sudo aide --check

# 6. Review user access
sudo lastlog
sudo last -n 50

Recovery (After Investigation)

# 1. Rotate ALL secrets
vault kv put secret/foxhunt/jwt secret="$(openssl rand -base64 64)"
vault kv put secret/foxhunt/kill_switch master_token="$(openssl rand -base64 32)"
# ... rotate all other secrets

# 2. Update firewall rules
sudo ufw default deny incoming
sudo ufw allow from <trusted-ip> to any port 50051

# 3. Apply security patches
sudo apt-get update
sudo apt-get upgrade -y

# 4. Restore network access (carefully)
sudo iptables -F
sudo systemctl restart foxhunt-*

# 5. Monitor closely for 48 hours
watch -n 60 './deployment/scripts/security-scan.sh'

---

Scenario 5: Network Failure/Partition

Trigger: Cannot reach exchange servers, inter-service communication failed Response Time: <10 seconds to halt trading

Immediate Actions

# 1. HALT TRADING (network issues = blind trading)
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

# 2. VERIFY NETWORK CONNECTIVITY
# Check external connectivity
ping -c 3 8.8.8.8  # Internet
ping -c 3 <exchange-server>  # Exchange

# Check inter-service connectivity
for service in trading backtesting ml_training; do
  curl -s http://$service:9092/metrics || echo "❌ Cannot reach $service"
done

# 3. CHECK NETWORK INTERFACE STATUS
ip link show
ip addr show

# 4. CHECK FIREWALL RULES
sudo iptables -L -n -v

Recovery Options

Option A: Network Restored Automatically

# Services will auto-recover when network returns
# Verify connectivity restored
ping -c 10 <exchange-server>

# Services should exit read-only mode automatically
curl http://localhost:9092/metrics | grep network_partition_active
# Expected: network_partition_active 0

# Resume trading after verification
curl -X POST http://localhost:8080/trading/resume \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

Option B: Manual Network Recovery

# 1. Restart network interface
sudo ifdown eth0
sudo ifup eth0

# 2. Restart network service
sudo systemctl restart networking

# 3. Verify connectivity
ping -c 5 <exchange-server>

# 4. Restart services
sudo systemctl restart foxhunt-*

# 5. Resume trading
curl -X POST http://localhost:8080/trading/resume \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

---

Scenario 6: High Latency Alert (>100μs p99)

Trigger: Order processing latency exceeds critical threshold Response Time: <5 minutes to investigate

Immediate Actions

# 1. VERIFY LATENCY SPIKE
curl http://localhost:9092/metrics | grep order_processing_duration_seconds

# 2. CHECK CPU FREQUENCY SCALING (should be "performance")
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# 3. CHECK SYSTEM LOAD
uptime
top -bn1 | head -20

# 4. CHECK DATABASE QUERY PERFORMANCE
psql $DATABASE_URL -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

# 5. CHECK NETWORK LATENCY
ping -c 10 <exchange-server>

Recovery Actions

Fix CPU Frequency Scaling:

sudo cpupower frequency-set -g performance

Fix Database Performance:

# Add missing indexes
psql $DATABASE_URL <<EOF
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_orders_created_at ON orders(created_at);
ANALYZE orders;
EOF

Scale Horizontally (if load too high):

# Kubernetes
kubectl scale deployment trading-service --replicas=5

# Docker Swarm
docker service scale foxhunt_trading_service=5

Restart Services (if degraded):

sudo systemctl restart foxhunt-trading

---

🟡 HIGH PRIORITY SCENARIOS (SEV-2)

Scenario 7: Disk Space Critical (>90%)

# 1. CHECK DISK USAGE
df -h

# 2. FIND LARGE FILES
sudo du -ah /var/log/foxhunt | sort -rh | head -20
sudo du -ah /opt/foxhunt | sort -rh | head -20

# 3. CLEAN UP LOGS (keep last 7 days)
sudo find /var/log/foxhunt -name "*.log" -mtime +7 -delete
sudo journalctl --vacuum-time=7d

# 4. CLEAN UP DOCKER (if using Docker)
docker system prune -af --volumes

# 5. EXPAND DISK (if possible)
sudo lvextend -L +100G /dev/vg0/root
sudo resize2fs /dev/vg0/root

Scenario 8: Memory Exhaustion (>90%)

# 1. IDENTIFY MEMORY HOGS
ps aux --sort=-%mem | head -10

# 2. RESTART SERVICES (frees memory)
sudo systemctl restart foxhunt-trading

# 3. INCREASE SWAP (temporary fix)
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# 4. INCREASE SERVICE MEMORY LIMITS
sudo systemctl edit foxhunt-trading
# Add: MemoryMax=4G
sudo systemctl restart foxhunt-trading

# 5. SCALE HORIZONTALLY (long-term fix)
kubectl scale deployment trading-service --replicas=5

Scenario 9: Redis Connection Failure

# 1. CHECK REDIS STATUS
redis-cli ping
sudo systemctl status redis

# 2. RESTART REDIS
sudo systemctl restart redis  # Bare-metal
docker-compose restart redis  # Docker

# 3. VERIFY CONNECTION FROM SERVICES
redis-cli -h redis ping

# 4. CHECK CONNECTION POOL
redis-cli info clients

# 5. RESTART SERVICES (reconnect to Redis)
sudo systemctl restart foxhunt-*

---

📞 Escalation Procedures

Escalation Matrix

Time Elapsed	Action
T+0	On-call engineer paged
T+5 min	If no response, escalate to backup on-call
T+10 min	Notify CTO/Technical Lead
T+15 min	If SEV-1, notify CEO
T+30 min	If unresolved, engage external support

Communication Protocol

Internal (Slack #foxhunt-incidents):

🚨 INCIDENT ALERT
Severity: SEV-X
Component: <service/system>
Impact: <trading halted/degraded/etc>
Detected: <timestamp>
On-Call: <name> responding
ETA: <estimate>

External (Stakeholders):

Subject: [INCIDENT] Foxhunt Trading System - <brief description>

Time: <timestamp>
Status: Investigating/Identified/Resolved
Impact: <clear impact description>
Actions: <what we're doing>
ETA: <when we expect resolution>
Contact: <on-call contact>

---

📊 Post-Incident Procedures

Immediate (Within 1 hour of resolution)

# 1. Document incident
cat > /var/log/foxhunt/incidents/incident-$(date +%Y%m%d-%H%M%S).md <<EOF
# Incident Report

**Date**: $(date)
**Severity**: SEV-X
**Duration**: X minutes
**Impact**: <description>
**Root Cause**: <cause>
**Resolution**: <what fixed it>
**Responders**: <names>
EOF

# 2. Send all-clear notification
./deployment/scripts/send-alert.sh "Incident resolved - system operational"

# 3. Monitor for recurrence
watch -n 60 './deployment/health_check.sh --mode comprehensive'

Post-Mortem (Within 48 hours)

1. Incident Timeline: Document minute-by-minute actions
2. Root Cause Analysis: 5 Whys methodology
3. Action Items: Prevent recurrence
4. Communication Review: How well did we communicate?
5. Documentation Updates: Update runbooks with lessons learned

---

🔧 Emergency Tools

Quick Diagnostic Script

#!/bin/bash
# /opt/foxhunt/bin/emergency-diagnostics.sh

echo "=== FOXHUNT EMERGENCY DIAGNOSTICS ==="
echo "Time: $(date)"
echo ""

echo "=== SERVICE STATUS ==="
systemctl status foxhunt-* --no-pager | grep -E "(Active|Main PID)"

echo ""
echo "=== SYSTEM RESOURCES ==="
echo "Load: $(uptime | awk -F'load average:' '{print $2}')"
echo "Memory: $(free -m | grep Mem | awk '{printf "%.1f%%", $3/$2*100}')"
echo "Disk: $(df -h / | tail -1 | awk '{print $5}')"

echo ""
echo "=== DATABASE ==="
psql $DATABASE_URL -c "SELECT 1;" && echo "✅ Connected" || echo "❌ Failed"

echo ""
echo "=== RECENT ERRORS ==="
journalctl -u foxhunt-* --since "5 minutes ago" | grep -i error | tail -10

echo ""
echo "=== NETWORK ==="
ping -c 3 8.8.8.8 && echo "✅ Internet OK" || echo "❌ Internet Failed"

---

Last Updated: 2025-10-07 Version: 1.0.0 Review Frequency: Monthly or after each incident