foxhunt/EMERGENCY_PROCEDURES.md

# Emergency Procedures - Foxhunt HFT Trading System

**Purpose**: Critical incident response procedures for immediate action
**Version**: 1.0.0
**Last Updated**: 2025-10-07

---

## 🚨 EMERGENCY CONTACTS

**Fill in before production deployment:**

| Role | Name | Phone | Email | Escalation Time |
|------|------|-------|-------|-----------------|
| On-Call Engineer | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | Immediate |
| Database Admin | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | 5 minutes |
| Security Lead | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | 10 minutes |
| CTO/Technical Lead | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | 15 minutes |
| CEO | [CONFIGURE] | [CONFIGURE] | [CONFIGURE] | 30 minutes |

**PagerDuty**: [CONFIGURE]
**Slack Channel**: #foxhunt-incidents

---

## 🎯 Severity Levels

### SEV-1: CRITICAL (Immediate Response)
- **Impact**: Trading completely halted, data loss, security breach
- **Response Time**: <5 minutes
- **Examples**: System down, database corruption, unauthorized access
- **Escalation**: Immediate page to on-call + CTO

### SEV-2: HIGH (Urgent Response)
- **Impact**: Significant degradation, trading impaired but operational
- **Response Time**: <15 minutes
- **Examples**: High latency (>500μs), service failure, data inconsistency
- **Escalation**: Page on-call, notify team in Slack

### SEV-3: MEDIUM (Priority Response)
- **Impact**: Minor degradation, non-critical component failure
- **Response Time**: <1 hour
- **Examples**: Non-critical service down, elevated error rate
- **Escalation**: Slack notification, log ticket

### SEV-4: LOW (Standard Response)
- **Impact**: No immediate impact, potential future issue
- **Response Time**: <4 hours (business hours)
- **Examples**: Warning alerts, resource usage trends
- **Escalation**: Standard ticket

---

## 🔴 CRITICAL SCENARIOS (SEV-1)

### Scenario 1: System-Wide Trading Halt

**Trigger**: All trading stopped, no orders processing
**Response Time**: <2 minutes to activate kill switch, <15 minutes to diagnose

#### Immediate Actions (First 60 seconds)

```bash
# 1. CONFIRM HALT - Check if trading actually stopped
curl http://localhost:9092/metrics | grep trading_active
# Expected: trading_active 0

# 2. ACTIVATE KILL SWITCH (if not already active)
export KILL_SWITCH_TOKEN="your-master-token"
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

# 3. VERIFY KILL SWITCH ACTIVE
curl http://localhost:8080/status
# Expected: {"status": "HALTED", "reason": "Emergency kill switch activated"}

# 4. NOTIFY STAKEHOLDERS
./deployment/scripts/send-emergency-alert.sh "Trading halted - investigating"
```

#### Diagnosis (Next 5 minutes)

```bash
# Check service status
docker-compose ps  # Docker
sudo systemctl status foxhunt-*  # Bare-metal

# Check recent errors
docker-compose logs --tail=100 trading_service | grep -i error
sudo journalctl -u foxhunt-trading --since "5 minutes ago" | grep -i error

# Check system resources
df -h  # Disk space
free -m  # Memory
uptime  # Load average

# Check database connectivity
psql $DATABASE_URL -c "SELECT 1;" || echo "❌ Database connection failed"

# Check network connectivity
ping -c 3 <exchange-server> || echo "❌ Network connectivity failed"
```

#### Decision Tree

**If database is down**:
```bash
# See: Scenario 2 - Database Failure
```

**If service crashed**:
```bash
# See: Scenario 3 - Service Crash
```

**If network issue**:
```bash
# See: Scenario 5 - Network Failure
```

**If cause unknown**:
```bash
# 1. Restart services in safe mode (read-only)
./deployment/scripts/restart-safe-mode.sh

# 2. Escalate to CTO
./deployment/scripts/escalate.sh --level=cto --message="Trading halt, cause unknown"

# 3. Begin detailed investigation
./deployment/scripts/capture-diagnostics.sh > /var/log/foxhunt/incident-$(date +%s).log
```

#### Recovery Procedure

```bash
# 1. Fix root cause (see specific scenarios)

# 2. Run validation tests
./deployment/scripts/pre-trading-validation.sh

# 3. Resume trading (only after validation passes)
curl -X POST http://localhost:8080/trading/resume \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Issue resolved - verified", "approved_by": "your-name"}'

# 4. Monitor closely for 30 minutes
watch -n 10 'curl -s http://localhost:9092/metrics | grep -E "(order_latency|error_rate)"'

# 5. Send all-clear notification
./deployment/scripts/send-emergency-alert.sh "Trading resumed - incident resolved"
```

---

### Scenario 2: Database Failure

**Trigger**: Cannot connect to PostgreSQL
**Response Time**: <5 minutes to failover

#### Immediate Actions

```bash
# 1. ACTIVATE KILL SWITCH
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

# 2. CHECK DATABASE STATUS
sudo systemctl status postgresql  # Bare-metal
docker-compose ps postgres  # Docker

# 3. CHECK REPLICATION STATUS (if configured)
psql -h replica-server -U foxhunt -c "SELECT pg_is_in_recovery();"
# Expected: t (true = replica)
```

#### Recovery Options

**Option A: Database is Running but Unresponsive**
```bash
# 1. Check connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# 2. Kill long-running queries
psql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'active' AND now() - query_start > interval '5 minutes';"

# 3. Restart PostgreSQL
sudo systemctl restart postgresql  # Bare-metal
docker-compose restart postgres  # Docker

# 4. Wait for startup
sleep 10

# 5. Test connection
psql $DATABASE_URL -c "SELECT 1;"
```

**Option B: Database Crashed - Restore from Replica**
```bash
# 1. Promote replica to primary
ssh replica-server "sudo -u postgres /usr/lib/postgresql/16/bin/pg_ctl promote -D /var/lib/postgresql/16/main"

# 2. Update connection strings
export NEW_DB_URL=postgresql://foxhunt:password@replica-server:5432/foxhunt

# 3. Update services
docker-compose down
sed -i "s|DATABASE_URL=.*|DATABASE_URL=$NEW_DB_URL|" .env
docker-compose up -d

# OR for bare-metal
sudo systemctl stop foxhunt-*
sudo systemctl set-environment DATABASE_URL="$NEW_DB_URL"
sudo systemctl start foxhunt-*

# 4. Verify connectivity
psql $NEW_DB_URL -c "SELECT count(*) FROM orders;"
```

**Option C: Complete Restore from Backup**
```bash
# 1. Download latest backup from S3
aws s3 cp s3://foxhunt-backups/postgres/latest.dump /tmp/restore.dump

# 2. Create new database
createdb foxhunt_restored

# 3. Restore
pg_restore -h localhost -U foxhunt -d foxhunt_restored -v /tmp/restore.dump

# 4. Swap databases
psql -U postgres <<EOF
ALTER DATABASE foxhunt RENAME TO foxhunt_old;
ALTER DATABASE foxhunt_restored RENAME TO foxhunt;
EOF

# 5. Restart services
sudo systemctl restart foxhunt-*

# Estimated downtime: 5-15 minutes (depends on backup size)
```

---

### Scenario 3: Service Crash/Unresponsive

**Trigger**: Service health check failing, not responding
**Response Time**: <1 minute to restart

#### Immediate Actions

```bash
# 1. IDENTIFY CRASHED SERVICE
docker-compose ps | grep -v "Up"  # Docker
sudo systemctl status foxhunt-* | grep failed  # Bare-metal

# 2. CHECK LOGS FOR CRASH REASON
docker-compose logs --tail=100 <service> | grep -E "(panic|error|fatal)"
sudo journalctl -u foxhunt-<service> -n 100 | grep -E "(panic|error|fatal)"

# 3. RESTART SERVICE
docker-compose restart <service>  # Docker
sudo systemctl restart foxhunt-<service>  # Bare-metal

# 4. VERIFY RESTART SUCCESSFUL
docker-compose ps <service>
sudo systemctl status foxhunt-<service>

# 5. CHECK FOR ERRORS AFTER RESTART
docker-compose logs --tail=50 <service>
sudo journalctl -u foxhunt-<service> -n 50
```

#### If Restart Fails

```bash
# 1. Check for port conflicts
sudo lsof -i :50052  # Trading service port
sudo kill -9 <PID>  # Kill conflicting process

# 2. Check disk space
df -h
# If >90% full, clean up logs
sudo find /var/log/foxhunt -name "*.log" -mtime +7 -delete

# 3. Check memory
free -m
# If memory exhausted, identify memory hog
ps aux --sort=-%mem | head -10

# 4. Try safe mode restart
docker-compose stop <service>
docker-compose up -d <service> --force-recreate

# OR bare-metal
sudo systemctl stop foxhunt-<service>
sudo rm -rf /tmp/foxhunt/*  # Clear temp files
sudo systemctl start foxhunt-<service>

# 5. If still failing, rollback
sudo cp /opt/foxhunt/bin/<service>.backup /opt/foxhunt/bin/<service>
sudo systemctl restart foxhunt-<service>
```

---

### Scenario 4: Security Breach Detected

**Trigger**: Unauthorized access, data exfiltration, suspicious activity
**Response Time**: IMMEDIATE - <30 seconds to isolate

#### Immediate Actions (CRITICAL - Do NOT delay)

```bash
# 1. ACTIVATE KILL SWITCH IMMEDIATELY
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

# 2. ISOLATE SYSTEM (disconnect from network)
# Bare-metal
sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
sudo iptables -P FORWARD DROP

# Docker
docker network disconnect foxhunt_foxhunt-network <container>

# 3. CAPTURE FORENSIC DATA
./deployment/scripts/capture-forensics.sh > /var/log/foxhunt/security-incident-$(date +%s).log

# 4. NOTIFY SECURITY TEAM IMMEDIATELY
./deployment/scripts/security-alert.sh --severity=critical --message="Security breach detected - system isolated"

# 5. PRESERVE EVIDENCE
sudo cp -r /var/log/foxhunt /var/log/foxhunt.incident-$(date +%s)
sudo tar -czf /tmp/incident-evidence-$(date +%s).tar.gz /var/log/foxhunt /opt/foxhunt/config
```

#### Investigation (Security Team)

```bash
# 1. Review audit logs
grep "$(date +%Y-%m-%d)" /var/log/foxhunt/audit.log | grep -E "(unauthorized|failed|suspicious)"

# 2. Check for unauthorized access
psql $DATABASE_URL -c "SELECT * FROM audit_logs WHERE action = 'login_failed' AND created_at > NOW() - INTERVAL '24 hours' ORDER BY created_at DESC;"

# 3. Review network connections
sudo netstat -anp | grep ESTABLISHED

# 4. Check for malware
sudo rkhunter --check
sudo chkrootkit

# 5. Check file integrity
sudo aide --check

# 6. Review user access
sudo lastlog
sudo last -n 50
```

#### Recovery (After Investigation)

```bash
# 1. Rotate ALL secrets
vault kv put secret/foxhunt/jwt secret="$(openssl rand -base64 64)"
vault kv put secret/foxhunt/kill_switch master_token="$(openssl rand -base64 32)"
# ... rotate all other secrets

# 2. Update firewall rules
sudo ufw default deny incoming
sudo ufw allow from <trusted-ip> to any port 50051

# 3. Apply security patches
sudo apt-get update
sudo apt-get upgrade -y

# 4. Restore network access (carefully)
sudo iptables -F
sudo systemctl restart foxhunt-*

# 5. Monitor closely for 48 hours
watch -n 60 './deployment/scripts/security-scan.sh'
```

---

### Scenario 5: Network Failure/Partition

**Trigger**: Cannot reach exchange servers, inter-service communication failed
**Response Time**: <10 seconds to halt trading

#### Immediate Actions

```bash
# 1. HALT TRADING (network issues = blind trading)
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"

# 2. VERIFY NETWORK CONNECTIVITY
# Check external connectivity
ping -c 3 8.8.8.8  # Internet
ping -c 3 <exchange-server>  # Exchange

# Check inter-service connectivity
for service in trading backtesting ml_training; do
  curl -s http://$service:9092/metrics || echo "❌ Cannot reach $service"
done

# 3. CHECK NETWORK INTERFACE STATUS
ip link show
ip addr show

# 4. CHECK FIREWALL RULES
sudo iptables -L -n -v
```

#### Recovery Options

**Option A: Network Restored Automatically**
```bash
# Services will auto-recover when network returns
# Verify connectivity restored
ping -c 10 <exchange-server>

# Services should exit read-only mode automatically
curl http://localhost:9092/metrics | grep network_partition_active
# Expected: network_partition_active 0

# Resume trading after verification
curl -X POST http://localhost:8080/trading/resume \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"
```

**Option B: Manual Network Recovery**
```bash
# 1. Restart network interface
sudo ifdown eth0
sudo ifup eth0

# 2. Restart network service
sudo systemctl restart networking

# 3. Verify connectivity
ping -c 5 <exchange-server>

# 4. Restart services
sudo systemctl restart foxhunt-*

# 5. Resume trading
curl -X POST http://localhost:8080/trading/resume \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_TOKEN"
```

---

### Scenario 6: High Latency Alert (>100μs p99)

**Trigger**: Order processing latency exceeds critical threshold
**Response Time**: <5 minutes to investigate

#### Immediate Actions

```bash
# 1. VERIFY LATENCY SPIKE
curl http://localhost:9092/metrics | grep order_processing_duration_seconds

# 2. CHECK CPU FREQUENCY SCALING (should be "performance")
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# 3. CHECK SYSTEM LOAD
uptime
top -bn1 | head -20

# 4. CHECK DATABASE QUERY PERFORMANCE
psql $DATABASE_URL -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

# 5. CHECK NETWORK LATENCY
ping -c 10 <exchange-server>
```

#### Recovery Actions

**Fix CPU Frequency Scaling**:
```bash
sudo cpupower frequency-set -g performance
```

**Fix Database Performance**:
```bash
# Add missing indexes
psql $DATABASE_URL <<EOF
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_orders_created_at ON orders(created_at);
ANALYZE orders;
EOF
```

**Scale Horizontally** (if load too high):
```bash
# Kubernetes
kubectl scale deployment trading-service --replicas=5

# Docker Swarm
docker service scale foxhunt_trading_service=5
```

**Restart Services** (if degraded):
```bash
sudo systemctl restart foxhunt-trading
```

---

## 🟡 HIGH PRIORITY SCENARIOS (SEV-2)

### Scenario 7: Disk Space Critical (>90%)

```bash
# 1. CHECK DISK USAGE
df -h

# 2. FIND LARGE FILES
sudo du -ah /var/log/foxhunt | sort -rh | head -20
sudo du -ah /opt/foxhunt | sort -rh | head -20

# 3. CLEAN UP LOGS (keep last 7 days)
sudo find /var/log/foxhunt -name "*.log" -mtime +7 -delete
sudo journalctl --vacuum-time=7d

# 4. CLEAN UP DOCKER (if using Docker)
docker system prune -af --volumes

# 5. EXPAND DISK (if possible)
sudo lvextend -L +100G /dev/vg0/root
sudo resize2fs /dev/vg0/root
```

### Scenario 8: Memory Exhaustion (>90%)

```bash
# 1. IDENTIFY MEMORY HOGS
ps aux --sort=-%mem | head -10

# 2. RESTART SERVICES (frees memory)
sudo systemctl restart foxhunt-trading

# 3. INCREASE SWAP (temporary fix)
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# 4. INCREASE SERVICE MEMORY LIMITS
sudo systemctl edit foxhunt-trading
# Add: MemoryMax=4G
sudo systemctl restart foxhunt-trading

# 5. SCALE HORIZONTALLY (long-term fix)
kubectl scale deployment trading-service --replicas=5
```

### Scenario 9: Redis Connection Failure

```bash
# 1. CHECK REDIS STATUS
redis-cli ping
sudo systemctl status redis

# 2. RESTART REDIS
sudo systemctl restart redis  # Bare-metal
docker-compose restart redis  # Docker

# 3. VERIFY CONNECTION FROM SERVICES
redis-cli -h redis ping

# 4. CHECK CONNECTION POOL
redis-cli info clients

# 5. RESTART SERVICES (reconnect to Redis)
sudo systemctl restart foxhunt-*
```

---

## 📞 Escalation Procedures

### Escalation Matrix

| Time Elapsed | Action |
|--------------|--------|
| T+0 | On-call engineer paged |
| T+5 min | If no response, escalate to backup on-call |
| T+10 min | Notify CTO/Technical Lead |
| T+15 min | If SEV-1, notify CEO |
| T+30 min | If unresolved, engage external support |

### Communication Protocol

**Internal (Slack #foxhunt-incidents)**:
```
🚨 INCIDENT ALERT
Severity: SEV-X
Component: <service/system>
Impact: <trading halted/degraded/etc>
Detected: <timestamp>
On-Call: <name> responding
ETA: <estimate>
```

**External (Stakeholders)**:
```
Subject: [INCIDENT] Foxhunt Trading System - <brief description>

Time: <timestamp>
Status: Investigating/Identified/Resolved
Impact: <clear impact description>
Actions: <what we're doing>
ETA: <when we expect resolution>
Contact: <on-call contact>
```

---

## 📊 Post-Incident Procedures

### Immediate (Within 1 hour of resolution)

```bash
# 1. Document incident
cat > /var/log/foxhunt/incidents/incident-$(date +%Y%m%d-%H%M%S).md <<EOF
# Incident Report

**Date**: $(date)
**Severity**: SEV-X
**Duration**: X minutes
**Impact**: <description>
**Root Cause**: <cause>
**Resolution**: <what fixed it>
**Responders**: <names>
EOF

# 2. Send all-clear notification
./deployment/scripts/send-alert.sh "Incident resolved - system operational"

# 3. Monitor for recurrence
watch -n 60 './deployment/health_check.sh --mode comprehensive'
```

### Post-Mortem (Within 48 hours)

1. **Incident Timeline**: Document minute-by-minute actions
2. **Root Cause Analysis**: 5 Whys methodology
3. **Action Items**: Prevent recurrence
4. **Communication Review**: How well did we communicate?
5. **Documentation Updates**: Update runbooks with lessons learned

---

## 🔧 Emergency Tools

### Quick Diagnostic Script

```bash
#!/bin/bash
# /opt/foxhunt/bin/emergency-diagnostics.sh

echo "=== FOXHUNT EMERGENCY DIAGNOSTICS ==="
echo "Time: $(date)"
echo ""

echo "=== SERVICE STATUS ==="
systemctl status foxhunt-* --no-pager | grep -E "(Active|Main PID)"

echo ""
echo "=== SYSTEM RESOURCES ==="
echo "Load: $(uptime | awk -F'load average:' '{print $2}')"
echo "Memory: $(free -m | grep Mem | awk '{printf "%.1f%%", $3/$2*100}')"
echo "Disk: $(df -h / | tail -1 | awk '{print $5}')"

echo ""
echo "=== DATABASE ==="
psql $DATABASE_URL -c "SELECT 1;" && echo "✅ Connected" || echo "❌ Failed"

echo ""
echo "=== RECENT ERRORS ==="
journalctl -u foxhunt-* --since "5 minutes ago" | grep -i error | tail -10

echo ""
echo "=== NETWORK ==="
ping -c 3 8.8.8.8 && echo "✅ Internet OK" || echo "❌ Internet Failed"
```

---

**Last Updated**: 2025-10-07
**Version**: 1.0.0
**Review Frequency**: Monthly or after each incident