foxhunt/docs/runbooks/incident-response.md

Incident Response Runbook

Last Updated: 2025-10-22 Scope: All production incidents On-Call Rotation: 24/7 coverage required

---

Severity Levels

Level	Definition	Response Time	Examples
P0	Critical - System down	5 min	Complete trading halt, data loss, security breach
P1	High - Severe degradation	15 min	API failures >10%, high latency P99 >1s, DB unavailable
P2	Medium - Partial degradation	1 hour	Single service degraded, non-critical feature broken
P3	Low - Minor issue	4 hours	Performance degradation <5%, non-urgent bugs
P4	Info - No impact	Next business day	Documentation updates, cosmetic issues

---

P0: Critical Incident Response

1. Initial Response (0-5 minutes)

# Step 1: Acknowledge incident
curl -X POST https://api.pagerduty.com/incidents/${INCIDENT_ID}/acknowledge \
  -H "Authorization: Token token=${PD_TOKEN}"

# Step 2: Declare incident in Slack
curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-Type: application/json' \
  -d '{"text":"🚨 P0 INCIDENT: Trading system down. War room: #incident-2025-10-22"}'

# Step 3: Create war room channel
# Manually create #incident-YYYY-MM-DD in Slack

# Step 4: Assess system health
kubectl get pods -n foxhunt --no-headers | awk '{if ($3 != "Running") print $0}'

2. Triage (5-10 minutes)

# Check service health
for svc in api-gateway trading-service backtesting-service ml-training-service trading-agent; do
    echo "=== $svc ==="
    kubectl logs -n foxhunt -l app.kubernetes.io/component=$svc --tail=50 | grep -i error
done

# Check infrastructure
kubectl top nodes
kubectl top pods -n foxhunt --sort-by=memory | head -20

# Check database
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM pg_stat_activity WHERE state='active';"

# Check metrics
curl -s http://prometheus:9090/api/v1/query?query=up{job=\"trading-service\"} | jq '.data.result'

3. Mitigation (10-30 minutes)

Service Restart:

# Restart unhealthy service
kubectl rollout restart deployment/foxhunt-trading-service -n foxhunt

# OR immediate pod deletion (faster but riskier)
kubectl delete pods -n foxhunt -l app.kubernetes.io/component=trading-service

Rollback:

# Quick rollback to previous version
kubectl rollout undo deployment/foxhunt-trading-service -n foxhunt

# Verify rollback
kubectl rollout status deployment/foxhunt-trading-service -n foxhunt

Circuit Breaker:

# Enable maintenance mode
kubectl create configmap foxhunt-maintenance -n foxhunt \
  --from-literal=enabled=true \
  --from-literal=message="Emergency maintenance. Trading suspended."

4. Recovery Verification (30-45 minutes)

# Health checks
for port in 8080 8081 8082 8095 8083; do
    curl -sf http://localhost:$port/health && echo "Port $port: OK" || echo "Port $port: FAILED"
done

# End-to-end test
tli auth login --username admin --password ${ADMIN_PASSWORD}
tli trade order submit --symbol ES.FUT --action BUY --quantity 1 --order-type MARKET --paper-trade

# Verify in database
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT * FROM orders ORDER BY created_at DESC LIMIT 1;"

5. Post-Incident (45-60 minutes)

# Notify resolution
curl -X POST https://api.pagerduty.com/incidents/${INCIDENT_ID}/resolve \
  -H "Authorization: Token token=${PD_TOKEN}"

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-Type: application/json' \
  -d '{"text":"✅ P0 RESOLVED: Trading system operational. Duration: 45 minutes."}'

# Schedule postmortem within 24 hours
# Use template: /home/jgrusewski/Work/foxhunt/docs/templates/incident-report.md

---

P1: High Severity Response

Response Procedure (15 minutes)

# 1. Assess scope
kubectl get pods -n foxhunt -o wide
kubectl logs -n foxhunt -l app.kubernetes.io/component=api-gateway --tail=100 | grep ERROR

# 2. Check metrics
curl -s "http://prometheus:9090/api/v1/query?query=rate(grpc_requests_total{code!~\"2..\"}[5m])" | jq '.data.result'

# 3. Scale if needed
kubectl scale deployment/foxhunt-api-gateway -n foxhunt --replicas=10

# 4. Monitor recovery
watch kubectl get hpa -n foxhunt

---

Common Incident Scenarios

Scenario 1: Database Connection Pool Exhausted

Symptoms:

ERROR: FATAL: remaining connection slots are reserved for non-replication superuser connections

Resolution:

# Check active connections
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# Terminate idle connections
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state='idle' AND state_change < now() - interval '5 minutes';"

# Increase max_connections (requires restart)
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "ALTER SYSTEM SET max_connections = 200;"

kubectl delete pod foxhunt-postgresql-0 -n foxhunt

Scenario 2: Redis Memory Full

Symptoms:

ERROR: OOM command not allowed when used memory > 'maxmemory'

Resolution:

# Check memory usage
kubectl exec -n foxhunt foxhunt-redis-master-0 -- redis-cli INFO memory | grep used_memory_human

# Evict keys (if eviction policy allows)
kubectl exec -n foxhunt foxhunt-redis-master-0 -- redis-cli FLUSHDB

# OR increase memory limit
kubectl patch statefulset foxhunt-redis-master -n foxhunt \
    -p '{"spec":{"template":{"spec":{"containers":[{"name":"redis","resources":{"limits":{"memory":"2Gi"}}}]}}}}'

Scenario 3: GPU Out of Memory

Symptoms:

ERROR: CUDA error: out of memory

Resolution:

# Check GPU memory
kubectl exec -n foxhunt foxhunt-ml-training-service-xxxxx -- nvidia-smi

# Kill GPU process
kubectl exec -n foxhunt foxhunt-ml-training-service-xxxxx -- nvidia-smi --gpu-reset

# Restart service
kubectl delete pod foxhunt-ml-training-service-xxxxx -n foxhunt

---

Incident Communication Template

Initial Report (within 5 minutes)

🚨 P0 INCIDENT: [Brief Description]

STATUS: Investigating
IMPACT: [Users affected, services down]
STARTED: 2025-10-22 10:00 UTC
INCIDENT COMMANDER: @john.doe
WAR ROOM: #incident-2025-10-22

Current actions:
- Checking service health
- Reviewing logs
- Escalating to engineering team

Update (every 15 minutes)

UPDATE 10:15 UTC:

ROOT CAUSE: Database connection pool exhausted
MITIGATION: Terminated idle connections, increased pool size
RECOVERY: Services restarting, ETA 5 minutes

Next update: 10:30 UTC

Resolution

✅ INCIDENT RESOLVED

DURATION: 45 minutes (10:00-10:45 UTC)
ROOT CAUSE: Database connection pool exhausted due to connection leak
RESOLUTION: Terminated idle connections, increased max_connections from 100 to 200
IMPACT: 150 users unable to submit orders for 45 minutes

FOLLOW-UP:
- [ ] Postmortem scheduled for 2025-10-23 10:00 UTC
- [ ] Fix connection leak in Trading Service v0.1.7
- [ ] Add monitoring alert for connection pool usage >80%

---

Escalation Matrix

Role	Primary	Secondary	Phone
Incident Commander	John Doe	Jane Smith	+1-555-0100
Engineering Lead	Alice Brown	Bob Wilson	+1-555-0101
Database Admin	Charlie Davis	Diana Evans	+1-555-0102
DevOps Lead	Eve Foster	Frank Green	+1-555-0103
Security Lead	Grace Harris	Henry Irving	+1-555-0104

Escalation Path:

1. On-call engineer (responds within 5 min)
2. Engineering Lead (escalate after 15 min)
3. CTO (escalate for P0 after 30 min or P1 after 2 hours)

---

End of Incident Response Runbook