foxhunt/docs/runbooks/incident-response.md

# Incident Response Runbook

**Last Updated**: 2025-10-22
**Scope**: All production incidents
**On-Call Rotation**: 24/7 coverage required

---

## Severity Levels

| Level | Definition | Response Time | Examples |
|-------|------------|---------------|----------|
| **P0** | Critical - System down | 5 min | Complete trading halt, data loss, security breach |
| **P1** | High - Severe degradation | 15 min | API failures >10%, high latency P99 >1s, DB unavailable |
| **P2** | Medium - Partial degradation | 1 hour | Single service degraded, non-critical feature broken |
| **P3** | Low - Minor issue | 4 hours | Performance degradation <5%, non-urgent bugs |
| **P4** | Info - No impact | Next business day | Documentation updates, cosmetic issues |

---

## P0: Critical Incident Response

### 1. Initial Response (0-5 minutes)

```bash
# Step 1: Acknowledge incident
curl -X POST https://api.pagerduty.com/incidents/${INCIDENT_ID}/acknowledge \
  -H "Authorization: Token token=${PD_TOKEN}"

# Step 2: Declare incident in Slack
curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-Type: application/json' \
  -d '{"text":"🚨 P0 INCIDENT: Trading system down. War room: #incident-2025-10-22"}'

# Step 3: Create war room channel
# Manually create #incident-YYYY-MM-DD in Slack

# Step 4: Assess system health
kubectl get pods -n foxhunt --no-headers | awk '{if ($3 != "Running") print $0}'
```

### 2. Triage (5-10 minutes)

```bash
# Check service health
for svc in api-gateway trading-service backtesting-service ml-training-service trading-agent; do
    echo "=== $svc ==="
    kubectl logs -n foxhunt -l app.kubernetes.io/component=$svc --tail=50 | grep -i error
done

# Check infrastructure
kubectl top nodes
kubectl top pods -n foxhunt --sort-by=memory | head -20

# Check database
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM pg_stat_activity WHERE state='active';"

# Check metrics
curl -s http://prometheus:9090/api/v1/query?query=up{job=\"trading-service\"} | jq '.data.result'
```

### 3. Mitigation (10-30 minutes)

**Service Restart**:
```bash
# Restart unhealthy service
kubectl rollout restart deployment/foxhunt-trading-service -n foxhunt

# OR immediate pod deletion (faster but riskier)
kubectl delete pods -n foxhunt -l app.kubernetes.io/component=trading-service
```

**Rollback**:
```bash
# Quick rollback to previous version
kubectl rollout undo deployment/foxhunt-trading-service -n foxhunt

# Verify rollback
kubectl rollout status deployment/foxhunt-trading-service -n foxhunt
```

**Circuit Breaker**:
```bash
# Enable maintenance mode
kubectl create configmap foxhunt-maintenance -n foxhunt \
  --from-literal=enabled=true \
  --from-literal=message="Emergency maintenance. Trading suspended."
```

### 4. Recovery Verification (30-45 minutes)

```bash
# Health checks
for port in 8080 8081 8082 8095 8083; do
    curl -sf http://localhost:$port/health && echo "Port $port: OK" || echo "Port $port: FAILED"
done

# End-to-end test
tli auth login --username admin --password ${ADMIN_PASSWORD}
tli trade order submit --symbol ES.FUT --action BUY --quantity 1 --order-type MARKET --paper-trade

# Verify in database
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT * FROM orders ORDER BY created_at DESC LIMIT 1;"
```

### 5. Post-Incident (45-60 minutes)

```bash
# Notify resolution
curl -X POST https://api.pagerduty.com/incidents/${INCIDENT_ID}/resolve \
  -H "Authorization: Token token=${PD_TOKEN}"

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-Type: application/json' \
  -d '{"text":"✅ P0 RESOLVED: Trading system operational. Duration: 45 minutes."}'

# Schedule postmortem within 24 hours
# Use template: /home/jgrusewski/Work/foxhunt/docs/templates/incident-report.md
```

---

## P1: High Severity Response

### Response Procedure (15 minutes)

```bash
# 1. Assess scope
kubectl get pods -n foxhunt -o wide
kubectl logs -n foxhunt -l app.kubernetes.io/component=api-gateway --tail=100 | grep ERROR

# 2. Check metrics
curl -s "http://prometheus:9090/api/v1/query?query=rate(grpc_requests_total{code!~\"2..\"}[5m])" | jq '.data.result'

# 3. Scale if needed
kubectl scale deployment/foxhunt-api-gateway -n foxhunt --replicas=10

# 4. Monitor recovery
watch kubectl get hpa -n foxhunt
```

---

## Common Incident Scenarios

### Scenario 1: Database Connection Pool Exhausted

**Symptoms**:
```
ERROR: FATAL: remaining connection slots are reserved for non-replication superuser connections
```

**Resolution**:
```bash
# Check active connections
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# Terminate idle connections
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state='idle' AND state_change < now() - interval '5 minutes';"

# Increase max_connections (requires restart)
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "ALTER SYSTEM SET max_connections = 200;"

kubectl delete pod foxhunt-postgresql-0 -n foxhunt
```

### Scenario 2: Redis Memory Full

**Symptoms**:
```
ERROR: OOM command not allowed when used memory > 'maxmemory'
```

**Resolution**:
```bash
# Check memory usage
kubectl exec -n foxhunt foxhunt-redis-master-0 -- redis-cli INFO memory | grep used_memory_human

# Evict keys (if eviction policy allows)
kubectl exec -n foxhunt foxhunt-redis-master-0 -- redis-cli FLUSHDB

# OR increase memory limit
kubectl patch statefulset foxhunt-redis-master -n foxhunt \
    -p '{"spec":{"template":{"spec":{"containers":[{"name":"redis","resources":{"limits":{"memory":"2Gi"}}}]}}}}'
```

### Scenario 3: GPU Out of Memory

**Symptoms**:
```
ERROR: CUDA error: out of memory
```

**Resolution**:
```bash
# Check GPU memory
kubectl exec -n foxhunt foxhunt-ml-training-service-xxxxx -- nvidia-smi

# Kill GPU process
kubectl exec -n foxhunt foxhunt-ml-training-service-xxxxx -- nvidia-smi --gpu-reset

# Restart service
kubectl delete pod foxhunt-ml-training-service-xxxxx -n foxhunt
```

---

## Incident Communication Template

### Initial Report (within 5 minutes)

```
🚨 P0 INCIDENT: [Brief Description]

STATUS: Investigating
IMPACT: [Users affected, services down]
STARTED: 2025-10-22 10:00 UTC
INCIDENT COMMANDER: @john.doe
WAR ROOM: #incident-2025-10-22

Current actions:
- Checking service health
- Reviewing logs
- Escalating to engineering team
```

### Update (every 15 minutes)

```
UPDATE 10:15 UTC:

ROOT CAUSE: Database connection pool exhausted
MITIGATION: Terminated idle connections, increased pool size
RECOVERY: Services restarting, ETA 5 minutes

Next update: 10:30 UTC
```

### Resolution

```
✅ INCIDENT RESOLVED

DURATION: 45 minutes (10:00-10:45 UTC)
ROOT CAUSE: Database connection pool exhausted due to connection leak
RESOLUTION: Terminated idle connections, increased max_connections from 100 to 200
IMPACT: 150 users unable to submit orders for 45 minutes

FOLLOW-UP:
- [ ] Postmortem scheduled for 2025-10-23 10:00 UTC
- [ ] Fix connection leak in Trading Service v0.1.7
- [ ] Add monitoring alert for connection pool usage >80%
```

---

## Escalation Matrix

| Role | Primary | Secondary | Phone |
|------|---------|-----------|-------|
| Incident Commander | John Doe | Jane Smith | +1-555-0100 |
| Engineering Lead | Alice Brown | Bob Wilson | +1-555-0101 |
| Database Admin | Charlie Davis | Diana Evans | +1-555-0102 |
| DevOps Lead | Eve Foster | Frank Green | +1-555-0103 |
| Security Lead | Grace Harris | Henry Irving | +1-555-0104 |

**Escalation Path**:
1. On-call engineer (responds within 5 min)
2. Engineering Lead (escalate after 15 min)
3. CTO (escalate for P0 after 30 min or P1 after 2 hours)

---

**End of Incident Response Runbook**