foxhunt/docs/monitoring/RUNBOOKS.md

# Operational Runbooks - Foxhunt HFT Trading System

**Last Updated**: 2025-10-07
**Version**: 1.0
**On-Call Contact**: #oncall-engineering Slack channel

---

## Table of Contents

1. [Trading Service Incidents](#1-trading-service-incidents)
2. [API Gateway Incidents](#2-api-gateway-incidents)
3. [ML Training Service Incidents](#3-ml-training-service-incidents)
4. [Backtesting Service Incidents](#4-backtesting-service-incidents)
5. [Database Incidents](#5-database-incidents)
6. [Infrastructure Incidents](#6-infrastructure-incidents)
7. [Incident Response Procedures](#7-incident-response-procedures)

---

## 1. Trading Service Incidents

### 1.1 High Order Latency (> 100μs p99)

**Alert**: `OrderLatencySLAViolation`
**Severity**: CRITICAL
**Impact**: Trading performance degraded, potential profit loss

#### Diagnosis Steps

1. **Check Current Latency**:
```bash
# Query Prometheus
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99, rate(trading_order_processing_microseconds_bucket[1m]))'

# Expected: < 100μs
# If > 150μs: CRITICAL emergency
```

2. **Identify Bottleneck**:
```bash
# Check CPU affinity
taskset -cp $(pgrep trading_service)

# Check CPU usage
top -p $(pgrep trading_service)

# Check network latency to broker
ping -c 10 broker.example.com

# Check database query latency
psql -U foxhunt -d foxhunt -c "SELECT * FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
```

3. **Check System Resources**:
```bash
# Memory
free -h

# CPU
mpstat 1 10

# Disk I/O
iostat -x 1 10

# Network
iftop -i eth0
```

#### Resolution Steps

**Immediate Actions** (< 5 minutes):
1. Verify CPU affinity is set correctly
2. Check for competing processes and kill if necessary
3. Verify network connectivity to broker
4. Check for database connection pool exhaustion

**Short-term Fixes** (< 30 minutes):
1. Restart trading service if memory leak suspected
2. Scale horizontally by adding more instances
3. Enable circuit breakers to shed load
4. Review recent code deployments for regressions

**Long-term Solutions**:
1. Optimize hot path code
2. Add caching for frequent database lookups
3. Review and tune database indexes
4. Upgrade hardware (CPU, network)

#### Post-Incident

- Document root cause
- Update monitoring thresholds if needed
- Create JIRA ticket for permanent fix
- Update this runbook with lessons learned

---

### 1.2 Trading Service Down

**Alert**: `TradingServiceDown`
**Severity**: CRITICAL
**Impact**: All trading operations halted

#### Diagnosis Steps

1. **Check Service Status**:
```bash
# Docker
docker ps | grep trading_service

# Kubernetes
kubectl get pods -l app=trading-service

# Systemd
systemctl status trading_service
```

2. **Check Logs**:
```bash
# Docker
docker logs --tail=100 trading_service

# Kubernetes
kubectl logs -l app=trading-service --tail=100

# Journald
journalctl -u trading_service -n 100 --no-pager
```

3. **Check Dependencies**:
```bash
# Database
psql -U foxhunt -d foxhunt -c "SELECT 1;"

# Redis
redis-cli ping

# Broker connectivity
telnet broker.example.com 9999
```

#### Resolution Steps

**Immediate Actions** (< 1 minute):
1. Attempt service restart
2. Check if process crashed or was killed (OOM)
3. Verify dependencies are healthy

**Service Restart**:
```bash
# Docker
docker restart trading_service

# Kubernetes
kubectl rollout restart deployment/trading-service

# Systemd
systemctl restart trading_service
```

**If Restart Fails**:
1. Check configuration files for errors
2. Verify database migrations are current
3. Check for port conflicts
4. Review recent deployments and rollback if needed

**Escalation**:
- If service won't start after 3 attempts → Page L2
- If crash cause unclear → Page L3
- If data corruption suspected → Page L4 + CTO

---

### 1.3 High Order Rejection Rate

**Alert**: `HighOrderRejectionRate`
**Severity**: WARNING
**Impact**: Strategy execution quality degraded

#### Diagnosis Steps

1. **Check Rejection Reasons**:
```bash
# Query rejection metrics
curl -s 'http://prometheus:9090/api/v1/query?query=rate(trading_orders_rejected_total[5m])' | jq

# Check database for recent rejections
psql -U foxhunt -d foxhunt -c "
SELECT rejection_reason, COUNT(*) as count
FROM order_rejections
WHERE created_at > NOW() - INTERVAL '10 minutes'
GROUP BY rejection_reason
ORDER BY count DESC
LIMIT 10;
"
```

2. **Identify Root Cause**:
- **Risk limit breach**: Check current positions and exposure
- **Validation errors**: Check order parameters
- **Broker rejections**: Check broker connectivity and account status
- **Compliance violations**: Check circuit breakers and regulatory constraints

#### Resolution Steps

**By Rejection Type**:

1. **Risk Limit Breach**:
```bash
# Check current exposure
curl -s 'http://trading-service:9091/api/risk/exposure'

# Reduce positions if needed
# (Manual intervention required - consult risk team)
```

2. **Validation Errors**:
```bash
# Review order validation logic
tail -f /var/log/trading_service/validation_errors.log

# Check for configuration drift
diff <(kubectl get configmap trading-config -o yaml) production_config.yaml
```

3. **Broker Rejections**:
```bash
# Check broker status
curl -s http://trading-service:9091/api/broker/status

# Verify account balance and margin
# Contact broker support if persistent
```

---

### 1.4 Position Limit Approaching

**Alert**: `PositionLimitApproaching`
**Severity**: WARNING
**Impact**: Risk of hitting position limits

#### Diagnosis Steps

```bash
# Check current positions
psql -U foxhunt -d foxhunt -c "
SELECT symbol, SUM(quantity) as total_position, MAX(position_limit) as limit,
       100.0 * SUM(quantity) / MAX(position_limit) as utilization_pct
FROM positions
GROUP BY symbol
HAVING 100.0 * SUM(quantity) / MAX(position_limit) > 80
ORDER BY utilization_pct DESC;
"
```

#### Resolution Steps

1. **Immediate**: Monitor position buildup closely
2. **Short-term**: Adjust strategy parameters to limit position sizes
3. **Long-term**: Review and adjust position limits with risk team

---

## 2. API Gateway Incidents

### 2.1 High Authentication Latency

**Alert**: `AuthLatencySLAViolation`
**Severity**: CRITICAL
**Impact**: All client requests delayed

#### Diagnosis Steps

1. **Check Auth Metrics**:
```bash
# Check cache hit rate
curl -s 'http://prometheus:9090/api/v1/query?query=100*rate(api_gateway_rbac_cache_hits[5m])/(rate(api_gateway_rbac_cache_hits[5m])+rate(api_gateway_rbac_cache_misses[5m]))'

# Check Redis latency
redis-cli --latency -h redis -p 6379

# Check database query time
psql -U foxhunt -d foxhunt -c "SELECT query, mean_exec_time FROM pg_stat_statements WHERE query LIKE '%rbac%' ORDER BY mean_exec_time DESC LIMIT 5;"
```

2. **Check Dependencies**:
```bash
# Redis health
redis-cli info | grep -E "used_memory|connected_clients|blocked_clients"

# Database connections
psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM pg_stat_activity;"
```

#### Resolution Steps

1. **Increase Cache Hit Rate**:
```bash
# Check cache size
redis-cli dbsize

# Increase cache expiry if needed (requires restart)
# Edit config: rbac_cache_ttl=3600
```

2. **Scale Redis**:
```bash
# Add more memory
# Or: Deploy Redis cluster for horizontal scaling
```

3. **Optimize Database Queries**:
```sql
-- Add index if missing
CREATE INDEX CONCURRENTLY idx_rbac_user_role ON user_roles(user_id, role_id);
```

---

### 2.2 Circuit Breaker Open

**Alert**: `CircuitBreakerOpen`
**Severity**: CRITICAL
**Impact**: Backend service {{ labels.service }} unavailable

#### Diagnosis Steps

```bash
# Check circuit breaker state
curl -s http://api-gateway:9090/api/circuit-breakers | jq

# Check backend service health
curl -s http://{{ labels.service }}:9091/health
```

#### Resolution Steps

1. **If Backend is Healthy**:
```bash
# Manually reset circuit breaker
curl -X POST http://api-gateway:9090/api/circuit-breakers/{{ labels.service }}/reset
```

2. **If Backend is Unhealthy**:
- See respective service runbook
- Circuit breaker will auto-close after backend recovers
- Monitor recovery: `watch -n 1 'curl -s http://api-gateway:9090/api/circuit-breakers/{{ labels.service }}'`

---

## 3. ML Training Service Incidents

### 3.1 Model Accuracy Degraded

**Alert**: `MLModelAccuracyDegraded`
**Severity**: CRITICAL
**Impact**: Trading strategy performance severely degraded

#### Diagnosis Steps

1. **Check Model Metrics**:
```bash
# Current accuracy
curl -s 'http://prometheus:9090/api/v1/query?query=ml_model_accuracy{model="mamba2"}'

# Model drift score
curl -s 'http://prometheus:9090/api/v1/query?query=ml_model_drift_score{model="mamba2"}'

# Prediction confidence
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.50, rate(ml_prediction_confidence_bucket[5m]))'
```

2. **Check Training Data**:
```bash
# Data freshness
psql -U foxhunt -d foxhunt -c "SELECT MAX(timestamp) FROM training_data;"

# Data quality checks
python /opt/foxhunt/ml/scripts/data_quality_check.py --last-24h
```

3. **Check for Distribution Shift**:
```bash
# Feature distribution analysis
python /opt/foxhunt/ml/scripts/feature_drift_analysis.py --model mamba2
```

#### Resolution Steps

**Immediate** (< 15 minutes):
1. Switch to backup model if available
2. Reduce model weight in ensemble
3. Enable conservative mode (wider stop-losses)

**Short-term** (< 2 hours):
1. Retrain model with recent data
2. Review feature engineering pipeline
3. Check for data quality issues

**Long-term**:
1. Implement online learning / continuous training
2. Add model monitoring dashboard
3. Automate model retraining triggers

#### Model Retraining Procedure

```bash
# Trigger retraining job
kubectl create job --from=cronjob/ml-training manual-retrain-$(date +%s)

# Monitor training progress
kubectl logs -f job/manual-retrain-<timestamp>

# Validate new model
python /opt/foxhunt/ml/scripts/model_validation.py --model-path s3://models/mamba2-latest

# Deploy new model
kubectl rollout restart deployment/ml-training-service
```

---

### 3.2 GPU Temperature High

**Alert**: `GPUTemperatureHigh`
**Severity**: CRITICAL
**Impact**: Risk of thermal throttling and hardware damage

#### Diagnosis Steps

```bash
# Check GPU status
nvidia-smi

# Check temperature history
curl -s 'http://prometheus:9090/api/v1/query?query=ml_gpu_temperature_celsius[10m]' | jq

# Check cooling system
sensors | grep -i fan
```

#### Resolution Steps

**Immediate**:
1. Reduce GPU workload:
```bash
# Pause non-critical training jobs
kubectl scale deployment/ml-training-batch --replicas=0

# Reduce inference load
# (Temporarily direct traffic to CPU inference)
```

2. Check physical cooling:
- Verify data center HVAC is working
- Check for blocked air vents
- Verify GPU fans are spinning

**If Temperature Remains Critical** (> 90°C):
1. Gracefully shut down GPU workloads
2. Contact hardware support
3. Failover to backup ML service (CPU-based)

---

### 3.3 Model Loading Failures

**Alert**: `ModelLoadingFailures`
**Severity**: CRITICAL
**Impact**: Unable to serve ML predictions

#### Diagnosis Steps

```bash
# Check S3 connectivity
aws s3 ls s3://foxhunt-models/

# Check model files
aws s3 ls s3://foxhunt-models/mamba2/ --recursive

# Check service logs
kubectl logs -l app=ml-training-service --tail=50 | grep -i "model.*load.*error"

# Check disk space
df -h /var/lib/models
```

#### Resolution Steps

1. **S3 Connection Issues**:
```bash
# Verify AWS credentials
aws sts get-caller-identity

# Check network connectivity
curl -I https://s3.amazonaws.com

# Check IAM permissions
aws s3api get-bucket-policy --bucket foxhunt-models
```

2. **Corrupted Model Files**:
```bash
# Re-download model
aws s3 sync s3://foxhunt-models/mamba2/latest /var/lib/models/mamba2/

# Verify checksums
md5sum /var/lib/models/mamba2/*.pt
```

3. **Disk Space**:
```bash
# Clean old model versions
find /var/lib/models -name "*.pt" -mtime +30 -delete

# Or: Expand disk
# (Requires infrastructure team)
```

---

## 4. Backtesting Service Incidents

### 4.1 Slow Backtest Execution

**Alert**: `BacktestExecutionTimeSlow`
**Severity**: WARNING
**Impact**: Strategy testing taking too long

#### Diagnosis Steps

```bash
# Check current backtests
curl -s http://backtesting-service:9092/api/backtests/active | jq

# Check resource usage
top -p $(pgrep backtesting_service)

# Check data loading time
tail -f /var/log/backtesting_service/performance.log | grep "parquet_read_time"
```

#### Resolution Steps

1. **Optimize Data Loading**:
```bash
# Check Parquet file size
ls -lh /data/historical/*.parquet

# Compact Parquet files if needed
python /opt/foxhunt/data/scripts/compact_parquet.py --input-dir /data/historical
```

2. **Reduce Concurrent Backtests**:
```bash
# Limit active backtests
curl -X PUT http://backtesting-service:9092/api/config/max_concurrent_backtests -d '{"value": 5}'
```

3. **Scale Resources**:
```bash
# Add more compute
kubectl scale deployment/backtesting-service --replicas=3
```

---

## 5. Database Incidents

### 5.1 PostgreSQL Connection Pool Exhaustion

**Alert**: `PostgreSQLConnectionPoolExhaustion`
**Severity**: CRITICAL
**Impact**: New database connections may fail

#### Diagnosis Steps

```bash
# Check current connections
psql -U foxhunt -d foxhunt -c "
SELECT client_addr, state, COUNT(*)
FROM pg_stat_activity
WHERE datname = 'foxhunt'
GROUP BY client_addr, state
ORDER BY count DESC;
"

# Check for long-running queries
psql -U foxhunt -d foxhunt -c "
SELECT pid, now() - query_start as duration, state, query
FROM pg_stat_activity
WHERE state != 'idle' AND query_start < now() - INTERVAL '1 minute'
ORDER BY duration DESC
LIMIT 10;
"

# Check for idle connections
psql -U foxhunt -d foxhunt -c "
SELECT state, COUNT(*)
FROM pg_stat_activity
GROUP BY state;
"
```

#### Resolution Steps

**Immediate**:
1. Kill idle connections:
```sql
-- Kill idle connections older than 5 minutes
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
  AND query_start < now() - INTERVAL '5 minutes'
  AND pid != pg_backend_pid();
```

2. Kill long-running queries (if safe):
```sql
-- Review first, then kill
SELECT pg_terminate_backend(<pid>);
```

**Short-term**:
1. Restart services with connection leaks
2. Increase max_connections temporarily:
```sql
-- Requires PostgreSQL restart
ALTER SYSTEM SET max_connections = 200;
-- Then: sudo systemctl restart postgresql
```

**Long-term**:
1. Implement connection pooling (PgBouncer)
2. Fix connection leaks in application code
3. Set idle timeout in applications

---

### 5.2 PostgreSQL Slow Queries

**Alert**: `PostgreSQLSlowQueries`
**Severity**: WARNING
**Impact**: Database performance degraded

#### Diagnosis Steps

```bash
# Identify slow queries
psql -U foxhunt -d foxhunt -c "
SELECT query, calls, mean_exec_time, max_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
"

# Check for missing indexes
psql -U foxhunt -d foxhunt -c "
SELECT schemaname, tablename, attname, n_distinct, correlation
FROM pg_stats
WHERE schemaname = 'public'
  AND n_distinct > 100
  AND correlation < 0.1
ORDER BY n_distinct DESC;
"

# Check for bloat
psql -U foxhunt -d foxhunt -c "
SELECT schemaname, tablename,
       pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;
"
```

#### Resolution Steps

1. **Add Missing Indexes**:
```sql
-- Analyze query plan first
EXPLAIN ANALYZE <slow_query>;

-- Add index if beneficial
CREATE INDEX CONCURRENTLY idx_<table>_<column> ON <table>(<column>);
```

2. **VACUUM/ANALYZE**:
```sql
-- For specific table
VACUUM ANALYZE <table>;

-- For entire database (during maintenance window)
VACUUMDB -U foxhunt -d foxhunt --analyze --verbose
```

3. **Optimize Query**:
- Rewrite query to use indexes
- Add query hints if needed
- Break complex queries into smaller chunks

---

### 5.3 Redis Down

**Alert**: `RedisDown`
**Severity**: CRITICAL
**Impact**: JWT revocation and caching unavailable

#### Diagnosis Steps

```bash
# Check Redis process
ps aux | grep redis-server

# Check logs
tail -f /var/log/redis/redis-server.log

# Check connectivity
redis-cli -h redis -p 6379 ping
```

#### Resolution Steps

1. **Restart Redis**:
```bash
# Docker
docker restart redis

# Systemd
sudo systemctl restart redis

# Check status
redis-cli ping
```

2. **If Restart Fails**:
```bash
# Check configuration
redis-cli config get '*'

# Check disk space
df -h /var/lib/redis

# Check memory
free -h

# Review logs for errors
tail -100 /var/log/redis/redis-server.log
```

3. **Restore from Backup** (if data corruption):
```bash
# Stop Redis
sudo systemctl stop redis

# Restore RDB file
sudo cp /backups/redis/dump.rdb /var/lib/redis/

# Start Redis
sudo systemctl start redis
```

---

## 6. Infrastructure Incidents

### 6.1 High CPU Utilization

**Alert**: `SystemCPUUtilizationCritical`
**Severity**: CRITICAL
**Impact**: All services performance degraded

#### Diagnosis Steps

```bash
# Identify top CPU consumers
top -b -n 1 | head -20

# Check per-process CPU
ps aux --sort=-%cpu | head -10

# Check CPU steal (for VMs)
mpstat 1 10 | grep -E "steal|Average"
```

#### Resolution Steps

1. **Immediate**:
```bash
# Kill non-essential processes
kill -9 <pid>

# Reduce process priority
renice +10 -p <pid>

# Disable non-critical cron jobs
crontab -e  # Comment out non-essential jobs
```

2. **Short-term**:
- Scale horizontally (add more instances)
- Optimize hot path code
- Enable CPU throttling on non-critical services

3. **Long-term**:
- Vertical scaling (larger instances)
- Code profiling and optimization
- Better resource isolation (cgroups, Kubernetes limits)

---

### 6.2 Disk Space Critical

**Alert**: `DiskSpaceCritical`
**Severity**: CRITICAL
**Impact**: Risk of service failures

#### Diagnosis Steps

```bash
# Check disk usage
df -h

# Find largest directories
du -h / | sort -rh | head -20

# Find large files
find / -type f -size +1G -exec ls -lh {} \; 2>/dev/null

# Check for deleted files still held by processes
lsof | grep deleted
```

#### Resolution Steps

**Immediate** (< 5 minutes):
```bash
# Clean logs
find /var/log -name "*.log" -mtime +7 -delete
journalctl --vacuum-time=7d

# Clean Docker
docker system prune -af --volumes

# Clean package cache
apt-get clean  # Ubuntu/Debian
yum clean all  # RHEL/CentOS

# Clean temp files
rm -rf /tmp/*
```

**Short-term**:
```bash
# Archive old data
tar -czf /backups/old_logs_$(date +%Y%m%d).tar.gz /var/log/*.log.1
rm -f /var/log/*.log.1

# Move to S3
aws s3 sync /data/historical s3://foxhunt-archives/historical/
rm -rf /data/historical/2024-*
```

**Long-term**:
- Implement log rotation policies
- Add disk space monitoring and auto-cleanup
- Expand disk capacity

---

### 6.3 Network Packet Loss

**Alert**: `NetworkPacketLossHigh`
**Severity**: CRITICAL
**Impact**: Network reliability degraded

#### Diagnosis Steps

```bash
# Check packet loss
ping -c 100 broker.example.com | grep loss

# Check interface statistics
ifconfig eth0 | grep -E "errors|dropped|overruns"

# Check network driver errors
ethtool -S eth0 | grep -E "error|drop"

# Check network congestion
netstat -s | grep -E "segment|retransmit"
```

#### Resolution Steps

1. **Immediate**:
```bash
# Check cable/connection physically
# Restart network interface
sudo ifdown eth0 && sudo ifup eth0

# Check switch port status
# (Requires network team)
```

2. **Short-term**:
- Failover to backup network interface
- Route traffic through alternate path
- Contact ISP/network provider

3. **Long-term**:
- Replace faulty hardware
- Upgrade network capacity
- Implement redundant network paths

---

## 7. Incident Response Procedures

### 7.1 Incident Classification

| Severity | Definition | Response Time | Escalation |
|----------|------------|---------------|------------|
| **P0 - Critical** | Trading halted, data loss, security breach | < 5 minutes | Immediate: CTO + CEO |
| **P1 - High** | Major service degradation, SLA breach | < 15 minutes | L2 + Manager |
| **P2 - Medium** | Minor service impact, approaching limits | < 1 hour | L1 + L2 |
| **P3 - Low** | No immediate impact, monitoring alert | < 4 hours | L1 only |

### 7.2 Incident Response Workflow

**1. ACKNOWLEDGE** (< 5 minutes)
```bash
# Acknowledge alert in PagerDuty
# Join #incidents Slack channel
# Post initial status: "Investigating [alert-name]"
```

**2. ASSESS** (< 15 minutes)
- Review alert details and metrics
- Check dashboards and logs
- Determine severity and impact
- Escalate if needed

**3. MITIGATE** (ASAP)
- Apply immediate fixes from runbook
- Communicate progress every 15 minutes
- Update status page if customer-facing

**4. RESOLVE**
- Verify issue is fully resolved
- Confirm metrics back to normal
- Document resolution steps

**5. POST-INCIDENT** (within 48 hours)
- Write incident report (template below)
- Schedule post-mortem meeting
- Create action items for prevention
- Update runbooks

### 7.3 Communication Template

**Initial Post** (#incidents channel):
```
🚨 **INCIDENT DETECTED**
Alert: [Alert Name]
Severity: [P0/P1/P2/P3]
Impact: [Description]
Assigned: @oncall
Status: Investigating
ETA: [Time]
```

**Update Posts** (every 15 minutes):
```
📊 **INCIDENT UPDATE**
Time: [HH:MM]
Status: [Investigating/Mitigating/Resolved]
Progress: [What we've done]
Next Steps: [What we're doing next]
ETA: [Updated estimate]
```

**Resolution Post**:
```
✅ **INCIDENT RESOLVED**
Alert: [Alert Name]
Duration: [X minutes]
Root Cause: [Brief description]
Fix Applied: [What we did]
Follow-up: [Ticket links]
Post-mortem: [Scheduled date/time]
```

### 7.4 Post-Incident Report Template

```markdown
# Incident Report: [Alert Name]

**Date**: YYYY-MM-DD
**Time**: HH:MM UTC
**Duration**: X minutes
**Severity**: P0/P1/P2/P3
**Responders**: @person1, @person2

## Summary
[One paragraph describing what happened]

## Impact
- **Users Affected**: X users/services
- **Revenue Impact**: $X (if applicable)
- **SLA Breach**: Yes/No (X% of error budget)

## Timeline
- **HH:MM** - Alert fired
- **HH:MM** - Incident acknowledged
- **HH:MM** - Root cause identified
- **HH:MM** - Fix applied
- **HH:MM** - Incident resolved

## Root Cause
[Detailed explanation of what went wrong and why]

## Resolution
[Detailed explanation of how it was fixed]

## Action Items
1. [ ] [Preventative measure] - @owner - [Due date]
2. [ ] [Monitoring improvement] - @owner - [Due date]
3. [ ] [Runbook update] - @owner - [Due date]

## Lessons Learned
- **What went well**: [Things that worked]
- **What went poorly**: [Things to improve]
- **Where we got lucky**: [Things that could have been worse]
```

---

## 8. Useful Commands Reference

### 8.1 Service Management

```bash
# Docker
docker ps | grep <service>
docker logs --tail=100 <container>
docker restart <container>
docker stats <container>

# Kubernetes
kubectl get pods -l app=<service>
kubectl logs -l app=<service> --tail=100
kubectl rollout restart deployment/<service>
kubectl describe pod <pod-name>

# Systemd
systemctl status <service>
systemctl restart <service>
journalctl -u <service> -n 100 --no-pager
```

### 8.2 Monitoring Queries

```bash
# Prometheus query
curl -s "http://prometheus:9090/api/v1/query?query=<promql>" | jq

# Check alert status
curl -s "http://alertmanager:9093/api/v2/alerts" | jq

# Check Grafana
curl -s "http://grafana:3000/api/dashboards/uid/<dashboard-uid>"
```

### 8.3 Database Queries

```bash
# PostgreSQL
psql -U foxhunt -d foxhunt

# Redis
redis-cli -h redis -p 6379

# Quick health check
psql -U foxhunt -d foxhunt -c "SELECT 1;"
redis-cli ping
```

---

## 9. Escalation Contacts

| Role | Primary | Secondary | PagerDuty Schedule |
|------|---------|-----------|-------------------|
| On-Call Engineer | Rotation | Rotation | 24/7 |
| Trading Platform Lead | John Doe | Jane Smith | Business hours |
| SRE Manager | Alice Brown | Bob Johnson | Business hours |
| VP Engineering | Charlie Davis | - | 24/7 (P0 only) |
| CTO | David Wilson | - | 24/7 (P0 only) |

**Emergency Hotline**: +1-555-FOXHUNT (24/7)
**Slack**: #oncall-engineering
**Email**: oncall@foxhunt.io

---

## 10. Runbook Maintenance

### 10.1 Update Process

1. After each incident, update relevant runbook section
2. Add new diagnosis steps if discovered
3. Document resolution steps that worked
4. Remove outdated information

### 10.2 Review Schedule

- **Weekly**: On-call engineer reviews recent incidents
- **Monthly**: Team reviews all runbooks for accuracy
- **Quarterly**: Complete runbook audit and update

### 10.3 Feedback

Submit runbook improvements via:
- GitHub PR to `docs/monitoring/RUNBOOKS.md`
- Slack: #platform-engineering
- Email: oncall@foxhunt.io

---

**Document Version History**:
- v1.0 (2025-10-07): Initial runbook creation
- Next Review: 2025-11-07