foxhunt/docs/monitoring/RUNBOOKS.md

Operational Runbooks - Foxhunt HFT Trading System

Last Updated: 2025-10-07 Version: 1.0 On-Call Contact: #oncall-engineering Slack channel

---

Table of Contents

1. Trading Service Incidents
2. API Gateway Incidents
3. ML Training Service Incidents
4. Backtesting Service Incidents
5. Database Incidents
6. Infrastructure Incidents
7. Incident Response Procedures

---

1. Trading Service Incidents

1.1 High Order Latency (> 100μs p99)

Alert: OrderLatencySLAViolation Severity: CRITICAL Impact: Trading performance degraded, potential profit loss

Diagnosis Steps

1. Check Current Latency:

# Query Prometheus
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99, rate(trading_order_processing_microseconds_bucket[1m]))'

# Expected: < 100μs
# If > 150μs: CRITICAL emergency

2. Identify Bottleneck:

# Check CPU affinity
taskset -cp $(pgrep trading_service)

# Check CPU usage
top -p $(pgrep trading_service)

# Check network latency to broker
ping -c 10 broker.example.com

# Check database query latency
psql -U foxhunt -d foxhunt -c "SELECT * FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

3. Check System Resources:

# Memory
free -h

# CPU
mpstat 1 10

# Disk I/O
iostat -x 1 10

# Network
iftop -i eth0

Resolution Steps

Immediate Actions (< 5 minutes):

1. Verify CPU affinity is set correctly
2. Check for competing processes and kill if necessary
3. Verify network connectivity to broker
4. Check for database connection pool exhaustion

Short-term Fixes (< 30 minutes):

1. Restart trading service if memory leak suspected
2. Scale horizontally by adding more instances
3. Enable circuit breakers to shed load
4. Review recent code deployments for regressions

Long-term Solutions:

1. Optimize hot path code
2. Add caching for frequent database lookups
3. Review and tune database indexes
4. Upgrade hardware (CPU, network)

Post-Incident

• Document root cause
• Update monitoring thresholds if needed
• Create JIRA ticket for permanent fix
• Update this runbook with lessons learned

---

1.2 Trading Service Down

Alert: TradingServiceDown Severity: CRITICAL Impact: All trading operations halted

Diagnosis Steps

1. Check Service Status:

# Docker
docker ps | grep trading_service

# Kubernetes
kubectl get pods -l app=trading-service

# Systemd
systemctl status trading_service

2. Check Logs:

# Docker
docker logs --tail=100 trading_service

# Kubernetes
kubectl logs -l app=trading-service --tail=100

# Journald
journalctl -u trading_service -n 100 --no-pager

3. Check Dependencies:

# Database
psql -U foxhunt -d foxhunt -c "SELECT 1;"

# Redis
redis-cli ping

# Broker connectivity
telnet broker.example.com 9999

Resolution Steps

Immediate Actions (< 1 minute):

1. Attempt service restart
2. Check if process crashed or was killed (OOM)
3. Verify dependencies are healthy

Service Restart:

# Docker
docker restart trading_service

# Kubernetes
kubectl rollout restart deployment/trading-service

# Systemd
systemctl restart trading_service

If Restart Fails:

1. Check configuration files for errors
2. Verify database migrations are current
3. Check for port conflicts
4. Review recent deployments and rollback if needed

Escalation:

• If service won't start after 3 attempts → Page L2
• If crash cause unclear → Page L3
• If data corruption suspected → Page L4 + CTO

---

1.3 High Order Rejection Rate

Alert: HighOrderRejectionRate Severity: WARNING Impact: Strategy execution quality degraded

Diagnosis Steps

1. Check Rejection Reasons:

# Query rejection metrics
curl -s 'http://prometheus:9090/api/v1/query?query=rate(trading_orders_rejected_total[5m])' | jq

# Check database for recent rejections
psql -U foxhunt -d foxhunt -c "
SELECT rejection_reason, COUNT(*) as count
FROM order_rejections
WHERE created_at > NOW() - INTERVAL '10 minutes'
GROUP BY rejection_reason
ORDER BY count DESC
LIMIT 10;
"

2. Identify Root Cause:

• Risk limit breach: Check current positions and exposure
• Validation errors: Check order parameters
• Broker rejections: Check broker connectivity and account status
• Compliance violations: Check circuit breakers and regulatory constraints

Resolution Steps

By Rejection Type:

1. Risk Limit Breach:

# Check current exposure
curl -s 'http://trading-service:9091/api/risk/exposure'

# Reduce positions if needed
# (Manual intervention required - consult risk team)

2. Validation Errors:

# Review order validation logic
tail -f /var/log/trading_service/validation_errors.log

# Check for configuration drift
diff <(kubectl get configmap trading-config -o yaml) production_config.yaml

3. Broker Rejections:

# Check broker status
curl -s http://trading-service:9091/api/broker/status

# Verify account balance and margin
# Contact broker support if persistent

---

1.4 Position Limit Approaching

Alert: PositionLimitApproaching Severity: WARNING Impact: Risk of hitting position limits

Diagnosis Steps

# Check current positions
psql -U foxhunt -d foxhunt -c "
SELECT symbol, SUM(quantity) as total_position, MAX(position_limit) as limit,
       100.0 * SUM(quantity) / MAX(position_limit) as utilization_pct
FROM positions
GROUP BY symbol
HAVING 100.0 * SUM(quantity) / MAX(position_limit) > 80
ORDER BY utilization_pct DESC;
"

Resolution Steps

1. Immediate: Monitor position buildup closely
2. Short-term: Adjust strategy parameters to limit position sizes
3. Long-term: Review and adjust position limits with risk team

---

2. API Gateway Incidents

2.1 High Authentication Latency

Alert: AuthLatencySLAViolation Severity: CRITICAL Impact: All client requests delayed

Diagnosis Steps

1. Check Auth Metrics:

# Check cache hit rate
curl -s 'http://prometheus:9090/api/v1/query?query=100*rate(api_gateway_rbac_cache_hits[5m])/(rate(api_gateway_rbac_cache_hits[5m])+rate(api_gateway_rbac_cache_misses[5m]))'

# Check Redis latency
redis-cli --latency -h redis -p 6379

# Check database query time
psql -U foxhunt -d foxhunt -c "SELECT query, mean_exec_time FROM pg_stat_statements WHERE query LIKE '%rbac%' ORDER BY mean_exec_time DESC LIMIT 5;"

2. Check Dependencies:

# Redis health
redis-cli info | grep -E "used_memory|connected_clients|blocked_clients"

# Database connections
psql -U foxhunt -d foxhunt -c "SELECT count(*) FROM pg_stat_activity;"

Resolution Steps

1. Increase Cache Hit Rate:

# Check cache size
redis-cli dbsize

# Increase cache expiry if needed (requires restart)
# Edit config: rbac_cache_ttl=3600

2. Scale Redis:

# Add more memory
# Or: Deploy Redis cluster for horizontal scaling

3. Optimize Database Queries:

-- Add index if missing
CREATE INDEX CONCURRENTLY idx_rbac_user_role ON user_roles(user_id, role_id);

---

2.2 Circuit Breaker Open

Alert: CircuitBreakerOpen Severity: CRITICAL Impact: Backend service {{ labels.service }} unavailable

Diagnosis Steps

# Check circuit breaker state
curl -s http://api-gateway:9090/api/circuit-breakers | jq

# Check backend service health
curl -s http://{{ labels.service }}:9091/health

Resolution Steps

1. If Backend is Healthy:

# Manually reset circuit breaker
curl -X POST http://api-gateway:9090/api/circuit-breakers/{{ labels.service }}/reset

2. If Backend is Unhealthy:

• See respective service runbook
• Circuit breaker will auto-close after backend recovers
• Monitor recovery: watch -n 1 'curl -s http://api-gateway:9090/api/circuit-breakers/{{ labels.service }}'

---

3. ML Training Service Incidents

3.1 Model Accuracy Degraded

Alert: MLModelAccuracyDegraded Severity: CRITICAL Impact: Trading strategy performance severely degraded

Diagnosis Steps

1. Check Model Metrics:

# Current accuracy
curl -s 'http://prometheus:9090/api/v1/query?query=ml_model_accuracy{model="mamba2"}'

# Model drift score
curl -s 'http://prometheus:9090/api/v1/query?query=ml_model_drift_score{model="mamba2"}'

# Prediction confidence
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.50, rate(ml_prediction_confidence_bucket[5m]))'

2. Check Training Data:

# Data freshness
psql -U foxhunt -d foxhunt -c "SELECT MAX(timestamp) FROM training_data;"

# Data quality checks
python /opt/foxhunt/ml/scripts/data_quality_check.py --last-24h

3. Check for Distribution Shift:

# Feature distribution analysis
python /opt/foxhunt/ml/scripts/feature_drift_analysis.py --model mamba2

Resolution Steps

Immediate (< 15 minutes):

1. Switch to backup model if available
2. Reduce model weight in ensemble
3. Enable conservative mode (wider stop-losses)

Short-term (< 2 hours):

1. Retrain model with recent data
2. Review feature engineering pipeline
3. Check for data quality issues

Long-term:

1. Implement online learning / continuous training
2. Add model monitoring dashboard
3. Automate model retraining triggers

Model Retraining Procedure

# Trigger retraining job
kubectl create job --from=cronjob/ml-training manual-retrain-$(date +%s)

# Monitor training progress
kubectl logs -f job/manual-retrain-<timestamp>

# Validate new model
python /opt/foxhunt/ml/scripts/model_validation.py --model-path s3://models/mamba2-latest

# Deploy new model
kubectl rollout restart deployment/ml-training-service

---

3.2 GPU Temperature High

Alert: GPUTemperatureHigh Severity: CRITICAL Impact: Risk of thermal throttling and hardware damage

Diagnosis Steps

# Check GPU status
nvidia-smi

# Check temperature history
curl -s 'http://prometheus:9090/api/v1/query?query=ml_gpu_temperature_celsius[10m]' | jq

# Check cooling system
sensors | grep -i fan

Resolution Steps

Immediate:

1. Reduce GPU workload:

# Pause non-critical training jobs
kubectl scale deployment/ml-training-batch --replicas=0

# Reduce inference load
# (Temporarily direct traffic to CPU inference)

2. Check physical cooling:

• Verify data center HVAC is working
• Check for blocked air vents
• Verify GPU fans are spinning

If Temperature Remains Critical (> 90°C):

1. Gracefully shut down GPU workloads
2. Contact hardware support
3. Failover to backup ML service (CPU-based)

---

3.3 Model Loading Failures

Alert: ModelLoadingFailures Severity: CRITICAL Impact: Unable to serve ML predictions

Diagnosis Steps

# Check S3 connectivity
aws s3 ls s3://foxhunt-models/

# Check model files
aws s3 ls s3://foxhunt-models/mamba2/ --recursive

# Check service logs
kubectl logs -l app=ml-training-service --tail=50 | grep -i "model.*load.*error"

# Check disk space
df -h /var/lib/models

Resolution Steps

1. S3 Connection Issues:

# Verify AWS credentials
aws sts get-caller-identity

# Check network connectivity
curl -I https://s3.amazonaws.com

# Check IAM permissions
aws s3api get-bucket-policy --bucket foxhunt-models

2. Corrupted Model Files:

# Re-download model
aws s3 sync s3://foxhunt-models/mamba2/latest /var/lib/models/mamba2/

# Verify checksums
md5sum /var/lib/models/mamba2/*.pt

3. Disk Space:

# Clean old model versions
find /var/lib/models -name "*.pt" -mtime +30 -delete

# Or: Expand disk
# (Requires infrastructure team)

---

4. Backtesting Service Incidents

4.1 Slow Backtest Execution

Alert: BacktestExecutionTimeSlow Severity: WARNING Impact: Strategy testing taking too long

Diagnosis Steps

# Check current backtests
curl -s http://backtesting-service:9092/api/backtests/active | jq

# Check resource usage
top -p $(pgrep backtesting_service)

# Check data loading time
tail -f /var/log/backtesting_service/performance.log | grep "parquet_read_time"

Resolution Steps

1. Optimize Data Loading:

# Check Parquet file size
ls -lh /data/historical/*.parquet

# Compact Parquet files if needed
python /opt/foxhunt/data/scripts/compact_parquet.py --input-dir /data/historical

2. Reduce Concurrent Backtests:

# Limit active backtests
curl -X PUT http://backtesting-service:9092/api/config/max_concurrent_backtests -d '{"value": 5}'

3. Scale Resources:

# Add more compute
kubectl scale deployment/backtesting-service --replicas=3

---

5. Database Incidents

5.1 PostgreSQL Connection Pool Exhaustion

Alert: PostgreSQLConnectionPoolExhaustion Severity: CRITICAL Impact: New database connections may fail

Diagnosis Steps

# Check current connections
psql -U foxhunt -d foxhunt -c "
SELECT client_addr, state, COUNT(*)
FROM pg_stat_activity
WHERE datname = 'foxhunt'
GROUP BY client_addr, state
ORDER BY count DESC;
"

# Check for long-running queries
psql -U foxhunt -d foxhunt -c "
SELECT pid, now() - query_start as duration, state, query
FROM pg_stat_activity
WHERE state != 'idle' AND query_start < now() - INTERVAL '1 minute'
ORDER BY duration DESC
LIMIT 10;
"

# Check for idle connections
psql -U foxhunt -d foxhunt -c "
SELECT state, COUNT(*)
FROM pg_stat_activity
GROUP BY state;
"

Resolution Steps

Immediate:

1. Kill idle connections:

-- Kill idle connections older than 5 minutes
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
  AND query_start < now() - INTERVAL '5 minutes'
  AND pid != pg_backend_pid();

2. Kill long-running queries (if safe):

-- Review first, then kill
SELECT pg_terminate_backend(<pid>);

Short-term:

1. Restart services with connection leaks
2. Increase max_connections temporarily:

-- Requires PostgreSQL restart
ALTER SYSTEM SET max_connections = 200;
-- Then: sudo systemctl restart postgresql

Long-term:

1. Implement connection pooling (PgBouncer)
2. Fix connection leaks in application code
3. Set idle timeout in applications

---

5.2 PostgreSQL Slow Queries

Alert: PostgreSQLSlowQueries Severity: WARNING Impact: Database performance degraded

Diagnosis Steps

# Identify slow queries
psql -U foxhunt -d foxhunt -c "
SELECT query, calls, mean_exec_time, max_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
"

# Check for missing indexes
psql -U foxhunt -d foxhunt -c "
SELECT schemaname, tablename, attname, n_distinct, correlation
FROM pg_stats
WHERE schemaname = 'public'
  AND n_distinct > 100
  AND correlation < 0.1
ORDER BY n_distinct DESC;
"

# Check for bloat
psql -U foxhunt -d foxhunt -c "
SELECT schemaname, tablename,
       pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) as size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;
"

Resolution Steps

1. Add Missing Indexes:

-- Analyze query plan first
EXPLAIN ANALYZE <slow_query>;

-- Add index if beneficial
CREATE INDEX CONCURRENTLY idx_<table>_<column> ON <table>(<column>);

2. VACUUM/ANALYZE:

-- For specific table
VACUUM ANALYZE <table>;

-- For entire database (during maintenance window)
VACUUMDB -U foxhunt -d foxhunt --analyze --verbose

3. Optimize Query:

• Rewrite query to use indexes
• Add query hints if needed
• Break complex queries into smaller chunks

---

5.3 Redis Down

Alert: RedisDown Severity: CRITICAL Impact: JWT revocation and caching unavailable

Diagnosis Steps

# Check Redis process
ps aux | grep redis-server

# Check logs
tail -f /var/log/redis/redis-server.log

# Check connectivity
redis-cli -h redis -p 6379 ping

Resolution Steps

1. Restart Redis:

# Docker
docker restart redis

# Systemd
sudo systemctl restart redis

# Check status
redis-cli ping

2. If Restart Fails:

# Check configuration
redis-cli config get '*'

# Check disk space
df -h /var/lib/redis

# Check memory
free -h

# Review logs for errors
tail -100 /var/log/redis/redis-server.log

3. Restore from Backup (if data corruption):

# Stop Redis
sudo systemctl stop redis

# Restore RDB file
sudo cp /backups/redis/dump.rdb /var/lib/redis/

# Start Redis
sudo systemctl start redis

---

6. Infrastructure Incidents

6.1 High CPU Utilization

Alert: SystemCPUUtilizationCritical Severity: CRITICAL Impact: All services performance degraded

Diagnosis Steps

# Identify top CPU consumers
top -b -n 1 | head -20

# Check per-process CPU
ps aux --sort=-%cpu | head -10

# Check CPU steal (for VMs)
mpstat 1 10 | grep -E "steal|Average"

Resolution Steps

1. Immediate:

# Kill non-essential processes
kill -9 <pid>

# Reduce process priority
renice +10 -p <pid>

# Disable non-critical cron jobs
crontab -e  # Comment out non-essential jobs

2. Short-term:

• Scale horizontally (add more instances)
• Optimize hot path code
• Enable CPU throttling on non-critical services

3. Long-term:

• Vertical scaling (larger instances)
• Code profiling and optimization
• Better resource isolation (cgroups, Kubernetes limits)

---

6.2 Disk Space Critical

Alert: DiskSpaceCritical Severity: CRITICAL Impact: Risk of service failures

Diagnosis Steps

# Check disk usage
df -h

# Find largest directories
du -h / | sort -rh | head -20

# Find large files
find / -type f -size +1G -exec ls -lh {} \; 2>/dev/null

# Check for deleted files still held by processes
lsof | grep deleted

Resolution Steps

Immediate (< 5 minutes):

# Clean logs
find /var/log -name "*.log" -mtime +7 -delete
journalctl --vacuum-time=7d

# Clean Docker
docker system prune -af --volumes

# Clean package cache
apt-get clean  # Ubuntu/Debian
yum clean all  # RHEL/CentOS

# Clean temp files
rm -rf /tmp/*

Short-term:

# Archive old data
tar -czf /backups/old_logs_$(date +%Y%m%d).tar.gz /var/log/*.log.1
rm -f /var/log/*.log.1

# Move to S3
aws s3 sync /data/historical s3://foxhunt-archives/historical/
rm -rf /data/historical/2024-*

Long-term:

• Implement log rotation policies
• Add disk space monitoring and auto-cleanup
• Expand disk capacity

---

6.3 Network Packet Loss

Alert: NetworkPacketLossHigh Severity: CRITICAL Impact: Network reliability degraded

Diagnosis Steps

# Check packet loss
ping -c 100 broker.example.com | grep loss

# Check interface statistics
ifconfig eth0 | grep -E "errors|dropped|overruns"

# Check network driver errors
ethtool -S eth0 | grep -E "error|drop"

# Check network congestion
netstat -s | grep -E "segment|retransmit"

Resolution Steps

1. Immediate:

# Check cable/connection physically
# Restart network interface
sudo ifdown eth0 && sudo ifup eth0

# Check switch port status
# (Requires network team)

2. Short-term:

• Failover to backup network interface
• Route traffic through alternate path
• Contact ISP/network provider

3. Long-term:

• Replace faulty hardware
• Upgrade network capacity
• Implement redundant network paths

---

7. Incident Response Procedures

7.1 Incident Classification

Severity	Definition	Response Time	Escalation
P0 - Critical	Trading halted, data loss, security breach	< 5 minutes	Immediate: CTO + CEO
P1 - High	Major service degradation, SLA breach	< 15 minutes	L2 + Manager
P2 - Medium	Minor service impact, approaching limits	< 1 hour	L1 + L2
P3 - Low	No immediate impact, monitoring alert	< 4 hours	L1 only

7.2 Incident Response Workflow

1. ACKNOWLEDGE (< 5 minutes)

# Acknowledge alert in PagerDuty
# Join #incidents Slack channel
# Post initial status: "Investigating [alert-name]"

2. ASSESS (< 15 minutes)

• Review alert details and metrics
• Check dashboards and logs
• Determine severity and impact
• Escalate if needed

3. MITIGATE (ASAP)

• Apply immediate fixes from runbook
• Communicate progress every 15 minutes
• Update status page if customer-facing

4. RESOLVE

• Verify issue is fully resolved
• Confirm metrics back to normal
• Document resolution steps

5. POST-INCIDENT (within 48 hours)

• Write incident report (template below)
• Schedule post-mortem meeting
• Create action items for prevention
• Update runbooks

7.3 Communication Template

Initial Post (#incidents channel):

🚨 **INCIDENT DETECTED**
Alert: [Alert Name]
Severity: [P0/P1/P2/P3]
Impact: [Description]
Assigned: @oncall
Status: Investigating
ETA: [Time]

Update Posts (every 15 minutes):

📊 **INCIDENT UPDATE**
Time: [HH:MM]
Status: [Investigating/Mitigating/Resolved]
Progress: [What we've done]
Next Steps: [What we're doing next]
ETA: [Updated estimate]

Resolution Post:

✅ **INCIDENT RESOLVED**
Alert: [Alert Name]
Duration: [X minutes]
Root Cause: [Brief description]
Fix Applied: [What we did]
Follow-up: [Ticket links]
Post-mortem: [Scheduled date/time]

7.4 Post-Incident Report Template

# Incident Report: [Alert Name]

**Date**: YYYY-MM-DD
**Time**: HH:MM UTC
**Duration**: X minutes
**Severity**: P0/P1/P2/P3
**Responders**: @person1, @person2

## Summary
[One paragraph describing what happened]

## Impact
- **Users Affected**: X users/services
- **Revenue Impact**: $X (if applicable)
- **SLA Breach**: Yes/No (X% of error budget)

## Timeline
- **HH:MM** - Alert fired
- **HH:MM** - Incident acknowledged
- **HH:MM** - Root cause identified
- **HH:MM** - Fix applied
- **HH:MM** - Incident resolved

## Root Cause
[Detailed explanation of what went wrong and why]

## Resolution
[Detailed explanation of how it was fixed]

## Action Items
1. [ ] [Preventative measure] - @owner - [Due date]
2. [ ] [Monitoring improvement] - @owner - [Due date]
3. [ ] [Runbook update] - @owner - [Due date]

## Lessons Learned
- **What went well**: [Things that worked]
- **What went poorly**: [Things to improve]
- **Where we got lucky**: [Things that could have been worse]

---

8. Useful Commands Reference

8.1 Service Management

# Docker
docker ps | grep <service>
docker logs --tail=100 <container>
docker restart <container>
docker stats <container>

# Kubernetes
kubectl get pods -l app=<service>
kubectl logs -l app=<service> --tail=100
kubectl rollout restart deployment/<service>
kubectl describe pod <pod-name>

# Systemd
systemctl status <service>
systemctl restart <service>
journalctl -u <service> -n 100 --no-pager

8.2 Monitoring Queries

# Prometheus query
curl -s "http://prometheus:9090/api/v1/query?query=<promql>" | jq

# Check alert status
curl -s "http://alertmanager:9093/api/v2/alerts" | jq

# Check Grafana
curl -s "http://grafana:3000/api/dashboards/uid/<dashboard-uid>"

8.3 Database Queries

# PostgreSQL
psql -U foxhunt -d foxhunt

# Redis
redis-cli -h redis -p 6379

# Quick health check
psql -U foxhunt -d foxhunt -c "SELECT 1;"
redis-cli ping

---

9. Escalation Contacts

Role	Primary	Secondary	PagerDuty Schedule
On-Call Engineer	Rotation	Rotation	24/7
Trading Platform Lead	John Doe	Jane Smith	Business hours
SRE Manager	Alice Brown	Bob Johnson	Business hours
VP Engineering	Charlie Davis	-	24/7 (P0 only)
CTO	David Wilson	-	24/7 (P0 only)

Emergency Hotline: +1-555-FOXHUNT (24/7) Slack: #oncall-engineering Email: oncall@foxhunt.io

---

10. Runbook Maintenance

10.1 Update Process

1. After each incident, update relevant runbook section
2. Add new diagnosis steps if discovered
3. Document resolution steps that worked
4. Remove outdated information

10.2 Review Schedule

• Weekly: On-call engineer reviews recent incidents
• Monthly: Team reviews all runbooks for accuracy
• Quarterly: Complete runbook audit and update

10.3 Feedback

Submit runbook improvements via:

• GitHub PR to docs/monitoring/RUNBOOKS.md
• Slack: #platform-engineering
• Email: oncall@foxhunt.io

---

Document Version History:

• v1.0 (2025-10-07): Initial runbook creation
• Next Review: 2025-11-07