foxhunt/docs/OPERATIONAL_RUNBOOK_V2.md

Foxhunt HFT Trading System - Operational Runbook

Version: 2.0 Last Updated: 2025-10-03 Wave: 71 - Production Operations Audience: System Operators, SREs, DevOps Engineers, Trading Desk

---

Quick Reference

Emergency Contacts

Role	Contact	Escalation
Trading Operations Lead	PagerDuty: trading-ops	Immediate for P0
Database Administrator	PagerDuty: dba-oncall	15 min for P1
Security Team	security@foxhunt.local	Immediate for breaches
Infrastructure Team	PagerDuty: infra-oncall	1 hour for P2
On-Call Engineer	PagerDuty: general-oncall	As per severity

Critical Commands (Bookmark This)

# Emergency stop all services
sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

# Health check all services
curl -s http://localhost:8080/health  # API Gateway
curl -s http://localhost:8081/health  # Trading Service
curl -s http://localhost:8082/health  # Backtesting Service
curl -s http://localhost:8083/health  # ML Training Service

# View real-time logs
sudo journalctl -u api_gateway -f
sudo journalctl -u trading_service -f

# Emergency rollback
sudo cp /usr/local/bin/api_gateway.bak /usr/local/bin/api_gateway
sudo systemctl restart api_gateway

# Database connection check
psql $DATABASE_URL -c "SELECT version();"

# Redis connection check
redis-cli -a $REDIS_PASSWORD ping

---

Table of Contents

1. Daily Startup Procedures
2. Service Monitoring
3. Configuration Management
4. Performance Monitoring
5. Log Management
6. Backup Verification
7. Emergency Procedures
8. Maintenance Windows
9. Common Troubleshooting Scenarios

---

Daily Startup Procedures

Pre-Market Startup Checklist

Execute 60 minutes before market open (e.g., 8:30 AM EST for 9:30 AM market open)

Step 1: Infrastructure Health Check (T-60min)

#!/bin/bash
# /opt/foxhunt/scripts/daily-startup-check.sh

echo "════════════════════════════════════════════════════════════"
echo "  Foxhunt HFT Trading System - Daily Startup Check"
echo "  $(date)"
echo "════════════════════════════════════════════════════════════"

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# 1. PostgreSQL Health
echo -e "\n1️⃣  Checking PostgreSQL..."
if sudo systemctl is-active --quiet postgresql; then
    echo -e "${GREEN}✓${NC} PostgreSQL service is running"
    if psql $DATABASE_URL -c "SELECT version();" > /dev/null 2>&1; then
        echo -e "${GREEN}✓${NC} Database connection successful"
    else
        echo -e "${RED}✗${NC} Database connection FAILED"
        exit 1
    fi
else
    echo -e "${RED}✗${NC} PostgreSQL service is NOT running"
    exit 1
fi

# 2. Redis Health
echo -e "\n2️⃣  Checking Redis..."
if sudo systemctl is-active --quiet redis-server; then
    echo -e "${GREEN}✓${NC} Redis service is running"
    if redis-cli -a $REDIS_PASSWORD ping | grep -q PONG; then
        echo -e "${GREEN}✓${NC} Redis connection successful"
    else
        echo -e "${RED}✗${NC} Redis connection FAILED"
        exit 1
    fi
else
    echo -e "${RED}✗${NC} Redis service is NOT running"
    exit 1
fi

# 3. Disk Space Check
echo -e "\n3️⃣  Checking Disk Space..."
DISK_USAGE=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -lt 80 ]; then
    echo -e "${GREEN}✓${NC} Disk usage: ${DISK_USAGE}%"
else
    echo -e "${YELLOW}⚠${NC}  WARNING: Disk usage above 80%: ${DISK_USAGE}%"
fi

# 4. Memory Check
echo -e "\n4️⃣  Checking Memory..."
MEMORY_USAGE=$(free | awk 'NR==2 {printf "%.0f", $3/$2 * 100}')
if [ $MEMORY_USAGE -lt 90 ]; then
    echo -e "${GREEN}✓${NC} Memory usage: ${MEMORY_USAGE}%"
else
    echo -e "${YELLOW}⚠${NC}  WARNING: Memory usage above 90%: ${MEMORY_USAGE}%"
fi

# 5. Network Connectivity
echo -e "\n5️⃣  Checking Network..."
if ping -c 3 8.8.8.8 > /dev/null 2>&1; then
    echo -e "${GREEN}✓${NC} Internet connectivity OK"
else
    echo -e "${RED}✗${NC} Internet connectivity FAILED"
    exit 1
fi

# 6. Time Synchronization (CRITICAL for HFT)
echo -e "\n6️⃣  Checking Time Synchronization..."
TIME_OFFSET=$(chronyc tracking | grep "System time" | awk '{print $4}')
if (( $(echo "$TIME_OFFSET < 0.0001" | bc -l) )); then
    echo -e "${GREEN}✓${NC} Time offset: ${TIME_OFFSET} seconds (< 100μs)"
else
    echo -e "${YELLOW}⚠${NC}  WARNING: Time offset: ${TIME_OFFSET} seconds (> 100μs)"
fi

# 7. Certificate Expiry Check
echo -e "\n7️⃣  Checking TLS Certificates..."
CERT_EXPIRY=$(openssl x509 -in /etc/foxhunt/certs/ca.crt -noout -enddate | cut -d= -f2)
CERT_EXPIRY_EPOCH=$(date -d "$CERT_EXPIRY" +%s)
CURRENT_EPOCH=$(date +%s)
DAYS_UNTIL_EXPIRY=$(( ($CERT_EXPIRY_EPOCH - $CURRENT_EPOCH) / 86400 ))

if [ $DAYS_UNTIL_EXPIRY -gt 30 ]; then
    echo -e "${GREEN}✓${NC} CA certificate expires in $DAYS_UNTIL_EXPIRY days"
else
    echo -e "${YELLOW}⚠${NC}  WARNING: CA certificate expires in $DAYS_UNTIL_EXPIRY days"
fi

echo -e "\n════════════════════════════════════════════════════════════"
echo -e "${GREEN}✓${NC} Infrastructure health check PASSED"
echo "════════════════════════════════════════════════════════════"

Failure Response:

• PostgreSQL down → See Database Recovery
• Redis down → See Cache Recovery
• Disk > 80% → See Disk Space Management
• Memory > 90% → See Memory Pressure
• Time offset > 100μs → See Time Sync Issues

Step 2: Service Startup (T-45min)

Start services in dependency order:

#!/bin/bash
# /opt/foxhunt/scripts/start-services.sh

echo "Starting Foxhunt Services..."

# Start API Gateway
echo "1. Starting API Gateway..."
sudo systemctl start api_gateway
sleep 5
if sudo systemctl is-active --quiet api_gateway; then
    echo "✓ API Gateway started"
else
    echo "✗ API Gateway failed to start"
    sudo journalctl -u api_gateway -n 50
    exit 1
fi

# Start Trading Service
echo "2. Starting Trading Service..."
sudo systemctl start trading_service
sleep 5
if sudo systemctl is-active --quiet trading_service; then
    echo "✓ Trading Service started"
else
    echo "✗ Trading Service failed to start"
    sudo journalctl -u trading_service -n 50
    exit 1
fi

# Start Backtesting Service
echo "3. Starting Backtesting Service..."
sudo systemctl start backtesting_service
sleep 5
if sudo systemctl is-active --quiet backtesting_service; then
    echo "✓ Backtesting Service started"
else
    echo "✗ Backtesting Service failed to start"
    sudo journalctl -u backtesting_service -n 50
    exit 1
fi

# Start ML Training Service
echo "4. Starting ML Training Service..."
sudo systemctl start ml_training_service
sleep 5
if sudo systemctl is-active --quiet ml_training_service; then
    echo "✓ ML Training Service started"
else
    echo "✗ ML Training Service failed to start"
    sudo journalctl -u ml_training_service -n 50
    exit 1
fi

echo ""
echo "✓ All services started successfully"

Step 3: Health Verification (T-35min)

#!/bin/bash
# /opt/foxhunt/scripts/verify-health.sh

echo "Verifying Service Health..."

# API Gateway
API_GATEWAY_HEALTH=$(curl -s http://localhost:8080/health)
if echo "$API_GATEWAY_HEALTH" | grep -q "ok"; then
    echo "✓ API Gateway health: OK"
else
    echo "✗ API Gateway health: FAILED"
    echo "   Response: $API_GATEWAY_HEALTH"
    exit 1
fi

# Trading Service
TRADING_SERVICE_HEALTH=$(curl -s http://localhost:8081/health)
if echo "$TRADING_SERVICE_HEALTH" | grep -q "ok"; then
    echo "✓ Trading Service health: OK"
else
    echo "✗ Trading Service health: FAILED"
    echo "   Response: $TRADING_SERVICE_HEALTH"
    exit 1
fi

# Backtesting Service
BACKTESTING_SERVICE_HEALTH=$(curl -s http://localhost:8082/health)
if echo "$BACKTESTING_SERVICE_HEALTH" | grep -q "ok"; then
    echo "✓ Backtesting Service health: OK"
else
    echo "✗ Backtesting Service health: FAILED"
    echo "   Response: $BACKTESTING_SERVICE_HEALTH"
    exit 1
fi

# ML Training Service
ML_TRAINING_SERVICE_HEALTH=$(curl -s http://localhost:8083/health)
if echo "$ML_TRAINING_SERVICE_HEALTH" | grep -q "ok"; then
    echo "✓ ML Training Service health: OK"
else
    echo "✗ ML Training Service health: FAILED"
    echo "   Response: $ML_TRAINING_SERVICE_HEALTH"
    exit 1
fi

echo ""
echo "✓ All services healthy and ready for trading"

Step 4: Smoke Tests (T-30min)

#!/bin/bash
# /opt/foxhunt/scripts/smoke-tests.sh

echo "Running Smoke Tests..."

# Test JWT authentication
echo "1. Testing JWT authentication..."
TOKEN=$(curl -s -X POST http://localhost:50051/auth/login \
    -H "Content-Type: application/json" \
    -d '{"username":"test_user","password":"test_password"}' | jq -r '.token')

if [ "$TOKEN" != "null" ] && [ -n "$TOKEN" ]; then
    echo "✓ JWT authentication working"
else
    echo "✗ JWT authentication FAILED"
    exit 1
fi

# Test order submission (dry-run)
echo "2. Testing order submission..."
ORDER_RESPONSE=$(curl -s -X POST http://localhost:50051/trading/order \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "symbol": "AAPL",
        "side": "buy",
        "quantity": 100,
        "order_type": "limit",
        "limit_price": 150.0,
        "dry_run": true
    }')

if echo "$ORDER_RESPONSE" | grep -q "order_id"; then
    echo "✓ Order submission working"
else
    echo "✗ Order submission FAILED"
    echo "   Response: $ORDER_RESPONSE"
    exit 1
fi

# Test portfolio query
echo "3. Testing portfolio query..."
PORTFOLIO=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:50051/trading/portfolio)
if echo "$PORTFOLIO" | grep -q "positions"; then
    echo "✓ Portfolio query working"
else
    echo "✗ Portfolio query FAILED"
    exit 1
fi

echo ""
echo "✓ All smoke tests passed - system ready for market open"

Post-Market Shutdown Checklist

Execute 30 minutes after market close (e.g., 4:30 PM EST for 4:00 PM close)

#!/bin/bash
# /opt/foxhunt/scripts/post-market-shutdown.sh

echo "Post-Market Shutdown Procedure..."

# 1. Graceful shutdown of services
echo "1. Shutting down services..."
sudo systemctl stop api_gateway
sudo systemctl stop trading_service
sudo systemctl stop backtesting_service
sudo systemctl stop ml_training_service

# 2. Backup databases
echo "2. Backing up databases..."
sudo -u postgres pg_dump foxhunt_production > /backups/foxhunt_$(date +%Y%m%d).sql
gzip /backups/foxhunt_$(date +%Y%m%d).sql

# 3. Archive logs
echo "3. Archiving logs..."
sudo tar -czf /backups/logs_$(date +%Y%m%d).tar.gz /var/log/foxhunt/

# 4. Upload backups to S3
echo "4. Uploading backups to S3..."
aws s3 cp /backups/foxhunt_$(date +%Y%m%d).sql.gz s3://foxhunt-backups/postgres/
aws s3 cp /backups/logs_$(date +%Y%m%d).tar.gz s3://foxhunt-backups/logs/

echo "✓ Post-market shutdown complete"

---

Service Monitoring

Key Performance Indicators (KPIs)

Application Metrics

Metric	Normal Range	Warning Threshold	Critical Threshold
API Gateway Latency (p99)	<5ms	>10ms	>20ms
Trading Service Latency (p99)	<10ms	>20ms	>50ms
Order Execution Rate	1000-10000 orders/min	<500 orders/min	<100 orders/min
JWT Validation Time	<10μs	>50μs	>100μs
Database Query Time (p99)	<10ms	>50ms	>100ms
Redis Response Time (p99)	<1ms	>5ms	>10ms
Error Rate	<0.1%	>1%	>5%

Infrastructure Metrics

Metric	Normal Range	Warning Threshold	Critical Threshold
CPU Usage	30-70%	>80%	>95%
Memory Usage	40-70%	>85%	>95%
Disk I/O Wait	<5%	>20%	>40%
Network Latency	<1ms	>5ms	>10ms
PostgreSQL Connections	20-100	>150	>180
Redis Memory Usage	<8GB	>14GB	>15GB

Business Metrics

Metric	Normal Range	Alert Condition
Fill Rate	>95%	<90%
P&L (daily)	Variable	<-$100K
Rejected Orders	<1%	>5%
Kill Switch Activations	0	>0 (investigate immediately)
Failed Logins	<10/hour	>100/hour

Dashboard Links

Grafana Dashboards:

• System Overview: http://grafana.foxhunt.local/d/overview
• Trading Metrics: http://grafana.foxhunt.local/d/trading
• Infrastructure: http://grafana.foxhunt.local/d/infrastructure
• Security: http://grafana.foxhunt.local/d/security

Prometheus Alerts:

• http://prometheus.foxhunt.local/alerts

ELK/Kibana:

• http://kibana.foxhunt.local

Alerting Configuration

PagerDuty Integration:

# alertmanager.yml
receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'PAGERDUTY_SERVICE_KEY'
        severity: 'critical'

  - name: 'slack-warnings'
    slack_configs:
      - api_url: 'SLACK_WEBHOOK_URL'
        channel: '#foxhunt-alerts'
        title: 'Warning: {{ .GroupLabels.alertname }}'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'slack-warnings'

  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      repeat_interval: 5m

---

Configuration Management

PostgreSQL NOTIFY/LISTEN Hot-Reload

How It Works:

1. Configuration changes are stored in config table
2. PostgreSQL trigger sends NOTIFY config_change event
3. Services listening on config_change channel receive notification
4. Services reload configuration from database without restart

Verify Hot-Reload is Working:

# Terminal 1: Monitor service logs
sudo journalctl -u trading_service -f

# Terminal 2: Update configuration
psql $DATABASE_URL -c "UPDATE config SET value = '200' WHERE key = 'max_order_size';"

# Terminal 1 should show:
# INFO trading_service: Configuration updated: max_order_size = 200

Manual Configuration Reload:

# Trigger configuration reload via NOTIFY
psql $DATABASE_URL -c "NOTIFY config_change, 'manual_reload';"

Configuration Update Procedure

-- Update configuration (triggers hot-reload)
UPDATE config SET value = '500', updated_at = NOW()
WHERE key = 'rate_limit_rps';

-- Verify update
SELECT key, value, updated_at FROM config WHERE key = 'rate_limit_rps';

⚠️ WARNING: Some configuration changes require service restart:

• TLS certificate paths
• Database connection strings
• Redis URLs
• gRPC bind addresses

---

Performance Monitoring

Real-Time Performance Monitoring

Monitor Trading Service Latency:

# Query Prometheus for p99 latency
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(trading_service_request_duration_seconds_bucket[5m]))' | jq .

Monitor Database Performance:

-- Top 10 slowest queries
SELECT query, calls, mean_exec_time, max_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;

-- Active queries
SELECT pid, usename, query, state, query_start
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY query_start;

Monitor Redis Performance:

# Redis INFO
redis-cli -a $REDIS_PASSWORD INFO stats

# Monitor commands in real-time
redis-cli -a $REDIS_PASSWORD MONITOR

Performance Tuning

Database Connection Pool Adjustment:

# Check current connections
psql $DATABASE_URL -c "SELECT count(*), application_name FROM pg_stat_activity GROUP BY application_name;"

# If connections are maxed out, adjust in .env:
# DATABASE_MAX_CONNECTIONS=40
sudo systemctl restart trading_service

Redis Memory Optimization:

# Check Redis memory usage
redis-cli -a $REDIS_PASSWORD INFO memory

# If memory usage is high, increase maxmemory:
redis-cli -a $REDIS_PASSWORD CONFIG SET maxmemory 20gb

---

Log Management

Log Locations

Service	Log Path
API Gateway	/var/log/foxhunt/api_gateway.log
Trading Service	/var/log/foxhunt/trading_service.log
Backtesting Service	/var/log/foxhunt/backtesting_service.log
ML Training Service	/var/log/foxhunt/ml_training_service.log
PostgreSQL	/var/log/postgresql/postgresql-*.log
Redis	/var/log/redis/redis-server.log
System	/var/log/syslog

Log Analysis

Search for Errors:

# Last 100 errors from all services
sudo journalctl -p err -n 100

# Errors in last hour
sudo journalctl -p err --since "1 hour ago"

# Search for specific error
sudo journalctl -u trading_service | grep "SQL"

Common Log Patterns:

# Failed login attempts
sudo grep "Authentication failed" /var/log/foxhunt/api_gateway.log

# Order rejections
sudo grep "Order rejected" /var/log/foxhunt/trading_service.log

# Database connection issues
sudo grep "connection refused" /var/log/foxhunt/*.log

Log Rotation

# Configure logrotate
sudo tee /etc/logrotate.d/foxhunt <<EOF
/var/log/foxhunt/*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    create 640 foxhunt foxhunt
    postrotate
        sudo systemctl reload api_gateway trading_service backtesting_service ml_training_service
    endscript
}
EOF

---

Backup Verification

Daily Backup Verification

#!/bin/bash
# /opt/foxhunt/scripts/verify-backups.sh

echo "Verifying Daily Backups..."

# Check PostgreSQL backup
LATEST_PG_BACKUP=$(ls -t /backups/foxhunt_*.sql.gz | head -1)
if [ -f "$LATEST_PG_BACKUP" ]; then
    BACKUP_SIZE=$(du -h "$LATEST_PG_BACKUP" | awk '{print $1}')
    BACKUP_AGE=$(( ($(date +%s) - $(stat -c %Y "$LATEST_PG_BACKUP")) / 3600 ))

    if [ $BACKUP_AGE -lt 24 ]; then
        echo "✓ PostgreSQL backup: $LATEST_PG_BACKUP ($BACKUP_SIZE, ${BACKUP_AGE}h old)"
    else
        echo "✗ PostgreSQL backup is too old: ${BACKUP_AGE} hours"
        exit 1
    fi
else
    echo "✗ No PostgreSQL backup found"
    exit 1
fi

# Verify backup integrity
gunzip -t "$LATEST_PG_BACKUP"
if [ $? -eq 0 ]; then
    echo "✓ Backup integrity verified"
else
    echo "✗ Backup integrity check FAILED"
    exit 1
fi

# Check S3 backups
S3_BACKUP_COUNT=$(aws s3 ls s3://foxhunt-backups/postgres/ | wc -l)
if [ $S3_BACKUP_COUNT -gt 30 ]; then
    echo "✓ S3 backups: $S3_BACKUP_COUNT files"
else
    echo "⚠ WARNING: Only $S3_BACKUP_COUNT S3 backups found (expected >30)"
fi

echo "✓ Backup verification complete"

Test Restore Procedure (Monthly)

#!/bin/bash
# /opt/foxhunt/scripts/test-restore.sh

echo "Testing Backup Restore (STAGING DATABASE ONLY)..."

# Create test database
psql $DATABASE_URL -c "CREATE DATABASE foxhunt_restore_test;"

# Restore latest backup
LATEST_BACKUP=$(ls -t /backups/foxhunt_*.sql.gz | head -1)
gunzip -c "$LATEST_BACKUP" | psql postgresql://foxhunt_user:PASSWORD@localhost/foxhunt_restore_test

# Verify restored data
RESTORED_TABLE_COUNT=$(psql postgresql://foxhunt_user:PASSWORD@localhost/foxhunt_restore_test -t -c "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';")

if [ $RESTORED_TABLE_COUNT -gt 20 ]; then
    echo "✓ Restore successful: $RESTORED_TABLE_COUNT tables restored"
else
    echo "✗ Restore FAILED: Only $RESTORED_TABLE_COUNT tables"
    exit 1
fi

# Cleanup
psql $DATABASE_URL -c "DROP DATABASE foxhunt_restore_test;"

echo "✓ Restore test complete"

---

Emergency Procedures

P0: Complete System Outage

Symptoms: All services down, no trading possible

Response:

# 1. Emergency stop (T+0min)
sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

# 2. Check infrastructure (T+1min)
sudo systemctl status postgresql
sudo systemctl status redis-server
df -h
free -h

# 3. Check logs for root cause (T+2min)
sudo journalctl -p err --since "10 minutes ago"

# 4. Attempt restart (T+5min)
sudo /opt/foxhunt/scripts/start-services.sh

# 5. If restart fails, rollback (T+10min)
sudo /opt/foxhunt/scripts/emergency-rollback.sh

# 6. Notify stakeholders (T+15min)
# Send notification to #foxhunt-incidents Slack channel

Database Recovery

Scenario: PostgreSQL crashed or corrupted

# Check PostgreSQL status
sudo systemctl status postgresql

# View PostgreSQL logs
sudo tail -100 /var/log/postgresql/postgresql-*.log

# Attempt restart
sudo systemctl restart postgresql

# If restart fails, restore from backup
sudo -u postgres pg_restore -d foxhunt_production /backups/foxhunt_20251003.sql.gz

Cache Recovery

Scenario: Redis crashed or data corruption

# Check Redis status
sudo systemctl status redis-server

# Attempt restart
sudo systemctl restart redis-server

# If restart fails, restore from RDB snapshot
sudo systemctl stop redis-server
sudo cp /backups/redis_20251003.rdb /var/lib/redis/dump.rdb
sudo chown redis:redis /var/lib/redis/dump.rdb
sudo systemctl start redis-server

Disk Space Management

Scenario: Disk usage > 80%

# Identify large files
sudo du -h /var/log /tmp /home | sort -rh | head -20

# Rotate logs immediately
sudo logrotate -f /etc/logrotate.d/foxhunt

# Archive and compress old logs
sudo tar -czf /backups/old_logs_$(date +%Y%m%d).tar.gz /var/log/foxhunt/*.log.1
sudo rm /var/log/foxhunt/*.log.1

# Clean up old backups (keep last 30 days)
find /backups -name "*.sql.gz" -mtime +30 -delete

Memory Pressure

Scenario: Memory usage > 90%

# Identify memory hogs
ps aux --sort=-%mem | head -20

# Restart services one by one
sudo systemctl restart api_gateway
sleep 30
sudo systemctl restart trading_service
sleep 30
sudo systemctl restart backtesting_service
sleep 30
sudo systemctl restart ml_training_service

# If issue persists, reboot server (during maintenance window)
sudo reboot

Time Sync Issues

Scenario: Time offset > 100μs (critical for MiFID II compliance)

# Check current time offset
chronyc tracking

# Force time sync
sudo chronyc -a makestep

# Restart chrony
sudo systemctl restart chrony

# Verify sync
chronyc tracking

# If offset still high, check time servers
chronyc sources

---

Maintenance Windows

Planned Maintenance Procedure

Recommended Window: Saturday 10:00 PM - Sunday 2:00 AM EST

Pre-Maintenance (T-24h)

• Notify all stakeholders
• Backup all databases and configurations
• Test rollback procedures in staging
• Prepare maintenance scripts
• Update change control ticket

During Maintenance

# 1. Stop services (T+0min)
sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

# 2. Backup current state (T+5min)
sudo -u postgres pg_dump foxhunt_production > /backups/pre_maintenance_$(date +%Y%m%d).sql

# 3. Apply updates (T+10min)
# - OS patches: sudo apt update && sudo apt upgrade
# - Service updates: deploy new binaries
# - Database migrations: psql $DATABASE_URL -f migration.sql

# 4. Restart services (T+60min)
sudo /opt/foxhunt/scripts/start-services.sh

# 5. Verify health (T+70min)
sudo /opt/foxhunt/scripts/verify-health.sh

# 6. Run smoke tests (T+80min)
sudo /opt/foxhunt/scripts/smoke-tests.sh

Post-Maintenance (T+120min)

• Verify all services healthy
• Monitor for 30 minutes
• Update change control ticket
• Notify stakeholders of completion

---

Common Troubleshooting Scenarios

Scenario 1: High API Gateway Latency

Symptoms: p99 latency > 20ms

Diagnosis:

# Check API Gateway metrics
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(api_gateway_request_duration_seconds_bucket[5m]))'

# Check for rate limiting
sudo journalctl -u api_gateway | grep "rate limit"

# Check Redis performance (JWT revocation)
redis-cli -a $REDIS_PASSWORD INFO stats

Resolution:

1. If Redis is slow, increase Redis memory or restart
2. If rate limiting is aggressive, adjust RATE_LIMIT_RPS
3. If database queries are slow, check PostgreSQL indexes

Scenario 2: Order Submission Failures

Symptoms: Orders rejected with "validation failed"

Diagnosis:

# Check trading service logs
sudo journalctl -u trading_service | grep "Order rejected"

# Check database connection
psql $DATABASE_URL -c "SELECT count(*) FROM orders WHERE created_at > NOW() - INTERVAL '1 hour';"

Resolution:

1. Verify input validation rules
2. Check risk limits in database
3. Verify kill switch is not active

Scenario 3: JWT Authentication Failures

Symptoms: Users cannot log in, "invalid token" errors

Diagnosis:

# Check API Gateway logs
sudo journalctl -u api_gateway | grep "Authentication failed"

# Check Redis connectivity (JWT revocation)
redis-cli -a $REDIS_PASSWORD ping

# Verify JWT secret is configured
echo $JWT_SECRET_FILE
cat $JWT_SECRET_FILE

Resolution:

1. Verify Redis is running and accessible
2. Rotate JWT secret if compromised
3. Clear revocation cache: redis-cli -a $REDIS_PASSWORD FLUSHDB

---

End of Operational Runbook

This document should be reviewed monthly and updated after every major incident.