foxhunt/docs/OPERATIONAL_RUNBOOK_V2.md

# Foxhunt HFT Trading System - Operational Runbook

**Version**: 2.0
**Last Updated**: 2025-10-03
**Wave**: 71 - Production Operations
**Audience**: System Operators, SREs, DevOps Engineers, Trading Desk

---

## Quick Reference

### Emergency Contacts

| Role | Contact | Escalation |
|------|---------|------------|
| Trading Operations Lead | PagerDuty: trading-ops | Immediate for P0 |
| Database Administrator | PagerDuty: dba-oncall | 15 min for P1 |
| Security Team | security@foxhunt.local | Immediate for breaches |
| Infrastructure Team | PagerDuty: infra-oncall | 1 hour for P2 |
| On-Call Engineer | PagerDuty: general-oncall | As per severity |

### Critical Commands (Bookmark This)

```bash
# Emergency stop all services
sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

# Health check all services
curl -s http://localhost:8080/health  # API Gateway
curl -s http://localhost:8081/health  # Trading Service
curl -s http://localhost:8082/health  # Backtesting Service
curl -s http://localhost:8083/health  # ML Training Service

# View real-time logs
sudo journalctl -u api_gateway -f
sudo journalctl -u trading_service -f

# Emergency rollback
sudo cp /usr/local/bin/api_gateway.bak /usr/local/bin/api_gateway
sudo systemctl restart api_gateway

# Database connection check
psql $DATABASE_URL -c "SELECT version();"

# Redis connection check
redis-cli -a $REDIS_PASSWORD ping
```

---

## Table of Contents

1. [Daily Startup Procedures](#daily-startup-procedures)
2. [Service Monitoring](#service-monitoring)
3. [Configuration Management](#configuration-management)
4. [Performance Monitoring](#performance-monitoring)
5. [Log Management](#log-management)
6. [Backup Verification](#backup-verification)
7. [Emergency Procedures](#emergency-procedures)
8. [Maintenance Windows](#maintenance-windows)
9. [Common Troubleshooting Scenarios](#common-troubleshooting-scenarios)

---

## Daily Startup Procedures

### Pre-Market Startup Checklist

**Execute 60 minutes before market open (e.g., 8:30 AM EST for 9:30 AM market open)**

#### Step 1: Infrastructure Health Check (T-60min)

```bash
#!/bin/bash
# /opt/foxhunt/scripts/daily-startup-check.sh

echo "════════════════════════════════════════════════════════════"
echo "  Foxhunt HFT Trading System - Daily Startup Check"
echo "  $(date)"
echo "════════════════════════════════════════════════════════════"

# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# 1. PostgreSQL Health
echo -e "\n1️⃣  Checking PostgreSQL..."
if sudo systemctl is-active --quiet postgresql; then
    echo -e "${GREEN}✓${NC} PostgreSQL service is running"
    if psql $DATABASE_URL -c "SELECT version();" > /dev/null 2>&1; then
        echo -e "${GREEN}✓${NC} Database connection successful"
    else
        echo -e "${RED}✗${NC} Database connection FAILED"
        exit 1
    fi
else
    echo -e "${RED}✗${NC} PostgreSQL service is NOT running"
    exit 1
fi

# 2. Redis Health
echo -e "\n2️⃣  Checking Redis..."
if sudo systemctl is-active --quiet redis-server; then
    echo -e "${GREEN}✓${NC} Redis service is running"
    if redis-cli -a $REDIS_PASSWORD ping | grep -q PONG; then
        echo -e "${GREEN}✓${NC} Redis connection successful"
    else
        echo -e "${RED}✗${NC} Redis connection FAILED"
        exit 1
    fi
else
    echo -e "${RED}✗${NC} Redis service is NOT running"
    exit 1
fi

# 3. Disk Space Check
echo -e "\n3️⃣  Checking Disk Space..."
DISK_USAGE=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -lt 80 ]; then
    echo -e "${GREEN}✓${NC} Disk usage: ${DISK_USAGE}%"
else
    echo -e "${YELLOW}⚠${NC}  WARNING: Disk usage above 80%: ${DISK_USAGE}%"
fi

# 4. Memory Check
echo -e "\n4️⃣  Checking Memory..."
MEMORY_USAGE=$(free | awk 'NR==2 {printf "%.0f", $3/$2 * 100}')
if [ $MEMORY_USAGE -lt 90 ]; then
    echo -e "${GREEN}✓${NC} Memory usage: ${MEMORY_USAGE}%"
else
    echo -e "${YELLOW}⚠${NC}  WARNING: Memory usage above 90%: ${MEMORY_USAGE}%"
fi

# 5. Network Connectivity
echo -e "\n5️⃣  Checking Network..."
if ping -c 3 8.8.8.8 > /dev/null 2>&1; then
    echo -e "${GREEN}✓${NC} Internet connectivity OK"
else
    echo -e "${RED}✗${NC} Internet connectivity FAILED"
    exit 1
fi

# 6. Time Synchronization (CRITICAL for HFT)
echo -e "\n6️⃣  Checking Time Synchronization..."
TIME_OFFSET=$(chronyc tracking | grep "System time" | awk '{print $4}')
if (( $(echo "$TIME_OFFSET < 0.0001" | bc -l) )); then
    echo -e "${GREEN}✓${NC} Time offset: ${TIME_OFFSET} seconds (< 100μs)"
else
    echo -e "${YELLOW}⚠${NC}  WARNING: Time offset: ${TIME_OFFSET} seconds (> 100μs)"
fi

# 7. Certificate Expiry Check
echo -e "\n7️⃣  Checking TLS Certificates..."
CERT_EXPIRY=$(openssl x509 -in /etc/foxhunt/certs/ca.crt -noout -enddate | cut -d= -f2)
CERT_EXPIRY_EPOCH=$(date -d "$CERT_EXPIRY" +%s)
CURRENT_EPOCH=$(date +%s)
DAYS_UNTIL_EXPIRY=$(( ($CERT_EXPIRY_EPOCH - $CURRENT_EPOCH) / 86400 ))

if [ $DAYS_UNTIL_EXPIRY -gt 30 ]; then
    echo -e "${GREEN}✓${NC} CA certificate expires in $DAYS_UNTIL_EXPIRY days"
else
    echo -e "${YELLOW}⚠${NC}  WARNING: CA certificate expires in $DAYS_UNTIL_EXPIRY days"
fi

echo -e "\n════════════════════════════════════════════════════════════"
echo -e "${GREEN}✓${NC} Infrastructure health check PASSED"
echo "════════════════════════════════════════════════════════════"
```

**Failure Response**:
- PostgreSQL down → See [Database Recovery](#database-recovery)
- Redis down → See [Cache Recovery](#cache-recovery)
- Disk > 80% → See [Disk Space Management](#disk-space-management)
- Memory > 90% → See [Memory Pressure](#memory-pressure)
- Time offset > 100μs → See [Time Sync Issues](#time-sync-issues)

#### Step 2: Service Startup (T-45min)

**Start services in dependency order**:

```bash
#!/bin/bash
# /opt/foxhunt/scripts/start-services.sh

echo "Starting Foxhunt Services..."

# Start API Gateway
echo "1. Starting API Gateway..."
sudo systemctl start api_gateway
sleep 5
if sudo systemctl is-active --quiet api_gateway; then
    echo "✓ API Gateway started"
else
    echo "✗ API Gateway failed to start"
    sudo journalctl -u api_gateway -n 50
    exit 1
fi

# Start Trading Service
echo "2. Starting Trading Service..."
sudo systemctl start trading_service
sleep 5
if sudo systemctl is-active --quiet trading_service; then
    echo "✓ Trading Service started"
else
    echo "✗ Trading Service failed to start"
    sudo journalctl -u trading_service -n 50
    exit 1
fi

# Start Backtesting Service
echo "3. Starting Backtesting Service..."
sudo systemctl start backtesting_service
sleep 5
if sudo systemctl is-active --quiet backtesting_service; then
    echo "✓ Backtesting Service started"
else
    echo "✗ Backtesting Service failed to start"
    sudo journalctl -u backtesting_service -n 50
    exit 1
fi

# Start ML Training Service
echo "4. Starting ML Training Service..."
sudo systemctl start ml_training_service
sleep 5
if sudo systemctl is-active --quiet ml_training_service; then
    echo "✓ ML Training Service started"
else
    echo "✗ ML Training Service failed to start"
    sudo journalctl -u ml_training_service -n 50
    exit 1
fi

echo ""
echo "✓ All services started successfully"
```

#### Step 3: Health Verification (T-35min)

```bash
#!/bin/bash
# /opt/foxhunt/scripts/verify-health.sh

echo "Verifying Service Health..."

# API Gateway
API_GATEWAY_HEALTH=$(curl -s http://localhost:8080/health)
if echo "$API_GATEWAY_HEALTH" | grep -q "ok"; then
    echo "✓ API Gateway health: OK"
else
    echo "✗ API Gateway health: FAILED"
    echo "   Response: $API_GATEWAY_HEALTH"
    exit 1
fi

# Trading Service
TRADING_SERVICE_HEALTH=$(curl -s http://localhost:8081/health)
if echo "$TRADING_SERVICE_HEALTH" | grep -q "ok"; then
    echo "✓ Trading Service health: OK"
else
    echo "✗ Trading Service health: FAILED"
    echo "   Response: $TRADING_SERVICE_HEALTH"
    exit 1
fi

# Backtesting Service
BACKTESTING_SERVICE_HEALTH=$(curl -s http://localhost:8082/health)
if echo "$BACKTESTING_SERVICE_HEALTH" | grep -q "ok"; then
    echo "✓ Backtesting Service health: OK"
else
    echo "✗ Backtesting Service health: FAILED"
    echo "   Response: $BACKTESTING_SERVICE_HEALTH"
    exit 1
fi

# ML Training Service
ML_TRAINING_SERVICE_HEALTH=$(curl -s http://localhost:8083/health)
if echo "$ML_TRAINING_SERVICE_HEALTH" | grep -q "ok"; then
    echo "✓ ML Training Service health: OK"
else
    echo "✗ ML Training Service health: FAILED"
    echo "   Response: $ML_TRAINING_SERVICE_HEALTH"
    exit 1
fi

echo ""
echo "✓ All services healthy and ready for trading"
```

#### Step 4: Smoke Tests (T-30min)

```bash
#!/bin/bash
# /opt/foxhunt/scripts/smoke-tests.sh

echo "Running Smoke Tests..."

# Test JWT authentication
echo "1. Testing JWT authentication..."
TOKEN=$(curl -s -X POST http://localhost:50051/auth/login \
    -H "Content-Type: application/json" \
    -d '{"username":"test_user","password":"test_password"}' | jq -r '.token')

if [ "$TOKEN" != "null" ] && [ -n "$TOKEN" ]; then
    echo "✓ JWT authentication working"
else
    echo "✗ JWT authentication FAILED"
    exit 1
fi

# Test order submission (dry-run)
echo "2. Testing order submission..."
ORDER_RESPONSE=$(curl -s -X POST http://localhost:50051/trading/order \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "symbol": "AAPL",
        "side": "buy",
        "quantity": 100,
        "order_type": "limit",
        "limit_price": 150.0,
        "dry_run": true
    }')

if echo "$ORDER_RESPONSE" | grep -q "order_id"; then
    echo "✓ Order submission working"
else
    echo "✗ Order submission FAILED"
    echo "   Response: $ORDER_RESPONSE"
    exit 1
fi

# Test portfolio query
echo "3. Testing portfolio query..."
PORTFOLIO=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:50051/trading/portfolio)
if echo "$PORTFOLIO" | grep -q "positions"; then
    echo "✓ Portfolio query working"
else
    echo "✗ Portfolio query FAILED"
    exit 1
fi

echo ""
echo "✓ All smoke tests passed - system ready for market open"
```

### Post-Market Shutdown Checklist

**Execute 30 minutes after market close (e.g., 4:30 PM EST for 4:00 PM close)**

```bash
#!/bin/bash
# /opt/foxhunt/scripts/post-market-shutdown.sh

echo "Post-Market Shutdown Procedure..."

# 1. Graceful shutdown of services
echo "1. Shutting down services..."
sudo systemctl stop api_gateway
sudo systemctl stop trading_service
sudo systemctl stop backtesting_service
sudo systemctl stop ml_training_service

# 2. Backup databases
echo "2. Backing up databases..."
sudo -u postgres pg_dump foxhunt_production > /backups/foxhunt_$(date +%Y%m%d).sql
gzip /backups/foxhunt_$(date +%Y%m%d).sql

# 3. Archive logs
echo "3. Archiving logs..."
sudo tar -czf /backups/logs_$(date +%Y%m%d).tar.gz /var/log/foxhunt/

# 4. Upload backups to S3
echo "4. Uploading backups to S3..."
aws s3 cp /backups/foxhunt_$(date +%Y%m%d).sql.gz s3://foxhunt-backups/postgres/
aws s3 cp /backups/logs_$(date +%Y%m%d).tar.gz s3://foxhunt-backups/logs/

echo "✓ Post-market shutdown complete"
```

---

## Service Monitoring

### Key Performance Indicators (KPIs)

#### Application Metrics

| Metric | Normal Range | Warning Threshold | Critical Threshold |
|--------|--------------|-------------------|-------------------|
| API Gateway Latency (p99) | <5ms | >10ms | >20ms |
| Trading Service Latency (p99) | <10ms | >20ms | >50ms |
| Order Execution Rate | 1000-10000 orders/min | <500 orders/min | <100 orders/min |
| JWT Validation Time | <10μs | >50μs | >100μs |
| Database Query Time (p99) | <10ms | >50ms | >100ms |
| Redis Response Time (p99) | <1ms | >5ms | >10ms |
| Error Rate | <0.1% | >1% | >5% |

#### Infrastructure Metrics

| Metric | Normal Range | Warning Threshold | Critical Threshold |
|--------|--------------|-------------------|-------------------|
| CPU Usage | 30-70% | >80% | >95% |
| Memory Usage | 40-70% | >85% | >95% |
| Disk I/O Wait | <5% | >20% | >40% |
| Network Latency | <1ms | >5ms | >10ms |
| PostgreSQL Connections | 20-100 | >150 | >180 |
| Redis Memory Usage | <8GB | >14GB | >15GB |

#### Business Metrics

| Metric | Normal Range | Alert Condition |
|--------|--------------|-----------------|
| Fill Rate | >95% | <90% |
| P&L (daily) | Variable | <-$100K |
| Rejected Orders | <1% | >5% |
| Kill Switch Activations | 0 | >0 (investigate immediately) |
| Failed Logins | <10/hour | >100/hour |

### Dashboard Links

**Grafana Dashboards**:
- System Overview: http://grafana.foxhunt.local/d/overview
- Trading Metrics: http://grafana.foxhunt.local/d/trading
- Infrastructure: http://grafana.foxhunt.local/d/infrastructure
- Security: http://grafana.foxhunt.local/d/security

**Prometheus Alerts**:
- http://prometheus.foxhunt.local/alerts

**ELK/Kibana**:
- http://kibana.foxhunt.local

### Alerting Configuration

**PagerDuty Integration**:

```yaml
# alertmanager.yml
receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'PAGERDUTY_SERVICE_KEY'
        severity: 'critical'

  - name: 'slack-warnings'
    slack_configs:
      - api_url: 'SLACK_WEBHOOK_URL'
        channel: '#foxhunt-alerts'
        title: 'Warning: {{ .GroupLabels.alertname }}'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'slack-warnings'

  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      repeat_interval: 5m
```

---

## Configuration Management

### PostgreSQL NOTIFY/LISTEN Hot-Reload

**How It Works**:

1. Configuration changes are stored in `config` table
2. PostgreSQL trigger sends `NOTIFY config_change` event
3. Services listening on `config_change` channel receive notification
4. Services reload configuration from database without restart

**Verify Hot-Reload is Working**:

```bash
# Terminal 1: Monitor service logs
sudo journalctl -u trading_service -f

# Terminal 2: Update configuration
psql $DATABASE_URL -c "UPDATE config SET value = '200' WHERE key = 'max_order_size';"

# Terminal 1 should show:
# INFO trading_service: Configuration updated: max_order_size = 200
```

**Manual Configuration Reload**:

```bash
# Trigger configuration reload via NOTIFY
psql $DATABASE_URL -c "NOTIFY config_change, 'manual_reload';"
```

### Configuration Update Procedure

```sql
-- Update configuration (triggers hot-reload)
UPDATE config SET value = '500', updated_at = NOW()
WHERE key = 'rate_limit_rps';

-- Verify update
SELECT key, value, updated_at FROM config WHERE key = 'rate_limit_rps';
```

**⚠️ WARNING**: Some configuration changes require service restart:
- TLS certificate paths
- Database connection strings
- Redis URLs
- gRPC bind addresses

---

## Performance Monitoring

### Real-Time Performance Monitoring

**Monitor Trading Service Latency**:

```bash
# Query Prometheus for p99 latency
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(trading_service_request_duration_seconds_bucket[5m]))' | jq .
```

**Monitor Database Performance**:

```sql
-- Top 10 slowest queries
SELECT query, calls, mean_exec_time, max_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;

-- Active queries
SELECT pid, usename, query, state, query_start
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY query_start;
```

**Monitor Redis Performance**:

```bash
# Redis INFO
redis-cli -a $REDIS_PASSWORD INFO stats

# Monitor commands in real-time
redis-cli -a $REDIS_PASSWORD MONITOR
```

### Performance Tuning

**Database Connection Pool Adjustment**:

```bash
# Check current connections
psql $DATABASE_URL -c "SELECT count(*), application_name FROM pg_stat_activity GROUP BY application_name;"

# If connections are maxed out, adjust in .env:
# DATABASE_MAX_CONNECTIONS=40
sudo systemctl restart trading_service
```

**Redis Memory Optimization**:

```bash
# Check Redis memory usage
redis-cli -a $REDIS_PASSWORD INFO memory

# If memory usage is high, increase maxmemory:
redis-cli -a $REDIS_PASSWORD CONFIG SET maxmemory 20gb
```

---

## Log Management

### Log Locations

| Service | Log Path |
|---------|----------|
| API Gateway | `/var/log/foxhunt/api_gateway.log` |
| Trading Service | `/var/log/foxhunt/trading_service.log` |
| Backtesting Service | `/var/log/foxhunt/backtesting_service.log` |
| ML Training Service | `/var/log/foxhunt/ml_training_service.log` |
| PostgreSQL | `/var/log/postgresql/postgresql-*.log` |
| Redis | `/var/log/redis/redis-server.log` |
| System | `/var/log/syslog` |

### Log Analysis

**Search for Errors**:

```bash
# Last 100 errors from all services
sudo journalctl -p err -n 100

# Errors in last hour
sudo journalctl -p err --since "1 hour ago"

# Search for specific error
sudo journalctl -u trading_service | grep "SQL"
```

**Common Log Patterns**:

```bash
# Failed login attempts
sudo grep "Authentication failed" /var/log/foxhunt/api_gateway.log

# Order rejections
sudo grep "Order rejected" /var/log/foxhunt/trading_service.log

# Database connection issues
sudo grep "connection refused" /var/log/foxhunt/*.log
```

### Log Rotation

```bash
# Configure logrotate
sudo tee /etc/logrotate.d/foxhunt <<EOF
/var/log/foxhunt/*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    create 640 foxhunt foxhunt
    postrotate
        sudo systemctl reload api_gateway trading_service backtesting_service ml_training_service
    endscript
}
EOF
```

---

## Backup Verification

### Daily Backup Verification

```bash
#!/bin/bash
# /opt/foxhunt/scripts/verify-backups.sh

echo "Verifying Daily Backups..."

# Check PostgreSQL backup
LATEST_PG_BACKUP=$(ls -t /backups/foxhunt_*.sql.gz | head -1)
if [ -f "$LATEST_PG_BACKUP" ]; then
    BACKUP_SIZE=$(du -h "$LATEST_PG_BACKUP" | awk '{print $1}')
    BACKUP_AGE=$(( ($(date +%s) - $(stat -c %Y "$LATEST_PG_BACKUP")) / 3600 ))

    if [ $BACKUP_AGE -lt 24 ]; then
        echo "✓ PostgreSQL backup: $LATEST_PG_BACKUP ($BACKUP_SIZE, ${BACKUP_AGE}h old)"
    else
        echo "✗ PostgreSQL backup is too old: ${BACKUP_AGE} hours"
        exit 1
    fi
else
    echo "✗ No PostgreSQL backup found"
    exit 1
fi

# Verify backup integrity
gunzip -t "$LATEST_PG_BACKUP"
if [ $? -eq 0 ]; then
    echo "✓ Backup integrity verified"
else
    echo "✗ Backup integrity check FAILED"
    exit 1
fi

# Check S3 backups
S3_BACKUP_COUNT=$(aws s3 ls s3://foxhunt-backups/postgres/ | wc -l)
if [ $S3_BACKUP_COUNT -gt 30 ]; then
    echo "✓ S3 backups: $S3_BACKUP_COUNT files"
else
    echo "⚠ WARNING: Only $S3_BACKUP_COUNT S3 backups found (expected >30)"
fi

echo "✓ Backup verification complete"
```

### Test Restore Procedure (Monthly)

```bash
#!/bin/bash
# /opt/foxhunt/scripts/test-restore.sh

echo "Testing Backup Restore (STAGING DATABASE ONLY)..."

# Create test database
psql $DATABASE_URL -c "CREATE DATABASE foxhunt_restore_test;"

# Restore latest backup
LATEST_BACKUP=$(ls -t /backups/foxhunt_*.sql.gz | head -1)
gunzip -c "$LATEST_BACKUP" | psql postgresql://foxhunt_user:PASSWORD@localhost/foxhunt_restore_test

# Verify restored data
RESTORED_TABLE_COUNT=$(psql postgresql://foxhunt_user:PASSWORD@localhost/foxhunt_restore_test -t -c "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';")

if [ $RESTORED_TABLE_COUNT -gt 20 ]; then
    echo "✓ Restore successful: $RESTORED_TABLE_COUNT tables restored"
else
    echo "✗ Restore FAILED: Only $RESTORED_TABLE_COUNT tables"
    exit 1
fi

# Cleanup
psql $DATABASE_URL -c "DROP DATABASE foxhunt_restore_test;"

echo "✓ Restore test complete"
```

---

## Emergency Procedures

### P0: Complete System Outage

**Symptoms**: All services down, no trading possible

**Response**:

```bash
# 1. Emergency stop (T+0min)
sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

# 2. Check infrastructure (T+1min)
sudo systemctl status postgresql
sudo systemctl status redis-server
df -h
free -h

# 3. Check logs for root cause (T+2min)
sudo journalctl -p err --since "10 minutes ago"

# 4. Attempt restart (T+5min)
sudo /opt/foxhunt/scripts/start-services.sh

# 5. If restart fails, rollback (T+10min)
sudo /opt/foxhunt/scripts/emergency-rollback.sh

# 6. Notify stakeholders (T+15min)
# Send notification to #foxhunt-incidents Slack channel
```

### Database Recovery

**Scenario**: PostgreSQL crashed or corrupted

```bash
# Check PostgreSQL status
sudo systemctl status postgresql

# View PostgreSQL logs
sudo tail -100 /var/log/postgresql/postgresql-*.log

# Attempt restart
sudo systemctl restart postgresql

# If restart fails, restore from backup
sudo -u postgres pg_restore -d foxhunt_production /backups/foxhunt_20251003.sql.gz
```

### Cache Recovery

**Scenario**: Redis crashed or data corruption

```bash
# Check Redis status
sudo systemctl status redis-server

# Attempt restart
sudo systemctl restart redis-server

# If restart fails, restore from RDB snapshot
sudo systemctl stop redis-server
sudo cp /backups/redis_20251003.rdb /var/lib/redis/dump.rdb
sudo chown redis:redis /var/lib/redis/dump.rdb
sudo systemctl start redis-server
```

### Disk Space Management

**Scenario**: Disk usage > 80%

```bash
# Identify large files
sudo du -h /var/log /tmp /home | sort -rh | head -20

# Rotate logs immediately
sudo logrotate -f /etc/logrotate.d/foxhunt

# Archive and compress old logs
sudo tar -czf /backups/old_logs_$(date +%Y%m%d).tar.gz /var/log/foxhunt/*.log.1
sudo rm /var/log/foxhunt/*.log.1

# Clean up old backups (keep last 30 days)
find /backups -name "*.sql.gz" -mtime +30 -delete
```

### Memory Pressure

**Scenario**: Memory usage > 90%

```bash
# Identify memory hogs
ps aux --sort=-%mem | head -20

# Restart services one by one
sudo systemctl restart api_gateway
sleep 30
sudo systemctl restart trading_service
sleep 30
sudo systemctl restart backtesting_service
sleep 30
sudo systemctl restart ml_training_service

# If issue persists, reboot server (during maintenance window)
sudo reboot
```

### Time Sync Issues

**Scenario**: Time offset > 100μs (critical for MiFID II compliance)

```bash
# Check current time offset
chronyc tracking

# Force time sync
sudo chronyc -a makestep

# Restart chrony
sudo systemctl restart chrony

# Verify sync
chronyc tracking

# If offset still high, check time servers
chronyc sources
```

---

## Maintenance Windows

### Planned Maintenance Procedure

**Recommended Window**: Saturday 10:00 PM - Sunday 2:00 AM EST

#### Pre-Maintenance (T-24h)

- [ ] Notify all stakeholders
- [ ] Backup all databases and configurations
- [ ] Test rollback procedures in staging
- [ ] Prepare maintenance scripts
- [ ] Update change control ticket

#### During Maintenance

```bash
# 1. Stop services (T+0min)
sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

# 2. Backup current state (T+5min)
sudo -u postgres pg_dump foxhunt_production > /backups/pre_maintenance_$(date +%Y%m%d).sql

# 3. Apply updates (T+10min)
# - OS patches: sudo apt update && sudo apt upgrade
# - Service updates: deploy new binaries
# - Database migrations: psql $DATABASE_URL -f migration.sql

# 4. Restart services (T+60min)
sudo /opt/foxhunt/scripts/start-services.sh

# 5. Verify health (T+70min)
sudo /opt/foxhunt/scripts/verify-health.sh

# 6. Run smoke tests (T+80min)
sudo /opt/foxhunt/scripts/smoke-tests.sh
```

#### Post-Maintenance (T+120min)

- [ ] Verify all services healthy
- [ ] Monitor for 30 minutes
- [ ] Update change control ticket
- [ ] Notify stakeholders of completion

---

## Common Troubleshooting Scenarios

### Scenario 1: High API Gateway Latency

**Symptoms**: p99 latency > 20ms

**Diagnosis**:

```bash
# Check API Gateway metrics
curl -s 'http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(api_gateway_request_duration_seconds_bucket[5m]))'

# Check for rate limiting
sudo journalctl -u api_gateway | grep "rate limit"

# Check Redis performance (JWT revocation)
redis-cli -a $REDIS_PASSWORD INFO stats
```

**Resolution**:

1. If Redis is slow, increase Redis memory or restart
2. If rate limiting is aggressive, adjust `RATE_LIMIT_RPS`
3. If database queries are slow, check PostgreSQL indexes

### Scenario 2: Order Submission Failures

**Symptoms**: Orders rejected with "validation failed"

**Diagnosis**:

```bash
# Check trading service logs
sudo journalctl -u trading_service | grep "Order rejected"

# Check database connection
psql $DATABASE_URL -c "SELECT count(*) FROM orders WHERE created_at > NOW() - INTERVAL '1 hour';"
```

**Resolution**:

1. Verify input validation rules
2. Check risk limits in database
3. Verify kill switch is not active

### Scenario 3: JWT Authentication Failures

**Symptoms**: Users cannot log in, "invalid token" errors

**Diagnosis**:

```bash
# Check API Gateway logs
sudo journalctl -u api_gateway | grep "Authentication failed"

# Check Redis connectivity (JWT revocation)
redis-cli -a $REDIS_PASSWORD ping

# Verify JWT secret is configured
echo $JWT_SECRET_FILE
cat $JWT_SECRET_FILE
```

**Resolution**:

1. Verify Redis is running and accessible
2. Rotate JWT secret if compromised
3. Clear revocation cache: `redis-cli -a $REDIS_PASSWORD FLUSHDB`

---

**End of Operational Runbook**

*This document should be reviewed monthly and updated after every major incident.*