foxhunt/docs/OPERATOR_RUNBOOK.md

Foxhunt HFT Trading System - Operator Runbook

Version: 1.0 Last Updated: 2025-10-03 Wave: 67 - Production Operations Audience: System Operators, SREs, DevOps Engineers

---

Quick Reference

Emergency Contacts

• Trading Operations Lead: [Contact via internal escalation system]
• Database Administrator: [Contact via internal escalation system]
• Security Team: [Contact via internal escalation system]
• On-Call Engineer: Check PagerDuty rotation

Critical Commands (Bookmark This)

# Emergency stop all services
/home/jgrusewski/Work/foxhunt/scripts/emergency-stop.sh

# Health check all services
/home/jgrusewski/Work/foxhunt/scripts/production-health-check.sh

# Rollback deployment
/home/jgrusewski/Work/foxhunt/scripts/emergency-rollback.sh

# View service logs
tail -f /var/log/foxhunt/trading_service.log

---

Table of Contents

1. Daily Startup Procedures
2. Service Monitoring
3. Configuration Management
4. Performance Monitoring
5. Log Management
6. Backup Verification
7. Emergency Procedures
8. Maintenance Windows

---

Daily Startup Procedures

Pre-Market Startup Checklist

Execute 60 minutes before market open

Step 1: Infrastructure Health Check (T-60min)

#!/bin/bash
# Daily startup checklist

echo "=== Foxhunt Daily Startup - $(date) ==="

# 1. Check PostgreSQL
echo "1. PostgreSQL Health..."
sudo systemctl status postgresql
psql $DATABASE_URL -c "SELECT version();" || echo "❌ PostgreSQL FAILED"

# 2. Check Redis
echo "2. Redis Health..."
redis-cli ping | grep PONG || echo "❌ Redis FAILED"

# 3. Check disk space (warn if < 20%)
echo "3. Disk Space..."
df -h | grep -E '(Filesystem|/home|/var)'
df -h / | awk 'NR==2 {if(int($5)>80) print "⚠️  WARNING: Disk usage above 80%"}'

# 4. Check memory
echo "4. Memory..."
free -h
free | awk 'NR==2 {if($3/$2 > 0.9) print "⚠️  WARNING: Memory usage above 90%"}'

# 5. Check network connectivity
echo "5. Network..."
ping -c 3 8.8.8.8 || echo "❌ Internet connectivity FAILED"

echo "=== Infrastructure Check Complete ==="

Save as: /home/jgrusewski/Work/foxhunt/scripts/daily-startup-check.sh

Failure Response:

• PostgreSQL down: See Database Recovery
• Redis down: See Cache Recovery
• Disk > 80%: See Disk Space Management
• Memory > 90%: See Memory Pressure

Step 2: Service Startup (T-45min)

Start services in deployment order:

# 1. Trading Service (Core - Start First)
echo "Starting Trading Service..."
cd /home/jgrusewski/Work/foxhunt
./target/release/trading_service \
    --config /etc/foxhunt/trading_service.toml \
    --env-file .env.production \
    2>&1 | tee -a /var/log/foxhunt/trading_service.log &

TRADING_PID=$!
echo "Trading Service PID: $TRADING_PID"

# Wait for health
sleep 30
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check || {
    echo "❌ Trading Service failed to start"
    exit 1
}

# 2. Backtesting Service
echo "Starting Backtesting Service..."
./target/release/backtesting_service \
    --config /etc/foxhunt/backtesting_service.toml \
    --env-file .env.production \
    2>&1 | tee -a /var/log/foxhunt/backtesting_service.log &

BACKTESTING_PID=$!
echo "Backtesting Service PID: $BACKTESTING_PID"

sleep 30
grpcurl -plaintext localhost:50052 grpc.health.v1.Health/Check || {
    echo "❌ Backtesting Service failed to start"
    exit 1
}

# 3. ML Training Service
echo "Starting ML Training Service..."
./target/release/ml_training_service \
    --config /etc/foxhunt/ml_training_service.toml \
    --env-file .env.production \
    2>&1 | tee -a /var/log/foxhunt/ml_training_service.log &

ML_PID=$!
echo "ML Training Service PID: $ML_PID"

sleep 30
grpcurl -plaintext localhost:50053 grpc.health.v1.Health/Check || {
    echo "❌ ML Training Service failed to start"
    exit 1
}

echo "=== All Services Started Successfully ==="
echo "Trading Service: PID $TRADING_PID"
echo "Backtesting Service: PID $BACKTESTING_PID"
echo "ML Training Service: PID $ML_PID"

# Save PIDs for monitoring
echo $TRADING_PID > /var/run/foxhunt/trading.pid
echo $BACKTESTING_PID > /var/run/foxhunt/backtesting.pid
echo $ML_PID > /var/run/foxhunt/ml_training.pid

Save as: /home/jgrusewski/Work/foxhunt/scripts/start-all-services.sh

Step 3: Monitoring Verification (T-30min)

# Verify Prometheus is scraping
curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job, health}'

# Check for any scrape failures
curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up")'

# Verify Grafana dashboards loading
curl -s localhost:3000/api/health | jq '.database'

Step 4: Final Validation (T-15min)

# Run comprehensive health check
./scripts/production-health-check.sh

# Check service logs for errors
tail -100 /var/log/foxhunt/trading_service.log | grep -i error
tail -100 /var/log/foxhunt/backtesting_service.log | grep -i error
tail -100 /var/log/foxhunt/ml_training_service.log | grep -i error

# If no errors, system is ready
echo "✅ System ready for market open"

Market Open Checklist:

• All services healthy
• Prometheus scraping
• Grafana dashboards loading
• No errors in logs (last 100 lines)
• Database connections stable
• Redis cache operational
• Backup verified (last 24h)

---

Service Monitoring

Real-Time Monitoring Dashboard

Primary Dashboard: Grafana http://localhost:3000/d/foxhunt-overview

Key Metrics to Watch:

Metric	Normal Range	Warning	Critical
gRPC Request Rate	100-1000 req/s	>2000 req/s	>5000 req/s
P99 Latency	<1ms	>5ms	>10ms
Database Connections	10-50	>80	>95
Memory Usage	30-60%	>75%	>90%
CPU Usage	20-50%	>70%	>85%
Error Rate	<0.1%	>1%	>5%

Service Health Monitoring

Every 5 minutes during trading hours:

# Quick health check loop
while true; do
    echo "=== Health Check $(date) ==="

    # Trading Service
    grpcurl -plaintext -max-time 5 localhost:50051 grpc.health.v1.Health/Check | \
        jq -r '.status' | grep -q "SERVING" && echo "✅ Trading" || echo "❌ Trading FAILED"

    # Backtesting Service
    grpcurl -plaintext -max-time 5 localhost:50052 grpc.health.v1.Health/Check | \
        jq -r '.status' | grep -q "SERVING" && echo "✅ Backtesting" || echo "❌ Backtesting FAILED"

    # ML Training Service
    grpcurl -plaintext -max-time 5 localhost:50053 grpc.health.v1.Health/Check | \
        jq -r '.status' | grep -q "SERVING" && echo "✅ ML Training" || echo "❌ ML Training FAILED"

    sleep 300  # 5 minutes
done

Automated Monitoring: Configure Prometheus alerts in /etc/prometheus/alerts.yml

Process Monitoring

# Check if services are running
pgrep -a trading_service || echo "⚠️  Trading Service not running"
pgrep -a backtesting_service || echo "⚠️  Backtesting Service not running"
pgrep -a ml_training_service || echo "⚠️  ML Training Service not running"

# Check memory usage by service
ps aux | grep -E '(trading_service|backtesting_service|ml_training_service)' | \
    awk '{print $11, "Memory:", $4"%", "CPU:", $3"%"}'

---

Configuration Management

Wave 66 Configuration System

Configuration Tiers (from Wave 66 Agent 11):

1. Compile-Time Constants ✅ Implemented

   • Location: /home/jgrusewski/Work/foxhunt/common/src/thresholds.rs
   • 120+ centralized constants
   • Change requires recompilation

2. Runtime Configuration 📋 Designed (Wave 67)

   • Location: Environment variables
   • Change requires service restart
   • See .env.production

3. Database Configuration 📋 Designed (Wave 68)

   • Location: PostgreSQL
   • Hot-reload via NOTIFY/LISTEN
   • No service restart required

Configuration Reload Procedure

Current State (Requires Restart):

# 1. Update configuration file
vim .env.production

# 2. Restart service (example: Trading Service)
# Get PID
TRADING_PID=$(cat /var/run/foxhunt/trading.pid)

# Graceful shutdown
kill -TERM $TRADING_PID

# Wait for clean shutdown (max 30s)
timeout 30 tail --pid=$TRADING_PID -f /dev/null

# Restart
./scripts/start-all-services.sh

# Verify
sleep 30
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

Future State (Hot-Reload - Wave 68):

• Database configuration changes trigger PostgreSQL NOTIFY
• Services receive LISTEN notification
• Configuration reloaded without restart
• Zero downtime configuration updates

Configuration Verification

# Verify current configuration
curl localhost:9090/api/config | jq '.data.yaml' | grep -E '(database|redis|cache)'

# Check environment variables
sudo -u foxhunt_service printenv | grep -E '(DATABASE|REDIS|CACHE)'

# Verify Wave 66 constants in use
grep -r "thresholds::" /home/jgrusewski/Work/foxhunt/services/trading_service/src/ | \
    head -10

---

Performance Monitoring

Key Performance Indicators

Real-Time Metrics (Prometheus):

# Query current latency
curl -s 'localhost:9090/api/v1/query?query=histogram_quantile(0.99, trading_order_latency_seconds)' | \
    jq '.data.result[0].value[1]'

# Query request rate
curl -s 'localhost:9090/api/v1/query?query=rate(grpc_server_handled_total[1m])' | \
    jq '.data.result[].value[1]'

# Query error rate
curl -s 'localhost:9090/api/v1/query?query=rate(grpc_server_handled_total{grpc_code!="OK"}[1m])' | \
    jq '.data.result[].value[1]'

Performance Baseline (from Wave 66 Test Report):

• adaptive-strategy: 69 tests in 0.10s
• common: 68 tests in 0.00s
• trading_engine: 281 tests in 2.23s
• Total: 418 tests in 2.33s

Production Performance (To Be Measured):

• See /home/jgrusewski/Work/foxhunt/docs/PERFORMANCE_BASELINES.md
• Run benchmarks: ./scripts/run-performance-benchmarks.sh
• Compare against baseline

Performance Degradation Response

If P99 latency > 10ms:

1. Check database query performance:

# Slow query log
tail -100 /var/log/postgresql/postgresql-14-main.log | grep "duration"

2. Check memory pressure:

free -h
sudo dmesg | tail -50 | grep -i "out of memory"

3. Check CPU usage:

top -b -n 1 | head -20
mpstat -P ALL 1 5

4. Check network latency:

ping -c 10 db-primary
ping -c 10 redis-cluster

5. If no obvious cause, collect diagnostic data:

./scripts/collect-performance-diagnostics.sh

---

Log Management

Log Locations

# Service logs
/var/log/foxhunt/trading_service.log
/var/log/foxhunt/backtesting_service.log
/var/log/foxhunt/ml_training_service.log

# System logs
/var/log/syslog
/var/log/postgresql/postgresql-14-main.log
/var/log/redis/redis-server.log

Log Rotation

Automated Rotation (logrotate):

# /etc/logrotate.d/foxhunt
/var/log/foxhunt/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 foxhunt_service foxhunt_service
    sharedscripts
    postrotate
        systemctl reload rsyslog > /dev/null 2>&1 || true
    endscript
}

Manual Log Analysis:

# Find errors in last hour
find /var/log/foxhunt/ -name "*.log" -mmin -60 -exec grep -H "ERROR" {} \;

# Count errors by type
grep "ERROR" /var/log/foxhunt/trading_service.log | \
    awk '{print $5}' | sort | uniq -c | sort -rn

# Tail all service logs
multitail /var/log/foxhunt/trading_service.log \
          /var/log/foxhunt/backtesting_service.log \
          /var/log/foxhunt/ml_training_service.log

Log Archiving

Daily Archive (Run during maintenance window):

# Archive logs older than 7 days
find /var/log/foxhunt/ -name "*.log.*" -mtime +7 -exec gzip {} \;

# Move to long-term storage
find /var/log/foxhunt/ -name "*.log.*.gz" -mtime +30 -exec mv {} /archive/foxhunt/logs/ \;

---

Backup Verification

Daily Backup Checklist

Every day at 02:00 AM (automated):

#!/bin/bash
# /home/jgrusewski/Work/foxhunt/scripts/verify-backup.sh

echo "=== Backup Verification $(date) ==="

# 1. Check PostgreSQL backup
LATEST_BACKUP=$(ls -t /backup/postgresql/ | head -1)
if [ -z "$LATEST_BACKUP" ]; then
    echo "❌ No PostgreSQL backup found"
    exit 1
fi

BACKUP_AGE=$(stat -c %Y "/backup/postgresql/$LATEST_BACKUP")
CURRENT_TIME=$(date +%s)
AGE_HOURS=$(( ($CURRENT_TIME - $BACKUP_AGE) / 3600 ))

if [ $AGE_HOURS -gt 24 ]; then
    echo "⚠️  WARNING: Latest backup is $AGE_HOURS hours old"
else
    echo "✅ PostgreSQL backup: $LATEST_BACKUP (${AGE_HOURS}h old)"
fi

# 2. Verify backup integrity
pg_restore --list "/backup/postgresql/$LATEST_BACKUP" > /dev/null 2>&1
if [ $? -eq 0 ]; then
    echo "✅ Backup integrity verified"
else
    echo "❌ Backup integrity check FAILED"
    exit 1
fi

# 3. Check backup size
BACKUP_SIZE=$(du -sh "/backup/postgresql/$LATEST_BACKUP" | awk '{print $1}')
echo "   Backup size: $BACKUP_SIZE"

# 4. Verify configuration backup
if [ -f "/backup/config/.env.production.backup" ]; then
    echo "✅ Configuration backup exists"
else
    echo "⚠️  WARNING: No configuration backup"
fi

echo "=== Backup Verification Complete ==="

Backup Restoration Test (Monthly):

# Restore to test database
pg_restore -d foxhunt_test /backup/postgresql/latest.dump

# Verify row counts match
psql foxhunt_production -c "SELECT COUNT(*) FROM orders;" > /tmp/prod_count
psql foxhunt_test -c "SELECT COUNT(*) FROM orders;" > /tmp/test_count
diff /tmp/prod_count /tmp/test_count || echo "⚠️  WARNING: Row counts differ"

---

Emergency Procedures

Emergency Stop (Market Close / Incident)

#!/bin/bash
# /home/jgrusewski/Work/foxhunt/scripts/emergency-stop.sh

echo "=== EMERGENCY STOP INITIATED $(date) ==="
echo "Reason: $1"

# 1. Stop accepting new orders (graceful)
curl -X POST localhost:50051/admin/pause-trading

# 2. Wait for in-flight orders to complete (max 30s)
sleep 30

# 3. Stop services (graceful shutdown)
for PID_FILE in /var/run/foxhunt/*.pid; do
    if [ -f "$PID_FILE" ]; then
        PID=$(cat "$PID_FILE")
        echo "Stopping PID $PID..."
        kill -TERM $PID
    fi
done

# 4. Wait for clean shutdown
sleep 10

# 5. Force kill if still running
pkill -9 -f trading_service
pkill -9 -f backtesting_service
pkill -9 -f ml_training_service

# 6. Verify all stopped
pgrep -a foxhunt && echo "⚠️  WARNING: Processes still running" || echo "✅ All services stopped"

# 7. Create incident report
echo "EMERGENCY STOP: $(date)" >> /var/log/foxhunt/incidents.log
echo "Reason: $1" >> /var/log/foxhunt/incidents.log

echo "=== EMERGENCY STOP COMPLETE ==="

Usage:

./scripts/emergency-stop.sh "Market anomaly detected"

Database Recovery

If PostgreSQL is unresponsive:

# 1. Check PostgreSQL status
sudo systemctl status postgresql

# 2. Check logs
sudo tail -100 /var/log/postgresql/postgresql-14-main.log

# 3. Restart PostgreSQL
sudo systemctl restart postgresql

# 4. Verify connections
psql $DATABASE_URL -c "SELECT 1;"

# 5. If restart fails, restore from backup
sudo -u postgres pg_restore -d foxhunt_production /backup/postgresql/latest.dump

Cache Recovery

If Redis is unresponsive:

# 1. Check Redis status
redis-cli ping

# 2. Restart Redis
sudo systemctl restart redis

# 3. Verify cluster health (if using Redis Cluster)
redis-cli cluster info

# 4. If data corruption, flush and rebuild
redis-cli FLUSHALL  # ⚠️  CAUTION: Deletes all cached data

Service Crash Recovery

If service crashes during trading hours:

# 1. Identify crashed service
pgrep -a trading_service || echo "Trading Service crashed"

# 2. Check crash logs
tail -200 /var/log/foxhunt/trading_service.log | grep -A 20 "FATAL\|panic"

# 3. Attempt automatic restart
./scripts/start-all-services.sh

# 4. If restart fails, investigate core dump
gdb target/release/trading_service /var/crash/core

---

Maintenance Windows

Weekly Maintenance (Sunday 02:00-04:00 AM)

Pre-Maintenance Checklist:

• Notify stakeholders 48h in advance
• Backup all databases
• Test rollback procedure
• Prepare maintenance scripts

Maintenance Tasks:

#!/bin/bash
# Weekly maintenance script

echo "=== Weekly Maintenance $(date) ==="

# 1. Database maintenance
echo "1. Database vacuum and analyze..."
psql $DATABASE_URL -c "VACUUM ANALYZE;"

# 2. Index rebuild (if needed)
echo "2. Checking index health..."
psql $DATABASE_URL -c "REINDEX DATABASE foxhunt_production;"

# 3. Log cleanup
echo "3. Cleaning old logs..."
find /var/log/foxhunt/ -name "*.log.*" -mtime +30 -delete

# 4. Temporary file cleanup
echo "4. Cleaning temp files..."
find /tmp/ -name "foxhunt-*" -mtime +7 -delete

# 5. Update dependencies (if applicable)
echo "5. Checking for security updates..."
cargo audit

# 6. Restart services (fresh start)
echo "6. Restarting all services..."
./scripts/emergency-stop.sh "Weekly maintenance"
sleep 10
./scripts/start-all-services.sh

echo "=== Maintenance Complete ==="

---

Troubleshooting Quick Reference

Common Issues:

Symptom	Likely Cause	Action
Service won't start	Port already in use	lsof -i :50051 and kill process
High latency	Database slow queries	Check pg_stat_statements
Memory leak	Configuration issue	Review Wave 66 constants
gRPC errors	Network/firewall	Check iptables, netstat
Authentication failing	Wave 63 not enabled	Enable .layer(auth_layer)

Detailed Troubleshooting: See /home/jgrusewski/Work/foxhunt/docs/TROUBLESHOOTING_GUIDE.md

---

Appendix: Service Architecture

┌─────────────────────────────────────────────┐
│          FOXHUNT SERVICE ARCHITECTURE       │
├─────────────────────────────────────────────┤
│                                              │
│  Trading Service (Port 50051)                │
│  ├─ Order Management                         │
│  ├─ Risk Management                          │
│  ├─ Position Tracking                        │
│  └─ Event Streaming                          │
│                                              │
│  Backtesting Service (Port 50052)            │
│  ├─ Strategy Testing                         │
│  ├─ Historical Data                          │
│  └─ Performance Analysis                     │
│                                              │
│  ML Training Service (Port 50053)            │
│  ├─ Model Training                           │
│  ├─ Model Management                         │
│  └─ Inference Engine                         │
│                                              │
│  TLI Client (Terminal UI)                    │
│  └─ gRPC connections to all services         │
│                                              │
└──────────────────────────────────────────────┘

---

Document Version: 1.0 Wave: 67 Agent 10 - Operator Runbook Maintained By: Foxhunt Operations Team Last Review: 2025-10-03

For Emergencies: Execute /home/jgrusewski/Work/foxhunt/scripts/emergency-stop.sh immediately.