foxhunt/docs/OPERATOR_RUNBOOK.md

# Foxhunt HFT Trading System - Operator Runbook

**Version**: 1.0
**Last Updated**: 2025-10-03
**Wave**: 67 - Production Operations
**Audience**: System Operators, SREs, DevOps Engineers

---

## Quick Reference

### Emergency Contacts
- **Trading Operations Lead**: [Contact via internal escalation system]
- **Database Administrator**: [Contact via internal escalation system]
- **Security Team**: [Contact via internal escalation system]
- **On-Call Engineer**: Check PagerDuty rotation

### Critical Commands (Bookmark This)
```bash
# Emergency stop all services
/home/jgrusewski/Work/foxhunt/scripts/emergency-stop.sh

# Health check all services
/home/jgrusewski/Work/foxhunt/scripts/production-health-check.sh

# Rollback deployment
/home/jgrusewski/Work/foxhunt/scripts/emergency-rollback.sh

# View service logs
tail -f /var/log/foxhunt/trading_service.log
```

---

## Table of Contents

1. [Daily Startup Procedures](#daily-startup-procedures)
2. [Service Monitoring](#service-monitoring)
3. [Configuration Management](#configuration-management)
4. [Performance Monitoring](#performance-monitoring)
5. [Log Management](#log-management)
6. [Backup Verification](#backup-verification)
7. [Emergency Procedures](#emergency-procedures)
8. [Maintenance Windows](#maintenance-windows)

---

## Daily Startup Procedures

### Pre-Market Startup Checklist

**Execute 60 minutes before market open**

#### Step 1: Infrastructure Health Check (T-60min)

```bash
#!/bin/bash
# Daily startup checklist

echo "=== Foxhunt Daily Startup - $(date) ==="

# 1. Check PostgreSQL
echo "1. PostgreSQL Health..."
sudo systemctl status postgresql
psql $DATABASE_URL -c "SELECT version();" || echo "❌ PostgreSQL FAILED"

# 2. Check Redis
echo "2. Redis Health..."
redis-cli ping | grep PONG || echo "❌ Redis FAILED"

# 3. Check disk space (warn if < 20%)
echo "3. Disk Space..."
df -h | grep -E '(Filesystem|/home|/var)'
df -h / | awk 'NR==2 {if(int($5)>80) print "⚠️  WARNING: Disk usage above 80%"}'

# 4. Check memory
echo "4. Memory..."
free -h
free | awk 'NR==2 {if($3/$2 > 0.9) print "⚠️  WARNING: Memory usage above 90%"}'

# 5. Check network connectivity
echo "5. Network..."
ping -c 3 8.8.8.8 || echo "❌ Internet connectivity FAILED"

echo "=== Infrastructure Check Complete ==="
```

**Save as**: `/home/jgrusewski/Work/foxhunt/scripts/daily-startup-check.sh`

**Failure Response**:
- PostgreSQL down: See [Database Recovery](#database-recovery)
- Redis down: See [Cache Recovery](#cache-recovery)
- Disk > 80%: See [Disk Space Management](#disk-space-management)
- Memory > 90%: See [Memory Pressure](#memory-pressure)

#### Step 2: Service Startup (T-45min)

**Start services in deployment order**:

```bash
# 1. Trading Service (Core - Start First)
echo "Starting Trading Service..."
cd /home/jgrusewski/Work/foxhunt
./target/release/trading_service \
    --config /etc/foxhunt/trading_service.toml \
    --env-file .env.production \
    2>&1 | tee -a /var/log/foxhunt/trading_service.log &

TRADING_PID=$!
echo "Trading Service PID: $TRADING_PID"

# Wait for health
sleep 30
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check || {
    echo "❌ Trading Service failed to start"
    exit 1
}

# 2. Backtesting Service
echo "Starting Backtesting Service..."
./target/release/backtesting_service \
    --config /etc/foxhunt/backtesting_service.toml \
    --env-file .env.production \
    2>&1 | tee -a /var/log/foxhunt/backtesting_service.log &

BACKTESTING_PID=$!
echo "Backtesting Service PID: $BACKTESTING_PID"

sleep 30
grpcurl -plaintext localhost:50052 grpc.health.v1.Health/Check || {
    echo "❌ Backtesting Service failed to start"
    exit 1
}

# 3. ML Training Service
echo "Starting ML Training Service..."
./target/release/ml_training_service \
    --config /etc/foxhunt/ml_training_service.toml \
    --env-file .env.production \
    2>&1 | tee -a /var/log/foxhunt/ml_training_service.log &

ML_PID=$!
echo "ML Training Service PID: $ML_PID"

sleep 30
grpcurl -plaintext localhost:50053 grpc.health.v1.Health/Check || {
    echo "❌ ML Training Service failed to start"
    exit 1
}

echo "=== All Services Started Successfully ==="
echo "Trading Service: PID $TRADING_PID"
echo "Backtesting Service: PID $BACKTESTING_PID"
echo "ML Training Service: PID $ML_PID"

# Save PIDs for monitoring
echo $TRADING_PID > /var/run/foxhunt/trading.pid
echo $BACKTESTING_PID > /var/run/foxhunt/backtesting.pid
echo $ML_PID > /var/run/foxhunt/ml_training.pid
```

**Save as**: `/home/jgrusewski/Work/foxhunt/scripts/start-all-services.sh`

#### Step 3: Monitoring Verification (T-30min)

```bash
# Verify Prometheus is scraping
curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job, health}'

# Check for any scrape failures
curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up")'

# Verify Grafana dashboards loading
curl -s localhost:3000/api/health | jq '.database'
```

#### Step 4: Final Validation (T-15min)

```bash
# Run comprehensive health check
./scripts/production-health-check.sh

# Check service logs for errors
tail -100 /var/log/foxhunt/trading_service.log | grep -i error
tail -100 /var/log/foxhunt/backtesting_service.log | grep -i error
tail -100 /var/log/foxhunt/ml_training_service.log | grep -i error

# If no errors, system is ready
echo "✅ System ready for market open"
```

**Market Open Checklist**:
- [ ] All services healthy
- [ ] Prometheus scraping
- [ ] Grafana dashboards loading
- [ ] No errors in logs (last 100 lines)
- [ ] Database connections stable
- [ ] Redis cache operational
- [ ] Backup verified (last 24h)

---

## Service Monitoring

### Real-Time Monitoring Dashboard

**Primary Dashboard**: Grafana `http://localhost:3000/d/foxhunt-overview`

**Key Metrics to Watch**:

| Metric | Normal Range | Warning | Critical |
|--------|--------------|---------|----------|
| gRPC Request Rate | 100-1000 req/s | >2000 req/s | >5000 req/s |
| P99 Latency | <1ms | >5ms | >10ms |
| Database Connections | 10-50 | >80 | >95 |
| Memory Usage | 30-60% | >75% | >90% |
| CPU Usage | 20-50% | >70% | >85% |
| Error Rate | <0.1% | >1% | >5% |

### Service Health Monitoring

**Every 5 minutes during trading hours**:

```bash
# Quick health check loop
while true; do
    echo "=== Health Check $(date) ==="

    # Trading Service
    grpcurl -plaintext -max-time 5 localhost:50051 grpc.health.v1.Health/Check | \
        jq -r '.status' | grep -q "SERVING" && echo "✅ Trading" || echo "❌ Trading FAILED"

    # Backtesting Service
    grpcurl -plaintext -max-time 5 localhost:50052 grpc.health.v1.Health/Check | \
        jq -r '.status' | grep -q "SERVING" && echo "✅ Backtesting" || echo "❌ Backtesting FAILED"

    # ML Training Service
    grpcurl -plaintext -max-time 5 localhost:50053 grpc.health.v1.Health/Check | \
        jq -r '.status' | grep -q "SERVING" && echo "✅ ML Training" || echo "❌ ML Training FAILED"

    sleep 300  # 5 minutes
done
```

**Automated Monitoring**: Configure Prometheus alerts in `/etc/prometheus/alerts.yml`

### Process Monitoring

```bash
# Check if services are running
pgrep -a trading_service || echo "⚠️  Trading Service not running"
pgrep -a backtesting_service || echo "⚠️  Backtesting Service not running"
pgrep -a ml_training_service || echo "⚠️  ML Training Service not running"

# Check memory usage by service
ps aux | grep -E '(trading_service|backtesting_service|ml_training_service)' | \
    awk '{print $11, "Memory:", $4"%", "CPU:", $3"%"}'
```

---

## Configuration Management

### Wave 66 Configuration System

**Configuration Tiers** (from Wave 66 Agent 11):

1. **Compile-Time Constants** ✅ Implemented
   - Location: `/home/jgrusewski/Work/foxhunt/common/src/thresholds.rs`
   - 120+ centralized constants
   - Change requires recompilation

2. **Runtime Configuration** 📋 Designed (Wave 67)
   - Location: Environment variables
   - Change requires service restart
   - See `.env.production`

3. **Database Configuration** 📋 Designed (Wave 68)
   - Location: PostgreSQL
   - Hot-reload via NOTIFY/LISTEN
   - No service restart required

### Configuration Reload Procedure

**Current State** (Requires Restart):

```bash
# 1. Update configuration file
vim .env.production

# 2. Restart service (example: Trading Service)
# Get PID
TRADING_PID=$(cat /var/run/foxhunt/trading.pid)

# Graceful shutdown
kill -TERM $TRADING_PID

# Wait for clean shutdown (max 30s)
timeout 30 tail --pid=$TRADING_PID -f /dev/null

# Restart
./scripts/start-all-services.sh

# Verify
sleep 30
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
```

**Future State** (Hot-Reload - Wave 68):
- Database configuration changes trigger PostgreSQL NOTIFY
- Services receive LISTEN notification
- Configuration reloaded without restart
- Zero downtime configuration updates

### Configuration Verification

```bash
# Verify current configuration
curl localhost:9090/api/config | jq '.data.yaml' | grep -E '(database|redis|cache)'

# Check environment variables
sudo -u foxhunt_service printenv | grep -E '(DATABASE|REDIS|CACHE)'

# Verify Wave 66 constants in use
grep -r "thresholds::" /home/jgrusewski/Work/foxhunt/services/trading_service/src/ | \
    head -10
```

---

## Performance Monitoring

### Key Performance Indicators

**Real-Time Metrics** (Prometheus):

```bash
# Query current latency
curl -s 'localhost:9090/api/v1/query?query=histogram_quantile(0.99, trading_order_latency_seconds)' | \
    jq '.data.result[0].value[1]'

# Query request rate
curl -s 'localhost:9090/api/v1/query?query=rate(grpc_server_handled_total[1m])' | \
    jq '.data.result[].value[1]'

# Query error rate
curl -s 'localhost:9090/api/v1/query?query=rate(grpc_server_handled_total{grpc_code!="OK"}[1m])' | \
    jq '.data.result[].value[1]'
```

**Performance Baseline** (from Wave 66 Test Report):
- adaptive-strategy: 69 tests in 0.10s
- common: 68 tests in 0.00s
- trading_engine: 281 tests in 2.23s
- **Total**: 418 tests in 2.33s

**Production Performance** (To Be Measured):
- See `/home/jgrusewski/Work/foxhunt/docs/PERFORMANCE_BASELINES.md`
- Run benchmarks: `./scripts/run-performance-benchmarks.sh`
- Compare against baseline

### Performance Degradation Response

**If P99 latency > 10ms**:

1. Check database query performance:
```bash
# Slow query log
tail -100 /var/log/postgresql/postgresql-14-main.log | grep "duration"
```

2. Check memory pressure:
```bash
free -h
sudo dmesg | tail -50 | grep -i "out of memory"
```

3. Check CPU usage:
```bash
top -b -n 1 | head -20
mpstat -P ALL 1 5
```

4. Check network latency:
```bash
ping -c 10 db-primary
ping -c 10 redis-cluster
```

5. If no obvious cause, collect diagnostic data:
```bash
./scripts/collect-performance-diagnostics.sh
```

---

## Log Management

### Log Locations

```bash
# Service logs
/var/log/foxhunt/trading_service.log
/var/log/foxhunt/backtesting_service.log
/var/log/foxhunt/ml_training_service.log

# System logs
/var/log/syslog
/var/log/postgresql/postgresql-14-main.log
/var/log/redis/redis-server.log
```

### Log Rotation

**Automated Rotation** (logrotate):

```bash
# /etc/logrotate.d/foxhunt
/var/log/foxhunt/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 foxhunt_service foxhunt_service
    sharedscripts
    postrotate
        systemctl reload rsyslog > /dev/null 2>&1 || true
    endscript
}
```

**Manual Log Analysis**:

```bash
# Find errors in last hour
find /var/log/foxhunt/ -name "*.log" -mmin -60 -exec grep -H "ERROR" {} \;

# Count errors by type
grep "ERROR" /var/log/foxhunt/trading_service.log | \
    awk '{print $5}' | sort | uniq -c | sort -rn

# Tail all service logs
multitail /var/log/foxhunt/trading_service.log \
          /var/log/foxhunt/backtesting_service.log \
          /var/log/foxhunt/ml_training_service.log
```

### Log Archiving

**Daily Archive** (Run during maintenance window):

```bash
# Archive logs older than 7 days
find /var/log/foxhunt/ -name "*.log.*" -mtime +7 -exec gzip {} \;

# Move to long-term storage
find /var/log/foxhunt/ -name "*.log.*.gz" -mtime +30 -exec mv {} /archive/foxhunt/logs/ \;
```

---

## Backup Verification

### Daily Backup Checklist

**Every day at 02:00 AM** (automated):

```bash
#!/bin/bash
# /home/jgrusewski/Work/foxhunt/scripts/verify-backup.sh

echo "=== Backup Verification $(date) ==="

# 1. Check PostgreSQL backup
LATEST_BACKUP=$(ls -t /backup/postgresql/ | head -1)
if [ -z "$LATEST_BACKUP" ]; then
    echo "❌ No PostgreSQL backup found"
    exit 1
fi

BACKUP_AGE=$(stat -c %Y "/backup/postgresql/$LATEST_BACKUP")
CURRENT_TIME=$(date +%s)
AGE_HOURS=$(( ($CURRENT_TIME - $BACKUP_AGE) / 3600 ))

if [ $AGE_HOURS -gt 24 ]; then
    echo "⚠️  WARNING: Latest backup is $AGE_HOURS hours old"
else
    echo "✅ PostgreSQL backup: $LATEST_BACKUP (${AGE_HOURS}h old)"
fi

# 2. Verify backup integrity
pg_restore --list "/backup/postgresql/$LATEST_BACKUP" > /dev/null 2>&1
if [ $? -eq 0 ]; then
    echo "✅ Backup integrity verified"
else
    echo "❌ Backup integrity check FAILED"
    exit 1
fi

# 3. Check backup size
BACKUP_SIZE=$(du -sh "/backup/postgresql/$LATEST_BACKUP" | awk '{print $1}')
echo "   Backup size: $BACKUP_SIZE"

# 4. Verify configuration backup
if [ -f "/backup/config/.env.production.backup" ]; then
    echo "✅ Configuration backup exists"
else
    echo "⚠️  WARNING: No configuration backup"
fi

echo "=== Backup Verification Complete ==="
```

**Backup Restoration Test** (Monthly):

```bash
# Restore to test database
pg_restore -d foxhunt_test /backup/postgresql/latest.dump

# Verify row counts match
psql foxhunt_production -c "SELECT COUNT(*) FROM orders;" > /tmp/prod_count
psql foxhunt_test -c "SELECT COUNT(*) FROM orders;" > /tmp/test_count
diff /tmp/prod_count /tmp/test_count || echo "⚠️  WARNING: Row counts differ"
```

---

## Emergency Procedures

### Emergency Stop (Market Close / Incident)

```bash
#!/bin/bash
# /home/jgrusewski/Work/foxhunt/scripts/emergency-stop.sh

echo "=== EMERGENCY STOP INITIATED $(date) ==="
echo "Reason: $1"

# 1. Stop accepting new orders (graceful)
curl -X POST localhost:50051/admin/pause-trading

# 2. Wait for in-flight orders to complete (max 30s)
sleep 30

# 3. Stop services (graceful shutdown)
for PID_FILE in /var/run/foxhunt/*.pid; do
    if [ -f "$PID_FILE" ]; then
        PID=$(cat "$PID_FILE")
        echo "Stopping PID $PID..."
        kill -TERM $PID
    fi
done

# 4. Wait for clean shutdown
sleep 10

# 5. Force kill if still running
pkill -9 -f trading_service
pkill -9 -f backtesting_service
pkill -9 -f ml_training_service

# 6. Verify all stopped
pgrep -a foxhunt && echo "⚠️  WARNING: Processes still running" || echo "✅ All services stopped"

# 7. Create incident report
echo "EMERGENCY STOP: $(date)" >> /var/log/foxhunt/incidents.log
echo "Reason: $1" >> /var/log/foxhunt/incidents.log

echo "=== EMERGENCY STOP COMPLETE ==="
```

**Usage**:
```bash
./scripts/emergency-stop.sh "Market anomaly detected"
```

### Database Recovery

**If PostgreSQL is unresponsive**:

```bash
# 1. Check PostgreSQL status
sudo systemctl status postgresql

# 2. Check logs
sudo tail -100 /var/log/postgresql/postgresql-14-main.log

# 3. Restart PostgreSQL
sudo systemctl restart postgresql

# 4. Verify connections
psql $DATABASE_URL -c "SELECT 1;"

# 5. If restart fails, restore from backup
sudo -u postgres pg_restore -d foxhunt_production /backup/postgresql/latest.dump
```

### Cache Recovery

**If Redis is unresponsive**:

```bash
# 1. Check Redis status
redis-cli ping

# 2. Restart Redis
sudo systemctl restart redis

# 3. Verify cluster health (if using Redis Cluster)
redis-cli cluster info

# 4. If data corruption, flush and rebuild
redis-cli FLUSHALL  # ⚠️  CAUTION: Deletes all cached data
```

### Service Crash Recovery

**If service crashes during trading hours**:

```bash
# 1. Identify crashed service
pgrep -a trading_service || echo "Trading Service crashed"

# 2. Check crash logs
tail -200 /var/log/foxhunt/trading_service.log | grep -A 20 "FATAL\|panic"

# 3. Attempt automatic restart
./scripts/start-all-services.sh

# 4. If restart fails, investigate core dump
gdb target/release/trading_service /var/crash/core
```

---

## Maintenance Windows

### Weekly Maintenance (Sunday 02:00-04:00 AM)

**Pre-Maintenance Checklist**:
- [ ] Notify stakeholders 48h in advance
- [ ] Backup all databases
- [ ] Test rollback procedure
- [ ] Prepare maintenance scripts

**Maintenance Tasks**:

```bash
#!/bin/bash
# Weekly maintenance script

echo "=== Weekly Maintenance $(date) ==="

# 1. Database maintenance
echo "1. Database vacuum and analyze..."
psql $DATABASE_URL -c "VACUUM ANALYZE;"

# 2. Index rebuild (if needed)
echo "2. Checking index health..."
psql $DATABASE_URL -c "REINDEX DATABASE foxhunt_production;"

# 3. Log cleanup
echo "3. Cleaning old logs..."
find /var/log/foxhunt/ -name "*.log.*" -mtime +30 -delete

# 4. Temporary file cleanup
echo "4. Cleaning temp files..."
find /tmp/ -name "foxhunt-*" -mtime +7 -delete

# 5. Update dependencies (if applicable)
echo "5. Checking for security updates..."
cargo audit

# 6. Restart services (fresh start)
echo "6. Restarting all services..."
./scripts/emergency-stop.sh "Weekly maintenance"
sleep 10
./scripts/start-all-services.sh

echo "=== Maintenance Complete ==="
```

---

## Troubleshooting Quick Reference

**Common Issues**:

| Symptom | Likely Cause | Action |
|---------|--------------|--------|
| Service won't start | Port already in use | `lsof -i :50051` and kill process |
| High latency | Database slow queries | Check `pg_stat_statements` |
| Memory leak | Configuration issue | Review Wave 66 constants |
| gRPC errors | Network/firewall | Check `iptables`, `netstat` |
| Authentication failing | Wave 63 not enabled | Enable `.layer(auth_layer)` |

**Detailed Troubleshooting**: See `/home/jgrusewski/Work/foxhunt/docs/TROUBLESHOOTING_GUIDE.md`

---

## Appendix: Service Architecture

```
┌─────────────────────────────────────────────┐
│          FOXHUNT SERVICE ARCHITECTURE       │
├─────────────────────────────────────────────┤
│                                              │
│  Trading Service (Port 50051)                │
│  ├─ Order Management                         │
│  ├─ Risk Management                          │
│  ├─ Position Tracking                        │
│  └─ Event Streaming                          │
│                                              │
│  Backtesting Service (Port 50052)            │
│  ├─ Strategy Testing                         │
│  ├─ Historical Data                          │
│  └─ Performance Analysis                     │
│                                              │
│  ML Training Service (Port 50053)            │
│  ├─ Model Training                           │
│  ├─ Model Management                         │
│  └─ Inference Engine                         │
│                                              │
│  TLI Client (Terminal UI)                    │
│  └─ gRPC connections to all services         │
│                                              │
└──────────────────────────────────────────────┘
```

---

**Document Version**: 1.0
**Wave**: 67 Agent 10 - Operator Runbook
**Maintained By**: Foxhunt Operations Team
**Last Review**: 2025-10-03

**For Emergencies**: Execute `/home/jgrusewski/Work/foxhunt/scripts/emergency-stop.sh` immediately.