foxhunt/docs/OPERATIONS_MANUAL.md

# Foxhunt HFT Trading System - Operations Manual

## Table of Contents

1. [Production Deployment](#production-deployment)
2. [System Startup & Shutdown](#system-startup--shutdown)
3. [Monitoring & Alerting](#monitoring--alerting)
4. [Performance Tuning](#performance-tuning)
5. [Troubleshooting](#troubleshooting)
6. [Backup & Recovery](#backup--recovery)
7. [Security Operations](#security-operations)
8. [Maintenance Procedures](#maintenance-procedures)
9. [Emergency Procedures](#emergency-procedures)
10. [Configuration Management](#configuration-management)

## Production Deployment

### Prerequisites

#### Hardware Requirements
```bash
# Trading Server Specifications
CPU: Intel Xeon Gold 6248R (24 cores, 3.0GHz) or AMD EPYC 7543 (32 cores, 2.8GHz)
Memory: 128GB DDR4-3200 ECC
Storage: 2TB NVMe SSD (Samsung 980 PRO or equivalent)
Network: 25Gbps Mellanox ConnectX-6 or Intel E810
OS: Ubuntu 22.04 LTS with real-time kernel

# Database Server Specifications  
CPU: Intel Xeon Gold 6258R (28 cores, 2.7GHz)
Memory: 256GB DDR4-3200 ECC
Storage: 4TB NVMe SSD for data, 1TB for logs
Network: 10Gbps for cluster communication
```

#### Software Dependencies
```bash
# System packages
sudo apt update && sudo apt install -y \
    build-essential \
    cmake \
    pkg-config \
    libssl-dev \
    libpq-dev \
    redis-server \
    postgresql-14 \
    influxdb \
    nginx \
    htop \
    iotop \
    perf \
    linux-tools-generic

# Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup default stable
rustup component add clippy rustfmt

# CUDA (optional, for GPU acceleration)
wget https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda_12.4.1_550.54.15_linux.run
sudo sh cuda_12.4.1_550.54.15_linux.run
```

### Environment Setup

#### 1. System Configuration
```bash
# Configure real-time kernel parameters
echo 'GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=2,3,4,5 rcu_nocbs=2,3,4,5 nohz_full=2,3,4,5"' | sudo tee -a /etc/default/grub
sudo update-grub

# Network optimization
echo 'net.core.rmem_max = 134217728' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 87380 134217728' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 134217728' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# CPU governor for consistent performance
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
```

#### 2. Database Setup
```bash
# PostgreSQL configuration
sudo -u postgres createdb foxhunt_production
sudo -u postgres createuser foxhunt_user
sudo -u postgres psql -c "ALTER USER foxhunt_user WITH ENCRYPTED PASSWORD 'secure_password';"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE foxhunt_production TO foxhunt_user;"

# InfluxDB setup
sudo systemctl start influxdb
sudo systemctl enable influxdb
influx setup --bucket foxhunt --org Foxhunt --retention 90d

# Redis configuration
sudo systemctl start redis-server
sudo systemctl enable redis-server
```

#### 3. Application Deployment
```bash
# Clone and build
git clone https://github.com/foxhunt-hft/foxhunt.git
cd foxhunt
git checkout production-hardening

# Build release version
export RUSTFLAGS="-C target-cpu=native -C target-feature=+avx2"
cargo build --release --features=simd,avx2,database-conversions

# Install systemd services
sudo cp deployment/systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload
```

#### 4. Configuration Files
```bash
# Production environment file
cp .env.example .env.production
vim .env.production
```

Example `.env.production`:
```env
# Environment
ENVIRONMENT=production
LOG_LEVEL=info
RUST_LOG=foxhunt=info,core=debug

# Database URLs
DATABASE_URL=postgresql://foxhunt_user:secure_password@localhost/foxhunt_production
INFLUXDB_URL=http://localhost:8086
REDIS_URL=redis://localhost:6379

# API Keys
POLYGON_API_KEY=your_polygon_api_key
ALPACA_API_KEY=your_alpaca_api_key
ALPACA_SECRET_KEY=your_alpaca_secret

# Broker Configuration
IB_HOST=localhost
IB_PORT=7497
IB_CLIENT_ID=1

# Performance Settings
MAX_LATENCY_US=50
ENABLE_SIMD=true
CPU_AFFINITY_CORES=2,3,4,5
MEMORY_POOL_SIZE_GB=8

# Risk Management
MAX_DAILY_LOSS=50000.00
MAX_POSITION_SIZE=1000000.00
VAR_CONFIDENCE_LEVEL=0.95

# Security
TLI_AUTH_SECRET=your_jwt_secret
TLS_CERT_PATH=/etc/foxhunt/tls/cert.pem
TLS_KEY_PATH=/etc/foxhunt/tls/key.pem
```

## System Startup & Shutdown

### Startup Sequence

#### 1. Infrastructure Services
```bash
# Start databases first
sudo systemctl start postgresql
sudo systemctl start influxdb  
sudo systemctl start redis-server

# Verify database connectivity
pg_isready -h localhost -p 5432
curl -f http://localhost:8086/ping
redis-cli ping
```

#### 2. Core Services
```bash
# Start in dependency order
sudo systemctl start foxhunt-core
sudo systemctl start foxhunt-data
sudo systemctl start foxhunt-risk
sudo systemctl start foxhunt-ml
sudo systemctl start foxhunt-tli

# Check service status
systemctl status foxhunt-*
```

#### 3. Health Verification
```bash
# System health check
curl -f http://localhost:8080/health

# TLI health check
grpcurl -plaintext localhost:50051 foxhunt.tli.HealthService/Check

# Performance verification
./target/release/foxhunt-bench --verify-latency
```

### Shutdown Sequence

#### 1. Graceful Application Shutdown
```bash
# Stop trading first to prevent new orders
sudo systemctl stop foxhunt-tli
sleep 10

# Stop core services
sudo systemctl stop foxhunt-ml
sudo systemctl stop foxhunt-risk
sudo systemctl stop foxhunt-data
sudo systemctl stop foxhunt-core
```

#### 2. Infrastructure Shutdown
```bash
# Stop databases last
sudo systemctl stop redis-server
sudo systemctl stop influxdb
sudo systemctl stop postgresql
```

### Emergency Shutdown
```bash
# Immediate halt of all trading
./scripts/emergency-halt.sh

# Force stop all services
sudo systemctl kill foxhunt-*
```

## Monitoring & Alerting

### Key Metrics to Monitor

#### Performance Metrics
- **Order Latency**: Target <50μs, Alert >100μs
- **Market Data Latency**: Target <5μs, Alert >20μs
- **CPU Usage**: Alert >80% on trading cores
- **Memory Usage**: Alert >90% of available
- **Network Latency**: Alert >1ms to exchanges

#### Trading Metrics
- **Orders per Second**: Monitor throughput
- **Fill Rate**: Track execution success
- **Slippage**: Monitor execution quality
- **PnL**: Real-time profit/loss tracking
- **Position Exposure**: Monitor risk limits

#### System Health
- **Service Uptime**: Alert on service failures
- **Database Connections**: Monitor pool utilization
- **Error Rates**: Alert on increased errors
- **Disk Usage**: Alert >85% full
- **Log Errors**: Monitor for critical errors

### Monitoring Setup

#### Prometheus Configuration
```yaml
# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'foxhunt'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 1s
    metrics_path: '/metrics'

  - job_name: 'system'
    static_configs:
      - targets: ['localhost:9100']
```

#### Grafana Dashboards
```bash
# Import pre-built dashboards
curl -X POST \
  http://admin:admin@localhost:3000/api/dashboards/db \
  -H 'Content-Type: application/json' \
  -d @monitoring/grafana/foxhunt-dashboard.json
```

#### Alert Rules
```yaml
# /etc/prometheus/alert_rules.yml
groups:
  - name: foxhunt.rules
    rules:
      - alert: HighLatency
        expr: foxhunt_order_latency_p99 > 100000  # 100μs
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "High order latency detected"
          
      - alert: ServiceDown
        expr: up{job="foxhunt"} == 0
        for: 10s
        labels:
          severity: critical
        annotations:
          summary: "Foxhunt service is down"
```

### Log Management

#### Log Locations
```bash
# Application logs
/var/log/foxhunt/core.log
/var/log/foxhunt/trading.log
/var/log/foxhunt/risk.log
/var/log/foxhunt/ml.log

# System logs
journalctl -u foxhunt-*
```

#### Log Rotation
```bash
# Configure logrotate
sudo tee /etc/logrotate.d/foxhunt << EOF
/var/log/foxhunt/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    create 644 foxhunt foxhunt
    postrotate
        systemctl reload foxhunt-*
    endscript
}
EOF
```

## Performance Tuning

### CPU Optimization

#### Core Isolation
```bash
# Isolate cores for trading threads
echo 2-5 | sudo tee /sys/devices/system/cpu/isolated

# Set CPU affinity for trading process
taskset -c 2,3 ./target/release/foxhunt-core &
TRADING_PID=$!

# Set real-time priority
sudo chrt -f -p 99 $TRADING_PID
```

#### CPU Governor
```bash
# Set performance governor
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Disable CPU idle states
sudo cpupower idle-set -D 0
```

### Memory Optimization

#### Huge Pages
```bash
# Configure huge pages
echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Mount hugetlbfs
sudo mount -t hugetlbfs none /mnt/huge
```

#### NUMA Awareness
```bash
# Check NUMA topology
numactl --hardware

# Bind process to NUMA node
numactl --cpunodebind=0 --membind=0 ./target/release/foxhunt-core
```

### Network Optimization

#### Interrupt Handling
```bash
# Bind network interrupts to specific CPUs
echo 1 | sudo tee /proc/irq/24/smp_affinity  # CPU 0
echo 2 | sudo tee /proc/irq/25/smp_affinity  # CPU 1
```

#### Network Buffer Tuning
```bash
# Increase network buffers
echo 'net.core.netdev_max_backlog = 5000' | sudo tee -a /etc/sysctl.conf
echo 'net.core.netdev_budget = 600' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
```

### Disk I/O Optimization

#### I/O Scheduler
```bash
# Set appropriate I/O scheduler for SSDs
echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler
```

#### Mount Options
```bash
# Optimize filesystem mount options
sudo mount -o remount,noatime,nodiratime /
```

## Troubleshooting

### Common Issues

#### High Latency
```bash
# Check system load
top -d 1
htop

# Check network latency
ping -c 10 exchange.hostname.com

# Check CPU frequency scaling
cat /proc/cpuinfo | grep MHz

# Check for context switches
sar -w 1 10
```

#### Memory Issues
```bash
# Check memory usage
free -h
cat /proc/meminfo

# Check for memory leaks
valgrind --tool=memcheck ./target/release/foxhunt-core

# Monitor memory allocation
pmap -x $(pgrep foxhunt-core)
```

#### Database Performance
```bash
# PostgreSQL performance
sudo -u postgres psql foxhunt_production -c "
SELECT query, calls, total_time, mean_time
FROM pg_stat_statements
ORDER BY total_time DESC LIMIT 10;"

# InfluxDB performance
influx query 'SHOW STATS'
```

#### Network Issues
```bash
# Check network statistics
ss -tuln
netstat -i
iftop

# Check dropped packets
cat /proc/net/dev

# Monitor network latency
mtr exchange.hostname.com
```

### Debugging Tools

#### System Profiling
```bash
# CPU profiling with perf
sudo perf record -g ./target/release/foxhunt-core
sudo perf report

# Memory profiling
heaptrack ./target/release/foxhunt-core
```

#### Application Debugging
```bash
# Enable debug logging
export RUST_LOG=debug
export FOXHUNT_LOG_LEVEL=trace

# Core dump analysis
ulimit -c unlimited
gdb ./target/release/foxhunt-core core
```

#### Network Debugging
```bash
# Packet capture
sudo tcpdump -i eth0 -w capture.pcap

# Network performance testing
iperf3 -c exchange.hostname.com
```

## Backup & Recovery

### Backup Strategy

#### Database Backups
```bash
# PostgreSQL backup
pg_dump -h localhost -U foxhunt_user foxhunt_production | gzip > backup_$(date +%Y%m%d_%H%M%S).sql.gz

# InfluxDB backup
influx backup /backup/influxdb/$(date +%Y%m%d_%H%M%S)

# Redis backup
redis-cli --rdb /backup/redis/dump_$(date +%Y%m%d_%H%M%S).rdb
```

#### Configuration Backups
```bash
# Backup configuration files
tar -czf config_backup_$(date +%Y%m%d_%H%M%S).tar.gz \
    .env.production \
    /etc/systemd/system/foxhunt-*.service \
    /etc/nginx/sites-available/foxhunt
```

#### Automated Backup Script
```bash
#!/bin/bash
# /usr/local/bin/foxhunt-backup.sh

BACKUP_DIR="/backup/foxhunt"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p "$BACKUP_DIR/$DATE"

# Database backups
pg_dump -h localhost -U foxhunt_user foxhunt_production | gzip > "$BACKUP_DIR/$DATE/postgres.sql.gz"
influx backup "$BACKUP_DIR/$DATE/influxdb"
redis-cli --rdb "$BACKUP_DIR/$DATE/redis.rdb"

# Configuration backup
tar -czf "$BACKUP_DIR/$DATE/config.tar.gz" .env.production /etc/systemd/system/foxhunt-*.service

# Cleanup old backups (keep 30 days)
find "$BACKUP_DIR" -type d -mtime +30 -exec rm -rf {} \;

# Upload to S3 (optional)
aws s3 sync "$BACKUP_DIR/$DATE" "s3://foxhunt-backups/$DATE"
```

### Recovery Procedures

#### Database Recovery
```bash
# PostgreSQL restore
sudo -u postgres createdb foxhunt_production_restore
gunzip -c backup_20240924_120000.sql.gz | sudo -u postgres psql foxhunt_production_restore

# InfluxDB restore
influx restore --bucket foxhunt /backup/influxdb/20240924_120000

# Redis restore
redis-cli --rdb dump_20240924_120000.rdb
```

#### Point-in-Time Recovery
```bash
# PostgreSQL PITR
sudo -u postgres pg_basebackup -D /var/lib/postgresql/14/main_backup -Ft -z -P
```

### Disaster Recovery

#### Recovery Time Objectives
- **Database Recovery**: <15 minutes
- **Application Recovery**: <5 minutes
- **Full System Recovery**: <30 minutes

#### Failover Procedures
```bash
# 1. Assess damage
systemctl status foxhunt-*
curl -f http://localhost:8080/health

# 2. Stop affected services
sudo systemctl stop foxhunt-*

# 3. Restore from backup
./scripts/restore-from-backup.sh latest

# 4. Restart services
sudo systemctl start foxhunt-*

# 5. Verify functionality
./scripts/verify-system-health.sh
```

## Security Operations

### Security Monitoring

#### Log Analysis
```bash
# Monitor authentication failures
grep "authentication failed" /var/log/foxhunt/*.log

# Check for suspicious API access
grep "401\|403" /var/log/nginx/access.log

# Monitor privilege escalation attempts
grep "sudo:" /var/log/auth.log
```

#### Network Security
```bash
# Monitor network connections
ss -tuln | grep :50051  # TLI gRPC port
ss -tuln | grep :8080   # Health check port

# Check firewall status
sudo ufw status verbose

# Monitor failed connections
grep "Connection refused" /var/log/syslog
```

### Certificate Management

#### TLS Certificate Renewal
```bash
# Check certificate expiry
openssl x509 -in /etc/foxhunt/tls/cert.pem -text -noout | grep "Not After"

# Renew certificates (if using Let's Encrypt)
sudo certbot renew --nginx

# Restart services after renewal
sudo systemctl reload nginx foxhunt-tli
```

### Access Control

#### User Management
```bash
# Add new user
sudo useradd -m -s /bin/bash -G foxhunt trader1
sudo passwd trader1

# Remove user access
sudo usermod -L trader1  # Lock account
sudo userdel trader1     # Delete account
```

#### API Key Rotation
```bash
# Generate new API keys
./scripts/generate-api-keys.sh

# Update configuration
vim .env.production

# Restart services
sudo systemctl restart foxhunt-*
```

## Maintenance Procedures

### Regular Maintenance

#### Daily Tasks
```bash
#!/bin/bash
# Daily maintenance script

# Check disk usage
df -h | grep -E '9[0-9]%' && echo "WARNING: High disk usage"

# Check log file sizes
find /var/log/foxhunt -name "*.log" -size +100M

# Verify backup completion
ls -la /backup/foxhunt/$(date +%Y%m%d)*

# Check system health
./scripts/health-check.sh
```

#### Weekly Tasks
```bash
#!/bin/bash
# Weekly maintenance script

# Update system packages (test environment first)
sudo apt list --upgradable

# Rotate logs manually if needed
sudo logrotate -f /etc/logrotate.d/foxhunt

# Check certificate expiry
./scripts/check-cert-expiry.sh

# Performance analysis
./scripts/performance-report.sh
```

#### Monthly Tasks
```bash
#!/bin/bash
# Monthly maintenance script

# Security updates
sudo apt update && sudo apt upgrade

# Database maintenance
sudo -u postgres vacuumdb --all --analyze --verbose

# Backup verification
./scripts/verify-backups.sh

# Performance tuning review
./scripts/performance-tuning-review.sh
```

### Software Updates

#### Update Procedure
```bash
# 1. Test in staging environment first
git checkout staging
cargo build --release
./scripts/run-integration-tests.sh

# 2. Schedule maintenance window
# 3. Create backup
./scripts/backup-system.sh

# 4. Update production
git checkout production-hardening
git pull origin production-hardening
cargo build --release

# 5. Deploy with zero downtime
./scripts/zero-downtime-deploy.sh

# 6. Verify deployment
./scripts/verify-deployment.sh
```

## Emergency Procedures

### Emergency Contacts

#### Internal Team
- **Lead Developer**: +1-555-0101 (24/7)
- **DevOps Engineer**: +1-555-0102 (24/7)
- **Risk Manager**: +1-555-0103 (Trading hours)
- **Compliance Officer**: +1-555-0104 (Business hours)

#### External Vendors
- **Polygon.io Support**: support@polygon.io
- **Interactive Brokers**: 877-442-2757
- **ICMarkets Support**: support@icmarkets.com

### Emergency Response

#### System Outage
```bash
# 1. Immediate assessment
./scripts/emergency-assessment.sh

# 2. Notify stakeholders
./scripts/send-alert.sh "CRITICAL: System outage detected"

# 3. Implement emergency procedures
./scripts/emergency-halt.sh     # Stop all trading
./scripts/emergency-recovery.sh # Begin recovery

# 4. Document incident
echo "$(date): System outage - investigating" >> /var/log/foxhunt/incidents.log
```

#### Security Incident
```bash
# 1. Isolate affected systems
sudo iptables -A INPUT -s suspicious_ip -j DROP

# 2. Preserve evidence
cp -r /var/log/foxhunt /backup/incident_$(date +%Y%m%d_%H%M%S)

# 3. Notify security team
./scripts/security-alert.sh "Security incident detected"

# 4. Begin forensic analysis
./scripts/forensic-analysis.sh
```

#### Trading Anomaly
```bash
# 1. Activate circuit breaker
curl -X POST http://localhost:8080/emergency/circuit-breaker

# 2. Halt all trading
curl -X POST http://localhost:8080/emergency/halt-trading

# 3. Assess positions
curl -X GET http://localhost:8080/positions/summary

# 4. Notify risk management
./scripts/risk-alert.sh "Trading anomaly detected"
```

## Configuration Management

### Environment Configuration

#### Configuration Files
```
config/
├── production.toml      # Production settings
├── staging.toml         # Staging settings
├── development.toml     # Development settings
└── local.toml          # Local development
```

#### Dynamic Configuration
```bash
# Update configuration without restart
curl -X POST http://localhost:8080/config/update \
  -H "Content-Type: application/json" \
  -d '{"max_position_size": 500000.00}'

# Verify configuration change
curl -X GET http://localhost:8080/config/current
```

### Version Control

#### Configuration Versioning
```bash
# Track configuration changes
git add config/production.toml
git commit -m "Update max position size to 500k"
git tag config-v1.2.3
```

#### Rollback Procedures
```bash
# Rollback configuration
git checkout config-v1.2.2 -- config/production.toml
sudo systemctl restart foxhunt-*

# Verify rollback
./scripts/verify-config.sh
```

This operations manual provides comprehensive procedures for managing the Foxhunt HFT system in production. Regular training and drill exercises should be conducted to ensure all operators are familiar with these procedures.