foxhunt/docs/COMPREHENSIVE_DEPLOYMENT_GUIDE.md

# Foxhunt HFT System - Comprehensive Deployment Guide

## Overview

This guide provides complete deployment instructions for the Foxhunt HFT trading system in production environments. The system is designed for ultra-low latency trading with sub-50μs execution times and enterprise-grade reliability.

**Version**: 1.0.0 Production  
**Last Updated**: 2025-09-24  
**Target Environment**: Production HFT Trading

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Infrastructure Requirements](#infrastructure-requirements)
3. [Security Setup](#security-setup)
4. [Database Configuration](#database-configuration)
5. [Service Deployment](#service-deployment)
6. [Performance Optimization](#performance-optimization)
7. [Monitoring & Alerting](#monitoring--alerting)
8. [Backup & Recovery](#backup--recovery)
9. [Troubleshooting](#troubleshooting)
10. [Maintenance Procedures](#maintenance-procedures)

## Prerequisites

### Hardware Requirements

**Minimum Production Configuration:**
```yaml
CPU: 
  - 2x Intel Xeon Gold 6248R (24 cores each, 3.0GHz base)
  - OR 2x AMD EPYC 7543 (32 cores each, 2.8GHz base)
  
Memory:
  - 256GB DDR4-3200 ECC (minimum)
  - 512GB DDR4-3200 ECC (recommended)
  
Storage:
  - 2x 2TB NVMe SSD (RAID 1 for OS/applications)
  - 4x 8TB NVMe SSD (RAID 10 for data)
  - Write latency < 100μs (99.9th percentile)
  
GPU (for ML Training):
  - 2x NVIDIA A100 80GB (minimum)
  - 4x NVIDIA H100 80GB (recommended)
  
Network:
  - 2x 25GbE network interfaces (redundant)
  - Direct market data feeds (dedicated lines)
  - Sub-1ms latency to exchange colocations
```

**Recommended Production Configuration:**
```yaml
CPU: 
  - 2x Intel Xeon Platinum 8380 (40 cores each, 2.3GHz base)
  - L3 Cache: 60MB per socket
  - Support for AVX-512
  
Memory:
  - 1TB DDR4-3200 ECC
  - 8-channel memory configuration
  
Storage:
  - 2x 4TB Intel Optane SSD (OS/applications)
  - 8x 15TB Samsung PM1743 NVMe (data storage)
  - Write latency < 50μs (99.9th percentile)
  
GPU:
  - 8x NVIDIA H100 80GB SXM
  - NVLink interconnect for multi-GPU training
  
Network:
  - 2x 100GbE InfiniBand interfaces
  - FPGA-based market data capture cards
  - Direct fiber connections to exchanges
```

### Software Prerequisites

```bash
# Operating System
Ubuntu 22.04 LTS Server (kernel 5.15+)
# OR
Red Hat Enterprise Linux 9.2

# System Dependencies
sudo apt update && sudo apt install -y \
    build-essential \
    cmake \
    pkg-config \
    openssl \
    libssl-dev \
    libpq-dev \
    protobuf-compiler \
    clang \
    llvm \
    libnuma-dev \
    hwloc \
    numactl

# Rust Toolchain (latest stable)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup toolchain install stable
rustup default stable
rustup component add clippy rustfmt

# NVIDIA Drivers & CUDA (for GPU acceleration)
wget https://developer.download.nvidia.com/compute/cuda/12.3.0/local_installers/cuda-repo-ubuntu2204-12-3-local_12.3.0-545.23.06-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-3-local_12.3.0-545.23.06-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-3-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt update
sudo apt install cuda-toolkit-12-3

# Docker & Docker Compose (for auxiliary services)
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
```

## Infrastructure Requirements

### Network Configuration

**Low-Latency Network Tuning:**
```bash
# Kernel network optimizations
echo 'net.core.rmem_max = 268435456' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 268435456' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 131072 268435456' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 268435456' >> /etc/sysctl.conf
echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_congestion_control = bbr' >> /etc/sysctl.conf
sysctl -p

# Network interface optimization
sudo ethtool -G eth0 rx 4096 tx 4096
sudo ethtool -K eth0 gro off gso off tso off
sudo ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
```

**CPU and Memory Optimization:**
```bash
# CPU frequency scaling
echo 'performance' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Disable CPU idle states
sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="[^"]*/& intel_idle.max_cstate=0 processor.max_cstate=1/' /etc/default/grub
sudo update-grub

# Huge pages configuration
echo 'vm.nr_hugepages = 8192' >> /etc/sysctl.conf
echo 'hugetlbfs /mnt/hugepages hugetlbfs mode=1770,gid=1000 0 0' >> /etc/fstab
sudo mkdir -p /mnt/hugepages
sudo mount -t hugetlbfs hugetlbfs /mnt/hugepages

# Memory optimization
echo 'vm.swappiness = 1' >> /etc/sysctl.conf
echo 'vm.dirty_ratio = 5' >> /etc/sysctl.conf
echo 'vm.dirty_background_ratio = 2' >> /etc/sysctl.conf
```

### Real-Time Kernel (Optional but Recommended)

```bash
# Install real-time kernel for ultra-low latency
sudo apt install linux-image-rt-amd64
sudo sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT="1>2"/' /etc/default/grub
sudo update-grub
# Reboot required
```

## Security Setup

### Certificate Management

```bash
# Generate production certificates
cd /opt/foxhunt/certs/production

# Generate CA private key
openssl genrsa -out ca-key.pem 4096

# Generate CA certificate
openssl req -new -x509 -days 3650 -key ca-key.pem -sha256 -out ca-cert.pem -subj \
    "/C=US/ST=NY/L=NYC/O=Foxhunt Trading/OU=HFT Infrastructure/CN=Foxhunt CA"

# Generate server private key
openssl genrsa -out foxhunt-key.pem 4096

# Generate server certificate signing request
openssl req -subj "/C=US/ST=NY/L=NYC/O=Foxhunt Trading/OU=HFT Infrastructure/CN=foxhunt.trading" \
    -sha256 -new -key foxhunt-key.pem -out foxhunt-csr.pem

# Generate server certificate
openssl x509 -req -days 365 -sha256 -in foxhunt-csr.pem -CA ca-cert.pem -CAkey ca-key.pem \
    -out foxhunt-cert.pem -CAcreateserial \
    -extensions v3_req -extfile <(cat <<EOF
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = foxhunt.trading
DNS.2 = localhost
IP.1 = 127.0.0.1
EOF
)

# Set appropriate permissions
chmod 400 ca-key.pem foxhunt-key.pem
chmod 444 ca-cert.pem foxhunt-cert.pem
```

### JWT Token Configuration

```bash
# Generate JWT signing key
openssl rand -base64 64 > /opt/foxhunt/certs/jwt-secret.key
chmod 400 /opt/foxhunt/certs/jwt-secret.key

# Generate encryption key for sensitive data
openssl rand -hex 32 > /opt/foxhunt/certs/encryption-key.key
chmod 400 /opt/foxhunt/certs/encryption-key.key
```

### Firewall Configuration

```bash
# Configure UFW firewall
sudo ufw --force reset
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH (secure port)
sudo ufw allow 2222/tcp

# Allow gRPC services (internal network only)
sudo ufw allow from 10.0.0.0/8 to any port 50051 proto tcp
sudo ufw allow from 172.16.0.0/12 to any port 50051 proto tcp
sudo ufw allow from 192.168.0.0/16 to any port 50051 proto tcp

# Allow monitoring ports (restricted)
sudo ufw allow from 10.0.0.0/8 to any port 9090 proto tcp  # Prometheus
sudo ufw allow from 10.0.0.0/8 to any port 3000 proto tcp  # Grafana

# Enable firewall
sudo ufw --force enable
```

## Database Configuration

### PostgreSQL Setup (Primary Database)

```bash
# Install PostgreSQL 15
sudo apt install postgresql-15 postgresql-contrib-15

# Configure PostgreSQL for HFT workloads
sudo -u postgres psql <<EOF
-- Create database and user
CREATE DATABASE foxhunt_trading;
CREATE USER foxhunt_user WITH ENCRYPTED PASSWORD 'secure_password_here';
GRANT ALL PRIVILEGES ON DATABASE foxhunt_trading TO foxhunt_user;

-- Performance tuning
ALTER SYSTEM SET shared_buffers = '8GB';
ALTER SYSTEM SET effective_cache_size = '24GB';
ALTER SYSTEM SET maintenance_work_mem = '2GB';
ALTER SYSTEM SET checkpoint_completion_target = 0.9;
ALTER SYSTEM SET wal_buffers = '64MB';
ALTER SYSTEM SET default_statistics_target = 100;
ALTER SYSTEM SET random_page_cost = 1.1;
ALTER SYSTEM SET effective_io_concurrency = 200;
ALTER SYSTEM SET work_mem = '256MB';
ALTER SYSTEM SET min_wal_size = '1GB';
ALTER SYSTEM SET max_wal_size = '4GB';

-- HFT-specific optimizations
ALTER SYSTEM SET synchronous_commit = off;
ALTER SYSTEM SET full_page_writes = off;
ALTER SYSTEM SET wal_compression = on;
ALTER SYSTEM SET wal_init_zero = off;
ALTER SYSTEM SET wal_recycle = off;

-- Connection pooling
ALTER SYSTEM SET max_connections = 200;
ALTER SYSTEM SET shared_preload_libraries = 'pg_stat_statements';

SELECT pg_reload_conf();
EOF

# Restart PostgreSQL
sudo systemctl restart postgresql
```

### InfluxDB Setup (Time-Series Metrics)

```bash
# Install InfluxDB 2.7
curl -LO https://dl.influxdata.com/influxdb/releases/influxdb2-2.7.0-amd64.deb
sudo dpkg -i influxdb2-2.7.0-amd64.deb

# Configure InfluxDB
sudo mkdir -p /etc/influxdb
sudo tee /etc/influxdb/config.toml > /dev/null <<EOF
[meta]
  dir = "/var/lib/influxdb/meta"

[data]
  dir = "/var/lib/influxdb/data"
  wal-dir = "/var/lib/influxdb/wal"
  
  # HFT optimizations
  cache-max-memory-size = "2g"
  cache-snapshot-memory-size = "100m"
  cache-snapshot-write-cold-duration = "1h"
  compact-full-write-cold-duration = "24h"
  max-series-per-database = 10000000
  max-values-per-tag = 1000000

[coordinator]
  write-timeout = "10s"
  max-concurrent-queries = 0
  query-timeout = "0s"
  log-queries-after = "0s"
  max-select-point = 0
  max-select-series = 0
  max-select-buckets = 0

[retention]
  enabled = true
  check-interval = "30m"

[shard-precreation]
  enabled = true
  check-interval = "10m"
  advance-period = "30m"

[admin]
  enabled = true
  bind-address = ":8086"
  https-enabled = false

[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = true
  log-enabled = true
  write-tracing = false
  pprof-enabled = false
  https-enabled = false
  max-row-limit = 10000
  max-connection-limit = 0
  shared-secret = "foxhunt_influx_secret"
  realm = "InfluxDB"

[logging]
  format = "auto"
  level = "info"
EOF

# Start InfluxDB
sudo systemctl enable influxdb
sudo systemctl start influxdb

# Initialize InfluxDB
influx setup \
  --username admin \
  --password secure_admin_password \
  --org foxhunt-trading \
  --bucket trading-metrics \
  --force
```

### Redis Setup (High-Performance Cache)

```bash
# Install Redis 7.0
curl -LO http://download.redis.io/redis-stable.tar.gz
tar xzf redis-stable.tar.gz
cd redis-stable
make && sudo make install

# Configure Redis for HFT
sudo tee /etc/redis/redis.conf > /dev/null <<EOF
# Network configuration
bind 127.0.0.1
port 6379
protected-mode yes
timeout 300
tcp-keepalive 300

# Memory configuration
maxmemory 16gb
maxmemory-policy allkeys-lru
save ""

# Performance optimizations
stop-writes-on-bgsave-error no
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis

# Logging
loglevel notice
logfile /var/log/redis/redis-server.log

# Security
requirepass secure_redis_password

# Advanced configuration
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
EOF

# Create Redis systemd service
sudo tee /etc/systemd/system/redis.service > /dev/null <<EOF
[Unit]
Description=Advanced key-value store
After=network.target

[Service]
Type=forking
User=redis
Group=redis
ExecStart=/usr/local/bin/redis-server /etc/redis/redis.conf
PIDFile=/var/run/redis/redis-server.pid
TimeoutStopSec=0
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Start Redis
sudo systemctl enable redis
sudo systemctl start redis
```

### ClickHouse Setup (OLAP Analytics)

```bash
# Install ClickHouse
curl https://clickhouse.com/install.sh | sh
sudo ./clickhouse install

# Configure ClickHouse for HFT analytics
sudo tee /etc/clickhouse-server/config.d/foxhunt.xml > /dev/null <<EOF
<clickhouse>
    <max_connections>4096</max_connections>
    <keep_alive_timeout>3</keep_alive_timeout>
    <max_concurrent_queries>100</max_concurrent_queries>
    <uncompressed_cache_size>8589934592</uncompressed_cache_size>
    <mark_cache_size>5368709120</mark_cache_size>
    <builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval>
    <max_session_timeout>3600</max_session_timeout>
    <default_session_timeout>60</default_session_timeout>
    
    <!-- Memory optimizations -->
    <max_memory_usage>32000000000</max_memory_usage>
    <max_bytes_before_external_group_by>16000000000</max_bytes_before_external_group_by>
    
    <!-- Performance settings -->
    <background_pool_size>16</background_pool_size>
    <background_merges_mutations_concurrency_ratio>2</background_merges_mutations_concurrency_ratio>
    
    <!-- Logging -->
    <logger>
        <level>information</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
    </logger>
</clickhouse>
EOF

# Start ClickHouse
sudo systemctl enable clickhouse-server
sudo systemctl start clickhouse-server
```

## Service Deployment

### Environment Configuration

```bash
# Create production environment file
sudo mkdir -p /opt/foxhunt/config/environments
sudo tee /opt/foxhunt/config/environments/.env.production > /dev/null <<EOF
# Production Environment Configuration
ENVIRONMENT=production
LOG_LEVEL=info
RUST_LOG=info

# Database Connections
DATABASE_URL="postgresql://foxhunt_user:secure_password_here@localhost:5432/foxhunt_trading"
INFLUXDB_URL="http://localhost:8086"
INFLUXDB_TOKEN="your_influxdb_token_here"
INFLUXDB_ORG="foxhunt-trading"
INFLUXDB_BUCKET="trading-metrics"
REDIS_URL="redis://:secure_redis_password@localhost:6379"
CLICKHOUSE_URL="http://localhost:8123"

# Security Configuration
JWT_SECRET_PATH="/opt/foxhunt/certs/jwt-secret.key"
ENCRYPTION_KEY_PATH="/opt/foxhunt/certs/encryption-key.key"
TLS_CERT_PATH="/opt/foxhunt/certs/production/foxhunt-cert.pem"
TLS_KEY_PATH="/opt/foxhunt/certs/production/foxhunt-key.pem"
TLS_CA_PATH="/opt/foxhunt/certs/production/ca-cert.pem"

# Performance Configuration
MAX_CRITICAL_LATENCY_US=50
MAX_TIMING_LATENCY_NS=14
ENABLE_SIMD=true
ENABLE_AVX2=true
ENABLE_GPU=true

# Trading Configuration
ENABLE_PAPER_TRADING=false
MAX_POSITION_SIZE=1000000
MAX_DAILY_LOSS=50000
RISK_CHECK_INTERVAL_MS=100

# Monitoring
PROMETHEUS_PORT=9090
METRICS_ENABLED=true
HEALTH_CHECK_INTERVAL_S=5

# Market Data
POLYGON_API_KEY="your_polygon_api_key"
ENABLE_REAL_TIME_DATA=true
DATA_RETENTION_DAYS=90

# Broker Configuration
ICMARKETS_DEMO=false
ICMARKETS_SERVER="live.icmarkets.com"
IB_TWS_PORT=7496
IB_GATEWAY_PORT=4001
EOF

# Set secure permissions
sudo chmod 600 /opt/foxhunt/config/environments/.env.production
```

### Service Compilation and Installation

```bash
# Clone and build Foxhunt
cd /opt
sudo git clone https://github.com/your-org/foxhunt.git
sudo chown -R foxhunt:foxhunt /opt/foxhunt
cd /opt/foxhunt

# Build in release mode with optimizations
export RUSTFLAGS="-C target-cpu=native -C opt-level=3 -C lto=fat"
cargo build --release --workspace

# Install systemd services
sudo cp deployment/systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload

# Create service user
sudo useradd -r -s /bin/false -d /opt/foxhunt foxhunt
sudo chown -R foxhunt:foxhunt /opt/foxhunt
```

### Service Startup Scripts

**Core Service:**
```bash
sudo tee /etc/systemd/system/foxhunt-core.service > /dev/null <<EOF
[Unit]
Description=Foxhunt Core Performance Infrastructure
After=network.target postgresql.service redis.service
Requires=postgresql.service redis.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
WorkingDirectory=/opt/foxhunt
Environment=RUST_LOG=info
EnvironmentFile=/opt/foxhunt/config/environments/.env.production
ExecStart=/opt/foxhunt/target/release/foxhunt-core
Restart=always
RestartSec=5
TimeoutStartSec=60
TimeoutStopSec=30

# Performance optimizations
Nice=-10
IOSchedulingClass=1
IOSchedulingPriority=4
CPUSchedulingPolicy=1
CPUSchedulingPriority=50

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/foxhunt/logs /opt/foxhunt/data

# Resource limits
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF
```

**TLI Service:**
```bash
sudo tee /etc/systemd/system/foxhunt-tli.service > /dev/null <<EOF
[Unit]
Description=Foxhunt Terminal Interface (TLI)
After=network.target foxhunt-core.service
Requires=foxhunt-core.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
WorkingDirectory=/opt/foxhunt
Environment=RUST_LOG=info
EnvironmentFile=/opt/foxhunt/config/environments/.env.production
ExecStart=/opt/foxhunt/target/release/tli
Restart=always
RestartSec=5
TimeoutStartSec=30
TimeoutStopSec=15

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/foxhunt/logs

# Resource limits
LimitNOFILE=8192

[Install]
WantedBy=multi-user.target
EOF
```

**ML Training Service:**
```bash
sudo tee /etc/systemd/system/foxhunt-ml.service > /dev/null <<EOF
[Unit]
Description=Foxhunt ML Training Service
After=network.target foxhunt-core.service
Requires=foxhunt-core.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
WorkingDirectory=/opt/foxhunt
Environment=RUST_LOG=info
Environment=CUDA_VISIBLE_DEVICES=0,1
EnvironmentFile=/opt/foxhunt/config/environments/.env.production
ExecStart=/opt/foxhunt/target/release/ml-training-service
Restart=always
RestartSec=10
TimeoutStartSec=120
TimeoutStopSec=30

# GPU access
DeviceAllow=/dev/nvidia0 rw
DeviceAllow=/dev/nvidia1 rw
DeviceAllow=/dev/nvidiactl rw
DeviceAllow=/dev/nvidia-modeset rw
DeviceAllow=/dev/nvidia-uvm rw
DeviceAllow=/dev/nvidia-uvm-tools rw

# Security
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/foxhunt/logs /opt/foxhunt/models /tmp

# Resource limits
LimitNOFILE=16384
LimitMEMLOCK=infinity
MemoryMax=64G

[Install]
WantedBy=multi-user.target
EOF
```

### Service Startup and Validation

```bash
# Enable and start all services
sudo systemctl enable foxhunt-core foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data
sudo systemctl start foxhunt-core
sleep 10
sudo systemctl start foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data

# Verify service status
sudo systemctl status foxhunt-core foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data

# Check service logs
sudo journalctl -u foxhunt-core -f --no-pager
sudo journalctl -u foxhunt-tli -f --no-pager

# Validate gRPC endpoints
grpcurl -plaintext localhost:50051 foxhunt.health.HealthService/Check
grpcurl -plaintext localhost:50051 foxhunt.trading.TradingService/GetSystemStatus
```

## Performance Optimization

### CPU Optimization

```bash
# Set CPU affinity for critical services
sudo tee /opt/foxhunt/scripts/set-affinity.sh > /dev/null <<'EOF'
#!/bin/bash

# Get core service PID
CORE_PID=$(systemctl show --property MainPID --value foxhunt-core)
TLI_PID=$(systemctl show --property MainPID --value foxhunt-tli)

# Assign cores (cores 0-7 for core service, 8-15 for TLI)
if [ "$CORE_PID" != "0" ]; then
    sudo taskset -cp 0-7 $CORE_PID
    echo "Core service (PID $CORE_PID) assigned to cores 0-7"
fi

if [ "$TLI_PID" != "0" ]; then
    sudo taskset -cp 8-15 $TLI_PID
    echo "TLI service (PID $TLI_PID) assigned to cores 8-15"
fi

# Set real-time priority for core service
if [ "$CORE_PID" != "0" ]; then
    sudo chrt -p -f 50 $CORE_PID
    echo "Core service set to real-time priority 50"
fi
EOF

chmod +x /opt/foxhunt/scripts/set-affinity.sh
sudo /opt/foxhunt/scripts/set-affinity.sh
```

### Memory Optimization

```bash
# Configure transparent huge pages
echo 'always' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo 'always' | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# NUMA optimization
sudo tee /opt/foxhunt/scripts/numa-optimize.sh > /dev/null <<'EOF'
#!/bin/bash

# Bind services to NUMA nodes
CORE_PID=$(systemctl show --property MainPID --value foxhunt-core)
ML_PID=$(systemctl show --property MainPID --value foxhunt-ml)

if [ "$CORE_PID" != "0" ]; then
    sudo numactl --cpunodebind=0 --membind=0 --pid=$CORE_PID
fi

if [ "$ML_PID" != "0" ]; then
    sudo numactl --cpunodebind=1 --membind=1 --pid=$ML_PID
fi
EOF

chmod +x /opt/foxhunt/scripts/numa-optimize.sh
sudo /opt/foxhunt/scripts/numa-optimize.sh
```

### Network Optimization

```bash
# Network interface optimization script
sudo tee /opt/foxhunt/scripts/network-optimize.sh > /dev/null <<'EOF'
#!/bin/bash

INTERFACE="eth0"  # Change to your primary interface

# Disable network features that add latency
sudo ethtool -K $INTERFACE gro off
sudo ethtool -K $INTERFACE gso off
sudo ethtool -K $INTERFACE tso off
sudo ethtool -K $INTERFACE ufo off
sudo ethtool -K $INTERFACE sg off
sudo ethtool -K $INTERFACE tx off
sudo ethtool -K $INTERFACE rx off

# Set interrupt coalescing to minimum
sudo ethtool -C $INTERFACE adaptive-rx off adaptive-tx off
sudo ethtool -C $INTERFACE rx-usecs 0 tx-usecs 0

# Increase ring buffer sizes
sudo ethtool -G $INTERFACE rx 4096 tx 4096

# Set network IRQ affinity
IRQ=$(cat /proc/interrupts | grep $INTERFACE | awk '{print $1}' | tr -d ':')
if [ ! -z "$IRQ" ]; then
    echo "2" | sudo tee /proc/irq/$IRQ/smp_affinity > /dev/null
    echo "Network IRQ $IRQ bound to CPU 1"
fi
EOF

chmod +x /opt/foxhunt/scripts/network-optimize.sh
sudo /opt/foxhunt/scripts/network-optimize.sh
```

## Monitoring & Alerting

### Prometheus Configuration

```yaml
# /opt/foxhunt/config/monitoring/prometheus.yml
global:
  scrape_interval: 1s
  evaluation_interval: 1s
  external_labels:
    cluster: 'foxhunt-production'
    environment: 'prod'

rule_files:
  - "hft-alerts.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

scrape_configs:
  - job_name: 'foxhunt-core'
    static_configs:
      - targets: ['localhost:9091']
    scrape_interval: 100ms  # High frequency for HFT
    
  - job_name: 'foxhunt-tli'
    static_configs:
      - targets: ['localhost:9092']
    scrape_interval: 1s
    
  - job_name: 'foxhunt-ml'
    static_configs:
      - targets: ['localhost:9093']
    scrape_interval: 5s
    
  - job_name: 'system-metrics'
    static_configs:
      - targets: ['localhost:9100']
    scrape_interval: 1s
    
  - job_name: 'postgresql'
    static_configs:
      - targets: ['localhost:9187']
      
  - job_name: 'redis'
    static_configs:
      - targets: ['localhost:9121']
```

### Critical Alerts Configuration

```yaml
# /opt/foxhunt/config/monitoring/hft-alerts.yml
groups:
  - name: trading.rules
    rules:
      - alert: HighLatency
        expr: trading_order_latency_microseconds > 50
        for: 1s
        labels:
          severity: critical
          service: trading
        annotations:
          summary: "Trading latency exceeded 50μs threshold"
          description: "Order execution latency is {{ $value }}μs"
          
      - alert: TimingSystemFailure
        expr: timing_tsc_reliability < 0.99
        for: 5s
        labels:
          severity: critical
          service: core
        annotations:
          summary: "TSC timing system unreliable"
          description: "TSC reliability dropped to {{ $value }}"
          
      - alert: CircuitBreakerTripped
        expr: risk_circuit_breaker_active == 1
        for: 0s
        labels:
          severity: critical
          service: risk
        annotations:
          summary: "Trading circuit breaker activated"
          description: "Emergency trading halt in effect"
          
      - alert: PositionLimitExceeded
        expr: risk_position_utilization > 0.95
        for: 30s
        labels:
          severity: warning
          service: risk
        annotations:
          summary: "Position limit near maximum"
          description: "Position utilization at {{ $value }}%"
          
      - alert: DatabaseConnectionFailure
        expr: database_connections_active == 0
        for: 10s
        labels:
          severity: critical
          service: persistence
        annotations:
          summary: "Database connection lost"
          description: "No active database connections"
          
      - alert: GPUUtilizationLow
        expr: ml_gpu_utilization < 0.1
        for: 300s
        labels:
          severity: warning
          service: ml
        annotations:
          summary: "GPU underutilized"
          description: "GPU utilization at {{ $value }}%"
          
      - alert: MemoryUsageHigh
        expr: system_memory_usage > 0.9
        for: 60s
        labels:
          severity: warning
          service: system
        annotations:
          summary: "High memory usage"
          description: "Memory usage at {{ $value }}%"
          
      - alert: NetworkLatencyHigh
        expr: network_roundtrip_latency_microseconds > 1000
        for: 30s
        labels:
          severity: warning
          service: network
        annotations:
          summary: "Network latency degraded"
          description: "Network roundtrip latency {{ $value }}μs"
```

### Grafana Dashboard Setup

```bash
# Install Grafana
sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana

# Configure Grafana
sudo tee /etc/grafana/grafana.ini > /dev/null <<EOF
[server]
http_port = 3000
domain = localhost

[security]
admin_user = admin
admin_password = secure_grafana_password

[dashboards]
default_home_dashboard_path = /etc/grafana/dashboards/foxhunt-overview.json
EOF

# Start Grafana
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
```

## Backup & Recovery

### Database Backup Strategy

```bash
# PostgreSQL backup script
sudo tee /opt/foxhunt/scripts/backup-postgresql.sh > /dev/null <<'EOF'
#!/bin/bash

BACKUP_DIR="/opt/foxhunt/backups/postgresql"
DATE=$(date +%Y%m%d_%H%M%S)
DB_NAME="foxhunt_trading"

mkdir -p $BACKUP_DIR

# Full backup
pg_dump -h localhost -U foxhunt_user -d $DB_NAME \
    --format=custom --compress=9 \
    --file=$BACKUP_DIR/foxhunt_full_$DATE.dump

# Incremental WAL backup
pg_basebackup -h localhost -U foxhunt_user -D $BACKUP_DIR/wal_$DATE \
    --format=tar --gzip --progress --verbose

# Cleanup old backups (keep 7 days)
find $BACKUP_DIR -name "*.dump" -mtime +7 -delete
find $BACKUP_DIR -name "wal_*" -mtime +7 -exec rm -rf {} \;

echo "Backup completed: $BACKUP_DIR/foxhunt_full_$DATE.dump"
EOF

chmod +x /opt/foxhunt/scripts/backup-postgresql.sh

# Schedule backups
echo "0 2 * * * /opt/foxhunt/scripts/backup-postgresql.sh" | sudo crontab -u foxhunt -
```

### System Configuration Backup

```bash
# Configuration backup script
sudo tee /opt/foxhunt/scripts/backup-config.sh > /dev/null <<'EOF'
#!/bin/bash

BACKUP_DIR="/opt/foxhunt/backups/config"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p $BACKUP_DIR

# Create configuration archive
tar -czf $BACKUP_DIR/config_$DATE.tar.gz \
    /opt/foxhunt/config \
    /opt/foxhunt/certs \
    /etc/systemd/system/foxhunt-*.service \
    /etc/sysctl.conf \
    /etc/security/limits.conf

echo "Configuration backup completed: $BACKUP_DIR/config_$DATE.tar.gz"
EOF

chmod +x /opt/foxhunt/scripts/backup-config.sh
```

### Disaster Recovery Procedures

```bash
# Recovery script template
sudo tee /opt/foxhunt/scripts/disaster-recovery.sh > /dev/null <<'EOF'
#!/bin/bash

echo "Foxhunt Disaster Recovery Procedure"
echo "==================================="

# 1. Stop all services
echo "Stopping all Foxhunt services..."
sudo systemctl stop foxhunt-*

# 2. Restore database
echo "Restoring PostgreSQL database..."
LATEST_BACKUP=$(ls -t /opt/foxhunt/backups/postgresql/*.dump | head -1)
if [ -f "$LATEST_BACKUP" ]; then
    sudo -u postgres dropdb foxhunt_trading
    sudo -u postgres createdb foxhunt_trading
    pg_restore -h localhost -U foxhunt_user -d foxhunt_trading $LATEST_BACKUP
    echo "Database restored from: $LATEST_BACKUP"
else
    echo "ERROR: No database backup found!"
    exit 1
fi

# 3. Restore configuration
echo "Restoring configuration..."
LATEST_CONFIG=$(ls -t /opt/foxhunt/backups/config/*.tar.gz | head -1)
if [ -f "$LATEST_CONFIG" ]; then
    tar -xzf $LATEST_CONFIG -C /
    echo "Configuration restored from: $LATEST_CONFIG"
else
    echo "WARNING: No configuration backup found!"
fi

# 4. Restart services
echo "Starting services in order..."
sudo systemctl start foxhunt-core
sleep 10
sudo systemctl start foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data

# 5. Verify system health
echo "Verifying system health..."
sleep 30
grpcurl -plaintext localhost:50051 foxhunt.health.HealthService/Check

echo "Disaster recovery completed!"
EOF

chmod +x /opt/foxhunt/scripts/disaster-recovery.sh
```

## Troubleshooting

### Common Issues and Solutions

**Issue: High Latency (>50μs)**
```bash
# Check CPU frequency scaling
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Verify real-time kernel
uname -r | grep rt

# Check network interface optimization
ethtool -k eth0 | grep -E "(gro|gso|tso)"

# Monitor CPU utilization
top -p $(pgrep foxhunt-core)
```

**Issue: Database Connection Errors**
```bash
# Check PostgreSQL status
sudo systemctl status postgresql
sudo -u postgres psql -c "SELECT version();"

# Check connection limits
sudo -u postgres psql -c "SHOW max_connections;"
sudo -u postgres psql -c "SELECT count(*) FROM pg_stat_activity;"

# Verify network connectivity
nc -zv localhost 5432
```

**Issue: GPU Not Detected**
```bash
# Check NVIDIA driver
nvidia-smi

# Verify CUDA installation
nvcc --version

# Check GPU permissions
ls -la /dev/nvidia*

# Verify systemd service configuration
sudo systemctl cat foxhunt-ml | grep -A5 "DeviceAllow"
```

**Issue: Memory Allocation Errors**
```bash
# Check huge pages
cat /proc/meminfo | grep Huge

# Verify memory limits
systemctl show foxhunt-core | grep Memory

# Check for memory leaks
valgrind --tool=massif ./target/release/foxhunt-core
```

### Log Analysis

```bash
# Aggregate log analysis
sudo journalctl -u foxhunt-* --since="1 hour ago" | grep -i error

# Performance log analysis
sudo journalctl -u foxhunt-core | grep "latency_us" | tail -100

# Real-time log monitoring
sudo journalctl -u foxhunt-core -f | grep -E "(ERROR|WARN|latency_us)"
```

### Performance Debugging

```bash
# CPU profiling
sudo perf record -g -p $(pgrep foxhunt-core)
sudo perf report

# Memory profiling
sudo valgrind --tool=massif --detailed-freq=1 ./target/release/foxhunt-core

# Network analysis
sudo tcpdump -i eth0 -n host exchange.com

# Latency measurement
sudo trace-cmd record -p function_graph -g do_IRQ -P $(pgrep foxhunt-core)
```

## Maintenance Procedures

### Regular Maintenance Tasks

**Daily:**
```bash
# Check service health
sudo systemctl is-active foxhunt-*

# Monitor disk usage
df -h | grep -E "(foxhunt|opt)"

# Check log rotation
sudo logrotate -f /etc/logrotate.d/foxhunt

# Verify backup completion
ls -la /opt/foxhunt/backups/postgresql/ | tail -5
```

**Weekly:**
```bash
# Update system packages
sudo apt update && sudo apt upgrade

# Rebuild indexes
sudo -u postgres psql foxhunt_trading -c "REINDEX DATABASE foxhunt_trading;"

# Clean old logs
find /opt/foxhunt/logs -name "*.log" -mtime +30 -delete

# Performance benchmarking
cd /opt/foxhunt && cargo bench
```

**Monthly:**
```bash
# Security updates
sudo unattended-upgrades

# Certificate renewal check
openssl x509 -in /opt/foxhunt/certs/production/foxhunt-cert.pem -noout -dates

# Database maintenance
sudo -u postgres psql foxhunt_trading -c "VACUUM ANALYZE;"

# System performance review
iostat -x 1 10
sar -u 1 10
```

### Update Procedures

```bash
# Production update script
sudo tee /opt/foxhunt/scripts/production-update.sh > /dev/null <<'EOF'
#!/bin/bash

echo "Foxhunt Production Update Procedure"
echo "=================================="

# 1. Pre-update backup
echo "Creating pre-update backup..."
/opt/foxhunt/scripts/backup-postgresql.sh
/opt/foxhunt/scripts/backup-config.sh

# 2. Stop services (graceful)
echo "Stopping services gracefully..."
sudo systemctl stop foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data
sleep 10
sudo systemctl stop foxhunt-core

# 3. Update code
echo "Updating codebase..."
cd /opt/foxhunt
git fetch origin
git checkout production
git pull origin production

# 4. Build new version
echo "Building updated version..."
cargo build --release --workspace

# 5. Run database migrations
echo "Running database migrations..."
./target/release/migration-tool migrate

# 6. Start services
echo "Starting services..."
sudo systemctl start foxhunt-core
sleep 15
sudo systemctl start foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data

# 7. Health verification
echo "Verifying system health..."
sleep 30
grpcurl -plaintext localhost:50051 foxhunt.health.HealthService/Check

# 8. Performance validation
echo "Running performance validation..."
timeout 60s ./target/release/performance-test --quick

echo "Update completed successfully!"
EOF

chmod +x /opt/foxhunt/scripts/production-update.sh
```

---

## Conclusion

This comprehensive deployment guide provides all necessary steps to deploy the Foxhunt HFT system in a production environment. The configuration emphasizes ultra-low latency performance, enterprise-grade security, and operational reliability suitable for high-frequency trading operations.

**Key Points:**
- Hardware requirements ensure sub-50μs latency capabilities
- Security configuration provides defense-in-depth protection
- Database setup optimized for HFT workloads
- Monitoring provides real-time visibility into system performance
- Backup and recovery procedures ensure business continuity
- Maintenance procedures keep the system running optimally

For additional support, refer to:
- [TLI Operations Manual](./TLI_OPERATIONS_MANUAL.md)
- [Performance Tuning Guide](./PERFORMANCE_TUNING.md)
- [Security Documentation](./SECURITY.md)
- [Disaster Recovery Plan](./DISASTER_RECOVERY.md)