foxhunt/docs/PRODUCTION_DEPLOYMENT.md

# Foxhunt HFT Trading System - Production Deployment Guide

## Overview

This guide provides comprehensive instructions for deploying the Foxhunt HFT Trading System to production environments. The deployment process is designed for zero-downtime deployments with comprehensive validation and rollback capabilities.

## Prerequisites

### Hardware Requirements
```
Production Server Specifications:
- CPU: Intel Xeon Gold 6248R (24 cores, 3.0GHz) or AMD EPYC 7543
- Memory: 128GB DDR4-3200 ECC
- Storage: 2TB NVMe SSD (Samsung 980 PRO or equivalent)
- Network: 25Gbps Mellanox ConnectX-6 or Intel E810
- OS: Ubuntu 22.04 LTS with real-time kernel
```

### Software Dependencies
```bash
# Install required packages
sudo apt update && sudo apt install -y \
    build-essential \
    cmake \
    pkg-config \
    libssl-dev \
    libpq-dev \
    redis-server \
    postgresql-14 \
    influxdb \
    nginx \
    docker.io \
    docker-compose

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env
rustup default stable
```

## Deployment Architecture

### Production Environment Layout
```
Production Infrastructure:
┌─────────────────────────────────────────────────────────┐
│                    Load Balancer                       │
│                  (HAProxy/Nginx)                       │
├─────────────────────────────────────────────────────────┤
│  Application Servers (3x for High Availability)        │
│  ├── Server-1: Primary Trading (Cores 2-5)             │
│  ├── Server-2: Risk Management (Cores 6-9)             │
│  └── Server-3: ML Inference (Cores 10-13)              │
├─────────────────────────────────────────────────────────┤
│  Database Cluster                                       │
│  ├── PostgreSQL Primary + 2 Replicas                   │
│  ├── InfluxDB Cluster (3 nodes)                        │
│  ├── Redis Cluster (3 masters + 3 replicas)           │
│  └── ClickHouse Cluster (2 shards, 2 replicas)        │
├─────────────────────────────────────────────────────────┤
│  Monitoring & Observability                            │
│  ├── Prometheus + Grafana                              │
│  ├── ELK Stack (Elasticsearch, Logstash, Kibana)      │
│  └── Jaeger Distributed Tracing                        │
└─────────────────────────────────────────────────────────┘
```

## Pre-Deployment Setup

### 1. Environment Configuration
```bash
# Create production environment file
cat > .env.production << EOF
# Environment
ENVIRONMENT=production
LOG_LEVEL=info
RUST_LOG=foxhunt=info,core=debug

# Database Configuration
DATABASE_URL=postgresql://foxhunt_user:${DB_PASSWORD}@db-primary:5432/foxhunt_production
REDIS_URL=redis://redis-cluster:6379
INFLUXDB_URL=http://influxdb-cluster:8086
CLICKHOUSE_URL=http://clickhouse-cluster:8123

# External APIs
POLYGON_API_KEY=${POLYGON_API_KEY}
ALPACA_API_KEY=${ALPACA_API_KEY}
ALPACA_SECRET_KEY=${ALPACA_SECRET_KEY}

# Broker Configuration
IB_HOST=ib-gateway.internal
IB_PORT=4001
IB_CLIENT_ID=1
IC_MARKETS_FIX_HOST=fix.icmarkets.com
IC_MARKETS_FIX_PORT=4448

# Performance Settings
MAX_LATENCY_US=50
ENABLE_SIMD=true
CPU_AFFINITY_CORES=2,3,4,5
MEMORY_POOL_SIZE_GB=16
ENABLE_RDTSC=true

# Risk Management
MAX_DAILY_LOSS=100000.00
MAX_POSITION_SIZE=2000000.00
VAR_CONFIDENCE_LEVEL=0.95
STRESS_TEST_SCENARIOS=10

# Security
TLI_AUTH_SECRET=${JWT_SECRET}
TLS_CERT_PATH=/etc/foxhunt/tls/cert.pem
TLS_KEY_PATH=/etc/foxhunt/tls/key.pem
ENABLE_MUTUAL_TLS=true

# Monitoring
PROMETHEUS_ENDPOINT=http://prometheus:9090
GRAFANA_ENDPOINT=http://grafana:3000
JAEGER_ENDPOINT=http://jaeger:14268

# High Availability
ENABLE_CLUSTERING=true
CLUSTER_NODES=foxhunt-1,foxhunt-2,foxhunt-3
LEADER_ELECTION=true
EOF
```

### 2. Security Setup
```bash
# Generate production certificates
mkdir -p /etc/foxhunt/tls
openssl req -x509 -newkey rsa:4096 -keyout /etc/foxhunt/tls/key.pem \
    -out /etc/foxhunt/tls/cert.pem -days 365 -nodes \
    -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/OU=Trading/CN=foxhunt.internal"

# Set proper permissions
chmod 600 /etc/foxhunt/tls/key.pem
chmod 644 /etc/foxhunt/tls/cert.pem
chown foxhunt:foxhunt /etc/foxhunt/tls/*

# Create JWT signing keys
openssl genrsa -out /etc/foxhunt/jwt-private.pem 2048
openssl rsa -in /etc/foxhunt/jwt-private.pem -pubout -out /etc/foxhunt/jwt-public.pem
```

### 3. Database Initialization
```bash
# PostgreSQL setup
sudo -u postgres createdb foxhunt_production
sudo -u postgres createuser foxhunt_user
sudo -u postgres psql -c "ALTER USER foxhunt_user WITH ENCRYPTED PASSWORD '${DB_PASSWORD}';"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE foxhunt_production TO foxhunt_user;"

# Run database migrations
./target/release/foxhunt-migrations --env production

# InfluxDB setup
influx setup --bucket foxhunt --org Foxhunt --retention 90d --token ${INFLUX_TOKEN}

# Redis cluster setup
redis-cli --cluster create \
    redis-1:6379 redis-2:6379 redis-3:6379 \
    redis-4:6379 redis-5:6379 redis-6:6379 \
    --cluster-replicas 1
```

## Deployment Process

### Step 1: Build Production Binaries
```bash
# Set production build flags
export RUSTFLAGS="-C target-cpu=native -C target-feature=+avx2,+fma"
export CARGO_PROFILE_RELEASE_LTO=fat
export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1

# Build all services with optimizations
cargo build --release --all-targets --features=production,simd,avx2

# Verify build artifacts
ls -la target/release/
```

### Step 2: Container Build and Registry Push
```bash
# Build production containers
docker build -f docker/Dockerfile.production \
    -t foxhunt/core:${VERSION} \
    --build-arg SERVICE=core .

docker build -f docker/Dockerfile.production \
    -t foxhunt/trading:${VERSION} \
    --build-arg SERVICE=trading .

docker build -f docker/Dockerfile.production \
    -t foxhunt/risk:${VERSION} \
    --build-arg SERVICE=risk .

docker build -f docker/Dockerfile.production \
    -t foxhunt/ml:${VERSION} \
    --build-arg SERVICE=ml .

docker build -f docker/Dockerfile.production \
    -t foxhunt/tli:${VERSION} \
    --build-arg SERVICE=tli .

# Push to registry
docker push foxhunt/core:${VERSION}
docker push foxhunt/trading:${VERSION}
docker push foxhunt/risk:${VERSION}
docker push foxhunt/ml:${VERSION}
docker push foxhunt/tli:${VERSION}
```

### Step 3: Infrastructure Deployment
```bash
# Deploy infrastructure with Terraform
cd deployment/terraform/production
terraform init
terraform plan -var="version=${VERSION}"
terraform apply -auto-approve

# Deploy Kubernetes infrastructure
kubectl apply -f deployment/k8s/namespace.yaml
kubectl apply -f deployment/k8s/secrets.yaml
kubectl apply -f deployment/k8s/configmaps.yaml
kubectl apply -f deployment/k8s/storage.yaml
```

### Step 4: Database Deployment
```bash
# Deploy database clusters
kubectl apply -f deployment/k8s/databases/postgresql-cluster.yaml
kubectl apply -f deployment/k8s/databases/redis-cluster.yaml
kubectl apply -f deployment/k8s/databases/influxdb-cluster.yaml
kubectl apply -f deployment/k8s/databases/clickhouse-cluster.yaml

# Wait for databases to be ready
kubectl wait --for=condition=ready pod -l app=postgresql --timeout=300s
kubectl wait --for=condition=ready pod -l app=redis --timeout=300s
kubectl wait --for=condition=ready pod -l app=influxdb --timeout=300s
kubectl wait --for=condition=ready pod -l app=clickhouse --timeout=300s
```

### Step 5: Application Deployment
```bash
# Deploy core services
kubectl apply -f deployment/k8s/services/core-service.yaml
kubectl wait --for=condition=available deployment/foxhunt-core --timeout=300s

# Deploy trading services
kubectl apply -f deployment/k8s/services/trading-service.yaml
kubectl wait --for=condition=available deployment/foxhunt-trading --timeout=300s

# Deploy risk management
kubectl apply -f deployment/k8s/services/risk-service.yaml
kubectl wait --for=condition=available deployment/foxhunt-risk --timeout=300s

# Deploy ML services
kubectl apply -f deployment/k8s/services/ml-service.yaml
kubectl wait --for=condition=available deployment/foxhunt-ml --timeout=300s

# Deploy TLI interface
kubectl apply -f deployment/k8s/services/tli-service.yaml
kubectl wait --for=condition=available deployment/foxhunt-tli --timeout=300s
```

### Step 6: Load Balancer and Ingress
```bash
# Deploy load balancer
kubectl apply -f deployment/k8s/networking/load-balancer.yaml

# Deploy ingress controller
kubectl apply -f deployment/k8s/networking/ingress.yaml

# Configure SSL termination
kubectl apply -f deployment/k8s/networking/ssl-config.yaml
```

### Step 7: Monitoring and Observability
```bash
# Deploy Prometheus
kubectl apply -f deployment/k8s/monitoring/prometheus.yaml

# Deploy Grafana
kubectl apply -f deployment/k8s/monitoring/grafana.yaml

# Deploy ELK Stack
kubectl apply -f deployment/k8s/monitoring/elasticsearch.yaml
kubectl apply -f deployment/k8s/monitoring/logstash.yaml
kubectl apply -f deployment/k8s/monitoring/kibana.yaml

# Deploy Jaeger
kubectl apply -f deployment/k8s/monitoring/jaeger.yaml
```

## Post-Deployment Validation

### Step 1: Health Checks
```bash
# Verify all services are running
kubectl get pods -n foxhunt

# Check service endpoints
curl -f https://foxhunt.internal/health
curl -f https://foxhunt.internal/api/v1/trading/health
curl -f https://foxhunt.internal/api/v1/risk/health
curl -f https://foxhunt.internal/api/v1/ml/health

# Verify database connectivity
./scripts/validate-database-connections.sh
```

### Step 2: Performance Validation
```bash
# Run latency benchmarks
./scripts/production-latency-test.sh

# Verify performance targets
./scripts/validate-performance-targets.sh

# Load testing
./scripts/production-load-test.sh --duration=300 --rps=1000
```

### Step 3: Security Validation
```bash
# SSL/TLS validation
./scripts/validate-ssl-certificates.sh

# Security scan
./scripts/production-security-scan.sh

# Penetration testing
./scripts/automated-pentest.sh
```

### Step 4: Integration Testing
```bash
# End-to-end integration tests
./scripts/production-integration-tests.sh

# Broker connectivity tests
./scripts/test-broker-connections.sh

# Market data validation
./scripts/validate-market-data-feeds.sh
```

## Zero-Downtime Deployment

### Blue-Green Deployment Strategy
```bash
#!/bin/bash
# Blue-Green deployment script

CURRENT_COLOR=$(kubectl get service foxhunt -o jsonpath='{.spec.selector.version}')
NEW_COLOR=$([ "$CURRENT_COLOR" = "blue" ] && echo "green" || echo "blue")

echo "Current deployment: $CURRENT_COLOR"
echo "Deploying to: $NEW_COLOR"

# Deploy new version to inactive environment
sed "s/{{COLOR}}/$NEW_COLOR/g" deployment/k8s/blue-green/deployment-template.yaml | kubectl apply -f -

# Wait for new deployment to be ready
kubectl wait --for=condition=available deployment/foxhunt-$NEW_COLOR --timeout=600s

# Run health checks on new deployment
./scripts/validate-deployment.sh $NEW_COLOR

# Switch traffic to new deployment
kubectl patch service foxhunt -p '{"spec":{"selector":{"version":"'$NEW_COLOR'"}}}'

# Monitor for issues
sleep 60
./scripts/monitor-deployment-health.sh

# Clean up old deployment
kubectl delete deployment foxhunt-$CURRENT_COLOR
```

### Rolling Update Strategy
```bash
# Configure rolling update strategy
kubectl patch deployment foxhunt-core -p '{
  "spec": {
    "strategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "maxUnavailable": 1,
        "maxSurge": 1
      }
    }
  }
}'

# Update image version
kubectl set image deployment/foxhunt-core core=foxhunt/core:${NEW_VERSION}

# Monitor rollout
kubectl rollout status deployment/foxhunt-core
```

## Configuration Management

### Environment-Specific Configurations
```bash
# Production configuration structure
config/
├── production/
│   ├── core.toml
│   ├── trading.toml
│   ├── risk.toml
│   ├── ml.toml
│   └── tli.toml
├── staging/
│   └── ...
└── development/
    └── ...
```

### Dynamic Configuration Updates
```bash
# Update configuration without restart
kubectl create configmap foxhunt-config \
    --from-file=config/production/ \
    --dry-run=client -o yaml | kubectl apply -f -

# Trigger configuration reload
kubectl rollout restart deployment/foxhunt-core
```

## Monitoring and Alerting Setup

### Prometheus Configuration
```yaml
# prometheus-config.yaml
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'foxhunt-core'
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names: ['foxhunt']
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: foxhunt-core
        action: keep

  - job_name: 'foxhunt-trading'
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names: ['foxhunt']
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: foxhunt-trading
        action: keep
```

### Grafana Dashboards
```bash
# Import pre-built dashboards
curl -X POST http://admin:admin@grafana:3000/api/dashboards/db \
  -H 'Content-Type: application/json' \
  -d @monitoring/grafana/foxhunt-overview.json

curl -X POST http://admin:admin@grafana:3000/api/dashboards/db \
  -H 'Content-Type: application/json' \
  -d @monitoring/grafana/foxhunt-performance.json
```

### Alert Rules
```yaml
# alert-rules.yaml
groups:
  - name: foxhunt.rules
    rules:
      - alert: HighOrderLatency
        expr: histogram_quantile(0.99, order_submission_latency_seconds_bucket) > 0.00005  # 50μs
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Order latency exceeding 50μs threshold"
          
      - alert: ServiceDown
        expr: up{job=~"foxhunt-.*"} == 0
        for: 10s
        labels:
          severity: critical
        annotations:
          summary: "Foxhunt service is down"
          
      - alert: HighMemoryUsage
        expr: process_resident_memory_bytes / node_memory_MemTotal_bytes > 0.8
        for: 60s
        labels:
          severity: warning
        annotations:
          summary: "High memory usage detected"
```

## Backup and Recovery

### Automated Backup Strategy
```bash
# Production backup script
#!/bin/bash
# /usr/local/bin/production-backup.sh

BACKUP_DIR="/backup/foxhunt/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Database backups
kubectl exec postgresql-primary-0 -- pg_dump foxhunt_production | gzip > "$BACKUP_DIR/postgresql.sql.gz"
kubectl exec influxdb-0 -- influx backup /tmp/backup && kubectl cp influxdb-0:/tmp/backup "$BACKUP_DIR/influxdb"
kubectl exec redis-0 -- redis-cli --rdb /tmp/dump.rdb && kubectl cp redis-0:/tmp/dump.rdb "$BACKUP_DIR/redis.rdb"

# Configuration backup
kubectl get configmaps -o yaml > "$BACKUP_DIR/configmaps.yaml"
kubectl get secrets -o yaml > "$BACKUP_DIR/secrets.yaml"

# Upload to S3
aws s3 sync "$BACKUP_DIR" "s3://foxhunt-backups/$(basename $BACKUP_DIR)"

echo "Backup completed: $BACKUP_DIR"
```

### Disaster Recovery Testing
```bash
# Monthly DR test
#!/bin/bash
# Test disaster recovery procedures

echo "Starting DR test..."

# Simulate primary site failure
./scripts/simulate-disaster.sh

# Activate DR site
./scripts/activate-dr-site.sh

# Validate DR functionality
./scripts/validate-dr-site.sh

# Failback to primary
./scripts/failback-to-primary.sh

echo "DR test completed"
```

## Security Hardening

### Network Security
```bash
# Configure network policies
kubectl apply -f deployment/k8s/security/network-policies.yaml

# Set up Web Application Firewall
kubectl apply -f deployment/k8s/security/waf-config.yaml

# Configure DDoS protection
kubectl apply -f deployment/k8s/security/ddos-protection.yaml
```

### Access Control
```bash
# Configure RBAC
kubectl apply -f deployment/k8s/security/rbac.yaml

# Set up service accounts
kubectl apply -f deployment/k8s/security/service-accounts.yaml

# Configure Pod Security Standards
kubectl apply -f deployment/k8s/security/pod-security.yaml
```

## Troubleshooting

### Common Deployment Issues

#### Service Discovery Problems
```bash
# Check DNS resolution
kubectl exec -it foxhunt-core-0 -- nslookup foxhunt-trading

# Verify service endpoints
kubectl get endpoints

# Check network connectivity
kubectl exec -it foxhunt-core-0 -- curl http://foxhunt-trading:8080/health
```

#### Performance Issues
```bash
# Check resource usage
kubectl top pods
kubectl top nodes

# Review metrics
curl http://prometheus:9090/api/v1/query?query=up

# Check logs
kubectl logs -f deployment/foxhunt-core
```

#### Database Connection Issues
```bash
# Check database status
kubectl exec -it postgresql-primary-0 -- psql -c "SELECT version();"

# Verify connection pools
kubectl logs deployment/foxhunt-core | grep "database"

# Test connectivity
kubectl exec -it foxhunt-core-0 -- ./scripts/test-db-connection.sh
```

## Rollback Procedures

### Automatic Rollback
```bash
# Rollback deployment
kubectl rollout undo deployment/foxhunt-core

# Rollback to specific revision
kubectl rollout undo deployment/foxhunt-core --to-revision=2

# Check rollback status
kubectl rollout status deployment/foxhunt-core
```

### Manual Rollback
```bash
# Emergency rollback script
#!/bin/bash
PREVIOUS_VERSION=$1

echo "Rolling back to version: $PREVIOUS_VERSION"

# Update all deployments
kubectl set image deployment/foxhunt-core core=foxhunt/core:$PREVIOUS_VERSION
kubectl set image deployment/foxhunt-trading trading=foxhunt/trading:$PREVIOUS_VERSION
kubectl set image deployment/foxhunt-risk risk=foxhunt/risk:$PREVIOUS_VERSION
kubectl set image deployment/foxhunt-ml ml=foxhunt/ml:$PREVIOUS_VERSION
kubectl set image deployment/foxhunt-tli tli=foxhunt/tli:$PREVIOUS_VERSION

# Wait for rollback completion
kubectl rollout status deployment/foxhunt-core
kubectl rollout status deployment/foxhunt-trading
kubectl rollout status deployment/foxhunt-risk
kubectl rollout status deployment/foxhunt-ml
kubectl rollout status deployment/foxhunt-tli

# Validate rollback
./scripts/validate-deployment.sh

echo "Rollback completed"
```

## Maintenance Windows

### Scheduled Maintenance
```bash
# Pre-maintenance checklist
./scripts/pre-maintenance-checklist.sh

# Put system in maintenance mode
kubectl scale deployment foxhunt-tli --replicas=0

# Perform maintenance
./scripts/maintenance-procedures.sh

# Bring system back online
kubectl scale deployment foxhunt-tli --replicas=3

# Post-maintenance validation
./scripts/post-maintenance-validation.sh
```

This production deployment guide provides comprehensive procedures for deploying and maintaining the Foxhunt HFT trading system in production environments with enterprise-grade reliability and performance.