foxhunt/docs/PRODUCTION_DEPLOYMENT_GUIDE_V2.md

# Foxhunt HFT Trading System - Production Deployment Guide

**Version**: 2.0
**Last Updated**: 2025-10-03
**Wave**: 71 - Complete Production Stack
**Status**: Production Ready with Full Security Stack

---

## Table of Contents

1. [Executive Summary](#executive-summary)
2. [System Architecture Overview](#system-architecture-overview)
3. [Prerequisites](#prerequisites)
4. [Pre-Deployment Checklist](#pre-deployment-checklist)
5. [Infrastructure Setup](#infrastructure-setup)
6. [Database Migration](#database-migration)
7. [TLS Certificate Configuration](#tls-certificate-configuration)
8. [Environment Configuration](#environment-configuration)
9. [Service Deployment](#service-deployment)
10. [Post-Deployment Verification](#post-deployment-verification)
11. [Performance Tuning](#performance-tuning)
12. [Disaster Recovery](#disaster-recovery)
13. [Rollback Procedures](#rollback-procedures)
14. [Production Deployment Pitfalls](#production-deployment-pitfalls)

---

## Executive Summary

This guide provides a complete, step-by-step procedure for deploying the Foxhunt HFT Trading System to production. The system consists of 4 main services with comprehensive security features designed for SOX/MiFID II compliance.

### Production Readiness Status

**✅ Production Ready Components**:
- API Gateway (6-layer authentication, rate limiting, audit logging)
- Trading Service (order execution, portfolio management, kill switch)
- Backtesting Service (strategy validation)
- ML Training Service (model lifecycle management)
- TLI (terminal client interface)

**✅ Security Features**:
- JWT authentication with revocation (Redis-backed)
- MFA/TOTP implementation (RFC 6238)
- RBAC with granular permissions
- Mutual TLS between services
- SQL injection protection (parameterized queries)
- Comprehensive audit trails (SOX/MiFID II)

**✅ Infrastructure**:
- PostgreSQL 16+ with NOTIFY/LISTEN for hot-reload
- Redis 7+ for JWT revocation and rate limiting
- S3 integration for ML model storage
- Docker deployment configurations

### Deployment Timeline

**Total Deployment Time**: 4-6 hours (first deployment)

| Phase | Duration | Status Gate |
|-------|----------|-------------|
| Infrastructure Setup | 60 min | PostgreSQL + Redis operational |
| Database Migration | 30 min | All migrations applied |
| Certificate Generation | 45 min | CA + service certs ready |
| Service Deployment | 90 min | All health checks passing |
| Verification | 60 min | Smoke tests complete |
| Performance Tuning | 60 min | Latency baselines measured |

---

## System Architecture Overview

### Service Topology

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         FOXHUNT HFT SYSTEM                              │
│                      Production Architecture                            │
└─────────────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────────────┐
│                          CLIENT LAYER                                    │
├──────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │  TLI (Terminal Client)                                            │  │
│  │  - Ratatui terminal UI                                            │  │
│  │  - gRPC client to 3 services                                      │  │
│  │  - JWT authentication                                             │  │
│  └─────────────────────┬────────────────────────────────────────────┘  │
│                        │                                                │
└────────────────────────┼────────────────────────────────────────────────┘
                         │
┌────────────────────────┼────────────────────────────────────────────────┐
│                        │         API GATEWAY LAYER                      │
├────────────────────────┼────────────────────────────────────────────────┤
│                        │                                                │
│  ┌─────────────────────▼──────────────────────────────────────────┐   │
│  │  API Gateway (0.0.0.0:50051)                                    │   │
│  │  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  │   │
│  │  6-Layer Security Stack:                                        │   │
│  │  1. JWT Validation (HS512, cached decoding key)                 │   │
│  │  2. JWT Revocation Check (Redis, <100μs)                        │   │
│  │  3. MFA/TOTP Validation (RFC 6238)                              │   │
│  │  4. RBAC Authorization (cached permissions)                     │   │
│  │  5. Rate Limiting (100 req/s per user)                          │   │
│  │  6. Audit Logging (SOX/MiFID II compliance)                     │   │
│  │  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  │   │
│  │  Performance: <10μs authentication overhead                     │   │
│  └──────────┬──────────────┬──────────────┬────────────────────────┘   │
│             │              │              │                            │
└─────────────┼──────────────┼──────────────┼────────────────────────────┘
              │              │              │
┌─────────────┼──────────────┼──────────────┼────────────────────────────┐
│             │              │              │   BACKEND SERVICES          │
├─────────────┼──────────────┼──────────────┼────────────────────────────┤
│             │              │              │                            │
│  ┌──────────▼─────────┐ ┌─▼────────────┐ ┌▼──────────────────────┐   │
│  │ Trading Service    │ │ Backtesting  │ │ ML Training Service   │   │
│  │ (50052)            │ │ Service      │ │ (50054)               │   │
│  │                    │ │ (50053)      │ │                       │   │
│  │ • Order execution  │ │              │ │ • Model training      │   │
│  │ • Portfolio mgmt   │ │ • Strategy   │ │ • S3 model storage    │   │
│  │ • Risk checks      │ │   validation │ │ • Version management  │   │
│  │ • Kill switch      │ │ • Historical │ │ • Metrics collection  │   │
│  │ • Compliance       │ │   replay     │ │                       │   │
│  └──────────┬─────────┘ └─┬────────────┘ └┬──────────────────────┘   │
│             │              │              │                            │
│             └──────────────┴──────────────┘                            │
│                            │                                           │
└────────────────────────────┼───────────────────────────────────────────┘
                             │
┌────────────────────────────┼───────────────────────────────────────────┐
│                            │      INFRASTRUCTURE LAYER                 │
├────────────────────────────┼───────────────────────────────────────────┤
│                            │                                           │
│  ┌─────────────────────────▼────────────────────────────────────┐    │
│  │  PostgreSQL 16+ Cluster (Primary + Replica)                  │    │
│  │  • Streaming replication (async)                             │    │
│  │  • NOTIFY/LISTEN for config hot-reload                       │    │
│  │  • Connection pooling (20 max per service)                   │    │
│  │  • Schema: users, roles, permissions, mfa_config, audit      │    │
│  └──────────────────────────────────────────────────────────────┘    │
│                                                                        │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  Redis 7+ Cluster (Sentinel/Cluster Mode)                    │    │
│  │  • JWT revocation blacklist                                  │    │
│  │  • Rate limiting (token bucket)                              │    │
│  │  • AOF persistence                                           │    │
│  └──────────────────────────────────────────────────────────────┘    │
│                                                                        │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  S3 (ML Model Storage)                                       │    │
│  │  • Server-side encryption (SSE-S3)                           │    │
│  │  • Model versioning                                          │    │
│  │  • Local cache (/cache/models/)                              │    │
│  └──────────────────────────────────────────────────────────────┘    │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────┐
│                       MONITORING & OBSERVABILITY                        │
├────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  Prometheus → Grafana → PagerDuty/Slack                                │
│  • Service metrics (latency, throughput, errors)                       │
│  • Infrastructure metrics (CPU, memory, disk, network)                 │
│  • Business metrics (orders/sec, fill rate, P&L)                       │
│                                                                         │
│  ELK Stack / Loki (Centralized Logging)                                │
│  • All service logs aggregated                                         │
│  • Audit trail immutability                                            │
│  • Security event monitoring                                           │
│                                                                         │
└────────────────────────────────────────────────────────────────────────┘
```

### Network Topology

```
┌─────────────────────────────────────────────────────────────────────┐
│                         NETWORK SEGMENTATION                        │
└─────────────────────────────────────────────────────────────────────┘

Internet/Exchanges
      │
      │ TLS 1.3
      ▼
┌──────────────────┐
│  Firewall/WAF    │
└────────┬─────────┘
         │
         ▼
┌─────────────────────────────────────────────┐
│  DMZ (Public Zone)                          │
│  ┌──────────────────┐                       │
│  │  API Gateway     │ 0.0.0.0:50051 (gRPC) │
│  │  Health: 8080    │ (HTTP health check)  │
│  └────────┬─────────┘                       │
└───────────┼──────────────────────────────────┘
            │ mTLS
            ▼
┌─────────────────────────────────────────────┐
│  Application Zone (Private)                 │
│  ┌──────────────┐  ┌──────────────┐        │
│  │ Trading Svc  │  │ Backtesting  │        │
│  │ :50052       │  │ Svc :50053   │        │
│  └──────┬───────┘  └──────┬───────┘        │
│         │                 │                 │
│  ┌──────▼─────────────────▼───────┐        │
│  │   ML Training Service           │        │
│  │   :50054                        │        │
│  └──────┬──────────────────────────┘        │
└─────────┼──────────────────────────────────┘
          │
          ▼
┌─────────────────────────────────────────────┐
│  Data Zone (Restricted)                     │
│  ┌───────────────┐  ┌──────────────┐       │
│  │ PostgreSQL    │  │ Redis        │       │
│  │ :5432         │  │ :6379        │       │
│  └───────────────┘  └──────────────┘       │
│                                             │
│  ┌──────────────────────────────────┐      │
│  │ S3 (via VPC Endpoint)            │      │
│  │ s3://foxhunt-models/             │      │
│  └──────────────────────────────────┘      │
└─────────────────────────────────────────────┘
```

---

## Prerequisites

### Hardware Requirements

#### API Gateway + Backend Services (Single Host Deployment)

**Minimum**:
```yaml
CPU: Intel Xeon Gold 6248R (24 cores @ 3.0GHz) OR AMD EPYC 7543 (32 cores @ 2.8GHz)
RAM: 128GB DDR4-3200 ECC
Storage: 2TB NVMe SSD (Samsung 980 PRO, WD Black SN850)
  - Write latency: <100μs (99.9th percentile)
  - Sequential read: >6000 MB/s
Network: 25Gbps NIC (Mellanox ConnectX-6, Intel E810)
  - Sub-1ms latency to exchange colocations
  - PTP/NTP for time synchronization (±100μs accuracy)
```

**Recommended (Distributed Deployment)**:
```yaml
API Gateway Host:
  CPU: 16 cores @ 3.5GHz+
  RAM: 64GB
  Network: 10Gbps dedicated

Trading Service Host:
  CPU: 32 cores @ 3.5GHz+ (CPU affinity for critical threads)
  RAM: 128GB
  Network: 25Gbps, kernel bypass (DPDK)

ML Training Service Host:
  GPU: 2x NVIDIA A100 80GB OR 1x H100 80GB
  CPU: 32 cores
  RAM: 256GB
  Storage: 4TB NVMe (model storage cache)

Database/Redis Host:
  CPU: 24 cores
  RAM: 256GB (large PostgreSQL shared_buffers)
  Storage: 4TB NVMe in RAID 10
```

### Operating System

**Recommended**: Ubuntu 22.04 LTS (Kernel 6.2+)

**OS-Level Hardening** (Critical for HFT):
```bash
# Real-time kernel for low latency
sudo apt install linux-image-lowlatency

# Network tuning
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

# File descriptor limits
echo "* soft nofile 1048576" | sudo tee -a /etc/security/limits.conf
echo "* hard nofile 1048576" | sudo tee -a /etc/security/limits.conf

# CPU isolation (replace 0-7 with CPU cores to isolate)
# Edit /etc/default/grub:
# GRUB_CMDLINE_LINUX="isolcpus=0-7 nohz_full=0-7 rcu_nocbs=0-7"
sudo update-grub
```

### Software Dependencies

```bash
# Update system
sudo apt update && sudo apt upgrade -y

# Essential build tools
sudo apt install -y \
    build-essential \
    cmake \
    pkg-config \
    libssl-dev \
    libpq-dev \
    protobuf-compiler \
    git \
    curl \
    wget

# PostgreSQL 16
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt update
sudo apt install -y postgresql-16 postgresql-client-16

# Redis 7
sudo apt install -y redis-server

# Docker (optional, for containerized deployment)
sudo apt install -y docker.io docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Rust toolchain (1.75+)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source ~/.cargo/env
rustup default stable

# Verify versions
cargo --version  # Should be 1.75.0+
psql --version   # Should be 16.x
redis-server --version  # Should be 7.x
```

### Time Synchronization (CRITICAL for HFT)

**⚠️ WARNING**: Skewed clocks can cause incorrect trades, regulatory violations (MiFID II), and audit failures.

```bash
# Install chrony (more accurate than ntpd)
sudo apt install -y chrony

# Configure chrony for financial market time sync
sudo tee /etc/chrony/chrony.conf <<EOF
# Use NIST time servers (replace with your region's time servers)
server time.nist.gov iburst
server time-a-g.nist.gov iburst
server time-b-g.nist.gov iburst

# For extreme accuracy, use PTP (Precision Time Protocol) hardware
# This requires PTP-capable NICs and switches

# Allow large time adjustments (be cautious in production)
makestep 1.0 3

# Log statistics
logdir /var/log/chrony
EOF

sudo systemctl restart chrony

# Verify time sync
chronyc tracking
# Expected: System time offset < 100μs for HFT
```

**PTP Configuration (for sub-microsecond accuracy)**:
```bash
# Install linuxptp
sudo apt install -y linuxptp

# Configure PTP on your network interface (replace ens192)
sudo ptp4l -i ens192 -m -s -H &
sudo phc2sys -s ens192 -m -w &
```

---

## Pre-Deployment Checklist

### Security Preparation

- [ ] **Secrets Manager** deployed (HashiCorp Vault, AWS Secrets Manager, etc.)
- [ ] **JWT Secret** generated (64+ bytes of cryptographic randomness)
  ```bash
  openssl rand -base64 64 > /secrets/jwt_secret.key
  chmod 600 /secrets/jwt_secret.key
  ```
- [ ] **Database Credentials** stored in secrets manager
- [ ] **Redis Password** generated and stored
- [ ] **S3 Access Keys** created with minimal IAM permissions
- [ ] **TLS CA Certificate** ready (or use Let's Encrypt)
- [ ] **Firewall Rules** defined (see Network Topology)

### Infrastructure Validation

- [ ] **PostgreSQL 16+** installed and accessible
- [ ] **Redis 7+** installed and accessible
- [ ] **S3 Bucket** created with versioning enabled
- [ ] **DNS Records** configured (if using domain names)
- [ ] **NTP/PTP** synchronized (time offset < 100μs)
- [ ] **Network Connectivity** tested (ping exchanges, databases)
- [ ] **Disk Space** available (minimum 500GB free)

### Compliance Documentation

- [ ] **SOX Compliance** checklist reviewed
- [ ] **MiFID II** timestamping requirements understood
- [ ] **Audit Trail** retention policy defined (7 years recommended)
- [ ] **Disaster Recovery** plan documented
- [ ] **Incident Response** plan reviewed

---

## Infrastructure Setup

### PostgreSQL Cluster Setup

#### 1. Primary Database Configuration

```bash
# Create production database and user
sudo -u postgres psql <<EOF
CREATE USER foxhunt_user WITH PASSWORD 'REPLACE_WITH_VAULT_PASSWORD';
CREATE DATABASE foxhunt_production OWNER foxhunt_user;

-- Grant necessary permissions
GRANT ALL PRIVILEGES ON DATABASE foxhunt_production TO foxhunt_user;

-- Enable required extensions
\c foxhunt_production
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements";

-- Grant extension usage
GRANT ALL ON SCHEMA public TO foxhunt_user;
EOF
```

#### 2. PostgreSQL Performance Tuning

Edit `/etc/postgresql/16/main/postgresql.conf`:

```ini
# Connection Settings
max_connections = 200
superuser_reserved_connections = 3

# Memory Settings (adjust based on available RAM)
shared_buffers = 32GB              # 25% of RAM for 128GB system
effective_cache_size = 96GB        # 75% of RAM
work_mem = 64MB                    # Per-operation memory
maintenance_work_mem = 2GB

# WAL Settings (for durability and replication)
wal_level = replica
wal_buffers = 16MB
max_wal_size = 4GB
min_wal_size = 1GB
checkpoint_completion_target = 0.9

# Query Planner
random_page_cost = 1.1             # For NVMe SSDs
effective_io_concurrency = 200

# Logging (for audit trails)
logging_collector = on
log_directory = '/var/log/postgresql'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_statement = 'mod'              # Log all DDL and DML
log_duration = on
log_line_prefix = '%t [%p]: user=%u,db=%d,app=%a,client=%h '

# NOTIFY/LISTEN for Config Hot-Reload
listen_addresses = '*'
```

Edit `/etc/postgresql/16/main/pg_hba.conf` (authentication):

```conf
# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   all             postgres                                peer
host    foxhunt_production  foxhunt_user    10.0.0.0/8          scram-sha-256
host    foxhunt_production  foxhunt_user    127.0.0.1/32        scram-sha-256
```

Restart PostgreSQL:
```bash
sudo systemctl restart postgresql
sudo systemctl enable postgresql
```

#### 3. Streaming Replication (Optional but Recommended)

**On Replica Host**:

```bash
# Stop PostgreSQL on replica
sudo systemctl stop postgresql

# Create base backup from primary
sudo -u postgres pg_basebackup -h PRIMARY_HOST -D /var/lib/postgresql/16/main -U replication -P -v -R

# Start replica
sudo systemctl start postgresql
```

**Verify Replication**:
```sql
-- On primary
SELECT * FROM pg_stat_replication;

-- On replica
SELECT pg_is_in_recovery();  -- Should return 't'
```

### Redis Cluster Setup

#### 1. Redis Configuration

Edit `/etc/redis/redis.conf`:

```conf
# Network
bind 0.0.0.0
protected-mode yes
port 6379
requirepass REPLACE_WITH_VAULT_PASSWORD

# Memory
maxmemory 16gb
maxmemory-policy allkeys-lru

# Persistence (for JWT revocation durability)
save 900 1
save 300 10
save 60 10000
appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# Performance
tcp-backlog 511
timeout 0
tcp-keepalive 300

# Logging
loglevel notice
logfile /var/log/redis/redis-server.log
```

Restart Redis:
```bash
sudo systemctl restart redis-server
sudo systemctl enable redis-server
```

#### 2. Redis Sentinel (High Availability)

**For production, deploy Redis Sentinel for automatic failover**:

```bash
# sentinel.conf
sentinel monitor foxhunt-redis MASTER_HOST 6379 2
sentinel auth-pass foxhunt-redis REPLACE_WITH_VAULT_PASSWORD
sentinel down-after-milliseconds foxhunt-redis 5000
sentinel failover-timeout foxhunt-redis 10000
sentinel parallel-syncs foxhunt-redis 1
```

Start Sentinel:
```bash
redis-sentinel /etc/redis/sentinel.conf &
```

---

## Database Migration

### Migration Files

The following migrations must be applied in order:

```bash
cd /home/jgrusewski/Work/foxhunt/database/migrations

# List all migrations
ls -1 *.sql | sort
```

**Migration Order**:
1. `009_security_api_keys.sql` - API key management
2. `010_compliance_audit_trails.sql` - Audit trail tables
3. `011_compliance_rules_dynamic.sql` - Dynamic compliance rules
4. `015_adaptive_strategy_config.sql` - Strategy configuration
5. `016_adaptive_strategy_seed_data.sql` - Seed data
6. `016_ml_training_data_tables.sql` - ML training tables
7. `017_mfa_totp_implementation.sql` - **MFA/TOTP (CRITICAL)**
8. `018_rbac_permissions.sql` - **RBAC (CRITICAL)**
9. `018_config_management_system.sql` - Config hot-reload
10. `019_config_notify_triggers.sql` - **PostgreSQL NOTIFY triggers (CRITICAL)**

### Apply Migrations

```bash
# Set database connection
export DATABASE_URL="postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production"

# Apply all migrations in order
cd /home/jgrusewski/Work/foxhunt/database/migrations

psql $DATABASE_URL -f 009_security_api_keys.sql
psql $DATABASE_URL -f 010_compliance_audit_trails.sql
psql $DATABASE_URL -f 011_compliance_rules_dynamic.sql
psql $DATABASE_URL -f 015_adaptive_strategy_config.sql
psql $DATABASE_URL -f 016_adaptive_strategy_seed_data.sql
psql $DATABASE_URL -f 016_ml_training_data_tables.sql
psql $DATABASE_URL -f 017_mfa_totp_implementation.sql
psql $DATABASE_URL -f 018_rbac_permissions.sql
psql $DATABASE_URL -f 018_config_management_system.sql
psql $DATABASE_URL -f 019_config_notify_triggers.sql

# Verify migrations
psql $DATABASE_URL -c "\dt"  # List all tables
```

### Verify Critical Tables

```sql
-- Verify RBAC tables
SELECT * FROM roles;
SELECT * FROM permissions;

-- Verify MFA tables
SELECT * FROM mfa_config LIMIT 5;

-- Verify audit trail
SELECT * FROM audit_events ORDER BY created_at DESC LIMIT 10;
```

### Seed Initial Data

```bash
# Create default admin user and roles
psql $DATABASE_URL <<EOF
-- Create admin role
INSERT INTO roles (name, description) VALUES
    ('admin', 'System administrator with full access'),
    ('trader', 'Trading desk user with order execution rights'),
    ('viewer', 'Read-only access to dashboards and reports')
ON CONFLICT (name) DO NOTHING;

-- Create default permissions
INSERT INTO permissions (endpoint, description) VALUES
    ('trading.submit_order', 'Submit trading orders'),
    ('trading.cancel_order', 'Cancel existing orders'),
    ('trading.view_portfolio', 'View portfolio positions'),
    ('backtesting.run_test', 'Run backtesting simulations'),
    ('ml.train_model', 'Train machine learning models'),
    ('system.admin', 'System administration access')
ON CONFLICT (endpoint) DO NOTHING;

-- Grant permissions to admin role
INSERT INTO role_permissions (role_id, permission_id)
SELECT r.id, p.id FROM roles r CROSS JOIN permissions p WHERE r.name = 'admin';

-- Create initial admin user (REPLACE PASSWORD HASH)
INSERT INTO users (username, email, password_hash) VALUES
    ('admin', 'admin@foxhunt.local', 'REPLACE_WITH_BCRYPT_HASH')
ON CONFLICT (username) DO NOTHING;

-- Grant admin role to admin user
INSERT INTO user_roles (user_id, role_id)
SELECT u.id, r.id FROM users u, roles r WHERE u.username = 'admin' AND r.name = 'admin';
EOF
```

---

## TLS Certificate Configuration

### 1. Generate Certificate Authority (CA)

```bash
# Create CA directory
mkdir -p /etc/foxhunt/certs
cd /etc/foxhunt/certs

# Generate CA private key
openssl genrsa -out ca.key 4096

# Generate CA certificate (valid for 10 years)
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt \
    -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=Foxhunt Root CA"

# Protect CA key
chmod 400 ca.key
chmod 644 ca.crt
```

### 2. Generate Service Certificates

**API Gateway Certificate**:

```bash
# Generate private key
openssl genrsa -out api_gateway.key 4096

# Generate CSR
openssl req -new -key api_gateway.key -out api_gateway.csr \
    -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=api-gateway.foxhunt.local"

# Sign with CA
openssl x509 -req -in api_gateway.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -out api_gateway.crt -days 825 -sha256

# Cleanup
rm api_gateway.csr
chmod 400 api_gateway.key
chmod 644 api_gateway.crt
```

**Repeat for each service**:
- `trading_service.{key,crt}`
- `backtesting_service.{key,crt}`
- `ml_training_service.{key,crt}`

### 3. Client Certificates (for mTLS)

```bash
# Generate client key
openssl genrsa -out client.key 4096

# Generate client CSR
openssl req -new -key client.key -out client.csr \
    -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=tli-client"

# Sign with CA
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -out client.crt -days 825 -sha256

# Cleanup
rm client.csr
```

### 4. Distribute Certificates

```bash
# Copy to service directories
sudo cp ca.crt /etc/foxhunt/certs/
sudo cp api_gateway.{key,crt} /etc/foxhunt/api_gateway/
sudo cp trading_service.{key,crt} /etc/foxhunt/trading_service/
sudo cp backtesting_service.{key,crt} /etc/foxhunt/backtesting_service/
sudo cp ml_training_service.{key,crt} /etc/foxhunt/ml_training_service/

# Set permissions
sudo chown foxhunt:foxhunt /etc/foxhunt/certs/*
sudo chmod 400 /etc/foxhunt/certs/*.key
sudo chmod 644 /etc/foxhunt/certs/*.crt
```

---

## Environment Configuration

### API Gateway Environment Variables

Create `/etc/foxhunt/api_gateway/.env`:

```bash
# Network
GATEWAY_BIND_ADDR=0.0.0.0:50051
HEALTH_PORT=8080

# JWT Configuration
JWT_SECRET_FILE=/secrets/jwt_secret.key
JWT_ISSUER=foxhunt-api-gateway
JWT_AUDIENCE=foxhunt-services
JWT_EXPIRY_SECONDS=3600

# Redis (JWT Revocation)
REDIS_URL=redis://:PASSWORD@localhost:6379/0

# Rate Limiting
RATE_LIMIT_RPS=100

# Audit Logging
ENABLE_AUDIT_LOGGING=true
AUDIT_LOG_FILE=/var/log/foxhunt/audit.log

# TLS
TLS_CERT_FILE=/etc/foxhunt/api_gateway/api_gateway.crt
TLS_KEY_FILE=/etc/foxhunt/api_gateway/api_gateway.key
TLS_CA_FILE=/etc/foxhunt/certs/ca.crt

# Logging
RUST_LOG=info,api_gateway=debug,tower_http=debug
```

### Trading Service Environment Variables

Create `/etc/foxhunt/trading_service/.env`:

```bash
# Database
DATABASE_URL=postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production

# Redis (Kill Switch)
REDIS_URL=redis://:PASSWORD@localhost:6379/1

# gRPC
GRPC_BIND_ADDR=0.0.0.0:50052
HEALTH_PORT=8081

# TLS
TLS_CERT_FILE=/etc/foxhunt/trading_service/trading_service.crt
TLS_KEY_FILE=/etc/foxhunt/trading_service/trading_service.key
TLS_CA_FILE=/etc/foxhunt/certs/ca.crt

# Performance
ENVIRONMENT=production
ENABLE_PERFORMANCE_MONITORING=true

# Logging
RUST_LOG=info,trading_service=debug
```

### Backtesting Service Environment Variables

Create `/etc/foxhunt/backtesting_service/.env`:

```bash
# Database
DATABASE_URL=postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production

# gRPC
GRPC_BIND_ADDR=0.0.0.0:50053
HEALTH_PORT=8082

# TLS
TLS_CERT_FILE=/etc/foxhunt/backtesting_service/backtesting_service.crt
TLS_KEY_FILE=/etc/foxhunt/backtesting_service/backtesting_service.key
TLS_CA_FILE=/etc/foxhunt/certs/ca.crt

# Logging
RUST_LOG=info,backtesting_service=debug
```

### ML Training Service Environment Variables

Create `/etc/foxhunt/ml_training_service/.env`:

```bash
# Database
DATABASE_URL=postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production

# S3 Configuration
S3_BUCKET=foxhunt-models
S3_REGION=us-east-1
AWS_ACCESS_KEY_ID=REPLACE_WITH_IAM_KEY
AWS_SECRET_ACCESS_KEY=REPLACE_WITH_IAM_SECRET

# Model Cache
MODEL_CACHE_DIR=/cache/models
MODEL_CACHE_SIZE_GB=100

# gRPC
GRPC_BIND_ADDR=0.0.0.0:50054
HEALTH_PORT=8083

# TLS
TLS_CERT_FILE=/etc/foxhunt/ml_training_service/ml_training_service.crt
TLS_KEY_FILE=/etc/foxhunt/ml_training_service/ml_training_service.key
TLS_CA_FILE=/etc/foxhunt/certs/ca.crt

# GPU
CUDA_VISIBLE_DEVICES=0,1

# Logging
RUST_LOG=info,ml_training_service=debug
```

---

## Service Deployment

### Build Release Binaries

```bash
cd /home/jgrusewski/Work/foxhunt

# Build all services in release mode with LTO
cargo build --release --workspace

# Binaries will be in target/release/:
# - api_gateway
# - trading_service
# - backtesting_service
# - ml_training_service
# - tli
```

### Deploy API Gateway

```bash
# Create service user
sudo useradd -r -s /bin/false foxhunt

# Copy binary
sudo cp target/release/api_gateway /usr/local/bin/
sudo chmod 755 /usr/local/bin/api_gateway
sudo chown foxhunt:foxhunt /usr/local/bin/api_gateway

# Create systemd service
sudo tee /etc/systemd/system/api_gateway.service <<EOF
[Unit]
Description=Foxhunt API Gateway
After=network.target postgresql.service redis.service
Requires=postgresql.service redis.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
EnvironmentFile=/etc/foxhunt/api_gateway/.env
ExecStart=/usr/local/bin/api_gateway
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=api_gateway

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/foxhunt

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable api_gateway
sudo systemctl start api_gateway

# Check status
sudo systemctl status api_gateway
sudo journalctl -u api_gateway -f
```

### Deploy Trading Service

```bash
# Copy binary
sudo cp target/release/trading_service /usr/local/bin/
sudo chmod 755 /usr/local/bin/trading_service
sudo chown foxhunt:foxhunt /usr/local/bin/trading_service

# Create systemd service
sudo tee /etc/systemd/system/trading_service.service <<EOF
[Unit]
Description=Foxhunt Trading Service
After=network.target postgresql.service redis.service
Requires=postgresql.service redis.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
EnvironmentFile=/etc/foxhunt/trading_service/.env
ExecStart=/usr/local/bin/trading_service
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=trading_service

# Performance tuning
CPUAffinity=0-7
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=99
LimitNOFILE=1048576

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable trading_service
sudo systemctl start trading_service

# Check status
sudo systemctl status trading_service
```

### Deploy Backtesting Service

```bash
# Copy binary
sudo cp target/release/backtesting_service /usr/local/bin/
sudo chmod 755 /usr/local/bin/backtesting_service
sudo chown foxhunt:foxhunt /usr/local/bin/backtesting_service

# Create systemd service
sudo tee /etc/systemd/system/backtesting_service.service <<EOF
[Unit]
Description=Foxhunt Backtesting Service
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
EnvironmentFile=/etc/foxhunt/backtesting_service/.env
ExecStart=/usr/local/bin/backtesting_service
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=backtesting_service

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable backtesting_service
sudo systemctl start backtesting_service
```

### Deploy ML Training Service

```bash
# Copy binary
sudo cp target/release/ml_training_service /usr/local/bin/
sudo chmod 755 /usr/local/bin/ml_training_service
sudo chown foxhunt:foxhunt /usr/local/bin/ml_training_service

# Create model cache directory
sudo mkdir -p /cache/models
sudo chown foxhunt:foxhunt /cache/models

# Create systemd service
sudo tee /etc/systemd/system/ml_training_service.service <<EOF
[Unit]
Description=Foxhunt ML Training Service
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
EnvironmentFile=/etc/foxhunt/ml_training_service/.env
ExecStart=/usr/local/bin/ml_training_service
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=ml_training_service

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable ml_training_service
sudo systemctl start ml_training_service
```

---

## Post-Deployment Verification

### 1. Health Check All Services

```bash
#!/bin/bash
# health_check.sh

echo "=== Foxhunt Production Health Check ==="
echo ""

# API Gateway
echo "1. API Gateway Health..."
curl -s http://localhost:8080/health || echo "❌ API Gateway FAILED"

# Trading Service
echo "2. Trading Service Health..."
curl -s http://localhost:8081/health || echo "❌ Trading Service FAILED"

# Backtesting Service
echo "3. Backtesting Service Health..."
curl -s http://localhost:8082/health || echo "❌ Backtesting Service FAILED"

# ML Training Service
echo "4. ML Training Service Health..."
curl -s http://localhost:8083/health || echo "❌ ML Training Service FAILED"

# PostgreSQL
echo "5. PostgreSQL..."
psql $DATABASE_URL -c "SELECT version();" || echo "❌ PostgreSQL FAILED"

# Redis
echo "6. Redis..."
redis-cli -a $REDIS_PASSWORD ping || echo "❌ Redis FAILED"

echo ""
echo "=== Health Check Complete ==="
```

### 2. Verify Service Logs

```bash
# Check for errors in all services
sudo journalctl -u api_gateway -u trading_service -u backtesting_service -u ml_training_service --since "5 minutes ago" | grep -i error

# Verify services are listening on correct ports
sudo netstat -tlnp | grep -E "(50051|50052|50053|50054|8080|8081|8082|8083)"
```

### 3. Connectivity Tests

**Test mTLS Between Services**:

```bash
# From TLI, test connection to API Gateway
cd /home/jgrusewski/Work/foxhunt
./target/release/tli --help  # Should connect successfully
```

**Test Database Connections**:

```bash
# Verify each service can connect to PostgreSQL
sudo -u foxhunt psql $DATABASE_URL -c "SELECT 'Connection successful';"
```

### 4. Smoke Tests

**Test API Gateway Authentication**:

```bash
# Generate JWT token (requires admin credentials)
curl -X POST http://localhost:50051/auth/login \
    -H "Content-Type: application/json" \
    -d '{"username":"admin","password":"ADMIN_PASSWORD"}'

# Use token to access protected endpoint
export TOKEN="eyJ..."
curl -H "Authorization: Bearer $TOKEN" http://localhost:50051/trading/portfolio
```

**Test Order Submission (Trading Service)**:

```bash
# Submit test order via API Gateway
curl -X POST http://localhost:50051/trading/order \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "symbol": "AAPL",
        "side": "buy",
        "quantity": 100,
        "order_type": "limit",
        "limit_price": 150.0
    }'
```

### 5. Performance Baselines

**Measure API Gateway Latency**:

```bash
# Install wrk for load testing
sudo apt install -y wrk

# Test authentication overhead
wrk -t4 -c100 -d30s --latency http://localhost:50051/health

# Expected: p99 latency < 10ms for health endpoint
```

**Measure Database Query Performance**:

```sql
-- Enable query timing
\timing on

-- Test critical queries
SELECT * FROM users LIMIT 10;
SELECT * FROM permissions WHERE endpoint = 'trading.submit_order';
```

---

## Performance Tuning

### PostgreSQL Connection Pooling

**Adjust per-service connection limits**:

```sql
-- Check current connections
SELECT count(*), application_name FROM pg_stat_activity GROUP BY application_name;

-- If connections are exhausted, increase max_connections in postgresql.conf
-- Or implement PgBouncer for connection pooling
```

**Deploy PgBouncer (Recommended for HFT)**:

```ini
# /etc/pgbouncer/pgbouncer.ini
[databases]
foxhunt_production = host=localhost port=5432 dbname=foxhunt_production

[pgbouncer]
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20
```

Update service `DATABASE_URL` to use PgBouncer: `postgresql://foxhunt_user:PASSWORD@localhost:6432/foxhunt_production`

### Redis Memory Optimization

```bash
# Monitor Redis memory usage
redis-cli -a $REDIS_PASSWORD INFO memory

# If memory usage is high, tune maxmemory-policy
redis-cli -a $REDIS_PASSWORD CONFIG SET maxmemory-policy allkeys-lfu
```

### gRPC Thread Pool Configuration

**Edit service code to adjust Tokio runtime**:

```rust
// In main.rs
#[tokio::main(worker_threads = 16)]  // Adjust based on CPU cores
async fn main() -> Result<()> {
    // ...
}
```

Rebuild and redeploy services after tuning.

### TCP Kernel Tuning (Already applied in Prerequisites)

Verify tuning is active:

```bash
sysctl net.ipv4.tcp_congestion_control  # Should be 'bbr'
sysctl net.core.rmem_max                # Should be 134217728
```

---

## Disaster Recovery

### Backup Strategy

**PostgreSQL Backups**:

```bash
# Daily full backup
sudo -u postgres pg_dump foxhunt_production > /backups/foxhunt_$(date +%Y%m%d).sql

# Compress and upload to S3
gzip /backups/foxhunt_$(date +%Y%m%d).sql
aws s3 cp /backups/foxhunt_$(date +%Y%m%d).sql.gz s3://foxhunt-backups/postgres/

# Retention: Keep last 30 days locally, 7 years in S3
```

**Redis Backups**:

```bash
# Trigger RDB snapshot
redis-cli -a $REDIS_PASSWORD BGSAVE

# Copy RDB file to backup location
sudo cp /var/lib/redis/dump.rdb /backups/redis_$(date +%Y%m%d).rdb
```

**Configuration Backups**:

```bash
# Backup all .env files and certificates
tar -czf /backups/config_$(date +%Y%m%d).tar.gz /etc/foxhunt/
```

### Restore Procedures

**Restore PostgreSQL**:

```bash
# Stop services
sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

# Drop and recreate database
sudo -u postgres psql <<EOF
DROP DATABASE foxhunt_production;
CREATE DATABASE foxhunt_production OWNER foxhunt_user;
EOF

# Restore from backup
gunzip -c /backups/foxhunt_20251003.sql.gz | sudo -u postgres psql foxhunt_production

# Restart services
sudo systemctl start api_gateway trading_service backtesting_service ml_training_service
```

**Restore Redis**:

```bash
sudo systemctl stop redis-server
sudo cp /backups/redis_20251003.rdb /var/lib/redis/dump.rdb
sudo chown redis:redis /var/lib/redis/dump.rdb
sudo systemctl start redis-server
```

### Recovery Time Objective (RTO) & Recovery Point Objective (RPO)

| Component | RTO | RPO |
|-----------|-----|-----|
| API Gateway | 5 minutes | N/A (stateless) |
| Trading Service | 5 minutes | N/A (stateless) |
| PostgreSQL | 30 minutes | 24 hours (daily backup) |
| Redis | 10 minutes | 1 hour (AOF) |

---

## Rollback Procedures

### Service Rollback

```bash
# Stop new version
sudo systemctl stop api_gateway

# Copy previous binary (assumes you keep old versions)
sudo cp /usr/local/bin/api_gateway.bak /usr/local/bin/api_gateway

# Start old version
sudo systemctl start api_gateway

# Verify
sudo systemctl status api_gateway
curl http://localhost:8080/health
```

### Database Rollback

**⚠️ WARNING**: Database rollbacks are complex. Test thoroughly in staging.

```bash
# If new migration broke things, restore from pre-migration backup
sudo -u postgres psql foxhunt_production < /backups/pre_migration_backup.sql
```

**For forward-only migrations** (recommended), write compensating migrations instead of rolling back.

---

## Production Deployment Pitfalls

### ⚠️ CRITICAL WARNINGS

#### 1. Inadequate Testing Environments
**Pitfall**: Deploying to production without a staging environment that mirrors production (data volume, network conditions, resource limits).

**Impact**: Production surprises, unexpected latency, crashes under load.

**Mitigation**:
- Deploy identical staging environment with realistic data volumes
- Load test staging with production-level traffic (100K req/s)
- Measure latency baselines in staging before production

#### 2. Secrets Management Mishaps
**Pitfall**: Hardcoded secrets, plain text files, lack of automated rotation.

**Impact**: Security breach, compromised system.

**Mitigation**:
- Never hardcode secrets in code or config files
- Use HashiCorp Vault or AWS Secrets Manager
- Implement automated secret rotation (JWT secrets every 90 days)
- Audit secret access regularly

#### 3. Time Synchronization Issues (NTP/PTP)
**Pitfall**: Relying on default OS time sync or inaccurate NTP sources.

**Impact**: Incorrect trade timestamps, regulatory violations (MiFID II), audit failures.

**Mitigation**:
- Deploy chrony with financial market time servers
- For sub-microsecond accuracy, use PTP with hardware support
- Monitor time drift continuously (alert if offset > 100μs)
- Verify time sync before every trading session

#### 4. Network Latency & Jitter
**Pitfall**: Overlooking network topology, physical proximity to exchanges, internal network hops.

**Impact**: Increased order latency, missed trading opportunities, competitive disadvantage.

**Mitigation**:
- Collocate infrastructure near exchanges
- Use dedicated network infrastructure (no shared VLANs)
- Implement kernel bypass (DPDK) for ultra-low latency
- Measure and monitor network jitter continuously

#### 5. Insufficient Resource Allocation
**Pitfall**: Under-provisioning CPU, memory, network bandwidth, or improper OS/kernel tuning.

**Impact**: High latency, service crashes, data loss.

**Mitigation**:
- Follow hardware requirements strictly (32+ cores, 128GB+ RAM)
- Apply OS-level tuning (see Prerequisites section)
- Reserve CPU cores for critical trading threads (CPU affinity)
- Monitor resource usage and scale proactively

#### 6. Observability Gaps
**Pitfall**: Deploying without comprehensive logging, metrics, distributed tracing.

**Impact**: Inability to diagnose production issues, extended downtime.

**Mitigation**:
- Deploy Prometheus + Grafana for metrics
- Deploy ELK stack or Loki for centralized logging
- Implement distributed tracing (OpenTelemetry)
- Create dashboards for critical KPIs (latency, throughput, error rate)

#### 7. Certificate Management Overheads
**Pitfall**: Expired mTLS certificates, difficult rotation, lack of automation.

**Impact**: Service outages, broken inter-service communication.

**Mitigation**:
- Implement automated certificate renewal (cert-manager, Let's Encrypt)
- Set up alerts for expiring certificates (30 days before expiry)
- Test certificate rotation in staging regularly
- Document manual renewal procedures for emergencies

#### 8. Database/Redis Misconfiguration
**Pitfall**: Lack of HA, improper caching strategies, insufficient connection pooling.

**Impact**: Service bottlenecks, data loss, downtime.

**Mitigation**:
- Deploy PostgreSQL streaming replication (async or sync)
- Deploy Redis Sentinel or Cluster for HA
- Use PgBouncer for connection pooling
- Tune PostgreSQL `shared_buffers`, `work_mem` based on workload

#### 9. Compliance Blind Spots
**Pitfall**: Not fully addressing SOX/MiFID II requirements for audit trails, data retention, access control.

**Impact**: Regulatory fines, legal liability, loss of trading license.

**Mitigation**:
- Enable comprehensive audit logging (all trades, logins, config changes)
- Implement immutable audit trails (PostgreSQL with no DELETE)
- Define data retention policies (7 years for audit trails)
- Conduct regular compliance audits

#### 10. Poor Rollback Strategy
**Pitfall**: Inability to quickly and safely revert to a previous stable state.

**Impact**: Extended downtime during deployment failures.

**Mitigation**:
- Keep previous binary versions (`api_gateway.bak`)
- Backup database before migrations (`pg_dump`)
- Test rollback procedures in staging monthly
- Document rollback steps in runbook

#### 11. Rust-Specific Pitfalls
**Pitfall**: Not building with `--release`, missing LTO, unreviewed `unsafe` blocks.

**Impact**: Slower performance, larger binaries, memory safety issues.

**Mitigation**:
- Always build with `cargo build --release` for production
- Enable LTO in `Cargo.toml`: `lto = "fat"`
- Strictly review any `unsafe` blocks (should be rare)
- Run `cargo clippy` and `cargo audit` in CI/CD

#### 12. JWT Secret Entropy
**Pitfall**: Using weak JWT secrets (< 32 bytes) or predictable values.

**Impact**: JWT forgery, unauthorized access.

**Mitigation**:
- Generate JWT secrets with `openssl rand -base64 64`
- Rotate JWT secrets every 90 days
- Never share JWT secrets across environments

---

## Appendix: Quick Reference

### Service Ports

| Service | gRPC Port | Health Port |
|---------|-----------|-------------|
| API Gateway | 50051 | 8080 |
| Trading Service | 50052 | 8081 |
| Backtesting Service | 50053 | 8082 |
| ML Training Service | 50054 | 8083 |

### Important File Locations

| Component | Path |
|-----------|------|
| Binaries | `/usr/local/bin/` |
| Config | `/etc/foxhunt/{service}/.env` |
| Certificates | `/etc/foxhunt/certs/` |
| Logs | `/var/log/foxhunt/` |
| Backups | `/backups/` |
| Model Cache | `/cache/models/` |

### Emergency Contacts

- **Trading Operations Lead**: [PagerDuty rotation]
- **Database Administrator**: [PagerDuty rotation]
- **Security Team**: security@foxhunt.local
- **On-Call Engineer**: [PagerDuty rotation]

---

**End of Production Deployment Guide**

*This document should be reviewed quarterly and updated after every major deployment.*