foxhunt/docs/PRODUCTION_DEPLOYMENT_GUIDE_V2.md

Foxhunt HFT Trading System - Production Deployment Guide

Version: 2.0 Last Updated: 2025-10-03 Wave: 71 - Complete Production Stack Status: Production Ready with Full Security Stack

---

Table of Contents

1. Executive Summary
2. System Architecture Overview
3. Prerequisites
4. Pre-Deployment Checklist
5. Infrastructure Setup
6. Database Migration
7. TLS Certificate Configuration
8. Environment Configuration
9. Service Deployment
10. Post-Deployment Verification
11. Performance Tuning
12. Disaster Recovery
13. Rollback Procedures
14. Production Deployment Pitfalls

---

Executive Summary

This guide provides a complete, step-by-step procedure for deploying the Foxhunt HFT Trading System to production. The system consists of 4 main services with comprehensive security features designed for SOX/MiFID II compliance.

Production Readiness Status

✅ Production Ready Components:

• API Gateway (6-layer authentication, rate limiting, audit logging)
• Trading Service (order execution, portfolio management, kill switch)
• Backtesting Service (strategy validation)
• ML Training Service (model lifecycle management)
• TLI (terminal client interface)

✅ Security Features:

• JWT authentication with revocation (Redis-backed)
• MFA/TOTP implementation (RFC 6238)
• RBAC with granular permissions
• Mutual TLS between services
• SQL injection protection (parameterized queries)
• Comprehensive audit trails (SOX/MiFID II)

✅ Infrastructure:

• PostgreSQL 16+ with NOTIFY/LISTEN for hot-reload
• Redis 7+ for JWT revocation and rate limiting
• S3 integration for ML model storage
• Docker deployment configurations

Deployment Timeline

Total Deployment Time: 4-6 hours (first deployment)

Phase	Duration	Status Gate
Infrastructure Setup	60 min	PostgreSQL + Redis operational
Database Migration	30 min	All migrations applied
Certificate Generation	45 min	CA + service certs ready
Service Deployment	90 min	All health checks passing
Verification	60 min	Smoke tests complete
Performance Tuning	60 min	Latency baselines measured

---

System Architecture Overview

Service Topology

┌─────────────────────────────────────────────────────────────────────────┐
│                         FOXHUNT HFT SYSTEM                              │
│                      Production Architecture                            │
└─────────────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────────────┐
│                          CLIENT LAYER                                    │
├──────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │  TLI (Terminal Client)                                            │  │
│  │  - Ratatui terminal UI                                            │  │
│  │  - gRPC client to 3 services                                      │  │
│  │  - JWT authentication                                             │  │
│  └─────────────────────┬────────────────────────────────────────────┘  │
│                        │                                                │
└────────────────────────┼────────────────────────────────────────────────┘
                         │
┌────────────────────────┼────────────────────────────────────────────────┐
│                        │         API GATEWAY LAYER                      │
├────────────────────────┼────────────────────────────────────────────────┤
│                        │                                                │
│  ┌─────────────────────▼──────────────────────────────────────────┐   │
│  │  API Gateway (0.0.0.0:50051)                                    │   │
│  │  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  │   │
│  │  6-Layer Security Stack:                                        │   │
│  │  1. JWT Validation (HS512, cached decoding key)                 │   │
│  │  2. JWT Revocation Check (Redis, <100μs)                        │   │
│  │  3. MFA/TOTP Validation (RFC 6238)                              │   │
│  │  4. RBAC Authorization (cached permissions)                     │   │
│  │  5. Rate Limiting (100 req/s per user)                          │   │
│  │  6. Audit Logging (SOX/MiFID II compliance)                     │   │
│  │  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  │   │
│  │  Performance: <10μs authentication overhead                     │   │
│  └──────────┬──────────────┬──────────────┬────────────────────────┘   │
│             │              │              │                            │
└─────────────┼──────────────┼──────────────┼────────────────────────────┘
              │              │              │
┌─────────────┼──────────────┼──────────────┼────────────────────────────┐
│             │              │              │   BACKEND SERVICES          │
├─────────────┼──────────────┼──────────────┼────────────────────────────┤
│             │              │              │                            │
│  ┌──────────▼─────────┐ ┌─▼────────────┐ ┌▼──────────────────────┐   │
│  │ Trading Service    │ │ Backtesting  │ │ ML Training Service   │   │
│  │ (50052)            │ │ Service      │ │ (50054)               │   │
│  │                    │ │ (50053)      │ │                       │   │
│  │ • Order execution  │ │              │ │ • Model training      │   │
│  │ • Portfolio mgmt   │ │ • Strategy   │ │ • S3 model storage    │   │
│  │ • Risk checks      │ │   validation │ │ • Version management  │   │
│  │ • Kill switch      │ │ • Historical │ │ • Metrics collection  │   │
│  │ • Compliance       │ │   replay     │ │                       │   │
│  └──────────┬─────────┘ └─┬────────────┘ └┬──────────────────────┘   │
│             │              │              │                            │
│             └──────────────┴──────────────┘                            │
│                            │                                           │
└────────────────────────────┼───────────────────────────────────────────┘
                             │
┌────────────────────────────┼───────────────────────────────────────────┐
│                            │      INFRASTRUCTURE LAYER                 │
├────────────────────────────┼───────────────────────────────────────────┤
│                            │                                           │
│  ┌─────────────────────────▼────────────────────────────────────┐    │
│  │  PostgreSQL 16+ Cluster (Primary + Replica)                  │    │
│  │  • Streaming replication (async)                             │    │
│  │  • NOTIFY/LISTEN for config hot-reload                       │    │
│  │  • Connection pooling (20 max per service)                   │    │
│  │  • Schema: users, roles, permissions, mfa_config, audit      │    │
│  └──────────────────────────────────────────────────────────────┘    │
│                                                                        │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  Redis 7+ Cluster (Sentinel/Cluster Mode)                    │    │
│  │  • JWT revocation blacklist                                  │    │
│  │  • Rate limiting (token bucket)                              │    │
│  │  • AOF persistence                                           │    │
│  └──────────────────────────────────────────────────────────────┘    │
│                                                                        │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  S3 (ML Model Storage)                                       │    │
│  │  • Server-side encryption (SSE-S3)                           │    │
│  │  • Model versioning                                          │    │
│  │  • Local cache (/cache/models/)                              │    │
│  └──────────────────────────────────────────────────────────────┘    │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────────────┐
│                       MONITORING & OBSERVABILITY                        │
├────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  Prometheus → Grafana → PagerDuty/Slack                                │
│  • Service metrics (latency, throughput, errors)                       │
│  • Infrastructure metrics (CPU, memory, disk, network)                 │
│  • Business metrics (orders/sec, fill rate, P&L)                       │
│                                                                         │
│  ELK Stack / Loki (Centralized Logging)                                │
│  • All service logs aggregated                                         │
│  • Audit trail immutability                                            │
│  • Security event monitoring                                           │
│                                                                         │
└────────────────────────────────────────────────────────────────────────┘

Network Topology

┌─────────────────────────────────────────────────────────────────────┐
│                         NETWORK SEGMENTATION                        │
└─────────────────────────────────────────────────────────────────────┘

Internet/Exchanges
      │
      │ TLS 1.3
      ▼
┌──────────────────┐
│  Firewall/WAF    │
└────────┬─────────┘
         │
         ▼
┌─────────────────────────────────────────────┐
│  DMZ (Public Zone)                          │
│  ┌──────────────────┐                       │
│  │  API Gateway     │ 0.0.0.0:50051 (gRPC) │
│  │  Health: 8080    │ (HTTP health check)  │
│  └────────┬─────────┘                       │
└───────────┼──────────────────────────────────┘
            │ mTLS
            ▼
┌─────────────────────────────────────────────┐
│  Application Zone (Private)                 │
│  ┌──────────────┐  ┌──────────────┐        │
│  │ Trading Svc  │  │ Backtesting  │        │
│  │ :50052       │  │ Svc :50053   │        │
│  └──────┬───────┘  └──────┬───────┘        │
│         │                 │                 │
│  ┌──────▼─────────────────▼───────┐        │
│  │   ML Training Service           │        │
│  │   :50054                        │        │
│  └──────┬──────────────────────────┘        │
└─────────┼──────────────────────────────────┘
          │
          ▼
┌─────────────────────────────────────────────┐
│  Data Zone (Restricted)                     │
│  ┌───────────────┐  ┌──────────────┐       │
│  │ PostgreSQL    │  │ Redis        │       │
│  │ :5432         │  │ :6379        │       │
│  └───────────────┘  └──────────────┘       │
│                                             │
│  ┌──────────────────────────────────┐      │
│  │ S3 (via VPC Endpoint)            │      │
│  │ s3://foxhunt-models/             │      │
│  └──────────────────────────────────┘      │
└─────────────────────────────────────────────┘

---

Prerequisites

Hardware Requirements

API Gateway + Backend Services (Single Host Deployment)

Minimum:

CPU: Intel Xeon Gold 6248R (24 cores @ 3.0GHz) OR AMD EPYC 7543 (32 cores @ 2.8GHz)
RAM: 128GB DDR4-3200 ECC
Storage: 2TB NVMe SSD (Samsung 980 PRO, WD Black SN850)
  - Write latency: <100μs (99.9th percentile)
  - Sequential read: >6000 MB/s
Network: 25Gbps NIC (Mellanox ConnectX-6, Intel E810)
  - Sub-1ms latency to exchange colocations
  - PTP/NTP for time synchronization (±100μs accuracy)

Recommended (Distributed Deployment):

API Gateway Host:
  CPU: 16 cores @ 3.5GHz+
  RAM: 64GB
  Network: 10Gbps dedicated

Trading Service Host:
  CPU: 32 cores @ 3.5GHz+ (CPU affinity for critical threads)
  RAM: 128GB
  Network: 25Gbps, kernel bypass (DPDK)

ML Training Service Host:
  GPU: 2x NVIDIA A100 80GB OR 1x H100 80GB
  CPU: 32 cores
  RAM: 256GB
  Storage: 4TB NVMe (model storage cache)

Database/Redis Host:
  CPU: 24 cores
  RAM: 256GB (large PostgreSQL shared_buffers)
  Storage: 4TB NVMe in RAID 10

Operating System

Recommended: Ubuntu 22.04 LTS (Kernel 6.2+)

OS-Level Hardening (Critical for HFT):

# Real-time kernel for low latency
sudo apt install linux-image-lowlatency

# Network tuning
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

# File descriptor limits
echo "* soft nofile 1048576" | sudo tee -a /etc/security/limits.conf
echo "* hard nofile 1048576" | sudo tee -a /etc/security/limits.conf

# CPU isolation (replace 0-7 with CPU cores to isolate)
# Edit /etc/default/grub:
# GRUB_CMDLINE_LINUX="isolcpus=0-7 nohz_full=0-7 rcu_nocbs=0-7"
sudo update-grub

Software Dependencies

# Update system
sudo apt update && sudo apt upgrade -y

# Essential build tools
sudo apt install -y \
    build-essential \
    cmake \
    pkg-config \
    libssl-dev \
    libpq-dev \
    protobuf-compiler \
    git \
    curl \
    wget

# PostgreSQL 16
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt update
sudo apt install -y postgresql-16 postgresql-client-16

# Redis 7
sudo apt install -y redis-server

# Docker (optional, for containerized deployment)
sudo apt install -y docker.io docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Rust toolchain (1.75+)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source ~/.cargo/env
rustup default stable

# Verify versions
cargo --version  # Should be 1.75.0+
psql --version   # Should be 16.x
redis-server --version  # Should be 7.x

Time Synchronization (CRITICAL for HFT)

⚠️ WARNING: Skewed clocks can cause incorrect trades, regulatory violations (MiFID II), and audit failures.

# Install chrony (more accurate than ntpd)
sudo apt install -y chrony

# Configure chrony for financial market time sync
sudo tee /etc/chrony/chrony.conf <<EOF
# Use NIST time servers (replace with your region's time servers)
server time.nist.gov iburst
server time-a-g.nist.gov iburst
server time-b-g.nist.gov iburst

# For extreme accuracy, use PTP (Precision Time Protocol) hardware
# This requires PTP-capable NICs and switches

# Allow large time adjustments (be cautious in production)
makestep 1.0 3

# Log statistics
logdir /var/log/chrony
EOF

sudo systemctl restart chrony

# Verify time sync
chronyc tracking
# Expected: System time offset < 100μs for HFT

PTP Configuration (for sub-microsecond accuracy):

# Install linuxptp
sudo apt install -y linuxptp

# Configure PTP on your network interface (replace ens192)
sudo ptp4l -i ens192 -m -s -H &
sudo phc2sys -s ens192 -m -w &

---

Pre-Deployment Checklist

Security Preparation

• Secrets Manager deployed (HashiCorp Vault, AWS Secrets Manager, etc.)
• JWT Secret generated (64+ bytes of cryptographic randomness)

  openssl rand -base64 64 > /secrets/jwt_secret.key
  chmod 600 /secrets/jwt_secret.key
  

• Database Credentials stored in secrets manager
• Redis Password generated and stored
• S3 Access Keys created with minimal IAM permissions
• TLS CA Certificate ready (or use Let's Encrypt)
• Firewall Rules defined (see Network Topology)

Infrastructure Validation

• PostgreSQL 16+ installed and accessible
• Redis 7+ installed and accessible
• S3 Bucket created with versioning enabled
• DNS Records configured (if using domain names)
• NTP/PTP synchronized (time offset < 100μs)
• Network Connectivity tested (ping exchanges, databases)
• Disk Space available (minimum 500GB free)

Compliance Documentation

• SOX Compliance checklist reviewed
• MiFID II timestamping requirements understood
• Audit Trail retention policy defined (7 years recommended)
• Disaster Recovery plan documented
• Incident Response plan reviewed

---

Infrastructure Setup

PostgreSQL Cluster Setup

1. Primary Database Configuration

# Create production database and user
sudo -u postgres psql <<EOF
CREATE USER foxhunt_user WITH PASSWORD 'REPLACE_WITH_VAULT_PASSWORD';
CREATE DATABASE foxhunt_production OWNER foxhunt_user;

-- Grant necessary permissions
GRANT ALL PRIVILEGES ON DATABASE foxhunt_production TO foxhunt_user;

-- Enable required extensions
\c foxhunt_production
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements";

-- Grant extension usage
GRANT ALL ON SCHEMA public TO foxhunt_user;
EOF

2. PostgreSQL Performance Tuning

Edit /etc/postgresql/16/main/postgresql.conf:

# Connection Settings
max_connections = 200
superuser_reserved_connections = 3

# Memory Settings (adjust based on available RAM)
shared_buffers = 32GB              # 25% of RAM for 128GB system
effective_cache_size = 96GB        # 75% of RAM
work_mem = 64MB                    # Per-operation memory
maintenance_work_mem = 2GB

# WAL Settings (for durability and replication)
wal_level = replica
wal_buffers = 16MB
max_wal_size = 4GB
min_wal_size = 1GB
checkpoint_completion_target = 0.9

# Query Planner
random_page_cost = 1.1             # For NVMe SSDs
effective_io_concurrency = 200

# Logging (for audit trails)
logging_collector = on
log_directory = '/var/log/postgresql'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_statement = 'mod'              # Log all DDL and DML
log_duration = on
log_line_prefix = '%t [%p]: user=%u,db=%d,app=%a,client=%h '

# NOTIFY/LISTEN for Config Hot-Reload
listen_addresses = '*'

Edit /etc/postgresql/16/main/pg_hba.conf (authentication):

# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   all             postgres                                peer
host    foxhunt_production  foxhunt_user    10.0.0.0/8          scram-sha-256
host    foxhunt_production  foxhunt_user    127.0.0.1/32        scram-sha-256

Restart PostgreSQL:

sudo systemctl restart postgresql
sudo systemctl enable postgresql

3. Streaming Replication (Optional but Recommended)

On Replica Host:

# Stop PostgreSQL on replica
sudo systemctl stop postgresql

# Create base backup from primary
sudo -u postgres pg_basebackup -h PRIMARY_HOST -D /var/lib/postgresql/16/main -U replication -P -v -R

# Start replica
sudo systemctl start postgresql

Verify Replication:

-- On primary
SELECT * FROM pg_stat_replication;

-- On replica
SELECT pg_is_in_recovery();  -- Should return 't'

Redis Cluster Setup

1. Redis Configuration

Edit /etc/redis/redis.conf:

# Network
bind 0.0.0.0
protected-mode yes
port 6379
requirepass REPLACE_WITH_VAULT_PASSWORD

# Memory
maxmemory 16gb
maxmemory-policy allkeys-lru

# Persistence (for JWT revocation durability)
save 900 1
save 300 10
save 60 10000
appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# Performance
tcp-backlog 511
timeout 0
tcp-keepalive 300

# Logging
loglevel notice
logfile /var/log/redis/redis-server.log

Restart Redis:

sudo systemctl restart redis-server
sudo systemctl enable redis-server

2. Redis Sentinel (High Availability)

For production, deploy Redis Sentinel for automatic failover:

# sentinel.conf
sentinel monitor foxhunt-redis MASTER_HOST 6379 2
sentinel auth-pass foxhunt-redis REPLACE_WITH_VAULT_PASSWORD
sentinel down-after-milliseconds foxhunt-redis 5000
sentinel failover-timeout foxhunt-redis 10000
sentinel parallel-syncs foxhunt-redis 1

Start Sentinel:

redis-sentinel /etc/redis/sentinel.conf &

---

Database Migration

Migration Files

The following migrations must be applied in order:

cd /home/jgrusewski/Work/foxhunt/database/migrations

# List all migrations
ls -1 *.sql | sort

Migration Order:

1. 009_security_api_keys.sql - API key management
2. 010_compliance_audit_trails.sql - Audit trail tables
3. 011_compliance_rules_dynamic.sql - Dynamic compliance rules
4. 015_adaptive_strategy_config.sql - Strategy configuration
5. 016_adaptive_strategy_seed_data.sql - Seed data
6. 016_ml_training_data_tables.sql - ML training tables
7. 017_mfa_totp_implementation.sql - MFA/TOTP (CRITICAL)
8. 018_rbac_permissions.sql - RBAC (CRITICAL)
9. 018_config_management_system.sql - Config hot-reload
10. 019_config_notify_triggers.sql - PostgreSQL NOTIFY triggers (CRITICAL)

Apply Migrations

# Set database connection
export DATABASE_URL="postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production"

# Apply all migrations in order
cd /home/jgrusewski/Work/foxhunt/database/migrations

psql $DATABASE_URL -f 009_security_api_keys.sql
psql $DATABASE_URL -f 010_compliance_audit_trails.sql
psql $DATABASE_URL -f 011_compliance_rules_dynamic.sql
psql $DATABASE_URL -f 015_adaptive_strategy_config.sql
psql $DATABASE_URL -f 016_adaptive_strategy_seed_data.sql
psql $DATABASE_URL -f 016_ml_training_data_tables.sql
psql $DATABASE_URL -f 017_mfa_totp_implementation.sql
psql $DATABASE_URL -f 018_rbac_permissions.sql
psql $DATABASE_URL -f 018_config_management_system.sql
psql $DATABASE_URL -f 019_config_notify_triggers.sql

# Verify migrations
psql $DATABASE_URL -c "\dt"  # List all tables

Verify Critical Tables

-- Verify RBAC tables
SELECT * FROM roles;
SELECT * FROM permissions;

-- Verify MFA tables
SELECT * FROM mfa_config LIMIT 5;

-- Verify audit trail
SELECT * FROM audit_events ORDER BY created_at DESC LIMIT 10;

Seed Initial Data

# Create default admin user and roles
psql $DATABASE_URL <<EOF
-- Create admin role
INSERT INTO roles (name, description) VALUES
    ('admin', 'System administrator with full access'),
    ('trader', 'Trading desk user with order execution rights'),
    ('viewer', 'Read-only access to dashboards and reports')
ON CONFLICT (name) DO NOTHING;

-- Create default permissions
INSERT INTO permissions (endpoint, description) VALUES
    ('trading.submit_order', 'Submit trading orders'),
    ('trading.cancel_order', 'Cancel existing orders'),
    ('trading.view_portfolio', 'View portfolio positions'),
    ('backtesting.run_test', 'Run backtesting simulations'),
    ('ml.train_model', 'Train machine learning models'),
    ('system.admin', 'System administration access')
ON CONFLICT (endpoint) DO NOTHING;

-- Grant permissions to admin role
INSERT INTO role_permissions (role_id, permission_id)
SELECT r.id, p.id FROM roles r CROSS JOIN permissions p WHERE r.name = 'admin';

-- Create initial admin user (REPLACE PASSWORD HASH)
INSERT INTO users (username, email, password_hash) VALUES
    ('admin', 'admin@foxhunt.local', 'REPLACE_WITH_BCRYPT_HASH')
ON CONFLICT (username) DO NOTHING;

-- Grant admin role to admin user
INSERT INTO user_roles (user_id, role_id)
SELECT u.id, r.id FROM users u, roles r WHERE u.username = 'admin' AND r.name = 'admin';
EOF

---

TLS Certificate Configuration

1. Generate Certificate Authority (CA)

# Create CA directory
mkdir -p /etc/foxhunt/certs
cd /etc/foxhunt/certs

# Generate CA private key
openssl genrsa -out ca.key 4096

# Generate CA certificate (valid for 10 years)
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt \
    -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=Foxhunt Root CA"

# Protect CA key
chmod 400 ca.key
chmod 644 ca.crt

2. Generate Service Certificates

API Gateway Certificate:

# Generate private key
openssl genrsa -out api_gateway.key 4096

# Generate CSR
openssl req -new -key api_gateway.key -out api_gateway.csr \
    -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=api-gateway.foxhunt.local"

# Sign with CA
openssl x509 -req -in api_gateway.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -out api_gateway.crt -days 825 -sha256

# Cleanup
rm api_gateway.csr
chmod 400 api_gateway.key
chmod 644 api_gateway.crt

Repeat for each service:

• trading_service.{key,crt}
• backtesting_service.{key,crt}
• ml_training_service.{key,crt}

3. Client Certificates (for mTLS)

# Generate client key
openssl genrsa -out client.key 4096

# Generate client CSR
openssl req -new -key client.key -out client.csr \
    -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=tli-client"

# Sign with CA
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -out client.crt -days 825 -sha256

# Cleanup
rm client.csr

4. Distribute Certificates

# Copy to service directories
sudo cp ca.crt /etc/foxhunt/certs/
sudo cp api_gateway.{key,crt} /etc/foxhunt/api_gateway/
sudo cp trading_service.{key,crt} /etc/foxhunt/trading_service/
sudo cp backtesting_service.{key,crt} /etc/foxhunt/backtesting_service/
sudo cp ml_training_service.{key,crt} /etc/foxhunt/ml_training_service/

# Set permissions
sudo chown foxhunt:foxhunt /etc/foxhunt/certs/*
sudo chmod 400 /etc/foxhunt/certs/*.key
sudo chmod 644 /etc/foxhunt/certs/*.crt

---

Environment Configuration

API Gateway Environment Variables

Create /etc/foxhunt/api_gateway/.env:

# Network
GATEWAY_BIND_ADDR=0.0.0.0:50051
HEALTH_PORT=8080

# JWT Configuration
JWT_SECRET_FILE=/secrets/jwt_secret.key
JWT_ISSUER=foxhunt-api-gateway
JWT_AUDIENCE=foxhunt-services
JWT_EXPIRY_SECONDS=3600

# Redis (JWT Revocation)
REDIS_URL=redis://:PASSWORD@localhost:6379/0

# Rate Limiting
RATE_LIMIT_RPS=100

# Audit Logging
ENABLE_AUDIT_LOGGING=true
AUDIT_LOG_FILE=/var/log/foxhunt/audit.log

# TLS
TLS_CERT_FILE=/etc/foxhunt/api_gateway/api_gateway.crt
TLS_KEY_FILE=/etc/foxhunt/api_gateway/api_gateway.key
TLS_CA_FILE=/etc/foxhunt/certs/ca.crt

# Logging
RUST_LOG=info,api_gateway=debug,tower_http=debug

Trading Service Environment Variables

Create /etc/foxhunt/trading_service/.env:

# Database
DATABASE_URL=postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production

# Redis (Kill Switch)
REDIS_URL=redis://:PASSWORD@localhost:6379/1

# gRPC
GRPC_BIND_ADDR=0.0.0.0:50052
HEALTH_PORT=8081

# TLS
TLS_CERT_FILE=/etc/foxhunt/trading_service/trading_service.crt
TLS_KEY_FILE=/etc/foxhunt/trading_service/trading_service.key
TLS_CA_FILE=/etc/foxhunt/certs/ca.crt

# Performance
ENVIRONMENT=production
ENABLE_PERFORMANCE_MONITORING=true

# Logging
RUST_LOG=info,trading_service=debug

Backtesting Service Environment Variables

Create /etc/foxhunt/backtesting_service/.env:

# Database
DATABASE_URL=postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production

# gRPC
GRPC_BIND_ADDR=0.0.0.0:50053
HEALTH_PORT=8082

# TLS
TLS_CERT_FILE=/etc/foxhunt/backtesting_service/backtesting_service.crt
TLS_KEY_FILE=/etc/foxhunt/backtesting_service/backtesting_service.key
TLS_CA_FILE=/etc/foxhunt/certs/ca.crt

# Logging
RUST_LOG=info,backtesting_service=debug

ML Training Service Environment Variables

Create /etc/foxhunt/ml_training_service/.env:

# Database
DATABASE_URL=postgresql://foxhunt_user:PASSWORD@localhost:5432/foxhunt_production

# S3 Configuration
S3_BUCKET=foxhunt-models
S3_REGION=us-east-1
AWS_ACCESS_KEY_ID=REPLACE_WITH_IAM_KEY
AWS_SECRET_ACCESS_KEY=REPLACE_WITH_IAM_SECRET

# Model Cache
MODEL_CACHE_DIR=/cache/models
MODEL_CACHE_SIZE_GB=100

# gRPC
GRPC_BIND_ADDR=0.0.0.0:50054
HEALTH_PORT=8083

# TLS
TLS_CERT_FILE=/etc/foxhunt/ml_training_service/ml_training_service.crt
TLS_KEY_FILE=/etc/foxhunt/ml_training_service/ml_training_service.key
TLS_CA_FILE=/etc/foxhunt/certs/ca.crt

# GPU
CUDA_VISIBLE_DEVICES=0,1

# Logging
RUST_LOG=info,ml_training_service=debug

---

Service Deployment

Build Release Binaries

cd /home/jgrusewski/Work/foxhunt

# Build all services in release mode with LTO
cargo build --release --workspace

# Binaries will be in target/release/:
# - api_gateway
# - trading_service
# - backtesting_service
# - ml_training_service
# - tli

Deploy API Gateway

# Create service user
sudo useradd -r -s /bin/false foxhunt

# Copy binary
sudo cp target/release/api_gateway /usr/local/bin/
sudo chmod 755 /usr/local/bin/api_gateway
sudo chown foxhunt:foxhunt /usr/local/bin/api_gateway

# Create systemd service
sudo tee /etc/systemd/system/api_gateway.service <<EOF
[Unit]
Description=Foxhunt API Gateway
After=network.target postgresql.service redis.service
Requires=postgresql.service redis.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
EnvironmentFile=/etc/foxhunt/api_gateway/.env
ExecStart=/usr/local/bin/api_gateway
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=api_gateway

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/foxhunt

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable api_gateway
sudo systemctl start api_gateway

# Check status
sudo systemctl status api_gateway
sudo journalctl -u api_gateway -f

Deploy Trading Service

# Copy binary
sudo cp target/release/trading_service /usr/local/bin/
sudo chmod 755 /usr/local/bin/trading_service
sudo chown foxhunt:foxhunt /usr/local/bin/trading_service

# Create systemd service
sudo tee /etc/systemd/system/trading_service.service <<EOF
[Unit]
Description=Foxhunt Trading Service
After=network.target postgresql.service redis.service
Requires=postgresql.service redis.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
EnvironmentFile=/etc/foxhunt/trading_service/.env
ExecStart=/usr/local/bin/trading_service
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=trading_service

# Performance tuning
CPUAffinity=0-7
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=99
LimitNOFILE=1048576

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable trading_service
sudo systemctl start trading_service

# Check status
sudo systemctl status trading_service

Deploy Backtesting Service

# Copy binary
sudo cp target/release/backtesting_service /usr/local/bin/
sudo chmod 755 /usr/local/bin/backtesting_service
sudo chown foxhunt:foxhunt /usr/local/bin/backtesting_service

# Create systemd service
sudo tee /etc/systemd/system/backtesting_service.service <<EOF
[Unit]
Description=Foxhunt Backtesting Service
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
EnvironmentFile=/etc/foxhunt/backtesting_service/.env
ExecStart=/usr/local/bin/backtesting_service
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=backtesting_service

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable backtesting_service
sudo systemctl start backtesting_service

Deploy ML Training Service

# Copy binary
sudo cp target/release/ml_training_service /usr/local/bin/
sudo chmod 755 /usr/local/bin/ml_training_service
sudo chown foxhunt:foxhunt /usr/local/bin/ml_training_service

# Create model cache directory
sudo mkdir -p /cache/models
sudo chown foxhunt:foxhunt /cache/models

# Create systemd service
sudo tee /etc/systemd/system/ml_training_service.service <<EOF
[Unit]
Description=Foxhunt ML Training Service
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=foxhunt
Group=foxhunt
EnvironmentFile=/etc/foxhunt/ml_training_service/.env
ExecStart=/usr/local/bin/ml_training_service
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=ml_training_service

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable ml_training_service
sudo systemctl start ml_training_service

---

Post-Deployment Verification

1. Health Check All Services

#!/bin/bash
# health_check.sh

echo "=== Foxhunt Production Health Check ==="
echo ""

# API Gateway
echo "1. API Gateway Health..."
curl -s http://localhost:8080/health || echo "❌ API Gateway FAILED"

# Trading Service
echo "2. Trading Service Health..."
curl -s http://localhost:8081/health || echo "❌ Trading Service FAILED"

# Backtesting Service
echo "3. Backtesting Service Health..."
curl -s http://localhost:8082/health || echo "❌ Backtesting Service FAILED"

# ML Training Service
echo "4. ML Training Service Health..."
curl -s http://localhost:8083/health || echo "❌ ML Training Service FAILED"

# PostgreSQL
echo "5. PostgreSQL..."
psql $DATABASE_URL -c "SELECT version();" || echo "❌ PostgreSQL FAILED"

# Redis
echo "6. Redis..."
redis-cli -a $REDIS_PASSWORD ping || echo "❌ Redis FAILED"

echo ""
echo "=== Health Check Complete ==="

2. Verify Service Logs

# Check for errors in all services
sudo journalctl -u api_gateway -u trading_service -u backtesting_service -u ml_training_service --since "5 minutes ago" | grep -i error

# Verify services are listening on correct ports
sudo netstat -tlnp | grep -E "(50051|50052|50053|50054|8080|8081|8082|8083)"

3. Connectivity Tests

Test mTLS Between Services:

# From TLI, test connection to API Gateway
cd /home/jgrusewski/Work/foxhunt
./target/release/tli --help  # Should connect successfully

Test Database Connections:

# Verify each service can connect to PostgreSQL
sudo -u foxhunt psql $DATABASE_URL -c "SELECT 'Connection successful';"

4. Smoke Tests

Test API Gateway Authentication:

# Generate JWT token (requires admin credentials)
curl -X POST http://localhost:50051/auth/login \
    -H "Content-Type: application/json" \
    -d '{"username":"admin","password":"ADMIN_PASSWORD"}'

# Use token to access protected endpoint
export TOKEN="eyJ..."
curl -H "Authorization: Bearer $TOKEN" http://localhost:50051/trading/portfolio

Test Order Submission (Trading Service):

# Submit test order via API Gateway
curl -X POST http://localhost:50051/trading/order \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
        "symbol": "AAPL",
        "side": "buy",
        "quantity": 100,
        "order_type": "limit",
        "limit_price": 150.0
    }'

5. Performance Baselines

Measure API Gateway Latency:

# Install wrk for load testing
sudo apt install -y wrk

# Test authentication overhead
wrk -t4 -c100 -d30s --latency http://localhost:50051/health

# Expected: p99 latency < 10ms for health endpoint

Measure Database Query Performance:

-- Enable query timing
\timing on

-- Test critical queries
SELECT * FROM users LIMIT 10;
SELECT * FROM permissions WHERE endpoint = 'trading.submit_order';

---

Performance Tuning

PostgreSQL Connection Pooling

Adjust per-service connection limits:

-- Check current connections
SELECT count(*), application_name FROM pg_stat_activity GROUP BY application_name;

-- If connections are exhausted, increase max_connections in postgresql.conf
-- Or implement PgBouncer for connection pooling

Deploy PgBouncer (Recommended for HFT):

# /etc/pgbouncer/pgbouncer.ini
[databases]
foxhunt_production = host=localhost port=5432 dbname=foxhunt_production

[pgbouncer]
listen_addr = 127.0.0.1
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20

Update service DATABASE_URL to use PgBouncer: postgresql://foxhunt_user:PASSWORD@localhost:6432/foxhunt_production

Redis Memory Optimization

# Monitor Redis memory usage
redis-cli -a $REDIS_PASSWORD INFO memory

# If memory usage is high, tune maxmemory-policy
redis-cli -a $REDIS_PASSWORD CONFIG SET maxmemory-policy allkeys-lfu

gRPC Thread Pool Configuration

Edit service code to adjust Tokio runtime:

// In main.rs
#[tokio::main(worker_threads = 16)]  // Adjust based on CPU cores
async fn main() -> Result<()> {
    // ...
}

Rebuild and redeploy services after tuning.

TCP Kernel Tuning (Already applied in Prerequisites)

Verify tuning is active:

sysctl net.ipv4.tcp_congestion_control  # Should be 'bbr'
sysctl net.core.rmem_max                # Should be 134217728

---

Disaster Recovery

Backup Strategy

PostgreSQL Backups:

# Daily full backup
sudo -u postgres pg_dump foxhunt_production > /backups/foxhunt_$(date +%Y%m%d).sql

# Compress and upload to S3
gzip /backups/foxhunt_$(date +%Y%m%d).sql
aws s3 cp /backups/foxhunt_$(date +%Y%m%d).sql.gz s3://foxhunt-backups/postgres/

# Retention: Keep last 30 days locally, 7 years in S3

Redis Backups:

# Trigger RDB snapshot
redis-cli -a $REDIS_PASSWORD BGSAVE

# Copy RDB file to backup location
sudo cp /var/lib/redis/dump.rdb /backups/redis_$(date +%Y%m%d).rdb

Configuration Backups:

# Backup all .env files and certificates
tar -czf /backups/config_$(date +%Y%m%d).tar.gz /etc/foxhunt/

Restore Procedures

Restore PostgreSQL:

# Stop services
sudo systemctl stop api_gateway trading_service backtesting_service ml_training_service

# Drop and recreate database
sudo -u postgres psql <<EOF
DROP DATABASE foxhunt_production;
CREATE DATABASE foxhunt_production OWNER foxhunt_user;
EOF

# Restore from backup
gunzip -c /backups/foxhunt_20251003.sql.gz | sudo -u postgres psql foxhunt_production

# Restart services
sudo systemctl start api_gateway trading_service backtesting_service ml_training_service

Restore Redis:

sudo systemctl stop redis-server
sudo cp /backups/redis_20251003.rdb /var/lib/redis/dump.rdb
sudo chown redis:redis /var/lib/redis/dump.rdb
sudo systemctl start redis-server

Recovery Time Objective (RTO) & Recovery Point Objective (RPO)

Component	RTO	RPO
API Gateway	5 minutes	N/A (stateless)
Trading Service	5 minutes	N/A (stateless)
PostgreSQL	30 minutes	24 hours (daily backup)
Redis	10 minutes	1 hour (AOF)

---

Rollback Procedures

Service Rollback

# Stop new version
sudo systemctl stop api_gateway

# Copy previous binary (assumes you keep old versions)
sudo cp /usr/local/bin/api_gateway.bak /usr/local/bin/api_gateway

# Start old version
sudo systemctl start api_gateway

# Verify
sudo systemctl status api_gateway
curl http://localhost:8080/health

Database Rollback

⚠️ WARNING: Database rollbacks are complex. Test thoroughly in staging.

# If new migration broke things, restore from pre-migration backup
sudo -u postgres psql foxhunt_production < /backups/pre_migration_backup.sql

For forward-only migrations (recommended), write compensating migrations instead of rolling back.

---

Production Deployment Pitfalls

⚠️ CRITICAL WARNINGS

1. Inadequate Testing Environments

Pitfall: Deploying to production without a staging environment that mirrors production (data volume, network conditions, resource limits).

Impact: Production surprises, unexpected latency, crashes under load.

Mitigation:

• Deploy identical staging environment with realistic data volumes
• Load test staging with production-level traffic (100K req/s)
• Measure latency baselines in staging before production

2. Secrets Management Mishaps

Pitfall: Hardcoded secrets, plain text files, lack of automated rotation.

Impact: Security breach, compromised system.

Mitigation:

• Never hardcode secrets in code or config files
• Use HashiCorp Vault or AWS Secrets Manager
• Implement automated secret rotation (JWT secrets every 90 days)
• Audit secret access regularly

3. Time Synchronization Issues (NTP/PTP)

Pitfall: Relying on default OS time sync or inaccurate NTP sources.

Impact: Incorrect trade timestamps, regulatory violations (MiFID II), audit failures.

Mitigation:

• Deploy chrony with financial market time servers
• For sub-microsecond accuracy, use PTP with hardware support
• Monitor time drift continuously (alert if offset > 100μs)
• Verify time sync before every trading session

4. Network Latency & Jitter

Pitfall: Overlooking network topology, physical proximity to exchanges, internal network hops.

Impact: Increased order latency, missed trading opportunities, competitive disadvantage.

Mitigation:

• Collocate infrastructure near exchanges
• Use dedicated network infrastructure (no shared VLANs)
• Implement kernel bypass (DPDK) for ultra-low latency
• Measure and monitor network jitter continuously

5. Insufficient Resource Allocation

Pitfall: Under-provisioning CPU, memory, network bandwidth, or improper OS/kernel tuning.

Impact: High latency, service crashes, data loss.

Mitigation:

• Follow hardware requirements strictly (32+ cores, 128GB+ RAM)
• Apply OS-level tuning (see Prerequisites section)
• Reserve CPU cores for critical trading threads (CPU affinity)
• Monitor resource usage and scale proactively

6. Observability Gaps

Pitfall: Deploying without comprehensive logging, metrics, distributed tracing.

Impact: Inability to diagnose production issues, extended downtime.

Mitigation:

• Deploy Prometheus + Grafana for metrics
• Deploy ELK stack or Loki for centralized logging
• Implement distributed tracing (OpenTelemetry)
• Create dashboards for critical KPIs (latency, throughput, error rate)

7. Certificate Management Overheads

Pitfall: Expired mTLS certificates, difficult rotation, lack of automation.

Impact: Service outages, broken inter-service communication.

Mitigation:

• Implement automated certificate renewal (cert-manager, Let's Encrypt)
• Set up alerts for expiring certificates (30 days before expiry)
• Test certificate rotation in staging regularly
• Document manual renewal procedures for emergencies

8. Database/Redis Misconfiguration

Pitfall: Lack of HA, improper caching strategies, insufficient connection pooling.

Impact: Service bottlenecks, data loss, downtime.

Mitigation:

• Deploy PostgreSQL streaming replication (async or sync)
• Deploy Redis Sentinel or Cluster for HA
• Use PgBouncer for connection pooling
• Tune PostgreSQL shared_buffers, work_mem based on workload

9. Compliance Blind Spots

Pitfall: Not fully addressing SOX/MiFID II requirements for audit trails, data retention, access control.

Impact: Regulatory fines, legal liability, loss of trading license.

Mitigation:

• Enable comprehensive audit logging (all trades, logins, config changes)
• Implement immutable audit trails (PostgreSQL with no DELETE)
• Define data retention policies (7 years for audit trails)
• Conduct regular compliance audits

10. Poor Rollback Strategy

Pitfall: Inability to quickly and safely revert to a previous stable state.

Impact: Extended downtime during deployment failures.

Mitigation:

• Keep previous binary versions (api_gateway.bak)
• Backup database before migrations (pg_dump)
• Test rollback procedures in staging monthly
• Document rollback steps in runbook

11. Rust-Specific Pitfalls

Pitfall: Not building with --release, missing LTO, unreviewed unsafe blocks.

Impact: Slower performance, larger binaries, memory safety issues.

Mitigation:

• Always build with cargo build --release for production
• Enable LTO in Cargo.toml: lto = "fat"
• Strictly review any unsafe blocks (should be rare)
• Run cargo clippy and cargo audit in CI/CD

12. JWT Secret Entropy

Pitfall: Using weak JWT secrets (< 32 bytes) or predictable values.

Impact: JWT forgery, unauthorized access.

Mitigation:

• Generate JWT secrets with openssl rand -base64 64
• Rotate JWT secrets every 90 days
• Never share JWT secrets across environments

---

Appendix: Quick Reference

Service Ports

Service	gRPC Port	Health Port
API Gateway	50051	8080
Trading Service	50052	8081
Backtesting Service	50053	8082
ML Training Service	50054	8083

Important File Locations

Component	Path
Binaries	/usr/local/bin/
Config	/etc/foxhunt/{service}/.env
Certificates	/etc/foxhunt/certs/
Logs	/var/log/foxhunt/
Backups	/backups/
Model Cache	/cache/models/

Emergency Contacts

• Trading Operations Lead: [PagerDuty rotation]
• Database Administrator: [PagerDuty rotation]
• Security Team: security@foxhunt.local
• On-Call Engineer: [PagerDuty rotation]

---

End of Production Deployment Guide

This document should be reviewed quarterly and updated after every major deployment.