foxhunt/docs/DEPLOYMENT.md

Foxhunt HFT System - Production Deployment Guide

Version: 1.0
Date: August 26, 2025
System Status: 85% Operational (11/13 services functional)

---

Table of Contents

1. System Requirements & Prerequisites
2. Infrastructure Deployment Options
3. Database Layer Setup
4. Core Service Deployment
5. Operations & Maintenance
6. Validation & Testing
7. Appendices

---

CRITICAL WARNING: HIGH-FREQUENCY TRADING SYSTEM

THIS SYSTEM HANDLES REAL MONEY TRADING OPERATIONS

Any configuration error can result in significant financial losses. This deployment guide must be followed exactly with complete validation at each step. When in doubt, HALT the deployment and consult the risk management team.

Financial Risk Mitigation:

• All deployments must maintain audit trails
• Position limits must be enforced at system level
• Circuit breakers must be tested and functional
• Disaster recovery procedures must be validated

---

Section 1: System Requirements & Prerequisites

1.1 Hardware Specifications (HFT-Optimized)

Minimum Production Requirements:

CPU:           Intel Xeon or AMD EPYC with >= 16 cores
               L3 Cache >= 32MB
               Base frequency >= 2.4GHz
               Support for CPU isolation and affinity

Memory:        64GB DDR4-3200 or higher
               NUMA-aware allocation
               Huge pages support (2MB/1GB)
               ECC memory required

Storage:       NVMe SSD >= 1TB (Primary)
               NVMe SSD >= 500GB (Logs/Temp)
               RAID 1 configuration for data protection
               >= 500K IOPS sustained

Network:       Dual 10GbE or single 25GbE minimum
               Low-latency NICs (Intel X710 or Mellanox)
               DPDK support preferred
               Precision Time Protocol (PTP) capable

Recommended Production Configuration:

CPU:           Intel Xeon Platinum 8380 (40 cores) or equivalent
Memory:        128GB DDR4-3200 with huge pages
Storage:       Dual NVMe in RAID 1 + separate WAL storage
Network:       Dual 25GbE with kernel bypass capabilities

1.2 Operating System Requirements

Base System:

• Ubuntu 22.04 LTS (Jammy) - Server Edition
• Real-time kernel (linux-image-rt-amd64)
• Kernel version >= 5.15 with PREEMPT_RT patches

Required Packages:

# System packages
apt-get install -y \
    linux-image-rt-amd64 \
    docker.io docker-compose-plugin \
    kubernetes-client \
    chrony \
    tuned \
    numactl \
    hwloc \
    cpuset \
    irqbalance \
    ethtool

# Performance monitoring
apt-get install -y \
    htop iotop \
    perf-tools-unstable \
    sysstat \
    nethogs \
    iftop

1.3 Kernel Optimizations for HFT

Boot Parameters (/etc/default/grub):

GRUB_CMDLINE_LINUX="
    isolcpus=2-15 
    nohz_full=2-15 
    rcu_nocbs=2-15 
    intel_idle.max_cstate=0 
    processor.max_cstate=0 
    intel_pstate=disable 
    nosoftlockup 
    nmi_watchdog=0 
    transparent_hugepage=never 
    default_hugepagesz=2M 
    hugepagesz=2M 
    hugepages=1024
"

Sysctl Optimizations (/etc/sysctl.d/99-hft-tuning.conf):

# Network performance
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_rmem = 4096 131072 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_congestion_control = bbr

# Memory management
vm.swappiness = 1
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.overcommit_memory = 1

# Process scheduling
kernel.sched_latency_ns = 1000000
kernel.sched_min_granularity_ns = 100000
kernel.sched_wakeup_granularity_ns = 50000

1.4 Network Configuration

Low Latency Network Setup:

# Disable interrupt coalescing
ethtool -C eth0 rx-usecs 0 tx-usecs 0

# Set ring buffer sizes
ethtool -G eth0 rx 4096 tx 4096

# CPU affinity for network interrupts
echo 2 > /proc/irq/24/smp_affinity  # NIC IRQ to isolated CPU

# Enable DPDK if supported
modprobe uio_pci_generic

PTP Time Synchronization:

# Install and configure chrony for PTP
systemctl enable chrony
echo "refclock PHC /dev/ptp0 poll 0 dpoll -2 offset 0" >> /etc/chrony/chrony.conf

1.5 Security Prerequisites

Certificate Management:

# Create certificate directory
mkdir -p /opt/foxhunt/certs/{ca,server,client}

# Generate CA certificate (production should use proper CA)
openssl genrsa -out /opt/foxhunt/certs/ca/ca-key.pem 4096
openssl req -new -x509 -days 365 -key /opt/foxhunt/certs/ca/ca-key.pem \
    -out /opt/foxhunt/certs/ca/ca.pem \
    -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=Foxhunt-CA"

Security Hardening:

# Firewall configuration
ufw --force enable
ufw default deny incoming
ufw default allow outgoing

# Allow necessary ports
ufw allow 22/tcp      # SSH
ufw allow 443/tcp     # HTTPS
ufw allow 8080/tcp    # Trading Engine API
ufw allow 5432/tcp    # PostgreSQL (internal network only)
ufw allow 6379/tcp    # Redis (internal network only)
ufw allow 8086/tcp    # InfluxDB (internal network only)

---

Section 2: Infrastructure Deployment Options

2.1 Deployment Architecture Decision Matrix

+------------------+------------------+------------------+
| Component        | Docker Swarm     | Kubernetes       |
+------------------+------------------+------------------+
| Trading Engine   | RECOMMENDED      | Optional         |
| Market Data      | RECOMMENDED      | Optional         |
| Risk Management  | RECOMMENDED      | Optional         |
| Databases        | RECOMMENDED      | Not Recommended  |
| Monitoring       | Optional         | RECOMMENDED      |
| Analytics        | Optional         | RECOMMENDED      |
+------------------+------------------+------------------+

Rationale: Core trading components require minimal latency overhead

2.2 Option A: Docker Swarm Production (Recommended for Core Trading)

Initialize Docker Swarm:

# On manager node
docker swarm init --advertise-addr <MANAGER-IP>

# Create production networks
docker network create \
    --driver overlay \
    --attachable \
    --opt encrypted=true \
    foxhunt-trading-prod

docker network create \
    --driver overlay \
    --attachable \
    foxhunt-monitoring-prod

Deploy Core Services:

# Navigate to deployment directory
cd /opt/foxhunt/ops/docker

# Set production environment
export FOXHUNT_ENV=production

# Load environment variables
source .env.production

# Deploy production stack
docker stack deploy -c docker-compose.prod.yml foxhunt-prod

CPU Affinity Configuration:

# Pin trading engine to cores 0-3
docker service update \
    --constraint-add node.role==manager \
    --placement-pref spread=node.id \
    foxhunt-prod_trading-engine

# Verify CPU assignment
docker exec $(docker ps -q -f name=trading-engine) \
    taskset -c -p 1

2.3 Option B: Kubernetes Production

Kubernetes Cluster Setup:

# Install kubectl if not present
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# Create namespace
kubectl create namespace foxhunt-trading-prod

# Apply RBAC
kubectl apply -f ops/kubernetes/manifests/

Deploy Trading Services:

# Navigate to Kubernetes manifests
cd /opt/foxhunt/ops/kubernetes/production

# Deploy in dependency order
kubectl apply -f namespace.yaml
kubectl apply -f secrets.yaml
kubectl apply -f configmap.yaml
kubectl apply -f trading-engine-deployment.yaml

# Verify deployment
kubectl get pods -n foxhunt-trading-prod
kubectl logs -f deployment/trading-engine -n foxhunt-trading-prod

2.4 Option C: Hybrid Deployment (Best Practice)

Core Trading on Docker Swarm:

# Deploy latency-critical services
docker stack deploy -c docker-compose-trading-core.yml foxhunt-trading

Supporting Services on Kubernetes:

# Deploy monitoring and analytics
kubectl apply -f ops/kubernetes/monitoring/

---

Section 3: Database Layer Setup

3.1 PostgreSQL Cluster Deployment

Primary Database Setup:

# Create data directories
mkdir -p /opt/foxhunt/data/postgres/{primary,replica}
chown -R 999:999 /opt/foxhunt/data/postgres

# Deploy PostgreSQL primary
docker service create \
    --name postgres-primary \
    --network foxhunt-trading-prod \
    --mount type=bind,source=/opt/foxhunt/data/postgres/primary,target=/var/lib/postgresql/data \
    --env POSTGRES_DB=hft_trading_prod \
    --env POSTGRES_USER=hft_user_prod \
    --env POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password \
    --secret postgres_password \
    --publish 5432:5432 \
    --replicas 1 \
    --constraint 'node.role == manager' \
    postgres:16-alpine

Database Schema Migration:

# Run migrations
docker exec -i postgres-primary psql -U hft_user_prod -d hft_trading_prod < migrations/001_initial_schema.sql
docker exec -i postgres-primary psql -U hft_user_prod -d hft_trading_prod < migrations/002_trading_tables.sql
docker exec -i postgres-primary psql -U hft_user_prod -d hft_trading_prod < migrations/003_indexes.sql

# Verify schema
docker exec -i postgres-primary psql -U hft_user_prod -d hft_trading_prod -c "\dt"

PostgreSQL HFT Optimizations:

-- High-performance settings
ALTER SYSTEM SET shared_buffers = '16GB';
ALTER SYSTEM SET effective_cache_size = '48GB';
ALTER SYSTEM SET maintenance_work_mem = '2GB';
ALTER SYSTEM SET checkpoint_segments = 64;
ALTER SYSTEM SET checkpoint_completion_target = 0.9;
ALTER SYSTEM SET wal_buffers = '64MB';
ALTER SYSTEM SET default_statistics_target = 1000;

-- Restart required
SELECT pg_reload_conf();

3.2 InfluxDB Time-Series Setup

InfluxDB Deployment:

# Create InfluxDB data directory
mkdir -p /opt/foxhunt/data/influxdb
chown -R 1000:1000 /opt/foxhunt/data/influxdb

# Deploy InfluxDB
docker service create \
    --name influxdb \
    --network foxhunt-trading-prod \
    --mount type=bind,source=/opt/foxhunt/data/influxdb,target=/var/lib/influxdb2 \
    --env DOCKER_INFLUXDB_INIT_MODE=setup \
    --env DOCKER_INFLUXDB_INIT_USERNAME=admin \
    --env DOCKER_INFLUXDB_INIT_PASSWORD_FILE=/run/secrets/influxdb_password \
    --env DOCKER_INFLUXDB_INIT_ORG=foxhunt-prod \
    --env DOCKER_INFLUXDB_INIT_BUCKET=market_data_prod \
    --secret influxdb_password \
    --secret influxdb_token \
    --publish 8086:8086 \
    influxdb:2.7-alpine

Market Data Schema Creation:

# Create market data bucket with appropriate retention
influx bucket create \
    --name market_data_realtime \
    --retention 7d \
    --org foxhunt-prod

influx bucket create \
    --name market_data_historical \
    --retention 2555d \
    --org foxhunt-prod

3.3 Redis Cache Configuration

Redis High-Performance Setup:

# Create Redis configuration
cat > /opt/foxhunt/configs/redis-prod.conf << EOF
# Memory settings
maxmemory 8gb
maxmemory-policy allkeys-lru

# Persistence settings
save 900 1
save 300 10
save 60 10000
appendonly yes
appendfsync everysec

# Network settings
tcp-keepalive 300
timeout 0

# Performance settings
hz 100
latency-monitor-threshold 100
EOF

# Deploy Redis
docker service create \
    --name redis-prod \
    --network foxhunt-trading-prod \
    --mount type=bind,source=/opt/foxhunt/configs/redis-prod.conf,target=/etc/redis/redis.conf,readonly \
    --mount type=bind,source=/opt/foxhunt/data/redis,target=/data \
    --publish 6379:6379 \
    redis:7-alpine redis-server /etc/redis/redis.conf --requirepass $(cat /run/secrets/redis_password)

Redis Performance Validation:

# Latency testing
redis-cli --latency-history -i 1

# Memory usage analysis
redis-cli info memory

# Performance benchmarking
redis-benchmark -h localhost -p 6379 -c 50 -n 100000

---

Section 4: Core Service Deployment

4.1 Service Dependency Chain

Dependency Flow (CRITICAL - Deploy in this order):

1. Security Service     ←─ Authentication & Authorization
   ↓
2. Persistence Service  ←─ Database connectivity layer  
   ↓
3. Market Data Service  ←─ External API integration
   ↓
4. Trading Engine       ←─ Core order processing
   ↓
5. Risk Management      ←─ Position monitoring
   ↓
6. Broker Connector     ←─ Order execution

4.2 Phase 1: Security Service Deployment

Deploy Security Service:

# Verify security prerequisites
ls -la /opt/foxhunt/certs/
docker secret ls | grep -E "(jwt|ca|server)"

# Deploy security service
docker service create \
    --name security-service \
    --network foxhunt-trading-prod \
    --env RUST_LOG=info,security=debug \
    --env JWT_SECRET_FILE=/run/secrets/jwt_secret \
    --env CA_CERT_FILE=/run/secrets/ca_cert \
    --secret jwt_secret \
    --secret ca_cert \
    --publish 8060:8060 \
    --replicas 1 \
    --constraint 'node.role == manager' \
    foxhunt/security-service:latest

# Health check
curl -f http://localhost:8060/health

4.3 Phase 2: Persistence Service

Deploy Persistence Layer:

# Deploy persistence service
docker service create \
    --name persistence-service \
    --network foxhunt-trading-prod \
    --env DATABASE_URL=postgresql://hft_user_prod:$(cat /run/secrets/postgres_password)@postgres-primary:5432/hft_trading_prod \
    --env REDIS_URL=redis://:$(cat /run/secrets/redis_password)@redis-prod:6379/0 \
    --env INFLUXDB_URL=http://influxdb:8086 \
    --env INFLUXDB_TOKEN_FILE=/run/secrets/influxdb_token \
    --secret postgres_password \
    --secret redis_password \
    --secret influxdb_token \
    --publish 8110:8110 \
    foxhunt/persistence:latest

# Verify database connectivity
curl http://localhost:8110/health/database

4.4 Phase 3: Market Data Service

External API Configuration:

# Verify external API credentials
docker secret ls | grep -E "(polygon|finnhub)"

# Deploy market data service
docker service create \
    --name market-data-service \
    --network foxhunt-trading-prod \
    --env RUST_LOG=info,market_data=debug \
    --env POLYGON_API_KEY_FILE=/run/secrets/polygon_api_key \
    --env FINNHUB_API_KEY_FILE=/run/secrets/finnhub_api_key \
    --env REDIS_URL=redis://:$(cat /run/secrets/redis_password)@redis-prod:6379/1 \
    --env PERSISTENCE_SERVICE_URL=http://persistence-service:8110 \
    --secret polygon_api_key \
    --secret finnhub_api_key \
    --secret redis_password \
    --publish 8090:8090 \
    --cpuset-cpus="4-7" \
    --memory=12g \
    foxhunt/market-data:latest

# Verify market data feed
curl http://localhost:8090/health
curl http://localhost:8090/market-data/AAPL/latest

4.5 Phase 4: Trading Engine (CRITICAL)

Trading Engine Deployment:

# CRITICAL: Verify all dependencies are healthy
curl -f http://localhost:8060/health  # Security
curl -f http://localhost:8110/health  # Persistence  
curl -f http://localhost:8090/health  # Market Data

# Deploy trading engine with maximum performance
docker service create \
    --name trading-engine \
    --network foxhunt-trading-prod \
    --env RUST_LOG=info,trading_engine=debug \
    --env DATABASE_URL=postgresql://hft_user_prod:$(cat /run/secrets/postgres_password)@postgres-primary:5432/hft_trading_prod \
    --env REDIS_URL=redis://:$(cat /run/secrets/redis_password)@redis-prod:6379/0 \
    --env SECURITY_SERVICE_URL=http://security-service:8060 \
    --env MARKET_DATA_SERVICE_URL=http://market-data-service:8090 \
    --env PERSISTENCE_SERVICE_URL=http://persistence-service:8110 \
    --env MAX_POSITION_SIZE=1000000 \
    --env ORDER_TIMEOUT_MS=5000 \
    --secret postgres_password \
    --secret redis_password \
    --secret jwt_secret \
    --publish 8080:8080 \
    --publish 8081:8081 \
    --cpuset-cpus="0-3" \
    --memory=8g \
    --ulimit nofile=1048576:1048576 \
    --ulimit memlock=-1:-1 \
    --constraint 'node.role == manager' \
    foxhunt/trading-engine:latest

# CRITICAL: Validate trading engine
curl -f http://localhost:8080/health
curl -f http://localhost:8080/ready
curl -f http://localhost:8081/metrics

4.6 Phase 5: Risk Management

Risk Management Service:

# Deploy risk management
docker service create \
    --name risk-management \
    --network foxhunt-trading-prod \
    --env RUST_LOG=info,risk=debug \
    --env TRADING_ENGINE_URL=http://trading-engine:8080 \
    --env PERSISTENCE_SERVICE_URL=http://persistence-service:8110 \
    --env MAX_DAILY_LOSS=50000 \
    --env POSITION_LIMIT_PERCENT=5 \
    --env RISK_CHECK_INTERVAL_MS=100 \
    --publish 8070:8070 \
    --cpuset-cpus="16-19" \
    foxhunt/risk-management:latest

# Verify risk controls
curl http://localhost:8070/health
curl http://localhost:8070/risk/current-limits

4.7 Phase 6: Broker Connector

Broker Integration:

# Deploy broker connector
docker service create \
    --name broker-connector \
    --network foxhunt-trading-prod \
    --env RUST_LOG=info,broker=debug \
    --env TRADING_ENGINE_URL=http://trading-engine:8080 \
    --env ICMARKETS_CLIENT_ID_FILE=/run/secrets/icmarkets_client_id \
    --env ICMARKETS_CLIENT_SECRET_FILE=/run/secrets/icmarkets_secret \
    --env FIX_CONFIG_FILE=/etc/broker/fix-config.xml \
    --secret icmarkets_client_id \
    --secret icmarkets_secret \
    --publish 8120:8120 \
    foxhunt/broker-connector:latest

# Verify broker connectivity
curl http://localhost:8120/health
curl http://localhost:8120/broker/status

---

Section 5: Operations & Maintenance

5.1 Monitoring Stack Deployment

Prometheus Configuration:

# Deploy Prometheus
docker service create \
    --name prometheus \
    --network foxhunt-monitoring-prod \
    --mount type=bind,source=/opt/foxhunt/configs/prometheus.yml,target=/etc/prometheus/prometheus.yml \
    --mount type=bind,source=/opt/foxhunt/data/prometheus,target=/prometheus \
    --publish 9090:9090 \
    prom/prometheus:v2.48.0 \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/prometheus \
    --storage.tsdb.retention.time=90d

Grafana Dashboard Setup:

# Deploy Grafana
docker service create \
    --name grafana \
    --network foxhunt-monitoring-prod \
    --env GF_SECURITY_ADMIN_PASSWORD_FILE=/run/secrets/grafana_password \
    --mount type=bind,source=/opt/foxhunt/configs/grafana,target=/etc/grafana/provisioning \
    --mount type=bind,source=/opt/foxhunt/data/grafana,target=/var/lib/grafana \
    --secret grafana_password \
    --publish 3000:3000 \
    grafana/grafana:10.2.0

Critical HFT Dashboards:

• Trading Performance: Latency, throughput, error rates
• Market Data: Feed latency, message rates, data quality
• Risk Metrics: Position exposure, P&L, limits
• System Health: CPU, memory, network, disk I/O

5.2 Performance Monitoring

Real-time Latency Monitoring:

# Enable latency monitoring
echo 'kernel.latencytop=1' >> /etc/sysctl.d/99-hft-tuning.conf

# Create latency monitoring script
cat > /opt/foxhunt/scripts/monitor-latency.sh << 'EOF'
#!/bin/bash
while true; do
    # Trading engine API latency
    curl -w "@curl-format.txt" -s -o /dev/null http://localhost:8080/health
    
    # Database query latency  
    docker exec postgres-primary psql -U hft_user_prod -d hft_trading_prod \
        -c "SELECT pg_stat_get_db_numbackends(oid) FROM pg_database WHERE datname='hft_trading_prod';" \
        > /dev/null
    
    sleep 1
done
EOF

Performance Baselines:

Target Performance Metrics:
- Order processing latency: < 1ms (99th percentile)
- Market data latency: < 10ms (99th percentile)  
- Database query latency: < 1ms (average)
- Memory allocation latency: < 100μs
- Network round-trip time: < 0.5ms (intra-datacenter)

5.3 Backup and Recovery

Automated Backup System:

# PostgreSQL backup script
cat > /opt/foxhunt/scripts/backup-postgres.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/opt/foxhunt/backups/postgres"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p ${BACKUP_DIR}

# Full database backup
docker exec postgres-primary pg_dump -U hft_user_prod hft_trading_prod | \
    gzip > ${BACKUP_DIR}/hft_trading_${TIMESTAMP}.sql.gz

# WAL archive backup
docker exec postgres-primary pg_basebackup -D /tmp/backup -F t -z -P -U hft_user_prod

# Retention policy (keep 30 days)
find ${BACKUP_DIR} -name "*.sql.gz" -mtime +30 -delete
EOF

# Schedule backups
echo "0 2 * * * /opt/foxhunt/scripts/backup-postgres.sh" | crontab -

Disaster Recovery Procedure:

# 1. Stop all trading services
docker service ls | grep foxhunt | awk '{print $2}' | xargs -I {} docker service rm {}

# 2. Restore database
gunzip -c /opt/foxhunt/backups/postgres/latest.sql.gz | \
    docker exec -i postgres-primary psql -U hft_user_prod -d hft_trading_prod

# 3. Verify data integrity
docker exec postgres-primary psql -U hft_user_prod -d hft_trading_prod \
    -c "SELECT COUNT(*) FROM trades WHERE created_at >= CURRENT_DATE;"

# 4. Restart services in dependency order
# (Follow Section 4 deployment sequence)

---

Section 6: Validation & Testing

6.1 Deployment Validation Checklist

System-Level Validation:

HARDWARE & OS:
[ ] CPU isolation configured (isolcpus parameter)
[ ] Huge pages allocated and available
[ ] Real-time kernel installed and active
[ ] Network interfaces optimized for low latency
[ ] PTP time synchronization operational
[ ] Firewall rules configured correctly

DATABASE LAYER:
[ ] PostgreSQL primary/replica cluster operational
[ ] Database schema migrations completed successfully  
[ ] InfluxDB time-series buckets created
[ ] Redis cache operational with correct memory limits
[ ] All database connections tested from services
[ ] Backup procedures tested and automated

SECURITY:
[ ] All certificates installed and valid
[ ] JWT authentication functional
[ ] Service-to-service mTLS operational
[ ] RBAC permissions configured correctly
[ ] External API keys configured and tested
[ ] Audit logging operational

SERVICES:
[ ] All 11 services deployed and healthy
[ ] Service dependency chain respected
[ ] gRPC communication operational
[ ] HTTP API endpoints responding
[ ] Metrics collection operational
[ ] Log aggregation functional

PERFORMANCE:
[ ] Order processing latency < 1ms
[ ] Market data latency < 10ms
[ ] Database query performance validated
[ ] Memory allocation optimized
[ ] CPU affinity assignments verified
[ ] Network throughput tested

6.2 Performance Benchmarking

Latency Testing Suite:

# Order processing latency test
cat > /opt/foxhunt/scripts/test-order-latency.sh << 'EOF'
#!/bin/bash
echo "Testing order processing latency..."

for i in {1..1000}; do
    start_time=$(date +%s%N)
    
    curl -s -X POST http://localhost:8080/orders \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $JWT_TOKEN" \
        -d '{
            "symbol": "AAPL",
            "quantity": 100,
            "side": "BUY",
            "order_type": "MARKET"
        }' > /dev/null
    
    end_time=$(date +%s%N)
    latency_ns=$((end_time - start_time))
    latency_us=$((latency_ns / 1000))
    
    echo "Order $i: ${latency_us}μs"
    
    if [ $latency_us -gt 1000 ]; then
        echo "WARNING: Latency exceeded 1ms threshold!"
    fi
done
EOF

chmod +x /opt/foxhunt/scripts/test-order-latency.sh

Throughput Testing:

# Market data throughput test
cat > /opt/foxhunt/scripts/test-market-data-throughput.sh << 'EOF'
#!/bin/bash
echo "Testing market data throughput..."

# Start throughput monitoring
start_time=$(date +%s)
start_messages=$(curl -s http://localhost:8090/metrics | grep "market_data_messages_total" | cut -d' ' -f2)

# Wait for test duration
sleep 60

# Calculate throughput
end_time=$(date +%s)
end_messages=$(curl -s http://localhost:8090/metrics | grep "market_data_messages_total" | cut -d' ' -f2)

duration=$((end_time - start_time))
message_count=$((end_messages - start_messages))
throughput=$((message_count / duration))

echo "Market data throughput: ${throughput} messages/second"

if [ $throughput -lt 10000 ]; then
    echo "WARNING: Throughput below 10k messages/second target!"
fi
EOF

6.3 Security Validation

Security Test Suite:

# Penetration testing checklist
cat > /opt/foxhunt/scripts/security-validation.sh << 'EOF'
#!/bin/bash
echo "Running security validation..."

# Test API authentication
echo "Testing API authentication..."
response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/orders)
if [ "$response" = "401" ]; then
    echo "✓ Unauthenticated requests properly rejected"
else
    echo "✗ Authentication bypass detected!"
fi

# Test JWT token validation
echo "Testing JWT validation..."
invalid_token="invalid.jwt.token"
response=$(curl -s -o /dev/null -w "%{http_code}" \
    -H "Authorization: Bearer $invalid_token" \
    http://localhost:8080/orders)
if [ "$response" = "401" ]; then
    echo "✓ Invalid JWT tokens properly rejected"
else
    echo "✗ JWT validation bypass detected!"
fi

# Test database connection security
echo "Testing database security..."
nmap -p 5432 localhost | grep -q "closed"
if [ $? -eq 0 ]; then
    echo "✓ Database port not exposed externally"
else
    echo "✗ Database port accessible from external network!"
fi
EOF

6.4 End-to-End Trading Simulation

Trading Workflow Test:

cat > /opt/foxhunt/scripts/e2e-trading-test.sh << 'EOF'
#!/bin/bash
set -e

echo "Starting end-to-end trading simulation..."

# 1. Authenticate and get JWT token
JWT_TOKEN=$(curl -s -X POST http://localhost:8060/auth/login \
    -H "Content-Type: application/json" \
    -d '{"username":"trader","password":"secure_password"}' | \
    jq -r '.token')

# 2. Check account balance
echo "Checking account balance..."
curl -s -H "Authorization: Bearer $JWT_TOKEN" \
    http://localhost:8080/account/balance

# 3. Get market data
echo "Fetching market data..."
curl -s http://localhost:8090/market-data/AAPL/latest

# 4. Place test order
echo "Placing test order..."
ORDER_ID=$(curl -s -X POST http://localhost:8080/orders \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $JWT_TOKEN" \
    -d '{
        "symbol": "AAPL",
        "quantity": 100,
        "side": "BUY", 
        "order_type": "LIMIT",
        "limit_price": 150.00
    }' | jq -r '.order_id')

# 5. Monitor order status
echo "Monitoring order $ORDER_ID..."
for i in {1..10}; do
    STATUS=$(curl -s -H "Authorization: Bearer $JWT_TOKEN" \
        http://localhost:8080/orders/$ORDER_ID | jq -r '.status')
    echo "Order status: $STATUS"
    
    if [ "$STATUS" = "FILLED" ] || [ "$STATUS" = "CANCELLED" ]; then
        break
    fi
    sleep 1
done

# 6. Check position
echo "Checking position..."
curl -s -H "Authorization: Bearer $JWT_TOKEN" \
    http://localhost:8080/positions/AAPL

echo "End-to-end test completed successfully!"
EOF

---

Section 7: Appendices

7.1 Configuration Templates

Environment Variables Template (.env.production):

# Production Environment Configuration
FOXHUNT_ENV=production
RUST_LOG=info
RUST_BACKTRACE=0

# Database Configuration  
POSTGRES_DB=hft_trading_prod
POSTGRES_USER=hft_user_prod
POSTGRES_PASSWORD=<SECURE_PASSWORD>

# Redis Configuration
REDIS_PASSWORD=<SECURE_PASSWORD>

# InfluxDB Configuration
INFLUXDB_ORG=foxhunt-prod
INFLUXDB_BUCKET=market_data_prod
INFLUXDB_ADMIN_PASSWORD=<SECURE_PASSWORD>
INFLUXDB_TOKEN=<SECURE_TOKEN>

# External API Keys
POLYGON_API_KEY=<POLYGON_PREMIUM_KEY>
FINNHUB_API_KEY=<FINNHUB_API_KEY>

# Broker Credentials
ICMARKETS_CLIENT_ID=<CLIENT_ID>
ICMARKETS_CLIENT_SECRET=<CLIENT_SECRET>

# Security
FOXHUNT_JWT_SECRET=<256_BIT_SECRET>

# Performance Tuning
DATA_FEED_BUFFER_SIZE=1048576
RISK_CHECK_INTERVAL_MS=100
MAX_POSITION_SIZE=1000000

7.2 Command Reference

Service Management Commands:

# View all services
docker service ls

# Check service logs
docker service logs -f foxhunt-prod_trading-engine

# Update service configuration
docker service update --env-add NEW_VAR=value foxhunt-prod_trading-engine

# Scale service
docker service scale foxhunt-prod_market-data=2

# Rolling restart
docker service update --force foxhunt-prod_trading-engine

# Remove service
docker service rm foxhunt-prod_trading-engine

Monitoring Commands:

# System performance
htop
iotop
nethogs

# Service health checks
curl http://localhost:8080/health
curl http://localhost:8080/ready
curl http://localhost:8080/metrics

# Database operations
docker exec -it postgres-primary psql -U hft_user_prod -d hft_trading_prod

# Redis operations
docker exec -it redis-prod redis-cli

# Log aggregation
docker logs --tail 100 -f $(docker ps -q -f name=trading-engine)

7.3 Emergency Procedures

Trading Halt Procedure:

#!/bin/bash
# Emergency trading halt - USE ONLY IN CRITICAL SITUATIONS

echo "INITIATING EMERGENCY TRADING HALT"
echo "Timestamp: $(date)"

# 1. Stop order processing
curl -X POST -H "Authorization: Bearer $ADMIN_JWT" \
    http://localhost:8080/admin/halt-trading

# 2. Cancel all open orders  
curl -X POST -H "Authorization: Bearer $ADMIN_JWT" \
    http://localhost:8080/admin/cancel-all-orders

# 3. Stop market data ingestion
docker service scale foxhunt-prod_market-data=0

# 4. Verify halt status
curl -H "Authorization: Bearer $ADMIN_JWT" \
    http://localhost:8080/admin/trading-status

echo "TRADING HALT COMPLETED"
echo "All trading activity suspended"
echo "Contact risk management team immediately"

Service Recovery:

#!/bin/bash
# Service recovery procedure

SERVICE_NAME=$1
if [ -z "$SERVICE_NAME" ]; then
    echo "Usage: $0 <service_name>"
    exit 1
fi

echo "Recovering service: $SERVICE_NAME"

# 1. Check service status
docker service ps $SERVICE_NAME

# 2. View recent logs
docker service logs --tail 50 $SERVICE_NAME

# 3. Restart service
docker service update --force $SERVICE_NAME

# 4. Wait for health check
sleep 10

# 5. Verify recovery
case $SERVICE_NAME in
    "foxhunt-prod_trading-engine")
        curl -f http://localhost:8080/health
        ;;
    "foxhunt-prod_market-data")
        curl -f http://localhost:8090/health
        ;;
    "foxhunt-prod_risk-management")
        curl -f http://localhost:8070/health
        ;;
esac

echo "Service recovery completed for $SERVICE_NAME"

7.4 Compliance Documentation

Audit Trail Requirements:

REGULATORY COMPLIANCE CHECKLIST:

TRADE REPORTING:
[ ] All trades logged with timestamp accuracy
[ ] Order lifecycle fully auditable
[ ] Position changes tracked with reasons
[ ] P&L calculations documented and verifiable

DATA RETENTION:
[ ] Trade data retained for 7 years minimum
[ ] Market data retained for regulatory periods
[ ] System logs retained for 2 years minimum
[ ] Backup verification performed monthly

RISK CONTROLS:
[ ] Position limits enforced at system level
[ ] Maximum loss limits configured and monitored
[ ] Circuit breakers tested and operational
[ ] Risk metrics calculated and reported real-time

ACCESS CONTROL:
[ ] All system access logged and monitored
[ ] Multi-factor authentication enforced
[ ] Privileged access regularly reviewed
[ ] Session management configured properly

BUSINESS CONTINUITY:
[ ] Disaster recovery procedures tested quarterly
[ ] Backup systems operational and validated
[ ] Network redundancy configured
[ ] Service availability meets SLA requirements

Contact Information:

CRITICAL ESCALATION CONTACTS:

Risk Management Emergency:
  Phone: [REDACTED]
  Email: risk-emergency@foxhunt.com
  Slack: #foxhunt-emergency

Technical Support:
  DevOps Team: devops@foxhunt.com
  Database Team: dba@foxhunt.com
  Security Team: security@foxhunt.com

Regulatory Compliance:
  Compliance Officer: compliance@foxhunt.com
  Legal Team: legal@foxhunt.com
  External Auditor: [REDACTED]

---

DEPLOYMENT SUCCESS CRITERIA

Financial Safety Validated:

• All circuit breakers tested and functional
• Position limits enforced at system level
• P&L monitoring operational
• Risk controls activated

Performance Requirements Met:

• Order processing latency < 1ms (99th percentile)
• Market data latency < 10ms (99th percentile)
• System availability > 99.99%
• Zero data loss during deployment

Security Posture Confirmed:

• All services authenticate via JWT
• Database connections encrypted
• External API access secured
• Audit logging operational

Operational Readiness Achieved:

• All 11 services healthy and responsive
• Monitoring and alerting functional
• Backup procedures automated and tested
• Disaster recovery validated

---

FINAL REMINDER: This is a financial trading system handling real money. Every step must be validated before proceeding. When in doubt, halt the deployment and consult the risk management team.

Deployment Status: Ready for production deployment with 85% system operational status.