foxhunt/docs/archive/infrastructure/DOCKER_DEPLOYMENT.md

Foxhunt HFT Trading System - Docker Deployment Guide

Wave 71 Agent 8: Complete Docker Compose Production Stack

This guide provides complete instructions for deploying the Foxhunt HFT trading system using Docker Compose.

Table of Contents

• Overview
• Architecture
• Prerequisites
• Quick Start
• Production Deployment
• Service Details
• Monitoring
• Troubleshooting
• Security

Overview

The Foxhunt Docker Compose stack includes:

• Infrastructure: PostgreSQL, Redis, InfluxDB, Vault, Prometheus, Grafana
• API Gateway (Wave 70): JWT authentication, rate limiting, request routing
• Backend Services: Trading, Backtesting, ML Training
• TLI Client (optional): Terminal interface for debugging

Architecture

Network Topology

                    ┌─────────────────────┐
                    │   External Access   │
                    │   (Port 50050)      │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │   API Gateway       │
                    │  (Authentication    │
                    │   Rate Limiting)    │
                    └──────────┬──────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        │                      │                      │
┌───────▼───────┐   ┌─────────▼─────────┐   ┌───────▼────────┐
│   Trading     │   │   Backtesting     │   │  ML Training   │
│   Service     │   │   Service         │   │  Service       │
│ (Port 50051)  │   │  (Port 50052)     │   │ (Port 50053)   │
└───────┬───────┘   └─────────┬─────────┘   └────────┬───────┘
        │                     │                       │
        └─────────────────────┼───────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
┌───────▼───────┐   ┌─────────▼─────────┐   ┌──────▼────────┐
│  PostgreSQL   │   │     Redis         │   │    Vault      │
│  (Database)   │   │    (Cache)        │   │  (Secrets)    │
└───────────────┘   └───────────────────┘   └───────────────┘

Service Communication

• External Network (foxhunt_external): API Gateway only
• Internal Network (foxhunt_internal): All services
• Backend services are NOT exposed to external networks
• All service communication uses gRPC with health checks

Prerequisites

System Requirements

• OS: Linux, macOS, or Windows with WSL2
• Docker: 24.0+ (with Compose V2)
• CPU: 8+ cores recommended (production: 16+ cores)
• RAM: 16GB minimum (production: 32GB+)
• Disk: 50GB+ free space

Software Installation

# Docker and Docker Compose
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

# Verify installation
docker --version
docker compose version

Quick Start

1. Clone Repository

git clone https://github.com/user/foxhunt.git
cd foxhunt

2. Configure Environment

# Copy environment template
cp .env.production.example .env.production

# Edit with your values (CRITICAL: Change all CHANGE_ME values)
nano .env.production

Minimum required changes:

• POSTGRES_PASSWORD
• JWT_SECRET (generate with: openssl rand -base64 32)
• INFLUXDB_PASSWORD
• VAULT_ROOT_TOKEN
• GRAFANA_ADMIN_PASSWORD

3. Start Infrastructure

# Start infrastructure services first
docker compose -f docker-compose.production.yml up -d postgres redis vault

# Wait for services to be healthy
docker compose -f docker-compose.production.yml ps

4. Initialize Database

# Run database migrations
docker compose -f docker-compose.production.yml exec postgres \
  psql -U foxhunt -d foxhunt -f /docker-entrypoint-initdb.d/001_trading_events.sql

5. Start All Services

# Start complete stack
docker compose -f docker-compose.production.yml up -d

# Check service health
docker compose -f docker-compose.production.yml ps
docker compose -f docker-compose.production.yml logs -f api_gateway

6. Verify Deployment

# Test API Gateway health
grpcurl -plaintext localhost:50050 grpc.health.v1.Health/Check

# Check Prometheus metrics
curl http://localhost:9091/metrics

# Access Grafana
open http://localhost:3000  # admin / [GRAFANA_ADMIN_PASSWORD]

Production Deployment

Security Hardening

1. Generate Strong Secrets

# JWT Secret (32+ bytes)
openssl rand -base64 32 > secrets/jwt_secret.txt

# PostgreSQL Password
openssl rand -base64 24 > secrets/postgres_password.txt

# Redis Password
openssl rand -base64 24 > secrets/redis_password.txt

2. TLS Certificates

# Generate self-signed certificates (development)
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout certs/server.key \
  -out certs/server.crt \
  -days 365 -subj "/CN=foxhunt.local"

# Production: Use Let's Encrypt or corporate CA

3. Configure Firewall

# Allow only API Gateway external port
sudo ufw allow 50050/tcp comment "API Gateway"
sudo ufw deny 50051:50053/tcp comment "Block backend services"

High Availability Setup

Database Replication

# docker-compose.ha.yml
services:
  postgres-primary:
    image: postgres:16-alpine
    environment:
      POSTGRES_REPLICATION_MODE: master

  postgres-replica:
    image: postgres:16-alpine
    environment:
      POSTGRES_REPLICATION_MODE: slave
      POSTGRES_MASTER_HOST: postgres-primary

Load Balancing

services:
  haproxy:
    image: haproxy:2.8-alpine
    ports:
      - "50050:50050"
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    depends_on:
      - api_gateway_1
      - api_gateway_2

Resource Optimization

Adjust Resource Limits

Edit .env.production:

# For high-frequency trading (HFT)
TRADING_SERVICE_CPU_LIMIT=8.0
TRADING_SERVICE_MEMORY_LIMIT=16G

# For backtesting workloads
BACKTESTING_SERVICE_CPU_LIMIT=4.0
BACKTESTING_SERVICE_MEMORY_LIMIT=8G

Enable CPU Pinning

services:
  trading_service:
    cpuset: "0-3"  # Bind to cores 0-3
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [cpu]

Service Details

API Gateway (Port 50050)

• Purpose: Central authentication and routing
• Features: JWT auth, rate limiting, MFA support
• Health: grpcurl -plaintext localhost:50050 grpc.health.v1.Health/Check
• Metrics: http://localhost:9091/metrics

Trading Service (Port 50051 - Internal)

• Purpose: Order execution and position management
• Dependencies: PostgreSQL, Redis, Vault
• Health: Internal only (via API Gateway)
• Metrics: http://[internal]:9092/metrics

Backtesting Service (Port 50052 - Internal)

• Purpose: Strategy backtesting
• Dependencies: PostgreSQL, historical data
• Health: Internal only (via API Gateway)
• Metrics: http://[internal]:9093/metrics

ML Training Service (Port 50053 - Internal)

• Purpose: Model training and inference
• Dependencies: PostgreSQL, S3, Redis
• Health: Internal only (via API Gateway)
• Metrics: http://[internal]:9094/metrics

Monitoring

Prometheus Metrics

All services expose Prometheus metrics:

# View all metrics endpoints
docker compose -f docker-compose.production.yml exec prometheus \
  cat /etc/prometheus/prometheus.yml

Grafana Dashboards

Access Grafana at http://localhost:3000:

1. HFT Trading Performance: Latency, throughput, order metrics
2. System Resources: CPU, memory, disk I/O
3. Service Health: gRPC health checks, error rates
4. Database Performance: Query times, connection pools

Log Aggregation

# View service logs
docker compose -f docker-compose.production.yml logs -f trading_service

# Filter by level
docker compose -f docker-compose.production.yml logs | grep ERROR

# Export logs
docker compose -f docker-compose.production.yml logs --since 1h > logs/trading-$(date +%Y%m%d).log

Troubleshooting

Service Won't Start

# Check service status
docker compose -f docker-compose.production.yml ps

# View detailed logs
docker compose -f docker-compose.production.yml logs trading_service

# Inspect container
docker inspect foxhunt-trading-service

Database Connection Errors

# Test PostgreSQL connection
docker compose -f docker-compose.production.yml exec postgres \
  psql -U foxhunt -c "SELECT version();"

# Check database URL
echo $DATABASE_URL

# Reset database
docker compose -f docker-compose.production.yml down -v
docker compose -f docker-compose.production.yml up -d postgres

Health Check Failures

# Install grpc_health_probe locally
wget https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.25/grpc_health_probe-linux-amd64
chmod +x grpc_health_probe-linux-amd64

# Test health check
./grpc_health_probe-linux-amd64 -addr localhost:50050

Performance Issues

# Check resource usage
docker stats

# View service metrics
curl http://localhost:9091/metrics | grep -E "(cpu|memory)"

# Analyze database performance
docker compose -f docker-compose.production.yml exec postgres \
  psql -U foxhunt -c "SELECT * FROM pg_stat_activity;"

Security

Best Practices

1. Never commit .env.production to version control
2. Use Docker secrets for production deployments
3. Enable TLS for all external connections
4. Implement network policies to restrict service communication
5. Regular security audits of dependencies and images
6. Enable audit logging for all critical operations
7. Use minimal base images (debian:bookworm-slim)
8. Run as non-root user (foxhunt:1000)

Vulnerability Scanning

# Scan images for vulnerabilities
docker scout quickview

# Detailed CVE report
docker scout cves foxhunt-api-gateway:latest

Access Control

# Restrict Docker socket access
sudo chmod 660 /var/run/docker.sock

# Use Docker rootless mode (advanced)
dockerd-rootless-setuptool.sh install

Maintenance

Backup Procedures

# Backup PostgreSQL
docker compose -f docker-compose.production.yml exec postgres \
  pg_dump -U foxhunt foxhunt > backups/foxhunt-$(date +%Y%m%d).sql

# Backup Redis
docker compose -f docker-compose.production.yml exec redis \
  redis-cli SAVE
docker cp foxhunt-redis:/data/dump.rdb backups/redis-$(date +%Y%m%d).rdb

# Backup volumes
docker run --rm -v postgres_data:/source -v $(pwd)/backups:/backup \
  alpine tar czf /backup/postgres_data-$(date +%Y%m%d).tar.gz -C /source .

Update Procedures

# Pull latest images
docker compose -f docker-compose.production.yml pull

# Graceful restart
docker compose -f docker-compose.production.yml up -d --force-recreate --no-deps api_gateway

# Full stack update
docker compose -f docker-compose.production.yml down
docker compose -f docker-compose.production.yml up -d

Support

For issues and questions:

• GitHub Issues: https://github.com/user/foxhunt/issues
• Documentation: ./docs/
• Deployment Checklist: ./deployment/DEPLOYMENT_CHECKLIST.md

---

Last Updated: 2025-10-03 (Wave 71 Agent 8) Docker Compose Version: 3.8 Tested Environments: Linux (Ubuntu 22.04), macOS (Docker Desktop 4.24+)