foxhunt/DOCKER_DEPLOYMENT.md

Foxhunt HFT Trading System - Docker Production Deployment

This document provides comprehensive instructions for deploying the Foxhunt HFT Trading System using Docker Compose in production environments.

🏗️ Architecture Overview

The system is deployed using a layered Docker Compose architecture:

┌─────────────────────────────────────────────────────────────┐
│                    Frontend Network                         │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │    Nginx    │  │     TLI     │  │      Grafana        │ │
│  │   (Proxy)   │  │ (Terminal)  │  │   (Monitoring)      │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                    Backend Network                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │   Trading   │  │ ML Training │  │    Backtesting      │ │
│  │   Service   │  │   Service   │  │      Service        │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                  Database Network                           │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │ PostgreSQL  │  │    Redis    │  │      InfluxDB       │ │
│  │ (Primary)   │  │  (Cache)    │  │   (Time Series)     │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                Infrastructure Network                       │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │    Vault    │  │ Prometheus  │  │    AlertManager     │ │
│  │ (Secrets)   │  │ (Metrics)   │  │     (Alerts)        │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

🚀 Quick Start

Prerequisites

• Docker Engine 24.0+
• Docker Compose 2.20+
• 32GB RAM minimum (64GB recommended)
• 20+ CPU cores for optimal HFT performance
• 500GB+ SSD storage
• Ubuntu 22.04 LTS (recommended)

1. Environment Setup

# Clone the repository
git clone <repository-url>
cd foxhunt

# Copy and customize environment file
cp .env.production .env.production.local
vim .env.production.local  # Configure your credentials

# Create data directories
sudo mkdir -p /opt/foxhunt/{config,data,models,backtests,checkpoints}
sudo mkdir -p /opt/foxhunt/{vault,postgres,redis,influxdb}/{data,logs}
sudo mkdir -p /opt/foxhunt/monitoring/{prometheus,grafana,alertmanager,loki,tempo}
sudo mkdir -p /var/log/foxhunt
sudo chown -R $(id -u):$(id -g) /opt/foxhunt /var/log/foxhunt

2. Quick Deployment

# Full production deployment
./deploy.sh

# Or deploy components separately
./deploy.sh --infrastructure-only    # Databases and Vault first
./deploy.sh --services-only         # Application services
./deploy.sh --monitoring-only       # Monitoring stack

3. Verify Deployment

# Run comprehensive health check
./health-check.sh --detailed --performance

# Check service logs
docker-compose -f docker-compose.production.yml logs -f

📋 Deployment Options

Infrastructure Only

Deploy just the foundational services (databases, Vault, caching):

docker-compose -f docker-compose.infrastructure.yml up -d

Services Included:

• HashiCorp Vault (secrets management)
• PostgreSQL (primary database)
• Redis (caching and pub/sub)
• InfluxDB (time series data)
• PgAdmin (database administration)
• Redis Commander (Redis administration)

Monitoring Only

Deploy the complete observability stack:

docker-compose -f docker-compose.monitoring.yml up -d

Services Included:

• Prometheus (metrics collection)
• Grafana (visualization)
• AlertManager (alerting)
• Loki (log aggregation)
• Tempo (distributed tracing)
• Node Exporter (system metrics)
• cAdvisor (container metrics)
• Uptime Kuma (uptime monitoring)

Full Production

Complete deployment with all services:

docker-compose -f docker-compose.production.yml up -d

All Services:

• Application services (Trading, ML, Backtesting, TLI)
• Infrastructure services (Vault, databases)
• Monitoring stack (Prometheus, Grafana, alerts)
• Reverse proxy (Nginx)

⚙️ Configuration

Environment Variables

Critical environment variables in .env.production:

# Database Credentials
POSTGRES_USER=foxhunt
POSTGRES_PASSWORD=YourSecurePassword123!
REDIS_PASSWORD=YourRedisPassword456!
INFLUXDB_TOKEN=your-influxdb-token-here

# Vault Configuration
VAULT_ROOT_TOKEN=your-vault-root-token
VAULT_FOXHUNT_PASSWORD=YourVaultPassword789!

# Trading System
FOXHUNT_ENV=production
MAX_POSITION_SIZE=1000000
MAX_DAILY_LOSS=50000
CIRCUIT_BREAKER_ENABLED=true

# Broker API Keys (Replace with real values)
ICMARKETS_USERNAME=your_username
IB_ACCOUNT=your_account
DATABENTO_API_KEY=your_api_key

Performance Tuning

The system includes HFT-optimized configurations:

CPU Affinity:

• Trading Service: Cores 2-5 (dedicated)
• ML Training: Cores 8-13 (GPU-optimized)
• Backtesting: Cores 14-17
• TLI: Cores 18-19

Memory Limits:

• Trading Service: 4GB
• ML Training: 16GB (with GPU support)
• PostgreSQL: 2GB
• Redis: 1GB

Network Optimization:

sysctls:
  - net.core.rmem_max=134217728
  - net.core.wmem_max=134217728
  - net.ipv4.tcp_rmem=4096 65536 134217728
  - net.ipv4.tcp_wmem=4096 65536 134217728

🔒 Security Features

Network Isolation

Services are isolated across multiple Docker networks:

• frontend-network: External access (TLI, Grafana, Nginx)
• backend-network: Service communication
• database-network: Database tier isolation
• infrastructure-network: Infrastructure services
• monitoring-network: Observability stack

Secrets Management

All sensitive data is managed through HashiCorp Vault:

# Initialize Vault (done automatically)
docker exec foxhunt-vault-prod vault operator init

# Store secrets
docker exec foxhunt-vault-prod vault kv put secret/foxhunt/trading \
    broker_password="your-password" \
    api_key="your-api-key"

Resource Limits

All containers have resource limits to prevent resource exhaustion:

mem_limit: 4g
memswap_limit: 4g
cpu_count: 4
cpu_percent: 400

📊 Monitoring and Observability

Access URLs

After deployment, access monitoring interfaces:

• Grafana: http://localhost:3000 (admin/admin)
• Prometheus: http://localhost:9090
• AlertManager: http://localhost:9093
• Vault UI: http://localhost:8200
• PgAdmin: http://localhost:5050

Key Metrics

The system monitors critical HFT metrics:

• Latency: Order processing latency (target: <10ms)
• Throughput: Orders per second
• Risk: VaR, drawdown, position sizes
• System: CPU, memory, disk usage
• Network: Connection status, data feed health

Alerting Rules

Critical alerts configured:

• TradingServiceDown: Trading service unavailable
• HighLatency: Order latency >10ms
• MaxPositionSizeExceeded: Position limit breach
• DailyLossThresholdReached: Loss limit reached
• CircuitBreakerTriggered: Emergency stop activated
• MarketDataStale: Data feed issues

🔧 Management Commands

Service Management

# View all services
docker-compose -f docker-compose.production.yml ps

# View logs
docker-compose -f docker-compose.production.yml logs -f trading-service

# Restart a service
docker-compose -f docker-compose.production.yml restart trading-service

# Scale a service
docker-compose -f docker-compose.production.yml up -d --scale backtesting-service=3

Health Monitoring

# Basic health check
./health-check.sh

# Detailed health check with performance metrics
./health-check.sh --detailed --performance

# Continuous monitoring
./health-check.sh --continuous

# JSON output for automation
./health-check.sh --json

Backup and Recovery

# Full backup
./backup.sh --full

# Incremental backup
./backup.sh --incremental

# Configuration only
./backup.sh --config-only

# List backups
./backup.sh --list

# Restore from backup
./backup.sh --restore backup_20240924_123456.tar.gz

🚨 Emergency Procedures

Circuit Breaker Activation

If system issues are detected:

# Emergency stop all trading
docker exec foxhunt-trading-prod curl -X POST http://localhost:8080/emergency/stop

# Check circuit breaker status
docker exec foxhunt-trading-prod curl http://localhost:8080/status/circuit-breaker

Service Recovery

# Stop all services
docker-compose -f docker-compose.production.yml down

# Start infrastructure first
docker-compose -f docker-compose.infrastructure.yml up -d

# Wait for databases to be healthy
./health-check.sh --detailed

# Start application services
docker-compose -f docker-compose.production.yml up -d

Data Recovery

# Stop services
docker-compose -f docker-compose.production.yml down

# Restore from backup
./backup.sh --restore /path/to/backup.tar.gz

# Restart services
./deploy.sh

🔍 Troubleshooting

Common Issues

Services Won't Start:

# Check Docker daemon
sudo systemctl status docker

# Check logs
docker-compose -f docker-compose.production.yml logs

# Check resource usage
docker system df
docker system prune  # Clean up if needed

High Latency:

# Check system load
htop

# Check network
netstat -i

# Check Docker networking
docker network ls
docker network inspect foxhunt-backend

Database Connection Issues:

# Test PostgreSQL
docker exec foxhunt-postgres-prod pg_isready -U foxhunt

# Test Redis
docker exec foxhunt-redis-prod redis-cli ping

# Check network connectivity
docker exec foxhunt-trading-prod nc -z foxhunt-postgres 5432

Performance Tuning

For High-Frequency Trading:

1. Enable CPU Isolation:

# Add to kernel parameters
sudo vim /etc/default/grub
# Add: isolcpus=2-19 nohz_full=2-19 rcu_nocbs=2-19
sudo update-grub
sudo reboot

2. Disable CPU Frequency Scaling:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

3. Optimize Network:

# Increase network buffers
echo 'net.core.rmem_max = 134217728' | sudo tee -a /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

📈 Scaling

Horizontal Scaling

# Scale backtesting service
docker-compose -f docker-compose.production.yml up -d --scale backtesting-service=3

# Scale ML training (with multiple GPUs)
docker-compose -f docker-compose.production.yml up -d --scale ml-training-service=2

Load Balancing

Nginx is configured for load balancing:

upstream trading_backend {
    server foxhunt-trading-1:8080;
    server foxhunt-trading-2:8080;
    server foxhunt-trading-3:8080;
}

🔐 Security Best Practices

1. Change Default Passwords: Update all default passwords in .env.production.local
2. Enable TLS: Configure TLS certificates for production
3. Network Firewall: Restrict external access to necessary ports only
4. Regular Updates: Keep Docker images and base OS updated
5. Audit Logs: Monitor all audit trails and access logs
6. Backup Encryption: Encrypt all backup files
7. Access Control: Use proper RBAC for all services

📞 Support

For deployment issues:

1. Check service logs: docker-compose logs <service-name>
2. Run health check: ./health-check.sh --detailed
3. Review monitoring dashboards in Grafana
4. Check system resources and network connectivity
5. Consult troubleshooting section above

📝 Changelog

• v1.0.0: Initial production deployment
• v1.1.0: Added monitoring stack
• v1.2.0: Enhanced security with Vault integration
• v1.3.0: Added backup and recovery automation

---

⚡ Production-Ready HFT Trading System with Docker Compose

This deployment provides enterprise-grade reliability, security, and performance optimized for high-frequency trading workloads.