foxhunt/DOCKER_DEPLOYMENT.md

# Foxhunt HFT Trading System - Docker Deployment Guide

**Wave 71 Agent 8: Complete Docker Compose Production Stack**

This guide provides complete instructions for deploying the Foxhunt HFT trading system using Docker Compose.

## Table of Contents

- [Overview](#overview)
- [Architecture](#architecture)
- [Prerequisites](#prerequisites)
- [Quick Start](#quick-start)
- [Production Deployment](#production-deployment)
- [Service Details](#service-details)
- [Monitoring](#monitoring)
- [Troubleshooting](#troubleshooting)
- [Security](#security)

## Overview

The Foxhunt Docker Compose stack includes:

- **Infrastructure**: PostgreSQL, Redis, InfluxDB, Vault, Prometheus, Grafana
- **API Gateway** (Wave 70): JWT authentication, rate limiting, request routing
- **Backend Services**: Trading, Backtesting, ML Training
- **TLI Client** (optional): Terminal interface for debugging

## Architecture

### Network Topology

```
                    ┌─────────────────────┐
                    │   External Access   │
                    │   (Port 50050)      │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │   API Gateway       │
                    │  (Authentication    │
                    │   Rate Limiting)    │
                    └──────────┬──────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        │                      │                      │
┌───────▼───────┐   ┌─────────▼─────────┐   ┌───────▼────────┐
│   Trading     │   │   Backtesting     │   │  ML Training   │
│   Service     │   │   Service         │   │  Service       │
│ (Port 50051)  │   │  (Port 50052)     │   │ (Port 50053)   │
└───────┬───────┘   └─────────┬─────────┘   └────────┬───────┘
        │                     │                       │
        └─────────────────────┼───────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
┌───────▼───────┐   ┌─────────▼─────────┐   ┌──────▼────────┐
│  PostgreSQL   │   │     Redis         │   │    Vault      │
│  (Database)   │   │    (Cache)        │   │  (Secrets)    │
└───────────────┘   └───────────────────┘   └───────────────┘
```

### Service Communication

- **External Network** (`foxhunt_external`): API Gateway only
- **Internal Network** (`foxhunt_internal`): All services
- Backend services are NOT exposed to external networks
- All service communication uses gRPC with health checks

## Prerequisites

### System Requirements

- **OS**: Linux, macOS, or Windows with WSL2
- **Docker**: 24.0+ (with Compose V2)
- **CPU**: 8+ cores recommended (production: 16+ cores)
- **RAM**: 16GB minimum (production: 32GB+)
- **Disk**: 50GB+ free space

### Software Installation

```bash
# Docker and Docker Compose
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

# Verify installation
docker --version
docker compose version
```

## Quick Start

### 1. Clone Repository

```bash
git clone https://github.com/user/foxhunt.git
cd foxhunt
```

### 2. Configure Environment

```bash
# Copy environment template
cp .env.production.example .env.production

# Edit with your values (CRITICAL: Change all CHANGE_ME values)
nano .env.production
```

**Minimum required changes:**
- `POSTGRES_PASSWORD`
- `JWT_SECRET` (generate with: `openssl rand -base64 32`)
- `INFLUXDB_PASSWORD`
- `VAULT_ROOT_TOKEN`
- `GRAFANA_ADMIN_PASSWORD`

### 3. Start Infrastructure

```bash
# Start infrastructure services first
docker compose -f docker-compose.production.yml up -d postgres redis vault

# Wait for services to be healthy
docker compose -f docker-compose.production.yml ps
```

### 4. Initialize Database

```bash
# Run database migrations
docker compose -f docker-compose.production.yml exec postgres \
  psql -U foxhunt -d foxhunt -f /docker-entrypoint-initdb.d/001_trading_events.sql
```

### 5. Start All Services

```bash
# Start complete stack
docker compose -f docker-compose.production.yml up -d

# Check service health
docker compose -f docker-compose.production.yml ps
docker compose -f docker-compose.production.yml logs -f api_gateway
```

### 6. Verify Deployment

```bash
# Test API Gateway health
grpcurl -plaintext localhost:50050 grpc.health.v1.Health/Check

# Check Prometheus metrics
curl http://localhost:9091/metrics

# Access Grafana
open http://localhost:3000  # admin / [GRAFANA_ADMIN_PASSWORD]
```

## Production Deployment

### Security Hardening

#### 1. Generate Strong Secrets

```bash
# JWT Secret (32+ bytes)
openssl rand -base64 32 > secrets/jwt_secret.txt

# PostgreSQL Password
openssl rand -base64 24 > secrets/postgres_password.txt

# Redis Password
openssl rand -base64 24 > secrets/redis_password.txt
```

#### 2. TLS Certificates

```bash
# Generate self-signed certificates (development)
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout certs/server.key \
  -out certs/server.crt \
  -days 365 -subj "/CN=foxhunt.local"

# Production: Use Let's Encrypt or corporate CA
```

#### 3. Configure Firewall

```bash
# Allow only API Gateway external port
sudo ufw allow 50050/tcp comment "API Gateway"
sudo ufw deny 50051:50053/tcp comment "Block backend services"
```

### High Availability Setup

#### Database Replication

```yaml
# docker-compose.ha.yml
services:
  postgres-primary:
    image: postgres:16-alpine
    environment:
      POSTGRES_REPLICATION_MODE: master

  postgres-replica:
    image: postgres:16-alpine
    environment:
      POSTGRES_REPLICATION_MODE: slave
      POSTGRES_MASTER_HOST: postgres-primary
```

#### Load Balancing

```yaml
services:
  haproxy:
    image: haproxy:2.8-alpine
    ports:
      - "50050:50050"
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    depends_on:
      - api_gateway_1
      - api_gateway_2
```

### Resource Optimization

#### Adjust Resource Limits

Edit `.env.production`:

```bash
# For high-frequency trading (HFT)
TRADING_SERVICE_CPU_LIMIT=8.0
TRADING_SERVICE_MEMORY_LIMIT=16G

# For backtesting workloads
BACKTESTING_SERVICE_CPU_LIMIT=4.0
BACKTESTING_SERVICE_MEMORY_LIMIT=8G
```

#### Enable CPU Pinning

```yaml
services:
  trading_service:
    cpuset: "0-3"  # Bind to cores 0-3
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [cpu]
```

## Service Details

### API Gateway (Port 50050)

- **Purpose**: Central authentication and routing
- **Features**: JWT auth, rate limiting, MFA support
- **Health**: `grpcurl -plaintext localhost:50050 grpc.health.v1.Health/Check`
- **Metrics**: `http://localhost:9091/metrics`

### Trading Service (Port 50051 - Internal)

- **Purpose**: Order execution and position management
- **Dependencies**: PostgreSQL, Redis, Vault
- **Health**: Internal only (via API Gateway)
- **Metrics**: `http://[internal]:9092/metrics`

### Backtesting Service (Port 50052 - Internal)

- **Purpose**: Strategy backtesting
- **Dependencies**: PostgreSQL, historical data
- **Health**: Internal only (via API Gateway)
- **Metrics**: `http://[internal]:9093/metrics`

### ML Training Service (Port 50053 - Internal)

- **Purpose**: Model training and inference
- **Dependencies**: PostgreSQL, S3, Redis
- **Health**: Internal only (via API Gateway)
- **Metrics**: `http://[internal]:9094/metrics`

## Monitoring

### Prometheus Metrics

All services expose Prometheus metrics:

```bash
# View all metrics endpoints
docker compose -f docker-compose.production.yml exec prometheus \
  cat /etc/prometheus/prometheus.yml
```

### Grafana Dashboards

Access Grafana at `http://localhost:3000`:

1. **HFT Trading Performance**: Latency, throughput, order metrics
2. **System Resources**: CPU, memory, disk I/O
3. **Service Health**: gRPC health checks, error rates
4. **Database Performance**: Query times, connection pools

### Log Aggregation

```bash
# View service logs
docker compose -f docker-compose.production.yml logs -f trading_service

# Filter by level
docker compose -f docker-compose.production.yml logs | grep ERROR

# Export logs
docker compose -f docker-compose.production.yml logs --since 1h > logs/trading-$(date +%Y%m%d).log
```

## Troubleshooting

### Service Won't Start

```bash
# Check service status
docker compose -f docker-compose.production.yml ps

# View detailed logs
docker compose -f docker-compose.production.yml logs trading_service

# Inspect container
docker inspect foxhunt-trading-service
```

### Database Connection Errors

```bash
# Test PostgreSQL connection
docker compose -f docker-compose.production.yml exec postgres \
  psql -U foxhunt -c "SELECT version();"

# Check database URL
echo $DATABASE_URL

# Reset database
docker compose -f docker-compose.production.yml down -v
docker compose -f docker-compose.production.yml up -d postgres
```

### Health Check Failures

```bash
# Install grpc_health_probe locally
wget https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.25/grpc_health_probe-linux-amd64
chmod +x grpc_health_probe-linux-amd64

# Test health check
./grpc_health_probe-linux-amd64 -addr localhost:50050
```

### Performance Issues

```bash
# Check resource usage
docker stats

# View service metrics
curl http://localhost:9091/metrics | grep -E "(cpu|memory)"

# Analyze database performance
docker compose -f docker-compose.production.yml exec postgres \
  psql -U foxhunt -c "SELECT * FROM pg_stat_activity;"
```

## Security

### Best Practices

1. **Never commit .env.production** to version control
2. **Use Docker secrets** for production deployments
3. **Enable TLS** for all external connections
4. **Implement network policies** to restrict service communication
5. **Regular security audits** of dependencies and images
6. **Enable audit logging** for all critical operations
7. **Use minimal base images** (debian:bookworm-slim)
8. **Run as non-root user** (foxhunt:1000)

### Vulnerability Scanning

```bash
# Scan images for vulnerabilities
docker scout quickview

# Detailed CVE report
docker scout cves foxhunt-api-gateway:latest
```

### Access Control

```bash
# Restrict Docker socket access
sudo chmod 660 /var/run/docker.sock

# Use Docker rootless mode (advanced)
dockerd-rootless-setuptool.sh install
```

## Maintenance

### Backup Procedures

```bash
# Backup PostgreSQL
docker compose -f docker-compose.production.yml exec postgres \
  pg_dump -U foxhunt foxhunt > backups/foxhunt-$(date +%Y%m%d).sql

# Backup Redis
docker compose -f docker-compose.production.yml exec redis \
  redis-cli SAVE
docker cp foxhunt-redis:/data/dump.rdb backups/redis-$(date +%Y%m%d).rdb

# Backup volumes
docker run --rm -v postgres_data:/source -v $(pwd)/backups:/backup \
  alpine tar czf /backup/postgres_data-$(date +%Y%m%d).tar.gz -C /source .
```

### Update Procedures

```bash
# Pull latest images
docker compose -f docker-compose.production.yml pull

# Graceful restart
docker compose -f docker-compose.production.yml up -d --force-recreate --no-deps api_gateway

# Full stack update
docker compose -f docker-compose.production.yml down
docker compose -f docker-compose.production.yml up -d
```

## Support

For issues and questions:

- GitHub Issues: https://github.com/user/foxhunt/issues
- Documentation: `./docs/`
- Deployment Checklist: `./deployment/DEPLOYMENT_CHECKLIST.md`

---

**Last Updated**: 2025-10-03 (Wave 71 Agent 8)
**Docker Compose Version**: 3.8
**Tested Environments**: Linux (Ubuntu 22.04), macOS (Docker Desktop 4.24+)