foxhunt/docs/deployment/TLS_CERTIFICATE_SETUP.md

TLS Certificate Setup Guide

Overview

Foxhunt HFT system uses TLS/mTLS for secure inter-service communication. This guide covers certificate generation, configuration, and troubleshooting for development and production environments.

Certificate Generation

Self-Signed Certificates (Development)

For local development and testing, self-signed certificates are sufficient:

# Create certificate directory
mkdir -p /tmp/foxhunt/certs
cd /tmp/foxhunt/certs

# Generate RSA 4096-bit private key and certificate
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout server.key \
  -out server.crt \
  -days 365 \
  -subj "/CN=localhost"

# Generate CA certificate for mTLS (mutual TLS)
openssl req -x509 -new -nodes -key server.key \
  -sha256 -days 365 -out ca.crt \
  -subj "/CN=Foxhunt CA"

# Set appropriate permissions
chmod 644 server.crt ca.crt
chmod 600 server.key

Production Certificates (Let's Encrypt or Corporate CA)

For production deployments, use properly signed certificates:

Option 1: Let's Encrypt (Free, Automated)

# Install certbot
sudo apt-get install certbot

# Generate certificate for domain
sudo certbot certonly --standalone \
  -d api.foxhunt.trading \
  -d trading.foxhunt.trading \
  --non-interactive \
  --agree-tos \
  --email ops@foxhunt.trading

# Copy certificates to Foxhunt directory
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/fullchain.pem /tmp/foxhunt/certs/server.crt
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/privkey.pem /tmp/foxhunt/certs/server.key
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/chain.pem /tmp/foxhunt/certs/ca.crt

# Set permissions
sudo chown foxhunt:foxhunt /tmp/foxhunt/certs/*
chmod 644 /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/ca.crt
chmod 600 /tmp/foxhunt/certs/server.key

Option 2: Corporate CA (Internal PKI)

# Request CSR from your CA
openssl req -new -newkey rsa:4096 -nodes \
  -keyout server.key \
  -out server.csr \
  -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=api.foxhunt.internal"

# Submit server.csr to your corporate CA
# Receive signed certificate (server.crt) and CA chain (ca.crt)

# Install certificates
cp server.crt server.key ca.crt /tmp/foxhunt/certs/
chmod 644 /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/ca.crt
chmod 600 /tmp/foxhunt/certs/server.key

Certificate Validation

Verify certificates are correctly generated:

# Verify certificate details
openssl x509 -in /tmp/foxhunt/certs/server.crt -text -noout

# Check expiration date
openssl x509 -in /tmp/foxhunt/certs/server.crt -noout -dates

# Verify private key matches certificate
openssl rsa -noout -modulus -in /tmp/foxhunt/certs/server.key | openssl md5
openssl x509 -noout -modulus -in /tmp/foxhunt/certs/server.crt | openssl md5
# Both MD5 hashes should match

# Verify CA chain
openssl verify -CAfile /tmp/foxhunt/certs/ca.crt /tmp/foxhunt/certs/server.crt

Docker Configuration

Volume Mounts

Update docker-compose.yml to mount certificates into containers:

services:
  api_gateway:
    volumes:
      - ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
    environment:
      - TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
      - TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
      - TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt

  trading_service:
    volumes:
      - ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
    environment:
      - TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
      - TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
      - TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt

  backtesting_service:
    volumes:
      - ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
    environment:
      - TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
      - TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
      - TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt

  ml_training_service:
    volumes:
      - ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
    environment:
      - TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
      - TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
      - TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt

Kubernetes Secrets (Production)

For Kubernetes deployments, use Secrets:

# Create TLS secret
kubectl create secret tls foxhunt-tls \
  --cert=/tmp/foxhunt/certs/server.crt \
  --key=/tmp/foxhunt/certs/server.key \
  --namespace=foxhunt

# Create CA secret
kubectl create secret generic foxhunt-ca \
  --from-file=ca.crt=/tmp/foxhunt/certs/ca.crt \
  --namespace=foxhunt

# Mount in deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
spec:
  template:
    spec:
      containers:
      - name: trading-service
        volumeMounts:
        - name: tls-certs
          mountPath: /tmp/foxhunt/certs
          readOnly: true
        env:
        - name: TLS_CERT_PATH
          value: /tmp/foxhunt/certs/tls.crt
        - name: TLS_KEY_PATH
          value: /tmp/foxhunt/certs/tls.key
      volumes:
      - name: tls-certs
        secret:
          secretName: foxhunt-tls

Certificate Rotation

Automated Rotation (Let's Encrypt)

# Create renewal script
cat > /etc/cron.daily/foxhunt-cert-renewal << 'EOF'
#!/bin/bash
# Renew Let's Encrypt certificates
certbot renew --quiet

# Copy to Foxhunt directory if renewed
if [ $? -eq 0 ]; then
  cp /etc/letsencrypt/live/api.foxhunt.trading/fullchain.pem /tmp/foxhunt/certs/server.crt
  cp /etc/letsencrypt/live/api.foxhunt.trading/privkey.pem /tmp/foxhunt/certs/server.key

  # Restart services to pick up new certificates
  docker-compose restart api_gateway trading_service backtesting_service ml_training_service
fi
EOF

chmod +x /etc/cron.daily/foxhunt-cert-renewal

Manual Rotation

# 1. Generate new certificates (follow generation steps above)

# 2. Backup old certificates
cp /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/server.crt.bak
cp /tmp/foxhunt/certs/server.key /tmp/foxhunt/certs/server.key.bak

# 3. Install new certificates
cp new-server.crt /tmp/foxhunt/certs/server.crt
cp new-server.key /tmp/foxhunt/certs/server.key

# 4. Restart services
docker-compose restart api_gateway trading_service backtesting_service ml_training_service

# 5. Verify services are healthy
docker-compose ps
curl -k https://localhost:50051/health

Troubleshooting

Common Issues

Error: "No such file or directory: /tmp/foxhunt/certs/server.crt"

Cause: Certificates not generated or volume mount incorrect

Fix:

# Verify certificates exist
ls -lh /tmp/foxhunt/certs/

# Check volume mount in docker-compose.yml
docker-compose config | grep -A 5 volumes

# Regenerate certificates if missing
cd /tmp/foxhunt/certs
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout server.key \
  -out server.crt \
  -days 365 \
  -subj "/CN=localhost"

Error: "Permission denied: /tmp/foxhunt/certs/server.key"

Cause: Incorrect file permissions

Fix:

# Set correct permissions
chmod 600 /tmp/foxhunt/certs/server.key
chmod 644 /tmp/foxhunt/certs/server.crt

# If running in Docker, ensure ownership matches container user
sudo chown 1000:1000 /tmp/foxhunt/certs/*

Error: "Certificate has expired"

Cause: Certificate validity period exceeded

Fix:

# Check expiration
openssl x509 -in /tmp/foxhunt/certs/server.crt -noout -dates

# Regenerate with longer validity (1 year)
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout server.key \
  -out server.crt \
  -days 365 \
  -subj "/CN=localhost"

# Restart services
docker-compose restart

Error: "Certificate verification failed"

Cause: CA certificate mismatch or missing

Fix:

# Verify certificate chain
openssl verify -CAfile /tmp/foxhunt/certs/ca.crt /tmp/foxhunt/certs/server.crt

# Ensure CA certificate is correct
openssl x509 -in /tmp/foxhunt/certs/ca.crt -text -noout

# Regenerate CA if needed
openssl req -x509 -new -nodes -key server.key \
  -sha256 -days 365 -out ca.crt \
  -subj "/CN=Foxhunt CA"

Error: "TLS handshake failed"

Cause: Protocol version mismatch or cipher suite incompatibility

Fix:

# Test TLS connection
openssl s_client -connect localhost:50051 -tls1_2

# Check supported ciphers
openssl ciphers -v 'HIGH:!aNULL:!MD5'

# Update service configuration to allow TLS 1.2+
# In service config (config/schemas.rs):
# tls_min_version: "TLS1_2"

Debugging Tools

# Test gRPC TLS connection
grpc_health_probe -addr=localhost:50051 \
  -tls \
  -tls-ca-cert=/tmp/foxhunt/certs/ca.crt \
  -tls-client-cert=/tmp/foxhunt/certs/server.crt \
  -tls-client-key=/tmp/foxhunt/certs/server.key

# View certificate details in service logs
docker logs foxhunt-api-gateway 2>&1 | grep -i tls

# Monitor TLS errors
journalctl -u docker -f | grep -i "tls\|certificate"

Security Best Practices

1. Key Protection:

   • Store private keys with 600 permissions (owner read/write only)
   • Never commit private keys to version control
   • Use hardware security modules (HSM) for production keys

2. Certificate Validation:

   • Enable strict certificate validation in production
   • Verify hostname matches certificate CN/SAN
   • Implement certificate pinning for critical services

3. Rotation Policy:

   • Rotate certificates every 90 days (Let's Encrypt default)
   • Monitor certificate expiration with alerts
   • Maintain certificate inventory in Vault

4. Cipher Suites:

   • Disable weak ciphers (MD5, DES, RC4)
   • Prefer forward secrecy (ECDHE, DHE)
   • Use TLS 1.2 or higher

Production Checklist

• Generate production certificates from trusted CA
• Configure automated renewal (certbot or corporate process)
• Set up certificate expiration monitoring
• Test TLS connections from all services
• Verify mTLS authentication works
• Document certificate locations in Vault
• Configure backup/restore procedures
• Set up alerts for TLS errors
• Test certificate rotation procedure
• Review security audit logs

References

• OpenSSL Documentation
• Let's Encrypt Best Practices
• gRPC TLS Guide
• Docker TLS Security