foxhunt/docs/deployment/TLS_CERTIFICATE_SETUP.md

# TLS Certificate Setup Guide

## Overview
Foxhunt HFT system uses TLS/mTLS for secure inter-service communication. This guide covers certificate generation, configuration, and troubleshooting for development and production environments.

## Certificate Generation

### Self-Signed Certificates (Development)

For local development and testing, self-signed certificates are sufficient:

```bash
# Create certificate directory
mkdir -p /tmp/foxhunt/certs
cd /tmp/foxhunt/certs

# Generate RSA 4096-bit private key and certificate
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout server.key \
  -out server.crt \
  -days 365 \
  -subj "/CN=localhost"

# Generate CA certificate for mTLS (mutual TLS)
openssl req -x509 -new -nodes -key server.key \
  -sha256 -days 365 -out ca.crt \
  -subj "/CN=Foxhunt CA"

# Set appropriate permissions
chmod 644 server.crt ca.crt
chmod 600 server.key
```

### Production Certificates (Let's Encrypt or Corporate CA)

For production deployments, use properly signed certificates:

#### Option 1: Let's Encrypt (Free, Automated)
```bash
# Install certbot
sudo apt-get install certbot

# Generate certificate for domain
sudo certbot certonly --standalone \
  -d api.foxhunt.trading \
  -d trading.foxhunt.trading \
  --non-interactive \
  --agree-tos \
  --email ops@foxhunt.trading

# Copy certificates to Foxhunt directory
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/fullchain.pem /tmp/foxhunt/certs/server.crt
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/privkey.pem /tmp/foxhunt/certs/server.key
sudo cp /etc/letsencrypt/live/api.foxhunt.trading/chain.pem /tmp/foxhunt/certs/ca.crt

# Set permissions
sudo chown foxhunt:foxhunt /tmp/foxhunt/certs/*
chmod 644 /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/ca.crt
chmod 600 /tmp/foxhunt/certs/server.key
```

#### Option 2: Corporate CA (Internal PKI)
```bash
# Request CSR from your CA
openssl req -new -newkey rsa:4096 -nodes \
  -keyout server.key \
  -out server.csr \
  -subj "/C=US/ST=NY/L=NYC/O=Foxhunt/CN=api.foxhunt.internal"

# Submit server.csr to your corporate CA
# Receive signed certificate (server.crt) and CA chain (ca.crt)

# Install certificates
cp server.crt server.key ca.crt /tmp/foxhunt/certs/
chmod 644 /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/ca.crt
chmod 600 /tmp/foxhunt/certs/server.key
```

### Certificate Validation

Verify certificates are correctly generated:

```bash
# Verify certificate details
openssl x509 -in /tmp/foxhunt/certs/server.crt -text -noout

# Check expiration date
openssl x509 -in /tmp/foxhunt/certs/server.crt -noout -dates

# Verify private key matches certificate
openssl rsa -noout -modulus -in /tmp/foxhunt/certs/server.key | openssl md5
openssl x509 -noout -modulus -in /tmp/foxhunt/certs/server.crt | openssl md5
# Both MD5 hashes should match

# Verify CA chain
openssl verify -CAfile /tmp/foxhunt/certs/ca.crt /tmp/foxhunt/certs/server.crt
```

## Docker Configuration

### Volume Mounts

Update `docker-compose.yml` to mount certificates into containers:

```yaml
services:
  api_gateway:
    volumes:
      - ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
    environment:
      - TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
      - TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
      - TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt

  trading_service:
    volumes:
      - ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
    environment:
      - TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
      - TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
      - TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt

  backtesting_service:
    volumes:
      - ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
    environment:
      - TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
      - TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
      - TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt

  ml_training_service:
    volumes:
      - ./tmp/foxhunt/certs:/tmp/foxhunt/certs:ro
    environment:
      - TLS_CERT_PATH=/tmp/foxhunt/certs/server.crt
      - TLS_KEY_PATH=/tmp/foxhunt/certs/server.key
      - TLS_CA_PATH=/tmp/foxhunt/certs/ca.crt
```

### Kubernetes Secrets (Production)

For Kubernetes deployments, use Secrets:

```yaml
# Create TLS secret
kubectl create secret tls foxhunt-tls \
  --cert=/tmp/foxhunt/certs/server.crt \
  --key=/tmp/foxhunt/certs/server.key \
  --namespace=foxhunt

# Create CA secret
kubectl create secret generic foxhunt-ca \
  --from-file=ca.crt=/tmp/foxhunt/certs/ca.crt \
  --namespace=foxhunt

# Mount in deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
spec:
  template:
    spec:
      containers:
      - name: trading-service
        volumeMounts:
        - name: tls-certs
          mountPath: /tmp/foxhunt/certs
          readOnly: true
        env:
        - name: TLS_CERT_PATH
          value: /tmp/foxhunt/certs/tls.crt
        - name: TLS_KEY_PATH
          value: /tmp/foxhunt/certs/tls.key
      volumes:
      - name: tls-certs
        secret:
          secretName: foxhunt-tls
```

## Certificate Rotation

### Automated Rotation (Let's Encrypt)

```bash
# Create renewal script
cat > /etc/cron.daily/foxhunt-cert-renewal << 'EOF'
#!/bin/bash
# Renew Let's Encrypt certificates
certbot renew --quiet

# Copy to Foxhunt directory if renewed
if [ $? -eq 0 ]; then
  cp /etc/letsencrypt/live/api.foxhunt.trading/fullchain.pem /tmp/foxhunt/certs/server.crt
  cp /etc/letsencrypt/live/api.foxhunt.trading/privkey.pem /tmp/foxhunt/certs/server.key

  # Restart services to pick up new certificates
  docker-compose restart api_gateway trading_service backtesting_service ml_training_service
fi
EOF

chmod +x /etc/cron.daily/foxhunt-cert-renewal
```

### Manual Rotation

```bash
# 1. Generate new certificates (follow generation steps above)

# 2. Backup old certificates
cp /tmp/foxhunt/certs/server.crt /tmp/foxhunt/certs/server.crt.bak
cp /tmp/foxhunt/certs/server.key /tmp/foxhunt/certs/server.key.bak

# 3. Install new certificates
cp new-server.crt /tmp/foxhunt/certs/server.crt
cp new-server.key /tmp/foxhunt/certs/server.key

# 4. Restart services
docker-compose restart api_gateway trading_service backtesting_service ml_training_service

# 5. Verify services are healthy
docker-compose ps
curl -k https://localhost:50051/health
```

## Troubleshooting

### Common Issues

#### Error: "No such file or directory: /tmp/foxhunt/certs/server.crt"

**Cause**: Certificates not generated or volume mount incorrect

**Fix**:
```bash
# Verify certificates exist
ls -lh /tmp/foxhunt/certs/

# Check volume mount in docker-compose.yml
docker-compose config | grep -A 5 volumes

# Regenerate certificates if missing
cd /tmp/foxhunt/certs
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout server.key \
  -out server.crt \
  -days 365 \
  -subj "/CN=localhost"
```

#### Error: "Permission denied: /tmp/foxhunt/certs/server.key"

**Cause**: Incorrect file permissions

**Fix**:
```bash
# Set correct permissions
chmod 600 /tmp/foxhunt/certs/server.key
chmod 644 /tmp/foxhunt/certs/server.crt

# If running in Docker, ensure ownership matches container user
sudo chown 1000:1000 /tmp/foxhunt/certs/*
```

#### Error: "Certificate has expired"

**Cause**: Certificate validity period exceeded

**Fix**:
```bash
# Check expiration
openssl x509 -in /tmp/foxhunt/certs/server.crt -noout -dates

# Regenerate with longer validity (1 year)
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout server.key \
  -out server.crt \
  -days 365 \
  -subj "/CN=localhost"

# Restart services
docker-compose restart
```

#### Error: "Certificate verification failed"

**Cause**: CA certificate mismatch or missing

**Fix**:
```bash
# Verify certificate chain
openssl verify -CAfile /tmp/foxhunt/certs/ca.crt /tmp/foxhunt/certs/server.crt

# Ensure CA certificate is correct
openssl x509 -in /tmp/foxhunt/certs/ca.crt -text -noout

# Regenerate CA if needed
openssl req -x509 -new -nodes -key server.key \
  -sha256 -days 365 -out ca.crt \
  -subj "/CN=Foxhunt CA"
```

#### Error: "TLS handshake failed"

**Cause**: Protocol version mismatch or cipher suite incompatibility

**Fix**:
```bash
# Test TLS connection
openssl s_client -connect localhost:50051 -tls1_2

# Check supported ciphers
openssl ciphers -v 'HIGH:!aNULL:!MD5'

# Update service configuration to allow TLS 1.2+
# In service config (config/schemas.rs):
# tls_min_version: "TLS1_2"
```

### Debugging Tools

```bash
# Test gRPC TLS connection
grpc_health_probe -addr=localhost:50051 \
  -tls \
  -tls-ca-cert=/tmp/foxhunt/certs/ca.crt \
  -tls-client-cert=/tmp/foxhunt/certs/server.crt \
  -tls-client-key=/tmp/foxhunt/certs/server.key

# View certificate details in service logs
docker logs foxhunt-api-gateway 2>&1 | grep -i tls

# Monitor TLS errors
journalctl -u docker -f | grep -i "tls\|certificate"
```

## Security Best Practices

1. **Key Protection**:
   - Store private keys with `600` permissions (owner read/write only)
   - Never commit private keys to version control
   - Use hardware security modules (HSM) for production keys

2. **Certificate Validation**:
   - Enable strict certificate validation in production
   - Verify hostname matches certificate CN/SAN
   - Implement certificate pinning for critical services

3. **Rotation Policy**:
   - Rotate certificates every 90 days (Let's Encrypt default)
   - Monitor certificate expiration with alerts
   - Maintain certificate inventory in Vault

4. **Cipher Suites**:
   - Disable weak ciphers (MD5, DES, RC4)
   - Prefer forward secrecy (ECDHE, DHE)
   - Use TLS 1.2 or higher

## Production Checklist

- [ ] Generate production certificates from trusted CA
- [ ] Configure automated renewal (certbot or corporate process)
- [ ] Set up certificate expiration monitoring
- [ ] Test TLS connections from all services
- [ ] Verify mTLS authentication works
- [ ] Document certificate locations in Vault
- [ ] Configure backup/restore procedures
- [ ] Set up alerts for TLS errors
- [ ] Test certificate rotation procedure
- [ ] Review security audit logs

## References

- [OpenSSL Documentation](https://www.openssl.org/docs/)
- [Let's Encrypt Best Practices](https://letsencrypt.org/docs/)
- [gRPC TLS Guide](https://grpc.io/docs/guides/auth/)
- [Docker TLS Security](https://docs.docker.com/engine/security/certificates/)