foxhunt/docs/DOCKER_SECRETS.md

# Docker Secrets Management Guide

**Last Updated**: 2025-10-12  
**Status**: Production Ready  
**Applies To**: Docker Swarm deployments  

---

## Overview

This document provides comprehensive guidance for managing secrets in the Foxhunt HFT trading system using Docker Swarm secrets. Docker secrets provide a secure way to store sensitive information like passwords, API keys, and certificates without exposing them in environment variables or configuration files.

**Security Benefits**:
- Secrets encrypted at rest
- Secrets encrypted in transit between manager and worker nodes
- Secrets mounted as read-only files in containers
- Secrets never exposed in `docker inspect` or environment variables
- Fine-grained access control per service

---

## Prerequisites

### 1. Docker Swarm Initialization

```bash
# Initialize Docker Swarm (run on manager node)
docker swarm init

# For multi-node clusters, add worker nodes:
docker swarm join-token worker
# Copy the token command and run on worker nodes
```

### 2. Required Secrets

The Foxhunt system requires the following secrets for production deployment:

| Secret Name | Purpose | Required By | Generation Method |
|------------|---------|-------------|-------------------|
| `foxhunt_jwt_secret` | JWT authentication | API Gateway | `openssl rand -base64 96` |
| `foxhunt_postgres_user` | PostgreSQL username | All services | Manual |
| `foxhunt_postgres_password` | PostgreSQL password | All services | `openssl rand -base64 32` |
| `foxhunt_redis_password` | Redis authentication | All services | `openssl rand -base64 32` |
| `foxhunt_vault_token` | Vault access token | All services | Vault CLI |
| `foxhunt_influxdb_token` | InfluxDB admin token | InfluxDB, Monitoring | `openssl rand -base64 32` |
| `foxhunt_aws_access_key_id` | AWS S3 access | Backtesting, ML | AWS Console |
| `foxhunt_aws_secret_access_key` | AWS S3 secret | Backtesting, ML | AWS Console |
| `foxhunt_benzinga_api_key` | Benzinga market data | Backtesting | Benzinga Dashboard |
| `foxhunt_tls_cert` | TLS certificate | All services | Certificate Authority |
| `foxhunt_tls_key` | TLS private key | All services | Certificate Authority |
| `foxhunt_tls_ca` | TLS CA bundle | All services | Certificate Authority |

---

## Creating Secrets

### Method 1: From File (Recommended for Certificates)

```bash
# Create secret from file
docker secret create foxhunt_tls_cert /path/to/tls/cert.pem
docker secret create foxhunt_tls_key /path/to/tls/key.pem
docker secret create foxhunt_tls_ca /path/to/tls/ca-bundle.pem
```

### Method 2: From Standard Input (Recommended for Passwords)

```bash
# Generate and create JWT secret
openssl rand -base64 96 | docker secret create foxhunt_jwt_secret -

# Create PostgreSQL credentials
echo "foxhunt_prod" | docker secret create foxhunt_postgres_user -
openssl rand -base64 32 | docker secret create foxhunt_postgres_password -

# Create Redis password
openssl rand -base64 32 | docker secret create foxhunt_redis_password -

# Create InfluxDB token
openssl rand -base64 32 | docker secret create foxhunt_influxdb_token -

# Create Vault token (from Vault CLI)
vault token create -format=json | jq -r '.auth.client_token' | \
  docker secret create foxhunt_vault_token -
```

### Method 3: From AWS Credentials

```bash
# Create AWS credentials from ~/.aws/credentials
cat ~/.aws/credentials | grep aws_access_key_id | cut -d'=' -f2 | tr -d ' ' | \
  docker secret create foxhunt_aws_access_key_id -

cat ~/.aws/credentials | grep aws_secret_access_key | cut -d'=' -f2 | tr -d ' ' | \
  docker secret create foxhunt_aws_secret_access_key -
```

### Method 4: From Environment Variables

```bash
# Create Benzinga API key from environment
echo "$BENZINGA_API_KEY" | docker secret create foxhunt_benzinga_api_key -
```

---

## Complete Setup Script

Create a script `setup-secrets.sh` for automated secret creation:

```bash
#!/bin/bash
set -euo pipefail

echo "🔐 Foxhunt Docker Secrets Setup"
echo "================================"

# Function to create secret from prompt
create_secret_from_input() {
    local secret_name=$1
    local prompt=$2
    
    if docker secret inspect "$secret_name" &>/dev/null; then
        echo "✓ Secret $secret_name already exists"
        return 0
    fi
    
    echo -n "$prompt: "
    read -s secret_value
    echo
    echo "$secret_value" | docker secret create "$secret_name" -
    echo "✓ Created secret: $secret_name"
}

# Function to create secret from file
create_secret_from_file() {
    local secret_name=$1
    local file_path=$2
    
    if docker secret inspect "$secret_name" &>/dev/null; then
        echo "✓ Secret $secret_name already exists"
        return 0
    fi
    
    if [ ! -f "$file_path" ]; then
        echo "❌ File not found: $file_path"
        return 1
    fi
    
    docker secret create "$secret_name" "$file_path"
    echo "✓ Created secret: $secret_name from $file_path"
}

# Function to generate and create secret
create_secret_generated() {
    local secret_name=$1
    local length=${2:-32}
    
    if docker secret inspect "$secret_name" &>/dev/null; then
        echo "✓ Secret $secret_name already exists"
        return 0
    fi
    
    openssl rand -base64 "$length" | docker secret create "$secret_name" -
    echo "✓ Generated secret: $secret_name"
}

echo ""
echo "Step 1: JWT Authentication"
echo "---------------------------"
create_secret_generated foxhunt_jwt_secret 96

echo ""
echo "Step 2: Database Credentials"
echo "-----------------------------"
create_secret_from_input foxhunt_postgres_user "PostgreSQL username"
create_secret_generated foxhunt_postgres_password 32

echo ""
echo "Step 3: Redis Password"
echo "----------------------"
create_secret_generated foxhunt_redis_password 32

echo ""
echo "Step 4: Vault Token"
echo "-------------------"
create_secret_from_input foxhunt_vault_token "Vault token"

echo ""
echo "Step 5: InfluxDB Token"
echo "----------------------"
create_secret_generated foxhunt_influxdb_token 32

echo ""
echo "Step 6: AWS Credentials"
echo "-----------------------"
create_secret_from_input foxhunt_aws_access_key_id "AWS Access Key ID"
create_secret_from_input foxhunt_aws_secret_access_key "AWS Secret Access Key"

echo ""
echo "Step 7: Benzinga API Key"
echo "------------------------"
create_secret_from_input foxhunt_benzinga_api_key "Benzinga API Key"

echo ""
echo "Step 8: TLS Certificates"
echo "------------------------"
create_secret_from_file foxhunt_tls_cert "./certs/server.crt"
create_secret_from_file foxhunt_tls_key "./certs/server.key"
create_secret_from_file foxhunt_tls_ca "./certs/ca-bundle.crt"

echo ""
echo "✅ All secrets created successfully!"
echo ""
echo "📋 List all secrets:"
docker secret ls | grep foxhunt

echo ""
echo "🚀 Ready to deploy:"
echo "   docker stack deploy -c docker-compose.prod.yml foxhunt"
```

Make the script executable and run it:

```bash
chmod +x setup-secrets.sh
./setup-secrets.sh
```

---

## Service Configuration

### Accessing Secrets in Services

Secrets are mounted as files in `/run/secrets/` inside containers. Services should read secrets from files instead of environment variables.

#### Example: Rust Code

```rust
use std::fs;

// Read JWT secret from file
pub fn load_jwt_secret() -> Result<String, std::io::Error> {
    let secret_path = std::env::var("JWT_SECRET_FILE")
        .unwrap_or_else(|_| "/run/secrets/jwt_secret".to_string());
    
    fs::read_to_string(secret_path)
        .map(|s| s.trim().to_string())
}

// Read database credentials
pub fn load_database_url() -> Result<String, std::io::Error> {
    let user = fs::read_to_string("/run/secrets/postgres_user")?
        .trim()
        .to_string();
    let password = fs::read_to_string("/run/secrets/postgres_password")?
        .trim()
        .to_string();
    
    Ok(format!(
        "postgresql://{}:{}@postgres:5432/foxhunt",
        user, password
    ))
}

// Read Redis password
pub fn load_redis_url() -> Result<String, std::io::Error> {
    let password = fs::read_to_string("/run/secrets/redis_password")?
        .trim()
        .to_string();
    
    Ok(format!("redis://:{}@redis:6379", password))
}
```

#### Example: Docker Compose Service

```yaml
services:
  api_gateway:
    image: foxhunt/api-gateway:latest
    secrets:
      - jwt_secret
      - postgres_password
      - postgres_user
      - redis_password
    environment:
      # Point to secret files instead of values
      - JWT_SECRET_FILE=/run/secrets/jwt_secret
      - DATABASE_URL_FILE=/run/secrets/postgres_password
      - REDIS_PASSWORD_FILE=/run/secrets/redis_password
```

---

## Secret Management Operations

### List Secrets

```bash
# List all secrets
docker secret ls

# Filter Foxhunt secrets
docker secret ls | grep foxhunt
```

### Inspect Secret Metadata

```bash
# View secret metadata (NOT the actual value)
docker secret inspect foxhunt_jwt_secret

# View secret metadata in JSON format
docker secret inspect foxhunt_jwt_secret --format json | jq
```

### Update Secrets (Rotation)

Docker secrets are immutable. To update a secret:

1. **Create new secret with different name**:
```bash
openssl rand -base64 96 | docker secret create foxhunt_jwt_secret_v2 -
```

2. **Update service to use new secret**:
```bash
# Update docker-compose.prod.yml to reference new secret
# Then deploy with:
docker stack deploy -c docker-compose.prod.yml foxhunt
```

3. **Remove old secret after validation**:
```bash
docker secret rm foxhunt_jwt_secret
```

### Delete Secrets

```bash
# Remove a single secret (requires services to be stopped)
docker secret rm foxhunt_jwt_secret

# Remove all Foxhunt secrets
docker secret ls | grep foxhunt | awk '{print $1}' | xargs docker secret rm
```

---

## Production Deployment

### 1. Initialize Swarm Cluster

```bash
# On manager node
docker swarm init --advertise-addr <MANAGER-IP>

# Add worker nodes (run on each worker)
docker swarm join --token <WORKER-TOKEN> <MANAGER-IP>:2377
```

### 2. Create Secrets

```bash
# Run setup script
./setup-secrets.sh

# Or manually create each secret
```

### 3. Deploy Stack

```bash
# Deploy with version tag
VERSION=v1.0.0 docker stack deploy -c docker-compose.prod.yml foxhunt

# Monitor deployment
watch docker stack ps foxhunt
```

### 4. Verify Secrets Access

```bash
# Check if services can access secrets
docker exec $(docker ps -q -f name=foxhunt_api_gateway) \
  ls -la /run/secrets/

# Verify secret content (should show file permissions)
docker exec $(docker ps -q -f name=foxhunt_api_gateway) \
  cat /run/secrets/jwt_secret
```

---

## Security Best Practices

### 1. Secret Rotation Policy

Implement regular secret rotation:

- **JWT secrets**: Rotate every 90 days
- **Database passwords**: Rotate every 180 days
- **API keys**: Rotate annually or when compromised
- **TLS certificates**: Renew before expiration (typically 1 year)

### 2. Access Control

```bash
# Only manager nodes can manage secrets
# Worker nodes only receive secrets for their assigned services

# Verify node roles
docker node ls

# Promote worker to manager (if needed)
docker node promote <NODE-ID>
```

### 3. Audit Logging

Enable Docker audit logging for secret access:

```bash
# Configure Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "authorization-plugins": ["authz-broker"],
  "audit": true
}
EOF

# Restart Docker
systemctl restart docker
```

### 4. Backup Secrets

```bash
# Backup secret IDs and metadata (NOT values)
docker secret ls --format "{{.ID}} {{.Name}}" > secrets-backup.txt

# Store actual secret values in secure vault
# DO NOT store in git or plain text files
```

### 5. Secret Injection Prevention

Never log or expose secrets:

```rust
// ❌ BAD: Logging secrets
println!("JWT secret: {}", jwt_secret);

// ✅ GOOD: Log only metadata
println!("JWT secret loaded: {} bytes", jwt_secret.len());

// ❌ BAD: Include in error messages
return Err(format!("Invalid secret: {}", secret));

// ✅ GOOD: Generic error
return Err("Invalid secret format".to_string());
```

---

## Troubleshooting

### Secret Not Found

```bash
# Verify secret exists
docker secret ls | grep <SECRET_NAME>

# Check service configuration
docker service inspect <SERVICE_NAME> --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}'
```

### Permission Denied

```bash
# Verify secret is assigned to service
docker service ps <SERVICE_NAME> --format '{{.Error}}'

# Check secret file permissions inside container
docker exec <CONTAINER_ID> ls -la /run/secrets/
```

### Secret Update Not Applied

```bash
# Force service update
docker service update --force <SERVICE_NAME>

# Or redeploy entire stack
docker stack deploy -c docker-compose.prod.yml foxhunt
```

### Secret Rotation Issues

```bash
# Create new version
docker secret create <SECRET_NAME>_v2 <NEW_VALUE>

# Update service to use new version
docker service update --secret-rm <SECRET_NAME> \
                     --secret-add source=<SECRET_NAME>_v2,target=<SECRET_NAME> \
                     <SERVICE_NAME>
```

---

## Kubernetes Alternative

For Kubernetes deployments, use Kubernetes Secrets instead:

```yaml
# Create Kubernetes secret
apiVersion: v1
kind: Secret
metadata:
  name: foxhunt-jwt-secret
type: Opaque
data:
  jwt_secret: <BASE64_ENCODED_VALUE>

# Reference in pod
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: api-gateway
    env:
    - name: JWT_SECRET
      valueFrom:
        secretKeyRef:
          name: foxhunt-jwt-secret
          key: jwt_secret
```

See `docs/KUBERNETES_DEPLOYMENT.md` for full Kubernetes guide.

---

## Migration from Environment Variables

To migrate from `.env` files to Docker secrets:

### 1. Audit Current Environment Variables

```bash
# List environment variables in running container
docker exec <CONTAINER_ID> env | grep -i secret
```

### 2. Create Secrets

```bash
# For each sensitive env var, create a secret
echo "$ENV_VAR_VALUE" | docker secret create <SECRET_NAME> -
```

### 3. Update Service Configuration

```yaml
# Before (environment variables)
services:
  api_gateway:
    environment:
      - JWT_SECRET=${JWT_SECRET}

# After (Docker secrets)
services:
  api_gateway:
    secrets:
      - jwt_secret
    environment:
      - JWT_SECRET_FILE=/run/secrets/jwt_secret
```

### 4. Update Application Code

```rust
// Before
let jwt_secret = std::env::var("JWT_SECRET")?;

// After
let jwt_secret = std::fs::read_to_string(
    std::env::var("JWT_SECRET_FILE")
        .unwrap_or("/run/secrets/jwt_secret".to_string())
)?.trim().to_string();
```

---

## Monitoring and Compliance

### Secret Access Monitoring

```bash
# Monitor secret access in container logs
docker service logs <SERVICE_NAME> | grep -i secret

# Check for failed secret reads
docker service ps <SERVICE_NAME> --filter "desired-state=shutdown"
```

### Compliance Reporting

Generate compliance report for secret rotation:

```bash
#!/bin/bash
echo "Secret Rotation Compliance Report"
echo "=================================="
echo ""

docker secret ls --format "{{.ID}},{{.Name}},{{.CreatedAt}}" | while IFS=',' read -r id name created; do
    age_days=$(( ($(date +%s) - $(date -d "$created" +%s)) / 86400 ))
    
    if [ $age_days -gt 90 ]; then
        echo "⚠️  $name: $age_days days old (ROTATION REQUIRED)"
    else
        echo "✅ $name: $age_days days old"
    fi
done
```

---

## References

- [Docker Secrets Documentation](https://docs.docker.com/engine/swarm/secrets/)
- [Docker Swarm Mode](https://docs.docker.com/engine/swarm/)
- [Foxhunt DEPLOYMENT.md](./DEPLOYMENT.md)
- [Foxhunt SECURITY.md](./SECURITY.md)
- [CLAUDE.md Production Deployment](../CLAUDE.md#production-deployment)

---

**Last Updated**: 2025-10-12  
**Maintained By**: Foxhunt Platform Team  
**Review Frequency**: Quarterly