foxhunt/docs/DOCKER_SECRETS.md

Docker Secrets Management Guide

Last Updated: 2025-10-12
Status: Production Ready
Applies To: Docker Swarm deployments

---

Overview

This document provides comprehensive guidance for managing secrets in the Foxhunt HFT trading system using Docker Swarm secrets. Docker secrets provide a secure way to store sensitive information like passwords, API keys, and certificates without exposing them in environment variables or configuration files.

Security Benefits:

• Secrets encrypted at rest
• Secrets encrypted in transit between manager and worker nodes
• Secrets mounted as read-only files in containers
• Secrets never exposed in docker inspect or environment variables
• Fine-grained access control per service

---

Prerequisites

1. Docker Swarm Initialization

# Initialize Docker Swarm (run on manager node)
docker swarm init

# For multi-node clusters, add worker nodes:
docker swarm join-token worker
# Copy the token command and run on worker nodes

2. Required Secrets

The Foxhunt system requires the following secrets for production deployment:

Secret Name	Purpose	Required By	Generation Method
foxhunt_jwt_secret	JWT authentication	API Gateway	openssl rand -base64 96
foxhunt_postgres_user	PostgreSQL username	All services	Manual
foxhunt_postgres_password	PostgreSQL password	All services	openssl rand -base64 32
foxhunt_redis_password	Redis authentication	All services	openssl rand -base64 32
foxhunt_vault_token	Vault access token	All services	Vault CLI
foxhunt_influxdb_token	InfluxDB admin token	InfluxDB, Monitoring	openssl rand -base64 32
foxhunt_aws_access_key_id	AWS S3 access	Backtesting, ML	AWS Console
foxhunt_aws_secret_access_key	AWS S3 secret	Backtesting, ML	AWS Console
foxhunt_benzinga_api_key	Benzinga market data	Backtesting	Benzinga Dashboard
foxhunt_tls_cert	TLS certificate	All services	Certificate Authority
foxhunt_tls_key	TLS private key	All services	Certificate Authority
foxhunt_tls_ca	TLS CA bundle	All services	Certificate Authority

---

Creating Secrets

Method 1: From File (Recommended for Certificates)

# Create secret from file
docker secret create foxhunt_tls_cert /path/to/tls/cert.pem
docker secret create foxhunt_tls_key /path/to/tls/key.pem
docker secret create foxhunt_tls_ca /path/to/tls/ca-bundle.pem

Method 2: From Standard Input (Recommended for Passwords)

# Generate and create JWT secret
openssl rand -base64 96 | docker secret create foxhunt_jwt_secret -

# Create PostgreSQL credentials
echo "foxhunt_prod" | docker secret create foxhunt_postgres_user -
openssl rand -base64 32 | docker secret create foxhunt_postgres_password -

# Create Redis password
openssl rand -base64 32 | docker secret create foxhunt_redis_password -

# Create InfluxDB token
openssl rand -base64 32 | docker secret create foxhunt_influxdb_token -

# Create Vault token (from Vault CLI)
vault token create -format=json | jq -r '.auth.client_token' | \
  docker secret create foxhunt_vault_token -

Method 3: From AWS Credentials

# Create AWS credentials from ~/.aws/credentials
cat ~/.aws/credentials | grep aws_access_key_id | cut -d'=' -f2 | tr -d ' ' | \
  docker secret create foxhunt_aws_access_key_id -

cat ~/.aws/credentials | grep aws_secret_access_key | cut -d'=' -f2 | tr -d ' ' | \
  docker secret create foxhunt_aws_secret_access_key -

Method 4: From Environment Variables

# Create Benzinga API key from environment
echo "$BENZINGA_API_KEY" | docker secret create foxhunt_benzinga_api_key -

---

Complete Setup Script

Create a script setup-secrets.sh for automated secret creation:

#!/bin/bash
set -euo pipefail

echo "🔐 Foxhunt Docker Secrets Setup"
echo "================================"

# Function to create secret from prompt
create_secret_from_input() {
    local secret_name=$1
    local prompt=$2
    
    if docker secret inspect "$secret_name" &>/dev/null; then
        echo "✓ Secret $secret_name already exists"
        return 0
    fi
    
    echo -n "$prompt: "
    read -s secret_value
    echo
    echo "$secret_value" | docker secret create "$secret_name" -
    echo "✓ Created secret: $secret_name"
}

# Function to create secret from file
create_secret_from_file() {
    local secret_name=$1
    local file_path=$2
    
    if docker secret inspect "$secret_name" &>/dev/null; then
        echo "✓ Secret $secret_name already exists"
        return 0
    fi
    
    if [ ! -f "$file_path" ]; then
        echo "❌ File not found: $file_path"
        return 1
    fi
    
    docker secret create "$secret_name" "$file_path"
    echo "✓ Created secret: $secret_name from $file_path"
}

# Function to generate and create secret
create_secret_generated() {
    local secret_name=$1
    local length=${2:-32}
    
    if docker secret inspect "$secret_name" &>/dev/null; then
        echo "✓ Secret $secret_name already exists"
        return 0
    fi
    
    openssl rand -base64 "$length" | docker secret create "$secret_name" -
    echo "✓ Generated secret: $secret_name"
}

echo ""
echo "Step 1: JWT Authentication"
echo "---------------------------"
create_secret_generated foxhunt_jwt_secret 96

echo ""
echo "Step 2: Database Credentials"
echo "-----------------------------"
create_secret_from_input foxhunt_postgres_user "PostgreSQL username"
create_secret_generated foxhunt_postgres_password 32

echo ""
echo "Step 3: Redis Password"
echo "----------------------"
create_secret_generated foxhunt_redis_password 32

echo ""
echo "Step 4: Vault Token"
echo "-------------------"
create_secret_from_input foxhunt_vault_token "Vault token"

echo ""
echo "Step 5: InfluxDB Token"
echo "----------------------"
create_secret_generated foxhunt_influxdb_token 32

echo ""
echo "Step 6: AWS Credentials"
echo "-----------------------"
create_secret_from_input foxhunt_aws_access_key_id "AWS Access Key ID"
create_secret_from_input foxhunt_aws_secret_access_key "AWS Secret Access Key"

echo ""
echo "Step 7: Benzinga API Key"
echo "------------------------"
create_secret_from_input foxhunt_benzinga_api_key "Benzinga API Key"

echo ""
echo "Step 8: TLS Certificates"
echo "------------------------"
create_secret_from_file foxhunt_tls_cert "./certs/server.crt"
create_secret_from_file foxhunt_tls_key "./certs/server.key"
create_secret_from_file foxhunt_tls_ca "./certs/ca-bundle.crt"

echo ""
echo "✅ All secrets created successfully!"
echo ""
echo "📋 List all secrets:"
docker secret ls | grep foxhunt

echo ""
echo "🚀 Ready to deploy:"
echo "   docker stack deploy -c docker-compose.prod.yml foxhunt"

Make the script executable and run it:

chmod +x setup-secrets.sh
./setup-secrets.sh

---

Service Configuration

Accessing Secrets in Services

Secrets are mounted as files in /run/secrets/ inside containers. Services should read secrets from files instead of environment variables.

Example: Rust Code

use std::fs;

// Read JWT secret from file
pub fn load_jwt_secret() -> Result<String, std::io::Error> {
    let secret_path = std::env::var("JWT_SECRET_FILE")
        .unwrap_or_else(|_| "/run/secrets/jwt_secret".to_string());
    
    fs::read_to_string(secret_path)
        .map(|s| s.trim().to_string())
}

// Read database credentials
pub fn load_database_url() -> Result<String, std::io::Error> {
    let user = fs::read_to_string("/run/secrets/postgres_user")?
        .trim()
        .to_string();
    let password = fs::read_to_string("/run/secrets/postgres_password")?
        .trim()
        .to_string();
    
    Ok(format!(
        "postgresql://{}:{}@postgres:5432/foxhunt",
        user, password
    ))
}

// Read Redis password
pub fn load_redis_url() -> Result<String, std::io::Error> {
    let password = fs::read_to_string("/run/secrets/redis_password")?
        .trim()
        .to_string();
    
    Ok(format!("redis://:{}@redis:6379", password))
}

Example: Docker Compose Service

services:
  api_gateway:
    image: foxhunt/api-gateway:latest
    secrets:
      - jwt_secret
      - postgres_password
      - postgres_user
      - redis_password
    environment:
      # Point to secret files instead of values
      - JWT_SECRET_FILE=/run/secrets/jwt_secret
      - DATABASE_URL_FILE=/run/secrets/postgres_password
      - REDIS_PASSWORD_FILE=/run/secrets/redis_password

---

Secret Management Operations

List Secrets

# List all secrets
docker secret ls

# Filter Foxhunt secrets
docker secret ls | grep foxhunt

Inspect Secret Metadata

# View secret metadata (NOT the actual value)
docker secret inspect foxhunt_jwt_secret

# View secret metadata in JSON format
docker secret inspect foxhunt_jwt_secret --format json | jq

Update Secrets (Rotation)

Docker secrets are immutable. To update a secret:

1. Create new secret with different name:

openssl rand -base64 96 | docker secret create foxhunt_jwt_secret_v2 -

2. Update service to use new secret:

# Update docker-compose.prod.yml to reference new secret
# Then deploy with:
docker stack deploy -c docker-compose.prod.yml foxhunt

3. Remove old secret after validation:

docker secret rm foxhunt_jwt_secret

Delete Secrets

# Remove a single secret (requires services to be stopped)
docker secret rm foxhunt_jwt_secret

# Remove all Foxhunt secrets
docker secret ls | grep foxhunt | awk '{print $1}' | xargs docker secret rm

---

Production Deployment

1. Initialize Swarm Cluster

# On manager node
docker swarm init --advertise-addr <MANAGER-IP>

# Add worker nodes (run on each worker)
docker swarm join --token <WORKER-TOKEN> <MANAGER-IP>:2377

2. Create Secrets

# Run setup script
./setup-secrets.sh

# Or manually create each secret

3. Deploy Stack

# Deploy with version tag
VERSION=v1.0.0 docker stack deploy -c docker-compose.prod.yml foxhunt

# Monitor deployment
watch docker stack ps foxhunt

4. Verify Secrets Access

# Check if services can access secrets
docker exec $(docker ps -q -f name=foxhunt_api_gateway) \
  ls -la /run/secrets/

# Verify secret content (should show file permissions)
docker exec $(docker ps -q -f name=foxhunt_api_gateway) \
  cat /run/secrets/jwt_secret

---

Security Best Practices

1. Secret Rotation Policy

Implement regular secret rotation:

• JWT secrets: Rotate every 90 days
• Database passwords: Rotate every 180 days
• API keys: Rotate annually or when compromised
• TLS certificates: Renew before expiration (typically 1 year)

2. Access Control

# Only manager nodes can manage secrets
# Worker nodes only receive secrets for their assigned services

# Verify node roles
docker node ls

# Promote worker to manager (if needed)
docker node promote <NODE-ID>

3. Audit Logging

Enable Docker audit logging for secret access:

# Configure Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "authorization-plugins": ["authz-broker"],
  "audit": true
}
EOF

# Restart Docker
systemctl restart docker

4. Backup Secrets

# Backup secret IDs and metadata (NOT values)
docker secret ls --format "{{.ID}} {{.Name}}" > secrets-backup.txt

# Store actual secret values in secure vault
# DO NOT store in git or plain text files

5. Secret Injection Prevention

Never log or expose secrets:

// ❌ BAD: Logging secrets
println!("JWT secret: {}", jwt_secret);

// ✅ GOOD: Log only metadata
println!("JWT secret loaded: {} bytes", jwt_secret.len());

// ❌ BAD: Include in error messages
return Err(format!("Invalid secret: {}", secret));

// ✅ GOOD: Generic error
return Err("Invalid secret format".to_string());

---

Troubleshooting

Secret Not Found

# Verify secret exists
docker secret ls | grep <SECRET_NAME>

# Check service configuration
docker service inspect <SERVICE_NAME> --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}'

Permission Denied

# Verify secret is assigned to service
docker service ps <SERVICE_NAME> --format '{{.Error}}'

# Check secret file permissions inside container
docker exec <CONTAINER_ID> ls -la /run/secrets/

Secret Update Not Applied

# Force service update
docker service update --force <SERVICE_NAME>

# Or redeploy entire stack
docker stack deploy -c docker-compose.prod.yml foxhunt

Secret Rotation Issues

# Create new version
docker secret create <SECRET_NAME>_v2 <NEW_VALUE>

# Update service to use new version
docker service update --secret-rm <SECRET_NAME> \
                     --secret-add source=<SECRET_NAME>_v2,target=<SECRET_NAME> \
                     <SERVICE_NAME>

---

Kubernetes Alternative

For Kubernetes deployments, use Kubernetes Secrets instead:

# Create Kubernetes secret
apiVersion: v1
kind: Secret
metadata:
  name: foxhunt-jwt-secret
type: Opaque
data:
  jwt_secret: <BASE64_ENCODED_VALUE>

# Reference in pod
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: api-gateway
    env:
    - name: JWT_SECRET
      valueFrom:
        secretKeyRef:
          name: foxhunt-jwt-secret
          key: jwt_secret

See docs/KUBERNETES_DEPLOYMENT.md for full Kubernetes guide.

---

Migration from Environment Variables

To migrate from .env files to Docker secrets:

1. Audit Current Environment Variables

# List environment variables in running container
docker exec <CONTAINER_ID> env | grep -i secret

2. Create Secrets

# For each sensitive env var, create a secret
echo "$ENV_VAR_VALUE" | docker secret create <SECRET_NAME> -

3. Update Service Configuration

# Before (environment variables)
services:
  api_gateway:
    environment:
      - JWT_SECRET=${JWT_SECRET}

# After (Docker secrets)
services:
  api_gateway:
    secrets:
      - jwt_secret
    environment:
      - JWT_SECRET_FILE=/run/secrets/jwt_secret

4. Update Application Code

// Before
let jwt_secret = std::env::var("JWT_SECRET")?;

// After
let jwt_secret = std::fs::read_to_string(
    std::env::var("JWT_SECRET_FILE")
        .unwrap_or("/run/secrets/jwt_secret".to_string())
)?.trim().to_string();

---

Monitoring and Compliance

Secret Access Monitoring

# Monitor secret access in container logs
docker service logs <SERVICE_NAME> | grep -i secret

# Check for failed secret reads
docker service ps <SERVICE_NAME> --filter "desired-state=shutdown"

Compliance Reporting

Generate compliance report for secret rotation:

#!/bin/bash
echo "Secret Rotation Compliance Report"
echo "=================================="
echo ""

docker secret ls --format "{{.ID}},{{.Name}},{{.CreatedAt}}" | while IFS=',' read -r id name created; do
    age_days=$(( ($(date +%s) - $(date -d "$created" +%s)) / 86400 ))
    
    if [ $age_days -gt 90 ]; then
        echo "⚠️  $name: $age_days days old (ROTATION REQUIRED)"
    else
        echo "✅ $name: $age_days days old"
    fi
done

---

References

• Docker Secrets Documentation
• Docker Swarm Mode
• Foxhunt DEPLOYMENT.md
• Foxhunt SECURITY.md
• CLAUDE.md Production Deployment

---

Last Updated: 2025-10-12
Maintained By: Foxhunt Platform Team
Review Frequency: Quarterly