foxhunt/docs/DEV_VS_PROD_CONFIG.md

Development vs Production Configuration

Comparison of docker-compose.yml (dev) and docker-compose.prod.yml (production)

---

Key Differences

1. Secrets Management

Aspect	Development	Production
Secret Storage	Environment variables in docker-compose.yml	Docker Swarm secrets (external)
Secret Access	Direct env vars	Mounted files in /run/secrets/
Secret Rotation	Manual .env file edit	Docker secret rotation
Visibility	Visible in docker inspect	NOT visible in docker inspect
Encryption	None	Encrypted at rest and in transit

Development Example:

services:
  api_gateway:
    environment:
      - JWT_SECRET=dev_secret_key_change_in_production
      - POSTGRES_PASSWORD=foxhunt_dev_password

Production Example:

services:
  api_gateway:
    secrets:
      - jwt_secret
      - postgres_password
    environment:
      - JWT_SECRET_FILE=/run/secrets/jwt_secret
      - DATABASE_URL_FILE=/run/secrets/postgres_password

---

2. Database Configuration

Parameter	Development	Production
Password	foxhunt_dev_password (hardcoded)	Docker secret
Connections	Default (100)	500
Shared Buffers	Default	4GB
WAL Settings	Default	Optimized (16MB buffers, 4GB max)
Synchronous Commit	on (default)	off (HFT optimization)
Persistence	Named volume	Bind mount (/mnt/data/foxhunt/postgres)

Production Tuning:

environment:
  POSTGRES_MAX_CONNECTIONS: 500
  POSTGRES_SHARED_BUFFERS: 4GB
  POSTGRES_EFFECTIVE_CACHE_SIZE: 12GB
  POSTGRES_SYNCHRONOUS_COMMIT: "off"  # 4.5x performance boost

---

3. Redis Configuration

Parameter	Development	Production
Authentication	None	Required (via secret)
Max Memory	Default	2GB with LRU eviction
Persistence	RDB only	RDB + AOF (every second)
Command	Default	Custom with optimization

Production Command:

command: >
  sh -c "redis-server 
  --requirepass $$(cat /run/secrets/redis_password)
  --maxmemory 2gb
  --maxmemory-policy allkeys-lru
  --appendonly yes
  --appendfsync everysec"

---

4. Deployment Configuration

Aspect	Development	Production
Orchestration	Docker Compose	Docker Swarm
Replicas	1 per service	1-3 (service-dependent)
Update Strategy	None	Rolling with zero-downtime
Rollback	Manual	Automatic on failure
Resource Limits	None	CPU/Memory limits set
Health Checks	Basic	Advanced with start periods

Production Deployment Config:

deploy:
  mode: replicated
  replicas: 3
  update_config:
    parallelism: 1
    delay: 10s
    order: start-first
  rollback_config:
    parallelism: 1
    delay: 10s
  resources:
    limits:
      cpus: '2'
      memory: 2G
    reservations:
      cpus: '1'
      memory: 1G
  restart_policy:
    condition: on-failure
    delay: 5s
    max_attempts: 3

---

5. Service Scaling

Service	Dev Replicas	Prod Replicas	Reasoning
API Gateway	1	3	High availability + load distribution
Trading Service	1	2	Stateless, can scale horizontally
Backtesting	1	1	Resource-intensive, single instance
ML Training	1	1	GPU-bound, single instance
PostgreSQL	1	1	Stateful, manager node only
Redis	1	1	Single instance with persistence

---

6. Networking

Aspect	Development	Production
Driver	bridge	overlay (multi-host)
Subnet	Auto-assigned	10.10.0.0/16
Encryption	No	Optional (enable for multi-datacenter)
Attachable	Default	Yes (for debugging)

Production Network:

networks:
  foxhunt-network:
    driver: overlay
    attachable: true
    ipam:
      config:
        - subnet: 10.10.0.0/16

---

7. Volume Management

Service	Development	Production
PostgreSQL	Named volume	Bind mount /mnt/data/foxhunt/postgres
Redis	Named volume	Bind mount /mnt/data/foxhunt/redis
ML Models	Local volume	Bind mount /mnt/data/foxhunt/models
Prometheus	Named volume	Bind mount /mnt/data/foxhunt/prometheus

Production Volume:

volumes:
  postgres_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/data/foxhunt/postgres

---

8. TLS/mTLS Configuration

Aspect	Development	Production
TLS Enabled	No	Yes (all services)
Certificate Source	N/A	Docker secrets
Certificate Validation	N/A	Mutual TLS between services

Production TLS:

services:
  api_gateway:
    secrets:
      - tls_cert
      - tls_key
      - tls_ca
    environment:
      - ENABLE_TLS=true
      - TLS_CERT_FILE=/run/secrets/tls_cert
      - TLS_KEY_FILE=/run/secrets/tls_key
      - TLS_CA_FILE=/run/secrets/tls_ca

---

9. Logging and Monitoring

Aspect	Development	Production
Log Level	RUST_LOG=info	RUST_LOG=info
Backtrace	RUST_BACKTRACE=1	RUST_BACKTRACE=0 (performance)
Audit Logging	Disabled	ENABLE_AUDIT_LOGGING=true
Metrics Retention	15 days	90 days
Prometheus Concurrency	50	100

---

10. Security Hardening

Feature	Development	Production
Grafana Signup	Allowed	GF_USERS_ALLOW_SIGN_UP=false
Grafana HTTPS	No	GF_SECURITY_COOKIE_SECURE=true
Rate Limiting	100 RPS	1000 RPS
Secret Rotation	Manual	Automated with Docker secrets
Node Placement	Any	Manager-only for stateful services

---

11. GPU Configuration

Aspect	Development	Production
Runtime	nvidia	nvidia
Device Selection	All	CUDA_VISIBLE_DEVICES=0
Node Placement	Any	node.labels.gpu == true
Resource Reservation	None	1 NVIDIA-GPU reserved

Production GPU Config:

ml_training_service:
  deploy:
    placement:
      constraints:
        - node.labels.gpu == true
    resources:
      reservations:
        generic_resources:
          - discrete_resource_spec:
              kind: 'NVIDIA-GPU'
              value: 1

---

Migration Checklist

Before Migrating to Production

• Generate strong secrets

  openssl rand -base64 96  # JWT (96 bytes)
  openssl rand -base64 32  # Passwords (32 bytes)
  

• Initialize Docker Swarm

  docker swarm init
  

• Create all required secrets

  ./scripts/setup-docker-secrets.sh --interactive
  

• Prepare persistent storage

  sudo mkdir -p /mnt/data/foxhunt/{postgres,redis,influxdb,vault,prometheus,grafana,models,checkpoints}
  sudo chown -R 1000:1000 /mnt/data/foxhunt
  

• Generate TLS certificates

  cd certs && ./generate-certs.sh
  

• Label GPU nodes (if using ML service)

  docker node update --label-add gpu=true <NODE_NAME>
  

• Configure external services

  • AWS credentials (S3 access)
  • Benzinga API key
  • Vault production token

• Update DNS/Load Balancer

  • Point production domain to API Gateway
  • Configure SSL termination if needed

Deployment

# 1. Deploy stack
VERSION=v1.0.0 docker stack deploy -c docker-compose.prod.yml foxhunt

# 2. Monitor deployment
watch docker stack ps foxhunt

# 3. Verify services
docker service ls | grep foxhunt

# 4. Check logs
docker service logs foxhunt_api_gateway -f

Post-Deployment

• Run health checks

  curl http://localhost:9090/-/healthy  # Prometheus
  curl http://localhost:8080/health     # Services
  

• Verify secret access

  docker exec $(docker ps -q -f name=foxhunt_api_gateway) ls /run/secrets/
  

• Test API authentication

  grpc_health_probe -addr=localhost:50051
  

• Monitor metrics

  • Open Grafana: http://localhost:3000
  • Check Prometheus targets: http://localhost:9090/targets

• Set up alerts

  • Configure alerting rules
  • Test alert notifications

---

Configuration Comparison Table

Configuration Item	Development Value	Production Value
JWT Secret	dev_secret_key_change_in_production	Docker secret (96 bytes)
DB Password	foxhunt_dev_password	Docker secret (32 bytes)
Redis Password	None	Docker secret (32 bytes)
API Gateway Replicas	1	3
Rate Limit (RPS)	100	1000
PostgreSQL Max Connections	100	500
PostgreSQL Shared Buffers	Default	4GB
Redis Max Memory	Unlimited	2GB
Prometheus Retention	15 days	90 days
TLS Enabled	No	Yes
Audit Logging	No	Yes
Volume Type	Named	Bind mount
Network Driver	bridge	overlay

---

Performance Impact

PostgreSQL Optimizations

• synchronous_commit=off: 4.5x throughput improvement (663 → 2,979 inserts/sec)
• Increased connections: Handles 500 concurrent connections
• Larger buffers: Better caching and reduced I/O

Redis Optimizations

• LRU eviction: Automatic memory management
• AOF persistence: Better durability with minimal overhead
• Authentication: Security without performance penalty

Service Scaling

• API Gateway (3 replicas): 3x request handling capacity
• Trading Service (2 replicas): 2x order processing capacity

---

Cost Considerations

Resource	Development	Production	Monthly Cost Estimate
Compute	1 server	3+ nodes	+$200-500/month
Storage	50GB	500GB-1TB	+$50-100/month
Network	Minimal	Load balancer + egress	+$30-50/month
Monitoring	Basic	Full stack	Included
Total	~$50/month	~$300-700/month	-

---

References

• Docker Secrets Documentation
• Docker Secrets Quick Start
• Deployment Guide
• TLS Setup Guide
• CLAUDE.md

---

Last Updated: 2025-10-12
Version: 1.0.0