foxhunt/docs/DEV_VS_PROD_CONFIG.md

# Development vs Production Configuration

**Comparison of docker-compose.yml (dev) and docker-compose.prod.yml (production)**

---

## Key Differences

### 1. Secrets Management

| Aspect | Development | Production |
|--------|-------------|------------|
| **Secret Storage** | Environment variables in `docker-compose.yml` | Docker Swarm secrets (external) |
| **Secret Access** | Direct env vars | Mounted files in `/run/secrets/` |
| **Secret Rotation** | Manual `.env` file edit | Docker secret rotation |
| **Visibility** | Visible in `docker inspect` | NOT visible in `docker inspect` |
| **Encryption** | None | Encrypted at rest and in transit |

**Development Example**:
```yaml
services:
  api_gateway:
    environment:
      - JWT_SECRET=dev_secret_key_change_in_production
      - POSTGRES_PASSWORD=foxhunt_dev_password
```

**Production Example**:
```yaml
services:
  api_gateway:
    secrets:
      - jwt_secret
      - postgres_password
    environment:
      - JWT_SECRET_FILE=/run/secrets/jwt_secret
      - DATABASE_URL_FILE=/run/secrets/postgres_password
```

---

### 2. Database Configuration

| Parameter | Development | Production |
|-----------|-------------|------------|
| **Password** | `foxhunt_dev_password` (hardcoded) | Docker secret |
| **Connections** | Default (100) | 500 |
| **Shared Buffers** | Default | 4GB |
| **WAL Settings** | Default | Optimized (16MB buffers, 4GB max) |
| **Synchronous Commit** | `on` (default) | `off` (HFT optimization) |
| **Persistence** | Named volume | Bind mount (`/mnt/data/foxhunt/postgres`) |

**Production Tuning**:
```yaml
environment:
  POSTGRES_MAX_CONNECTIONS: 500
  POSTGRES_SHARED_BUFFERS: 4GB
  POSTGRES_EFFECTIVE_CACHE_SIZE: 12GB
  POSTGRES_SYNCHRONOUS_COMMIT: "off"  # 4.5x performance boost
```

---

### 3. Redis Configuration

| Parameter | Development | Production |
|-----------|-------------|------------|
| **Authentication** | None | Required (via secret) |
| **Max Memory** | Default | 2GB with LRU eviction |
| **Persistence** | RDB only | RDB + AOF (every second) |
| **Command** | Default | Custom with optimization |

**Production Command**:
```yaml
command: >
  sh -c "redis-server 
  --requirepass $$(cat /run/secrets/redis_password)
  --maxmemory 2gb
  --maxmemory-policy allkeys-lru
  --appendonly yes
  --appendfsync everysec"
```

---

### 4. Deployment Configuration

| Aspect | Development | Production |
|--------|-------------|------------|
| **Orchestration** | Docker Compose | Docker Swarm |
| **Replicas** | 1 per service | 1-3 (service-dependent) |
| **Update Strategy** | None | Rolling with zero-downtime |
| **Rollback** | Manual | Automatic on failure |
| **Resource Limits** | None | CPU/Memory limits set |
| **Health Checks** | Basic | Advanced with start periods |

**Production Deployment Config**:
```yaml
deploy:
  mode: replicated
  replicas: 3
  update_config:
    parallelism: 1
    delay: 10s
    order: start-first
  rollback_config:
    parallelism: 1
    delay: 10s
  resources:
    limits:
      cpus: '2'
      memory: 2G
    reservations:
      cpus: '1'
      memory: 1G
  restart_policy:
    condition: on-failure
    delay: 5s
    max_attempts: 3
```

---

### 5. Service Scaling

| Service | Dev Replicas | Prod Replicas | Reasoning |
|---------|--------------|---------------|-----------|
| **API Gateway** | 1 | 3 | High availability + load distribution |
| **Trading Service** | 1 | 2 | Stateless, can scale horizontally |
| **Backtesting** | 1 | 1 | Resource-intensive, single instance |
| **ML Training** | 1 | 1 | GPU-bound, single instance |
| **PostgreSQL** | 1 | 1 | Stateful, manager node only |
| **Redis** | 1 | 1 | Single instance with persistence |

---

### 6. Networking

| Aspect | Development | Production |
|--------|-------------|------------|
| **Driver** | bridge | overlay (multi-host) |
| **Subnet** | Auto-assigned | 10.10.0.0/16 |
| **Encryption** | No | Optional (enable for multi-datacenter) |
| **Attachable** | Default | Yes (for debugging) |

**Production Network**:
```yaml
networks:
  foxhunt-network:
    driver: overlay
    attachable: true
    ipam:
      config:
        - subnet: 10.10.0.0/16
```

---

### 7. Volume Management

| Service | Development | Production |
|---------|-------------|------------|
| **PostgreSQL** | Named volume | Bind mount `/mnt/data/foxhunt/postgres` |
| **Redis** | Named volume | Bind mount `/mnt/data/foxhunt/redis` |
| **ML Models** | Local volume | Bind mount `/mnt/data/foxhunt/models` |
| **Prometheus** | Named volume | Bind mount `/mnt/data/foxhunt/prometheus` |

**Production Volume**:
```yaml
volumes:
  postgres_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/data/foxhunt/postgres
```

---

### 8. TLS/mTLS Configuration

| Aspect | Development | Production |
|--------|-------------|------------|
| **TLS Enabled** | No | Yes (all services) |
| **Certificate Source** | N/A | Docker secrets |
| **Certificate Validation** | N/A | Mutual TLS between services |

**Production TLS**:
```yaml
services:
  api_gateway:
    secrets:
      - tls_cert
      - tls_key
      - tls_ca
    environment:
      - ENABLE_TLS=true
      - TLS_CERT_FILE=/run/secrets/tls_cert
      - TLS_KEY_FILE=/run/secrets/tls_key
      - TLS_CA_FILE=/run/secrets/tls_ca
```

---

### 9. Logging and Monitoring

| Aspect | Development | Production |
|--------|-------------|------------|
| **Log Level** | `RUST_LOG=info` | `RUST_LOG=info` |
| **Backtrace** | `RUST_BACKTRACE=1` | `RUST_BACKTRACE=0` (performance) |
| **Audit Logging** | Disabled | `ENABLE_AUDIT_LOGGING=true` |
| **Metrics Retention** | 15 days | 90 days |
| **Prometheus Concurrency** | 50 | 100 |

---

### 10. Security Hardening

| Feature | Development | Production |
|---------|-------------|------------|
| **Grafana Signup** | Allowed | `GF_USERS_ALLOW_SIGN_UP=false` |
| **Grafana HTTPS** | No | `GF_SECURITY_COOKIE_SECURE=true` |
| **Rate Limiting** | 100 RPS | 1000 RPS |
| **Secret Rotation** | Manual | Automated with Docker secrets |
| **Node Placement** | Any | Manager-only for stateful services |

---

### 11. GPU Configuration

| Aspect | Development | Production |
|--------|-------------|------------|
| **Runtime** | nvidia | nvidia |
| **Device Selection** | All | CUDA_VISIBLE_DEVICES=0 |
| **Node Placement** | Any | `node.labels.gpu == true` |
| **Resource Reservation** | None | 1 NVIDIA-GPU reserved |

**Production GPU Config**:
```yaml
ml_training_service:
  deploy:
    placement:
      constraints:
        - node.labels.gpu == true
    resources:
      reservations:
        generic_resources:
          - discrete_resource_spec:
              kind: 'NVIDIA-GPU'
              value: 1
```

---

## Migration Checklist

### Before Migrating to Production

- [ ] **Generate strong secrets**
  ```bash
  openssl rand -base64 96  # JWT (96 bytes)
  openssl rand -base64 32  # Passwords (32 bytes)
  ```

- [ ] **Initialize Docker Swarm**
  ```bash
  docker swarm init
  ```

- [ ] **Create all required secrets**
  ```bash
  ./scripts/setup-docker-secrets.sh --interactive
  ```

- [ ] **Prepare persistent storage**
  ```bash
  sudo mkdir -p /mnt/data/foxhunt/{postgres,redis,influxdb,vault,prometheus,grafana,models,checkpoints}
  sudo chown -R 1000:1000 /mnt/data/foxhunt
  ```

- [ ] **Generate TLS certificates**
  ```bash
  cd certs && ./generate-certs.sh
  ```

- [ ] **Label GPU nodes** (if using ML service)
  ```bash
  docker node update --label-add gpu=true <NODE_NAME>
  ```

- [ ] **Configure external services**
  - AWS credentials (S3 access)
  - Benzinga API key
  - Vault production token

- [ ] **Update DNS/Load Balancer**
  - Point production domain to API Gateway
  - Configure SSL termination if needed

### Deployment

```bash
# 1. Deploy stack
VERSION=v1.0.0 docker stack deploy -c docker-compose.prod.yml foxhunt

# 2. Monitor deployment
watch docker stack ps foxhunt

# 3. Verify services
docker service ls | grep foxhunt

# 4. Check logs
docker service logs foxhunt_api_gateway -f
```

### Post-Deployment

- [ ] **Run health checks**
  ```bash
  curl http://localhost:9090/-/healthy  # Prometheus
  curl http://localhost:8080/health     # Services
  ```

- [ ] **Verify secret access**
  ```bash
  docker exec $(docker ps -q -f name=foxhunt_api_gateway) ls /run/secrets/
  ```

- [ ] **Test API authentication**
  ```bash
  grpc_health_probe -addr=localhost:50051
  ```

- [ ] **Monitor metrics**
  - Open Grafana: http://localhost:3000
  - Check Prometheus targets: http://localhost:9090/targets

- [ ] **Set up alerts**
  - Configure alerting rules
  - Test alert notifications

---

## Configuration Comparison Table

| Configuration Item | Development Value | Production Value |
|-------------------|------------------|------------------|
| JWT Secret | `dev_secret_key_change_in_production` | Docker secret (96 bytes) |
| DB Password | `foxhunt_dev_password` | Docker secret (32 bytes) |
| Redis Password | None | Docker secret (32 bytes) |
| API Gateway Replicas | 1 | 3 |
| Rate Limit (RPS) | 100 | 1000 |
| PostgreSQL Max Connections | 100 | 500 |
| PostgreSQL Shared Buffers | Default | 4GB |
| Redis Max Memory | Unlimited | 2GB |
| Prometheus Retention | 15 days | 90 days |
| TLS Enabled | No | Yes |
| Audit Logging | No | Yes |
| Volume Type | Named | Bind mount |
| Network Driver | bridge | overlay |

---

## Performance Impact

### PostgreSQL Optimizations
- **synchronous_commit=off**: 4.5x throughput improvement (663 → 2,979 inserts/sec)
- **Increased connections**: Handles 500 concurrent connections
- **Larger buffers**: Better caching and reduced I/O

### Redis Optimizations
- **LRU eviction**: Automatic memory management
- **AOF persistence**: Better durability with minimal overhead
- **Authentication**: Security without performance penalty

### Service Scaling
- **API Gateway (3 replicas)**: 3x request handling capacity
- **Trading Service (2 replicas)**: 2x order processing capacity

---

## Cost Considerations

| Resource | Development | Production | Monthly Cost Estimate |
|----------|-------------|------------|----------------------|
| Compute | 1 server | 3+ nodes | +$200-500/month |
| Storage | 50GB | 500GB-1TB | +$50-100/month |
| Network | Minimal | Load balancer + egress | +$30-50/month |
| Monitoring | Basic | Full stack | Included |
| **Total** | ~$50/month | ~$300-700/month | - |

---

## References

- [Docker Secrets Documentation](./DOCKER_SECRETS.md)
- [Docker Secrets Quick Start](./DOCKER_SECRETS_QUICKSTART.md)
- [Deployment Guide](./DEPLOYMENT.md)
- [TLS Setup Guide](./TLS_SETUP.md)
- [CLAUDE.md](../CLAUDE.md)

---

**Last Updated**: 2025-10-12  
**Version**: 1.0.0