foxhunt/PRODUCTION_DEPLOYMENT_RUNBOOK.md

Production Deployment Runbook - Foxhunt HFT Trading System

Version: 1.0.0 Last Updated: 2025-10-07 Status: Production Ready (93-94% deployment readiness) Owner: DevOps Team

---

🎯 Document Purpose

This runbook provides comprehensive step-by-step procedures for deploying, operating, and maintaining the Foxhunt HFT Trading System in production environments. It covers all operational scenarios from initial deployment through disaster recovery.

Critical Reference: This document incorporates findings from Agent 96's Docker E2E validation and addresses all identified deployment blockers.

---

📋 Table of Contents

1. Prerequisites
2. Initial Deployment
3. Rolling Updates
4. Scaling Procedures
5. Disaster Recovery
6. Monitoring and Alerting
7. Troubleshooting
8. Security Procedures

---

Prerequisites

1. Infrastructure Requirements

Hardware (Bare-Metal Deployment - Recommended for HFT)

• CPU: 16+ cores (Intel Xeon or AMD EPYC, 3.0+ GHz)
• RAM: 64GB+ minimum, 128GB recommended
• Storage: NVMe SSD 1TB+ (IOPS: 100K+ read, 50K+ write)
• Network: 10Gbps+ low-latency network adapter
• GPU: NVIDIA RTX 3050 Ti or better (for ML inference)
  • CUDA 12.3+ support
  • 4GB+ VRAM minimum

Software Dependencies

# Operating System
Ubuntu 22.04 LTS (kernel 5.15+)

# Container Runtime (if using Docker)
Docker Engine 24.0+
Docker Compose 2.20+
nvidia-docker2 (for GPU support)

# Database
PostgreSQL 16+ with TimescaleDB extension
Redis 7.0+

# Monitoring
Prometheus 2.45+
Grafana 10.0+
InfluxDB 2.7+ (time-series metrics)

# Secrets Management
HashiCorp Vault 1.15+

# Build Tools (for bare-metal)
Rust 1.89+
CUDA Toolkit 12.3+ (for GPU services)
protobuf-compiler 3.0+

2. Environment Variables

Agent 96 Finding: Missing environment variables caused service failures. All variables below are REQUIRED.

Core Infrastructure (All Services)

# Database Configuration
DATABASE_URL=postgresql://foxhunt:<password>@<host>:5432/foxhunt
REDIS_URL=redis://<host>:6379
VAULT_ADDR=http://<vault-host>:8200
VAULT_TOKEN=<vault-token>

# Logging
RUST_LOG=info
RUST_BACKTRACE=1

# Environment
FOXHUNT_ENV=production

Authentication Services (Trading, API Gateway)

# JWT Configuration
JWT_SECRET=<64+ character base64 string>
# Generate: openssl rand -base64 64
# MUST contain: uppercase, lowercase, numbers, symbols

# Production: Use file-based secrets instead
JWT_SECRET_FILE=/run/secrets/jwt_secret

# JWT Claims
JWT_ISSUER=foxhunt-api-gateway
JWT_AUDIENCE=foxhunt-services

API Keys (Backtesting Service)

Agent 96 Critical Finding: Missing BENZINGA_API_KEY blocked backtesting service startup.

# Required for Backtesting Service
BENZINGA_API_KEY=<benzinga-pro-api-key>
# Obtain from: https://www.benzinga.com/apis

# Optional: Databento API (already configured)
DATABENTO_API_KEY=<databento-api-key>

Security Tokens (Production)

# Kill Switch Authentication
KILL_SWITCH_MASTER_TOKEN=<secure-random-token>
# Generate: openssl rand -base64 32

# Production: Use file-based secrets
KILL_SWITCH_MASTER_TOKEN_FILE=/run/secrets/kill_switch_token

Rate Limiting (API Gateway)

RATE_LIMIT_RPS=100
ENABLE_AUDIT_LOGGING=true

AWS S3 Configuration (Model Storage)

AWS_ACCESS_KEY_ID=<aws-access-key>
AWS_SECRET_ACCESS_KEY=<aws-secret-key>
AWS_REGION=us-east-1

# S3 Buckets
S3_MODEL_STORAGE_BUCKET=foxhunt-models
S3_CHECKPOINT_BUCKET=foxhunt-checkpoints
S3_ARCHIVAL_BUCKET=foxhunt-archives

GPU Configuration (ML Services)

# CUDA Environment
CUDA_HOME=/usr/local/cuda
LD_LIBRARY_PATH=$CUDA_HOME/lib64:$CUDA_HOME/targets/x86_64-linux/lib
PATH=$CUDA_HOME/bin:$PATH

# CUDA Build Flags (Docker)
CUDARC_CUDA_VERSION=13000
CUDA_COMPUTE_CAP=86  # RTX 3050 Ti

# Docker Runtime
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=compute,utility

3. Secrets Management (Vault Configuration)

Initialize Vault Secrets

# Start Vault (development mode for initial setup)
docker run -d --name vault -p 8200:8200 \
  -e VAULT_DEV_ROOT_TOKEN_ID=foxhunt-dev-root \
  hashicorp/vault:1.15 server -dev

# Set Vault address
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=foxhunt-dev-root

# Store JWT secret
vault kv put secret/foxhunt/jwt \
  secret="$(openssl rand -base64 64)"

# Store kill switch token
vault kv put secret/foxhunt/kill_switch \
  master_token="$(openssl rand -base64 32)"

# Store API keys
vault kv put secret/foxhunt/api_keys \
  benzinga="<your-benzinga-api-key>" \
  databento="<your-databento-api-key>"

# Store AWS credentials
vault kv put secret/foxhunt/aws \
  access_key_id="<aws-access-key>" \
  secret_access_key="<aws-secret-key>"

# Verify secrets stored
vault kv list secret/foxhunt

Production Vault Configuration

# Production: Use proper Vault seal (not dev mode)
# See: https://www.vaultproject.io/docs/configuration/seal

# Enable AppRole authentication
vault auth enable approle

# Create policy for Foxhunt services
vault policy write foxhunt-services - <<EOF
path "secret/data/foxhunt/*" {
  capabilities = ["read"]
}
EOF

# Create AppRole
vault write auth/approle/role/foxhunt-services \
  secret_id_ttl=24h \
  token_ttl=1h \
  token_max_ttl=4h \
  policies="foxhunt-services"

# Get credentials for services
vault read auth/approle/role/foxhunt-services/role-id
vault write -f auth/approle/role/foxhunt-services/secret-id

4. Network Requirements

Firewall Rules

# External Access (from internet/clients)
Allow TCP 50051  # API Gateway gRPC (TLS encrypted)
Allow TCP 443    # API Gateway HTTPS (optional)
Allow TCP 3000   # Grafana (restrict to internal network)

# Internal Service Communication (service mesh)
Allow TCP 50050-50054  # gRPC inter-service
Allow TCP 5432         # PostgreSQL
Allow TCP 6379         # Redis
Allow TCP 8200         # Vault
Allow TCP 8086         # InfluxDB
Allow TCP 9090         # Prometheus

# Metrics Collection (internal only)
Allow TCP 9091-9094    # Service metrics endpoints

# Deny all other inbound traffic

Load Balancer Configuration

# For API Gateway (external)
Frontend: TCP 50051 (TLS passthrough)
Backend: Multiple API Gateway instances
Health Check: gRPC health probe on 50051
Timeout: 30s
Keep-Alive: enabled

# Sticky sessions for WebSocket connections
Session Affinity: Client IP hash

5. GPU Requirements (ML Services)

Agent 96 Finding: Docker images missing NVIDIA runtime libraries.

Host Setup

# Install NVIDIA drivers
sudo apt-get update
sudo apt-get install -y nvidia-driver-535

# Verify GPU
nvidia-smi

# Install nvidia-docker2
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker

# Test GPU access in container
docker run --rm --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi

Docker Runtime Configuration

# /etc/docker/daemon.json
{
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  }
}

# Restart Docker
sudo systemctl restart docker

---

Initial Deployment

Deployment Mode Selection

Mode 1: Bare-Metal (RECOMMENDED for HFT)

• Latency: <50μs order processing
• Throughput: 50K+ orders/second
• Use Case: Production trading, maximum performance
• See: Bare-Metal Deployment

Mode 2: Docker Compose

• Latency: ~100μs order processing
• Throughput: 10K+ orders/second
• Use Case: Staging, testing, development
• See: Docker Deployment

Mode 3: Kubernetes

• Latency: ~200μs order processing
• Throughput: Horizontally scalable
• Use Case: Cloud, multi-region, high availability
• See: Kubernetes Deployment

---

Bare-Metal Deployment

Performance: Sub-50μs latency, direct hardware access

Step 1: Build Optimized Binaries

# Clone repository
cd /opt
sudo git clone <repository-url> foxhunt
sudo chown -R foxhunt:foxhunt foxhunt
cd foxhunt

# Build with CPU-specific optimizations
export RUSTFLAGS="-C target-cpu=native -C opt-level=3"
cargo build --release --workspace

# Verify binaries
ls -lh target/release/ | grep -E "(trading|backtesting|ml_training|api_gateway)"

# Expected output:
# trading_service       ~50MB
# backtesting_service   ~55MB
# ml_training_service   ~80MB
# api_gateway          ~45MB

Step 2: Install Binaries

# Create installation directories
sudo mkdir -p /opt/foxhunt/{bin,config,logs,data,models,checkpoints}

# Install binaries
sudo cp target/release/trading_service /opt/foxhunt/bin/
sudo cp target/release/backtesting_service /opt/foxhunt/bin/
sudo cp target/release/ml_training_service /opt/foxhunt/bin/
sudo cp target/release/api_gateway /opt/foxhunt/bin/

# Set permissions
sudo chown -R foxhunt:foxhunt /opt/foxhunt
sudo chmod +x /opt/foxhunt/bin/*

Step 3: Configure SystemD Services

Agent 96 Note: Services must start in correct order (infra → backend → gateway).

# Generate SystemD service files
./deployment/create_systemd_services.sh

# Service files created in /etc/systemd/system/:
# - foxhunt-trading.service
# - foxhunt-backtesting.service
# - foxhunt-ml-training.service
# - foxhunt-api-gateway.service

# Reload SystemD
sudo systemctl daemon-reload

# Enable services (auto-start on boot)
sudo systemctl enable foxhunt-trading
sudo systemctl enable foxhunt-backtesting
sudo systemctl enable foxhunt-ml-training
sudo systemctl enable foxhunt-api-gateway

Step 4: Database Setup

# Start PostgreSQL (if not running)
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Create database and user
sudo -u postgres psql <<EOF
CREATE USER foxhunt WITH PASSWORD '<secure-password>';
CREATE DATABASE foxhunt OWNER foxhunt;
\c foxhunt
CREATE EXTENSION IF NOT EXISTS timescaledb;
GRANT ALL PRIVILEGES ON DATABASE foxhunt TO foxhunt;
EOF

# Run migrations
cd /opt/foxhunt
export DATABASE_URL=postgresql://foxhunt:<password>@localhost:5432/foxhunt
cargo sqlx migrate run

# Verify migrations
psql $DATABASE_URL -c '\dt'

# Expected tables:
# - orders
# - positions
# - market_data
# - audit_logs
# - ml_models
# - configurations
# (+ 12 more from 18 migrations)

Step 5: Start Services (Correct Order)

Critical: Services must start in dependency order to avoid panics.

# 1. Infrastructure services
sudo systemctl start redis
sudo systemctl start vault
sudo systemctl status redis vault

# 2. Backend services (parallel)
sudo systemctl start foxhunt-trading
sudo systemctl start foxhunt-backtesting
sudo systemctl start foxhunt-ml-training

# Wait for backend health checks (30 seconds)
sleep 30

# 3. API Gateway (depends on all backends)
sudo systemctl start foxhunt-api-gateway

# Verify all services running
sudo systemctl status foxhunt-*

Step 6: Health Validation

# Run comprehensive health check
./deployment/health_check.sh --mode comprehensive --verbose

# Expected output:
# ✅ Trading Service: HEALTHY (gRPC port 50052)
# ✅ Backtesting Service: HEALTHY (gRPC port 50053)
# ✅ ML Training Service: HEALTHY (gRPC port 50054)
# ✅ API Gateway: HEALTHY (gRPC port 50051)
# ✅ PostgreSQL: Connected
# ✅ Redis: Connected
# ✅ Vault: Sealed/Unsealed status
# ✅ Memory usage: <500MB per service
# ✅ CPU usage: <10% idle

Step 7: Smoke Tests

# Test API Gateway
grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

# Expected: {"status": "SERVING"}

# Test Trading Service (via Gateway)
# Requires TLI or gRPC client

# Check metrics endpoints
curl http://localhost:9092/metrics | grep foxhunt_
curl http://localhost:9093/metrics | grep foxhunt_
curl http://localhost:9094/metrics | grep foxhunt_
curl http://localhost:9091/metrics | grep foxhunt_

# Verify logs for errors
sudo journalctl -u foxhunt-* --since "5 minutes ago" | grep -i error

# Should return: No errors

---

Docker Deployment

Agent 96 Findings: 3 critical Dockerfile issues must be fixed before deployment.

Pre-Deployment Fixes (REQUIRED)

Fix 1: Dockerfile Path Corrections

# Issue: Dockerfiles reference crates/config (doesn't exist)
# Fix: Update all Dockerfiles

# Edit each Dockerfile:
# - services/trading_service/Dockerfile
# - services/backtesting_service/Dockerfile
# - services/ml_training_service/Dockerfile
# - services/api_gateway/Dockerfile

# Find and replace:
# OLD: COPY crates/config ./crates/config
# NEW: COPY config ./config

Fix 2: ML Training Service Entry Point

# Issue: Container expects subcommand but doesn't provide default
# Fix: Add CMD to Dockerfile

# Edit: services/ml_training_service/Dockerfile
# Add before final line:
CMD ["ml_training_service", "serve"]

Fix 3: Add Benzinga API Key

# Issue: Backtesting service requires BENZINGA_API_KEY
# Fix: Add to docker-compose.yml or .env file

# Edit: docker-compose.yml
backtesting_service:
  environment:
    - BENZINGA_API_KEY=${BENZINGA_API_KEY}

# Or add to .env:
echo "BENZINGA_API_KEY=<your-key>" >> .env

Step 1: Build Docker Images

cd /opt/foxhunt

# Build all services
./deployment/build_docker_images.sh --service all --enable-gpu

# Or build individually
docker build -f services/trading_service/Dockerfile -t foxhunt-trading-service:latest .
docker build -f services/backtesting_service/Dockerfile -t foxhunt-backtesting-service:latest .
docker build -f services/ml_training_service/Dockerfile -t foxhunt-ml-training-service:latest .
docker build -f services/api_gateway/Dockerfile -t foxhunt-api-gateway:latest .

# Verify images built
docker images | grep foxhunt

# Expected:
# foxhunt-trading-service      latest    <id>    119MB
# foxhunt-backtesting-service  latest    <id>    120MB
# foxhunt-ml-training-service  latest    <id>    2.24GB (includes CUDA)
# foxhunt-api-gateway         latest    <id>    119MB

Step 2: Configure Environment

# Create production environment file
cp .env.example .env.production

# Edit .env.production with production values
vim .env.production

# Required variables (from Agent 96 findings):
DATABASE_URL=postgresql://foxhunt:<password>@postgres:5432/foxhunt
REDIS_URL=redis://redis:6379
VAULT_ADDR=http://vault:8200
VAULT_TOKEN=<production-token>
JWT_SECRET=<generate-with-openssl-rand-base64-64>
BENZINGA_API_KEY=<your-benzinga-api-key>
KILL_SWITCH_MASTER_TOKEN=<generate-with-openssl-rand-base64-32>
AWS_ACCESS_KEY_ID=<aws-key>
AWS_SECRET_ACCESS_KEY=<aws-secret>

# Validate all required variables set
./deployment/scripts/validate-env.sh .env.production

Step 3: Start Infrastructure Services

# Start database and caching layers
docker-compose up -d postgres redis vault influxdb

# Wait for health checks
docker-compose ps

# Verify all healthy (retry_count 0/5)
# foxhunt-postgres    healthy
# foxhunt-redis       healthy
# foxhunt-vault       healthy
# foxhunt-influxdb    healthy

Step 4: Run Database Migrations

# Connect to running postgres container
docker-compose exec postgres psql -U foxhunt -d foxhunt

# Or run migrations from host
export DATABASE_URL=postgresql://foxhunt:<password>@localhost:5432/foxhunt
cargo sqlx migrate run

# Verify 18 migrations applied
psql $DATABASE_URL -c "SELECT version FROM _sqlx_migrations ORDER BY version;"

Step 5: Start Monitoring Stack

# Start Prometheus and Grafana
docker-compose up -d prometheus grafana

# Verify access
curl http://localhost:9090/-/healthy    # Prometheus
curl http://localhost:3000/api/health   # Grafana

# Login to Grafana
# URL: http://localhost:3000
# Username: admin
# Password: foxhunt123 (from docker-compose.yml)

Step 6: Start Application Services (Correct Order)

# Start backend services (can run in parallel)
docker-compose up -d trading_service backtesting_service ml_training_service

# Wait for services to initialize (30-60 seconds for GPU services)
sleep 60

# Check logs for successful startup
docker-compose logs trading_service | grep "gRPC server started"
docker-compose logs backtesting_service | grep "Repository layer initialized"
docker-compose logs ml_training_service | grep "ML training service started"

# Start API Gateway (after backends healthy)
docker-compose up -d api_gateway

# Verify all services running
docker-compose ps

# Expected:
# foxhunt-trading-service      Up      0.0.0.0:50052->50051/tcp
# foxhunt-backtesting-service  Up      0.0.0.0:50053->50052/tcp
# foxhunt-ml-training-service  Up      0.0.0.0:50054->50053/tcp
# foxhunt-api-gateway         Up      0.0.0.0:50051->50050/tcp

Step 7: Validate Deployment

# Check service health
for port in 50051 50052 50053 50054; do
  docker run --rm --network host \
    fullstorydev/grpcurl:latest \
    -plaintext localhost:$port grpc.health.v1.Health/Check
done

# Expected: {"status": "SERVING"} for all

# Check GPU access (ML Training Service)
docker-compose exec ml_training_service nvidia-smi

# Expected: GPU 0: NVIDIA RTX 3050 Ti

# Monitor resource usage
docker stats --no-stream

# Expected: <500MB memory per service

---

Kubernetes Deployment

Use Case: Cloud deployment, horizontal scaling, high availability

Step 1: Build and Push Images

# Set registry
export REGISTRY=your-registry.com/foxhunt

# Build and push all images
./deployment/build_docker_images.sh \
  --service all \
  --enable-gpu \
  --push \
  --registry $REGISTRY

# Verify images pushed
docker images | grep $REGISTRY

Step 2: Create Kubernetes Manifests

# Create namespace
kubectl create namespace foxhunt

# Create secrets
kubectl create secret generic foxhunt-secrets -n foxhunt \
  --from-literal=database-url="postgresql://..." \
  --from-literal=jwt-secret="$(openssl rand -base64 64)" \
  --from-literal=benzinga-api-key="<key>" \
  --from-literal=kill-switch-token="$(openssl rand -base64 32)"

# Create ConfigMap
kubectl create configmap foxhunt-config -n foxhunt \
  --from-literal=rust-log=info \
  --from-literal=rate-limit-rps=100

Step 3: Deploy Infrastructure

# k8s/infrastructure.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: foxhunt
spec:
  selector:
    app: postgres
  ports:
    - port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: foxhunt
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: timescale/timescaledb:latest-pg16
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_DB
          value: foxhunt
        - name: POSTGRES_USER
          value: foxhunt
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: foxhunt-secrets
              key: postgres-password
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi

# Apply infrastructure
kubectl apply -f k8s/infrastructure.yaml

# Verify
kubectl get pods -n foxhunt

Step 4: Deploy Application Services

# k8s/trading-service.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
  namespace: foxhunt
spec:
  replicas: 3
  selector:
    matchLabels:
      app: trading-service
  template:
    metadata:
      labels:
        app: trading-service
    spec:
      containers:
      - name: trading-service
        image: your-registry.com/foxhunt/trading-service:latest
        ports:
        - containerPort: 50051
          name: grpc
        - containerPort: 9092
          name: metrics
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: foxhunt-secrets
              key: database-url
        - name: REDIS_URL
          value: redis://redis:6379
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: foxhunt-secrets
              key: jwt-secret
        resources:
          requests:
            memory: "512Mi"
            cpu: "1000m"
          limits:
            memory: "2Gi"
            cpu: "2000m"
        livenessProbe:
          exec:
            command: ["/usr/local/bin/grpc_health_probe", "-addr=:50051"]
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command: ["/usr/local/bin/grpc_health_probe", "-addr=:50051"]
          initialDelaySeconds: 15
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: trading-service
  namespace: foxhunt
spec:
  selector:
    app: trading-service
  ports:
  - name: grpc
    port: 50051
    targetPort: 50051
  - name: metrics
    port: 9092
    targetPort: 9092

# Deploy services
kubectl apply -f k8s/trading-service.yaml
kubectl apply -f k8s/backtesting-service.yaml
kubectl apply -f k8s/ml-training-service.yaml
kubectl apply -f k8s/api-gateway.yaml

# Verify
kubectl get pods -n foxhunt -l app=trading-service
kubectl logs -n foxhunt -l app=trading-service

Step 5: Configure Ingress

# k8s/ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foxhunt-ingress
  namespace: foxhunt
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.foxhunt.trading
    secretName: foxhunt-tls
  rules:
  - host: api.foxhunt.trading
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-gateway
            port:
              number: 50051

# Apply ingress
kubectl apply -f k8s/ingress.yaml

# Get external IP
kubectl get ingress -n foxhunt foxhunt-ingress

Step 6: Set Up Auto-Scaling

# k8s/hpa.yaml
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: trading-service-hpa
  namespace: foxhunt
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: trading-service
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

# Apply HPA
kubectl apply -f k8s/hpa.yaml

# Monitor scaling
kubectl get hpa -n foxhunt -w

---

Rolling Updates

Zero-Downtime Update Procedure

Strategy: Blue-Green Deployment

# Step 1: Build new version
export NEW_VERSION=v1.1.0
./deployment/build_release.sh --version $NEW_VERSION

# Step 2: Deploy to "green" environment (parallel to "blue")
# Bare-metal: Deploy to separate servers
# Docker: Use docker-compose with different project name
# Kubernetes: Deploy to separate namespace

# Step 3: Run smoke tests on green
./deployment/health_check.sh --target green --mode comprehensive

# Step 4: Switch traffic (load balancer/DNS)
# Update load balancer backend to green environment

# Step 5: Monitor for errors (rollback window: 15 minutes)
watch -n 5 'kubectl get pods -n foxhunt-green'
watch -n 5 'curl http://green.foxhunt.trading/metrics | grep error_rate'

# Step 6: If successful, decommission blue environment
# If errors, rollback to blue

Rollback Procedure

# Immediate rollback (switch load balancer back to blue)
# Total time: <30 seconds

# Kubernetes rollback
kubectl rollout undo deployment/trading-service -n foxhunt
kubectl rollout status deployment/trading-service -n foxhunt

# SystemD rollback
sudo systemctl stop foxhunt-trading
sudo cp /opt/foxhunt/bin/trading_service.backup /opt/foxhunt/bin/trading_service
sudo systemctl start foxhunt-trading

# Docker rollback
docker-compose pull trading_service:previous
docker-compose up -d trading_service

Database Migration Sequencing

Critical: Database migrations must be backward-compatible for zero-downtime deployments.

# Step 1: Create backward-compatible migration
# migrations/019_add_new_column.sql
BEGIN;
  -- Add column with default value (safe)
  ALTER TABLE orders ADD COLUMN new_field VARCHAR(255) DEFAULT 'default_value';
COMMIT;

# Step 2: Apply migration (old code still works)
cargo sqlx migrate run

# Step 3: Deploy new application code
# (old code ignores new column, new code uses it)

# Step 4: After deployment stable, remove default
# migrations/020_remove_default.sql
BEGIN;
  ALTER TABLE orders ALTER COLUMN new_field DROP DEFAULT;
COMMIT;

---

Scaling Procedures

Horizontal Scaling (Adding Instances)

Stateless Services (Trading, Backtesting, API Gateway)

Bare-Metal:

# Add new server to load balancer pool
# Install service on new server (same as initial deployment)
# Start service
sudo systemctl start foxhunt-trading

# Verify health before adding to pool
curl http://new-server:9092/metrics

# Add to load balancer
# Update Nginx upstream configuration
# nginx/upstream.conf
upstream trading_service {
  server trading-1.foxhunt.internal:50051;
  server trading-2.foxhunt.internal:50051;
  server trading-3.foxhunt.internal:50051;  # New server
}

# Reload Nginx
sudo nginx -s reload

Kubernetes:

# Scale deployment
kubectl scale deployment trading-service -n foxhunt --replicas=5

# Or use HPA (automatic scaling)
kubectl autoscale deployment trading-service -n foxhunt \
  --min=3 --max=10 --cpu-percent=70

Docker Swarm:

# Scale service
docker service scale foxhunt_trading_service=5

# Verify
docker service ps foxhunt_trading_service

Stateful Services (PostgreSQL, Redis)

PostgreSQL Read Replicas:

# Create read replica
pg_basebackup -h primary.foxhunt.internal -D /var/lib/postgresql/replica \
  -U replication -P -v -R

# Start replica
sudo systemctl start postgresql-replica

# Update connection pool to use replica for reads
# services/trading_service/src/config.rs
DatabaseConfig {
  write_url: "postgresql://primary:5432/foxhunt",
  read_urls: vec![
    "postgresql://replica-1:5432/foxhunt",
    "postgresql://replica-2:5432/foxhunt",
  ],
}

Redis Cluster:

# Create 6-node cluster (3 masters, 3 replicas)
redis-cli --cluster create \
  redis-1:6379 redis-2:6379 redis-3:6379 \
  redis-4:6379 redis-5:6379 redis-6:6379 \
  --cluster-replicas 1

# Verify cluster
redis-cli -c cluster nodes

Vertical Scaling (Resource Adjustments)

Increase CPU/Memory (Kubernetes)

# Update deployment resources
kubectl patch deployment trading-service -n foxhunt -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"trading-service","resources":{"requests":{"cpu":"2000m","memory":"4Gi"},"limits":{"cpu":"4000m","memory":"8Gi"}}}]}}}}'

# Verify change
kubectl get deployment trading-service -n foxhunt -o yaml | grep -A 5 resources

Increase Disk (Bare-Metal)

# Resize LVM volume (PostgreSQL data)
sudo lvextend -L +100G /dev/vg0/postgres
sudo resize2fs /dev/vg0/postgres

# Verify
df -h /var/lib/postgresql

Auto-Scaling Policies

CPU-Based Scaling (Kubernetes)

# Already configured in HPA (see Kubernetes Deployment)
# Scales from 3 to 10 replicas based on 70% CPU utilization

Custom Metrics Scaling

# k8s/hpa-custom.yaml
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: trading-service-hpa
  namespace: foxhunt
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: trading-service
  minReplicas: 3
  maxReplicas: 20
  metrics:
  # CPU
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  # Memory
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  # Custom: Order processing latency
  - type: Pods
    pods:
      metric:
        name: order_processing_latency_p99_seconds
      target:
        type: AverageValue
        averageValue: "0.05"  # 50ms
  # Custom: Queue depth
  - type: Pods
    pods:
      metric:
        name: order_queue_depth
      target:
        type: AverageValue
        averageValue: "100"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # 5 minute cooldown
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
      selectPolicy: Max

---

Disaster Recovery

Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO)

Component	RTO	RPO	Current Status
Database	<5 min	<1 min	⚠️ Streaming replication needed
Trading Service	<1 min	0 (stateless)	✅ Ready (stateless)
ML Training Service	<2 min	<5 min	⚠️ Checkpoint recovery needed
Risk Engine	<30 sec	0 (real-time)	✅ Ready (stateless)
Market Data	<10 sec	0 (real-time)	⚠️ Feed switching needed
API Gateway	<1 min	0 (stateless)	✅ Ready (stateless)

Scenario 1: Database Failure

Impact: Complete data persistence loss Recovery: <5 minutes

Detection

# Automatic: PostgreSQL connection pool alerts
# Manual: Check database connectivity
psql $DATABASE_URL -c "SELECT 1;" || echo "Database down!"

Recovery Procedure

# Step 1: Promote read replica to primary (if available)
# On replica server:
sudo systemctl stop postgresql-replica
sudo -u postgres /usr/lib/postgresql/16/bin/pg_ctl promote -D /var/lib/postgresql/16/main

# Verify promotion
psql -h replica-server -c "SELECT pg_is_in_recovery();"
# Expected: f (false = not in recovery = primary)

# Step 2: Update connection strings
export OLD_PRIMARY=primary.foxhunt.internal
export NEW_PRIMARY=replica-1.foxhunt.internal

# Update environment variables (all services)
sudo systemctl set-environment DATABASE_URL=postgresql://foxhunt:password@$NEW_PRIMARY:5432/foxhunt

# Restart services to pick up new connection
sudo systemctl restart foxhunt-*

# Step 3: Verify data consistency
psql -h $NEW_PRIMARY -c "SELECT MAX(created_at) FROM orders;"
psql -h $NEW_PRIMARY -c "SELECT count(*) FROM positions;"

# Step 4: Create new replica from promoted primary
# (See Horizontal Scaling - PostgreSQL Read Replicas)

Backup Restoration (if no replica available)

# Step 1: Restore from backup
# Backups location: S3 bucket foxhunt-backups/postgres/
aws s3 cp s3://foxhunt-backups/postgres/latest.dump /tmp/postgres-restore.dump

# Step 2: Create new database
sudo -u postgres createdb foxhunt_restored

# Step 3: Restore data
pg_restore -h localhost -U foxhunt -d foxhunt_restored -v /tmp/postgres-restore.dump

# Step 4: Rename databases (quick switchover)
sudo -u postgres psql <<EOF
ALTER DATABASE foxhunt RENAME TO foxhunt_old;
ALTER DATABASE foxhunt_restored RENAME TO foxhunt;
EOF

# Step 5: Restart services
sudo systemctl restart foxhunt-*

# Estimated downtime: 2-5 minutes (depends on database size)

Scenario 2: Service Crash/Failure

Impact: Service unavailable Recovery: <1 minute (automatic restart)

Detection

# SystemD automatic restart (configured in service files)
# Restart=always
# RestartSec=10s

# Manual check
sudo systemctl status foxhunt-trading

Recovery Procedure

# Step 1: Automatic restart (SystemD handles)
# No action needed if service restarts successfully

# Step 2: If restart fails, check logs
sudo journalctl -u foxhunt-trading -n 100 --no-pager

# Step 3: Common issues and fixes

# Issue: Port already in use
sudo lsof -i :50052
sudo kill -9 <PID>
sudo systemctl start foxhunt-trading

# Issue: Database connection failure
# Verify database is running
sudo systemctl status postgresql

# Issue: Corrupted cache/state
sudo rm -rf /tmp/foxhunt/*
sudo systemctl start foxhunt-trading

# Step 4: If service still fails, rollback
sudo systemctl stop foxhunt-trading
sudo cp /opt/foxhunt/bin/trading_service.backup /opt/foxhunt/bin/trading_service
sudo systemctl start foxhunt-trading

Scenario 3: Network Partition (Split Brain)

Impact: Service isolation, data inconsistency risk Recovery: <3 minutes

Detection

# Services detect partition via health checks
# Logs will show: "Network partition detected - entering read-only mode"

# Manual detection
for service in trading backtesting ml_training; do
  curl http://$service.foxhunt.internal:9092/metrics | grep network_partition_active
done

Recovery Procedure

# Step 1: Services automatically enter read-only mode
# No trading operations performed during partition

# Step 2: Restore network connectivity
# Fix network infrastructure (routers, switches, firewalls)

# Step 3: Verify connectivity restored
for service in trading-1 trading-2 trading-3; do
  ping -c 3 $service.foxhunt.internal
done

# Step 4: Services automatically reconcile state
# PostgreSQL replication catches up
# Redis cluster nodes rejoin

# Step 5: Verify data consistency
psql -h primary -c "SELECT count(*) FROM orders;" > /tmp/primary_count
psql -h replica -c "SELECT count(*) FROM orders;" > /tmp/replica_count
diff /tmp/primary_count /tmp/replica_count
# Expected: No difference

# Step 6: Resume normal operations
# Services automatically exit read-only mode after health checks pass

Scenario 4: Complete System Failure (Data Center Outage)

Impact: All services offline Recovery: <15 minutes (cold start)

Recovery Procedure

# Step 1: Restore infrastructure
# Power on servers
# Verify network connectivity

# Step 2: Start infrastructure services
sudo systemctl start postgresql
sudo systemctl start redis
sudo systemctl start vault

# Wait for health
sleep 30

# Step 3: Unseal Vault (if using production seal)
export VAULT_ADDR=http://localhost:8200
vault operator unseal <unseal-key-1>
vault operator unseal <unseal-key-2>
vault operator unseal <unseal-key-3>

# Verify unsealed
vault status

# Step 4: Start application services (in order)
sudo systemctl start foxhunt-trading
sudo systemctl start foxhunt-backtesting
sudo systemctl start foxhunt-ml-training
sleep 30
sudo systemctl start foxhunt-api-gateway

# Step 5: Verify full system health
./deployment/health_check.sh --mode comprehensive

# Step 6: Run data integrity checks
./deployment/scripts/data-integrity-check.sh

# Step 7: Resume trading operations
# Notify operations team
# Monitor for anomalies for 1 hour

Backup Procedures

Database Backups

# Automated daily backup (cron job)
# /etc/cron.daily/foxhunt-backup

#!/bin/bash
set -euo pipefail

BACKUP_DIR=/var/backups/foxhunt
S3_BUCKET=s3://foxhunt-backups/postgres
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup
pg_dump -h localhost -U foxhunt -Fc foxhunt > $BACKUP_DIR/foxhunt_$DATE.dump

# Compress
gzip $BACKUP_DIR/foxhunt_$DATE.dump

# Upload to S3
aws s3 cp $BACKUP_DIR/foxhunt_$DATE.dump.gz $S3_BUCKET/

# Keep local backups for 7 days
find $BACKUP_DIR -name "*.dump.gz" -mtime +7 -delete

# Verify backup
pg_restore --list $BACKUP_DIR/foxhunt_$DATE.dump.gz > /dev/null

Configuration Backups

# Backup configurations to S3 daily
#!/bin/bash
S3_BUCKET=s3://foxhunt-backups/config
DATE=$(date +%Y%m%d)

# Backup configuration files
tar -czf /tmp/foxhunt-config-$DATE.tar.gz /opt/foxhunt/config

# Upload to S3
aws s3 cp /tmp/foxhunt-config-$DATE.tar.gz $S3_BUCKET/

# Cleanup
rm /tmp/foxhunt-config-$DATE.tar.gz

Model Backups

# ML models are already stored in S3 (foxhunt-models bucket)
# Checkpoints stored in S3 (foxhunt-checkpoints bucket)
# Retention: Latest 10 checkpoints per model

# Verify model backups
aws s3 ls s3://foxhunt-models/ --recursive | grep -E "mamba2|dqn|ppo|tft"

# Verify checkpoint backups
aws s3 ls s3://foxhunt-checkpoints/ --recursive | tail -10

---

Monitoring and Alerting

Critical Metrics

System Health Metrics

Metric	Warning Threshold	Critical Threshold	Action
Order latency (p99)	>50μs	>100μs	Scale horizontally
Error rate	>0.1%	>1%	Page on-call engineer
Memory usage	>80%	>90%	Restart service, add memory
CPU usage	>70%	>90%	Scale horizontally
Disk usage	>80%	>90%	Expand disk, archive data
Database connections	>80%	>95%	Increase pool size
Queue depth	>1000	>5000	Scale workers

Grafana Dashboards

Access: http://localhost:3000 (admin / foxhunt123)

Pre-configured Dashboards:

1. HFT Trading Performance

   • Order latency (p50, p95, p99)
   • Throughput (orders/second)
   • Error rates by error type
   • Position counts

2. System Resources

   • CPU usage per service
   • Memory usage per service
   • Disk I/O
   • Network bandwidth

3. Database Performance

   • Connection pool utilization
   • Query latency
   • Active queries
   • Replication lag

4. ML Model Performance

   • Inference latency
   • Model accuracy
   • Cache hit rate
   • GPU utilization

Prometheus Alerts

Alert Rules (config/prometheus/rules/alerts.yml):

groups:
- name: foxhunt_critical
  interval: 30s
  rules:
  # Trading latency
  - alert: HighOrderLatency
    expr: histogram_quantile(0.99, rate(order_processing_duration_seconds_bucket[5m])) > 0.0001
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High order processing latency"
      description: "P99 latency is {{ $value }}s (threshold: 100μs)"

  # Error rate
  - alert: HighErrorRate
    expr: rate(foxhunt_errors_total[5m]) > 0.01
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "High error rate detected"
      description: "Error rate is {{ $value }} (threshold: 1%)"

  # Service down
  - alert: ServiceDown
    expr: up{job=~"foxhunt.*"} == 0
    for: 30s
    labels:
      severity: critical
    annotations:
      summary: "Service {{ $labels.job }} is down"
      description: "Service has been down for 30 seconds"

  # Database connections
  - alert: HighDatabaseConnections
    expr: pg_stat_database_numbackends / pg_settings_max_connections > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High database connection usage"
      description: "{{ $value | humanizePercentage }} of max connections in use"

  # Disk usage
  - alert: HighDiskUsage
    expr: (node_filesystem_size_bytes - node_filesystem_free_bytes) / node_filesystem_size_bytes > 0.9
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High disk usage on {{ $labels.instance }}"
      description: "Disk usage is {{ $value | humanizePercentage }}"

  # GPU utilization (ML services)
  - alert: GPUNotDetected
    expr: nvidia_gpu_count == 0
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "GPU not detected in ML service"
      description: "ML service running without GPU acceleration"

Alert Routing (AlertManager)

Configuration (config/prometheus/alertmanager.yml):

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'default'
  routes:
  # Critical alerts -> Page on-call
  - match:
      severity: critical
    receiver: 'pagerduty'
    continue: true

  # Warning alerts -> Slack
  - match:
      severity: warning
    receiver: 'slack'

receivers:
- name: 'default'
  email_configs:
  - to: 'ops@foxhunt.trading'
    from: 'alerts@foxhunt.trading'
    smarthost: 'smtp.gmail.com:587'
    auth_username: 'alerts@foxhunt.trading'
    auth_password: '<app-password>'

- name: 'pagerduty'
  pagerduty_configs:
  - service_key: '<pagerduty-service-key>'
    description: '{{ .GroupLabels.alertname }}: {{ .Annotations.summary }}'

- name: 'slack'
  slack_configs:
  - api_url: '<slack-webhook-url>'
    channel: '#foxhunt-alerts'
    title: '{{ .GroupLabels.alertname }}'
    text: '{{ .Annotations.description }}'

Log Aggregation

Locations:

• SystemD logs: journalctl -u foxhunt-* -f
• Application logs: /opt/foxhunt/logs/*.log
• Docker logs: docker-compose logs -f

Log Levels:

• ERROR: Failures requiring immediate attention
• WARN: Potential issues, degraded performance
• INFO: Normal operations, state changes
• DEBUG: Detailed diagnostics (disabled in production)

Important Log Patterns:

# Monitor errors in real-time
journalctl -u foxhunt-* -f | grep ERROR

# Check for panics
journalctl -u foxhunt-* --since "1 hour ago" | grep -i panic

# Monitor database connection issues
journalctl -u foxhunt-trading -f | grep "database connection"

# Check for configuration errors (Agent 96 finding)
journalctl -u foxhunt-* | grep "Configuration error"

---

Troubleshooting

Common Issues and Solutions

Issue 1: Service Won't Start

Agent 96 Finding: Backtesting service failed due to missing BENZINGA_API_KEY.

Symptoms:

sudo systemctl status foxhunt-trading
# Output: failed (code=exited, status=1)

Diagnosis:

# Check logs for error
sudo journalctl -u foxhunt-trading -n 50 --no-pager

# Common errors:
# - "Configuration error in field 'api_key'"
# - "Failed to connect to database"
# - "Port already in use"
# - "Failed to load JWT secret"

Solutions:

Missing Environment Variable:

# Check environment variables
sudo systemctl show foxhunt-trading | grep Environment

# Add missing variable
sudo systemctl edit foxhunt-trading
# Add in override:
[Service]
Environment="BENZINGA_API_KEY=<your-key>"

# Restart
sudo systemctl restart foxhunt-trading

Port Already in Use:

# Find process using port
sudo lsof -i :50052

# Kill process
sudo kill -9 <PID>

# Restart service
sudo systemctl start foxhunt-trading

Database Connection Failure:

# Verify database is running
sudo systemctl status postgresql

# Test connection
psql $DATABASE_URL -c "SELECT 1;"

# Check connection pool settings
# trading_service/src/config.rs
# Increase pool size if needed

JWT Secret Validation Failure:

# Issue: JWT secret doesn't meet requirements (64+ chars, mixed case, symbols)
# Generate new secret
openssl rand -base64 64 | tr -d '\n'

# Update environment variable
export JWT_SECRET="<new-secret>"

# Restart service
sudo systemctl restart foxhunt-trading

Issue 2: High Latency

Symptoms:

# Prometheus metric shows high latency
curl http://localhost:9092/metrics | grep order_processing_duration_seconds

# Grafana dashboard shows p99 >100μs

Diagnosis:

# Check CPU frequency scaling (should be performance mode for HFT)
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Check CPU affinity
ps -eLo pid,tid,psr,comm | grep trading_service

# Check network latency
ping -c 10 <exchange-server>

# Check database query performance
psql $DATABASE_URL -c "SELECT query, mean_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

Solutions:

CPU Frequency Scaling:

# Set performance governor
sudo cpupower frequency-set -g performance

# Verify
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Expected: performance

CPU Affinity:

# Pin trading service to specific cores (avoid context switching)
sudo systemctl edit foxhunt-trading
# Add:
[Service]
CPUAffinity=0-7  # Use cores 0-7 exclusively

# Restart
sudo systemctl restart foxhunt-trading

Database Query Optimization:

# Add indexes for slow queries
psql $DATABASE_URL <<EOF
CREATE INDEX CONCURRENTLY idx_orders_created_at ON orders(created_at);
CREATE INDEX CONCURRENTLY idx_positions_symbol ON positions(symbol);
ANALYZE orders;
ANALYZE positions;
EOF

Network Latency:

# Use DPDK for kernel bypass (advanced)
# See: https://www.dpdk.org/

# Or enable TCP tuning
sudo sysctl -w net.ipv4.tcp_low_latency=1
sudo sysctl -w net.ipv4.tcp_fastopen=3
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728

Issue 3: Memory Issues

Symptoms:

# Service OOM killed
journalctl -u foxhunt-trading | grep "Out of memory"

# Docker container restarting
docker ps | grep Restarting

Diagnosis:

# Check memory usage
docker stats --no-stream foxhunt-trading-service

# Or for bare-metal
ps aux | grep trading_service | awk '{print $6}'

# Check for memory leaks
valgrind --leak-check=full /opt/foxhunt/bin/trading_service

Solutions:

Increase Memory Limit:

# Docker
docker-compose.yml:
  trading_service:
    deploy:
      resources:
        limits:
          memory: 2G  # Increase from 500M

# SystemD
sudo systemctl edit foxhunt-trading
[Service]
MemoryMax=2G

sudo systemctl restart foxhunt-trading

Fix Memory Leak (if identified):

# Review recent code changes
# Use Rust memory profiling tools: heaptrack, massif

# Check for:
# - Unbounded collections (Vec, HashMap without cleanup)
# - Circular references (Rc/Arc cycles)
# - Not dropping file handles

Issue 4: GPU Access Problems

Agent 96 Finding: ML Training Service requires NVIDIA runtime in Docker.

Symptoms:

# nvidia-smi not found
docker exec foxhunt-ml-training-service nvidia-smi
# Output: executable file not found in $PATH

Diagnosis:

# Check GPU on host
nvidia-smi

# Check Docker runtime
docker info | grep -i runtime

# Check if container has GPU access
docker run --rm --gpus all nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi

Solutions:

Install nvidia-docker2:

# See Prerequisites -> GPU Requirements section

Update Dockerfile (Agent 96 fix):

# Change base image to include CUDA runtime
FROM nvidia/cuda:12.3.0-runtime-ubuntu22.04 as runtime

# Install CUDA libraries
RUN apt-get update && apt-get install -y \
    cuda-runtime-12-3 \
    && rm -rf /var/lib/apt/lists/*

Update docker-compose.yml:

ml_training_service:
  runtime: nvidia
  environment:
    - NVIDIA_VISIBLE_DEVICES=all
    - NVIDIA_DRIVER_CAPABILITIES=compute,utility

Rebuild and restart:

docker-compose build ml_training_service
docker-compose up -d ml_training_service

# Verify GPU access
docker-compose exec ml_training_service nvidia-smi

Issue 5: Configuration Errors

Agent 96 Finding: Missing configuration values cause service startup failures.

Symptoms:

# Service logs show configuration error
journalctl -u foxhunt-backtesting | grep "Configuration error"
# Output: "Configuration error in field 'api_key': Benzinga API key is required"

Diagnosis:

# Check all environment variables
sudo systemctl show foxhunt-backtesting | grep Environment

# Validate configuration
/opt/foxhunt/bin/backtesting_service config validate

Solutions:

Add Missing Variables:

# Create environment file
sudo vim /etc/foxhunt/backtesting.env

# Add:
BENZINGA_API_KEY=<your-key>
DATABASE_URL=postgresql://...
REDIS_URL=redis://...

# Update SystemD service
sudo systemctl edit foxhunt-backtesting
[Service]
EnvironmentFile=/etc/foxhunt/backtesting.env

# Restart
sudo systemctl restart foxhunt-backtesting

Validate All Services:

# Run configuration validation for all services
for service in trading backtesting ml_training api_gateway; do
  echo "Validating $service..."
  /opt/foxhunt/bin/${service}_service config validate
done

Issue 6: Database Connection Pool Exhausted

Symptoms:

# Error in logs
journalctl -u foxhunt-trading | grep "connection pool"
# Output: "Failed to acquire connection from pool: timeout"

Diagnosis:

# Check active connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"

# Check max connections
psql $DATABASE_URL -c "SHOW max_connections;"

# Check connection pool configuration
# services/trading_service/src/config.rs

Solutions:

Increase Pool Size:

# Edit configuration
# services/trading_service/src/config.rs
DatabaseConfig {
  max_connections: 100,  // Increase from default (20)
  min_connections: 10,
  connection_timeout: Duration::from_secs(30),
}

# Rebuild and restart
cargo build --release -p trading_service
sudo systemctl restart foxhunt-trading

Increase PostgreSQL Max Connections:

# Edit postgresql.conf
sudo vim /etc/postgresql/16/main/postgresql.conf

# Change:
max_connections = 200  # Increase from 100

# Restart PostgreSQL
sudo systemctl restart postgresql

Fix Connection Leaks:

# Audit code for connections not being returned to pool
# Ensure all database operations use connection pooling

# Check for long-running queries
psql $DATABASE_URL -c "SELECT pid, now() - query_start as duration, query FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC;"

# Kill long-running queries if needed
psql $DATABASE_URL -c "SELECT pg_terminate_backend(<PID>);"

---

Security Procedures

JWT Secret Rotation

Frequency: Every 90 days or after suspected compromise

Procedure:

# Step 1: Generate new secret
NEW_JWT_SECRET=$(openssl rand -base64 64 | tr -d '\n')

# Step 2: Store in Vault
vault kv put secret/foxhunt/jwt \
  secret="$NEW_JWT_SECRET" \
  previous_secret="$OLD_JWT_SECRET"  # Keep for transition period

# Step 3: Update services with dual-key support
# Services will accept both old and new tokens during transition

# Step 4: After 24 hours (all clients refreshed), remove old secret
vault kv put secret/foxhunt/jwt \
  secret="$NEW_JWT_SECRET"

# Step 5: Verify all clients using new secret
curl http://localhost:9091/metrics | grep jwt_validation_success_total

Vault Token Management

Procedure:

# Step 1: Rotate AppRole secret
vault write -f auth/approle/role/foxhunt-services/secret-id

# Step 2: Get new secret ID
NEW_SECRET_ID=$(vault write -f -format=json auth/approle/role/foxhunt-services/secret-id | jq -r .data.secret_id)

# Step 3: Update services
sudo systemctl set-environment VAULT_SECRET_ID="$NEW_SECRET_ID"

# Step 4: Restart services to pick up new secret
sudo systemctl restart foxhunt-*

# Step 5: Revoke old secret (after verifying new one works)
vault write auth/approle/role/foxhunt-services/secret-id-accessor/<old-accessor> destroy=true

TLS Certificate Renewal

Frequency: Every 90 days (Let's Encrypt) or per certificate lifetime

Procedure:

# Using cert-manager (Kubernetes)
# Automatic renewal - no action needed

# Using Let's Encrypt (bare-metal)
sudo certbot renew --nginx

# Manual renewal
sudo certbot certonly --nginx -d api.foxhunt.trading

# Reload Nginx
sudo nginx -s reload

# Verify new certificate
echo | openssl s_client -servername api.foxhunt.trading -connect api.foxhunt.trading:443 2>/dev/null | openssl x509 -noout -dates

Audit Log Review

Frequency: Daily (automated), detailed review weekly

Procedure:

# Automated daily check for suspicious activity
cat <<'EOF' > /etc/cron.daily/foxhunt-audit-check
#!/bin/bash
set -euo pipefail

AUDIT_LOG=/var/log/foxhunt/audit.log
REPORT=/tmp/audit-report-$(date +%Y%m%d).txt

# Check for failed authentication attempts
echo "Failed auth attempts:" >> $REPORT
grep "authentication failed" $AUDIT_LOG | wc -l >> $REPORT

# Check for unauthorized access attempts
echo "Unauthorized access:" >> $REPORT
grep "unauthorized" $AUDIT_LOG | wc -l >> $REPORT

# Check for privileged operations
echo "Privileged operations:" >> $REPORT
grep "kill_switch activated" $AUDIT_LOG >> $REPORT

# Email report
mail -s "Foxhunt Daily Audit Report" ops@foxhunt.trading < $REPORT
EOF

chmod +x /etc/cron.daily/foxhunt-audit-check

Security Incident Response

Detection:

# Automated alerts from Prometheus/AlertManager
# Manual review of logs
# External security notifications

Immediate Response (first 15 minutes):

# Step 1: Activate kill switch (halt all trading)
curl -X POST http://localhost:8080/emergency/kill \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_MASTER_TOKEN"

# Step 2: Isolate affected services
sudo systemctl stop foxhunt-trading
sudo systemctl stop foxhunt-api-gateway

# Step 3: Enable read-only mode for remaining services
curl -X POST http://localhost:8082/mode/readonly

# Step 4: Capture forensic data
./deployment/scripts/capture-forensics.sh > /var/log/foxhunt/incident-$(date +%s).log

# Step 5: Notify security team
# Send alert with incident details

Investigation (next 1-4 hours):

# Review audit logs
grep "$(date +%Y-%m-%d)" /var/log/foxhunt/audit.log | grep -E "(failed|unauthorized|suspicious)"

# Check for unauthorized access
psql $DATABASE_URL -c "SELECT * FROM audit_logs WHERE action = 'unauthorized' AND created_at > NOW() - INTERVAL '24 hours';"

# Review network connections
sudo netstat -anp | grep ESTABLISHED | grep trading_service

# Check for malware/rootkits
sudo rkhunter --check
sudo chkrootkit

Remediation:

# Rotate all secrets (see above procedures)
# Apply security patches
# Update firewall rules if needed
# Review and update security policies

Recovery:

# After remediation complete, resume operations
curl -X POST http://localhost:8080/trading/resume \
  -H "X-Kill-Switch-Token: $KILL_SWITCH_MASTER_TOKEN"

# Monitor closely for 24 hours
watch -n 60 './deployment/health_check.sh --mode comprehensive'

---

Appendix A: Service Ports Reference

Service	External Port	Internal Port	Metrics Port	Health Port
PostgreSQL	5432	5432	-	-
Redis	6379	6379	-	-
Vault	8200	8200	-	-
InfluxDB	8086	8086	-	-
Prometheus	9090	9090	-	-
Grafana	3000	3000	-	-
Trading Service	50052	50051	9092	8080
Backtesting Service	50053	50052	9093	-
ML Training Service	50054	50053	9094	-
API Gateway	50051	50050	9091	-

Appendix B: Environment Variables Reference

See Prerequisites -> Environment Variables for complete reference.

Appendix C: Commands Quick Reference

# Start all services
docker-compose up -d

# Stop all services
docker-compose down

# View logs
docker-compose logs -f <service>
journalctl -u foxhunt-<service> -f

# Health check
./deployment/health_check.sh --mode comprehensive

# Restart service
sudo systemctl restart foxhunt-<service>
docker-compose restart <service>

# Check metrics
curl http://localhost:9092/metrics

# Database connection
psql $DATABASE_URL

# Vault login
export VAULT_ADDR=http://localhost:8200
vault login <token>

---

Document Version: 1.0.0 Last Updated: 2025-10-07 Next Review: After Wave 125 Phase 3B completion Maintainer: DevOps Team Contact: ops@foxhunt.trading