foxhunt/LOAD_BALANCING_SCALING.md

# Load Balancing & Horizontal Scaling Guide

**Last Updated**: 2025-10-07
**Status**: Production Ready
**Audience**: DevOps, SRE, Platform Engineers

---

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Load Balancer Architecture](#load-balancer-architecture)
3. [API Gateway Load Balancing](#api-gateway-load-balancing)
4. [Backend Service Scaling](#backend-service-scaling)
5. [Horizontal Pod Autoscaling (HPA)](#horizontal-pod-autoscaling-hpa)
6. [Database Scaling](#database-scaling)
7. [Redis Scaling](#redis-scaling)
8. [Performance Targets](#performance-targets)
9. [Cost Optimization](#cost-optimization)
10. [Monitoring & Alerting](#monitoring--alerting)

---

## Architecture Overview

### Scaling Topology

```
                        Internet
                           |
                           v
                   ┌───────────────┐
                   │  Load Balancer │ (nginx/HAProxy/AWS ALB)
                   │  TLS Termination│
                   │  DDoS Protection│
                   └───────┬────────┘
                           |
         ┌─────────────────┼─────────────────┐
         v                 v                 v
    ┌─────────┐      ┌─────────┐      ┌─────────┐
    │ Gateway │      │ Gateway │      │ Gateway │
    │  Pod 1  │      │  Pod 2  │      │  Pod N  │
    └────┬────┘      └────┬────┘      └────┬────┘
         |                |                 |
         └────────────────┼─────────────────┘
                          |
         ┌────────────────┼────────────────┐
         v                v                v
    ┌─────────┐      ┌──────────┐    ┌──────────┐
    │ Trading │      │Backtesting│   │ ML Train │
    │ Service │      │  Service  │   │  Service │
    │ (1-N)   │      │  (1-M)    │   │  (1-K)   │
    └────┬────┘      └─────┬─────┘   └────┬─────┘
         |                 |               |
         └─────────────────┼───────────────┘
                           |
         ┌─────────────────┼─────────────────┐
         v                 v                 v
    ┌──────────┐     ┌──────────┐     ┌──────────┐
    │PostgreSQL│     │  Redis   │     │InfluxDB  │
    │(Primary +│     │ Cluster  │     │  Cluster │
    │ Replicas)│     │          │     │          │
    └──────────┘     └──────────┘     └──────────┘
```

### Key Principles

1. **Stateless Services**: All application services are stateless for horizontal scaling
2. **External State**: State stored in PostgreSQL, Redis, or S3
3. **Health-Based Routing**: Load balancer only routes to healthy instances
4. **Graceful Degradation**: System continues with reduced capacity during failures
5. **Auto-Scaling**: Automatic scaling based on metrics (CPU, memory, request rate)

---

## Load Balancer Architecture

### Recommended Options

#### 1. **nginx** (Recommended for Kubernetes)
- **Pros**: Excellent gRPC support, lightweight, Kubernetes Ingress controller
- **Cons**: More complex configuration for advanced routing
- **Use Case**: Kubernetes deployments, high-performance edge routing

#### 2. **HAProxy** (Recommended for VM/Bare Metal)
- **Pros**: Battle-tested, advanced health checks, transparent gRPC support
- **Cons**: Steeper learning curve
- **Use Case**: VM-based deployments, multi-datacenter setups

#### 3. **AWS Application Load Balancer (ALB)**
- **Pros**: Managed service, auto-scaling, AWS integration
- **Cons**: Vendor lock-in, limited gRPC support (requires workarounds)
- **Use Case**: AWS-hosted deployments

#### 4. **Envoy Proxy** (Advanced)
- **Pros**: Native gRPC support, service mesh ready, advanced observability
- **Cons**: Complex configuration, resource overhead
- **Use Case**: Microservices mesh, multi-protocol routing

### gRPC Load Balancing Requirements

**Critical**: gRPC uses HTTP/2 with persistent connections, requiring **Layer 7 (L7) load balancing**.

**Why L4 Fails**:
```
Client ─── [L4 LB] ───> Backend 1 (gets ALL traffic)
    │                   Backend 2 (idle)
    │                   Backend 3 (idle)
    └─ Single TCP connection = Single backend
```

**Why L7 Works**:
```
Client ─── [L7 LB] ───> Backend 1 (33% requests)
    │                   Backend 2 (33% requests)
    │                   Backend 3 (34% requests)
    └─ Request-level balancing across HTTP/2 stream
```

### Health Check Configuration

**Health Check Requirements**:
1. **Protocol**: gRPC Health Checking Protocol (standard)
2. **Interval**: Every 5 seconds
3. **Timeout**: 2 seconds
4. **Threshold**: 2 consecutive failures = unhealthy
5. **Path**: `grpc.health.v1.Health/Check`

**Example Health Check**:
```bash
# Using grpc_health_probe
grpc_health_probe -addr=api-gateway:50051

# Response
status: SERVING
```

### Session Affinity (Sticky Sessions)

**When to Use**:
- Stateful operations (rare in our architecture)
- WebSocket connections (if added later)
- User-specific caching

**When NOT to Use**:
- Stateless services (Trading, Backtesting, ML Training)
- High-availability scenarios (sticky sessions reduce failover capability)

**Configuration**:
```nginx
# nginx session affinity (if needed)
upstream api_gateway {
    ip_hash;  # Session affinity based on client IP
    server gateway-1:50051;
    server gateway-2:50051;
}
```

---

## API Gateway Load Balancing

### Single Entry Point Pattern

**Architecture**:
```
External Clients (TLI, Mobile, Web)
         |
         v
   Load Balancer (TLS termination)
         |
         v
   API Gateway Cluster (N instances)
         |
         v
   Backend Services (Trading, Backtesting, ML)
```

### TLS Termination at Load Balancer

**Benefits**:
1. **Reduced Backend Load**: API Gateway doesn't handle TLS handshakes
2. **Certificate Management**: Single point for certificate renewal
3. **Simplified Backend**: Backend services can use plaintext gRPC
4. **Performance**: Hardware-accelerated TLS at load balancer

**Configuration**:
```nginx
# nginx TLS termination
server {
    listen 443 ssl http2;

    ssl_certificate /etc/ssl/certs/foxhunt.crt;
    ssl_certificate_key /etc/ssl/private/foxhunt.key;

    ssl_protocols TLSv1.3 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        grpc_pass grpc://api_gateway_backend;
    }
}
```

### Rate Limiting at Edge

**Why Edge Rate Limiting**:
1. **DDoS Protection**: Block attack traffic before it reaches backend
2. **Resource Efficiency**: Save backend resources for legitimate requests
3. **Fairness**: Prevent single client from monopolizing resources

**Rate Limit Tiers**:
```nginx
# nginx rate limiting by tier
limit_req_zone $jwt_user_tier zone=tier1:10m rate=100r/s;  # Free tier
limit_req_zone $jwt_user_tier zone=tier2:10m rate=500r/s;  # Premium
limit_req_zone $jwt_user_tier zone=tier3:10m rate=2000r/s; # Enterprise

location /api {
    limit_req zone=tier1 burst=20 nodelay;
    grpc_pass grpc://api_gateway_backend;
}
```

**Rate Limit Configuration**:
| Tier       | Rate Limit | Burst | Typical Use Case |
|------------|------------|-------|------------------|
| Free       | 100 req/s  | 20    | Retail traders   |
| Premium    | 500 req/s  | 100   | Active traders   |
| Enterprise | 2000 req/s | 500   | HFT firms        |
| Internal   | Unlimited  | N/A   | Service-to-service |

### DDoS Protection

**Layer 3/4 Protection** (Network/Transport):
```nginx
# nginx connection limits
limit_conn_zone $binary_remote_addr zone=addr:10m;
limit_conn addr 10;  # Max 10 concurrent connections per IP
```

**Layer 7 Protection** (Application):
```nginx
# Rate limiting (see above)
# Request size limits
client_max_body_size 1M;

# Timeout limits
grpc_read_timeout 30s;
grpc_send_timeout 30s;
```

**Additional Measures**:
1. **IP Blacklisting**: Block known malicious IPs
2. **Geo-Blocking**: Restrict access by country (if needed)
3. **Challenge-Response**: CAPTCHA for suspicious traffic
4. **Cloud-Based DDoS Protection**: Cloudflare, AWS Shield

---

## Backend Service Scaling

### Trading Service: Stateless Scaling

**Characteristics**:
- **Stateless**: No in-memory state (all state in PostgreSQL/Redis)
- **Horizontal Scaling**: Add instances linearly for more capacity
- **Auto-Scaling Trigger**: CPU > 70%, Request Rate > 1000 req/s

**Scaling Strategy**:
```yaml
# Kubernetes HPA for Trading Service
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: trading-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: trading-service
  minReplicas: 3  # Always maintain 3 for HA
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: grpc_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"
```

**Resource Requirements**:
```yaml
resources:
  requests:
    cpu: 500m      # 0.5 CPU core
    memory: 1Gi    # 1GB RAM
  limits:
    cpu: 2000m     # 2 CPU cores
    memory: 4Gi    # 4GB RAM
```

**Expected Performance Per Instance**:
- **Throughput**: 1,000-2,000 requests/second
- **Latency**: P99 < 5ms (order submission)
- **Concurrent Connections**: 1,000-5,000

### Backtesting Service: Job Queue Pattern

**Characteristics**:
- **Long-Running Jobs**: Backtests can take minutes to hours
- **CPU-Intensive**: High CPU usage during simulation
- **Job Queue**: Use Redis queue for job distribution

**Scaling Strategy**:
```yaml
# Kubernetes HPA for Backtesting Service (worker pool)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backtesting-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backtesting-service
  minReplicas: 2   # Minimum workers
  maxReplicas: 10  # Maximum workers
  metrics:
  - type: Pods
    pods:
      metric:
        name: redis_queue_length  # Scale on queue depth
      target:
        type: AverageValue
        averageValue: "5"  # 5 jobs per worker
```

**Job Queue Design**:
```
Client Request → API Gateway → Backtesting Service (Job Scheduler)
                                      ↓
                              Redis Job Queue
                                      ↓
                    Worker Pool (N instances)
                         ↓           ↓           ↓
                    Worker 1    Worker 2    Worker N
                         ↓           ↓           ↓
                    Results stored in PostgreSQL
```

**Resource Requirements** (per worker):
```yaml
resources:
  requests:
    cpu: 2000m     # 2 CPU cores (CPU-intensive)
    memory: 4Gi    # 4GB RAM (large datasets)
  limits:
    cpu: 4000m     # 4 CPU cores
    memory: 8Gi    # 8GB RAM
```

### ML Training Service: GPU Resource Allocation

**Characteristics**:
- **GPU-Dependent**: Requires NVIDIA GPU (RTX 3050 Ti or better)
- **Long-Running Jobs**: Training can take hours to days
- **Resource-Intensive**: High GPU memory, CPU, and RAM usage

**Scaling Strategy** (Kubernetes with NVIDIA GPU Operator):
```yaml
# ML Training Service with GPU affinity
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-training-service
spec:
  replicas: 2  # Manual scaling (GPU resources expensive)
  template:
    spec:
      containers:
      - name: ml-training-service
        image: foxhunt/ml-training-service:latest
        resources:
          requests:
            cpu: 4000m          # 4 CPU cores
            memory: 16Gi        # 16GB RAM
            nvidia.com/gpu: 1   # 1 GPU per pod
          limits:
            cpu: 8000m          # 8 CPU cores
            memory: 32Gi        # 32GB RAM
            nvidia.com/gpu: 1
        env:
        - name: CUDA_VISIBLE_DEVICES
          value: "0"
```

**GPU Scheduling Considerations**:
1. **Node Affinity**: Schedule only on GPU-enabled nodes
2. **GPU Sharing**: Use NVIDIA MPS for multi-tenant GPU sharing (if needed)
3. **Preemption**: Low-priority training jobs can be preempted for inference
4. **Cost**: GPU instances are 5-10x more expensive than CPU

**Auto-Scaling NOT Recommended**:
- GPU instances take 2-5 minutes to provision
- Training jobs are scheduled, not real-time
- Use job queue pattern (like Backtesting Service)

### Database Connection Pooling

**Why Connection Pooling**:
- PostgreSQL connection limit: ~100-500 connections
- Each service instance needs 5-20 connections
- 20 service instances × 20 connections = 400 connections (nearing limit)

**Solution: PgBouncer**:
```
Service Instances (100+) → PgBouncer (pools to 50 connections) → PostgreSQL
```

**PgBouncer Configuration**:
```ini
# pgbouncer.ini
[databases]
foxhunt = host=postgres port=5432 dbname=foxhunt

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt

# Connection pooling
pool_mode = transaction  # Transaction-level pooling
max_client_conn = 1000   # Max connections from services
default_pool_size = 25   # Pool size per database
reserve_pool_size = 10   # Emergency pool
```

**Service Configuration** (use PgBouncer instead of direct PostgreSQL):
```bash
# Before (direct PostgreSQL)
DATABASE_URL=postgresql://foxhunt:password@postgres:5432/foxhunt

# After (via PgBouncer)
DATABASE_URL=postgresql://foxhunt:password@pgbouncer:6432/foxhunt
```

---

## Horizontal Pod Autoscaling (HPA)

### Metrics for Scaling Decisions

**CPU-Based Scaling** (Most Common):
```yaml
metrics:
- type: Resource
  resource:
    name: cpu
    target:
      type: Utilization
      averageUtilization: 70  # Scale when CPU > 70%
```

**Memory-Based Scaling**:
```yaml
metrics:
- type: Resource
  resource:
    name: memory
    target:
      type: Utilization
      averageUtilization: 80  # Scale when memory > 80%
```

**Custom Metrics** (Request Rate):
```yaml
metrics:
- type: Pods
  pods:
    metric:
      name: grpc_requests_per_second
    target:
      type: AverageValue
      averageValue: "1000"  # Scale when > 1000 req/s per pod
```

**Custom Metrics** (Queue Depth):
```yaml
metrics:
- type: Pods
  pods:
    metric:
      name: redis_queue_length
    target:
      type: AverageValue
      averageValue: "5"  # Scale when > 5 jobs per worker
```

### Scale-Up Thresholds

**Aggressive Scale-Up** (Trading Service - latency-sensitive):
```yaml
behavior:
  scaleUp:
    stabilizationWindowSeconds: 30  # Wait 30s before scaling up
    policies:
    - type: Percent
      value: 100  # Double replicas
      periodSeconds: 15
    - type: Pods
      value: 4    # Or add 4 pods
      periodSeconds: 15
    selectPolicy: Max  # Use whichever scales faster
```

**Conservative Scale-Up** (Backtesting Service - cost-sensitive):
```yaml
behavior:
  scaleUp:
    stabilizationWindowSeconds: 60  # Wait 60s before scaling up
    policies:
    - type: Percent
      value: 50   # Add 50% more replicas
      periodSeconds: 60
    - type: Pods
      value: 2    # Or add 2 pods
      periodSeconds: 60
    selectPolicy: Min  # Use whichever scales slower
```

### Scale-Down Thresholds

**Slow Scale-Down** (Prevent flapping):
```yaml
behavior:
  scaleDown:
    stabilizationWindowSeconds: 300  # Wait 5 minutes before scaling down
    policies:
    - type: Percent
      value: 25   # Remove 25% of replicas
      periodSeconds: 60
    - type: Pods
      value: 1    # Or remove 1 pod
      periodSeconds: 60
    selectPolicy: Min  # Use whichever scales slower
```

**Why Slow Scale-Down**:
1. **Prevent Flapping**: Avoid rapid up/down cycles (costs CPU, disrupts connections)
2. **Traffic Spikes**: Allow system to handle sudden load increases
3. **Cost**: Over-provisioning by 10-20% is cheaper than SLA violations

### Min/Max Replica Counts

**Recommended Configurations**:

| Service           | Min Replicas | Max Replicas | Reasoning |
|-------------------|--------------|--------------|-----------|
| API Gateway       | 3            | 20           | HA (3) + burst capacity |
| Trading Service   | 3            | 20           | HA (3) + 50K orders/sec |
| Backtesting       | 2            | 10           | Workers for job queue |
| ML Training       | 1            | 3            | GPU resources expensive |

**Min Replicas Rationale**:
- **3 for HA**: Survive 1 node failure + 1 rolling update
- **2 for Workers**: Minimum for job queue parallelism
- **1 for GPU**: GPU resources too expensive for idle HA

**Max Replicas Rationale**:
- **20 for Trading**: Based on performance testing (50K orders/sec ÷ 2.5K orders/sec per pod)
- **10 for Backtesting**: Rarely need more than 10 concurrent backtests
- **3 for ML Training**: GPU cluster size limit

---

## Database Scaling

### Read Replicas for Read-Heavy Workloads

**When to Use Read Replicas**:
- **Read:Write Ratio**: > 3:1 (75%+ reads)
- **Query Patterns**: Analytics, reporting, backtesting
- **Write Latency**: Not affected by read load

**PostgreSQL Streaming Replication**:
```
Primary (Write) ──────> Replica 1 (Read)
    │                   Replica 2 (Read)
    │                   Replica 3 (Read)
    └─ Async replication (< 100ms lag)
```

**Configuration**:
```yaml
# Kubernetes StatefulSet with read replicas
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-replica
spec:
  replicas: 2  # 2 read replicas
  serviceName: postgres-replica
  template:
    spec:
      containers:
      - name: postgres
        image: timescale/timescaledb:latest-pg15
        env:
        - name: POSTGRES_REPLICA_MODE
          value: "replica"
        - name: POSTGRES_MASTER_HOST
          value: "postgres-primary"
```

**Application-Level Read/Write Splitting**:
```rust
// trading_engine/src/persistence/postgres.rs
pub struct PostgresPool {
    write_pool: sqlx::PgPool,  // Primary
    read_pool: sqlx::PgPool,   // Read replica(s)
}

impl PostgresPool {
    pub async fn execute_write(&self, query: Query) -> Result<()> {
        // All writes go to primary
        self.write_pool.execute(query).await
    }

    pub async fn execute_read(&self, query: Query) -> Result<Rows> {
        // Reads go to replica (or primary if replica unavailable)
        self.read_pool.fetch_all(query).await
    }
}
```

### Connection Pooling (PgBouncer)

**See "Backend Service Scaling" section for PgBouncer configuration.**

**Benefits**:
1. **Reduced Connection Overhead**: Reuse connections instead of creating new ones
2. **Connection Limit Management**: Protect PostgreSQL from connection exhaustion
3. **Transaction Pooling**: Release connections immediately after transaction

**Pooling Modes**:
```ini
# Transaction pooling (recommended for stateless services)
pool_mode = transaction  # Connection released after transaction

# Session pooling (for stateful operations)
pool_mode = session      # Connection held for entire session

# Statement pooling (most aggressive, use with caution)
pool_mode = statement    # Connection released after each statement
```

### Sharding Strategy (Future - If Needed)

**When to Shard**:
- Database size > 1TB
- Write throughput > 10K writes/sec
- Read replicas insufficient

**Sharding Strategies**:

1. **By Asset Class** (Natural partition):
```sql
-- Shard 1: Equities
-- Shard 2: Options
-- Shard 3: Futures
-- Shard 4: Crypto
```

2. **By Time** (TimescaleDB hypertables - automatic):
```sql
-- TimescaleDB automatically partitions by time
CREATE TABLE market_data (
    time TIMESTAMPTZ NOT NULL,
    symbol TEXT NOT NULL,
    price DOUBLE PRECISION
);

SELECT create_hypertable('market_data', 'time');
-- Automatic partitioning by month/week
```

3. **By User ID** (Hash-based):
```sql
-- Shard based on user_id hash
Shard = user_id % num_shards
```

**Recommendation**: Start with TimescaleDB hypertables (time-based partitioning). Only implement custom sharding if necessary.

### TimescaleDB Hypertables

**Already Configured** (from migrations):
```sql
-- Migration 002: TimescaleDB extension
CREATE EXTENSION IF NOT EXISTS timescaledb;

-- Migration 003: Market data hypertable
SELECT create_hypertable('market_data', 'timestamp',
    chunk_time_interval => INTERVAL '1 day');

-- Automatic compression for old data
ALTER TABLE market_data SET (
    timescaledb.compress,
    timescaledb.compress_orderby = 'timestamp DESC',
    timescaledb.compress_segmentby = 'symbol'
);
```

**Benefits**:
- **Automatic Partitioning**: By time (day/week/month)
- **Compression**: Old data automatically compressed (10x space savings)
- **Parallel Queries**: TimescaleDB parallelizes across chunks
- **Retention Policies**: Automatic data deletion after N days

---

## Redis Scaling

### Redis Cluster for Horizontal Scaling

**When to Use Redis Cluster**:
- Dataset size > 10GB (single node RAM limit)
- Throughput > 100K ops/sec
- High availability required

**Redis Cluster Architecture**:
```
Client ──> Redis Cluster Proxy
             │
             ├──> Shard 1 (Master + Replica)
             ├──> Shard 2 (Master + Replica)
             └──> Shard 3 (Master + Replica)
```

**Configuration** (docker-compose.yml):
```yaml
services:
  redis-cluster:
    image: redis:7-alpine
    command: redis-server --cluster-enabled yes --cluster-config-file nodes.conf
    ports:
      - "7000-7005:7000-7005"
    volumes:
      - redis-cluster-data:/data

  # Create cluster (run once)
  redis-cluster-init:
    image: redis:7-alpine
    command: |
      redis-cli --cluster create \
        redis-1:7000 redis-2:7001 redis-3:7002 \
        redis-4:7003 redis-5:7004 redis-6:7005 \
        --cluster-replicas 1
    depends_on:
      - redis-cluster
```

**Client Configuration**:
```rust
// Use redis-cluster-async crate
use redis_cluster_async::{Client, ClusterClient};

let client = ClusterClient::new(vec![
    "redis://redis-1:7000",
    "redis://redis-2:7001",
    "redis://redis-3:7002",
])?;

// Client automatically routes to correct shard
let mut conn = client.get_connection().await?;
conn.set("key", "value").await?;
```

### Redis Sentinel for High Availability

**When to Use Sentinel** (Alternative to Cluster):
- Single master + replicas (no sharding)
- Automatic failover required
- Dataset fits in single node RAM

**Sentinel Architecture**:
```
Client ──> Sentinel (monitors health)
             │
             v
         Primary ──> Replica 1
                 └─> Replica 2

(If Primary fails, Sentinel promotes Replica 1)
```

**Configuration**:
```yaml
# docker-compose.yml
services:
  redis-primary:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  redis-replica-1:
    image: redis:7-alpine
    command: redis-server --replicaof redis-primary 6379

  redis-sentinel:
    image: redis:7-alpine
    command: redis-sentinel /etc/redis/sentinel.conf
    volumes:
      - ./redis-sentinel.conf:/etc/redis/sentinel.conf
```

```conf
# redis-sentinel.conf
sentinel monitor mymaster redis-primary 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 10000
```

### Cache Warming Strategies

**Cold Cache Problem**:
- Cache miss rate spikes after restart
- Database overload from cache misses
- Slow response times until cache warms

**Solution 1: Lazy Loading** (Default):
```rust
// Check cache first, load from DB on miss
async fn get_user_config(user_id: &str, cache: &RedisPool, db: &PgPool) -> Result<UserConfig> {
    // Try cache first
    if let Some(config) = cache.get(user_id).await? {
        return Ok(config);
    }

    // Cache miss: Load from database
    let config = db.fetch_one("SELECT * FROM user_config WHERE user_id = $1", user_id).await?;

    // Warm cache for next request
    cache.set(user_id, &config, Duration::from_secs(3600)).await?;

    Ok(config)
}
```

**Solution 2: Proactive Warming**:
```rust
// Warm cache on startup with most-used data
async fn warm_cache(cache: &RedisPool, db: &PgPool) -> Result<()> {
    // Load top 1000 most active users
    let active_users = db.fetch_all(
        "SELECT user_id, config FROM user_config ORDER BY last_active DESC LIMIT 1000"
    ).await?;

    for user in active_users {
        cache.set(&user.user_id, &user.config, Duration::from_secs(3600)).await?;
    }

    Ok(())
}
```

**Solution 3: Cache Priming** (Scheduled):
```rust
// Periodically refresh cache for hot data
async fn prime_cache_job(cache: &RedisPool, db: &PgPool) -> Result<()> {
    loop {
        // Refresh market data cache every minute
        let latest_quotes = db.fetch_all("SELECT * FROM latest_quotes").await?;
        for quote in latest_quotes {
            cache.set(&quote.symbol, &quote, Duration::from_secs(60)).await?;
        }

        tokio::time::sleep(Duration::from_secs(60)).await;
    }
}
```

### Eviction Policies

**Redis Eviction Policies**:
```conf
# redis.conf
maxmemory 2gb
maxmemory-policy allkeys-lru  # Evict least recently used keys
```

**Policy Options**:
| Policy            | Description | Use Case |
|-------------------|-------------|----------|
| `noeviction`      | Return error when memory full | Never (avoid data loss) |
| `allkeys-lru`     | Evict least recently used keys | **Recommended for cache** |
| `allkeys-lfu`     | Evict least frequently used keys | Hot key workloads |
| `volatile-lru`    | Evict LRU keys with TTL set | Mixed cache + persistent data |
| `volatile-ttl`    | Evict keys with shortest TTL | Time-sensitive data |

**Recommendation**: Use `allkeys-lru` for cache workloads.

---

## Performance Targets

### Latency Targets (from PERFORMANCE_BENCHMARKS.md)

| Operation | P50 | P99 | P99.9 |
|-----------|-----|-----|-------|
| Order Submission | < 1ms | < 5ms | < 10ms |
| Market Data Ingestion | < 500μs | < 2ms | < 5ms |
| Position Query | < 2ms | < 10ms | < 20ms |
| Risk Check | < 1ms | < 5ms | < 10ms |

### Throughput Targets

| Service | Target Throughput | Peak Capacity |
|---------|------------------|---------------|
| API Gateway | 10K req/s | 50K req/s |
| Trading Service | 5K orders/s | 20K orders/s |
| Market Data Ingestion | 100K events/s | 500K events/s |
| Backtesting Service | 10 concurrent backtests | 50 concurrent backtests |

**Scaling Math**:
```
# Trading Service Example
Target: 20K orders/sec
Per-instance capacity: 2K orders/sec
Required instances: 20K ÷ 2K = 10 instances
Safety margin (20%): 10 × 1.2 = 12 instances
```

### Concurrent User Limits

| User Tier | Concurrent Connections | Rate Limit |
|-----------|----------------------|------------|
| Free | 1 | 100 req/s |
| Premium | 5 | 500 req/s |
| Enterprise | 50 | 2000 req/s |

### Resource Utilization Targets

**Target Utilization** (for auto-scaling):
- **CPU**: 60-70% (allows headroom for bursts)
- **Memory**: 70-80% (allows headroom for spikes)
- **Disk**: < 80% (alerts at 80%, critical at 90%)
- **Network**: < 50% (network saturation kills performance)

**Why NOT 90% Utilization**:
1. **No Headroom**: No capacity for traffic spikes
2. **Slow Auto-Scaling**: Takes 30-60s to add instances
3. **Latency Degradation**: High CPU = increased latency
4. **OOM Kills**: High memory usage risks out-of-memory crashes

---

## Cost Optimization

### Right-Sizing Instances

**CPU/Memory Ratios by Service**:

| Service | CPU | Memory | Ratio | Instance Type (AWS) |
|---------|-----|--------|-------|---------------------|
| API Gateway | 1 core | 2GB | 1:2 | t3.small |
| Trading Service | 2 cores | 4GB | 1:2 | t3.medium |
| Backtesting Service | 4 cores | 8GB | 1:2 | c6i.xlarge (compute-optimized) |
| ML Training Service | 8 cores | 32GB + GPU | 1:4 | g4dn.2xlarge (GPU) |

**How to Right-Size**:
1. **Monitor Actual Usage**: Use Prometheus metrics for 2 weeks
2. **Identify Bottlenecks**: CPU-bound vs memory-bound vs I/O-bound
3. **Adjust Instance Types**: Match instance type to workload
4. **Iterate**: Re-evaluate every quarter

### Spot Instances for Non-Critical Workloads

**What are Spot Instances**:
- Spare cloud capacity sold at 50-90% discount
- Can be terminated with 2-minute notice
- **Only use for fault-tolerant workloads**

**Recommended Use Cases**:
```yaml
# Kubernetes node groups
nodeGroups:
  # Production workloads (on-demand)
  - name: production-on-demand
    instanceType: t3.medium
    desiredCapacity: 10
    labels:
      workload: production

  # Backtesting workers (spot instances)
  - name: backtesting-spot
    instanceType: c6i.xlarge
    desiredCapacity: 5
    instancesDistribution:
      onDemandBaseCapacity: 1  # Always keep 1 on-demand
      onDemandPercentageAboveBaseCapacity: 0  # Rest are spot
      spotAllocationStrategy: capacity-optimized
    labels:
      workload: backtesting
```

**Never Use Spot For**:
- Trading Service (real-time, SLA-critical)
- API Gateway (user-facing)
- Databases (stateful)

**Good Use Cases**:
- Backtesting workers (jobs can retry)
- ML training (checkpointing allows resume)
- Batch processing

### Reserved Instances for Base Load

**What are Reserved Instances**:
- 1-year or 3-year commitment
- 30-60% discount vs on-demand
- Best for predictable, steady workload

**Recommended Strategy**:
```
Total Capacity Needed: 20 instances
├── Base Load (always needed): 10 instances → Reserved (50% savings)
├── Typical Load: +5 instances → On-Demand
└── Peak Load: +5 instances → Auto-Scaling (On-Demand or Spot)
```

**Cost Comparison** (AWS t3.medium, us-east-1):
| Pricing Model | Cost/month | Savings |
|---------------|------------|---------|
| On-Demand | $30.37 | 0% |
| Reserved (1yr, no upfront) | $18.25 | 40% |
| Reserved (3yr, all upfront) | $11.68 | 62% |
| Spot (average) | $9.11 | 70% |

### Auto-Scaling for Burst Capacity

**Cost-Optimized Auto-Scaling**:
```yaml
# HPA with cost awareness
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: trading-service-hpa
spec:
  minReplicas: 10  # Reserved instances (base load)
  maxReplicas: 20  # Auto-scale to 20 (on-demand/spot)
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

  # Slow scale-down to avoid flapping
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600  # 10 minutes
      policies:
      - type: Pods
        value: 1
        periodSeconds: 120  # Remove 1 pod every 2 minutes
```

**Cost Math**:
```
Scenario 1: No Auto-Scaling (always 20 instances)
Cost: 20 × $30.37 = $607.40/month

Scenario 2: Auto-Scaling (10 reserved, 10 auto-scale)
Base: 10 × $18.25 = $182.50 (reserved)
Burst: 10 × $30.37 × 30% = $91.11 (on-demand, 30% of time)
Total: $273.61/month
Savings: $333.79/month (55%)
```

---

## Monitoring & Alerting

### Key Metrics to Monitor

**Service Health**:
```promql
# Service availability (should be > 99.9%)
up{job="api-gateway"} == 1

# Request rate
rate(grpc_requests_total[5m])

# Error rate (should be < 0.1%)
rate(grpc_requests_total{status="error"}[5m]) / rate(grpc_requests_total[5m])

# Latency (P99 should be < 5ms)
histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket[5m]))
```

**Auto-Scaling Metrics**:
```promql
# Current replica count
kube_deployment_status_replicas{deployment="trading-service"}

# Desired replica count (from HPA)
kube_horizontalpodautoscaler_status_desired_replicas{hpa="trading-service-hpa"}

# CPU utilization
avg(rate(container_cpu_usage_seconds_total{pod=~"trading-service.*"}[5m])) * 100

# Memory utilization
avg(container_memory_working_set_bytes{pod=~"trading-service.*"}) /
avg(container_spec_memory_limit_bytes{pod=~"trading-service.*"}) * 100
```

**Database Metrics**:
```promql
# Connection pool usage (should be < 80%)
pg_stat_database_numbackends / pg_settings_max_connections * 100

# Replication lag (should be < 100ms)
pg_stat_replication_replay_lag_bytes

# Query latency
rate(pg_stat_database_blks_read[5m])
```

### Alerts to Configure

**Critical Alerts** (PagerDuty):
```yaml
# Service down
- alert: ServiceDown
  expr: up{job=~".*service"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Service {{ $labels.job }} is down"

# High error rate
- alert: HighErrorRate
  expr: rate(grpc_requests_total{status="error"}[5m]) / rate(grpc_requests_total[5m]) > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High error rate on {{ $labels.service }}: {{ $value | humanizePercentage }}"

# High latency
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket[5m])) > 0.1
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High P99 latency on {{ $labels.service }}: {{ $value }}s"
```

**Warning Alerts** (Slack):
```yaml
# High CPU usage (approaching auto-scale threshold)
- alert: HighCPU
  expr: avg(rate(container_cpu_usage_seconds_total[5m])) by (pod) > 0.8
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "High CPU on {{ $labels.pod }}: {{ $value | humanizePercentage }}"

# Database connection pool approaching limit
- alert: HighDatabaseConnections
  expr: pg_stat_database_numbackends / pg_settings_max_connections > 0.8
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Database connection pool at {{ $value | humanizePercentage }}"
```

### Dashboards

**Recommended Grafana Dashboards**:
1. **Service Overview**: Request rate, error rate, latency (RED metrics)
2. **Auto-Scaling**: Current/desired replicas, CPU/memory utilization
3. **Database**: Connection pool, query latency, replication lag
4. **Load Balancer**: Active connections, requests per backend
5. **Cost**: Instance count, spot vs on-demand usage

---

## Quick Reference

### Scaling Decision Matrix

| Symptom | Root Cause | Solution |
|---------|-----------|----------|
| High CPU (> 80%) | Not enough compute | Scale up replicas |
| High memory (> 90%) | Memory leak or under-provisioned | Increase memory limits or fix leak |
| High latency + low CPU | Database bottleneck | Add read replicas or optimize queries |
| High error rate | Service overload | Scale up replicas or rate limit clients |
| Database connection exhaustion | Too many service instances | Add PgBouncer connection pooling |
| Redis memory full | Cache size exceeds capacity | Add Redis Cluster or increase eviction |

### Scaling Commands

**Kubernetes**:
```bash
# Manual scaling
kubectl scale deployment trading-service --replicas=10

# Check HPA status
kubectl get hpa

# Describe HPA (shows scaling events)
kubectl describe hpa trading-service-hpa

# View pod resource usage
kubectl top pods

# View node resource usage
kubectl top nodes
```

**Docker Compose** (development):
```bash
# Scale service
docker-compose up -d --scale trading_service=5

# View resource usage
docker stats
```

### Load Balancer Health Checks

**nginx**:
```bash
# Check nginx status
curl http://localhost/nginx_status

# Test gRPC endpoint
grpc_health_probe -addr=localhost:443 -tls
```

**HAProxy**:
```bash
# Check HAProxy stats
curl http://localhost:8404/stats

# Test backend health
echo "show servers state" | socat /var/run/haproxy.sock stdio
```

---

## Additional Resources

- **Kubernetes HPA**: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
- **nginx gRPC Load Balancing**: https://www.nginx.com/blog/nginx-1-13-10-grpc/
- **PostgreSQL Connection Pooling**: https://www.pgbouncer.org/
- **Redis Cluster**: https://redis.io/docs/manual/scaling/
- **Performance Benchmarks**: See `PERFORMANCE_BENCHMARKS.md`
- **Scaling Playbook**: See `SCALING_PLAYBOOK.md`

---

**Last Updated**: 2025-10-07
**Version**: 1.0
**Owner**: Platform Engineering Team