foxhunt/LOAD_BALANCING_SCALING.md

Load Balancing & Horizontal Scaling Guide

Last Updated: 2025-10-07 Status: Production Ready Audience: DevOps, SRE, Platform Engineers

---

Table of Contents

1. Architecture Overview
2. Load Balancer Architecture
3. API Gateway Load Balancing
4. Backend Service Scaling
5. Horizontal Pod Autoscaling (HPA)
6. Database Scaling
7. Redis Scaling
8. Performance Targets
9. Cost Optimization
10. Monitoring & Alerting

---

Architecture Overview

Scaling Topology

                        Internet
                           |
                           v
                   ┌───────────────┐
                   │  Load Balancer │ (nginx/HAProxy/AWS ALB)
                   │  TLS Termination│
                   │  DDoS Protection│
                   └───────┬────────┘
                           |
         ┌─────────────────┼─────────────────┐
         v                 v                 v
    ┌─────────┐      ┌─────────┐      ┌─────────┐
    │ Gateway │      │ Gateway │      │ Gateway │
    │  Pod 1  │      │  Pod 2  │      │  Pod N  │
    └────┬────┘      └────┬────┘      └────┬────┘
         |                |                 |
         └────────────────┼─────────────────┘
                          |
         ┌────────────────┼────────────────┐
         v                v                v
    ┌─────────┐      ┌──────────┐    ┌──────────┐
    │ Trading │      │Backtesting│   │ ML Train │
    │ Service │      │  Service  │   │  Service │
    │ (1-N)   │      │  (1-M)    │   │  (1-K)   │
    └────┬────┘      └─────┬─────┘   └────┬─────┘
         |                 |               |
         └─────────────────┼───────────────┘
                           |
         ┌─────────────────┼─────────────────┐
         v                 v                 v
    ┌──────────┐     ┌──────────┐     ┌──────────┐
    │PostgreSQL│     │  Redis   │     │InfluxDB  │
    │(Primary +│     │ Cluster  │     │  Cluster │
    │ Replicas)│     │          │     │          │
    └──────────┘     └──────────┘     └──────────┘

Key Principles

1. Stateless Services: All application services are stateless for horizontal scaling
2. External State: State stored in PostgreSQL, Redis, or S3
3. Health-Based Routing: Load balancer only routes to healthy instances
4. Graceful Degradation: System continues with reduced capacity during failures
5. Auto-Scaling: Automatic scaling based on metrics (CPU, memory, request rate)

---

Load Balancer Architecture

Recommended Options

1. nginx (Recommended for Kubernetes)

• Pros: Excellent gRPC support, lightweight, Kubernetes Ingress controller
• Cons: More complex configuration for advanced routing
• Use Case: Kubernetes deployments, high-performance edge routing

2. HAProxy (Recommended for VM/Bare Metal)

• Pros: Battle-tested, advanced health checks, transparent gRPC support
• Cons: Steeper learning curve
• Use Case: VM-based deployments, multi-datacenter setups

3. AWS Application Load Balancer (ALB)

• Pros: Managed service, auto-scaling, AWS integration
• Cons: Vendor lock-in, limited gRPC support (requires workarounds)
• Use Case: AWS-hosted deployments

4. Envoy Proxy (Advanced)

• Pros: Native gRPC support, service mesh ready, advanced observability
• Cons: Complex configuration, resource overhead
• Use Case: Microservices mesh, multi-protocol routing

gRPC Load Balancing Requirements

Critical: gRPC uses HTTP/2 with persistent connections, requiring Layer 7 (L7) load balancing.

Why L4 Fails:

Client ─── [L4 LB] ───> Backend 1 (gets ALL traffic)
    │                   Backend 2 (idle)
    │                   Backend 3 (idle)
    └─ Single TCP connection = Single backend

Why L7 Works:

Client ─── [L7 LB] ───> Backend 1 (33% requests)
    │                   Backend 2 (33% requests)
    │                   Backend 3 (34% requests)
    └─ Request-level balancing across HTTP/2 stream

Health Check Configuration

Health Check Requirements:

1. Protocol: gRPC Health Checking Protocol (standard)
2. Interval: Every 5 seconds
3. Timeout: 2 seconds
4. Threshold: 2 consecutive failures = unhealthy
5. Path: grpc.health.v1.Health/Check

Example Health Check:

# Using grpc_health_probe
grpc_health_probe -addr=api-gateway:50051

# Response
status: SERVING

Session Affinity (Sticky Sessions)

When to Use:

• Stateful operations (rare in our architecture)
• WebSocket connections (if added later)
• User-specific caching

When NOT to Use:

• Stateless services (Trading, Backtesting, ML Training)
• High-availability scenarios (sticky sessions reduce failover capability)

Configuration:

# nginx session affinity (if needed)
upstream api_gateway {
    ip_hash;  # Session affinity based on client IP
    server gateway-1:50051;
    server gateway-2:50051;
}

---

API Gateway Load Balancing

Single Entry Point Pattern

Architecture:

External Clients (TLI, Mobile, Web)
         |
         v
   Load Balancer (TLS termination)
         |
         v
   API Gateway Cluster (N instances)
         |
         v
   Backend Services (Trading, Backtesting, ML)

TLS Termination at Load Balancer

Benefits:

1. Reduced Backend Load: API Gateway doesn't handle TLS handshakes
2. Certificate Management: Single point for certificate renewal
3. Simplified Backend: Backend services can use plaintext gRPC
4. Performance: Hardware-accelerated TLS at load balancer

Configuration:

# nginx TLS termination
server {
    listen 443 ssl http2;

    ssl_certificate /etc/ssl/certs/foxhunt.crt;
    ssl_certificate_key /etc/ssl/private/foxhunt.key;

    ssl_protocols TLSv1.3 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        grpc_pass grpc://api_gateway_backend;
    }
}

Rate Limiting at Edge

Why Edge Rate Limiting:

1. DDoS Protection: Block attack traffic before it reaches backend
2. Resource Efficiency: Save backend resources for legitimate requests
3. Fairness: Prevent single client from monopolizing resources

Rate Limit Tiers:

# nginx rate limiting by tier
limit_req_zone $jwt_user_tier zone=tier1:10m rate=100r/s;  # Free tier
limit_req_zone $jwt_user_tier zone=tier2:10m rate=500r/s;  # Premium
limit_req_zone $jwt_user_tier zone=tier3:10m rate=2000r/s; # Enterprise

location /api {
    limit_req zone=tier1 burst=20 nodelay;
    grpc_pass grpc://api_gateway_backend;
}

Rate Limit Configuration:

Tier	Rate Limit	Burst	Typical Use Case
Free	100 req/s	20	Retail traders
Premium	500 req/s	100	Active traders
Enterprise	2000 req/s	500	HFT firms
Internal	Unlimited	N/A	Service-to-service

DDoS Protection

Layer 3/4 Protection (Network/Transport):

# nginx connection limits
limit_conn_zone $binary_remote_addr zone=addr:10m;
limit_conn addr 10;  # Max 10 concurrent connections per IP

Layer 7 Protection (Application):

# Rate limiting (see above)
# Request size limits
client_max_body_size 1M;

# Timeout limits
grpc_read_timeout 30s;
grpc_send_timeout 30s;

Additional Measures:

1. IP Blacklisting: Block known malicious IPs
2. Geo-Blocking: Restrict access by country (if needed)
3. Challenge-Response: CAPTCHA for suspicious traffic
4. Cloud-Based DDoS Protection: Cloudflare, AWS Shield

---

Backend Service Scaling

Trading Service: Stateless Scaling

Characteristics:

• Stateless: No in-memory state (all state in PostgreSQL/Redis)
• Horizontal Scaling: Add instances linearly for more capacity
• Auto-Scaling Trigger: CPU > 70%, Request Rate > 1000 req/s

Scaling Strategy:

# Kubernetes HPA for Trading Service
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: trading-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: trading-service
  minReplicas: 3  # Always maintain 3 for HA
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: grpc_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

Resource Requirements:

resources:
  requests:
    cpu: 500m      # 0.5 CPU core
    memory: 1Gi    # 1GB RAM
  limits:
    cpu: 2000m     # 2 CPU cores
    memory: 4Gi    # 4GB RAM

Expected Performance Per Instance:

• Throughput: 1,000-2,000 requests/second
• Latency: P99 < 5ms (order submission)
• Concurrent Connections: 1,000-5,000

Backtesting Service: Job Queue Pattern

Characteristics:

• Long-Running Jobs: Backtests can take minutes to hours
• CPU-Intensive: High CPU usage during simulation
• Job Queue: Use Redis queue for job distribution

Scaling Strategy:

# Kubernetes HPA for Backtesting Service (worker pool)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backtesting-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backtesting-service
  minReplicas: 2   # Minimum workers
  maxReplicas: 10  # Maximum workers
  metrics:
  - type: Pods
    pods:
      metric:
        name: redis_queue_length  # Scale on queue depth
      target:
        type: AverageValue
        averageValue: "5"  # 5 jobs per worker

Job Queue Design:

Client Request → API Gateway → Backtesting Service (Job Scheduler)
                                      ↓
                              Redis Job Queue
                                      ↓
                    Worker Pool (N instances)
                         ↓           ↓           ↓
                    Worker 1    Worker 2    Worker N
                         ↓           ↓           ↓
                    Results stored in PostgreSQL

Resource Requirements (per worker):

resources:
  requests:
    cpu: 2000m     # 2 CPU cores (CPU-intensive)
    memory: 4Gi    # 4GB RAM (large datasets)
  limits:
    cpu: 4000m     # 4 CPU cores
    memory: 8Gi    # 8GB RAM

ML Training Service: GPU Resource Allocation

Characteristics:

• GPU-Dependent: Requires NVIDIA GPU (RTX 3050 Ti or better)
• Long-Running Jobs: Training can take hours to days
• Resource-Intensive: High GPU memory, CPU, and RAM usage

Scaling Strategy (Kubernetes with NVIDIA GPU Operator):

# ML Training Service with GPU affinity
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-training-service
spec:
  replicas: 2  # Manual scaling (GPU resources expensive)
  template:
    spec:
      containers:
      - name: ml-training-service
        image: foxhunt/ml-training-service:latest
        resources:
          requests:
            cpu: 4000m          # 4 CPU cores
            memory: 16Gi        # 16GB RAM
            nvidia.com/gpu: 1   # 1 GPU per pod
          limits:
            cpu: 8000m          # 8 CPU cores
            memory: 32Gi        # 32GB RAM
            nvidia.com/gpu: 1
        env:
        - name: CUDA_VISIBLE_DEVICES
          value: "0"

GPU Scheduling Considerations:

1. Node Affinity: Schedule only on GPU-enabled nodes
2. GPU Sharing: Use NVIDIA MPS for multi-tenant GPU sharing (if needed)
3. Preemption: Low-priority training jobs can be preempted for inference
4. Cost: GPU instances are 5-10x more expensive than CPU

Auto-Scaling NOT Recommended:

• GPU instances take 2-5 minutes to provision
• Training jobs are scheduled, not real-time
• Use job queue pattern (like Backtesting Service)

Database Connection Pooling

Why Connection Pooling:

• PostgreSQL connection limit: ~100-500 connections
• Each service instance needs 5-20 connections
• 20 service instances × 20 connections = 400 connections (nearing limit)

Solution: PgBouncer:

Service Instances (100+) → PgBouncer (pools to 50 connections) → PostgreSQL

PgBouncer Configuration:

# pgbouncer.ini
[databases]
foxhunt = host=postgres port=5432 dbname=foxhunt

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt

# Connection pooling
pool_mode = transaction  # Transaction-level pooling
max_client_conn = 1000   # Max connections from services
default_pool_size = 25   # Pool size per database
reserve_pool_size = 10   # Emergency pool

Service Configuration (use PgBouncer instead of direct PostgreSQL):

# Before (direct PostgreSQL)
DATABASE_URL=postgresql://foxhunt:password@postgres:5432/foxhunt

# After (via PgBouncer)
DATABASE_URL=postgresql://foxhunt:password@pgbouncer:6432/foxhunt

---

Horizontal Pod Autoscaling (HPA)

Metrics for Scaling Decisions

CPU-Based Scaling (Most Common):

metrics:
- type: Resource
  resource:
    name: cpu
    target:
      type: Utilization
      averageUtilization: 70  # Scale when CPU > 70%

Memory-Based Scaling:

metrics:
- type: Resource
  resource:
    name: memory
    target:
      type: Utilization
      averageUtilization: 80  # Scale when memory > 80%

Custom Metrics (Request Rate):

metrics:
- type: Pods
  pods:
    metric:
      name: grpc_requests_per_second
    target:
      type: AverageValue
      averageValue: "1000"  # Scale when > 1000 req/s per pod

Custom Metrics (Queue Depth):

metrics:
- type: Pods
  pods:
    metric:
      name: redis_queue_length
    target:
      type: AverageValue
      averageValue: "5"  # Scale when > 5 jobs per worker

Scale-Up Thresholds

Aggressive Scale-Up (Trading Service - latency-sensitive):

behavior:
  scaleUp:
    stabilizationWindowSeconds: 30  # Wait 30s before scaling up
    policies:
    - type: Percent
      value: 100  # Double replicas
      periodSeconds: 15
    - type: Pods
      value: 4    # Or add 4 pods
      periodSeconds: 15
    selectPolicy: Max  # Use whichever scales faster

Conservative Scale-Up (Backtesting Service - cost-sensitive):

behavior:
  scaleUp:
    stabilizationWindowSeconds: 60  # Wait 60s before scaling up
    policies:
    - type: Percent
      value: 50   # Add 50% more replicas
      periodSeconds: 60
    - type: Pods
      value: 2    # Or add 2 pods
      periodSeconds: 60
    selectPolicy: Min  # Use whichever scales slower

Scale-Down Thresholds

Slow Scale-Down (Prevent flapping):

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300  # Wait 5 minutes before scaling down
    policies:
    - type: Percent
      value: 25   # Remove 25% of replicas
      periodSeconds: 60
    - type: Pods
      value: 1    # Or remove 1 pod
      periodSeconds: 60
    selectPolicy: Min  # Use whichever scales slower

Why Slow Scale-Down:

1. Prevent Flapping: Avoid rapid up/down cycles (costs CPU, disrupts connections)
2. Traffic Spikes: Allow system to handle sudden load increases
3. Cost: Over-provisioning by 10-20% is cheaper than SLA violations

Min/Max Replica Counts

Recommended Configurations:

Service	Min Replicas	Max Replicas	Reasoning
API Gateway	3	20	HA (3) + burst capacity
Trading Service	3	20	HA (3) + 50K orders/sec
Backtesting	2	10	Workers for job queue
ML Training	1	3	GPU resources expensive

Min Replicas Rationale:

• 3 for HA: Survive 1 node failure + 1 rolling update
• 2 for Workers: Minimum for job queue parallelism
• 1 for GPU: GPU resources too expensive for idle HA

Max Replicas Rationale:

• 20 for Trading: Based on performance testing (50K orders/sec ÷ 2.5K orders/sec per pod)
• 10 for Backtesting: Rarely need more than 10 concurrent backtests
• 3 for ML Training: GPU cluster size limit

---

Database Scaling

Read Replicas for Read-Heavy Workloads

When to Use Read Replicas:

• Read:Write Ratio: > 3:1 (75%+ reads)
• Query Patterns: Analytics, reporting, backtesting
• Write Latency: Not affected by read load

PostgreSQL Streaming Replication:

Primary (Write) ──────> Replica 1 (Read)
    │                   Replica 2 (Read)
    │                   Replica 3 (Read)
    └─ Async replication (< 100ms lag)

Configuration:

# Kubernetes StatefulSet with read replicas
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-replica
spec:
  replicas: 2  # 2 read replicas
  serviceName: postgres-replica
  template:
    spec:
      containers:
      - name: postgres
        image: timescale/timescaledb:latest-pg15
        env:
        - name: POSTGRES_REPLICA_MODE
          value: "replica"
        - name: POSTGRES_MASTER_HOST
          value: "postgres-primary"

Application-Level Read/Write Splitting:

// trading_engine/src/persistence/postgres.rs
pub struct PostgresPool {
    write_pool: sqlx::PgPool,  // Primary
    read_pool: sqlx::PgPool,   // Read replica(s)
}

impl PostgresPool {
    pub async fn execute_write(&self, query: Query) -> Result<()> {
        // All writes go to primary
        self.write_pool.execute(query).await
    }

    pub async fn execute_read(&self, query: Query) -> Result<Rows> {
        // Reads go to replica (or primary if replica unavailable)
        self.read_pool.fetch_all(query).await
    }
}

Connection Pooling (PgBouncer)

See "Backend Service Scaling" section for PgBouncer configuration.

Benefits:

1. Reduced Connection Overhead: Reuse connections instead of creating new ones
2. Connection Limit Management: Protect PostgreSQL from connection exhaustion
3. Transaction Pooling: Release connections immediately after transaction

Pooling Modes:

# Transaction pooling (recommended for stateless services)
pool_mode = transaction  # Connection released after transaction

# Session pooling (for stateful operations)
pool_mode = session      # Connection held for entire session

# Statement pooling (most aggressive, use with caution)
pool_mode = statement    # Connection released after each statement

Sharding Strategy (Future - If Needed)

When to Shard:

• Database size > 1TB
• Write throughput > 10K writes/sec
• Read replicas insufficient

Sharding Strategies:

1. By Asset Class (Natural partition):

-- Shard 1: Equities
-- Shard 2: Options
-- Shard 3: Futures
-- Shard 4: Crypto

2. By Time (TimescaleDB hypertables - automatic):

-- TimescaleDB automatically partitions by time
CREATE TABLE market_data (
    time TIMESTAMPTZ NOT NULL,
    symbol TEXT NOT NULL,
    price DOUBLE PRECISION
);

SELECT create_hypertable('market_data', 'time');
-- Automatic partitioning by month/week

3. By User ID (Hash-based):

-- Shard based on user_id hash
Shard = user_id % num_shards

Recommendation: Start with TimescaleDB hypertables (time-based partitioning). Only implement custom sharding if necessary.

TimescaleDB Hypertables

Already Configured (from migrations):

-- Migration 002: TimescaleDB extension
CREATE EXTENSION IF NOT EXISTS timescaledb;

-- Migration 003: Market data hypertable
SELECT create_hypertable('market_data', 'timestamp',
    chunk_time_interval => INTERVAL '1 day');

-- Automatic compression for old data
ALTER TABLE market_data SET (
    timescaledb.compress,
    timescaledb.compress_orderby = 'timestamp DESC',
    timescaledb.compress_segmentby = 'symbol'
);

Benefits:

• Automatic Partitioning: By time (day/week/month)
• Compression: Old data automatically compressed (10x space savings)
• Parallel Queries: TimescaleDB parallelizes across chunks
• Retention Policies: Automatic data deletion after N days

---

Redis Scaling

Redis Cluster for Horizontal Scaling

When to Use Redis Cluster:

• Dataset size > 10GB (single node RAM limit)
• Throughput > 100K ops/sec
• High availability required

Redis Cluster Architecture:

Client ──> Redis Cluster Proxy
             │
             ├──> Shard 1 (Master + Replica)
             ├──> Shard 2 (Master + Replica)
             └──> Shard 3 (Master + Replica)

Configuration (docker-compose.yml):

services:
  redis-cluster:
    image: redis:7-alpine
    command: redis-server --cluster-enabled yes --cluster-config-file nodes.conf
    ports:
      - "7000-7005:7000-7005"
    volumes:
      - redis-cluster-data:/data

  # Create cluster (run once)
  redis-cluster-init:
    image: redis:7-alpine
    command: |
      redis-cli --cluster create \
        redis-1:7000 redis-2:7001 redis-3:7002 \
        redis-4:7003 redis-5:7004 redis-6:7005 \
        --cluster-replicas 1
    depends_on:
      - redis-cluster

Client Configuration:

// Use redis-cluster-async crate
use redis_cluster_async::{Client, ClusterClient};

let client = ClusterClient::new(vec![
    "redis://redis-1:7000",
    "redis://redis-2:7001",
    "redis://redis-3:7002",
])?;

// Client automatically routes to correct shard
let mut conn = client.get_connection().await?;
conn.set("key", "value").await?;

Redis Sentinel for High Availability

When to Use Sentinel (Alternative to Cluster):

• Single master + replicas (no sharding)
• Automatic failover required
• Dataset fits in single node RAM

Sentinel Architecture:

Client ──> Sentinel (monitors health)
             │
             v
         Primary ──> Replica 1
                 └─> Replica 2

(If Primary fails, Sentinel promotes Replica 1)

Configuration:

# docker-compose.yml
services:
  redis-primary:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  redis-replica-1:
    image: redis:7-alpine
    command: redis-server --replicaof redis-primary 6379

  redis-sentinel:
    image: redis:7-alpine
    command: redis-sentinel /etc/redis/sentinel.conf
    volumes:
      - ./redis-sentinel.conf:/etc/redis/sentinel.conf

# redis-sentinel.conf
sentinel monitor mymaster redis-primary 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 10000

Cache Warming Strategies

Cold Cache Problem:

• Cache miss rate spikes after restart
• Database overload from cache misses
• Slow response times until cache warms

Solution 1: Lazy Loading (Default):

// Check cache first, load from DB on miss
async fn get_user_config(user_id: &str, cache: &RedisPool, db: &PgPool) -> Result<UserConfig> {
    // Try cache first
    if let Some(config) = cache.get(user_id).await? {
        return Ok(config);
    }

    // Cache miss: Load from database
    let config = db.fetch_one("SELECT * FROM user_config WHERE user_id = $1", user_id).await?;

    // Warm cache for next request
    cache.set(user_id, &config, Duration::from_secs(3600)).await?;

    Ok(config)
}

Solution 2: Proactive Warming:

// Warm cache on startup with most-used data
async fn warm_cache(cache: &RedisPool, db: &PgPool) -> Result<()> {
    // Load top 1000 most active users
    let active_users = db.fetch_all(
        "SELECT user_id, config FROM user_config ORDER BY last_active DESC LIMIT 1000"
    ).await?;

    for user in active_users {
        cache.set(&user.user_id, &user.config, Duration::from_secs(3600)).await?;
    }

    Ok(())
}

Solution 3: Cache Priming (Scheduled):

// Periodically refresh cache for hot data
async fn prime_cache_job(cache: &RedisPool, db: &PgPool) -> Result<()> {
    loop {
        // Refresh market data cache every minute
        let latest_quotes = db.fetch_all("SELECT * FROM latest_quotes").await?;
        for quote in latest_quotes {
            cache.set(&quote.symbol, &quote, Duration::from_secs(60)).await?;
        }

        tokio::time::sleep(Duration::from_secs(60)).await;
    }
}

Eviction Policies

Redis Eviction Policies:

# redis.conf
maxmemory 2gb
maxmemory-policy allkeys-lru  # Evict least recently used keys

Policy Options:

Policy	Description	Use Case
noeviction	Return error when memory full	Never (avoid data loss)
allkeys-lru	Evict least recently used keys	Recommended for cache
allkeys-lfu	Evict least frequently used keys	Hot key workloads
volatile-lru	Evict LRU keys with TTL set	Mixed cache + persistent data
volatile-ttl	Evict keys with shortest TTL	Time-sensitive data

Recommendation: Use allkeys-lru for cache workloads.

---

Performance Targets

Latency Targets (from PERFORMANCE_BENCHMARKS.md)

Operation	P50	P99	P99.9
Order Submission	< 1ms	< 5ms	< 10ms
Market Data Ingestion	< 500μs	< 2ms	< 5ms
Position Query	< 2ms	< 10ms	< 20ms
Risk Check	< 1ms	< 5ms	< 10ms

Throughput Targets

Service	Target Throughput	Peak Capacity
API Gateway	10K req/s	50K req/s
Trading Service	5K orders/s	20K orders/s
Market Data Ingestion	100K events/s	500K events/s
Backtesting Service	10 concurrent backtests	50 concurrent backtests

Scaling Math:

# Trading Service Example
Target: 20K orders/sec
Per-instance capacity: 2K orders/sec
Required instances: 20K ÷ 2K = 10 instances
Safety margin (20%): 10 × 1.2 = 12 instances

Concurrent User Limits

User Tier	Concurrent Connections	Rate Limit
Free	1	100 req/s
Premium	5	500 req/s
Enterprise	50	2000 req/s

Resource Utilization Targets

Target Utilization (for auto-scaling):

• CPU: 60-70% (allows headroom for bursts)
• Memory: 70-80% (allows headroom for spikes)
• Disk: < 80% (alerts at 80%, critical at 90%)
• Network: < 50% (network saturation kills performance)

Why NOT 90% Utilization:

1. No Headroom: No capacity for traffic spikes
2. Slow Auto-Scaling: Takes 30-60s to add instances
3. Latency Degradation: High CPU = increased latency
4. OOM Kills: High memory usage risks out-of-memory crashes

---

Cost Optimization

Right-Sizing Instances

CPU/Memory Ratios by Service:

Service	CPU	Memory	Ratio	Instance Type (AWS)
API Gateway	1 core	2GB	1:2	t3.small
Trading Service	2 cores	4GB	1:2	t3.medium
Backtesting Service	4 cores	8GB	1:2	c6i.xlarge (compute-optimized)
ML Training Service	8 cores	32GB + GPU	1:4	g4dn.2xlarge (GPU)

How to Right-Size:

1. Monitor Actual Usage: Use Prometheus metrics for 2 weeks
2. Identify Bottlenecks: CPU-bound vs memory-bound vs I/O-bound
3. Adjust Instance Types: Match instance type to workload
4. Iterate: Re-evaluate every quarter

Spot Instances for Non-Critical Workloads

What are Spot Instances:

• Spare cloud capacity sold at 50-90% discount
• Can be terminated with 2-minute notice
• Only use for fault-tolerant workloads

Recommended Use Cases:

# Kubernetes node groups
nodeGroups:
  # Production workloads (on-demand)
  - name: production-on-demand
    instanceType: t3.medium
    desiredCapacity: 10
    labels:
      workload: production

  # Backtesting workers (spot instances)
  - name: backtesting-spot
    instanceType: c6i.xlarge
    desiredCapacity: 5
    instancesDistribution:
      onDemandBaseCapacity: 1  # Always keep 1 on-demand
      onDemandPercentageAboveBaseCapacity: 0  # Rest are spot
      spotAllocationStrategy: capacity-optimized
    labels:
      workload: backtesting

Never Use Spot For:

• Trading Service (real-time, SLA-critical)
• API Gateway (user-facing)
• Databases (stateful)

Good Use Cases:

• Backtesting workers (jobs can retry)
• ML training (checkpointing allows resume)
• Batch processing

Reserved Instances for Base Load

What are Reserved Instances:

• 1-year or 3-year commitment
• 30-60% discount vs on-demand
• Best for predictable, steady workload

Recommended Strategy:

Total Capacity Needed: 20 instances
├── Base Load (always needed): 10 instances → Reserved (50% savings)
├── Typical Load: +5 instances → On-Demand
└── Peak Load: +5 instances → Auto-Scaling (On-Demand or Spot)

Cost Comparison (AWS t3.medium, us-east-1):

Pricing Model	Cost/month	Savings
On-Demand	$30.37	0%
Reserved (1yr, no upfront)	$18.25	40%
Reserved (3yr, all upfront)	$11.68	62%
Spot (average)	$9.11	70%

Auto-Scaling for Burst Capacity

Cost-Optimized Auto-Scaling:

# HPA with cost awareness
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: trading-service-hpa
spec:
  minReplicas: 10  # Reserved instances (base load)
  maxReplicas: 20  # Auto-scale to 20 (on-demand/spot)
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

  # Slow scale-down to avoid flapping
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600  # 10 minutes
      policies:
      - type: Pods
        value: 1
        periodSeconds: 120  # Remove 1 pod every 2 minutes

Cost Math:

Scenario 1: No Auto-Scaling (always 20 instances)
Cost: 20 × $30.37 = $607.40/month

Scenario 2: Auto-Scaling (10 reserved, 10 auto-scale)
Base: 10 × $18.25 = $182.50 (reserved)
Burst: 10 × $30.37 × 30% = $91.11 (on-demand, 30% of time)
Total: $273.61/month
Savings: $333.79/month (55%)

---

Monitoring & Alerting

Key Metrics to Monitor

Service Health:

# Service availability (should be > 99.9%)
up{job="api-gateway"} == 1

# Request rate
rate(grpc_requests_total[5m])

# Error rate (should be < 0.1%)
rate(grpc_requests_total{status="error"}[5m]) / rate(grpc_requests_total[5m])

# Latency (P99 should be < 5ms)
histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket[5m]))

Auto-Scaling Metrics:

# Current replica count
kube_deployment_status_replicas{deployment="trading-service"}

# Desired replica count (from HPA)
kube_horizontalpodautoscaler_status_desired_replicas{hpa="trading-service-hpa"}

# CPU utilization
avg(rate(container_cpu_usage_seconds_total{pod=~"trading-service.*"}[5m])) * 100

# Memory utilization
avg(container_memory_working_set_bytes{pod=~"trading-service.*"}) /
avg(container_spec_memory_limit_bytes{pod=~"trading-service.*"}) * 100

Database Metrics:

# Connection pool usage (should be < 80%)
pg_stat_database_numbackends / pg_settings_max_connections * 100

# Replication lag (should be < 100ms)
pg_stat_replication_replay_lag_bytes

# Query latency
rate(pg_stat_database_blks_read[5m])

Alerts to Configure

Critical Alerts (PagerDuty):

# Service down
- alert: ServiceDown
  expr: up{job=~".*service"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Service {{ $labels.job }} is down"

# High error rate
- alert: HighErrorRate
  expr: rate(grpc_requests_total{status="error"}[5m]) / rate(grpc_requests_total[5m]) > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High error rate on {{ $labels.service }}: {{ $value | humanizePercentage }}"

# High latency
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(grpc_request_duration_seconds_bucket[5m])) > 0.1
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High P99 latency on {{ $labels.service }}: {{ $value }}s"

Warning Alerts (Slack):

# High CPU usage (approaching auto-scale threshold)
- alert: HighCPU
  expr: avg(rate(container_cpu_usage_seconds_total[5m])) by (pod) > 0.8
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "High CPU on {{ $labels.pod }}: {{ $value | humanizePercentage }}"

# Database connection pool approaching limit
- alert: HighDatabaseConnections
  expr: pg_stat_database_numbackends / pg_settings_max_connections > 0.8
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Database connection pool at {{ $value | humanizePercentage }}"

Dashboards

Recommended Grafana Dashboards:

1. Service Overview: Request rate, error rate, latency (RED metrics)
2. Auto-Scaling: Current/desired replicas, CPU/memory utilization
3. Database: Connection pool, query latency, replication lag
4. Load Balancer: Active connections, requests per backend
5. Cost: Instance count, spot vs on-demand usage

---

Quick Reference

Scaling Decision Matrix

Symptom	Root Cause	Solution
High CPU (> 80%)	Not enough compute	Scale up replicas
High memory (> 90%)	Memory leak or under-provisioned	Increase memory limits or fix leak
High latency + low CPU	Database bottleneck	Add read replicas or optimize queries
High error rate	Service overload	Scale up replicas or rate limit clients
Database connection exhaustion	Too many service instances	Add PgBouncer connection pooling
Redis memory full	Cache size exceeds capacity	Add Redis Cluster or increase eviction

Scaling Commands

Kubernetes:

# Manual scaling
kubectl scale deployment trading-service --replicas=10

# Check HPA status
kubectl get hpa

# Describe HPA (shows scaling events)
kubectl describe hpa trading-service-hpa

# View pod resource usage
kubectl top pods

# View node resource usage
kubectl top nodes

Docker Compose (development):

# Scale service
docker-compose up -d --scale trading_service=5

# View resource usage
docker stats

Load Balancer Health Checks

nginx:

# Check nginx status
curl http://localhost/nginx_status

# Test gRPC endpoint
grpc_health_probe -addr=localhost:443 -tls

HAProxy:

# Check HAProxy stats
curl http://localhost:8404/stats

# Test backend health
echo "show servers state" | socat /var/run/haproxy.sock stdio

---

Additional Resources

• Kubernetes HPA: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
• nginx gRPC Load Balancing: https://www.nginx.com/blog/nginx-1-13-10-grpc/
• PostgreSQL Connection Pooling: https://www.pgbouncer.org/
• Redis Cluster: https://redis.io/docs/manual/scaling/
• Performance Benchmarks: See PERFORMANCE_BENCHMARKS.md
• Scaling Playbook: See SCALING_PLAYBOOK.md

---

Last Updated: 2025-10-07 Version: 1.0 Owner: Platform Engineering Team