foxhunt/SCALING_PLAYBOOK.md

Scaling Playbook - Foxhunt HFT Trading System

Last Updated: 2025-10-07 Status: Production Ready Audience: DevOps, SRE, Platform Engineers

---

Table of Contents

1. Overview
2. When to Scale
3. Monitoring Metrics
4. Scaling Decisions
5. Cost vs Performance Tradeoffs
6. Scaling Procedures
7. Troubleshooting
8. Emergency Response

---

Overview

This playbook provides operational guidance for scaling the Foxhunt HFT Trading System. It covers:

• When to scale (metrics and thresholds)
• How to scale (manual and automated procedures)
• Why to scale (performance, cost, reliability)
• What to monitor (metrics and alerts)

Scaling Philosophy

1. Proactive Scaling: Scale before problems occur (monitor trends)
2. Cost-Aware Scaling: Balance performance with cost
3. Automated First: Use HPA for routine scaling, manual for exceptions
4. Graceful Degradation: System should degrade gracefully under load
5. Measured Response: Base decisions on metrics, not gut feeling

---

When to Scale

Scale-Up Triggers (Add Capacity)

Critical (Immediate Action Required)

Metric	Threshold	Action	Timeline
P99 Latency	> 100ms	Scale up immediately	< 5 minutes
Error Rate	> 1%	Scale up immediately	< 5 minutes
CPU Utilization	> 90%	Scale up immediately	< 5 minutes
Memory Utilization	> 95%	Scale up immediately	< 2 minutes
Service Down	Any instance down	Replace/scale up	< 2 minutes

Example Alert:

CRITICAL: API Gateway P99 Latency at 150ms (threshold: 100ms)
Current replicas: 3, CPU: 92%, Memory: 78%
Action: Scale up to 6 replicas immediately

Warning (Plan Scaling Within 1 Hour)

Metric	Threshold	Action	Timeline
P99 Latency	> 50ms	Plan scale-up	< 1 hour
Error Rate	> 0.5%	Plan scale-up	< 1 hour
CPU Utilization	> 80%	Plan scale-up	< 1 hour
Memory Utilization	> 85%	Plan scale-up	< 1 hour
Request Queue Depth	> 100	Plan scale-up	< 30 minutes

Example Alert:

WARNING: Trading Service CPU at 85% (threshold: 80%)
Current replicas: 5, Request rate: 8K req/s
Action: Review traffic patterns, plan scale-up to 7 replicas

Proactive (Plan Scaling Within 24 Hours)

Metric	Threshold	Action	Timeline
Traffic Growth	> 20% week-over-week	Plan capacity expansion	< 1 week
CPU Utilization Trend	Increasing 10%/week	Plan scale-up	< 1 week
Peak Traffic Approaching	Predicted > 80% capacity	Pre-scale before peak	< 24 hours
Scheduled Events	Known high-traffic events	Pre-scale 1 hour before	< 1 hour before event

Example:

INFO: Traffic growing 25% week-over-week (5K → 6.25K req/s)
Current capacity: 10K req/s (10 replicas)
Projected capacity need (4 weeks): 12.5K req/s
Action: Plan to add 2 replicas (12 total) within 2 weeks

Scale-Down Triggers (Reduce Capacity)

Safe to Scale Down

Metric	Threshold	Action	Timeline
CPU Utilization	< 40% for 30 minutes	Scale down by 25%	< 10 minutes
Memory Utilization	< 50% for 30 minutes	Scale down by 25%	< 10 minutes
Request Rate	< 50% of capacity	Scale down by 25%	< 10 minutes
Off-Peak Hours	Night/weekend (known pattern)	Scale down to min replicas	Scheduled

Example:

INFO: API Gateway CPU at 35% for 45 minutes (threshold: 40% for 30m)
Current replicas: 10, Request rate: 2K req/s (capacity: 20K req/s)
Action: Scale down to 8 replicas (save 20% cost, maintain 10K req/s capacity)

Never Scale Down

• Error rate > 0.1%
• P99 latency > 20ms
• Recent scale-up (< 10 minutes ago)
• During peak traffic hours
• Less than minReplicas (HA requirement)

---

Monitoring Metrics

Service-Level Metrics (RED Metrics)

Request Rate:

# Total request rate across all pods
sum(rate(grpc_requests_total{service="trading-service"}[5m]))

# Per-pod request rate
rate(grpc_requests_total{service="trading-service"}[5m])

Error Rate:

# Error percentage
sum(rate(grpc_requests_total{service="trading-service",status="error"}[5m])) /
sum(rate(grpc_requests_total{service="trading-service"}[5m])) * 100

# Alert if > 0.5%

Latency (Duration):

# P99 latency
histogram_quantile(0.99,
  rate(grpc_request_duration_seconds_bucket{service="trading-service"}[5m]))

# P50, P95, P99.9
histogram_quantile(0.50, ...)
histogram_quantile(0.95, ...)
histogram_quantile(0.999, ...)

Resource Utilization Metrics

CPU:

# Average CPU utilization across pods
avg(rate(container_cpu_usage_seconds_total{pod=~"trading-service.*"}[5m])) * 100

# Per-pod CPU
rate(container_cpu_usage_seconds_total{pod="trading-service-1"}[5m]) * 100

Memory:

# Average memory utilization
avg(container_memory_working_set_bytes{pod=~"trading-service.*"}) /
avg(container_spec_memory_limit_bytes{pod=~"trading-service.*"}) * 100

# Memory usage in MB
container_memory_working_set_bytes{pod="trading-service-1"} / 1024 / 1024

Scaling Metrics

Replica Count:

# Current replicas
kube_deployment_status_replicas{deployment="trading-service"}

# Desired replicas (from HPA)
kube_horizontalpodautoscaler_status_desired_replicas{hpa="trading-service-hpa"}

# Available replicas (healthy)
kube_deployment_status_replicas_available{deployment="trading-service"}

Scaling Events:

# View recent scaling events
kubectl get events -n foxhunt \
  --field-selector involvedObject.kind=HorizontalPodAutoscaler \
  --sort-by='.lastTimestamp'

# Example output:
# 5m ago: ScaledUp from 5 to 7 replicas (CPU: 85%)
# 15m ago: ScaledDown from 10 to 8 replicas (CPU: 35%)

Database Metrics

Connection Pool:

# PostgreSQL active connections
pg_stat_database_numbackends{datname="foxhunt"}

# Connection pool utilization
pg_stat_database_numbackends / pg_settings_max_connections * 100

# Alert if > 80%

Replication Lag:

# Replication lag (should be < 100ms)
pg_stat_replication_replay_lag_bytes

# Convert to milliseconds
pg_stat_replication_replay_lag_seconds * 1000

Query Performance:

# Slow queries (> 100ms)
pg_stat_statements_mean_exec_time_seconds{query=~".*"} > 0.1

# Top 10 slowest queries
topk(10, pg_stat_statements_mean_exec_time_seconds)

---

Scaling Decisions

Decision Matrix

Symptom	Root Cause	Solution	Cost Impact
High CPU (> 80%)	Insufficient compute	Scale up replicas	High (+20% cost per replica)
High Memory (> 90%)	Memory leak or under-provisioned	Fix leak OR increase memory limits	Medium (10-20% cost increase)
High Latency + Low CPU	Database bottleneck	Add read replicas	Medium (+30% DB cost)
High Error Rate	Service overload	Scale up replicas	High (+20% cost per replica)
Connection Exhaustion	Too many service instances	Add PgBouncer	Low (<5% cost increase)
Redis Memory Full	Cache size exceeds capacity	Add Redis Cluster	Medium (+50% Redis cost)
Queue Depth High	Worker shortage	Scale up workers	High (+20% per worker)
Disk I/O High	Storage bottleneck	Upgrade disk (SSD → NVMe)	High (2-3x storage cost)

Decision Tree

Start: Is there a performance issue?
  ├─ Yes: What metric is degraded?
  │   ├─ Latency (P99 > 100ms)
  │   │   ├─ CPU high? → Scale up replicas
  │   │   ├─ Database slow? → Add read replicas or optimize queries
  │   │   └─ Network slow? → Check load balancer, network config
  │   ├─ Error Rate (> 0.5%)
  │   │   ├─ Service overload? → Scale up replicas
  │   │   ├─ Database errors? → Check connection pool, add replicas
  │   │   └─ External API errors? → Implement retry, circuit breaker
  │   └─ Throughput (cannot handle load)
  │       ├─ CPU bottleneck? → Scale up replicas
  │       ├─ Database bottleneck? → Optimize queries, add replicas
  │       └─ Network bottleneck? → Upgrade network, load balancer
  └─ No: Is cost optimization possible?
      ├─ Over-provisioned (CPU < 40%)? → Scale down replicas
      ├─ Off-peak hours? → Scale down to minReplicas
      └─ Right-size instances? → Switch to smaller instance types

Example Scenarios

Scenario 1: Trading Service High CPU

Symptoms:

• CPU: 92% (threshold: 80%)
• P99 Latency: 75ms (threshold: 100ms, target: 5ms)
• Error Rate: 0.3% (threshold: 0.5%)
• Current replicas: 5
• Request rate: 10K orders/sec

Analysis:

• CPU high but latency still acceptable
• Error rate below threshold
• Approaching capacity (10K orders/sec ÷ 5 = 2K orders/sec per pod)

Decision: Scale up proactively

• Target CPU: 70% (with 20% headroom)
• Required replicas: 10K orders/sec ÷ (2K × 0.7) = 7.14 → 8 replicas
• Cost impact: +60% (5 → 8 replicas)

Action:

# Manual scale-up
kubectl scale deployment trading-service --replicas=8 -n foxhunt

# Verify scaling
watch kubectl get pods -n foxhunt -l app=trading-service

Scenario 2: Database Connection Exhaustion

Symptoms:

• Error: "FATAL: remaining connection slots are reserved"
• PostgreSQL connections: 95/100
• Services running: API Gateway (3), Trading (10), Backtesting (5)
• Expected connections: (3 + 10 + 5) × 10 = 180 (exceeds limit!)

Analysis:

• Connection pool not configured
• Each service instance uses 10 connections
• Total connections exceed PostgreSQL limit

Decision: Add PgBouncer connection pooler

• PgBouncer pools 180 connections → 50 PostgreSQL connections
• No need to reduce service instances

Action:

# Deploy PgBouncer
kubectl apply -f pgbouncer-deployment.yaml

# Update service connection strings
DATABASE_URL=postgresql://foxhunt:password@pgbouncer:6432/foxhunt

Scenario 3: Pre-Scaling for Known Event

Symptoms:

• Scheduled event: Market open (9:30 AM ET)
• Historical data: Traffic spikes 5x (2K → 10K req/s)
• Current capacity: 5 replicas × 2K req/s = 10K req/s (at limit!)
• Risk: Latency spike, potential errors

Analysis:

• Need capacity for 10K req/s with headroom
• Target: 15K req/s capacity (50% headroom)
• Required replicas: 15K ÷ 2K = 7.5 → 8 replicas

Decision: Pre-scale 30 minutes before event

• Scale up to 8 replicas at 9:00 AM
• Monitor during event (9:30 AM - 10:00 AM)
• Scale down to 5 replicas at 11:00 AM (if traffic normalizes)

Action:

# Pre-scale at 9:00 AM
kubectl scale deployment trading-service --replicas=8 -n foxhunt

# Monitor during event
watch kubectl top pods -n foxhunt

# Scale down at 11:00 AM (if safe)
kubectl scale deployment trading-service --replicas=5 -n foxhunt

---

Cost vs Performance Tradeoffs

Cost Models

Instance Cost (AWS t3.medium, us-east-1):

Pricing Model	$/hour	$/month	Savings
On-Demand	$0.0416	$30.37	0%
Reserved (1yr)	$0.025	$18.25	40%
Reserved (3yr)	$0.016	$11.68	62%
Spot (avg)	$0.0125	$9.11	70%

Scaling Cost Example (Trading Service):

Base: 5 replicas × $30.37/mo = $151.85/mo (on-demand)
Peak: 10 replicas × $30.37/mo = $303.70/mo (on-demand)

Strategy 1: All On-Demand
Cost: $303.70/mo (worst case)

Strategy 2: Base Reserved + Peak On-Demand
Base: 5 replicas × $18.25/mo = $91.25/mo (reserved)
Peak: 5 replicas × $30.37/mo × 30% uptime = $45.56/mo (on-demand)
Total: $136.81/mo
Savings: $166.89/mo (55% vs all on-demand)

Strategy 3: Base Reserved + Peak Spot
Base: 5 replicas × $18.25/mo = $91.25/mo (reserved)
Peak: 5 replicas × $9.11/mo × 30% uptime = $13.67/mo (spot)
Total: $104.92/mo
Savings: $198.78/mo (65% vs all on-demand)

Performance Targets vs Cost

Target	Replicas	Cost/Month	Latency (P99)	Throughput
Minimum (HA)	3	$91.11	10-20ms	6K req/s
Balanced	5	$151.85	5-10ms	10K req/s
High Performance	10	$303.70	2-5ms	20K req/s
Peak Capacity	20	$607.40	< 2ms	40K req/s

Recommendation: Use "Balanced" (5 replicas) as baseline, scale to 10-20 for peak traffic.

Cost Optimization Strategies

1. Right-Size Instances

Before:

Instance: t3.large (2 vCPU, 8GB RAM)
Utilization: CPU 35%, Memory 40%
Cost: $60.74/mo

After:

Instance: t3.medium (2 vCPU, 4GB RAM)
Utilization: CPU 70%, Memory 80%
Cost: $30.37/mo
Savings: $30.37/mo (50%)

2. Use Reserved Instances for Base Load

Before (all on-demand):

10 instances × $30.37/mo = $303.70/mo

After (5 reserved + 5 on-demand):

5 reserved × $18.25/mo = $91.25/mo
5 on-demand × $30.37/mo × 30% uptime = $45.56/mo
Total: $136.81/mo
Savings: $166.89/mo (55%)

3. Use Spot Instances for Non-Critical Workloads

Backtesting Workers (spot instances):

Before: 10 workers × $60.74/mo (c6i.xlarge on-demand) = $607.40/mo
After: 10 workers × $18.22/mo (c6i.xlarge spot, 70% discount) = $182.20/mo
Savings: $425.20/mo (70%)

Caution: Only use spot for fault-tolerant workloads (backtesting, ML training). Never for trading service!

4. Auto-Scale During Off-Peak Hours

Peak Hours (9:00 AM - 4:00 PM ET, weekdays):

Replicas: 10 (high traffic)
Cost: 10 × $30.37/mo × 35% time = $106.30/mo

Off-Peak Hours (nights, weekends):

Replicas: 3 (minimal traffic)
Cost: 3 × $30.37/mo × 65% time = $59.22/mo

Total Cost: $165.52/mo (vs $303.70/mo always 10 replicas) Savings: $138.18/mo (45%)

---

Scaling Procedures

Manual Scaling (Kubernetes)

Scale Up

# 1. Check current replica count
kubectl get deployment trading-service -n foxhunt

# 2. Scale up
kubectl scale deployment trading-service --replicas=10 -n foxhunt

# 3. Verify new pods are running
kubectl get pods -n foxhunt -l app=trading-service

# 4. Wait for pods to be ready (STATUS: Running, READY: 1/1)
kubectl wait --for=condition=ready pod -l app=trading-service -n foxhunt --timeout=5m

# 5. Verify load is distributed
kubectl top pods -n foxhunt -l app=trading-service

Scale Down

# 1. Verify load is low (CPU < 50%, latency normal)
kubectl top pods -n foxhunt -l app=trading-service

# 2. Drain pods gracefully (one at a time)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# 3. Scale down
kubectl scale deployment trading-service --replicas=5 -n foxhunt

# 4. Verify no errors or latency spikes
# (Monitor Prometheus/Grafana for 5 minutes)

Automated Scaling (HPA)

Enable HPA

# 1. Apply HPA manifest
kubectl apply -f config/k8s/hpa.yaml

# 2. Verify HPA is active
kubectl get hpa -n foxhunt

# 3. Check HPA status
kubectl describe hpa trading-service-hpa -n foxhunt

# Expected output:
# Current replicas: 5
# Desired replicas: 5
# Metrics: CPU 65% (target: 70%)

Modify HPA Thresholds

# 1. Edit HPA
kubectl edit hpa trading-service-hpa -n foxhunt

# 2. Change CPU threshold (e.g., 70% → 60%)
# metrics:
# - type: Resource
#   resource:
#     name: cpu
#     target:
#       type: Utilization
#       averageUtilization: 60  # Changed from 70

# 3. Save and exit (HPA updates immediately)

# 4. Verify change
kubectl describe hpa trading-service-hpa -n foxhunt

Disable HPA (Emergency)

# 1. Delete HPA (keeps current replica count)
kubectl delete hpa trading-service-hpa -n foxhunt

# 2. Verify HPA is gone
kubectl get hpa -n foxhunt

# 3. Deployment is now manually managed
kubectl scale deployment trading-service --replicas=<desired> -n foxhunt

Database Scaling

Add Read Replica

# 1. Create read replica (PostgreSQL)
# (Cloud provider specific: AWS RDS, GCP Cloud SQL, etc.)

# Example (Kubernetes StatefulSet):
kubectl scale statefulset postgres-replica --replicas=2 -n foxhunt

# 2. Update application to use read replica for read queries
# (See "Application-Level Read/Write Splitting" in LOAD_BALANCING_SCALING.md)

# 3. Verify replication lag
psql -h postgres-replica-1 -c "SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn();"

Scale Redis Cluster

# 1. Add Redis node
redis-cli --cluster add-node new-node:6379 existing-node:6379

# 2. Rebalance cluster (distribute keys)
redis-cli --cluster rebalance existing-node:6379

# 3. Verify cluster status
redis-cli --cluster check existing-node:6379

---

Troubleshooting

Issue: HPA Not Scaling Up

Symptoms:

• CPU > 80% but HPA not scaling
• Desired replicas = Current replicas (no change)

Possible Causes:

1. Stabilization window: HPA waiting before scaling
2. Resource quota exceeded: Namespace/cluster out of resources
3. Node capacity: No nodes available for new pods

Diagnosis:

# 1. Check HPA status
kubectl describe hpa trading-service-hpa -n foxhunt
# Look for: "ScalingLimited" or "BackoffBoth" events

# 2. Check resource quotas
kubectl describe resourcequota -n foxhunt

# 3. Check node capacity
kubectl describe nodes | grep -A 5 "Allocated resources"

Solutions:

• Wait for stabilization window to pass (30-60s)
• Increase resource quotas (if insufficient)
• Add nodes to cluster (if no capacity)

Issue: Pods Evicted After Scale-Up

Symptoms:

• New pods created, then immediately evicted
• Pod status: "Evicted"

Possible Causes:

1. Resource pressure: Node out of memory/disk
2. Resource limits too high: Pod requests exceed node capacity

Diagnosis:

# 1. Check pod status
kubectl describe pod <pod-name> -n foxhunt
# Look for: "Reason: Evicted"

# 2. Check node resources
kubectl top nodes

# 3. Check pod resource requests
kubectl get pod <pod-name> -n foxhunt -o yaml | grep -A 10 resources

Solutions:

• Reduce resource requests/limits (if too high)
• Add nodes with more resources
• Delete unused pods to free resources

Issue: High Latency Despite Scaling Up

Symptoms:

• Replicas increased (5 → 10) but P99 latency still high (> 100ms)
• CPU/memory usage normal

Possible Causes:

1. Database bottleneck: Queries slow, connection pool exhausted
2. Network bottleneck: Load balancer misconfigured
3. External API bottleneck: Dependency slow

Diagnosis:

# 1. Check database query performance
psql -h postgres -c "SELECT query, mean_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

# 2. Check database connection pool
psql -h postgres -c "SELECT count(*) FROM pg_stat_activity;"

# 3. Check external API latency
# (Prometheus query)
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="external-api"}[5m]))

Solutions:

• Optimize slow database queries (add indexes, rewrite)
• Add database read replicas
• Add PgBouncer connection pooler
• Implement caching for external API calls

---

Emergency Response

Incident: Service Completely Down

Severity: P0 (Critical) Response Time: Immediate

Symptoms:

• All pods in CrashLoopBackOff
• Service unreachable (health check failing)
• Error rate: 100%

Immediate Actions (within 5 minutes):

1. Check pod status:

   kubectl get pods -n foxhunt -l app=trading-service
   kubectl logs <pod-name> -n foxhunt --tail=100
   

2. Rollback to last working version:

   kubectl rollout undo deployment trading-service -n foxhunt
   kubectl rollout status deployment trading-service -n foxhunt
   

3. Scale up to compensate:

   kubectl scale deployment trading-service --replicas=10 -n foxhunt
   

4. Notify stakeholders:

   • Alert: "Trading Service down, rollback in progress"
   • ETA: 5 minutes to restore service

Root Cause Analysis (after service restored):

• Check deployment changes (git log, CI/CD)
• Check resource limits (CPU, memory, disk)
• Check external dependencies (database, Redis, external APIs)

Incident: Database Connection Exhaustion

Severity: P1 (High) Response Time: < 15 minutes

Symptoms:

• Errors: "FATAL: remaining connection slots are reserved"
• Services unable to connect to database
• Error rate: 50-100%

Immediate Actions:

1. Kill idle connections:

   SELECT pg_terminate_backend(pid)
   FROM pg_stat_activity
   WHERE state = 'idle' AND state_change < now() - interval '5 minutes';
   

2. Scale down services temporarily:

   # Reduce replicas to free connections
   kubectl scale deployment trading-service --replicas=3 -n foxhunt
   kubectl scale deployment backtesting-service --replicas=1 -n foxhunt
   

3. Deploy PgBouncer (if not already):

   kubectl apply -f pgbouncer-deployment.yaml
   

4. Update connection strings (rolling update):

   kubectl set env deployment/trading-service DATABASE_URL=postgresql://foxhunt:password@pgbouncer:6432/foxhunt
   

Incident: Traffic Spike (DDoS or Viral Event)

Severity: P1 (High) Response Time: < 10 minutes

Symptoms:

• Request rate 10x normal (100K req/s vs 10K req/s)
• P99 latency > 1s
• Error rate > 5%

Immediate Actions:

1. Enable aggressive rate limiting:

   # nginx config
   limit_req_zone $binary_remote_addr zone=ddos:10m rate=10r/s;
   limit_req zone=ddos burst=20 nodelay;
   

2. Scale up to max capacity:

   kubectl scale deployment api-gateway --replicas=20 -n foxhunt
   kubectl scale deployment trading-service --replicas=20 -n foxhunt
   

3. Enable DDoS protection (if available):

   • Cloudflare: Enable "I'm Under Attack" mode
   • AWS: Enable AWS Shield Advanced
   • Manual: Block suspicious IPs

4. Shed non-critical traffic:

   # Reject non-authenticated requests
   location /api {
       if ($http_authorization = "") {
           return 503 "Service temporarily unavailable";
       }
   }
   

Post-Incident:

• Analyze traffic patterns (legitimate vs attack)
• Implement permanent DDoS protection
• Review capacity planning

---

Quick Reference

Common Commands

# View current replica count
kubectl get deployment -n foxhunt

# Scale deployment
kubectl scale deployment <name> --replicas=<count> -n foxhunt

# View HPA status
kubectl get hpa -n foxhunt
kubectl describe hpa <name> -n foxhunt

# View pod resource usage
kubectl top pods -n foxhunt

# View scaling events
kubectl get events -n foxhunt --field-selector involvedObject.kind=HorizontalPodAutoscaler

# View pod logs
kubectl logs <pod-name> -n foxhunt --tail=100

# Rollback deployment
kubectl rollout undo deployment <name> -n foxhunt

Metrics Cheat Sheet

Metric	Good	Warning	Critical
CPU	< 70%	70-85%	> 85%
Memory	< 75%	75-90%	> 90%
P99 Latency	< 10ms	10-50ms	> 50ms
Error Rate	< 0.1%	0.1-0.5%	> 0.5%
Replica Count	>= minReplicas	< minReplicas	0

---

Additional Resources

• Load Balancing: See LOAD_BALANCING_SCALING.md
• Performance Benchmarks: See PERFORMANCE_BENCHMARKS.md
• Monitoring Dashboards: Grafana (http://localhost:3000)
• Prometheus Alerts: Alertmanager (http://localhost:9093)

---

Last Updated: 2025-10-07 Version: 1.0 Owner: Platform Engineering Team