foxhunt/SCALING_PLAYBOOK.md

# Scaling Playbook - Foxhunt HFT Trading System

**Last Updated**: 2025-10-07
**Status**: Production Ready
**Audience**: DevOps, SRE, Platform Engineers

---

## Table of Contents

1. [Overview](#overview)
2. [When to Scale](#when-to-scale)
3. [Monitoring Metrics](#monitoring-metrics)
4. [Scaling Decisions](#scaling-decisions)
5. [Cost vs Performance Tradeoffs](#cost-vs-performance-tradeoffs)
6. [Scaling Procedures](#scaling-procedures)
7. [Troubleshooting](#troubleshooting)
8. [Emergency Response](#emergency-response)

---

## Overview

This playbook provides operational guidance for scaling the Foxhunt HFT Trading System. It covers:
- **When** to scale (metrics and thresholds)
- **How** to scale (manual and automated procedures)
- **Why** to scale (performance, cost, reliability)
- **What** to monitor (metrics and alerts)

### Scaling Philosophy

1. **Proactive Scaling**: Scale before problems occur (monitor trends)
2. **Cost-Aware Scaling**: Balance performance with cost
3. **Automated First**: Use HPA for routine scaling, manual for exceptions
4. **Graceful Degradation**: System should degrade gracefully under load
5. **Measured Response**: Base decisions on metrics, not gut feeling

---

## When to Scale

### Scale-Up Triggers (Add Capacity)

#### Critical (Immediate Action Required)

| Metric | Threshold | Action | Timeline |
|--------|-----------|--------|----------|
| P99 Latency | > 100ms | Scale up immediately | < 5 minutes |
| Error Rate | > 1% | Scale up immediately | < 5 minutes |
| CPU Utilization | > 90% | Scale up immediately | < 5 minutes |
| Memory Utilization | > 95% | Scale up immediately | < 2 minutes |
| Service Down | Any instance down | Replace/scale up | < 2 minutes |

**Example Alert**:
```
CRITICAL: API Gateway P99 Latency at 150ms (threshold: 100ms)
Current replicas: 3, CPU: 92%, Memory: 78%
Action: Scale up to 6 replicas immediately
```

#### Warning (Plan Scaling Within 1 Hour)

| Metric | Threshold | Action | Timeline |
|--------|-----------|--------|----------|
| P99 Latency | > 50ms | Plan scale-up | < 1 hour |
| Error Rate | > 0.5% | Plan scale-up | < 1 hour |
| CPU Utilization | > 80% | Plan scale-up | < 1 hour |
| Memory Utilization | > 85% | Plan scale-up | < 1 hour |
| Request Queue Depth | > 100 | Plan scale-up | < 30 minutes |

**Example Alert**:
```
WARNING: Trading Service CPU at 85% (threshold: 80%)
Current replicas: 5, Request rate: 8K req/s
Action: Review traffic patterns, plan scale-up to 7 replicas
```

#### Proactive (Plan Scaling Within 24 Hours)

| Metric | Threshold | Action | Timeline |
|--------|-----------|--------|----------|
| Traffic Growth | > 20% week-over-week | Plan capacity expansion | < 1 week |
| CPU Utilization Trend | Increasing 10%/week | Plan scale-up | < 1 week |
| Peak Traffic Approaching | Predicted > 80% capacity | Pre-scale before peak | < 24 hours |
| Scheduled Events | Known high-traffic events | Pre-scale 1 hour before | < 1 hour before event |

**Example**:
```
INFO: Traffic growing 25% week-over-week (5K → 6.25K req/s)
Current capacity: 10K req/s (10 replicas)
Projected capacity need (4 weeks): 12.5K req/s
Action: Plan to add 2 replicas (12 total) within 2 weeks
```

### Scale-Down Triggers (Reduce Capacity)

#### Safe to Scale Down

| Metric | Threshold | Action | Timeline |
|--------|-----------|--------|----------|
| CPU Utilization | < 40% for 30 minutes | Scale down by 25% | < 10 minutes |
| Memory Utilization | < 50% for 30 minutes | Scale down by 25% | < 10 minutes |
| Request Rate | < 50% of capacity | Scale down by 25% | < 10 minutes |
| Off-Peak Hours | Night/weekend (known pattern) | Scale down to min replicas | Scheduled |

**Example**:
```
INFO: API Gateway CPU at 35% for 45 minutes (threshold: 40% for 30m)
Current replicas: 10, Request rate: 2K req/s (capacity: 20K req/s)
Action: Scale down to 8 replicas (save 20% cost, maintain 10K req/s capacity)
```

#### Never Scale Down

- Error rate > 0.1%
- P99 latency > 20ms
- Recent scale-up (< 10 minutes ago)
- During peak traffic hours
- Less than minReplicas (HA requirement)

---

## Monitoring Metrics

### Service-Level Metrics (RED Metrics)

**Request Rate**:
```promql
# Total request rate across all pods
sum(rate(grpc_requests_total{service="trading-service"}[5m]))

# Per-pod request rate
rate(grpc_requests_total{service="trading-service"}[5m])
```

**Error Rate**:
```promql
# Error percentage
sum(rate(grpc_requests_total{service="trading-service",status="error"}[5m])) /
sum(rate(grpc_requests_total{service="trading-service"}[5m])) * 100

# Alert if > 0.5%
```

**Latency (Duration)**:
```promql
# P99 latency
histogram_quantile(0.99,
  rate(grpc_request_duration_seconds_bucket{service="trading-service"}[5m]))

# P50, P95, P99.9
histogram_quantile(0.50, ...)
histogram_quantile(0.95, ...)
histogram_quantile(0.999, ...)
```

### Resource Utilization Metrics

**CPU**:
```promql
# Average CPU utilization across pods
avg(rate(container_cpu_usage_seconds_total{pod=~"trading-service.*"}[5m])) * 100

# Per-pod CPU
rate(container_cpu_usage_seconds_total{pod="trading-service-1"}[5m]) * 100
```

**Memory**:
```promql
# Average memory utilization
avg(container_memory_working_set_bytes{pod=~"trading-service.*"}) /
avg(container_spec_memory_limit_bytes{pod=~"trading-service.*"}) * 100

# Memory usage in MB
container_memory_working_set_bytes{pod="trading-service-1"} / 1024 / 1024
```

### Scaling Metrics

**Replica Count**:
```promql
# Current replicas
kube_deployment_status_replicas{deployment="trading-service"}

# Desired replicas (from HPA)
kube_horizontalpodautoscaler_status_desired_replicas{hpa="trading-service-hpa"}

# Available replicas (healthy)
kube_deployment_status_replicas_available{deployment="trading-service"}
```

**Scaling Events**:
```bash
# View recent scaling events
kubectl get events -n foxhunt \
  --field-selector involvedObject.kind=HorizontalPodAutoscaler \
  --sort-by='.lastTimestamp'

# Example output:
# 5m ago: ScaledUp from 5 to 7 replicas (CPU: 85%)
# 15m ago: ScaledDown from 10 to 8 replicas (CPU: 35%)
```

### Database Metrics

**Connection Pool**:
```promql
# PostgreSQL active connections
pg_stat_database_numbackends{datname="foxhunt"}

# Connection pool utilization
pg_stat_database_numbackends / pg_settings_max_connections * 100

# Alert if > 80%
```

**Replication Lag**:
```promql
# Replication lag (should be < 100ms)
pg_stat_replication_replay_lag_bytes

# Convert to milliseconds
pg_stat_replication_replay_lag_seconds * 1000
```

**Query Performance**:
```promql
# Slow queries (> 100ms)
pg_stat_statements_mean_exec_time_seconds{query=~".*"} > 0.1

# Top 10 slowest queries
topk(10, pg_stat_statements_mean_exec_time_seconds)
```

---

## Scaling Decisions

### Decision Matrix

| Symptom | Root Cause | Solution | Cost Impact |
|---------|-----------|----------|-------------|
| **High CPU (> 80%)** | Insufficient compute | Scale up replicas | **High** (+20% cost per replica) |
| **High Memory (> 90%)** | Memory leak or under-provisioned | Fix leak OR increase memory limits | **Medium** (10-20% cost increase) |
| **High Latency + Low CPU** | Database bottleneck | Add read replicas | **Medium** (+30% DB cost) |
| **High Error Rate** | Service overload | Scale up replicas | **High** (+20% cost per replica) |
| **Connection Exhaustion** | Too many service instances | Add PgBouncer | **Low** (<5% cost increase) |
| **Redis Memory Full** | Cache size exceeds capacity | Add Redis Cluster | **Medium** (+50% Redis cost) |
| **Queue Depth High** | Worker shortage | Scale up workers | **High** (+20% per worker) |
| **Disk I/O High** | Storage bottleneck | Upgrade disk (SSD → NVMe) | **High** (2-3x storage cost) |

### Decision Tree

```
Start: Is there a performance issue?
  ├─ Yes: What metric is degraded?
  │   ├─ Latency (P99 > 100ms)
  │   │   ├─ CPU high? → Scale up replicas
  │   │   ├─ Database slow? → Add read replicas or optimize queries
  │   │   └─ Network slow? → Check load balancer, network config
  │   ├─ Error Rate (> 0.5%)
  │   │   ├─ Service overload? → Scale up replicas
  │   │   ├─ Database errors? → Check connection pool, add replicas
  │   │   └─ External API errors? → Implement retry, circuit breaker
  │   └─ Throughput (cannot handle load)
  │       ├─ CPU bottleneck? → Scale up replicas
  │       ├─ Database bottleneck? → Optimize queries, add replicas
  │       └─ Network bottleneck? → Upgrade network, load balancer
  └─ No: Is cost optimization possible?
      ├─ Over-provisioned (CPU < 40%)? → Scale down replicas
      ├─ Off-peak hours? → Scale down to minReplicas
      └─ Right-size instances? → Switch to smaller instance types
```

### Example Scenarios

#### Scenario 1: Trading Service High CPU

**Symptoms**:
- CPU: 92% (threshold: 80%)
- P99 Latency: 75ms (threshold: 100ms, target: 5ms)
- Error Rate: 0.3% (threshold: 0.5%)
- Current replicas: 5
- Request rate: 10K orders/sec

**Analysis**:
- CPU high but latency still acceptable
- Error rate below threshold
- Approaching capacity (10K orders/sec ÷ 5 = 2K orders/sec per pod)

**Decision**: Scale up proactively
- Target CPU: 70% (with 20% headroom)
- Required replicas: 10K orders/sec ÷ (2K × 0.7) = 7.14 → 8 replicas
- Cost impact: +60% (5 → 8 replicas)

**Action**:
```bash
# Manual scale-up
kubectl scale deployment trading-service --replicas=8 -n foxhunt

# Verify scaling
watch kubectl get pods -n foxhunt -l app=trading-service
```

#### Scenario 2: Database Connection Exhaustion

**Symptoms**:
- Error: "FATAL: remaining connection slots are reserved"
- PostgreSQL connections: 95/100
- Services running: API Gateway (3), Trading (10), Backtesting (5)
- Expected connections: (3 + 10 + 5) × 10 = 180 (exceeds limit!)

**Analysis**:
- Connection pool not configured
- Each service instance uses 10 connections
- Total connections exceed PostgreSQL limit

**Decision**: Add PgBouncer connection pooler
- PgBouncer pools 180 connections → 50 PostgreSQL connections
- No need to reduce service instances

**Action**:
```bash
# Deploy PgBouncer
kubectl apply -f pgbouncer-deployment.yaml

# Update service connection strings
DATABASE_URL=postgresql://foxhunt:password@pgbouncer:6432/foxhunt
```

#### Scenario 3: Pre-Scaling for Known Event

**Symptoms**:
- Scheduled event: Market open (9:30 AM ET)
- Historical data: Traffic spikes 5x (2K → 10K req/s)
- Current capacity: 5 replicas × 2K req/s = 10K req/s (at limit!)
- Risk: Latency spike, potential errors

**Analysis**:
- Need capacity for 10K req/s with headroom
- Target: 15K req/s capacity (50% headroom)
- Required replicas: 15K ÷ 2K = 7.5 → 8 replicas

**Decision**: Pre-scale 30 minutes before event
- Scale up to 8 replicas at 9:00 AM
- Monitor during event (9:30 AM - 10:00 AM)
- Scale down to 5 replicas at 11:00 AM (if traffic normalizes)

**Action**:
```bash
# Pre-scale at 9:00 AM
kubectl scale deployment trading-service --replicas=8 -n foxhunt

# Monitor during event
watch kubectl top pods -n foxhunt

# Scale down at 11:00 AM (if safe)
kubectl scale deployment trading-service --replicas=5 -n foxhunt
```

---

## Cost vs Performance Tradeoffs

### Cost Models

**Instance Cost** (AWS t3.medium, us-east-1):
| Pricing Model | $/hour | $/month | Savings |
|---------------|--------|---------|---------|
| On-Demand | $0.0416 | $30.37 | 0% |
| Reserved (1yr) | $0.025 | $18.25 | 40% |
| Reserved (3yr) | $0.016 | $11.68 | 62% |
| Spot (avg) | $0.0125 | $9.11 | 70% |

**Scaling Cost Example** (Trading Service):
```
Base: 5 replicas × $30.37/mo = $151.85/mo (on-demand)
Peak: 10 replicas × $30.37/mo = $303.70/mo (on-demand)

Strategy 1: All On-Demand
Cost: $303.70/mo (worst case)

Strategy 2: Base Reserved + Peak On-Demand
Base: 5 replicas × $18.25/mo = $91.25/mo (reserved)
Peak: 5 replicas × $30.37/mo × 30% uptime = $45.56/mo (on-demand)
Total: $136.81/mo
Savings: $166.89/mo (55% vs all on-demand)

Strategy 3: Base Reserved + Peak Spot
Base: 5 replicas × $18.25/mo = $91.25/mo (reserved)
Peak: 5 replicas × $9.11/mo × 30% uptime = $13.67/mo (spot)
Total: $104.92/mo
Savings: $198.78/mo (65% vs all on-demand)
```

### Performance Targets vs Cost

| Target | Replicas | Cost/Month | Latency (P99) | Throughput |
|--------|----------|------------|---------------|------------|
| Minimum (HA) | 3 | $91.11 | 10-20ms | 6K req/s |
| Balanced | 5 | $151.85 | 5-10ms | 10K req/s |
| High Performance | 10 | $303.70 | 2-5ms | 20K req/s |
| Peak Capacity | 20 | $607.40 | < 2ms | 40K req/s |

**Recommendation**: Use "Balanced" (5 replicas) as baseline, scale to 10-20 for peak traffic.

### Cost Optimization Strategies

#### 1. Right-Size Instances

**Before**:
```
Instance: t3.large (2 vCPU, 8GB RAM)
Utilization: CPU 35%, Memory 40%
Cost: $60.74/mo
```

**After**:
```
Instance: t3.medium (2 vCPU, 4GB RAM)
Utilization: CPU 70%, Memory 80%
Cost: $30.37/mo
Savings: $30.37/mo (50%)
```

#### 2. Use Reserved Instances for Base Load

**Before** (all on-demand):
```
10 instances × $30.37/mo = $303.70/mo
```

**After** (5 reserved + 5 on-demand):
```
5 reserved × $18.25/mo = $91.25/mo
5 on-demand × $30.37/mo × 30% uptime = $45.56/mo
Total: $136.81/mo
Savings: $166.89/mo (55%)
```

#### 3. Use Spot Instances for Non-Critical Workloads

**Backtesting Workers** (spot instances):
```
Before: 10 workers × $60.74/mo (c6i.xlarge on-demand) = $607.40/mo
After: 10 workers × $18.22/mo (c6i.xlarge spot, 70% discount) = $182.20/mo
Savings: $425.20/mo (70%)
```

**Caution**: Only use spot for fault-tolerant workloads (backtesting, ML training). Never for trading service!

#### 4. Auto-Scale During Off-Peak Hours

**Peak Hours** (9:00 AM - 4:00 PM ET, weekdays):
```
Replicas: 10 (high traffic)
Cost: 10 × $30.37/mo × 35% time = $106.30/mo
```

**Off-Peak Hours** (nights, weekends):
```
Replicas: 3 (minimal traffic)
Cost: 3 × $30.37/mo × 65% time = $59.22/mo
```

**Total Cost**: $165.52/mo (vs $303.70/mo always 10 replicas)
**Savings**: $138.18/mo (45%)

---

## Scaling Procedures

### Manual Scaling (Kubernetes)

#### Scale Up

```bash
# 1. Check current replica count
kubectl get deployment trading-service -n foxhunt

# 2. Scale up
kubectl scale deployment trading-service --replicas=10 -n foxhunt

# 3. Verify new pods are running
kubectl get pods -n foxhunt -l app=trading-service

# 4. Wait for pods to be ready (STATUS: Running, READY: 1/1)
kubectl wait --for=condition=ready pod -l app=trading-service -n foxhunt --timeout=5m

# 5. Verify load is distributed
kubectl top pods -n foxhunt -l app=trading-service
```

#### Scale Down

```bash
# 1. Verify load is low (CPU < 50%, latency normal)
kubectl top pods -n foxhunt -l app=trading-service

# 2. Drain pods gracefully (one at a time)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# 3. Scale down
kubectl scale deployment trading-service --replicas=5 -n foxhunt

# 4. Verify no errors or latency spikes
# (Monitor Prometheus/Grafana for 5 minutes)
```

### Automated Scaling (HPA)

#### Enable HPA

```bash
# 1. Apply HPA manifest
kubectl apply -f config/k8s/hpa.yaml

# 2. Verify HPA is active
kubectl get hpa -n foxhunt

# 3. Check HPA status
kubectl describe hpa trading-service-hpa -n foxhunt

# Expected output:
# Current replicas: 5
# Desired replicas: 5
# Metrics: CPU 65% (target: 70%)
```

#### Modify HPA Thresholds

```bash
# 1. Edit HPA
kubectl edit hpa trading-service-hpa -n foxhunt

# 2. Change CPU threshold (e.g., 70% → 60%)
# metrics:
# - type: Resource
#   resource:
#     name: cpu
#     target:
#       type: Utilization
#       averageUtilization: 60  # Changed from 70

# 3. Save and exit (HPA updates immediately)

# 4. Verify change
kubectl describe hpa trading-service-hpa -n foxhunt
```

#### Disable HPA (Emergency)

```bash
# 1. Delete HPA (keeps current replica count)
kubectl delete hpa trading-service-hpa -n foxhunt

# 2. Verify HPA is gone
kubectl get hpa -n foxhunt

# 3. Deployment is now manually managed
kubectl scale deployment trading-service --replicas=<desired> -n foxhunt
```

### Database Scaling

#### Add Read Replica

```bash
# 1. Create read replica (PostgreSQL)
# (Cloud provider specific: AWS RDS, GCP Cloud SQL, etc.)

# Example (Kubernetes StatefulSet):
kubectl scale statefulset postgres-replica --replicas=2 -n foxhunt

# 2. Update application to use read replica for read queries
# (See "Application-Level Read/Write Splitting" in LOAD_BALANCING_SCALING.md)

# 3. Verify replication lag
psql -h postgres-replica-1 -c "SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn();"
```

#### Scale Redis Cluster

```bash
# 1. Add Redis node
redis-cli --cluster add-node new-node:6379 existing-node:6379

# 2. Rebalance cluster (distribute keys)
redis-cli --cluster rebalance existing-node:6379

# 3. Verify cluster status
redis-cli --cluster check existing-node:6379
```

---

## Troubleshooting

### Issue: HPA Not Scaling Up

**Symptoms**:
- CPU > 80% but HPA not scaling
- Desired replicas = Current replicas (no change)

**Possible Causes**:
1. **Stabilization window**: HPA waiting before scaling
2. **Resource quota exceeded**: Namespace/cluster out of resources
3. **Node capacity**: No nodes available for new pods

**Diagnosis**:
```bash
# 1. Check HPA status
kubectl describe hpa trading-service-hpa -n foxhunt
# Look for: "ScalingLimited" or "BackoffBoth" events

# 2. Check resource quotas
kubectl describe resourcequota -n foxhunt

# 3. Check node capacity
kubectl describe nodes | grep -A 5 "Allocated resources"
```

**Solutions**:
- Wait for stabilization window to pass (30-60s)
- Increase resource quotas (if insufficient)
- Add nodes to cluster (if no capacity)

### Issue: Pods Evicted After Scale-Up

**Symptoms**:
- New pods created, then immediately evicted
- Pod status: "Evicted"

**Possible Causes**:
1. **Resource pressure**: Node out of memory/disk
2. **Resource limits too high**: Pod requests exceed node capacity

**Diagnosis**:
```bash
# 1. Check pod status
kubectl describe pod <pod-name> -n foxhunt
# Look for: "Reason: Evicted"

# 2. Check node resources
kubectl top nodes

# 3. Check pod resource requests
kubectl get pod <pod-name> -n foxhunt -o yaml | grep -A 10 resources
```

**Solutions**:
- Reduce resource requests/limits (if too high)
- Add nodes with more resources
- Delete unused pods to free resources

### Issue: High Latency Despite Scaling Up

**Symptoms**:
- Replicas increased (5 → 10) but P99 latency still high (> 100ms)
- CPU/memory usage normal

**Possible Causes**:
1. **Database bottleneck**: Queries slow, connection pool exhausted
2. **Network bottleneck**: Load balancer misconfigured
3. **External API bottleneck**: Dependency slow

**Diagnosis**:
```bash
# 1. Check database query performance
psql -h postgres -c "SELECT query, mean_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

# 2. Check database connection pool
psql -h postgres -c "SELECT count(*) FROM pg_stat_activity;"

# 3. Check external API latency
# (Prometheus query)
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="external-api"}[5m]))
```

**Solutions**:
- Optimize slow database queries (add indexes, rewrite)
- Add database read replicas
- Add PgBouncer connection pooler
- Implement caching for external API calls

---

## Emergency Response

### Incident: Service Completely Down

**Severity**: P0 (Critical)
**Response Time**: Immediate

**Symptoms**:
- All pods in CrashLoopBackOff
- Service unreachable (health check failing)
- Error rate: 100%

**Immediate Actions** (within 5 minutes):

1. **Check pod status**:
   ```bash
   kubectl get pods -n foxhunt -l app=trading-service
   kubectl logs <pod-name> -n foxhunt --tail=100
   ```

2. **Rollback to last working version**:
   ```bash
   kubectl rollout undo deployment trading-service -n foxhunt
   kubectl rollout status deployment trading-service -n foxhunt
   ```

3. **Scale up to compensate**:
   ```bash
   kubectl scale deployment trading-service --replicas=10 -n foxhunt
   ```

4. **Notify stakeholders**:
   - Alert: "Trading Service down, rollback in progress"
   - ETA: 5 minutes to restore service

**Root Cause Analysis** (after service restored):
- Check deployment changes (git log, CI/CD)
- Check resource limits (CPU, memory, disk)
- Check external dependencies (database, Redis, external APIs)

### Incident: Database Connection Exhaustion

**Severity**: P1 (High)
**Response Time**: < 15 minutes

**Symptoms**:
- Errors: "FATAL: remaining connection slots are reserved"
- Services unable to connect to database
- Error rate: 50-100%

**Immediate Actions**:

1. **Kill idle connections**:
   ```sql
   SELECT pg_terminate_backend(pid)
   FROM pg_stat_activity
   WHERE state = 'idle' AND state_change < now() - interval '5 minutes';
   ```

2. **Scale down services temporarily**:
   ```bash
   # Reduce replicas to free connections
   kubectl scale deployment trading-service --replicas=3 -n foxhunt
   kubectl scale deployment backtesting-service --replicas=1 -n foxhunt
   ```

3. **Deploy PgBouncer** (if not already):
   ```bash
   kubectl apply -f pgbouncer-deployment.yaml
   ```

4. **Update connection strings** (rolling update):
   ```bash
   kubectl set env deployment/trading-service DATABASE_URL=postgresql://foxhunt:password@pgbouncer:6432/foxhunt
   ```

### Incident: Traffic Spike (DDoS or Viral Event)

**Severity**: P1 (High)
**Response Time**: < 10 minutes

**Symptoms**:
- Request rate 10x normal (100K req/s vs 10K req/s)
- P99 latency > 1s
- Error rate > 5%

**Immediate Actions**:

1. **Enable aggressive rate limiting**:
   ```nginx
   # nginx config
   limit_req_zone $binary_remote_addr zone=ddos:10m rate=10r/s;
   limit_req zone=ddos burst=20 nodelay;
   ```

2. **Scale up to max capacity**:
   ```bash
   kubectl scale deployment api-gateway --replicas=20 -n foxhunt
   kubectl scale deployment trading-service --replicas=20 -n foxhunt
   ```

3. **Enable DDoS protection** (if available):
   - Cloudflare: Enable "I'm Under Attack" mode
   - AWS: Enable AWS Shield Advanced
   - Manual: Block suspicious IPs

4. **Shed non-critical traffic**:
   ```nginx
   # Reject non-authenticated requests
   location /api {
       if ($http_authorization = "") {
           return 503 "Service temporarily unavailable";
       }
   }
   ```

**Post-Incident**:
- Analyze traffic patterns (legitimate vs attack)
- Implement permanent DDoS protection
- Review capacity planning

---

## Quick Reference

### Common Commands

```bash
# View current replica count
kubectl get deployment -n foxhunt

# Scale deployment
kubectl scale deployment <name> --replicas=<count> -n foxhunt

# View HPA status
kubectl get hpa -n foxhunt
kubectl describe hpa <name> -n foxhunt

# View pod resource usage
kubectl top pods -n foxhunt

# View scaling events
kubectl get events -n foxhunt --field-selector involvedObject.kind=HorizontalPodAutoscaler

# View pod logs
kubectl logs <pod-name> -n foxhunt --tail=100

# Rollback deployment
kubectl rollout undo deployment <name> -n foxhunt
```

### Metrics Cheat Sheet

| Metric | Good | Warning | Critical |
|--------|------|---------|----------|
| CPU | < 70% | 70-85% | > 85% |
| Memory | < 75% | 75-90% | > 90% |
| P99 Latency | < 10ms | 10-50ms | > 50ms |
| Error Rate | < 0.1% | 0.1-0.5% | > 0.5% |
| Replica Count | >= minReplicas | < minReplicas | 0 |

---

## Additional Resources

- **Load Balancing**: See `LOAD_BALANCING_SCALING.md`
- **Performance Benchmarks**: See `PERFORMANCE_BENCHMARKS.md`
- **Monitoring Dashboards**: Grafana (http://localhost:3000)
- **Prometheus Alerts**: Alertmanager (http://localhost:9093)

---

**Last Updated**: 2025-10-07
**Version**: 1.0
**Owner**: Platform Engineering Team