# High Latency Troubleshooting Guide

**Last Updated**: 2025-10-22
**Target**: P99 latency <500ms

---

## Diagnosis

### Step 1: Identify Latency Source

```bash
# Check API Gateway latency
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8080/health
# Response time: 1250ms (HIGH)

# Check database query performance
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT query, mean_exec_time, calls FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

# Check service-to-service latency
kubectl logs -n foxhunt -l app.kubernetes.io/component=trading-service | grep "request_duration_ms" | tail -100
```

### Step 2: Check Resource Utilization

```bash
# CPU usage
kubectl top pods -n foxhunt --sort-by=cpu | head -20

# Memory usage
kubectl top pods -n foxhunt --sort-by=memory | head -20

# Disk I/O (PostgreSQL)
kubectl exec -n foxhunt foxhunt-postgresql-0 -- iostat -x 1 5
```

### Step 3: Analyze Slow Queries

```bash
# Enable slow query logging (PostgreSQL)
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "ALTER SYSTEM SET log_min_duration_statement = 100;"  # Log queries >100ms

kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
    psql -U foxhunt -d foxhunt -c "SELECT pg_reload_conf();"

# View slow queries
kubectl logs -n foxhunt foxhunt-postgresql-0 | grep "duration:" | tail -50
```

---

## Common Causes

### Cause 1: Missing Database Index

**Symptom**: Queries take >1s, high CPU on database

**Diagnosis**:
```sql
-- Find sequential scans on large tables
SELECT schemaname, tablename, seq_scan, seq_tup_read, idx_scan 
FROM pg_stat_user_tables 
WHERE seq_scan > 0 AND seq_tup_read > 100000 
ORDER BY seq_tup_read DESC 
LIMIT 10;
```

**Resolution**:
```sql
-- Create missing index
CREATE INDEX CONCURRENTLY idx_orders_symbol_created_at ON orders(symbol, created_at DESC);

-- Verify index usage
EXPLAIN ANALYZE SELECT * FROM orders WHERE symbol='ES.FUT' ORDER BY created_at DESC LIMIT 100;
```

### Cause 2: Connection Pool Exhausted

**Symptom**: `timeout acquiring connection from pool`

**Diagnosis**:
```bash
kubectl logs -n foxhunt -l app.kubernetes.io/component=trading-service | grep "connection pool"
```

**Resolution**:
```bash
# Increase pool size via environment variable
kubectl set env deployment/foxhunt-trading-service -n foxhunt DATABASE_MAX_CONNECTIONS=50

# OR scale horizontally
kubectl scale deployment/foxhunt-trading-service -n foxhunt --replicas=10
```

### Cause 3: Network Latency

**Symptom**: High latency between services

**Diagnosis**:
```bash
# Test inter-service latency
kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- \
    ping foxhunt-postgresql.foxhunt.svc.cluster.local

# Check for packet loss
kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- \
    mtr --report --report-cycles 10 foxhunt-postgresql.foxhunt.svc.cluster.local
```

**Resolution**:
```bash
# Verify services are in same availability zone
kubectl get pods -n foxhunt -o wide

# Use node affinity to colocate services
```

---

**End of High Latency Troubleshooting Guide**