foxhunt/testing/api-gateway-load/QUICK_START.md

# API Gateway Load Testing - Quick Start Guide

## Prerequisites

1. **Docker Services Running**:
   ```bash
   docker ps | grep -E "(redis|postgres)"
   # Should show: api_gateway_test_redis (port 6380)
   # Should show: api_gateway_test_postgres (port 5433)
   ```

2. **API Gateway Service Running**:
   ```bash
   curl http://localhost:50050/health
   # Should return: {"status":"ok"}
   ```

3. **Load Test Runner Built**:
   ```bash
   cargo build --release -p api_gateway_load_tests
   ```

## Quick Test Commands

### 1. Normal Load Test (1K clients, 60 seconds)
```bash
cd services/api_gateway/load_tests
cargo run --release --bin load_test_runner -- normal
```

**Output**: `normal_load_report.html` + 3 SVG charts

### 2. Spike Load Test (0→10K clients, 70 seconds)
```bash
cargo run --release --bin load_test_runner -- spike
```

**Output**: `spike_load_report.html` + 3 SVG charts

### 3. Stress Test (Find breaking point)
```bash
cargo run --release --bin load_test_runner -- stress
```

**Output**: `stress_test_report.html` + 3 SVG charts

### 4. Sustained Load Test (100 clients, 24 hours)
```bash
# Run in screen/tmux session
screen -S load_test
cargo run --release --bin load_test_runner -- sustained
# Detach: Ctrl+A, D
```

**Output**: `sustained_load_report.html` + 3 SVG charts

### 5. Run All Tests Sequentially
```bash
cargo run --release --bin load_test_runner -- all
```

**Output**: All 3 HTML reports + 9 SVG charts

## Custom Test Parameters

### Normal Load (Custom)
```bash
cargo run --release --bin load_test_runner -- normal \
    --gateway-url http://localhost:50050 \
    --num-clients 500 \
    --duration-secs 120
```

### Spike Load (Custom)
```bash
cargo run --release --bin load_test_runner -- spike \
    --gateway-url http://localhost:50050 \
    --target-clients 5000 \
    --ramp-up-secs 15 \
    --sustain-secs 90
```

### Stress Test (Custom)
```bash
cargo run --release --bin load_test_runner -- stress \
    --gateway-url http://localhost:50050 \
    --initial-clients 200 \
    --increment 200 \
    --increment-interval-secs 30 \
    --max-p99-latency-ms 100.0 \
    --max-error-rate-pct 10.0
```

## Viewing Reports

### In Browser
```bash
firefox normal_load_report.html &
firefox spike_load_report.html &
firefox stress_test_report.html &
```

### Extract Key Metrics
```bash
# P99 Latency
grep -A1 "P99 Latency" normal_load_report.html | grep "value"

# Error Rate
grep -A1 "Error Rate" normal_load_report.html | grep "value"

# Requests/Second
grep -A1 "Requests/Second" normal_load_report.html | grep "value"
```

## Performance Targets

| Metric | Target | How to Verify |
|--------|--------|---------------|
| P99 Latency | <10μs | Check HTML report summary |
| Throughput | >100K req/s | Check HTML report summary |
| Error Rate | <0.1% | Check HTML report summary |
| P50 Latency | <2μs | Check latency statistics table |
| P90 Latency | <5μs | Check latency statistics table |

## Troubleshooting

### Test Fails to Connect
```bash
# Check API Gateway is running
curl http://localhost:50050/health

# Check port binding
netstat -tlnp | grep 50050

# Start API Gateway
cargo run --release -p api_gateway
```

### High Latency Results
```bash
# Check system load
htop

# Check API Gateway logs
docker logs foxhunt-api-gateway --tail 100

# Check PostgreSQL connections
psql -h localhost -p 5433 -U postgres -c 'SELECT count(*) FROM pg_stat_activity;'
```

### High Error Rate
```bash
# Check API Gateway errors
journalctl -u foxhunt-api-gateway -n 100

# Check backend service health
curl http://localhost:50051/health  # Trading
curl http://localhost:50052/health  # Backtesting
curl http://localhost:50053/health  # ML Training
```

## Monitoring During Tests

### Real-time Metrics
```bash
# Watch Prometheus metrics
watch -n 1 "curl -s http://localhost:50050/metrics | grep -E '(requests_total|latency)'"

# Monitor Docker stats
docker stats foxhunt-api-gateway --no-stream
```

### System Resources
```bash
# CPU usage
top -p $(pgrep api_gateway)

# Memory usage
ps aux | grep api_gateway

# Network usage
iftop -i lo
```

## Quick Validation (5 minutes)

```bash
# 1. Start services
docker-compose up -d redis postgres
cargo run --release -p api_gateway &

# 2. Wait for health
sleep 5
curl http://localhost:50050/health

# 3. Run quick test (100 clients, 30 seconds)
cd services/api_gateway/load_tests
cargo run --release --bin load_test_runner -- normal \
    --num-clients 100 \
    --duration-secs 30

# 4. View report
firefox normal_load_report.html
```

## Expected Results

### Good Performance
```
Total Requests:      6,000,000+
Requests/Second:     100,000+
Error Rate:          <0.1%
P99 Latency:         <10μs
Circuit Breaker:     0 activations
```

### Needs Optimization
```
Total Requests:      60,000 - 600,000
Requests/Second:     1,000 - 10,000
Error Rate:          0.5% - 5%
P99 Latency:         10-100ms
Circuit Breaker:     May activate
```

## Report Files

After running tests, you'll find:
```
normal_load_report.html         # Main HTML dashboard
normal_load_report.rps.svg      # Requests/second chart
normal_load_report.latency.svg  # P99 latency chart
normal_load_report.errors.svg   # Error rate chart
```

## Next Steps

1. Review HTML reports for performance insights
2. Check "Capacity Recommendation" section for bottlenecks
3. Implement optimization recommendations
4. Re-run tests to validate improvements
5. Document baseline performance for regression testing