foxhunt/testing/api-gateway-load/QUICK_START.md

API Gateway Load Testing - Quick Start Guide

Prerequisites

1. Docker Services Running:

   docker ps | grep -E "(redis|postgres)"
   # Should show: api_gateway_test_redis (port 6380)
   # Should show: api_gateway_test_postgres (port 5433)
   

2. API Gateway Service Running:

   curl http://localhost:50050/health
   # Should return: {"status":"ok"}
   

3. Load Test Runner Built:

   cargo build --release -p api_gateway_load_tests
   

Quick Test Commands

1. Normal Load Test (1K clients, 60 seconds)

cd services/api_gateway/load_tests
cargo run --release --bin load_test_runner -- normal

Output: normal_load_report.html + 3 SVG charts

2. Spike Load Test (0→10K clients, 70 seconds)

cargo run --release --bin load_test_runner -- spike

Output: spike_load_report.html + 3 SVG charts

3. Stress Test (Find breaking point)

cargo run --release --bin load_test_runner -- stress

Output: stress_test_report.html + 3 SVG charts

4. Sustained Load Test (100 clients, 24 hours)

# Run in screen/tmux session
screen -S load_test
cargo run --release --bin load_test_runner -- sustained
# Detach: Ctrl+A, D

Output: sustained_load_report.html + 3 SVG charts

5. Run All Tests Sequentially

cargo run --release --bin load_test_runner -- all

Output: All 3 HTML reports + 9 SVG charts

Custom Test Parameters

Normal Load (Custom)

cargo run --release --bin load_test_runner -- normal \
    --gateway-url http://localhost:50050 \
    --num-clients 500 \
    --duration-secs 120

Spike Load (Custom)

cargo run --release --bin load_test_runner -- spike \
    --gateway-url http://localhost:50050 \
    --target-clients 5000 \
    --ramp-up-secs 15 \
    --sustain-secs 90

Stress Test (Custom)

cargo run --release --bin load_test_runner -- stress \
    --gateway-url http://localhost:50050 \
    --initial-clients 200 \
    --increment 200 \
    --increment-interval-secs 30 \
    --max-p99-latency-ms 100.0 \
    --max-error-rate-pct 10.0

Viewing Reports

In Browser

firefox normal_load_report.html &
firefox spike_load_report.html &
firefox stress_test_report.html &

Extract Key Metrics

# P99 Latency
grep -A1 "P99 Latency" normal_load_report.html | grep "value"

# Error Rate
grep -A1 "Error Rate" normal_load_report.html | grep "value"

# Requests/Second
grep -A1 "Requests/Second" normal_load_report.html | grep "value"

Performance Targets

Metric	Target	How to Verify
P99 Latency	<10μs	Check HTML report summary
Throughput	>100K req/s	Check HTML report summary
Error Rate	<0.1%	Check HTML report summary
P50 Latency	<2μs	Check latency statistics table
P90 Latency	<5μs	Check latency statistics table

Troubleshooting

Test Fails to Connect

# Check API Gateway is running
curl http://localhost:50050/health

# Check port binding
netstat -tlnp | grep 50050

# Start API Gateway
cargo run --release -p api_gateway

High Latency Results

# Check system load
htop

# Check API Gateway logs
docker logs foxhunt-api-gateway --tail 100

# Check PostgreSQL connections
psql -h localhost -p 5433 -U postgres -c 'SELECT count(*) FROM pg_stat_activity;'

High Error Rate

# Check API Gateway errors
journalctl -u foxhunt-api-gateway -n 100

# Check backend service health
curl http://localhost:50051/health  # Trading
curl http://localhost:50052/health  # Backtesting
curl http://localhost:50053/health  # ML Training

Monitoring During Tests

Real-time Metrics

# Watch Prometheus metrics
watch -n 1 "curl -s http://localhost:50050/metrics | grep -E '(requests_total|latency)'"

# Monitor Docker stats
docker stats foxhunt-api-gateway --no-stream

System Resources

# CPU usage
top -p $(pgrep api_gateway)

# Memory usage
ps aux | grep api_gateway

# Network usage
iftop -i lo

Quick Validation (5 minutes)

# 1. Start services
docker-compose up -d redis postgres
cargo run --release -p api_gateway &

# 2. Wait for health
sleep 5
curl http://localhost:50050/health

# 3. Run quick test (100 clients, 30 seconds)
cd services/api_gateway/load_tests
cargo run --release --bin load_test_runner -- normal \
    --num-clients 100 \
    --duration-secs 30

# 4. View report
firefox normal_load_report.html

Expected Results

Good Performance

Total Requests:      6,000,000+
Requests/Second:     100,000+
Error Rate:          <0.1%
P99 Latency:         <10μs
Circuit Breaker:     0 activations

Needs Optimization

Total Requests:      60,000 - 600,000
Requests/Second:     1,000 - 10,000
Error Rate:          0.5% - 5%
P99 Latency:         10-100ms
Circuit Breaker:     May activate

Report Files

After running tests, you'll find:

normal_load_report.html         # Main HTML dashboard
normal_load_report.rps.svg      # Requests/second chart
normal_load_report.latency.svg  # P99 latency chart
normal_load_report.errors.svg   # Error rate chart

Next Steps

1. Review HTML reports for performance insights
2. Check "Capacity Recommendation" section for bottlenecks
3. Implement optimization recommendations
4. Re-run tests to validate improvements
5. Document baseline performance for regression testing