foxhunt/services/api_gateway/load_tests/README.md

# API Gateway Load Testing Framework

Comprehensive load testing infrastructure for validating API Gateway performance under high concurrency.

## Overview

This framework provides four test scenarios with detailed metrics collection, time-series analysis, and HTML report generation:

1. **Normal Load**: 1K concurrent clients for 60 seconds
2. **Spike Load**: 0→10K clients in 10s, sustain 60s
3. **Sustained Load**: 100 clients for 24 hours (endurance test)
4. **Stress Test**: Incrementally increase load until failure

## Architecture

```
┌─────────────────────┐
│  Test Orchestrator  │
└──────────┬──────────┘
           │
           ├─────────────────┬─────────────────┬─────────────────┐
           │                 │                 │                 │
    ┌──────▼──────┐   ┌──────▼──────┐   ┌──────▼──────┐   ┌──────▼──────┐
    │   Normal    │   │    Spike    │   │  Sustained  │   │   Stress    │
    │    Load     │   │    Load     │   │    Load     │   │    Test     │
    └──────┬──────┘   └──────┬──────┘   └──────┬──────┘   └──────┬──────┘
           │                 │                 │                 │
           └─────────────────┴─────────────────┴─────────────────┘
                                      │
                          ┌───────────▼───────────┐
                          │  Virtual Client Pool  │
                          │  (Authenticated HTTP  │
                          │   + Mixed Workload)   │
                          └───────────┬───────────┘
                                      │
                          ┌───────────▼───────────┐
                          │  Metrics Collector    │
                          │  - HDR Histogram      │
                          │  - Time Series Data   │
                          │  - Per-Service Stats  │
                          └───────────┬───────────┘
                                      │
                          ┌───────────▼───────────┐
                          │  Report Generator     │
                          │  - HTML + SVG Charts  │
                          │  - Capacity Analysis  │
                          └───────────────────────┘
```

## Installation

```bash
cd /home/jgrusewski/Work/foxhunt/services/api_gateway/load_tests
cargo build --release
```

## Usage

### Run Individual Scenarios

```bash
# Normal load test (1K clients, 60s)
cargo run --release -- normal \
  --gateway-url http://localhost:50050 \
  --num-clients 1000 \
  --duration-secs 60

# Spike load test (0→10K in 10s, sustain 60s)
cargo run --release -- spike \
  --gateway-url http://localhost:50050 \
  --target-clients 10000 \
  --ramp-up-secs 10 \
  --sustain-secs 60

# Sustained load test (100 clients, 24h)
cargo run --release -- sustained \
  --gateway-url http://localhost:50050 \
  --num-clients 100 \
  --duration-secs 86400

# Stress test (incrementally increase until failure)
cargo run --release -- stress \
  --gateway-url http://localhost:50050 \
  --initial-clients 100 \
  --increment 100 \
  --increment-interval-secs 60 \
  --max-p99-latency-ms 50.0 \
  --max-error-rate-pct 5.0
```

### Run All Scenarios

```bash
cargo run --release -- all --gateway-url http://localhost:50050
```

## Metrics Collected

### Latency Statistics
- **Min/Max/Mean**: Full latency range
- **Percentiles**: P50, P90, P95, P99, P99.9
- **Standard Deviation**: Latency consistency

### Request Breakdown
- **Total Requests**: Aggregate count
- **Successful**: 2xx responses
- **Failed**: 4xx/5xx errors
- **Timeout**: Connection/request timeouts
- **Rate Limited**: 429 responses
- **Circuit Breaker**: 503 responses

### Time Series Data (1-second intervals)
- Requests per second (RPS)
- P99 latency
- Error rate percentage
- Active client count

### Per-Service Statistics
- **Trading Service**: Order submission, position queries
- **Backtesting Service**: Backtest execution
- **ML Training Service**: Model training requests

## Workload Distribution

Mixed workload simulates realistic usage:

- **60%** - Order submissions
- **30%** - Position queries
- **8%** - Backtesting requests
- **2%** - ML training requests

Each client has random think time (1-50ms) between requests to simulate human behavior.

## Report Generation

HTML reports are automatically generated with:

1. **Summary Cards**: Total requests, RPS, error rate, P99 latency
2. **Latency Table**: All percentiles with statistics
3. **Request Breakdown**: Success/failure categorization
4. **Performance Charts** (SVG):
   - Requests per second over time
   - P99 latency over time
   - Error rate over time
5. **Capacity Recommendations**: Based on observed performance

## Success Criteria

### Normal Load
- **Target**: <10ms P99 latency, 0% errors
- **Pass**: Error rate < 1%, P99 < 10ms
- **Fail**: Error rate ≥ 5%, P99 ≥ 50ms

### Spike Load
- **Target**: Graceful handling, circuit breakers activate
- **Pass**: Error rate < 10%, circuit breakers respond correctly
- **Fail**: System crashes, uncontrolled cascading failures

### Sustained Load
- **Target**: No memory leaks, stable latency
- **Pass**: Latency drift < 5%, error rate stddev < 2%
- **Fail**: Latency increases > 10%, memory exhaustion

### Stress Test
- **Target**: Identify capacity limits
- **Pass**: Breaking point identified with clear bottleneck
- **Fail**: Undefined behavior, data corruption

## Example Report Output

```
Load Test Report: Normal Load Test
Test Period: 2025-10-03 12:00:00 to 2025-10-03 12:01:00
Duration: 60 seconds (0.02 hours)

Summary:
┌──────────────────┬──────────┐
│ Total Requests   │  120,000 │
│ Requests/Second  │   2,000  │
│ Error Rate       │   0.12%  │
│ P99 Latency      │   8.5ms  │
└──────────────────┴──────────┘

Latency Statistics:
┌──────────┬──────────┐
│ P50      │  3.2ms   │
│ P90      │  5.1ms   │
│ P95      │  6.8ms   │
│ P99      │  8.5ms   │
│ P99.9    │  12.3ms  │
└──────────┴──────────┘

Capacity Recommendation:
✓ System handled 1,000 clients with 0.12% error rate
✓ P99 latency well within 10ms target
✓ Safe for production at this load level
```

## Load Generator Resources

### Requirements
- **CPU**: 4+ cores for 1K clients, 8+ cores for 10K clients
- **Memory**: 2GB for 1K clients, 8GB for 10K clients
- **Network**: Low-latency connection to API Gateway

### Monitoring Load Generator

The framework monitors its own resource usage to ensure the load generator doesn't become a bottleneck. If you see warnings about load generator CPU/memory, consider:

1. Running on a larger machine
2. Distributing load across multiple generators
3. Reducing concurrent client count

## Advanced Configuration

### Custom JWT Token

Set authentication parameters in the code:

```rust
let auth_client = AuthenticatedClient::new(
    gateway_url,
    "your-jwt-secret",
    "user-id",
    "username"
).await?;
```

### Custom Test Duration

All scenarios support custom durations:

```bash
# Extended normal load test (5 minutes)
cargo run --release -- normal --duration-secs 300

# Long-running stress test
cargo run --release -- stress --increment-interval-secs 300
```

## Troubleshooting

### Connection Refused
```
Error: Connection refused (os error 111)
```
**Solution**: Ensure API Gateway is running at `http://localhost:50050`

### Too Many Open Files
```
Error: Too many open files (os error 24)
```
**Solution**: Increase system file descriptor limit:
```bash
ulimit -n 10000
```

### Memory Exhaustion
```
Error: Cannot allocate memory
```
**Solution**: Reduce `--num-clients` or run on a larger machine

### High Latency from Generator
```
Warning: Load generator CPU > 80%, results may be unreliable
```
**Solution**: Use a more powerful machine or reduce client count

## Integration with CI/CD

### GitHub Actions Example

```yaml
name: Load Testing

on:
  schedule:
    - cron: '0 0 * * 0'  # Weekly

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Start API Gateway
        run: |
          docker-compose up -d api_gateway
          sleep 10
      - name: Run Load Tests
        run: |
          cd services/api_gateway/load_tests
          cargo run --release -- normal
      - name: Upload Reports
        uses: actions/upload-artifact@v3
        with:
          name: load-test-reports
          path: |
            normal_load_report.html
            *.svg
```

## Performance Baseline

Expected results for reference hardware (AWS c5.4xlarge):

| Scenario       | Clients | RPS   | P99 Latency | Error Rate |
|----------------|---------|-------|-------------|------------|
| Normal Load    | 1,000   | 2,000 | 8ms         | <0.1%      |
| Spike Load     | 10,000  | 8,000 | 25ms        | <2%        |
| Sustained Load | 100     | 200   | 5ms         | <0.01%     |
| Stress Test    | 5,000   | 5,000 | 45ms        | Breaking   |

## License

Part of the Foxhunt HFT Trading System - MIT OR Apache-2.0