foxhunt/services/load_tests/README.md

# Load Tests - Trading Service Throughput Validation

## Overview

Comprehensive load testing suite for validating trading service throughput and performance under various load scenarios.

## Test Scenarios

### 1. Sustained Load (10,000 orders/sec for 60s)
- **Target**: 10,000 orders/second sustained throughput
- **Duration**: 60 seconds
- **Concurrent Clients**: 100
- **Validates**: System stability under sustained load

### 2. Peak Burst (50,000 orders/sec for 10s)
- **Target**: 50,000 orders/second peak burst
- **Duration**: 10 seconds
- **Concurrent Clients**: 500
- **Validates**: System behavior under peak load spikes

### 3. Market Data Streaming (1M updates)
- **Target**: 1,000,000 concurrent market data updates
- **Streams**: 1,000 concurrent streams
- **Duration**: 30 seconds
- **Validates**: Streaming infrastructure capacity

### 4. Connection Pool Saturation (1,000 clients)
- **Target**: 1,000 concurrent clients
- **Requests per Client**: 100
- **Validates**: Connection pool management and resource limits

## Usage

### Run All Tests
```bash
cargo run -p load_tests --release -- --scenario all
```

### Run Individual Scenarios
```bash
# Sustained load
cargo run -p load_tests --release -- --scenario sustained

# Peak burst
cargo run -p load_tests --release -- --scenario burst

# Streaming
cargo run -p load_tests --release -- --scenario streaming

# Connection pool
cargo run -p load_tests --release -- --scenario pool
```

### Custom Configuration
```bash
cargo run -p load_tests --release -- \
  --scenario sustained \
  --url http://trading-service:50052 \
  --output /path/to/report.md \
  --verbose
```

## Metrics Collected

### Throughput Metrics
- Requests per second (sustained and peak)
- Total requests processed
- Success/failure rates

### Latency Distribution
- P50 (median) latency
- P95 latency
- P99 latency
- Maximum latency

### Resource Usage
- Memory consumption (average)
- Connection pool utilization
- Stream management overhead

## Output Report

Test results are saved as Markdown reports containing:
- Executive summary
- Detailed metrics breakdown
- Latency distribution charts
- Resource usage analysis
- Performance recommendations

Default output: `/tmp/WAVE_120_AGENT_5_LOAD_TESTING.md`

## Prerequisites

1. **Trading Service Running**:
   ```bash
   docker-compose up -d trading_service
   # OR
   cargo run -p trading_service
   ```

2. **Database Available**:
   ```bash
   docker-compose up -d postgres redis
   ```

3. **Sufficient System Resources**:
   - 8GB+ RAM recommended
   - Multi-core CPU for parallel clients
   - Network bandwidth for 50k+ req/sec

## Architecture

### Components

- **Scenarios**: Test scenario implementations
  - `sustained_load.rs`: 10k orders/sec for 60s
  - `burst_load.rs`: 50k orders/sec for 10s
  - `streaming_load.rs`: 1M market data updates
  - `pool_saturation.rs`: 1000 concurrent clients
  - `comprehensive.rs`: All scenarios sequentially

- **Clients**: gRPC client implementations
  - `trading_client.rs`: Trading service client wrapper

- **Metrics**: Performance measurement
  - `metrics.rs`: HDR histogram-based metrics collection
  - `monitor.rs`: System resource monitoring

### Load Generation Pattern

```rust
// Concurrent client pattern
for client_id in 0..NUM_CLIENTS {
    tokio::spawn(async move {
        let client = TradingClient::connect(url).await?;

        // Submit orders with rate limiting
        while duration_remaining {
            client.submit_order(...).await?;
            tokio::time::sleep(rate_limit).await;
        }
    });
}
```

## Performance Targets

### Sustained Load
- ✅ Throughput: ≥9,000 req/sec
- ✅ Error Rate: <1%
- ✅ P95 Latency: <10ms

### Peak Burst
- ✅ Throughput: ≥40,000 req/sec
- ✅ Error Rate: <5%
- ✅ P99 Latency: <50ms

### Streaming
- ✅ Updates: ≥900k received
- ✅ Concurrent Streams: 1000
- ✅ Stream Stability: <1% failures

### Connection Pool
- ✅ Concurrent Connections: 1000
- ✅ Error Rate: <5%
- ✅ P99 Latency: <100ms

## Troubleshooting

### Connection Refused
```bash
# Verify trading service is running
grpc_health_probe -addr=localhost:50052
```

### High Error Rates
- Check system resource limits (ulimit, file descriptors)
- Verify database connection pool size
- Review trading service logs for errors

### Memory Issues
- Reduce concurrent clients
- Enable connection pooling
- Check for memory leaks in trading service

## Integration with CI/CD

```yaml
# .github/workflows/load-test.yml
- name: Run Load Tests
  run: |
    docker-compose up -d
    cargo run -p load_tests --release -- --scenario all

- name: Upload Report
  uses: actions/upload-artifact@v3
  with:
    name: load-test-report
    path: /tmp/WAVE_120_AGENT_5_LOAD_TESTING.md
```

## Wave 120 Objectives

**Agent 5 Tasks**:
- ✅ Create load_tests package
- ✅ Implement 4 throughput scenarios
- ✅ Measure latency, throughput, error rates
- ✅ Monitor memory usage
- ⏳ Run tests against live service
- ⏳ Generate performance report

**Expected Outcomes**:
- Validate 10k orders/sec sustained capacity
- Confirm 50k orders/sec peak burst handling
- Verify 1M concurrent stream updates
- Validate 1000+ concurrent client support