foxhunt/testing/load/README.md

# Load Tests - Minimal Dependency Crate

**Purpose**: Fast-compiling load tests for Foxhunt Trading Service

**Compilation Time**: 20-30 seconds (vs 120-180s in original tests/ crate)

**Dependency Reduction**: 86% (5 deps vs 36 deps)

---

## Quick Start

### Rust Load Tests (Direct Trading Service - Port 50052)

```bash
cd tests/load_tests
cargo test --release -- --nocapture
```

### Authenticated ghz Load Tests (API Gateway - Port 50051)

```bash
cd tests/load_tests

# Quick authentication test (1 request)
./ghz_quick_auth_test.sh

# Full authenticated load test suite
./ghz_authenticated.sh
```

---

## Test Suites

### 1. Rust Load Tests (Minimal Dependencies)

**Target**: Trading Service direct (port 50052)  
**Auth**: Not required (direct backend access)

#### Run Specific Test

```bash
# Baseline latency (1000 sequential orders)
cargo test --release test_1_baseline_latency -- --nocapture

# Concurrent connections (100 clients, 100 orders each)
cargo test --release test_2_concurrent_connections -- --nocapture

# Database performance (5000 orders)
cargo test --release test_4_database_performance -- --nocapture

# Resource monitoring (health + metrics)
cargo test --release test_5_resource_monitoring -- --nocapture

# Production readiness assessment
cargo test --release test_6_production_readiness -- --nocapture
```

#### Run Sustained Load Test (Ignored by Default)

```bash
# 5-minute sustained load (50 clients, 200 orders/sec each = 10K total)
cargo test --release test_3_sustained_load -- --ignored --nocapture
```

### 2. Authenticated ghz Load Tests (Shell Scripts)

**Target**: API Gateway (port 50051)  
**Auth**: JWT tokens (auto-generated)  
**Protocol**: gRPC with metadata

#### Prerequisites

1. **Install ghz** (if not already installed):
```bash
# Ubuntu/Debian
wget https://github.com/bojand/ghz/releases/download/v0.117.0/ghz-linux-x86_64.tar.gz
tar -xzf ghz-linux-x86_64.tar.gz
sudo mv ghz /usr/local/bin/

# MacOS
brew install ghz

# Arch Linux
yay -S ghz
```

2. **Install jq** (optional, for result parsing):
```bash
sudo apt-get install jq  # Ubuntu/Debian
brew install jq          # MacOS
```

3. **Start API Gateway**:
```bash
docker-compose up -d api_gateway postgres trading_service
```

4. **Configure JWT Secret** (already in .env):
```bash
# Verify JWT_SECRET is set
grep JWT_SECRET .env
```

#### Available Scripts

##### Quick Authentication Test
```bash
# Verify JWT auth works (1 request only)
./ghz_quick_auth_test.sh
```

**Output**: Single authenticated request to validate setup

##### Full Authenticated Load Suite
```bash
# Run all 4 test scenarios (baseline, medium, high, sustained)
./ghz_authenticated.sh
```

**Test Scenarios**:
1. **Baseline**: 1,000 requests @ 100 RPS (10 concurrent)
2. **Medium**: 5,000 requests @ 500 RPS (50 concurrent)
3. **High**: 10,000 requests @ 1,000 RPS (100 concurrent)
4. **Sustained**: 2 minutes @ 500 RPS (60,000 total requests)

**Output Files**: `results/baseline_authenticated_*.json`, etc.

#### JWT Token Generation

The scripts automatically generate JWT tokens using:
```bash
# Manual token generation (if needed)
./tests/e2e_helpers/jwt_token_generator.sh [username] [role]

# Example
./tests/e2e_helpers/jwt_token_generator.sh "load_test_user" "trader"
```

**Token Features**:
- 1-hour expiration
- Includes trading permissions (submit_order, view_positions, cancel_order)
- Signed with JWT_SECRET from .env
- Includes jti, role, sub fields (required by API Gateway)

#### Results Analysis

**JSON Output** (with jq installed):
```bash
# View summary of latest test
jq '.' tests/load_tests/results/baseline_authenticated_*.json | tail -1
```

**Metrics Collected**:
- Total requests
- Success rate (%)
- P50, P95, P99 latency (ms)
- Throughput (req/s)
- Error distribution

**Monitoring Endpoints**:
- Prometheus: http://localhost:9091/metrics
- Grafana: http://localhost:3000

---

## Prerequisites

### Infrastructure Running

```bash
# For Rust tests (Trading Service direct)
docker-compose up -d postgres trading_service

# For ghz tests (API Gateway)
docker-compose up -d postgres trading_service api_gateway

# Verify services healthy
docker-compose ps
```

### Service Endpoints

| Service | Protocol | Port | Auth | Used By |
|---------|----------|------|------|---------|
| Trading Service | gRPC | 50052 | No | Rust tests |
| API Gateway | gRPC | 50051 | JWT | ghz scripts |
| Health (Trading) | HTTP | 8081 | No | test_5 |
| Metrics (Trading) | HTTP | 9092 | No | test_5 |
| Metrics (Gateway) | HTTP | 9091 | No | Monitoring |

---

## Test Details

### Rust Test Suite

#### Test 1: Baseline Latency
- **Orders**: 1,000 sequential
- **Purpose**: Single-client latency baseline
- **Metrics**: P50, P95, P99 latency + throughput

#### Test 2: Concurrent Connections
- **Clients**: 100 concurrent
- **Orders per client**: 100
- **Total orders**: 10,000
- **Purpose**: Concurrency stress test
- **Metrics**: Latency distribution + success rate

#### Test 3: Sustained Load (Ignored)
- **Duration**: 5 minutes
- **Clients**: 50 concurrent
- **Target rate**: 10,000 orders/sec total
- **Purpose**: Sustained load validation
- **Metrics**: Long-term stability

#### Test 4: Database Performance
- **Orders**: 5,000
- **Purpose**: Database write throughput
- **Target**: >2,000 writes/sec

#### Test 5: Resource Monitoring
- **Purpose**: Health + metrics validation
- **Checks**: HTTP health endpoint, Prometheus metrics
- **Requires**: `health-checks` feature

#### Test 6: Production Readiness
- **Clients**: 50 concurrent
- **Orders per client**: 200
- **Total orders**: 10,000
- **Criteria**:
  - Success rate >= 99%
  - Throughput >= 5,000 orders/sec
  - P99 latency < 100ms

### ghz Authenticated Test Suite

#### Test 1: Baseline Authenticated Load
- **Requests**: 1,000
- **RPS**: 100
- **Concurrency**: 10
- **Purpose**: Verify JWT auth + baseline latency
- **Expected**: 100% success, <50ms P99

#### Test 2: Medium Authenticated Load
- **Requests**: 5,000
- **RPS**: 500
- **Concurrency**: 50
- **Purpose**: Medium load with authentication
- **Expected**: >99% success, <100ms P99

#### Test 3: High Authenticated Load
- **Requests**: 10,000
- **RPS**: 1,000
- **Concurrency**: 100
- **Purpose**: High throughput with JWT overhead
- **Expected**: >95% success, <150ms P99

#### Test 4: Sustained Authenticated Load
- **Duration**: 2 minutes
- **RPS**: 500
- **Concurrency**: 50
- **Total**: ~60,000 requests
- **Purpose**: Long-term stability validation
- **Expected**: >99% success, stable latency

---

## Features

### Default (No Features)
- Core gRPC load testing (tests 1-4, 6)
- Dependencies: `tokio`, `tonic`, `uuid`

### `health-checks` (Optional)
```bash
cargo test --release --features health-checks
```
- Enables test_5 (resource monitoring)
- Adds `reqwest` dependency
- HTTP health + metrics checks

---

## Performance Targets

| Metric | Target | Typical (Direct) | Typical (Gateway) |
|--------|--------|------------------|-------------------|
| Success Rate | >= 99% | 99.5-100% | 99-100% |
| Throughput | >= 5K orders/sec | 7-10K | 5-7K |
| P50 Latency | < 20ms | 10-15ms | 15-25ms |
| P99 Latency | < 100ms | 30-50ms | 50-100ms |
| DB Writes/sec | >= 2K | 2.5-3K | 2-2.5K |

**Note**: API Gateway adds ~5-10ms latency due to JWT validation and proxying.

---

## Troubleshooting

### "Connection refused" Error

**For Rust tests (port 50052)**:
```bash
docker-compose up -d trading_service
docker-compose ps  # Verify "Up" status
```

**For ghz tests (port 50051)**:
```bash
docker-compose up -d api_gateway
docker-compose ps  # Verify "Up" status
```

### "Failed to generate JWT token"

**Check JWT_SECRET**:
```bash
# Verify secret exists
grep JWT_SECRET .env

# If missing, add to .env
echo 'JWT_SECRET=your-secret-key-here' >> .env
```

### "Too many open files" Error
```bash
ulimit -n 4096  # Increase file descriptor limit
```

### Authentication Failures (401 errors)

**Check token format**:
```bash
# Generate test token
./tests/e2e_helpers/jwt_token_generator.sh test_user trader

# Verify token has 3 parts (header.payload.signature)
```

**Check API Gateway logs**:
```bash
docker-compose logs api_gateway | grep -i "auth\|jwt\|401"
```

### High Latency

**Check**:
1. PostgreSQL synchronous_commit setting
2. Network latency (localhost vs Docker)
3. System load (CPU, memory)
4. API Gateway JWT validation overhead

**Optimize PostgreSQL**:
```sql
-- In PostgreSQL
ALTER SYSTEM SET synchronous_commit = off;
SELECT pg_reload_conf();
```

---

## Compilation Time Comparison

| Crate | Dependencies | Compile Time | Speedup |
|-------|--------------|--------------|---------|
| `tests/` (original) | 36 | 120-180s | Baseline |
| `tests/load_tests` | 5 | 20-30s | **6x faster** |
| ghz scripts | N/A | 0s | **Instant** |

---

## Architecture

### Rust Tests (Minimal Dependencies)
```toml
[dependencies]
tokio = { workspace = true }           # Async runtime
tonic = { workspace = true }           # gRPC client
tonic-prost = { workspace = true }     # Protobuf runtime
prost = { workspace = true }           # Protobuf types
uuid = { workspace = true }            # Order IDs
reqwest = { optional = true }          # HTTP (feature-gated)
```

### ghz Scripts (Shell + OpenSSL)
```bash
# Dependencies
- bash
- ghz (gRPC load testing)
- openssl (JWT signing)
- jq (optional, result parsing)
- nc (netcat, connectivity check)
```

### Build Process
1. `build.rs` compiles `trading.proto` from Trading Service
2. Generated code included via `tonic::include_proto!("trading")`
3. No heavy dependencies (ML, database clients, test frameworks)

---

## CI/CD Integration

### GitHub Actions
```yaml
- name: Run Rust Load Tests
  run: |
    docker-compose up -d postgres trading_service
    cd tests/load_tests
    cargo test --release --features health-checks

- name: Run Authenticated ghz Tests
  run: |
    docker-compose up -d api_gateway postgres trading_service
    cd tests/load_tests
    ./ghz_quick_auth_test.sh
    ./ghz_authenticated.sh
```

### GitLab CI
```yaml
rust_load_tests:
  script:
    - docker-compose up -d postgres trading_service
    - cd tests/load_tests
    - cargo test --release --features health-checks

ghz_load_tests:
  script:
    - docker-compose up -d api_gateway postgres trading_service
    - cd tests/load_tests
    - ./ghz_authenticated.sh
```

---

## Related Documentation

- [LOAD_TEST_DEPENDENCY_OPTIMIZATION.md](../../LOAD_TEST_DEPENDENCY_OPTIMIZATION.md) - Detailed analysis
- [LOAD_TEST_OPTIMIZATION_SUMMARY.md](../../LOAD_TEST_OPTIMIZATION_SUMMARY.md) - Implementation summary
- [TESTING_PLAN.md](../../TESTING_PLAN.md) - Overall testing strategy
- [WAVE_132_AUTH_VALIDATION_SUMMARY.md](../../WAVE_132_AUTH_VALIDATION_SUMMARY.md) - JWT authentication validation

---

## Summary

| Test Type | Target | Auth | Compilation | Execution | Use Case |
|-----------|--------|------|-------------|-----------|----------|
| Rust Tests | Trading Service (50052) | No | 20-30s | Fast | Backend performance |
| ghz Scripts | API Gateway (50051) | JWT | 0s | Fast | End-to-end auth flow |

**Recommendation**: Use **both** test types for comprehensive validation:
1. **Rust tests** for backend performance benchmarks
2. **ghz scripts** for authenticated API Gateway validation

---

**Status**: ✅ Production Ready  
**Rust Tests**: < 30 seconds compilation ✅  
**ghz Scripts**: Instant execution ✅  
**JWT Authentication**: Fully validated ✅