foxhunt/testing/load

Load Tests - Minimal Dependency Crate

Purpose: Fast-compiling load tests for Foxhunt Trading Service

Compilation Time: 20-30 seconds (vs 120-180s in original tests/ crate)

Dependency Reduction: 86% (5 deps vs 36 deps)

---

Quick Start

Rust Load Tests (Direct Trading Service - Port 50052)

cd tests/load_tests
cargo test --release -- --nocapture

Authenticated ghz Load Tests (API Gateway - Port 50051)

cd tests/load_tests

# Quick authentication test (1 request)
./ghz_quick_auth_test.sh

# Full authenticated load test suite
./ghz_authenticated.sh

---

Test Suites

1. Rust Load Tests (Minimal Dependencies)

Target: Trading Service direct (port 50052)
Auth: Not required (direct backend access)

Run Specific Test

# Baseline latency (1000 sequential orders)
cargo test --release test_1_baseline_latency -- --nocapture

# Concurrent connections (100 clients, 100 orders each)
cargo test --release test_2_concurrent_connections -- --nocapture

# Database performance (5000 orders)
cargo test --release test_4_database_performance -- --nocapture

# Resource monitoring (health + metrics)
cargo test --release test_5_resource_monitoring -- --nocapture

# Production readiness assessment
cargo test --release test_6_production_readiness -- --nocapture

Run Sustained Load Test (Ignored by Default)

# 5-minute sustained load (50 clients, 200 orders/sec each = 10K total)
cargo test --release test_3_sustained_load -- --ignored --nocapture

2. Authenticated ghz Load Tests (Shell Scripts)

Target: API Gateway (port 50051)
Auth: JWT tokens (auto-generated)
Protocol: gRPC with metadata

Prerequisites

1. Install ghz (if not already installed):

# Ubuntu/Debian
wget https://github.com/bojand/ghz/releases/download/v0.117.0/ghz-linux-x86_64.tar.gz
tar -xzf ghz-linux-x86_64.tar.gz
sudo mv ghz /usr/local/bin/

# MacOS
brew install ghz

# Arch Linux
yay -S ghz

2. Install jq (optional, for result parsing):

sudo apt-get install jq  # Ubuntu/Debian
brew install jq          # MacOS

3. Start API Gateway:

docker-compose up -d api_gateway postgres trading_service

4. Configure JWT Secret (already in .env):

# Verify JWT_SECRET is set
grep JWT_SECRET .env

Available Scripts

Quick Authentication Test

# Verify JWT auth works (1 request only)
./ghz_quick_auth_test.sh

Output: Single authenticated request to validate setup

Full Authenticated Load Suite

# Run all 4 test scenarios (baseline, medium, high, sustained)
./ghz_authenticated.sh

Test Scenarios:

1. Baseline: 1,000 requests @ 100 RPS (10 concurrent)
2. Medium: 5,000 requests @ 500 RPS (50 concurrent)
3. High: 10,000 requests @ 1,000 RPS (100 concurrent)
4. Sustained: 2 minutes @ 500 RPS (60,000 total requests)

Output Files: results/baseline_authenticated_*.json, etc.

JWT Token Generation

The scripts automatically generate JWT tokens using:

# Manual token generation (if needed)
./tests/e2e_helpers/jwt_token_generator.sh [username] [role]

# Example
./tests/e2e_helpers/jwt_token_generator.sh "load_test_user" "trader"

Token Features:

• 1-hour expiration
• Includes trading permissions (submit_order, view_positions, cancel_order)
• Signed with JWT_SECRET from .env
• Includes jti, role, sub fields (required by API Gateway)

Results Analysis

JSON Output (with jq installed):

# View summary of latest test
jq '.' tests/load_tests/results/baseline_authenticated_*.json | tail -1

Metrics Collected:

• Total requests
• Success rate (%)
• P50, P95, P99 latency (ms)
• Throughput (req/s)
• Error distribution

Monitoring Endpoints:

• Prometheus: http://localhost:9091/metrics
• Grafana: http://localhost:3000

---

Prerequisites

Infrastructure Running

# For Rust tests (Trading Service direct)
docker-compose up -d postgres trading_service

# For ghz tests (API Gateway)
docker-compose up -d postgres trading_service api_gateway

# Verify services healthy
docker-compose ps

Service Endpoints

Service	Protocol	Port	Auth	Used By
Trading Service	gRPC	50052	No	Rust tests
API Gateway	gRPC	50051	JWT	ghz scripts
Health (Trading)	HTTP	8081	No	test_5
Metrics (Trading)	HTTP	9092	No	test_5
Metrics (Gateway)	HTTP	9091	No	Monitoring

---

Test Details

Rust Test Suite

Test 1: Baseline Latency

• Orders: 1,000 sequential
• Purpose: Single-client latency baseline
• Metrics: P50, P95, P99 latency + throughput

Test 2: Concurrent Connections

• Clients: 100 concurrent
• Orders per client: 100
• Total orders: 10,000
• Purpose: Concurrency stress test
• Metrics: Latency distribution + success rate

Test 3: Sustained Load (Ignored)

• Duration: 5 minutes
• Clients: 50 concurrent
• Target rate: 10,000 orders/sec total
• Purpose: Sustained load validation
• Metrics: Long-term stability

Test 4: Database Performance

• Orders: 5,000
• Purpose: Database write throughput
• Target: >2,000 writes/sec

Test 5: Resource Monitoring

• Purpose: Health + metrics validation
• Checks: HTTP health endpoint, Prometheus metrics
• Requires: health-checks feature

Test 6: Production Readiness

• Clients: 50 concurrent
• Orders per client: 200
• Total orders: 10,000
• Criteria:
  • Success rate >= 99%
  • Throughput >= 5,000 orders/sec
  • P99 latency < 100ms

ghz Authenticated Test Suite

Test 1: Baseline Authenticated Load

• Requests: 1,000
• RPS: 100
• Concurrency: 10
• Purpose: Verify JWT auth + baseline latency
• Expected: 100% success, <50ms P99

Test 2: Medium Authenticated Load

• Requests: 5,000
• RPS: 500
• Concurrency: 50
• Purpose: Medium load with authentication
• Expected: >99% success, <100ms P99

Test 3: High Authenticated Load

• Requests: 10,000
• RPS: 1,000
• Concurrency: 100
• Purpose: High throughput with JWT overhead
• Expected: >95% success, <150ms P99

Test 4: Sustained Authenticated Load

• Duration: 2 minutes
• RPS: 500
• Concurrency: 50
• Total: ~60,000 requests
• Purpose: Long-term stability validation
• Expected: >99% success, stable latency

---

Features

Default (No Features)

• Core gRPC load testing (tests 1-4, 6)
• Dependencies: tokio, tonic, uuid

health-checks (Optional)

cargo test --release --features health-checks

• Enables test_5 (resource monitoring)
• Adds reqwest dependency
• HTTP health + metrics checks

---

Performance Targets

Metric	Target	Typical (Direct)	Typical (Gateway)
Success Rate	>= 99%	99.5-100%	99-100%
Throughput	>= 5K orders/sec	7-10K	5-7K
P50 Latency	< 20ms	10-15ms	15-25ms
P99 Latency	< 100ms	30-50ms	50-100ms
DB Writes/sec	>= 2K	2.5-3K	2-2.5K

Note: API Gateway adds ~5-10ms latency due to JWT validation and proxying.

---

Troubleshooting

"Connection refused" Error

For Rust tests (port 50052):

docker-compose up -d trading_service
docker-compose ps  # Verify "Up" status

For ghz tests (port 50051):

docker-compose up -d api_gateway
docker-compose ps  # Verify "Up" status

"Failed to generate JWT token"

Check JWT_SECRET:

# Verify secret exists
grep JWT_SECRET .env

# If missing, add to .env
echo 'JWT_SECRET=your-secret-key-here' >> .env

"Too many open files" Error

ulimit -n 4096  # Increase file descriptor limit

Authentication Failures (401 errors)

Check token format:

# Generate test token
./tests/e2e_helpers/jwt_token_generator.sh test_user trader

# Verify token has 3 parts (header.payload.signature)

Check API Gateway logs:

docker-compose logs api_gateway | grep -i "auth\|jwt\|401"

High Latency

Check:

1. PostgreSQL synchronous_commit setting
2. Network latency (localhost vs Docker)
3. System load (CPU, memory)
4. API Gateway JWT validation overhead

Optimize PostgreSQL:

-- In PostgreSQL
ALTER SYSTEM SET synchronous_commit = off;
SELECT pg_reload_conf();

---

Compilation Time Comparison

Crate	Dependencies	Compile Time	Speedup
tests/ (original)	36	120-180s	Baseline
tests/load_tests	5	20-30s	6x faster
ghz scripts	N/A	0s	Instant

---

Architecture

Rust Tests (Minimal Dependencies)

[dependencies]
tokio = { workspace = true }           # Async runtime
tonic = { workspace = true }           # gRPC client
tonic-prost = { workspace = true }     # Protobuf runtime
prost = { workspace = true }           # Protobuf types
uuid = { workspace = true }            # Order IDs
reqwest = { optional = true }          # HTTP (feature-gated)

ghz Scripts (Shell + OpenSSL)

# Dependencies
- bash
- ghz (gRPC load testing)
- openssl (JWT signing)
- jq (optional, result parsing)
- nc (netcat, connectivity check)

Build Process

1. build.rs compiles trading.proto from Trading Service
2. Generated code included via tonic::include_proto!("trading")
3. No heavy dependencies (ML, database clients, test frameworks)

---

CI/CD Integration

GitHub Actions

- name: Run Rust Load Tests
  run: |
    docker-compose up -d postgres trading_service
    cd tests/load_tests
    cargo test --release --features health-checks

- name: Run Authenticated ghz Tests
  run: |
    docker-compose up -d api_gateway postgres trading_service
    cd tests/load_tests
    ./ghz_quick_auth_test.sh
    ./ghz_authenticated.sh

GitLab CI

rust_load_tests:
  script:
    - docker-compose up -d postgres trading_service
    - cd tests/load_tests
    - cargo test --release --features health-checks

ghz_load_tests:
  script:
    - docker-compose up -d api_gateway postgres trading_service
    - cd tests/load_tests
    - ./ghz_authenticated.sh

---

Related Documentation

• LOAD_TEST_DEPENDENCY_OPTIMIZATION.md - Detailed analysis
• LOAD_TEST_OPTIMIZATION_SUMMARY.md - Implementation summary
• TESTING_PLAN.md - Overall testing strategy
• WAVE_132_AUTH_VALIDATION_SUMMARY.md - JWT authentication validation

---

Summary

Test Type	Target	Auth	Compilation	Execution	Use Case
Rust Tests	Trading Service (50052)	No	20-30s	Fast	Backend performance
ghz Scripts	API Gateway (50051)	JWT	0s	Fast	End-to-end auth flow

Recommendation: Use both test types for comprehensive validation:

1. Rust tests for backend performance benchmarks
2. ghz scripts for authenticated API Gateway validation

---

Status: ✅ Production Ready
Rust Tests: < 30 seconds compilation ✅
ghz Scripts: Instant execution ✅
JWT Authentication: Fully validated ✅