gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: create/update README.md for all services, CLI, and testing crates

Browse Source 
Create 3 missing service READMEs (api_gateway, data_acquisition_service,
trading_agent_service). Create bin/fxt/README.md. Create
testing/service-integration/README.md. Update existing service and testing
READMEs to standard template. Delete 4 subdirectory READMEs from
testing/integration/ and testing/e2e/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-01 22:50:38 +01:00
parent b57df47ddc
commit e2a0576ef5
19 changed files with 225 additions and 5814 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
bin/fxt/README.md Normal file
View File

@@ -0,0 +1,18 @@
# fxt
Foxhunt CLI -- command-line interface for the Foxhunt HFT trading system.
## Commands
- `fxt auth` -- authentication and token management
- `fxt trade` -- order submission and management
- `fxt train` -- model training job management
- `fxt tune` -- hyperparameter optimization
- `fxt agent` -- trading agent control
- `fxt model` -- model listing and inspection
- `fxt backtest-ml` -- ML backtesting workflows
## Configuration
- `FXT_CONFIG` -- config file path (default: `~/.config/fxt/config.toml`)
- `TRADING_SERVICE_URL` -- gRPC endpoint for trading service

21
services/api_gateway/README.md Normal file
View File

@@ -0,0 +1,21 @@
# api_gateway
API Gateway service with 6-layer authentication, RBAC, rate limiting, and gRPC proxy routing.
## Key Types
- `ApiGatewayService` -- main service implementation
- `RoleBasedAccessControl` -- RBAC authorization
- `RateLimiter` -- 3-tier rate limiting (auth/trading/compute)
- `CircuitBreaker` -- upstream service protection
## Configuration
- `JWT_SECRET` -- JWT signing secret (min 32 chars)
- `UPSTREAM_TRADING_URL` -- trading service gRPC endpoint
- `UPSTREAM_ML_URL` -- ML service gRPC endpoint
## Features
- `mfa` (default) -- multi-factor authentication support
- Build without: `cargo check -p api_gateway --no-default-features --features minimal`

53
services/backtesting_service/README.md
View File

@@ -1,49 +1,28 @@
# Backtesting Service
# backtesting_service
## Overview
Strategy backtesting engine with historical data replay, performance reporting, and results persistence.
The `backtesting_service` offers an independent and isolated environment for rigorously testing and validating trading strategies against historical market data. It provides a robust platform for simulating trading performance, analyzing strategy efficacy, and generating comprehensive performance reports before live deployment.
## Key Types
## Features
- `BacktestingServiceImpl` -- main gRPC service
- `DataReplayEngine` -- historical market data replay
- `PerformanceReporter` -- Sharpe, drawdown, win rate metrics
* **Independent Backtesting Service**: Operates autonomously, allowing for parallel and isolated strategy evaluations.
* **gRPC API for Backtest Execution**: Exposes a clear API for submitting and managing backtesting jobs.
* **Strategy Testing and Validation**: Enables comprehensive testing of various trading strategies under different market conditions.
* **Performance Reporting**: Generates detailed reports including metrics like P&L, Sharpe ratio, drawdown, and win rate.
* **Data Replay Engine**: Accurately replays historical market data, simulating real-world order book dynamics and trade execution.
* **Results Persistence**: Stores backtesting results and reports for historical analysis and comparison.
## gRPC Endpoints
## gRPC API
- `RunBacktest` -- submit backtest configuration and strategy
- `GetBacktestResults` -- retrieve results for completed backtests
- `ListAvailableStrategies` -- list registered strategies
- `GetBacktestReport` -- detailed performance report
The `backtesting_service` exposes a gRPC API for initiating and retrieving backtest results. Key endpoints include:
- `RunBacktest` - Submit backtest configuration and strategy
- `GetBacktestResults` - Retrieve results for completed backtests
- `ListAvailableStrategies` - List registered strategies
- `GetBacktestReport` - Get detailed performance report
## Configuration
## Running the service
To run the `backtesting_service` binary:
```bash
cargo run --bin backtesting_service
```
## Data Requirements
The service requires historical market data in Parquet format:
- Data should be stored in the configured data directory
- Supports tick data, order book snapshots, and OHLCV candles
- Data must include instrument, timestamp, and price/quantity fields
- `GRPC_PORT` -- gRPC listen port
- `DATABASE_URL` -- PostgreSQL connection string
- Historical data directory with Parquet tick/OHLCV files
## Testing
To run the tests for the `backtesting_service` crate:
```bash
cargo test --package backtesting_service
SQLX_OFFLINE=true cargo test -p backtesting_service --lib
```
## Documentation
Comprehensive API documentation is available at [docs.rs/backtesting_service](https://docs.rs/backtesting_service).

1263
services/broker_gateway_service/README.md
View File

File diff suppressed because it is too large Load Diff

15
services/data_acquisition_service/README.md Normal file
View File

@@ -0,0 +1,15 @@
# data_acquisition_service
Automated Databento data downloading, validation, and S3 upload for HFT trading.
## Key Types
- `DataAcquisitionServiceImpl` -- main gRPC service
- `Downloader` -- Databento API client
- `Uploader` -- S3 artifact upload
- `Validator` -- data quality checks
## Configuration
- `DATABENTO_API_KEY` -- Databento API credentials
- `S3_BUCKET` -- target storage bucket

43
services/ml_training_service/README.md
View File

@@ -1,48 +1,27 @@
# ml_training_service
Model training orchestration and lifecycle management for the Foxhunt HFT trading system. Manages training jobs for DQN, PPO, TFT, Mamba2, TLOB, and Liquid models with progress tracking, resource allocation, and model artifact storage.
Model training orchestration and lifecycle management for DQN, PPO, TFT, Mamba2, TLOB, and Liquid models with progress tracking and artifact storage.
## Building
## Key Types
```bash
# Default (minimal features)
cargo build --release -p ml_training_service
# With GPU acceleration (requires CUDA)
cargo build --release -p ml_training_service --features gpu
# With mock training data (testing only, bypasses database)
cargo build --release -p ml_training_service --features mock-data
```
- `MlTrainingServiceImpl` -- main gRPC service
- `JobTracker` -- training job state machine
- `CheckpointManager` -- model artifact persistence
## Features
| Feature | Default | Description |
|-------------|---------|--------------------------------------------------|
| `minimal` | Yes | Minimal ML feature set for financial models |
| `gpu` | No | SIMD GPU acceleration (requires CUDA) |
| `debug` | No | Additional debug logging |
| `mock-data` | No | Use mock training data instead of PostgreSQL |
- `minimal` (default) -- minimal ML feature set for financial models
- `gpu` -- SIMD GPU acceleration (requires CUDA)
- `mock-data` -- mock training data (testing, bypasses database)
## Configuration
The gRPC listen port is set via the `GRPC_PORT` environment variable. Prometheus metrics are exposed on port 9094.
PostgreSQL (via sqlx) is used for job metadata, training history, and state management. Set the connection string with `DATABASE_URL`.
## Running
```bash
GRPC_PORT=50053 DATABASE_URL="postgresql://user:pass@localhost:5432/foxhunt_training" \
./target/release/ml_training_service serve
```
- `GRPC_PORT` -- gRPC listen port
- `DATABASE_URL` -- PostgreSQL for job metadata and training history
- Prometheus metrics on port 9094
## Testing
```bash
# Unit tests (offline, no database required)
SQLX_OFFLINE=true cargo test -p ml_training_service --lib
# Integration tests (requires running PostgreSQL)
cargo test -p ml_training_service
```

16
services/trading_agent_service/README.md Normal file
View File

@@ -0,0 +1,16 @@
# trading_agent_service
Portfolio management with autonomous universe selection, asset allocation, and order generation.
## Key Types
- `TradingAgentServiceImpl` -- main gRPC service
- `AutonomousUniverseManager` -- dynamic universe scaling
- `AssetSelector` -- instrument selection
- `Allocator` -- position sizing and allocation
- `OrderGenerator` -- order creation from signals
## Configuration
- `ML_SERVICE_URL` -- ensemble ML service endpoint
- `TRADING_SERVICE_URL` -- order execution endpoint

88
services/trading_service/README.md
View File

@@ -1,82 +1,32 @@
# Trading Service
# trading_service
## Overview
Core execution engine for order placement, position keeping, risk integration, and real-time P&L tracking.
The `trading_service` is the core execution engine for the Foxhunt HFT platform. It manages the entire lifecycle of trading operations, from order placement and execution to real-time position keeping and risk management. This service is critical for high-frequency, low-latency trading activities, ensuring compliance and optimal performance.
## Key Types
## Features
- `TradingServiceImpl` -- main gRPC service
- `OrderExecutor` -- high-throughput order handling
- `PositionManager` -- real-time position and P&L tracking
- `RiskIntegration` -- pre-trade and post-trade compliance
* **Order Execution**: Handles high-throughput order placement, modification, and cancellation across various exchanges.
* **Position Management**: Maintains real-time tracking of all open positions, including P&L calculations and exposure.
* **Risk Integration**: Integrates with upstream risk systems to enforce pre-trade and post-trade compliance checks.
* **Compliance Checks**: Automatically applies regulatory and internal compliance rules to all trading activities.
* **Market Data Subscriptions**: Subscribes to and processes real-time market data feeds for informed decision-making.
* **Real-time P&L Tracking**: Provides immediate profit and loss updates for active strategies and overall portfolio.
* **Health Checks and Metrics**: Exposes endpoints for monitoring service health and operational metrics.
## gRPC Endpoints
## gRPC API
The `trading_service` exposes a gRPC API for interacting with its core functionalities. Key endpoints include:
- `PlaceOrder` - Submit new orders
- `CancelOrder` - Cancel existing orders
- `GetPosition` - Query current positions
- `SubscribeMarketData` - Subscribe to market data feeds
- `GetPnlUpdates` - Retrieve real-time P&L updates
## Running the service
To run the `trading_service` binary:
```bash
cargo run --bin trading_service
```
- `PlaceOrder` -- submit new orders
- `CancelOrder` -- cancel existing orders
- `GetPosition` -- query current positions
- `SubscribeMarketData` -- market data feeds
- `GetPnlUpdates` -- real-time P&L updates
## Configuration
The service is configured via the central `config` crate with PostgreSQL backend. Key configuration includes:
- Database connection strings
- Risk parameters and limits
- Broker connection settings
- gRPC server port and TLS settings
### TLS Configuration (Agent S3)
The Trading Service supports TLS 1.3 with optional mutual TLS (mTLS) for secure gRPC communications.
**Environment Variables**:
```bash
TLS_ENABLED=false # Enable TLS (default: false)
TLS_CERT_PATH=/app/certs/trading_service/server.crt # Server certificate
TLS_KEY_PATH=/app/certs/trading_service/server.key # Server private key
TLS_CA_PATH=/app/certs/trading_service/ca.crt # CA certificate
TLS_REQUIRE_CLIENT_CERT=false # Require client certs (default: false)
```
**Certificate Directory Structure**:
```
/app/certs/trading_service/
├── server.crt # Server certificate
├── server.key # Server private key
└── ca.crt # CA certificate for client verification
```
**Features**:
- TLS 1.3 encryption for all gRPC traffic
- Mutual TLS (mTLS) support for client certificate authentication
- 6-layer certificate validation (expiration, purpose, constraints, extensions, SANs, revocation)
- Role-based access control (RBAC) via certificate Organizational Unit (OU)
- CRL (Certificate Revocation List) support
**Security Note**: TLS is disabled by default for development. Enable `TLS_ENABLED=true` for production deployments.
- `GRPC_PORT` -- gRPC listen port
- `DATABASE_URL` -- PostgreSQL connection string
- `TLS_ENABLED` -- enable TLS 1.3 (default: false)
- `TLS_CERT_PATH`, `TLS_KEY_PATH`, `TLS_CA_PATH` -- TLS certificates
- `TLS_REQUIRE_CLIENT_CERT` -- mutual TLS (default: false)
## Testing
To run the tests for the `trading_service` crate:
```bash
cargo test --package trading_service
SQLX_OFFLINE=true cargo test -p trading_service --lib
```
## Documentation
Comprehensive API documentation is available at [docs.rs/trading_service](https://docs.rs/trading_service).

320
testing/api-gateway-load/README.md
View File

@@ -1,316 +1,24 @@
# API Gateway Load Testing Framework
# api-gateway-load
Comprehensive load testing infrastructure for validating API Gateway performance under high concurrency.
Load testing framework for API Gateway with 4 scenarios: normal (1K clients), spike (10K ramp), sustained (24h), and stress (incremental to failure).
## Overview
## Key Types
This framework provides four test scenarios with detailed metrics collection, time-series analysis, and HTML report generation:
- `TestOrchestrator` -- scenario coordination
- `VirtualClientPool` -- authenticated HTTP client generation
- `MetricsCollector` -- HDR histogram latency tracking
- `ReportGenerator` -- HTML + SVG chart output
1. **Normal Load**: 1K concurrent clients for 60 seconds
2. **Spike Load**: 0→10K clients in 10s, sustain 60s
3. **Sustained Load**: 100 clients for 24 hours (endurance test)
4. **Stress Test**: Incrementally increase load until failure
## Scenarios
## Architecture
```
┌─────────────────────┐
│ Test Orchestrator │
└──────────┬──────────┘
│
├─────────────────┬─────────────────┬─────────────────┐
│ │ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ Normal │ │ Spike │ │ Sustained │ │ Stress │
│ Load │ │ Load │ │ Load │ │ Test │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │ │
└─────────────────┴─────────────────┴─────────────────┘
│
┌───────────▼───────────┐
│ Virtual Client Pool │
│ (Authenticated HTTP │
│ + Mixed Workload) │
└───────────┬───────────┘
│
┌───────────▼───────────┐
│ Metrics Collector │
│ - HDR Histogram │
│ - Time Series Data │
│ - Per-Service Stats │
└───────────┬───────────┘
│
┌───────────▼───────────┐
│ Report Generator │
│ - HTML + SVG Charts │
│ - Capacity Analysis │
└───────────────────────┘
```
## Installation
```bash
cd /home/jgrusewski/Work/foxhunt/services/api_gateway/load_tests
cargo build --release
```
- `normal` -- 1K concurrent clients, 60s, target <10ms P99
- `spike` -- 0 to 10K in 10s, sustain 60s
- `sustained` -- 100 clients, 24h endurance
- `stress` -- incremental increase until failure
## Usage
### Run Individual Scenarios
```bash
# Normal load test (1K clients, 60s)
cargo run --release -- normal \
--gateway-url http://localhost:50050 \
--num-clients 1000 \
--duration-secs 60
# Spike load test (0→10K in 10s, sustain 60s)
cargo run --release -- spike \
--gateway-url http://localhost:50050 \
--target-clients 10000 \
--ramp-up-secs 10 \
--sustain-secs 60
# Sustained load test (100 clients, 24h)
cargo run --release -- sustained \
--gateway-url http://localhost:50050 \
--num-clients 100 \
--duration-secs 86400
# Stress test (incrementally increase until failure)
cargo run --release -- stress \
--gateway-url http://localhost:50050 \
--initial-clients 100 \
--increment 100 \
--increment-interval-secs 60 \
--max-p99-latency-ms 50.0 \
--max-error-rate-pct 5.0
cargo run --release -p api-gateway-load -- normal --gateway-url http://localhost:50050
cargo run --release -p api-gateway-load -- all --gateway-url http://localhost:50050
```
### Run All Scenarios
```bash
cargo run --release -- all --gateway-url http://localhost:50050
```
## Metrics Collected
### Latency Statistics
- **Min/Max/Mean**: Full latency range
- **Percentiles**: P50, P90, P95, P99, P99.9
- **Standard Deviation**: Latency consistency
### Request Breakdown
- **Total Requests**: Aggregate count
- **Successful**: 2xx responses
- **Failed**: 4xx/5xx errors
- **Timeout**: Connection/request timeouts
- **Rate Limited**: 429 responses
- **Circuit Breaker**: 503 responses
### Time Series Data (1-second intervals)
- Requests per second (RPS)
- P99 latency
- Error rate percentage
- Active client count
### Per-Service Statistics
- **Trading Service**: Order submission, position queries
- **Backtesting Service**: Backtest execution
- **ML Training Service**: Model training requests
## Workload Distribution
Mixed workload simulates realistic usage:
- **60%** - Order submissions
- **30%** - Position queries
- **8%** - Backtesting requests
- **2%** - ML training requests
Each client has random think time (1-50ms) between requests to simulate human behavior.
## Report Generation
HTML reports are automatically generated with:
1. **Summary Cards**: Total requests, RPS, error rate, P99 latency
2. **Latency Table**: All percentiles with statistics
3. **Request Breakdown**: Success/failure categorization
4. **Performance Charts** (SVG):
- Requests per second over time
- P99 latency over time
- Error rate over time
5. **Capacity Recommendations**: Based on observed performance
## Success Criteria
### Normal Load
- **Target**: <10ms P99 latency, 0% errors
- **Pass**: Error rate < 1%, P99 < 10ms
- **Fail**: Error rate ≥ 5%, P99 ≥ 50ms
### Spike Load
- **Target**: Graceful handling, circuit breakers activate
- **Pass**: Error rate < 10%, circuit breakers respond correctly
- **Fail**: System crashes, uncontrolled cascading failures
### Sustained Load
- **Target**: No memory leaks, stable latency
- **Pass**: Latency drift < 5%, error rate stddev < 2%
- **Fail**: Latency increases > 10%, memory exhaustion
### Stress Test
- **Target**: Identify capacity limits
- **Pass**: Breaking point identified with clear bottleneck
- **Fail**: Undefined behavior, data corruption
## Example Report Output
```
Load Test Report: Normal Load Test
Test Period: 2025-10-03 12:00:00 to 2025-10-03 12:01:00
Duration: 60 seconds (0.02 hours)
Summary:
┌──────────────────┬──────────┐
│ Total Requests │ 120,000 │
│ Requests/Second │ 2,000 │
│ Error Rate │ 0.12% │
│ P99 Latency │ 8.5ms │
└──────────────────┴──────────┘
Latency Statistics:
┌──────────┬──────────┐
│ P50 │ 3.2ms │
│ P90 │ 5.1ms │
│ P95 │ 6.8ms │
│ P99 │ 8.5ms │
│ P99.9 │ 12.3ms │
└──────────┴──────────┘
Capacity Recommendation:
✓ System handled 1,000 clients with 0.12% error rate
✓ P99 latency well within 10ms target
✓ Safe for production at this load level
```
## Load Generator Resources
### Requirements
- **CPU**: 4+ cores for 1K clients, 8+ cores for 10K clients
- **Memory**: 2GB for 1K clients, 8GB for 10K clients
- **Network**: Low-latency connection to API Gateway
### Monitoring Load Generator
The framework monitors its own resource usage to ensure the load generator doesn't become a bottleneck. If you see warnings about load generator CPU/memory, consider:
1. Running on a larger machine
2. Distributing load across multiple generators
3. Reducing concurrent client count
## Advanced Configuration
### Custom JWT Token
Set authentication parameters in the code:
```rust
let auth_client = AuthenticatedClient::new(
gateway_url,
"your-jwt-secret",
"user-id",
"username"
).await?;
```
### Custom Test Duration
All scenarios support custom durations:
```bash
# Extended normal load test (5 minutes)
cargo run --release -- normal --duration-secs 300
# Long-running stress test
cargo run --release -- stress --increment-interval-secs 300
```
## Troubleshooting
### Connection Refused
```
Error: Connection refused (os error 111)
```
**Solution**: Ensure API Gateway is running at `http://localhost:50050`
### Too Many Open Files
```
Error: Too many open files (os error 24)
```
**Solution**: Increase system file descriptor limit:
```bash
ulimit -n 10000
```
### Memory Exhaustion
```
Error: Cannot allocate memory
```
**Solution**: Reduce `--num-clients` or run on a larger machine
### High Latency from Generator
```
Warning: Load generator CPU > 80%, results may be unreliable
```
**Solution**: Use a more powerful machine or reduce client count
## Integration with CI/CD
### GitHub Actions Example
```yaml
name: Load Testing
on:
schedule:
- cron: '0 0 * * 0' # Weekly
jobs:
load-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Start API Gateway
run: |
docker-compose up -d api_gateway
sleep 10
- name: Run Load Tests
run: |
cd services/api_gateway/load_tests
cargo run --release -- normal
- name: Upload Reports
uses: actions/upload-artifact@v3
with:
name: load-test-reports
path: |
normal_load_report.html
*.svg
```
## Performance Baseline
Expected results for reference hardware (AWS c5.4xlarge):
| Scenario | Clients | RPS | P99 Latency | Error Rate |
|----------------|---------|-------|-------------|------------|
| Normal Load | 1,000 | 2,000 | 8ms | <0.1% |
| Spike Load | 10,000 | 8,000 | 25ms | <2% |
| Sustained Load | 100 | 200 | 5ms | <0.01% |
| Stress Test | 5,000 | 5,000 | 45ms | Breaking |
## License
Part of the Foxhunt HFT Trading System - MIT OR Apache-2.0

403
testing/e2e/README.md
View File

@@ -1,398 +1,23 @@
# Foxhunt E2E Testing Framework
# foxhunt-e2e
A comprehensive End-to-End testing framework for the Foxhunt High-Frequency Trading system. This framework tests the complete integration between TLI client, all three services (Trading, Backtesting, ML Training), database interactions, ML model inference, and complete trading workflows.
End-to-end testing framework covering service orchestration, gRPC clients, database integration, ML pipeline, and complete trading workflows.
## 🎯 Overview
## Key Types
The E2E testing framework provides:
- `E2ETestFramework` -- core orchestrator
- `ServiceManager` -- automated service startup/shutdown
- `DatabaseTestHarness` -- transaction-isolated DB testing
- `MLPipelineTestHarness` -- mock ML model inference
- **Service Orchestration**: Automated startup/shutdown of all services
- **gRPC Client Testing**: Authentication, streaming, and error handling
- **Database Integration**: Transaction management and configuration hot-reload
- **ML Pipeline Testing**: Model inference, training, and ensemble predictions
- **Complete Workflow Testing**: End-to-end trading scenarios
- **Performance Benchmarking**: Load testing and performance metrics
- **Corrode-MCP Integration**: Advanced test execution and reporting
## Test Categories
## 🏗️ Architecture
- Service startup, shutdown, recovery
- gRPC client connections, streaming, error handling
- ML inference, training, ensemble predictions
- Trading workflows, order lifecycle, risk, emergency stop
```
tests/e2e/
├── Cargo.toml # Project configuration
├── build.rs # gRPC proto compilation
├── src/
│ ├── lib.rs # Main library and test macros
│ ├── framework.rs # Core E2E testing framework
│ ├── services.rs # Service management and orchestration
│ ├── clients.rs # gRPC test clients
│ ├── database.rs # Database testing harness
│ ├── ml_pipeline.rs # ML model testing framework
│ ├── workflows.rs # Complete trading workflow tests
│ ├── utils.rs # Test utilities and data generation
│ ├── corrode.rs # Corrode-MCP integration
│ └── bin/
│ ├── test_runner.rs # Test execution runner
│ └── service_orchestrator.rs # Service management tool
├── tests/
│ └── integration_test.rs # Example integration tests
└── README.md # This file
```
## 🚀 Quick Start
### Prerequisites
1. **Rust Toolchain**: Ensure you have Rust 1.75+ installed
2. **PostgreSQL**: Running instance for database tests
3. **Corrode-MCP**: Install corrode for advanced test execution
## Usage
```bash
# Install corrode-mcp (if not already installed)
cargo install corrode-mcp
# Set up environment
export DATABASE_URL="postgresql://localhost/foxhunt_test"
export RUST_LOG="info"
SQLX_OFFLINE=true cargo test -p foxhunt-e2e --lib
```
### Running Tests
#### Option 1: Using Test Runner (Recommended)
```bash
# Build the test runner
cargo build --bin test_runner --release
# Run all E2E tests
./target/release/test_runner run --test all
# Run specific test categories
./target/release/test_runner run --test trading --parallel 2
./target/release/test_runner run --test ml --verbose
./target/release/test_runner run --test smoke --fail-fast
# List available tests
./target/release/test_runner list
# Generate test report
./target/release/test_runner report --results-dir ./test-results --format html
```
#### Option 2: Using Service Orchestrator
```bash
# Build the service orchestrator
cargo build --bin service_orchestrator --release
# Start all services for testing
./target/release/service_orchestrator start --services all --wait
# Check service status
./target/release/service_orchestrator status
# Run specific tests against running services
cargo test --package foxhunt-e2e
# Stop services when done
./target/release/service_orchestrator stop --services all
```
#### Option 3: Direct Cargo Testing
```bash
# Run all integration tests
cargo test --package foxhunt-e2e
# Run specific test
cargo test --package foxhunt-e2e test_complete_trading_workflow
# Run with output
cargo test --package foxhunt-e2e -- --nocapture
```
## 📋 Test Categories
### 🔧 Service Tests
- **service_startup**: Verify all services start and respond to health checks
- **service_shutdown**: Test graceful service shutdown
- **service_recovery**: Test service recovery after failures
### 🗄️ Database Tests
- **database_integration**: Test PostgreSQL integration and queries
- **database_migrations**: Test database schema migrations
- **database_performance**: Test database query performance
### 📡 gRPC Tests
- **grpc_clients**: Test all gRPC client connections and authentication
- **grpc_streaming**: Test streaming gRPC calls (market data, order updates)
- **grpc_error_handling**: Test gRPC error scenarios and recovery
### 🤖 ML Pipeline Tests
- **ml_inference**: Test ML model inference pipelines
- **ml_training**: Test ML model training workflows
- **ml_ensemble**: Test ensemble prediction workflows
### 💼 Trading Tests
- **trading_workflows**: Complete trading workflow tests
- **order_lifecycle**: Order submission to execution lifecycle
- **risk_management**: Risk management and safety mechanisms
- **emergency_stop**: Emergency stop and kill switch tests
### 🎯 Full Suite
- **all**: Run complete E2E test suite
- **smoke**: Run smoke tests for quick validation
- **performance**: Run performance and load tests
## 🛠️ Framework Components
### E2ETestFramework
The core framework that orchestrates all components:
```rust
use foxhunt_e2e::{e2e_test, framework::E2ETestFramework};
e2e_test!(my_test, |framework: E2ETestFramework| async {
// Your test logic here
let tli_client = framework.get_tli_client().await?;
let health = framework.check_services_health().await?;
assert!(health.all_healthy);
Ok(())
});
```
### Service Management
Automated service lifecycle management:
```rust
use foxhunt_e2e::services::ServiceManager;
let mut manager = ServiceManager::new();
manager.start_all_services().await?;
// Tests run here
manager.stop_all_services().await?;
```
### gRPC Clients
Type-safe gRPC client implementations:
```rust
use foxhunt_e2e::clients::{TradingServiceClient, MLTrainingServiceClient};
let mut trading = TradingServiceClient::new("http://localhost:50051").await?;
let portfolio = trading.get_portfolio().await?;
let mut ml = MLTrainingServiceClient::new("http://localhost:50053").await?;
let prediction = ml.predict(features).await?;
```
### Database Testing
Transaction-isolated database testing:
```rust
use foxhunt_e2e::database::DatabaseTestHarness;
let db = DatabaseTestHarness::new().await?;
let mut tx = db.begin_test_transaction().await?;
// Database operations here - will auto-rollback
```
### ML Pipeline Testing
Mock ML models for testing:
```rust
use foxhunt_e2e::ml_pipeline::MLPipelineTestHarness;
let ml = MLPipelineTestHarness::new().await?;
let result = ml.test_model_inference("mamba", features).await?;
let ensemble = ml.test_ensemble_prediction(features).await?;
```
## 🎛️ Configuration
### Environment Variables
- `DATABASE_URL`: PostgreSQL connection string for test database
- `RUST_LOG`: Log level (debug, info, warn, error)
- `FOXHUNT_TEST_MODE`: Set to "true" for test mode
- `CUDA_VISIBLE_DEVICES`: GPU configuration for ML tests
- `TORCH_DEVICE`: PyTorch device (cpu/cuda) for ML tests
### Test Configuration
```toml
# tests/e2e/Cargo.toml
[package.metadata.e2e]
default_timeout = 600
max_parallel_sessions = 4
service_startup_timeout = 120
database_url = "postgresql://localhost/foxhunt_test"
```
## 📊 Performance Benchmarks
The framework includes comprehensive performance testing:
### Order Submission Performance
- Target: >10 orders/second
- Success rate: >90%
- Latency: <100ms average
### ML Inference Performance
- Target: >20 inferences/second
- Latency: <50ms average
- GPU utilization monitoring
### Database Performance
- Query execution time monitoring
- Connection pool performance
- Transaction throughput
## 🔍 Debugging and Troubleshooting
### Enable Debug Logging
```bash
export RUST_LOG=debug
cargo test --package foxhunt-e2e -- --nocapture
```
### Service Logs
```bash
# View service logs
./target/release/service_orchestrator logs trading --follow
# Check service status
./target/release/service_orchestrator status
```
### Database Issues
```bash
# Check database connection
psql $DATABASE_URL -c "SELECT 1;"
# Reset test database
dropdb foxhunt_test && createdb foxhunt_test
```
### Common Issues
1. **Service startup timeouts**: Increase `startup_timeout` in service configs
2. **gRPC connection errors**: Verify services are running and ports are correct
3. **Database connection failures**: Check PostgreSQL is running and credentials
4. **ML model loading errors**: Ensure model files exist or use mock models
## 🧪 Writing Custom Tests
### Basic Test Structure
```rust
use foxhunt_e2e::{e2e_test, framework::E2ETestFramework};
use anyhow::Result;
e2e_test!(test_my_feature, |framework: E2ETestFramework| async {
// Test setup
let client = framework.get_tli_client().await?;
// Test execution
let result = client.my_operation().await?;
// Assertions
assert!(result.success, "Operation failed");
// Cleanup (automatic)
Ok(())
});
```
### Advanced Test Features
```rust
e2e_test!(test_complex_workflow, |framework: E2ETestFramework| async {
// Use test data generator
let mut generator = TestDataGenerator::new();
let market_data = generator.generate_market_data()?;
// Measure performance
let (result, duration) = TestUtils::measure_execution_time(|| async {
// Your operation here
Ok(42)
}).await?;
// Database testing
let db = &framework.database_harness;
let mut tx = db.begin_test_transaction().await?;
// Database operations...
// ML testing
let ml = &framework.ml_pipeline;
let prediction = ml.test_ensemble_prediction(features).await?;
Ok(())
});
```
## 📈 Continuous Integration
### GitHub Actions Example
```yaml
name: E2E Tests
on: [push, pull_request]
jobs:
e2e-tests:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: foxhunt_test
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v3
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
- name: Install corrode-mcp
run: cargo install corrode-mcp
- name: Run E2E tests
env:
DATABASE_URL: postgresql://postgres:postgres@localhost/foxhunt_test
RUST_LOG: info
run: |
cargo build --bin service_orchestrator --release
./target/release/service_orchestrator start --services all --wait --background &
sleep 10
cargo test --package foxhunt-e2e
```
## 🤝 Contributing
1. **Add new tests**: Create new test functions using the `e2e_test!` macro
2. **Extend framework**: Add new components to the framework modules
3. **Improve performance**: Optimize test execution and resource usage
4. **Documentation**: Update this README and code documentation
### Test Naming Convention
- `test_[component]_[scenario]`: e.g., `test_trading_order_lifecycle`
- Use descriptive names that explain what is being tested
- Group related tests in the same file
### Code Style
- Follow Rust standard formatting (`cargo fmt`)
- Add comprehensive error handling
- Include informative log messages
- Write clear assertions with descriptive failure messages
## 📝 License
This E2E testing framework is part of the Foxhunt HFT Trading System and follows the same license terms as the main project.

386
testing/e2e/integration/README.md
View File

@@ -1,386 +0,0 @@
# Foxhunt E2E Integration Test Suite
Comprehensive end-to-end integration tests for the Foxhunt HFT Trading System.
## 🚀 Quick Start
```bash
# 1. Start PostgreSQL
docker run -d --name foxhunt-postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=foxhunt \
-p 5433:5432 postgres:15
# 2. Run all tests
export DATABASE_URL="postgresql://postgres:postgres@localhost:5433/foxhunt"
./e2e_test_suite.sh
```
## 📋 Test Suite Overview
| Test | Description | Duration | Status |
|------|-------------|----------|--------|
| **auth_flow_test.sh** | Full authentication flow (JWT + MFA + RBAC) | ~12s | ✅ Complete |
| **trading_flow_test.sh** | Complete trading lifecycle | ~8s | ✅ Complete |
| **hot_reload_test.sh** | Configuration hot-reload (NOTIFY) | ~6s | ✅ Complete |
| **backtesting_flow_test.sh** | Backtesting workflow | TBD | 🚧 Future |
| **ml_training_flow_test.sh** | ML training workflow | TBD | 🚧 Future |
## 🔐 Test 1: Authentication Flow
**Tests**: Login → JWT → MFA → RBAC → Authenticated Request
### What It Validates
- ✅ Trading service accessibility
- ✅ User creation with bcrypt password hashing
- ✅ JWT token generation (HS256 signature)
- ✅ TOTP/MFA infrastructure
- ✅ RBAC permission validation
- ✅ Token expiration and structure
- ✅ Security (invalid credential rejection)
- ✅ Authentication audit trail
### Usage
```bash
./auth_flow_test.sh
```
### Output
Generates JWT token stored in:
- `/tmp/foxhunt_test_token_<PID>`
- `$FOXHUNT_JWT_TOKEN` environment variable
## 💹 Test 2: Trading Flow
**Tests**: Order Submission → Risk Checks → Execution → Position Update
### What It Validates
- ✅ Order submission validation
- ✅ Pre-trade risk limits (max order size)
- ✅ Order execution simulation
- ✅ Position calculation (quantity + avg price)
- ✅ Complete audit trail
- ✅ SOX compliance validation
### Usage
```bash
./trading_flow_test.sh
```
### Trade Lifecycle
```
Order Submit → Risk Check → Execution → Position Update → Audit
```
## 🔄 Test 3: Configuration Hot-Reload
**Tests**: Config Update → PostgreSQL NOTIFY → Service Reload
### What It Validates
- ✅ PostgreSQL NOTIFY/LISTEN mechanism
- ✅ Configuration change detection
- ✅ Hot-reload without service restart
- ✅ Change history audit trail
- ✅ Connection stability
- ✅ Performance (< 100ms latency)
### Usage
```bash
./hot_reload_test.sh
```
### Hot-Reload Flow
```
UPDATE config → Trigger → NOTIFY → Service → Reload
```
## 🎯 Master Test Suite
**Script**: `e2e_test_suite.sh`
Orchestrates all tests with:
- Pre-flight checks (database, tools)
- Sequential test execution
- Result aggregation and reporting
- Colored output with progress tracking
### Usage
```bash
# Run all tests
./e2e_test_suite.sh
# With custom configuration
export TRADING_SERVICE_HOST="localhost"
export TRADING_SERVICE_PORT="50051"
export DATABASE_URL="postgresql://postgres:postgres@localhost:5433/foxhunt"
./e2e_test_suite.sh
```
### Expected Output
```
╔════════════════════════════════════════════════════════════════╗
║ Foxhunt HFT E2E Integration Test Suite ║
╚════════════════════════════════════════════════════════════════╝
[Test 1] Full Authentication Flow
═══════════════════════════════════════════════════════
✅ Full Authentication Flow PASSED (12s)
[Test 2] Complete Trading Flow
═══════════════════════════════════════════════════════
✅ Complete Trading Flow PASSED (8s)
[Test 3] Configuration Hot-Reload
═══════════════════════════════════════════════════════
✅ Configuration Hot-Reload PASSED (6s)
╔════════════════════════════════════════════════════════════════╗
║ Test Suite Summary ║
╚════════════════════════════════════════════════════════════════╝
Total Tests: 3
Passed: 3
Failed: 0
Skipped: 0
Success Rate: 100.0%
╔════════════════════════════════════════════════════════════════╗
║ ALL E2E TESTS PASSED! ✅ ║
╚════════════════════════════════════════════════════════════════╝
```
## 🔧 Prerequisites
### Required Services
1. **PostgreSQL Database** (port 5433)
```bash
docker run -d --name foxhunt-postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=foxhunt \
-p 5433:5432 postgres:15
```
2. **Trading Service** (optional for full tests)
```bash
DATABASE_URL=postgresql://postgres:postgres@localhost:5433/foxhunt \
cargo run --bin trading_service
```
### Required Tools
```bash
# Ubuntu/Debian
sudo apt-get install -y \
postgresql-client \
bc \
jq \
netcat-openbsd \
oath-toolkit
# Verify
psql --version
bc --version
jq --version
nc -h
oathtool --version
```
## 🌍 Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `TRADING_SERVICE_HOST` | localhost | Trading service hostname |
| `TRADING_SERVICE_PORT` | 50051 | Trading service gRPC port |
| `DATABASE_URL` | postgresql://postgres:postgres@localhost:5433/foxhunt | PostgreSQL connection string |
| `REDIS_URL` | redis://localhost:6379 | Redis connection string (optional) |
## 📊 Performance Benchmarks
| Test | Duration | Target |
|------|----------|--------|
| Authentication Flow | ~12s | < 15s |
| Trading Flow | ~8s | < 10s |
| Hot-Reload | ~6s | < 10s |
| **Total Suite** | **~26s** | **< 60s** |
### Hot-Reload Latency
| Operation | Latency |
|-----------|---------|
| Config Update | < 50ms |
| NOTIFY Propagation | < 10ms |
| Service Reload | < 40ms |
| **End-to-End** | **< 100ms** |
## 🛡️ Security & Compliance
### Security Features
- ✅ Bcrypt password hashing (cost factor 12)
- ✅ JWT with HMAC-SHA256 signing
- ✅ TOTP/MFA infrastructure
- ✅ SQL injection prevention (parameterized queries)
- ✅ Token expiration validation
### Regulatory Compliance
- ✅ **SOX**: Immutable audit trails with timestamps
- ✅ **MiFID II**: Order lifecycle tracking
- ✅ **Data Retention**: Configurable retention policies
## 🐛 Troubleshooting
### Database Connection Failed
```bash
# Check PostgreSQL is running
docker ps | grep postgres
# Start if needed
docker run -d --name foxhunt-postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=foxhunt \
-p 5433:5432 postgres:15
```
### Trading Service Not Accessible
```bash
# Check if running
ps aux | grep trading_service
# Start if needed
DATABASE_URL=postgresql://postgres:postgres@localhost:5433/foxhunt \
cargo run --bin trading_service
```
### Missing Tools
```bash
# Install all required tools
sudo apt-get install -y postgresql-client bc jq netcat-openbsd oath-toolkit
```
### Permission Denied
```bash
# Make scripts executable
chmod +x *.sh
```
## 📚 Documentation
- **Full Documentation**: `/home/jgrusewski/Work/foxhunt/docs/WAVE75_AGENT11_E2E_TESTING.md`
- **Project Instructions**: `/home/jgrusewski/Work/foxhunt/CLAUDE.md`
## 🔮 Future Enhancements
### Planned Tests
1. **Backtesting Flow**
- Strategy creation
- Backtest execution
- Results retrieval
2. **ML Training Flow**
- Training job submission
- Model deployment
- Inference validation
3. **WebSocket Streaming**
- Real-time market data
- Order events
- Position updates
4. **Kill Switch**
- Emergency shutdown
- State synchronization
- Service recovery
5. **Load Testing**
- Concurrent orders
- Rate limiting
- Performance degradation
### CI/CD Integration
```yaml
# GitHub Actions example
- name: Run E2E Tests
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5433/foxhunt
run: |
cd tests/e2e/integration
./e2e_test_suite.sh
```
## ✅ Success Criteria
- ✅ All core tests passing (3/5 implemented)
- ✅ Complete trading flow validated
- ✅ Inter-service communication working
- ✅ Hot-reload functional (< 100ms)
- ✅ Audit trails persisted correctly
- ✅ SOX compliance validated
- ✅ Security measures validated
## 📝 Files
```
/home/jgrusewski/Work/foxhunt/tests/e2e/integration/
├── e2e_test_suite.sh # Master orchestration script
├── auth_flow_test.sh # Authentication test
├── trading_flow_test.sh # Trading flow test
├── hot_reload_test.sh # Hot-reload test
└── README.md # This file
```
## 🎓 Usage Examples
### Run Single Test
```bash
./auth_flow_test.sh
```
### Run with Custom Database
```bash
export DATABASE_URL="postgresql://user:pass@host:5432/dbname"
./e2e_test_suite.sh
```
### Debug Mode
```bash
set -x # Enable debug output
./e2e_test_suite.sh
```
### Continuous Testing
```bash
# Run tests every 5 minutes
while true; do
./e2e_test_suite.sh
sleep 300
done
```
---
**Wave 75 Agent 11 - End-to-End Integration Testing**
**Status**: ✅ Production Ready
**Date**: 2025-10-03

1524
testing/integration/README.md
View File

File diff suppressed because it is too large Load Diff

292
testing/integration/chaos/README.md
View File

@@ -1,292 +0,0 @@
# Foxhunt Chaos Engineering Framework
A comprehensive chaos engineering framework specifically designed for high-frequency trading (HFT) systems with sub-100ms recovery requirements.
## 🎯 Overview
This framework provides systematic failure injection and recovery validation for the Foxhunt HFT trading system, focusing on:
- **MLTrainingService Resilience**: Kill/restart scenarios with checkpoint recovery
- **Performance Validation**: Sub-100ms recovery time requirements
- **Failure Injection**: Network, memory, GPU, disk, and database failures
- **Automated Testing**: Nightly chaos job scheduling with reporting
- **HFT-Specific Requirements**: Ultra-low latency validation and monitoring
## 🏗️ Architecture
```
tests/chaos/
├── chaos_framework.rs # Core chaos orchestration engine
├── ml_training_chaos.rs # ML-specific chaos tests
├── nightly_chaos_runner.rs # Automated scheduling and execution
├── chaos_cli.rs # Command-line interface
├── examples/
│ ├── usage_examples.rs # Comprehensive usage examples
│ └── chaos_config.toml # Configuration template
└── README.md # This file
```
## 🚀 Quick Start
### 1. Basic ML Service Kill Test
```rust
use foxhunt_tests::chaos::*;
#[tokio::main]
async fn main() -> Result<()> {
// Initialize chaos framework
let runner = initialize_foxhunt_chaos().await?;
// Run quick ML service resilience test
let results = run_quick_chaos_test().await?;
println!("Chaos test results: {:?}", results);
Ok(())
}
```
### 2. Command Line Usage
```bash
# Run a single process kill experiment
cargo run --bin foxhunt-chaos run --experiment-type process-kill --service ml_training_service
# Run comprehensive ML test suite
cargo run --bin foxhunt-chaos ml-suite --endpoint http://localhost:8080 --generate-report
# Start nightly scheduler
cargo run --bin foxhunt-chaos schedule --time 02:00 --exclude-weekends --webhook https://hooks.slack.com/...
# Validate system readiness
cargo run --bin foxhunt-chaos validate --check-ml-service --check-database
```
### 3. Configuration File
```toml
# chaos_config.toml
[general]
enabled = true
schedule_time = "02:00"
max_duration_hours = 3
[ml_chaos_config]
ml_service_endpoint = "http://localhost:8080"
max_recovery_time_ms = 100 # HFT requirement
model_types = ["tlob", "dqn", "mamba2"]
```
## 🧪 Supported Failure Types
### Process Failures
- **SIGTERM/SIGKILL**: Graceful and forceful process termination
- **Service Restart**: Automatic restart with configurable delays
### Resource Exhaustion
- **Memory Pressure**: Configurable memory consumption (2GB-8GB)
- **GPU Exhaustion**: GPU memory filling (80%-95% capacity)
- **CPU Throttling**: CPU limit enforcement (25%-75%)
### Infrastructure Failures
- **Network Partitions**: Port-specific network isolation
- **Disk I/O Failures**: File system failure injection
- **Database Disconnections**: Connection pool exhaustion
## 📊 ML Model Support
The framework supports chaos testing across all Foxhunt ML models:
| Model | Type | Recovery Target | Checkpoint Interval |
|-------|------|----------------|-------------------|
| **TLOB** | Transformer | 25ms | 30s |
| **MAMBA-2** | State Space | 40ms | 60s |
| **DQN** | Deep Q-Learning | 80ms | 120s |
| **PPO** | Policy Optimization | 60ms | 90s |
| **Liquid** | Neural Network | 35ms | 45s |
| **TFT** | Temporal Fusion | 95ms | 180s |
## 🕒 Nightly Automation
### Scheduling Features
- **Configurable Time**: Any time zone and schedule
- **Weekend Exclusion**: Skip weekends for production safety
- **Retry Logic**: Automatic retry on failure with exponential backoff
- **Notification Integration**: Slack/Teams webhooks for alerts
### Alert Thresholds
- **Critical**: SLA violations (>100ms recovery)
- **Warning**: Checkpoint failures or performance regressions
- **Info**: Successful completion notifications
## 📈 Performance Requirements
### HFT Latency Targets
- **Order Processing**: <50μs end-to-end
- **Market Data**: <30μs ingestion latency
- **Risk Calculation**: <25μs computation
- **Recovery Time**: <100ms system restoration
### Validation Metrics
- **P50/P95/P99 Latency**: Histogram tracking
- **Recovery Time Distribution**: Statistical analysis
- **Checkpoint Integrity**: Binary validation
- **Performance Regression**: Pre/post comparison
## 🔧 Integration
### Test Infrastructure Integration
```rust
// In your tests/lib.rs
pub mod chaos;
#[tokio::test]
async fn test_ml_service_resilience() {
use crate::chaos::*;
let results = run_quick_chaos_test().await.unwrap();
assert!(!results.is_empty());
}
```
### CI/CD Integration
```yaml
# .github/workflows/chaos.yml
name: Chaos Engineering
on:
schedule:
- cron: '0 2 * * *' # Run at 2 AM daily
jobs:
chaos-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Run Chaos Tests
run: cargo test --test chaos_integration
```
## 📋 Example Scenarios
### 1. ML Training Kill/Restart
```rust
let experiment = ChaosExperiment {
name: "TLOB Training Kill Test".to_string(),
target_service: "ml_training_service".to_string(),
failure_type: FailureType::ProcessKill {
signal: Signal::SIGTERM,
delay_before_restart_ms: 2000,
},
max_recovery_time_ms: 25, // TLOB target
// ...
};
```
### 2. GPU Memory Exhaustion
```rust
let experiment = ChaosExperiment {
name: "GPU Memory Pressure Test".to_string(),
failure_type: FailureType::GpuResourceExhaustion {
memory_fill_percent: 95,
duration_ms: 20000,
},
max_recovery_time_ms: 150, // Relaxed for GPU
// ...
};
```
### 3. Network Partition
```rust
let experiment = ChaosExperiment {
name: "Database Partition Test".to_string(),
failure_type: FailureType::NetworkPartition {
target_ports: vec![5432, 6379], // PostgreSQL, Redis
duration_ms: 15000,
},
// ...
};
```
## 🛡️ Safety Features
### Production Safeguards
- **Weekend Exclusion**: Automatic weekend skipping
- **Duration Limits**: Maximum 3-hour chaos windows
- **Recovery Timeouts**: Automatic experiment termination
- **Checkpoint Validation**: Pre/post integrity checks
### Monitoring Integration
- **Real-time Alerting**: Immediate notification of failures
- **Performance Tracking**: Latency histogram recording
- **Report Generation**: Automated markdown reports
- **Event Streaming**: Live experiment status updates
## 📊 Reporting
### Automated Reports
```markdown
# ML Training Chaos Engineering Report
**Generated:** 2025-01-21 02:30:00 UTC
**Total Tests:** 18
## Summary
- ✅ **Successful:** 16 (88.9%)
- ❌ **Failed:** 2 (11.1%)
- 📊 **Success Rate:** 88.9%
## Results by Model Type
- **tlob:** 6/6 (100.0%)
- **dqn:** 5/6 (83.3%)
- **mamba2:** 5/6 (83.3%)
## Performance Analysis
- ✅ **No Performance Regressions Detected**
- **Average Recovery Time:** 45.2ms
- **Max Recovery Time:** 78ms
```
## 🔍 Troubleshooting
### Common Issues
1. **Service Not Found**
```bash
# Check service status
systemctl status ml_training_service
```
2. **GPU Not Available**
```bash
# Verify GPU access
nvidia-smi
```
3. **Permission Errors**
```bash
# Check chaos framework permissions
sudo usermod -a -G docker $USER
```
### Debug Mode
```bash
# Enable verbose logging
cargo run --bin foxhunt-chaos --verbose run --experiment-type process-kill
```
## 🤝 Contributing
1. **Add New Failure Types**: Extend `FailureType` enum
2. **ML Model Support**: Add new models to `ModelType`
3. **Monitoring Integration**: Extend metrics collection
4. **Custom Experiments**: Create domain-specific tests
## 📝 License
This chaos engineering framework is part of the Foxhunt HFT trading system and follows the same licensing terms.
---
**⚠️ Important**: This framework is designed specifically for HFT systems with sub-100ms recovery requirements. Always test in non-production environments first and ensure proper safeguards are in place.

257
testing/integration/e2e_helpers/README.md
View File

@@ -1,257 +0,0 @@
# E2E Testing Helpers
Helper scripts and utilities for end-to-end testing of the Foxhunt HFT Trading System.
## JWT Token Generator
**File**: `jwt_token_generator.sh`
Generates valid JWT tokens for testing API Gateway authentication. The token structure matches the production API Gateway implementation.
### Quick Start
```bash
# Generate default trader token
./jwt_token_generator.sh
# Generate admin token
./jwt_token_generator.sh admin_user admin "api.access,system.admin"
# Generate token with 10-minute expiration
./jwt_token_generator.sh test_user trader "api.access" 600
# Use in curl request
TOKEN=$(./jwt_token_generator.sh)
curl -H "Authorization: Bearer $TOKEN" http://localhost:50051/api/v1/orders
```
### Arguments
| Position | Name | Default | Description |
|----------|------|---------|-------------|
| 1 | user_id | `test_user_123` | User identifier |
| 2 | role | `trader` | User role (trader, admin, viewer) |
| 3 | permissions | `api.access` | Comma-separated permissions |
| 4 | ttl_seconds | `3600` | Token expiration in seconds |
### Environment Variables
- `JWT_SECRET` - JWT signing secret (default: test secret matching API Gateway)
**Production**: Set `JWT_SECRET` environment variable to match your deployment:
```bash
export JWT_SECRET="your-production-secret-key"
./jwt_token_generator.sh
```
### Token Structure
Generated tokens include the following claims (matching `services/api_gateway/tests/common/mod.rs`):
**Standard JWT Claims**:
- `sub` - Subject (user ID)
- `iat` - Issued at (Unix timestamp)
- `exp` - Expiration (Unix timestamp)
- `nbf` - Not before (Unix timestamp)
- `iss` - Issuer (`foxhunt-api-gateway`)
- `aud` - Audience (`foxhunt-services`)
- `jti` - JWT ID (UUID, for revocation support)
**Foxhunt-Specific Claims**:
- `roles` - User roles array (RBAC)
- `permissions` - Granular permissions array
- `token_type` - Token type (`access` or `refresh`)
- `session_id` - Session identifier (UUID)
### Common Use Cases
#### 1. Test API Gateway Authentication
```bash
# Generate token
TOKEN=$(./jwt_token_generator.sh)
# Test health endpoint (no auth required)
curl http://localhost:8080/health
# Test authenticated endpoint
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:50051/api/v1/orders
```
#### 2. Test Role-Based Access Control (RBAC)
```bash
# Trader role (limited permissions)
TRADER_TOKEN=$(./jwt_token_generator.sh trader_user trader "api.access")
# Admin role (full permissions)
ADMIN_TOKEN=$(./jwt_token_generator.sh admin_user admin "api.access,system.admin")
# Test trader permissions
curl -H "Authorization: Bearer $TRADER_TOKEN" \
http://localhost:50051/api/v1/orders
# Test admin permissions
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:50051/api/v1/config
```
#### 3. Test Token Expiration
```bash
# Short-lived token (30 seconds)
SHORT_TOKEN=$(./jwt_token_generator.sh test_user trader "api.access" 30)
# Use immediately (should succeed)
curl -H "Authorization: Bearer $SHORT_TOKEN" \
http://localhost:50051/api/v1/orders
# Wait 31 seconds
sleep 31
# Use again (should fail with 401 Unauthorized)
curl -H "Authorization: Bearer $SHORT_TOKEN" \
http://localhost:50051/api/v1/orders
```
#### 4. Load Testing with Multiple Users
```bash
# Generate 10 unique user tokens
for i in {1..10}; do
TOKEN=$(./jwt_token_generator.sh "user_$i" trader "api.access")
echo "$TOKEN" > "token_$i.txt"
done
# Use in load test
TOKEN=$(cat token_1.txt)
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:50051/api/v1/orders
```
#### 5. Integration Test Scripts
```bash
#!/bin/bash
# test_trading_flow.sh
# Generate authentication token
TOKEN=$(./jwt_token_generator.sh)
# Submit order
ORDER_RESPONSE=$(curl -s -X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"symbol": "BTC/USD", "quantity": 1.0, "price": 50000.0}' \
http://localhost:50051/api/v1/orders)
# Extract order ID
ORDER_ID=$(echo "$ORDER_RESPONSE" | jq -r '.order_id')
# Check order status
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:50051/api/v1/orders/$ORDER_ID"
```
### Validation
The generated tokens are validated against the same structure used in Rust tests:
**Source**: `services/api_gateway/tests/common/mod.rs` (lines 28-62)
```rust
pub fn generate_test_token(
user_id: &str,
roles: Vec<String>,
permissions: Vec<String>,
ttl_seconds: u64,
) -> Result<(String, String)> {
// ... (matches this script's implementation)
}
```
### Dependencies
**Required**:
- Python 3.x
- PyJWT library: `pip install pyjwt`
**Installation**:
```bash
# Ubuntu/Debian
sudo apt-get install python3 python3-pip
pip3 install pyjwt
# macOS
brew install python3
pip3 install pyjwt
# Verify installation
python3 -c "import jwt; print('PyJWT installed:', jwt.__version__)"
```
### Troubleshooting
#### Error: "PyJWT library not found"
```bash
# Install PyJWT
pip3 install pyjwt
# Or use system package manager
sudo apt-get install python3-jwt # Debian/Ubuntu
```
#### Error: "python3 not found"
```bash
# Install Python 3
sudo apt-get install python3 # Debian/Ubuntu
brew install python3 # macOS
```
#### Token Validation Fails
Ensure JWT_SECRET matches your API Gateway configuration:
```bash
# Check API Gateway secret (default for development)
export JWT_SECRET="test-secret-must-be-at-least-64-characters-long-for-security-validation-ok-1234567890"
# Generate token with correct secret
TOKEN=$(./jwt_token_generator.sh)
```
#### Token Expired
Default expiration is 1 hour (3600 seconds). Generate fresh token:
```bash
# Generate new token
TOKEN=$(./jwt_token_generator.sh)
# Or increase TTL to 24 hours
TOKEN=$(./jwt_token_generator.sh test_user trader "api.access" 86400)
```
### Security Notes
- **Development Only**: The default JWT secret is for testing only
- **Production**: Always use a strong, randomly-generated secret
- **Storage**: Never commit tokens or secrets to version control
- **Expiration**: Use short-lived tokens (15-60 minutes) in production
- **Revocation**: Tokens include `jti` claim for server-side revocation
### Related Files
- **API Gateway Auth**: `services/api_gateway/src/auth/interceptor.rs`
- **JWT Service**: `services/api_gateway/src/auth/jwt/service.rs`
- **Test Utilities**: `services/api_gateway/tests/common/mod.rs`
- **E2E Tests**: `services/api_gateway/tests/e2e_tests.rs`
### References
- JWT Standard: [RFC 7519](https://tools.ietf.org/html/rfc7519)
- PyJWT Documentation: https://pyjwt.readthedocs.io/
- API Gateway RBAC: `services/api_gateway/src/auth/README.md`

359
testing/integration/smoke_tests/README.md
View File

@@ -1,359 +0,0 @@
# Smoke Tests - End-to-End System Validation
## Overview
Automated smoke test suite for validating complete Foxhunt HFT system functionality after deployment. These tests provide quick validation that all critical components are operational.
## Test Categories
### 1. Infrastructure Health (`infrastructure_health.rs`)
Tests core infrastructure services:
- **PostgreSQL**: Connection, schema validation, TimescaleDB extension
- **Redis**: Connection, SET/GET operations, key expiration
- **Vault**: Connectivity and health status
- **InfluxDB**: Ping endpoint and availability
- **Prometheus**: Health endpoint and metrics availability
- **Grafana**: API health check
### 2. Service Health (`service_health.rs`)
Tests gRPC microservices:
- **Trading Service**: HTTP health check, gRPC connection (port 50052)
- **API Gateway**: gRPC connection (port 50051)
- **Backtesting Service**: gRPC connection (port 50053) - BLOCKED
- **ML Training Service**: gRPC connection (port 50054) - BLOCKED
- **Service Ports**: Validates all services are listening
- **Response Times**: Measures connection latency
- **Metrics Endpoints**: Validates Prometheus exporters (ports 9091-9094)
### 3. Authentication Flow (`authentication_flow.rs`)
Tests authentication and security:
- **JWT Generation**: Token creation with claims
- **JWT Validation**: Signature verification and claim extraction
- **JWT Expiration**: Expired token rejection
- **JWT Signature**: Invalid signature detection
- **Session Storage**: Redis-based session management
- **Token Revocation**: Revocation list checking
- **Rate Limiting**: Request throttling with Redis
### 4. Basic Order Flow (`basic_order_flow.rs`)
Tests core trading operations:
- **Order Submission**: Insert orders into PostgreSQL
- **Order Query**: Retrieve order by ID
- **Order Cancellation**: Update order status to cancelled
- **Position Management**: Create and query positions
- **Order History**: Query user's order history
- **Complete Lifecycle**: Full order flow from submission to cleanup
## Usage
### Run All Smoke Tests
```bash
# Using the automated script (recommended)
./run_smoke_tests.sh
# Using Cargo directly
cargo test --test smoke_tests --features smoke-tests
```
### Run Specific Categories
```bash
# Infrastructure only
./run_smoke_tests.sh --category infrastructure
# Service health only
./run_smoke_tests.sh --category service
# Authentication only
./run_smoke_tests.sh --category authentication
# Order flow only
./run_smoke_tests.sh --category order_flow
```
### Fast Mode (Critical Tests Only)
```bash
# Run only infrastructure and service health checks
./run_smoke_tests.sh --fast
```
### Verbose Mode
```bash
# Run with debug logging
./run_smoke_tests.sh --verbose
```
## Environment Variables
All tests use environment variables with sensible defaults:
```bash
# Infrastructure
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
REDIS_URL=redis://localhost:6379
VAULT_ADDR=http://localhost:8200
INFLUXDB_URL=http://localhost:8086
# Services
API_GATEWAY_URL=http://localhost:50051
TRADING_SERVICE_URL=http://localhost:50052
BACKTESTING_SERVICE_URL=http://localhost:50053
ML_TRAINING_SERVICE_URL=http://localhost:50054
# Monitoring
PROMETHEUS_URL=http://localhost:9090
GRAFANA_URL=http://localhost:3000
# Authentication
JWT_SECRET=dev_secret_key_change_in_production
# Logging
RUST_LOG=info # Set to 'debug' for verbose output
```
## Known Issues and Blockers
### Blocked Tests (Agent 96 Findings)
1. **Backtesting Service** - Configuration issues prevent deployment
- Test marked with `#[ignore]` attribute
- Skips gracefully when service unavailable
2. **ML Training Service** - Configuration issues prevent deployment
- Test marked with `#[ignore]` attribute
- Skips gracefully when service unavailable
### Working Tests
- ✅ Infrastructure Health (PostgreSQL, Redis, Vault, InfluxDB, Prometheus, Grafana)
- ✅ Trading Service Health (confirmed working by Agent 96)
- ✅ API Gateway Health
- ✅ Authentication Flow (JWT, Sessions, Rate Limiting)
- ✅ Basic Order Flow (Database operations)
## Test Architecture
### Graceful Failure Handling
Tests use the `skip_if_unavailable!` macro to handle service unavailability:
```rust
skip_if_unavailable!("Service Name", {
// Test code here
result
});
```
This allows tests to:
- Pass when services are available
- Skip gracefully when services are down (connection refused, timeout)
- Fail hard for actual test failures
### Timeout Management
All tests have configurable timeouts:
- **Standard smoke tests**: 10 seconds
- **Infrastructure tests**: 5 seconds
Timeouts prevent hanging tests and provide quick feedback.
### Parallel Execution
Tests can run in parallel by default, but use `--test-threads=1` for sequential execution when debugging:
```bash
cargo test --test smoke_tests -- --test-threads=1 --nocapture
```
## Integration with CI/CD
### Docker Deployment Validation
After deploying with Docker Compose:
```bash
# 1. Start all services
docker-compose up -d
# 2. Wait for health checks
docker-compose ps
# 3. Run smoke tests
./run_smoke_tests.sh
# 4. Check exit code
if [ $? -eq 0 ]; then
echo "Deployment validated - ready for production"
else
echo "Deployment issues detected - review logs"
fi
```
### Kubernetes Deployment Validation
```bash
# 1. Deploy to cluster
kubectl apply -f k8s/
# 2. Wait for pods
kubectl wait --for=condition=ready pod -l app=foxhunt --timeout=300s
# 3. Port forward services
kubectl port-forward svc/api-gateway 50051:50051 &
kubectl port-forward svc/trading-service 50052:50051 &
# 4. Run smoke tests
./run_smoke_tests.sh
# 5. Cleanup port forwards
pkill -f "kubectl port-forward"
```
## Metrics and Reporting
### Test Pass Rate
The smoke test runner reports:
- Total categories tested
- Passed/failed counts
- Pass percentage
- Exit code (0 = success, 1 = failure)
### Sample Output
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Foxhunt HFT System - Smoke Test Suite
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Environment Configuration:
Database: postgresql://foxhunt:***@localhost:5432/foxhunt
Redis: redis://localhost:6379
Vault: http://localhost:8200
API Gateway: http://localhost:50051
Trading Service: http://localhost:50052
Log Level: info
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Testing: Infrastructure Health
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ PostgreSQL connection successful (TimescaleDB: true)
✅ Redis connection and operations successful
✅ Vault connectivity successful (status: 200)
✅ Infrastructure Health - PASSED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Smoke Test Summary
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total Categories: 4
Passed: 4
Failed: 0
Pass Rate: 100%
✅ All smoke tests passed!
System is ready for deployment.
```
## Future Enhancements
### Planned Test Categories (Not Implemented)
5. **Market Data Tests** (blocked by service availability)
- Subscribe to market data stream
- Receive quote updates
- Historical data queries
6. **ML Inference Tests** (blocked by service availability)
- Model prediction requests
- Feature engineering pipeline
- Inference latency validation (<100ms)
7. **Compliance Tests** (requires ML service)
- Audit log creation
- Best execution analysis
- Risk check validation
8. **Advanced Monitoring** (requires full deployment)
- Alert manager connectivity
- Custom dashboard validation
- Log aggregation checks
## Troubleshooting
### Common Issues
1. **Connection Refused**
```bash
# Check if services are running
docker-compose ps
# Restart failed services
docker-compose restart trading_service
```
2. **Database Schema Missing**
```bash
# Run migrations
cargo sqlx migrate run
```
3. **Redis Connection Failed**
```bash
# Check Redis is running
redis-cli ping
# Restart Redis
docker-compose restart redis
```
4. **JWT Secret Not Set**
```bash
# Set JWT secret
export JWT_SECRET="your-secret-key-here"
```
### Debug Mode
Run tests with full debug output:
```bash
RUST_LOG=debug ./run_smoke_tests.sh --verbose
```
This shows:
- Full test execution flow
- Connection attempts
- Query execution
- Error details
## Contributing
When adding new smoke tests:
1. Create test file in `tests/smoke_tests/`
2. Add module to `tests/smoke_tests/mod.rs`
3. Use `skip_if_unavailable!` macro for graceful failures
4. Add timeout with `with_timeout()` wrapper
5. Update this README with test documentation
6. Update `run_smoke_tests.sh` with new category
### Test Template
```rust
#[tokio::test]
async fn test_new_feature() {
let result = with_timeout(async {
// Your test code here
Ok(())
}).await;
skip_if_unavailable!("Service Name", { result });
}
```

471
testing/load/README.md
View File

@@ -1,464 +1,25 @@
# Load Tests - Minimal Dependency Crate
# load
**Purpose**: Fast-compiling load tests for Foxhunt Trading Service
**Compilation Time**: 20-30 seconds (vs 120-180s in original tests/ crate)
**Dependency Reduction**: 86% (5 deps vs 36 deps)
---
## Quick Start
### Rust Load Tests (Direct Trading Service - Port 50052)
```bash
cd tests/load_tests
cargo test --release -- --nocapture
```
### Authenticated ghz Load Tests (API Gateway - Port 50051)
```bash
cd tests/load_tests
# Quick authentication test (1 request)
./ghz_quick_auth_test.sh
# Full authenticated load test suite
./ghz_authenticated.sh
```
---
Minimal-dependency load tests for Trading Service throughput validation. Rust gRPC tests (direct port 50052) plus authenticated ghz shell scripts (API Gateway port 50051).
## Test Suites
### 1. Rust Load Tests (Minimal Dependencies)
### Rust Tests (direct trading service)
**Target**: Trading Service direct (port 50052)
**Auth**: Not required (direct backend access)
- `test_1_baseline_latency` -- 1K sequential orders
- `test_2_concurrent_connections` -- 100 clients x 100 orders
- `test_3_sustained_load` (ignored) -- 5 min, 10K orders/sec
- `test_4_database_performance` -- 5K order DB writes
- `test_5_resource_monitoring` -- health + metrics (feature `health-checks`)
- `test_6_production_readiness` -- 50 clients x 200 orders, success >= 99%
#### Run Specific Test
### ghz Scripts (API Gateway with JWT)
- `ghz_quick_auth_test.sh` -- single authenticated request
- `ghz_authenticated.sh` -- baseline/medium/high/sustained scenarios
## Usage
```bash
# Baseline latency (1000 sequential orders)
cargo test --release test_1_baseline_latency -- --nocapture
# Concurrent connections (100 clients, 100 orders each)
cargo test --release test_2_concurrent_connections -- --nocapture
# Database performance (5000 orders)
cargo test --release test_4_database_performance -- --nocapture
# Resource monitoring (health + metrics)
cargo test --release test_5_resource_monitoring -- --nocapture
# Production readiness assessment
cargo test --release test_6_production_readiness -- --nocapture
cargo test -p load --release -- --nocapture
```
#### Run Sustained Load Test (Ignored by Default)
```bash
# 5-minute sustained load (50 clients, 200 orders/sec each = 10K total)
cargo test --release test_3_sustained_load -- --ignored --nocapture
```
### 2. Authenticated ghz Load Tests (Shell Scripts)
**Target**: API Gateway (port 50051)
**Auth**: JWT tokens (auto-generated)
**Protocol**: gRPC with metadata
#### Prerequisites
1. **Install ghz** (if not already installed):
```bash
# Ubuntu/Debian
wget https://github.com/bojand/ghz/releases/download/v0.117.0/ghz-linux-x86_64.tar.gz
tar -xzf ghz-linux-x86_64.tar.gz
sudo mv ghz /usr/local/bin/
# MacOS
brew install ghz
# Arch Linux
yay -S ghz
```
2. **Install jq** (optional, for result parsing):
```bash
sudo apt-get install jq # Ubuntu/Debian
brew install jq # MacOS
```
3. **Start API Gateway**:
```bash
docker-compose up -d api_gateway postgres trading_service
```
4. **Configure JWT Secret** (already in .env):
```bash
# Verify JWT_SECRET is set
grep JWT_SECRET .env
```
#### Available Scripts
##### Quick Authentication Test
```bash
# Verify JWT auth works (1 request only)
./ghz_quick_auth_test.sh
```
**Output**: Single authenticated request to validate setup
##### Full Authenticated Load Suite
```bash
# Run all 4 test scenarios (baseline, medium, high, sustained)
./ghz_authenticated.sh
```
**Test Scenarios**:
1. **Baseline**: 1,000 requests @ 100 RPS (10 concurrent)
2. **Medium**: 5,000 requests @ 500 RPS (50 concurrent)
3. **High**: 10,000 requests @ 1,000 RPS (100 concurrent)
4. **Sustained**: 2 minutes @ 500 RPS (60,000 total requests)
**Output Files**: `results/baseline_authenticated_*.json`, etc.
#### JWT Token Generation
The scripts automatically generate JWT tokens using:
```bash
# Manual token generation (if needed)
./tests/e2e_helpers/jwt_token_generator.sh [username] [role]
# Example
./tests/e2e_helpers/jwt_token_generator.sh "load_test_user" "trader"
```
**Token Features**:
- 1-hour expiration
- Includes trading permissions (submit_order, view_positions, cancel_order)
- Signed with JWT_SECRET from .env
- Includes jti, role, sub fields (required by API Gateway)
#### Results Analysis
**JSON Output** (with jq installed):
```bash
# View summary of latest test
jq '.' tests/load_tests/results/baseline_authenticated_*.json | tail -1
```
**Metrics Collected**:
- Total requests
- Success rate (%)
- P50, P95, P99 latency (ms)
- Throughput (req/s)
- Error distribution
**Monitoring Endpoints**:
- Prometheus: http://localhost:9091/metrics
- Grafana: http://localhost:3000
---
## Prerequisites
### Infrastructure Running
```bash
# For Rust tests (Trading Service direct)
docker-compose up -d postgres trading_service
# For ghz tests (API Gateway)
docker-compose up -d postgres trading_service api_gateway
# Verify services healthy
docker-compose ps
```
### Service Endpoints
| Service | Protocol | Port | Auth | Used By |
|---------|----------|------|------|---------|
| Trading Service | gRPC | 50052 | No | Rust tests |
| API Gateway | gRPC | 50051 | JWT | ghz scripts |
| Health (Trading) | HTTP | 8081 | No | test_5 |
| Metrics (Trading) | HTTP | 9092 | No | test_5 |
| Metrics (Gateway) | HTTP | 9091 | No | Monitoring |
---
## Test Details
### Rust Test Suite
#### Test 1: Baseline Latency
- **Orders**: 1,000 sequential
- **Purpose**: Single-client latency baseline
- **Metrics**: P50, P95, P99 latency + throughput
#### Test 2: Concurrent Connections
- **Clients**: 100 concurrent
- **Orders per client**: 100
- **Total orders**: 10,000
- **Purpose**: Concurrency stress test
- **Metrics**: Latency distribution + success rate
#### Test 3: Sustained Load (Ignored)
- **Duration**: 5 minutes
- **Clients**: 50 concurrent
- **Target rate**: 10,000 orders/sec total
- **Purpose**: Sustained load validation
- **Metrics**: Long-term stability
#### Test 4: Database Performance
- **Orders**: 5,000
- **Purpose**: Database write throughput
- **Target**: >2,000 writes/sec
#### Test 5: Resource Monitoring
- **Purpose**: Health + metrics validation
- **Checks**: HTTP health endpoint, Prometheus metrics
- **Requires**: `health-checks` feature
#### Test 6: Production Readiness
- **Clients**: 50 concurrent
- **Orders per client**: 200
- **Total orders**: 10,000
- **Criteria**:
- Success rate >= 99%
- Throughput >= 5,000 orders/sec
- P99 latency < 100ms
### ghz Authenticated Test Suite
#### Test 1: Baseline Authenticated Load
- **Requests**: 1,000
- **RPS**: 100
- **Concurrency**: 10
- **Purpose**: Verify JWT auth + baseline latency
- **Expected**: 100% success, <50ms P99
#### Test 2: Medium Authenticated Load
- **Requests**: 5,000
- **RPS**: 500
- **Concurrency**: 50
- **Purpose**: Medium load with authentication
- **Expected**: >99% success, <100ms P99
#### Test 3: High Authenticated Load
- **Requests**: 10,000
- **RPS**: 1,000
- **Concurrency**: 100
- **Purpose**: High throughput with JWT overhead
- **Expected**: >95% success, <150ms P99
#### Test 4: Sustained Authenticated Load
- **Duration**: 2 minutes
- **RPS**: 500
- **Concurrency**: 50
- **Total**: ~60,000 requests
- **Purpose**: Long-term stability validation
- **Expected**: >99% success, stable latency
---
## Features
### Default (No Features)
- Core gRPC load testing (tests 1-4, 6)
- Dependencies: `tokio`, `tonic`, `uuid`
### `health-checks` (Optional)
```bash
cargo test --release --features health-checks
```
- Enables test_5 (resource monitoring)
- Adds `reqwest` dependency
- HTTP health + metrics checks
---
## Performance Targets
| Metric | Target | Typical (Direct) | Typical (Gateway) |
|--------|--------|------------------|-------------------|
| Success Rate | >= 99% | 99.5-100% | 99-100% |
| Throughput | >= 5K orders/sec | 7-10K | 5-7K |
| P50 Latency | < 20ms | 10-15ms | 15-25ms |
| P99 Latency | < 100ms | 30-50ms | 50-100ms |
| DB Writes/sec | >= 2K | 2.5-3K | 2-2.5K |
**Note**: API Gateway adds ~5-10ms latency due to JWT validation and proxying.
---
## Troubleshooting
### "Connection refused" Error
**For Rust tests (port 50052)**:
```bash
docker-compose up -d trading_service
docker-compose ps # Verify "Up" status
```
**For ghz tests (port 50051)**:
```bash
docker-compose up -d api_gateway
docker-compose ps # Verify "Up" status
```
### "Failed to generate JWT token"
**Check JWT_SECRET**:
```bash
# Verify secret exists
grep JWT_SECRET .env
# If missing, add to .env
echo 'JWT_SECRET=your-secret-key-here' >> .env
```
### "Too many open files" Error
```bash
ulimit -n 4096 # Increase file descriptor limit
```
### Authentication Failures (401 errors)
**Check token format**:
```bash
# Generate test token
./tests/e2e_helpers/jwt_token_generator.sh test_user trader
# Verify token has 3 parts (header.payload.signature)
```
**Check API Gateway logs**:
```bash
docker-compose logs api_gateway | grep -i "auth\|jwt\|401"
```
### High Latency
**Check**:
1. PostgreSQL synchronous_commit setting
2. Network latency (localhost vs Docker)
3. System load (CPU, memory)
4. API Gateway JWT validation overhead
**Optimize PostgreSQL**:
```sql
-- In PostgreSQL
ALTER SYSTEM SET synchronous_commit = off;
SELECT pg_reload_conf();
```
---
## Compilation Time Comparison
| Crate | Dependencies | Compile Time | Speedup |
|-------|--------------|--------------|---------|
| `tests/` (original) | 36 | 120-180s | Baseline |
| `tests/load_tests` | 5 | 20-30s | **6x faster** |
| ghz scripts | N/A | 0s | **Instant** |
---
## Architecture
### Rust Tests (Minimal Dependencies)
```toml
[dependencies]
tokio = { workspace = true } # Async runtime
tonic = { workspace = true } # gRPC client
tonic-prost = { workspace = true } # Protobuf runtime
prost = { workspace = true } # Protobuf types
uuid = { workspace = true } # Order IDs
reqwest = { optional = true } # HTTP (feature-gated)
```
### ghz Scripts (Shell + OpenSSL)
```bash
# Dependencies
- bash
- ghz (gRPC load testing)
- openssl (JWT signing)
- jq (optional, result parsing)
- nc (netcat, connectivity check)
```
### Build Process
1. `build.rs` compiles `trading.proto` from Trading Service
2. Generated code included via `tonic::include_proto!("trading")`
3. No heavy dependencies (ML, database clients, test frameworks)
---
## CI/CD Integration
### GitHub Actions
```yaml
- name: Run Rust Load Tests
run: |
docker-compose up -d postgres trading_service
cd tests/load_tests
cargo test --release --features health-checks
- name: Run Authenticated ghz Tests
run: |
docker-compose up -d api_gateway postgres trading_service
cd tests/load_tests
./ghz_quick_auth_test.sh
./ghz_authenticated.sh
```
### GitLab CI
```yaml
rust_load_tests:
script:
- docker-compose up -d postgres trading_service
- cd tests/load_tests
- cargo test --release --features health-checks
ghz_load_tests:
script:
- docker-compose up -d api_gateway postgres trading_service
- cd tests/load_tests
- ./ghz_authenticated.sh
```
---
## Related Documentation
- [LOAD_TEST_DEPENDENCY_OPTIMIZATION.md](../../LOAD_TEST_DEPENDENCY_OPTIMIZATION.md) - Detailed analysis
- [LOAD_TEST_OPTIMIZATION_SUMMARY.md](../../LOAD_TEST_OPTIMIZATION_SUMMARY.md) - Implementation summary
- [TESTING_PLAN.md](../../TESTING_PLAN.md) - Overall testing strategy
- [WAVE_132_AUTH_VALIDATION_SUMMARY.md](../../WAVE_132_AUTH_VALIDATION_SUMMARY.md) - JWT authentication validation
---
## Summary
| Test Type | Target | Auth | Compilation | Execution | Use Case |
|-----------|--------|------|-------------|-----------|----------|
| Rust Tests | Trading Service (50052) | No | 20-30s | Fast | Backend performance |
| ghz Scripts | API Gateway (50051) | JWT | 0s | Fast | End-to-end auth flow |
**Recommendation**: Use **both** test types for comprehensive validation:
1. **Rust tests** for backend performance benchmarks
2. **ghz scripts** for authenticated API Gateway validation
---
**Status**: ✅ Production Ready
**Rust Tests**: < 30 seconds compilation ✅
**ghz Scripts**: Instant execution ✅
**JWT Authentication**: Fully validated ✅

9
testing/service-integration/README.md Normal file
View File

@@ -0,0 +1,9 @@
# service-integration
Cross-service integration tests for gRPC service interactions.
## Usage
```bash
SQLX_OFFLINE=true cargo test -p service-integration --lib
```

213
testing/service-load/README.md
View File

@@ -1,216 +1,17 @@
# Load Tests - Trading Service Throughput Validation
# service-load
## Overview
Trading service throughput load tests with 4 scenarios: sustained (10K/s for 60s), peak burst (50K/s for 10s), market data streaming (1M updates), and connection pool saturation (1K clients).
Comprehensive load testing suite for validating trading service throughput and performance under various load scenarios.
## Scenarios
## Test Scenarios
### 1. Sustained Load (10,000 orders/sec for 60s)
- **Target**: 10,000 orders/second sustained throughput
- **Duration**: 60 seconds
- **Concurrent Clients**: 100
- **Validates**: System stability under sustained load
### 2. Peak Burst (50,000 orders/sec for 10s)
- **Target**: 50,000 orders/second peak burst
- **Duration**: 10 seconds
- **Concurrent Clients**: 500
- **Validates**: System behavior under peak load spikes
### 3. Market Data Streaming (1M updates)
- **Target**: 1,000,000 concurrent market data updates
- **Streams**: 1,000 concurrent streams
- **Duration**: 30 seconds
- **Validates**: Streaming infrastructure capacity
### 4. Connection Pool Saturation (1,000 clients)
- **Target**: 1,000 concurrent clients
- **Requests per Client**: 100
- **Validates**: Connection pool management and resource limits
- `sustained` -- 10K orders/sec, 60s, 100 clients
- `burst` -- 50K orders/sec, 10s, 500 clients
- `streaming` -- 1M market data updates, 1K streams
- `pool` -- 1K concurrent clients, 100 requests each
## Usage
### Run All Tests
```bash
cargo run -p load_tests --release -- --scenario all
```
### Run Individual Scenarios
```bash
# Sustained load
cargo run -p load_tests --release -- --scenario sustained
# Peak burst
cargo run -p load_tests --release -- --scenario burst
# Streaming
cargo run -p load_tests --release -- --scenario streaming
# Connection pool
cargo run -p load_tests --release -- --scenario pool
```
### Custom Configuration
```bash
cargo run -p load_tests --release -- \
--scenario sustained \
--url http://trading-service:50052 \
--output /path/to/report.md \
--verbose
```
## Metrics Collected
### Throughput Metrics
- Requests per second (sustained and peak)
- Total requests processed
- Success/failure rates
### Latency Distribution
- P50 (median) latency
- P95 latency
- P99 latency
- Maximum latency
### Resource Usage
- Memory consumption (average)
- Connection pool utilization
- Stream management overhead
## Output Report
Test results are saved as Markdown reports containing:
- Executive summary
- Detailed metrics breakdown
- Latency distribution charts
- Resource usage analysis
- Performance recommendations
Default output: `/tmp/WAVE_120_AGENT_5_LOAD_TESTING.md`
## Prerequisites
1. **Trading Service Running**:
```bash
docker-compose up -d trading_service
# OR
cargo run -p trading_service
```
2. **Database Available**:
```bash
docker-compose up -d postgres redis
```
3. **Sufficient System Resources**:
- 8GB+ RAM recommended
- Multi-core CPU for parallel clients
- Network bandwidth for 50k+ req/sec
## Architecture
### Components
- **Scenarios**: Test scenario implementations
- `sustained_load.rs`: 10k orders/sec for 60s
- `burst_load.rs`: 50k orders/sec for 10s
- `streaming_load.rs`: 1M market data updates
- `pool_saturation.rs`: 1000 concurrent clients
- `comprehensive.rs`: All scenarios sequentially
- **Clients**: gRPC client implementations
- `trading_client.rs`: Trading service client wrapper
- **Metrics**: Performance measurement
- `metrics.rs`: HDR histogram-based metrics collection
- `monitor.rs`: System resource monitoring
### Load Generation Pattern
```rust
// Concurrent client pattern
for client_id in 0..NUM_CLIENTS {
tokio::spawn(async move {
let client = TradingClient::connect(url).await?;
// Submit orders with rate limiting
while duration_remaining {
client.submit_order(...).await?;
tokio::time::sleep(rate_limit).await;
}
});
}
```
## Performance Targets
### Sustained Load
- ✅ Throughput: ≥9,000 req/sec
- ✅ Error Rate: <1%
- ✅ P95 Latency: <10ms
### Peak Burst
- ✅ Throughput: ≥40,000 req/sec
- ✅ Error Rate: <5%
- ✅ P99 Latency: <50ms
### Streaming
- ✅ Updates: ≥900k received
- ✅ Concurrent Streams: 1000
- ✅ Stream Stability: <1% failures
### Connection Pool
- ✅ Concurrent Connections: 1000
- ✅ Error Rate: <5%
- ✅ P99 Latency: <100ms
## Troubleshooting
### Connection Refused
```bash
# Verify trading service is running
grpc_health_probe -addr=localhost:50052
```
### High Error Rates
- Check system resource limits (ulimit, file descriptors)
- Verify database connection pool size
- Review trading service logs for errors
### Memory Issues
- Reduce concurrent clients
- Enable connection pooling
- Check for memory leaks in trading service
## Integration with CI/CD
```yaml
# .github/workflows/load-test.yml
- name: Run Load Tests
run: |
docker-compose up -d
cargo run -p load_tests --release -- --scenario all
- name: Upload Report
uses: actions/upload-artifact@v3
with:
name: load-test-report
path: /tmp/WAVE_120_AGENT_5_LOAD_TESTING.md
```
## Wave 120 Objectives
**Agent 5 Tasks**:
- ✅ Create load_tests package
- ✅ Implement 4 throughput scenarios
- ✅ Measure latency, throughput, error rates
- ✅ Monitor memory usage
- ⏳ Run tests against live service
- ⏳ Generate performance report
**Expected Outcomes**:
- Validate 10k orders/sec sustained capacity
- Confirm 50k orders/sec peak burst handling
- Verify 1M concurrent stream updates
- Validate 1000+ concurrent client support

288
testing/test-common/README.md
View File

@@ -1,282 +1,30 @@
# Test Common Crate
# test-common
Shared test fixtures, builders, and utilities for the Foxhunt project.
Shared test fixtures, builders, and assertions for the Foxhunt workspace. Consolidates duplicate test code across 39+ files.
This crate consolidates **duplicate test code across 39+ test files**, saving approximately **3,500+ LOC** and significantly improving test maintainability.
## Key Types
## 📦 What's Included
- `TestDb` -- isolated test database with automatic cleanup
- `MockOrderBuilder` -- fluent order/trade builder
- `BarBuilder` -- OHLCV bar generation with realistic price movements
- `PositionBuilder` -- mock positions with P&L calculations
### Fixtures (`fixtures/`)
## Fixtures
#### Database (`database.rs`)
- **TestDb**: Isolated test database with automatic schema cleanup
- **setup_test_data()**: Pre-populate test data
- Pattern consolidates **39 duplicate `setup_test_db()` functions**
- `generate_ohlcv_bars()` -- realistic OHLCV data
- `generate_random_walk()` -- Monte Carlo price simulations
- `generate_crisis_returns()` -- 2008 crisis patterns
- `generate_order_book()` -- order book levels
```rust
use test_common::TestDb;
## Assertions
#[tokio::test]
async fn my_test() {
let db = TestDb::with_migrations().await;
let result = sqlx::query("SELECT * FROM users")
.fetch_all(db.pool())
.await;
// Test continues...
}
```
- `assert_approx_eq!` -- percentage tolerance comparison
- `assert_ohlc_valid` -- OHLC bar validation
- `assert_var_exceedances` -- VaR exceedance checks
#### Orders (`orders.rs`)
- **MockOrderBuilder**: Fluent API for creating test orders
- **MockTradeBuilder**: Fluent API for creating test trades
- Pattern consolidates **22 duplicate `create_mock_order()` functions**
```rust
use test_common::MockOrderBuilder;
let order = MockOrderBuilder::new()
.symbol("TSLA")
.sell()
.quantity(50.0)
.limit_price(200.0)
.filled()
.build();
```
#### Market Data (`market_data.rs`)
- **generate_ohlcv_bars()**: Realistic OHLCV data with price movements
- **generate_random_walk()**: Monte Carlo price simulations
- **generate_crisis_returns()**: 2008 Financial Crisis patterns
- **generate_covid_crash_returns()**: March 2020 patterns
- **generate_order_book()**: Order book levels
- Pattern consolidates **15+ duplicate bar/tick generators**
```rust
use test_common::generate_ohlcv_bars;
let bars = generate_ohlcv_bars(100, 150.0);
assert_eq!(bars.len(), 100);
```
#### Config (`config.rs`)
- **TestConfig**: General test configuration
- **MLTestConfig**: ML training test config
- **RiskTestConfig**: Risk engine test config
```rust
use test_common::fixtures::config::RiskTestConfig;
let config = RiskTestConfig::conservative();
assert_eq!(config.circuit_breaker_threshold, 0.05);
```
#### Network (`network.rs`)
- **MockHttpResponse**: HTTP response builder
- **mock_market_data_response()**: API response fixtures
- **mock_websocket_*_message()**: WebSocket message fixtures
### Builders (`builders/`)
#### BarBuilder (`bar_builder.rs`)
Fluent API for creating OHLCV bars with realistic data:
```rust
use test_common::BarBuilder;
let bars = BarBuilder::new()
.price(150.0)
.bullish()
.build_series(100, 5); // 100 bars, 5 minutes apart
```
#### OrderBuilder (`order_builder.rs`)
Re-export of `MockOrderBuilder` for convenience.
#### PositionBuilder (`position_builder.rs`)
Create mock positions with P&L calculations:
```rust
use test_common::PositionBuilder;
let pos = PositionBuilder::new()
.symbol("NVDA")
.long(100.0)
.entry_price(450.0)
.profitable(10.0) // 10% profit
.build();
assert_eq!(pos.pnl(), 4500.0);
```
### Assertions (`assertions/`)
Domain-specific assertions for financial testing:
```rust
use test_common::assert_approx_eq;
use test_common::assertions::assert_ohlc_valid;
// Percentage tolerance
assert_approx_eq!(100.0, 101.0, 0.02); // 2% tolerance
// OHLC validation
assert_ohlc_valid(open, high, low, close);
// VaR exceedances
assert_var_exceedances(&returns, var, 0.99, 0.1);
// P&L validation
assert_pnl(entry_price, current_price, quantity, expected, 0.01);
```
## 🚀 Quick Start
### Add to your test crate
Update your crate's `Cargo.toml`:
## Usage
```toml
[dev-dependencies]
test_common = { path = "../../tests/test_common" }
test_common = { path = "../../testing/test-common" }
```
### Import and use
```rust
use test_common::{TestDb, MockOrderBuilder, generate_ohlcv_bars};
#[tokio::test]
async fn comprehensive_test() {
// Setup database
let db = TestDb::with_migrations().await;
// Create test data
let bars = generate_ohlcv_bars(100, 150.0);
let order = MockOrderBuilder::new()
.symbol("AAPL")
.buy()
.build();
// Run your tests...
}
```
## 📊 Impact Analysis
### LOC Savings by Pattern
| Pattern | Files | Avg LOC/File | Total Saved |
|---------|-------|--------------|-------------|
| `setup_test_db()` | 39 | 15 | 585 |
| `create_mock_order()` | 22 | 25 | 550 |
| `create_mock_bars()` | 15 | 35 | 525 |
| Database setup helpers | 39 | 20 | 780 |
| Market data generators | 15 | 30 | 450 |
| Config builders | 10 | 15 | 150 |
| Mock responses | 8 | 20 | 160 |
| Custom assertions | 12 | 18 | 216 |
| **TOTAL** | **160+** | **~22** | **~3,416** |
### Files Consolidated
#### ML Training Service Tests (15 files)
- `checkpoint_manager_tests.rs`
- `job_tracker_test.rs`
- `integration_tests.rs`
- `model_lifecycle_edge_cases.rs`
- `training_error_recovery_tests.rs`
- And 10+ more integration tests
#### Risk Tests (18 files)
- `risk_comprehensive_tests.rs` (1,197 LOC)
- `risk_var_calculations_tests.rs` (855 LOC)
- `portfolio_optimization_tests.rs` (1,002 LOC)
- And 15+ more risk/compliance tests
#### ML Tests (10+ files)
- `feature_cache_tests.rs`
- `dqn_feature_cache_test.rs`
- `inference_optimization_tests.rs`
- And more
## 🔧 Migration Guide
### Before (Duplicate Code)
```rust
// In every test file:
async fn setup_test_db() -> PgPool {
let database_url = std::env::var("DATABASE_URL")
.unwrap_or_else(|_| "postgresql://...".to_string());
PgPool::connect(&database_url).await.unwrap()
}
fn create_mock_bars(count: usize) -> Vec<OHLCVBar> {
let mut bars = Vec::new();
// 30+ lines of duplicate bar generation...
bars
}
#[tokio::test]
async fn my_test() {
let pool = setup_test_db().await;
let bars = create_mock_bars(100);
// Test logic...
}
```
### After (Using test_common)
```rust
use test_common::{TestDb, generate_ohlcv_bars};
#[tokio::test]
async fn my_test() {
let db = TestDb::with_migrations().await;
let bars = generate_ohlcv_bars(100, 150.0);
// Test logic...
}
```
**Result**: Reduced from ~60 LOC to ~8 LOC per test file!
## 📈 Benefits
1. **Maintainability**: Update fixtures once, benefit everywhere
2. **Consistency**: All tests use identical setup patterns
3. **Discoverability**: Single location for test utilities
4. **Type Safety**: Builder patterns prevent invalid test data
5. **Documentation**: Well-documented patterns and examples
6. **Speed**: Pre-compiled fixtures load faster than copies
## 🧪 Test Coverage
The `test_common` crate itself has **90%+ test coverage**:
```bash
cd tests/test_common
cargo test
```
All fixtures and builders include their own unit tests to ensure correctness.
## 📝 Contributing
When adding new test patterns:
1. Check if similar code exists in 3+ test files
2. Extract to appropriate module in `test_common`
3. Add builder pattern if applicable
4. Include unit tests
5. Document with examples
6. Update this README
## 🔗 Related Documentation
- [Risk Tests](../../risk/tests/README.md)
- [ML Tests](../../ml/tests/README.md)
- [Database Schema](../../migrations/README.md)
## 📜 License
Part of the Foxhunt trading system. Same license as parent project.