foxhunt/tests/e2e

Foxhunt E2E Testing Framework

A comprehensive End-to-End testing framework for the Foxhunt High-Frequency Trading system. This framework tests the complete integration between TLI client, all three services (Trading, Backtesting, ML Training), database interactions, ML model inference, and complete trading workflows.

🎯 Overview

The E2E testing framework provides:

• Service Orchestration: Automated startup/shutdown of all services
• gRPC Client Testing: Authentication, streaming, and error handling
• Database Integration: Transaction management and configuration hot-reload
• ML Pipeline Testing: Model inference, training, and ensemble predictions
• Complete Workflow Testing: End-to-end trading scenarios
• Performance Benchmarking: Load testing and performance metrics
• Corrode-MCP Integration: Advanced test execution and reporting

🏗️ Architecture

tests/e2e/
├── Cargo.toml                    # Project configuration
├── build.rs                     # gRPC proto compilation
├── src/
│   ├── lib.rs                   # Main library and test macros
│   ├── framework.rs             # Core E2E testing framework
│   ├── services.rs              # Service management and orchestration
│   ├── clients.rs               # gRPC test clients
│   ├── database.rs              # Database testing harness
│   ├── ml_pipeline.rs           # ML model testing framework
│   ├── workflows.rs             # Complete trading workflow tests
│   ├── utils.rs                 # Test utilities and data generation
│   ├── corrode.rs               # Corrode-MCP integration
│   └── bin/
│       ├── test_runner.rs       # Test execution runner
│       └── service_orchestrator.rs # Service management tool
├── tests/
│   └── integration_test.rs      # Example integration tests
└── README.md                    # This file

🚀 Quick Start

Prerequisites

1. Rust Toolchain: Ensure you have Rust 1.75+ installed
2. PostgreSQL: Running instance for database tests
3. Corrode-MCP: Install corrode for advanced test execution

# Install corrode-mcp (if not already installed)
cargo install corrode-mcp

# Set up environment
export DATABASE_URL="postgresql://localhost/foxhunt_test"
export RUST_LOG="info"

Running Tests

Option 1: Using Test Runner (Recommended)

# Build the test runner
cargo build --bin test_runner --release

# Run all E2E tests
./target/release/test_runner run --test all

# Run specific test categories
./target/release/test_runner run --test trading --parallel 2
./target/release/test_runner run --test ml --verbose
./target/release/test_runner run --test smoke --fail-fast

# List available tests
./target/release/test_runner list

# Generate test report
./target/release/test_runner report --results-dir ./test-results --format html

Option 2: Using Service Orchestrator

# Build the service orchestrator
cargo build --bin service_orchestrator --release

# Start all services for testing
./target/release/service_orchestrator start --services all --wait

# Check service status
./target/release/service_orchestrator status

# Run specific tests against running services
cargo test --package foxhunt-e2e

# Stop services when done
./target/release/service_orchestrator stop --services all

Option 3: Direct Cargo Testing

# Run all integration tests
cargo test --package foxhunt-e2e

# Run specific test
cargo test --package foxhunt-e2e test_complete_trading_workflow

# Run with output
cargo test --package foxhunt-e2e -- --nocapture

📋 Test Categories

🔧 Service Tests

• service_startup: Verify all services start and respond to health checks
• service_shutdown: Test graceful service shutdown
• service_recovery: Test service recovery after failures

🗄️ Database Tests

• database_integration: Test PostgreSQL integration and queries
• database_migrations: Test database schema migrations
• database_performance: Test database query performance

📡 gRPC Tests

• grpc_clients: Test all gRPC client connections and authentication
• grpc_streaming: Test streaming gRPC calls (market data, order updates)
• grpc_error_handling: Test gRPC error scenarios and recovery

🤖 ML Pipeline Tests

• ml_inference: Test ML model inference pipelines
• ml_training: Test ML model training workflows
• ml_ensemble: Test ensemble prediction workflows

💼 Trading Tests

• trading_workflows: Complete trading workflow tests
• order_lifecycle: Order submission to execution lifecycle
• risk_management: Risk management and safety mechanisms
• emergency_stop: Emergency stop and kill switch tests

🎯 Full Suite

• all: Run complete E2E test suite
• smoke: Run smoke tests for quick validation
• performance: Run performance and load tests

🛠️ Framework Components

E2ETestFramework

The core framework that orchestrates all components:

use foxhunt_e2e::{e2e_test, framework::E2ETestFramework};

e2e_test!(my_test, |framework: E2ETestFramework| async {
    // Your test logic here
    let tli_client = framework.get_tli_client().await?;
    let health = framework.check_services_health().await?;
    assert!(health.all_healthy);
    Ok(())
});

Service Management

Automated service lifecycle management:

use foxhunt_e2e::services::ServiceManager;

let mut manager = ServiceManager::new();
manager.start_all_services().await?;
// Tests run here
manager.stop_all_services().await?;

gRPC Clients

Type-safe gRPC client implementations:

use foxhunt_e2e::clients::{TradingServiceClient, MLTrainingServiceClient};

let mut trading = TradingServiceClient::new("http://localhost:50051").await?;
let portfolio = trading.get_portfolio().await?;

let mut ml = MLTrainingServiceClient::new("http://localhost:50053").await?;
let prediction = ml.predict(features).await?;

Database Testing

Transaction-isolated database testing:

use foxhunt_e2e::database::DatabaseTestHarness;

let db = DatabaseTestHarness::new().await?;
let mut tx = db.begin_test_transaction().await?;
// Database operations here - will auto-rollback

ML Pipeline Testing

Mock ML models for testing:

use foxhunt_e2e::ml_pipeline::MLPipelineTestHarness;

let ml = MLPipelineTestHarness::new().await?;
let result = ml.test_model_inference("mamba", features).await?;
let ensemble = ml.test_ensemble_prediction(features).await?;

🎛️ Configuration

Environment Variables

• DATABASE_URL: PostgreSQL connection string for test database
• RUST_LOG: Log level (debug, info, warn, error)
• FOXHUNT_TEST_MODE: Set to "true" for test mode
• CUDA_VISIBLE_DEVICES: GPU configuration for ML tests
• TORCH_DEVICE: PyTorch device (cpu/cuda) for ML tests

Test Configuration

# tests/e2e/Cargo.toml
[package.metadata.e2e]
default_timeout = 600
max_parallel_sessions = 4
service_startup_timeout = 120
database_url = "postgresql://localhost/foxhunt_test"

📊 Performance Benchmarks

The framework includes comprehensive performance testing:

Order Submission Performance

• Target: >10 orders/second
• Success rate: >90%
• Latency: <100ms average

ML Inference Performance

• Target: >20 inferences/second
• Latency: <50ms average
• GPU utilization monitoring

Database Performance

• Query execution time monitoring
• Connection pool performance
• Transaction throughput

🔍 Debugging and Troubleshooting

Enable Debug Logging

export RUST_LOG=debug
cargo test --package foxhunt-e2e -- --nocapture

Service Logs

# View service logs
./target/release/service_orchestrator logs trading --follow

# Check service status
./target/release/service_orchestrator status

Database Issues

# Check database connection
psql $DATABASE_URL -c "SELECT 1;"

# Reset test database
dropdb foxhunt_test && createdb foxhunt_test

Common Issues

1. Service startup timeouts: Increase startup_timeout in service configs
2. gRPC connection errors: Verify services are running and ports are correct
3. Database connection failures: Check PostgreSQL is running and credentials
4. ML model loading errors: Ensure model files exist or use mock models

🧪 Writing Custom Tests

Basic Test Structure

use foxhunt_e2e::{e2e_test, framework::E2ETestFramework};
use anyhow::Result;

e2e_test!(test_my_feature, |framework: E2ETestFramework| async {
    // Test setup
    let client = framework.get_tli_client().await?;
    
    // Test execution
    let result = client.my_operation().await?;
    
    // Assertions
    assert!(result.success, "Operation failed");
    
    // Cleanup (automatic)
    Ok(())
});

Advanced Test Features

e2e_test!(test_complex_workflow, |framework: E2ETestFramework| async {
    // Use test data generator
    let mut generator = TestDataGenerator::new();
    let market_data = generator.generate_market_data()?;
    
    // Measure performance
    let (result, duration) = TestUtils::measure_execution_time(|| async {
        // Your operation here
        Ok(42)
    }).await?;
    
    // Database testing
    let db = &framework.database_harness;
    let mut tx = db.begin_test_transaction().await?;
    // Database operations...
    
    // ML testing
    let ml = &framework.ml_pipeline;
    let prediction = ml.test_ensemble_prediction(features).await?;
    
    Ok(())
});

📈 Continuous Integration

GitHub Actions Example

name: E2E Tests
on: [push, pull_request]

jobs:
  e2e-tests:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: foxhunt_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    
    steps:
      - uses: actions/checkout@v3
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      
      - name: Install corrode-mcp
        run: cargo install corrode-mcp
      
      - name: Run E2E tests
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost/foxhunt_test
          RUST_LOG: info
        run: |
          cargo build --bin service_orchestrator --release
          ./target/release/service_orchestrator start --services all --wait --background &
          sleep 10
          cargo test --package foxhunt-e2e

🤝 Contributing

1. Add new tests: Create new test functions using the e2e_test! macro
2. Extend framework: Add new components to the framework modules
3. Improve performance: Optimize test execution and resource usage
4. Documentation: Update this README and code documentation

Test Naming Convention

• test_[component]_[scenario]: e.g., test_trading_order_lifecycle
• Use descriptive names that explain what is being tested
• Group related tests in the same file

Code Style

• Follow Rust standard formatting (cargo fmt)
• Add comprehensive error handling
• Include informative log messages
• Write clear assertions with descriptive failure messages

📝 License

This E2E testing framework is part of the Foxhunt HFT Trading System and follows the same license terms as the main project.