foxhunt/tests/e2e/README.md

# Foxhunt E2E Testing Framework

A comprehensive End-to-End testing framework for the Foxhunt High-Frequency Trading system. This framework tests the complete integration between TLI client, all three services (Trading, Backtesting, ML Training), database interactions, ML model inference, and complete trading workflows.

## 🎯 Overview

The E2E testing framework provides:

- **Service Orchestration**: Automated startup/shutdown of all services
- **gRPC Client Testing**: Authentication, streaming, and error handling
- **Database Integration**: Transaction management and configuration hot-reload
- **ML Pipeline Testing**: Model inference, training, and ensemble predictions
- **Complete Workflow Testing**: End-to-end trading scenarios
- **Performance Benchmarking**: Load testing and performance metrics
- **Corrode-MCP Integration**: Advanced test execution and reporting

## 🏗️ Architecture

```
tests/e2e/
├── Cargo.toml                    # Project configuration
├── build.rs                     # gRPC proto compilation
├── src/
│   ├── lib.rs                   # Main library and test macros
│   ├── framework.rs             # Core E2E testing framework
│   ├── services.rs              # Service management and orchestration
│   ├── clients.rs               # gRPC test clients
│   ├── database.rs              # Database testing harness
│   ├── ml_pipeline.rs           # ML model testing framework
│   ├── workflows.rs             # Complete trading workflow tests
│   ├── utils.rs                 # Test utilities and data generation
│   ├── corrode.rs               # Corrode-MCP integration
│   └── bin/
│       ├── test_runner.rs       # Test execution runner
│       └── service_orchestrator.rs # Service management tool
├── tests/
│   └── integration_test.rs      # Example integration tests
└── README.md                    # This file
```

## 🚀 Quick Start

### Prerequisites

1. **Rust Toolchain**: Ensure you have Rust 1.75+ installed
2. **PostgreSQL**: Running instance for database tests
3. **Corrode-MCP**: Install corrode for advanced test execution

```bash
# Install corrode-mcp (if not already installed)
cargo install corrode-mcp

# Set up environment
export DATABASE_URL="postgresql://localhost/foxhunt_test"
export RUST_LOG="info"
```

### Running Tests

#### Option 1: Using Test Runner (Recommended)

```bash
# Build the test runner
cargo build --bin test_runner --release

# Run all E2E tests
./target/release/test_runner run --test all

# Run specific test categories
./target/release/test_runner run --test trading --parallel 2
./target/release/test_runner run --test ml --verbose
./target/release/test_runner run --test smoke --fail-fast

# List available tests
./target/release/test_runner list

# Generate test report
./target/release/test_runner report --results-dir ./test-results --format html
```

#### Option 2: Using Service Orchestrator

```bash
# Build the service orchestrator
cargo build --bin service_orchestrator --release

# Start all services for testing
./target/release/service_orchestrator start --services all --wait

# Check service status
./target/release/service_orchestrator status

# Run specific tests against running services
cargo test --package foxhunt-e2e

# Stop services when done
./target/release/service_orchestrator stop --services all
```

#### Option 3: Direct Cargo Testing

```bash
# Run all integration tests
cargo test --package foxhunt-e2e

# Run specific test
cargo test --package foxhunt-e2e test_complete_trading_workflow

# Run with output
cargo test --package foxhunt-e2e -- --nocapture
```

## 📋 Test Categories

### 🔧 Service Tests
- **service_startup**: Verify all services start and respond to health checks
- **service_shutdown**: Test graceful service shutdown
- **service_recovery**: Test service recovery after failures

### 🗄️ Database Tests
- **database_integration**: Test PostgreSQL integration and queries
- **database_migrations**: Test database schema migrations
- **database_performance**: Test database query performance

### 📡 gRPC Tests
- **grpc_clients**: Test all gRPC client connections and authentication
- **grpc_streaming**: Test streaming gRPC calls (market data, order updates)
- **grpc_error_handling**: Test gRPC error scenarios and recovery

### 🤖 ML Pipeline Tests
- **ml_inference**: Test ML model inference pipelines
- **ml_training**: Test ML model training workflows
- **ml_ensemble**: Test ensemble prediction workflows

### 💼 Trading Tests
- **trading_workflows**: Complete trading workflow tests
- **order_lifecycle**: Order submission to execution lifecycle
- **risk_management**: Risk management and safety mechanisms
- **emergency_stop**: Emergency stop and kill switch tests

### 🎯 Full Suite
- **all**: Run complete E2E test suite
- **smoke**: Run smoke tests for quick validation
- **performance**: Run performance and load tests

## 🛠️ Framework Components

### E2ETestFramework
The core framework that orchestrates all components:

```rust
use foxhunt_e2e::{e2e_test, framework::E2ETestFramework};

e2e_test!(my_test, |framework: E2ETestFramework| async {
    // Your test logic here
    let tli_client = framework.get_tli_client().await?;
    let health = framework.check_services_health().await?;
    assert!(health.all_healthy);
    Ok(())
});
```

### Service Management
Automated service lifecycle management:

```rust
use foxhunt_e2e::services::ServiceManager;

let mut manager = ServiceManager::new();
manager.start_all_services().await?;
// Tests run here
manager.stop_all_services().await?;
```

### gRPC Clients
Type-safe gRPC client implementations:

```rust
use foxhunt_e2e::clients::{TradingServiceClient, MLTrainingServiceClient};

let mut trading = TradingServiceClient::new("http://localhost:50051").await?;
let portfolio = trading.get_portfolio().await?;

let mut ml = MLTrainingServiceClient::new("http://localhost:50053").await?;
let prediction = ml.predict(features).await?;
```

### Database Testing
Transaction-isolated database testing:

```rust
use foxhunt_e2e::database::DatabaseTestHarness;

let db = DatabaseTestHarness::new().await?;
let mut tx = db.begin_test_transaction().await?;
// Database operations here - will auto-rollback
```

### ML Pipeline Testing
Mock ML models for testing:

```rust
use foxhunt_e2e::ml_pipeline::MLPipelineTestHarness;

let ml = MLPipelineTestHarness::new().await?;
let result = ml.test_model_inference("mamba", features).await?;
let ensemble = ml.test_ensemble_prediction(features).await?;
```

## 🎛️ Configuration

### Environment Variables

- `DATABASE_URL`: PostgreSQL connection string for test database
- `RUST_LOG`: Log level (debug, info, warn, error)
- `FOXHUNT_TEST_MODE`: Set to "true" for test mode
- `CUDA_VISIBLE_DEVICES`: GPU configuration for ML tests
- `TORCH_DEVICE`: PyTorch device (cpu/cuda) for ML tests

### Test Configuration

```toml
# tests/e2e/Cargo.toml
[package.metadata.e2e]
default_timeout = 600
max_parallel_sessions = 4
service_startup_timeout = 120
database_url = "postgresql://localhost/foxhunt_test"
```

## 📊 Performance Benchmarks

The framework includes comprehensive performance testing:

### Order Submission Performance
- Target: >10 orders/second
- Success rate: >90%
- Latency: <100ms average

### ML Inference Performance
- Target: >20 inferences/second
- Latency: <50ms average
- GPU utilization monitoring

### Database Performance
- Query execution time monitoring
- Connection pool performance
- Transaction throughput

## 🔍 Debugging and Troubleshooting

### Enable Debug Logging
```bash
export RUST_LOG=debug
cargo test --package foxhunt-e2e -- --nocapture
```

### Service Logs
```bash
# View service logs
./target/release/service_orchestrator logs trading --follow

# Check service status
./target/release/service_orchestrator status
```

### Database Issues
```bash
# Check database connection
psql $DATABASE_URL -c "SELECT 1;"

# Reset test database
dropdb foxhunt_test && createdb foxhunt_test
```

### Common Issues

1. **Service startup timeouts**: Increase `startup_timeout` in service configs
2. **gRPC connection errors**: Verify services are running and ports are correct
3. **Database connection failures**: Check PostgreSQL is running and credentials
4. **ML model loading errors**: Ensure model files exist or use mock models

## 🧪 Writing Custom Tests

### Basic Test Structure

```rust
use foxhunt_e2e::{e2e_test, framework::E2ETestFramework};
use anyhow::Result;

e2e_test!(test_my_feature, |framework: E2ETestFramework| async {
    // Test setup
    let client = framework.get_tli_client().await?;
    
    // Test execution
    let result = client.my_operation().await?;
    
    // Assertions
    assert!(result.success, "Operation failed");
    
    // Cleanup (automatic)
    Ok(())
});
```

### Advanced Test Features

```rust
e2e_test!(test_complex_workflow, |framework: E2ETestFramework| async {
    // Use test data generator
    let mut generator = TestDataGenerator::new();
    let market_data = generator.generate_market_data()?;
    
    // Measure performance
    let (result, duration) = TestUtils::measure_execution_time(|| async {
        // Your operation here
        Ok(42)
    }).await?;
    
    // Database testing
    let db = &framework.database_harness;
    let mut tx = db.begin_test_transaction().await?;
    // Database operations...
    
    // ML testing
    let ml = &framework.ml_pipeline;
    let prediction = ml.test_ensemble_prediction(features).await?;
    
    Ok(())
});
```

## 📈 Continuous Integration

### GitHub Actions Example

```yaml
name: E2E Tests
on: [push, pull_request]

jobs:
  e2e-tests:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: foxhunt_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    
    steps:
      - uses: actions/checkout@v3
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      
      - name: Install corrode-mcp
        run: cargo install corrode-mcp
      
      - name: Run E2E tests
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost/foxhunt_test
          RUST_LOG: info
        run: |
          cargo build --bin service_orchestrator --release
          ./target/release/service_orchestrator start --services all --wait --background &
          sleep 10
          cargo test --package foxhunt-e2e
```

## 🤝 Contributing

1. **Add new tests**: Create new test functions using the `e2e_test!` macro
2. **Extend framework**: Add new components to the framework modules
3. **Improve performance**: Optimize test execution and resource usage
4. **Documentation**: Update this README and code documentation

### Test Naming Convention

- `test_[component]_[scenario]`: e.g., `test_trading_order_lifecycle`
- Use descriptive names that explain what is being tested
- Group related tests in the same file

### Code Style

- Follow Rust standard formatting (`cargo fmt`)
- Add comprehensive error handling
- Include informative log messages
- Write clear assertions with descriptive failure messages

## 📝 License

This E2E testing framework is part of the Foxhunt HFT Trading System and follows the same license terms as the main project.