foxhunt/testing/e2e/E2E_TEST_GUIDE.md

Foxhunt E2E Test Suite - Comprehensive Guide

📋 Overview

This directory contains comprehensive end-to-end (E2E) integration tests for the Foxhunt HFT Trading System. The tests validate complete workflows across multiple services, ensuring system reliability, performance, and correctness.

🎯 Test Categories

1. Core Trading Flow Tests (full_trading_flow_e2e.rs)

Complete trading workflow validation:

• ✅ Market data subscription
• ✅ Order submission and validation
• ✅ Risk management checks
• ✅ Order execution and fills
• ✅ Position updates
• ✅ P&L calculation
• ✅ Account balance updates
• ✅ Order lifecycle with cancellation
• ✅ Risk limit enforcement

Key Tests:

• test_complete_trading_workflow - Full end-to-end trading flow
• test_order_lifecycle_with_cancellation - Order management
• test_risk_limit_enforcement - Risk controls

2. ML Inference Tests (ml_inference_e2e.rs)

Machine learning model integration:

• ✅ Market data → feature extraction
• ✅ Real-time model inference (DQN, PPO, MAMBA, TFT, TLOB)
• ✅ Ensemble prediction aggregation
• ✅ Trading signal generation
• ✅ Model performance monitoring
• ✅ Prediction accuracy validation
• ✅ Batch vs streaming consistency

Key Tests:

• test_complete_ml_inference_pipeline - Full ML pipeline
• test_ml_model_failover - Graceful degradation
• test_ml_performance_benchmarks - Performance validation

3. Risk Management Tests (risk_management_e2e.rs)

Comprehensive risk system validation:

• ✅ VaR (Value at Risk) calculations
• ✅ Position risk assessment
• ✅ Portfolio exposure monitoring
• ✅ Circuit breaker activation
• ✅ Emergency stop functionality
• ✅ Risk alert system
• ✅ Compliance monitoring

Key Tests:

• test_complete_risk_management_system - Full risk system
• Portfolio VaR calculation
• Position risk assessment
• Risk metrics validation

4. Multi-Service Integration (NEW: multi_service_integration.rs)

Cross-service workflow validation:

• ✅ Trading Service + ML Training Service integration
• ✅ Trading Service + Backtesting Service integration
• ✅ Full multi-service data flow
• ✅ Service coordination and communication
• ✅ Configuration transfer between services

Key Tests:

• test_trading_ml_integration - Trading + ML coordination
• test_trading_backtesting_integration - Trading + Backtesting
• test_full_multi_service_workflow - Complete workflow

5. Error Handling & Recovery (NEW: error_handling_recovery.rs)

System resilience validation:

• ✅ Invalid order handling
• ✅ Service timeout handling
• ✅ ML model failure graceful degradation
• ✅ Concurrent error scenarios
• ✅ Data validation and sanitization

Key Tests:

• test_invalid_order_handling - Input validation
• test_service_timeout_handling - Timeout management
• test_ml_model_failure_graceful_degradation - Failover
• test_concurrent_error_handling - Concurrent resilience
• test_data_validation_and_sanitization - Edge cases

6. Performance & Load Tests (NEW: performance_load_tests.rs)

System performance validation:

• ✅ Order submission throughput
• ✅ Concurrent order processing
• ✅ Market data processing throughput
• ✅ ML inference performance
• ✅ Latency percentiles (p50, p95, p99)
• ✅ Sustained load testing

Key Tests:

• test_order_submission_throughput - Order rate validation
• test_concurrent_order_processing - Concurrent user simulation
• test_market_data_processing_throughput - Data pipeline
• test_ml_inference_performance - ML performance
• test_latency_percentiles - SLA validation
• test_sustained_load - Endurance testing

7. Simplified Integration Tests (NEW: simplified_integration_test.rs)

Basic unit-like integration tests:

• ✅ Type and structure validation
• ✅ Market data structures
• ✅ Order validation logic
• ✅ Risk calculation logic
• ✅ Feature extraction logic
• ✅ Concurrent operations
• ✅ Error handling patterns
• ✅ Data serialization
• ✅ Timestamp handling
• ✅ Collection operations

Key Tests:

• Basic type validation without services
• Standalone logic testing
• No external dependencies required

🏗️ Test Infrastructure

Core Components

E2ETestFramework (src/framework.rs)

Main orchestration framework providing:

• Service lifecycle management
• gRPC client connections
• Database testing harness
• ML pipeline testing
• Performance monitoring
• Test data management

ServiceManager (src/services.rs)

Service orchestration:

• Start/stop all services
• Health monitoring
• Port management
• Process lifecycle

MLPipelineTestHarness (src/ml_pipeline.rs)

ML testing infrastructure:

• Model health checks
• Feature extraction
• Inference testing
• Ensemble predictions
• Performance metrics

PerformanceTracker (src/performance.rs)

Performance monitoring:

• Metric recording
• Latency tracking
• Throughput measurement
• Report generation

Test Utilities

Test Data Generation

• generate_market_data() - Realistic market ticks
• generate_test_order() - Order generation
• generate_comprehensive_market_data() - Multi-symbol data
• generate_validation_market_data() - Known patterns

Helper Functions

• wait_for_condition() - Async condition polling
• Market data processing utilities
• Order validation helpers

🚀 Running Tests

Run All E2E Tests

cargo test -p e2e_tests --no-fail-fast

Run Specific Test Suite

# Trading flow tests
cargo test -p e2e_tests --test full_trading_flow_e2e

# ML inference tests
cargo test -p e2e_tests --test ml_inference_e2e

# Multi-service integration
cargo test -p e2e_tests --test multi_service_integration

# Error handling tests
cargo test -p e2e_tests --test error_handling_recovery

# Performance tests
cargo test -p e2e_tests --test performance_load_tests

# Simplified tests (no services required)
cargo test -p e2e_tests --test simplified_integration_test

Run Specific Test

cargo test -p e2e_tests --test full_trading_flow_e2e test_complete_trading_workflow

Run with Logging

RUST_LOG=info cargo test -p e2e_tests --test full_trading_flow_e2e -- --nocapture

Run in Release Mode (Performance)

cargo test -p e2e_tests --release --test performance_load_tests

📊 Test Coverage Summary

Existing Tests (Original)

• Full Trading Flow: 3 comprehensive tests
• ML Inference: 3 model pipeline tests
• Risk Management: Complete risk system validation
• Config Hot Reload: Configuration management
• Compliance & Regulatory: SOX, MiFID II compliance
• Emergency Shutdown: Failover scenarios
• Data Flow Performance: Pipeline validation
• Order Lifecycle & Risk: Combined testing

Total Existing: ~50+ test scenarios across 14 files

New Tests Added

1. Simplified Integration - 10 basic tests
2. Multi-Service Integration - 3 service coordination tests
3. Error Handling & Recovery - 5 resilience tests
4. Performance & Load Tests - 6 performance tests

Total New: 24 new test scenarios

Combined Total

• ~74+ test scenarios across 18 test files
• 7 major test categories
• Complete system coverage

🎯 Test Objectives

Functional Testing

• ✅ Order submission and execution
• ✅ Risk management and compliance
• ✅ ML model inference and predictions
• ✅ Configuration management
• ✅ Data flow and processing

Integration Testing

• ✅ Service-to-service communication
• ✅ gRPC API validation
• ✅ Database interactions
• ✅ Multi-service workflows

Performance Testing

• ✅ Throughput measurement
• ✅ Latency validation
• ✅ Load testing
• ✅ Concurrent operations
• ✅ Resource utilization

Reliability Testing

• ✅ Error handling
• ✅ Failure recovery
• ✅ Graceful degradation
• ✅ Circuit breakers
• ✅ Timeout handling

📈 Performance Targets

Latency SLAs

• p50 (median): < 50ms
• p95: < 100ms
• p99: < 200ms

Throughput Targets

• Order submission: > 10 orders/sec
• Market data processing: > 1,000 ticks/sec
• ML inference: < 100ms (batch)

Reliability Targets

• Success rate: > 95%
• Uptime: 99.9%
• Error recovery: < 1s

🔧 Test Configuration

Environment Variables

# Service endpoints
TRADING_SERVICE_URL=http://localhost:50051
BACKTESTING_SERVICE_URL=http://localhost:50052
ML_TRAINING_SERVICE_URL=http://localhost:50053

# Database
DATABASE_URL=postgresql://localhost/foxhunt_test

# Test settings
E2E_TEST_TIMEOUT=300  # seconds
E2E_LOG_LEVEL=info

Test Data

• Market data generated programmatically
• No Redis/Postgres required for basic tests
• Mocks available for offline testing

🐛 Troubleshooting

Common Issues

Test Compilation Errors

# Clean and rebuild
cargo clean
cargo build -p e2e_tests

Service Connection Failures

• Verify services are running
• Check port availability
• Review service health endpoints

Timeout Issues

• Increase test timeouts
• Check system resources
• Review service logs

Debug Mode

RUST_LOG=debug cargo test -p e2e_tests -- --nocapture

📝 Adding New Tests

Basic Structure

use e2e_tests::{e2e_test, E2ETestFramework};

e2e_test!(
    test_my_feature,
    |mut framework: E2ETestFramework| async {
        // Test implementation
        Ok(())
    }
);

Best Practices

1. Use the e2e_test! macro for standardization
2. Record performance metrics
3. Add comprehensive assertions
4. Include cleanup logic
5. Document test purpose and coverage

🎓 Test Patterns

Pattern 1: Service Health Check

let health = framework.check_services_health().await?;
assert!(health.all_healthy);

Pattern 2: Client Retrieval

let trading_client = framework.get_trading_client().await?;

Pattern 3: Performance Tracking

framework.performance_tracker.record_metric("metric_name", value)?;

Pattern 4: Error Handling

match result {
    Ok(response) => { /* handle success */ },
    Err(e) => { /* validate error */ }
}

📊 Test Results

Viewing Results

Test results include:

• Pass/fail status
• Execution time
• Performance metrics
• Error details
• Coverage information

Metrics Dashboard

Performance metrics are recorded and can be analyzed:

• Latency distributions
• Throughput trends
• Error rates
• Resource utilization

🔮 Future Enhancements

Planned Additions

1. Chaos Engineering Tests

   • Random service failures
   • Network partition simulation
   • Resource exhaustion scenarios

2. Extended Performance Tests

   • Soak testing (24+ hours)
   • Spike testing
   • Stress testing to breaking point

3. Security Tests

   • Authentication validation
   • Authorization checks
   • Input sanitization
   • SQL injection prevention

4. Compliance Tests

   • Extended regulatory scenarios
   • Audit trail validation
   • Best execution verification

📚 Related Documentation

• Architecture: /docs/architecture/
• API Documentation: /docs/api/
• Deployment Guide: /docs/deployment/
• Monitoring Guide: /docs/monitoring/

🤝 Contributing

When adding new tests:

1. Follow existing patterns
2. Add documentation
3. Update this guide
4. Include performance metrics
5. Test locally before committing

📞 Support

For issues or questions:

• Review test output logs
• Check service status
• Consult architecture documentation
• Review related test files

---

Last Updated: 2025-10-01 Test Coverage: 74+ scenarios across 18 files Status: Active Development