foxhunt/testing/e2e/E2E_TEST_GUIDE.md

# Foxhunt E2E Test Suite - Comprehensive Guide

## 📋 Overview

This directory contains comprehensive end-to-end (E2E) integration tests for the Foxhunt HFT Trading System. The tests validate complete workflows across multiple services, ensuring system reliability, performance, and correctness.

## 🎯 Test Categories

### 1. **Core Trading Flow Tests** (`full_trading_flow_e2e.rs`)
Complete trading workflow validation:
- ✅ Market data subscription
- ✅ Order submission and validation
- ✅ Risk management checks
- ✅ Order execution and fills
- ✅ Position updates
- ✅ P&L calculation
- ✅ Account balance updates
- ✅ Order lifecycle with cancellation
- ✅ Risk limit enforcement

**Key Tests:**
- `test_complete_trading_workflow` - Full end-to-end trading flow
- `test_order_lifecycle_with_cancellation` - Order management
- `test_risk_limit_enforcement` - Risk controls

### 2. **ML Inference Tests** (`ml_inference_e2e.rs`)
Machine learning model integration:
- ✅ Market data → feature extraction
- ✅ Real-time model inference (DQN, PPO, MAMBA, TFT, TLOB)
- ✅ Ensemble prediction aggregation
- ✅ Trading signal generation
- ✅ Model performance monitoring
- ✅ Prediction accuracy validation
- ✅ Batch vs streaming consistency

**Key Tests:**
- `test_complete_ml_inference_pipeline` - Full ML pipeline
- `test_ml_model_failover` - Graceful degradation
- `test_ml_performance_benchmarks` - Performance validation

### 3. **Risk Management Tests** (`risk_management_e2e.rs`)
Comprehensive risk system validation:
- ✅ VaR (Value at Risk) calculations
- ✅ Position risk assessment
- ✅ Portfolio exposure monitoring
- ✅ Circuit breaker activation
- ✅ Emergency stop functionality
- ✅ Risk alert system
- ✅ Compliance monitoring

**Key Tests:**
- `test_complete_risk_management_system` - Full risk system
- Portfolio VaR calculation
- Position risk assessment
- Risk metrics validation

### 4. **Multi-Service Integration** (NEW: `multi_service_integration.rs`)
Cross-service workflow validation:
- ✅ Trading Service + ML Training Service integration
- ✅ Trading Service + Backtesting Service integration
- ✅ Full multi-service data flow
- ✅ Service coordination and communication
- ✅ Configuration transfer between services

**Key Tests:**
- `test_trading_ml_integration` - Trading + ML coordination
- `test_trading_backtesting_integration` - Trading + Backtesting
- `test_full_multi_service_workflow` - Complete workflow

### 5. **Error Handling & Recovery** (NEW: `error_handling_recovery.rs`)
System resilience validation:
- ✅ Invalid order handling
- ✅ Service timeout handling
- ✅ ML model failure graceful degradation
- ✅ Concurrent error scenarios
- ✅ Data validation and sanitization

**Key Tests:**
- `test_invalid_order_handling` - Input validation
- `test_service_timeout_handling` - Timeout management
- `test_ml_model_failure_graceful_degradation` - Failover
- `test_concurrent_error_handling` - Concurrent resilience
- `test_data_validation_and_sanitization` - Edge cases

### 6. **Performance & Load Tests** (NEW: `performance_load_tests.rs`)
System performance validation:
- ✅ Order submission throughput
- ✅ Concurrent order processing
- ✅ Market data processing throughput
- ✅ ML inference performance
- ✅ Latency percentiles (p50, p95, p99)
- ✅ Sustained load testing

**Key Tests:**
- `test_order_submission_throughput` - Order rate validation
- `test_concurrent_order_processing` - Concurrent user simulation
- `test_market_data_processing_throughput` - Data pipeline
- `test_ml_inference_performance` - ML performance
- `test_latency_percentiles` - SLA validation
- `test_sustained_load` - Endurance testing

### 7. **Simplified Integration Tests** (NEW: `simplified_integration_test.rs`)
Basic unit-like integration tests:
- ✅ Type and structure validation
- ✅ Market data structures
- ✅ Order validation logic
- ✅ Risk calculation logic
- ✅ Feature extraction logic
- ✅ Concurrent operations
- ✅ Error handling patterns
- ✅ Data serialization
- ✅ Timestamp handling
- ✅ Collection operations

**Key Tests:**
- Basic type validation without services
- Standalone logic testing
- No external dependencies required

## 🏗️ Test Infrastructure

### Core Components

#### **E2ETestFramework** (`src/framework.rs`)
Main orchestration framework providing:
- Service lifecycle management
- gRPC client connections
- Database testing harness
- ML pipeline testing
- Performance monitoring
- Test data management

#### **ServiceManager** (`src/services.rs`)
Service orchestration:
- Start/stop all services
- Health monitoring
- Port management
- Process lifecycle

#### **MLPipelineTestHarness** (`src/ml_pipeline.rs`)
ML testing infrastructure:
- Model health checks
- Feature extraction
- Inference testing
- Ensemble predictions
- Performance metrics

#### **PerformanceTracker** (`src/performance.rs`)
Performance monitoring:
- Metric recording
- Latency tracking
- Throughput measurement
- Report generation

### Test Utilities

#### **Test Data Generation**
- `generate_market_data()` - Realistic market ticks
- `generate_test_order()` - Order generation
- `generate_comprehensive_market_data()` - Multi-symbol data
- `generate_validation_market_data()` - Known patterns

#### **Helper Functions**
- `wait_for_condition()` - Async condition polling
- Market data processing utilities
- Order validation helpers

## 🚀 Running Tests

### Run All E2E Tests
```bash
cargo test -p e2e_tests --no-fail-fast
```

### Run Specific Test Suite
```bash
# Trading flow tests
cargo test -p e2e_tests --test full_trading_flow_e2e

# ML inference tests
cargo test -p e2e_tests --test ml_inference_e2e

# Multi-service integration
cargo test -p e2e_tests --test multi_service_integration

# Error handling tests
cargo test -p e2e_tests --test error_handling_recovery

# Performance tests
cargo test -p e2e_tests --test performance_load_tests

# Simplified tests (no services required)
cargo test -p e2e_tests --test simplified_integration_test
```

### Run Specific Test
```bash
cargo test -p e2e_tests --test full_trading_flow_e2e test_complete_trading_workflow
```

### Run with Logging
```bash
RUST_LOG=info cargo test -p e2e_tests --test full_trading_flow_e2e -- --nocapture
```

### Run in Release Mode (Performance)
```bash
cargo test -p e2e_tests --release --test performance_load_tests
```

## 📊 Test Coverage Summary

### Existing Tests (Original)
- **Full Trading Flow**: 3 comprehensive tests
- **ML Inference**: 3 model pipeline tests
- **Risk Management**: Complete risk system validation
- **Config Hot Reload**: Configuration management
- **Compliance & Regulatory**: SOX, MiFID II compliance
- **Emergency Shutdown**: Failover scenarios
- **Data Flow Performance**: Pipeline validation
- **Order Lifecycle & Risk**: Combined testing

**Total Existing**: ~50+ test scenarios across 14 files

### New Tests Added
1. **Simplified Integration** - 10 basic tests
2. **Multi-Service Integration** - 3 service coordination tests
3. **Error Handling & Recovery** - 5 resilience tests
4. **Performance & Load Tests** - 6 performance tests

**Total New**: 24 new test scenarios

### Combined Total
- **~74+ test scenarios** across 18 test files
- **7 major test categories**
- **Complete system coverage**

## 🎯 Test Objectives

### Functional Testing
- ✅ Order submission and execution
- ✅ Risk management and compliance
- ✅ ML model inference and predictions
- ✅ Configuration management
- ✅ Data flow and processing

### Integration Testing
- ✅ Service-to-service communication
- ✅ gRPC API validation
- ✅ Database interactions
- ✅ Multi-service workflows

### Performance Testing
- ✅ Throughput measurement
- ✅ Latency validation
- ✅ Load testing
- ✅ Concurrent operations
- ✅ Resource utilization

### Reliability Testing
- ✅ Error handling
- ✅ Failure recovery
- ✅ Graceful degradation
- ✅ Circuit breakers
- ✅ Timeout handling

## 📈 Performance Targets

### Latency SLAs
- **p50 (median)**: < 50ms
- **p95**: < 100ms
- **p99**: < 200ms

### Throughput Targets
- **Order submission**: > 10 orders/sec
- **Market data processing**: > 1,000 ticks/sec
- **ML inference**: < 100ms (batch)

### Reliability Targets
- **Success rate**: > 95%
- **Uptime**: 99.9%
- **Error recovery**: < 1s

## 🔧 Test Configuration

### Environment Variables
```bash
# Service endpoints
TRADING_SERVICE_URL=http://localhost:50051
BACKTESTING_SERVICE_URL=http://localhost:50052
ML_TRAINING_SERVICE_URL=http://localhost:50053

# Database
DATABASE_URL=postgresql://localhost/foxhunt_test

# Test settings
E2E_TEST_TIMEOUT=300  # seconds
E2E_LOG_LEVEL=info
```

### Test Data
- Market data generated programmatically
- No Redis/Postgres required for basic tests
- Mocks available for offline testing

## 🐛 Troubleshooting

### Common Issues

#### Test Compilation Errors
```bash
# Clean and rebuild
cargo clean
cargo build -p e2e_tests
```

#### Service Connection Failures
- Verify services are running
- Check port availability
- Review service health endpoints

#### Timeout Issues
- Increase test timeouts
- Check system resources
- Review service logs

### Debug Mode
```bash
RUST_LOG=debug cargo test -p e2e_tests -- --nocapture
```

## 📝 Adding New Tests

### Basic Structure
```rust
use e2e_tests::{e2e_test, E2ETestFramework};

e2e_test!(
    test_my_feature,
    |mut framework: E2ETestFramework| async {
        // Test implementation
        Ok(())
    }
);
```

### Best Practices
1. Use the `e2e_test!` macro for standardization
2. Record performance metrics
3. Add comprehensive assertions
4. Include cleanup logic
5. Document test purpose and coverage

## 🎓 Test Patterns

### Pattern 1: Service Health Check
```rust
let health = framework.check_services_health().await?;
assert!(health.all_healthy);
```

### Pattern 2: Client Retrieval
```rust
let trading_client = framework.get_trading_client().await?;
```

### Pattern 3: Performance Tracking
```rust
framework.performance_tracker.record_metric("metric_name", value)?;
```

### Pattern 4: Error Handling
```rust
match result {
    Ok(response) => { /* handle success */ },
    Err(e) => { /* validate error */ }
}
```

## 📊 Test Results

### Viewing Results
Test results include:
- Pass/fail status
- Execution time
- Performance metrics
- Error details
- Coverage information

### Metrics Dashboard
Performance metrics are recorded and can be analyzed:
- Latency distributions
- Throughput trends
- Error rates
- Resource utilization

## 🔮 Future Enhancements

### Planned Additions
1. **Chaos Engineering Tests**
   - Random service failures
   - Network partition simulation
   - Resource exhaustion scenarios

2. **Extended Performance Tests**
   - Soak testing (24+ hours)
   - Spike testing
   - Stress testing to breaking point

3. **Security Tests**
   - Authentication validation
   - Authorization checks
   - Input sanitization
   - SQL injection prevention

4. **Compliance Tests**
   - Extended regulatory scenarios
   - Audit trail validation
   - Best execution verification

## 📚 Related Documentation

- **Architecture**: `/docs/architecture/`
- **API Documentation**: `/docs/api/`
- **Deployment Guide**: `/docs/deployment/`
- **Monitoring Guide**: `/docs/monitoring/`

## 🤝 Contributing

When adding new tests:
1. Follow existing patterns
2. Add documentation
3. Update this guide
4. Include performance metrics
5. Test locally before committing

## 📞 Support

For issues or questions:
- Review test output logs
- Check service status
- Consult architecture documentation
- Review related test files

---

**Last Updated**: 2025-10-01
**Test Coverage**: 74+ scenarios across 18 files
**Status**: Active Development