foxhunt/docs/testing/E2E_INTEGRATION_TESTS.md

# End-to-End Integration Testing Documentation

**Last Updated**: 2025-10-08
**Test Suite Version**: 1.0
**Author**: Agent 112

---

## Overview

This document describes the comprehensive end-to-end (E2E) integration test suite for the Foxhunt HFT Trading System. The test suite validates complete client → gateway → service flows across all microservices.

### Test Coverage Summary

| Test Suite | Test Count | Coverage Area |
|------------|-----------|---------------|
| Trading Service E2E | 15 tests | Order lifecycle, positions, streaming |
| Backtesting Service E2E | 12 tests | Backtest execution, monitoring, results |
| ML Training Service E2E | 12 tests | Training jobs, progress, configuration |
| Service Health & Resilience | 15 tests | Health checks, degradation, failover |
| **TOTAL** | **54 tests** | **Full service integration** |

---

## Architecture

### Test Flow Topology

```
┌─────────────────────────────────────────────────────────┐
│                    Test Client                          │
│            (Generated gRPC Clients)                     │
└──────────────────────┬──────────────────────────────────┘
                       │ JWT Authentication
                       ▼
┌─────────────────────────────────────────────────────────┐
│                 API Gateway (Port 50051)                │
│          Auth Validation + Request Routing              │
└───┬──────────────────┬──────────────────┬───────────────┘
    │                  │                  │
    ▼                  ▼                  ▼
┌──────────┐    ┌──────────────┐    ┌────────────────┐
│ Trading  │    │ Backtesting  │    │  ML Training   │
│ Service  │    │   Service    │    │    Service     │
│Port 50052│    │  Port 50053  │    │  Port 50054    │
└──────────┘    └──────────────┘    └────────────────┘
```

### Authentication Flow

All E2E tests use JWT-based authentication:

1. **Token Generation**: Tests generate JWT tokens with:
   - User ID (e.g., `test_trader_001`)
   - Roles (e.g., `["trader", "admin"]`)
   - Permissions (e.g., `["api.access", "trading.submit"]`)
   - Session ID (unique per test)
   - JTI (JWT ID for revocation tracking)

2. **Token Validation**: API Gateway validates:
   - Signature using shared secret
   - Expiration timestamp
   - Required permissions
   - Revocation status (Redis check)

3. **Request Context**: Authenticated requests include:
   - User context in request extensions
   - Authorization header: `Bearer <token>`
   - Audit logging enabled

---

## Test Suites

### 1. Trading Service E2E Tests

**Location**: `services/integration_tests/tests/trading_service_e2e.rs`
**Test Count**: 15 tests

#### Section 1: Order Submission (5 tests)

1. **test_e2e_order_submission_market_order**
   - Validates: Market order submission through API Gateway
   - Verifies: Order ID returned, success status
   - Flow: Client → API Gateway → Trading Service

2. **test_e2e_order_submission_limit_order**
   - Validates: Limit order with price specification
   - Verifies: Price propagation, order acceptance

3. **test_e2e_order_submission_without_auth**
   - Validates: Authentication enforcement
   - Verifies: Unauthenticated requests rejected with 401

4. **test_e2e_order_cancellation**
   - Validates: Order lifecycle management
   - Verifies: Submit → Cancel flow, status updates

5. **test_e2e_order_status_query**
   - Validates: Order status retrieval
   - Verifies: Status reflects order lifecycle

#### Section 2: Position Management (3 tests)

6. **test_e2e_get_all_positions**
   - Validates: Portfolio position retrieval
   - Verifies: All positions returned

7. **test_e2e_get_position_by_symbol**
   - Validates: Symbol-filtered position query
   - Verifies: Correct position data

8. **test_e2e_get_account_info**
   - Validates: Account balance and metrics
   - Verifies: Buying power, cash balance calculation

#### Section 3: Real-Time Streaming (4 tests)

9. **test_e2e_market_data_subscription**
   - Validates: Real-time market data streaming
   - Verifies: Trade and quote events received

10. **test_e2e_order_updates_subscription**
    - Validates: Order status streaming
    - Verifies: Real-time order fills and updates

11. **test_e2e_concurrent_order_submissions**
    - Validates: Concurrent order handling
    - Verifies: 80%+ success rate for 10 parallel orders

12. **test_e2e_gateway_request_routing**
    - Validates: Multi-endpoint routing
    - Verifies: Different request types routed correctly

#### Section 4: Error Handling (3 tests)

13. **test_e2e_invalid_symbol_handling**
    - Validates: Symbol validation
    - Verifies: Invalid symbols rejected

14. **test_e2e_negative_quantity_validation**
    - Validates: Input validation
    - Verifies: Negative quantities rejected

15. **test_e2e_gateway_timeout_handling**
    - Validates: Timeout management
    - Verifies: Graceful timeout handling

---

### 2. Backtesting Service E2E Tests

**Location**: `services/integration_tests/tests/backtesting_service_e2e.rs`
**Test Count**: 12 tests

#### Section 1: Backtest Lifecycle (5 tests)

1. **test_e2e_backtest_start**
   - Validates: Backtest job creation
   - Verifies: Strategy parameters, date range, initial capital

2. **test_e2e_backtest_status**
   - Validates: Progress monitoring
   - Verifies: Status transitions, progress percentage

3. **test_e2e_backtest_stop**
   - Validates: Graceful job termination
   - Verifies: Partial results saved

4. **test_e2e_backtest_results**
   - Validates: Results retrieval after completion
   - Verifies: Sharpe ratio, returns, trade metrics

5. **test_e2e_backtest_list**
   - Validates: Historical backtest listing
   - Verifies: Pagination, filtering

#### Section 2: Real-Time Monitoring (3 tests)

6. **test_e2e_backtest_progress_subscription**
   - Validates: Streaming progress updates
   - Verifies: Equity curve, PnL updates

7. **test_e2e_backtest_filtering_by_strategy**
   - Validates: Strategy-based filtering
   - Verifies: Correct strategy match

8. **test_e2e_backtest_filtering_by_status**
   - Validates: Status-based filtering
   - Verifies: COMPLETED status filter

#### Section 3: Error Handling (4 tests)

9. **test_e2e_backtest_invalid_date_range**
   - Validates: Date validation
   - Verifies: Start > End rejected

10. **test_e2e_backtest_invalid_capital**
    - Validates: Capital validation
    - Verifies: Negative capital rejected

11. **test_e2e_backtest_nonexistent_status**
    - Validates: Error handling
    - Verifies: 404 for non-existent backtests

12. **test_e2e_backtest_unauthenticated_access**
    - Validates: Security enforcement
    - Verifies: Auth required for all operations

---

### 3. ML Training Service E2E Tests

**Location**: `services/integration_tests/tests/ml_training_service_e2e.rs`
**Test Count**: 12 tests

#### Section 1: Training Job Lifecycle (5 tests)

1. **test_e2e_training_job_start**
   - Validates: Training job creation
   - Verifies: Hyperparameters, resource allocation

2. **test_e2e_training_job_stop**
   - Validates: Job termination
   - Verifies: Graceful shutdown, cleanup

3. **test_e2e_list_training_jobs**
   - Validates: Job listing
   - Verifies: Pagination, metadata

4. **test_e2e_filter_training_jobs_by_model**
   - Validates: Model-based filtering
   - Verifies: DQN/PPO/MAMBA filtering

5. **test_e2e_filter_training_jobs_by_status**
   - Validates: Status filtering
   - Verifies: COMPLETED/RUNNING/FAILED

#### Section 2: Real-Time Monitoring (3 tests)

6. **test_e2e_watch_training_progress**
   - Validates: Streaming training metrics
   - Verifies: Loss, accuracy, epoch progress

7. **test_e2e_resource_utilization**
   - Validates: GPU/CPU monitoring
   - Verifies: Resource availability

8. **test_e2e_stream_resource_metrics**
   - Validates: Real-time resource streaming
   - Verifies: GPU utilization, active jobs

#### Section 3: Configuration (4 tests)

9. **test_e2e_validate_training_config**
   - Validates: Pre-flight validation
   - Verifies: Config warnings/errors

10. **test_e2e_get_training_templates**
    - Validates: Template retrieval
    - Verifies: Model-specific templates

11. **test_e2e_filter_templates_by_model_type**
    - Validates: Template filtering
    - Verifies: DQN/PPO templates

12. **test_e2e_training_with_auto_deploy**
    - Validates: Auto-deployment
    - Verifies: Model deployment on completion

---

### 4. Service Health & Resilience E2E Tests

**Location**: `services/integration_tests/tests/service_health_resilience_e2e.rs`
**Test Count**: 15 tests

#### Section 1: System Health Monitoring (5 tests)

1. **test_e2e_system_health_all_services**
   - Validates: Overall system health
   - Verifies: All services reporting status

2. **test_e2e_system_health_specific_service**
   - Validates: Individual service health
   - Verifies: Trading service status

3. **test_e2e_health_check_interval**
   - Validates: Health update frequency
   - Verifies: Timestamps updating

4. **test_e2e_health_status_transitions**
   - Validates: Status change events
   - Verifies: HEALTHY → DEGRADED transitions

5. **test_e2e_degraded_service_detection**
   - Validates: Degradation detection
   - Verifies: System remains operational

#### Section 2: Graceful Degradation (5 tests)

6. **test_e2e_trading_service_available_backtesting_optional**
   - Validates: Core vs optional services
   - Verifies: Trading works when backtesting down

7. **test_e2e_partial_service_failure_handling**
   - Validates: Partial failure tolerance
   - Verifies: System reports known state

8. **test_e2e_circuit_breaker_validation**
   - Validates: Circuit breaker activation
   - Verifies: Protection from cascading failures

9. **test_e2e_timeout_handling**
   - Validates: Request timeout management
   - Verifies: Graceful timeout responses

10. **test_e2e_retry_logic_validation**
    - Validates: Exponential backoff retries
    - Verifies: 3 retries with backoff

#### Section 3: Service Discovery (5 tests)

11. **test_e2e_api_gateway_routing**
    - Validates: Multi-service routing
    - Verifies: Correct backend selection

12. **test_e2e_service_discovery**
    - Validates: Service registration
    - Verifies: Service metadata available

13. **test_e2e_concurrent_service_requests**
    - Validates: Concurrent request handling
    - Verifies: 80%+ success rate

14. **test_e2e_load_balancing_verification**
    - Validates: Load distribution
    - Verifies: Response time variance

15. **test_e2e_service_failover**
    - Validates: Failover configuration
    - Verifies: Redundancy setup

---

## Running the Tests

### Prerequisites

1. **Infrastructure Services Running**:
   ```bash
   docker-compose up -d postgres redis vault influxdb
   ```

2. **Application Services Running**:
   ```bash
   cargo run -p api_gateway &
   cargo run -p trading_service &
   cargo run -p backtesting_service &
   cargo run -p ml_training_service &
   ```

3. **Database Migrations**:
   ```bash
   cargo sqlx migrate run
   ```

### Execution

#### Run All E2E Tests

```bash
cargo test -p integration_tests --test '*_e2e' -- --ignored
```

#### Run Specific Test Suite

```bash
# Trading service only
cargo test -p integration_tests --test trading_service_e2e -- --ignored

# Backtesting service only
cargo test -p integration_tests --test backtesting_service_e2e -- --ignored

# ML training service only
cargo test -p integration_tests --test ml_training_service_e2e -- --ignored

# Health & resilience only
cargo test -p integration_tests --test service_health_resilience_e2e -- --ignored
```

#### Run Single Test

```bash
cargo test -p integration_tests test_e2e_order_submission_market_order -- --ignored --nocapture
```

### Test Flags

- `--ignored`: Required - all E2E tests are marked with `#[ignore]`
- `--nocapture`: Shows println output
- `--test-threads=1`: Serialize tests (avoid port conflicts)

---

## Test Configuration

### Environment Variables

```bash
# API Gateway endpoint
API_GATEWAY_ADDR=http://localhost:50051

# JWT configuration
JWT_SECRET=dev_secret_key_change_in_production
JWT_ISSUER=foxhunt-api-gateway
JWT_AUDIENCE=foxhunt-services

# Service endpoints (for direct testing)
TRADING_SERVICE_ADDR=http://localhost:50052
BACKTESTING_SERVICE_ADDR=http://localhost:50053
ML_TRAINING_SERVICE_ADDR=http://localhost:50054

# Database
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Redis
REDIS_URL=redis://localhost:6379
```

### Test Users

| User ID | Roles | Permissions |
|---------|-------|-------------|
| test_trader_001 | trader | api.access, trading.submit, trading.view |
| test_backtester_001 | analyst | api.access, backtesting.execute, backtesting.view |
| test_ml_engineer_001 | ml_engineer | api.access, ml.train, ml.view, ml.manage |
| health_monitor | admin | api.access, system.monitor |
| resilience_tester | admin | api.access, system.admin |

---

## Expected Results

### Success Criteria

1. **Authentication**:
   - ✅ All requests with valid JWT succeed
   - ✅ Unauthenticated requests return 401
   - ✅ Insufficient permissions return 403

2. **Trading Service**:
   - ✅ Orders submitted and tracked
   - ✅ Positions retrieved correctly
   - ✅ Real-time streams functional

3. **Backtesting Service**:
   - ✅ Backtests start and complete
   - ✅ Results include performance metrics
   - ✅ Progress updates received

4. **ML Training Service**:
   - ✅ Training jobs start and stop
   - ✅ Resource monitoring works
   - ✅ Templates available

5. **Health & Resilience**:
   - ✅ All services report health
   - ✅ Graceful degradation functional
   - ✅ Circuit breakers activate

### Performance Targets

| Metric | Target | Measured By |
|--------|--------|-------------|
| Authentication Latency | <10μs | test_e2e_successful_authentication_flow |
| Order Submission | <100ms | test_e2e_order_submission_market_order |
| Concurrent Requests | 80%+ success | test_e2e_concurrent_order_submissions |
| Stream Latency | <1s first event | test_e2e_market_data_subscription |

---

## Troubleshooting

### Common Issues

#### 1. Connection Refused

**Symptom**: `Connection refused (os error 111)`

**Solution**:
```bash
# Check services are running
docker-compose ps
lsof -i :50051  # API Gateway
lsof -i :50052  # Trading Service
```

#### 2. Authentication Failures

**Symptom**: `Unauthenticated` errors

**Solution**:
- Verify JWT_SECRET matches across services
- Check token expiration (tokens valid for 1 hour)
- Ensure Redis is accessible for revocation checks

#### 3. Test Timeouts

**Symptom**: Tests hang or timeout

**Solution**:
```bash
# Increase timeout in test
timeout(StdDuration::from_secs(30), ...)

# Check service health
curl http://localhost:9091/metrics  # API Gateway metrics
```

#### 4. gRPC Errors

**Symptom**: `transport error` or `h2 error`

**Solution**:
- Ensure HTTP/2 is enabled
- Check for port conflicts
- Verify proto compatibility

### Debug Logging

Enable debug output:

```bash
RUST_LOG=debug cargo test -p integration_tests -- --ignored --nocapture
```

Trace specific components:

```bash
RUST_LOG=integration_tests=trace,tonic=debug cargo test ...
```

---

## Test Maintenance

### Adding New Tests

1. **Create test function**:
   ```rust
   #[tokio::test]
   #[ignore] // Requires running services
   async fn test_e2e_new_feature() -> Result<()> {
       // Test implementation
   }
   ```

2. **Use authentication helper**:
   ```rust
   let mut client = create_authenticated_client().await?;
   ```

3. **Add assertions**:
   ```rust
   assert!(result.is_ok(), "Operation should succeed");
   ```

4. **Document in this file**

### Updating Proto Files

When proto files change:

1. Update proto files in `tli/proto/`
2. Rebuild integration tests: `cargo build -p integration_tests`
3. Update test assertions for new fields
4. Re-run tests to validate

### CI/CD Integration

For CI pipelines:

```yaml
# .github/workflows/e2e-tests.yml
- name: Run E2E Tests
  run: |
    docker-compose up -d
    sleep 10  # Wait for services
    cargo test -p integration_tests -- --ignored
```

---

## Metrics & Reporting

### Test Execution Report

After running tests:

```bash
# Generate coverage report
cargo llvm-cov --html --output-dir coverage_e2e

# View results
open coverage_e2e/index.html
```

### Expected Coverage

| Component | Current | Target |
|-----------|---------|--------|
| API Gateway routing | ~60% | 75% |
| Service authentication | ~80% | 90% |
| Order lifecycle | ~70% | 85% |
| Backtest execution | ~65% | 80% |
| ML training flow | ~60% | 75% |

---

## Future Enhancements

### Planned Additions

1. **Performance Tests**:
   - Latency percentiles (p50, p95, p99)
   - Throughput benchmarks
   - Load testing integration

2. **Chaos Engineering**:
   - Network partition simulation
   - Service crash recovery
   - Database failover

3. **Security Tests**:
   - JWT tampering attempts
   - Permission escalation
   - Rate limiting validation

4. **Integration with CI/CD**:
   - Automated nightly runs
   - Performance regression detection
   - Test result dashboards

---

## References

- [gRPC Testing Best Practices](https://grpc.io/docs/guides/testing/)
- [Tonic Client Documentation](https://docs.rs/tonic/)
- [Tokio Test Utilities](https://docs.rs/tokio-test/)
- [JWT Testing Patterns](https://jwt.io/introduction)

---

**Document Version**: 1.0
**Last Review**: 2025-10-08
**Next Review**: 2025-10-15