foxhunt/docs/testing/E2E_INTEGRATION_TESTS.md

End-to-End Integration Testing Documentation

Last Updated: 2025-10-08 Test Suite Version: 1.0 Author: Agent 112

---

Overview

This document describes the comprehensive end-to-end (E2E) integration test suite for the Foxhunt HFT Trading System. The test suite validates complete client → gateway → service flows across all microservices.

Test Coverage Summary

Test Suite	Test Count	Coverage Area
Trading Service E2E	15 tests	Order lifecycle, positions, streaming
Backtesting Service E2E	12 tests	Backtest execution, monitoring, results
ML Training Service E2E	12 tests	Training jobs, progress, configuration
Service Health & Resilience	15 tests	Health checks, degradation, failover
TOTAL	54 tests	Full service integration

---

Architecture

Test Flow Topology

┌─────────────────────────────────────────────────────────┐
│                    Test Client                          │
│            (Generated gRPC Clients)                     │
└──────────────────────┬──────────────────────────────────┘
                       │ JWT Authentication
                       ▼
┌─────────────────────────────────────────────────────────┐
│                 API Gateway (Port 50051)                │
│          Auth Validation + Request Routing              │
└───┬──────────────────┬──────────────────┬───────────────┘
    │                  │                  │
    ▼                  ▼                  ▼
┌──────────┐    ┌──────────────┐    ┌────────────────┐
│ Trading  │    │ Backtesting  │    │  ML Training   │
│ Service  │    │   Service    │    │    Service     │
│Port 50052│    │  Port 50053  │    │  Port 50054    │
└──────────┘    └──────────────┘    └────────────────┘

Authentication Flow

All E2E tests use JWT-based authentication:

1. Token Generation: Tests generate JWT tokens with:

   • User ID (e.g., test_trader_001)
   • Roles (e.g., ["trader", "admin"])
   • Permissions (e.g., ["api.access", "trading.submit"])
   • Session ID (unique per test)
   • JTI (JWT ID for revocation tracking)

2. Token Validation: API Gateway validates:

   • Signature using shared secret
   • Expiration timestamp
   • Required permissions
   • Revocation status (Redis check)

3. Request Context: Authenticated requests include:

   • User context in request extensions
   • Authorization header: Bearer <token>
   • Audit logging enabled

---

Test Suites

1. Trading Service E2E Tests

Location: services/integration_tests/tests/trading_service_e2e.rs Test Count: 15 tests

Section 1: Order Submission (5 tests)

1. test_e2e_order_submission_market_order

   • Validates: Market order submission through API Gateway
   • Verifies: Order ID returned, success status
   • Flow: Client → API Gateway → Trading Service

2. test_e2e_order_submission_limit_order

   • Validates: Limit order with price specification
   • Verifies: Price propagation, order acceptance

3. test_e2e_order_submission_without_auth

   • Validates: Authentication enforcement
   • Verifies: Unauthenticated requests rejected with 401

4. test_e2e_order_cancellation

   • Validates: Order lifecycle management
   • Verifies: Submit → Cancel flow, status updates

5. test_e2e_order_status_query

   • Validates: Order status retrieval
   • Verifies: Status reflects order lifecycle

Section 2: Position Management (3 tests)

6. test_e2e_get_all_positions

   • Validates: Portfolio position retrieval
   • Verifies: All positions returned

7. test_e2e_get_position_by_symbol

   • Validates: Symbol-filtered position query
   • Verifies: Correct position data

8. test_e2e_get_account_info

   • Validates: Account balance and metrics
   • Verifies: Buying power, cash balance calculation

Section 3: Real-Time Streaming (4 tests)

9. test_e2e_market_data_subscription

   • Validates: Real-time market data streaming
   • Verifies: Trade and quote events received

10. test_e2e_order_updates_subscription

    • Validates: Order status streaming
    • Verifies: Real-time order fills and updates

11. test_e2e_concurrent_order_submissions

    • Validates: Concurrent order handling
    • Verifies: 80%+ success rate for 10 parallel orders

12. test_e2e_gateway_request_routing

    • Validates: Multi-endpoint routing
    • Verifies: Different request types routed correctly

Section 4: Error Handling (3 tests)

13. test_e2e_invalid_symbol_handling

    • Validates: Symbol validation
    • Verifies: Invalid symbols rejected

14. test_e2e_negative_quantity_validation

    • Validates: Input validation
    • Verifies: Negative quantities rejected

15. test_e2e_gateway_timeout_handling

    • Validates: Timeout management
    • Verifies: Graceful timeout handling

---

2. Backtesting Service E2E Tests

Location: services/integration_tests/tests/backtesting_service_e2e.rs Test Count: 12 tests

Section 1: Backtest Lifecycle (5 tests)

1. test_e2e_backtest_start

   • Validates: Backtest job creation
   • Verifies: Strategy parameters, date range, initial capital

2. test_e2e_backtest_status

   • Validates: Progress monitoring
   • Verifies: Status transitions, progress percentage

3. test_e2e_backtest_stop

   • Validates: Graceful job termination
   • Verifies: Partial results saved

4. test_e2e_backtest_results

   • Validates: Results retrieval after completion
   • Verifies: Sharpe ratio, returns, trade metrics

5. test_e2e_backtest_list

   • Validates: Historical backtest listing
   • Verifies: Pagination, filtering

Section 2: Real-Time Monitoring (3 tests)

6. test_e2e_backtest_progress_subscription

   • Validates: Streaming progress updates
   • Verifies: Equity curve, PnL updates

7. test_e2e_backtest_filtering_by_strategy

   • Validates: Strategy-based filtering
   • Verifies: Correct strategy match

8. test_e2e_backtest_filtering_by_status

   • Validates: Status-based filtering
   • Verifies: COMPLETED status filter

Section 3: Error Handling (4 tests)

9. test_e2e_backtest_invalid_date_range

   • Validates: Date validation
   • Verifies: Start > End rejected

10. test_e2e_backtest_invalid_capital

    • Validates: Capital validation
    • Verifies: Negative capital rejected

11. test_e2e_backtest_nonexistent_status

    • Validates: Error handling
    • Verifies: 404 for non-existent backtests

12. test_e2e_backtest_unauthenticated_access

    • Validates: Security enforcement
    • Verifies: Auth required for all operations

---

3. ML Training Service E2E Tests

Location: services/integration_tests/tests/ml_training_service_e2e.rs Test Count: 12 tests

Section 1: Training Job Lifecycle (5 tests)

1. test_e2e_training_job_start

   • Validates: Training job creation
   • Verifies: Hyperparameters, resource allocation

2. test_e2e_training_job_stop

   • Validates: Job termination
   • Verifies: Graceful shutdown, cleanup

3. test_e2e_list_training_jobs

   • Validates: Job listing
   • Verifies: Pagination, metadata

4. test_e2e_filter_training_jobs_by_model

   • Validates: Model-based filtering
   • Verifies: DQN/PPO/MAMBA filtering

5. test_e2e_filter_training_jobs_by_status

   • Validates: Status filtering
   • Verifies: COMPLETED/RUNNING/FAILED

Section 2: Real-Time Monitoring (3 tests)

6. test_e2e_watch_training_progress

   • Validates: Streaming training metrics
   • Verifies: Loss, accuracy, epoch progress

7. test_e2e_resource_utilization

   • Validates: GPU/CPU monitoring
   • Verifies: Resource availability

8. test_e2e_stream_resource_metrics

   • Validates: Real-time resource streaming
   • Verifies: GPU utilization, active jobs

Section 3: Configuration (4 tests)

9. test_e2e_validate_training_config

   • Validates: Pre-flight validation
   • Verifies: Config warnings/errors

10. test_e2e_get_training_templates

    • Validates: Template retrieval
    • Verifies: Model-specific templates

11. test_e2e_filter_templates_by_model_type

    • Validates: Template filtering
    • Verifies: DQN/PPO templates

12. test_e2e_training_with_auto_deploy

    • Validates: Auto-deployment
    • Verifies: Model deployment on completion

---

4. Service Health & Resilience E2E Tests

Location: services/integration_tests/tests/service_health_resilience_e2e.rs Test Count: 15 tests

Section 1: System Health Monitoring (5 tests)

1. test_e2e_system_health_all_services

   • Validates: Overall system health
   • Verifies: All services reporting status

2. test_e2e_system_health_specific_service

   • Validates: Individual service health
   • Verifies: Trading service status

3. test_e2e_health_check_interval

   • Validates: Health update frequency
   • Verifies: Timestamps updating

4. test_e2e_health_status_transitions

   • Validates: Status change events
   • Verifies: HEALTHY → DEGRADED transitions

5. test_e2e_degraded_service_detection

   • Validates: Degradation detection
   • Verifies: System remains operational

Section 2: Graceful Degradation (5 tests)

6. test_e2e_trading_service_available_backtesting_optional

   • Validates: Core vs optional services
   • Verifies: Trading works when backtesting down

7. test_e2e_partial_service_failure_handling

   • Validates: Partial failure tolerance
   • Verifies: System reports known state

8. test_e2e_circuit_breaker_validation

   • Validates: Circuit breaker activation
   • Verifies: Protection from cascading failures

9. test_e2e_timeout_handling

   • Validates: Request timeout management
   • Verifies: Graceful timeout responses

10. test_e2e_retry_logic_validation

    • Validates: Exponential backoff retries
    • Verifies: 3 retries with backoff

Section 3: Service Discovery (5 tests)

11. test_e2e_api_gateway_routing

    • Validates: Multi-service routing
    • Verifies: Correct backend selection

12. test_e2e_service_discovery

    • Validates: Service registration
    • Verifies: Service metadata available

13. test_e2e_concurrent_service_requests

    • Validates: Concurrent request handling
    • Verifies: 80%+ success rate

14. test_e2e_load_balancing_verification

    • Validates: Load distribution
    • Verifies: Response time variance

15. test_e2e_service_failover

    • Validates: Failover configuration
    • Verifies: Redundancy setup

---

Running the Tests

Prerequisites

1. Infrastructure Services Running:

   docker-compose up -d postgres redis vault influxdb
   

2. Application Services Running:

   cargo run -p api_gateway &
   cargo run -p trading_service &
   cargo run -p backtesting_service &
   cargo run -p ml_training_service &
   

3. Database Migrations:

   cargo sqlx migrate run
   

Execution

Run All E2E Tests

cargo test -p integration_tests --test '*_e2e' -- --ignored

Run Specific Test Suite

# Trading service only
cargo test -p integration_tests --test trading_service_e2e -- --ignored

# Backtesting service only
cargo test -p integration_tests --test backtesting_service_e2e -- --ignored

# ML training service only
cargo test -p integration_tests --test ml_training_service_e2e -- --ignored

# Health & resilience only
cargo test -p integration_tests --test service_health_resilience_e2e -- --ignored

Run Single Test

cargo test -p integration_tests test_e2e_order_submission_market_order -- --ignored --nocapture

Test Flags

• --ignored: Required - all E2E tests are marked with #[ignore]
• --nocapture: Shows println output
• --test-threads=1: Serialize tests (avoid port conflicts)

---

Test Configuration

Environment Variables

# API Gateway endpoint
API_GATEWAY_ADDR=http://localhost:50051

# JWT configuration
JWT_SECRET=dev_secret_key_change_in_production
JWT_ISSUER=foxhunt-api-gateway
JWT_AUDIENCE=foxhunt-services

# Service endpoints (for direct testing)
TRADING_SERVICE_ADDR=http://localhost:50052
BACKTESTING_SERVICE_ADDR=http://localhost:50053
ML_TRAINING_SERVICE_ADDR=http://localhost:50054

# Database
DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Redis
REDIS_URL=redis://localhost:6379

Test Users

User ID	Roles	Permissions
test_trader_001	trader	api.access, trading.submit, trading.view
test_backtester_001	analyst	api.access, backtesting.execute, backtesting.view
test_ml_engineer_001	ml_engineer	api.access, ml.train, ml.view, ml.manage
health_monitor	admin	api.access, system.monitor
resilience_tester	admin	api.access, system.admin

---

Expected Results

Success Criteria

1. Authentication:

   • ✅ All requests with valid JWT succeed
   • ✅ Unauthenticated requests return 401
   • ✅ Insufficient permissions return 403

2. Trading Service:

   • ✅ Orders submitted and tracked
   • ✅ Positions retrieved correctly
   • ✅ Real-time streams functional

3. Backtesting Service:

   • ✅ Backtests start and complete
   • ✅ Results include performance metrics
   • ✅ Progress updates received

4. ML Training Service:

   • ✅ Training jobs start and stop
   • ✅ Resource monitoring works
   • ✅ Templates available

5. Health & Resilience:

   • ✅ All services report health
   • ✅ Graceful degradation functional
   • ✅ Circuit breakers activate

Performance Targets

Metric	Target	Measured By
Authentication Latency	<10μs	test_e2e_successful_authentication_flow
Order Submission	<100ms	test_e2e_order_submission_market_order
Concurrent Requests	80%+ success	test_e2e_concurrent_order_submissions
Stream Latency	<1s first event	test_e2e_market_data_subscription

---

Troubleshooting

Common Issues

1. Connection Refused

Symptom: Connection refused (os error 111)

Solution:

# Check services are running
docker-compose ps
lsof -i :50051  # API Gateway
lsof -i :50052  # Trading Service

2. Authentication Failures

Symptom: Unauthenticated errors

Solution:

• Verify JWT_SECRET matches across services
• Check token expiration (tokens valid for 1 hour)
• Ensure Redis is accessible for revocation checks

3. Test Timeouts

Symptom: Tests hang or timeout

Solution:

# Increase timeout in test
timeout(StdDuration::from_secs(30), ...)

# Check service health
curl http://localhost:9091/metrics  # API Gateway metrics

4. gRPC Errors

Symptom: transport error or h2 error

Solution:

• Ensure HTTP/2 is enabled
• Check for port conflicts
• Verify proto compatibility

Debug Logging

Enable debug output:

RUST_LOG=debug cargo test -p integration_tests -- --ignored --nocapture

Trace specific components:

RUST_LOG=integration_tests=trace,tonic=debug cargo test ...

---

Test Maintenance

Adding New Tests

1. Create test function:

   #[tokio::test]
   #[ignore] // Requires running services
   async fn test_e2e_new_feature() -> Result<()> {
       // Test implementation
   }
   

2. Use authentication helper:

   let mut client = create_authenticated_client().await?;
   

3. Add assertions:

   assert!(result.is_ok(), "Operation should succeed");
   

4. Document in this file

Updating Proto Files

When proto files change:

1. Update proto files in tli/proto/
2. Rebuild integration tests: cargo build -p integration_tests
3. Update test assertions for new fields
4. Re-run tests to validate

CI/CD Integration

For CI pipelines:

# .github/workflows/e2e-tests.yml
- name: Run E2E Tests
  run: |
    docker-compose up -d
    sleep 10  # Wait for services
    cargo test -p integration_tests -- --ignored

---

Metrics & Reporting

Test Execution Report

After running tests:

# Generate coverage report
cargo llvm-cov --html --output-dir coverage_e2e

# View results
open coverage_e2e/index.html

Expected Coverage

Component	Current	Target
API Gateway routing	~60%	75%
Service authentication	~80%	90%
Order lifecycle	~70%	85%
Backtest execution	~65%	80%
ML training flow	~60%	75%

---

Future Enhancements

Planned Additions

1. Performance Tests:

   • Latency percentiles (p50, p95, p99)
   • Throughput benchmarks
   • Load testing integration

2. Chaos Engineering:

   • Network partition simulation
   • Service crash recovery
   • Database failover

3. Security Tests:

   • JWT tampering attempts
   • Permission escalation
   • Rate limiting validation

4. Integration with CI/CD:

   • Automated nightly runs
   • Performance regression detection
   • Test result dashboards

---

References

• gRPC Testing Best Practices
• Tonic Client Documentation
• Tokio Test Utilities
• JWT Testing Patterns

---

Document Version: 1.0 Last Review: 2025-10-08 Next Review: 2025-10-15