foxhunt/bin/fxt/tests/INTEGRATION_TEST_GUIDE.md

TLI Integration Test Guide

Overview

This guide provides comprehensive documentation for running and understanding the TLI (Terminal Line Interface) system integration tests. The test suite covers service communication, database operations, end-to-end workflows, error handling, and performance testing.

Test Structure

tli/tests/integration/
├── mod.rs                          # Common test utilities and configuration
├── service_integration_tests.rs    # Service-to-service communication tests
├── database_integration_tests.rs   # Database operation and integrity tests
├── end_to_end_tests.rs             # Complete workflow scenario tests
├── error_handling_tests.rs         # Resilience and error recovery tests
└── performance_tests.rs            # Load testing and performance validation

Prerequisites

Environment Setup

1. Required Environment Variables

   # Optional: PostgreSQL testing (if available)
   export TEST_POSTGRES_URL="postgresql://user:password@localhost:5432/test_db"
   
   # Optional: InfluxDB testing (if available)
   export TEST_INFLUX_URL="http://localhost:8086"
   export TEST_INFLUX_TOKEN="your_influx_token"
   export TEST_INFLUX_ORG="test_org"
   
   # Optional: Enable real-time tests
   export TLI_ENABLE_RT_TESTS="true"
   
   # Logging level
   export RUST_LOG="info"
   

2. Database Setup (Optional)

   # PostgreSQL (if testing database integration)
   createdb test_db
   
   # InfluxDB (if testing time-series integration)
   influx bucket create -n test_bucket -o test_org
   

Dependencies

All test dependencies are defined in Cargo.toml:

• tokio-test - Async testing utilities
• tempfile - Temporary file management
• wiremock - HTTP mocking for network tests
• fake - Test data generation
• proptest - Property-based testing
• criterion - Performance benchmarking

Test Categories

1. Service Integration Tests

File: service_integration_tests.rs

Tests communication between TLI clients and backend services:

Trading Service Tests

• Order submission and cancellation
• Position monitoring
• Market data subscription
• Risk management validation
• Portfolio analytics

Backtesting Service Tests

• Backtest execution workflow
• Strategy optimization
• Data management
• Performance reporting

Configuration Management Tests

• Configuration CRUD operations
• Hot-reload functionality
• Validation workflows

Event Streaming Tests

• Real-time event subscription
• Event filtering and routing
• Stream reconnection handling

Run Command:

cargo test --test integration service_integration_tests

2. Database Integration Tests

File: database_integration_tests.rs

Tests database operations across different storage systems:

SQLite Configuration Tests

• Configuration CRUD operations
• Bulk operations and transactions
• Version management
• Backup and restore functionality

PostgreSQL Event Tests (Optional)

• Event storage and retrieval
• Event aggregation and statistics
• Cleanup and archival processes
• Query performance

InfluxDB Time-Series Tests (Optional)

• Market data storage
• Performance metrics collection
• Time-based queries and aggregations

Transaction Integrity Tests

• Rollback scenarios
• Multi-database consistency
• Connection pool management
• Timeout handling

Run Command:

cargo test --test integration database_integration_tests

3. End-to-End Scenario Tests

File: end_to_end_tests.rs

Tests complete business workflows:

Order Lifecycle Tests

• Complete order submission → execution → settlement
• Order modification and cancellation
• Batch order operations
• Position and portfolio updates

Backtesting Workflow Tests

• Data availability validation
• Backtest creation and execution
• Strategy optimization workflows
• Walk-forward analysis
• Report generation

Configuration Management Flow

• Configuration updates with hot-reload
• Validation and approval workflows
• History tracking

Security Authentication Flow

• User registration and authentication
• Role-based access control
• Session management
• Token refresh and logout

Event Storage and Retrieval

• Event generation through operations
• Event filtering and querying
• Event statistics and analytics

Run Command:

cargo test --test integration end_to_end_tests

4. Error Handling and Resilience Tests

File: error_handling_tests.rs

Tests system behavior under failure conditions:

Service Unavailability Tests

• Single service failures
• Partial system degradation
• Service recovery scenarios

Network Failure Tests

• Connection timeouts
• Connection refused handling
• Intermittent network failures

Database Failure Tests

• Connection failures
• Transaction rollback scenarios
• Recovery mechanisms

Invalid Data Tests

• Malformed requests
• Data validation failures
• Boundary condition handling

Resilience Pattern Tests

• Circuit breaker activation
• Retry with exponential backoff
• Rate limiting handling
• Graceful degradation

Run Command:

cargo test --test integration error_handling_tests

5. Performance and Load Tests

File: performance_tests.rs

Tests system performance under various load conditions:

Concurrent Request Tests

• High-concurrency order submission
• Mixed operation load testing
• Connection pool stress testing

High-Frequency Trading Simulation

• Ultra-low latency requirements
• High-throughput order flow
• Market data streaming performance

Stress Tests

• Memory usage under load
• Resource limit validation
• Connection pool exhaustion

Backtesting Performance

• Concurrent backtest execution
• Large dataset processing
• Resource utilization monitoring

Run Command:

cargo test --test integration performance_tests

Running Tests

All Integration Tests

# Run all integration tests
cargo test --test integration

# Run with logging
RUST_LOG=info cargo test --test integration

# Run with nocapture to see output
cargo test --test integration -- --nocapture

Specific Test Categories

# Service integration only
cargo test --test integration service_integration

# Database tests only
cargo test --test integration database_integration

# End-to-end scenarios only
cargo test --test integration end_to_end

# Error handling only
cargo test --test integration error_handling

# Performance tests only
cargo test --test integration performance

Individual Test Functions

# Specific test function
cargo test --test integration test_complete_order_lifecycle

# Tests matching pattern
cargo test --test integration order_lifecycle

Test Configuration

Performance Test Configuration

# High-performance testing
export TLI_PERF_MAX_CONCURRENT=100
export TLI_PERF_TEST_DURATION=30
export TLI_PERF_TARGET_LATENCY_MS=10

# Memory-constrained testing
export TLI_PERF_MEMORY_LIMIT_MB=256

Database Test Configuration

# Skip database tests if no external databases available
export TLI_SKIP_DATABASE_TESTS="true"

# Use specific database for testing
export TLI_TEST_DATABASE_URL="sqlite::memory:"

Mock Servers

The test suite includes sophisticated mock servers that simulate real services:

MockTradingServer

• Simulates trading service responses
• Configurable failure modes
• Performance testing optimizations
• Event generation

MockBacktestingServer

• Simulates backtesting workflows
• Progress tracking simulation
• Result generation
• Resource usage simulation

MockRiskServer

• Risk calculation simulation
• Limit validation
• Alert generation

Failure Modes

• ServiceUnavailable - 503 responses
• DatabaseError - Database connection failures
• TransactionFailure - Transaction rollback scenarios
• IntermittentFailure - Random failures
• RateLimited - 429 responses
• PartialDegradation - Reduced functionality

Test Data Management

Temporary Resources

• SQLite databases created in temporary directories
• Automatic cleanup after tests
• Isolated test environments

Generated Test Data

• Fake market data using fake crate
• Random but consistent test symbols
• Realistic order and position data

Configuration Fixtures

• Predefined configuration sets
• Validation test cases
• Performance baselines

Performance Benchmarks

Latency Targets

• Order submission: < 10ms P99
• Position queries: < 5ms P99
• Configuration updates: < 100ms P99

Throughput Targets

• Order submission: > 1000 RPS
• Market data updates: > 10,000/second
• Event processing: > 5,000/second

Resource Limits

• Memory usage: < 512MB under load
• CPU usage: < 80% sustained
• Connection pool: 100 concurrent connections

Troubleshooting

Common Issues

1. Port Conflicts

   Error: Address already in use
   

   Solution: Change mock_server_port in test configuration or kill conflicting processes.

2. Database Connection Failures

   Error: Failed to connect to database
   

   Solution: Verify database URLs and credentials, or disable database tests.

3. Timeout Failures

   Error: Operation timed out
   

   Solution: Increase timeout values in test configuration or check system performance.

4. Memory Issues

   Error: Cannot allocate memory
   

   Solution: Reduce concurrent test limits or increase available memory.

Debug Mode

# Enable debug logging
RUST_LOG=debug cargo test --test integration

# Enable trace logging for specific modules
RUST_LOG=tli=trace cargo test --test integration

# Run single test with full output
cargo test --test integration test_name -- --exact --nocapture

Performance Debugging

# Run with performance profiling
cargo test --test integration performance_tests --release

# Memory usage tracking
valgrind --tool=massif cargo test --test integration

# CPU profiling
perf record cargo test --test integration performance_tests

Continuous Integration

GitHub Actions Configuration

name: Integration Tests
on: [push, pull_request]

jobs:
  integration-tests:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:14
        env:
          POSTGRES_PASSWORD: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v3
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - name: Run integration tests
        env:
          TEST_POSTGRES_URL: postgresql://postgres:test@localhost:5432/postgres
          RUST_LOG: info
        run: cargo test --test integration

Test Reports

• JUnit XML output for CI systems
• Performance benchmark results
• Coverage reports
• Memory usage analysis

Contributing

Adding New Tests

1. Service Integration Tests

   • Add to appropriate module in service_integration_tests.rs
   • Follow existing patterns for setup/teardown
   • Include both success and failure scenarios

2. Database Tests

   • Add to database_integration_tests.rs
   • Handle optional database availability
   • Test both SQLite and PostgreSQL when available

3. End-to-End Tests

   • Add complete workflow tests to end_to_end_tests.rs
   • Include authentication and authorization
   • Test realistic user scenarios

4. Error Handling Tests

   • Add to error_handling_tests.rs
   • Test specific failure modes
   • Validate recovery mechanisms

5. Performance Tests

   • Add to performance_tests.rs
   • Define performance targets
   • Include resource usage monitoring

Test Naming Conventions

• Use descriptive names: test_complete_order_lifecycle
• Group related tests: order_lifecycle_tests module
• Include test type: test_concurrent_order_submission

Documentation

• Document test purpose and scope
• Include setup requirements
• Specify performance expectations
• Add troubleshooting notes

Metrics and Monitoring

Test Metrics Collected

• Execution time per test
• Memory usage patterns
• Network request latencies
• Database query performance
• Error rates and types

Performance Baselines

• Baseline measurements for regression detection
• Performance trend analysis
• Resource usage monitoring
• Capacity planning data

Reporting

• Test execution reports
• Performance benchmark results
• Coverage analysis
• Failure trend analysis

Security Considerations

Test Data

• No real credentials in tests
• Temporary databases only
• Isolated test environments
• Automatic cleanup

Network Security

• Mock servers only bind to localhost
• No external network access required
• Encrypted communication testing
• Authentication flow validation

Data Protection

• No persistent sensitive data
• Memory cleanup after tests
• Secure random data generation
• Audit trail testing

---

For additional support or questions about the integration tests, please refer to the main TLI documentation or contact the development team.