foxhunt/tli/tests/INTEGRATION_TEST_GUIDE.md

# TLI Integration Test Guide

## Overview

This guide provides comprehensive documentation for running and understanding the TLI (Terminal Line Interface) system integration tests. The test suite covers service communication, database operations, end-to-end workflows, error handling, and performance testing.

## Test Structure

```
tli/tests/integration/
├── mod.rs                          # Common test utilities and configuration
├── service_integration_tests.rs    # Service-to-service communication tests
├── database_integration_tests.rs   # Database operation and integrity tests
├── end_to_end_tests.rs             # Complete workflow scenario tests
├── error_handling_tests.rs         # Resilience and error recovery tests
└── performance_tests.rs            # Load testing and performance validation
```

## Prerequisites

### Environment Setup

1. **Required Environment Variables**
   ```bash
   # Optional: PostgreSQL testing (if available)
   export TEST_POSTGRES_URL="postgresql://user:password@localhost:5432/test_db"

   # Optional: InfluxDB testing (if available)
   export TEST_INFLUX_URL="http://localhost:8086"
   export TEST_INFLUX_TOKEN="your_influx_token"
   export TEST_INFLUX_ORG="test_org"

   # Optional: Enable real-time tests
   export TLI_ENABLE_RT_TESTS="true"

   # Logging level
   export RUST_LOG="info"
   ```

2. **Database Setup** (Optional)
   ```bash
   # PostgreSQL (if testing database integration)
   createdb test_db

   # InfluxDB (if testing time-series integration)
   influx bucket create -n test_bucket -o test_org
   ```

### Dependencies

All test dependencies are defined in `Cargo.toml`:
- `tokio-test` - Async testing utilities
- `tempfile` - Temporary file management
- `wiremock` - HTTP mocking for network tests
- `fake` - Test data generation
- `proptest` - Property-based testing
- `criterion` - Performance benchmarking

## Test Categories

### 1. Service Integration Tests

**File**: `service_integration_tests.rs`

Tests communication between TLI clients and backend services:

#### Trading Service Tests
- Order submission and cancellation
- Position monitoring
- Market data subscription
- Risk management validation
- Portfolio analytics

#### Backtesting Service Tests
- Backtest execution workflow
- Strategy optimization
- Data management
- Performance reporting

#### Configuration Management Tests
- Configuration CRUD operations
- Hot-reload functionality
- Validation workflows

#### Event Streaming Tests
- Real-time event subscription
- Event filtering and routing
- Stream reconnection handling

**Run Command**:
```bash
cargo test --test integration service_integration_tests
```

### 2. Database Integration Tests

**File**: `database_integration_tests.rs`

Tests database operations across different storage systems:

#### SQLite Configuration Tests
- Configuration CRUD operations
- Bulk operations and transactions
- Version management
- Backup and restore functionality

#### PostgreSQL Event Tests (Optional)
- Event storage and retrieval
- Event aggregation and statistics
- Cleanup and archival processes
- Query performance

#### InfluxDB Time-Series Tests (Optional)
- Market data storage
- Performance metrics collection
- Time-based queries and aggregations

#### Transaction Integrity Tests
- Rollback scenarios
- Multi-database consistency
- Connection pool management
- Timeout handling

**Run Command**:
```bash
cargo test --test integration database_integration_tests
```

### 3. End-to-End Scenario Tests

**File**: `end_to_end_tests.rs`

Tests complete business workflows:

#### Order Lifecycle Tests
- Complete order submission → execution → settlement
- Order modification and cancellation
- Batch order operations
- Position and portfolio updates

#### Backtesting Workflow Tests
- Data availability validation
- Backtest creation and execution
- Strategy optimization workflows
- Walk-forward analysis
- Report generation

#### Configuration Management Flow
- Configuration updates with hot-reload
- Validation and approval workflows
- History tracking

#### Security Authentication Flow
- User registration and authentication
- Role-based access control
- Session management
- Token refresh and logout

#### Event Storage and Retrieval
- Event generation through operations
- Event filtering and querying
- Event statistics and analytics

**Run Command**:
```bash
cargo test --test integration end_to_end_tests
```

### 4. Error Handling and Resilience Tests

**File**: `error_handling_tests.rs`

Tests system behavior under failure conditions:

#### Service Unavailability Tests
- Single service failures
- Partial system degradation
- Service recovery scenarios

#### Network Failure Tests
- Connection timeouts
- Connection refused handling
- Intermittent network failures

#### Database Failure Tests
- Connection failures
- Transaction rollback scenarios
- Recovery mechanisms

#### Invalid Data Tests
- Malformed requests
- Data validation failures
- Boundary condition handling

#### Resilience Pattern Tests
- Circuit breaker activation
- Retry with exponential backoff
- Rate limiting handling
- Graceful degradation

**Run Command**:
```bash
cargo test --test integration error_handling_tests
```

### 5. Performance and Load Tests

**File**: `performance_tests.rs`

Tests system performance under various load conditions:

#### Concurrent Request Tests
- High-concurrency order submission
- Mixed operation load testing
- Connection pool stress testing

#### High-Frequency Trading Simulation
- Ultra-low latency requirements
- High-throughput order flow
- Market data streaming performance

#### Stress Tests
- Memory usage under load
- Resource limit validation
- Connection pool exhaustion

#### Backtesting Performance
- Concurrent backtest execution
- Large dataset processing
- Resource utilization monitoring

**Run Command**:
```bash
cargo test --test integration performance_tests
```

## Running Tests

### All Integration Tests
```bash
# Run all integration tests
cargo test --test integration

# Run with logging
RUST_LOG=info cargo test --test integration

# Run with nocapture to see output
cargo test --test integration -- --nocapture
```

### Specific Test Categories
```bash
# Service integration only
cargo test --test integration service_integration

# Database tests only
cargo test --test integration database_integration

# End-to-end scenarios only
cargo test --test integration end_to_end

# Error handling only
cargo test --test integration error_handling

# Performance tests only
cargo test --test integration performance
```

### Individual Test Functions
```bash
# Specific test function
cargo test --test integration test_complete_order_lifecycle

# Tests matching pattern
cargo test --test integration order_lifecycle
```

### Test Configuration

#### Performance Test Configuration
```bash
# High-performance testing
export TLI_PERF_MAX_CONCURRENT=100
export TLI_PERF_TEST_DURATION=30
export TLI_PERF_TARGET_LATENCY_MS=10

# Memory-constrained testing
export TLI_PERF_MEMORY_LIMIT_MB=256
```

#### Database Test Configuration
```bash
# Skip database tests if no external databases available
export TLI_SKIP_DATABASE_TESTS="true"

# Use specific database for testing
export TLI_TEST_DATABASE_URL="sqlite::memory:"
```

## Mock Servers

The test suite includes sophisticated mock servers that simulate real services:

### MockTradingServer
- Simulates trading service responses
- Configurable failure modes
- Performance testing optimizations
- Event generation

### MockBacktestingServer
- Simulates backtesting workflows
- Progress tracking simulation
- Result generation
- Resource usage simulation

### MockRiskServer
- Risk calculation simulation
- Limit validation
- Alert generation

### Failure Modes
- `ServiceUnavailable` - 503 responses
- `DatabaseError` - Database connection failures
- `TransactionFailure` - Transaction rollback scenarios
- `IntermittentFailure` - Random failures
- `RateLimited` - 429 responses
- `PartialDegradation` - Reduced functionality

## Test Data Management

### Temporary Resources
- SQLite databases created in temporary directories
- Automatic cleanup after tests
- Isolated test environments

### Generated Test Data
- Fake market data using `fake` crate
- Random but consistent test symbols
- Realistic order and position data

### Configuration Fixtures
- Predefined configuration sets
- Validation test cases
- Performance baselines

## Performance Benchmarks

### Latency Targets
- Order submission: < 10ms P99
- Position queries: < 5ms P99
- Configuration updates: < 100ms P99

### Throughput Targets
- Order submission: > 1000 RPS
- Market data updates: > 10,000/second
- Event processing: > 5,000/second

### Resource Limits
- Memory usage: < 512MB under load
- CPU usage: < 80% sustained
- Connection pool: 100 concurrent connections

## Troubleshooting

### Common Issues

1. **Port Conflicts**
   ```
   Error: Address already in use
   ```
   Solution: Change `mock_server_port` in test configuration or kill conflicting processes.

2. **Database Connection Failures**
   ```
   Error: Failed to connect to database
   ```
   Solution: Verify database URLs and credentials, or disable database tests.

3. **Timeout Failures**
   ```
   Error: Operation timed out
   ```
   Solution: Increase timeout values in test configuration or check system performance.

4. **Memory Issues**
   ```
   Error: Cannot allocate memory
   ```
   Solution: Reduce concurrent test limits or increase available memory.

### Debug Mode
```bash
# Enable debug logging
RUST_LOG=debug cargo test --test integration

# Enable trace logging for specific modules
RUST_LOG=tli=trace cargo test --test integration

# Run single test with full output
cargo test --test integration test_name -- --exact --nocapture
```

### Performance Debugging
```bash
# Run with performance profiling
cargo test --test integration performance_tests --release

# Memory usage tracking
valgrind --tool=massif cargo test --test integration

# CPU profiling
perf record cargo test --test integration performance_tests
```

## Continuous Integration

### GitHub Actions Configuration
```yaml
name: Integration Tests
on: [push, pull_request]

jobs:
  integration-tests:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:14
        env:
          POSTGRES_PASSWORD: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v3
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - name: Run integration tests
        env:
          TEST_POSTGRES_URL: postgresql://postgres:test@localhost:5432/postgres
          RUST_LOG: info
        run: cargo test --test integration
```

### Test Reports
- JUnit XML output for CI systems
- Performance benchmark results
- Coverage reports
- Memory usage analysis

## Contributing

### Adding New Tests

1. **Service Integration Tests**
   - Add to appropriate module in `service_integration_tests.rs`
   - Follow existing patterns for setup/teardown
   - Include both success and failure scenarios

2. **Database Tests**
   - Add to `database_integration_tests.rs`
   - Handle optional database availability
   - Test both SQLite and PostgreSQL when available

3. **End-to-End Tests**
   - Add complete workflow tests to `end_to_end_tests.rs`
   - Include authentication and authorization
   - Test realistic user scenarios

4. **Error Handling Tests**
   - Add to `error_handling_tests.rs`
   - Test specific failure modes
   - Validate recovery mechanisms

5. **Performance Tests**
   - Add to `performance_tests.rs`
   - Define performance targets
   - Include resource usage monitoring

### Test Naming Conventions
- Use descriptive names: `test_complete_order_lifecycle`
- Group related tests: `order_lifecycle_tests` module
- Include test type: `test_concurrent_order_submission`

### Documentation
- Document test purpose and scope
- Include setup requirements
- Specify performance expectations
- Add troubleshooting notes

## Metrics and Monitoring

### Test Metrics Collected
- Execution time per test
- Memory usage patterns
- Network request latencies
- Database query performance
- Error rates and types

### Performance Baselines
- Baseline measurements for regression detection
- Performance trend analysis
- Resource usage monitoring
- Capacity planning data

### Reporting
- Test execution reports
- Performance benchmark results
- Coverage analysis
- Failure trend analysis

## Security Considerations

### Test Data
- No real credentials in tests
- Temporary databases only
- Isolated test environments
- Automatic cleanup

### Network Security
- Mock servers only bind to localhost
- No external network access required
- Encrypted communication testing
- Authentication flow validation

### Data Protection
- No persistent sensitive data
- Memory cleanup after tests
- Secure random data generation
- Audit trail testing

---

For additional support or questions about the integration tests, please refer to the main TLI documentation or contact the development team.