foxhunt/tests/README.md

# Foxhunt HFT Trading System - Comprehensive Test Documentation

## 🎯 Overview

This document provides comprehensive test documentation for the Foxhunt High-Frequency Trading (HFT) system, covering test architecture, execution guides, mock strategies, CI/CD pipeline integration, and coverage requirements.

## 📋 Table of Contents

1. [Test Architecture](#test-architecture)
2. [Test Running Guide](#test-running-guide)
3. [Mock Strategies Documentation](#mock-strategies-documentation)
4. [CI/CD Test Pipeline](#cicd-test-pipeline)
5. [Coverage Requirements](#coverage-requirements)
6. [Performance Requirements](#performance-requirements)
7. [Troubleshooting](#troubleshooting)

---

## 🏗️ Test Architecture

### System Overview

The Foxhunt test architecture is designed for enterprise-grade HFT systems with zero-tolerance for failures in production. The architecture follows a layered approach with comprehensive coverage across all system components.

```
┌─────────────────────────────────────────────────────────────┐
│                    TEST ARCHITECTURE                        │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │    E2E      │  │Integration  │  │Performance  │        │
│  │   Tests     │  │   Tests     │  │   Tests     │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │    Unit     │  │    Chaos    │  │ Compliance  │        │
│  │   Tests     │  │Engineering  │  │   Tests     │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │   Mock      │  │  Security   │  │   Load      │        │
│  │ Strategies  │  │   Tests     │  │   Tests     │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
└─────────────────────────────────────────────────────────────┘
```

### Test Layer Structure

#### 1. Unit Tests (`/tests/unit/`)
- **Purpose**: Test individual components in isolation
- **Coverage**: Core trading algorithms, ML models, risk calculations
- **Performance Target**: <1ms per test
- **Safety**: Zero panic operations, comprehensive error handling

**Key Components:**
```rust
// Core financial calculations with precision validation
tests/unit/financial_calculation_precision.rs
- Price calculation accuracy (14+ decimal places)
- Volume calculation correctness
- P&L computation validation

// ML model accuracy validation
tests/unit/ml_model_accuracy_validation.rs
- MAMBA-2 SSM model testing
- TLOB Transformer validation
- DQN/PPO reinforcement learning tests

// Risk management validation
tests/unit/risk_management_tests.rs
- VaR calculations
- Kelly criterion sizing
- Position limit enforcement
```

#### 2. Integration Tests (`/tests/integration/`)
- **Purpose**: Test service-to-service communication
- **Coverage**: gRPC interfaces, database operations, message passing
- **Performance Target**: <100ms per integration test
- **Safety**: Circuit breaker validation, timeout handling

**Key Components:**
```rust
// Service communication validation
tests/integration/service_communication_validation.rs
- Trading Service ↔ TLI communication
- ML Training Service ↔ Trading Service data flow
- Backtesting Service ↔ Data Service integration

// Database integration tests
tests/integration/database_integration.rs
- PostgreSQL connection pooling
- Real-time configuration updates
- Transaction consistency validation
```

#### 3. End-to-End Tests (`/tests/e2e/`)
- **Purpose**: Complete trading workflow validation
- **Coverage**: Full order lifecycle from signal to execution
- **Performance Target**: <500ms complete workflow
- **Safety**: Production-like scenarios, error recovery

#### 4. Performance Tests (`/tests/performance/`)
- **Purpose**: HFT latency and throughput validation
- **Coverage**: Sub-microsecond operations, high-frequency scenarios
- **Performance Target**: Meet HFT requirements (see Performance Requirements)

#### 5. Chaos Engineering Tests (`/tests/chaos/`)
- **Purpose**: System resilience under failure conditions
- **Coverage**: Network partitions, service crashes, resource exhaustion
- **Performance Target**: <1s recovery time for critical services

### Test Framework Components

#### Safety Framework (`framework/production_test_safety.rs`)

```rust
/// Production-safe test result type eliminating panic operations
pub type TestResult<T> = Result<T, TestSafetyError>;

/// Comprehensive error handling for all test failure modes
#[derive(Debug, thiserror::Error)]
pub enum TestSafetyError {
    #[error("Integration failure in {service}.{operation}: {details}")]
    IntegrationFailure {
        service: String,
        operation: String,
        details: String,
    },
    #[error("Performance requirement not met: {operation} took {actual_ns}ns, max allowed {max_ns}ns")]
    PerformanceViolation {
        operation: String,
        actual_ns: u64,
        max_ns: u64,
    },
    #[error("Timeout exceeded: {operation} timed out after {timeout_ms}ms")]
    TimeoutExceeded {
        operation: String,
        timeout_ms: u64,
    },
    // ... additional error types
}
```

#### Performance Validator

```rust
/// HFT performance validation with hardware-level precision
pub struct HftPerformanceValidator;

impl HftPerformanceValidator {
    /// Validates operation meets HFT latency requirements
    /// Uses RDTSC for sub-nanosecond timing precision
    pub fn validate_latency(operation: &str, duration_ns: u64) -> TestResult<()> {
        let requirement = match operation {
            "order_validation" => 10_000,      // 10μs max
            "risk_check" => 50_000,            // 50μs max
            "market_data_processing" => 1_000,  // 1μs max
            "position_update" => 5_000,         // 5μs max
            "price_calculation" => 2_000,       // 2μs max
            _ => return Err(TestSafetyError::UnknownOperation { operation: operation.to_string() })
        };

        if duration_ns > requirement {
            return Err(TestSafetyError::PerformanceViolation {
                operation: operation.to_string(),
                actual_ns: duration_ns,
                max_ns: requirement,
            });
        }

        Ok(())
    }
}
```

---

## 🚀 Test Running Guide

### Quick Start

#### Prerequisites
```bash
# Set environment variables
export DATABASE_URL="postgresql://localhost/foxhunt_test"
export RUST_LOG=debug
export CUDA_VISIBLE_DEVICES=0  # For GPU tests

# Install test database
psql -c "CREATE DATABASE foxhunt_test;"
psql foxhunt_test -f tests/fixtures/test_schema.sql
```

#### Basic Test Commands

```bash
# Run all tests (comprehensive suite)
cargo test --workspace

# Run tests with coverage
cargo test --workspace -- --nocapture
cargo tarpaulin --out Html --output-dir coverage/

# Run specific test suites
cargo test --package tests unit_tests
cargo test --package tests integration_tests
cargo test --package tests e2e_tests
```

### Test Execution Modes

#### 1. Development Mode (Fast Feedback)
```bash
# Quick unit tests only (30-60 seconds)
cargo test --package tests --lib unit

# With file watching for continuous testing
cargo watch -x "test --package tests --lib unit"
```

#### 2. Integration Mode (Service Validation)
```bash
# Integration tests with service startup (2-5 minutes)
./scripts/start_test_services.sh
cargo test --package tests integration
./scripts/stop_test_services.sh
```

#### 3. Performance Mode (HFT Validation)
```bash
# Performance and benchmarking tests (5-10 minutes)
cargo test --package tests performance --release
cargo bench --package tests

# GPU-accelerated ML model tests
cargo test --package tests --features cuda gpu_tests
```

#### 4. Chaos Engineering Mode (Resilience Testing)
```bash
# Chaos engineering tests (10-15 minutes)
cargo test --package tests chaos --release -- --test-threads=1

# With network simulation
sudo cargo test --package tests chaos::network_partition
```

#### 5. Production Validation Mode (Full Suite)
```bash
# Complete production readiness validation (30-45 minutes)
./scripts/run_production_tests.sh

# Expected output for production readiness:
# ✅ ALL TESTS PASSED - System ready for production deployment!
```

### Service-Specific Test Commands

#### Trading Service Tests
```bash
# Core trading logic
cargo test --package services --bin trading_service

# Trading service integration
cargo test --package tests trading_service_integration

# Performance validation
cargo bench trading_latency
```

#### ML Training Service Tests
```bash
# ML model accuracy tests
cargo test --package tests ml_model_accuracy_validation

# GPU performance tests
cargo test --package tests gpu_performance --features cuda

# Model inference benchmarks
cargo bench ml_inference
```

#### TLI (Terminal Line Interface) Tests
```bash
# TLI functionality tests
cargo test --package tli

# TLI performance tests
cargo bench tli_performance_validation
```

### Environment-Specific Testing

#### Docker Environment
```bash
# Start test environment
docker-compose -f docker-compose.test.yml up -d

# Run containerized tests
docker-compose -f docker-compose.test.yml run tests

# Cleanup
docker-compose -f docker-compose.test.yml down
```

#### Kubernetes Environment
```bash
# Deploy test cluster
kubectl apply -f tests/k8s/test-namespace.yaml
kubectl apply -f tests/k8s/

# Run distributed tests
kubectl run test-runner --image=foxhunt:test --command -- cargo test

# Monitor test execution
kubectl logs -f test-runner
```

---

## 🎭 Mock Strategies Documentation

### Overview

The Foxhunt system uses sophisticated mocking strategies to simulate real trading environments while maintaining deterministic, repeatable tests.

### Mock Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                   MOCK ARCHITECTURE                         │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │   Market    │  │   Broker    │  │ Data Feed   │        │
│  │   Mocks     │  │   Mocks     │  │   Mocks     │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │  External   │  │    Time     │  │  Network    │        │
│  │ API Mocks   │  │   Mocks     │  │   Mocks     │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
└─────────────────────────────────────────────────────────────┘
```

### 1. Market Data Mocks (`/tests/mocks/market_data.rs`)

#### Realistic Market Simulation
```rust
/// High-fidelity market data simulator for HFT testing
pub struct MarketDataMock {
    /// Tick-by-tick price movements with microsecond precision
    tick_generator: TickGenerator,
    /// Order book depth simulation (L2 data)
    order_book: OrderBookSimulator,
    /// Market microstructure effects
    microstructure: MicrostructureSimulator,
}

impl MarketDataMock {
    /// Creates realistic EUR/USD market conditions
    pub fn eurusd_realistic() -> Self {
        Self {
            tick_generator: TickGenerator::new()
                .with_spread(0.00001)  // 0.1 pip spread
                .with_volatility(0.0012)  // 12 bps daily vol
                .with_frequency(1000), // 1000 ticks/second
            order_book: OrderBookSimulator::new()
                .with_depth(10)  // 10 levels deep
                .with_liquidity(1_000_000), // $1M per level
            microstructure: MicrostructureSimulator::new()
                .with_latency_distribution(LatencyDist::Normal(50, 10)), // 50μs ± 10μs
        }
    }
}
```

#### Market Scenario Simulation
```rust
/// Predefined market scenarios for testing
pub enum MarketScenario {
    /// Normal trading conditions
    Normal,
    /// High volatility period (e.g., NFP release)
    HighVolatility,
    /// Low liquidity conditions (e.g., holiday trading)
    LowLiquidity,
    /// Flash crash scenario
    FlashCrash,
    /// Market closure/opening
    MarketTransition,
}

impl MarketDataMock {
    /// Simulates specific market scenarios with realistic parameters
    pub fn with_scenario(scenario: MarketScenario) -> Self {
        match scenario {
            MarketScenario::HighVolatility => {
                Self::eurusd_realistic()
                    .with_volatility(0.008)  // 80 bps (5x normal)
                    .with_frequency(5000)    // 5x tick rate
                    .with_spread_widening(3.0)  // 3x wider spreads
            },
            MarketScenario::FlashCrash => {
                Self::eurusd_realistic()
                    .with_price_shock(-0.02)  // 200 pip drop
                    .with_liquidity_drain(0.1)  // 90% liquidity removal
                    .with_recovery_time(300)  // 5-minute recovery
            },
            // ... other scenarios
        }
    }
}
```

### 2. Broker API Mocks (`/tests/mocks/broker.rs`)

#### Interactive Brokers Mock
```rust
/// Mock Interactive Brokers TWS API
pub struct IBApiMock {
    /// Connection simulation with realistic latency
    connection: MockConnection,
    /// Account information simulation
    account: AccountSimulator,
    /// Order execution simulation
    execution_engine: ExecutionSimulator,
}

impl IBApiMock {
    /// Simulates realistic order execution with market impact
    pub async fn place_order(&mut self, order: Order) -> TestResult<OrderResponse> {
        // Simulate network latency (realistic: 1-5ms)
        self.simulate_latency(Duration::from_micros(2500)).await;

        // Simulate order validation (realistic: broker-side checks)
        self.validate_order(&order)?;

        // Simulate execution with realistic slippage
        let execution = self.execution_engine.execute_with_slippage(order).await?;

        Ok(OrderResponse {
            order_id: execution.order_id,
            status: OrderStatus::Filled,
            fill_price: execution.fill_price,
            fill_time: SystemTime::now(),
            commission: self.calculate_commission(&execution),
        })
    }
}
```

#### Order Execution Simulation
```rust
/// Realistic order execution simulation including market impact
pub struct ExecutionSimulator {
    /// Market impact model
    impact_model: MarketImpactModel,
    /// Slippage simulation
    slippage_model: SlippageModel,
}

impl ExecutionSimulator {
    /// Executes order with realistic market conditions
    pub async fn execute_with_slippage(&self, order: Order) -> TestResult<Execution> {
        let market_price = self.get_current_price(order.symbol).await?;

        // Calculate market impact based on order size
        let impact = self.impact_model.calculate_impact(
            order.quantity,
            self.get_average_daily_volume(order.symbol).await?
        );

        // Apply slippage based on market conditions
        let slippage = self.slippage_model.calculate_slippage(
            order.quantity,
            self.get_current_spread(order.symbol).await?
        );

        let fill_price = match order.side {
            OrderSide::Buy => market_price + impact + slippage,
            OrderSide::Sell => market_price - impact - slippage,
        };

        Ok(Execution {
            order_id: order.id,
            fill_price,
            fill_quantity: order.quantity,
            fill_time: SystemTime::now(),
        })
    }
}
```

### 3. Time Mocks (`/tests/mocks/time.rs`)

#### Deterministic Time Control
```rust
/// Mock time provider for deterministic testing
pub struct MockTimeProvider {
    /// Current mock time
    current_time: Arc<Mutex<SystemTime>>,
    /// Time advancement step size
    step_size: Duration,
}

impl MockTimeProvider {
    /// Advances time by specified duration
    pub fn advance_time(&self, duration: Duration) -> TestResult<()> {
        let mut current = self.current_time.lock()
            .map_err(|_| TestSafetyError::TimeProviderError)?;
        *current += duration;
        Ok(())
    }

    /// Fast-forwards through market session
    pub fn fast_forward_market_session(&self) -> TestResult<()> {
        // Simulate 8-hour trading session in 1 second
        self.advance_time(Duration::from_hours(8))
    }
}
```

### 4. Network Mocks (`/tests/mocks/network.rs`)

#### Network Condition Simulation
```rust
/// Simulates various network conditions for resilience testing
pub struct NetworkConditionMock {
    /// Latency simulation
    latency: LatencySimulator,
    /// Packet loss simulation
    packet_loss: PacketLossSimulator,
    /// Bandwidth simulation
    bandwidth: BandwidthSimulator,
}

impl NetworkConditionMock {
    /// Simulates poor network conditions
    pub fn poor_connection() -> Self {
        Self {
            latency: LatencySimulator::new()
                .with_base_latency(Duration::from_millis(50))
                .with_jitter(Duration::from_millis(20))
                .with_spikes(0.05), // 5% of requests have 500ms spike
            packet_loss: PacketLossSimulator::new()
                .with_loss_rate(0.01), // 1% packet loss
            bandwidth: BandwidthSimulator::new()
                .with_throughput(1_000_000), // 1Mbps
        }
    }
}
```

### Mock Testing Patterns

#### 1. Dependency Injection Pattern
```rust
/// Service with mockable dependencies
pub struct TradingService<T: TimeProvider, M: MarketDataProvider, B: BrokerApi> {
    time_provider: T,
    market_data: M,
    broker: B,
}

impl TradingService<MockTimeProvider, MarketDataMock, IBApiMock> {
    /// Creates service with all mocks for testing
    pub fn with_mocks() -> Self {
        Self {
            time_provider: MockTimeProvider::new(),
            market_data: MarketDataMock::eurusd_realistic(),
            broker: IBApiMock::new(),
        }
    }
}
```

#### 2. Scenario-Based Testing
```rust
#[cfg(test)]
mod scenario_tests {
    use super::*;

    #[tokio::test]
    async fn test_high_volatility_scenario() -> TestResult<()> {
        let mut trading_service = TradingService::with_mocks();

        // Configure high volatility market conditions
        trading_service.market_data = MarketDataMock::with_scenario(
            MarketScenario::HighVolatility
        );

        // Execute trading strategy
        let result = trading_service.execute_strategy().await?;

        // Verify appropriate risk management response
        assert!(result.position_size < normal_position_size * 0.5);
        assert!(result.stop_loss_tighter_than_normal);

        Ok(())
    }
}
```

---

## 🔄 CI/CD Test Pipeline

### Overview

The CI/CD pipeline ensures comprehensive testing at every stage of development, from commit to production deployment.

### Pipeline Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                  CI/CD PIPELINE STAGES                      │
├─────────────────────────────────────────────────────────────┤
│  Commit → Fast Tests → Integration → Performance → Deploy   │
│    ↓         ↓           ↓              ↓           ↓      │
│  Lint    Unit Tests   Service     Benchmarks   Production  │
│  Check   (30-60s)    Tests       (5-10min)    Validation  │
│  (5s)                (2-5min)                   (30min)    │
└─────────────────────────────────────────────────────────────┘
```

### GitHub Actions Configuration

#### Main Workflow (`.github/workflows/ci.yml`)
```yaml
name: Foxhunt HFT CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  RUST_VERSION: 1.75
  DATABASE_URL: postgresql://postgres:postgres@localhost:5432/foxhunt_test
  CARGO_TERM_COLOR: always

jobs:
  # Stage 1: Fast Feedback (30-60 seconds)
  fast_feedback:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Cache Cargo Dependencies
        uses: actions/cache@v3
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target/
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}

      - name: Install Rust Toolchain
        uses: actions-rs/toolchain@v1
        with:
          toolchain: ${{ env.RUST_VERSION }}
          components: clippy, rustfmt
          override: true

      - name: Code Formatting Check
        run: cargo fmt --all -- --check

      - name: Clippy Analysis (Zero Panic Policy)
        run: |
          cargo clippy --workspace --all-targets --all-features -- \
            -D warnings \
            -D clippy::unwrap_used \
            -D clippy::expect_used \
            -D clippy::panic

      - name: Security Audit
        uses: actions-rs/audit@v1

      - name: Unit Tests (Fast)
        run: cargo test --workspace --lib --bins --tests unit
        env:
          RUST_BACKTRACE: 1

  # Stage 2: Integration Testing (2-5 minutes)
  integration_tests:
    runs-on: ubuntu-latest
    needs: fast_feedback
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: foxhunt_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432

      redis:
        image: redis:7
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379

    steps:
      - uses: actions/checkout@v4

      - name: Setup Test Database
        run: |
          psql $DATABASE_URL -c "CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";"
          psql $DATABASE_URL -f tests/fixtures/test_schema.sql

      - name: Integration Tests
        run: cargo test --workspace integration
        env:
          RUST_BACKTRACE: 1
          TEST_DATABASE_URL: ${{ env.DATABASE_URL }}

      - name: Service Communication Tests
        run: cargo test --package tests service_communication_validation

      - name: Database Integration Tests
        run: cargo test --package tests database_integration

  # Stage 3: Performance Testing (5-10 minutes)
  performance_tests:
    runs-on: ubuntu-latest
    needs: integration_tests
    steps:
      - uses: actions/checkout@v4

      - name: Install Performance Dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y linux-tools-generic

      - name: HFT Performance Validation
        run: |
          cargo test --package tests performance --release -- --nocapture
          cargo bench --workspace
        env:
          RUST_BACKTRACE: 1

      - name: Latency Benchmarks
        run: |
          echo "=== Trading Latency Benchmarks ==="
          cargo bench trading_latency
          echo "=== Order Processing Benchmarks ==="
          cargo bench order_processing
          echo "=== Risk Calculation Benchmarks ==="
          cargo bench risk_calculations

      - name: Performance Regression Check
        run: |
          # Compare with baseline performance metrics
          python scripts/check_performance_regression.py

  # Stage 4: GPU Testing (CUDA-enabled runners)
  gpu_tests:
    runs-on: [self-hosted, gpu]  # Requires GPU-enabled runner
    needs: fast_feedback
    if: contains(github.event.head_commit.message, '[gpu]') || github.ref == 'refs/heads/main'

    steps:
      - uses: actions/checkout@v4

      - name: GPU Environment Setup
        run: |
          nvidia-smi
          export CUDA_VISIBLE_DEVICES=0

      - name: GPU ML Model Tests
        run: |
          cargo test --package tests --features cuda gpu_tests
          cargo test --package ml --features cuda

      - name: ML Performance Benchmarks
        run: cargo bench ml_inference --features cuda

  # Stage 5: Chaos Engineering (Optional, on schedule)
  chaos_tests:
    runs-on: ubuntu-latest
    needs: integration_tests
    if: github.event_name == 'schedule' || contains(github.event.head_commit.message, '[chaos]')

    steps:
      - uses: actions/checkout@v4

      - name: Chaos Engineering Tests
        run: |
          cargo test --package tests chaos --release -- --test-threads=1
        timeout-minutes: 30

      - name: Network Partition Tests
        run: |
          sudo cargo test --package tests chaos::network_partition
        timeout-minutes: 15

  # Stage 6: Production Readiness Validation
  production_validation:
    runs-on: ubuntu-latest
    needs: [integration_tests, performance_tests]
    if: github.ref == 'refs/heads/main'

    steps:
      - uses: actions/checkout@v4

      - name: Comprehensive Production Tests
        run: |
          ./scripts/run_production_tests.sh
        timeout-minutes: 45

      - name: Production Readiness Check
        run: |
          if ./scripts/run_production_tests.sh | grep -q "✅ ALL TESTS PASSED"; then
            echo "✅ System ready for production deployment"
            echo "production_ready=true" >> $GITHUB_OUTPUT
          else
            echo "❌ System not ready for production"
            echo "production_ready=false" >> $GITHUB_OUTPUT
            exit 1
          fi
        id: readiness_check

      - name: Generate Test Report
        run: |
          ./scripts/generate_test_report.sh > test_report.md

      - name: Upload Test Report
        uses: actions/upload-artifact@v3
        with:
          name: test-report
          path: test_report.md

  # Stage 7: Deployment Gate
  deployment_gate:
    runs-on: ubuntu-latest
    needs: production_validation
    if: github.ref == 'refs/heads/main' && needs.production_validation.outputs.production_ready == 'true'

    steps:
      - name: Production Deployment Authorization
        run: |
          echo "🚀 Production deployment authorized"
          echo "All tests passed - system ready for production"

      - name: Trigger Deployment
        uses: peter-evans/repository-dispatch@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          event-type: deploy-production
          client-payload: |
            {
              "ref": "${{ github.ref }}",
              "sha": "${{ github.sha }}",
              "test_status": "passed"
            }
```

#### Performance Regression Monitoring
```yaml
# .github/workflows/performance-monitoring.yml
name: Performance Regression Monitoring

on:
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM
  workflow_dispatch:

jobs:
  performance_baseline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Benchmark Current Performance
        run: |
          cargo bench --workspace > benchmark_results.txt

      - name: Compare with Baseline
        run: |
          python scripts/performance_regression_analysis.py

      - name: Alert on Regression
        if: failure()
        uses: 8398a7/action-slack@v3
        with:
          status: failure
          text: "🚨 Performance regression detected in Foxhunt HFT system"
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}
```

### Branch-Specific Testing

#### Feature Branch Testing
```yaml
# Lightweight testing for feature branches
- name: Feature Branch Tests
  if: github.ref != 'refs/heads/main'
  run: |
    cargo test --workspace --lib unit
    cargo test --package tests integration::basic
```

#### Release Branch Testing
```yaml
# Comprehensive testing for release branches
- name: Release Validation
  if: startsWith(github.ref, 'refs/heads/release/')
  run: |
    ./scripts/run_comprehensive_tests.sh
    ./scripts/validate_production_readiness.sh
```

### Test Metrics Collection

#### Test Execution Metrics
```yaml
- name: Collect Test Metrics
  run: |
    echo "test_duration=$(date +%s)" >> $GITHUB_ENV
    cargo test --workspace -- --format json > test_results.json

    # Parse test results
    python scripts/parse_test_results.py test_results.json
```

#### Coverage Collection
```yaml
- name: Code Coverage
  run: |
    cargo install cargo-tarpaulin
    cargo tarpaulin --out Xml --output-dir coverage/

- name: Upload Coverage
  uses: codecov/codecov-action@v3
  with:
    file: coverage/tarpaulin-report.xml
    fail_ci_if_error: true
```

---

## 📊 Coverage Requirements

### Coverage Targets

The Foxhunt system maintains strict coverage requirements to ensure production reliability:

| Component | Line Coverage | Branch Coverage | Function Coverage | Requirements |
|-----------|---------------|-----------------|-------------------|--------------|
| **Core Trading** | ≥95% | ≥90% | 100% | Critical path |
| **Risk Management** | ≥98% | ≥95% | 100% | Zero tolerance |
| **ML Models** | ≥85% | ≥80% | ≥95% | Model validation |
| **Data Processing** | ≥90% | ≥85% | ≥95% | Data integrity |
| **Services** | ≥90% | ≥85% | ≥95% | Service reliability |
| **Utilities** | ≥80% | ≥75% | ≥90% | Support functions |

### Coverage Analysis

#### Core Components Coverage

```bash
# Generate detailed coverage report
cargo tarpaulin --workspace --out Html --output-dir coverage/ \
  --exclude-files "tests/*" "benches/*" \
  --timeout 300

# Critical components detailed analysis
cargo tarpaulin --packages foxhunt-core,risk,ml \
  --out Xml --output-dir coverage/critical/
```

#### Coverage Report Structure
```
coverage/
├── index.html                 # Main coverage dashboard
├── core/                      # Core trading system coverage
│   ├── trading/              # Trading algorithms
│   ├── risk/                 # Risk management
│   └── compliance/           # Compliance systems
├── ml/                       # ML model coverage
│   ├── models/               # Individual model coverage
│   └── inference/            # Inference pipeline coverage
├── services/                 # Service coverage
└── integration/              # Integration test coverage
```

#### Coverage Enforcement

```rust
// Coverage enforcement in CI
#[cfg(test)]
mod coverage_requirements {
    use super::*;

    #[test]
    fn enforce_critical_path_coverage() {
        // This test fails if critical paths are not fully covered
        let coverage = get_line_coverage("core/src/trading/");
        assert!(
            coverage >= 0.95,
            "Critical trading path coverage {} below required 95%",
            coverage
        );
    }

    #[test]
    fn enforce_risk_management_coverage() {
        // Risk management must have near-perfect coverage
        let coverage = get_line_coverage("risk/src/");
        assert!(
            coverage >= 0.98,
            "Risk management coverage {} below required 98%",
            coverage
        );
    }
}
```

### Coverage Exclusions

#### Justified Exclusions
```rust
// Example of justified coverage exclusion
impl OrderManager {
    pub fn process_order(&self, order: Order) -> Result<OrderResult, OrderError> {
        // Normal processing logic (covered by tests)
        match self.validate_order(&order) {
            Ok(_) => self.execute_order(order),
            Err(e) => {
                // Emergency logging - exclude from coverage
                #[cfg(not(tarpaulin_include))]
                emergency_log!("Critical order validation failure: {}", e);
                Err(e)
            }
        }
    }
}
```

#### Coverage Configuration (`.cargo/config.toml`)
```toml
[env]
# Coverage exclusion patterns
TARPAULIN_EXCLUDE = [
    "tests/*",
    "benches/*",
    "examples/*",
    "*/main.rs",
    "*emergency_log*"
]
```

### Differential Coverage

#### Pull Request Coverage
```yaml
- name: Differential Coverage Check
  run: |
    # Check coverage only on changed files
    git diff --name-only origin/main...HEAD | \
      grep "\.rs$" | \
      xargs cargo tarpaulin --files

    # Ensure new code meets coverage standards
    python scripts/check_differential_coverage.py
```

#### Coverage Regression Prevention
```bash
#!/bin/bash
# scripts/check_coverage_regression.sh

BASELINE_COVERAGE=$(cat coverage/baseline.txt)
CURRENT_COVERAGE=$(cargo tarpaulin --workspace --output-dir /tmp | grep "Coverage:" | cut -d' ' -f2)

if (( $(echo "$CURRENT_COVERAGE < $BASELINE_COVERAGE - 1.0" | bc -l) )); then
    echo "❌ Coverage regression detected: $CURRENT_COVERAGE% < $BASELINE_COVERAGE%"
    exit 1
else
    echo "✅ Coverage maintained: $CURRENT_COVERAGE%"
fi
```

---

## ⚡ Performance Requirements

### HFT Performance Standards

The Foxhunt system must meet strict latency requirements for high-frequency trading:

| Operation | Latency Requirement | Throughput Requirement | Validation Method |
|-----------|-------------------|----------------------|------------------|
| **Order Validation** | <10μs (99.9% ile) | >100K orders/sec | Hardware timing |
| **Risk Check** | <50μs (99.9% ile) | >50K checks/sec | RDTSC validation |
| **Market Data Processing** | <1μs (99.9% ile) | >1M ticks/sec | Lock-free queues |
| **Position Update** | <5μs (99.9% ile) | >200K updates/sec | Atomic operations |
| **Price Calculation** | <2μs (99.9% ile) | >500K calcs/sec | SIMD validation |
| **ML Inference** | <100μs (99.9% ile) | >10K predictions/sec | GPU acceleration |

### Performance Test Implementation

#### Latency Testing with Hardware Precision
```rust
/// Hardware-level latency measurement using RDTSC
pub struct HardwareTimer {
    cpu_frequency: u64,
}

impl HardwareTimer {
    pub fn new() -> Self {
        Self {
            cpu_frequency: Self::detect_cpu_frequency(),
        }
    }

    /// Measures operation latency with nanosecond precision
    pub fn measure<F, T>(&self, operation: F) -> (T, Duration)
    where
        F: FnOnce() -> T,
    {
        unsafe {
            let start = core::arch::x86_64::_rdtsc();
            let result = operation();
            let end = core::arch::x86_64::_rdtsc();

            let cycles = end - start;
            let nanos = (cycles * 1_000_000_000) / self.cpu_frequency;

            (result, Duration::from_nanos(nanos))
        }
    }
}

#[cfg(test)]
mod performance_tests {
    use super::*;

    #[test]
    fn test_order_validation_latency() -> TestResult<()> {
        let timer = HardwareTimer::new();
        let order_manager = OrderManager::new();
        let test_order = create_test_order();

        // Warm up CPU caches
        for _ in 0..1000 {
            let _ = order_manager.validate_order(&test_order);
        }

        // Measure actual latency (1000 iterations for statistical significance)
        let mut latencies = Vec::with_capacity(1000);
        for _ in 0..1000 {
            let (_, latency) = timer.measure(|| {
                order_manager.validate_order(&test_order)
            });
            latencies.push(latency.as_nanos() as u64);
        }

        // Statistical analysis
        let p99_9 = percentile(&mut latencies, 99.9);
        let mean = latencies.iter().sum::<u64>() / latencies.len() as u64;

        // Validate performance requirements
        HftPerformanceValidator::validate_latency("order_validation", p99_9)?;

        println!("Order Validation Performance:");
        println!("  Mean: {}ns", mean);
        println!("  99.9%ile: {}ns (requirement: <10,000ns)", p99_9);

        Ok(())
    }
}
```

#### Throughput Testing
```rust
#[test]
fn test_order_processing_throughput() -> TestResult<()> {
    let order_manager = OrderManager::new();
    let test_orders: Vec<Order> = (0..100_000)
        .map(|_| create_test_order())
        .collect();

    let start = Instant::now();

    // Process orders in parallel to measure throughput
    let results: Vec<_> = test_orders
        .par_iter()
        .map(|order| order_manager.process_order(order))
        .collect();

    let duration = start.elapsed();
    let throughput = test_orders.len() as f64 / duration.as_secs_f64();

    // Validate throughput requirement
    assert!(
        throughput >= 100_000.0,
        "Order processing throughput {}orders/sec below required 100K/sec",
        throughput as u64
    );

    println!("Order Processing Throughput: {:.0} orders/sec", throughput);
    Ok(())
}
```

### Memory Performance Testing
```rust
#[test]
fn test_memory_allocation_performance() -> TestResult<()> {
    use std::alloc::{GlobalAlloc, Layout, System};
    use std::time::Instant;

    // Test memory pool allocation performance
    let memory_pool = MemoryPool::new(1024 * 1024); // 1MB pool

    let timer = HardwareTimer::new();
    let mut allocation_times = Vec::with_capacity(10000);

    for _ in 0..10000 {
        let (_, duration) = timer.measure(|| {
            let ptr = memory_pool.allocate(64); // Allocate 64-byte order structure
            memory_pool.deallocate(ptr);
        });
        allocation_times.push(duration.as_nanos() as u64);
    }

    let p99 = percentile(&mut allocation_times, 99.0);

    // Memory allocation must be <100ns for HFT
    assert!(
        p99 < 100,
        "Memory allocation P99 latency {}ns exceeds 100ns requirement",
        p99
    );

    Ok(())
}
```

---

## 🔧 Troubleshooting

### Common Test Issues

#### 1. Database Connection Issues
```bash
# Problem: Tests fail with database connection errors
# Solution:
export DATABASE_URL="postgresql://localhost/foxhunt_test"
psql -c "CREATE DATABASE foxhunt_test;"
psql foxhunt_test -f tests/fixtures/test_schema.sql

# For Docker environments:
docker-compose -f docker-compose.test.yml up postgres -d
```

#### 2. GPU Tests Failing
```bash
# Problem: CUDA tests fail on CI
# Solution: Check GPU availability
nvidia-smi
export CUDA_VISIBLE_DEVICES=0

# Skip GPU tests if no GPU available:
cargo test --workspace -- --skip gpu_tests
```

#### 3. Performance Tests Inconsistent
```bash
# Problem: Performance tests show inconsistent results
# Solution: Isolate CPU cores and disable frequency scaling
sudo cpufreq-set -g performance
sudo taskset -c 0-3 cargo test performance --release

# Set CPU affinity in tests:
core_affinity::set_for_current(CoreId { id: 0 });
```

#### 4. Integration Tests Timeout
```bash
# Problem: Integration tests timeout
# Solution: Increase timeout and check service health
export TEST_TIMEOUT=300  # 5 minutes
./scripts/check_service_health.sh

# Debug service startup:
RUST_LOG=debug cargo test integration --nocapture
```

### Test Environment Setup

#### Development Environment
```bash
#!/bin/bash
# scripts/setup_test_environment.sh

echo "Setting up Foxhunt test environment..."

# Database setup
createdb foxhunt_test
psql foxhunt_test -f tests/fixtures/test_schema.sql

# Redis setup
redis-server --daemonize yes --port 6380

# Environment variables
export DATABASE_URL="postgresql://localhost/foxhunt_test"
export REDIS_URL="redis://localhost:6380"
export RUST_LOG=debug
export TEST_ENV=development

# Performance optimizations
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space

echo "✅ Test environment ready"
```

#### CI Environment Debug
```bash
# Debug CI failures
export RUST_BACKTRACE=full
export RUST_LOG=trace
cargo test --workspace --verbose -- --nocapture

# Generate detailed test report
cargo test --workspace -- --format json > test_results.json
python scripts/analyze_test_failures.py test_results.json
```

### Performance Debugging

#### Latency Spikes Investigation
```rust
#[cfg(test)]
mod performance_debug {
    #[test]
    fn debug_latency_spikes() -> TestResult<()> {
        let timer = HardwareTimer::new();
        let mut latencies = Vec::new();

        // Measure with detailed timing
        for i in 0..10000 {
            let (_, latency) = timer.measure(|| {
                // Your operation here
                expensive_operation()
            });

            let nanos = latency.as_nanos() as u64;
            latencies.push((i, nanos));

            // Log spikes for investigation
            if nanos > 50_000 { // >50μs spike
                println!("Latency spike at iteration {}: {}ns", i, nanos);
                // Additional debugging info
                print_cpu_state();
                print_memory_state();
            }
        }

        Ok(())
    }
}
```

### Test Data Management

#### Test Data Generation
```rust
/// Generates realistic test data for HFT scenarios
pub struct TestDataGenerator {
    rng: ChaCha8Rng,
}

impl TestDataGenerator {
    /// Creates realistic EUR/USD order flow for testing
    pub fn generate_eurusd_orders(&mut self, count: usize) -> Vec<Order> {
        (0..count)
            .map(|_| Order {
                symbol: "EURUSD".to_string(),
                side: if self.rng.gen_bool(0.5) { OrderSide::Buy } else { OrderSide::Sell },
                quantity: Decimal::from_f64(
                    self.rng.gen_range(1_000.0..1_000_000.0)
                ).unwrap(),
                price: Decimal::from_f64(
                    self.rng.gen_range(1.0500..1.1500)
                ).unwrap(),
                order_type: OrderType::Limit,
                time_in_force: TimeInForce::GoodTillCancel,
                timestamp: SystemTime::now(),
            })
            .collect()
    }
}
```

### Monitoring Test Health

#### Test Metrics Dashboard
```rust
/// Collects test execution metrics
pub struct TestMetricsCollector {
    execution_times: HashMap<String, Vec<Duration>>,
    failure_counts: HashMap<String, u64>,
    memory_usage: Vec<usize>,
}

impl TestMetricsCollector {
    pub fn record_test_execution(&mut self, test_name: &str, duration: Duration) {
        self.execution_times
            .entry(test_name.to_string())
            .or_default()
            .push(duration);
    }

    pub fn generate_health_report(&self) -> TestHealthReport {
        TestHealthReport {
            total_tests: self.execution_times.len(),
            average_execution_time: self.calculate_average_time(),
            slowest_tests: self.get_slowest_tests(10),
            failure_rate: self.calculate_failure_rate(),
            memory_trend: self.analyze_memory_trend(),
        }
    }
}
```

---

## 📈 Advanced Testing Strategies

### Property-Based Testing

#### Financial Property Validation
```rust
use proptest::prelude::*;

proptest! {
    #[test]
    fn test_position_value_conservation(
        orders in prop::collection::vec(arbitrary_order(), 1..100)
    ) {
        let mut position_manager = PositionManager::new();
        let initial_value = position_manager.total_value();

        // Apply all orders
        for order in orders {
            position_manager.apply_order(order)?;
        }

        // Property: Total value should be conserved (minus commissions)
        let final_value = position_manager.total_value();
        let commissions = position_manager.total_commissions();

        prop_assert_eq!(
            initial_value,
            final_value + commissions,
            "Position value not conserved"
        );
    }
}
```

### Mutation Testing

#### Code Robustness Validation
```bash
# Install cargo-mutagen for mutation testing
cargo install cargo-mutagen

# Run mutation tests on critical components
cargo mutagen --package foxhunt-core --package risk

# Analyze mutation test results
python scripts/analyze_mutation_results.py
```

### Stress Testing

#### System Limits Exploration
```rust
#[test]
fn stress_test_order_processing() -> TestResult<()> {
    let order_manager = OrderManager::new();
    let orders_per_second = [1_000, 10_000, 50_000, 100_000, 200_000];

    for &rate in &orders_per_second {
        println!("Testing {} orders/second...", rate);

        let start = Instant::now();
        let orders = generate_orders(rate);

        let results: Result<Vec<_>, _> = orders
            .into_iter()
            .map(|order| order_manager.process_order(order))
            .collect();

        match results {
            Ok(_) => {
                println!("✅ Successfully processed {} orders/second", rate);
            },
            Err(e) => {
                println!("❌ Failed at {} orders/second: {}", rate, e);
                break;
            }
        }
    }

    Ok(())
}
```

This comprehensive test documentation provides a complete guide to testing the Foxhunt HFT Trading System, covering architecture, execution, mocking strategies, CI/CD integration, and coverage requirements. The documentation ensures production-ready testing practices with zero-tolerance for failures.