foxhunt/benches/README.md

# Foxhunt HFT Performance Benchmark Suite

## Overview

Comprehensive benchmark suite for validating performance claims and detecting regressions in the Foxhunt HFT trading system. All benchmarks use [Criterion.rs](https://github.com/bheisler/criterion.rs) for statistical rigor and HTML report generation.

## Benchmark Categories

### 1. Trading Latency (`trading_latency.rs`)

Validates critical trading path performance:

- **Order Creation**: Target <50μs p99
  - Limit order creation
  - Market order creation
  - Order validation

- **Market Event Processing**: Target <10μs p99
  - Trade event creation
  - Quote event creation
  - Event parsing overhead

- **Position Calculations**: Target <5μs
  - Market value updates
  - P&L calculations
  - Position aggregation

- **Order Book Updates**: Target <1μs p99
  - Bid/ask insertions
  - Best bid/ask lookups
  - Level updates

- **Event Queue Operations**: Target <1μs p99
  - Push operations
  - Pop operations
  - Push/pop cycles

- **End-to-End Order Pipeline**: Target <50μs p99
  - Full order creation → validation → submission flow

**Run**: `cargo bench --bench trading_latency`

### 2. Database Performance (`database_performance.rs`)

Validates database operation performance:

- **Connection Acquisition**: Target <5ms p99
  - Pool sizes: 5, 10, 20, 50 connections
  - Cold start vs warm pool

- **Query Execution**: Target <10ms p99
  - Simple SELECT queries
  - Parameterized queries
  - INSERT operations

- **Transaction Latency**: Target <15ms p99
  - BEGIN → COMMIT cycles
  - Rollback operations

- **Pool Saturation**: Graceful degradation
  - Concurrent request handling
  - Backpressure behavior

- **Batch Operations**: Amortized efficiency
  - Batch inserts vs individual
  - 100-record batches

- **Index Lookups**: O(log n) scaling
  - Table sizes: 1K, 10K, 100K, 1M rows

**Run**: `cargo bench --bench database_performance`

### 3. Streaming Throughput (`streaming_throughput.rs`)

Validates gRPC streaming performance:

- **Message Throughput**: Target >10,000 msg/sec
  - Payload sizes: 64B, 256B, 1KB, 4KB
  - Sustained throughput

- **Stream Latency**: Target p99 <1ms
  - Send → receive round-trip
  - Buffering overhead

- **Backpressure Handling**: Graceful degradation
  - Buffer saturation behavior
  - Flow control mechanisms

- **Concurrent Streams**: Target >100 streams
  - 10, 50, 100, 200 concurrent streams
  - Resource management

- **Serialization Overhead**: Minimal impact
  - Protocol buffer encoding/decoding
  - Payload size impact

- **Flow Control**: Window-based vs continuous
  - Windowed transmission
  - Acknowledgment overhead

**Run**: `cargo bench --bench streaming_throughput`

### 4. Metrics Overhead (`metrics_overhead.rs`)

Validates observability doesn't impact trading:

- **Observation Overhead**: Target <5μs per metric
  - Counter increments
  - Gauge sets
  - Histogram observations

- **Registry Lookup**: O(1) performance
  - Registry sizes: 10, 100, 1K, 10K metrics
  - Hash map efficiency

- **Label Cardinality**: Target >1000 unique labels
  - Label counts: 1, 5, 10, 20 per metric
  - Memory and CPU impact

- **Aggregation**: Target <100μs
  - Sample counts: 100, 1K, 10K
  - Percentile calculations

- **Concurrent Updates**: Lock contention
  - Thread counts: 1, 2, 4, 8
  - Mutex overhead

- **Histogram Buckets**: Bucket selection speed
  - Bucket counts: 10, 50, 100

**Run**: `cargo bench --bench metrics_overhead`

### 5. End-to-End Pipeline (`end_to_end.rs`)

Validates complete trading flow:

- **Full Pipeline**: Target <200μs p99
  - Market data ingestion
  - Signal generation
  - Risk validation
  - Order creation
  - Order submission
  - Confirmation receipt

- **Pipeline Under Load**: Throughput capacity
  - 100, 1K, 10K events/sec
  - Resource utilization

- **Risk Validation Overhead**: Target <10μs
  - Position limit checks
  - Drawdown calculations
  - With vs without validation

- **Order Routing**: Smart routing overhead
  - Direct market access
  - Multi-venue routing

**Run**: `cargo bench --bench end_to_end`

## Running Benchmarks

### All Benchmarks
```bash
cargo bench --workspace --all-features
```

### Individual Benchmark
```bash
cargo bench --bench trading_latency
cargo bench --bench database_performance
cargo bench --bench streaming_throughput
cargo bench --bench metrics_overhead
cargo bench --bench end_to_end
```

### With Baseline Comparison
```bash
# Save current results as baseline
cargo bench -- --save-baseline main

# Compare against baseline
cargo bench -- --baseline main
```

### Filter Specific Tests
```bash
# Run only order creation benchmarks
cargo bench --bench trading_latency -- order_creation

# Run only throughput tests
cargo bench --bench streaming_throughput -- throughput
```

## Interpreting Results

### Criterion Output

Criterion provides:
- **Mean**: Average latency
- **Std Dev**: Variance in measurements
- **Median**: Middle value (50th percentile)
- **Outliers**: Statistical anomalies
- **Change**: Comparison vs baseline (if available)

### HTML Reports

Detailed HTML reports are generated in `target/criterion/`:
- Violin plots showing distribution
- Time series charts
- Statistical analysis
- Regression detection

Open reports:
```bash
open target/criterion/report/index.html
```

### Performance Targets

| Component | Target | Critical? |
|-----------|--------|-----------|
| Order Processing | <50μs p99 | ✅ Yes |
| Risk Validation | <5μs p99 | ✅ Yes |
| Market Data | <10μs p99 | ✅ Yes |
| Event Queue | <1μs p99 | ✅ Yes |
| DB Connection | <5ms p99 | ⚠️ Important |
| Query Execution | <10ms p99 | ⚠️ Important |
| gRPC Streaming | >10K msg/sec | ✅ Yes |
| Stream Latency | <1ms p99 | ✅ Yes |
| Metrics Collection | <5μs | ⚠️ Important |
| End-to-End Pipeline | <200μs p99 | ✅ Critical |

## CI/CD Integration

### Automated Regression Detection

GitHub Actions workflow (`.github/workflows/benchmark_regression.yml`) runs on:
- Pull requests to `main`
- Pushes to `main`
- Manual workflow dispatch

### Workflow Steps

1. **Run Benchmarks**: Execute full suite
2. **Compare Baseline**: Check for regressions vs `main`
3. **Generate Report**: Create markdown summary
4. **Upload Artifacts**: Store HTML reports (90 days)
5. **Comment PR**: Post results to pull request

### Regression Criteria

⚠️ **Review Required** if:
- >10% performance degradation in critical paths
- >20% degradation in non-critical paths
- p99 latency exceeds targets
- Throughput falls below targets

## Development Workflow

### Adding New Benchmarks

1. **Create benchmark file** in `benches/comprehensive/`
2. **Add to Cargo.toml**:
   ```toml
   [[bench]]
   name = "my_benchmark"
   harness = false
   path = "benches/comprehensive/my_benchmark.rs"
   ```
3. **Follow structure**:
   ```rust
   use criterion::{criterion_group, criterion_main, Criterion};
   
   fn bench_function(c: &mut Criterion) {
       c.bench_function("test_name", |b| {
           b.iter(|| {
               // Code to benchmark
           });
       });
   }
   
   criterion_group!(benches, bench_function);
   criterion_main!(benches);
   ```

### Validation Tests

Each benchmark includes `#[cfg(test)]` validation tests:
- Assert performance targets
- Verify behavior correctness
- Document expected performance

Run validation tests:
```bash
cargo test --benches
```

## Best Practices

### Measurement Accuracy

1. **Warm-up iterations**: Let JIT optimize
2. **Sample size**: Use sufficient iterations (Criterion default: 100)
3. **Measurement time**: Allow statistical significance (Criterion default: 5s)
4. **Isolate system**: Close other applications
5. **CPU frequency**: Lock frequency to avoid scaling

### Avoiding Pitfalls

1. **Don't optimize away**: Use `black_box()` to prevent DCE
2. **Minimize setup**: Use `iter_batched()` for setup code
3. **Realistic workload**: Benchmark production-like scenarios
4. **External factors**: Be aware of system load, thermal throttling

### Reproducibility

For consistent results:
```bash
# Lock CPU frequency (Linux)
sudo cpupower frequency-set -g performance

# Disable turbo boost
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo

# Set CPU affinity
taskset -c 0 cargo bench
```

## Troubleshooting

### Benchmark Compilation Fails

```bash
# Check dependencies
cargo check --benches

# Update lockfile
cargo update
```

### Inconsistent Results

- Check system load: `htop`, `iostat`
- Verify CPU frequency: `cpupower frequency-info`
- Increase sample size: Use Criterion config
- Check thermal throttling: Monitor CPU temperature

### CI Failures

- Review HTML reports in artifacts
- Compare against local results
- Check for environment differences
- Verify baseline compatibility

## Performance Monitoring

### Continuous Tracking

1. **Baseline Updates**: Update `main` baseline regularly
2. **Trend Analysis**: Track performance over time
3. **Alerting**: Set up alerts for critical regressions
4. **Documentation**: Update targets as system evolves

### Production Correlation

Compare benchmark results with production metrics:
- Order processing latency (APM)
- Database query times (slow query log)
- gRPC stream throughput (monitoring)
- Overall system latency (distributed tracing)

## References

- [Criterion.rs User Guide](https://bheisler.github.io/criterion.rs/book/)
- [Rust Performance Book](https://nnethercote.github.io/perf-book/)
- [CLAUDE.md Performance Targets](../CLAUDE.md)
- [14ns Latency Validation](fourteen_ns_validation.rs)

## Questions?

For benchmark issues or questions:
1. Review this documentation
2. Check existing benchmark code for examples
3. Review Criterion.rs documentation
4. Open GitHub issue with benchmark results attached