foxhunt/benches

Foxhunt HFT Performance Benchmark Suite

Overview

Comprehensive benchmark suite for validating performance claims and detecting regressions in the Foxhunt HFT trading system. All benchmarks use Criterion.rs for statistical rigor and HTML report generation.

Benchmark Categories

1. Trading Latency (trading_latency.rs)

Validates critical trading path performance:

• Order Creation: Target <50μs p99

  • Limit order creation
  • Market order creation
  • Order validation

• Market Event Processing: Target <10μs p99

  • Trade event creation
  • Quote event creation
  • Event parsing overhead

• Position Calculations: Target <5μs

  • Market value updates
  • P&L calculations
  • Position aggregation

• Order Book Updates: Target <1μs p99

  • Bid/ask insertions
  • Best bid/ask lookups
  • Level updates

• Event Queue Operations: Target <1μs p99

  • Push operations
  • Pop operations
  • Push/pop cycles

• End-to-End Order Pipeline: Target <50μs p99

  • Full order creation → validation → submission flow

Run: cargo bench --bench trading_latency

2. Database Performance (database_performance.rs)

Validates database operation performance:

• Connection Acquisition: Target <5ms p99

  • Pool sizes: 5, 10, 20, 50 connections
  • Cold start vs warm pool

• Query Execution: Target <10ms p99

  • Simple SELECT queries
  • Parameterized queries
  • INSERT operations

• Transaction Latency: Target <15ms p99

  • BEGIN → COMMIT cycles
  • Rollback operations

• Pool Saturation: Graceful degradation

  • Concurrent request handling
  • Backpressure behavior

• Batch Operations: Amortized efficiency

  • Batch inserts vs individual
  • 100-record batches

• Index Lookups: O(log n) scaling

  • Table sizes: 1K, 10K, 100K, 1M rows

Run: cargo bench --bench database_performance

3. Streaming Throughput (streaming_throughput.rs)

Validates gRPC streaming performance:

• Message Throughput: Target >10,000 msg/sec

  • Payload sizes: 64B, 256B, 1KB, 4KB
  • Sustained throughput

• Stream Latency: Target p99 <1ms

  • Send → receive round-trip
  • Buffering overhead

• Backpressure Handling: Graceful degradation

  • Buffer saturation behavior
  • Flow control mechanisms

• Concurrent Streams: Target >100 streams

  • 10, 50, 100, 200 concurrent streams
  • Resource management

• Serialization Overhead: Minimal impact

  • Protocol buffer encoding/decoding
  • Payload size impact

• Flow Control: Window-based vs continuous

  • Windowed transmission
  • Acknowledgment overhead

Run: cargo bench --bench streaming_throughput

4. Metrics Overhead (metrics_overhead.rs)

Validates observability doesn't impact trading:

• Observation Overhead: Target <5μs per metric

  • Counter increments
  • Gauge sets
  • Histogram observations

• Registry Lookup: O(1) performance

  • Registry sizes: 10, 100, 1K, 10K metrics
  • Hash map efficiency

• Label Cardinality: Target >1000 unique labels

  • Label counts: 1, 5, 10, 20 per metric
  • Memory and CPU impact

• Aggregation: Target <100μs

  • Sample counts: 100, 1K, 10K
  • Percentile calculations

• Concurrent Updates: Lock contention

  • Thread counts: 1, 2, 4, 8
  • Mutex overhead

• Histogram Buckets: Bucket selection speed

  • Bucket counts: 10, 50, 100

Run: cargo bench --bench metrics_overhead

5. End-to-End Pipeline (end_to_end.rs)

Validates complete trading flow:

• Full Pipeline: Target <200μs p99

  • Market data ingestion
  • Signal generation
  • Risk validation
  • Order creation
  • Order submission
  • Confirmation receipt

• Pipeline Under Load: Throughput capacity

  • 100, 1K, 10K events/sec
  • Resource utilization

• Risk Validation Overhead: Target <10μs

  • Position limit checks
  • Drawdown calculations
  • With vs without validation

• Order Routing: Smart routing overhead

  • Direct market access
  • Multi-venue routing

Run: cargo bench --bench end_to_end

Running Benchmarks

All Benchmarks

cargo bench --workspace --all-features

Individual Benchmark

cargo bench --bench trading_latency
cargo bench --bench database_performance
cargo bench --bench streaming_throughput
cargo bench --bench metrics_overhead
cargo bench --bench end_to_end

With Baseline Comparison

# Save current results as baseline
cargo bench -- --save-baseline main

# Compare against baseline
cargo bench -- --baseline main

Filter Specific Tests

# Run only order creation benchmarks
cargo bench --bench trading_latency -- order_creation

# Run only throughput tests
cargo bench --bench streaming_throughput -- throughput

Interpreting Results

Criterion Output

Criterion provides:

• Mean: Average latency
• Std Dev: Variance in measurements
• Median: Middle value (50th percentile)
• Outliers: Statistical anomalies
• Change: Comparison vs baseline (if available)

HTML Reports

Detailed HTML reports are generated in target/criterion/:

• Violin plots showing distribution
• Time series charts
• Statistical analysis
• Regression detection

Open reports:

open target/criterion/report/index.html

Performance Targets

Component	Target	Critical?
Order Processing	<50μs p99	✅ Yes
Risk Validation	<5μs p99	✅ Yes
Market Data	<10μs p99	✅ Yes
Event Queue	<1μs p99	✅ Yes
DB Connection	<5ms p99	⚠️ Important
Query Execution	<10ms p99	⚠️ Important
gRPC Streaming	>10K msg/sec	✅ Yes
Stream Latency	<1ms p99	✅ Yes
Metrics Collection	<5μs	⚠️ Important
End-to-End Pipeline	<200μs p99	✅ Critical

CI/CD Integration

Automated Regression Detection

GitHub Actions workflow (.github/workflows/benchmark_regression.yml) runs on:

• Pull requests to main
• Pushes to main
• Manual workflow dispatch

Workflow Steps

1. Run Benchmarks: Execute full suite
2. Compare Baseline: Check for regressions vs main
3. Generate Report: Create markdown summary
4. Upload Artifacts: Store HTML reports (90 days)
5. Comment PR: Post results to pull request

Regression Criteria

⚠️ Review Required if:

• 10% performance degradation in critical paths

• 20% degradation in non-critical paths

• p99 latency exceeds targets
• Throughput falls below targets

Development Workflow

Adding New Benchmarks

1. Create benchmark file in benches/comprehensive/
2. Add to Cargo.toml:

   [[bench]]
   name = "my_benchmark"
   harness = false
   path = "benches/comprehensive/my_benchmark.rs"
   

3. Follow structure:

   use criterion::{criterion_group, criterion_main, Criterion};
   
   fn bench_function(c: &mut Criterion) {
       c.bench_function("test_name", |b| {
           b.iter(|| {
               // Code to benchmark
           });
       });
   }
   
   criterion_group!(benches, bench_function);
   criterion_main!(benches);
   

Validation Tests

Each benchmark includes #[cfg(test)] validation tests:

• Assert performance targets
• Verify behavior correctness
• Document expected performance

Run validation tests:

cargo test --benches

Best Practices

Measurement Accuracy

1. Warm-up iterations: Let JIT optimize
2. Sample size: Use sufficient iterations (Criterion default: 100)
3. Measurement time: Allow statistical significance (Criterion default: 5s)
4. Isolate system: Close other applications
5. CPU frequency: Lock frequency to avoid scaling

Avoiding Pitfalls

1. Don't optimize away: Use black_box() to prevent DCE
2. Minimize setup: Use iter_batched() for setup code
3. Realistic workload: Benchmark production-like scenarios
4. External factors: Be aware of system load, thermal throttling

Reproducibility

For consistent results:

# Lock CPU frequency (Linux)
sudo cpupower frequency-set -g performance

# Disable turbo boost
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo

# Set CPU affinity
taskset -c 0 cargo bench

Troubleshooting

Benchmark Compilation Fails

# Check dependencies
cargo check --benches

# Update lockfile
cargo update

Inconsistent Results

• Check system load: htop, iostat
• Verify CPU frequency: cpupower frequency-info
• Increase sample size: Use Criterion config
• Check thermal throttling: Monitor CPU temperature

CI Failures

• Review HTML reports in artifacts
• Compare against local results
• Check for environment differences
• Verify baseline compatibility

Performance Monitoring

Continuous Tracking

1. Baseline Updates: Update main baseline regularly
2. Trend Analysis: Track performance over time
3. Alerting: Set up alerts for critical regressions
4. Documentation: Update targets as system evolves

Production Correlation

Compare benchmark results with production metrics:

• Order processing latency (APM)
• Database query times (slow query log)
• gRPC stream throughput (monitoring)
• Overall system latency (distributed tracing)

References

• Criterion.rs User Guide
• Rust Performance Book
• CLAUDE.md Performance Targets
• 14ns Latency Validation

Questions?

For benchmark issues or questions:

1. Review this documentation
2. Check existing benchmark code for examples
3. Review Criterion.rs documentation
4. Open GitHub issue with benchmark results attached