foxhunt/docs/archive/performance/PERFORMANCE_BENCHMARKS.md

Performance Benchmarks - Foxhunt HFT System

Overview

The Foxhunt HFT system includes three comprehensive benchmark suites to validate performance across all critical components:

1. e2e_performance.rs - Order lifecycle end-to-end performance
2. e2e_latency.rs - Cross-service latency profiling
3. comprehensive_performance.rs - Component-level comprehensive validation (NEW in Wave 125)

---

Quick Start

Run All Benchmarks

# Run all trading_engine benchmarks (3 suites, ~44 benchmarks)
cargo bench -p trading_engine

# View HTML report with graphs
open target/criterion/report/index.html

Run Specific Suite

# Comprehensive performance suite only
cargo bench --bench comprehensive_performance

# E2E performance suite only
cargo bench --bench e2e_performance

# E2E latency suite only
cargo bench --bench e2e_latency

Run Specific Benchmark Category

# Order processing benchmarks
cargo bench --bench comprehensive_performance order_processing

# Throughput validation
cargo bench --bench comprehensive_performance throughput

# Memory efficiency
cargo bench --bench comprehensive_performance memory

Use Helper Script

# Run all benchmarks with organized output
./run_performance_benchmarks.sh

---

Benchmark Suites

1. End-to-End Performance (e2e_performance.rs)

Purpose: Validate complete order lifecycle performance

Categories:

1. Order Lifecycle (3 benchmarks)

   • Single order submission
   • Batch order submission (10, 100, 1000)
   • Latency distribution with HDR histogram

2. Execution Pipeline (3 benchmarks)

   • Execution processing
   • Execution with position update
   • Concurrent execution (10, 100, 1000)

3. Settlement Flow (3 benchmarks)

   • PnL calculation
   • Full settlement flow (PnL + compliance + portfolio)
   • Batch settlement (100, 1K, 10K)

4. Throughput Tests (2 benchmarks)

   • Sustained throughput (1 second)
   • Burst handling (1K, 10K, 100K)

5. Memory Efficiency (1 benchmark)

   • Memory per order (100K orders)

6. Component Breakdown (1 benchmark)

   • Individual component latency profiling

7. Target Validation (1 benchmark)

   • Comprehensive target validation

Total: 14 benchmarks

Targets:

• Order lifecycle: P99 < 100μs
• Execution pipeline: P99 < 50μs
• Settlement: P99 < 75μs
• Throughput: > 100K orders/sec
• Memory: < 100B per order

2. End-to-End Latency (e2e_latency.rs)

Purpose: Measure cross-service latency and integration overhead

Categories:

1. API Gateway (1 benchmark)

   • API Gateway → Trading Service (target: <5ms)

2. Order Flow (1 benchmark)

   • Order placement → Fill (target: <10ms)

3. Strategy (1 benchmark)

   • Market data → Strategy decision (target: <2ms)

4. Risk (1 benchmark)

   • Risk check latency (target: <1ms)

5. Component Breakdown (1 benchmark)

   • Auth, rate limit, position lookup, ML inference

6. Load Testing (2 benchmarks)

   • Concurrent latency (10, 50, 100 concurrent)
   • Latency by load (100, 1K, 10K orders)

7. Full E2E (1 benchmark)

   • Complete cycle: Market data → Fill

Total: 10 benchmarks

Targets:

• API Gateway → Trading: P99 < 5ms
• Order → Fill: P99 < 10ms
• Market data → Decision: P99 < 2ms
• Risk check: P99 < 1ms

3. Comprehensive Performance (comprehensive_performance.rs) - NEW

Purpose: Component-level validation with comprehensive target coverage

Categories:

1. Order Processing (4 benchmarks)

   • Submission latency
   • Cancellation latency
   • Modification latency
   • Batch submission (10, 100, 1K, 10K)

2. Position Management (2 benchmarks)

   • Position update latency
   • Portfolio risk calculation

3. Market Data (2 benchmarks)

   • Ingestion throughput
   • Order book update latency

4. Risk Management (2 benchmarks)

   • Pre-trade risk check
   • Post-trade risk validation

5. Compliance (1 benchmark)

   • Audit logging overhead

6. Throughput (2 benchmarks)

   • Sustained load (1 second)
   • Burst handling (1K, 10K, 50K)

7. Memory (1 benchmark)

   • Per-operation allocation (100K orders)

8. Comprehensive Validation (1 benchmark)

   • All targets in one benchmark

Total: 20+ benchmarks

Targets:

• Order operations: P99 < 100μs
• Position updates: P99 < 50μs
• Market data: 50K+ events/sec, P99 < 20μs
• Risk checks: P99 < 50μs
• Audit overhead: < 10μs
• Throughput: 50K+ orders/sec
• Memory: < 100B/order

---

Performance Targets Summary

Component	Target	Benchmark Suite	Benchmark Name
Order submission	P99 < 100μs	comprehensive	order_processing/submission_latency
Order cancellation	P99 < 100μs	comprehensive	order_processing/cancellation_latency
Order modification	P99 < 100μs	comprehensive	order_processing/modification_latency
Position update	P99 < 50μs	comprehensive	position_management/update_latency
Portfolio risk	P99 < 50μs	comprehensive	position_management/risk_calculation
Market data ingestion	50K+ events/sec	comprehensive	market_data/ingestion_throughput
Order book update	P99 < 20μs	comprehensive	market_data/order_book_update
Pre-trade risk	P99 < 50μs	comprehensive	risk_management/pre_trade_check
Post-trade risk	P99 < 50μs	comprehensive	risk_management/post_trade_validation
Audit overhead	< 10μs	comprehensive	compliance/audit_logging_overhead
Sustained throughput	50K+ orders/sec	comprehensive	throughput/sustained_load
Burst handling	50K+ orders/sec	comprehensive	throughput/burst_handling
Memory per order	< 100B	comprehensive	memory/per_operation_allocation
API Gateway latency	P99 < 5ms	e2e_latency	api_gateway_to_trading
Order to fill	P99 < 10ms	e2e_latency	order_to_fill
Market → decision	P99 < 2ms	e2e_latency	market_data_to_decision
Risk check	P99 < 1ms	e2e_latency	risk_check

Total: 17 performance targets, all covered by benchmarks

---

Advanced Usage

Baseline Comparison

# Save baseline
cargo bench --bench comprehensive_performance -- --save-baseline v1.0

# Compare against baseline
cargo bench --bench comprehensive_performance -- --baseline v1.0

# See performance regression/improvement

Export Results

# CSV format
cargo bench --bench comprehensive_performance -- --output-format csv > results.csv

# JSON format
cargo bench --bench comprehensive_performance -- --output-format json > results.json

Filter Benchmarks

# Run only order processing benchmarks
cargo bench --bench comprehensive_performance order_processing

# Run only specific benchmark
cargo bench --bench comprehensive_performance order_processing/submission_latency

Configure Sample Size

# Quick run (fewer samples)
cargo bench --bench comprehensive_performance -- --sample-size 10

# Thorough run (more samples, more accurate)
cargo bench --bench comprehensive_performance -- --sample-size 200

Warm-up Configuration

# Longer warm-up for more stable results
cargo bench --bench comprehensive_performance -- --warm-up-time 10

# Skip warm-up for faster results
cargo bench --bench comprehensive_performance -- --warm-up-time 0

---

Interpreting Results

Latency Metrics

Percentiles:

• P50 (Median): Typical case - 50% of operations faster than this
• P90: 90% of operations faster than this
• P95: 95% of operations faster than this
• P99: Critical for HFT - 99% of operations faster than this
• P99.9: Worst-case handling - only 0.1% slower
• Max: Absolute worst case observed

What to Focus On:

• P99: Primary target for HFT systems
• Mean: Average performance
• Max: Identify outliers and tail latency

Throughput Metrics

Measurement:

• Operations per second (ops/sec)
• Orders per second (orders/sec)
• Events per second (events/sec)

What to Focus On:

• Sustained throughput: Can the system maintain this rate?
• Burst handling: How does latency degrade under load?

Memory Metrics

Measurement:

• Bytes per operation
• Total memory delta (KB/MB)
• Extrapolation to 1M operations

What to Focus On:

• Per-operation allocation: Should be <100B for HFT
• Memory leaks: Delta should be reasonable for operation count

---

Troubleshooting

Benchmarks Taking Too Long

# Reduce sample size
cargo bench -- --sample-size 10

# Reduce measurement time
cargo bench -- --measurement-time 1

Compilation Errors

# Check compilation first
cargo check -p trading_engine --benches

# Clean and rebuild
cargo clean
cargo build -p trading_engine --benches --release

Memory Benchmarks Show 0KB (Non-Linux)

Issue: Memory tracking only works on Linux (VmRSS)

Solution: Run memory benchmarks on Linux environment:

# Run in Docker Linux container
docker run -v $(pwd):/workspace rust:latest cargo bench --bench comprehensive_performance memory

High Variance in Results

Causes:

• Other processes running
• CPU throttling/power saving
• Cold caches

Solutions:

# Increase sample size
cargo bench -- --sample-size 200

# Longer warm-up
cargo bench -- --warm-up-time 10

# Isolate CPU cores (Linux)
taskset -c 0-3 cargo bench

---

CI/CD Integration

GitHub Actions Example

name: Performance Benchmarks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - name: Run benchmarks
        run: cargo bench -p trading_engine
      - name: Upload results
        uses: actions/upload-artifact@v2
        with:
          name: benchmark-results
          path: target/criterion/

Regression Detection

# In CI, compare against main branch baseline
git checkout main
cargo bench -- --save-baseline main

git checkout feature-branch
cargo bench -- --baseline main

# Parse results for >10% regression

---

Performance Dashboard

Criterion HTML Report

# Generate and view
cargo bench
open target/criterion/report/index.html

Features:

• Latency distribution graphs
• Throughput comparisons
• Statistical analysis
• Baseline comparisons
• Per-benchmark details

Custom Dashboard (Future)

# Export to CSV and import to Grafana
cargo bench -- --output-format csv > results.csv

# Or push to time-series database
./scripts/push_benchmark_results.sh

---

Best Practices

1. Consistent Environment

• Run on same hardware
• Close unnecessary applications
• Use release builds (--release is default for benchmarks)

2. Multiple Runs

• Don't rely on single run
• Criterion automatically runs multiple samples
• Check for high variance

3. Baseline Tracking

• Always save baselines for comparison
• Track performance over time
• Detect regressions early

4. Focus on P99

• P99 is critical for HFT
• Mean can be misleading
• Max shows worst case

5. Real-World Workloads

• Use realistic data sizes
• Test concurrent scenarios
• Include error handling paths

---

Files

Benchmark Files

• /trading_engine/benches/e2e_performance.rs - Order lifecycle benchmarks
• /trading_engine/benches/e2e_latency.rs - Cross-service latency benchmarks
• /trading_engine/benches/comprehensive_performance.rs - Component-level benchmarks

Helper Scripts

• /run_performance_benchmarks.sh - Run all benchmarks with organized output

Documentation

• /PERFORMANCE_BENCHMARKS.md - This file
• /tmp/WAVE_125_AGENT_90_PERFORMANCE.md - Comprehensive benchmark suite report

---

FAQ

Q: How long do benchmarks take? A: Full suite (~44 benchmarks) takes 30-60 minutes. Individual categories take 2-5 minutes.

Q: Can I run benchmarks in parallel? A: No, benchmarks should run sequentially for accurate results. Parallel runs interfere with measurements.

Q: Why are my results different from expected? A: Results vary by hardware, OS, load, etc. Focus on relative comparisons and trends.

Q: Should I run benchmarks on CI? A: Yes, but use baseline comparison to detect regressions. CI environments may have different absolute performance.

Q: How do I know if I meet targets? A: Benchmarks include automated target validation with ✅/❌ indicators in output.

Q: What if a benchmark fails? A: Check compilation first, then verify test data is valid. Most failures are configuration issues, not actual performance problems.

---

Support

For issues or questions:

1. Check troubleshooting section above
2. Review benchmark code for expected behavior
3. Compare against expected results in documentation
4. Check Criterion.rs documentation: https://bheisler.github.io/criterion.rs/

---

Last Updated: Wave 125 - Agent 90 Status: Production-ready comprehensive benchmark suite Coverage: 17 performance targets, all validated