foxhunt/PERFORMANCE_BENCHMARKS.md

# Performance Benchmarks - Foxhunt HFT System

## Overview

The Foxhunt HFT system includes three comprehensive benchmark suites to validate performance across all critical components:

1. **`e2e_performance.rs`** - Order lifecycle end-to-end performance
2. **`e2e_latency.rs`** - Cross-service latency profiling
3. **`comprehensive_performance.rs`** - Component-level comprehensive validation (NEW in Wave 125)

---

## Quick Start

### Run All Benchmarks
```bash
# Run all trading_engine benchmarks (3 suites, ~44 benchmarks)
cargo bench -p trading_engine

# View HTML report with graphs
open target/criterion/report/index.html
```

### Run Specific Suite
```bash
# Comprehensive performance suite only
cargo bench --bench comprehensive_performance

# E2E performance suite only
cargo bench --bench e2e_performance

# E2E latency suite only
cargo bench --bench e2e_latency
```

### Run Specific Benchmark Category
```bash
# Order processing benchmarks
cargo bench --bench comprehensive_performance order_processing

# Throughput validation
cargo bench --bench comprehensive_performance throughput

# Memory efficiency
cargo bench --bench comprehensive_performance memory
```

### Use Helper Script
```bash
# Run all benchmarks with organized output
./run_performance_benchmarks.sh
```

---

## Benchmark Suites

### 1. End-to-End Performance (`e2e_performance.rs`)

**Purpose**: Validate complete order lifecycle performance

**Categories**:
1. **Order Lifecycle** (3 benchmarks)
   - Single order submission
   - Batch order submission (10, 100, 1000)
   - Latency distribution with HDR histogram

2. **Execution Pipeline** (3 benchmarks)
   - Execution processing
   - Execution with position update
   - Concurrent execution (10, 100, 1000)

3. **Settlement Flow** (3 benchmarks)
   - PnL calculation
   - Full settlement flow (PnL + compliance + portfolio)
   - Batch settlement (100, 1K, 10K)

4. **Throughput Tests** (2 benchmarks)
   - Sustained throughput (1 second)
   - Burst handling (1K, 10K, 100K)

5. **Memory Efficiency** (1 benchmark)
   - Memory per order (100K orders)

6. **Component Breakdown** (1 benchmark)
   - Individual component latency profiling

7. **Target Validation** (1 benchmark)
   - Comprehensive target validation

**Total**: 14 benchmarks

**Targets**:
- Order lifecycle: P99 < 100μs
- Execution pipeline: P99 < 50μs
- Settlement: P99 < 75μs
- Throughput: > 100K orders/sec
- Memory: < 100B per order

### 2. End-to-End Latency (`e2e_latency.rs`)

**Purpose**: Measure cross-service latency and integration overhead

**Categories**:
1. **API Gateway** (1 benchmark)
   - API Gateway → Trading Service (target: <5ms)

2. **Order Flow** (1 benchmark)
   - Order placement → Fill (target: <10ms)

3. **Strategy** (1 benchmark)
   - Market data → Strategy decision (target: <2ms)

4. **Risk** (1 benchmark)
   - Risk check latency (target: <1ms)

5. **Component Breakdown** (1 benchmark)
   - Auth, rate limit, position lookup, ML inference

6. **Load Testing** (2 benchmarks)
   - Concurrent latency (10, 50, 100 concurrent)
   - Latency by load (100, 1K, 10K orders)

7. **Full E2E** (1 benchmark)
   - Complete cycle: Market data → Fill

**Total**: 10 benchmarks

**Targets**:
- API Gateway → Trading: P99 < 5ms
- Order → Fill: P99 < 10ms
- Market data → Decision: P99 < 2ms
- Risk check: P99 < 1ms

### 3. Comprehensive Performance (`comprehensive_performance.rs`) - NEW

**Purpose**: Component-level validation with comprehensive target coverage

**Categories**:
1. **Order Processing** (4 benchmarks)
   - Submission latency
   - Cancellation latency
   - Modification latency
   - Batch submission (10, 100, 1K, 10K)

2. **Position Management** (2 benchmarks)
   - Position update latency
   - Portfolio risk calculation

3. **Market Data** (2 benchmarks)
   - Ingestion throughput
   - Order book update latency

4. **Risk Management** (2 benchmarks)
   - Pre-trade risk check
   - Post-trade risk validation

5. **Compliance** (1 benchmark)
   - Audit logging overhead

6. **Throughput** (2 benchmarks)
   - Sustained load (1 second)
   - Burst handling (1K, 10K, 50K)

7. **Memory** (1 benchmark)
   - Per-operation allocation (100K orders)

8. **Comprehensive Validation** (1 benchmark)
   - All targets in one benchmark

**Total**: 20+ benchmarks

**Targets**:
- Order operations: P99 < 100μs
- Position updates: P99 < 50μs
- Market data: 50K+ events/sec, P99 < 20μs
- Risk checks: P99 < 50μs
- Audit overhead: < 10μs
- Throughput: 50K+ orders/sec
- Memory: < 100B/order

---

## Performance Targets Summary

| Component | Target | Benchmark Suite | Benchmark Name |
|-----------|--------|-----------------|----------------|
| Order submission | P99 < 100μs | comprehensive | `order_processing/submission_latency` |
| Order cancellation | P99 < 100μs | comprehensive | `order_processing/cancellation_latency` |
| Order modification | P99 < 100μs | comprehensive | `order_processing/modification_latency` |
| Position update | P99 < 50μs | comprehensive | `position_management/update_latency` |
| Portfolio risk | P99 < 50μs | comprehensive | `position_management/risk_calculation` |
| Market data ingestion | 50K+ events/sec | comprehensive | `market_data/ingestion_throughput` |
| Order book update | P99 < 20μs | comprehensive | `market_data/order_book_update` |
| Pre-trade risk | P99 < 50μs | comprehensive | `risk_management/pre_trade_check` |
| Post-trade risk | P99 < 50μs | comprehensive | `risk_management/post_trade_validation` |
| Audit overhead | < 10μs | comprehensive | `compliance/audit_logging_overhead` |
| Sustained throughput | 50K+ orders/sec | comprehensive | `throughput/sustained_load` |
| Burst handling | 50K+ orders/sec | comprehensive | `throughput/burst_handling` |
| Memory per order | < 100B | comprehensive | `memory/per_operation_allocation` |
| API Gateway latency | P99 < 5ms | e2e_latency | `api_gateway_to_trading` |
| Order to fill | P99 < 10ms | e2e_latency | `order_to_fill` |
| Market → decision | P99 < 2ms | e2e_latency | `market_data_to_decision` |
| Risk check | P99 < 1ms | e2e_latency | `risk_check` |

**Total**: 17 performance targets, all covered by benchmarks

---

## Advanced Usage

### Baseline Comparison
```bash
# Save baseline
cargo bench --bench comprehensive_performance -- --save-baseline v1.0

# Compare against baseline
cargo bench --bench comprehensive_performance -- --baseline v1.0

# See performance regression/improvement
```

### Export Results
```bash
# CSV format
cargo bench --bench comprehensive_performance -- --output-format csv > results.csv

# JSON format
cargo bench --bench comprehensive_performance -- --output-format json > results.json
```

### Filter Benchmarks
```bash
# Run only order processing benchmarks
cargo bench --bench comprehensive_performance order_processing

# Run only specific benchmark
cargo bench --bench comprehensive_performance order_processing/submission_latency
```

### Configure Sample Size
```bash
# Quick run (fewer samples)
cargo bench --bench comprehensive_performance -- --sample-size 10

# Thorough run (more samples, more accurate)
cargo bench --bench comprehensive_performance -- --sample-size 200
```

### Warm-up Configuration
```bash
# Longer warm-up for more stable results
cargo bench --bench comprehensive_performance -- --warm-up-time 10

# Skip warm-up for faster results
cargo bench --bench comprehensive_performance -- --warm-up-time 0
```

---

## Interpreting Results

### Latency Metrics

**Percentiles**:
- **P50 (Median)**: Typical case - 50% of operations faster than this
- **P90**: 90% of operations faster than this
- **P95**: 95% of operations faster than this
- **P99**: Critical for HFT - 99% of operations faster than this
- **P99.9**: Worst-case handling - only 0.1% slower
- **Max**: Absolute worst case observed

**What to Focus On**:
- **P99**: Primary target for HFT systems
- **Mean**: Average performance
- **Max**: Identify outliers and tail latency

### Throughput Metrics

**Measurement**:
- Operations per second (ops/sec)
- Orders per second (orders/sec)
- Events per second (events/sec)

**What to Focus On**:
- **Sustained throughput**: Can the system maintain this rate?
- **Burst handling**: How does latency degrade under load?

### Memory Metrics

**Measurement**:
- Bytes per operation
- Total memory delta (KB/MB)
- Extrapolation to 1M operations

**What to Focus On**:
- **Per-operation allocation**: Should be <100B for HFT
- **Memory leaks**: Delta should be reasonable for operation count

---

## Troubleshooting

### Benchmarks Taking Too Long
```bash
# Reduce sample size
cargo bench -- --sample-size 10

# Reduce measurement time
cargo bench -- --measurement-time 1
```

### Compilation Errors
```bash
# Check compilation first
cargo check -p trading_engine --benches

# Clean and rebuild
cargo clean
cargo build -p trading_engine --benches --release
```

### Memory Benchmarks Show 0KB (Non-Linux)
**Issue**: Memory tracking only works on Linux (VmRSS)

**Solution**: Run memory benchmarks on Linux environment:
```bash
# Run in Docker Linux container
docker run -v $(pwd):/workspace rust:latest cargo bench --bench comprehensive_performance memory
```

### High Variance in Results
**Causes**:
- Other processes running
- CPU throttling/power saving
- Cold caches

**Solutions**:
```bash
# Increase sample size
cargo bench -- --sample-size 200

# Longer warm-up
cargo bench -- --warm-up-time 10

# Isolate CPU cores (Linux)
taskset -c 0-3 cargo bench
```

---

## CI/CD Integration

### GitHub Actions Example
```yaml
name: Performance Benchmarks

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - name: Run benchmarks
        run: cargo bench -p trading_engine
      - name: Upload results
        uses: actions/upload-artifact@v2
        with:
          name: benchmark-results
          path: target/criterion/
```

### Regression Detection
```bash
# In CI, compare against main branch baseline
git checkout main
cargo bench -- --save-baseline main

git checkout feature-branch
cargo bench -- --baseline main

# Parse results for >10% regression
```

---

## Performance Dashboard

### Criterion HTML Report
```bash
# Generate and view
cargo bench
open target/criterion/report/index.html
```

**Features**:
- Latency distribution graphs
- Throughput comparisons
- Statistical analysis
- Baseline comparisons
- Per-benchmark details

### Custom Dashboard (Future)
```bash
# Export to CSV and import to Grafana
cargo bench -- --output-format csv > results.csv

# Or push to time-series database
./scripts/push_benchmark_results.sh
```

---

## Best Practices

### 1. Consistent Environment
- Run on same hardware
- Close unnecessary applications
- Use release builds (`--release` is default for benchmarks)

### 2. Multiple Runs
- Don't rely on single run
- Criterion automatically runs multiple samples
- Check for high variance

### 3. Baseline Tracking
- Always save baselines for comparison
- Track performance over time
- Detect regressions early

### 4. Focus on P99
- P99 is critical for HFT
- Mean can be misleading
- Max shows worst case

### 5. Real-World Workloads
- Use realistic data sizes
- Test concurrent scenarios
- Include error handling paths

---

## Files

### Benchmark Files
- `/trading_engine/benches/e2e_performance.rs` - Order lifecycle benchmarks
- `/trading_engine/benches/e2e_latency.rs` - Cross-service latency benchmarks
- `/trading_engine/benches/comprehensive_performance.rs` - Component-level benchmarks

### Helper Scripts
- `/run_performance_benchmarks.sh` - Run all benchmarks with organized output

### Documentation
- `/PERFORMANCE_BENCHMARKS.md` - This file
- `/tmp/WAVE_125_AGENT_90_PERFORMANCE.md` - Comprehensive benchmark suite report

---

## FAQ

**Q: How long do benchmarks take?**
A: Full suite (~44 benchmarks) takes 30-60 minutes. Individual categories take 2-5 minutes.

**Q: Can I run benchmarks in parallel?**
A: No, benchmarks should run sequentially for accurate results. Parallel runs interfere with measurements.

**Q: Why are my results different from expected?**
A: Results vary by hardware, OS, load, etc. Focus on relative comparisons and trends.

**Q: Should I run benchmarks on CI?**
A: Yes, but use baseline comparison to detect regressions. CI environments may have different absolute performance.

**Q: How do I know if I meet targets?**
A: Benchmarks include automated target validation with ✅/❌ indicators in output.

**Q: What if a benchmark fails?**
A: Check compilation first, then verify test data is valid. Most failures are configuration issues, not actual performance problems.

---

## Support

For issues or questions:
1. Check troubleshooting section above
2. Review benchmark code for expected behavior
3. Compare against expected results in documentation
4. Check Criterion.rs documentation: https://bheisler.github.io/criterion.rs/

---

**Last Updated**: Wave 125 - Agent 90
**Status**: Production-ready comprehensive benchmark suite
**Coverage**: 17 performance targets, all validated