foxhunt/PERFORMANCE_REGRESSION_QUICKSTART.md

# Performance Regression Testing - Quick Start Guide

**Status**: ✅ Production Ready
**Time to Setup**: 5 minutes
**Time to Run**: 10 minutes

---

## 1. Quick Start (5 Minutes)

### 1.1 Run Benchmarks

```bash
# Run performance regression suite
cargo bench --bench performance_regression

# Expected output:
# - 8 benchmark groups
# - ~40 individual benchmarks
# - 10 minutes runtime
# - HTML report at target/criterion/report/index.html
```

### 1.2 View Results

```bash
# Open HTML report
open target/criterion/report/index.html

# Check metrics:
# ✅ ML Prediction: P50 < 20μs, P99 < 50μs
# ✅ Hot-Swap: P50 < 1μs
# ✅ DB Writes: >1000/sec
# ✅ Backtest: >1100 bars/sec
# ✅ Order Processing: P99 < 100μs
# ✅ Risk Validation: P99 < 50μs
```

---

## 2. Record Baseline (1 Command)

```bash
# Record baseline for main branch
./scripts/record_baseline_metrics.sh main

# Output files:
# - performance_metrics/metrics_main_*.json
# - performance_metrics/baseline_summary_main.md
# - target/criterion/baselines/main/
```

---

## 3. Compare Performance (1 Command)

```bash
# After making changes, compare against baseline
cargo bench --bench performance_regression -- --baseline main

# Criterion will show:
# - Green: Performance improved
# - White: No significant change
# - Red: Performance regressed >5%
```

---

## 4. Generate Flame Graphs (Optional)

```bash
# Generate flame graphs for profiling (Linux only)
./scripts/generate_flame_graphs.sh

# View interactive flame graphs
open flame_graphs/index.html

# Or profile specific benchmark
./scripts/generate_flame_graphs.sh ml_prediction 60
```

---

## 5. CI Integration (Automatic)

Performance regression checks run automatically on:
- ✅ Pull requests (compare vs main)
- ✅ Main branch commits (save new baseline)

**CI Fails If**:
- Performance degrades >10%
- Critical metrics miss targets

---

## 6. Common Workflows

### 6.1 Before Committing

```bash
# 1. Run benchmarks
cargo bench --bench performance_regression

# 2. Check for regressions
cargo bench --bench performance_regression -- --baseline main

# 3. If regression found, investigate
./scripts/generate_flame_graphs.sh <benchmark-name> 60
```

### 6.2 After Optimization

```bash
# 1. Record pre-optimization baseline
cargo bench --bench performance_regression -- --save-baseline pre_opt

# 2. Make optimization changes

# 3. Compare improvement
cargo bench --bench performance_regression -- --baseline pre_opt

# 4. Save new baseline if satisfied
cargo bench --bench performance_regression -- --save-baseline main
```

### 6.3 Investigating Regression

```bash
# 1. Identify problematic benchmark
cargo bench --bench performance_regression -- --baseline main

# 2. Generate flame graph
./scripts/generate_flame_graphs.sh <benchmark-name> 60

# 3. Analyze flame graph
open flame_graphs/flamegraph_<benchmark-name>_*.svg

# 4. Look for wide frames (high CPU time)
# 5. Fix identified bottlenecks
# 6. Re-run benchmarks to verify
```

---

## 7. Benchmark Descriptions

| Benchmark | Target | Purpose |
|-----------|--------|---------|
| `ml_prediction_latency` | P50 20μs, P99 50μs | ML model inference speed |
| `hot_swap_latency` | P50 1μs | Model switching overhead |
| `database_writes` | 1000/sec | DB write throughput |
| `backtest_performance` | 1100 bars/sec | Strategy backtesting speed |
| `order_processing` | P99 100μs | Order operations latency |
| `risk_validation` | P99 50μs | Risk check overhead |
| `memory_allocation` | <1μs | Allocation overhead |
| `concurrent_access` | <10μs | Lock contention |

---

## 8. Troubleshooting

### 8.1 Benchmarks Taking Too Long

```bash
# Use quick mode (fewer samples)
cargo bench --bench performance_regression -- --sample-size 10
```

### 8.2 High Variance in Results

```bash
# Close other applications
# Disable CPU frequency scaling
# Use longer measurement time
cargo bench --bench performance_regression -- --measurement-time 60
```

### 8.3 Flame Graphs Not Working

```bash
# Flame graphs require Linux
# Install perf tools
sudo apt-get install linux-tools-$(uname -r)

# Set perf permissions
sudo sysctl kernel.perf_event_paranoid=-1
```

---

## 9. Key Files

| File | Purpose |
|------|---------|
| `benches/performance_regression.rs` | Main benchmark suite |
| `scripts/record_baseline_metrics.sh` | Baseline recording |
| `scripts/generate_flame_graphs.sh` | Flame graph generation |
| `target/criterion/baselines/` | Saved baselines |
| `target/criterion/report/` | HTML reports |
| `performance_metrics/` | Metrics JSON/markdown |
| `flame_graphs/` | Generated flame graphs |

---

## 10. Next Steps

✅ **Immediate**: Run benchmarks to establish baseline
```bash
cargo bench --bench performance_regression
./scripts/record_baseline_metrics.sh main
```

✅ **Before PR**: Compare against baseline
```bash
cargo bench --bench performance_regression -- --baseline main
```

✅ **Optimization**: Generate flame graphs
```bash
./scripts/generate_flame_graphs.sh
```

---

## 11. Help

**Full Documentation**: See `PERFORMANCE_REGRESSION_TEST_REPORT.md`

**Issues**:
- Compilation errors: `cargo clean && cargo build --bench performance_regression`
- CI failures: Review PR comments and HTML reports
- Performance questions: Check flame graphs

**Links**:
- [Criterion.rs Docs](https://bheisler.github.io/criterion.rs/)
- [Flame Graphs Guide](https://www.brendangregg.com/flamegraphs.html)

---

**Last Updated**: 2025-10-14
**Status**: ✅ Production Ready