foxhunt/PERFORMANCE_REGRESSION_QUICKSTART.md

Performance Regression Testing - Quick Start Guide

Status: ✅ Production Ready Time to Setup: 5 minutes Time to Run: 10 minutes

---

1. Quick Start (5 Minutes)

1.1 Run Benchmarks

# Run performance regression suite
cargo bench --bench performance_regression

# Expected output:
# - 8 benchmark groups
# - ~40 individual benchmarks
# - 10 minutes runtime
# - HTML report at target/criterion/report/index.html

1.2 View Results

# Open HTML report
open target/criterion/report/index.html

# Check metrics:
# ✅ ML Prediction: P50 < 20μs, P99 < 50μs
# ✅ Hot-Swap: P50 < 1μs
# ✅ DB Writes: >1000/sec
# ✅ Backtest: >1100 bars/sec
# ✅ Order Processing: P99 < 100μs
# ✅ Risk Validation: P99 < 50μs

---

2. Record Baseline (1 Command)

# Record baseline for main branch
./scripts/record_baseline_metrics.sh main

# Output files:
# - performance_metrics/metrics_main_*.json
# - performance_metrics/baseline_summary_main.md
# - target/criterion/baselines/main/

---

3. Compare Performance (1 Command)

# After making changes, compare against baseline
cargo bench --bench performance_regression -- --baseline main

# Criterion will show:
# - Green: Performance improved
# - White: No significant change
# - Red: Performance regressed >5%

---

4. Generate Flame Graphs (Optional)

# Generate flame graphs for profiling (Linux only)
./scripts/generate_flame_graphs.sh

# View interactive flame graphs
open flame_graphs/index.html

# Or profile specific benchmark
./scripts/generate_flame_graphs.sh ml_prediction 60

---

5. CI Integration (Automatic)

Performance regression checks run automatically on:

• ✅ Pull requests (compare vs main)
• ✅ Main branch commits (save new baseline)

CI Fails If:

• Performance degrades >10%
• Critical metrics miss targets

---

6. Common Workflows

6.1 Before Committing

# 1. Run benchmarks
cargo bench --bench performance_regression

# 2. Check for regressions
cargo bench --bench performance_regression -- --baseline main

# 3. If regression found, investigate
./scripts/generate_flame_graphs.sh <benchmark-name> 60

6.2 After Optimization

# 1. Record pre-optimization baseline
cargo bench --bench performance_regression -- --save-baseline pre_opt

# 2. Make optimization changes

# 3. Compare improvement
cargo bench --bench performance_regression -- --baseline pre_opt

# 4. Save new baseline if satisfied
cargo bench --bench performance_regression -- --save-baseline main

6.3 Investigating Regression

# 1. Identify problematic benchmark
cargo bench --bench performance_regression -- --baseline main

# 2. Generate flame graph
./scripts/generate_flame_graphs.sh <benchmark-name> 60

# 3. Analyze flame graph
open flame_graphs/flamegraph_<benchmark-name>_*.svg

# 4. Look for wide frames (high CPU time)
# 5. Fix identified bottlenecks
# 6. Re-run benchmarks to verify

---

7. Benchmark Descriptions

Benchmark	Target	Purpose
ml_prediction_latency	P50 20μs, P99 50μs	ML model inference speed
hot_swap_latency	P50 1μs	Model switching overhead
database_writes	1000/sec	DB write throughput
backtest_performance	1100 bars/sec	Strategy backtesting speed
order_processing	P99 100μs	Order operations latency
risk_validation	P99 50μs	Risk check overhead
memory_allocation	<1μs	Allocation overhead
concurrent_access	<10μs	Lock contention

---

8. Troubleshooting

8.1 Benchmarks Taking Too Long

# Use quick mode (fewer samples)
cargo bench --bench performance_regression -- --sample-size 10

8.2 High Variance in Results

# Close other applications
# Disable CPU frequency scaling
# Use longer measurement time
cargo bench --bench performance_regression -- --measurement-time 60

8.3 Flame Graphs Not Working

# Flame graphs require Linux
# Install perf tools
sudo apt-get install linux-tools-$(uname -r)

# Set perf permissions
sudo sysctl kernel.perf_event_paranoid=-1

---

9. Key Files

File	Purpose
benches/performance_regression.rs	Main benchmark suite
scripts/record_baseline_metrics.sh	Baseline recording
scripts/generate_flame_graphs.sh	Flame graph generation
target/criterion/baselines/	Saved baselines
target/criterion/report/	HTML reports
performance_metrics/	Metrics JSON/markdown
flame_graphs/	Generated flame graphs

---

10. Next Steps

✅ Immediate: Run benchmarks to establish baseline

cargo bench --bench performance_regression
./scripts/record_baseline_metrics.sh main

✅ Before PR: Compare against baseline

cargo bench --bench performance_regression -- --baseline main

✅ Optimization: Generate flame graphs

./scripts/generate_flame_graphs.sh

---

11. Help

Full Documentation: See PERFORMANCE_REGRESSION_TEST_REPORT.md

Issues:

• Compilation errors: cargo clean && cargo build --bench performance_regression
• CI failures: Review PR comments and HTML reports
• Performance questions: Check flame graphs

Links:

• Criterion.rs Docs
• Flame Graphs Guide

---

Last Updated: 2025-10-14 Status: ✅ Production Ready