foxhunt/docs/archive/wave_d/reports/OPTIMIZATION_QUICK_START.md

Rust Compiler Optimization - Quick Start Guide

Generated: 2025-10-25 Current Status: Baseline (922x faster than targets) Target: 1,032-1,180x faster than targets (12-28% improvement)

---

🚀 TL;DR - Execute This Week

# 1. Install cargo-pgo (5 minutes)
cargo install cargo-pgo

# 2. Run PGO pipeline (20-30 minutes)
./scripts/implement_pgo.sh

# 3. Add musl target for static linking (5 minutes)
rustup target add x86_64-unknown-linux-musl
cargo build --release --target x86_64-unknown-linux-musl

# 4. Test native CPU builds locally (5 minutes)
cargo build --release --config target.x86_64-unknown-linux-gnu.local
cargo bench --bench performance_regression

# Expected result: 7-22% performance improvement (average: 14%)

---

📋 Step-by-Step Implementation

Step 1: Profile-Guided Optimization (PGO)

Expected Gain: 5-15% (average: 10%) Time Required: 1-2 hours

# Install cargo-pgo
cargo install cargo-pgo

# Option A: Automated (recommended)
./scripts/implement_pgo.sh

# Option B: Manual
# 1. Build instrumented binary
RUSTFLAGS="-C target-cpu=native" cargo pgo build --release

# 2. Run representative workloads to generate profile data
# Workload 1: Backtesting
cargo run --release -p backtesting_service -- --symbol ES.FUT --duration 180d

# Workload 2: ML training
cargo run --release -p ml --example train_tft_parquet -- \
  --parquet-file test_data/ES_FUT_180d.parquet --epochs 10

# Workload 3: Benchmarks
cargo bench --bench performance_regression

# 3. Build optimized binary
RUSTFLAGS="-C target-cpu=native" cargo pgo optimize --release

# 4. Validate improvements
cargo bench --bench performance_regression -- --baseline pgo-before

Validation: Check target/criterion/ for improvement reports

---

Step 2: Static Linking (musl)

Expected Gain: <1% latency, 5-10% jitter reduction Time Required: 30 minutes

# Add musl target
rustup target add x86_64-unknown-linux-musl

# Build static binary
cargo build --release --target x86_64-unknown-linux-musl

# Verify static linking (should show "not a dynamic executable")
ldd target/x86_64-unknown-linux-musl/release/api_gateway

# Test binary
./target/x86_64-unknown-linux-musl/release/api_gateway --help

Benefits:

• Zero dynamic dependencies
• Reduced jitter from PLT/GOT indirection
• Faster startup time

---

Step 3: Native CPU Targeting (Local Builds)

Expected Gain: 0-5% (average: 2.5%) Time Required: 15 minutes

# Option 1: Use .cargo/config.toml.optimized (recommended)
cp .cargo/config.toml .cargo/config.toml.backup
cp .cargo/config.toml.optimized .cargo/config.toml

# Option 2: Manual RUSTFLAGS
export RUSTFLAGS="-C target-cpu=native"
cargo build --release

# Benchmark improvements
cargo bench --bench performance_regression -- --baseline before-native --save-baseline after-native

Local CPU Features (i7-11800H):

• AVX-512F, AVX-512DQ, AVX-512BW, AVX-512VL
• ADX (Multi-Precision Add-Carry)
• SHA-NI (Hardware SHA hashing)
• VAES, VPCLMULQDQ (Advanced crypto)

Warning: Do NOT use target-cpu=native for Runpod deployment (incompatible with V100 CPU)

---

Step 4: Separate Profiling/Production Builds

Expected Gain: ~1% (frame pointer overhead) Time Required: 15 minutes

# Option 1: Use Cargo.toml.optimized (recommended)
cp Cargo.toml Cargo.toml.backup

# Edit Cargo.toml and add these profiles (see Cargo.toml.optimized for full config):
# [profile.release-profile]   # For profiling (keeps frame pointers)
# [profile.release-production] # For production (no frame pointers)

# Build production binary
cargo build --release --profile release-production

# Verify binary size
ls -lh target/release-production/

Usage:

• Profiling: cargo build --release --profile release-profile (with frame pointers)
• Production: cargo build --release --profile release-production (no overhead)

---

📊 Validation Checklist

After Each Optimization

• Run full benchmark suite: cargo bench --bench performance_regression
• Check for regressions: Compare with baseline in target/criterion/
• Verify binary still works: ./target/release/api_gateway --help
• Test critical paths: Order submission, authentication, ML inference
• Check binary size: ls -lh target/release/

Expected Results

Optimization	Latency	Throughput	Jitter	Binary Size
PGO	5-15%	3-8%	2-5%	+10-20%
Static Linking	<1%	0%	5-10%	+5-10%
Native CPU	0-5%	1-3%	1-2%	0%
No Frame Ptrs	~1%	~0.5%	<1%	-5%

Total: 7-22% latency improvement (average: 14%)

---

🛠️ Configuration Files

Updated .cargo/config.toml

# PRODUCTION - Runpod deployment (x86-64-v3)
[target.x86_64-unknown-linux-gnu]
rustflags = [
    "-C", "target-cpu=x86-64-v3",
    "-C", "target-feature=+avx2,+fma,+bmi2",
    "-C", "opt-level=3",
    "-C", "codegen-units=1",
]

# LOCAL - Maximum performance (native CPU)
[target.x86_64-unknown-linux-gnu.local]
rustflags = [
    "-C", "target-cpu=native",  # USE ALL CPU FEATURES
    "-C", "opt-level=3",
    "-C", "codegen-units=1",
]

# STATIC - Zero dynamic dependencies (musl)
[target.x86_64-unknown-linux-musl]
rustflags = [
    "-C", "target-cpu=native",
    "-C", "link-arg=-static",
    "-C", "opt-level=3",
]

Updated Cargo.toml

[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
panic = "abort"
strip = false  # Keep symbols for BOLT
overflow-checks = false

[profile.release-pgo]
inherits = "release"
# Used with cargo-pgo

[profile.release-production]
inherits = "release"
strip = true  # Remove symbols for production

---

🧪 Benchmarking Commands

Before Optimization

# Establish baseline
cargo build --release
cargo bench --bench performance_regression -- --save-baseline before-opt

# Key benchmarks
cargo bench --bench trading_latency -- --save-baseline before-opt
cargo bench --bench database_performance -- --save-baseline before-opt
cargo bench --bench full_trading_cycle -- --save-baseline before-opt

After Optimization

# Build optimized binary (PGO + Native)
RUSTFLAGS="-C target-cpu=native" cargo pgo optimize --release

# Compare with baseline
cargo bench --bench performance_regression -- --baseline before-opt
cargo bench --bench trading_latency -- --baseline before-opt
cargo bench --bench database_performance -- --baseline before-opt
cargo bench --bench full_trading_cycle -- --baseline before-opt

# Generate report
cargo bench -- --baseline before-opt --save-baseline after-opt

Expected Output

Order Matching          time:   [0.850 μs 0.900 μs 0.950 μs]
                        change: [-15.0% -10.0% -5.0%] (p = 0.00 < 0.05)
                        Performance has improved.

Authentication          time:   [3.50 μs 3.70 μs 3.90 μs]
                        change: [-20.0% -15.0% -10.0%] (p = 0.00 < 0.05)
                        Performance has improved.

---

🎯 Success Metrics

Phase 1 Targets (This Week)

• PGO pipeline operational
• Static binaries build successfully
• Native CPU builds tested locally
• Separate profiles configured
• Overall: 7-22% latency improvement validated
• Jitter: 5-15% reduction in P99 latency
• No regressions in functionality

Key Benchmarks

Metric	Baseline	Target (Phase 1)	Status
Order Matching	1-6μs	0.85-5.1μs	🎯 15%
Authentication	4.4μs	3.5μs	🎯 20%
Order Submission	15.96ms	12.8ms	🎯 20%
API Gateway	21-488μs	17-390μs	🎯 20%
DBN Loading	0.70ms	0.56ms	🎯 20%

---

🚧 Troubleshooting

PGO Build Fails

# Error: cargo-pgo not found
cargo install cargo-pgo

# Error: Profile data not found
# Solution: Run representative workloads to generate profiles
cargo bench --bench performance_regression

Static Build Fails

# Error: musl target not found
rustup target add x86_64-unknown-linux-musl

# Error: OpenSSL not found
# Solution: Use rustls instead of openssl
# In Cargo.toml: reqwest = { features = ["rustls-tls"] }

Native CPU Build Incompatible with Runpod

# Solution: Use separate profiles
# Local: cargo build --release --config target.x86_64-unknown-linux-gnu.local
# Runpod: cargo build --release (uses x86-64-v3 baseline)

---

📚 Next Steps (Week 2+)

Priority 5: Allocator Optimization (1-3% gain)

# Test mimalloc
cargo add mimalloc
# Add to main.rs: #[global_allocator] static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
cargo bench -- --save-baseline mimalloc

# Test jemalloc
cargo add jemallocator
# Add to main.rs: #[global_allocator] static GLOBAL: jemallocator::Jemalloc = jemallocator::Jemalloc;
cargo bench -- --save-baseline jemalloc

# Compare
./scripts/compare_allocators.py

Priority 6: BOLT Post-Link Optimization (2-8% gain)

# Research phase (Week 3)
# - Study LLVM BOLT documentation
# - Set up BOLT toolchain
# - Test on simple examples

# Integration phase (Week 4)
# - Integrate BOLT into build pipeline
# - Generate runtime profiles with perf
# - Apply BOLT optimizations
# - Validate improvements

---

📞 Support & Resources

Documentation:

• Full Analysis: AGENT_15_RUST_COMPILER_OPTIMIZATION_ANALYSIS.md
• Benchmark Estimates: OPTIMIZATION_BENCHMARK_ESTIMATES.md
• Optimized Configs: .cargo/config.toml.optimized, Cargo.toml.optimized

Scripts:

• PGO Implementation: scripts/implement_pgo.sh

External Resources:

• Rust Performance Book: https://nnethercote.github.io/perf-book/
• cargo-pgo: https://github.com/Kobzol/cargo-pgo
• LLVM BOLT: https://github.com/llvm/llvm-project/tree/main/bolt

---

✅ Completion Checklist

Week 1 (Quick Wins):

• Day 1-2: PGO implementation (5-15% gain)
• Day 3: Static linking (jitter reduction)
• Day 4: Native CPU targeting (0-5% gain)
• Day 5: Separate profiles (~1% gain)

Week 2 (Allocator):

• Benchmark allocators (mimalloc vs jemalloc)
• Integrate chosen allocator (1-3% gain)

Weeks 3-4 (BOLT):

• Research LLVM BOLT
• Integrate BOLT optimization (2-8% gain)

Total Expected Improvement: 12-28% (average: 22%) Final Performance: 1,032-1,180x faster than targets (average: 1,125x)

---

🎉 Ready to start? Run: ./scripts/implement_pgo.sh