# Rust Compiler Optimization - Quick Start Guide

**Generated**: 2025-10-25
**Current Status**: Baseline (922x faster than targets)
**Target**: 1,032-1,180x faster than targets (12-28% improvement)

---

## 🚀 TL;DR - Execute This Week

```bash
# 1. Install cargo-pgo (5 minutes)
cargo install cargo-pgo

# 2. Run PGO pipeline (20-30 minutes)
./scripts/implement_pgo.sh

# 3. Add musl target for static linking (5 minutes)
rustup target add x86_64-unknown-linux-musl
cargo build --release --target x86_64-unknown-linux-musl

# 4. Test native CPU builds locally (5 minutes)
cargo build --release --config target.x86_64-unknown-linux-gnu.local
cargo bench --bench performance_regression

# Expected result: 7-22% performance improvement (average: 14%)
```

---

## 📋 Step-by-Step Implementation

### Step 1: Profile-Guided Optimization (PGO)

**Expected Gain**: 5-15% (average: 10%)
**Time Required**: 1-2 hours

```bash
# Install cargo-pgo
cargo install cargo-pgo

# Option A: Automated (recommended)
./scripts/implement_pgo.sh

# Option B: Manual
# 1. Build instrumented binary
RUSTFLAGS="-C target-cpu=native" cargo pgo build --release

# 2. Run representative workloads to generate profile data
# Workload 1: Backtesting
cargo run --release -p backtesting_service -- --symbol ES.FUT --duration 180d

# Workload 2: ML training
cargo run --release -p ml --example train_tft_parquet -- \
  --parquet-file test_data/ES_FUT_180d.parquet --epochs 10

# Workload 3: Benchmarks
cargo bench --bench performance_regression

# 3. Build optimized binary
RUSTFLAGS="-C target-cpu=native" cargo pgo optimize --release

# 4. Validate improvements
cargo bench --bench performance_regression -- --baseline pgo-before
```

**Validation**: Check `target/criterion/` for improvement reports

---

### Step 2: Static Linking (musl)

**Expected Gain**: <1% latency, 5-10% jitter reduction
**Time Required**: 30 minutes

```bash
# Add musl target
rustup target add x86_64-unknown-linux-musl

# Build static binary
cargo build --release --target x86_64-unknown-linux-musl

# Verify static linking (should show "not a dynamic executable")
ldd target/x86_64-unknown-linux-musl/release/api_gateway

# Test binary
./target/x86_64-unknown-linux-musl/release/api_gateway --help
```

**Benefits**:
- Zero dynamic dependencies
- Reduced jitter from PLT/GOT indirection
- Faster startup time

---

### Step 3: Native CPU Targeting (Local Builds)

**Expected Gain**: 0-5% (average: 2.5%)
**Time Required**: 15 minutes

```bash
# Option 1: Use .cargo/config.toml.optimized (recommended)
cp .cargo/config.toml .cargo/config.toml.backup
cp .cargo/config.toml.optimized .cargo/config.toml

# Option 2: Manual RUSTFLAGS
export RUSTFLAGS="-C target-cpu=native"
cargo build --release

# Benchmark improvements
cargo bench --bench performance_regression -- --baseline before-native --save-baseline after-native
```

**Local CPU Features** (i7-11800H):
- AVX-512F, AVX-512DQ, AVX-512BW, AVX-512VL
- ADX (Multi-Precision Add-Carry)
- SHA-NI (Hardware SHA hashing)
- VAES, VPCLMULQDQ (Advanced crypto)

**Warning**: Do NOT use `target-cpu=native` for Runpod deployment (incompatible with V100 CPU)

---

### Step 4: Separate Profiling/Production Builds

**Expected Gain**: ~1% (frame pointer overhead)
**Time Required**: 15 minutes

```bash
# Option 1: Use Cargo.toml.optimized (recommended)
cp Cargo.toml Cargo.toml.backup

# Edit Cargo.toml and add these profiles (see Cargo.toml.optimized for full config):
# [profile.release-profile]   # For profiling (keeps frame pointers)
# [profile.release-production] # For production (no frame pointers)

# Build production binary
cargo build --release --profile release-production

# Verify binary size
ls -lh target/release-production/
```

**Usage**:
- **Profiling**: `cargo build --release --profile release-profile` (with frame pointers)
- **Production**: `cargo build --release --profile release-production` (no overhead)

---

## 📊 Validation Checklist

### After Each Optimization

- [ ] Run full benchmark suite: `cargo bench --bench performance_regression`
- [ ] Check for regressions: Compare with baseline in `target/criterion/`
- [ ] Verify binary still works: `./target/release/api_gateway --help`
- [ ] Test critical paths: Order submission, authentication, ML inference
- [ ] Check binary size: `ls -lh target/release/`

### Expected Results

| Optimization | Latency | Throughput | Jitter | Binary Size |
|--------------|---------|------------|--------|-------------|
| PGO | 5-15% | 3-8% | 2-5% | +10-20% |
| Static Linking | <1% | 0% | 5-10% | +5-10% |
| Native CPU | 0-5% | 1-3% | 1-2% | 0% |
| No Frame Ptrs | ~1% | ~0.5% | <1% | -5% |

**Total**: 7-22% latency improvement (average: 14%)

---

## 🛠️ Configuration Files

### Updated .cargo/config.toml

```toml
# PRODUCTION - Runpod deployment (x86-64-v3)
[target.x86_64-unknown-linux-gnu]
rustflags = [
    "-C", "target-cpu=x86-64-v3",
    "-C", "target-feature=+avx2,+fma,+bmi2",
    "-C", "opt-level=3",
    "-C", "codegen-units=1",
]

# LOCAL - Maximum performance (native CPU)
[target.x86_64-unknown-linux-gnu.local]
rustflags = [
    "-C", "target-cpu=native",  # USE ALL CPU FEATURES
    "-C", "opt-level=3",
    "-C", "codegen-units=1",
]

# STATIC - Zero dynamic dependencies (musl)
[target.x86_64-unknown-linux-musl]
rustflags = [
    "-C", "target-cpu=native",
    "-C", "link-arg=-static",
    "-C", "opt-level=3",
]
```

### Updated Cargo.toml

```toml
[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
panic = "abort"
strip = false  # Keep symbols for BOLT
overflow-checks = false

[profile.release-pgo]
inherits = "release"
# Used with cargo-pgo

[profile.release-production]
inherits = "release"
strip = true  # Remove symbols for production
```

---

## 🧪 Benchmarking Commands

### Before Optimization

```bash
# Establish baseline
cargo build --release
cargo bench --bench performance_regression -- --save-baseline before-opt

# Key benchmarks
cargo bench --bench trading_latency -- --save-baseline before-opt
cargo bench --bench database_performance -- --save-baseline before-opt
cargo bench --bench full_trading_cycle -- --save-baseline before-opt
```

### After Optimization

```bash
# Build optimized binary (PGO + Native)
RUSTFLAGS="-C target-cpu=native" cargo pgo optimize --release

# Compare with baseline
cargo bench --bench performance_regression -- --baseline before-opt
cargo bench --bench trading_latency -- --baseline before-opt
cargo bench --bench database_performance -- --baseline before-opt
cargo bench --bench full_trading_cycle -- --baseline before-opt

# Generate report
cargo bench -- --baseline before-opt --save-baseline after-opt
```

### Expected Output

```
Order Matching          time:   [0.850 μs 0.900 μs 0.950 μs]
                        change: [-15.0% -10.0% -5.0%] (p = 0.00 < 0.05)
                        Performance has improved.

Authentication          time:   [3.50 μs 3.70 μs 3.90 μs]
                        change: [-20.0% -15.0% -10.0%] (p = 0.00 < 0.05)
                        Performance has improved.
```

---

## 🎯 Success Metrics

### Phase 1 Targets (This Week)

- [ ] PGO pipeline operational
- [ ] Static binaries build successfully
- [ ] Native CPU builds tested locally
- [ ] Separate profiles configured
- [ ] **Overall**: 7-22% latency improvement validated
- [ ] **Jitter**: 5-15% reduction in P99 latency
- [ ] **No regressions** in functionality

### Key Benchmarks

| Metric | Baseline | Target (Phase 1) | Status |
|--------|----------|------------------|--------|
| Order Matching | 1-6μs | 0.85-5.1μs | 🎯 15% |
| Authentication | 4.4μs | 3.5μs | 🎯 20% |
| Order Submission | 15.96ms | 12.8ms | 🎯 20% |
| API Gateway | 21-488μs | 17-390μs | 🎯 20% |
| DBN Loading | 0.70ms | 0.56ms | 🎯 20% |

---

## 🚧 Troubleshooting

### PGO Build Fails

```bash
# Error: cargo-pgo not found
cargo install cargo-pgo

# Error: Profile data not found
# Solution: Run representative workloads to generate profiles
cargo bench --bench performance_regression
```

### Static Build Fails

```bash
# Error: musl target not found
rustup target add x86_64-unknown-linux-musl

# Error: OpenSSL not found
# Solution: Use rustls instead of openssl
# In Cargo.toml: reqwest = { features = ["rustls-tls"] }
```

### Native CPU Build Incompatible with Runpod

```bash
# Solution: Use separate profiles
# Local: cargo build --release --config target.x86_64-unknown-linux-gnu.local
# Runpod: cargo build --release (uses x86-64-v3 baseline)
```

---

## 📚 Next Steps (Week 2+)

### Priority 5: Allocator Optimization (1-3% gain)

```bash
# Test mimalloc
cargo add mimalloc
# Add to main.rs: #[global_allocator] static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
cargo bench -- --save-baseline mimalloc

# Test jemalloc
cargo add jemallocator
# Add to main.rs: #[global_allocator] static GLOBAL: jemallocator::Jemalloc = jemallocator::Jemalloc;
cargo bench -- --save-baseline jemalloc

# Compare
./scripts/compare_allocators.py
```

### Priority 6: BOLT Post-Link Optimization (2-8% gain)

```bash
# Research phase (Week 3)
# - Study LLVM BOLT documentation
# - Set up BOLT toolchain
# - Test on simple examples

# Integration phase (Week 4)
# - Integrate BOLT into build pipeline
# - Generate runtime profiles with perf
# - Apply BOLT optimizations
# - Validate improvements
```

---

## 📞 Support & Resources

**Documentation**:
- Full Analysis: `AGENT_15_RUST_COMPILER_OPTIMIZATION_ANALYSIS.md`
- Benchmark Estimates: `OPTIMIZATION_BENCHMARK_ESTIMATES.md`
- Optimized Configs: `.cargo/config.toml.optimized`, `Cargo.toml.optimized`

**Scripts**:
- PGO Implementation: `scripts/implement_pgo.sh`

**External Resources**:
- Rust Performance Book: https://nnethercote.github.io/perf-book/
- cargo-pgo: https://github.com/Kobzol/cargo-pgo
- LLVM BOLT: https://github.com/llvm/llvm-project/tree/main/bolt

---

## ✅ Completion Checklist

**Week 1 (Quick Wins)**:
- [ ] Day 1-2: PGO implementation (5-15% gain)
- [ ] Day 3: Static linking (jitter reduction)
- [ ] Day 4: Native CPU targeting (0-5% gain)
- [ ] Day 5: Separate profiles (~1% gain)

**Week 2 (Allocator)**:
- [ ] Benchmark allocators (mimalloc vs jemalloc)
- [ ] Integrate chosen allocator (1-3% gain)

**Weeks 3-4 (BOLT)**:
- [ ] Research LLVM BOLT
- [ ] Integrate BOLT optimization (2-8% gain)

**Total Expected Improvement**: 12-28% (average: 22%)
**Final Performance**: 1,032-1,180x faster than targets (average: 1,125x)

---

🎉 **Ready to start? Run**: `./scripts/implement_pgo.sh`