foxhunt/AGENT_D38_PROFILING_QUICK_REFERENCE.md

Agent D38: Profiling Quick Reference Guide

Quick Start: How to profile the 225-feature pipeline and identify bottlenecks

---

🚀 Quick Commands

# 1. Run basic profiling test (no flamegraph)
cargo test -p ml --test wave_d_profiling_test --release -- --ignored --nocapture

# 2. Generate CPU flamegraph (BEST FOR BOTTLENECK ANALYSIS)
cargo flamegraph --test wave_d_profiling_test -p ml --release -- --ignored --nocapture
# Opens flamegraph.svg in browser - look for wide bars (>5% CPU)

# 3. Cache profiling (verify <5% miss rate)
perf stat -e cache-references,cache-misses \
    cargo test -p ml --test wave_d_profiling_test --release -- --ignored --nocapture

---

📊 Reading Profiling Results

CPU Flamegraph (flamegraph.svg)

┌────────────────────────────────────────────────────┐
│ Feature225Profiler::extract_features (100%)        │  ← Total function
├────────────────────────┬───────────────────────────┤
│ Wave C: 60% CPU        │  Wave D: 40% CPU          │  ← Stages
│  ⚠️  HOTSPOT          │                           │
├────────────┬───────────┴─────┬─────────────────────┤
│ price: 25% │ volume: 20%     │ CUSUM: 15%          │  ← Sub-functions
└────────────┴─────────────────┴─────────────────────┘

What to Look For:

• Wide bars >20% CPU = Hotspots (optimize these first)
• Deep stacks = Inlining opportunities
• Unexpected libraries = Dependency overhead

Cache Profiling Output

Performance counter stats:
  12,345,678 cache-references
     617,284 cache-misses           # 5.00% miss rate

Targets:

• ✅ <5% miss rate = Good cache locality
• ❌ >5% miss rate = Consider ring buffers, SoA layout

---

🎯 Performance Targets

Metric	Target	Status
Total P99 latency	<100μs	⏳ TBD
Wave C P99	<40μs	⏳ TBD
Wave D P99	<25μs	⏳ TBD
Cache miss rate	<5%	⏳ TBD
No single stage CPU%	<50%	⏳ TBD

---

🔧 Common Optimizations

1. SIMD Vectorization (30-50% speedup)

// Before (scalar)
for i in 0..data.len() {
    sum += data[i];
}

// After (SIMD AVX2)
use std::simd::f64x4;
let chunks = data.chunks_exact(4);
let simd_sum = chunks.map(|c| f64x4::from_slice(c)).sum();

2. Pre-Allocated Buffers (10-15% speedup)

// Before
let mut features = Vec::new();  // Reallocations!

// After
let mut features = Vec::with_capacity(225);  // Zero reallocations

3. Ring Buffer (5-10% speedup via cache hits)

// Before (VecDeque = non-contiguous)
let mut window = VecDeque::with_capacity(50);

// After (contiguous ring buffer)
let mut window = vec![0.0; 50];
let mut head = 0;

---

📁 Key Files

• Profiling Test: /home/jgrusewski/Work/foxhunt/ml/tests/wave_d_profiling_test.rs
• Full Report: /home/jgrusewski/Work/foxhunt/AGENT_D38_PROFILING_ANALYSIS_REPORT.md
• Flamegraph Output: ./flamegraph.svg (generated by cargo flamegraph)

---

🐛 Troubleshooting

Profiling test fails with "No DBN test data found"

# Check if DBN files exist
ls -lh /home/jgrusewski/Work/foxhunt/test_data/real/databento/

# If missing, download from Databento or use synthetic data

Flamegraph command not found

cargo install flamegraph

Perf permission denied

# Temporary (current session)
sudo sysctl -w kernel.perf_event_paranoid=-1

# Permanent
echo "kernel.perf_event_paranoid=-1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

---

📈 Next Steps After Profiling

1. Identify hotspot (flamegraph wide bar >20% CPU)
2. Apply optimization (SIMD, pre-alloc, or cache-friendly)
3. Re-profile to validate improvement
4. Repeat until P99 <100μs

---

Created: 2025-10-18 | Agent: D38