foxhunt/AGENT_D38_PROFILING_QUICK_REFERENCE.md

# Agent D38: Profiling Quick Reference Guide

**Quick Start**: How to profile the 225-feature pipeline and identify bottlenecks

---

## 🚀 Quick Commands

```bash
# 1. Run basic profiling test (no flamegraph)
cargo test -p ml --test wave_d_profiling_test --release -- --ignored --nocapture

# 2. Generate CPU flamegraph (BEST FOR BOTTLENECK ANALYSIS)
cargo flamegraph --test wave_d_profiling_test -p ml --release -- --ignored --nocapture
# Opens flamegraph.svg in browser - look for wide bars (>5% CPU)

# 3. Cache profiling (verify <5% miss rate)
perf stat -e cache-references,cache-misses \
    cargo test -p ml --test wave_d_profiling_test --release -- --ignored --nocapture
```

---

## 📊 Reading Profiling Results

### CPU Flamegraph (flamegraph.svg)

```
┌────────────────────────────────────────────────────┐
│ Feature225Profiler::extract_features (100%)        │  ← Total function
├────────────────────────┬───────────────────────────┤
│ Wave C: 60% CPU        │  Wave D: 40% CPU          │  ← Stages
│  ⚠️  HOTSPOT          │                           │
├────────────┬───────────┴─────┬─────────────────────┤
│ price: 25% │ volume: 20%     │ CUSUM: 15%          │  ← Sub-functions
└────────────┴─────────────────┴─────────────────────┘
```

**What to Look For**:
- **Wide bars >20% CPU** = Hotspots (optimize these first)
- **Deep stacks** = Inlining opportunities
- **Unexpected libraries** = Dependency overhead

### Cache Profiling Output

```
Performance counter stats:
  12,345,678 cache-references
     617,284 cache-misses           # 5.00% miss rate
```

**Targets**:
- ✅ **<5% miss rate** = Good cache locality
- ❌ **>5% miss rate** = Consider ring buffers, SoA layout

---

## 🎯 Performance Targets

| **Metric**              | **Target**  | **Status** |
|-------------------------|-------------|------------|
| Total P99 latency       | <100μs      | ⏳ TBD     |
| Wave C P99              | <40μs       | ⏳ TBD     |
| Wave D P99              | <25μs       | ⏳ TBD     |
| Cache miss rate         | <5%         | ⏳ TBD     |
| No single stage CPU%    | <50%        | ⏳ TBD     |

---

## 🔧 Common Optimizations

### 1. SIMD Vectorization (30-50% speedup)

```rust
// Before (scalar)
for i in 0..data.len() {
    sum += data[i];
}

// After (SIMD AVX2)
use std::simd::f64x4;
let chunks = data.chunks_exact(4);
let simd_sum = chunks.map(|c| f64x4::from_slice(c)).sum();
```

### 2. Pre-Allocated Buffers (10-15% speedup)

```rust
// Before
let mut features = Vec::new();  // Reallocations!

// After
let mut features = Vec::with_capacity(225);  // Zero reallocations
```

### 3. Ring Buffer (5-10% speedup via cache hits)

```rust
// Before (VecDeque = non-contiguous)
let mut window = VecDeque::with_capacity(50);

// After (contiguous ring buffer)
let mut window = vec![0.0; 50];
let mut head = 0;
```

---

## 📁 Key Files

- **Profiling Test**: `/home/jgrusewski/Work/foxhunt/ml/tests/wave_d_profiling_test.rs`
- **Full Report**: `/home/jgrusewski/Work/foxhunt/AGENT_D38_PROFILING_ANALYSIS_REPORT.md`
- **Flamegraph Output**: `./flamegraph.svg` (generated by `cargo flamegraph`)

---

## 🐛 Troubleshooting

### Profiling test fails with "No DBN test data found"

```bash
# Check if DBN files exist
ls -lh /home/jgrusewski/Work/foxhunt/test_data/real/databento/

# If missing, download from Databento or use synthetic data
```

### Flamegraph command not found

```bash
cargo install flamegraph
```

### Perf permission denied

```bash
# Temporary (current session)
sudo sysctl -w kernel.perf_event_paranoid=-1

# Permanent
echo "kernel.perf_event_paranoid=-1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
```

---

## 📈 Next Steps After Profiling

1. **Identify hotspot** (flamegraph wide bar >20% CPU)
2. **Apply optimization** (SIMD, pre-alloc, or cache-friendly)
3. **Re-profile** to validate improvement
4. **Repeat** until P99 <100μs

---

**Created**: 2025-10-18 | **Agent**: D38