foxhunt/docs/PERFORMANCE_TUNING.md

# Foxhunt HFT Trading System - Performance Tuning Guide

## Table of Contents

1. [Performance Targets](#performance-targets)
2. [System-Level Optimizations](#system-level-optimizations)
3. [CPU Optimization](#cpu-optimization)
4. [Memory Optimization](#memory-optimization)
5. [Network Optimization](#network-optimization)
6. [Storage Optimization](#storage-optimization)
7. [Application-Level Tuning](#application-level-tuning)
8. [Database Performance](#database-performance)
9. [Monitoring & Profiling](#monitoring--profiling)
10. [Benchmarking & Testing](#benchmarking--testing)

## Performance Targets

### Latency Requirements
- **Order Submission**: <50μs (50 microseconds)
- **Risk Checks**: <10μs (10 microseconds)
- **Market Data Processing**: <5μs (5 microseconds)
- **Timing Operations**: <14ns (14 nanoseconds)
- **End-to-End Trading**: <100μs (100 microseconds)

### Throughput Requirements
- **Orders per Second**: >10,000
- **Market Data Messages**: >100,000/sec
- **Risk Calculations**: >1,000/sec
- **Database Transactions**: >5,000/sec

### Resource Utilization Targets
- **CPU Usage**: <70% on trading cores
- **Memory Usage**: <80% of available RAM
- **Network Utilization**: <60% of bandwidth
- **Disk I/O**: <50% of IOPS capacity

## System-Level Optimizations

### Operating System Configuration

#### Kernel Parameters
```bash
# /etc/sysctl.conf - System-wide performance tuning

# Network performance
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.rmem_default = 8388608
net.core.wmem_default = 8388608
net.core.netdev_max_backlog = 5000
net.core.netdev_budget = 600
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_low_latency = 1

# Memory management
vm.swappiness = 1
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.vfs_cache_pressure = 50

# File system
fs.file-max = 2097152
fs.nr_open = 1048576

# Apply settings
sudo sysctl -p
```

#### Real-Time Kernel Configuration
```bash
# Install real-time kernel
sudo apt install linux-image-rt-generic linux-headers-rt-generic

# Boot parameters for HFT optimization
# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=2,3,4,5 rcu_nocbs=2,3,4,5 nohz_full=2,3,4,5 intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll"

sudo update-grub
```

### CPU Governor and Frequency Scaling
```bash
# Set performance governor for consistent performance
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Disable CPU idle states
sudo cpupower idle-set -D 0

# Set minimum CPU frequency to maximum
cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
```

### Hardware Configuration

#### BIOS/UEFI Settings
```
Performance Configuration:
- CPU Power Management: Disabled
- C-States: Disabled
- Turbo Boost: Enabled
- Hyper-Threading: Enabled (if beneficial for workload)
- Intel SpeedStep: Disabled
- EIST: Disabled

Memory Configuration:
- Memory Operating Mode: Performance
- NUMA: Enabled
- Memory RAS: Disabled (for performance)

Power Management:
- Power Profile: Maximum Performance
- CPU Power Management: Disabled
```

## CPU Optimization

### CPU Affinity Management

#### Core Allocation Strategy
```bash
# Core allocation for HFT workload:
# Core 0-1: OS and system processes
# Core 2-3: Trading engine (isolated)
# Core 4-5: Risk management
# Core 6-7: Market data processing
# Core 8+: ML inference and background tasks

# Isolate trading cores
echo 2-3 | sudo tee /sys/devices/system/cpu/isolated

# Set CPU affinity for critical processes
./scripts/set-cpu-affinity.sh
```

#### CPU Affinity Script
```bash
#!/bin/bash
# /usr/local/bin/set-cpu-affinity.sh

# Trading engine on dedicated cores
taskset -c 2,3 systemctl restart foxhunt-core

# Risk management
taskset -c 4,5 systemctl restart foxhunt-risk

# Market data processing
taskset -c 6,7 systemctl restart foxhunt-data

# ML inference
taskset -c 8-11 systemctl restart foxhunt-ml

# Set real-time priority for trading processes
sudo chrt -f -p 99 $(pgrep foxhunt-core)
sudo chrt -f -p 90 $(pgrep foxhunt-risk)
sudo chrt -f -p 80 $(pgrep foxhunt-data)
```

### SIMD Optimization

#### AVX2/AVX-512 Detection and Usage
```bash
# Check CPU features
lscpu | grep -E "avx|sse"
cat /proc/cpuinfo | grep flags

# Build with CPU-specific optimizations
export RUSTFLAGS="-C target-cpu=native -C target-feature=+avx2,+fma"
cargo build --release

# For AVX-512 capable systems
export RUSTFLAGS="-C target-cpu=native -C target-feature=+avx512f,+avx512dq"
```

#### SIMD Performance Validation
```rust
// Benchmark SIMD operations
#[cfg(test)]
mod simd_benchmarks {
    use criterion::{black_box, criterion_group, criterion_main, Criterion};
    use crate::simd::SimdPriceOps;
    
    fn benchmark_price_calculations(c: &mut Criterion) {
        let simd_ops = SimdPriceOps::new().unwrap();
        let prices = vec![100.0f32; 1000];
        
        c.bench_function("simd_price_adjustment", |b| {
            b.iter(|| simd_ops.apply_adjustment(black_box(&prices), black_box(0.001)))
        });
    }
    
    criterion_group!(benches, benchmark_price_calculations);
    criterion_main!(benches);
}
```

### Context Switch Minimization

#### Thread Pool Configuration
```rust
// Optimize thread pool for minimal context switching
use rayon::ThreadPoolBuilder;

let thread_pool = ThreadPoolBuilder::new()
    .num_threads(4)  // Match isolated cores
    .thread_name(|index| format!("hft-worker-{}", index))
    .build()
    .unwrap();

// Pin threads to specific cores
thread_pool.install(|| {
    // CPU-intensive work here
});
```

## Memory Optimization

### Memory Layout and Allocation

#### Large Pages Configuration
```bash
# Configure transparent huge pages
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Configure explicit huge pages
echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Mount hugetlbfs
sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs none /mnt/huge -o uid=foxhunt,gid=foxhunt,mode=755

# Add to /etc/fstab for persistence
echo "none /mnt/huge hugetlbfs uid=foxhunt,gid=foxhunt,mode=755 0 0" | sudo tee -a /etc/fstab
```

#### NUMA Optimization
```bash
# Check NUMA topology
numactl --hardware

# Bind process to specific NUMA node
numactl --cpunodebind=0 --membind=0 ./target/release/foxhunt-core

# Check NUMA policy
numactl --show
```

### Memory Pool Management

#### Pre-allocated Memory Pools
```rust
use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicUsize, Ordering};

// Custom allocator for performance monitoring
struct PerformanceAllocator;

unsafe impl GlobalAlloc for PerformanceAllocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        let ptr = System.alloc(layout);
        if !ptr.is_null() {
            ALLOCATED_BYTES.fetch_add(layout.size(), Ordering::Relaxed);
        }
        ptr
    }
    
    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        System.dealloc(ptr, layout);
        ALLOCATED_BYTES.fetch_sub(layout.size(), Ordering::Relaxed);
    }
}

static ALLOCATED_BYTES: AtomicUsize = AtomicUsize::new(0);

#[global_allocator]
static ALLOCATOR: PerformanceAllocator = PerformanceAllocator;
```

#### Memory-Mapped Files
```rust
use memmap2::MmapOptions;
use std::fs::OpenOptions;

// Memory-map large datasets for performance
fn create_memory_mapped_data() -> Result<memmap2::Mmap, Box<dyn std::error::Error>> {
    let file = OpenOptions::new()
        .read(true)
        .write(true)
        .create(true)
        .open("/mnt/huge/market_data.bin")?;
    
    file.set_len(1024 * 1024 * 1024)?; // 1GB
    
    let mmap = unsafe {
        MmapOptions::new()
            .map(&file)?
    };
    
    Ok(mmap)
}
```

### Cache Optimization

#### Cache-Friendly Data Structures
```rust
#[repr(C, align(64))]  // Cache line alignment
pub struct CacheAlignedPrice {
    pub value: f64,
    pub timestamp: u64,
    _padding: [u8; 48],  // Pad to cache line boundary
}

// Cache-friendly order book structure
#[repr(C)]
pub struct OrderBookLevel {
    pub price: f64,
    pub quantity: f64,
    pub orders: u32,
    pub timestamp: u64,
}
```

## Network Optimization

### Network Interface Configuration

#### High-Performance Network Settings
```bash
# Optimize network interface (replace eth0 with actual interface)
INTERFACE="eth0"

# Set ring buffer sizes
sudo ethtool -G $INTERFACE rx 4096 tx 4096

# Enable hardware offloading
sudo ethtool -K $INTERFACE gso on
sudo ethtool -K $INTERFACE tso on
sudo ethtool -K $INTERFACE lro on
sudo ethtool -K $INTERFACE gro on

# Set interrupt coalescing
sudo ethtool -C $INTERFACE rx-usecs 1 tx-usecs 1

# Check current settings
sudo ethtool -g $INTERFACE
sudo ethtool -k $INTERFACE
sudo ethtool -c $INTERFACE
```

#### Network Queue Management
```bash
# Configure multiple queues for multi-core processing
sudo ethtool -L $INTERFACE combined 4

# Set CPU affinity for network interrupts
echo 1 | sudo tee /proc/irq/24/smp_affinity  # NIC queue 0 -> CPU 1
echo 2 | sudo tee /proc/irq/25/smp_affinity  # NIC queue 1 -> CPU 2
echo 4 | sudo tee /proc/irq/26/smp_affinity  # NIC queue 2 -> CPU 3
echo 8 | sudo tee /proc/irq/27/smp_affinity  # NIC queue 3 -> CPU 4
```

### TCP/UDP Optimization

#### Low-Latency Socket Configuration
```rust
use std::net::{TcpStream, SocketAddr};
use socket2::{Socket, Domain, Type, Protocol};

fn create_optimized_socket(addr: SocketAddr) -> Result<TcpStream, Box<dyn std::error::Error>> {
    let socket = Socket::new(Domain::IPV4, Type::STREAM, Some(Protocol::TCP))?;
    
    // Enable TCP_NODELAY for immediate sends
    socket.set_nodelay(true)?;
    
    // Set socket buffer sizes
    socket.set_recv_buffer_size(1024 * 1024)?;  // 1MB
    socket.set_send_buffer_size(1024 * 1024)?;  // 1MB
    
    // Enable address reuse
    socket.set_reuse_address(true)?;
    
    // Set keep-alive
    socket.set_keepalive(true)?;
    
    socket.connect(&addr.into())?;
    
    Ok(socket.into())
}
```

#### Kernel Bypass Networking (DPDK)
```bash
# Install DPDK for kernel bypass
wget https://fast.dpdk.org/rel/dpdk-23.11.tar.xz
tar xf dpdk-23.11.tar.xz
cd dpdk-23.11

# Build DPDK
meson setup build
ninja -C build
sudo ninja -C build install

# Bind network interface to DPDK
sudo modprobe uio_pci_generic
sudo dpdk-devbind.py --bind=uio_pci_generic 0000:02:00.0
```

## Storage Optimization

### File System Optimization

#### File System Selection and Mounting
```bash
# Format with optimal settings for performance
sudo mkfs.ext4 -F -E stride=32,stripe-width=128 /dev/nvme0n1

# Mount with performance optimizations
sudo mount -t ext4 -o noatime,nodiratime,data=writeback,barrier=0,nobh /dev/nvme0n1 /var/lib/foxhunt

# Add to /etc/fstab
echo "/dev/nvme0n1 /var/lib/foxhunt ext4 noatime,nodiratime,data=writeback,barrier=0,nobh 0 0" | sudo tee -a /etc/fstab
```

#### I/O Scheduler Configuration
```bash
# Set appropriate I/O scheduler for SSDs
echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler

# For traditional HDDs, use CFQ
echo cfq | sudo tee /sys/block/sda/queue/scheduler

# Optimize queue depth
echo 32 | sudo tee /sys/block/nvme0n1/queue/nr_requests
```

### Database Storage Optimization

#### PostgreSQL Storage Configuration
```bash
# PostgreSQL configuration for performance
# /etc/postgresql/14/main/postgresql.conf

# Memory settings
shared_buffers = 32GB                    # 25% of system RAM
effective_cache_size = 96GB              # 75% of system RAM
work_mem = 256MB                         # For complex queries
maintenance_work_mem = 2GB               # For maintenance operations

# Checkpoint settings
checkpoint_completion_target = 0.9
wal_buffers = 16MB
max_wal_size = 4GB
min_wal_size = 1GB

# Connection settings
max_connections = 200
shared_preload_libraries = 'pg_stat_statements'

# Logging (disable in production)
log_statement = 'none'
log_min_duration_statement = -1
```

#### InfluxDB Storage Optimization
```toml
# /etc/influxdb/influxdb.conf

[data]
  dir = "/var/lib/influxdb/data"
  engine = "tsm1"
  max-series-per-database = 10000000
  max-values-per-tag = 1000000

[wal]
  dir = "/var/lib/influxdb/wal"
  fsync-delay = "0s"
  
[cache]
  max-memory-size = "2g"
  snapshot-memory-size = "256m"
  
[compaction]
  throughput-bytes-per-second = "100m"
  
[retention]
  enabled = true
  check-interval = "30m"
```

## Application-Level Tuning

### Rust Compiler Optimizations

#### Build Configuration
```toml
# Cargo.toml - Profile optimizations

[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
panic = "abort"
strip = true

[profile.release-with-debug]
inherits = "release"
debug = true
strip = false

# Target-specific optimizations
[target.'cfg(target_arch = "x86_64")']
rustflags = [
    "-C", "target-cpu=native",
    "-C", "target-feature=+avx2,+fma,+sse4.2",
    "-C", "link-arg=-fuse-ld=lld",
]
```

#### Compile-Time Features
```bash
# Build with maximum optimizations
export RUSTFLAGS="-C target-cpu=native -C target-feature=+avx2,+fma -C link-arg=-fuse-ld=lld"
cargo build --release --features=simd,avx2,lto

# Profile-guided optimization
cargo pgo build --release
./target/release/foxhunt-benchmark  # Generate profile data
cargo pgo optimize --release
```

### Lock-Free Programming

#### Atomic Operations Optimization
```rust
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;

pub struct HighFrequencyCounter {
    counter: AtomicU64,
}

impl HighFrequencyCounter {
    pub fn increment(&self) -> u64 {
        // Use relaxed ordering for maximum performance
        self.counter.fetch_add(1, Ordering::Relaxed)
    }
    
    pub fn get(&self) -> u64 {
        // Acquire ordering for reading
        self.counter.load(Ordering::Acquire)
    }
}

// Lock-free queue implementation
use crossbeam::queue::ArrayQueue;

pub struct LockFreeOrderQueue {
    queue: Arc<ArrayQueue<Order>>,
}

impl LockFreeOrderQueue {
    pub fn new(capacity: usize) -> Self {
        Self {
            queue: Arc::new(ArrayQueue::new(capacity)),
        }
    }
    
    pub fn push(&self, order: Order) -> Result<(), Order> {
        self.queue.push(order)
    }
    
    pub fn pop(&self) -> Option<Order> {
        self.queue.pop()
    }
}
```

### Memory Access Patterns

#### Cache-Aware Programming
```rust
// Optimize for cache locality
#[derive(Clone, Copy)]
#[repr(C, align(64))]  // Cache line alignment
pub struct PriceLevel {
    pub price: f64,
    pub quantity: f64,
    pub timestamp: u64,
    _padding: [u8; 40],  // Pad to cache line size
}

// Array of Structures vs Structure of Arrays
pub struct AoSOrderBook {
    levels: Vec<PriceLevel>,  // Better for random access
}

pub struct SoAOrderBook {
    prices: Vec<f64>,      // Better for bulk operations
    quantities: Vec<f64>,
    timestamps: Vec<u64>,
}

// Prefetch data for better cache performance
#[cfg(target_arch = "x86_64")]
unsafe fn prefetch_data(ptr: *const u8) {
    use std::arch::x86_64::_mm_prefetch;
    _mm_prefetch(ptr as *const i8, std::arch::x86_64::_MM_HINT_T0);
}
```

## Database Performance

### PostgreSQL Optimization

#### Query Optimization
```sql
-- Optimize critical trading queries
EXPLAIN (ANALYZE, BUFFERS) 
SELECT * FROM orders 
WHERE symbol = 'AAPL' 
  AND status = 'PENDING' 
  AND created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC;

-- Create partial indexes for better performance
CREATE INDEX CONCURRENTLY idx_orders_active 
ON orders (symbol, created_at DESC) 
WHERE status IN ('PENDING', 'PARTIALLY_FILLED');

-- Optimize order execution query
CREATE INDEX CONCURRENTLY idx_orders_execution 
ON orders (order_id, status) 
WHERE status != 'CANCELLED';
```

#### Connection Pooling
```rust
use deadpool_postgres::{Config, Pool, Runtime};
use tokio_postgres::NoTls;

// Optimized connection pool configuration
let mut cfg = Config::new();
cfg.host = Some("localhost".to_string());
cfg.dbname = Some("foxhunt_production".to_string());
cfg.user = Some("foxhunt_user".to_string());
cfg.password = Some("secure_password".to_string());

// Pool sizing for high-frequency trading
cfg.pool = Some(deadpool_postgres::PoolConfig {
    max_size: 50,           // Maximum connections
    timeouts: deadpool_postgres::Timeouts {
        wait: Some(std::time::Duration::from_millis(100)),
        create: Some(std::time::Duration::from_millis(1000)),
        recycle: Some(std::time::Duration::from_millis(100)),
    },
    ..Default::default()
});

let pool = cfg.create_pool(Some(Runtime::Tokio1), NoTls)?;
```

#### Database Maintenance
```bash
#!/bin/bash
# Automated database maintenance script

# Analyze statistics daily
sudo -u postgres psql foxhunt_production -c "ANALYZE;"

# Vacuum weekly (during maintenance window)
sudo -u postgres psql foxhunt_production -c "VACUUM (ANALYZE, VERBOSE);"

# Reindex monthly
sudo -u postgres psql foxhunt_production -c "REINDEX DATABASE foxhunt_production;"

# Update statistics
sudo -u postgres psql foxhunt_production -c "
UPDATE pg_stat_statements 
SET calls = 0, total_time = 0, mean_time = 0;"
```

### InfluxDB Optimization

#### Schema Design for Performance
```sql
-- Optimize measurement schema
CREATE RETENTION POLICY "high_frequency" ON "foxhunt" DURATION 7d REPLICATION 1 DEFAULT;
CREATE RETENTION POLICY "daily_aggregates" ON "foxhunt" DURATION 90d REPLICATION 1;

-- Continuous queries for downsampling
CREATE CONTINUOUS QUERY "downsample_trades" ON "foxhunt"
BEGIN
  SELECT mean("price") AS "mean_price", 
         sum("quantity") AS "total_quantity"
  INTO "daily_aggregates"."trades_1m"
  FROM "trades"
  GROUP BY time(1m), "symbol"
END;
```

#### Write Optimization
```rust
use influxdb::{Client, Query, Timestamp};
use influxdb::InfluxDbWriteable;

// Batch writes for better performance
#[derive(InfluxDbWriteable)]
struct Trade {
    time: Timestamp,
    #[influxdb(tag)]
    symbol: String,
    #[influxdb(field)]
    price: f64,
    #[influxdb(field)]
    quantity: f64,
}

async fn batch_write_trades(client: &Client, trades: Vec<Trade>) -> Result<(), Box<dyn std::error::Error>> {
    let query = trades
        .into_iter()
        .fold(Query::write_query(Timestamp::Now, "trades"), |query, trade| {
            query.add_query(trade)
        });
    
    client.query(&query).await?;
    Ok(())
}
```

## Monitoring & Profiling

### Performance Monitoring Setup

#### Real-Time Performance Metrics
```rust
use prometheus::{Counter, Histogram, Gauge, register_counter, register_histogram, register_gauge};

lazy_static! {
    static ref ORDER_LATENCY: Histogram = register_histogram!(
        "order_submission_latency_seconds",
        "Time taken to submit an order",
        vec![0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01]  // μs to s buckets
    ).unwrap();
    
    static ref ORDERS_PROCESSED: Counter = register_counter!(
        "orders_processed_total",
        "Total number of orders processed"
    ).unwrap();
    
    static ref ACTIVE_CONNECTIONS: Gauge = register_gauge!(
        "active_connections",
        "Number of active connections"
    ).unwrap();
}

// Measure and record latency
fn submit_order_with_metrics(order: Order) -> Result<(), Error> {
    let timer = ORDER_LATENCY.start_timer();
    
    let result = submit_order(order);
    
    timer.observe_duration();
    ORDERS_PROCESSED.inc();
    
    result
}
```

#### System Performance Monitoring
```bash
#!/bin/bash
# /usr/local/bin/performance-monitor.sh

# CPU performance
echo "=== CPU Performance ==="
top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1

# Memory usage
echo "=== Memory Usage ==="
free -h | grep Mem | awk '{print "Used: " $3 " / " $2 " (" $3/$2*100 "%)"}'

# Network statistics
echo "=== Network Performance ==="
sar -n DEV 1 1 | grep -E "(eth0|ens|enp)"

# Disk I/O
echo "=== Disk I/O ==="
iostat -x 1 1 | grep -E "(nvme|sda)"

# Process-specific metrics
echo "=== Foxhunt Processes ==="
ps aux | grep foxhunt | awk '{print $1, $2, $3, $4, $11}'
```

### Profiling Tools

#### CPU Profiling with perf
```bash
# Profile CPU usage for specific process
sudo perf record -g -p $(pgrep foxhunt-core) -- sleep 30
sudo perf report

# System-wide profiling
sudo perf record -g -a -- sleep 10

# Memory profiling
sudo perf record -e cache-misses,cache-references -g ./target/release/foxhunt-core
```

#### Rust-Specific Profiling
```bash
# Install profiling tools
cargo install cargo-profdata
cargo install flamegraph

# Generate flame graphs
cargo flamegraph --bin foxhunt-core

# Profile with callgrind
valgrind --tool=callgrind --callgrind-out-file=callgrind.out ./target/release/foxhunt-core
kcachegrind callgrind.out
```

#### Memory Profiling
```bash
# Heap profiling with heaptrack
heaptrack ./target/release/foxhunt-core
heaptrack_gui heaptrack.*.gz

# Memory leak detection with valgrind
valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all ./target/release/foxhunt-core
```

## Benchmarking & Testing

### Latency Benchmarking

#### Order Submission Benchmark
```rust
use criterion::{black_box, criterion_group, criterion_main, Criterion, BatchSize};
use std::time::Instant;

fn benchmark_order_submission(c: &mut Criterion) {
    let trading_engine = TradingEngine::new().unwrap();
    
    c.bench_function("order_submission", |b| {
        b.iter_batched(
            || create_test_order(),
            |order| {
                let start = Instant::now();
                let result = trading_engine.submit_order(black_box(order));
                let duration = start.elapsed();
                
                // Assert latency requirement
                assert!(duration.as_nanos() < 50_000); // 50μs
                result
            },
            BatchSize::SmallInput,
        )
    });
}

fn benchmark_risk_check(c: &mut Criterion) {
    let risk_engine = RiskEngine::new().unwrap();
    
    c.bench_function("risk_check", |b| {
        b.iter_batched(
            || create_test_order(),
            |order| {
                let start = Instant::now();
                let result = risk_engine.check_order(black_box(&order));
                let duration = start.elapsed();
                
                // Assert latency requirement
                assert!(duration.as_nanos() < 10_000); // 10μs
                result
            },
            BatchSize::SmallInput,
        )
    });
}

criterion_group!(benches, benchmark_order_submission, benchmark_risk_check);
criterion_main!(benches);
```

### Throughput Testing

#### Load Testing Script
```bash
#!/bin/bash
# Load testing for throughput validation

DURATION=60  # Test duration in seconds
RATE=1000    # Orders per second

echo "Starting load test: $RATE orders/second for $DURATION seconds"

# Start monitoring
./scripts/start-performance-monitoring.sh &
MONITOR_PID=$!

# Generate load
for i in $(seq 1 $RATE); do
    {
        for j in $(seq 1 $DURATION); do
            curl -X POST http://localhost:8080/orders \
                -H "Content-Type: application/json" \
                -d '{"symbol":"AAPL","side":"buy","quantity":100,"price":150.00}' &
            sleep 0.001  # 1ms between requests
        done
        wait
    } &
done

wait

# Stop monitoring
kill $MONITOR_PID

echo "Load test completed"
./scripts/generate-performance-report.sh
```

### Stress Testing

#### Memory Stress Test
```bash
#!/bin/bash
# Memory stress testing

echo "Starting memory stress test"

# Generate large datasets
./target/release/foxhunt-core --mode=stress-test --memory-size=8GB &
STRESS_PID=$!

# Monitor memory usage
while kill -0 $STRESS_PID 2>/dev/null; do
    MEMORY_USAGE=$(ps -p $STRESS_PID -o %mem --no-headers)
    echo "Memory usage: ${MEMORY_USAGE}%"
    
    if (( $(echo "$MEMORY_USAGE > 90" | bc -l) )); then
        echo "WARNING: High memory usage detected"
    fi
    
    sleep 1
done

echo "Memory stress test completed"
```

#### Network Stress Test
```bash
#!/bin/bash
# Network throughput testing

# Test network bandwidth
iperf3 -c exchange-gateway.com -t 60 -P 4

# Test packet rate
hping3 -c 10000 -i u1000 exchange-gateway.com

# Monitor network statistics during test
watch -n 1 'cat /proc/net/dev | grep eth0'
```

### Performance Regression Testing

#### Automated Performance CI
```yaml
# .github/workflows/performance.yml
name: Performance Tests

on:
  push:
    branches: [main, production-hardening]
  pull_request:
    branches: [main]

jobs:
  performance:
    runs-on: [self-hosted, hft-performance]
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Build optimized binary
      run: |
        export RUSTFLAGS="-C target-cpu=native"
        cargo build --release
        
    - name: Run latency benchmarks
      run: |
        cargo bench --bench latency_tests
        
    - name: Run throughput tests
      run: |
        ./scripts/throughput-test.sh
        
    - name: Performance regression check
      run: |
        ./scripts/check-performance-regression.sh
```

### Continuous Performance Monitoring

#### Performance Baseline Tracking
```bash
#!/bin/bash
# /usr/local/bin/performance-baseline.sh

BASELINE_FILE="/var/log/foxhunt/performance_baseline.json"
CURRENT_METRICS="/tmp/current_performance.json"

# Collect current performance metrics
{
    echo "{"
    echo "  \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\","
    echo "  \"order_latency_p99\": $(curl -s http://localhost:9090/api/v1/query?query=histogram_quantile%280.99%2C%20order_submission_latency_seconds_bucket%29 | jq -r '.data.result[0].value[1]'),"
    echo "  \"throughput_ops\": $(curl -s http://localhost:9090/api/v1/query?query=rate%28orders_processed_total%5B1m%5D%29 | jq -r '.data.result[0].value[1]'),"
    echo "  \"cpu_usage\": $(top -bn1 | grep \"Cpu(s)\" | awk '{print $2}' | cut -d'%' -f1),"
    echo "  \"memory_usage\": $(free | grep Mem | awk '{printf \"%.2f\", $3/$2 * 100.0}')"
    echo "}"
} > $CURRENT_METRICS

# Compare with baseline
if [ -f "$BASELINE_FILE" ]; then
    ./scripts/compare-performance.py "$BASELINE_FILE" "$CURRENT_METRICS"
else
    cp "$CURRENT_METRICS" "$BASELINE_FILE"
    echo "Performance baseline established"
fi
```

This performance tuning guide provides comprehensive optimization strategies for achieving ultra-low latency in the Foxhunt HFT trading system. Regular monitoring and continuous optimization are essential for maintaining peak performance in production environments.