foxhunt/docs/archive/performance/LOAD_TEST_DEPENDENCY_OPTIMIZATION.md

Load Test Dependency Optimization Report

Date: 2025-10-11 Target: tests/load_test_trading_service.rs Goal: Reduce compilation time from >2 minutes to <2 minutes

---

Analysis Summary

Dependencies Actually Used in load_test_trading_service.rs

Based on code analysis, the load test ONLY uses:

// Standard library (no external deps)
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime};

// External dependencies
use tokio::time::timeout;
use tonic::transport::Channel;
use tonic::{Request, Status};
use uuid::Uuid;
use reqwest;  // Only for health checks in test_5

// Generated protobuf code
pub mod trading {
    tonic::include_proto!("trading");
}

Total external dependencies needed: 4 crates

• tokio (async runtime)
• tonic (gRPC client)
• uuid (order ID generation)
• reqwest (HTTP health checks)

---

Current tests/Cargo.toml Analysis

Dependencies Currently Declared (36 total)

Core async (3):

• ✅ tokio - USED
• ❌ tokio-test - UNUSED in load test
• ❌ tokio-stream - UNUSED in load test

Foxhunt crates (8):

• ❌ trading_engine - UNUSED in load test
• ❌ risk - UNUSED in load test
• ❌ risk-data - UNUSED in load test
• ❌ ml - UNUSED in load test
• ❌ data - UNUSED in load test
• ❌ tli - UNUSED in load test
• ❌ common - UNUSED in load test
• ❌ config - UNUSED in load test
• ❌ trading_service - UNUSED in load test

Serialization (4):

• ❌ serde - UNUSED in load test
• ❌ serde_json - UNUSED in load test
• ❌ toml - UNUSED in load test
• ❌ chrono - UNUSED in load test

Math (1):

• ❌ num - UNUSED in load test

Concurrency (2):

• ❌ crossbeam - UNUSED in load test
• ❌ arc-swap - UNUSED in load test

Async utilities (2):

• ❌ async-trait - UNUSED in load test
• ❌ futures - UNUSED in load test

gRPC (2):

• ✅ tonic - USED
• ❌ tonic-health - UNUSED in load test

HTTP (1):

• ✅ reqwest - USED (minimally, only test_5)

JWT (1):

• ❌ jsonwebtoken - UNUSED in load test

Database (3):

• ❌ sqlx - UNUSED in load test
• ✅ uuid - USED
• ❌ rust_decimal - UNUSED in load test
• ❌ rust_decimal_macros - UNUSED in load test

Error handling (2):

• ❌ anyhow - UNUSED in load test
• ❌ thiserror - UNUSED in load test

Environment (1):

• ❌ dotenvy - UNUSED in load test

CLI (1):

• ❌ clap - UNUSED in load test

Additional testing (5):

• ❌ rand - UNUSED in load test
• ❌ rand_distr - UNUSED in load test (used elsewhere)
• ❌ parking_lot - UNUSED in load test (used elsewhere)
• ❌ hdrhistogram - UNUSED in load test (used in e2e/benches)
• ❌ lazy_static - UNUSED in load test (used in fixtures)

Performance (2):

• ❌ criterion - UNUSED in load test (used in benches)
• ❌ quickcheck - UNUSED in load test

Database integration (2):

• ❌ redis - UNUSED in load test
• ❌ influxdb2 - UNUSED in load test

Monitoring (2):

• ❌ tracing - UNUSED in load test
• ❌ tracing-subscriber - UNUSED in load test

File system (1):

• ❌ tempfile - UNUSED in load test

Memory profiling (2):

• ❌ dhat - UNUSED in load test
• ❌ jemalloc_pprof - UNUSED in load test

Dev Dependencies (3):

• ❌ tempfile - Duplicate, UNUSED in load test
• ❌ serial_test - UNUSED in load test
• ❌ rstest - UNUSED in load test

---

Recommendations

Option 1: Create Dedicated Load Test Crate (RECOMMENDED)

Create a minimal tests/load_tests/Cargo.toml:

[package]
name = "load_tests"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "load_test_trading_service"
path = "src/load_test_trading_service.rs"

[dependencies]
# Minimal dependencies for load testing
tokio = { workspace = true }
tonic = { workspace = true }
prost = { workspace = true }
uuid = { workspace = true }
reqwest = { workspace = true, features = ["json"] }

[build-dependencies]
tonic-prost-build = { workspace = true }

Benefits:

• Compile time: ~20-30 seconds (90% reduction)
• Isolated from heavy testing dependencies
• Clear separation of concerns
• Can run independently

Implementation:

mkdir -p tests/load_tests/src
mv tests/load_test_trading_service.rs tests/load_tests/src/
# Create minimal Cargo.toml above
# Add to workspace members in root Cargo.toml

Option 2: Feature-Flag Heavy Dependencies

Modify tests/Cargo.toml to make heavy deps optional:

[dependencies]
# Always needed
tokio.workspace = true
tonic.workspace = true
uuid.workspace = true

# Optional for specific test types
reqwest = { workspace = true, optional = true }
criterion = { workspace = true, optional = true }
hdrhistogram = { workspace = true, optional = true }
# ... etc

[features]
default = []
load-tests = ["reqwest"]
performance-tests = ["criterion", "hdrhistogram"]
integration-tests = ["redis", "influxdb2", "sqlx"]

Run load tests with:

cargo test --test load_test_trading_service --features load-tests

Benefits:

• Reduces compilation when running specific test types
• Keeps all tests in one crate
• Flexible feature combinations

Option 3: Optimize Current Cargo.toml

Remove unused dependencies from tests/Cargo.toml:

Remove these (32 dependencies):

# REMOVE - not used in load test
tokio-test, tokio-stream
trading_engine, risk, risk-data, ml, data, tli, common, config, trading_service
serde, serde_json, toml, chrono
num
crossbeam, arc-swap
async-trait, futures
tonic-health
jsonwebtoken
sqlx, rust_decimal, rust_decimal_macros
anyhow, thiserror
dotenvy
clap
rand, rand_distr, parking_lot, hdrhistogram, lazy_static
criterion, quickcheck
redis, influxdb2
tracing, tracing-subscriber
tempfile (dev-dep duplicate)
serial_test, rstest
dhat, jemalloc_pprof

Keep only (4 dependencies):

[dependencies]
tokio.workspace = true
tonic.workspace = true
uuid.workspace = true
reqwest = { workspace = true, features = ["json"], optional = true }

[features]
smoke-tests = ["reqwest"]

Benefits:

• Immediate 70-80% compilation time reduction
• Still supports load testing
• Breaking change: other tests need migration

---

Compilation Time Estimates

Approach	Est. Compilation Time	Change	Complexity
Current	120-180 seconds	Baseline	-
Option 1 (Dedicated crate)	20-30 seconds	-85%	Medium
Option 2 (Feature flags)	40-60 seconds	-65%	Low
Option 3 (Remove unused)	30-50 seconds	-75%	High (breaking)

---

Root Cause Analysis

The load test timeout issue is caused by:

1. Dependency cascade: tests/Cargo.toml includes 36 dependencies
2. Heavy Foxhunt crates: ml, data, trading_engine bring in dozens of transitive deps
3. Unnecessary test frameworks: criterion, quickcheck, proptest compile slowly
4. Database clients: sqlx, redis, influxdb2 add significant compile time

The load test file itself is minimal (500 lines, 4 direct dependencies), but Cargo compiles ALL dependencies declared in Cargo.toml regardless of whether they're used.

---

Implementation Priority

Immediate (< 30 minutes):

1. ✅ Create dedicated tests/load_tests crate (Option 1)
2. ✅ Move load_test_trading_service.rs
3. ✅ Add minimal Cargo.toml
4. ✅ Test compilation time

Short-term (1-2 hours):

1. Implement feature flags for other test types (Option 2)
2. Audit other test files for similar issues
3. Update CI/CD to use feature flags

Long-term (1 week):

1. Restructure entire tests/ crate into focused sub-crates
2. Migrate benchmark tests to benches/
3. Document dependency optimization guidelines

---

Verification Plan

# Before optimization
time cargo build --test load_test_trading_service --release

# After Option 1 (dedicated crate)
time cargo build --bin load_test_trading_service --release

# Expected: 20-30 seconds (< 2 minute target ✅)

---

Additional Optimizations

Profile Tuning

Current [profile.test] in root Cargo.toml:

[profile.test]
opt-level = 1
debug = true
debug-assertions = true
overflow-checks = true
lto = false
incremental = true
codegen-units = 256  # ← Very high, slows linking

Optimize for faster compilation:

[profile.test]
opt-level = 0  # Faster compilation, slower runtime (OK for tests)
debug = true
debug-assertions = true
overflow-checks = true
lto = false
incremental = true
codegen-units = 16  # Faster linking (was 256)

Workspace Optimization

Enable sparse index in .cargo/config.toml:

[registries.crates-io]
protocol = "sparse"

Enable parallel frontend in .cargo/config.toml:

[build]
pipelining = true

---

Conclusion

Recommendation: Implement Option 1 (Dedicated crate)

Rationale:

• Cleanest separation
• Fastest compilation (20-30s, well under 2min target)
• No breaking changes to existing tests
• Easy to maintain
• Can be done in < 30 minutes

Next Steps:

1. Create tests/load_tests directory
2. Move load_test_trading_service.rs
3. Create minimal Cargo.toml
4. Add to workspace members
5. Test compilation time
6. Update documentation

---

Author: Claude (Foxhunt HFT) Status: Ready for implementation