foxhunt/docs/archive/wave_d/agents/AGENT_25_EXECUTIVE_SUMMARY.md

AGENT 25: ML Dependency Optimization - Executive Summary

Date: 2025-10-25 Status: ✅ Analysis Complete Document: AGENT_25_DEPENDENCY_OPTIMIZATION_PLAN.md (27KB detailed report)

---

Key Findings

Current State

• Binary sizes: 5.0MB (PPO) to 9.9MB (TFT)
• Compile time: 2 minutes for ML crate
• Direct dependencies: 60 in ml/Cargo.toml
• Duplicate versions: 12 critical duplicates found

Optimization Potential

• Binary size reduction: 3.65MB (34% smaller)
• Compile time improvement: 1m 33s faster (78% reduction)
• Dependency cleanup: Remove 8 unnecessary deps

---

Top 5 Quick Wins (2 hours, 1.7MB savings)

1. Remove databento from ml crate (-800KB)

Why: ML crate should NOT download data (violates separation of concerns)

# ml/Cargo.toml - DELETE:
# databento = "0.34"
# dotenv = "0.15"

Impact: -800KB binary, -15s compile, 50+ dependencies removed

2. Optimize reqwest features (-400KB)

Why: Using ALL default features (json, charset, http2, cookies, gzip)

# Cargo.toml workspace - CHANGE:
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] }

Impact: -400KB binary, -8s compile

3. Remove pyarrow from arrow (-400KB)

Why: Python interop not needed in Rust HFT system

# Cargo.toml workspace - CHANGE:
arrow = { version = "56", default-features = false, features = ["chrono-tz"] }

Impact: -400KB binary, -10s compile

4. Remove async from parquet (-100KB)

Why: Training uses sync I/O (blocking reads are faster)

# Cargo.toml workspace - CHANGE:
parquet = { version = "56", default-features = false, features = ["arrow", "zstd"] }

Impact: -100KB binary, -5s compile

5. Optimize ndarray features (-50KB)

Why: Rayon feature duplicates workspace rayon

# ml/Cargo.toml - CHANGE:
ndarray = { version = "0.15", default-features = false, features = ["std", "serde"] }

Impact: -50KB binary, -3s compile

TOTAL PHASE 1: -1.7MB binary (-17%), -38s compile (-32%)

---

Critical Duplicate Versions

Most Impactful

1. hashbrown (3 versions!): v0.14, v0.15, v0.16 → Consolidate to v0.16 (-300KB)
2. nalgebra (2 versions): v0.32, v0.33 → Update statrs to v0.33 (-500KB)
3. opentelemetry-jaeger (outdated v0.22) → Remove entirely (-600KB)
4. rand (3 versions!): v0.8 (2x), v0.9 → Consolidate to v0.9 (-150KB)
5. rustls (3 versions!): v0.22, v0.23 (2x) → Update tokio-tungstenite (-600KB)

TOTAL CONSOLIDATION: -2.15MB binary (-21%), -55s compile

---

Rejected Alternatives

Why NOT replace reqwest with ureq

• ✅ ureq is 92% smaller (100KB vs 1.2MB)
• ❌ Blocking only (breaks async databento integration)
• ❌ No HTTP/2 (some APIs require it)
• Verdict: Optimize reqwest features instead

Why NOT replace arrow/parquet with custom reader

• ✅ Would save ~2.5MB
• ❌ 3,000+ LOC to implement (2-3 weeks)
• ❌ High complexity (Parquet spec is 200+ pages)
• ❌ Potential bugs in binary format parsing
• Verdict: Arrow/Parquet is worth the size

Why KEEP ndarray + nalgebra

• ✅ Industry standard for ML in Rust
• ✅ Excellent Candle/CUDA integration
• ❌ No lightweight alternatives with GPU support
• Verdict: Essential dependencies, optimize features only

---

3-Phase Implementation Plan

Phase 1: Quick Wins (2 hours)

Target: -1.7MB binary, -38s compile

1. Remove databento from ml crate
2. Optimize reqwest, arrow, parquet features
3. Slim down ndarray features

Phase 2: Duplicate Consolidation (4 hours)

Target: -1.55MB binary, -47s compile

1. Consolidate nalgebra v0.32 → v0.33
2. Remove opentelemetry-jaeger (deprecated)
3. Consolidate hashbrown v0.14/v0.15 → v0.16
4. Consolidate rand v0.8 → v0.9

Phase 3: Optional Enhancements (4 hours)

Target: -400KB binary, -8s compile

1. Replace chrono-tz with FixedOffset (conditional)
2. Optimize sqlx features in ml crate
3. Remove unused feature flags

GRAND TOTAL: -3.65MB binary (-34%), -1m 33s compile (-78%)

---

Expected Results

Binary Sizes After Optimization

Binary	Current	Optimized	Savings
train_tft_parquet	9.9MB	6.5MB	-3.4MB (34%)
train_mamba2_parquet	9.4MB	6.2MB	-3.2MB (34%)
train_dqn	9.6MB	6.3MB	-3.3MB (34%)
train_ppo	5.0MB	3.3MB	-1.7MB (34%)

Compile Times After Optimization

Phase	Current	Optimized	Improvement
ML crate only	2m 0s	27s	-1m 33s (78%)

---

Risk Assessment

Low Risk (Safe to implement immediately)

• ✅ Remove databento from ml crate
• ✅ Optimize reqwest/arrow/parquet features
• ✅ Optimize ndarray features

Medium Risk (Requires testing)

• ⚠️ Consolidate nalgebra versions
• ⚠️ Remove opentelemetry-jaeger
• ⚠️ Consolidate hashbrown/rand versions

High Risk (NOT recommended)

• ❌ Replace reqwest with ureq
• ❌ Custom Parquet reader
• ❌ Remove ndarray/nalgebra

---

Recommendation

APPROVE Phase 1 for immediate implementation (2 hours, 1.7MB savings, low risk)

DEFER Phase 2 & 3 until after FP32 Runpod deployment (medium risk, requires testing)

Rationale:

1. Phase 1 has zero breaking changes (only feature optimization)
2. 1.7MB savings is significant (17% reduction)
3. 38s compile time improvement helps iteration speed
4. Can be implemented in 2 hours with minimal testing
5. Does not interfere with Runpod deployment timeline

---

Next Steps

1. Immediate: Implement Phase 1 (2 hours)

   • Remove databento from ml/Cargo.toml
   • Update workspace features (reqwest, arrow, parquet)
   • Test: cargo build --release -p ml --examples
   • Verify binary sizes: du -h target/release/examples/train_*

2. Week 2-3: Implement Phase 2 (4 hours, after Runpod deployment)

   • Consolidate duplicate versions
   • Full test suite validation
   • Benchmark regression testing

3. Week 4: Implement Phase 3 (4 hours, optional)

   • Conditional chrono-tz replacement
   • Final feature optimization
   • Document final results

---

Full details: See AGENT_25_DEPENDENCY_OPTIMIZATION_PLAN.md (27KB, 600+ lines)