foxhunt/docs/archive/wave_d/reports/DATABENTO_DEPENDENCY_ANALYSIS.md

Databento Dependency Analysis - ML Crate

Date: 2025-10-25
Agent: Binary Optimization Quick Win Analysis
Status: ❌ CANNOT REMOVE - Dependency is heavily used

Summary

The databento dependency CANNOT be removed from the ml crate. Analysis shows extensive usage across:

• 6 core data loaders in ml/src/data_loaders/
• 15+ training examples in ml/examples/
• 1 benchmark module in ml/src/benchmark/

Recommendation: Keep databento dependency. Focus on other binary optimization strategies.

---

Usage Analysis

Core Source Files (ml/src/)

Data Loaders (6 files - CRITICAL)

1. ml/src/data_loaders/calibration.rs

   • Uses DbnSequenceLoader for INT8 calibration
   • Line 116: use super::DbnSequenceLoader;
   • Line 127-142: DBN file loading and copying

2. ml/src/data_loaders/tlob_loader.rs

   • Uses dbn::decode::{DbnDecoder, DbnMetadata, DecodeRecordRef}
   • Line 216: let mut decoder = DbnDecoder::new(reader)
   • TLOB model requires MBP-10 data from Databento

3. ml/src/data_loaders/streaming_dbn_loader.rs (52 references)

   • Primary streaming loader: StreamingDbnLoader struct
   • Line 43-44: Imports DbnParser and DbnDecoder
   • Lines 237-272: DBN file discovery and streaming logic
   • CRITICAL: Used for memory-efficient large dataset training

4. ml/src/data_loaders/dbn_tick_adapter.rs (34 references)

   • DBNTickAdapter struct for converting DBN bars to ticks
   • Line 64-65: Imports DbnParser and DbnDecoder
   • Line 210: let mut decoder = DbnDecoder::new(reader)
   • CRITICAL: Required for alternative bar sampling (tick/volume/dollar bars)

5. ml/src/data_loaders/dbn_sequence_loader.rs (93 references)

   • Core loader: DbnSequenceLoader struct
   • Line 33-34: Imports DbnParser and DbnDecoder
   • Lines 417-440: DBN file discovery and sequence loading
   • CRITICAL: Used by MAMBA-2, TFT, and all sequence-based models

6. ml/src/real_data_loader.rs (13 references)

   • RealDataLoader for production data loading
   • Line 33: use dbn::decode::{DbnDecoder, DecodeRecordRef};
   • Lines 173-220: DBN file parsing
   • PRODUCTION: Used in live trading data pipeline

Other Modules

• ml/src/tlob/mbp10_feature_extractor.rs: Uses data::providers::databento::mbp10::Mbp10Snapshot
• ml/src/model_registry.rs: Documents "databento_2024_Q4" data source
• ml/src/benchmark/performance_tracker.rs: Tracks dbn_load_time_ms metric
• ml/src/benchmark/data_loader.rs: DbnDataLoader struct for benchmarking

---

Examples (ml/examples/)

15+ training/validation examples depend on databento:

Critical Training Examples

1. download_training_data.rs (DIRECT API USAGE)

   • Line 27-28: use databento::historical::timeseries::GetRangeParams;
   • Line 28: use databento::{Compression, HistoricalClient};
   • CRITICAL: Downloads market data from Databento API

2. train_mamba2.rs

   • Line 38: use ml::data_loaders::DbnSequenceLoader;
   • Line 74: Default path test_data/real/databento/ml_training_small
   • Lines 164-182: DBN sequence loading for MAMBA-2 training

3. train_tft_dbn.rs

   • Line 23: use dbn::decode::{DbnDecoder, DecodeRecordRef};
   • Line 362: async fn load_dbn_ohlcv_bars()
   • Tests at lines 649-678 validate DBN loading

4. train_liquid_dbn.rs

   • Line 18: use ml::data_loaders::dbn_sequence_loader::DbnSequenceLoader;
   • Line 43: Creates DbnSequenceLoader

Validation Examples

• validate_databento_files.rs: Full DBN file validation suite
• validate_225_features_databento.rs: 225-feature validation with DBN data
• validate_features_1_50.rs: Feature validation (lines 27-32)
• validate_wave_c_features_51_150.rs: Wave C feature validation

Benchmark Examples

• benchmark_streaming_vs_batch.rs: Compares StreamingDbnLoader vs DbnSequenceLoader
• tft_int8_calibration.rs: INT8 calibration with DBN data (lines 176-191)

Backtest Examples

• wave_comparison_full_features.rs: Wave comparison backtest (line 65)
• wave_c_backtest.rs: Wave C backtest (lines 150-162)
• comprehensive_model_backtest.rs: Uses DbnParser (line 702)
• backtest_ensemble.rs: Ensemble backtest (line 430)

---

Cargo.toml Dependencies

# ml/Cargo.toml lines 131-132
dbn.workspace = true  # Databento Binary format for real market data loading
databento = "0.34"    # Databento API client for downloading data (includes async by default)

Why Both Dependencies?

1. dbn: Low-level binary format parser (workspace dependency)

   • Used by: DbnDecoder, DbnMetadata, DecodeRecordRef
   • Purpose: Parse .dbn binary files (OHLCV, MBP-10, trades)

2. databento: High-level API client (version 0.34)

   • Used by: download_training_data.rs example
   • Purpose: Download market data from Databento API
   • Features: Historical timeseries, real-time streaming

---

Impact Analysis

If Removed: Broken Functionality

🔴 Critical Breakages

1. All ML model training would fail (MAMBA-2, TFT, DQN, PPO)

   • No way to load historical market data
   • DbnSequenceLoader is the primary training data source

2. Alternative bar sampling would break (Wave B)

   • Tick/volume/dollar/imbalance/run bars require DBN ticks
   • DBNTickAdapter unusable

3. INT8 quantization calibration would fail (Wave QAT)

   • calibration.rs requires DBN data for observer calibration

4. TLOB model would be unusable

   • Requires MBP-10 order book data from Databento

5. Data downloading would be impossible

   • download_training_data.rs is the only way to acquire new data
   • No alternative data source exists

⚠️ Secondary Breakages

• Performance benchmarks would fail (benchmark_streaming_vs_batch.rs)
• Validation tests would fail (15+ examples)
• Production data pipeline would break (RealDataLoader)

---

Alternative Optimization Strategies

Since databento cannot be removed, consider these alternatives:

1. Binary Stripping

# Strip debug symbols from release builds
cargo build --release -p ml --example train_tft_parquet
strip target/release/examples/train_tft_parquet
# Expected savings: 20-30% (debug symbols)

2. Feature Reduction

Check if databento = "0.34" has optional features that can be disabled:

# Investigate:
databento = { version = "0.34", default-features = false, features = ["historical"] }

3. LTO (Link-Time Optimization)

Already enabled in Cargo.toml workspace settings:

[profile.release]
lto = "fat"
codegen-units = 1

4. Dynamic Linking (CUDA/cuDNN)

• CUDA libraries are already dynamically linked
• Databento/dbn are Rust crates (statically linked by default)

5. Dependency Audit

Run cargo tree -p ml | grep databento to see transitive dependencies:

cargo tree -p ml -e normal | grep -E "(databento|dbn)"

---

Conclusion

CANNOT REMOVE DATABENTO: The dependency is foundational to the ML training pipeline.

Binary Size Impact:

• databento crate: ~50-100KB compiled
• dbn crate: ~30-50KB compiled
• Total: ~80-150KB (negligible vs 50MB total binary size)

Recommendation: Focus on:

1. LTO optimization (already enabled)
2. Strip debug symbols (strip command)
3. Feature flag optimization (investigate databento optional features)
4. Dead code elimination (clippy + manual review)

Next Quick Win: Investigate other unused dependencies (e.g., arrayfire, petgraph) that may have larger impact.

---

References

• CLAUDE.md: Documents DBN data loading as critical infrastructure
• Wave D Documentation: Relies on DBN data for regime detection validation
• Wave C Implementation: All 201 features extracted from DBN market data
• Runpod Deployment: Uses pre-uploaded .parquet files (converted from DBN)