20aa345a7c
Databento historical bulk DBN files emit symbol mappings only in the metadata header (date-range form), not as in-stream SymbolMappingMsg records. The strict 'no mapping = error' branch from the previous commit blocked the front-month smoke (alpha-perception-vstrr) at Q1 2024 where the dominant id 5602 has no streaming mapping entry. Volume-leader detection itself doesn't depend on the symbol; the regex check was defense-in-depth. Log-warn on symbol-mismatch and proceed with id=metadata-only when no streaming mapping exists. Calendar-spread anomalies still surface in logs without blocking training.
data
Market data ingestion, broker integration, and feature extraction.
Providers
- Databento — historical and real-time market data via DBN format
- Benzinga — news and fundamentals feed
Broker Integrations
- IB TWS — Interactive Brokers TWS/Gateway socket connection
- ICMarkets — FIX 4.4 protocol integration
Key Modules
brokers— broker adapters (IB TWS, ICMarkets FIX)providers— data provider clients (Databento, Benzinga)parquet_persistence— Parquet read/write for tick and bar datareplay— market data replay for backtestingtraining_pipeline— data preparation for ML model trainingfeatures— technical indicator and feature computationvalidation— data quality checks and schema validation
Cargo Features
| Feature | Default | Description |
|---|---|---|
databento |
yes | Databento provider support |
benzinga |
yes | Benzinga provider support |
icmarkets |
yes | ICMarkets FIX 4.4 integration |
redis-cache |
no | Redis caching layer |
ib |
no | Interactive Brokers adapter |
mock |
no | Mock providers for testing |