Files

jgrusewski 6df3284d0d fix(data): MBP-10 decoder corrupted the inside quote (level 0)

Both parse_mbp10_file and parse_mbp10_streaming wrote the single MBP-10
update event's (price,size) into levels[0] via update_level(0,...) and
then copied the authoritative book from mbp10.levels[1..] — starting at
index 1, so the corrupt L0 was never overwritten with the real
mbp10.levels[0]. Result on real cluster data (ES.FUT 2025-Q1 front-month,
2M records): 14.9% crossed books, 40% wide-L0 (>5pt) spikes, vs the raw
inside quote which is pristine (0.016% crossed, 0% wide, 0.25pt median).
Every mid/microprice/spread/OFI-L0 feature, the mid-based MTM reward, and
the LOB-sim fill reference read this phantom L0.

Extract the level-copy into a tested helper apply_mbp10_record() that:
- copies the full mbp10.levels[0..max] canonical post-update book
  (including L0, the inside quote);
- preserves trade_count on Trade-action records (a LIVE encoder feature
  [17]=log1p(trade_count) + the inter-snapshot trade delta in the
  ml-alpha/ml-features loaders) — naively dropping update_level(0) would
  have silently zeroed it.

Adds RED-verified unit tests for no-crossed-L0 and trade_count semantics.
cargo test -p data --lib: 377 passed.

Sidecars (.predecoded.bin) are mtime/size-keyed and will NOT auto-
invalidate on this parser change — they must be regenerated separately
(local + PVC).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-05 14:48:05 +02:00

benches

fix(clippy): resolve all clippy warnings across entire workspace (--all-targets)

2026-03-13 10:18:35 +01:00

data

refactor: restructure repo — crates/, bin/, testing/ layout

2026-02-25 11:56:00 +01:00

src

fix(data): MBP-10 decoder corrupted the inside quote (level 0)

2026-06-05 14:48:05 +02:00

tests

fix(data,ml-backtesting): DBN nanoprice scaling + skip-on-corrupt-top-of-book

2026-05-20 09:37:07 +02:00

Cargo.toml

build: per-package CGU + dep dedup — workspace builds ~30% faster

2026-05-01 01:00:52 +02:00

README.md

docs: create/update README.md for all 17 crates

2026-03-01 22:47:39 +01:00

README.md

data

Market data ingestion, broker integration, and feature extraction.

Providers

Databento — historical and real-time market data via DBN format
Benzinga — news and fundamentals feed

Broker Integrations

IB TWS — Interactive Brokers TWS/Gateway socket connection
ICMarkets — FIX 4.4 protocol integration

Key Modules

brokers — broker adapters (IB TWS, ICMarkets FIX)
providers — data provider clients (Databento, Benzinga)
parquet_persistence — Parquet read/write for tick and bar data
replay — market data replay for backtesting
training_pipeline — data preparation for ML model training
features — technical indicator and feature computation
validation — data quality checks and schema validation

Cargo Features

Feature	Default	Description
`databento`	yes	Databento provider support
`benzinga`	yes	Benzinga provider support
`icmarkets`	yes	ICMarkets FIX 4.4 integration
`redis-cache`	no	Redis caching layer
`ib`	no	Interactive Brokers adapter
`mock`	no	Mock providers for testing