Both parse_mbp10_file and parse_mbp10_streaming wrote the single MBP-10
update event's (price,size) into levels[0] via update_level(0,...) and
then copied the authoritative book from mbp10.levels[1..] — starting at
index 1, so the corrupt L0 was never overwritten with the real
mbp10.levels[0]. Result on real cluster data (ES.FUT 2025-Q1 front-month,
2M records): 14.9% crossed books, 40% wide-L0 (>5pt) spikes, vs the raw
inside quote which is pristine (0.016% crossed, 0% wide, 0.25pt median).
Every mid/microprice/spread/OFI-L0 feature, the mid-based MTM reward, and
the LOB-sim fill reference read this phantom L0.
Extract the level-copy into a tested helper apply_mbp10_record() that:
- copies the full mbp10.levels[0..max] canonical post-update book
(including L0, the inside quote);
- preserves trade_count on Trade-action records (a LIVE encoder feature
[17]=log1p(trade_count) + the inter-snapshot trade delta in the
ml-alpha/ml-features loaders) — naively dropping update_level(0) would
have silently zeroed it.
Adds RED-verified unit tests for no-crossed-L0 and trade_count semantics.
cargo test -p data --lib: 377 passed.
Sidecars (.predecoded.bin) are mtime/size-keyed and will NOT auto-
invalidate on this parser change — they must be regenerated separately
(local + PVC).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
6df3284d0d