jgrusewski e7ce4395e8 perf(precompute): parallel trades load + predecoded sidecar cache ...

Flamegraph of precompute_features on 1Q ES showed 62% of CPU time in
zstd decompression, 6% in DBN FSM parsing, and only 2% in the actual
feature math — single-threaded zstd was the bottleneck, not compute.

Two fixes:

1. Per-quarter parallelism on the volume-bar trades loop (was sequential
   `for file in &trade_files`); brings it in line with the OFI path that
   already used par_iter.

2. Predecoded sidecar cache in `crates/ml-features/src/predecoded.rs`:
   first call to a `.dbn.zst` writes a bincode'd Vec<Mbp10Snapshot> or
   Vec<DbnTrade> under `<output_dir>/predecoded/`. Subsequent calls
   deserialize the sidecar and skip zstd entirely. An mtime+size header
   self-invalidates the sidecar when the source changes — no manual
   flush needed when a quarter is re-downloaded.

   Local 1Q ES results:
   - cold (writes sidecar): 40.7s (was 39.3s; +1.4s for write)
   - warm (HIT):             4.7s  (8.7× faster)
   - zstd in flat perf:      62% → 0% of CPU samples
   - sidecar disk per Q:     ~150MB

The sidecar layer also auto-dedupes within a single run: the OFI section
re-loads trades, but the second call hits the sidecar that the
volume-bar section wrote moments earlier.

CLI: `--rebuild-predecoded` purges sidecars for cold-path testing or
after a wire-format change to Mbp10Snapshot / DbnTrade. Sidecars also
self-invalidate on format-version mismatch so old caches are skipped
silently rather than mis-deserializing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-05-16 14:11:26 +02:00

284 KiB

Raw Blame History

View Raw