db9fd1b2da
Bottleneck B of two: replaces the sequential `ImbalanceBarSampler` walk
in `mbp10_to_imbalance_bars` with a two-pass parallel decomposition.
Bit-identical to sequential — verified by 6/6 passing tests in
`crates/ml-features/tests/imbalance_bars_parallel_bit_equiv_test.rs` at
both dense (T=20, ~23k bars) and sparse (T=500, ~300 bars) emission
regimes, exact 0.0 diff on every f64 field.
Why two-pass (NOT time-bucket sharding)
=======================================
Time-bucket sharding (the SP20 OFI pattern in
`compute_ofi_per_bar_parallel`) bit-equates to sequential because OFI
features derive from BOUNDED rolling windows (VPIN ≤50, Kyle ≤100,
trade-imb ≤100, OFI-stats ≤300). After ≥window-size warmup updates,
two rolling buffers started from different initial states converge
bit-identically.
`ImbalanceBarSampler` is fundamentally different: its `cumulative_imbalance`
is a path integral that resets only when crossing ±threshold. There is
NO bounded-lookback window. Two replays of the same trade tape starting
from different cum_imb offsets emit at DIFFERENT trade indices, and the
phase offset is NOT guaranteed to vanish at any future trade — even
after many emissions, a bounded phase difference can persist
indefinitely. An earlier time-bucket-sharded prototype with WARMUP=10000
trades passed K=4/K=8 at threshold=20 (dense emissions, ~30 emissions
per shard's warmup window) but FAILED at threshold=500 with a 1-bar
count drift (golden=303, parallel=304) — exactly the kind of phase-
offset divergence predicted by the path-integral analysis.
Two-pass decomposition
======================
Pass 1 (sequential, lightweight): walk the trade tape ONCE tracking only
cum_imb, prev_price, last_direction. Record the trade index of every
emission-triggering trade as the end of a segment. No OHLCV state, no
Vec<OHLCVBar> allocation, no per-trade conditional bar construction.
O(N) simple arithmetic — for 50k-2M trades it runs in 1-15ms.
Pass 2 (parallel rayon): each emission segment [boundary[i-1]+1,
boundary[i]] produces exactly one bar via independent OHLCV reduction
over its trade slice. Segments are fully independent; `par_iter()` over
segment ranges is trivially correct.
Bit-equivalence guarantee
=========================
Pass 1 mirrors `ImbalanceBarSampler::update` arithmetic verbatim:
- zero-volume early-return (alternative_bars.rs:484-486)
- direction tie-break (alternative_bars.rs:495-506)
- cum_imb update (alternative_bars.rs:508-510)
- emission threshold check (alternative_bars.rs:522-523)
- prev_price/last_direction kept across emissions (reset() at :558-566)
- NO trailing-partial-bar flush (matches sequential exactly)
Pass 2's per-segment OHLCV reduction matches the sampler's per-bar OHLCV
update verbatim: open = first non-zero-volume trade in segment, close =
last non-zero-volume trade, high/low = max/min over non-zero-volume
trades, volume = sum, timestamp = open's timestamp.
Performance (release-mode bench, 16-thread rayon pool)
======================================================
n=100k: seq=1.39ms, par=1.88ms, speedup=0.74x (rayon overhead dominates)
n=500k: seq=8.04ms, par=3.35ms, speedup=2.40x
n=2M: seq=27.59ms, par=12.39ms, speedup=2.23x
Speedup is bounded by Pass 1 (sequential, ~5-7ms at 2M trades) since
Pass 2 (parallel, ~2-3ms at 2M trades on 16 threads) is ~3x faster
than Pass 1's sequential floor. At production scale (50k-500k trades
per `mbp10_to_imbalance_bars` call) we get 2-2.4x speedup over the
all-sequential baseline.
The `min_bars_per_task` cutoff falls back to a sequential Pass 2 when
segment count < 256 (the rayon spawn overhead exceeds the parallel
benefit). Tunable via the second function arg.
Files changed
=============
- crates/ml-features/src/alternative_bars.rs (+1 -1):
derive `Clone` on `ImbalanceBarSampler` for parallel-shard use
(carried through this commit even though Pass 1's hand-rolled walk
in `mbp10_loader.rs` doesn't need it — keeps the type clonable for
future test/bench infrastructure).
- crates/ml-features/src/lib.rs (+3 -2):
re-export `imbalance_bars_parallel`, `imbalance_bars_sequential`,
`DEFAULT_IMBALANCE_BAR_MIN_BARS_PER_TASK`.
- crates/ml-features/src/mbp10_loader.rs (+253 -15):
new `imbalance_bars_parallel`, `imbalance_bars_sequential`,
`DEFAULT_IMBALANCE_BAR_MIN_BARS_PER_TASK` const. Replace the inline
sampler-walk loop in `mbp10_to_imbalance_bars` with a call into
`imbalance_bars_parallel`. Module note explains why time-bucket
sharding doesn't apply.
- crates/ml-features/tests/imbalance_bars_parallel_bit_equiv_test.rs
(+272 NEW): hermetic synthetic-trade bit-equivalence tests at:
* default cutoff (par_iter Pass 2)
* forced par_iter (min_bars_per_task=1)
* forced sequential Pass 2 (min_bars_per_task=usize::MAX)
* empty input
* small input (Pass 2 below par cutoff)
* high-threshold sparse emissions (the case that broke the
time-bucket sharding prototype). All pass exact 0.0 diff.
Also note: 293/293 ml-features lib tests pass (no regression).
Pairs with the file-level par_iter trade extraction in
8f5c64e10 (Bottleneck A). Together they parallelise both halves of
`mbp10_to_imbalance_bars`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
79 lines
3.3 KiB
Rust
79 lines
3.3 KiB
Rust
//! Feature Engineering for Foxhunt ML models
|
|
//!
|
|
//! Core feature extractors: technical indicators, price patterns, volume analysis,
|
|
//! microstructure proxies, OFI (Order Flow Imbalance), normalization, and more.
|
|
|
|
#![allow(clippy::module_name_repetitions)]
|
|
#![allow(clippy::integer_division)]
|
|
#![deny(
|
|
clippy::unwrap_used,
|
|
clippy::expect_used,
|
|
clippy::panic
|
|
)]
|
|
#![allow(clippy::shadow_reuse, clippy::shadow_same, clippy::shadow_unrelated)] // Tensor ops: let x = x.relu() is idiomatic
|
|
#![allow(clippy::non_ascii_literal)] // Math symbols in ML documentation and error messages
|
|
#![allow(clippy::partial_pub_fields)] // ML config structs: some fields are pub API, some internal
|
|
#![allow(clippy::same_name_method)] // Intentional: inherent methods shadow trait defaults for ML-specific behavior
|
|
#![allow(clippy::indexing_slicing)] // Tensor/matrix indexing with bounds guaranteed by construction
|
|
#![allow(clippy::similar_names)] // ML naming: min_val/max_val, state/states are conventional
|
|
|
|
// Re-export core types at crate root for internal use
|
|
pub use ml_core::MLError;
|
|
pub use ml_core::types::OHLCVBar;
|
|
|
|
// Public modules (21 total)
|
|
pub mod adx_features;
|
|
pub mod alternative_bars;
|
|
pub mod bar_resampler;
|
|
pub mod barrier_optimization;
|
|
pub mod config;
|
|
pub mod ewma;
|
|
pub mod feature_extraction;
|
|
pub mod mbp10_loader;
|
|
pub mod microstructure;
|
|
pub mod microstructure_features;
|
|
pub mod normalization;
|
|
pub mod ofi_calculator;
|
|
pub mod pipeline;
|
|
pub mod position_features;
|
|
pub mod price_features;
|
|
pub mod regime_adx;
|
|
pub mod statistical_features;
|
|
pub mod time_features;
|
|
pub mod trades_loader;
|
|
pub mod types;
|
|
pub mod volume_features;
|
|
|
|
// Re-exports
|
|
pub use config::{FeatureConfig, FeatureGroup, FeatureIndices, FeaturePhase};
|
|
pub use alternative_bars::{
|
|
DollarBarSampler, ImbalanceBarSampler, RunBarSampler, TickBarSampler, VolumeBarSampler,
|
|
};
|
|
pub use barrier_optimization::{BarrierOptimizer, BarrierParams, OptimizationResult};
|
|
pub use ewma::{AdaptiveThreshold, EWMACalculator};
|
|
pub use price_features::PriceFeatureExtractor;
|
|
pub use volume_features::VolumeFeatureExtractor;
|
|
pub use adx_features::AdxFeatureExtractor;
|
|
pub use regime_adx::RegimeADXFeatures;
|
|
pub use microstructure_features::{
|
|
BuySellImbalance, HighLowSpread, InterArrivalTime, KyleLambda, MicrostructureFeature,
|
|
PriceImpact, TickCount, VarianceRatio, VolumeWeightedSpread,
|
|
};
|
|
pub use time_features::TimeFeatureExtractor;
|
|
pub use statistical_features::StatisticalFeatureExtractor;
|
|
pub use normalization::{FeatureNormalizer, NormalizationStats, RingBuffer};
|
|
pub use pipeline::{FeatureExtractionPipeline, PipelinePerformance};
|
|
pub use position_features::PositionFeatures;
|
|
pub use ofi_calculator::{
|
|
compute_ofi_per_bar_parallel, compute_ofi_per_bar_sequential, OFICalculator, OFIFeatures,
|
|
PerBarOfiInputs, DEFAULT_OFI_PARALLEL_WARMUP_BARS, PER_BAR_OFI_DIM,
|
|
};
|
|
pub use mbp10_loader::{
|
|
extract_trades_from_dbn_file, extract_trades_from_snapshots, get_recent_snapshots,
|
|
get_snapshots_for_timestamp, imbalance_bars_parallel, imbalance_bars_sequential,
|
|
load_mbp10_snapshots_sync, mbp10_to_imbalance_bars, Mbp10Trade,
|
|
DEFAULT_IMBALANCE_BAR_MIN_BARS_PER_TASK,
|
|
};
|
|
pub use trades_loader::{build_volume_bars, filter_front_month, get_trades_for_bar, load_trades_sync, DbnTrade, DEFAULT_VOLUME_BAR_SIZE};
|
|
pub use bar_resampler::BarResampler;
|