diff --git a/docs/superpowers/specs/2026-04-16-tick-microstructure-intelligence-design.md b/docs/superpowers/specs/2026-04-16-tick-microstructure-intelligence-design.md new file mode 100644 index 000000000..60456041d --- /dev/null +++ b/docs/superpowers/specs/2026-04-16-tick-microstructure-intelligence-design.md @@ -0,0 +1,293 @@ +# Tick-Level Microstructure Intelligence — Design Spec + +**Goal:** Extract 12 new predictive features from raw MBP-10 tick data (1000-5000 ticks/bar) to give the DQN model sub-bar microstructure awareness. At inference, use H100 idle time between bars for speculative pre-computation and mid-bar regime detection. Feature vector grows from 50 (42+8 OFI) to 62 (42+20 OFI) dimensions. + +**Core Principle:** Bar-level models see WHERE price went. Tick-level models see HOW it got there — the microstructure path within the bar predicts the next bar. + +--- + +## Data Source + +**Databento** — single source for all tick data (historical + live). + +- Dataset: `GLBX.MDP3` (CME Globex MDP 3.0) +- Schema: `mbp-10` (10-level order book snapshots) +- Symbol: `ES` (E-mini S&P 500 Futures) +- Historical: 148GB on PVC at `/mnt/training-data/futures-baseline` +- Live: Databento Rust crate (`databento`), 6.1μs median feed latency +- Format: `Mbp10Snapshot` — 10 `BidAskPair` levels (bid_px, bid_sz, bid_ct, ask_px, ask_sz, ask_ct) +- Trades: from same Databento feed, Lee-Ready classification for buy/sell + +IBKR is execution only (max 9 depth levels, subscription caps — unsuitable for tick data). + +--- + +## The 12 New Microstructure Features + +OFI_DIM: 8→20. Existing 8 features (OFI_L1, OFI_L5, depth_imbalance, VPIN, Kyle's Lambda, bid_slope, ask_slope, trade_imbalance) stay at indices 0-7. New features at indices 8-19. + +All features are **incremental** — O(1) per tick via running statistics (EMAs, counters, running sums). Same computation for precompute (batch over historical ticks) and inference (streaming from Databento live). + +### PEARLS (Cont 2014, Easley 2012) + +| Index | Feature | Incremental State | Formula | +|-------|---------|-------------------|---------| +| 8 | **OFI Trajectory** | LinearRegressor (online Welford) | Slope of OFI_L1 over all ticks in bar. Front-loaded buying → continuation. Back-loaded → reversal. | +| 9 | **Realized Variance** | Running sum of squared returns | Σ(log(mid_t/mid_{t-1})²). True intra-bar volatility — bar range is a poor proxy. | +| 10 | **Trade Arrival Intensity** | Hawkes exponential kernel EMA | λ(t) = μ + α·Σexp(-β(t-t_i)). Self-exciting clusters. Rising intensity → momentum. | + +### GEMS (microstructure theory) + +| Index | Feature | Incremental State | Formula | +|-------|---------|-------------------|---------| +| 11 | **Book Pressure Gradient** | Weighted running sum | Σ_l(exp(-0.3l)·(bid_sz_l - ask_sz_l)) / Σ(bid_sz_l + ask_sz_l). Deep vs surface pressure across 10 levels. | +| 12 | **Spread Dynamics** | Running min/max/mean of spread | (max_spread - min_spread) / mean_spread. Widening = market makers pulling back = volatility incoming. | +| 13 | **Aggression Ratio** | Two counters (ask_trades, total) | trades_at_ask / total_trades (Lee-Ready). > 0.6 = strong buy pressure. | +| 14 | **Queue Depletion Asymmetry** | Running depletion rates bid/ask | (bid_depletion - ask_depletion) / max. Reveals hidden institutional flow direction. | +| 15 | **Depth Renewal Rate** | Counters (new_orders, cancels) at top 3 | new_placed / canceled. Market makers refreshing = confidence. Not refreshing = about to pull. | + +### NOVELS (new combinations) + +| Index | Feature | Incremental State | Formula | +|-------|---------|-------------------|---------| +| 16 | **Intra-Bar Momentum** | Two accumulators (first/second half returns) | corr(returns_first_half, returns_second_half). Positive = continuation. Negative = reversal. Detects regime WITHIN the bar. | +| 17 | **Microstructure Regime Score** | Composite of aggression, spread, book thickness | sigmoid(aggression × spread_widen × thin_book). "Institutional sweep mode" (high) vs "market making mode" (low). | +| 18 | **Order Flow Acceleration** | EMA of OFI deltas | d(OFI_L1)/dt — is buying accelerating or decelerating? Acceleration → breakout. Deceleration → pullback. | +| 19 | **Toxicity Gradient** | EMA of VPIN deltas | d(VPIN)/dt. Catches the ONSET of informed flow — earlier signal than VPIN level alone. | + +--- + +## IncrementalMicrostructureCalculator + +New struct extending the existing `OFICalculator` in `crates/ml-features/src/ofi_calculator.rs`. + +```rust +pub struct IncrementalMicrostructureCalculator { + /// Existing 8 OFI features + ofi: OFICalculator, + + /// Running state for 12 new features (all O(1) per tick) + ofi_trajectory: OnlineLinearRegressor, // slope of OFI_L1 over time + realized_var_sum: f64, // sum of squared log returns + prev_mid: f64, // previous midpoint for returns + hawkes_intensity: f64, // exponential kernel EMA + last_trade_time_ns: u64, // for Hawkes decay + spread_min: f64, // running min/max/mean + spread_max: f64, + spread_sum: f64, + trades_at_ask: u64, // aggression counters + trades_total: u64, + bid_depletion: f64, // queue depletion EMAs + ask_depletion: f64, + prev_bid_sz_top3: u64, // for depletion rate + prev_ask_sz_top3: u64, + new_orders_top3: u64, // depth renewal counters + cancels_top3: u64, + returns_first_half: Vec, // for intra-bar momentum + returns_second_half: Vec, + ofi_prev: f64, // for acceleration + ofi_delta_ema: f64, + vpin_prev: f64, // for toxicity gradient + vpin_delta_ema: f64, + tick_count: u64, + bar_start_ns: u64, + bar_duration_ns: u64, // expected bar duration (60s) +} +``` + +**Interface:** +```rust +impl IncrementalMicrostructureCalculator { + pub fn new(bar_duration_ns: u64) -> Self; + pub fn update(&mut self, snapshot: &Mbp10Snapshot, trade: Option<&Trade>) -> (); + pub fn snapshot(&self) -> MicrostructureFeatures; // [20] f64 + pub fn reset(&mut self); // call at bar boundary +} +``` + +`update()` is O(1) — no allocations, no loops over history. Called once per MBP-10 event. +`snapshot()` finalizes the 20 features. Called at bar close. +`reset()` clears running state for the next bar. Called after snapshot. + +--- + +## fxcache Format Change + +Single format. OFI_DIM: 8→20. **All existing fxcache files must be regenerated.** + +``` +FxCacheHeader (64 bytes): + magic [u8; 8] = b"FXCACHE\0" + version u16 = 1 + feat_dim u16 = 42 + target_dim u16 = 4 + ofi_dim u16 = 20 ← was 8 + bar_count u64 + cache_key [u8; 32] + reserved [u8; 8] + +Record: [i64 ts][66 × f32] = 272 bytes/bar ← was 224 + 66 = 42 features + 4 targets + 20 OFI +``` + +Constants change in `crates/ml/src/fxcache.rs`: +```rust +const OFI_DIM: usize = 20; // was 8 +const RECORD_F64_COUNT: usize = 66; // was 54 (42+4+20) +``` + +--- + +## Precompute Pipeline + +Extend `crates/ml/examples/precompute_features.rs`: + +``` +For each minute bar [bar_start, bar_end): + 1. Collect Mbp10Snapshots from DBN files within time range + 2. Collect Trades from trades DBN files within time range + 3. Feed into IncrementalMicrostructureCalculator.update() per tick + 4. calculator.snapshot() → [20] features + 5. Write to fxcache record alongside 42 features + 4 targets +``` + +The existing OFI calculation path in `precompute_features.rs` already loads MBP-10 snapshots per bar and calls `OFICalculator`. The change: replace `OFICalculator` with `IncrementalMicrostructureCalculator` which computes all 20 features (including the existing 8). + +**Precompute time:** 4M bars × ~2000 ticks/bar = 8B ticks × <1μs = ~8 seconds. Negligible. + +--- + +## Model Integration + +The DQN's feature input grows by 12 dimensions: + +| Component | Change | +|-----------|--------| +| `fxcache.rs` | OFI_DIM: 8→20, RECORD_F64_COUNT: 54→66 | +| `precompute_features.rs` | Use IncrementalMicrostructureCalculator | +| Experience collector (`gpu_experience_collector.rs`) | Load 20 OFI features per bar (was 8) | +| DQN config | state_dim or OFI concat dimension grows by 12 | +| Trunk w_s1 | Input dimension grows by 12 (via state_dim config) | +| ISV feature gating (Phase 3) | Controls which of 20 microstructure features matter per regime | + +The additional 12 dimensions are handled automatically by the existing bottleneck/VSN architecture — the trunk learns which features matter. ISV feature gating (when implemented) provides regime-adaptive control. + +--- + +## Inference: H100 Between-Bar Compute + +### Databento Live → Incremental Calculator + +``` +Databento Live (GLBX.MDP3, mbp-10, ES) + │ + │ databento Rust crate → Mbp10Snapshot per event + │ 6.1μs median latency + │ + ▼ +IncrementalMicrostructureCalculator.update() + │ <1μs per tick, O(1), no allocation + │ + ▼ At bar close: +calculator.snapshot() → microstructure_state [20] + │ + ▼ +DQN forward pass on H100 → action +``` + +### Between-Bar Speculative Compute (NOVEL N10) + +The H100 sits idle 99.9998% of inference time. Use it: + +**Every 5 seconds (12× per bar):** + +1. **Snapshot** intermediate microstructure_state [20] +2. **ISV signal update** → micro-regime detection (regime_velocity, stability) +3. **Speculative trunk forward** → pre-compute h_s2 from intermediate features +4. **Cache** the speculative h_s2 + ISV state + +**At bar close:** +- If actual features within 0.5σ of speculative: **use cached h_s2, run branch heads only** (~0.01ms) +- If features deviate significantly: **discard cache, full forward** (~0.1ms) +- Free option: helps when right, costs nothing when wrong + +### Mid-Bar Regime Shift Early Warning (NOVEL N11) + +With 12 regime assessments per bar (every 5s): + +1. `regime_velocity` spikes mid-bar → regime shift detected 30s before bar close +2. ISV automatically reduces `risk_budget` via risk branch +3. Speculative "escape action" (direction=Flat) pre-computed and cached +4. At bar close: if shift confirmed → execute escape instantly. False alarm → proceed normally. + +30-second early warning that no bar-level model can match. + +--- + +## Inference Service Architecture + +New component in `services/trading-agent/` (or extend existing): + +```rust +struct TickProcessor { + calculator: IncrementalMicrostructureCalculator, + databento_client: databento::LiveClient, + h100_channel: tokio::sync::mpsc::Sender, +} + +impl TickProcessor { + /// Subscribe to Databento live MBP-10 for ES + async fn run(&mut self) { + let subscription = self.databento_client + .subscribe("GLBX.MDP3", "mbp-10", &["ES"]) + .await; + + while let Some(event) = subscription.next().await { + self.calculator.update(&event.snapshot, event.trade.as_ref()); + + // Every 5s: send intermediate snapshot to H100 for speculative compute + if self.calculator.tick_count % SPECULATIVE_INTERVAL == 0 { + let snapshot = self.calculator.snapshot(); + self.h100_channel.send(snapshot).await; + } + } + } + + /// Called at bar close — returns final features + fn bar_close(&mut self) -> MicrostructureFeatures { + let features = self.calculator.snapshot(); + self.calculator.reset(); + features + } +} +``` + +--- + +## Implementation Order + +1. `IncrementalMicrostructureCalculator` — 12 new features in `ofi_calculator.rs` +2. `fxcache.rs` — OFI_DIM 8→20, record size 224→272 +3. `precompute_features.rs` — use new calculator, output 20 OFI +4. Regenerate all fxcache files on PVC +5. Experience collector — load 20 OFI, widen state buffers +6. DQN config — state_dim grows by 12 +7. Trunk w_s1 resize — automatic via config +8. Databento live integration — TickProcessor service +9. H100 between-bar speculative compute +10. Smoke test + verification + +--- + +## Risks + +| Risk | Mitigation | +|------|-----------| +| Existing fxcache files incompatible | Regenerate all. Precompute is ~8 seconds for 4M bars. | +| 12 new features → overfitting on training data | ISV feature gating suppresses irrelevant features. 12 features / 4M bars = negligible risk. | +| IncrementalMicrostructureCalculator state drift | Reset at every bar boundary. Running statistics are bounded by definition. | +| Databento live feed drops/gaps | Calculator handles missing ticks gracefully (EMA continues with last value). At bar close, features reflect whatever ticks arrived. | +| Speculative trunk cache miss rate high | Free option. Cache hit → 10× faster. Cache miss → normal path. Measure hit rate and tune threshold. | +| Precompute changes bar alignment | Use same bar boundaries as existing pipeline. Ticks are assigned to bars by timestamp range. | +| Hawkes intensity numerically unstable | Clamp intensity to [0, 100]. Exponential decay with β=1.0 prevents unbounded growth. | +| Lee-Ready classification errors | Standard approach in literature. Errors are symmetric (equal chance of misclassifying buy as sell). Net effect averages out over 2000 ticks. |