foxhunt/docs/superpowers/specs/2026-04-16-tick-microstructure-intelligence-design.md

Tick-Level Microstructure Intelligence — Design Spec

Goal: Extract 12 new predictive features from raw MBP-10 tick data (1000-5000 ticks/bar) to give the DQN model sub-bar microstructure awareness. At inference, use H100 idle time between bars for speculative pre-computation and mid-bar regime detection. Feature vector grows from 50 (42+8 OFI) to 62 (42+20 OFI) dimensions.

Core Principle: Bar-level models see WHERE price went. Tick-level models see HOW it got there — the microstructure path within the bar predicts the next bar.

---

Data Source

Databento — single source for all tick data (historical + live).

• Dataset: GLBX.MDP3 (CME Globex MDP 3.0)
• Schema: mbp-10 (10-level order book snapshots)
• Symbol: ES (E-mini S&P 500 Futures)
• Historical: 148GB on PVC at /mnt/training-data/futures-baseline
• Live: Databento Rust crate (databento), 6.1μs median feed latency
• Format: Mbp10Snapshot — 10 BidAskPair levels (bid_px, bid_sz, bid_ct, ask_px, ask_sz, ask_ct)
• Trades: from same Databento feed, Lee-Ready classification for buy/sell

IBKR is execution only (max 9 depth levels, subscription caps — unsuitable for tick data).

---

The 12 New Microstructure Features

OFI_DIM: 8→20. Existing 8 features (OFI_L1, OFI_L5, depth_imbalance, VPIN, Kyle's Lambda, bid_slope, ask_slope, trade_imbalance) stay at indices 0-7. New features at indices 8-19.

All features are incremental — O(1) per tick via running statistics (EMAs, counters, running sums). Same computation for precompute (batch over historical ticks) and inference (streaming from Databento live).

PEARLS (Cont 2014, Easley 2012)

Index	Feature	Incremental State	Formula
8	OFI Trajectory	LinearRegressor (online Welford)	Slope of OFI_L1 over all ticks in bar. Front-loaded buying → continuation. Back-loaded → reversal.
9	Realized Variance	Running sum of squared returns	Σ(log(mid_t/mid_{t-1})²). True intra-bar volatility — bar range is a poor proxy.
10	Trade Arrival Intensity	Hawkes exponential kernel EMA	λ(t) = μ + α·Σexp(-β(t-t_i)). Self-exciting clusters. Rising intensity → momentum.

GEMS (microstructure theory)

Index	Feature	Incremental State	Formula
11	Book Pressure Gradient	Weighted running sum	Σ_l(exp(-0.3l)·(bid_sz_l - ask_sz_l)) / Σ(bid_sz_l + ask_sz_l). Deep vs surface pressure across 10 levels.
12	Spread Dynamics	Running min/max/mean of spread	(max_spread - min_spread) / mean_spread. Widening = market makers pulling back = volatility incoming.
13	Aggression Ratio	Two counters (ask_trades, total)	trades_at_ask / total_trades (Lee-Ready). > 0.6 = strong buy pressure.
14	Queue Depletion Asymmetry	Running depletion rates bid/ask	(bid_depletion - ask_depletion) / max. Reveals hidden institutional flow direction.
15	Order Count Flux	Running delta of bid_ct/ask_ct at top 3	Δbid_ct>0 → new order proxy, Δbid_ct<0 → cancel/fill proxy. Approximation: MBP-10 snapshots can't distinguish cancel from fill. Treat as market maker activity volatility — high flux = active refresh, low flux = about to pull.

NOVELS (new combinations)

Index	Feature	Incremental State	Formula
16	Intra-Bar Momentum	Two accumulators (first/second half returns)	corr(returns_first_half, returns_second_half). Positive = continuation. Negative = reversal. Detects regime WITHIN the bar.
17	Microstructure Regime Score	Composite of aggression, spread, book thickness	sigmoid(aggression × spread_widen × thin_book). "Institutional sweep mode" (high) vs "market making mode" (low).
18	Order Flow Acceleration	EMA of OFI deltas	d(OFI_L1)/dt — is buying accelerating or decelerating? Acceleration → breakout. Deceleration → pullback.
19	Toxicity Gradient	EMA of VPIN deltas	d(VPIN)/dt. Catches the ONSET of informed flow — earlier signal than VPIN level alone.

---

IncrementalMicrostructureCalculator

New struct extending the existing OFICalculator in crates/ml-features/src/ofi_calculator.rs.

pub struct IncrementalMicrostructureCalculator {
    /// Existing 8 OFI features
    ofi: OFICalculator,
    
    /// Running state for 12 new features (all O(1) per tick)
    ofi_trajectory: OnlineLinearRegressor,  // slope of OFI_L1 over time
    realized_var_sum: f64,                   // sum of squared log returns
    prev_mid: f64,                           // previous midpoint for returns
    hawkes_intensity: f64,                   // exponential kernel EMA
    last_trade_time_ns: u64,                 // for Hawkes decay
    spread_min: f64,                         // running min/max/mean
    spread_max: f64,
    spread_sum: f64,
    trades_at_ask: u64,                      // aggression counters
    trades_total: u64,
    bid_depletion: f64,                      // queue depletion EMAs
    ask_depletion: f64,
    prev_bid_sz_top3: u64,                   // for depletion rate
    prev_ask_sz_top3: u64,
    new_orders_top3: u64,                    // depth renewal counters
    cancels_top3: u64,
    // Intra-bar momentum: Welford online covariance (O(1) per tick, no allocation)
    half1_sum_x: f64, half1_sum_x2: f64, half1_n: u64,
    half2_sum_x: f64, half2_sum_x2: f64, half2_n: u64,
    cross_sum_xy: f64,  // for correlation between halves
    ofi_prev: f64,                           // for acceleration
    ofi_delta_ema: f64,
    vpin_prev: f64,                          // for toxicity gradient
    vpin_delta_ema: f64,
    tick_count: u64,
    bar_start_ns: u64,
    bar_duration_ns: u64,                    // expected bar duration (60s)
}

Interface:

impl IncrementalMicrostructureCalculator {
    pub fn new(bar_duration_ns: u64) -> Self;
    pub fn update(&mut self, snapshot: &Mbp10Snapshot, trade: Option<&Trade>) -> ();
    pub fn snapshot(&self) -> MicrostructureFeatures;  // [20] f64 — pure read, non-mutating, idempotent
    pub fn reset(&mut self);  // call at bar boundary — clears all running state
}

update() is O(1) — no allocations, no loops over history. Called once per MBP-10 event. snapshot() finalizes the 20 features. Called at bar close. reset() clears running state for the next bar. Called after snapshot.

---

fxcache Format Change

Single format. OFI_DIM: 8→20. All existing fxcache files must be regenerated.

FxCacheHeader (64 bytes):
  magic       [u8; 8]   = b"FXCACHE\0"
  version     u16       = 1
  feat_dim    u16       = 42
  target_dim  u16       = 4
  ofi_dim     u16       = 20          ← was 8
  bar_count   u64
  cache_key   [u8; 32]
  reserved    [u8; 8]

Record: [i64 ts][66 × f32] = 272 bytes/bar   ← was 224
  66 = 42 features + 4 targets + 20 OFI

Constants change in crates/ml/src/fxcache.rs:

const OFI_DIM: usize = 20;              // was 8
const RECORD_F64_COUNT: usize = 66;      // was 54 (42+4+20)

CRITICAL: OFI_DIM=8 is load-bearing in 6+ files. All must be updated atomically:

File	Location	What to change
fxcache.rs:124	validate()	Reads ofi_dim from header — will reject old files. Code and data deploy atomically.
fxcache.rs:195,377	Vec<[f64; OFI_DIM]>	Compile-time array generic — all callers cascade.
precompute_features.rs:329,342,434	[f64; 8] literals	Change to [f64; OFI_DIM] or use fxcache::OFI_DIM.
gpu_experience_collector.rs:747	let ofi_dim = if ... { 8 }	Hardcoded detection. Must use OFI_DIM constant.
cuda_pipeline/mod.rs:402,477,516	if ofi { 8 } else { 0 }	Same hardcoded literal.
constructor.rs:52-53,255	raw_state_dim = 74 (42+8+16+8)	Must become 42+8+16+OFI_DIM. Aligned: (86+7)&!7 = 88.
hyperopt/adapters/dqn.rs:1437	Literal 8	Use OFI_DIM.
Test files	[0.0; 8], vec![[0.0; 8]; n]	All OFI fixtures must use OFI_DIM.

Deploy rule: Code and fxcache data must deploy atomically. New code with OFI_DIM=20 will reject old fxcache (ofi_dim=8 in header). Regenerate fxcache BEFORE deploying new code, or deploy both in the same Argo workflow step.

---

Precompute Pipeline

Extend crates/ml/examples/precompute_features.rs:

For each minute bar [bar_start, bar_end):
  1. Collect Mbp10Snapshots from DBN files within time range
  2. Collect Trades from trades DBN files within time range
  3. Feed into IncrementalMicrostructureCalculator.update() per tick
  4. calculator.snapshot() → [20] features
  5. Write to fxcache record alongside 42 features + 4 targets

The existing OFI calculation path in precompute_features.rs already loads MBP-10 snapshots per bar and calls OFICalculator. The change: replace OFICalculator with IncrementalMicrostructureCalculator which computes all 20 features (including the existing 8).

Precompute time: Calculator CPU time is ~8 seconds (8B ticks × <1μs). But disk I/O dominates — 148GB MBP-10 at ~500MB/s = ~5 minutes read time. Total precompute: ~5-10 minutes depending on I/O. Set Argo workflow timeout accordingly.

---

Model Integration

The DQN's feature input grows by 12 dimensions:

Component	Change
fxcache.rs	OFI_DIM: 8→20, RECORD_F64_COUNT: 54→66
precompute_features.rs	Use IncrementalMicrostructureCalculator
Experience collector (gpu_experience_collector.rs)	Load 20 OFI features per bar (was 8)
DQN config	state_dim or OFI concat dimension grows by 12
Trunk w_s1	Input dimension grows by 12 (via state_dim config)
ISV feature gating (Phase 3)	Controls which of 20 microstructure features matter per regime

The additional 12 dimensions are handled automatically by the existing bottleneck/VSN architecture — the trunk learns which features matter. ISV feature gating (when implemented) provides regime-adaptive control.

---

Inference: H100 Between-Bar Compute

Databento Live → Incremental Calculator

Databento Live (GLBX.MDP3, mbp-10, ES)
    │
    │ databento Rust crate → Mbp10Snapshot per event
    │ 6.1μs median latency
    │
    ▼
IncrementalMicrostructureCalculator.update()
    │ <1μs per tick, O(1), no allocation
    │
    ▼ At bar close:
calculator.snapshot() → microstructure_state [20]
    │
    ▼
DQN forward pass on H100 → action

Between-Bar Speculative Compute (NOVEL N10)

The H100 sits idle 99.9998% of inference time. Use it:

Every 5 seconds (12× per bar):

1. Snapshot intermediate microstructure_state [20]
2. ISV signal update → micro-regime detection (regime_velocity, stability)
3. Speculative trunk forward → pre-compute h_s2 from intermediate features
4. Cache the speculative h_s2 + ISV state

At bar close:

• If actual features within 0.5σ of speculative: use cached h_s2, run branch heads only (~0.01ms)
• If features deviate significantly: discard cache, full forward (~0.1ms)
• Free option: helps when right, costs nothing when wrong

Mid-Bar Regime Shift Early Warning (NOVEL N11)

With 12 regime assessments per bar (every 5s):

1. regime_velocity spikes mid-bar → regime shift detected 30s before bar close
2. ISV automatically reduces risk_budget via risk branch
3. Speculative "escape action" (direction=Flat) pre-computed and cached
4. At bar close: if shift confirmed → execute escape instantly. False alarm → proceed normally.

30-second early warning that no bar-level model can match.

---

Inference Service Architecture

New component in services/trading-agent/ (or extend existing):

struct TickProcessor {
    calculator: IncrementalMicrostructureCalculator,
    databento_client: databento::LiveClient,
    h100_channel: tokio::sync::mpsc::Sender<MicrostructureSnapshot>,
}

impl TickProcessor {
    /// Subscribe to Databento live MBP-10 for ES
    async fn run(&mut self) {
        let subscription = self.databento_client
            .subscribe("GLBX.MDP3", "mbp-10", &["ES"])
            .await;
        
        while let Some(event) = subscription.next().await {
            self.calculator.update(&event.snapshot, event.trade.as_ref());
            
            // Every 5s: send intermediate snapshot to H100 for speculative compute
            if self.calculator.tick_count % SPECULATIVE_INTERVAL == 0 {
                let snapshot = self.calculator.snapshot();
                self.h100_channel.send(snapshot).await;
            }
        }
    }
    
    /// Called at bar close — returns final features
    fn bar_close(&mut self) -> MicrostructureFeatures {
        let features = self.calculator.snapshot();
        self.calculator.reset();
        features
    }
}

---

Implementation Order

1. IncrementalMicrostructureCalculator — 12 new features in ofi_calculator.rs
2. fxcache.rs — OFI_DIM 8→20, record size 224→272
3. precompute_features.rs — use new calculator, output 20 OFI
4. Regenerate all fxcache files on PVC
5. Experience collector — load 20 OFI, widen state buffers
6. DQN config — state_dim grows by 12
7. Trunk w_s1 resize — automatic via config
8. Databento live integration — TickProcessor service
9. H100 between-bar speculative compute
10. Smoke test + verification

---

Risks

Risk	Mitigation
Existing fxcache files incompatible	Regenerate all (~5-10 min). Code and data deploy atomically — new code rejects old fxcache.
OFI_DIM=8 hardcoded in 6+ files	All sites enumerated in spec. Must update atomically in single commit. Use shared constant, not literals.
12 new features → overfitting on training data	ISV feature gating suppresses irrelevant features. 12 features / 4M bars = negligible risk.
IncrementalMicrostructureCalculator state drift	Reset at every bar boundary. Running statistics are bounded by definition.
Databento live feed drops/gaps	Calculator handles missing ticks gracefully (EMA continues with last value). At bar close, features reflect whatever ticks arrived.
Speculative trunk cache miss rate high	Free option. Cache hit → 10× faster. Cache miss → normal path. Measure hit rate and tune threshold.
Precompute changes bar alignment	Use same bar boundaries as existing pipeline. Ticks are assigned to bars by timestamp range.
Hawkes intensity numerically unstable	Clamp intensity to [0, 100]. Exponential decay with β=1.0 prevents unbounded growth.
Lee-Ready classification errors	Standard approach in literature. Errors are symmetric (equal chance of misclassifying buy as sell). Net effect averages out over 2000 ticks.