spec: Tick-Level Microstructure Intelligence — 12 new features from MBP-10
OFI_DIM: 8→20. IncrementalMicrostructureCalculator: O(1) per tick. Pearls: OFI trajectory, realized variance, Hawkes trade intensity. Gems: book pressure gradient, spread dynamics, aggression ratio, queue depletion, depth renewal rate. Novels: intra-bar momentum, microstructure regime score, order flow acceleration, toxicity gradient, speculative H100 inference (N10), mid-bar regime early warning (N11). Databento (GLBX.MDP3, mbp-10) for all tick data. IBKR execution only. fxcache single format, regenerate all files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,293 @@
|
||||
# Tick-Level Microstructure Intelligence — Design Spec
|
||||
|
||||
**Goal:** Extract 12 new predictive features from raw MBP-10 tick data (1000-5000 ticks/bar) to give the DQN model sub-bar microstructure awareness. At inference, use H100 idle time between bars for speculative pre-computation and mid-bar regime detection. Feature vector grows from 50 (42+8 OFI) to 62 (42+20 OFI) dimensions.
|
||||
|
||||
**Core Principle:** Bar-level models see WHERE price went. Tick-level models see HOW it got there — the microstructure path within the bar predicts the next bar.
|
||||
|
||||
---
|
||||
|
||||
## Data Source
|
||||
|
||||
**Databento** — single source for all tick data (historical + live).
|
||||
|
||||
- Dataset: `GLBX.MDP3` (CME Globex MDP 3.0)
|
||||
- Schema: `mbp-10` (10-level order book snapshots)
|
||||
- Symbol: `ES` (E-mini S&P 500 Futures)
|
||||
- Historical: 148GB on PVC at `/mnt/training-data/futures-baseline`
|
||||
- Live: Databento Rust crate (`databento`), 6.1μs median feed latency
|
||||
- Format: `Mbp10Snapshot` — 10 `BidAskPair` levels (bid_px, bid_sz, bid_ct, ask_px, ask_sz, ask_ct)
|
||||
- Trades: from same Databento feed, Lee-Ready classification for buy/sell
|
||||
|
||||
IBKR is execution only (max 9 depth levels, subscription caps — unsuitable for tick data).
|
||||
|
||||
---
|
||||
|
||||
## The 12 New Microstructure Features
|
||||
|
||||
OFI_DIM: 8→20. Existing 8 features (OFI_L1, OFI_L5, depth_imbalance, VPIN, Kyle's Lambda, bid_slope, ask_slope, trade_imbalance) stay at indices 0-7. New features at indices 8-19.
|
||||
|
||||
All features are **incremental** — O(1) per tick via running statistics (EMAs, counters, running sums). Same computation for precompute (batch over historical ticks) and inference (streaming from Databento live).
|
||||
|
||||
### PEARLS (Cont 2014, Easley 2012)
|
||||
|
||||
| Index | Feature | Incremental State | Formula |
|
||||
|-------|---------|-------------------|---------|
|
||||
| 8 | **OFI Trajectory** | LinearRegressor (online Welford) | Slope of OFI_L1 over all ticks in bar. Front-loaded buying → continuation. Back-loaded → reversal. |
|
||||
| 9 | **Realized Variance** | Running sum of squared returns | Σ(log(mid_t/mid_{t-1})²). True intra-bar volatility — bar range is a poor proxy. |
|
||||
| 10 | **Trade Arrival Intensity** | Hawkes exponential kernel EMA | λ(t) = μ + α·Σexp(-β(t-t_i)). Self-exciting clusters. Rising intensity → momentum. |
|
||||
|
||||
### GEMS (microstructure theory)
|
||||
|
||||
| Index | Feature | Incremental State | Formula |
|
||||
|-------|---------|-------------------|---------|
|
||||
| 11 | **Book Pressure Gradient** | Weighted running sum | Σ_l(exp(-0.3l)·(bid_sz_l - ask_sz_l)) / Σ(bid_sz_l + ask_sz_l). Deep vs surface pressure across 10 levels. |
|
||||
| 12 | **Spread Dynamics** | Running min/max/mean of spread | (max_spread - min_spread) / mean_spread. Widening = market makers pulling back = volatility incoming. |
|
||||
| 13 | **Aggression Ratio** | Two counters (ask_trades, total) | trades_at_ask / total_trades (Lee-Ready). > 0.6 = strong buy pressure. |
|
||||
| 14 | **Queue Depletion Asymmetry** | Running depletion rates bid/ask | (bid_depletion - ask_depletion) / max. Reveals hidden institutional flow direction. |
|
||||
| 15 | **Depth Renewal Rate** | Counters (new_orders, cancels) at top 3 | new_placed / canceled. Market makers refreshing = confidence. Not refreshing = about to pull. |
|
||||
|
||||
### NOVELS (new combinations)
|
||||
|
||||
| Index | Feature | Incremental State | Formula |
|
||||
|-------|---------|-------------------|---------|
|
||||
| 16 | **Intra-Bar Momentum** | Two accumulators (first/second half returns) | corr(returns_first_half, returns_second_half). Positive = continuation. Negative = reversal. Detects regime WITHIN the bar. |
|
||||
| 17 | **Microstructure Regime Score** | Composite of aggression, spread, book thickness | sigmoid(aggression × spread_widen × thin_book). "Institutional sweep mode" (high) vs "market making mode" (low). |
|
||||
| 18 | **Order Flow Acceleration** | EMA of OFI deltas | d(OFI_L1)/dt — is buying accelerating or decelerating? Acceleration → breakout. Deceleration → pullback. |
|
||||
| 19 | **Toxicity Gradient** | EMA of VPIN deltas | d(VPIN)/dt. Catches the ONSET of informed flow — earlier signal than VPIN level alone. |
|
||||
|
||||
---
|
||||
|
||||
## IncrementalMicrostructureCalculator
|
||||
|
||||
New struct extending the existing `OFICalculator` in `crates/ml-features/src/ofi_calculator.rs`.
|
||||
|
||||
```rust
|
||||
pub struct IncrementalMicrostructureCalculator {
|
||||
/// Existing 8 OFI features
|
||||
ofi: OFICalculator,
|
||||
|
||||
/// Running state for 12 new features (all O(1) per tick)
|
||||
ofi_trajectory: OnlineLinearRegressor, // slope of OFI_L1 over time
|
||||
realized_var_sum: f64, // sum of squared log returns
|
||||
prev_mid: f64, // previous midpoint for returns
|
||||
hawkes_intensity: f64, // exponential kernel EMA
|
||||
last_trade_time_ns: u64, // for Hawkes decay
|
||||
spread_min: f64, // running min/max/mean
|
||||
spread_max: f64,
|
||||
spread_sum: f64,
|
||||
trades_at_ask: u64, // aggression counters
|
||||
trades_total: u64,
|
||||
bid_depletion: f64, // queue depletion EMAs
|
||||
ask_depletion: f64,
|
||||
prev_bid_sz_top3: u64, // for depletion rate
|
||||
prev_ask_sz_top3: u64,
|
||||
new_orders_top3: u64, // depth renewal counters
|
||||
cancels_top3: u64,
|
||||
returns_first_half: Vec<f64>, // for intra-bar momentum
|
||||
returns_second_half: Vec<f64>,
|
||||
ofi_prev: f64, // for acceleration
|
||||
ofi_delta_ema: f64,
|
||||
vpin_prev: f64, // for toxicity gradient
|
||||
vpin_delta_ema: f64,
|
||||
tick_count: u64,
|
||||
bar_start_ns: u64,
|
||||
bar_duration_ns: u64, // expected bar duration (60s)
|
||||
}
|
||||
```
|
||||
|
||||
**Interface:**
|
||||
```rust
|
||||
impl IncrementalMicrostructureCalculator {
|
||||
pub fn new(bar_duration_ns: u64) -> Self;
|
||||
pub fn update(&mut self, snapshot: &Mbp10Snapshot, trade: Option<&Trade>) -> ();
|
||||
pub fn snapshot(&self) -> MicrostructureFeatures; // [20] f64
|
||||
pub fn reset(&mut self); // call at bar boundary
|
||||
}
|
||||
```
|
||||
|
||||
`update()` is O(1) — no allocations, no loops over history. Called once per MBP-10 event.
|
||||
`snapshot()` finalizes the 20 features. Called at bar close.
|
||||
`reset()` clears running state for the next bar. Called after snapshot.
|
||||
|
||||
---
|
||||
|
||||
## fxcache Format Change
|
||||
|
||||
Single format. OFI_DIM: 8→20. **All existing fxcache files must be regenerated.**
|
||||
|
||||
```
|
||||
FxCacheHeader (64 bytes):
|
||||
magic [u8; 8] = b"FXCACHE\0"
|
||||
version u16 = 1
|
||||
feat_dim u16 = 42
|
||||
target_dim u16 = 4
|
||||
ofi_dim u16 = 20 ← was 8
|
||||
bar_count u64
|
||||
cache_key [u8; 32]
|
||||
reserved [u8; 8]
|
||||
|
||||
Record: [i64 ts][66 × f32] = 272 bytes/bar ← was 224
|
||||
66 = 42 features + 4 targets + 20 OFI
|
||||
```
|
||||
|
||||
Constants change in `crates/ml/src/fxcache.rs`:
|
||||
```rust
|
||||
const OFI_DIM: usize = 20; // was 8
|
||||
const RECORD_F64_COUNT: usize = 66; // was 54 (42+4+20)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Precompute Pipeline
|
||||
|
||||
Extend `crates/ml/examples/precompute_features.rs`:
|
||||
|
||||
```
|
||||
For each minute bar [bar_start, bar_end):
|
||||
1. Collect Mbp10Snapshots from DBN files within time range
|
||||
2. Collect Trades from trades DBN files within time range
|
||||
3. Feed into IncrementalMicrostructureCalculator.update() per tick
|
||||
4. calculator.snapshot() → [20] features
|
||||
5. Write to fxcache record alongside 42 features + 4 targets
|
||||
```
|
||||
|
||||
The existing OFI calculation path in `precompute_features.rs` already loads MBP-10 snapshots per bar and calls `OFICalculator`. The change: replace `OFICalculator` with `IncrementalMicrostructureCalculator` which computes all 20 features (including the existing 8).
|
||||
|
||||
**Precompute time:** 4M bars × ~2000 ticks/bar = 8B ticks × <1μs = ~8 seconds. Negligible.
|
||||
|
||||
---
|
||||
|
||||
## Model Integration
|
||||
|
||||
The DQN's feature input grows by 12 dimensions:
|
||||
|
||||
| Component | Change |
|
||||
|-----------|--------|
|
||||
| `fxcache.rs` | OFI_DIM: 8→20, RECORD_F64_COUNT: 54→66 |
|
||||
| `precompute_features.rs` | Use IncrementalMicrostructureCalculator |
|
||||
| Experience collector (`gpu_experience_collector.rs`) | Load 20 OFI features per bar (was 8) |
|
||||
| DQN config | state_dim or OFI concat dimension grows by 12 |
|
||||
| Trunk w_s1 | Input dimension grows by 12 (via state_dim config) |
|
||||
| ISV feature gating (Phase 3) | Controls which of 20 microstructure features matter per regime |
|
||||
|
||||
The additional 12 dimensions are handled automatically by the existing bottleneck/VSN architecture — the trunk learns which features matter. ISV feature gating (when implemented) provides regime-adaptive control.
|
||||
|
||||
---
|
||||
|
||||
## Inference: H100 Between-Bar Compute
|
||||
|
||||
### Databento Live → Incremental Calculator
|
||||
|
||||
```
|
||||
Databento Live (GLBX.MDP3, mbp-10, ES)
|
||||
│
|
||||
│ databento Rust crate → Mbp10Snapshot per event
|
||||
│ 6.1μs median latency
|
||||
│
|
||||
▼
|
||||
IncrementalMicrostructureCalculator.update()
|
||||
│ <1μs per tick, O(1), no allocation
|
||||
│
|
||||
▼ At bar close:
|
||||
calculator.snapshot() → microstructure_state [20]
|
||||
│
|
||||
▼
|
||||
DQN forward pass on H100 → action
|
||||
```
|
||||
|
||||
### Between-Bar Speculative Compute (NOVEL N10)
|
||||
|
||||
The H100 sits idle 99.9998% of inference time. Use it:
|
||||
|
||||
**Every 5 seconds (12× per bar):**
|
||||
|
||||
1. **Snapshot** intermediate microstructure_state [20]
|
||||
2. **ISV signal update** → micro-regime detection (regime_velocity, stability)
|
||||
3. **Speculative trunk forward** → pre-compute h_s2 from intermediate features
|
||||
4. **Cache** the speculative h_s2 + ISV state
|
||||
|
||||
**At bar close:**
|
||||
- If actual features within 0.5σ of speculative: **use cached h_s2, run branch heads only** (~0.01ms)
|
||||
- If features deviate significantly: **discard cache, full forward** (~0.1ms)
|
||||
- Free option: helps when right, costs nothing when wrong
|
||||
|
||||
### Mid-Bar Regime Shift Early Warning (NOVEL N11)
|
||||
|
||||
With 12 regime assessments per bar (every 5s):
|
||||
|
||||
1. `regime_velocity` spikes mid-bar → regime shift detected 30s before bar close
|
||||
2. ISV automatically reduces `risk_budget` via risk branch
|
||||
3. Speculative "escape action" (direction=Flat) pre-computed and cached
|
||||
4. At bar close: if shift confirmed → execute escape instantly. False alarm → proceed normally.
|
||||
|
||||
30-second early warning that no bar-level model can match.
|
||||
|
||||
---
|
||||
|
||||
## Inference Service Architecture
|
||||
|
||||
New component in `services/trading-agent/` (or extend existing):
|
||||
|
||||
```rust
|
||||
struct TickProcessor {
|
||||
calculator: IncrementalMicrostructureCalculator,
|
||||
databento_client: databento::LiveClient,
|
||||
h100_channel: tokio::sync::mpsc::Sender<MicrostructureSnapshot>,
|
||||
}
|
||||
|
||||
impl TickProcessor {
|
||||
/// Subscribe to Databento live MBP-10 for ES
|
||||
async fn run(&mut self) {
|
||||
let subscription = self.databento_client
|
||||
.subscribe("GLBX.MDP3", "mbp-10", &["ES"])
|
||||
.await;
|
||||
|
||||
while let Some(event) = subscription.next().await {
|
||||
self.calculator.update(&event.snapshot, event.trade.as_ref());
|
||||
|
||||
// Every 5s: send intermediate snapshot to H100 for speculative compute
|
||||
if self.calculator.tick_count % SPECULATIVE_INTERVAL == 0 {
|
||||
let snapshot = self.calculator.snapshot();
|
||||
self.h100_channel.send(snapshot).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Called at bar close — returns final features
|
||||
fn bar_close(&mut self) -> MicrostructureFeatures {
|
||||
let features = self.calculator.snapshot();
|
||||
self.calculator.reset();
|
||||
features
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. `IncrementalMicrostructureCalculator` — 12 new features in `ofi_calculator.rs`
|
||||
2. `fxcache.rs` — OFI_DIM 8→20, record size 224→272
|
||||
3. `precompute_features.rs` — use new calculator, output 20 OFI
|
||||
4. Regenerate all fxcache files on PVC
|
||||
5. Experience collector — load 20 OFI, widen state buffers
|
||||
6. DQN config — state_dim grows by 12
|
||||
7. Trunk w_s1 resize — automatic via config
|
||||
8. Databento live integration — TickProcessor service
|
||||
9. H100 between-bar speculative compute
|
||||
10. Smoke test + verification
|
||||
|
||||
---
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Existing fxcache files incompatible | Regenerate all. Precompute is ~8 seconds for 4M bars. |
|
||||
| 12 new features → overfitting on training data | ISV feature gating suppresses irrelevant features. 12 features / 4M bars = negligible risk. |
|
||||
| IncrementalMicrostructureCalculator state drift | Reset at every bar boundary. Running statistics are bounded by definition. |
|
||||
| Databento live feed drops/gaps | Calculator handles missing ticks gracefully (EMA continues with last value). At bar close, features reflect whatever ticks arrived. |
|
||||
| Speculative trunk cache miss rate high | Free option. Cache hit → 10× faster. Cache miss → normal path. Measure hit rate and tune threshold. |
|
||||
| Precompute changes bar alignment | Use same bar boundaries as existing pipeline. Ticks are assigned to bars by timestamp range. |
|
||||
| Hawkes intensity numerically unstable | Clamp intensity to [0, 100]. Exponential decay with β=1.0 prevents unbounded growth. |
|
||||
| Lee-Ready classification errors | Standard approach in literature. Errors are symmetric (equal chance of misclassifying buy as sell). Net effect averages out over 2000 ticks. |
|
||||
Reference in New Issue
Block a user