foxhunt/docs/superpowers/specs/2026-04-16-tick-microstructure-intelligence-design.md

# Tick-Level Microstructure Intelligence — Design Spec

**Goal:** Extract 12 new predictive features from raw MBP-10 tick data (1000-5000 ticks/bar) to give the DQN model sub-bar microstructure awareness. At inference, use H100 idle time between bars for speculative pre-computation and mid-bar regime detection. Feature vector grows from 50 (42+8 OFI) to 62 (42+20 OFI) dimensions.

**Core Principle:** Bar-level models see WHERE price went. Tick-level models see HOW it got there — the microstructure path within the bar predicts the next bar.

---

## Data Source

**Databento** — single source for all tick data (historical + live).

- Dataset: `GLBX.MDP3` (CME Globex MDP 3.0)
- Schema: `mbp-10` (10-level order book snapshots)
- Symbol: `ES` (E-mini S&P 500 Futures)
- Historical: 148GB on PVC at `/mnt/training-data/futures-baseline`
- Live: Databento Rust crate (`databento`), 6.1μs median feed latency
- Format: `Mbp10Snapshot` — 10 `BidAskPair` levels (bid_px, bid_sz, bid_ct, ask_px, ask_sz, ask_ct)
- Trades: from same Databento feed, Lee-Ready classification for buy/sell

IBKR is execution only (max 9 depth levels, subscription caps — unsuitable for tick data).

---

## The 12 New Microstructure Features

OFI_DIM: 8→20. Existing 8 features (OFI_L1, OFI_L5, depth_imbalance, VPIN, Kyle's Lambda, bid_slope, ask_slope, trade_imbalance) stay at indices 0-7. New features at indices 8-19.

All features are **incremental** — O(1) per tick via running statistics (EMAs, counters, running sums). Same computation for precompute (batch over historical ticks) and inference (streaming from Databento live).

### PEARLS (Cont 2014, Easley 2012)

| Index | Feature | Incremental State | Formula |
|-------|---------|-------------------|---------|
| 8 | **OFI Trajectory** | LinearRegressor (online Welford) | Slope of OFI_L1 over all ticks in bar. Front-loaded buying → continuation. Back-loaded → reversal. |
| 9 | **Realized Variance** | Running sum of squared returns | Σ(log(mid_t/mid_{t-1})²). True intra-bar volatility — bar range is a poor proxy. |
| 10 | **Trade Arrival Intensity** | Hawkes exponential kernel EMA | λ(t) = μ + α·Σexp(-β(t-t_i)). Self-exciting clusters. Rising intensity → momentum. |

### GEMS (microstructure theory)

| Index | Feature | Incremental State | Formula |
|-------|---------|-------------------|---------|
| 11 | **Book Pressure Gradient** | Weighted running sum | Σ_l(exp(-0.3l)·(bid_sz_l - ask_sz_l)) / Σ(bid_sz_l + ask_sz_l). Deep vs surface pressure across 10 levels. |
| 12 | **Spread Dynamics** | Running min/max/mean of spread | (max_spread - min_spread) / mean_spread. Widening = market makers pulling back = volatility incoming. |
| 13 | **Aggression Ratio** | Two counters (ask_trades, total) | trades_at_ask / total_trades (Lee-Ready). > 0.6 = strong buy pressure. |
| 14 | **Queue Depletion Asymmetry** | Running depletion rates bid/ask | (bid_depletion - ask_depletion) / max. Reveals hidden institutional flow direction. |
| 15 | **Order Count Flux** | Running delta of bid_ct/ask_ct at top 3 | Δbid_ct>0 → new order proxy, Δbid_ct<0 → cancel/fill proxy. Approximation: MBP-10 snapshots can't distinguish cancel from fill. Treat as market maker activity volatility — high flux = active refresh, low flux = about to pull. |

### NOVELS (new combinations)

| Index | Feature | Incremental State | Formula |
|-------|---------|-------------------|---------|
| 16 | **Intra-Bar Momentum** | Two accumulators (first/second half returns) | corr(returns_first_half, returns_second_half). Positive = continuation. Negative = reversal. Detects regime WITHIN the bar. |
| 17 | **Microstructure Regime Score** | Composite of aggression, spread, book thickness | sigmoid(aggression × spread_widen × thin_book). "Institutional sweep mode" (high) vs "market making mode" (low). |
| 18 | **Order Flow Acceleration** | EMA of OFI deltas | d(OFI_L1)/dt — is buying accelerating or decelerating? Acceleration → breakout. Deceleration → pullback. |
| 19 | **Toxicity Gradient** | EMA of VPIN deltas | d(VPIN)/dt. Catches the ONSET of informed flow — earlier signal than VPIN level alone. |

---

## IncrementalMicrostructureCalculator

New struct extending the existing `OFICalculator` in `crates/ml-features/src/ofi_calculator.rs`.

```rust
pub struct IncrementalMicrostructureCalculator {
    /// Existing 8 OFI features
    ofi: OFICalculator,
    
    /// Running state for 12 new features (all O(1) per tick)
    ofi_trajectory: OnlineLinearRegressor,  // slope of OFI_L1 over time
    realized_var_sum: f64,                   // sum of squared log returns
    prev_mid: f64,                           // previous midpoint for returns
    hawkes_intensity: f64,                   // exponential kernel EMA
    last_trade_time_ns: u64,                 // for Hawkes decay
    spread_min: f64,                         // running min/max/mean
    spread_max: f64,
    spread_sum: f64,
    trades_at_ask: u64,                      // aggression counters
    trades_total: u64,
    bid_depletion: f64,                      // queue depletion EMAs
    ask_depletion: f64,
    prev_bid_sz_top3: u64,                   // for depletion rate
    prev_ask_sz_top3: u64,
    new_orders_top3: u64,                    // depth renewal counters
    cancels_top3: u64,
    // Intra-bar momentum: Welford online covariance (O(1) per tick, no allocation)
    half1_sum_x: f64, half1_sum_x2: f64, half1_n: u64,
    half2_sum_x: f64, half2_sum_x2: f64, half2_n: u64,
    cross_sum_xy: f64,  // for correlation between halves
    ofi_prev: f64,                           // for acceleration
    ofi_delta_ema: f64,
    vpin_prev: f64,                          // for toxicity gradient
    vpin_delta_ema: f64,
    tick_count: u64,
    bar_start_ns: u64,
    bar_duration_ns: u64,                    // expected bar duration (60s)
}
```

**Interface:**
```rust
impl IncrementalMicrostructureCalculator {
    pub fn new(bar_duration_ns: u64) -> Self;
    pub fn update(&mut self, snapshot: &Mbp10Snapshot, trade: Option<&Trade>) -> ();
    pub fn snapshot(&self) -> MicrostructureFeatures;  // [20] f64 — pure read, non-mutating, idempotent
    pub fn reset(&mut self);  // call at bar boundary — clears all running state
}
```

`update()` is O(1) — no allocations, no loops over history. Called once per MBP-10 event.
`snapshot()` finalizes the 20 features. Called at bar close.
`reset()` clears running state for the next bar. Called after snapshot.

---

## fxcache Format Change

Single format. OFI_DIM: 8→20. **All existing fxcache files must be regenerated.**

```
FxCacheHeader (64 bytes):
  magic       [u8; 8]   = b"FXCACHE\0"
  version     u16       = 1
  feat_dim    u16       = 42
  target_dim  u16       = 4
  ofi_dim     u16       = 20          ← was 8
  bar_count   u64
  cache_key   [u8; 32]
  reserved    [u8; 8]

Record: [i64 ts][66 × f32] = 272 bytes/bar   ← was 224
  66 = 42 features + 4 targets + 20 OFI
```

Constants change in `crates/ml/src/fxcache.rs`:
```rust
const OFI_DIM: usize = 20;              // was 8
const RECORD_F64_COUNT: usize = 66;      // was 54 (42+4+20)
```

**CRITICAL: OFI_DIM=8 is load-bearing in 6+ files.** All must be updated atomically:

| File | Location | What to change |
|------|----------|----------------|
| `fxcache.rs:124` | `validate()` | Reads `ofi_dim` from header — will reject old files. Code and data deploy atomically. |
| `fxcache.rs:195,377` | `Vec<[f64; OFI_DIM]>` | Compile-time array generic — all callers cascade. |
| `precompute_features.rs:329,342,434` | `[f64; 8]` literals | Change to `[f64; OFI_DIM]` or use `fxcache::OFI_DIM`. |
| `gpu_experience_collector.rs:747` | `let ofi_dim = if ... { 8 }` | Hardcoded detection. Must use `OFI_DIM` constant. |
| `cuda_pipeline/mod.rs:402,477,516` | `if ofi { 8 } else { 0 }` | Same hardcoded literal. |
| `constructor.rs:52-53,255` | `raw_state_dim = 74` (42+8+16+8) | Must become `42+8+16+OFI_DIM`. Aligned: `(86+7)&!7 = 88`. |
| `hyperopt/adapters/dqn.rs:1437` | Literal `8` | Use `OFI_DIM`. |
| Test files | `[0.0; 8]`, `vec![[0.0; 8]; n]` | All OFI fixtures must use `OFI_DIM`. |

**Deploy rule:** Code and fxcache data must deploy atomically. New code with OFI_DIM=20 will reject old fxcache (ofi_dim=8 in header). Regenerate fxcache BEFORE deploying new code, or deploy both in the same Argo workflow step.

---

## Precompute Pipeline

Extend `crates/ml/examples/precompute_features.rs`:

```
For each minute bar [bar_start, bar_end):
  1. Collect Mbp10Snapshots from DBN files within time range
  2. Collect Trades from trades DBN files within time range
  3. Feed into IncrementalMicrostructureCalculator.update() per tick
  4. calculator.snapshot() → [20] features
  5. Write to fxcache record alongside 42 features + 4 targets
```

The existing OFI calculation path in `precompute_features.rs` already loads MBP-10 snapshots per bar and calls `OFICalculator`. The change: replace `OFICalculator` with `IncrementalMicrostructureCalculator` which computes all 20 features (including the existing 8).

**Precompute time:** Calculator CPU time is ~8 seconds (8B ticks × <1μs). But disk I/O dominates — 148GB MBP-10 at ~500MB/s = ~5 minutes read time. Total precompute: **~5-10 minutes** depending on I/O. Set Argo workflow timeout accordingly.

---

## Model Integration

The DQN's feature input grows by 12 dimensions:

| Component | Change |
|-----------|--------|
| `fxcache.rs` | OFI_DIM: 8→20, RECORD_F64_COUNT: 54→66 |
| `precompute_features.rs` | Use IncrementalMicrostructureCalculator |
| Experience collector (`gpu_experience_collector.rs`) | Load 20 OFI features per bar (was 8) |
| DQN config | state_dim or OFI concat dimension grows by 12 |
| Trunk w_s1 | Input dimension grows by 12 (via state_dim config) |
| ISV feature gating (Phase 3) | Controls which of 20 microstructure features matter per regime |

The additional 12 dimensions are handled automatically by the existing bottleneck/VSN architecture — the trunk learns which features matter. ISV feature gating (when implemented) provides regime-adaptive control.

---

## Inference: H100 Between-Bar Compute

### Databento Live → Incremental Calculator

```
Databento Live (GLBX.MDP3, mbp-10, ES)
    │
    │ databento Rust crate → Mbp10Snapshot per event
    │ 6.1μs median latency
    │
    ▼
IncrementalMicrostructureCalculator.update()
    │ <1μs per tick, O(1), no allocation
    │
    ▼ At bar close:
calculator.snapshot() → microstructure_state [20]
    │
    ▼
DQN forward pass on H100 → action
```

### Between-Bar Speculative Compute (NOVEL N10)

The H100 sits idle 99.9998% of inference time. Use it:

**Every 5 seconds (12× per bar):**

1. **Snapshot** intermediate microstructure_state [20]
2. **ISV signal update** → micro-regime detection (regime_velocity, stability)
3. **Speculative trunk forward** → pre-compute h_s2 from intermediate features
4. **Cache** the speculative h_s2 + ISV state

**At bar close:**
- If actual features within 0.5σ of speculative: **use cached h_s2, run branch heads only** (~0.01ms)
- If features deviate significantly: **discard cache, full forward** (~0.1ms)
- Free option: helps when right, costs nothing when wrong

### Mid-Bar Regime Shift Early Warning (NOVEL N11)

With 12 regime assessments per bar (every 5s):

1. `regime_velocity` spikes mid-bar → regime shift detected 30s before bar close
2. ISV automatically reduces `risk_budget` via risk branch
3. Speculative "escape action" (direction=Flat) pre-computed and cached
4. At bar close: if shift confirmed → execute escape instantly. False alarm → proceed normally.

30-second early warning that no bar-level model can match.

---

## Inference Service Architecture

New component in `services/trading-agent/` (or extend existing):

```rust
struct TickProcessor {
    calculator: IncrementalMicrostructureCalculator,
    databento_client: databento::LiveClient,
    h100_channel: tokio::sync::mpsc::Sender<MicrostructureSnapshot>,
}

impl TickProcessor {
    /// Subscribe to Databento live MBP-10 for ES
    async fn run(&mut self) {
        let subscription = self.databento_client
            .subscribe("GLBX.MDP3", "mbp-10", &["ES"])
            .await;
        
        while let Some(event) = subscription.next().await {
            self.calculator.update(&event.snapshot, event.trade.as_ref());
            
            // Every 5s: send intermediate snapshot to H100 for speculative compute
            if self.calculator.tick_count % SPECULATIVE_INTERVAL == 0 {
                let snapshot = self.calculator.snapshot();
                self.h100_channel.send(snapshot).await;
            }
        }
    }
    
    /// Called at bar close — returns final features
    fn bar_close(&mut self) -> MicrostructureFeatures {
        let features = self.calculator.snapshot();
        self.calculator.reset();
        features
    }
}
```

---

## Implementation Order

1. `IncrementalMicrostructureCalculator` — 12 new features in `ofi_calculator.rs`
2. `fxcache.rs` — OFI_DIM 8→20, record size 224→272
3. `precompute_features.rs` — use new calculator, output 20 OFI
4. Regenerate all fxcache files on PVC
5. Experience collector — load 20 OFI, widen state buffers
6. DQN config — state_dim grows by 12
7. Trunk w_s1 resize — automatic via config
8. Databento live integration — TickProcessor service
9. H100 between-bar speculative compute
10. Smoke test + verification

---

## Risks

| Risk | Mitigation |
|------|-----------|
| Existing fxcache files incompatible | Regenerate all (~5-10 min). Code and data deploy atomically — new code rejects old fxcache. |
| OFI_DIM=8 hardcoded in 6+ files | All sites enumerated in spec. Must update atomically in single commit. Use shared constant, not literals. |
| 12 new features → overfitting on training data | ISV feature gating suppresses irrelevant features. 12 features / 4M bars = negligible risk. |
| IncrementalMicrostructureCalculator state drift | Reset at every bar boundary. Running statistics are bounded by definition. |
| Databento live feed drops/gaps | Calculator handles missing ticks gracefully (EMA continues with last value). At bar close, features reflect whatever ticks arrived. |
| Speculative trunk cache miss rate high | Free option. Cache hit → 10× faster. Cache miss → normal path. Measure hit rate and tune threshold. |
| Precompute changes bar alignment | Use same bar boundaries as existing pipeline. Ticks are assigned to bars by timestamp range. |
| Hawkes intensity numerically unstable | Clamp intensity to [0, 100]. Exponential decay with β=1.0 prevents unbounded growth. |
| Lee-Ready classification errors | Standard approach in literature. Errors are symmetric (equal chance of misclassifying buy as sell). Net effect averages out over 2000 ticks. |