foxhunt/MBP10_QUICK_REFERENCE.md

# MBP-10 Quick Reference Guide

## One-Minute Overview

**MBP-10** = Market By Price with 10 price levels (best bid/ask to 10th level)

**Location**: `/home/jgrusewski/Work/foxhunt/data/src/providers/databento/mbp10.rs`

**Purpose**: Extract microstructure features for TLOB ML model training

---

## Core Types

### BidAskPair
```rust
struct BidAskPair {
    bid_px: i64,        // Price (fixed-point 1e-12)
    bid_sz: u32,        // Volume
    bid_ct: u32,        // Order count
    ask_px: i64,        // Price (fixed-point 1e-12)
    ask_sz: u32,        // Volume
    ask_ct: u32,        // Order count
}

// Quick methods
bid.bid_price()         // f64 price
bid.ask_price()         // f64 price
bid.is_valid()          // Check non-zero
```

### Mbp10Snapshot
```rust
struct Mbp10Snapshot {
    symbol: String,
    timestamp: u64,
    levels: Vec<BidAskPair>,  // Always 10 levels
    sequence: u32,
    trade_count: u32,
}

// Quick methods
snapshot.mid_price()              // (best_bid + best_ask) / 2
snapshot.spread()                 // best_ask - best_bid
snapshot.volume_imbalance()       // [-1, 1] balance metric
snapshot.calculate_vwap()         // Volume-weighted avg price
snapshot.total_bid_volume()       // Sum of all bid volumes
snapshot.total_ask_volume()       // Sum of all ask volumes
snapshot.depth()                  // Count of valid levels
```

---

## Feature Extraction Summary

**Output**: 51-dimensional feature vector

| Category | Count | Description |
|----------|-------|-------------|
| Price Levels | 20 | Bid/Ask for each of 10 levels (normalized) |
| Volume Levels | 10 | Log-scaled volumes for each level |
| Microstructure | 21 | Spread, imbalance, depth, liquidity, VWAP |
| **Total** | **51** | Full microstructure snapshot |

---

## Common Operations

### Create Snapshot
```rust
let mut levels = vec![];
for i in 0..10 {
    levels.push(BidAskPair {
        bid_px: BidAskPair::price_from_f64(100.0 - i as f64 * 0.01),
        bid_sz: 1000,
        bid_ct: 5,
        ask_px: BidAskPair::price_from_f64(100.05 + i as f64 * 0.01),
        ask_sz: 800,
        ask_ct: 3,
    });
}

let snapshot = Mbp10Snapshot::new("ES.FUT".into(), timestamp, levels, seq, trades);
```

### Extract Key Metrics
```rust
let mid = snapshot.mid_price();
let spread_bps = snapshot.spread() / mid * 10000.0;
let imbalance = snapshot.volume_imbalance();  // Range: [-1, 1]
let depth = snapshot.depth();
```

### Build Features
```rust
let mid = snapshot.mid_price();
let mut features = Vec::new();

// Price levels (normalized)
for level in &snapshot.levels {
    features.push(((level.bid_price() - mid) / mid) as f32);
    features.push(((level.ask_price() - mid) / mid) as f32);
}

// Volume levels (log-scaled)
for level in &snapshot.levels {
    features.push((level.bid_sz as f32 + 1.0).ln());
    features.push((level.ask_sz as f32 + 1.0).ln());
}

// Microstructure
features.push(snapshot.spread() as f32);
features.push(snapshot.volume_imbalance() as f32);
features.push(snapshot.calculate_vwap() as f32);
// ... continue for 51 total
```

### Update During Stream
```rust
snapshot.update_level(
    0,                          // Level
    OrderBookAction::Add,       // Action (Add/Modify/Cancel/Trade)
    BidAskPair::price_from_f64(100.50),
    1000,                       // Size
    5,                          // Order count
    true,                       // Bid side
);
```

---

## Price Conversion Cheat Sheet

```rust
// String price → Fixed-point (1e-12 scaling)
let fixed = BidAskPair::price_from_f64(150.50);
// Result: 150500000000000

// Fixed-point → String price
let price = BidAskPair::price_to_f64(150500000000000);
// Result: 150.5

// Example prices
100.00  = 100000000000000
150.55  = 150550000000000
4500.75 = 4500750000000000
```

---

## Data Quality Checks

```rust
// Validate snapshot
if !snapshot.levels.iter().all(|l| l.is_valid()) {
    // Some levels are empty
}

// Check spread sanity
let spread_bps = snapshot.spread() / snapshot.mid_price() * 10000.0;
if spread_bps > 1000.0 {
    // Unreasonable spread (>1%)
}

// Check for crossover (bug detection)
let (bid, ask) = snapshot.get_best_bid_ask();
if bid >= ask {
    // ERROR: Price crossover
}
```

---

## ML Training Pipeline

```
1. Load DBN file
2. Parse Mbp10Snapshot from each record
3. Extract 51-dim features
4. Create labels (price direction, etc.)
5. Feed to TLOB model
```

---

## Performance

| Operation | Time |
|-----------|------|
| `mid_price()` | <100ns |
| `volume_imbalance()` | ~500ns |
| `calculate_vwap()` | ~1.2μs |
| Extract 51 features | ~5-10μs |
| Update level | <200ns |

**Throughput**: 50K+ feature vectors/sec

---

## Related Files

- **Implementation**: `/home/jgrusewski/Work/foxhunt/data/src/providers/databento/mbp10.rs`
- **Feature Extraction**: `/home/jgrusewski/Work/foxhunt/ml/src/features/`
- **TLOB Model**: `/home/jgrusewski/Work/foxhunt/ml/src/tlob/`
- **DBN Streaming**: `/home/jgrusewski/Work/foxhunt/data/src/providers/databento/dbn_parser.rs`

---

## Common Mistakes to Avoid

1. **Forgetting the 1e-12 scaling**
   ```rust
   // WRONG
   let price = snapshot.levels[0].bid_px as f64;  // Will be huge
   
   // RIGHT
   let price = BidAskPair::price_to_f64(snapshot.levels[0].bid_px);
   ```

2. **Division by zero in normalization**
   ```rust
   // WRONG
   let normalized = (price - mid) / mid;  // If mid == 0
   
   // RIGHT
   let normalized = (price - mid) / (mid + 1e-8);
   ```

3. **Forgetting to validate levels**
   ```rust
   // WRONG - assumes all levels valid
   for level in &snapshot.levels {
       // Use data...
   }
   
   // RIGHT
   for level in &snapshot.levels {
       if level.is_valid() {
           // Use data...
       }
   }
   ```

4. **Ignoring zero volumes**
   ```rust
   // WRONG
   let log_vol = (snapshot.levels[0].bid_sz as f32).ln();  // NaN if 0
   
   // RIGHT
   let log_vol = (snapshot.levels[0].bid_sz as f32 + 1.0).ln();
   ```

---

## Test Commands

```bash
# Run MBP-10 tests
cargo test --lib data::providers::databento::mbp10

# Test feature extraction
cargo test --lib ml::features

# Integration tests
cargo test --test ml_readiness -- --nocapture
```