diff --git a/docs/superpowers/specs/2026-04-23-tlob-feature-integration.md b/docs/superpowers/specs/2026-04-23-tlob-feature-integration.md new file mode 100644 index 000000000..3e773369b --- /dev/null +++ b/docs/superpowers/specs/2026-04-23-tlob-feature-integration.md @@ -0,0 +1,253 @@ +# TLOB-Style Order-Book Features vs Foxhunt OFI — Diff & Integration Plan + +**Date:** 2026-04-23 +**Status:** Phase A (diff + plan, no code changes) +**Scope:** Rust-only, native cudarc/cuBLAS path. Explicitly excludes ONNX and the `ml-supervised` dependency. + +--- + +## 1. Background + +- Foxhunt's DQN state vector is 96 elements (see `crates/ml-core/src/state_layout.rs`), of which slots `[42..62)` (`OFI_START..OFI_START + OFI_DIM`) carry the 20-dim OFI feature block. +- The 20 OFI slots are populated in `crates/ml/examples/precompute_features.rs` during the fxcache build and persisted per bar via `crates/ml/src/fxcache.rs` (version 4, magic `FXCACHE\0`). +- `ml-supervised/src/tlob/` still ships three non-ONNX modules: `features.rs` (51-feature `TLOBFeatureExtractor`), `mbp10_feature_extractor.rs` (Mbp10Snapshot → 51-feature adapter), and `analytics.rs` (stub struct, no logic). The broken `crates/ml/src/trainers/tlob.rs` consumer was removed in `ac959f807`. + +The goal of this document is to decide which TLOB features are worth migrating into Foxhunt's fxcache without bringing in the supervised crate or ONNX. + +--- + +## 2. Current Foxhunt OFI Inventory (20 features) + +The 20-element OFI vector written per bar into `.fxcache` records is assembled in `precompute_features.rs` (lines 460–527) and is read back by the DQN trunk via `ofi_embed_build_input` in `experience_kernels.cu` (lines 6148–6173). + +| Slot | Name | Source | Computation | Consumer | +|---:|---|---|---|---| +| 0 | `ofi_level1` | `OFICalculator::calc_ofi_l1` | Cont (2010) best-level OFI, z-score-normalized across a 300-snapshot rolling window (`OFIStats`) | DQN OFI embedding | +| 1 | `ofi_level5` | `OFICalculator::calc_ofi_l5` | Exponentially weighted multi-level OFI across levels 0..5 with weights `exp(-0.5·l)` | DQN OFI embedding | +| 2 | `depth_imbalance` | `OFICalculator::calc_depth_imbalance` | `(Σbid_sz − Σask_sz) / Σtotal`, clamped `[-1, 1]` | DQN OFI embedding | +| 3 | `vpin` | `VPINCalculator::compute` | Easley et al. 2012; needs `feed_trade()` to be non-zero | DQN OFI embedding | +| 4 | `kyle_lambda` | `KyleLambdaCalculator::compute` | Kyle (1985) regression `Δprice ~ signed_volume`; needs `feed_trade()` | DQN OFI embedding | +| 5 | `bid_slope` | `OFICalculator::calc_slopes` | Linear regression slope of cumulative bid volume vs level | DQN OFI embedding | +| 6 | `ask_slope` | `OFICalculator::calc_slopes` | Linear regression slope of cumulative ask volume vs level | DQN OFI embedding | +| 7 | `trade_imbalance` | `TradeImbalanceTracker` | Real buy/sell pressure if `feed_trade()` fed, else tick-rule | DQN OFI embedding | +| 8 | `delta_ofi_level1` | `precompute_features.rs:509` | `ofi[bar][0] − ofi[bar-1][0]` (lag-1 temporal delta) | DQN OFI embedding | +| 9 | `delta_ofi_level5` | same | delta of slot 1 | DQN OFI embedding | +| 10 | `delta_depth_imbalance` | same | delta of slot 2 | DQN OFI embedding | +| 11 | `delta_vpin` | same | delta of slot 3 | DQN OFI embedding | +| 12 | `delta_kyle_lambda` | same | delta of slot 4 | DQN OFI embedding | +| 13 | `delta_bid_slope` | same | delta of slot 5 | DQN OFI embedding | +| 14 | `delta_ask_slope` | same | delta of slot 6 | DQN OFI embedding | +| 15 | `delta_trade_imbalance` | same | delta of slot 7 | DQN OFI embedding | +| 16 | `book_aggression` | `precompute_features.rs:492–503` | Center-of-mass asymmetry `(sell_COM − buy_COM) / 10` across all 10 MBP-10 levels | DQN OFI embedding | +| 17 | `log_bar_duration` | `precompute_features.rs:517–527` | `ln(bar_duration_secs) / 10.0` | DQN OFI embedding | +| 18 | `ofi_acceleration` | `MicrostructureState::snapshot()[10]` | EMA of ΔOFI_L1 (from `ml-features/ofi_calculator.rs:951–957`) | **Persisted but unread** — `ofi_embed_build_input` only reads `[0..18)` | +| 19 | `toxicity_gradient` | `MicrostructureState::snapshot()[11]` | EMA of ΔVPIN (from `ml-features/ofi_calculator.rs:959–966`) | **Persisted but unread** | + +**Trunk-input dimensionality:** The OFI embedding MLP (`GpuDqnTrainer::launch_ofi_embed_forward`, `gpu_dqn_trainer.rs:3550–3591`) reads 18 of these 20 slots and projects to a 10-dim latent. Slots 18–19 live in the fxcache but are currently not consumed by the model. + +**Microstructure features that exist in `MicrostructureState` but are not persisted:** indices 0–9 of `MicrostructureState::snapshot()[..12]` produce: + +- `[0]` OFI Trajectory (online linreg slope of OFI_L1 vs tick index) +- `[1]` Realized Variance (Σ log-returns²) +- `[2]` Hawkes Trade Intensity (μ + α·e^(-β·Δt)) +- `[3]` Book Pressure (weighted `(bid-ask) / total` across 10 levels, different weighting from `book_aggression`) +- `[4]` Spread Dynamics `(max−min) / mean` across bar +- `[5]` Aggression Ratio (trades_at_ask / total_trades) +- `[6]` Queue Depletion Asymmetry (EMA bid/ask size drops) +- `[7]` Order Count Flux (ct_increase / (ct_increase + ct_decrease)) +- `[8]` Intra-Bar Momentum (sign-weighted product of first- and second-half mean returns) +- `[9]` Microstructure Regime Score (sigmoid of aggression × spread_dyn × thin_book) +- `[10]` OFI Acceleration (persisted as slot 18) +- `[11]` Toxicity Gradient (persisted as slot 19) + +Indices `[0..10)` of `MicrostructureState` are computed every snapshot but discarded before they reach fxcache. They are *not* new TLOB features — they are **pre-existing Foxhunt work** that has been left on the table. + +--- + +## 3. TLOB Extractor Inventory (51 features) + +The `TLOBFeatureExtractor` in `crates/ml-supervised/src/tlob/features.rs` produces a 51-element vector in five category blocks. Below is every name it emits, along with its actual behavior from the code (not its docstring aspiration). + +### 3.1 Price block — indices 0..10 (`extract_price_features`) + +| Idx | Name | Actual computation | +|---:|---|---| +| 0 | `spread_bps` | `normalize((ask − bid), 0, 1000)` — note: raw units, not basis points, despite the name | +| 1..9 | `level_1_spread` .. `level_9_spread` | `normalize((ask_i − bid_i), 0, 2000)` for the lowest 4 actual levels, zero otherwise — `TLOBFeatures` only stores 5 levels, so slots 5..9 are always zero. Importance pre-weights decay linearly by index. | + +### 3.2 Volume block — indices 10..22 (`extract_volume_features`) + +| Idx | Name | Actual computation | +|---:|---|---| +| 10 | `volume_imbalance` | `(Σbid_vol − Σask_vol) / (Σbid_vol + Σask_vol)` | +| 11..21 | `volume_feature_1..volume_feature_11` | **Placeholder**: each slot is the hard-coded constant `normalize(0.1·(i+1), 0, 10)` — no live volume data is read. | + +### 3.3 Microstructure block — indices 22..37 (`extract_microstructure_features`) + +| Idx | Name | Actual computation | +|---:|---|---| +| 22 | `vpin_score` | Single-snapshot VPIN: `|Σbid − Σask| / (Σbid + Σask)` — point-in-time estimate, no bucket history | +| 23 | `order_flow_toxicity` | `0.6 · (spread/mid) + 0.4 · |vol_imbalance|` | +| 24 | `book_pressure` | `(Σbid − Σask) / (Σbid + Σask)` — identical to slot 10 `volume_imbalance` | +| 25..36 | `microstructure_1..microstructure_12` | `normalize(microstructure_features[i % len], -1, 1)` — loops over whatever the upstream adapter placed in `TLOBFeatures.microstructure_features`. For the Mbp10 adapter, the source is the 11-element vector built in `mbp10_feature_extractor.rs:86–99` (see §3.6). | + +### 3.4 Technical block — indices 37..45 (`extract_technical_features`) + +| Idx | Name | Actual computation | +|---:|---|---| +| 37 | `momentum` | `normalize(features.momentum, -0.1, 0.1)` — upstream value | +| 38 | `volatility` | `normalize(features.volatility, 0, 0.5)` — upstream value | +| 39..44 | `technical_1..technical_6` | `normalize(sin(0.1·i), -1, 1)` — **pure sine-wave placeholders**, no market signal. | + +### 3.5 Time block — indices 45..51 (`extract_time_features`) + +| Idx | Name | Actual computation | +|---:|---|---| +| 45 | `time_since_update` | Hard-coded `0.5` — explicit mock in `features.rs:392`. | +| 46 | `time_of_day_pattern` | `(14..=21 UTC) ? 0.8 + 0.2·sin((hour-14)/7) : 0.2` — US cash-session indicator | +| 47 | `session_phase` | Bucketed UTC hour (14..16 = 0.9, 17..19 = 1.0, 20..21 = 0.7, else 0.3) | +| 48..50 | `time_feature_1..time_feature_3` | `normalize(sin(ts_us·(i+1) / 1e6), -1, 1)` — sinusoid of the raw timestamp | + +### 3.6 What the Mbp10 adapter actually supplies + +`ml-supervised/src/tlob/mbp10_feature_extractor.rs::extract_features_from_mbp10` constructs `TLOBFeatures.microstructure_features` as an 11-element vector: + +1. `spread` (raw, from `Mbp10Snapshot::spread()`) +2. `volume_imbalance` +3. `book_pressure` (weighted across all present levels) +4. `order_imbalance` (bid_ct vs ask_ct) +5. `vwap − mid_price` +6. `weighted_mid − mid_price` +7. `depth` (total level count) +8. `spread_bps` (`spread/mid × 10_000`, clamped 0..1000) +9. `price_impact` (`spread · (Σbid+Σask) / 1000`) +10. `ln(bid_order_count)` +11. `ln(ask_order_count)` + +This vector is fed to `TLOBFeatureExtractor` and appears in slots 25..36 (wrapped by `i % 11`, so the 12th slot wraps back to index 0). `features.volatility` (slot 38) is set to `spread / mid_price` and `features.momentum` (slot 37) is set to `volume_imbalance` — both coarse proxies rather than real time-series derivations. + +--- + +## 4. Overlap Matrix — TLOB vs Foxhunt OFI + +Each TLOB slot is classified as **Duplicate** (already computed by Foxhunt, possibly more rigorously), **Placeholder** (TLOB code emits a constant or a sine wave — no real signal), or **Novel** (genuinely new information worth integrating). + +| TLOB idx | Name | Status | Foxhunt equivalent / justification | +|---:|---|---|---| +| 0 | `spread_bps` | Duplicate (weaker) | Foxhunt's `Mbp10Snapshot::spread()` and `book_aggression` (slot 16) carry spread information; additionally, Mbp10-adapter slot 8 (spread in bps) is already wrapped into slots 25..36 | +| 1..9 | `level_1..9_spread` | Novel (partial) | **Per-level spread** is not currently persisted. Foxhunt's `ofi_level5` is a multi-level OFI but compresses everything into a scalar; the raw per-level structure is dropped. Candidate for migration (up to 5 real levels — MBP-10 has 10 available, but the TLOB extractor is hard-capped to 5). | +| 10 | `volume_imbalance` | Duplicate | Foxhunt slot 2 (`depth_imbalance`) is the identical formula. | +| 11..21 | `volume_feature_1..11` | Placeholder | Constants in source (`0.1·(i+1)`). No migration value. | +| 22 | `vpin_score` | Duplicate (weaker) | Foxhunt slot 3 (`vpin`) uses rolling signed-volume buckets and real trade data; TLOB's version is a snapshot-only `|imbalance| / total`. The Foxhunt version dominates. | +| 23 | `order_flow_toxicity` | Novel (borderline) | A linear blend `0.6·(spread/mid) + 0.4·|vol_imbalance|`. Foxhunt tracks both components separately but not this specific combiner. Low information value given the components are already present. | +| 24 | `book_pressure` | Duplicate | Identical formula to TLOB slot 10 and Foxhunt slot 2 (depth imbalance). Foxhunt also has a **weighted** book pressure in `MicrostructureState::calc_book_pressure` (`exp(-0.3·l)` weighting) that is computed but not persisted — see §5. | +| 25 (via adapter idx 0) | spread (raw) | Duplicate | Same as TLOB slot 0 | +| 26 (adapter idx 1) | volume_imbalance | Duplicate | Foxhunt slot 2 | +| 27 (adapter idx 2) | book_pressure (weighted across 10 levels) | Novel | Weighted book pressure across all available levels. Different weighting scheme from `book_aggression` (slot 16) and from `MicrostructureState::calc_book_pressure` — the TLOB/Mbp10-adapter version uses uniform `(bid − ask) / total`, while the Foxhunt `MicrostructureState` uses `exp(-0.3·l)` weighting. **The Mbp10-adapter version is the weaker of the two.** The Foxhunt version is already computed internally; the remedy is to persist `MicrostructureState::snapshot()[3]`, not to copy TLOB. | +| 28 (adapter idx 3) | `order_imbalance` (bid_ct vs ask_ct) | Novel | Count-based imbalance (number of resting orders) is not in Foxhunt OFI. Order counts (`bid_ct`, `ask_ct`) are available from MBP-10 but currently only consumed by `MicrostructureState::ct_increase_count` flux (slot 15, not persisted). | +| 29 (adapter idx 4) | `vwap − mid_price` | Novel | Deviation of VWAP (from the book) vs mid-price is not currently persisted. `Mbp10Snapshot::calculate_vwap()` already exists in `crates/data`. | +| 30 (adapter idx 5) | `weighted_mid − mid_price` | Novel | Weighted-mid deviation — microprice-like residual. Not in Foxhunt OFI. | +| 31 (adapter idx 6) | `depth` | Novel (trivial) | Count of populated levels in the snapshot. Information value is low because Databento MBP-10 always delivers 10 levels, but non-zero only when `bid_sz > 0`. Probably not worth a dedicated slot. | +| 32 (adapter idx 7) | `spread_bps` (0..1000 clamped) | Duplicate | Same signal as TLOB slot 0. | +| 33 (adapter idx 8) | `price_impact` (`spread · Σvolume / 1000`) | Novel | Naive price-impact proxy. Foxhunt's `kyle_lambda` (slot 4) is a superior, regression-based impact estimate. This slot adds only a heuristic; duplicate-ish. | +| 34 (adapter idx 9) | `ln(bid_order_count)` | Novel | Log of total bid order count. Not in Foxhunt OFI. | +| 35 (adapter idx 10) | `ln(ask_order_count)` | Novel | Log of total ask order count. Not in Foxhunt OFI. | +| 36 (adapter wraps to idx 0: spread) | | Duplicate | Wrap of slot 25. | +| 37 | `momentum` | Duplicate | Foxhunt has multi-timeframe momentum features in the 42-dim market block (see `ml-features/price_features.rs`). | +| 38 | `volatility` | Duplicate (weaker) | Foxhunt has EWMA volatility via `ml-features/ewma.rs` and realized variance via `MicrostructureState::snapshot()[1]` (unpersisted). | +| 39..44 | `technical_1..6` | Placeholder | Sine-wave constants. | +| 45 | `time_since_update` | Placeholder | Hard-coded `0.5`. | +| 46 | `time_of_day_pattern` | Novel | UTC-hour-bucketed US-session flag with a smooth sine. Foxhunt does have time features in `ml-features/time_features.rs` — need to verify overlap, but a session-phase indicator is reasonable. | +| 47 | `session_phase` | Novel | Step-function variant of slot 46. Similar overlap caveats. | +| 48..50 | `time_feature_1..3` | Placeholder | Sinusoids of raw timestamp. No market signal. | + +### Overlap summary + +| Bucket | Count | +|---|---| +| Duplicate (already in Foxhunt OFI or 42-dim market block) | 12 slots | +| Placeholder (hardcoded constants or sine waves) | 23 slots | +| Novel (genuinely new information) | **16 slots** — but see deduplication below | + +### Novel-feature deduplication + +After collapsing redundancy within the novel set, the shortlist of genuinely distinct new features is: + +1. **Per-level spreads** (`level_1..5_spread`) — 5 features. Only 5 are meaningful because `TLOBFeatures` caps at 5 levels. +2. **Count-based imbalance** (`order_imbalance` = `(bid_ct − ask_ct) / (bid_ct + ask_ct)`) — 1 feature. +3. **VWAP − mid deviation** — 1 feature. +4. **Weighted-mid − mid deviation** (microprice residual) — 1 feature. +5. **Log bid/ask order counts** — 2 features. +6. **Session-phase / time-of-day indicator** — 1 or 2 features (overlapping — either the step-function or the smooth sine, not both). + +That collapses to **10 genuinely novel features**. Every other "novel" slot in TLOB is either a weaker re-implementation of something Foxhunt already has, or a placeholder. + +### Foxhunt-internal features worth persisting *before* adding TLOB slots + +The `MicrostructureState` structure already computes **10 features that are unpersisted** (slots 0–9 of `snapshot()`): realized variance, Hawkes trade intensity, weighted book pressure, spread dynamics, aggression ratio, queue-depletion asymmetry, order-count flux, intra-bar momentum, microstructure regime score, and OFI trajectory. These dominate most of the TLOB "novel" candidates in information content and require **zero new math** — only a change in the fxcache write path. Any integration pass should prioritize persisting these before pulling TLOB-adapter slots. + +--- + +## 5. Phase B Integration Plan (Draft, ≤ 10 novel features) + +This plan is provisional. Before implementation, the overlap matrix above should be reviewed; a reasonable first migration is 6–10 slots, selected from the §4 shortlist. + +### 5.1 Target OFI layout + +Bump `OFI_DIM` from 20 → either **28** (add 8 new slots, keeps 8-alignment) or **32** (add 12 new slots, maintains tensor-core alignment and leaves headroom). Recommendation: **28** — covers the high-value Foxhunt-internal slots first and one or two TLOB-style additions. The recommended new slots, in precedence order: + +1. `MicrostructureState::snapshot()[0..10]` — 10 already-computed Foxhunt features. Select the 6 highest-value (e.g. realized variance, Hawkes intensity, weighted book pressure, spread dynamics, aggression ratio, queue-depletion asymmetry) if picking 28. +2. `(bid_ct − ask_ct) / (bid_ct + ask_ct)` — count-based imbalance (TLOB-novel). +3. `(weighted_mid − mid) / mid` — microprice residual (TLOB-novel). + +The precise slot assignment and a final 8-slot vs 12-slot choice should be decided during Phase B kickoff. + +### 5.2 Files to update + +| File | Change | +|---|---| +| `crates/ml-core/src/state_layout.rs` | Bump `OFI_DIM` (20 → 28 or 32). Verify `STATE_DIM` stays 8-aligned (new STATE_DIM becomes 104 or 108; pick 104/112 to stay `% 8 == 0`, or 128 to stay `% 128 == 0` at the padded level). | +| `crates/ml/src/fxcache.rs` | Bump `OFI_DIM` constant, bump `FXCACHE_VERSION` from 4 → 5. `validate()` will reject v4 files at load; the `ensure-fxcache` Argo step regenerates. Update the format doc comment, `RECORD_F64_COUNT`, and the test fixtures in `has_ofi_tests` / `discover_tests`. | +| `crates/ml/examples/precompute_features.rs` | Extend the `ofi_per_bar` row from `[0.0; 20]` to `[0.0; OFI_DIM]` and fill the new slots at the point where `micro_state.snapshot()` is consumed. Ensure `MicrostructureState::snapshot()` values that were previously dropped are now written. No new math required for the Foxhunt-internal candidates. | +| `crates/ml-features/src/ofi_calculator.rs` | No changes required — all source computations already exist in `MicrostructureState`. | +| `crates/ml/src/cuda_pipeline/experience_kernels.cu` (`ofi_embed_build_input`) | Extend the input-extraction kernel from reading 18 slots to reading the new `OFI_DIM`. Update the `[B, 18]` scratch buffer to `[B, OFI_DIM]` everywhere it is allocated and referenced (see `gpu_dqn_trainer.rs:8217` and related). The cuBLAS SGEMM dimensions change: `input[B, OFI_DIM] @ W^T[OFI_DIM, 10]` — the 10-dim output stays the same unless a wider embedding is desired. | +| `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` | Update the `ofi_embed_input_buf` allocation (line 8217), the `ofi_embed_w` weight tensor shape (`OFI_DIM → 10`), Xavier init sizes, backward-pass kernels, and every hard-coded `18` referring to the OFI input width. | +| `crates/ml-core/src/state_layout.rs` (headers) + `state_layout.cuh` | Keep the CUDA header in sync so `SL_OFI_START` + `SL_OFI_DIM` match. | +| `crates/ml/src/cuda_pipeline/c51_loss_kernel.cu`, `gpu_dqn_trainer.rs` builders | Any hard-coded `20` or `18` for OFI slice widths must be swept — `grep -rn '\\b18\\b\\|\\b20\\b' crates/ml/src/cuda_pipeline` before the final commit. | +| Trunk parameter count (`w_b1fc`, `w_b1bias`, etc.) | If the OFI embedding width is unchanged (still → 10), trunk shapes are unaffected. If the OFI embed output width is increased, the downstream shared-trunk input dim grows, requiring a param-tensor resize and smoke-test re-run. | + +### 5.3 Cache-migration path + +The fxcache already includes a strict version guard (`FxCacheHeader::validate()` rejects non-current `FXCACHE_VERSION`). Steps: + +1. Bump `FXCACHE_VERSION` from 4 → 5 in a single commit with the precompute change. Existing `.fxcache` files become invalid. +2. The `ensure-fxcache` Argo step detects the error and triggers regeneration from MBP-10 + trades (148 GB + 3.3 GB on the PVC). +3. Regeneration time on the L40S pool: previously ~45 min for 4M bars; the incremental microstructure work is already done per tick, so expected overhead is <10%. +4. The per-fold `norm_stats.json` files alongside each `.fxcache` (see `norm_stats_path_for_key`) are regenerated automatically by the same precompute step — no separate migration. +5. Downstream consumers: supervised and RL trainers both call `load_fxcache`, which fails loudly on mismatch. No silent format-drift failure mode. + +### 5.4 Smoke-test verification + +After the dimension bump: + +- `SQLX_OFFLINE=true cargo check --workspace` must pass (catches all hard-coded `20`/`18`). +- `SQLX_OFFLINE=true cargo test -p ml --lib` — FxCache roundtrip tests in `fxcache.rs` should be updated to use the new `OFI_DIM` and pass. +- `FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture` — the magnitude-distribution smoke test (20 epochs on RTX 3050 Ti, recently restored to passing in commit `2fb30f098`) must continue to pass. A dimension change that breaks the trunk forward will show up as `NaN` Q-values or a shape-mismatch crash before training starts. +- The `multi_fold_convergence` smoke test (`crates/ml/src/trainers/dqn/smoke_tests/multi_fold_convergence.rs`) must continue to pass. + +### 5.5 Risks and migration notes + +- **cuBLAS alignment.** The existing design uses `STATE_DIM = 96` because `96 % 8 == 0` (required for tensor-core SGEMM). Growing `OFI_DIM` by 8 → `STATE_DIM = 104` → padded to 128 works. Growing by 12 → `STATE_DIM = 108` is **not** 8-aligned and requires 4 more padding bytes (`PADDING_DIM: 4 → 4` stays OK if the new total is 112; 108 → needs `PADDING_DIM = 4` to reach 112). Static asserts at the bottom of `state_layout.rs` will catch any miscount. +- **Hard-coded `20` and `18`.** The OFI width appears in multiple CUDA kernels and Rust call sites (see `grep` sweep in §5.2). A single-commit integration must sweep all of them — leaving one behind produces silent memory-boundary bugs that smoke tests may not catch if they only exercise the forward pass. +- **No new dependencies.** All source data (`bid_ct`, `ask_ct`, weighted mid, VWAP) is already accessible via `data::providers::databento::mbp10::Mbp10Snapshot`. `MicrostructureState` is in `ml-features` which is already a DQN dependency. No new crate or ONNX import is required. +- **Do not copy from `ml-supervised`.** The TLOB extractor in `ml-supervised/src/tlob/features.rs` is 30–40 % placeholder code (sine waves, hard-coded constants). Copying it wholesale would inject dead features into the DQN state. The integration pattern should be: re-implement the handful of genuinely novel calculations directly against `Mbp10Snapshot` inside `ml-features`, following the existing `OFICalculator`/`MicrostructureState` style. + +--- + +## 6. Recommendation + +- The TLOB 51-feature extractor has roughly **10 genuinely novel features** relative to Foxhunt's 20 OFI + 42 market + 12 `MicrostructureState` features. Most of the "novel" TLOB slots are either placeholders or weaker re-implementations. +- The highest-impact Phase B move is to **persist the 10 `MicrostructureState` features that are already computed but dropped** (slots 0..9 of `snapshot()`), not to import TLOB. This is low-risk: zero new math, a dimension bump, a cache regen, and a smoke-test pass. +- If the Phase B budget allows an 8-slot expansion (`OFI_DIM` 20 → 28), a sensible split is: 6 top-ranked `MicrostructureState` features + 2 TLOB-novel additions (order-count imbalance + microprice residual). This keeps the integration focused on real signal and avoids pulling in any part of the `ml-supervised` TLOB crate. + +Phase B should be scoped after review of this document.