docs(tlob): feature-diff doc for TLOB 51 vs OFI 20 — Phase A

Compares the 51-feature TLOBFeatureExtractor (crates/ml-supervised/src/tlob/
features.rs + mbp10_feature_extractor.rs) against Foxhunt's 20-slot OFI
vector persisted in .fxcache.

Findings: of TLOB's 51 slots, 23 are placeholders (sine waves / hardcoded
constants), 12 duplicate existing OFI or 42-dim market features, and only
~10 carry genuinely novel information. Meanwhile the existing
MicrostructureState struct in ml-features already computes 10 features
(realized variance, Hawkes intensity, weighted book pressure, spread
dynamics, aggression ratio, queue-depletion asymmetry, order-count flux,
intra-bar momentum, regime score, OFI trajectory) that are dropped before
reaching fxcache — these dominate the TLOB novel candidates and require
zero new math.

Phase B plan (drafted inline): bump OFI_DIM 20→28, bump FXCACHE_VERSION
4→5, prioritize persisting the 10 Foxhunt-internal MicrostructureState
features before any TLOB-derived additions, sweep all hard-coded 18/20
constants across CUDA kernels + gpu_dqn_trainer.rs, and rely on the
ensure-fxcache Argo step for cache regeneration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-04-23 09:12:01 +02:00
parent d9fee6ef8d
commit 746b8b6754

View File

@@ -0,0 +1,253 @@
# TLOB-Style Order-Book Features vs Foxhunt OFI — Diff & Integration Plan
**Date:** 2026-04-23
**Status:** Phase A (diff + plan, no code changes)
**Scope:** Rust-only, native cudarc/cuBLAS path. Explicitly excludes ONNX and the `ml-supervised` dependency.
---
## 1. Background
- Foxhunt's DQN state vector is 96 elements (see `crates/ml-core/src/state_layout.rs`), of which slots `[42..62)` (`OFI_START..OFI_START + OFI_DIM`) carry the 20-dim OFI feature block.
- The 20 OFI slots are populated in `crates/ml/examples/precompute_features.rs` during the fxcache build and persisted per bar via `crates/ml/src/fxcache.rs` (version 4, magic `FXCACHE\0`).
- `ml-supervised/src/tlob/` still ships three non-ONNX modules: `features.rs` (51-feature `TLOBFeatureExtractor`), `mbp10_feature_extractor.rs` (Mbp10Snapshot → 51-feature adapter), and `analytics.rs` (stub struct, no logic). The broken `crates/ml/src/trainers/tlob.rs` consumer was removed in `ac959f807`.
The goal of this document is to decide which TLOB features are worth migrating into Foxhunt's fxcache without bringing in the supervised crate or ONNX.
---
## 2. Current Foxhunt OFI Inventory (20 features)
The 20-element OFI vector written per bar into `.fxcache` records is assembled in `precompute_features.rs` (lines 460527) and is read back by the DQN trunk via `ofi_embed_build_input` in `experience_kernels.cu` (lines 61486173).
| Slot | Name | Source | Computation | Consumer |
|---:|---|---|---|---|
| 0 | `ofi_level1` | `OFICalculator::calc_ofi_l1` | Cont (2010) best-level OFI, z-score-normalized across a 300-snapshot rolling window (`OFIStats`) | DQN OFI embedding |
| 1 | `ofi_level5` | `OFICalculator::calc_ofi_l5` | Exponentially weighted multi-level OFI across levels 0..5 with weights `exp(-0.5·l)` | DQN OFI embedding |
| 2 | `depth_imbalance` | `OFICalculator::calc_depth_imbalance` | `(Σbid_sz Σask_sz) / Σtotal`, clamped `[-1, 1]` | DQN OFI embedding |
| 3 | `vpin` | `VPINCalculator::compute` | Easley et al. 2012; needs `feed_trade()` to be non-zero | DQN OFI embedding |
| 4 | `kyle_lambda` | `KyleLambdaCalculator::compute` | Kyle (1985) regression `Δprice ~ signed_volume`; needs `feed_trade()` | DQN OFI embedding |
| 5 | `bid_slope` | `OFICalculator::calc_slopes` | Linear regression slope of cumulative bid volume vs level | DQN OFI embedding |
| 6 | `ask_slope` | `OFICalculator::calc_slopes` | Linear regression slope of cumulative ask volume vs level | DQN OFI embedding |
| 7 | `trade_imbalance` | `TradeImbalanceTracker` | Real buy/sell pressure if `feed_trade()` fed, else tick-rule | DQN OFI embedding |
| 8 | `delta_ofi_level1` | `precompute_features.rs:509` | `ofi[bar][0] ofi[bar-1][0]` (lag-1 temporal delta) | DQN OFI embedding |
| 9 | `delta_ofi_level5` | same | delta of slot 1 | DQN OFI embedding |
| 10 | `delta_depth_imbalance` | same | delta of slot 2 | DQN OFI embedding |
| 11 | `delta_vpin` | same | delta of slot 3 | DQN OFI embedding |
| 12 | `delta_kyle_lambda` | same | delta of slot 4 | DQN OFI embedding |
| 13 | `delta_bid_slope` | same | delta of slot 5 | DQN OFI embedding |
| 14 | `delta_ask_slope` | same | delta of slot 6 | DQN OFI embedding |
| 15 | `delta_trade_imbalance` | same | delta of slot 7 | DQN OFI embedding |
| 16 | `book_aggression` | `precompute_features.rs:492503` | Center-of-mass asymmetry `(sell_COM buy_COM) / 10` across all 10 MBP-10 levels | DQN OFI embedding |
| 17 | `log_bar_duration` | `precompute_features.rs:517527` | `ln(bar_duration_secs) / 10.0` | DQN OFI embedding |
| 18 | `ofi_acceleration` | `MicrostructureState::snapshot()[10]` | EMA of ΔOFI_L1 (from `ml-features/ofi_calculator.rs:951957`) | **Persisted but unread**`ofi_embed_build_input` only reads `[0..18)` |
| 19 | `toxicity_gradient` | `MicrostructureState::snapshot()[11]` | EMA of ΔVPIN (from `ml-features/ofi_calculator.rs:959966`) | **Persisted but unread** |
**Trunk-input dimensionality:** The OFI embedding MLP (`GpuDqnTrainer::launch_ofi_embed_forward`, `gpu_dqn_trainer.rs:35503591`) reads 18 of these 20 slots and projects to a 10-dim latent. Slots 1819 live in the fxcache but are currently not consumed by the model.
**Microstructure features that exist in `MicrostructureState` but are not persisted:** indices 09 of `MicrostructureState::snapshot()[..12]` produce:
- `[0]` OFI Trajectory (online linreg slope of OFI_L1 vs tick index)
- `[1]` Realized Variance (Σ log-returns²)
- `[2]` Hawkes Trade Intensity (μ + α·e^(-β·Δt))
- `[3]` Book Pressure (weighted `(bid-ask) / total` across 10 levels, different weighting from `book_aggression`)
- `[4]` Spread Dynamics `(maxmin) / mean` across bar
- `[5]` Aggression Ratio (trades_at_ask / total_trades)
- `[6]` Queue Depletion Asymmetry (EMA bid/ask size drops)
- `[7]` Order Count Flux (ct_increase / (ct_increase + ct_decrease))
- `[8]` Intra-Bar Momentum (sign-weighted product of first- and second-half mean returns)
- `[9]` Microstructure Regime Score (sigmoid of aggression × spread_dyn × thin_book)
- `[10]` OFI Acceleration (persisted as slot 18)
- `[11]` Toxicity Gradient (persisted as slot 19)
Indices `[0..10)` of `MicrostructureState` are computed every snapshot but discarded before they reach fxcache. They are *not* new TLOB features — they are **pre-existing Foxhunt work** that has been left on the table.
---
## 3. TLOB Extractor Inventory (51 features)
The `TLOBFeatureExtractor` in `crates/ml-supervised/src/tlob/features.rs` produces a 51-element vector in five category blocks. Below is every name it emits, along with its actual behavior from the code (not its docstring aspiration).
### 3.1 Price block — indices 0..10 (`extract_price_features`)
| Idx | Name | Actual computation |
|---:|---|---|
| 0 | `spread_bps` | `normalize((ask bid), 0, 1000)` — note: raw units, not basis points, despite the name |
| 1..9 | `level_1_spread` .. `level_9_spread` | `normalize((ask_i bid_i), 0, 2000)` for the lowest 4 actual levels, zero otherwise — `TLOBFeatures` only stores 5 levels, so slots 5..9 are always zero. Importance pre-weights decay linearly by index. |
### 3.2 Volume block — indices 10..22 (`extract_volume_features`)
| Idx | Name | Actual computation |
|---:|---|---|
| 10 | `volume_imbalance` | `(Σbid_vol Σask_vol) / (Σbid_vol + Σask_vol)` |
| 11..21 | `volume_feature_1..volume_feature_11` | **Placeholder**: each slot is the hard-coded constant `normalize(0.1·(i+1), 0, 10)` — no live volume data is read. |
### 3.3 Microstructure block — indices 22..37 (`extract_microstructure_features`)
| Idx | Name | Actual computation |
|---:|---|---|
| 22 | `vpin_score` | Single-snapshot VPIN: `|Σbid Σask| / (Σbid + Σask)` — point-in-time estimate, no bucket history |
| 23 | `order_flow_toxicity` | `0.6 · (spread/mid) + 0.4 · |vol_imbalance|` |
| 24 | `book_pressure` | `(Σbid Σask) / (Σbid + Σask)` — identical to slot 10 `volume_imbalance` |
| 25..36 | `microstructure_1..microstructure_12` | `normalize(microstructure_features[i % len], -1, 1)` — loops over whatever the upstream adapter placed in `TLOBFeatures.microstructure_features`. For the Mbp10 adapter, the source is the 11-element vector built in `mbp10_feature_extractor.rs:8699` (see §3.6). |
### 3.4 Technical block — indices 37..45 (`extract_technical_features`)
| Idx | Name | Actual computation |
|---:|---|---|
| 37 | `momentum` | `normalize(features.momentum, -0.1, 0.1)` — upstream value |
| 38 | `volatility` | `normalize(features.volatility, 0, 0.5)` — upstream value |
| 39..44 | `technical_1..technical_6` | `normalize(sin(0.1·i), -1, 1)`**pure sine-wave placeholders**, no market signal. |
### 3.5 Time block — indices 45..51 (`extract_time_features`)
| Idx | Name | Actual computation |
|---:|---|---|
| 45 | `time_since_update` | Hard-coded `0.5` — explicit mock in `features.rs:392`. |
| 46 | `time_of_day_pattern` | `(14..=21 UTC) ? 0.8 + 0.2·sin((hour-14)/7) : 0.2` — US cash-session indicator |
| 47 | `session_phase` | Bucketed UTC hour (14..16 = 0.9, 17..19 = 1.0, 20..21 = 0.7, else 0.3) |
| 48..50 | `time_feature_1..time_feature_3` | `normalize(sin(ts_us·(i+1) / 1e6), -1, 1)` — sinusoid of the raw timestamp |
### 3.6 What the Mbp10 adapter actually supplies
`ml-supervised/src/tlob/mbp10_feature_extractor.rs::extract_features_from_mbp10` constructs `TLOBFeatures.microstructure_features` as an 11-element vector:
1. `spread` (raw, from `Mbp10Snapshot::spread()`)
2. `volume_imbalance`
3. `book_pressure` (weighted across all present levels)
4. `order_imbalance` (bid_ct vs ask_ct)
5. `vwap mid_price`
6. `weighted_mid mid_price`
7. `depth` (total level count)
8. `spread_bps` (`spread/mid × 10_000`, clamped 0..1000)
9. `price_impact` (`spread · (Σbid+Σask) / 1000`)
10. `ln(bid_order_count)`
11. `ln(ask_order_count)`
This vector is fed to `TLOBFeatureExtractor` and appears in slots 25..36 (wrapped by `i % 11`, so the 12th slot wraps back to index 0). `features.volatility` (slot 38) is set to `spread / mid_price` and `features.momentum` (slot 37) is set to `volume_imbalance` — both coarse proxies rather than real time-series derivations.
---
## 4. Overlap Matrix — TLOB vs Foxhunt OFI
Each TLOB slot is classified as **Duplicate** (already computed by Foxhunt, possibly more rigorously), **Placeholder** (TLOB code emits a constant or a sine wave — no real signal), or **Novel** (genuinely new information worth integrating).
| TLOB idx | Name | Status | Foxhunt equivalent / justification |
|---:|---|---|---|
| 0 | `spread_bps` | Duplicate (weaker) | Foxhunt's `Mbp10Snapshot::spread()` and `book_aggression` (slot 16) carry spread information; additionally, Mbp10-adapter slot 8 (spread in bps) is already wrapped into slots 25..36 |
| 1..9 | `level_1..9_spread` | Novel (partial) | **Per-level spread** is not currently persisted. Foxhunt's `ofi_level5` is a multi-level OFI but compresses everything into a scalar; the raw per-level structure is dropped. Candidate for migration (up to 5 real levels — MBP-10 has 10 available, but the TLOB extractor is hard-capped to 5). |
| 10 | `volume_imbalance` | Duplicate | Foxhunt slot 2 (`depth_imbalance`) is the identical formula. |
| 11..21 | `volume_feature_1..11` | Placeholder | Constants in source (`0.1·(i+1)`). No migration value. |
| 22 | `vpin_score` | Duplicate (weaker) | Foxhunt slot 3 (`vpin`) uses rolling signed-volume buckets and real trade data; TLOB's version is a snapshot-only `|imbalance| / total`. The Foxhunt version dominates. |
| 23 | `order_flow_toxicity` | Novel (borderline) | A linear blend `0.6·(spread/mid) + 0.4·|vol_imbalance|`. Foxhunt tracks both components separately but not this specific combiner. Low information value given the components are already present. |
| 24 | `book_pressure` | Duplicate | Identical formula to TLOB slot 10 and Foxhunt slot 2 (depth imbalance). Foxhunt also has a **weighted** book pressure in `MicrostructureState::calc_book_pressure` (`exp(-0.3·l)` weighting) that is computed but not persisted — see §5. |
| 25 (via adapter idx 0) | spread (raw) | Duplicate | Same as TLOB slot 0 |
| 26 (adapter idx 1) | volume_imbalance | Duplicate | Foxhunt slot 2 |
| 27 (adapter idx 2) | book_pressure (weighted across 10 levels) | Novel | Weighted book pressure across all available levels. Different weighting scheme from `book_aggression` (slot 16) and from `MicrostructureState::calc_book_pressure` — the TLOB/Mbp10-adapter version uses uniform `(bid ask) / total`, while the Foxhunt `MicrostructureState` uses `exp(-0.3·l)` weighting. **The Mbp10-adapter version is the weaker of the two.** The Foxhunt version is already computed internally; the remedy is to persist `MicrostructureState::snapshot()[3]`, not to copy TLOB. |
| 28 (adapter idx 3) | `order_imbalance` (bid_ct vs ask_ct) | Novel | Count-based imbalance (number of resting orders) is not in Foxhunt OFI. Order counts (`bid_ct`, `ask_ct`) are available from MBP-10 but currently only consumed by `MicrostructureState::ct_increase_count` flux (slot 15, not persisted). |
| 29 (adapter idx 4) | `vwap mid_price` | Novel | Deviation of VWAP (from the book) vs mid-price is not currently persisted. `Mbp10Snapshot::calculate_vwap()` already exists in `crates/data`. |
| 30 (adapter idx 5) | `weighted_mid mid_price` | Novel | Weighted-mid deviation — microprice-like residual. Not in Foxhunt OFI. |
| 31 (adapter idx 6) | `depth` | Novel (trivial) | Count of populated levels in the snapshot. Information value is low because Databento MBP-10 always delivers 10 levels, but non-zero only when `bid_sz > 0`. Probably not worth a dedicated slot. |
| 32 (adapter idx 7) | `spread_bps` (0..1000 clamped) | Duplicate | Same signal as TLOB slot 0. |
| 33 (adapter idx 8) | `price_impact` (`spread · Σvolume / 1000`) | Novel | Naive price-impact proxy. Foxhunt's `kyle_lambda` (slot 4) is a superior, regression-based impact estimate. This slot adds only a heuristic; duplicate-ish. |
| 34 (adapter idx 9) | `ln(bid_order_count)` | Novel | Log of total bid order count. Not in Foxhunt OFI. |
| 35 (adapter idx 10) | `ln(ask_order_count)` | Novel | Log of total ask order count. Not in Foxhunt OFI. |
| 36 (adapter wraps to idx 0: spread) | | Duplicate | Wrap of slot 25. |
| 37 | `momentum` | Duplicate | Foxhunt has multi-timeframe momentum features in the 42-dim market block (see `ml-features/price_features.rs`). |
| 38 | `volatility` | Duplicate (weaker) | Foxhunt has EWMA volatility via `ml-features/ewma.rs` and realized variance via `MicrostructureState::snapshot()[1]` (unpersisted). |
| 39..44 | `technical_1..6` | Placeholder | Sine-wave constants. |
| 45 | `time_since_update` | Placeholder | Hard-coded `0.5`. |
| 46 | `time_of_day_pattern` | Novel | UTC-hour-bucketed US-session flag with a smooth sine. Foxhunt does have time features in `ml-features/time_features.rs` — need to verify overlap, but a session-phase indicator is reasonable. |
| 47 | `session_phase` | Novel | Step-function variant of slot 46. Similar overlap caveats. |
| 48..50 | `time_feature_1..3` | Placeholder | Sinusoids of raw timestamp. No market signal. |
### Overlap summary
| Bucket | Count |
|---|---|
| Duplicate (already in Foxhunt OFI or 42-dim market block) | 12 slots |
| Placeholder (hardcoded constants or sine waves) | 23 slots |
| Novel (genuinely new information) | **16 slots** — but see deduplication below |
### Novel-feature deduplication
After collapsing redundancy within the novel set, the shortlist of genuinely distinct new features is:
1. **Per-level spreads** (`level_1..5_spread`) — 5 features. Only 5 are meaningful because `TLOBFeatures` caps at 5 levels.
2. **Count-based imbalance** (`order_imbalance` = `(bid_ct ask_ct) / (bid_ct + ask_ct)`) — 1 feature.
3. **VWAP mid deviation** — 1 feature.
4. **Weighted-mid mid deviation** (microprice residual) — 1 feature.
5. **Log bid/ask order counts** — 2 features.
6. **Session-phase / time-of-day indicator** — 1 or 2 features (overlapping — either the step-function or the smooth sine, not both).
That collapses to **10 genuinely novel features**. Every other "novel" slot in TLOB is either a weaker re-implementation of something Foxhunt already has, or a placeholder.
### Foxhunt-internal features worth persisting *before* adding TLOB slots
The `MicrostructureState` structure already computes **10 features that are unpersisted** (slots 09 of `snapshot()`): realized variance, Hawkes trade intensity, weighted book pressure, spread dynamics, aggression ratio, queue-depletion asymmetry, order-count flux, intra-bar momentum, microstructure regime score, and OFI trajectory. These dominate most of the TLOB "novel" candidates in information content and require **zero new math** — only a change in the fxcache write path. Any integration pass should prioritize persisting these before pulling TLOB-adapter slots.
---
## 5. Phase B Integration Plan (Draft, ≤ 10 novel features)
This plan is provisional. Before implementation, the overlap matrix above should be reviewed; a reasonable first migration is 610 slots, selected from the §4 shortlist.
### 5.1 Target OFI layout
Bump `OFI_DIM` from 20 → either **28** (add 8 new slots, keeps 8-alignment) or **32** (add 12 new slots, maintains tensor-core alignment and leaves headroom). Recommendation: **28** — covers the high-value Foxhunt-internal slots first and one or two TLOB-style additions. The recommended new slots, in precedence order:
1. `MicrostructureState::snapshot()[0..10]` — 10 already-computed Foxhunt features. Select the 6 highest-value (e.g. realized variance, Hawkes intensity, weighted book pressure, spread dynamics, aggression ratio, queue-depletion asymmetry) if picking 28.
2. `(bid_ct ask_ct) / (bid_ct + ask_ct)` — count-based imbalance (TLOB-novel).
3. `(weighted_mid mid) / mid` — microprice residual (TLOB-novel).
The precise slot assignment and a final 8-slot vs 12-slot choice should be decided during Phase B kickoff.
### 5.2 Files to update
| File | Change |
|---|---|
| `crates/ml-core/src/state_layout.rs` | Bump `OFI_DIM` (20 → 28 or 32). Verify `STATE_DIM` stays 8-aligned (new STATE_DIM becomes 104 or 108; pick 104/112 to stay `% 8 == 0`, or 128 to stay `% 128 == 0` at the padded level). |
| `crates/ml/src/fxcache.rs` | Bump `OFI_DIM` constant, bump `FXCACHE_VERSION` from 4 → 5. `validate()` will reject v4 files at load; the `ensure-fxcache` Argo step regenerates. Update the format doc comment, `RECORD_F64_COUNT`, and the test fixtures in `has_ofi_tests` / `discover_tests`. |
| `crates/ml/examples/precompute_features.rs` | Extend the `ofi_per_bar` row from `[0.0; 20]` to `[0.0; OFI_DIM]` and fill the new slots at the point where `micro_state.snapshot()` is consumed. Ensure `MicrostructureState::snapshot()` values that were previously dropped are now written. No new math required for the Foxhunt-internal candidates. |
| `crates/ml-features/src/ofi_calculator.rs` | No changes required — all source computations already exist in `MicrostructureState`. |
| `crates/ml/src/cuda_pipeline/experience_kernels.cu` (`ofi_embed_build_input`) | Extend the input-extraction kernel from reading 18 slots to reading the new `OFI_DIM`. Update the `[B, 18]` scratch buffer to `[B, OFI_DIM]` everywhere it is allocated and referenced (see `gpu_dqn_trainer.rs:8217` and related). The cuBLAS SGEMM dimensions change: `input[B, OFI_DIM] @ W^T[OFI_DIM, 10]` — the 10-dim output stays the same unless a wider embedding is desired. |
| `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` | Update the `ofi_embed_input_buf` allocation (line 8217), the `ofi_embed_w` weight tensor shape (`OFI_DIM → 10`), Xavier init sizes, backward-pass kernels, and every hard-coded `18` referring to the OFI input width. |
| `crates/ml-core/src/state_layout.rs` (headers) + `state_layout.cuh` | Keep the CUDA header in sync so `SL_OFI_START` + `SL_OFI_DIM` match. |
| `crates/ml/src/cuda_pipeline/c51_loss_kernel.cu`, `gpu_dqn_trainer.rs` builders | Any hard-coded `20` or `18` for OFI slice widths must be swept — `grep -rn '\\b18\\b\\|\\b20\\b' crates/ml/src/cuda_pipeline` before the final commit. |
| Trunk parameter count (`w_b1fc`, `w_b1bias`, etc.) | If the OFI embedding width is unchanged (still → 10), trunk shapes are unaffected. If the OFI embed output width is increased, the downstream shared-trunk input dim grows, requiring a param-tensor resize and smoke-test re-run. |
### 5.3 Cache-migration path
The fxcache already includes a strict version guard (`FxCacheHeader::validate()` rejects non-current `FXCACHE_VERSION`). Steps:
1. Bump `FXCACHE_VERSION` from 4 → 5 in a single commit with the precompute change. Existing `.fxcache` files become invalid.
2. The `ensure-fxcache` Argo step detects the error and triggers regeneration from MBP-10 + trades (148 GB + 3.3 GB on the PVC).
3. Regeneration time on the L40S pool: previously ~45 min for 4M bars; the incremental microstructure work is already done per tick, so expected overhead is <10%.
4. The per-fold `norm_stats.json` files alongside each `.fxcache` (see `norm_stats_path_for_key`) are regenerated automatically by the same precompute step — no separate migration.
5. Downstream consumers: supervised and RL trainers both call `load_fxcache`, which fails loudly on mismatch. No silent format-drift failure mode.
### 5.4 Smoke-test verification
After the dimension bump:
- `SQLX_OFFLINE=true cargo check --workspace` must pass (catches all hard-coded `20`/`18`).
- `SQLX_OFFLINE=true cargo test -p ml --lib` — FxCache roundtrip tests in `fxcache.rs` should be updated to use the new `OFI_DIM` and pass.
- `FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture` — the magnitude-distribution smoke test (20 epochs on RTX 3050 Ti, recently restored to passing in commit `2fb30f098`) must continue to pass. A dimension change that breaks the trunk forward will show up as `NaN` Q-values or a shape-mismatch crash before training starts.
- The `multi_fold_convergence` smoke test (`crates/ml/src/trainers/dqn/smoke_tests/multi_fold_convergence.rs`) must continue to pass.
### 5.5 Risks and migration notes
- **cuBLAS alignment.** The existing design uses `STATE_DIM = 96` because `96 % 8 == 0` (required for tensor-core SGEMM). Growing `OFI_DIM` by 8 → `STATE_DIM = 104` → padded to 128 works. Growing by 12 → `STATE_DIM = 108` is **not** 8-aligned and requires 4 more padding bytes (`PADDING_DIM: 4 → 4` stays OK if the new total is 112; 108 → needs `PADDING_DIM = 4` to reach 112). Static asserts at the bottom of `state_layout.rs` will catch any miscount.
- **Hard-coded `20` and `18`.** The OFI width appears in multiple CUDA kernels and Rust call sites (see `grep` sweep in §5.2). A single-commit integration must sweep all of them — leaving one behind produces silent memory-boundary bugs that smoke tests may not catch if they only exercise the forward pass.
- **No new dependencies.** All source data (`bid_ct`, `ask_ct`, weighted mid, VWAP) is already accessible via `data::providers::databento::mbp10::Mbp10Snapshot`. `MicrostructureState` is in `ml-features` which is already a DQN dependency. No new crate or ONNX import is required.
- **Do not copy from `ml-supervised`.** The TLOB extractor in `ml-supervised/src/tlob/features.rs` is 3040 % placeholder code (sine waves, hard-coded constants). Copying it wholesale would inject dead features into the DQN state. The integration pattern should be: re-implement the handful of genuinely novel calculations directly against `Mbp10Snapshot` inside `ml-features`, following the existing `OFICalculator`/`MicrostructureState` style.
---
## 6. Recommendation
- The TLOB 51-feature extractor has roughly **10 genuinely novel features** relative to Foxhunt's 20 OFI + 42 market + 12 `MicrostructureState` features. Most of the "novel" TLOB slots are either placeholders or weaker re-implementations.
- The highest-impact Phase B move is to **persist the 10 `MicrostructureState` features that are already computed but dropped** (slots 0..9 of `snapshot()`), not to import TLOB. This is low-risk: zero new math, a dimension bump, a cache regen, and a smoke-test pass.
- If the Phase B budget allows an 8-slot expansion (`OFI_DIM` 20 → 28), a sensible split is: 6 top-ranked `MicrostructureState` features + 2 TLOB-novel additions (order-count imbalance + microprice residual). This keeps the integration focused on real signal and avoids pulling in any part of the `ml-supervised` TLOB crate.
Phase B should be scoped after review of this document.