# Volume Bar Pipeline — Design Spec

## Problem

The fxcache training data has INTERLEAVED OHLCV bars from multiple ES futures contract months (ESH4, ESM4, ESU4, ESZ4...). The OHLCV DBN loader loads all contract months and sorts by timestamp, creating fake ±60 point price jumps between consecutive bars. This produces garbage reward signals — the model has never seen clean data. WinRate is 26% (below random 33%) even with zero transaction costs.

The OHLCV DBN files are the wrong data source. We have MBP-10 and trades DBN data from Databento that is higher quality and already used for OFI features.

## Solution

Replace the OHLCV-based bar construction with volume bars built from trades data, aligned with MBP-10 snapshots for order book features.

### Volume Bars

Aggregate trades until cumulative volume reaches 100 contracts. Each bar records:
- **Open**: first trade price in the bar
- **High**: max trade price in the bar
- **Low**: min trade price in the bar
- **Close**: last trade price in the bar
- **Volume**: total contracts (always ≥100, may exceed if a single trade crosses the threshold)
- **Timestamp**: last trade's timestamp (nanoseconds since epoch)

**Why 100 contracts**: ES trades ~1.5M contracts/day during regular hours (~6.5h). At 100 contracts/bar: ~15,000 bars/day, ~23 bars/minute during active trading, 0-1 overnight. This provides sub-minute resolution during liquid periods and naturally filters dead overnight periods. Suitable for MFT with 5-60 minute hold periods.

**Why volume bars over time bars**: equal information content per bar (no empty overnight bars), better statistical properties (closer to i.i.d.), self-adjusting for liquidity. Standard approach from Lopez de Prado's "Advances in Financial Machine Learning."

### Instrument Filtering

Trades DBN files from Databento contain all contract months for a symbol. The pipeline must select only the front-month continuous contract:

1. Group trades by `instrument_id`
2. Per session (trading day), select the `instrument_id` with the highest cumulative volume
3. On roll days (when front month switches), use the NEW front month from the roll point onward
4. Reject trades from non-front-month contracts

### MBP-10 Alignment

For each volume bar, snap the most recent MBP-10 snapshot (closest timestamp ≤ bar timestamp):
- Compute OFI (Order Flow Imbalance) from the book delta between consecutive bar-aligned snapshots
- Extract mid_price as a feature (separate from the trade-price-based OHLCV)
- If no MBP-10 snapshot exists within 60s of a bar, use the previous bar's snapshot (forward-fill)

### Targets

`targets[4] = [close, next_close, close, next_close]` — same layout as current fxcache. Close = last trade price of the volume bar. Next_close = last trade price of the next volume bar.

### Feature Extraction

The existing 42-feature extraction pipeline (SMA, RSI, ATR, Bollinger, momentum, etc.) operates on the volume bars exactly as it does on time bars. The 50-bar warmup period becomes "last 50 volume bars" instead of "last 50 minutes." Features naturally adapt — ATR on volume bars measures volatility per unit of market activity.

### fxcache Format

Unchanged. Same 64-byte header + `[i64 timestamp, f64×42 features, f64×4 targets, f64×8 ofi]` per bar. The CUDA kernels don't change — they just read features and prices. Drop-in replacement.

## Files Modified

- `crates/ml/examples/precompute_features.rs` — replace OHLCV DBN loading (Step 2) with trades-based volume bar construction + instrument filtering. Keep MBP-10 OFI path but snap to volume bar timestamps.
- `crates/ml/src/trainers/dqn/data_loading.rs` — update `decode_ohlcv_records` to use trades DBN → volume bars. Add instrument_id filtering. Update `collect_dbn_files_filtered` to look for trades files.
- `crates/data/src/providers/databento/dbn_parser.rs` — add `parse_trades_to_volume_bars()` function that reads trades DBN, filters by instrument_id, and aggregates into volume bars.
- `crates/ml/src/fxcache.rs` — no format changes. May need to update cache key computation to include bar_type=volume and bar_size=100.

## Files NOT Modified

- CUDA kernels — they read from fxcache, don't care about bar type
- Training loop — unchanged
- Feature extraction (`features/extraction.rs`) — operates on OHLCVBars, unchanged
- OFI calculator — unchanged, just receives differently-timed snapshots

## Testing

### Unit Tests

1. **Volume bar construction**: synthetic trades → verify OHLCV. Cases:
   - Exactly 100 contracts → 1 bar
   - 250 contracts in 3 trades → 2 bars (100 + 150, second bar closes at cumulative boundary)
   - Single trade of 500 contracts → 5 bars
   - Zero-volume gap → no bars produced

2. **Instrument filtering**: 3 instrument_ids with known volumes → verify only the highest-volume is kept. Test roll where front month switches mid-session.

3. **MBP-10 snap alignment**: volume bars at t=[100, 200, 300], snapshots at t=[90, 150, 250, 310] → verify bars get snapshots [90, 150, 250]. Test no-snapshot-available case.

### Integration Tests

4. **Price continuity (regression gate)**: load real ES trades DBN, build volume bars, assert NO consecutive bars with >0.5% price jump. This catches the interleaved-contracts bug.

5. **fxcache round-trip**: build fxcache from volume bars, load back, verify features/targets match. Check feature statistics (mean, std) are in expected ranges.

### Smoke Test

6. **Training signal**: build fxcache from clean volume bars, run 10-epoch smoke test, verify:
   - WinRate > 30% (above random — proves direction signal exists)
   - Q-value gap > 0.01 (model differentiates direction)
   - No price jumps > 0.5% in consecutive bars (data integrity)

Test 4 (price continuity) MUST run as part of the precompute binary and fail loudly if violated.

## Success Criteria

- fxcache built from volume bars with zero interleaved-contract artifacts
- Consecutive bar price continuity: max |close[t+1] - close[t]| / close[t] < 0.5%
- Local smoke test WinRate > 30% within 10 epochs
- H100 training shows Q-value gap > 0.01 and PF trending toward 1.0+
