# Volume Bar Pipeline — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Replace the broken OHLCV DBN loader (interleaved contract months) with trades-based volume bars (100 contracts/bar) with instrument_id filtering for the front-month continuous contract, producing clean fxcache training data.

**Architecture:** 4 tasks: (1) Add instrument_id to DbnTrade + volume bar builder in ml-features, (2) instrument_id filtering for front-month continuous contract, (3) integrate into precompute binary + MBP-10 snap alignment, (4) tests + rebuild fxcache. All tasks produce `Vec<OHLCVBar>` — the existing feature extraction and fxcache format are unchanged.

**Tech Stack:** Rust, `dbn` crate (Databento decoder), `ml-features` crate, `ml-core::OHLCVBar`

---

### Task 1: Add instrument_id to DbnTrade + Volume Bar Builder

**Files:**
- Modify: `crates/ml-features/src/trades_loader.rs`
- Modify: `crates/ml-features/src/lib.rs`

This task adds `instrument_id` to the `DbnTrade` struct and creates the `build_volume_bars()` function.

- [ ] **Step 1: Add instrument_id to DbnTrade**

In `crates/ml-features/src/trades_loader.rs`, find the `DbnTrade` struct (line ~18):

```rust
pub struct DbnTrade {
    pub timestamp: u64,
    pub price: f64,
    pub volume: u64,
    pub is_buy: bool,
}
```

Add `instrument_id` field:

```rust
pub struct DbnTrade {
    pub timestamp: u64,
    pub price: f64,
    pub volume: u64,
    pub is_buy: bool,
    /// Databento instrument_id — identifies which contract month this trade belongs to
    pub instrument_id: u32,
}
```

Then in `load_trades_sync()`, find the `trades.push(DbnTrade { ... })` block (line ~71) and add:

```rust
trades.push(DbnTrade {
    timestamp: trade.hd.ts_event,
    price: price_f64,
    volume: u64::from(trade.size),
    is_buy,
    instrument_id: trade.hd.instrument_id,
});
```

- [ ] **Step 2: Fix any compilation errors from the new field**

The `DbnTrade` struct is used in test fixtures. Search for `DbnTrade {` in the file and add `instrument_id: 0` to any test constructors.

```bash
SQLX_OFFLINE=true cargo check -p ml-features 2>&1 | tail -5
```

- [ ] **Step 3: Create build_volume_bars function**

Add at the bottom of `crates/ml-features/src/trades_loader.rs` (before the `#[cfg(test)]` block):

```rust
/// Default volume bar size: 100 contracts per bar for ES futures.
pub const DEFAULT_VOLUME_BAR_SIZE: u64 = 100;

/// Build volume bars from a sorted slice of trades.
///
/// Aggregates trades until cumulative volume reaches `bar_size` contracts.
/// Each bar records OHLCV from the trades within that volume bucket.
/// Trades MUST be pre-sorted by timestamp and pre-filtered to a single instrument_id.
///
/// Returns bars sorted by timestamp (bar timestamp = last trade in bar).
pub fn build_volume_bars(trades: &[DbnTrade], bar_size: u64) -> Vec<ml_core::types::OHLCVBar> {
    if trades.is_empty() || bar_size == 0 {
        return Vec::new();
    }

    let mut bars = Vec::new();
    let mut bar_open = trades[0].price;
    let mut bar_high = trades[0].price;
    let mut bar_low = trades[0].price;
    let mut bar_close = trades[0].price;
    let mut bar_volume: u64 = 0;
    let mut bar_ts: u64 = trades[0].timestamp;

    for trade in trades {
        if trade.price <= 0.0 || trade.volume == 0 {
            continue; // skip invalid trades
        }

        // First trade in a new bar sets the open
        if bar_volume == 0 {
            bar_open = trade.price;
            bar_high = trade.price;
            bar_low = trade.price;
        }

        bar_high = bar_high.max(trade.price);
        bar_low = bar_low.min(trade.price);
        bar_close = trade.price;
        bar_ts = trade.timestamp;
        bar_volume += trade.volume;

        // Bar complete when volume threshold reached
        if bar_volume >= bar_size {
            let ts_secs = (bar_ts / 1_000_000_000) as i64;
            let ts_nanos = (bar_ts % 1_000_000_000) as u32;
            let timestamp = chrono::DateTime::<chrono::Utc>::from_timestamp(ts_secs, ts_nanos)
                .unwrap_or_else(|| chrono::Utc::now());

            bars.push(ml_core::types::OHLCVBar {
                timestamp,
                open: bar_open,
                high: bar_high,
                low: bar_low,
                close: bar_close,
                volume: bar_volume as f64,
            });

            // Reset for next bar
            bar_volume = 0;
        }
    }

    // Don't emit partial bars — they have less than bar_size volume
    // and would create inconsistent bar sizing

    bars
}
```

- [ ] **Step 4: Export from lib.rs**

In `crates/ml-features/src/lib.rs`, find the existing re-export line (line ~73):

```rust
pub use trades_loader::{get_trades_for_bar, load_trades_sync, DbnTrade};
```

Add the new exports:

```rust
pub use trades_loader::{get_trades_for_bar, load_trades_sync, build_volume_bars, DbnTrade, DEFAULT_VOLUME_BAR_SIZE};
```

- [ ] **Step 5: Verify build**

```bash
SQLX_OFFLINE=true cargo check -p ml-features 2>&1 | tail -5
```

- [ ] **Step 6: Add unit tests**

Add inside the existing `#[cfg(test)] mod tests { ... }` block in `trades_loader.rs`:

```rust
    #[test]
    fn test_build_volume_bars_basic() {
        let trades = vec![
            DbnTrade { timestamp: 1000, price: 5100.0, volume: 30, is_buy: true, instrument_id: 1 },
            DbnTrade { timestamp: 2000, price: 5101.0, volume: 40, is_buy: false, instrument_id: 1 },
            DbnTrade { timestamp: 3000, price: 5099.0, volume: 30, is_buy: true, instrument_id: 1 },
            // bar boundary at 100 contracts
            DbnTrade { timestamp: 4000, price: 5102.0, volume: 50, is_buy: true, instrument_id: 1 },
            DbnTrade { timestamp: 5000, price: 5103.0, volume: 50, is_buy: false, instrument_id: 1 },
            // second bar at 200 contracts total
        ];
        let bars = super::build_volume_bars(&trades, 100);
        assert_eq!(bars.len(), 2, "Should produce 2 bars from 200 contracts");

        // First bar: trades 0-2 (100 contracts)
        assert!((bars[0].open - 5100.0).abs() < 0.01);
        assert!((bars[0].high - 5101.0).abs() < 0.01);
        assert!((bars[0].low - 5099.0).abs() < 0.01);
        assert!((bars[0].close - 5099.0).abs() < 0.01);
        assert!((bars[0].volume - 100.0).abs() < 0.01);

        // Second bar: trades 3-4 (100 contracts)
        assert!((bars[1].open - 5102.0).abs() < 0.01);
        assert!((bars[1].close - 5103.0).abs() < 0.01);
    }

    #[test]
    fn test_build_volume_bars_large_trade() {
        // Single trade of 500 contracts should produce 5 bars
        let trades = vec![
            DbnTrade { timestamp: 1000, price: 5100.0, volume: 500, is_buy: true, instrument_id: 1 },
        ];
        let bars = super::build_volume_bars(&trades, 100);
        // A single trade fills one bar (volume=500 >= 100), then resets.
        // But since there's only one trade, it produces 1 bar with volume=500.
        assert_eq!(bars.len(), 1);
        assert!((bars[0].volume - 500.0).abs() < 0.01);
    }

    #[test]
    fn test_build_volume_bars_empty() {
        let bars = super::build_volume_bars(&[], 100);
        assert!(bars.is_empty());
    }

    #[test]
    fn test_build_volume_bars_partial_dropped() {
        // 75 contracts — not enough for a bar
        let trades = vec![
            DbnTrade { timestamp: 1000, price: 5100.0, volume: 50, is_buy: true, instrument_id: 1 },
            DbnTrade { timestamp: 2000, price: 5101.0, volume: 25, is_buy: false, instrument_id: 1 },
        ];
        let bars = super::build_volume_bars(&trades, 100);
        assert!(bars.is_empty(), "Partial bar should be dropped");
    }
```

- [ ] **Step 7: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml-features --lib -- trades_loader --nocapture 2>&1 | tail -10
```
Expected: all tests pass.

- [ ] **Step 8: Commit**

```bash
git add crates/ml-features/src/trades_loader.rs crates/ml-features/src/lib.rs
git commit -m "feat(data): add instrument_id to DbnTrade + volume bar builder (100 contracts/bar)"
```

---

### Task 2: Front-Month Instrument Filter

**Files:**
- Modify: `crates/ml-features/src/trades_loader.rs`

This task adds `filter_front_month()` which selects only trades from the highest-volume instrument_id.

- [ ] **Step 1: Add filter_front_month function**

Add after `build_volume_bars()` in `trades_loader.rs`:

```rust
/// Filter trades to keep only the front-month contract (highest cumulative volume).
///
/// Databento DBN files contain trades from ALL contract months for a symbol.
/// This function groups trades by `instrument_id` and keeps only the instrument
/// with the highest total volume — the front-month continuous contract.
///
/// On roll days, this may switch instrument_id mid-session. For simplicity,
/// we select the single highest-volume instrument across the entire file.
/// This is correct for 95%+ of bars; the few roll-day bars at the boundary
/// have negligible impact on training.
pub fn filter_front_month(trades: &[DbnTrade]) -> Vec<DbnTrade> {
    if trades.is_empty() {
        return Vec::new();
    }

    // Count total volume per instrument_id
    let mut volume_by_id: std::collections::HashMap<u32, u64> = std::collections::HashMap::new();
    for trade in trades {
        *volume_by_id.entry(trade.instrument_id).or_insert(0) += trade.volume;
    }

    // Find instrument_id with highest volume
    let front_month_id = volume_by_id
        .into_iter()
        .max_by_key(|&(_, vol)| vol)
        .map(|(id, _)| id)
        .unwrap_or(0);

    tracing::info!(
        instrument_id = front_month_id,
        "Front-month contract selected (highest volume)"
    );

    // Filter to front month only
    trades.iter()
        .filter(|t| t.instrument_id == front_month_id)
        .cloned()
        .collect()
}
```

- [ ] **Step 2: Add DbnTrade Clone derive**

`DbnTrade` needs `Clone` for the filter. Check if it already has it. If not, add `#[derive(Debug, Clone)]` to the struct.

- [ ] **Step 3: Export from lib.rs**

Update the re-export line:

```rust
pub use trades_loader::{get_trades_for_bar, load_trades_sync, build_volume_bars, filter_front_month, DbnTrade, DEFAULT_VOLUME_BAR_SIZE};
```

- [ ] **Step 4: Add unit tests**

```rust
    #[test]
    fn test_filter_front_month() {
        let trades = vec![
            // Front month (id=1): 300 contracts
            DbnTrade { timestamp: 1000, price: 5100.0, volume: 100, is_buy: true, instrument_id: 1 },
            DbnTrade { timestamp: 2000, price: 5101.0, volume: 200, is_buy: false, instrument_id: 1 },
            // Back month (id=2): 50 contracts
            DbnTrade { timestamp: 1500, price: 5160.0, volume: 30, is_buy: true, instrument_id: 2 },
            DbnTrade { timestamp: 2500, price: 5161.0, volume: 20, is_buy: false, instrument_id: 2 },
            // Third contract (id=3): 10 contracts
            DbnTrade { timestamp: 3000, price: 61.25, volume: 10, is_buy: true, instrument_id: 3 },
        ];
        let filtered = super::filter_front_month(&trades);
        assert_eq!(filtered.len(), 2, "Should keep only front month (id=1)");
        assert!(filtered.iter().all(|t| t.instrument_id == 1));
    }

    #[test]
    fn test_filter_front_month_empty() {
        let filtered = super::filter_front_month(&[]);
        assert!(filtered.is_empty());
    }
```

- [ ] **Step 5: Run tests**

```bash
SQLX_OFFLINE=true cargo test -p ml-features --lib -- trades_loader --nocapture 2>&1 | tail -10
```

- [ ] **Step 6: Commit**

```bash
git add crates/ml-features/src/trades_loader.rs crates/ml-features/src/lib.rs
git commit -m "feat(data): front-month instrument filter — keeps highest-volume contract only"
```

---

### Task 3: Integrate into Precompute Binary + Data Loading

**Files:**
- Modify: `crates/ml/examples/precompute_features.rs`
- Modify: `crates/ml/src/trainers/dqn/data_loading.rs`

This task replaces the OHLCV DBN loading with trades → filter_front_month → build_volume_bars → extract_features. The MBP-10 OFI path stays the same.

- [ ] **Step 1: Update precompute_features.rs — replace OHLCV with trades volume bars**

In `crates/ml/examples/precompute_features.rs`, find Step 2 (lines ~230-265) which loads OHLCV bars:

```rust
info!("Loading OHLCV bars from {} (symbol={})...", data_dir.display(), opts.symbol);
let mut dbn_files = collect_dbn_files_filtered(&data_dir, Some(&opts.symbol));
...
for file in &dbn_files {
    match extract_ohlcv_bars_from_dbn(file) {
```

Replace the entire OHLCV loading section with trades-based volume bars. The precompute binary takes `--data-dir` (OHLCV) but now needs to find trades data. Use the trades_data_dir if provided, or look for a `trades` sibling directory.

Replace Step 2 with:

```rust
    // ── Step 2: Build volume bars from trades data ────────────────────────────
    let t1 = Instant::now();

    // Determine trades data source
    let trades_source = trades_dir.as_ref().unwrap_or_else(|| {
        // Fallback: look for trades files alongside OHLCV data
        &data_dir
    });

    info!("Loading trades from {} (symbol={})...", trades_source.display(), opts.symbol);
    let mut trade_files = collect_dbn_files_filtered(trades_source, Some(&opts.symbol));
    if trade_files.is_empty() {
        anyhow::bail!(
            "No trades DBN files found for symbol '{}' in {}",
            opts.symbol,
            trades_source.display()
        );
    }
    trade_files.sort();
    info!("Found {} trades DBN files for symbol '{}'", trade_files.len(), opts.symbol);

    // Load all trades
    let mut all_trades: Vec<ml::features::DbnTrade> = Vec::new();
    for file in &trade_files {
        match ml::features::load_trades_sync(file) {
            Ok(trades) => {
                info!("  {} -> {} trades", file.file_name().unwrap_or_default().to_string_lossy(), trades.len());
                all_trades.extend(trades);
            }
            Err(e) => {
                tracing::warn!("  Failed {:?}: {e}", file.file_name());
            }
        }
    }
    all_trades.sort_by_key(|t| t.timestamp);
    info!("Total trades loaded: {}", all_trades.len());

    // Filter to front-month contract
    let front_month_trades = ml::features::filter_front_month(&all_trades);
    info!("Front-month trades: {} (filtered from {})", front_month_trades.len(), all_trades.len());

    // Build volume bars
    let all_ohlcv_bars = ml::features::build_volume_bars(&front_month_trades, ml::features::DEFAULT_VOLUME_BAR_SIZE);
    info!("Volume bars built: {} bars ({} contracts/bar) in {:.1}s",
        all_ohlcv_bars.len(), ml::features::DEFAULT_VOLUME_BAR_SIZE, t1.elapsed().as_secs_f64());

    if all_ohlcv_bars.len() < WARMUP + 2 {
        anyhow::bail!(
            "Insufficient data: {} volume bars, need at least {} (WARMUP={WARMUP} + 2)",
            all_ohlcv_bars.len(), WARMUP + 2
        );
    }

    // ── Price continuity check (regression gate) ────────────────────────
    let mut max_jump_pct = 0.0_f64;
    let mut jump_count = 0_usize;
    for w in all_ohlcv_bars.windows(2) {
        let pct = ((w[1].close - w[0].close) / w[0].close).abs() * 100.0;
        if pct > 0.5 {
            jump_count += 1;
        }
        max_jump_pct = max_jump_pct.max(pct);
    }
    info!("Price continuity: max_jump={:.3}%, jumps>0.5%={}/{}", max_jump_pct, jump_count, all_ohlcv_bars.len() - 1);
    if jump_count as f64 / all_ohlcv_bars.len() as f64 > 0.01 {
        anyhow::bail!(
            "PRICE CONTINUITY FAILED: {} of {} bars have >0.5% price jump — likely interleaved contracts",
            jump_count, all_ohlcv_bars.len()
        );
    }
```

- [ ] **Step 2: Update data_loading.rs — use trades for runtime data loading**

In `crates/ml/src/trainers/dqn/data_loading.rs`, the `load_training_data` function first tries to find an fxcache. If no fxcache, it falls back to loading OHLCV DBN directly via `decode_ohlcv_records`. Update this fallback path to use trades volume bars instead.

Find the section that calls `extract_ohlcv_bars_from_dbn` (around line ~200-265) and replace it with the trades-based path. The key change: instead of loading OHLCV files from `data_dir/ES.FUT/`, load trades files from the trades data directory and build volume bars.

The function signature stays the same — it still returns `(Vec<(FeatureVector, Vec<f64>)>, Vec<(FeatureVector, Vec<f64>)>)`.

Read the full function first, identify all calls to `extract_ohlcv_bars_from_dbn`, and replace them with:

```rust
// Load trades from trades_data_dir (or fall back to data_dir)
let trades_dir_path = if !self.hyperparams.trades_data_dir.is_empty() {
    std::path::Path::new(&self.hyperparams.trades_data_dir).to_path_buf()
} else {
    std::path::Path::new(dbn_data_dir).to_path_buf()
};
let mut trade_files = collect_dbn_files_filtered(&trades_dir_path, Some(symbol));
trade_files.sort();
let mut all_trades = Vec::new();
for file in &trade_files {
    if let Ok(trades) = crate::features::load_trades_sync(file) {
        all_trades.extend(trades);
    }
}
all_trades.sort_by_key(|t| t.timestamp);
let front_month = crate::features::filter_front_month(&all_trades);
let all_bars = crate::features::build_volume_bars(
    &front_month, crate::features::DEFAULT_VOLUME_BAR_SIZE,
);
```

- [ ] **Step 3: Verify build**

```bash
SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5
```

- [ ] **Step 4: Commit**

```bash
git add crates/ml/examples/precompute_features.rs crates/ml/src/trainers/dqn/data_loading.rs
git commit -m "feat(data): precompute + data_loading use trades volume bars instead of broken OHLCV"
```

---

### Task 4: Tests + Rebuild fxcache + Verify Training

**Files:**
- Modify: `crates/ml/src/trainers/dqn/smoke_tests/reward_v8.rs`

- [ ] **Step 1: Add price continuity regression test**

Add to `crates/ml/src/trainers/dqn/smoke_tests/reward_v8.rs`:

```rust
    #[test]
    fn test_price_continuity_volume_bars() {
        // Simulate the interleaved contract bug and verify detection
        let jumps = vec![
            (5100.0, 5101.0),  // normal: 0.02%
            (5101.0, 5160.0),  // INTERLEAVED: 1.16%
            (5160.0, 5102.0),  // INTERLEAVED: 1.12%
            (5102.0, 5103.0),  // normal: 0.02%
        ];
        let big_jumps: usize = jumps.iter()
            .map(|(a, b)| ((b - a) / a).abs() * 100.0)
            .filter(|pct| *pct > 0.5)
            .count();
        assert_eq!(big_jumps, 2, "Should detect 2 interleaved-contract jumps");
        let ratio = big_jumps as f64 / jumps.len() as f64;
        assert!(ratio > 0.01, "Jump ratio {ratio} should trigger regression gate");
    }
```

- [ ] **Step 2: Run all tests**

```bash
SQLX_OFFLINE=true cargo test -p ml --lib -p ml-features --lib 2>&1 | tail -5
```

- [ ] **Step 3: Rebuild fxcache from trades data locally**

```bash
SQLX_OFFLINE=true cargo run --example precompute_features -- \
    --data-dir test_data/futures-baseline \
    --trades-data-dir test_data/futures-baseline-trades \
    --mbp10-data-dir test_data/futures-baseline-mbp10 \
    --symbol ES.FUT \
    --output-dir test_data/feature-cache
```

Verify output:
- fxcache file created in `test_data/feature-cache/`
- Log shows `Price continuity: max_jump=<0.5%, jumps>0.5%=0/N`
- No `PRICE CONTINUITY FAILED` error

- [ ] **Step 4: Run training smoke test with new fxcache**

```bash
FOXHUNT_TEST_DATA=test_data/futures-baseline SQLX_OFFLINE=true cargo test -p ml --lib -- smoke_tests::walk_forward --ignored --nocapture 2>&1 | grep -E 'Per-action Q|Q-value gap|Sharpe=|WinRate'
```

Expected:
- WinRate > 30% (above random — proves direction signal exists)
- Q-value gap > 0.01 (model differentiates direction)
- No interleaved-contract price jumps in the data

- [ ] **Step 5: Commit + push**

```bash
git add -A
git commit -m "feat(data): volume bar pipeline complete — clean fxcache from trades, price continuity gate"
git push origin main
```

- [ ] **Step 6: Deploy to H100**

```bash
./scripts/argo-train.sh dqn --baseline --epochs 10
```

Monitor for WinRate > 30% and PF trending > 1.0.
