foxhunt/DATABENTO_0.34_MIGRATION_GUIDE.md

# Databento 0.34+ Schema Enum API Migration Guide

## Overview

The new databento crate (0.34.1+) API requires using the **`Schema` enum** from the `dbn` crate instead of `String` for schema parameters. This document provides a comprehensive guide for updating outdated examples to use the new API pattern.

**Current Versions in Use:**
- `databento = "0.34"` (ML module)
- `dbn = "0.42.0"` (Data module)

---

## 1. Schema Enum Usage - How to Properly Use Schema Types

### New API Pattern (CORRECT)

```rust
use dbn::Schema;
use std::str::FromStr;

// Method 1: Parse from string (recommended for dynamic schemas)
let schema_enum = Schema::from_str("mbp-10")
    .context("Failed to parse schema")?;

// Method 2: Use direct enum variants (when schema is known at compile time)
let schema_enum = Schema::Mbp10;  // Level 2 Order Book - 10 levels
let schema_enum = Schema::Ohlcv1M;  // OHLCV 1-minute bars
let schema_enum = Schema::Trades;  // Trade records
let schema_enum = Schema::Tbbo;  // BEST BID/ASK (Top of Book)
let schema_enum = Schema::Mbo;  // Market By Order
```

### Available Schema Enum Variants

```rust
pub enum Schema {
    // Trade data
    Trades,           // Individual trade records
    
    // Quote data
    Tbbo,             // Top of Book (best bid/ask)
    Nbbo,             // National Best Bid/Offer
    
    // Order book (MBP variants)
    Mbp1,             // Market By Price, 1 level (bid/ask)
    Mbp10,            // Market By Price, 10 levels (L2 order book)
    
    // Market By Order (L3 order book)
    Mbo,              // Individual order level detail
    
    // OHLCV bars (multiple resolutions)
    Ohlcv1S,          // 1-second bars
    Ohlcv1M,          // 1-minute bars
    Ohlcv1H,          // 1-hour bars
    Ohlcv1D,          // 1-day bars
    
    // Additional schemas (dataset dependent)
    // Check databento documentation for complete list
}
```

### OLD API (INCORRECT - DEPRECATED)

```rust
// ❌ WRONG - This pattern no longer works with databento 0.34+
let schema_string = "mbp-10";  // String type - NO LONGER ACCEPTED
let response = client.timeseries()
    .get_range(&params)
    .schema(schema_string)  // ❌ Type error: expected Schema enum
    .await?;
```

---

## 2. Date Range Handling - The New API Pattern

### Working Example (from download_l2_data.rs)

The new databento API uses the `DateTimeRange` type from the `time` crate instead of `.start()` method:

```rust
use time::{PrimitiveDateTime, Date, Time, UtcOffset};
use databento::historical::DateTimeRange;

// Parse date string to Date type
let date_obj = Date::parse(date, &time::format_description::parse("[year]-[month]-[day]")?)?;

// Create PrimitiveDateTime for start of day (UTC midnight)
let start_dt = PrimitiveDateTime::new(date_obj, Time::MIDNIGHT)
    .assume_offset(UtcOffset::UTC);

// Create end of day (24 hours later)
let end_dt = start_dt + time::Duration::days(1);

// Convert tuple to DateTimeRange
let date_time_range: DateTimeRange = (start_dt, end_dt).into();

// Use in GetRangeParams
let params = GetRangeParams::builder()
    .dataset("GLBX.MDP3".to_string())
    .symbols(vec![symbol.to_string()])
    .schema(Schema::Mbp10)
    .date_time_range(date_time_range)  // ✅ CORRECT
    .build();

// NO MORE .start() and .end() methods - they don't exist in 0.34+
// ❌ WRONG: params.start(start_dt).end(end_dt)
```

### Chrono Integration (Alternative Pattern)

```rust
use chrono::{DateTime, Utc, NaiveDate};

// Using chrono for date parsing
let date = "2024-01-02";
let naive_date = NaiveDate::parse_from_str(date, "%Y-%m-%d")?;
let start = DateTime::<Utc>::from_naive_utc_and_offset(
    naive_date.and_hms_opt(0, 0, 0).unwrap(),
    Utc
);
let end = start + chrono::Duration::days(1);

// If GetRangeParams accepts chrono types, use directly
// Otherwise, convert to time crate types
```

---

## 3. AsyncDbnDecoder Response Handling

### Pattern from Working Example

The new API returns a response type that implements `AsyncRead`. You must read it asynchronously:

```rust
use tokio::io::AsyncReadExt;
use databento::HistoricalClient;

// Make the request (returns AsyncDbnDecoder)
let mut decoder = client
    .timeseries()
    .get_range(&params)
    .await?;

// Read all data into buffer
let mut buffer = Vec::new();
let mut temp_buf = vec![0u8; 8192];  // 8KB read chunks

loop {
    // AsyncReadExt trait provides read() method
    let n = decoder.get_mut().read(&mut temp_buf).await?;
    if n == 0 {
        break;  // EOF
    }
    buffer.extend_from_slice(&temp_buf[..n]);
}

// Now buffer contains the complete DBN-encoded data
let size = buffer.len() as u64;

// Validate size (should be >1 KB for a trading day)
if size < 1024 {
    return Ok(None);  // Likely no data (holiday/no trading)
}

// Write to file
fs::write(&output_file, &buffer)?;
```

### Key Points:
- `decoder` implements `tokio::io::AsyncRead`
- Must use `.get_mut()` to access the inner reader
- Use `AsyncReadExt::read()` for async reading
- Read into a buffer in chunks to handle large files
- No automatic decoding - you get raw DBN binary data

---

## 4. Working Example Patterns

### Complete Minimal Example

```rust
use anyhow::{Context, Result};
use databento::historical::timeseries::GetRangeParams;
use databento::{HistoricalClient, historical::DateTimeRange};
use dbn::Schema;
use std::str::FromStr;
use tokio::io::AsyncReadExt;
use time::{PrimitiveDateTime, Date, Time, UtcOffset};

#[tokio::main]
async fn main() -> Result<()> {
    // 1. Initialize client
    let api_key = std::env::var("DATABENTO_API_KEY")?;
    let mut client = HistoricalClient::builder()
        .key(api_key)?
        .build()?;

    // 2. Parse schema using Schema enum
    let schema = Schema::from_str("mbp-10")?;

    // 3. Create date range using time crate
    let date_str = "2024-01-02";
    let date_obj = Date::parse(
        date_str, 
        &time::format_description::parse("[year]-[month]-[day]")?
    )?;
    let start_dt = PrimitiveDateTime::new(date_obj, Time::MIDNIGHT)
        .assume_offset(UtcOffset::UTC);
    let end_dt = start_dt + time::Duration::days(1);
    let date_time_range: DateTimeRange = (start_dt, end_dt).into();

    // 4. Build request with Schema enum
    let params = GetRangeParams::builder()
        .dataset("GLBX.MDP3".to_string())
        .symbols(vec!["ES.FUT".to_string()])
        .schema(schema)  // ✅ Schema enum, not String
        .date_time_range(date_time_range)  // ✅ DateTimeRange, not .start()/.end()
        .build();

    // 5. Execute request and handle AsyncDbnDecoder
    let mut decoder = client.timeseries().get_range(&params).await?;

    // 6. Read data asynchronously
    let mut buffer = Vec::new();
    let mut temp_buf = vec![0u8; 8192];
    loop {
        let n = decoder.get_mut().read(&mut temp_buf).await?;
        if n == 0 { break; }
        buffer.extend_from_slice(&temp_buf[..n]);
    }

    println!("Downloaded {} bytes", buffer.len());
    Ok(())
}
```

### From Real Codebase: download_l2_data.rs

**File:** `/home/jgrusewski/Work/foxhunt/ml/examples/download_l2_data.rs`

Key code sections:

```rust
// Lines 34-36: Import Schema enum
use databento::historical::timeseries::GetRangeParams;
use databento::{HistoricalClient, historical::DateTimeRange};
use dbn::Schema;

// Lines 138-139: Parse schema from string
let schema_enum = Schema::from_str("mbp-10")
    .context("Failed to parse schema")?;

// Lines 132-136: Create DateTimeRange (NOT .start()/.end())
let date_obj = Date::parse(date, &time::format_description::parse("[year]-[month]-[day]")?)?;
let start_dt = PrimitiveDateTime::new(date_obj, Time::MIDNIGHT).assume_offset(UtcOffset::UTC);
let end_dt = start_dt + time::Duration::days(1);
let date_time_range: DateTimeRange = (start_dt, end_dt).into();

// Lines 142-147: Build params with Schema enum
let params = GetRangeParams::builder()
    .dataset("GLBX.MDP3".to_string())
    .symbols(vec![symbol.to_string()])
    .schema(schema_enum)  // ✅ Schema enum
    .date_time_range(date_time_range)  // ✅ DateTimeRange
    .build();

// Lines 154-165: Handle AsyncDbnDecoder response
let mut decoder = client.timeseries().get_range(&params).await?;
let mut buffer = Vec::new();
let mut temp_buf = vec![0u8; 8192];
loop {
    let n = decoder.get_mut().read(&mut temp_buf).await?;
    if n == 0 { break; }
    buffer.extend_from_slice(&temp_buf[..n]);
}
```

---

## 5. Common Migration Issues & Fixes

### Issue 1: Type Mismatch - String Instead of Schema Enum

```rust
// ❌ WRONG
let params = GetRangeParams::builder()
    .schema("mbp-10".to_string())  // Type error!
    .build();

// ✅ CORRECT
use dbn::Schema;
use std::str::FromStr;

let params = GetRangeParams::builder()
    .schema(Schema::from_str("mbp-10")?)  // Parse to enum
    .build();

// ✅ OR use direct variant
let params = GetRangeParams::builder()
    .schema(Schema::Mbp10)  // Direct enum
    .build();
```

### Issue 2: No .start() Method on Builder

```rust
// ❌ WRONG - method doesn't exist
let params = GetRangeParams::builder()
    .start(start_dt)
    .end(end_dt)
    .build();

// ✅ CORRECT - use date_time_range
use databento::historical::DateTimeRange;
use time::{PrimitiveDateTime, UtcOffset};

let range: DateTimeRange = (start_dt, end_dt).into();
let params = GetRangeParams::builder()
    .date_time_range(range)
    .build();
```

### Issue 3: Not Handling AsyncRead Response

```rust
// ❌ WRONG - treat response as synchronous
let data = client.timeseries().get_range(&params).await?;
// Compiler error: response is AsyncReader, not Vec<u8>

// ✅ CORRECT - use async read methods
use tokio::io::AsyncReadExt;

let mut decoder = client.timeseries().get_range(&params).await?;
let mut buffer = Vec::new();
let mut temp_buf = vec![0u8; 8192];
loop {
    let n = decoder.get_mut().read(&mut temp_buf).await?;
    if n == 0 { break; }
    buffer.extend_from_slice(&temp_buf[..n]);
}
```

---

## 6. Cargo Dependencies for New API

```toml
# Cargo.toml
[dependencies]
databento = "0.34"  # Historical client with Schema support
dbn = "0.42.0"      # Schema enum and DBN decoding
tokio = { version = "1", features = ["full"] }
tokio-io = "0.1"    # For AsyncReadExt
time = "0.3"        # For DateTimeRange handling
chrono = "0.4"      # Alternative date handling
anyhow = "1"        # Error handling
```

---

## 7. Updated Examples in Codebase

### Example 1: ml/examples/download_l2_data.rs

**Status:** ✅ WORKING - Uses new API correctly

**Key patterns:**
- Uses `Schema::from_str()` for schema parsing
- Uses `time` crate for `DateTimeRange` 
- Properly handles `AsyncDbnDecoder` with async read

**Location:** `/home/jgrusewski/Work/foxhunt/ml/examples/download_l2_data.rs`

### Example 2: data/examples/download_mbp10_data.rs

**Status:** ⚠️ PARTIAL - Uses HTTP batch API (different pattern)

**Uses:**
- Batch submit API (different from direct timeseries download)
- Direct HTTP requests instead of client SDK
- Raw HTTP JSON API instead of Schema enum

**Location:** `/home/jgrusewski/Work/foxhunt/data/examples/download_mbp10_data.rs`

### Example 3: data/examples/test_databento_download.rs

**Status:** ⚠️ PARTIAL - Uses HTTP timeseries API

**Uses:**
- Direct HTTP GET requests with query parameters
- Schema specified as string in query parameter
- Not using databento client SDK

**Location:** `/home/jgrusewski/Work/foxhunt/data/examples/test_databento_download.rs`

---

## 8. Migration Checklist

When updating old databento examples to 0.34+:

- [ ] Replace `schema: "string"` with `schema(Schema::from_str("string")?)`
- [ ] Replace `.start(dt).end(dt)` with `.date_time_range((start, end).into())`
- [ ] Add `use dbn::Schema;` import
- [ ] Add `use databento::historical::DateTimeRange;` import
- [ ] Add `use tokio::io::AsyncReadExt;` for response handling
- [ ] Change synchronous response handling to async with buffer loop
- [ ] Use `decoder.get_mut().read(&mut buf).await?` instead of expecting Vec<u8>
- [ ] Update Cargo.toml dependencies (databento = "0.34"+, dbn = "0.42"+)
- [ ] Test with real API key against Databento staging environment

---

## 9. Quick Reference - API Comparison

| Old Pattern (Pre-0.34) | New Pattern (0.34+) |
|---|---|
| `.schema("mbp-10")` | `.schema(Schema::from_str("mbp-10")?)` |
| `.schema("ohlcv-1m")` | `.schema(Schema::Ohlcv1M)` |
| `.start(dt).end(dt)` | `.date_time_range((start, end).into())` |
| Response: `Vec<u8>` (sync) | Response: `AsyncDbnDecoder` (async) |
| `response.bytes()` | `decoder.get_mut().read(&mut buf).await?` |
| String schema type | `dbn::Schema` enum |
| Direct date assignment | `time::DateTimeRange` tuple |

---

## 10. Resources

**Official Documentation:**
- Databento Python: https://github.com/databento/databento-rs
- DBN Format: https://databento.com/docs/api/encoding/dbn
- Schema Reference: https://databento.com/docs/api/reference/data/schema

**File Locations in Foxhunt:**
- Working example: `/home/jgrusewski/Work/foxhunt/ml/examples/download_l2_data.rs`
- Parser wrapper: `/home/jgrusewski/Work/foxhunt/data/src/providers/databento/parser.rs`
- Client wrapper: `/home/jgrusewski/Work/foxhunt/data/src/providers/databento/client.rs`
- Module: `/home/jgrusewski/Work/foxhunt/data/src/providers/databento/mod.rs`