foxhunt/DATABENTO_0.34_MIGRATION_GUIDE.md

Databento 0.34+ Schema Enum API Migration Guide

Overview

The new databento crate (0.34.1+) API requires using the Schema enum from the dbn crate instead of String for schema parameters. This document provides a comprehensive guide for updating outdated examples to use the new API pattern.

Current Versions in Use:

• databento = "0.34" (ML module)
• dbn = "0.42.0" (Data module)

---

1. Schema Enum Usage - How to Properly Use Schema Types

New API Pattern (CORRECT)

use dbn::Schema;
use std::str::FromStr;

// Method 1: Parse from string (recommended for dynamic schemas)
let schema_enum = Schema::from_str("mbp-10")
    .context("Failed to parse schema")?;

// Method 2: Use direct enum variants (when schema is known at compile time)
let schema_enum = Schema::Mbp10;  // Level 2 Order Book - 10 levels
let schema_enum = Schema::Ohlcv1M;  // OHLCV 1-minute bars
let schema_enum = Schema::Trades;  // Trade records
let schema_enum = Schema::Tbbo;  // BEST BID/ASK (Top of Book)
let schema_enum = Schema::Mbo;  // Market By Order

Available Schema Enum Variants

pub enum Schema {
    // Trade data
    Trades,           // Individual trade records
    
    // Quote data
    Tbbo,             // Top of Book (best bid/ask)
    Nbbo,             // National Best Bid/Offer
    
    // Order book (MBP variants)
    Mbp1,             // Market By Price, 1 level (bid/ask)
    Mbp10,            // Market By Price, 10 levels (L2 order book)
    
    // Market By Order (L3 order book)
    Mbo,              // Individual order level detail
    
    // OHLCV bars (multiple resolutions)
    Ohlcv1S,          // 1-second bars
    Ohlcv1M,          // 1-minute bars
    Ohlcv1H,          // 1-hour bars
    Ohlcv1D,          // 1-day bars
    
    // Additional schemas (dataset dependent)
    // Check databento documentation for complete list
}

OLD API (INCORRECT - DEPRECATED)

// ❌ WRONG - This pattern no longer works with databento 0.34+
let schema_string = "mbp-10";  // String type - NO LONGER ACCEPTED
let response = client.timeseries()
    .get_range(&params)
    .schema(schema_string)  // ❌ Type error: expected Schema enum
    .await?;

---

2. Date Range Handling - The New API Pattern

Working Example (from download_l2_data.rs)

The new databento API uses the DateTimeRange type from the time crate instead of .start() method:

use time::{PrimitiveDateTime, Date, Time, UtcOffset};
use databento::historical::DateTimeRange;

// Parse date string to Date type
let date_obj = Date::parse(date, &time::format_description::parse("[year]-[month]-[day]")?)?;

// Create PrimitiveDateTime for start of day (UTC midnight)
let start_dt = PrimitiveDateTime::new(date_obj, Time::MIDNIGHT)
    .assume_offset(UtcOffset::UTC);

// Create end of day (24 hours later)
let end_dt = start_dt + time::Duration::days(1);

// Convert tuple to DateTimeRange
let date_time_range: DateTimeRange = (start_dt, end_dt).into();

// Use in GetRangeParams
let params = GetRangeParams::builder()
    .dataset("GLBX.MDP3".to_string())
    .symbols(vec![symbol.to_string()])
    .schema(Schema::Mbp10)
    .date_time_range(date_time_range)  // ✅ CORRECT
    .build();

// NO MORE .start() and .end() methods - they don't exist in 0.34+
// ❌ WRONG: params.start(start_dt).end(end_dt)

Chrono Integration (Alternative Pattern)

use chrono::{DateTime, Utc, NaiveDate};

// Using chrono for date parsing
let date = "2024-01-02";
let naive_date = NaiveDate::parse_from_str(date, "%Y-%m-%d")?;
let start = DateTime::<Utc>::from_naive_utc_and_offset(
    naive_date.and_hms_opt(0, 0, 0).unwrap(),
    Utc
);
let end = start + chrono::Duration::days(1);

// If GetRangeParams accepts chrono types, use directly
// Otherwise, convert to time crate types

---

3. AsyncDbnDecoder Response Handling

Pattern from Working Example

The new API returns a response type that implements AsyncRead. You must read it asynchronously:

use tokio::io::AsyncReadExt;
use databento::HistoricalClient;

// Make the request (returns AsyncDbnDecoder)
let mut decoder = client
    .timeseries()
    .get_range(&params)
    .await?;

// Read all data into buffer
let mut buffer = Vec::new();
let mut temp_buf = vec![0u8; 8192];  // 8KB read chunks

loop {
    // AsyncReadExt trait provides read() method
    let n = decoder.get_mut().read(&mut temp_buf).await?;
    if n == 0 {
        break;  // EOF
    }
    buffer.extend_from_slice(&temp_buf[..n]);
}

// Now buffer contains the complete DBN-encoded data
let size = buffer.len() as u64;

// Validate size (should be >1 KB for a trading day)
if size < 1024 {
    return Ok(None);  // Likely no data (holiday/no trading)
}

// Write to file
fs::write(&output_file, &buffer)?;

Key Points:

• decoder implements tokio::io::AsyncRead
• Must use .get_mut() to access the inner reader
• Use AsyncReadExt::read() for async reading
• Read into a buffer in chunks to handle large files
• No automatic decoding - you get raw DBN binary data

---

4. Working Example Patterns

Complete Minimal Example

use anyhow::{Context, Result};
use databento::historical::timeseries::GetRangeParams;
use databento::{HistoricalClient, historical::DateTimeRange};
use dbn::Schema;
use std::str::FromStr;
use tokio::io::AsyncReadExt;
use time::{PrimitiveDateTime, Date, Time, UtcOffset};

#[tokio::main]
async fn main() -> Result<()> {
    // 1. Initialize client
    let api_key = std::env::var("DATABENTO_API_KEY")?;
    let mut client = HistoricalClient::builder()
        .key(api_key)?
        .build()?;

    // 2. Parse schema using Schema enum
    let schema = Schema::from_str("mbp-10")?;

    // 3. Create date range using time crate
    let date_str = "2024-01-02";
    let date_obj = Date::parse(
        date_str, 
        &time::format_description::parse("[year]-[month]-[day]")?
    )?;
    let start_dt = PrimitiveDateTime::new(date_obj, Time::MIDNIGHT)
        .assume_offset(UtcOffset::UTC);
    let end_dt = start_dt + time::Duration::days(1);
    let date_time_range: DateTimeRange = (start_dt, end_dt).into();

    // 4. Build request with Schema enum
    let params = GetRangeParams::builder()
        .dataset("GLBX.MDP3".to_string())
        .symbols(vec!["ES.FUT".to_string()])
        .schema(schema)  // ✅ Schema enum, not String
        .date_time_range(date_time_range)  // ✅ DateTimeRange, not .start()/.end()
        .build();

    // 5. Execute request and handle AsyncDbnDecoder
    let mut decoder = client.timeseries().get_range(&params).await?;

    // 6. Read data asynchronously
    let mut buffer = Vec::new();
    let mut temp_buf = vec![0u8; 8192];
    loop {
        let n = decoder.get_mut().read(&mut temp_buf).await?;
        if n == 0 { break; }
        buffer.extend_from_slice(&temp_buf[..n]);
    }

    println!("Downloaded {} bytes", buffer.len());
    Ok(())
}

From Real Codebase: download_l2_data.rs

File: /home/jgrusewski/Work/foxhunt/ml/examples/download_l2_data.rs

Key code sections:

// Lines 34-36: Import Schema enum
use databento::historical::timeseries::GetRangeParams;
use databento::{HistoricalClient, historical::DateTimeRange};
use dbn::Schema;

// Lines 138-139: Parse schema from string
let schema_enum = Schema::from_str("mbp-10")
    .context("Failed to parse schema")?;

// Lines 132-136: Create DateTimeRange (NOT .start()/.end())
let date_obj = Date::parse(date, &time::format_description::parse("[year]-[month]-[day]")?)?;
let start_dt = PrimitiveDateTime::new(date_obj, Time::MIDNIGHT).assume_offset(UtcOffset::UTC);
let end_dt = start_dt + time::Duration::days(1);
let date_time_range: DateTimeRange = (start_dt, end_dt).into();

// Lines 142-147: Build params with Schema enum
let params = GetRangeParams::builder()
    .dataset("GLBX.MDP3".to_string())
    .symbols(vec![symbol.to_string()])
    .schema(schema_enum)  // ✅ Schema enum
    .date_time_range(date_time_range)  // ✅ DateTimeRange
    .build();

// Lines 154-165: Handle AsyncDbnDecoder response
let mut decoder = client.timeseries().get_range(&params).await?;
let mut buffer = Vec::new();
let mut temp_buf = vec![0u8; 8192];
loop {
    let n = decoder.get_mut().read(&mut temp_buf).await?;
    if n == 0 { break; }
    buffer.extend_from_slice(&temp_buf[..n]);
}

---

5. Common Migration Issues & Fixes

Issue 1: Type Mismatch - String Instead of Schema Enum

// ❌ WRONG
let params = GetRangeParams::builder()
    .schema("mbp-10".to_string())  // Type error!
    .build();

// ✅ CORRECT
use dbn::Schema;
use std::str::FromStr;

let params = GetRangeParams::builder()
    .schema(Schema::from_str("mbp-10")?)  // Parse to enum
    .build();

// ✅ OR use direct variant
let params = GetRangeParams::builder()
    .schema(Schema::Mbp10)  // Direct enum
    .build();

Issue 2: No .start() Method on Builder

// ❌ WRONG - method doesn't exist
let params = GetRangeParams::builder()
    .start(start_dt)
    .end(end_dt)
    .build();

// ✅ CORRECT - use date_time_range
use databento::historical::DateTimeRange;
use time::{PrimitiveDateTime, UtcOffset};

let range: DateTimeRange = (start_dt, end_dt).into();
let params = GetRangeParams::builder()
    .date_time_range(range)
    .build();

Issue 3: Not Handling AsyncRead Response

// ❌ WRONG - treat response as synchronous
let data = client.timeseries().get_range(&params).await?;
// Compiler error: response is AsyncReader, not Vec<u8>

// ✅ CORRECT - use async read methods
use tokio::io::AsyncReadExt;

let mut decoder = client.timeseries().get_range(&params).await?;
let mut buffer = Vec::new();
let mut temp_buf = vec![0u8; 8192];
loop {
    let n = decoder.get_mut().read(&mut temp_buf).await?;
    if n == 0 { break; }
    buffer.extend_from_slice(&temp_buf[..n]);
}

---

6. Cargo Dependencies for New API

# Cargo.toml
[dependencies]
databento = "0.34"  # Historical client with Schema support
dbn = "0.42.0"      # Schema enum and DBN decoding
tokio = { version = "1", features = ["full"] }
tokio-io = "0.1"    # For AsyncReadExt
time = "0.3"        # For DateTimeRange handling
chrono = "0.4"      # Alternative date handling
anyhow = "1"        # Error handling

---

7. Updated Examples in Codebase

Example 1: ml/examples/download_l2_data.rs

Status: ✅ WORKING - Uses new API correctly

Key patterns:

• Uses Schema::from_str() for schema parsing
• Uses time crate for DateTimeRange
• Properly handles AsyncDbnDecoder with async read

Location: /home/jgrusewski/Work/foxhunt/ml/examples/download_l2_data.rs

Example 2: data/examples/download_mbp10_data.rs

Status: ⚠️ PARTIAL - Uses HTTP batch API (different pattern)

Uses:

• Batch submit API (different from direct timeseries download)
• Direct HTTP requests instead of client SDK
• Raw HTTP JSON API instead of Schema enum

Location: /home/jgrusewski/Work/foxhunt/data/examples/download_mbp10_data.rs

Example 3: data/examples/test_databento_download.rs

Status: ⚠️ PARTIAL - Uses HTTP timeseries API

Uses:

• Direct HTTP GET requests with query parameters
• Schema specified as string in query parameter
• Not using databento client SDK

Location: /home/jgrusewski/Work/foxhunt/data/examples/test_databento_download.rs

---

8. Migration Checklist

When updating old databento examples to 0.34+:

• Replace schema: "string" with schema(Schema::from_str("string")?)
• Replace .start(dt).end(dt) with .date_time_range((start, end).into())
• Add use dbn::Schema; import
• Add use databento::historical::DateTimeRange; import
• Add use tokio::io::AsyncReadExt; for response handling
• Change synchronous response handling to async with buffer loop
• Use decoder.get_mut().read(&mut buf).await? instead of expecting Vec
• Update Cargo.toml dependencies (databento = "0.34"+, dbn = "0.42"+)
• Test with real API key against Databento staging environment

---

9. Quick Reference - API Comparison

Old Pattern (Pre-0.34)	New Pattern (0.34+)
.schema("mbp-10")	.schema(Schema::from_str("mbp-10")?)
.schema("ohlcv-1m")	.schema(Schema::Ohlcv1M)
.start(dt).end(dt)	.date_time_range((start, end).into())
Response: Vec<u8> (sync)	Response: AsyncDbnDecoder (async)
response.bytes()	decoder.get_mut().read(&mut buf).await?
String schema type	dbn::Schema enum
Direct date assignment	time::DateTimeRange tuple

---

10. Resources

Official Documentation:

• Databento Python: https://github.com/databento/databento-rs
• DBN Format: https://databento.com/docs/api/encoding/dbn
• Schema Reference: https://databento.com/docs/api/reference/data/schema

File Locations in Foxhunt:

• Working example: /home/jgrusewski/Work/foxhunt/ml/examples/download_l2_data.rs
• Parser wrapper: /home/jgrusewski/Work/foxhunt/data/src/providers/databento/parser.rs
• Client wrapper: /home/jgrusewski/Work/foxhunt/data/src/providers/databento/client.rs
• Module: /home/jgrusewski/Work/foxhunt/data/src/providers/databento/mod.rs