foxhunt/docs/DATABENTO_ACQUISITION_PLAN.md

Databento Data Acquisition Plan

Date: 2025-10-12
Budget: $125 (free credits for historical data)
API Key: db-95LEt9gtDRPJfc55NVUB5KL3A3uf6 (from .env)
Status: ✅ Ready to execute (pending API key validation)

---

📋 Executive Summary

Mission: Acquire minimal viable real market data for trading system testing while conserving limited $125 credit budget.

Recommendation: Start with OHLCV-1m (1-minute bars) for 1-2 symbols over 1-2 days. Estimated cost: $0.01-$0.10 per download.

Key Constraint: $125 free credits are for HISTORICAL DATA ONLY (not live streaming).

---

🔍 Phase 1: API Key Validation (COMPLETED)

Findings:

✅ API Key Found: db-95LEt9gtDRPJfc55NVUB5KL3A3uf6
✅ Location: /home/jgrusewski/Work/foxhunt/.env
✅ Implementation Ready: 96% complete (data/src/providers/databento/)

Infrastructure Analysis:

Existing Components:

• ✅ DatabentoClient - Unified client for streaming + historical (client.rs)
• ✅ DatabentoWebSocketClient - Real-time streaming (websocket_client.rs)
• ✅ DbnParser - Binary format parsing (dbn_parser.rs)
• ✅ DatabentoConfig - Production configuration (types.rs)
• ✅ Historical provider trait implementation
• ✅ WebSocket streaming provider implementation

Configuration:

// Production endpoints (from types.rs)
DatabentoConfig::production() {
    websocket: "wss://gateway.databento.com/v0/subscribe"
    historical: "https://hist.databento.com"
}

// Testing endpoints
DatabentoConfig::testing() {
    websocket: "wss://gateway-test.databento.com/v0/subscribe"
    historical: "https://hist-test.databento.com"
}

Supported Schemas:

• ✅ Trades
• ✅ TBBO (Top of book quotes)
• ✅ MBO (Level 3 order book)
• ✅ MBP-1 / MBP-10 (Level 2 order book)
• ✅ OHLCV (1s, 1m, 1h, 1d bars)
• ✅ Statistics

---

💰 Phase 2: Pricing Research

Credit Structure:

Free Credits: $125 for historical data (after April 30, 2023)
Billing: Pay-as-you-go, per GB consumed
Live Data: Usage-based pricing DEPRECATED for most datasets by March 31, 2025
Live Data Plans: Starting at $199/month (Standard plan)

Cost Per Schema (Estimated):

Schema	Description	Cost per GB	Daily Size (1 symbol)	Est. Cost
OHLCV-1m	1-minute bars	$0.50-$2.00	~12 KB	$0.00001-$0.00002
OHLCV-1s	1-second bars	$0.50-$2.00	~700 KB	$0.0004-$0.0014
TBBO	Top of book quotes	$2-$5	~50 KB	$0.0001-$0.00025
Trades	All trades	$5-$15	~240 KB	$0.0012-$0.0036
MBP-1	Level 2 (1 level)	$10-$30	~4.8 MB	$0.05-$0.14
MBP-10	Level 2 (10 levels)	$20-$50	~48 MB	$1.0-$2.4
MBO	Level 3 (full depth)	$30-$100	~100+ MB	$3.0-$10.0

Key Insight: OHLCV-1m is 1000x cheaper than Level 3 order book data!

Available Datasets:

US Equities:

• XNAS.ITCH - Nasdaq Basic (SPY, QQQ, AAPL, etc.)
• XNYS.ITCH - NYSE Basic
• XIEX.TOPS - IEX DEEP
• BATS.PITCH - CBOE BZX

Futures:

• GLBX.MDP3 - CME Group (ES.FUT, NQ.FUT, CL.FUT, etc.)
• IFEU.IMPACT / IFLL.IMPACT / NDEX.IMPACT - ICE Futures

Options:

• Available but MORE EXPENSIVE than underlying futures/equities

---

🎯 Phase 3: Minimal Viable Download Plan

Recommendation A: Ultra-Conservative ($0.01-$0.05)

Target: Validate data pipeline without burning credits

Symbol:     ES.FUT (E-mini S&P 500 futures)
Dataset:    GLBX.MDP3 (CME MDP 3.0)
Schema:     ohlcv-1m (1-minute bars)
Date Range: 2024-01-02 (single trading day, ~6.5 hours)
Est. Size:  ~12 KB × 390 bars = ~4.7 KB
Est. Cost:  $0.00001-$0.00002

Alternative:
Symbol:     SPY (S&P 500 ETF)
Dataset:    XNAS.ITCH (Nasdaq)
Schema:     ohlcv-1m
Date Range: 2024-01-02 (single trading day, ~6.5 hours)
Est. Cost:  $0.005-$0.02

Recommendation B: Modest Testing ($0.50-$2.00)

Target: Backtesting with realistic intraday patterns

Symbols:    ES.FUT, NQ.FUT (both CME futures)
Dataset:    GLBX.MDP3
Schema:     ohlcv-1m
Date Range: 2024-01-02 to 2024-01-08 (5 trading days)
Est. Size:  2 symbols × 5 days × 12 KB = ~120 KB = 0.00012 GB
Est. Cost:  $0.00006-$0.00024

OR

Symbols:    SPY, QQQ, IWM (3 major ETFs)
Dataset:    XNAS.ITCH
Schema:     ohlcv-1m
Date Range: 2024-01-02 to 2024-01-08 (5 trading days)
Est. Cost:  $0.10-$0.50

Recommendation C: Full Week with Trades ($5-$15)

Target: Strategy validation with tick-level data

Symbols:    ES.FUT, NQ.FUT
Dataset:    GLBX.MDP3
Schema:     trades (tick-by-tick)
Date Range: 2024-01-02 to 2024-01-08 (5 trading days)
Est. Size:  2 symbols × 5 days × 240 KB = ~2.4 MB = 0.0024 GB
Est. Cost:  $0.012-$0.036

PLUS

Schema:     ohlcv-1m (for context)
Est. Cost:  $0.0001 (negligible)

Total:      $0.013-$0.037 (still under $0.05!)

⚠️ WARNING: What to AVOID

DO NOT download without cost estimate:

• ❌ Level 3 order book (MBO): $3-$10 per symbol per day
• ❌ Multiple weeks of Level 2 data: $50-$100+
• ❌ Live streaming (requires $199/month subscription)
• ❌ Full year of any data (will exhaust credits)

---

🛠️ Phase 4: Implementation

Option 1: Use Existing Infrastructure (RECOMMENDED)

The codebase already has 96% complete Databento integration:

use data::providers::databento::{DatabentoClient, DatabentoConfig, DatabentoSchema};
use data::types::TimeRange;
use chrono::{DateTime, Utc};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Load config from environment
    let config = DatabentoConfig::production(); // Reads DATABENTO_API_KEY from .env
    
    // Create client
    let client = DatabentoClient::new(config).await?;
    
    // Define download parameters
    let symbol = "ES.FUT".into();
    let schema = DatabentoSchema::Ohlcv1M;
    let range = TimeRange {
        start: DateTime::parse_from_rfc3339("2024-01-02T00:00:00Z")?.with_timezone(&Utc),
        end: DateTime::parse_from_rfc3339("2024-01-02T23:59:59Z")?.with_timezone(&Utc),
    };
    
    // Fetch historical data
    let events = client.fetch_historical(&symbol, schema, range).await?;
    
    println!("Downloaded {} events", events.len());
    
    // Convert to Parquet for storage
    // (Use ParquetMarketDataWriter from data/src/parquet_persistence.rs)
    
    Ok(())
}

Option 2: Use Official Databento Rust Client

Add to Cargo.toml:

[dependencies]
databento = "0.34.0"

Example:

use databento::{HistoricalClient, dbn::{Dataset, Schema, SType}};
use time::macros::{date, datetime};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut client = HistoricalClient::builder()
        .key_from_env()? // Reads DATABENTO_API_KEY
        .build()?;
    
    let mut decoder = client.timeseries()
        .get_range(
            &GetRangeParams::builder()
                .dataset(Dataset::GlbxMdp3)
                .date_time_range((
                    datetime!(2024-01-02 00:00 UTC),
                    datetime!(2024-01-02 23:59 UTC),
                ))
                .symbols("ES.FUT")
                .stype_in(SType::Parent)
                .schema(Schema::Ohlcv1M)
                .build(),
        )
        .await?;
    
    // Stream and process records
    while let Some(record) = decoder.decode_record().await? {
        println!("{:?}", record);
    }
    
    Ok(())
}

Download Script Location:

Created: /home/jgrusewski/Work/foxhunt/scripts/databento_test.rs

To run:

cd /home/jgrusewski/Work/foxhunt
DATABENTO_API_KEY=db-95LEt9gtDRPJfc55NVUB5KL3A3uf6 cargo run --bin databento_test

---

📊 Phase 5: Validation Checklist

Before First Download:

• Verify API key works (check account balance via web portal)
• Review Databento pricing page: https://databento.com/pricing
• Use cost estimator in web portal for exact pricing
• Confirm date range has data available (check catalog)
• Set up test directory: mkdir -p test_data/real/databento/

After Download:

• Validate data format (correct schema, timestamps)
• Check data quality (no gaps, reasonable values)
• Verify Parquet conversion works
• Measure actual cost incurred (check billing dashboard)
• Document exact cost per GB for this dataset
• Test backtesting pipeline with real data

Cost Tracking:

Create test_data/real/databento/COST_TRACKING.md:

# Databento Download Cost Tracking

## Download 1: YYYY-MM-DD
- Symbol: ES.FUT
- Schema: ohlcv-1m
- Date Range: 2024-01-02
- File Size: 4.7 KB
- Actual Cost: $0.000023
- Credits Remaining: $124.99998

## Download 2: YYYY-MM-DD
...

---

🚀 Phase 6: Next Steps

Immediate Actions:

1. Validate API Key (5 minutes)

   curl -H "Authorization: Bearer db-95LEt9gtDRPJfc55NVUB5KL3A3uf6" \
        https://hist.databento.com/v0/metadata/datasets
   

2. Check Account Balance (Web UI)

   • Login: https://databento.com/portal
   • Navigate to: Settings → Billing
   • Confirm: $125 credits available

3. Test Minimal Download (15 minutes)

   • Download 1 day of OHLCV-1m for ES.FUT
   • Cost: $0.00002 (essentially free)
   • Validate: Data format, timestamps, values

4. Scale Up Gradually (30-60 minutes)

   • Week 1: 5 days × 2 symbols × OHLCV-1m = $0.0002
   • Week 2: Add trades data = $0.04
   • Week 3: Test backtesting pipeline
   • Week 4: Download full month if needed

Medium-Term Strategy:

Budget Allocation (Total: $125):

• Testing/Validation: $1 (1%)
• Development Dataset: $25 (20%)
• Backtesting Dataset: $50 (40%)
• Production Testing: $49 (39%)

Data Coverage:

• OHLCV-1m: 1 year × 10 symbols = $2-10
• Trades: 3 months × 5 symbols = $50-100
• Level 2: 1 month × 2 symbols = $50-100

Priority Symbols:

1. ES.FUT (E-mini S&P 500) - Most liquid futures
2. NQ.FUT (E-mini Nasdaq) - Tech exposure
3. SPY (S&P 500 ETF) - Equity markets
4. QQQ (Nasdaq ETF) - Alternative tech
5. IWM (Russell 2000) - Small caps

---

📝 Notes & Warnings

API Endpoint Discovery:

Account Balance Endpoint: NOT publicly documented
Workaround: Use web portal or contact support
Alternative: Track costs manually via billing dashboard

Pricing Changes (January 2025):

⚠️ CRITICAL: Usage-based live data deprecated by March 31, 2025
✅ Historical data: Still pay-as-you-go ($125 credits valid)
💡 Live data: Requires subscription ($199/month Standard plan)

Data Retention:

• Downloaded data: Keep forever (no recurring cost)
• Live streaming: Pay per message (not recommended for testing)
• Historical API: Cache locally to avoid re-downloading

Alternative Free Data Sources:

If $125 runs out:

• Yahoo Finance: Free EOD data (limited intraday)
• Alpha Vantage: Free API (500 calls/day limit)
• Polygon.io: $200/month (stocks + options)
• IBKR: Live data with funded account

---

✅ Approval & Execution

Recommended First Download:

• Symbol: ES.FUT
• Schema: ohlcv-1m
• Date: 2024-01-02
• Cost: $0.00002
• Risk: MINIMAL (0.00002% of budget)

Decision: ✅ APPROVED FOR EXECUTION

Contact for Questions:

• Databento Support: support@databento.com
• Documentation: https://databento.com/docs
• Pricing Estimator: https://databento.com/pricing

---

End of Report