Files
foxhunt/databento_pricing_research.md
jgrusewski e8a68ee39f Download 360 DBN files (36.3 MB) using Rust databento client
- Created data/examples/download_ml_training_data.rs using reqwest + Databento HTTP API
- Downloaded 90 days × 4 symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT)
- Files saved to test_data/real/databento/ml_training/
- Total: 360 files, 15 MB compressed DBN format
- Used existing Rust pattern from download_nq_fut.rs
- API key loaded from .env file
- 100% success rate (360/360 files)
- Ready for ML training benchmarks

Next: Create simplified training benchmark for RTX 3050 Ti GPU measurements
2025-10-13 13:30:02 +02:00

24 KiB
Raw Blame History

Databento Pricing Research - Small Scale Trading

Research Date: 2025-10-12
Target Audience: Limited budget deployments, small-scale traders, startups


Executive Summary

Databento offers a $125 free credit for historical data (best entry point) and tiered subscription plans starting at $179-199/month for live streaming. Historical data uses pay-as-you-go pricing ($/GB), while live data is transitioning from usage-based to subscription-only by March 2025.

Best Option for Budget-Constrained Trading: Start with $125 free credits for historical data, then upgrade to subscription plans only when live data is required.


Free Tier / Sandbox

Available: YES - $125 Free Credits

  • Credit Amount: $125 in free data credits for historical data
  • Eligibility: All users who registered after April 30, 2023
  • Limitations:
    • Historical data only (no live/real-time data)
    • Credits expire (check account for expiration date)
    • Pay-as-you-go pricing deducts from credits
    • Cannot be used for live streaming subscriptions
  • How to Access:
    1. Sign up at databento.com
    2. Credits automatically applied to account
    3. Use Historical API to consume credits
    4. Monitor usage in portal dashboard

Testing Without Spending

  • Use $125 credits to download historical tick data, OHLCV bars, order books
  • Test backtesting strategies with real exchange data
  • Experiment with different schemas (trades, MBP-1, OHLCV) before committing
  • No sandbox/demo mode - you work with real production data using credits

Minimum Cost Entry

Absolute Cheapest Option: $0 (Free Credits)

What You Get with $125 Credits:

  • Historical data priced per GB (varies by dataset/schema)
  • Billing based on uncompressed binary encoding size
  • Data metered by bytes consumed (not time period)
  • Multiple schemas available at different price points

After Free Credits: Usage-Based Historical Data

Pricing Model: $/GB (varies by dataset and schema)

  • Cheapest schemas: OHLCV bars (aggregate data) - lowest GB/day
  • Mid-range: Trades, BBO (best bid/offer)
  • Most expensive: Full order book (MBO - market by order)

Estimated Costs (based on research, exact pricing in Data Catalog):

  • OHLCV-1m bars: ~$0.10-0.50 per symbol per day (very low data volume)
  • Trades schema: ~$1-5 per symbol per day (moderate volume)
  • MBP-1 (BBO): ~$2-8 per symbol per day (moderate-high volume)
  • MBO (full book): ~$10-50+ per symbol per day (high volume)

Important Notes:

  • Prices vary significantly by venue (CME, Nasdaq, OPRA, etc.)
  • Crypto datasets may have different pricing than equities/futures
  • Use Data Catalog price estimator for exact costs
  • Historical data remains pay-as-you-go (no subscription required)

Scaling Costs

Historical Data Estimates (Approximate)

Scenario 1: Ultra-Budget (OHLCV bars only)

  • 1 symbol, 1 day: $0.20 (BTC futures, 1-minute bars)
  • 5 symbols, 1 week: $7.00 (5 × 7 × $0.20)
  • 10 symbols, 1 month: $60 (10 × 30 × $0.20)

Scenario 2: Moderate (Trades schema)

  • 1 symbol, 1 day: $2.50 (BTC futures, all trades)
  • 5 symbols, 1 week: $87.50 (5 × 7 × $2.50)
  • 10 symbols, 1 month: $750 (10 × 30 × $2.50)

Scenario 3: Advanced (MBP-1 / BBO)

  • 1 symbol, 1 day: $5.00 (tick-level BBO updates)
  • 5 symbols, 1 week: $175 (5 × 7 × $5.00)
  • 10 symbols, 1 month: $1,500 (10 × 30 × $5.00)

Scenario 4: Professional (Full Order Book - MBO)

  • 1 symbol, 1 day: $25 (every order book change)
  • 5 symbols, 1 week: $875 (5 × 7 × $25)
  • 10 symbols, 1 month: $7,500 (10 × 30 × $25)

Live Data Subscription Plans (NEW 2025)

⚠️ MAJOR CHANGE: Usage-based live data being phased out by March 31, 2025

Subscription Tiers (Effective January 13, 2025)

Plan Monthly Cost Description Best For
Usage-Based Pay per message ⚠️ Being deprecated (March 2025) Legacy users only
Standard $179-199/month Cost-effective flat rate Small traders, startups
Plus Contact sales Mid-tier enterprise Growing firms
Unlimited Contact sales Full enterprise Large institutions

Dataset-Specific Pricing:

  • CME (Futures): $179/month Standard plan
  • OPRA (Options): $199/month Standard plan
  • Databento US Equities: $199/month Standard plan
  • Other datasets: Check Data Catalog for specific pricing

What's Included (Standard Plan):

  • Real-time streaming data (last 24 hours intraday)
  • Personal/non-professional use license
  • No long-term contracts (cancel anytime)
  • Pass-through exchange fees (no Databento markup)
  • 2.9% Stripe processing fee on license fees

Grandfathering Policy:

  • Existing users with active licenses can keep usage-based pricing
  • Access terminates if license is cancelled
  • New users cannot access usage-based live data

Cheapest Dataset Recommendations

For Historical Data (Using Free Credits)

1. CME Bitcoin Futures (BTC) RECOMMENDED

  • Why: Single liquid crypto futures contract
  • Venue: CME GLBX.MDP3
  • Schema: Start with OHLCV-1m bars (cheapest), upgrade to Trades as needed
  • Cost: ~$0.20-2.50/day depending on schema
  • Data Depth: 12+ years of history available
  • Use Case: Crypto futures trading, low-cost backtesting

2. CME Micro E-mini S&P 500 (MES)

  • Why: Fractional equity index exposure, lower data volume than ES
  • Schema: OHLCV-1m or Trades
  • Cost: ~$0.30-3.00/day depending on schema
  • Use Case: Equity index trading, smaller contract size

3. Single Equity (e.g., AAPL, MSFT)

  • Why: Test stock trading strategies with real data
  • Venue: Nasdaq TotalView-ITCH or Databento US Equities
  • Schema: Trades (mid-range cost)
  • Cost: ~$1-5/day per symbol
  • Note: Equities require subscription for live data ($199/month Standard)

4. Crypto Exchange Data (if available)

  • Check Data Catalog for direct crypto exchange feeds (Coinbase, Kraken, etc.)
  • May have different pricing than CME crypto futures
  • Contact Databento for crypto venue availability

Per-Symbol vs Flat Rate

Historical Data: Always per-symbol, per-GB pricing

  • More symbols = higher cost (linear scaling)
  • Optimize by selecting only needed symbols
  • Use OHLCV bars to minimize data volume

Live Data: Subscription plans (post-March 2025)

  • Flat monthly rate per dataset
  • Unlimited symbols within dataset (subject to exchange rules)
  • Better value for multi-symbol strategies

Discounts Available

Student Discount: YES (Indirect)

UC Berkeley Partnership Example:

  • Databento partners with top universities (UC Berkeley MFE program)
  • Academic institutions can create team accounts at no additional per-user cost
  • Students get access to historical data through university programs
  • How to Access: Check if your university has a Databento partnership
  • Contact: Reach out to Databento sales for academic/student pricing

Individual Students:

  • No explicit student discount advertised
  • $125 free credits available to all users (including students)
  • May negotiate custom pricing for academic research projects

Startup Discount: ⚠️ POSSIBLE (Contact Sales)

Startup Program:

  • Databento markets specifically to startups on their website
  • "Explore for free, onboard in minutes, pay for what you use" messaging
  • Likely custom pricing for early-stage companies
  • Requirements: Unknown (not publicly documented)
  • How to Apply: Contact Databento sales team directly
  • Link: https://databento.com/startups

Indicators of Startup-Friendly Pricing:

  • No long-term contracts (cancel anytime)
  • Pay-as-you-go historical data (no upfront commitment)
  • $125 free credits to test before buying
  • Standard plan at $179-199/month (vs $1000+ enterprise competitors)

Volume Discount: YES (Bulk Purchase)

Historical Data:

  • Bulk historical data downloads may qualify for discounts
  • Contact sales for large multi-year data purchases
  • Custom pricing for research institutions, hedge funds

Live Data Subscriptions:

  • Plus and Unlimited tiers for higher volume users
  • Enterprise pricing available (contact sales)
  • Multi-dataset bundles may offer savings

Other Discount Opportunities

1. Multiple Datasets: Bundle pricing possible (contact sales) 2. Annual Prepayment: Potential discount vs monthly billing (unconfirmed) 3. Non-Profit/Research: May qualify for academic pricing 4. Credit Farming Warning: ⚠️ Do NOT create multiple accounts for free credits (Databento detects and bans)


Data Type Cost Comparison

Schemas Ranked by Cost (Cheapest to Most Expensive)

Schema Description Data Volume Cost Tier Best For
OHLCV-1m 1-minute bars (O/H/L/C/V) Lowest $ Backtesting, low-freq strategies
OHLCV-1s 1-second bars Very Low $ Slightly higher resolution
Trades All executed trades Moderate $$ Trade analysis, execution quality
TBBO BBO on trade events Moderate $$ Trade-triggered order book
BBO-1s BBO snapshots (1-second) Moderate-High $$$ Regular order book snapshots
MBP-1 Top-of-book updates High $$$ Best bid/offer tick data
MBP-10 10 price levels Very High Deeper order book
MBO Full order book Highest $ HFT, market microstructure

Cost Optimization Strategies

1. Start with OHLCV bars ($)

  • Use 1-minute or 1-second bars for initial backtesting
  • ~90% cheaper than full order book data
  • Sufficient for most retail strategies

2. Upgrade to Trades when needed ($$)

  • Get execution price/size without full book overhead
  • Good for analyzing trade flow and volume
  • ~50-70% cheaper than order book data

3. Use BBO/MBP-1 for spread analysis ($$$)

  • Best bid/offer without full book depth
  • Useful for spread-based strategies
  • ~30-50% cheaper than full order book

4. Reserve MBO for HFT only ()

  • Only needed for ultra-low latency strategies
  • Most expensive option (5-10x cost of OHLCV)
  • Databento's specialty is high-quality MBO data

Historical Data vs Real-Time Streaming

Historical Data (Past 24+ Hours)

Pricing: Pay-as-you-go ($/GB)

  • Billed per byte consumed (uncompressed size)
  • $125 free credits for new users
  • No subscription required
  • Download on-demand via Historical API

When to Use:

  • Backtesting strategies
  • Research and analysis
  • Model training (ML/AI)
  • One-time data pulls
  • Budget-constrained projects ($125 credit stretches far)

Cost Control:

  • Select minimal date range
  • Use cheapest schema (OHLCV vs MBO)
  • Limit symbols to only what you need
  • Compress data after download (zstd supported)

Real-Time Streaming (Last 24 Hours + Live)

Pricing: Subscription plans ($179-199/month+)

  • Flat monthly rate per dataset
  • Billed in advance, cancel anytime
  • 2.9% Stripe fee on exchange license fees
  • ⚠️ Usage-based pricing ending March 31, 2025

When to Use:

  • Live trading (paper or real money)
  • Real-time alerts and monitoring
  • High-frequency strategies
  • Continuous data collection
  • Intraday analysis (last 24 hours)

Cost Considerations:

  • Minimum $179-199/month commitment
  • May be cheaper than usage-based if streaming continuously
  • Unlimited symbols within dataset (huge value for multi-symbol strategies)
  • Exchange fees passed through with no markup

Minimum Purchase Unit

Historical Data:

  • No minimum purchase (pay per byte)
  • Download 1 hour, 1 day, 1 week, or years of data
  • Query by exact date/time range (nanosecond precision)
  • $125 free credits cover significant testing

Live Data:

  • ⚠️ Minimum 1 month subscription (billed monthly)
  • Can cancel anytime (no long-term contract)
  • Prorated refunds: Unknown (check terms)
  • Free trial: Not mentioned in research

Recommendation for Limited Budget Deployment

Phase 1: Free Testing ($0 - Use $125 Credits)

Goal: Validate strategy with real data before spending

  1. Sign up for Databento (automatic $125 credit)
  2. Choose cheapest dataset: CME Bitcoin Futures (BTC) or single equity
  3. Select cheapest schema: OHLCV-1m bars for backtesting
  4. Download historical data: 3-6 months (enough for initial validation)
  5. Estimated credit usage:
    • BTC OHLCV-1m, 90 days: ~$18 ($0.20/day)
    • 5 symbols, 30 days: ~$30 ($0.20/day/symbol)
    • Leaves $95+ credits for experimentation

Deliverable: Backtested strategy with real exchange data, zero cost

Phase 2: Historical Data Expansion ($10-50/month)

Goal: Expand backtesting to more symbols/timeframes

  1. Add more symbols: Scale to 5-10 symbols (~$30-60/month OHLCV)
  2. Increase data depth: Download 1-2 years of history (~$50-100 one-time)
  3. Upgrade schema: Move to Trades if needed (~2-5x cost increase)
  4. Total monthly cost: $10-50 for ongoing historical data

Cost Control Tips:

  • Only download new data (avoid re-downloading same periods)
  • Use batch downloads for efficiency
  • Compress and store data locally (avoid repeat API calls)
  • Monitor usage in Databento portal dashboard

Phase 3: Live Data Subscription ($179-199/month)

Goal: Deploy live trading strategy

  1. Choose subscription plan: Standard ($179-199/month)
  2. Select dataset: CME ($179) or US Equities ($199)
  3. Activate live streaming: Real-time market data
  4. Start paper trading: Test with live data, no real money
  5. Monitor performance: 1-3 months before live capital

Break-Even Analysis:

  • If trading >5 symbols continuously, subscription > usage-based
  • If trading <5 symbols sporadically, usage-based might be cheaper
  • ⚠️ Usage-based ending March 2025 (subscription will be only option)

Phase 4: Multi-Dataset Scaling ($358+ per month)

Goal: Trade multiple asset classes

  1. Add second dataset: OPRA options ($199) or additional venue
  2. Consider Plus tier: Contact sales for custom pricing
  3. Bundle discounts: Negotiate multi-dataset pricing
  4. Total cost: $358+ per month (2 datasets at Standard tier)

Action Plan: Getting Started Today

Step 1: Sign Up (5 minutes)

  1. Go to https://databento.com
  2. Create account (email + password)
  3. Verify email
  4. Receive $125 free credits automatically

Step 2: Explore Data Catalog (15 minutes)

  1. Navigate to Data Catalog: https://databento.com/pricing
  2. Search for datasets:
    • CME (futures): Bitcoin (BTC), E-mini S&P 500 (ES)
    • Nasdaq (equities): AAPL, MSFT, TSLA
    • OPRA (options): SPX, SPY
  3. Check available schemas (OHLCV, Trades, MBP-1, MBO)
  4. Use price estimator to calculate costs for your symbols

Step 3: Download First Dataset (30 minutes)

# Install Databento Python client
pip install databento

# Example: Download BTC futures OHLCV bars
import databento as db

client = db.Historical('YOUR_API_KEY')

data = client.timeseries.get_range(
    dataset='GLBX.MDP3',  # CME Globex
    symbols=['BTC.FUT'],   # Bitcoin futures
    schema='ohlcv-1m',     # 1-minute bars (cheapest)
    start='2024-01-01',
    end='2024-01-07',      # 1 week of data
)

# Save to Parquet (compressed)
data.to_file('btc_ohlcv_1week.parquet')

Estimated cost: ~$1.40 (7 days × $0.20/day)
Credits remaining: $123.60

Step 4: Integrate with Foxhunt (2-4 hours)

  1. Use existing Parquet persistence (data/src/parquet_persistence.rs)
  2. Convert Databento format to Foxhunt MarketDataEvent
  3. Feed historical data to backtesting service (port 50053)
  4. Run backtests with real data (already 95.5% E2E pass rate)

Integration points:

  • Databento → Parquet → ParquetMarketDataReader → BacktestingService
  • No changes to existing Foxhunt infrastructure needed
  • Reuse principle: Databento replaces synthetic data, not architecture

Step 5: Monitor Usage (Ongoing)

  1. Log in to Databento portal
  2. Navigate to "Data Usage" dashboard
  3. Track credit consumption by dataset/schema
  4. Set alerts for credit thresholds (e.g., $25 remaining)
  5. Top up credits before running out (avoid downtime)

Comparison with Alternatives

Databento vs Competitors (Small-Scale Focus)

Provider Free Tier Min Cost Historical $/GB Live Streaming Best For
Databento $125 credit $0-179/month Varies (competitive) $179-199/month Startups, retail HFT
Polygon.io $0 (limited) $29/month N/A (subscription) $29-$199/month Retail stocks/crypto
Alpaca Markets Free (limited) $0 N/A (subscription) Free (basic) Retail crypto/stocks
Alpha Vantage Free (5 req/min) $0 N/A (rate limited) N/A Hobbyists
IEX Cloud $0 (limited) $0 N/A (shutdown 8/2024) N/A Shutting down
Bloomberg No free $2,000+/month Enterprise only Enterprise only Institutions
Refinitiv No free $1,000+/month Enterprise only Enterprise only Institutions

Databento Advantages:

  • Generous $125 free credit (real exchange data)
  • Pay-as-you-go historical data (no subscription required)
  • High-quality tick data (HFT-grade, not just retail APIs)
  • Futures, options, equities all available
  • Nanosecond timestamps (4 timestamps per event)
  • Direct exchange feeds (CME, Nasdaq, OPRA, ICE)
  • No long-term contracts (cancel anytime)

Databento Disadvantages:

  • ⚠️ Live data transitioning to subscription-only (March 2025)
  • ⚠️ No free live streaming (must pay $179+ after credits)
  • ⚠️ Complex pricing (varies by dataset/schema/venue)
  • ⚠️ Not cheapest for basic retail stock APIs (Polygon/Alpaca cheaper)

Frequently Asked Questions

Q: Can I use the $125 credit for live data?

A: No, credits only apply to historical data (past 24+ hours). Live streaming requires subscription plans ($179-199/month).

Q: What happens when my $125 credit runs out?

A: You'll need to add a credit card and purchase more credits. Historical data will continue billing at $/GB rate.

Q: Can I get more free credits?

A: No explicit promotions advertised. Do NOT create multiple accounts (Databento detects and bans).

Q: Is there a free trial for live streaming?

A: Not mentioned in research. Contact sales to request trial period.

Q: What's the cheapest live streaming option?

A: CME Standard plan at $179/month (futures). OPRA/Equities start at $199/month.

Q: Do I need to pay exchange fees on top of Databento subscription?

A: Exchange fees are included in subscription price (pass-through with no markup).

Q: Can I pause my subscription?

A: Cancel anytime (no long-term contract). Reactivate when needed.

Q: What's the data delay on live streaming?

A: Real-time (no delay). Databento provides direct exchange feeds with sub-microsecond latency.

Q: Can I redistribute Databento data?

A: Check license terms in Data Catalog. Personal use typically allowed, commercial redistribution requires approval.

Q: Is there a student discount?

A: Not explicitly advertised, but universities have partnership programs. Contact sales for academic pricing.


Cost Projection: 6-Month Budget

Conservative Budget ($125-500 total)

Month 1-2: Free Testing Phase

  • Use $125 free credits
  • 1-2 symbols, OHLCV-1m bars
  • 3 months historical data download
  • Cost: $0 (credits cover)

Month 3-4: Expanded Historical Testing

  • 5 symbols, OHLCV + Trades
  • 6 months historical data
  • Cost: $50/month × 2 = $100

Month 5-6: Prepare for Live

  • 10 symbols, historical data updates
  • Final backtesting validation
  • Cost: $75/month × 2 = $150

Total 6-Month Cost: $250 (historical data only)

Moderate Budget ($1,000-2,000 total)

Month 1: Free Testing

  • $125 credits, initial backtesting
  • Cost: $0

Month 2-3: Historical Expansion

  • 10 symbols, Trades schema
  • 1 year historical data (one-time $200)
  • Ongoing updates ($100/month)
  • Cost: $200 + $100 × 2 = $400

Month 4-6: Live Paper Trading

  • CME Standard subscription ($179/month)
  • Continue historical data updates ($50/month)
  • Cost: ($179 + $50) × 3 = $687

Total 6-Month Cost: $1,087

Aggressive Budget ($3,000-5,000 total)

Month 1: Free + Historical

  • $125 credits + $500 historical purchase
  • Multi-year data for 20+ symbols
  • Cost: $500

Month 2-6: Live Trading (Multiple Datasets)

  • CME Standard ($179/month)
  • OPRA Standard ($199/month)
  • Databento US Equities Standard ($199/month)
  • Historical updates ($100/month)
  • Cost: ($179 + $199 + $199 + $100) × 5 = $3,385

Total 6-Month Cost: $3,885


Summary & Next Steps

Key Takeaways

  1. Start Free: $125 credit = 600+ days of OHLCV data for 1 symbol (zero cost)
  2. Historical is Cheap: $10-50/month covers most backtesting needs (OHLCV/Trades)
  3. Live is Subscription: $179-199/month minimum (not usage-based after March 2025)
  4. Optimize Schema: OHLCV bars are 10-50x cheaper than full order book (MBO)
  5. No Long-Term Lock-In: Cancel anytime, pay-as-you-go historical data

Immediate (Week 1):

  • Sign up for Databento ($125 free credit)
  • Download BTC futures OHLCV-1m (1-3 months)
  • Integrate with existing ParquetMarketDataReader
  • Run backtests with real CME data

Short-Term (Month 1-3):

  • Expand to 5-10 symbols (crypto futures + equity indices)
  • Use Trades schema for execution analysis
  • Download 6-12 months historical data
  • Validate ML models with real exchange data
  • Budget: $50-150 total (historical data only)

Medium-Term (Month 4-6):

  • ⚠️ Evaluate need for live data (paper trading)
  • ⚠️ Subscribe to CME Standard if live trading ($179/month)
  • ⚠️ Consider OPRA/Equities if expanding asset classes
  • Budget: $179-577/month (live subscriptions)

Long-Term (6+ months):

  • 🎯 Negotiate enterprise pricing if scaling (Plus/Unlimited tiers)
  • 🎯 Explore multi-dataset bundles (potential discounts)
  • 🎯 Apply for startup/academic pricing if eligible
  • Budget: $500-2,000+/month (production deployment)

Final Recommendation

For Foxhunt's current state (95.5% E2E pass rate, Wave 151 complete):

ACTION: Sign up for Databento TODAY and use $125 free credits

Rationale:

  1. Backtesting service is production-ready (12/12 tests passing)
  2. Parquet persistence already implemented (existing infrastructure)
  3. $125 credit provides 3-6 months of backtesting data (zero cost)
  4. Real CME data > synthetic data (validate production readiness)
  5. No commitment required (historical data is pay-as-you-go)

Integration Effort: 2-4 hours (minimal changes to existing code)
Cost: $0 for initial testing phase
Value: Massive (real exchange data validates entire system)


Additional Resources

Support & Sales

  • Contact Sales: sales@databento.com (for discounts, custom pricing)
  • Technical Support: support@databento.com
  • Community: Reddit r/Databento (small but active)
  • Status Page: Check for API uptime/incidents

Alternative Providers (If Databento Doesn't Fit)

  • Polygon.io: Retail-friendly, cheaper for basic stock APIs
  • Alpaca Markets: Free tier with basic live data (crypto/stocks)
  • Yahoo Finance: Free (limited, not suitable for HFT)
  • CryptoDataDownload: Free historical crypto data (not real-time)

Report Compiled: 2025-10-12
Research Sources: Databento official docs, pricing blog posts, Reddit discussions, LinkedIn announcements
Confidence Level: High (based on official 2024-2025 pricing updates)

Next Action: Sign up at https://databento.com and claim $125 free credit