foxhunt/databento_sample_data.md

Databento Sample Data Availability

Report Date: 2025-10-12
Status: ✅ FREE SAMPLE DATA FOUND (GitHub test files)

---

Executive Summary

Good News: Databento provides 81 test DBN files in their GitHub repository for free download, covering all major data types. Additionally, new users receive $125 in free credits for historical data.

Best Strategy:

1. Download GitHub test files (immediate, free)
2. Use $125 free credits for real market data testing
3. Validate pipeline with GitHub samples before spending credits

---

Official Sample Data

GitHub Test Files (FREE ✅)

Location: https://github.com/databento/dbn/tree/main/tests/data

Available: Yes - 81 test files
Cost: Free (open source)
Size: Small test samples (~KB to low MB range)
Symbols: Test data (not real symbols, but valid DBN format)
Date Range: Various test scenarios
Data Types: Comprehensive coverage

Available Data Types:

• ✅ BBO (Best Bid/Offer): bbo-1m, bbo-1s variants
• ✅ MBO (Market By Order): Full order book data
• ✅ MBP-1 (Market By Price - Top of Book)
• ✅ MBP-10 (Market By Price - 10 levels)
• ✅ OHLCV: 1d, 1h, 1m, 1s candlestick bars
• ✅ Trades: Trade data
• ✅ TBBO: Trade with BBO
• ✅ Definition: Instrument definitions
• ✅ Imbalance: Order imbalances
• ✅ Statistics: Market statistics
• ✅ Status: Market status events
• ✅ CBBO/CMBP: Consolidated variants

File Formats:

• .dbn - Uncompressed DBN binary
• .dbn.zst - Zstandard compressed (v1, v2, v3)
• .dbn.frag - DBN fragments (no metadata header)
• .dbz - Legacy compressed format

---

Free Credits Program

Available: Yes - $125 free credits
Eligibility: All new Databento accounts
Usage: Historical data only (not live data)
Expiration: No expiration mentioned
Source: https://databento.com/docs/faqs/usage-pricing-and-data-credits

What $125 Gets You:

• Historical data billed per byte consumed
• Pricing varies by dataset ($/GB)
• Example: ~$0.50-$5 per GB depending on venue/schema
• Estimate: 25-250 GB of historical data

---

Format Details

DBN (Databento Binary Encoding)

File Format: Proprietary binary with optional Zstandard compression
Compression: Yes (.dbn.zst files ~70-90% size reduction)
Parser: ✅ ALREADY EXISTS in our codebase

Existing Infrastructure:

// /home/jgrusewski/Work/foxhunt/data/src/dbn_parser.rs
pub struct DbnParser {
    // Parses DBN format into MarketDataEvent
}

impl DbnParser {
    pub async fn parse_file(&self, path: &Path) -> Result<Vec<MarketDataEvent>>;
    pub async fn parse_stream(&self, reader: impl AsyncRead) -> Result<Vec<MarketDataEvent>>;
}

Status: ✅ Parser ready, just needs testing with sample files

---

Testing Strategy

Phase 1: GitHub Sample Validation (FREE)

Timeline: 1-2 hours
Cost: $0

# 1. Clone DBN repository
git clone https://github.com/databento/dbn.git
cd dbn/tests/data

# 2. Download sample files (examples)
wget https://raw.githubusercontent.com/databento/dbn/main/tests/data/mbo.dbn.zst
wget https://raw.githubusercontent.com/databento/dbn/main/tests/data/ohlcv-1d.dbn.zst
wget https://raw.githubusercontent.com/databento/dbn/main/tests/data/trades.dbn.zst

# 3. Copy to foxhunt test directory
cp *.dbn* /home/jgrusewski/Work/foxhunt/test_data/databento/

# 4. Test with existing DbnParser
cargo test -p data test_dbn_parser

Expected Outcome:

• ✅ Validate DbnParser works with real DBN files
• ✅ Identify any format compatibility issues
• ✅ Benchmark parsing performance
• ✅ Confirm event conversion to MarketDataEvent

Phase 2: Free Credits Testing (PAID - using free credits)

Timeline: 2-4 hours
Cost: $0 (uses free $125 credits)

# 1. Sign up for Databento account
# https://databento.com (automatic $125 credits)

# 2. Generate API key
# Portal: https://databento.com/portal

# 3. Download 1-day sample
export DATABENTO_API_KEY="db-YOUR_KEY"

# Python example (can convert to Rust later)
pip install databento databento-dbn

python << 'EOF'
import databento as db
client = db.Historical()

# Download BTC futures (small dataset)
client.timeseries.get_range(
    dataset='GLBX.MDP3',  # CME Globex
    symbols=['BTC.FUT'],
    schema='trades',
    start='2024-01-02',
    end='2024-01-02',
    path='test_data/databento/btc_trades_20240102.dbn.zst'
)

# Cost estimate: ~$0.10-$1.00 for 1 day
EOF

Expected Outcome:

• ✅ Real market data in DBN format
• ✅ Validate backtesting pipeline end-to-end
• ✅ Test data quality and completeness
• ✅ Measure parsing performance on larger files

Phase 3: Alternative Free Data (BACKUP)

If GitHub samples insufficient OR want real data without spending credits:

Option A: Use existing Kaggle data (already have)

• Location: /home/jgrusewski/Work/foxhunt/test_data/
• Format: CSV (convert to Parquet)
• Cost: $0

Option B: Public crypto exchanges (websocket capture)

• Binance, Coinbase APIs
• Capture 1-day sample yourself
• Cost: $0 (time investment)

Option C: Request demo from Databento

• Contact: support@databento.com
• Enterprise demo datasets
• Cost: $0 (requires approval)

---

Recommended Testing Workflow

Week 1: GitHub Samples + Parser Validation

Day 1-2: GitHub Sample Testing

# Clone and test with all 81 sample files
for file in tests/data/*.dbn*; do
    cargo test -- --nocapture test_parse_file "$file"
done

Deliverables:

• ✅ DbnParser compatibility report
• ✅ Performance benchmarks (parse time per file)
• ✅ Event conversion validation
• ✅ Identified any format issues

Week 2: Real Data Testing ($0.50-$5 from free credits)

Day 3-4: Small Real Dataset

# Download 1-3 days of real data
# Target symbols: BTC, ETH futures (high volume)
# Target cost: <$5 from free credits

Deliverables:

• ✅ Backtesting service validation
• ✅ ML training pipeline test
• ✅ Performance metrics (realistic data)
• ✅ Data quality assessment

Week 3: Full Pipeline Integration

Day 5-7: End-to-End Testing

# Run full backtesting workflow
cargo test -p backtesting_service --test e2e_dbn_replay

Deliverables:

• ✅ Complete data pipeline validated
• ✅ Backtesting metrics verified
• ✅ Ready for production data purchase

---

Alternative Testing (If No Free Samples Work)

Minimum Purchase Strategy

If must purchase immediately:

Option 1: 1-Day Minimum ($0.50-$2)

• Dataset: CME Globex (GLBX.MDP3)
• Symbol: ES (S&P 500 futures) or BTC
• Schema: Trades or MBP-1
• Date: Recent weekday

Option 2: Free Tier Strategy

• Some venues offer free data with delay
• Check: DBEQ.BASIC (Databento Equities Basic)
• May have free tier or heavily discounted pricing

Option 3: Partner with Databento

• Startup program: https://databento.com/startups
• Potential: Additional credits or discounts
• Requires: Application and approval

---

Next Steps

Immediate Actions (Next 30 minutes)

1. Clone DBN Repository:

   cd /tmp
   git clone https://github.com/databento/dbn.git
   ls -lh dbn/tests/data/*.dbn* | head -20
   

2. Copy Sample Files to Foxhunt:

   mkdir -p /home/jgrusewski/Work/foxhunt/test_data/databento/samples
   cp dbn/tests/data/*.dbn* /home/jgrusewski/Work/foxhunt/test_data/databento/samples/
   

3. Test DbnParser:

   cd /home/jgrusewski/Work/foxhunt
   cargo test -p data test_dbn_parser -- --nocapture
   

Short-term Actions (This Week)

4. Sign Up for Databento Account (if not already):

   • URL: https://databento.com
   • Claim $125 free credits
   • Generate API key

5. Download 1-Day Real Sample (costs ~$0.50-$1):

   # Use Python client (easiest)
   pip install databento databento-dbn
   # Download small real dataset
   

6. Validate Backtesting Pipeline:

   cargo test -p backtesting_service --test e2e_databento
   

Medium-term Actions (Next 2 Weeks)

7. Performance Benchmarking:

   • Measure parse time vs Parquet
   • Compare data quality
   • Validate event accuracy

8. Integration Testing:

   • ML training pipeline
   • Backtesting service
   • Feature engineering

9. Cost Analysis:

   • Estimate monthly data costs
   • Compare Databento vs alternatives
   • ROI calculation

---

Cost-Benefit Analysis

Free Testing Path

Activity	Cost	Time	Risk
GitHub samples (81 files)	$0	1h	None
DbnParser validation	$0	2h	None
Sign up ($125 credits)	$0	5min	None
1-day real test	$0*	30min	None
Full pipeline test	$0*	4h	None
Total	$0*	~8h	None

* Uses free $125 credits (does not require payment)

Minimum Purchase Path (if needed)

Activity	Cost	Time	Risk
1-day purchase	$0.50-$2	30min	Low
1-week purchase	$3-$15	1h	Medium
1-month purchase	$50-$200	2h	High

Recommendation: ✅ START WITH FREE PATH (GitHub + $125 credits)

---

Success Criteria

Phase 1 Success (GitHub Samples)

• ✅ DbnParser successfully parses all 81 test files
• ✅ Zero parsing errors or format incompatibilities
• ✅ Event conversion produces valid MarketDataEvent structs
• ✅ Performance: <100ms per file (test data is small)

Phase 2 Success (Real Data)

• ✅ Successfully download 1-3 days real market data
• ✅ Data quality: No gaps, accurate timestamps, valid prices
• ✅ Backtesting service processes data correctly
• ✅ Performance: Parse 1GB in <5 seconds

Phase 3 Success (Full Pipeline)

• ✅ ML training pipeline generates features correctly
• ✅ Backtesting produces accurate metrics (Sharpe, PnL, drawdown)
• ✅ All E2E tests passing (100%)
• ✅ Ready for production data subscription

---

Parser Compatibility

Existing DbnParser Capabilities

Location: /home/jgrusewski/Work/foxhunt/data/src/dbn_parser.rs

Supported:

• ✅ DBN binary format parsing
• ✅ Zstandard decompression
• ✅ Event conversion to MarketDataEvent
• ✅ Async/streaming support

May Need Updates (TBD after testing):

• ⚠️ DBN format version compatibility (v1, v2, v3)
• ⚠️ Fragment handling (.dbn.frag files)
• ⚠️ Legacy DBZ format support
• ⚠️ Error handling for corrupt files

Testing Will Reveal:

• Actual compatibility issues
• Performance bottlenecks
• Missing features

---

Additional Resources

Official Documentation

• DBN Spec: https://databento.com/docs/standards-and-conventions/databento-binary-encoding
• Quickstart: https://databento.com/docs/quickstart
• Python Client: https://github.com/databento/databento-python
• Rust Client: https://github.com/databento/databento-rs
• DBN Repo: https://github.com/databento/dbn

Tools

• dbn-cli: Command-line tool for DBN files

  cargo install dbn-cli
  dbn some.dbn.zst --json  # Convert to JSON
  dbn some.dbn.zst --csv   # Convert to CSV
  

Community

• Support: support@databento.com
• Slack: Community slack (link on website)
• GitHub: File issues on respective repositories

---

Risk Assessment

Low Risk ✅

• GitHub samples are free and unlimited
• $125 credits have no expiration
• No credit card required until exceeding free credits
• Can test extensively before any payment

Medium Risk ⚠️

• GitHub samples may not be representative of real data
• Free credits expire (no confirmed timeline)
• May need to purchase if requirements exceed $125

High Risk ❌

• None identified for initial testing phase

---

Conclusion

✅ READY TO PROCEED WITH FREE TESTING

Immediate Action: Download GitHub test files and validate DbnParser

Timeline: Can start testing immediately (no payment required)

Cost: $0 for comprehensive testing (81 samples + $125 credits)

Recommendation:

1. ✅ Clone DBN repo now (5 minutes)
2. ✅ Test all 81 samples (1-2 hours)
3. ✅ Sign up for Databento if samples work (5 minutes)
4. ✅ Download 1-day real data using credits ($0)
5. ✅ Validate full pipeline (2-4 hours)

Total Time Investment: ~8 hours
Total Cost: $0 (uses existing free resources)

---

Report Generated: 2025-10-12
Status: ✅ FREE SAMPLE DATA CONFIRMED
Next Step: Clone GitHub repository and begin testing