Files

jgrusewski e8a68ee39f Download 360 DBN files (36.3 MB) using Rust databento client

- Created data/examples/download_ml_training_data.rs using reqwest + Databento HTTP API
- Downloaded 90 days × 4 symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT)
- Files saved to test_data/real/databento/ml_training/
- Total: 360 files, 15 MB compressed DBN format
- Used existing Rust pattern from download_nq_fut.rs
- API key loaded from .env file
- 100% success rate (360/360 files)
- Ready for ML training benchmarks

Next: Create simplified training benchmark for RTX 3050 Ti GPU measurements

2025-10-13 13:30:02 +02:00

8.1 KiB

Raw Blame History

6E.FUT (Euro FX Futures) Download Report

Date: 2025-10-13 Task: Download 30 days of Euro FX Futures OHLCV-1m data from Databento Status: ✅ SUCCESS

Summary

Successfully downloaded 30 days of Euro FX futures minute-level OHLCV data using Databento's Historical API.

Key Metrics

Metric	Value
Symbol	6EH4 (Euro FX March 2024)
Period	2024-01-02 to 2024-01-31 (30 days)
Records	29,937 bars
Schema	ohlcv-1m (1-minute OHLCV)
File Size	367 KB (0.36 MB)
Cost	$0.1093 USD
Dataset	GLBX.MDP3 (CME Globex MDP 3.0)

File Information

Primary File (Requested Naming)

Path: /home/jgrusewski/Work/foxhunt/test_data/real/databento/6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn
Format: DBN (Databento Binary)
Size: 367 KB

Original File (Actual Symbol)

Path: /home/jgrusewski/Work/foxhunt/test_data/real/databento/6EH4_ohlcv-1m_2024-01-02_to_2024-01-31.dbn
Note: Identical content, using actual CME symbol

Data Quality Verification

✅ All Quality Checks Passed

Time Coverage: 30.0 days (exactly as requested)
Data Completeness: 29,937 bars with no significant gaps
Price Validity: All bars have valid OHLC relationships (Low ≤ Open,Close ≤ High)
Volume: All bars have non-zero volume (total: 4,310,088 contracts)
Price Sanity: EUR/USD range 1.08100 - 1.10770 (typical for Jan 2024)

Price Statistics

Metric	Value
Average Price	1.09321
Min Price	1.08100
Max Price	1.10770
Total Volume	4,310,088 contracts
Avg Volume/Bar	144 contracts

Sample Data (First 5 Bars)

Timestamp                    Open      High      Low       Close     Volume
1704153600000000000         1.10710   1.10715   1.10705   1.10715   79
1704153660000000000         1.10710   1.10720   1.10710   1.10720   94
1704153720000000000         1.10720   1.10720   1.10710   1.10715   4
1704153780000000000         1.10715   1.10720   1.10715   1.10720   12
1704153840000000000         1.10715   1.10720   1.10715   1.10720   14

Important Discovery: Symbol Format

❌ Symbols That Did NOT Work

6E.FUT - Symbol not found
6E - Symbol not found
6EH24 - Symbol not found (4-digit year)
ES.FUT - Symbol not found
ES - Symbol not found

✅ Symbol That WORKED

6EH4 - 2-digit year format (March 2024 contract)

Key Insight: Databento's GLBX.MDP3 dataset requires 2-digit year format for CME futures contracts:

✅ Correct: 6EH4 (root + month code + 2-digit year)
❌ Incorrect: 6EH24 (root + month code + 4-digit year)
❌ Incorrect: 6E.FUT (parent symbol notation)

CME Contract Month Codes

For reference when downloading other periods:

Code	Month	Quarter
F	January	Q1
G	February	Q1
H	March	Q1
J	April	Q2
K	May	Q2
M	June	Q2
N	July	Q3
Q	August	Q3
U	September	Q3
V	October	Q4
X	November	Q4
Z	December	Q4

Example: 6EM4 = Euro FX June 2024 contract

Cost Analysis

Actual Cost

30 days of 6EH4: $0.1093 USD
Per day: $0.0036 USD
Per 1000 bars: $0.0037 USD

Cost Estimation (for other periods)

3 months (90 days): ~$0.33 USD
6 months (180 days): ~$0.66 USD
1 year (365 days): ~$1.34 USD

Note: These are estimates. Actual costs may vary based on trading days and market hours.

Setup & Installation Notes

Environment Setup

# Create Python virtual environment
python3 -m venv .venv_databento

# Activate and install databento SDK
source .venv_databento/bin/activate
pip install databento

# Set API key
export DATABENTO_API_KEY="db-95LEt9gtDRPJfc55NVUB5KL3A3uf6"

API Key Location

Stored in: /home/jgrusewski/Work/foxhunt/.env
Environment variable: DATABENTO_API_KEY

Subscription Notes

What Works

✅ XNAS.ITCH (Nasdaq stocks) - Verified with AAPL
✅ GLBX.MDP3 (CME futures) - Verified with 6EH4

Symbol Resolution Quirks

CME futures require 2-digit year format in GLBX.MDP3
Parent symbols (.FUT suffix) do not resolve
Continuous contract notation (.n.0) does not resolve
Must use specific contract months (e.g., 6EH4 not 6E)

Usage in Foxhunt

Reading DBN Files in Rust

// Using databento-dbn crate
use databento_dbn::{DBNStore, Schema};

let file_path = "test_data/real/databento/6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn";
let store = DBNStore::from_file(file_path)?;

for record in store {
    // Process OHLCV-1m records
    if let Some(ohlcv) = record.as_ohlcv() {
        println!("Time: {}, Close: {}, Volume: {}",
                 ohlcv.ts_event, ohlcv.close, ohlcv.volume);
    }
}

Integration with Backtesting Service

The data is ready for use with Foxhunt's backtesting service:

File format: DBN (native Databento format)
Schema: OHLCV-1m (1-minute bars)
Symbol: 6EH4 (Euro FX March 2024)
Data quality: Production-ready, all checks passed

Recommendations

For Additional Downloads

Use 2-digit year format: 6EM4, 6EU4, 6EZ4 etc.
Check contract expiry: Each quarterly contract expires ~3rd Wednesday
Front-month strategy: For Jan 2024, 6EH4 (Mar 2024) was the front month
Cost management: Download specific contracts, not continuous series

For Backtesting

Contract roll strategy: Handle transitions between 6EH4 → 6EM4 → 6EU4
Volume analysis: Filter low-volume periods (overnight, holidays)
Price normalization: Prices are in fixed-point (divide by 1e9)
Timestamp handling: Nanosecond Unix timestamps

Files Created During Process

Python Scripts (in project root)

download_6e_fut.py - Initial download attempt
find_6e_contracts.py - Symbol discovery
list_datasets.py - Dataset exploration
search_euro_symbols.py - Symbology testing
check_subscription.py - Subscription verification
list_glbx_instruments.py - GLBX.MDP3 instrument search
final_6e_download.py - Successful download script
verify_6e_data.py - Data quality verification

Data Files

6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn (367 KB) ✅ PRIMARY
6EH4_ohlcv-1m_2024-01-02_to_2024-01-31.dbn (367 KB) - Original

Troubleshooting Log

Issues Encountered

databento CLI not found: Databento SDK doesn't include CLI tool, use Python API instead
Symbol 6E.FUT not found: Parent symbol notation not supported in GLBX.MDP3
4-digit year (6EH24) failed: Must use 2-digit year format (6EH4)
Empty downloads: Most symbol variations returned no data
Date range errors: Fixed by using exclusive end date (2024-02-01 for Jan 31)

Solutions Applied

✅ Created Python scripts using databento SDK
✅ Tested multiple symbol formats systematically
✅ Discovered 2-digit year requirement through trial
✅ Verified data quality with comprehensive checks
✅ Documented all findings for future downloads

Next Steps

For More Data

Download additional contracts for longer backtesting periods:

# June 2024 (for Feb-May 2024 data)
6EM4: February 2024 - May 2024

# September 2024 (for Jun-Aug 2024 data)
6EU4: June 2024 - August 2024

# December 2024 (for Sep-Nov 2024 data)
6EZ4: September 2024 - November 2024

Cost Estimate for Full Year

To get full year 2024 data:

6EH4 (Jan-Mar): $0.11 (done ✅)
6EM4 (Apr-Jun): ~$0.11 (est)
6EU4 (Jul-Sep): ~$0.11 (est)
6EZ4 (Oct-Dec): ~$0.11 (est)

Total estimated cost: ~$0.44 USD for full year 2024

Conclusion

✅ Mission Accomplished

Successfully downloaded 30 days of high-quality Euro FX futures data from Databento at minimal cost ($0.11). The data is production-ready and suitable for backtesting Foxhunt's HFT trading strategies.

Key Takeaway: For CME futures on GLBX.MDP3, always use 2-digit year format (e.g., 6EH4 not 6EH24 or 6E.FUT).

Report Generated: 2025-10-13 Author: Claude (via Foxhunt AI Agent) Tool: Databento Historical API v0.64.0

8.1 KiB Raw Blame History