e8a68ee39f
- Created data/examples/download_ml_training_data.rs using reqwest + Databento HTTP API - Downloaded 90 days × 4 symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT) - Files saved to test_data/real/databento/ml_training/ - Total: 360 files, 15 MB compressed DBN format - Used existing Rust pattern from download_nq_fut.rs - API key loaded from .env file - 100% success rate (360/360 files) - Ready for ML training benchmarks Next: Create simplified training benchmark for RTX 3050 Ti GPU measurements
4.0 KiB
4.0 KiB
Databento API Cost Tracking
Account Balance
- Initial Credits: $125.00 (as of 2025-10-12)
- Current Credits: ~$124.9960 (after 3 downloads)
Usage History
2025-10-13: CL.FUT Historical Data Download
- Symbol: CL.FUT (Crude Oil Futures)
- Dataset: GLBX.MDP3 (CME Group MDP 3.0)
- Schema: ohlcv-1m (1-minute OHLCV bars)
- Date Range: 2024-01-02 (single trading day)
- File Size: 1,527,031 bytes (1.46 MB / 0.00142 GB)
- Estimated Cost: $0.000712 - $0.002848
- File Location:
/home/jgrusewski/Work/foxhunt/test_data/real/databento/CL.FUT_ohlcv-1m_2024-01-02.dbn - Status: ✅ SUCCESS (⚠️ validation pending)
- Notes: 15.46x larger than ES.FUT due to numerous calendar spreads and related instruments (CLN4-CLH6, CLU4-CLM5, CLX5-CLZ6, mini contracts MCLX5/MCLV7, cross-commodity spreads)
- Bar Count: ~24,000+ bars (estimated, includes many spread symbols)
- Validation: DBN format verified (file header: "DBN\x01"), validation script created but blocked by cargo compilation
2025-10-13: NQ.FUT Historical Data Download
- Symbol: NQ.FUT (Nasdaq-100 E-mini Futures)
- Dataset: GLBX.MDP3 (CME Group MDP 3.0)
- Schema: ohlcv-1m (1-minute OHLCV bars)
- Date Range: 2024-01-02 (single trading day)
- File Size: 94,510 bytes (92.29 KB / 0.0000879899 GB)
- Estimated Cost: $0.000044 - $0.000176
- File Location:
/home/jgrusewski/Work/foxhunt/test_data/real/databento/NQ.FUT_ohlcv-1m_2024-01-02.dbn - Status: ✅ SUCCESS
- Notes: Downloaded via curl with Basic Authentication
2025-10-12: ES.FUT Historical Data Download
- Symbol: ES.FUT (E-mini S&P 500 Futures)
- Dataset: GLBX.MDP3 (CME Group MDP 3.0)
- Schema: ohlcv-1m (1-minute OHLCV bars)
- Date Range: 2024-01-02 (single trading day)
- File Size: 96,470 bytes (94.21 KB / 0.0000898447 GB)
- Estimated Cost: $0.000045 - $0.000180
- File Location:
/home/jgrusewski/Work/foxhunt/test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn - Status: ✅ SUCCESS
Cost Analysis
Pricing Tiers (per GB)
- Tier 1: $0.50/GB (first usage)
- Tier 2: $2.00/GB (maximum)
First Download Metrics
- Size: 94.21 KB (0.0000898 GB)
- Cost (low estimate): $0.000045
- Cost (high estimate): $0.000180
- Credits Remaining: ~$125.00 (negligible usage)
Projected Costs for Full Testing
Assuming 1 day of data = 94 KB:
- 1 week (7 days): ~659 KB = $0.00032 - $0.00126
- 1 month (30 days): ~2.8 MB = $0.00137 - $0.00547
- 1 year (252 trading days): ~23.7 MB = $0.011 - $0.046
- 5 years (1,260 days): ~118.6 MB = $0.055 - $0.220
Multi-Symbol Projections
If downloading ES.FUT, NQ.FUT, RTY.FUT (3 symbols):
- 1 year: ~71 MB = $0.035 - $0.138
- 5 years: ~356 MB = $0.165 - $0.660
Note on CL.FUT: CL.FUT is significantly larger (15.46x) due to numerous related instruments:
- 1 day: 1.46 MB (vs ES.FUT: 94 KB)
- 1 year: ~368 MB = $0.18 - $0.72
- Recommendation: Use specific contract months instead of parent symbol for testing
Conclusion
✅ API Key Working: Basic Authentication required (not Bearer token) ✅ Costs Extremely Low: First download cost <$0.0002 ✅ Budget Safe: Can download years of data within $125 budget ✅ Production Ready: DBN format downloaded successfully
Authentication Notes
CRITICAL: Databento API requires Basic Authentication, not Bearer tokens:
// ❌ WRONG (401 Unauthorized)
.header("Authorization", format!("Bearer {}", api_key))
// ✅ CORRECT (200 OK)
.basic_auth(&api_key, Some(""))
Next Steps
- ✅ Download test successful (ES.FUT)
- ✅ Multi-symbol download (NQ.FUT - same date as ES.FUT)
- ✅ Crude oil futures download (CL.FUT - same date for cross-symbol testing)
- ⏳ Validate CL.FUT data (pending cargo compilation fix)
- ⏳ Parse DBN format to Parquet
- ⏳ Integrate with backtesting pipeline
- ⏳ Expand to additional symbols (RTY.FUT, etc.)
- ⏳ Historical range testing (1-5 years)