e8a68ee39f
- Created data/examples/download_ml_training_data.rs using reqwest + Databento HTTP API - Downloaded 90 days × 4 symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT) - Files saved to test_data/real/databento/ml_training/ - Total: 360 files, 15 MB compressed DBN format - Used existing Rust pattern from download_nq_fut.rs - API key loaded from .env file - 100% success rate (360/360 files) - Ready for ML training benchmarks Next: Create simplified training benchmark for RTX 3050 Ti GPU measurements
98 lines
4.0 KiB
Markdown
98 lines
4.0 KiB
Markdown
# Databento API Cost Tracking
|
|
|
|
## Account Balance
|
|
- **Initial Credits**: $125.00 (as of 2025-10-12)
|
|
- **Current Credits**: ~$124.9960 (after 3 downloads)
|
|
|
|
## Usage History
|
|
|
|
### 2025-10-13: CL.FUT Historical Data Download
|
|
- **Symbol**: CL.FUT (Crude Oil Futures)
|
|
- **Dataset**: GLBX.MDP3 (CME Group MDP 3.0)
|
|
- **Schema**: ohlcv-1m (1-minute OHLCV bars)
|
|
- **Date Range**: 2024-01-02 (single trading day)
|
|
- **File Size**: 1,527,031 bytes (1.46 MB / 0.00142 GB)
|
|
- **Estimated Cost**: $0.000712 - $0.002848
|
|
- **File Location**: `/home/jgrusewski/Work/foxhunt/test_data/real/databento/CL.FUT_ohlcv-1m_2024-01-02.dbn`
|
|
- **Status**: ✅ SUCCESS (⚠️ validation pending)
|
|
- **Notes**: 15.46x larger than ES.FUT due to numerous calendar spreads and related instruments (CLN4-CLH6, CLU4-CLM5, CLX5-CLZ6, mini contracts MCLX5/MCLV7, cross-commodity spreads)
|
|
- **Bar Count**: ~24,000+ bars (estimated, includes many spread symbols)
|
|
- **Validation**: DBN format verified (file header: "DBN\x01"), validation script created but blocked by cargo compilation
|
|
|
|
### 2025-10-13: NQ.FUT Historical Data Download
|
|
- **Symbol**: NQ.FUT (Nasdaq-100 E-mini Futures)
|
|
- **Dataset**: GLBX.MDP3 (CME Group MDP 3.0)
|
|
- **Schema**: ohlcv-1m (1-minute OHLCV bars)
|
|
- **Date Range**: 2024-01-02 (single trading day)
|
|
- **File Size**: 94,510 bytes (92.29 KB / 0.0000879899 GB)
|
|
- **Estimated Cost**: $0.000044 - $0.000176
|
|
- **File Location**: `/home/jgrusewski/Work/foxhunt/test_data/real/databento/NQ.FUT_ohlcv-1m_2024-01-02.dbn`
|
|
- **Status**: ✅ SUCCESS
|
|
- **Notes**: Downloaded via curl with Basic Authentication
|
|
|
|
### 2025-10-12: ES.FUT Historical Data Download
|
|
- **Symbol**: ES.FUT (E-mini S&P 500 Futures)
|
|
- **Dataset**: GLBX.MDP3 (CME Group MDP 3.0)
|
|
- **Schema**: ohlcv-1m (1-minute OHLCV bars)
|
|
- **Date Range**: 2024-01-02 (single trading day)
|
|
- **File Size**: 96,470 bytes (94.21 KB / 0.0000898447 GB)
|
|
- **Estimated Cost**: $0.000045 - $0.000180
|
|
- **File Location**: `/home/jgrusewski/Work/foxhunt/test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn`
|
|
- **Status**: ✅ SUCCESS
|
|
|
|
## Cost Analysis
|
|
|
|
### Pricing Tiers (per GB)
|
|
- **Tier 1**: $0.50/GB (first usage)
|
|
- **Tier 2**: $2.00/GB (maximum)
|
|
|
|
### First Download Metrics
|
|
- **Size**: 94.21 KB (0.0000898 GB)
|
|
- **Cost (low estimate)**: $0.000045
|
|
- **Cost (high estimate)**: $0.000180
|
|
- **Credits Remaining**: ~$125.00 (negligible usage)
|
|
|
|
### Projected Costs for Full Testing
|
|
Assuming 1 day of data = 94 KB:
|
|
- **1 week (7 days)**: ~659 KB = $0.00032 - $0.00126
|
|
- **1 month (30 days)**: ~2.8 MB = $0.00137 - $0.00547
|
|
- **1 year (252 trading days)**: ~23.7 MB = $0.011 - $0.046
|
|
- **5 years (1,260 days)**: ~118.6 MB = $0.055 - $0.220
|
|
|
|
### Multi-Symbol Projections
|
|
If downloading ES.FUT, NQ.FUT, RTY.FUT (3 symbols):
|
|
- **1 year**: ~71 MB = $0.035 - $0.138
|
|
- **5 years**: ~356 MB = $0.165 - $0.660
|
|
|
|
**Note on CL.FUT**:
|
|
CL.FUT is significantly larger (15.46x) due to numerous related instruments:
|
|
- **1 day**: 1.46 MB (vs ES.FUT: 94 KB)
|
|
- **1 year**: ~368 MB = $0.18 - $0.72
|
|
- **Recommendation**: Use specific contract months instead of parent symbol for testing
|
|
|
|
## Conclusion
|
|
✅ **API Key Working**: Basic Authentication required (not Bearer token)
|
|
✅ **Costs Extremely Low**: First download cost <$0.0002
|
|
✅ **Budget Safe**: Can download years of data within $125 budget
|
|
✅ **Production Ready**: DBN format downloaded successfully
|
|
|
|
## Authentication Notes
|
|
**CRITICAL**: Databento API requires **Basic Authentication**, not Bearer tokens:
|
|
```rust
|
|
// ❌ WRONG (401 Unauthorized)
|
|
.header("Authorization", format!("Bearer {}", api_key))
|
|
|
|
// ✅ CORRECT (200 OK)
|
|
.basic_auth(&api_key, Some(""))
|
|
```
|
|
|
|
## Next Steps
|
|
1. ✅ Download test successful (ES.FUT)
|
|
2. ✅ Multi-symbol download (NQ.FUT - same date as ES.FUT)
|
|
3. ✅ Crude oil futures download (CL.FUT - same date for cross-symbol testing)
|
|
4. ⏳ Validate CL.FUT data (pending cargo compilation fix)
|
|
5. ⏳ Parse DBN format to Parquet
|
|
6. ⏳ Integrate with backtesting pipeline
|
|
7. ⏳ Expand to additional symbols (RTY.FUT, etc.)
|
|
8. ⏳ Historical range testing (1-5 years)
|