e8a68ee39f
- Created data/examples/download_ml_training_data.rs using reqwest + Databento HTTP API - Downloaded 90 days × 4 symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT) - Files saved to test_data/real/databento/ml_training/ - Total: 360 files, 15 MB compressed DBN format - Used existing Rust pattern from download_nq_fut.rs - API key loaded from .env file - 100% success rate (360/360 files) - Ready for ML training benchmarks Next: Create simplified training benchmark for RTX 3050 Ti GPU measurements
147 lines
4.2 KiB
Markdown
147 lines
4.2 KiB
Markdown
# Databento Gold Futures Data Download
|
|
|
|
## Summary
|
|
|
|
Successfully downloaded 30 days of Gold Futures (GC) OHLCV-1m data from Databento.
|
|
|
|
## Download Details
|
|
|
|
- **Date Range**: 2024-01-02 to 2024-01-31 (30 calendar days)
|
|
- **Symbol**: GC.c.0 (continuous front-month contract)
|
|
- **Dataset**: GLBX.MDP3
|
|
- **Schema**: ohlcv-1m (1-minute OHLCV bars)
|
|
- **Cost**: $0.00 (free data)
|
|
- **Output File**: `GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.dbn`
|
|
- **File Size**: 11 KB (11,138 bytes)
|
|
- **Record Count**: 781 bars
|
|
|
|
## Data Characteristics
|
|
|
|
### Coverage
|
|
- **Actual date range in data**: 2024-01-02 08:19:00 UTC to 2024-01-30 23:35:00 UTC
|
|
- **Trading days covered**: 29 days
|
|
- **Average bars per day**: 39 (varies from 2 to 675)
|
|
|
|
### Trading Hours
|
|
Gold futures trade primarily during CME hours. The data shows activity concentrated around:
|
|
- **Most active**: 14:00-16:00 UTC (~67-71 bars/hour)
|
|
- **Moderate activity**: 11:00-18:00 UTC
|
|
- **Limited activity**: 19:00-08:00 UTC
|
|
|
|
### Price Statistics
|
|
- **Price range**: $2,005.30 - $2,073.70
|
|
- **Average close**: $2,033.90 ± $6.46
|
|
- **Max single-bar return**: 0.79%
|
|
- **Min single-bar return**: -1.42%
|
|
- **Volatility (std dev)**: 0.126%
|
|
|
|
### Volume Statistics
|
|
- **Average volume**: 6 contracts
|
|
- **Median volume**: 2 contracts
|
|
- **Max volume**: 114 contracts
|
|
- **Zero volume bars**: 0 (0%)
|
|
|
|
## Data Quality
|
|
|
|
✅ **Passed all quality checks**:
|
|
- No missing OHLC values (1 missing value total, likely metadata)
|
|
- No duplicate timestamps
|
|
- Consistent OHLC relationships (High ≥ Open/Close, Low ≤ Open/Close)
|
|
- No price spikes exceeding 20% threshold (max change: 1.42%)
|
|
- All volume bars have positive or zero volume
|
|
|
|
## Technical Notes
|
|
|
|
### Symbology Issues
|
|
Attempted to download parent symbol `GC.FUT` and specific contract months (GCG24, GCH24, etc.) but encountered symbology errors:
|
|
```
|
|
422 symbology_invalid_request
|
|
None of the symbols could be resolved
|
|
```
|
|
|
|
Only the continuous contract `GC.c.0` (front-month) resolved successfully.
|
|
|
|
### Data Sparsity
|
|
The dataset has relatively sparse coverage (781 bars over 30 days, average ~26 bars/day). This is likely due to:
|
|
1. Limited trading hours for gold futures
|
|
2. Using free/sample tier data
|
|
3. OHLCV-1m aggregation only including bars with activity
|
|
|
|
January 30th had significantly more data (675 bars) compared to other days (2-19 bars), suggesting variable data availability or increased trading activity on that day.
|
|
|
|
## Usage
|
|
|
|
### Loading in Python (databento)
|
|
```python
|
|
import databento as db
|
|
|
|
# Load from file
|
|
store = db.DBNStore.from_file('GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.dbn')
|
|
|
|
# Convert to DataFrame
|
|
df = store.to_df()
|
|
print(f"Loaded {len(df)} bars")
|
|
|
|
# Access OHLCV data
|
|
print(df[['open', 'high', 'low', 'close', 'volume']].head())
|
|
```
|
|
|
|
### Loading in Rust (databento-dbn crate)
|
|
```rust
|
|
use databento_dbn::{decode::DbnDecoder, DBNStore};
|
|
|
|
// Load DBN file
|
|
let file = std::fs::File::open("GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.dbn")?;
|
|
let mut decoder = DbnDecoder::new(file)?;
|
|
|
|
// Iterate over records
|
|
for record in decoder {
|
|
let record = record?;
|
|
// Process OHLCV bar
|
|
}
|
|
```
|
|
|
|
## Cost Tracking
|
|
|
|
| Item | Cost |
|
|
|------|------|
|
|
| Data download (30 days, OHLCV-1m) | $0.00 |
|
|
| **Total** | **$0.00** |
|
|
|
|
The data was free, likely due to:
|
|
- Using continuous contract symbology
|
|
- Free tier / sample data
|
|
- Limited historical depth (only 30 days)
|
|
|
|
## Recommendations
|
|
|
|
For production use:
|
|
1. Consider subscribing to paid tier for denser data coverage
|
|
2. Explore specific contract months if symbology issues are resolved
|
|
3. Verify trading hours align with strategy requirements
|
|
4. Test with longer date ranges to assess data quality consistency
|
|
|
|
## Files
|
|
|
|
```
|
|
test_data/real/databento/
|
|
├── GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.dbn # 11 KB, 781 bars
|
|
└── README.md # This file
|
|
```
|
|
|
|
## Download Scripts
|
|
|
|
The following Python scripts were used for this download:
|
|
- `download_gc_timeseries.py` - Main download script (successful)
|
|
- `analyze_gc_data.py` - Data quality analysis
|
|
- `check_gc_symbols.py` - Symbology debugging
|
|
- `download_gc_specific_contract.py` - Attempted specific contracts
|
|
|
|
All scripts are located in the project root directory.
|
|
|
|
---
|
|
|
|
**Date**: 2025-10-13
|
|
**Downloaded by**: Databento Python SDK v0.64.0
|
|
**API Key**: db-95LEt...uf6 (masked)
|