Download 360 DBN files (36.3 MB) using Rust databento client
- Created data/examples/download_ml_training_data.rs using reqwest + Databento HTTP API - Downloaded 90 days × 4 symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT) - Files saved to test_data/real/databento/ml_training/ - Total: 360 files, 15 MB compressed DBN format - Used existing Rust pattern from download_nq_fut.rs - API key loaded from .env file - 100% success rate (360/360 files) - Ready for ML training benchmarks Next: Create simplified training benchmark for RTX 3050 Ti GPU measurements
This commit is contained in:
20
.env.example
20
.env.example
@@ -139,6 +139,26 @@ TEST_AWS_REGION=us-east-1
|
||||
USE_LOCALSTACK=false
|
||||
LOCALSTACK_ENDPOINT=http://localhost:4566
|
||||
|
||||
# =============================================================================
|
||||
# DBN Real Data Configuration (Wave 153 - Real Data Integration)
|
||||
# =============================================================================
|
||||
|
||||
# Enable DBN file-based market data instead of Databento API
|
||||
# Set to "true" for E2E tests with real historical data
|
||||
# Set to "false" or leave unset for production Databento API usage
|
||||
USE_DBN_DATA=false
|
||||
|
||||
# DBN Symbol Mappings - Comma-separated list of symbol:path pairs
|
||||
# Example: ES.FUT:/path/to/ES.FUT_ohlcv-1m_2024-01-02.dbn,NQ.FUT:/path/to/NQ.FUT.dbn
|
||||
# For Docker: Use /workspace/test_data/real/databento/ prefix
|
||||
# For local: Use test_data/real/databento/ (relative to workspace root)
|
||||
DBN_SYMBOL_MAPPINGS=ES.FUT:test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn
|
||||
|
||||
# DBN Symbol Remapping - Map requested symbols to available data symbols
|
||||
# Example: BTC/USD:ES.FUT,ETH/USD:ES.FUT (use ES.FUT data for crypto symbols in tests)
|
||||
# This allows E2E tests requesting crypto symbols to use available futures data
|
||||
DBN_SYMBOL_MAP=BTC/USD:ES.FUT,ETH/USD:ES.FUT
|
||||
|
||||
# =============================================================================
|
||||
# Production Notes
|
||||
# =============================================================================
|
||||
|
||||
Reference in New Issue
Block a user