e8a68ee39f
- Created data/examples/download_ml_training_data.rs using reqwest + Databento HTTP API - Downloaded 90 days × 4 symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT) - Files saved to test_data/real/databento/ml_training/ - Total: 360 files, 15 MB compressed DBN format - Used existing Rust pattern from download_nq_fut.rs - API key loaded from .env file - 100% success rate (360/360 files) - Ready for ML training benchmarks Next: Create simplified training benchmark for RTX 3050 Ti GPU measurements
2.9 KiB
2.9 KiB
Kaggle Data Source - Quick Reference Card
Wave 153 Bake-Off
Date: 2025-10-12 | Score: 9.5/10 ⭐⭐⭐⭐⭐ | Status: 🏆 BEST FREE OPTION
📊 At a Glance
| Metric | BTC/USD | ETH/USD |
|---|---|---|
| Size | 262 MB | 22 MB |
| Rows | ~3.8M | ~320K |
| Last Update | 4 days ago | 1 month ago |
| Update Freq | Daily | Monthly |
| Exchanges | 7 | 7 |
| Usability | 10.0/10 | 10.0/10 |
| Quality | 9.5/10 | 9.5/10 |
🎯 Quick Decision Matrix
| Your Need | Use Kaggle? |
|---|---|
| Backtesting | ✅ YES - Perfect |
| ML Training | ✅ YES - Excellent |
| Live Trading | ❌ NO - 4-day lag |
| Academic Research | ✅ YES - Free + cited |
| Quick Prototype | ✅ YES - Easy CSV |
| HFT | ❌ NO - 1-min only |
📥 Download (30 seconds)
# Install
pip install kaggle
# Download
kaggle datasets download -d imranbukhari/comprehensive-btcusd-1m-data
kaggle datasets download -d imranbukhari/comprehensive-ethusd-1m-data
# Extract
unzip comprehensive-btcusd-1m-data.zip
unzip comprehensive-ethusd-1m-data.zip
Recommended File: *_Combined_Index.csv
✅ Top 5 Strengths
- FREE - $0 cost
- Quality - 9.5/10 score
- Multi-Exchange - 7 sources
- Continuous - No gaps
- Maintained - Daily updates (BTC)
⚠️ Top 5 Limitations
- Preprocessed - Not raw
- Lag - 4 days (BTC)
- Not Real-Time - Unsuitable for live
- Large - 262MB download
- Kaggle Account - Required (free)
🔗 URLs
BTC: https://kaggle.com/datasets/imranbukhari/comprehensive-btcusd-1m-data
ETH: https://kaggle.com/datasets/imranbukhari/comprehensive-ethusd-1m-data
📋 5-Minute Validation
import pandas as pd
df = pd.read_csv('BTCUSD_1m_Combined_Index.csv', parse_dates=['timestamp'])
# 1. Basic stats
print(f"Rows: {len(df):,}")
print(f"Range: {df['timestamp'].min()} to {df['timestamp'].max()}")
# 2. OHLCV check
violations = ~((df['high'] >= df['low']) & (df['high'] >= df['open']) & (df['high'] >= df['close']))
print(f"OHLCV Violations: {violations.sum()}")
# 3. Completeness
daily = df.groupby(df['timestamp'].dt.date).size()
print(f"Completeness: {(daily == 1440).mean() * 100:.1f}%")
# 4. Volume
print(f"Zero Volume: {(df['volume'] == 0).mean() * 100:.1f}%")
🏆 Final Verdict
BEST FREE OPTION for backtesting and ML training.
Use Kaggle if: Free data, historical analysis, ML training
Skip Kaggle if: Need real-time, live trading, tick data
Hybrid Strategy: Kaggle (training) + Binance API (live)
📞 Quick Links
- Full Report:
analysis_report.json - Executive Summary:
EXECUTIVE_SUMMARY.md - Technical Details:
TECHNICAL_DETAILS.md - This Card:
QUICK_REFERENCE.md
Wave 153 | Agent: Kaggle Analysis | Status: ✅ COMPLETE