gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e8a68ee39fcc2d5631bb1fba86865c0d59fb4ed3
foxhunt/test_data/real/databento/DATA_QUALITY_VALIDATION_REPORT.md
jgrusewski e8a68ee39f Download 360 DBN files (36.3 MB) using Rust databento client 
- Created data/examples/download_ml_training_data.rs using reqwest + Databento HTTP API
- Downloaded 90 days × 4 symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT)
- Files saved to test_data/real/databento/ml_training/
- Total: 360 files, 15 MB compressed DBN format
- Used existing Rust pattern from download_nq_fut.rs
- API key loaded from .env file
- 100% success rate (360/360 files)
- Ready for ML training benchmarks

Next: Create simplified training benchmark for RTX 3050 Ti GPU measurements
2025-10-13 13:30:02 +02:00

8.8 KiB
Raw Blame History

Data Quality Validation Report

Date: 2025-10-13 Symbols: GC (Gold), ZN (Treasury), 6E (Euro FX) Period: January 2-31, 2024 (29 days) Schema: OHLCV-1m (1-minute bars) Dataset: GLBX.MDP3 (CME Globex)

Executive Summary

✅ 2 of 3 symbols PRODUCTION READY (ZN.FUT, 6E.FUT) ⚠️ 1 symbol ACCEPTABLE quality (GC - low liquidity/sparse data)

Production Readiness by Symbol

Symbol Bars Quality Score OHLCV Violations Zero Volumes Large Gaps Price Spikes Production Ready
GC (Gold) 781 ACCEPTABLE 0 0 (0.0%) 225 (28.8%) 0 ⚠️ REVIEW REQUIRED
ZN.FUT (Treasury) 28,935 EXCELLENT 0 0 (0.0%) 197 (0.7%) 0 ✅ YES
6E.FUT (Euro FX) 29,937 EXCELLENT 0 0 (0.0%) 73 (0.2%) 0 ✅ YES

Detailed Analysis by Symbol

1. GC (Gold Futures - Continuous Contract)

File: GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.dbn (11 KB compressed → 43 KB uncompressed)

Statistics

  • Total bars: 781 bars over 29 days (~28 bars/day average)
  • Coverage: 2024-01-02 08:19:00 UTC to 2024-01-30 23:35:00 UTC (28 days, 15 hours)
  • Price range: $2,005.29 - $2,073.69 (avg: $2,033.89)
  • Expected range: $2,000.00 - $2,100.00 ✅ WITHIN BOUNDS
  • Volume: Total 4,475 contracts (avg: 5.7/bar, max: 114, min: 1)

Quality Metrics

  • ✅ OHLCV violations: 0 (perfect bar integrity)
  • ✅ Zero volumes: 0 (0.0%)
  • ⚠️ Large gaps (>2 min): 225 (28.8%) - HIGH
  • ✅ Price spikes (>20%): 0

Assessment

Quality Score: ACCEPTABLE Production Readiness: ⚠️ REVIEW REQUIRED

Issues:

  1. Very sparse data: Only 781 bars over 29 days indicates low liquidity
  2. High gap frequency: 28.8% of bars have >2-minute gaps (avg ~3.5 hours between bars)
  3. Continuous contract: Symbol format GC.c.0 suggests rolled continuous contract

Recommendations:

  • ⚠️ NOT recommended for high-frequency strategies (1-minute bars too sparse)
  • ✅ Suitable for lower-frequency strategies (hourly, daily)
  • 💡 Consider downloading specific contract (e.g., GCG24) for better liquidity
  • 💡 Alternatively, use 5-minute or 1-hour bars for continuous contracts

2. ZN.FUT (10-Year Treasury Note Futures)

File: ZN.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn (315 KB compressed → 1.6 MB uncompressed)

Statistics

  • Total bars: 28,935 bars over 29 days (~998 bars/day = ~16.6 hours/day)
  • Coverage: 2024-01-02 00:00:00 UTC to 2024-01-31 23:59:00 UTC (29 days, 23 hours)
  • Price range: $110.82 - $112.79 (avg: $111.76)
  • Expected range: $110.00 - $113.00 ✅ WITHIN BOUNDS
  • Volume: Total 5,022,468 contracts (avg: 173.6/bar, max: 4,890, min: 1)

Quality Metrics

  • ✅ OHLCV violations: 0 (perfect bar integrity)
  • ✅ Zero volumes: 0 (0.0%)
  • ✅ Large gaps (>2 min): 197 (0.7%) - EXCELLENT (expected for overnight/market close)
  • ✅ Price spikes (>20%): 0

Assessment

Quality Score: EXCELLENT 🌟 Production Readiness: ✅ PRODUCTION READY

Strengths:

  • ✅ High data density (998 bars/day)
  • ✅ Near-continuous coverage during trading hours
  • ✅ Good liquidity (avg 174 contracts/bar)
  • ✅ Zero quality violations
  • ✅ Minimal gaps (0.7% - expected during non-trading hours)

Recommendations:

  • ✅ APPROVED for production backtesting
  • ✅ Suitable for high-frequency strategies (sub-minute execution)
  • ✅ Suitable for all timeframes (1-min to daily)

3. 6E.FUT (Euro FX Futures - EUR/USD)

File: 6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn (367 KB compressed → 1.7 MB uncompressed)

Statistics

  • Total bars: 29,937 bars over 29 days (~1,032 bars/day = ~17.2 hours/day)
  • Coverage: 2024-01-02 00:00:00 UTC to 2024-01-31 23:59:00 UTC (29 days, 23 hours)
  • Price range: $1.0796 - $1.0987 (avg: $1.0892)
  • Expected range: $1.08 - $1.11 ✅ WITHIN BOUNDS
  • Volume: Total 4,310,088 contracts (avg: 143.8/bar, max: 7,272, min: 1)

Quality Metrics

  • ✅ OHLCV violations: 0 (perfect bar integrity)
  • ✅ Zero volumes: 0 (0.0%)
  • ✅ Large gaps (>2 min): 73 (0.2%) - EXCELLENT (expected for overnight/market close)
  • ✅ Price spikes (>20%): 0

Assessment

Quality Score: EXCELLENT 🌟 Production Readiness: ✅ PRODUCTION READY

Strengths:

  • ✅ High data density (1,032 bars/day)
  • ✅ Near-continuous coverage during trading hours
  • ✅ Good liquidity (avg 144 contracts/bar)
  • ✅ Zero quality violations
  • ✅ Minimal gaps (0.2% - expected during non-trading hours)
  • ✅ Stable FX market (low volatility, no spikes)

Recommendations:

  • ✅ APPROVED for production backtesting
  • ✅ Suitable for high-frequency strategies (sub-minute execution)
  • ✅ Suitable for all timeframes (1-min to daily)
  • ✅ Ideal for FX algo trading (24-hour market coverage)

Technical Notes

Data Format

  • Schema: OHLCV-1m (1-minute candlestick bars)
  • Dataset: GLBX.MDP3 (CME Globex Market Data Platform v3)
  • Compression: Zstandard (required decompression for dbn 0.42.0)
  • Encoding: DBN version 1 binary format

Validation Methodology

  • OHLCV Relationships: High ≥ {Open, Close, Low}, Low ≤ {Open, Close, High}
  • Price Spike Threshold: >20% change between consecutive bars
  • Large Gap Threshold: >120 seconds (2 minutes) between 1-minute bars
  • Zero Volume Detection: Exact match (volume = 0)
  • Price Range Validation: Asset-specific expected ranges with 10% tolerance

Quality Score Criteria

  • EXCELLENT: 0 OHLCV violations, <10% zero volumes, <5% gaps, 0 price spikes
  • GOOD: <5 violations, <20% zero volumes, <10% gaps, 0 price spikes
  • ACCEPTABLE: <10 violations
  • POOR: ≥10 violations

Recommendations

Immediate Actions

  1. ZN.FUT (Treasury) ✅

    • Status: PRODUCTION READY
    • Action: Proceed with backtesting strategies
    • Suitable for: All strategy types (HFT, swing, position)

  2. 6E.FUT (Euro FX) ✅

    • Status: PRODUCTION READY
    • Action: Proceed with backtesting strategies
    • Suitable for: All strategy types, especially FX-focused

  3. GC (Gold) ⚠️

    • Status: ACCEPTABLE (requires review)
    • Action:
      • Option A: Use for lower-frequency strategies (hourly+) ✅
      • Option B: Download specific contract (e.g., GCG24, GCJ24) for better liquidity
      • Option C: Request 5-minute or 1-hour bars for continuous contract

Future Data Acquisitions

Recommended Additional Symbols:

  • CL.FUT (Crude Oil) - High liquidity, 24-hour trading
  • NQ.FUT (NASDAQ-100 E-mini) - Tech index futures
  • RTY.FUT (Russell 2000) - Small-cap futures
  • Specific GC contracts: GCG24 (Feb 2024), GCJ24 (Apr 2024) for better liquidity

Data Quality Preferences:

  • ✅ Prefer uncompressed DBN files (or use dbn 0.43+ with compression support)
  • ✅ Request specific contracts over continuous for high-frequency work
  • ✅ Validate compression format before download (Zstandard requires explicit handling)

Appendix: File Inventory

Downloaded Files

File Size (Compressed) Size (Uncompressed) Bars Quality Status
GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.dbn 11 KB 43 KB 781 ACCEPTABLE ⚠️ Review
ZN.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn 315 KB 1.6 MB 28,935 EXCELLENT ✅ Ready
6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn 367 KB 1.7 MB 29,937 EXCELLENT ✅ Ready

Decompressed Files (for dbn 0.42.0 compatibility)

  • GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.uncompressed.dbn (43 KB)
  • ZN.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.uncompressed.dbn (1.6 MB)
  • 6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.uncompressed.dbn (1.7 MB)

Note: Decompression required because dbn crate 0.42.0 does not support Zstandard-compressed files via DbnDecoder::from_file(). Consider upgrading to dbn 0.43+ for native compression support.

Validation Tool

Created: services/backtesting_service/examples/validate_multi_symbol.rs

Run validation:

cargo run -p backtesting_service --example validate_multi_symbol

Features:

  • Multi-symbol validation in single run
  • Asset-specific price range validation
  • Comprehensive quality metrics (OHLCV, gaps, spikes, volumes)
  • Production readiness assessment
  • Formatted summary table with recommendations

Report Generated: 2025-10-13 Validation Tool: validate_multi_symbol.rs Total Symbols Validated: 3 Production Ready: 2 (66.7%) Review Required: 1 (33.3%)

Reference in New Issue View Git Blame Copy Permalink