# Data Quality Validation Report
**Date**: 2025-10-13
**Symbols**: GC (Gold), ZN (Treasury), 6E (Euro FX)
**Period**: January 2-31, 2024 (29 days)
**Schema**: OHLCV-1m (1-minute bars)
**Dataset**: GLBX.MDP3 (CME Globex)

---

## Executive Summary

✅ **2 of 3 symbols PRODUCTION READY** (ZN.FUT, 6E.FUT)
⚠️ **1 symbol ACCEPTABLE quality** (GC - low liquidity/sparse data)

### Production Readiness by Symbol

| Symbol | Bars | Quality Score | OHLCV Violations | Zero Volumes | Large Gaps | Price Spikes | Production Ready |
|--------|------|---------------|------------------|--------------|------------|--------------|------------------|
| **GC** (Gold) | 781 | ACCEPTABLE | 0 | 0 (0.0%) | 225 (28.8%) | 0 | ⚠️ **REVIEW REQUIRED** |
| **ZN.FUT** (Treasury) | 28,935 | **EXCELLENT** | 0 | 0 (0.0%) | 197 (0.7%) | 0 | ✅ **YES** |
| **6E.FUT** (Euro FX) | 29,937 | **EXCELLENT** | 0 | 0 (0.0%) | 73 (0.2%) | 0 | ✅ **YES** |

---

## Detailed Analysis by Symbol

### 1. GC (Gold Futures - Continuous Contract)

**File**: `GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.dbn` (11 KB compressed → 43 KB uncompressed)

#### Statistics
- **Total bars**: 781 bars over 29 days (~28 bars/day average)
- **Coverage**: 2024-01-02 08:19:00 UTC to 2024-01-30 23:35:00 UTC (28 days, 15 hours)
- **Price range**: $2,005.29 - $2,073.69 (avg: $2,033.89)
- **Expected range**: $2,000.00 - $2,100.00 ✅ **WITHIN BOUNDS**
- **Volume**: Total 4,475 contracts (avg: 5.7/bar, max: 114, min: 1)

#### Quality Metrics
- ✅ **OHLCV violations**: 0 (perfect bar integrity)
- ✅ **Zero volumes**: 0 (0.0%)
- ⚠️ **Large gaps (>2 min)**: 225 (28.8%) - **HIGH**
- ✅ **Price spikes (>20%)**: 0

#### Assessment
**Quality Score**: ACCEPTABLE
**Production Readiness**: ⚠️ **REVIEW REQUIRED**

**Issues**:
1. **Very sparse data**: Only 781 bars over 29 days indicates low liquidity
2. **High gap frequency**: 28.8% of bars have >2-minute gaps (avg ~3.5 hours between bars)
3. **Continuous contract**: Symbol format `GC.c.0` suggests rolled continuous contract

**Recommendations**:
- ⚠️ **NOT recommended for high-frequency strategies** (1-minute bars too sparse)
- ✅ **Suitable for lower-frequency strategies** (hourly, daily)
- 💡 Consider downloading **specific contract** (e.g., `GCG24`) for better liquidity
- 💡 Alternatively, use **5-minute or 1-hour bars** for continuous contracts

---

### 2. ZN.FUT (10-Year Treasury Note Futures)

**File**: `ZN.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn` (315 KB compressed → 1.6 MB uncompressed)

#### Statistics
- **Total bars**: 28,935 bars over 29 days (~998 bars/day = ~16.6 hours/day)
- **Coverage**: 2024-01-02 00:00:00 UTC to 2024-01-31 23:59:00 UTC (29 days, 23 hours)
- **Price range**: $110.82 - $112.79 (avg: $111.76)
- **Expected range**: $110.00 - $113.00 ✅ **WITHIN BOUNDS**
- **Volume**: Total 5,022,468 contracts (avg: 173.6/bar, max: 4,890, min: 1)

#### Quality Metrics
- ✅ **OHLCV violations**: 0 (perfect bar integrity)
- ✅ **Zero volumes**: 0 (0.0%)
- ✅ **Large gaps (>2 min)**: 197 (0.7%) - **EXCELLENT** (expected for overnight/market close)
- ✅ **Price spikes (>20%)**: 0

#### Assessment
**Quality Score**: **EXCELLENT** 🌟
**Production Readiness**: ✅ **PRODUCTION READY**

**Strengths**:
- ✅ High data density (998 bars/day)
- ✅ Near-continuous coverage during trading hours
- ✅ Good liquidity (avg 174 contracts/bar)
- ✅ Zero quality violations
- ✅ Minimal gaps (0.7% - expected during non-trading hours)

**Recommendations**:
- ✅ **APPROVED for production backtesting**
- ✅ **Suitable for high-frequency strategies** (sub-minute execution)
- ✅ **Suitable for all timeframes** (1-min to daily)

---

### 3. 6E.FUT (Euro FX Futures - EUR/USD)

**File**: `6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn` (367 KB compressed → 1.7 MB uncompressed)

#### Statistics
- **Total bars**: 29,937 bars over 29 days (~1,032 bars/day = ~17.2 hours/day)
- **Coverage**: 2024-01-02 00:00:00 UTC to 2024-01-31 23:59:00 UTC (29 days, 23 hours)
- **Price range**: $1.0796 - $1.0987 (avg: $1.0892)
- **Expected range**: $1.08 - $1.11 ✅ **WITHIN BOUNDS**
- **Volume**: Total 4,310,088 contracts (avg: 143.8/bar, max: 7,272, min: 1)

#### Quality Metrics
- ✅ **OHLCV violations**: 0 (perfect bar integrity)
- ✅ **Zero volumes**: 0 (0.0%)
- ✅ **Large gaps (>2 min)**: 73 (0.2%) - **EXCELLENT** (expected for overnight/market close)
- ✅ **Price spikes (>20%)**: 0

#### Assessment
**Quality Score**: **EXCELLENT** 🌟
**Production Readiness**: ✅ **PRODUCTION READY**

**Strengths**:
- ✅ High data density (1,032 bars/day)
- ✅ Near-continuous coverage during trading hours
- ✅ Good liquidity (avg 144 contracts/bar)
- ✅ Zero quality violations
- ✅ Minimal gaps (0.2% - expected during non-trading hours)
- ✅ Stable FX market (low volatility, no spikes)

**Recommendations**:
- ✅ **APPROVED for production backtesting**
- ✅ **Suitable for high-frequency strategies** (sub-minute execution)
- ✅ **Suitable for all timeframes** (1-min to daily)
- ✅ **Ideal for FX algo trading** (24-hour market coverage)

---

## Technical Notes

### Data Format
- **Schema**: OHLCV-1m (1-minute candlestick bars)
- **Dataset**: GLBX.MDP3 (CME Globex Market Data Platform v3)
- **Compression**: Zstandard (required decompression for dbn 0.42.0)
- **Encoding**: DBN version 1 binary format

### Validation Methodology
- **OHLCV Relationships**: High ≥ {Open, Close, Low}, Low ≤ {Open, Close, High}
- **Price Spike Threshold**: >20% change between consecutive bars
- **Large Gap Threshold**: >120 seconds (2 minutes) between 1-minute bars
- **Zero Volume Detection**: Exact match (volume = 0)
- **Price Range Validation**: Asset-specific expected ranges with 10% tolerance

### Quality Score Criteria
- **EXCELLENT**: 0 OHLCV violations, <10% zero volumes, <5% gaps, 0 price spikes
- **GOOD**: <5 violations, <20% zero volumes, <10% gaps, 0 price spikes
- **ACCEPTABLE**: <10 violations
- **POOR**: ≥10 violations

---

## Recommendations

### Immediate Actions

1. **ZN.FUT (Treasury)** ✅
   - **Status**: PRODUCTION READY
   - **Action**: Proceed with backtesting strategies
   - **Suitable for**: All strategy types (HFT, swing, position)

2. **6E.FUT (Euro FX)** ✅
   - **Status**: PRODUCTION READY
   - **Action**: Proceed with backtesting strategies
   - **Suitable for**: All strategy types, especially FX-focused

3. **GC (Gold)** ⚠️
   - **Status**: ACCEPTABLE (requires review)
   - **Action**:
     - **Option A**: Use for lower-frequency strategies (hourly+) ✅
     - **Option B**: Download specific contract (e.g., `GCG24`, `GCJ24`) for better liquidity
     - **Option C**: Request 5-minute or 1-hour bars for continuous contract

### Future Data Acquisitions

**Recommended Additional Symbols**:
- **CL.FUT** (Crude Oil) - High liquidity, 24-hour trading
- **NQ.FUT** (NASDAQ-100 E-mini) - Tech index futures
- **RTY.FUT** (Russell 2000) - Small-cap futures
- **Specific GC contracts**: `GCG24` (Feb 2024), `GCJ24` (Apr 2024) for better liquidity

**Data Quality Preferences**:
- ✅ **Prefer uncompressed DBN files** (or use dbn 0.43+ with compression support)
- ✅ **Request specific contracts** over continuous for high-frequency work
- ✅ **Validate compression format** before download (Zstandard requires explicit handling)

---

## Appendix: File Inventory

### Downloaded Files

| File | Size (Compressed) | Size (Uncompressed) | Bars | Quality | Status |
|------|-------------------|---------------------|------|---------|--------|
| `GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.dbn` | 11 KB | 43 KB | 781 | ACCEPTABLE | ⚠️ Review |
| `ZN.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn` | 315 KB | 1.6 MB | 28,935 | EXCELLENT | ✅ Ready |
| `6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.dbn` | 367 KB | 1.7 MB | 29,937 | EXCELLENT | ✅ Ready |

### Decompressed Files (for dbn 0.42.0 compatibility)

- `GC_continuous_ohlcv-1m_2024-01-02_to_2024-01-31.uncompressed.dbn` (43 KB)
- `ZN.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.uncompressed.dbn` (1.6 MB)
- `6E.FUT_ohlcv-1m_2024-01-02_to_2024-01-31.uncompressed.dbn` (1.7 MB)

**Note**: Decompression required because `dbn` crate 0.42.0 does not support Zstandard-compressed files via `DbnDecoder::from_file()`. Consider upgrading to dbn 0.43+ for native compression support.

---

## Validation Tool

**Created**: `services/backtesting_service/examples/validate_multi_symbol.rs`

**Run validation**:
```bash
cargo run -p backtesting_service --example validate_multi_symbol
```

**Features**:
- Multi-symbol validation in single run
- Asset-specific price range validation
- Comprehensive quality metrics (OHLCV, gaps, spikes, volumes)
- Production readiness assessment
- Formatted summary table with recommendations

---

**Report Generated**: 2025-10-13
**Validation Tool**: `validate_multi_symbol.rs`
**Total Symbols Validated**: 3
**Production Ready**: 2 (66.7%)
**Review Required**: 1 (33.3%)