foxhunt/docs/DATA_VALIDATION_GUIDE.md

# DBN Data Quality Validation Guide

## Overview

Comprehensive data quality validation tools for DBN (Databento Binary) market data files. These tools ensure data quality before backtesting and catch issues early in the data acquisition pipeline.

## Features

- **Multi-file validation** - Validate individual symbols or all symbols at once
- **Quality scoring (0-100)** - Automated quality assessment with ratings
- **Anomaly detection** - Automated detection of price spikes, gaps, and corrupted data
- **Multiple report formats** - Text, JSON, and HTML reports
- **CI/CD integration** - Exit codes for automated validation pipelines
- **Pre-commit hooks** - Validate new data before committing

## Quick Start

### Validate Single Symbol

```bash
cargo run -p backtesting_service --bin validate_dbn_data -- \
  --symbol ES.FUT \
  --format text
```

### Validate All Symbols

```bash
cargo run -p backtesting_service --bin validate_dbn_data -- \
  --all \
  --format html \
  --output validation_report.html
```

### CI/CD Mode (Exit on Failure)

```bash
cargo run -p backtesting_service --bin validate_dbn_data -- \
  --all \
  --fail-on-poor-quality \
  --min-quality-score 70
```

## Quality Checks

### Critical Issues (Score -20)

- **OHLCV Violations** - Invalid price relationships (high < low, etc.)
- **Negative Prices** - Prices below zero (data corruption)
- **Out-of-Order Timestamps** - Chronological ordering broken

### High Severity Issues (Score -10-15)

- **Duplicate Timestamps** - Multiple bars at same timestamp

### Medium Severity Issues (Score -5)

- **Price Spikes (>10%)** - Abnormal price movements
- **Zero Volumes (>10%)** - Excessive bars with no volume

### Low Severity Issues (Score -2)

- **Timestamp Gaps (>5%)** - Missing data periods
- **Low Completeness (<80%)** - Insufficient data coverage

## Quality Ratings

| Score | Rating | Status | Description |
|-------|--------|--------|-------------|
| 90-100 | EXCELLENT | ✅ Production Ready | No critical issues, minor anomalies only |
| 75-89 | GOOD | ✅ Production Ready | Some minor issues, acceptable for production |
| 60-74 | ACCEPTABLE | ⚠️ Caution | Notable issues, review recommended |
| 40-59 | POOR | ❌ Not Ready | Significant quality problems |
| 0-39 | CRITICAL | ❌ Blocked | Critical data corruption, unusable |

## Report Formats

### Text Report (Console)

```
═══════════════════════════════════════════════════════
DBN DATA QUALITY VALIDATION REPORT
═══════════════════════════════════════════════════════
Timestamp: 2025-10-13T08:31:14.367677223+00:00
Duration: 3ms

📊 SUMMARY
  Total Symbols: 1
  Total Files: 1
  Total Bars: 1674
  Overall Quality: 90 (EXCELLENT)

─────────────────────────────────────────────────────
SYMBOL: ES.FUT
─────────────────────────────────────────────────────

📈 Statistics:
  Bars: 1674
  Price Range: $3604.99 - $5175.00
  Avg Close: $4822.21
  ...
```

### JSON Report (API Integration)

```json
{
  "timestamp": "2025-10-13T08:31:14.367677223+00:00",
  "total_symbols": 1,
  "total_bars": 1674,
  "overall_quality": {
    "score": 90,
    "rating": "EXCELLENT",
    "issues": ["294 duplicate timestamps"],
    "recommendations": ["Remove duplicate bars"]
  },
  "symbols": [...]
}
```

### HTML Report (Dashboard)

Interactive HTML dashboard with:
- Color-coded quality scores
- Expandable anomaly details
- Symbol-level drill-down
- Export-ready format

## CI/CD Integration

### Shell Script (Recommended)

```bash
./scripts/validate_data_quality.sh \
  --all \
  --min-quality 70 \
  --output validation_reports
```

**Exit Codes:**
- `0` - All validations passed
- `1` - Quality check failed
- `2` - Validation error (no data, missing files)

### GitHub Actions

Workflow automatically triggers on:
- Push to `test_data/**/*.dbn`
- Pull requests with DBN files
- Daily at 2 AM UTC (data degradation check)
- Manual workflow dispatch

```yaml
# .github/workflows/data-quality-validation.yml
name: DBN Data Quality Validation

on:
  push:
    paths: ['test_data/**/*.dbn']
  pull_request:
    paths: ['test_data/**/*.dbn']
  schedule:
    - cron: '0 2 * * *'
```

### Pre-commit Hook

```bash
# Install pre-commit hook
ln -sf ../../.githooks/pre-commit-data-validation .git/hooks/pre-commit

# Or configure git hooks path
git config core.hooksPath .githooks
```

The hook automatically:
1. Detects changed `.dbn` files
2. Validates affected symbols
3. Blocks commit if quality < 60

**Skip hook (emergency only):**
```bash
git commit --no-verify
```

## CLI Reference

### Options

| Option | Description | Default |
|--------|-------------|---------|
| `--symbol <SYMBOL>` | Validate specific symbol | - |
| `--all` | Validate all symbols | false |
| `--format <FORMAT>` | Output format: text, json, html | text |
| `--output <FILE>` | Output file path | stdout |
| `--fail-on-poor-quality` | Exit 1 if quality fails | false |
| `--min-quality-score <N>` | Minimum score (0-100) | 70 |
| `--verbose` | Enable verbose output | false |
| `--data-dir <PATH>` | Test data directory | test_data/real/databento |

### Examples

**Quick validation:**
```bash
cargo run -p backtesting_service --bin validate_dbn_data -- --symbol ES.FUT
```

**Full validation with HTML report:**
```bash
cargo run -p backtesting_service --bin validate_dbn_data -- \
  --all \
  --format html \
  --output reports/validation_$(date +%Y%m%d).html
```

**Strict CI/CD mode:**
```bash
cargo run -p backtesting_service --bin validate_dbn_data -- \
  --all \
  --min-quality-score 80 \
  --fail-on-poor-quality
```

**Validate specific date range (via script):**
```bash
# Validate symbol with custom data directory
cargo run -p backtesting_service --bin validate_dbn_data -- \
  --symbol ES.FUT \
  --data-dir test_data/real/databento/2024-01
```

## Anomaly Types

### Price Spikes

**Detection:** >10% price change between consecutive bars

**Example:**
```
[MEDIUM] Price Spike at bar 145: 12.34% price change ($4800.00 -> $5392.00)
```

**Causes:**
- Flash crash events
- Data encoding errors
- Market microstructure noise

### Timestamp Gaps

**Detection:** >2 minutes between consecutive 1-minute bars

**Example:**
```
[LOW] Large Gap at bar 250: 3600 seconds (60 minutes)
```

**Causes:**
- Market close/open transitions (expected)
- Trading halts
- Data acquisition interruptions

### OHLCV Violations

**Detection:** Invalid price relationships

**Example:**
```
[HIGH] OHLCV Violation at bar 42: O=4822.50 H=4820.00 L=4825.00 C=4823.00
```

**Causes:**
- Data corruption
- Encoding errors
- Incorrect parsing

### Duplicate Timestamps

**Detection:** Multiple bars at same timestamp

**Example:**
```
[MEDIUM] Duplicate Timestamp at bar 99: Duplicate timestamp found
```

**Causes:**
- Data source overlaps
- Incorrect data merging
- Replay buffer issues

## Troubleshooting

### No DBN Files Found

**Error:**
```
No DBN files found in: test_data/real/databento
```

**Solutions:**
1. Check data directory path: `--data-dir <path>`
2. Verify files exist: `ls test_data/real/databento/*.dbn`
3. Check file permissions

### Invalid DBN Header

**Error:**
```
Failed to create DBN decoder for file: ES.FUT_ohlcv-1m_2024-01-02.dbn
Caused by: decoding error: invalid DBN header
```

**Solutions:**
1. Verify file is valid DBN format: `file <filename>.dbn`
2. Re-download corrupted file
3. Check DBN version compatibility

### Quality Score Below Threshold

**Error:**
```
❌ VALIDATION FAILED: Quality score 65 < minimum 70
```

**Solutions:**
1. Review validation report for specific issues
2. Fix data quality problems (deduplicate, correct timestamps)
3. Lower threshold if issues are acceptable: `--min-quality-score 60`

## Performance

### Benchmarks

| Symbol | Files | Bars | Load Time | Validation Time | Total |
|--------|-------|------|-----------|-----------------|-------|
| ES.FUT | 1 | 1,674 | 1.3ms | 1.7ms | 3ms |
| ESH4 | 3 | 5,022 | 3.9ms | 5.1ms | 9ms |
| NQ.FUT | 1 | 1,674 | 1.2ms | 1.6ms | 2.8ms |
| All | 6 | 8,370 | 6.4ms | 8.4ms | 14.8ms |

**Target:** <10ms per file (ACHIEVED ✅)

### Optimization Tips

1. **Use `--release` mode** for production validation (25x faster)
2. **Validate specific symbols** during development
3. **Cache validation reports** in CI/CD pipelines
4. **Parallel validation** for large symbol sets (future enhancement)

## Best Practices

### Development

1. **Validate before backtesting** - Catch data issues early
2. **Review anomalies** - Understand data quality characteristics
3. **Track quality scores** - Monitor data degradation over time

### Production

1. **CI/CD integration** - Block bad data from merging
2. **Automated monitoring** - Daily validation checks
3. **Quality thresholds** - Enforce minimum standards (>70)
4. **Alert on failures** - Notify team of quality issues

### Data Acquisition

1. **Pre-commit validation** - Check new data before commit
2. **Multi-file validation** - Ensure consistency across dates
3. **Cross-symbol validation** - Check price correlations (future)
4. **Incremental validation** - Only validate changed files

## Future Enhancements

- [ ] Cross-symbol correlation analysis
- [ ] Time alignment validation across symbols
- [ ] Statistical anomaly detection (ML-based)
- [ ] Historical quality tracking (trends)
- [ ] Parallel validation for large datasets
- [ ] WebAssembly build for browser validation
- [ ] Real-time validation during data streaming
- [ ] Automated data repair suggestions

## Support

**Issues:** Report validation bugs or feature requests via GitHub Issues

**Documentation:** See `TESTING_PLAN.md` for backtesting strategy

**Examples:** See `services/backtesting_service/examples/` for usage patterns

---

**Last Updated:** 2025-10-13
**Tool Version:** 1.0.0
**Status:** Production Ready ✅