# Data Quality Validation

## Quick Reference

### Validate Data

```bash
# Single symbol
cargo run -p backtesting_service --bin validate_dbn_data -- --symbol ES.FUT

# All symbols
cargo run -p backtesting_service --bin validate_dbn_data -- --all

# With HTML report
./scripts/validate_data_quality.sh --all --output reports/
```

### CI/CD Integration

```bash
# Run validation in CI/CD (exits 1 on failure)
./scripts/validate_data_quality.sh --all --fail-on-poor-quality --min-quality 70
```

### Pre-commit Hook

```bash
# Install validation hook
git config core.hooksPath .githooks

# Now new DBN files will be validated before commit
git add test_data/real/databento/ES.FUT_new.dbn
git commit -m "Add new data"  # ← Validation runs automatically
```

## Quality Standards

| Score | Rating | Status |
|-------|--------|--------|
| 90-100 | EXCELLENT | ✅ Production Ready |
| 75-89 | GOOD | ✅ Production Ready |
| 60-74 | ACCEPTABLE | ⚠️ Review Recommended |
| 40-59 | POOR | ❌ Not Production Ready |
| 0-39 | CRITICAL | ❌ Data Corrupted |

## Validation Checks

- ✅ OHLCV relationship validation
- ✅ Price range sanity checks
- ✅ Volume validation (>0, realistic ranges)
- ✅ Timestamp continuity (no gaps, correct ordering)
- ✅ Anomaly detection (spikes, duplicates, corruption)
- ✅ Data completeness metrics

## Documentation

See [DATA_VALIDATION_GUIDE.md](docs/DATA_VALIDATION_GUIDE.md) for:
- Detailed usage instructions
- CI/CD integration guide
- Troubleshooting tips
- Performance benchmarks
- Best practices

## Files

```
services/backtesting_service/
  src/bin/validate_dbn_data.rs    # Validation tool (binary)

scripts/
  validate_data_quality.sh         # CI/CD integration script

.githooks/
  pre-commit-data-validation       # Git pre-commit hook

.github/workflows/
  data-quality-validation.yml      # GitHub Actions workflow

docs/
  DATA_VALIDATION_GUIDE.md         # Complete documentation
```

## Status

**Production Ready** ✅
- Comprehensive validation suite
- Multiple report formats (text, JSON, HTML)
- CI/CD integration with exit codes
- Pre-commit hook for data integrity
- Automated anomaly detection

## Quick Test

```bash
# Test validation tool
cargo run -p backtesting_service --bin validate_dbn_data -- \
  --symbol ES.FUT \
  --format html \
  --output /tmp/validation_report.html

# View report
xdg-open /tmp/validation_report.html
```

---

**Agent 18 Complete** - Comprehensive data quality validation tools deployed