foxhunt/docs/DATA_VALIDATION_GUIDE.md

DBN Data Quality Validation Guide

Overview

Comprehensive data quality validation tools for DBN (Databento Binary) market data files. These tools ensure data quality before backtesting and catch issues early in the data acquisition pipeline.

Features

• Multi-file validation - Validate individual symbols or all symbols at once
• Quality scoring (0-100) - Automated quality assessment with ratings
• Anomaly detection - Automated detection of price spikes, gaps, and corrupted data
• Multiple report formats - Text, JSON, and HTML reports
• CI/CD integration - Exit codes for automated validation pipelines
• Pre-commit hooks - Validate new data before committing

Quick Start

Validate Single Symbol

cargo run -p backtesting_service --bin validate_dbn_data -- \
  --symbol ES.FUT \
  --format text

Validate All Symbols

cargo run -p backtesting_service --bin validate_dbn_data -- \
  --all \
  --format html \
  --output validation_report.html

CI/CD Mode (Exit on Failure)

cargo run -p backtesting_service --bin validate_dbn_data -- \
  --all \
  --fail-on-poor-quality \
  --min-quality-score 70

Quality Checks

Critical Issues (Score -20)

• OHLCV Violations - Invalid price relationships (high < low, etc.)
• Negative Prices - Prices below zero (data corruption)
• Out-of-Order Timestamps - Chronological ordering broken

High Severity Issues (Score -10-15)

• Duplicate Timestamps - Multiple bars at same timestamp

Medium Severity Issues (Score -5)

• Price Spikes (>10%) - Abnormal price movements
• Zero Volumes (>10%) - Excessive bars with no volume

Low Severity Issues (Score -2)

• Timestamp Gaps (>5%) - Missing data periods
• Low Completeness (<80%) - Insufficient data coverage

Quality Ratings

Score	Rating	Status	Description
90-100	EXCELLENT	✅ Production Ready	No critical issues, minor anomalies only
75-89	GOOD	✅ Production Ready	Some minor issues, acceptable for production
60-74	ACCEPTABLE	⚠️ Caution	Notable issues, review recommended
40-59	POOR	❌ Not Ready	Significant quality problems
0-39	CRITICAL	❌ Blocked	Critical data corruption, unusable

Report Formats

Text Report (Console)

═══════════════════════════════════════════════════════
DBN DATA QUALITY VALIDATION REPORT
═══════════════════════════════════════════════════════
Timestamp: 2025-10-13T08:31:14.367677223+00:00
Duration: 3ms

📊 SUMMARY
  Total Symbols: 1
  Total Files: 1
  Total Bars: 1674
  Overall Quality: 90 (EXCELLENT)

─────────────────────────────────────────────────────
SYMBOL: ES.FUT
─────────────────────────────────────────────────────

📈 Statistics:
  Bars: 1674
  Price Range: $3604.99 - $5175.00
  Avg Close: $4822.21
  ...

JSON Report (API Integration)

{
  "timestamp": "2025-10-13T08:31:14.367677223+00:00",
  "total_symbols": 1,
  "total_bars": 1674,
  "overall_quality": {
    "score": 90,
    "rating": "EXCELLENT",
    "issues": ["294 duplicate timestamps"],
    "recommendations": ["Remove duplicate bars"]
  },
  "symbols": [...]
}

HTML Report (Dashboard)

Interactive HTML dashboard with:

• Color-coded quality scores
• Expandable anomaly details
• Symbol-level drill-down
• Export-ready format

CI/CD Integration

Shell Script (Recommended)

./scripts/validate_data_quality.sh \
  --all \
  --min-quality 70 \
  --output validation_reports

Exit Codes:

• 0 - All validations passed
• 1 - Quality check failed
• 2 - Validation error (no data, missing files)

GitHub Actions

Workflow automatically triggers on:

• Push to test_data/**/*.dbn
• Pull requests with DBN files
• Daily at 2 AM UTC (data degradation check)
• Manual workflow dispatch

# .github/workflows/data-quality-validation.yml
name: DBN Data Quality Validation

on:
  push:
    paths: ['test_data/**/*.dbn']
  pull_request:
    paths: ['test_data/**/*.dbn']
  schedule:
    - cron: '0 2 * * *'

Pre-commit Hook

# Install pre-commit hook
ln -sf ../../.githooks/pre-commit-data-validation .git/hooks/pre-commit

# Or configure git hooks path
git config core.hooksPath .githooks

The hook automatically:

1. Detects changed .dbn files
2. Validates affected symbols
3. Blocks commit if quality < 60

Skip hook (emergency only):

git commit --no-verify

CLI Reference

Options

Option	Description	Default
--symbol <SYMBOL>	Validate specific symbol	-
--all	Validate all symbols	false
--format <FORMAT>	Output format: text, json, html	text
--output <FILE>	Output file path	stdout
--fail-on-poor-quality	Exit 1 if quality fails	false
--min-quality-score <N>	Minimum score (0-100)	70
--verbose	Enable verbose output	false
--data-dir <PATH>	Test data directory	test_data/real/databento

Examples

Quick validation:

cargo run -p backtesting_service --bin validate_dbn_data -- --symbol ES.FUT

Full validation with HTML report:

cargo run -p backtesting_service --bin validate_dbn_data -- \
  --all \
  --format html \
  --output reports/validation_$(date +%Y%m%d).html

Strict CI/CD mode:

cargo run -p backtesting_service --bin validate_dbn_data -- \
  --all \
  --min-quality-score 80 \
  --fail-on-poor-quality

Validate specific date range (via script):

# Validate symbol with custom data directory
cargo run -p backtesting_service --bin validate_dbn_data -- \
  --symbol ES.FUT \
  --data-dir test_data/real/databento/2024-01

Anomaly Types

Price Spikes

Detection: >10% price change between consecutive bars

Example:

[MEDIUM] Price Spike at bar 145: 12.34% price change ($4800.00 -> $5392.00)

Causes:

• Flash crash events
• Data encoding errors
• Market microstructure noise

Timestamp Gaps

Detection: >2 minutes between consecutive 1-minute bars

Example:

[LOW] Large Gap at bar 250: 3600 seconds (60 minutes)

Causes:

• Market close/open transitions (expected)
• Trading halts
• Data acquisition interruptions

OHLCV Violations

Detection: Invalid price relationships

Example:

[HIGH] OHLCV Violation at bar 42: O=4822.50 H=4820.00 L=4825.00 C=4823.00

Causes:

• Data corruption
• Encoding errors
• Incorrect parsing

Duplicate Timestamps

Detection: Multiple bars at same timestamp

Example:

[MEDIUM] Duplicate Timestamp at bar 99: Duplicate timestamp found

Causes:

• Data source overlaps
• Incorrect data merging
• Replay buffer issues

Troubleshooting

No DBN Files Found

Error:

No DBN files found in: test_data/real/databento

Solutions:

1. Check data directory path: --data-dir <path>
2. Verify files exist: ls test_data/real/databento/*.dbn
3. Check file permissions

Invalid DBN Header

Error:

Failed to create DBN decoder for file: ES.FUT_ohlcv-1m_2024-01-02.dbn
Caused by: decoding error: invalid DBN header

Solutions:

1. Verify file is valid DBN format: file <filename>.dbn
2. Re-download corrupted file
3. Check DBN version compatibility

Quality Score Below Threshold

Error:

❌ VALIDATION FAILED: Quality score 65 < minimum 70

Solutions:

1. Review validation report for specific issues
2. Fix data quality problems (deduplicate, correct timestamps)
3. Lower threshold if issues are acceptable: --min-quality-score 60

Performance

Benchmarks

Symbol	Files	Bars	Load Time	Validation Time	Total
ES.FUT	1	1,674	1.3ms	1.7ms	3ms
ESH4	3	5,022	3.9ms	5.1ms	9ms
NQ.FUT	1	1,674	1.2ms	1.6ms	2.8ms
All	6	8,370	6.4ms	8.4ms	14.8ms

Target: <10ms per file (ACHIEVED ✅)

Optimization Tips

1. Use --release mode for production validation (25x faster)
2. Validate specific symbols during development
3. Cache validation reports in CI/CD pipelines
4. Parallel validation for large symbol sets (future enhancement)

Best Practices

Development

1. Validate before backtesting - Catch data issues early
2. Review anomalies - Understand data quality characteristics
3. Track quality scores - Monitor data degradation over time

Production

1. CI/CD integration - Block bad data from merging
2. Automated monitoring - Daily validation checks
3. Quality thresholds - Enforce minimum standards (>70)
4. Alert on failures - Notify team of quality issues

Data Acquisition

1. Pre-commit validation - Check new data before commit
2. Multi-file validation - Ensure consistency across dates
3. Cross-symbol validation - Check price correlations (future)
4. Incremental validation - Only validate changed files

Future Enhancements

• Cross-symbol correlation analysis
• Time alignment validation across symbols
• Statistical anomaly detection (ML-based)
• Historical quality tracking (trends)
• Parallel validation for large datasets
• WebAssembly build for browser validation
• Real-time validation during data streaming
• Automated data repair suggestions

Support

Issues: Report validation bugs or feature requests via GitHub Issues

Documentation: See TESTING_PLAN.md for backtesting strategy

Examples: See services/backtesting_service/examples/ for usage patterns

---

Last Updated: 2025-10-13 Tool Version: 1.0.0 Status: Production Ready ✅