foxhunt/docs/DBN_TROUBLESHOOTING.md

DBN Troubleshooting Guide

Version: 1.0 Last Updated: 2025-10-13

---

Table of Contents

1. Common Errors
2. Data Quality Issues
3. Performance Problems
4. File Format Issues
5. Integration Issues
6. Debugging Tools

---

Common Errors

Error: "DBN file not found"

Symptom:

Error: DBN file not found: test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

Cause: File path is incorrect or relative paths are not resolving correctly.

Solution 1: Use workspace-relative paths

fn get_test_file_path() -> String {
    let workspace_root = std::env::current_dir()
        .unwrap()
        .ancestors()
        .find(|p| p.join("Cargo.toml").exists() && p.join("test_data").exists())
        .expect("Could not find workspace root");

    workspace_root
        .join("test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn")
        .to_string_lossy()
        .to_string()
}

Solution 2: Use absolute paths

let file_path = "/home/jgrusewski/Work/foxhunt/test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn";

Solution 3: Verify file exists

ls -lh test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn
# Should output: -rw-rw-r-- 1 user user 95K Oct 12 23:22 ES.FUT_ohlcv-1m_2024-01-02.dbn

---

Error: "No DBN file configured for symbol"

Symptom:

Error: No DBN file configured for symbol: UNKNOWN

Cause: Symbol not in file mapping.

Solution:

// Check available symbols first
let symbols = data_source.available_symbols();
println!("Available symbols: {:?}", symbols);

// Add missing symbol
data_source.add_symbol_mapping(
    "UNKNOWN".to_string(),
    "path/to/file.dbn".to_string()
);

---

Error: "Invalid message length"

Symptom:

WARN Invalid message length: 0 at offset 1234

Cause: Corrupted DBN file or incorrect file format.

Diagnosis:

# Check file size (should be > 0)
ls -lh test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

# Check file is not empty
du -h test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

# Try to open with dbn CLI tool (if available)
dbn info test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

Solution: Re-download or regenerate the DBN file.

---

Error: "Failed to decode DBN record"

Symptom:

Error: Failed to decode DBN record at offset 5678

Cause: File format mismatch or version incompatibility.

Solution:

// Enable version upgrades
let mut decoder = DbnDecoder::from_file(file_path)?;
decoder.set_upgrade_policy(VersionUpgradePolicy::UpgradeToV3)?;

Check DBN version compatibility:

• Foxhunt uses dbn crate v0.9+
• Supports DBN format v1, v2, v3
• Automatic version upgrading enabled by default

---

Data Quality Issues

Issue: Unrealistic Prices

Symptom:

Bar 150: ES.FUT price 47.4275 outside realistic range (3000-6000)

Cause: Price encoding anomaly (7 decimal places instead of 9).

Automatic Fix: The system automatically detects and corrects these:

// Price anomaly detection (automatic in load_ohlcv_bars)
if pct_change > 0.5 && close_f64 < 1000.0 {
    let corrected_close = close_f64 * 100.0;
    if corrected_close >= 3000.0 && corrected_close <= 6000.0 {
        // Apply 100x correction
        open_f64 *= 100.0;
        high_f64 *= 100.0;
        low_f64 *= 100.0;
        close_f64 = corrected_close;
    }
}

Verification:

// Check correction logs
RUST_LOG=debug cargo test -p backtesting_service test_load_real_dbn_file

// Look for lines like:
// DEBUG Applied 100x price correction at bar 150 (99% change, $47.43 -> $4742.75)

Manual Validation:

async fn validate_prices() -> Result<()> {
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;

    for (i, bar) in bars.iter().enumerate() {
        let close_f64 = bar.close.to_string().parse::<f64>().unwrap();

        if close_f64 < 3000.0 || close_f64 > 6000.0 {
            eprintln!("WARN: Bar {} has unusual price: ${:.2}", i, close_f64);
        }
    }

    Ok(())
}

---

Issue: OHLCV Relationship Violations

Symptom:

Quality issue at bar 42: high < low

Diagnosis:

fn diagnose_ohlcv_issues(bars: &[MarketData]) {
    for (i, bar) in bars.iter().enumerate() {
        if bar.high < bar.low {
            eprintln!("Bar {}: high ({}) < low ({})", i, bar.high, bar.low);
        }
        if bar.high < bar.open || bar.high < bar.close {
            eprintln!("Bar {}: high not highest value", i);
        }
        if bar.low > bar.open || bar.low > bar.close {
            eprintln!("Bar {}: low not lowest value", i);
        }
    }
}

Solution: This indicates data corruption. Options:

1. Re-download the DBN file
2. Skip corrupted bars
3. Contact data provider

// Skip corrupted bars
let valid_bars: Vec<_> = bars.into_iter()
    .filter(|bar| {
        bar.high >= bar.low &&
        bar.high >= bar.open &&
        bar.high >= bar.close &&
        bar.low <= bar.open &&
        bar.low <= bar.close
    })
    .collect();

---

Issue: Missing or Sparse Data

Symptom:

Expected ~390 bars, got 150

Diagnosis:

async fn analyze_data_density() -> Result<()> {
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;

    if bars.len() < 350 {
        eprintln!("WARN: Expected ~390 bars (1-minute, full trading day), got {}", bars.len());
    }

    // Check for gaps
    for i in 1..bars.len() {
        let gap = bars[i].timestamp - bars[i-1].timestamp;
        if gap > chrono::Duration::minutes(5) {
            eprintln!("Data gap detected at bar {}: {} minute gap", i, gap.num_minutes());
        }
    }

    Ok(())
}

Common Causes:

1. Partial day data (market opened late, closed early)
2. Data feed interruption
3. Filter applied during download

Solution: Verify data source coverage and re-download if needed.

---

Issue: Timestamp Anomalies

Symptom:

Bar 200: Timestamp 2024-01-02 15:30:00 (jumped backwards)

Diagnosis:

fn check_timestamp_ordering(bars: &[MarketData]) {
    for i in 1..bars.len() {
        if bars[i].timestamp < bars[i-1].timestamp {
            eprintln!(
                "Timestamp ordering issue at bar {}: {} < {}",
                i,
                bars[i].timestamp,
                bars[i-1].timestamp
            );
        }
    }
}

Solution: Sort bars after loading (already done by load_ohlcv_bars):

// Automatic sorting
bars.sort_by(|a, b| a.timestamp.cmp(&b.timestamp));

---

Performance Problems

Issue: Slow Loading (>100ms for ~400 bars)

Target: <10ms for ~400 bars Actual: >100ms

Diagnosis:

use std::time::Instant;

let start = Instant::now();
let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
let duration = start.elapsed();

println!("Loaded {} bars in {:?} ({:.2}ms)",
    bars.len(),
    duration,
    duration.as_secs_f64() * 1000.0
);

if duration.as_millis() > 100 {
    eprintln!("WARN: Performance target missed");
}

Common Causes & Solutions:

1. Cold File System Cache

Solution: Warm up cache first

// Warm-up run
let _ = data_source.load_ohlcv_bars("ES.FUT").await?;

// Timed run (will be faster)
let start = Instant::now();
let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
let duration = start.elapsed();

2. Disk I/O Bottleneck

Check disk performance:

# Test read speed
dd if=test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn of=/dev/null bs=1M

# Should be >100 MB/s on modern SSD

Solution: Use SSD storage, not HDD

3. Debug Logging Overhead

Disable verbose logging:

# ❌ Slow (debug logging)
RUST_LOG=debug cargo run

# ✅ Fast (info or warn only)
RUST_LOG=info cargo run

4. Memory Allocations

Pre-allocate vectors:

// ✅ Good (pre-allocated)
let mut bars = Vec::with_capacity(400);

// ❌ Slow (reallocates multiple times)
let mut bars = Vec::new();

---

Issue: High Memory Usage

Symptom: Process uses >1GB RAM for loading 10 files.

Diagnosis:

use sysinfo::{System, SystemExt, ProcessExt};

let mut sys = System::new_all();
sys.refresh_all();

if let Some(process) = sys.process(sysinfo::get_current_pid().unwrap()) {
    println!("Memory usage: {} MB", process.memory() / 1024);
}

Solutions:

1. Disable Caching

let data_source = DbnDataSource::new(file_mapping)
    .await?
    .with_cache_limit(0); // No caching

2. Stream Processing

// Process files one at a time
for symbol in symbols {
    let bars = data_source.load_ohlcv_bars(&symbol).await?;
    process_bars(&bars);
    // Bars dropped here, memory freed
}

3. Limit LRU Cache Size

let data_source = DbnDataSource::new(file_mapping)
    .await?
    .with_cache_limit(3); // Only cache last 3 symbols

---

File Format Issues

Issue: Unknown File Format

Symptom:

Error: Failed to create DBN decoder for file: unknown format

Verify file format:

# Check file magic bytes
xxd -l 16 test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

# Should start with DBN magic bytes (0x64 0x62 0x6e)

Verify DBN schema:

# If you have dbn CLI tool
dbn info test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

# Expected output:
# Schema: OHLCV-1m
# Compression: none
# Records: 390

---

Issue: Unsupported Schema

Symptom:

WARN Unknown message type: 0x52

Supported Schemas:

• ✅ OHLCV-1m (1-minute bars) - Fully supported
• ✅ Trade messages - Supported via DbnParser
• ✅ Quote messages - Supported via DbnParser
• ❌ MBO (Market By Order) - Not yet implemented
• ❌ MBP (Market By Price) - Not yet implemented

Workaround: Convert unsupported schema to OHLCV using databento tools:

# Convert trades to OHLCV-1m
dbn convert --schema ohlcv-1m --interval 1m input.dbn output.dbn

---

Integration Issues

Issue: Repository Interface Not Working

Symptom:

Error: the trait bound `DbnMarketDataRepository: MarketDataRepository` is not satisfied

Solution: Import the trait

use backtesting_service::repositories::MarketDataRepository;

// Now trait methods available
let data = repo.load_historical_data(&symbols, start_time, end_time).await?;

---

Issue: Timestamp Format Mismatch

Symptom:

Error: Invalid timestamp: expected nanoseconds, got seconds

DBN uses nanoseconds (i64):

// ✅ Correct (nanoseconds since Unix epoch)
let start_time = 1704153600_000_000_000i64; // 2024-01-02 00:00:00

// ❌ Wrong (seconds)
let start_time = 1704153600i64;

Convert DateTime to nanoseconds:

use chrono::{TimeZone, Utc};

let dt = Utc.with_ymd_and_hms(2024, 1, 2, 0, 0, 0).unwrap();
let nanos = dt.timestamp_nanos_opt().unwrap();

---

Issue: Symbol Mapping Not Working

Symptom:

Loaded 0 bars for BTC/USD (expected ES.FUT data)

Diagnosis:

// Check symbol mappings
let repo = DbnMarketDataRepository::new_with_mappings(
    file_mapping,
    symbol_mappings
).await?;

// Add debug logging
RUST_LOG=debug cargo run
// Look for: "Symbol mapping: BTC/USD -> ES.FUT"

Solution: Verify mapping configuration

let mut symbol_mappings = HashMap::new();
symbol_mappings.insert("BTC/USD".to_string(), "ES.FUT".to_string());
symbol_mappings.insert("ETH/USD".to_string(), "ES.FUT".to_string());

let repo = DbnMarketDataRepository::new_with_mappings(
    file_mapping,
    symbol_mappings // ← Make sure this is passed!
).await?;

---

Debugging Tools

Tool 1: Data Validation Script

// Save as: scripts/validate_dbn_data.rs

use backtesting_service::dbn_data_source::DbnDataSource;
use std::collections::HashMap;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let mut file_mapping = HashMap::new();
    file_mapping.insert(
        "ES.FUT".to_string(),
        "test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn".to_string()
    );

    let data_source = DbnDataSource::new(file_mapping).await?;
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;

    println!("=== DBN Data Validation Report ===");
    println!("File: ES.FUT_ohlcv-1m_2024-01-02.dbn");
    println!("Bars loaded: {}", bars.len());

    // Check expected count
    if bars.len() < 350 || bars.len() > 450 {
        eprintln!("⚠️  WARN: Expected ~390 bars, got {}", bars.len());
    } else {
        println!("✅ Bar count in expected range (350-450)");
    }

    // Validate OHLCV relationships
    let mut quality_issues = 0;
    for (i, bar) in bars.iter().enumerate() {
        if !(bar.high >= bar.low &&
             bar.high >= bar.open &&
             bar.high >= bar.close &&
             bar.low <= bar.open &&
             bar.low <= bar.close) {
            quality_issues += 1;
            if quality_issues <= 3 {
                eprintln!("⚠️  Bar {} OHLCV violation", i);
            }
        }
    }

    if quality_issues == 0 {
        println!("✅ All bars passed OHLCV validation");
    } else {
        eprintln!("❌ {} bars failed OHLCV validation", quality_issues);
    }

    // Check price ranges
    let mut price_issues = 0;
    for (i, bar) in bars.iter().enumerate() {
        let close_f64 = bar.close.to_string().parse::<f64>().unwrap();
        if close_f64 < 3000.0 || close_f64 > 6000.0 {
            price_issues += 1;
            if price_issues <= 3 {
                eprintln!("⚠️  Bar {} unusual price: ${:.2}", i, close_f64);
            }
        }
    }

    if price_issues == 0 {
        println!("✅ All prices in realistic range (3000-6000)");
    } else {
        eprintln!("❌ {} bars with unusual prices", price_issues);
    }

    // Check timestamp ordering
    let mut timestamp_issues = 0;
    for i in 1..bars.len() {
        if bars[i].timestamp < bars[i-1].timestamp {
            timestamp_issues += 1;
        }
    }

    if timestamp_issues == 0 {
        println!("✅ Timestamps properly ordered");
    } else {
        eprintln!("❌ {} timestamp ordering issues", timestamp_issues);
    }

    println!("\n=== Summary ===");
    println!("First bar: {} @ {}", bars[0].close, bars[0].timestamp);
    println!("Last bar:  {} @ {}", bars[bars.len()-1].close, bars[bars.len()-1].timestamp);

    Ok(())
}

Run with:

cargo run --bin validate_dbn_data

---

Tool 2: Performance Profiler

// Save as: scripts/profile_dbn_loading.rs

use backtesting_service::dbn_data_source::DbnDataSource;
use std::collections::HashMap;
use std::time::Instant;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    println!("=== DBN Loading Performance Profile ===\n");

    let mut file_mapping = HashMap::new();
    file_mapping.insert(
        "ES.FUT".to_string(),
        "test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn".to_string()
    );

    // Test 1: Cold load
    println!("Test 1: Cold load (first time)");
    let data_source = DbnDataSource::new(file_mapping.clone()).await?;
    let start = Instant::now();
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
    let cold_duration = start.elapsed();
    println!("  Duration: {:?} ({:.2}ms)", cold_duration, cold_duration.as_secs_f64() * 1000.0);
    println!("  Bars: {}", bars.len());

    // Test 2: Warm load
    println!("\nTest 2: Warm load (cached in OS)");
    let start = Instant::now();
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
    let warm_duration = start.elapsed();
    println!("  Duration: {:?} ({:.2}ms)", warm_duration, warm_duration.as_secs_f64() * 1000.0);
    println!("  Bars: {}", bars.len());

    // Test 3: With caching enabled
    println!("\nTest 3: With LRU cache (immediate)");
    let data_source = DbnDataSource::new(file_mapping.clone())
        .await?
        .with_cache_limit(10);

    // Prime cache
    let _ = data_source.load_ohlcv_bars("ES.FUT").await?;

    let start = Instant::now();
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
    let cached_duration = start.elapsed();
    println!("  Duration: {:?} ({:.2}ms)", cached_duration, cached_duration.as_secs_f64() * 1000.0);
    println!("  Bars: {}", bars.len());

    // Performance summary
    println!("\n=== Performance Summary ===");
    println!("Target: <10ms");
    println!("Cold load:   {:.2}ms {}",
        cold_duration.as_secs_f64() * 1000.0,
        if cold_duration.as_millis() < 10 { "✅" } else { "⚠️" }
    );
    println!("Warm load:   {:.2}ms {}",
        warm_duration.as_secs_f64() * 1000.0,
        if warm_duration.as_millis() < 10 { "✅" } else { "⚠️" }
    );
    println!("Cached load: {:.2}ms ✅",
        cached_duration.as_secs_f64() * 1000.0
    );

    // Throughput
    let bars_per_sec = (bars.len() as f64 / warm_duration.as_secs_f64()) as u64;
    println!("\nThroughput: {} bars/sec", bars_per_sec);

    Ok(())
}

Run with:

cargo run --bin profile_dbn_loading

---

Tool 3: Data Explorer

// Save as: scripts/explore_dbn_data.rs

use backtesting_service::dbn_data_source::DbnDataSource;
use std::collections::HashMap;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let symbol = std::env::args().nth(1).unwrap_or_else(|| "ES.FUT".to_string());
    let file_path = std::env::args().nth(2).unwrap_or_else(|| {
        "test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn".to_string()
    });

    println!("=== DBN Data Explorer ===");
    println!("Symbol: {}", symbol);
    println!("File: {}\n", file_path);

    let mut file_mapping = HashMap::new();
    file_mapping.insert(symbol.clone(), file_path);

    let data_source = DbnDataSource::new(file_mapping).await?;
    let bars = data_source.load_ohlcv_bars(&symbol).await?;

    println!("Total bars: {}", bars.len());
    println!("Time range: {} to {}", bars[0].timestamp, bars[bars.len()-1].timestamp);

    // Sample bars
    println!("\n=== First 5 bars ===");
    for (i, bar) in bars.iter().take(5).enumerate() {
        println!("[{:03}] {} @ {} | O:{} H:{} L:{} C:{} V:{}",
            i,
            bar.symbol,
            bar.timestamp.format("%Y-%m-%d %H:%M:%S"),
            bar.open,
            bar.high,
            bar.low,
            bar.close,
            bar.volume
        );
    }

    println!("\n=== Last 5 bars ===");
    for (i, bar) in bars.iter().rev().take(5).rev().enumerate() {
        let idx = bars.len() - 5 + i;
        println!("[{:03}] {} @ {} | O:{} H:{} L:{} C:{} V:{}",
            idx,
            bar.symbol,
            bar.timestamp.format("%Y-%m-%d %H:%M:%S"),
            bar.open,
            bar.high,
            bar.low,
            bar.close,
            bar.volume
        );
    }

    // Statistics
    let closes: Vec<f64> = bars.iter()
        .map(|b| b.close.to_string().parse().unwrap())
        .collect();

    let mean = closes.iter().sum::<f64>() / closes.len() as f64;
    let min = closes.iter().cloned().fold(f64::INFINITY, f64::min);
    let max = closes.iter().cloned().fold(f64::NEG_INFINITY, f64::max);

    println!("\n=== Statistics ===");
    println!("Mean close: ${:.2}", mean);
    println!("Min close:  ${:.2}", min);
    println!("Max close:  ${:.2}", max);
    println!("Range:      ${:.2}", max - min);

    Ok(())
}

Run with:

cargo run --bin explore_dbn_data ES.FUT test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

---

Getting Help

Check Logs

# Enable debug logging
RUST_LOG=debug cargo test -p backtesting_service dbn_integration_tests

# Filter for DBN-specific logs
RUST_LOG=backtesting_service::dbn_data_source=debug cargo run

Run Test Suite

# All DBN tests
cargo test -p backtesting_service dbn_integration_tests

# Specific test
cargo test -p backtesting_service test_load_real_dbn_file

# With output
cargo test -p backtesting_service test_load_real_dbn_file -- --nocapture

File a Bug Report

Include:

1. Error message (full stack trace)
2. DBN file info (name, size, schema)
3. Code snippet (minimal reproduction)
4. System info (OS, Rust version)
5. Logs (with RUST_LOG=debug)

# Generate system info
rustc --version
cargo --version
uname -a

---

Last Updated: 2025-10-13 Version: 1.0