foxhunt/docs/DBN_TROUBLESHOOTING.md

# DBN Troubleshooting Guide

**Version**: 1.0
**Last Updated**: 2025-10-13

---

## Table of Contents

1. [Common Errors](#common-errors)
2. [Data Quality Issues](#data-quality-issues)
3. [Performance Problems](#performance-problems)
4. [File Format Issues](#file-format-issues)
5. [Integration Issues](#integration-issues)
6. [Debugging Tools](#debugging-tools)

---

## Common Errors

### Error: "DBN file not found"

**Symptom**:
```
Error: DBN file not found: test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn
```

**Cause**: File path is incorrect or relative paths are not resolving correctly.

**Solution 1**: Use workspace-relative paths
```rust
fn get_test_file_path() -> String {
    let workspace_root = std::env::current_dir()
        .unwrap()
        .ancestors()
        .find(|p| p.join("Cargo.toml").exists() && p.join("test_data").exists())
        .expect("Could not find workspace root");

    workspace_root
        .join("test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn")
        .to_string_lossy()
        .to_string()
}
```

**Solution 2**: Use absolute paths
```rust
let file_path = "/home/jgrusewski/Work/foxhunt/test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn";
```

**Solution 3**: Verify file exists
```bash
ls -lh test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn
# Should output: -rw-rw-r-- 1 user user 95K Oct 12 23:22 ES.FUT_ohlcv-1m_2024-01-02.dbn
```

---

### Error: "No DBN file configured for symbol"

**Symptom**:
```
Error: No DBN file configured for symbol: UNKNOWN
```

**Cause**: Symbol not in file mapping.

**Solution**:
```rust
// Check available symbols first
let symbols = data_source.available_symbols();
println!("Available symbols: {:?}", symbols);

// Add missing symbol
data_source.add_symbol_mapping(
    "UNKNOWN".to_string(),
    "path/to/file.dbn".to_string()
);
```

---

### Error: "Invalid message length"

**Symptom**:
```
WARN Invalid message length: 0 at offset 1234
```

**Cause**: Corrupted DBN file or incorrect file format.

**Diagnosis**:
```bash
# Check file size (should be > 0)
ls -lh test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

# Check file is not empty
du -h test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

# Try to open with dbn CLI tool (if available)
dbn info test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn
```

**Solution**: Re-download or regenerate the DBN file.

---

### Error: "Failed to decode DBN record"

**Symptom**:
```
Error: Failed to decode DBN record at offset 5678
```

**Cause**: File format mismatch or version incompatibility.

**Solution**:
```rust
// Enable version upgrades
let mut decoder = DbnDecoder::from_file(file_path)?;
decoder.set_upgrade_policy(VersionUpgradePolicy::UpgradeToV3)?;
```

**Check DBN version compatibility**:
- Foxhunt uses `dbn` crate v0.9+
- Supports DBN format v1, v2, v3
- Automatic version upgrading enabled by default

---

## Data Quality Issues

### Issue: Unrealistic Prices

**Symptom**:
```
Bar 150: ES.FUT price 47.4275 outside realistic range (3000-6000)
```

**Cause**: Price encoding anomaly (7 decimal places instead of 9).

**Automatic Fix**: The system automatically detects and corrects these:

```rust
// Price anomaly detection (automatic in load_ohlcv_bars)
if pct_change > 0.5 && close_f64 < 1000.0 {
    let corrected_close = close_f64 * 100.0;
    if corrected_close >= 3000.0 && corrected_close <= 6000.0 {
        // Apply 100x correction
        open_f64 *= 100.0;
        high_f64 *= 100.0;
        low_f64 *= 100.0;
        close_f64 = corrected_close;
    }
}
```

**Verification**:
```rust
// Check correction logs
RUST_LOG=debug cargo test -p backtesting_service test_load_real_dbn_file

// Look for lines like:
// DEBUG Applied 100x price correction at bar 150 (99% change, $47.43 -> $4742.75)
```

**Manual Validation**:
```rust
async fn validate_prices() -> Result<()> {
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;

    for (i, bar) in bars.iter().enumerate() {
        let close_f64 = bar.close.to_string().parse::<f64>().unwrap();

        if close_f64 < 3000.0 || close_f64 > 6000.0 {
            eprintln!("WARN: Bar {} has unusual price: ${:.2}", i, close_f64);
        }
    }

    Ok(())
}
```

---

### Issue: OHLCV Relationship Violations

**Symptom**:
```
Quality issue at bar 42: high < low
```

**Diagnosis**:
```rust
fn diagnose_ohlcv_issues(bars: &[MarketData]) {
    for (i, bar) in bars.iter().enumerate() {
        if bar.high < bar.low {
            eprintln!("Bar {}: high ({}) < low ({})", i, bar.high, bar.low);
        }
        if bar.high < bar.open || bar.high < bar.close {
            eprintln!("Bar {}: high not highest value", i);
        }
        if bar.low > bar.open || bar.low > bar.close {
            eprintln!("Bar {}: low not lowest value", i);
        }
    }
}
```

**Solution**: This indicates data corruption. Options:
1. Re-download the DBN file
2. Skip corrupted bars
3. Contact data provider

```rust
// Skip corrupted bars
let valid_bars: Vec<_> = bars.into_iter()
    .filter(|bar| {
        bar.high >= bar.low &&
        bar.high >= bar.open &&
        bar.high >= bar.close &&
        bar.low <= bar.open &&
        bar.low <= bar.close
    })
    .collect();
```

---

### Issue: Missing or Sparse Data

**Symptom**:
```
Expected ~390 bars, got 150
```

**Diagnosis**:
```rust
async fn analyze_data_density() -> Result<()> {
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;

    if bars.len() < 350 {
        eprintln!("WARN: Expected ~390 bars (1-minute, full trading day), got {}", bars.len());
    }

    // Check for gaps
    for i in 1..bars.len() {
        let gap = bars[i].timestamp - bars[i-1].timestamp;
        if gap > chrono::Duration::minutes(5) {
            eprintln!("Data gap detected at bar {}: {} minute gap", i, gap.num_minutes());
        }
    }

    Ok(())
}
```

**Common Causes**:
1. Partial day data (market opened late, closed early)
2. Data feed interruption
3. Filter applied during download

**Solution**: Verify data source coverage and re-download if needed.

---

### Issue: Timestamp Anomalies

**Symptom**:
```
Bar 200: Timestamp 2024-01-02 15:30:00 (jumped backwards)
```

**Diagnosis**:
```rust
fn check_timestamp_ordering(bars: &[MarketData]) {
    for i in 1..bars.len() {
        if bars[i].timestamp < bars[i-1].timestamp {
            eprintln!(
                "Timestamp ordering issue at bar {}: {} < {}",
                i,
                bars[i].timestamp,
                bars[i-1].timestamp
            );
        }
    }
}
```

**Solution**: Sort bars after loading (already done by `load_ohlcv_bars`):
```rust
// Automatic sorting
bars.sort_by(|a, b| a.timestamp.cmp(&b.timestamp));
```

---

## Performance Problems

### Issue: Slow Loading (>100ms for ~400 bars)

**Target**: <10ms for ~400 bars
**Actual**: >100ms

**Diagnosis**:
```rust
use std::time::Instant;

let start = Instant::now();
let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
let duration = start.elapsed();

println!("Loaded {} bars in {:?} ({:.2}ms)",
    bars.len(),
    duration,
    duration.as_secs_f64() * 1000.0
);

if duration.as_millis() > 100 {
    eprintln!("WARN: Performance target missed");
}
```

**Common Causes & Solutions**:

#### 1. Cold File System Cache

**Solution**: Warm up cache first
```rust
// Warm-up run
let _ = data_source.load_ohlcv_bars("ES.FUT").await?;

// Timed run (will be faster)
let start = Instant::now();
let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
let duration = start.elapsed();
```

#### 2. Disk I/O Bottleneck

**Check disk performance**:
```bash
# Test read speed
dd if=test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn of=/dev/null bs=1M

# Should be >100 MB/s on modern SSD
```

**Solution**: Use SSD storage, not HDD

#### 3. Debug Logging Overhead

**Disable verbose logging**:
```bash
# ❌ Slow (debug logging)
RUST_LOG=debug cargo run

# ✅ Fast (info or warn only)
RUST_LOG=info cargo run
```

#### 4. Memory Allocations

**Pre-allocate vectors**:
```rust
// ✅ Good (pre-allocated)
let mut bars = Vec::with_capacity(400);

// ❌ Slow (reallocates multiple times)
let mut bars = Vec::new();
```

---

### Issue: High Memory Usage

**Symptom**: Process uses >1GB RAM for loading 10 files.

**Diagnosis**:
```rust
use sysinfo::{System, SystemExt, ProcessExt};

let mut sys = System::new_all();
sys.refresh_all();

if let Some(process) = sys.process(sysinfo::get_current_pid().unwrap()) {
    println!("Memory usage: {} MB", process.memory() / 1024);
}
```

**Solutions**:

#### 1. Disable Caching

```rust
let data_source = DbnDataSource::new(file_mapping)
    .await?
    .with_cache_limit(0); // No caching
```

#### 2. Stream Processing

```rust
// Process files one at a time
for symbol in symbols {
    let bars = data_source.load_ohlcv_bars(&symbol).await?;
    process_bars(&bars);
    // Bars dropped here, memory freed
}
```

#### 3. Limit LRU Cache Size

```rust
let data_source = DbnDataSource::new(file_mapping)
    .await?
    .with_cache_limit(3); // Only cache last 3 symbols
```

---

## File Format Issues

### Issue: Unknown File Format

**Symptom**:
```
Error: Failed to create DBN decoder for file: unknown format
```

**Verify file format**:
```bash
# Check file magic bytes
xxd -l 16 test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

# Should start with DBN magic bytes (0x64 0x62 0x6e)
```

**Verify DBN schema**:
```bash
# If you have dbn CLI tool
dbn info test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

# Expected output:
# Schema: OHLCV-1m
# Compression: none
# Records: 390
```

---

### Issue: Unsupported Schema

**Symptom**:
```
WARN Unknown message type: 0x52
```

**Supported Schemas**:
- ✅ OHLCV-1m (1-minute bars) - **Fully supported**
- ✅ Trade messages - Supported via DbnParser
- ✅ Quote messages - Supported via DbnParser
- ❌ MBO (Market By Order) - Not yet implemented
- ❌ MBP (Market By Price) - Not yet implemented

**Workaround**: Convert unsupported schema to OHLCV using databento tools:
```bash
# Convert trades to OHLCV-1m
dbn convert --schema ohlcv-1m --interval 1m input.dbn output.dbn
```

---

## Integration Issues

### Issue: Repository Interface Not Working

**Symptom**:
```
Error: the trait bound `DbnMarketDataRepository: MarketDataRepository` is not satisfied
```

**Solution**: Import the trait
```rust
use backtesting_service::repositories::MarketDataRepository;

// Now trait methods available
let data = repo.load_historical_data(&symbols, start_time, end_time).await?;
```

---

### Issue: Timestamp Format Mismatch

**Symptom**:
```
Error: Invalid timestamp: expected nanoseconds, got seconds
```

**DBN uses nanoseconds** (i64):
```rust
// ✅ Correct (nanoseconds since Unix epoch)
let start_time = 1704153600_000_000_000i64; // 2024-01-02 00:00:00

// ❌ Wrong (seconds)
let start_time = 1704153600i64;
```

**Convert DateTime to nanoseconds**:
```rust
use chrono::{TimeZone, Utc};

let dt = Utc.with_ymd_and_hms(2024, 1, 2, 0, 0, 0).unwrap();
let nanos = dt.timestamp_nanos_opt().unwrap();
```

---

### Issue: Symbol Mapping Not Working

**Symptom**:
```
Loaded 0 bars for BTC/USD (expected ES.FUT data)
```

**Diagnosis**:
```rust
// Check symbol mappings
let repo = DbnMarketDataRepository::new_with_mappings(
    file_mapping,
    symbol_mappings
).await?;

// Add debug logging
RUST_LOG=debug cargo run
// Look for: "Symbol mapping: BTC/USD -> ES.FUT"
```

**Solution**: Verify mapping configuration
```rust
let mut symbol_mappings = HashMap::new();
symbol_mappings.insert("BTC/USD".to_string(), "ES.FUT".to_string());
symbol_mappings.insert("ETH/USD".to_string(), "ES.FUT".to_string());

let repo = DbnMarketDataRepository::new_with_mappings(
    file_mapping,
    symbol_mappings // ← Make sure this is passed!
).await?;
```

---

## Debugging Tools

### Tool 1: Data Validation Script

```rust
// Save as: scripts/validate_dbn_data.rs

use backtesting_service::dbn_data_source::DbnDataSource;
use std::collections::HashMap;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let mut file_mapping = HashMap::new();
    file_mapping.insert(
        "ES.FUT".to_string(),
        "test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn".to_string()
    );

    let data_source = DbnDataSource::new(file_mapping).await?;
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;

    println!("=== DBN Data Validation Report ===");
    println!("File: ES.FUT_ohlcv-1m_2024-01-02.dbn");
    println!("Bars loaded: {}", bars.len());

    // Check expected count
    if bars.len() < 350 || bars.len() > 450 {
        eprintln!("⚠️  WARN: Expected ~390 bars, got {}", bars.len());
    } else {
        println!("✅ Bar count in expected range (350-450)");
    }

    // Validate OHLCV relationships
    let mut quality_issues = 0;
    for (i, bar) in bars.iter().enumerate() {
        if !(bar.high >= bar.low &&
             bar.high >= bar.open &&
             bar.high >= bar.close &&
             bar.low <= bar.open &&
             bar.low <= bar.close) {
            quality_issues += 1;
            if quality_issues <= 3 {
                eprintln!("⚠️  Bar {} OHLCV violation", i);
            }
        }
    }

    if quality_issues == 0 {
        println!("✅ All bars passed OHLCV validation");
    } else {
        eprintln!("❌ {} bars failed OHLCV validation", quality_issues);
    }

    // Check price ranges
    let mut price_issues = 0;
    for (i, bar) in bars.iter().enumerate() {
        let close_f64 = bar.close.to_string().parse::<f64>().unwrap();
        if close_f64 < 3000.0 || close_f64 > 6000.0 {
            price_issues += 1;
            if price_issues <= 3 {
                eprintln!("⚠️  Bar {} unusual price: ${:.2}", i, close_f64);
            }
        }
    }

    if price_issues == 0 {
        println!("✅ All prices in realistic range (3000-6000)");
    } else {
        eprintln!("❌ {} bars with unusual prices", price_issues);
    }

    // Check timestamp ordering
    let mut timestamp_issues = 0;
    for i in 1..bars.len() {
        if bars[i].timestamp < bars[i-1].timestamp {
            timestamp_issues += 1;
        }
    }

    if timestamp_issues == 0 {
        println!("✅ Timestamps properly ordered");
    } else {
        eprintln!("❌ {} timestamp ordering issues", timestamp_issues);
    }

    println!("\n=== Summary ===");
    println!("First bar: {} @ {}", bars[0].close, bars[0].timestamp);
    println!("Last bar:  {} @ {}", bars[bars.len()-1].close, bars[bars.len()-1].timestamp);

    Ok(())
}
```

Run with:
```bash
cargo run --bin validate_dbn_data
```

---

### Tool 2: Performance Profiler

```rust
// Save as: scripts/profile_dbn_loading.rs

use backtesting_service::dbn_data_source::DbnDataSource;
use std::collections::HashMap;
use std::time::Instant;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    println!("=== DBN Loading Performance Profile ===\n");

    let mut file_mapping = HashMap::new();
    file_mapping.insert(
        "ES.FUT".to_string(),
        "test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn".to_string()
    );

    // Test 1: Cold load
    println!("Test 1: Cold load (first time)");
    let data_source = DbnDataSource::new(file_mapping.clone()).await?;
    let start = Instant::now();
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
    let cold_duration = start.elapsed();
    println!("  Duration: {:?} ({:.2}ms)", cold_duration, cold_duration.as_secs_f64() * 1000.0);
    println!("  Bars: {}", bars.len());

    // Test 2: Warm load
    println!("\nTest 2: Warm load (cached in OS)");
    let start = Instant::now();
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
    let warm_duration = start.elapsed();
    println!("  Duration: {:?} ({:.2}ms)", warm_duration, warm_duration.as_secs_f64() * 1000.0);
    println!("  Bars: {}", bars.len());

    // Test 3: With caching enabled
    println!("\nTest 3: With LRU cache (immediate)");
    let data_source = DbnDataSource::new(file_mapping.clone())
        .await?
        .with_cache_limit(10);

    // Prime cache
    let _ = data_source.load_ohlcv_bars("ES.FUT").await?;

    let start = Instant::now();
    let bars = data_source.load_ohlcv_bars("ES.FUT").await?;
    let cached_duration = start.elapsed();
    println!("  Duration: {:?} ({:.2}ms)", cached_duration, cached_duration.as_secs_f64() * 1000.0);
    println!("  Bars: {}", bars.len());

    // Performance summary
    println!("\n=== Performance Summary ===");
    println!("Target: <10ms");
    println!("Cold load:   {:.2}ms {}",
        cold_duration.as_secs_f64() * 1000.0,
        if cold_duration.as_millis() < 10 { "✅" } else { "⚠️" }
    );
    println!("Warm load:   {:.2}ms {}",
        warm_duration.as_secs_f64() * 1000.0,
        if warm_duration.as_millis() < 10 { "✅" } else { "⚠️" }
    );
    println!("Cached load: {:.2}ms ✅",
        cached_duration.as_secs_f64() * 1000.0
    );

    // Throughput
    let bars_per_sec = (bars.len() as f64 / warm_duration.as_secs_f64()) as u64;
    println!("\nThroughput: {} bars/sec", bars_per_sec);

    Ok(())
}
```

Run with:
```bash
cargo run --bin profile_dbn_loading
```

---

### Tool 3: Data Explorer

```rust
// Save as: scripts/explore_dbn_data.rs

use backtesting_service::dbn_data_source::DbnDataSource;
use std::collections::HashMap;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let symbol = std::env::args().nth(1).unwrap_or_else(|| "ES.FUT".to_string());
    let file_path = std::env::args().nth(2).unwrap_or_else(|| {
        "test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn".to_string()
    });

    println!("=== DBN Data Explorer ===");
    println!("Symbol: {}", symbol);
    println!("File: {}\n", file_path);

    let mut file_mapping = HashMap::new();
    file_mapping.insert(symbol.clone(), file_path);

    let data_source = DbnDataSource::new(file_mapping).await?;
    let bars = data_source.load_ohlcv_bars(&symbol).await?;

    println!("Total bars: {}", bars.len());
    println!("Time range: {} to {}", bars[0].timestamp, bars[bars.len()-1].timestamp);

    // Sample bars
    println!("\n=== First 5 bars ===");
    for (i, bar) in bars.iter().take(5).enumerate() {
        println!("[{:03}] {} @ {} | O:{} H:{} L:{} C:{} V:{}",
            i,
            bar.symbol,
            bar.timestamp.format("%Y-%m-%d %H:%M:%S"),
            bar.open,
            bar.high,
            bar.low,
            bar.close,
            bar.volume
        );
    }

    println!("\n=== Last 5 bars ===");
    for (i, bar) in bars.iter().rev().take(5).rev().enumerate() {
        let idx = bars.len() - 5 + i;
        println!("[{:03}] {} @ {} | O:{} H:{} L:{} C:{} V:{}",
            idx,
            bar.symbol,
            bar.timestamp.format("%Y-%m-%d %H:%M:%S"),
            bar.open,
            bar.high,
            bar.low,
            bar.close,
            bar.volume
        );
    }

    // Statistics
    let closes: Vec<f64> = bars.iter()
        .map(|b| b.close.to_string().parse().unwrap())
        .collect();

    let mean = closes.iter().sum::<f64>() / closes.len() as f64;
    let min = closes.iter().cloned().fold(f64::INFINITY, f64::min);
    let max = closes.iter().cloned().fold(f64::NEG_INFINITY, f64::max);

    println!("\n=== Statistics ===");
    println!("Mean close: ${:.2}", mean);
    println!("Min close:  ${:.2}", min);
    println!("Max close:  ${:.2}", max);
    println!("Range:      ${:.2}", max - min);

    Ok(())
}
```

Run with:
```bash
cargo run --bin explore_dbn_data ES.FUT test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn
```

---

## Getting Help

### Check Logs

```bash
# Enable debug logging
RUST_LOG=debug cargo test -p backtesting_service dbn_integration_tests

# Filter for DBN-specific logs
RUST_LOG=backtesting_service::dbn_data_source=debug cargo run
```

### Run Test Suite

```bash
# All DBN tests
cargo test -p backtesting_service dbn_integration_tests

# Specific test
cargo test -p backtesting_service test_load_real_dbn_file

# With output
cargo test -p backtesting_service test_load_real_dbn_file -- --nocapture
```

### File a Bug Report

Include:
1. Error message (full stack trace)
2. DBN file info (name, size, schema)
3. Code snippet (minimal reproduction)
4. System info (OS, Rust version)
5. Logs (with RUST_LOG=debug)

```bash
# Generate system info
rustc --version
cargo --version
uname -a
```

---

**Last Updated**: 2025-10-13
**Version**: 1.0