foxhunt/AGENT_M17_IMPLEMENTATION_REFERENCE.md

Agent M17: MarketDataRepository Implementation Reference

File Locations

Main Implementations

foxhunt/
├── services/backtesting_service/src/
│   ├── repositories.rs                    # Trait definitions (18-45)
│   ├── repository_impl.rs                 # DataProviderMDR, BenzingaNR (24-286)
│   └── dbn_repository.rs                  # DbnMarketDataRepository (1,049 LOC)
│
├── services/trading_service/src/
│   ├── repositories.rs                    # Trait definitions
│   └── repository_impls.rs                # PostgresMarketDataRepository (567-750+)
│
└── services/ml_training_service/src/
    └── repository.rs                      # PostgresMlDataRepository

MarketDataRepository Trait

Definition Location: services/backtesting_service/src/repositories.rs (lines 17-45)

#[async_trait]
pub trait MarketDataRepository: Send + Sync {
    async fn load_historical_data(
        &self,
        symbols: &[String],
        start_time: i64,          // nanoseconds
        end_time: i64,            // nanoseconds
    ) -> Result<Vec<crate::strategy_engine::MarketData>>;

    async fn check_data_availability(
        &self,
        symbols: &[String],
        start_time: i64,
        end_time: i64,
    ) -> Result<HashMap<String, bool>>;
}

Implementation Details

1. DbnMarketDataRepository

File: services/backtesting_service/src/dbn_repository.rs

Structure:

• Lines 1-55: Module docs and struct definition
• Lines 56-569: Core implementation methods
• Lines 570-703: MarketDataRepository trait impl
• Lines 705-1048: Comprehensive tests (14 test functions)

Key Methods:

// Trait required
pub async fn load_historical_data(...)
pub async fn check_data_availability(...)

// Extra features
pub async fn load_by_time_range(start: DateTime, end: DateTime)
pub async fn load_with_volume_filter(min_volume: Decimal)
pub async fn load_regime_samples(regime_type: &str, count: usize)
pub async fn get_date_range(symbol: &str)
pub fn resample_bars(bars: &[MarketData], target_minutes: u32)
pub fn calculate_rolling_stats(bars: &[MarketData], window_size: usize)
pub fn generate_summary_stats(bars: &[MarketData])

Data Source:

pub struct DbnMarketDataRepository {
    data_source: Arc<DbnDataSource>,
    symbol_mappings: HashMap<String, String>,
}

Test Coverage:

• test_dbn_repository_creation
• test_check_data_availability
• test_load_by_time_range
• test_load_with_volume_filter
• test_load_regime_samples_trending
• test_load_regime_samples_ranging
• test_load_regime_samples_invalid
• test_get_date_range
• test_resample_bars
• test_calculate_rolling_stats
• test_generate_summary_stats
• test_empty_bars_edge_cases
• test_performance_target

2. DataProviderMarketDataRepository

File: services/backtesting_service/src/repository_impl.rs (lines 24-105)

Structure:

pub struct DataProviderMarketDataRepository {
    databento_provider: Arc<DatabentoHistoricalProvider>,
}

impl DataProviderMarketDataRepository {
    pub async fn new() -> Result<Self>
}

Implementation Flow:

1. Create DatabentoHistoricalProvider (async)
2. Convert timestamps: nanoseconds → DateTime
3. Create TimeRange for date range
4. Call provider.fetch() with OHLCV schema
5. Convert MarketDataEvent::Bar → MarketData
6. Sort by timestamp
7. Return to caller

3. PostgresMarketDataRepository

File: services/trading_service/src/repository_impls.rs (lines 567-750+)

Purpose: Real-time market data storage (NOT historical retrieval for backtesting)

Structure:

pub struct PostgresMarketDataRepository {
    pool: PgPool,
}

impl PostgresMarketDataRepository {
    pub fn new(pool: PgPool) -> Self
}

Methods:

// Store operations
pub async fn store_market_tick(tick: &MarketTick) -> Result<()>
pub async fn store_order_book(symbol: &str, order_book: &OrderBook) -> Result<()>
pub async fn store_market_event(event: &MarketDataEvent) -> Result<()>

// Query operations
pub async fn get_order_book(symbol: &str, depth: i32) -> Result<OrderBook>
pub async fn get_latest_prices(symbols: &[String]) -> Result<Vec<MarketTick>>
pub async fn get_historical_data(symbol: &str, from: i64, to: i64) -> Result<Vec<MarketTick>>
pub async fn get_order_book_level_count(symbol: &str, price: f64, side: OrderSide) -> Result<i32>

Database Tables:

• market_ticks - Individual ticks (symbol, price, quantity, side, timestamp)
• order_book_levels - Order book snapshots (symbol, side, price, quantity, timestamp)

Factory Function

Location: services/backtesting_service/src/repository_impl.rs (lines 287-365)

pub async fn create_repositories(
    storage_manager: Arc<StorageManager>,
) -> Result<DefaultRepositories> {
    let use_dbn_data = std::env::var("USE_DBN_DATA")
        .ok()
        .and_then(|v| v.parse::<bool>().ok())
        .unwrap_or(false);

    let market_data: Box<dyn MarketDataRepository> = if use_dbn_data {
        // Parse DBN_SYMBOL_MAPPINGS
        // Parse DBN_SYMBOL_MAP
        Box::new(DbnMarketDataRepository::new_with_mappings(...).await?)
    } else {
        Box::new(DataProviderMarketDataRepository::new().await?)
    };

    let trading = Box::new(StorageManagerTradingRepository::new(storage_manager));
    let news = Box::new(BenzingaNewsRepository::new().await?);

    Ok(DefaultRepositories {
        market_data,
        trading,
        news,
    })
}

Environment Variables

For DBN-based Backtesting

USE_DBN_DATA=true
DBN_SYMBOL_MAPPINGS="ES.FUT:test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn,NQ.FUT:test_data/real/databento/NQ.FUT_ohlcv-1m_2024-01-02.dbn"
DBN_SYMBOL_MAP="BTC/USD:ES.FUT,ETH/USD:NQ.FUT"

For Databento API

USE_DBN_DATA=false
# API key comes from Vault (config crate)

Type Definitions

MarketData (from strategy_engine)

pub struct MarketData {
    pub symbol: String,
    pub timestamp: DateTime<Utc>,
    pub open: Decimal,
    pub high: Decimal,
    pub low: Decimal,
    pub close: Decimal,
    pub volume: Decimal,
    pub timeframe: TimeFrame,
}

pub enum TimeFrame {
    Minute,
    FiveMinute,
    FifteenMinute,
    Hour,
    Daily,
    Weekly,
}

MarketTick (from trading service)

pub struct MarketTick {
    pub symbol: String,
    pub price: f64,
    pub quantity: f64,
    pub timestamp: i64,              // Unix seconds
    pub side: Option<OrderSide>,
}

OrderBook

pub struct OrderBook {
    pub symbol: String,
    pub bids: Vec<PriceLevel>,
    pub asks: Vec<PriceLevel>,
    pub timestamp: i64,
}

pub struct PriceLevel {
    pub price: f64,
    pub size: f64,
}

Mock Implementations

Location 1: repositories.rs (lines 190-212)

pub struct MockMarketDataRepository;

impl MarketDataRepository for MockMarketDataRepository {
    async fn load_historical_data(...) -> Result<Vec<MarketData>> {
        Ok(vec![])
    }
    
    async fn check_data_availability(...) -> Result<HashMap<String, bool>> {
        Ok(HashMap::new())
    }
}

Location 2: mock_repositories.rs (test module)

pub struct MockMarketDataRepository {
    pub data: Arc<RwLock<Vec<MarketData>>>,
}

impl MockMarketDataRepository {
    pub fn new() -> Self
    pub fn with_data(data: Vec<MarketData>) -> Self
}

Integration Points

Data Crate Providers

use data::providers::databento::{DatabentoConfig, DatabentoHistoricalProvider};
use data::providers::traits::{HistoricalProvider, HistoricalSchema};
use data::types::TimeRange;

Common Types

use common::MarketDataEvent;
use common::Symbol;
use common::OrderSide;
use common::OrderStatus;
use common::OrderType;

Storage Integration

use crate::storage::StorageManager;

Performance Targets

Operation	Target	Implementation Status
Load 400 bars	<10ms	✅ Verified in test_performance_target
Store tick	<1ms	⚠️ Not measured
Query order book	<5ms	⚠️ Not measured
Databento fetch	<100ms	✅ By API SLA

Test Commands

# Test DbnMarketDataRepository
cargo test -p backtesting_service dbn_repository

# Test DataProviderMarketDataRepository
cargo test -p backtesting_service repository_impl

# Test PostgresMarketDataRepository
cargo test -p trading_service repository_impls

# Run all backtesting service tests
cargo test -p backtesting_service

Key Files to Know

1. Trait Definition

   • services/backtesting_service/src/repositories.rs (lines 13-45)

2. DbnMarketDataRepository

   • services/backtesting_service/src/dbn_repository.rs (complete file)
   • Related: services/backtesting_service/src/dbn_data_source.rs

3. Provider Implementations

   • services/backtesting_service/src/repository_impl.rs (lines 24-105)
   • Related: data/src/providers/databento/mod.rs

4. Trading Service Storage

   • services/trading_service/src/repository_impls.rs (lines 567-750+)

5. Factory & Dependency Injection

   • services/backtesting_service/src/repository_impl.rs (lines 287-365)

---

Last Updated: 2025-10-18 Agent: M17 Status: ✅ Complete and Verified