foxhunt/AGENT_M17_IMPLEMENTATION_REFERENCE.md

# Agent M17: MarketDataRepository Implementation Reference

## File Locations

### Main Implementations

```
foxhunt/
├── services/backtesting_service/src/
│   ├── repositories.rs                    # Trait definitions (18-45)
│   ├── repository_impl.rs                 # DataProviderMDR, BenzingaNR (24-286)
│   └── dbn_repository.rs                  # DbnMarketDataRepository (1,049 LOC)
│
├── services/trading_service/src/
│   ├── repositories.rs                    # Trait definitions
│   └── repository_impls.rs                # PostgresMarketDataRepository (567-750+)
│
└── services/ml_training_service/src/
    └── repository.rs                      # PostgresMlDataRepository
```

## MarketDataRepository Trait

**Definition Location**: `services/backtesting_service/src/repositories.rs` (lines 17-45)

```rust
#[async_trait]
pub trait MarketDataRepository: Send + Sync {
    async fn load_historical_data(
        &self,
        symbols: &[String],
        start_time: i64,          // nanoseconds
        end_time: i64,            // nanoseconds
    ) -> Result<Vec<crate::strategy_engine::MarketData>>;

    async fn check_data_availability(
        &self,
        symbols: &[String],
        start_time: i64,
        end_time: i64,
    ) -> Result<HashMap<String, bool>>;
}
```

## Implementation Details

### 1. DbnMarketDataRepository

**File**: `services/backtesting_service/src/dbn_repository.rs`

**Structure**:
- Lines 1-55: Module docs and struct definition
- Lines 56-569: Core implementation methods
- Lines 570-703: MarketDataRepository trait impl
- Lines 705-1048: Comprehensive tests (14 test functions)

**Key Methods**:
```rust
// Trait required
pub async fn load_historical_data(...)
pub async fn check_data_availability(...)

// Extra features
pub async fn load_by_time_range(start: DateTime, end: DateTime)
pub async fn load_with_volume_filter(min_volume: Decimal)
pub async fn load_regime_samples(regime_type: &str, count: usize)
pub async fn get_date_range(symbol: &str)
pub fn resample_bars(bars: &[MarketData], target_minutes: u32)
pub fn calculate_rolling_stats(bars: &[MarketData], window_size: usize)
pub fn generate_summary_stats(bars: &[MarketData])
```

**Data Source**:
```rust
pub struct DbnMarketDataRepository {
    data_source: Arc<DbnDataSource>,
    symbol_mappings: HashMap<String, String>,
}
```

**Test Coverage**:
- test_dbn_repository_creation
- test_check_data_availability
- test_load_by_time_range
- test_load_with_volume_filter
- test_load_regime_samples_trending
- test_load_regime_samples_ranging
- test_load_regime_samples_invalid
- test_get_date_range
- test_resample_bars
- test_calculate_rolling_stats
- test_generate_summary_stats
- test_empty_bars_edge_cases
- test_performance_target

### 2. DataProviderMarketDataRepository

**File**: `services/backtesting_service/src/repository_impl.rs` (lines 24-105)

**Structure**:
```rust
pub struct DataProviderMarketDataRepository {
    databento_provider: Arc<DatabentoHistoricalProvider>,
}

impl DataProviderMarketDataRepository {
    pub async fn new() -> Result<Self>
}
```

**Implementation Flow**:
1. Create DatabentoHistoricalProvider (async)
2. Convert timestamps: nanoseconds → DateTime
3. Create TimeRange for date range
4. Call provider.fetch() with OHLCV schema
5. Convert MarketDataEvent::Bar → MarketData
6. Sort by timestamp
7. Return to caller

### 3. PostgresMarketDataRepository

**File**: `services/trading_service/src/repository_impls.rs` (lines 567-750+)

**Purpose**: Real-time market data storage (NOT historical retrieval for backtesting)

**Structure**:
```rust
pub struct PostgresMarketDataRepository {
    pool: PgPool,
}

impl PostgresMarketDataRepository {
    pub fn new(pool: PgPool) -> Self
}
```

**Methods**:
```rust
// Store operations
pub async fn store_market_tick(tick: &MarketTick) -> Result<()>
pub async fn store_order_book(symbol: &str, order_book: &OrderBook) -> Result<()>
pub async fn store_market_event(event: &MarketDataEvent) -> Result<()>

// Query operations
pub async fn get_order_book(symbol: &str, depth: i32) -> Result<OrderBook>
pub async fn get_latest_prices(symbols: &[String]) -> Result<Vec<MarketTick>>
pub async fn get_historical_data(symbol: &str, from: i64, to: i64) -> Result<Vec<MarketTick>>
pub async fn get_order_book_level_count(symbol: &str, price: f64, side: OrderSide) -> Result<i32>
```

**Database Tables**:
- `market_ticks` - Individual ticks (symbol, price, quantity, side, timestamp)
- `order_book_levels` - Order book snapshots (symbol, side, price, quantity, timestamp)

## Factory Function

**Location**: `services/backtesting_service/src/repository_impl.rs` (lines 287-365)

```rust
pub async fn create_repositories(
    storage_manager: Arc<StorageManager>,
) -> Result<DefaultRepositories> {
    let use_dbn_data = std::env::var("USE_DBN_DATA")
        .ok()
        .and_then(|v| v.parse::<bool>().ok())
        .unwrap_or(false);

    let market_data: Box<dyn MarketDataRepository> = if use_dbn_data {
        // Parse DBN_SYMBOL_MAPPINGS
        // Parse DBN_SYMBOL_MAP
        Box::new(DbnMarketDataRepository::new_with_mappings(...).await?)
    } else {
        Box::new(DataProviderMarketDataRepository::new().await?)
    };

    let trading = Box::new(StorageManagerTradingRepository::new(storage_manager));
    let news = Box::new(BenzingaNewsRepository::new().await?);

    Ok(DefaultRepositories {
        market_data,
        trading,
        news,
    })
}
```

## Environment Variables

### For DBN-based Backtesting

```bash
USE_DBN_DATA=true
DBN_SYMBOL_MAPPINGS="ES.FUT:test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn,NQ.FUT:test_data/real/databento/NQ.FUT_ohlcv-1m_2024-01-02.dbn"
DBN_SYMBOL_MAP="BTC/USD:ES.FUT,ETH/USD:NQ.FUT"
```

### For Databento API

```bash
USE_DBN_DATA=false
# API key comes from Vault (config crate)
```

## Type Definitions

### MarketData (from strategy_engine)

```rust
pub struct MarketData {
    pub symbol: String,
    pub timestamp: DateTime<Utc>,
    pub open: Decimal,
    pub high: Decimal,
    pub low: Decimal,
    pub close: Decimal,
    pub volume: Decimal,
    pub timeframe: TimeFrame,
}

pub enum TimeFrame {
    Minute,
    FiveMinute,
    FifteenMinute,
    Hour,
    Daily,
    Weekly,
}
```

### MarketTick (from trading service)

```rust
pub struct MarketTick {
    pub symbol: String,
    pub price: f64,
    pub quantity: f64,
    pub timestamp: i64,              // Unix seconds
    pub side: Option<OrderSide>,
}
```

### OrderBook

```rust
pub struct OrderBook {
    pub symbol: String,
    pub bids: Vec<PriceLevel>,
    pub asks: Vec<PriceLevel>,
    pub timestamp: i64,
}

pub struct PriceLevel {
    pub price: f64,
    pub size: f64,
}
```

## Mock Implementations

### Location 1: repositories.rs (lines 190-212)
```rust
pub struct MockMarketDataRepository;

impl MarketDataRepository for MockMarketDataRepository {
    async fn load_historical_data(...) -> Result<Vec<MarketData>> {
        Ok(vec![])
    }
    
    async fn check_data_availability(...) -> Result<HashMap<String, bool>> {
        Ok(HashMap::new())
    }
}
```

### Location 2: mock_repositories.rs (test module)
```rust
pub struct MockMarketDataRepository {
    pub data: Arc<RwLock<Vec<MarketData>>>,
}

impl MockMarketDataRepository {
    pub fn new() -> Self
    pub fn with_data(data: Vec<MarketData>) -> Self
}
```

## Integration Points

### Data Crate Providers

```rust
use data::providers::databento::{DatabentoConfig, DatabentoHistoricalProvider};
use data::providers::traits::{HistoricalProvider, HistoricalSchema};
use data::types::TimeRange;
```

### Common Types

```rust
use common::MarketDataEvent;
use common::Symbol;
use common::OrderSide;
use common::OrderStatus;
use common::OrderType;
```

### Storage Integration

```rust
use crate::storage::StorageManager;
```

## Performance Targets

| Operation | Target | Implementation Status |
|---|---|---|
| Load 400 bars | <10ms | ✅ Verified in test_performance_target |
| Store tick | <1ms | ⚠️ Not measured |
| Query order book | <5ms | ⚠️ Not measured |
| Databento fetch | <100ms | ✅ By API SLA |

## Test Commands

```bash
# Test DbnMarketDataRepository
cargo test -p backtesting_service dbn_repository

# Test DataProviderMarketDataRepository
cargo test -p backtesting_service repository_impl

# Test PostgresMarketDataRepository
cargo test -p trading_service repository_impls

# Run all backtesting service tests
cargo test -p backtesting_service
```

## Key Files to Know

1. **Trait Definition**
   - `services/backtesting_service/src/repositories.rs` (lines 13-45)

2. **DbnMarketDataRepository**
   - `services/backtesting_service/src/dbn_repository.rs` (complete file)
   - Related: `services/backtesting_service/src/dbn_data_source.rs`

3. **Provider Implementations**
   - `services/backtesting_service/src/repository_impl.rs` (lines 24-105)
   - Related: `data/src/providers/databento/mod.rs`

4. **Trading Service Storage**
   - `services/trading_service/src/repository_impls.rs` (lines 567-750+)

5. **Factory & Dependency Injection**
   - `services/backtesting_service/src/repository_impl.rs` (lines 287-365)

---

**Last Updated**: 2025-10-18
**Agent**: M17
**Status**: ✅ Complete and Verified