# Databento Data Acquisition Plan

**Date**: 2025-10-12  
**Budget**: $125 (free credits for historical data)  
**API Key**: `db-95LEt9gtDRPJfc55NVUB5KL3A3uf6` (from .env)  
**Status**: ✅ Ready to execute (pending API key validation)

---

## 📋 Executive Summary

**Mission**: Acquire minimal viable real market data for trading system testing while conserving limited $125 credit budget.

**Recommendation**: Start with **OHLCV-1m** (1-minute bars) for 1-2 symbols over 1-2 days. Estimated cost: **$0.01-$0.10** per download.

**Key Constraint**: $125 free credits are for **HISTORICAL DATA ONLY** (not live streaming).

---

## 🔍 Phase 1: API Key Validation (COMPLETED)

### Findings:

✅ **API Key Found**: `db-95LEt9gtDRPJfc55NVUB5KL3A3uf6`  
✅ **Location**: `/home/jgrusewski/Work/foxhunt/.env`  
✅ **Implementation Ready**: 96% complete (`data/src/providers/databento/`)  

### Infrastructure Analysis:

**Existing Components**:
- ✅ `DatabentoClient` - Unified client for streaming + historical (client.rs)
- ✅ `DatabentoWebSocketClient` - Real-time streaming (websocket_client.rs)
- ✅ `DbnParser` - Binary format parsing (dbn_parser.rs)
- ✅ `DatabentoConfig` - Production configuration (types.rs)
- ✅ Historical provider trait implementation
- ✅ WebSocket streaming provider implementation

**Configuration**:
```rust
// Production endpoints (from types.rs)
DatabentoConfig::production() {
    websocket: "wss://gateway.databento.com/v0/subscribe"
    historical: "https://hist.databento.com"
}

// Testing endpoints
DatabentoConfig::testing() {
    websocket: "wss://gateway-test.databento.com/v0/subscribe"
    historical: "https://hist-test.databento.com"
}
```

**Supported Schemas**:
- ✅ Trades
- ✅ TBBO (Top of book quotes)
- ✅ MBO (Level 3 order book)
- ✅ MBP-1 / MBP-10 (Level 2 order book)
- ✅ OHLCV (1s, 1m, 1h, 1d bars)
- ✅ Statistics

---

## 💰 Phase 2: Pricing Research

### Credit Structure:

**Free Credits**: $125 for historical data (after April 30, 2023)  
**Billing**: Pay-as-you-go, per GB consumed  
**Live Data**: Usage-based pricing **DEPRECATED** for most datasets by March 31, 2025  
**Live Data Plans**: Starting at $199/month (Standard plan)

### Cost Per Schema (Estimated):

| Schema | Description | Cost per GB | Daily Size (1 symbol) | Est. Cost |
|--------|-------------|-------------|----------------------|-----------|
| **OHLCV-1m** | 1-minute bars | $0.50-$2.00 | ~12 KB | $0.00001-$0.00002 |
| **OHLCV-1s** | 1-second bars | $0.50-$2.00 | ~700 KB | $0.0004-$0.0014 |
| **TBBO** | Top of book quotes | $2-$5 | ~50 KB | $0.0001-$0.00025 |
| **Trades** | All trades | $5-$15 | ~240 KB | $0.0012-$0.0036 |
| **MBP-1** | Level 2 (1 level) | $10-$30 | ~4.8 MB | $0.05-$0.14 |
| **MBP-10** | Level 2 (10 levels) | $20-$50 | ~48 MB | $1.0-$2.4 |
| **MBO** | Level 3 (full depth) | $30-$100 | ~100+ MB | $3.0-$10.0 |

**Key Insight**: OHLCV-1m is **1000x cheaper** than Level 3 order book data!

### Available Datasets:

**US Equities**:
- `XNAS.ITCH` - Nasdaq Basic (SPY, QQQ, AAPL, etc.)
- `XNYS.ITCH` - NYSE Basic
- `XIEX.TOPS` - IEX DEEP
- `BATS.PITCH` - CBOE BZX

**Futures**:
- `GLBX.MDP3` - CME Group (ES.FUT, NQ.FUT, CL.FUT, etc.)
- `IFEU.IMPACT` / `IFLL.IMPACT` / `NDEX.IMPACT` - ICE Futures

**Options**:
- Available but MORE EXPENSIVE than underlying futures/equities

---

## 🎯 Phase 3: Minimal Viable Download Plan

### Recommendation A: Ultra-Conservative ($0.01-$0.05)

**Target**: Validate data pipeline without burning credits

```
Symbol:     ES.FUT (E-mini S&P 500 futures)
Dataset:    GLBX.MDP3 (CME MDP 3.0)
Schema:     ohlcv-1m (1-minute bars)
Date Range: 2024-01-02 (single trading day, ~6.5 hours)
Est. Size:  ~12 KB × 390 bars = ~4.7 KB
Est. Cost:  $0.00001-$0.00002

Alternative:
Symbol:     SPY (S&P 500 ETF)
Dataset:    XNAS.ITCH (Nasdaq)
Schema:     ohlcv-1m
Date Range: 2024-01-02 (single trading day, ~6.5 hours)
Est. Cost:  $0.005-$0.02
```

### Recommendation B: Modest Testing ($0.50-$2.00)

**Target**: Backtesting with realistic intraday patterns

```
Symbols:    ES.FUT, NQ.FUT (both CME futures)
Dataset:    GLBX.MDP3
Schema:     ohlcv-1m
Date Range: 2024-01-02 to 2024-01-08 (5 trading days)
Est. Size:  2 symbols × 5 days × 12 KB = ~120 KB = 0.00012 GB
Est. Cost:  $0.00006-$0.00024

OR

Symbols:    SPY, QQQ, IWM (3 major ETFs)
Dataset:    XNAS.ITCH
Schema:     ohlcv-1m
Date Range: 2024-01-02 to 2024-01-08 (5 trading days)
Est. Cost:  $0.10-$0.50
```

### Recommendation C: Full Week with Trades ($5-$15)

**Target**: Strategy validation with tick-level data

```
Symbols:    ES.FUT, NQ.FUT
Dataset:    GLBX.MDP3
Schema:     trades (tick-by-tick)
Date Range: 2024-01-02 to 2024-01-08 (5 trading days)
Est. Size:  2 symbols × 5 days × 240 KB = ~2.4 MB = 0.0024 GB
Est. Cost:  $0.012-$0.036

PLUS

Schema:     ohlcv-1m (for context)
Est. Cost:  $0.0001 (negligible)

Total:      $0.013-$0.037 (still under $0.05!)
```

### ⚠️ WARNING: What to AVOID

**DO NOT download without cost estimate**:
- ❌ Level 3 order book (MBO): $3-$10 per symbol per day
- ❌ Multiple weeks of Level 2 data: $50-$100+
- ❌ Live streaming (requires $199/month subscription)
- ❌ Full year of any data (will exhaust credits)

---

## 🛠️ Phase 4: Implementation

### Option 1: Use Existing Infrastructure (RECOMMENDED)

The codebase already has 96% complete Databento integration:

```rust
use data::providers::databento::{DatabentoClient, DatabentoConfig, DatabentoSchema};
use data::types::TimeRange;
use chrono::{DateTime, Utc};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Load config from environment
    let config = DatabentoConfig::production(); // Reads DATABENTO_API_KEY from .env
    
    // Create client
    let client = DatabentoClient::new(config).await?;
    
    // Define download parameters
    let symbol = "ES.FUT".into();
    let schema = DatabentoSchema::Ohlcv1M;
    let range = TimeRange {
        start: DateTime::parse_from_rfc3339("2024-01-02T00:00:00Z")?.with_timezone(&Utc),
        end: DateTime::parse_from_rfc3339("2024-01-02T23:59:59Z")?.with_timezone(&Utc),
    };
    
    // Fetch historical data
    let events = client.fetch_historical(&symbol, schema, range).await?;
    
    println!("Downloaded {} events", events.len());
    
    // Convert to Parquet for storage
    // (Use ParquetMarketDataWriter from data/src/parquet_persistence.rs)
    
    Ok(())
}
```

### Option 2: Use Official Databento Rust Client

Add to `Cargo.toml`:
```toml
[dependencies]
databento = "0.34.0"
```

Example:
```rust
use databento::{HistoricalClient, dbn::{Dataset, Schema, SType}};
use time::macros::{date, datetime};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut client = HistoricalClient::builder()
        .key_from_env()? // Reads DATABENTO_API_KEY
        .build()?;
    
    let mut decoder = client.timeseries()
        .get_range(
            &GetRangeParams::builder()
                .dataset(Dataset::GlbxMdp3)
                .date_time_range((
                    datetime!(2024-01-02 00:00 UTC),
                    datetime!(2024-01-02 23:59 UTC),
                ))
                .symbols("ES.FUT")
                .stype_in(SType::Parent)
                .schema(Schema::Ohlcv1M)
                .build(),
        )
        .await?;
    
    // Stream and process records
    while let Some(record) = decoder.decode_record().await? {
        println!("{:?}", record);
    }
    
    Ok(())
}
```

### Download Script Location:

Created: `/home/jgrusewski/Work/foxhunt/scripts/databento_test.rs`

**To run**:
```bash
cd /home/jgrusewski/Work/foxhunt
DATABENTO_API_KEY=db-95LEt9gtDRPJfc55NVUB5KL3A3uf6 cargo run --bin databento_test
```

---

## 📊 Phase 5: Validation Checklist

### Before First Download:

- [ ] Verify API key works (check account balance via web portal)
- [ ] Review Databento pricing page: https://databento.com/pricing
- [ ] Use cost estimator in web portal for exact pricing
- [ ] Confirm date range has data available (check catalog)
- [ ] Set up test directory: `mkdir -p test_data/real/databento/`

### After Download:

- [ ] Validate data format (correct schema, timestamps)
- [ ] Check data quality (no gaps, reasonable values)
- [ ] Verify Parquet conversion works
- [ ] Measure actual cost incurred (check billing dashboard)
- [ ] Document exact cost per GB for this dataset
- [ ] Test backtesting pipeline with real data

### Cost Tracking:

Create `test_data/real/databento/COST_TRACKING.md`:
```markdown
# Databento Download Cost Tracking

## Download 1: YYYY-MM-DD
- Symbol: ES.FUT
- Schema: ohlcv-1m
- Date Range: 2024-01-02
- File Size: 4.7 KB
- Actual Cost: $0.000023
- Credits Remaining: $124.99998

## Download 2: YYYY-MM-DD
...
```

---

## 🚀 Phase 6: Next Steps

### Immediate Actions:

1. **Validate API Key** (5 minutes)
   ```bash
   curl -H "Authorization: Bearer db-95LEt9gtDRPJfc55NVUB5KL3A3uf6" \
        https://hist.databento.com/v0/metadata/datasets
   ```

2. **Check Account Balance** (Web UI)
   - Login: https://databento.com/portal
   - Navigate to: Settings → Billing
   - Confirm: $125 credits available

3. **Test Minimal Download** (15 minutes)
   - Download 1 day of OHLCV-1m for ES.FUT
   - Cost: $0.00002 (essentially free)
   - Validate: Data format, timestamps, values

4. **Scale Up Gradually** (30-60 minutes)
   - Week 1: 5 days × 2 symbols × OHLCV-1m = $0.0002
   - Week 2: Add trades data = $0.04
   - Week 3: Test backtesting pipeline
   - Week 4: Download full month if needed

### Medium-Term Strategy:

**Budget Allocation** (Total: $125):
- Testing/Validation: $1 (1%)
- Development Dataset: $25 (20%)
- Backtesting Dataset: $50 (40%)
- Production Testing: $49 (39%)

**Data Coverage**:
- OHLCV-1m: 1 year × 10 symbols = $2-10
- Trades: 3 months × 5 symbols = $50-100
- Level 2: 1 month × 2 symbols = $50-100

**Priority Symbols**:
1. ES.FUT (E-mini S&P 500) - Most liquid futures
2. NQ.FUT (E-mini Nasdaq) - Tech exposure
3. SPY (S&P 500 ETF) - Equity markets
4. QQQ (Nasdaq ETF) - Alternative tech
5. IWM (Russell 2000) - Small caps

---

## 📝 Notes & Warnings

### API Endpoint Discovery:

**Account Balance Endpoint**: NOT publicly documented  
**Workaround**: Use web portal or contact support  
**Alternative**: Track costs manually via billing dashboard

### Pricing Changes (January 2025):

⚠️ **CRITICAL**: Usage-based live data deprecated by March 31, 2025  
✅ **Historical data**: Still pay-as-you-go ($125 credits valid)  
💡 **Live data**: Requires subscription ($199/month Standard plan)

### Data Retention:

- Downloaded data: **Keep forever** (no recurring cost)
- Live streaming: **Pay per message** (not recommended for testing)
- Historical API: **Cache locally** to avoid re-downloading

### Alternative Free Data Sources:

If $125 runs out:
- Yahoo Finance: Free EOD data (limited intraday)
- Alpha Vantage: Free API (500 calls/day limit)
- Polygon.io: $200/month (stocks + options)
- IBKR: Live data with funded account

---

## ✅ Approval & Execution

**Recommended First Download**:
- Symbol: ES.FUT
- Schema: ohlcv-1m
- Date: 2024-01-02
- Cost: $0.00002
- Risk: MINIMAL (0.00002% of budget)

**Decision**: ✅ APPROVED FOR EXECUTION

**Contact for Questions**:
- Databento Support: support@databento.com
- Documentation: https://databento.com/docs
- Pricing Estimator: https://databento.com/pricing

---

**End of Report**