foxhunt/docs/DATABENTO_GUIDELINES.md

# Databento Credit Management Guidelines

**Version**: 1.0
**Date**: 2025-10-13
**Budget**: $125.00 (free credits for historical data)
**Credits Remaining**: ~$124.9960 (after 3 downloads)
**Status**: ACTIVE - Credit Conservation Required

---

## Executive Summary

This document provides comprehensive guidelines for **cost-effective and strategic acquisition of market data** from Databento while conserving the limited $125 credit budget. The goal is to maximize data value while preventing accidental credit exhaustion through careful planning, prioritization, and budget tracking.

**Key Principle**: **Download once, use many times**. Every download should be justified and planned.

---

## 1. Credit Budget Overview

### Current Status

| Metric | Value | Notes |
|--------|-------|-------|
| **Initial Credits** | $125.00 | Free tier for historical data only |
| **Used to Date** | $0.0040 | 3 downloads (ES.FUT, NQ.FUT, CL.FUT) |
| **Remaining** | $124.9960 | 99.997% available |
| **Expiration** | Unknown | Check Databento portal for expiration date |

### Usage History

1. **2024-01-02 ES.FUT** (96.47 KB): $0.000045 - $0.000180
2. **2024-01-02 NQ.FUT** (92.29 KB): $0.000044 - $0.000176
3. **2024-01-02 CL.FUT** (1.46 MB): $0.000712 - $0.002848
4. **2024-01-03-05 ESH4** (3 files, ~57 KB): $0.00009 - $0.00036

**Total Actual Cost**: $0.001-$0.003 (rounded to $0.004 high estimate)

---

## 2. Pricing Structure

### Schema Cost Tiers (Per GB)

Databento charges by **uncompressed data size**, not time period. Different schemas have vastly different costs:

| Schema | Description | Cost/GB | Size/Day (1 symbol) | Est. Cost/Day | Use Case |
|--------|-------------|---------|---------------------|---------------|----------|
| **OHLCV-1m** | 1-minute bars | $0.50-$2.00 | ~12 KB | $0.00001 | Backtesting, low-freq strategies |
| **OHLCV-1s** | 1-second bars | $0.50-$2.00 | ~700 KB | $0.0004 | Higher resolution analysis |
| **TBBO** | Top of book | $2-$5 | ~50 KB | $0.0001 | Basic order book |
| **Trades** | All trades | $5-$15 | ~240 KB | $0.0012 | Trade analysis, execution |
| **MBP-1** | L2 (1 level) | $10-$30 | ~4.8 MB | $0.05 | Spread trading |
| **MBP-10** | L2 (10 levels) | $20-$50 | ~48 MB | $1.0 | Deep book analysis |
| **MBO** | L3 (full depth) | $30-$100 | ~100+ MB | $3.0+ | HFT, market microstructure |

**Critical Insight**: OHLCV-1m is **1,000-10,000x cheaper** than Level 3 order book data!

### Symbol Type Modifiers

The `stype_in` parameter affects data volume and cost:

| Symbol Type | Description | Cost Multiplier | Example |
|-------------|-------------|-----------------|---------|
| **continuous** | Front contract only | 1.0x | ES.FUT → ESH4 (March 2024) |
| **parent** | All related instruments | **10-60x** | CL.FUT includes 60+ spreads |
| **specific contract** | Single month | 1.0x | ESH4 (March 2024 only) |

**Warning**: Using `parent` on commodities with many spreads (CL.FUT, GC.FUT) can increase costs by 15-60x!

---

## 3. Budget Allocation Strategy

### Recommended Allocation ($125 Total)

| Phase | Purpose | Budget | Remaining | Priority |
|-------|---------|--------|-----------|----------|
| **Testing** | API validation, parser testing | $1 (1%) | $124 | CRITICAL |
| **Development** | Multi-symbol, multi-regime testing | $25 (20%) | $99 | HIGH |
| **Backtesting** | Strategy validation dataset | $50 (40%) | $49 | HIGH |
| **Production Prep** | Extended historical data | $49 (39%) | $0 | MEDIUM |

### Phase-Specific Guidelines

#### Phase 1: Testing ($0-1, COMPLETE)

✅ **Status**: COMPLETE
✅ **Used**: $0.004
✅ **Deliverables**: ES.FUT, NQ.FUT, CL.FUT (1 day each)

**Lessons Learned**:
- OHLCV-1m is extremely cheap (~$0.00002/day/symbol)
- Parent symbols on commodities are 15x larger
- DBN format validated, parser working

#### Phase 2: Development ($1-25, IN PROGRESS)

🎯 **Current Status**: 4 days ES.FUT downloaded
🎯 **Goal**: Multi-symbol, multi-regime testing data
🎯 **Remaining Budget**: $24.996

**Recommended Downloads**:

1. **Regime Diversity** (Priority: HIGH)
   - ES.FUT: 1-2 weeks of different market conditions
   - Cost: $0.001-0.002 per week
   - Use: Adaptive strategy regime detection testing

2. **Cross-Symbol Testing** (Priority: HIGH)
   - NQ.FUT, RTY.FUT: 5 days each (tech, small-cap)
   - Cost: $0.0002 per symbol
   - Use: Multi-asset strategy validation

3. **Commodity Testing** (Priority: MEDIUM)
   - GC.FUT, ZC.FUT: 3 days each (gold, corn - use specific contracts!)
   - Cost: $0.00006 per day (specific contract)
   - Use: Cross-asset correlation analysis

#### Phase 3: Backtesting ($25-75, NOT STARTED)

🔒 **Status**: BLOCKED until Phase 2 complete
🔒 **Requirements**: Validated strategies, identified data gaps

**Target Dataset**:
- 10 symbols × 30 days × OHLCV-1m: ~$0.006
- 5 symbols × 30 days × Trades: ~$0.18
- 2 symbols × 30 days × MBP-1: ~$3.00
- **Total**: ~$3-5 for comprehensive backtesting dataset

#### Phase 4: Production Prep ($75-125, NOT STARTED)

🔒 **Status**: BLOCKED until strategies validated
🔒 **Requirements**: Positive Sharpe ratio (>1.5), validated strategies

**Extended Historical Data**:
- 6-12 months of key symbols
- Multiple market regimes (trending, ranging, volatile, crisis)
- Out-of-sample validation dataset

---

## 4. Cost Optimization Best Practices

### 4.1 Schema Selection

✅ **DO: Start with cheapest schema**
```bash
# Example: OHLCV-1m for initial testing
schema=ohlcv-1m  # ~$0.00001/day
```

❌ **DON'T: Jump to expensive schemas**
```bash
# Example: MBO for testing (wasteful)
schema=mbo  # ~$3.00/day (300,000x more expensive!)
```

**Upgrade Path**:
1. OHLCV-1m → Validate strategy logic
2. OHLCV-1s → Add resolution if needed
3. Trades → Add execution analysis
4. MBP-1/MBO → Only if strategy requires order book

### 4.2 Symbol Type Selection

✅ **DO: Use specific contracts**
```bash
# Example: Specific contract month
symbols=ESH4  # March 2024 E-mini S&P 500
stype_in=continuous
```

❌ **DON'T: Use parent on complex instruments**
```bash
# Example: Parent symbol on commodity with many spreads
symbols=CL.FUT
stype_in=parent  # Downloads 60+ related instruments (15x cost!)
```

**Symbol Type Decision Tree**:
```
Need front contract only? → Use specific contract (ESH4)
Need rollover analysis? → Use continuous
Need spread analysis? → Use parent (BUT BUDGET CAREFULLY!)
```

### 4.3 Date Range Selection

✅ **DO: Download minimal viable range**
```bash
# Example: Single day for testing
start=2024-01-02T00:00:00Z
end=2024-01-03T00:00:00Z  # 1 day
```

❌ **DON'T: Download full year upfront**
```bash
# Example: Full year without plan (wasteful)
start=2024-01-01T00:00:00Z
end=2025-01-01T00:00:00Z  # 252 days (may not need all!)
```

**Date Selection Strategy**:
1. Download 1 day → Validate pipeline
2. Download 1 week → Test strategy logic
3. Download 1 month → Validate Sharpe ratio
4. Download 3-6 months → Out-of-sample testing
5. Download 1+ years → Production dataset (only if profitable!)

### 4.4 Symbol Prioritization

**Liquidity-Based Prioritization**:

| Tier | Symbols | Liquidity | Priority | Use Case |
|------|---------|-----------|----------|----------|
| **Tier 1** | ES, SPY, NQ, QQQ | Ultra-high | CRITICAL | Core equity strategies |
| **Tier 2** | RTY, IWM, CL, GC | High | HIGH | Diversification |
| **Tier 3** | ZC, ZS, HG, SI | Medium | MEDIUM | Commodity strategies |
| **Tier 4** | Exotic spreads, options | Low | LOW | Advanced strategies only |

**Download Order**:
1. Tier 1: 1-2 symbols (ES, NQ)
2. Validate strategies work
3. Tier 2: Add 2-3 symbols (RTY, CL, GC)
4. Validate cross-asset performance
5. Tier 3+: Only if strategies require specific assets

### 4.5 Avoid Redundant Downloads

✅ **DO: Cache locally**
```bash
# Save downloaded files permanently
test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn
```

❌ **DON'T: Re-download same data**
```bash
# Check if file exists before downloading
if [ ! -f "ES.FUT_ohlcv-1m_2024-01-02.dbn" ]; then
    curl ...  # Download only if missing
fi
```

**File Naming Convention**:
```
{SYMBOL}_{SCHEMA}_{START_DATE}.dbn
Examples:
- ES.FUT_ohlcv-1m_2024-01-02.dbn
- NQ.FUT_trades_2024-01-02.dbn
- ESH4_mbo_2024-01-02.dbn
```

---

## 5. Budget Alerts and Thresholds

### Alert Levels

| Threshold | Credits Used | Remaining | Action Required |
|-----------|--------------|-----------|-----------------|
| **GREEN** | <$25 | >$100 | Continue as planned |
| **YELLOW** | $25-$75 | $50-$100 | Review spending, prioritize critical downloads |
| **ORANGE** | $75-$100 | $25-$50 | **HALT non-critical downloads**, executive approval required |
| **RED** | $100-$125 | $0-$25 | **EMERGENCY HALT**, preserve for critical fixes only |

### Monitoring Commands

**Check Credit Balance** (via Databento Portal):
1. Login: https://databento.com/portal
2. Navigate: Settings → Billing
3. View: "Credits Remaining"

**Track Cumulative Usage** (local tracking):
```bash
# View usage log
cat /home/jgrusewski/Work/foxhunt/COST_TRACKING.md

# Calculate total spent
grep "Estimated Cost" COST_TRACKING.md | awk '{sum+=$NF} END {print "Total: $"sum}'
```

### Automated Budget Alerts

**Create alert script** (recommended):
```bash
#!/bin/bash
# scripts/check_databento_budget.sh

USED=$(grep "Estimated Cost" COST_TRACKING.md | awk '{sum+=$NF} END {print sum}')
REMAINING=$(echo "125 - $USED" | bc)

if (( $(echo "$USED > 100" | bc -l) )); then
    echo "🚨 RED ALERT: $USED spent, only \$$REMAINING remaining!"
    exit 1
elif (( $(echo "$USED > 75" | bc -l) )); then
    echo "🟠 ORANGE ALERT: $USED spent, \$$REMAINING remaining"
    exit 0
elif (( $(echo "$USED > 25" | bc -l) )); then
    echo "🟡 YELLOW ALERT: $USED spent, \$$REMAINING remaining"
    exit 0
else
    echo "✅ GREEN: $USED spent, \$$REMAINING remaining"
    exit 0
fi
```

**Usage**:
```bash
# Run before each download
./scripts/check_databento_budget.sh

# Add to pre-commit hook (optional)
```

---

## 6. Cost Estimation Methodology

### Formula

```
Estimated Cost = File Size (GB) × Price per GB

Price per GB:
- Low estimate: $0.50 (OHLCV schemas)
- High estimate: $2.00 (OHLCV schemas)
- Mid estimate: $1.25 (use for planning)
```

### Size Estimation (Before Download)

**Based on historical data**:

| Schema | Size/Bar | Bars/Day | Size/Day (1 symbol) |
|--------|----------|----------|---------------------|
| OHLCV-1m | ~60 bytes | 390-1440 | ~25-90 KB |
| OHLCV-1s | ~60 bytes | 23400-86400 | ~1.4-5 MB |
| Trades | ~120 bytes | ~2000 | ~240 KB |
| MBP-1 | ~200 bytes | ~24000 | ~4.8 MB |

**Example Calculation**:
```
Download Plan: ES.FUT, 5 days, OHLCV-1m

Estimated Size: 5 days × 90 KB/day = 450 KB = 0.00044 GB
Estimated Cost (low): 0.00044 × $0.50 = $0.00022
Estimated Cost (high): 0.00044 × $2.00 = $0.00088
Budget: Use high estimate ($0.00088) for safety
```

### Post-Download Verification

**After each download**:
1. Check actual file size: `ls -lh test_data/real/databento/`
2. Calculate actual cost: `File Size (GB) × $0.50-$2.00`
3. Update `COST_TRACKING.md` with actual cost
4. Compare estimated vs actual (improve future estimates)

---

## 7. Prioritization Matrix

### Decision Framework

When deciding whether to download data, use this matrix:

| Factor | Weight | Scoring |
|--------|--------|---------|
| **Strategy Requirement** | 40% | 0 = Nice-to-have, 10 = Critical blocker |
| **Data Uniqueness** | 30% | 0 = Already have similar, 10 = Unique regime |
| **Cost Efficiency** | 20% | 0 = >$5, 10 = <$0.01 |
| **Immediate Use** | 10% | 0 = Future use, 10 = Needed today |

**Total Score**: (Factor1 × 0.4) + (Factor2 × 0.3) + (Factor3 × 0.2) + (Factor4 × 0.1)

**Decision Thresholds**:
- **Score >7.5**: APPROVE (high priority)
- **Score 5.0-7.5**: REVIEW (medium priority, conditional approval)
- **Score <5.0**: DEFER (low priority, wait until budget permits)

### Example Prioritization

**Scenario 1: Additional ES.FUT trending day**
- Strategy Requirement: 8 (need more regime diversity)
- Data Uniqueness: 6 (have 1 trending day, want 2nd for validation)
- Cost Efficiency: 10 (<$0.00002)
- Immediate Use: 9 (testing adaptive strategy today)
- **Score**: (8×0.4) + (6×0.3) + (10×0.2) + (9×0.1) = **7.9 → APPROVE**

**Scenario 2: Full year of MBO data for CL.FUT**
- Strategy Requirement: 3 (not currently using L3 order book)
- Data Uniqueness: 7 (unique high-resolution data)
- Cost Efficiency: 0 (~$750 for full year)
- Immediate Use: 2 (future research project)
- **Score**: (3×0.4) + (7×0.3) + (0×0.2) + (2×0.1) = **3.5 → DEFER**

---

## 8. Alternatives to Databento

### Free Data Sources

When credits are low or data needs are basic:

| Source | Cost | Data Quality | Coverage | Best For |
|--------|------|--------------|----------|----------|
| **Yahoo Finance** | Free | Moderate | EOD + delayed intraday | Daily strategies, backtesting |
| **Alpha Vantage** | Free (500 calls/day) | Good | 1-minute bars (limited) | Retail stocks, basic testing |
| **Kaggle Datasets** | Free | Varies | Pre-packaged datasets | Research, ML training |
| **Polygon.io** | $29-199/month | Good | Stocks, crypto, forex | Retail trading APIs |
| **IBKR** | Free (with funded account) | Excellent | Real-time (with account) | Live trading, paper trading |

**When to Use Alternatives**:
- ✅ EOD (end-of-day) data: Use Yahoo Finance
- ✅ Basic backtesting: Use Kaggle datasets
- ✅ Live paper trading: Use IBKR paper account
- ❌ High-resolution tick data: Databento required
- ❌ HFT-grade timestamps: Databento required

### Synthetic Data

When real data is unavailable or too expensive:

**Acceptable Use Cases**:
- Unit testing (parser validation, schema testing)
- Integration testing (pipeline throughput, error handling)
- Load testing (high-volume ingestion)

**Unacceptable Use Cases**:
- Strategy backtesting (results will be unrealistic)
- ML model training (garbage in, garbage out)
- Performance validation (latency, Sharpe ratio)

---

## 9. Download Planning Checklist

Before every download, complete this checklist:

### Pre-Download

- [ ] **Justification**: Why is this data needed? (Strategy requirement, testing, research)
- [ ] **Alternatives Checked**: Can I use existing data or free sources?
- [ ] **Schema Justified**: Why this schema? (Start with cheapest, upgrade if needed)
- [ ] **Symbol Type Verified**: Using `continuous` or specific contract (not `parent` on commodities)?
- [ ] **Date Range Minimized**: Downloading smallest viable range?
- [ ] **Budget Check**: Current credits remaining >20% buffer for this download?
- [ ] **Cost Estimated**: Calculated high estimate cost?
- [ ] **File Path Planned**: Where will this be saved?
- [ ] **Use Case Documented**: How will this data be used?

### Post-Download

- [ ] **File Verified**: File exists and has expected size?
- [ ] **Format Validated**: DBN header correct, parseable?
- [ ] **Data Quality**: Spot-check prices, volumes, timestamps?
- [ ] **Cost Tracked**: Actual cost logged in `COST_TRACKING.md`?
- [ ] **Pipeline Tested**: Data successfully integrated into backtesting/ML pipeline?
- [ ] **Documentation Updated**: Download details documented?

---

## 10. Emergency Procedures

### Scenario: Credit Exhaustion

**IF credits run out before critical work complete**:

1. **Immediate Actions**:
   - HALT all downloads
   - Assess remaining critical needs
   - Review COST_TRACKING.md for spending patterns

2. **Options**:
   - **Option A**: Purchase additional credits (credit card required)
   - **Option B**: Switch to free data sources (Yahoo, Alpha Vantage)
   - **Option C**: Use synthetic data for non-critical testing
   - **Option D**: Wait for credit renewal (if applicable)

3. **Prevention**:
   - Maintain 20% credit buffer ($25)
   - Require approval for downloads >$5
   - Review spending weekly

### Scenario: Accidental Large Download

**IF accidentally downloaded expensive data (MBO, parent symbol, full year)**:

1. **Immediate Actions**:
   - Check file size: `ls -lh test_data/real/databento/`
   - Estimate cost: `File Size (GB) × $2.00` (high estimate)
   - Check credits remaining (Databento portal)

2. **Damage Control**:
   - **IF credits remaining >$50**: Continue, but adjust future plans
   - **IF credits remaining $25-50**: HALT non-critical downloads
   - **IF credits remaining <$25**: **EMERGENCY HALT**, executive approval required

3. **Prevention**:
   - Use planning script (scripts/plan_databento_download.sh)
   - Start with 1-day download, verify cost before scaling
   - Double-check symbol type (avoid `parent` on commodities)

---

## 11. Success Metrics

### Key Performance Indicators (KPIs)

| Metric | Target | Current | Status |
|--------|--------|---------|--------|
| **Credits Remaining** | >$75 (60%) | $124.996 (99.997%) | ✅ EXCELLENT |
| **Cost per Download** | <$1.00 average | $0.001 | ✅ EXCELLENT |
| **Data Reuse Rate** | >5 uses per download | TBD | 📊 Measuring |
| **Download Efficiency** | <5% wasted downloads | 0% | ✅ EXCELLENT |
| **Budget Overruns** | 0 incidents | 0 | ✅ PERFECT |

### Monthly Review

**Conduct monthly review** (or after $25 spent):

1. **Spending Analysis**:
   - Total credits used this month
   - Cost per download (average, min, max)
   - Most expensive downloads (identify waste)

2. **Data Utilization**:
   - Which downloads were used in backtesting?
   - Which downloads were never used? (waste)
   - Reuse rate (uses per download)

3. **Adjustment Recommendations**:
   - Schema optimization opportunities
   - Symbol prioritization changes
   - Budget allocation adjustments

---

## 12. Contact and Support

### Databento Support

- **Technical Support**: support@databento.com
- **Sales (Discounts)**: sales@databento.com
- **Documentation**: https://databento.com/docs
- **Pricing Estimator**: https://databento.com/pricing

### Internal Escalation

**Budget Concerns**:
1. Yellow Alert ($25-75 used): Inform team, review plan
2. Orange Alert ($75-100 used): Require approval for all downloads
3. Red Alert ($100+ used): Emergency halt, executive decision

**Technical Issues**:
1. Parser failures: Check `data/src/providers/databento/` implementation
2. Cost overruns: Review `scripts/plan_databento_download.sh` estimates
3. Data quality: Validate with `data/examples/validate_*.rs` scripts

---

## 13. Quick Reference

### Common Download Commands

**Single day, single symbol (OHLCV-1m)**:
```bash
curl -u "${DATABENTO_API_KEY}:" \
  "https://hist.databento.com/v0/timeseries.get_range?\
dataset=GLBX.MDP3&\
symbols=ES.FUT&\
schema=ohlcv-1m&\
start=2024-01-02T00:00:00Z&\
end=2024-01-03T00:00:00Z&\
encoding=dbn&\
stype_in=continuous" \
  --output test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn
```

**Multi-symbol download**:
```bash
# Use comma-separated symbols
symbols=ES.FUT,NQ.FUT,RTY.FUT
```

**Estimate cost before download**:
```bash
./scripts/plan_databento_download.sh \
  --symbols "ES.FUT,NQ.FUT" \
  --days 5 \
  --schema ohlcv-1m
```

### Decision Tree

```
Need data? → Check existing files first
  ↓
Existing data sufficient? → Use existing (STOP)
  ↓ NO
Free source available? → Use free source (Yahoo, Kaggle)
  ↓ NO
Estimate cost: <$0.01? → APPROVE (download immediately)
  ↓ NO
Estimate cost: $0.01-$0.10? → REVIEW (check budget, prioritize)
  ↓ NO
Estimate cost: >$0.10? → REQUIRE APPROVAL (justify + review)
  ↓
Download planned? → Use scripts/plan_databento_download.sh
  ↓
Download → Verify file, update COST_TRACKING.md
  ↓
Use data → Mark as used, track reuse rate
```

---

## 14. Version History

| Version | Date | Changes | Author |
|---------|------|---------|--------|
| 1.0 | 2025-10-13 | Initial guidelines | Agent 23 |

---

## 15. Appendix: Cost Projections

### Scenario A: Conservative Testing ($0-25)

**Target**: Validate strategies with minimal data

- 5 symbols (ES, NQ, RTY, CL, GC)
- 5 days each (total: 25 days)
- OHLCV-1m schema
- **Estimated Cost**: 25 days × $0.00002/day = **$0.0005**
- **Credits Remaining**: $124.9995

### Scenario B: Comprehensive Backtesting ($25-75)

**Target**: Full strategy validation dataset

- 10 symbols
- 30 days each (total: 300 days)
- OHLCV-1m schema
- **Estimated Cost**: 300 days × $0.00002/day = **$0.006**
- **Credits Remaining**: $124.994

**With Trades schema** (for execution analysis):
- 5 symbols
- 30 days each (total: 150 days)
- Trades schema
- **Estimated Cost**: 150 days × $0.0012/day = **$0.18**
- **Total with OHLCV**: $0.006 + $0.18 = **$0.186**
- **Credits Remaining**: $124.814

### Scenario C: Production Dataset ($75-125)

**Target**: Extended historical data for production

- 10 symbols
- 252 days (1 year)
- OHLCV-1m + Trades
- **OHLCV Cost**: 2520 days × $0.00002 = **$0.0504**
- **Trades Cost**: 1260 days × $0.0012 = **$1.512**
- **Total**: **$1.5624**
- **Credits Remaining**: $123.4376

**Adding Level 2 (MBP-1)** for 2 symbols:
- 2 symbols × 30 days = 60 days
- **MBP-1 Cost**: 60 days × $0.05/day = **$3.00**
- **Total**: $1.5624 + $3.00 = **$4.5624**
- **Credits Remaining**: $120.4376

---

**Conclusion**: With careful planning, the $125 credit budget can support:
- ✅ Comprehensive testing (Scenarios A+B): <$1
- ✅ Full production dataset (Scenario C): ~$5
- ✅ Multiple strategy iterations: ~$10-20
- ✅ Reserve for emergencies: $100+

**Key Success Factor**: Schema selection (OHLCV vs MBO = 1000x cost difference)

---

**Document Status**: APPROVED FOR USE
**Next Review**: After $25 credits used or 2025-11-13 (whichever comes first)

---

**END OF GUIDELINES**