foxhunt/docs/DATABENTO_GUIDELINES.md

Databento Credit Management Guidelines

Version: 1.0 Date: 2025-10-13 Budget: $125.00 (free credits for historical data) Credits Remaining: ~$124.9960 (after 3 downloads) Status: ACTIVE - Credit Conservation Required

---

Executive Summary

This document provides comprehensive guidelines for cost-effective and strategic acquisition of market data from Databento while conserving the limited $125 credit budget. The goal is to maximize data value while preventing accidental credit exhaustion through careful planning, prioritization, and budget tracking.

Key Principle: Download once, use many times. Every download should be justified and planned.

---

1. Credit Budget Overview

Current Status

Metric	Value	Notes
Initial Credits	$125.00	Free tier for historical data only
Used to Date	$0.0040	3 downloads (ES.FUT, NQ.FUT, CL.FUT)
Remaining	$124.9960	99.997% available
Expiration	Unknown	Check Databento portal for expiration date

Usage History

1. 2024-01-02 ES.FUT (96.47 KB): $0.000045 - $0.000180
2. 2024-01-02 NQ.FUT (92.29 KB): $0.000044 - $0.000176
3. 2024-01-02 CL.FUT (1.46 MB): $0.000712 - $0.002848
4. 2024-01-03-05 ESH4 (3 files, ~57 KB): $0.00009 - $0.00036

Total Actual Cost: $0.001-$0.003 (rounded to $0.004 high estimate)

---

2. Pricing Structure

Schema Cost Tiers (Per GB)

Databento charges by uncompressed data size, not time period. Different schemas have vastly different costs:

Schema	Description	Cost/GB	Size/Day (1 symbol)	Est. Cost/Day	Use Case
OHLCV-1m	1-minute bars	$0.50-$2.00	~12 KB	$0.00001	Backtesting, low-freq strategies
OHLCV-1s	1-second bars	$0.50-$2.00	~700 KB	$0.0004	Higher resolution analysis
TBBO	Top of book	$2-$5	~50 KB	$0.0001	Basic order book
Trades	All trades	$5-$15	~240 KB	$0.0012	Trade analysis, execution
MBP-1	L2 (1 level)	$10-$30	~4.8 MB	$0.05	Spread trading
MBP-10	L2 (10 levels)	$20-$50	~48 MB	$1.0	Deep book analysis
MBO	L3 (full depth)	$30-$100	~100+ MB	$3.0+	HFT, market microstructure

Critical Insight: OHLCV-1m is 1,000-10,000x cheaper than Level 3 order book data!

Symbol Type Modifiers

The stype_in parameter affects data volume and cost:

Symbol Type	Description	Cost Multiplier	Example
continuous	Front contract only	1.0x	ES.FUT → ESH4 (March 2024)
parent	All related instruments	10-60x	CL.FUT includes 60+ spreads
specific contract	Single month	1.0x	ESH4 (March 2024 only)

Warning: Using parent on commodities with many spreads (CL.FUT, GC.FUT) can increase costs by 15-60x!

---

3. Budget Allocation Strategy

Recommended Allocation ($125 Total)

Phase	Purpose	Budget	Remaining	Priority
Testing	API validation, parser testing	$1 (1%)	$124	CRITICAL
Development	Multi-symbol, multi-regime testing	$25 (20%)	$99	HIGH
Backtesting	Strategy validation dataset	$50 (40%)	$49	HIGH
Production Prep	Extended historical data	$49 (39%)	$0	MEDIUM

Phase-Specific Guidelines

Phase 1: Testing ($0-1, COMPLETE)

✅ Status: COMPLETE ✅ Used: $0.004 ✅ Deliverables: ES.FUT, NQ.FUT, CL.FUT (1 day each)

Lessons Learned:

• OHLCV-1m is extremely cheap (~$0.00002/day/symbol)
• Parent symbols on commodities are 15x larger
• DBN format validated, parser working

Phase 2: Development ($1-25, IN PROGRESS)

🎯 Current Status: 4 days ES.FUT downloaded 🎯 Goal: Multi-symbol, multi-regime testing data 🎯 Remaining Budget: $24.996

Recommended Downloads:

1. Regime Diversity (Priority: HIGH)

   • ES.FUT: 1-2 weeks of different market conditions
   • Cost: $0.001-0.002 per week
   • Use: Adaptive strategy regime detection testing

2. Cross-Symbol Testing (Priority: HIGH)

   • NQ.FUT, RTY.FUT: 5 days each (tech, small-cap)
   • Cost: $0.0002 per symbol
   • Use: Multi-asset strategy validation

3. Commodity Testing (Priority: MEDIUM)

   • GC.FUT, ZC.FUT: 3 days each (gold, corn - use specific contracts!)
   • Cost: $0.00006 per day (specific contract)
   • Use: Cross-asset correlation analysis

Phase 3: Backtesting ($25-75, NOT STARTED)

🔒 Status: BLOCKED until Phase 2 complete 🔒 Requirements: Validated strategies, identified data gaps

Target Dataset:

• 10 symbols × 30 days × OHLCV-1m: ~$0.006
• 5 symbols × 30 days × Trades: ~$0.18
• 2 symbols × 30 days × MBP-1: ~$3.00
• Total: ~$3-5 for comprehensive backtesting dataset

Phase 4: Production Prep ($75-125, NOT STARTED)

🔒 Status: BLOCKED until strategies validated 🔒 Requirements: Positive Sharpe ratio (>1.5), validated strategies

Extended Historical Data:

• 6-12 months of key symbols
• Multiple market regimes (trending, ranging, volatile, crisis)
• Out-of-sample validation dataset

---

4. Cost Optimization Best Practices

4.1 Schema Selection

✅ DO: Start with cheapest schema

# Example: OHLCV-1m for initial testing
schema=ohlcv-1m  # ~$0.00001/day

❌ DON'T: Jump to expensive schemas

# Example: MBO for testing (wasteful)
schema=mbo  # ~$3.00/day (300,000x more expensive!)

Upgrade Path:

1. OHLCV-1m → Validate strategy logic
2. OHLCV-1s → Add resolution if needed
3. Trades → Add execution analysis
4. MBP-1/MBO → Only if strategy requires order book

4.2 Symbol Type Selection

✅ DO: Use specific contracts

# Example: Specific contract month
symbols=ESH4  # March 2024 E-mini S&P 500
stype_in=continuous

❌ DON'T: Use parent on complex instruments

# Example: Parent symbol on commodity with many spreads
symbols=CL.FUT
stype_in=parent  # Downloads 60+ related instruments (15x cost!)

Symbol Type Decision Tree:

Need front contract only? → Use specific contract (ESH4)
Need rollover analysis? → Use continuous
Need spread analysis? → Use parent (BUT BUDGET CAREFULLY!)

4.3 Date Range Selection

✅ DO: Download minimal viable range

# Example: Single day for testing
start=2024-01-02T00:00:00Z
end=2024-01-03T00:00:00Z  # 1 day

❌ DON'T: Download full year upfront

# Example: Full year without plan (wasteful)
start=2024-01-01T00:00:00Z
end=2025-01-01T00:00:00Z  # 252 days (may not need all!)

Date Selection Strategy:

1. Download 1 day → Validate pipeline
2. Download 1 week → Test strategy logic
3. Download 1 month → Validate Sharpe ratio
4. Download 3-6 months → Out-of-sample testing
5. Download 1+ years → Production dataset (only if profitable!)

4.4 Symbol Prioritization

Liquidity-Based Prioritization:

Tier	Symbols	Liquidity	Priority	Use Case
Tier 1	ES, SPY, NQ, QQQ	Ultra-high	CRITICAL	Core equity strategies
Tier 2	RTY, IWM, CL, GC	High	HIGH	Diversification
Tier 3	ZC, ZS, HG, SI	Medium	MEDIUM	Commodity strategies
Tier 4	Exotic spreads, options	Low	LOW	Advanced strategies only

Download Order:

1. Tier 1: 1-2 symbols (ES, NQ)
2. Validate strategies work
3. Tier 2: Add 2-3 symbols (RTY, CL, GC)
4. Validate cross-asset performance
5. Tier 3+: Only if strategies require specific assets

4.5 Avoid Redundant Downloads

✅ DO: Cache locally

# Save downloaded files permanently
test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

❌ DON'T: Re-download same data

# Check if file exists before downloading
if [ ! -f "ES.FUT_ohlcv-1m_2024-01-02.dbn" ]; then
    curl ...  # Download only if missing
fi

File Naming Convention:

{SYMBOL}_{SCHEMA}_{START_DATE}.dbn
Examples:
- ES.FUT_ohlcv-1m_2024-01-02.dbn
- NQ.FUT_trades_2024-01-02.dbn
- ESH4_mbo_2024-01-02.dbn

---

5. Budget Alerts and Thresholds

Alert Levels

Threshold	Credits Used	Remaining	Action Required
GREEN	<$25	>$100	Continue as planned
YELLOW	$25-$75	$50-$100	Review spending, prioritize critical downloads
ORANGE	$75-$100	$25-$50	HALT non-critical downloads, executive approval required
RED	$100-$125	$0-$25	EMERGENCY HALT, preserve for critical fixes only

Monitoring Commands

Check Credit Balance (via Databento Portal):

1. Login: https://databento.com/portal
2. Navigate: Settings → Billing
3. View: "Credits Remaining"

Track Cumulative Usage (local tracking):

# View usage log
cat /home/jgrusewski/Work/foxhunt/COST_TRACKING.md

# Calculate total spent
grep "Estimated Cost" COST_TRACKING.md | awk '{sum+=$NF} END {print "Total: $"sum}'

Automated Budget Alerts

Create alert script (recommended):

#!/bin/bash
# scripts/check_databento_budget.sh

USED=$(grep "Estimated Cost" COST_TRACKING.md | awk '{sum+=$NF} END {print sum}')
REMAINING=$(echo "125 - $USED" | bc)

if (( $(echo "$USED > 100" | bc -l) )); then
    echo "🚨 RED ALERT: $USED spent, only \$$REMAINING remaining!"
    exit 1
elif (( $(echo "$USED > 75" | bc -l) )); then
    echo "🟠 ORANGE ALERT: $USED spent, \$$REMAINING remaining"
    exit 0
elif (( $(echo "$USED > 25" | bc -l) )); then
    echo "🟡 YELLOW ALERT: $USED spent, \$$REMAINING remaining"
    exit 0
else
    echo "✅ GREEN: $USED spent, \$$REMAINING remaining"
    exit 0
fi

Usage:

# Run before each download
./scripts/check_databento_budget.sh

# Add to pre-commit hook (optional)

---

6. Cost Estimation Methodology

Formula

Estimated Cost = File Size (GB) × Price per GB

Price per GB:
- Low estimate: $0.50 (OHLCV schemas)
- High estimate: $2.00 (OHLCV schemas)
- Mid estimate: $1.25 (use for planning)

Size Estimation (Before Download)

Based on historical data:

Schema	Size/Bar	Bars/Day	Size/Day (1 symbol)
OHLCV-1m	~60 bytes	390-1440	~25-90 KB
OHLCV-1s	~60 bytes	23400-86400	~1.4-5 MB
Trades	~120 bytes	~2000	~240 KB
MBP-1	~200 bytes	~24000	~4.8 MB

Example Calculation:

Download Plan: ES.FUT, 5 days, OHLCV-1m

Estimated Size: 5 days × 90 KB/day = 450 KB = 0.00044 GB
Estimated Cost (low): 0.00044 × $0.50 = $0.00022
Estimated Cost (high): 0.00044 × $2.00 = $0.00088
Budget: Use high estimate ($0.00088) for safety

Post-Download Verification

After each download:

1. Check actual file size: ls -lh test_data/real/databento/
2. Calculate actual cost: File Size (GB) × $0.50-$2.00
3. Update COST_TRACKING.md with actual cost
4. Compare estimated vs actual (improve future estimates)

---

7. Prioritization Matrix

Decision Framework

When deciding whether to download data, use this matrix:

Factor	Weight	Scoring
Strategy Requirement	40%	0 = Nice-to-have, 10 = Critical blocker
Data Uniqueness	30%	0 = Already have similar, 10 = Unique regime
Cost Efficiency	20%	0 = >$5, 10 = <$0.01
Immediate Use	10%	0 = Future use, 10 = Needed today

Total Score: (Factor1 × 0.4) + (Factor2 × 0.3) + (Factor3 × 0.2) + (Factor4 × 0.1)

Decision Thresholds:

• Score >7.5: APPROVE (high priority)
• Score 5.0-7.5: REVIEW (medium priority, conditional approval)
• Score <5.0: DEFER (low priority, wait until budget permits)

Example Prioritization

Scenario 1: Additional ES.FUT trending day

• Strategy Requirement: 8 (need more regime diversity)
• Data Uniqueness: 6 (have 1 trending day, want 2nd for validation)
• Cost Efficiency: 10 (<$0.00002)
• Immediate Use: 9 (testing adaptive strategy today)
• Score: (8×0.4) + (6×0.3) + (10×0.2) + (9×0.1) = 7.9 → APPROVE

Scenario 2: Full year of MBO data for CL.FUT

• Strategy Requirement: 3 (not currently using L3 order book)
• Data Uniqueness: 7 (unique high-resolution data)
• Cost Efficiency: 0 (~$750 for full year)
• Immediate Use: 2 (future research project)
• Score: (3×0.4) + (7×0.3) + (0×0.2) + (2×0.1) = 3.5 → DEFER

---

8. Alternatives to Databento

Free Data Sources

When credits are low or data needs are basic:

Source	Cost	Data Quality	Coverage	Best For
Yahoo Finance	Free	Moderate	EOD + delayed intraday	Daily strategies, backtesting
Alpha Vantage	Free (500 calls/day)	Good	1-minute bars (limited)	Retail stocks, basic testing
Kaggle Datasets	Free	Varies	Pre-packaged datasets	Research, ML training
Polygon.io	$29-199/month	Good	Stocks, crypto, forex	Retail trading APIs
IBKR	Free (with funded account)	Excellent	Real-time (with account)	Live trading, paper trading

When to Use Alternatives:

• ✅ EOD (end-of-day) data: Use Yahoo Finance
• ✅ Basic backtesting: Use Kaggle datasets
• ✅ Live paper trading: Use IBKR paper account
• ❌ High-resolution tick data: Databento required
• ❌ HFT-grade timestamps: Databento required

Synthetic Data

When real data is unavailable or too expensive:

Acceptable Use Cases:

• Unit testing (parser validation, schema testing)
• Integration testing (pipeline throughput, error handling)
• Load testing (high-volume ingestion)

Unacceptable Use Cases:

• Strategy backtesting (results will be unrealistic)
• ML model training (garbage in, garbage out)
• Performance validation (latency, Sharpe ratio)

---

9. Download Planning Checklist

Before every download, complete this checklist:

Pre-Download

• Justification: Why is this data needed? (Strategy requirement, testing, research)
• Alternatives Checked: Can I use existing data or free sources?
• Schema Justified: Why this schema? (Start with cheapest, upgrade if needed)
• Symbol Type Verified: Using continuous or specific contract (not parent on commodities)?
• Date Range Minimized: Downloading smallest viable range?
• Budget Check: Current credits remaining >20% buffer for this download?
• Cost Estimated: Calculated high estimate cost?
• File Path Planned: Where will this be saved?
• Use Case Documented: How will this data be used?

Post-Download

• File Verified: File exists and has expected size?
• Format Validated: DBN header correct, parseable?
• Data Quality: Spot-check prices, volumes, timestamps?
• Cost Tracked: Actual cost logged in COST_TRACKING.md?
• Pipeline Tested: Data successfully integrated into backtesting/ML pipeline?
• Documentation Updated: Download details documented?

---

10. Emergency Procedures

Scenario: Credit Exhaustion

IF credits run out before critical work complete:

1. Immediate Actions:

   • HALT all downloads
   • Assess remaining critical needs
   • Review COST_TRACKING.md for spending patterns

2. Options:

   • Option A: Purchase additional credits (credit card required)
   • Option B: Switch to free data sources (Yahoo, Alpha Vantage)
   • Option C: Use synthetic data for non-critical testing
   • Option D: Wait for credit renewal (if applicable)

3. Prevention:

   • Maintain 20% credit buffer ($25)
   • Require approval for downloads >$5
   • Review spending weekly

Scenario: Accidental Large Download

IF accidentally downloaded expensive data (MBO, parent symbol, full year):

1. Immediate Actions:

   • Check file size: ls -lh test_data/real/databento/
   • Estimate cost: File Size (GB) × $2.00 (high estimate)
   • Check credits remaining (Databento portal)

2. Damage Control:

   • IF credits remaining >$50: Continue, but adjust future plans
   • IF credits remaining $25-50: HALT non-critical downloads
   • IF credits remaining <$25: EMERGENCY HALT, executive approval required

3. Prevention:

   • Use planning script (scripts/plan_databento_download.sh)
   • Start with 1-day download, verify cost before scaling
   • Double-check symbol type (avoid parent on commodities)

---

11. Success Metrics

Key Performance Indicators (KPIs)

Metric	Target	Current	Status
Credits Remaining	>$75 (60%)	$124.996 (99.997%)	✅ EXCELLENT
Cost per Download	<$1.00 average	$0.001	✅ EXCELLENT
Data Reuse Rate	>5 uses per download	TBD	📊 Measuring
Download Efficiency	<5% wasted downloads	0%	✅ EXCELLENT
Budget Overruns	0 incidents	0	✅ PERFECT

Monthly Review

Conduct monthly review (or after $25 spent):

1. Spending Analysis:

   • Total credits used this month
   • Cost per download (average, min, max)
   • Most expensive downloads (identify waste)

2. Data Utilization:

   • Which downloads were used in backtesting?
   • Which downloads were never used? (waste)
   • Reuse rate (uses per download)

3. Adjustment Recommendations:

   • Schema optimization opportunities
   • Symbol prioritization changes
   • Budget allocation adjustments

---

12. Contact and Support

Databento Support

• Technical Support: support@databento.com
• Sales (Discounts): sales@databento.com
• Documentation: https://databento.com/docs
• Pricing Estimator: https://databento.com/pricing

Internal Escalation

Budget Concerns:

1. Yellow Alert ($25-75 used): Inform team, review plan
2. Orange Alert ($75-100 used): Require approval for all downloads
3. Red Alert ($100+ used): Emergency halt, executive decision

Technical Issues:

1. Parser failures: Check data/src/providers/databento/ implementation
2. Cost overruns: Review scripts/plan_databento_download.sh estimates
3. Data quality: Validate with data/examples/validate_*.rs scripts

---

13. Quick Reference

Common Download Commands

Single day, single symbol (OHLCV-1m):

curl -u "${DATABENTO_API_KEY}:" \
  "https://hist.databento.com/v0/timeseries.get_range?\
dataset=GLBX.MDP3&\
symbols=ES.FUT&\
schema=ohlcv-1m&\
start=2024-01-02T00:00:00Z&\
end=2024-01-03T00:00:00Z&\
encoding=dbn&\
stype_in=continuous" \
  --output test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn

Multi-symbol download:

# Use comma-separated symbols
symbols=ES.FUT,NQ.FUT,RTY.FUT

Estimate cost before download:

./scripts/plan_databento_download.sh \
  --symbols "ES.FUT,NQ.FUT" \
  --days 5 \
  --schema ohlcv-1m

Decision Tree

Need data? → Check existing files first
  ↓
Existing data sufficient? → Use existing (STOP)
  ↓ NO
Free source available? → Use free source (Yahoo, Kaggle)
  ↓ NO
Estimate cost: <$0.01? → APPROVE (download immediately)
  ↓ NO
Estimate cost: $0.01-$0.10? → REVIEW (check budget, prioritize)
  ↓ NO
Estimate cost: >$0.10? → REQUIRE APPROVAL (justify + review)
  ↓
Download planned? → Use scripts/plan_databento_download.sh
  ↓
Download → Verify file, update COST_TRACKING.md
  ↓
Use data → Mark as used, track reuse rate

---

14. Version History

Version	Date	Changes	Author
1.0	2025-10-13	Initial guidelines	Agent 23

---

15. Appendix: Cost Projections

Scenario A: Conservative Testing ($0-25)

Target: Validate strategies with minimal data

• 5 symbols (ES, NQ, RTY, CL, GC)
• 5 days each (total: 25 days)
• OHLCV-1m schema
• Estimated Cost: 25 days × $0.00002/day = $0.0005
• Credits Remaining: $124.9995

Scenario B: Comprehensive Backtesting ($25-75)

Target: Full strategy validation dataset

• 10 symbols
• 30 days each (total: 300 days)
• OHLCV-1m schema
• Estimated Cost: 300 days × $0.00002/day = $0.006
• Credits Remaining: $124.994

With Trades schema (for execution analysis):

• 5 symbols
• 30 days each (total: 150 days)
• Trades schema
• Estimated Cost: 150 days × $0.0012/day = $0.18
• Total with OHLCV: $0.006 + $0.18 = $0.186
• Credits Remaining: $124.814

Scenario C: Production Dataset ($75-125)

Target: Extended historical data for production

• 10 symbols
• 252 days (1 year)
• OHLCV-1m + Trades
• OHLCV Cost: 2520 days × $0.00002 = $0.0504
• Trades Cost: 1260 days × $0.0012 = $1.512
• Total: $1.5624
• Credits Remaining: $123.4376

Adding Level 2 (MBP-1) for 2 symbols:

• 2 symbols × 30 days = 60 days
• MBP-1 Cost: 60 days × $0.05/day = $3.00
• Total: $1.5624 + $3.00 = $4.5624
• Credits Remaining: $120.4376

---

Conclusion: With careful planning, the $125 credit budget can support:

• ✅ Comprehensive testing (Scenarios A+B): <$1
• ✅ Full production dataset (Scenario C): ~$5
• ✅ Multiple strategy iterations: ~$10-20
• ✅ Reserve for emergencies: $100+

Key Success Factor: Schema selection (OHLCV vs MBO = 1000x cost difference)

---

Document Status: APPROVED FOR USE Next Review: After $25 credits used or 2025-11-13 (whichever comes first)

---

END OF GUIDELINES