foxhunt/databento_scaling_plan.md

Databento Progressive Scaling Plan - Foxhunt HFT System

Version: 1.0
Date: 2025-10-12
Status: APPROVED FOR IMMEDIATE STAGE 1 EXECUTION
System Status: 100% Production Ready (21/22 E2E tests passing, 95.5% pass rate)

---

Executive Summary

This plan defines a 5-stage progressive adoption strategy for Databento market data, scaling from minimal testing ($0-50) to full production deployment ($5K-15K+/month). The approach prioritizes risk minimization through validated milestones and cost control through staged investment.

Key Principles

1. Test What You Fly: Use production-grade L3/MBO data from Stage 1 to validate architecture early
2. Fail Fast, Fail Cheap: Discover technical issues at $50 cost, not $5,000 cost
3. Progressive Validation: Each stage has clear go/no-go criteria before advancing
4. Budget Transparency: Separate data costs, exchange fees, and infrastructure costs
5. Risk Isolation: 100% E2E test pass rate required before live capital deployment

Investment Summary

Stage	Data (Databento)	Exchange Fees	Infrastructure (Colo)	Total Monthly	Risk Level
Testing	$0-50 (one-time)	$0	$0	$0-50	Minimal
Prototype	$50-200	$0	$0	$50-200	Low
Alpha	$200-500	$0-500*	$0	$200-1,000	Medium
Beta	$500-1,500	$500-2,500	$1,000-5,000	$2,000-9,000	Medium-High
Production	$1,500-5,000	$2,500+	$1,000-5,000+	$5,000-15,000+	High

*Exchange fees may start in Alpha if using licensed real-time data for paper trading.

---

Stage 1: Technical Testing ($0-50, 1 Week)

Objective

Validate that the Foxhunt system can ingest, parse, and process Databento L3/MBO data without falling behind.

Scope

• Dataset: 1 symbol (e.g., BTC-USD), 1 trading day, L3/MBO depth
• Data Schema: Market By Order (MBO) - full order flow with add/cancel/modify events
• Environment: Development (local or cloud dev servers)
• Capital at Risk: $0 (historical data only)

Success Criteria (Go/No-Go)

✅ PASS: Advance to Stage 2
❌ FAIL: Return to free Kaggle data, re-evaluate architecture

Criterion	Target	Measurement Method
Data Integrity	100% messages parsed	Verify all DBN messages parse correctly, zero checksum failures
Timestamp Precision	Nanosecond accuracy	Validate no timestamp truncation in storage pipeline
Ingestion Performance	>1x real-time speed	Process full day's data faster than real-time (e.g., 1 day in <1 hour)
Order Book Reconstruction	100% accuracy	Validate order book state matches expected snapshots
ML Feature Extraction	Zero errors	Confirm MAMBA-2/DQN/PPO models can consume MBO features
Storage Format	Parquet write success	Verify Parquet persistence works with MBO schema

Technical Validation Checklist

# 1. Download 1-day MBO data from Databento
databento historical download \
  --dataset GLBX.MDP3 \
  --symbols BTC-USD \
  --start 2025-10-01 \
  --end 2025-10-02 \
  --schema mbo \
  --output-format dbn

# 2. Run parser validation
cargo test -p data --test test_databento_mbo_parser -- --nocapture

# 3. Run ingestion benchmark
cargo bench -p data --bench databento_ingestion

# 4. Validate order book reconstruction
cargo test -p trading_engine --test test_order_book_replay

# 5. Run ML feature extraction
cargo test -p ml --test test_mbo_feature_extraction

Expected Costs

• Data: $20-50 (1 symbol, 1 day MBO historical)
• Compute: $0 (use existing dev infrastructure)
• Exchange Fees: $0 (historical data only)
• Total: $20-50 one-time

Timeline

Day	Activity	Owner	Deliverable
1	Download MBO data, set up parser	Data Team	DBN files ingested
2-3	Run ingestion tests, validate performance	Trading Team	Benchmark results
4	Order book reconstruction validation	Trading Team	Accuracy report
5	ML feature extraction tests	ML Team	Feature validation
6-7	Review results, go/no-go decision	All Teams	Decision document

Rollback Plan

IF any success criterion fails:

1. Document specific failure (parser error, performance issue, etc.)
2. Determine if issue is fixable (software bug) or architectural (data volume too high)
3. IF fixable: Fix and retry Stage 1 (cost: +$50)
4. IF architectural: Abandon Databento, continue with Kaggle data
5. Cost limit: $200 maximum for Stage 1 retries before abandoning

Risk Assessment

Risk	Probability	Impact	Mitigation
Parser fails on MBO schema	Low	Medium	Test with small subset first
Performance <1x real-time	Medium	High	Profile code, optimize bottlenecks
Storage format incompatible	Low	Medium	Validate Parquet schema early
ML models can't consume features	Low	High	Test feature extraction first

---

Stage 2: Prototype Backtesting ($50-200, 2 Weeks)

Objective

Validate that trading strategies show positive edge on realistic, high-quality market data.

Scope

• Dataset: 2-3 symbols (BTC-USD, ETH-USD, SOL-USD), 1 week, L3/MBO depth
• Data Schema: MBO with full order flow
• Environment: Development with Backtesting Service
• Capital at Risk: $0 (backtesting only)

Success Criteria (Go/No-Go)

✅ PASS: Advance to Stage 3 (Alpha)
❌ FAIL: Return to Stage 1 for more testing OR re-evaluate strategy (NOT data quality)

Criterion	Target	Measurement Method
Sharpe Ratio	>1.5	Out-of-sample backtest results
Maximum Drawdown	<20%	Risk metrics from Backtesting Service
Statistical Significance	p-value <0.05	Validate results not due to chance/overfitting
Strategy Latency	<10ms P99	Measure decision-to-order latency
Win Rate	>55%	Percentage of profitable trades
PnL per Trade	Positive net of fees	Include realistic slippage + commission

Technical Validation Checklist

# 1. Download 1-week MBO data (3 symbols)
databento historical download \
  --dataset GLBX.MDP3 \
  --symbols BTC-USD,ETH-USD,SOL-USD \
  --start 2025-10-01 \
  --end 2025-10-08 \
  --schema mbo

# 2. Run backtest with MAMBA-2 strategy
cargo run -p backtesting_service -- \
  --strategy mamba2 \
  --data-source databento \
  --symbols BTC-USD,ETH-USD,SOL-USD \
  --start-date 2025-10-01 \
  --end-date 2025-10-08

# 3. Run backtest with DQN strategy
cargo run -p backtesting_service -- \
  --strategy dqn \
  --data-source databento \
  --symbols BTC-USD,ETH-USD,SOL-USD

# 4. Run backtest with PPO strategy
cargo run -p backtesting_service -- \
  --strategy ppo \
  --data-source databento \
  --symbols BTC-USD,ETH-USD,SOL-USD

# 5. Generate performance report
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \
  -f scripts/generate_backtest_report.sql

Expected Costs

• Data: $100-200 (3 symbols, 1 week MBO historical)
• Compute: $0 (use existing dev infrastructure)
• Exchange Fees: $0 (historical data only)
• Total: $100-200 one-time

Timeline

Week	Activity	Owner	Deliverable
1	Download data, run backtests	Data + Trading Teams	Backtest results
2	Analyze results, optimize strategies	ML Team	Performance report
End of 2	Go/no-go decision	All Teams	Decision document

Decision Tree

Backtest Results
├─ Sharpe >1.5, Drawdown <20%, Win Rate >55%
│  └─ ✅ ADVANCE TO STAGE 3 (Alpha)
│
├─ Sharpe <1.0, Negative PnL
│  ├─ Data quality issue? (unlikely)
│  │  └─ Return to Stage 1, test with different symbol/period
│  │
│  └─ Strategy issue? (likely)
│     └─ Re-evaluate strategy, optimize ML models, NOT a Databento problem
│
└─ Sharpe 1.0-1.5, Moderate performance
   └─ Extend testing period (buy +1 week data, $100-150)
      ├─ Improved results → Advance to Stage 3
      └─ No improvement → Return to strategy optimization

Rollback Plan

IF strategies fail to show edge:

1. First: Assume it's a strategy problem, NOT a data problem
2. Analyze trade-by-trade performance, identify failure modes
3. Optimize ML model parameters (learning rate, architecture, features)
4. Re-run backtests with optimized strategies (no additional data cost)
5. IF still failing: Consider that the strategy may not work (accept this reality)
6. Cost limit: $500 maximum for Stage 2 before abandoning strategy

Risk Assessment

Risk	Probability	Impact	Mitigation
Strategies show no edge	Medium	High	Accept reality, optimize or pivot
Overfitting to data	Medium	High	Use walk-forward validation
Backtest ≠ live performance	High	Critical	Model slippage conservatively
Insufficient data (1 week)	Low	Medium	Extend to 2-4 weeks if needed

---

Stage 3: Alpha - Paper Trading ($200-1,000/month, 1-3 Months)

Objective

Validate strategies in live market conditions with paper trading (simulated orders). Confirm backtest results translate to real-time performance.

Scope

• Dataset: 5 symbols (BTC, ETH, SOL, AVAX, MATIC), 1 month real-time + historical
• Data Schema: L3/MBO live stream + historical MBO for replay/analysis
• Environment: Production infrastructure (no colocation yet)
• Capital at Risk: $0 (paper trading only, NO REAL MONEY)

Prerequisites (MUST BE MET)

🚨 CRITICAL GO/NO-GO CHECKPOINT 🚨

Prerequisite	Status	Verification Method
100% E2E Test Pass Rate	⚠️ 95.5% (21/22)	cargo test --workspace --test e2e_* must show 22/22 passing
Stage 2 Backtest Success	Pending	Sharpe >1.5, Drawdown <20% validated
System Stability	✅ Validated	4/4 services healthy, zero compilation errors
Monitoring Operational	✅ Validated	Prometheus/Grafana dashboards live

ACTION REQUIRED BEFORE STAGE 3:

1. Fix failing E2E test (progress subscription, backtesting service)
2. Validate fix with full test suite run
3. Document test failure root cause and resolution

Success Criteria (Go/No-Go)

✅ PASS: Advance to Stage 4 (Beta - REAL MONEY)
❌ FAIL: Return to Stage 2 for strategy refinement

Criterion	Target	Measurement Method
Live vs Backtest Alignment	<15% deviation	Compare live paper PnL to backtest predictions
System Uptime	>99.9%	Trading Service availability (43.2 min downtime max/month)
Order Fill Rate	>95%	Percentage of paper orders "filled" at expected prices
Latency (E2E)	<100ms P99	Order submission to acknowledgment
Data Feed Latency	<500μs P99	Databento feed delay (ts_recv - ts_event)
Profitability	Positive net PnL	After realistic fees + slippage

Expected Costs

Month 1: $200-500

• Data (Databento): $200-400 (5 symbols, real-time MBO + historical backfill)
• Exchange Fees: $0-100 (may apply for real-time feeds depending on venue)
• Compute: $0 (use existing infrastructure)
• Total: $200-500/month

Months 2-3 (if extended): $400-1,000/month

• Data: $300-500/month (continuous real-time feed)
• Exchange Fees: $100-500/month (if using licensed data for paper trading)
• Compute: $0 (existing infrastructure sufficient)

Rollback Plan

IF paper trading fails to meet criteria:

1. Analyze failure mode: Slippage? Latency? Market impact? Data quality?
2. Slippage/Latency issue: Optimize order routing, consider colocation (Stage 4 requirement anyway)
3. Strategy issue: Return to Stage 2, re-run backtests with more conservative assumptions
4. Data quality issue: (unlikely) Test with different symbols/exchanges
5. Cost limit: $1,500 maximum for Stage 3 before returning to Stage 2

---

Stage 4: Beta - Live Trading ($2,000-9,000/month, 3-6 Months)

Objective

Deploy strategies with REAL CAPITAL on a limited scale. Validate profitability with actual execution costs (slippage, fees, market impact).

Scope

• Dataset: 10-20 symbols, continuous real-time MBO + historical for analysis
• Data Schema: L3/MBO live stream, full depth
• Environment: Production with colocation (Equinix NY4, CME Aurora, or equivalent)
• Capital at Risk: $10,000-50,000 (controlled allocation per strategy/symbol)

Prerequisites (MUST BE MET)

🚨 CRITICAL GO/NO-GO CHECKPOINT 🚨

Prerequisite	Status	Verification Method
Stage 3 Success	Pending	Paper trading PnL >0, <15% backtest deviation
100% E2E Test Pass Rate	⚠️ 95.5% (21/22)	MUST BE 100% before real money
Colocation Setup	Not Started	Latency from colo <1ms to exchange
Risk Management Validated	✅ Implemented	VaR, position limits, circuit breakers tested
Regulatory Compliance	Pending	Broker account, API keys, compliance checks

Success Criteria (Go/No-Go)

✅ PASS: Advance to Stage 5 (Production)
❌ FAIL: Return to Stage 3 (paper trading) OR reduce to Stage 2 (backtesting)

Criterion	Target	Measurement Method
ROI	>2:1 sustained (3 months)	(PnL / Total Costs) > 2.0
Sharpe Ratio	>1.5	Live trading Sharpe over 3-month period
Maximum Drawdown	<15%	Worst peak-to-trough decline in capital
System Uptime	>99.95%	Trading Service availability (<21.6 min downtime/month)
Order Fill Rate	>98%	Percentage of orders filled (not rejected/failed)
Latency (E2E)	<10ms P99	From signal to order acknowledgment
Data Feed Latency	<100μs P99	Databento feed delay (colocated)

Expected Costs

Monthly Recurring ($2,000-9,000/month):

Cost Center	Low End	High End	Notes
Data (Databento)	$500	$1,500	10-20 symbols, real-time MBO + historical
Exchange Fees	$500	$2,500	Varies by venue (Nasdaq, CME, etc.)
Colocation	$1,000	$5,000	Rack space, power, bandwidth
Compute	$0	$0	Use colocated servers (capex amortized)
Total	$2,000	$9,000	Average: ~$5,000/month

One-Time Setup ($5,000-15,000):

• Server hardware: $3,000-8,000 (2-4 servers with GPU)
• Network equipment: $1,000-3,000 (switches, NICs)
• Setup fees: $1,000-4,000 (colo provider, exchange connectivity)

Circuit Breakers (Automated Safety)

Triggered by:

• Daily loss >5% of capital → Halt trading for 1 hour, require manual override
• Drawdown >10% → Reduce position sizes by 50%
• Drawdown >15% → Halt trading for 24 hours, require manual review
• Drawdown >20% → EMERGENCY HALT, cease all trading, close positions
• Latency >50ms P99 sustained for 5 minutes → Halt trading, investigate
• Order fill rate <95% for 1 hour → Halt trading, investigate execution quality

---

Stage 5: Production - Full Deployment ($5,000-15,000+/month, Continuous)

Objective

Scale to full production with 50+ symbols, institutional-grade operations, and target ROI >5:1.

Scope

• Dataset: 50+ symbols across multiple exchanges, full L3/MBO depth
• Data Schema: MBO for all symbols, L1/L2 for additional monitoring
• Environment: Production colocation with redundancy and disaster recovery
• Capital at Risk: $100,000-500,000+ (depends on strategy capacity and risk tolerance)

Prerequisites (MUST BE MET)

🚨 CRITICAL GO/NO-GO CHECKPOINT 🚨

Prerequisite	Status	Verification Method
Stage 4 Success	Pending	ROI >2:1 sustained for 3-6 months
100% E2E Test Pass Rate	⚠️ 95.5% (21/22)	MUST BE 100%
Regulatory Approval	Pending	All compliance requirements met
Infrastructure Redundancy	Pending	Failover tested, <1 min recovery time
Operational Runbooks	Pending	24/7 on-call, incident response procedures

Success Criteria (Ongoing)

Criterion	Target	Measurement Method
ROI	>5:1 sustained (6+ months)	(PnL / Total Costs) > 5.0
Sharpe Ratio	>2.0	Live trading Sharpe over 6-month period
Maximum Drawdown	<10%	Worst peak-to-trough decline in capital
System Uptime	>99.99%	Trading Service availability (<4.3 min downtime/month)
Order Fill Rate	>99%	Percentage of orders filled
Latency (E2E)	<5ms P99	From signal to order acknowledgment
Data Feed Latency	<50μs P99	Databento feed delay (colocated, optimized)

Expected Costs

Monthly Recurring ($5,000-15,000+/month):

Cost Center	Low End	High End	Notes
Data (Databento)	$1,500	$5,000	50+ symbols, multiple exchanges, full MBO
Exchange Fees	$2,500	$10,000	Nasdaq TotalView, CME, etc. (major cost driver)
Colocation	$1,000	$5,000	Primary + secondary sites
Compute	$0	$0	Capex amortized
Monitoring/Ops	$500	$2,000	PagerDuty, Datadog, operational overhead
Total	$5,500	$22,000	Average: ~$10,000-15,000/month

Note: Exchange fees are the largest variable. Nasdaq TotalView alone can be $5,000-10,000/month depending on contract.

---

Cost Control Measures

Budget Limits (Hard Caps)

Stage	Monthly Cap	Cumulative Cap	Enforcement
Testing	$50 (one-time)	$200 (with retries)	Manual approval for >$50
Prototype	$200/month	$500 (total)	Manual approval for >$200/month
Alpha	$1,000/month	$3,000 (total)	Automatic alerts at $800/month
Beta	$9,000/month	$54,000 (6 months)	Automatic alerts at $7,000/month
Production	$15,000/month	No cap (ongoing)	Automatic alerts at $12,000/month

Automated Cost Controls

Databento API Limits:

# Set hard daily limit
curl -X POST https://api.databento.com/v1/account/limits \
  -H "Authorization: Bearer $API_KEY" \
  -d '{"daily_limit_usd": 50}'  # Stage 1: $50/day max

# Set alert threshold
curl -X POST https://api.databento.com/v1/account/alerts \
  -H "Authorization: Bearer $API_KEY" \
  -d '{"alert_threshold_usd": 40, "alert_email": "team@example.com"}'

Contingency Budget

Reserve funds for unexpected costs:

• Stage 1-2: $500 reserve (for retries/debugging)
• Stage 3: $1,000 reserve (for extended testing)
• Stage 4: $5,000 reserve (for optimization/fixes)
• Stage 5: $10,000 reserve (for unexpected issues)

---

Contingency Plans

Plan A: Stage 1 Technical Failure

Scenario: Parser fails, performance <1x real-time, or order book reconstruction errors

Actions:

1. Document specific failure (logs, error messages, performance metrics)
2. Determine if fixable:
   • Software bug: Fix code, retry Stage 1 (cost: +$50)
   • Architecture issue: System can't handle MBO volume, major rework required
3. IF retries fail (>$200 spent): Abandon Databento, continue with Kaggle data
4. Decision timeline: 1 week maximum for diagnosis and fix

Cost: $50-200 total

Plan B: Stage 2 Strategy Failure

Scenario: Backtests show negative PnL, Sharpe <1.0, or high drawdown

Actions:

1. First assumption: Strategy problem, NOT data quality problem
2. Analyze trade-by-trade performance, identify failure modes
3. Optimize ML models:
   • Hyperparameter tuning (learning rate, architecture)
   • Feature engineering (add/remove features)
   • Training data augmentation
4. Re-run backtests with optimized strategies (no additional data cost)
5. IF still failing: Accept that strategy may not work, pivot to different approach

Cost: $100-500 for additional data (if needed)

Plan C: Stage 3 Paper Trading Misalignment

Scenario: Live paper trading PnL deviates >25% from backtest predictions

Actions:

1. Analyze deviation sources:
   • Slippage underestimated: Refine slippage model, retry paper trading
   • Latency issues: Optimize code, consider colocation (Stage 4 requirement)
   • Market regime change: Validate strategy adapts, extend testing period
2. Extend paper trading for +1 month (cost: +$500)
3. IF still misaligned: Return to Stage 2 with more conservative assumptions

Cost: $500-1,000 for extended testing

Plan D: Stage 4 Capital Loss

Scenario: Live trading results in significant capital loss

Severity Levels:

Level 1: Small Loss (<10% capital, e.g., <$5K)

• Pause trading, analyze by symbol/strategy
• Remove underperformers, resume with reduced scope
• Cost: ~$2,000-3,000 for 1-month retry

Level 2: Moderate Loss (10-20% capital, e.g., $5K-10K)

• HALT live trading immediately
• Return to Stage 3 (paper trading) for 1 month
• Identify and fix failure mode
• Cost: ~$500 Stage 3 + $2,000-3,000 Stage 4 retry

Level 3: Large Loss (>20% capital, e.g., >$10K)

• EMERGENCY HALT - Circuit breaker triggers
• Cease all trading, close positions immediately
• Conduct post-mortem (bug? market event? strategy flaw?)
• IF bug: Fix, extensive testing, retry Stage 3-4
• IF strategy flaw: Abandon approach, major rework or pivot
• Cost: Accept loss, return to Stage 2 or exit HFT

---

Quick Reference

Stage Summary Table

Stage	Budget	Duration	Symbols	Data Depth	Capital Risk	Key Milestone
Testing	$0-50	1 week	1	L3/MBO	$0	Parser validated
Prototype	$50-200	2 weeks	2-3	L3/MBO	$0	Strategy shows edge
Alpha	$200-1K/mo	1-3 months	5	L3/MBO	$0	Paper trading profitable
Beta	$2K-9K/mo	3-6 months	10-20	L3/MBO	$10K-50K	ROI >2:1 sustained
Production	$5K-15K+/mo	Continuous	50+	L3/MBO	$100K-500K+	ROI >5:1 sustained

Critical Go/No-Go Gates

Stage 1 → Stage 2: ✅ Data integrity 100%, Performance >1x real-time
Stage 2 → Stage 3: ✅ Sharpe >1.5, Drawdown <20%, Win Rate >55%
Stage 3 → Stage 4: ✅ 100% E2E tests, Paper trading profitable, Backtest alignment <15%
Stage 4 → Stage 5: ✅ ROI >2:1 sustained (3-6 months), Drawdown <15%
Stage 5 → Continue: ✅ ROI >5:1 sustained (6+ months), Drawdown <10%

---

Document Control

Version History:

Version	Date	Author	Changes
1.0	2025-10-12	AI Agent	Initial version, approved for Stage 1 execution

Review Schedule:

• After each stage completion
• Quarterly for production stage
• Ad-hoc for major incidents or market changes

Approval Status: ✅ APPROVED FOR IMMEDIATE STAGE 1 EXECUTION

Next Review Date: After Stage 1 completion (expected: 2025-10-19)

---

END OF DOCUMENT