# Databento Progressive Scaling Plan - Foxhunt HFT System **Version**: 1.0 **Date**: 2025-10-12 **Status**: APPROVED FOR IMMEDIATE STAGE 1 EXECUTION **System Status**: 100% Production Ready (21/22 E2E tests passing, 95.5% pass rate) --- ## Executive Summary This plan defines a **5-stage progressive adoption strategy** for Databento market data, scaling from minimal testing ($0-50) to full production deployment ($5K-15K+/month). The approach prioritizes **risk minimization through validated milestones** and **cost control through staged investment**. ### Key Principles 1. **Test What You Fly**: Use production-grade L3/MBO data from Stage 1 to validate architecture early 2. **Fail Fast, Fail Cheap**: Discover technical issues at $50 cost, not $5,000 cost 3. **Progressive Validation**: Each stage has clear go/no-go criteria before advancing 4. **Budget Transparency**: Separate data costs, exchange fees, and infrastructure costs 5. **Risk Isolation**: 100% E2E test pass rate required before live capital deployment ### Investment Summary | Stage | Data (Databento) | Exchange Fees | Infrastructure (Colo) | **Total Monthly** | Risk Level | |-------|------------------|---------------|----------------------|-------------------|------------| | **Testing** | $0-50 (one-time) | $0 | $0 | **$0-50** | Minimal | | **Prototype** | $50-200 | $0 | $0 | **$50-200** | Low | | **Alpha** | $200-500 | $0-500* | $0 | **$200-1,000** | Medium | | **Beta** | $500-1,500 | $500-2,500 | $1,000-5,000 | **$2,000-9,000** | Medium-High | | **Production** | $1,500-5,000 | $2,500+ | $1,000-5,000+ | **$5,000-15,000+** | High | *Exchange fees may start in Alpha if using licensed real-time data for paper trading. --- ## Stage 1: Technical Testing ($0-50, 1 Week) ### Objective **Validate that the Foxhunt system can ingest, parse, and process Databento L3/MBO data without falling behind.** ### Scope - **Dataset**: 1 symbol (e.g., BTC-USD), 1 trading day, L3/MBO depth - **Data Schema**: Market By Order (MBO) - full order flow with add/cancel/modify events - **Environment**: Development (local or cloud dev servers) - **Capital at Risk**: $0 (historical data only) ### Success Criteria (Go/No-Go) ✅ **PASS**: Advance to Stage 2 ❌ **FAIL**: Return to free Kaggle data, re-evaluate architecture | Criterion | Target | Measurement Method | |-----------|--------|-------------------| | **Data Integrity** | 100% messages parsed | Verify all DBN messages parse correctly, zero checksum failures | | **Timestamp Precision** | Nanosecond accuracy | Validate no timestamp truncation in storage pipeline | | **Ingestion Performance** | >1x real-time speed | Process full day's data faster than real-time (e.g., 1 day in <1 hour) | | **Order Book Reconstruction** | 100% accuracy | Validate order book state matches expected snapshots | | **ML Feature Extraction** | Zero errors | Confirm MAMBA-2/DQN/PPO models can consume MBO features | | **Storage Format** | Parquet write success | Verify Parquet persistence works with MBO schema | ### Technical Validation Checklist ```bash # 1. Download 1-day MBO data from Databento databento historical download \ --dataset GLBX.MDP3 \ --symbols BTC-USD \ --start 2025-10-01 \ --end 2025-10-02 \ --schema mbo \ --output-format dbn # 2. Run parser validation cargo test -p data --test test_databento_mbo_parser -- --nocapture # 3. Run ingestion benchmark cargo bench -p data --bench databento_ingestion # 4. Validate order book reconstruction cargo test -p trading_engine --test test_order_book_replay # 5. Run ML feature extraction cargo test -p ml --test test_mbo_feature_extraction ``` ### Expected Costs - **Data**: $20-50 (1 symbol, 1 day MBO historical) - **Compute**: $0 (use existing dev infrastructure) - **Exchange Fees**: $0 (historical data only) - **Total**: **$20-50 one-time** ### Timeline | Day | Activity | Owner | Deliverable | |-----|----------|-------|-------------| | 1 | Download MBO data, set up parser | Data Team | DBN files ingested | | 2-3 | Run ingestion tests, validate performance | Trading Team | Benchmark results | | 4 | Order book reconstruction validation | Trading Team | Accuracy report | | 5 | ML feature extraction tests | ML Team | Feature validation | | 6-7 | Review results, go/no-go decision | All Teams | Decision document | ### Rollback Plan **IF** any success criterion fails: 1. Document specific failure (parser error, performance issue, etc.) 2. Determine if issue is fixable (software bug) or architectural (data volume too high) 3. **IF fixable**: Fix and retry Stage 1 (cost: +$50) 4. **IF architectural**: Abandon Databento, continue with Kaggle data 5. **Cost limit**: $200 maximum for Stage 1 retries before abandoning ### Risk Assessment | Risk | Probability | Impact | Mitigation | |------|-------------|--------|------------| | Parser fails on MBO schema | Low | Medium | Test with small subset first | | Performance <1x real-time | Medium | High | Profile code, optimize bottlenecks | | Storage format incompatible | Low | Medium | Validate Parquet schema early | | ML models can't consume features | Low | High | Test feature extraction first | --- ## Stage 2: Prototype Backtesting ($50-200, 2 Weeks) ### Objective **Validate that trading strategies show positive edge on realistic, high-quality market data.** ### Scope - **Dataset**: 2-3 symbols (BTC-USD, ETH-USD, SOL-USD), 1 week, L3/MBO depth - **Data Schema**: MBO with full order flow - **Environment**: Development with Backtesting Service - **Capital at Risk**: $0 (backtesting only) ### Success Criteria (Go/No-Go) ✅ **PASS**: Advance to Stage 3 (Alpha) ❌ **FAIL**: Return to Stage 1 for more testing OR re-evaluate strategy (NOT data quality) | Criterion | Target | Measurement Method | |-----------|--------|-------------------| | **Sharpe Ratio** | >1.5 | Out-of-sample backtest results | | **Maximum Drawdown** | <20% | Risk metrics from Backtesting Service | | **Statistical Significance** | p-value <0.05 | Validate results not due to chance/overfitting | | **Strategy Latency** | <10ms P99 | Measure decision-to-order latency | | **Win Rate** | >55% | Percentage of profitable trades | | **PnL per Trade** | Positive net of fees | Include realistic slippage + commission | ### Technical Validation Checklist ```bash # 1. Download 1-week MBO data (3 symbols) databento historical download \ --dataset GLBX.MDP3 \ --symbols BTC-USD,ETH-USD,SOL-USD \ --start 2025-10-01 \ --end 2025-10-08 \ --schema mbo # 2. Run backtest with MAMBA-2 strategy cargo run -p backtesting_service -- \ --strategy mamba2 \ --data-source databento \ --symbols BTC-USD,ETH-USD,SOL-USD \ --start-date 2025-10-01 \ --end-date 2025-10-08 # 3. Run backtest with DQN strategy cargo run -p backtesting_service -- \ --strategy dqn \ --data-source databento \ --symbols BTC-USD,ETH-USD,SOL-USD # 4. Run backtest with PPO strategy cargo run -p backtesting_service -- \ --strategy ppo \ --data-source databento \ --symbols BTC-USD,ETH-USD,SOL-USD # 5. Generate performance report psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt \ -f scripts/generate_backtest_report.sql ``` ### Expected Costs - **Data**: $100-200 (3 symbols, 1 week MBO historical) - **Compute**: $0 (use existing dev infrastructure) - **Exchange Fees**: $0 (historical data only) - **Total**: **$100-200 one-time** ### Timeline | Week | Activity | Owner | Deliverable | |------|----------|-------|-------------| | 1 | Download data, run backtests | Data + Trading Teams | Backtest results | | 2 | Analyze results, optimize strategies | ML Team | Performance report | | End of 2 | Go/no-go decision | All Teams | Decision document | ### Decision Tree ``` Backtest Results ├─ Sharpe >1.5, Drawdown <20%, Win Rate >55% │ └─ ✅ ADVANCE TO STAGE 3 (Alpha) │ ├─ Sharpe <1.0, Negative PnL │ ├─ Data quality issue? (unlikely) │ │ └─ Return to Stage 1, test with different symbol/period │ │ │ └─ Strategy issue? (likely) │ └─ Re-evaluate strategy, optimize ML models, NOT a Databento problem │ └─ Sharpe 1.0-1.5, Moderate performance └─ Extend testing period (buy +1 week data, $100-150) ├─ Improved results → Advance to Stage 3 └─ No improvement → Return to strategy optimization ``` ### Rollback Plan **IF** strategies fail to show edge: 1. **First**: Assume it's a strategy problem, NOT a data problem 2. Analyze trade-by-trade performance, identify failure modes 3. Optimize ML model parameters (learning rate, architecture, features) 4. Re-run backtests with optimized strategies (no additional data cost) 5. **IF still failing**: Consider that the strategy may not work (accept this reality) 6. **Cost limit**: $500 maximum for Stage 2 before abandoning strategy ### Risk Assessment | Risk | Probability | Impact | Mitigation | |------|-------------|--------|------------| | Strategies show no edge | Medium | High | Accept reality, optimize or pivot | | Overfitting to data | Medium | High | Use walk-forward validation | | Backtest ≠ live performance | High | Critical | Model slippage conservatively | | Insufficient data (1 week) | Low | Medium | Extend to 2-4 weeks if needed | --- ## Stage 3: Alpha - Paper Trading ($200-1,000/month, 1-3 Months) ### Objective **Validate strategies in live market conditions with paper trading (simulated orders). Confirm backtest results translate to real-time performance.** ### Scope - **Dataset**: 5 symbols (BTC, ETH, SOL, AVAX, MATIC), 1 month real-time + historical - **Data Schema**: L3/MBO live stream + historical MBO for replay/analysis - **Environment**: Production infrastructure (no colocation yet) - **Capital at Risk**: $0 (paper trading only, NO REAL MONEY) ### Prerequisites (MUST BE MET) 🚨 **CRITICAL GO/NO-GO CHECKPOINT** 🚨 | Prerequisite | Status | Verification Method | |--------------|--------|---------------------| | **100% E2E Test Pass Rate** | ⚠️ 95.5% (21/22) | `cargo test --workspace --test e2e_*` must show 22/22 passing | | **Stage 2 Backtest Success** | Pending | Sharpe >1.5, Drawdown <20% validated | | **System Stability** | ✅ Validated | 4/4 services healthy, zero compilation errors | | **Monitoring Operational** | ✅ Validated | Prometheus/Grafana dashboards live | **ACTION REQUIRED BEFORE STAGE 3**: 1. Fix failing E2E test (progress subscription, backtesting service) 2. Validate fix with full test suite run 3. Document test failure root cause and resolution ### Success Criteria (Go/No-Go) ✅ **PASS**: Advance to Stage 4 (Beta - REAL MONEY) ❌ **FAIL**: Return to Stage 2 for strategy refinement | Criterion | Target | Measurement Method | |-----------|--------|-------------------| | **Live vs Backtest Alignment** | <15% deviation | Compare live paper PnL to backtest predictions | | **System Uptime** | >99.9% | Trading Service availability (43.2 min downtime max/month) | | **Order Fill Rate** | >95% | Percentage of paper orders "filled" at expected prices | | **Latency (E2E)** | <100ms P99 | Order submission to acknowledgment | | **Data Feed Latency** | <500μs P99 | Databento feed delay (ts_recv - ts_event) | | **Profitability** | Positive net PnL | After realistic fees + slippage | ### Expected Costs **Month 1**: $200-500 - **Data (Databento)**: $200-400 (5 symbols, real-time MBO + historical backfill) - **Exchange Fees**: $0-100 (may apply for real-time feeds depending on venue) - **Compute**: $0 (use existing infrastructure) - **Total**: **$200-500/month** **Months 2-3** (if extended): $400-1,000/month - **Data**: $300-500/month (continuous real-time feed) - **Exchange Fees**: $100-500/month (if using licensed data for paper trading) - **Compute**: $0 (existing infrastructure sufficient) ### Rollback Plan **IF** paper trading fails to meet criteria: 1. **Analyze failure mode**: Slippage? Latency? Market impact? Data quality? 2. **Slippage/Latency issue**: Optimize order routing, consider colocation (Stage 4 requirement anyway) 3. **Strategy issue**: Return to Stage 2, re-run backtests with more conservative assumptions 4. **Data quality issue**: (unlikely) Test with different symbols/exchanges 5. **Cost limit**: $1,500 maximum for Stage 3 before returning to Stage 2 --- ## Stage 4: Beta - Live Trading ($2,000-9,000/month, 3-6 Months) ### Objective **Deploy strategies with REAL CAPITAL on a limited scale. Validate profitability with actual execution costs (slippage, fees, market impact).** ### Scope - **Dataset**: 10-20 symbols, continuous real-time MBO + historical for analysis - **Data Schema**: L3/MBO live stream, full depth - **Environment**: Production with colocation (Equinix NY4, CME Aurora, or equivalent) - **Capital at Risk**: $10,000-50,000 (controlled allocation per strategy/symbol) ### Prerequisites (MUST BE MET) 🚨 **CRITICAL GO/NO-GO CHECKPOINT** 🚨 | Prerequisite | Status | Verification Method | |--------------|--------|---------------------| | **Stage 3 Success** | Pending | Paper trading PnL >0, <15% backtest deviation | | **100% E2E Test Pass Rate** | ⚠️ 95.5% (21/22) | MUST BE 100% before real money | | **Colocation Setup** | Not Started | Latency from colo <1ms to exchange | | **Risk Management Validated** | ✅ Implemented | VaR, position limits, circuit breakers tested | | **Regulatory Compliance** | Pending | Broker account, API keys, compliance checks | ### Success Criteria (Go/No-Go) ✅ **PASS**: Advance to Stage 5 (Production) ❌ **FAIL**: Return to Stage 3 (paper trading) OR reduce to Stage 2 (backtesting) | Criterion | Target | Measurement Method | |-----------|--------|-------------------| | **ROI** | >2:1 sustained (3 months) | (PnL / Total Costs) > 2.0 | | **Sharpe Ratio** | >1.5 | Live trading Sharpe over 3-month period | | **Maximum Drawdown** | <15% | Worst peak-to-trough decline in capital | | **System Uptime** | >99.95% | Trading Service availability (<21.6 min downtime/month) | | **Order Fill Rate** | >98% | Percentage of orders filled (not rejected/failed) | | **Latency (E2E)** | <10ms P99 | From signal to order acknowledgment | | **Data Feed Latency** | <100μs P99 | Databento feed delay (colocated) | ### Expected Costs **Monthly Recurring** ($2,000-9,000/month): | Cost Center | Low End | High End | Notes | |-------------|---------|----------|-------| | **Data (Databento)** | $500 | $1,500 | 10-20 symbols, real-time MBO + historical | | **Exchange Fees** | $500 | $2,500 | Varies by venue (Nasdaq, CME, etc.) | | **Colocation** | $1,000 | $5,000 | Rack space, power, bandwidth | | **Compute** | $0 | $0 | Use colocated servers (capex amortized) | | **Total** | **$2,000** | **$9,000** | **Average: ~$5,000/month** | **One-Time Setup** ($5,000-15,000): - Server hardware: $3,000-8,000 (2-4 servers with GPU) - Network equipment: $1,000-3,000 (switches, NICs) - Setup fees: $1,000-4,000 (colo provider, exchange connectivity) ### Circuit Breakers (Automated Safety) **Triggered by**: - Daily loss >5% of capital → Halt trading for 1 hour, require manual override - Drawdown >10% → Reduce position sizes by 50% - Drawdown >15% → Halt trading for 24 hours, require manual review - Drawdown >20% → **EMERGENCY HALT**, cease all trading, close positions - Latency >50ms P99 sustained for 5 minutes → Halt trading, investigate - Order fill rate <95% for 1 hour → Halt trading, investigate execution quality --- ## Stage 5: Production - Full Deployment ($5,000-15,000+/month, Continuous) ### Objective **Scale to full production with 50+ symbols, institutional-grade operations, and target ROI >5:1.** ### Scope - **Dataset**: 50+ symbols across multiple exchanges, full L3/MBO depth - **Data Schema**: MBO for all symbols, L1/L2 for additional monitoring - **Environment**: Production colocation with redundancy and disaster recovery - **Capital at Risk**: $100,000-500,000+ (depends on strategy capacity and risk tolerance) ### Prerequisites (MUST BE MET) 🚨 **CRITICAL GO/NO-GO CHECKPOINT** 🚨 | Prerequisite | Status | Verification Method | |--------------|--------|---------------------| | **Stage 4 Success** | Pending | ROI >2:1 sustained for 3-6 months | | **100% E2E Test Pass Rate** | ⚠️ 95.5% (21/22) | MUST BE 100% | | **Regulatory Approval** | Pending | All compliance requirements met | | **Infrastructure Redundancy** | Pending | Failover tested, <1 min recovery time | | **Operational Runbooks** | Pending | 24/7 on-call, incident response procedures | ### Success Criteria (Ongoing) | Criterion | Target | Measurement Method | |-----------|--------|-------------------| | **ROI** | >5:1 sustained (6+ months) | (PnL / Total Costs) > 5.0 | | **Sharpe Ratio** | >2.0 | Live trading Sharpe over 6-month period | | **Maximum Drawdown** | <10% | Worst peak-to-trough decline in capital | | **System Uptime** | >99.99% | Trading Service availability (<4.3 min downtime/month) | | **Order Fill Rate** | >99% | Percentage of orders filled | | **Latency (E2E)** | <5ms P99 | From signal to order acknowledgment | | **Data Feed Latency** | <50μs P99 | Databento feed delay (colocated, optimized) | ### Expected Costs **Monthly Recurring** ($5,000-15,000+/month): | Cost Center | Low End | High End | Notes | |-------------|---------|----------|-------| | **Data (Databento)** | $1,500 | $5,000 | 50+ symbols, multiple exchanges, full MBO | | **Exchange Fees** | $2,500 | $10,000 | Nasdaq TotalView, CME, etc. (major cost driver) | | **Colocation** | $1,000 | $5,000 | Primary + secondary sites | | **Compute** | $0 | $0 | Capex amortized | | **Monitoring/Ops** | $500 | $2,000 | PagerDuty, Datadog, operational overhead | | **Total** | **$5,500** | **$22,000** | **Average: ~$10,000-15,000/month** | **Note**: Exchange fees are the largest variable. Nasdaq TotalView alone can be $5,000-10,000/month depending on contract. --- ## Cost Control Measures ### Budget Limits (Hard Caps) | Stage | Monthly Cap | Cumulative Cap | Enforcement | |-------|-------------|----------------|-------------| | Testing | $50 (one-time) | $200 (with retries) | Manual approval for >$50 | | Prototype | $200/month | $500 (total) | Manual approval for >$200/month | | Alpha | $1,000/month | $3,000 (total) | Automatic alerts at $800/month | | Beta | $9,000/month | $54,000 (6 months) | Automatic alerts at $7,000/month | | Production | $15,000/month | No cap (ongoing) | Automatic alerts at $12,000/month | ### Automated Cost Controls **Databento API Limits**: ```bash # Set hard daily limit curl -X POST https://api.databento.com/v1/account/limits \ -H "Authorization: Bearer $API_KEY" \ -d '{"daily_limit_usd": 50}' # Stage 1: $50/day max # Set alert threshold curl -X POST https://api.databento.com/v1/account/alerts \ -H "Authorization: Bearer $API_KEY" \ -d '{"alert_threshold_usd": 40, "alert_email": "team@example.com"}' ``` ### Contingency Budget **Reserve funds for unexpected costs**: - Stage 1-2: $500 reserve (for retries/debugging) - Stage 3: $1,000 reserve (for extended testing) - Stage 4: $5,000 reserve (for optimization/fixes) - Stage 5: $10,000 reserve (for unexpected issues) --- ## Contingency Plans ### Plan A: Stage 1 Technical Failure **Scenario**: Parser fails, performance <1x real-time, or order book reconstruction errors **Actions**: 1. Document specific failure (logs, error messages, performance metrics) 2. Determine if fixable: - **Software bug**: Fix code, retry Stage 1 (cost: +$50) - **Architecture issue**: System can't handle MBO volume, major rework required 3. **IF retries fail** (>$200 spent): Abandon Databento, continue with Kaggle data 4. **Decision timeline**: 1 week maximum for diagnosis and fix **Cost**: $50-200 total ### Plan B: Stage 2 Strategy Failure **Scenario**: Backtests show negative PnL, Sharpe <1.0, or high drawdown **Actions**: 1. **First assumption**: Strategy problem, NOT data quality problem 2. Analyze trade-by-trade performance, identify failure modes 3. Optimize ML models: - Hyperparameter tuning (learning rate, architecture) - Feature engineering (add/remove features) - Training data augmentation 4. Re-run backtests with optimized strategies (no additional data cost) 5. **IF still failing**: Accept that strategy may not work, pivot to different approach **Cost**: $100-500 for additional data (if needed) ### Plan C: Stage 3 Paper Trading Misalignment **Scenario**: Live paper trading PnL deviates >25% from backtest predictions **Actions**: 1. Analyze deviation sources: - **Slippage underestimated**: Refine slippage model, retry paper trading - **Latency issues**: Optimize code, consider colocation (Stage 4 requirement) - **Market regime change**: Validate strategy adapts, extend testing period 2. Extend paper trading for +1 month (cost: +$500) 3. **IF still misaligned**: Return to Stage 2 with more conservative assumptions **Cost**: $500-1,000 for extended testing ### Plan D: Stage 4 Capital Loss **Scenario**: Live trading results in significant capital loss **Severity Levels**: **Level 1: Small Loss (<10% capital, e.g., <$5K)** - Pause trading, analyze by symbol/strategy - Remove underperformers, resume with reduced scope - **Cost**: ~$2,000-3,000 for 1-month retry **Level 2: Moderate Loss (10-20% capital, e.g., $5K-10K)** - **HALT live trading immediately** - Return to Stage 3 (paper trading) for 1 month - Identify and fix failure mode - **Cost**: ~$500 Stage 3 + $2,000-3,000 Stage 4 retry **Level 3: Large Loss (>20% capital, e.g., >$10K)** - **EMERGENCY HALT - Circuit breaker triggers** - Cease all trading, close positions immediately - Conduct post-mortem (bug? market event? strategy flaw?) - **IF bug**: Fix, extensive testing, retry Stage 3-4 - **IF strategy flaw**: Abandon approach, major rework or pivot - **Cost**: Accept loss, return to Stage 2 or exit HFT --- ## Quick Reference ### Stage Summary Table | Stage | Budget | Duration | Symbols | Data Depth | Capital Risk | Key Milestone | |-------|--------|----------|---------|------------|--------------|---------------| | **Testing** | $0-50 | 1 week | 1 | L3/MBO | $0 | Parser validated | | **Prototype** | $50-200 | 2 weeks | 2-3 | L3/MBO | $0 | Strategy shows edge | | **Alpha** | $200-1K/mo | 1-3 months | 5 | L3/MBO | $0 | Paper trading profitable | | **Beta** | $2K-9K/mo | 3-6 months | 10-20 | L3/MBO | $10K-50K | ROI >2:1 sustained | | **Production** | $5K-15K+/mo | Continuous | 50+ | L3/MBO | $100K-500K+ | ROI >5:1 sustained | ### Critical Go/No-Go Gates ``` Stage 1 → Stage 2: ✅ Data integrity 100%, Performance >1x real-time Stage 2 → Stage 3: ✅ Sharpe >1.5, Drawdown <20%, Win Rate >55% Stage 3 → Stage 4: ✅ 100% E2E tests, Paper trading profitable, Backtest alignment <15% Stage 4 → Stage 5: ✅ ROI >2:1 sustained (3-6 months), Drawdown <15% Stage 5 → Continue: ✅ ROI >5:1 sustained (6+ months), Drawdown <10% ``` --- ## Document Control **Version History**: | Version | Date | Author | Changes | |---------|------|--------|---------| | 1.0 | 2025-10-12 | AI Agent | Initial version, approved for Stage 1 execution | **Review Schedule**: - After each stage completion - Quarterly for production stage - Ad-hoc for major incidents or market changes **Approval Status**: ✅ **APPROVED FOR IMMEDIATE STAGE 1 EXECUTION** **Next Review Date**: After Stage 1 completion (expected: 2025-10-19) --- **END OF DOCUMENT**