foxhunt/docs/archive/data_management/REAL_DATA_NEXT_STEPS.md

Real Data Integration - Next Steps Roadmap

Date: 2025-10-13 Status: Real Data Integration Complete ✅ Phase: Trading Strategy Development

---

🎯 Executive Summary

With real data integration complete, the focus shifts to expanding data coverage and validating strategies with production-grade market data.

Current Status:

• ✅ DBN integration operational (0.70ms, 14x faster than target)
• ✅ 6 DBN files acquired (ES.FUT, ESH4, NQ.FUT, CL.FUT)
• ✅ 19/19 tests passing (100%)
• ✅ 29,000+ lines of documentation

Immediate Priority: Expand data coverage to 5-10 symbols, 30-90 days

---

📋 Phase 1: Data Coverage Expansion (1-2 Weeks)

Priority 1: Additional Futures Symbols (HIGH)

Target: 5-10 futures symbols across multiple asset classes

Symbols to Acquire:

1. ES.FUT - E-mini S&P 500 (already have 1 day ✅)

   • Expand to 30-90 days
   • Target: Q4 2024 (bull market) + Q1 2025 (volatile)

2. NQ.FUT - E-mini Nasdaq 100 (already have 1 day ✅)

   • Expand to 30-90 days
   • High correlation with tech sector

3. CL.FUT - Crude Oil (already have 1 day ✅)

   • Expand to 30-90 days
   • Energy sector, geopolitical volatility

4. GC.FUT - Gold Futures (NEW)

   • 30-90 days
   • Safe haven asset, inflation hedge

5. ZN.FUT - 10-Year Treasury Note (NEW)

   • 30-90 days
   • Interest rate sensitivity, bond market

6. 6E.FUT - Euro FX (NEW, Optional)

   • 30-90 days
   • Currency markets, forex volatility

Data Requirements:

• Format: DBN (Databento Binary) 1-minute OHLCV
• Venue: CME Group (ES, NQ, CL, GC, ZN) + ICE (6E)
• Dataset: ohlcv-1m schema
• Date Range: 30-90 consecutive days (Jan-Mar 2024 or Oct-Dec 2024)

Effort: 2-3 days

• Day 1: Databento API setup, credit purchase (~$50-100)
• Day 2: Download scripts, file organization, validation
• Day 3: Test integration, documentation updates

Success Criteria:

• ✅ 5-10 symbols with 30+ days each
• ✅ All files validated with automatic anomaly correction
• ✅ Test coverage updated to use new symbols
• ✅ Documentation updated with new data inventory

Priority 2: Multi-Day Integration Tests (MEDIUM)

Goal: Validate multi-day loading and continuity

Tasks:

1. Create 30-day continuous backtest test
2. Validate timestamp ordering across days
3. Test overnight gaps and weekends
4. Validate data quality across entire range

Files to Update:

• services/backtesting_service/tests/dbn_multi_day_tests.rs
• services/integration_tests/tests/backtest_real_data_test.rs (NEW)

Effort: 1-2 days

Success Criteria:

• ✅ 30-day continuous backtest passing
• ✅ Gap handling validated (weekends, holidays)
• ✅ Cross-day metrics correct (PnL, drawdown)

Priority 3: Automated Data Pipeline (MEDIUM)

Goal: Automate data acquisition and validation

Components:

1. Download Script (scripts/download_dbn_data.sh)

   • Databento API integration
   • Bulk download support
   • Progress tracking

2. Validation Script (scripts/validate_dbn_quality.sh)

   • Automatic anomaly detection
   • OHLCV validation
   • Summary statistics

3. Update Script (scripts/update_test_data.sh)

   • Weekly/monthly data refresh
   • Version control (git LFS for large files)

Effort: 2-3 days

Success Criteria:

• ✅ One-command data acquisition
• ✅ Automatic validation on download
• ✅ CI/CD integration (weekly data updates)

---

📋 Phase 2: Strategy Backtesting (1-2 Weeks)

Priority 1: Backtest Existing Strategies (HIGH)

Strategies to Test:

1. Moving Average Crossover (trading_engine/src/strategies/moving_average_crossover.rs)

   • Test with ES.FUT 30-day dataset
   • Validate signals vs synthetic data
   • Measure Sharpe, drawdown, win rate

2. Adaptive Strategy (trading_engine/src/strategies/adaptive_strategy.rs)

   • Test regime detection with real volatility
   • Validate trending/ranging/sideways classification
   • Compare vs synthetic regime transitions

3. Mean Reversion (if implemented)

   • Test with ranging market data
   • Validate entry/exit timing

4. Momentum (if implemented)

   • Test with trending market data
   • Validate breakout detection

Effort: 3-5 days (1-2 days per strategy)

Success Criteria:

• ✅ All strategies tested with 30+ days of real data
• ✅ Performance metrics documented (Sharpe, drawdown, PnL, win rate)
• ✅ Edge cases identified (gaps, outliers, extreme volatility)
• ✅ Comparison report: synthetic vs real data performance

Priority 2: Multi-Symbol Portfolio Backtest (MEDIUM)

Goal: Test portfolio-level strategies with real cross-asset data

Scenarios:

1. Sector Rotation: ES.FUT (equities) vs ZN.FUT (bonds)
2. Risk Parity: ES, NQ, CL, GC diversification
3. Hedge Strategies: Long ES, short NQ during tech sell-off

Effort: 2-3 days

Success Criteria:

• ✅ Multi-symbol backtest infrastructure operational
• ✅ Portfolio-level metrics (total PnL, correlation, beta)
• ✅ Rebalancing logic validated

Priority 3: Transaction Cost & Slippage (MEDIUM)

Goal: Model realistic execution costs

Implementation:

1. Add bid-ask spread simulation
2. Model slippage based on volume
3. Include commission structure (CME fees)
4. Validate impact on strategy profitability

Effort: 2-3 days

Success Criteria:

• ✅ Realistic cost model implemented
• ✅ All strategies re-tested with costs
• ✅ Profitability adjusted for slippage

---

📋 Phase 3: ML Model Validation (1-2 Weeks)

Priority 1: MAMBA-2 Model (HIGH)

Goal: Test state space model with real market data

Tasks:

1. Load ES.FUT 30-day dataset
2. Generate features (price, volume, volatility)
3. Run inference on all bars
4. Measure prediction accuracy vs real outcomes
5. Compare vs synthetic data performance

Effort: 2-3 days

Success Criteria:

• ✅ MAMBA-2 inference operational with real data
• ✅ Prediction accuracy measured (MAE, RMSE, R²)
• ✅ Overfitting identified (if any)
• ✅ Hyperparameter adjustments documented

Priority 2: DQN/PPO Reinforcement Learning (HIGH)

Goal: Test RL agents with real market environments

Tasks:

1. Create environment using real data replay
2. Train DQN/PPO agents on 30-day datasets
3. Validate action selection vs real outcomes
4. Measure reward accumulation
5. Compare vs synthetic environment performance

Effort: 3-4 days

Success Criteria:

• ✅ DQN/PPO training operational with real data
• ✅ Cumulative reward measured
• ✅ Policy convergence validated
• ✅ Overfitting identified (if any)

Priority 3: TFT Forecasting (MEDIUM)

Goal: Test temporal fusion transformer for price prediction

Tasks:

1. Load multi-day dataset for time series
2. Generate temporal features
3. Train TFT on historical data
4. Validate forecasting accuracy
5. Measure inference latency

Effort: 2-3 days

Success Criteria:

• ✅ TFT training operational with real data
• ✅ Forecast accuracy measured (MAPE, MAE)
• ✅ Multi-step ahead forecasting validated

Priority 4: Feature Engineering Validation (MEDIUM)

Goal: Validate feature extraction with real market data

Tasks:

1. Test technical indicators (SMA, EMA, RSI, MACD)
2. Validate microstructure features (bid-ask, order imbalance)
3. Test TLOB (Temporal Limit Order Book) features
4. Measure feature importance with real data

Effort: 1-2 days

Success Criteria:

• ✅ All feature extractors operational
• ✅ Feature distributions analyzed (vs synthetic)
• ✅ Feature importance ranking documented

---

📋 Phase 4: Replace Mock Data in Tests (1 Week)

Priority 1: E2E Test Conversion (HIGH)

Goal: Replace synthetic data with real DBN data in all E2E tests

Files to Update:

• services/integration_tests/tests/backtest_integration_test.rs
• services/integration_tests/tests/ml_pipeline_test.rs
• services/integration_tests/tests/trading_service_test.rs

Effort: 2-3 days

Success Criteria:

• ✅ All E2E tests use real DBN data
• ✅ 22/22 tests still passing (100%)
• ✅ Test execution time acceptable (<5 min)

Priority 2: Unit Test Conversion (MEDIUM)

Goal: Replace mock data generators with real data helpers

Files to Update:

• trading_engine/tests/strategy_tests.rs
• ml/tests/model_tests.rs
• data/tests/feature_extraction_tests.rs

Effort: 2-3 days

Success Criteria:

• ✅ Unit tests use real data samples
• ✅ Test coverage maintained (>95%)
• ✅ Edge cases validated with real anomalies

---

📋 Phase 5: Performance Optimization (Optional, 1 Week)

Priority 1: Metadata Caching (MEDIUM)

Goal: Cache file metadata for faster date range queries

Implementation:

• Add first_ts and last_ts to FileEntry
• Parse first/last record on initial load
• Skip files outside date range

Effort: 1-2 days

Success Criteria:

• ✅ 3x speedup for single-day queries on multi-day datasets
• ✅ Backward compatibility maintained

Priority 2: Parallel File Loading (LOW)

Goal: Load multiple files concurrently

Implementation:

• Use tokio::task::JoinSet
• Load files in parallel
• Merge results after completion

Effort: 1-2 days

Success Criteria:

• ✅ Near-linear speedup (3 files in ~1ms instead of 2.1ms)
• ✅ Thread-safe decoder handling

Priority 3: LRU Cache Implementation (LOW)

Goal: Avoid reloading same files for repeated queries

Implementation:

• Replace HashMap with lru::LruCache
• Configurable cache size
• Eviction policy

Effort: 1 day

Success Criteria:

• ✅ Near-instant repeated queries (cache hit = 0.001ms)
• ✅ Memory bounded (configurable limit)

---

📅 Timeline Summary

Phase	Duration	Priority	Status
Phase 1: Data Expansion	1-2 weeks	HIGH	🔜 Next
Phase 2: Strategy Backtesting	1-2 weeks	HIGH	⏸️ Pending data
Phase 3: ML Validation	1-2 weeks	HIGH	⏸️ Pending data
Phase 4: Mock Data Replacement	1 week	MEDIUM	⏸️ Pending validation
Phase 5: Performance Optimization	1 week	LOW	⏸️ Optional

Total Duration: 4-7 weeks (depending on parallel execution)

---

🎯 Success Metrics

Data Coverage

• Target: 5-10 symbols, 30-90 days each
• Current: 4 symbols, 1-4 days each
• Gap: 1-6 additional symbols, 26-89 additional days per symbol

Strategy Performance

• Target: All strategies validated with real data
• Current: Zero strategies tested with real data
• Gap: 3-5 strategies need validation

ML Model Accuracy

• Target: Accuracy measured on real data, overfitting identified
• Current: No ML models tested with real data
• Gap: 4 models (MAMBA-2, DQN, PPO, TFT) need validation

Test Coverage

• Target: 100% real data in tests
• Current: 19/19 backtesting tests use real data, 0/22 E2E tests
• Gap: 22 E2E tests need conversion

---

📝 Deliverables by Phase

Phase 1 Deliverables

• 5-10 DBN files (30-90 days each)
• Updated data inventory documentation
• Automated download/validation scripts
• Multi-day integration tests (30+ days)

Phase 2 Deliverables

• Strategy backtest reports (Sharpe, drawdown, PnL)
• Synthetic vs real data comparison
• Multi-symbol portfolio backtest results
• Transaction cost model implementation

Phase 3 Deliverables

• MAMBA-2 accuracy report (real data)
• DQN/PPO training results (real data)
• TFT forecasting accuracy report
• Feature importance analysis (real vs synthetic)

Phase 4 Deliverables

• E2E tests converted to real data
• Unit tests converted to real data
• 22/22 E2E tests passing (100%)
• Test execution time optimized

Phase 5 Deliverables (Optional)

• Metadata caching implemented
• Parallel file loading implemented
• LRU cache implemented
• Performance benchmark report

---

🚀 Getting Started

Immediate Next Step: Expand Data Coverage

Action: Acquire 5-10 symbols with 30-90 days each

Execution Plan:

# Step 1: Set up Databento API credentials
export DATABENTO_API_KEY="your_api_key_here"

# Step 2: Run download script (to be created)
./scripts/download_dbn_data.sh --symbols ES.FUT,NQ.FUT,CL.FUT,GC.FUT,ZN.FUT --start 2024-01-01 --end 2024-03-31

# Step 3: Validate data quality
./scripts/validate_dbn_quality.sh test_data/real/databento/*.dbn

# Step 4: Run tests
cargo test -p backtesting_service --test dbn_multi_day_tests
cargo test -p backtesting_service --test dbn_integration_tests

# Step 5: Update documentation
# - Update CLAUDE.md with new data inventory
# - Update DBN_INTEGRATION_GUIDE.md with new examples

Expected Outcome:

• ✅ 5-10 symbols with 30-90 days each acquired
• ✅ All files validated with automatic anomaly correction
• ✅ Tests passing with expanded data coverage
• ✅ Documentation updated

Timeline: 2-3 days

---

📞 Questions & Support

For Data Acquisition:

• See docs/DATABENTO_GUIDELINES.md
• See docs/DBN_INTEGRATION_GUIDE.md
• Contact: Databento support (support@databento.com)

For Strategy Backtesting:

• See docs/DBN_INTEGRATION_GUIDE.md (Integration Examples)
• See services/backtesting_service/DBN_REPOSITORY_USAGE.md
• Review docs/examples/dbn_backtesting_integration.rs

For ML Model Validation:

• See TESTING_PLAN.md
• See ml/README.md (if exists)
• Review docs/examples/dbn_basic_loading.rs

For Performance Optimization:

• See services/backtesting_service/tests/fixtures/PERFORMANCE.md
• See docs/DBN_TROUBLESHOOTING.md

---

Roadmap Created: 2025-10-13 Status: Real Data Integration Complete ✅ Next Milestone: Data Coverage Expansion (5-10 symbols, 30-90 days) Expected Completion: 1-2 weeks