# Backtesting Service Feature Integration Investigation ## Executive Summary The Backtesting Service currently uses a **simplified feature extraction pipeline** that is NOT integrated with Wave C features (alternative bars, fractional differentiation, meta-labeling, barrier optimization). Features are extracted at strategy runtime but NOT persisted or validated against actual market outcomes during backtesting. This creates a critical gap between: 1. **Live trading** - Uses `SharedMLStrategy` with full ML inference 2. **Backtesting** - Uses simplified local feature extraction with static parameters 3. **ML training** - Uses 256-feature vectors from `UnifiedFeatureExtractor` in data crate --- ## 1. BACKTESTING ARCHITECTURE ### 1.1 Core Components ``` Backtesting Service Flow: ┌─────────────────────┐ │ DBN Data Source │ ← Loads OHLCV bars from real DBN files (0.70ms) └──────────┬──────────┘ │ ▼ ┌──────────────────────────────────────┐ │ StrategyEngine::execute_backtest() │ ├──────────────────────────────────────┤ │ 1. Load market data (via repository) │ │ 2. For each market data point: │ │ - Call strategy.execute() │ │ - Generate TradeSignals │ │ - Execute trades in portfolio │ │ 3. Calculate performance metrics │ └──────────┬──────────────────────────┘ │ ▼ ┌──────────────────────────────────────┐ │ PerformanceAnalyzer │ ├──────────────────────────────────────┤ │ - Sharpe Ratio │ │ - Drawdown Analysis │ │ - Win Rate │ │ - PnL Calculation │ └──────────────────────────────────────┘ ``` **Key File**: `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/strategy_engine.rs` ### 1.2 Available Strategies 1. **MovingAverageCrossoverStrategy** (Lines 349-404) - Simplistic trigger_price parameter - No feature extraction - No access to UnifiedFeatureExtractor 2. **BuyAndHoldStrategy** (Lines 406-447) - Static allocation-based - No strategy logic or features 3. **NewsAwareStrategy** (Lines 449-526) - Simulated sentiment/momentum (hardcoded values: 0.2, 55.0) - Would benefit from features but doesn't actually extract them - Comment at line 462-464: "This is a simplified example - in reality, the strategy would use the UnifiedFeatureExtractor" 4. **MLPoweredStrategy** (ml_strategy_engine.rs) - Uses SharedMLStrategy from common crate - Delegates to ML models (DQN, PPO, MAMBA-2, TFT) - But still has local MLFeatureExtractor as fallback ### 1.3 Feature Extraction - DISCONNECTED **Current Location 1: StrategyEngine** - Lines 549-554: Creates UnifiedFeatureExtractor with default config - Lines 685-689: **NOT ACTUALLY USED** - just initialized but never called - Comment at line 686: "In production, this would properly convert NewsEvent to the format expected by UnifiedFeatureExtractor" **Current Location 2: MLStrategyEngine.MLFeatureExtractor** - Lines 72-173 (ml_strategy_engine.rs): Local feature extractor - Extracts 8 basic features: 1. Price return 2. Short-term MA ratio 3. Price volatility 4. Volume ratio 5. Volume MA ratio 6. Hour of day 7. Day of week 8. All normalized via tanh() normalization **Problem**: These 8 features are extracted locally WITHOUT integration with: - 18 Wave A technical indicators (RSI, MACD, Bollinger, ATR, ADX, CCI, Stochastic, etc.) - UnifiedFeatureExtractor (256 features in data crate) - Alternative bars (Wave B) - Fractional differentiation (Wave C) - Meta-labeling (Wave C) --- ## 2. PERFORMANCE METRICS CALCULATION ### 2.1 Metrics Computed **File**: `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/performance.rs` ```rust PerformanceMetrics { total_return: f64, // Line 135-144 annualized_return: f64, // Line 236-245 sharpe_ratio: f64, // Line 253-254, 479-502 sortino_ratio: f64, // Line 257, 506-537 max_drawdown: f64, // Line 260, 540-560 volatility: f64, // Line 253 win_rate: f64, // Line 152-161 profit_factor: f64, // Line 163-182 // ... more metrics } ``` ### 2.2 Sharpe Ratio Implementation **File**: `performance.rs`, Lines 479-502 ```rust fn calculate_volatility_and_sharpe(&self, returns: &[f64], duration_years: f64) -> (f64, f64) { if returns.is_empty() || duration_years <= 0.0 { return (0.0, 0.0); } let mean_return = returns.iter().sum::() / returns.len() as f64; let variance = returns.iter() .map(|r| (r - mean_return).powi(2)) .sum::() / returns.len() as f64; let volatility = variance.sqrt(); let annualized_volatility = volatility * (252.0_f64).sqrt(); // 252 trading days let excess_return = mean_return - self.config.risk_free_rate / 252.0; // Daily risk-free rate let sharpe_ratio = if annualized_volatility > 0.0 { excess_return * (252.0_f64).sqrt() / annualized_volatility } else { 0.0 }; (annualized_volatility, sharpe_ratio) } ``` **Key Points**: - Standard formula: (Return - Risk-Free Rate) / Volatility - Annualized using 252 trading days - Risk-free rate from config - Applied to trade-level returns ### 2.3 Drawdown Calculation **File**: `performance.rs`, Lines 540-560 ```rust fn calculate_max_drawdown(&self, trades: &[BacktestTrade], initial_capital: f64) -> (f64, f64) { let mut running_equity = initial_capital; let mut peak_equity = initial_capital; let mut max_drawdown = 0.0; for trade in trades { running_equity += trade.pnl.to_f64().unwrap_or(0.0); if running_equity > peak_equity { peak_equity = running_equity; } let current_drawdown = (peak_equity - running_equity) / peak_equity; if current_drawdown > max_drawdown { max_drawdown = current_drawdown; } } (max_drawdown, max_drawdown_duration) } ``` **Calculation**: - Tracks running portfolio equity after each trade - Tracks peak equity - Drawdown = (Peak - Current) / Peak - Returns maximum drawdown as percentage ### 2.4 Win Rate Tracking **File**: `performance.rs`, Lines 146-161 ```rust let winning_trades: Vec<&BacktestTrade> = trades.iter().filter(|t| t.pnl > Decimal::ZERO).collect(); let losing_trades: Vec<&BacktestTrade> = trades.iter().filter(|t| t.pnl < Decimal::ZERO).collect(); let win_rate = if trades.is_empty() { 0.0 } else { let result = (winning_trades.len() as f64 / trades.len() as f64) * 100.0; if !result.is_finite() { 0.0 } else { result } }; ``` **Calculation**: (Winning Trades) / (Total Trades) * 100% ### 2.5 PnL Calculation **File**: `strategy_engine.rs`, Lines 183-298 Per-trade PnL: ```rust let proceeds = quantity * adjusted_price - commission; let cost_basis = position.avg_price * quantity; let pnl = proceeds - cost_basis; ``` Cumulative: Sum of all trade PnLs --- ## 3. DBN INTEGRATION ### 3.1 Data Loading **File**: `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/dbn_data_source.rs` ``` DBN File → DbnDataSource → MarketData struct └─ Lines 41-58: strategy_engine.rs ``` **Performance**: 0.70ms for 1,674 bars (14x faster than 10ms target) **Automatic Price Correction**: 96.4% spike reduction - Fixes bars encoded with 7 decimal places instead of 9 - Context-aware anomaly detection ### 3.2 Market Data Structure ```rust pub struct MarketData { pub symbol: String, pub timestamp: DateTime, pub open: Decimal, pub high: Decimal, pub low: Decimal, pub close: Decimal, pub volume: Decimal, pub timeframe: TimeFrame, } ``` **Problem**: Only OHLCV data - no alternative bars (dollar, volume, run, tick, imbalance) --- ## 4. ML STRATEGY INTEGRATION ### 4.1 SharedMLStrategy Usage **File**: `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/ml_strategy_engine.rs` ```rust impl MLPoweredStrategy { pub fn new(name: String, lookback_periods: usize) -> Self { let min_confidence_threshold = 0.6; let strategy = Arc::new(SharedMLStrategy::new(lookback_periods, min_confidence_threshold)); // ... } pub async fn get_ensemble_prediction(&mut self, market_data: &MarketData) -> Result> { let price = market_data.close.to_f64().unwrap_or(0.0); let volume = market_data.volume.to_f64().unwrap_or(0.0); let timestamp = market_data.timestamp; let common_predictions = self.strategy.get_ensemble_prediction(price, volume, timestamp).await?; // ... } } ``` **Status**: ✅ Uses SharedMLStrategy (ONE SINGLE SYSTEM) **BUT**: Backtesting doesn't validate predictions against actual outcomes! - Lines 473-486 (ml_strategy_engine.rs): ```rust if let Some(prev_price) = previous_price { let current_price = data_point.close.to_f64().unwrap_or(prev_price); let actual_return = (current_price - prev_price) / prev_price; ml_strategy.validate_predictions(&predictions, actual_return).await; } ``` - ⚠️ **PROBLEM**: Real trades are NOT generated, so no performance feedback loop! ### 4.2 Model Performance Tracking **Structure**: MLModelPerformance (Lines 39-59, ml_strategy_engine.rs) ```rust pub struct MLModelPerformance { pub model_id: String, pub total_predictions: u64, pub correct_predictions: u64, pub avg_latency_us: f64, pub avg_confidence: f64, pub accuracy_percentage: f64, pub returns: Vec, pub sharpe_ratio: f64, pub max_drawdown: f64, } ``` **Gap**: Predictions validated but NOT applied to trading decisions! --- ## 5. CRITICAL GAPS FOR WAVE C INTEGRATION ### 5.1 What's Missing | Feature | Status | Location | Gap | |---------|--------|----------|-----| | Alternative Bars | ❌ Not integrated | ml/src/features/alternative_bars.rs | Backtesting uses time-based OHLCV only | | Fractional Differentiation | ❌ Not integrated | Not yet implemented | Needed for stationarity | | Meta-Labeling | ❌ Not integrated | ml/src/labeling/meta_labeling_engine.rs | No precision improvement mechanism | | Barrier Optimization | ❌ Partially tested | ml/src/features/barrier_optimization.rs | Not used in backtesting strategies | | Dollar Bars | ❌ Not integrated | ml/src/features/alternative_bars.rs | Would reduce noise vs. time-bars | | Volume Bars | ❌ Not integrated | ml/src/features/alternative_bars.rs | Better for market regimes | | Run Bars | ❌ Not integrated | ml/src/features/alternative_bars.rs | Detects directional persistence | ### 5.2 Data Flow for Wave C Integration ``` Current (Isolated): DBN Time-Bars → StrategyEngine → Simplified Features (8) → Trade Signals ↓ Performance Metrics (disconnected from ML) Needed (Wave C): DBN OHLCV ↓ Alternative Bars (dollar/volume/run/tick/imbalance) ↓ Fractional Differentiation (d=0.5 for stationarity) ↓ UnifiedFeatureExtractor (256 features + 18 technical indicators) ↓ Meta-Labeling Engine (primary labels from barriers, secondary from ML) ↓ StrategyEngine with full feature vectors ↓ Performance Validation with actual vs. predicted ``` ### 5.3 Feature Extraction Integration Points **Location 1: StrategyEngine** (strategy_engine.rs, Line 311) ```rust feature_extractor: Arc, ``` - **Status**: Initialized but never called - **Action**: Replace with actual feature extraction calls **Location 2: MLStrategyEngine** (ml_strategy_engine.rs, Lines 74-172) ```rust pub fn extract_features(&mut self, market_data: &MarketData) -> Vec { ``` - **Status**: Local 8-feature extraction - **Action**: Delegate to UnifiedFeatureExtractor (256 features) + alternative bars **Location 3: NewsAwareStrategy** (strategy_engine.rs, Line 462-464) ```rust // In reality, the strategy would use the UnifiedFeatureExtractor ``` - **Status**: TODO comment - **Action**: Implement proper feature extraction --- ## 6. CURRENT TEST COVERAGE ### 6.1 Strategy Tests **File**: `services/backtesting_service/tests/strategy_engine_tests.rs` - Tests: MA crossover, buy-and-hold, basic execution - **Gap**: No tests for feature extraction or Wave C features ### 6.2 ML Strategy Tests **File**: `services/backtesting_service/tests/ml_strategy_backtest_test.rs` - Tests: ML strategy initialization and basic execution - **Gap**: No validation of feature vectors or prediction quality ### 6.3 Performance Metrics Tests **File**: `services/backtesting_service/tests/performance_metrics.rs` - Tests: Sharpe calculation, drawdown calculation, win rate - **Gap**: No tests comparing Wave A vs Wave C features ### 6.4 Alternative Bars Tests **Location**: `ml/tests/alternative_bars_integration_test.rs` - Tests: Dollar bars, volume bars, run bars, tick bars, imbalance bars - Status: 19/19 tests passing (100%) - **Gap**: NOT integrated into backtesting service ### 6.5 Barrier Label Tests **Location**: `ml/tests/barrier_label_validation_test.rs` - Tests: Triple barrier labeling accuracy - Status: Tests passing - **Gap**: NOT used in backtesting for strategy signals --- ## 7. RECOMMENDED INTEGRATION APPROACH ### Phase 1: Feature Extraction Consolidation (Week 1) 1. **Update MarketData to support multiple bar types** ```rust pub struct MarketData { pub symbol: String, pub timestamp: DateTime, pub price_point: PricePoint, // NEW: supports OHLCV + bar metadata pub volume: Decimal, pub bar_type: BarType, // NEW: Time, Dollar, Volume, Run, Tick, Imbalance } ``` 2. **Integrate UnifiedFeatureExtractor into StrategyEngine** - Replace 8-feature local extraction with 256-feature UnifiedFeatureExtractor - Add alternative bar conversion layer 3. **Create DbnAlternativeBarsConverter** ```rust pub struct DbnAlternativeBarsConverter { dbn_source: DbnDataSource, alternative_bars: Arc, } impl DbnAlternativeBarsConverter { pub async fn load_dollar_bars(symbol: &str, threshold: f64) -> Vec pub async fn load_volume_bars(symbol: &str, threshold: u64) -> Vec pub async fn load_run_bars(symbol: &str, threshold: i32) -> Vec } ``` ### Phase 2: Strategy Enhancements (Week 2) 1. **Update strategies to use full feature vectors** ```rust impl StrategyExecutor for AdaptiveStrategy { fn execute(&self, market_data: &MarketData, features: &FeatureVector) { // Use 256 features + 18 technical indicators } } ``` 2. **Implement meta-labeling in backtesting** ```rust pub struct MetaLabeledBacktest { base_strategy: Box, meta_labeler: MetaLabelingEngine, } ``` 3. **Add fractional differentiation preprocessing** ```rust pub struct FractionallyDifferencedMarketData { original: Vec, differentiated: Vec>, d_exponent: f64, // 0.0-1.0 } ``` ### Phase 3: Validation & Backtesting (Week 3) 1. **Implement prediction-to-trade mapping** ```rust async fn execute_ml_backtest(&self, context: &BacktestContext) { // Generate features // Get ML predictions // Generate signals with confidence thresholds // Execute trades // Validate predictions vs actual returns // Persist performance metrics } ``` 2. **Add Wave A/B/C comparison suite** ```rust pub struct FeatureEngineeringComparison { wave_a_results: BacktestResult, // 18 indicators wave_b_results: BacktestResult, // + alternative bars wave_c_results: BacktestResult, // + fractional diff + meta-labels } ``` 3. **Create comprehensive test suite** - Unit tests for each feature type - Integration tests for backtesting pipeline - E2E tests for full feature→trade→metrics flow --- ## 8. CURRENT PERFORMANCE ### 8.1 Backtesting Performance | Metric | Value | Target | Status | |--------|-------|--------|--------| | DBN Load Time | 0.70ms | <10ms | ✅ 14x better | | Execution Speed | <5s | <5s | ✅ Acceptable | | Memory Usage | <100MB | <1GB | ✅ Excellent | | Feature Extraction | 2μs/bar | <100μs | ✅ 50x better | ### 8.2 Current Test Results ``` Backtesting Service Tests: 19/19 (100%) ML Models: 584/584 (100%) Alternative Bars: 19/19 (100%) Barrier Labeling: Tests passing Meta-Labeling: Tests passing ``` --- ## 9. IMPLEMENTATION CHECKLIST FOR WAVE C - [ ] Create DbnAlternativeBarsConverter - [ ] Update MarketData struct for bar type support - [ ] Integrate UnifiedFeatureExtractor into StrategyEngine - [ ] Add fractional differentiation layer - [ ] Implement meta-labeling in backtesting - [ ] Create feature comparison utilities - [ ] Add comprehensive test suite (50+ tests) - [ ] Update performance metrics for feature-level analysis - [ ] Document feature extraction pipeline - [ ] Validate against real data (ES.FUT, NQ.FUT, ZN.FUT) - [ ] Generate comparison reports (Wave A vs B vs C) --- ## 10. KEY FILES SUMMARY | File | Purpose | Status | |------|---------|--------| | strategy_engine.rs | Strategy execution | ⚠️ Features initialized but unused | | ml_strategy_engine.rs | ML strategy wrapper | ⚠️ Local 8-feature extractor (outdated) | | performance.rs | Metrics calculation | ✅ Comprehensive (Sharpe, drawdown, etc.) | | dbn_data_source.rs | DBN loading | ✅ Production-ready (0.70ms) | | unified_feature_extractor.rs | 256-feature extraction | ❌ Not integrated into backtesting | | alternative_bars.rs | Alternative sampling | ❌ Tested but not used | | meta_labeling_engine.rs | Precision improvement | ❌ Tested but not integrated | | barrier_optimization.rs | Triple barrier tuning | ⚠️ Tested, not used in backtesting |