foxhunt/docs/archive/backtesting/BACKTESTING_FEATURES_INVESTIGATION.md

Backtesting Service Feature Integration Investigation

Executive Summary

The Backtesting Service currently uses a simplified feature extraction pipeline that is NOT integrated with Wave C features (alternative bars, fractional differentiation, meta-labeling, barrier optimization). Features are extracted at strategy runtime but NOT persisted or validated against actual market outcomes during backtesting. This creates a critical gap between:

1. Live trading - Uses SharedMLStrategy with full ML inference
2. Backtesting - Uses simplified local feature extraction with static parameters
3. ML training - Uses 256-feature vectors from UnifiedFeatureExtractor in data crate

---

1. BACKTESTING ARCHITECTURE

1.1 Core Components

Backtesting Service Flow:
┌─────────────────────┐
│  DBN Data Source    │  ← Loads OHLCV bars from real DBN files (0.70ms)
└──────────┬──────────┘
           │
           ▼
┌──────────────────────────────────────┐
│  StrategyEngine::execute_backtest()  │
├──────────────────────────────────────┤
│ 1. Load market data (via repository) │
│ 2. For each market data point:       │
│    - Call strategy.execute()         │
│    - Generate TradeSignals           │
│    - Execute trades in portfolio     │
│ 3. Calculate performance metrics     │
└──────────┬──────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│  PerformanceAnalyzer                 │
├──────────────────────────────────────┤
│ - Sharpe Ratio                       │
│ - Drawdown Analysis                  │
│ - Win Rate                           │
│ - PnL Calculation                    │
└──────────────────────────────────────┘

Key File: /home/jgrusewski/Work/foxhunt/services/backtesting_service/src/strategy_engine.rs

1.2 Available Strategies

1. MovingAverageCrossoverStrategy (Lines 349-404)

   • Simplistic trigger_price parameter
   • No feature extraction
   • No access to UnifiedFeatureExtractor

2. BuyAndHoldStrategy (Lines 406-447)

   • Static allocation-based
   • No strategy logic or features

3. NewsAwareStrategy (Lines 449-526)

   • Simulated sentiment/momentum (hardcoded values: 0.2, 55.0)
   • Would benefit from features but doesn't actually extract them
   • Comment at line 462-464: "This is a simplified example - in reality, the strategy would use the UnifiedFeatureExtractor"

4. MLPoweredStrategy (ml_strategy_engine.rs)

   • Uses SharedMLStrategy from common crate
   • Delegates to ML models (DQN, PPO, MAMBA-2, TFT)
   • But still has local MLFeatureExtractor as fallback

1.3 Feature Extraction - DISCONNECTED

Current Location 1: StrategyEngine

• Lines 549-554: Creates UnifiedFeatureExtractor with default config
• Lines 685-689: NOT ACTUALLY USED - just initialized but never called
• Comment at line 686: "In production, this would properly convert NewsEvent to the format expected by UnifiedFeatureExtractor"

Current Location 2: MLStrategyEngine.MLFeatureExtractor

• Lines 72-173 (ml_strategy_engine.rs): Local feature extractor
• Extracts 8 basic features:
  1. Price return
  2. Short-term MA ratio
  3. Price volatility
  4. Volume ratio
  5. Volume MA ratio
  6. Hour of day
  7. Day of week
  8. All normalized via tanh() normalization

Problem: These 8 features are extracted locally WITHOUT integration with:

• 18 Wave A technical indicators (RSI, MACD, Bollinger, ATR, ADX, CCI, Stochastic, etc.)
• UnifiedFeatureExtractor (256 features in data crate)
• Alternative bars (Wave B)
• Fractional differentiation (Wave C)
• Meta-labeling (Wave C)

---

2. PERFORMANCE METRICS CALCULATION

2.1 Metrics Computed

File: /home/jgrusewski/Work/foxhunt/services/backtesting_service/src/performance.rs

PerformanceMetrics {
    total_return: f64,           // Line 135-144
    annualized_return: f64,      // Line 236-245
    sharpe_ratio: f64,           // Line 253-254, 479-502
    sortino_ratio: f64,          // Line 257, 506-537
    max_drawdown: f64,           // Line 260, 540-560
    volatility: f64,             // Line 253
    win_rate: f64,               // Line 152-161
    profit_factor: f64,          // Line 163-182
    // ... more metrics
}

2.2 Sharpe Ratio Implementation

File: performance.rs, Lines 479-502

fn calculate_volatility_and_sharpe(&self, returns: &[f64], duration_years: f64) -> (f64, f64) {
    if returns.is_empty() || duration_years <= 0.0 {
        return (0.0, 0.0);
    }

    let mean_return = returns.iter().sum::<f64>() / returns.len() as f64;
    let variance = returns.iter()
        .map(|r| (r - mean_return).powi(2))
        .sum::<f64>() / returns.len() as f64;

    let volatility = variance.sqrt();
    let annualized_volatility = volatility * (252.0_f64).sqrt(); // 252 trading days

    let excess_return = mean_return - self.config.risk_free_rate / 252.0; // Daily risk-free rate
    let sharpe_ratio = if annualized_volatility > 0.0 {
        excess_return * (252.0_f64).sqrt() / annualized_volatility
    } else {
        0.0
    };

    (annualized_volatility, sharpe_ratio)
}

Key Points:

• Standard formula: (Return - Risk-Free Rate) / Volatility
• Annualized using 252 trading days
• Risk-free rate from config
• Applied to trade-level returns

2.3 Drawdown Calculation

File: performance.rs, Lines 540-560

fn calculate_max_drawdown(&self, trades: &[BacktestTrade], initial_capital: f64) -> (f64, f64) {
    let mut running_equity = initial_capital;
    let mut peak_equity = initial_capital;
    let mut max_drawdown = 0.0;

    for trade in trades {
        running_equity += trade.pnl.to_f64().unwrap_or(0.0);

        if running_equity > peak_equity {
            peak_equity = running_equity;
        }

        let current_drawdown = (peak_equity - running_equity) / peak_equity;
        if current_drawdown > max_drawdown {
            max_drawdown = current_drawdown;
        }
    }

    (max_drawdown, max_drawdown_duration)
}

Calculation:

• Tracks running portfolio equity after each trade
• Tracks peak equity
• Drawdown = (Peak - Current) / Peak
• Returns maximum drawdown as percentage

2.4 Win Rate Tracking

File: performance.rs, Lines 146-161

let winning_trades: Vec<&BacktestTrade> =
    trades.iter().filter(|t| t.pnl > Decimal::ZERO).collect();

let losing_trades: Vec<&BacktestTrade> =
    trades.iter().filter(|t| t.pnl < Decimal::ZERO).collect();

let win_rate = if trades.is_empty() {
    0.0
} else {
    let result = (winning_trades.len() as f64 / trades.len() as f64) * 100.0;
    if !result.is_finite() {
        0.0
    } else {
        result
    }
};

Calculation: (Winning Trades) / (Total Trades) * 100%

2.5 PnL Calculation

File: strategy_engine.rs, Lines 183-298

Per-trade PnL:

let proceeds = quantity * adjusted_price - commission;
let cost_basis = position.avg_price * quantity;
let pnl = proceeds - cost_basis;

Cumulative: Sum of all trade PnLs

---

3. DBN INTEGRATION

3.1 Data Loading

File: /home/jgrusewski/Work/foxhunt/services/backtesting_service/src/dbn_data_source.rs

DBN File → DbnDataSource → MarketData struct
                         └─ Lines 41-58: strategy_engine.rs

Performance: 0.70ms for 1,674 bars (14x faster than 10ms target)

Automatic Price Correction: 96.4% spike reduction

• Fixes bars encoded with 7 decimal places instead of 9
• Context-aware anomaly detection

3.2 Market Data Structure

pub struct MarketData {
    pub symbol: String,
    pub timestamp: DateTime<Utc>,
    pub open: Decimal,
    pub high: Decimal,
    pub low: Decimal,
    pub close: Decimal,
    pub volume: Decimal,
    pub timeframe: TimeFrame,
}

Problem: Only OHLCV data - no alternative bars (dollar, volume, run, tick, imbalance)

---

4. ML STRATEGY INTEGRATION

4.1 SharedMLStrategy Usage

File: /home/jgrusewski/Work/foxhunt/services/backtesting_service/src/ml_strategy_engine.rs

impl MLPoweredStrategy {
    pub fn new(name: String, lookback_periods: usize) -> Self {
        let min_confidence_threshold = 0.6;
        let strategy = Arc::new(SharedMLStrategy::new(lookback_periods, min_confidence_threshold));
        // ...
    }

    pub async fn get_ensemble_prediction(&mut self, market_data: &MarketData) -> Result<Vec<MLPrediction>> {
        let price = market_data.close.to_f64().unwrap_or(0.0);
        let volume = market_data.volume.to_f64().unwrap_or(0.0);
        let timestamp = market_data.timestamp;

        let common_predictions = self.strategy.get_ensemble_prediction(price, volume, timestamp).await?;
        // ...
    }
}

Status: ✅ Uses SharedMLStrategy (ONE SINGLE SYSTEM)

BUT: Backtesting doesn't validate predictions against actual outcomes!

• Lines 473-486 (ml_strategy_engine.rs):

  if let Some(prev_price) = previous_price {
      let current_price = data_point.close.to_f64().unwrap_or(prev_price);
      let actual_return = (current_price - prev_price) / prev_price;
      ml_strategy.validate_predictions(&predictions, actual_return).await;
  }
  

• ⚠️ PROBLEM: Real trades are NOT generated, so no performance feedback loop!

4.2 Model Performance Tracking

Structure: MLModelPerformance (Lines 39-59, ml_strategy_engine.rs)

pub struct MLModelPerformance {
    pub model_id: String,
    pub total_predictions: u64,
    pub correct_predictions: u64,
    pub avg_latency_us: f64,
    pub avg_confidence: f64,
    pub accuracy_percentage: f64,
    pub returns: Vec<f64>,
    pub sharpe_ratio: f64,
    pub max_drawdown: f64,
}

Gap: Predictions validated but NOT applied to trading decisions!

---

5. CRITICAL GAPS FOR WAVE C INTEGRATION

5.1 What's Missing

Feature	Status	Location	Gap
Alternative Bars	❌ Not integrated	ml/src/features/alternative_bars.rs	Backtesting uses time-based OHLCV only
Fractional Differentiation	❌ Not integrated	Not yet implemented	Needed for stationarity
Meta-Labeling	❌ Not integrated	ml/src/labeling/meta_labeling_engine.rs	No precision improvement mechanism
Barrier Optimization	❌ Partially tested	ml/src/features/barrier_optimization.rs	Not used in backtesting strategies
Dollar Bars	❌ Not integrated	ml/src/features/alternative_bars.rs	Would reduce noise vs. time-bars
Volume Bars	❌ Not integrated	ml/src/features/alternative_bars.rs	Better for market regimes
Run Bars	❌ Not integrated	ml/src/features/alternative_bars.rs	Detects directional persistence

5.2 Data Flow for Wave C Integration

Current (Isolated):
DBN Time-Bars → StrategyEngine → Simplified Features (8) → Trade Signals
                                                              ↓
                                                        Performance Metrics
                                                        (disconnected from ML)

Needed (Wave C):
DBN OHLCV
    ↓
Alternative Bars (dollar/volume/run/tick/imbalance)
    ↓
Fractional Differentiation (d=0.5 for stationarity)
    ↓
UnifiedFeatureExtractor (256 features + 18 technical indicators)
    ↓
Meta-Labeling Engine (primary labels from barriers, secondary from ML)
    ↓
StrategyEngine with full feature vectors
    ↓
Performance Validation with actual vs. predicted

5.3 Feature Extraction Integration Points

Location 1: StrategyEngine (strategy_engine.rs, Line 311)

feature_extractor: Arc<UnifiedFeatureExtractor>,

• Status: Initialized but never called
• Action: Replace with actual feature extraction calls

Location 2: MLStrategyEngine (ml_strategy_engine.rs, Lines 74-172)

pub fn extract_features(&mut self, market_data: &MarketData) -> Vec<f64> {

• Status: Local 8-feature extraction
• Action: Delegate to UnifiedFeatureExtractor (256 features) + alternative bars

Location 3: NewsAwareStrategy (strategy_engine.rs, Line 462-464)

// In reality, the strategy would use the UnifiedFeatureExtractor

• Status: TODO comment
• Action: Implement proper feature extraction

---

6. CURRENT TEST COVERAGE

6.1 Strategy Tests

File: services/backtesting_service/tests/strategy_engine_tests.rs

• Tests: MA crossover, buy-and-hold, basic execution
• Gap: No tests for feature extraction or Wave C features

6.2 ML Strategy Tests

File: services/backtesting_service/tests/ml_strategy_backtest_test.rs

• Tests: ML strategy initialization and basic execution
• Gap: No validation of feature vectors or prediction quality

6.3 Performance Metrics Tests

File: services/backtesting_service/tests/performance_metrics.rs

• Tests: Sharpe calculation, drawdown calculation, win rate
• Gap: No tests comparing Wave A vs Wave C features

6.4 Alternative Bars Tests

Location: ml/tests/alternative_bars_integration_test.rs

• Tests: Dollar bars, volume bars, run bars, tick bars, imbalance bars
• Status: 19/19 tests passing (100%)
• Gap: NOT integrated into backtesting service

6.5 Barrier Label Tests

Location: ml/tests/barrier_label_validation_test.rs

• Tests: Triple barrier labeling accuracy
• Status: Tests passing
• Gap: NOT used in backtesting for strategy signals

---

7. RECOMMENDED INTEGRATION APPROACH

Phase 1: Feature Extraction Consolidation (Week 1)

1. Update MarketData to support multiple bar types

   pub struct MarketData {
       pub symbol: String,
       pub timestamp: DateTime<Utc>,
       pub price_point: PricePoint,  // NEW: supports OHLCV + bar metadata
       pub volume: Decimal,
       pub bar_type: BarType,  // NEW: Time, Dollar, Volume, Run, Tick, Imbalance
   }
   

2. Integrate UnifiedFeatureExtractor into StrategyEngine

   • Replace 8-feature local extraction with 256-feature UnifiedFeatureExtractor
   • Add alternative bar conversion layer

3. Create DbnAlternativeBarsConverter

   pub struct DbnAlternativeBarsConverter {
       dbn_source: DbnDataSource,
       alternative_bars: Arc<AlternativeBars>,
   }
   
   impl DbnAlternativeBarsConverter {
       pub async fn load_dollar_bars(symbol: &str, threshold: f64) -> Vec<MarketData>
       pub async fn load_volume_bars(symbol: &str, threshold: u64) -> Vec<MarketData>
       pub async fn load_run_bars(symbol: &str, threshold: i32) -> Vec<MarketData>
   }
   

Phase 2: Strategy Enhancements (Week 2)

1. Update strategies to use full feature vectors

   impl StrategyExecutor for AdaptiveStrategy {
       fn execute(&self, market_data: &MarketData, features: &FeatureVector) {
           // Use 256 features + 18 technical indicators
       }
   }
   

2. Implement meta-labeling in backtesting

   pub struct MetaLabeledBacktest {
       base_strategy: Box<dyn StrategyExecutor>,
       meta_labeler: MetaLabelingEngine,
   }
   

3. Add fractional differentiation preprocessing

   pub struct FractionallyDifferencedMarketData {
       original: Vec<MarketData>,
       differentiated: Vec<Vec<f64>>,
       d_exponent: f64,  // 0.0-1.0
   }
   

Phase 3: Validation & Backtesting (Week 3)

1. Implement prediction-to-trade mapping

   async fn execute_ml_backtest(&self, context: &BacktestContext) {
       // Generate features
       // Get ML predictions
       // Generate signals with confidence thresholds
       // Execute trades
       // Validate predictions vs actual returns
       // Persist performance metrics
   }
   

2. Add Wave A/B/C comparison suite

   pub struct FeatureEngineeringComparison {
       wave_a_results: BacktestResult,  // 18 indicators
       wave_b_results: BacktestResult,  // + alternative bars
       wave_c_results: BacktestResult,  // + fractional diff + meta-labels
   }
   

3. Create comprehensive test suite

   • Unit tests for each feature type
   • Integration tests for backtesting pipeline
   • E2E tests for full feature→trade→metrics flow

---

8. CURRENT PERFORMANCE

8.1 Backtesting Performance

Metric	Value	Target	Status
DBN Load Time	0.70ms	<10ms	✅ 14x better
Execution Speed	<5s	<5s	✅ Acceptable
Memory Usage	<100MB	<1GB	✅ Excellent
Feature Extraction	2μs/bar	<100μs	✅ 50x better

8.2 Current Test Results

Backtesting Service Tests: 19/19 (100%)
ML Models: 584/584 (100%)
Alternative Bars: 19/19 (100%)
Barrier Labeling: Tests passing
Meta-Labeling: Tests passing

---

9. IMPLEMENTATION CHECKLIST FOR WAVE C

• Create DbnAlternativeBarsConverter
• Update MarketData struct for bar type support
• Integrate UnifiedFeatureExtractor into StrategyEngine
• Add fractional differentiation layer
• Implement meta-labeling in backtesting
• Create feature comparison utilities
• Add comprehensive test suite (50+ tests)
• Update performance metrics for feature-level analysis
• Document feature extraction pipeline
• Validate against real data (ES.FUT, NQ.FUT, ZN.FUT)
• Generate comparison reports (Wave A vs B vs C)

---

10. KEY FILES SUMMARY

File	Purpose	Status
strategy_engine.rs	Strategy execution	⚠️ Features initialized but unused
ml_strategy_engine.rs	ML strategy wrapper	⚠️ Local 8-feature extractor (outdated)
performance.rs	Metrics calculation	✅ Comprehensive (Sharpe, drawdown, etc.)
dbn_data_source.rs	DBN loading	✅ Production-ready (0.70ms)
unified_feature_extractor.rs	256-feature extraction	❌ Not integrated into backtesting
alternative_bars.rs	Alternative sampling	❌ Tested but not used
meta_labeling_engine.rs	Precision improvement	❌ Tested but not integrated
barrier_optimization.rs	Triple barrier tuning	⚠️ Tested, not used in backtesting