foxhunt/BACKTESTING_FEATURES_INVESTIGATION.md

# Backtesting Service Feature Integration Investigation

## Executive Summary

The Backtesting Service currently uses a **simplified feature extraction pipeline** that is NOT integrated with Wave C features (alternative bars, fractional differentiation, meta-labeling, barrier optimization). Features are extracted at strategy runtime but NOT persisted or validated against actual market outcomes during backtesting. This creates a critical gap between:

1. **Live trading** - Uses `SharedMLStrategy` with full ML inference
2. **Backtesting** - Uses simplified local feature extraction with static parameters
3. **ML training** - Uses 256-feature vectors from `UnifiedFeatureExtractor` in data crate

---

## 1. BACKTESTING ARCHITECTURE

### 1.1 Core Components

```
Backtesting Service Flow:
┌─────────────────────┐
│  DBN Data Source    │  ← Loads OHLCV bars from real DBN files (0.70ms)
└──────────┬──────────┘
           │
           ▼
┌──────────────────────────────────────┐
│  StrategyEngine::execute_backtest()  │
├──────────────────────────────────────┤
│ 1. Load market data (via repository) │
│ 2. For each market data point:       │
│    - Call strategy.execute()         │
│    - Generate TradeSignals           │
│    - Execute trades in portfolio     │
│ 3. Calculate performance metrics     │
└──────────┬──────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│  PerformanceAnalyzer                 │
├──────────────────────────────────────┤
│ - Sharpe Ratio                       │
│ - Drawdown Analysis                  │
│ - Win Rate                           │
│ - PnL Calculation                    │
└──────────────────────────────────────┘
```

**Key File**: `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/strategy_engine.rs`

### 1.2 Available Strategies

1. **MovingAverageCrossoverStrategy** (Lines 349-404)
   - Simplistic trigger_price parameter
   - No feature extraction
   - No access to UnifiedFeatureExtractor

2. **BuyAndHoldStrategy** (Lines 406-447)
   - Static allocation-based
   - No strategy logic or features

3. **NewsAwareStrategy** (Lines 449-526)
   - Simulated sentiment/momentum (hardcoded values: 0.2, 55.0)
   - Would benefit from features but doesn't actually extract them
   - Comment at line 462-464: "This is a simplified example - in reality, the strategy would use the UnifiedFeatureExtractor"

4. **MLPoweredStrategy** (ml_strategy_engine.rs)
   - Uses SharedMLStrategy from common crate
   - Delegates to ML models (DQN, PPO, MAMBA-2, TFT)
   - But still has local MLFeatureExtractor as fallback

### 1.3 Feature Extraction - DISCONNECTED

**Current Location 1: StrategyEngine**
- Lines 549-554: Creates UnifiedFeatureExtractor with default config
- Lines 685-689: **NOT ACTUALLY USED** - just initialized but never called
- Comment at line 686: "In production, this would properly convert NewsEvent to the format expected by UnifiedFeatureExtractor"

**Current Location 2: MLStrategyEngine.MLFeatureExtractor**
- Lines 72-173 (ml_strategy_engine.rs): Local feature extractor
- Extracts 8 basic features:
  1. Price return
  2. Short-term MA ratio
  3. Price volatility
  4. Volume ratio
  5. Volume MA ratio
  6. Hour of day
  7. Day of week
  8. All normalized via tanh() normalization

**Problem**: These 8 features are extracted locally WITHOUT integration with:
- 18 Wave A technical indicators (RSI, MACD, Bollinger, ATR, ADX, CCI, Stochastic, etc.)
- UnifiedFeatureExtractor (256 features in data crate)
- Alternative bars (Wave B)
- Fractional differentiation (Wave C)
- Meta-labeling (Wave C)

---

## 2. PERFORMANCE METRICS CALCULATION

### 2.1 Metrics Computed

**File**: `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/performance.rs`

```rust
PerformanceMetrics {
    total_return: f64,           // Line 135-144
    annualized_return: f64,      // Line 236-245
    sharpe_ratio: f64,           // Line 253-254, 479-502
    sortino_ratio: f64,          // Line 257, 506-537
    max_drawdown: f64,           // Line 260, 540-560
    volatility: f64,             // Line 253
    win_rate: f64,               // Line 152-161
    profit_factor: f64,          // Line 163-182
    // ... more metrics
}
```

### 2.2 Sharpe Ratio Implementation

**File**: `performance.rs`, Lines 479-502

```rust
fn calculate_volatility_and_sharpe(&self, returns: &[f64], duration_years: f64) -> (f64, f64) {
    if returns.is_empty() || duration_years <= 0.0 {
        return (0.0, 0.0);
    }

    let mean_return = returns.iter().sum::<f64>() / returns.len() as f64;
    let variance = returns.iter()
        .map(|r| (r - mean_return).powi(2))
        .sum::<f64>() / returns.len() as f64;

    let volatility = variance.sqrt();
    let annualized_volatility = volatility * (252.0_f64).sqrt(); // 252 trading days

    let excess_return = mean_return - self.config.risk_free_rate / 252.0; // Daily risk-free rate
    let sharpe_ratio = if annualized_volatility > 0.0 {
        excess_return * (252.0_f64).sqrt() / annualized_volatility
    } else {
        0.0
    };

    (annualized_volatility, sharpe_ratio)
}
```

**Key Points**:
- Standard formula: (Return - Risk-Free Rate) / Volatility
- Annualized using 252 trading days
- Risk-free rate from config
- Applied to trade-level returns

### 2.3 Drawdown Calculation

**File**: `performance.rs`, Lines 540-560

```rust
fn calculate_max_drawdown(&self, trades: &[BacktestTrade], initial_capital: f64) -> (f64, f64) {
    let mut running_equity = initial_capital;
    let mut peak_equity = initial_capital;
    let mut max_drawdown = 0.0;

    for trade in trades {
        running_equity += trade.pnl.to_f64().unwrap_or(0.0);

        if running_equity > peak_equity {
            peak_equity = running_equity;
        }

        let current_drawdown = (peak_equity - running_equity) / peak_equity;
        if current_drawdown > max_drawdown {
            max_drawdown = current_drawdown;
        }
    }

    (max_drawdown, max_drawdown_duration)
}
```

**Calculation**:
- Tracks running portfolio equity after each trade
- Tracks peak equity
- Drawdown = (Peak - Current) / Peak
- Returns maximum drawdown as percentage

### 2.4 Win Rate Tracking

**File**: `performance.rs`, Lines 146-161

```rust
let winning_trades: Vec<&BacktestTrade> =
    trades.iter().filter(|t| t.pnl > Decimal::ZERO).collect();

let losing_trades: Vec<&BacktestTrade> =
    trades.iter().filter(|t| t.pnl < Decimal::ZERO).collect();

let win_rate = if trades.is_empty() {
    0.0
} else {
    let result = (winning_trades.len() as f64 / trades.len() as f64) * 100.0;
    if !result.is_finite() {
        0.0
    } else {
        result
    }
};
```

**Calculation**: (Winning Trades) / (Total Trades) * 100%

### 2.5 PnL Calculation

**File**: `strategy_engine.rs`, Lines 183-298

Per-trade PnL:
```rust
let proceeds = quantity * adjusted_price - commission;
let cost_basis = position.avg_price * quantity;
let pnl = proceeds - cost_basis;
```

Cumulative: Sum of all trade PnLs

---

## 3. DBN INTEGRATION

### 3.1 Data Loading

**File**: `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/dbn_data_source.rs`

```
DBN File → DbnDataSource → MarketData struct
                         └─ Lines 41-58: strategy_engine.rs
```

**Performance**: 0.70ms for 1,674 bars (14x faster than 10ms target)

**Automatic Price Correction**: 96.4% spike reduction
- Fixes bars encoded with 7 decimal places instead of 9
- Context-aware anomaly detection

### 3.2 Market Data Structure

```rust
pub struct MarketData {
    pub symbol: String,
    pub timestamp: DateTime<Utc>,
    pub open: Decimal,
    pub high: Decimal,
    pub low: Decimal,
    pub close: Decimal,
    pub volume: Decimal,
    pub timeframe: TimeFrame,
}
```

**Problem**: Only OHLCV data - no alternative bars (dollar, volume, run, tick, imbalance)

---

## 4. ML STRATEGY INTEGRATION

### 4.1 SharedMLStrategy Usage

**File**: `/home/jgrusewski/Work/foxhunt/services/backtesting_service/src/ml_strategy_engine.rs`

```rust
impl MLPoweredStrategy {
    pub fn new(name: String, lookback_periods: usize) -> Self {
        let min_confidence_threshold = 0.6;
        let strategy = Arc::new(SharedMLStrategy::new(lookback_periods, min_confidence_threshold));
        // ...
    }

    pub async fn get_ensemble_prediction(&mut self, market_data: &MarketData) -> Result<Vec<MLPrediction>> {
        let price = market_data.close.to_f64().unwrap_or(0.0);
        let volume = market_data.volume.to_f64().unwrap_or(0.0);
        let timestamp = market_data.timestamp;

        let common_predictions = self.strategy.get_ensemble_prediction(price, volume, timestamp).await?;
        // ...
    }
}
```

**Status**: ✅ Uses SharedMLStrategy (ONE SINGLE SYSTEM)

**BUT**: Backtesting doesn't validate predictions against actual outcomes!
- Lines 473-486 (ml_strategy_engine.rs):
  ```rust
  if let Some(prev_price) = previous_price {
      let current_price = data_point.close.to_f64().unwrap_or(prev_price);
      let actual_return = (current_price - prev_price) / prev_price;
      ml_strategy.validate_predictions(&predictions, actual_return).await;
  }
  ```
- ⚠️ **PROBLEM**: Real trades are NOT generated, so no performance feedback loop!

### 4.2 Model Performance Tracking

**Structure**: MLModelPerformance (Lines 39-59, ml_strategy_engine.rs)
```rust
pub struct MLModelPerformance {
    pub model_id: String,
    pub total_predictions: u64,
    pub correct_predictions: u64,
    pub avg_latency_us: f64,
    pub avg_confidence: f64,
    pub accuracy_percentage: f64,
    pub returns: Vec<f64>,
    pub sharpe_ratio: f64,
    pub max_drawdown: f64,
}
```

**Gap**: Predictions validated but NOT applied to trading decisions!

---

## 5. CRITICAL GAPS FOR WAVE C INTEGRATION

### 5.1 What's Missing

| Feature | Status | Location | Gap |
|---------|--------|----------|-----|
| Alternative Bars | ❌ Not integrated | ml/src/features/alternative_bars.rs | Backtesting uses time-based OHLCV only |
| Fractional Differentiation | ❌ Not integrated | Not yet implemented | Needed for stationarity |
| Meta-Labeling | ❌ Not integrated | ml/src/labeling/meta_labeling_engine.rs | No precision improvement mechanism |
| Barrier Optimization | ❌ Partially tested | ml/src/features/barrier_optimization.rs | Not used in backtesting strategies |
| Dollar Bars | ❌ Not integrated | ml/src/features/alternative_bars.rs | Would reduce noise vs. time-bars |
| Volume Bars | ❌ Not integrated | ml/src/features/alternative_bars.rs | Better for market regimes |
| Run Bars | ❌ Not integrated | ml/src/features/alternative_bars.rs | Detects directional persistence |

### 5.2 Data Flow for Wave C Integration

```
Current (Isolated):
DBN Time-Bars → StrategyEngine → Simplified Features (8) → Trade Signals
                                                              ↓
                                                        Performance Metrics
                                                        (disconnected from ML)

Needed (Wave C):
DBN OHLCV
    ↓
Alternative Bars (dollar/volume/run/tick/imbalance)
    ↓
Fractional Differentiation (d=0.5 for stationarity)
    ↓
UnifiedFeatureExtractor (256 features + 18 technical indicators)
    ↓
Meta-Labeling Engine (primary labels from barriers, secondary from ML)
    ↓
StrategyEngine with full feature vectors
    ↓
Performance Validation with actual vs. predicted
```

### 5.3 Feature Extraction Integration Points

**Location 1: StrategyEngine** (strategy_engine.rs, Line 311)
```rust
feature_extractor: Arc<UnifiedFeatureExtractor>,
```
- **Status**: Initialized but never called
- **Action**: Replace with actual feature extraction calls

**Location 2: MLStrategyEngine** (ml_strategy_engine.rs, Lines 74-172)
```rust
pub fn extract_features(&mut self, market_data: &MarketData) -> Vec<f64> {
```
- **Status**: Local 8-feature extraction
- **Action**: Delegate to UnifiedFeatureExtractor (256 features) + alternative bars

**Location 3: NewsAwareStrategy** (strategy_engine.rs, Line 462-464)
```rust
// In reality, the strategy would use the UnifiedFeatureExtractor
```
- **Status**: TODO comment
- **Action**: Implement proper feature extraction

---

## 6. CURRENT TEST COVERAGE

### 6.1 Strategy Tests

**File**: `services/backtesting_service/tests/strategy_engine_tests.rs`
- Tests: MA crossover, buy-and-hold, basic execution
- **Gap**: No tests for feature extraction or Wave C features

### 6.2 ML Strategy Tests

**File**: `services/backtesting_service/tests/ml_strategy_backtest_test.rs`
- Tests: ML strategy initialization and basic execution
- **Gap**: No validation of feature vectors or prediction quality

### 6.3 Performance Metrics Tests

**File**: `services/backtesting_service/tests/performance_metrics.rs`
- Tests: Sharpe calculation, drawdown calculation, win rate
- **Gap**: No tests comparing Wave A vs Wave C features

### 6.4 Alternative Bars Tests

**Location**: `ml/tests/alternative_bars_integration_test.rs`
- Tests: Dollar bars, volume bars, run bars, tick bars, imbalance bars
- Status: 19/19 tests passing (100%)
- **Gap**: NOT integrated into backtesting service

### 6.5 Barrier Label Tests

**Location**: `ml/tests/barrier_label_validation_test.rs`
- Tests: Triple barrier labeling accuracy
- Status: Tests passing
- **Gap**: NOT used in backtesting for strategy signals

---

## 7. RECOMMENDED INTEGRATION APPROACH

### Phase 1: Feature Extraction Consolidation (Week 1)

1. **Update MarketData to support multiple bar types**
   ```rust
   pub struct MarketData {
       pub symbol: String,
       pub timestamp: DateTime<Utc>,
       pub price_point: PricePoint,  // NEW: supports OHLCV + bar metadata
       pub volume: Decimal,
       pub bar_type: BarType,  // NEW: Time, Dollar, Volume, Run, Tick, Imbalance
   }
   ```

2. **Integrate UnifiedFeatureExtractor into StrategyEngine**
   - Replace 8-feature local extraction with 256-feature UnifiedFeatureExtractor
   - Add alternative bar conversion layer

3. **Create DbnAlternativeBarsConverter**
   ```rust
   pub struct DbnAlternativeBarsConverter {
       dbn_source: DbnDataSource,
       alternative_bars: Arc<AlternativeBars>,
   }
   
   impl DbnAlternativeBarsConverter {
       pub async fn load_dollar_bars(symbol: &str, threshold: f64) -> Vec<MarketData>
       pub async fn load_volume_bars(symbol: &str, threshold: u64) -> Vec<MarketData>
       pub async fn load_run_bars(symbol: &str, threshold: i32) -> Vec<MarketData>
   }
   ```

### Phase 2: Strategy Enhancements (Week 2)

1. **Update strategies to use full feature vectors**
   ```rust
   impl StrategyExecutor for AdaptiveStrategy {
       fn execute(&self, market_data: &MarketData, features: &FeatureVector) {
           // Use 256 features + 18 technical indicators
       }
   }
   ```

2. **Implement meta-labeling in backtesting**
   ```rust
   pub struct MetaLabeledBacktest {
       base_strategy: Box<dyn StrategyExecutor>,
       meta_labeler: MetaLabelingEngine,
   }
   ```

3. **Add fractional differentiation preprocessing**
   ```rust
   pub struct FractionallyDifferencedMarketData {
       original: Vec<MarketData>,
       differentiated: Vec<Vec<f64>>,
       d_exponent: f64,  // 0.0-1.0
   }
   ```

### Phase 3: Validation & Backtesting (Week 3)

1. **Implement prediction-to-trade mapping**
   ```rust
   async fn execute_ml_backtest(&self, context: &BacktestContext) {
       // Generate features
       // Get ML predictions
       // Generate signals with confidence thresholds
       // Execute trades
       // Validate predictions vs actual returns
       // Persist performance metrics
   }
   ```

2. **Add Wave A/B/C comparison suite**
   ```rust
   pub struct FeatureEngineeringComparison {
       wave_a_results: BacktestResult,  // 18 indicators
       wave_b_results: BacktestResult,  // + alternative bars
       wave_c_results: BacktestResult,  // + fractional diff + meta-labels
   }
   ```

3. **Create comprehensive test suite**
   - Unit tests for each feature type
   - Integration tests for backtesting pipeline
   - E2E tests for full feature→trade→metrics flow

---

## 8. CURRENT PERFORMANCE

### 8.1 Backtesting Performance

| Metric | Value | Target | Status |
|--------|-------|--------|--------|
| DBN Load Time | 0.70ms | <10ms | ✅ 14x better |
| Execution Speed | <5s | <5s | ✅ Acceptable |
| Memory Usage | <100MB | <1GB | ✅ Excellent |
| Feature Extraction | 2μs/bar | <100μs | ✅ 50x better |

### 8.2 Current Test Results

```
Backtesting Service Tests: 19/19 (100%)
ML Models: 584/584 (100%)
Alternative Bars: 19/19 (100%)
Barrier Labeling: Tests passing
Meta-Labeling: Tests passing
```

---

## 9. IMPLEMENTATION CHECKLIST FOR WAVE C

- [ ] Create DbnAlternativeBarsConverter
- [ ] Update MarketData struct for bar type support
- [ ] Integrate UnifiedFeatureExtractor into StrategyEngine
- [ ] Add fractional differentiation layer
- [ ] Implement meta-labeling in backtesting
- [ ] Create feature comparison utilities
- [ ] Add comprehensive test suite (50+ tests)
- [ ] Update performance metrics for feature-level analysis
- [ ] Document feature extraction pipeline
- [ ] Validate against real data (ES.FUT, NQ.FUT, ZN.FUT)
- [ ] Generate comparison reports (Wave A vs B vs C)

---

## 10. KEY FILES SUMMARY

| File | Purpose | Status |
|------|---------|--------|
| strategy_engine.rs | Strategy execution | ⚠️ Features initialized but unused |
| ml_strategy_engine.rs | ML strategy wrapper | ⚠️ Local 8-feature extractor (outdated) |
| performance.rs | Metrics calculation | ✅ Comprehensive (Sharpe, drawdown, etc.) |
| dbn_data_source.rs | DBN loading | ✅ Production-ready (0.70ms) |
| unified_feature_extractor.rs | 256-feature extraction | ❌ Not integrated into backtesting |
| alternative_bars.rs | Alternative sampling | ❌ Tested but not used |
| meta_labeling_engine.rs | Precision improvement | ❌ Tested but not integrated |
| barrier_optimization.rs | Triple barrier tuning | ⚠️ Tested, not used in backtesting |