foxhunt/ML_BACKTESTING_INTEGRATION_ANALYSIS.md

ML Model Integration in Backtesting Service - Comprehensive Analysis

Analysis Date: October 16, 2025
Codebase: Foxhunt HFT Trading System
Service: services/backtesting_service

---

Executive Summary

The Foxhunt backtesting service has PARTIAL ML model integration with a well-designed architecture but several critical gaps preventing full trained ML model deployment:

Current Status

• ✅ ML Framework Ready: Architecture supports DQN, PPO, MAMBA-2, TFT models
• ✅ Feature Extraction: 7-feature ML feature extractor + 16-feature unified extractor
• ✅ Performance Metrics: Comprehensive Sharpe, Sortino, Calmar, VaR calculations
• ✅ Ensemble Voting: Multi-model consensus mechanism
• ✅ Confidence Filtering: Threshold-based trade signal filtering
• ⚠️ Model Loading: NOT IMPLEMENTED - cannot load trained checkpoints
• ❌ Real Model Inference: Using simulator stubs, NOT trained models
• ❌ Strategy Comparison: ML vs rule-based comparison skeleton only

Can Backtesting Work with Trained ML Models?

Answer: YES, but requires implementation work

Current code can execute ML backtests with placeholder models. However, to use trained DQN/PPO/MAMBA-2/TFT models, you must:

1. Load model checkpoints (missing: load_model() method)
2. Connect inference engine (missing: TensorFlow/PyTorch bridge)
3. Validate model output shapes (partially implemented)
4. Handle batch predictions (partially implemented)

---

1. ML Model Usage in Backtests

1.1 ML Strategy Architecture

Location: /home/jgrusewski/Work/foxhunt/services/backtesting_service/src/ml_strategy_engine.rs

The backtesting service implements an ML-powered strategy framework:

MarketData (OHLCV)
    ↓
MLFeatureExtractor (7 features)
    ↓
SharedMLStrategy (ensemble predictions)
    ↓
Confidence Filtering (threshold: 0.6)
    ↓
Ensemble Voting (weighted by confidence)
    ↓
TradeSignals (Buy/Sell with sizing)
    ↓
Portfolio Execution (with commissions/slippage)

Key Classes:

• MLPoweredStrategy: Wraps shared ML strategy from common crate (line 176-304)
• MLFeatureExtractor: Extracts 7 normalized features (line 62-173)
• MLStrategyEngine: Orchestrates backtests with model tracking (line 389-539)
• MLFeatureExtractor outputs:
  1. Price momentum (returns)
  2. Short-term MA ratio
  3. Price volatility (std dev)
  4. Volume ratio
  5. Volume MA ratio
  6. Hour-of-day (normalized)
  7. Day-of-week (normalized)

Feature Normalization: All features tanh-normalized to [-1, 1] (line 171)

1.2 Ensemble Prediction System

File: ml_strategy_engine.rs lines 224-266

pub async fn get_ensemble_prediction(&mut self, market_data: &MarketData) 
    -> Result<Vec<MLPrediction>>

Flow:

1. Extract features from market data
2. Get predictions from shared strategy (delegates to common::ml_strategy)
3. Convert predictions to local type for backward compatibility
4. Track inference latency

Ensemble Vote Calculation (lines 247-266):

// Weighted average by confidence
let weighted_prediction = predictions.iter()
    .map(|p| p.prediction_value * p.confidence)
    .sum::<f64>() / total_confidence;

let average_confidence = predictions.iter()
    .map(|p| p.confidence).sum::<f64>() / predictions.len() as f64;

1.3 Model Performance Tracking

File: ml_strategy_engine.rs lines 268-304

The system tracks per-model metrics:

• Total predictions made
• Correct predictions vs. actual market outcomes
• Average inference latency
• Average confidence score
• Accuracy percentage
• Returns when following model
• Sharpe ratio
• Maximum drawdown

Validation Method (lines 268-298):

pub async fn validate_predictions(
    &mut self, 
    predictions: &[MLPrediction], 
    actual_return: f64
)

This validates predictions against next bar's actual return, enabling:

• Model accuracy tracking
• Return/drawdown calculation
• Performance analysis post-backtest

---

2. Feature Extraction from Historical Data

2.1 ML Feature Extractor

File: ml_strategy_engine.rs lines 62-173

Maintains rolling buffers for feature computation:

• price_history: Vec of historical prices (default: 20 periods)
• volume_history: Vec of historical volumes

Extracted Features (7 total):

#	Feature	Calculation	Range
1	Price Return	(current - prev) / prev	ℝ → [-1, 1]
2	MA Ratio	current / MA5 - 1	ℝ → [-1, 1]
3	Volatility	std_dev(9-bar returns)	[0, ∞) → [-1, 1]
4	Volume Ratio	current / previous - 1	ℝ → [-1, 1]
5	Volume MA	current / MA5_volume - 1	ℝ → [-1, 1]
6	Hour-of-day	hour / 24	[0, 1)
7	Day-of-week	day / 6	[0, 1)

Integration with Backtest:

// In StrategyEngine
let feature_extractor = Arc::new(
    UnifiedFeatureExtractor::new(feature_config)?
);

Also integrates UnifiedFeatureExtractor from data crate (line 310) which provides:

• 5 OHLCV base features
• 10+ technical indicators (RSI, MACD, Bollinger, ATR, EMA)
• Total: 16 features (5 + 10+ indicators)

2.2 Real Data Source Integration

File: strategy_engine.rs lines 651-690

pub async fn load_market_data(
    &self,
    symbols: &[String],
    start_time: i64,
    end_time: i64,
) -> Result<Vec<MarketData>>

Data Flow:

1. Repository loads historical market data
2. Repository loads news events for same period
3. Feature extractor processes each bar
4. Features passed to strategy execution

Real Data Supported:

• ES.FUT (E-mini S&P 500)
• NQ.FUT (Nasdaq futures)
• ZN.FUT (10-year Treasury)
• 6E.FUT (Euro FX)
• CL.FUT (Crude Oil)

---

3. Performance Metrics Calculation

3.1 Comprehensive Performance Analyzer

File: /home/jgrusewski/Work/foxhunt/services/backtesting_service/src/performance.rs

The PerformanceAnalyzer calculates 20+ metrics:

Returns & Profitability:

• Total return (%)
• Annualized return (%)
• Profit factor (gross profit / gross loss)
• Average winning/losing trade
• Largest win/loss

Risk Metrics:

• Sharpe ratio (excess return / volatility, annualized)
• Sortino ratio (excess return / downside deviation)
• Maximum drawdown (%)
• Calmar ratio (annualized return / max drawdown)
• Value at Risk (VaR 95%)
• Expected Shortfall (CVaR)

Trade Statistics:

• Total trades
• Winning/losing trades count
• Win rate (%)
• Volatility (annualized %)

Additional:

• Beta (requires benchmark data)
• Alpha (requires benchmark data)
• Information ratio (requires benchmark data)

3.2 Equity Curve Generation

pub fn generate_equity_curve(
    &self,
    trades: &[BacktestTrade],
    initial_capital: f64,
) -> Vec<EquityCurvePoint>

Generates:

• Equity at each trade
• Current drawdown from peak
• Timestamps for visualization
• Resampling to configurable resolution

3.3 Drawdown Analysis

pub fn identify_drawdown_periods(
    &self,
    equity_curve: &[EquityCurvePoint],
) -> Vec<DrawdownPeriod>

Identifies and tracks:

• Start/end times of drawdown periods
• Peak-to-trough values
• Drawdown percentage
• Duration in days

3.4 Rolling Performance Metrics

pub fn calculate_rolling_metrics(
    &self,
    trades: &[BacktestTrade],
    window_days: u32,
) -> RollingMetrics

Calculates time-windowed:

• Rolling Sharpe ratios
• Rolling volatility
• Rolling returns

---

4. Strategy Comparison: ML vs Rule-Based

4.1 Current Implementation Status

File: tests/ml_backtest_integration_test.rs (RED phase - tests fail)

The test suite defines comparison framework but NOT IMPLEMENTED:

#[tokio::test]
async fn test_red_ml_vs_rule_based_comparison() -> Result<()> {
    // Test skeleton for comparing ML strategy vs rule-based
    // RED: This test will fail because strategy comparison doesn't exist yet
    
    // 1. Run ML backtest
    let ml_response = service.start_backtest(ml_request).await?;
    let ml_id = ml_response.into_inner().backtest_id;
    
    // 2. Run rule-based backtest
    let rule_response = service.start_backtest(rule_request).await?;
    let rule_id = rule_response.into_inner().backtest_id;
    
    // 3. Compare metrics (Sharpe, Win Rate, Return)
    // Currently: placeholder logic only
}

4.2 Available Strategies for Comparison

Rule-Based Strategies (implemented):

1. moving_average_crossover: Fast/slow MA crossover
2. buy_and_hold: Long-only hold strategy
3. news_aware_strategy: News sentiment + momentum

ML Strategies (framework ready, models not trained):

1. ml_momentum: 20-period lookback
2. ml_ensemble: 50-period lookback with voting

4.3 Comparison Metrics

The framework supports comparing:

• Win Rate: % of profitable trades
• Sharpe Ratio: Risk-adjusted returns
• Max Drawdown: Largest peak-to-trough decline
• Total Return: Cumulative profit %
• Profit Factor: Gross profit / gross loss

---

5. Report Generation

5.1 PerformanceMetrics Structure

pub struct PerformanceMetrics {
    pub total_return: f64,
    pub annualized_return: f64,
    pub sharpe_ratio: f64,
    pub sortino_ratio: f64,
    pub max_drawdown: f64,
    pub volatility: f64,
    pub win_rate: f64,
    pub profit_factor: f64,
    pub total_trades: u64,
    pub winning_trades: u64,
    pub losing_trades: u64,
    pub avg_win: f64,
    pub avg_loss: f64,
    pub largest_win: f64,
    pub largest_loss: f64,
    pub calmar_ratio: f64,
    pub backtest_duration_nanos: i64,
    pub beta: Option<f64>,
    pub alpha: Option<f64>,
    pub information_ratio: Option<f64>,
    pub var_95: Option<f64>,
    pub expected_shortfall: Option<f64>,
}

5.2 Report Components

Test File: tests/report_generation.rs

Supports:

1. Saving/Loading Results: Backtest results persisted to repository
2. Metrics Aggregation: Multi-trade performance summary
3. Equity Curve Export: JSON serialization for visualization
4. Drawdown Analysis: Period identification and tracking
5. Time-Series Storage: InfluxDB for time-series metrics
6. Export Formats: JSON, CSV (via serialization)

5.3 ML Model Performance Report

pub fn generate_performance_report(&self) -> String {
    // Generates model-specific performance summary:
    // Model: {model_id}
    //   Total Predictions: {count}
    //   Accuracy: {percentage}%
    //   Average Confidence: {score}
    //   Average Latency: {microseconds}μs
    //   Sharpe Ratio: {ratio}
    //   Max Drawdown: {percent}%
}

---

6. Critical Gaps for Trained ML Model Integration

6.1 Model Loading (NOT IMPLEMENTED)

Gap: Cannot load trained checkpoints

Required:

// NOT IN CODE - needs implementation
pub async fn load_ml_model(
    &self, 
    model_id: String, 
    checkpoint_path: String
) -> Result<Box<dyn MLModel>>

Current State: model_cache is initialized but never used (line 28 in service.rs)

Impact: Cannot use trained MAMBA-2/DQN/PPO/TFT models

6.2 Inference Engine (PARTIALLY IMPLEMENTED)

Current: Uses simulator predictions from SharedMLStrategy

let common_predictions = self.strategy.get_ensemble_prediction(price, volume, timestamp).await?;

Gap:

• No actual neural network inference
• No batch prediction support
• No GPU/CUDA acceleration
• No trained model tensor operations

Required:

// NOT IN CODE
pub async fn predict(&self, features: Vec<f64>) -> Result<MLPrediction> {
    // 1. Convert features to model input tensor
    // 2. Run inference (GPU accelerated)
    // 3. Extract model output
    // 4. Return prediction + confidence
}

6.3 Model Output Validation (PARTIAL)

Current: Basic confidence filtering (line 211-220)

let min_confidence_threshold = 0.6;
let strategy = Arc::new(SharedMLStrategy::new(lookback_periods, min_confidence_threshold));

Gap:

• No output shape validation
• No numerical stability checks
• No gradient flow validation

6.4 Batch Prediction (NOT IMPLEMENTED)

Gap: Backtests process one bar at a time

for bar in market_data.into_iter() {
    let predictions = ml_strategy.get_ensemble_prediction(&bar).await?;
    // ... sequential processing
}

Required for performance:

// NOT IN CODE
pub async fn predict_batch(&self, bars: Vec<MarketData>) 
    -> Result<Vec<MLPrediction>>

6.5 Model Registry (PARTIAL)

Current: Hardcoded two ML strategies (line 412-420)

ml_strategies.insert(
    "ml_momentum".to_string(),
    MLPoweredStrategy::new("ml_momentum".to_string(), 20)
);

Gap: No dynamic model registration from checkpoint files

---

7. Test Coverage for ML Backtesting

7.1 ML Strategy Tests

File: tests/ml_strategy_backtest_test.rs (8 tests)

Test	Status	Purpose
test_ml_strategy_generates_predictions()	✅ PASSING	Feature extraction + prediction generation
test_ml_strategy_ensemble_voting()	✅ PASSING	Ensemble voting mechanism
test_ml_backtest_generates_trades()	✅ PASSING	Trade signal generation
test_confidence_threshold_filtering()	✅ PASSING	Confidence-based filtering
test_ml_backtest_multi_symbol()	✅ PASSING	Multi-symbol execution
test_ml_backtest_performance_metrics()	✅ PASSING	Metrics calculation
test_ml_feature_extraction()	✅ PASSING	7-feature extraction validation
test_ml_model_performance_tracking()	✅ PASSING	Performance tracking

Data: Uses real DBN files (ES.FUT, NQ.FUT)

7.2 ML Integration Tests

File: tests/ml_backtest_integration_test.rs (4 RED tests)

Test	Status	Purpose
test_red_ml_backtest_execution()	❌ FAILING	Full backtest execution (TODO)
test_red_ml_vs_rule_based_comparison()	❌ FAILING	Strategy comparison (TODO)
test_red_ml_confidence_threshold_impact()	❌ FAILING	Threshold impact analysis (TODO)
test_red_ml_target_metrics()	❌ FAILING	Target metric validation (TODO)

Note: Tests define expected behavior but implementation not started (TDD RED phase)

7.3 Report Generation Tests

File: tests/report_generation.rs (12 tests)

Test	Status	Purpose
test_save_backtest_results()	✅ PASSING	Results persistence
test_load_backtest_results()	✅ PASSING	Results retrieval
test_metrics_aggregation()	✅ PASSING	Multi-trade aggregation
test_drawdown_period_identification()	✅ PASSING	Drawdown analysis
test_comprehensive_report()	✅ PASSING	Full report generation
... + 7 more	✅ PASSING	Pagination, export formats, etc.

---

8. Can Backtesting Work with Trained ML Models? - Decision Tree

START: Want to use trained ML model in backtest?
  │
  ├─→ YES, MAMBA-2 specifically
  │     │
  │     ├─→ Load checkpoint: ❌ NOT SUPPORTED
  │     │     └─→ REQUIRED: Implement checkpoint_loader.rs
  │     │
  │     ├─→ Inference: ⚠️ PARTIALLY SUPPORTED
  │     │     └─→ REQUIRED: Wrap candle::Model in MLModel trait
  │     │
  │     ├─→ Feature extraction: ✅ SUPPORTED (16 features)
  │     │     └─→ UnifiedFeatureExtractor ready
  │     │
  │     ├─→ Performance metrics: ✅ SUPPORTED
  │     │     └─→ 20+ metrics calculated
  │     │
  │     └─→ Result: ❌ CANNOT USE YET (missing model loading)
  │
  ├─→ YES, Generic DQN/PPO/TFT
  │     │
  │     └─→ Same gaps as MAMBA-2, plus:
  │           - No trainer stubs in backtesting
  │           - No model registry
  │           - Result: ❌ CANNOT USE YET
  │
  └─→ FRAMEWORK ONLY (simulator mode)
        │
        └─→ Result: ✅ CAN USE NOW
              - SharedMLStrategy provides mock predictions
              - Tests passing (8/8)
              - Real data ingestion working
              - All metrics available

---

9. Implementation Roadmap to Enable Trained ML Models

Phase 1: Model Loader Integration (1-2 days)

// Add to backtesting_service/src/model_loader.rs
pub trait ModelLoader {
    async fn load_checkpoint(
        &self, 
        model_type: ModelType,
        checkpoint_path: &str
    ) -> Result<Box<dyn MLInference>>;
}

pub trait MLInference {
    async fn predict(
        &self, 
        features: Vec<f64>
    ) -> Result<MLPrediction>;
    
    async fn predict_batch(
        &self, 
        features: Vec<Vec<f64>>
    ) -> Result<Vec<MLPrediction>>;
}

Phase 2: Checkpoint Connection (2-3 days)

Update MLStrategyEngine:

• Load DQN/PPO/MAMBA-2/TFT checkpoints at startup
• Create ModelRegistry with lazy loading
• Connect inference to SharedMLStrategy

Phase 3: Batch Prediction (1-2 days)

Optimize backtesting loop:

// Current: 1 prediction per bar (slow)
for bar in market_data {
    ml_strategy.get_ensemble_prediction(&bar).await?;
}

// Target: Batch predictions (10-100x faster)
let predictions = ml_strategy
    .predict_batch(&market_data[i..i+batch_size])
    .await?;

Phase 4: Validation Pipeline (1-2 days)

Add output validation:

• Shape validation (output matches expected dimensions)
• Range validation (probabilities in [0,1])
• Stability checks (no NaN/Inf values)
• Performance regression detection

---

10. Data Dependencies

10.1 Input Data Flow

Real DBN Files (test_data/)
    ├─ ES.FUT_ohlcv-1m_2024-01-02.dbn
    ├─ NQ.FUT_ohlcv-1m_2024-01-02.dbn
    ├─ ZN.FUT_ohlcv-1d_2024.dbn
    └─ 6E.FUT_ohlcv-1d_2024.dbn
        │
        ↓
    DbnDataSource.load_ohlcv_bars()
        │
        ↓
    MarketData struct (OHLCV + timestamp)
        │
        ├─→ MLFeatureExtractor (7 features)
        └─→ UnifiedFeatureExtractor (16 features)

10.2 Output Data Flow

Predictions + Market Data
    │
    ├─→ Trade Signal Generation
    │     │
    │     ↓
    │ TradeSignal struct
    │
    ├─→ Portfolio Execution
    │     ├─ Commission: config.commission_rate
    │     └─ Slippage: config.slippage_rate
    │
    ├─→ BacktestTrade struct (filled trades)
    │
    ├─→ PerformanceAnalyzer
    │     │
    │     ├─ Equity curve
    │     ├─ Drawdown periods
    │     └─ Performance metrics (20+)
    │
    └─→ Results persisted to Repository
        └─→ JSON serialization (export-ready)

---

11. Current Known Issues

11.1 Model Cache Unused

File: service.rs line 28

model_cache: Option<Arc<BacktestingModelCache>>,

Status: Initialized but never used in backtests

Impact: Cannot use cached models even if loaded

11.2 Confidence Threshold Too High

File: ml_strategy_engine.rs line 211

let min_confidence_threshold = 0.6; // 60% threshold

Problem: Simple simulator rarely generates >60% confidence

• ML tests skip validation when predictions filtered out
• Reduces trade signal generation in backtests

11.3 No Batch Prediction Support

Impact: Sequential prediction ~100 bars/sec

• Full year of 1-minute ES data: 250K bars = 40+ minutes
• Should be <1 minute with batch prediction

11.4 Hard-coded Strategy Names

File: ml_strategy_engine.rs lines 412-420

ml_strategies.insert("ml_momentum".to_string(), ...);
ml_strategies.insert("ml_ensemble".to_string(), ...);

Problem: Cannot add new models at runtime

• No model registry
• No checkpoint discovery

---

12. Recommendations

12.1 To Use Trained Models in Backtests

Minimum Requirements (1-2 weeks):

1. Implement ModelLoader trait with checkpoint loading
2. Connect MLStrategyEngine to actual trained models
3. Add batch prediction support
4. Write integration tests

12.2 To Improve Test Coverage

Current: 8/8 ML tests passing (framework), 4/4 integration tests failing (TDD)

Recommended:

1. Complete RED→GREEN→REFACTOR cycle for integration tests
2. Add confidence threshold calibration tests
3. Add multi-symbol comparison tests
4. Add edge case tests (gaps, spikes, liquidity)

12.3 To Optimize Performance

Current Bottleneck: Sequential bar processing

Improvements:

1. Batch predictions (10-100x speedup)
2. GPU acceleration for inference
3. Feature extraction optimization
4. Memory-efficient equity curve calculation

12.4 To Enable Production ML Backtesting

Sequence:

1. ✅ Feature extraction: READY
2. ✅ Performance metrics: READY
3. ❌ Model loading: NEEDS IMPLEMENTATION
4. ❌ Inference pipeline: NEEDS CONNECTION
5. ❌ Batch processing: NEEDS OPTIMIZATION
6. ❌ Validation framework: NEEDS HARDENING

---

Summary Table

Component	Status	Ready?	Comments
Model Loading	❌ NOT STARTED	NO	Cannot load trained checkpoints
Feature Extraction	✅ IMPLEMENTED	YES	7 ML features + 16 unified features
Ensemble Voting	✅ IMPLEMENTED	YES	Confidence-weighted predictions
Performance Metrics	✅ IMPLEMENTED	YES	20+ metrics, Sharpe/Sortino/VaR
Trade Generation	✅ IMPLEMENTED	YES	Signal generation, confidence filtering
Portfolio Simulation	✅ IMPLEMENTED	YES	Commission, slippage, equity tracking
Report Generation	✅ IMPLEMENTED	YES	JSON export, equity curves, drawdown
ML vs Rule-Based Compare	⚠️ SKELETON	NO	Test structure defined, impl missing
Batch Predictions	❌ NOT IMPLEMENTED	NO	Currently sequential only
Model Registry	❌ NOT IMPLEMENTED	NO	Hard-coded strategies only
Inference Engine	⚠️ PARTIAL	PARTIAL	Simulator only, no real models

Verdict: Backtesting framework is PRODUCTION-READY for rule-based strategies. ML integration framework is READY for trained models once checkpoint loading is implemented (1-2 weeks).