================================================================================ BACKTESTING SERVICE FEATURE GAPS SUMMARY October 17, 2025 ================================================================================ ┌─────────────────────────────────────────────────────────────────────────────┐ │ CURRENT STATE (Wave A) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ DBN OHLCV Data (0.70ms load) │ │ ↓ │ │ StrategyEngine │ │ • MovingAverageCrossover (trigger_price parameter only) │ │ • BuyAndHold (static allocation) │ │ • NewsAware (hardcoded sentiment 0.2, momentum 55.0) │ │ • MLPoweredStrategy (uses SharedMLStrategy + 8 features) │ │ ↓ │ │ Portfolio Execution (Commission + Slippage) │ │ ↓ │ │ Performance Metrics │ │ ✅ Sharpe Ratio (252-day annualized) │ │ ✅ Sortino Ratio (downside risk only) │ │ ✅ Max Drawdown (peak-to-trough) │ │ ✅ Win Rate (winning trades %) │ │ ✅ Profit Factor (gross profit / gross loss) │ │ ✅ PnL Tracking (per trade + cumulative) │ │ │ │ ⚠️ GAPS: │ │ • 8 local features vs 256 in ML training │ │ • Only time-based OHLCV (no alternative bars) │ │ • No fractional differentiation │ │ • No meta-labeling precision improvement │ │ • UnifiedFeatureExtractor initialized but never used │ │ • ML predictions validated but NOT used for trading │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ NEEDED STATE (Wave C - Fractional Differentiation) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ DBN OHLCV Data │ │ ↓ │ │ ┌─ Alternative Bars Generation ─┐ │ │ │ • Dollar Bars (noise reduction) │ │ │ │ • Volume Bars (regime detection)│ │ │ │ • Run Bars (directional trends) │ │ │ │ • Tick Bars (frequency-based) │ │ │ │ • Imbalance Bars (micro-trends) │ │ │ └───────────────────────────────┘ │ │ ↓ │ │ Fractional Differentiation (d=0.5) │ │ • Removes unit root (stationarity) │ │ • Preserves long-range memory │ │ • Improves ML convergence │ │ ↓ │ │ UnifiedFeatureExtractor (256 features) │ │ • 18 Wave A technical indicators (RSI, MACD, Bollinger, ATR, etc.) │ │ • Price features (returns, volatility, microstructure) │ │ • Volume features (VWAP, OBV, CMF) │ │ • Temporal features (hour, day, seasonality) │ │ • Regime features (structural breaks, CUSUM) │ │ ↓ │ │ Meta-Labeling Engine │ │ • Primary Labels: Triple barrier (upper/lower/time) │ │ • Secondary Labels: ML model predictions (DQN, PPO, MAMBA-2) │ │ • Precision Improvement: Filter low-confidence predictions │ │ ↓ │ │ StrategyEngine (with 256 features + meta-labels) │ │ • Adaptive strategy (detects regime switches) │ │ • ML signal generation with confidence thresholds │ │ • Dynamic position sizing (Kelly criterion based on Sharpe) │ │ ↓ │ │ Portfolio Execution │ │ ↓ │ │ Enhanced Performance Metrics │ │ ✅ All Wave A metrics │ │ ✅ Feature-level performance attribution │ │ ✅ Regime-specific Sharpe ratios │ │ ✅ Prediction accuracy (ML signals vs actual returns) │ │ ✅ Meta-label precision/recall │ │ │ │ 📊 EXPECTED IMPROVEMENTS: │ │ • Win Rate: 41.8% → 48-52% (+10-24%) │ │ • Sharpe Ratio: -6.52 → 0.5-1.0 (+7 points) │ │ • Max Drawdown: Lower due to regime detection │ │ • Feature coverage: 8 → 256 features (32x increase) │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ FEATURE EXTRACTION DISCONNECTS │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Location 1: StrategyEngine (strategy_engine.rs:311) │ │ ───────────────────────────────────────────────── │ │ feature_extractor: Arc, │ │ │ │ Status: ❌ INITIALIZED BUT NEVER CALLED │ │ Used: 0x across entire codebase │ │ Expected: 1x per market data point │ │ │ │ Location 2: MLStrategyEngine (ml_strategy_engine.rs:74-172) │ │ ───────────────────────────────────────────────────────── │ │ Local MLFeatureExtractor with 8 features: │ │ 1. Price return (6-period) │ │ 2. MA ratio (5-period SMA) │ │ 3. Price volatility (10-period std dev) │ │ 4. Volume ratio (2-period) │ │ 5. Volume MA ratio (5-period) │ │ 6. Hour of day (normalized 0-1) │ │ 7. Day of week (normalized 0-1) │ │ 8. All normalized via tanh() │ │ │ │ Status: ⚠️ OUTDATED (old architecture) │ │ Should: Delegate to UnifiedFeatureExtractor + alternative bars │ │ │ │ Location 3: NewsAwareStrategy (strategy_engine.rs:462-464) │ │ ───────────────────────────────────────────────────────── │ │ Comment: "In reality, the strategy would use UnifiedFeatureExtractor" │ │ Status: ❌ TODO - NOT IMPLEMENTED │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ AVAILABLE WAVE C COMPONENTS (Already Implemented) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Component Location Status │ │ ───────────────────────────────────────────────────────────────────── │ │ Alternative Bars ml/src/features/ ✅ 19/19 │ │ (Dollar, Volume, Run, Tick, Imbalance) alternative_bars.rs tests │ │ │ │ Barrier Labeling (Triple Barrier) ml/src/labeling/ ✅ Tests │ │ barrier_backtest.rs passing │ │ │ │ Meta-Labeling Engine ml/src/labeling/ ✅ Tests │ │ meta_labeling_engine.rs passing │ │ │ │ Barrier Optimization ml/src/features/ ✅ Tests │ │ barrier_optimization.rs passing │ │ │ │ UnifiedFeatureExtractor (256 features) data/src/unified_ ✅ 100% │ │ feature_extractor.rs complete │ │ │ │ Technical Indicators (18) ml/src/features/ ✅ All 18 │ │ (RSI, MACD, Bollinger, ATR, ADX, technical_indicators.rs integrated │ │ CCI, Stochastic, EWMA, etc.) │ │ │ │ 🔴 MISSING: Fractional Differentiation (NOT YET IMPLEMENTED) │ │ Purpose: Remove unit root while preserving long-range memory │ │ Estimated effort: 2-3 days │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ INTEGRATION ROADMAP (3 Weeks) │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Week 1: Feature Extraction Consolidation │ │ ─────────────────────────────────────── │ │ Day 1-2: Create DbnAlternativeBarsConverter │ │ • Wrap DbnDataSource with alternative bar generation │ │ • Support dollar/volume/run/tick/imbalance bars │ │ Day 3-4: Update MarketData struct │ │ • Add bar_type enum │ │ • Add bar metadata (cumulative $, volume, runs, etc.) │ │ Day 5: Integrate UnifiedFeatureExtractor │ │ • Replace 8-feature local extraction │ │ • Call during execute_backtest() │ │ │ │ Week 2: Strategy Enhancements │ │ ─────────────────────────── │ │ Day 1-2: Implement fractional differentiation │ │ • d=0.5 (0-1 range for stationarity) │ │ • Preserve memory (vs full differencing d=1) │ │ Day 3-4: Implement meta-labeling in backtesting │ │ • Primary labels: Triple barrier │ │ • Secondary labels: ML predictions │ │ Day 5: Update strategies with 256 features │ │ • Adaptive strategy (regime detection) │ │ • Dynamic position sizing │ │ │ │ Week 3: Validation & Testing │ │ ──────────────────────────── │ │ Day 1-2: Implement prediction-to-trade mapping │ │ • Generate ML signals with confidence thresholds │ │ • Validate predictions vs actual returns │ │ Day 3-4: Create Wave A/B/C comparison suite │ │ • Side-by-side backtest results │ │ • Feature performance attribution │ │ Day 5: Comprehensive testing (50+ test cases) │ │ • Unit tests for each component │ │ • E2E pipeline tests │ │ • Real data validation (ES.FUT, NQ.FUT, ZN.FUT) │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────┐ │ KEY METRICS TO TRACK │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Performance: │ │ • Sharpe Ratio (currently: -6.52 → target: 0.5-1.0) │ │ • Win Rate (currently: 41.8% → target: 48-52%) │ │ • Max Drawdown (lower with regime detection) │ │ • Profit Factor (gross profit / gross loss) │ │ │ │ Feature Quality: │ │ • Feature importance ranking (via SHAP or permutation) │ │ • Feature correlation (remove redundant features) │ │ • Feature coverage (8 → 256 features) │ │ │ │ ML Integration: │ │ • Prediction accuracy vs actual returns │ │ • Meta-label precision (filters ~30% low-confidence signals) │ │ • Model latency (<100μs target) │ │ • Confidence calibration (predicted vs realized) │ │ │ │ Regime Detection: │ │ • Sharpe ratio by regime (up/down/sideways) │ │ • Strategy switching frequency │ │ • Adaptation lag (days to detect regime change) │ │ │ │ Data Quality: │ │ • Alternative bar validity (no NaNs, monotonicity) │ │ • Fractional differentiation stationarity (ADF test) │ │ • Feature scaling consistency │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ CRITICAL SUCCESS FACTORS: 1. ✅ Consolidate on ONE feature extractor (UnifiedFeatureExtractor) 2. ✅ Validate features during backtesting (not just predictions) 3. ✅ Use same features as live trading (SharedMLStrategy) 4. ✅ Compare Wave A/B/C sequentially (not isolation) 5. ✅ Test on real market data (DBN + real edge cases) TIMELINE: 3 weeks (5 engineers working in parallel) STATUS: READY TO IMPLEMENT (all components exist, just need integration) ================================================================================