Files

jgrusewski 30241e858d docs: backtesting vertical slice production readiness design

8-task plan to bridge disconnected backtesting pipeline layers:
DBN parser → feature extraction → ML inference → position tracking → PnL

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-21 15:21:22 +01:00

8.7 KiB

Raw Blame History

Backtesting Vertical Slice — Production Readiness Design

Date: 2026-02-21 Status: Approved Goal: Load .dbn file + trained model checkpoints → run backtest → get real PnL numbers with trade-by-trade history.

Problem Statement

The backtesting pipeline has 8 disconnected layers — each independently functional but no connectors between them. The result: backtesting has never actually run ML inference on real data. Every layer produces output, but nothing flows end-to-end.

Current Broken Flow

.dbn file → [DbnParser] → ProcessedMessage    ✗ NO CONVERTER    → MarketEvent
Parquet   → [ParquetDataLoader]               ✗ NO CONVERTER    → MarketEvent
MarketEvent → [AdaptiveStrategyRunner]        ✗ WRONG FEATURES  → 3-5 dim (models need 51)
Features → [GlobalRegistry.predict_selected]  ✗ EMPTY REGISTRY  → ModelNotFound
Prediction → [generate_signal]                ✓ WORKS           → TradingSignal
Signal → [execute_order]                      ✓ WORKS           → cash update
Position → [PositionTracker]                  ✗ STUB Ok(())     → no tracking
PnL                                           ✗ NEVER UPDATED   → always 0

Design Decisions

1. Data Source: DBN files via DbnParser

The DbnParser in data/src/providers/databento/dbn_parser.rs already correctly parses MBO, MBP1, MBP10, Trade, and OHLCV records from .dbn files. We add a converter layer (ProcessedMessage → MarketEvent) and a replay engine that feeds events into the existing StrategyTester.

Alternative considered: Parquet replay via ParquetDataLoader. Rejected because it adds a conversion step (DBN→Parquet) and ParquetMarketDataEvent has a different schema than MarketEvent.

2. Feature Extraction: Production 51-dim via FeatureExtractor

Replace the local 3-5 feature extractor in strategy_runner.rs with ProductionFeatureExtractorAdapter from ml/src/features/production_adapter.rs, which produces exactly 51 features matching DQN/PPO training expectations.

Alternative considered: UnifiedFeatureExtractor from data/ (50-100+ features). Rejected because ML models were trained on the 51-dim layout from ml::features::extraction::FeatureExtractor. Feature dimension mismatch would produce garbage predictions.

3. Model Loading: Direct safetensors → model constructors

Load .safetensors checkpoints directly into model structs (DQN, PPO, TFT, Mamba2) using existing load_from_safetensors() methods, then wrap in Arc<dyn MLModel> for the global registry.

Alternative considered: S3-based model_loader crate. Rejected for backtesting because it only returns raw bytes with no deserialization, and backtesting should work from local files without S3 dependency.

4. Position Tracking: Fix existing stubs

The PositionTracker struct exists with correct fields but empty method bodies. We implement the actual accounting rather than adding a new tracker, keeping the existing StrategyTester integration intact.

Architecture

.dbn file
    ↓ DbnParser::parse_batch() (exists, data/ crate)
Vec<ProcessedMessage>
    ↓ NEW: dbn_to_market_event() converter (backtesting/ crate)
Vec<MarketEvent>
    ↓ NEW: DbnReplayEngine::start_replay() → mpsc channel
ReplayEvent stream
    ↓ StrategyTester event loop (exists)
AdaptiveStrategyRunner.on_market_event()
    ↓ MODIFIED: uses ProductionFeatureExtractorAdapter (51-dim)
Features { values: Vec<f64> [51], names, timestamp }
    ↓ GlobalRegistry.predict_selected() (exists)
    ↓ NEW: BacktestModelLoader populates registry at startup
Vec<ModelPrediction>
    ↓ generate_signal() + RiskManager (exist)
TradingSignal
    ↓ execute_order() (exists)
    ↓ FIXED: PositionTracker (real accounting + trade recording)
BacktestResult { trades, total_pnl, sharpe, max_drawdown, win_rate }

Components

Task 1: DBN → MarketEvent Converter

File: backtesting/src/dbn_converter.rs

Convert data::providers::databento::dbn_parser::ProcessedMessage variants to trading_engine::types::events::MarketEvent variants:

ProcessedMessage::Trade { price, quantity, timestamp, symbol } → MarketEvent::Trade { .. }
ProcessedMessage::Ohlcv { open, high, low, close, volume, timestamp, symbol } → MarketEvent::Bar { .. }
ProcessedMessage::Quote { bid_price, ask_price, .. } → MarketEvent::OrderBookUpdate { .. }

Must handle: price type conversions (f64 → Decimal), timestamp formats, symbol normalization.

Task 2: DBN Replay Engine

File: backtesting/src/dbn_replay.rs

pub struct DbnReplayEngine {
    events: Vec<MarketEvent>,  // pre-loaded and sorted by timestamp
}

impl DbnReplayEngine {
    pub fn from_dbn_file(path: &Path) -> Result<Self>;
    pub fn start_replay(&self, tx: mpsc::UnboundedSender<ReplayEvent>) -> JoinHandle<()>;
}

Integrates with StrategyTester by implementing the same channel-based feed pattern as MarketReplay.

Task 3: Wire Production Feature Extractor

File: Modify backtesting/src/strategy_runner.rs

Replace the local FeatureExtractor (3-5 features) with ProductionFeatureExtractorAdapter from ml/src/features/production_adapter.rs. The adapter needs a warmup period of 50 bars — during warmup, no predictions are made. After warmup, each market event produces a 51-dim feature vector.

Must handle: the adapter expects OHLCV bars, but on_market_event() receives individual trades. Buffer trades into 1-minute bars before feeding to the extractor.

Task 4: Model Checkpoint Loader

File: backtesting/src/model_loader.rs

pub fn load_model_from_checkpoint(
    model_type: &str,    // "DQN", "PPO", "TFT", "MAMBA-2"
    checkpoint_path: &Path,
    config: &ModelConfig, // per-model config (state_dim, hidden_dims, etc.)
) -> Result<Arc<dyn MLModel>, MLError>;

Uses:

DQN: DQN::new(config) then load_from_safetensors(path)
PPO: PPO::new(config) then load actor weights
TFT: TemporalFusionTransformer::new(config) then load weights
Mamba2: Mamba2SSM::new(config, &device) then load weights

Task 5: Registry Startup

File: backtesting/src/model_loader.rs (same file as Task 4)

pub struct BacktestModelLoader;

impl BacktestModelLoader {
    /// Load models from config and register in global registry
    pub fn load_models(models: &[ModelSpec]) -> Result<()> {
        for spec in models {
            let model = load_model_from_checkpoint(&spec.model_type, &spec.path, &spec.config)?;
            get_global_registry().register(model);
        }
        Ok(())
    }
}

Called by StrategyTester::run_test() before starting the event replay loop.

Task 6: Fix PositionTracker

File: Modify backtesting/src/strategy_tester.rs

pub fn update_position(&mut self, symbol: &str, price: Decimal, quantity: i64) -> Result<()> {
    // Real accounting: track entry price, current price, unrealized PnL
    // Handle position flip (long → short), partial closes, averaging
}

pub fn record_trade(&mut self, trade: TradeRecord) -> Result<()> {
    // Push to self.trades vec with: timestamp, symbol, side, price, quantity, realized_pnl
}

Task 7: Fix PnL Tracking