From 30241e858df70ca1f05dc5e2f00f6d6244b1ffa9 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 21 Feb 2026 15:21:22 +0100 Subject: [PATCH] docs: backtesting vertical slice production readiness design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 8-task plan to bridge disconnected backtesting pipeline layers: DBN parser → feature extraction → ML inference → position tracking → PnL Co-Authored-By: Claude Opus 4.6 --- ...02-21-backtesting-vertical-slice-design.md | 205 ++++++++++++++++++ 1 file changed, 205 insertions(+) create mode 100644 docs/plans/2026-02-21-backtesting-vertical-slice-design.md diff --git a/docs/plans/2026-02-21-backtesting-vertical-slice-design.md b/docs/plans/2026-02-21-backtesting-vertical-slice-design.md new file mode 100644 index 000000000..fd319a2c3 --- /dev/null +++ b/docs/plans/2026-02-21-backtesting-vertical-slice-design.md @@ -0,0 +1,205 @@ +# Backtesting Vertical Slice — Production Readiness Design + +> **Date:** 2026-02-21 +> **Status:** Approved +> **Goal:** Load `.dbn` file + trained model checkpoints → run backtest → get real PnL numbers with trade-by-trade history. + +## Problem Statement + +The backtesting pipeline has 8 disconnected layers — each independently functional but no connectors between them. The result: backtesting has never actually run ML inference on real data. Every layer produces output, but nothing flows end-to-end. + +### Current Broken Flow + +``` +.dbn file → [DbnParser] → ProcessedMessage ✗ NO CONVERTER → MarketEvent +Parquet → [ParquetDataLoader] ✗ NO CONVERTER → MarketEvent +MarketEvent → [AdaptiveStrategyRunner] ✗ WRONG FEATURES → 3-5 dim (models need 51) +Features → [GlobalRegistry.predict_selected] ✗ EMPTY REGISTRY → ModelNotFound +Prediction → [generate_signal] ✓ WORKS → TradingSignal +Signal → [execute_order] ✓ WORKS → cash update +Position → [PositionTracker] ✗ STUB Ok(()) → no tracking +PnL ✗ NEVER UPDATED → always 0 +``` + +## Design Decisions + +### 1. Data Source: DBN files via DbnParser + +The `DbnParser` in `data/src/providers/databento/dbn_parser.rs` already correctly parses MBO, MBP1, MBP10, Trade, and OHLCV records from `.dbn` files. We add a converter layer (`ProcessedMessage` → `MarketEvent`) and a replay engine that feeds events into the existing `StrategyTester`. + +**Alternative considered:** Parquet replay via `ParquetDataLoader`. Rejected because it adds a conversion step (DBN→Parquet) and `ParquetMarketDataEvent` has a different schema than `MarketEvent`. + +### 2. Feature Extraction: Production 51-dim via FeatureExtractor + +Replace the local 3-5 feature extractor in `strategy_runner.rs` with `ProductionFeatureExtractorAdapter` from `ml/src/features/production_adapter.rs`, which produces exactly 51 features matching DQN/PPO training expectations. + +**Alternative considered:** `UnifiedFeatureExtractor` from `data/` (50-100+ features). Rejected because ML models were trained on the 51-dim layout from `ml::features::extraction::FeatureExtractor`. Feature dimension mismatch would produce garbage predictions. + +### 3. Model Loading: Direct safetensors → model constructors + +Load `.safetensors` checkpoints directly into model structs (DQN, PPO, TFT, Mamba2) using existing `load_from_safetensors()` methods, then wrap in `Arc` for the global registry. + +**Alternative considered:** S3-based `model_loader` crate. Rejected for backtesting because it only returns raw bytes with no deserialization, and backtesting should work from local files without S3 dependency. + +### 4. Position Tracking: Fix existing stubs + +The `PositionTracker` struct exists with correct fields but empty method bodies. We implement the actual accounting rather than adding a new tracker, keeping the existing `StrategyTester` integration intact. + +## Architecture + +``` +.dbn file + ↓ DbnParser::parse_batch() (exists, data/ crate) +Vec + ↓ NEW: dbn_to_market_event() converter (backtesting/ crate) +Vec + ↓ NEW: DbnReplayEngine::start_replay() → mpsc channel +ReplayEvent stream + ↓ StrategyTester event loop (exists) +AdaptiveStrategyRunner.on_market_event() + ↓ MODIFIED: uses ProductionFeatureExtractorAdapter (51-dim) +Features { values: Vec [51], names, timestamp } + ↓ GlobalRegistry.predict_selected() (exists) + ↓ NEW: BacktestModelLoader populates registry at startup +Vec + ↓ generate_signal() + RiskManager (exist) +TradingSignal + ↓ execute_order() (exists) + ↓ FIXED: PositionTracker (real accounting + trade recording) +BacktestResult { trades, total_pnl, sharpe, max_drawdown, win_rate } +``` + +## Components + +### Task 1: DBN → MarketEvent Converter + +**File:** `backtesting/src/dbn_converter.rs` + +Convert `data::providers::databento::dbn_parser::ProcessedMessage` variants to `trading_engine::types::events::MarketEvent` variants: +- `ProcessedMessage::Trade { price, quantity, timestamp, symbol }` → `MarketEvent::Trade { .. }` +- `ProcessedMessage::Ohlcv { open, high, low, close, volume, timestamp, symbol }` → `MarketEvent::Bar { .. }` +- `ProcessedMessage::Quote { bid_price, ask_price, .. }` → `MarketEvent::OrderBookUpdate { .. }` + +Must handle: price type conversions (f64 → Decimal), timestamp formats, symbol normalization. + +### Task 2: DBN Replay Engine + +**File:** `backtesting/src/dbn_replay.rs` + +```rust +pub struct DbnReplayEngine { + events: Vec, // pre-loaded and sorted by timestamp +} + +impl DbnReplayEngine { + pub fn from_dbn_file(path: &Path) -> Result; + pub fn start_replay(&self, tx: mpsc::UnboundedSender) -> JoinHandle<()>; +} +``` + +Integrates with `StrategyTester` by implementing the same channel-based feed pattern as `MarketReplay`. + +### Task 3: Wire Production Feature Extractor + +**File:** Modify `backtesting/src/strategy_runner.rs` + +Replace the local `FeatureExtractor` (3-5 features) with `ProductionFeatureExtractorAdapter` from `ml/src/features/production_adapter.rs`. The adapter needs a warmup period of 50 bars — during warmup, no predictions are made. After warmup, each market event produces a 51-dim feature vector. + +Must handle: the adapter expects OHLCV bars, but `on_market_event()` receives individual trades. Buffer trades into 1-minute bars before feeding to the extractor. + +### Task 4: Model Checkpoint Loader + +**File:** `backtesting/src/model_loader.rs` + +```rust +pub fn load_model_from_checkpoint( + model_type: &str, // "DQN", "PPO", "TFT", "MAMBA-2" + checkpoint_path: &Path, + config: &ModelConfig, // per-model config (state_dim, hidden_dims, etc.) +) -> Result, MLError>; +``` + +Uses: +- DQN: `DQN::new(config)` then `load_from_safetensors(path)` +- PPO: `PPO::new(config)` then load actor weights +- TFT: `TemporalFusionTransformer::new(config)` then load weights +- Mamba2: `Mamba2SSM::new(config, &device)` then load weights + +### Task 5: Registry Startup + +**File:** `backtesting/src/model_loader.rs` (same file as Task 4) + +```rust +pub struct BacktestModelLoader; + +impl BacktestModelLoader { + /// Load models from config and register in global registry + pub fn load_models(models: &[ModelSpec]) -> Result<()> { + for spec in models { + let model = load_model_from_checkpoint(&spec.model_type, &spec.path, &spec.config)?; + get_global_registry().register(model); + } + Ok(()) + } +} +``` + +Called by `StrategyTester::run_test()` before starting the event replay loop. + +### Task 6: Fix PositionTracker + +**File:** Modify `backtesting/src/strategy_tester.rs` + +```rust +pub fn update_position(&mut self, symbol: &str, price: Decimal, quantity: i64) -> Result<()> { + // Real accounting: track entry price, current price, unrealized PnL + // Handle position flip (long → short), partial closes, averaging +} + +pub fn record_trade(&mut self, trade: TradeRecord) -> Result<()> { + // Push to self.trades vec with: timestamp, symbol, side, price, quantity, realized_pnl +} +``` + +### Task 7: Fix PnL Tracking + +**File:** Modify `backtesting/src/strategy_runner.rs` + +- Update `PerformanceTracker.total_pnl` on each trade execution +- Increment `winning_trades` counter when realized PnL > 0 +- Populate `StrategyResult.trades` from recorded trade history in `finalize()` +- Compute `performance_timeline` (equity curve snapshots at regular intervals) + +### Task 8: Integration Test + +**File:** `backtesting/tests/dbn_backtest_integration.rs` + +End-to-end test: +1. Create synthetic `.dbn` file with known price pattern (trending up) +2. Create DQN model with random weights, save checkpoint +3. Configure backtest: load model, set risk limits, set position sizing +4. Run backtest over synthetic data +5. Assert: features are 51-dim, predictions are in valid range, trades are recorded, PnL is computed, position tracking is accurate + +## Constraints + +- **Build:** `SQLX_OFFLINE=true cargo check -p backtesting` +- **VRAM:** RTX 3050 Ti 4GB — max 2-3 models loaded simultaneously +- **Clippy:** `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]` +- **No PostgreSQL:** Tests must work offline (no DB dependency) + +## Phase 2 Preview (Trading Service Vertical Slice) + +After Phase 1 validates the strategy through backtesting: +- Fix `fetch_features_for_symbol()` stub in trading service +- Wire ensemble coordinator to real feature cache +- Connect risk engine to `risk/` crate +- Fix paper trading hardcoded prices +- Enable OpenTelemetry tracing + +## Phase 3 Preview (Hardening) + +- mTLS certificate verification +- OCSP revocation checking +- Execution algorithms (TWAP/VWAP) +- Broker connectivity (IB TWS / AMP Futures FIX)