docs: backtesting vertical slice production readiness design

8-task plan to bridge disconnected backtesting pipeline layers: DBN parser → feature extraction → ML inference → position tracking → PnL Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 15:21:22 +01:00
parent e2a712593a
commit 30241e858d
1 changed files with 205 additions and 0 deletions
--- a/docs/plans/2026-02-21-backtesting-vertical-slice-design.md
+++ b/docs/plans/2026-02-21-backtesting-vertical-slice-design.md
@@ -0,0 +1,205 @@
+# Backtesting Vertical Slice — Production Readiness Design
+
+> **Date:** 2026-02-21
+> **Status:** Approved
+> **Goal:** Load `.dbn` file + trained model checkpoints → run backtest → get real PnL numbers with trade-by-trade history.
+
+## Problem Statement
+
+The backtesting pipeline has 8 disconnected layers — each independently functional but no connectors between them. The result: backtesting has never actually run ML inference on real data. Every layer produces output, but nothing flows end-to-end.
+
+### Current Broken Flow
+
+```
+.dbn file → [DbnParser] → ProcessedMessage    ✗ NO CONVERTER    → MarketEvent
+Parquet   → [ParquetDataLoader]               ✗ NO CONVERTER    → MarketEvent
+MarketEvent → [AdaptiveStrategyRunner]        ✗ WRONG FEATURES  → 3-5 dim (models need 51)
+Features → [GlobalRegistry.predict_selected]  ✗ EMPTY REGISTRY  → ModelNotFound
+Prediction → [generate_signal]                ✓ WORKS           → TradingSignal
+Signal → [execute_order]                      ✓ WORKS           → cash update
+Position → [PositionTracker]                  ✗ STUB Ok(())     → no tracking
+PnL                                           ✗ NEVER UPDATED   → always 0
+```
+
+## Design Decisions
+
+### 1. Data Source: DBN files via DbnParser
+
+The `DbnParser` in `data/src/providers/databento/dbn_parser.rs` already correctly parses MBO, MBP1, MBP10, Trade, and OHLCV records from `.dbn` files. We add a converter layer (`ProcessedMessage` → `MarketEvent`) and a replay engine that feeds events into the existing `StrategyTester`.
+
+**Alternative considered:** Parquet replay via `ParquetDataLoader`. Rejected because it adds a conversion step (DBN→Parquet) and `ParquetMarketDataEvent` has a different schema than `MarketEvent`.
+
+### 2. Feature Extraction: Production 51-dim via FeatureExtractor
+
+Replace the local 3-5 feature extractor in `strategy_runner.rs` with `ProductionFeatureExtractorAdapter` from `ml/src/features/production_adapter.rs`, which produces exactly 51 features matching DQN/PPO training expectations.
+
+**Alternative considered:** `UnifiedFeatureExtractor` from `data/` (50-100+ features). Rejected because ML models were trained on the 51-dim layout from `ml::features::extraction::FeatureExtractor`. Feature dimension mismatch would produce garbage predictions.
+
+### 3. Model Loading: Direct safetensors → model constructors
+
+Load `.safetensors` checkpoints directly into model structs (DQN, PPO, TFT, Mamba2) using existing `load_from_safetensors()` methods, then wrap in `Arc<dyn MLModel>` for the global registry.
+
+**Alternative considered:** S3-based `model_loader` crate. Rejected for backtesting because it only returns raw bytes with no deserialization, and backtesting should work from local files without S3 dependency.
+
+### 4. Position Tracking: Fix existing stubs
+
+The `PositionTracker` struct exists with correct fields but empty method bodies. We implement the actual accounting rather than adding a new tracker, keeping the existing `StrategyTester` integration intact.
+
+## Architecture
+
+```
+.dbn file
+    ↓ DbnParser::parse_batch() (exists, data/ crate)
+Vec<ProcessedMessage>
+    ↓ NEW: dbn_to_market_event() converter (backtesting/ crate)
+Vec<MarketEvent>
+    ↓ NEW: DbnReplayEngine::start_replay() → mpsc channel
+ReplayEvent stream
+    ↓ StrategyTester event loop (exists)
+AdaptiveStrategyRunner.on_market_event()
+    ↓ MODIFIED: uses ProductionFeatureExtractorAdapter (51-dim)
+Features { values: Vec<f64> [51], names, timestamp }
+    ↓ GlobalRegistry.predict_selected() (exists)
+    ↓ NEW: BacktestModelLoader populates registry at startup
+Vec<ModelPrediction>
+    ↓ generate_signal() + RiskManager (exist)
+TradingSignal
+    ↓ execute_order() (exists)
+    ↓ FIXED: PositionTracker (real accounting + trade recording)
+BacktestResult { trades, total_pnl, sharpe, max_drawdown, win_rate }
+```
+
+## Components
+
+### Task 1: DBN → MarketEvent Converter
+
+**File:** `backtesting/src/dbn_converter.rs`
+
+Convert `data::providers::databento::dbn_parser::ProcessedMessage` variants to `trading_engine::types::events::MarketEvent` variants:
+- `ProcessedMessage::Trade { price, quantity, timestamp, symbol }` → `MarketEvent::Trade { .. }`
+- `ProcessedMessage::Ohlcv { open, high, low, close, volume, timestamp, symbol }` → `MarketEvent::Bar { .. }`
+- `ProcessedMessage::Quote { bid_price, ask_price, .. }` → `MarketEvent::OrderBookUpdate { .. }`
+
+Must handle: price type conversions (f64 → Decimal), timestamp formats, symbol normalization.
+
+### Task 2: DBN Replay Engine
+
+**File:** `backtesting/src/dbn_replay.rs`
+
+```rust
+pub struct DbnReplayEngine {
+    events: Vec<MarketEvent>,  // pre-loaded and sorted by timestamp
+}
+
+impl DbnReplayEngine {
+    pub fn from_dbn_file(path: &Path) -> Result<Self>;
+    pub fn start_replay(&self, tx: mpsc::UnboundedSender<ReplayEvent>) -> JoinHandle<()>;
+}
+```
+
+Integrates with `StrategyTester` by implementing the same channel-based feed pattern as `MarketReplay`.
+
+### Task 3: Wire Production Feature Extractor
+
+**File:** Modify `backtesting/src/strategy_runner.rs`
+
+Replace the local `FeatureExtractor` (3-5 features) with `ProductionFeatureExtractorAdapter` from `ml/src/features/production_adapter.rs`. The adapter needs a warmup period of 50 bars — during warmup, no predictions are made. After warmup, each market event produces a 51-dim feature vector.
+
+Must handle: the adapter expects OHLCV bars, but `on_market_event()` receives individual trades. Buffer trades into 1-minute bars before feeding to the extractor.
+
+### Task 4: Model Checkpoint Loader
+
+**File:** `backtesting/src/model_loader.rs`
+
+```rust
+pub fn load_model_from_checkpoint(
+    model_type: &str,    // "DQN", "PPO", "TFT", "MAMBA-2"
+    checkpoint_path: &Path,
+    config: &ModelConfig, // per-model config (state_dim, hidden_dims, etc.)
+) -> Result<Arc<dyn MLModel>, MLError>;
+```
+
+Uses:
+- DQN: `DQN::new(config)` then `load_from_safetensors(path)`
+- PPO: `PPO::new(config)` then load actor weights
+- TFT: `TemporalFusionTransformer::new(config)` then load weights
+- Mamba2: `Mamba2SSM::new(config, &device)` then load weights
+
+### Task 5: Registry Startup
+
+**File:** `backtesting/src/model_loader.rs` (same file as Task 4)
+
+```rust
+pub struct BacktestModelLoader;
+
+impl BacktestModelLoader {
+    /// Load models from config and register in global registry
+    pub fn load_models(models: &[ModelSpec]) -> Result<()> {
+        for spec in models {
+            let model = load_model_from_checkpoint(&spec.model_type, &spec.path, &spec.config)?;
+            get_global_registry().register(model);
+        }
+        Ok(())
+    }
+}
+```
+
+Called by `StrategyTester::run_test()` before starting the event replay loop.
+
+### Task 6: Fix PositionTracker
+
+**File:** Modify `backtesting/src/strategy_tester.rs`
+
+```rust
+pub fn update_position(&mut self, symbol: &str, price: Decimal, quantity: i64) -> Result<()> {
+    // Real accounting: track entry price, current price, unrealized PnL
+    // Handle position flip (long → short), partial closes, averaging
+}
+
+pub fn record_trade(&mut self, trade: TradeRecord) -> Result<()> {
+    // Push to self.trades vec with: timestamp, symbol, side, price, quantity, realized_pnl
+}
+```
+
+### Task 7: Fix PnL Tracking
+
+**File:** Modify `backtesting/src/strategy_runner.rs`
+
+- Update `PerformanceTracker.total_pnl` on each trade execution
+- Increment `winning_trades` counter when realized PnL > 0
+- Populate `StrategyResult.trades` from recorded trade history in `finalize()`
+- Compute `performance_timeline` (equity curve snapshots at regular intervals)
+
+### Task 8: Integration Test
+
+**File:** `backtesting/tests/dbn_backtest_integration.rs`
+
+End-to-end test:
+1. Create synthetic `.dbn` file with known price pattern (trending up)
+2. Create DQN model with random weights, save checkpoint
+3. Configure backtest: load model, set risk limits, set position sizing
+4. Run backtest over synthetic data
+5. Assert: features are 51-dim, predictions are in valid range, trades are recorded, PnL is computed, position tracking is accurate
+
+## Constraints
+
+- **Build:** `SQLX_OFFLINE=true cargo check -p backtesting`
+- **VRAM:** RTX 3050 Ti 4GB — max 2-3 models loaded simultaneously
+- **Clippy:** `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]`
+- **No PostgreSQL:** Tests must work offline (no DB dependency)
+
+## Phase 2 Preview (Trading Service Vertical Slice)
+
+After Phase 1 validates the strategy through backtesting:
+- Fix `fetch_features_for_symbol()` stub in trading service
+- Wire ensemble coordinator to real feature cache
+- Connect risk engine to `risk/` crate
+- Fix paper trading hardcoded prices
+- Enable OpenTelemetry tracing
+
+## Phase 3 Preview (Hardening)
+
+- mTLS certificate verification
+- OCSP revocation checking
+- Execution algorithms (TWAP/VWAP)
+- Broker connectivity (IB TWS / AMP Futures FIX)