diff --git a/docs/plans/2026-02-21-real-data-backtest-and-deploy-design.md b/docs/plans/2026-02-21-real-data-backtest-and-deploy-design.md new file mode 100644 index 000000000..967c3a406 --- /dev/null +++ b/docs/plans/2026-02-21-real-data-backtest-and-deploy-design.md @@ -0,0 +1,101 @@ +# Real-Data Ensemble Backtest & Hyperopt Deploy Design + +**Date**: 2026-02-21 +**Goal**: Prove the 4-model ensemble works end-to-end on real 6E.FUT data, then close the hyperopt-to-production gap. + +## Problem + +All pipeline components exist (4 models, ensemble, paper broker, PnL tracker, validation harness, hyperopt) but they've never been connected end-to-end on real market data. Ensemble tests use mocks for Mamba2/TFT. Paper trading tests use synthetic data. Hyperopt saves best params but doesn't deploy them. + +## Phase A: Real-Data Ensemble Backtest + +### Data Flow + +``` +Real 6E.FUT bars (DBN files from test_data/real/databento/ml_training_small) + │ + ▼ +Feature extraction (51-dim FeatureVector per bar) + │ + ▼ +InferenceEnsemble (DQN + PPO + Mamba2 + TFT, random weights) + │ warm-up: first 50 bars fill sequence buffers, no trading + ▼ +TradeSignal (Buy/Sell/Hold, threshold ±0.1) + │ + ▼ +PaperBroker ($100k, 1bps slippage, $2.50 commission) + │ + ▼ +PnLTracker → Sharpe, max drawdown, cumulative return +``` + +### Key Decisions + +- **Dataset**: `test_data/real/databento/ml_training_small` (~361 files of 6E.FUT) +- **Model weights**: Random (proving pipeline correctness, not alpha) +- **Warm-up**: 50 bars to fill Mamba2/TFT sequence buffers before trading starts +- **Feature extraction**: Reuse existing real_data_loader to build FeatureVectors from OHLCV bars +- **Success criteria**: Pipeline completes without panics, all metrics are finite and bounded. Positive Sharpe NOT required (random weights). + +### What This Proves + +- All 4 real model adapters work together in ensemble +- Feature extraction → prediction → signal → fill → P&L path is connected +- Sequence models (Mamba2, TFT) handle real data dimensions correctly +- PaperBroker produces realistic fill simulation +- PnLTracker computes valid Sharpe/drawdown from real price movements + +## Phase B: Hyperopt → Deploy → Validate + +### Data Flow + +``` +Best params JSON (from ml/hyperopt_results/) + │ + ▼ +DQNParams deserialization → DQNHyperparameters + │ + ▼ +DQNTrainer::train(real_data) → checkpoint.safetensors + │ + ▼ +DqnInferenceAdapter::from_checkpoint(path) ← NEW CAPABILITY + │ + ▼ +InferenceEnsemble (trained DQN + random PPO/Mamba2/TFT) + │ + ▼ +ValidationHarness (walk-forward, 5 folds) + │ + ▼ +ValidationVerdict: Pass / Marginal / Fail + DSR p-value, PBO ratio, permutation p-value +``` + +### Key Decisions + +- **Start with DQN only** — it has 16+ hyperopt runs with best Sharpe up to 3.16 +- **New capability needed**: `DqnInferenceAdapter::from_checkpoint(path)` to load trained weights +- **Validation**: Walk-forward with 5 folds, DSR p<0.05, PBO<0.25 +- **PPO/Mamba2/TFT deployment**: Deferred until DQN proves the pattern works + +### What This Proves + +- Hyperopt results can be deployed to the inference pipeline +- Trained models produce statistically validated alpha (or not — that's valuable info too) +- The full loop: optimize → train → deploy → validate is closed + +## Constraints + +- GPU: RTX 3050 Ti, 4GB VRAM — batch sizes capped at 230 +- `SQLX_OFFLINE=true` for all cargo commands +- Clippy denials: `unwrap_used`, `expect_used`, `panic`, `indexing_slicing` +- Graceful skip if test data not found (CI-safe) + +## Non-Goals + +- Live trading or real broker integration +- Multi-asset support (6E.FUT only) +- Optimizing for positive Sharpe in Phase A (random weights) +- Deploying all 4 models from hyperopt (DQN first)