From 66966806c75417e43ce075e4c14bbadbe22e8ce9 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 23 Feb 2026 14:17:22 +0100 Subject: [PATCH] docs: add trading universe & data organization design Define futures baseline universe (ES, NQ, ZN, 6E) with micro contract mapping for limited capital. Reorganize scattered test data into cache structure. Download spec for 730 days of OHLCV-1m. Co-Authored-By: Claude Opus 4.6 --- ...ading-universe-data-organization-design.md | 246 ++++++++++++++++++ 1 file changed, 246 insertions(+) create mode 100644 docs/plans/2026-02-23-trading-universe-data-organization-design.md diff --git a/docs/plans/2026-02-23-trading-universe-data-organization-design.md b/docs/plans/2026-02-23-trading-universe-data-organization-design.md new file mode 100644 index 000000000..5f8e392d0 --- /dev/null +++ b/docs/plans/2026-02-23-trading-universe-data-organization-design.md @@ -0,0 +1,246 @@ +# Trading Universe & Data Organization Design + +**Date**: 2026-02-23 +**Status**: Approved +**Goal**: Define the baseline futures trading universe, reorganize data into the cache structure, and wire universe config into the data pipeline. + +## Problem + +- `AssetUniverse::us_starter()` lists equities (SPY, AAPL, etc.) but we have zero equity data +- All existing data is CME futures (ES, NQ, ZN, 6E) from Jan-May 2024 — almost 2 years stale +- Test data is scattered across `test_data/` in multiple formats and locations (root-level, ml_training/, corrupted/) +- No connection between `AssetUniverse` and `DatasetSpec` — the universe doesn't feed the pipeline +- Limited initial capital requires micro contract mapping (train on ES, trade MES) +- No download specification for acquiring the remaining ~640 trading days to reach 730d + +## Strategic Decisions + +- **Baseline universe**: 4 CME futures (ES, NQ, ZN, 6E) — we have data and IBKR supports them +- **Micro contract execution**: Train on full-size contracts (identical price data), execute on micros for limited capital +- **Data window**: 730 days ending today (March 2024 → February 2026) for regime diversity +- **Schema**: OHLCV-1m only for baseline (MBP-10 deferred to Phase 2) +- **Download**: Manual via Databento CLI for now, automated download is future work + +## Part 1: Universe Definition + +### `futures_baseline()` Preset + +| Training Symbol | Trading Symbol | Exchange | Asset Class | Why | +|----------------|---------------|----------|-------------|-----| +| ES.FUT | MES.FUT | GLBX.MDP3 | Future | Benchmark equity index, most liquid | +| NQ.FUT | MNQ.FUT | GLBX.MDP3 | Future | Tech-heavy, different character from ES | +| ZN.FUT | ZN.FUT | GLBX.MDP3 | Future | Uncorrelated to equities, less HFT | +| 6E.FUT | M6E.FUT | GLBX.MDP3 | Future | EU exposure, FX = different regime dynamics | + +### New Field: `trading_symbol` + +```rust +pub struct UniverseAsset { + pub symbol: String, // Training/data symbol (ES.FUT) + pub trading_symbol: Option, // Execution symbol (MES.FUT), None = same as symbol + pub exchange: String, + pub asset_class: AssetClass, + pub sector: Option, + pub min_daily_volume: f64, + pub enabled: bool, +} +``` + +When `trading_symbol` is `Some(...)`, the system trains on `symbol` data but routes execution orders to `trading_symbol`. This is the micro contract mapping. + +### GPU Compatibility (RTX 3050 Ti 4GB) + +- 730d × 4 symbols × ~400 bars/day = ~1.17M bars total +- At 128 features × f32 = 512 bytes/bar → ~570MB raw features (fits in RAM) +- Each model trains sequentially, peak GPU ~350MB (TFT) — well within 4GB +- `GpuCapabilities` module dynamically adjusts batch sizes + +## Part 2: Data Organization + +### Current State (scattered) + +``` +test_data/ 2.7GB total +├── ES_FUT_180d.parquet ← duplicate consolidated +├── ES_FUT_180d.dbn ← duplicate consolidated +├── ES_FUT_small.parquet ← fixture +├── NQ_FUT_180d.parquet ← duplicate consolidated +├── ZN_FUT_90d_clean.parquet ← duplicate consolidated +├── 6E_FUT_180d.dbn ← duplicate consolidated +├── real/databento/ +│ ├── ml_training/ ← 360 daily DBN files (90 days × 4 symbols) +│ │ ├── corrupted/ ← 90 days bad ES data +│ │ ├── ES.FUT_ohlcv-1m_*.dbn +│ │ ├── NQ.FUT_ohlcv-1m_*.dbn +│ │ ├── ZN.FUT_ohlcv-1m_*.dbn +│ │ └── 6E.FUT_ohlcv-1m_*.dbn +│ ├── ml_training_small/ ← 4 days 6E fixture +│ └── GC_continuous_*.dbn ← partial, not in universe +├── mbp10/ ← 7 days CME order book (keep for Phase 2) +└── various .tmp, _unseen, etc. ← stale +``` + +### Target State (organized) + +``` +data/ +└── cache/ + ├── manifest.json + ├── ES.FUT/ + │ ├── ohlcv-1m_2024-01-02.dbn + │ ├── ... + │ └── ohlcv-1m_2026-02-21.dbn (after download) + ├── NQ.FUT/ + │ └── ... + ├── ZN.FUT/ + │ └── ... + └── 6E.FUT/ + └── ... + +test_data/ +├── mbp10/ (keep for Phase 2) +├── fixtures/ (small synthetic files for unit tests) +│ ├── ES_FUT_small.parquet +│ ├── ZN_FUT_small.parquet +│ └── NQ_FUT_small.parquet +└── README.md (what's what) +``` + +### What Gets Deleted + +- `test_data/real/databento/ml_training/corrupted/` — 90 days bad ES data +- Root-level consolidated files: `*_180d.parquet`, `*_180d.dbn`, `*_decompressed.dbn` +- `ES_FUT_unseen*.dbn`, `ES_FUT_unseen.parquet` — stale eval data +- `GC_continuous_*.dbn` — not in baseline universe +- `real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn.tmp` — temp file +- `real/databento/nq_180d/` — duplicate consolidated + +### What Gets Moved + +- `test_data/real/databento/ml_training/_ohlcv-1m_*.dbn` → `data/cache//` +- `test_data/*_small.parquet` → `test_data/fixtures/` + +## Part 3: Download Specification + +### Config File: `config/universe-futures-baseline.toml` + +```toml +[universe] +name = "futures-baseline" +description = "CME futures baseline: 4 symbols, 730 days, OHLCV-1m" +date_range_start = "2024-03-01" +date_range_end = "2026-02-23" +bar_size = "1m" +databento_dataset = "GLBX.MDP3" +databento_schema = "ohlcv-1m" + +[[symbols]] +symbol = "ES.FUT" +trading_symbol = "MES.FUT" +asset_class = "Future" +min_daily_volume = 1000000.0 + +[[symbols]] +symbol = "NQ.FUT" +trading_symbol = "MNQ.FUT" +asset_class = "Future" +min_daily_volume = 500000.0 + +[[symbols]] +symbol = "ZN.FUT" +asset_class = "Future" +min_daily_volume = 500000.0 + +[[symbols]] +symbol = "6E.FUT" +trading_symbol = "M6E.FUT" +asset_class = "Future" +min_daily_volume = 200000.0 +``` + +### Download Gap Calculation + +We have 90 days (Jan 2 - May 6, 2024). We need ~730 trading days ending Feb 23, 2026. + +Missing ranges per symbol: +- 2024-05-07 → 2026-02-23 (~460 trading days) + +The `DatasetManager::find_gaps()` method handles this — it compares the manifest against the required range and returns what to download. + +## Part 4: Pipeline Wiring + +### `DatasetSpec::from_universe()` + +```rust +impl DatasetSpec { + pub fn from_universe(universe: &AssetUniverse, mode: DatasetMode) -> Self { + let symbols = universe.eligible().iter().map(|a| SymbolSpec { + symbol: a.symbol.clone(), + exchange: a.exchange.clone(), + asset_class: a.asset_class.clone(), + }).collect(); + + Self { + name: format!("{}-{:?}", "futures-baseline", mode), + symbols, + date_range: mode.to_date_range(), + bar_size: BarSize::OneMinute, + split_ratio: SplitRatio::default(), + mode, + } + } +} +``` + +### `AssetUniverse::from_config()` + +```rust +impl AssetUniverse { + pub fn from_config(path: &Path) -> Result { + // Parse TOML config file into AssetUniverse + } +} +``` + +## Integration Points + +| From | To | How | +|------|-----|-----| +| Universe config (TOML) | `AssetUniverse` | `from_config()` parses TOML | +| `AssetUniverse` | `DatasetSpec` | `from_universe()` creates spec with correct symbols | +| `DatasetSpec` | `DatasetManager` | `prepare()` checks cache, identifies gaps | +| Cache manifest | Gap detection | `find_gaps()` returns missing date ranges | +| `trading_symbol` | Execution routing | Ensemble predicts on `symbol`, orders go to `trading_symbol` | + +## Per-Mode Data Estimates (4 symbols, OHLCV-1m) + +| Mode | Duration | Bars/Symbol | Total Bars | Disk (DBN) | Features (RAM) | +|------|----------|-------------|------------|------------|----------------| +| Dev | 30 days | ~12K | ~48K | ~15MB | ~24MB | +| Backtest | 180 days | ~72K | ~288K | ~90MB | ~140MB | +| Full | 730 days | ~292K | ~1.17M | ~360MB | ~570MB | + +## Fallback Behavior + +1. Download incomplete → use what's cached, log warning about reduced date range +2. Symbol has no data → skip from active set, warn +3. Trading symbol unavailable at broker → fall back to full-size contract, reduce position size +4. All symbols below threshold → stay flat (no trading) + +## What This Does NOT Include + +- Automated Databento API download (manual CLI for now) +- MBP-10 order book integration (Phase 2) +- EU futures: FDAX, FESX, FGBL (Phase 2, after baseline proves out) +- Mid-cap equities (Phase 3, only if prediction accuracy justifies) +- Broker-specific execution routing (separate work) + +## Testing + +- `futures_baseline()` preset: correct symbols, classes, volumes +- `trading_symbol` mapping: ES→MES, NQ→MNQ, 6E→M6E, ZN→ZN +- `from_config()`: parse TOML, handle missing fields, validate +- `from_universe()`: correct DatasetSpec generation per mode +- Cache manifest after data move: correct ranges, file paths +- Data integrity: all moved DBN files still parseable +- All tests run on CPU, no GPU or Databento API required