diff --git a/docs/plans/2026-02-23-trading-universe-data-organization-design.md b/docs/plans/2026-02-23-trading-universe-data-organization-design.md new file mode 100644 index 000000000..5f8e392d0 --- /dev/null +++ b/docs/plans/2026-02-23-trading-universe-data-organization-design.md @@ -0,0 +1,246 @@ +# Trading Universe & Data Organization Design + +**Date**: 2026-02-23 +**Status**: Approved +**Goal**: Define the baseline futures trading universe, reorganize data into the cache structure, and wire universe config into the data pipeline. + +## Problem + +- `AssetUniverse::us_starter()` lists equities (SPY, AAPL, etc.) but we have zero equity data +- All existing data is CME futures (ES, NQ, ZN, 6E) from Jan-May 2024 — almost 2 years stale +- Test data is scattered across `test_data/` in multiple formats and locations (root-level, ml_training/, corrupted/) +- No connection between `AssetUniverse` and `DatasetSpec` — the universe doesn't feed the pipeline +- Limited initial capital requires micro contract mapping (train on ES, trade MES) +- No download specification for acquiring the remaining ~640 trading days to reach 730d + +## Strategic Decisions + +- **Baseline universe**: 4 CME futures (ES, NQ, ZN, 6E) — we have data and IBKR supports them +- **Micro contract execution**: Train on full-size contracts (identical price data), execute on micros for limited capital +- **Data window**: 730 days ending today (March 2024 → February 2026) for regime diversity +- **Schema**: OHLCV-1m only for baseline (MBP-10 deferred to Phase 2) +- **Download**: Manual via Databento CLI for now, automated download is future work + +## Part 1: Universe Definition + +### `futures_baseline()` Preset + +| Training Symbol | Trading Symbol | Exchange | Asset Class | Why | +|----------------|---------------|----------|-------------|-----| +| ES.FUT | MES.FUT | GLBX.MDP3 | Future | Benchmark equity index, most liquid | +| NQ.FUT | MNQ.FUT | GLBX.MDP3 | Future | Tech-heavy, different character from ES | +| ZN.FUT | ZN.FUT | GLBX.MDP3 | Future | Uncorrelated to equities, less HFT | +| 6E.FUT | M6E.FUT | GLBX.MDP3 | Future | EU exposure, FX = different regime dynamics | + +### New Field: `trading_symbol` + +```rust +pub struct UniverseAsset { + pub symbol: String, // Training/data symbol (ES.FUT) + pub trading_symbol: Option, // Execution symbol (MES.FUT), None = same as symbol + pub exchange: String, + pub asset_class: AssetClass, + pub sector: Option, + pub min_daily_volume: f64, + pub enabled: bool, +} +``` + +When `trading_symbol` is `Some(...)`, the system trains on `symbol` data but routes execution orders to `trading_symbol`. This is the micro contract mapping. + +### GPU Compatibility (RTX 3050 Ti 4GB) + +- 730d × 4 symbols × ~400 bars/day = ~1.17M bars total +- At 128 features × f32 = 512 bytes/bar → ~570MB raw features (fits in RAM) +- Each model trains sequentially, peak GPU ~350MB (TFT) — well within 4GB +- `GpuCapabilities` module dynamically adjusts batch sizes + +## Part 2: Data Organization + +### Current State (scattered) + +``` +test_data/ 2.7GB total +├── ES_FUT_180d.parquet ← duplicate consolidated +├── ES_FUT_180d.dbn ← duplicate consolidated +├── ES_FUT_small.parquet ← fixture +├── NQ_FUT_180d.parquet ← duplicate consolidated +├── ZN_FUT_90d_clean.parquet ← duplicate consolidated +├── 6E_FUT_180d.dbn ← duplicate consolidated +├── real/databento/ +│ ├── ml_training/ ← 360 daily DBN files (90 days × 4 symbols) +│ │ ├── corrupted/ ← 90 days bad ES data +│ │ ├── ES.FUT_ohlcv-1m_*.dbn +│ │ ├── NQ.FUT_ohlcv-1m_*.dbn +│ │ ├── ZN.FUT_ohlcv-1m_*.dbn +│ │ └── 6E.FUT_ohlcv-1m_*.dbn +│ ├── ml_training_small/ ← 4 days 6E fixture +│ └── GC_continuous_*.dbn ← partial, not in universe +├── mbp10/ ← 7 days CME order book (keep for Phase 2) +└── various .tmp, _unseen, etc. ← stale +``` + +### Target State (organized) + +``` +data/ +└── cache/ + ├── manifest.json + ├── ES.FUT/ + │ ├── ohlcv-1m_2024-01-02.dbn + │ ├── ... + │ └── ohlcv-1m_2026-02-21.dbn (after download) + ├── NQ.FUT/ + │ └── ... + ├── ZN.FUT/ + │ └── ... + └── 6E.FUT/ + └── ... + +test_data/ +├── mbp10/ (keep for Phase 2) +├── fixtures/ (small synthetic files for unit tests) +│ ├── ES_FUT_small.parquet +│ ├── ZN_FUT_small.parquet +│ └── NQ_FUT_small.parquet +└── README.md (what's what) +``` + +### What Gets Deleted + +- `test_data/real/databento/ml_training/corrupted/` — 90 days bad ES data +- Root-level consolidated files: `*_180d.parquet`, `*_180d.dbn`, `*_decompressed.dbn` +- `ES_FUT_unseen*.dbn`, `ES_FUT_unseen.parquet` — stale eval data +- `GC_continuous_*.dbn` — not in baseline universe +- `real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn.tmp` — temp file +- `real/databento/nq_180d/` — duplicate consolidated + +### What Gets Moved + +- `test_data/real/databento/ml_training/_ohlcv-1m_*.dbn` → `data/cache//` +- `test_data/*_small.parquet` → `test_data/fixtures/` + +## Part 3: Download Specification + +### Config File: `config/universe-futures-baseline.toml` + +```toml +[universe] +name = "futures-baseline" +description = "CME futures baseline: 4 symbols, 730 days, OHLCV-1m" +date_range_start = "2024-03-01" +date_range_end = "2026-02-23" +bar_size = "1m" +databento_dataset = "GLBX.MDP3" +databento_schema = "ohlcv-1m" + +[[symbols]] +symbol = "ES.FUT" +trading_symbol = "MES.FUT" +asset_class = "Future" +min_daily_volume = 1000000.0 + +[[symbols]] +symbol = "NQ.FUT" +trading_symbol = "MNQ.FUT" +asset_class = "Future" +min_daily_volume = 500000.0 + +[[symbols]] +symbol = "ZN.FUT" +asset_class = "Future" +min_daily_volume = 500000.0 + +[[symbols]] +symbol = "6E.FUT" +trading_symbol = "M6E.FUT" +asset_class = "Future" +min_daily_volume = 200000.0 +``` + +### Download Gap Calculation + +We have 90 days (Jan 2 - May 6, 2024). We need ~730 trading days ending Feb 23, 2026. + +Missing ranges per symbol: +- 2024-05-07 → 2026-02-23 (~460 trading days) + +The `DatasetManager::find_gaps()` method handles this — it compares the manifest against the required range and returns what to download. + +## Part 4: Pipeline Wiring + +### `DatasetSpec::from_universe()` + +```rust +impl DatasetSpec { + pub fn from_universe(universe: &AssetUniverse, mode: DatasetMode) -> Self { + let symbols = universe.eligible().iter().map(|a| SymbolSpec { + symbol: a.symbol.clone(), + exchange: a.exchange.clone(), + asset_class: a.asset_class.clone(), + }).collect(); + + Self { + name: format!("{}-{:?}", "futures-baseline", mode), + symbols, + date_range: mode.to_date_range(), + bar_size: BarSize::OneMinute, + split_ratio: SplitRatio::default(), + mode, + } + } +} +``` + +### `AssetUniverse::from_config()` + +```rust +impl AssetUniverse { + pub fn from_config(path: &Path) -> Result { + // Parse TOML config file into AssetUniverse + } +} +``` + +## Integration Points + +| From | To | How | +|------|-----|-----| +| Universe config (TOML) | `AssetUniverse` | `from_config()` parses TOML | +| `AssetUniverse` | `DatasetSpec` | `from_universe()` creates spec with correct symbols | +| `DatasetSpec` | `DatasetManager` | `prepare()` checks cache, identifies gaps | +| Cache manifest | Gap detection | `find_gaps()` returns missing date ranges | +| `trading_symbol` | Execution routing | Ensemble predicts on `symbol`, orders go to `trading_symbol` | + +## Per-Mode Data Estimates (4 symbols, OHLCV-1m) + +| Mode | Duration | Bars/Symbol | Total Bars | Disk (DBN) | Features (RAM) | +|------|----------|-------------|------------|------------|----------------| +| Dev | 30 days | ~12K | ~48K | ~15MB | ~24MB | +| Backtest | 180 days | ~72K | ~288K | ~90MB | ~140MB | +| Full | 730 days | ~292K | ~1.17M | ~360MB | ~570MB | + +## Fallback Behavior + +1. Download incomplete → use what's cached, log warning about reduced date range +2. Symbol has no data → skip from active set, warn +3. Trading symbol unavailable at broker → fall back to full-size contract, reduce position size +4. All symbols below threshold → stay flat (no trading) + +## What This Does NOT Include + +- Automated Databento API download (manual CLI for now) +- MBP-10 order book integration (Phase 2) +- EU futures: FDAX, FESX, FGBL (Phase 2, after baseline proves out) +- Mid-cap equities (Phase 3, only if prediction accuracy justifies) +- Broker-specific execution routing (separate work) + +## Testing + +- `futures_baseline()` preset: correct symbols, classes, volumes +- `trading_symbol` mapping: ES→MES, NQ→MNQ, 6E→M6E, ZN→ZN +- `from_config()`: parse TOML, handle missing fields, validate +- `from_universe()`: correct DatasetSpec generation per mode +- Cache manifest after data move: correct ranges, file paths +- Data integrity: all moved DBN files still parseable +- All tests run on CPU, no GPU or Databento API required