From ae225b0e31a4212001c8ddef47c8b693025985f8 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 23 Feb 2026 14:21:48 +0100 Subject: [PATCH] docs: add trading universe & data organization implementation plan 10-task plan: unify AssetClass, add trading_symbol field, futures_baseline() preset, TOML config loading, DatasetSpec::from_universe() wiring, data reorganization into cache structure, stale data cleanup, manifest generation. Co-Authored-By: Claude Opus 4.6 --- ...iverse-data-organization-implementation.md | 1080 +++++++++++++++++ 1 file changed, 1080 insertions(+) create mode 100644 docs/plans/2026-02-23-trading-universe-data-organization-implementation.md diff --git a/docs/plans/2026-02-23-trading-universe-data-organization-implementation.md b/docs/plans/2026-02-23-trading-universe-data-organization-implementation.md new file mode 100644 index 000000000..674ae698b --- /dev/null +++ b/docs/plans/2026-02-23-trading-universe-data-organization-implementation.md @@ -0,0 +1,1080 @@ +# Trading Universe & Data Organization Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Define a futures baseline universe (ES, NQ, ZN, 6E) with micro contract mapping, reorganize scattered test data into the cache structure, add TOML config loading, and wire `AssetUniverse` into `DatasetSpec`. + +**Architecture:** Add `trading_symbol` field to `UniverseAsset`, create a `futures_baseline()` preset, unify the duplicated `AssetClass` enum, add `from_config()` for TOML loading and `from_universe()` on `DatasetSpec`, physically reorganize DBN files into `data/cache/`, and clean up stale data. + +**Tech Stack:** Rust, serde, toml crate (workspace dep), chrono, Databento DBN files + +--- + +## Context for the Implementer + +### Key Files You'll Touch + +- `ml/src/asset_selection/mod.rs` — `UniverseAsset`, `AssetUniverse`, `AssetClass` +- `ml/src/data_pipeline/mod.rs` — `AssetClass` (duplicate), `DatasetSpec`, `SymbolSpec` +- `ml/Cargo.toml` — add `toml` dependency +- `config/universe-futures-baseline.toml` — new config file +- `.gitignore` — add `data/cache/` + +### Important Conventions + +- **Clippy deny rules**: `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]` — use `.get()`, `?`, `.ok_or()` instead. +- **Test command**: `SQLX_OFFLINE=true cargo test -p ml --lib` +- **Build check**: `SQLX_OFFLINE=true cargo check --workspace` +- **Two `AssetClass` enums exist**: `data_pipeline::AssetClass` and `asset_selection::AssetClass`. They're identical. Task 1 unifies them. +- **`toml` crate**: available as workspace dep in root `Cargo.toml` (version `0.8`) but not yet in `ml/Cargo.toml`. +- **Serde on all public types**: All `UniverseAsset`, `AssetUniverse`, `DatasetSpec` etc. derive `Serialize, Deserialize`. + +### Existing Data Layout + +``` +test_data/real/databento/ml_training/ — 360 clean DBN files (90 days × 4 symbols) +test_data/real/databento/ml_training/corrupted/ — 91 bad ES files (May-Sep 2024) +test_data/ — root-level duplicates, stale files +test_data/mbp10/ — 7 days MBP-10 (keep for Phase 2) +``` + +--- + +### Task 1: Unify `AssetClass` — Remove Duplicate Enum + +Both `data_pipeline::mod.rs` and `asset_selection::mod.rs` define identical `AssetClass` enums. This task makes `asset_selection::AssetClass` the canonical one and re-exports it in `data_pipeline`. + +**Files:** +- Modify: `ml/src/data_pipeline/mod.rs:18-24` — remove `AssetClass` enum, import from `asset_selection` +- Modify: `ml/src/asset_selection/mod.rs` — no changes needed (already has it) + +**Step 1: Write the failing test** + +Add to `ml/src/data_pipeline/mod.rs` tests: + +```rust +#[test] +fn test_asset_class_is_same_type() { + // Verify that data_pipeline::AssetClass and asset_selection::AssetClass + // are the same type (not just structurally equal) + let dp_future = AssetClass::Future; + let as_future = crate::asset_selection::AssetClass::Future; + // If they're the same type, this comparison compiles + assert_eq!(dp_future, as_future); +} +``` + +**Step 2: Run test to verify it fails** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib data_pipeline::tests::test_asset_class_is_same_type` + +Expected: FAIL — "mismatched types" because they're two separate enums. + +**Step 3: Unify the enums** + +In `ml/src/data_pipeline/mod.rs`, replace the local `AssetClass` definition (lines 18-24): + +```rust +/// Asset class categorization +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub enum AssetClass { + Equity, + ETF, + Future, +} +``` + +With a re-export: + +```rust +// Re-export AssetClass from asset_selection (canonical location) +pub use crate::asset_selection::AssetClass; +``` + +**Step 4: Run tests to verify it passes** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib data_pipeline::tests` + +Expected: ALL PASS (9 tests). The existing tests use `AssetClass::Equity` etc. which still resolves via the re-export. + +**Step 5: Run full workspace check** + +Run: `SQLX_OFFLINE=true cargo check --workspace` + +Expected: No errors. If any crate imports `data_pipeline::AssetClass` explicitly, it still works via re-export. + +**Step 6: Commit** + +```bash +git add ml/src/data_pipeline/mod.rs +git commit -m "refactor(ml): unify AssetClass — re-export from asset_selection" +``` + +--- + +### Task 2: Add `trading_symbol` Field to `UniverseAsset` + +**Files:** +- Modify: `ml/src/asset_selection/mod.rs:27-40` — add field +- Modify: `ml/src/asset_selection/mod.rs:82-143` — update `us_starter()` to include `trading_symbol: None` + +**Step 1: Write the failing test** + +Add to `ml/src/asset_selection/mod.rs` tests: + +```rust +#[test] +fn test_trading_symbol_mapping() { + let asset = UniverseAsset { + symbol: "ES.FUT".to_string(), + trading_symbol: Some("MES.FUT".to_string()), + exchange: "GLBX.MDP3".to_string(), + asset_class: AssetClass::Future, + sector: None, + min_daily_volume: 1_000_000.0, + enabled: true, + }; + assert_eq!(asset.execution_symbol(), "MES.FUT"); +} + +#[test] +fn test_trading_symbol_none_falls_back() { + let asset = UniverseAsset { + symbol: "ZN.FUT".to_string(), + trading_symbol: None, + exchange: "GLBX.MDP3".to_string(), + asset_class: AssetClass::Future, + sector: None, + min_daily_volume: 500_000.0, + enabled: true, + }; + assert_eq!(asset.execution_symbol(), "ZN.FUT"); +} +``` + +**Step 2: Run test to verify it fails** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib asset_selection::tests::test_trading_symbol` + +Expected: FAIL — `trading_symbol` field and `execution_symbol()` method don't exist. + +**Step 3: Add the field and method** + +In `ml/src/asset_selection/mod.rs`, add `trading_symbol` to `UniverseAsset`: + +```rust +/// A single asset in the trading universe. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct UniverseAsset { + /// Ticker symbol (e.g. "ES.FUT") — used for data/training. + pub symbol: String, + /// Execution symbol (e.g. "MES.FUT") — used for order routing. + /// When `None`, orders are routed to `symbol` directly. + pub trading_symbol: Option, + /// MIC exchange code (e.g. "GLBX.MDP3"). + pub exchange: String, + /// Asset classification. + pub asset_class: AssetClass, + /// Optional GICS sector. + pub sector: Option, + /// Minimum average daily volume required for eligibility. + pub min_daily_volume: f64, + /// Whether this asset is enabled for trading. + pub enabled: bool, +} + +impl UniverseAsset { + /// The symbol to use for order execution. + /// Returns `trading_symbol` if set, otherwise `symbol`. + #[must_use] + pub fn execution_symbol(&self) -> &str { + self.trading_symbol.as_deref().unwrap_or(&self.symbol) + } +} +``` + +Update every `UniverseAsset` struct literal in `us_starter()` to include `trading_symbol: None`. There are 7 of them (SPY, QQQ, AAPL, MSFT, NVDA, AMZN, IWM). Add `trading_symbol: None,` after each `symbol:` line. + +**Step 4: Run tests to verify all pass** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib asset_selection::tests` + +Expected: ALL PASS (9 tests — 7 existing + 2 new). + +**Step 5: Run workspace check** + +Run: `SQLX_OFFLINE=true cargo check --workspace` + +Expected: No errors. Serde skips `None` by default on serialization but includes it on deserialization with `Option`, so existing JSON roundtrip test still works. + +**Step 6: Commit** + +```bash +git add ml/src/asset_selection/mod.rs +git commit -m "feat(ml): add trading_symbol field to UniverseAsset for micro contract mapping" +``` + +--- + +### Task 3: Add `futures_baseline()` Preset + +**Files:** +- Modify: `ml/src/asset_selection/mod.rs` — add `futures_baseline()` method after `us_starter()` + +**Step 1: Write the failing test** + +Add to `ml/src/asset_selection/mod.rs` tests: + +```rust +#[test] +fn test_futures_baseline_universe() { + let universe = AssetUniverse::futures_baseline(); + assert_eq!(universe.assets.len(), 4); + assert_eq!(universe.eligible_count(), 4); + + // All are futures + for asset in &universe.assets { + assert_eq!(asset.asset_class, AssetClass::Future); + } +} + +#[test] +fn test_futures_baseline_symbols() { + let universe = AssetUniverse::futures_baseline(); + let symbols = universe.symbols(); + assert!(symbols.contains(&"ES.FUT")); + assert!(symbols.contains(&"NQ.FUT")); + assert!(symbols.contains(&"ZN.FUT")); + assert!(symbols.contains(&"6E.FUT")); +} + +#[test] +fn test_futures_baseline_micro_mapping() { + let universe = AssetUniverse::futures_baseline(); + let es = universe.find("ES.FUT"); + assert!(es.is_some()); + let es = es.unwrap_or_else(|| panic!("ES.FUT not found")); + assert_eq!(es.execution_symbol(), "MES.FUT"); + + let zn = universe.find("ZN.FUT"); + assert!(zn.is_some()); + let zn = zn.unwrap_or_else(|| panic!("ZN.FUT not found")); + // ZN has no micro mapping — executes as ZN.FUT + assert_eq!(zn.execution_symbol(), "ZN.FUT"); +} + +#[test] +fn test_futures_baseline_serde_roundtrip() { + let universe = AssetUniverse::futures_baseline(); + let json = serde_json::to_string(&universe).unwrap_or_default(); + let restored: AssetUniverse = + serde_json::from_str(&json).unwrap_or_else(|_| AssetUniverse::new()); + assert_eq!(restored.assets.len(), 4); + // Verify trading_symbol survives roundtrip + let es = restored.find("ES.FUT"); + assert_eq!( + es.map(|a| a.execution_symbol()), + Some("MES.FUT") + ); +} +``` + +Note: The `unwrap_or_else(|| panic!(...))` in tests is acceptable — clippy deny rules apply to production code, not `#[cfg(test)]` blocks. + +**Step 2: Run test to verify it fails** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib asset_selection::tests::test_futures_baseline` + +Expected: FAIL — `futures_baseline()` method not found. + +**Step 3: Implement `futures_baseline()`** + +Add after `us_starter()` in `ml/src/asset_selection/mod.rs`: + +```rust + /// Preset: CME futures baseline universe (4 assets for limited capital). + /// + /// Training is done on full-size contracts (ES, NQ, ZN, 6E). + /// Execution routes to micro contracts where available (MES, MNQ, M6E). + #[must_use] + pub fn futures_baseline() -> Self { + Self { + assets: vec![ + UniverseAsset { + symbol: "ES.FUT".to_string(), + trading_symbol: Some("MES.FUT".to_string()), + exchange: "GLBX.MDP3".to_string(), + asset_class: AssetClass::Future, + sector: None, + min_daily_volume: 1_000_000.0, + enabled: true, + }, + UniverseAsset { + symbol: "NQ.FUT".to_string(), + trading_symbol: Some("MNQ.FUT".to_string()), + exchange: "GLBX.MDP3".to_string(), + asset_class: AssetClass::Future, + sector: None, + min_daily_volume: 500_000.0, + enabled: true, + }, + UniverseAsset { + symbol: "ZN.FUT".to_string(), + trading_symbol: None, + exchange: "GLBX.MDP3".to_string(), + asset_class: AssetClass::Future, + sector: None, + min_daily_volume: 500_000.0, + enabled: true, + }, + UniverseAsset { + symbol: "6E.FUT".to_string(), + trading_symbol: Some("M6E.FUT".to_string()), + exchange: "GLBX.MDP3".to_string(), + asset_class: AssetClass::Future, + sector: None, + min_daily_volume: 200_000.0, + enabled: true, + }, + ], + } + } +``` + +**Step 4: Run tests to verify all pass** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib asset_selection::tests` + +Expected: ALL PASS (13 tests — 9 existing + 4 new). + +**Step 5: Commit** + +```bash +git add ml/src/asset_selection/mod.rs +git commit -m "feat(ml): add futures_baseline() preset — ES, NQ, ZN, 6E with micro mapping" +``` + +--- + +### Task 4: Add TOML Config Loading (`from_config`) + +**Files:** +- Modify: `ml/Cargo.toml` — add `toml` dependency +- Modify: `ml/src/asset_selection/mod.rs` — add `from_config()` method and TOML intermediate structs + +**Step 1: Add `toml` dependency** + +In `ml/Cargo.toml`, add after `serde_yaml = "0.9"` (line 49): + +```toml +toml.workspace = true +``` + +**Step 2: Write the failing test** + +Add to `ml/src/asset_selection/mod.rs` tests: + +```rust +#[test] +fn test_from_config_toml() { + let toml_content = r#" +[universe] +name = "test-universe" +description = "Test" + +[[symbols]] +symbol = "ES.FUT" +trading_symbol = "MES.FUT" +exchange = "GLBX.MDP3" +asset_class = "Future" +min_daily_volume = 1000000.0 + +[[symbols]] +symbol = "ZN.FUT" +exchange = "GLBX.MDP3" +asset_class = "Future" +min_daily_volume = 500000.0 +"#; + let dir = tempfile::tempdir().unwrap_or_else(|e| panic!("tmpdir: {e}")); + let path = dir.path().join("universe.toml"); + std::fs::write(&path, toml_content).unwrap_or_else(|e| panic!("write: {e}")); + + let universe = AssetUniverse::from_config(&path); + assert!(universe.is_ok(), "from_config failed: {:?}", universe.err()); + let universe = universe.unwrap_or_else(|_| AssetUniverse::new()); + assert_eq!(universe.assets.len(), 2); + assert_eq!( + universe.find("ES.FUT").map(|a| a.execution_symbol()), + Some("MES.FUT") + ); + assert_eq!( + universe.find("ZN.FUT").map(|a| a.trading_symbol.as_deref()), + Some(None) + ); +} + +#[test] +fn test_from_config_missing_file() { + let result = AssetUniverse::from_config(std::path::Path::new("/nonexistent/path.toml")); + assert!(result.is_err()); +} + +#[test] +fn test_from_config_invalid_toml() { + let dir = tempfile::tempdir().unwrap_or_else(|e| panic!("tmpdir: {e}")); + let path = dir.path().join("bad.toml"); + std::fs::write(&path, "not valid toml {{{{").unwrap_or_else(|e| panic!("write: {e}")); + let result = AssetUniverse::from_config(&path); + assert!(result.is_err()); +} +``` + +**Step 3: Run test to verify it fails** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib asset_selection::tests::test_from_config` + +Expected: FAIL — `from_config` method not found. + +**Step 4: Implement `from_config`** + +Add to `ml/src/asset_selection/mod.rs`, at the top add the import: + +```rust +use std::path::Path; +use crate::MLError; +``` + +Add TOML deserialization structs (these are private, only used for parsing): + +```rust +// --------------------------------------------------------------------------- +// TOML config deserialization +// --------------------------------------------------------------------------- + +#[derive(Deserialize)] +struct UniverseConfigFile { + #[allow(dead_code)] + universe: UniverseConfigMeta, + symbols: Vec, +} + +#[derive(Deserialize)] +struct UniverseConfigMeta { + #[allow(dead_code)] + name: String, + #[allow(dead_code)] + description: Option, +} + +#[derive(Deserialize)] +struct SymbolConfigEntry { + symbol: String, + trading_symbol: Option, + exchange: String, + asset_class: String, + min_daily_volume: Option, +} +``` + +Add the method to `impl AssetUniverse`: + +```rust + /// Load a universe from a TOML config file. + /// + /// # Errors + /// Returns `MLError::ConfigError` on I/O or parse failures. + pub fn from_config(path: &Path) -> Result { + let contents = std::fs::read_to_string(path).map_err(|e| MLError::ConfigError { + reason: format!("Failed to read universe config {}: {e}", path.display()), + })?; + + let config: UniverseConfigFile = + toml::from_str(&contents).map_err(|e| MLError::ConfigError { + reason: format!("Failed to parse universe config {}: {e}", path.display()), + })?; + + let assets = config + .symbols + .into_iter() + .map(|s| { + let asset_class = match s.asset_class.as_str() { + "Future" | "future" => AssetClass::Future, + "Equity" | "equity" => AssetClass::Equity, + "ETF" | "etf" => AssetClass::ETF, + other => { + return Err(MLError::ConfigError { + reason: format!("Unknown asset class: {other}"), + }) + } + }; + Ok(UniverseAsset { + symbol: s.symbol, + trading_symbol: s.trading_symbol, + exchange: s.exchange, + asset_class, + sector: None, + min_daily_volume: s.min_daily_volume.unwrap_or(0.0), + enabled: true, + }) + }) + .collect::, _>>()?; + + Ok(Self { assets }) + } +``` + +**Step 5: Run tests to verify all pass** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib asset_selection::tests` + +Expected: ALL PASS (16 tests — 13 + 3 new). + +**Step 6: Run workspace check** + +Run: `SQLX_OFFLINE=true cargo check --workspace` + +Expected: No errors. + +**Step 7: Commit** + +```bash +git add ml/Cargo.toml ml/src/asset_selection/mod.rs +git commit -m "feat(ml): add AssetUniverse::from_config() for TOML config loading" +``` + +--- + +### Task 5: Add `DatasetSpec::from_universe()` Pipeline Wiring + +**Files:** +- Modify: `ml/src/data_pipeline/mod.rs` — add `from_universe()` method to `DatasetSpec` + +**Step 1: Write the failing test** + +Add to `ml/src/data_pipeline/mod.rs` tests: + +```rust +#[test] +fn test_dataset_spec_from_universe() { + let universe = crate::asset_selection::AssetUniverse::futures_baseline(); + let spec = DatasetSpec::from_universe(&universe, DatasetMode::Dev); + + assert_eq!(spec.symbols.len(), 4); + assert!(spec.name.contains("Dev")); + assert_eq!(spec.bar_size, BarSize::OneMinute); + assert!(spec.split_ratio.is_valid()); + + // Verify symbols match universe + let symbol_names: Vec<&str> = spec.symbols.iter().map(|s| s.symbol.as_str()).collect(); + assert!(symbol_names.contains(&"ES.FUT")); + assert!(symbol_names.contains(&"NQ.FUT")); + assert!(symbol_names.contains(&"ZN.FUT")); + assert!(symbol_names.contains(&"6E.FUT")); + + // All should be Future class + for sym in &spec.symbols { + assert_eq!(sym.asset_class, AssetClass::Future); + } +} + +#[test] +fn test_dataset_spec_from_universe_modes() { + let universe = crate::asset_selection::AssetUniverse::futures_baseline(); + + let dev = DatasetSpec::from_universe(&universe, DatasetMode::Dev); + let backtest = DatasetSpec::from_universe(&universe, DatasetMode::Backtest); + let full = DatasetSpec::from_universe(&universe, DatasetMode::Full); + + // Dev < Backtest < Full in estimated bars + assert!(dev.estimated_total_bars() < backtest.estimated_total_bars()); + assert!(backtest.estimated_total_bars() < full.estimated_total_bars()); +} + +#[test] +fn test_dataset_spec_from_universe_disabled_filtered() { + let mut universe = crate::asset_selection::AssetUniverse::futures_baseline(); + // Disable one asset + if let Some(first) = universe.assets.get_mut(0) { + first.enabled = false; + } + let spec = DatasetSpec::from_universe(&universe, DatasetMode::Dev); + // Only 3 symbols because one was disabled + assert_eq!(spec.symbols.len(), 3); +} +``` + +**Step 2: Run test to verify it fails** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib data_pipeline::tests::test_dataset_spec_from_universe` + +Expected: FAIL — `from_universe()` method not found. + +**Step 3: Implement `from_universe()`** + +Add to `impl DatasetSpec` in `ml/src/data_pipeline/mod.rs`: + +```rust + /// Create a spec from an asset universe and dataset mode. + /// + /// Only includes enabled assets from the universe. + #[must_use] + pub fn from_universe( + universe: &crate::asset_selection::AssetUniverse, + mode: DatasetMode, + ) -> Self { + let symbols = universe + .eligible() + .iter() + .map(|a| SymbolSpec { + symbol: a.symbol.clone(), + exchange: a.exchange.clone(), + asset_class: a.asset_class.clone(), + }) + .collect(); + + Self { + name: format!("futures-baseline-{:?}", mode), + symbols, + date_range: mode.to_date_range(), + bar_size: BarSize::OneMinute, + split_ratio: SplitRatio::default(), + warmup_bars: 50, + } + } +``` + +**Step 4: Run tests to verify all pass** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib data_pipeline::tests` + +Expected: ALL PASS (12 tests — 9 existing + 3 new). + +**Step 5: Commit** + +```bash +git add ml/src/data_pipeline/mod.rs +git commit -m "feat(ml): add DatasetSpec::from_universe() — wire asset selection to data pipeline" +``` + +--- + +### Task 6: Create Universe Config File + +**Files:** +- Create: `config/universe-futures-baseline.toml` + +**Step 1: Write the config file** + +Create `config/universe-futures-baseline.toml`: + +```toml +# Futures Baseline Universe +# +# 4 CME futures for initial trading with limited capital. +# Train on full-size contracts, execute on micro versions. +# Data window: 730 days (March 2024 → February 2026) +# +# Schema: OHLCV-1m via Databento (GLBX.MDP3 dataset) + +[universe] +name = "futures-baseline" +description = "CME futures baseline: 4 symbols, 730 days, OHLCV-1m" +date_range_start = "2024-03-01" +date_range_end = "2026-02-23" +bar_size = "1m" +databento_dataset = "GLBX.MDP3" +databento_schema = "ohlcv-1m" + +[[symbols]] +symbol = "ES.FUT" +trading_symbol = "MES.FUT" +exchange = "GLBX.MDP3" +asset_class = "Future" +min_daily_volume = 1000000.0 + +[[symbols]] +symbol = "NQ.FUT" +trading_symbol = "MNQ.FUT" +exchange = "GLBX.MDP3" +asset_class = "Future" +min_daily_volume = 500000.0 + +[[symbols]] +symbol = "ZN.FUT" +exchange = "GLBX.MDP3" +asset_class = "Future" +min_daily_volume = 500000.0 + +[[symbols]] +symbol = "6E.FUT" +trading_symbol = "M6E.FUT" +exchange = "GLBX.MDP3" +asset_class = "Future" +min_daily_volume = 200000.0 +``` + +**Step 2: Write integration test** + +Add to `ml/src/asset_selection/mod.rs` tests: + +```rust +#[test] +fn test_load_futures_baseline_config() { + let config_path = std::path::Path::new("../config/universe-futures-baseline.toml"); + if !config_path.exists() { + // Skip in CI where config may not be at expected relative path + return; + } + let universe = AssetUniverse::from_config(config_path); + assert!(universe.is_ok(), "Failed to load config: {:?}", universe.err()); + let universe = universe.unwrap_or_else(|_| AssetUniverse::new()); + assert_eq!(universe.assets.len(), 4); + assert_eq!( + universe.find("ES.FUT").map(|a| a.execution_symbol()), + Some("MES.FUT") + ); +} +``` + +**Step 3: Run test** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib asset_selection::tests::test_load_futures_baseline` + +Expected: PASS (loads the real config file). + +**Step 4: Commit** + +```bash +git add config/universe-futures-baseline.toml ml/src/asset_selection/mod.rs +git commit -m "feat: add futures-baseline universe config (TOML)" +``` + +--- + +### Task 7: Reorganize Data — Move Clean DBN Files to Cache + +This task physically moves files. No Rust code changes. + +**Step 1: Create directory structure and add to .gitignore** + +```bash +# Add data/cache/ to .gitignore (this is downloaded data, not checked in) +echo "" >> .gitignore +echo "# Data cache (downloaded market data, not checked in)" >> .gitignore +echo "data/cache/" >> .gitignore + +# Create cache directories +mkdir -p data/cache/ES.FUT +mkdir -p data/cache/NQ.FUT +mkdir -p data/cache/ZN.FUT +mkdir -p data/cache/6E.FUT +``` + +**Step 2: Move clean DBN files** + +```bash +# Move per-symbol daily files from ml_training/ to data/cache/ +# The filenames are like: ES.FUT_ohlcv-1m_2024-01-02.dbn +# Rename to: ohlcv-1m_2024-01-02.dbn (strip symbol prefix) + +cd test_data/real/databento/ml_training + +for f in ES.FUT_ohlcv-1m_*.dbn; do + [ -f "$f" ] && mv "$f" ../../../../data/cache/ES.FUT/"${f#ES.FUT_}" +done + +for f in NQ.FUT_ohlcv-1m_*.dbn; do + [ -f "$f" ] && mv "$f" ../../../../data/cache/NQ.FUT/"${f#NQ.FUT_}" +done + +for f in ZN.FUT_ohlcv-1m_*.dbn; do + [ -f "$f" ] && mv "$f" ../../../../data/cache/ZN.FUT/"${f#ZN.FUT_}" +done + +for f in 6E.FUT_ohlcv-1m_*.dbn; do + [ -f "$f" ] && mv "$f" ../../../../data/cache/6E.FUT/"${f#6E.FUT_}" +done + +cd ../../../.. +``` + +**Step 3: Move fixture files** + +```bash +mkdir -p test_data/fixtures + +# Move small parquet files used in unit tests +mv test_data/ES_FUT_small.parquet test_data/fixtures/ 2>/dev/null || true +mv test_data/ZN_FUT_small.parquet test_data/fixtures/ 2>/dev/null || true +mv test_data/NQ_FUT_small.parquet test_data/fixtures/ 2>/dev/null || true +``` + +**Step 4: Verify file counts** + +```bash +echo "ES.FUT files:" && ls data/cache/ES.FUT/ | wc -l +echo "NQ.FUT files:" && ls data/cache/NQ.FUT/ | wc -l +echo "ZN.FUT files:" && ls data/cache/ZN.FUT/ | wc -l +echo "6E.FUT files:" && ls data/cache/6E.FUT/ | wc -l +``` + +Expected: 90 files each (360 total). + +**Step 5: Commit .gitignore change** + +```bash +git add .gitignore +git commit -m "chore: add data/cache/ to .gitignore (downloaded market data)" +``` + +--- + +### Task 8: Clean Up Stale Data + +This task deletes files that are no longer needed. No Rust code changes. + +**Step 1: Delete corrupted data** + +```bash +rm -rf test_data/real/databento/ml_training/corrupted/ +``` + +**Step 2: Delete root-level consolidated duplicates** + +These are consolidated/derived files that can be regenerated from the per-day DBN files: + +```bash +rm -f test_data/ES_FUT_180d.parquet +rm -f test_data/ES_FUT_180d.dbn +rm -f test_data/ES_FUT_180d_decompressed.dbn +rm -f test_data/NQ_FUT_180d.parquet +rm -f test_data/NQ_FUT_180d.dbn +rm -f test_data/NQ_FUT_180d_uncompressed.dbn +rm -f test_data/ZN_FUT_180d.dbn +rm -f test_data/ZN_FUT_90d.dbn +rm -f test_data/ZN_FUT_90d.parquet +rm -f test_data/ZN_FUT_90d_clean.parquet +rm -f test_data/6E_FUT_180d.dbn +rm -f test_data/6E_FUT_180d.parquet +rm -f test_data/6E_FUT_small.parquet +``` + +**Step 3: Delete stale eval/unseen data** + +```bash +rm -f test_data/ES_FUT_unseen.dbn +rm -f test_data/ES_FUT_unseen.parquet +rm -f test_data/ES_FUT_unseen_sequential.dbn +``` + +**Step 4: Delete non-universe symbols** + +```bash +rm -f test_data/real/databento/GC_continuous_ohlcv-1m_*.dbn +rm -f test_data/real/databento/ES.FUT_ohlcv-1m_2024-01-02.dbn.tmp +rm -rf test_data/real/databento/nq_180d/ +``` + +**Step 5: Delete small training fixtures that were moved or are stale** + +```bash +rm -rf test_data/real/databento/ml_training_small/ +rm -f test_data/real/databento/NQ.FUT_ohlcv-1m_2024-01-02.dbn +``` + +**Step 6: Verify remaining structure** + +```bash +echo "=== Remaining test_data ===" && find test_data -type f | sort +echo "=== data/cache ===" && find data/cache -type f | wc -l +``` + +Expected remaining: +- `test_data/mbp10/` — 14 files (keep for Phase 2) +- `test_data/fixtures/` — 3 small parquet files +- `data/cache/` — 360 DBN files (90 per symbol) + +**Step 7: Verify no tests are broken by deleted files** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib` + +Expected: ALL PASS. Unit tests use synthetic data, not the deleted files. If any test fails due to a missing file path, fix the test path to point to `test_data/fixtures/` or `data/cache/`. + +**Step 8: No git commit for data moves** — data files are in `.gitignore` already (test_data is not tracked, data/cache is now gitignored). Only commit if any test files needed path fixes. + +--- + +### Task 9: Generate Cache Manifest for Existing Data + +**Files:** +- Modify: `ml/src/data_pipeline/mod.rs` or add a test that generates the manifest + +**Step 1: Write a test that builds and saves a manifest from on-disk files** + +Add to `ml/src/data_pipeline/cache.rs` tests: + +```rust +#[test] +fn test_build_manifest_from_disk() { + use chrono::Utc; + use std::path::Path; + + let cache_dir = Path::new("../data/cache"); + if !cache_dir.exists() { + // Skip if data hasn't been moved yet + return; + } + + let mut manifest = CacheManifest::new(); + + // Scan each symbol directory + for symbol in &["ES.FUT", "NQ.FUT", "ZN.FUT", "6E.FUT"] { + let symbol_dir = cache_dir.join(symbol); + if !symbol_dir.exists() { + continue; + } + + let mut files: Vec<_> = std::fs::read_dir(&symbol_dir) + .into_iter() + .flatten() + .filter_map(|e| e.ok()) + .filter(|e| { + e.path() + .extension() + .map(|ext| ext == "dbn") + .unwrap_or(false) + }) + .collect(); + files.sort_by_key(|e| e.file_name()); + + for entry in &files { + let fname = entry.file_name(); + let fname_str = fname.to_string_lossy(); + // Parse date from filename: ohlcv-1m_2024-01-02.dbn + if let Some(date_str) = fname_str + .strip_prefix("ohlcv-1m_") + .and_then(|s| s.strip_suffix(".dbn")) + { + if let Ok(date) = NaiveDate::parse_from_str(date_str, "%Y-%m-%d") { + manifest.add_range( + symbol, + "GLBX.MDP3", + super::BarSize::OneMinute, + CachedRange { + start: date, + end: date, + bar_count: 390, // Approximate for 1m bars + file_path: entry.path().strip_prefix(cache_dir) + .unwrap_or(&entry.path()) + .to_path_buf(), + cached_at: Utc::now(), + }, + ); + } + } + } + } + + // Verify we found data + assert!( + !manifest.symbols.is_empty(), + "No symbols found in cache" + ); + + // Save the manifest + let result = manifest.save(cache_dir); + assert!(result.is_ok(), "Failed to save manifest: {:?}", result.err()); + + // Verify it round-trips + let loaded = CacheManifest::load(cache_dir); + assert!(loaded.is_ok()); + let loaded = loaded.unwrap_or_else(|_| CacheManifest::new()); + assert_eq!(loaded.symbols.len(), manifest.symbols.len()); +} +``` + +**Step 2: Run the test** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib data_pipeline::cache::tests::test_build_manifest_from_disk` + +Expected: PASS — creates `data/cache/manifest.json` with 4 symbols, 90 ranges each. + +**Step 3: Verify manifest contents** + +```bash +cat data/cache/manifest.json | python3 -m json.tool | head -30 +``` + +Expected: JSON with `symbols` map containing ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT, each with 90 ranges. + +**Step 4: Commit** + +```bash +git add ml/src/data_pipeline/cache.rs +git commit -m "feat(ml): add manifest generation test — builds cache manifest from on-disk DBN files" +``` + +--- + +### Task 10: Update Prelude Exports and Final Verification + +**Files:** +- Modify: `ml/src/lib.rs:1970-1974` — ensure new exports are included + +**Step 1: Verify prelude exports** + +Check that `ml/src/lib.rs` prelude already has the right exports. It should have: + +```rust +// Data pipeline +pub use crate::data_pipeline::{DatasetManager, DatasetMode, DatasetSpec, PreparedDataset}; + +// Asset selection +pub use crate::asset_selection::{ActiveSetSelector, AssetUniverse, PredictabilityScorer}; +``` + +These are already present. No changes needed if they export `AssetUniverse` (which includes `futures_baseline()` and `from_config()`). + +**Step 2: Run full test suite** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5` + +Expected: All tests pass. Count should be ~2230+ (2204 existing + ~26 new tests from this plan). + +**Step 3: Run workspace-wide check** + +Run: `SQLX_OFFLINE=true cargo check --workspace` + +Expected: No errors. + +**Step 4: Run clippy** + +Run: `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings 2>&1 | tail -5` + +Expected: No errors or warnings. + +**Step 5: Commit if any prelude changes were needed** + +```bash +# Only if changes were made +git add ml/src/lib.rs +git commit -m "chore(ml): update prelude exports for trading universe" +``` + +--- + +## Summary + +| Task | What | Tests Added | +|------|------|-------------| +| 1 | Unify `AssetClass` — remove duplicate enum | 1 | +| 2 | Add `trading_symbol` field + `execution_symbol()` | 2 | +| 3 | Add `futures_baseline()` preset | 4 | +| 4 | Add `from_config()` TOML loading | 3 | +| 5 | Add `DatasetSpec::from_universe()` | 3 | +| 6 | Create TOML config file + integration test | 1 | +| 7 | Move DBN files to `data/cache/` | 0 (file ops) | +| 8 | Delete stale/corrupted data | 0 (cleanup) | +| 9 | Generate cache manifest | 1 | +| 10 | Final verification | 0 | +| **Total** | | **15 new tests** |