diff --git a/docs/plans/2026-03-08-ml-crate-split-phase2-design.md b/docs/plans/2026-03-08-ml-crate-split-phase2-design.md new file mode 100644 index 000000000..a3a499649 --- /dev/null +++ b/docs/plans/2026-03-08-ml-crate-split-phase2-design.md @@ -0,0 +1,95 @@ +# ML Crate Split Phase 2 — Domain-Aligned Extraction + +**Date**: 2026-03-08 +**Approach**: B (domain-aligned extraction, ml becomes orchestration layer) +**Target**: 91K LOC → ~12K LOC (87% reduction) + +## Context + +Phase 1 (`feature/ml-crate-split`, merged as `121962b7`) extracted types and shared +infrastructure into 23 sub-crates. The heavy implementation code (trainers, hyperopt +adapters, CUDA pipeline, model architectures) remained in the main `ml` crate. + +## Extraction Groups + +### Group 1: Trainers → model sub-crates (18K LOC) + +| Source | Destination | +|--------|-------------| +| `trainers/dqn/` (trainer, config, data_loading) | `ml-dqn` | +| `trainers/tft/` | `ml-supervised` | +| `trainers/ppo.rs` | `ml-ppo` | +| `trainers/online_learning.rs` | `ml-core` | +| `trainers/mod.rs` (registry glue) | stays in `ml` | + +### Group 2: Hyperopt adapters → ml-hyperopt (12.7K LOC) + +Move all 12 adapters + campaign/shared_data/tests into `ml-hyperopt`. +`ml-hyperopt` gains deps on `ml-dqn`, `ml-ppo`, `ml-supervised`. + +### Group 3: Model impl remnants → model sub-crates (~5.7K LOC) + +| Source | Destination | +|--------|-------------| +| `ml/src/dqn/` | merge into `ml-dqn` | +| `ml/src/tft/` (training, trainable_adapter) | merge into `ml-supervised` | +| `ml/src/ppo/` | merge into `ml-ppo` | +| `ml/src/liquid/`, `tgnn/`, `kan/`, `xlstm/`, `diffusion/`, `mamba/`, `tlob/` | merge into `ml-supervised` | + +### Group 4: Infrastructure → existing sub-crates (~11K LOC) + +| Source | Destination | +|--------|-------------| +| `cuda_pipeline/` (4K) | `ml-core` | +| `data_loaders/` (4.4K) + `data_pipeline/` (1.5K) + `training/` (1.2K) | `ml-data` | +| `features/` remaining (4.3K) | merge into `ml-features` | +| `ensemble/` remaining (3.4K) | merge into `ml-ensemble` | +| `flash_attention/` (1.2K) | `ml-core` | +| `microstructure/` (682) | `ml-features` | + +### Group 5: New sub-crates (8.3K LOC) + +| Source | Destination | +|--------|-------------| +| `benchmark/` (6K) | new `ml-benchmark` | +| `deployment/` (2.3K) | new `ml-deployment` | + +### Group 6: Small modules → existing homes + +- `model_registry/` + `registry/` (1.1K) → `ml-core` +- Thin re-export stubs (backtesting, validation, etc.) → deleted, direct imports + +## What stays in ml (~12K LOC — orchestration layer) + +- `lib.rs` — re-exports, From impls, prelude (~500) +- `integration/` — inference engine, performance monitor (3.7K) +- `inference.rs` + `inference_validator.rs` — inference orchestration (1.8K) +- `model_factory.rs` — model instantiation (~500) +- `training_pipeline.rs` — training orchestration (847) +- `portfolio_transformer.rs` — portfolio transformer (814) +- `preprocessing.rs` — data preprocessing (754) +- `examples.rs` — usage examples (892) +- `bridge.rs` — type system bridge (~400) +- `trainers/mod.rs` — trainer registry/factory (~500) +- `batch_processing.rs`, `walk_forward.rs`, `data_loader.rs` — misc glue (~600) + +## Dependency Graph (post-split) + +``` +ml-core (foundation) + ↑ +ml-dqn, ml-ppo, ml-supervised (models + trainers) + ↑ +ml-hyperopt (optimizer framework + all model adapters) + ↑ +ml (orchestration: inference, factory, pipeline) +``` + +## Execution Strategy + +- Work in a git worktree for isolation +- Swarm of parallel agents per extraction group +- Each group: move code, update imports, fix Cargo.toml deps, run clippy+tests +- Groups 1-3 are interdependent (trainer moves affect hyperopt adapter imports) +- Groups 4-6 are independent of each other and of 1-3 +- Final: update all downstream crates (services, binaries) that import from `ml`