c208217791
Plan to reduce ml crate from 91K to ~12K LOC by extracting trainers into model sub-crates, hyperopt adapters into ml-hyperopt, and infrastructure into existing sub-crates. ml becomes an orchestration layer owning inference, model factory, and training pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3.5 KiB
3.5 KiB
ML Crate Split Phase 2 — Domain-Aligned Extraction
Date: 2026-03-08 Approach: B (domain-aligned extraction, ml becomes orchestration layer) Target: 91K LOC → ~12K LOC (87% reduction)
Context
Phase 1 (feature/ml-crate-split, merged as 121962b7) extracted types and shared
infrastructure into 23 sub-crates. The heavy implementation code (trainers, hyperopt
adapters, CUDA pipeline, model architectures) remained in the main ml crate.
Extraction Groups
Group 1: Trainers → model sub-crates (18K LOC)
| Source | Destination |
|---|---|
trainers/dqn/ (trainer, config, data_loading) |
ml-dqn |
trainers/tft/ |
ml-supervised |
trainers/ppo.rs |
ml-ppo |
trainers/online_learning.rs |
ml-core |
trainers/mod.rs (registry glue) |
stays in ml |
Group 2: Hyperopt adapters → ml-hyperopt (12.7K LOC)
Move all 12 adapters + campaign/shared_data/tests into ml-hyperopt.
ml-hyperopt gains deps on ml-dqn, ml-ppo, ml-supervised.
Group 3: Model impl remnants → model sub-crates (~5.7K LOC)
| Source | Destination |
|---|---|
ml/src/dqn/ |
merge into ml-dqn |
ml/src/tft/ (training, trainable_adapter) |
merge into ml-supervised |
ml/src/ppo/ |
merge into ml-ppo |
ml/src/liquid/, tgnn/, kan/, xlstm/, diffusion/, mamba/, tlob/ |
merge into ml-supervised |
Group 4: Infrastructure → existing sub-crates (~11K LOC)
| Source | Destination |
|---|---|
cuda_pipeline/ (4K) |
ml-core |
data_loaders/ (4.4K) + data_pipeline/ (1.5K) + training/ (1.2K) |
ml-data |
features/ remaining (4.3K) |
merge into ml-features |
ensemble/ remaining (3.4K) |
merge into ml-ensemble |
flash_attention/ (1.2K) |
ml-core |
microstructure/ (682) |
ml-features |
Group 5: New sub-crates (8.3K LOC)
| Source | Destination |
|---|---|
benchmark/ (6K) |
new ml-benchmark |
deployment/ (2.3K) |
new ml-deployment |
Group 6: Small modules → existing homes
model_registry/+registry/(1.1K) →ml-core- Thin re-export stubs (backtesting, validation, etc.) → deleted, direct imports
What stays in ml (~12K LOC — orchestration layer)
lib.rs— re-exports, From impls, prelude (~500)integration/— inference engine, performance monitor (3.7K)inference.rs+inference_validator.rs— inference orchestration (1.8K)model_factory.rs— model instantiation (~500)training_pipeline.rs— training orchestration (847)portfolio_transformer.rs— portfolio transformer (814)preprocessing.rs— data preprocessing (754)examples.rs— usage examples (892)bridge.rs— type system bridge (~400)trainers/mod.rs— trainer registry/factory (~500)batch_processing.rs,walk_forward.rs,data_loader.rs— misc glue (~600)
Dependency Graph (post-split)
ml-core (foundation)
↑
ml-dqn, ml-ppo, ml-supervised (models + trainers)
↑
ml-hyperopt (optimizer framework + all model adapters)
↑
ml (orchestration: inference, factory, pipeline)
Execution Strategy
- Work in a git worktree for isolation
- Swarm of parallel agents per extraction group
- Each group: move code, update imports, fix Cargo.toml deps, run clippy+tests
- Groups 1-3 are interdependent (trainer moves affect hyperopt adapter imports)
- Groups 4-6 are independent of each other and of 1-3
- Final: update all downstream crates (services, binaries) that import from
ml