docs(ml): add design doc and implementation plan for crate split
Split the monolithic ml crate (260K lines, 55s compile) into 5 crates: ml-core, ml-rl, ml-supervised, ml-infra, ml (facade). 15-task plan with full module inventory and import migration guide. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
231
docs/plans/2026-03-07-ml-crate-split-design.md
Normal file
231
docs/plans/2026-03-07-ml-crate-split-design.md
Normal file
@@ -0,0 +1,231 @@
|
||||
# ML Crate Split — Design Document
|
||||
|
||||
## Problem
|
||||
|
||||
The `ml` crate is 260K lines / 530 files. It compiles as a single `rustc` invocation taking **55.1s** — 69% of the total workspace build. Only 4 of 16 CPU cores are utilized during this phase. Incremental builds after touching any file in `ml/src/` recompile all 260K lines.
|
||||
|
||||
## Goal
|
||||
|
||||
Split `ml` into sub-crates that compile in parallel, reducing:
|
||||
- Clean build time from ~55s to ~35-40s
|
||||
- Incremental rebuild after editing a model from ~55s to ~10-15s
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
ml-core (shared types, traits, utilities, features, data)
|
||||
├─→ ml-rl (DQN + PPO + cuda_pipeline) ── parallel ──┐
|
||||
├─→ ml-supervised (TFT, Mamba, Liquid, etc.) ── parallel ──┤
|
||||
│ ↓
|
||||
└──────────────────────────→ ml-infra (hyperopt, ensemble, trainers)
|
||||
↓
|
||||
ml (thin facade, re-exports)
|
||||
```
|
||||
|
||||
`ml-rl` and `ml-supervised` compile **fully in parallel** — they have zero dependencies on each other.
|
||||
|
||||
## Crate Assignment (Every Module Accounted For)
|
||||
|
||||
### ml-core (~50K lines)
|
||||
|
||||
Shared infrastructure that 2+ other sub-crates depend on.
|
||||
|
||||
**From lib.rs (types defined inline):**
|
||||
- `Trade`, `MarketDataSnapshot`, `FeatureVector`, `IntegerTensor`, `UpdateSummary`
|
||||
- `HealthStatus`, `MarketRegime`, `CommonTypeError`, `CommonError`
|
||||
- `MLError`, `MLResult`, `UnifiedMLResult`
|
||||
- `ModelPrediction`, `InferenceResult`, `ModelMetadata`, `Features`, `MLModel` (traits)
|
||||
- `PRECISION_FACTOR`, `MAX_INFERENCE_LATENCY_US`
|
||||
- `get_training_device()`, `get_training_device_at()`
|
||||
- `HFTPerformanceProfile`, `ParallelExecutor`, `LatencyOptimizer`, `ModelRegistry` helpers
|
||||
- All `pub use` re-exports (`ErrorCategory`, `Adam`, `ModelType`)
|
||||
|
||||
**Modules (moved as-is):**
|
||||
- `error.rs` (66 lines)
|
||||
- `types/` (49 lines) — `OHLCVBar` etc.
|
||||
- `common/` (1,435 lines) — `FactoredAction`, `ExposureLevel`, `CircuitBreaker`
|
||||
- `config/` (818 lines)
|
||||
- `cuda_compat.rs` (511 lines)
|
||||
- `tensor_ops.rs` (143 lines)
|
||||
- `traits.rs` (199 lines)
|
||||
- `model.rs` (4 lines)
|
||||
- `optimizers/` (261 lines) — `Adam`
|
||||
- `gradient_accumulation.rs` (274 lines)
|
||||
- `gradient_utils.rs` (169 lines)
|
||||
- `gpu/` (836 lines) — capabilities, `DeviceConfig`, `memory_profile`
|
||||
- `safety/` (6,467 lines) — `MLSafetyManager`, tensor safety
|
||||
- `security/` (1,407 lines)
|
||||
- `memory_optimization/` (4,576 lines) — quantization (used by TFT)
|
||||
- `checkpoint/` (7,867 lines) — `Checkpointable` trait, storage
|
||||
- `training/` (1,565 lines) — `UnifiedTrainable`, `CheckpointMetadata`, `TrainingMetrics`
|
||||
- `training.rs` (85 lines)
|
||||
- `features/` (20,256 lines) — feature extraction, normalization, OFI, MBP-10
|
||||
- `feature_cache.rs` (349 lines)
|
||||
- `preprocessing.rs` (754 lines)
|
||||
- `data_loader.rs` (713 lines)
|
||||
- `data_loaders/` (4,365 lines) — `DbnSequenceLoader`, calibration
|
||||
- `data_pipeline/` (1,512 lines)
|
||||
- `data_validation/` (3,529 lines)
|
||||
- `evaluation/` (1,011 lines) — `EvaluationEngine`, `PerformanceMetrics`
|
||||
- `metrics/` (420 lines)
|
||||
- `performance.rs` (481 lines)
|
||||
- `observability/` (1,193 lines)
|
||||
|
||||
**Extracted from dqn/ → ml-core shared modules:**
|
||||
- `dqn/mixed_precision.rs` (722 lines) → `ml-core/src/mixed_precision.rs`
|
||||
- `dqn/xavier_init.rs` (200 lines) → `ml-core/src/xavier_init.rs`
|
||||
- `dqn/portfolio_tracker.rs` (931 lines) → `ml-core/src/portfolio_tracker.rs`
|
||||
- `dqn/action_space.rs` (245 lines) → `ml-core/src/action_space.rs`
|
||||
- `dqn/order_router.rs` (175 lines) → `ml-core/src/order_router.rs`
|
||||
|
||||
**NOT extracting:**
|
||||
- `dqn/circuit_breaker.rs` — already a re-export of `common/circuit_breaker.rs` (which stays in ml-core)
|
||||
- `liquid/FixedPoint` — only used by mamba (both in ml-supervised), no extraction needed
|
||||
|
||||
### ml-rl (~44K lines)
|
||||
|
||||
Reinforcement learning models. Depends on: `ml-core`.
|
||||
|
||||
- `dqn/` (33,116 lines minus 2,273 extracted = ~30,843 lines)
|
||||
- `ppo/` (14,182 lines)
|
||||
- `cuda_pipeline/` (4,431 lines) — GPU data pipeline for DQN/PPO trainers
|
||||
|
||||
**Import changes:**
|
||||
- `crate::dqn::mixed_precision::training_dtype` → `ml_core::mixed_precision::training_dtype`
|
||||
- `crate::dqn::xavier_init::linear_xavier` → `ml_core::xavier_init::linear_xavier`
|
||||
- `crate::dqn::portfolio_tracker` → `ml_core::portfolio_tracker`
|
||||
- `crate::dqn::action_space` → `ml_core::action_space`
|
||||
- `crate::dqn::order_router` → `ml_core::order_router`
|
||||
- `crate::dqn::circuit_breaker` → `ml_core::common::circuit_breaker`
|
||||
- All `crate::MLError` → `ml_core::MLError`
|
||||
- All `crate::training::unified_trainer` → `ml_core::training::unified_trainer`
|
||||
|
||||
### ml-supervised (~36K lines)
|
||||
|
||||
Supervised/generative models. Depends on: `ml-core`. **Zero dependency on ml-rl.**
|
||||
|
||||
- `tft/` (11,440 lines)
|
||||
- `mamba/` (6,465 lines)
|
||||
- `liquid/` (5,228 lines) — includes `FixedPoint`
|
||||
- `tgnn/` (4,410 lines)
|
||||
- `tlob/` (2,189 lines)
|
||||
- `kan/` (1,397 lines)
|
||||
- `xlstm/` (1,404 lines)
|
||||
- `diffusion/` (1,317 lines)
|
||||
- `transformers/` (821 lines) — shared transformer utilities
|
||||
- `flash_attention/` (1,225 lines) — shared attention implementation
|
||||
|
||||
**Import changes:**
|
||||
- Same pattern as ml-rl: `crate::X` → `ml_core::X` for shared items
|
||||
- `crate::liquid::FixedPoint` stays as `crate::liquid::FixedPoint` (both in same crate)
|
||||
- `crate::dqn::mixed_precision::training_dtype` → `ml_core::mixed_precision::training_dtype`
|
||||
|
||||
### ml-infra (~67K lines)
|
||||
|
||||
Training infrastructure. Depends on: `ml-core`, `ml-rl`, `ml-supervised`.
|
||||
|
||||
- `hyperopt/` (19,934 lines) — Bayesian optimization, adapters for all 10 models
|
||||
- `ensemble/` (13,744 lines) — inference adapters, confidence, voting
|
||||
- `trainers/` (17,772 lines) — `DQNTrainer`, `PPOTrainer`, unified training loop
|
||||
- `benchmark/` (5,969 lines) — model benchmarks
|
||||
- `deployment/` (8,190 lines) — A/B testing, hot-swap, versioning
|
||||
- `training_pipeline.rs` (847 lines)
|
||||
- `walk_forward.rs` (485 lines)
|
||||
|
||||
**Import changes:**
|
||||
- `crate::dqn::DQN` → `ml_rl::dqn::DQN`
|
||||
- `crate::ppo::PPO` → `ml_rl::ppo::PPO`
|
||||
- `crate::tft::TemporalFusionTransformer` → `ml_supervised::tft::TemporalFusionTransformer`
|
||||
- (and so on for all model types)
|
||||
- Shared items: `crate::X` → `ml_core::X`
|
||||
|
||||
### ml (thin facade, ~30K lines)
|
||||
|
||||
Re-exports everything from sub-crates. Contains modules that cross sub-crate boundaries or are only used externally.
|
||||
|
||||
**Modules (moved here):**
|
||||
- `inference.rs` (1,677 lines) — depends on TFT (supervised) + safety (core)
|
||||
- `inference_validator.rs` (528 lines)
|
||||
- `model_factory.rs` (339 lines) — creates any model type
|
||||
- `model_loader_integration.rs` (725 lines)
|
||||
- `model_registry/` (558 lines) + `model_registry.rs` (1,350 lines)
|
||||
- `registry/` (527 lines)
|
||||
- `bridge.rs` (356 lines)
|
||||
- `batch_processing.rs` (710 lines)
|
||||
- `benchmarks.rs` (733 lines)
|
||||
- `examples.rs` (892 lines)
|
||||
- `portfolio_transformer.rs` (814 lines)
|
||||
- `validation/` (4,337 lines) — depends on DQN + PPO types
|
||||
- `risk/` (5,023 lines) — ML risk (Kelly, position sizing)
|
||||
- `regime/` (4,943 lines) + `regime_detection/` (1,196 lines)
|
||||
- `microstructure/` (2,598 lines)
|
||||
- `labeling/` (3,340 lines)
|
||||
- `universe/` (1,731 lines)
|
||||
- `asset_selection/` (1,184 lines)
|
||||
- `backtesting/` (1,120 lines)
|
||||
- `paper_trading/` (389 lines)
|
||||
- `stress_testing/` (1,254 lines)
|
||||
- `integration/` (3,678 lines)
|
||||
- `tests/` (1,844 lines)
|
||||
- `gpu_benchmarks/` (21 lines)
|
||||
- `cuda_common/` (0 lines — empty)
|
||||
|
||||
**Re-exports (preserves external API):**
|
||||
```rust
|
||||
// ml/src/lib.rs
|
||||
pub use ml_core::*; // All types, errors, traits
|
||||
pub use ml_rl::dqn; // DQN module
|
||||
pub use ml_rl::ppo; // PPO module
|
||||
pub use ml_rl::cuda_pipeline; // GPU pipeline
|
||||
pub use ml_supervised::tft; // All supervised models
|
||||
pub use ml_supervised::mamba;
|
||||
// ... etc for all model modules
|
||||
pub use ml_infra::hyperopt;
|
||||
pub use ml_infra::ensemble;
|
||||
pub use ml_infra::trainers;
|
||||
// ... etc
|
||||
```
|
||||
|
||||
## Feature Flags
|
||||
|
||||
| Feature | Crate | Reason |
|
||||
|---------|-------|--------|
|
||||
| `cuda` | `ml-core` (propagated) | candle-core/cuda, candle-nn/cuda |
|
||||
| `minimal-inference` | `ml` facade | controls which sub-crates are included |
|
||||
| `financial` | `ml` facade | feature gate for services |
|
||||
| `s3-storage` | `ml-core` | AWS SDK deps |
|
||||
| `high-precision` | `ml-core` | rust_decimal feature |
|
||||
| `mimalloc-allocator` | `ml-core` | allocator |
|
||||
| `simd` | `ml-core` | SIMD features |
|
||||
| `gc` | `ml-core` | GC features |
|
||||
| `nccl` | `ml-rl` | multi-GPU for RL training |
|
||||
|
||||
## External Consumer Migration
|
||||
|
||||
All external consumers (`trading_service`, `ml_training_service`, etc.) continue to depend on `ml` (the facade). The facade re-exports everything, so **zero import changes** in downstream crates. The workspace `Cargo.toml` entry stays: `ml = { path = "crates/ml" }`.
|
||||
|
||||
## Projected Compile Times
|
||||
|
||||
| Phase | Time | Parallelism |
|
||||
|-------|------|-------------|
|
||||
| ml-core | ~15s | serial (first) |
|
||||
| ml-rl + ml-supervised | ~13s | **2-way parallel** |
|
||||
| ml-infra | ~18s | serial (after models) |
|
||||
| ml facade | ~3s | serial (last) |
|
||||
| **Total clean** | **~49s** | vs 55s current |
|
||||
| **Edit in dqn (incremental)** | **~13s** | vs 55s current |
|
||||
| **Edit in tft (incremental)** | **~10s** | vs 55s current |
|
||||
| **Edit in features (incremental)** | **~10s** | vs 55s current |
|
||||
|
||||
## Risks
|
||||
|
||||
1. **Circular dependencies**: `validation/` imports both DQN and PPO types → placed in facade
|
||||
2. **Test placement**: Tests that cross model boundaries stay in the facade crate
|
||||
3. **Feature flag propagation**: `cuda` must be forwarded from facade through sub-crates
|
||||
4. **Re-export completeness**: Must verify every `use ml::X` in the workspace still resolves
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No stubs — every function is wired to its real implementation
|
||||
- No new functionality — pure mechanical restructuring
|
||||
- No API changes for external consumers
|
||||
910
docs/plans/2026-03-07-ml-crate-split-plan.md
Normal file
910
docs/plans/2026-03-07-ml-crate-split-plan.md
Normal file
@@ -0,0 +1,910 @@
|
||||
# ML Crate Split Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Split the monolithic `ml` crate (260K lines, 55s compile) into 5 crates that compile in parallel, reducing incremental rebuilds from 55s to ~10-15s.
|
||||
|
||||
**Architecture:** 4 sub-crates (`ml-core`, `ml-rl`, `ml-supervised`, `ml-infra`) + thin `ml` facade that re-exports everything. External consumers see no API changes.
|
||||
|
||||
**Tech Stack:** Rust workspace, Cargo features, `pub use` re-exports.
|
||||
|
||||
**Reference:** See `docs/plans/2026-03-07-ml-crate-split-design.md` for full module inventory and dependency analysis.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Preparation (in worktree `.worktrees/ml-crate-split`, branch `feature/ml-crate-split`)
|
||||
|
||||
### Task 1: Verify Clean Baseline
|
||||
|
||||
**Files:** None modified
|
||||
|
||||
**Step 1: Run full workspace check**
|
||||
|
||||
Run: `cd /home/jgrusewski/Work/foxhunt/.worktrees/ml-crate-split && SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5`
|
||||
Expected: `Finished` with 0 errors
|
||||
|
||||
**Step 2: Run ml tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5`
|
||||
Expected: `test result: ok` with 2506+ tests
|
||||
|
||||
**Step 3: Record baseline**
|
||||
|
||||
Note the exact test count and build time for comparison after split.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Create Sub-Crate Directory Structure
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-core/Cargo.toml`
|
||||
- Create: `crates/ml-core/src/lib.rs`
|
||||
- Create: `crates/ml-rl/Cargo.toml`
|
||||
- Create: `crates/ml-rl/src/lib.rs`
|
||||
- Create: `crates/ml-supervised/Cargo.toml`
|
||||
- Create: `crates/ml-supervised/src/lib.rs`
|
||||
- Create: `crates/ml-infra/Cargo.toml`
|
||||
- Create: `crates/ml-infra/src/lib.rs`
|
||||
- Modify: `Cargo.toml` (workspace root — add members + workspace deps)
|
||||
|
||||
**Step 1: Create crate directories**
|
||||
|
||||
```bash
|
||||
mkdir -p crates/ml-core/src crates/ml-rl/src crates/ml-supervised/src crates/ml-infra/src
|
||||
```
|
||||
|
||||
**Step 2: Add workspace members**
|
||||
|
||||
In root `Cargo.toml`, add to `[workspace] members`:
|
||||
```toml
|
||||
"crates/ml-core",
|
||||
"crates/ml-rl",
|
||||
"crates/ml-supervised",
|
||||
"crates/ml-infra",
|
||||
```
|
||||
|
||||
And add workspace dependency entries:
|
||||
```toml
|
||||
ml-core = { path = "crates/ml-core" }
|
||||
ml-rl = { path = "crates/ml-rl" }
|
||||
ml-supervised = { path = "crates/ml-supervised" }
|
||||
ml-infra = { path = "crates/ml-infra" }
|
||||
```
|
||||
|
||||
**Step 3: Create minimal Cargo.toml for each sub-crate**
|
||||
|
||||
Each sub-crate gets a `Cargo.toml` with:
|
||||
- Same edition/rust-version as `ml`
|
||||
- Dependencies subset (only what its modules need)
|
||||
- Feature flags relevant to its modules
|
||||
- `ml-core` as dependency for `ml-rl`, `ml-supervised`, `ml-infra`
|
||||
|
||||
See the detailed Cargo.toml specs in Tasks 5-8.
|
||||
|
||||
**Step 4: Create stub lib.rs for each**
|
||||
|
||||
Each gets an empty `pub mod` list initially:
|
||||
```rust
|
||||
// crates/ml-core/src/lib.rs
|
||||
// Placeholder — modules moved in subsequent tasks
|
||||
```
|
||||
|
||||
**Step 5: Verify workspace compiles**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5`
|
||||
Expected: Compiles (empty crates are valid)
|
||||
|
||||
**Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml-core crates/ml-rl crates/ml-supervised crates/ml-infra Cargo.toml
|
||||
git commit -m "feat(ml): scaffold sub-crate directory structure for ml split"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Build ml-core (Foundation)
|
||||
|
||||
### Task 3: Extract Shared Items from dqn/ to Top-Level Modules
|
||||
|
||||
These items live in `dqn/` but are used by PPO, trainers, ensemble, hyperopt — they belong in `ml-core`.
|
||||
|
||||
**Files:**
|
||||
- Move: `crates/ml/src/dqn/mixed_precision.rs` → `crates/ml/src/mixed_precision.rs`
|
||||
- Move: `crates/ml/src/dqn/xavier_init.rs` → `crates/ml/src/xavier_init.rs`
|
||||
- Move: `crates/ml/src/dqn/portfolio_tracker.rs` → `crates/ml/src/portfolio_tracker.rs`
|
||||
- Move: `crates/ml/src/dqn/action_space.rs` → `crates/ml/src/action_space.rs`
|
||||
- Move: `crates/ml/src/dqn/order_router.rs` → `crates/ml/src/order_router.rs`
|
||||
- Modify: `crates/ml/src/dqn/mod.rs` — replace module declarations with re-exports
|
||||
- Modify: `crates/ml/src/lib.rs` — add `pub mod` for new top-level modules
|
||||
- Modify: All files importing these via `crate::dqn::*` path
|
||||
|
||||
**Step 1: Move files**
|
||||
|
||||
```bash
|
||||
cd crates/ml/src
|
||||
mv dqn/mixed_precision.rs mixed_precision.rs
|
||||
mv dqn/xavier_init.rs xavier_init.rs
|
||||
mv dqn/portfolio_tracker.rs portfolio_tracker.rs
|
||||
mv dqn/action_space.rs action_space.rs
|
||||
mv dqn/order_router.rs order_router.rs
|
||||
```
|
||||
|
||||
**Step 2: Update dqn/mod.rs**
|
||||
|
||||
Replace the old `pub mod` declarations with re-exports so existing `crate::dqn::X` paths still work:
|
||||
```rust
|
||||
// In dqn/mod.rs, REPLACE:
|
||||
// pub mod mixed_precision;
|
||||
// pub mod xavier_init;
|
||||
// pub mod portfolio_tracker;
|
||||
// pub mod action_space;
|
||||
// pub mod order_router;
|
||||
// WITH:
|
||||
pub use crate::mixed_precision;
|
||||
pub use crate::xavier_init;
|
||||
pub use crate::portfolio_tracker;
|
||||
pub use crate::action_space;
|
||||
pub use crate::order_router;
|
||||
```
|
||||
|
||||
**Step 3: Add modules to ml/lib.rs**
|
||||
|
||||
Add these lines to `lib.rs`:
|
||||
```rust
|
||||
pub mod mixed_precision;
|
||||
pub mod xavier_init;
|
||||
pub mod portfolio_tracker;
|
||||
pub mod action_space;
|
||||
pub mod order_router;
|
||||
```
|
||||
|
||||
**Step 4: Fix internal imports in moved files**
|
||||
|
||||
In each moved file, change `use crate::dqn::` imports to `use crate::` where referencing peer modules (e.g., `mixed_precision.rs` may import from `dqn` internals — fix those paths).
|
||||
|
||||
Specific fixes needed:
|
||||
- `action_space.rs`: has no `crate::dqn::` imports (only std/serde) — no changes needed
|
||||
- `xavier_init.rs`: uses `candle_core`/`candle_nn` only — no changes needed
|
||||
- `mixed_precision.rs`: uses `candle_core` only — no changes needed
|
||||
- `portfolio_tracker.rs`: check for `crate::dqn::` imports and fix
|
||||
- `order_router.rs`: imports `crate::dqn::action_space::*` → change to `crate::action_space::*`; imports `crate::common::action::*` → keep as-is
|
||||
|
||||
**Step 5: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5`
|
||||
Expected: Compiles with 0 errors (re-exports preserve all paths)
|
||||
|
||||
**Step 6: Run tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5`
|
||||
Expected: Same test count as baseline, 0 failures
|
||||
|
||||
**Step 7: Run clippy**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings 2>&1 | tail -10`
|
||||
Expected: 0 errors, 0 warnings
|
||||
|
||||
**Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add -A crates/ml/
|
||||
git commit -m "refactor(ml): extract shared items from dqn/ to top-level modules
|
||||
|
||||
Move mixed_precision, xavier_init, portfolio_tracker, action_space,
|
||||
order_router from dqn/ to ml/src/ root. These are shared infrastructure
|
||||
used by PPO, trainers, hyperopt, and ensemble — not DQN-specific.
|
||||
Re-export from dqn/mod.rs for backward compatibility."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Write ml-core Cargo.toml
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-core/Cargo.toml`
|
||||
|
||||
**Step 1: Write Cargo.toml**
|
||||
|
||||
Build by examining `crates/ml/Cargo.toml` and including ONLY dependencies used by ml-core modules. Key deps:
|
||||
- `candle-core`, `candle-nn`, `candle-optimisers` (for mixed_precision, xavier_init, Adam, gpu)
|
||||
- `common`, `config`, `data` (workspace crates)
|
||||
- `tokio`, `serde`, `thiserror`, `anyhow`, `chrono`, `tracing` (core utilities)
|
||||
- `arrow`, `parquet` (for features/)
|
||||
- `dbn`, `databento` (for data_loaders/)
|
||||
- `rust_decimal` (financial types)
|
||||
- `nalgebra`, `ndarray`, `num-traits` (for features math)
|
||||
- `dashmap`, `lru` (for caching in features/)
|
||||
- `sha2`, `hmac`, `hex` (for security/)
|
||||
- `flate2`, `memmap2` (for checkpoint/)
|
||||
- NOT: `argmin` (hyperopt only), NOT model-specific deps
|
||||
|
||||
Feature flags:
|
||||
```toml
|
||||
[features]
|
||||
default = ["cuda"]
|
||||
cuda = ["candle-core/cuda", "candle-core/cudnn", "candle-nn/cuda", "candle-nn/cudnn"]
|
||||
s3-storage = ["aws-config", "aws-sdk-s3", "aws-types", "aws-credential-types", "urlencoding"]
|
||||
high-precision = ["rust_decimal/serde-float"]
|
||||
mimalloc-allocator = ["mimalloc"]
|
||||
simd = []
|
||||
gc = []
|
||||
financial = []
|
||||
minimal-inference = []
|
||||
```
|
||||
|
||||
**Step 2: Verify it parses**
|
||||
|
||||
Run: `cargo check -p ml-core 2>&1 | head -5` (will fail on missing modules, that's OK — just verifying Cargo.toml syntax)
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/ml-core/Cargo.toml
|
||||
git commit -m "feat(ml-core): write Cargo.toml with dependency subset"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Move Core Modules to ml-core
|
||||
|
||||
This is the largest single task. Move all modules assigned to ml-core.
|
||||
|
||||
**Files:**
|
||||
- Move: ~30 modules from `crates/ml/src/` to `crates/ml-core/src/`
|
||||
- Modify: `crates/ml-core/src/lib.rs` — declare all modules
|
||||
- Modify: `crates/ml/src/lib.rs` — remove moved modules, add `pub use ml_core::*`
|
||||
|
||||
**Step 1: Move type definitions from lib.rs**
|
||||
|
||||
Extract the type definitions (`Trade`, `MarketRegime`, `CommonError`, `CommonTypeError`, `MLError`, `HealthStatus`, `MarketDataSnapshot`, `FeatureVector`, `IntegerTensor`, `UpdateSummary`, `ModelPrediction`, `InferenceResult`, `ModelMetadata`, `Features`, `MLModel`, `HFTPerformanceProfile`, `ParallelExecutor`, `LatencyOptimizer`, `ModelRegistry` helpers, and all `From` impls for `MLError`) from `crates/ml/src/lib.rs` into `crates/ml-core/src/lib.rs`.
|
||||
|
||||
Keep the crate-level attributes (`#![deny(...)]`, `#![allow(...)]`) in both — ml-core gets the canonical set, ml facade gets a minimal subset.
|
||||
|
||||
**Step 2: Move modules**
|
||||
|
||||
```bash
|
||||
# Core infrastructure
|
||||
mv crates/ml/src/error.rs crates/ml-core/src/
|
||||
mv crates/ml/src/types crates/ml-core/src/
|
||||
mv crates/ml/src/common crates/ml-core/src/
|
||||
mv crates/ml/src/config crates/ml-core/src/
|
||||
mv crates/ml/src/cuda_compat.rs crates/ml-core/src/
|
||||
mv crates/ml/src/tensor_ops.rs crates/ml-core/src/
|
||||
mv crates/ml/src/traits.rs crates/ml-core/src/
|
||||
mv crates/ml/src/model.rs crates/ml-core/src/
|
||||
mv crates/ml/src/optimizers crates/ml-core/src/
|
||||
mv crates/ml/src/gradient_accumulation.rs crates/ml-core/src/
|
||||
mv crates/ml/src/gradient_utils.rs crates/ml-core/src/
|
||||
mv crates/ml/src/gpu crates/ml-core/src/
|
||||
mv crates/ml/src/safety crates/ml-core/src/
|
||||
mv crates/ml/src/security crates/ml-core/src/
|
||||
mv crates/ml/src/memory_optimization crates/ml-core/src/
|
||||
mv crates/ml/src/checkpoint crates/ml-core/src/
|
||||
mv crates/ml/src/training crates/ml-core/src/
|
||||
mv crates/ml/src/training.rs crates/ml-core/src/
|
||||
|
||||
# Extracted shared items
|
||||
mv crates/ml/src/mixed_precision.rs crates/ml-core/src/
|
||||
mv crates/ml/src/xavier_init.rs crates/ml-core/src/
|
||||
mv crates/ml/src/portfolio_tracker.rs crates/ml-core/src/
|
||||
mv crates/ml/src/action_space.rs crates/ml-core/src/
|
||||
mv crates/ml/src/order_router.rs crates/ml-core/src/
|
||||
|
||||
# Features and data
|
||||
mv crates/ml/src/features crates/ml-core/src/
|
||||
mv crates/ml/src/feature_cache.rs crates/ml-core/src/
|
||||
mv crates/ml/src/preprocessing.rs crates/ml-core/src/
|
||||
mv crates/ml/src/data_loader.rs crates/ml-core/src/
|
||||
mv crates/ml/src/data_loaders crates/ml-core/src/
|
||||
mv crates/ml/src/data_pipeline crates/ml-core/src/
|
||||
mv crates/ml/src/data_validation crates/ml-core/src/
|
||||
mv crates/ml/src/evaluation crates/ml-core/src/
|
||||
|
||||
# Metrics and monitoring
|
||||
mv crates/ml/src/metrics crates/ml-core/src/
|
||||
mv crates/ml/src/performance.rs crates/ml-core/src/
|
||||
mv crates/ml/src/observability crates/ml-core/src/
|
||||
```
|
||||
|
||||
**Step 3: Update ml-core/src/lib.rs**
|
||||
|
||||
Declare all moved modules with `pub mod`. Include all type definitions and re-exports. Copy the relevant `use` imports (serde, chrono, rust_decimal, etc.).
|
||||
|
||||
**Step 4: Update all `use crate::` imports in moved files**
|
||||
|
||||
Every file in `crates/ml-core/src/` that uses `use crate::` — these references are now correct (they refer to ml-core's own modules). Verify no moved file imports something that stayed in `ml`.
|
||||
|
||||
Potential issues:
|
||||
- `safety/` imports from `crate::dqn::` (but mixed_precision is now in ml-core root)
|
||||
- `features/` may import from model-specific modules → fix or feature-gate
|
||||
- `data_loaders/` imports `crate::types::OHLCVBar` → still in ml-core ✓
|
||||
- `evaluation/` imports `crate::features::` → still in ml-core ✓
|
||||
- `training/unified_trainer.rs` — verify it only imports items now in ml-core
|
||||
- `checkpoint/` — verify no model-specific imports
|
||||
|
||||
Run `grep -rn 'use crate::dqn\|use crate::ppo\|use crate::tft\|use crate::mamba\|use crate::ensemble\|use crate::hyperopt\|use crate::trainers' crates/ml-core/src/` to find any imports of modules that are NOT in ml-core. These must be removed/refactored.
|
||||
|
||||
**Step 5: Update ml/src/lib.rs**
|
||||
|
||||
Replace removed `pub mod` declarations with re-exports:
|
||||
```rust
|
||||
// Add dependency
|
||||
pub use ml_core;
|
||||
|
||||
// Re-export everything from ml-core
|
||||
pub use ml_core::*;
|
||||
// Explicitly re-export modules for path compatibility
|
||||
pub use ml_core::common;
|
||||
pub use ml_core::config;
|
||||
pub use ml_core::checkpoint;
|
||||
// ... etc for all modules that external consumers access by path (ml::checkpoint::X)
|
||||
```
|
||||
|
||||
**Step 6: Add ml-core dependency to ml/Cargo.toml**
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
ml-core = { workspace = true }
|
||||
```
|
||||
|
||||
**Step 7: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml-core 2>&1 | tail -5`
|
||||
Expected: ml-core compiles
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5`
|
||||
Expected: ml compiles (re-exports resolve)
|
||||
|
||||
**Step 8: Fix any compilation errors**
|
||||
|
||||
Iterate on import fixes until both `ml-core` and `ml` compile cleanly.
|
||||
|
||||
**Step 9: Run tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5`
|
||||
Expected: All tests still pass
|
||||
|
||||
**Step 10: Run clippy**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo clippy -p ml-core -p ml -- -D warnings 2>&1 | tail -10`
|
||||
Expected: 0 errors, 0 warnings
|
||||
|
||||
**Step 11: Commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "refactor(ml): move core modules to ml-core crate
|
||||
|
||||
Move types, errors, traits, features, data loaders, checkpoint,
|
||||
safety, GPU utils, and shared infrastructure to ml-core.
|
||||
Re-export from ml facade for backward compatibility."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Build ml-rl (RL Models)
|
||||
|
||||
### Task 6: Write ml-rl Cargo.toml
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-rl/Cargo.toml`
|
||||
|
||||
**Step 1: Write Cargo.toml**
|
||||
|
||||
Dependencies: `ml-core` + `candle-core`, `candle-nn` + RL-specific deps (`crossbeam`, etc.).
|
||||
|
||||
Feature flags:
|
||||
```toml
|
||||
[features]
|
||||
default = ["cuda"]
|
||||
cuda = ["ml-core/cuda"]
|
||||
nccl = ["cuda"]
|
||||
```
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Move RL Modules to ml-rl
|
||||
|
||||
**Files:**
|
||||
- Move: `crates/ml/src/dqn/` → `crates/ml-rl/src/dqn/`
|
||||
- Move: `crates/ml/src/ppo/` → `crates/ml-rl/src/ppo/`
|
||||
- Move: `crates/ml/src/cuda_pipeline/` → `crates/ml-rl/src/cuda_pipeline/`
|
||||
- Modify: `crates/ml-rl/src/lib.rs`
|
||||
- Modify: `crates/ml/src/lib.rs`
|
||||
|
||||
**Step 1: Move modules**
|
||||
|
||||
```bash
|
||||
mv crates/ml/src/dqn crates/ml-rl/src/
|
||||
mv crates/ml/src/ppo crates/ml-rl/src/
|
||||
mv crates/ml/src/cuda_pipeline crates/ml-rl/src/
|
||||
```
|
||||
|
||||
**Step 2: Write ml-rl/src/lib.rs**
|
||||
|
||||
```rust
|
||||
// Same clippy attributes as ml-core
|
||||
pub mod dqn;
|
||||
pub mod ppo;
|
||||
pub mod cuda_pipeline;
|
||||
```
|
||||
|
||||
**Step 3: Fix all imports in ml-rl**
|
||||
|
||||
In every file under `crates/ml-rl/src/`:
|
||||
- `use crate::MLError` → `use ml_core::MLError`
|
||||
- `use crate::mixed_precision` → `use ml_core::mixed_precision`
|
||||
- `use crate::xavier_init` → `use ml_core::xavier_init`
|
||||
- `use crate::portfolio_tracker` → `use ml_core::portfolio_tracker`
|
||||
- `use crate::action_space` → `use ml_core::action_space`
|
||||
- `use crate::order_router` → `use ml_core::order_router`
|
||||
- `use crate::common::` → `use ml_core::common::`
|
||||
- `use crate::cuda_compat::` → `use ml_core::cuda_compat::`
|
||||
- `use crate::gradient_accumulation::` → `use ml_core::gradient_accumulation::`
|
||||
- `use crate::training::unified_trainer::` → `use ml_core::training::unified_trainer::`
|
||||
- `use crate::safety::` → `use ml_core::safety::`
|
||||
- `use crate::tensor_ops::` → `use ml_core::tensor_ops::`
|
||||
- `use crate::gpu::` → `use ml_core::gpu::`
|
||||
- `use crate::checkpoint::` → `use ml_core::checkpoint::`
|
||||
- `use crate::features::` → `use ml_core::features::`
|
||||
- `use crate::evaluation::` → `use ml_core::evaluation::`
|
||||
- `use crate::Adam` → `use ml_core::Adam`
|
||||
- `use crate::PRECISION_FACTOR` → `use ml_core::PRECISION_FACTOR`
|
||||
- `use crate::{MLError, MLResult}` → `use ml_core::{MLError, MLResult}`
|
||||
|
||||
Keep `use crate::dqn::` and `use crate::ppo::` as-is (these reference ml-rl's own modules).
|
||||
|
||||
Remove the re-exports from `dqn/mod.rs` added in Task 3 (they re-exported from `crate::` which was ml, now those modules are in ml-core):
|
||||
```rust
|
||||
// REMOVE these from dqn/mod.rs:
|
||||
// pub use crate::mixed_precision;
|
||||
// pub use crate::xavier_init;
|
||||
// etc.
|
||||
// REPLACE WITH:
|
||||
pub use ml_core::mixed_precision;
|
||||
pub use ml_core::xavier_init;
|
||||
pub use ml_core::portfolio_tracker;
|
||||
pub use ml_core::action_space;
|
||||
pub use ml_core::order_router;
|
||||
```
|
||||
|
||||
Also fix PPO re-exports in `ppo/mod.rs`:
|
||||
```rust
|
||||
// ppo/circuit_breaker.rs re-exports from crate::common::circuit_breaker
|
||||
// Change to: ml_core::common::circuit_breaker
|
||||
```
|
||||
|
||||
Handle cross-RL deps:
|
||||
- PPO imports `crate::dqn::circuit_breaker` → becomes `crate::dqn::circuit_breaker` (still works via dqn re-export) OR `ml_core::common::circuit_breaker` (direct)
|
||||
- PPO imports `crate::dqn::mixed_precision::training_dtype` → `ml_core::mixed_precision::training_dtype`
|
||||
- PPO imports `crate::dqn::portfolio_tracker` → `ml_core::portfolio_tracker`
|
||||
- PPO imports `crate::dqn::xavier_init::linear_xavier` → `ml_core::xavier_init::linear_xavier`
|
||||
- cuda_pipeline imports `crate::dqn::mixed_precision` → `ml_core::mixed_precision`
|
||||
- cuda_pipeline imports `crate::dqn::replay_buffer_type::GpuBatch` → `crate::dqn::replay_buffer_type::GpuBatch` (stays in ml-rl)
|
||||
|
||||
Watch for: `use crate::trainers::DQNTrainer` in dqn/ — this is a reverse dep. If it exists, it's likely in test code. If in non-test code, it needs to be removed/refactored (trainers will be in ml-infra).
|
||||
|
||||
Watch for: `use crate::hyperopt::adapters::PPOTrainer` in ppo/ — same issue. Check if test-only.
|
||||
|
||||
**Step 4: Update ml/src/lib.rs**
|
||||
|
||||
Add re-exports:
|
||||
```rust
|
||||
pub use ml_rl::dqn;
|
||||
pub use ml_rl::ppo;
|
||||
pub use ml_rl::cuda_pipeline;
|
||||
```
|
||||
|
||||
Add dependency to `ml/Cargo.toml`:
|
||||
```toml
|
||||
ml-rl = { workspace = true }
|
||||
```
|
||||
|
||||
**Step 5: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml-rl 2>&1 | tail -10`
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10`
|
||||
|
||||
**Step 6: Fix errors iteratively**
|
||||
|
||||
Keep fixing imports until both compile.
|
||||
|
||||
**Step 7: Run tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5`
|
||||
|
||||
**Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "refactor(ml): move DQN, PPO, cuda_pipeline to ml-rl crate"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Build ml-supervised (Supervised Models)
|
||||
|
||||
### Task 8: Write ml-supervised Cargo.toml
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-supervised/Cargo.toml`
|
||||
|
||||
Dependencies: `ml-core` + `candle-core`, `candle-nn`.
|
||||
|
||||
Feature flags:
|
||||
```toml
|
||||
[features]
|
||||
default = ["cuda"]
|
||||
cuda = ["ml-core/cuda"]
|
||||
```
|
||||
|
||||
**Step 1: Commit**
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Move Supervised Models to ml-supervised
|
||||
|
||||
**Files:**
|
||||
- Move: `crates/ml/src/tft/` → `crates/ml-supervised/src/tft/`
|
||||
- Move: `crates/ml/src/mamba/` → `crates/ml-supervised/src/mamba/`
|
||||
- Move: `crates/ml/src/liquid/` → `crates/ml-supervised/src/liquid/`
|
||||
- Move: `crates/ml/src/tgnn/` → `crates/ml-supervised/src/tgnn/`
|
||||
- Move: `crates/ml/src/tlob/` → `crates/ml-supervised/src/tlob/`
|
||||
- Move: `crates/ml/src/kan/` → `crates/ml-supervised/src/kan/`
|
||||
- Move: `crates/ml/src/xlstm/` → `crates/ml-supervised/src/xlstm/`
|
||||
- Move: `crates/ml/src/diffusion/` → `crates/ml-supervised/src/diffusion/`
|
||||
- Move: `crates/ml/src/transformers/` → `crates/ml-supervised/src/transformers/`
|
||||
- Move: `crates/ml/src/flash_attention/` → `crates/ml-supervised/src/flash_attention/`
|
||||
- Modify: `crates/ml-supervised/src/lib.rs`
|
||||
- Modify: `crates/ml/src/lib.rs`
|
||||
|
||||
**Step 1: Move modules**
|
||||
|
||||
```bash
|
||||
for mod in tft mamba liquid tgnn tlob kan xlstm diffusion transformers flash_attention; do
|
||||
mv crates/ml/src/$mod crates/ml-supervised/src/
|
||||
done
|
||||
```
|
||||
|
||||
**Step 2: Write ml-supervised/src/lib.rs**
|
||||
|
||||
```rust
|
||||
pub mod tft;
|
||||
pub mod mamba;
|
||||
pub mod liquid;
|
||||
pub mod tgnn;
|
||||
pub mod tlob;
|
||||
pub mod kan;
|
||||
pub mod xlstm;
|
||||
pub mod diffusion;
|
||||
pub mod transformers;
|
||||
pub mod flash_attention;
|
||||
```
|
||||
|
||||
**Step 3: Fix all imports**
|
||||
|
||||
Same pattern as Task 7: `use crate::X` → `use ml_core::X` for anything from ml-core.
|
||||
|
||||
Specific cross-model deps to handle:
|
||||
- `tft` imports `crate::liquid::FixedPoint` → `crate::liquid::FixedPoint` (both in ml-supervised ✓)
|
||||
- `mamba` imports `crate::liquid::FixedPoint` → `crate::liquid::FixedPoint` (both in ml-supervised ✓)
|
||||
- `tft` imports `crate::inference::RealInferenceError` → This module stays in ml facade! Need to either:
|
||||
- Move `RealInferenceError` type to ml-core, OR
|
||||
- Remove this import if it's only used in a few places and can be refactored
|
||||
- All models import `crate::dqn::mixed_precision::training_dtype` → `ml_core::mixed_precision::training_dtype`
|
||||
- All models import `crate::training::unified_trainer::*` → `ml_core::training::unified_trainer::*`
|
||||
|
||||
**Step 4: Update ml/src/lib.rs**
|
||||
|
||||
```rust
|
||||
pub use ml_supervised::tft;
|
||||
pub use ml_supervised::mamba;
|
||||
pub use ml_supervised::liquid;
|
||||
pub use ml_supervised::tgnn;
|
||||
pub use ml_supervised::tlob;
|
||||
pub use ml_supervised::kan;
|
||||
pub use ml_supervised::xlstm;
|
||||
pub use ml_supervised::diffusion;
|
||||
pub use ml_supervised::transformers;
|
||||
pub use ml_supervised::flash_attention;
|
||||
```
|
||||
|
||||
Add dependency: `ml-supervised = { workspace = true }`
|
||||
|
||||
**Step 5: Verify and fix**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml-supervised 2>&1 | tail -10`
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10`
|
||||
|
||||
**Step 6: Run tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5`
|
||||
|
||||
**Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "refactor(ml): move 8 supervised models to ml-supervised crate
|
||||
|
||||
TFT, Mamba, Liquid, TGNN, TLOB, KAN, xLSTM, Diffusion plus shared
|
||||
transformers and flash_attention infrastructure."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Build ml-infra (Training Infrastructure)
|
||||
|
||||
### Task 10: Write ml-infra Cargo.toml
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml-infra/Cargo.toml`
|
||||
|
||||
Dependencies: `ml-core`, `ml-rl`, `ml-supervised` + `argmin` (for hyperopt), model-specific deps.
|
||||
|
||||
Feature flags:
|
||||
```toml
|
||||
[features]
|
||||
default = ["cuda"]
|
||||
cuda = ["ml-core/cuda", "ml-rl/cuda", "ml-supervised/cuda"]
|
||||
```
|
||||
|
||||
**Step 1: Commit**
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Move Infrastructure Modules to ml-infra
|
||||
|
||||
**Files:**
|
||||
- Move: `crates/ml/src/hyperopt/` → `crates/ml-infra/src/hyperopt/`
|
||||
- Move: `crates/ml/src/ensemble/` → `crates/ml-infra/src/ensemble/`
|
||||
- Move: `crates/ml/src/trainers/` → `crates/ml-infra/src/trainers/`
|
||||
- Move: `crates/ml/src/benchmark/` → `crates/ml-infra/src/benchmark/`
|
||||
- Move: `crates/ml/src/deployment/` → `crates/ml-infra/src/deployment/`
|
||||
- Move: `crates/ml/src/training_pipeline.rs` → `crates/ml-infra/src/training_pipeline.rs`
|
||||
- Move: `crates/ml/src/walk_forward.rs` → `crates/ml-infra/src/walk_forward.rs`
|
||||
- Modify: `crates/ml-infra/src/lib.rs`
|
||||
- Modify: `crates/ml/src/lib.rs`
|
||||
|
||||
**Step 1: Move modules**
|
||||
|
||||
```bash
|
||||
for mod in hyperopt ensemble trainers benchmark deployment; do
|
||||
mv crates/ml/src/$mod crates/ml-infra/src/
|
||||
done
|
||||
mv crates/ml/src/training_pipeline.rs crates/ml-infra/src/
|
||||
mv crates/ml/src/walk_forward.rs crates/ml-infra/src/
|
||||
```
|
||||
|
||||
**Step 2: Write ml-infra/src/lib.rs**
|
||||
|
||||
```rust
|
||||
pub mod hyperopt;
|
||||
pub mod ensemble;
|
||||
pub mod trainers;
|
||||
pub mod benchmark;
|
||||
pub mod deployment;
|
||||
pub mod training_pipeline;
|
||||
pub mod walk_forward;
|
||||
```
|
||||
|
||||
**Step 3: Fix all imports**
|
||||
|
||||
This is the most complex crate — it imports from all three sub-crates:
|
||||
- `use crate::dqn::DQN` → `use ml_rl::dqn::DQN`
|
||||
- `use crate::dqn::DQNConfig` → `use ml_rl::dqn::DQNConfig`
|
||||
- `use crate::ppo::PPO` → `use ml_rl::ppo::PPO`
|
||||
- `use crate::tft::*` → `use ml_supervised::tft::*`
|
||||
- `use crate::mamba::*` → `use ml_supervised::mamba::*`
|
||||
- `use crate::liquid::*` → `use ml_supervised::liquid::*`
|
||||
- `use crate::diffusion::*` → `use ml_supervised::diffusion::*`
|
||||
- `use crate::kan::*` → `use ml_supervised::kan::*`
|
||||
- `use crate::xlstm::*` → `use ml_supervised::xlstm::*`
|
||||
- `use crate::tgnn::*` → `use ml_supervised::tgnn::*`
|
||||
- `use crate::tlob::*` → `use ml_supervised::tlob::*`
|
||||
- `use crate::cuda_pipeline::*` → `use ml_rl::cuda_pipeline::*`
|
||||
- All shared items: `use crate::X` → `use ml_core::X`
|
||||
|
||||
**Step 4: Update ml/src/lib.rs**
|
||||
|
||||
```rust
|
||||
pub use ml_infra::hyperopt;
|
||||
pub use ml_infra::ensemble;
|
||||
pub use ml_infra::trainers;
|
||||
pub use ml_infra::benchmark;
|
||||
pub use ml_infra::deployment;
|
||||
pub use ml_infra::training_pipeline;
|
||||
pub use ml_infra::walk_forward;
|
||||
```
|
||||
|
||||
Add dependency: `ml-infra = { workspace = true }`
|
||||
|
||||
**Step 5: Verify and fix**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml-infra 2>&1 | tail -10`
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10`
|
||||
|
||||
**Step 6: Run tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5`
|
||||
|
||||
**Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "refactor(ml): move hyperopt, ensemble, trainers to ml-infra crate"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Finalize Facade and Verify
|
||||
|
||||
### Task 12: Clean Up ml Facade
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/lib.rs` — final cleanup
|
||||
- Modify: `crates/ml/Cargo.toml` — update dependencies
|
||||
|
||||
**Step 1: Update ml/Cargo.toml**
|
||||
|
||||
- Add: `ml-core`, `ml-rl`, `ml-supervised`, `ml-infra` as dependencies
|
||||
- Remove: all dependencies that are now only used by sub-crates
|
||||
- Keep: dependencies used by remaining facade modules (`inference.rs`, `validation/`, `risk/`, etc.)
|
||||
|
||||
**Step 2: Verify ml/src/lib.rs has complete re-exports**
|
||||
|
||||
Every module path that external consumers use (e.g., `ml::dqn::DQNConfig`, `ml::ensemble::EnsembleCoordinator`) must resolve. Create a checklist from the external consumer import scan (Task 1 research).
|
||||
|
||||
**Step 3: Fix remaining facade module imports**
|
||||
|
||||
Modules still in ml facade (`inference.rs`, `validation/`, `risk/`, `regime/`, etc.) need import updates:
|
||||
- `use crate::dqn::` — these now come from `ml_rl` via re-export, so `use crate::dqn::` still works ✓
|
||||
- `use crate::tft::` — same via re-export from `ml_supervised` ✓
|
||||
- `use crate::ensemble::` — same via re-export from `ml_infra` ✓
|
||||
- `use crate::MLError` — from `ml_core` via `pub use ml_core::*` ✓
|
||||
|
||||
This should work because the facade re-exports everything. But verify with compilation.
|
||||
|
||||
**Step 4: Verify compilation**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5`
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "refactor(ml): finalize facade with complete re-exports"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 13: Full Workspace Verification
|
||||
|
||||
**Files:** None modified
|
||||
|
||||
**Step 1: Full workspace check**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -10`
|
||||
Expected: 0 errors across all workspace members
|
||||
|
||||
**Step 2: Full workspace clippy**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo clippy --workspace -- -D warnings 2>&1 | tail -20`
|
||||
Expected: 0 errors, 0 warnings
|
||||
|
||||
**Step 3: Run ml tests**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5`
|
||||
Expected: Same test count as baseline (2506+), 0 failures
|
||||
|
||||
**Step 4: Run sub-crate tests**
|
||||
|
||||
Run each in parallel:
|
||||
```bash
|
||||
SQLX_OFFLINE=true cargo test -p ml-core --lib 2>&1 | tail -5
|
||||
SQLX_OFFLINE=true cargo test -p ml-rl --lib 2>&1 | tail -5
|
||||
SQLX_OFFLINE=true cargo test -p ml-supervised --lib 2>&1 | tail -5
|
||||
SQLX_OFFLINE=true cargo test -p ml-infra --lib 2>&1 | tail -5
|
||||
```
|
||||
Expected: Tests distributed across sub-crates, total matches baseline
|
||||
|
||||
**Step 5: Run downstream service checks**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
SQLX_OFFLINE=true cargo check -p trading_service 2>&1 | tail -3
|
||||
SQLX_OFFLINE=true cargo check -p ml_training_service 2>&1 | tail -3
|
||||
SQLX_OFFLINE=true cargo check -p backtesting_service 2>&1 | tail -3
|
||||
SQLX_OFFLINE=true cargo check -p trading_agent_service 2>&1 | tail -3
|
||||
```
|
||||
Expected: All compile with 0 errors (they use `ml` facade, re-exports preserve paths)
|
||||
|
||||
**Step 6: Verify compile time improvement**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo build -p ml --timings 2>&1 | tail -5`
|
||||
Open the timings HTML report. Compare per-crate times against the baseline 55.1s.
|
||||
|
||||
**Step 7: Commit if any fixes were needed**
|
||||
|
||||
---
|
||||
|
||||
### Task 14: Update common dev-dependency
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/common/Cargo.toml`
|
||||
|
||||
**Step 1: Update dev-dependency**
|
||||
|
||||
`common` has `ml = { path = "../ml" }` as a dev-dependency. Update the path since ml is still at the same location:
|
||||
```toml
|
||||
ml = { path = "../ml" } # Should still work — verify
|
||||
```
|
||||
|
||||
If paths changed, update accordingly.
|
||||
|
||||
**Step 2: Verify**
|
||||
|
||||
Run: `SQLX_OFFLINE=true cargo test -p common --lib 2>&1 | tail -5`
|
||||
|
||||
**Step 3: Commit if changed**
|
||||
|
||||
---
|
||||
|
||||
### Task 15: Update Memory and Documentation
|
||||
|
||||
**Files:**
|
||||
- Modify: `/home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/MEMORY.md`
|
||||
|
||||
**Step 1: Update MEMORY.md**
|
||||
|
||||
Update the Architecture section to reflect the new 5-crate ML structure. Update the ML Model Training section with new crate paths. Remove any stale references to `crates/ml/src/dqn/mixed_precision.rs` etc.
|
||||
|
||||
**Step 2: Final commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "refactor(ml): complete crate split — 5 crates, parallel compilation
|
||||
|
||||
Split the monolithic ml crate (260K lines, 55s compile) into:
|
||||
- ml-core: shared types, traits, features, data, checkpoint, safety
|
||||
- ml-rl: DQN + PPO + cuda_pipeline
|
||||
- ml-supervised: TFT, Mamba, Liquid, TGNN, TLOB, KAN, xLSTM, Diffusion
|
||||
- ml-infra: hyperopt, ensemble, trainers, benchmark, deployment
|
||||
- ml: thin facade re-exporting everything
|
||||
|
||||
Zero API changes for downstream consumers.
|
||||
Incremental rebuild after model edit: ~55s → ~10-15s."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution Checklist
|
||||
|
||||
| Task | Description | Est. Size | Depends On |
|
||||
|------|-------------|-----------|------------|
|
||||
| 1 | Verify clean baseline | verify | — |
|
||||
| 2 | Create sub-crate dirs + workspace | scaffold | 1 |
|
||||
| 3 | Extract shared items from dqn/ | refactor | 2 |
|
||||
| 4 | Write ml-core Cargo.toml | config | 2 |
|
||||
| 5 | Move core modules to ml-core | **large** | 3, 4 |
|
||||
| 6 | Write ml-rl Cargo.toml | config | 4 |
|
||||
| 7 | Move RL modules to ml-rl | **large** | 5, 6 |
|
||||
| 8 | Write ml-supervised Cargo.toml | config | 4 |
|
||||
| 9 | Move supervised models | **large** | 5, 8 |
|
||||
| 10 | Write ml-infra Cargo.toml | config | 6, 8 |
|
||||
| 11 | Move infra modules | **large** | 7, 9, 10 |
|
||||
| 12 | Clean up ml facade | medium | 11 |
|
||||
| 13 | Full verification | verify | 12 |
|
||||
| 14 | Update common dev-dep | small | 12 |
|
||||
| 15 | Update memory + final commit | small | 13 |
|
||||
|
||||
## Critical Invariants (Verify After Every Task)
|
||||
|
||||
1. **No stubs**: Every function must be wired to its real implementation
|
||||
2. **No lost code**: Every module in the inventory must be accounted for
|
||||
3. **No API changes**: `use ml::X::Y` must work for all external consumers
|
||||
4. **Tests pass**: Same test count, 0 failures
|
||||
5. **Clippy clean**: 0 errors, 0 warnings
|
||||
6. **Feature flags work**: `--features cuda`, `--no-default-features`, `--features financial` all compile
|
||||
Reference in New Issue
Block a user