diff --git a/docs/plans/2026-03-07-ml-crate-split-design.md b/docs/plans/2026-03-07-ml-crate-split-design.md new file mode 100644 index 000000000..1bb8df69a --- /dev/null +++ b/docs/plans/2026-03-07-ml-crate-split-design.md @@ -0,0 +1,231 @@ +# ML Crate Split — Design Document + +## Problem + +The `ml` crate is 260K lines / 530 files. It compiles as a single `rustc` invocation taking **55.1s** — 69% of the total workspace build. Only 4 of 16 CPU cores are utilized during this phase. Incremental builds after touching any file in `ml/src/` recompile all 260K lines. + +## Goal + +Split `ml` into sub-crates that compile in parallel, reducing: +- Clean build time from ~55s to ~35-40s +- Incremental rebuild after editing a model from ~55s to ~10-15s + +## Architecture + +``` +ml-core (shared types, traits, utilities, features, data) + ├─→ ml-rl (DQN + PPO + cuda_pipeline) ── parallel ──┐ + ├─→ ml-supervised (TFT, Mamba, Liquid, etc.) ── parallel ──┤ + │ ↓ + └──────────────────────────→ ml-infra (hyperopt, ensemble, trainers) + ↓ + ml (thin facade, re-exports) +``` + +`ml-rl` and `ml-supervised` compile **fully in parallel** — they have zero dependencies on each other. + +## Crate Assignment (Every Module Accounted For) + +### ml-core (~50K lines) + +Shared infrastructure that 2+ other sub-crates depend on. + +**From lib.rs (types defined inline):** +- `Trade`, `MarketDataSnapshot`, `FeatureVector`, `IntegerTensor`, `UpdateSummary` +- `HealthStatus`, `MarketRegime`, `CommonTypeError`, `CommonError` +- `MLError`, `MLResult`, `UnifiedMLResult` +- `ModelPrediction`, `InferenceResult`, `ModelMetadata`, `Features`, `MLModel` (traits) +- `PRECISION_FACTOR`, `MAX_INFERENCE_LATENCY_US` +- `get_training_device()`, `get_training_device_at()` +- `HFTPerformanceProfile`, `ParallelExecutor`, `LatencyOptimizer`, `ModelRegistry` helpers +- All `pub use` re-exports (`ErrorCategory`, `Adam`, `ModelType`) + +**Modules (moved as-is):** +- `error.rs` (66 lines) +- `types/` (49 lines) — `OHLCVBar` etc. +- `common/` (1,435 lines) — `FactoredAction`, `ExposureLevel`, `CircuitBreaker` +- `config/` (818 lines) +- `cuda_compat.rs` (511 lines) +- `tensor_ops.rs` (143 lines) +- `traits.rs` (199 lines) +- `model.rs` (4 lines) +- `optimizers/` (261 lines) — `Adam` +- `gradient_accumulation.rs` (274 lines) +- `gradient_utils.rs` (169 lines) +- `gpu/` (836 lines) — capabilities, `DeviceConfig`, `memory_profile` +- `safety/` (6,467 lines) — `MLSafetyManager`, tensor safety +- `security/` (1,407 lines) +- `memory_optimization/` (4,576 lines) — quantization (used by TFT) +- `checkpoint/` (7,867 lines) — `Checkpointable` trait, storage +- `training/` (1,565 lines) — `UnifiedTrainable`, `CheckpointMetadata`, `TrainingMetrics` +- `training.rs` (85 lines) +- `features/` (20,256 lines) — feature extraction, normalization, OFI, MBP-10 +- `feature_cache.rs` (349 lines) +- `preprocessing.rs` (754 lines) +- `data_loader.rs` (713 lines) +- `data_loaders/` (4,365 lines) — `DbnSequenceLoader`, calibration +- `data_pipeline/` (1,512 lines) +- `data_validation/` (3,529 lines) +- `evaluation/` (1,011 lines) — `EvaluationEngine`, `PerformanceMetrics` +- `metrics/` (420 lines) +- `performance.rs` (481 lines) +- `observability/` (1,193 lines) + +**Extracted from dqn/ → ml-core shared modules:** +- `dqn/mixed_precision.rs` (722 lines) → `ml-core/src/mixed_precision.rs` +- `dqn/xavier_init.rs` (200 lines) → `ml-core/src/xavier_init.rs` +- `dqn/portfolio_tracker.rs` (931 lines) → `ml-core/src/portfolio_tracker.rs` +- `dqn/action_space.rs` (245 lines) → `ml-core/src/action_space.rs` +- `dqn/order_router.rs` (175 lines) → `ml-core/src/order_router.rs` + +**NOT extracting:** +- `dqn/circuit_breaker.rs` — already a re-export of `common/circuit_breaker.rs` (which stays in ml-core) +- `liquid/FixedPoint` — only used by mamba (both in ml-supervised), no extraction needed + +### ml-rl (~44K lines) + +Reinforcement learning models. Depends on: `ml-core`. + +- `dqn/` (33,116 lines minus 2,273 extracted = ~30,843 lines) +- `ppo/` (14,182 lines) +- `cuda_pipeline/` (4,431 lines) — GPU data pipeline for DQN/PPO trainers + +**Import changes:** +- `crate::dqn::mixed_precision::training_dtype` → `ml_core::mixed_precision::training_dtype` +- `crate::dqn::xavier_init::linear_xavier` → `ml_core::xavier_init::linear_xavier` +- `crate::dqn::portfolio_tracker` → `ml_core::portfolio_tracker` +- `crate::dqn::action_space` → `ml_core::action_space` +- `crate::dqn::order_router` → `ml_core::order_router` +- `crate::dqn::circuit_breaker` → `ml_core::common::circuit_breaker` +- All `crate::MLError` → `ml_core::MLError` +- All `crate::training::unified_trainer` → `ml_core::training::unified_trainer` + +### ml-supervised (~36K lines) + +Supervised/generative models. Depends on: `ml-core`. **Zero dependency on ml-rl.** + +- `tft/` (11,440 lines) +- `mamba/` (6,465 lines) +- `liquid/` (5,228 lines) — includes `FixedPoint` +- `tgnn/` (4,410 lines) +- `tlob/` (2,189 lines) +- `kan/` (1,397 lines) +- `xlstm/` (1,404 lines) +- `diffusion/` (1,317 lines) +- `transformers/` (821 lines) — shared transformer utilities +- `flash_attention/` (1,225 lines) — shared attention implementation + +**Import changes:** +- Same pattern as ml-rl: `crate::X` → `ml_core::X` for shared items +- `crate::liquid::FixedPoint` stays as `crate::liquid::FixedPoint` (both in same crate) +- `crate::dqn::mixed_precision::training_dtype` → `ml_core::mixed_precision::training_dtype` + +### ml-infra (~67K lines) + +Training infrastructure. Depends on: `ml-core`, `ml-rl`, `ml-supervised`. + +- `hyperopt/` (19,934 lines) — Bayesian optimization, adapters for all 10 models +- `ensemble/` (13,744 lines) — inference adapters, confidence, voting +- `trainers/` (17,772 lines) — `DQNTrainer`, `PPOTrainer`, unified training loop +- `benchmark/` (5,969 lines) — model benchmarks +- `deployment/` (8,190 lines) — A/B testing, hot-swap, versioning +- `training_pipeline.rs` (847 lines) +- `walk_forward.rs` (485 lines) + +**Import changes:** +- `crate::dqn::DQN` → `ml_rl::dqn::DQN` +- `crate::ppo::PPO` → `ml_rl::ppo::PPO` +- `crate::tft::TemporalFusionTransformer` → `ml_supervised::tft::TemporalFusionTransformer` +- (and so on for all model types) +- Shared items: `crate::X` → `ml_core::X` + +### ml (thin facade, ~30K lines) + +Re-exports everything from sub-crates. Contains modules that cross sub-crate boundaries or are only used externally. + +**Modules (moved here):** +- `inference.rs` (1,677 lines) — depends on TFT (supervised) + safety (core) +- `inference_validator.rs` (528 lines) +- `model_factory.rs` (339 lines) — creates any model type +- `model_loader_integration.rs` (725 lines) +- `model_registry/` (558 lines) + `model_registry.rs` (1,350 lines) +- `registry/` (527 lines) +- `bridge.rs` (356 lines) +- `batch_processing.rs` (710 lines) +- `benchmarks.rs` (733 lines) +- `examples.rs` (892 lines) +- `portfolio_transformer.rs` (814 lines) +- `validation/` (4,337 lines) — depends on DQN + PPO types +- `risk/` (5,023 lines) — ML risk (Kelly, position sizing) +- `regime/` (4,943 lines) + `regime_detection/` (1,196 lines) +- `microstructure/` (2,598 lines) +- `labeling/` (3,340 lines) +- `universe/` (1,731 lines) +- `asset_selection/` (1,184 lines) +- `backtesting/` (1,120 lines) +- `paper_trading/` (389 lines) +- `stress_testing/` (1,254 lines) +- `integration/` (3,678 lines) +- `tests/` (1,844 lines) +- `gpu_benchmarks/` (21 lines) +- `cuda_common/` (0 lines — empty) + +**Re-exports (preserves external API):** +```rust +// ml/src/lib.rs +pub use ml_core::*; // All types, errors, traits +pub use ml_rl::dqn; // DQN module +pub use ml_rl::ppo; // PPO module +pub use ml_rl::cuda_pipeline; // GPU pipeline +pub use ml_supervised::tft; // All supervised models +pub use ml_supervised::mamba; +// ... etc for all model modules +pub use ml_infra::hyperopt; +pub use ml_infra::ensemble; +pub use ml_infra::trainers; +// ... etc +``` + +## Feature Flags + +| Feature | Crate | Reason | +|---------|-------|--------| +| `cuda` | `ml-core` (propagated) | candle-core/cuda, candle-nn/cuda | +| `minimal-inference` | `ml` facade | controls which sub-crates are included | +| `financial` | `ml` facade | feature gate for services | +| `s3-storage` | `ml-core` | AWS SDK deps | +| `high-precision` | `ml-core` | rust_decimal feature | +| `mimalloc-allocator` | `ml-core` | allocator | +| `simd` | `ml-core` | SIMD features | +| `gc` | `ml-core` | GC features | +| `nccl` | `ml-rl` | multi-GPU for RL training | + +## External Consumer Migration + +All external consumers (`trading_service`, `ml_training_service`, etc.) continue to depend on `ml` (the facade). The facade re-exports everything, so **zero import changes** in downstream crates. The workspace `Cargo.toml` entry stays: `ml = { path = "crates/ml" }`. + +## Projected Compile Times + +| Phase | Time | Parallelism | +|-------|------|-------------| +| ml-core | ~15s | serial (first) | +| ml-rl + ml-supervised | ~13s | **2-way parallel** | +| ml-infra | ~18s | serial (after models) | +| ml facade | ~3s | serial (last) | +| **Total clean** | **~49s** | vs 55s current | +| **Edit in dqn (incremental)** | **~13s** | vs 55s current | +| **Edit in tft (incremental)** | **~10s** | vs 55s current | +| **Edit in features (incremental)** | **~10s** | vs 55s current | + +## Risks + +1. **Circular dependencies**: `validation/` imports both DQN and PPO types → placed in facade +2. **Test placement**: Tests that cross model boundaries stay in the facade crate +3. **Feature flag propagation**: `cuda` must be forwarded from facade through sub-crates +4. **Re-export completeness**: Must verify every `use ml::X` in the workspace still resolves + +## Non-Goals + +- No stubs — every function is wired to its real implementation +- No new functionality — pure mechanical restructuring +- No API changes for external consumers diff --git a/docs/plans/2026-03-07-ml-crate-split-plan.md b/docs/plans/2026-03-07-ml-crate-split-plan.md new file mode 100644 index 000000000..096592d71 --- /dev/null +++ b/docs/plans/2026-03-07-ml-crate-split-plan.md @@ -0,0 +1,910 @@ +# ML Crate Split Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Split the monolithic `ml` crate (260K lines, 55s compile) into 5 crates that compile in parallel, reducing incremental rebuilds from 55s to ~10-15s. + +**Architecture:** 4 sub-crates (`ml-core`, `ml-rl`, `ml-supervised`, `ml-infra`) + thin `ml` facade that re-exports everything. External consumers see no API changes. + +**Tech Stack:** Rust workspace, Cargo features, `pub use` re-exports. + +**Reference:** See `docs/plans/2026-03-07-ml-crate-split-design.md` for full module inventory and dependency analysis. + +--- + +## Phase 1: Preparation (in worktree `.worktrees/ml-crate-split`, branch `feature/ml-crate-split`) + +### Task 1: Verify Clean Baseline + +**Files:** None modified + +**Step 1: Run full workspace check** + +Run: `cd /home/jgrusewski/Work/foxhunt/.worktrees/ml-crate-split && SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5` +Expected: `Finished` with 0 errors + +**Step 2: Run ml tests** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5` +Expected: `test result: ok` with 2506+ tests + +**Step 3: Record baseline** + +Note the exact test count and build time for comparison after split. + +--- + +### Task 2: Create Sub-Crate Directory Structure + +**Files:** +- Create: `crates/ml-core/Cargo.toml` +- Create: `crates/ml-core/src/lib.rs` +- Create: `crates/ml-rl/Cargo.toml` +- Create: `crates/ml-rl/src/lib.rs` +- Create: `crates/ml-supervised/Cargo.toml` +- Create: `crates/ml-supervised/src/lib.rs` +- Create: `crates/ml-infra/Cargo.toml` +- Create: `crates/ml-infra/src/lib.rs` +- Modify: `Cargo.toml` (workspace root — add members + workspace deps) + +**Step 1: Create crate directories** + +```bash +mkdir -p crates/ml-core/src crates/ml-rl/src crates/ml-supervised/src crates/ml-infra/src +``` + +**Step 2: Add workspace members** + +In root `Cargo.toml`, add to `[workspace] members`: +```toml +"crates/ml-core", +"crates/ml-rl", +"crates/ml-supervised", +"crates/ml-infra", +``` + +And add workspace dependency entries: +```toml +ml-core = { path = "crates/ml-core" } +ml-rl = { path = "crates/ml-rl" } +ml-supervised = { path = "crates/ml-supervised" } +ml-infra = { path = "crates/ml-infra" } +``` + +**Step 3: Create minimal Cargo.toml for each sub-crate** + +Each sub-crate gets a `Cargo.toml` with: +- Same edition/rust-version as `ml` +- Dependencies subset (only what its modules need) +- Feature flags relevant to its modules +- `ml-core` as dependency for `ml-rl`, `ml-supervised`, `ml-infra` + +See the detailed Cargo.toml specs in Tasks 5-8. + +**Step 4: Create stub lib.rs for each** + +Each gets an empty `pub mod` list initially: +```rust +// crates/ml-core/src/lib.rs +// Placeholder — modules moved in subsequent tasks +``` + +**Step 5: Verify workspace compiles** + +Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5` +Expected: Compiles (empty crates are valid) + +**Step 6: Commit** + +```bash +git add crates/ml-core crates/ml-rl crates/ml-supervised crates/ml-infra Cargo.toml +git commit -m "feat(ml): scaffold sub-crate directory structure for ml split" +``` + +--- + +## Phase 2: Build ml-core (Foundation) + +### Task 3: Extract Shared Items from dqn/ to Top-Level Modules + +These items live in `dqn/` but are used by PPO, trainers, ensemble, hyperopt — they belong in `ml-core`. + +**Files:** +- Move: `crates/ml/src/dqn/mixed_precision.rs` → `crates/ml/src/mixed_precision.rs` +- Move: `crates/ml/src/dqn/xavier_init.rs` → `crates/ml/src/xavier_init.rs` +- Move: `crates/ml/src/dqn/portfolio_tracker.rs` → `crates/ml/src/portfolio_tracker.rs` +- Move: `crates/ml/src/dqn/action_space.rs` → `crates/ml/src/action_space.rs` +- Move: `crates/ml/src/dqn/order_router.rs` → `crates/ml/src/order_router.rs` +- Modify: `crates/ml/src/dqn/mod.rs` — replace module declarations with re-exports +- Modify: `crates/ml/src/lib.rs` — add `pub mod` for new top-level modules +- Modify: All files importing these via `crate::dqn::*` path + +**Step 1: Move files** + +```bash +cd crates/ml/src +mv dqn/mixed_precision.rs mixed_precision.rs +mv dqn/xavier_init.rs xavier_init.rs +mv dqn/portfolio_tracker.rs portfolio_tracker.rs +mv dqn/action_space.rs action_space.rs +mv dqn/order_router.rs order_router.rs +``` + +**Step 2: Update dqn/mod.rs** + +Replace the old `pub mod` declarations with re-exports so existing `crate::dqn::X` paths still work: +```rust +// In dqn/mod.rs, REPLACE: +// pub mod mixed_precision; +// pub mod xavier_init; +// pub mod portfolio_tracker; +// pub mod action_space; +// pub mod order_router; +// WITH: +pub use crate::mixed_precision; +pub use crate::xavier_init; +pub use crate::portfolio_tracker; +pub use crate::action_space; +pub use crate::order_router; +``` + +**Step 3: Add modules to ml/lib.rs** + +Add these lines to `lib.rs`: +```rust +pub mod mixed_precision; +pub mod xavier_init; +pub mod portfolio_tracker; +pub mod action_space; +pub mod order_router; +``` + +**Step 4: Fix internal imports in moved files** + +In each moved file, change `use crate::dqn::` imports to `use crate::` where referencing peer modules (e.g., `mixed_precision.rs` may import from `dqn` internals — fix those paths). + +Specific fixes needed: +- `action_space.rs`: has no `crate::dqn::` imports (only std/serde) — no changes needed +- `xavier_init.rs`: uses `candle_core`/`candle_nn` only — no changes needed +- `mixed_precision.rs`: uses `candle_core` only — no changes needed +- `portfolio_tracker.rs`: check for `crate::dqn::` imports and fix +- `order_router.rs`: imports `crate::dqn::action_space::*` → change to `crate::action_space::*`; imports `crate::common::action::*` → keep as-is + +**Step 5: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5` +Expected: Compiles with 0 errors (re-exports preserve all paths) + +**Step 6: Run tests** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5` +Expected: Same test count as baseline, 0 failures + +**Step 7: Run clippy** + +Run: `SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings 2>&1 | tail -10` +Expected: 0 errors, 0 warnings + +**Step 8: Commit** + +```bash +git add -A crates/ml/ +git commit -m "refactor(ml): extract shared items from dqn/ to top-level modules + +Move mixed_precision, xavier_init, portfolio_tracker, action_space, +order_router from dqn/ to ml/src/ root. These are shared infrastructure +used by PPO, trainers, hyperopt, and ensemble — not DQN-specific. +Re-export from dqn/mod.rs for backward compatibility." +``` + +--- + +### Task 4: Write ml-core Cargo.toml + +**Files:** +- Create: `crates/ml-core/Cargo.toml` + +**Step 1: Write Cargo.toml** + +Build by examining `crates/ml/Cargo.toml` and including ONLY dependencies used by ml-core modules. Key deps: +- `candle-core`, `candle-nn`, `candle-optimisers` (for mixed_precision, xavier_init, Adam, gpu) +- `common`, `config`, `data` (workspace crates) +- `tokio`, `serde`, `thiserror`, `anyhow`, `chrono`, `tracing` (core utilities) +- `arrow`, `parquet` (for features/) +- `dbn`, `databento` (for data_loaders/) +- `rust_decimal` (financial types) +- `nalgebra`, `ndarray`, `num-traits` (for features math) +- `dashmap`, `lru` (for caching in features/) +- `sha2`, `hmac`, `hex` (for security/) +- `flate2`, `memmap2` (for checkpoint/) +- NOT: `argmin` (hyperopt only), NOT model-specific deps + +Feature flags: +```toml +[features] +default = ["cuda"] +cuda = ["candle-core/cuda", "candle-core/cudnn", "candle-nn/cuda", "candle-nn/cudnn"] +s3-storage = ["aws-config", "aws-sdk-s3", "aws-types", "aws-credential-types", "urlencoding"] +high-precision = ["rust_decimal/serde-float"] +mimalloc-allocator = ["mimalloc"] +simd = [] +gc = [] +financial = [] +minimal-inference = [] +``` + +**Step 2: Verify it parses** + +Run: `cargo check -p ml-core 2>&1 | head -5` (will fail on missing modules, that's OK — just verifying Cargo.toml syntax) + +**Step 3: Commit** + +```bash +git add crates/ml-core/Cargo.toml +git commit -m "feat(ml-core): write Cargo.toml with dependency subset" +``` + +--- + +### Task 5: Move Core Modules to ml-core + +This is the largest single task. Move all modules assigned to ml-core. + +**Files:** +- Move: ~30 modules from `crates/ml/src/` to `crates/ml-core/src/` +- Modify: `crates/ml-core/src/lib.rs` — declare all modules +- Modify: `crates/ml/src/lib.rs` — remove moved modules, add `pub use ml_core::*` + +**Step 1: Move type definitions from lib.rs** + +Extract the type definitions (`Trade`, `MarketRegime`, `CommonError`, `CommonTypeError`, `MLError`, `HealthStatus`, `MarketDataSnapshot`, `FeatureVector`, `IntegerTensor`, `UpdateSummary`, `ModelPrediction`, `InferenceResult`, `ModelMetadata`, `Features`, `MLModel`, `HFTPerformanceProfile`, `ParallelExecutor`, `LatencyOptimizer`, `ModelRegistry` helpers, and all `From` impls for `MLError`) from `crates/ml/src/lib.rs` into `crates/ml-core/src/lib.rs`. + +Keep the crate-level attributes (`#![deny(...)]`, `#![allow(...)]`) in both — ml-core gets the canonical set, ml facade gets a minimal subset. + +**Step 2: Move modules** + +```bash +# Core infrastructure +mv crates/ml/src/error.rs crates/ml-core/src/ +mv crates/ml/src/types crates/ml-core/src/ +mv crates/ml/src/common crates/ml-core/src/ +mv crates/ml/src/config crates/ml-core/src/ +mv crates/ml/src/cuda_compat.rs crates/ml-core/src/ +mv crates/ml/src/tensor_ops.rs crates/ml-core/src/ +mv crates/ml/src/traits.rs crates/ml-core/src/ +mv crates/ml/src/model.rs crates/ml-core/src/ +mv crates/ml/src/optimizers crates/ml-core/src/ +mv crates/ml/src/gradient_accumulation.rs crates/ml-core/src/ +mv crates/ml/src/gradient_utils.rs crates/ml-core/src/ +mv crates/ml/src/gpu crates/ml-core/src/ +mv crates/ml/src/safety crates/ml-core/src/ +mv crates/ml/src/security crates/ml-core/src/ +mv crates/ml/src/memory_optimization crates/ml-core/src/ +mv crates/ml/src/checkpoint crates/ml-core/src/ +mv crates/ml/src/training crates/ml-core/src/ +mv crates/ml/src/training.rs crates/ml-core/src/ + +# Extracted shared items +mv crates/ml/src/mixed_precision.rs crates/ml-core/src/ +mv crates/ml/src/xavier_init.rs crates/ml-core/src/ +mv crates/ml/src/portfolio_tracker.rs crates/ml-core/src/ +mv crates/ml/src/action_space.rs crates/ml-core/src/ +mv crates/ml/src/order_router.rs crates/ml-core/src/ + +# Features and data +mv crates/ml/src/features crates/ml-core/src/ +mv crates/ml/src/feature_cache.rs crates/ml-core/src/ +mv crates/ml/src/preprocessing.rs crates/ml-core/src/ +mv crates/ml/src/data_loader.rs crates/ml-core/src/ +mv crates/ml/src/data_loaders crates/ml-core/src/ +mv crates/ml/src/data_pipeline crates/ml-core/src/ +mv crates/ml/src/data_validation crates/ml-core/src/ +mv crates/ml/src/evaluation crates/ml-core/src/ + +# Metrics and monitoring +mv crates/ml/src/metrics crates/ml-core/src/ +mv crates/ml/src/performance.rs crates/ml-core/src/ +mv crates/ml/src/observability crates/ml-core/src/ +``` + +**Step 3: Update ml-core/src/lib.rs** + +Declare all moved modules with `pub mod`. Include all type definitions and re-exports. Copy the relevant `use` imports (serde, chrono, rust_decimal, etc.). + +**Step 4: Update all `use crate::` imports in moved files** + +Every file in `crates/ml-core/src/` that uses `use crate::` — these references are now correct (they refer to ml-core's own modules). Verify no moved file imports something that stayed in `ml`. + +Potential issues: +- `safety/` imports from `crate::dqn::` (but mixed_precision is now in ml-core root) +- `features/` may import from model-specific modules → fix or feature-gate +- `data_loaders/` imports `crate::types::OHLCVBar` → still in ml-core ✓ +- `evaluation/` imports `crate::features::` → still in ml-core ✓ +- `training/unified_trainer.rs` — verify it only imports items now in ml-core +- `checkpoint/` — verify no model-specific imports + +Run `grep -rn 'use crate::dqn\|use crate::ppo\|use crate::tft\|use crate::mamba\|use crate::ensemble\|use crate::hyperopt\|use crate::trainers' crates/ml-core/src/` to find any imports of modules that are NOT in ml-core. These must be removed/refactored. + +**Step 5: Update ml/src/lib.rs** + +Replace removed `pub mod` declarations with re-exports: +```rust +// Add dependency +pub use ml_core; + +// Re-export everything from ml-core +pub use ml_core::*; +// Explicitly re-export modules for path compatibility +pub use ml_core::common; +pub use ml_core::config; +pub use ml_core::checkpoint; +// ... etc for all modules that external consumers access by path (ml::checkpoint::X) +``` + +**Step 6: Add ml-core dependency to ml/Cargo.toml** + +```toml +[dependencies] +ml-core = { workspace = true } +``` + +**Step 7: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p ml-core 2>&1 | tail -5` +Expected: ml-core compiles + +Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5` +Expected: ml compiles (re-exports resolve) + +**Step 8: Fix any compilation errors** + +Iterate on import fixes until both `ml-core` and `ml` compile cleanly. + +**Step 9: Run tests** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5` +Expected: All tests still pass + +**Step 10: Run clippy** + +Run: `SQLX_OFFLINE=true cargo clippy -p ml-core -p ml -- -D warnings 2>&1 | tail -10` +Expected: 0 errors, 0 warnings + +**Step 11: Commit** + +```bash +git add -A +git commit -m "refactor(ml): move core modules to ml-core crate + +Move types, errors, traits, features, data loaders, checkpoint, +safety, GPU utils, and shared infrastructure to ml-core. +Re-export from ml facade for backward compatibility." +``` + +--- + +## Phase 3: Build ml-rl (RL Models) + +### Task 6: Write ml-rl Cargo.toml + +**Files:** +- Create: `crates/ml-rl/Cargo.toml` + +**Step 1: Write Cargo.toml** + +Dependencies: `ml-core` + `candle-core`, `candle-nn` + RL-specific deps (`crossbeam`, etc.). + +Feature flags: +```toml +[features] +default = ["cuda"] +cuda = ["ml-core/cuda"] +nccl = ["cuda"] +``` + +**Step 2: Commit** + +--- + +### Task 7: Move RL Modules to ml-rl + +**Files:** +- Move: `crates/ml/src/dqn/` → `crates/ml-rl/src/dqn/` +- Move: `crates/ml/src/ppo/` → `crates/ml-rl/src/ppo/` +- Move: `crates/ml/src/cuda_pipeline/` → `crates/ml-rl/src/cuda_pipeline/` +- Modify: `crates/ml-rl/src/lib.rs` +- Modify: `crates/ml/src/lib.rs` + +**Step 1: Move modules** + +```bash +mv crates/ml/src/dqn crates/ml-rl/src/ +mv crates/ml/src/ppo crates/ml-rl/src/ +mv crates/ml/src/cuda_pipeline crates/ml-rl/src/ +``` + +**Step 2: Write ml-rl/src/lib.rs** + +```rust +// Same clippy attributes as ml-core +pub mod dqn; +pub mod ppo; +pub mod cuda_pipeline; +``` + +**Step 3: Fix all imports in ml-rl** + +In every file under `crates/ml-rl/src/`: +- `use crate::MLError` → `use ml_core::MLError` +- `use crate::mixed_precision` → `use ml_core::mixed_precision` +- `use crate::xavier_init` → `use ml_core::xavier_init` +- `use crate::portfolio_tracker` → `use ml_core::portfolio_tracker` +- `use crate::action_space` → `use ml_core::action_space` +- `use crate::order_router` → `use ml_core::order_router` +- `use crate::common::` → `use ml_core::common::` +- `use crate::cuda_compat::` → `use ml_core::cuda_compat::` +- `use crate::gradient_accumulation::` → `use ml_core::gradient_accumulation::` +- `use crate::training::unified_trainer::` → `use ml_core::training::unified_trainer::` +- `use crate::safety::` → `use ml_core::safety::` +- `use crate::tensor_ops::` → `use ml_core::tensor_ops::` +- `use crate::gpu::` → `use ml_core::gpu::` +- `use crate::checkpoint::` → `use ml_core::checkpoint::` +- `use crate::features::` → `use ml_core::features::` +- `use crate::evaluation::` → `use ml_core::evaluation::` +- `use crate::Adam` → `use ml_core::Adam` +- `use crate::PRECISION_FACTOR` → `use ml_core::PRECISION_FACTOR` +- `use crate::{MLError, MLResult}` → `use ml_core::{MLError, MLResult}` + +Keep `use crate::dqn::` and `use crate::ppo::` as-is (these reference ml-rl's own modules). + +Remove the re-exports from `dqn/mod.rs` added in Task 3 (they re-exported from `crate::` which was ml, now those modules are in ml-core): +```rust +// REMOVE these from dqn/mod.rs: +// pub use crate::mixed_precision; +// pub use crate::xavier_init; +// etc. +// REPLACE WITH: +pub use ml_core::mixed_precision; +pub use ml_core::xavier_init; +pub use ml_core::portfolio_tracker; +pub use ml_core::action_space; +pub use ml_core::order_router; +``` + +Also fix PPO re-exports in `ppo/mod.rs`: +```rust +// ppo/circuit_breaker.rs re-exports from crate::common::circuit_breaker +// Change to: ml_core::common::circuit_breaker +``` + +Handle cross-RL deps: +- PPO imports `crate::dqn::circuit_breaker` → becomes `crate::dqn::circuit_breaker` (still works via dqn re-export) OR `ml_core::common::circuit_breaker` (direct) +- PPO imports `crate::dqn::mixed_precision::training_dtype` → `ml_core::mixed_precision::training_dtype` +- PPO imports `crate::dqn::portfolio_tracker` → `ml_core::portfolio_tracker` +- PPO imports `crate::dqn::xavier_init::linear_xavier` → `ml_core::xavier_init::linear_xavier` +- cuda_pipeline imports `crate::dqn::mixed_precision` → `ml_core::mixed_precision` +- cuda_pipeline imports `crate::dqn::replay_buffer_type::GpuBatch` → `crate::dqn::replay_buffer_type::GpuBatch` (stays in ml-rl) + +Watch for: `use crate::trainers::DQNTrainer` in dqn/ — this is a reverse dep. If it exists, it's likely in test code. If in non-test code, it needs to be removed/refactored (trainers will be in ml-infra). + +Watch for: `use crate::hyperopt::adapters::PPOTrainer` in ppo/ — same issue. Check if test-only. + +**Step 4: Update ml/src/lib.rs** + +Add re-exports: +```rust +pub use ml_rl::dqn; +pub use ml_rl::ppo; +pub use ml_rl::cuda_pipeline; +``` + +Add dependency to `ml/Cargo.toml`: +```toml +ml-rl = { workspace = true } +``` + +**Step 5: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p ml-rl 2>&1 | tail -10` +Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10` + +**Step 6: Fix errors iteratively** + +Keep fixing imports until both compile. + +**Step 7: Run tests** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5` + +**Step 8: Commit** + +```bash +git add -A +git commit -m "refactor(ml): move DQN, PPO, cuda_pipeline to ml-rl crate" +``` + +--- + +## Phase 4: Build ml-supervised (Supervised Models) + +### Task 8: Write ml-supervised Cargo.toml + +**Files:** +- Create: `crates/ml-supervised/Cargo.toml` + +Dependencies: `ml-core` + `candle-core`, `candle-nn`. + +Feature flags: +```toml +[features] +default = ["cuda"] +cuda = ["ml-core/cuda"] +``` + +**Step 1: Commit** + +--- + +### Task 9: Move Supervised Models to ml-supervised + +**Files:** +- Move: `crates/ml/src/tft/` → `crates/ml-supervised/src/tft/` +- Move: `crates/ml/src/mamba/` → `crates/ml-supervised/src/mamba/` +- Move: `crates/ml/src/liquid/` → `crates/ml-supervised/src/liquid/` +- Move: `crates/ml/src/tgnn/` → `crates/ml-supervised/src/tgnn/` +- Move: `crates/ml/src/tlob/` → `crates/ml-supervised/src/tlob/` +- Move: `crates/ml/src/kan/` → `crates/ml-supervised/src/kan/` +- Move: `crates/ml/src/xlstm/` → `crates/ml-supervised/src/xlstm/` +- Move: `crates/ml/src/diffusion/` → `crates/ml-supervised/src/diffusion/` +- Move: `crates/ml/src/transformers/` → `crates/ml-supervised/src/transformers/` +- Move: `crates/ml/src/flash_attention/` → `crates/ml-supervised/src/flash_attention/` +- Modify: `crates/ml-supervised/src/lib.rs` +- Modify: `crates/ml/src/lib.rs` + +**Step 1: Move modules** + +```bash +for mod in tft mamba liquid tgnn tlob kan xlstm diffusion transformers flash_attention; do + mv crates/ml/src/$mod crates/ml-supervised/src/ +done +``` + +**Step 2: Write ml-supervised/src/lib.rs** + +```rust +pub mod tft; +pub mod mamba; +pub mod liquid; +pub mod tgnn; +pub mod tlob; +pub mod kan; +pub mod xlstm; +pub mod diffusion; +pub mod transformers; +pub mod flash_attention; +``` + +**Step 3: Fix all imports** + +Same pattern as Task 7: `use crate::X` → `use ml_core::X` for anything from ml-core. + +Specific cross-model deps to handle: +- `tft` imports `crate::liquid::FixedPoint` → `crate::liquid::FixedPoint` (both in ml-supervised ✓) +- `mamba` imports `crate::liquid::FixedPoint` → `crate::liquid::FixedPoint` (both in ml-supervised ✓) +- `tft` imports `crate::inference::RealInferenceError` → This module stays in ml facade! Need to either: + - Move `RealInferenceError` type to ml-core, OR + - Remove this import if it's only used in a few places and can be refactored +- All models import `crate::dqn::mixed_precision::training_dtype` → `ml_core::mixed_precision::training_dtype` +- All models import `crate::training::unified_trainer::*` → `ml_core::training::unified_trainer::*` + +**Step 4: Update ml/src/lib.rs** + +```rust +pub use ml_supervised::tft; +pub use ml_supervised::mamba; +pub use ml_supervised::liquid; +pub use ml_supervised::tgnn; +pub use ml_supervised::tlob; +pub use ml_supervised::kan; +pub use ml_supervised::xlstm; +pub use ml_supervised::diffusion; +pub use ml_supervised::transformers; +pub use ml_supervised::flash_attention; +``` + +Add dependency: `ml-supervised = { workspace = true }` + +**Step 5: Verify and fix** + +Run: `SQLX_OFFLINE=true cargo check -p ml-supervised 2>&1 | tail -10` +Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10` + +**Step 6: Run tests** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5` + +**Step 7: Commit** + +```bash +git add -A +git commit -m "refactor(ml): move 8 supervised models to ml-supervised crate + +TFT, Mamba, Liquid, TGNN, TLOB, KAN, xLSTM, Diffusion plus shared +transformers and flash_attention infrastructure." +``` + +--- + +## Phase 5: Build ml-infra (Training Infrastructure) + +### Task 10: Write ml-infra Cargo.toml + +**Files:** +- Create: `crates/ml-infra/Cargo.toml` + +Dependencies: `ml-core`, `ml-rl`, `ml-supervised` + `argmin` (for hyperopt), model-specific deps. + +Feature flags: +```toml +[features] +default = ["cuda"] +cuda = ["ml-core/cuda", "ml-rl/cuda", "ml-supervised/cuda"] +``` + +**Step 1: Commit** + +--- + +### Task 11: Move Infrastructure Modules to ml-infra + +**Files:** +- Move: `crates/ml/src/hyperopt/` → `crates/ml-infra/src/hyperopt/` +- Move: `crates/ml/src/ensemble/` → `crates/ml-infra/src/ensemble/` +- Move: `crates/ml/src/trainers/` → `crates/ml-infra/src/trainers/` +- Move: `crates/ml/src/benchmark/` → `crates/ml-infra/src/benchmark/` +- Move: `crates/ml/src/deployment/` → `crates/ml-infra/src/deployment/` +- Move: `crates/ml/src/training_pipeline.rs` → `crates/ml-infra/src/training_pipeline.rs` +- Move: `crates/ml/src/walk_forward.rs` → `crates/ml-infra/src/walk_forward.rs` +- Modify: `crates/ml-infra/src/lib.rs` +- Modify: `crates/ml/src/lib.rs` + +**Step 1: Move modules** + +```bash +for mod in hyperopt ensemble trainers benchmark deployment; do + mv crates/ml/src/$mod crates/ml-infra/src/ +done +mv crates/ml/src/training_pipeline.rs crates/ml-infra/src/ +mv crates/ml/src/walk_forward.rs crates/ml-infra/src/ +``` + +**Step 2: Write ml-infra/src/lib.rs** + +```rust +pub mod hyperopt; +pub mod ensemble; +pub mod trainers; +pub mod benchmark; +pub mod deployment; +pub mod training_pipeline; +pub mod walk_forward; +``` + +**Step 3: Fix all imports** + +This is the most complex crate — it imports from all three sub-crates: +- `use crate::dqn::DQN` → `use ml_rl::dqn::DQN` +- `use crate::dqn::DQNConfig` → `use ml_rl::dqn::DQNConfig` +- `use crate::ppo::PPO` → `use ml_rl::ppo::PPO` +- `use crate::tft::*` → `use ml_supervised::tft::*` +- `use crate::mamba::*` → `use ml_supervised::mamba::*` +- `use crate::liquid::*` → `use ml_supervised::liquid::*` +- `use crate::diffusion::*` → `use ml_supervised::diffusion::*` +- `use crate::kan::*` → `use ml_supervised::kan::*` +- `use crate::xlstm::*` → `use ml_supervised::xlstm::*` +- `use crate::tgnn::*` → `use ml_supervised::tgnn::*` +- `use crate::tlob::*` → `use ml_supervised::tlob::*` +- `use crate::cuda_pipeline::*` → `use ml_rl::cuda_pipeline::*` +- All shared items: `use crate::X` → `use ml_core::X` + +**Step 4: Update ml/src/lib.rs** + +```rust +pub use ml_infra::hyperopt; +pub use ml_infra::ensemble; +pub use ml_infra::trainers; +pub use ml_infra::benchmark; +pub use ml_infra::deployment; +pub use ml_infra::training_pipeline; +pub use ml_infra::walk_forward; +``` + +Add dependency: `ml-infra = { workspace = true }` + +**Step 5: Verify and fix** + +Run: `SQLX_OFFLINE=true cargo check -p ml-infra 2>&1 | tail -10` +Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10` + +**Step 6: Run tests** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5` + +**Step 7: Commit** + +```bash +git add -A +git commit -m "refactor(ml): move hyperopt, ensemble, trainers to ml-infra crate" +``` + +--- + +## Phase 6: Finalize Facade and Verify + +### Task 12: Clean Up ml Facade + +**Files:** +- Modify: `crates/ml/src/lib.rs` — final cleanup +- Modify: `crates/ml/Cargo.toml` — update dependencies + +**Step 1: Update ml/Cargo.toml** + +- Add: `ml-core`, `ml-rl`, `ml-supervised`, `ml-infra` as dependencies +- Remove: all dependencies that are now only used by sub-crates +- Keep: dependencies used by remaining facade modules (`inference.rs`, `validation/`, `risk/`, etc.) + +**Step 2: Verify ml/src/lib.rs has complete re-exports** + +Every module path that external consumers use (e.g., `ml::dqn::DQNConfig`, `ml::ensemble::EnsembleCoordinator`) must resolve. Create a checklist from the external consumer import scan (Task 1 research). + +**Step 3: Fix remaining facade module imports** + +Modules still in ml facade (`inference.rs`, `validation/`, `risk/`, `regime/`, etc.) need import updates: +- `use crate::dqn::` — these now come from `ml_rl` via re-export, so `use crate::dqn::` still works ✓ +- `use crate::tft::` — same via re-export from `ml_supervised` ✓ +- `use crate::ensemble::` — same via re-export from `ml_infra` ✓ +- `use crate::MLError` — from `ml_core` via `pub use ml_core::*` ✓ + +This should work because the facade re-exports everything. But verify with compilation. + +**Step 4: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5` + +**Step 5: Commit** + +```bash +git add -A +git commit -m "refactor(ml): finalize facade with complete re-exports" +``` + +--- + +### Task 13: Full Workspace Verification + +**Files:** None modified + +**Step 1: Full workspace check** + +Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -10` +Expected: 0 errors across all workspace members + +**Step 2: Full workspace clippy** + +Run: `SQLX_OFFLINE=true cargo clippy --workspace -- -D warnings 2>&1 | tail -20` +Expected: 0 errors, 0 warnings + +**Step 3: Run ml tests** + +Run: `SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5` +Expected: Same test count as baseline (2506+), 0 failures + +**Step 4: Run sub-crate tests** + +Run each in parallel: +```bash +SQLX_OFFLINE=true cargo test -p ml-core --lib 2>&1 | tail -5 +SQLX_OFFLINE=true cargo test -p ml-rl --lib 2>&1 | tail -5 +SQLX_OFFLINE=true cargo test -p ml-supervised --lib 2>&1 | tail -5 +SQLX_OFFLINE=true cargo test -p ml-infra --lib 2>&1 | tail -5 +``` +Expected: Tests distributed across sub-crates, total matches baseline + +**Step 5: Run downstream service checks** + +Run: +```bash +SQLX_OFFLINE=true cargo check -p trading_service 2>&1 | tail -3 +SQLX_OFFLINE=true cargo check -p ml_training_service 2>&1 | tail -3 +SQLX_OFFLINE=true cargo check -p backtesting_service 2>&1 | tail -3 +SQLX_OFFLINE=true cargo check -p trading_agent_service 2>&1 | tail -3 +``` +Expected: All compile with 0 errors (they use `ml` facade, re-exports preserve paths) + +**Step 6: Verify compile time improvement** + +Run: `SQLX_OFFLINE=true cargo build -p ml --timings 2>&1 | tail -5` +Open the timings HTML report. Compare per-crate times against the baseline 55.1s. + +**Step 7: Commit if any fixes were needed** + +--- + +### Task 14: Update common dev-dependency + +**Files:** +- Modify: `crates/common/Cargo.toml` + +**Step 1: Update dev-dependency** + +`common` has `ml = { path = "../ml" }` as a dev-dependency. Update the path since ml is still at the same location: +```toml +ml = { path = "../ml" } # Should still work — verify +``` + +If paths changed, update accordingly. + +**Step 2: Verify** + +Run: `SQLX_OFFLINE=true cargo test -p common --lib 2>&1 | tail -5` + +**Step 3: Commit if changed** + +--- + +### Task 15: Update Memory and Documentation + +**Files:** +- Modify: `/home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/MEMORY.md` + +**Step 1: Update MEMORY.md** + +Update the Architecture section to reflect the new 5-crate ML structure. Update the ML Model Training section with new crate paths. Remove any stale references to `crates/ml/src/dqn/mixed_precision.rs` etc. + +**Step 2: Final commit** + +```bash +git add -A +git commit -m "refactor(ml): complete crate split — 5 crates, parallel compilation + +Split the monolithic ml crate (260K lines, 55s compile) into: +- ml-core: shared types, traits, features, data, checkpoint, safety +- ml-rl: DQN + PPO + cuda_pipeline +- ml-supervised: TFT, Mamba, Liquid, TGNN, TLOB, KAN, xLSTM, Diffusion +- ml-infra: hyperopt, ensemble, trainers, benchmark, deployment +- ml: thin facade re-exporting everything + +Zero API changes for downstream consumers. +Incremental rebuild after model edit: ~55s → ~10-15s." +``` + +--- + +## Execution Checklist + +| Task | Description | Est. Size | Depends On | +|------|-------------|-----------|------------| +| 1 | Verify clean baseline | verify | — | +| 2 | Create sub-crate dirs + workspace | scaffold | 1 | +| 3 | Extract shared items from dqn/ | refactor | 2 | +| 4 | Write ml-core Cargo.toml | config | 2 | +| 5 | Move core modules to ml-core | **large** | 3, 4 | +| 6 | Write ml-rl Cargo.toml | config | 4 | +| 7 | Move RL modules to ml-rl | **large** | 5, 6 | +| 8 | Write ml-supervised Cargo.toml | config | 4 | +| 9 | Move supervised models | **large** | 5, 8 | +| 10 | Write ml-infra Cargo.toml | config | 6, 8 | +| 11 | Move infra modules | **large** | 7, 9, 10 | +| 12 | Clean up ml facade | medium | 11 | +| 13 | Full verification | verify | 12 | +| 14 | Update common dev-dep | small | 12 | +| 15 | Update memory + final commit | small | 13 | + +## Critical Invariants (Verify After Every Task) + +1. **No stubs**: Every function must be wired to its real implementation +2. **No lost code**: Every module in the inventory must be accounted for +3. **No API changes**: `use ml::X::Y` must work for all external consumers +4. **Tests pass**: Same test count, 0 failures +5. **Clippy clean**: 0 errors, 0 warnings +6. **Feature flags work**: `--features cuda`, `--no-default-features`, `--features financial` all compile