foxhunt/docs/plans/2026-03-08-ml-crate-split-phase2.md

# ML Crate Split Phase 2 — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Reduce ml crate from 91K to ~12K LOC by extracting trainers, hyperopt adapters, model impls, and infrastructure into existing sub-crates.

**Architecture:** Extract in dependency order: foundational modules first (cuda_pipeline, features, flash_attention), then model impls, then trainers (which depend on foundations), then hyperopt/ensemble adapters (which depend on trainers). `ml` becomes an orchestration layer owning inference, model factory, and training pipeline.

**Tech Stack:** Rust workspace, Cargo.toml workspace deps, `pub use` re-exports for backward compat

**Worktree:** `.worktrees/ml-split-p2` on branch `feature/ml-crate-split-phase2`

**Build command:** `SQLX_OFFLINE=true cargo check --workspace` (no DB required)
**Test command:** `SQLX_OFFLINE=true cargo test -p <crate> --lib`
**Clippy:** `SQLX_OFFLINE=true cargo clippy --workspace -- -D warnings`

---

## Dependency Order & Parallelism

```
[Parallel Group A]          [Parallel Group B]       [Parallel Group C]
Task 1: cuda_pipeline→core  Task 3: model impls→     Task 4: benchmark→
Task 1b: flash_attn→core       sub-crates               ml-benchmark
Task 1c: features→ml-feat   Task 3b: dqn/→ml-dqn     Task 4b: deployment→
Task 1d: microstructure→     Task 3c: ppo/→ml-ppo        ml-deployment
    ml-features              Task 3d: model stubs→
                                ml-supervised
         │                        │
         ▼                        ▼
[Sequential: Task 2]     [Sequential: Task 5]
data_loaders→ml-data      trainers→model sub-crates
data_pipeline→ml-data      (depends on A + B)
training/→ml-data               │
                                ▼
                        [Sequential: Task 6]
                        hyperopt adapters→ml-hyperopt
                        ensemble adapters→ml-ensemble
                        (depends on Task 5)
                                │
                                ▼
                        [Final: Task 7]
                        Cleanup ml lib.rs,
                        update services, fix all imports
```

---

### Task 1: Foundation Extractions → ml-core & ml-features

**Parallel subtasks — no interdependencies.**

#### Task 1a: cuda_pipeline/ → ml-core (4,084 LOC)

**Files to move:**
- `crates/ml/src/cuda_pipeline/mod.rs` (1,122)
- `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (795)
- `crates/ml/src/cuda_pipeline/gpu_ppo_collector.rs` (662)
- `crates/ml/src/cuda_pipeline/gpu_weights.rs` (887)
- `crates/ml/src/cuda_pipeline/gpu_portfolio.rs` (254)
- `crates/ml/src/cuda_pipeline/double_buffer.rs` (161)
- `crates/ml/src/cuda_pipeline/multi_gpu.rs` (203)

**Steps:**
1. Create `crates/ml-core/src/cuda_pipeline/` directory
2. Move all 7 files, update `mod.rs` module paths
3. Replace `use crate::MLError` → `use crate::error::MLError` (already in ml-core)
4. Replace `use crate::dqn::mixed_precision::training_dtype` → `use crate::mixed_precision::training_dtype` (already in ml-core)
5. Add `pub mod cuda_pipeline;` to `crates/ml-core/src/lib.rs`
6. In `crates/ml/src/lib.rs`: replace `pub mod cuda_pipeline;` with `pub use ml_core::cuda_pipeline;`
7. Run: `SQLX_OFFLINE=true cargo check -p ml-core -p ml`
8. Commit: `refactor(ml): move cuda_pipeline to ml-core`

#### Task 1b: flash_attention/ → ml-core (1,225 LOC)

**Files to move:**
- `crates/ml/src/flash_attention/*.rs`

**Steps:**
1. Create `crates/ml-core/src/flash_attention/`
2. Move files, fix `crate::` → appropriate ml-core paths
3. Add `pub mod flash_attention;` to ml-core lib.rs
4. In ml lib.rs: replace `pub mod flash_attention;` with `pub use ml_core::flash_attention;`
5. Run: `SQLX_OFFLINE=true cargo check -p ml-core -p ml`
6. Commit: `refactor(ml): move flash_attention to ml-core`

#### Task 1c: features/ remainder → ml-features (4,348 LOC)

**Files to move:**
- `crates/ml/src/features/extraction.rs` (1,285)
- `crates/ml/src/features/regime_adaptive.rs` (673)
- `crates/ml/src/features/multi_timeframe.rs` (630)
- `crates/ml/src/features/sample_weights.rs` (362)
- `crates/ml/src/features/unified.rs` (532)
- `crates/ml/src/features/regime_cusum.rs` (372)
- `crates/ml/src/features/regime_transition.rs` (333)
- `crates/ml/src/features/production_adapter.rs` (123)
- `crates/ml/src/features/mod.rs` (38)

**Steps:**
1. Check what `ml-features` already has vs. what's in `ml/src/features/`
2. Move non-duplicate files into `crates/ml-features/src/`
3. Fix imports: `use crate::` → `use crate::` (within ml-features) or add deps
4. Update ml-features Cargo.toml if new deps needed (e.g., ml-regime)
5. In ml lib.rs: replace `pub mod features;` with `pub use ml_features as features;` (or re-export)
6. Run: `SQLX_OFFLINE=true cargo check -p ml-features -p ml`
7. Commit: `refactor(ml): merge remaining features into ml-features`

#### Task 1d: microstructure/ → ml-features (682 LOC)

**Files to move:**
- `crates/ml/src/microstructure/*.rs`

**Steps:**
1. Move to `crates/ml-features/src/microstructure/`
2. Fix imports
3. Re-export from ml: `pub use ml_features::microstructure;`
4. Run: `SQLX_OFFLINE=true cargo check -p ml-features -p ml`
5. Commit: `refactor(ml): move microstructure to ml-features`

---

### Task 2: Data Modules → ml-data (7,041 LOC)

**Depends on:** Task 1c (features in ml-features)

**Files to move:**
- `crates/ml/src/data_loaders/*.rs` (4,365 — 7 files)
- `crates/ml/src/data_pipeline/*.rs` (1,500)
- `crates/ml/src/training/unified_data_loader.rs` + other training/ files (1,176)

**Steps:**
1. Add `ml-features` dependency to `crates/ml-data/Cargo.toml`
2. Add `dbn`, `zstd` dependencies if not already present
3. Move `data_loaders/` to `crates/ml-data/src/data_loaders/`
4. Move `data_pipeline/` to `crates/ml-data/src/data_pipeline/`
5. Move `training/` to `crates/ml-data/src/training/`
6. Fix imports: `use crate::features::` → `use ml_features::`, `use crate::types::` → `use ml_core::types::`
7. Fix `ensemble::MarketRegime` import in dbn_sequence_loader (may need re-export from ml-core)
8. In ml lib.rs: replace module declarations with re-exports
9. Run: `SQLX_OFFLINE=true cargo check -p ml-data -p ml`
10. Run: `SQLX_OFFLINE=true cargo test -p ml-data --lib`
11. Commit: `refactor(ml): move data_loaders, data_pipeline, training to ml-data`

---

### Task 3: Model Implementation Remnants → Model Sub-crates (~5,700 LOC)

**Parallel subtasks — independent per model.**

#### Task 3a: dqn/ → ml-dqn (959 LOC)

**Files:** `crates/ml/src/dqn/*.rs` (action_space, circuit_breaker, curiosity, logging, mixed_precision duplicate check, order_routing, portfolio_tracker, regime_conditional, reward, target_update)

**Steps:**
1. Check for duplicates: compare `ml/src/dqn/` contents with `crates/ml-dqn/src/`
2. Move non-duplicate files to `crates/ml-dqn/src/`
3. For duplicates: verify ml-dqn version is canonical, delete from ml/src/dqn/
4. Re-export from ml: `pub use ml_dqn as dqn;` (already done partially)
5. Run: `SQLX_OFFLINE=true cargo check -p ml-dqn -p ml`
6. Commit: `refactor(ml): merge dqn remnants into ml-dqn`

#### Task 3b: ppo/ → ml-ppo (1,004 LOC)

**Files:** `crates/ml/src/ppo/mod.rs`, `trainable_adapter.rs`, `stress_testing.rs`

**Steps:**
1. Move to `crates/ml-ppo/src/`
2. Fix imports
3. Re-export from ml
4. Check + test
5. Commit: `refactor(ml): merge ppo remnants into ml-ppo`

#### Task 3c: Supervised model dirs → ml-supervised (~4,900 LOC)

**Files:**
- `crates/ml/src/tft/` (1,796) — training.rs, trainable_adapter.rs
- `crates/ml/src/liquid/` (672)
- `crates/ml/src/tgnn/` (633)
- `crates/ml/src/kan/` (517)
- `crates/ml/src/xlstm/` (439)
- `crates/ml/src/diffusion/` (443)
- `crates/ml/src/mamba/` (399)
- `crates/ml/src/tlob/` (700)
- `crates/ml/src/transformers/` (821)

**Steps:**
1. For each model dir: compare with `ml-supervised/src/<model>/`
2. Move training logic and trainable adapters to `ml-supervised`
3. Move `transformers/` to `ml-supervised/src/transformers/`
4. Re-export from ml
5. Run: `SQLX_OFFLINE=true cargo check -p ml-supervised -p ml`
6. Commit: `refactor(ml): merge supervised model remnants into ml-supervised`

---

### Task 4: New Sub-crates (8,305 LOC)

**Parallel — independent of other tasks.**

#### Task 4a: benchmark/ → ml-benchmark (5,969 LOC)

**Steps:**
1. Create `crates/ml-benchmark/` with Cargo.toml
2. Add to workspace members in root Cargo.toml
3. Move `crates/ml/src/benchmark/` + `benchmarks.rs` to new crate
4. Add dependencies: ml-core, ml-dqn, ml-ppo, ml-supervised (model benchmarks reference models)
5. Re-export from ml: `pub use ml_benchmark as benchmark;`
6. Run: `SQLX_OFFLINE=true cargo check -p ml-benchmark -p ml`
7. Commit: `refactor(ml): extract benchmark into ml-benchmark`

#### Task 4b: deployment/ → ml-deployment (2,336 LOC)

**Steps:**
1. Create `crates/ml-deployment/` with Cargo.toml
2. Add to workspace members
3. Move `crates/ml/src/deployment/` to new crate
4. Add dependencies: ml-core
5. Re-export from ml
6. Check + test
7. Commit: `refactor(ml): extract deployment into ml-deployment`

---

### Task 5: Trainers → Model Sub-crates (18,034 LOC)

**Depends on:** Tasks 1 (foundations extracted), Task 3 (model impls merged)

This is the largest and most complex extraction. The trainers have heavy cross-module dependencies. After Tasks 1+3, those dependencies are now in sub-crates, so trainers can follow.

#### Task 5a: trainers/dqn/ → ml-dqn (9,554 LOC)

**Files:**
- `trainers/dqn/trainer.rs` (5,177)
- `trainers/dqn/config.rs` (1,223)
- `trainers/dqn/data_loading.rs` (920)
- `trainers/dqn/monitoring.rs` (284)
- `trainers/dqn/early_stopping.rs` (256)
- `trainers/dqn/features.rs` (163)
- `trainers/dqn/financials.rs` (178)
- `trainers/dqn/lr_scheduler.rs` (228)
- `trainers/dqn/risk.rs` (145)
- `trainers/dqn/statistics.rs` (134)
- `trainers/dqn/mod.rs` (45)

**Steps:**
1. Add dependencies to `crates/ml-dqn/Cargo.toml`:
   - `ml-features = { workspace = true }` (feature extraction)
   - `ml-labeling = { workspace = true }` (triple barrier)
   - `risk = { workspace = true }` (DrawdownMonitor, position limiter)
   - Any missing deps (evaluation metrics — may need to move evaluation too)
2. Create `crates/ml-dqn/src/trainer/` directory
3. Move all 11 files from `crates/ml/src/trainers/dqn/` → `crates/ml-dqn/src/trainer/`
4. Fix imports:
   - `use crate::cuda_pipeline::` → `use ml_core::cuda_pipeline::`
   - `use crate::dqn::` → `use crate::` (now in same crate)
   - `use crate::evaluation::` → determine home (may need to move evaluation first)
   - `use crate::features::` → `use ml_features::`
   - `use crate::labeling::` → `use ml_labeling::`
   - `use crate::memory_optimization::` → `use ml_core::memory_optimization::`
5. Add `pub mod trainer;` to ml-dqn lib.rs
6. In ml trainers/mod.rs: `pub use ml_dqn::trainer as dqn;` (or similar re-export)
7. Run: `SQLX_OFFLINE=true cargo check -p ml-dqn -p ml`
8. Run: `SQLX_OFFLINE=true cargo test -p ml-dqn --lib`
9. Commit: `refactor(ml): move DQN trainer to ml-dqn`

**Note:** The `evaluation/` module (metrics) is used by DQN trainer. It should move to ml-dqn or ml-core depending on whether other trainers also use it. Check with: `grep -r 'crate::evaluation' crates/ml/src/trainers/`

#### Task 5b: trainers/ppo.rs → ml-ppo (1,832 LOC)

**Steps:**
1. Add deps to ml-ppo Cargo.toml: ml-core (cuda_pipeline, batch_size_resolver)
2. Move `trainers/ppo.rs` → `crates/ml-ppo/src/trainer.rs`
3. Fix imports: `use crate::cuda_pipeline::PpoGpuData` → `use ml_core::cuda_pipeline::PpoGpuData`
4. Fix: `use crate::dqn::mixed_precision::training_dtype` → `use ml_core::mixed_precision::training_dtype`
5. Add `pub mod trainer;` to ml-ppo lib.rs
6. Re-export from ml
7. Check + test
8. Commit: `refactor(ml): move PPO trainer to ml-ppo`

#### Task 5c: trainers/tft/ → ml-supervised (2,957 LOC)

**Steps:**
1. Add deps to ml-supervised: ml-checkpoint, ml-core (memory_optimization)
2. Move `trainers/tft/` → `crates/ml-supervised/src/tft/trainer/` (or tft_trainer/)
3. Fix imports
4. Re-export from ml
5. Check + test
6. Commit: `refactor(ml): move TFT trainer to ml-supervised`

#### Task 5d: Remaining trainers (3,691 LOC)

**Files:**
- `trainers/mamba2.rs` (581) → ml-supervised
- `trainers/liquid.rs` (530) → ml-supervised
- `trainers/tlob.rs` (769) → ml-supervised
- `trainers/online_learning.rs` (982) → ml-core
- `trainers/curriculum.rs` (565) → ml-core (shared training infrastructure)
- `trainers/validation_metrics.rs` (454) → ml-core (shared)
- `trainers/tft_parquet.rs` (302) → ml-supervised

**Steps:**
1. Move model-specific trainers to ml-supervised
2. Move shared training infra to ml-core
3. Fix imports in each
4. Update ml trainers/mod.rs to re-export from sub-crates
5. Check + test
6. Commit: `refactor(ml): move remaining trainers to sub-crates`

---

### Task 6: Adapters → ml-hyperopt & ml-ensemble (~16K LOC)

**Depends on:** Task 5 (trainers in sub-crates)

#### Task 6a: hyperopt/adapters/ → ml-hyperopt (12,752 LOC + 1,295 supporting)

**Steps:**
1. Add dependencies to `crates/ml-hyperopt/Cargo.toml`:
   - `ml-dqn = { workspace = true }` (DQN adapter)
   - `ml-ppo = { workspace = true }` (PPO adapter)
   - `ml-supervised = { workspace = true }` (8 supervised adapters)
   - `ml-features = { workspace = true }` (feature extraction)
   - `ml-data = { workspace = true }` (data loading)
   - `dbn`, `zstd` (DBN file loading)
2. Create `crates/ml-hyperopt/src/adapters/` directory
3. Move all 14 adapter files + mod.rs
4. Move `campaign.rs`, `shared_data.rs`, `tests_argmin.rs`
5. Fix imports: `use crate::trainers::dqn::` → `use ml_dqn::trainer::`
6. Fix: `use crate::evaluation::` → determine new path
7. Update ml hyperopt/mod.rs to re-export from ml-hyperopt
8. Run: `SQLX_OFFLINE=true cargo check -p ml-hyperopt -p ml`
9. Run: `SQLX_OFFLINE=true cargo test -p ml-hyperopt --lib`
10. Commit: `refactor(ml): move hyperopt adapters to ml-hyperopt`

#### Task 6b: ensemble/adapters/ → ml-ensemble (3,427 LOC)

**Steps:**
1. Add deps to ml-ensemble: ml-dqn, ml-ppo, ml-supervised, ml-features
2. Move `ensemble/adapters/` to `crates/ml-ensemble/src/adapters/`
3. Move `ensemble/model_adapter.rs` to ml-ensemble
4. Fix imports: model-specific `use crate::tft::` → `use ml_supervised::tft::`
5. Fix: `use crate::dqn::mixed_precision::` → `use ml_core::mixed_precision::`
6. Re-export from ml
7. Check + test
8. Commit: `refactor(ml): move ensemble adapters to ml-ensemble`

---

### Task 7: Final Cleanup

**Depends on:** All previous tasks

#### Task 7a: Small module extraction

Move remaining small modules to existing sub-crates:
- `model_registry/` + `model_registry.rs` + `registry/` → ml-core or ml-checkpoint
- `evaluation/` → ml-core (shared metrics used by multiple trainers)
- `data_validation/` stub → delete (ml-data-validation crate exists)
- Other stubs (backtesting/, checkpoint/, labeling/, etc.) → delete re-export stubs

#### Task 7b: Update ml lib.rs

Rewrite `crates/ml/src/lib.rs` to be a clean orchestration facade:
- Remove all extracted `pub mod` declarations
- Replace with `pub use` re-exports from sub-crates
- Keep local modules: integration/, inference.rs, model_factory.rs, training_pipeline.rs, etc.
- Keep `From` impls that bridge local types to ml-core types

#### Task 7c: Update downstream consumers

Check and fix imports in:
- `services/trading_service/` (uses `ml::` paths)
- `services/trading_agent_service/`
- `services/backtesting_service/`
- `services/ml_training_service/`
- `bin/fxt/` (CLI binary)
- Training binaries

For each: `grep -r 'use ml::' <service>/src/` and fix any broken paths.

#### Task 7d: Workspace Cargo.toml

- Add new crates to `[workspace.members]`: ml-benchmark, ml-deployment
- Add workspace deps if needed
- Verify feature flag propagation (cuda feature must chain through all sub-crates)

#### Task 7e: Final verification

```bash
# Full workspace check
SQLX_OFFLINE=true cargo check --workspace

# Full workspace clippy
SQLX_OFFLINE=true cargo clippy --workspace -- -D warnings

# Run all ml-* tests
for crate in crates/ml-*/; do
  name=$(basename "$crate")
  echo "Testing $name..."
  SQLX_OFFLINE=true cargo test -p "$name" --lib 2>&1 | tail -1
done

# Run ml orchestration crate tests
SQLX_OFFLINE=true cargo test -p ml --lib

# Verify line count reduction
find crates/ml/src -name '*.rs' -type f | xargs wc -l | tail -1
# Expected: ~12,000 (down from 91,000)
```

Commit: `refactor(ml): finalize phase 2 split — ml reduced to orchestration layer`

---

## Key Risks & Mitigations

1. **Circular dependencies**: Sub-crates must never depend on `ml`. All deps flow: ml → sub-crates → ml-core. Verified: no sub-crate currently depends on ml.

2. **evaluation module**: Used by DQN trainer AND hyperopt adapters. Must decide home (ml-core or ml-dqn) BEFORE moving trainers. Check: `grep -r 'crate::evaluation' crates/ml/src/`

3. **mixed_precision::training_dtype**: Referenced from DQN, PPO, ensemble. Already in ml-core. Verify re-export path works.

4. **Feature flags**: cuda feature must propagate. After split, `ml`'s `[features] cuda = [...]` must include all new sub-crate cuda features.

5. **Re-export backward compat**: Services use `ml::trainers::dqn::DQNTrainer`. After move, must re-export: `pub use ml_dqn::trainer as dqn;` in trainers/mod.rs.

6. **Test count regression**: Track test counts before/after. Expected: same total, redistributed across sub-crates.

## Pre-Split Baseline

```bash
# Record before starting
find crates/ml/src -name '*.rs' -type f | xargs wc -l | tail -1  # ~91,000
SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | grep 'test result'
```