foxhunt/docs/plans/2026-03-07-ml-crate-split-plan.md

ML Crate Split Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Split the monolithic ml crate (260K lines, 55s compile) into 6 crates that compile in parallel, reducing incremental rebuilds from 55s to ~6-12s.

Architecture: 5 sub-crates (ml-core, ml-dqn, ml-ppo, ml-supervised, ml-infra) + thin ml facade. DQN, PPO, and supervised models compile in 3-way parallel. External consumers see no API changes.

Tech Stack: Rust workspace, Cargo features, pub use re-exports.

Reference: See docs/plans/2026-03-07-ml-crate-split-design.md for full module inventory and dependency analysis.

---

Phase 1: Preparation (in worktree .worktrees/ml-crate-split, branch feature/ml-crate-split)

Task 1: Verify Clean Baseline

Files: None modified

Step 1: Run full workspace check

Run: cd /home/jgrusewski/Work/foxhunt/.worktrees/ml-crate-split && SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5 Expected: Finished with 0 errors

Step 2: Run ml tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5 Expected: test result: ok with 2506+ tests

Step 3: Record baseline

Note the exact test count and build time for comparison after split.

---

Task 2: Create Sub-Crate Directory Structure

Files:

• Create: crates/ml-core/Cargo.toml
• Create: crates/ml-core/src/lib.rs
• Create: crates/ml-dqn/Cargo.toml
• Create: crates/ml-dqn/src/lib.rs
• Create: crates/ml-ppo/Cargo.toml
• Create: crates/ml-ppo/src/lib.rs
• Create: crates/ml-supervised/Cargo.toml
• Create: crates/ml-supervised/src/lib.rs
• Create: crates/ml-infra/Cargo.toml
• Create: crates/ml-infra/src/lib.rs
• Modify: Cargo.toml (workspace root — add members + workspace deps)

Step 1: Create crate directories

mkdir -p crates/ml-core/src crates/ml-dqn/src crates/ml-ppo/src crates/ml-supervised/src crates/ml-infra/src

Step 2: Add workspace members

In root Cargo.toml, add to [workspace] members:

"crates/ml-core",
"crates/ml-dqn",
"crates/ml-ppo",
"crates/ml-supervised",
"crates/ml-infra",

And add workspace dependency entries:

ml-core = { path = "crates/ml-core" }
ml-dqn = { path = "crates/ml-dqn" }
ml-ppo = { path = "crates/ml-ppo" }
ml-supervised = { path = "crates/ml-supervised" }
ml-infra = { path = "crates/ml-infra" }

Step 3: Create minimal Cargo.toml for each sub-crate

Each sub-crate gets a Cargo.toml with:

• Same edition/rust-version as ml
• Dependencies subset (only what its modules need)
• Feature flags relevant to its modules
• ml-core as dependency for ml-dqn, ml-ppo, ml-supervised, ml-infra

See the detailed Cargo.toml specs in Tasks 4-10.

Step 4: Create stub lib.rs for each

Each gets an empty pub mod list initially:

// crates/ml-core/src/lib.rs
// Placeholder — modules moved in subsequent tasks

Step 5: Verify workspace compiles

Run: SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -5 Expected: Compiles (empty crates are valid)

Step 6: Commit

git add crates/ml-core crates/ml-dqn crates/ml-ppo crates/ml-supervised crates/ml-infra Cargo.toml
git commit -m "feat(ml): scaffold sub-crate directory structure for ml split"

---

Phase 2: Build ml-core (Foundation)

Task 3: Extract Shared Items from dqn/ to Top-Level Modules

These items live in dqn/ but are used by PPO, trainers, ensemble, hyperopt — they belong in ml-core. Also extract TradingAction from dqn/agent.rs to enable DQN/PPO independence.

Files:

• Move: crates/ml/src/dqn/mixed_precision.rs → crates/ml/src/mixed_precision.rs
• Move: crates/ml/src/dqn/xavier_init.rs → crates/ml/src/xavier_init.rs
• Move: crates/ml/src/dqn/portfolio_tracker.rs → crates/ml/src/portfolio_tracker.rs
• Move: crates/ml/src/dqn/action_space.rs → crates/ml/src/action_space.rs
• Move: crates/ml/src/dqn/order_router.rs → crates/ml/src/order_router.rs
• Modify: crates/ml/src/dqn/agent.rs — move TradingAction enum to new file
• Create: crates/ml/src/trading_action.rs — TradingAction enum (Buy/Sell/Hold)
• Modify: crates/ml/src/dqn/mod.rs — replace module declarations with re-exports
• Modify: crates/ml/src/lib.rs — add pub mod for new top-level modules
• Modify: All files importing these via crate::dqn::* path

Step 1: Move files

cd crates/ml/src
mv dqn/mixed_precision.rs mixed_precision.rs
mv dqn/xavier_init.rs xavier_init.rs
mv dqn/portfolio_tracker.rs portfolio_tracker.rs
mv dqn/action_space.rs action_space.rs
mv dqn/order_router.rs order_router.rs

Step 2: Extract TradingAction from dqn/agent.rs

Create crates/ml/src/trading_action.rs with the TradingAction enum (copied from dqn/agent.rs). In dqn/agent.rs, replace the enum with pub use crate::trading_action::TradingAction;. This is the only PPO→DQN dependency remaining after the other extractions — making it shared enables full DQN/PPO independence.

Step 3: Update dqn/mod.rs

Replace the old pub mod declarations with re-exports so existing crate::dqn::X paths still work:

// In dqn/mod.rs, REPLACE:
//   pub mod mixed_precision;
//   pub mod xavier_init;
//   pub mod portfolio_tracker;
//   pub mod action_space;
//   pub mod order_router;
// WITH:
pub use crate::mixed_precision;
pub use crate::xavier_init;
pub use crate::portfolio_tracker;
pub use crate::action_space;
pub use crate::order_router;
pub use crate::trading_action::TradingAction;

Step 4: Add modules to ml/lib.rs

Add these lines to lib.rs:

pub mod mixed_precision;
pub mod xavier_init;
pub mod portfolio_tracker;
pub mod action_space;
pub mod order_router;
pub mod trading_action;

Step 5: Fix internal imports in moved files

In each moved file, change use crate::dqn:: imports to use crate:: where referencing peer modules.

Specific fixes needed:

• action_space.rs: has no crate::dqn:: imports (only std/serde) — no changes needed
• xavier_init.rs: uses candle_core/candle_nn only — no changes needed
• mixed_precision.rs: uses candle_core only — no changes needed
• portfolio_tracker.rs: check for crate::dqn:: imports and fix
• order_router.rs: imports crate::dqn::action_space::* → change to crate::action_space::*; imports crate::common::action::* → keep as-is
• PPO test files using crate::dqn::TradingAction → change to crate::trading_action::TradingAction

Step 5: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5 Expected: Compiles with 0 errors (re-exports preserve all paths)

Step 6: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5 Expected: Same test count as baseline, 0 failures

Step 7: Run clippy

Run: SQLX_OFFLINE=true cargo clippy -p ml -- -D warnings 2>&1 | tail -10 Expected: 0 errors, 0 warnings

Step 8: Commit

git add -A crates/ml/
git commit -m "refactor(ml): extract shared items from dqn/ to top-level modules

Move mixed_precision, xavier_init, portfolio_tracker, action_space,
order_router from dqn/ to ml/src/ root. These are shared infrastructure
used by PPO, trainers, hyperopt, and ensemble — not DQN-specific.
Re-export from dqn/mod.rs for backward compatibility."

---

Task 4: Write ml-core Cargo.toml

Files:

• Create: crates/ml-core/Cargo.toml

Step 1: Write Cargo.toml

Build by examining crates/ml/Cargo.toml and including ONLY dependencies used by ml-core modules. Key deps:

• candle-core, candle-nn, candle-optimisers (for mixed_precision, xavier_init, Adam, gpu)
• common, config, data (workspace crates)
• tokio, serde, thiserror, anyhow, chrono, tracing (core utilities)
• arrow, parquet (for features/)
• dbn, databento (for data_loaders/)
• rust_decimal (financial types)
• nalgebra, ndarray, num-traits (for features math)
• dashmap, lru (for caching in features/)
• sha2, hmac, hex (for security/)
• flate2, memmap2 (for checkpoint/)
• NOT: argmin (hyperopt only), NOT model-specific deps

Feature flags:

[features]
default = ["cuda"]
cuda = ["candle-core/cuda", "candle-core/cudnn", "candle-nn/cuda", "candle-nn/cudnn"]
s3-storage = ["aws-config", "aws-sdk-s3", "aws-types", "aws-credential-types", "urlencoding"]
high-precision = ["rust_decimal/serde-float"]
mimalloc-allocator = ["mimalloc"]
simd = []
gc = []
financial = []
minimal-inference = []

Step 2: Verify it parses

Run: cargo check -p ml-core 2>&1 | head -5 (will fail on missing modules, that's OK — just verifying Cargo.toml syntax)

Step 3: Commit

git add crates/ml-core/Cargo.toml
git commit -m "feat(ml-core): write Cargo.toml with dependency subset"

---

Task 5: Move Core Modules to ml-core

This is the largest single task. Move all modules assigned to ml-core.

Files:

• Move: ~30 modules from crates/ml/src/ to crates/ml-core/src/
• Modify: crates/ml-core/src/lib.rs — declare all modules
• Modify: crates/ml/src/lib.rs — remove moved modules, add pub use ml_core::*

Step 1: Move type definitions from lib.rs

Extract the type definitions (Trade, MarketRegime, CommonError, CommonTypeError, MLError, HealthStatus, MarketDataSnapshot, FeatureVector, IntegerTensor, UpdateSummary, ModelPrediction, InferenceResult, ModelMetadata, Features, MLModel, HFTPerformanceProfile, ParallelExecutor, LatencyOptimizer, ModelRegistry helpers, and all From impls for MLError) from crates/ml/src/lib.rs into crates/ml-core/src/lib.rs.

Keep the crate-level attributes (#![deny(...)], #![allow(...)]) in both — ml-core gets the canonical set, ml facade gets a minimal subset.

Step 2: Move modules

# Core infrastructure
mv crates/ml/src/error.rs crates/ml-core/src/
mv crates/ml/src/types crates/ml-core/src/
mv crates/ml/src/common crates/ml-core/src/
mv crates/ml/src/config crates/ml-core/src/
mv crates/ml/src/cuda_compat.rs crates/ml-core/src/
mv crates/ml/src/tensor_ops.rs crates/ml-core/src/
mv crates/ml/src/traits.rs crates/ml-core/src/
mv crates/ml/src/model.rs crates/ml-core/src/
mv crates/ml/src/optimizers crates/ml-core/src/
mv crates/ml/src/gradient_accumulation.rs crates/ml-core/src/
mv crates/ml/src/gradient_utils.rs crates/ml-core/src/
mv crates/ml/src/gpu crates/ml-core/src/
mv crates/ml/src/safety crates/ml-core/src/
mv crates/ml/src/security crates/ml-core/src/
mv crates/ml/src/memory_optimization crates/ml-core/src/
mv crates/ml/src/checkpoint crates/ml-core/src/
mv crates/ml/src/training crates/ml-core/src/
mv crates/ml/src/training.rs crates/ml-core/src/

# Extracted shared items
mv crates/ml/src/mixed_precision.rs crates/ml-core/src/
mv crates/ml/src/xavier_init.rs crates/ml-core/src/
mv crates/ml/src/portfolio_tracker.rs crates/ml-core/src/
mv crates/ml/src/action_space.rs crates/ml-core/src/
mv crates/ml/src/order_router.rs crates/ml-core/src/

# Features and data
mv crates/ml/src/features crates/ml-core/src/
mv crates/ml/src/feature_cache.rs crates/ml-core/src/
mv crates/ml/src/preprocessing.rs crates/ml-core/src/
mv crates/ml/src/data_loader.rs crates/ml-core/src/
mv crates/ml/src/data_loaders crates/ml-core/src/
mv crates/ml/src/data_pipeline crates/ml-core/src/
mv crates/ml/src/data_validation crates/ml-core/src/
mv crates/ml/src/evaluation crates/ml-core/src/

# Metrics and monitoring
mv crates/ml/src/metrics crates/ml-core/src/
mv crates/ml/src/performance.rs crates/ml-core/src/
mv crates/ml/src/observability crates/ml-core/src/

Step 3: Update ml-core/src/lib.rs

Declare all moved modules with pub mod. Include all type definitions and re-exports. Copy the relevant use imports (serde, chrono, rust_decimal, etc.).

Step 4: Update all use crate:: imports in moved files

Every file in crates/ml-core/src/ that uses use crate:: — these references are now correct (they refer to ml-core's own modules). Verify no moved file imports something that stayed in ml.

Potential issues:

• safety/ imports from crate::dqn:: (but mixed_precision is now in ml-core root)
• features/ may import from model-specific modules → fix or feature-gate
• data_loaders/ imports crate::types::OHLCVBar → still in ml-core ✓
• evaluation/ imports crate::features:: → still in ml-core ✓
• training/unified_trainer.rs — verify it only imports items now in ml-core
• checkpoint/ — verify no model-specific imports

Run grep -rn 'use crate::dqn\|use crate::ppo\|use crate::tft\|use crate::mamba\|use crate::ensemble\|use crate::hyperopt\|use crate::trainers' crates/ml-core/src/ to find any imports of modules that are NOT in ml-core. These must be removed/refactored.

Step 5: Update ml/src/lib.rs

Replace removed pub mod declarations with re-exports:

// Add dependency
pub use ml_core;

// Re-export everything from ml-core
pub use ml_core::*;
// Explicitly re-export modules for path compatibility
pub use ml_core::common;
pub use ml_core::config;
pub use ml_core::checkpoint;
// ... etc for all modules that external consumers access by path (ml::checkpoint::X)

Step 6: Add ml-core dependency to ml/Cargo.toml

[dependencies]
ml-core = { workspace = true }

Step 7: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml-core 2>&1 | tail -5 Expected: ml-core compiles

Run: SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5 Expected: ml compiles (re-exports resolve)

Step 8: Fix any compilation errors

Iterate on import fixes until both ml-core and ml compile cleanly.

Step 9: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5 Expected: All tests still pass

Step 10: Run clippy

Run: SQLX_OFFLINE=true cargo clippy -p ml-core -p ml -- -D warnings 2>&1 | tail -10 Expected: 0 errors, 0 warnings

Step 11: Commit

git add -A
git commit -m "refactor(ml): move core modules to ml-core crate

Move types, errors, traits, features, data loaders, checkpoint,
safety, GPU utils, and shared infrastructure to ml-core.
Re-export from ml facade for backward compatibility."

---

Phase 3: Build ml-dqn and ml-ppo (RL Models — Separate Crates)

Task 6: Write ml-dqn Cargo.toml

Files:

• Create: crates/ml-dqn/Cargo.toml

Step 1: Write Cargo.toml

Dependencies: ml-core + candle-core, candle-nn + DQN-specific deps (crossbeam, etc.).

Feature flags:

[features]
default = ["cuda"]
cuda = ["ml-core/cuda"]
nccl = ["cuda"]

Step 2: Commit

---

Task 7a: Move DQN + cuda_pipeline to ml-dqn

Files:

• Move: crates/ml/src/dqn/ → crates/ml-dqn/src/dqn/
• Move: crates/ml/src/cuda_pipeline/ → crates/ml-dqn/src/cuda_pipeline/
• Modify: crates/ml-dqn/src/lib.rs
• Modify: crates/ml/src/lib.rs

Step 1: Move modules

mv crates/ml/src/dqn crates/ml-dqn/src/
mv crates/ml/src/cuda_pipeline crates/ml-dqn/src/

Step 2: Write ml-dqn/src/lib.rs

// Same clippy attributes as ml-core
pub mod dqn;
pub mod cuda_pipeline;

Step 3: Fix all imports in ml-dqn

In every file under crates/ml-dqn/src/:

• use crate::MLError → use ml_core::MLError
• use crate::mixed_precision → use ml_core::mixed_precision
• use crate::xavier_init → use ml_core::xavier_init
• use crate::portfolio_tracker → use ml_core::portfolio_tracker
• use crate::action_space → use ml_core::action_space
• use crate::order_router → use ml_core::order_router
• use crate::trading_action → use ml_core::trading_action
• use crate::common:: → use ml_core::common::
• use crate::cuda_compat:: → use ml_core::cuda_compat::
• use crate::gradient_accumulation:: → use ml_core::gradient_accumulation::
• use crate::training::unified_trainer:: → use ml_core::training::unified_trainer::
• use crate::safety:: → use ml_core::safety::
• use crate::tensor_ops:: → use ml_core::tensor_ops::
• use crate::gpu:: → use ml_core::gpu::
• use crate::checkpoint:: → use ml_core::checkpoint::
• use crate::features:: → use ml_core::features::
• use crate::evaluation:: → use ml_core::evaluation::
• use crate::Adam → use ml_core::Adam
• use crate::PRECISION_FACTOR → use ml_core::PRECISION_FACTOR
• use crate::{MLError, MLResult} → use ml_core::{MLError, MLResult}

Keep use crate::dqn:: as-is (references ml-dqn's own dqn module). Keep use crate::cuda_pipeline:: as-is (references ml-dqn's own cuda_pipeline module).

Update the re-exports from dqn/mod.rs added in Task 3:

// REPLACE crate:: references WITH ml_core::
pub use ml_core::mixed_precision;
pub use ml_core::xavier_init;
pub use ml_core::portfolio_tracker;
pub use ml_core::action_space;
pub use ml_core::order_router;
pub use ml_core::trading_action::TradingAction;

cuda_pipeline imports:

• crate::dqn::mixed_precision → ml_core::mixed_precision
• crate::dqn::replay_buffer_type::GpuBatch → crate::dqn::replay_buffer_type::GpuBatch (stays in ml-dqn ✓)

Watch for: use crate::trainers::DQNTrainer in dqn/ — reverse dep, likely test-only. Remove or cfg(test) gate with a comment.

Step 4: Update ml/src/lib.rs

Add re-exports:

pub use ml_dqn::dqn;
pub use ml_dqn::cuda_pipeline;

Add dependency to ml/Cargo.toml:

ml-dqn = { workspace = true }

Step 5: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml-dqn 2>&1 | tail -10 Run: SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10

Step 6: Fix errors iteratively

Keep fixing imports until both compile.

Step 7: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5

Step 8: Commit

git add -A
git commit -m "refactor(ml): move DQN + cuda_pipeline to ml-dqn crate"

---

Task 7b: Write ml-ppo Cargo.toml and Move PPO

Files:

• Create: crates/ml-ppo/Cargo.toml
• Move: crates/ml/src/ppo/ → crates/ml-ppo/src/ppo/
• Modify: crates/ml-ppo/src/lib.rs
• Modify: crates/ml/src/lib.rs

Step 1: Write ml-ppo Cargo.toml

Dependencies: ml-core + candle-core, candle-nn. NO dependency on ml-dqn.

Feature flags:

[features]
default = ["cuda"]
cuda = ["ml-core/cuda"]

Step 2: Move PPO

mv crates/ml/src/ppo crates/ml-ppo/src/

Step 3: Write ml-ppo/src/lib.rs

// Same clippy attributes as ml-core
pub mod ppo;

Step 4: Fix all imports in ml-ppo

Same pattern as ml-dqn: use crate::X → use ml_core::X for all shared items.

Key changes:

• use crate::dqn::circuit_breaker → use ml_core::common::circuit_breaker (direct path, no dqn dependency)
• use crate::dqn::mixed_precision::training_dtype → use ml_core::mixed_precision::training_dtype
• use crate::dqn::portfolio_tracker → use ml_core::portfolio_tracker
• use crate::dqn::xavier_init::linear_xavier → use ml_core::xavier_init::linear_xavier
• use crate::dqn::TradingAction → use ml_core::trading_action::TradingAction (test-only)

Fix ppo/circuit_breaker.rs:

// Change: pub use crate::common::circuit_breaker::*
// To: pub use ml_core::common::circuit_breaker::*

Watch for: use crate::hyperopt::adapters::PPOTrainer in ppo/ — reverse dep, likely test-only.

Step 5: Update ml/src/lib.rs

pub use ml_ppo::ppo;

Add dependency: ml-ppo = { workspace = true }

Step 6: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml-ppo 2>&1 | tail -10 Run: SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10

Step 7: Verify DQN and PPO are independent

Run: SQLX_OFFLINE=true cargo tree -p ml-ppo --depth 1 | grep ml-dqn Expected: NO output (ml-ppo must NOT depend on ml-dqn)

Step 8: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5

Step 9: Commit

git add -A
git commit -m "refactor(ml): move PPO to ml-ppo crate (independent of ml-dqn)"

---

Phase 4: Build ml-supervised (Supervised Models)

Task 8: Write ml-supervised Cargo.toml

Files:

• Create: crates/ml-supervised/Cargo.toml

Dependencies: ml-core + candle-core, candle-nn.

Feature flags:

[features]
default = ["cuda"]
cuda = ["ml-core/cuda"]

Step 1: Commit

---

Task 9: Move Supervised Models to ml-supervised

Files:

• Move: crates/ml/src/tft/ → crates/ml-supervised/src/tft/
• Move: crates/ml/src/mamba/ → crates/ml-supervised/src/mamba/
• Move: crates/ml/src/liquid/ → crates/ml-supervised/src/liquid/
• Move: crates/ml/src/tgnn/ → crates/ml-supervised/src/tgnn/
• Move: crates/ml/src/tlob/ → crates/ml-supervised/src/tlob/
• Move: crates/ml/src/kan/ → crates/ml-supervised/src/kan/
• Move: crates/ml/src/xlstm/ → crates/ml-supervised/src/xlstm/
• Move: crates/ml/src/diffusion/ → crates/ml-supervised/src/diffusion/
• Move: crates/ml/src/transformers/ → crates/ml-supervised/src/transformers/
• Move: crates/ml/src/flash_attention/ → crates/ml-supervised/src/flash_attention/
• Modify: crates/ml-supervised/src/lib.rs
• Modify: crates/ml/src/lib.rs

Step 1: Move modules

for mod in tft mamba liquid tgnn tlob kan xlstm diffusion transformers flash_attention; do
  mv crates/ml/src/$mod crates/ml-supervised/src/
done

Step 2: Write ml-supervised/src/lib.rs

pub mod tft;
pub mod mamba;
pub mod liquid;
pub mod tgnn;
pub mod tlob;
pub mod kan;
pub mod xlstm;
pub mod diffusion;
pub mod transformers;
pub mod flash_attention;

Step 3: Fix all imports

Same pattern as Task 7: use crate::X → use ml_core::X for anything from ml-core.

Specific cross-model deps to handle:

• tft imports crate::liquid::FixedPoint → crate::liquid::FixedPoint (both in ml-supervised ✓)
• mamba imports crate::liquid::FixedPoint → crate::liquid::FixedPoint (both in ml-supervised ✓)
• tft imports crate::inference::RealInferenceError → This module stays in ml facade! Need to either:
  • Move RealInferenceError type to ml-core, OR
  • Remove this import if it's only used in a few places and can be refactored
• All models import crate::dqn::mixed_precision::training_dtype → ml_core::mixed_precision::training_dtype
• All models import crate::training::unified_trainer::* → ml_core::training::unified_trainer::*

Step 4: Update ml/src/lib.rs

pub use ml_supervised::tft;
pub use ml_supervised::mamba;
pub use ml_supervised::liquid;
pub use ml_supervised::tgnn;
pub use ml_supervised::tlob;
pub use ml_supervised::kan;
pub use ml_supervised::xlstm;
pub use ml_supervised::diffusion;
pub use ml_supervised::transformers;
pub use ml_supervised::flash_attention;

Add dependency: ml-supervised = { workspace = true }

Step 5: Verify and fix

Run: SQLX_OFFLINE=true cargo check -p ml-supervised 2>&1 | tail -10 Run: SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10

Step 6: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5

Step 7: Commit

git add -A
git commit -m "refactor(ml): move 8 supervised models to ml-supervised crate

TFT, Mamba, Liquid, TGNN, TLOB, KAN, xLSTM, Diffusion plus shared
transformers and flash_attention infrastructure."

---

Phase 5: Build ml-infra (Training Infrastructure)

Task 10: Write ml-infra Cargo.toml

Files:

• Create: crates/ml-infra/Cargo.toml

Dependencies: ml-core, ml-dqn, ml-ppo, ml-supervised + argmin (for hyperopt), model-specific deps.

Feature flags:

[features]
default = ["cuda"]
cuda = ["ml-core/cuda", "ml-dqn/cuda", "ml-ppo/cuda", "ml-supervised/cuda"]

Step 1: Commit

---

Task 11: Move Infrastructure Modules to ml-infra

Files:

• Move: crates/ml/src/hyperopt/ → crates/ml-infra/src/hyperopt/
• Move: crates/ml/src/ensemble/ → crates/ml-infra/src/ensemble/
• Move: crates/ml/src/trainers/ → crates/ml-infra/src/trainers/
• Move: crates/ml/src/benchmark/ → crates/ml-infra/src/benchmark/
• Move: crates/ml/src/deployment/ → crates/ml-infra/src/deployment/
• Move: crates/ml/src/training_pipeline.rs → crates/ml-infra/src/training_pipeline.rs
• Move: crates/ml/src/walk_forward.rs → crates/ml-infra/src/walk_forward.rs
• Modify: crates/ml-infra/src/lib.rs
• Modify: crates/ml/src/lib.rs

Step 1: Move modules

for mod in hyperopt ensemble trainers benchmark deployment; do
  mv crates/ml/src/$mod crates/ml-infra/src/
done
mv crates/ml/src/training_pipeline.rs crates/ml-infra/src/
mv crates/ml/src/walk_forward.rs crates/ml-infra/src/

Step 2: Write ml-infra/src/lib.rs

pub mod hyperopt;
pub mod ensemble;
pub mod trainers;
pub mod benchmark;
pub mod deployment;
pub mod training_pipeline;
pub mod walk_forward;

Step 3: Fix all imports

This is the most complex crate — it imports from all four model sub-crates:

• use crate::dqn::DQN → use ml_dqn::dqn::DQN
• use crate::dqn::DQNConfig → use ml_dqn::dqn::DQNConfig
• use crate::ppo::PPO → use ml_ppo::ppo::PPO
• use crate::cuda_pipeline::* → use ml_dqn::cuda_pipeline::*
• use crate::tft::* → use ml_supervised::tft::*
• use crate::mamba::* → use ml_supervised::mamba::*
• use crate::liquid::* → use ml_supervised::liquid::*
• use crate::diffusion::* → use ml_supervised::diffusion::*
• use crate::kan::* → use ml_supervised::kan::*
• use crate::xlstm::* → use ml_supervised::xlstm::*
• use crate::tgnn::* → use ml_supervised::tgnn::*
• use crate::tlob::* → use ml_supervised::tlob::*
• All shared items: use crate::X → use ml_core::X

Step 4: Update ml/src/lib.rs

pub use ml_infra::hyperopt;
pub use ml_infra::ensemble;
pub use ml_infra::trainers;
pub use ml_infra::benchmark;
pub use ml_infra::deployment;
pub use ml_infra::training_pipeline;
pub use ml_infra::walk_forward;

Add dependency: ml-infra = { workspace = true }

Step 5: Verify and fix

Run: SQLX_OFFLINE=true cargo check -p ml-infra 2>&1 | tail -10 Run: SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -10

Step 6: Run tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5

Step 7: Commit

git add -A
git commit -m "refactor(ml): move hyperopt, ensemble, trainers to ml-infra crate"

---

Phase 6: Finalize Facade and Verify

Task 12: Clean Up ml Facade

Files:

• Modify: crates/ml/src/lib.rs — final cleanup
• Modify: crates/ml/Cargo.toml — update dependencies

Step 1: Update ml/Cargo.toml

• Add: ml-core, ml-dqn, ml-ppo, ml-supervised, ml-infra as dependencies
• Remove: all dependencies that are now only used by sub-crates
• Keep: dependencies used by remaining facade modules (inference.rs, validation/, risk/, etc.)

Step 2: Verify ml/src/lib.rs has complete re-exports

Every module path that external consumers use (e.g., ml::dqn::DQNConfig, ml::ensemble::EnsembleCoordinator) must resolve. Create a checklist from the external consumer import scan (Task 1 research).

Step 3: Fix remaining facade module imports

Modules still in ml facade (inference.rs, validation/, risk/, regime/, etc.) need import updates:

• use crate::dqn:: — these now come from ml_dqn via re-export, so use crate::dqn:: still works ✓
• use crate::ppo:: — same via re-export from ml_ppo ✓
• use crate::tft:: — same via re-export from ml_supervised ✓
• use crate::ensemble:: — same via re-export from ml_infra ✓
• use crate::MLError — from ml_core via pub use ml_core::* ✓

This should work because the facade re-exports everything. But verify with compilation.

Step 4: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml 2>&1 | tail -5

Step 5: Commit

git add -A
git commit -m "refactor(ml): finalize facade with complete re-exports"

---

Task 13: Full Workspace Verification

Files: None modified

Step 1: Full workspace check

Run: SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -10 Expected: 0 errors across all workspace members

Step 2: Full workspace clippy

Run: SQLX_OFFLINE=true cargo clippy --workspace -- -D warnings 2>&1 | tail -20 Expected: 0 errors, 0 warnings

Step 3: Run ml tests

Run: SQLX_OFFLINE=true cargo test -p ml --lib 2>&1 | tail -5 Expected: Same test count as baseline (2506+), 0 failures

Step 4: Run sub-crate tests

Run each in parallel:

SQLX_OFFLINE=true cargo test -p ml-core --lib 2>&1 | tail -5
SQLX_OFFLINE=true cargo test -p ml-dqn --lib 2>&1 | tail -5
SQLX_OFFLINE=true cargo test -p ml-ppo --lib 2>&1 | tail -5
SQLX_OFFLINE=true cargo test -p ml-supervised --lib 2>&1 | tail -5
SQLX_OFFLINE=true cargo test -p ml-infra --lib 2>&1 | tail -5

Expected: Tests distributed across sub-crates, total matches baseline

Step 5: Run downstream service checks

Run:

SQLX_OFFLINE=true cargo check -p trading_service 2>&1 | tail -3
SQLX_OFFLINE=true cargo check -p ml_training_service 2>&1 | tail -3
SQLX_OFFLINE=true cargo check -p backtesting_service 2>&1 | tail -3
SQLX_OFFLINE=true cargo check -p trading_agent_service 2>&1 | tail -3

Expected: All compile with 0 errors (they use ml facade, re-exports preserve paths)

Step 6: Verify compile time improvement

Run: SQLX_OFFLINE=true cargo build -p ml --timings 2>&1 | tail -5 Open the timings HTML report. Compare per-crate times against the baseline 55.1s.

Step 7: Commit if any fixes were needed

---

Task 14: Update common dev-dependency

Files:

• Modify: crates/common/Cargo.toml

Step 1: Update dev-dependency

common has ml = { path = "../ml" } as a dev-dependency. Update the path since ml is still at the same location:

ml = { path = "../ml" }  # Should still work — verify

If paths changed, update accordingly.

Step 2: Verify

Run: SQLX_OFFLINE=true cargo test -p common --lib 2>&1 | tail -5

Step 3: Commit if changed

---

Task 15: Update Memory and Documentation

Files:

• Modify: /home/jgrusewski/.claude/projects/-home-jgrusewski-Work-foxhunt/memory/MEMORY.md

Step 1: Update MEMORY.md

Update the Architecture section to reflect the new 6-crate ML structure. Update the ML Model Training section with new crate paths. Remove any stale references to crates/ml/src/dqn/mixed_precision.rs etc.

Step 2: Final commit

git add -A
git commit -m "refactor(ml): complete crate split — 6 crates, 3-way parallel compilation

Split the monolithic ml crate (260K lines, 55s compile) into:
- ml-core: shared types, traits, features, data, checkpoint, safety
- ml-dqn: DQN + cuda_pipeline
- ml-ppo: PPO (independent of ml-dqn)
- ml-supervised: TFT, Mamba, Liquid, TGNN, TLOB, KAN, xLSTM, Diffusion
- ml-infra: hyperopt, ensemble, trainers, benchmark, deployment
- ml: thin facade re-exporting everything

3-way parallel compilation (ml-dqn + ml-ppo + ml-supervised).
Zero API changes for downstream consumers.
Incremental rebuild after model edit: ~55s → ~6-12s."

---

Execution Checklist

Task	Description	Est. Size	Depends On
1	Verify clean baseline	verify	—
2	Create sub-crate dirs + workspace	scaffold	1
3	Extract shared items from dqn/ + TradingAction	refactor	2
4	Write ml-core Cargo.toml	config	2
5	Move core modules to ml-core	large	3, 4
6	Write ml-dqn Cargo.toml	config	4
7a	Move DQN + cuda_pipeline to ml-dqn	large	5, 6
7b	Write ml-ppo Cargo.toml + move PPO	large	5
8	Write ml-supervised Cargo.toml	config	4
9	Move supervised models	large	5, 8
10	Write ml-infra Cargo.toml	config	6, 7b, 8
11	Move infra modules	large	7a, 7b, 9, 10
12	Clean up ml facade	medium	11
13	Full verification	verify	12
14	Update common dev-dep	small	12
15	Update memory + final commit	small	13

Note: Tasks 7a, 7b, and 9 can execute in parallel (they move independent module sets).

Critical Invariants (Verify After Every Task)

1. No stubs: Every function must be wired to its real implementation
2. No lost code: Every module in the inventory must be accounted for
3. No API changes: use ml::X::Y must work for all external consumers
4. Tests pass: Same test count, 0 failures
5. Clippy clean: 0 errors, 0 warnings
6. Feature flags work: --features cuda, --no-default-features, --features financial all compile