foxhunt/docs/plans/2026-02-20-codebase-deep-clean-implementation.md

Codebase Deep Clean — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Systematically clean a Rust codebase that accumulated technical debt across 30 waves of AI-assisted development — removing dead code, consolidating duplicates, reorganizing tests, splitting oversized files, and restructuring the folder layout.

Architecture: Bottom-Up strategy. Each task is a safe, isolated layer that builds on the previous. Every commit must pass SQLX_OFFLINE=true cargo check --workspace and cargo test -p ml --lib. The pre-commit hook enforces workspace compilation.

Tech Stack: Rust (Cargo workspace), sqlx (offline mode), candle (ML framework)

---

Task 1: Documentation Purge

Files:

• Delete: entire docs/ directory (982 files) EXCEPT docs/plans/ (this plan)
• Delete: archive/ directory if it exists at project root

Step 1: Preserve the plans directory

# Move plans out temporarily
cp -r docs/plans /tmp/foxhunt-plans-backup

Step 2: Delete docs and archive

rm -rf docs/
rm -rf archive/

Step 3: Restore plans

mkdir -p docs/plans
cp -r /tmp/foxhunt-plans-backup/* docs/plans/
rm -rf /tmp/foxhunt-plans-backup

Step 4: Verify compilation

SQLX_OFFLINE=true cargo check --workspace

Expected: passes (no code references docs)

Step 5: Commit

git add -A docs/ archive/
git commit -m "chore: purge all documentation except implementation plans

Remove 982 AI-generated markdown files and archive directory.
No code references these files. Plans directory preserved."

---

Task 2: Dead Code Removal — Dead Files

Files to delete:

• ml/src/error_consolidated.rs (344 lines — declared in lib.rs but never imported)
• ml/src/operations.rs (234 lines — never imported, tensor_ops.rs is the active version)
• ml/src/operations_safe.rs (323 lines — never imported)
• ml/src/ops_production.rs (732 lines — never imported, largest dead file)
• ml/src/dqn/multi_step_new.rs (126 lines — experimental variant, never imported)
• ml/src/dqn/reward_elite.rs (522 lines — not declared in dqn/mod.rs)
• ml/src/dqn/reward_coordinator.rs (568 lines — not declared in dqn/mod.rs)
• ml/src/dqn/intrinsic_rewards.rs (499 lines — not declared in dqn/mod.rs)
• ml/src/dqn/reward_simple_pnl.rs (538 lines — not declared in dqn/mod.rs)
• ml/src/dqn/reward_sharpe_tests.rs (381 lines — orphaned test file, not a module)
• ml/src/dqn/reward/tests/reward_sharpe_tests.rs (duplicate orphaned test)

Files to modify:

• ml/src/lib.rs — remove 4 dead pub mod declarations

Step 1: Delete the 11 dead files

rm ml/src/error_consolidated.rs
rm ml/src/operations.rs
rm ml/src/operations_safe.rs
rm ml/src/ops_production.rs
rm ml/src/dqn/multi_step_new.rs
rm ml/src/dqn/reward_elite.rs
rm ml/src/dqn/reward_coordinator.rs
rm ml/src/dqn/intrinsic_rewards.rs
rm ml/src/dqn/reward_simple_pnl.rs
rm ml/src/dqn/reward_sharpe_tests.rs
rm -rf ml/src/dqn/reward/tests/

Step 2: Remove dead module declarations from ml/src/lib.rs

Remove these 4 lines (approx lines 1070, 1076, 1085-1086):

// DELETE: pub mod error_consolidated;
// DELETE: pub mod operations;
// DELETE: pub mod operations_safe; // Safe operations module
// DELETE: pub mod ops_production; // Production ML operations

Step 3: Verify compilation

SQLX_OFFLINE=true cargo check --workspace

Expected: passes — these files had zero imports.

Step 4: Run tests

cargo test -p ml --lib 2>&1 | tail -5

Expected: all tests pass.

Step 5: Commit

git add -A
git commit -m "chore(ml): remove 11 dead code files (~4,267 lines)

Files had zero imports across the entire workspace:
- error_consolidated.rs, operations.rs, operations_safe.rs, ops_production.rs
- dqn/multi_step_new.rs, reward_elite.rs, reward_coordinator.rs
- dqn/intrinsic_rewards.rs, reward_simple_pnl.rs, reward_sharpe_tests.rs
- dqn/reward/tests/ (orphaned duplicate)
Also removed 4 dead pub mod declarations from lib.rs."

---

Task 3: Dead Code Removal — Orphan Source Files

Investigation: These files exist in ml/src/ but have NO mod declaration in lib.rs:

• ml/src/integration_test.rs
• ml/src/lib_test.rs
• ml/src/test_common.rs
• ml/src/test_fixtures.rs

Also check:

• ml/src/dqn/tests/factored_integration_tests.rs — exists in directory but NOT declared in ml/src/dqn/tests/mod.rs
• ml/src/hyperopt/adapters/tests/ — exists but NOT declared in ml/src/hyperopt/adapters/mod.rs

Step 1: Verify these files are truly unreachable

# Confirm no mod declaration in lib.rs
grep -n "integration_test\|lib_test\|test_common\|test_fixtures" ml/src/lib.rs

# Confirm factored_integration_tests not declared
grep -n "factored_integration_tests" ml/src/dqn/tests/mod.rs

# Confirm hyperopt tests not declared
grep -n "tests" ml/src/hyperopt/adapters/mod.rs

Expected: no output for any of these (confirming the files are dead).

Step 2: Delete orphan files

rm ml/src/integration_test.rs
rm ml/src/lib_test.rs
rm ml/src/test_common.rs
rm ml/src/test_fixtures.rs
rm ml/src/dqn/tests/factored_integration_tests.rs
rm -rf ml/src/hyperopt/adapters/tests/

Step 3: Verify compilation and tests

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml --lib 2>&1 | tail -5

Step 4: Commit

git add -A
git commit -m "chore(ml): remove orphan source files with no mod declarations

Files existed on disk but were never compiled — no mod declaration
in lib.rs, dqn/tests/mod.rs, or hyperopt/adapters/mod.rs."

---

Task 4: Dead Code Removal — #[allow(dead_code)] Audit

Step 1: Find all #[allow(dead_code)] markers in ml/

grep -rn '#\[allow(dead_code)\]' ml/src/ | head -30

Step 2: For each marker, determine if the item is actually used

For each file+item found:

• Search for use references across the workspace
• If referenced: remove #[allow(dead_code)] attribute (it's not dead)
• If not referenced: delete the entire item (struct/fn/const/etc.)

Step 3: Verify compilation and tests

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml --lib 2>&1 | tail -5

Step 4: Commit

git add -A
git commit -m "chore(ml): audit and clean #[allow(dead_code)] markers

Remove attribute where code is actually used, delete truly dead items."

---

Task 5: Module Consolidation — Deprecated Types

Files:

• Modify: common/src/features/types.rs:8 — FeatureVector54 definition
• Modify: common/src/lib.rs:87 — re-export of FeatureVector54
• Modify: ml/src/features/extraction.rs:64-65 — deprecated alias
• Modify: ml/tests/dqn_call_sites_remaining_test.rs:9 — uses FeatureVector54

Step 1: Check all usages of FeatureVector54

grep -rn "FeatureVector54" --include="*.rs" .

Examine each hit. If only used in:

• type definitions (declaring the alias) — safe to remove
• test files — update to use FeatureVector (51-dim)
• doc comments or markdown — already deleted in Task 1

Step 2: Update test file

In ml/tests/dqn_call_sites_remaining_test.rs:9, change:

// FROM:
use ml::types::FeatureVector54;
// TO:
use ml::features::extraction::FeatureVector;

Update any array literals from [f64; 54] to [f64; 51] in the same file.

Step 3: Remove deprecated alias from ml/src/features/extraction.rs

Remove lines 64-65:

// DELETE: #[deprecated(since = "WAVE 10", note = "Use FeatureVector (51 dimensions) instead")]
// DELETE: pub type FeatureVector54 = [f64; 54];

Also remove FeatureVector43 if unused (line 68).

Step 4: Remove from common/src/features/types.rs

Remove line 8:

// DELETE: pub type FeatureVector54 = [f64; 54];

Step 5: Remove re-export from common/src/lib.rs

Line 87 — remove FeatureVector54 from the re-export list. Change:

// FROM:
    FeatureVector54, BarData,
// TO:
    BarData,

Step 6: Verify compilation and tests

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml --lib 2>&1 | tail -5

Expected: deprecation warning disappears.

Step 7: Commit

git add -A
git commit -m "chore: remove deprecated FeatureVector54 type alias

Dimension was reduced from 54 to 51 in WAVE 10. All usages now
use FeatureVector ([f64; 51]) directly."

---

Task 6: Test Reorganization — Move Integration Tests

The tests in ml/src/dqn/tests/ and ml/src/trainers/dqn/tests/ are declared as #[cfg(test)] submodules. They use use crate:: imports which gives them internal access. Moving to ml/tests/ requires converting use crate:: → use ml:: (only public API accessible).

IMPORTANT: Before moving, verify each test file only uses pub items. If a test accesses private internals, it must stay as an inline #[cfg(test)] module.

Step 1: Audit test imports for private access

For each test file in ml/src/dqn/tests/ and ml/src/trainers/dqn/tests/:

grep "use crate::" ml/src/dqn/tests/*.rs ml/src/trainers/dqn/tests/*.rs

Check each import: is the item pub in its module? If not, the test must stay in-tree.

Step 2: For movable tests — create destination directories

mkdir -p ml/tests/dqn
mkdir -p ml/tests/trainers/dqn

Step 3: Move test files and convert imports

For each file being moved:

1. Copy to new location
2. Replace use crate:: with use ml:: throughout
3. Remove from source directory
4. Update parent mod.rs to remove the #[cfg(test)] mod <name>; declaration

Example for ml/src/dqn/tests/activation_tests.rs:

cp ml/src/dqn/tests/activation_tests.rs ml/tests/dqn/activation_tests.rs
# Edit: replace "use crate::" with "use ml::" in the new file
rm ml/src/dqn/tests/activation_tests.rs
# Edit: remove "mod activation_tests;" from ml/src/dqn/tests/mod.rs

Repeat for all movable test files.

Step 4: Clean up empty test directories

If ml/src/dqn/tests/mod.rs has no more module declarations, remove the directory:

rm -rf ml/src/dqn/tests/

Update ml/src/dqn/mod.rs to remove mod tests; declaration (line 150).

Same for ml/src/trainers/dqn/tests/ — update ml/src/trainers/dqn/mod.rs to remove mod tests; (line 39).

Step 5: Rename test files

Apply naming conventions during the move:

• target_update_comprehensive_tests.rs → target_update_tests.rs
• p0_integration_tests.rs → integration_tests.rs
• p1_integration_tests.rs → p1_tests.rs
• ensemble_uncertainty_hyperopt_tests.rs → ensemble_hyperopt_tests.rs
• dqn_wave26_params_test.rs → dqn_params_test.rs (already moved in Task 3)

Step 6: Verify compilation and tests

SQLX_OFFLINE=true cargo check --workspace
cargo test -p ml 2>&1 | tail -20

Expected: all tests still pass, now running from ml/tests/ instead of ml/src/.

Step 7: Commit

git add -A
git commit -m "refactor(ml): reorganize tests — move integration tests to ml/tests/

Move test files from ml/src/*/tests/ to ml/tests/. Convert
crate-internal imports to public API imports. Rename files to
follow naming conventions (no wave/priority prefixes)."

---

Task 7: Split Large File — trainers/dqn/trainer.rs (4,755 LOC)

This is the largest file. Read it first to identify logical boundaries.

Step 1: Read and map the file structure

grep -n "^pub struct\|^pub enum\|^pub fn\|^impl\|^pub trait\|^mod " ml/src/trainers/dqn/trainer.rs | head -40

Identify logical sections:

• Configuration structs → config.rs
• Training statistics / metrics → stats.rs
• Early stopping logic → early_stopping.rs
• Logging / reporting → logging.rs
• Core trainer struct + train loop → stays in trainer.rs

Step 2: Extract config types to ml/src/trainers/dqn/config.rs

Create new file with all config-related structs. Add pub use re-exports in mod.rs.

Step 3: Extract stats types to ml/src/trainers/dqn/stats.rs

Move training statistics, metrics structs, and their impl blocks.

Step 4: Extract early stopping to ml/src/trainers/dqn/early_stopping.rs

Move EarlyStopping struct and its implementation.

Step 5: Extract logging to ml/src/trainers/dqn/logging.rs

Move logging/reporting functions and related types.

Step 6: Update ml/src/trainers/dqn/mod.rs

Add new module declarations and re-exports:

mod config;
mod early_stopping;
mod logging;
mod stats;
pub mod trainer;

pub use config::*;
pub use early_stopping::*;
pub use logging::*;
pub use stats::*;
pub use trainer::*;

Step 7: Update imports in trainer.rs

Replace self-contained type references with use super::config::* etc.

Step 8: Verify compilation and tests

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml 2>&1 | tail -10

Step 9: Commit

git add -A
git commit -m "refactor(ml): split trainers/dqn/trainer.rs (4,755 LOC) into sub-modules

Extract config, stats, early_stopping, and logging into separate
files. Core trainer struct and train loop remain in trainer.rs."

---

Task 8: Split Large File — hyperopt/adapters/dqn.rs (3,543 LOC)

Step 1: Read and map the file structure

grep -n "^pub struct\|^pub enum\|^pub fn\|^impl\|^pub trait" ml/src/hyperopt/adapters/dqn.rs | head -30

Step 2: Extract parameter space definitions to dqn_params.rs

Step 3: Extract search strategy implementations to dqn_search.rs

Step 4: Update ml/src/hyperopt/adapters/mod.rs

Add sub-module declarations. Preserve existing public re-exports on line 61:

pub use dqn::{DQNMetrics, DQNParams, DQNTrainer};

Step 5: Verify compilation and tests

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml 2>&1 | tail -10

Step 6: Commit

git add -A
git commit -m "refactor(ml): split hyperopt/adapters/dqn.rs (3,543 LOC) into sub-modules

Extract parameter space definitions and search strategies."

---

Task 9: Split Large File — mamba/mod.rs (3,247 LOC)

Step 1: Read and map the file

grep -n "^pub struct\|^pub enum\|^pub fn\|^impl\|^pub trait" ml/src/mamba/mod.rs | head -30

Step 2: Extract Mamba2 model struct and forward pass to model.rs

Step 3: Extract layer implementations to layers.rs

Step 4: Extract config to config.rs

Step 5: Rewrite mod.rs as re-export hub

Keep only module declarations and pub use re-exports.

Step 6: Verify compilation and tests

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml 2>&1 | tail -10

Step 7: Commit

git add -A
git commit -m "refactor(ml): split mamba/mod.rs (3,247 LOC) into sub-modules

Extract model, layers, and config into separate files."

---

Task 10: Split Large File — dqn/dqn.rs (2,405 LOC)

Step 1: Read and map the file

grep -n "^pub struct\|^pub enum\|^pub fn\|^impl\|^pub trait" ml/src/dqn/dqn.rs | head -30

Step 2: Extract network architecture to network.rs

Step 3: Extract training operations to training_ops.rs

Step 4: Update ml/src/dqn/mod.rs

Existing line 71 exports pub use dqn::{DQN, DQNConfig}. The DQN and DQNConfig must remain accessible from the same path after the split.

Step 5: Verify compilation and tests

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml 2>&1 | tail -10

Step 6: Commit

git add -A
git commit -m "refactor(ml): split dqn/dqn.rs (2,405 LOC) into sub-modules

Extract network architecture and training operations."

---

Task 11: Split Lower-Priority Large Files (>1,500 LOC)

Apply the same pattern to:

11a. ml/src/ppo/ppo.rs (1,742 LOC)

• Extract config → ppo/config.rs
• Extract network → ppo/network.rs
• Keep core PPO logic in ppo.rs

11b. ml/src/inference.rs (1,660 LOC)

• Extract engine → inference_engine.rs or deployment/engine.rs
• Extract metrics → inference_metrics.rs

11c. ml/src/dqn/reward.rs (1,317 LOC)

• Extract into reward/ module tree: reward/mod.rs, reward/config.rs, reward/functions.rs

Each sub-split gets its own verify + commit cycle.

---

Task 12: Folder Structure — Remove Zero-Import Root Files

From the analysis, these root files have ZERO imports across the workspace:

• ml/src/benchmarks.rs (730 lines, 0 importers)
• ml/src/model_loader_integration.rs (726 lines, 0 importers)
• ml/src/production.rs (61 lines, 0 importers — stub only)
• ml/src/validation.rs (137 lines, 0 importers)

Step 1: Verify zero imports

grep -rn "use crate::benchmarks\|use ml::benchmarks" ml/ services/ common/
grep -rn "use crate::model_loader_integration\|use ml::model_loader_integration" ml/ services/ common/
grep -rn "use crate::production\|use ml::production" ml/ services/ common/
grep -rn "use crate::validation\b\|use ml::validation\b" ml/ services/ common/

Step 2: Delete confirmed dead files and their mod declarations from lib.rs

Step 3: Verify and commit

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml --lib 2>&1 | tail -5
git add -A
git commit -m "chore(ml): remove 4 unused root-level modules (~1,654 lines)

benchmarks.rs, model_loader_integration.rs, production.rs,
validation.rs — all had zero imports across the workspace."

---

Task 13: Folder Structure — Move Low-Connectivity Root Files

Files with 1-2 importers. Move the file, update all import paths. No re-exports.

Approach for every move: Move file → cargo check --workspace → compiler lists every broken import → fix them all → verify → commit.

Phase 1 (1 importer each):

File	Lines	Destination	Importer to update
batch_processing.rs	710	training/batch_processing.rs	unsafe_validation_tests.rs
bridge.rs	361	Keep at root (used by inference.rs)	—
portfolio_transformer.rs	814	features/portfolio_transformer.rs	portfolio_integration.rs
qat_metrics_exporter.rs	424	memory_optimization/quantization_metrics.rs	self-ref only

Phase 2 (2 importers):

File	Lines	Destination	Importers to update
examples.rs	900	DELETE (dead — only self-references)	verify first
feature_cache.rs	350	features/cache.rs	2 files
inference_validator.rs	529	Keep alongside inference.rs	2 files
regime_detection.rs	118	regime/detection.rs	2 files

For each move:

1. Move the file with git mv
2. Add mod declaration in destination's mod.rs
3. Remove old mod declaration from lib.rs
4. SQLX_OFFLINE=true cargo check --workspace — compiler lists every broken import
5. Fix every use crate::old_module → use crate::new_module::path across the workspace
6. Re-run cargo check --workspace — expect zero errors
7. Commit

---

Task 14: Folder Structure — Move Hub Modules

Same approach as Task 13: move + fix all imports. No re-exports, no aliases. The compiler is the safety net — cargo check enumerates every broken path.

14a. performance.rs (481 lines, 4 importers) → benchmark/performance.rs

Move file, fix 4 import sites.

14b. cuda_compat.rs (487 lines, 10 importers) → stays at root

The 10 importers are all within ml/src/ neural modules (attention, residual, mamba, etc.). This is a cross-cutting utility used by many neural layers — keeping at root is the correct domain decision, not a workaround. No move needed.

14c. inference.rs (1,661 lines, 6 importers)

Will be split in Task 11b. Defer move until after split.

14d. preprocessing.rs (458 lines, 11 importers) → features/preprocessing.rs

Move file, fix 11 import sites. Use cargo check output to find them all.

git mv ml/src/preprocessing.rs ml/src/features/preprocessing.rs
# Add `pub mod preprocessing;` to ml/src/features/mod.rs
# Remove `pub mod preprocessing;` from ml/src/lib.rs
SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep "error\[E0432\]"
# Fix each broken import: use crate::preprocessing → use crate::features::preprocessing

14e. real_data_loader.rs (649 lines, 27 importers) → data_loaders/real_data.rs

Move file, fix 27 import sites. Mechanical but safe — compiler catches everything.

git mv ml/src/real_data_loader.rs ml/src/data_loaders/real_data.rs
SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep "error\[E0432\]"
# Fix all 27 broken imports

14f. training.rs (550 lines, 40 importers) → training/core.rs

Highest-volume move. Move file, fix 40 import sites.

git mv ml/src/training.rs ml/src/training/core.rs
# Add `pub mod core;` to ml/src/training/mod.rs (create if needed)
# Remove `pub mod training;` from ml/src/lib.rs, add `pub mod training;` pointing to directory
SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep "error\[E0432\]"
# Fix all 40 broken imports: use crate::training:: → use crate::training::core::

For each move: commit separately so git revert targets one module at a time.

---

Task 15: File Naming Normalization

Rename remaining files with poor names (applies to wherever they ended up after moves):

Current	Proposed
rainbow_agent_impl.rs	rainbow_agent.rs

Step 1: Check for conflicts

ls ml/src/dqn/rainbow_agent.rs 2>&1

Step 2: Rename with git mv

git mv ml/src/dqn/rainbow_agent_impl.rs ml/src/dqn/rainbow_agent.rs

Step 3: Update mod declaration in dqn/mod.rs

// FROM: mod rainbow_agent_impl;
// TO:   mod rainbow_agent;

Update any pub use rainbow_agent_impl:: to pub use rainbow_agent::.

Step 4: Verify and commit

SQLX_OFFLINE=true cargo check --workspace && cargo test -p ml 2>&1 | tail -5
git add -A
git commit -m "refactor(ml): normalize file names — remove _impl suffix"

---

Task 16: Final Cleanup

Step 1: Remove empty directories

find ml/src -type d -empty -print
# Review, then:
find ml/src -type d -empty -delete

Step 2: Check cuda_common/ directory

NOTE: Investigation found cuda_common/ is NOT empty (contains kernel_fusion.cu). Do NOT delete. Verify:

ls -la ml/src/cuda_common/

Step 3: Run full test suite

SQLX_OFFLINE=true cargo check --workspace
cargo test -p ml 2>&1 | tail -20

Step 4: Count remaining files and lines

find ml/src -name "*.rs" | wc -l
find ml/src -name "*.rs" -exec wc -l {} + | tail -1
ls ml/src/*.rs | wc -l

Document the before/after numbers.

Step 5: Final commit

git add -A
git commit -m "chore(ml): final cleanup — remove empty directories

Complete codebase deep clean:
- Removed ~X dead files (~Y lines)
- Split Z oversized files into sub-modules
- Reorganized tests into ml/tests/
- Restructured ml/src/ root from N files to M files"

---

Risk Notes

• Task 6 (test moves): Tests using use crate:: have crate-internal access. After moving to ml/tests/, they can only access pub items via use ml::. Verify each test before moving.
• Tasks 7-11 (file splits): Read the full file first. Identify impl blocks that span multiple logical sections — these may need to be split across files with use super::* to access the parent type.
• Tasks 13-14 (folder moves): Move file, then use cargo check --workspace to find every broken import. Fix them all — no re-exports, no aliases. The compiler catches everything; it's mechanical work, not risky work. Commit each module move separately for easy rollback.
• Pre-commit hook: Runs SQLX_OFFLINE=true cargo check --workspace. If new sqlx queries are added, the .sqlx/ cache must be updated with a running database.