From cec36d307220f478f02df1db722d8801bc7300e1 Mon Sep 17 00:00:00 2001
From: jgrusewski <jgrusewski@localhost>
Date: Fri, 20 Feb 2026 13:08:10 +0100
Subject: [PATCH] docs: Add codebase deep clean design document

Comprehensive cleanup plan covering: documentation purge, dead code
removal (~4,846 lines across 11 files), module consolidation,
test reorganization, large file splitting, and folder restructuring.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 .../2026-02-20-codebase-deep-clean-design.md  | 148 ++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100644 docs/plans/2026-02-20-codebase-deep-clean-design.md

diff --git a/docs/plans/2026-02-20-codebase-deep-clean-design.md b/docs/plans/2026-02-20-codebase-deep-clean-design.md
new file mode 100644
index 000000000..3dd482587
--- /dev/null
+++ b/docs/plans/2026-02-20-codebase-deep-clean-design.md
@@ -0,0 +1,148 @@
+# Codebase Deep Clean Design
+
+**Date**: 2026-02-20
+**Branch**: feature/codebase-cleanup
+**Strategy**: Bottom-Up (safe layers first, each step builds on previous)
+
+## Context
+
+After 30 waves of AI-assisted development, the ml/ crate has accumulated:
+- 982 markdown docs (mostly AI-generated noise)
+- ~4,846 lines of confirmed dead code across 11 files
+- Duplicate module implementations (errors, ops, rewards, multi-step)
+- 4 files exceeding 2,400 lines
+- 37 standalone files cluttering ml/src/ root
+- Tests scattered in source directories
+
+## Safety Gate
+
+Every step must pass `cargo check` + `cargo test` before committing.
+
+---
+
+## Step 1: Documentation Purge
+
+Delete the entire `docs/` directory (982 files). Clean slate.
+
+No code references these files. Commit immediately after deletion.
+
+## Step 2: Dead Code Removal
+
+Remove 11 files (4,846 lines) with zero imports:
+
+| File | Lines |
+|---|---|
+| ml/src/error_consolidated.rs | 344 |
+| ml/src/operations.rs | 234 |
+| ml/src/operations_safe.rs | 323 |
+| ml/src/ops_production.rs | 732 |
+| ml/src/dqn/multi_step_new.rs | 126 |
+| ml/src/dqn/reward_elite.rs | 522 |
+| ml/src/dqn/reward_coordinator.rs | 568 |
+| ml/src/dqn/intrinsic_rewards.rs | 499 |
+| ml/src/dqn/reward_simple_pnl.rs | 538 |
+| ml/src/dqn/reward_sharpe_tests.rs | 381 |
+| ml/src/dqn/reward/tests/reward_sharpe_tests.rs | ~381 |
+
+Also remove dead `pub mod` declarations from ml/src/lib.rs for:
+error_consolidated, operations, operations_safe, ops_production.
+
+Audit all `#[allow(dead_code)]` markers — remove attribute if code is used, delete code if truly dead.
+
+## Step 3: Module Consolidation
+
+- Fix deprecated `FeatureVector54` → `FeatureVector51` in common/src/lib.rs
+- Remove the deprecated type alias if nothing references it
+- Verify remaining ops files serve distinct purposes (tensor_ops, gradient_utils, safety/tensor_ops)
+
+## Step 4: Test Reorganization
+
+Move integration tests from ml/src/ to ml/tests/:
+
+| From | To |
+|---|---|
+| ml/src/dqn/tests/*.rs | ml/tests/dqn/ |
+| ml/src/trainers/dqn/tests/*.rs | ml/tests/trainers/dqn/ |
+| ml/src/hyperopt/adapters/tests/*.rs | ml/tests/hyperopt/ |
+| ml/src/integration_test.rs | ml/tests/ |
+| ml/src/lib_test.rs | ml/tests/ |
+| ml/src/test_common.rs | ml/tests/common/mod.rs |
+| ml/src/test_fixtures.rs | ml/tests/common/fixtures.rs |
+
+Keep inline `#[cfg(test)] mod tests` blocks — that's idiomatic Rust.
+
+## Step 5: Split Large Files
+
+### 5a. trainer.rs (4,755 LOC) → trainers/dqn/
+- trainer.rs (core struct, train loop)
+- config.rs (training configuration)
+- stats.rs (training statistics)
+- early_stopping.rs (early stopping logic)
+- logging.rs (training logging)
+
+### 5b. hyperopt/adapters/dqn.rs (3,543 LOC)
+- dqn.rs (core adapter, trimmed)
+- dqn_params.rs (parameter space definitions)
+- dqn_search.rs (search strategies)
+
+### 5c. mamba/mod.rs (3,247 LOC)
+- mod.rs (re-exports)
+- model.rs (Mamba2 model, forward pass)
+- layers.rs (layer implementations)
+- config.rs (configuration)
+
+### 5d. dqn/dqn.rs (2,405 LOC)
+- dqn.rs (core DQN struct, trimmed)
+- network.rs (architecture)
+- training_ops.rs (training operations)
+
+### 5e. Lower priority (>1,500 LOC)
+- ppo/ppo.rs (1,742) — split similarly
+- inference.rs (1,660) — split into engine + validator
+- dqn/reward.rs (1,317) — split into reward/ module tree
+
+Each split = separate commit for easy rollback.
+
+## Step 6: Folder Structure & Naming
+
+### Move root files into parent modules
+
+| File | Destination |
+|---|---|
+| batch_processing.rs | training/ |
+| benchmarks.rs | benchmark/ |
+| bridge.rs | integration/ |
+| cuda_compat.rs | common/ |
+| examples.rs | Remove or tests/ |
+| feature_cache.rs | features/cache.rs |
+| inference.rs | deployment/ |
+| inference_validator.rs | deployment/ |
+| model_loader_integration.rs | integration/model_loader.rs |
+| performance.rs | benchmark/ |
+| portfolio_transformer.rs | features/ |
+| preprocessing.rs | features/ |
+| production.rs | deployment/ |
+| qat_metrics_exporter.rs | memory_optimization/quantization_metrics.rs |
+| real_data_loader.rs | data_loaders/real_data.rs |
+| regime_detection.rs | regime/detection.rs |
+| training.rs | training/ (merge) |
+| training_pipeline.rs | training/pipeline.rs |
+| validation.rs | data_validation/ |
+
+Goal: ml/src/ root → ~8 files (lib.rs, error.rs, traits.rs, model.rs, model_factory.rs, model_registry.rs, random_model.rs).
+
+### File naming normalization
+
+| Current | Proposed |
+|---|---|
+| dqn_wave26_params_test.rs | dqn_params_test.rs |
+| p0_integration_tests.rs | integration_tests.rs |
+| target_update_comprehensive_tests.rs | target_update_tests.rs |
+| ensemble_uncertainty_hyperopt_tests.rs | ensemble_hyperopt_tests.rs |
+| rainbow_agent_impl.rs | rainbow_agent.rs (if no conflict) |
+| qat_metrics_exporter.rs | quantization_metrics.rs |
+| model_loader_integration.rs | model_loader.rs |
+
+Rules: snake_case, no wave/priority prefixes, no _new/_safe/_production/_consolidated suffixes, tests use `<module>_tests.rs`.
+
+Remove empty `cuda_common/` directory.