# Codebase Deep Clean Design

**Date**: 2026-02-20
**Branch**: feature/codebase-cleanup
**Strategy**: Bottom-Up (safe layers first, each step builds on previous)

## Context

After 30 waves of AI-assisted development, the ml/ crate has accumulated:
- 982 markdown docs (mostly AI-generated noise)
- ~4,846 lines of confirmed dead code across 11 files
- Duplicate module implementations (errors, ops, rewards, multi-step)
- 4 files exceeding 2,400 lines
- 37 standalone files cluttering ml/src/ root
- Tests scattered in source directories

## Safety Gate

Every step must pass `cargo check` + `cargo test` before committing.

---

## Step 1: Documentation Purge

Delete the entire `docs/` directory (982 files). Clean slate.

No code references these files. Commit immediately after deletion.

## Step 2: Dead Code Removal

Remove 11 files (4,846 lines) with zero imports:

| File | Lines |
|---|---|
| ml/src/error_consolidated.rs | 344 |
| ml/src/operations.rs | 234 |
| ml/src/operations_safe.rs | 323 |
| ml/src/ops_production.rs | 732 |
| ml/src/dqn/multi_step_new.rs | 126 |
| ml/src/dqn/reward_elite.rs | 522 |
| ml/src/dqn/reward_coordinator.rs | 568 |
| ml/src/dqn/intrinsic_rewards.rs | 499 |
| ml/src/dqn/reward_simple_pnl.rs | 538 |
| ml/src/dqn/reward_sharpe_tests.rs | 381 |
| ml/src/dqn/reward/tests/reward_sharpe_tests.rs | ~381 |

Also remove dead `pub mod` declarations from ml/src/lib.rs for:
error_consolidated, operations, operations_safe, ops_production.

Audit all `#[allow(dead_code)]` markers — remove attribute if code is used, delete code if truly dead.

## Step 3: Module Consolidation

- Fix deprecated `FeatureVector54` → `FeatureVector51` in common/src/lib.rs
- Remove the deprecated type alias if nothing references it
- Verify remaining ops files serve distinct purposes (tensor_ops, gradient_utils, safety/tensor_ops)

## Step 4: Test Reorganization

Move integration tests from ml/src/ to ml/tests/:

| From | To |
|---|---|
| ml/src/dqn/tests/*.rs | ml/tests/dqn/ |
| ml/src/trainers/dqn/tests/*.rs | ml/tests/trainers/dqn/ |
| ml/src/hyperopt/adapters/tests/*.rs | ml/tests/hyperopt/ |
| ml/src/integration_test.rs | ml/tests/ |
| ml/src/lib_test.rs | ml/tests/ |
| ml/src/test_common.rs | ml/tests/common/mod.rs |
| ml/src/test_fixtures.rs | ml/tests/common/fixtures.rs |

Keep inline `#[cfg(test)] mod tests` blocks — that's idiomatic Rust.

## Step 5: Split Large Files

### 5a. trainer.rs (4,755 LOC) → trainers/dqn/
- trainer.rs (core struct, train loop)
- config.rs (training configuration)
- stats.rs (training statistics)
- early_stopping.rs (early stopping logic)
- logging.rs (training logging)

### 5b. hyperopt/adapters/dqn.rs (3,543 LOC)
- dqn.rs (core adapter, trimmed)
- dqn_params.rs (parameter space definitions)
- dqn_search.rs (search strategies)

### 5c. mamba/mod.rs (3,247 LOC)
- mod.rs (re-exports)
- model.rs (Mamba2 model, forward pass)
- layers.rs (layer implementations)
- config.rs (configuration)

### 5d. dqn/dqn.rs (2,405 LOC)
- dqn.rs (core DQN struct, trimmed)
- network.rs (architecture)
- training_ops.rs (training operations)

### 5e. Lower priority (>1,500 LOC)
- ppo/ppo.rs (1,742) — split similarly
- inference.rs (1,660) — split into engine + validator
- dqn/reward.rs (1,317) — split into reward/ module tree

Each split = separate commit for easy rollback.

## Step 6: Folder Structure & Naming

### Move root files into parent modules

| File | Destination |
|---|---|
| batch_processing.rs | training/ |
| benchmarks.rs | benchmark/ |
| bridge.rs | integration/ |
| cuda_compat.rs | common/ |
| examples.rs | Remove or tests/ |
| feature_cache.rs | features/cache.rs |
| inference.rs | deployment/ |
| inference_validator.rs | deployment/ |
| model_loader_integration.rs | integration/model_loader.rs |
| performance.rs | benchmark/ |
| portfolio_transformer.rs | features/ |
| preprocessing.rs | features/ |
| production.rs | deployment/ |
| qat_metrics_exporter.rs | memory_optimization/quantization_metrics.rs |
| real_data_loader.rs | data_loaders/real_data.rs |
| regime_detection.rs | regime/detection.rs |
| training.rs | training/ (merge) |
| training_pipeline.rs | training/pipeline.rs |
| validation.rs | data_validation/ |

Goal: ml/src/ root → ~8 files (lib.rs, error.rs, traits.rs, model.rs, model_factory.rs, model_registry.rs, random_model.rs).

### File naming normalization

| Current | Proposed |
|---|---|
| dqn_wave26_params_test.rs | dqn_params_test.rs |
| p0_integration_tests.rs | integration_tests.rs |
| target_update_comprehensive_tests.rs | target_update_tests.rs |
| ensemble_uncertainty_hyperopt_tests.rs | ensemble_hyperopt_tests.rs |
| rainbow_agent_impl.rs | rainbow_agent.rs (if no conflict) |
| qat_metrics_exporter.rs | quantization_metrics.rs |
| model_loader_integration.rs | model_loader.rs |

Rules: snake_case, no wave/priority prefixes, no _new/_safe/_production/_consolidated suffixes, tests use `<module>_tests.rs`.

Remove empty `cuda_common/` directory.