foxhunt/docs/plans/2026-02-20-codebase-deep-clean-design.md

Codebase Deep Clean Design

Date: 2026-02-20 Branch: feature/codebase-cleanup Strategy: Bottom-Up (safe layers first, each step builds on previous)

Context

After 30 waves of AI-assisted development, the ml/ crate has accumulated:

• 982 markdown docs (mostly AI-generated noise)
• ~4,846 lines of confirmed dead code across 11 files
• Duplicate module implementations (errors, ops, rewards, multi-step)
• 4 files exceeding 2,400 lines
• 37 standalone files cluttering ml/src/ root
• Tests scattered in source directories

Safety Gate

Every step must pass cargo check + cargo test before committing.

---

Step 1: Documentation Purge

Delete the entire docs/ directory (982 files). Clean slate.

No code references these files. Commit immediately after deletion.

Step 2: Dead Code Removal

Remove 11 files (4,846 lines) with zero imports:

File	Lines
ml/src/error_consolidated.rs	344
ml/src/operations.rs	234
ml/src/operations_safe.rs	323
ml/src/ops_production.rs	732
ml/src/dqn/multi_step_new.rs	126
ml/src/dqn/reward_elite.rs	522
ml/src/dqn/reward_coordinator.rs	568
ml/src/dqn/intrinsic_rewards.rs	499
ml/src/dqn/reward_simple_pnl.rs	538
ml/src/dqn/reward_sharpe_tests.rs	381
ml/src/dqn/reward/tests/reward_sharpe_tests.rs	~381

Also remove dead pub mod declarations from ml/src/lib.rs for: error_consolidated, operations, operations_safe, ops_production.

Audit all #[allow(dead_code)] markers — remove attribute if code is used, delete code if truly dead.

Step 3: Module Consolidation

• Fix deprecated FeatureVector54 → FeatureVector51 in common/src/lib.rs
• Remove the deprecated type alias if nothing references it
• Verify remaining ops files serve distinct purposes (tensor_ops, gradient_utils, safety/tensor_ops)

Step 4: Test Reorganization

Move integration tests from ml/src/ to ml/tests/:

From	To
ml/src/dqn/tests/*.rs	ml/tests/dqn/
ml/src/trainers/dqn/tests/*.rs	ml/tests/trainers/dqn/
ml/src/hyperopt/adapters/tests/*.rs	ml/tests/hyperopt/
ml/src/integration_test.rs	ml/tests/
ml/src/lib_test.rs	ml/tests/
ml/src/test_common.rs	ml/tests/common/mod.rs
ml/src/test_fixtures.rs	ml/tests/common/fixtures.rs

Keep inline #[cfg(test)] mod tests blocks — that's idiomatic Rust.

Step 5: Split Large Files

5a. trainer.rs (4,755 LOC) → trainers/dqn/

• trainer.rs (core struct, train loop)
• config.rs (training configuration)
• stats.rs (training statistics)
• early_stopping.rs (early stopping logic)
• logging.rs (training logging)

5b. hyperopt/adapters/dqn.rs (3,543 LOC)

• dqn.rs (core adapter, trimmed)
• dqn_params.rs (parameter space definitions)
• dqn_search.rs (search strategies)

5c. mamba/mod.rs (3,247 LOC)

• mod.rs (re-exports)
• model.rs (Mamba2 model, forward pass)
• layers.rs (layer implementations)
• config.rs (configuration)

5d. dqn/dqn.rs (2,405 LOC)

• dqn.rs (core DQN struct, trimmed)
• network.rs (architecture)
• training_ops.rs (training operations)

5e. Lower priority (>1,500 LOC)

• ppo/ppo.rs (1,742) — split similarly
• inference.rs (1,660) — split into engine + validator
• dqn/reward.rs (1,317) — split into reward/ module tree

Each split = separate commit for easy rollback.

Step 6: Folder Structure & Naming

Move root files into parent modules

File	Destination
batch_processing.rs	training/
benchmarks.rs	benchmark/
bridge.rs	integration/
cuda_compat.rs	common/
examples.rs	Remove or tests/
feature_cache.rs	features/cache.rs
inference.rs	deployment/
inference_validator.rs	deployment/
model_loader_integration.rs	integration/model_loader.rs
performance.rs	benchmark/
portfolio_transformer.rs	features/
preprocessing.rs	features/
production.rs	deployment/
qat_metrics_exporter.rs	memory_optimization/quantization_metrics.rs
real_data_loader.rs	data_loaders/real_data.rs
regime_detection.rs	regime/detection.rs
training.rs	training/ (merge)
training_pipeline.rs	training/pipeline.rs
validation.rs	data_validation/

Goal: ml/src/ root → ~8 files (lib.rs, error.rs, traits.rs, model.rs, model_factory.rs, model_registry.rs, random_model.rs).

File naming normalization

Current	Proposed
dqn_wave26_params_test.rs	dqn_params_test.rs
p0_integration_tests.rs	integration_tests.rs
target_update_comprehensive_tests.rs	target_update_tests.rs
ensemble_uncertainty_hyperopt_tests.rs	ensemble_hyperopt_tests.rs
rainbow_agent_impl.rs	rainbow_agent.rs (if no conflict)
qat_metrics_exporter.rs	quantization_metrics.rs
model_loader_integration.rs	model_loader.rs

Rules: snake_case, no wave/priority prefixes, no _new/_safe/_production/_consolidated suffixes, tests use <module>_tests.rs.

Remove empty cuda_common/ directory.