From cbd36fb7eb0a79c8447a01898723c32d59ffe817 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 1 Mar 2026 19:22:22 +0100 Subject: [PATCH] docs: add codebase consolidation & DRY cleanup design Co-Authored-By: Claude Opus 4.6 --- ...026-03-01-codebase-consolidation-design.md | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 docs/plans/2026-03-01-codebase-consolidation-design.md diff --git a/docs/plans/2026-03-01-codebase-consolidation-design.md b/docs/plans/2026-03-01-codebase-consolidation-design.md new file mode 100644 index 000000000..f6171f22e --- /dev/null +++ b/docs/plans/2026-03-01-codebase-consolidation-design.md @@ -0,0 +1,144 @@ +# Codebase Consolidation & DRY Cleanup — Design + +**Date**: 2026-03-01 +**Branch**: `feature/codebase-consolidation` +**Estimated savings**: ~13,100 lines + +## Context + +Following the ML inference cleanup (-8,281 lines), a comprehensive codebase audit +revealed 4 major cleanup areas across 543K LOC (crates+services). The `ml` crate +alone is 247K LOC but is well-structured; the biggest opportunities are in +shared infrastructure, compliance boilerplate, and service-level glue code. + +## Area 1: DRY Violations (~600 lines saved) + +### Problem + +Three type definitions exist as exact duplicates in multiple crates: + +| Type | Canonical Location | Duplicate Location | +|---|---|---| +| `RetryStrategy` enum | `common/src/error.rs:159` | `trading_engine/src/types/error.rs:18` | +| `ErrorSeverity` enum | `common/src/error.rs:132` | `data/src/error.rs:441`, `database/src/error.rs:119` | +| `RetryConfig` struct | `common/src/resilience/retry.rs:40` | `storage/src/model_helpers.rs:141` | + +### Design + +1. Keep canonical definitions in `common/` +2. Delete duplicates from `trading_engine`, `data`, `database`, `storage` +3. Re-export from `common` crate root if not already +4. Update all `use` statements in consuming code +5. Verify with `cargo check --workspace` + +### Risk + +Low. These are exact duplicates with identical APIs. The only risk is missed +import sites, which the compiler will catch. + +## Area 2: `common/src/types.rs` Split (structural, ~4,909 LOC) + +### Problem + +`common/src/types.rs` is a 4,909-line god file containing 6+ unrelated domains: +type aliases, service types, market data types, order types, connection types, +error types, and currency types. This makes the file hard to navigate and +violates the project's 500-line-per-file convention. + +### Design + +Split into focused modules under `common/src/types/`: + +| Module | Content | Est. LOC | +|---|---|---| +| `aliases.rs` | `AsyncResult`, `SharedHashMap`, `PositionMap`, etc. | ~80 | +| `service.rs` | `ServiceId`, `ServiceStatus`, `ConfigVersion` | ~120 | +| `market_data.rs` | `MarketDataEvent`, `QuoteEvent`, `TradeEvent`, `BarEvent`, orderbook types | ~1,500 | +| `order.rs` | `OrderType`, `OrderSide`, `OrderStatus`, `OrderEvent`, `BrokerType` | ~400 | +| `connection.rs` | `ConnectionInfo`, `ConnectionEvent`, `ConnectionStatus`, `ResourceLimits` | ~200 | +| `error.rs` | `CommonTypeError` + trait impls | ~200 | +| `currency.rs` | `Currency` enum + Display/Default | ~100 | +| `mod.rs` | `pub use` re-exports for backward compatibility | ~30 | + +### Backward Compatibility + +`types/mod.rs` will `pub use` everything, so existing `use common::types::*` +and `use common::types::QuoteEvent` imports continue working unchanged. + +### Risk + +Low. Pure structural refactor with re-exports. No API changes. + +## Area 3: `trading_engine/compliance/` Audit (~11,000 lines saved) + +### Problem + +The compliance directory contains 16,068 LOC across 9 files: + +- `iso27001_compliance.rs` (3,217) — ISMS scope, organization info, facilities +- `compliance_reporting.rs` (2,406) — Report generation +- `sox_compliance.rs` (2,262) — Sarbanes-Oxley controls +- `audit_trails.rs` (1,847) — Audit trail storage +- `transaction_reporting.rs` (1,714) — MiFID/Dodd-Frank TCA +- `automated_reporting.rs` (1,628) — Automated report scheduling +- `best_execution.rs` (1,101) — 29 struct/enums for best execution analysis +- `mod.rs` (1,107) — Module orchestration +- `regulatory_api.rs` (786) — API endpoints + +Much of this appears to be enterprise scaffolding (ISO27001 `OrganizationInfo`, +`Location`, `FacilityType`, `ContactInfo`, `ISMSScope`) that is unlikely to be +called from production code paths in an HFT futures trading system. + +### Design + +1. **Audit usage**: grep for each public type/function to find callers outside tests +2. **Categorize**: active (called in production) vs dormant (tests-only or zero callers) +3. **Delete dormant code**: Remove types and functions with zero production callers +4. **Consolidate active code**: Merge remaining functions into fewer, smaller files +5. **Keep essentials**: audit trail logging, basic compliance checks that risk/ references + +### Risk + +Medium. Need to verify each type's usage carefully. Some compliance types may be +referenced transitively through trait bounds. The `#[allow(unused_variables)]` +at the top of `best_execution.rs` is a red flag suggesting it may be entirely unused. + +## Area 4: `trading_service/enhanced_ml.rs` Cleanup (~1,500 lines saved) + +### Problem + +At 3,027 LOC, this file implements the ML gRPC service with: +- Runtime model tracking (`RuntimeModelInfo`) duplicating `ml::ModelMetadata` +- Model loading code that should delegate to `EnsembleModelAdapter` +- Known `Status::unavailable` stubs for retrain and feature_importance +- PPO/TFT inference code with hardcoded fallbacks + +### Design + +1. Audit for legacy patterns matching deleted `MLFeatureExtractor`/`SimpleDQNAdapter` +2. Consolidate `RuntimeModelInfo` with existing `ml::ModelMetadata` types +3. Simplify model loading to use `ml::ensemble::build_production_strategy()` +4. Remove dead code paths that duplicate what `EnsembleModelAdapter` now handles +5. Keep the gRPC service interface — just slim the implementation + +### Risk + +Medium. This file implements active gRPC endpoints. Changes must preserve the +proto contract. Focus on internal simplification, not API changes. + +## Execution Order + +1. **DRY violations** (creates canonical shared types) +2. **types.rs split** (structural, no deps on other areas) +3. **compliance/ audit** (largest deletion target) +4. **enhanced_ml.rs cleanup** (builds on Area 1 patterns) + +## Verification + +After each area: +- `SQLX_OFFLINE=true cargo check --workspace` — zero errors +- `SQLX_OFFLINE=true cargo clippy -p --lib -- -D warnings` +- `SQLX_OFFLINE=true cargo test -p --lib` +- `git diff --stat main` — track cumulative line savings + +Final target: **-13,000+ net lines deleted** while preserving all production functionality.