foxhunt/docs/codebase-cleanup/CLEANUP_ANALYSIS_REPORT.md

# Foxhunt Codebase Cleanup Analysis Report

**Date**: 2025-11-27
**Analyzer**: Claude Code (SPARC Methodology)
**Project Size**: 58GB total, 527,700 LOC Rust code

---

## Executive Summary

The Foxhunt HFT trading system is a substantial Rust codebase with integrated ML/RL components. Analysis reveals **excellent code quality** in core modules but significant **technical debt** from rapid AI-assisted development, primarily manifesting as:

1. **506 working files in root folder** (should be 0)
2. **51GB target directory** (needs cleanup)
3. **3-4% code duplication** (~15,000-20,000 lines)
4. **22-27 unused dependencies**
5. **Arrow v55/v56 version conflict** causing 24 duplicate crates
6. **40% test coverage in critical risk management code**

### Quick Impact Summary

| Cleanup Action | Disk Savings | Build Impact | Risk Level |
|----------------|--------------|--------------|------------|
| Root folder cleanup | ~50MB | None | ✅ LOW |
| Target directory clean | ~51GB | Rebuild needed | ✅ LOW |
| Remove unused deps | ~100MB | -5% build time | ✅ LOW |
| Fix Arrow conflict | ~2GB | -15% build time | 🟡 MEDIUM |
| Consolidate duplication | ~8,400 LOC | None | 🟡 MEDIUM |

---

## Phase 1: Root Folder Cleanup (CRITICAL)

### Current State

```
Root folder file count:
- Markdown/Text files: 474
- Shell scripts: 27
- Python files: 4
- Rust files: 1
- JSON files: 2 (example_backtest_results.json, sqlx-data.json)
- Temp files: 2 (=2.11.0, =2.12.0)
TOTAL: 510 files that should NOT be in root
```

### Files Categories

**Agent Reports (300+ files)**:
- AGENT_*_*.md/txt - Development session reports
- WAVE*_*.md/txt - Development wave summaries
- DQN_*.md/txt - DQN development artifacts
- BUG*_*.md/txt - Bug investigation reports

**Deployment Scripts (27 files)**:
- deploy_*.sh - Kubernetes/RunPod deployment scripts
- monitor_*.sh - Pod monitoring scripts
- terminate_*.sh - Cleanup scripts
- test_*.sh - Manual test scripts

**Temporary/Debug Files**:
- =2.11.0, =2.12.0 - Leftover from failed commands
- *.py - One-off validation scripts
- *.rs (standalone) - Debug inspection scripts

### Recommended Actions

```bash
# Create archive directory
mkdir -p archive/reports archive/scripts archive/temp

# Move agent reports
mv AGENT*.md AGENT*.txt archive/reports/
mv WAVE*.md WAVE*.txt archive/reports/
mv DQN_*.md DQN_*.txt archive/reports/
mv BUG*.md BUG*.txt archive/reports/

# Move scripts
mv deploy*.sh archive/scripts/
mv monitor*.sh archive/scripts/
mv terminate*.sh archive/scripts/
mv test_*.sh archive/scripts/

# Move temp files
mv '=2.11.0' '=2.12.0' archive/temp/
mv *.py archive/temp/
mv inspect_safetensors.rs check_validation_data.rs archive/temp/

# Clean JSON
mv example_backtest_results.json archive/temp/

# Add to .gitignore
echo "archive/" >> .gitignore
```

### Files to KEEP in root:
- CLAUDE.md (project instructions)
- Cargo.toml, Cargo.lock
- clippy.toml
- justfile, Makefile
- .env.example, .gitignore
- .gitlab-ci.yml, docker-compose.yml

---

## Phase 2: Target Directory Cleanup

**Current Size**: 51GB

```bash
# Full clean (recommended for fresh builds)
cargo clean

# Selective clean (preserves dependencies)
cargo clean --release
cargo clean --profile test

# After cleanup, run incremental build
cargo build --workspace
```

**Expected After Cleanup**: 0GB → ~5GB after rebuild

---

## Phase 3: Unused Dependencies

### High Confidence Removals (22 dependencies)

**backtesting/Cargo.toml** (6 deps):
- thiserror (using anyhow instead)
- crossbeam (unused concurrency)
- ndarray (using nalgebra)
- bincode (unused serialization)
- prometheus (metrics not implemented)
- fastrand (using rand)

**storage/Cargo.toml** (6 deps):
- tokio_util (unused codec)
- rustc_hash (unused hasher)
- indexmap (unused ordered map)
- fs2 (unused file locks)
- dashmap (unused concurrent map)
- backon (unused retry)

**ml/Cargo.toml** (2 deps):
- arrayfire (using candle)
- argmin_math (using candle optimizers)

**data/Cargo.toml** (3 deps):
- hex (unused encoding)
- md5 (unused hashing)
- nonzero (unused type)

**Test Dependencies** (5 deps across 7 crates):
- tokio_test
- rstest
- test_case

### Verification Required (5 dependencies)

**market-data/Cargo.toml**:
- tokio, anyhow, tracing (may be transitive)

**data/Cargo.toml**:
- webpki_roots, xml_rs (check reqwest usage)

### Validation Command

```bash
# Install cargo-udeps for accurate detection
cargo install cargo-udeps
cargo +nightly udeps --workspace --all-targets
```

---

## Phase 4: Version Conflicts

### Arrow v55/v56 Conflict

**Impact**: 24 duplicate crates compiled, +2GB disk, +15% build time

**Root Cause**:
- `parquet = "56"` pulls Arrow v56
- Some transitive dependency pulls Arrow v55

**Solution in Cargo.toml**:
```toml
[patch.crates-io]
# Force Arrow v56 throughout workspace
arrow = { version = "56" }
arrow-array = { version = "56" }
arrow-schema = { version = "56" }
```

**Verification**:
```bash
cargo tree -d | grep arrow
# Should show single arrow version after fix
```

---

## Phase 5: Code Duplication Consolidation

### Priority 1: Error Handling (1,200 LOC saved)

**Files with 95%+ identical code**:
- ml/src/error_consolidated.rs (345 lines)
- risk/src/error_consolidated.rs (473 lines)
- data/src/error_consolidated.rs (288 lines)
- tli/src/error_consolidated.rs (522 lines)

**Solution**:
Create `common/src/error/service_error_trait.rs`:
```rust
pub trait ServiceErrorExt: From<CommonError> {
    fn error_code(&self) -> &'static str;
    fn category(&self) -> ErrorCategory;
    fn severity(&self) -> Severity;
    fn retry_strategy(&self) -> RetryStrategy;
}

macro_rules! define_service_error {
    ($name:ident { $($variant:tt)* }) => {
        // Generate standard ServiceError boilerplate
    };
}
```

### Priority 2: Test Fixtures (3,500 LOC saved)

**Pattern**: `setup_test_db()`, `create_mock_order()` repeated 30+ times

**Solution**:
Create `tests/test_common/`:
```
tests/test_common/
├── fixtures/
│   ├── database.rs      # Shared DB setup
│   ├── orders.rs        # Mock orders
│   ├── market_data.rs   # Mock market data
│   └── config.rs        # Test configs
└── builders.rs          # Builder patterns
```

### Priority 3: Configuration Structs (1,800 LOC saved)

**Pattern**: `DatabaseConfig`, `TlsConfig` duplicated in 8+ crates

**Solution**:
```rust
// config/src/common_configs.rs
pub struct DatabaseConfig { ... }
pub struct TlsConfig { ... }
pub struct RedisConfig { ... }

// In other crates:
pub use config::common_configs::{DatabaseConfig, TlsConfig};
```

---

## Phase 6: ML Module Refactoring

### Large Files Requiring Split

| File | Lines | Target |
|------|-------|--------|
| trainers/dqn.rs | 4,975 | <1,000 |
| mamba/mod.rs | 3,247 | <1,000 |
| hyperopt/adapters/dqn.rs | 3,162 | <1,000 |
| trainers/tft.rs | 2,915 | <1,000 |

### Recommended Structure

```
ml/src/trainers/
├── dqn/
│   ├── mod.rs           # Public API
│   ├── core.rs          # Training loop
│   ├── checkpointing.rs # Save/load
│   ├── metrics.rs       # Training metrics
│   └── hyperopt.rs      # Hyperparameter tuning
├── ppo/
│   └── (similar structure)
└── shared/
    ├── training_loop.rs # Common loop logic
    └── checkpoint.rs    # Common checkpoint logic
```

### Preprocessing Consolidation

**Current**: 90 files with `normalize|standardize` logic

**Target**: Single `ml/src/features/preprocessing.rs`:
```rust
pub trait FeaturePreprocessor {
    fn normalize(&self, features: &[f64]) -> Result<Vec<f64>>;
    fn standardize(&self, features: &[f64]) -> Result<Vec<f64>>;
    fn clip(&self, features: &[f64], min: f64, max: f64) -> Result<Vec<f64>>;
}
```

---

## Phase 7: Test Coverage Gaps

### Critical Untested Code (RISK TO CAPITAL)

**risk/src/risk_engine.rs** (47 functions, 0 tests):
- Pre-trade risk validation
- Position limit enforcement
- Margin calculations

**risk/src/kelly_sizing.rs** (10 functions, 0 tests):
- Kelly criterion calculations
- Position sizing

**risk/src/var_calculator/** (VaR calculations untested):
- parametric.rs
- monte_carlo.rs
- expected_shortfall.rs

**trading_engine/src/trading/engine.rs** (Core execution untested):
- Order submission
- Partial fill handling
- Position updates

### Required Test Files

```
tests/risk/
├── risk_engine_tests.rs      # 20+ unit tests
├── kelly_sizing_tests.rs     # 15+ unit tests
├── var_calculator_tests.rs   # 30+ unit tests
└── circuit_breaker_tests.rs  # 10+ scenario tests

tests/trading/
├── order_execution_tests.rs  # 25+ unit tests
├── position_manager_tests.rs # 15+ unit tests
└── integration_tests.rs      # E2E scenarios
```

---

## Implementation Timeline

### Week 1: Quick Wins (Low Risk)
- [ ] Archive 506 root folder files
- [ ] Clean target directory
- [ ] Remove 22 unused dependencies
- [ ] Delete placeholder modules in ml/src/regime/

### Week 2: Build Optimization
- [ ] Fix Arrow version conflict
- [ ] Add workspace-level dependency patches
- [ ] Verify build time improvements

### Week 3-4: Test Coverage
- [ ] Add risk_engine unit tests
- [ ] Add kelly_sizing tests
- [ ] Add trading execution tests
- [ ] Add VaR calculator tests

### Week 5-6: Code Consolidation
- [ ] Consolidate error handling (4 modules)
- [ ] Create test fixtures library
- [ ] Migrate 50% of test files

### Week 7-8: ML Refactoring
- [ ] Split large trainer files
- [ ] Consolidate preprocessing logic
- [ ] Implement DeviceManager

---

## Metrics & Success Criteria

### Before Cleanup
- Root folder files: 510
- Target directory: 51GB
- Unused dependencies: 22-27
- Code duplication: 3-4%
- Risk module test coverage: 40%
- Build time (clean): 15-20 min

### After Cleanup (Target)
- Root folder files: <10
- Target directory: <5GB (after rebuild)
- Unused dependencies: 0
- Code duplication: <1%
- Risk module test coverage: >80%
- Build time (clean): 12-15 min

---

## Risk Assessment

### Safe to Execute Immediately
✅ Root folder cleanup (no code changes)
✅ Target directory clean (rebuild recovers)
✅ Delete placeholder modules (empty files)
✅ Remove test dependencies (tests only)

### Requires Verification
🟡 Remove production dependencies (run tests first)
🟡 Arrow version patch (verify all features work)
🟡 Error handling consolidation (gradual migration)

### High Risk (Defer)
🔴 Configuration struct consolidation (widespread imports)
🔴 ML trainer refactoring (active development)
🔴 Trading engine changes (business critical)

---

## Commands Summary

```bash
# Phase 1: Root cleanup
mkdir -p archive/{reports,scripts,temp}
mv AGENT*.md AGENT*.txt WAVE*.md WAVE*.txt DQN_*.md DQN_*.txt BUG*.md BUG*.txt archive/reports/
mv deploy*.sh monitor*.sh terminate*.sh test_*.sh archive/scripts/
mv '=2.11.0' '=2.12.0' *.py example_backtest_results.json archive/temp/

# Phase 2: Target cleanup
cargo clean

# Phase 3: Verify build
cargo build --workspace
cargo test --workspace

# Phase 4: Check dependencies
cargo install cargo-udeps
cargo +nightly udeps --workspace
```

---

## Appendix: File Inventory

### Files to Archive (Sample)

```
AGENT_14_BACKTESTING_INTEGRATION_INVESTIGATION.md
AGENT_15_WAVE12_IMPLEMENTATION_REPORT.md
AGENT_16_HANDOFF.txt
...
WAVE_12_CAMPAIGN_SUMMARY.md
WAVE_16C_SMOKE_TEST_REPORT.md
...
DQN_HYPEROPT_RESULTS_SUMMARY.md
DQN_VALIDATION_SYSTEM_REPORT.md
...
BUG17_P1_IMPLEMENTATION_REPORT.md
BUG24_BUG25_TDD_REPORT.md
```

### Files to Keep

```
CLAUDE.md              # Project instructions
Cargo.toml             # Workspace manifest
Cargo.lock             # Dependency lock
clippy.toml            # Linting config
justfile               # Task runner
Makefile               # Build tasks
.gitignore             # Git ignore
.gitlab-ci.yml         # CI config
docker-compose.yml     # Docker config
.env.example           # Environment template
sqlx-data.json         # SQLx offline mode
```