foxhunt/docs/codebase-cleanup/CLEANUP_ANALYSIS_REPORT.md

Foxhunt Codebase Cleanup Analysis Report

Date: 2025-11-27 Analyzer: Claude Code (SPARC Methodology) Project Size: 58GB total, 527,700 LOC Rust code

---

Executive Summary

The Foxhunt HFT trading system is a substantial Rust codebase with integrated ML/RL components. Analysis reveals excellent code quality in core modules but significant technical debt from rapid AI-assisted development, primarily manifesting as:

1. 506 working files in root folder (should be 0)
2. 51GB target directory (needs cleanup)
3. 3-4% code duplication (~15,000-20,000 lines)
4. 22-27 unused dependencies
5. Arrow v55/v56 version conflict causing 24 duplicate crates
6. 40% test coverage in critical risk management code

Quick Impact Summary

Cleanup Action	Disk Savings	Build Impact	Risk Level
Root folder cleanup	~50MB	None	✅ LOW
Target directory clean	~51GB	Rebuild needed	✅ LOW
Remove unused deps	~100MB	-5% build time	✅ LOW
Fix Arrow conflict	~2GB	-15% build time	🟡 MEDIUM
Consolidate duplication	~8,400 LOC	None	🟡 MEDIUM

---

Phase 1: Root Folder Cleanup (CRITICAL)

Current State

Root folder file count:
- Markdown/Text files: 474
- Shell scripts: 27
- Python files: 4
- Rust files: 1
- JSON files: 2 (example_backtest_results.json, sqlx-data.json)
- Temp files: 2 (=2.11.0, =2.12.0)
TOTAL: 510 files that should NOT be in root

Files Categories

Agent Reports (300+ files):

• AGENT__.md/txt - Development session reports
• WAVE*_*.md/txt - Development wave summaries
• DQN_*.md/txt - DQN development artifacts
• BUG*_*.md/txt - Bug investigation reports

Deployment Scripts (27 files):

• deploy_*.sh - Kubernetes/RunPod deployment scripts
• monitor_*.sh - Pod monitoring scripts
• terminate_*.sh - Cleanup scripts
• test_*.sh - Manual test scripts

Temporary/Debug Files:

• =2.11.0, =2.12.0 - Leftover from failed commands
• *.py - One-off validation scripts
• *.rs (standalone) - Debug inspection scripts

Recommended Actions

# Create archive directory
mkdir -p archive/reports archive/scripts archive/temp

# Move agent reports
mv AGENT*.md AGENT*.txt archive/reports/
mv WAVE*.md WAVE*.txt archive/reports/
mv DQN_*.md DQN_*.txt archive/reports/
mv BUG*.md BUG*.txt archive/reports/

# Move scripts
mv deploy*.sh archive/scripts/
mv monitor*.sh archive/scripts/
mv terminate*.sh archive/scripts/
mv test_*.sh archive/scripts/

# Move temp files
mv '=2.11.0' '=2.12.0' archive/temp/
mv *.py archive/temp/
mv inspect_safetensors.rs check_validation_data.rs archive/temp/

# Clean JSON
mv example_backtest_results.json archive/temp/

# Add to .gitignore
echo "archive/" >> .gitignore

Files to KEEP in root:

• CLAUDE.md (project instructions)
• Cargo.toml, Cargo.lock
• clippy.toml
• justfile, Makefile
• .env.example, .gitignore
• .gitlab-ci.yml, docker-compose.yml

---

Phase 2: Target Directory Cleanup

Current Size: 51GB

# Full clean (recommended for fresh builds)
cargo clean

# Selective clean (preserves dependencies)
cargo clean --release
cargo clean --profile test

# After cleanup, run incremental build
cargo build --workspace

Expected After Cleanup: 0GB → ~5GB after rebuild

---

Phase 3: Unused Dependencies

High Confidence Removals (22 dependencies)

backtesting/Cargo.toml (6 deps):

• thiserror (using anyhow instead)
• crossbeam (unused concurrency)
• ndarray (using nalgebra)
• bincode (unused serialization)
• prometheus (metrics not implemented)
• fastrand (using rand)

storage/Cargo.toml (6 deps):

• tokio_util (unused codec)
• rustc_hash (unused hasher)
• indexmap (unused ordered map)
• fs2 (unused file locks)
• dashmap (unused concurrent map)
• backon (unused retry)

ml/Cargo.toml (2 deps):

• arrayfire (using candle)
• argmin_math (using candle optimizers)

data/Cargo.toml (3 deps):

• hex (unused encoding)
• md5 (unused hashing)
• nonzero (unused type)

Test Dependencies (5 deps across 7 crates):

• tokio_test
• rstest
• test_case

Verification Required (5 dependencies)

market-data/Cargo.toml:

• tokio, anyhow, tracing (may be transitive)

data/Cargo.toml:

• webpki_roots, xml_rs (check reqwest usage)

Validation Command

# Install cargo-udeps for accurate detection
cargo install cargo-udeps
cargo +nightly udeps --workspace --all-targets

---

Phase 4: Version Conflicts

Arrow v55/v56 Conflict

Impact: 24 duplicate crates compiled, +2GB disk, +15% build time

Root Cause:

• parquet = "56" pulls Arrow v56
• Some transitive dependency pulls Arrow v55

Solution in Cargo.toml:

[patch.crates-io]
# Force Arrow v56 throughout workspace
arrow = { version = "56" }
arrow-array = { version = "56" }
arrow-schema = { version = "56" }

Verification:

cargo tree -d | grep arrow
# Should show single arrow version after fix

---

Phase 5: Code Duplication Consolidation

Priority 1: Error Handling (1,200 LOC saved)

Files with 95%+ identical code:

• ml/src/error_consolidated.rs (345 lines)
• risk/src/error_consolidated.rs (473 lines)
• data/src/error_consolidated.rs (288 lines)
• tli/src/error_consolidated.rs (522 lines)

Solution: Create common/src/error/service_error_trait.rs:

pub trait ServiceErrorExt: From<CommonError> {
    fn error_code(&self) -> &'static str;
    fn category(&self) -> ErrorCategory;
    fn severity(&self) -> Severity;
    fn retry_strategy(&self) -> RetryStrategy;
}

macro_rules! define_service_error {
    ($name:ident { $($variant:tt)* }) => {
        // Generate standard ServiceError boilerplate
    };
}

Priority 2: Test Fixtures (3,500 LOC saved)

Pattern: setup_test_db(), create_mock_order() repeated 30+ times

Solution: Create tests/test_common/:

tests/test_common/
├── fixtures/
│   ├── database.rs      # Shared DB setup
│   ├── orders.rs        # Mock orders
│   ├── market_data.rs   # Mock market data
│   └── config.rs        # Test configs
└── builders.rs          # Builder patterns

Priority 3: Configuration Structs (1,800 LOC saved)

Pattern: DatabaseConfig, TlsConfig duplicated in 8+ crates

Solution:

// config/src/common_configs.rs
pub struct DatabaseConfig { ... }
pub struct TlsConfig { ... }
pub struct RedisConfig { ... }

// In other crates:
pub use config::common_configs::{DatabaseConfig, TlsConfig};

---

Phase 6: ML Module Refactoring

Large Files Requiring Split

File	Lines	Target
trainers/dqn.rs	4,975	<1,000
mamba/mod.rs	3,247	<1,000
hyperopt/adapters/dqn.rs	3,162	<1,000
trainers/tft.rs	2,915	<1,000

Recommended Structure

ml/src/trainers/
├── dqn/
│   ├── mod.rs           # Public API
│   ├── core.rs          # Training loop
│   ├── checkpointing.rs # Save/load
│   ├── metrics.rs       # Training metrics
│   └── hyperopt.rs      # Hyperparameter tuning
├── ppo/
│   └── (similar structure)
└── shared/
    ├── training_loop.rs # Common loop logic
    └── checkpoint.rs    # Common checkpoint logic

Preprocessing Consolidation

Current: 90 files with normalize|standardize logic

Target: Single ml/src/features/preprocessing.rs:

pub trait FeaturePreprocessor {
    fn normalize(&self, features: &[f64]) -> Result<Vec<f64>>;
    fn standardize(&self, features: &[f64]) -> Result<Vec<f64>>;
    fn clip(&self, features: &[f64], min: f64, max: f64) -> Result<Vec<f64>>;
}

---

Phase 7: Test Coverage Gaps

Critical Untested Code (RISK TO CAPITAL)

risk/src/risk_engine.rs (47 functions, 0 tests):

• Pre-trade risk validation
• Position limit enforcement
• Margin calculations

risk/src/kelly_sizing.rs (10 functions, 0 tests):

• Kelly criterion calculations
• Position sizing

risk/src/var_calculator/ (VaR calculations untested):

• parametric.rs
• monte_carlo.rs
• expected_shortfall.rs

trading_engine/src/trading/engine.rs (Core execution untested):

• Order submission
• Partial fill handling
• Position updates

Required Test Files

tests/risk/
├── risk_engine_tests.rs      # 20+ unit tests
├── kelly_sizing_tests.rs     # 15+ unit tests
├── var_calculator_tests.rs   # 30+ unit tests
└── circuit_breaker_tests.rs  # 10+ scenario tests

tests/trading/
├── order_execution_tests.rs  # 25+ unit tests
├── position_manager_tests.rs # 15+ unit tests
└── integration_tests.rs      # E2E scenarios

---

Implementation Timeline

Week 1: Quick Wins (Low Risk)

• Archive 506 root folder files
• Clean target directory
• Remove 22 unused dependencies
• Delete placeholder modules in ml/src/regime/

Week 2: Build Optimization

• Fix Arrow version conflict
• Add workspace-level dependency patches
• Verify build time improvements

Week 3-4: Test Coverage

• Add risk_engine unit tests
• Add kelly_sizing tests
• Add trading execution tests
• Add VaR calculator tests

Week 5-6: Code Consolidation

• Consolidate error handling (4 modules)
• Create test fixtures library
• Migrate 50% of test files

Week 7-8: ML Refactoring

• Split large trainer files
• Consolidate preprocessing logic
• Implement DeviceManager

---

Metrics & Success Criteria

Before Cleanup

• Root folder files: 510
• Target directory: 51GB
• Unused dependencies: 22-27
• Code duplication: 3-4%
• Risk module test coverage: 40%
• Build time (clean): 15-20 min

After Cleanup (Target)

• Root folder files: <10
• Target directory: <5GB (after rebuild)
• Unused dependencies: 0
• Code duplication: <1%
• Risk module test coverage: >80%
• Build time (clean): 12-15 min

---

Risk Assessment

Safe to Execute Immediately

✅ Root folder cleanup (no code changes) ✅ Target directory clean (rebuild recovers) ✅ Delete placeholder modules (empty files) ✅ Remove test dependencies (tests only)

Requires Verification

🟡 Remove production dependencies (run tests first) 🟡 Arrow version patch (verify all features work) 🟡 Error handling consolidation (gradual migration)

High Risk (Defer)

🔴 Configuration struct consolidation (widespread imports) 🔴 ML trainer refactoring (active development) 🔴 Trading engine changes (business critical)

---

Commands Summary

# Phase 1: Root cleanup
mkdir -p archive/{reports,scripts,temp}
mv AGENT*.md AGENT*.txt WAVE*.md WAVE*.txt DQN_*.md DQN_*.txt BUG*.md BUG*.txt archive/reports/
mv deploy*.sh monitor*.sh terminate*.sh test_*.sh archive/scripts/
mv '=2.11.0' '=2.12.0' *.py example_backtest_results.json archive/temp/

# Phase 2: Target cleanup
cargo clean

# Phase 3: Verify build
cargo build --workspace
cargo test --workspace

# Phase 4: Check dependencies
cargo install cargo-udeps
cargo +nightly udeps --workspace

---

Appendix: File Inventory

Files to Archive (Sample)

AGENT_14_BACKTESTING_INTEGRATION_INVESTIGATION.md
AGENT_15_WAVE12_IMPLEMENTATION_REPORT.md
AGENT_16_HANDOFF.txt
...
WAVE_12_CAMPAIGN_SUMMARY.md
WAVE_16C_SMOKE_TEST_REPORT.md
...
DQN_HYPEROPT_RESULTS_SUMMARY.md
DQN_VALIDATION_SYSTEM_REPORT.md
...
BUG17_P1_IMPLEMENTATION_REPORT.md
BUG24_BUG25_TDD_REPORT.md

Files to Keep

CLAUDE.md              # Project instructions
Cargo.toml             # Workspace manifest
Cargo.lock             # Dependency lock
clippy.toml            # Linting config
justfile               # Task runner
Makefile               # Build tasks
.gitignore             # Git ignore
.gitlab-ci.yml         # CI config
docker-compose.yml     # Docker config
.env.example           # Environment template
sqlx-data.json         # SQLx offline mode