foxhunt/docs/plans/2026-03-01-workspace-cleanup-implementation.md

Workspace Cleanup — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Delete ~92 scattered .md files, create/update READMEs for all 36 crates/services/testing, add lib.rs doc comments. Keep docs/plans/.

Architecture: Three phases: (1) bulk delete stale docs, (2) create/update READMEs with standard template, (3) add lib.rs doc comments. No Rust code changes except //! comments. Worktree cleanup is a separate non-git operation.

Tech Stack: git rm, file creation, SQLX_OFFLINE=true cargo check --workspace

---

Task 1: Delete all scattered .md files

Delete every .md file that is NOT a root-level README.md for a crate/service/testing directory. Also delete .serena tool artifacts. Keep docs/plans/.

Step 1: Delete scattered .md in crates/

cd /home/jgrusewski/Work/foxhunt
git rm \
  crates/data/tests/EDGE_CASE_FINDINGS.md \
  crates/database/QUICK_START.md \
  crates/ml/PERFORMANCE_TRACKING.md \
  crates/ml/MAMBA2_CONFIGURATION_FIX.md \
  crates/ml/PERFORMANCE_QUICK_START.md \
  crates/ml/MAMBA2_DIMENSION_ANALYSIS.md \
  crates/ml/QUICKSTART_GPU_BENCHMARK.md \
  crates/ml/docs/codebase-cleanup/DQN_PARAMETER_CONSISTENCY_AUDIT.md \
  crates/ml/docs/dqn_hyperopt_parameter_audit.md \
  crates/ml/docs/GPU_BENCHMARK_GUIDE.md \
  crates/ml/docs/dqn_hyperparameters_analysis.md \
  crates/ml/docs/QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md \
  crates/ml/docs/QAT_GUIDE.md \
  crates/ml/hyperparams/README.md \
  crates/ml/profiling_reports/TFT_INT8_MEMORY_PROFILING_GUIDE.md \
  crates/ml/profiling_reports/MEMORY_COMPARISON_CHART.md \
  crates/ml/src/benchmark/TFT_BENCHMARK_README.md \
  crates/ml/src/checkpoint/README.md \
  crates/ml/tests/TFT_TEST_REPORT.md \
  crates/ml/tests/MAMBA_TEST_COVERAGE.md \
  crates/ml/trained_models/production/tft_real_data/TRAINING_REPORT.md \
  crates/risk/docs/TEST_COVERAGE_REPORT.md \
  crates/storage/tests/S3_TEST_COVERAGE.md \
  crates/trading_engine/docs/audit_trail_persistence_usage.md \
  crates/trading_engine/docs/TEST_COVERAGE_REPORT.md

Step 2: Delete .serena tool artifacts

git rm -r crates/trading_engine/src/types/.serena/

Step 3: Delete scattered .md in services/

git rm \
  services/api_gateway/tests/INTEGRATION_TEST_REPORT.md \
  services/api_gateway/tests/RATE_LIMITER_TEST_REPORT.md \
  services/api_gateway/src/metrics/README.md \
  services/api_gateway/tests/README.md \
  services/api_gateway/benches/README.md \
  services/api_gateway/METRICS_DEPLOYMENT.md \
  services/api_gateway/METRICS_ARCHITECTURE.md \
  services/api_gateway/REVOCATION_CACHE_USAGE.md \
  services/api_gateway/BENCHMARKS.md \
  services/api_gateway/ML_TRAINING_PROXY_INTEGRATION.md \
  services/api_gateway/RATE_LIMITER_IMPLEMENTATION.md \
  services/backtesting_service/tests/SERVICE_TESTS_REPORT.md \
  services/backtesting_service/tests/COVERAGE_MAPPING.md \
  services/backtesting_service/tests/fixtures/README.md \
  services/backtesting_service/tests/fixtures/PERFORMANCE.md \
  services/backtesting_service/tests/fixtures/QUICKSTART.md \
  services/backtesting_service/tests/fixtures/ARCHITECTURE.md \
  services/backtesting_service/docs/DBN_LOADING_PERFORMANCE_REPORT.md \
  services/backtesting_service/DBN_REPOSITORY_USAGE.md \
  services/backtesting_service/ADVANCED_QUERIES_QUICKREF.md \
  services/broker_gateway_service/DEPLOYMENT.md \
  services/broker_gateway_service/MONITORING_IMPLEMENTATION.md \
  services/broker_gateway_service/METRICS_QUICK_REF.md \
  services/broker_gateway_service/docs/API.md \
  services/broker_gateway_service/docs/TROUBLESHOOTING.md \
  services/broker_gateway_service/docs/DEPLOYMENT.md \
  services/ml_training_service/HYPERPARAMETER_TUNING.md \
  services/ml_training_service/TRIAL_EXECUTOR_USAGE.md \
  services/ml_training_service/tests/fixtures/README.md \
  services/trading_service/src/streaming/README.md \
  services/trading_service/docs/ml_integration_design.md \
  services/trading_service/tests/common/README.md \
  services/trading_agent_service/README_TLS.md

Step 4: Delete scattered .md in bin/ and testing/

git rm \
  bin/fxt/CONFIG_FILE_SUPPORT.md \
  bin/fxt/TUNE_COMMAND_README.md \
  bin/fxt/docs/AUTHENTICATION.md \
  bin/fxt/docs/ML_TRADING_COMMANDS.md \
  bin/fxt/docs/USAGE.md \
  bin/fxt/tests/INTEGRATION_TEST_GUIDE.md \
  bin/fxt/tests/TEST_EXECUTION_README.md \
  testing/api-gateway-load/CHEAT_SHEET.md \
  testing/api-gateway-load/QUICK_START.md \
  testing/e2e/E2E_TEST_GUIDE.md \
  testing/integration/e2e_helpers/INFRASTRUCTURE_VALIDATION.md \
  testing/integration/e2e_helpers/USAGE_EXAMPLES.md \
  testing/integration/e2e_helpers/QUICKSTART.md \
  testing/integration/e2e_helpers/VALIDATION_REPORT.md \
  testing/integration/e2e_helpers/QUICK_START_INFRASTRUCTURE.md \
  testing/integration/fixtures/README.md \
  testing/service-integration/README_DBN_INTEGRATION.md

Step 5: Keep docs/plans/

The 137 plan docs in docs/plans/ are kept as historical reference. No action needed.

Step 6: Remove empty directories left behind

# git auto-removes empty dirs on `git rm`, but check for any remaining
find crates/ services/ bin/ testing/ -type d -empty -delete 2>/dev/null
# Also remove docs/plans/ dir itself if empty
rmdir docs/plans 2>/dev/null || true

Step 7: Commit

git add -A
git commit -m "cleanup: delete 92 scattered .md files and .serena artifacts"

---

Task 2: Remove stale git worktrees

Not a git-tracked change, but frees ~13GB disk.

Step 1: List worktrees

git worktree list

Step 2: Remove all worktrees in .claude/worktrees/

git worktree remove --force .claude/worktrees/real-metrics-overhaul
git worktree remove --force .claude/worktrees/autonomous-agents
git worktree remove --force .claude/worktrees/numeric-type-unification
git worktree remove --force .claude/worktrees/training-deploy

Step 3: Prune stale worktree references

git worktree prune

Step 4: Verify

git worktree list
# Should show only main worktree

---

Task 3: Create/update crate READMEs (17 crates)

For each crate, create or update README.md using this template:

# <crate-name>

<Cargo.toml description>

## Key Types

- `TypeA` — brief description
- `TypeB` — brief description

## Usage

Brief usage notes or example.

Step 1: Create 8 missing crate READMEs

Create README.md in each of these directories with content based on their Cargo.toml description and key types:

crates/config/README.md:

# config

Configuration management for Foxhunt HFT trading system.

## Key Types

- `FoxhuntConfig` — top-level configuration struct
- `MlConfig` — ML model and training configuration
- `RiskConfig` — risk management parameters
- `DatabaseConfig` — PostgreSQL connection settings
- `StorageConfig` — S3/object storage configuration

## Usage

```rust
use config::FoxhuntConfig;
let config = FoxhuntConfig::load("config/foxhunt.toml")?;

**crates/ctrader-openapi/README.md:**
```markdown
# ctrader-openapi

cTrader Open API client — Protobuf over TCP+TLS for order routing, account queries, and market data.

## Key Types

- `CtraderConnection` — TLS-wrapped TCP connection to cTrader
- `CtraderCodec` — Protobuf message framing
- `RateLimiter` — request rate limiting
- `AuthManager` — OAuth2 authentication flow

## Usage

Broker gateway integration for order routing via cTrader Open API protocol.

crates/market-data/README.md:

# market-data

Market data repository for the Foxhunt HFT trading system.

## Key Types

- `MarketDataRepository` — PostgreSQL-backed market data storage
- Price, orderbook, indicator, and model data access methods

## Usage

```rust
use market_data::MarketDataRepository;
let repo = MarketDataRepository::new(pool).await?;

**crates/ml-data/README.md:**
```markdown
# ml-data

Production-ready ML data management for HFT trading systems.

## Key Types

- `MlDataConfig` — ML data repository configuration
- `FeaturesRepository` — feature storage and retrieval
- `ModelsRepository` — model metadata and versioning
- `PerformanceRepository` — model performance tracking
- `TrainingRepository` — training run management

## Usage

```rust
use ml_data::{MlDataConfig, FeaturesRepository};

**crates/model_loader/README.md:**
```markdown
# model_loader

ML model loading and caching infrastructure.

## Key Types

- `AsyncModelLoader` — async model loading with LRU cache
- `LoadedModelInfo` — loaded model metadata and weights
- `ModelType` — re-exported from common (10 model variants)

## Usage

```rust
use model_loader::AsyncModelLoader;
let loader = AsyncModelLoader::new(cache_size);
let model = loader.load("path/to/checkpoint").await?;

**crates/risk-data/README.md:**
```markdown
# risk-data

Risk data repository for high-frequency trading risk management.

## Key Types

- `RiskDataConfig` — repository configuration
- `VarRepository` — Value-at-Risk data storage
- `LimitsRepository` — position and risk limit management
- `ComplianceRepository` — compliance record storage

## Usage

```rust
use risk_data::{RiskDataConfig, VarRepository};

**crates/trading-data/README.md:**
```markdown
# trading-data

Trading data repository layer for the high-frequency trading system.

## Key Types

- `PostgresOrderRepository` — PostgreSQL-backed order storage
- `OrderRepository` trait — order CRUD operations
- Position and execution tracking

## Usage

```rust
use trading_data::PostgresOrderRepository;
let repo = PostgresOrderRepository::new(pool).await?;

**crates/training_uploader/README.md:**
```markdown
# training_uploader

K8s sidecar that uploads training artifacts to S3 and reports completion via gRPC.

## Usage

Deployed as a sidecar container alongside training jobs. Watches for training
artifacts (checkpoints, metrics) and uploads them to the configured S3 bucket.

## Configuration

- `S3_BUCKET` — target bucket for artifacts
- `TRAINING_SERVICE_URL` — gRPC endpoint for completion reporting

Step 2: Update 9 existing crate READMEs

Read each existing README and update to the standard template format. Strip stale content, keep accurate descriptions. Existing READMEs to update:

• crates/backtesting/README.md
• crates/common/README.md
• crates/data/README.md
• crates/database/README.md
• crates/ml/README.md
• crates/risk/README.md
• crates/storage/README.md
• crates/trading_engine/README.md
• crates/web-gateway/README.md

For each: read current content, trim to standard template (title, purpose, key types, usage), remove bloated sections.

Step 3: Verify no compilation impact

Run: SQLX_OFFLINE=true cargo check --workspace Expected: Clean (README changes can't break compilation)

Step 4: Commit

git add crates/*/README.md
git commit -m "docs: create/update README.md for all 17 crates"

---

Task 4: Create/update service, binary, and testing READMEs

Step 1: Create 3 missing service READMEs

services/api_gateway/README.md:

# api_gateway

API Gateway service with 6-layer authentication, RBAC, rate limiting, and gRPC proxy routing.

## Key Types

- `ApiGatewayService` — main service implementation
- `RoleBasedAccessControl` — RBAC authorization
- `RateLimiter` — 3-tier rate limiting (auth/trading/compute)
- `CircuitBreaker` — upstream service protection

## Configuration

- `JWT_SECRET` — JWT signing secret (min 32 chars)
- `UPSTREAM_TRADING_URL` — trading service gRPC endpoint
- `UPSTREAM_ML_URL` — ML service gRPC endpoint

## Features

- `mfa` (default) — multi-factor authentication support
- Build without: `cargo check -p api_gateway --no-default-features --features minimal`

services/data_acquisition_service/README.md:

# data_acquisition_service

Automated Databento data downloading, validation, and S3 upload for HFT trading.

## Key Types

- `DataAcquisitionServiceImpl` — main gRPC service
- `Downloader` — Databento API client
- `Uploader` — S3 artifact upload
- `Validator` — data quality checks

## Configuration

- `DATABENTO_API_KEY` — Databento API credentials
- `S3_BUCKET` — target storage bucket

services/trading_agent_service/README.md:

# trading_agent_service

Portfolio management with autonomous universe selection, asset allocation, and order generation.

## Key Types

- `TradingAgentServiceImpl` — main gRPC service
- `AutonomousUniverseManager` — dynamic universe scaling
- `AssetSelector` — instrument selection
- `Allocator` — position sizing and allocation
- `OrderGenerator` — order creation from signals

## Configuration

- `ML_SERVICE_URL` — ensemble ML service endpoint
- `TRADING_SERVICE_URL` — order execution endpoint

Step 2: Update 4 existing service READMEs

Read and update to standard template:

• services/backtesting_service/README.md
• services/broker_gateway_service/README.md
• services/ml_training_service/README.md
• services/trading_service/README.md

For services/tests/ — this is a disabled test service, skip README creation.

Step 3: Create bin/fxt/README.md

# fxt

Foxhunt CLI — command-line interface for the Foxhunt HFT trading system.

## Commands

- `fxt auth` — authentication and token management
- `fxt trade` — order submission and management
- `fxt train` — model training job management
- `fxt tune` — hyperparameter optimization
- `fxt agent` — trading agent control
- `fxt model` — model listing and inspection
- `fxt backtest-ml` — ML backtesting workflows

## Configuration

- `FXT_CONFIG` — config file path (default: `~/.config/fxt/config.toml`)
- `TRADING_SERVICE_URL` — gRPC endpoint for trading service

Step 4: Update 10 testing crate READMEs

Read existing testing READMEs and update to standard template. For those with existing README.md, trim to standard format. Testing crates:

• testing/api-gateway-load/README.md (exists)
• testing/e2e/README.md (exists)
• testing/harness/ (no README — skip, not a published crate)
• testing/integration/README.md (exists)
• testing/load/README.md (exists)
• testing/service-integration/ (no README — create minimal)
• testing/service-load/README.md (exists)
• testing/stress/ (no README — skip)
• testing/test-common/README.md (exists)
• testing/vault-integration/ (no README — skip)

Also delete subdirectory READMEs that survived (chaos, smoke_tests, e2e_helpers — only keep root README per crate):

• testing/integration/chaos/README.md
• testing/integration/e2e_helpers/README.md
• testing/integration/smoke_tests/README.md
• testing/e2e/integration/README.md

Step 5: Commit

git add services/*/README.md bin/fxt/README.md testing/*/README.md
git rm testing/integration/chaos/README.md testing/integration/e2e_helpers/README.md testing/integration/smoke_tests/README.md testing/e2e/integration/README.md 2>/dev/null || true
git commit -m "docs: create/update README.md for all services, CLI, and testing crates"

---

Task 5: Add lib.rs doc comments to 7 crates

Add //! module-level doc comments to the top of src/lib.rs for crates that are missing them.

Step 1: Add doc comments

For each crate, add a single //! line after any #![...] attributes but before the first use/mod statement.

crates/backtesting/src/lib.rs — add after #![allow(missing_docs)]:

//! Backtesting engine for strategy evaluation with walk-forward analysis.

crates/ctrader-openapi/src/lib.rs — add after #![deny(...)]:

//! cTrader Open API client — Protobuf over TCP+TLS for order routing and market data.

crates/data/src/lib.rs — add after #![allow(unused_extern_crates)]:

//! Data processing and validation for the Foxhunt HFT trading system.

crates/ml/src/lib.rs — add after #![deny(...)]:

//! Machine learning models and ensemble inference for HFT signal generation.

crates/risk/src/lib.rs — add after the comment block:

//! Risk management — VaR, Kelly criterion, circuit breakers, and kill switches.

crates/trading_engine/src/lib.rs — add after #![deny(...)]:

//! Trading engine — FIX 4.4, IB TWS, order management, and execution.

crates/training_uploader/src/main.rs — add after #![allow(...)]:

//! K8s sidecar that uploads training artifacts to S3 and reports completion via gRPC.

Step 2: Verify compilation

Run: SQLX_OFFLINE=true cargo check --workspace Expected: Clean (doc comments can't break compilation)

Step 3: Commit

git add crates/backtesting/src/lib.rs crates/ctrader-openapi/src/lib.rs crates/data/src/lib.rs crates/ml/src/lib.rs crates/risk/src/lib.rs crates/trading_engine/src/lib.rs crates/training_uploader/src/main.rs
git commit -m "docs: add lib.rs doc comments to 7 crates"

---

Task 6: Final verification

Step 1: Workspace build

Run: SQLX_OFFLINE=true cargo check --workspace Expected: zero errors

Step 2: Clippy on workspace

Run: SQLX_OFFLINE=true cargo clippy --workspace --lib -- -D warnings Expected: zero warnings

Step 3: Verify no stray .md files remain

# List all non-README .md files in code directories (should be empty or near-empty)
find crates/ services/ bin/ testing/ -name "*.md" ! -name "README.md" | sort

Expected: empty output (all scattered docs deleted)

Step 4: Verify every crate/service has README

for dir in crates/*/; do
  [ -f "$dir/README.md" ] && echo "OK $dir" || echo "MISSING $dir"
done
for dir in services/*/; do
  [ -f "$dir/README.md" ] && echo "OK $dir" || echo "MISSING $dir"
done
[ -f bin/fxt/README.md ] && echo "OK bin/fxt" || echo "MISSING bin/fxt"

Expected: all OK (services/tests/ may be MISSING — that's acceptable, it's a disabled service)

Step 5: Line count verification

git diff --stat HEAD~4
# Should show significant net negative lines from doc deletion