foxhunt/docs/plans/2026-03-01-workspace-cleanup-implementation.md

# Workspace Cleanup — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Delete ~92 scattered .md files, create/update READMEs for all 36 crates/services/testing, add lib.rs doc comments. Keep docs/plans/.

**Architecture:** Three phases: (1) bulk delete stale docs, (2) create/update READMEs with standard template, (3) add lib.rs doc comments. No Rust code changes except `//!` comments. Worktree cleanup is a separate non-git operation.

**Tech Stack:** git rm, file creation, `SQLX_OFFLINE=true cargo check --workspace`

---

### Task 1: Delete all scattered .md files

Delete every .md file that is NOT a root-level README.md for a crate/service/testing directory. Also delete `.serena` tool artifacts. Keep `docs/plans/`.

**Step 1: Delete scattered .md in crates/**

```bash
cd /home/jgrusewski/Work/foxhunt
git rm \
  crates/data/tests/EDGE_CASE_FINDINGS.md \
  crates/database/QUICK_START.md \
  crates/ml/PERFORMANCE_TRACKING.md \
  crates/ml/MAMBA2_CONFIGURATION_FIX.md \
  crates/ml/PERFORMANCE_QUICK_START.md \
  crates/ml/MAMBA2_DIMENSION_ANALYSIS.md \
  crates/ml/QUICKSTART_GPU_BENCHMARK.md \
  crates/ml/docs/codebase-cleanup/DQN_PARAMETER_CONSISTENCY_AUDIT.md \
  crates/ml/docs/dqn_hyperopt_parameter_audit.md \
  crates/ml/docs/GPU_BENCHMARK_GUIDE.md \
  crates/ml/docs/dqn_hyperparameters_analysis.md \
  crates/ml/docs/QAT_GRADIENT_CHECKPOINTING_WORKAROUND.md \
  crates/ml/docs/QAT_GUIDE.md \
  crates/ml/hyperparams/README.md \
  crates/ml/profiling_reports/TFT_INT8_MEMORY_PROFILING_GUIDE.md \
  crates/ml/profiling_reports/MEMORY_COMPARISON_CHART.md \
  crates/ml/src/benchmark/TFT_BENCHMARK_README.md \
  crates/ml/src/checkpoint/README.md \
  crates/ml/tests/TFT_TEST_REPORT.md \
  crates/ml/tests/MAMBA_TEST_COVERAGE.md \
  crates/ml/trained_models/production/tft_real_data/TRAINING_REPORT.md \
  crates/risk/docs/TEST_COVERAGE_REPORT.md \
  crates/storage/tests/S3_TEST_COVERAGE.md \
  crates/trading_engine/docs/audit_trail_persistence_usage.md \
  crates/trading_engine/docs/TEST_COVERAGE_REPORT.md
```

**Step 2: Delete .serena tool artifacts**

```bash
git rm -r crates/trading_engine/src/types/.serena/
```

**Step 3: Delete scattered .md in services/**

```bash
git rm \
  services/api_gateway/tests/INTEGRATION_TEST_REPORT.md \
  services/api_gateway/tests/RATE_LIMITER_TEST_REPORT.md \
  services/api_gateway/src/metrics/README.md \
  services/api_gateway/tests/README.md \
  services/api_gateway/benches/README.md \
  services/api_gateway/METRICS_DEPLOYMENT.md \
  services/api_gateway/METRICS_ARCHITECTURE.md \
  services/api_gateway/REVOCATION_CACHE_USAGE.md \
  services/api_gateway/BENCHMARKS.md \
  services/api_gateway/ML_TRAINING_PROXY_INTEGRATION.md \
  services/api_gateway/RATE_LIMITER_IMPLEMENTATION.md \
  services/backtesting_service/tests/SERVICE_TESTS_REPORT.md \
  services/backtesting_service/tests/COVERAGE_MAPPING.md \
  services/backtesting_service/tests/fixtures/README.md \
  services/backtesting_service/tests/fixtures/PERFORMANCE.md \
  services/backtesting_service/tests/fixtures/QUICKSTART.md \
  services/backtesting_service/tests/fixtures/ARCHITECTURE.md \
  services/backtesting_service/docs/DBN_LOADING_PERFORMANCE_REPORT.md \
  services/backtesting_service/DBN_REPOSITORY_USAGE.md \
  services/backtesting_service/ADVANCED_QUERIES_QUICKREF.md \
  services/broker_gateway_service/DEPLOYMENT.md \
  services/broker_gateway_service/MONITORING_IMPLEMENTATION.md \
  services/broker_gateway_service/METRICS_QUICK_REF.md \
  services/broker_gateway_service/docs/API.md \
  services/broker_gateway_service/docs/TROUBLESHOOTING.md \
  services/broker_gateway_service/docs/DEPLOYMENT.md \
  services/ml_training_service/HYPERPARAMETER_TUNING.md \
  services/ml_training_service/TRIAL_EXECUTOR_USAGE.md \
  services/ml_training_service/tests/fixtures/README.md \
  services/trading_service/src/streaming/README.md \
  services/trading_service/docs/ml_integration_design.md \
  services/trading_service/tests/common/README.md \
  services/trading_agent_service/README_TLS.md
```

**Step 4: Delete scattered .md in bin/ and testing/**

```bash
git rm \
  bin/fxt/CONFIG_FILE_SUPPORT.md \
  bin/fxt/TUNE_COMMAND_README.md \
  bin/fxt/docs/AUTHENTICATION.md \
  bin/fxt/docs/ML_TRADING_COMMANDS.md \
  bin/fxt/docs/USAGE.md \
  bin/fxt/tests/INTEGRATION_TEST_GUIDE.md \
  bin/fxt/tests/TEST_EXECUTION_README.md \
  testing/api-gateway-load/CHEAT_SHEET.md \
  testing/api-gateway-load/QUICK_START.md \
  testing/e2e/E2E_TEST_GUIDE.md \
  testing/integration/e2e_helpers/INFRASTRUCTURE_VALIDATION.md \
  testing/integration/e2e_helpers/USAGE_EXAMPLES.md \
  testing/integration/e2e_helpers/QUICKSTART.md \
  testing/integration/e2e_helpers/VALIDATION_REPORT.md \
  testing/integration/e2e_helpers/QUICK_START_INFRASTRUCTURE.md \
  testing/integration/fixtures/README.md \
  testing/service-integration/README_DBN_INTEGRATION.md
```

**Step 5: Keep docs/plans/**

The 137 plan docs in `docs/plans/` are kept as historical reference. No action needed.

**Step 6: Remove empty directories left behind**

```bash
# git auto-removes empty dirs on `git rm`, but check for any remaining
find crates/ services/ bin/ testing/ -type d -empty -delete 2>/dev/null
# Also remove docs/plans/ dir itself if empty
rmdir docs/plans 2>/dev/null || true
```

**Step 7: Commit**

```bash
git add -A
git commit -m "cleanup: delete 92 scattered .md files and .serena artifacts"
```

---

### Task 2: Remove stale git worktrees

Not a git-tracked change, but frees ~13GB disk.

**Step 1: List worktrees**

```bash
git worktree list
```

**Step 2: Remove all worktrees in .claude/worktrees/**

```bash
git worktree remove --force .claude/worktrees/real-metrics-overhaul
git worktree remove --force .claude/worktrees/autonomous-agents
git worktree remove --force .claude/worktrees/numeric-type-unification
git worktree remove --force .claude/worktrees/training-deploy
```

**Step 3: Prune stale worktree references**

```bash
git worktree prune
```

**Step 4: Verify**

```bash
git worktree list
# Should show only main worktree
```

---

### Task 3: Create/update crate READMEs (17 crates)

For each crate, create or update `README.md` using this template:

```markdown
# <crate-name>

<Cargo.toml description>

## Key Types

- `TypeA` — brief description
- `TypeB` — brief description

## Usage

Brief usage notes or example.
```

**Step 1: Create 8 missing crate READMEs**

Create `README.md` in each of these directories with content based on their Cargo.toml description and key types:

**crates/config/README.md:**
```markdown
# config

Configuration management for Foxhunt HFT trading system.

## Key Types

- `FoxhuntConfig` — top-level configuration struct
- `MlConfig` — ML model and training configuration
- `RiskConfig` — risk management parameters
- `DatabaseConfig` — PostgreSQL connection settings
- `StorageConfig` — S3/object storage configuration

## Usage

```rust
use config::FoxhuntConfig;
let config = FoxhuntConfig::load("config/foxhunt.toml")?;
```
```

**crates/ctrader-openapi/README.md:**
```markdown
# ctrader-openapi

cTrader Open API client — Protobuf over TCP+TLS for order routing, account queries, and market data.

## Key Types

- `CtraderConnection` — TLS-wrapped TCP connection to cTrader
- `CtraderCodec` — Protobuf message framing
- `RateLimiter` — request rate limiting
- `AuthManager` — OAuth2 authentication flow

## Usage

Broker gateway integration for order routing via cTrader Open API protocol.
```

**crates/market-data/README.md:**
```markdown
# market-data

Market data repository for the Foxhunt HFT trading system.

## Key Types

- `MarketDataRepository` — PostgreSQL-backed market data storage
- Price, orderbook, indicator, and model data access methods

## Usage

```rust
use market_data::MarketDataRepository;
let repo = MarketDataRepository::new(pool).await?;
```
```

**crates/ml-data/README.md:**
```markdown
# ml-data

Production-ready ML data management for HFT trading systems.

## Key Types

- `MlDataConfig` — ML data repository configuration
- `FeaturesRepository` — feature storage and retrieval
- `ModelsRepository` — model metadata and versioning
- `PerformanceRepository` — model performance tracking
- `TrainingRepository` — training run management

## Usage

```rust
use ml_data::{MlDataConfig, FeaturesRepository};
```
```

**crates/model_loader/README.md:**
```markdown
# model_loader

ML model loading and caching infrastructure.

## Key Types

- `AsyncModelLoader` — async model loading with LRU cache
- `LoadedModelInfo` — loaded model metadata and weights
- `ModelType` — re-exported from common (10 model variants)

## Usage

```rust
use model_loader::AsyncModelLoader;
let loader = AsyncModelLoader::new(cache_size);
let model = loader.load("path/to/checkpoint").await?;
```
```

**crates/risk-data/README.md:**
```markdown
# risk-data

Risk data repository for high-frequency trading risk management.

## Key Types

- `RiskDataConfig` — repository configuration
- `VarRepository` — Value-at-Risk data storage
- `LimitsRepository` — position and risk limit management
- `ComplianceRepository` — compliance record storage

## Usage

```rust
use risk_data::{RiskDataConfig, VarRepository};
```
```

**crates/trading-data/README.md:**
```markdown
# trading-data

Trading data repository layer for the high-frequency trading system.

## Key Types

- `PostgresOrderRepository` — PostgreSQL-backed order storage
- `OrderRepository` trait — order CRUD operations
- Position and execution tracking

## Usage

```rust
use trading_data::PostgresOrderRepository;
let repo = PostgresOrderRepository::new(pool).await?;
```
```

**crates/training_uploader/README.md:**
```markdown
# training_uploader

K8s sidecar that uploads training artifacts to S3 and reports completion via gRPC.

## Usage

Deployed as a sidecar container alongside training jobs. Watches for training
artifacts (checkpoints, metrics) and uploads them to the configured S3 bucket.

## Configuration

- `S3_BUCKET` — target bucket for artifacts
- `TRAINING_SERVICE_URL` — gRPC endpoint for completion reporting
```

**Step 2: Update 9 existing crate READMEs**

Read each existing README and update to the standard template format. Strip stale content, keep accurate descriptions. Existing READMEs to update:
- `crates/backtesting/README.md`
- `crates/common/README.md`
- `crates/data/README.md`
- `crates/database/README.md`
- `crates/ml/README.md`
- `crates/risk/README.md`
- `crates/storage/README.md`
- `crates/trading_engine/README.md`
- `crates/web-gateway/README.md`

For each: read current content, trim to standard template (title, purpose, key types, usage), remove bloated sections.

**Step 3: Verify no compilation impact**

Run: `SQLX_OFFLINE=true cargo check --workspace`
Expected: Clean (README changes can't break compilation)

**Step 4: Commit**

```bash
git add crates/*/README.md
git commit -m "docs: create/update README.md for all 17 crates"
```

---

### Task 4: Create/update service, binary, and testing READMEs

**Step 1: Create 3 missing service READMEs**

**services/api_gateway/README.md:**
```markdown
# api_gateway

API Gateway service with 6-layer authentication, RBAC, rate limiting, and gRPC proxy routing.

## Key Types

- `ApiGatewayService` — main service implementation
- `RoleBasedAccessControl` — RBAC authorization
- `RateLimiter` — 3-tier rate limiting (auth/trading/compute)
- `CircuitBreaker` — upstream service protection

## Configuration

- `JWT_SECRET` — JWT signing secret (min 32 chars)
- `UPSTREAM_TRADING_URL` — trading service gRPC endpoint
- `UPSTREAM_ML_URL` — ML service gRPC endpoint

## Features

- `mfa` (default) — multi-factor authentication support
- Build without: `cargo check -p api_gateway --no-default-features --features minimal`
```

**services/data_acquisition_service/README.md:**
```markdown
# data_acquisition_service

Automated Databento data downloading, validation, and S3 upload for HFT trading.

## Key Types

- `DataAcquisitionServiceImpl` — main gRPC service
- `Downloader` — Databento API client
- `Uploader` — S3 artifact upload
- `Validator` — data quality checks

## Configuration

- `DATABENTO_API_KEY` — Databento API credentials
- `S3_BUCKET` — target storage bucket
```

**services/trading_agent_service/README.md:**
```markdown
# trading_agent_service

Portfolio management with autonomous universe selection, asset allocation, and order generation.

## Key Types

- `TradingAgentServiceImpl` — main gRPC service
- `AutonomousUniverseManager` — dynamic universe scaling
- `AssetSelector` — instrument selection
- `Allocator` — position sizing and allocation
- `OrderGenerator` — order creation from signals

## Configuration

- `ML_SERVICE_URL` — ensemble ML service endpoint
- `TRADING_SERVICE_URL` — order execution endpoint
```

**Step 2: Update 4 existing service READMEs**

Read and update to standard template:
- `services/backtesting_service/README.md`
- `services/broker_gateway_service/README.md`
- `services/ml_training_service/README.md`
- `services/trading_service/README.md`

For `services/tests/` — this is a disabled test service, skip README creation.

**Step 3: Create bin/fxt/README.md**

```markdown
# fxt

Foxhunt CLI — command-line interface for the Foxhunt HFT trading system.

## Commands

- `fxt auth` — authentication and token management
- `fxt trade` — order submission and management
- `fxt train` — model training job management
- `fxt tune` — hyperparameter optimization
- `fxt agent` — trading agent control
- `fxt model` — model listing and inspection
- `fxt backtest-ml` — ML backtesting workflows

## Configuration

- `FXT_CONFIG` — config file path (default: `~/.config/fxt/config.toml`)
- `TRADING_SERVICE_URL` — gRPC endpoint for trading service
```

**Step 4: Update 10 testing crate READMEs**

Read existing testing READMEs and update to standard template. For those with existing README.md, trim to standard format. Testing crates:
- `testing/api-gateway-load/README.md` (exists)
- `testing/e2e/README.md` (exists)
- `testing/harness/` (no README — skip, not a published crate)
- `testing/integration/README.md` (exists)
- `testing/load/README.md` (exists)
- `testing/service-integration/` (no README — create minimal)
- `testing/service-load/README.md` (exists)
- `testing/stress/` (no README — skip)
- `testing/test-common/README.md` (exists)
- `testing/vault-integration/` (no README — skip)

Also delete subdirectory READMEs that survived (chaos, smoke_tests, e2e_helpers — only keep root README per crate):
- `testing/integration/chaos/README.md`
- `testing/integration/e2e_helpers/README.md`
- `testing/integration/smoke_tests/README.md`
- `testing/e2e/integration/README.md`

**Step 5: Commit**

```bash
git add services/*/README.md bin/fxt/README.md testing/*/README.md
git rm testing/integration/chaos/README.md testing/integration/e2e_helpers/README.md testing/integration/smoke_tests/README.md testing/e2e/integration/README.md 2>/dev/null || true
git commit -m "docs: create/update README.md for all services, CLI, and testing crates"
```

---

### Task 5: Add lib.rs doc comments to 7 crates

Add `//!` module-level doc comments to the top of `src/lib.rs` for crates that are missing them.

**Step 1: Add doc comments**

For each crate, add a single `//!` line after any `#![...]` attributes but before the first `use`/`mod` statement.

**crates/backtesting/src/lib.rs** — add after `#![allow(missing_docs)]`:
```rust
//! Backtesting engine for strategy evaluation with walk-forward analysis.
```

**crates/ctrader-openapi/src/lib.rs** — add after `#![deny(...)]`:
```rust
//! cTrader Open API client — Protobuf over TCP+TLS for order routing and market data.
```

**crates/data/src/lib.rs** — add after `#![allow(unused_extern_crates)]`:
```rust
//! Data processing and validation for the Foxhunt HFT trading system.
```

**crates/ml/src/lib.rs** — add after `#![deny(...)]`:
```rust
//! Machine learning models and ensemble inference for HFT signal generation.
```

**crates/risk/src/lib.rs** — add after the comment block:
```rust
//! Risk management — VaR, Kelly criterion, circuit breakers, and kill switches.
```

**crates/trading_engine/src/lib.rs** — add after `#![deny(...)]`:
```rust
//! Trading engine — FIX 4.4, IB TWS, order management, and execution.
```

**crates/training_uploader/src/main.rs** — add after `#![allow(...)]`:
```rust
//! K8s sidecar that uploads training artifacts to S3 and reports completion via gRPC.
```

**Step 2: Verify compilation**

Run: `SQLX_OFFLINE=true cargo check --workspace`
Expected: Clean (doc comments can't break compilation)

**Step 3: Commit**

```bash
git add crates/backtesting/src/lib.rs crates/ctrader-openapi/src/lib.rs crates/data/src/lib.rs crates/ml/src/lib.rs crates/risk/src/lib.rs crates/trading_engine/src/lib.rs crates/training_uploader/src/main.rs
git commit -m "docs: add lib.rs doc comments to 7 crates"
```

---

### Task 6: Final verification

**Step 1: Workspace build**

Run: `SQLX_OFFLINE=true cargo check --workspace`
Expected: zero errors

**Step 2: Clippy on workspace**

Run: `SQLX_OFFLINE=true cargo clippy --workspace --lib -- -D warnings`
Expected: zero warnings

**Step 3: Verify no stray .md files remain**

```bash
# List all non-README .md files in code directories (should be empty or near-empty)
find crates/ services/ bin/ testing/ -name "*.md" ! -name "README.md" | sort
```

Expected: empty output (all scattered docs deleted)

**Step 4: Verify every crate/service has README**

```bash
for dir in crates/*/; do
  [ -f "$dir/README.md" ] && echo "OK $dir" || echo "MISSING $dir"
done
for dir in services/*/; do
  [ -f "$dir/README.md" ] && echo "OK $dir" || echo "MISSING $dir"
done
[ -f bin/fxt/README.md ] && echo "OK bin/fxt" || echo "MISSING bin/fxt"
```

Expected: all OK (services/tests/ may be MISSING — that's acceptable, it's a disabled service)

**Step 5: Line count verification**

```bash
git diff --stat HEAD~4
# Should show significant net negative lines from doc deletion
```