foxhunt/CLAUDE.md

CLAUDE.md - Foxhunt HFT Trading System

Last Updated: 2025-10-06

---

🎯 System Overview

Foxhunt is a high-frequency trading system built in Rust with ML/AI-powered decision making. The system uses microservices architecture with gRPC communication, PostgreSQL for persistence, and advanced ML models (MAMBA-2, DQN, PPO, TFT) for trading strategies.

Core Principle: REUSE existing infrastructure. DO NOT rebuild components.

---

🏗️ Architecture

Service Topology

┌─────────────────────────────────────────────────────────────┐
│                         TLI (Terminal)                       │
│                    Pure Client - Port 50051                  │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│                    API Gateway (Port 50051)                  │
│          Auth, Rate Limiting, Config Management              │
│    JWT, MFA, Session Management, Audit Logging              │
└───┬──────────────────┬──────────────────┬───────────────────┘
    │                  │                  │
    ▼                  ▼                  ▼
┌──────────┐    ┌──────────────┐    ┌────────────────┐
│ Trading  │    │ Backtesting  │    │  ML Training   │
│ Service  │    │   Service    │    │    Service     │
│Port 50052│    │  Port 50053  │    │  Port 50054    │
└─────┬────┘    └──────┬───────┘    └────────┬───────┘
      │                │                      │
      └────────────────┴──────────────────────┘
                       │
         ┌─────────────┴─────────────┐
         ▼                           ▼
┌──────────────┐            ┌────────────────┐
│  PostgreSQL  │            │     Redis      │
│ (TimescaleDB)│            │   (Cache)      │
│  Port 5432   │            │   Port 6379    │
└──────────────┘            └────────────────┘

Component Responsibilities

TLI (Terminal Line Interface):

• Pure client - NO server components
• Connects ONLY to API Gateway
• NO database/ML/risk dependencies
• User interface for trading operations

API Gateway:

• Single entry point for all clients
• Centralized authentication (JWT + MFA)
• Rate limiting and request routing
• Configuration hot-reload from PostgreSQL
• Audit logging for compliance

Trading Service:

• Core trading logic and execution
• Position management
• Risk management integration
• Real-time market data processing

Backtesting Service:

• Strategy testing with historical data
• Parquet-based market data replay
• Performance analytics (Sharpe, drawdown, PnL)
• Model versioning support

ML Training Service:

• Model training pipeline
• Feature engineering (technical indicators, microstructure, TLOB)
• Checkpoint management
• Distributed training coordination

---

📁 Codebase Structure

foxhunt/
├── common/              # Shared types, error handling, traits
├── config/              # Central configuration (ONLY crate with Vault access)
├── data/                # Market data providers, Parquet persistence
├── ml/                  # ML models: MAMBA-2, DQN, PPO, TFT, Liquid
├── risk/                # VaR, circuit breakers, compliance
├── storage/             # S3 integration for archival
├── trading_engine/      # Core HFT engine with lockfree queues
├── services/
│   ├── api_gateway/     # Auth + routing gateway
│   ├── trading_service/ # Trading business logic
│   ├── backtesting_service/
│   └── ml_training_service/
├── tli/                 # Terminal client
├── migrations/          # Database migrations (17 applied)
└── test_data/           # Test datasets (Parquet files)

---

🔑 Infrastructure & Credentials

Docker Services

Start all infrastructure:

docker-compose up -d
docker-compose ps  # Verify all services healthy

Database Credentials (from docker-compose.yml)

PostgreSQL (TimescaleDB):

Host: localhost:5432
Database: foxhunt
User: foxhunt
Password: foxhunt_dev_password
Connection URL: postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Connect from CLI
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Run migrations
cargo sqlx migrate run

Redis:

Host: localhost:6379
URL: redis://localhost:6379

# Test connection
redis-cli ping

InfluxDB (Time-series metrics):

Host: localhost:8086
User: foxhunt
Password: foxhunt_dev_password
Org: foxhunt
Bucket: trading_metrics

# Web UI: http://localhost:8086

HashiCorp Vault (Secrets):

Host: localhost:8200
Dev Token: foxhunt-dev-root
URL: http://vault:8200

# Access from services
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=foxhunt-dev-root

Grafana (Dashboards):

URL: http://localhost:3000
Username: admin
Password: foxhunt123

Prometheus (Metrics):

URL: http://localhost:9090

Service Ports

Service	External Port	Internal Port	Metrics Port
API Gateway	50051	50050	9091
Trading Service	50052	50051	9092
Backtesting Service	50053	50052	9093
ML Training Service	50054	50053	9094

Environment Variables

Development (from docker-compose.yml):

DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt
REDIS_URL=redis://redis:6379
VAULT_ADDR=http://vault:8200
VAULT_TOKEN=foxhunt-dev-root
JWT_SECRET=dev_secret_key_change_in_production
RUST_LOG=info
RUST_BACKTRACE=1

Production (use Vault for secrets):

# Load from .env (never commit this file!)
cp .env.example .env
# Edit .env with production credentials

GPU/CUDA Configuration (ML Inference)

CUDA Environment (RTX 3050 Ti - enabled in Wave 115):

# CUDA environment variables (already in ~/.bashrc)
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$CUDA_HOME/targets/x86_64-linux/lib:$LD_LIBRARY_PATH
export PATH=$CUDA_HOME/bin:$PATH

# Verify CUDA availability
nvidia-smi  # Check GPU status
nvcc --version  # CUDA compiler version (12.8/12.9/13.0)

ML Crate CUDA Support:

# ml/Cargo.toml (Wave 115: CUDA enabled)
[dependencies]
candle-core = { version = "0.9", features = ["cuda"] }  # GPU acceleration
candle-nn = { version = "0.9" }
candle-optimisers = { version = "0.9" }

[features]
cuda = ["candle-core/cuda", "candle-core/cudnn"]  # Optional for CI/Docker

Usage in Code:

// ml/src/inference.rs
use candle_core::{Device, Tensor};

// GPU device selection (automatic fallback to CPU)
let device = Device::cuda_if_available(0)?;  // Use GPU 0 if available

// Create tensor on GPU
let input = Tensor::new(&[1.0, 2.0, 3.0], &device)?;

// All candle operations automatically use GPU when device is CUDA
let output = model.forward(&input)?;  // Runs on GPU

Testing with GPU:

# Run ML tests (GPU-enabled)
cargo test -p ml --lib

# Slow GPU tests are marked with #[ignore]
cargo test -p ml --lib -- --ignored  # Run slow GPU tests explicitly

# Check GPU utilization during tests
watch -n 1 nvidia-smi  # Monitor GPU usage in real-time

Docker GPU Support (for production):

# docker-compose.yml (add for ML training service)
services:
  ml_training_service:
    runtime: nvidia  # NVIDIA Container Runtime
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility

Performance Impact:

• ML inference: CPU → GPU (RTX 3050 Ti)
• Model loading: ~60s (3 models with GPU initialization)
• Inference latency: 10-50x faster for large models
• MAMBA-2, TFT, DQN all GPU-accelerated

Troubleshooting:

# If GPU not detected
nvidia-smi  # Verify GPU visible
nvcc --version  # Verify CUDA installed
echo $CUDA_HOME  # Should be /usr/local/cuda
echo $LD_LIBRARY_PATH  # Should include CUDA libs

# Rebuild ml crate with CUDA
cargo clean -p ml
cargo build -p ml --features cuda

# Check candle GPU support
cargo test -p ml --lib test_model_loading_multiple_models -- --nocapture

---

🚫 Critical Architectural Rules

1. Configuration Management

• ONLY the config crate accesses Vault directly
• NO type aliases or backward compatibility layers
• Services import: use config::{ServiceConfig, ConfigManager};
• NEVER create foxhunt-config-crate or foxhunt-* prefixed crates

2. TLI Architecture

• TLI is a PURE CLIENT - NO server components
• NO WebSocketServer, NO HealthServer
• NO database/ML/risk dependencies
• Connects ONLY to API Gateway (port 50051)

3. Service Boundaries

• API Gateway: Server for TLI, client for backend services
• Trading Service: Monolithic business logic
• Backtesting/ML Services: Independent, specialized services
• All inter-service communication via gRPC

4. Error Handling Patterns

// CommonError factory methods (common/src/error.rs)
CommonError::config("message")           // Configuration errors
CommonError::network("message")          // Network errors
CommonError::service(ErrorCategory, "msg") // Service errors
CommonError::validation("message")       // Validation errors
CommonError::internal("message")         // Internal errors

// StorageError variants (storage/src/error.rs)
StorageError::ConfigError { message }    // Config errors
StorageError::IoError { message }        // I/O errors
StorageError::NetworkError { message }   // Network errors
// NO StorageError::Common variant!

5. Common Compilation Fixes

// Use ::std::core:: not core:: when local crate shadows std
use ::std::core::mem;

// Add async-stream when needed
async-stream = "0.3"

// NO direct vault access outside config crate
// ❌ use vault_service::...
// ✅ use config::ConfigManager;

---

🧪 Testing Infrastructure (REUSE)

See TESTING_PLAN.md for comprehensive testing strategy.

Existing Components

Parquet Market Data Replay:

// data/src/parquet_persistence.rs
let writer = ParquetMarketDataWriter::new(...);
writer.write_event(market_event).await?;

let reader = ParquetMarketDataReader::new(...);
let events = reader.read_file("test.parquet").await?;

Backtesting Service (gRPC):

let client = BacktestingServiceClient::connect("http://localhost:50053").await?;
let response = client.start_backtest(request).await?;

Feature Engineering:

// data/src/training_pipeline.rs
let processor = FeatureProcessor::new(config);
let features = processor.process_batch(&market_data).await?;

Test Database Setup

# 1. Start PostgreSQL
docker-compose up -d postgres

# 2. Run migrations
cargo sqlx migrate run

# 3. Verify schema
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\dt'

SQLx Offline Mode

For CI/CD without live database:

# Generate metadata
cargo sqlx prepare --workspace

# Enable offline mode
echo 'SQLX_OFFLINE=true' >> .cargo/config.toml

---

🛠️ Development Workflow

Initial Setup

# 1. Clone repository
git clone <repo-url>
cd foxhunt

# 2. Start infrastructure
docker-compose up -d

# 3. Wait for services to be healthy
docker-compose ps

# 4. Run database migrations
cargo sqlx migrate run

# 5. Build workspace
cargo build --workspace

# 6. Run tests
cargo test --workspace

Common Commands

# Build all services
cargo build --workspace --release

# Run specific service
cargo run -p trading_service

# Test specific package
cargo test -p ml

# Check compilation (fast)
cargo check --workspace

# Run linter
cargo clippy --workspace -- -D warnings

# Measure test coverage
cargo llvm-cov --html --output-dir coverage_report

# Clean build artifacts
cargo clean

Running Services

# Via Docker Compose (recommended)
docker-compose up -d api_gateway trading_service backtesting_service ml_training_service

# Via Cargo (development)
cargo run -p api_gateway &
cargo run -p trading_service &
cargo run -p backtesting_service &
cargo run -p ml_training_service &

---

📊 Current Status

Production Readiness: 89.5% (5.5% from deployment)

Complete (100%):

• ✅ Monitoring: Prometheus alerts, Grafana dashboards
• ✅ Documentation: 85K+ lines comprehensive docs
• ✅ Reliability: Circuit breakers, chaos testing
• ✅ Scalability: Horizontal scaling, load balancing
• ✅ Deployment: All 4 services compile + Docker validated

In Progress:

• 🟡 Testing: 46.28% coverage (up from 37.83%, target: 60%)
• 🟡 Compliance: 83% SOX/MiFID II (target: 100%)
• 🟡 Performance: 36% (auth validated, full cycle pending)
• 🟡 Security: CVSS 5.9 (1 mitigated vulnerability)

Recent Achievements

Wave 117 (15 agents) - ZERO COVERAGE ELIMINATION:

• 463 new tests: ~11,700 lines of test code added
• Coverage impact: 37.83% → 46.28% (+8.45% absolute, +22.3% relative)
• Compliance tests: 219 tests (audit trails, SOX, MiFID II, best execution)
• Persistence tests: 132 tests (Redis, ClickHouse, PostgreSQL)
• Config tests: 113 tests (runtime, schemas, structures)
• Zero coverage reduced: 8,698 → ~6,500 lines (-25.3%)
• Service coverage measured: API Gateway 20.19% baseline established
• Production readiness: 87.8% → 89.5% (+1.7%)

Wave 116 (12 agents) - BASELINE CORRECTION:

• 211 new tests: ~7,000 lines of test code added
• ML model tests: 136 tests (MAMBA-2, DQN, PPO, TFT, Liquid) - 70-75% coverage
• Backtesting tests: 62 tests (service, strategy, analytics) - 70-80% coverage
• SQLx unblocked: 11 queries converted to runtime (service coverage enabled)
• Critical discovery: Wave 115's 47.03% was incomplete (only 3 packages)
• Accurate baseline: 37.83% full workspace (includes trading_engine 25,190 lines)
• Zero coverage areas: 8,698 lines identified (compliance, persistence, config)
• Production readiness: 90.5% → 87.8% (revised to accurate measurement)

Wave 115 (13 agents):

• CUDA GPU support: RTX 3050 Ti enabled for ML inference
• Test failures: 26 → 0 fixed (100% pass rate achieved)
• Warnings: 939 → 452 eliminated (-487, -52%)
• Testing: 29.8% → 47.03% (incomplete - only 3 packages measured)

Wave 114 (10 agents):

• Service compilation: 96+ errors fixed → 0 errors (100% success)
• Common package coverage: 26.03% measured
• Trading engine tests: 26 errors fixed
• Production readiness: 90.0% → 90.5% (+0.5%)

Wave 113 (39 agents):

• Coverage unblocked: 29.8% → 47.03% (+17.23%)
• Security hardening: 67% vulnerability reduction
• Test suite: 1,532 tests validated (98.3% pass rate)
• Dependencies: 942 → 933 crates (-9)

Known Issues

1. CUDA 13.0 Incompatibility (CRITICAL BLOCKER): Service coverage blocked

   • Error: cudarc v0.16.6 doesn't support CUDA 13.0
   • Impact: Cannot measure Trading/Backtesting/ML Service coverage
   • Workaround: Use --no-default-features to disable CUDA for coverage
   • Fix effort: 1-2 days (add feature flags to ml crate)

2. Remaining Zero Coverage Areas (~6,500 lines - reduced from 8,698)

   • Compliance: ~1,400 lines remaining (Wave 117 covered 70-80%)
   • Persistence: ~800 lines remaining (ClickHouse blocked by mockito)
   • Config: ~200 lines remaining (circular dependency blocks 425 lines)
   • Trading Engine Core: ~3,000 lines (order matching, lockfree queues)
   • Risk Engine: ~1,100 lines (circuit breakers, VaR calculations)
   • Impact: Need 60% target (currently 46.28%)

3. Test Failures: 7 tests (0.4%) - down from 1,653 total

   • Data package: 5 failures (config default value mismatches)
   • ML package: 2 failures (GPU/SIMD threshold issues)
   • Redis integration: ✅ Fixed by Wave 117 Agent 7
   • Fix effort: 4-6 hours

4. Mockito 1.7.0 Compatibility: ClickHouse tests blocked

   • Error: Mock HTTP server returns "HTTP 501 Not Implemented"
   • Impact: 36 ClickHouse tests compile but fail at runtime (0% pass rate)
   • Solution: Downgrade to mockito 0.31.x OR switch to wiremock
   • Fix effort: 1-2 hours

5. Config Compilation Timeout: Circular dependency blocks 58 tests

   • Impact: 425 lines unmeasured (structures.rs, schemas.rs)
   • Solution: Resolve circular dependency in config package
   • Fix effort: 1-2 days

6. Documentation Warnings: 452 warnings (missing docs)

   • Pre-commit hook blocks commits at 50 warning threshold
   • Fix effort: 1-2 weeks for full documentation

---

🚀 Next Priorities (Wave 118 - Path to 95% Production)

Current: 89.5% production readiness, 46.28% coverage Target: 95% production readiness, 60-70% coverage Timeline: 3-4 weeks

Phase 1: Fix Critical Blockers (1-2 days) - IMMEDIATE

Goal: Unblock service coverage measurement and fix test failures

1. CUDA 13.0 Incompatibility (Priority 1 - 1-2 days):

   • Add feature flags to ml crate (default = [], cuda = [...])
   • Measure service coverage with --no-default-features
   • Target: Trading Service, Backtesting Service, ML Training Service
   • Expected Impact: Unblocks 3 services, enables accurate baseline

2. Test Failures (4-6 hours):

   • Fix 5 data package failures (config default values)
   • Fix 2 ml package failures (GPU/SIMD thresholds)
   • Target: 99.6% → 100% pass rate
   • Expected Impact: +0.4% reliability

3. Mockito 1.7.0 Issue (1-2 hours):

   • Downgrade to mockito 0.31.x OR switch to wiremock
   • Unblock 36 ClickHouse tests
   • Expected Impact: +800 lines persistence coverage

4. Config Compilation (1-2 days):

   • Resolve circular dependency in config package
   • Unblock 58 tests (425 lines)
   • Expected Impact: +1.5% config coverage

Phase 2: Trading Engine Core Coverage (1 week) → +5-7% coverage

Goal: Test core HFT trading logic (~3,000 lines at 0%)

1. Order Matching Engine:

   • Limit orders, market orders, stop orders
   • Price-time priority, order book management
   • Partial fills, order cancellation

2. Lockfree Queue Performance:

   • SPSC/MPMC queue implementations
   • Latency benchmarks (<1μs target)
   • Concurrent access patterns

3. Market Data Processing:

   • L2 order book updates
   • Trade execution confirmation
   • Market microstructure features

Expected Impact: 46.28% → 51-53% coverage (+5-7%)

Phase 3: Risk Engine Core Coverage (3-5 days) → +2-3% coverage

Goal: Test risk management and compliance (~1,100 lines at 0%)

1. Circuit Breakers:

   • Price movement limits
   • Volume spike detection
   • Position limit enforcement

2. VaR Calculations:

   • Historical simulation
   • Monte Carlo simulation
   • Parametric VaR

3. Real-time Risk Monitoring:

   • Position delta, gamma, vega
   • Portfolio P&L tracking
   • Margin requirement calculation

Expected Impact: 51-53% → 54-56% coverage (+2-3%)

Phase 4: Service E2E Integration (1 week) → +4-6% coverage

Goal: End-to-end service testing and performance validation

1. Service Coverage Measurement (now unblocked):

   • Measure Trading Service, Backtesting Service, ML Training Service
   • Establish accurate baselines per service
   • Target: 30-40% service coverage average

2. E2E Performance Benchmarks:

   • Full order lifecycle latency (target: <5ms p99)
   • Load testing (1K orders/second sustained)
   • Stress testing (10K orders/second peak)
   • Gain: +40% performance score (36% → 80%)

3. Integration Test Suite:

   • API Gateway → Trading Service flow
   • Trading Service → ML Training Service
   • Backtesting Service end-to-end

Expected Impact: 54-56% → 60% coverage (+4-6%)

---

📖 Documentation

Architecture & Development

• CLAUDE.md: This file - architecture fundamentals
• TESTING_PLAN.md: ML testing strategy with crypto data
• .env.example: Environment variable template

Wave Reports (Latest)

• WAVE_116_FINAL_SUMMARY.md: 12-agent coverage expansion (211 tests, baseline correction)
• WAVE115_FINAL_SUMMARY.md: CUDA enablement + test failure fixes (13 agents)
• WAVE114_FINAL_REPORT.md: Service compilation fixes (Phase 2)
• WAVE113_FINAL_SUMMARY.md: Coverage unblocking & security
• WAVE112_FINAL_STATUS.md: Systematic compilation fix

Technical Documentation

• migrations/README.md: Database schema changes
• docs/: Detailed component documentation
• README.md: Project overview

---

🔒 Security Best Practices

Development

• ✅ All .env files gitignored
• ✅ No hardcoded credentials in source
• ✅ API keys from environment variables
• ✅ Docker secrets for production

Production

• Use Vault for all secrets (not environment variables)
• Enable MFA for critical operations
• Rotate JWT secrets regularly
• Use TLS for all gRPC communication
• Enable audit logging (ENABLE_AUDIT_LOGGING=true)

Current Vulnerabilities

• RSA Marvin Attack (CVSS 5.9): Mitigated (PostgreSQL-only, no MySQL)
• 2 unmaintained dependencies (low risk): instant, paste

---

🐛 Anti-Workaround Protocol

FORBIDDEN Approaches

❌ NEVER create stubs or placeholders ❌ NEVER create fallback/compatibility layers ❌ NEVER skip features to avoid fixing them ❌ NEVER estimate when you can measure

REQUIRED Approaches

✅ ALWAYS fix root causes ✅ ALWAYS proper rewrites, not simplifications ✅ ALWAYS complete implementations ✅ ALWAYS reuse existing infrastructure

Examples

Bad:

// ❌ Stub implementation
pub fn read_file(&self, filename: &str) -> Result<Vec<MarketDataEvent>> {
    warn!("Not implemented yet");
    Ok(Vec::new())
}

Good:

// ✅ Complete implementation
pub async fn read_file(&self, filename: &str) -> Result<Vec<MarketDataEvent>> {
    let file = tokio::fs::File::open(filepath).await?;
    let builder = ParquetRecordBatchReaderBuilder::try_new(file).await?;
    // ... full Arrow-based Parquet reading
}

---

📞 Quick Reference

Docker Services

docker-compose up -d         # Start all services
docker-compose ps            # Check status
docker-compose logs -f <service>  # View logs
docker-compose down          # Stop all services

Database Operations

# PostgreSQL
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
cargo sqlx migrate run
cargo sqlx migrate revert

# Redis
redis-cli -h localhost -p 6379

Service Health Checks

# API Gateway
grpc_health_probe -addr=localhost:50051

# Trading Service
grpc_health_probe -addr=localhost:50052

# All services via Prometheus
curl http://localhost:9090/api/v1/targets

Coverage Measurement

# Workspace coverage
cargo llvm-cov --html --output-dir coverage_report

# Specific package
cargo llvm-cov -p ml --html --output-dir coverage_ml

# View report
open coverage_report/index.html

---

🎓 Learning Resources

Rust + Async

• Tokio Tutorial
• Async Book

gRPC + Tonic

• Tonic Documentation
• gRPC Health Checking

HFT + Trading

• Market microstructure theory
• Order book dynamics
• Latency optimization techniques

ML/AI

• MAMBA-2: State space models
• DQN: Deep Q-learning
• PPO: Proximal Policy Optimization
• TFT: Temporal Fusion Transformer

---

Last Updated: 2025-10-06 Production Status: 89.5% (5.5% from deployment) Next Milestone: Wave 118 - Fix critical blockers (CUDA, test failures, mockito, config)