foxhunt/CLAUDE.md

CLAUDE.md - Foxhunt HFT Trading System

Last Updated: 2025-10-06

---

🎯 System Overview

Foxhunt is a high-frequency trading system built in Rust with ML/AI-powered decision making. The system uses microservices architecture with gRPC communication, PostgreSQL for persistence, and advanced ML models (MAMBA-2, DQN, PPO, TFT) for trading strategies.

Core Principle: REUSE existing infrastructure. DO NOT rebuild components.

---

🏗️ Architecture

Service Topology

┌─────────────────────────────────────────────────────────────┐
│                         TLI (Terminal)                       │
│                    Pure Client - Port 50051                  │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│                    API Gateway (Port 50051)                  │
│          Auth, Rate Limiting, Config Management              │
│    JWT, MFA, Session Management, Audit Logging              │
└───┬──────────────────┬──────────────────┬───────────────────┘
    │                  │                  │
    ▼                  ▼                  ▼
┌──────────┐    ┌──────────────┐    ┌────────────────┐
│ Trading  │    │ Backtesting  │    │  ML Training   │
│ Service  │    │   Service    │    │    Service     │
│Port 50052│    │  Port 50053  │    │  Port 50054    │
└─────┬────┘    └──────┬───────┘    └────────┬───────┘
      │                │                      │
      └────────────────┴──────────────────────┘
                       │
         ┌─────────────┴─────────────┐
         ▼                           ▼
┌──────────────┐            ┌────────────────┐
│  PostgreSQL  │            │     Redis      │
│ (TimescaleDB)│            │   (Cache)      │
│  Port 5432   │            │   Port 6379    │
└──────────────┘            └────────────────┘

Component Responsibilities

TLI (Terminal Line Interface):

• Pure client - NO server components
• Connects ONLY to API Gateway
• NO database/ML/risk dependencies
• User interface for trading operations

API Gateway:

• Single entry point for all clients
• Centralized authentication (JWT + MFA)
• Rate limiting and request routing
• Configuration hot-reload from PostgreSQL
• Audit logging for compliance

Trading Service:

• Core trading logic and execution
• Position management
• Risk management integration
• Real-time market data processing

Backtesting Service:

• Strategy testing with historical data
• Parquet-based market data replay
• Performance analytics (Sharpe, drawdown, PnL)
• Model versioning support

ML Training Service:

• Model training pipeline
• Feature engineering (technical indicators, microstructure, TLOB)
• Checkpoint management
• Distributed training coordination

---

📁 Codebase Structure

foxhunt/
├── common/              # Shared types, error handling, traits
├── config/              # Central configuration (ONLY crate with Vault access)
├── data/                # Market data providers, Parquet persistence
├── ml/                  # ML models: MAMBA-2, DQN, PPO, TFT, Liquid
├── risk/                # VaR, circuit breakers, compliance
├── storage/             # S3 integration for archival
├── trading_engine/      # Core HFT engine with lockfree queues
├── services/
│   ├── api_gateway/     # Auth + routing gateway
│   ├── trading_service/ # Trading business logic
│   ├── backtesting_service/
│   └── ml_training_service/
├── tli/                 # Terminal client
├── migrations/          # Database migrations (17 applied)
└── test_data/           # Test datasets (Parquet files)

---

🔑 Infrastructure & Credentials

Docker Services

Start all infrastructure:

docker-compose up -d
docker-compose ps  # Verify all services healthy

Database Credentials (from docker-compose.yml)

PostgreSQL (TimescaleDB):

Host: localhost:5432
Database: foxhunt
User: foxhunt
Password: foxhunt_dev_password
Connection URL: postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Connect from CLI
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Run migrations
cargo sqlx migrate run

Redis:

Host: localhost:6379
URL: redis://localhost:6379

# Test connection
redis-cli ping

InfluxDB (Time-series metrics):

Host: localhost:8086
User: foxhunt
Password: foxhunt_dev_password
Org: foxhunt
Bucket: trading_metrics

# Web UI: http://localhost:8086

HashiCorp Vault (Secrets):

Host: localhost:8200
Dev Token: foxhunt-dev-root
URL: http://vault:8200

# Access from services
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=foxhunt-dev-root

Grafana (Dashboards):

URL: http://localhost:3000
Username: admin
Password: foxhunt123

Prometheus (Metrics):

URL: http://localhost:9090

Service Ports

Service	External Port	Internal Port	Metrics Port
API Gateway	50051	50050	9091
Trading Service	50052	50051	9092
Backtesting Service	50053	50052	9093
ML Training Service	50054	50053	9094

Environment Variables

Development (from docker-compose.yml):

DATABASE_URL=postgresql://foxhunt:foxhunt_dev_password@postgres:5432/foxhunt
REDIS_URL=redis://redis:6379
VAULT_ADDR=http://vault:8200
VAULT_TOKEN=foxhunt-dev-root
JWT_SECRET=dev_secret_key_change_in_production
RUST_LOG=info
RUST_BACKTRACE=1

Production (use Vault for secrets):

# Load from .env (never commit this file!)
cp .env.example .env
# Edit .env with production credentials

GPU/CUDA Configuration (ML Inference)

CUDA Environment (RTX 3050 Ti - enabled in Wave 115):

# CUDA environment variables (already in ~/.bashrc)
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$CUDA_HOME/targets/x86_64-linux/lib:$LD_LIBRARY_PATH
export PATH=$CUDA_HOME/bin:$PATH

# Verify CUDA availability
nvidia-smi  # Check GPU status
nvcc --version  # CUDA compiler version (12.8/12.9/13.0)

ML Crate CUDA Support:

# ml/Cargo.toml (Wave 115: CUDA enabled)
[dependencies]
candle-core = { version = "0.9", features = ["cuda"] }  # GPU acceleration
candle-nn = { version = "0.9" }
candle-optimisers = { version = "0.9" }

[features]
cuda = ["candle-core/cuda", "candle-core/cudnn"]  # Optional for CI/Docker

Usage in Code:

// ml/src/inference.rs
use candle_core::{Device, Tensor};

// GPU device selection (automatic fallback to CPU)
let device = Device::cuda_if_available(0)?;  // Use GPU 0 if available

// Create tensor on GPU
let input = Tensor::new(&[1.0, 2.0, 3.0], &device)?;

// All candle operations automatically use GPU when device is CUDA
let output = model.forward(&input)?;  // Runs on GPU

Testing with GPU:

# Run ML tests (GPU-enabled)
cargo test -p ml --lib

# Slow GPU tests are marked with #[ignore]
cargo test -p ml --lib -- --ignored  # Run slow GPU tests explicitly

# Check GPU utilization during tests
watch -n 1 nvidia-smi  # Monitor GPU usage in real-time

Docker GPU Support (for production):

# docker-compose.yml (add for ML training service)
services:
  ml_training_service:
    runtime: nvidia  # NVIDIA Container Runtime
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility

Performance Impact:

• ML inference: CPU → GPU (RTX 3050 Ti)
• Model loading: ~60s (3 models with GPU initialization)
• Inference latency: 10-50x faster for large models
• MAMBA-2, TFT, DQN all GPU-accelerated

Troubleshooting:

# If GPU not detected
nvidia-smi  # Verify GPU visible
nvcc --version  # Verify CUDA installed
echo $CUDA_HOME  # Should be /usr/local/cuda
echo $LD_LIBRARY_PATH  # Should include CUDA libs

# Rebuild ml crate with CUDA
cargo clean -p ml
cargo build -p ml --features cuda

# Check candle GPU support
cargo test -p ml --lib test_model_loading_multiple_models -- --nocapture

---

🚫 Critical Architectural Rules

1. Configuration Management

• ONLY the config crate accesses Vault directly
• NO type aliases or backward compatibility layers
• Services import: use config::{ServiceConfig, ConfigManager};
• NEVER create foxhunt-config-crate or foxhunt-* prefixed crates

2. TLI Architecture

• TLI is a PURE CLIENT - NO server components
• NO WebSocketServer, NO HealthServer
• NO database/ML/risk dependencies
• Connects ONLY to API Gateway (port 50051)

3. Service Boundaries

• API Gateway: Server for TLI, client for backend services
• Trading Service: Monolithic business logic
• Backtesting/ML Services: Independent, specialized services
• All inter-service communication via gRPC

4. Error Handling Patterns

// CommonError factory methods (common/src/error.rs)
CommonError::config("message")           // Configuration errors
CommonError::network("message")          // Network errors
CommonError::service(ErrorCategory, "msg") // Service errors
CommonError::validation("message")       // Validation errors
CommonError::internal("message")         // Internal errors

// StorageError variants (storage/src/error.rs)
StorageError::ConfigError { message }    // Config errors
StorageError::IoError { message }        // I/O errors
StorageError::NetworkError { message }   // Network errors
// NO StorageError::Common variant!

5. Common Compilation Fixes

// Use ::std::core:: not core:: when local crate shadows std
use ::std::core::mem;

// Add async-stream when needed
async-stream = "0.3"

// NO direct vault access outside config crate
// ❌ use vault_service::...
// ✅ use config::ConfigManager;

---

🧪 Testing Infrastructure (REUSE)

See TESTING_PLAN.md for comprehensive testing strategy.

Existing Components

Parquet Market Data Replay:

// data/src/parquet_persistence.rs
let writer = ParquetMarketDataWriter::new(...);
writer.write_event(market_event).await?;

let reader = ParquetMarketDataReader::new(...);
let events = reader.read_file("test.parquet").await?;

Backtesting Service (gRPC):

let client = BacktestingServiceClient::connect("http://localhost:50053").await?;
let response = client.start_backtest(request).await?;

Feature Engineering:

// data/src/training_pipeline.rs
let processor = FeatureProcessor::new(config);
let features = processor.process_batch(&market_data).await?;

Test Database Setup

# 1. Start PostgreSQL
docker-compose up -d postgres

# 2. Run migrations
cargo sqlx migrate run

# 3. Verify schema
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt -c '\dt'

SQLx Offline Mode

For CI/CD without live database:

# Generate metadata
cargo sqlx prepare --workspace

# Enable offline mode
echo 'SQLX_OFFLINE=true' >> .cargo/config.toml

---

🛠️ Development Workflow

Initial Setup

# 1. Clone repository
git clone <repo-url>
cd foxhunt

# 2. Start infrastructure
docker-compose up -d

# 3. Wait for services to be healthy
docker-compose ps

# 4. Run database migrations
cargo sqlx migrate run

# 5. Build workspace
cargo build --workspace

# 6. Run tests
cargo test --workspace

Common Commands

# Build all services
cargo build --workspace --release

# Run specific service
cargo run -p trading_service

# Test specific package
cargo test -p ml

# Check compilation (fast)
cargo check --workspace

# Run linter
cargo clippy --workspace -- -D warnings

# Measure test coverage
cargo llvm-cov --html --output-dir coverage_report

# Clean build artifacts
cargo clean

Running Services

# Via Docker Compose (recommended)
docker-compose up -d api_gateway trading_service backtesting_service ml_training_service

# Via Cargo (development)
cargo run -p api_gateway &
cargo run -p trading_service &
cargo run -p backtesting_service &
cargo run -p ml_training_service &

---

📊 Current Status

Production Readiness: 87.8% (7.2% from deployment) ⚠️ Revised Baseline

Complete (100%):

• ✅ Monitoring: Prometheus alerts, Grafana dashboards
• ✅ Documentation: 85K+ lines comprehensive docs
• ✅ Reliability: Circuit breakers, chaos testing
• ✅ Scalability: Horizontal scaling, load balancing
• ✅ Deployment: All 4 services compile + Docker validated

In Progress:

• 🟡 Testing: 37.83% coverage (accurate full workspace measurement)
• 🟡 Compliance: 83% SOX/MiFID II (target: 100%)
• 🟡 Performance: 36% (auth validated, full cycle pending)
• 🟡 Security: CVSS 5.9 (1 mitigated vulnerability)

Recent Achievements

Wave 116 (12 agents) ⚠️ BASELINE CORRECTION:

• 211 new tests: ~7,000 lines of test code added
• ML model tests: 136 tests (MAMBA-2, DQN, PPO, TFT, Liquid) - 70-75% coverage
• Backtesting tests: 62 tests (service, strategy, analytics) - 70-80% coverage
• SQLx unblocked: 11 queries converted to runtime (service coverage enabled)
• Critical discovery: Wave 115's 47.03% was incomplete (only 3 packages)
• Accurate baseline: 37.83% full workspace (includes trading_engine 25,190 lines)
• Zero coverage areas: 8,698 lines identified (compliance, persistence, config)
• Production readiness: 90.5% → 87.8% (revised to accurate measurement)

Wave 115 (13 agents):

• CUDA GPU support: RTX 3050 Ti enabled for ML inference
• Test failures: 26 → 0 fixed (100% pass rate achieved)
• Warnings: 939 → 452 eliminated (-487, -52%)
• Testing: 29.8% → 47.03% (incomplete - only 3 packages measured)

Wave 114 (10 agents):

• Service compilation: 96+ errors fixed → 0 errors (100% success)
• Common package coverage: 26.03% measured
• Trading engine tests: 26 errors fixed
• Production readiness: 90.0% → 90.5% (+0.5%)

Wave 113 (39 agents):

• Coverage unblocked: 29.8% → 47.03% (+17.23%)
• Security hardening: 67% vulnerability reduction
• Test suite: 1,532 tests validated (98.3% pass rate)
• Dependencies: 942 → 933 crates (-9)

Known Issues

1. Zero Coverage Areas (8,698 lines - 34.5% of measured codebase)

   • Compliance: 4,621 lines (audit_trails, reporting, sox, mifid)
   • Persistence: 2,735 lines (redis, clickhouse, postgres)
   • Config: 1,342 lines
   • Impact: Largest blocker to 60% coverage target

2. Test Failures: 1 test (0.7%) - Redis connection

   • trading_engine Redis connection test
   • Fix effort: 1-2 hours

3. ML/Backtesting Coverage Unmeasured: Compilation timeout (>3 min)

   • Large dependency trees (candle, nalgebra)
   • Solution: Separate measurement runs OR faster CI

4. Documentation Warnings: 452 warnings (missing docs)

   • Pre-commit hook blocks commits at 50 warning threshold
   • Fix effort: 1-2 weeks for full documentation

---

🚀 Next Priorities (Wave 117 - Path to 95% Production)

Current: 87.8% production readiness, 37.83% coverage Target: 95% production readiness, 60-70% coverage Timeline: 4-6 weeks

Priority 1: Zero Coverage Areas (2-3 weeks) → +13-15% coverage

Goal: Test 8,698 lines of untested code

1. Compliance modules (+4,621 lines):

   • Audit trails, transaction reporting
   • SOX compliance, MiFID II compliance
   • Automated reporting systems

2. Persistence modules (+2,735 lines):

   • Redis caching layer
   • ClickHouse analytics
   • PostgreSQL repository layer

3. Config modules (+1,342 lines):

   • Hot-reload validation
   • Schema validation
   • Configuration parsing

Expected Impact: 37.83% → 52% coverage (+14.17%)

Priority 2: Service Coverage Measurement (1-2 hours)

• SQLx unblocked by Wave 116 (Agent 9)
• Run service tests with PostgreSQL
• Gain: Validate service coverage targets

Priority 3: Fix Remaining Test Failure (1-2 hours)

• 1 Redis connection test failing
• Quick win for 100% pass rate
• Gain: +0.7% test reliability

Priority 4: E2E Performance Benchmarks (1-2 days)

• Performance only 36% (auth validated, full cycle untested)
• Latency profiling, load testing
• Gain: +40% performance score (36% → 80%)

---

📖 Documentation

Architecture & Development

• CLAUDE.md: This file - architecture fundamentals
• TESTING_PLAN.md: ML testing strategy with crypto data
• .env.example: Environment variable template

Wave Reports (Latest)

• WAVE_116_FINAL_SUMMARY.md: 12-agent coverage expansion (211 tests, baseline correction)
• WAVE115_FINAL_SUMMARY.md: CUDA enablement + test failure fixes (13 agents)
• WAVE114_FINAL_REPORT.md: Service compilation fixes (Phase 2)
• WAVE113_FINAL_SUMMARY.md: Coverage unblocking & security
• WAVE112_FINAL_STATUS.md: Systematic compilation fix

Technical Documentation

• migrations/README.md: Database schema changes
• docs/: Detailed component documentation
• README.md: Project overview

---

🔒 Security Best Practices

Development

• ✅ All .env files gitignored
• ✅ No hardcoded credentials in source
• ✅ API keys from environment variables
• ✅ Docker secrets for production

Production

• Use Vault for all secrets (not environment variables)
• Enable MFA for critical operations
• Rotate JWT secrets regularly
• Use TLS for all gRPC communication
• Enable audit logging (ENABLE_AUDIT_LOGGING=true)

Current Vulnerabilities

• RSA Marvin Attack (CVSS 5.9): Mitigated (PostgreSQL-only, no MySQL)
• 2 unmaintained dependencies (low risk): instant, paste

---

🐛 Anti-Workaround Protocol

FORBIDDEN Approaches

❌ NEVER create stubs or placeholders ❌ NEVER create fallback/compatibility layers ❌ NEVER skip features to avoid fixing them ❌ NEVER estimate when you can measure

REQUIRED Approaches

✅ ALWAYS fix root causes ✅ ALWAYS proper rewrites, not simplifications ✅ ALWAYS complete implementations ✅ ALWAYS reuse existing infrastructure

Examples

Bad:

// ❌ Stub implementation
pub fn read_file(&self, filename: &str) -> Result<Vec<MarketDataEvent>> {
    warn!("Not implemented yet");
    Ok(Vec::new())
}

Good:

// ✅ Complete implementation
pub async fn read_file(&self, filename: &str) -> Result<Vec<MarketDataEvent>> {
    let file = tokio::fs::File::open(filepath).await?;
    let builder = ParquetRecordBatchReaderBuilder::try_new(file).await?;
    // ... full Arrow-based Parquet reading
}

---

📞 Quick Reference

Docker Services

docker-compose up -d         # Start all services
docker-compose ps            # Check status
docker-compose logs -f <service>  # View logs
docker-compose down          # Stop all services

Database Operations

# PostgreSQL
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt
cargo sqlx migrate run
cargo sqlx migrate revert

# Redis
redis-cli -h localhost -p 6379

Service Health Checks

# API Gateway
grpc_health_probe -addr=localhost:50051

# Trading Service
grpc_health_probe -addr=localhost:50052

# All services via Prometheus
curl http://localhost:9090/api/v1/targets

Coverage Measurement

# Workspace coverage
cargo llvm-cov --html --output-dir coverage_report

# Specific package
cargo llvm-cov -p ml --html --output-dir coverage_ml

# View report
open coverage_report/index.html

---

🎓 Learning Resources

Rust + Async

• Tokio Tutorial
• Async Book

gRPC + Tonic

• Tonic Documentation
• gRPC Health Checking

HFT + Trading

• Market microstructure theory
• Order book dynamics
• Latency optimization techniques

ML/AI

• MAMBA-2: State space models
• DQN: Deep Q-learning
• PPO: Proximal Policy Optimization
• TFT: Temporal Fusion Transformer

---

Last Updated: 2025-10-06 Production Status: 90.5% (4.5% from deployment) Next Milestone: Wave 115 - ML testing infrastructure + test failure fixes