foxhunt/docs/SYSTEM_ARCHITECTURE.md

# Foxhunt HFT Trading System - System Architecture

## Overview

Foxhunt is a high-frequency trading (HFT) system designed for ultra-low latency operations with sub-50μs execution times. The system employs a modular architecture with specialized components for performance, machine learning, risk management, and compliance.

**Last Updated**: 2025-09-24 - Production-Ready Status
**Current Version**: 1.0.0 Production
**Performance Status**: 14ns timing precision achieved, sub-50μs target latency validated

## Architecture Principles

### Performance-First Design
- **Ultra-Low Latency**: Target <50μs end-to-end order execution (validated in production)
- **Hardware Timing**: 14ns precision RDTSC-based timing (measured)
- **SIMD Optimization**: AVX2/AVX-512 vectorization with runtime detection
- **Lock-Free Structures**: Zero-contention concurrent operations
- **CPU Affinity**: Dedicated cores for critical trading threads
- **Small Batch Processing**: Optimized batch operations for HFT workloads

### Enterprise Safety & Reliability
- **Mathematical Safety**: NaN/Infinity detection and gradient clipping
- **Financial Type Safety**: Unified decimal types preventing precision loss
- **Circuit Breakers**: Automatic trading halts with atomic kill switches
- **Graceful Degradation**: Fallback mechanisms for component failures
- **Real-time Monitoring**: Continuous health and performance tracking
- **Data Persistence**: Multi-tier storage (PostgreSQL, InfluxDB, Redis, ClickHouse)

### Regulatory Compliance
- **SOX Compliance**: Financial controls and comprehensive audit trails
- **MiFID II**: Transaction reporting and best execution compliance
- **Real-time Risk Management**: Position limits, VaR, and exposure controls
- **Audit Trail**: Complete transaction traceability and regulatory reporting

## System Components

```
┌─────────────────────────────────────────────────────────────────────────────────┐
│                           Foxhunt HFT System v1.0.0                           │
│                         Production-Ready Architecture                          │
├─────────────────────────────────────────────────────────────────────────────────┤
│                        TLI (Terminal Interface)                                │
│          gRPC Services + Real-time Streaming + Security Layer                  │
│     Trading • Config • Health • ML Training • Resource Management              │
├─────────────────┬─────────────────┬─────────────────┬─────────────────────────┤
│      Core       │    ML Models    │  Risk & Safety  │    Data & Persistence   │
│   Performance   │   & Training    │   Management    │       Management        │
│   (14ns timing) │    Pipeline     │                 │                         │
├─────────────────┼─────────────────┼─────────────────┼─────────────────────────┤
│ • Types System  │ • TLOB Trans.   │ • Risk Engine   │ • Multi-tier Storage    │
│ • RDTSC Timing  │ • MAMBA SSM     │ • VaR Calculator│   - PostgreSQL (ACID)  │
│ • SIMD/AVX2     │ • DQN/PPO RL    │ • Kelly Sizing  │   - InfluxDB (Metrics)  │
│ • Lock-free     │ • Liquid NN     │ • Position Track│   - Redis (Cache)       │
│ • CPU Affinity  │ • TFT (Temporal)│ • Stress Test   │   - ClickHouse (OLAP)   │
│ • Small Batch   │ • Ensemble      │ • Circuit Break │ • Real-time Streams     │
│   Optimizer     │ • Training API  │ • Atomic Kill   │   - Polygon.io          │
│ • Event System  │ • Model Registry│ • Compliance    │   - Broker Feeds        │
│ • Config Mgmt   │ • Safety Ctrl   │ • SOX/MiFID II  │ • Event Sourcing        │
└─────────────────┴─────────────────┴─────────────────┴─────────────────────────┘
                                       │
                           ┌───────────┴───────────┐
                                                      │  ICMarkets • IB TWS   │
                           │   FIX 4.4 • REST API  │
                           └───────────────────────┘
                         │ • Interactive     │
                         │   Brokers (TWS)   │
                         │ • ICMarkets       │
                         │   (FIX 4.4)       │
                         │ • Order Routing   │
                         └───────────────────┘
```

## Core Performance Module

### Architecture

The core module provides the foundational infrastructure for ultra-low latency operations:

```rust
core/
├── types/           # Financial types with optimized memory layout
├── timing/          # RDTSC-based nanosecond precision timing
├── simd/            # AVX2/AVX-512 vectorized operations
├── lockfree/        # Lock-free data structures (SPSC, MPSC)
├── affinity/        # CPU core binding and real-time scheduling
├── events/          # High-performance event processing
├── config/          # Environment-based configuration
├── trading/         # Core trading engine and order management
└── brokers/         # Broker connectivity and FIX protocol
```

### Key Features

- **14ns Timing Precision**: Hardware timestamp counter (RDTSC) for ultra-precise latency measurement
- **SIMD Acceleration**: Automatic AVX2/AVX-512 detection with vectorized price calculations
- **Lock-Free Queues**: Single-producer single-consumer (SPSC) and multi-producer single-consumer (MPSC) queues
- **CPU Affinity Management**: Dedicated cores for trading threads to avoid context switching
- **Small Batch Optimization**: Batch processing for improved throughput without latency penalty

### Performance Characteristics

| Component | Latency Target | Throughput Target |
|-----------|---------------|-------------------|
| Order Submission | <50μs | >10,000/sec |
| Risk Checks | <10μs | >1,000/sec |
| Market Data Processing | <5μs | >100,000/sec |
| Timing Operations | <14ns | Continuous |

## Machine Learning Module

### Architecture

Advanced ML models for trading signal generation and market prediction:

```rust
ml/
├── tlob/            # TLOB (Time-Limit Order Book) Transformer
├── mamba/           # MAMBA-2 State Space Models
├── dqn/             # Deep Q-Network reinforcement learning
├── ppo/             # Proximal Policy Optimization
├── liquid/          # Liquid Neural Networks
├── tft/             # Temporal Fusion Transformer
├── features/        # Feature engineering pipeline
├── training/        # Model training infrastructure
├── inference/       # Real-time inference engine
└── benchmarks/      # Performance testing
```

### Model Specifications

#### TLOB Transformer
- **Purpose**: Order book level prediction
- **Architecture**: Multi-head attention with temporal encoding
- **Input**: L2 order book snapshots, trade history
- **Output**: Price movement probabilities
- **Latency**: <100μs inference time

#### MAMBA State Space Model
- **Purpose**: Long sequence modeling for market regimes
- **Architecture**: Selective state space with hardware-aware optimizations
- **Input**: Multi-timeframe market data
- **Output**: Regime classifications and trend predictions
- **Memory**: Constant O(1) memory complexity

#### DQN Agent
- **Purpose**: Reinforcement learning for position sizing
- **Architecture**: Double DQN with prioritized experience replay
- **State Space**: Portfolio state, market features, risk metrics
- **Action Space**: Position sizes and hold/exit decisions
- **Training**: Continuous online learning

### GPU Acceleration

The system supports CUDA acceleration for ML inference:

```rust
// GPU feature detection
if cuda_available() {
    let gpu_model = TlobTransformer::new_gpu(config)?;
} else {
    let cpu_model = TlobTransformer::new_cpu(config)?;
}
```

## Risk Management Module

### Architecture

Comprehensive risk management with real-time monitoring:

```rust
risk/
├── risk_engine.rs      # Central risk management engine
├── position_tracker.rs # Real-time position tracking
├── var_calculator.rs   # Value-at-Risk calculations
├── kelly_sizing.rs     # Optimal position sizing
├── compliance.rs       # Regulatory compliance
├── circuit_breaker.rs  # Emergency trading halts
├── stress_tester.rs    # Portfolio stress testing
└── safety/            # Atomic kill switches and safety mechanisms
```

### Risk Controls

#### Pre-Trade Checks
1. **Position Limits**: Maximum position sizes per symbol/sector
2. **Concentration Limits**: Maximum portfolio allocation percentages
3. **Correlation Limits**: Maximum correlated position exposure
4. **Liquidity Checks**: Minimum market liquidity requirements
5. **Volatility Filters**: Maximum allowed volatility exposure

#### Post-Trade Monitoring
1. **Real-time PnL**: Continuous profit/loss tracking
2. **Drawdown Monitoring**: Maximum drawdown thresholds
3. **VaR Calculation**: Daily Value-at-Risk assessment
4. **Stress Testing**: Scenario-based portfolio analysis
5. **Margin Monitoring**: Real-time margin requirement tracking

#### Emergency Procedures
```rust
// Circuit breaker activation
if portfolio_loss > max_daily_loss {
    circuit_breaker.emergency_halt().await?;
    notify_risk_team().await?;
}

// Atomic kill switch
if system_anomaly_detected() {
    atomic_kill_switch.activate().await?;
    liquidate_all_positions().await?;
}
```

## Data Management Module

### Architecture

Real-time and historical market data management:

```rust
data/
├── polygon.rs       # Polygon.io integration
├── providers/       # Multiple data source providers
├── cache/          # High-performance data caching
├── streaming/      # Real-time data streaming
├── historical/     # Historical data management
└── aggregation/    # Multi-source data aggregation
```

### Data Flow

```
Market Data Sources → Data Providers → Cache Layer → Trading Engine
       │                   │              │             │
   Polygon.io          Aggregation     Redis Cache   Order Logic
   Alpaca              Validation      Memory Pool   Risk Checks
   IEX Cloud           Normalization   Lock-free Q   ML Features
```

### Performance Specifications

- **Market Data Latency**: <1ms from exchange to application
- **Cache Hit Rate**: >99% for frequently accessed symbols
- **Storage**: InfluxDB for time-series, PostgreSQL for relational
- **Throughput**: >1M market data updates/second

## TLI (Terminal Interface) Module

### Architecture

Remote management and monitoring interface:

```rust
tli/
├── server/          # gRPC server implementation
├── client/          # Client SDK and CLI tools
├── dashboard/       # Web-based dashboard
├── auth/           # Authentication and authorization
├── health/         # System health monitoring
└── config/         # Configuration management
```

### gRPC Services

```protobuf
service TradingService {
    rpc SubmitOrder(OrderRequest) returns (OrderResponse);
    rpc GetPositions(PositionRequest) returns (PositionResponse);
    rpc GetSystemHealth(HealthRequest) returns (HealthResponse);
    rpc UpdateConfig(ConfigRequest) returns (ConfigResponse);
}

service RiskService {
    rpc GetRiskMetrics(RiskRequest) returns (RiskResponse);
    rpc UpdateRiskLimits(LimitRequest) returns (LimitResponse);
    rpc TriggerStressTest(StressRequest) returns (StressResponse);
}

service MLService {
    rpc GetPredictions(PredictionRequest) returns (PredictionResponse);
    rpc UpdateModel(ModelRequest) returns (ModelResponse);
    rpc GetModelMetrics(MetricsRequest) returns (MetricsResponse);
}
```

### Dashboard Features

- **Real-time Monitoring**: Live system metrics and performance
- **Order Management**: Order submission and execution tracking
- **Risk Dashboard**: Real-time risk metrics and limits
- **Performance Analytics**: Latency histograms and throughput charts
- **Configuration Management**: Dynamic parameter updates

## Broker Integration

### Supported Brokers

#### Interactive Brokers (TWS)
- **Protocol**: TWS API over TCP
- **Features**: Full order management, market data, account info
- **Latency**: ~5-15ms to exchange
- **Redundancy**: Multiple gateway connections

#### ICMarkets
- **Protocol**: FIX 4.4
- **Features**: Direct market access, institutional rates
- **Latency**: ~1-5ms to exchange
- **Connectivity**: Co-located servers available

### Order Routing

```rust
// Smart order routing with latency optimization
let router = OrderRouter::new()
    .add_venue(Venue::InteractiveBrokers, ib_config)
    .add_venue(Venue::ICMarkets, ic_config)
    .with_routing_strategy(RoutingStrategy::LowestLatency)
    .build()?;

let execution = router.route_order(order).await?;
```

## Data Storage Architecture

### Time-Series Data (InfluxDB)
- **Market Data**: Real-time and historical price/volume data
- **Performance Metrics**: Latency measurements, throughput stats
- **Trading Metrics**: Order flow, execution statistics
- **System Metrics**: CPU usage, memory consumption, network I/O

### Relational Data (PostgreSQL)
- **Configuration**: System and strategy parameters
- **Audit Logs**: Complete audit trail for compliance
- **User Management**: Authentication and authorization data
- **Reference Data**: Symbol mappings, exchange calendars

### Caching Layer (Redis)
- **Hot Data**: Frequently accessed market data
- **Session Data**: User sessions and temporary state
- **Rate Limiting**: API rate limiting counters
- **Feature Cache**: Pre-computed ML features

## Security Architecture

### Authentication & Authorization
- **JWT Tokens**: Stateless authentication with configurable expiry
- **Role-Based Access**: Granular permissions for different user types
- **API Keys**: Service-to-service authentication
- **Session Management**: Secure session handling with automatic timeout

### Network Security
- **TLS Encryption**: All communications encrypted with TLS 1.3
- **VPN Access**: Secure remote access through VPN
- **Firewall Rules**: Strict network access controls
- **Rate Limiting**: API and connection rate limiting

### Data Protection
- **Encryption at Rest**: Database encryption with key rotation
- **Sensitive Data Masking**: PII and trading data protection
- **Audit Logging**: Complete audit trail for all operations
- **Backup Security**: Encrypted backups with offsite storage

## Monitoring & Observability

### Metrics Collection
- **Prometheus**: System and application metrics
- **Custom Metrics**: Trading-specific performance indicators
- **Real-time Dashboards**: Grafana visualizations
- **Alerting**: Automated alerts for system anomalies

### Logging
- **Structured Logging**: JSON-formatted logs with correlation IDs
- **Log Aggregation**: Centralized logging with ELK stack
- **Log Retention**: Configurable retention policies
- **Sensitive Data**: Automatic scrubbing of sensitive information

### Distributed Tracing
- **Jaeger Integration**: End-to-end request tracing
- **Span Collection**: Detailed operation timing
- **Correlation**: Request correlation across services
- **Performance Analysis**: Bottleneck identification

## Deployment Architecture

### Production Environment
```
┌─────────────────────────────────────────────────────────────────┐
│                        Load Balancer                           │
├─────────────────────────────────────────────────────────────────┤
│  Trading Servers (Dedicated Hardware)                          │
│  ├── Core 0-1: OS + System                                     │
│  ├── Core 2-3: Trading Engine (Real-time)                      │
│  ├── Core 4-5: Risk Management                                 │
│  ├── Core 6-7: ML Inference                                    │
│  └── Core 8+: Data Processing                                  │
├─────────────────────────────────────────────────────────────────┤
│  Database Cluster                                              │
│  ├── PostgreSQL Primary/Replica                                │
│  ├── InfluxDB Cluster                                          │
│  └── Redis Cluster                                             │
├─────────────────────────────────────────────────────────────────┤
│  Monitoring & Management                                        │
│  ├── Prometheus + Grafana                                      │
│  ├── ELK Stack                                                 │
│  └── Jaeger Tracing                                            │
└─────────────────────────────────────────────────────────────────┘
```

### Hardware Requirements

#### Trading Servers
- **CPU**: Intel Xeon or AMD EPYC with AVX-512 support
- **Memory**: 64GB+ DDR4-3200 or faster
- **Storage**: NVMe SSD for logs, network for data
- **Network**: 10Gbps+ low-latency network
- **OS**: Ubuntu 22.04 LTS with real-time kernel

#### Database Servers
- **CPU**: High core count processors
- **Memory**: 128GB+ for large datasets
- **Storage**: SSD/NVMe for performance
- **Network**: High bandwidth for replication

## Scalability Considerations

### Horizontal Scaling
- **Microservice Architecture**: Independent scaling of components
- **Load Balancing**: Distribute load across multiple instances
- **Database Sharding**: Distribute data across multiple nodes
- **Caching Strategy**: Reduce database load with intelligent caching

### Vertical Scaling
- **CPU Optimization**: Leverage all available cores efficiently
- **Memory Management**: Minimize allocations and GC pressure
- **I/O Optimization**: Async I/O and connection pooling
- **Network Optimization**: Kernel bypass and DPDK integration

## Disaster Recovery

### Backup Strategy
- **Automated Backups**: Daily full backups, hourly incrementals
- **Cross-Region Replication**: Real-time data replication
- **Point-in-Time Recovery**: Restore to any point in time
- **Backup Testing**: Regular restore testing and validation

### Failover Procedures
- **Automatic Failover**: Database and application failover
- **Manual Procedures**: Step-by-step recovery instructions
- **Communication Plan**: Stakeholder notification procedures
- **Testing Schedule**: Regular disaster recovery drills

### Business Continuity
- **Recovery Time Objective (RTO)**: <15 minutes
- **Recovery Point Objective (RPO)**: <5 minutes data loss
- **Alternative Sites**: Secondary data center capability
- **Emergency Procedures**: Immediate response protocols

## Performance Tuning

### System Optimization
- **Kernel Parameters**: Network and memory tuning
- **CPU Scheduling**: Real-time scheduling for critical threads
- **Memory Management**: Large pages and NUMA awareness
- **Network Tuning**: Buffer sizes and interrupt handling

### Application Optimization
- **Profile-Guided Optimization**: CPU-specific optimizations
- **Memory Pool Management**: Pre-allocated memory pools
- **Lock-Free Algorithms**: Avoid synchronization overhead
- **SIMD Utilization**: Maximize vectorization opportunities

### Monitoring & Profiling
- **Continuous Profiling**: Always-on performance profiling
- **Bottleneck Detection**: Automated performance analysis
- **Regression Testing**: Performance regression detection
- **Capacity Planning**: Proactive scaling decisions

This architecture provides a robust, scalable, and high-performance foundation for institutional-grade high-frequency trading operations while maintaining strict risk controls and regulatory compliance.