foxhunt/deployment

Foxhunt HFT Trading System - Production Deployment

Overview

This deployment infrastructure provides production-ready deployment automation for the Foxhunt HFT trading system with sub-30μs latency requirements and zero-downtime deployment capabilities.

Architecture

Production Deployment Architecture:
├── SystemD Services (Hardware Optimized)
├── Docker Compose (Development Environment)
├── Ansible Automation (Production Deployment)
├── Zero-Downtime Scripts (Canary Deployment)
├── Health Monitoring (Prometheus/Grafana)
├── Log Aggregation (Loki/Custom Pipeline)
└── Emergency Rollback (Sub-30s Recovery)

Quick Start

Development Environment

# Start development environment
cd deployment/docker
docker-compose up -d

# With performance profiling
docker-compose -f docker-compose.yml -f docker-compose.profiling.yml up -d

# View logs
docker-compose logs -f foxhunt-core-dev

# Stop environment
docker-compose down -v

Production Deployment

# Deploy to production using Ansible
cd deployment/ansible
ansible-playbook -i inventory/production deploy-foxhunt.yml -e version=v1.2.3

# Zero-downtime deployment
cd deployment/scripts
./zero-downtime-deploy.sh v1.2.3

# Validate deployment
./production-validation.sh

# Emergency rollback if needed
./emergency-rollback.sh

Directory Structure

deployment/
├── systemd/                    # SystemD service templates
│   ├── foxhunt-core.service    # Core trading service
│   ├── foxhunt-tli.service     # Trading Layer Interface
│   ├── foxhunt-ml.service      # ML services
│   ├── foxhunt-risk.service    # Risk management
│   └── foxhunt-data.service    # Data service
├── docker/                     # Development environment
│   ├── docker-compose.yml      # Main development stack
│   ├── docker-compose.profiling.yml  # Performance testing
│   ├── config/dev.toml         # Development configuration
│   └── secrets/                # API keys and secrets
├── ansible/                    # Production automation
│   ├── deploy-foxhunt.yml      # Main deployment playbook
│   ├── tasks/                  # Task files
│   └── roles/                  # Ansible roles
├── scripts/                    # Deployment scripts
│   ├── zero-downtime-deploy.sh      # Zero-downtime deployment
│   ├── emergency-rollback.sh        # Emergency rollback
│   ├── production-validation.sh     # Deployment validation
│   ├── automated-deployment-tests.sh # Deployment testing
│   └── log-pipeline.sh              # Log aggregation
└── monitoring/                 # Monitoring configuration
    ├── prometheus.yml          # Prometheus config
    ├── loki-config.yml         # Loki config
    ├── alerts/                 # Alert rules
    └── grafana/                # Grafana datasources

SystemD Services

Hardware Optimizations

Each service is configured with specific CPU affinity and performance optimizations:

• foxhunt-core: CPU cores 2-5, real-time scheduling (FIFO, priority 90)
• foxhunt-tli: CPU cores 0-1, highest priority (FIFO, priority 95)
• foxhunt-ml: CPU cores 6-9, GPU access, batch scheduling
• foxhunt-risk: CPU cores 10-11, real-time scheduling (FIFO, priority 85)
• foxhunt-data: CPU cores 12-15, I/O optimized

Installation

# Copy service files
sudo cp deployment/systemd/*.service /etc/systemd/system/

# Reload systemd
sudo systemctl daemon-reload

# Enable services
sudo systemctl enable foxhunt-core foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data

# Start services
sudo systemctl start foxhunt-core

Zero-Downtime Deployment

Canary Deployment Strategy

The deployment process follows a hardware-aware canary approach:

1. Phase 1: Deploy to standby hardware first
2. Phase 2: Performance validation with real market data
3. Phase 3: Traffic cutover using load balancer
4. Phase 4: Monitor latency for 5 minutes before declaring success

Usage

# Standard canary deployment
./zero-downtime-deploy.sh v1.2.3

# Blue-green deployment (when implemented)
./zero-downtime-deploy.sh v1.2.3 --strategy blue-green

# Validation only (dry run)
./zero-downtime-deploy.sh v1.2.3 --validate-only

# Skip performance validation (emergency)
./zero-downtime-deploy.sh v1.2.3 --skip-performance

Emergency Rollback

Critical requirement: Complete rollback in under 30 seconds.

Automatic Rollback Triggers

• Order latency > 30μs for more than 1 second
• Service health check failures
• Performance degradation below thresholds

Manual Rollback

# Emergency rollback to last known good version
./emergency-rollback.sh

# Validate rollback prerequisites only
./emergency-rollback.sh --validate-only

# Force rollback without confirmations
./emergency-rollback.sh --force

Monitoring & Alerting

Key Metrics

• Latency: Order processing latency (target: <30μs)
• Throughput: Orders processed per minute (target: >1000)
• Health: Service availability and responsiveness
• Resources: CPU, memory, network utilization

Alert Thresholds

Critical Alerts:
- Order latency > 30μs
- Service down > 5s
- High error rate > 1%

Warning Alerts:
- Throughput < 1000 ops/min
- Memory usage > 80%
- Data latency > 100ms

Accessing Monitoring

# Prometheus metrics
curl http://localhost:9090

# Grafana dashboards
http://localhost:3000 (admin/admin)

# Service health endpoints
curl http://localhost:8080/health  # Core
curl http://localhost:8081/health  # TLI
curl http://localhost:8082/health  # ML
curl http://localhost:8083/health  # Risk
curl http://localhost:8084/health  # Data

Log Aggregation

Real-Time Log Processing

The log pipeline extracts metrics and forwards logs to centralized storage:

• Latency extraction: Automatic detection and forwarding
• Trading metrics: Order execution tracking
• Error detection: Automatic error classification
• Performance monitoring: System resource tracking

Usage

# Start log aggregation pipeline
./log-pipeline.sh

# View aggregated logs
curl http://localhost:3100/loki/api/v1/query?query={service="foxhunt-core"}

Performance Optimization

Hardware Requirements

• CPU: 16+ cores with isolation support
• Memory: 32GB+ with hugepage support
• Network: Low-latency network interface
• Storage: NVMe SSD for logs and data

System Tuning

The Ansible playbooks automatically configure:

• CPU isolation (isolcpus, nohz_full, rcu_nocbs)
• Hugepages (2MB pages, 1024 pages = 2GB)
• Network IRQ affinity
• NUMA topology awareness
• Memory locking limits
• CPU governor (performance mode)

Security

Service Hardening

• Process isolation: Dedicated user/group
• File permissions: Restricted access to binaries and configs
• Network security: Service-specific port binding
• Memory protection: No new privileges, protected directories

Secrets Management

# API keys stored in secure locations
/opt/foxhunt/secrets/databento.key   # Databento API key
/opt/foxhunt/secrets/benzinga.key    # Benzinga Pro API key
/opt/foxhunt/secrets/ib-creds.enc   # Interactive Brokers credentials

Testing

Automated Deployment Tests

# Run comprehensive deployment tests
./automated-deployment-tests.sh

# Test specific components
docker-compose config                    # Docker configuration
ansible-playbook --syntax-check *.yml   # Ansible syntax
bash -n *.sh                            # Script syntax

Performance Validation

# Full production validation
./production-validation.sh

# Quick health check
curl -f http://localhost:8080/health && echo "✓ Healthy"

Troubleshooting

Common Issues

1. High Latency

   # Check CPU affinity
   taskset -p $(pgrep foxhunt-core)
   
   # Verify hugepages
   cat /proc/meminfo | grep Huge
   
   # Check system load
   htop
   

2. Service Start Failures

   # Check service logs
   journalctl -u foxhunt-core -f
   
   # Verify binary permissions
   ls -la /opt/foxhunt/bin/
   
   # Check configuration
   foxhunt-core --validate-config
   

3. Deployment Failures

   # Check deployment logs
   tail -f /var/log/foxhunt/deployment-*.log
   
   # Verify prerequisites
   ./zero-downtime-deploy.sh --validate-only
   
   # Emergency rollback
   ./emergency-rollback.sh
   

Log Locations

/var/log/foxhunt/foxhunt.log        # Application logs
/var/log/foxhunt/deployment-*.log   # Deployment logs
/var/log/foxhunt/rollback-*.log     # Rollback logs
/var/log/foxhunt/validation-*.log   # Validation logs

Production Checklist

Before deploying to production:

• Hardware optimizations configured
• SystemD services tested
• Monitoring dashboards configured
• Alert rules validated
• Emergency rollback tested
• Performance thresholds validated
• Security hardening applied
• Backup procedures tested
• Documentation updated
• Team training completed

Support

For deployment issues:

1. Check the relevant log files
2. Run the validation scripts
3. Consult the troubleshooting section
4. Review monitoring dashboards
5. Consider emergency rollback if needed

---

SUCCESS CRITERIA:

• Sub-30μs latency maintained through deployments
• Zero trading downtime during updates
• <30-second emergency rollback capability
• Complete audit trail of all operations