This deployment infrastructure provides production-ready deployment automation for the Foxhunt HFT trading system with sub-30μs latency requirements and zero-downtime deployment capabilities.

Architecture

Production Deployment Architecture:
├── SystemD Services (Hardware Optimized)
├── Docker Compose (Development Environment)
├── Ansible Automation (Production Deployment)
├── Zero-Downtime Scripts (Canary Deployment)
├── Health Monitoring (Prometheus/Grafana)
├── Log Aggregation (Loki/Custom Pipeline)
└── Emergency Rollback (Sub-30s Recovery)

Quick Start

Development Environment

# Start development environment
cd deployment/docker
docker-compose up -d

# With performance profiling
docker-compose -f docker-compose.yml -f docker-compose.profiling.yml up -d

# View logs
docker-compose logs -f foxhunt-core-dev

# Stop environment
docker-compose down -v

Production Deployment

# Deploy to production using Ansible
cd deployment/ansible
ansible-playbook -i inventory/production deploy-foxhunt.yml -e version=v1.2.3

# Zero-downtime deployment
cd deployment/scripts
./zero-downtime-deploy.sh v1.2.3

# Validate deployment
./production-validation.sh

# Emergency rollback if needed
./emergency-rollback.sh

Directory Structure

deployment/
├── systemd/                    # SystemD service templates
│   ├── foxhunt-core.service    # Core trading service
│   ├── foxhunt-tli.service     # Trading Layer Interface
│   ├── foxhunt-ml.service      # ML services
│   ├── foxhunt-risk.service    # Risk management
│   └── foxhunt-data.service    # Data service
├── docker/                     # Development environment
│   ├── docker-compose.yml      # Main development stack
│   ├── docker-compose.profiling.yml  # Performance testing
│   ├── config/dev.toml         # Development configuration
│   └── secrets/                # API keys and secrets
├── ansible/                    # Production automation
│   ├── deploy-foxhunt.yml      # Main deployment playbook
│   ├── tasks/                  # Task files
│   └── roles/                  # Ansible roles
├── scripts/                    # Deployment scripts
│   ├── zero-downtime-deploy.sh      # Zero-downtime deployment
│   ├── emergency-rollback.sh        # Emergency rollback
│   ├── production-validation.sh     # Deployment validation
│   ├── automated-deployment-tests.sh # Deployment testing
│   └── log-pipeline.sh              # Log aggregation
└── monitoring/                 # Monitoring configuration
    ├── prometheus.yml          # Prometheus config
    ├── loki-config.yml         # Loki config
    ├── alerts/                 # Alert rules
    └── grafana/                # Grafana datasources

SystemD Services

Hardware Optimizations

Each service is configured with specific CPU affinity and performance optimizations:

foxhunt-core: CPU cores 2-5, real-time scheduling (FIFO, priority 90)
foxhunt-tli: CPU cores 0-1, highest priority (FIFO, priority 95)
foxhunt-ml: CPU cores 6-9, GPU access, batch scheduling
foxhunt-risk: CPU cores 10-11, real-time scheduling (FIFO, priority 85)
foxhunt-data: CPU cores 12-15, I/O optimized

Installation

# Copy service files
sudo cp deployment/systemd/*.service /etc/systemd/system/

# Reload systemd
sudo systemctl daemon-reload

# Enable services
sudo systemctl enable foxhunt-core foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data

# Start services
sudo systemctl start foxhunt-core

Zero-Downtime Deployment

Canary Deployment Strategy

The deployment process follows a hardware-aware canary approach:

Phase 1: Deploy to standby hardware first
Phase 2: Performance validation with real market data
Phase 3: Traffic cutover using load balancer
Phase 4: Monitor latency for 5 minutes before declaring success

Usage

# Standard canary deployment
./zero-downtime-deploy.sh v1.2.3

# Blue-green deployment (when implemented)
./zero-downtime-deploy.sh v1.2.3 --strategy blue-green

# Validation only (dry run)
./zero-downtime-deploy.sh v1.2.3 --validate-only

# Skip performance validation (emergency)
./zero-downtime-deploy.sh v1.2.3 --skip-performance

Emergency Rollback

Critical requirement: Complete rollback in under 30 seconds.

Automatic Rollback Triggers

Order latency > 30μs for more than 1 second
Service health check failures
Performance degradation below thresholds

Manual Rollback

# Emergency rollback to last known good version
./emergency-rollback.sh

# Validate rollback prerequisites only
./emergency-rollback.sh --validate-only

# Force rollback without confirmations
./emergency-rollback.sh --force

Monitoring & Alerting

Key Metrics

Latency: Order processing latency (target: <30μs)
Throughput: Orders processed per minute (target: >1000)
Health: Service availability and responsiveness
Resources: CPU, memory, network utilization

Alert Thresholds

Critical Alerts:
- Order latency > 30μs
- Service down > 5s
- High error rate > 1%

Warning Alerts:
- Throughput < 1000 ops/min
- Memory usage > 80%
- Data latency > 100ms

Accessing Monitoring

# Prometheus metrics
curl http://localhost:9090

# Grafana dashboards
http://localhost:3000 (admin/admin)

# Service health endpoints
curl http://localhost:8080/health  # Core
curl http://localhost:8081/health  # TLI
curl http://localhost:8082/health  # ML
curl http://localhost:8083/health  # Risk
curl http://localhost:8084/health  # Data

Log Aggregation

Real-Time Log Processing

The log pipeline extracts metrics and forwards logs to centralized storage:

Latency extraction: Automatic detection and forwarding
Trading metrics: Order execution tracking
Error detection: Automatic error classification
Performance monitoring: System resource tracking

Usage

# Start log aggregation pipeline
./log-pipeline.sh

# View aggregated logs
curl http://localhost:3100/loki/api/v1/query?query={service="foxhunt-core"}

Performance Optimization

Hardware Requirements

CPU: 16+ cores with isolation support
Memory: 32GB+ with hugepage support
Network: Low-latency network interface
Storage: NVMe SSD for logs and data

System Tuning

The Ansible playbooks automatically configure:

CPU isolation (isolcpus, nohz_full, rcu_nocbs)
Hugepages (2MB pages, 1024 pages = 2GB)
Network IRQ affinity
NUMA topology awareness
Memory locking limits
CPU governor (performance mode)

Security

Service Hardening

Process isolation: Dedicated user/group
File permissions: Restricted access to binaries and configs
Network security: Service-specific port binding
Memory protection: No new privileges, protected directories

Secrets Management

# API keys stored in secure locations
/opt/foxhunt/secrets/databento.key   # Databento API key
/opt/foxhunt/secrets/benzinga.key    # Benzinga Pro API key
/opt/foxhunt/secrets/ib-creds.enc   # Interactive Brokers credentials

Testing

Automated Deployment Tests

# Run comprehensive deployment tests
./automated-deployment-tests.sh

# Test specific components
docker-compose config                    # Docker configuration
ansible-playbook --syntax-check *.yml   # Ansible syntax
bash -n *.sh                            # Script syntax

Performance Validation

# Full production validation
./production-validation.sh

# Quick health check
curl -f http://localhost:8080/health && echo "✓ Healthy"

Troubleshooting

Common Issues

High Latency

# Check CPU affinity
taskset -p $(pgrep foxhunt-core)

# Verify hugepages
cat /proc/meminfo | grep Huge

# Check system load
htop

Service Start Failures

# Check service logs
journalctl -u foxhunt-core -f

# Verify binary permissions
ls -la /opt/foxhunt/bin/

# Check configuration
foxhunt-core --validate-config

Deployment Failures

# Check deployment logs
tail -f /var/log/foxhunt/deployment-*.log

# Verify prerequisites
./zero-downtime-deploy.sh --validate-only

# Emergency rollback
./emergency-rollback.sh

Log Locations

/var/log/foxhunt/foxhunt.log        # Application logs
/var/log/foxhunt/deployment-*.log   # Deployment logs
/var/log/foxhunt/rollback-*.log     # Rollback logs
/var/log/foxhunt/validation-*.log   # Validation logs

Production Checklist

Before deploying to production:

Hardware optimizations configured
SystemD services tested
Monitoring dashboards configured
Alert rules validated
Emergency rollback tested
Performance thresholds validated
Security hardening applied
Backup procedures tested
Documentation updated
Team training completed

Support

For deployment issues:

Check the relevant log files
Run the validation scripts
Consult the troubleshooting section
Review monitoring dashboards
Consider emergency rollback if needed

SUCCESS CRITERIA:

Sub-30μs latency maintained through deployments
Zero trading downtime during updates
<30-second emergency rollback capability
Complete audit trail of all operations