foxhunt/deployment/README.md

# Foxhunt HFT Trading System - Production Deployment

## Overview

This deployment infrastructure provides production-ready deployment automation for the Foxhunt HFT trading system with sub-30μs latency requirements and zero-downtime deployment capabilities.

## Architecture

```
Production Deployment Architecture:
├── SystemD Services (Hardware Optimized)
├── Docker Compose (Development Environment)
├── Ansible Automation (Production Deployment)
├── Zero-Downtime Scripts (Canary Deployment)
├── Health Monitoring (Prometheus/Grafana)
├── Log Aggregation (Loki/Custom Pipeline)
└── Emergency Rollback (Sub-30s Recovery)
```

## Quick Start

### Development Environment

```bash
# Start development environment
cd deployment/docker
docker-compose up -d

# With performance profiling
docker-compose -f docker-compose.yml -f docker-compose.profiling.yml up -d

# View logs
docker-compose logs -f foxhunt-core-dev

# Stop environment
docker-compose down -v
```

### Production Deployment

```bash
# Deploy to production using Ansible
cd deployment/ansible
ansible-playbook -i inventory/production deploy-foxhunt.yml -e version=v1.2.3

# Zero-downtime deployment
cd deployment/scripts
./zero-downtime-deploy.sh v1.2.3

# Validate deployment
./production-validation.sh

# Emergency rollback if needed
./emergency-rollback.sh
```

## Directory Structure

```
deployment/
├── systemd/                    # SystemD service templates
│   ├── foxhunt-core.service    # Core trading service
│   ├── foxhunt-tli.service     # Trading Layer Interface
│   ├── foxhunt-ml.service      # ML services
│   ├── foxhunt-risk.service    # Risk management
│   └── foxhunt-data.service    # Data service
├── docker/                     # Development environment
│   ├── docker-compose.yml      # Main development stack
│   ├── docker-compose.profiling.yml  # Performance testing
│   ├── config/dev.toml         # Development configuration
│   └── secrets/                # API keys and secrets
├── ansible/                    # Production automation
│   ├── deploy-foxhunt.yml      # Main deployment playbook
│   ├── tasks/                  # Task files
│   └── roles/                  # Ansible roles
├── scripts/                    # Deployment scripts
│   ├── zero-downtime-deploy.sh      # Zero-downtime deployment
│   ├── emergency-rollback.sh        # Emergency rollback
│   ├── production-validation.sh     # Deployment validation
│   ├── automated-deployment-tests.sh # Deployment testing
│   └── log-pipeline.sh              # Log aggregation
└── monitoring/                 # Monitoring configuration
    ├── prometheus.yml          # Prometheus config
    ├── loki-config.yml         # Loki config
    ├── alerts/                 # Alert rules
    └── grafana/                # Grafana datasources
```

## SystemD Services

### Hardware Optimizations

Each service is configured with specific CPU affinity and performance optimizations:

- **foxhunt-core**: CPU cores 2-5, real-time scheduling (FIFO, priority 90)
- **foxhunt-tli**: CPU cores 0-1, highest priority (FIFO, priority 95)
- **foxhunt-ml**: CPU cores 6-9, GPU access, batch scheduling
- **foxhunt-risk**: CPU cores 10-11, real-time scheduling (FIFO, priority 85)
- **foxhunt-data**: CPU cores 12-15, I/O optimized

### Installation

```bash
# Copy service files
sudo cp deployment/systemd/*.service /etc/systemd/system/

# Reload systemd
sudo systemctl daemon-reload

# Enable services
sudo systemctl enable foxhunt-core foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data

# Start services
sudo systemctl start foxhunt-core
```

## Zero-Downtime Deployment

### Canary Deployment Strategy

The deployment process follows a hardware-aware canary approach:

1. **Phase 1**: Deploy to standby hardware first
2. **Phase 2**: Performance validation with real market data
3. **Phase 3**: Traffic cutover using load balancer
4. **Phase 4**: Monitor latency for 5 minutes before declaring success

### Usage

```bash
# Standard canary deployment
./zero-downtime-deploy.sh v1.2.3

# Blue-green deployment (when implemented)
./zero-downtime-deploy.sh v1.2.3 --strategy blue-green

# Validation only (dry run)
./zero-downtime-deploy.sh v1.2.3 --validate-only

# Skip performance validation (emergency)
./zero-downtime-deploy.sh v1.2.3 --skip-performance
```

## Emergency Rollback

Critical requirement: Complete rollback in under 30 seconds.

### Automatic Rollback Triggers

- Order latency > 30μs for more than 1 second
- Service health check failures
- Performance degradation below thresholds

### Manual Rollback

```bash
# Emergency rollback to last known good version
./emergency-rollback.sh

# Validate rollback prerequisites only
./emergency-rollback.sh --validate-only

# Force rollback without confirmations
./emergency-rollback.sh --force
```

## Monitoring & Alerting

### Key Metrics

- **Latency**: Order processing latency (target: <30μs)
- **Throughput**: Orders processed per minute (target: >1000)
- **Health**: Service availability and responsiveness
- **Resources**: CPU, memory, network utilization

### Alert Thresholds

```yaml
Critical Alerts:
- Order latency > 30μs
- Service down > 5s
- High error rate > 1%

Warning Alerts:
- Throughput < 1000 ops/min
- Memory usage > 80%
- Data latency > 100ms
```

### Accessing Monitoring

```bash
# Prometheus metrics
curl http://localhost:9090

# Grafana dashboards
http://localhost:3000 (admin/admin)

# Service health endpoints
curl http://localhost:8080/health  # Core
curl http://localhost:8081/health  # TLI
curl http://localhost:8082/health  # ML
curl http://localhost:8083/health  # Risk
curl http://localhost:8084/health  # Data
```

## Log Aggregation

### Real-Time Log Processing

The log pipeline extracts metrics and forwards logs to centralized storage:

- **Latency extraction**: Automatic detection and forwarding
- **Trading metrics**: Order execution tracking
- **Error detection**: Automatic error classification
- **Performance monitoring**: System resource tracking

### Usage

```bash
# Start log aggregation pipeline
./log-pipeline.sh

# View aggregated logs
curl http://localhost:3100/loki/api/v1/query?query={service="foxhunt-core"}
```

## Performance Optimization

### Hardware Requirements

- **CPU**: 16+ cores with isolation support
- **Memory**: 32GB+ with hugepage support
- **Network**: Low-latency network interface
- **Storage**: NVMe SSD for logs and data

### System Tuning

The Ansible playbooks automatically configure:

- CPU isolation (`isolcpus`, `nohz_full`, `rcu_nocbs`)
- Hugepages (2MB pages, 1024 pages = 2GB)
- Network IRQ affinity
- NUMA topology awareness
- Memory locking limits
- CPU governor (performance mode)

## Security

### Service Hardening

- **Process isolation**: Dedicated user/group
- **File permissions**: Restricted access to binaries and configs
- **Network security**: Service-specific port binding
- **Memory protection**: No new privileges, protected directories

### Secrets Management

```bash
# API keys stored in secure locations
/opt/foxhunt/secrets/databento.key   # Databento API key
/opt/foxhunt/secrets/benzinga.key    # Benzinga Pro API key
/opt/foxhunt/secrets/ib-creds.enc   # Interactive Brokers credentials
```

## Testing

### Automated Deployment Tests

```bash
# Run comprehensive deployment tests
./automated-deployment-tests.sh

# Test specific components
docker-compose config                    # Docker configuration
ansible-playbook --syntax-check *.yml   # Ansible syntax
bash -n *.sh                            # Script syntax
```

### Performance Validation

```bash
# Full production validation
./production-validation.sh

# Quick health check
curl -f http://localhost:8080/health && echo "✓ Healthy"
```

## Troubleshooting

### Common Issues

1. **High Latency**
   ```bash
   # Check CPU affinity
   taskset -p $(pgrep foxhunt-core)
   
   # Verify hugepages
   cat /proc/meminfo | grep Huge
   
   # Check system load
   htop
   ```

2. **Service Start Failures**
   ```bash
   # Check service logs
   journalctl -u foxhunt-core -f
   
   # Verify binary permissions
   ls -la /opt/foxhunt/bin/
   
   # Check configuration
   foxhunt-core --validate-config
   ```

3. **Deployment Failures**
   ```bash
   # Check deployment logs
   tail -f /var/log/foxhunt/deployment-*.log
   
   # Verify prerequisites
   ./zero-downtime-deploy.sh --validate-only
   
   # Emergency rollback
   ./emergency-rollback.sh
   ```

### Log Locations

```bash
/var/log/foxhunt/foxhunt.log        # Application logs
/var/log/foxhunt/deployment-*.log   # Deployment logs
/var/log/foxhunt/rollback-*.log     # Rollback logs
/var/log/foxhunt/validation-*.log   # Validation logs
```

## Production Checklist

Before deploying to production:

- [ ] Hardware optimizations configured
- [ ] SystemD services tested
- [ ] Monitoring dashboards configured
- [ ] Alert rules validated
- [ ] Emergency rollback tested
- [ ] Performance thresholds validated
- [ ] Security hardening applied
- [ ] Backup procedures tested
- [ ] Documentation updated
- [ ] Team training completed

## Support

For deployment issues:

1. Check the relevant log files
2. Run the validation scripts
3. Consult the troubleshooting section
4. Review monitoring dashboards
5. Consider emergency rollback if needed

---

**SUCCESS CRITERIA:**
- Sub-30μs latency maintained through deployments
- Zero trading downtime during updates
- <30-second emergency rollback capability
- Complete audit trail of all operations