foxhunt/docs/CI_CD_PIPELINE_GUIDE.md

# Foxhunt HFT CI/CD Pipeline Guide

## Overview

This document provides comprehensive guidance for the Foxhunt HFT Trading System CI/CD pipeline, designed specifically for high-frequency trading environments with strict performance, security, and compliance requirements.

## Architecture Overview

### Pipeline Components

1. **GitHub Actions Workflow** - Automated CI/CD orchestration
2. **Security Scanning** - cargo auditable and cargo geiger integration
3. **Performance Validation** - HFT latency and throughput verification
4. **Blue-Green Deployment** - Zero-downtime production releases
5. **Canary Traffic Splitting** - Risk-controlled rollouts with 1% initial traffic
6. **Compliance Reporting** - Regulatory audit trail generation
7. **Emergency Rollback** - Rapid recovery mechanisms

### Deployment Strategies

#### Canary Deployment (Default)
- **Initial Traffic**: 1% of production traffic
- **Monitoring Period**: 5-15 minutes
- **Auto-promotion**: Based on performance metrics
- **Rollback**: Automated on failure detection

#### Blue-Green Deployment
- **Zero Downtime**: Instant traffic switching
- **Full Environment**: Complete service stack deployment
- **Validation**: Comprehensive health checks
- **Rollback**: Immediate traffic reversion

## Prerequisites

### Infrastructure Requirements

- **Operating System**: Ubuntu 20.04+ or RHEL 8+
- **Container Runtime**: Docker 20.10+ with BuildKit
- **Load Balancer**: nginx 1.20+ with stream module
- **Monitoring**: Prometheus and Grafana stack
- **Storage**: 100GB+ available for releases and logs

### Security Requirements

- **GPG Signing**: All production commits must be signed
- **RBAC**: Role-based access control for deployments
- **Secrets Management**: GitHub Secrets for sensitive data
- **Network Security**: VPN/private networks for production

### Performance Requirements

- **CPU**: 16+ cores with CPU affinity support
- **Memory**: 32GB+ RAM for HFT workloads
- **Network**: 10Gbps+ low-latency networking
- **Storage**: NVMe SSD for sub-microsecond I/O

## Configuration

### Environment Variables

Set the following secrets in GitHub repository settings:

```bash
# Required secrets
GITHUB_TOKEN                 # GitHub Actions access
FOXHUNT_ALERT_WEBHOOK       # Slack/Teams webhook for alerts
POLYGON_API_KEY             # Market data API access
GRAFANA_ADMIN_PASSWORD      # Monitoring access

# Optional secrets
FOXHUNT_SSH_KEY             # Production server access
DOCKER_REGISTRY_TOKEN       # Container registry access
COMPLIANCE_WEBHOOK          # Regulatory reporting endpoint
```

### Deployment Configuration

Edit `deployment/config/production.toml`:

```toml
[deployment]
strategy = "canary"           # canary, blue-green, or validate-only
environment = "production"    # staging, production
canary_percentage = 1.0       # Initial canary traffic (1-99%)

[performance]
max_latency_us = 30          # Maximum acceptable latency
min_throughput_ops = 100000  # Minimum throughput requirement
validation_timeout = 300     # Performance test duration

[monitoring]
health_check_interval = 5    # Health check frequency (seconds)
metrics_retention = 2592000  # 30 days retention
alert_threshold_cpu = 80     # CPU usage alert threshold
alert_threshold_memory = 8192 # Memory usage alert threshold (MB)

[compliance]
audit_retention_days = 2555  # 7 years for regulatory compliance
generate_reports = true      # Enable compliance reporting
digital_signatures = true   # Enable report signing
```

## Deployment Workflows

### Automatic Deployment (Production)

Triggered on push to `production` or `production-hardening` branch:

1. **Security Audit** - Vulnerability scanning
2. **Build & Test** - Compilation and test execution
3. **Performance Validation** - Latency/throughput verification
4. **Docker Build** - Container image creation
5. **Production Deployment** - Canary or blue-green strategy
6. **Post-deployment Monitoring** - Health and performance verification
7. **Compliance Reporting** - Regulatory documentation

### Manual Deployment

Use GitHub Actions workflow dispatch:

```bash
# Navigate to Actions tab in GitHub
# Select "Foxhunt HFT CI/CD Pipeline"
# Click "Run workflow"
# Configure parameters:
#   - Deployment strategy: canary/blue-green/validate-only
#   - Environment: staging/production
#   - Canary percentage: 1-100
```

### Emergency Procedures

#### Emergency Rollback

For critical production issues:

```bash
# On production server
sudo /opt/foxhunt/deployment/scripts/emergency-rollback.sh \
  --reason "Critical latency spike detected" \
  --force
```

#### Emergency Stop

To immediately halt all trading operations:

```bash
# Stop all services
for service in foxhunt-core foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data; do
  sudo systemctl stop $service
done

# Verify all stopped
sudo systemctl status foxhunt-*
```

## Monitoring and Alerting

### Real-time Monitoring

The deployment includes comprehensive monitoring:

- **Service Health**: HTTP health checks every 5 seconds
- **Performance Metrics**: Latency and throughput tracking
- **System Resources**: CPU, memory, disk, and network monitoring
- **Business Metrics**: Order processing and risk calculations

### Alert Thresholds

#### Critical Alerts (Immediate Response)
- Any service failure
- Latency > 100μs sustained
- CPU > 95% for 5+ minutes
- Memory > 90% usage
- Disk > 95% full

#### Warning Alerts (Monitor Closely)
- Latency > 50μs sustained
- Throughput < 50% of baseline
- CPU > 80% for 10+ minutes
- Error rate > 1%

### Monitoring Commands

```bash
# Real-time deployment monitoring
/opt/foxhunt/deployment/scripts/deployment-monitoring.sh --duration 300

# Generate status report
/opt/foxhunt/deployment/scripts/deployment-monitoring.sh --report-only

# Check specific service
curl -f http://localhost:8080/health
curl -s http://localhost:8080/metrics | grep latency
```

## Performance Validation

### Automated Performance Tests

The pipeline includes automated performance validation:

```bash
# Run performance benchmarks
cargo bench --workspace

# Validate against thresholds
python3 scripts/validate-performance.py benchmark-results.txt
```

### Performance Thresholds

| Component | Max Latency | Min Throughput | Max Std Dev |
|-----------|-------------|----------------|-------------|
| Trading Engine | 30μs | 100,000 ops/sec | 10μs |
| Order Processing | 25μs | 150,000 ops/sec | 8μs |
| Risk Calculations | 20μs | 200,000 ops/sec | 5μs |
| ML Inference | 50μs | 50,000 ops/sec | 20μs |

### Performance Monitoring

```bash
# Real-time latency monitoring
watch -n 1 'curl -s http://localhost:8080/metrics | grep -E "(latency|throughput)"'

# Historical performance analysis
python3 scripts/analyze-performance-trends.py /var/log/foxhunt/performance/
```

## Security and Compliance

### Security Scanning

Automated security scans include:

1. **cargo audit** - Known vulnerability scanning
2. **cargo geiger** - Unsafe code detection
3. **Docker security** - Container vulnerability scanning
4. **Dependency audit** - Third-party package security

### Compliance Features

- **Audit Trail**: Complete deployment history
- **Digital Signatures**: Cryptographic integrity verification
- **Change Control**: Automated change management records
- **Regulatory Reporting**: SOC2, ISO 27001, MiFID II compliance

### Compliance Reports

```bash
# Generate compliance report
python3 scripts/generate-compliance-report.py \
  --sha $(git rev-parse HEAD) \
  --status success \
  --output compliance-report.json
```

## Troubleshooting

### Common Issues

#### Deployment Failures

**Symptom**: Pipeline fails at deployment stage
**Diagnosis**: Check deployment logs
```bash
tail -f /home/jgrusewski/Work/foxhunt/logs/deployment-*.log
```
**Resolution**: Verify service health and rollback if necessary

#### Performance Validation Failures

**Symptom**: Latency thresholds exceeded
**Diagnosis**: Check system resources and service metrics
```bash
/opt/foxhunt/deployment/scripts/deployment-monitoring.sh --once
```
**Resolution**: Investigate resource bottlenecks or consider rollback

#### Health Check Failures

**Symptom**: Services fail health checks
**Diagnosis**: Check service logs and configuration
```bash
journalctl -u foxhunt-core -f
curl -v http://localhost:8080/health
```
**Resolution**: Fix service configuration or dependencies

### Log Locations

- **Deployment Logs**: `/home/jgrusewski/Work/foxhunt/logs/`
- **Service Logs**: `/var/log/foxhunt/`
- **Nginx Logs**: `/var/log/nginx/foxhunt_*.log`
- **System Logs**: `journalctl -u foxhunt-*`

### Recovery Procedures

#### Full System Recovery

1. Stop all services
2. Identify last known good version
3. Execute emergency rollback
4. Validate system health
5. Notify stakeholders

```bash
# Emergency recovery script
sudo /opt/foxhunt/deployment/scripts/emergency-rollback.sh \
  --reason "Full system recovery" \
  --force
```

## Maintenance

### Regular Maintenance Tasks

#### Daily
- [ ] Review deployment logs
- [ ] Check performance metrics
- [ ] Validate backup integrity

#### Weekly  
- [ ] Update security dependencies
- [ ] Review compliance reports
- [ ] Test rollback procedures

#### Monthly
- [ ] Performance baseline updates
- [ ] Security audit review
- [ ] Disaster recovery testing

### Capacity Planning

Monitor these metrics for capacity planning:

- **CPU utilization trends**
- **Memory usage patterns** 
- **Network I/O growth**
- **Storage utilization**
- **Request volume trends**

## Support and Escalation

### Support Tiers

1. **L1 Support**: Basic monitoring and health checks
2. **L2 Support**: Performance analysis and configuration
3. **L3 Support**: Architecture changes and emergency response

### Escalation Procedures

#### Severity 1 (Critical)
- **Response Time**: 15 minutes
- **Resolution Time**: 1 hour
- **Notification**: Immediate alert to on-call team

#### Severity 2 (High)
- **Response Time**: 1 hour  
- **Resolution Time**: 4 hours
- **Notification**: Standard alert channels

#### Severity 3 (Medium)
- **Response Time**: 4 hours
- **Resolution Time**: 24 hours
- **Notification**: Standard queues

### Contact Information

- **Emergency Hotline**: Available 24/7 for Severity 1 issues
- **Slack Channel**: #foxhunt-ops for real-time communication
- **Email**: foxhunt-ops@company.com for non-urgent issues

---

**Document Version**: 1.0  
**Last Updated**: 2025-01-21  
**Review Schedule**: Quarterly  
**Owner**: DevOps Team