foxhunt/docs/CI_CD_PIPELINE_GUIDE.md

Foxhunt HFT CI/CD Pipeline Guide

Overview

This document provides comprehensive guidance for the Foxhunt HFT Trading System CI/CD pipeline, designed specifically for high-frequency trading environments with strict performance, security, and compliance requirements.

Architecture Overview

Pipeline Components

1. GitHub Actions Workflow - Automated CI/CD orchestration
2. Security Scanning - cargo auditable and cargo geiger integration
3. Performance Validation - HFT latency and throughput verification
4. Blue-Green Deployment - Zero-downtime production releases
5. Canary Traffic Splitting - Risk-controlled rollouts with 1% initial traffic
6. Compliance Reporting - Regulatory audit trail generation
7. Emergency Rollback - Rapid recovery mechanisms

Deployment Strategies

Canary Deployment (Default)

• Initial Traffic: 1% of production traffic
• Monitoring Period: 5-15 minutes
• Auto-promotion: Based on performance metrics
• Rollback: Automated on failure detection

Blue-Green Deployment

• Zero Downtime: Instant traffic switching
• Full Environment: Complete service stack deployment
• Validation: Comprehensive health checks
• Rollback: Immediate traffic reversion

Prerequisites

Infrastructure Requirements

• Operating System: Ubuntu 20.04+ or RHEL 8+
• Container Runtime: Docker 20.10+ with BuildKit
• Load Balancer: nginx 1.20+ with stream module
• Monitoring: Prometheus and Grafana stack
• Storage: 100GB+ available for releases and logs

Security Requirements

• GPG Signing: All production commits must be signed
• RBAC: Role-based access control for deployments
• Secrets Management: GitHub Secrets for sensitive data
• Network Security: VPN/private networks for production

Performance Requirements

• CPU: 16+ cores with CPU affinity support
• Memory: 32GB+ RAM for HFT workloads
• Network: 10Gbps+ low-latency networking
• Storage: NVMe SSD for sub-microsecond I/O

Configuration

Environment Variables

Set the following secrets in GitHub repository settings:

# Required secrets
GITHUB_TOKEN                 # GitHub Actions access
FOXHUNT_ALERT_WEBHOOK       # Slack/Teams webhook for alerts
POLYGON_API_KEY             # Market data API access
GRAFANA_ADMIN_PASSWORD      # Monitoring access

# Optional secrets
FOXHUNT_SSH_KEY             # Production server access
DOCKER_REGISTRY_TOKEN       # Container registry access
COMPLIANCE_WEBHOOK          # Regulatory reporting endpoint

Deployment Configuration

Edit deployment/config/production.toml:

[deployment]
strategy = "canary"           # canary, blue-green, or validate-only
environment = "production"    # staging, production
canary_percentage = 1.0       # Initial canary traffic (1-99%)

[performance]
max_latency_us = 30          # Maximum acceptable latency
min_throughput_ops = 100000  # Minimum throughput requirement
validation_timeout = 300     # Performance test duration

[monitoring]
health_check_interval = 5    # Health check frequency (seconds)
metrics_retention = 2592000  # 30 days retention
alert_threshold_cpu = 80     # CPU usage alert threshold
alert_threshold_memory = 8192 # Memory usage alert threshold (MB)

[compliance]
audit_retention_days = 2555  # 7 years for regulatory compliance
generate_reports = true      # Enable compliance reporting
digital_signatures = true   # Enable report signing

Deployment Workflows

Automatic Deployment (Production)

Triggered on push to production or production-hardening branch:

1. Security Audit - Vulnerability scanning
2. Build & Test - Compilation and test execution
3. Performance Validation - Latency/throughput verification
4. Docker Build - Container image creation
5. Production Deployment - Canary or blue-green strategy
6. Post-deployment Monitoring - Health and performance verification
7. Compliance Reporting - Regulatory documentation

Manual Deployment

Use GitHub Actions workflow dispatch:

# Navigate to Actions tab in GitHub
# Select "Foxhunt HFT CI/CD Pipeline"
# Click "Run workflow"
# Configure parameters:
#   - Deployment strategy: canary/blue-green/validate-only
#   - Environment: staging/production
#   - Canary percentage: 1-100

Emergency Procedures

Emergency Rollback

For critical production issues:

# On production server
sudo /opt/foxhunt/deployment/scripts/emergency-rollback.sh \
  --reason "Critical latency spike detected" \
  --force

Emergency Stop

To immediately halt all trading operations:

# Stop all services
for service in foxhunt-core foxhunt-tli foxhunt-ml foxhunt-risk foxhunt-data; do
  sudo systemctl stop $service
done

# Verify all stopped
sudo systemctl status foxhunt-*

Monitoring and Alerting

Real-time Monitoring

The deployment includes comprehensive monitoring:

• Service Health: HTTP health checks every 5 seconds
• Performance Metrics: Latency and throughput tracking
• System Resources: CPU, memory, disk, and network monitoring
• Business Metrics: Order processing and risk calculations

Alert Thresholds

Critical Alerts (Immediate Response)

• Any service failure
• Latency > 100μs sustained
• CPU > 95% for 5+ minutes
• Memory > 90% usage
• Disk > 95% full

Warning Alerts (Monitor Closely)

• Latency > 50μs sustained
• Throughput < 50% of baseline
• CPU > 80% for 10+ minutes
• Error rate > 1%

Monitoring Commands

# Real-time deployment monitoring
/opt/foxhunt/deployment/scripts/deployment-monitoring.sh --duration 300

# Generate status report
/opt/foxhunt/deployment/scripts/deployment-monitoring.sh --report-only

# Check specific service
curl -f http://localhost:8080/health
curl -s http://localhost:8080/metrics | grep latency

Performance Validation

Automated Performance Tests

The pipeline includes automated performance validation:

# Run performance benchmarks
cargo bench --workspace

# Validate against thresholds
python3 scripts/validate-performance.py benchmark-results.txt

Performance Thresholds

Component	Max Latency	Min Throughput	Max Std Dev
Trading Engine	30μs	100,000 ops/sec	10μs
Order Processing	25μs	150,000 ops/sec	8μs
Risk Calculations	20μs	200,000 ops/sec	5μs
ML Inference	50μs	50,000 ops/sec	20μs

Performance Monitoring

# Real-time latency monitoring
watch -n 1 'curl -s http://localhost:8080/metrics | grep -E "(latency|throughput)"'

# Historical performance analysis
python3 scripts/analyze-performance-trends.py /var/log/foxhunt/performance/

Security and Compliance

Security Scanning

Automated security scans include:

1. cargo audit - Known vulnerability scanning
2. cargo geiger - Unsafe code detection
3. Docker security - Container vulnerability scanning
4. Dependency audit - Third-party package security

Compliance Features

• Audit Trail: Complete deployment history
• Digital Signatures: Cryptographic integrity verification
• Change Control: Automated change management records
• Regulatory Reporting: SOC2, ISO 27001, MiFID II compliance

Compliance Reports

# Generate compliance report
python3 scripts/generate-compliance-report.py \
  --sha $(git rev-parse HEAD) \
  --status success \
  --output compliance-report.json

Troubleshooting

Common Issues

Deployment Failures

Symptom: Pipeline fails at deployment stage Diagnosis: Check deployment logs

tail -f /home/jgrusewski/Work/foxhunt/logs/deployment-*.log

Resolution: Verify service health and rollback if necessary

Performance Validation Failures

Symptom: Latency thresholds exceeded Diagnosis: Check system resources and service metrics

/opt/foxhunt/deployment/scripts/deployment-monitoring.sh --once

Resolution: Investigate resource bottlenecks or consider rollback

Health Check Failures

Symptom: Services fail health checks Diagnosis: Check service logs and configuration

journalctl -u foxhunt-core -f
curl -v http://localhost:8080/health

Resolution: Fix service configuration or dependencies

Log Locations

• Deployment Logs: /home/jgrusewski/Work/foxhunt/logs/
• Service Logs: /var/log/foxhunt/
• Nginx Logs: /var/log/nginx/foxhunt_*.log
• System Logs: journalctl -u foxhunt-*

Recovery Procedures

Full System Recovery

1. Stop all services
2. Identify last known good version
3. Execute emergency rollback
4. Validate system health
5. Notify stakeholders

# Emergency recovery script
sudo /opt/foxhunt/deployment/scripts/emergency-rollback.sh \
  --reason "Full system recovery" \
  --force

Maintenance

Regular Maintenance Tasks

Daily

• Review deployment logs
• Check performance metrics
• Validate backup integrity

Weekly

• Update security dependencies
• Review compliance reports
• Test rollback procedures

Monthly

• Performance baseline updates
• Security audit review
• Disaster recovery testing

Capacity Planning

Monitor these metrics for capacity planning:

• CPU utilization trends
• Memory usage patterns
• Network I/O growth
• Storage utilization
• Request volume trends

Support and Escalation

Support Tiers

1. L1 Support: Basic monitoring and health checks
2. L2 Support: Performance analysis and configuration
3. L3 Support: Architecture changes and emergency response

Escalation Procedures

Severity 1 (Critical)

• Response Time: 15 minutes
• Resolution Time: 1 hour
• Notification: Immediate alert to on-call team

Severity 2 (High)

• Response Time: 1 hour
• Resolution Time: 4 hours
• Notification: Standard alert channels

Severity 3 (Medium)

• Response Time: 4 hours
• Resolution Time: 24 hours
• Notification: Standard queues

Contact Information

• Emergency Hotline: Available 24/7 for Severity 1 issues
• Slack Channel: #foxhunt-ops for real-time communication
• Email: foxhunt-ops@company.com for non-urgent issues

---

Document Version: 1.0
Last Updated: 2025-01-21
Review Schedule: Quarterly
Owner: DevOps Team