foxhunt/CI_CD_PIPELINE.md

# CI/CD Pipeline Documentation

**Last Updated**: 2025-10-07
**Version**: 1.0.0
**Owner**: Platform Team

---

## Table of Contents

1. [Pipeline Overview](#pipeline-overview)
2. [Testing Stage](#testing-stage)
3. [Build Stage](#build-stage)
4. [Deployment Stage](#deployment-stage)
5. [Security Scanning](#security-scanning)
6. [Performance Benchmarks](#performance-benchmarks)
7. [GitOps Integration](#gitops-integration)
8. [Workflow Examples](#workflow-examples)
9. [Troubleshooting](#troubleshooting)

---

## Pipeline Overview

### Pipeline Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                         CI/CD Pipeline                           │
└─────────────────────────────────────────────────────────────────┘
                                │
                    ┌───────────┴───────────┐
                    ▼                       ▼
              ┌──────────┐           ┌──────────┐
              │   Test   │           │  Build   │
              │  Stage   │           │  Stage   │
              └────┬─────┘           └────┬─────┘
                   │                      │
         ┌─────────┴─────────┐           │
         ▼                   ▼           ▼
    ┌─────────┐         ┌─────────┐ ┌─────────┐
    │  Unit   │         │  Integ  │ │ Docker  │
    │  Tests  │         │  Tests  │ │  Build  │
    └────┬────┘         └────┬────┘ └────┬────┘
         │                   │           │
         └─────────┬─────────┘           │
                   ▼                     ▼
              ┌──────────┐         ┌──────────┐
              │ Security │         │  Deploy  │
              │   Scan   │         │  Stage   │
              └────┬─────┘         └────┬─────┘
                   │                    │
                   │      ┌─────────────┴─────────────┐
                   │      ▼             ▼             ▼
                   │  ┌──────┐     ┌─────────┐  ┌──────┐
                   │  │ Dev  │     │ Staging │  │ Prod │
                   │  └──────┘     └─────────┘  └──────┘
                   │      │             │            │
                   └──────┴─────────────┴────────────┘
                                │
                                ▼
                          ┌──────────┐
                          │ Validate │
                          │  Health  │
                          └──────────┘
```

### Pipeline Stages

| Stage | Purpose | Duration | Auto/Manual |
|-------|---------|----------|-------------|
| **Test** | Run unit, integration, and coverage tests | 5-10 min | Auto |
| **Security Scan** | Vulnerability scanning, SAST/DAST | 3-5 min | Auto |
| **Build** | Docker image building and tagging | 8-12 min | Auto |
| **Deploy Dev** | Deploy to development environment | 2-3 min | Auto |
| **Deploy Staging** | Deploy to staging environment | 3-5 min | Manual approval |
| **Performance Test** | Run automated benchmarks | 5-10 min | Auto (staging) |
| **Deploy Prod** | Deploy to production environment | 5-8 min | Manual approval |
| **Rollback** | Automated rollback on health check failure | 1-2 min | Auto (on failure) |

### Branching Strategy

```
main (production)
  │
  ├── develop (staging)
  │     │
  │     ├── feature/agent-98-ci-cd-docs
  │     ├── feature/performance-optimization
  │     └── feature/new-ml-model
  │
  └── hotfix/critical-bug-fix
```

**Branch Protection Rules**:
- **main**: Requires 2 approvals, all checks pass, no force push
- **develop**: Requires 1 approval, all checks pass
- **feature/***: No restrictions, auto-delete after merge

**Merge Strategy**:
- Feature → Develop: Squash merge
- Develop → Main: Merge commit (preserves history)
- Hotfix → Main: Cherry-pick to develop

### Automated vs Manual Gates

**Automated Gates** (block merge on failure):
- ✅ All unit tests pass (>99% pass rate required)
- ✅ Integration tests pass
- ✅ Code coverage ≥60% (enforced)
- ✅ No critical/high security vulnerabilities
- ✅ Docker build succeeds
- ✅ Linting passes (`cargo clippy`)
- ✅ Formatting passes (`cargo fmt --check`)

**Manual Gates** (require approval):
- 🔒 Deploy to staging (team lead approval)
- 🔒 Deploy to production (2 approvals: team lead + DevOps)
- 🔒 Database migrations (DBA approval)
- 🔒 Infrastructure changes (DevOps approval)

### Integration Points

**GitHub Actions**: Primary CI/CD platform
- Workflows: `.github/workflows/*.yml`
- Secrets: GitHub Secrets (encrypted)
- Artifacts: GitHub Packages / Docker Registry

**GitLab CI**: Alternative/fallback
- Configuration: `.gitlab-ci.yml`
- Runners: Kubernetes-based GitLab Runners
- Registry: GitLab Container Registry

**Kubernetes**: Deployment target
- Manifests: `k8s/*.yaml`
- Helm charts: `helm/foxhunt/`
- ArgoCD: GitOps continuous deployment

**Observability**:
- Prometheus: Metrics collection
- Grafana: Dashboards and alerts
- Loki: Log aggregation
- Jaeger: Distributed tracing

---

## Testing Stage

### Test Execution

**Unit Tests** (5-7 minutes):
```bash
# Run all unit tests with coverage
cargo llvm-cov --workspace --lcov --output-path lcov.info

# Requirements:
# - Pass rate: >99% (336/337+ tests passing)
# - Coverage: ≥60% workspace coverage
# - No panics or segfaults
```

**Integration Tests** (3-5 minutes):
```bash
# Run integration tests (requires Docker services)
docker-compose up -d postgres redis vault
cargo test --workspace --test '*' -- --test-threads=1

# Tests:
# - API Gateway E2E (services/api_gateway/tests/e2e_tests.rs)
# - Trading Service E2E (services/trading_service/tests/integration_e2e_tests.rs)
# - ML Training Service (services/ml_training_service/tests/integration_tests.rs)
# - Backtesting Service (services/backtesting_service/tests/integration_tests.rs)
```

**GPU Tests** (optional, 2-3 minutes):
```bash
# Run slow GPU tests (if CUDA available)
cargo test -p ml --lib -- --ignored --test-threads=1

# Skipped in CI if GPU not available
# Uses CPU fallback automatically
```

### Coverage Reporting

**Coverage Threshold Enforcement**:
```yaml
# .github/workflows/test.yml
- name: Check Coverage Threshold
  run: |
    COVERAGE=$(cargo llvm-cov --workspace --summary-only | grep -oP 'TOTAL.*\K[0-9.]+(?=%)')
    echo "Coverage: $COVERAGE%"
    if (( $(echo "$COVERAGE < 60.0" | bc -l) )); then
      echo "❌ Coverage $COVERAGE% below threshold 60%"
      exit 1
    fi
    echo "✅ Coverage $COVERAGE% meets threshold"
```

**Coverage Report Artifacts**:
- `lcov.info`: LCOV format (for Codecov/Coveralls)
- `coverage_report/`: HTML report (browsable)
- `coverage-summary.json`: JSON summary for badges

**Coverage Badges**:
```markdown
[![Coverage](https://img.shields.io/codecov/c/github/foxhunt/foxhunt)](https://codecov.io/gh/foxhunt/foxhunt)
```

### Test Pass Rate Requirements

| Environment | Pass Rate | Coverage | Action on Failure |
|-------------|-----------|----------|-------------------|
| **Feature Branch** | ≥99% | ≥55% | Block merge, notify author |
| **Develop** | ≥99.5% | ≥60% | Block merge, alert team |
| **Main** | 100% | ≥60% | Block merge, rollback if deployed |

**Flaky Test Handling**:
```yaml
# Retry flaky tests up to 3 times
- name: Run Tests with Retry
  uses: nick-invision/retry@v2
  with:
    timeout_minutes: 15
    max_attempts: 3
    command: cargo test --workspace
```

---

## Build Stage

### Docker Image Building

**Multi-Stage Build Strategy** (from Agent 94):
```dockerfile
# Stage 1: Build dependencies
FROM rust:1.75-slim AS chef
RUN cargo install cargo-chef

# Stage 2: Plan dependencies
FROM chef AS planner
WORKDIR /app
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# Stage 3: Build dependencies (cached layer)
FROM chef AS builder
WORKDIR /app
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

# Stage 4: Build application
COPY . .
RUN cargo build --release -p trading_service

# Stage 5: Runtime image
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates libssl3 && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/trading_service /usr/local/bin/
CMD ["trading_service"]
```

**Build Optimization**:
- Dependency caching: `cargo-chef` reduces rebuild time 80%
- Layer caching: Only rebuild changed layers
- Multi-platform: Build for `linux/amd64` and `linux/arm64`
- Image size: ~50MB final runtime image

### Image Tagging Strategy

**Tagging Scheme**:
```bash
# Git commit SHA (immutable)
foxhunt/trading-service:abc123def456

# Semantic version (tagged releases)
foxhunt/trading-service:v1.2.3

# Environment tags (mutable)
foxhunt/trading-service:latest        # main branch
foxhunt/trading-service:develop       # develop branch
foxhunt/trading-service:staging       # staging environment
foxhunt/trading-service:production    # production environment

# Full example
foxhunt/trading-service:v1.2.3-abc123def456
```

**Tagging Logic**:
```yaml
- name: Docker Metadata
  id: meta
  uses: docker/metadata-action@v5
  with:
    images: |
      foxhunt/trading-service
      ghcr.io/foxhunt/trading-service
    tags: |
      type=ref,event=branch
      type=ref,event=pr
      type=semver,pattern={{version}}
      type=semver,pattern={{major}}.{{minor}}
      type=sha,prefix={{branch}}-
```

### Image Registry

**Primary Registry: GitHub Container Registry (GHCR)**:
```bash
# Login
echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_ACTOR --password-stdin

# Push
docker push ghcr.io/foxhunt/trading-service:v1.2.3
```

**Alternative Registries**:
- **AWS ECR**: Production deployments (high availability)
- **Docker Hub**: Public images (documentation examples)
- **GitLab Registry**: Fallback/self-hosted option

**Registry Configuration**:
```yaml
# .github/workflows/build.yml
- name: Push to Multiple Registries
  run: |
    # GitHub Container Registry (primary)
    docker push ghcr.io/foxhunt/trading-service:${{ steps.meta.outputs.tags }}

    # AWS ECR (production)
    aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_REGISTRY
    docker tag ghcr.io/foxhunt/trading-service:$TAG $ECR_REGISTRY/trading-service:$TAG
    docker push $ECR_REGISTRY/trading-service:$TAG
```

### Build Artifact Storage

**Artifacts Stored**:
1. Docker images (tagged and pushed to registries)
2. Coverage reports (uploaded to Codecov)
3. Test results (JUnit XML format)
4. Build logs (GitHub Actions artifacts)
5. Binary releases (GitHub Releases for tagged versions)

**Artifact Retention**:
- **Docker images**: 90 days for non-production tags
- **Coverage reports**: Permanent (Codecov)
- **Test results**: 30 days (GitHub Actions)
- **Build logs**: 90 days (GitHub Actions)
- **Releases**: Permanent (GitHub Releases)

---

## Deployment Stage

### Deployment Environments

| Environment | Purpose | Auto-Deploy | Approval | Rollback |
|-------------|---------|-------------|----------|----------|
| **Development** | Feature testing, rapid iteration | ✅ Yes (on push to feature/*) | None | Auto (on failure) |
| **Staging** | Pre-production validation | ✅ Yes (on merge to develop) | Team Lead | Auto (on failure) |
| **Production** | Live trading | ❌ Manual | 2 approvals | Manual (with auto option) |

### Development Environment

**Auto-Deploy Trigger**:
```yaml
# Deploy on push to any feature branch
on:
  push:
    branches:
      - 'feature/**'
      - 'develop'
```

**Deployment Steps**:
1. Build Docker image
2. Push to dev registry
3. Update Kubernetes deployment (dev namespace)
4. Wait for rollout completion
5. Run smoke tests
6. Notify team on Slack

**Dev Environment Configuration**:
```yaml
# k8s/dev/trading-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
  namespace: foxhunt-dev
spec:
  replicas: 1  # Single replica for dev
  strategy:
    type: Recreate  # Fast deployment, downtime acceptable
  template:
    spec:
      containers:
      - name: trading-service
        image: ghcr.io/foxhunt/trading-service:develop
        env:
        - name: RUST_LOG
          value: "debug"  # Verbose logging in dev
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: foxhunt-dev-secrets
              key: database-url
```

### Staging Environment

**Deploy Trigger**:
```yaml
# Deploy on merge to develop (with approval)
on:
  pull_request:
    types: [closed]
    branches:
      - develop
```

**Approval Workflow**:
```yaml
- name: Request Staging Deployment Approval
  uses: trstringer/manual-approval@v1
  with:
    secret: ${{ secrets.GITHUB_TOKEN }}
    approvers: team-leads
    minimum-approvals: 1
    issue-title: "Deploy to Staging: ${{ github.sha }}"
```

**Staging Deployment Steps**:
1. Run full test suite (unit + integration)
2. Security scan (no critical/high vulnerabilities)
3. Build Docker image
4. Push to staging registry
5. **Manual approval required**
6. Update Kubernetes deployment (staging namespace)
7. Run performance benchmarks
8. Run smoke tests
9. Validate metrics (Prometheus alerts)
10. Notify team on Slack

**Staging Configuration**:
```yaml
# k8s/staging/trading-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
  namespace: foxhunt-staging
spec:
  replicas: 2  # Multi-replica for load testing
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0  # Zero-downtime deployment
      maxSurge: 1
  template:
    spec:
      containers:
      - name: trading-service
        image: ghcr.io/foxhunt/trading-service:staging-${{ github.sha }}
        env:
        - name: RUST_LOG
          value: "info"
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1Gi
```

### Production Environment

**Deploy Trigger**:
```yaml
# Manual trigger only (no auto-deploy to production)
on:
  workflow_dispatch:
    inputs:
      version:
        description: 'Version to deploy (e.g., v1.2.3)'
        required: true
      reason:
        description: 'Deployment reason'
        required: true
```

**Approval Workflow**:
```yaml
- name: Request Production Deployment Approval
  uses: trstringer/manual-approval@v1
  with:
    secret: ${{ secrets.GITHUB_TOKEN }}
    approvers: team-leads,devops-team
    minimum-approvals: 2  # Requires 2 approvals
    issue-title: "🚀 Deploy to Production: ${{ github.event.inputs.version }}"
    issue-body: |
      **Version**: ${{ github.event.inputs.version }}
      **Reason**: ${{ github.event.inputs.reason }}
      **Commit**: ${{ github.sha }}
      **Triggered by**: ${{ github.actor }}
```

**Production Deployment Steps**:
1. Verify staging deployment successful (last 24h)
2. Security scan (zero critical vulnerabilities)
3. Build Docker image (tagged with version)
4. Push to production registry (ECR)
5. **2 manual approvals required**
6. Database migrations (if any, with DBA approval)
7. Blue-Green deployment:
   - Deploy to "green" environment
   - Run health checks
   - Run smoke tests
   - Shift traffic gradually (10% → 50% → 100%)
   - Monitor metrics for 15 minutes
8. Decommission "blue" environment
9. Create GitHub release
10. Notify team on Slack

**Production Configuration**:
```yaml
# k8s/prod/trading-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
  namespace: foxhunt-prod
spec:
  replicas: 4  # High availability
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # Maintain 75% capacity during rollout
      maxSurge: 1
  template:
    spec:
      containers:
      - name: trading-service
        image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/foxhunt/trading-service:v1.2.3
        env:
        - name: RUST_LOG
          value: "warn"  # Minimal logging in production
        - name: ENABLE_AUDIT_LOGGING
          value: "true"
        resources:
          requests:
            cpu: 2000m
            memory: 4Gi
          limits:
            cpu: 4000m
            memory: 8Gi
      affinity:
        podAntiAffinity:  # Spread across nodes
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - trading-service
            topologyKey: kubernetes.io/hostname
```

### Health Check Validation

**Kubernetes Liveness Probe**:
```yaml
livenessProbe:
  exec:
    command:
    - /usr/local/bin/grpc_health_probe
    - -addr=:50051
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
```

**Kubernetes Readiness Probe**:
```yaml
readinessProbe:
  exec:
    command:
    - /usr/local/bin/grpc_health_probe
    - -addr=:50051
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2
```

**Post-Deployment Health Checks**:
```bash
#!/bin/bash
# scripts/health_check.sh

SERVICE_URL="https://api.foxhunt.io"
MAX_RETRIES=30
RETRY_DELAY=10

for i in $(seq 1 $MAX_RETRIES); do
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" $SERVICE_URL/health)

  if [ "$HTTP_STATUS" -eq 200 ]; then
    echo "✅ Health check passed (attempt $i/$MAX_RETRIES)"
    exit 0
  fi

  echo "⏳ Health check failed (attempt $i/$MAX_RETRIES), retrying in ${RETRY_DELAY}s..."
  sleep $RETRY_DELAY
done

echo "❌ Health check failed after $MAX_RETRIES attempts"
exit 1
```

### Rollback Automation

**Automatic Rollback Triggers**:
- Health check failure after deployment
- Error rate >5% (Prometheus alert)
- P99 latency >100ms (Prometheus alert)
- Memory usage >90% (Kubernetes OOM)

**Rollback Process**:
```yaml
- name: Automatic Rollback on Failure
  if: failure()
  run: |
    echo "🔄 Deployment failed, initiating rollback..."
    kubectl rollout undo deployment/trading-service -n foxhunt-prod
    kubectl rollout status deployment/trading-service -n foxhunt-prod

    # Notify team
    curl -X POST $SLACK_WEBHOOK -d '{
      "text": "⚠️ Production deployment failed, rollback completed",
      "attachments": [{
        "color": "danger",
        "fields": [
          {"title": "Version", "value": "${{ github.event.inputs.version }}"},
          {"title": "Commit", "value": "${{ github.sha }}"},
          {"title": "Actor", "value": "${{ github.actor }}"}
        ]
      }]
    }'
```

**Manual Rollback**:
```bash
# Rollback to previous version
kubectl rollout undo deployment/trading-service -n foxhunt-prod

# Rollback to specific revision
kubectl rollout undo deployment/trading-service -n foxhunt-prod --to-revision=42

# Check rollout status
kubectl rollout status deployment/trading-service -n foxhunt-prod

# Verify rollback
kubectl describe deployment trading-service -n foxhunt-prod | grep Image:
```

---

## Security Scanning

### Dependency Vulnerability Scanning

**Tool**: Cargo Audit (RustSec Advisory Database)

```yaml
- name: Security Audit
  run: |
    cargo install cargo-audit
    cargo audit --deny warnings --ignore RUSTSEC-2020-0071
```

**Current Known Vulnerabilities**:
- **RSA Marvin Attack (CVSS 5.9)**: Mitigated (PostgreSQL-only, no MySQL)
- **Unmaintained**: `instant`, `paste` (low risk)

**Vulnerability Response SLA**:
| Severity | Response Time | Fix Time | Action |
|----------|---------------|----------|--------|
| **Critical** | 1 hour | 24 hours | Block all deployments |
| **High** | 4 hours | 72 hours | Block production deployments |
| **Medium** | 1 day | 1 week | Warning, track in issue |
| **Low** | 1 week | 1 month | Track in backlog |

### Docker Image Scanning

**Tool**: Trivy (Aqua Security)

```yaml
- name: Scan Docker Image
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: 'ghcr.io/foxhunt/trading-service:${{ github.sha }}'
    format: 'sarif'
    output: 'trivy-results.sarif'
    severity: 'CRITICAL,HIGH'
    exit-code: '1'  # Fail on critical/high vulnerabilities
```

**Scan Coverage**:
- OS package vulnerabilities (Debian base image)
- Application dependencies (Rust crates)
- Embedded secrets detection
- Malware scanning

### SAST/DAST Integration

**Static Application Security Testing (SAST)**:

Tool: **Clippy** (Rust linter with security rules)
```yaml
- name: Run Clippy Security Lints
  run: |
    cargo clippy --workspace --all-targets -- \
      -D clippy::unwrap_used \
      -D clippy::expect_used \
      -D clippy::panic \
      -D clippy::todo \
      -D clippy::unimplemented \
      -W clippy::all
```

Tool: **Cargo Geiger** (Unsafe code detection)
```yaml
- name: Unsafe Code Audit
  run: |
    cargo install cargo-geiger
    cargo geiger --all-features --output-format GitHubMarkdown >> $GITHUB_STEP_SUMMARY
```

**Dynamic Application Security Testing (DAST)**:

Tool: **OWASP ZAP** (API security testing)
```yaml
- name: DAST Scan
  uses: zaproxy/action-api-scan@v0.4.0
  with:
    target: 'https://staging.foxhunt.io'
    rules_file_name: '.zap/rules.tsv'
    fail_action: true
```

**DAST Test Coverage**:
- API authentication bypass
- SQL injection (gRPC parameters)
- XSS in error messages
- CSRF protection
- Rate limiting validation

### Secret Detection

**Tool**: TruffleHog (Git secret scanning)

```yaml
- name: Scan for Secrets
  uses: trufflesecurity/trufflehog@main
  with:
    path: ./
    base: ${{ github.event.repository.default_branch }}
    head: HEAD
    extra_args: --only-verified
```

**Prevented Secrets**:
- API keys (AWS, Databento, Alpaca)
- Database credentials
- JWT secrets
- Private keys (RSA, ED25519)
- OAuth tokens

**Secret Management**:
- Development: Environment variables (`.env` gitignored)
- Staging/Production: HashiCorp Vault
- CI/CD: GitHub Secrets (encrypted at rest)

---

## Performance Benchmarks

### Automated Benchmark Execution

**Criterion Benchmarks** (latency measurement):
```yaml
- name: Run Criterion Benchmarks
  run: |
    cargo bench --workspace --bench e2e_latency -- --save-baseline ${{ github.sha }}
    cargo bench --workspace --bench order_matching -- --save-baseline ${{ github.sha }}
```

**Benchmark Suites**:
1. **E2E Latency** (`trading_engine/benches/e2e_latency.rs`):
   - Order submission → execution latency
   - Target: <100μs p99

2. **Order Matching** (`trading_engine/benches/order_matching.rs`):
   - Matching engine throughput
   - Target: 50K+ orders/sec

3. **Lock-Free Queue** (`trading_engine/benches/lockfree_queue.rs`):
   - Queue enqueue/dequeue latency
   - Target: <1μs p50

4. **ML Inference** (`services/trading_service/tests/performance_benchmarks.rs`):
   - MAMBA-2, DQN, PPO, TFT inference latency
   - Target: <10ms p99 (GPU), <50ms (CPU)

### Performance Regression Detection

**Baseline Comparison**:
```yaml
- name: Compare Benchmarks
  run: |
    cargo bench --workspace -- --baseline ${{ github.event.before }}

    # Fail if >10% regression
    python scripts/compare_benchmarks.py \
      --baseline ${{ github.event.before }} \
      --current ${{ github.sha }} \
      --threshold 0.10
```

**Regression Thresholds**:
| Metric | Threshold | Action |
|--------|-----------|--------|
| **Latency** | +10% | Block merge, alert team |
| **Throughput** | -10% | Block merge, alert team |
| **Memory** | +20% | Warning, track in issue |
| **CPU** | +15% | Warning, track in issue |

### Benchmark Reporting

**Report Format**:
```markdown
## Benchmark Results

| Benchmark | Baseline | Current | Change | Status |
|-----------|----------|---------|--------|--------|
| E2E Latency (p99) | 87μs | 92μs | +5.7% | ✅ PASS |
| Order Matching | 52K ops/s | 51K ops/s | -1.9% | ✅ PASS |
| ML Inference (p99) | 8.2ms | 7.9ms | -3.7% | ✅ PASS |

**Detailed Results**: [View Full Report](https://github.com/foxhunt/foxhunt/actions/runs/123456)
```

**Storage**:
- Criterion output: `target/criterion/`
- Historical data: GitHub Actions artifacts (90 days)
- Long-term tracking: InfluxDB (permanent)

**Visualization**:
```yaml
- name: Publish Benchmark Results
  run: |
    # Upload to InfluxDB
    python scripts/upload_benchmarks.py \
      --file target/criterion/e2e_latency/base/estimates.json \
      --influx-url $INFLUX_URL \
      --token $INFLUX_TOKEN
```

---

## GitOps Integration

### Infrastructure as Code (Terraform)

**Directory Structure**:
```
terraform/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   └── production/
├── modules/
│   ├── eks/          # Kubernetes cluster
│   ├── rds/          # PostgreSQL database
│   ├── elasticache/  # Redis cache
│   └── s3/           # Object storage
└── global/
    ├── iam/          # IAM roles and policies
    └── route53/      # DNS configuration
```

**Terraform Workflow**:
```yaml
- name: Terraform Plan
  run: |
    cd terraform/environments/${{ inputs.environment }}
    terraform init
    terraform plan -out=tfplan

- name: Terraform Apply
  if: github.event.inputs.approve == 'true'
  run: |
    cd terraform/environments/${{ inputs.environment }}
    terraform apply tfplan
```

**State Management**:
- Backend: S3 + DynamoDB (state locking)
- Versioning: Enabled (rollback capability)
- Encryption: AES-256 at rest

### Kubernetes Manifests

**Directory Structure**:
```
k8s/
├── base/                    # Kustomize base
│   ├── api-gateway/
│   ├── trading-service/
│   ├── backtesting-service/
│   └── ml-training-service/
├── overlays/
│   ├── dev/                # Dev environment overrides
│   ├── staging/            # Staging overrides
│   └── production/         # Production overrides
└── helm/
    └── foxhunt/            # Helm chart
        ├── Chart.yaml
        ├── values.yaml
        └── templates/
```

**Kustomize Example**:
```yaml
# k8s/base/trading-service/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - servicemonitor.yaml
  - hpa.yaml

configMapGenerator:
  - name: trading-service-config
    files:
      - config.toml

secretGenerator:
  - name: trading-service-secrets
    envs:
      - secrets.env
```

**Environment Overlays**:
```yaml
# k8s/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base/trading-service

patchesStrategicMerge:
  - replica-patch.yaml  # Increase replicas to 4
  - resource-patch.yaml # Increase CPU/memory limits
```

### Configuration Management

**ConfigMap Hot-Reload** (from CLAUDE.md):
```yaml
# k8s/base/trading-service/deployment.yaml
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        # Force pod restart on ConfigMap change
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      containers:
      - name: trading-service
        env:
        - name: CONFIG_RELOAD_ENABLED
          value: "true"
        - name: CONFIG_RELOAD_INTERVAL_SECS
          value: "60"
        volumeMounts:
        - name: config
          mountPath: /etc/foxhunt
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: trading-service-config
```

**Secret Management (External Secrets Operator)**:
```yaml
# k8s/base/trading-service/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: trading-service-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: trading-service-secrets
    creationPolicy: Owner
  data:
  - secretKey: database-url
    remoteRef:
      key: foxhunt/production/database
      property: url
  - secretKey: jwt-secret
    remoteRef:
      key: foxhunt/production/auth
      property: jwt_secret
```

### GitOps Workflows

**ArgoCD Application**:
```yaml
# argocd/trading-service.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: trading-service
  namespace: argocd
spec:
  project: foxhunt
  source:
    repoURL: https://github.com/foxhunt/foxhunt
    targetRevision: main
    path: k8s/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: foxhunt-prod
  syncPolicy:
    automated:
      prune: true      # Delete removed resources
      selfHeal: true   # Auto-sync on drift
      allowEmpty: false
    syncOptions:
    - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
```

**GitOps Flow**:
```
1. Developer merges PR to main
2. GitHub Actions builds Docker image
3. GitHub Actions updates k8s/overlays/production/kustomization.yaml
4. ArgoCD detects change (poll interval: 3 minutes)
5. ArgoCD syncs new image to production
6. Kubernetes performs rolling update
7. ArgoCD reports sync status to Slack
```

**Deployment Notifications**:
```yaml
# argocd/notifications.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [slack-deployment-success]

  template.slack-deployment-success: |
    message: |
      ✅ Application {{.app.metadata.name}} deployed successfully!
      Version: {{.app.status.sync.revision}}
      Environment: {{.app.spec.destination.namespace}}
```

---

## Workflow Examples

See the following workflow files:
- `.github/workflows/test.yml` - Test pipeline
- `.github/workflows/build.yml` - Docker build pipeline
- `.github/workflows/deploy.yml` - Deployment pipeline
- `.gitlab-ci.yml` - GitLab CI example

---

## Troubleshooting

### Common Issues

**1. Coverage Threshold Failure**
```
Error: Coverage 58.3% below threshold 60%
```

**Solution**:
```bash
# Identify uncovered areas
cargo llvm-cov --html --output-dir coverage_report
open coverage_report/index.html

# Add tests for uncovered code
cargo test -p <package> --lib -- --nocapture
```

**2. Docker Build Failure**
```
Error: failed to solve: executor failed running [/bin/sh -c cargo build --release]
```

**Solution**:
```bash
# Check Dockerfile syntax
docker build --no-cache -t test .

# Verify dependencies
cargo check --workspace

# Clear cache and rebuild
docker system prune -a
```

**3. Health Check Timeout**
```
Error: Health check failed after 30 retries
```

**Solution**:
```bash
# Check pod logs
kubectl logs -n foxhunt-prod deployment/trading-service --tail=100

# Check pod status
kubectl describe pod -n foxhunt-prod -l app=trading-service

# Manually test health check
kubectl exec -it -n foxhunt-prod <pod-name> -- grpc_health_probe -addr=:50051
```

**4. Rollback Failed**
```
Error: No previous revision available
```

**Solution**:
```bash
# Check rollout history
kubectl rollout history deployment/trading-service -n foxhunt-prod

# Manually rollback to specific revision
kubectl rollout undo deployment/trading-service -n foxhunt-prod --to-revision=42
```

### Pipeline Debugging

**Enable Debug Logging**:
```yaml
# .github/workflows/test.yml
env:
  RUST_LOG: debug
  RUST_BACKTRACE: 1
  ACTIONS_RUNNER_DEBUG: true
  ACTIONS_STEP_DEBUG: true
```

**SSH into GitHub Actions Runner**:
```yaml
- name: Setup tmate session
  uses: mxschmitt/action-tmate@v3
  if: failure()
```

**Local Pipeline Simulation**:
```bash
# Install act (GitHub Actions local runner)
brew install act

# Run workflow locally
act -j test --secret-file .env.secrets

# Debug specific step
act -j test --step "Run Tests" -v
```

### Contact and Support

**Team Contacts**:
- DevOps Lead: @devops-lead (Slack)
- Platform Team: #platform-team (Slack)
- On-Call: @oncall (PagerDuty)

**Documentation**:
- Architecture: `CLAUDE.md`
- Testing: `TESTING_PLAN.md`
- Deployment: `DEPLOYMENT_GUIDE.md`

**Incident Response**:
- Critical: Page on-call engineer (PagerDuty)
- High: Alert in #incidents channel
- Medium: Create issue, assign to team
- Low: Track in backlog

---

**Last Updated**: 2025-10-07
**Maintained By**: Platform Team
**Review Cycle**: Quarterly