foxhunt/CI_CD_PIPELINE.md

CI/CD Pipeline Documentation

Last Updated: 2025-10-07 Version: 1.0.0 Owner: Platform Team

---

Table of Contents

1. Pipeline Overview
2. Testing Stage
3. Build Stage
4. Deployment Stage
5. Security Scanning
6. Performance Benchmarks
7. GitOps Integration
8. Workflow Examples
9. Troubleshooting

---

Pipeline Overview

Pipeline Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         CI/CD Pipeline                           │
└─────────────────────────────────────────────────────────────────┘
                                │
                    ┌───────────┴───────────┐
                    ▼                       ▼
              ┌──────────┐           ┌──────────┐
              │   Test   │           │  Build   │
              │  Stage   │           │  Stage   │
              └────┬─────┘           └────┬─────┘
                   │                      │
         ┌─────────┴─────────┐           │
         ▼                   ▼           ▼
    ┌─────────┐         ┌─────────┐ ┌─────────┐
    │  Unit   │         │  Integ  │ │ Docker  │
    │  Tests  │         │  Tests  │ │  Build  │
    └────┬────┘         └────┬────┘ └────┬────┘
         │                   │           │
         └─────────┬─────────┘           │
                   ▼                     ▼
              ┌──────────┐         ┌──────────┐
              │ Security │         │  Deploy  │
              │   Scan   │         │  Stage   │
              └────┬─────┘         └────┬─────┘
                   │                    │
                   │      ┌─────────────┴─────────────┐
                   │      ▼             ▼             ▼
                   │  ┌──────┐     ┌─────────┐  ┌──────┐
                   │  │ Dev  │     │ Staging │  │ Prod │
                   │  └──────┘     └─────────┘  └──────┘
                   │      │             │            │
                   └──────┴─────────────┴────────────┘
                                │
                                ▼
                          ┌──────────┐
                          │ Validate │
                          │  Health  │
                          └──────────┘

Pipeline Stages

Stage	Purpose	Duration	Auto/Manual
Test	Run unit, integration, and coverage tests	5-10 min	Auto
Security Scan	Vulnerability scanning, SAST/DAST	3-5 min	Auto
Build	Docker image building and tagging	8-12 min	Auto
Deploy Dev	Deploy to development environment	2-3 min	Auto
Deploy Staging	Deploy to staging environment	3-5 min	Manual approval
Performance Test	Run automated benchmarks	5-10 min	Auto (staging)
Deploy Prod	Deploy to production environment	5-8 min	Manual approval
Rollback	Automated rollback on health check failure	1-2 min	Auto (on failure)

Branching Strategy

main (production)
  │
  ├── develop (staging)
  │     │
  │     ├── feature/agent-98-ci-cd-docs
  │     ├── feature/performance-optimization
  │     └── feature/new-ml-model
  │
  └── hotfix/critical-bug-fix

Branch Protection Rules:

• main: Requires 2 approvals, all checks pass, no force push
• develop: Requires 1 approval, all checks pass
• feature/*: No restrictions, auto-delete after merge

Merge Strategy:

• Feature → Develop: Squash merge
• Develop → Main: Merge commit (preserves history)
• Hotfix → Main: Cherry-pick to develop

Automated vs Manual Gates

Automated Gates (block merge on failure):

• ✅ All unit tests pass (>99% pass rate required)
• ✅ Integration tests pass
• ✅ Code coverage ≥60% (enforced)
• ✅ No critical/high security vulnerabilities
• ✅ Docker build succeeds
• ✅ Linting passes (cargo clippy)
• ✅ Formatting passes (cargo fmt --check)

Manual Gates (require approval):

• 🔒 Deploy to staging (team lead approval)
• 🔒 Deploy to production (2 approvals: team lead + DevOps)
• 🔒 Database migrations (DBA approval)
• 🔒 Infrastructure changes (DevOps approval)

Integration Points

GitHub Actions: Primary CI/CD platform

• Workflows: .github/workflows/*.yml
• Secrets: GitHub Secrets (encrypted)
• Artifacts: GitHub Packages / Docker Registry

GitLab CI: Alternative/fallback

• Configuration: .gitlab-ci.yml
• Runners: Kubernetes-based GitLab Runners
• Registry: GitLab Container Registry

Kubernetes: Deployment target

• Manifests: k8s/*.yaml
• Helm charts: helm/foxhunt/
• ArgoCD: GitOps continuous deployment

Observability:

• Prometheus: Metrics collection
• Grafana: Dashboards and alerts
• Loki: Log aggregation
• Jaeger: Distributed tracing

---

Testing Stage

Test Execution

Unit Tests (5-7 minutes):

# Run all unit tests with coverage
cargo llvm-cov --workspace --lcov --output-path lcov.info

# Requirements:
# - Pass rate: >99% (336/337+ tests passing)
# - Coverage: ≥60% workspace coverage
# - No panics or segfaults

Integration Tests (3-5 minutes):

# Run integration tests (requires Docker services)
docker-compose up -d postgres redis vault
cargo test --workspace --test '*' -- --test-threads=1

# Tests:
# - API Gateway E2E (services/api_gateway/tests/e2e_tests.rs)
# - Trading Service E2E (services/trading_service/tests/integration_e2e_tests.rs)
# - ML Training Service (services/ml_training_service/tests/integration_tests.rs)
# - Backtesting Service (services/backtesting_service/tests/integration_tests.rs)

GPU Tests (optional, 2-3 minutes):

# Run slow GPU tests (if CUDA available)
cargo test -p ml --lib -- --ignored --test-threads=1

# Skipped in CI if GPU not available
# Uses CPU fallback automatically

Coverage Reporting

Coverage Threshold Enforcement:

# .github/workflows/test.yml
- name: Check Coverage Threshold
  run: |
    COVERAGE=$(cargo llvm-cov --workspace --summary-only | grep -oP 'TOTAL.*\K[0-9.]+(?=%)')
    echo "Coverage: $COVERAGE%"
    if (( $(echo "$COVERAGE < 60.0" | bc -l) )); then
      echo "❌ Coverage $COVERAGE% below threshold 60%"
      exit 1
    fi
    echo "✅ Coverage $COVERAGE% meets threshold"

Coverage Report Artifacts:

• lcov.info: LCOV format (for Codecov/Coveralls)
• coverage_report/: HTML report (browsable)
• coverage-summary.json: JSON summary for badges

Coverage Badges:

[![Coverage](https://img.shields.io/codecov/c/github/foxhunt/foxhunt)](https://codecov.io/gh/foxhunt/foxhunt)

Test Pass Rate Requirements

Environment	Pass Rate	Coverage	Action on Failure
Feature Branch	≥99%	≥55%	Block merge, notify author
Develop	≥99.5%	≥60%	Block merge, alert team
Main	100%	≥60%	Block merge, rollback if deployed

Flaky Test Handling:

# Retry flaky tests up to 3 times
- name: Run Tests with Retry
  uses: nick-invision/retry@v2
  with:
    timeout_minutes: 15
    max_attempts: 3
    command: cargo test --workspace

---

Build Stage

Docker Image Building

Multi-Stage Build Strategy (from Agent 94):

# Stage 1: Build dependencies
FROM rust:1.75-slim AS chef
RUN cargo install cargo-chef

# Stage 2: Plan dependencies
FROM chef AS planner
WORKDIR /app
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# Stage 3: Build dependencies (cached layer)
FROM chef AS builder
WORKDIR /app
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

# Stage 4: Build application
COPY . .
RUN cargo build --release -p trading_service

# Stage 5: Runtime image
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates libssl3 && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/trading_service /usr/local/bin/
CMD ["trading_service"]

Build Optimization:

• Dependency caching: cargo-chef reduces rebuild time 80%
• Layer caching: Only rebuild changed layers
• Multi-platform: Build for linux/amd64 and linux/arm64
• Image size: ~50MB final runtime image

Image Tagging Strategy

Tagging Scheme:

# Git commit SHA (immutable)
foxhunt/trading-service:abc123def456

# Semantic version (tagged releases)
foxhunt/trading-service:v1.2.3

# Environment tags (mutable)
foxhunt/trading-service:latest        # main branch
foxhunt/trading-service:develop       # develop branch
foxhunt/trading-service:staging       # staging environment
foxhunt/trading-service:production    # production environment

# Full example
foxhunt/trading-service:v1.2.3-abc123def456

Tagging Logic:

- name: Docker Metadata
  id: meta
  uses: docker/metadata-action@v5
  with:
    images: |
      foxhunt/trading-service
      ghcr.io/foxhunt/trading-service
    tags: |
      type=ref,event=branch
      type=ref,event=pr
      type=semver,pattern={{version}}
      type=semver,pattern={{major}}.{{minor}}
      type=sha,prefix={{branch}}-

Image Registry

Primary Registry: GitHub Container Registry (GHCR):

# Login
echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_ACTOR --password-stdin

# Push
docker push ghcr.io/foxhunt/trading-service:v1.2.3

Alternative Registries:

• AWS ECR: Production deployments (high availability)
• Docker Hub: Public images (documentation examples)
• GitLab Registry: Fallback/self-hosted option

Registry Configuration:

# .github/workflows/build.yml
- name: Push to Multiple Registries
  run: |
    # GitHub Container Registry (primary)
    docker push ghcr.io/foxhunt/trading-service:${{ steps.meta.outputs.tags }}

    # AWS ECR (production)
    aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_REGISTRY
    docker tag ghcr.io/foxhunt/trading-service:$TAG $ECR_REGISTRY/trading-service:$TAG
    docker push $ECR_REGISTRY/trading-service:$TAG

Build Artifact Storage

Artifacts Stored:

1. Docker images (tagged and pushed to registries)
2. Coverage reports (uploaded to Codecov)
3. Test results (JUnit XML format)
4. Build logs (GitHub Actions artifacts)
5. Binary releases (GitHub Releases for tagged versions)

Artifact Retention:

• Docker images: 90 days for non-production tags
• Coverage reports: Permanent (Codecov)
• Test results: 30 days (GitHub Actions)
• Build logs: 90 days (GitHub Actions)
• Releases: Permanent (GitHub Releases)

---

Deployment Stage

Deployment Environments

Environment	Purpose	Auto-Deploy	Approval	Rollback
Development	Feature testing, rapid iteration	✅ Yes (on push to feature/*)	None	Auto (on failure)
Staging	Pre-production validation	✅ Yes (on merge to develop)	Team Lead	Auto (on failure)
Production	Live trading	❌ Manual	2 approvals	Manual (with auto option)

Development Environment

Auto-Deploy Trigger:

# Deploy on push to any feature branch
on:
  push:
    branches:
      - 'feature/**'
      - 'develop'

Deployment Steps:

1. Build Docker image
2. Push to dev registry
3. Update Kubernetes deployment (dev namespace)
4. Wait for rollout completion
5. Run smoke tests
6. Notify team on Slack

Dev Environment Configuration:

# k8s/dev/trading-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
  namespace: foxhunt-dev
spec:
  replicas: 1  # Single replica for dev
  strategy:
    type: Recreate  # Fast deployment, downtime acceptable
  template:
    spec:
      containers:
      - name: trading-service
        image: ghcr.io/foxhunt/trading-service:develop
        env:
        - name: RUST_LOG
          value: "debug"  # Verbose logging in dev
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: foxhunt-dev-secrets
              key: database-url

Staging Environment

Deploy Trigger:

# Deploy on merge to develop (with approval)
on:
  pull_request:
    types: [closed]
    branches:
      - develop

Approval Workflow:

- name: Request Staging Deployment Approval
  uses: trstringer/manual-approval@v1
  with:
    secret: ${{ secrets.GITHUB_TOKEN }}
    approvers: team-leads
    minimum-approvals: 1
    issue-title: "Deploy to Staging: ${{ github.sha }}"

Staging Deployment Steps:

1. Run full test suite (unit + integration)
2. Security scan (no critical/high vulnerabilities)
3. Build Docker image
4. Push to staging registry
5. Manual approval required
6. Update Kubernetes deployment (staging namespace)
7. Run performance benchmarks
8. Run smoke tests
9. Validate metrics (Prometheus alerts)
10. Notify team on Slack

Staging Configuration:

# k8s/staging/trading-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
  namespace: foxhunt-staging
spec:
  replicas: 2  # Multi-replica for load testing
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0  # Zero-downtime deployment
      maxSurge: 1
  template:
    spec:
      containers:
      - name: trading-service
        image: ghcr.io/foxhunt/trading-service:staging-${{ github.sha }}
        env:
        - name: RUST_LOG
          value: "info"
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1Gi

Production Environment

Deploy Trigger:

# Manual trigger only (no auto-deploy to production)
on:
  workflow_dispatch:
    inputs:
      version:
        description: 'Version to deploy (e.g., v1.2.3)'
        required: true
      reason:
        description: 'Deployment reason'
        required: true

Approval Workflow:

- name: Request Production Deployment Approval
  uses: trstringer/manual-approval@v1
  with:
    secret: ${{ secrets.GITHUB_TOKEN }}
    approvers: team-leads,devops-team
    minimum-approvals: 2  # Requires 2 approvals
    issue-title: "🚀 Deploy to Production: ${{ github.event.inputs.version }}"
    issue-body: |
      **Version**: ${{ github.event.inputs.version }}
      **Reason**: ${{ github.event.inputs.reason }}
      **Commit**: ${{ github.sha }}
      **Triggered by**: ${{ github.actor }}

Production Deployment Steps:

1. Verify staging deployment successful (last 24h)
2. Security scan (zero critical vulnerabilities)
3. Build Docker image (tagged with version)
4. Push to production registry (ECR)
5. 2 manual approvals required
6. Database migrations (if any, with DBA approval)
7. Blue-Green deployment:
   • Deploy to "green" environment
   • Run health checks
   • Run smoke tests
   • Shift traffic gradually (10% → 50% → 100%)
   • Monitor metrics for 15 minutes
8. Decommission "blue" environment
9. Create GitHub release
10. Notify team on Slack

Production Configuration:

# k8s/prod/trading-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trading-service
  namespace: foxhunt-prod
spec:
  replicas: 4  # High availability
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # Maintain 75% capacity during rollout
      maxSurge: 1
  template:
    spec:
      containers:
      - name: trading-service
        image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/foxhunt/trading-service:v1.2.3
        env:
        - name: RUST_LOG
          value: "warn"  # Minimal logging in production
        - name: ENABLE_AUDIT_LOGGING
          value: "true"
        resources:
          requests:
            cpu: 2000m
            memory: 4Gi
          limits:
            cpu: 4000m
            memory: 8Gi
      affinity:
        podAntiAffinity:  # Spread across nodes
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - trading-service
            topologyKey: kubernetes.io/hostname

Health Check Validation

Kubernetes Liveness Probe:

livenessProbe:
  exec:
    command:
    - /usr/local/bin/grpc_health_probe
    - -addr=:50051
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

Kubernetes Readiness Probe:

readinessProbe:
  exec:
    command:
    - /usr/local/bin/grpc_health_probe
    - -addr=:50051
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2

Post-Deployment Health Checks:

#!/bin/bash
# scripts/health_check.sh

SERVICE_URL="https://api.foxhunt.io"
MAX_RETRIES=30
RETRY_DELAY=10

for i in $(seq 1 $MAX_RETRIES); do
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" $SERVICE_URL/health)

  if [ "$HTTP_STATUS" -eq 200 ]; then
    echo "✅ Health check passed (attempt $i/$MAX_RETRIES)"
    exit 0
  fi

  echo "⏳ Health check failed (attempt $i/$MAX_RETRIES), retrying in ${RETRY_DELAY}s..."
  sleep $RETRY_DELAY
done

echo "❌ Health check failed after $MAX_RETRIES attempts"
exit 1

Rollback Automation

Automatic Rollback Triggers:

• Health check failure after deployment
• Error rate >5% (Prometheus alert)
• P99 latency >100ms (Prometheus alert)
• Memory usage >90% (Kubernetes OOM)

Rollback Process:

- name: Automatic Rollback on Failure
  if: failure()
  run: |
    echo "🔄 Deployment failed, initiating rollback..."
    kubectl rollout undo deployment/trading-service -n foxhunt-prod
    kubectl rollout status deployment/trading-service -n foxhunt-prod

    # Notify team
    curl -X POST $SLACK_WEBHOOK -d '{
      "text": "⚠️ Production deployment failed, rollback completed",
      "attachments": [{
        "color": "danger",
        "fields": [
          {"title": "Version", "value": "${{ github.event.inputs.version }}"},
          {"title": "Commit", "value": "${{ github.sha }}"},
          {"title": "Actor", "value": "${{ github.actor }}"}
        ]
      }]
    }'

Manual Rollback:

# Rollback to previous version
kubectl rollout undo deployment/trading-service -n foxhunt-prod

# Rollback to specific revision
kubectl rollout undo deployment/trading-service -n foxhunt-prod --to-revision=42

# Check rollout status
kubectl rollout status deployment/trading-service -n foxhunt-prod

# Verify rollback
kubectl describe deployment trading-service -n foxhunt-prod | grep Image:

---

Security Scanning

Dependency Vulnerability Scanning

Tool: Cargo Audit (RustSec Advisory Database)

- name: Security Audit
  run: |
    cargo install cargo-audit
    cargo audit --deny warnings --ignore RUSTSEC-2020-0071

Current Known Vulnerabilities:

• RSA Marvin Attack (CVSS 5.9): Mitigated (PostgreSQL-only, no MySQL)
• Unmaintained: instant, paste (low risk)

Vulnerability Response SLA:

Severity	Response Time	Fix Time	Action
Critical	1 hour	24 hours	Block all deployments
High	4 hours	72 hours	Block production deployments
Medium	1 day	1 week	Warning, track in issue
Low	1 week	1 month	Track in backlog

Docker Image Scanning

Tool: Trivy (Aqua Security)

- name: Scan Docker Image
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: 'ghcr.io/foxhunt/trading-service:${{ github.sha }}'
    format: 'sarif'
    output: 'trivy-results.sarif'
    severity: 'CRITICAL,HIGH'
    exit-code: '1'  # Fail on critical/high vulnerabilities

Scan Coverage:

• OS package vulnerabilities (Debian base image)
• Application dependencies (Rust crates)
• Embedded secrets detection
• Malware scanning

SAST/DAST Integration

Static Application Security Testing (SAST):

Tool: Clippy (Rust linter with security rules)

- name: Run Clippy Security Lints
  run: |
    cargo clippy --workspace --all-targets -- \
      -D clippy::unwrap_used \
      -D clippy::expect_used \
      -D clippy::panic \
      -D clippy::todo \
      -D clippy::unimplemented \
      -W clippy::all

Tool: Cargo Geiger (Unsafe code detection)

- name: Unsafe Code Audit
  run: |
    cargo install cargo-geiger
    cargo geiger --all-features --output-format GitHubMarkdown >> $GITHUB_STEP_SUMMARY

Dynamic Application Security Testing (DAST):

Tool: OWASP ZAP (API security testing)

- name: DAST Scan
  uses: zaproxy/action-api-scan@v0.4.0
  with:
    target: 'https://staging.foxhunt.io'
    rules_file_name: '.zap/rules.tsv'
    fail_action: true

DAST Test Coverage:

• API authentication bypass
• SQL injection (gRPC parameters)
• XSS in error messages
• CSRF protection
• Rate limiting validation

Secret Detection

Tool: TruffleHog (Git secret scanning)

- name: Scan for Secrets
  uses: trufflesecurity/trufflehog@main
  with:
    path: ./
    base: ${{ github.event.repository.default_branch }}
    head: HEAD
    extra_args: --only-verified

Prevented Secrets:

• API keys (AWS, Databento, Alpaca)
• Database credentials
• JWT secrets
• Private keys (RSA, ED25519)
• OAuth tokens

Secret Management:

• Development: Environment variables (.env gitignored)
• Staging/Production: HashiCorp Vault
• CI/CD: GitHub Secrets (encrypted at rest)

---

Performance Benchmarks

Automated Benchmark Execution

Criterion Benchmarks (latency measurement):

- name: Run Criterion Benchmarks
  run: |
    cargo bench --workspace --bench e2e_latency -- --save-baseline ${{ github.sha }}
    cargo bench --workspace --bench order_matching -- --save-baseline ${{ github.sha }}

Benchmark Suites:

1. E2E Latency (trading_engine/benches/e2e_latency.rs):

   • Order submission → execution latency
   • Target: <100μs p99

2. Order Matching (trading_engine/benches/order_matching.rs):

   • Matching engine throughput
   • Target: 50K+ orders/sec

3. Lock-Free Queue (trading_engine/benches/lockfree_queue.rs):

   • Queue enqueue/dequeue latency
   • Target: <1μs p50

4. ML Inference (services/trading_service/tests/performance_benchmarks.rs):

   • MAMBA-2, DQN, PPO, TFT inference latency
   • Target: <10ms p99 (GPU), <50ms (CPU)

Performance Regression Detection

Baseline Comparison:

- name: Compare Benchmarks
  run: |
    cargo bench --workspace -- --baseline ${{ github.event.before }}

    # Fail if >10% regression
    python scripts/compare_benchmarks.py \
      --baseline ${{ github.event.before }} \
      --current ${{ github.sha }} \
      --threshold 0.10

Regression Thresholds:

Metric	Threshold	Action
Latency	+10%	Block merge, alert team
Throughput	-10%	Block merge, alert team
Memory	+20%	Warning, track in issue
CPU	+15%	Warning, track in issue

Benchmark Reporting

Report Format:

## Benchmark Results

| Benchmark | Baseline | Current | Change | Status |
|-----------|----------|---------|--------|--------|
| E2E Latency (p99) | 87μs | 92μs | +5.7% | ✅ PASS |
| Order Matching | 52K ops/s | 51K ops/s | -1.9% | ✅ PASS |
| ML Inference (p99) | 8.2ms | 7.9ms | -3.7% | ✅ PASS |

**Detailed Results**: [View Full Report](https://github.com/foxhunt/foxhunt/actions/runs/123456)

Storage:

• Criterion output: target/criterion/
• Historical data: GitHub Actions artifacts (90 days)
• Long-term tracking: InfluxDB (permanent)

Visualization:

- name: Publish Benchmark Results
  run: |
    # Upload to InfluxDB
    python scripts/upload_benchmarks.py \
      --file target/criterion/e2e_latency/base/estimates.json \
      --influx-url $INFLUX_URL \
      --token $INFLUX_TOKEN

---

GitOps Integration

Infrastructure as Code (Terraform)

Directory Structure:

terraform/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   └── production/
├── modules/
│   ├── eks/          # Kubernetes cluster
│   ├── rds/          # PostgreSQL database
│   ├── elasticache/  # Redis cache
│   └── s3/           # Object storage
└── global/
    ├── iam/          # IAM roles and policies
    └── route53/      # DNS configuration

Terraform Workflow:

- name: Terraform Plan
  run: |
    cd terraform/environments/${{ inputs.environment }}
    terraform init
    terraform plan -out=tfplan

- name: Terraform Apply
  if: github.event.inputs.approve == 'true'
  run: |
    cd terraform/environments/${{ inputs.environment }}
    terraform apply tfplan

State Management:

• Backend: S3 + DynamoDB (state locking)
• Versioning: Enabled (rollback capability)
• Encryption: AES-256 at rest

Kubernetes Manifests

Directory Structure:

k8s/
├── base/                    # Kustomize base
│   ├── api-gateway/
│   ├── trading-service/
│   ├── backtesting-service/
│   └── ml-training-service/
├── overlays/
│   ├── dev/                # Dev environment overrides
│   ├── staging/            # Staging overrides
│   └── production/         # Production overrides
└── helm/
    └── foxhunt/            # Helm chart
        ├── Chart.yaml
        ├── values.yaml
        └── templates/

Kustomize Example:

# k8s/base/trading-service/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - servicemonitor.yaml
  - hpa.yaml

configMapGenerator:
  - name: trading-service-config
    files:
      - config.toml

secretGenerator:
  - name: trading-service-secrets
    envs:
      - secrets.env

Environment Overlays:

# k8s/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base/trading-service

patchesStrategicMerge:
  - replica-patch.yaml  # Increase replicas to 4
  - resource-patch.yaml # Increase CPU/memory limits

Configuration Management

ConfigMap Hot-Reload (from CLAUDE.md):

# k8s/base/trading-service/deployment.yaml
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        # Force pod restart on ConfigMap change
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
    spec:
      containers:
      - name: trading-service
        env:
        - name: CONFIG_RELOAD_ENABLED
          value: "true"
        - name: CONFIG_RELOAD_INTERVAL_SECS
          value: "60"
        volumeMounts:
        - name: config
          mountPath: /etc/foxhunt
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: trading-service-config

Secret Management (External Secrets Operator):

# k8s/base/trading-service/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: trading-service-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: trading-service-secrets
    creationPolicy: Owner
  data:
  - secretKey: database-url
    remoteRef:
      key: foxhunt/production/database
      property: url
  - secretKey: jwt-secret
    remoteRef:
      key: foxhunt/production/auth
      property: jwt_secret

GitOps Workflows

ArgoCD Application:

# argocd/trading-service.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: trading-service
  namespace: argocd
spec:
  project: foxhunt
  source:
    repoURL: https://github.com/foxhunt/foxhunt
    targetRevision: main
    path: k8s/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: foxhunt-prod
  syncPolicy:
    automated:
      prune: true      # Delete removed resources
      selfHeal: true   # Auto-sync on drift
      allowEmpty: false
    syncOptions:
    - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

GitOps Flow:

1. Developer merges PR to main
2. GitHub Actions builds Docker image
3. GitHub Actions updates k8s/overlays/production/kustomization.yaml
4. ArgoCD detects change (poll interval: 3 minutes)
5. ArgoCD syncs new image to production
6. Kubernetes performs rolling update
7. ArgoCD reports sync status to Slack

Deployment Notifications:

# argocd/notifications.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [slack-deployment-success]

  template.slack-deployment-success: |
    message: |
      ✅ Application {{.app.metadata.name}} deployed successfully!
      Version: {{.app.status.sync.revision}}
      Environment: {{.app.spec.destination.namespace}}

---

Workflow Examples

See the following workflow files:

• .github/workflows/test.yml - Test pipeline
• .github/workflows/build.yml - Docker build pipeline
• .github/workflows/deploy.yml - Deployment pipeline
• .gitlab-ci.yml - GitLab CI example

---

Troubleshooting

Common Issues

1. Coverage Threshold Failure

Error: Coverage 58.3% below threshold 60%

Solution:

# Identify uncovered areas
cargo llvm-cov --html --output-dir coverage_report
open coverage_report/index.html

# Add tests for uncovered code
cargo test -p <package> --lib -- --nocapture

2. Docker Build Failure

Error: failed to solve: executor failed running [/bin/sh -c cargo build --release]

Solution:

# Check Dockerfile syntax
docker build --no-cache -t test .

# Verify dependencies
cargo check --workspace

# Clear cache and rebuild
docker system prune -a

3. Health Check Timeout

Error: Health check failed after 30 retries

Solution:

# Check pod logs
kubectl logs -n foxhunt-prod deployment/trading-service --tail=100

# Check pod status
kubectl describe pod -n foxhunt-prod -l app=trading-service

# Manually test health check
kubectl exec -it -n foxhunt-prod <pod-name> -- grpc_health_probe -addr=:50051

4. Rollback Failed

Error: No previous revision available

Solution:

# Check rollout history
kubectl rollout history deployment/trading-service -n foxhunt-prod

# Manually rollback to specific revision
kubectl rollout undo deployment/trading-service -n foxhunt-prod --to-revision=42

Pipeline Debugging

Enable Debug Logging:

# .github/workflows/test.yml
env:
  RUST_LOG: debug
  RUST_BACKTRACE: 1
  ACTIONS_RUNNER_DEBUG: true
  ACTIONS_STEP_DEBUG: true

SSH into GitHub Actions Runner:

- name: Setup tmate session
  uses: mxschmitt/action-tmate@v3
  if: failure()

Local Pipeline Simulation:

# Install act (GitHub Actions local runner)
brew install act

# Run workflow locally
act -j test --secret-file .env.secrets

# Debug specific step
act -j test --step "Run Tests" -v

Contact and Support

Team Contacts:

• DevOps Lead: @devops-lead (Slack)
• Platform Team: #platform-team (Slack)
• On-Call: @oncall (PagerDuty)

Documentation:

• Architecture: CLAUDE.md
• Testing: TESTING_PLAN.md
• Deployment: DEPLOYMENT_GUIDE.md

Incident Response:

• Critical: Page on-call engineer (PagerDuty)
• High: Alert in #incidents channel
• Medium: Create issue, assign to team
• Low: Track in backlog

---

Last Updated: 2025-10-07 Maintained By: Platform Team Review Cycle: Quarterly