foxhunt/docs/deployment/zero-downtime-deployment.md

# Zero-Downtime Deployment Guide

**Last Updated**: 2025-10-22
**Target**: Production environments
**Expected Downtime**: 0 seconds
**Deployment Time**: 15-25 minutes

---

## Overview

This guide details strategies for deploying Foxhunt updates without service interruptions using rolling updates, blue-green deployments, and canary releases.

## Rolling Update (Default Strategy)

### Kubernetes RollingUpdate

```yaml
# Deployment strategy configuration
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2         # Max 2 extra pods during update
      maxUnavailable: 1   # Max 1 pod unavailable during update
```

### Deployment Procedure

```bash
# Update image tag
kubectl set image deployment/foxhunt-trading-service \
  trading-service=ghcr.io/your-org/foxhunt/trading-service:v0.1.6 \
  -n foxhunt

# Monitor rollout
kubectl rollout status deployment/foxhunt-trading-service -n foxhunt
# Waiting for deployment "foxhunt-trading-service" rollout to finish: 2 out of 5 new replicas have been updated...
# deployment "foxhunt-trading-service" successfully rolled out

# Verify no downtime
while true; do
  curl -sf http://api.foxhunt.example.com/health && echo " OK" || echo " FAILED"
  sleep 1
done
```

---

## Blue-Green Deployment

### Setup

```bash
# Create Blue (current) and Green (new) deployments
cat > blue-green-deployment.yaml <<'EOF'
---
# Blue Deployment (Current - v0.1.5)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foxhunt-trading-service-blue
  namespace: foxhunt
  labels:
    version: v0.1.5
    deployment: blue
spec:
  replicas: 5
  selector:
    matchLabels:
      app: trading-service
      version: v0.1.5
  template:
    metadata:
      labels:
        app: trading-service
        version: v0.1.5
    spec:
      containers:
      - name: trading-service
        image: ghcr.io/your-org/foxhunt/trading-service:v0.1.5
        ports:
        - containerPort: 50052

---
# Green Deployment (New - v0.1.6)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foxhunt-trading-service-green
  namespace: foxhunt
  labels:
    version: v0.1.6
    deployment: green
spec:
  replicas: 5
  selector:
    matchLabels:
      app: trading-service
      version: v0.1.6
  template:
    metadata:
      labels:
        app: trading-service
        version: v0.1.6
    spec:
      containers:
      - name: trading-service
        image: ghcr.io/your-org/foxhunt/trading-service:v0.1.6
        ports:
        - containerPort: 50052

---
# Service (points to Blue initially)
apiVersion: v1
kind: Service
metadata:
  name: foxhunt-trading-service
  namespace: foxhunt
spec:
  selector:
    app: trading-service
    version: v0.1.5  # Points to Blue
  ports:
  - port: 50052
    targetPort: 50052
EOF

kubectl apply -f blue-green-deployment.yaml
```

### Switch Traffic to Green

```bash
# Verify Green deployment is healthy
kubectl get pods -n foxhunt -l version=v0.1.6
# NAME                                             READY   STATUS    RESTARTS   AGE
# foxhunt-trading-service-green-abc123             1/1     Running   0          2m
# foxhunt-trading-service-green-def456             1/1     Running   0          2m
# foxhunt-trading-service-green-ghi789             1/1     Running   0          2m

# Run smoke tests on Green
kubectl port-forward -n foxhunt svc/foxhunt-trading-service-green 50052:50052 &
grpc_health_probe -addr=localhost:50052
# status: SERVING

# Switch service to Green (instant cutover)
kubectl patch service foxhunt-trading-service -n foxhunt \
  -p '{"spec":{"selector":{"version":"v0.1.6"}}}'

# Verify traffic switched
kubectl describe service foxhunt-trading-service -n foxhunt | grep Selector
# Selector:  app=trading-service,version=v0.1.6

# Monitor for 10 minutes, then delete Blue
sleep 600
kubectl delete deployment foxhunt-trading-service-blue -n foxhunt
```

### Rollback (if issues detected)

```bash
# Instant rollback to Blue
kubectl patch service foxhunt-trading-service -n foxhunt \
  -p '{"spec":{"selector":{"version":"v0.1.5"}}}'

# Traffic immediately returns to Blue
```

---

## Canary Deployment

### Istio Canary with Traffic Splitting

```bash
# Install Istio (if not already installed)
istioctl install --set profile=default -y

# Create VirtualService for canary
cat > canary-virtualservice.yaml <<'EOF'
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: trading-service-canary
  namespace: foxhunt
spec:
  hosts:
  - foxhunt-trading-service
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: foxhunt-trading-service
        subset: v0.1.6
      weight: 100
  - route:
    - destination:
        host: foxhunt-trading-service
        subset: v0.1.5
      weight: 95  # 95% to stable
    - destination:
        host: foxhunt-trading-service
        subset: v0.1.6
      weight: 5   # 5% to canary

---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: trading-service-canary
  namespace: foxhunt
spec:
  host: foxhunt-trading-service
  subsets:
  - name: v0.1.5
    labels:
      version: v0.1.5
  - name: v0.1.6
    labels:
      version: v0.1.6
EOF

kubectl apply -f canary-virtualservice.yaml
```

### Gradual Traffic Shift

```bash
# Start with 5% canary traffic
kubectl patch virtualservice trading-service-canary -n foxhunt --type=json \
  -p='[{"op": "replace", "path": "/spec/http/1/route/0/weight", "value": 95},
       {"op": "replace", "path": "/spec/http/1/route/1/weight", "value": 5}]'

# Monitor canary metrics for 15 minutes
# If metrics are good, increase to 25%
kubectl patch virtualservice trading-service-canary -n foxhunt --type=json \
  -p='[{"op": "replace", "path": "/spec/http/1/route/0/weight", "value": 75},
       {"op": "replace", "path": "/spec/http/1/route/1/weight", "value": 25}]'

# Continue incrementing: 5% → 25% → 50% → 75% → 100%

# Final cutover (100% to canary)
kubectl patch virtualservice trading-service-canary -n foxhunt --type=json \
  -p='[{"op": "replace", "path": "/spec/http/1/route/0/weight", "value": 0},
       {"op": "replace", "path": "/spec/http/1/route/1/weight", "value": 100}]'
```

### Automated Canary with Flagger

```bash
# Install Flagger
kubectl apply -k github.com/fluxcd/flagger//kustomize/istio

# Create Canary resource
cat > trading-service-canary.yaml <<'EOF'
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: foxhunt-trading-service
  namespace: foxhunt
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: foxhunt-trading-service
  service:
    port: 50052
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m
    webhooks:
    - name: load-test
      url: http://flagger-loadtester/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://foxhunt-trading-service:50052/"
EOF

kubectl apply -f trading-service-canary.yaml

# Flagger automatically manages canary rollout
# Monitor progress
kubectl describe canary foxhunt-trading-service -n foxhunt
```

---

## Database Migration with Zero Downtime

### Backward-Compatible Migrations

```sql
-- BAD: Breaking change (immediate column drop)
ALTER TABLE orders DROP COLUMN legacy_field;

-- GOOD: Multi-step migration
-- Step 1 (Deploy with code supporting both old/new schema)
ALTER TABLE orders ADD COLUMN new_field TEXT;
UPDATE orders SET new_field = legacy_field WHERE new_field IS NULL;

-- Step 2 (Wait 24 hours, deploy code using only new_field)
-- Step 3 (After verifying no errors, drop old column)
ALTER TABLE orders DROP COLUMN legacy_field;
```

### Migration Deployment Procedure

```bash
# Step 1: Apply additive migration
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
  psql -U foxhunt -d foxhunt < migrations/045_add_new_field.up.sql

# Step 2: Deploy application code (supports both schemas)
kubectl set image deployment/foxhunt-trading-service \
  trading-service=ghcr.io/your-org/foxhunt/trading-service:v0.1.6-migration \
  -n foxhunt

# Step 3: Wait 24 hours, verify no errors

# Step 4: Deploy final code (uses only new schema)
kubectl set image deployment/foxhunt-trading-service \
  trading-service=ghcr.io/your-org/foxhunt/trading-service:v0.1.6-final \
  -n foxhunt

# Step 5: Drop old column (after 7 days)
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
  psql -U foxhunt -d foxhunt < migrations/046_drop_legacy_field.up.sql
```

---

## Pre-Deployment Checklist

- [ ] All tests pass in CI/CD pipeline
- [ ] Database migrations are backward-compatible
- [ ] Rollback plan documented and tested
- [ ] Monitoring dashboards configured for new version
- [ ] Alert thresholds reviewed and updated
- [ ] On-call engineer notified and available
- [ ] Deployment window scheduled (prefer off-peak hours)
- [ ] Backup created within last 1 hour
- [ ] Canary deployment enabled for high-risk changes

---

**End of Zero-Downtime Deployment Guide**