Files

jgrusewski 7458f1be01 feat(wave12): E2E validation complete - 225-feature pipeline ready

✅ Validation Results:
- PPO training: 24.2s (1 epoch, 950 samples, dim=225)
- Feature extraction: 105μs/bar (9.5x faster than target)
- Model checkpoint: 293KB (147KB actor + 146KB critic)
- GPU memory: 145MB used (96.4% headroom)
- Zero dimension mismatches

📊 Success Criteria (5/5):
✅ Feature dimension = 225 (Wave C 201 + Wave D 24)
✅ Model state_dim = 225
✅ Training completed without errors
✅ Checkpoint saved successfully
✅ No dimension mismatch errors

📁 Training Data Ready:
- ES.FUT: 2.9MB, 180 days
- NQ.FUT: 4.4MB, 180 days
- 6E.FUT: 2.8MB, 180 days
- ZN.FUT: 65KB, 90 days (clean)

🚀 Next: Full production model retraining (4 models, ~10min GPU time)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-22 22:48:04 +02:00

8.9 KiB

Raw Blame History

Zero-Downtime Deployment Guide

Last Updated: 2025-10-22 Target: Production environments Expected Downtime: 0 seconds Deployment Time: 15-25 minutes

Overview

This guide details strategies for deploying Foxhunt updates without service interruptions using rolling updates, blue-green deployments, and canary releases.

Rolling Update (Default Strategy)

Kubernetes RollingUpdate

# Deployment strategy configuration
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2         # Max 2 extra pods during update
      maxUnavailable: 1   # Max 1 pod unavailable during update

Deployment Procedure

# Update image tag
kubectl set image deployment/foxhunt-trading-service \
  trading-service=ghcr.io/your-org/foxhunt/trading-service:v0.1.6 \
  -n foxhunt

# Monitor rollout
kubectl rollout status deployment/foxhunt-trading-service -n foxhunt
# Waiting for deployment "foxhunt-trading-service" rollout to finish: 2 out of 5 new replicas have been updated...
# deployment "foxhunt-trading-service" successfully rolled out

# Verify no downtime
while true; do
  curl -sf http://api.foxhunt.example.com/health && echo " OK" || echo " FAILED"
  sleep 1
done

Blue-Green Deployment

Setup

# Create Blue (current) and Green (new) deployments
cat > blue-green-deployment.yaml <<'EOF'
---
# Blue Deployment (Current - v0.1.5)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foxhunt-trading-service-blue
  namespace: foxhunt
  labels:
    version: v0.1.5
    deployment: blue
spec:
  replicas: 5
  selector:
    matchLabels:
      app: trading-service
      version: v0.1.5
  template:
    metadata:
      labels:
        app: trading-service
        version: v0.1.5
    spec:
      containers:
      - name: trading-service
        image: ghcr.io/your-org/foxhunt/trading-service:v0.1.5
        ports:
        - containerPort: 50052

---
# Green Deployment (New - v0.1.6)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foxhunt-trading-service-green
  namespace: foxhunt
  labels:
    version: v0.1.6
    deployment: green
spec:
  replicas: 5
  selector:
    matchLabels:
      app: trading-service
      version: v0.1.6
  template:
    metadata:
      labels:
        app: trading-service
        version: v0.1.6
    spec:
      containers:
      - name: trading-service
        image: ghcr.io/your-org/foxhunt/trading-service:v0.1.6
        ports:
        - containerPort: 50052

---
# Service (points to Blue initially)
apiVersion: v1
kind: Service
metadata:
  name: foxhunt-trading-service
  namespace: foxhunt
spec:
  selector:
    app: trading-service
    version: v0.1.5  # Points to Blue
  ports:
  - port: 50052
    targetPort: 50052
EOF

kubectl apply -f blue-green-deployment.yaml

Switch Traffic to Green

# Verify Green deployment is healthy
kubectl get pods -n foxhunt -l version=v0.1.6
# NAME                                             READY   STATUS    RESTARTS   AGE
# foxhunt-trading-service-green-abc123             1/1     Running   0          2m
# foxhunt-trading-service-green-def456             1/1     Running   0          2m
# foxhunt-trading-service-green-ghi789             1/1     Running   0          2m

# Run smoke tests on Green
kubectl port-forward -n foxhunt svc/foxhunt-trading-service-green 50052:50052 &
grpc_health_probe -addr=localhost:50052
# status: SERVING

# Switch service to Green (instant cutover)
kubectl patch service foxhunt-trading-service -n foxhunt \
  -p '{"spec":{"selector":{"version":"v0.1.6"}}}'

# Verify traffic switched
kubectl describe service foxhunt-trading-service -n foxhunt | grep Selector
# Selector:  app=trading-service,version=v0.1.6

# Monitor for 10 minutes, then delete Blue
sleep 600
kubectl delete deployment foxhunt-trading-service-blue -n foxhunt

Rollback (if issues detected)

# Instant rollback to Blue
kubectl patch service foxhunt-trading-service -n foxhunt \
  -p '{"spec":{"selector":{"version":"v0.1.5"}}}'

# Traffic immediately returns to Blue

Canary Deployment

Istio Canary with Traffic Splitting

# Install Istio (if not already installed)
istioctl install --set profile=default -y

# Create VirtualService for canary
cat > canary-virtualservice.yaml <<'EOF'
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: trading-service-canary
  namespace: foxhunt
spec:
  hosts:
  - foxhunt-trading-service
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: foxhunt-trading-service
        subset: v0.1.6
      weight: 100
  - route:
    - destination:
        host: foxhunt-trading-service
        subset: v0.1.5
      weight: 95  # 95% to stable
    - destination:
        host: foxhunt-trading-service
        subset: v0.1.6
      weight: 5   # 5% to canary

---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: trading-service-canary
  namespace: foxhunt
spec:
  host: foxhunt-trading-service
  subsets:
  - name: v0.1.5
    labels:
      version: v0.1.5
  - name: v0.1.6
    labels:
      version: v0.1.6
EOF

kubectl apply -f canary-virtualservice.yaml

Gradual Traffic Shift

# Start with 5% canary traffic
kubectl patch virtualservice trading-service-canary -n foxhunt --type=json \
  -p='[{"op": "replace", "path": "/spec/http/1/route/0/weight", "value": 95},
       {"op": "replace", "path": "/spec/http/1/route/1/weight", "value": 5}]'

# Monitor canary metrics for 15 minutes
# If metrics are good, increase to 25%
kubectl patch virtualservice trading-service-canary -n foxhunt --type=json \
  -p='[{"op": "replace", "path": "/spec/http/1/route/0/weight", "value": 75},
       {"op": "replace", "path": "/spec/http/1/route/1/weight", "value": 25}]'

# Continue incrementing: 5% → 25% → 50% → 75% → 100%

# Final cutover (100% to canary)
kubectl patch virtualservice trading-service-canary -n foxhunt --type=json \
  -p='[{"op": "replace", "path": "/spec/http/1/route/0/weight", "value": 0},
       {"op": "replace", "path": "/spec/http/1/route/1/weight", "value": 100}]'

Automated Canary with Flagger

# Install Flagger
kubectl apply -k github.com/fluxcd/flagger//kustomize/istio

# Create Canary resource
cat > trading-service-canary.yaml <<'EOF'
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: foxhunt-trading-service
  namespace: foxhunt
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: foxhunt-trading-service
  service:
    port: 50052
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m
    webhooks:
    - name: load-test
      url: http://flagger-loadtester/
      timeout: 5s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://foxhunt-trading-service:50052/"
EOF

kubectl apply -f trading-service-canary.yaml

# Flagger automatically manages canary rollout
# Monitor progress
kubectl describe canary foxhunt-trading-service -n foxhunt

Database Migration with Zero Downtime

Backward-Compatible Migrations

-- BAD: Breaking change (immediate column drop)
ALTER TABLE orders DROP COLUMN legacy_field;

-- GOOD: Multi-step migration
-- Step 1 (Deploy with code supporting both old/new schema)
ALTER TABLE orders ADD COLUMN new_field TEXT;
UPDATE orders SET new_field = legacy_field WHERE new_field IS NULL;

-- Step 2 (Wait 24 hours, deploy code using only new_field)
-- Step 3 (After verifying no errors, drop old column)
ALTER TABLE orders DROP COLUMN legacy_field;

Migration Deployment Procedure

# Step 1: Apply additive migration
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
  psql -U foxhunt -d foxhunt < migrations/045_add_new_field.up.sql

# Step 2: Deploy application code (supports both schemas)
kubectl set image deployment/foxhunt-trading-service \
  trading-service=ghcr.io/your-org/foxhunt/trading-service:v0.1.6-migration \
  -n foxhunt

# Step 3: Wait 24 hours, verify no errors

# Step 4: Deploy final code (uses only new schema)
kubectl set image deployment/foxhunt-trading-service \
  trading-service=ghcr.io/your-org/foxhunt/trading-service:v0.1.6-final \
  -n foxhunt

# Step 5: Drop old column (after 7 days)
kubectl exec -n foxhunt foxhunt-postgresql-0 -- \
  psql -U foxhunt -d foxhunt < migrations/046_drop_legacy_field.up.sql

Pre-Deployment Checklist

All tests pass in CI/CD pipeline
Database migrations are backward-compatible
Rollback plan documented and tested
Monitoring dashboards configured for new version
Alert thresholds reviewed and updated
On-call engineer notified and available
Deployment window scheduled (prefer off-peak hours)
Backup created within last 1 hour
Canary deployment enabled for high-risk changes

End of Zero-Downtime Deployment Guide

8.9 KiB Raw Blame History