gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
46fea9a0e39fe82db8a7eda4cc43bdfd92cc299a
foxhunt/docs/deployment/kubernetes-deployment.md
jgrusewski 7458f1be01 feat(wave12): E2E validation complete - 225-feature pipeline ready 
✅ Validation Results:
- PPO training: 24.2s (1 epoch, 950 samples, dim=225)
- Feature extraction: 105μs/bar (9.5x faster than target)
- Model checkpoint: 293KB (147KB actor + 146KB critic)
- GPU memory: 145MB used (96.4% headroom)
- Zero dimension mismatches

📊 Success Criteria (5/5):
✅ Feature dimension = 225 (Wave C 201 + Wave D 24)
✅ Model state_dim = 225
✅ Training completed without errors
✅ Checkpoint saved successfully
✅ No dimension mismatch errors

📁 Training Data Ready:
- ES.FUT: 2.9MB, 180 days
- NQ.FUT: 4.4MB, 180 days
- 6E.FUT: 2.8MB, 180 days
- ZN.FUT: 65KB, 90 days (clean)

🚀 Next: Full production model retraining (4 models, ~10min GPU time)

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 22:48:04 +02:00

1145 lines
26 KiB
Markdown
Raw Blame History

# Kubernetes Deployment Guide
**Last Updated**: 2025-10-22
**Target Environment**: Production, Multi-Region
**Prerequisites**: Kubernetes 1.27+, Helm 3.12+, kubectl
**Deployment Time**: 30-45 minutes
---
## Table of Contents
1. [Overview](#overview)
2. [Prerequisites](#prerequisites)
3. [Cluster Setup](#cluster-setup)
4. [Helm Chart Deployment](#helm-chart-deployment)
5. [Service Configuration](#service-configuration)
6. [Scaling Configuration](#scaling-configuration)
7. [Health Checks and Readiness](#health-checks-and-readiness)
8. [Monitoring and Observability](#monitoring-and-observability)
9. [Troubleshooting](#troubleshooting)
---
## Overview
This guide covers deploying Foxhunt to Kubernetes using Helm charts. The deployment supports:
- **High Availability**: 3+ replicas per service
- **Auto-scaling**: HPA based on CPU, memory, and custom metrics
- **Service Mesh**: Istio for traffic management
- **Secrets Management**: External Secrets Operator + Vault
- **Observability**: Prometheus, Grafana, Jaeger
- **GPU Workloads**: ML training service with GPU node pools
### Architecture Diagram
```mermaid
graph TB
subgraph "Ingress Layer"
INGRESS[Ingress Controller<br/>NGINX/Istio]
end
subgraph "Application Layer"
direction LR
GATEWAY[API Gateway<br/>3 replicas]
TRADING[Trading Service<br/>5 replicas]
BACKTEST[Backtesting<br/>2 replicas]
ML[ML Training<br/>1 GPU replica]
AGENT[Trading Agent<br/>3 replicas]
end
subgraph "Data Layer"
POSTGRES[(PostgreSQL<br/>StatefulSet)]
REDIS[(Redis<br/>StatefulSet)]
end
subgraph "Platform Services"
VAULT[Vault]
PROMETHEUS[Prometheus]
GRAFANA[Grafana]
end
INGRESS --> GATEWAY
GATEWAY --> TRADING
GATEWAY --> BACKTEST
GATEWAY --> ML
GATEWAY --> AGENT
TRADING --> POSTGRES
TRADING --> REDIS
BACKTEST --> POSTGRES
ML --> POSTGRES
AGENT --> POSTGRES
AGENT --> REDIS
TRADING --> VAULT
BACKTEST --> VAULT
ML --> VAULT
AGENT --> VAULT
```
---
## Prerequisites
### 1. Cluster Requirements
**Minimum** (Development):
- **Nodes**: 3 worker nodes
- **vCPUs**: 8 cores per node (24 total)
- **RAM**: 16 GB per node (48 GB total)
- **Storage**: 100 GB SSD per node
- **Kubernetes**: 1.27+
**Recommended** (Production):
- **Nodes**: 6 worker nodes (3 app + 2 data + 1 GPU)
- **vCPUs**: 16 cores per node
- **RAM**: 32 GB per node
- **Storage**: 500 GB NVMe SSD per node
- **Kubernetes**: 1.28+
- **GPU Nodes**: 1+ nodes with NVIDIA T4/A100 GPUs
### 2. Software Requirements
```bash
# Install kubectl
curl -LO "https://dl.k8s.io/release/v1.28.0/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
# Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version
# Install Istio CLI (optional)
curl -L https://istio.io/downloadIstio | sh -
sudo mv istio-*/bin/istioctl /usr/local/bin/
istioctl version
```
### 3. Cluster Access Validation
```bash
# Verify cluster access
kubectl cluster-info
# Kubernetes control plane is running at https://...
# Check nodes
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# node-1 Ready master 30d v1.28.0
# node-2 Ready worker 30d v1.28.0
# node-3 Ready worker 30d v1.28.0
# Check available resources
kubectl top nodes
# NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
# node-2 500m 6% 4096Mi 25%
# node-3 500m 6% 4096Mi 25%
```
---
## Cluster Setup
### 1. Create Namespace
```bash
# Create foxhunt namespace
kubectl create namespace foxhunt
# Label namespace for monitoring
kubectl label namespace foxhunt monitoring=enabled
# Verify namespace
kubectl get namespace foxhunt
# NAME STATUS AGE
# foxhunt Active 1m
```
### 2. Configure Node Pools
**Application Node Pool** (general workloads):
```bash
# Label application nodes
kubectl label nodes node-2 node-3 node-4 workload-type=application
# Verify labels
kubectl get nodes -l workload-type=application
```
**Data Node Pool** (stateful workloads):
```bash
# Label data nodes (with local SSD)
kubectl label nodes node-5 node-6 workload-type=data
# Add taints to prevent non-data workloads
kubectl taint nodes node-5 node-6 workload-type=data:NoSchedule
```
**GPU Node Pool** (ML training):
```bash
# Label GPU nodes
kubectl label nodes gpu-node-1 workload-type=gpu accelerator=nvidia-t4
# Add taints for GPU workloads only
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
# Verify GPU availability
kubectl describe node gpu-node-1 | grep nvidia.com/gpu
# nvidia.com/gpu: 1
```
### 3. Install NVIDIA GPU Operator (if using GPUs)
```bash
# Add NVIDIA Helm repository
helm repo add nvidia https://nvidia.github.io/gpu-operator
helm repo update
# Install GPU Operator
helm install gpu-operator nvidia/gpu-operator \
--namespace gpu-operator-resources \
--create-namespace \
--set driver.enabled=true
# Verify GPU operator
kubectl get pods -n gpu-operator-resources
# NAME READY STATUS RESTARTS AGE
# gpu-feature-discovery-xxxxx 1/1 Running 0 2m
# gpu-operator-xxxxx 1/1 Running 0 2m
# nvidia-container-toolkit-daemonset-xxxxx 1/1 Running 0 2m
# nvidia-dcgm-exporter-xxxxx 1/1 Running 0 2m
# nvidia-device-plugin-daemonset-xxxxx 1/1 Running 0 2m
# nvidia-driver-daemonset-xxxxx 1/1 Running 0 2m
```
### 4. Install Istio Service Mesh (optional)
```bash
# Install Istio with minimal profile
istioctl install --set profile=default -y
# Enable sidecar injection for foxhunt namespace
kubectl label namespace foxhunt istio-injection=enabled
# Verify Istio installation
kubectl get pods -n istio-system
# NAME READY STATUS RESTARTS AGE
# istio-ingressgateway-xxxxx 1/1 Running 0 2m
# istiod-xxxxx 1/1 Running 0 2m
```
### 5. Install External Secrets Operator
```bash
# Add External Secrets Helm repository
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
# Install External Secrets Operator
helm install external-secrets \
external-secrets/external-secrets \
--namespace external-secrets-system \
--create-namespace
# Verify installation
kubectl get pods -n external-secrets-system
# NAME READY STATUS RESTARTS AGE
# external-secrets-xxxxx 1/1 Running 0 1m
# external-secrets-cert-controller-xxxxx 1/1 Running 0 1m
# external-secrets-webhook-xxxxx 1/1 Running 0 1m
```
---
## Helm Chart Deployment
### 1. Create Helm Chart Structure
```bash
# Create Helm chart directory
mkdir -p helm/foxhunt
cd helm/foxhunt
# Create chart structure
cat > Chart.yaml <<EOF
apiVersion: v2
name: foxhunt
description: Foxhunt HFT Trading System
type: application
version: 0.1.0
appVersion: "0.1.0"
dependencies:
- name: postgresql
version: 12.10.0
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
- name: redis
version: 18.0.0
repository: https://charts.bitnami.com/bitnami
condition: redis.enabled
EOF
# Create default values
cat > values.yaml <<EOF
# Global configuration
global:
environment: production
imageRegistry: ghcr.io/your-org
imagePullPolicy: IfNotPresent
storageClass: fast-ssd
# API Gateway
apiGateway:
enabled: true
replicaCount: 3
image:
repository: foxhunt/api-gateway
tag: "0.1.0"
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 1Gi
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
service:
type: ClusterIP
grpcPort: 50051
httpPort: 8080
metricsPort: 9091
# Trading Service
tradingService:
enabled: true
replicaCount: 5
image:
repository: foxhunt/trading-service
tag: "0.1.0"
resources:
requests:
cpu: 1000m
memory: 1Gi
limits:
cpu: 4000m
memory: 2Gi
autoscaling:
enabled: true
minReplicas: 5
maxReplicas: 20
targetCPUUtilizationPercentage: 70
service:
type: ClusterIP
grpcPort: 50052
httpPort: 8081
metricsPort: 9092
# Backtesting Service
backtestingService:
enabled: true
replicaCount: 2
image:
repository: foxhunt/backtesting-service
tag: "0.1.0"
resources:
requests:
cpu: 2000m
memory: 2Gi
limits:
cpu: 8000m
memory: 8Gi
autoscaling:
enabled: false # CPU-intensive, fixed replicas
service:
type: ClusterIP
grpcPort: 50053
httpPort: 8082
metricsPort: 9093
# ML Training Service
mlTrainingService:
enabled: true
replicaCount: 1
image:
repository: foxhunt/ml-training-service
tag: "0.1.0"
resources:
requests:
cpu: 4000m
memory: 8Gi
nvidia.com/gpu: 1
limits:
cpu: 16000m
memory: 16Gi
nvidia.com/gpu: 1
nodeSelector:
workload-type: gpu
accelerator: nvidia-t4
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
service:
type: ClusterIP
grpcPort: 50054
httpPort: 8095
metricsPort: 9094
# Trading Agent Service
tradingAgent:
enabled: true
replicaCount: 3
image:
repository: foxhunt/trading-agent
tag: "0.1.0"
resources:
requests:
cpu: 1000m
memory: 1Gi
limits:
cpu: 4000m
memory: 2Gi
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 15
targetCPUUtilizationPercentage: 70
service:
type: ClusterIP
grpcPort: 50055
httpPort: 8083
metricsPort: 9095
# PostgreSQL (TimescaleDB)
postgresql:
enabled: true
auth:
username: foxhunt
password: "" # Set via external secret
database: foxhunt
primary:
persistence:
enabled: true
size: 100Gi
storageClass: fast-ssd
resources:
requests:
cpu: 2000m
memory: 4Gi
limits:
cpu: 8000m
memory: 16Gi
nodeSelector:
workload-type: data
tolerations:
- key: workload-type
operator: Equal
value: data
effect: NoSchedule
# Redis
redis:
enabled: true
auth:
enabled: true
password: "" # Set via external secret
master:
persistence:
enabled: true
size: 10Gi
storageClass: fast-ssd
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 2Gi
# Vault (external - not managed by this chart)
vault:
enabled: false
address: "http://vault.vault-system.svc.cluster.local:8200"
# Monitoring
monitoring:
enabled: true
prometheus:
enabled: true
grafana:
enabled: true
# Ingress
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/backend-protocol: GRPC
hosts:
- host: api.foxhunt.example.com
paths:
- path: /
pathType: Prefix
service:
name: api-gateway
port: 50051
tls:
- secretName: foxhunt-tls
hosts:
- api.foxhunt.example.com
EOF
```
### 2. Create Kubernetes Manifests
**API Gateway Deployment**:
```bash
cat > templates/api-gateway-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "foxhunt.fullname" . }}-api-gateway
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
app.kubernetes.io/component: api-gateway
spec:
{{- if not .Values.apiGateway.autoscaling.enabled }}
replicas: {{ .Values.apiGateway.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "foxhunt.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: api-gateway
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9091"
prometheus.io/path: "/metrics"
labels:
{{- include "foxhunt.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: api-gateway
spec:
serviceAccountName: {{ include "foxhunt.serviceAccountName" . }}
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
containers:
- name: api-gateway
image: "{{ .Values.global.imageRegistry }}/{{ .Values.apiGateway.image.repository }}:{{ .Values.apiGateway.image.tag }}"
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
ports:
- name: grpc
containerPort: 50051
protocol: TCP
- name: http
containerPort: 8080
protocol: TCP
- name: metrics
containerPort: 9091
protocol: TCP
env:
- name: RUST_LOG
value: "info"
- name: ENVIRONMENT
value: {{ .Values.global.environment }}
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: foxhunt-secrets
key: database-url
- name: REDIS_URL
valueFrom:
secretKeyRef:
name: foxhunt-secrets
key: redis-url
- name: VAULT_ADDR
value: {{ .Values.vault.address }}
- name: VAULT_TOKEN
valueFrom:
secretKeyRef:
name: foxhunt-secrets
key: vault-token
livenessProbe:
exec:
command:
- grpc_health_probe
- -addr=:50051
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
exec:
command:
- grpc_health_probe
- -addr=:50051
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 2
resources:
{{- toYaml .Values.apiGateway.resources | nindent 10 }}
volumeMounts:
- name: config
mountPath: /app/config
readOnly: true
volumes:
- name: config
configMap:
name: {{ include "foxhunt.fullname" . }}-config
EOF
```
**HorizontalPodAutoscaler**:
```bash
cat > templates/api-gateway-hpa.yaml <<'EOF'
{{- if .Values.apiGateway.autoscaling.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "foxhunt.fullname" . }}-api-gateway
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
app.kubernetes.io/component: api-gateway
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "foxhunt.fullname" . }}-api-gateway
minReplicas: {{ .Values.apiGateway.autoscaling.minReplicas }}
maxReplicas: {{ .Values.apiGateway.autoscaling.maxReplicas }}
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.apiGateway.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
- type: Pods
pods:
metric:
name: grpc_requests_per_second
target:
type: AverageValue
averageValue: "100"
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50
periodSeconds: 60
- type: Pods
value: 2
periodSeconds: 60
selectPolicy: Min
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
- type: Pods
value: 4
periodSeconds: 15
selectPolicy: Max
{{- end }}
EOF
```
**Service**:
```bash
cat > templates/api-gateway-service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: {{ include "foxhunt.fullname" . }}-api-gateway
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
app.kubernetes.io/component: api-gateway
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9091"
spec:
type: {{ .Values.apiGateway.service.type }}
ports:
- port: {{ .Values.apiGateway.service.grpcPort }}
targetPort: grpc
protocol: TCP
name: grpc
- port: {{ .Values.apiGateway.service.httpPort }}
targetPort: http
protocol: TCP
name: http
- port: {{ .Values.apiGateway.service.metricsPort }}
targetPort: metrics
protocol: TCP
name: metrics
selector:
{{- include "foxhunt.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: api-gateway
EOF
```
**PodDisruptionBudget**:
```bash
cat > templates/api-gateway-pdb.yaml <<'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ include "foxhunt.fullname" . }}-api-gateway
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
app.kubernetes.io/component: api-gateway
spec:
minAvailable: 2
selector:
matchLabels:
{{- include "foxhunt.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: api-gateway
EOF
```
### 3. Create External Secret for Vault Integration
```bash
cat > templates/external-secret.yaml <<'EOF'
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: {{ include "foxhunt.fullname" . }}-secrets
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: foxhunt-secrets
creationPolicy: Owner
data:
- secretKey: database-url
remoteRef:
key: secret/foxhunt/database
property: url
- secretKey: redis-url
remoteRef:
key: secret/foxhunt/redis
property: url
- secretKey: vault-token
remoteRef:
key: secret/foxhunt/vault
property: token
- secretKey: jwt-secret
remoteRef:
key: secret/foxhunt/jwt
property: secret
EOF
```
### 4. Deploy Helm Chart
```bash
# Add dependencies
helm dependency update
# Dry-run deployment
helm install foxhunt . \
--namespace foxhunt \
--dry-run \
--debug
# Install chart
helm install foxhunt . \
--namespace foxhunt \
--create-namespace \
--values values.yaml \
--set postgresql.auth.password=$(openssl rand -base64 32) \
--set redis.auth.password=$(openssl rand -base64 32) \
--timeout 10m \
--wait
# Verify deployment
helm list -n foxhunt
# NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
# foxhunt foxhunt 1 2025-10-22 10:00:00.000000000 +0000 UTC deployed foxhunt-0.1.0 0.1.0
# Check pod status
kubectl get pods -n foxhunt
# NAME READY STATUS RESTARTS AGE
# foxhunt-api-gateway-xxxxx 1/1 Running 0 2m
# foxhunt-trading-service-xxxxx 1/1 Running 0 2m
# foxhunt-backtesting-service-xxxxx 1/1 Running 0 2m
# foxhunt-ml-training-service-xxxxx 1/1 Running 0 2m
# foxhunt-trading-agent-xxxxx 1/1 Running 0 2m
# foxhunt-postgresql-0 1/1 Running 0 2m
# foxhunt-redis-master-0 1/1 Running 0 2m
```
---
## Service Configuration
### 1. ConfigMap for Application Configuration
```bash
cat > templates/configmap.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "foxhunt.fullname" . }}-config
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
data:
config.toml: |
[server]
environment = "{{ .Values.global.environment }}"
[logging]
level = "info"
format = "json"
[metrics]
enabled = true
prometheus_port = 9091
[health]
enabled = true
http_port = 8080
[trading]
max_position_size = 1000
risk_limit_pct = 2.0
[ml]
model_path = "/app/ml/trained_models"
feature_count = 225
gpu_enabled = true
EOF
kubectl apply -f templates/configmap.yaml -n foxhunt
```
### 2. ServiceAccount with RBAC
```bash
cat > templates/serviceaccount.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "foxhunt.serviceAccountName" . }}
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ include "foxhunt.fullname" . }}
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ include "foxhunt.fullname" . }}
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ include "foxhunt.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ include "foxhunt.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
EOF
kubectl apply -f templates/serviceaccount.yaml -n foxhunt
```
---
## Scaling Configuration
### 1. Vertical Pod Autoscaler (VPA)
```bash
# Install VPA (if not already installed)
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh
# Create VPA for Trading Service
cat > vpa-trading-service.yaml <<'EOF'
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: foxhunt-trading-service-vpa
namespace: foxhunt
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: foxhunt-trading-service
updatePolicy:
updateMode: "Auto" # Or "Recreate" for stateful services
resourcePolicy:
containerPolicies:
- containerName: trading-service
minAllowed:
cpu: 500m
memory: 512Mi
maxAllowed:
cpu: 8000m
memory: 8Gi
controlledResources:
- cpu
- memory
EOF
kubectl apply -f vpa-trading-service.yaml
```
### 2. Cluster Autoscaler Configuration
```bash
# Configure Cluster Autoscaler (AWS EKS example)
cat > cluster-autoscaler.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: cluster-autoscaler
template:
metadata:
labels:
app: cluster-autoscaler
spec:
serviceAccountName: cluster-autoscaler
containers:
- image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.28.0
name: cluster-autoscaler
command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/foxhunt
- --balance-similar-node-groups
- --skip-nodes-with-system-pods=false
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 100m
memory: 300Mi
EOF
kubectl apply -f cluster-autoscaler.yaml
```
### 3. PodPriority for Critical Services
```bash
cat > pod-priority-classes.yaml <<'EOF'
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: foxhunt-critical
value: 1000000
globalDefault: false
description: "Critical services: API Gateway, Trading Service"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: foxhunt-high
value: 100000
globalDefault: false
description: "High priority: Trading Agent, ML Training"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: foxhunt-normal
value: 10000
globalDefault: true
description: "Normal priority: Backtesting"
EOF
kubectl apply -f pod-priority-classes.yaml
# Update deployments to use priority classes
# Add to Deployment spec.template.spec:
# priorityClassName: foxhunt-critical
```
---
## Health Checks and Readiness
### 1. Liveness and Readiness Probes
**gRPC Health Probe** (already configured in deployment):
```yaml
livenessProbe:
exec:
command:
- grpc_health_probe
- -addr=:50051
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
exec:
command:
- grpc_health_probe
- -addr=:50051
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 2
```
### 2. Startup Probe for Slow-Starting Services
```yaml
# Add to ML Training Service deployment
startupProbe:
exec:
command:
- grpc_health_probe
- -addr=:50054
initialDelaySeconds: 0
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 30 # 5 minutes max startup time
```
---
## Monitoring and Observability
### 1. ServiceMonitor for Prometheus
```bash
cat > templates/servicemonitor.yaml <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "foxhunt.fullname" . }}
labels:
{{- include "foxhunt.labels" . | nindent 4 }}
spec:
selector:
matchLabels:
{{- include "foxhunt.selectorLabels" . | nindent 6 }}
endpoints:
- port: metrics
interval: 15s
path: /metrics
namespaceSelector:
matchNames:
- {{ .Release.Namespace }}
EOF
kubectl apply -f templates/servicemonitor.yaml -n foxhunt
```
### 2. Grafana Dashboards
See [grafana-setup.md](../monitoring/grafana-setup.md) for dashboard configuration.
---
## Troubleshooting
### 1. Pod Stuck in Pending State
```bash
# Describe pod to see events
kubectl describe pod <pod-name> -n foxhunt
# Common issues:
# - Insufficient resources: Check node capacity
kubectl top nodes
# - Unschedulable due to taints: Check tolerations
kubectl get pod <pod-name> -n foxhunt -o yaml | grep -A 10 tolerations
# - PVC not bound: Check PersistentVolumeClaims
kubectl get pvc -n foxhunt
```
### 2. CrashLoopBackOff
```bash
# Check logs
kubectl logs <pod-name> -n foxhunt --previous
# Common issues:
# - Database connection failed: Verify DATABASE_URL secret
kubectl get secret foxhunt-secrets -n foxhunt -o yaml
# - Missing GPU: Verify GPU node availability
kubectl get nodes -l accelerator=nvidia-t4
```
### 3. High Memory Usage
```bash
# Check memory usage
kubectl top pods -n foxhunt --sort-by=memory
# Adjust memory limits
helm upgrade foxhunt . \
--namespace foxhunt \
--set tradingService.resources.limits.memory=4Gi \
--reuse-values
```
---
**End of Kubernetes Deployment Guide**
Reference in New Issue View Git Blame Copy Permalink