f3012611f0
Replace standalone Prometheus/node-exporter/kube-state-metrics with kube-prometheus-stack Helm chart. Add ServiceMonitor CRDs, PrometheusRule CRDs (HFT + broker alerts), comprehensive network policies, and wire AlertManager to Mattermost via incoming webhook. - Delete old standalone prometheus.yaml, node-exporter.yaml, kube-state-metrics.yaml - Add prometheus-stack-values.yaml (Helm values with Scaleway Kapsule tuning) - Add service-monitors.yaml (foxhunt-services + dcgm-exporter ServiceMonitors) - Add prometheus-rules-hft.yaml and prometheus-rules-broker.yaml (PrometheusRule CRDs) - Rewrite prometheus network policy for operator-managed pods (DNS, kube-state-metrics, operator, alertmanager sidecar, monitoring namespace for DCGM) - Add alertmanager network policy (Mattermost egress) - Upgrade Grafana: AlertManager datasource, unified alerting, fix nodeSelector - Fix Tempo: tolerations for GPU nodes, resource tuning Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
44 lines
1.2 KiB
YAML
44 lines
1.2 KiB
YAML
# ServiceMonitors for Foxhunt services and GPU metrics
|
|
# Applied separately from the Helm chart (additionalServiceMonitors not rendered by chart)
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: foxhunt-services
|
|
namespace: foxhunt
|
|
labels:
|
|
app.kubernetes.io/part-of: foxhunt
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/part-of: foxhunt
|
|
namespaceSelector:
|
|
matchNames: [foxhunt]
|
|
endpoints:
|
|
- port: metrics
|
|
interval: 15s
|
|
# Override job label to "foxhunt-services" (matching old standalone config)
|
|
# so all existing dashboard queries continue to work
|
|
relabelings:
|
|
- targetLabel: job
|
|
replacement: foxhunt-services
|
|
---
|
|
# DCGM Exporter (GPU metrics from nvidia-gpu-operator DaemonSet)
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: ServiceMonitor
|
|
metadata:
|
|
name: dcgm-exporter
|
|
namespace: foxhunt
|
|
labels:
|
|
app.kubernetes.io/part-of: foxhunt
|
|
spec:
|
|
selector:
|
|
matchExpressions:
|
|
- key: app.kubernetes.io/name
|
|
operator: In
|
|
values: [dcgm-exporter]
|
|
namespaceSelector:
|
|
matchNames: [kube-system, monitoring, foxhunt]
|
|
endpoints:
|
|
- port: metrics
|
|
interval: 15s
|