jgrusewski
|
99f54e3f94
|
fix(infra): resolve postgres disk-full cascade, add SPOF mitigations
Postgres PVC hit 100% → crash-looped → GitLab webservice couldn't
verify SSH keys → all git operations failed. Root cause: 10Gi PVC
outgrown by GitLab database.
Fixes applied:
- Expand postgres PVC 10Gi → 20Gi
- Expand minio PVC 100Gi → 150Gi (was 81.8% full)
- Expand prometheus PVC 2Gi → 10Gi, retentionSize 1500MB → 8GB
- Scale gitlab-webservice to 2 replicas for HA
- Add PVC autoscaler CronJob (every 15min, auto-expand at 85%)
- Add daily postgres backup CronJob (pg_dump → MinIO, 7d + 4w retention)
- Add PrometheusRule storage alerts (75% warning, 90% critical)
- Add network policies for maintenance job egress
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-03-28 18:55:25 +01:00 |
|
jgrusewski
|
f3012611f0
|
feat(monitoring): migrate to kube-prometheus-stack Helm chart
Replace standalone Prometheus/node-exporter/kube-state-metrics with
kube-prometheus-stack Helm chart. Add ServiceMonitor CRDs, PrometheusRule
CRDs (HFT + broker alerts), comprehensive network policies, and wire
AlertManager to Mattermost via incoming webhook.
- Delete old standalone prometheus.yaml, node-exporter.yaml, kube-state-metrics.yaml
- Add prometheus-stack-values.yaml (Helm values with Scaleway Kapsule tuning)
- Add service-monitors.yaml (foxhunt-services + dcgm-exporter ServiceMonitors)
- Add prometheus-rules-hft.yaml and prometheus-rules-broker.yaml (PrometheusRule CRDs)
- Rewrite prometheus network policy for operator-managed pods (DNS, kube-state-metrics,
operator, alertmanager sidecar, monitoring namespace for DCGM)
- Add alertmanager network policy (Mattermost egress)
- Upgrade Grafana: AlertManager datasource, unified alerting, fix nodeSelector
- Fix Tempo: tolerations for GPU nodes, resource tuning
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
2026-03-14 11:33:46 +01:00 |
|