jgrusewski
99f54e3f94
fix(infra): resolve postgres disk-full cascade, add SPOF mitigations
Postgres PVC hit 100% → crash-looped → GitLab webservice couldn't
verify SSH keys → all git operations failed. Root cause: 10Gi PVC
outgrown by GitLab database.
Fixes applied:
- Expand postgres PVC 10Gi → 20Gi
- Expand minio PVC 100Gi → 150Gi (was 81.8% full)
- Expand prometheus PVC 2Gi → 10Gi, retentionSize 1500MB → 8GB
- Scale gitlab-webservice to 2 replicas for HA
- Add PVC autoscaler CronJob (every 15min, auto-expand at 85%)
- Add daily postgres backup CronJob (pg_dump → MinIO, 7d + 4w retention)
- Add PrometheusRule storage alerts (75% warning, 90% critical)
- Add network policies for maintenance job egress
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>