foxhunt

Files

jgrusewski 99f54e3f94 fix(infra): resolve postgres disk-full cascade, add SPOF mitigations

Postgres PVC hit 100% → crash-looped → GitLab webservice couldn't
verify SSH keys → all git operations failed. Root cause: 10Gi PVC
outgrown by GitLab database.

Fixes applied:
- Expand postgres PVC 10Gi → 20Gi
- Expand minio PVC 100Gi → 150Gi (was 81.8% full)
- Expand prometheus PVC 2Gi → 10Gi, retentionSize 1500MB → 8GB
- Scale gitlab-webservice to 2 replicas for HA
- Add PVC autoscaler CronJob (every 15min, auto-expand at 85%)
- Add daily postgres backup CronJob (pg_dump → MinIO, 7d + 4w retention)
- Add PrometheusRule storage alerts (75% warning, 90% critical)
- Add network policies for maintenance job egress

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-28 18:55:25 +01:00

dashboards

fix(dashboards): update datasource UIDs and folder annotations for Helm Prometheus

2026-03-14 11:34:14 +01:00

alloy-values.yaml

refactor(infra): update all manifests for pool rename (always-on→services, ci-build→ci-compile/ci-training)

2026-02-27 10:33:34 +01:00

APPLY-ORDER.md

fix(infra): remove unused L4 (ci-rl) pool — consolidate all training on ci-training

2026-03-01 14:02:54 +01:00

dcgm-exporter.yaml

fix(infra): replace gitlab.com/prometheus annotations with prometheus.io