99f54e3f94
Postgres PVC hit 100% → crash-looped → GitLab webservice couldn't verify SSH keys → all git operations failed. Root cause: 10Gi PVC outgrown by GitLab database. Fixes applied: - Expand postgres PVC 10Gi → 20Gi - Expand minio PVC 100Gi → 150Gi (was 81.8% full) - Expand prometheus PVC 2Gi → 10Gi, retentionSize 1500MB → 8GB - Scale gitlab-webservice to 2 replicas for HA - Add PVC autoscaler CronJob (every 15min, auto-expand at 85%) - Add daily postgres backup CronJob (pg_dump → MinIO, 7d + 4w retention) - Add PrometheusRule storage alerts (75% warning, 90% critical) - Add network policies for maintenance job egress Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
57 lines
2.6 KiB
YAML
57 lines
2.6 KiB
YAML
# Storage alert rules — volume usage warnings before auto-expand kicks in
|
|
# Apply: kubectl apply -f prometheus-rules-storage.yaml
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: foxhunt-storage-alerts
|
|
namespace: foxhunt
|
|
labels:
|
|
app.kubernetes.io/part-of: foxhunt
|
|
role: alert-rules
|
|
spec:
|
|
groups:
|
|
- name: storage_volume_usage
|
|
interval: 60s
|
|
rules:
|
|
# PVC usage > 75% — warning (auto-expand triggers at 85%)
|
|
- alert: PVCUsageHigh
|
|
expr: |
|
|
100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
|
|
/ kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 75
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
component: storage
|
|
annotations:
|
|
summary: "PVC {{ $labels.persistentvolumeclaim }} at {{ $value | printf \"%.0f\" }}% usage"
|
|
description: "Volume {{ $labels.persistentvolumeclaim }} is {{ $value | printf \"%.0f\" }}% full. Auto-expand triggers at 85%."
|
|
|
|
# PVC usage > 90% — critical (auto-expand should have already acted)
|
|
- alert: PVCUsageCritical
|
|
expr: |
|
|
100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
|
|
/ kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 90
|
|
for: 2m
|
|
labels:
|
|
severity: critical
|
|
component: storage
|
|
annotations:
|
|
summary: "PVC {{ $labels.persistentvolumeclaim }} CRITICAL at {{ $value | printf \"%.0f\" }}%"
|
|
description: "Volume {{ $labels.persistentvolumeclaim }} is {{ $value | printf \"%.0f\" }}% full despite auto-expand. Investigate immediately."
|
|
action: "Check pvc-autoscaler CronJob logs. Manual expand: kubectl patch pvc {{ $labels.persistentvolumeclaim }} -n foxhunt --type merge -p '{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"XXGi\"}}}}'"
|
|
|
|
# PVC approaching max auto-expand limit (>80Gi of 100Gi max)
|
|
- alert: PVCApproachingMaxSize
|
|
expr: |
|
|
kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 80 * 1024 * 1024 * 1024
|
|
and
|
|
100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
|
|
/ kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 70
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
component: storage
|
|
annotations:
|
|
summary: "PVC {{ $labels.persistentvolumeclaim }} is large and growing"
|
|
description: "Volume is >80Gi and >70% full. Auto-expand max is 100Gi. Consider manual intervention or data cleanup."
|