jgrusewski 99f54e3f94 fix(infra): resolve postgres disk-full cascade, add SPOF mitigations ...

Postgres PVC hit 100% → crash-looped → GitLab webservice couldn't
verify SSH keys → all git operations failed. Root cause: 10Gi PVC
outgrown by GitLab database.

Fixes applied:
- Expand postgres PVC 10Gi → 20Gi
- Expand minio PVC 100Gi → 150Gi (was 81.8% full)
- Expand prometheus PVC 2Gi → 10Gi, retentionSize 1500MB → 8GB
- Scale gitlab-webservice to 2 replicas for HA
- Add PVC autoscaler CronJob (every 15min, auto-expand at 85%)
- Add daily postgres backup CronJob (pg_dump → MinIO, 7d + 4w retention)
- Add PrometheusRule storage alerts (75% warning, 90% critical)
- Add network policies for maintenance job egress

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-28 18:55:25 +01:00

..

argo

fix: stop deleting feature cache on every hyperopt run

2026-03-25 22:03:48 +01:00

databases

fix(infra): resolve postgres disk-full cascade, add SPOF mitigations

2026-03-28 18:55:25 +01:00

dns

fix(infra): node-bootstrap DaemonSet + GitLab package auth + latest tag

2026-03-07 21:53:31 +01:00

gitlab

fix(infra): resolve postgres disk-full cascade, add SPOF mitigations

2026-03-28 18:55:25 +01:00

gpu-overlays

fix(infra): replace gitlab.com/prometheus annotations with prometheus.io

2026-03-08 18:39:24 +01:00

jobs

feat(infra): migrate container registry from SCW to internal GitLab

2026-03-05 13:52:24 +01:00

mattermost

feat(infra): deploy Mattermost on platform nodes with CI webhook egress

2026-03-12 09:20:32 +01:00

minio

fix(infra): resolve postgres disk-full cascade, add SPOF mitigations

2026-03-28 18:55:25 +01:00

monitoring

fix(infra): resolve postgres disk-full cascade, add SPOF mitigations

2026-03-28 18:55:25 +01:00

network-policies

fix(infra): resolve postgres disk-full cascade, add SPOF mitigations

2026-03-28 18:55:25 +01:00

secrets

feat(infra): QuestDB metrics sink + monitoring network policies

2026-03-08 02:42:53 +01:00

services

fix: tune log levels across workspace — demote noisy warn to debug/trace

2026-03-14 11:35:15 +01:00

stalwart

feat(stalwart): fallback-admin via K8s secret for reliable auth

2026-03-12 23:01:42 +01:00

storage

feat(infra): replace PVC binary distribution with MinIO S3 fetch

2026-03-04 16:46:57 +01:00

tailscale

feat(infra): migrate container registry from SCW to internal GitLab

2026-03-05 13:52:24 +01:00

training

feat: TOML training profile system — config-driven hyperparameters

2026-03-21 23:35:41 +01:00

gpu-taint.yaml

infra: split GPU pool into training (H100) and inference (L4), add auto-taint

2026-02-24 17:39:56 +01:00

namespace.yaml

fix: comprehensive audit — real brokers, deployment fixes, production safety

2026-02-25 00:32:10 +01:00