From 4825866ac69b7aa92893ea06f9e7bbfad9dfd8eb Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 21 Jun 2026 18:50:43 +0200 Subject: [PATCH] =?UTF-8?q?docs(2E):=20include=20alertmanager=20(useful)?= =?UTF-8?q?=20=E2=80=94=20operator=20brings=20it=20up;=20receivers=20a=20f?= =?UTF-8?q?ollow-up?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/2026-06-21-observability-2e.md | 27 +++++++++---------- .../2026-06-21-observability-2e-design.md | 23 +++++++++------- 2 files changed, 26 insertions(+), 24 deletions(-) diff --git a/docs/superpowers/plans/2026-06-21-observability-2e.md b/docs/superpowers/plans/2026-06-21-observability-2e.md index c6408c8..b0b6ee1 100644 --- a/docs/superpowers/plans/2026-06-21-observability-2e.md +++ b/docs/superpowers/plans/2026-06-21-observability-2e.md @@ -18,26 +18,23 @@ netpols; add a dagster nginx block + DNS record. --- -## Task 1: Re-enable core metrics (prometheus + grafana + kube-state-metrics) +## Task 1: Re-enable core metrics (prometheus + alertmanager + grafana + kube-state-metrics) - [ ] **Step 1: Scale up the normal deployments** ```bash kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=1 ``` -Expected: 3 deployments scaled. +Expected: 3 deployments scaled. The operator then reconciles the Prometheus **and Alertmanager** CRs +(both `replicas=1`) → brings both statefulsets up automatically (no manual scale needed for them). -- [ ] **Step 2: Keep alertmanager OFF via its CR (operator would otherwise bring it to 1)** -```bash -kubectl patch alertmanager prometheus-stack-kube-prom-alertmanager -n foxhunt --type merge -p '{"spec":{"replicas":0}}' 2>&1 | tail -1 -``` -Expected: `alertmanager.monitoring.coreos.com/... patched` (or already 0). - -- [ ] **Step 3: Wait for the operator to bring prometheus up + pods ready** +- [ ] **Step 2: Wait for the operator to bring prometheus + alertmanager up + pods ready** ```bash i=0; until kubectl get sts prometheus-prometheus-stack-kube-prom-prometheus -n foxhunt -o jsonpath='{.status.readyReplicas}' 2>/dev/null | grep -q 1 || [ $i -ge 36 ]; do sleep 5; i=$((i+1)); done -kubectl get pods -n foxhunt | grep -iE "prometheus-stack-kube-prom-prometheus|kube-state-metrics|grafana|kube-prom-operator" | grep -v node-exporter | awk '{print $1,$3}' +kubectl get pods -n foxhunt | grep -iE "prometheus-stack-kube-prom-prometheus|alertmanager|kube-state-metrics|grafana|kube-prom-operator" | grep -v node-exporter | awk '{print $1,$3}' ``` -Expected: `prometheus-prometheus-stack-...-0`, `grafana`, `kube-state-metrics`, `kube-prom-operator` pods Running; alertmanager NOT present. +Expected: `prometheus-...-0`, `alertmanager-...-0`, `grafana`, `kube-state-metrics`, `kube-prom-operator` +pods Running. (If alertmanager stays at 0, its CR replicas may be 0 → `kubectl patch alertmanager +prometheus-stack-kube-prom-alertmanager -n foxhunt --type merge -p '{"spec":{"replicas":1}}'`.) --- @@ -174,7 +171,7 @@ cd /home/jgrusewski/Work/fxhnt git add infra/k8s/monitoring/grafana-netpol.yaml infra/k8s/orchestration/dagster.yaml infra/k8s/gitlab/tailscale-proxy.yaml git commit -m "feat(infra): re-enable prometheus+grafana, expose grafana.fxhnt.ai + dagster.fxhnt.ai (Phase 2E) -- scaled prometheus-stack operator/kube-state-metrics + grafana back to 1 (alertmanager kept 0) +- scaled prometheus-stack operator/kube-state-metrics + grafana back to 1; operator brings prometheus + alertmanager up - grafana-netpol: admit tailscale proxy -> grafana:3000 (was 502) - dagster.yaml netpol: + ingress proxy -> :3000; nginx dagster.fxhnt.ai block (ws); DNS A record @@ -194,8 +191,8 @@ GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master ## Task 5: Final verification ```bash -echo "=== pods (metrics core up, alertmanager/loki/tempo absent) ===" -kubectl get pods -n foxhunt | grep -iE "prometheus|grafana|kube-state" | grep -v node-exporter | awk '{print $1,$3}' +echo "=== pods (metrics core + alertmanager up; loki/tempo absent) ===" +kubectl get pods -n foxhunt | grep -iE "prometheus|alertmanager|grafana|kube-state" | grep -v node-exporter | awk '{print $1,$3}' echo "=== UIs ===" GP=$(kubectl get secret grafana-admin -n foxhunt -o jsonpath='{.data.admin-password}'|base64 -d 2>/dev/null) curl -sk -m 12 -o /dev/null -w "grafana login: %{http_code}\n" https://grafana.fxhnt.ai/login @@ -212,7 +209,7 @@ Expected: prometheus/grafana/kube-state pods Running; grafana 200; dagster 200/3 - Metrics: `kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=0` (back to dormant). ## Acceptance criteria (from spec) -- prometheus/kube-state-metrics/operator/grafana Running; alertmanager/loki/tempo stay 0. ✅ T1, T5 +- prometheus/kube-state-metrics/operator/alertmanager/grafana Running; loki/tempo stay 0. ✅ T1, T5 - grafana.fxhnt.ai 200 + prometheus datasource. ✅ T2, T5 - dagster.fxhnt.ai 200 (UI). ✅ T3, T5 - git/dashboard still 200. ✅ T3, T5 diff --git a/docs/superpowers/specs/2026-06-21-observability-2e-design.md b/docs/superpowers/specs/2026-06-21-observability-2e-design.md index 8b35831..531845a 100644 --- a/docs/superpowers/specs/2026-06-21-observability-2e-design.md +++ b/docs/superpowers/specs/2026-06-21-observability-2e-design.md @@ -16,13 +16,17 @@ proxy so they're reachable like `git`/`dashboard`/`minio`. ## Scope — re-enable (core metrics only) ON: `prometheus` (via operator), `prometheus-stack-kube-prom-operator`, -`prometheus-stack-kube-state-metrics`, `grafana`, node-exporters (already running). -OFF (stay at 0): alertmanager, loki, tempo, alloy, promtail, pushgateway, dcgm-exporter (GPU, dead). +`prometheus-stack-kube-state-metrics`, `alertmanager`, `grafana`, node-exporters (already running). +OFF (stay at 0): loki, tempo, alloy, promtail, pushgateway, dcgm-exporter (GPU, dead). -Re-enable **via the committed Helm values** (persistent), not a transient `kubectl scale`: -- `prometheus-stack`: ensure operator + kube-state-metrics + prometheus (prometheusSpec.replicas=1) on; - `alertmanager.enabled=false`. The operator reconciles the Prometheus CR → prometheus statefulset to 1. -- `grafana`: helm release values already `replicas: 1` (was manually scaled to 0) → restore to 1. +Re-enable by scaling the operator + kube-state-metrics + grafana deployments back to 1; the operator then +reconciles the Prometheus **and Alertmanager** CRs (both `replicas=1`) → brings those statefulsets up. The +configured prometheus-rules (hft/broker/storage) fire into Alertmanager. + +**Note — alert notification routing is a follow-up:** Alertmanager will run + collect firing alerts, but +*where* they get sent (Mattermost / email via Stalwart / webhook) needs a receiver config + channel choice. +2E gets Alertmanager running with rules active (visible in Grafana unified-alerting + the in-cluster +Alertmanager UI); wiring a notification receiver is a small follow-up. ## Scope — expose via tailnet proxy (`infra/k8s/gitlab/tailscale-proxy.yaml`) @@ -43,7 +47,7 @@ pod. Grafana → Prometheus (in-cluster) → scrapes node-exporters + kube-state ## Risks / safety -- **Resource:** ~1–1.5 GiB for prometheus+grafana+kube-state-metrics. Headroom exists (GitLab freed 5.5 GiB). +- **Resource:** ~1.5–2 GiB for prometheus+grafana+kube-state-metrics+alertmanager. Headroom exists (GitLab freed 5.5 GiB). - **Proxy edit:** adding two server blocks + restarting the proxy — verify `git`/`dashboard` stay 200 after (the proxy fronts everything). nginx resolves upstreams at boot, so the new upstreams (grafana/dagster svcs) must exist before restart — they do. @@ -53,10 +57,11 @@ pod. Grafana → Prometheus (in-cluster) → scrapes node-exporters + kube-state (same as the other `*.fxhnt.ai` services) — acceptable for a single-maintainer private platform. ## Out of scope -Logs (Loki), traces (Tempo), alerting (Alertmanager), GPU metrics (dcgm) — the heavier options not chosen. +Logs (Loki), traces (Tempo), GPU metrics (dcgm) — the heavier options not chosen. Alert notification +receiver/routing config (Mattermost/email/webhook) — a follow-up after Alertmanager is up. ## Acceptance criteria -- `prometheus`, `kube-state-metrics`, operator, `grafana` pods Running; alertmanager/loki/tempo stay 0. +- `prometheus`, `kube-state-metrics`, operator, `alertmanager`, `grafana` pods Running; loki/tempo stay 0. - `grafana.fxhnt.ai` returns 200 (login page); a dashboard shows live pod CPU/RAM; Prometheus targets up. - `dagster.fxhnt.ai` returns 200 (Dagster run UI loads, live updates work). - `git.fxhnt.ai` + `dashboard.fxhnt.ai` still 200 (proxy not regressed).