gitadmin/fxhnt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(2E): include alertmanager (useful) — operator brings it up; receivers a follow-up

Browse Source
This commit is contained in:
jgrusewski
2026-06-21 18:50:43 +02:00
parent df1b3409b5
commit 4825866ac6
2 changed files with 26 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
docs/superpowers/plans/2026-06-21-observability-2e.md
View File

@@ -18,26 +18,23 @@ netpols; add a dagster nginx block + DNS record.
---
## Task 1: Re-enable core metrics (prometheus + grafana + kube-state-metrics)
## Task 1: Re-enable core metrics (prometheus + alertmanager + grafana + kube-state-metrics)
- [ ] **Step 1: Scale up the normal deployments**
```bash
kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=1
```
Expected: 3 deployments scaled.
Expected: 3 deployments scaled. The operator then reconciles the Prometheus **and Alertmanager** CRs
(both `replicas=1`) → brings both statefulsets up automatically (no manual scale needed for them).
- [ ] **Step 2: Keep alertmanager OFF via its CR (operator would otherwise bring it to 1)**
```bash
kubectl patch alertmanager prometheus-stack-kube-prom-alertmanager -n foxhunt --type merge -p '{"spec":{"replicas":0}}' 2>&1 | tail -1
```
Expected: `alertmanager.monitoring.coreos.com/... patched` (or already 0).
- [ ] **Step 3: Wait for the operator to bring prometheus up + pods ready**
- [ ] **Step 2: Wait for the operator to bring prometheus + alertmanager up + pods ready**
```bash
i=0; until kubectl get sts prometheus-prometheus-stack-kube-prom-prometheus -n foxhunt -o jsonpath='{.status.readyReplicas}' 2>/dev/null | grep -q 1 || [ $i -ge 36 ]; do sleep 5; i=$((i+1)); done
kubectl get pods -n foxhunt | grep -iE "prometheus-stack-kube-prom-prometheus|kube-state-metrics|grafana|kube-prom-operator" | grep -v node-exporter | awk '{print $1,$3}'
kubectl get pods -n foxhunt | grep -iE "prometheus-stack-kube-prom-prometheus|alertmanager|kube-state-metrics|grafana|kube-prom-operator" | grep -v node-exporter | awk '{print $1,$3}'
```
Expected: `prometheus-prometheus-stack-...-0`, `grafana`, `kube-state-metrics`, `kube-prom-operator` pods Running; alertmanager NOT present.
Expected: `prometheus-...-0`, `alertmanager-...-0`, `grafana`, `kube-state-metrics`, `kube-prom-operator`
pods Running. (If alertmanager stays at 0, its CR replicas may be 0 → `kubectl patch alertmanager
prometheus-stack-kube-prom-alertmanager -n foxhunt --type merge -p '{"spec":{"replicas":1}}'`.)
---
@@ -174,7 +171,7 @@ cd /home/jgrusewski/Work/fxhnt
git add infra/k8s/monitoring/grafana-netpol.yaml infra/k8s/orchestration/dagster.yaml infra/k8s/gitlab/tailscale-proxy.yaml
git commit -m "feat(infra): re-enable prometheus+grafana, expose grafana.fxhnt.ai + dagster.fxhnt.ai (Phase 2E)
- scaled prometheus-stack operator/kube-state-metrics + grafana back to 1 (alertmanager kept 0)
- scaled prometheus-stack operator/kube-state-metrics + grafana back to 1; operator brings prometheus + alertmanager up
- grafana-netpol: admit tailscale proxy -> grafana:3000 (was 502)
- dagster.yaml netpol: + ingress proxy -> :3000; nginx dagster.fxhnt.ai block (ws); DNS A record
@@ -194,8 +191,8 @@ GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master
## Task 5: Final verification
```bash
echo "=== pods (metrics core up, alertmanager/loki/tempo absent) ==="
kubectl get pods -n foxhunt | grep -iE "prometheus|grafana|kube-state" | grep -v node-exporter | awk '{print $1,$3}'
echo "=== pods (metrics core + alertmanager up; loki/tempo absent) ==="
kubectl get pods -n foxhunt | grep -iE "prometheus|alertmanager|grafana|kube-state" | grep -v node-exporter | awk '{print $1,$3}'
echo "=== UIs ==="
GP=$(kubectl get secret grafana-admin -n foxhunt -o jsonpath='{.data.admin-password}'|base64 -d 2>/dev/null)
curl -sk -m 12 -o /dev/null -w "grafana login: %{http_code}\n" https://grafana.fxhnt.ai/login
@@ -212,7 +209,7 @@ Expected: prometheus/grafana/kube-state pods Running; grafana 200; dagster 200/3
- Metrics: `kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=0` (back to dormant).
## Acceptance criteria (from spec)
- prometheus/kube-state-metrics/operator/grafana Running; alertmanager/loki/tempo stay 0. ✅ T1, T5
- prometheus/kube-state-metrics/operator/alertmanager/grafana Running; loki/tempo stay 0. ✅ T1, T5
- grafana.fxhnt.ai 200 + prometheus datasource. ✅ T2, T5
- dagster.fxhnt.ai 200 (UI). ✅ T3, T5
- git/dashboard still 200. ✅ T3, T5

23
docs/superpowers/specs/2026-06-21-observability-2e-design.md
View File

@@ -16,13 +16,17 @@ proxy so they're reachable like `git`/`dashboard`/`minio`.
## Scope — re-enable (core metrics only)
ON: `prometheus` (via operator), `prometheus-stack-kube-prom-operator`,
`prometheus-stack-kube-state-metrics`, `grafana`, node-exporters (already running).
OFF (stay at 0): alertmanager, loki, tempo, alloy, promtail, pushgateway, dcgm-exporter (GPU, dead).
`prometheus-stack-kube-state-metrics`, `alertmanager`, `grafana`, node-exporters (already running).
OFF (stay at 0): loki, tempo, alloy, promtail, pushgateway, dcgm-exporter (GPU, dead).
Re-enable **via the committed Helm values** (persistent), not a transient `kubectl scale`:
- `prometheus-stack`: ensure operator + kube-state-metrics + prometheus (prometheusSpec.replicas=1) on;
`alertmanager.enabled=false`. The operator reconciles the Prometheus CR → prometheus statefulset to 1.
- `grafana`: helm release values already `replicas: 1` (was manually scaled to 0) → restore to 1.
Re-enable by scaling the operator + kube-state-metrics + grafana deployments back to 1; the operator then
reconciles the Prometheus **and Alertmanager** CRs (both `replicas=1`) → brings those statefulsets up. The
configured prometheus-rules (hft/broker/storage) fire into Alertmanager.
**Note — alert notification routing is a follow-up:** Alertmanager will run + collect firing alerts, but
*where* they get sent (Mattermost / email via Stalwart / webhook) needs a receiver config + channel choice.
2E gets Alertmanager running with rules active (visible in Grafana unified-alerting + the in-cluster
Alertmanager UI); wiring a notification receiver is a small follow-up.
## Scope — expose via tailnet proxy (`infra/k8s/gitlab/tailscale-proxy.yaml`)
@@ -43,7 +47,7 @@ pod. Grafana → Prometheus (in-cluster) → scrapes node-exporters + kube-state
## Risks / safety
- **Resource:** ~1–1.5 GiB for prometheus+grafana+kube-state-metrics. Headroom exists (GitLab freed 5.5 GiB).
- **Resource:** ~1.5–2 GiB for prometheus+grafana+kube-state-metrics+alertmanager. Headroom exists (GitLab freed 5.5 GiB).
- **Proxy edit:** adding two server blocks + restarting the proxy — verify `git`/`dashboard` stay 200
after (the proxy fronts everything). nginx resolves upstreams at boot, so the new upstreams
(grafana/dagster svcs) must exist before restart — they do.
@@ -53,10 +57,11 @@ pod. Grafana → Prometheus (in-cluster) → scrapes node-exporters + kube-state
(same as the other `*.fxhnt.ai` services) — acceptable for a single-maintainer private platform.
## Out of scope
Logs (Loki), traces (Tempo), alerting (Alertmanager), GPU metrics (dcgm) — the heavier options not chosen.
Logs (Loki), traces (Tempo), GPU metrics (dcgm) — the heavier options not chosen. Alert notification
receiver/routing config (Mattermost/email/webhook) — a follow-up after Alertmanager is up.
## Acceptance criteria
- `prometheus`, `kube-state-metrics`, operator, `grafana` pods Running; alertmanager/loki/tempo stay 0.
- `prometheus`, `kube-state-metrics`, operator, `alertmanager`, `grafana` pods Running; loki/tempo stay 0.
- `grafana.fxhnt.ai` returns 200 (login page); a dashboard shows live pod CPU/RAM; Prometheus targets up.
- `dagster.fxhnt.ai` returns 200 (Dagster run UI loads, live updates work).
- `git.fxhnt.ai` + `dashboard.fxhnt.ai` still 200 (proxy not regressed).