feat(infra): re-enable prometheus+grafana+alertmanager, expose grafana.fxhnt.ai + dagster.fxhnt.ai (Phase 2E)

- scaled prometheus-stack operator/kube-state-metrics + grafana to 1; operator brought
  prometheus + alertmanager up. node-exporters already running. loki/tempo/dcgm stay 0.
- grafana-netpol: admit tailscale proxy -> grafana:3000 (fixes the 502)
- dagster.yaml netpol: + ingress proxy -> :3000; nginx dagster.fxhnt.ai block (websocket)
- dns module: + dagster A record (imported live; plan=No changes — no out-of-band drift)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/superpowers/plans/2026-06-21-observability-2e.md

@@ -0,0 +1,216 @@
+# Re-enable Observability + Expose dagster.fxhnt.ai — Implementation Plan (Phase 2E)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> Additive (scale-up + wiring); the proxy edit is gated on `*.fxhnt.ai` staying healthy. Steps use
+> checkbox (`- [ ]`) syntax.
+
+**Goal:** Turn the scaled-to-zero Prometheus+Grafana core back on and expose `grafana.fxhnt.ai` +
+`dagster.fxhnt.ai` over the tailnet proxy.
+
+**Architecture:** `kubectl scale` the operator/kube-state-metrics/grafana back to 1 (operator brings
+prometheus up; alertmanager kept at 0 via its CR); admit the tailscale proxy to grafana + dagster via
+netpols; add a dagster nginx block + DNS record.
+
+**Tech Stack:** kube-prometheus-stack (operator), Grafana, Dagster, nginx tailscale proxy, Scaleway DNS.
+
+**Repo:** `~/Work/fxhnt` (IaC home). **Spec:** `docs/superpowers/specs/2026-06-21-observability-2e-design.md`.
+**Branch:** `chore/observability-2e` (already created).
+
+---
+
+## Task 1: Re-enable core metrics (prometheus + alertmanager + grafana + kube-state-metrics)
+
+- [ ] **Step 1: Scale up the normal deployments**
+```bash
+kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=1
+```
+Expected: 3 deployments scaled. The operator then reconciles the Prometheus **and Alertmanager** CRs
+(both `replicas=1`) → brings both statefulsets up automatically (no manual scale needed for them).
+
+- [ ] **Step 2: Wait for the operator to bring prometheus + alertmanager up + pods ready**
+```bash
+i=0; until kubectl get sts prometheus-prometheus-stack-kube-prom-prometheus -n foxhunt -o jsonpath='{.status.readyReplicas}' 2>/dev/null | grep -q 1 || [ $i -ge 36 ]; do sleep 5; i=$((i+1)); done
+kubectl get pods -n foxhunt | grep -iE "prometheus-stack-kube-prom-prometheus|alertmanager|kube-state-metrics|grafana|kube-prom-operator" | grep -v node-exporter | awk '{print $1,$3}'
+```
+Expected: `prometheus-...-0`, `alertmanager-...-0`, `grafana`, `kube-state-metrics`, `kube-prom-operator`
+pods Running. (If alertmanager stays at 0, its CR replicas may be 0 → `kubectl patch alertmanager
+prometheus-stack-kube-prom-alertmanager -n foxhunt --type merge -p '{"spec":{"replicas":1}}'`.)
+
+---
+
+## Task 2: Expose grafana.fxhnt.ai (admit the proxy)
+
+**Files:** Create `infra/k8s/monitoring/grafana-netpol.yaml`
+
+- [ ] **Step 1: Add a netpol admitting the tailscale proxy → grafana:3000 (grafana pod port)**
+
+First find grafana's pod port:
+```bash
+kubectl get deploy grafana -n foxhunt -o jsonpath='{.spec.template.spec.containers[0].ports[*].containerPort}{"\n"}'
+```
+(Grafana listens on 3000 in-pod; svc maps 80→3000.) Create `infra/k8s/monitoring/grafana-netpol.yaml`:
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: grafana-proxy-ingress
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: grafana
+  policyTypes: [Ingress]
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tailscale-gitlab-proxy
+      ports:
+        - { port: 3000, protocol: TCP }
+```
+(If Step-1 shows a different containerPort, use that.)
+
+- [ ] **Step 2: Apply + verify grafana.fxhnt.ai**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+kubectl apply -f infra/k8s/monitoring/grafana-netpol.yaml 2>&1 | tail -1
+curl -sk -m 12 -o /dev/null -w "grafana.fxhnt.ai: %{http_code}\n" https://grafana.fxhnt.ai/login 2>&1
+```
+Expected: `200` (Grafana login). **If 502/000:** grafana pod not ready yet, or the proxy can't reach it —
+check the grafana label matches the netpol `podSelector` and the proxy nginx grafana block points at
+`grafana.foxhunt.svc:80`.
+
+- [ ] **Step 3: Verify Prometheus datasource works (grafana → prometheus in-cluster)**
+```bash
+GP=$(kubectl get secret grafana-admin -n foxhunt -o jsonpath='{.data.admin-password}'|base64 -d 2>/dev/null)
+curl -sk -u "admin:$GP" "https://grafana.fxhnt.ai/api/datasources" 2>/dev/null | python3 -c "import sys,json;[print(d['name'],d['type'],d['url']) for d in json.load(sys.stdin)]" 2>/dev/null || echo "(check datasource manually in UI)"
+```
+Expected: a Prometheus datasource listed. **If grafana can't reach prometheus**, add an egress allow
+(grafana → prometheus:9090) — only if a datasource health check fails.
+
+---
+
+## Task 3: Expose dagster.fxhnt.ai
+
+**Files:** Modify `infra/k8s/orchestration/dagster.yaml` (its NetworkPolicy), `infra/k8s/gitlab/tailscale-proxy.yaml`
+
+- [ ] **Step 1: Add an Ingress rule to the dagster NetworkPolicy (in dagster.yaml)**
+
+The dagster NetworkPolicy (in `dagster.yaml`) is currently `policyTypes: [Egress]` only. Add `Ingress`
+and a rule admitting the proxy → the webserver pod port `3000`:
+```yaml
+  policyTypes: [Ingress, Egress]
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tailscale-gitlab-proxy
+      ports:
+        - { port: 3000, protocol: TCP }
+```
+(Keep the existing `egress:` block. The dagster webserver listens on pod port 3000; svc maps 80→3000.)
+
+- [ ] **Step 2: Add the dagster nginx server block (with WebSocket upgrade)**
+
+In `infra/k8s/gitlab/tailscale-proxy.yaml`, after the Grafana server block, add:
+```
+    # Dagster — dagster.fxhnt.ai
+    server {
+        listen 443 ssl;
+        server_name dagster.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        location / {
+            proxy_pass http://dagster.foxhunt.svc.cluster.local:80;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 3600s;
+        }
+    }
+```
+
+- [ ] **Step 3: Create the Scaleway DNS A record `dagster → 100.95.225.27`**
+```bash
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+scw dns record add fxhnt.ai name=dagster type=A data=100.95.225.27 ttl=300 2>&1 | tail -2
+```
+Expected: record created. (Mirror via the dns terragrunt module later if you want it IaC-managed — out of scope here.)
+
+- [ ] **Step 4: Apply the netpol + proxy, restart proxy, GATE on platform health**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+kubectl apply -f infra/k8s/orchestration/dagster.yaml 2>&1 | grep -iE "networkpolicy|configured" | tail -2
+kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml >/dev/null 2>&1
+kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
+kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s 2>&1 | tail -1
+# Cilium can hold a stale endpoint after rollout — force-recreate the proxy pod (proven pattern)
+kubectl delete pod -n foxhunt -l app.kubernetes.io/name=tailscale-gitlab-proxy --force --grace-period=0 2>&1 | tail -1
+kubectl wait --for=condition=ready pod -n foxhunt -l app.kubernetes.io/name=tailscale-gitlab-proxy --timeout=120s 2>&1 | tail -1
+for h in git dashboard grafana dagster; do curl -sk -m 12 -o /dev/null -w "  $h.fxhnt.ai: %{http_code}\n" https://$h.fxhnt.ai/ 2>&1; done
+```
+Expected: `git`/`dashboard` 200 (not regressed); `grafana` 200; `dagster` 200/302. **STOP + revert proxy if
+git/dashboard go 000.**
+
+---
+
+## Task 4: Commit + push + merge
+
+- [ ] **Step 1: Commit fxhnt + push branch**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add infra/k8s/monitoring/grafana-netpol.yaml infra/k8s/orchestration/dagster.yaml infra/k8s/gitlab/tailscale-proxy.yaml
+git commit -m "feat(infra): re-enable prometheus+grafana, expose grafana.fxhnt.ai + dagster.fxhnt.ai (Phase 2E)
+
+- scaled prometheus-stack operator/kube-state-metrics + grafana back to 1; operator brings prometheus + alertmanager up
+- grafana-netpol: admit tailscale proxy -> grafana:3000 (was 502)
+- dagster.yaml netpol: + ingress proxy -> :3000; nginx dagster.fxhnt.ai block (ws); DNS A record
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/observability-2e 2>&1 | tail -2
+```
+
+- [ ] **Step 2: Merge → master + push**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git checkout master && git merge --no-ff chore/observability-2e -m "Merge: re-enable observability + expose grafana/dagster (Phase 2E)" 2>&1 | tail -2
+git branch -d chore/observability-2e
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1
+```
+
+---
+
+## Task 5: Final verification
+```bash
+echo "=== pods (metrics core + alertmanager up; loki/tempo absent) ==="
+kubectl get pods -n foxhunt | grep -iE "prometheus|alertmanager|grafana|kube-state" | grep -v node-exporter | awk '{print $1,$3}'
+echo "=== UIs ==="
+GP=$(kubectl get secret grafana-admin -n foxhunt -o jsonpath='{.data.admin-password}'|base64 -d 2>/dev/null)
+curl -sk -m 12 -o /dev/null -w "grafana login: %{http_code}\n" https://grafana.fxhnt.ai/login
+curl -sk -m 12 -o /dev/null -w "dagster: %{http_code}\n" https://dagster.fxhnt.ai/
+echo "=== prometheus targets healthy (via grafana datasource proxy) ==="
+curl -sk -u "admin:$GP" "https://grafana.fxhnt.ai/api/datasources" 2>/dev/null | grep -o '"type":"prometheus"' | head -1
+```
+Expected: prometheus/grafana/kube-state pods Running; grafana 200; dagster 200/302; prometheus datasource present.
+
+---
+
+## Rollback
+- Proxy: `git checkout infra/k8s/gitlab/tailscale-proxy.yaml` + apply + restart.
+- Metrics: `kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=0` (back to dormant).
+
+## Acceptance criteria (from spec)
+- prometheus/kube-state-metrics/operator/alertmanager/grafana Running; loki/tempo stay 0. ✅ T1, T5
+- grafana.fxhnt.ai 200 + prometheus datasource. ✅ T2, T5
+- dagster.fxhnt.ai 200 (UI). ✅ T3, T5
+- git/dashboard still 200. ✅ T3, T5
+- committed + pushed; DNS record created. ✅ T3, T4

docs/superpowers/specs/2026-06-21-observability-2e-design.md

@@ -0,0 +1,68 @@
+# Re-enable Observability (Prometheus + Grafana) + Expose dagster.fxhnt.ai — Design (Phase 2E)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** fxhnt (IaC home as of 2D)
+**Part of:** Phase 2 platform consolidation ([[project_phase2_gitlab_to_gitea]]). 2A–2D DONE. 2E is the
+optional observability follow-up.
+
+## Motivation
+
+GitLab's removal (2C) freed ~5.5 GiB. The `grafana` + `prometheus-stack` (kube-prometheus-stack) Helm
+releases are already installed but **scaled to zero** (only node-exporters run). Re-enable the *core
+metrics* slice to monitor the cluster + fund, and expose Grafana and the Dagster UI over the tailnet
+proxy so they're reachable like `git`/`dashboard`/`minio`.
+
+## Scope — re-enable (core metrics only)
+
+ON: `prometheus` (via operator), `prometheus-stack-kube-prom-operator`,
+`prometheus-stack-kube-state-metrics`, `alertmanager`, `grafana`, node-exporters (already running).
+OFF (stay at 0): loki, tempo, alloy, promtail, pushgateway, dcgm-exporter (GPU, dead).
+
+Re-enable by scaling the operator + kube-state-metrics + grafana deployments back to 1; the operator then
+reconciles the Prometheus **and Alertmanager** CRs (both `replicas=1`) → brings those statefulsets up. The
+configured prometheus-rules (hft/broker/storage) fire into Alertmanager.
+
+**Note — alert notification routing is a follow-up:** Alertmanager will run + collect firing alerts, but
+*where* they get sent (Mattermost / email via Stalwart / webhook) needs a receiver config + channel choice.
+2E gets Alertmanager running with rules active (visible in Grafana unified-alerting + the in-cluster
+Alertmanager UI); wiring a notification receiver is a small follow-up.
+
+## Scope — expose via tailnet proxy (`infra/k8s/gitlab/tailscale-proxy.yaml`)
+
+1. **`grafana.fxhnt.ai`** — svc (`grafana:80`) + nginx block + DNS A record already exist. Missing piece:
+   a NetworkPolicy admitting the `tailscale-gitlab-proxy` pod → `grafana:80` (same fix as minio/gitea;
+   that's why grafana was 502/unreachable). Grafana Prometheus datasource already configured
+   (`grafana-datasources.yaml`); admin from the `grafana-admin` secret.
+2. **`dagster.fxhnt.ai`** — NEW:
+   - nginx server block `dagster.fxhnt.ai` → `dagster.foxhunt.svc.cluster.local:80` (Dagster webserver),
+     with websocket headers (Dagster UI uses WebSockets for live run updates).
+   - Scaleway DNS A record `dagster → 100.95.225.27` (live proxy node IP).
+   - dagster NetworkPolicy admitting the `tailscale-gitlab-proxy` → `:80` (dagster webserver container port).
+
+## Data flow
+
+Browser (tailnet) → `grafana.fxhnt.ai`/`dagster.fxhnt.ai` → nginx (tailscale-gitlab-proxy) → svc →
+pod. Grafana → Prometheus (in-cluster) → scrapes node-exporters + kube-state-metrics + service-monitors.
+
+## Risks / safety
+
+- **Resource:** ~1.5–2 GiB for prometheus+grafana+kube-state-metrics+alertmanager. Headroom exists (GitLab freed 5.5 GiB).
+- **Proxy edit:** adding two server blocks + restarting the proxy — verify `git`/`dashboard` stay 200
+  after (the proxy fronts everything). nginx resolves upstreams at boot, so the new upstreams
+  (grafana/dagster svcs) must exist before restart — they do.
+- **netpol pattern:** admit-the-proxy is the proven pattern (postgres/gitea/minio). Low risk.
+- **No data destroyed** — purely additive (scale-up + wiring).
+- **Dagster auth:** the Dagster UI has no built-in auth; it's only reachable over the private tailnet
+  (same as the other `*.fxhnt.ai` services) — acceptable for a single-maintainer private platform.
+
+## Out of scope
+Logs (Loki), traces (Tempo), GPU metrics (dcgm) — the heavier options not chosen. Alert notification
+receiver/routing config (Mattermost/email/webhook) — a follow-up after Alertmanager is up.
+
+## Acceptance criteria
+- `prometheus`, `kube-state-metrics`, operator, `alertmanager`, `grafana` pods Running; loki/tempo stay 0.
+- `grafana.fxhnt.ai` returns 200 (login page); a dashboard shows live pod CPU/RAM; Prometheus targets up.
+- `dagster.fxhnt.ai` returns 200 (Dagster run UI loads, live updates work).
+- `git.fxhnt.ai` + `dashboard.fxhnt.ai` still 200 (proxy not regressed).
+- Changes committed to fxhnt (values + proxy + netpols) + pushed; DNS record created.

infra/k8s/gitlab/tailscale-proxy.yaml

@@ -199,6 +199,28 @@ data:
         }
     }
 
+    # Dagster — dagster.fxhnt.ai
+    server {
+        listen 443 ssl;
+        server_name dagster.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        location / {
+            proxy_pass http://dagster.foxhunt.svc.cluster.local:80;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 3600s;
+        }
+    }
+
     # fxhnt cockpit — dashboard.fxhnt.ai
     # Proxies to the cockpit's Tailscale node (peer-to-peer over the tailnet) rather than the cluster Service:
     # the pod CIDR 100.64.0.0/15 overlaps Tailscale CGNAT, so this kernel-mode proxy can't reach platform-pool

infra/k8s/monitoring/grafana-netpol.yaml

@@ -0,0 +1,19 @@
+# Admit the tailscale proxy to Grafana so grafana.fxhnt.ai works (default-deny-all otherwise blocks it —
+# this was the 502). Same admit-the-proxy pattern as postgres/gitea/minio. Grafana pod listens on 3000.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: grafana-proxy-ingress
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: grafana
+  policyTypes: [Ingress]
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tailscale-gitlab-proxy
+      ports:
+        - { port: 3000, protocol: TCP }

infra/k8s/orchestration/dagster.yaml

@@ -156,7 +156,10 @@ kind: NetworkPolicy
 metadata: { name: dagster, namespace: foxhunt, labels: { app.kubernetes.io/part-of: foxhunt } }
 spec:
   podSelector: { matchLabels: { app.kubernetes.io/name: dagster } }
-  policyTypes: [Egress]
+  policyTypes: [Ingress, Egress]
+  ingress:
+    - from: [{ podSelector: { matchLabels: { app.kubernetes.io/name: tailscale-gitlab-proxy } } }]   # dagster.fxhnt.ai
+      ports: [{ port: 3000, protocol: TCP }]
   egress:
     - to: [{ podSelector: { matchLabels: { app.kubernetes.io/name: postgres } } }]
       ports: [{ port: 5432, protocol: TCP }]

infra/modules/dns/main.tf

@@ -56,3 +56,11 @@ resource "scaleway_domain_record" "mail" {
   ttl      = 300
 }
 
+resource "scaleway_domain_record" "dagster" {
+  dns_zone = var.dns_zone
+  name     = "dagster"
+  type     = "A"
+  data     = var.git_ip  # Tailscale proxy — Dagster orchestrator UI (Phase 2E)
+  ttl      = 300
+}
+