# Re-enable Observability (Prometheus + Grafana) + Expose dagster.fxhnt.ai — Design (Phase 2E)

**Date:** 2026-06-21
**Status:** Design (approved)
**Repo:** fxhnt (IaC home as of 2D)
**Part of:** Phase 2 platform consolidation ([[project_phase2_gitlab_to_gitea]]). 2A–2D DONE. 2E is the
optional observability follow-up.

## Motivation

GitLab's removal (2C) freed ~5.5 GiB. The `grafana` + `prometheus-stack` (kube-prometheus-stack) Helm
releases are already installed but **scaled to zero** (only node-exporters run). Re-enable the *core
metrics* slice to monitor the cluster + fund, and expose Grafana and the Dagster UI over the tailnet
proxy so they're reachable like `git`/`dashboard`/`minio`.

## Scope — re-enable (core metrics only)

ON: `prometheus` (via operator), `prometheus-stack-kube-prom-operator`,
`prometheus-stack-kube-state-metrics`, `alertmanager`, `grafana`, node-exporters (already running).
OFF (stay at 0): loki, tempo, alloy, promtail, pushgateway, dcgm-exporter (GPU, dead).

Re-enable by scaling the operator + kube-state-metrics + grafana deployments back to 1; the operator then
reconciles the Prometheus **and Alertmanager** CRs (both `replicas=1`) → brings those statefulsets up. The
configured prometheus-rules (hft/broker/storage) fire into Alertmanager.

**Note — alert notification routing is a follow-up:** Alertmanager will run + collect firing alerts, but
*where* they get sent (Mattermost / email via Stalwart / webhook) needs a receiver config + channel choice.
2E gets Alertmanager running with rules active (visible in Grafana unified-alerting + the in-cluster
Alertmanager UI); wiring a notification receiver is a small follow-up.

## Scope — expose via tailnet proxy (`infra/k8s/gitlab/tailscale-proxy.yaml`)

1. **`grafana.fxhnt.ai`** — svc (`grafana:80`) + nginx block + DNS A record already exist. Missing piece:
   a NetworkPolicy admitting the `tailscale-gitlab-proxy` pod → `grafana:80` (same fix as minio/gitea;
   that's why grafana was 502/unreachable). Grafana Prometheus datasource already configured
   (`grafana-datasources.yaml`); admin from the `grafana-admin` secret.
2. **`dagster.fxhnt.ai`** — NEW:
   - nginx server block `dagster.fxhnt.ai` → `dagster.foxhunt.svc.cluster.local:80` (Dagster webserver),
     with websocket headers (Dagster UI uses WebSockets for live run updates).
   - Scaleway DNS A record `dagster → 100.95.225.27` (live proxy node IP).
   - dagster NetworkPolicy admitting the `tailscale-gitlab-proxy` → `:80` (dagster webserver container port).

## Data flow

Browser (tailnet) → `grafana.fxhnt.ai`/`dagster.fxhnt.ai` → nginx (tailscale-gitlab-proxy) → svc →
pod. Grafana → Prometheus (in-cluster) → scrapes node-exporters + kube-state-metrics + service-monitors.

## Risks / safety

- **Resource:** ~1.5–2 GiB for prometheus+grafana+kube-state-metrics+alertmanager. Headroom exists (GitLab freed 5.5 GiB).
- **Proxy edit:** adding two server blocks + restarting the proxy — verify `git`/`dashboard` stay 200
  after (the proxy fronts everything). nginx resolves upstreams at boot, so the new upstreams
  (grafana/dagster svcs) must exist before restart — they do.
- **netpol pattern:** admit-the-proxy is the proven pattern (postgres/gitea/minio). Low risk.
- **No data destroyed** — purely additive (scale-up + wiring).
- **Dagster auth:** the Dagster UI has no built-in auth; it's only reachable over the private tailnet
  (same as the other `*.fxhnt.ai` services) — acceptable for a single-maintainer private platform.

## Out of scope
Logs (Loki), traces (Tempo), GPU metrics (dcgm) — the heavier options not chosen. Alert notification
receiver/routing config (Mattermost/email/webhook) — a follow-up after Alertmanager is up.

## Acceptance criteria
- `prometheus`, `kube-state-metrics`, operator, `alertmanager`, `grafana` pods Running; loki/tempo stay 0.
- `grafana.fxhnt.ai` returns 200 (login page); a dashboard shows live pod CPU/RAM; Prometheus targets up.
- `dagster.fxhnt.ai` returns 200 (Dagster run UI loads, live updates work).
- `git.fxhnt.ai` + `dashboard.fxhnt.ai` still 200 (proxy not regressed).
- Changes committed to fxhnt (values + proxy + netpols) + pushed; DNS record created.
