# Re-enable Observability + Expose dagster.fxhnt.ai — Implementation Plan (Phase 2E)

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
> Additive (scale-up + wiring); the proxy edit is gated on `*.fxhnt.ai` staying healthy. Steps use
> checkbox (`- [ ]`) syntax.

**Goal:** Turn the scaled-to-zero Prometheus+Grafana core back on and expose `grafana.fxhnt.ai` +
`dagster.fxhnt.ai` over the tailnet proxy.

**Architecture:** `kubectl scale` the operator/kube-state-metrics/grafana back to 1 (operator brings
prometheus up; alertmanager kept at 0 via its CR); admit the tailscale proxy to grafana + dagster via
netpols; add a dagster nginx block + DNS record.

**Tech Stack:** kube-prometheus-stack (operator), Grafana, Dagster, nginx tailscale proxy, Scaleway DNS.

**Repo:** `~/Work/fxhnt` (IaC home). **Spec:** `docs/superpowers/specs/2026-06-21-observability-2e-design.md`.
**Branch:** `chore/observability-2e` (already created).

---

## Task 1: Re-enable core metrics (prometheus + alertmanager + grafana + kube-state-metrics)

- [ ] **Step 1: Scale up the normal deployments**
```bash
kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=1
```
Expected: 3 deployments scaled. The operator then reconciles the Prometheus **and Alertmanager** CRs
(both `replicas=1`) → brings both statefulsets up automatically (no manual scale needed for them).

- [ ] **Step 2: Wait for the operator to bring prometheus + alertmanager up + pods ready**
```bash
i=0; until kubectl get sts prometheus-prometheus-stack-kube-prom-prometheus -n foxhunt -o jsonpath='{.status.readyReplicas}' 2>/dev/null | grep -q 1 || [ $i -ge 36 ]; do sleep 5; i=$((i+1)); done
kubectl get pods -n foxhunt | grep -iE "prometheus-stack-kube-prom-prometheus|alertmanager|kube-state-metrics|grafana|kube-prom-operator" | grep -v node-exporter | awk '{print $1,$3}'
```
Expected: `prometheus-...-0`, `alertmanager-...-0`, `grafana`, `kube-state-metrics`, `kube-prom-operator`
pods Running. (If alertmanager stays at 0, its CR replicas may be 0 → `kubectl patch alertmanager
prometheus-stack-kube-prom-alertmanager -n foxhunt --type merge -p '{"spec":{"replicas":1}}'`.)

---

## Task 2: Expose grafana.fxhnt.ai (admit the proxy)

**Files:** Create `infra/k8s/monitoring/grafana-netpol.yaml`

- [ ] **Step 1: Add a netpol admitting the tailscale proxy → grafana:3000 (grafana pod port)**

First find grafana's pod port:
```bash
kubectl get deploy grafana -n foxhunt -o jsonpath='{.spec.template.spec.containers[0].ports[*].containerPort}{"\n"}'
```
(Grafana listens on 3000 in-pod; svc maps 80→3000.) Create `infra/k8s/monitoring/grafana-netpol.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: grafana-proxy-ingress
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: grafana
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tailscale-gitlab-proxy
      ports:
        - { port: 3000, protocol: TCP }
```
(If Step-1 shows a different containerPort, use that.)

- [ ] **Step 2: Apply + verify grafana.fxhnt.ai**
```bash
cd /home/jgrusewski/Work/fxhnt
kubectl apply -f infra/k8s/monitoring/grafana-netpol.yaml 2>&1 | tail -1
curl -sk -m 12 -o /dev/null -w "grafana.fxhnt.ai: %{http_code}\n" https://grafana.fxhnt.ai/login 2>&1
```
Expected: `200` (Grafana login). **If 502/000:** grafana pod not ready yet, or the proxy can't reach it —
check the grafana label matches the netpol `podSelector` and the proxy nginx grafana block points at
`grafana.foxhunt.svc:80`.

- [ ] **Step 3: Verify Prometheus datasource works (grafana → prometheus in-cluster)**
```bash
GP=$(kubectl get secret grafana-admin -n foxhunt -o jsonpath='{.data.admin-password}'|base64 -d 2>/dev/null)
curl -sk -u "admin:$GP" "https://grafana.fxhnt.ai/api/datasources" 2>/dev/null | python3 -c "import sys,json;[print(d['name'],d['type'],d['url']) for d in json.load(sys.stdin)]" 2>/dev/null || echo "(check datasource manually in UI)"
```
Expected: a Prometheus datasource listed. **If grafana can't reach prometheus**, add an egress allow
(grafana → prometheus:9090) — only if a datasource health check fails.

---

## Task 3: Expose dagster.fxhnt.ai

**Files:** Modify `infra/k8s/orchestration/dagster.yaml` (its NetworkPolicy), `infra/k8s/gitlab/tailscale-proxy.yaml`

- [ ] **Step 1: Add an Ingress rule to the dagster NetworkPolicy (in dagster.yaml)**

The dagster NetworkPolicy (in `dagster.yaml`) is currently `policyTypes: [Egress]` only. Add `Ingress`
and a rule admitting the proxy → the webserver pod port `3000`:
```yaml
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tailscale-gitlab-proxy
      ports:
        - { port: 3000, protocol: TCP }
```
(Keep the existing `egress:` block. The dagster webserver listens on pod port 3000; svc maps 80→3000.)

- [ ] **Step 2: Add the dagster nginx server block (with WebSocket upgrade)**

In `infra/k8s/gitlab/tailscale-proxy.yaml`, after the Grafana server block, add:
```
    # Dagster — dagster.fxhnt.ai
    server {
        listen 443 ssl;
        server_name dagster.fxhnt.ai;

        ssl_certificate /etc/nginx/certs/tls.crt;
        ssl_certificate_key /etc/nginx/certs/tls.key;
        ssl_protocols TLSv1.2 TLSv1.3;

        location / {
            proxy_pass http://dagster.foxhunt.svc.cluster.local:80;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 3600s;
        }
    }
```

- [ ] **Step 3: Create the Scaleway DNS A record `dagster → 100.95.225.27`**
```bash
export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
scw dns record add fxhnt.ai name=dagster type=A data=100.95.225.27 ttl=300 2>&1 | tail -2
```
Expected: record created. (Mirror via the dns terragrunt module later if you want it IaC-managed — out of scope here.)

- [ ] **Step 4: Apply the netpol + proxy, restart proxy, GATE on platform health**
```bash
cd /home/jgrusewski/Work/fxhnt
kubectl apply -f infra/k8s/orchestration/dagster.yaml 2>&1 | grep -iE "networkpolicy|configured" | tail -2
kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml >/dev/null 2>&1
kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s 2>&1 | tail -1
# Cilium can hold a stale endpoint after rollout — force-recreate the proxy pod (proven pattern)
kubectl delete pod -n foxhunt -l app.kubernetes.io/name=tailscale-gitlab-proxy --force --grace-period=0 2>&1 | tail -1
kubectl wait --for=condition=ready pod -n foxhunt -l app.kubernetes.io/name=tailscale-gitlab-proxy --timeout=120s 2>&1 | tail -1
for h in git dashboard grafana dagster; do curl -sk -m 12 -o /dev/null -w "  $h.fxhnt.ai: %{http_code}\n" https://$h.fxhnt.ai/ 2>&1; done
```
Expected: `git`/`dashboard` 200 (not regressed); `grafana` 200; `dagster` 200/302. **STOP + revert proxy if
git/dashboard go 000.**

---

## Task 4: Commit + push + merge

- [ ] **Step 1: Commit fxhnt + push branch**
```bash
cd /home/jgrusewski/Work/fxhnt
git add infra/k8s/monitoring/grafana-netpol.yaml infra/k8s/orchestration/dagster.yaml infra/k8s/gitlab/tailscale-proxy.yaml
git commit -m "feat(infra): re-enable prometheus+grafana, expose grafana.fxhnt.ai + dagster.fxhnt.ai (Phase 2E)

- scaled prometheus-stack operator/kube-state-metrics + grafana back to 1; operator brings prometheus + alertmanager up
- grafana-netpol: admit tailscale proxy -> grafana:3000 (was 502)
- dagster.yaml netpol: + ingress proxy -> :3000; nginx dagster.fxhnt.ai block (ws); DNS A record

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/observability-2e 2>&1 | tail -2
```

- [ ] **Step 2: Merge → master + push**
```bash
cd /home/jgrusewski/Work/fxhnt
git checkout master && git merge --no-ff chore/observability-2e -m "Merge: re-enable observability + expose grafana/dagster (Phase 2E)" 2>&1 | tail -2
git branch -d chore/observability-2e
GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1
```

---

## Task 5: Final verification
```bash
echo "=== pods (metrics core + alertmanager up; loki/tempo absent) ==="
kubectl get pods -n foxhunt | grep -iE "prometheus|alertmanager|grafana|kube-state" | grep -v node-exporter | awk '{print $1,$3}'
echo "=== UIs ==="
GP=$(kubectl get secret grafana-admin -n foxhunt -o jsonpath='{.data.admin-password}'|base64 -d 2>/dev/null)
curl -sk -m 12 -o /dev/null -w "grafana login: %{http_code}\n" https://grafana.fxhnt.ai/login
curl -sk -m 12 -o /dev/null -w "dagster: %{http_code}\n" https://dagster.fxhnt.ai/
echo "=== prometheus targets healthy (via grafana datasource proxy) ==="
curl -sk -u "admin:$GP" "https://grafana.fxhnt.ai/api/datasources" 2>/dev/null | grep -o '"type":"prometheus"' | head -1
```
Expected: prometheus/grafana/kube-state pods Running; grafana 200; dagster 200/302; prometheus datasource present.

---

## Rollback
- Proxy: `git checkout infra/k8s/gitlab/tailscale-proxy.yaml` + apply + restart.
- Metrics: `kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=0` (back to dormant).

## Acceptance criteria (from spec)
- prometheus/kube-state-metrics/operator/alertmanager/grafana Running; loki/tempo stay 0. ✅ T1, T5
- grafana.fxhnt.ai 200 + prometheus datasource. ✅ T2, T5
- dagster.fxhnt.ai 200 (UI). ✅ T3, T5
- git/dashboard still 200. ✅ T3, T5
- committed + pushed; DNS record created. ✅ T3, T4
