# Kanidm SSO + NetBird Mesh Implementation Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Deploy Kanidm (passwordless WebAuthn SSO) for 6 Tailnet-exposed services and NetBird (self-hosted mesh VPN) alongside Tailscale — zero breaking changes.

**Architecture:** Kanidm provides centralized OIDC identity (RS256, WebAuthn-first) for Grafana, Mattermost, GitLab CE, Stalwart Mail, FXT Web Dashboard, and FXT CLI. NetBird runs as a parallel mesh VPN with Kanidm OIDC auth, side-by-side with Tailscale. All K8s manifests go under `infra/k8s/`, DNS records in `infra/modules/dns/`, nginx routing in the existing tailscale-proxy.

**Tech Stack:** Kanidm 1.9.2, NetBird 0.66.0, Scaleway Kapsule (DEV1-L), Cilium NetworkPolicy, nginx reverse proxy, Terraform (Scaleway DNS)

**Spec:** `docs/superpowers/specs/2026-03-14-kanidm-sso-design.md`

---

## Chunk 1: Kanidm Core Infrastructure

### Task 1: Create Kanidm K8s Manifests

**Files:**
- Create: `infra/k8s/kanidm/kanidm.yaml` (ConfigMap + PVC + Deployment + Service)
- Create: `infra/k8s/network-policies/kanidm.yaml`

- [ ] **Step 1: Create kanidm directory**

```bash
ls infra/k8s/  # verify parent exists
mkdir -p infra/k8s/kanidm
```

- [ ] **Step 2: Write Kanidm deployment manifest**

Create `infra/k8s/kanidm/kanidm.yaml` with all resources in one file:

```yaml
# Kanidm Identity Provider — WebAuthn-first, Rust-native
# Provides OIDC SSO for all Tailnet-exposed services
apiVersion: v1
kind: ConfigMap
metadata:
  name: kanidm-config
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: kanidm
    app.kubernetes.io/part-of: foxhunt
data:
  server.toml: |
    [server]
    domain = "auth.fxhnt.ai"
    origin = "https://auth.fxhnt.ai"
    tls_chain = "/data/certs/tls.crt"
    tls_key = "/data/certs/tls.key"
    bindaddress = "[::]:8443"
    db_path = "/data/kanidm.db"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kanidm-data
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: kanidm
    app.kubernetes.io/part-of: foxhunt
spec:
  storageClassName: scw-bssd
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kanidm
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: kanidm
    app.kubernetes.io/part-of: foxhunt
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: kanidm
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kanidm
        app.kubernetes.io/part-of: foxhunt
    spec:
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      containers:
        - name: kanidm
          image: kanidm/server:1.9.2
          ports:
            - containerPort: 8443
              protocol: TCP
              name: https
          resources:
            requests:
              cpu: 25m
              memory: 48Mi
            limits:
              cpu: 100m
              memory: 96Mi
          volumeMounts:
            - name: data
              mountPath: /data
            - name: config
              mountPath: /data/server.toml
              subPath: server.toml
              readOnly: true
            - name: tls-certs
              mountPath: /data/certs
              readOnly: true
          readinessProbe:
            tcpSocket:
              port: 8443
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            tcpSocket:
              port: 8443
            initialDelaySeconds: 10
            periodSeconds: 30
      volumes:
        - name: config
          configMap:
            name: kanidm-config
        - name: tls-certs
          secret:
            secretName: gitlab-tls-cert
        - name: data
          persistentVolumeClaim:
            claimName: kanidm-data
---
apiVersion: v1
kind: Service
metadata:
  name: kanidm
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: kanidm
    app.kubernetes.io/part-of: foxhunt
spec:
  selector:
    app.kubernetes.io/name: kanidm
  ports:
    - port: 8443
      targetPort: 8443
      protocol: TCP
      name: https
```

- [ ] **Step 3: Write Kanidm network policy**

Create `infra/k8s/network-policies/kanidm.yaml`:

```yaml
# Kanidm network policy
# Ingress: nginx proxy (OIDC flows), foxhunt services (token validation)
# Egress: DNS only (Kanidm is the identity source, not a consumer)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: kanidm
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: kanidm
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Tailscale proxy (nginx) — OIDC redirects from all Tailnet services
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tailscale-gitlab-proxy
      ports:
        - protocol: TCP
          port: 8443
    # In-cluster services that validate tokens directly
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/part-of: foxhunt
      ports:
        - protocol: TCP
          port: 8443
  egress:
    # DNS resolution
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
```

- [ ] **Step 4: Validate YAML syntax**

```bash
kubectl apply --dry-run=client -f infra/k8s/kanidm/kanidm.yaml
kubectl apply --dry-run=client -f infra/k8s/network-policies/kanidm.yaml
```

Expected: `configured (dry run)` for all resources, no errors.

- [ ] **Step 5: Commit**

```bash
git add infra/k8s/kanidm/kanidm.yaml infra/k8s/network-policies/kanidm.yaml
git commit -m "feat(kanidm): add K8s manifests for Kanidm identity provider

Deployment, Service, PVC (1Gi scw-bssd), ConfigMap (server.toml),
and NetworkPolicy. Pinned to kanidm/server:1.9.2. Recreate strategy
for RWO PVC. Platform node pool."
```

---

### Task 2: Add DNS Records (Terraform)

**Files:**
- Modify: `infra/modules/dns/main.tf`

- [ ] **Step 1: Add `auth` and `netbird` DNS records**

Append to `infra/modules/dns/main.tf` after the existing `mail` record (follows existing pattern — A record → `100.90.76.85`, TTL 300):

```hcl
resource "scaleway_domain_record" "auth" {
  dns_zone = var.dns_zone
  name     = "auth"
  type     = "A"
  data     = var.git_ip
  ttl      = 300
}

resource "scaleway_domain_record" "netbird" {
  dns_zone = var.dns_zone
  name     = "netbird"
  type     = "A"
  data     = var.git_ip
  ttl      = 300
}
```

- [ ] **Step 2: Validate terraform syntax**

```bash
cd infra/live/production/dns && terragrunt validate
```

Expected: `Success! The configuration is valid.`

- [ ] **Step 3: Commit**

```bash
git add infra/modules/dns/main.tf
git commit -m "feat(dns): add auth.fxhnt.ai and netbird.fxhnt.ai A records"
```

---

### Task 3: Add Nginx Proxy Blocks

**Files:**
- Modify: `infra/k8s/gitlab/tailscale-proxy.yaml`

- [ ] **Step 1: Add Kanidm server block to nginx config**

In `infra/k8s/gitlab/tailscale-proxy.yaml`, add a new server block in the nginx ConfigMap data (after the existing `minio.fxhnt.ai` block). Follow the exact same pattern as existing blocks:

```nginx
    # Kanidm Identity Provider — auth.fxhnt.ai
    server {
        listen 443 ssl;
        server_name auth.fxhnt.ai;

        ssl_certificate /etc/nginx/certs/tls.crt;
        ssl_certificate_key /etc/nginx/certs/tls.key;

        location / {
            # Kanidm uses self-signed internal TLS — external TLS terminates at nginx
            proxy_pass https://kanidm.foxhunt.svc.cluster.local:8443;
            proxy_ssl_verify off;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
```

- [ ] **Step 2: Add NetBird server block**

Add another server block after the Kanidm one:

```nginx
    # NetBird Dashboard + Management — netbird.fxhnt.ai
    server {
        listen 443 ssl;
        server_name netbird.fxhnt.ai;

        ssl_certificate /etc/nginx/certs/tls.crt;
        ssl_certificate_key /etc/nginx/certs/tls.key;

        location / {
            proxy_pass http://netbird-dashboard.foxhunt.svc.cluster.local:80;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

        # Management API
        location /api {
            proxy_pass http://netbird-server.foxhunt.svc.cluster.local:443;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
        }

        # Signal (gRPC)
        location /signalexchange.SignalExchange/ {
            grpc_pass grpc://netbird-server.foxhunt.svc.cluster.local:10000;
        }
    }
```

- [ ] **Step 3: Validate proxy YAML**

```bash
kubectl apply --dry-run=client -f infra/k8s/gitlab/tailscale-proxy.yaml
```

Expected: `configured (dry run)`.

**Note**: nginx will pick up these changes after `rollout restart` in Task 8a.

- [ ] **Step 4: Commit**

```bash
git add infra/k8s/gitlab/tailscale-proxy.yaml
git commit -m "feat(proxy): add nginx upstream blocks for auth.fxhnt.ai and netbird.fxhnt.ai"
```

---

### Task 4: Add Kanidm Monitoring (Prometheus Alert Rule)

**Files:**
- Modify: `infra/k8s/monitoring/prometheus-rules-hft.yaml`

- [ ] **Step 1: Add KanidmDown alert rule**

Add to the `hft_infrastructure` group in `infra/k8s/monitoring/prometheus-rules-hft.yaml` (after the PostgreSQLDown rule). Uses kube-state-metrics (already deployed) — no blackbox exporter needed:

```yaml
        - alert: KanidmDown
          expr: kube_deployment_status_replicas_available{namespace="foxhunt", deployment="kanidm"} == 0
          for: 30s
          labels:
            severity: critical
            component: identity
            impact: auth_failure
          annotations:
            summary: "Kanidm identity provider is down — SSO broken for all services"
        - alert: KanidmDiskSpaceFillingUp
          expr: (kubelet_volume_stats_used_bytes{persistentvolumeclaim="kanidm-data"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="kanidm-data"}) * 100 > 80
          for: 1h
          labels:
            severity: warning
            component: identity
          annotations:
            summary: "Kanidm PVC over 80% full — plan resize (scw-bssd requires Recreate strategy downtime)"
```

- [ ] **Step 2: Validate YAML**

```bash
kubectl apply --dry-run=client -f infra/k8s/monitoring/prometheus-rules-hft.yaml
```

- [ ] **Step 3: Commit**

```bash
git add infra/k8s/monitoring/prometheus-rules-hft.yaml
git commit -m "feat(monitoring): add KanidmDown critical alert rule"
```

---

### Task 5: Add Kanidm Backup CronJob

**Files:**
- Create: `infra/k8s/kanidm/backup-cronjob.yaml`

**Note**: Kanidm's PVC is RWO (scw-bssd block storage). A separate pod cannot mount it
concurrently. Instead, the CronJob uses `kubectl exec` to run `kanidm database backup`
inside the running Kanidm pod, then copies the output to MinIO via a sidecar rclone step.

- [ ] **Step 1: Write backup CronJob manifest**

Create `infra/k8s/kanidm/backup-cronjob.yaml`:

```yaml
# Daily Kanidm database backup to MinIO S3
# Uses kubectl exec into running Kanidm pod (avoids RWO PVC concurrent mount)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kanidm-backup
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: kanidm-backup
    app.kubernetes.io/part-of: foxhunt
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kanidm-backup
  namespace: foxhunt
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kanidm-backup
  namespace: foxhunt
subjects:
  - kind: ServiceAccount
    name: kanidm-backup
    namespace: foxhunt
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kanidm-backup
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kanidm-backup
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: kanidm-backup
    app.kubernetes.io/part-of: foxhunt
spec:
  schedule: "0 3 * * *"  # 3am daily
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kanidm-backup
          nodeSelector:
            k8s.scaleway.com/pool-name: platform
          restartPolicy: OnFailure
          # Two containers share an emptyDir: kubectl dumps backup, rclone uploads.
          # bitnami/kubectl does NOT include rclone, so they must be separate images.
          containers:
            - name: kubectl-backup
              image: bitnami/kubectl:1.31
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  TIMESTAMP=$(date +%Y%m%d-%H%M%S)
                  BACKUP_FILE="kanidm-${TIMESTAMP}.db"
                  POD=$(kubectl -n foxhunt get pod -l app.kubernetes.io/name=kanidm -o jsonpath='{.items[0].metadata.name}')
                  echo "Backing up Kanidm DB from pod $POD..."
                  kubectl -n foxhunt exec "$POD" -- kanidm database backup -o /tmp/kanidm-backup.db
                  kubectl -n foxhunt cp "$POD:/tmp/kanidm-backup.db" "/backup/${BACKUP_FILE}"
                  kubectl -n foxhunt exec "$POD" -- rm -f /tmp/kanidm-backup.db
                  echo "Backup dumped to /backup/${BACKUP_FILE} — rclone container will upload."
              resources:
                requests:
                  cpu: 10m
                  memory: 32Mi
                limits:
                  cpu: 100m
                  memory: 64Mi
              volumeMounts:
                - name: backup-staging
                  mountPath: /backup
            - name: rclone-upload
              image: rclone/rclone:1.66
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  echo "Waiting for backup file..."
                  TIMEOUT=120; ELAPSED=0
                  while [ -z "$(ls /backup/kanidm-*.db 2>/dev/null)" ]; do
                    sleep 2; ELAPSED=$((ELAPSED+2))
                    if [ "$ELAPSED" -ge "$TIMEOUT" ]; then echo "Timeout waiting for backup"; exit 1; fi
                  done
                  BACKUP_FILE=$(ls /backup/kanidm-*.db | head -n 1)
                  echo "Uploading ${BACKUP_FILE} to S3..."
                  rclone copy "${BACKUP_FILE}" minio:foxhunt-kanidm-backup/ \
                    --s3-provider=Minio \
                    --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
                    --s3-access-key-id=$(cat /secrets/access-key) \
                    --s3-secret-access-key=$(cat /secrets/secret-key) \
                    --s3-no-check-bucket
                  echo "Upload complete: ${BACKUP_FILE} -> s3://foxhunt-kanidm-backup/"
              resources:
                requests:
                  cpu: 10m
                  memory: 32Mi
                limits:
                  cpu: 100m
                  memory: 64Mi
              volumeMounts:
                - name: backup-staging
                  mountPath: /backup
                - name: minio-credentials
                  mountPath: /secrets
                  readOnly: true
          volumes:
            - name: backup-staging
              emptyDir: {}
            - name: minio-credentials
              secret:
                secretName: minio-credentials
```

- [ ] **Step 2: Validate YAML**

```bash
kubectl apply --dry-run=client -f infra/k8s/kanidm/backup-cronjob.yaml
```

- [ ] **Step 3: Commit**

```bash
git add infra/k8s/kanidm/backup-cronjob.yaml
git commit -m "feat(kanidm): add daily backup CronJob via kubectl exec to MinIO S3"
```

---

## Chunk 2: NetBird Infrastructure

### Task 6: Create NetBird K8s Manifests

**Files:**
- Create: `infra/k8s/netbird/netbird.yaml` (PVC + Deployment + Services for server and dashboard)
- Create: `infra/k8s/network-policies/netbird.yaml`

- [ ] **Step 1: Create netbird directory**

```bash
mkdir -p infra/k8s/netbird
```

- [ ] **Step 2: Write NetBird deployment manifest**

Create `infra/k8s/netbird/netbird.yaml`:

```yaml
# NetBird — self-hosted WireGuard mesh VPN (parallel with Tailscale)
# All-in-one server: management + signal + relay + STUN
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: netbird-data
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/part-of: foxhunt
spec:
  storageClassName: scw-bssd
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: netbird-server
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/component: server
    app.kubernetes.io/part-of: foxhunt
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: netbird
      app.kubernetes.io/component: server
  template:
    metadata:
      labels:
        app.kubernetes.io/name: netbird
        app.kubernetes.io/component: server
        app.kubernetes.io/part-of: foxhunt
    spec:
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      containers:
        - name: netbird
          image: netbirdio/netbird-server:0.66.0
          ports:
            - containerPort: 443
              protocol: TCP
              name: management
            - containerPort: 10000
              protocol: TCP
              name: signal
            - containerPort: 3478
              protocol: UDP
              name: stun
          env:
            - name: NETBIRD_DOMAIN
              value: "netbird.fxhnt.ai"
          resources:
            requests:
              cpu: 25m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 256Mi
          volumeMounts:
            - name: data
              mountPath: /var/lib/netbird
          readinessProbe:
            tcpSocket:
              port: 443
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            tcpSocket:
              port: 443
            initialDelaySeconds: 15
            periodSeconds: 30
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: netbird-data
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: netbird-dashboard
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/component: dashboard
    app.kubernetes.io/part-of: foxhunt
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: netbird
      app.kubernetes.io/component: dashboard
  template:
    metadata:
      labels:
        app.kubernetes.io/name: netbird
        app.kubernetes.io/component: dashboard
        app.kubernetes.io/part-of: foxhunt
    spec:
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      containers:
        - name: dashboard
          image: netbirdio/dashboard:v2.33.0
          ports:
            - containerPort: 80
              protocol: TCP
              name: http
          env:
            - name: NETBIRD_MGMT_API_ENDPOINT
              value: "https://netbird.fxhnt.ai/api"
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 50m
              memory: 64Mi
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: netbird-server
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/component: server
    app.kubernetes.io/part-of: foxhunt
spec:
  selector:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/component: server
  ports:
    - port: 443
      targetPort: 443
      protocol: TCP
      name: management
    - port: 10000
      targetPort: 10000
      protocol: TCP
      name: signal
---
# STUN/TURN — NodePort for UDP (nginx can't proxy UDP)
apiVersion: v1
kind: Service
metadata:
  name: netbird-stun
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/component: stun
    app.kubernetes.io/part-of: foxhunt
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/component: server
  ports:
    - port: 3478
      targetPort: 3478
      protocol: UDP
      name: stun
      nodePort: 31478
---
apiVersion: v1
kind: Service
metadata:
  name: netbird-dashboard
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/component: dashboard
    app.kubernetes.io/part-of: foxhunt
spec:
  selector:
    app.kubernetes.io/name: netbird
    app.kubernetes.io/component: dashboard
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
```

- [ ] **Step 3: Write NetBird network policy**

Create `infra/k8s/network-policies/netbird.yaml`:

```yaml
# NetBird network policy
# Ingress: nginx proxy (dashboard/API/signal), STUN/TURN (UDP from peers)
# Egress: DNS, Kanidm OIDC
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: netbird
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: netbird
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Nginx proxy (dashboard + management API + signal)
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tailscale-gitlab-proxy
      ports:
        - protocol: TCP
          port: 80
        - protocol: TCP
          port: 443
        - protocol: TCP
          port: 10000
    # STUN/TURN from external peers (via NodePort)
    - ports:
        - protocol: UDP
          port: 3478
    # Peer connections from foxhunt namespace
    - from:
        - podSelector: {}
      ports:
        - protocol: TCP
          port: 443
  egress:
    # DNS resolution
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # Kanidm OIDC (token validation)
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: kanidm
      ports:
        - protocol: TCP
          port: 8443
```

- [ ] **Step 4: Validate YAML syntax**

```bash
kubectl apply --dry-run=client -f infra/k8s/netbird/netbird.yaml
kubectl apply --dry-run=client -f infra/k8s/network-policies/netbird.yaml
```

- [ ] **Step 5: Commit**

```bash
git add infra/k8s/netbird/netbird.yaml infra/k8s/network-policies/netbird.yaml
git commit -m "feat(netbird): add K8s manifests for self-hosted NetBird mesh VPN

Server (management+signal+relay+STUN), dashboard, PVC, services
(ClusterIP + NodePort for STUN UDP), and NetworkPolicy.
Side-by-side with Tailscale — no breaking changes."
```

---

### Task 7: Add NetBird Monitoring Alert

**Files:**
- Modify: `infra/k8s/monitoring/prometheus-rules-hft.yaml`

- [ ] **Step 1: Add NetBirdDown alert rule**

Add to the `hft_infrastructure` group (after the KanidmDown rule added in Task 4):

```yaml
        - alert: NetBirdDown
          expr: up{job="netbird"} == 0
          for: 2m
          labels:
            severity: medium
            component: network
            impact: vpn_evaluation
          annotations:
            summary: "NetBird mesh VPN is down (evaluation — Tailscale primary unaffected)"
```

- [ ] **Step 2: Add NetBird scrape config to prometheus-stack-values.yaml**

Add to `additionalScrapeConfigs` in `infra/k8s/monitoring/prometheus-stack-values.yaml` (after the Tempo scrape config):

```yaml
      # NetBird management metrics (verify endpoint exists after deploy — see Task 9 Step 3)
      - job_name: netbird
        metrics_path: /metrics
        static_configs:
          - targets: ['netbird-server.foxhunt.svc.cluster.local:443']
```

- [ ] **Step 3: Add Prometheus egress to NetBird in prometheus network policy**

Add to `infra/k8s/network-policies/prometheus.yaml` egress rules (after the Argo Workflows block):

```yaml
    # NetBird management metrics
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: netbird
      ports:
        - protocol: TCP
          port: 443
```

- [ ] **Step 4: Validate all modified YAML files**

```bash
kubectl apply --dry-run=client -f infra/k8s/monitoring/prometheus-rules-hft.yaml
kubectl apply --dry-run=client -f infra/k8s/network-policies/prometheus.yaml
```

- [ ] **Step 5: Commit**

```bash
git add infra/k8s/monitoring/prometheus-rules-hft.yaml \
        infra/k8s/monitoring/prometheus-stack-values.yaml \
        infra/k8s/network-policies/prometheus.yaml
git commit -m "feat(monitoring): add NetBird scrape config and alert rule"
```

---

## Chunk 3: Deploy & Configure Identity

### Task 8a: Apply DNS + Proxy (prerequisite for auth.fxhnt.ai)

**Prerequisite**: Tasks 1-7 committed and pushed.

- [ ] **Step 1: Verify TLS cert secret exists**

```bash
kubectl -n foxhunt get secret gitlab-tls-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -dates
```

Expected: `subject=...*.fxhnt.ai...` with valid dates.

- [ ] **Step 2: Apply updated nginx proxy**

```bash
kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml
kubectl -n foxhunt rollout restart deployment/tailscale-gitlab-proxy
kubectl -n foxhunt rollout status deployment/tailscale-gitlab-proxy --timeout=60s
```

- [ ] **Step 3: Apply DNS via Terraform**

```bash
cd infra/live/production/dns && terragrunt apply -auto-approve
```

Verify:
```bash
dig auth.fxhnt.ai +short    # Should return 100.90.76.85
dig netbird.fxhnt.ai +short  # Should return 100.90.76.85
```

---

### Task 8b: Deploy Kanidm Pod

- [ ] **Step 1: Apply Kanidm manifests**

```bash
kubectl apply -f infra/k8s/kanidm/kanidm.yaml
kubectl apply -f infra/k8s/network-policies/kanidm.yaml
kubectl apply -f infra/k8s/kanidm/backup-cronjob.yaml
```

- [ ] **Step 2: Wait for pod ready**

```bash
kubectl -n foxhunt rollout status deployment/kanidm --timeout=120s
kubectl -n foxhunt get pods -l app.kubernetes.io/name=kanidm
```

Expected: `1/1 Running`.

- [ ] **Step 3: Retrieve admin password**

```bash
# Get the auto-generated admin password from first-boot logs
kubectl -n foxhunt logs deployment/kanidm | grep -i "admin password"
# If pod restarted and logs are lost:
# kubectl -n foxhunt logs deployment/kanidm --previous | grep -i "admin password"
# Or reset: kubectl -n foxhunt exec -it deployment/kanidm -- kanidm recover-account admin
```

Save the password securely — needed for initial setup.

- [ ] **Step 4: Create person account and groups**

Run each command interactively (one at a time via kubectl exec):

```bash
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm login -D admin

kubectl -n foxhunt exec -it deployment/kanidm -- kanidm person create jgrusewski "Jan Grusewski" -D admin

kubectl -n foxhunt exec -it deployment/kanidm -- kanidm group add-members idm_admins jgrusewski -D admin

kubectl -n foxhunt exec -it deployment/kanidm -- kanidm group create foxhunt_users -D admin

kubectl -n foxhunt exec -it deployment/kanidm -- kanidm group add-members foxhunt_users jgrusewski -D admin
```

- [ ] **Step 5: Register WebAuthn credential**

Open `https://auth.fxhnt.ai` in browser. Log in with admin password, then register Touch ID / platform authenticator via the Kanidm web UI.

---

### Task 8c: Create OIDC Clients & Secrets

- [ ] **Step 1: Create all 7 OIDC clients**

Run each command interactively (one at a time):

```bash
# Grafana
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 create grafana "Grafana" https://grafana.fxhnt.ai -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 update-scope-map grafana foxhunt_users openid profile email -D admin

# Mattermost
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 create mattermost "Mattermost" https://chat.fxhnt.ai -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 update-scope-map mattermost foxhunt_users openid profile email -D admin

# GitLab
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 create gitlab "GitLab" https://git.fxhnt.ai -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 update-scope-map gitlab foxhunt_users openid profile email -D admin

# Stalwart
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 create stalwart "Stalwart Mail" https://mail.fxhnt.ai -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 update-scope-map stalwart foxhunt_users openid profile email -D admin

# FXT Web Dashboard
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 create fxt-web "FXT Dashboard" https://dashboard.fxhnt.ai -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 update-scope-map fxt-web foxhunt_users openid profile email -D admin

# FXT API
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 create fxt-api "FXT API" https://api.fxhnt.ai -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 update-scope-map fxt-api foxhunt_users openid profile email -D admin

# NetBird
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 create netbird "NetBird" https://netbird.fxhnt.ai -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 update-scope-map netbird foxhunt_users openid profile email groups -D admin
```

- [ ] **Step 2: Retrieve client secrets and store in K8s secret**

```bash
# Get each client secret (run one at a time, note the output)
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 show-basic-secret grafana -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 show-basic-secret mattermost -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 show-basic-secret gitlab -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 show-basic-secret stalwart -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 show-basic-secret fxt-web -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 show-basic-secret fxt-api -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm system oauth2 show-basic-secret netbird -D admin

# Create K8s secret with all OIDC client secrets (replace <secret> with actual values)
kubectl -n foxhunt create secret generic kanidm-oidc-secrets \
  --from-literal=KANIDM_GRAFANA_SECRET='<secret>' \
  --from-literal=KANIDM_MATTERMOST_SECRET='<secret>' \
  --from-literal=KANIDM_GITLAB_SECRET='<secret>' \
  --from-literal=KANIDM_STALWART_SECRET='<secret>' \
  --from-literal=KANIDM_FXT_WEB_SECRET='<secret>' \
  --from-literal=KANIDM_FXT_API_SECRET='<secret>' \
  --from-literal=KANIDM_NETBIRD_SECRET='<secret>'
```

- [ ] **Step 3: Create service account for FXT CI**

```bash
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm service-account create fxt-ci "FXT CI Service Account" -D admin
kubectl -n foxhunt exec -it deployment/kanidm -- kanidm service-account api-token generate fxt-ci "ci-token" -D admin
# Save the generated token securely
```

- [ ] **Step 4: Verify OIDC discovery endpoints**

```bash
curl -sk https://auth.fxhnt.ai/oauth2/openid/grafana/.well-known/openid-configuration | jq .
```

Expected: JSON with `issuer`, `authorization_endpoint`, `token_endpoint`, `jwks_uri` fields.

---

### Task 9: Deploy NetBird to Cluster

- [ ] **Step 1: Apply NetBird manifests**

```bash
kubectl apply -f infra/k8s/netbird/netbird.yaml
kubectl apply -f infra/k8s/network-policies/netbird.yaml
```

- [ ] **Step 2: Wait for pods ready**

```bash
kubectl -n foxhunt rollout status deployment/netbird-server --timeout=120s
kubectl -n foxhunt rollout status deployment/netbird-dashboard --timeout=120s
```

Expected: Both `1/1 Running`.

- [ ] **Step 3: Verify NetBird metrics endpoint**

```bash
kubectl -n foxhunt exec deployment/netbird-server -- wget -qO- http://localhost:443/metrics 2>/dev/null | head -5
```

If no metrics on port 443, check if NetBird exposes metrics on a different port (e.g., 9090). Update the Prometheus scrape config in Task 7 Step 2 accordingly.

- [ ] **Step 4: Configure NetBird OIDC with Kanidm**

Access NetBird dashboard at `https://netbird.fxhnt.ai`, configure the generic OIDC provider:
- Type: Generic OIDC
- Client ID: `netbird`
- Client Secret: (from `kanidm-oidc-secrets`)
- Issuer: `https://auth.fxhnt.ai/oauth2/openid/netbird`

- [ ] **Step 5: Verify NetBird login via Kanidm**

Open `https://netbird.fxhnt.ai` → should redirect to `auth.fxhnt.ai` → WebAuthn prompt → logged into NetBird dashboard.

- [ ] **Step 6: Install NetBird client on workstation**

Install NetBird client alongside Tailscale (both can coexist — different WireGuard interfaces):

```bash
# macOS
brew install netbird/tap/netbird
# Connect to self-hosted management
netbird up --management-url https://netbird.fxhnt.ai
```

Verify: `netbird status` shows connected. Tailscale still works: `tailscale status` unchanged.

---

## Chunk 4: Service OIDC Integrations

### Task 10: Integrate Grafana with Kanidm OIDC

**Files:**
- Modify: `infra/k8s/gitlab/grafana-values.yaml`

- [ ] **Step 1: Add OIDC config to Grafana values**

Add to `grafana.ini` section in `infra/k8s/gitlab/grafana-values.yaml`:

```yaml
  auth.generic_oauth:
    enabled: true
    name: Kanidm
    client_id: grafana
    client_secret: $__env{KANIDM_GRAFANA_SECRET}
    scopes: openid profile email
    auth_url: https://auth.fxhnt.ai/ui/oauth2
    token_url: https://auth.fxhnt.ai/oauth2/token
    api_url: https://auth.fxhnt.ai/oauth2/openid/grafana/userinfo
    use_pkce: true
    role_attribute_path: "contains(groups[*], 'idm_admins') && 'Admin' || 'Viewer'"
```

Add environment variable from secret:

```yaml
envFromSecrets:
  - name: kanidm-oidc-secrets
    optional: false
```

**Note**: Keep `disable_login_form: false` during Phase 2 (additive, no breakage). Set to `true` in Phase 3 hardening.

- [ ] **Step 2: Validate and commit**

```bash
kubectl apply --dry-run=client -f infra/k8s/gitlab/grafana-values.yaml 2>/dev/null || echo "Helm values - no dry-run needed"
git add infra/k8s/gitlab/grafana-values.yaml
git commit -m "feat(grafana): add Kanidm OIDC authentication (additive, login form kept)"
```

- [ ] **Step 3: Apply and verify**

```bash
helm upgrade grafana grafana/grafana -n foxhunt -f infra/k8s/gitlab/grafana-values.yaml
```

Open `https://grafana.fxhnt.ai` — should show "Login with Kanidm" button alongside existing login form.

---

### Task 11: Integrate Mattermost with Kanidm OIDC

**Files:**
- Modify: `infra/k8s/mattermost/mattermost.yaml` (Deployment env vars)

Mattermost Team Edition uses `MM_*` env vars for configuration (no Helm, no ConfigMap for settings).

- [ ] **Step 1: Add GitLabSettings OIDC env vars to Mattermost deployment**

In `infra/k8s/mattermost/mattermost.yaml`, add these env vars to the container spec
(after the existing `MM_FILESETTINGS_DIRECTORY` entry):

```yaml
            # Kanidm OIDC via GitLab adapter (Phase 2 — additive, email login kept)
            # WARNING: KANIDM_MATTERMOST_SECRET MUST be defined before MM_GITLABSETTINGS_SECRET.
            # K8s $(VAR) substitution is order-dependent — moving it below will silently
            # produce an empty string, breaking OIDC authentication.
            - name: KANIDM_MATTERMOST_SECRET
              valueFrom:
                secretKeyRef:
                  name: kanidm-oidc-secrets
                  key: KANIDM_MATTERMOST_SECRET
            - name: MM_GITLABSETTINGS_ENABLE
              value: "true"
            - name: MM_GITLABSETTINGS_ID
              value: "mattermost"
            - name: MM_GITLABSETTINGS_SECRET
              value: "$(KANIDM_MATTERMOST_SECRET)"
            - name: MM_GITLABSETTINGS_AUTHENDPOINT
              value: "https://auth.fxhnt.ai/ui/oauth2"
            - name: MM_GITLABSETTINGS_TOKENENDPOINT
              value: "https://auth.fxhnt.ai/oauth2/token"
            - name: MM_GITLABSETTINGS_USERAPIENDPOINT
              value: "https://auth.fxhnt.ai/oauth2/openid/mattermost/userinfo"
            - name: MM_GITLABSETTINGS_SCOPE
              value: "openid profile email"
```

**Important**: Test the userinfo response — Mattermost expects `username` field, Kanidm returns `preferred_username`. If login fails, switch to `MM_OPENIDSETTINGS_*` env vars instead of `MM_GITLABSETTINGS_*`.

- [ ] **Step 2: Validate YAML**

```bash
kubectl apply --dry-run=client -f infra/k8s/mattermost/mattermost.yaml
```

- [ ] **Step 3: Apply and verify**

```bash
kubectl apply -f infra/k8s/mattermost/mattermost.yaml
kubectl -n foxhunt rollout restart deployment/mattermost
kubectl -n foxhunt rollout status deployment/mattermost --timeout=120s
```

Open `https://chat.fxhnt.ai` — should show "Login with GitLab" button alongside existing email login.

- [ ] **Step 4: Commit**

```bash
git add infra/k8s/mattermost/mattermost.yaml
git commit -m "feat(mattermost): add Kanidm OIDC authentication via GitLab adapter"
```

---

### Task 12: Integrate GitLab CE with Kanidm OIDC

**Files:**
- Modify: `infra/k8s/gitlab/values.yaml`

- [ ] **Step 1: Create GitLab OIDC secret**

```bash
kubectl -n foxhunt create secret generic gitlab-oidc-kanidm \
  --from-literal=provider='{
    "name": "openid_connect",
    "label": "Kanidm",
    "args": {
      "name": "openid_connect",
      "scope": ["openid", "profile", "email"],
      "response_type": "code",
      "issuer": "https://auth.fxhnt.ai/oauth2/openid/gitlab",
      "client_auth_method": "basic",
      "discovery": true,
      "uid_field": "sub",
      "pkce": true,
      "client_options": {
        "identifier": "gitlab",
        "secret": "<from kanidm-oidc-secrets>",
        "redirect_uri": "https://git.fxhnt.ai/users/auth/openid_connect/callback"
      }
    }
  }'
```

- [ ] **Step 2: Add OmniAuth config to GitLab Helm values**

Add to `global.appConfig` in `infra/k8s/gitlab/values.yaml`:

```yaml
    omniauth:
      enabled: true
      allowSingleSignOn: ['openid_connect']
      blockAutoCreatedUsers: false
      providers:
        - secret: gitlab-oidc-kanidm
          key: provider
```

**Note**: Do NOT set `autoSignInWithProvider` yet — Phase 2 is additive. Set it in Phase 3.

- [ ] **Step 3: Upgrade GitLab and verify**

```bash
helm upgrade gitlab gitlab/gitlab -n foxhunt -f infra/k8s/gitlab/values.yaml
```

Open `https://git.fxhnt.ai` — should show "Sign in with Kanidm" button alongside existing login.

- [ ] **Step 4: Commit**

```bash
git add infra/k8s/gitlab/values.yaml
git commit -m "feat(gitlab): add Kanidm OIDC via OmniAuth openid_connect (additive)"
```

---

### Task 13: Integrate Stalwart Mail with Kanidm OIDC

**Files:**
- Modify: `infra/k8s/stalwart/stalwart.yaml` (ConfigMap `stalwart-config`, `config.toml` section)

Stalwart uses a ConfigMap-mounted `config.toml` at `/opt/stalwart/etc/config.toml`.
The OIDC secret comes from the `kanidm-oidc-secrets` K8s secret via an env var, referenced
in TOML with `%{env:KANIDM_STALWART_SECRET}%`.

- [ ] **Step 1: Add KANIDM_STALWART_SECRET env var to Stalwart deployment**

In `infra/k8s/stalwart/stalwart.yaml`, add this env var to the container spec (after the existing `ADMIN_SECRET` entry):

```yaml
            - name: KANIDM_STALWART_SECRET
              valueFrom:
                secretKeyRef:
                  name: kanidm-oidc-secrets
                  key: KANIDM_STALWART_SECRET
```

- [ ] **Step 2: Add OIDC auth backend to stalwart-config ConfigMap**

In `infra/k8s/stalwart/stalwart.yaml`, append to the `config.toml` data in the `stalwart-config` ConfigMap (after the `[certificate.default]` section):

```toml

    [authentication.oidc]
    enable = true
    issuer-url = "https://auth.fxhnt.ai/oauth2/openid/stalwart"
    client-id = "stalwart"
    client-secret = "%{env:KANIDM_STALWART_SECRET}%"
    scopes = ["openid", "profile", "email"]
    account-mapping = "email"
```

- [ ] **Step 3: Validate YAML**

```bash
kubectl apply --dry-run=client -f infra/k8s/stalwart/stalwart.yaml
```

- [ ] **Step 4: Apply and verify**

```bash
kubectl apply -f infra/k8s/stalwart/stalwart.yaml
kubectl -n foxhunt rollout restart deployment/stalwart
kubectl -n foxhunt rollout status deployment/stalwart --timeout=120s
```

Test: Open `https://mail.fxhnt.ai` admin UI — should show OIDC login option. Also test IMAP login with OIDC token (via mail client XOAUTH2 support).

- [ ] **Step 5: Commit**

```bash
git add infra/k8s/stalwart/stalwart.yaml
git commit -m "feat(stalwart): add Kanidm OIDC authentication backend"
```

---

## Chunk 5: Apply Monitoring

### Task 14: Apply Monitoring Changes

**Note**: DNS and nginx proxy were already applied in Task 8a. This task applies the remaining monitoring configuration.

- [ ] **Step 1: Apply updated monitoring config**

```bash
kubectl apply -f infra/k8s/monitoring/prometheus-rules-hft.yaml
helm upgrade prometheus-stack prometheus-community/kube-prometheus-stack \
  -n foxhunt -f infra/k8s/monitoring/prometheus-stack-values.yaml
kubectl apply -f infra/k8s/network-policies/prometheus.yaml
```

- [ ] **Step 2: Validate all services accessible**

```bash
curl -sk https://auth.fxhnt.ai/status          # Kanidm health
curl -sk https://netbird.fxhnt.ai/              # NetBird dashboard
curl -sk https://grafana.fxhnt.ai/              # Grafana (unchanged)
curl -sk https://chat.fxhnt.ai/                 # Mattermost (unchanged)
curl -sk https://git.fxhnt.ai/                  # GitLab (unchanged)
```

---

## Chunk 6: Phase 3 Hardening (Future — After Validation)

> **Do NOT execute this chunk until all Phase 2 integrations are validated working for at least 1 week.**

### Task 15: Disable Old Auth Methods

- [ ] **Step 1: Grafana — disable login form**

In `infra/k8s/gitlab/grafana-values.yaml`, change the `auth` section:

```yaml
  auth:
    disable_login_form: true
```

Keep break-glass: `admin` user remains with local password in K8s secret.

- [ ] **Step 2: Mattermost — disable email signup**

In `infra/k8s/mattermost/mattermost.yaml`, add env var:

```yaml
            - name: MM_EMAILSETTINGS_ENABLESIGNUPWITHEMAIL
              value: "false"
```

- [ ] **Step 3: GitLab — auto-redirect to Kanidm**

In `infra/k8s/gitlab/values.yaml`, update the `omniauth` section:

```yaml
    omniauth:
      enabled: true
      allowSingleSignOn: ['openid_connect']
      blockAutoCreatedUsers: false
      autoSignInWithProvider: openid_connect
      providers:
        - secret: gitlab-oidc-kanidm
          key: provider
```

Keep break-glass: `root` user with local password.

- [ ] **Step 4: FXT API — replace HS256 with RS256 JWKS validation**

**Files:**
- Modify: `services/api/src/auth/jwt/service.rs` (lines 345-346)
- Modify: `services/api/src/auth/interceptor.rs` (lines 336-338)

In `services/api/src/auth/jwt/service.rs`, replace the HS256 validation:

```rust
// BEFORE (HS256 — shared secret):
let key = DecodingKey::from_secret(self.config.jwt_secret.as_ref());
let mut validation = Validation::new(Algorithm::HS256);

// AFTER (RS256 — JWKS public key from Kanidm):
let key = self.jwks_decoding_key.as_ref()
    .ok_or_else(|| anyhow::anyhow!("JWKS key not loaded — Kanidm unreachable?"))?;
let mut validation = Validation::new(Algorithm::RS256);
```

Add a JWKS fetcher that caches the public key from Kanidm on startup and refreshes on a TTL + reactively on unknown `kid`:

```rust
// In JwtConfig or a new JwksCache struct:
// Fetch: GET https://auth.fxhnt.ai/oauth2/openid/fxt-api/public_key.jwk
// Parse: jsonwebtoken::DecodingKey::from_rsa_components(n, e)
// Cache: Arc<RwLock<DecodingKey>>, background refresh every 5 min
// Reactive refresh: if token validation fails with "kid not found", trigger
//   immediate out-of-band JWKS fetch. If new key matches, retry validation.
//   Rate-limit reactive refreshes to 1 per 10s to prevent abuse.
// Env: KANIDM_JWKS_URL (defaults to https://auth.fxhnt.ai/oauth2/openid/fxt-api/public_key.jwk)
// Env: JWKS_REFRESH_INTERVAL_SECS (default 300)
```

In `services/api/src/auth/interceptor.rs`, same change:

```rust
// BEFORE:
let decoding_key = Arc::new(DecodingKey::from_secret(secret.as_bytes()));
let mut validation = Validation::new(Algorithm::HS256);

// AFTER:
let decoding_key = Arc::new(jwks_cache.current_key());
let mut validation = Validation::new(Algorithm::RS256);
```

After migration, remove `JWT_SECRET` / `JWT_SECRET_FILE` env vars and Vault secret path from all service deployments.

- [ ] **Step 5: Commit hardening changes**

```bash
git add infra/k8s/gitlab/grafana-values.yaml \
        infra/k8s/mattermost/mattermost.yaml \
        infra/k8s/gitlab/values.yaml \
        services/api/src/auth/jwt/service.rs \
        services/api/src/auth/interceptor.rs
git commit -m "feat(security): Phase 3 hardening — disable local auth, RS256-only JWT

Disable login forms (Grafana, Mattermost email signup, GitLab auto-redirect).
Replace HS256 shared-secret JWT with RS256 JWKS from Kanidm.
Break-glass: admin/root local accounts kept on Grafana and GitLab."
```
