foxhunt/docs/superpowers/specs/2026-03-14-kanidm-sso-design.md

Kanidm SSO/MFA + NetBird Mesh — Passwordless Zero-Trust Identity & Network

Date: 2026-03-14 Status: Approved Scope: 6 Tailnet-exposed services + NetBird parallel mesh (side-by-side with Tailscale)

Problem

6 separate credential sets across Grafana, Mattermost, GitLab CE, Stalwart Mail, FXT Web Dashboard, and FXT CLI/gRPC. No SSO, no MFA, no passwordless auth. Shared HS256 JWT secret for FXT services. Redis, QuestDB, Prometheus, Loki, Tempo have no authentication (network-isolated only — acceptable).

Decision

Kanidm — Rust-native, WebAuthn-first identity provider.

Why Kanidm over alternatives

Criteria	Kanidm	Keycloak	Authentik	Authelia
RAM	~50MB	~384MB	~300MB	~30MB
Passwordless-first	Default	Configurable	No (MFA only)	No (MFA only)
External deps	None	PostgreSQL	PG + Redis	Optional Redis
Device auth flow	Possible	Native	Limited	Limited
OIDC maturity	Solid	Mature	Mature	Good

Deciding factors: DEV1-L nodes (3 × 8GB = 24GB total RAM) make resource efficiency critical. Keycloak at 384MB is 1.5% of cluster vs Kanidm at 50MB (0.2%). Authelia can't do passwordless-first. All 6 target services support OIDC.

Architecture

Deployment

• Single pod on platform node pool (DEV1-L)
• Container: kanidm/server:1.9.2 (pinned, ~50MB RSS)
• Embedded database (no PostgreSQL/Redis dependency)
• PVC: 1Gi scw-bssd for identity database
• Deployment strategy: Recreate (RWO PVC — cannot run two pods simultaneously)
• TLS: *.fxhnt.ai wildcard cert, exposed as auth.fxhnt.ai
• Tailscale access via existing nginx proxy (100.90.76.85)

Kanidm server.toml

Required configuration (Kanidm cannot start without these):

[server]
domain = "auth.fxhnt.ai"
origin = "https://auth.fxhnt.ai"
tls_chain = "/data/certs/tls.crt"
tls_key = "/data/certs/tls.key"
bindaddress = "[::]:8443"
db_path = "/data/kanidm.db"

Mount the *.fxhnt.ai wildcard cert into the pod at /data/certs/ from the existing gitlab-tls-cert K8s secret. Mount server.toml from a ConfigMap.

Resource Budget

Resource	Request	Limit
CPU	25m	100m
Memory	48Mi	96Mi
PVC	1Gi	—

Identity Model

• 1 person account: WebAuthn (Touch ID) as sole authenticator. No password.
• Service accounts: API tokens for headless flows (CI, scripts). Client credentials grant.
• Groups: idm_admins (full admin), foxhunt_users (OIDC login allowed).
• 7 OIDC clients: grafana, mattermost, gitlab, stalwart, fxt-web, fxt-api, netbird.

Token Architecture

Property	Value
Signing	RS256 (asymmetric — services validate with JWKS)
ID token lifetime	15 minutes
Access token lifetime	8 hours (interactive), 24h (service account)
Refresh token	7 days, rotate on use
JWKS endpoint	Per-client: https://auth.fxhnt.ai/oauth2/openid/{client}/public_key.jwk
Discovery	Per-client: https://auth.fxhnt.ai/oauth2/openid/{client}/.well-known/openid-configuration

Service Integrations

1. Grafana — Native OIDC (auth.generic_oauth)

grafana.ini:
  auth.generic_oauth:
    enabled: true
    name: Kanidm
    client_id: grafana
    client_secret: <from K8s secret>
    scopes: openid profile email
    auth_url: https://auth.fxhnt.ai/ui/oauth2
    token_url: https://auth.fxhnt.ai/oauth2/token
    api_url: https://auth.fxhnt.ai/oauth2/openid/grafana/userinfo
    use_pkce: true
    role_attribute_path: "contains(groups[*], 'idm_admins') && 'Admin' || 'Viewer'"
  auth:
    disable_login_form: true

2. Mattermost — GitLab OAuth adapter (Slack-compatible OIDC)

config:
  GitLabSettings:
    Enable: true
    Id: mattermost
    Secret: <from K8s secret>
    AuthEndpoint: https://auth.fxhnt.ai/ui/oauth2
    TokenEndpoint: https://auth.fxhnt.ai/oauth2/token
    UserApiEndpoint: https://auth.fxhnt.ai/oauth2/openid/mattermost/userinfo
    Scope: openid profile email

Note: Mattermost's GitLab adapter expects username (GitLab's field name) in the userinfo response. Kanidm uses standard OIDC preferred_username. Test the userinfo response shape during Phase 2 — may need Kanidm's claim mapping or a Mattermost OpenIdSettings config instead.

3. GitLab CE — OmniAuth openid_connect

K8s secret gitlab-oidc-kanidm:

{
  "name": "openid_connect",
  "label": "Kanidm",
  "args": {
    "name": "openid_connect",
    "scope": ["openid", "profile", "email"],
    "response_type": "code",
    "issuer": "https://auth.fxhnt.ai/oauth2/openid/gitlab",
    "client_auth_method": "basic",
    "discovery": true,
    "uid_field": "sub",
    "pkce": true,
    "client_options": {
      "identifier": "gitlab",
      "secret": "<client-secret>",
      "redirect_uri": "https://git.fxhnt.ai/users/auth/openid_connect/callback"
    }
  }
}

4. Stalwart Mail — OIDC auth backend

[authentication.oidc]
enable = true
issuer-url = "https://auth.fxhnt.ai/oauth2/openid/stalwart"
client-id = "stalwart"
client-secret = "<from env>"
scopes = ["openid", "profile", "email"]
account-mapping = "email"

5. FXT Web Dashboard — OIDC Authorization Code + PKCE

const config: OidcClientSettings = {
  authority: 'https://auth.fxhnt.ai/oauth2/openid/fxt-web',
  client_id: 'fxt-web',
  redirect_uri: 'https://dashboard.fxhnt.ai/callback',
  scope: 'openid profile email',
  response_type: 'code',
};

6. FXT CLI/gRPC — Device Authorization Grant

Interactive: fxt login → browser opens → Touch ID → token cached 8h. Headless/CI: service account fxt-ci with API token (client credentials grant).

Proxy Routing

Existing tailscale-gitlab-proxy nginx gets a new server block:

server {
    listen 443 ssl;
    server_name auth.fxhnt.ai;

    ssl_certificate /etc/nginx/certs/tls.crt;
    ssl_certificate_key /etc/nginx/certs/tls.key;

    location / {
        # Kanidm uses self-signed internal TLS — external TLS terminates at nginx
        proxy_pass https://kanidm.foxhunt.svc.cluster.local:8443;
        proxy_ssl_verify off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        # WebSocket support (Kanidm admin UI, WebAuthn flow)
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

DNS: auth.fxhnt.ai A record → 100.90.76.85 (Scaleway DNS module).

Network Policy

spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: kanidm
  policyTypes: [Ingress, Egress]
  ingress:
    # Tailscale proxy (nginx) — OIDC redirects from all Tailnet services
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tailscale-gitlab-proxy
      ports:
        - { protocol: TCP, port: 8443 }
    # In-cluster services that validate tokens directly
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/part-of: foxhunt
      ports:
        - { protocol: TCP, port: 8443 }
  egress:
    - to:
        - namespaceSelector: {}
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

Migration Strategy

Phase 1: Deploy Kanidm (Day 1)

1. Deploy Kanidm pod + PVC + Service + NetworkPolicy
2. Add auth.fxhnt.ai DNS + nginx upstream
3. Create person account with WebAuthn (Touch ID)
4. Create service account fxt-ci
5. Verify OIDC endpoints

Phase 2: Integrate Services — Additive (Day 2-3)

OIDC added alongside existing auth (no breakage):

Order	Service	Reason
1	Grafana	Easiest OIDC, low risk
2	FXT Web Dashboard	Own code, fast iteration
3	Mattermost	GitLab OAuth adapter
4	FXT CLI/gRPC	Device auth flow
5	GitLab CE	OmniAuth — more moving parts
6	Stalwart Mail	Least documented integration

Phase 3: Harden — Disable Old Auth (Day 4+)

1. Disable login forms (Grafana, Mattermost, GitLab)
2. Remove HS256 shared JWT secret from FXT
3. Keep break-glass local admin on GitLab + Grafana

Break-Glass Emergency Access

• GitLab: root local account
• Grafana: admin local account (K8s secret)
• All others: kubectl port-forward

Operational Concerns

Backup

• Daily CronJob: kanidm database backup → MinIO S3 (foxhunt-kanidm-backup bucket)
• CronJob pod mounts Kanidm PVC (ReadOnly) + MinIO credentials secret
• Uses kanidm database backup -o /tmp/kanidm.backup then rclone copy to S3
• No Kanidm auth needed — backup reads the DB file directly (offline-safe)
• RPO: 24h, RTO: ~5 minutes

Monitoring

• Kanidm does not expose a native Prometheus metrics endpoint
• Use Prometheus blackbox exporter probe against https://kanidm.foxhunt.svc.cluster.local:8443/status (health check)
• Alert: KanidmDown — probe_success{job="kanidm"} == 0 for 30s, severity critical

Session Revocation

• RS256 tokens are validated statelessly (JWKS only) — no per-request callback to Kanidm
• Blast radius on key compromise: access tokens remain valid until expiry (8h interactive, 24h service)
• Mitigation: reduce access token lifetime to 1h if faster revocation is needed; refresh tokens can be revoked server-side immediately via kanidm session CLI
• Break-glass: rotate the RS256 signing key (kanidm key rotate) to invalidate ALL tokens instantly

Upgrades

• Image pinned to kanidm/server:1.9.2 — upgrade deliberately after reading changelogs
• Embedded DB migrations run automatically on startup
• Test OIDC flow after each upgrade
• Kanidm is pre-2.0 — expect breaking changes between minor versions

Key Rotation

• OIDC client secrets: kanidm system oauth2 CLI
• Service account tokens: kanidm service-account api-token generate
• RS256 signing keys: kanidm key rotate

NetBird — Parallel Mesh (Side-by-Side with Tailscale)

Tailscale remains the primary mesh VPN. NetBird runs alongside it for evaluation. Both meshes connect the same devices. Traffic gradually shifts to NetBird as confidence builds. Zero breaking changes — Tailscale continues working unchanged.

Why NetBird

• Fully open-source (AGPLv3) — entire stack self-hosted, no proprietary control plane
• Built-in SSO/MFA via OIDC (integrates with Kanidm via generic OIDC connector)
• Device posture checks (zero-trust at network layer)
• Quantum-resistant mesh (post-quantum crypto, since v0.65)
• Platform coverage: macOS, Windows, Android (8.0+), iOS (14.0+), Linux
• Built-in reverse proxy (since Feb 2026)

Architecture

Your Devices
  ├── Tailscale client (primary)  ─── Tailscale cloud control ─── 100.x.x.x
  └── NetBird client (parallel)   ─── Self-hosted control     ─── 10.x.x.x (NetBird CIDR)
                                      (on platform pool)

Cluster:
  ┌─────────────────────────────────┐
  │  NetBird Server (all-in-one)    │
  │  ├─ Management API              │
  │  ├─ Signal server               │
  │  ├─ Relay (STUN/TURN)           │
  │  └─ Dashboard UI                │
  │  Port 443 (HTTPS) + 3478 (UDP) │
  │  OIDC: Kanidm (generic OIDC)   │
  └─────────────────────────────────┘

Deployment

• Image: netbirdio/netbird-server:0.66.0 (all-in-one: management + signal + relay + STUN)
• Dashboard: netbirdio/dashboard:v2.33.0
• Pool: platform node pool (DEV1-L)
• PVC: 1Gi scw-bssd (RWO) at /var/lib/netbird
• Strategy: Recreate (RWO PVC)
• DNS: netbird.fxhnt.ai A record → 100.90.76.85 (via existing nginx proxy)

Resource Budget

Component	CPU Req	CPU Limit	Mem Req	Mem Limit
Server	25m	200m	64Mi	256Mi
Dashboard	10m	50m	16Mi	64Mi
Total	35m	250m	80Mi	320Mi

~80MB baseline — comparable to Kanidm's 50MB. Together ~130MB for the full zero-trust stack.

Kanidm OIDC Integration

NetBird's generic OIDC connector connects to Kanidm:

# NetBird management config
idp:
  type: generic
  clientId: netbird
  clientSecret: <from K8s secret>
  issuer: https://auth.fxhnt.ai/oauth2/openid/netbird

Required OIDC claims from Kanidm: sub (required), email (recommended), name (recommended), groups (for group sync).

Redirect URI: https://netbird.fxhnt.ai/oauth2/callback/{connector-id}

Create OIDC client netbird in Kanidm (same as the other 6 clients).

Nginx Proxy Block

# NetBird Dashboard + Management API
server {
    listen 443 ssl;
    server_name netbird.fxhnt.ai;

    ssl_certificate /etc/nginx/certs/tls.crt;
    ssl_certificate_key /etc/nginx/certs/tls.key;

    location / {
        proxy_pass http://netbird-dashboard.foxhunt.svc.cluster.local:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    # Management API
    location /api {
        proxy_pass http://netbird-server.foxhunt.svc.cluster.local:443;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }

    # Signal (gRPC)
    location /signalexchange.SignalExchange/ {
        grpc_pass grpc://netbird-server.foxhunt.svc.cluster.local:10000;
    }
}

STUN/TURN: UDP port 3478 — needs a separate NodePort or hostNetwork since nginx can't proxy UDP. Use a dedicated Service with type: NodePort on UDP 3478.

Network Policy

spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: netbird
  policyTypes: [Ingress, Egress]
  ingress:
    # Nginx proxy (HTTPS dashboard + management API)
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tailscale-gitlab-proxy
      ports:
        - { protocol: TCP, port: 80 }    # dashboard
        - { protocol: TCP, port: 443 }   # management API
        - { protocol: TCP, port: 10000 } # signal (gRPC)
    # STUN/TURN from external (all peers)
    - ports:
        - { protocol: UDP, port: 3478 }
    # Peer connections from foxhunt namespace
    - from:
        - podSelector: {}
      ports:
        - { protocol: TCP, port: 443 }
  egress:
    # DNS
    - to:
        - namespaceSelector: {}
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    # Kanidm OIDC (token validation)
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: kanidm
      ports:
        - { protocol: TCP, port: 8443 }

Side-by-Side Strategy

Aspect	Tailscale (primary)	NetBird (parallel)
Status	Primary — no changes	Evaluation
Control plane	Tailscale cloud (proprietary)	Self-hosted (platform pool)
IP range	100.x.x.x (CGNAT)	10.x.x.x (NetBird default)
Client install	Already on all devices	Install alongside Tailscale
DNS	*.fxhnt.ai → Tailscale IP	netbird.fxhnt.ai → same Tailscale IP (dashboard only)
Service access	All services via Tailscale IPs	Test access via NetBird IPs
Auth	Tailscale SSO	Kanidm OIDC
Breakage risk	Zero — nothing changes	Additive only

Migration Path (Future)

1. Now: Both clients on all devices, Tailscale primary
2. Evaluate: Access services through NetBird, compare latency/reliability
3. Confidence: Move non-critical services (Grafana, Mattermost) to NetBird-only DNS
4. Full migration: Redirect *.fxhnt.ai DNS to NetBird IPs, decommission Tailscale
5. Never forced: If NetBird disappoints, remove it. Tailscale never touched.

Monitoring

• Prometheus scrape: NetBird management exposes metrics
• Alert: NetBirdDown — up{job="netbird"} == 0 for 2m, severity medium (not critical — Tailscale is primary)