# Kanidm SSO/MFA + NetBird Mesh — Passwordless Zero-Trust Identity & Network

**Date**: 2026-03-14
**Status**: Approved
**Scope**: 6 Tailnet-exposed services + NetBird parallel mesh (side-by-side with Tailscale)

## Problem

6 separate credential sets across Grafana, Mattermost, GitLab CE, Stalwart Mail, FXT Web Dashboard, and FXT CLI/gRPC. No SSO, no MFA, no passwordless auth. Shared HS256 JWT secret for FXT services. Redis, QuestDB, Prometheus, Loki, Tempo have no authentication (network-isolated only — acceptable).

## Decision

**Kanidm** — Rust-native, WebAuthn-first identity provider.

### Why Kanidm over alternatives

| Criteria | Kanidm | Keycloak | Authentik | Authelia |
|----------|--------|----------|-----------|----------|
| RAM | ~50MB | ~384MB | ~300MB | ~30MB |
| Passwordless-first | **Default** | Configurable | No (MFA only) | No (MFA only) |
| External deps | None | PostgreSQL | PG + Redis | Optional Redis |
| Device auth flow | Possible | Native | Limited | Limited |
| OIDC maturity | Solid | Mature | Mature | Good |

**Deciding factors**: DEV1-L nodes (3 × 8GB = 24GB total RAM) make resource efficiency critical. Keycloak at 384MB is 1.5% of cluster vs Kanidm at 50MB (0.2%). Authelia can't do passwordless-first. All 6 target services support OIDC.

## Architecture

### Deployment

- Single pod on `platform` node pool (DEV1-L)
- Container: `kanidm/server:1.9.2` (pinned, ~50MB RSS)
- Embedded database (no PostgreSQL/Redis dependency)
- PVC: 1Gi `scw-bssd` for identity database
- Deployment strategy: **Recreate** (RWO PVC — cannot run two pods simultaneously)
- TLS: `*.fxhnt.ai` wildcard cert, exposed as `auth.fxhnt.ai`
- Tailscale access via existing nginx proxy (100.90.76.85)

### Kanidm `server.toml`

Required configuration (Kanidm cannot start without these):

```toml
[server]
domain = "auth.fxhnt.ai"
origin = "https://auth.fxhnt.ai"
tls_chain = "/data/certs/tls.crt"
tls_key = "/data/certs/tls.key"
bindaddress = "[::]:8443"
db_path = "/data/kanidm.db"
```

Mount the `*.fxhnt.ai` wildcard cert into the pod at `/data/certs/` from the existing `gitlab-tls-cert` K8s secret. Mount `server.toml` from a ConfigMap.

### Resource Budget

| Resource | Request | Limit |
|----------|---------|-------|
| CPU | 25m | 100m |
| Memory | 48Mi | 96Mi |
| PVC | 1Gi | — |

### Identity Model

- **1 person account**: WebAuthn (Touch ID) as sole authenticator. No password.
- **Service accounts**: API tokens for headless flows (CI, scripts). Client credentials grant.
- **Groups**: `idm_admins` (full admin), `foxhunt_users` (OIDC login allowed).
- **7 OIDC clients**: grafana, mattermost, gitlab, stalwart, fxt-web, fxt-api, netbird.

### Token Architecture

| Property | Value |
|----------|-------|
| Signing | RS256 (asymmetric — services validate with JWKS) |
| ID token lifetime | 15 minutes |
| Access token lifetime | 8 hours (interactive), 24h (service account) |
| Refresh token | 7 days, rotate on use |
| JWKS endpoint | Per-client: `https://auth.fxhnt.ai/oauth2/openid/{client}/public_key.jwk` |
| Discovery | Per-client: `https://auth.fxhnt.ai/oauth2/openid/{client}/.well-known/openid-configuration` |

## Service Integrations

### 1. Grafana — Native OIDC (`auth.generic_oauth`)

```yaml
grafana.ini:
  auth.generic_oauth:
    enabled: true
    name: Kanidm
    client_id: grafana
    client_secret: <from K8s secret>
    scopes: openid profile email
    auth_url: https://auth.fxhnt.ai/ui/oauth2
    token_url: https://auth.fxhnt.ai/oauth2/token
    api_url: https://auth.fxhnt.ai/oauth2/openid/grafana/userinfo
    use_pkce: true
    role_attribute_path: "contains(groups[*], 'idm_admins') && 'Admin' || 'Viewer'"
  auth:
    disable_login_form: true
```

### 2. Mattermost — GitLab OAuth adapter (Slack-compatible OIDC)

```yaml
config:
  GitLabSettings:
    Enable: true
    Id: mattermost
    Secret: <from K8s secret>
    AuthEndpoint: https://auth.fxhnt.ai/ui/oauth2
    TokenEndpoint: https://auth.fxhnt.ai/oauth2/token
    UserApiEndpoint: https://auth.fxhnt.ai/oauth2/openid/mattermost/userinfo
    Scope: openid profile email
```

**Note**: Mattermost's GitLab adapter expects `username` (GitLab's field name) in the userinfo response. Kanidm uses standard OIDC `preferred_username`. Test the userinfo response shape during Phase 2 — may need Kanidm's claim mapping or a Mattermost `OpenIdSettings` config instead.

### 3. GitLab CE — OmniAuth `openid_connect`

K8s secret `gitlab-oidc-kanidm`:
```json
{
  "name": "openid_connect",
  "label": "Kanidm",
  "args": {
    "name": "openid_connect",
    "scope": ["openid", "profile", "email"],
    "response_type": "code",
    "issuer": "https://auth.fxhnt.ai/oauth2/openid/gitlab",
    "client_auth_method": "basic",
    "discovery": true,
    "uid_field": "sub",
    "pkce": true,
    "client_options": {
      "identifier": "gitlab",
      "secret": "<client-secret>",
      "redirect_uri": "https://git.fxhnt.ai/users/auth/openid_connect/callback"
    }
  }
}
```

### 4. Stalwart Mail — OIDC auth backend

```toml
[authentication.oidc]
enable = true
issuer-url = "https://auth.fxhnt.ai/oauth2/openid/stalwart"
client-id = "stalwart"
client-secret = "<from env>"
scopes = ["openid", "profile", "email"]
account-mapping = "email"
```

### 5. FXT Web Dashboard — OIDC Authorization Code + PKCE

```typescript
const config: OidcClientSettings = {
  authority: 'https://auth.fxhnt.ai/oauth2/openid/fxt-web',
  client_id: 'fxt-web',
  redirect_uri: 'https://dashboard.fxhnt.ai/callback',
  scope: 'openid profile email',
  response_type: 'code',
};
```

### 6. FXT CLI/gRPC — Device Authorization Grant

Interactive: `fxt login` → browser opens → Touch ID → token cached 8h.
Headless/CI: service account `fxt-ci` with API token (client credentials grant).

## Proxy Routing

Existing `tailscale-gitlab-proxy` nginx gets a new server block:

```nginx
server {
    listen 443 ssl;
    server_name auth.fxhnt.ai;

    ssl_certificate /etc/nginx/certs/tls.crt;
    ssl_certificate_key /etc/nginx/certs/tls.key;

    location / {
        # Kanidm uses self-signed internal TLS — external TLS terminates at nginx
        proxy_pass https://kanidm.foxhunt.svc.cluster.local:8443;
        proxy_ssl_verify off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        # WebSocket support (Kanidm admin UI, WebAuthn flow)
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}
```

DNS: `auth.fxhnt.ai` A record → `100.90.76.85` (Scaleway DNS module).

## Network Policy

```yaml
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: kanidm
  policyTypes: [Ingress, Egress]
  ingress:
    # Tailscale proxy (nginx) — OIDC redirects from all Tailnet services
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tailscale-gitlab-proxy
      ports:
        - { protocol: TCP, port: 8443 }
    # In-cluster services that validate tokens directly
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/part-of: foxhunt
      ports:
        - { protocol: TCP, port: 8443 }
  egress:
    - to:
        - namespaceSelector: {}
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
```

## Migration Strategy

### Phase 1: Deploy Kanidm (Day 1)
1. Deploy Kanidm pod + PVC + Service + NetworkPolicy
2. Add `auth.fxhnt.ai` DNS + nginx upstream
3. Create person account with WebAuthn (Touch ID)
4. Create service account `fxt-ci`
5. Verify OIDC endpoints

### Phase 2: Integrate Services — Additive (Day 2-3)
OIDC added alongside existing auth (no breakage):

| Order | Service | Reason |
|-------|---------|--------|
| 1 | Grafana | Easiest OIDC, low risk |
| 2 | FXT Web Dashboard | Own code, fast iteration |
| 3 | Mattermost | GitLab OAuth adapter |
| 4 | FXT CLI/gRPC | Device auth flow |
| 5 | GitLab CE | OmniAuth — more moving parts |
| 6 | Stalwart Mail | Least documented integration |

### Phase 3: Harden — Disable Old Auth (Day 4+)
1. Disable login forms (Grafana, Mattermost, GitLab)
2. Remove HS256 shared JWT secret from FXT
3. Keep break-glass local admin on GitLab + Grafana

### Break-Glass Emergency Access
- GitLab: root local account
- Grafana: admin local account (K8s secret)
- All others: kubectl port-forward

## Operational Concerns

### Backup
- Daily CronJob: `kanidm database backup` → MinIO S3 (`foxhunt-kanidm-backup` bucket)
- CronJob pod mounts Kanidm PVC (ReadOnly) + MinIO credentials secret
- Uses `kanidm database backup -o /tmp/kanidm.backup` then `rclone copy` to S3
- No Kanidm auth needed — backup reads the DB file directly (offline-safe)
- RPO: 24h, RTO: ~5 minutes

### Monitoring
- Kanidm does not expose a native Prometheus metrics endpoint
- Use Prometheus blackbox exporter probe against `https://kanidm.foxhunt.svc.cluster.local:8443/status` (health check)
- Alert: `KanidmDown` — `probe_success{job="kanidm"} == 0` for 30s, severity critical

### Session Revocation
- RS256 tokens are validated statelessly (JWKS only) — no per-request callback to Kanidm
- **Blast radius on key compromise**: access tokens remain valid until expiry (8h interactive, 24h service)
- Mitigation: reduce access token lifetime to 1h if faster revocation is needed; refresh tokens can be revoked server-side immediately via `kanidm session` CLI
- Break-glass: rotate the RS256 signing key (`kanidm key rotate`) to invalidate ALL tokens instantly

### Upgrades
- Image pinned to `kanidm/server:1.9.2` — upgrade deliberately after reading changelogs
- Embedded DB migrations run automatically on startup
- Test OIDC flow after each upgrade
- Kanidm is pre-2.0 — expect breaking changes between minor versions

### Key Rotation
- OIDC client secrets: `kanidm system oauth2` CLI
- Service account tokens: `kanidm service-account api-token generate`
- RS256 signing keys: `kanidm key rotate`

## NetBird — Parallel Mesh (Side-by-Side with Tailscale)

**Tailscale remains the primary mesh VPN.** NetBird runs alongside it for evaluation. Both meshes connect the same devices. Traffic gradually shifts to NetBird as confidence builds. Zero breaking changes — Tailscale continues working unchanged.

### Why NetBird

- Fully open-source (AGPLv3) — entire stack self-hosted, no proprietary control plane
- Built-in SSO/MFA via OIDC (integrates with Kanidm via generic OIDC connector)
- Device posture checks (zero-trust at network layer)
- Quantum-resistant mesh (post-quantum crypto, since v0.65)
- Platform coverage: macOS, Windows, Android (8.0+), iOS (14.0+), Linux
- Built-in reverse proxy (since Feb 2026)

### Architecture

```
Your Devices
  ├── Tailscale client (primary)  ─── Tailscale cloud control ─── 100.x.x.x
  └── NetBird client (parallel)   ─── Self-hosted control     ─── 10.x.x.x (NetBird CIDR)
                                      (on platform pool)

Cluster:
  ┌─────────────────────────────────┐
  │  NetBird Server (all-in-one)    │
  │  ├─ Management API              │
  │  ├─ Signal server               │
  │  ├─ Relay (STUN/TURN)           │
  │  └─ Dashboard UI                │
  │  Port 443 (HTTPS) + 3478 (UDP) │
  │  OIDC: Kanidm (generic OIDC)   │
  └─────────────────────────────────┘
```

### Deployment

- **Image**: `netbirdio/netbird-server:0.66.0` (all-in-one: management + signal + relay + STUN)
- **Dashboard**: `netbirdio/dashboard:v2.33.0`
- **Pool**: `platform` node pool (DEV1-L)
- **PVC**: 1Gi `scw-bssd` (RWO) at `/var/lib/netbird`
- **Strategy**: Recreate (RWO PVC)
- **DNS**: `netbird.fxhnt.ai` A record → `100.90.76.85` (via existing nginx proxy)

### Resource Budget

| Component | CPU Req | CPU Limit | Mem Req | Mem Limit |
|-----------|---------|-----------|---------|-----------|
| Server | 25m | 200m | 64Mi | 256Mi |
| Dashboard | 10m | 50m | 16Mi | 64Mi |
| **Total** | **35m** | **250m** | **80Mi** | **320Mi** |

~80MB baseline — comparable to Kanidm's 50MB. Together ~130MB for the full zero-trust stack.

### Kanidm OIDC Integration

NetBird's generic OIDC connector connects to Kanidm:

```yaml
# NetBird management config
idp:
  type: generic
  clientId: netbird
  clientSecret: <from K8s secret>
  issuer: https://auth.fxhnt.ai/oauth2/openid/netbird
```

Required OIDC claims from Kanidm: `sub` (required), `email` (recommended), `name` (recommended), `groups` (for group sync).

Redirect URI: `https://netbird.fxhnt.ai/oauth2/callback/{connector-id}`

Create OIDC client `netbird` in Kanidm (same as the other 6 clients).

### Nginx Proxy Block

```nginx
# NetBird Dashboard + Management API
server {
    listen 443 ssl;
    server_name netbird.fxhnt.ai;

    ssl_certificate /etc/nginx/certs/tls.crt;
    ssl_certificate_key /etc/nginx/certs/tls.key;

    location / {
        proxy_pass http://netbird-dashboard.foxhunt.svc.cluster.local:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    # Management API
    location /api {
        proxy_pass http://netbird-server.foxhunt.svc.cluster.local:443;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }

    # Signal (gRPC)
    location /signalexchange.SignalExchange/ {
        grpc_pass grpc://netbird-server.foxhunt.svc.cluster.local:10000;
    }
}
```

**STUN/TURN**: UDP port 3478 — needs a separate NodePort or hostNetwork since nginx can't proxy UDP. Use a dedicated Service with `type: NodePort` on UDP 3478.

### Network Policy

```yaml
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: netbird
  policyTypes: [Ingress, Egress]
  ingress:
    # Nginx proxy (HTTPS dashboard + management API)
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tailscale-gitlab-proxy
      ports:
        - { protocol: TCP, port: 80 }    # dashboard
        - { protocol: TCP, port: 443 }   # management API
        - { protocol: TCP, port: 10000 } # signal (gRPC)
    # STUN/TURN from external (all peers)
    - ports:
        - { protocol: UDP, port: 3478 }
    # Peer connections from foxhunt namespace
    - from:
        - podSelector: {}
      ports:
        - { protocol: TCP, port: 443 }
  egress:
    # DNS
    - to:
        - namespaceSelector: {}
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    # Kanidm OIDC (token validation)
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: kanidm
      ports:
        - { protocol: TCP, port: 8443 }
```

### Side-by-Side Strategy

| Aspect | Tailscale (primary) | NetBird (parallel) |
|--------|--------------------|--------------------|
| Status | **Primary — no changes** | Evaluation |
| Control plane | Tailscale cloud (proprietary) | Self-hosted (platform pool) |
| IP range | 100.x.x.x (CGNAT) | 10.x.x.x (NetBird default) |
| Client install | Already on all devices | Install alongside Tailscale |
| DNS | `*.fxhnt.ai` → Tailscale IP | `netbird.fxhnt.ai` → same Tailscale IP (dashboard only) |
| Service access | All services via Tailscale IPs | Test access via NetBird IPs |
| Auth | Tailscale SSO | Kanidm OIDC |
| Breakage risk | **Zero** — nothing changes | Additive only |

### Migration Path (Future)

1. **Now**: Both clients on all devices, Tailscale primary
2. **Evaluate**: Access services through NetBird, compare latency/reliability
3. **Confidence**: Move non-critical services (Grafana, Mattermost) to NetBird-only DNS
4. **Full migration**: Redirect `*.fxhnt.ai` DNS to NetBird IPs, decommission Tailscale
5. **Never forced**: If NetBird disappoints, remove it. Tailscale never touched.

### Monitoring

- Prometheus scrape: NetBird management exposes metrics
- Alert: `NetBirdDown` — `up{job="netbird"} == 0` for 2m, severity medium (not critical — Tailscale is primary)
