diff --git a/docs/plans/2026-02-27-cockpit-nat-gateway-design.md b/docs/plans/2026-02-27-cockpit-nat-gateway-design.md new file mode 100644 index 000000000..436a2f1ff --- /dev/null +++ b/docs/plans/2026-02-27-cockpit-nat-gateway-design.md @@ -0,0 +1,172 @@ +# K8s Metrics (Cockpit) + NAT Gateway Design + +**Date:** 2026-02-27 +**Status:** Approved + +## Problem + +1. **No metrics visibility** — the bundled GitLab Prometheus only scrapes GitLab internals. No node CPU/memory, no GPU metrics, no pod resource usage in any dashboard. +2. **All nodes have public IPs** — unnecessary attack surface. GPU/CI/dev pools don't need direct internet exposure. + +## Solution + +Two workstreams: Scaleway Cockpit for metrics, Public Gateway for NAT. + +--- + +## Workstream 1: NAT Gateway + +### Architecture + +``` +Internet + | +[Public Gateway VPC-GW-S] ── public IP (flexible) + | +[foxhunt-pn private network] + | +[Kapsule nodes — all private-only] + | +[Tailscale subnet router] ── NAT traversal (DERP relays) +``` + +### Terraform Changes + +**New module: `infra/modules/public-gateway/`** + +Resources: +- `scaleway_vpc_public_gateway_ip` — flexible public IP for the gateway +- `scaleway_vpc_public_gateway` — gateway instance (VPC-GW-S type) +- `scaleway_vpc_gateway_network` — attach to `foxhunt-pn` with masquerade + push_default_route + +**New live config: `infra/live/production/public-gateway/terragrunt.hcl`** + +Inputs: +- `private_network_id` — from kapsule module output (needs new output) +- `gateway_type` — "VPC-GW-S" + +**Modify: `infra/modules/kapsule/main.tf`** + +- Add `public_ip_disabled = true` to all 6 pool resources +- Add new output: `private_network_id` (for public-gateway module dependency) + +### Rollout Strategy (Zero-Downtime) + +1. Apply public-gateway module first (creates gateway, attaches to network) +2. Apply kapsule with `public_ip_disabled = true` on all pools +3. For always-on pool: temporarily set `max_size = 2` → new node joins with private IP → drain old node → set `max_size = 1` +4. For gitlab pool: same approach (temporarily `max_size = 2`, drain old, scale back) +5. Autoscaling pools (gpu-training, gpu-inference, ci-build, gpu-dev): next scale-up automatically gets private IP + +### Tailscale Impact + +None. Tailscale is designed for NAT traversal. The subnet router and gitlab proxy will continue working via DERP relays. No config changes needed. + +### Cost + +| Item | Monthly | +|------|---------| +| VPC-GW-S | ~EUR 14.50 | +| Flexible IP | ~EUR 3.00 | +| **Subtotal** | **~EUR 17.50** | + +--- + +## Workstream 2: Metrics (Scaleway Cockpit) + +### Architecture + +``` +[Grafana Alloy DaemonSet] ──remote_write──> [Cockpit Mimir] + scrapes: | + - node-exporter (CPU, mem, disk, net) v + - kube-state-metrics (pod states) [Cockpit Grafana] + - kubelet/cAdvisor (container metrics) (managed, pre-built dashboards) + - DCGM Exporter (GPU util, VRAM, temp) + +[GitLab Prometheus] ──remote_write──> [Cockpit Mimir] + (GitLab/CI metrics) +``` + +### What Gets Deployed + +#### 1. Cockpit Terraform Resources (`infra/modules/cockpit/`) + +- `data "scaleway_cockpit"` — get Cockpit endpoints +- `scaleway_cockpit_source` — custom metrics data source for K8s data plane +- `scaleway_cockpit_token` — push token for Alloy + Prometheus remote_write +- `scaleway_cockpit_grafana_user` — admin user for Cockpit Grafana + +#### 2. Grafana Alloy via k8s-monitoring Helm (`infra/k8s/monitoring/alloy-values.yaml`) + +The `grafana/k8s-monitoring` Helm chart deploys: +- **Grafana Alloy** — DaemonSet, one pod per node, scrapes all targets and pushes to Cockpit +- **node-exporter** — DaemonSet for host-level CPU/memory/disk/network metrics +- **kube-state-metrics** — Deployment for K8s object state (pod status, deployments, etc.) + +Configured with Cockpit's remote_write endpoint and push token via `X-TOKEN` header. + +#### 3. DCGM Exporter (`infra/k8s/monitoring/dcgm-exporter.yaml`) + +- DaemonSet with `nodeSelector` for GPU pools (gpu-training, gpu-inference, ci-build, gpu-dev) +- Exposes GPU metrics: utilization, memory used/free, temperature, power draw, PCIe throughput +- Alloy scrapes the DCGM Exporter pods automatically + +#### 4. GitLab Prometheus Remote Write + +Modify `infra/k8s/gitlab/values.yaml` to add `remote_write` config to the bundled Prometheus: +- Pushes GitLab application metrics + CI job metrics to Cockpit +- Keeps local scraping for GitLab internal use + +### Metrics Collected + +| Source | Metrics | Nodes | +|--------|---------|-------| +| node-exporter | CPU, memory, disk I/O, network, filesystem | All 6 pools | +| kube-state-metrics | Pod states, deployment status, resource requests/limits | Cluster-wide | +| kubelet/cAdvisor | Container CPU/memory/network per pod | All 6 pools | +| DCGM Exporter | GPU utilization, VRAM, temperature, power, PCIe | GPU pools only | +| GitLab Prometheus | GitLab Rails, Sidekiq, Gitaly, CI job metrics | GitLab pool | + +### Access + +- **Cockpit Grafana**: `cockpit.scaleway.com` → pre-built K8s dashboards +- **Existing Grafana** (`grafana.fxhnt.ai`): can add Cockpit as Prometheus-compatible datasource later if desired + +### Cost + +| Item | Monthly | +|------|---------| +| Custom metrics (~6 nodes, moderate cardinality) | ~EUR 5-15 | +| **Subtotal** | **~EUR 5-15** | + +--- + +## Total Cost Impact + +| Component | Monthly | +|-----------|---------| +| Public Gateway + IP | ~EUR 17.50 | +| Cockpit custom metrics | ~EUR 5-15 | +| **Total** | **~EUR 22-33** | + +## File Changes Summary + +### New Files +- `infra/modules/public-gateway/main.tf` — gateway + IP + network attachment +- `infra/modules/public-gateway/variables.tf` +- `infra/modules/public-gateway/outputs.tf` +- `infra/live/production/public-gateway/terragrunt.hcl` +- `infra/modules/cockpit/main.tf` — Cockpit source + token + Grafana user +- `infra/modules/cockpit/variables.tf` +- `infra/modules/cockpit/outputs.tf` +- `infra/live/production/cockpit/terragrunt.hcl` +- `infra/k8s/monitoring/alloy-values.yaml` — k8s-monitoring Helm values +- `infra/k8s/monitoring/dcgm-exporter.yaml` — DCGM DaemonSet + +### Modified Files +- `infra/modules/kapsule/main.tf` — add `public_ip_disabled`, output `private_network_id` +- `infra/modules/kapsule/variables.tf` — new variable +- `infra/modules/kapsule/outputs.tf` — new output +- `infra/live/production/kapsule/terragrunt.hcl` — pass new variable +- `infra/k8s/gitlab/values.yaml` — add Prometheus remote_write to Cockpit