From 4860395dbfc17ef3eda8a96188887fb0bcd4cf16 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 27 Feb 2026 00:30:06 +0100 Subject: [PATCH] docs: add implementation plan for Cockpit metrics + NAT gateway 10 tasks across 2 workstreams (NAT + Metrics), mostly parallelizable. Includes Terraform modules, K8s manifests, Helm values, and runbook. Co-Authored-By: Claude Opus 4.6 --- ...2-27-cockpit-nat-gateway-implementation.md | 795 ++++++++++++++++++ 1 file changed, 795 insertions(+) create mode 100644 docs/plans/2026-02-27-cockpit-nat-gateway-implementation.md diff --git a/docs/plans/2026-02-27-cockpit-nat-gateway-implementation.md b/docs/plans/2026-02-27-cockpit-nat-gateway-implementation.md new file mode 100644 index 000000000..ec0189b7e --- /dev/null +++ b/docs/plans/2026-02-27-cockpit-nat-gateway-implementation.md @@ -0,0 +1,795 @@ +# Cockpit Metrics + NAT Gateway Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Deploy a Scaleway Public Gateway (NAT) so all K8s nodes use private-only IPs, and set up Scaleway Cockpit with Grafana Alloy + DCGM Exporter for full cluster metrics (CPU, memory, disk, network, GPU). + +**Architecture:** Two independent workstreams. Workstream 1 creates a Terraform module for the Public Gateway, attaches it to the existing VPC private network, and disables public IPs on all node pools. Workstream 2 creates a Terraform module for Cockpit credentials, then deploys K8s manifests for Grafana Alloy (via k8s-monitoring Helm chart) and DCGM Exporter, plus configures GitLab's bundled Prometheus to remote_write to Cockpit. + +**Tech Stack:** Terraform (Scaleway provider ~2.46), Terragrunt, Kubernetes YAML, Helm values, Scaleway Cockpit (Mimir), Grafana Alloy, NVIDIA DCGM Exporter + +**Design doc:** `docs/plans/2026-02-27-cockpit-nat-gateway-design.md` + +--- + +## Task 1: Public Gateway Terraform Module + +Creates the NAT gateway, reserves a public IP, and attaches it to the existing private network. + +**Files:** +- Create: `infra/modules/public-gateway/main.tf` +- Create: `infra/modules/public-gateway/variables.tf` +- Create: `infra/modules/public-gateway/outputs.tf` + +**Step 1: Create `infra/modules/public-gateway/variables.tf`** + +```hcl +variable "region" { + description = "Scaleway region" + type = string +} + +variable "zone" { + description = "Scaleway zone for the gateway" + type = string +} + +variable "project_id" { + description = "Scaleway project ID" + type = string +} + +variable "private_network_id" { + description = "ID of the existing VPC private network to attach the gateway to" + type = string +} + +variable "gateway_type" { + description = "Public Gateway instance type (VPC-GW-S, VPC-GW-M)" + type = string + default = "VPC-GW-S" +} + +variable "bastion_enabled" { + description = "Enable SSH bastion on the gateway" + type = bool + default = false +} +``` + +**Step 2: Create `infra/modules/public-gateway/main.tf`** + +```hcl +resource "scaleway_vpc_public_gateway_ip" "foxhunt" { + zone = var.zone +} + +resource "scaleway_vpc_public_gateway" "foxhunt" { + name = "foxhunt-gw" + type = var.gateway_type + zone = var.zone + bastion_enabled = var.bastion_enabled + ip_id = scaleway_vpc_public_gateway_ip.foxhunt.id +} + +resource "scaleway_vpc_gateway_network" "foxhunt" { + gateway_id = scaleway_vpc_public_gateway.foxhunt.id + private_network_id = var.private_network_id + enable_masquerade = true + + ipam_config { + push_default_route = true + } +} +``` + +**Step 3: Create `infra/modules/public-gateway/outputs.tf`** + +```hcl +output "gateway_id" { + description = "ID of the Public Gateway" + value = scaleway_vpc_public_gateway.foxhunt.id +} + +output "gateway_ip" { + description = "Public IP address of the gateway" + value = scaleway_vpc_public_gateway_ip.foxhunt.address +} +``` + +**Step 4: Validate module syntax** + +Run: `cd infra/modules/public-gateway && terraform init -backend=false && terraform validate` +Expected: "Success! The configuration is valid." + +**Step 5: Commit** + +```bash +git add infra/modules/public-gateway/ +git commit -m "feat(infra): add public-gateway Terraform module for NAT" +``` + +--- + +## Task 2: Export Private Network ID from Kapsule Module + +The public-gateway module needs the private network ID. Add it as an output from the kapsule module. + +**Files:** +- Modify: `infra/modules/kapsule/outputs.tf` + +**Step 1: Add private_network_id output** + +Append to `infra/modules/kapsule/outputs.tf` (after the existing `gitlab_pool_id` output): + +```hcl +output "private_network_id" { + description = "ID of the VPC private network the cluster is attached to" + value = scaleway_vpc_private_network.foxhunt.id +} +``` + +**Step 2: Commit** + +```bash +git add infra/modules/kapsule/outputs.tf +git commit -m "feat(infra): export private_network_id from kapsule module" +``` + +--- + +## Task 3: Disable Public IPs on All Node Pools + +Add `public_ip_disabled` to every pool in the kapsule module, controlled by a single variable. + +**Files:** +- Modify: `infra/modules/kapsule/variables.tf` +- Modify: `infra/modules/kapsule/main.tf` +- Modify: `infra/live/production/kapsule/terragrunt.hcl` + +**Step 1: Add variable to `infra/modules/kapsule/variables.tf`** + +Append at the end of the file: + +```hcl +variable "public_ip_disabled" { + description = "Disable public IPs on all node pools (requires Public Gateway for egress)" + type = bool + default = false +} +``` + +**Step 2: Add `public_ip_disabled` to each pool in `infra/modules/kapsule/main.tf`** + +Add `public_ip_disabled = var.public_ip_disabled` to each of the 6 pool resources. The attribute goes after `autohealing = true` in each block: + +For `scaleway_k8s_pool.always_on` (line ~37, after `autohealing = true`): +```hcl + public_ip_disabled = var.public_ip_disabled +``` + +For `scaleway_k8s_pool.gpu_training` (line ~51, after `autohealing = true`): +```hcl + public_ip_disabled = var.public_ip_disabled +``` + +For `scaleway_k8s_pool.gpu_inference` (line ~70, after `autohealing = true`): +```hcl + public_ip_disabled = var.public_ip_disabled +``` + +For `scaleway_k8s_pool.gitlab` (line ~88, after `autohealing = true`): +```hcl + public_ip_disabled = var.public_ip_disabled +``` + +For `scaleway_k8s_pool.ci_build` (line ~102, after `autohealing = true`): +```hcl + public_ip_disabled = var.public_ip_disabled +``` + +For `scaleway_k8s_pool.dev` (line ~121, after `autohealing = true`): +```hcl + public_ip_disabled = var.public_ip_disabled +``` + +**Step 3: Set the variable in `infra/live/production/kapsule/terragrunt.hcl`** + +Add to the `inputs` block (after the `dev_pool_max_size` line): + +```hcl + # NAT gateway handles egress — no public IPs on nodes + public_ip_disabled = true +``` + +**Step 4: Validate kapsule module syntax** + +Run: `cd infra/modules/kapsule && terraform init -backend=false && terraform validate` +Expected: "Success! The configuration is valid." + +**Step 5: Commit** + +```bash +git add infra/modules/kapsule/variables.tf infra/modules/kapsule/main.tf infra/live/production/kapsule/terragrunt.hcl +git commit -m "feat(infra): disable public IPs on all Kapsule node pools" +``` + +--- + +## Task 4: Public Gateway Terragrunt Live Config + +Create the live production config that wires the public-gateway module to the existing private network. + +**Files:** +- Create: `infra/live/production/public-gateway/terragrunt.hcl` + +**Step 1: Create the terragrunt config** + +```hcl +include "root" { + path = find_in_parent_folders("root.hcl") +} + +terraform { + source = "../../../modules/public-gateway" +} + +dependency "kapsule" { + config_path = "../kapsule" +} + +inputs = { + private_network_id = dependency.kapsule.outputs.private_network_id + gateway_type = "VPC-GW-S" +} +``` + +Note: This is the first Terragrunt `dependency` block in the project. Terragrunt will automatically read the kapsule state to get the private_network_id output. + +**Step 2: Commit** + +```bash +git add infra/live/production/public-gateway/terragrunt.hcl +git commit -m "feat(infra): add public-gateway live config with kapsule dependency" +``` + +--- + +## Task 5: Cockpit Terraform Module + +Creates the Cockpit data source, push token, and Grafana user. + +**Files:** +- Create: `infra/modules/cockpit/main.tf` +- Create: `infra/modules/cockpit/variables.tf` +- Create: `infra/modules/cockpit/outputs.tf` + +**Step 1: Create `infra/modules/cockpit/variables.tf`** + +```hcl +variable "project_id" { + description = "Scaleway project ID" + type = string +} + +variable "grafana_login" { + description = "Login for Cockpit Grafana admin user" + type = string + default = "foxhunt-admin" +} + +variable "grafana_role" { + description = "Role for the Cockpit Grafana user (editor or viewer)" + type = string + default = "editor" +} +``` + +**Step 2: Create `infra/modules/cockpit/main.tf`** + +```hcl +data "scaleway_cockpit" "main" { + project_id = var.project_id +} + +# Custom metrics data source for K8s data plane +resource "scaleway_cockpit_source" "k8s_metrics" { + project_id = var.project_id + name = "kapsule-data-plane" + type = "metrics" +} + +# Push token for Grafana Alloy + Prometheus remote_write +resource "scaleway_cockpit_token" "k8s_push" { + project_id = var.project_id + name = "kapsule-push-token" + + scopes { + query_metrics = true + write_metrics = true + query_logs = true + write_logs = true + } +} + +# Grafana user for dashboard access +resource "scaleway_cockpit_grafana_user" "admin" { + project_id = var.project_id + login = var.grafana_login + role = var.grafana_role +} +``` + +**Step 3: Create `infra/modules/cockpit/outputs.tf`** + +```hcl +output "metrics_push_url" { + description = "Cockpit Mimir remote_write URL for Alloy/Prometheus" + value = scaleway_cockpit_source.k8s_metrics.push_url +} + +output "push_token" { + description = "Token for authenticating remote_write requests" + value = scaleway_cockpit_token.k8s_push.secret_key + sensitive = true +} + +output "grafana_url" { + description = "Cockpit Grafana dashboard URL" + value = data.scaleway_cockpit.main.endpoints[0].grafana_url +} + +output "grafana_password" { + description = "Generated password for the Grafana user" + value = scaleway_cockpit_grafana_user.admin.password + sensitive = true +} +``` + +**Step 4: Validate module syntax** + +Run: `cd infra/modules/cockpit && terraform init -backend=false && terraform validate` +Expected: "Success! The configuration is valid." + +**Step 5: Commit** + +```bash +git add infra/modules/cockpit/ +git commit -m "feat(infra): add cockpit Terraform module for metrics push" +``` + +--- + +## Task 6: Cockpit Terragrunt Live Config + +**Files:** +- Create: `infra/live/production/cockpit/terragrunt.hcl` + +**Step 1: Create the terragrunt config** + +```hcl +include "root" { + path = find_in_parent_folders("root.hcl") +} + +terraform { + source = "../../../modules/cockpit" +} + +inputs = {} +``` + +Note: `project_id` is already provided by the root.hcl common inputs. No dependencies needed. + +**Step 2: Commit** + +```bash +git add infra/live/production/cockpit/terragrunt.hcl +git commit -m "feat(infra): add cockpit live config" +``` + +--- + +## Task 7: Grafana Alloy (k8s-monitoring) Helm Values + +The `grafana/k8s-monitoring` Helm chart deploys Alloy + node-exporter + kube-state-metrics. This values file configures it to push to Cockpit. + +**Important:** The actual `metrics_push_url` and `push_token` values must be filled in after `terragrunt apply` on the cockpit module. Use placeholders that the operator replaces. + +**Files:** +- Create: `infra/k8s/monitoring/alloy-values.yaml` + +**Step 1: Create the Helm values file** + +```yaml +# Grafana k8s-monitoring Helm chart values +# Chart: grafana/k8s-monitoring +# Deploys: Alloy (DaemonSet) + node-exporter + kube-state-metrics +# +# Install: +# helm repo add grafana https://grafana.github.io/helm-charts +# helm install k8s-monitoring grafana/k8s-monitoring \ +# --namespace monitoring --create-namespace \ +# -f infra/k8s/monitoring/alloy-values.yaml +# +# After terragrunt apply on cockpit module, get the values: +# cd infra/live/production/cockpit && terragrunt output -json +# Then create the K8s secret: +# kubectl -n monitoring create secret generic cockpit-push-token \ +# --from-literal=token= + +cluster: + name: foxhunt + +externalServices: + prometheus: + host: "COCKPIT_METRICS_PUSH_URL" + # URL from: cd infra/live/production/cockpit && terragrunt output metrics_push_url + # Format: https://.metrics.cockpit.fr-par.scw.cloud/api/v1/push + authMode: raw + secret: + create: false + name: cockpit-push-token + namespace: monitoring + customHeaders: + - name: X-TOKEN + valueFrom: + secretKeyRef: + name: cockpit-push-token + key: token + +metrics: + enabled: true + cost: + enabled: false + node-exporter: + enabled: true + kube-state-metrics: + enabled: true + kubelet: + enabled: true + cadvisor: + enabled: true + apiserver: + enabled: true + +# Alloy DaemonSet — runs on every node +alloy-metrics: + alloy: + resources: + requests: + cpu: 50m + memory: 128Mi + limits: + cpu: 200m + memory: 256Mi + +# node-exporter — host metrics (CPU, memory, disk, network) +prometheus-node-exporter: + resources: + requests: + cpu: 25m + memory: 32Mi + limits: + cpu: 100m + memory: 64Mi + +# kube-state-metrics — K8s object state +kube-state-metrics: + resources: + requests: + cpu: 25m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + nodeSelector: + k8s.scaleway.com/pool-name: always-on + +# Disable components we don't need +logs: + enabled: false +traces: + enabled: false +opencost: + enabled: false +``` + +**Step 2: Commit** + +```bash +git add infra/k8s/monitoring/alloy-values.yaml +git commit -m "feat(infra): add Grafana Alloy k8s-monitoring Helm values for Cockpit" +``` + +--- + +## Task 8: DCGM Exporter for GPU Metrics + +DaemonSet that runs on GPU nodes only and exposes NVIDIA GPU metrics for Alloy to scrape. + +**Files:** +- Create: `infra/k8s/monitoring/dcgm-exporter.yaml` + +**Step 1: Create the DCGM Exporter manifest** + +```yaml +# NVIDIA DCGM Exporter — GPU metrics for Cockpit +# Runs on all GPU node pools (gpu-training, gpu-inference, ci-build, gpu-dev) +# Alloy auto-discovers and scrapes these pods via annotations +--- +apiVersion: v1 +kind: Namespace +metadata: + name: monitoring + labels: + app.kubernetes.io/part-of: foxhunt +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: dcgm-exporter + namespace: monitoring + labels: + app.kubernetes.io/name: dcgm-exporter + app.kubernetes.io/part-of: foxhunt +spec: + selector: + matchLabels: + app.kubernetes.io/name: dcgm-exporter + template: + metadata: + labels: + app.kubernetes.io/name: dcgm-exporter + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "9400" + prometheus.io/path: "/metrics" + spec: + # Only schedule on nodes that have GPUs + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: nvidia.com/gpu.present + operator: In + values: + - "true" + tolerations: + # Tolerate GPU taints so the exporter can run on GPU nodes + - operator: Exists + effect: NoSchedule + containers: + - name: dcgm-exporter + image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04 + ports: + - name: metrics + containerPort: 9400 + env: + - name: DCGM_EXPORTER_KUBERNETES + value: "true" + resources: + requests: + cpu: 25m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + securityContext: + runAsNonRoot: false + runAsUser: 0 + volumeMounts: + - name: device-metrics + mountPath: /var/lib/dcgm + volumes: + - name: device-metrics + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: dcgm-exporter + namespace: monitoring + labels: + app.kubernetes.io/name: dcgm-exporter + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "9400" +spec: + type: ClusterIP + selector: + app.kubernetes.io/name: dcgm-exporter + ports: + - name: metrics + port: 9400 + targetPort: 9400 +``` + +**Step 2: Commit** + +```bash +git add infra/k8s/monitoring/dcgm-exporter.yaml +git commit -m "feat(infra): add DCGM Exporter DaemonSet for GPU metrics" +``` + +--- + +## Task 9: GitLab Prometheus Remote Write to Cockpit + +Configure the bundled GitLab Prometheus to also push metrics to Cockpit, so GitLab application and CI metrics appear alongside cluster metrics. + +**Files:** +- Modify: `infra/k8s/gitlab/values.yaml` + +**Step 1: Add remote_write config to the prometheus section** + +In `infra/k8s/gitlab/values.yaml`, locate the `prometheus:` section (starts at line 71). Add `remoteWrite` after `server.resources.limits` (after line 93, before `gitlab-runner:`): + +```yaml + remoteWrite: + - url: "COCKPIT_METRICS_PUSH_URL" + # URL from: cd infra/live/production/cockpit && terragrunt output metrics_push_url + headers: + X-TOKEN: "COCKPIT_PUSH_TOKEN" + # Token from: cd infra/live/production/cockpit && terragrunt output -raw push_token + writeRelabelConfigs: + - sourceLabels: [__name__] + regex: "gitlab_.*|gitaly_.*|sidekiq_.*|ruby_.*|puma_.*" + action: keep +``` + +The `writeRelabelConfigs` filter ensures only GitLab-relevant metrics are pushed (not duplicating K8s metrics that Alloy already handles). + +**Step 2: Commit** + +```bash +git add infra/k8s/gitlab/values.yaml +git commit -m "feat(infra): add Prometheus remote_write to Cockpit in GitLab Helm values" +``` + +--- + +## Task 10: Apply Instructions Document + +Create a runbook for the operator to apply all changes in the correct order. + +**Files:** +- Create: `infra/k8s/monitoring/APPLY-ORDER.md` + +**Step 1: Create the runbook** + +```markdown +# Cockpit + NAT Gateway — Apply Order + +## Prerequisites +- `scw` CLI authenticated +- `terragrunt` installed +- `kubectl` access to foxhunt cluster +- `helm` installed + +## Phase 1: Cockpit (metrics backend) + +1. Apply Cockpit Terraform module: + ```bash + cd infra/live/production/cockpit + terragrunt apply + ``` + +2. Note the outputs: + ```bash + terragrunt output metrics_push_url + terragrunt output -raw push_token + terragrunt output grafana_url + terragrunt output -raw grafana_password + ``` + +3. Create the K8s secret for the push token: + ```bash + kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f - + kubectl -n monitoring create secret generic cockpit-push-token \ + --from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token) + ``` + +4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`: + - Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output + +5. Deploy DCGM Exporter: + ```bash + kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml + ``` + +6. Deploy Grafana Alloy (k8s-monitoring): + ```bash + helm repo add grafana https://grafana.github.io/helm-charts + helm repo update + helm install k8s-monitoring grafana/k8s-monitoring \ + --namespace monitoring \ + -f infra/k8s/monitoring/alloy-values.yaml + ``` + +7. Verify metrics flow: + - Open Cockpit Grafana (URL from step 2) + - Login with foxhunt-admin / (password from step 2) + - Check "Kubernetes / Compute Resources / Cluster" dashboard + +## Phase 2: GitLab Prometheus remote_write + +1. Update placeholders in `infra/k8s/gitlab/values.yaml`: + - Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN` + +2. Upgrade GitLab Helm release: + ```bash + helm upgrade gitlab gitlab/gitlab \ + --namespace foxhunt \ + -f infra/k8s/gitlab/values.yaml + ``` + +## Phase 3: NAT Gateway + +1. Apply Public Gateway module: + ```bash + cd infra/live/production/public-gateway + terragrunt apply + ``` + This creates the gateway and attaches it to the private network. + Existing nodes still have public IPs — nothing breaks yet. + +2. Apply Kapsule pool changes (disables public IPs): + ```bash + cd infra/live/production/kapsule + terragrunt apply + ``` + **WARNING**: This may recreate nodes. For zero-downtime: + +3. Zero-downtime node migration for always-on pool: + ```bash + # Temporarily allow 2 nodes + # Edit terragrunt.hcl: always_on max_size → 2 (need variable change) + # Or use SCW CLI: + scw k8s pool update max-size=2 size=2 + # Wait for new node (private IP) to be Ready + kubectl get nodes -w + # Drain old node + kubectl drain --ignore-daemonsets --delete-emptydir-data + # Scale back + scw k8s pool update max-size=1 size=1 + ``` + +4. Repeat for gitlab pool (if not autoscaling, same process). + +5. Autoscaling pools (gpu-training, gpu-inference, ci-build, gpu-dev): + - No action needed — next scale-up automatically uses private IP. + +6. Verify Tailscale still works: + ```bash + tailscale status # should show foxhunt-kapsule and foxhunt-gitlab nodes + curl -k https://git.fxhnt.ai # should still resolve via Tailscale + ``` +``` + +**Step 2: Commit** + +```bash +git add infra/k8s/monitoring/APPLY-ORDER.md +git commit -m "docs(infra): add apply-order runbook for Cockpit + NAT gateway" +``` + +--- + +## Execution Order Summary + +| Task | Workstream | Description | Depends On | +|------|-----------|-------------|------------| +| 1 | NAT | Public Gateway Terraform module | — | +| 2 | NAT | Export private_network_id from kapsule | — | +| 3 | NAT | Disable public IPs on all pools | — | +| 4 | NAT | Public Gateway Terragrunt live config | Task 1, 2 | +| 5 | Metrics | Cockpit Terraform module | — | +| 6 | Metrics | Cockpit Terragrunt live config | Task 5 | +| 7 | Metrics | Alloy k8s-monitoring Helm values | — | +| 8 | Metrics | DCGM Exporter DaemonSet | — | +| 9 | Metrics | GitLab Prometheus remote_write | — | +| 10 | Ops | Apply-order runbook | All above | + +**Parallelizable:** Tasks 1-3 and 5-9 are all independent and can be done in parallel. Task 4 depends on 1+2. Task 10 is last.