foxhunt/docs/plans/2026-02-27-cockpit-nat-gateway-implementation.md

Cockpit Metrics + NAT Gateway Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Deploy a Scaleway Public Gateway (NAT) so all K8s nodes use private-only IPs, and set up Scaleway Cockpit with Grafana Alloy + DCGM Exporter for full cluster metrics (CPU, memory, disk, network, GPU).

Architecture: Two independent workstreams. Workstream 1 creates a Terraform module for the Public Gateway, attaches it to the existing VPC private network, and disables public IPs on all node pools. Workstream 2 creates a Terraform module for Cockpit credentials, then deploys K8s manifests for Grafana Alloy (via k8s-monitoring Helm chart) and DCGM Exporter, plus configures GitLab's bundled Prometheus to remote_write to Cockpit.

Tech Stack: Terraform (Scaleway provider ~2.46), Terragrunt, Kubernetes YAML, Helm values, Scaleway Cockpit (Mimir), Grafana Alloy, NVIDIA DCGM Exporter

Design doc: docs/plans/2026-02-27-cockpit-nat-gateway-design.md

---

Task 1: Public Gateway Terraform Module

Creates the NAT gateway, reserves a public IP, and attaches it to the existing private network.

Files:

• Create: infra/modules/public-gateway/main.tf
• Create: infra/modules/public-gateway/variables.tf
• Create: infra/modules/public-gateway/outputs.tf

Step 1: Create infra/modules/public-gateway/variables.tf

variable "region" {
  description = "Scaleway region"
  type        = string
}

variable "zone" {
  description = "Scaleway zone for the gateway"
  type        = string
}

variable "project_id" {
  description = "Scaleway project ID"
  type        = string
}

variable "private_network_id" {
  description = "ID of the existing VPC private network to attach the gateway to"
  type        = string
}

variable "gateway_type" {
  description = "Public Gateway instance type (VPC-GW-S, VPC-GW-M)"
  type        = string
  default     = "VPC-GW-S"
}

variable "bastion_enabled" {
  description = "Enable SSH bastion on the gateway"
  type        = bool
  default     = false
}

Step 2: Create infra/modules/public-gateway/main.tf

resource "scaleway_vpc_public_gateway_ip" "foxhunt" {
  zone = var.zone
}

resource "scaleway_vpc_public_gateway" "foxhunt" {
  name            = "foxhunt-gw"
  type            = var.gateway_type
  zone            = var.zone
  bastion_enabled = var.bastion_enabled
  ip_id           = scaleway_vpc_public_gateway_ip.foxhunt.id
}

resource "scaleway_vpc_gateway_network" "foxhunt" {
  gateway_id         = scaleway_vpc_public_gateway.foxhunt.id
  private_network_id = var.private_network_id
  enable_masquerade  = true

  ipam_config {
    push_default_route = true
  }
}

Step 3: Create infra/modules/public-gateway/outputs.tf

output "gateway_id" {
  description = "ID of the Public Gateway"
  value       = scaleway_vpc_public_gateway.foxhunt.id
}

output "gateway_ip" {
  description = "Public IP address of the gateway"
  value       = scaleway_vpc_public_gateway_ip.foxhunt.address
}

Step 4: Validate module syntax

Run: cd infra/modules/public-gateway && terraform init -backend=false && terraform validate Expected: "Success! The configuration is valid."

Step 5: Commit

git add infra/modules/public-gateway/
git commit -m "feat(infra): add public-gateway Terraform module for NAT"

---

Task 2: Export Private Network ID from Kapsule Module

The public-gateway module needs the private network ID. Add it as an output from the kapsule module.

Files:

• Modify: infra/modules/kapsule/outputs.tf

Step 1: Add private_network_id output

Append to infra/modules/kapsule/outputs.tf (after the existing gitlab_pool_id output):

output "private_network_id" {
  description = "ID of the VPC private network the cluster is attached to"
  value       = scaleway_vpc_private_network.foxhunt.id
}

Step 2: Commit

git add infra/modules/kapsule/outputs.tf
git commit -m "feat(infra): export private_network_id from kapsule module"

---

Task 3: Disable Public IPs on All Node Pools

Add public_ip_disabled to every pool in the kapsule module, controlled by a single variable.

Files:

• Modify: infra/modules/kapsule/variables.tf
• Modify: infra/modules/kapsule/main.tf
• Modify: infra/live/production/kapsule/terragrunt.hcl

Step 1: Add variable to infra/modules/kapsule/variables.tf

Append at the end of the file:

variable "public_ip_disabled" {
  description = "Disable public IPs on all node pools (requires Public Gateway for egress)"
  type        = bool
  default     = false
}

Step 2: Add public_ip_disabled to each pool in infra/modules/kapsule/main.tf

Add public_ip_disabled = var.public_ip_disabled to each of the 6 pool resources. The attribute goes after autohealing = true in each block:

For scaleway_k8s_pool.always_on (line ~37, after autohealing = true):

  public_ip_disabled = var.public_ip_disabled

For scaleway_k8s_pool.gpu_training (line ~51, after autohealing = true):

  public_ip_disabled = var.public_ip_disabled

For scaleway_k8s_pool.gpu_inference (line ~70, after autohealing = true):

  public_ip_disabled = var.public_ip_disabled

For scaleway_k8s_pool.gitlab (line ~88, after autohealing = true):

  public_ip_disabled = var.public_ip_disabled

For scaleway_k8s_pool.ci_build (line ~102, after autohealing = true):

  public_ip_disabled = var.public_ip_disabled

For scaleway_k8s_pool.dev (line ~121, after autohealing = true):

  public_ip_disabled = var.public_ip_disabled

Step 3: Set the variable in infra/live/production/kapsule/terragrunt.hcl

Add to the inputs block (after the dev_pool_max_size line):

  # NAT gateway handles egress — no public IPs on nodes
  public_ip_disabled = true

Step 4: Validate kapsule module syntax

Run: cd infra/modules/kapsule && terraform init -backend=false && terraform validate Expected: "Success! The configuration is valid."

Step 5: Commit

git add infra/modules/kapsule/variables.tf infra/modules/kapsule/main.tf infra/live/production/kapsule/terragrunt.hcl
git commit -m "feat(infra): disable public IPs on all Kapsule node pools"

---

Task 4: Public Gateway Terragrunt Live Config

Create the live production config that wires the public-gateway module to the existing private network.

Files:

• Create: infra/live/production/public-gateway/terragrunt.hcl

Step 1: Create the terragrunt config

include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../../../modules/public-gateway"
}

dependency "kapsule" {
  config_path = "../kapsule"
}

inputs = {
  private_network_id = dependency.kapsule.outputs.private_network_id
  gateway_type       = "VPC-GW-S"
}

Note: This is the first Terragrunt dependency block in the project. Terragrunt will automatically read the kapsule state to get the private_network_id output.

Step 2: Commit

git add infra/live/production/public-gateway/terragrunt.hcl
git commit -m "feat(infra): add public-gateway live config with kapsule dependency"

---

Task 5: Cockpit Terraform Module

Creates the Cockpit data source, push token, and Grafana user.

Files:

• Create: infra/modules/cockpit/main.tf
• Create: infra/modules/cockpit/variables.tf
• Create: infra/modules/cockpit/outputs.tf

Step 1: Create infra/modules/cockpit/variables.tf

variable "project_id" {
  description = "Scaleway project ID"
  type        = string
}

variable "grafana_login" {
  description = "Login for Cockpit Grafana admin user"
  type        = string
  default     = "foxhunt-admin"
}

variable "grafana_role" {
  description = "Role for the Cockpit Grafana user (editor or viewer)"
  type        = string
  default     = "editor"
}

Step 2: Create infra/modules/cockpit/main.tf

data "scaleway_cockpit" "main" {
  project_id = var.project_id
}

# Custom metrics data source for K8s data plane
resource "scaleway_cockpit_source" "k8s_metrics" {
  project_id = var.project_id
  name       = "kapsule-data-plane"
  type       = "metrics"
}

# Push token for Grafana Alloy + Prometheus remote_write
resource "scaleway_cockpit_token" "k8s_push" {
  project_id = var.project_id
  name       = "kapsule-push-token"

  scopes {
    query_metrics = true
    write_metrics = true
    query_logs    = true
    write_logs    = true
  }
}

# Grafana user for dashboard access
resource "scaleway_cockpit_grafana_user" "admin" {
  project_id = var.project_id
  login      = var.grafana_login
  role       = var.grafana_role
}

Step 3: Create infra/modules/cockpit/outputs.tf

output "metrics_push_url" {
  description = "Cockpit Mimir remote_write URL for Alloy/Prometheus"
  value       = scaleway_cockpit_source.k8s_metrics.push_url
}

output "push_token" {
  description = "Token for authenticating remote_write requests"
  value       = scaleway_cockpit_token.k8s_push.secret_key
  sensitive   = true
}

output "grafana_url" {
  description = "Cockpit Grafana dashboard URL"
  value       = data.scaleway_cockpit.main.endpoints[0].grafana_url
}

output "grafana_password" {
  description = "Generated password for the Grafana user"
  value       = scaleway_cockpit_grafana_user.admin.password
  sensitive   = true
}

Step 4: Validate module syntax

Run: cd infra/modules/cockpit && terraform init -backend=false && terraform validate Expected: "Success! The configuration is valid."

Step 5: Commit

git add infra/modules/cockpit/
git commit -m "feat(infra): add cockpit Terraform module for metrics push"

---

Task 6: Cockpit Terragrunt Live Config

Files:

• Create: infra/live/production/cockpit/terragrunt.hcl

Step 1: Create the terragrunt config

include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../../../modules/cockpit"
}

inputs = {}

Note: project_id is already provided by the root.hcl common inputs. No dependencies needed.

Step 2: Commit

git add infra/live/production/cockpit/terragrunt.hcl
git commit -m "feat(infra): add cockpit live config"

---

Task 7: Grafana Alloy (k8s-monitoring) Helm Values

The grafana/k8s-monitoring Helm chart deploys Alloy + node-exporter + kube-state-metrics. This values file configures it to push to Cockpit.

Important: The actual metrics_push_url and push_token values must be filled in after terragrunt apply on the cockpit module. Use placeholders that the operator replaces.

Files:

• Create: infra/k8s/monitoring/alloy-values.yaml

Step 1: Create the Helm values file

# Grafana k8s-monitoring Helm chart values
# Chart: grafana/k8s-monitoring
# Deploys: Alloy (DaemonSet) + node-exporter + kube-state-metrics
#
# Install:
#   helm repo add grafana https://grafana.github.io/helm-charts
#   helm install k8s-monitoring grafana/k8s-monitoring \
#     --namespace monitoring --create-namespace \
#     -f infra/k8s/monitoring/alloy-values.yaml
#
# After terragrunt apply on cockpit module, get the values:
#   cd infra/live/production/cockpit && terragrunt output -json
# Then create the K8s secret:
#   kubectl -n monitoring create secret generic cockpit-push-token \
#     --from-literal=token=<push_token from terragrunt output>

cluster:
  name: foxhunt

externalServices:
  prometheus:
    host: "COCKPIT_METRICS_PUSH_URL"
    # URL from: cd infra/live/production/cockpit && terragrunt output metrics_push_url
    # Format: https://<source-id>.metrics.cockpit.fr-par.scw.cloud/api/v1/push
    authMode: raw
    secret:
      create: false
      name: cockpit-push-token
      namespace: monitoring
    customHeaders:
      - name: X-TOKEN
        valueFrom:
          secretKeyRef:
            name: cockpit-push-token
            key: token

metrics:
  enabled: true
  cost:
    enabled: false
  node-exporter:
    enabled: true
  kube-state-metrics:
    enabled: true
  kubelet:
    enabled: true
  cadvisor:
    enabled: true
  apiserver:
    enabled: true

# Alloy DaemonSet — runs on every node
alloy-metrics:
  alloy:
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        cpu: 200m
        memory: 256Mi

# node-exporter — host metrics (CPU, memory, disk, network)
prometheus-node-exporter:
  resources:
    requests:
      cpu: 25m
      memory: 32Mi
    limits:
      cpu: 100m
      memory: 64Mi

# kube-state-metrics — K8s object state
kube-state-metrics:
  resources:
    requests:
      cpu: 25m
      memory: 64Mi
    limits:
      cpu: 100m
      memory: 128Mi
  nodeSelector:
    k8s.scaleway.com/pool-name: always-on

# Disable components we don't need
logs:
  enabled: false
traces:
  enabled: false
opencost:
  enabled: false

Step 2: Commit

git add infra/k8s/monitoring/alloy-values.yaml
git commit -m "feat(infra): add Grafana Alloy k8s-monitoring Helm values for Cockpit"

---

Task 8: DCGM Exporter for GPU Metrics

DaemonSet that runs on GPU nodes only and exposes NVIDIA GPU metrics for Alloy to scrape.

Files:

• Create: infra/k8s/monitoring/dcgm-exporter.yaml

Step 1: Create the DCGM Exporter manifest

# NVIDIA DCGM Exporter — GPU metrics for Cockpit
# Runs on all GPU node pools (gpu-training, gpu-inference, ci-build, gpu-dev)
# Alloy auto-discovers and scrapes these pods via annotations
---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    app.kubernetes.io/part-of: foxhunt
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: dcgm-exporter
    app.kubernetes.io/part-of: foxhunt
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: dcgm-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: dcgm-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9400"
        prometheus.io/path: "/metrics"
    spec:
      # Only schedule on nodes that have GPUs
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nvidia.com/gpu.present
                    operator: In
                    values:
                      - "true"
      tolerations:
        # Tolerate GPU taints so the exporter can run on GPU nodes
        - operator: Exists
          effect: NoSchedule
      containers:
        - name: dcgm-exporter
          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
          ports:
            - name: metrics
              containerPort: 9400
          env:
            - name: DCGM_EXPORTER_KUBERNETES
              value: "true"
          resources:
            requests:
              cpu: 25m
              memory: 64Mi
            limits:
              cpu: 100m
              memory: 128Mi
          securityContext:
            runAsNonRoot: false
            runAsUser: 0
          volumeMounts:
            - name: device-metrics
              mountPath: /var/lib/dcgm
      volumes:
        - name: device-metrics
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: dcgm-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: dcgm-exporter
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9400"
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: dcgm-exporter
  ports:
    - name: metrics
      port: 9400
      targetPort: 9400

Step 2: Commit

git add infra/k8s/monitoring/dcgm-exporter.yaml
git commit -m "feat(infra): add DCGM Exporter DaemonSet for GPU metrics"

---

Task 9: GitLab Prometheus Remote Write to Cockpit

Configure the bundled GitLab Prometheus to also push metrics to Cockpit, so GitLab application and CI metrics appear alongside cluster metrics.

Files:

• Modify: infra/k8s/gitlab/values.yaml

Step 1: Add remote_write config to the prometheus section

In infra/k8s/gitlab/values.yaml, locate the prometheus: section (starts at line 71). Add remoteWrite after server.resources.limits (after line 93, before gitlab-runner:):

    remoteWrite:
      - url: "COCKPIT_METRICS_PUSH_URL"
        # URL from: cd infra/live/production/cockpit && terragrunt output metrics_push_url
        headers:
          X-TOKEN: "COCKPIT_PUSH_TOKEN"
          # Token from: cd infra/live/production/cockpit && terragrunt output -raw push_token
        writeRelabelConfigs:
          - sourceLabels: [__name__]
            regex: "gitlab_.*|gitaly_.*|sidekiq_.*|ruby_.*|puma_.*"
            action: keep

The writeRelabelConfigs filter ensures only GitLab-relevant metrics are pushed (not duplicating K8s metrics that Alloy already handles).

Step 2: Commit

git add infra/k8s/gitlab/values.yaml
git commit -m "feat(infra): add Prometheus remote_write to Cockpit in GitLab Helm values"

---

Task 10: Apply Instructions Document

Create a runbook for the operator to apply all changes in the correct order.

Files:

• Create: infra/k8s/monitoring/APPLY-ORDER.md

Step 1: Create the runbook

# Cockpit + NAT Gateway — Apply Order

## Prerequisites
- `scw` CLI authenticated
- `terragrunt` installed
- `kubectl` access to foxhunt cluster
- `helm` installed

## Phase 1: Cockpit (metrics backend)

1. Apply Cockpit Terraform module:
   ```bash
   cd infra/live/production/cockpit
   terragrunt apply

2. Note the outputs:

   terragrunt output metrics_push_url
   terragrunt output -raw push_token
   terragrunt output grafana_url
   terragrunt output -raw grafana_password
   

3. Create the K8s secret for the push token:

   kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
   kubectl -n monitoring create secret generic cockpit-push-token \
     --from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
   

4. Update placeholders in infra/k8s/monitoring/alloy-values.yaml:

   • Replace COCKPIT_METRICS_PUSH_URL with the metrics_push_url output

5. Deploy DCGM Exporter:

   kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
   

6. Deploy Grafana Alloy (k8s-monitoring):

   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   helm install k8s-monitoring grafana/k8s-monitoring \
     --namespace monitoring \
     -f infra/k8s/monitoring/alloy-values.yaml
   

7. Verify metrics flow:

   • Open Cockpit Grafana (URL from step 2)
   • Login with foxhunt-admin / (password from step 2)
   • Check "Kubernetes / Compute Resources / Cluster" dashboard

Phase 2: GitLab Prometheus remote_write

1. Update placeholders in infra/k8s/gitlab/values.yaml:

   • Replace COCKPIT_METRICS_PUSH_URL and COCKPIT_PUSH_TOKEN

2. Upgrade GitLab Helm release:

   helm upgrade gitlab gitlab/gitlab \
     --namespace foxhunt \
     -f infra/k8s/gitlab/values.yaml
   

Phase 3: NAT Gateway

1. Apply Public Gateway module:

   cd infra/live/production/public-gateway
   terragrunt apply
   

   This creates the gateway and attaches it to the private network. Existing nodes still have public IPs — nothing breaks yet.

2. Apply Kapsule pool changes (disables public IPs):

   cd infra/live/production/kapsule
   terragrunt apply
   

   WARNING: This may recreate nodes. For zero-downtime:

3. Zero-downtime node migration for always-on pool:

   # Temporarily allow 2 nodes
   # Edit terragrunt.hcl: always_on max_size → 2 (need variable change)
   # Or use SCW CLI:
   scw k8s pool update <always-on-pool-id> max-size=2 size=2
   # Wait for new node (private IP) to be Ready
   kubectl get nodes -w
   # Drain old node
   kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
   # Scale back
   scw k8s pool update <always-on-pool-id> max-size=1 size=1
   

4. Repeat for gitlab pool (if not autoscaling, same process).

5. Autoscaling pools (gpu-training, gpu-inference, ci-build, gpu-dev):

   • No action needed — next scale-up automatically uses private IP.

6. Verify Tailscale still works:

   tailscale status  # should show foxhunt-kapsule and foxhunt-gitlab nodes
   curl -k https://git.fxhnt.ai  # should still resolve via Tailscale
   

**Step 2: Commit**

```bash
git add infra/k8s/monitoring/APPLY-ORDER.md
git commit -m "docs(infra): add apply-order runbook for Cockpit + NAT gateway"

---

Execution Order Summary

Task	Workstream	Description	Depends On
1	NAT	Public Gateway Terraform module	—
2	NAT	Export private_network_id from kapsule	—
3	NAT	Disable public IPs on all pools	—
4	NAT	Public Gateway Terragrunt live config	Task 1, 2
5	Metrics	Cockpit Terraform module	—
6	Metrics	Cockpit Terragrunt live config	Task 5
7	Metrics	Alloy k8s-monitoring Helm values	—
8	Metrics	DCGM Exporter DaemonSet	—
9	Metrics	GitLab Prometheus remote_write	—
10	Ops	Apply-order runbook	All above

Parallelizable: Tasks 1-3 and 5-9 are all independent and can be done in parallel. Task 4 depends on 1+2. Task 10 is last.