# Cockpit Metrics + NAT Gateway Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Deploy a Scaleway Public Gateway (NAT) so all K8s nodes use private-only IPs, and set up Scaleway Cockpit with Grafana Alloy + DCGM Exporter for full cluster metrics (CPU, memory, disk, network, GPU).

**Architecture:** Two independent workstreams. Workstream 1 creates a Terraform module for the Public Gateway, attaches it to the existing VPC private network, and disables public IPs on all node pools. Workstream 2 creates a Terraform module for Cockpit credentials, then deploys K8s manifests for Grafana Alloy (via k8s-monitoring Helm chart) and DCGM Exporter, plus configures GitLab's bundled Prometheus to remote_write to Cockpit.

**Tech Stack:** Terraform (Scaleway provider ~2.46), Terragrunt, Kubernetes YAML, Helm values, Scaleway Cockpit (Mimir), Grafana Alloy, NVIDIA DCGM Exporter

**Design doc:** `docs/plans/2026-02-27-cockpit-nat-gateway-design.md`

---

## Task 1: Public Gateway Terraform Module

Creates the NAT gateway, reserves a public IP, and attaches it to the existing private network.

**Files:**
- Create: `infra/modules/public-gateway/main.tf`
- Create: `infra/modules/public-gateway/variables.tf`
- Create: `infra/modules/public-gateway/outputs.tf`

**Step 1: Create `infra/modules/public-gateway/variables.tf`**

```hcl
variable "region" {
  description = "Scaleway region"
  type        = string
}

variable "zone" {
  description = "Scaleway zone for the gateway"
  type        = string
}

variable "project_id" {
  description = "Scaleway project ID"
  type        = string
}

variable "private_network_id" {
  description = "ID of the existing VPC private network to attach the gateway to"
  type        = string
}

variable "gateway_type" {
  description = "Public Gateway instance type (VPC-GW-S, VPC-GW-M)"
  type        = string
  default     = "VPC-GW-S"
}

variable "bastion_enabled" {
  description = "Enable SSH bastion on the gateway"
  type        = bool
  default     = false
}
```

**Step 2: Create `infra/modules/public-gateway/main.tf`**

```hcl
resource "scaleway_vpc_public_gateway_ip" "foxhunt" {
  zone = var.zone
}

resource "scaleway_vpc_public_gateway" "foxhunt" {
  name            = "foxhunt-gw"
  type            = var.gateway_type
  zone            = var.zone
  bastion_enabled = var.bastion_enabled
  ip_id           = scaleway_vpc_public_gateway_ip.foxhunt.id
}

resource "scaleway_vpc_gateway_network" "foxhunt" {
  gateway_id         = scaleway_vpc_public_gateway.foxhunt.id
  private_network_id = var.private_network_id
  enable_masquerade  = true

  ipam_config {
    push_default_route = true
  }
}
```

**Step 3: Create `infra/modules/public-gateway/outputs.tf`**

```hcl
output "gateway_id" {
  description = "ID of the Public Gateway"
  value       = scaleway_vpc_public_gateway.foxhunt.id
}

output "gateway_ip" {
  description = "Public IP address of the gateway"
  value       = scaleway_vpc_public_gateway_ip.foxhunt.address
}
```

**Step 4: Validate module syntax**

Run: `cd infra/modules/public-gateway && terraform init -backend=false && terraform validate`
Expected: "Success! The configuration is valid."

**Step 5: Commit**

```bash
git add infra/modules/public-gateway/
git commit -m "feat(infra): add public-gateway Terraform module for NAT"
```

---

## Task 2: Export Private Network ID from Kapsule Module

The public-gateway module needs the private network ID. Add it as an output from the kapsule module.

**Files:**
- Modify: `infra/modules/kapsule/outputs.tf`

**Step 1: Add private_network_id output**

Append to `infra/modules/kapsule/outputs.tf` (after the existing `gitlab_pool_id` output):

```hcl
output "private_network_id" {
  description = "ID of the VPC private network the cluster is attached to"
  value       = scaleway_vpc_private_network.foxhunt.id
}
```

**Step 2: Commit**

```bash
git add infra/modules/kapsule/outputs.tf
git commit -m "feat(infra): export private_network_id from kapsule module"
```

---

## Task 3: Disable Public IPs on All Node Pools

Add `public_ip_disabled` to every pool in the kapsule module, controlled by a single variable.

**Files:**
- Modify: `infra/modules/kapsule/variables.tf`
- Modify: `infra/modules/kapsule/main.tf`
- Modify: `infra/live/production/kapsule/terragrunt.hcl`

**Step 1: Add variable to `infra/modules/kapsule/variables.tf`**

Append at the end of the file:

```hcl
variable "public_ip_disabled" {
  description = "Disable public IPs on all node pools (requires Public Gateway for egress)"
  type        = bool
  default     = false
}
```

**Step 2: Add `public_ip_disabled` to each pool in `infra/modules/kapsule/main.tf`**

Add `public_ip_disabled = var.public_ip_disabled` to each of the 6 pool resources. The attribute goes after `autohealing = true` in each block:

For `scaleway_k8s_pool.always_on` (line ~37, after `autohealing = true`):
```hcl
  public_ip_disabled = var.public_ip_disabled
```

For `scaleway_k8s_pool.gpu_training` (line ~51, after `autohealing = true`):
```hcl
  public_ip_disabled = var.public_ip_disabled
```

For `scaleway_k8s_pool.gpu_inference` (line ~70, after `autohealing = true`):
```hcl
  public_ip_disabled = var.public_ip_disabled
```

For `scaleway_k8s_pool.gitlab` (line ~88, after `autohealing = true`):
```hcl
  public_ip_disabled = var.public_ip_disabled
```

For `scaleway_k8s_pool.ci_build` (line ~102, after `autohealing = true`):
```hcl
  public_ip_disabled = var.public_ip_disabled
```

For `scaleway_k8s_pool.dev` (line ~121, after `autohealing = true`):
```hcl
  public_ip_disabled = var.public_ip_disabled
```

**Step 3: Set the variable in `infra/live/production/kapsule/terragrunt.hcl`**

Add to the `inputs` block (after the `dev_pool_max_size` line):

```hcl
  # NAT gateway handles egress — no public IPs on nodes
  public_ip_disabled = true
```

**Step 4: Validate kapsule module syntax**

Run: `cd infra/modules/kapsule && terraform init -backend=false && terraform validate`
Expected: "Success! The configuration is valid."

**Step 5: Commit**

```bash
git add infra/modules/kapsule/variables.tf infra/modules/kapsule/main.tf infra/live/production/kapsule/terragrunt.hcl
git commit -m "feat(infra): disable public IPs on all Kapsule node pools"
```

---

## Task 4: Public Gateway Terragrunt Live Config

Create the live production config that wires the public-gateway module to the existing private network.

**Files:**
- Create: `infra/live/production/public-gateway/terragrunt.hcl`

**Step 1: Create the terragrunt config**

```hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../../../modules/public-gateway"
}

dependency "kapsule" {
  config_path = "../kapsule"
}

inputs = {
  private_network_id = dependency.kapsule.outputs.private_network_id
  gateway_type       = "VPC-GW-S"
}
```

Note: This is the first Terragrunt `dependency` block in the project. Terragrunt will automatically read the kapsule state to get the private_network_id output.

**Step 2: Commit**

```bash
git add infra/live/production/public-gateway/terragrunt.hcl
git commit -m "feat(infra): add public-gateway live config with kapsule dependency"
```

---

## Task 5: Cockpit Terraform Module

Creates the Cockpit data source, push token, and Grafana user.

**Files:**
- Create: `infra/modules/cockpit/main.tf`
- Create: `infra/modules/cockpit/variables.tf`
- Create: `infra/modules/cockpit/outputs.tf`

**Step 1: Create `infra/modules/cockpit/variables.tf`**

```hcl
variable "project_id" {
  description = "Scaleway project ID"
  type        = string
}

variable "grafana_login" {
  description = "Login for Cockpit Grafana admin user"
  type        = string
  default     = "foxhunt-admin"
}

variable "grafana_role" {
  description = "Role for the Cockpit Grafana user (editor or viewer)"
  type        = string
  default     = "editor"
}
```

**Step 2: Create `infra/modules/cockpit/main.tf`**

```hcl
data "scaleway_cockpit" "main" {
  project_id = var.project_id
}

# Custom metrics data source for K8s data plane
resource "scaleway_cockpit_source" "k8s_metrics" {
  project_id = var.project_id
  name       = "kapsule-data-plane"
  type       = "metrics"
}

# Push token for Grafana Alloy + Prometheus remote_write
resource "scaleway_cockpit_token" "k8s_push" {
  project_id = var.project_id
  name       = "kapsule-push-token"

  scopes {
    query_metrics = true
    write_metrics = true
    query_logs    = true
    write_logs    = true
  }
}

# Grafana user for dashboard access
resource "scaleway_cockpit_grafana_user" "admin" {
  project_id = var.project_id
  login      = var.grafana_login
  role       = var.grafana_role
}
```

**Step 3: Create `infra/modules/cockpit/outputs.tf`**

```hcl
output "metrics_push_url" {
  description = "Cockpit Mimir remote_write URL for Alloy/Prometheus"
  value       = scaleway_cockpit_source.k8s_metrics.push_url
}

output "push_token" {
  description = "Token for authenticating remote_write requests"
  value       = scaleway_cockpit_token.k8s_push.secret_key
  sensitive   = true
}

output "grafana_url" {
  description = "Cockpit Grafana dashboard URL"
  value       = data.scaleway_cockpit.main.endpoints[0].grafana_url
}

output "grafana_password" {
  description = "Generated password for the Grafana user"
  value       = scaleway_cockpit_grafana_user.admin.password
  sensitive   = true
}
```

**Step 4: Validate module syntax**

Run: `cd infra/modules/cockpit && terraform init -backend=false && terraform validate`
Expected: "Success! The configuration is valid."

**Step 5: Commit**

```bash
git add infra/modules/cockpit/
git commit -m "feat(infra): add cockpit Terraform module for metrics push"
```

---

## Task 6: Cockpit Terragrunt Live Config

**Files:**
- Create: `infra/live/production/cockpit/terragrunt.hcl`

**Step 1: Create the terragrunt config**

```hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../../../modules/cockpit"
}

inputs = {}
```

Note: `project_id` is already provided by the root.hcl common inputs. No dependencies needed.

**Step 2: Commit**

```bash
git add infra/live/production/cockpit/terragrunt.hcl
git commit -m "feat(infra): add cockpit live config"
```

---

## Task 7: Grafana Alloy (k8s-monitoring) Helm Values

The `grafana/k8s-monitoring` Helm chart deploys Alloy + node-exporter + kube-state-metrics. This values file configures it to push to Cockpit.

**Important:** The actual `metrics_push_url` and `push_token` values must be filled in after `terragrunt apply` on the cockpit module. Use placeholders that the operator replaces.

**Files:**
- Create: `infra/k8s/monitoring/alloy-values.yaml`

**Step 1: Create the Helm values file**

```yaml
# Grafana k8s-monitoring Helm chart values
# Chart: grafana/k8s-monitoring
# Deploys: Alloy (DaemonSet) + node-exporter + kube-state-metrics
#
# Install:
#   helm repo add grafana https://grafana.github.io/helm-charts
#   helm install k8s-monitoring grafana/k8s-monitoring \
#     --namespace monitoring --create-namespace \
#     -f infra/k8s/monitoring/alloy-values.yaml
#
# After terragrunt apply on cockpit module, get the values:
#   cd infra/live/production/cockpit && terragrunt output -json
# Then create the K8s secret:
#   kubectl -n monitoring create secret generic cockpit-push-token \
#     --from-literal=token=<push_token from terragrunt output>

cluster:
  name: foxhunt

externalServices:
  prometheus:
    host: "COCKPIT_METRICS_PUSH_URL"
    # URL from: cd infra/live/production/cockpit && terragrunt output metrics_push_url
    # Format: https://<source-id>.metrics.cockpit.fr-par.scw.cloud/api/v1/push
    authMode: raw
    secret:
      create: false
      name: cockpit-push-token
      namespace: monitoring
    customHeaders:
      - name: X-TOKEN
        valueFrom:
          secretKeyRef:
            name: cockpit-push-token
            key: token

metrics:
  enabled: true
  cost:
    enabled: false
  node-exporter:
    enabled: true
  kube-state-metrics:
    enabled: true
  kubelet:
    enabled: true
  cadvisor:
    enabled: true
  apiserver:
    enabled: true

# Alloy DaemonSet — runs on every node
alloy-metrics:
  alloy:
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        cpu: 200m
        memory: 256Mi

# node-exporter — host metrics (CPU, memory, disk, network)
prometheus-node-exporter:
  resources:
    requests:
      cpu: 25m
      memory: 32Mi
    limits:
      cpu: 100m
      memory: 64Mi

# kube-state-metrics — K8s object state
kube-state-metrics:
  resources:
    requests:
      cpu: 25m
      memory: 64Mi
    limits:
      cpu: 100m
      memory: 128Mi
  nodeSelector:
    k8s.scaleway.com/pool-name: always-on

# Disable components we don't need
logs:
  enabled: false
traces:
  enabled: false
opencost:
  enabled: false
```

**Step 2: Commit**

```bash
git add infra/k8s/monitoring/alloy-values.yaml
git commit -m "feat(infra): add Grafana Alloy k8s-monitoring Helm values for Cockpit"
```

---

## Task 8: DCGM Exporter for GPU Metrics

DaemonSet that runs on GPU nodes only and exposes NVIDIA GPU metrics for Alloy to scrape.

**Files:**
- Create: `infra/k8s/monitoring/dcgm-exporter.yaml`

**Step 1: Create the DCGM Exporter manifest**

```yaml
# NVIDIA DCGM Exporter — GPU metrics for Cockpit
# Runs on all GPU node pools (gpu-training, gpu-inference, ci-build, gpu-dev)
# Alloy auto-discovers and scrapes these pods via annotations
---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    app.kubernetes.io/part-of: foxhunt
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: dcgm-exporter
    app.kubernetes.io/part-of: foxhunt
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: dcgm-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: dcgm-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9400"
        prometheus.io/path: "/metrics"
    spec:
      # Only schedule on nodes that have GPUs
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nvidia.com/gpu.present
                    operator: In
                    values:
                      - "true"
      tolerations:
        # Tolerate GPU taints so the exporter can run on GPU nodes
        - operator: Exists
          effect: NoSchedule
      containers:
        - name: dcgm-exporter
          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
          ports:
            - name: metrics
              containerPort: 9400
          env:
            - name: DCGM_EXPORTER_KUBERNETES
              value: "true"
          resources:
            requests:
              cpu: 25m
              memory: 64Mi
            limits:
              cpu: 100m
              memory: 128Mi
          securityContext:
            runAsNonRoot: false
            runAsUser: 0
          volumeMounts:
            - name: device-metrics
              mountPath: /var/lib/dcgm
      volumes:
        - name: device-metrics
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: dcgm-exporter
  namespace: monitoring
  labels:
    app.kubernetes.io/name: dcgm-exporter
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9400"
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: dcgm-exporter
  ports:
    - name: metrics
      port: 9400
      targetPort: 9400
```

**Step 2: Commit**

```bash
git add infra/k8s/monitoring/dcgm-exporter.yaml
git commit -m "feat(infra): add DCGM Exporter DaemonSet for GPU metrics"
```

---

## Task 9: GitLab Prometheus Remote Write to Cockpit

Configure the bundled GitLab Prometheus to also push metrics to Cockpit, so GitLab application and CI metrics appear alongside cluster metrics.

**Files:**
- Modify: `infra/k8s/gitlab/values.yaml`

**Step 1: Add remote_write config to the prometheus section**

In `infra/k8s/gitlab/values.yaml`, locate the `prometheus:` section (starts at line 71). Add `remoteWrite` after `server.resources.limits` (after line 93, before `gitlab-runner:`):

```yaml
    remoteWrite:
      - url: "COCKPIT_METRICS_PUSH_URL"
        # URL from: cd infra/live/production/cockpit && terragrunt output metrics_push_url
        headers:
          X-TOKEN: "COCKPIT_PUSH_TOKEN"
          # Token from: cd infra/live/production/cockpit && terragrunt output -raw push_token
        writeRelabelConfigs:
          - sourceLabels: [__name__]
            regex: "gitlab_.*|gitaly_.*|sidekiq_.*|ruby_.*|puma_.*"
            action: keep
```

The `writeRelabelConfigs` filter ensures only GitLab-relevant metrics are pushed (not duplicating K8s metrics that Alloy already handles).

**Step 2: Commit**

```bash
git add infra/k8s/gitlab/values.yaml
git commit -m "feat(infra): add Prometheus remote_write to Cockpit in GitLab Helm values"
```

---

## Task 10: Apply Instructions Document

Create a runbook for the operator to apply all changes in the correct order.

**Files:**
- Create: `infra/k8s/monitoring/APPLY-ORDER.md`

**Step 1: Create the runbook**

```markdown
# Cockpit + NAT Gateway — Apply Order

## Prerequisites
- `scw` CLI authenticated
- `terragrunt` installed
- `kubectl` access to foxhunt cluster
- `helm` installed

## Phase 1: Cockpit (metrics backend)

1. Apply Cockpit Terraform module:
   ```bash
   cd infra/live/production/cockpit
   terragrunt apply
   ```

2. Note the outputs:
   ```bash
   terragrunt output metrics_push_url
   terragrunt output -raw push_token
   terragrunt output grafana_url
   terragrunt output -raw grafana_password
   ```

3. Create the K8s secret for the push token:
   ```bash
   kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
   kubectl -n monitoring create secret generic cockpit-push-token \
     --from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
   ```

4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`:
   - Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output

5. Deploy DCGM Exporter:
   ```bash
   kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
   ```

6. Deploy Grafana Alloy (k8s-monitoring):
   ```bash
   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   helm install k8s-monitoring grafana/k8s-monitoring \
     --namespace monitoring \
     -f infra/k8s/monitoring/alloy-values.yaml
   ```

7. Verify metrics flow:
   - Open Cockpit Grafana (URL from step 2)
   - Login with foxhunt-admin / (password from step 2)
   - Check "Kubernetes / Compute Resources / Cluster" dashboard

## Phase 2: GitLab Prometheus remote_write

1. Update placeholders in `infra/k8s/gitlab/values.yaml`:
   - Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN`

2. Upgrade GitLab Helm release:
   ```bash
   helm upgrade gitlab gitlab/gitlab \
     --namespace foxhunt \
     -f infra/k8s/gitlab/values.yaml
   ```

## Phase 3: NAT Gateway

1. Apply Public Gateway module:
   ```bash
   cd infra/live/production/public-gateway
   terragrunt apply
   ```
   This creates the gateway and attaches it to the private network.
   Existing nodes still have public IPs — nothing breaks yet.

2. Apply Kapsule pool changes (disables public IPs):
   ```bash
   cd infra/live/production/kapsule
   terragrunt apply
   ```
   **WARNING**: This may recreate nodes. For zero-downtime:

3. Zero-downtime node migration for always-on pool:
   ```bash
   # Temporarily allow 2 nodes
   # Edit terragrunt.hcl: always_on max_size → 2 (need variable change)
   # Or use SCW CLI:
   scw k8s pool update <always-on-pool-id> max-size=2 size=2
   # Wait for new node (private IP) to be Ready
   kubectl get nodes -w
   # Drain old node
   kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
   # Scale back
   scw k8s pool update <always-on-pool-id> max-size=1 size=1
   ```

4. Repeat for gitlab pool (if not autoscaling, same process).

5. Autoscaling pools (gpu-training, gpu-inference, ci-build, gpu-dev):
   - No action needed — next scale-up automatically uses private IP.

6. Verify Tailscale still works:
   ```bash
   tailscale status  # should show foxhunt-kapsule and foxhunt-gitlab nodes
   curl -k https://git.fxhnt.ai  # should still resolve via Tailscale
   ```
```

**Step 2: Commit**

```bash
git add infra/k8s/monitoring/APPLY-ORDER.md
git commit -m "docs(infra): add apply-order runbook for Cockpit + NAT gateway"
```

---

## Execution Order Summary

| Task | Workstream | Description | Depends On |
|------|-----------|-------------|------------|
| 1 | NAT | Public Gateway Terraform module | — |
| 2 | NAT | Export private_network_id from kapsule | — |
| 3 | NAT | Disable public IPs on all pools | — |
| 4 | NAT | Public Gateway Terragrunt live config | Task 1, 2 |
| 5 | Metrics | Cockpit Terraform module | — |
| 6 | Metrics | Cockpit Terragrunt live config | Task 5 |
| 7 | Metrics | Alloy k8s-monitoring Helm values | — |
| 8 | Metrics | DCGM Exporter DaemonSet | — |
| 9 | Metrics | GitLab Prometheus remote_write | — |
| 10 | Ops | Apply-order runbook | All above |

**Parallelizable:** Tasks 1-3 and 5-9 are all independent and can be done in parallel. Task 4 depends on 1+2. Task 10 is last.
