docs: add implementation plan for Cockpit metrics + NAT gateway
10 tasks across 2 workstreams (NAT + Metrics), mostly parallelizable. Includes Terraform modules, K8s manifests, Helm values, and runbook. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
795
docs/plans/2026-02-27-cockpit-nat-gateway-implementation.md
Normal file
795
docs/plans/2026-02-27-cockpit-nat-gateway-implementation.md
Normal file
@@ -0,0 +1,795 @@
|
||||
# Cockpit Metrics + NAT Gateway Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Deploy a Scaleway Public Gateway (NAT) so all K8s nodes use private-only IPs, and set up Scaleway Cockpit with Grafana Alloy + DCGM Exporter for full cluster metrics (CPU, memory, disk, network, GPU).
|
||||
|
||||
**Architecture:** Two independent workstreams. Workstream 1 creates a Terraform module for the Public Gateway, attaches it to the existing VPC private network, and disables public IPs on all node pools. Workstream 2 creates a Terraform module for Cockpit credentials, then deploys K8s manifests for Grafana Alloy (via k8s-monitoring Helm chart) and DCGM Exporter, plus configures GitLab's bundled Prometheus to remote_write to Cockpit.
|
||||
|
||||
**Tech Stack:** Terraform (Scaleway provider ~2.46), Terragrunt, Kubernetes YAML, Helm values, Scaleway Cockpit (Mimir), Grafana Alloy, NVIDIA DCGM Exporter
|
||||
|
||||
**Design doc:** `docs/plans/2026-02-27-cockpit-nat-gateway-design.md`
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Public Gateway Terraform Module
|
||||
|
||||
Creates the NAT gateway, reserves a public IP, and attaches it to the existing private network.
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/modules/public-gateway/main.tf`
|
||||
- Create: `infra/modules/public-gateway/variables.tf`
|
||||
- Create: `infra/modules/public-gateway/outputs.tf`
|
||||
|
||||
**Step 1: Create `infra/modules/public-gateway/variables.tf`**
|
||||
|
||||
```hcl
|
||||
variable "region" {
|
||||
description = "Scaleway region"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "zone" {
|
||||
description = "Scaleway zone for the gateway"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "Scaleway project ID"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "private_network_id" {
|
||||
description = "ID of the existing VPC private network to attach the gateway to"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "gateway_type" {
|
||||
description = "Public Gateway instance type (VPC-GW-S, VPC-GW-M)"
|
||||
type = string
|
||||
default = "VPC-GW-S"
|
||||
}
|
||||
|
||||
variable "bastion_enabled" {
|
||||
description = "Enable SSH bastion on the gateway"
|
||||
type = bool
|
||||
default = false
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Create `infra/modules/public-gateway/main.tf`**
|
||||
|
||||
```hcl
|
||||
resource "scaleway_vpc_public_gateway_ip" "foxhunt" {
|
||||
zone = var.zone
|
||||
}
|
||||
|
||||
resource "scaleway_vpc_public_gateway" "foxhunt" {
|
||||
name = "foxhunt-gw"
|
||||
type = var.gateway_type
|
||||
zone = var.zone
|
||||
bastion_enabled = var.bastion_enabled
|
||||
ip_id = scaleway_vpc_public_gateway_ip.foxhunt.id
|
||||
}
|
||||
|
||||
resource "scaleway_vpc_gateway_network" "foxhunt" {
|
||||
gateway_id = scaleway_vpc_public_gateway.foxhunt.id
|
||||
private_network_id = var.private_network_id
|
||||
enable_masquerade = true
|
||||
|
||||
ipam_config {
|
||||
push_default_route = true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 3: Create `infra/modules/public-gateway/outputs.tf`**
|
||||
|
||||
```hcl
|
||||
output "gateway_id" {
|
||||
description = "ID of the Public Gateway"
|
||||
value = scaleway_vpc_public_gateway.foxhunt.id
|
||||
}
|
||||
|
||||
output "gateway_ip" {
|
||||
description = "Public IP address of the gateway"
|
||||
value = scaleway_vpc_public_gateway_ip.foxhunt.address
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4: Validate module syntax**
|
||||
|
||||
Run: `cd infra/modules/public-gateway && terraform init -backend=false && terraform validate`
|
||||
Expected: "Success! The configuration is valid."
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/modules/public-gateway/
|
||||
git commit -m "feat(infra): add public-gateway Terraform module for NAT"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Export Private Network ID from Kapsule Module
|
||||
|
||||
The public-gateway module needs the private network ID. Add it as an output from the kapsule module.
|
||||
|
||||
**Files:**
|
||||
- Modify: `infra/modules/kapsule/outputs.tf`
|
||||
|
||||
**Step 1: Add private_network_id output**
|
||||
|
||||
Append to `infra/modules/kapsule/outputs.tf` (after the existing `gitlab_pool_id` output):
|
||||
|
||||
```hcl
|
||||
output "private_network_id" {
|
||||
description = "ID of the VPC private network the cluster is attached to"
|
||||
value = scaleway_vpc_private_network.foxhunt.id
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/modules/kapsule/outputs.tf
|
||||
git commit -m "feat(infra): export private_network_id from kapsule module"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Disable Public IPs on All Node Pools
|
||||
|
||||
Add `public_ip_disabled` to every pool in the kapsule module, controlled by a single variable.
|
||||
|
||||
**Files:**
|
||||
- Modify: `infra/modules/kapsule/variables.tf`
|
||||
- Modify: `infra/modules/kapsule/main.tf`
|
||||
- Modify: `infra/live/production/kapsule/terragrunt.hcl`
|
||||
|
||||
**Step 1: Add variable to `infra/modules/kapsule/variables.tf`**
|
||||
|
||||
Append at the end of the file:
|
||||
|
||||
```hcl
|
||||
variable "public_ip_disabled" {
|
||||
description = "Disable public IPs on all node pools (requires Public Gateway for egress)"
|
||||
type = bool
|
||||
default = false
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Add `public_ip_disabled` to each pool in `infra/modules/kapsule/main.tf`**
|
||||
|
||||
Add `public_ip_disabled = var.public_ip_disabled` to each of the 6 pool resources. The attribute goes after `autohealing = true` in each block:
|
||||
|
||||
For `scaleway_k8s_pool.always_on` (line ~37, after `autohealing = true`):
|
||||
```hcl
|
||||
public_ip_disabled = var.public_ip_disabled
|
||||
```
|
||||
|
||||
For `scaleway_k8s_pool.gpu_training` (line ~51, after `autohealing = true`):
|
||||
```hcl
|
||||
public_ip_disabled = var.public_ip_disabled
|
||||
```
|
||||
|
||||
For `scaleway_k8s_pool.gpu_inference` (line ~70, after `autohealing = true`):
|
||||
```hcl
|
||||
public_ip_disabled = var.public_ip_disabled
|
||||
```
|
||||
|
||||
For `scaleway_k8s_pool.gitlab` (line ~88, after `autohealing = true`):
|
||||
```hcl
|
||||
public_ip_disabled = var.public_ip_disabled
|
||||
```
|
||||
|
||||
For `scaleway_k8s_pool.ci_build` (line ~102, after `autohealing = true`):
|
||||
```hcl
|
||||
public_ip_disabled = var.public_ip_disabled
|
||||
```
|
||||
|
||||
For `scaleway_k8s_pool.dev` (line ~121, after `autohealing = true`):
|
||||
```hcl
|
||||
public_ip_disabled = var.public_ip_disabled
|
||||
```
|
||||
|
||||
**Step 3: Set the variable in `infra/live/production/kapsule/terragrunt.hcl`**
|
||||
|
||||
Add to the `inputs` block (after the `dev_pool_max_size` line):
|
||||
|
||||
```hcl
|
||||
# NAT gateway handles egress — no public IPs on nodes
|
||||
public_ip_disabled = true
|
||||
```
|
||||
|
||||
**Step 4: Validate kapsule module syntax**
|
||||
|
||||
Run: `cd infra/modules/kapsule && terraform init -backend=false && terraform validate`
|
||||
Expected: "Success! The configuration is valid."
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/modules/kapsule/variables.tf infra/modules/kapsule/main.tf infra/live/production/kapsule/terragrunt.hcl
|
||||
git commit -m "feat(infra): disable public IPs on all Kapsule node pools"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Public Gateway Terragrunt Live Config
|
||||
|
||||
Create the live production config that wires the public-gateway module to the existing private network.
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/live/production/public-gateway/terragrunt.hcl`
|
||||
|
||||
**Step 1: Create the terragrunt config**
|
||||
|
||||
```hcl
|
||||
include "root" {
|
||||
path = find_in_parent_folders("root.hcl")
|
||||
}
|
||||
|
||||
terraform {
|
||||
source = "../../../modules/public-gateway"
|
||||
}
|
||||
|
||||
dependency "kapsule" {
|
||||
config_path = "../kapsule"
|
||||
}
|
||||
|
||||
inputs = {
|
||||
private_network_id = dependency.kapsule.outputs.private_network_id
|
||||
gateway_type = "VPC-GW-S"
|
||||
}
|
||||
```
|
||||
|
||||
Note: This is the first Terragrunt `dependency` block in the project. Terragrunt will automatically read the kapsule state to get the private_network_id output.
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/live/production/public-gateway/terragrunt.hcl
|
||||
git commit -m "feat(infra): add public-gateway live config with kapsule dependency"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Cockpit Terraform Module
|
||||
|
||||
Creates the Cockpit data source, push token, and Grafana user.
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/modules/cockpit/main.tf`
|
||||
- Create: `infra/modules/cockpit/variables.tf`
|
||||
- Create: `infra/modules/cockpit/outputs.tf`
|
||||
|
||||
**Step 1: Create `infra/modules/cockpit/variables.tf`**
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
description = "Scaleway project ID"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "grafana_login" {
|
||||
description = "Login for Cockpit Grafana admin user"
|
||||
type = string
|
||||
default = "foxhunt-admin"
|
||||
}
|
||||
|
||||
variable "grafana_role" {
|
||||
description = "Role for the Cockpit Grafana user (editor or viewer)"
|
||||
type = string
|
||||
default = "editor"
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Create `infra/modules/cockpit/main.tf`**
|
||||
|
||||
```hcl
|
||||
data "scaleway_cockpit" "main" {
|
||||
project_id = var.project_id
|
||||
}
|
||||
|
||||
# Custom metrics data source for K8s data plane
|
||||
resource "scaleway_cockpit_source" "k8s_metrics" {
|
||||
project_id = var.project_id
|
||||
name = "kapsule-data-plane"
|
||||
type = "metrics"
|
||||
}
|
||||
|
||||
# Push token for Grafana Alloy + Prometheus remote_write
|
||||
resource "scaleway_cockpit_token" "k8s_push" {
|
||||
project_id = var.project_id
|
||||
name = "kapsule-push-token"
|
||||
|
||||
scopes {
|
||||
query_metrics = true
|
||||
write_metrics = true
|
||||
query_logs = true
|
||||
write_logs = true
|
||||
}
|
||||
}
|
||||
|
||||
# Grafana user for dashboard access
|
||||
resource "scaleway_cockpit_grafana_user" "admin" {
|
||||
project_id = var.project_id
|
||||
login = var.grafana_login
|
||||
role = var.grafana_role
|
||||
}
|
||||
```
|
||||
|
||||
**Step 3: Create `infra/modules/cockpit/outputs.tf`**
|
||||
|
||||
```hcl
|
||||
output "metrics_push_url" {
|
||||
description = "Cockpit Mimir remote_write URL for Alloy/Prometheus"
|
||||
value = scaleway_cockpit_source.k8s_metrics.push_url
|
||||
}
|
||||
|
||||
output "push_token" {
|
||||
description = "Token for authenticating remote_write requests"
|
||||
value = scaleway_cockpit_token.k8s_push.secret_key
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
output "grafana_url" {
|
||||
description = "Cockpit Grafana dashboard URL"
|
||||
value = data.scaleway_cockpit.main.endpoints[0].grafana_url
|
||||
}
|
||||
|
||||
output "grafana_password" {
|
||||
description = "Generated password for the Grafana user"
|
||||
value = scaleway_cockpit_grafana_user.admin.password
|
||||
sensitive = true
|
||||
}
|
||||
```
|
||||
|
||||
**Step 4: Validate module syntax**
|
||||
|
||||
Run: `cd infra/modules/cockpit && terraform init -backend=false && terraform validate`
|
||||
Expected: "Success! The configuration is valid."
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/modules/cockpit/
|
||||
git commit -m "feat(infra): add cockpit Terraform module for metrics push"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Cockpit Terragrunt Live Config
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/live/production/cockpit/terragrunt.hcl`
|
||||
|
||||
**Step 1: Create the terragrunt config**
|
||||
|
||||
```hcl
|
||||
include "root" {
|
||||
path = find_in_parent_folders("root.hcl")
|
||||
}
|
||||
|
||||
terraform {
|
||||
source = "../../../modules/cockpit"
|
||||
}
|
||||
|
||||
inputs = {}
|
||||
```
|
||||
|
||||
Note: `project_id` is already provided by the root.hcl common inputs. No dependencies needed.
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/live/production/cockpit/terragrunt.hcl
|
||||
git commit -m "feat(infra): add cockpit live config"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Grafana Alloy (k8s-monitoring) Helm Values
|
||||
|
||||
The `grafana/k8s-monitoring` Helm chart deploys Alloy + node-exporter + kube-state-metrics. This values file configures it to push to Cockpit.
|
||||
|
||||
**Important:** The actual `metrics_push_url` and `push_token` values must be filled in after `terragrunt apply` on the cockpit module. Use placeholders that the operator replaces.
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/k8s/monitoring/alloy-values.yaml`
|
||||
|
||||
**Step 1: Create the Helm values file**
|
||||
|
||||
```yaml
|
||||
# Grafana k8s-monitoring Helm chart values
|
||||
# Chart: grafana/k8s-monitoring
|
||||
# Deploys: Alloy (DaemonSet) + node-exporter + kube-state-metrics
|
||||
#
|
||||
# Install:
|
||||
# helm repo add grafana https://grafana.github.io/helm-charts
|
||||
# helm install k8s-monitoring grafana/k8s-monitoring \
|
||||
# --namespace monitoring --create-namespace \
|
||||
# -f infra/k8s/monitoring/alloy-values.yaml
|
||||
#
|
||||
# After terragrunt apply on cockpit module, get the values:
|
||||
# cd infra/live/production/cockpit && terragrunt output -json
|
||||
# Then create the K8s secret:
|
||||
# kubectl -n monitoring create secret generic cockpit-push-token \
|
||||
# --from-literal=token=<push_token from terragrunt output>
|
||||
|
||||
cluster:
|
||||
name: foxhunt
|
||||
|
||||
externalServices:
|
||||
prometheus:
|
||||
host: "COCKPIT_METRICS_PUSH_URL"
|
||||
# URL from: cd infra/live/production/cockpit && terragrunt output metrics_push_url
|
||||
# Format: https://<source-id>.metrics.cockpit.fr-par.scw.cloud/api/v1/push
|
||||
authMode: raw
|
||||
secret:
|
||||
create: false
|
||||
name: cockpit-push-token
|
||||
namespace: monitoring
|
||||
customHeaders:
|
||||
- name: X-TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: cockpit-push-token
|
||||
key: token
|
||||
|
||||
metrics:
|
||||
enabled: true
|
||||
cost:
|
||||
enabled: false
|
||||
node-exporter:
|
||||
enabled: true
|
||||
kube-state-metrics:
|
||||
enabled: true
|
||||
kubelet:
|
||||
enabled: true
|
||||
cadvisor:
|
||||
enabled: true
|
||||
apiserver:
|
||||
enabled: true
|
||||
|
||||
# Alloy DaemonSet — runs on every node
|
||||
alloy-metrics:
|
||||
alloy:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
|
||||
# node-exporter — host metrics (CPU, memory, disk, network)
|
||||
prometheus-node-exporter:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
|
||||
# kube-state-metrics — K8s object state
|
||||
kube-state-metrics:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: always-on
|
||||
|
||||
# Disable components we don't need
|
||||
logs:
|
||||
enabled: false
|
||||
traces:
|
||||
enabled: false
|
||||
opencost:
|
||||
enabled: false
|
||||
```
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/monitoring/alloy-values.yaml
|
||||
git commit -m "feat(infra): add Grafana Alloy k8s-monitoring Helm values for Cockpit"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: DCGM Exporter for GPU Metrics
|
||||
|
||||
DaemonSet that runs on GPU nodes only and exposes NVIDIA GPU metrics for Alloy to scrape.
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/k8s/monitoring/dcgm-exporter.yaml`
|
||||
|
||||
**Step 1: Create the DCGM Exporter manifest**
|
||||
|
||||
```yaml
|
||||
# NVIDIA DCGM Exporter — GPU metrics for Cockpit
|
||||
# Runs on all GPU node pools (gpu-training, gpu-inference, ci-build, gpu-dev)
|
||||
# Alloy auto-discovers and scrapes these pods via annotations
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: dcgm-exporter
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9400"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
# Only schedule on nodes that have GPUs
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: nvidia.com/gpu.present
|
||||
operator: In
|
||||
values:
|
||||
- "true"
|
||||
tolerations:
|
||||
# Tolerate GPU taints so the exporter can run on GPU nodes
|
||||
- operator: Exists
|
||||
effect: NoSchedule
|
||||
containers:
|
||||
- name: dcgm-exporter
|
||||
image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
|
||||
ports:
|
||||
- name: metrics
|
||||
containerPort: 9400
|
||||
env:
|
||||
- name: DCGM_EXPORTER_KUBERNETES
|
||||
value: "true"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
runAsNonRoot: false
|
||||
runAsUser: 0
|
||||
volumeMounts:
|
||||
- name: device-metrics
|
||||
mountPath: /var/lib/dcgm
|
||||
volumes:
|
||||
- name: device-metrics
|
||||
emptyDir: {}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: dcgm-exporter
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9400"
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
ports:
|
||||
- name: metrics
|
||||
port: 9400
|
||||
targetPort: 9400
|
||||
```
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/monitoring/dcgm-exporter.yaml
|
||||
git commit -m "feat(infra): add DCGM Exporter DaemonSet for GPU metrics"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 9: GitLab Prometheus Remote Write to Cockpit
|
||||
|
||||
Configure the bundled GitLab Prometheus to also push metrics to Cockpit, so GitLab application and CI metrics appear alongside cluster metrics.
|
||||
|
||||
**Files:**
|
||||
- Modify: `infra/k8s/gitlab/values.yaml`
|
||||
|
||||
**Step 1: Add remote_write config to the prometheus section**
|
||||
|
||||
In `infra/k8s/gitlab/values.yaml`, locate the `prometheus:` section (starts at line 71). Add `remoteWrite` after `server.resources.limits` (after line 93, before `gitlab-runner:`):
|
||||
|
||||
```yaml
|
||||
remoteWrite:
|
||||
- url: "COCKPIT_METRICS_PUSH_URL"
|
||||
# URL from: cd infra/live/production/cockpit && terragrunt output metrics_push_url
|
||||
headers:
|
||||
X-TOKEN: "COCKPIT_PUSH_TOKEN"
|
||||
# Token from: cd infra/live/production/cockpit && terragrunt output -raw push_token
|
||||
writeRelabelConfigs:
|
||||
- sourceLabels: [__name__]
|
||||
regex: "gitlab_.*|gitaly_.*|sidekiq_.*|ruby_.*|puma_.*"
|
||||
action: keep
|
||||
```
|
||||
|
||||
The `writeRelabelConfigs` filter ensures only GitLab-relevant metrics are pushed (not duplicating K8s metrics that Alloy already handles).
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/gitlab/values.yaml
|
||||
git commit -m "feat(infra): add Prometheus remote_write to Cockpit in GitLab Helm values"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 10: Apply Instructions Document
|
||||
|
||||
Create a runbook for the operator to apply all changes in the correct order.
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/k8s/monitoring/APPLY-ORDER.md`
|
||||
|
||||
**Step 1: Create the runbook**
|
||||
|
||||
```markdown
|
||||
# Cockpit + NAT Gateway — Apply Order
|
||||
|
||||
## Prerequisites
|
||||
- `scw` CLI authenticated
|
||||
- `terragrunt` installed
|
||||
- `kubectl` access to foxhunt cluster
|
||||
- `helm` installed
|
||||
|
||||
## Phase 1: Cockpit (metrics backend)
|
||||
|
||||
1. Apply Cockpit Terraform module:
|
||||
```bash
|
||||
cd infra/live/production/cockpit
|
||||
terragrunt apply
|
||||
```
|
||||
|
||||
2. Note the outputs:
|
||||
```bash
|
||||
terragrunt output metrics_push_url
|
||||
terragrunt output -raw push_token
|
||||
terragrunt output grafana_url
|
||||
terragrunt output -raw grafana_password
|
||||
```
|
||||
|
||||
3. Create the K8s secret for the push token:
|
||||
```bash
|
||||
kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
|
||||
kubectl -n monitoring create secret generic cockpit-push-token \
|
||||
--from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
|
||||
```
|
||||
|
||||
4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`:
|
||||
- Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output
|
||||
|
||||
5. Deploy DCGM Exporter:
|
||||
```bash
|
||||
kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
|
||||
```
|
||||
|
||||
6. Deploy Grafana Alloy (k8s-monitoring):
|
||||
```bash
|
||||
helm repo add grafana https://grafana.github.io/helm-charts
|
||||
helm repo update
|
||||
helm install k8s-monitoring grafana/k8s-monitoring \
|
||||
--namespace monitoring \
|
||||
-f infra/k8s/monitoring/alloy-values.yaml
|
||||
```
|
||||
|
||||
7. Verify metrics flow:
|
||||
- Open Cockpit Grafana (URL from step 2)
|
||||
- Login with foxhunt-admin / (password from step 2)
|
||||
- Check "Kubernetes / Compute Resources / Cluster" dashboard
|
||||
|
||||
## Phase 2: GitLab Prometheus remote_write
|
||||
|
||||
1. Update placeholders in `infra/k8s/gitlab/values.yaml`:
|
||||
- Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN`
|
||||
|
||||
2. Upgrade GitLab Helm release:
|
||||
```bash
|
||||
helm upgrade gitlab gitlab/gitlab \
|
||||
--namespace foxhunt \
|
||||
-f infra/k8s/gitlab/values.yaml
|
||||
```
|
||||
|
||||
## Phase 3: NAT Gateway
|
||||
|
||||
1. Apply Public Gateway module:
|
||||
```bash
|
||||
cd infra/live/production/public-gateway
|
||||
terragrunt apply
|
||||
```
|
||||
This creates the gateway and attaches it to the private network.
|
||||
Existing nodes still have public IPs — nothing breaks yet.
|
||||
|
||||
2. Apply Kapsule pool changes (disables public IPs):
|
||||
```bash
|
||||
cd infra/live/production/kapsule
|
||||
terragrunt apply
|
||||
```
|
||||
**WARNING**: This may recreate nodes. For zero-downtime:
|
||||
|
||||
3. Zero-downtime node migration for always-on pool:
|
||||
```bash
|
||||
# Temporarily allow 2 nodes
|
||||
# Edit terragrunt.hcl: always_on max_size → 2 (need variable change)
|
||||
# Or use SCW CLI:
|
||||
scw k8s pool update <always-on-pool-id> max-size=2 size=2
|
||||
# Wait for new node (private IP) to be Ready
|
||||
kubectl get nodes -w
|
||||
# Drain old node
|
||||
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
|
||||
# Scale back
|
||||
scw k8s pool update <always-on-pool-id> max-size=1 size=1
|
||||
```
|
||||
|
||||
4. Repeat for gitlab pool (if not autoscaling, same process).
|
||||
|
||||
5. Autoscaling pools (gpu-training, gpu-inference, ci-build, gpu-dev):
|
||||
- No action needed — next scale-up automatically uses private IP.
|
||||
|
||||
6. Verify Tailscale still works:
|
||||
```bash
|
||||
tailscale status # should show foxhunt-kapsule and foxhunt-gitlab nodes
|
||||
curl -k https://git.fxhnt.ai # should still resolve via Tailscale
|
||||
```
|
||||
```
|
||||
|
||||
**Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/monitoring/APPLY-ORDER.md
|
||||
git commit -m "docs(infra): add apply-order runbook for Cockpit + NAT gateway"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution Order Summary
|
||||
|
||||
| Task | Workstream | Description | Depends On |
|
||||
|------|-----------|-------------|------------|
|
||||
| 1 | NAT | Public Gateway Terraform module | — |
|
||||
| 2 | NAT | Export private_network_id from kapsule | — |
|
||||
| 3 | NAT | Disable public IPs on all pools | — |
|
||||
| 4 | NAT | Public Gateway Terragrunt live config | Task 1, 2 |
|
||||
| 5 | Metrics | Cockpit Terraform module | — |
|
||||
| 6 | Metrics | Cockpit Terragrunt live config | Task 5 |
|
||||
| 7 | Metrics | Alloy k8s-monitoring Helm values | — |
|
||||
| 8 | Metrics | DCGM Exporter DaemonSet | — |
|
||||
| 9 | Metrics | GitLab Prometheus remote_write | — |
|
||||
| 10 | Ops | Apply-order runbook | All above |
|
||||
|
||||
**Parallelizable:** Tasks 1-3 and 5-9 are all independent and can be done in parallel. Task 4 depends on 1+2. Task 10 is last.
|
||||
Reference in New Issue
Block a user