docs: add Cockpit metrics + NAT gateway design
Covers two workstreams: 1. Scaleway Public Gateway for NAT (all pools private-only) 2. Scaleway Cockpit + Alloy + DCGM Exporter for full metrics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
172
docs/plans/2026-02-27-cockpit-nat-gateway-design.md
Normal file
172
docs/plans/2026-02-27-cockpit-nat-gateway-design.md
Normal file
@@ -0,0 +1,172 @@
|
||||
# K8s Metrics (Cockpit) + NAT Gateway Design
|
||||
|
||||
**Date:** 2026-02-27
|
||||
**Status:** Approved
|
||||
|
||||
## Problem
|
||||
|
||||
1. **No metrics visibility** — the bundled GitLab Prometheus only scrapes GitLab internals. No node CPU/memory, no GPU metrics, no pod resource usage in any dashboard.
|
||||
2. **All nodes have public IPs** — unnecessary attack surface. GPU/CI/dev pools don't need direct internet exposure.
|
||||
|
||||
## Solution
|
||||
|
||||
Two workstreams: Scaleway Cockpit for metrics, Public Gateway for NAT.
|
||||
|
||||
---
|
||||
|
||||
## Workstream 1: NAT Gateway
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Internet
|
||||
|
|
||||
[Public Gateway VPC-GW-S] ── public IP (flexible)
|
||||
|
|
||||
[foxhunt-pn private network]
|
||||
|
|
||||
[Kapsule nodes — all private-only]
|
||||
|
|
||||
[Tailscale subnet router] ── NAT traversal (DERP relays)
|
||||
```
|
||||
|
||||
### Terraform Changes
|
||||
|
||||
**New module: `infra/modules/public-gateway/`**
|
||||
|
||||
Resources:
|
||||
- `scaleway_vpc_public_gateway_ip` — flexible public IP for the gateway
|
||||
- `scaleway_vpc_public_gateway` — gateway instance (VPC-GW-S type)
|
||||
- `scaleway_vpc_gateway_network` — attach to `foxhunt-pn` with masquerade + push_default_route
|
||||
|
||||
**New live config: `infra/live/production/public-gateway/terragrunt.hcl`**
|
||||
|
||||
Inputs:
|
||||
- `private_network_id` — from kapsule module output (needs new output)
|
||||
- `gateway_type` — "VPC-GW-S"
|
||||
|
||||
**Modify: `infra/modules/kapsule/main.tf`**
|
||||
|
||||
- Add `public_ip_disabled = true` to all 6 pool resources
|
||||
- Add new output: `private_network_id` (for public-gateway module dependency)
|
||||
|
||||
### Rollout Strategy (Zero-Downtime)
|
||||
|
||||
1. Apply public-gateway module first (creates gateway, attaches to network)
|
||||
2. Apply kapsule with `public_ip_disabled = true` on all pools
|
||||
3. For always-on pool: temporarily set `max_size = 2` → new node joins with private IP → drain old node → set `max_size = 1`
|
||||
4. For gitlab pool: same approach (temporarily `max_size = 2`, drain old, scale back)
|
||||
5. Autoscaling pools (gpu-training, gpu-inference, ci-build, gpu-dev): next scale-up automatically gets private IP
|
||||
|
||||
### Tailscale Impact
|
||||
|
||||
None. Tailscale is designed for NAT traversal. The subnet router and gitlab proxy will continue working via DERP relays. No config changes needed.
|
||||
|
||||
### Cost
|
||||
|
||||
| Item | Monthly |
|
||||
|------|---------|
|
||||
| VPC-GW-S | ~EUR 14.50 |
|
||||
| Flexible IP | ~EUR 3.00 |
|
||||
| **Subtotal** | **~EUR 17.50** |
|
||||
|
||||
---
|
||||
|
||||
## Workstream 2: Metrics (Scaleway Cockpit)
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
[Grafana Alloy DaemonSet] ──remote_write──> [Cockpit Mimir]
|
||||
scrapes: |
|
||||
- node-exporter (CPU, mem, disk, net) v
|
||||
- kube-state-metrics (pod states) [Cockpit Grafana]
|
||||
- kubelet/cAdvisor (container metrics) (managed, pre-built dashboards)
|
||||
- DCGM Exporter (GPU util, VRAM, temp)
|
||||
|
||||
[GitLab Prometheus] ──remote_write──> [Cockpit Mimir]
|
||||
(GitLab/CI metrics)
|
||||
```
|
||||
|
||||
### What Gets Deployed
|
||||
|
||||
#### 1. Cockpit Terraform Resources (`infra/modules/cockpit/`)
|
||||
|
||||
- `data "scaleway_cockpit"` — get Cockpit endpoints
|
||||
- `scaleway_cockpit_source` — custom metrics data source for K8s data plane
|
||||
- `scaleway_cockpit_token` — push token for Alloy + Prometheus remote_write
|
||||
- `scaleway_cockpit_grafana_user` — admin user for Cockpit Grafana
|
||||
|
||||
#### 2. Grafana Alloy via k8s-monitoring Helm (`infra/k8s/monitoring/alloy-values.yaml`)
|
||||
|
||||
The `grafana/k8s-monitoring` Helm chart deploys:
|
||||
- **Grafana Alloy** — DaemonSet, one pod per node, scrapes all targets and pushes to Cockpit
|
||||
- **node-exporter** — DaemonSet for host-level CPU/memory/disk/network metrics
|
||||
- **kube-state-metrics** — Deployment for K8s object state (pod status, deployments, etc.)
|
||||
|
||||
Configured with Cockpit's remote_write endpoint and push token via `X-TOKEN` header.
|
||||
|
||||
#### 3. DCGM Exporter (`infra/k8s/monitoring/dcgm-exporter.yaml`)
|
||||
|
||||
- DaemonSet with `nodeSelector` for GPU pools (gpu-training, gpu-inference, ci-build, gpu-dev)
|
||||
- Exposes GPU metrics: utilization, memory used/free, temperature, power draw, PCIe throughput
|
||||
- Alloy scrapes the DCGM Exporter pods automatically
|
||||
|
||||
#### 4. GitLab Prometheus Remote Write
|
||||
|
||||
Modify `infra/k8s/gitlab/values.yaml` to add `remote_write` config to the bundled Prometheus:
|
||||
- Pushes GitLab application metrics + CI job metrics to Cockpit
|
||||
- Keeps local scraping for GitLab internal use
|
||||
|
||||
### Metrics Collected
|
||||
|
||||
| Source | Metrics | Nodes |
|
||||
|--------|---------|-------|
|
||||
| node-exporter | CPU, memory, disk I/O, network, filesystem | All 6 pools |
|
||||
| kube-state-metrics | Pod states, deployment status, resource requests/limits | Cluster-wide |
|
||||
| kubelet/cAdvisor | Container CPU/memory/network per pod | All 6 pools |
|
||||
| DCGM Exporter | GPU utilization, VRAM, temperature, power, PCIe | GPU pools only |
|
||||
| GitLab Prometheus | GitLab Rails, Sidekiq, Gitaly, CI job metrics | GitLab pool |
|
||||
|
||||
### Access
|
||||
|
||||
- **Cockpit Grafana**: `cockpit.scaleway.com` → pre-built K8s dashboards
|
||||
- **Existing Grafana** (`grafana.fxhnt.ai`): can add Cockpit as Prometheus-compatible datasource later if desired
|
||||
|
||||
### Cost
|
||||
|
||||
| Item | Monthly |
|
||||
|------|---------|
|
||||
| Custom metrics (~6 nodes, moderate cardinality) | ~EUR 5-15 |
|
||||
| **Subtotal** | **~EUR 5-15** |
|
||||
|
||||
---
|
||||
|
||||
## Total Cost Impact
|
||||
|
||||
| Component | Monthly |
|
||||
|-----------|---------|
|
||||
| Public Gateway + IP | ~EUR 17.50 |
|
||||
| Cockpit custom metrics | ~EUR 5-15 |
|
||||
| **Total** | **~EUR 22-33** |
|
||||
|
||||
## File Changes Summary
|
||||
|
||||
### New Files
|
||||
- `infra/modules/public-gateway/main.tf` — gateway + IP + network attachment
|
||||
- `infra/modules/public-gateway/variables.tf`
|
||||
- `infra/modules/public-gateway/outputs.tf`
|
||||
- `infra/live/production/public-gateway/terragrunt.hcl`
|
||||
- `infra/modules/cockpit/main.tf` — Cockpit source + token + Grafana user
|
||||
- `infra/modules/cockpit/variables.tf`
|
||||
- `infra/modules/cockpit/outputs.tf`
|
||||
- `infra/live/production/cockpit/terragrunt.hcl`
|
||||
- `infra/k8s/monitoring/alloy-values.yaml` — k8s-monitoring Helm values
|
||||
- `infra/k8s/monitoring/dcgm-exporter.yaml` — DCGM DaemonSet
|
||||
|
||||
### Modified Files
|
||||
- `infra/modules/kapsule/main.tf` — add `public_ip_disabled`, output `private_network_id`
|
||||
- `infra/modules/kapsule/variables.tf` — new variable
|
||||
- `infra/modules/kapsule/outputs.tf` — new output
|
||||
- `infra/live/production/kapsule/terragrunt.hcl` — pass new variable
|
||||
- `infra/k8s/gitlab/values.yaml` — add Prometheus remote_write to Cockpit
|
||||
Reference in New Issue
Block a user