# K8s Metrics (Cockpit) + NAT Gateway Design

**Date:** 2026-02-27
**Status:** Approved

## Problem

1. **No metrics visibility** — the bundled GitLab Prometheus only scrapes GitLab internals. No node CPU/memory, no GPU metrics, no pod resource usage in any dashboard.
2. **All nodes have public IPs** — unnecessary attack surface. GPU/CI/dev pools don't need direct internet exposure.

## Solution

Two workstreams: Scaleway Cockpit for metrics, Public Gateway for NAT.

---

## Workstream 1: NAT Gateway

### Architecture

```
Internet
    |
[Public Gateway VPC-GW-S] ── public IP (flexible)
    |
[foxhunt-pn private network]
    |
[Kapsule nodes — all private-only]
    |
[Tailscale subnet router] ── NAT traversal (DERP relays)
```

### Terraform Changes

**New module: `infra/modules/public-gateway/`**

Resources:
- `scaleway_vpc_public_gateway_ip` — flexible public IP for the gateway
- `scaleway_vpc_public_gateway` — gateway instance (VPC-GW-S type)
- `scaleway_vpc_gateway_network` — attach to `foxhunt-pn` with masquerade + push_default_route

**New live config: `infra/live/production/public-gateway/terragrunt.hcl`**

Inputs:
- `private_network_id` — from kapsule module output (needs new output)
- `gateway_type` — "VPC-GW-S"

**Modify: `infra/modules/kapsule/main.tf`**

- Add `public_ip_disabled = true` to all 6 pool resources
- Add new output: `private_network_id` (for public-gateway module dependency)

### Rollout Strategy (Zero-Downtime)

1. Apply public-gateway module first (creates gateway, attaches to network)
2. Apply kapsule with `public_ip_disabled = true` on all pools
3. For always-on pool: temporarily set `max_size = 2` → new node joins with private IP → drain old node → set `max_size = 1`
4. For gitlab pool: same approach (temporarily `max_size = 2`, drain old, scale back)
5. Autoscaling pools (gpu-training, gpu-inference, ci-build, gpu-dev): next scale-up automatically gets private IP

### Tailscale Impact

None. Tailscale is designed for NAT traversal. The subnet router and gitlab proxy will continue working via DERP relays. No config changes needed.

### Cost

| Item | Monthly |
|------|---------|
| VPC-GW-S | ~EUR 14.50 |
| Flexible IP | ~EUR 3.00 |
| **Subtotal** | **~EUR 17.50** |

---

## Workstream 2: Metrics (Scaleway Cockpit)

### Architecture

```
[Grafana Alloy DaemonSet] ──remote_write──> [Cockpit Mimir]
    scrapes:                                      |
    - node-exporter (CPU, mem, disk, net)         v
    - kube-state-metrics (pod states)        [Cockpit Grafana]
    - kubelet/cAdvisor (container metrics)    (managed, pre-built dashboards)
    - DCGM Exporter (GPU util, VRAM, temp)

[GitLab Prometheus] ──remote_write──> [Cockpit Mimir]
    (GitLab/CI metrics)
```

### What Gets Deployed

#### 1. Cockpit Terraform Resources (`infra/modules/cockpit/`)

- `data "scaleway_cockpit"` — get Cockpit endpoints
- `scaleway_cockpit_source` — custom metrics data source for K8s data plane
- `scaleway_cockpit_token` — push token for Alloy + Prometheus remote_write
- `scaleway_cockpit_grafana_user` — admin user for Cockpit Grafana

#### 2. Grafana Alloy via k8s-monitoring Helm (`infra/k8s/monitoring/alloy-values.yaml`)

The `grafana/k8s-monitoring` Helm chart deploys:
- **Grafana Alloy** — DaemonSet, one pod per node, scrapes all targets and pushes to Cockpit
- **node-exporter** — DaemonSet for host-level CPU/memory/disk/network metrics
- **kube-state-metrics** — Deployment for K8s object state (pod status, deployments, etc.)

Configured with Cockpit's remote_write endpoint and push token via `X-TOKEN` header.

#### 3. DCGM Exporter (`infra/k8s/monitoring/dcgm-exporter.yaml`)

- DaemonSet with `nodeSelector` for GPU pools (gpu-training, gpu-inference, ci-build, gpu-dev)
- Exposes GPU metrics: utilization, memory used/free, temperature, power draw, PCIe throughput
- Alloy scrapes the DCGM Exporter pods automatically

#### 4. GitLab Prometheus Remote Write

Modify `infra/k8s/gitlab/values.yaml` to add `remote_write` config to the bundled Prometheus:
- Pushes GitLab application metrics + CI job metrics to Cockpit
- Keeps local scraping for GitLab internal use

### Metrics Collected

| Source | Metrics | Nodes |
|--------|---------|-------|
| node-exporter | CPU, memory, disk I/O, network, filesystem | All 6 pools |
| kube-state-metrics | Pod states, deployment status, resource requests/limits | Cluster-wide |
| kubelet/cAdvisor | Container CPU/memory/network per pod | All 6 pools |
| DCGM Exporter | GPU utilization, VRAM, temperature, power, PCIe | GPU pools only |
| GitLab Prometheus | GitLab Rails, Sidekiq, Gitaly, CI job metrics | GitLab pool |

### Access

- **Cockpit Grafana**: `cockpit.scaleway.com` → pre-built K8s dashboards
- **Existing Grafana** (`grafana.fxhnt.ai`): can add Cockpit as Prometheus-compatible datasource later if desired

### Cost

| Item | Monthly |
|------|---------|
| Custom metrics (~6 nodes, moderate cardinality) | ~EUR 5-15 |
| **Subtotal** | **~EUR 5-15** |

---

## Total Cost Impact

| Component | Monthly |
|-----------|---------|
| Public Gateway + IP | ~EUR 17.50 |
| Cockpit custom metrics | ~EUR 5-15 |
| **Total** | **~EUR 22-33** |

## File Changes Summary

### New Files
- `infra/modules/public-gateway/main.tf` — gateway + IP + network attachment
- `infra/modules/public-gateway/variables.tf`
- `infra/modules/public-gateway/outputs.tf`
- `infra/live/production/public-gateway/terragrunt.hcl`
- `infra/modules/cockpit/main.tf` — Cockpit source + token + Grafana user
- `infra/modules/cockpit/variables.tf`
- `infra/modules/cockpit/outputs.tf`
- `infra/live/production/cockpit/terragrunt.hcl`
- `infra/k8s/monitoring/alloy-values.yaml` — k8s-monitoring Helm values
- `infra/k8s/monitoring/dcgm-exporter.yaml` — DCGM DaemonSet

### Modified Files
- `infra/modules/kapsule/main.tf` — add `public_ip_disabled`, output `private_network_id`
- `infra/modules/kapsule/variables.tf` — new variable
- `infra/modules/kapsule/outputs.tf` — new output
- `infra/live/production/kapsule/terragrunt.hcl` — pass new variable
- `infra/k8s/gitlab/values.yaml` — add Prometheus remote_write to Cockpit
