feat(infra): standalone Prometheus deployment for foxhunt namespace

- ServiceAccount + ClusterRole (nodes, pods, endpoints, metrics) - ConfigMap with 7 scrape jobs: self, foxhunt-services, annotated-pods, gitlab-annotated-pods, node-exporter, kube-state-metrics, dcgm-exporter, pushgateway - Deployment (Prometheus v3.8.1, 14d retention, 1500MB size limit) - PVC (2Gi scw-bssd), Service (port 80 → 9090) Also includes pods-panel-versioning design doc. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 13:24:52 +01:00
parent 8aa8e6db6a
commit 7c8fa991cc
2 changed files with 380 additions and 0 deletions
--- a/docs/plans/2026-03-04-pods-panel-versioning-design.md
+++ b/docs/plans/2026-03-04-pods-panel-versioning-design.md
@@ -0,0 +1,104 @@
+# Pods Panel, CalVer Versioning & Cluster Resources Fix
+
+**Date**: 2026-03-04
+**Status**: Approved
+
+## 1. CalVer Auto-Versioning
+
+**Scheme**: `YYYY.MM.PIPELINE_IID` (e.g. `2026.03.408`)
+
+### Flow
+
+1. CI compile jobs compute `FOXHUNT_VERSION` from `CI_COMMIT_TIMESTAMP` year.month + `CI_PIPELINE_IID`
+2. Passed to rustc via `FOXHUNT_BUILD_VERSION` env var
+3. `common/src/build_info.rs` exposes `pub fn version() -> &'static str` using `option_env!("FOXHUNT_BUILD_VERSION").unwrap_or(env!("CARGO_PKG_VERSION"))`
+4. Each service populates `ServiceStatus.version` from `build_info::version()`
+5. fxt shows version in TUI header bar and Services cockpit column
+
+### Files changed
+
+- `.gitlab-ci.yml`: compute + export `FOXHUNT_BUILD_VERSION` in compile jobs (~10 lines)
+- `crates/common/src/build_info.rs`: new module, `version()` function
+- `crates/common/src/lib.rs`: re-export `build_info`
+- `services/api_gateway/src/grpc/monitoring_handler.rs`: populate `version` field from `build_info::version()`
+- All 7 service `main.rs` files: log version at startup
+- `bin/fxt/src/tui/cockpits/overview.rs`: show fxt version in header
+
+## 2. Pods Cockpit (Key 7)
+
+### Proto
+
+New streaming RPC in `monitoring.proto`:
+
+```protobuf
+rpc SubscribeClusterPods(SubscribeClusterPodsRequest) returns (stream ClusterPodsResponse);
+
+message SubscribeClusterPodsRequest {
+  uint32 interval_seconds = 1;  // default 5
+}
+
+message ClusterPodsResponse {
+  repeated PodInfo pods = 1;
+  int64 timestamp = 2;
+}
+
+message PodInfo {
+  string name = 1;
+  string service = 2;        // app.kubernetes.io/name label
+  string status = 3;         // Running/Pending/CrashLoopBackOff/Completed/etc
+  int32 restarts = 4;
+  string age = 5;
+  string node = 6;
+  float cpu_millicores = 7;  // from metrics-server (0 if unavailable)
+  float memory_mb = 8;       // from metrics-server (0 if unavailable)
+  bool ready = 9;
+  string version = 10;       // from FOXHUNT_BUILD_VERSION env or image tag
+}
+```
+
+### api_gateway implementation
+
+- Add `kube` crate dependency (K8s client library for Rust)
+- New `pods_handler.rs` module: list/watch pods in `foxhunt` namespace via K8s API
+- Extracts `app.kubernetes.io/name` label as service grouping key
+- Streams pod state every 5s (default interval)
+- Pod CPU/memory from metrics-server API (`/apis/metrics.k8s.io/v1beta1/namespaces/foxhunt/pods`)
+
+### TUI cockpit
+
+- New `cockpits/pods.rs` as 7th cockpit (key 7)
+- **Default view**: service-grouped summary rows (`api-gateway: 1/1 Running, v2026.03.408`)
+- **Expanded view**: Tab/Enter on a service row expands to show individual pod rows with name, restarts, age, node, CPU/mem
+- Navigation: Up/Down to select service, Enter to expand/collapse
+
+### NetworkPolicy
+
+- `api-gateway.yaml`: add egress to K8s API (`172.16.0.0/16:6443`) for pod listing
+
+### RBAC
+
+- New Role + RoleBinding granting api_gateway SA `get`/`list`/`watch` on pods in foxhunt namespace
+
+## 3. Cluster Resources Fix
+
+### Root cause
+
+`monitoring_handler.rs:372-375` hardcodes `cpu_percent: 0.0, memory_used_mb: 0.0, memory_total_mb: 0.0` with comment "not yet scraped".
+
+### Fix
+
+Query Prometheus for node-exporter metrics:
+
+- CPU: `sum(rate(node_cpu_seconds_total{mode!="idle"}[2m])) / count(node_cpu_seconds_total{mode="idle"}) * 100`
+- RAM used: `sum(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / 1024 / 1024`
+- RAM total: `sum(node_memory_MemTotal_bytes) / 1024 / 1024`
+
+### Pre-requisite
+
+Prometheus NetworkPolicy needs egress to node IPs on port 9100 (node-exporter). Currently these targets are DOWN due to missing egress rule. Add `ipBlock: 172.16.0.0/16` port 9100 to prometheus NetworkPolicy.
+
+## Dependencies
+
+- Prometheus networking fixes (completed 2026-03-04): Service selector, NetworkPolicy, api_gateway egress
+- node-exporter egress fix (part of this work)
+- `kube` crate added to api_gateway
--- a/infra/k8s/monitoring/prometheus.yaml
+++ b/infra/k8s/monitoring/prometheus.yaml
@@ -0,0 +1,276 @@
+# Standalone Prometheus for foxhunt services
+# Runs on the foxhunt node pool alongside all other services.
+# Replaces dependency on gitlab-prometheus-server (gitlab pool).
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: prometheus
+  namespace: foxhunt
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: foxhunt-prometheus
+rules:
+  - apiGroups: [""]
+    resources: [nodes, nodes/proxy, nodes/metrics, services, endpoints, pods]
+    verbs: [get, list, watch]
+  - apiGroups: [discovery.k8s.io]
+    resources: [endpointslices]
+    verbs: [get, list, watch]
+  - nonResourceURLs: [/metrics]
+    verbs: [get]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: foxhunt-prometheus
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: foxhunt-prometheus
+subjects:
+  - kind: ServiceAccount
+    name: prometheus
+    namespace: foxhunt
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: prometheus-config
+  namespace: foxhunt
+data:
+  prometheus.yml: |
+    global:
+      scrape_interval: 15s
+      evaluation_interval: 15s
+      scrape_timeout: 10s
+
+    scrape_configs:
+      # Self-monitoring
+      - job_name: prometheus
+        static_configs:
+          - targets: ['localhost:9090']
+
+      # Foxhunt services (metrics port)
+      - job_name: foxhunt-services
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_part_of]
+            action: keep
+            regex: foxhunt
+          - source_labels: [__meta_kubernetes_endpoint_port_name]
+            action: keep
+            regex: metrics
+          - source_labels: [__meta_kubernetes_service_name]
+            target_label: service
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+
+      # Pod annotation-based scraping (prometheus.io/scrape: "true")
+      - job_name: annotated-pods
+        kubernetes_sd_configs:
+          - role: pod
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+            action: keep
+            regex: "true"
+          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
+            action: replace
+            regex: (.+)
+            target_label: __metrics_path__
+          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
+            action: replace
+            regex: ([^:]+)(?::\d+)?;(\d+)
+            replacement: $1:$2
+            target_label: __address__
+          - action: labelmap
+            regex: __meta_kubernetes_pod_label_(.+)
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+          - source_labels: [__meta_kubernetes_pod_name]
+            target_label: pod
+
+      # GitLab annotation-based scraping (gitlab.com/prometheus_scrape: "true")
+      - job_name: gitlab-annotated-pods
+        kubernetes_sd_configs:
+          - role: pod
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_pod_annotation_gitlab_com_prometheus_scrape]
+            action: keep
+            regex: "true"
+          - source_labels: [__meta_kubernetes_pod_annotation_gitlab_com_prometheus_path]
+            action: replace
+            regex: (.+)
+            target_label: __metrics_path__
+          - source_labels: [__address__, __meta_kubernetes_pod_annotation_gitlab_com_prometheus_port]
+            action: replace
+            regex: ([^:]+)(?::\d+)?;(\d+)
+            replacement: $1:$2
+            target_label: __address__
+          - action: labelmap
+            regex: __meta_kubernetes_pod_label_(.+)
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+          - source_labels: [__meta_kubernetes_pod_name]
+            target_label: kubernetes_pod_name
+
+      # Node-exporter (host metrics)
+      - job_name: node-exporter
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
+            action: keep
+            regex: node-exporter
+          - source_labels: [__meta_kubernetes_endpoints_name]
+            action: keep
+            regex: node-exporter
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+
+      # kube-state-metrics
+      - job_name: kube-state-metrics
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
+            action: keep
+            regex: kube-state-metrics
+          - source_labels: [__meta_kubernetes_endpoint_port_name]
+            action: keep
+            regex: http
+
+      # GPU metrics (DCGM exporter)
+      - job_name: dcgm-exporter
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [kube-system, monitoring]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_name]
+            action: keep
+            regex: .*dcgm.*
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+
+      # Pushgateway (training job metrics)
+      - job_name: pushgateway
+        honor_labels: true
+        static_configs:
+          - targets: ['pushgateway.foxhunt.svc.cluster.local:9091']
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: prometheus
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: prometheus
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: prometheus
+  strategy:
+    type: Recreate  # single-replica, avoid PVC contention
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: prometheus
+        app.kubernetes.io/part-of: foxhunt
+    spec:
+      serviceAccountName: prometheus
+      nodeSelector:
+        k8s.scaleway.com/pool-name: foxhunt
+      securityContext:
+        fsGroup: 65534
+        runAsNonRoot: true
+        runAsUser: 65534
+      containers:
+        - name: prometheus
+          image: quay.io/prometheus/prometheus:v3.8.1
+          args:
+            - --config.file=/etc/prometheus/prometheus.yml
+            - --storage.tsdb.path=/prometheus
+            - --storage.tsdb.retention.time=14d
+            - --storage.tsdb.retention.size=1500MB
+            - --web.enable-lifecycle
+            - --web.console.libraries=/usr/share/prometheus/console_libraries
+            - --web.console.templates=/usr/share/prometheus/consoles
+          ports:
+            - containerPort: 9090
+              name: http
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: 500m
+              memory: 512Mi
+          volumeMounts:
+            - name: config
+              mountPath: /etc/prometheus
+            - name: data
+              mountPath: /prometheus
+          readinessProbe:
+            httpGet:
+              path: /-/ready
+              port: 9090
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /-/healthy
+              port: 9090
+            initialDelaySeconds: 10
+            periodSeconds: 15
+      volumes:
+        - name: config
+          configMap:
+            name: prometheus-config
+        - name: data
+          persistentVolumeClaim:
+            claimName: prometheus-data
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: prometheus-data
+  namespace: foxhunt
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: scw-bssd
+  resources:
+    requests:
+      storage: 2Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: prometheus
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: prometheus
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    app.kubernetes.io/name: prometheus
+    app.kubernetes.io/part-of: foxhunt
+  ports:
+    - name: http
+      port: 80
+      targetPort: 9090