diff --git a/docs/plans/2026-03-04-pods-panel-versioning-design.md b/docs/plans/2026-03-04-pods-panel-versioning-design.md new file mode 100644 index 000000000..595586c2c --- /dev/null +++ b/docs/plans/2026-03-04-pods-panel-versioning-design.md @@ -0,0 +1,104 @@ +# Pods Panel, CalVer Versioning & Cluster Resources Fix + +**Date**: 2026-03-04 +**Status**: Approved + +## 1. CalVer Auto-Versioning + +**Scheme**: `YYYY.MM.PIPELINE_IID` (e.g. `2026.03.408`) + +### Flow + +1. CI compile jobs compute `FOXHUNT_VERSION` from `CI_COMMIT_TIMESTAMP` year.month + `CI_PIPELINE_IID` +2. Passed to rustc via `FOXHUNT_BUILD_VERSION` env var +3. `common/src/build_info.rs` exposes `pub fn version() -> &'static str` using `option_env!("FOXHUNT_BUILD_VERSION").unwrap_or(env!("CARGO_PKG_VERSION"))` +4. Each service populates `ServiceStatus.version` from `build_info::version()` +5. fxt shows version in TUI header bar and Services cockpit column + +### Files changed + +- `.gitlab-ci.yml`: compute + export `FOXHUNT_BUILD_VERSION` in compile jobs (~10 lines) +- `crates/common/src/build_info.rs`: new module, `version()` function +- `crates/common/src/lib.rs`: re-export `build_info` +- `services/api_gateway/src/grpc/monitoring_handler.rs`: populate `version` field from `build_info::version()` +- All 7 service `main.rs` files: log version at startup +- `bin/fxt/src/tui/cockpits/overview.rs`: show fxt version in header + +## 2. Pods Cockpit (Key 7) + +### Proto + +New streaming RPC in `monitoring.proto`: + +```protobuf +rpc SubscribeClusterPods(SubscribeClusterPodsRequest) returns (stream ClusterPodsResponse); + +message SubscribeClusterPodsRequest { + uint32 interval_seconds = 1; // default 5 +} + +message ClusterPodsResponse { + repeated PodInfo pods = 1; + int64 timestamp = 2; +} + +message PodInfo { + string name = 1; + string service = 2; // app.kubernetes.io/name label + string status = 3; // Running/Pending/CrashLoopBackOff/Completed/etc + int32 restarts = 4; + string age = 5; + string node = 6; + float cpu_millicores = 7; // from metrics-server (0 if unavailable) + float memory_mb = 8; // from metrics-server (0 if unavailable) + bool ready = 9; + string version = 10; // from FOXHUNT_BUILD_VERSION env or image tag +} +``` + +### api_gateway implementation + +- Add `kube` crate dependency (K8s client library for Rust) +- New `pods_handler.rs` module: list/watch pods in `foxhunt` namespace via K8s API +- Extracts `app.kubernetes.io/name` label as service grouping key +- Streams pod state every 5s (default interval) +- Pod CPU/memory from metrics-server API (`/apis/metrics.k8s.io/v1beta1/namespaces/foxhunt/pods`) + +### TUI cockpit + +- New `cockpits/pods.rs` as 7th cockpit (key 7) +- **Default view**: service-grouped summary rows (`api-gateway: 1/1 Running, v2026.03.408`) +- **Expanded view**: Tab/Enter on a service row expands to show individual pod rows with name, restarts, age, node, CPU/mem +- Navigation: Up/Down to select service, Enter to expand/collapse + +### NetworkPolicy + +- `api-gateway.yaml`: add egress to K8s API (`172.16.0.0/16:6443`) for pod listing + +### RBAC + +- New Role + RoleBinding granting api_gateway SA `get`/`list`/`watch` on pods in foxhunt namespace + +## 3. Cluster Resources Fix + +### Root cause + +`monitoring_handler.rs:372-375` hardcodes `cpu_percent: 0.0, memory_used_mb: 0.0, memory_total_mb: 0.0` with comment "not yet scraped". + +### Fix + +Query Prometheus for node-exporter metrics: + +- CPU: `sum(rate(node_cpu_seconds_total{mode!="idle"}[2m])) / count(node_cpu_seconds_total{mode="idle"}) * 100` +- RAM used: `sum(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / 1024 / 1024` +- RAM total: `sum(node_memory_MemTotal_bytes) / 1024 / 1024` + +### Pre-requisite + +Prometheus NetworkPolicy needs egress to node IPs on port 9100 (node-exporter). Currently these targets are DOWN due to missing egress rule. Add `ipBlock: 172.16.0.0/16` port 9100 to prometheus NetworkPolicy. + +## Dependencies + +- Prometheus networking fixes (completed 2026-03-04): Service selector, NetworkPolicy, api_gateway egress +- node-exporter egress fix (part of this work) +- `kube` crate added to api_gateway diff --git a/infra/k8s/monitoring/prometheus.yaml b/infra/k8s/monitoring/prometheus.yaml new file mode 100644 index 000000000..4fc425ac2 --- /dev/null +++ b/infra/k8s/monitoring/prometheus.yaml @@ -0,0 +1,276 @@ +# Standalone Prometheus for foxhunt services +# Runs on the foxhunt node pool alongside all other services. +# Replaces dependency on gitlab-prometheus-server (gitlab pool). +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: prometheus + namespace: foxhunt +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: foxhunt-prometheus +rules: + - apiGroups: [""] + resources: [nodes, nodes/proxy, nodes/metrics, services, endpoints, pods] + verbs: [get, list, watch] + - apiGroups: [discovery.k8s.io] + resources: [endpointslices] + verbs: [get, list, watch] + - nonResourceURLs: [/metrics] + verbs: [get] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: foxhunt-prometheus +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: foxhunt-prometheus +subjects: + - kind: ServiceAccount + name: prometheus + namespace: foxhunt +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config + namespace: foxhunt +data: + prometheus.yml: | + global: + scrape_interval: 15s + evaluation_interval: 15s + scrape_timeout: 10s + + scrape_configs: + # Self-monitoring + - job_name: prometheus + static_configs: + - targets: ['localhost:9090'] + + # Foxhunt services (metrics port) + - job_name: foxhunt-services + kubernetes_sd_configs: + - role: endpoints + namespaces: + names: [foxhunt] + relabel_configs: + - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_part_of] + action: keep + regex: foxhunt + - source_labels: [__meta_kubernetes_endpoint_port_name] + action: keep + regex: metrics + - source_labels: [__meta_kubernetes_service_name] + target_label: service + - source_labels: [__meta_kubernetes_namespace] + target_label: namespace + + # Pod annotation-based scraping (prometheus.io/scrape: "true") + - job_name: annotated-pods + kubernetes_sd_configs: + - role: pod + namespaces: + names: [foxhunt] + relabel_configs: + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] + action: keep + regex: "true" + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] + action: replace + regex: (.+) + target_label: __metrics_path__ + - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] + action: replace + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + target_label: __address__ + - action: labelmap + regex: __meta_kubernetes_pod_label_(.+) + - source_labels: [__meta_kubernetes_namespace] + target_label: namespace + - source_labels: [__meta_kubernetes_pod_name] + target_label: pod + + # GitLab annotation-based scraping (gitlab.com/prometheus_scrape: "true") + - job_name: gitlab-annotated-pods + kubernetes_sd_configs: + - role: pod + namespaces: + names: [foxhunt] + relabel_configs: + - source_labels: [__meta_kubernetes_pod_annotation_gitlab_com_prometheus_scrape] + action: keep + regex: "true" + - source_labels: [__meta_kubernetes_pod_annotation_gitlab_com_prometheus_path] + action: replace + regex: (.+) + target_label: __metrics_path__ + - source_labels: [__address__, __meta_kubernetes_pod_annotation_gitlab_com_prometheus_port] + action: replace + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + target_label: __address__ + - action: labelmap + regex: __meta_kubernetes_pod_label_(.+) + - source_labels: [__meta_kubernetes_namespace] + target_label: namespace + - source_labels: [__meta_kubernetes_pod_name] + target_label: kubernetes_pod_name + + # Node-exporter (host metrics) + - job_name: node-exporter + kubernetes_sd_configs: + - role: endpoints + namespaces: + names: [foxhunt] + relabel_configs: + - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name] + action: keep + regex: node-exporter + - source_labels: [__meta_kubernetes_endpoints_name] + action: keep + regex: node-exporter + - source_labels: [__meta_kubernetes_pod_node_name] + target_label: node + + # kube-state-metrics + - job_name: kube-state-metrics + kubernetes_sd_configs: + - role: endpoints + namespaces: + names: [foxhunt] + relabel_configs: + - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name] + action: keep + regex: kube-state-metrics + - source_labels: [__meta_kubernetes_endpoint_port_name] + action: keep + regex: http + + # GPU metrics (DCGM exporter) + - job_name: dcgm-exporter + kubernetes_sd_configs: + - role: endpoints + namespaces: + names: [kube-system, monitoring] + relabel_configs: + - source_labels: [__meta_kubernetes_service_name] + action: keep + regex: .*dcgm.* + - source_labels: [__meta_kubernetes_pod_node_name] + target_label: node + + # Pushgateway (training job metrics) + - job_name: pushgateway + honor_labels: true + static_configs: + - targets: ['pushgateway.foxhunt.svc.cluster.local:9091'] +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: prometheus + namespace: foxhunt + labels: + app.kubernetes.io/name: prometheus + app.kubernetes.io/part-of: foxhunt +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: prometheus + strategy: + type: Recreate # single-replica, avoid PVC contention + template: + metadata: + labels: + app.kubernetes.io/name: prometheus + app.kubernetes.io/part-of: foxhunt + spec: + serviceAccountName: prometheus + nodeSelector: + k8s.scaleway.com/pool-name: foxhunt + securityContext: + fsGroup: 65534 + runAsNonRoot: true + runAsUser: 65534 + containers: + - name: prometheus + image: quay.io/prometheus/prometheus:v3.8.1 + args: + - --config.file=/etc/prometheus/prometheus.yml + - --storage.tsdb.path=/prometheus + - --storage.tsdb.retention.time=14d + - --storage.tsdb.retention.size=1500MB + - --web.enable-lifecycle + - --web.console.libraries=/usr/share/prometheus/console_libraries + - --web.console.templates=/usr/share/prometheus/consoles + ports: + - containerPort: 9090 + name: http + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi + volumeMounts: + - name: config + mountPath: /etc/prometheus + - name: data + mountPath: /prometheus + readinessProbe: + httpGet: + path: /-/ready + port: 9090 + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: + path: /-/healthy + port: 9090 + initialDelaySeconds: 10 + periodSeconds: 15 + volumes: + - name: config + configMap: + name: prometheus-config + - name: data + persistentVolumeClaim: + claimName: prometheus-data +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: prometheus-data + namespace: foxhunt +spec: + accessModes: [ReadWriteOnce] + storageClassName: scw-bssd + resources: + requests: + storage: 2Gi +--- +apiVersion: v1 +kind: Service +metadata: + name: prometheus + namespace: foxhunt + labels: + app.kubernetes.io/name: prometheus + app.kubernetes.io/part-of: foxhunt +spec: + selector: + app.kubernetes.io/name: prometheus + app.kubernetes.io/part-of: foxhunt + ports: + - name: http + port: 80 + targetPort: 9090