7c8fa991cc
- ServiceAccount + ClusterRole (nodes, pods, endpoints, metrics) - ConfigMap with 7 scrape jobs: self, foxhunt-services, annotated-pods, gitlab-annotated-pods, node-exporter, kube-state-metrics, dcgm-exporter, pushgateway - Deployment (Prometheus v3.8.1, 14d retention, 1500MB size limit) - PVC (2Gi scw-bssd), Service (port 80 → 9090) Also includes pods-panel-versioning design doc. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
105 lines
3.8 KiB
Markdown
105 lines
3.8 KiB
Markdown
# Pods Panel, CalVer Versioning & Cluster Resources Fix
|
|
|
|
**Date**: 2026-03-04
|
|
**Status**: Approved
|
|
|
|
## 1. CalVer Auto-Versioning
|
|
|
|
**Scheme**: `YYYY.MM.PIPELINE_IID` (e.g. `2026.03.408`)
|
|
|
|
### Flow
|
|
|
|
1. CI compile jobs compute `FOXHUNT_VERSION` from `CI_COMMIT_TIMESTAMP` year.month + `CI_PIPELINE_IID`
|
|
2. Passed to rustc via `FOXHUNT_BUILD_VERSION` env var
|
|
3. `common/src/build_info.rs` exposes `pub fn version() -> &'static str` using `option_env!("FOXHUNT_BUILD_VERSION").unwrap_or(env!("CARGO_PKG_VERSION"))`
|
|
4. Each service populates `ServiceStatus.version` from `build_info::version()`
|
|
5. fxt shows version in TUI header bar and Services cockpit column
|
|
|
|
### Files changed
|
|
|
|
- `.gitlab-ci.yml`: compute + export `FOXHUNT_BUILD_VERSION` in compile jobs (~10 lines)
|
|
- `crates/common/src/build_info.rs`: new module, `version()` function
|
|
- `crates/common/src/lib.rs`: re-export `build_info`
|
|
- `services/api_gateway/src/grpc/monitoring_handler.rs`: populate `version` field from `build_info::version()`
|
|
- All 7 service `main.rs` files: log version at startup
|
|
- `bin/fxt/src/tui/cockpits/overview.rs`: show fxt version in header
|
|
|
|
## 2. Pods Cockpit (Key 7)
|
|
|
|
### Proto
|
|
|
|
New streaming RPC in `monitoring.proto`:
|
|
|
|
```protobuf
|
|
rpc SubscribeClusterPods(SubscribeClusterPodsRequest) returns (stream ClusterPodsResponse);
|
|
|
|
message SubscribeClusterPodsRequest {
|
|
uint32 interval_seconds = 1; // default 5
|
|
}
|
|
|
|
message ClusterPodsResponse {
|
|
repeated PodInfo pods = 1;
|
|
int64 timestamp = 2;
|
|
}
|
|
|
|
message PodInfo {
|
|
string name = 1;
|
|
string service = 2; // app.kubernetes.io/name label
|
|
string status = 3; // Running/Pending/CrashLoopBackOff/Completed/etc
|
|
int32 restarts = 4;
|
|
string age = 5;
|
|
string node = 6;
|
|
float cpu_millicores = 7; // from metrics-server (0 if unavailable)
|
|
float memory_mb = 8; // from metrics-server (0 if unavailable)
|
|
bool ready = 9;
|
|
string version = 10; // from FOXHUNT_BUILD_VERSION env or image tag
|
|
}
|
|
```
|
|
|
|
### api_gateway implementation
|
|
|
|
- Add `kube` crate dependency (K8s client library for Rust)
|
|
- New `pods_handler.rs` module: list/watch pods in `foxhunt` namespace via K8s API
|
|
- Extracts `app.kubernetes.io/name` label as service grouping key
|
|
- Streams pod state every 5s (default interval)
|
|
- Pod CPU/memory from metrics-server API (`/apis/metrics.k8s.io/v1beta1/namespaces/foxhunt/pods`)
|
|
|
|
### TUI cockpit
|
|
|
|
- New `cockpits/pods.rs` as 7th cockpit (key 7)
|
|
- **Default view**: service-grouped summary rows (`api-gateway: 1/1 Running, v2026.03.408`)
|
|
- **Expanded view**: Tab/Enter on a service row expands to show individual pod rows with name, restarts, age, node, CPU/mem
|
|
- Navigation: Up/Down to select service, Enter to expand/collapse
|
|
|
|
### NetworkPolicy
|
|
|
|
- `api-gateway.yaml`: add egress to K8s API (`172.16.0.0/16:6443`) for pod listing
|
|
|
|
### RBAC
|
|
|
|
- New Role + RoleBinding granting api_gateway SA `get`/`list`/`watch` on pods in foxhunt namespace
|
|
|
|
## 3. Cluster Resources Fix
|
|
|
|
### Root cause
|
|
|
|
`monitoring_handler.rs:372-375` hardcodes `cpu_percent: 0.0, memory_used_mb: 0.0, memory_total_mb: 0.0` with comment "not yet scraped".
|
|
|
|
### Fix
|
|
|
|
Query Prometheus for node-exporter metrics:
|
|
|
|
- CPU: `sum(rate(node_cpu_seconds_total{mode!="idle"}[2m])) / count(node_cpu_seconds_total{mode="idle"}) * 100`
|
|
- RAM used: `sum(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / 1024 / 1024`
|
|
- RAM total: `sum(node_memory_MemTotal_bytes) / 1024 / 1024`
|
|
|
|
### Pre-requisite
|
|
|
|
Prometheus NetworkPolicy needs egress to node IPs on port 9100 (node-exporter). Currently these targets are DOWN due to missing egress rule. Add `ipBlock: 172.16.0.0/16` port 9100 to prometheus NetworkPolicy.
|
|
|
|
## Dependencies
|
|
|
|
- Prometheus networking fixes (completed 2026-03-04): Service selector, NetworkPolicy, api_gateway egress
|
|
- node-exporter egress fix (part of this work)
|
|
- `kube` crate added to api_gateway
|