947 lines
27 KiB
Markdown
947 lines
27 KiB
Markdown
# Pods Panel, CalVer Versioning & Monitoring Fixes — Implementation Plan
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Add CalVer auto-versioning to all builds, a 7th TUI cockpit showing K8s pod status, and fix cluster resource monitoring.
|
|
|
|
**Architecture:** CI computes `YYYY.MM.PIPELINE_IID` and passes it via env var. Each binary reads it at compile time via `option_env!()`. A new gRPC streaming RPC in api_gateway uses the `kube` crate to list/watch pods. The TUI gets a 7th cockpit (key 7) rendering pod state by service group.
|
|
|
|
**Tech Stack:** Rust (workspace), tonic/prost (gRPC), ratatui (TUI), kube-rs (K8s client), GitLab CI, Prometheus
|
|
|
|
---
|
|
|
|
## Task 1: CalVer Build Version — `common/src/build_info.rs` ✅ (if done) / TODO
|
|
|
|
**Files:**
|
|
- Create: `crates/common/src/build_info.rs`
|
|
- Modify: `crates/common/src/lib.rs`
|
|
|
|
**Step 1: Create `build_info.rs`**
|
|
|
|
```rust
|
|
// crates/common/src/build_info.rs
|
|
//! Build-time version information.
|
|
//!
|
|
//! CI sets `FOXHUNT_BUILD_VERSION` (e.g. `2026.03.408`).
|
|
//! Local dev falls back to `CARGO_PKG_VERSION`.
|
|
|
|
/// Build version string. Set at compile time.
|
|
///
|
|
/// CI: `FOXHUNT_BUILD_VERSION=2026.03.408`
|
|
/// Local: falls back to Cargo.toml `version` field.
|
|
pub fn version() -> &'static str {
|
|
option_env!("FOXHUNT_BUILD_VERSION").unwrap_or(env!("CARGO_PKG_VERSION"))
|
|
}
|
|
```
|
|
|
|
**Step 2: Register module in `lib.rs`**
|
|
|
|
In `crates/common/src/lib.rs`, add after the `pub mod tls;` line:
|
|
|
|
```rust
|
|
pub mod build_info;
|
|
```
|
|
|
|
**Step 3: Verify it compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p common`
|
|
Expected: compiles, `version()` returns "1.0.0" (Cargo.toml workspace version)
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add crates/common/src/build_info.rs crates/common/src/lib.rs
|
|
git commit -m "feat(common): add build_info module with CalVer version function"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 2: CI Version Computation — `.gitlab-ci.yml`
|
|
|
|
**Files:**
|
|
- Modify: `.gitlab-ci.yml` (global variables + compile job scripts)
|
|
|
|
**Step 1: Add version computation to global variables**
|
|
|
|
In the `variables:` block (after line 59), add:
|
|
|
|
```yaml
|
|
# CalVer: YYYY.MM.<pipeline_iid> — monotonically increasing per pipeline
|
|
# Computed at job start; override in pipeline vars to pin a version.
|
|
FOXHUNT_BUILD_VERSION: "${CI_COMMIT_TIMESTAMP:0:4}.${CI_COMMIT_TIMESTAMP:5:2}.${CI_PIPELINE_IID}"
|
|
```
|
|
|
|
**Note:** GitLab CE doesn't support bash substring in variables. We must compute it in the script section instead.
|
|
|
|
**Step 1 (revised): Add version computation to compile scripts**
|
|
|
|
In `compile-services` script section, add before `cargo build`:
|
|
|
|
```yaml
|
|
- |
|
|
# CalVer: YYYY.MM.<pipeline_iid>
|
|
YEAR=$(date -d "$CI_COMMIT_TIMESTAMP" +%Y 2>/dev/null || date +%Y)
|
|
MONTH=$(date -d "$CI_COMMIT_TIMESTAMP" +%m 2>/dev/null || date +%m)
|
|
export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.${CI_PIPELINE_IID:-0}"
|
|
echo "Build version: $FOXHUNT_BUILD_VERSION"
|
|
```
|
|
|
|
Add the same block to `compile-training` script section, before `cargo build`.
|
|
|
|
Also add to `compile-fxt` (if it exists as a separate job, or it's part of compile-services).
|
|
|
|
**Step 2: Verify locally**
|
|
|
|
Run: `FOXHUNT_BUILD_VERSION=2026.03.999 SQLX_OFFLINE=true cargo check -p common`
|
|
Expected: compiles. `version()` would return "2026.03.999" at runtime.
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add .gitlab-ci.yml
|
|
git commit -m "feat(ci): compute CalVer FOXHUNT_BUILD_VERSION in compile jobs"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 3: Populate `ServiceStatus.version` in api_gateway
|
|
|
|
**Files:**
|
|
- Modify: `services/api_gateway/src/grpc/monitoring_handler.rs`
|
|
- Dependency: `common::build_info::version()`
|
|
|
|
**Step 1: Find `check_backend_health` or `ServiceStatus` construction**
|
|
|
|
In `monitoring_handler.rs`, find where `ServiceStatus` is built (around line 280-306). The `version` field should be populated.
|
|
|
|
Look for the pattern:
|
|
```rust
|
|
version: None, // or ..Default::default()
|
|
```
|
|
|
|
Replace with:
|
|
```rust
|
|
version: Some(common::build_info::version().to_owned()),
|
|
```
|
|
|
|
This only sets the api_gateway's own version. Each downstream service would need to expose its version in its health response. For now, api_gateway reports its own version for all discovered services (they're all built from the same commit).
|
|
|
|
**Step 2: Verify it compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p api_gateway`
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add services/api_gateway/src/grpc/monitoring_handler.rs
|
|
git commit -m "feat(api_gateway): populate ServiceStatus.version from build_info"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 4: Cluster Resources Fix (DONE)
|
|
|
|
> Already completed during monitoring debugging session.
|
|
|
|
**Files changed:**
|
|
- `services/api_gateway/src/grpc/monitoring_handler.rs`: Added `query_cluster_resources()` method, wired into `build_response()` replacing hardcoded zeros.
|
|
- `infra/k8s/network-policies/prometheus.yaml`: Added port 9100 to `172.16.0.0/16` ipBlock, port 8080 to foxhunt podSelector, port 9400 to kube-system namespace.
|
|
- `infra/k8s/network-policies/prometheus-cilium.yaml` (NEW): CiliumNetworkPolicy for Prometheus → hostNetwork pods (node-exporter).
|
|
|
|
**Verify: Already verified** — Prometheus returns real CPU/RAM/GPU values. api_gateway monitoring tests pass (9/9).
|
|
|
|
**Commit:**
|
|
|
|
```bash
|
|
git add services/api_gateway/src/grpc/monitoring_handler.rs
|
|
git add infra/k8s/network-policies/prometheus.yaml
|
|
git add infra/k8s/network-policies/prometheus-cilium.yaml
|
|
git commit -m "fix(monitoring): wire cluster resources, fix Prometheus networking"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 5: Proto — `SubscribeClusterPods` RPC
|
|
|
|
**Files:**
|
|
- Modify: `proto/monitoring.proto`
|
|
|
|
**Step 1: Add messages and RPC**
|
|
|
|
After the `GetEpochHistory` RPC (line 54), add:
|
|
|
|
```protobuf
|
|
// Kubernetes pod status — streams every N seconds
|
|
rpc SubscribeClusterPods(SubscribeClusterPodsRequest) returns (stream ClusterPodsResponse);
|
|
```
|
|
|
|
After `GetEpochHistoryResponse` (line 425), add:
|
|
|
|
```protobuf
|
|
// ============================================================================
|
|
// Cluster Pod Messages
|
|
// ============================================================================
|
|
|
|
message SubscribeClusterPodsRequest {
|
|
uint32 interval_seconds = 1; // 0 = server default (5s)
|
|
}
|
|
|
|
message ClusterPodsResponse {
|
|
repeated PodInfo pods = 1;
|
|
int64 timestamp = 2;
|
|
}
|
|
|
|
message PodInfo {
|
|
string name = 1;
|
|
string service = 2; // app.kubernetes.io/name label
|
|
string namespace = 3;
|
|
string status = 4; // Running/Pending/CrashLoopBackOff/Completed/etc
|
|
int32 restarts = 5;
|
|
string age = 6; // human-readable: "2d5h", "3m"
|
|
string node = 7;
|
|
bool ready = 8;
|
|
string version = 9; // from FOXHUNT_BUILD_VERSION env or empty
|
|
}
|
|
```
|
|
|
|
**Step 2: Verify proto compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p api_gateway`
|
|
Expected: compile error for unimplemented `subscribe_cluster_pods` — that's expected, we implement it next.
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add proto/monitoring.proto
|
|
git commit -m "feat(proto): add SubscribeClusterPods streaming RPC"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 6: api_gateway — `kube` crate + pods handler
|
|
|
|
**Files:**
|
|
- Modify: `services/api_gateway/Cargo.toml`
|
|
- Create: `services/api_gateway/src/grpc/pods_handler.rs`
|
|
- Modify: `services/api_gateway/src/grpc/monitoring_handler.rs` (implement the new RPC)
|
|
|
|
**Step 1: Add `kube` dependency**
|
|
|
|
In `services/api_gateway/Cargo.toml`, add after the `redis` dependency:
|
|
|
|
```toml
|
|
# Kubernetes API client for pod listing
|
|
kube = { version = "0.99", features = ["client", "runtime", "derive"] }
|
|
k8s-openapi = { version = "0.24", features = ["latest"] }
|
|
```
|
|
|
|
**Step 2: Create pods_handler.rs**
|
|
|
|
```rust
|
|
// services/api_gateway/src/grpc/pods_handler.rs
|
|
//! Kubernetes pod listing for the Pods cockpit.
|
|
|
|
use anyhow::Result;
|
|
use k8s_openapi::api::core::v1::Pod;
|
|
use kube::{Api, Client};
|
|
|
|
/// Fetch all pods in a namespace and convert to proto `PodInfo`.
|
|
pub async fn list_pods(namespace: &str) -> Result<Vec<super::monitoring::PodInfo>> {
|
|
let client = Client::try_default().await?;
|
|
let pods: Api<Pod> = Api::namespaced(client, namespace);
|
|
let pod_list = pods.list(&Default::default()).await?;
|
|
|
|
let mut infos = Vec::with_capacity(pod_list.items.len());
|
|
for pod in pod_list.items {
|
|
let meta = &pod.metadata;
|
|
let spec = pod.spec.as_ref();
|
|
let status = pod.status.as_ref();
|
|
|
|
let name = meta.name.clone().unwrap_or_default();
|
|
let labels = meta.labels.as_ref();
|
|
let service = labels
|
|
.and_then(|l| l.get("app.kubernetes.io/name"))
|
|
.cloned()
|
|
.unwrap_or_default();
|
|
|
|
let phase = status
|
|
.and_then(|s| s.phase.clone())
|
|
.unwrap_or_else(|| "Unknown".into());
|
|
|
|
let ready = status
|
|
.and_then(|s| s.conditions.as_ref())
|
|
.map(|conds| conds.iter().any(|c| c.type_ == "Ready" && c.status == "True"))
|
|
.unwrap_or(false);
|
|
|
|
let restarts: i32 = status
|
|
.and_then(|s| s.container_statuses.as_ref())
|
|
.map(|cs| cs.iter().map(|c| c.restart_count).sum())
|
|
.unwrap_or(0);
|
|
|
|
let node = spec
|
|
.and_then(|s| s.node_name.clone())
|
|
.unwrap_or_default();
|
|
|
|
let creation = meta.creation_timestamp.as_ref().map(|t| t.0);
|
|
let age = creation
|
|
.map(|t| format_age(t))
|
|
.unwrap_or_else(|| "?".into());
|
|
|
|
// Try to get version from FOXHUNT_BUILD_VERSION env var in first container
|
|
let version = spec
|
|
.and_then(|s| s.containers.first())
|
|
.and_then(|c| c.env.as_ref())
|
|
.and_then(|envs| {
|
|
envs.iter()
|
|
.find(|e| e.name == "FOXHUNT_BUILD_VERSION")
|
|
.and_then(|e| e.value.clone())
|
|
})
|
|
.unwrap_or_default();
|
|
|
|
infos.push(super::monitoring::PodInfo {
|
|
name,
|
|
service,
|
|
namespace: namespace.into(),
|
|
status: phase,
|
|
restarts,
|
|
age,
|
|
node: short_node_name(&node),
|
|
ready,
|
|
version,
|
|
});
|
|
}
|
|
|
|
// Sort by service name, then pod name
|
|
infos.sort_by(|a, b| (&a.service, &a.name).cmp(&(&b.service, &b.name)));
|
|
Ok(infos)
|
|
}
|
|
|
|
fn short_node_name(full: &str) -> String {
|
|
// scw-foxhunt-foxhunt-e4027b8e... → foxhunt
|
|
// scw-foxhunt-ci-training-h100-... → ci-training-h100
|
|
full.strip_prefix("scw-foxhunt-")
|
|
.unwrap_or(full)
|
|
.split('-')
|
|
.take(3)
|
|
.collect::<Vec<_>>()
|
|
.join("-")
|
|
.chars()
|
|
.take(20)
|
|
.collect()
|
|
}
|
|
|
|
fn format_age(created: chrono::DateTime<chrono::Utc>) -> String {
|
|
let dur = chrono::Utc::now() - created;
|
|
let secs = dur.num_seconds();
|
|
if secs < 60 {
|
|
format!("{secs}s")
|
|
} else if secs < 3600 {
|
|
format!("{}m", secs / 60)
|
|
} else if secs < 86400 {
|
|
format!("{}h{}m", secs / 3600, (secs % 3600) / 60)
|
|
} else {
|
|
format!("{}d{}h", secs / 86400, (secs % 86400) / 3600)
|
|
}
|
|
}
|
|
```
|
|
|
|
**Step 3: Wire into monitoring_handler.rs**
|
|
|
|
Add the `subscribe_cluster_pods` implementation to the `MonitoringService` impl block. This follows the same streaming pattern as `stream_training_metrics`:
|
|
|
|
```rust
|
|
type SubscribeClusterPodsStream =
|
|
Pin<Box<dyn Stream<Item = Result<ClusterPodsResponse, Status>> + Send>>;
|
|
|
|
async fn subscribe_cluster_pods(
|
|
&self,
|
|
request: Request<SubscribeClusterPodsRequest>,
|
|
) -> Result<Response<Self::SubscribeClusterPodsStream>, Status> {
|
|
let req = request.into_inner();
|
|
let interval_secs = if req.interval_seconds == 0 { 5 } else { req.interval_seconds.clamp(1, 60) };
|
|
|
|
let stream = async_stream::try_stream! {
|
|
let mut interval = tokio::time::interval(Duration::from_secs(u64::from(interval_secs)));
|
|
loop {
|
|
interval.tick().await;
|
|
match pods_handler::list_pods("foxhunt").await {
|
|
Ok(pods) => {
|
|
yield ClusterPodsResponse {
|
|
pods,
|
|
timestamp: chrono::Utc::now().timestamp(),
|
|
};
|
|
}
|
|
Err(e) => {
|
|
tracing::warn!("Pod listing failed: {e}");
|
|
yield ClusterPodsResponse {
|
|
pods: vec![],
|
|
timestamp: chrono::Utc::now().timestamp(),
|
|
};
|
|
}
|
|
}
|
|
}
|
|
};
|
|
|
|
Ok(Response::new(Box::pin(stream)))
|
|
}
|
|
```
|
|
|
|
**Step 4: Verify it compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p api_gateway`
|
|
|
|
**Step 5: Commit**
|
|
|
|
```bash
|
|
git add services/api_gateway/Cargo.toml
|
|
git add services/api_gateway/src/grpc/pods_handler.rs
|
|
git add services/api_gateway/src/grpc/monitoring_handler.rs
|
|
git commit -m "feat(api_gateway): implement SubscribeClusterPods with kube crate"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 7: RBAC for api_gateway pod listing
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/api-gateway.yaml` (or create separate RBAC file)
|
|
|
|
**Step 1: Add Role + RoleBinding**
|
|
|
|
Create or append to the api_gateway K8s resources:
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: Role
|
|
metadata:
|
|
name: api-gateway-pod-reader
|
|
namespace: foxhunt
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: [pods]
|
|
verbs: [get, list, watch]
|
|
- apiGroups: [metrics.k8s.io]
|
|
resources: [pods]
|
|
verbs: [get, list]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: api-gateway-pod-reader
|
|
namespace: foxhunt
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: api-gateway-pod-reader
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: api-gateway
|
|
namespace: foxhunt
|
|
```
|
|
|
|
**Step 2: Apply**
|
|
|
|
Run: `kubectl apply -f infra/k8s/services/api-gateway.yaml`
|
|
|
|
**Step 3: Verify**
|
|
|
|
Run: `kubectl auth can-i list pods -n foxhunt --as=system:serviceaccount:foxhunt:api-gateway`
|
|
Expected: `yes`
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add infra/k8s/services/api-gateway.yaml
|
|
git commit -m "feat(infra): RBAC for api_gateway pod listing"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 8: api_gateway NetworkPolicy — K8s API egress
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/network-policies/api-gateway.yaml`
|
|
|
|
**Step 1: Add K8s API egress**
|
|
|
|
Add egress rule to allow api_gateway → K8s API for pod listing:
|
|
|
|
```yaml
|
|
# Kubernetes API (pod listing for Pods cockpit)
|
|
- to:
|
|
- ipBlock:
|
|
cidr: 10.32.0.0/12
|
|
ports:
|
|
- protocol: TCP
|
|
port: 443
|
|
- to:
|
|
- ipBlock:
|
|
cidr: 172.16.0.0/16
|
|
ports:
|
|
- protocol: TCP
|
|
port: 6443
|
|
```
|
|
|
|
**Step 2: Apply and verify**
|
|
|
|
Run: `kubectl apply -f infra/k8s/network-policies/api-gateway.yaml`
|
|
Then: `kubectl exec deploy/api-gateway -n foxhunt -- wget -qO- --timeout=3 https://kubernetes.default.svc/api/v1/namespaces/foxhunt/pods 2>&1 | head -5`
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add infra/k8s/network-policies/api-gateway.yaml
|
|
git commit -m "feat(infra): api-gateway K8s API egress for pod listing"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 9: TUI — PodData structs + StateUpdate variant
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/tui/state.rs`
|
|
- Modify: `bin/fxt/src/tui/data_fetcher.rs`
|
|
- Modify: `bin/fxt/src/tui/event_loop.rs`
|
|
|
|
**Step 1: Add pod data structs to `state.rs`**
|
|
|
|
After `AlertData` struct (around line 312):
|
|
|
|
```rust
|
|
/// A single K8s pod.
|
|
pub struct PodData {
|
|
pub name: String,
|
|
pub service: String,
|
|
pub status: String,
|
|
pub restarts: i32,
|
|
pub age: String,
|
|
pub node: String,
|
|
pub ready: bool,
|
|
pub version: String,
|
|
}
|
|
|
|
/// Pods grouped by service.
|
|
pub struct PodGroupData {
|
|
pub service: String,
|
|
pub pods: Vec<PodData>,
|
|
pub ready_count: usize,
|
|
pub total_count: usize,
|
|
}
|
|
```
|
|
|
|
Add `pods: Vec<PodGroupData>` field to `AppState` struct, and `pods: Vec::new()` to its Default impl. Also add `selected_pod_group: Option<usize>` and `pod_group_expanded: bool` for navigation.
|
|
|
|
**Step 2: Add `StateUpdate::Pods` variant to `data_fetcher.rs`**
|
|
|
|
```rust
|
|
/// Cluster pod statuses (from SubscribeClusterPods).
|
|
Pods(Vec<PodGroupData>),
|
|
```
|
|
|
|
**Step 3: Handle in `apply_update` in `event_loop.rs`**
|
|
|
|
```rust
|
|
StateUpdate::Pods(pods) => {
|
|
state.pods = pods;
|
|
}
|
|
```
|
|
|
|
**Step 4: Verify it compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
|
|
**Step 5: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/tui/state.rs bin/fxt/src/tui/data_fetcher.rs bin/fxt/src/tui/event_loop.rs
|
|
git commit -m "feat(fxt): add PodData/PodGroupData structs and StateUpdate::Pods"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 10: TUI — `spawn_pods_stream` in data_fetcher
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/tui/data_fetcher.rs`
|
|
|
|
**Step 1: Add `spawn_pods_stream` method**
|
|
|
|
Follow the same pattern as `spawn_training_metrics`. The stream subscribes to `SubscribeClusterPods` and groups pods by service:
|
|
|
|
```rust
|
|
fn spawn_pods_stream(&self, client: &FoxhuntClient) {
|
|
let mut mon = client.monitoring();
|
|
let tx = self.tx.clone();
|
|
let cancel = self.cancel.clone();
|
|
|
|
tokio::spawn(async move {
|
|
let mut backoff = Backoff::new();
|
|
loop {
|
|
let req = Request::new(monitoring::SubscribeClusterPodsRequest {
|
|
interval_seconds: 5,
|
|
});
|
|
match mon.subscribe_cluster_pods(req).await {
|
|
Ok(resp) => {
|
|
backoff.reset();
|
|
let mut stream = resp.into_inner();
|
|
loop {
|
|
tokio::select! {
|
|
_ = cancel.cancelled() => return,
|
|
msg = stream.message() => {
|
|
match msg {
|
|
Ok(Some(m)) => {
|
|
let groups = group_pods(m.pods);
|
|
let _ = tx.send(StateUpdate::Pods(groups));
|
|
}
|
|
Ok(None) => break,
|
|
Err(e) => {
|
|
warn!("Pods stream error: {e}");
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
Err(e) => {
|
|
warn!("Pods stream connect failed: {e}");
|
|
}
|
|
}
|
|
if backoff.attempts >= MAX_RECONNECT_ATTEMPTS { return; }
|
|
tokio::select! {
|
|
_ = cancel.cancelled() => return,
|
|
_ = tokio::time::sleep(backoff.next_delay()) => {}
|
|
}
|
|
}
|
|
});
|
|
}
|
|
```
|
|
|
|
Add a `group_pods` helper:
|
|
|
|
```rust
|
|
fn group_pods(pods: Vec<monitoring::PodInfo>) -> Vec<PodGroupData> {
|
|
use std::collections::BTreeMap;
|
|
let mut groups: BTreeMap<String, Vec<PodData>> = BTreeMap::new();
|
|
for p in pods {
|
|
let svc = if p.service.is_empty() { "(other)".into() } else { p.service.clone() };
|
|
groups.entry(svc).or_default().push(PodData {
|
|
name: p.name,
|
|
service: p.service,
|
|
status: p.status,
|
|
restarts: p.restarts,
|
|
age: p.age,
|
|
node: p.node,
|
|
ready: p.ready,
|
|
version: p.version,
|
|
});
|
|
}
|
|
groups.into_iter().map(|(service, pods)| {
|
|
let ready_count = pods.iter().filter(|p| p.ready).count();
|
|
let total_count = pods.len();
|
|
PodGroupData { service, pods, ready_count, total_count }
|
|
}).collect()
|
|
}
|
|
```
|
|
|
|
**Step 2: Register in `start()` method**
|
|
|
|
Add `self.spawn_pods_stream(client);` to the `start()` method.
|
|
|
|
**Step 3: Verify it compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/tui/data_fetcher.rs
|
|
git commit -m "feat(fxt): add spawn_pods_stream with service grouping"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 11: TUI — Pods Cockpit
|
|
|
|
**Files:**
|
|
- Create: `bin/fxt/src/tui/cockpits/pods.rs`
|
|
- Modify: `bin/fxt/src/tui/cockpits/mod.rs`
|
|
|
|
**Step 1: Create `pods.rs`**
|
|
|
|
```rust
|
|
//! Pods cockpit — Kubernetes pod status grouped by service.
|
|
|
|
use ratatui::layout::{Constraint, Direction, Layout, Rect};
|
|
use ratatui::style::{Modifier, Style};
|
|
use ratatui::text::{Line, Span};
|
|
use ratatui::widgets::{Block, Borders, Paragraph, Row, Table};
|
|
|
|
use super::super::state::AppState;
|
|
use super::super::theme;
|
|
use super::Cockpit;
|
|
|
|
pub struct PodsCockpit;
|
|
|
|
impl Cockpit for PodsCockpit {
|
|
fn name(&self) -> &str {
|
|
"Pods"
|
|
}
|
|
|
|
fn render(&self, frame: &mut ratatui::Frame, area: Rect, state: &AppState) {
|
|
let block = Block::default()
|
|
.borders(Borders::ALL)
|
|
.title(" Cluster Pods ")
|
|
.title_style(theme::title_style())
|
|
.border_style(Style::default().fg(theme::BORDER));
|
|
|
|
if state.pods.is_empty() {
|
|
let msg = Paragraph::new("Waiting for pod data...")
|
|
.style(theme::muted_style())
|
|
.block(block);
|
|
frame.render_widget(msg, area);
|
|
return;
|
|
}
|
|
|
|
let header = Row::new(vec![
|
|
"Service", "Ready", "Status", "Version", "Restarts", "Age", "Node",
|
|
])
|
|
.style(Style::default().fg(theme::SECONDARY).add_modifier(Modifier::BOLD));
|
|
|
|
let mut rows = Vec::new();
|
|
for group in &state.pods {
|
|
// Summary row for the service group
|
|
let ready_str = format!("{}/{}", group.ready_count, group.total_count);
|
|
let status = if group.ready_count == group.total_count {
|
|
Span::styled("Running", Style::default().fg(theme::SUCCESS))
|
|
} else if group.ready_count == 0 {
|
|
Span::styled("Down", Style::default().fg(theme::ERROR))
|
|
} else {
|
|
Span::styled("Degraded", Style::default().fg(theme::WARNING))
|
|
};
|
|
|
|
let version = group.pods.first()
|
|
.map(|p| p.version.as_str())
|
|
.unwrap_or("");
|
|
|
|
let total_restarts: i32 = group.pods.iter().map(|p| p.restarts).sum();
|
|
let restart_style = if total_restarts > 0 {
|
|
Style::default().fg(theme::WARNING)
|
|
} else {
|
|
theme::muted_style()
|
|
};
|
|
|
|
rows.push(Row::new(vec![
|
|
Line::from(Span::styled(
|
|
&group.service,
|
|
Style::default().fg(theme::PRIMARY).add_modifier(Modifier::BOLD),
|
|
)),
|
|
Line::from(ready_str),
|
|
Line::from(status),
|
|
Line::from(Span::styled(version, theme::muted_style())),
|
|
Line::from(Span::styled(
|
|
format!("{total_restarts}"),
|
|
restart_style,
|
|
)),
|
|
Line::from(Span::styled(
|
|
group.pods.first().map(|p| p.age.as_str()).unwrap_or(""),
|
|
theme::muted_style(),
|
|
)),
|
|
Line::from(Span::styled(
|
|
group.pods.first().map(|p| p.node.as_str()).unwrap_or(""),
|
|
theme::muted_style(),
|
|
)),
|
|
]));
|
|
|
|
// Expanded: show individual pods if more than 1 in group
|
|
if group.pods.len() > 1 {
|
|
for pod in &group.pods {
|
|
let pod_status_style = if pod.ready {
|
|
Style::default().fg(theme::SUCCESS)
|
|
} else {
|
|
Style::default().fg(theme::ERROR)
|
|
};
|
|
let pod_restart_style = if pod.restarts > 0 {
|
|
Style::default().fg(theme::WARNING)
|
|
} else {
|
|
theme::muted_style()
|
|
};
|
|
rows.push(Row::new(vec![
|
|
Line::from(Span::styled(
|
|
format!(" {}", short_pod_name(&pod.name)),
|
|
theme::muted_style(),
|
|
)),
|
|
Line::from(if pod.ready { "✓" } else { "✗" }),
|
|
Line::from(Span::styled(&pod.status, pod_status_style)),
|
|
Line::from(""),
|
|
Line::from(Span::styled(
|
|
format!("{}", pod.restarts),
|
|
pod_restart_style,
|
|
)),
|
|
Line::from(Span::styled(&pod.age, theme::muted_style())),
|
|
Line::from(Span::styled(&pod.node, theme::muted_style())),
|
|
]));
|
|
}
|
|
}
|
|
}
|
|
|
|
let widths = [
|
|
Constraint::Length(28), // Service
|
|
Constraint::Length(7), // Ready
|
|
Constraint::Length(12), // Status
|
|
Constraint::Length(14), // Version
|
|
Constraint::Length(9), // Restarts
|
|
Constraint::Length(8), // Age
|
|
Constraint::Fill(1), // Node
|
|
];
|
|
|
|
let table = Table::new(rows, widths)
|
|
.header(header)
|
|
.block(block);
|
|
|
|
frame.render_widget(table, area);
|
|
}
|
|
}
|
|
|
|
/// Shorten pod names: `api-gateway-7686d84bdd-zl4mn` → `zl4mn`
|
|
fn short_pod_name(name: &str) -> &str {
|
|
name.rsplit('-').next().unwrap_or(name)
|
|
}
|
|
```
|
|
|
|
**Step 2: Register in `cockpits/mod.rs`**
|
|
|
|
```rust
|
|
pub mod pods;
|
|
```
|
|
|
|
**Step 3: Verify it compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/tui/cockpits/pods.rs bin/fxt/src/tui/cockpits/mod.rs
|
|
git commit -m "feat(fxt): add Pods cockpit with service-grouped table"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 12: TUI — Wire Cockpit 7 into Event Loop
|
|
|
|
**Files:**
|
|
- Modify: `bin/fxt/src/tui/event_loop.rs`
|
|
|
|
**Step 1: Update COCKPIT_COUNT**
|
|
|
|
```rust
|
|
const COCKPIT_COUNT: usize = 7;
|
|
```
|
|
|
|
**Step 2: Add PodsCockpit to cockpits vec**
|
|
|
|
```rust
|
|
use super::cockpits::pods::PodsCockpit;
|
|
|
|
// In event_loop():
|
|
let cockpits: Vec<Box<dyn Cockpit>> = vec![
|
|
Box::new(OverviewCockpit),
|
|
Box::new(TrainingCockpit),
|
|
Box::new(TradingCockpit),
|
|
Box::new(ServicesCockpit),
|
|
Box::new(RiskCockpit),
|
|
Box::new(DataCockpit),
|
|
Box::new(PodsCockpit),
|
|
];
|
|
```
|
|
|
|
**Step 3: Update key handler range**
|
|
|
|
Change:
|
|
```rust
|
|
KeyCode::Char(c @ '1'..='6') => {
|
|
```
|
|
To:
|
|
```rust
|
|
KeyCode::Char(c @ '1'..='7') => {
|
|
```
|
|
|
|
**Step 4: Update status bar**
|
|
|
|
In `render_status_bar`, change `"1-6"` to `"1-7"`:
|
|
```rust
|
|
Span::styled("1-7", Style::default().fg(theme::SECONDARY)),
|
|
```
|
|
|
|
**Step 5: Update help overlay**
|
|
|
|
Update the cockpit list text:
|
|
```rust
|
|
Line::from(vec![Span::styled(
|
|
" Cockpits: Overview | Training | Trading",
|
|
theme::muted_style(),
|
|
)]),
|
|
Line::from(vec![Span::styled(
|
|
" Services | Risk | Data | Pods",
|
|
theme::muted_style(),
|
|
)]),
|
|
```
|
|
|
|
**Step 6: Update AppState doc comment**
|
|
|
|
In `state.rs`, change `Active cockpit index (0..=5)` to `Active cockpit index (0..=6)` and `six cockpit views` to `seven cockpit views`.
|
|
|
|
**Step 7: Verify it compiles**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check -p fxt`
|
|
|
|
**Step 8: Run tests**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo test -p fxt --lib`
|
|
|
|
**Step 9: Commit**
|
|
|
|
```bash
|
|
git add bin/fxt/src/tui/event_loop.rs bin/fxt/src/tui/state.rs
|
|
git commit -m "feat(fxt): wire Pods cockpit as key 7, update event loop"
|
|
```
|
|
|
|
---
|
|
|
|
## Task 13: Final Integration Build + Deploy
|
|
|
|
**Step 1: Full workspace check**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo check --workspace`
|
|
|
|
**Step 2: Run api_gateway + fxt tests**
|
|
|
|
Run: `SQLX_OFFLINE=true cargo test -p api_gateway --lib && SQLX_OFFLINE=true cargo test -p fxt --lib`
|
|
|
|
**Step 3: Commit any remaining changes**
|
|
|
|
```bash
|
|
git status
|
|
git add -A
|
|
git commit -m "chore: final integration for pods panel + CalVer + monitoring fixes"
|
|
```
|
|
|
|
**Step 4: Push and trigger CI**
|
|
|
|
Push branch `worktree-pods-versioning` to GitLab to trigger a CI pipeline.
|
|
|
|
---
|
|
|
|
## Dependencies Graph
|
|
|
|
```
|
|
Task 1 (build_info) ─┬─→ Task 3 (version in ServiceStatus)
|
|
└─→ Task 2 (CI version)
|
|
Task 4 (cluster resources) ── already done
|
|
Task 5 (proto) ──→ Task 6 (pods handler) ──→ Task 7 (RBAC) + Task 8 (NetworkPolicy)
|
|
Task 9 (state structs) ──→ Task 10 (data fetcher) ──→ Task 11 (cockpit) ──→ Task 12 (event loop)
|
|
All ──→ Task 13 (integration)
|
|
```
|