foxhunt/docs/plans/2026-03-04-pods-panel-versioning.md

Pods Panel, CalVer Versioning & Monitoring Fixes — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Add CalVer auto-versioning to all builds, a 7th TUI cockpit showing K8s pod status, and fix cluster resource monitoring.

Architecture: CI computes YYYY.MM.PIPELINE_IID and passes it via env var. Each binary reads it at compile time via option_env!(). A new gRPC streaming RPC in api_gateway uses the kube crate to list/watch pods. The TUI gets a 7th cockpit (key 7) rendering pod state by service group.

Tech Stack: Rust (workspace), tonic/prost (gRPC), ratatui (TUI), kube-rs (K8s client), GitLab CI, Prometheus

---

Task 1: CalVer Build Version — common/src/build_info.rs ✅ (if done) / TODO

Files:

• Create: crates/common/src/build_info.rs
• Modify: crates/common/src/lib.rs

Step 1: Create build_info.rs

// crates/common/src/build_info.rs
//! Build-time version information.
//!
//! CI sets `FOXHUNT_BUILD_VERSION` (e.g. `2026.03.408`).
//! Local dev falls back to `CARGO_PKG_VERSION`.

/// Build version string. Set at compile time.
///
/// CI: `FOXHUNT_BUILD_VERSION=2026.03.408`
/// Local: falls back to Cargo.toml `version` field.
pub fn version() -> &'static str {
    option_env!("FOXHUNT_BUILD_VERSION").unwrap_or(env!("CARGO_PKG_VERSION"))
}

Step 2: Register module in lib.rs

In crates/common/src/lib.rs, add after the pub mod tls; line:

pub mod build_info;

Step 3: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p common Expected: compiles, version() returns "1.0.0" (Cargo.toml workspace version)

Step 4: Commit

git add crates/common/src/build_info.rs crates/common/src/lib.rs
git commit -m "feat(common): add build_info module with CalVer version function"

---

Task 2: CI Version Computation — .gitlab-ci.yml

Files:

• Modify: .gitlab-ci.yml (global variables + compile job scripts)

Step 1: Add version computation to global variables

In the variables: block (after line 59), add:

  # CalVer: YYYY.MM.<pipeline_iid> — monotonically increasing per pipeline
  # Computed at job start; override in pipeline vars to pin a version.
  FOXHUNT_BUILD_VERSION: "${CI_COMMIT_TIMESTAMP:0:4}.${CI_COMMIT_TIMESTAMP:5:2}.${CI_PIPELINE_IID}"

Note: GitLab CE doesn't support bash substring in variables. We must compute it in the script section instead.

Step 1 (revised): Add version computation to compile scripts

In compile-services script section, add before cargo build:

    - |
      # CalVer: YYYY.MM.<pipeline_iid>
      YEAR=$(date -d "$CI_COMMIT_TIMESTAMP" +%Y 2>/dev/null || date +%Y)
      MONTH=$(date -d "$CI_COMMIT_TIMESTAMP" +%m 2>/dev/null || date +%m)
      export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.${CI_PIPELINE_IID:-0}"
      echo "Build version: $FOXHUNT_BUILD_VERSION"

Add the same block to compile-training script section, before cargo build.

Also add to compile-fxt (if it exists as a separate job, or it's part of compile-services).

Step 2: Verify locally

Run: FOXHUNT_BUILD_VERSION=2026.03.999 SQLX_OFFLINE=true cargo check -p common Expected: compiles. version() would return "2026.03.999" at runtime.

Step 3: Commit

git add .gitlab-ci.yml
git commit -m "feat(ci): compute CalVer FOXHUNT_BUILD_VERSION in compile jobs"

---

Task 3: Populate ServiceStatus.version in api_gateway

Files:

• Modify: services/api_gateway/src/grpc/monitoring_handler.rs
• Dependency: common::build_info::version()

Step 1: Find check_backend_health or ServiceStatus construction

In monitoring_handler.rs, find where ServiceStatus is built (around line 280-306). The version field should be populated.

Look for the pattern:

version: None, // or ..Default::default()

Replace with:

version: Some(common::build_info::version().to_owned()),

This only sets the api_gateway's own version. Each downstream service would need to expose its version in its health response. For now, api_gateway reports its own version for all discovered services (they're all built from the same commit).

Step 2: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p api_gateway

Step 3: Commit

git add services/api_gateway/src/grpc/monitoring_handler.rs
git commit -m "feat(api_gateway): populate ServiceStatus.version from build_info"

---

Task 4: Cluster Resources Fix (DONE)

Already completed during monitoring debugging session.

Files changed:

• services/api_gateway/src/grpc/monitoring_handler.rs: Added query_cluster_resources() method, wired into build_response() replacing hardcoded zeros.
• infra/k8s/network-policies/prometheus.yaml: Added port 9100 to 172.16.0.0/16 ipBlock, port 8080 to foxhunt podSelector, port 9400 to kube-system namespace.
• infra/k8s/network-policies/prometheus-cilium.yaml (NEW): CiliumNetworkPolicy for Prometheus → hostNetwork pods (node-exporter).

Verify: Already verified — Prometheus returns real CPU/RAM/GPU values. api_gateway monitoring tests pass (9/9).

Commit:

git add services/api_gateway/src/grpc/monitoring_handler.rs
git add infra/k8s/network-policies/prometheus.yaml
git add infra/k8s/network-policies/prometheus-cilium.yaml
git commit -m "fix(monitoring): wire cluster resources, fix Prometheus networking"

---

Task 5: Proto — SubscribeClusterPods RPC

Files:

• Modify: proto/monitoring.proto

Step 1: Add messages and RPC

After the GetEpochHistory RPC (line 54), add:

  // Kubernetes pod status — streams every N seconds
  rpc SubscribeClusterPods(SubscribeClusterPodsRequest) returns (stream ClusterPodsResponse);

After GetEpochHistoryResponse (line 425), add:

// ============================================================================
// Cluster Pod Messages
// ============================================================================

message SubscribeClusterPodsRequest {
  uint32 interval_seconds = 1;  // 0 = server default (5s)
}

message ClusterPodsResponse {
  repeated PodInfo pods = 1;
  int64 timestamp = 2;
}

message PodInfo {
  string name = 1;
  string service = 2;         // app.kubernetes.io/name label
  string namespace = 3;
  string status = 4;          // Running/Pending/CrashLoopBackOff/Completed/etc
  int32 restarts = 5;
  string age = 6;             // human-readable: "2d5h", "3m"
  string node = 7;
  bool ready = 8;
  string version = 9;         // from FOXHUNT_BUILD_VERSION env or empty
}

Step 2: Verify proto compiles

Run: SQLX_OFFLINE=true cargo check -p api_gateway Expected: compile error for unimplemented subscribe_cluster_pods — that's expected, we implement it next.

Step 3: Commit

git add proto/monitoring.proto
git commit -m "feat(proto): add SubscribeClusterPods streaming RPC"

---

Task 6: api_gateway — kube crate + pods handler

Files:

• Modify: services/api_gateway/Cargo.toml
• Create: services/api_gateway/src/grpc/pods_handler.rs
• Modify: services/api_gateway/src/grpc/monitoring_handler.rs (implement the new RPC)

Step 1: Add kube dependency

In services/api_gateway/Cargo.toml, add after the redis dependency:

# Kubernetes API client for pod listing
kube = { version = "0.99", features = ["client", "runtime", "derive"] }
k8s-openapi = { version = "0.24", features = ["latest"] }

Step 2: Create pods_handler.rs

// services/api_gateway/src/grpc/pods_handler.rs
//! Kubernetes pod listing for the Pods cockpit.

use anyhow::Result;
use k8s_openapi::api::core::v1::Pod;
use kube::{Api, Client};

/// Fetch all pods in a namespace and convert to proto `PodInfo`.
pub async fn list_pods(namespace: &str) -> Result<Vec<super::monitoring::PodInfo>> {
    let client = Client::try_default().await?;
    let pods: Api<Pod> = Api::namespaced(client, namespace);
    let pod_list = pods.list(&Default::default()).await?;

    let mut infos = Vec::with_capacity(pod_list.items.len());
    for pod in pod_list.items {
        let meta = &pod.metadata;
        let spec = pod.spec.as_ref();
        let status = pod.status.as_ref();

        let name = meta.name.clone().unwrap_or_default();
        let labels = meta.labels.as_ref();
        let service = labels
            .and_then(|l| l.get("app.kubernetes.io/name"))
            .cloned()
            .unwrap_or_default();

        let phase = status
            .and_then(|s| s.phase.clone())
            .unwrap_or_else(|| "Unknown".into());

        let ready = status
            .and_then(|s| s.conditions.as_ref())
            .map(|conds| conds.iter().any(|c| c.type_ == "Ready" && c.status == "True"))
            .unwrap_or(false);

        let restarts: i32 = status
            .and_then(|s| s.container_statuses.as_ref())
            .map(|cs| cs.iter().map(|c| c.restart_count).sum())
            .unwrap_or(0);

        let node = spec
            .and_then(|s| s.node_name.clone())
            .unwrap_or_default();

        let creation = meta.creation_timestamp.as_ref().map(|t| t.0);
        let age = creation
            .map(|t| format_age(t))
            .unwrap_or_else(|| "?".into());

        // Try to get version from FOXHUNT_BUILD_VERSION env var in first container
        let version = spec
            .and_then(|s| s.containers.first())
            .and_then(|c| c.env.as_ref())
            .and_then(|envs| {
                envs.iter()
                    .find(|e| e.name == "FOXHUNT_BUILD_VERSION")
                    .and_then(|e| e.value.clone())
            })
            .unwrap_or_default();

        infos.push(super::monitoring::PodInfo {
            name,
            service,
            namespace: namespace.into(),
            status: phase,
            restarts,
            age,
            node: short_node_name(&node),
            ready,
            version,
        });
    }

    // Sort by service name, then pod name
    infos.sort_by(|a, b| (&a.service, &a.name).cmp(&(&b.service, &b.name)));
    Ok(infos)
}

fn short_node_name(full: &str) -> String {
    // scw-foxhunt-foxhunt-e4027b8e... → foxhunt
    // scw-foxhunt-ci-training-h100-... → ci-training-h100
    full.strip_prefix("scw-foxhunt-")
        .unwrap_or(full)
        .split('-')
        .take(3)
        .collect::<Vec<_>>()
        .join("-")
        .chars()
        .take(20)
        .collect()
}

fn format_age(created: chrono::DateTime<chrono::Utc>) -> String {
    let dur = chrono::Utc::now() - created;
    let secs = dur.num_seconds();
    if secs < 60 {
        format!("{secs}s")
    } else if secs < 3600 {
        format!("{}m", secs / 60)
    } else if secs < 86400 {
        format!("{}h{}m", secs / 3600, (secs % 3600) / 60)
    } else {
        format!("{}d{}h", secs / 86400, (secs % 86400) / 3600)
    }
}

Step 3: Wire into monitoring_handler.rs

Add the subscribe_cluster_pods implementation to the MonitoringService impl block. This follows the same streaming pattern as stream_training_metrics:

type SubscribeClusterPodsStream =
    Pin<Box<dyn Stream<Item = Result<ClusterPodsResponse, Status>> + Send>>;

async fn subscribe_cluster_pods(
    &self,
    request: Request<SubscribeClusterPodsRequest>,
) -> Result<Response<Self::SubscribeClusterPodsStream>, Status> {
    let req = request.into_inner();
    let interval_secs = if req.interval_seconds == 0 { 5 } else { req.interval_seconds.clamp(1, 60) };

    let stream = async_stream::try_stream! {
        let mut interval = tokio::time::interval(Duration::from_secs(u64::from(interval_secs)));
        loop {
            interval.tick().await;
            match pods_handler::list_pods("foxhunt").await {
                Ok(pods) => {
                    yield ClusterPodsResponse {
                        pods,
                        timestamp: chrono::Utc::now().timestamp(),
                    };
                }
                Err(e) => {
                    tracing::warn!("Pod listing failed: {e}");
                    yield ClusterPodsResponse {
                        pods: vec![],
                        timestamp: chrono::Utc::now().timestamp(),
                    };
                }
            }
        }
    };

    Ok(Response::new(Box::pin(stream)))
}

Step 4: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p api_gateway

Step 5: Commit

git add services/api_gateway/Cargo.toml
git add services/api_gateway/src/grpc/pods_handler.rs
git add services/api_gateway/src/grpc/monitoring_handler.rs
git commit -m "feat(api_gateway): implement SubscribeClusterPods with kube crate"

---

Task 7: RBAC for api_gateway pod listing

Files:

• Modify: infra/k8s/services/api-gateway.yaml (or create separate RBAC file)

Step 1: Add Role + RoleBinding

Create or append to the api_gateway K8s resources:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: api-gateway-pod-reader
  namespace: foxhunt
rules:
  - apiGroups: [""]
    resources: [pods]
    verbs: [get, list, watch]
  - apiGroups: [metrics.k8s.io]
    resources: [pods]
    verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: api-gateway-pod-reader
  namespace: foxhunt
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: api-gateway-pod-reader
subjects:
  - kind: ServiceAccount
    name: api-gateway
    namespace: foxhunt

Step 2: Apply

Run: kubectl apply -f infra/k8s/services/api-gateway.yaml

Step 3: Verify

Run: kubectl auth can-i list pods -n foxhunt --as=system:serviceaccount:foxhunt:api-gateway Expected: yes

Step 4: Commit

git add infra/k8s/services/api-gateway.yaml
git commit -m "feat(infra): RBAC for api_gateway pod listing"

---

Task 8: api_gateway NetworkPolicy — K8s API egress

Files:

• Modify: infra/k8s/network-policies/api-gateway.yaml

Step 1: Add K8s API egress

Add egress rule to allow api_gateway → K8s API for pod listing:

    # Kubernetes API (pod listing for Pods cockpit)
    - to:
        - ipBlock:
            cidr: 10.32.0.0/12
      ports:
        - protocol: TCP
          port: 443
    - to:
        - ipBlock:
            cidr: 172.16.0.0/16
      ports:
        - protocol: TCP
          port: 6443

Step 2: Apply and verify

Run: kubectl apply -f infra/k8s/network-policies/api-gateway.yaml Then: kubectl exec deploy/api-gateway -n foxhunt -- wget -qO- --timeout=3 https://kubernetes.default.svc/api/v1/namespaces/foxhunt/pods 2>&1 | head -5

Step 3: Commit

git add infra/k8s/network-policies/api-gateway.yaml
git commit -m "feat(infra): api-gateway K8s API egress for pod listing"

---

Task 9: TUI — PodData structs + StateUpdate variant

Files:

• Modify: bin/fxt/src/tui/state.rs
• Modify: bin/fxt/src/tui/data_fetcher.rs
• Modify: bin/fxt/src/tui/event_loop.rs

Step 1: Add pod data structs to state.rs

After AlertData struct (around line 312):

/// A single K8s pod.
pub struct PodData {
    pub name: String,
    pub service: String,
    pub status: String,
    pub restarts: i32,
    pub age: String,
    pub node: String,
    pub ready: bool,
    pub version: String,
}

/// Pods grouped by service.
pub struct PodGroupData {
    pub service: String,
    pub pods: Vec<PodData>,
    pub ready_count: usize,
    pub total_count: usize,
}

Add pods: Vec<PodGroupData> field to AppState struct, and pods: Vec::new() to its Default impl. Also add selected_pod_group: Option<usize> and pod_group_expanded: bool for navigation.

Step 2: Add StateUpdate::Pods variant to data_fetcher.rs

/// Cluster pod statuses (from SubscribeClusterPods).
Pods(Vec<PodGroupData>),

Step 3: Handle in apply_update in event_loop.rs

StateUpdate::Pods(pods) => {
    state.pods = pods;
}

Step 4: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p fxt

Step 5: Commit

git add bin/fxt/src/tui/state.rs bin/fxt/src/tui/data_fetcher.rs bin/fxt/src/tui/event_loop.rs
git commit -m "feat(fxt): add PodData/PodGroupData structs and StateUpdate::Pods"

---

Task 10: TUI — spawn_pods_stream in data_fetcher

Files:

• Modify: bin/fxt/src/tui/data_fetcher.rs

Step 1: Add spawn_pods_stream method

Follow the same pattern as spawn_training_metrics. The stream subscribes to SubscribeClusterPods and groups pods by service:

fn spawn_pods_stream(&self, client: &FoxhuntClient) {
    let mut mon = client.monitoring();
    let tx = self.tx.clone();
    let cancel = self.cancel.clone();

    tokio::spawn(async move {
        let mut backoff = Backoff::new();
        loop {
            let req = Request::new(monitoring::SubscribeClusterPodsRequest {
                interval_seconds: 5,
            });
            match mon.subscribe_cluster_pods(req).await {
                Ok(resp) => {
                    backoff.reset();
                    let mut stream = resp.into_inner();
                    loop {
                        tokio::select! {
                            _ = cancel.cancelled() => return,
                            msg = stream.message() => {
                                match msg {
                                    Ok(Some(m)) => {
                                        let groups = group_pods(m.pods);
                                        let _ = tx.send(StateUpdate::Pods(groups));
                                    }
                                    Ok(None) => break,
                                    Err(e) => {
                                        warn!("Pods stream error: {e}");
                                        break;
                                    }
                                }
                            }
                        }
                    }
                }
                Err(e) => {
                    warn!("Pods stream connect failed: {e}");
                }
            }
            if backoff.attempts >= MAX_RECONNECT_ATTEMPTS { return; }
            tokio::select! {
                _ = cancel.cancelled() => return,
                _ = tokio::time::sleep(backoff.next_delay()) => {}
            }
        }
    });
}

Add a group_pods helper:

fn group_pods(pods: Vec<monitoring::PodInfo>) -> Vec<PodGroupData> {
    use std::collections::BTreeMap;
    let mut groups: BTreeMap<String, Vec<PodData>> = BTreeMap::new();
    for p in pods {
        let svc = if p.service.is_empty() { "(other)".into() } else { p.service.clone() };
        groups.entry(svc).or_default().push(PodData {
            name: p.name,
            service: p.service,
            status: p.status,
            restarts: p.restarts,
            age: p.age,
            node: p.node,
            ready: p.ready,
            version: p.version,
        });
    }
    groups.into_iter().map(|(service, pods)| {
        let ready_count = pods.iter().filter(|p| p.ready).count();
        let total_count = pods.len();
        PodGroupData { service, pods, ready_count, total_count }
    }).collect()
}

Step 2: Register in start() method

Add self.spawn_pods_stream(client); to the start() method.

Step 3: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p fxt

Step 4: Commit

git add bin/fxt/src/tui/data_fetcher.rs
git commit -m "feat(fxt): add spawn_pods_stream with service grouping"

---

Task 11: TUI — Pods Cockpit

Files:

• Create: bin/fxt/src/tui/cockpits/pods.rs
• Modify: bin/fxt/src/tui/cockpits/mod.rs

Step 1: Create pods.rs

//! Pods cockpit — Kubernetes pod status grouped by service.

use ratatui::layout::{Constraint, Direction, Layout, Rect};
use ratatui::style::{Modifier, Style};
use ratatui::text::{Line, Span};
use ratatui::widgets::{Block, Borders, Paragraph, Row, Table};

use super::super::state::AppState;
use super::super::theme;
use super::Cockpit;

pub struct PodsCockpit;

impl Cockpit for PodsCockpit {
    fn name(&self) -> &str {
        "Pods"
    }

    fn render(&self, frame: &mut ratatui::Frame, area: Rect, state: &AppState) {
        let block = Block::default()
            .borders(Borders::ALL)
            .title(" Cluster Pods ")
            .title_style(theme::title_style())
            .border_style(Style::default().fg(theme::BORDER));

        if state.pods.is_empty() {
            let msg = Paragraph::new("Waiting for pod data...")
                .style(theme::muted_style())
                .block(block);
            frame.render_widget(msg, area);
            return;
        }

        let header = Row::new(vec![
            "Service", "Ready", "Status", "Version", "Restarts", "Age", "Node",
        ])
        .style(Style::default().fg(theme::SECONDARY).add_modifier(Modifier::BOLD));

        let mut rows = Vec::new();
        for group in &state.pods {
            // Summary row for the service group
            let ready_str = format!("{}/{}", group.ready_count, group.total_count);
            let status = if group.ready_count == group.total_count {
                Span::styled("Running", Style::default().fg(theme::SUCCESS))
            } else if group.ready_count == 0 {
                Span::styled("Down", Style::default().fg(theme::ERROR))
            } else {
                Span::styled("Degraded", Style::default().fg(theme::WARNING))
            };

            let version = group.pods.first()
                .map(|p| p.version.as_str())
                .unwrap_or("");

            let total_restarts: i32 = group.pods.iter().map(|p| p.restarts).sum();
            let restart_style = if total_restarts > 0 {
                Style::default().fg(theme::WARNING)
            } else {
                theme::muted_style()
            };

            rows.push(Row::new(vec![
                Line::from(Span::styled(
                    &group.service,
                    Style::default().fg(theme::PRIMARY).add_modifier(Modifier::BOLD),
                )),
                Line::from(ready_str),
                Line::from(status),
                Line::from(Span::styled(version, theme::muted_style())),
                Line::from(Span::styled(
                    format!("{total_restarts}"),
                    restart_style,
                )),
                Line::from(Span::styled(
                    group.pods.first().map(|p| p.age.as_str()).unwrap_or(""),
                    theme::muted_style(),
                )),
                Line::from(Span::styled(
                    group.pods.first().map(|p| p.node.as_str()).unwrap_or(""),
                    theme::muted_style(),
                )),
            ]));

            // Expanded: show individual pods if more than 1 in group
            if group.pods.len() > 1 {
                for pod in &group.pods {
                    let pod_status_style = if pod.ready {
                        Style::default().fg(theme::SUCCESS)
                    } else {
                        Style::default().fg(theme::ERROR)
                    };
                    let pod_restart_style = if pod.restarts > 0 {
                        Style::default().fg(theme::WARNING)
                    } else {
                        theme::muted_style()
                    };
                    rows.push(Row::new(vec![
                        Line::from(Span::styled(
                            format!("  {}", short_pod_name(&pod.name)),
                            theme::muted_style(),
                        )),
                        Line::from(if pod.ready { "✓" } else { "✗" }),
                        Line::from(Span::styled(&pod.status, pod_status_style)),
                        Line::from(""),
                        Line::from(Span::styled(
                            format!("{}", pod.restarts),
                            pod_restart_style,
                        )),
                        Line::from(Span::styled(&pod.age, theme::muted_style())),
                        Line::from(Span::styled(&pod.node, theme::muted_style())),
                    ]));
                }
            }
        }

        let widths = [
            Constraint::Length(28), // Service
            Constraint::Length(7),  // Ready
            Constraint::Length(12), // Status
            Constraint::Length(14), // Version
            Constraint::Length(9),  // Restarts
            Constraint::Length(8),  // Age
            Constraint::Fill(1),   // Node
        ];

        let table = Table::new(rows, widths)
            .header(header)
            .block(block);

        frame.render_widget(table, area);
    }
}

/// Shorten pod names: `api-gateway-7686d84bdd-zl4mn` → `zl4mn`
fn short_pod_name(name: &str) -> &str {
    name.rsplit('-').next().unwrap_or(name)
}

Step 2: Register in cockpits/mod.rs

pub mod pods;

Step 3: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p fxt

Step 4: Commit

git add bin/fxt/src/tui/cockpits/pods.rs bin/fxt/src/tui/cockpits/mod.rs
git commit -m "feat(fxt): add Pods cockpit with service-grouped table"

---

Task 12: TUI — Wire Cockpit 7 into Event Loop

Files:

• Modify: bin/fxt/src/tui/event_loop.rs

Step 1: Update COCKPIT_COUNT

const COCKPIT_COUNT: usize = 7;

Step 2: Add PodsCockpit to cockpits vec

use super::cockpits::pods::PodsCockpit;

// In event_loop():
let cockpits: Vec<Box<dyn Cockpit>> = vec![
    Box::new(OverviewCockpit),
    Box::new(TrainingCockpit),
    Box::new(TradingCockpit),
    Box::new(ServicesCockpit),
    Box::new(RiskCockpit),
    Box::new(DataCockpit),
    Box::new(PodsCockpit),
];

Step 3: Update key handler range

Change:

KeyCode::Char(c @ '1'..='6') => {

To:

KeyCode::Char(c @ '1'..='7') => {

Step 4: Update status bar

In render_status_bar, change "1-6" to "1-7":

Span::styled("1-7", Style::default().fg(theme::SECONDARY)),

Step 5: Update help overlay

Update the cockpit list text:

Line::from(vec![Span::styled(
    "  Cockpits: Overview | Training | Trading",
    theme::muted_style(),
)]),
Line::from(vec![Span::styled(
    "            Services | Risk     | Data | Pods",
    theme::muted_style(),
)]),

Step 6: Update AppState doc comment

In state.rs, change Active cockpit index (0..=5) to Active cockpit index (0..=6) and six cockpit views to seven cockpit views.

Step 7: Verify it compiles

Run: SQLX_OFFLINE=true cargo check -p fxt

Step 8: Run tests

Run: SQLX_OFFLINE=true cargo test -p fxt --lib

Step 9: Commit

git add bin/fxt/src/tui/event_loop.rs bin/fxt/src/tui/state.rs
git commit -m "feat(fxt): wire Pods cockpit as key 7, update event loop"

---

Task 13: Final Integration Build + Deploy

Step 1: Full workspace check

Run: SQLX_OFFLINE=true cargo check --workspace

Step 2: Run api_gateway + fxt tests

Run: SQLX_OFFLINE=true cargo test -p api_gateway --lib && SQLX_OFFLINE=true cargo test -p fxt --lib

Step 3: Commit any remaining changes

git status
git add -A
git commit -m "chore: final integration for pods panel + CalVer + monitoring fixes"

Step 4: Push and trigger CI

Push branch worktree-pods-versioning to GitLab to trigger a CI pipeline.

---

Dependencies Graph

Task 1 (build_info) ─┬─→ Task 3 (version in ServiceStatus)
                      └─→ Task 2 (CI version)
Task 4 (cluster resources) ── already done
Task 5 (proto) ──→ Task 6 (pods handler) ──→ Task 7 (RBAC) + Task 8 (NetworkPolicy)
Task 9 (state structs) ──→ Task 10 (data fetcher) ──→ Task 11 (cockpit) ──→ Task 12 (event loop)
All ──→ Task 13 (integration)