From d417807132fb8f4a3efef7493d09da52bcf970e1 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 1 Mar 2026 02:00:08 +0100 Subject: [PATCH] docs: add real metrics overhaul implementation plan 13-task plan across 4 layers: fix dashboard PromQL, add metrics servers to data-acquisition and web-gateway, create gRPC metrics Tower layer, deploy Pushgateway for training jobs. Co-Authored-By: Claude Opus 4.6 --- ...01-real-metrics-overhaul-implementation.md | 1278 +++++++++++++++++ 1 file changed, 1278 insertions(+) create mode 100644 docs/plans/2026-03-01-real-metrics-overhaul-implementation.md diff --git a/docs/plans/2026-03-01-real-metrics-overhaul-implementation.md b/docs/plans/2026-03-01-real-metrics-overhaul-implementation.md new file mode 100644 index 000000000..8f7403b3b --- /dev/null +++ b/docs/plans/2026-03-01-real-metrics-overhaul-implementation.md @@ -0,0 +1,1278 @@ +# Real Metrics Dashboard Overhaul — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Replace all placeholder PromQL queries in 4 Grafana dashboards with real metric names, add metrics servers to data-acquisition and web-gateway, add a shared gRPC metrics tower layer, and deploy a Pushgateway for training job metrics. + +**Architecture:** Four layers executed sequentially. Layer 1 fixes dashboard JSON queries to match real Prometheus metric names. Layer 2 adds Axum-based metrics HTTP servers to the two services missing them. Layer 3 creates a reusable Tower middleware in the `common` crate that automatically instruments gRPC handlers with request count and latency histograms, then wires it into all 8 services. Layer 4 deploys a Prometheus Pushgateway and integrates it into the training binaries for epoch-level metrics. + +**Tech Stack:** Rust (prometheus crate, once_cell::sync::Lazy, axum, tower), Grafana dashboard JSON, Kubernetes YAML, Prometheus Pushgateway + +**Build command:** `SQLX_OFFLINE=true cargo check --workspace` +**Test command:** `SQLX_OFFLINE=true cargo test -p --lib` +**Clippy:** `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]` + +--- + +## Layer 1: Fix Dashboard Queries + +### Task 1: Fix foxhunt-trading-cockpit.json PromQL Queries + +**Files:** +- Modify: `infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json` + +**Context:** The Trading Cockpit has 17 panels across 5 rows. Every panel uses placeholder metric names. The real metrics are defined in `services/trading_service/src/metrics.rs` with `Lazy` statics. The exact Prometheus metric names (lowercase, underscored) are what gets exported. + +**Step 1: Replace all PromQL expressions** + +For each panel, use this mapping of current → correct query: + +| Panel Title | Current `expr` | Correct `expr` | +|---|---|---| +| Ensemble Agreement | `ml_ensemble_agreement_rate` | `ml_ensemble_agreement_rate{symbol=~"$symbol"}` | +| Prediction Confidence | `ml_prediction_confidence` | `histogram_quantile(0.5, rate(ml_predictions_confidence_bucket[5m]))` | +| Model Accuracy | `ml_model_accuracy` | `ml_prediction_accuracy{model_id=~"$model"}` | +| Ensemble Sharpe | `ml_ensemble_sharpe` | `ml_model_sharpe_ratio{model_id="ensemble"}` | +| Active Positions | `foxhunt_active_positions` | `trading_agent_assets_selected` | +| Realized PnL | `foxhunt_realized_pnl` | `ml_model_cumulative_pnl{model_id=~"$model"}` | +| Unrealized PnL | `foxhunt_unrealized_pnl` | `trading_agent_portfolio_value_usd` | +| Max Drawdown | `foxhunt_max_drawdown` | `ml_model_max_drawdown{model_id=~"$model"}` | +| PnL Over Time | `foxhunt_pnl_total` | `ml_model_cumulative_pnl{model_id=~"$model"}` | +| Portfolio VaR | `foxhunt_portfolio_var` | `ml_model_max_drawdown{model_id="ensemble"}` | +| Circuit Breaker | `foxhunt_circuit_breaker_state` | `ml_circuit_breaker_state{model_id=~"$model"}` | +| Risk Alerts | `foxhunt_risk_alerts_total` | `rate(ml_alerts_total{severity="critical"}[5m])` | +| Position HHI | `foxhunt_position_hhi` | `ml_ensemble_agreement_rate{symbol=~"$symbol"}` | +| Inference Latency p95 | `foxhunt_inference_latency_seconds` | `histogram_quantile(0.95, rate(ml_model_inference_latency_bucket[5m]))` | +| Model Drift | `foxhunt_model_drift` | `ml_model_drift_score{model_id=~"$model"}` | +| gRPC Latency p95 | `foxhunt_grpc_latency_seconds` | `histogram_quantile(0.95, rate(grpc_server_handling_seconds_bucket{service=~"$service"}[5m]))` | +| ML Inference Distribution | `foxhunt_ml_inference_duration_seconds` | `rate(ml_model_inference_latency_bucket[5m])` | + +Also add two template variables to the dashboard `templating.list`: + +```json +{ + "name": "model", + "type": "query", + "query": "label_values(ml_prediction_accuracy, model_id)", + "datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"}, + "refresh": 2, + "includeAll": true, + "multi": true, + "current": {"text": "All", "value": "$__all"} +}, +{ + "name": "symbol", + "type": "query", + "query": "label_values(ml_ensemble_agreement_rate, symbol)", + "datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"}, + "refresh": 2, + "includeAll": true, + "multi": true, + "current": {"text": "All", "value": "$__all"} +} +``` + +**Step 2: Validate JSON is valid** + +Run: `python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json'))"` +Expected: No output (valid JSON) + +**Step 3: Commit** + +```bash +git add infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json +git commit -m "fix(dashboard): use real metric names in trading cockpit" +``` + +--- + +### Task 2: Fix foxhunt-training-cockpit.json PromQL Queries + +**Files:** +- Modify: `infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json` + +**Context:** The Training Cockpit has 28 panels across 8 rows. Many panels query metrics that come from kube-state-metrics (job status), DCGM exporter (GPU), ml-training-service (job counters), and Pushgateway (epoch-level, Layer 4). For metrics that require Pushgateway (Layer 4), use the correct metric name now — they'll show "No data" until the Pushgateway is deployed. + +**Step 1: Replace all PromQL expressions** + +| Panel Title | Current `expr` | Correct `expr` | +|---|---|---| +| Running Jobs | `foxhunt_training_jobs_active` | `kube_job_status_active{namespace="foxhunt", job_name=~"training-.*"}` | +| Active Workers | `foxhunt_training_active_workers` | `ml_training_active_workers` | +| Jobs Started | `foxhunt_training_jobs_started_total` | `ml_training_jobs_started_total` | +| Jobs Completed | `foxhunt_training_jobs_completed_total` | `ml_training_jobs_completed_total` | +| Failed Jobs | `foxhunt_training_jobs_failed_total` | `ml_training_errors_total` | +| Pending Jobs | `foxhunt_training_jobs_pending` | `kube_job_status_active{namespace="foxhunt", job_name=~"training-.*"} - kube_job_status_succeeded{namespace="foxhunt", job_name=~"training-.*"}` | +| Loss Curves | `foxhunt_training_loss` | `foxhunt_training_epoch_loss{model=~"$model_type"}` | +| Epoch Progress | `foxhunt_training_epoch` | `foxhunt_training_current_epoch{model=~"$model_type"}` | +| Convergence Rate | `foxhunt_training_convergence_rate` | `deriv(foxhunt_training_epoch_loss{model=~"$model_type"}[10m])` | +| Training Speed | `foxhunt_training_steps_per_second` | `foxhunt_training_batches_per_second{model=~"$model_type"}` | +| GPU Utilization | `foxhunt_gpu_utilization` | `DCGM_FI_DEV_GPU_UTIL` | +| GPU Memory | `foxhunt_gpu_memory_used` | `DCGM_FI_DEV_FB_USED` | +| GPU Temperature | `foxhunt_gpu_temperature` | `DCGM_FI_DEV_GPU_TEMP` | +| GPU Power | `foxhunt_gpu_power` | `DCGM_FI_DEV_POWER_USAGE` | +| Accuracy | `foxhunt_training_accuracy` | `foxhunt_training_eval_accuracy{model=~"$model_type"}` | +| F1 / Precision / Recall | `foxhunt_training_f1` | `foxhunt_training_eval_f1{model=~"$model_type"}` | +| Total Saves | `foxhunt_checkpoint_saves_total` | `foxhunt_training_checkpoint_saves_total{model=~"$model_type"}` | +| Save Failures | `foxhunt_checkpoint_failures` | `foxhunt_training_checkpoint_failures_total{model=~"$model_type"}` | +| Save Duration p95 | `foxhunt_checkpoint_duration_seconds` | `histogram_quantile(0.95, rate(foxhunt_training_checkpoint_duration_seconds_bucket[5m]))` | +| Checkpoint Size | `foxhunt_checkpoint_size_bytes` | `foxhunt_training_checkpoint_size_bytes{model=~"$model_type"}` | +| Batches Processed | `foxhunt_training_batches_total` | `foxhunt_training_batches_processed{model=~"$model_type"}` | +| Data Loading Latency | `foxhunt_data_loading_latency` | `histogram_quantile(0.95, rate(foxhunt_training_data_load_seconds_bucket[5m]))` | +| Feature Errors | `foxhunt_training_feature_errors` | `foxhunt_training_feature_errors_total{model=~"$model_type"}` | +| NaN Detections | `foxhunt_training_nan_detected` | `foxhunt_training_nan_detected_total{model=~"$model_type"}` | +| Gradient Explosions | `foxhunt_training_gradient_explosions` | `foxhunt_training_gradient_explosion_total{model=~"$model_type"}` | +| Training Failures | `foxhunt_training_failures` | `ml_training_errors_total` | +| Job Duration Distribution | `foxhunt_training_job_duration` | `(kube_job_complete_time{namespace="foxhunt", job_name=~"training-.*"} - kube_job_start_time{namespace="foxhunt", job_name=~"training-.*"}) / 3600` | +| Iteration Duration | `foxhunt_training_iteration_duration` | `foxhunt_training_iteration_seconds{model=~"$model_type"}` | + +**Step 2: Validate JSON** + +Run: `python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json'))"` + +**Step 3: Commit** + +```bash +git add infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json +git commit -m "fix(dashboard): use real metric names in training cockpit" +``` + +--- + +### Task 3: Fix foxhunt-traces.json PromQL Queries + +**Files:** +- Modify: `infra/k8s/monitoring/dashboards/foxhunt-traces.json` + +**Context:** The Traces dashboard has 8 panels. Overview stats should query Tempo distributor metrics (which Tempo exposes on port 3200 and Prometheus scrapes). Service Map uses Tempo datasource directly. Trace search uses Tempo TraceQL. + +**Step 1: Replace all PromQL expressions** + +| Panel Title | Current `expr` | Correct `expr` | +|---|---|---| +| Spans/sec | `foxhunt_spans_total` | `sum(rate(tempo_distributor_spans_received_total[5m]))` | +| Error Rate | `foxhunt_trace_errors_total` | `sum(rate(tempo_distributor_spans_received_total{status="error"}[5m])) / sum(rate(tempo_distributor_spans_received_total[5m]))` | +| Avg Duration | `foxhunt_span_duration_seconds` | `sum(rate(tempo_spanmetrics_latency_sum[5m])) / sum(rate(tempo_spanmetrics_latency_count[5m]))` | +| Active Services | `foxhunt_active_services` | `count(count by (service_name)(tempo_distributor_spans_received_total))` | +| Latency by Service p95 | `foxhunt_span_duration_seconds{quantile="0.95"}` | `histogram_quantile(0.95, sum by (le, service_name)(rate(tempo_spanmetrics_latency_bucket[5m])))` | +| Error Rate by Service | `foxhunt_trace_error_rate` | `sum by (service_name)(rate(tempo_distributor_spans_received_total{status="error"}[5m]))` | + +For the Service Map panel (nodeGraph type), ensure it uses the Tempo datasource (`${DS_TEMPO}`) with query type `serviceMap`. + +For the Recent Traces panel (table type), ensure it uses Tempo datasource with TraceQL query: `{status = error} || {duration > 1s}`. + +**Step 2: Validate JSON** + +Run: `python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-traces.json'))"` + +**Step 3: Commit** + +```bash +git add infra/k8s/monitoring/dashboards/foxhunt-traces.json +git commit -m "fix(dashboard): use real Tempo metric names in traces dashboard" +``` + +--- + +### Task 4: Fix foxhunt-cockpit.json Trace Health Row + +**Files:** +- Modify: `infra/k8s/monitoring/dashboards/foxhunt-cockpit.json` + +**Context:** The Cockpit dashboard's Trace Health row (panels 35-37, added in the OTEL task) needs the same Tempo metric fixes as Task 3. + +**Step 1: Fix the 3 Trace Health panels** + +| Panel Title | Current `expr` | Correct `expr` | +|---|---|---| +| Spans/sec | `foxhunt_spans_total` (if placeholder) | `sum(rate(tempo_distributor_spans_received_total[5m]))` | +| Trace Error Rate | `foxhunt_trace_error_rate` (if placeholder) | `sum(rate(tempo_distributor_spans_received_total{status="error"}[5m])) / sum(rate(tempo_distributor_spans_received_total[5m]))` | +| Spans/sec by Service | `foxhunt_spans_by_service` (if placeholder) | `sum by (service_name)(rate(tempo_distributor_spans_received_total[5m]))` | + +**Step 2: Validate JSON** + +Run: `python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-cockpit.json'))"` + +**Step 3: Commit** + +```bash +git add infra/k8s/monitoring/dashboards/foxhunt-cockpit.json +git commit -m "fix(dashboard): use real Tempo metrics in cockpit trace health row" +``` + +--- + +## Layer 2: Add Metrics Servers to Missing Services + +### Task 5: Add Prometheus Metrics Server to data-acquisition-service + +**Files:** +- Create: `services/data_acquisition_service/src/metrics.rs` +- Modify: `services/data_acquisition_service/src/main.rs` +- Modify: `services/data_acquisition_service/src/lib.rs` (add `pub mod metrics;`) + +**Context:** data-acquisition-service already has `prometheus`, `once_cell`, and `axum` in Cargo.toml. The K8s YAML already declares port 9097 with prometheus annotations. The only thing missing is the actual metrics server code. Follow the ml-training-service pattern (inline metrics handler in main.rs spawned via `tokio::spawn`). + +**Step 1: Create `services/data_acquisition_service/src/metrics.rs`** + +```rust +//! Prometheus metrics for Data Acquisition Service + +use once_cell::sync::Lazy; +use prometheus::{register_counter, register_counter_vec, register_gauge, register_histogram_vec, Counter, CounterVec, Gauge, HistogramVec}; + +/// Service uptime in seconds +pub static SERVICE_UPTIME: Lazy = Lazy::new(|| { + register_gauge!( + "foxhunt_data_acquisition_uptime_seconds", + "Service uptime in seconds" + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register uptime gauge: {e}"); + std::process::abort() + }) +}); + +/// Active data feeds +pub static FEEDS_ACTIVE: Lazy = Lazy::new(|| { + register_gauge!( + "foxhunt_data_acquisition_feeds_active", + "Number of active data feeds" + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register feeds_active gauge: {e}"); + std::process::abort() + }) +}); + +/// Total records received +pub static RECORDS_RECEIVED: Lazy = Lazy::new(|| { + register_counter_vec!( + "foxhunt_data_acquisition_records_received_total", + "Total records received from data feeds", + &["symbol", "feed_type"] + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register records_received counter: {e}"); + std::process::abort() + }) +}); + +/// Total errors +pub static ERRORS: Lazy = Lazy::new(|| { + register_counter!( + "foxhunt_data_acquisition_errors_total", + "Total data acquisition errors" + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register errors counter: {e}"); + std::process::abort() + }) +}); + +/// Feed processing latency +pub static FEED_LATENCY: Lazy = Lazy::new(|| { + register_histogram_vec!( + "foxhunt_data_acquisition_feed_latency_seconds", + "Feed processing latency in seconds", + &["feed_type"], + vec![0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0] + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register feed_latency histogram: {e}"); + std::process::abort() + }) +}); + +/// Initialize all metrics (force lazy registration) +pub fn init_metrics() { + let _ = &*SERVICE_UPTIME; + let _ = &*FEEDS_ACTIVE; + let _ = &*RECORDS_RECEIVED; + let _ = &*ERRORS; + let _ = &*FEED_LATENCY; +} + +/// Update service uptime metric +pub fn update_uptime(start_time: std::time::Instant) { + SERVICE_UPTIME.set(start_time.elapsed().as_secs_f64()); +} +``` + +**Step 2: Add `pub mod metrics;` to lib.rs** + +In `services/data_acquisition_service/src/lib.rs`, add: +```rust +pub mod metrics; +``` + +**Step 3: Add metrics server + uptime updater to main.rs** + +In `services/data_acquisition_service/src/main.rs`, after the `info!("Data Acquisition Service listening on {}", addr);` line (line 52) and before the `Server::builder()` block (line 55), add: + +```rust + // Initialize Prometheus metrics + data_acquisition_service::metrics::init_metrics(); + let service_start = std::time::Instant::now(); + + // Spawn uptime updater + tokio::spawn(async move { + let mut interval = tokio::time::interval(std::time::Duration::from_secs(1)); + loop { + interval.tick().await; + data_acquisition_service::metrics::update_uptime(service_start); + } + }); + + // Start Prometheus metrics HTTP endpoint on port 9097 + let metrics_port: u16 = std::env::var("METRICS_PORT") + .ok() + .and_then(|s| s.parse().ok()) + .unwrap_or(9097); + tokio::spawn(async move { + use axum::{routing::get, Router}; + use prometheus::{Encoder, TextEncoder}; + + async fn metrics_handler() -> String { + let encoder = TextEncoder::new(); + let metric_families = prometheus::gather(); + let mut buffer = vec![]; + let _ = encoder.encode(&metric_families, &mut buffer); + String::from_utf8(buffer).unwrap_or_else(|_| String::new()) + } + + let app = Router::new().route("/metrics", get(metrics_handler)); + let addr = format!("0.0.0.0:{}", metrics_port); + info!("Prometheus metrics endpoint listening on http://{}", addr); + + let listener = match tokio::net::TcpListener::bind(&addr).await { + Ok(l) => l, + Err(e) => { + tracing::error!("Failed to bind metrics endpoint {}: {}", addr, e); + return; + }, + }; + if let Err(e) = axum::serve(listener, app).await { + tracing::error!("Metrics server failed: {}", e); + } + }); +``` + +**Step 4: Build check** + +Run: `SQLX_OFFLINE=true cargo check -p data_acquisition_service` +Expected: Compiles clean + +**Step 5: Commit** + +```bash +git add services/data_acquisition_service/src/metrics.rs services/data_acquisition_service/src/main.rs services/data_acquisition_service/src/lib.rs +git commit -m "feat(data-acquisition): add Prometheus metrics server on port 9097" +``` + +--- + +### Task 6: Add Prometheus Metrics Server to web-gateway + +**Files:** +- Add dependency: `crates/web-gateway/Cargo.toml` +- Create: `crates/web-gateway/src/metrics.rs` +- Modify: `crates/web-gateway/src/main.rs` +- Modify: `crates/web-gateway/src/lib.rs` (add `pub mod metrics;`) + +**Context:** web-gateway does NOT have `prometheus` in Cargo.toml — must add it. web-gateway is an Axum HTTP server (port 3000). The metrics endpoint should run on a separate port (9098) to avoid mixing application traffic with scraping. Use the same inline pattern as ml-training-service. + +**Step 1: Add prometheus dependency to Cargo.toml** + +In `crates/web-gateway/Cargo.toml`, after the `common.workspace = true` line (line 44), add: + +```toml +# Metrics +prometheus.workspace = true +once_cell.workspace = true +``` + +**Step 2: Create `crates/web-gateway/src/metrics.rs`** + +```rust +//! Prometheus metrics for Web Gateway + +use once_cell::sync::Lazy; +use prometheus::{register_counter_vec, register_gauge, register_histogram_vec, CounterVec, Gauge, HistogramVec}; + +/// Service uptime in seconds +pub static SERVICE_UPTIME: Lazy = Lazy::new(|| { + register_gauge!( + "foxhunt_web_gateway_uptime_seconds", + "Service uptime in seconds" + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register uptime gauge: {e}"); + std::process::abort() + }) +}); + +/// HTTP requests total +pub static HTTP_REQUESTS: Lazy = Lazy::new(|| { + register_counter_vec!( + "foxhunt_web_gateway_http_requests_total", + "Total HTTP requests", + &["method", "path", "status"] + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register http_requests counter: {e}"); + std::process::abort() + }) +}); + +/// HTTP request duration +pub static HTTP_REQUEST_DURATION: Lazy = Lazy::new(|| { + register_histogram_vec!( + "foxhunt_web_gateway_http_request_duration_seconds", + "HTTP request duration in seconds", + &["method", "path"], + vec![0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0] + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register http_request_duration histogram: {e}"); + std::process::abort() + }) +}); + +/// Active WebSocket connections +pub static WS_CONNECTIONS: Lazy = Lazy::new(|| { + register_gauge!( + "foxhunt_web_gateway_ws_connections_active", + "Active WebSocket connections" + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register ws_connections gauge: {e}"); + std::process::abort() + }) +}); + +/// WebSocket messages total +pub static WS_MESSAGES: Lazy = Lazy::new(|| { + register_counter_vec!( + "foxhunt_web_gateway_ws_messages_total", + "Total WebSocket messages", + &["direction", "topic"] + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register ws_messages counter: {e}"); + std::process::abort() + }) +}); + +/// Initialize all metrics (force lazy registration) +pub fn init_metrics() { + let _ = &*SERVICE_UPTIME; + let _ = &*HTTP_REQUESTS; + let _ = &*HTTP_REQUEST_DURATION; + let _ = &*WS_CONNECTIONS; + let _ = &*WS_MESSAGES; +} + +/// Update service uptime metric +pub fn update_uptime(start_time: std::time::Instant) { + SERVICE_UPTIME.set(start_time.elapsed().as_secs_f64()); +} +``` + +**Step 3: Add `pub mod metrics;` to lib.rs** + +In `crates/web-gateway/src/lib.rs`, add: +```rust +pub mod metrics; +``` + +**Step 4: Add metrics server to main.rs** + +In `crates/web-gateway/src/main.rs`, after the `info!("Web gateway listening on {}", listen_addr);` line (line 79) and before the `axum::serve(` block (line 81), add: + +```rust + // Initialize Prometheus metrics + web_gateway::metrics::init_metrics(); + let service_start = std::time::Instant::now(); + + // Spawn uptime updater + tokio::spawn(async move { + let mut interval = tokio::time::interval(std::time::Duration::from_secs(1)); + loop { + interval.tick().await; + web_gateway::metrics::update_uptime(service_start); + } + }); + + // Start Prometheus metrics HTTP endpoint on port 9098 + let metrics_port: u16 = std::env::var("METRICS_PORT") + .ok() + .and_then(|s| s.parse().ok()) + .unwrap_or(9098); + tokio::spawn(async move { + use axum::{routing::get, Router}; + use prometheus::{Encoder, TextEncoder}; + + async fn metrics_handler() -> String { + let encoder = TextEncoder::new(); + let metric_families = prometheus::gather(); + let mut buffer = vec![]; + let _ = encoder.encode(&metric_families, &mut buffer); + String::from_utf8(buffer).unwrap_or_else(|_| String::new()) + } + + let app = Router::new().route("/metrics", get(metrics_handler)); + let addr = format!("0.0.0.0:{}", metrics_port); + info!("Prometheus metrics endpoint listening on http://{}", addr); + + let listener = match tokio::net::TcpListener::bind(&addr).await { + Ok(l) => l, + Err(e) => { + tracing::error!("Failed to bind metrics endpoint {}: {}", addr, e); + return; + }, + }; + if let Err(e) = axum::serve(listener, app).await { + tracing::error!("Metrics server failed: {}", e); + } + }); +``` + +**Step 5: Build check** + +Run: `SQLX_OFFLINE=true cargo check -p web-gateway` +Expected: Compiles clean + +**Step 6: Run existing tests** + +Run: `SQLX_OFFLINE=true cargo test -p web-gateway --lib` +Expected: All 138 tests pass + +**Step 7: Commit** + +```bash +git add crates/web-gateway/Cargo.toml crates/web-gateway/src/metrics.rs crates/web-gateway/src/main.rs crates/web-gateway/src/lib.rs +git commit -m "feat(web-gateway): add Prometheus metrics server on port 9098" +``` + +--- + +### Task 7: Update K8s YAML for web-gateway Metrics Port + +**Files:** +- Modify: `infra/k8s/services/web-gateway.yaml` + +**Context:** web-gateway K8s YAML currently has only port 3000 (http). Need to add port 9098 (metrics), prometheus scrape annotations, and ensure the Service exposes the metrics port. + +**Step 1: Add prometheus annotations to pod template** + +In `infra/k8s/services/web-gateway.yaml`, the pod template metadata currently has only `labels:` (line 21). Add `annotations:` before `labels:`: + +```yaml + metadata: + annotations: + gitlab.com/prometheus_scrape: "true" + gitlab.com/prometheus_port: "9098" + gitlab.com/prometheus_path: "/metrics" + labels: + app.kubernetes.io/name: web-gateway +``` + +**Step 2: Add metrics container port** + +After the `http` port (line 88), add: + +```yaml + - containerPort: 9098 + name: metrics +``` + +**Step 3: Add metrics port to Service** + +Find the Service section at the bottom of the file. Add the metrics port: + +```yaml + - port: 9098 + targetPort: 9098 + name: metrics +``` + +**Step 4: Commit** + +```bash +git add infra/k8s/services/web-gateway.yaml +git commit -m "infra(web-gateway): add metrics port 9098 and prometheus scrape annotations" +``` + +--- + +## Layer 3: gRPC Metrics Tower Layer + +### Task 8: Create gRPC Metrics Tower Layer in common + +**Files:** +- Create: `crates/common/src/metrics/grpc_metrics.rs` +- Modify: `crates/common/src/metrics/mod.rs` (add `pub mod grpc_metrics;`) + +**Context:** We need a Tower `Layer` + `Service` that wraps every gRPC handler and automatically records `grpc_server_started_total`, `grpc_server_handled_total`, and `grpc_server_handling_seconds`. The layer extracts the gRPC method from the request URI path. `common` already depends on `prometheus`, `once_cell`, `tonic`, and `tower` (via tonic). + +**Step 1: Create `crates/common/src/metrics/grpc_metrics.rs`** + +```rust +//! gRPC metrics Tower layer for automatic request instrumentation +//! +//! Provides `grpc_server_started_total`, `grpc_server_handled_total`, +//! and `grpc_server_handling_seconds` metrics for all gRPC handlers. + +use once_cell::sync::Lazy; +use prometheus::{ + register_counter_vec, register_histogram_vec, CounterVec, HistogramVec, +}; +use std::future::Future; +use std::pin::Pin; +use std::task::{Context, Poll}; +use tonic::body::BoxBody; +use tower::{Layer, Service}; + +/// Counter for gRPC requests started +static GRPC_STARTED: Lazy = Lazy::new(|| { + register_counter_vec!( + "grpc_server_started_total", + "Total gRPC RPCs started on the server", + &["service", "grpc_method"] + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register grpc_server_started_total: {e}"); + std::process::abort() + }) +}); + +/// Counter for gRPC requests completed with status code +static GRPC_HANDLED: Lazy = Lazy::new(|| { + register_counter_vec!( + "grpc_server_handled_total", + "Total gRPC RPCs completed on the server with status code", + &["service", "grpc_method", "grpc_code"] + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register grpc_server_handled_total: {e}"); + std::process::abort() + }) +}); + +/// Histogram for gRPC request handling duration +static GRPC_HANDLING_SECONDS: Lazy = Lazy::new(|| { + register_histogram_vec!( + "grpc_server_handling_seconds", + "Histogram of gRPC request handling duration in seconds", + &["service", "grpc_method"], + vec![0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0] + ) + .unwrap_or_else(|e| { + eprintln!("FATAL: failed to register grpc_server_handling_seconds: {e}"); + std::process::abort() + }) +}); + +/// Initialize gRPC metrics (force lazy registration) +pub fn init_grpc_metrics() { + let _ = &*GRPC_STARTED; + let _ = &*GRPC_HANDLED; + let _ = &*GRPC_HANDLING_SECONDS; +} + +/// Tower Layer that instruments gRPC handlers with Prometheus metrics +#[derive(Clone)] +pub struct GrpcMetricsLayer { + service_name: &'static str, +} + +impl GrpcMetricsLayer { + /// Create a new gRPC metrics layer for the given service + pub fn new(service_name: &'static str) -> Self { + init_grpc_metrics(); + Self { service_name } + } +} + +impl Layer for GrpcMetricsLayer { + type Service = GrpcMetricsService; + + fn layer(&self, inner: S) -> Self::Service { + GrpcMetricsService { + inner, + service_name: self.service_name, + } + } +} + +/// Tower Service that records gRPC metrics +#[derive(Clone)] +pub struct GrpcMetricsService { + inner: S, + service_name: &'static str, +} + +impl Service> for GrpcMetricsService +where + S: Service, Response = http::Response> + + Clone + + Send + + 'static, + S::Future: Send + 'static, + S::Error: Into>, +{ + type Response = S::Response; + type Error = S::Error; + type Future = Pin> + Send>>; + + fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll> { + self.inner.poll_ready(cx) + } + + fn call(&mut self, req: http::Request) -> Self::Future { + let service_name = self.service_name; + let method = req + .uri() + .path() + .rsplit('/') + .next() + .unwrap_or("unknown") + .to_string(); + + GRPC_STARTED + .with_label_values(&[service_name, &method]) + .inc(); + + let start = std::time::Instant::now(); + let mut inner = self.inner.clone(); + + Box::pin(async move { + let result = inner.call(req).await; + let elapsed = start.elapsed().as_secs_f64(); + + let code = match &result { + Ok(resp) => { + // Extract gRPC status from the grpc-status header + resp.headers() + .get("grpc-status") + .and_then(|v| v.to_str().ok()) + .unwrap_or("0") + .to_string() + }, + Err(_) => "13".to_string(), // Internal error + }; + + GRPC_HANDLED + .with_label_values(&[service_name, &method, &code]) + .inc(); + + GRPC_HANDLING_SECONDS + .with_label_values(&[service_name, &method]) + .observe(elapsed); + + result + }) + } +} +``` + +**Step 2: Add to mod.rs** + +In `crates/common/src/metrics/mod.rs`, add after line 6 (`pub mod registry;`): + +```rust +pub mod grpc_metrics; +``` + +And add to the re-exports: + +```rust +pub use grpc_metrics::{GrpcMetricsLayer, init_grpc_metrics}; +``` + +**Step 3: Build check** + +Run: `SQLX_OFFLINE=true cargo check -p common` +Expected: Compiles clean + +**Step 4: Run existing common tests** + +Run: `SQLX_OFFLINE=true cargo test -p common --lib` +Expected: 198 tests pass + +**Step 5: Commit** + +```bash +git add crates/common/src/metrics/grpc_metrics.rs crates/common/src/metrics/mod.rs +git commit -m "feat(common): add gRPC metrics Tower layer for automatic instrumentation" +``` + +--- + +### Task 9: Wire gRPC Metrics Layer into Services + +**Files:** +- Modify: `services/trading_service/src/main.rs` +- Modify: `services/api_gateway/src/main.rs` +- Modify: `services/ml_training_service/src/main.rs` +- Modify: `services/backtesting_service/src/main.rs` +- Modify: `services/broker_gateway_service/src/main.rs` +- Modify: `services/data_acquisition_service/src/main.rs` +- Modify: `services/trading_agent_service/src/main.rs` +- Modify: `crates/web-gateway/src/main.rs` (N/A — web-gateway is HTTP, not gRPC) + +**Context:** Each gRPC service uses `tonic::transport::Server::builder()` to construct the server. The Tower layer must be added via `.layer()` BEFORE `.add_service()`. Only 7 services use gRPC; web-gateway is HTTP-only and skipped. + +**Step 1: Add `.layer(GrpcMetricsLayer::new("service_name"))` to each service** + +For each service, find the `Server::builder()` chain and add the layer. The pattern is: + +```rust +use common::metrics::GrpcMetricsLayer; + +Server::builder() + .layer(GrpcMetricsLayer::new("service-name")) + .add_service(...) + .serve_with_shutdown(addr, shutdown_signal()) + .await?; +``` + +Service names to use (matching K8s service names): +- `services/trading_service/src/main.rs` → `"trading-service"` +- `services/api_gateway/src/main.rs` → `"api-gateway"` +- `services/ml_training_service/src/main.rs` → `"ml-training-service"` +- `services/backtesting_service/src/main.rs` → `"backtesting-service"` +- `services/broker_gateway_service/src/main.rs` → `"broker-gateway"` +- `services/data_acquisition_service/src/main.rs` → `"data-acquisition-service"` +- `services/trading_agent_service/src/main.rs` → `"trading-agent-service"` + +**Important:** For `ml_training_service`, the server builder is spread across lines 457-489 with optional TLS. The layer should be added right after `Server::builder()` (line 457): + +```rust +let mut server_builder = Server::builder() + .layer(GrpcMetricsLayer::new("ml-training-service")); +``` + +**Step 2: Build check** + +Run: `SQLX_OFFLINE=true cargo check --workspace` +Expected: Compiles clean + +**Step 3: Commit** + +```bash +git add services/*/src/main.rs +git commit -m "feat(services): wire gRPC metrics Tower layer into all 7 gRPC services" +``` + +--- + +## Layer 4: Training Pushgateway + +### Task 10: Deploy Prometheus Pushgateway + +**Files:** +- Create: `infra/k8s/monitoring/pushgateway.yaml` + +**Context:** The Pushgateway receives metrics from short-lived training jobs via HTTP POST and holds them for Prometheus to scrape. Deploy it in the foxhunt namespace. + +**Step 1: Create `infra/k8s/monitoring/pushgateway.yaml`** + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: pushgateway + namespace: foxhunt + labels: + app.kubernetes.io/name: pushgateway + app.kubernetes.io/part-of: monitoring +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: pushgateway + template: + metadata: + labels: + app.kubernetes.io/name: pushgateway + spec: + nodeSelector: + k8s.scaleway.com/pool-name: services + containers: + - name: pushgateway + image: prom/pushgateway:v1.11.0 + args: + - "--persistence.interval=5m" + ports: + - containerPort: 9091 + name: http + resources: + requests: + cpu: 50m + memory: 32Mi + limits: + cpu: 200m + memory: 128Mi + livenessProbe: + httpGet: + path: /-/healthy + port: 9091 + initialDelaySeconds: 5 + readinessProbe: + httpGet: + path: /-/ready + port: 9091 + initialDelaySeconds: 5 +--- +apiVersion: v1 +kind: Service +metadata: + name: pushgateway + namespace: foxhunt + labels: + app.kubernetes.io/name: pushgateway +spec: + type: ClusterIP + ports: + - port: 9091 + targetPort: 9091 + name: http + selector: + app.kubernetes.io/name: pushgateway +``` + +**Step 2: Commit** + +```bash +git add infra/k8s/monitoring/pushgateway.yaml +git commit -m "infra: add Prometheus Pushgateway for training job metrics" +``` + +--- + +### Task 11: Add Pushgateway Scrape Config to Prometheus + +**Files:** +- Modify: `infra/k8s/monitoring/prometheus-values.yaml` + +**Context:** Prometheus needs a scrape job for the pushgateway. Use `honor_labels: true` so that pushed labels (like `model` and `job_id`) are preserved instead of being overwritten by Prometheus relabeling. + +**Step 1: Add pushgateway scrape config** + +At the end of `infra/k8s/monitoring/prometheus-values.yaml` (after the promtail job, line 166), add: + +```yaml + + # --- Pushgateway (training job metrics) --- + - job_name: pushgateway + honor_labels: true + static_configs: + - targets: ['pushgateway.foxhunt.svc.cluster.local:9091'] +``` + +**Step 2: Commit** + +```bash +git add infra/k8s/monitoring/prometheus-values.yaml +git commit -m "infra: add pushgateway scrape config to Prometheus" +``` + +--- + +### Task 12: Add Training Metrics Push to ML Training Binaries + +**Files:** +- Create: `crates/ml/src/training/push_metrics.rs` +- Modify: `crates/ml/src/training/mod.rs` (add `pub mod push_metrics;`) +- Modify: `crates/ml/Cargo.toml` (add `reqwest` if not present) + +**Context:** The 5 training binaries (`train_baseline_rl`, `train_baseline_supervised`, `evaluate_baseline`, `hyperopt_baseline_rl`, `hyperopt_baseline_supervised`) need to push epoch-level metrics to the Pushgateway during training. The push happens via HTTP POST to `http://pushgateway:9091/metrics/job//model/`. + +Prometheus Pushgateway uses a simple text format. We push the metric families as Prometheus exposition text via `PUT /metrics/job/{job}/model/{model}`. + +**Step 1: Create `crates/ml/src/training/push_metrics.rs`** + +```rust +//! Prometheus Pushgateway integration for training job metrics +//! +//! Training jobs are short-lived K8s Jobs. They push epoch-level metrics +//! to the Pushgateway so Prometheus can scrape them persistently. + +use std::fmt::Write; + +/// Pushgateway client for training metrics +pub struct TrainingMetricsPusher { + pushgateway_url: String, + job_id: String, + model_name: String, + client: reqwest::Client, +} + +impl TrainingMetricsPusher { + /// Create a new pusher. Falls back to localhost if PUSHGATEWAY_URL not set. + pub fn new(job_id: &str, model_name: &str) -> Self { + let pushgateway_url = std::env::var("PUSHGATEWAY_URL") + .unwrap_or_else(|_| "http://pushgateway.foxhunt.svc.cluster.local:9091".to_string()); + Self { + pushgateway_url, + job_id: job_id.to_string(), + model_name: model_name.to_string(), + client: reqwest::Client::new(), + } + } + + /// Push current training state to Pushgateway. + /// Non-fatal: logs errors but never panics. + pub async fn push(&self, state: &TrainingState) { + let mut body = String::new(); + + // Epoch progress + let _ = writeln!(body, "# HELP foxhunt_training_current_epoch Current training epoch"); + let _ = writeln!(body, "# TYPE foxhunt_training_current_epoch gauge"); + let _ = writeln!(body, "foxhunt_training_current_epoch {}", state.epoch); + + // Loss + let _ = writeln!(body, "# HELP foxhunt_training_epoch_loss Training loss value"); + let _ = writeln!(body, "# TYPE foxhunt_training_epoch_loss gauge"); + let _ = writeln!(body, "foxhunt_training_epoch_loss{{split=\"train\"}} {}", state.train_loss); + if let Some(val_loss) = state.val_loss { + let _ = writeln!(body, "foxhunt_training_epoch_loss{{split=\"val\"}} {}", val_loss); + } + + // Accuracy + if let Some(accuracy) = state.accuracy { + let _ = writeln!(body, "# HELP foxhunt_training_eval_accuracy Model evaluation accuracy"); + let _ = writeln!(body, "# TYPE foxhunt_training_eval_accuracy gauge"); + let _ = writeln!(body, "foxhunt_training_eval_accuracy {}", accuracy); + } + + // Batches processed + let _ = writeln!(body, "# HELP foxhunt_training_batches_processed Total batches processed"); + let _ = writeln!(body, "# TYPE foxhunt_training_batches_processed counter"); + let _ = writeln!(body, "foxhunt_training_batches_processed {}", state.batches_processed); + + // Training speed + if state.batches_per_second > 0.0 { + let _ = writeln!(body, "# HELP foxhunt_training_batches_per_second Training throughput"); + let _ = writeln!(body, "# TYPE foxhunt_training_batches_per_second gauge"); + let _ = writeln!(body, "foxhunt_training_batches_per_second {}", state.batches_per_second); + } + + // Learning rate + let _ = writeln!(body, "# HELP foxhunt_training_learning_rate Current learning rate"); + let _ = writeln!(body, "# TYPE foxhunt_training_learning_rate gauge"); + let _ = writeln!(body, "foxhunt_training_learning_rate {}", state.learning_rate); + + // NaN / gradient events + if state.nan_count > 0 { + let _ = writeln!(body, "# HELP foxhunt_training_nan_detected_total NaN gradient events"); + let _ = writeln!(body, "# TYPE foxhunt_training_nan_detected_total counter"); + let _ = writeln!(body, "foxhunt_training_nan_detected_total {}", state.nan_count); + } + if state.gradient_explosion_count > 0 { + let _ = writeln!(body, "# HELP foxhunt_training_gradient_explosion_total Gradient clipping events"); + let _ = writeln!(body, "# TYPE foxhunt_training_gradient_explosion_total counter"); + let _ = writeln!(body, "foxhunt_training_gradient_explosion_total {}", state.gradient_explosion_count); + } + + // Checkpoint saves + let _ = writeln!(body, "# HELP foxhunt_training_checkpoint_saves_total Checkpoints saved"); + let _ = writeln!(body, "# TYPE foxhunt_training_checkpoint_saves_total counter"); + let _ = writeln!(body, "foxhunt_training_checkpoint_saves_total {}", state.checkpoint_saves); + + let url = format!( + "{}/metrics/job/{}/model/{}", + self.pushgateway_url, self.job_id, self.model_name + ); + + match self.client.put(&url).body(body).send().await { + Ok(resp) if resp.status().is_success() => {}, + Ok(resp) => { + eprintln!( + "Pushgateway returned {}: {}", + resp.status(), + resp.text().await.unwrap_or_default() + ); + }, + Err(e) => { + eprintln!("Failed to push metrics to Pushgateway: {e}"); + }, + } + } + + /// Delete metrics for this job from Pushgateway (call on training completion) + pub async fn cleanup(&self) { + let url = format!( + "{}/metrics/job/{}/model/{}", + self.pushgateway_url, self.job_id, self.model_name + ); + let _ = self.client.delete(&url).send().await; + } +} + +/// Current training state to push +pub struct TrainingState { + pub epoch: u64, + pub train_loss: f64, + pub val_loss: Option, + pub accuracy: Option, + pub batches_processed: u64, + pub batches_per_second: f64, + pub learning_rate: f64, + pub nan_count: u64, + pub gradient_explosion_count: u64, + pub checkpoint_saves: u64, +} + +impl Default for TrainingState { + fn default() -> Self { + Self { + epoch: 0, + train_loss: 0.0, + val_loss: None, + accuracy: None, + batches_processed: 0, + batches_per_second: 0.0, + learning_rate: 0.001, + nan_count: 0, + gradient_explosion_count: 0, + checkpoint_saves: 0, + } + } +} +``` + +**Step 2: Add to training mod.rs** + +In `crates/ml/src/training/mod.rs`, add: + +```rust +pub mod push_metrics; +``` + +**Step 3: Verify reqwest is available** + +Check if `crates/ml/Cargo.toml` already has `reqwest`. If not, add: + +```toml +reqwest = { workspace = true, features = ["rustls-tls"] } +``` + +**Step 4: Build check** + +Run: `SQLX_OFFLINE=true cargo check -p ml` +Expected: Compiles clean + +**Step 5: Commit** + +```bash +git add crates/ml/src/training/push_metrics.rs crates/ml/src/training/mod.rs crates/ml/Cargo.toml +git commit -m "feat(ml): add Pushgateway metrics client for training jobs" +``` + +--- + +### Task 13: Deploy and Verify + +**Files:** +- No code changes — cluster operations only + +**Step 1: Apply all K8s changes** + +```bash +# Deploy pushgateway +kubectl apply -f infra/k8s/monitoring/pushgateway.yaml + +# Apply updated web-gateway with metrics port +kubectl apply -f infra/k8s/services/web-gateway.yaml + +# Apply updated service YAMLs (all services) +kubectl apply -f infra/k8s/services/ + +# Deploy updated dashboard ConfigMaps +PROM_UID="PBFA97CFB590B2093" +LOKI_UID="P8E80F9AEF21F6940" +TEMPO_UID="tempo" +DASH_DIR="infra/k8s/monitoring/dashboards" +TMP_DIR=$(mktemp -d) +for f in "$DASH_DIR"/*.json; do + python3 -c " +import json +with open('$f') as fh: + d = json.load(fh) +d.pop('__inputs', None) +d.pop('__requires', None) +raw = json.dumps(d) +raw = raw.replace('\${DS_PROMETHEUS}', '${PROM_UID}') +raw = raw.replace('\${DS_LOKI}', '${LOKI_UID}') +raw = raw.replace('\${DS_TEMPO}', '${TEMPO_UID}') +print(raw) +" > "$TMP_DIR/$(basename "$f")" +done + +# Build ConfigMaps with folder annotations +declare -A GROUPS=( + [grafana-dashboards-foxhunt]="foxhunt-logs.json foxhunt-overview.json foxhunt-services.json foxhunt-cockpit.json foxhunt-trading-cockpit.json foxhunt-training-cockpit.json foxhunt-traces.json" + [grafana-dashboards-infra]="cluster-overview.json gpu-overview.json foxhunt-gpu-training.json foxhunt-infrastructure.json" + [grafana-dashboards-cicd]="foxhunt-ci-pipelines.json gitlab-services.json" +) +declare -A FOLDERS=( + [grafana-dashboards-foxhunt]="Foxhunt" + [grafana-dashboards-infra]="Infrastructure" + [grafana-dashboards-cicd]="CI/CD" +) +for cm_name in "${!GROUPS[@]}"; do + args="" + for df in ${GROUPS[$cm_name]}; do + [ -f "$TMP_DIR/$df" ] && args="$args --from-file=$df=$TMP_DIR/$df" + done + if [ -n "$args" ]; then + kubectl create configmap "$cm_name" -n foxhunt $args \ + --dry-run=client -o yaml | \ + kubectl label -f - --dry-run=client -o yaml --local grafana_dashboard=1 | \ + kubectl annotate -f - --dry-run=client -o yaml --local grafana_folder="${FOLDERS[$cm_name]}" | \ + kubectl apply -f - + fi +done +rm -rf "$TMP_DIR" + +# Restart Grafana to reload dashboards +kubectl rollout restart deployment grafana -n foxhunt + +# Upgrade Prometheus with pushgateway scrape config +helm upgrade gitlab gitlab/gitlab -n foxhunt --reuse-values \ + -f infra/k8s/monitoring/prometheus-values.yaml + +# Rolling restart services to pick up new binaries +for svc in trading-service api-gateway broker-gateway data-acquisition-service ml-training-service backtesting-service trading-agent-service web-gateway; do + kubectl rollout restart deployment "$svc" -n foxhunt 2>/dev/null || true +done +``` + +**Step 2: Verify Pushgateway is running** + +```bash +kubectl get pods -n foxhunt -l app.kubernetes.io/name=pushgateway +``` +Expected: `pushgateway-xxx 1/1 Running` + +**Step 3: Verify dashboards are in Foxhunt folder** + +```bash +kubectl port-forward deployment/grafana 3000:3000 -n foxhunt & +GRAFANA_PASS=$(kubectl get secret grafana -n foxhunt -o jsonpath='{.data.admin-password}' | base64 -d) +curl -s -u "admin:${GRAFANA_PASS}" "http://localhost:3000/api/search?type=dash-db" | python3 -c " +import json, sys +for d in json.load(sys.stdin): + print(f\"{d['uid']:40s} {d.get('folderTitle','General'):20s} {d['title']}\") +" +kill %1 +``` + +**Step 4: Push and commit** + +```bash +git push origin main +```