# Real Metrics Dashboard Overhaul — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Replace all placeholder PromQL queries in 4 Grafana dashboards with real metric names, add metrics servers to data-acquisition and web-gateway, add a shared gRPC metrics tower layer, and deploy a Pushgateway for training job metrics.

**Architecture:** Four layers executed sequentially. Layer 1 fixes dashboard JSON queries to match real Prometheus metric names. Layer 2 adds Axum-based metrics HTTP servers to the two services missing them. Layer 3 creates a reusable Tower middleware in the `common` crate that automatically instruments gRPC handlers with request count and latency histograms, then wires it into all 8 services. Layer 4 deploys a Prometheus Pushgateway and integrates it into the training binaries for epoch-level metrics.

**Tech Stack:** Rust (prometheus crate, once_cell::sync::Lazy, axum, tower), Grafana dashboard JSON, Kubernetes YAML, Prometheus Pushgateway

**Build command:** `SQLX_OFFLINE=true cargo check --workspace`
**Test command:** `SQLX_OFFLINE=true cargo test -p <crate> --lib`
**Clippy:** `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]`

---

## Layer 1: Fix Dashboard Queries

### Task 1: Fix foxhunt-trading-cockpit.json PromQL Queries

**Files:**
- Modify: `infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json`

**Context:** The Trading Cockpit has 17 panels across 5 rows. Every panel uses placeholder metric names. The real metrics are defined in `services/trading_service/src/metrics.rs` with `Lazy<CounterVec/GaugeVec/HistogramVec>` statics. The exact Prometheus metric names (lowercase, underscored) are what gets exported.

**Step 1: Replace all PromQL expressions**

For each panel, use this mapping of current → correct query:

| Panel Title | Current `expr` | Correct `expr` |
|---|---|---|
| Ensemble Agreement | `ml_ensemble_agreement_rate` | `ml_ensemble_agreement_rate{symbol=~"$symbol"}` |
| Prediction Confidence | `ml_prediction_confidence` | `histogram_quantile(0.5, rate(ml_predictions_confidence_bucket[5m]))` |
| Model Accuracy | `ml_model_accuracy` | `ml_prediction_accuracy{model_id=~"$model"}` |
| Ensemble Sharpe | `ml_ensemble_sharpe` | `ml_model_sharpe_ratio{model_id="ensemble"}` |
| Active Positions | `foxhunt_active_positions` | `trading_agent_assets_selected` |
| Realized PnL | `foxhunt_realized_pnl` | `ml_model_cumulative_pnl{model_id=~"$model"}` |
| Unrealized PnL | `foxhunt_unrealized_pnl` | `trading_agent_portfolio_value_usd` |
| Max Drawdown | `foxhunt_max_drawdown` | `ml_model_max_drawdown{model_id=~"$model"}` |
| PnL Over Time | `foxhunt_pnl_total` | `ml_model_cumulative_pnl{model_id=~"$model"}` |
| Portfolio VaR | `foxhunt_portfolio_var` | `ml_model_max_drawdown{model_id="ensemble"}` |
| Circuit Breaker | `foxhunt_circuit_breaker_state` | `ml_circuit_breaker_state{model_id=~"$model"}` |
| Risk Alerts | `foxhunt_risk_alerts_total` | `rate(ml_alerts_total{severity="critical"}[5m])` |
| Position HHI | `foxhunt_position_hhi` | `ml_ensemble_agreement_rate{symbol=~"$symbol"}` |
| Inference Latency p95 | `foxhunt_inference_latency_seconds` | `histogram_quantile(0.95, rate(ml_model_inference_latency_bucket[5m]))` |
| Model Drift | `foxhunt_model_drift` | `ml_model_drift_score{model_id=~"$model"}` |
| gRPC Latency p95 | `foxhunt_grpc_latency_seconds` | `histogram_quantile(0.95, rate(grpc_server_handling_seconds_bucket{service=~"$service"}[5m]))` |
| ML Inference Distribution | `foxhunt_ml_inference_duration_seconds` | `rate(ml_model_inference_latency_bucket[5m])` |

Also add two template variables to the dashboard `templating.list`:

```json
{
  "name": "model",
  "type": "query",
  "query": "label_values(ml_prediction_accuracy, model_id)",
  "datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
  "refresh": 2,
  "includeAll": true,
  "multi": true,
  "current": {"text": "All", "value": "$__all"}
},
{
  "name": "symbol",
  "type": "query",
  "query": "label_values(ml_ensemble_agreement_rate, symbol)",
  "datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
  "refresh": 2,
  "includeAll": true,
  "multi": true,
  "current": {"text": "All", "value": "$__all"}
}
```

**Step 2: Validate JSON is valid**

Run: `python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json'))"`
Expected: No output (valid JSON)

**Step 3: Commit**

```bash
git add infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json
git commit -m "fix(dashboard): use real metric names in trading cockpit"
```

---

### Task 2: Fix foxhunt-training-cockpit.json PromQL Queries

**Files:**
- Modify: `infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json`

**Context:** The Training Cockpit has 28 panels across 8 rows. Many panels query metrics that come from kube-state-metrics (job status), DCGM exporter (GPU), ml-training-service (job counters), and Pushgateway (epoch-level, Layer 4). For metrics that require Pushgateway (Layer 4), use the correct metric name now — they'll show "No data" until the Pushgateway is deployed.

**Step 1: Replace all PromQL expressions**

| Panel Title | Current `expr` | Correct `expr` |
|---|---|---|
| Running Jobs | `foxhunt_training_jobs_active` | `kube_job_status_active{namespace="foxhunt", job_name=~"training-.*"}` |
| Active Workers | `foxhunt_training_active_workers` | `ml_training_active_workers` |
| Jobs Started | `foxhunt_training_jobs_started_total` | `ml_training_jobs_started_total` |
| Jobs Completed | `foxhunt_training_jobs_completed_total` | `ml_training_jobs_completed_total` |
| Failed Jobs | `foxhunt_training_jobs_failed_total` | `ml_training_errors_total` |
| Pending Jobs | `foxhunt_training_jobs_pending` | `kube_job_status_active{namespace="foxhunt", job_name=~"training-.*"} - kube_job_status_succeeded{namespace="foxhunt", job_name=~"training-.*"}` |
| Loss Curves | `foxhunt_training_loss` | `foxhunt_training_epoch_loss{model=~"$model_type"}` |
| Epoch Progress | `foxhunt_training_epoch` | `foxhunt_training_current_epoch{model=~"$model_type"}` |
| Convergence Rate | `foxhunt_training_convergence_rate` | `deriv(foxhunt_training_epoch_loss{model=~"$model_type"}[10m])` |
| Training Speed | `foxhunt_training_steps_per_second` | `foxhunt_training_batches_per_second{model=~"$model_type"}` |
| GPU Utilization | `foxhunt_gpu_utilization` | `DCGM_FI_DEV_GPU_UTIL` |
| GPU Memory | `foxhunt_gpu_memory_used` | `DCGM_FI_DEV_FB_USED` |
| GPU Temperature | `foxhunt_gpu_temperature` | `DCGM_FI_DEV_GPU_TEMP` |
| GPU Power | `foxhunt_gpu_power` | `DCGM_FI_DEV_POWER_USAGE` |
| Accuracy | `foxhunt_training_accuracy` | `foxhunt_training_eval_accuracy{model=~"$model_type"}` |
| F1 / Precision / Recall | `foxhunt_training_f1` | `foxhunt_training_eval_f1{model=~"$model_type"}` |
| Total Saves | `foxhunt_checkpoint_saves_total` | `foxhunt_training_checkpoint_saves_total{model=~"$model_type"}` |
| Save Failures | `foxhunt_checkpoint_failures` | `foxhunt_training_checkpoint_failures_total{model=~"$model_type"}` |
| Save Duration p95 | `foxhunt_checkpoint_duration_seconds` | `histogram_quantile(0.95, rate(foxhunt_training_checkpoint_duration_seconds_bucket[5m]))` |
| Checkpoint Size | `foxhunt_checkpoint_size_bytes` | `foxhunt_training_checkpoint_size_bytes{model=~"$model_type"}` |
| Batches Processed | `foxhunt_training_batches_total` | `foxhunt_training_batches_processed{model=~"$model_type"}` |
| Data Loading Latency | `foxhunt_data_loading_latency` | `histogram_quantile(0.95, rate(foxhunt_training_data_load_seconds_bucket[5m]))` |
| Feature Errors | `foxhunt_training_feature_errors` | `foxhunt_training_feature_errors_total{model=~"$model_type"}` |
| NaN Detections | `foxhunt_training_nan_detected` | `foxhunt_training_nan_detected_total{model=~"$model_type"}` |
| Gradient Explosions | `foxhunt_training_gradient_explosions` | `foxhunt_training_gradient_explosion_total{model=~"$model_type"}` |
| Training Failures | `foxhunt_training_failures` | `ml_training_errors_total` |
| Job Duration Distribution | `foxhunt_training_job_duration` | `(kube_job_complete_time{namespace="foxhunt", job_name=~"training-.*"} - kube_job_start_time{namespace="foxhunt", job_name=~"training-.*"}) / 3600` |
| Iteration Duration | `foxhunt_training_iteration_duration` | `foxhunt_training_iteration_seconds{model=~"$model_type"}` |

**Step 2: Validate JSON**

Run: `python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json'))"`

**Step 3: Commit**

```bash
git add infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json
git commit -m "fix(dashboard): use real metric names in training cockpit"
```

---

### Task 3: Fix foxhunt-traces.json PromQL Queries

**Files:**
- Modify: `infra/k8s/monitoring/dashboards/foxhunt-traces.json`

**Context:** The Traces dashboard has 8 panels. Overview stats should query Tempo distributor metrics (which Tempo exposes on port 3200 and Prometheus scrapes). Service Map uses Tempo datasource directly. Trace search uses Tempo TraceQL.

**Step 1: Replace all PromQL expressions**

| Panel Title | Current `expr` | Correct `expr` |
|---|---|---|
| Spans/sec | `foxhunt_spans_total` | `sum(rate(tempo_distributor_spans_received_total[5m]))` |
| Error Rate | `foxhunt_trace_errors_total` | `sum(rate(tempo_distributor_spans_received_total{status="error"}[5m])) / sum(rate(tempo_distributor_spans_received_total[5m]))` |
| Avg Duration | `foxhunt_span_duration_seconds` | `sum(rate(tempo_spanmetrics_latency_sum[5m])) / sum(rate(tempo_spanmetrics_latency_count[5m]))` |
| Active Services | `foxhunt_active_services` | `count(count by (service_name)(tempo_distributor_spans_received_total))` |
| Latency by Service p95 | `foxhunt_span_duration_seconds{quantile="0.95"}` | `histogram_quantile(0.95, sum by (le, service_name)(rate(tempo_spanmetrics_latency_bucket[5m])))` |
| Error Rate by Service | `foxhunt_trace_error_rate` | `sum by (service_name)(rate(tempo_distributor_spans_received_total{status="error"}[5m]))` |

For the Service Map panel (nodeGraph type), ensure it uses the Tempo datasource (`${DS_TEMPO}`) with query type `serviceMap`.

For the Recent Traces panel (table type), ensure it uses Tempo datasource with TraceQL query: `{status = error} || {duration > 1s}`.

**Step 2: Validate JSON**

Run: `python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-traces.json'))"`

**Step 3: Commit**

```bash
git add infra/k8s/monitoring/dashboards/foxhunt-traces.json
git commit -m "fix(dashboard): use real Tempo metric names in traces dashboard"
```

---

### Task 4: Fix foxhunt-cockpit.json Trace Health Row

**Files:**
- Modify: `infra/k8s/monitoring/dashboards/foxhunt-cockpit.json`

**Context:** The Cockpit dashboard's Trace Health row (panels 35-37, added in the OTEL task) needs the same Tempo metric fixes as Task 3.

**Step 1: Fix the 3 Trace Health panels**

| Panel Title | Current `expr` | Correct `expr` |
|---|---|---|
| Spans/sec | `foxhunt_spans_total` (if placeholder) | `sum(rate(tempo_distributor_spans_received_total[5m]))` |
| Trace Error Rate | `foxhunt_trace_error_rate` (if placeholder) | `sum(rate(tempo_distributor_spans_received_total{status="error"}[5m])) / sum(rate(tempo_distributor_spans_received_total[5m]))` |
| Spans/sec by Service | `foxhunt_spans_by_service` (if placeholder) | `sum by (service_name)(rate(tempo_distributor_spans_received_total[5m]))` |

**Step 2: Validate JSON**

Run: `python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-cockpit.json'))"`

**Step 3: Commit**

```bash
git add infra/k8s/monitoring/dashboards/foxhunt-cockpit.json
git commit -m "fix(dashboard): use real Tempo metrics in cockpit trace health row"
```

---

## Layer 2: Add Metrics Servers to Missing Services

### Task 5: Add Prometheus Metrics Server to data-acquisition-service

**Files:**
- Create: `services/data_acquisition_service/src/metrics.rs`
- Modify: `services/data_acquisition_service/src/main.rs`
- Modify: `services/data_acquisition_service/src/lib.rs` (add `pub mod metrics;`)

**Context:** data-acquisition-service already has `prometheus`, `once_cell`, and `axum` in Cargo.toml. The K8s YAML already declares port 9097 with prometheus annotations. The only thing missing is the actual metrics server code. Follow the ml-training-service pattern (inline metrics handler in main.rs spawned via `tokio::spawn`).

**Step 1: Create `services/data_acquisition_service/src/metrics.rs`**

```rust
//! Prometheus metrics for Data Acquisition Service

use once_cell::sync::Lazy;
use prometheus::{register_counter, register_counter_vec, register_gauge, register_histogram_vec, Counter, CounterVec, Gauge, HistogramVec};

/// Service uptime in seconds
pub static SERVICE_UPTIME: Lazy<Gauge> = Lazy::new(|| {
    register_gauge!(
        "foxhunt_data_acquisition_uptime_seconds",
        "Service uptime in seconds"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register uptime gauge: {e}");
        std::process::abort()
    })
});

/// Active data feeds
pub static FEEDS_ACTIVE: Lazy<Gauge> = Lazy::new(|| {
    register_gauge!(
        "foxhunt_data_acquisition_feeds_active",
        "Number of active data feeds"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register feeds_active gauge: {e}");
        std::process::abort()
    })
});

/// Total records received
pub static RECORDS_RECEIVED: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "foxhunt_data_acquisition_records_received_total",
        "Total records received from data feeds",
        &["symbol", "feed_type"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register records_received counter: {e}");
        std::process::abort()
    })
});

/// Total errors
pub static ERRORS: Lazy<Counter> = Lazy::new(|| {
    register_counter!(
        "foxhunt_data_acquisition_errors_total",
        "Total data acquisition errors"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register errors counter: {e}");
        std::process::abort()
    })
});

/// Feed processing latency
pub static FEED_LATENCY: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "foxhunt_data_acquisition_feed_latency_seconds",
        "Feed processing latency in seconds",
        &["feed_type"],
        vec![0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register feed_latency histogram: {e}");
        std::process::abort()
    })
});

/// Initialize all metrics (force lazy registration)
pub fn init_metrics() {
    let _ = &*SERVICE_UPTIME;
    let _ = &*FEEDS_ACTIVE;
    let _ = &*RECORDS_RECEIVED;
    let _ = &*ERRORS;
    let _ = &*FEED_LATENCY;
}

/// Update service uptime metric
pub fn update_uptime(start_time: std::time::Instant) {
    SERVICE_UPTIME.set(start_time.elapsed().as_secs_f64());
}
```

**Step 2: Add `pub mod metrics;` to lib.rs**

In `services/data_acquisition_service/src/lib.rs`, add:
```rust
pub mod metrics;
```

**Step 3: Add metrics server + uptime updater to main.rs**

In `services/data_acquisition_service/src/main.rs`, after the `info!("Data Acquisition Service listening on {}", addr);` line (line 52) and before the `Server::builder()` block (line 55), add:

```rust
    // Initialize Prometheus metrics
    data_acquisition_service::metrics::init_metrics();
    let service_start = std::time::Instant::now();

    // Spawn uptime updater
    tokio::spawn(async move {
        let mut interval = tokio::time::interval(std::time::Duration::from_secs(1));
        loop {
            interval.tick().await;
            data_acquisition_service::metrics::update_uptime(service_start);
        }
    });

    // Start Prometheus metrics HTTP endpoint on port 9097
    let metrics_port: u16 = std::env::var("METRICS_PORT")
        .ok()
        .and_then(|s| s.parse().ok())
        .unwrap_or(9097);
    tokio::spawn(async move {
        use axum::{routing::get, Router};
        use prometheus::{Encoder, TextEncoder};

        async fn metrics_handler() -> String {
            let encoder = TextEncoder::new();
            let metric_families = prometheus::gather();
            let mut buffer = vec![];
            let _ = encoder.encode(&metric_families, &mut buffer);
            String::from_utf8(buffer).unwrap_or_else(|_| String::new())
        }

        let app = Router::new().route("/metrics", get(metrics_handler));
        let addr = format!("0.0.0.0:{}", metrics_port);
        info!("Prometheus metrics endpoint listening on http://{}", addr);

        let listener = match tokio::net::TcpListener::bind(&addr).await {
            Ok(l) => l,
            Err(e) => {
                tracing::error!("Failed to bind metrics endpoint {}: {}", addr, e);
                return;
            },
        };
        if let Err(e) = axum::serve(listener, app).await {
            tracing::error!("Metrics server failed: {}", e);
        }
    });
```

**Step 4: Build check**

Run: `SQLX_OFFLINE=true cargo check -p data_acquisition_service`
Expected: Compiles clean

**Step 5: Commit**

```bash
git add services/data_acquisition_service/src/metrics.rs services/data_acquisition_service/src/main.rs services/data_acquisition_service/src/lib.rs
git commit -m "feat(data-acquisition): add Prometheus metrics server on port 9097"
```

---

### Task 6: Add Prometheus Metrics Server to web-gateway

**Files:**
- Add dependency: `crates/web-gateway/Cargo.toml`
- Create: `crates/web-gateway/src/metrics.rs`
- Modify: `crates/web-gateway/src/main.rs`
- Modify: `crates/web-gateway/src/lib.rs` (add `pub mod metrics;`)

**Context:** web-gateway does NOT have `prometheus` in Cargo.toml — must add it. web-gateway is an Axum HTTP server (port 3000). The metrics endpoint should run on a separate port (9098) to avoid mixing application traffic with scraping. Use the same inline pattern as ml-training-service.

**Step 1: Add prometheus dependency to Cargo.toml**

In `crates/web-gateway/Cargo.toml`, after the `common.workspace = true` line (line 44), add:

```toml
# Metrics
prometheus.workspace = true
once_cell.workspace = true
```

**Step 2: Create `crates/web-gateway/src/metrics.rs`**

```rust
//! Prometheus metrics for Web Gateway

use once_cell::sync::Lazy;
use prometheus::{register_counter_vec, register_gauge, register_histogram_vec, CounterVec, Gauge, HistogramVec};

/// Service uptime in seconds
pub static SERVICE_UPTIME: Lazy<Gauge> = Lazy::new(|| {
    register_gauge!(
        "foxhunt_web_gateway_uptime_seconds",
        "Service uptime in seconds"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register uptime gauge: {e}");
        std::process::abort()
    })
});

/// HTTP requests total
pub static HTTP_REQUESTS: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "foxhunt_web_gateway_http_requests_total",
        "Total HTTP requests",
        &["method", "path", "status"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register http_requests counter: {e}");
        std::process::abort()
    })
});

/// HTTP request duration
pub static HTTP_REQUEST_DURATION: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "foxhunt_web_gateway_http_request_duration_seconds",
        "HTTP request duration in seconds",
        &["method", "path"],
        vec![0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register http_request_duration histogram: {e}");
        std::process::abort()
    })
});

/// Active WebSocket connections
pub static WS_CONNECTIONS: Lazy<Gauge> = Lazy::new(|| {
    register_gauge!(
        "foxhunt_web_gateway_ws_connections_active",
        "Active WebSocket connections"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register ws_connections gauge: {e}");
        std::process::abort()
    })
});

/// WebSocket messages total
pub static WS_MESSAGES: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "foxhunt_web_gateway_ws_messages_total",
        "Total WebSocket messages",
        &["direction", "topic"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register ws_messages counter: {e}");
        std::process::abort()
    })
});

/// Initialize all metrics (force lazy registration)
pub fn init_metrics() {
    let _ = &*SERVICE_UPTIME;
    let _ = &*HTTP_REQUESTS;
    let _ = &*HTTP_REQUEST_DURATION;
    let _ = &*WS_CONNECTIONS;
    let _ = &*WS_MESSAGES;
}

/// Update service uptime metric
pub fn update_uptime(start_time: std::time::Instant) {
    SERVICE_UPTIME.set(start_time.elapsed().as_secs_f64());
}
```

**Step 3: Add `pub mod metrics;` to lib.rs**

In `crates/web-gateway/src/lib.rs`, add:
```rust
pub mod metrics;
```

**Step 4: Add metrics server to main.rs**

In `crates/web-gateway/src/main.rs`, after the `info!("Web gateway listening on {}", listen_addr);` line (line 79) and before the `axum::serve(` block (line 81), add:

```rust
    // Initialize Prometheus metrics
    web_gateway::metrics::init_metrics();
    let service_start = std::time::Instant::now();

    // Spawn uptime updater
    tokio::spawn(async move {
        let mut interval = tokio::time::interval(std::time::Duration::from_secs(1));
        loop {
            interval.tick().await;
            web_gateway::metrics::update_uptime(service_start);
        }
    });

    // Start Prometheus metrics HTTP endpoint on port 9098
    let metrics_port: u16 = std::env::var("METRICS_PORT")
        .ok()
        .and_then(|s| s.parse().ok())
        .unwrap_or(9098);
    tokio::spawn(async move {
        use axum::{routing::get, Router};
        use prometheus::{Encoder, TextEncoder};

        async fn metrics_handler() -> String {
            let encoder = TextEncoder::new();
            let metric_families = prometheus::gather();
            let mut buffer = vec![];
            let _ = encoder.encode(&metric_families, &mut buffer);
            String::from_utf8(buffer).unwrap_or_else(|_| String::new())
        }

        let app = Router::new().route("/metrics", get(metrics_handler));
        let addr = format!("0.0.0.0:{}", metrics_port);
        info!("Prometheus metrics endpoint listening on http://{}", addr);

        let listener = match tokio::net::TcpListener::bind(&addr).await {
            Ok(l) => l,
            Err(e) => {
                tracing::error!("Failed to bind metrics endpoint {}: {}", addr, e);
                return;
            },
        };
        if let Err(e) = axum::serve(listener, app).await {
            tracing::error!("Metrics server failed: {}", e);
        }
    });
```

**Step 5: Build check**

Run: `SQLX_OFFLINE=true cargo check -p web-gateway`
Expected: Compiles clean

**Step 6: Run existing tests**

Run: `SQLX_OFFLINE=true cargo test -p web-gateway --lib`
Expected: All 138 tests pass

**Step 7: Commit**

```bash
git add crates/web-gateway/Cargo.toml crates/web-gateway/src/metrics.rs crates/web-gateway/src/main.rs crates/web-gateway/src/lib.rs
git commit -m "feat(web-gateway): add Prometheus metrics server on port 9098"
```

---

### Task 7: Update K8s YAML for web-gateway Metrics Port

**Files:**
- Modify: `infra/k8s/services/web-gateway.yaml`

**Context:** web-gateway K8s YAML currently has only port 3000 (http). Need to add port 9098 (metrics), prometheus scrape annotations, and ensure the Service exposes the metrics port.

**Step 1: Add prometheus annotations to pod template**

In `infra/k8s/services/web-gateway.yaml`, the pod template metadata currently has only `labels:` (line 21). Add `annotations:` before `labels:`:

```yaml
    metadata:
      annotations:
        gitlab.com/prometheus_scrape: "true"
        gitlab.com/prometheus_port: "9098"
        gitlab.com/prometheus_path: "/metrics"
      labels:
        app.kubernetes.io/name: web-gateway
```

**Step 2: Add metrics container port**

After the `http` port (line 88), add:

```yaml
            - containerPort: 9098
              name: metrics
```

**Step 3: Add metrics port to Service**

Find the Service section at the bottom of the file. Add the metrics port:

```yaml
    - port: 9098
      targetPort: 9098
      name: metrics
```

**Step 4: Commit**

```bash
git add infra/k8s/services/web-gateway.yaml
git commit -m "infra(web-gateway): add metrics port 9098 and prometheus scrape annotations"
```

---

## Layer 3: gRPC Metrics Tower Layer

### Task 8: Create gRPC Metrics Tower Layer in common

**Files:**
- Create: `crates/common/src/metrics/grpc_metrics.rs`
- Modify: `crates/common/src/metrics/mod.rs` (add `pub mod grpc_metrics;`)

**Context:** We need a Tower `Layer` + `Service` that wraps every gRPC handler and automatically records `grpc_server_started_total`, `grpc_server_handled_total`, and `grpc_server_handling_seconds`. The layer extracts the gRPC method from the request URI path. `common` already depends on `prometheus`, `once_cell`, `tonic`, and `tower` (via tonic).

**Step 1: Create `crates/common/src/metrics/grpc_metrics.rs`**

```rust
//! gRPC metrics Tower layer for automatic request instrumentation
//!
//! Provides `grpc_server_started_total`, `grpc_server_handled_total`,
//! and `grpc_server_handling_seconds` metrics for all gRPC handlers.

use once_cell::sync::Lazy;
use prometheus::{
    register_counter_vec, register_histogram_vec, CounterVec, HistogramVec,
};
use std::future::Future;
use std::pin::Pin;
use std::task::{Context, Poll};
use tonic::body::BoxBody;
use tower::{Layer, Service};

/// Counter for gRPC requests started
static GRPC_STARTED: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "grpc_server_started_total",
        "Total gRPC RPCs started on the server",
        &["service", "grpc_method"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register grpc_server_started_total: {e}");
        std::process::abort()
    })
});

/// Counter for gRPC requests completed with status code
static GRPC_HANDLED: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "grpc_server_handled_total",
        "Total gRPC RPCs completed on the server with status code",
        &["service", "grpc_method", "grpc_code"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register grpc_server_handled_total: {e}");
        std::process::abort()
    })
});

/// Histogram for gRPC request handling duration
static GRPC_HANDLING_SECONDS: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "grpc_server_handling_seconds",
        "Histogram of gRPC request handling duration in seconds",
        &["service", "grpc_method"],
        vec![0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register grpc_server_handling_seconds: {e}");
        std::process::abort()
    })
});

/// Initialize gRPC metrics (force lazy registration)
pub fn init_grpc_metrics() {
    let _ = &*GRPC_STARTED;
    let _ = &*GRPC_HANDLED;
    let _ = &*GRPC_HANDLING_SECONDS;
}

/// Tower Layer that instruments gRPC handlers with Prometheus metrics
#[derive(Clone)]
pub struct GrpcMetricsLayer {
    service_name: &'static str,
}

impl GrpcMetricsLayer {
    /// Create a new gRPC metrics layer for the given service
    pub fn new(service_name: &'static str) -> Self {
        init_grpc_metrics();
        Self { service_name }
    }
}

impl<S> Layer<S> for GrpcMetricsLayer {
    type Service = GrpcMetricsService<S>;

    fn layer(&self, inner: S) -> Self::Service {
        GrpcMetricsService {
            inner,
            service_name: self.service_name,
        }
    }
}

/// Tower Service that records gRPC metrics
#[derive(Clone)]
pub struct GrpcMetricsService<S> {
    inner: S,
    service_name: &'static str,
}

impl<S> Service<http::Request<BoxBody>> for GrpcMetricsService<S>
where
    S: Service<http::Request<BoxBody>, Response = http::Response<BoxBody>>
        + Clone
        + Send
        + 'static,
    S::Future: Send + 'static,
    S::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
{
    type Response = S::Response;
    type Error = S::Error;
    type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;

    fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
        self.inner.poll_ready(cx)
    }

    fn call(&mut self, req: http::Request<BoxBody>) -> Self::Future {
        let service_name = self.service_name;
        let method = req
            .uri()
            .path()
            .rsplit('/')
            .next()
            .unwrap_or("unknown")
            .to_string();

        GRPC_STARTED
            .with_label_values(&[service_name, &method])
            .inc();

        let start = std::time::Instant::now();
        let mut inner = self.inner.clone();

        Box::pin(async move {
            let result = inner.call(req).await;
            let elapsed = start.elapsed().as_secs_f64();

            let code = match &result {
                Ok(resp) => {
                    // Extract gRPC status from the grpc-status header
                    resp.headers()
                        .get("grpc-status")
                        .and_then(|v| v.to_str().ok())
                        .unwrap_or("0")
                        .to_string()
                },
                Err(_) => "13".to_string(), // Internal error
            };

            GRPC_HANDLED
                .with_label_values(&[service_name, &method, &code])
                .inc();

            GRPC_HANDLING_SECONDS
                .with_label_values(&[service_name, &method])
                .observe(elapsed);

            result
        })
    }
}
```

**Step 2: Add to mod.rs**

In `crates/common/src/metrics/mod.rs`, add after line 6 (`pub mod registry;`):

```rust
pub mod grpc_metrics;
```

And add to the re-exports:

```rust
pub use grpc_metrics::{GrpcMetricsLayer, init_grpc_metrics};
```

**Step 3: Build check**

Run: `SQLX_OFFLINE=true cargo check -p common`
Expected: Compiles clean

**Step 4: Run existing common tests**

Run: `SQLX_OFFLINE=true cargo test -p common --lib`
Expected: 198 tests pass

**Step 5: Commit**

```bash
git add crates/common/src/metrics/grpc_metrics.rs crates/common/src/metrics/mod.rs
git commit -m "feat(common): add gRPC metrics Tower layer for automatic instrumentation"
```

---

### Task 9: Wire gRPC Metrics Layer into Services

**Files:**
- Modify: `services/trading_service/src/main.rs`
- Modify: `services/api_gateway/src/main.rs`
- Modify: `services/ml_training_service/src/main.rs`
- Modify: `services/backtesting_service/src/main.rs`
- Modify: `services/broker_gateway_service/src/main.rs`
- Modify: `services/data_acquisition_service/src/main.rs`
- Modify: `services/trading_agent_service/src/main.rs`
- Modify: `crates/web-gateway/src/main.rs` (N/A — web-gateway is HTTP, not gRPC)

**Context:** Each gRPC service uses `tonic::transport::Server::builder()` to construct the server. The Tower layer must be added via `.layer()` BEFORE `.add_service()`. Only 7 services use gRPC; web-gateway is HTTP-only and skipped.

**Step 1: Add `.layer(GrpcMetricsLayer::new("service_name"))` to each service**

For each service, find the `Server::builder()` chain and add the layer. The pattern is:

```rust
use common::metrics::GrpcMetricsLayer;

Server::builder()
    .layer(GrpcMetricsLayer::new("service-name"))
    .add_service(...)
    .serve_with_shutdown(addr, shutdown_signal())
    .await?;
```

Service names to use (matching K8s service names):
- `services/trading_service/src/main.rs` → `"trading-service"`
- `services/api_gateway/src/main.rs` → `"api-gateway"`
- `services/ml_training_service/src/main.rs` → `"ml-training-service"`
- `services/backtesting_service/src/main.rs` → `"backtesting-service"`
- `services/broker_gateway_service/src/main.rs` → `"broker-gateway"`
- `services/data_acquisition_service/src/main.rs` → `"data-acquisition-service"`
- `services/trading_agent_service/src/main.rs` → `"trading-agent-service"`

**Important:** For `ml_training_service`, the server builder is spread across lines 457-489 with optional TLS. The layer should be added right after `Server::builder()` (line 457):

```rust
let mut server_builder = Server::builder()
    .layer(GrpcMetricsLayer::new("ml-training-service"));
```

**Step 2: Build check**

Run: `SQLX_OFFLINE=true cargo check --workspace`
Expected: Compiles clean

**Step 3: Commit**

```bash
git add services/*/src/main.rs
git commit -m "feat(services): wire gRPC metrics Tower layer into all 7 gRPC services"
```

---

## Layer 4: Training Pushgateway

### Task 10: Deploy Prometheus Pushgateway

**Files:**
- Create: `infra/k8s/monitoring/pushgateway.yaml`

**Context:** The Pushgateway receives metrics from short-lived training jobs via HTTP POST and holds them for Prometheus to scrape. Deploy it in the foxhunt namespace.

**Step 1: Create `infra/k8s/monitoring/pushgateway.yaml`**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pushgateway
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: pushgateway
    app.kubernetes.io/part-of: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: pushgateway
  template:
    metadata:
      labels:
        app.kubernetes.io/name: pushgateway
    spec:
      nodeSelector:
        k8s.scaleway.com/pool-name: services
      containers:
        - name: pushgateway
          image: prom/pushgateway:v1.11.0
          args:
            - "--persistence.interval=5m"
          ports:
            - containerPort: 9091
              name: http
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              cpu: 200m
              memory: 128Mi
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9091
            initialDelaySeconds: 5
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9091
            initialDelaySeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: pushgateway
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: pushgateway
spec:
  type: ClusterIP
  ports:
    - port: 9091
      targetPort: 9091
      name: http
  selector:
    app.kubernetes.io/name: pushgateway
```

**Step 2: Commit**

```bash
git add infra/k8s/monitoring/pushgateway.yaml
git commit -m "infra: add Prometheus Pushgateway for training job metrics"
```

---

### Task 11: Add Pushgateway Scrape Config to Prometheus

**Files:**
- Modify: `infra/k8s/monitoring/prometheus-values.yaml`

**Context:** Prometheus needs a scrape job for the pushgateway. Use `honor_labels: true` so that pushed labels (like `model` and `job_id`) are preserved instead of being overwritten by Prometheus relabeling.

**Step 1: Add pushgateway scrape config**

At the end of `infra/k8s/monitoring/prometheus-values.yaml` (after the promtail job, line 166), add:

```yaml

      # --- Pushgateway (training job metrics) ---
      - job_name: pushgateway
        honor_labels: true
        static_configs:
          - targets: ['pushgateway.foxhunt.svc.cluster.local:9091']
```

**Step 2: Commit**

```bash
git add infra/k8s/monitoring/prometheus-values.yaml
git commit -m "infra: add pushgateway scrape config to Prometheus"
```

---

### Task 12: Add Training Metrics Push to ML Training Binaries

**Files:**
- Create: `crates/ml/src/training/push_metrics.rs`
- Modify: `crates/ml/src/training/mod.rs` (add `pub mod push_metrics;`)
- Modify: `crates/ml/Cargo.toml` (add `reqwest` if not present)

**Context:** The 5 training binaries (`train_baseline_rl`, `train_baseline_supervised`, `evaluate_baseline`, `hyperopt_baseline_rl`, `hyperopt_baseline_supervised`) need to push epoch-level metrics to the Pushgateway during training. The push happens via HTTP POST to `http://pushgateway:9091/metrics/job/<job_name>/model/<model_name>`.

Prometheus Pushgateway uses a simple text format. We push the metric families as Prometheus exposition text via `PUT /metrics/job/{job}/model/{model}`.

**Step 1: Create `crates/ml/src/training/push_metrics.rs`**

```rust
//! Prometheus Pushgateway integration for training job metrics
//!
//! Training jobs are short-lived K8s Jobs. They push epoch-level metrics
//! to the Pushgateway so Prometheus can scrape them persistently.

use std::fmt::Write;

/// Pushgateway client for training metrics
pub struct TrainingMetricsPusher {
    pushgateway_url: String,
    job_id: String,
    model_name: String,
    client: reqwest::Client,
}

impl TrainingMetricsPusher {
    /// Create a new pusher. Falls back to localhost if PUSHGATEWAY_URL not set.
    pub fn new(job_id: &str, model_name: &str) -> Self {
        let pushgateway_url = std::env::var("PUSHGATEWAY_URL")
            .unwrap_or_else(|_| "http://pushgateway.foxhunt.svc.cluster.local:9091".to_string());
        Self {
            pushgateway_url,
            job_id: job_id.to_string(),
            model_name: model_name.to_string(),
            client: reqwest::Client::new(),
        }
    }

    /// Push current training state to Pushgateway.
    /// Non-fatal: logs errors but never panics.
    pub async fn push(&self, state: &TrainingState) {
        let mut body = String::new();

        // Epoch progress
        let _ = writeln!(body, "# HELP foxhunt_training_current_epoch Current training epoch");
        let _ = writeln!(body, "# TYPE foxhunt_training_current_epoch gauge");
        let _ = writeln!(body, "foxhunt_training_current_epoch {}", state.epoch);

        // Loss
        let _ = writeln!(body, "# HELP foxhunt_training_epoch_loss Training loss value");
        let _ = writeln!(body, "# TYPE foxhunt_training_epoch_loss gauge");
        let _ = writeln!(body, "foxhunt_training_epoch_loss{{split=\"train\"}} {}", state.train_loss);
        if let Some(val_loss) = state.val_loss {
            let _ = writeln!(body, "foxhunt_training_epoch_loss{{split=\"val\"}} {}", val_loss);
        }

        // Accuracy
        if let Some(accuracy) = state.accuracy {
            let _ = writeln!(body, "# HELP foxhunt_training_eval_accuracy Model evaluation accuracy");
            let _ = writeln!(body, "# TYPE foxhunt_training_eval_accuracy gauge");
            let _ = writeln!(body, "foxhunt_training_eval_accuracy {}", accuracy);
        }

        // Batches processed
        let _ = writeln!(body, "# HELP foxhunt_training_batches_processed Total batches processed");
        let _ = writeln!(body, "# TYPE foxhunt_training_batches_processed counter");
        let _ = writeln!(body, "foxhunt_training_batches_processed {}", state.batches_processed);

        // Training speed
        if state.batches_per_second > 0.0 {
            let _ = writeln!(body, "# HELP foxhunt_training_batches_per_second Training throughput");
            let _ = writeln!(body, "# TYPE foxhunt_training_batches_per_second gauge");
            let _ = writeln!(body, "foxhunt_training_batches_per_second {}", state.batches_per_second);
        }

        // Learning rate
        let _ = writeln!(body, "# HELP foxhunt_training_learning_rate Current learning rate");
        let _ = writeln!(body, "# TYPE foxhunt_training_learning_rate gauge");
        let _ = writeln!(body, "foxhunt_training_learning_rate {}", state.learning_rate);

        // NaN / gradient events
        if state.nan_count > 0 {
            let _ = writeln!(body, "# HELP foxhunt_training_nan_detected_total NaN gradient events");
            let _ = writeln!(body, "# TYPE foxhunt_training_nan_detected_total counter");
            let _ = writeln!(body, "foxhunt_training_nan_detected_total {}", state.nan_count);
        }
        if state.gradient_explosion_count > 0 {
            let _ = writeln!(body, "# HELP foxhunt_training_gradient_explosion_total Gradient clipping events");
            let _ = writeln!(body, "# TYPE foxhunt_training_gradient_explosion_total counter");
            let _ = writeln!(body, "foxhunt_training_gradient_explosion_total {}", state.gradient_explosion_count);
        }

        // Checkpoint saves
        let _ = writeln!(body, "# HELP foxhunt_training_checkpoint_saves_total Checkpoints saved");
        let _ = writeln!(body, "# TYPE foxhunt_training_checkpoint_saves_total counter");
        let _ = writeln!(body, "foxhunt_training_checkpoint_saves_total {}", state.checkpoint_saves);

        let url = format!(
            "{}/metrics/job/{}/model/{}",
            self.pushgateway_url, self.job_id, self.model_name
        );

        match self.client.put(&url).body(body).send().await {
            Ok(resp) if resp.status().is_success() => {},
            Ok(resp) => {
                eprintln!(
                    "Pushgateway returned {}: {}",
                    resp.status(),
                    resp.text().await.unwrap_or_default()
                );
            },
            Err(e) => {
                eprintln!("Failed to push metrics to Pushgateway: {e}");
            },
        }
    }

    /// Delete metrics for this job from Pushgateway (call on training completion)
    pub async fn cleanup(&self) {
        let url = format!(
            "{}/metrics/job/{}/model/{}",
            self.pushgateway_url, self.job_id, self.model_name
        );
        let _ = self.client.delete(&url).send().await;
    }
}

/// Current training state to push
pub struct TrainingState {
    pub epoch: u64,
    pub train_loss: f64,
    pub val_loss: Option<f64>,
    pub accuracy: Option<f64>,
    pub batches_processed: u64,
    pub batches_per_second: f64,
    pub learning_rate: f64,
    pub nan_count: u64,
    pub gradient_explosion_count: u64,
    pub checkpoint_saves: u64,
}

impl Default for TrainingState {
    fn default() -> Self {
        Self {
            epoch: 0,
            train_loss: 0.0,
            val_loss: None,
            accuracy: None,
            batches_processed: 0,
            batches_per_second: 0.0,
            learning_rate: 0.001,
            nan_count: 0,
            gradient_explosion_count: 0,
            checkpoint_saves: 0,
        }
    }
}
```

**Step 2: Add to training mod.rs**

In `crates/ml/src/training/mod.rs`, add:

```rust
pub mod push_metrics;
```

**Step 3: Verify reqwest is available**

Check if `crates/ml/Cargo.toml` already has `reqwest`. If not, add:

```toml
reqwest = { workspace = true, features = ["rustls-tls"] }
```

**Step 4: Build check**

Run: `SQLX_OFFLINE=true cargo check -p ml`
Expected: Compiles clean

**Step 5: Commit**

```bash
git add crates/ml/src/training/push_metrics.rs crates/ml/src/training/mod.rs crates/ml/Cargo.toml
git commit -m "feat(ml): add Pushgateway metrics client for training jobs"
```

---

### Task 13: Deploy and Verify

**Files:**
- No code changes — cluster operations only

**Step 1: Apply all K8s changes**

```bash
# Deploy pushgateway
kubectl apply -f infra/k8s/monitoring/pushgateway.yaml

# Apply updated web-gateway with metrics port
kubectl apply -f infra/k8s/services/web-gateway.yaml

# Apply updated service YAMLs (all services)
kubectl apply -f infra/k8s/services/

# Deploy updated dashboard ConfigMaps
PROM_UID="PBFA97CFB590B2093"
LOKI_UID="P8E80F9AEF21F6940"
TEMPO_UID="tempo"
DASH_DIR="infra/k8s/monitoring/dashboards"
TMP_DIR=$(mktemp -d)
for f in "$DASH_DIR"/*.json; do
  python3 -c "
import json
with open('$f') as fh:
    d = json.load(fh)
d.pop('__inputs', None)
d.pop('__requires', None)
raw = json.dumps(d)
raw = raw.replace('\${DS_PROMETHEUS}', '${PROM_UID}')
raw = raw.replace('\${DS_LOKI}', '${LOKI_UID}')
raw = raw.replace('\${DS_TEMPO}', '${TEMPO_UID}')
print(raw)
" > "$TMP_DIR/$(basename "$f")"
done

# Build ConfigMaps with folder annotations
declare -A GROUPS=(
  [grafana-dashboards-foxhunt]="foxhunt-logs.json foxhunt-overview.json foxhunt-services.json foxhunt-cockpit.json foxhunt-trading-cockpit.json foxhunt-training-cockpit.json foxhunt-traces.json"
  [grafana-dashboards-infra]="cluster-overview.json gpu-overview.json foxhunt-gpu-training.json foxhunt-infrastructure.json"
  [grafana-dashboards-cicd]="foxhunt-ci-pipelines.json gitlab-services.json"
)
declare -A FOLDERS=(
  [grafana-dashboards-foxhunt]="Foxhunt"
  [grafana-dashboards-infra]="Infrastructure"
  [grafana-dashboards-cicd]="CI/CD"
)
for cm_name in "${!GROUPS[@]}"; do
  args=""
  for df in ${GROUPS[$cm_name]}; do
    [ -f "$TMP_DIR/$df" ] && args="$args --from-file=$df=$TMP_DIR/$df"
  done
  if [ -n "$args" ]; then
    kubectl create configmap "$cm_name" -n foxhunt $args \
      --dry-run=client -o yaml | \
      kubectl label -f - --dry-run=client -o yaml --local grafana_dashboard=1 | \
      kubectl annotate -f - --dry-run=client -o yaml --local grafana_folder="${FOLDERS[$cm_name]}" | \
      kubectl apply -f -
  fi
done
rm -rf "$TMP_DIR"

# Restart Grafana to reload dashboards
kubectl rollout restart deployment grafana -n foxhunt

# Upgrade Prometheus with pushgateway scrape config
helm upgrade gitlab gitlab/gitlab -n foxhunt --reuse-values \
  -f infra/k8s/monitoring/prometheus-values.yaml

# Rolling restart services to pick up new binaries
for svc in trading-service api-gateway broker-gateway data-acquisition-service ml-training-service backtesting-service trading-agent-service web-gateway; do
  kubectl rollout restart deployment "$svc" -n foxhunt 2>/dev/null || true
done
```

**Step 2: Verify Pushgateway is running**

```bash
kubectl get pods -n foxhunt -l app.kubernetes.io/name=pushgateway
```
Expected: `pushgateway-xxx   1/1   Running`

**Step 3: Verify dashboards are in Foxhunt folder**

```bash
kubectl port-forward deployment/grafana 3000:3000 -n foxhunt &
GRAFANA_PASS=$(kubectl get secret grafana -n foxhunt -o jsonpath='{.data.admin-password}' | base64 -d)
curl -s -u "admin:${GRAFANA_PASS}" "http://localhost:3000/api/search?type=dash-db" | python3 -c "
import json, sys
for d in json.load(sys.stdin):
    print(f\"{d['uid']:40s} {d.get('folderTitle','General'):20s} {d['title']}\")
"
kill %1
```

**Step 4: Push and commit**

```bash
git push origin main
```
