gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
ae704a7b7346da17039e4aeef8265f65c6df40ee
foxhunt/docs/plans/2026-03-01-real-metrics-overhaul-implementation.md
jgrusewski d417807132 docs: add real metrics overhaul implementation plan 
13-task plan across 4 layers: fix dashboard PromQL, add metrics
servers to data-acquisition and web-gateway, create gRPC metrics
Tower layer, deploy Pushgateway for training jobs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 02:00:08 +01:00

45 KiB
Raw Blame History

Real Metrics Dashboard Overhaul — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Replace all placeholder PromQL queries in 4 Grafana dashboards with real metric names, add metrics servers to data-acquisition and web-gateway, add a shared gRPC metrics tower layer, and deploy a Pushgateway for training job metrics.

Architecture: Four layers executed sequentially. Layer 1 fixes dashboard JSON queries to match real Prometheus metric names. Layer 2 adds Axum-based metrics HTTP servers to the two services missing them. Layer 3 creates a reusable Tower middleware in the common crate that automatically instruments gRPC handlers with request count and latency histograms, then wires it into all 8 services. Layer 4 deploys a Prometheus Pushgateway and integrates it into the training binaries for epoch-level metrics.

Tech Stack: Rust (prometheus crate, once_cell::sync::Lazy, axum, tower), Grafana dashboard JSON, Kubernetes YAML, Prometheus Pushgateway

Build command: SQLX_OFFLINE=true cargo check --workspace Test command: SQLX_OFFLINE=true cargo test -p <crate> --lib Clippy: #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]

Layer 1: Fix Dashboard Queries

Task 1: Fix foxhunt-trading-cockpit.json PromQL Queries

Files:

  • Modify: infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json

Context: The Trading Cockpit has 17 panels across 5 rows. Every panel uses placeholder metric names. The real metrics are defined in services/trading_service/src/metrics.rs with Lazy<CounterVec/GaugeVec/HistogramVec> statics. The exact Prometheus metric names (lowercase, underscored) are what gets exported.

Step 1: Replace all PromQL expressions

For each panel, use this mapping of current → correct query:

Panel Title Current expr Correct expr
Ensemble Agreement ml_ensemble_agreement_rate ml_ensemble_agreement_rate{symbol=~"$symbol"}
Prediction Confidence ml_prediction_confidence histogram_quantile(0.5, rate(ml_predictions_confidence_bucket[5m]))
Model Accuracy ml_model_accuracy ml_prediction_accuracy{model_id=~"$model"}
Ensemble Sharpe ml_ensemble_sharpe ml_model_sharpe_ratio{model_id="ensemble"}
Active Positions foxhunt_active_positions trading_agent_assets_selected
Realized PnL foxhunt_realized_pnl ml_model_cumulative_pnl{model_id=~"$model"}
Unrealized PnL foxhunt_unrealized_pnl trading_agent_portfolio_value_usd
Max Drawdown foxhunt_max_drawdown ml_model_max_drawdown{model_id=~"$model"}
PnL Over Time foxhunt_pnl_total ml_model_cumulative_pnl{model_id=~"$model"}
Portfolio VaR foxhunt_portfolio_var ml_model_max_drawdown{model_id="ensemble"}
Circuit Breaker foxhunt_circuit_breaker_state ml_circuit_breaker_state{model_id=~"$model"}
Risk Alerts foxhunt_risk_alerts_total rate(ml_alerts_total{severity="critical"}[5m])
Position HHI foxhunt_position_hhi ml_ensemble_agreement_rate{symbol=~"$symbol"}
Inference Latency p95 foxhunt_inference_latency_seconds histogram_quantile(0.95, rate(ml_model_inference_latency_bucket[5m]))
Model Drift foxhunt_model_drift ml_model_drift_score{model_id=~"$model"}
gRPC Latency p95 foxhunt_grpc_latency_seconds histogram_quantile(0.95, rate(grpc_server_handling_seconds_bucket{service=~"$service"}[5m]))
ML Inference Distribution foxhunt_ml_inference_duration_seconds rate(ml_model_inference_latency_bucket[5m])

Also add two template variables to the dashboard templating.list:

{
  "name": "model",
  "type": "query",
  "query": "label_values(ml_prediction_accuracy, model_id)",
  "datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
  "refresh": 2,
  "includeAll": true,
  "multi": true,
  "current": {"text": "All", "value": "$__all"}
},
{
  "name": "symbol",
  "type": "query",
  "query": "label_values(ml_ensemble_agreement_rate, symbol)",
  "datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
  "refresh": 2,
  "includeAll": true,
  "multi": true,
  "current": {"text": "All", "value": "$__all"}
}

Step 2: Validate JSON is valid

Run: python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json'))" Expected: No output (valid JSON)

Step 3: Commit

git add infra/k8s/monitoring/dashboards/foxhunt-trading-cockpit.json
git commit -m "fix(dashboard): use real metric names in trading cockpit"

Task 2: Fix foxhunt-training-cockpit.json PromQL Queries

Files:

  • Modify: infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json

Context: The Training Cockpit has 28 panels across 8 rows. Many panels query metrics that come from kube-state-metrics (job status), DCGM exporter (GPU), ml-training-service (job counters), and Pushgateway (epoch-level, Layer 4). For metrics that require Pushgateway (Layer 4), use the correct metric name now — they'll show "No data" until the Pushgateway is deployed.

Step 1: Replace all PromQL expressions

Panel Title Current expr Correct expr
Running Jobs foxhunt_training_jobs_active kube_job_status_active{namespace="foxhunt", job_name=~"training-.*"}
Active Workers foxhunt_training_active_workers ml_training_active_workers
Jobs Started foxhunt_training_jobs_started_total ml_training_jobs_started_total
Jobs Completed foxhunt_training_jobs_completed_total ml_training_jobs_completed_total
Failed Jobs foxhunt_training_jobs_failed_total ml_training_errors_total
Pending Jobs foxhunt_training_jobs_pending kube_job_status_active{namespace="foxhunt", job_name=~"training-.*"} - kube_job_status_succeeded{namespace="foxhunt", job_name=~"training-.*"}
Loss Curves foxhunt_training_loss foxhunt_training_epoch_loss{model=~"$model_type"}
Epoch Progress foxhunt_training_epoch foxhunt_training_current_epoch{model=~"$model_type"}
Convergence Rate foxhunt_training_convergence_rate deriv(foxhunt_training_epoch_loss{model=~"$model_type"}[10m])
Training Speed foxhunt_training_steps_per_second foxhunt_training_batches_per_second{model=~"$model_type"}
GPU Utilization foxhunt_gpu_utilization DCGM_FI_DEV_GPU_UTIL
GPU Memory foxhunt_gpu_memory_used DCGM_FI_DEV_FB_USED
GPU Temperature foxhunt_gpu_temperature DCGM_FI_DEV_GPU_TEMP
GPU Power foxhunt_gpu_power DCGM_FI_DEV_POWER_USAGE
Accuracy foxhunt_training_accuracy foxhunt_training_eval_accuracy{model=~"$model_type"}
F1 / Precision / Recall foxhunt_training_f1 foxhunt_training_eval_f1{model=~"$model_type"}
Total Saves foxhunt_checkpoint_saves_total foxhunt_training_checkpoint_saves_total{model=~"$model_type"}
Save Failures foxhunt_checkpoint_failures foxhunt_training_checkpoint_failures_total{model=~"$model_type"}
Save Duration p95 foxhunt_checkpoint_duration_seconds histogram_quantile(0.95, rate(foxhunt_training_checkpoint_duration_seconds_bucket[5m]))
Checkpoint Size foxhunt_checkpoint_size_bytes foxhunt_training_checkpoint_size_bytes{model=~"$model_type"}
Batches Processed foxhunt_training_batches_total foxhunt_training_batches_processed{model=~"$model_type"}
Data Loading Latency foxhunt_data_loading_latency histogram_quantile(0.95, rate(foxhunt_training_data_load_seconds_bucket[5m]))
Feature Errors foxhunt_training_feature_errors foxhunt_training_feature_errors_total{model=~"$model_type"}
NaN Detections foxhunt_training_nan_detected foxhunt_training_nan_detected_total{model=~"$model_type"}
Gradient Explosions foxhunt_training_gradient_explosions foxhunt_training_gradient_explosion_total{model=~"$model_type"}
Training Failures foxhunt_training_failures ml_training_errors_total
Job Duration Distribution foxhunt_training_job_duration (kube_job_complete_time{namespace="foxhunt", job_name=~"training-.*"} - kube_job_start_time{namespace="foxhunt", job_name=~"training-.*"}) / 3600
Iteration Duration foxhunt_training_iteration_duration foxhunt_training_iteration_seconds{model=~"$model_type"}

Step 2: Validate JSON

Run: python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json'))"

Step 3: Commit

git add infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json
git commit -m "fix(dashboard): use real metric names in training cockpit"

Task 3: Fix foxhunt-traces.json PromQL Queries

Files:

  • Modify: infra/k8s/monitoring/dashboards/foxhunt-traces.json

Context: The Traces dashboard has 8 panels. Overview stats should query Tempo distributor metrics (which Tempo exposes on port 3200 and Prometheus scrapes). Service Map uses Tempo datasource directly. Trace search uses Tempo TraceQL.

Step 1: Replace all PromQL expressions

Panel Title Current expr Correct expr
Spans/sec foxhunt_spans_total sum(rate(tempo_distributor_spans_received_total[5m]))
Error Rate foxhunt_trace_errors_total sum(rate(tempo_distributor_spans_received_total{status="error"}[5m])) / sum(rate(tempo_distributor_spans_received_total[5m]))
Avg Duration foxhunt_span_duration_seconds sum(rate(tempo_spanmetrics_latency_sum[5m])) / sum(rate(tempo_spanmetrics_latency_count[5m]))
Active Services foxhunt_active_services count(count by (service_name)(tempo_distributor_spans_received_total))
Latency by Service p95 foxhunt_span_duration_seconds{quantile="0.95"} histogram_quantile(0.95, sum by (le, service_name)(rate(tempo_spanmetrics_latency_bucket[5m])))
Error Rate by Service foxhunt_trace_error_rate sum by (service_name)(rate(tempo_distributor_spans_received_total{status="error"}[5m]))

For the Service Map panel (nodeGraph type), ensure it uses the Tempo datasource (${DS_TEMPO}) with query type serviceMap.

For the Recent Traces panel (table type), ensure it uses Tempo datasource with TraceQL query: {status = error} || {duration > 1s}.

Step 2: Validate JSON

Run: python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-traces.json'))"

Step 3: Commit

git add infra/k8s/monitoring/dashboards/foxhunt-traces.json
git commit -m "fix(dashboard): use real Tempo metric names in traces dashboard"

Task 4: Fix foxhunt-cockpit.json Trace Health Row

Files:

  • Modify: infra/k8s/monitoring/dashboards/foxhunt-cockpit.json

Context: The Cockpit dashboard's Trace Health row (panels 35-37, added in the OTEL task) needs the same Tempo metric fixes as Task 3.

Step 1: Fix the 3 Trace Health panels

Panel Title Current expr Correct expr
Spans/sec foxhunt_spans_total (if placeholder) sum(rate(tempo_distributor_spans_received_total[5m]))
Trace Error Rate foxhunt_trace_error_rate (if placeholder) sum(rate(tempo_distributor_spans_received_total{status="error"}[5m])) / sum(rate(tempo_distributor_spans_received_total[5m]))
Spans/sec by Service foxhunt_spans_by_service (if placeholder) sum by (service_name)(rate(tempo_distributor_spans_received_total[5m]))

Step 2: Validate JSON

Run: python3 -c "import json; json.load(open('infra/k8s/monitoring/dashboards/foxhunt-cockpit.json'))"

Step 3: Commit

git add infra/k8s/monitoring/dashboards/foxhunt-cockpit.json
git commit -m "fix(dashboard): use real Tempo metrics in cockpit trace health row"

Layer 2: Add Metrics Servers to Missing Services

Task 5: Add Prometheus Metrics Server to data-acquisition-service

Files:

  • Create: services/data_acquisition_service/src/metrics.rs
  • Modify: services/data_acquisition_service/src/main.rs
  • Modify: services/data_acquisition_service/src/lib.rs (add pub mod metrics;)

Context: data-acquisition-service already has prometheus, once_cell, and axum in Cargo.toml. The K8s YAML already declares port 9097 with prometheus annotations. The only thing missing is the actual metrics server code. Follow the ml-training-service pattern (inline metrics handler in main.rs spawned via tokio::spawn).

Step 1: Create services/data_acquisition_service/src/metrics.rs

//! Prometheus metrics for Data Acquisition Service

use once_cell::sync::Lazy;
use prometheus::{register_counter, register_counter_vec, register_gauge, register_histogram_vec, Counter, CounterVec, Gauge, HistogramVec};

/// Service uptime in seconds
pub static SERVICE_UPTIME: Lazy<Gauge> = Lazy::new(|| {
    register_gauge!(
        "foxhunt_data_acquisition_uptime_seconds",
        "Service uptime in seconds"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register uptime gauge: {e}");
        std::process::abort()
    })
});

/// Active data feeds
pub static FEEDS_ACTIVE: Lazy<Gauge> = Lazy::new(|| {
    register_gauge!(
        "foxhunt_data_acquisition_feeds_active",
        "Number of active data feeds"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register feeds_active gauge: {e}");
        std::process::abort()
    })
});

/// Total records received
pub static RECORDS_RECEIVED: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "foxhunt_data_acquisition_records_received_total",
        "Total records received from data feeds",
        &["symbol", "feed_type"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register records_received counter: {e}");
        std::process::abort()
    })
});

/// Total errors
pub static ERRORS: Lazy<Counter> = Lazy::new(|| {
    register_counter!(
        "foxhunt_data_acquisition_errors_total",
        "Total data acquisition errors"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register errors counter: {e}");
        std::process::abort()
    })
});

/// Feed processing latency
pub static FEED_LATENCY: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "foxhunt_data_acquisition_feed_latency_seconds",
        "Feed processing latency in seconds",
        &["feed_type"],
        vec![0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register feed_latency histogram: {e}");
        std::process::abort()
    })
});

/// Initialize all metrics (force lazy registration)
pub fn init_metrics() {
    let _ = &*SERVICE_UPTIME;
    let _ = &*FEEDS_ACTIVE;
    let _ = &*RECORDS_RECEIVED;
    let _ = &*ERRORS;
    let _ = &*FEED_LATENCY;
}

/// Update service uptime metric
pub fn update_uptime(start_time: std::time::Instant) {
    SERVICE_UPTIME.set(start_time.elapsed().as_secs_f64());
}

Step 2: Add pub mod metrics; to lib.rs

In services/data_acquisition_service/src/lib.rs, add:

pub mod metrics;

Step 3: Add metrics server + uptime updater to main.rs

In services/data_acquisition_service/src/main.rs, after the info!("Data Acquisition Service listening on {}", addr); line (line 52) and before the Server::builder() block (line 55), add:

    // Initialize Prometheus metrics
    data_acquisition_service::metrics::init_metrics();
    let service_start = std::time::Instant::now();

    // Spawn uptime updater
    tokio::spawn(async move {
        let mut interval = tokio::time::interval(std::time::Duration::from_secs(1));
        loop {
            interval.tick().await;
            data_acquisition_service::metrics::update_uptime(service_start);
        }
    });

    // Start Prometheus metrics HTTP endpoint on port 9097
    let metrics_port: u16 = std::env::var("METRICS_PORT")
        .ok()
        .and_then(|s| s.parse().ok())
        .unwrap_or(9097);
    tokio::spawn(async move {
        use axum::{routing::get, Router};
        use prometheus::{Encoder, TextEncoder};

        async fn metrics_handler() -> String {
            let encoder = TextEncoder::new();
            let metric_families = prometheus::gather();
            let mut buffer = vec![];
            let _ = encoder.encode(&metric_families, &mut buffer);
            String::from_utf8(buffer).unwrap_or_else(|_| String::new())
        }

        let app = Router::new().route("/metrics", get(metrics_handler));
        let addr = format!("0.0.0.0:{}", metrics_port);
        info!("Prometheus metrics endpoint listening on http://{}", addr);

        let listener = match tokio::net::TcpListener::bind(&addr).await {
            Ok(l) => l,
            Err(e) => {
                tracing::error!("Failed to bind metrics endpoint {}: {}", addr, e);
                return;
            },
        };
        if let Err(e) = axum::serve(listener, app).await {
            tracing::error!("Metrics server failed: {}", e);
        }
    });

Step 4: Build check

Run: SQLX_OFFLINE=true cargo check -p data_acquisition_service Expected: Compiles clean

Step 5: Commit

git add services/data_acquisition_service/src/metrics.rs services/data_acquisition_service/src/main.rs services/data_acquisition_service/src/lib.rs
git commit -m "feat(data-acquisition): add Prometheus metrics server on port 9097"

Task 6: Add Prometheus Metrics Server to web-gateway

Files:

  • Add dependency: crates/web-gateway/Cargo.toml
  • Create: crates/web-gateway/src/metrics.rs
  • Modify: crates/web-gateway/src/main.rs
  • Modify: crates/web-gateway/src/lib.rs (add pub mod metrics;)

Context: web-gateway does NOT have prometheus in Cargo.toml — must add it. web-gateway is an Axum HTTP server (port 3000). The metrics endpoint should run on a separate port (9098) to avoid mixing application traffic with scraping. Use the same inline pattern as ml-training-service.

Step 1: Add prometheus dependency to Cargo.toml

In crates/web-gateway/Cargo.toml, after the common.workspace = true line (line 44), add:

# Metrics
prometheus.workspace = true
once_cell.workspace = true

Step 2: Create crates/web-gateway/src/metrics.rs

//! Prometheus metrics for Web Gateway

use once_cell::sync::Lazy;
use prometheus::{register_counter_vec, register_gauge, register_histogram_vec, CounterVec, Gauge, HistogramVec};

/// Service uptime in seconds
pub static SERVICE_UPTIME: Lazy<Gauge> = Lazy::new(|| {
    register_gauge!(
        "foxhunt_web_gateway_uptime_seconds",
        "Service uptime in seconds"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register uptime gauge: {e}");
        std::process::abort()
    })
});

/// HTTP requests total
pub static HTTP_REQUESTS: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "foxhunt_web_gateway_http_requests_total",
        "Total HTTP requests",
        &["method", "path", "status"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register http_requests counter: {e}");
        std::process::abort()
    })
});

/// HTTP request duration
pub static HTTP_REQUEST_DURATION: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "foxhunt_web_gateway_http_request_duration_seconds",
        "HTTP request duration in seconds",
        &["method", "path"],
        vec![0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register http_request_duration histogram: {e}");
        std::process::abort()
    })
});

/// Active WebSocket connections
pub static WS_CONNECTIONS: Lazy<Gauge> = Lazy::new(|| {
    register_gauge!(
        "foxhunt_web_gateway_ws_connections_active",
        "Active WebSocket connections"
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register ws_connections gauge: {e}");
        std::process::abort()
    })
});

/// WebSocket messages total
pub static WS_MESSAGES: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "foxhunt_web_gateway_ws_messages_total",
        "Total WebSocket messages",
        &["direction", "topic"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register ws_messages counter: {e}");
        std::process::abort()
    })
});

/// Initialize all metrics (force lazy registration)
pub fn init_metrics() {
    let _ = &*SERVICE_UPTIME;
    let _ = &*HTTP_REQUESTS;
    let _ = &*HTTP_REQUEST_DURATION;
    let _ = &*WS_CONNECTIONS;
    let _ = &*WS_MESSAGES;
}

/// Update service uptime metric
pub fn update_uptime(start_time: std::time::Instant) {
    SERVICE_UPTIME.set(start_time.elapsed().as_secs_f64());
}

Step 3: Add pub mod metrics; to lib.rs

In crates/web-gateway/src/lib.rs, add:

pub mod metrics;

Step 4: Add metrics server to main.rs

In crates/web-gateway/src/main.rs, after the info!("Web gateway listening on {}", listen_addr); line (line 79) and before the axum::serve( block (line 81), add:

    // Initialize Prometheus metrics
    web_gateway::metrics::init_metrics();
    let service_start = std::time::Instant::now();

    // Spawn uptime updater
    tokio::spawn(async move {
        let mut interval = tokio::time::interval(std::time::Duration::from_secs(1));
        loop {
            interval.tick().await;
            web_gateway::metrics::update_uptime(service_start);
        }
    });

    // Start Prometheus metrics HTTP endpoint on port 9098
    let metrics_port: u16 = std::env::var("METRICS_PORT")
        .ok()
        .and_then(|s| s.parse().ok())
        .unwrap_or(9098);
    tokio::spawn(async move {
        use axum::{routing::get, Router};
        use prometheus::{Encoder, TextEncoder};

        async fn metrics_handler() -> String {
            let encoder = TextEncoder::new();
            let metric_families = prometheus::gather();
            let mut buffer = vec![];
            let _ = encoder.encode(&metric_families, &mut buffer);
            String::from_utf8(buffer).unwrap_or_else(|_| String::new())
        }

        let app = Router::new().route("/metrics", get(metrics_handler));
        let addr = format!("0.0.0.0:{}", metrics_port);
        info!("Prometheus metrics endpoint listening on http://{}", addr);

        let listener = match tokio::net::TcpListener::bind(&addr).await {
            Ok(l) => l,
            Err(e) => {
                tracing::error!("Failed to bind metrics endpoint {}: {}", addr, e);
                return;
            },
        };
        if let Err(e) = axum::serve(listener, app).await {
            tracing::error!("Metrics server failed: {}", e);
        }
    });

Step 5: Build check

Run: SQLX_OFFLINE=true cargo check -p web-gateway Expected: Compiles clean

Step 6: Run existing tests

Run: SQLX_OFFLINE=true cargo test -p web-gateway --lib Expected: All 138 tests pass

Step 7: Commit

git add crates/web-gateway/Cargo.toml crates/web-gateway/src/metrics.rs crates/web-gateway/src/main.rs crates/web-gateway/src/lib.rs
git commit -m "feat(web-gateway): add Prometheus metrics server on port 9098"

Task 7: Update K8s YAML for web-gateway Metrics Port

Files:

  • Modify: infra/k8s/services/web-gateway.yaml

Context: web-gateway K8s YAML currently has only port 3000 (http). Need to add port 9098 (metrics), prometheus scrape annotations, and ensure the Service exposes the metrics port.

Step 1: Add prometheus annotations to pod template

In infra/k8s/services/web-gateway.yaml, the pod template metadata currently has only labels: (line 21). Add annotations: before labels::

    metadata:
      annotations:
        gitlab.com/prometheus_scrape: "true"
        gitlab.com/prometheus_port: "9098"
        gitlab.com/prometheus_path: "/metrics"
      labels:
        app.kubernetes.io/name: web-gateway

Step 2: Add metrics container port

After the http port (line 88), add:

            - containerPort: 9098
              name: metrics

Step 3: Add metrics port to Service

Find the Service section at the bottom of the file. Add the metrics port:

    - port: 9098
      targetPort: 9098
      name: metrics

Step 4: Commit

git add infra/k8s/services/web-gateway.yaml
git commit -m "infra(web-gateway): add metrics port 9098 and prometheus scrape annotations"

Layer 3: gRPC Metrics Tower Layer

Task 8: Create gRPC Metrics Tower Layer in common

Files:

  • Create: crates/common/src/metrics/grpc_metrics.rs
  • Modify: crates/common/src/metrics/mod.rs (add pub mod grpc_metrics;)

Context: We need a Tower Layer + Service that wraps every gRPC handler and automatically records grpc_server_started_total, grpc_server_handled_total, and grpc_server_handling_seconds. The layer extracts the gRPC method from the request URI path. common already depends on prometheus, once_cell, tonic, and tower (via tonic).

Step 1: Create crates/common/src/metrics/grpc_metrics.rs

//! gRPC metrics Tower layer for automatic request instrumentation
//!
//! Provides `grpc_server_started_total`, `grpc_server_handled_total`,
//! and `grpc_server_handling_seconds` metrics for all gRPC handlers.

use once_cell::sync::Lazy;
use prometheus::{
    register_counter_vec, register_histogram_vec, CounterVec, HistogramVec,
};
use std::future::Future;
use std::pin::Pin;
use std::task::{Context, Poll};
use tonic::body::BoxBody;
use tower::{Layer, Service};

/// Counter for gRPC requests started
static GRPC_STARTED: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "grpc_server_started_total",
        "Total gRPC RPCs started on the server",
        &["service", "grpc_method"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register grpc_server_started_total: {e}");
        std::process::abort()
    })
});

/// Counter for gRPC requests completed with status code
static GRPC_HANDLED: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "grpc_server_handled_total",
        "Total gRPC RPCs completed on the server with status code",
        &["service", "grpc_method", "grpc_code"]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register grpc_server_handled_total: {e}");
        std::process::abort()
    })
});

/// Histogram for gRPC request handling duration
static GRPC_HANDLING_SECONDS: Lazy<HistogramVec> = Lazy::new(|| {
    register_histogram_vec!(
        "grpc_server_handling_seconds",
        "Histogram of gRPC request handling duration in seconds",
        &["service", "grpc_method"],
        vec![0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0]
    )
    .unwrap_or_else(|e| {
        eprintln!("FATAL: failed to register grpc_server_handling_seconds: {e}");
        std::process::abort()
    })
});

/// Initialize gRPC metrics (force lazy registration)
pub fn init_grpc_metrics() {
    let _ = &*GRPC_STARTED;
    let _ = &*GRPC_HANDLED;
    let _ = &*GRPC_HANDLING_SECONDS;
}

/// Tower Layer that instruments gRPC handlers with Prometheus metrics
#[derive(Clone)]
pub struct GrpcMetricsLayer {
    service_name: &'static str,
}

impl GrpcMetricsLayer {
    /// Create a new gRPC metrics layer for the given service
    pub fn new(service_name: &'static str) -> Self {
        init_grpc_metrics();
        Self { service_name }
    }
}

impl<S> Layer<S> for GrpcMetricsLayer {
    type Service = GrpcMetricsService<S>;

    fn layer(&self, inner: S) -> Self::Service {
        GrpcMetricsService {
            inner,
            service_name: self.service_name,
        }
    }
}

/// Tower Service that records gRPC metrics
#[derive(Clone)]
pub struct GrpcMetricsService<S> {
    inner: S,
    service_name: &'static str,
}

impl<S> Service<http::Request<BoxBody>> for GrpcMetricsService<S>
where
    S: Service<http::Request<BoxBody>, Response = http::Response<BoxBody>>
        + Clone
        + Send
        + 'static,
    S::Future: Send + 'static,
    S::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
{
    type Response = S::Response;
    type Error = S::Error;
    type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;

    fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
        self.inner.poll_ready(cx)
    }

    fn call(&mut self, req: http::Request<BoxBody>) -> Self::Future {
        let service_name = self.service_name;
        let method = req
            .uri()
            .path()
            .rsplit('/')
            .next()
            .unwrap_or("unknown")
            .to_string();

        GRPC_STARTED
            .with_label_values(&[service_name, &method])
            .inc();

        let start = std::time::Instant::now();
        let mut inner = self.inner.clone();

        Box::pin(async move {
            let result = inner.call(req).await;
            let elapsed = start.elapsed().as_secs_f64();

            let code = match &result {
                Ok(resp) => {
                    // Extract gRPC status from the grpc-status header
                    resp.headers()
                        .get("grpc-status")
                        .and_then(|v| v.to_str().ok())
                        .unwrap_or("0")
                        .to_string()
                },
                Err(_) => "13".to_string(), // Internal error
            };

            GRPC_HANDLED
                .with_label_values(&[service_name, &method, &code])
                .inc();

            GRPC_HANDLING_SECONDS
                .with_label_values(&[service_name, &method])
                .observe(elapsed);

            result
        })
    }
}

Step 2: Add to mod.rs

In crates/common/src/metrics/mod.rs, add after line 6 (pub mod registry;):

pub mod grpc_metrics;

And add to the re-exports:

pub use grpc_metrics::{GrpcMetricsLayer, init_grpc_metrics};

Step 3: Build check

Run: SQLX_OFFLINE=true cargo check -p common Expected: Compiles clean

Step 4: Run existing common tests

Run: SQLX_OFFLINE=true cargo test -p common --lib Expected: 198 tests pass

Step 5: Commit

git add crates/common/src/metrics/grpc_metrics.rs crates/common/src/metrics/mod.rs
git commit -m "feat(common): add gRPC metrics Tower layer for automatic instrumentation"

Task 9: Wire gRPC Metrics Layer into Services

Files:

  • Modify: services/trading_service/src/main.rs
  • Modify: services/api_gateway/src/main.rs
  • Modify: services/ml_training_service/src/main.rs
  • Modify: services/backtesting_service/src/main.rs
  • Modify: services/broker_gateway_service/src/main.rs
  • Modify: services/data_acquisition_service/src/main.rs
  • Modify: services/trading_agent_service/src/main.rs
  • Modify: crates/web-gateway/src/main.rs (N/A — web-gateway is HTTP, not gRPC)

Context: Each gRPC service uses tonic::transport::Server::builder() to construct the server. The Tower layer must be added via .layer() BEFORE .add_service(). Only 7 services use gRPC; web-gateway is HTTP-only and skipped.

Step 1: Add .layer(GrpcMetricsLayer::new("service_name")) to each service

For each service, find the Server::builder() chain and add the layer. The pattern is:

use common::metrics::GrpcMetricsLayer;

Server::builder()
    .layer(GrpcMetricsLayer::new("service-name"))
    .add_service(...)
    .serve_with_shutdown(addr, shutdown_signal())
    .await?;

Service names to use (matching K8s service names):

  • services/trading_service/src/main.rs → "trading-service"
  • services/api_gateway/src/main.rs → "api-gateway"
  • services/ml_training_service/src/main.rs → "ml-training-service"
  • services/backtesting_service/src/main.rs → "backtesting-service"
  • services/broker_gateway_service/src/main.rs → "broker-gateway"
  • services/data_acquisition_service/src/main.rs → "data-acquisition-service"
  • services/trading_agent_service/src/main.rs → "trading-agent-service"

Important: For ml_training_service, the server builder is spread across lines 457-489 with optional TLS. The layer should be added right after Server::builder() (line 457):

let mut server_builder = Server::builder()
    .layer(GrpcMetricsLayer::new("ml-training-service"));

Step 2: Build check

Run: SQLX_OFFLINE=true cargo check --workspace Expected: Compiles clean

Step 3: Commit

git add services/*/src/main.rs
git commit -m "feat(services): wire gRPC metrics Tower layer into all 7 gRPC services"

Layer 4: Training Pushgateway

Task 10: Deploy Prometheus Pushgateway

Files:

  • Create: infra/k8s/monitoring/pushgateway.yaml

Context: The Pushgateway receives metrics from short-lived training jobs via HTTP POST and holds them for Prometheus to scrape. Deploy it in the foxhunt namespace.

Step 1: Create infra/k8s/monitoring/pushgateway.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pushgateway
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: pushgateway
    app.kubernetes.io/part-of: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: pushgateway
  template:
    metadata:
      labels:
        app.kubernetes.io/name: pushgateway
    spec:
      nodeSelector:
        k8s.scaleway.com/pool-name: services
      containers:
        - name: pushgateway
          image: prom/pushgateway:v1.11.0
          args:
            - "--persistence.interval=5m"
          ports:
            - containerPort: 9091
              name: http
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              cpu: 200m
              memory: 128Mi
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9091
            initialDelaySeconds: 5
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9091
            initialDelaySeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: pushgateway
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: pushgateway
spec:
  type: ClusterIP
  ports:
    - port: 9091
      targetPort: 9091
      name: http
  selector:
    app.kubernetes.io/name: pushgateway

Step 2: Commit

git add infra/k8s/monitoring/pushgateway.yaml
git commit -m "infra: add Prometheus Pushgateway for training job metrics"

Task 11: Add Pushgateway Scrape Config to Prometheus

Files:

  • Modify: infra/k8s/monitoring/prometheus-values.yaml

Context: Prometheus needs a scrape job for the pushgateway. Use honor_labels: true so that pushed labels (like model and job_id) are preserved instead of being overwritten by Prometheus relabeling.

Step 1: Add pushgateway scrape config

At the end of infra/k8s/monitoring/prometheus-values.yaml (after the promtail job, line 166), add:


      # --- Pushgateway (training job metrics) ---
      - job_name: pushgateway
        honor_labels: true
        static_configs:
          - targets: ['pushgateway.foxhunt.svc.cluster.local:9091']

Step 2: Commit

git add infra/k8s/monitoring/prometheus-values.yaml
git commit -m "infra: add pushgateway scrape config to Prometheus"

Task 12: Add Training Metrics Push to ML Training Binaries

Files:

  • Create: crates/ml/src/training/push_metrics.rs
  • Modify: crates/ml/src/training/mod.rs (add pub mod push_metrics;)
  • Modify: crates/ml/Cargo.toml (add reqwest if not present)

Context: The 5 training binaries (train_baseline_rl, train_baseline_supervised, evaluate_baseline, hyperopt_baseline_rl, hyperopt_baseline_supervised) need to push epoch-level metrics to the Pushgateway during training. The push happens via HTTP POST to http://pushgateway:9091/metrics/job/<job_name>/model/<model_name>.

Prometheus Pushgateway uses a simple text format. We push the metric families as Prometheus exposition text via PUT /metrics/job/{job}/model/{model}.

Step 1: Create crates/ml/src/training/push_metrics.rs

//! Prometheus Pushgateway integration for training job metrics
//!
//! Training jobs are short-lived K8s Jobs. They push epoch-level metrics
//! to the Pushgateway so Prometheus can scrape them persistently.

use std::fmt::Write;

/// Pushgateway client for training metrics
pub struct TrainingMetricsPusher {
    pushgateway_url: String,
    job_id: String,
    model_name: String,
    client: reqwest::Client,
}

impl TrainingMetricsPusher {
    /// Create a new pusher. Falls back to localhost if PUSHGATEWAY_URL not set.
    pub fn new(job_id: &str, model_name: &str) -> Self {
        let pushgateway_url = std::env::var("PUSHGATEWAY_URL")
            .unwrap_or_else(|_| "http://pushgateway.foxhunt.svc.cluster.local:9091".to_string());
        Self {
            pushgateway_url,
            job_id: job_id.to_string(),
            model_name: model_name.to_string(),
            client: reqwest::Client::new(),
        }
    }

    /// Push current training state to Pushgateway.
    /// Non-fatal: logs errors but never panics.
    pub async fn push(&self, state: &TrainingState) {
        let mut body = String::new();

        // Epoch progress
        let _ = writeln!(body, "# HELP foxhunt_training_current_epoch Current training epoch");
        let _ = writeln!(body, "# TYPE foxhunt_training_current_epoch gauge");
        let _ = writeln!(body, "foxhunt_training_current_epoch {}", state.epoch);

        // Loss
        let _ = writeln!(body, "# HELP foxhunt_training_epoch_loss Training loss value");
        let _ = writeln!(body, "# TYPE foxhunt_training_epoch_loss gauge");
        let _ = writeln!(body, "foxhunt_training_epoch_loss{{split=\"train\"}} {}", state.train_loss);
        if let Some(val_loss) = state.val_loss {
            let _ = writeln!(body, "foxhunt_training_epoch_loss{{split=\"val\"}} {}", val_loss);
        }

        // Accuracy
        if let Some(accuracy) = state.accuracy {
            let _ = writeln!(body, "# HELP foxhunt_training_eval_accuracy Model evaluation accuracy");
            let _ = writeln!(body, "# TYPE foxhunt_training_eval_accuracy gauge");
            let _ = writeln!(body, "foxhunt_training_eval_accuracy {}", accuracy);
        }

        // Batches processed
        let _ = writeln!(body, "# HELP foxhunt_training_batches_processed Total batches processed");
        let _ = writeln!(body, "# TYPE foxhunt_training_batches_processed counter");
        let _ = writeln!(body, "foxhunt_training_batches_processed {}", state.batches_processed);

        // Training speed
        if state.batches_per_second > 0.0 {
            let _ = writeln!(body, "# HELP foxhunt_training_batches_per_second Training throughput");
            let _ = writeln!(body, "# TYPE foxhunt_training_batches_per_second gauge");
            let _ = writeln!(body, "foxhunt_training_batches_per_second {}", state.batches_per_second);
        }

        // Learning rate
        let _ = writeln!(body, "# HELP foxhunt_training_learning_rate Current learning rate");
        let _ = writeln!(body, "# TYPE foxhunt_training_learning_rate gauge");
        let _ = writeln!(body, "foxhunt_training_learning_rate {}", state.learning_rate);

        // NaN / gradient events
        if state.nan_count > 0 {
            let _ = writeln!(body, "# HELP foxhunt_training_nan_detected_total NaN gradient events");
            let _ = writeln!(body, "# TYPE foxhunt_training_nan_detected_total counter");
            let _ = writeln!(body, "foxhunt_training_nan_detected_total {}", state.nan_count);
        }
        if state.gradient_explosion_count > 0 {
            let _ = writeln!(body, "# HELP foxhunt_training_gradient_explosion_total Gradient clipping events");
            let _ = writeln!(body, "# TYPE foxhunt_training_gradient_explosion_total counter");
            let _ = writeln!(body, "foxhunt_training_gradient_explosion_total {}", state.gradient_explosion_count);
        }

        // Checkpoint saves
        let _ = writeln!(body, "# HELP foxhunt_training_checkpoint_saves_total Checkpoints saved");
        let _ = writeln!(body, "# TYPE foxhunt_training_checkpoint_saves_total counter");
        let _ = writeln!(body, "foxhunt_training_checkpoint_saves_total {}", state.checkpoint_saves);

        let url = format!(
            "{}/metrics/job/{}/model/{}",
            self.pushgateway_url, self.job_id, self.model_name
        );

        match self.client.put(&url).body(body).send().await {
            Ok(resp) if resp.status().is_success() => {},
            Ok(resp) => {
                eprintln!(
                    "Pushgateway returned {}: {}",
                    resp.status(),
                    resp.text().await.unwrap_or_default()
                );
            },
            Err(e) => {
                eprintln!("Failed to push metrics to Pushgateway: {e}");
            },
        }
    }

    /// Delete metrics for this job from Pushgateway (call on training completion)
    pub async fn cleanup(&self) {
        let url = format!(
            "{}/metrics/job/{}/model/{}",
            self.pushgateway_url, self.job_id, self.model_name
        );
        let _ = self.client.delete(&url).send().await;
    }
}

/// Current training state to push
pub struct TrainingState {
    pub epoch: u64,
    pub train_loss: f64,
    pub val_loss: Option<f64>,
    pub accuracy: Option<f64>,
    pub batches_processed: u64,
    pub batches_per_second: f64,
    pub learning_rate: f64,
    pub nan_count: u64,
    pub gradient_explosion_count: u64,
    pub checkpoint_saves: u64,
}

impl Default for TrainingState {
    fn default() -> Self {
        Self {
            epoch: 0,
            train_loss: 0.0,
            val_loss: None,
            accuracy: None,
            batches_processed: 0,
            batches_per_second: 0.0,
            learning_rate: 0.001,
            nan_count: 0,
            gradient_explosion_count: 0,
            checkpoint_saves: 0,
        }
    }
}

Step 2: Add to training mod.rs

In crates/ml/src/training/mod.rs, add:

pub mod push_metrics;

Step 3: Verify reqwest is available

Check if crates/ml/Cargo.toml already has reqwest. If not, add:

reqwest = { workspace = true, features = ["rustls-tls"] }

Step 4: Build check

Run: SQLX_OFFLINE=true cargo check -p ml Expected: Compiles clean

Step 5: Commit

git add crates/ml/src/training/push_metrics.rs crates/ml/src/training/mod.rs crates/ml/Cargo.toml
git commit -m "feat(ml): add Pushgateway metrics client for training jobs"

Task 13: Deploy and Verify

Files:

  • No code changes — cluster operations only

Step 1: Apply all K8s changes

# Deploy pushgateway
kubectl apply -f infra/k8s/monitoring/pushgateway.yaml

# Apply updated web-gateway with metrics port
kubectl apply -f infra/k8s/services/web-gateway.yaml

# Apply updated service YAMLs (all services)
kubectl apply -f infra/k8s/services/

# Deploy updated dashboard ConfigMaps
PROM_UID="PBFA97CFB590B2093"
LOKI_UID="P8E80F9AEF21F6940"
TEMPO_UID="tempo"
DASH_DIR="infra/k8s/monitoring/dashboards"
TMP_DIR=$(mktemp -d)
for f in "$DASH_DIR"/*.json; do
  python3 -c "
import json
with open('$f') as fh:
    d = json.load(fh)
d.pop('__inputs', None)
d.pop('__requires', None)
raw = json.dumps(d)
raw = raw.replace('\${DS_PROMETHEUS}', '${PROM_UID}')
raw = raw.replace('\${DS_LOKI}', '${LOKI_UID}')
raw = raw.replace('\${DS_TEMPO}', '${TEMPO_UID}')
print(raw)
" > "$TMP_DIR/$(basename "$f")"
done

# Build ConfigMaps with folder annotations
declare -A GROUPS=(
  [grafana-dashboards-foxhunt]="foxhunt-logs.json foxhunt-overview.json foxhunt-services.json foxhunt-cockpit.json foxhunt-trading-cockpit.json foxhunt-training-cockpit.json foxhunt-traces.json"
  [grafana-dashboards-infra]="cluster-overview.json gpu-overview.json foxhunt-gpu-training.json foxhunt-infrastructure.json"
  [grafana-dashboards-cicd]="foxhunt-ci-pipelines.json gitlab-services.json"
)
declare -A FOLDERS=(
  [grafana-dashboards-foxhunt]="Foxhunt"
  [grafana-dashboards-infra]="Infrastructure"
  [grafana-dashboards-cicd]="CI/CD"
)
for cm_name in "${!GROUPS[@]}"; do
  args=""
  for df in ${GROUPS[$cm_name]}; do
    [ -f "$TMP_DIR/$df" ] && args="$args --from-file=$df=$TMP_DIR/$df"
  done
  if [ -n "$args" ]; then
    kubectl create configmap "$cm_name" -n foxhunt $args \
      --dry-run=client -o yaml | \
      kubectl label -f - --dry-run=client -o yaml --local grafana_dashboard=1 | \
      kubectl annotate -f - --dry-run=client -o yaml --local grafana_folder="${FOLDERS[$cm_name]}" | \
      kubectl apply -f -
  fi
done
rm -rf "$TMP_DIR"

# Restart Grafana to reload dashboards
kubectl rollout restart deployment grafana -n foxhunt

# Upgrade Prometheus with pushgateway scrape config
helm upgrade gitlab gitlab/gitlab -n foxhunt --reuse-values \
  -f infra/k8s/monitoring/prometheus-values.yaml

# Rolling restart services to pick up new binaries
for svc in trading-service api-gateway broker-gateway data-acquisition-service ml-training-service backtesting-service trading-agent-service web-gateway; do
  kubectl rollout restart deployment "$svc" -n foxhunt 2>/dev/null || true
done

Step 2: Verify Pushgateway is running

kubectl get pods -n foxhunt -l app.kubernetes.io/name=pushgateway

Expected: pushgateway-xxx 1/1 Running

Step 3: Verify dashboards are in Foxhunt folder

kubectl port-forward deployment/grafana 3000:3000 -n foxhunt &
GRAFANA_PASS=$(kubectl get secret grafana -n foxhunt -o jsonpath='{.data.admin-password}' | base64 -d)
curl -s -u "admin:${GRAFANA_PASS}" "http://localhost:3000/api/search?type=dash-db" | python3 -c "
import json, sys
for d in json.load(sys.stdin):
    print(f\"{d['uid']:40s} {d.get('folderTitle','General'):20s} {d['title']}\")
"
kill %1

Step 4: Push and commit

git push origin main
Reference in New Issue View Git Blame Copy Permalink