foxhunt/docs/plans/2026-03-01-real-metrics-overhaul-design.md

Real Metrics Dashboard Overhaul — Design

Problem

The 4 dashboards deployed on 2026-03-01 (Trading Cockpit, Training Cockpit, Traces, updated Cockpit) use placeholder foxhunt_* metric names in their PromQL queries. The services emit metrics under service-specific prefixes (trading_*, api_gateway_*, ML_*). Two services have no metrics server at all. No gRPC middleware exists for automatic request/response instrumentation. Training jobs can't push epoch-level metrics to Prometheus.

Current State

Services with Metrics Servers

Service	Port	Custom Metrics	Notes
api-gateway	9091	48	Auth pipeline, JWT, RBAC, config reload, routing
trading-service	9092	10	Latency gauges, ring buffer, uptime
ml-training-service	9094	5	Jobs started/completed, errors, workers, uptime
backtesting-service	9093	4	Backtests started/completed, errors, uptime
trading-agent-service	9095	3	Portfolio value, assets selected, universe size
broker-gateway	9096	0 custom	process_* only, metrics.rs has definitions but unused

Services WITHOUT Metrics Servers

Service	Issue
data-acquisition-service	No metrics server on 9097 (port declared in K8s YAML but nothing listens)
web-gateway	No metrics port at all

ML Metrics (Defined but Dormant)

trading-service/src/metrics.rs and ml_metrics.rs define ~35 ML metrics (predictions, accuracy, ensemble, PnL, drawdown, circuit breaker, drift) using Lazy<CounterVec/GaugeVec/HistogramVec>. These are registered with the global Prometheus registry but only populated when ML predictions flow through the trading-service. They will automatically appear on /metrics once traffic runs.

Training Metrics Gap

Training runs are K8s batch Jobs (job-template.yaml). They terminate after training completes. Prometheus can scrape them while running but misses data after completion. No Pushgateway exists to persist epoch-level metrics (loss, accuracy, learning rate).

Design — Four Layers

Layer 1: Fix Dashboard Queries

Update all 4 dashboards to use real metric names. Complete mapping:

Trading Cockpit (foxhunt-trading-cockpit.json)

Panel	Current Query	Real Query
Ensemble Agreement	foxhunt_ensemble_agreement_rate	ML_ENSEMBLE_AGREEMENT_RATE{symbol=~"$symbol"}
Prediction Confidence	foxhunt_prediction_confidence	histogram_quantile(0.5, ML_PREDICTIONS_CONFIDENCE_bucket)
Model Accuracy	foxhunt_model_accuracy	ML_MODEL_ACCURACY{model_id=~"$model"}
Ensemble Sharpe	foxhunt_ensemble_sharpe	ML_MODEL_SHARPE_RATIO{model_id="ensemble"}
Active Positions	foxhunt_active_positions	trading_agent_assets_selected
Realized PnL	foxhunt_realized_pnl	ML_MODEL_CUMULATIVE_PNL{model_id=~"$model"}
Unrealized PnL	foxhunt_unrealized_pnl	trading_agent_portfolio_value_usd
Max Drawdown	foxhunt_max_drawdown	ML_MODEL_MAX_DRAWDOWN{model_id=~"$model"}
PnL Over Time	foxhunt_pnl_total	ML_MODEL_CUMULATIVE_PNL{model_id=~"$model"}
Portfolio VaR	foxhunt_portfolio_var	foxhunt_risk_var_portfolio (needs Layer 2)
Circuit Breaker	foxhunt_circuit_breaker_state	ML_CIRCUIT_BREAKER_TRANSITIONS{model_id=~"$model"}
Risk Alerts	foxhunt_risk_alerts_total	ML_ALERTS_TOTAL{severity="critical"}
Position HHI	foxhunt_position_hhi	foxhunt_risk_position_hhi (needs Layer 2)
Inference Latency p95	foxhunt_inference_latency_seconds	histogram_quantile(0.95, ML_MODEL_INFERENCE_LATENCY_bucket)
Model Drift	foxhunt_model_drift	ML_MODEL_DRIFT_SCORE{model_id=~"$model"}
gRPC Latency p95	foxhunt_grpc_latency_seconds	histogram_quantile(0.95, grpc_server_handling_seconds_bucket{service=~"$service"}) (needs Layer 3)
ML Inference Distribution	foxhunt_ml_inference_duration_seconds	ML_MODEL_INFERENCE_LATENCY_bucket

Training Cockpit (foxhunt-training-cockpit.json)

Panel	Current Query	Real Query
Running Jobs	foxhunt_training_jobs_active	kube_job_status_active{namespace="foxhunt",job_name=~"training-.*"}
Active Workers	foxhunt_training_active_workers	ml_training_active_workers
Jobs Started/Completed/Failed	foxhunt_training_jobs_*	ml_training_jobs_started_total / ml_training_jobs_completed_total / ml_training_errors_total
Pending Jobs	foxhunt_training_jobs_pending	kube_job_status_active{namespace="foxhunt"} - kube_job_status_succeeded{namespace="foxhunt"}
Loss Curves	foxhunt_training_loss	foxhunt_training_epoch_loss{model=~"$model_type"} (needs Layer 4 Pushgateway)
Epoch Progress	foxhunt_training_epoch	foxhunt_training_current_epoch{model=~"$model_type"} (needs Layer 4)
Convergence Rate	foxhunt_training_convergence_rate	deriv(foxhunt_training_epoch_loss[5m]) (needs Layer 4)
Training Speed	foxhunt_training_steps_per_second	foxhunt_training_batches_per_second{model=~"$model_type"} (needs Layer 4)
GPU Utilization/Memory/Temp/Power	foxhunt_gpu_*	DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE
Accuracy / F1	foxhunt_training_accuracy	foxhunt_training_eval_accuracy{model=~"$model_type"} (needs Layer 4)
Checkpoint Saves/Failures/Duration/Size	foxhunt_checkpoint_*	foxhunt_training_checkpoint_* (needs Layer 4)
Batches Processed	foxhunt_training_batches_total	foxhunt_training_batches_processed{model=~"$model_type"} (needs Layer 4)
Data Loading Latency	foxhunt_data_loading_latency	foxhunt_training_data_load_seconds{model=~"$model_type"} (needs Layer 4)
Feature/NaN/Gradient Errors	foxhunt_training_*_errors	foxhunt_training_nan_detected / foxhunt_training_gradient_explosion (needs Layer 4)
Job Duration	foxhunt_training_job_duration	kube_job_complete_time - kube_job_start_time

Traces Dashboard (foxhunt-traces.json)

Panel	Current Query	Real Query
Spans/sec	foxhunt_spans_total	rate(tempo_distributor_spans_received_total[5m])
Error Rate	foxhunt_trace_errors_total	rate(tempo_distributor_spans_received_total{status="error"}[5m]) or Tempo-derived
Avg Duration	foxhunt_span_duration_seconds	Tempo TraceQL: `{}
Active Services	foxhunt_active_services	count(count by (service_name)(tempo_distributor_spans_received_total))
Service Map	nodeGraph from Tempo	Tempo service graph (already correct)
Latency by Service p95	foxhunt_span_duration_seconds	histogram_quantile(0.95, tempo_span_metrics_duration_bucket{service=~"$service"})

Updated Cockpit (foxhunt-cockpit.json — Trace Health row)

Panels already use Tempo metrics (tempo_distributor_spans_received_total) — mostly correct, minor fixes needed.

Layer 2: Add Metrics Servers to Missing Services

data-acquisition-service (port 9097)

Add metrics.rs + Axum HTTP server following the trading-service pattern:

foxhunt_data_acquisition_feeds_active              gauge     Active data feeds
foxhunt_data_acquisition_records_received_total     counter   Records received {symbol, feed_type}
foxhunt_data_acquisition_errors_total               counter   Errors {error_type}
foxhunt_data_acquisition_feed_latency_seconds       histogram Feed processing latency
foxhunt_data_acquisition_last_record_timestamp      gauge     Last record timestamp {symbol}
foxhunt_data_acquisition_uptime_seconds             counter   Service uptime

web-gateway (port 9098)

Add metrics.rs + handler in Axum router (web-gateway already uses Axum):

foxhunt_web_gateway_http_requests_total             counter   HTTP requests {method, path, status}
foxhunt_web_gateway_http_request_duration_seconds    histogram Request latency {method, path}
foxhunt_web_gateway_ws_connections_active            gauge     Active WebSocket connections
foxhunt_web_gateway_ws_messages_total                counter   WS messages {direction, topic}
foxhunt_web_gateway_uptime_seconds                   counter   Service uptime

K8s YAML Updates

• Add metrics named port (9098) to web-gateway deployment
• Fix data-acquisition to actually bind metrics on 9097
• Ensure app.kubernetes.io/part-of: foxhunt label on all services

Layer 3: gRPC Metrics Tower Layer

Add crates/common/src/metrics/grpc_metrics.rs with a tonic Tower layer:

grpc_server_handled_total{service, grpc_method, grpc_code}     counter
grpc_server_handling_seconds{service, grpc_method}              histogram  [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0]
grpc_server_started_total{service, grpc_method}                 counter

This uses the tower::Layer + tower::Service pattern to wrap every gRPC handler automatically. Each service adds one line in main.rs:

Server::builder()
    .layer(common::metrics::grpc_metrics_layer("trading-service"))
    .add_service(...)

Layer 4: Training Pushgateway

Deploy Prometheus Pushgateway

Add infra/k8s/monitoring/pushgateway.yaml:

• Image: prom/pushgateway:v1.11.0
• Port: 9091 (internal, behind ClusterIP service)
• Prometheus scrape config: job pushgateway, honor_labels true
• Retention: ephemeral (no persistence needed, Prometheus scrapes frequently)

Training Binary Integration

Add push_metrics() to the training binaries (train_baseline_rl, train_baseline_supervised):

foxhunt_training_current_epoch{model, job_id}          gauge     Current epoch number
foxhunt_training_epoch_loss{model, job_id, split}      gauge     Loss value (train/val)
foxhunt_training_eval_accuracy{model, job_id}           gauge     Eval accuracy
foxhunt_training_batches_processed{model, job_id}       counter   Batches processed
foxhunt_training_batches_per_second{model, job_id}      gauge     Training throughput
foxhunt_training_data_load_seconds{model, job_id}       histogram Data loading latency
foxhunt_training_checkpoint_saves_total{model, job_id}  counter   Checkpoints saved
foxhunt_training_checkpoint_duration_seconds{model}     histogram Checkpoint save time
foxhunt_training_nan_detected{model, job_id}            counter   NaN gradient events
foxhunt_training_gradient_explosion{model, job_id}      counter   Gradient clipping events
foxhunt_training_learning_rate{model, job_id}           gauge     Current learning rate

Push frequency: every epoch end + every 60 seconds during training. Library: prometheus crate push_metrics() to http://pushgateway:9091.

Pushgateway Lifecycle

• Training job starts → pushes initial metrics with job_id label
• During training → pushes every epoch + every 60s
• Training job ends → Pushgateway retains last values until Prometheus scrapes
• Prometheus scrapes Pushgateway every 15s → data persisted in Prometheus TSDB
• Old job_id groups cleaned up by Pushgateway TTL or manual cleanup

What This Does NOT Cover

• Real-time P&L / position data: Requires the trading engine to be connected to a live broker with real market data. The metrics infrastructure will be ready, but values will be 0 until live trading starts.
• QuestDB integration for dashboards: QuestDB stores ML prediction history for the feedback loop. Grafana can query QuestDB via its PostgreSQL datasource, but that's a separate datasource setup, not Prometheus.
• Alerting rules: We add the metrics and dashboards. Alertmanager rules for SLA/threshold alerts are a follow-up.

Metric Naming Convention

All new metrics follow: foxhunt_<service>_<metric>_<unit>

• foxhunt_ prefix for all application metrics
• Service names: data_acquisition, web_gateway, training
• Units: _seconds, _total, _bytes
• Counter suffix: always _total
• No foxhunt_ prefix on gRPC layer metrics (standard grpc_server_* names for compatibility with existing Grafana community dashboards)

File Changes Summary

Layer	Files Created	Files Modified
L1: Fix Queries	0	4 dashboard JSONs
L2: Missing Metrics	2 (metrics.rs × 2)	2 main.rs + 2 K8s YAMLs
L3: gRPC Layer	1 (grpc_metrics.rs)	8 service main.rs + common/metrics/mod.rs
L4: Pushgateway	1 (pushgateway.yaml)	ml training binaries, prometheus config, .gitlab-ci.yml