# Real Metrics Dashboard Overhaul — Design

## Problem

The 4 dashboards deployed on 2026-03-01 (Trading Cockpit, Training Cockpit, Traces, updated Cockpit) use placeholder `foxhunt_*` metric names in their PromQL queries. The services emit metrics under service-specific prefixes (`trading_*`, `api_gateway_*`, `ML_*`). Two services have no metrics server at all. No gRPC middleware exists for automatic request/response instrumentation. Training jobs can't push epoch-level metrics to Prometheus.

## Current State

### Services with Metrics Servers

| Service | Port | Custom Metrics | Notes |
|---|---|---|---|
| api-gateway | 9091 | 48 | Auth pipeline, JWT, RBAC, config reload, routing |
| trading-service | 9092 | 10 | Latency gauges, ring buffer, uptime |
| ml-training-service | 9094 | 5 | Jobs started/completed, errors, workers, uptime |
| backtesting-service | 9093 | 4 | Backtests started/completed, errors, uptime |
| trading-agent-service | 9095 | 3 | Portfolio value, assets selected, universe size |
| broker-gateway | 9096 | 0 custom | process_* only, metrics.rs has definitions but unused |

### Services WITHOUT Metrics Servers

| Service | Issue |
|---|---|
| data-acquisition-service | No metrics server on 9097 (port declared in K8s YAML but nothing listens) |
| web-gateway | No metrics port at all |

### ML Metrics (Defined but Dormant)

`trading-service/src/metrics.rs` and `ml_metrics.rs` define ~35 ML metrics (predictions, accuracy, ensemble, PnL, drawdown, circuit breaker, drift) using `Lazy<CounterVec/GaugeVec/HistogramVec>`. These are registered with the global Prometheus registry but only populated when ML predictions flow through the trading-service. They will automatically appear on `/metrics` once traffic runs.

### Training Metrics Gap

Training runs are K8s batch Jobs (`job-template.yaml`). They terminate after training completes. Prometheus can scrape them while running but misses data after completion. No Pushgateway exists to persist epoch-level metrics (loss, accuracy, learning rate).

## Design — Four Layers

### Layer 1: Fix Dashboard Queries

Update all 4 dashboards to use real metric names. Complete mapping:

#### Trading Cockpit (`foxhunt-trading-cockpit.json`)

| Panel | Current Query | Real Query |
|---|---|---|
| Ensemble Agreement | `foxhunt_ensemble_agreement_rate` | `ML_ENSEMBLE_AGREEMENT_RATE{symbol=~"$symbol"}` |
| Prediction Confidence | `foxhunt_prediction_confidence` | `histogram_quantile(0.5, ML_PREDICTIONS_CONFIDENCE_bucket)` |
| Model Accuracy | `foxhunt_model_accuracy` | `ML_MODEL_ACCURACY{model_id=~"$model"}` |
| Ensemble Sharpe | `foxhunt_ensemble_sharpe` | `ML_MODEL_SHARPE_RATIO{model_id="ensemble"}` |
| Active Positions | `foxhunt_active_positions` | `trading_agent_assets_selected` |
| Realized PnL | `foxhunt_realized_pnl` | `ML_MODEL_CUMULATIVE_PNL{model_id=~"$model"}` |
| Unrealized PnL | `foxhunt_unrealized_pnl` | `trading_agent_portfolio_value_usd` |
| Max Drawdown | `foxhunt_max_drawdown` | `ML_MODEL_MAX_DRAWDOWN{model_id=~"$model"}` |
| PnL Over Time | `foxhunt_pnl_total` | `ML_MODEL_CUMULATIVE_PNL{model_id=~"$model"}` |
| Portfolio VaR | `foxhunt_portfolio_var` | `foxhunt_risk_var_portfolio` (needs Layer 2) |
| Circuit Breaker | `foxhunt_circuit_breaker_state` | `ML_CIRCUIT_BREAKER_TRANSITIONS{model_id=~"$model"}` |
| Risk Alerts | `foxhunt_risk_alerts_total` | `ML_ALERTS_TOTAL{severity="critical"}` |
| Position HHI | `foxhunt_position_hhi` | `foxhunt_risk_position_hhi` (needs Layer 2) |
| Inference Latency p95 | `foxhunt_inference_latency_seconds` | `histogram_quantile(0.95, ML_MODEL_INFERENCE_LATENCY_bucket)` |
| Model Drift | `foxhunt_model_drift` | `ML_MODEL_DRIFT_SCORE{model_id=~"$model"}` |
| gRPC Latency p95 | `foxhunt_grpc_latency_seconds` | `histogram_quantile(0.95, grpc_server_handling_seconds_bucket{service=~"$service"})` (needs Layer 3) |
| ML Inference Distribution | `foxhunt_ml_inference_duration_seconds` | `ML_MODEL_INFERENCE_LATENCY_bucket` |

#### Training Cockpit (`foxhunt-training-cockpit.json`)

| Panel | Current Query | Real Query |
|---|---|---|
| Running Jobs | `foxhunt_training_jobs_active` | `kube_job_status_active{namespace="foxhunt",job_name=~"training-.*"}` |
| Active Workers | `foxhunt_training_active_workers` | `ml_training_active_workers` |
| Jobs Started/Completed/Failed | `foxhunt_training_jobs_*` | `ml_training_jobs_started_total` / `ml_training_jobs_completed_total` / `ml_training_errors_total` |
| Pending Jobs | `foxhunt_training_jobs_pending` | `kube_job_status_active{namespace="foxhunt"} - kube_job_status_succeeded{namespace="foxhunt"}` |
| Loss Curves | `foxhunt_training_loss` | `foxhunt_training_epoch_loss{model=~"$model_type"}` (needs Layer 4 Pushgateway) |
| Epoch Progress | `foxhunt_training_epoch` | `foxhunt_training_current_epoch{model=~"$model_type"}` (needs Layer 4) |
| Convergence Rate | `foxhunt_training_convergence_rate` | `deriv(foxhunt_training_epoch_loss[5m])` (needs Layer 4) |
| Training Speed | `foxhunt_training_steps_per_second` | `foxhunt_training_batches_per_second{model=~"$model_type"}` (needs Layer 4) |
| GPU Utilization/Memory/Temp/Power | `foxhunt_gpu_*` | `DCGM_FI_DEV_GPU_UTIL`, `DCGM_FI_DEV_FB_USED`, `DCGM_FI_DEV_GPU_TEMP`, `DCGM_FI_DEV_POWER_USAGE` |
| Accuracy / F1 | `foxhunt_training_accuracy` | `foxhunt_training_eval_accuracy{model=~"$model_type"}` (needs Layer 4) |
| Checkpoint Saves/Failures/Duration/Size | `foxhunt_checkpoint_*` | `foxhunt_training_checkpoint_*` (needs Layer 4) |
| Batches Processed | `foxhunt_training_batches_total` | `foxhunt_training_batches_processed{model=~"$model_type"}` (needs Layer 4) |
| Data Loading Latency | `foxhunt_data_loading_latency` | `foxhunt_training_data_load_seconds{model=~"$model_type"}` (needs Layer 4) |
| Feature/NaN/Gradient Errors | `foxhunt_training_*_errors` | `foxhunt_training_nan_detected` / `foxhunt_training_gradient_explosion` (needs Layer 4) |
| Job Duration | `foxhunt_training_job_duration` | `kube_job_complete_time - kube_job_start_time` |

#### Traces Dashboard (`foxhunt-traces.json`)

| Panel | Current Query | Real Query |
|---|---|---|
| Spans/sec | `foxhunt_spans_total` | `rate(tempo_distributor_spans_received_total[5m])` |
| Error Rate | `foxhunt_trace_errors_total` | `rate(tempo_distributor_spans_received_total{status="error"}[5m])` or Tempo-derived |
| Avg Duration | `foxhunt_span_duration_seconds` | Tempo TraceQL: `{} | avg(duration)` |
| Active Services | `foxhunt_active_services` | `count(count by (service_name)(tempo_distributor_spans_received_total))` |
| Service Map | nodeGraph from Tempo | Tempo service graph (already correct) |
| Latency by Service p95 | `foxhunt_span_duration_seconds` | `histogram_quantile(0.95, tempo_span_metrics_duration_bucket{service=~"$service"})` |

#### Updated Cockpit (`foxhunt-cockpit.json` — Trace Health row)

Panels already use Tempo metrics (`tempo_distributor_spans_received_total`) — mostly correct, minor fixes needed.

### Layer 2: Add Metrics Servers to Missing Services

#### data-acquisition-service (port 9097)

Add `metrics.rs` + Axum HTTP server following the trading-service pattern:

```
foxhunt_data_acquisition_feeds_active              gauge     Active data feeds
foxhunt_data_acquisition_records_received_total     counter   Records received {symbol, feed_type}
foxhunt_data_acquisition_errors_total               counter   Errors {error_type}
foxhunt_data_acquisition_feed_latency_seconds       histogram Feed processing latency
foxhunt_data_acquisition_last_record_timestamp      gauge     Last record timestamp {symbol}
foxhunt_data_acquisition_uptime_seconds             counter   Service uptime
```

#### web-gateway (port 9098)

Add `metrics.rs` + handler in Axum router (web-gateway already uses Axum):

```
foxhunt_web_gateway_http_requests_total             counter   HTTP requests {method, path, status}
foxhunt_web_gateway_http_request_duration_seconds    histogram Request latency {method, path}
foxhunt_web_gateway_ws_connections_active            gauge     Active WebSocket connections
foxhunt_web_gateway_ws_messages_total                counter   WS messages {direction, topic}
foxhunt_web_gateway_uptime_seconds                   counter   Service uptime
```

#### K8s YAML Updates

- Add `metrics` named port (9098) to web-gateway deployment
- Fix data-acquisition to actually bind metrics on 9097
- Ensure `app.kubernetes.io/part-of: foxhunt` label on all services

### Layer 3: gRPC Metrics Tower Layer

Add `crates/common/src/metrics/grpc_metrics.rs` with a tonic Tower layer:

```
grpc_server_handled_total{service, grpc_method, grpc_code}     counter
grpc_server_handling_seconds{service, grpc_method}              histogram  [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0]
grpc_server_started_total{service, grpc_method}                 counter
```

This uses the `tower::Layer` + `tower::Service` pattern to wrap every gRPC handler automatically. Each service adds one line in main.rs:

```rust
Server::builder()
    .layer(common::metrics::grpc_metrics_layer("trading-service"))
    .add_service(...)
```

### Layer 4: Training Pushgateway

#### Deploy Prometheus Pushgateway

Add `infra/k8s/monitoring/pushgateway.yaml`:
- Image: `prom/pushgateway:v1.11.0`
- Port: 9091 (internal, behind ClusterIP service)
- Prometheus scrape config: job `pushgateway`, honor_labels true
- Retention: ephemeral (no persistence needed, Prometheus scrapes frequently)

#### Training Binary Integration

Add `push_metrics()` to the training binaries (`train_baseline_rl`, `train_baseline_supervised`):

```
foxhunt_training_current_epoch{model, job_id}          gauge     Current epoch number
foxhunt_training_epoch_loss{model, job_id, split}      gauge     Loss value (train/val)
foxhunt_training_eval_accuracy{model, job_id}           gauge     Eval accuracy
foxhunt_training_batches_processed{model, job_id}       counter   Batches processed
foxhunt_training_batches_per_second{model, job_id}      gauge     Training throughput
foxhunt_training_data_load_seconds{model, job_id}       histogram Data loading latency
foxhunt_training_checkpoint_saves_total{model, job_id}  counter   Checkpoints saved
foxhunt_training_checkpoint_duration_seconds{model}     histogram Checkpoint save time
foxhunt_training_nan_detected{model, job_id}            counter   NaN gradient events
foxhunt_training_gradient_explosion{model, job_id}      counter   Gradient clipping events
foxhunt_training_learning_rate{model, job_id}           gauge     Current learning rate
```

Push frequency: every epoch end + every 60 seconds during training.
Library: `prometheus` crate `push_metrics()` to `http://pushgateway:9091`.

#### Pushgateway Lifecycle

- Training job starts → pushes initial metrics with `job_id` label
- During training → pushes every epoch + every 60s
- Training job ends → Pushgateway retains last values until Prometheus scrapes
- Prometheus scrapes Pushgateway every 15s → data persisted in Prometheus TSDB
- Old `job_id` groups cleaned up by Pushgateway TTL or manual cleanup

## What This Does NOT Cover

- **Real-time P&L / position data**: Requires the trading engine to be connected to a live broker with real market data. The metrics infrastructure will be ready, but values will be 0 until live trading starts.
- **QuestDB integration for dashboards**: QuestDB stores ML prediction history for the feedback loop. Grafana can query QuestDB via its PostgreSQL datasource, but that's a separate datasource setup, not Prometheus.
- **Alerting rules**: We add the metrics and dashboards. Alertmanager rules for SLA/threshold alerts are a follow-up.

## Metric Naming Convention

All new metrics follow: `foxhunt_<service>_<metric>_<unit>`

- `foxhunt_` prefix for all application metrics
- Service names: `data_acquisition`, `web_gateway`, `training`
- Units: `_seconds`, `_total`, `_bytes`
- Counter suffix: always `_total`
- No `foxhunt_` prefix on gRPC layer metrics (standard `grpc_server_*` names for compatibility with existing Grafana community dashboards)

## File Changes Summary

| Layer | Files Created | Files Modified |
|---|---|---|
| L1: Fix Queries | 0 | 4 dashboard JSONs |
| L2: Missing Metrics | 2 (metrics.rs × 2) | 2 main.rs + 2 K8s YAMLs |
| L3: gRPC Layer | 1 (grpc_metrics.rs) | 8 service main.rs + common/metrics/mod.rs |
| L4: Pushgateway | 1 (pushgateway.yaml) | ml training binaries, prometheus config, .gitlab-ci.yml |
