docs: add real metrics overhaul design
Four-layer plan: fix dashboard queries to use real metric names, add metrics servers to data-acquisition and web-gateway, add shared gRPC metrics tower layer, deploy Pushgateway for training job metrics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
211
docs/plans/2026-03-01-real-metrics-overhaul-design.md
Normal file
211
docs/plans/2026-03-01-real-metrics-overhaul-design.md
Normal file
@@ -0,0 +1,211 @@
|
||||
# Real Metrics Dashboard Overhaul — Design
|
||||
|
||||
## Problem
|
||||
|
||||
The 4 dashboards deployed on 2026-03-01 (Trading Cockpit, Training Cockpit, Traces, updated Cockpit) use placeholder `foxhunt_*` metric names in their PromQL queries. The services emit metrics under service-specific prefixes (`trading_*`, `api_gateway_*`, `ML_*`). Two services have no metrics server at all. No gRPC middleware exists for automatic request/response instrumentation. Training jobs can't push epoch-level metrics to Prometheus.
|
||||
|
||||
## Current State
|
||||
|
||||
### Services with Metrics Servers
|
||||
|
||||
| Service | Port | Custom Metrics | Notes |
|
||||
|---|---|---|---|
|
||||
| api-gateway | 9091 | 48 | Auth pipeline, JWT, RBAC, config reload, routing |
|
||||
| trading-service | 9092 | 10 | Latency gauges, ring buffer, uptime |
|
||||
| ml-training-service | 9094 | 5 | Jobs started/completed, errors, workers, uptime |
|
||||
| backtesting-service | 9093 | 4 | Backtests started/completed, errors, uptime |
|
||||
| trading-agent-service | 9095 | 3 | Portfolio value, assets selected, universe size |
|
||||
| broker-gateway | 9096 | 0 custom | process_* only, metrics.rs has definitions but unused |
|
||||
|
||||
### Services WITHOUT Metrics Servers
|
||||
|
||||
| Service | Issue |
|
||||
|---|---|
|
||||
| data-acquisition-service | No metrics server on 9097 (port declared in K8s YAML but nothing listens) |
|
||||
| web-gateway | No metrics port at all |
|
||||
|
||||
### ML Metrics (Defined but Dormant)
|
||||
|
||||
`trading-service/src/metrics.rs` and `ml_metrics.rs` define ~35 ML metrics (predictions, accuracy, ensemble, PnL, drawdown, circuit breaker, drift) using `Lazy<CounterVec/GaugeVec/HistogramVec>`. These are registered with the global Prometheus registry but only populated when ML predictions flow through the trading-service. They will automatically appear on `/metrics` once traffic runs.
|
||||
|
||||
### Training Metrics Gap
|
||||
|
||||
Training runs are K8s batch Jobs (`job-template.yaml`). They terminate after training completes. Prometheus can scrape them while running but misses data after completion. No Pushgateway exists to persist epoch-level metrics (loss, accuracy, learning rate).
|
||||
|
||||
## Design — Four Layers
|
||||
|
||||
### Layer 1: Fix Dashboard Queries
|
||||
|
||||
Update all 4 dashboards to use real metric names. Complete mapping:
|
||||
|
||||
#### Trading Cockpit (`foxhunt-trading-cockpit.json`)
|
||||
|
||||
| Panel | Current Query | Real Query |
|
||||
|---|---|---|
|
||||
| Ensemble Agreement | `foxhunt_ensemble_agreement_rate` | `ML_ENSEMBLE_AGREEMENT_RATE{symbol=~"$symbol"}` |
|
||||
| Prediction Confidence | `foxhunt_prediction_confidence` | `histogram_quantile(0.5, ML_PREDICTIONS_CONFIDENCE_bucket)` |
|
||||
| Model Accuracy | `foxhunt_model_accuracy` | `ML_MODEL_ACCURACY{model_id=~"$model"}` |
|
||||
| Ensemble Sharpe | `foxhunt_ensemble_sharpe` | `ML_MODEL_SHARPE_RATIO{model_id="ensemble"}` |
|
||||
| Active Positions | `foxhunt_active_positions` | `trading_agent_assets_selected` |
|
||||
| Realized PnL | `foxhunt_realized_pnl` | `ML_MODEL_CUMULATIVE_PNL{model_id=~"$model"}` |
|
||||
| Unrealized PnL | `foxhunt_unrealized_pnl` | `trading_agent_portfolio_value_usd` |
|
||||
| Max Drawdown | `foxhunt_max_drawdown` | `ML_MODEL_MAX_DRAWDOWN{model_id=~"$model"}` |
|
||||
| PnL Over Time | `foxhunt_pnl_total` | `ML_MODEL_CUMULATIVE_PNL{model_id=~"$model"}` |
|
||||
| Portfolio VaR | `foxhunt_portfolio_var` | `foxhunt_risk_var_portfolio` (needs Layer 2) |
|
||||
| Circuit Breaker | `foxhunt_circuit_breaker_state` | `ML_CIRCUIT_BREAKER_TRANSITIONS{model_id=~"$model"}` |
|
||||
| Risk Alerts | `foxhunt_risk_alerts_total` | `ML_ALERTS_TOTAL{severity="critical"}` |
|
||||
| Position HHI | `foxhunt_position_hhi` | `foxhunt_risk_position_hhi` (needs Layer 2) |
|
||||
| Inference Latency p95 | `foxhunt_inference_latency_seconds` | `histogram_quantile(0.95, ML_MODEL_INFERENCE_LATENCY_bucket)` |
|
||||
| Model Drift | `foxhunt_model_drift` | `ML_MODEL_DRIFT_SCORE{model_id=~"$model"}` |
|
||||
| gRPC Latency p95 | `foxhunt_grpc_latency_seconds` | `histogram_quantile(0.95, grpc_server_handling_seconds_bucket{service=~"$service"})` (needs Layer 3) |
|
||||
| ML Inference Distribution | `foxhunt_ml_inference_duration_seconds` | `ML_MODEL_INFERENCE_LATENCY_bucket` |
|
||||
|
||||
#### Training Cockpit (`foxhunt-training-cockpit.json`)
|
||||
|
||||
| Panel | Current Query | Real Query |
|
||||
|---|---|---|
|
||||
| Running Jobs | `foxhunt_training_jobs_active` | `kube_job_status_active{namespace="foxhunt",job_name=~"training-.*"}` |
|
||||
| Active Workers | `foxhunt_training_active_workers` | `ml_training_active_workers` |
|
||||
| Jobs Started/Completed/Failed | `foxhunt_training_jobs_*` | `ml_training_jobs_started_total` / `ml_training_jobs_completed_total` / `ml_training_errors_total` |
|
||||
| Pending Jobs | `foxhunt_training_jobs_pending` | `kube_job_status_active{namespace="foxhunt"} - kube_job_status_succeeded{namespace="foxhunt"}` |
|
||||
| Loss Curves | `foxhunt_training_loss` | `foxhunt_training_epoch_loss{model=~"$model_type"}` (needs Layer 4 Pushgateway) |
|
||||
| Epoch Progress | `foxhunt_training_epoch` | `foxhunt_training_current_epoch{model=~"$model_type"}` (needs Layer 4) |
|
||||
| Convergence Rate | `foxhunt_training_convergence_rate` | `deriv(foxhunt_training_epoch_loss[5m])` (needs Layer 4) |
|
||||
| Training Speed | `foxhunt_training_steps_per_second` | `foxhunt_training_batches_per_second{model=~"$model_type"}` (needs Layer 4) |
|
||||
| GPU Utilization/Memory/Temp/Power | `foxhunt_gpu_*` | `DCGM_FI_DEV_GPU_UTIL`, `DCGM_FI_DEV_FB_USED`, `DCGM_FI_DEV_GPU_TEMP`, `DCGM_FI_DEV_POWER_USAGE` |
|
||||
| Accuracy / F1 | `foxhunt_training_accuracy` | `foxhunt_training_eval_accuracy{model=~"$model_type"}` (needs Layer 4) |
|
||||
| Checkpoint Saves/Failures/Duration/Size | `foxhunt_checkpoint_*` | `foxhunt_training_checkpoint_*` (needs Layer 4) |
|
||||
| Batches Processed | `foxhunt_training_batches_total` | `foxhunt_training_batches_processed{model=~"$model_type"}` (needs Layer 4) |
|
||||
| Data Loading Latency | `foxhunt_data_loading_latency` | `foxhunt_training_data_load_seconds{model=~"$model_type"}` (needs Layer 4) |
|
||||
| Feature/NaN/Gradient Errors | `foxhunt_training_*_errors` | `foxhunt_training_nan_detected` / `foxhunt_training_gradient_explosion` (needs Layer 4) |
|
||||
| Job Duration | `foxhunt_training_job_duration` | `kube_job_complete_time - kube_job_start_time` |
|
||||
|
||||
#### Traces Dashboard (`foxhunt-traces.json`)
|
||||
|
||||
| Panel | Current Query | Real Query |
|
||||
|---|---|---|
|
||||
| Spans/sec | `foxhunt_spans_total` | `rate(tempo_distributor_spans_received_total[5m])` |
|
||||
| Error Rate | `foxhunt_trace_errors_total` | `rate(tempo_distributor_spans_received_total{status="error"}[5m])` or Tempo-derived |
|
||||
| Avg Duration | `foxhunt_span_duration_seconds` | Tempo TraceQL: `{} | avg(duration)` |
|
||||
| Active Services | `foxhunt_active_services` | `count(count by (service_name)(tempo_distributor_spans_received_total))` |
|
||||
| Service Map | nodeGraph from Tempo | Tempo service graph (already correct) |
|
||||
| Latency by Service p95 | `foxhunt_span_duration_seconds` | `histogram_quantile(0.95, tempo_span_metrics_duration_bucket{service=~"$service"})` |
|
||||
|
||||
#### Updated Cockpit (`foxhunt-cockpit.json` — Trace Health row)
|
||||
|
||||
Panels already use Tempo metrics (`tempo_distributor_spans_received_total`) — mostly correct, minor fixes needed.
|
||||
|
||||
### Layer 2: Add Metrics Servers to Missing Services
|
||||
|
||||
#### data-acquisition-service (port 9097)
|
||||
|
||||
Add `metrics.rs` + Axum HTTP server following the trading-service pattern:
|
||||
|
||||
```
|
||||
foxhunt_data_acquisition_feeds_active gauge Active data feeds
|
||||
foxhunt_data_acquisition_records_received_total counter Records received {symbol, feed_type}
|
||||
foxhunt_data_acquisition_errors_total counter Errors {error_type}
|
||||
foxhunt_data_acquisition_feed_latency_seconds histogram Feed processing latency
|
||||
foxhunt_data_acquisition_last_record_timestamp gauge Last record timestamp {symbol}
|
||||
foxhunt_data_acquisition_uptime_seconds counter Service uptime
|
||||
```
|
||||
|
||||
#### web-gateway (port 9098)
|
||||
|
||||
Add `metrics.rs` + handler in Axum router (web-gateway already uses Axum):
|
||||
|
||||
```
|
||||
foxhunt_web_gateway_http_requests_total counter HTTP requests {method, path, status}
|
||||
foxhunt_web_gateway_http_request_duration_seconds histogram Request latency {method, path}
|
||||
foxhunt_web_gateway_ws_connections_active gauge Active WebSocket connections
|
||||
foxhunt_web_gateway_ws_messages_total counter WS messages {direction, topic}
|
||||
foxhunt_web_gateway_uptime_seconds counter Service uptime
|
||||
```
|
||||
|
||||
#### K8s YAML Updates
|
||||
|
||||
- Add `metrics` named port (9098) to web-gateway deployment
|
||||
- Fix data-acquisition to actually bind metrics on 9097
|
||||
- Ensure `app.kubernetes.io/part-of: foxhunt` label on all services
|
||||
|
||||
### Layer 3: gRPC Metrics Tower Layer
|
||||
|
||||
Add `crates/common/src/metrics/grpc_metrics.rs` with a tonic Tower layer:
|
||||
|
||||
```
|
||||
grpc_server_handled_total{service, grpc_method, grpc_code} counter
|
||||
grpc_server_handling_seconds{service, grpc_method} histogram [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0]
|
||||
grpc_server_started_total{service, grpc_method} counter
|
||||
```
|
||||
|
||||
This uses the `tower::Layer` + `tower::Service` pattern to wrap every gRPC handler automatically. Each service adds one line in main.rs:
|
||||
|
||||
```rust
|
||||
Server::builder()
|
||||
.layer(common::metrics::grpc_metrics_layer("trading-service"))
|
||||
.add_service(...)
|
||||
```
|
||||
|
||||
### Layer 4: Training Pushgateway
|
||||
|
||||
#### Deploy Prometheus Pushgateway
|
||||
|
||||
Add `infra/k8s/monitoring/pushgateway.yaml`:
|
||||
- Image: `prom/pushgateway:v1.11.0`
|
||||
- Port: 9091 (internal, behind ClusterIP service)
|
||||
- Prometheus scrape config: job `pushgateway`, honor_labels true
|
||||
- Retention: ephemeral (no persistence needed, Prometheus scrapes frequently)
|
||||
|
||||
#### Training Binary Integration
|
||||
|
||||
Add `push_metrics()` to the training binaries (`train_baseline_rl`, `train_baseline_supervised`):
|
||||
|
||||
```
|
||||
foxhunt_training_current_epoch{model, job_id} gauge Current epoch number
|
||||
foxhunt_training_epoch_loss{model, job_id, split} gauge Loss value (train/val)
|
||||
foxhunt_training_eval_accuracy{model, job_id} gauge Eval accuracy
|
||||
foxhunt_training_batches_processed{model, job_id} counter Batches processed
|
||||
foxhunt_training_batches_per_second{model, job_id} gauge Training throughput
|
||||
foxhunt_training_data_load_seconds{model, job_id} histogram Data loading latency
|
||||
foxhunt_training_checkpoint_saves_total{model, job_id} counter Checkpoints saved
|
||||
foxhunt_training_checkpoint_duration_seconds{model} histogram Checkpoint save time
|
||||
foxhunt_training_nan_detected{model, job_id} counter NaN gradient events
|
||||
foxhunt_training_gradient_explosion{model, job_id} counter Gradient clipping events
|
||||
foxhunt_training_learning_rate{model, job_id} gauge Current learning rate
|
||||
```
|
||||
|
||||
Push frequency: every epoch end + every 60 seconds during training.
|
||||
Library: `prometheus` crate `push_metrics()` to `http://pushgateway:9091`.
|
||||
|
||||
#### Pushgateway Lifecycle
|
||||
|
||||
- Training job starts → pushes initial metrics with `job_id` label
|
||||
- During training → pushes every epoch + every 60s
|
||||
- Training job ends → Pushgateway retains last values until Prometheus scrapes
|
||||
- Prometheus scrapes Pushgateway every 15s → data persisted in Prometheus TSDB
|
||||
- Old `job_id` groups cleaned up by Pushgateway TTL or manual cleanup
|
||||
|
||||
## What This Does NOT Cover
|
||||
|
||||
- **Real-time P&L / position data**: Requires the trading engine to be connected to a live broker with real market data. The metrics infrastructure will be ready, but values will be 0 until live trading starts.
|
||||
- **QuestDB integration for dashboards**: QuestDB stores ML prediction history for the feedback loop. Grafana can query QuestDB via its PostgreSQL datasource, but that's a separate datasource setup, not Prometheus.
|
||||
- **Alerting rules**: We add the metrics and dashboards. Alertmanager rules for SLA/threshold alerts are a follow-up.
|
||||
|
||||
## Metric Naming Convention
|
||||
|
||||
All new metrics follow: `foxhunt_<service>_<metric>_<unit>`
|
||||
|
||||
- `foxhunt_` prefix for all application metrics
|
||||
- Service names: `data_acquisition`, `web_gateway`, `training`
|
||||
- Units: `_seconds`, `_total`, `_bytes`
|
||||
- Counter suffix: always `_total`
|
||||
- No `foxhunt_` prefix on gRPC layer metrics (standard `grpc_server_*` names for compatibility with existing Grafana community dashboards)
|
||||
|
||||
## File Changes Summary
|
||||
|
||||
| Layer | Files Created | Files Modified |
|
||||
|---|---|---|
|
||||
| L1: Fix Queries | 0 | 4 dashboard JSONs |
|
||||
| L2: Missing Metrics | 2 (metrics.rs × 2) | 2 main.rs + 2 K8s YAMLs |
|
||||
| L3: gRPC Layer | 1 (grpc_metrics.rs) | 8 service main.rs + common/metrics/mod.rs |
|
||||
| L4: Pushgateway | 1 (pushgateway.yaml) | ml training binaries, prometheus config, .gitlab-ci.yml |
|
||||
Reference in New Issue
Block a user