diff --git a/docs/plans/2026-03-01-real-metrics-overhaul-design.md b/docs/plans/2026-03-01-real-metrics-overhaul-design.md new file mode 100644 index 000000000..87a8bd352 --- /dev/null +++ b/docs/plans/2026-03-01-real-metrics-overhaul-design.md @@ -0,0 +1,211 @@ +# Real Metrics Dashboard Overhaul — Design + +## Problem + +The 4 dashboards deployed on 2026-03-01 (Trading Cockpit, Training Cockpit, Traces, updated Cockpit) use placeholder `foxhunt_*` metric names in their PromQL queries. The services emit metrics under service-specific prefixes (`trading_*`, `api_gateway_*`, `ML_*`). Two services have no metrics server at all. No gRPC middleware exists for automatic request/response instrumentation. Training jobs can't push epoch-level metrics to Prometheus. + +## Current State + +### Services with Metrics Servers + +| Service | Port | Custom Metrics | Notes | +|---|---|---|---| +| api-gateway | 9091 | 48 | Auth pipeline, JWT, RBAC, config reload, routing | +| trading-service | 9092 | 10 | Latency gauges, ring buffer, uptime | +| ml-training-service | 9094 | 5 | Jobs started/completed, errors, workers, uptime | +| backtesting-service | 9093 | 4 | Backtests started/completed, errors, uptime | +| trading-agent-service | 9095 | 3 | Portfolio value, assets selected, universe size | +| broker-gateway | 9096 | 0 custom | process_* only, metrics.rs has definitions but unused | + +### Services WITHOUT Metrics Servers + +| Service | Issue | +|---|---| +| data-acquisition-service | No metrics server on 9097 (port declared in K8s YAML but nothing listens) | +| web-gateway | No metrics port at all | + +### ML Metrics (Defined but Dormant) + +`trading-service/src/metrics.rs` and `ml_metrics.rs` define ~35 ML metrics (predictions, accuracy, ensemble, PnL, drawdown, circuit breaker, drift) using `Lazy`. These are registered with the global Prometheus registry but only populated when ML predictions flow through the trading-service. They will automatically appear on `/metrics` once traffic runs. + +### Training Metrics Gap + +Training runs are K8s batch Jobs (`job-template.yaml`). They terminate after training completes. Prometheus can scrape them while running but misses data after completion. No Pushgateway exists to persist epoch-level metrics (loss, accuracy, learning rate). + +## Design — Four Layers + +### Layer 1: Fix Dashboard Queries + +Update all 4 dashboards to use real metric names. Complete mapping: + +#### Trading Cockpit (`foxhunt-trading-cockpit.json`) + +| Panel | Current Query | Real Query | +|---|---|---| +| Ensemble Agreement | `foxhunt_ensemble_agreement_rate` | `ML_ENSEMBLE_AGREEMENT_RATE{symbol=~"$symbol"}` | +| Prediction Confidence | `foxhunt_prediction_confidence` | `histogram_quantile(0.5, ML_PREDICTIONS_CONFIDENCE_bucket)` | +| Model Accuracy | `foxhunt_model_accuracy` | `ML_MODEL_ACCURACY{model_id=~"$model"}` | +| Ensemble Sharpe | `foxhunt_ensemble_sharpe` | `ML_MODEL_SHARPE_RATIO{model_id="ensemble"}` | +| Active Positions | `foxhunt_active_positions` | `trading_agent_assets_selected` | +| Realized PnL | `foxhunt_realized_pnl` | `ML_MODEL_CUMULATIVE_PNL{model_id=~"$model"}` | +| Unrealized PnL | `foxhunt_unrealized_pnl` | `trading_agent_portfolio_value_usd` | +| Max Drawdown | `foxhunt_max_drawdown` | `ML_MODEL_MAX_DRAWDOWN{model_id=~"$model"}` | +| PnL Over Time | `foxhunt_pnl_total` | `ML_MODEL_CUMULATIVE_PNL{model_id=~"$model"}` | +| Portfolio VaR | `foxhunt_portfolio_var` | `foxhunt_risk_var_portfolio` (needs Layer 2) | +| Circuit Breaker | `foxhunt_circuit_breaker_state` | `ML_CIRCUIT_BREAKER_TRANSITIONS{model_id=~"$model"}` | +| Risk Alerts | `foxhunt_risk_alerts_total` | `ML_ALERTS_TOTAL{severity="critical"}` | +| Position HHI | `foxhunt_position_hhi` | `foxhunt_risk_position_hhi` (needs Layer 2) | +| Inference Latency p95 | `foxhunt_inference_latency_seconds` | `histogram_quantile(0.95, ML_MODEL_INFERENCE_LATENCY_bucket)` | +| Model Drift | `foxhunt_model_drift` | `ML_MODEL_DRIFT_SCORE{model_id=~"$model"}` | +| gRPC Latency p95 | `foxhunt_grpc_latency_seconds` | `histogram_quantile(0.95, grpc_server_handling_seconds_bucket{service=~"$service"})` (needs Layer 3) | +| ML Inference Distribution | `foxhunt_ml_inference_duration_seconds` | `ML_MODEL_INFERENCE_LATENCY_bucket` | + +#### Training Cockpit (`foxhunt-training-cockpit.json`) + +| Panel | Current Query | Real Query | +|---|---|---| +| Running Jobs | `foxhunt_training_jobs_active` | `kube_job_status_active{namespace="foxhunt",job_name=~"training-.*"}` | +| Active Workers | `foxhunt_training_active_workers` | `ml_training_active_workers` | +| Jobs Started/Completed/Failed | `foxhunt_training_jobs_*` | `ml_training_jobs_started_total` / `ml_training_jobs_completed_total` / `ml_training_errors_total` | +| Pending Jobs | `foxhunt_training_jobs_pending` | `kube_job_status_active{namespace="foxhunt"} - kube_job_status_succeeded{namespace="foxhunt"}` | +| Loss Curves | `foxhunt_training_loss` | `foxhunt_training_epoch_loss{model=~"$model_type"}` (needs Layer 4 Pushgateway) | +| Epoch Progress | `foxhunt_training_epoch` | `foxhunt_training_current_epoch{model=~"$model_type"}` (needs Layer 4) | +| Convergence Rate | `foxhunt_training_convergence_rate` | `deriv(foxhunt_training_epoch_loss[5m])` (needs Layer 4) | +| Training Speed | `foxhunt_training_steps_per_second` | `foxhunt_training_batches_per_second{model=~"$model_type"}` (needs Layer 4) | +| GPU Utilization/Memory/Temp/Power | `foxhunt_gpu_*` | `DCGM_FI_DEV_GPU_UTIL`, `DCGM_FI_DEV_FB_USED`, `DCGM_FI_DEV_GPU_TEMP`, `DCGM_FI_DEV_POWER_USAGE` | +| Accuracy / F1 | `foxhunt_training_accuracy` | `foxhunt_training_eval_accuracy{model=~"$model_type"}` (needs Layer 4) | +| Checkpoint Saves/Failures/Duration/Size | `foxhunt_checkpoint_*` | `foxhunt_training_checkpoint_*` (needs Layer 4) | +| Batches Processed | `foxhunt_training_batches_total` | `foxhunt_training_batches_processed{model=~"$model_type"}` (needs Layer 4) | +| Data Loading Latency | `foxhunt_data_loading_latency` | `foxhunt_training_data_load_seconds{model=~"$model_type"}` (needs Layer 4) | +| Feature/NaN/Gradient Errors | `foxhunt_training_*_errors` | `foxhunt_training_nan_detected` / `foxhunt_training_gradient_explosion` (needs Layer 4) | +| Job Duration | `foxhunt_training_job_duration` | `kube_job_complete_time - kube_job_start_time` | + +#### Traces Dashboard (`foxhunt-traces.json`) + +| Panel | Current Query | Real Query | +|---|---|---| +| Spans/sec | `foxhunt_spans_total` | `rate(tempo_distributor_spans_received_total[5m])` | +| Error Rate | `foxhunt_trace_errors_total` | `rate(tempo_distributor_spans_received_total{status="error"}[5m])` or Tempo-derived | +| Avg Duration | `foxhunt_span_duration_seconds` | Tempo TraceQL: `{} | avg(duration)` | +| Active Services | `foxhunt_active_services` | `count(count by (service_name)(tempo_distributor_spans_received_total))` | +| Service Map | nodeGraph from Tempo | Tempo service graph (already correct) | +| Latency by Service p95 | `foxhunt_span_duration_seconds` | `histogram_quantile(0.95, tempo_span_metrics_duration_bucket{service=~"$service"})` | + +#### Updated Cockpit (`foxhunt-cockpit.json` — Trace Health row) + +Panels already use Tempo metrics (`tempo_distributor_spans_received_total`) — mostly correct, minor fixes needed. + +### Layer 2: Add Metrics Servers to Missing Services + +#### data-acquisition-service (port 9097) + +Add `metrics.rs` + Axum HTTP server following the trading-service pattern: + +``` +foxhunt_data_acquisition_feeds_active gauge Active data feeds +foxhunt_data_acquisition_records_received_total counter Records received {symbol, feed_type} +foxhunt_data_acquisition_errors_total counter Errors {error_type} +foxhunt_data_acquisition_feed_latency_seconds histogram Feed processing latency +foxhunt_data_acquisition_last_record_timestamp gauge Last record timestamp {symbol} +foxhunt_data_acquisition_uptime_seconds counter Service uptime +``` + +#### web-gateway (port 9098) + +Add `metrics.rs` + handler in Axum router (web-gateway already uses Axum): + +``` +foxhunt_web_gateway_http_requests_total counter HTTP requests {method, path, status} +foxhunt_web_gateway_http_request_duration_seconds histogram Request latency {method, path} +foxhunt_web_gateway_ws_connections_active gauge Active WebSocket connections +foxhunt_web_gateway_ws_messages_total counter WS messages {direction, topic} +foxhunt_web_gateway_uptime_seconds counter Service uptime +``` + +#### K8s YAML Updates + +- Add `metrics` named port (9098) to web-gateway deployment +- Fix data-acquisition to actually bind metrics on 9097 +- Ensure `app.kubernetes.io/part-of: foxhunt` label on all services + +### Layer 3: gRPC Metrics Tower Layer + +Add `crates/common/src/metrics/grpc_metrics.rs` with a tonic Tower layer: + +``` +grpc_server_handled_total{service, grpc_method, grpc_code} counter +grpc_server_handling_seconds{service, grpc_method} histogram [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0] +grpc_server_started_total{service, grpc_method} counter +``` + +This uses the `tower::Layer` + `tower::Service` pattern to wrap every gRPC handler automatically. Each service adds one line in main.rs: + +```rust +Server::builder() + .layer(common::metrics::grpc_metrics_layer("trading-service")) + .add_service(...) +``` + +### Layer 4: Training Pushgateway + +#### Deploy Prometheus Pushgateway + +Add `infra/k8s/monitoring/pushgateway.yaml`: +- Image: `prom/pushgateway:v1.11.0` +- Port: 9091 (internal, behind ClusterIP service) +- Prometheus scrape config: job `pushgateway`, honor_labels true +- Retention: ephemeral (no persistence needed, Prometheus scrapes frequently) + +#### Training Binary Integration + +Add `push_metrics()` to the training binaries (`train_baseline_rl`, `train_baseline_supervised`): + +``` +foxhunt_training_current_epoch{model, job_id} gauge Current epoch number +foxhunt_training_epoch_loss{model, job_id, split} gauge Loss value (train/val) +foxhunt_training_eval_accuracy{model, job_id} gauge Eval accuracy +foxhunt_training_batches_processed{model, job_id} counter Batches processed +foxhunt_training_batches_per_second{model, job_id} gauge Training throughput +foxhunt_training_data_load_seconds{model, job_id} histogram Data loading latency +foxhunt_training_checkpoint_saves_total{model, job_id} counter Checkpoints saved +foxhunt_training_checkpoint_duration_seconds{model} histogram Checkpoint save time +foxhunt_training_nan_detected{model, job_id} counter NaN gradient events +foxhunt_training_gradient_explosion{model, job_id} counter Gradient clipping events +foxhunt_training_learning_rate{model, job_id} gauge Current learning rate +``` + +Push frequency: every epoch end + every 60 seconds during training. +Library: `prometheus` crate `push_metrics()` to `http://pushgateway:9091`. + +#### Pushgateway Lifecycle + +- Training job starts → pushes initial metrics with `job_id` label +- During training → pushes every epoch + every 60s +- Training job ends → Pushgateway retains last values until Prometheus scrapes +- Prometheus scrapes Pushgateway every 15s → data persisted in Prometheus TSDB +- Old `job_id` groups cleaned up by Pushgateway TTL or manual cleanup + +## What This Does NOT Cover + +- **Real-time P&L / position data**: Requires the trading engine to be connected to a live broker with real market data. The metrics infrastructure will be ready, but values will be 0 until live trading starts. +- **QuestDB integration for dashboards**: QuestDB stores ML prediction history for the feedback loop. Grafana can query QuestDB via its PostgreSQL datasource, but that's a separate datasource setup, not Prometheus. +- **Alerting rules**: We add the metrics and dashboards. Alertmanager rules for SLA/threshold alerts are a follow-up. + +## Metric Naming Convention + +All new metrics follow: `foxhunt___` + +- `foxhunt_` prefix for all application metrics +- Service names: `data_acquisition`, `web_gateway`, `training` +- Units: `_seconds`, `_total`, `_bytes` +- Counter suffix: always `_total` +- No `foxhunt_` prefix on gRPC layer metrics (standard `grpc_server_*` names for compatibility with existing Grafana community dashboards) + +## File Changes Summary + +| Layer | Files Created | Files Modified | +|---|---|---| +| L1: Fix Queries | 0 | 4 dashboard JSONs | +| L2: Missing Metrics | 2 (metrics.rs × 2) | 2 main.rs + 2 K8s YAMLs | +| L3: gRPC Layer | 1 (grpc_metrics.rs) | 8 service main.rs + common/metrics/mod.rs | +| L4: Pushgateway | 1 (pushgateway.yaml) | ml training binaries, prometheus config, .gitlab-ci.yml |