46 lines
1.9 KiB
Markdown
46 lines
1.9 KiB
Markdown
# Training Metrics in common::metrics
|
|
|
|
**Date:** 2026-03-01
|
|
**Status:** Approved
|
|
|
|
## Goal
|
|
|
|
Emit Prometheus metrics from training binaries so the Training Cockpit Grafana dashboard shows live loss curves, epoch progress, checkpoint stats, and error counters.
|
|
|
|
## Architecture
|
|
|
|
Two new files in `crates/common/src/metrics/`:
|
|
|
|
### `server.rs` — General-purpose Prometheus HTTP server
|
|
|
|
- `start_metrics_server(port: u16)` — spawns daemon thread with `std::net::TcpListener` on `0.0.0.0:{port}`
|
|
- Responds to `GET /metrics` with `gather_metrics()` (Prometheus text format)
|
|
- 404 for all other paths
|
|
- Reusable by any standalone binary (training, tools, future services)
|
|
|
|
### `training_metrics.rs` — Training-specific registration + typed helpers
|
|
|
|
- `init_training_metrics()` — registers all 18 dashboard-expected metrics
|
|
- 13 typed helper functions wrapping `set_gauge_vec`, `increment_counter_vec`, `observe_histogram_vec`
|
|
- Labels: `model` + `fold` on most metrics, `model` only on histograms, none on `active_workers`
|
|
|
|
## 18 Dashboard-Expected Metrics
|
|
|
|
**Gauges** (model, fold): current_epoch, epoch_loss, validation_loss, batches_per_second, batches_processed, iteration_seconds, eval_accuracy, eval_precision, eval_recall, eval_f1, checkpoint_size_bytes
|
|
|
|
**Counters** (model, fold): checkpoint_saves_total, checkpoint_failures_total, nan_detected_total, gradient_explosion_total, feature_errors_total
|
|
|
|
**Histograms** (model): checkpoint_duration_seconds, data_load_seconds
|
|
|
|
**Gauge** (no labels): active_workers
|
|
|
|
## Callers
|
|
|
|
- `train_baseline_supervised.rs` — full instrumentation (epoch, loss, checkpoint, NaN, data_load)
|
|
- `train_baseline_rl.rs` — full instrumentation for DQN and PPO loops
|
|
- `hyperopt_baseline_rl.rs` / `hyperopt_baseline_supervised.rs` — lifecycle only (init, server, active_workers)
|
|
|
|
## Port
|
|
|
|
9094 — matches existing K8s job annotations in `infra/k8s/training/job-template.yaml`.
|