# Training Metrics in common::metrics

**Date:** 2026-03-01
**Status:** Approved

## Goal

Emit Prometheus metrics from training binaries so the Training Cockpit Grafana dashboard shows live loss curves, epoch progress, checkpoint stats, and error counters.

## Architecture

Two new files in `crates/common/src/metrics/`:

### `server.rs` — General-purpose Prometheus HTTP server

- `start_metrics_server(port: u16)` — spawns daemon thread with `std::net::TcpListener` on `0.0.0.0:{port}`
- Responds to `GET /metrics` with `gather_metrics()` (Prometheus text format)
- 404 for all other paths
- Reusable by any standalone binary (training, tools, future services)

### `training_metrics.rs` — Training-specific registration + typed helpers

- `init_training_metrics()` — registers all 18 dashboard-expected metrics
- 13 typed helper functions wrapping `set_gauge_vec`, `increment_counter_vec`, `observe_histogram_vec`
- Labels: `model` + `fold` on most metrics, `model` only on histograms, none on `active_workers`

## 18 Dashboard-Expected Metrics

**Gauges** (model, fold): current_epoch, epoch_loss, validation_loss, batches_per_second, batches_processed, iteration_seconds, eval_accuracy, eval_precision, eval_recall, eval_f1, checkpoint_size_bytes

**Counters** (model, fold): checkpoint_saves_total, checkpoint_failures_total, nan_detected_total, gradient_explosion_total, feature_errors_total

**Histograms** (model): checkpoint_duration_seconds, data_load_seconds

**Gauge** (no labels): active_workers

## Callers

- `train_baseline_supervised.rs` — full instrumentation (epoch, loss, checkpoint, NaN, data_load)
- `train_baseline_rl.rs` — full instrumentation for DQN and PPO loops
- `hyperopt_baseline_rl.rs` / `hyperopt_baseline_supervised.rs` — lifecycle only (init, server, active_workers)

## Port

9094 — matches existing K8s job annotations in `infra/k8s/training/job-template.yaml`.
