foxhunt/docs/plans/2026-03-01-training-metrics-common-design.md

Training Metrics in common::metrics

Date: 2026-03-01 Status: Approved

Goal

Emit Prometheus metrics from training binaries so the Training Cockpit Grafana dashboard shows live loss curves, epoch progress, checkpoint stats, and error counters.

Architecture

Two new files in crates/common/src/metrics/:

server.rs — General-purpose Prometheus HTTP server

• start_metrics_server(port: u16) — spawns daemon thread with std::net::TcpListener on 0.0.0.0:{port}
• Responds to GET /metrics with gather_metrics() (Prometheus text format)
• 404 for all other paths
• Reusable by any standalone binary (training, tools, future services)

training_metrics.rs — Training-specific registration + typed helpers

• init_training_metrics() — registers all 18 dashboard-expected metrics
• 13 typed helper functions wrapping set_gauge_vec, increment_counter_vec, observe_histogram_vec
• Labels: model + fold on most metrics, model only on histograms, none on active_workers

18 Dashboard-Expected Metrics

Gauges (model, fold): current_epoch, epoch_loss, validation_loss, batches_per_second, batches_processed, iteration_seconds, eval_accuracy, eval_precision, eval_recall, eval_f1, checkpoint_size_bytes

Counters (model, fold): checkpoint_saves_total, checkpoint_failures_total, nan_detected_total, gradient_explosion_total, feature_errors_total

Histograms (model): checkpoint_duration_seconds, data_load_seconds

Gauge (no labels): active_workers

Callers

• train_baseline_supervised.rs — full instrumentation (epoch, loss, checkpoint, NaN, data_load)
• train_baseline_rl.rs — full instrumentation for DQN and PPO loops
• hyperopt_baseline_rl.rs / hyperopt_baseline_supervised.rs — lifecycle only (init, server, active_workers)

Port

9094 — matches existing K8s job annotations in infra/k8s/training/job-template.yaml.