Commit Graph

6 Commits

Author SHA1 Message Date
jgrusewski
ab218121b6 feat(monitoring): wire 12 new metric→proto field mappings in group_into_sessions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:09:54 +01:00
jgrusewski
7983302a52 proto(monitoring): add 12 new TrainingSession fields for diagnostics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:00:36 +01:00
jgrusewski
2f97402d5d fix(monitoring): use active_workers metric instead of kube_job query
CI training jobs run as GitLab runner pods, not K8s Jobs, so the
kube_job_status_active query always returned 0. Switch to
foxhunt_training_active_workers which is emitted by all training
binaries regardless of deployment method.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:21:15 +01:00
jgrusewski
85725abd8f feat(monitoring): per-epoch Prometheus metrics + CI scrape plumbing
- DQN/PPO trainers now record epoch, loss, val_loss, batch/s every epoch
- CI training jobs get Prometheus scrape annotations via runner overrides
- Allow Prometheus (foxhunt namespace) to scrape foxhunt-ci pods
- Skip no-model metrics in monitoring service session grouping

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:10:52 +01:00
jgrusewski
9e20337ee4 feat(monitoring): add monitoring_service crate with Prometheus-to-gRPC bridge
Prometheus client queries training/hyperopt/GPU/K8s metrics, gRPC service
exposes unary + server-streaming RPCs, 4 unit tests, zero clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 21:35:38 +01:00
jgrusewski
33d3cf2dd2 feat(monitoring): add monitoring.proto with gRPC service definition
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 21:21:29 +01:00