Files
foxhunt/infra/scripts
jgrusewski e89fbc2b4d fix(metrics): wire training pod scraping to Grafana dashboard
Three root causes for "no metrics" on the training dashboard:

1. Dashboard template variables ($model, $fold) sourced from
   foxhunt_training_current_epoch which isn't emitted until the first
   epoch completes. Switch to foxhunt_training_step which fires from
   step 500 onward.

2. train.sh pod template missing Prometheus annotations
   (prometheus.io/scrape, port, path). Also add the
   app.kubernetes.io/component label to the eval manifest so
   evaluation pods are discoverable too.

3. DQN and PPO trainers only called set_epoch() at the END of each
   epoch. Move the call to the TOP of the epoch loop so the gauge
   exists from the first training iteration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:02:29 +01:00
..