fix(metrics): wire training pod scraping to Grafana dashboard

Three root causes for "no metrics" on the training dashboard:

1. Dashboard template variables ($model, $fold) sourced from
   foxhunt_training_current_epoch which isn't emitted until the first
   epoch completes. Switch to foxhunt_training_step which fires from
   step 500 onward.

2. train.sh pod template missing Prometheus annotations
   (prometheus.io/scrape, port, path). Also add the
   app.kubernetes.io/component label to the eval manifest so
   evaluation pods are discoverable too.

3. DQN and PPO trainers only called set_epoch() at the END of each
   epoch. Move the call to the TOP of the epoch loop so the gauge
   exists from the first training iteration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-03-09 20:59:39 +01:00
parent 9d6a04ab4a
commit e89fbc2b4d
4 changed files with 19 additions and 2 deletions

View File

@@ -196,6 +196,10 @@ spec:
foxhunt/job-type: training
foxhunt/model: ${model}
foxhunt/preset: ${PRESET}
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9094"
prometheus.io/path: "/metrics"
spec:
restartPolicy: Never
nodeSelector:
@@ -341,7 +345,12 @@ spec:
metadata:
labels:
app.kubernetes.io/part-of: foxhunt
app.kubernetes.io/component: training-workflow
foxhunt/job-type: evaluate
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9094"
prometheus.io/path: "/metrics"
spec:
restartPolicy: Never
nodeSelector: