fix(metrics): wire training pod scraping to Grafana dashboard
Three root causes for "no metrics" on the training dashboard: 1. Dashboard template variables ($model, $fold) sourced from foxhunt_training_current_epoch which isn't emitted until the first epoch completes. Switch to foxhunt_training_step which fires from step 500 onward. 2. train.sh pod template missing Prometheus annotations (prometheus.io/scrape, port, path). Also add the app.kubernetes.io/component label to the eval manifest so evaluation pods are discoverable too. 3. DQN and PPO trainers only called set_epoch() at the END of each epoch. Move the call to the TOP of the epoch loop so the gauge exists from the first training iteration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -196,6 +196,10 @@ spec:
|
||||
foxhunt/job-type: training
|
||||
foxhunt/model: ${model}
|
||||
foxhunt/preset: ${PRESET}
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9094"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
nodeSelector:
|
||||
@@ -341,7 +345,12 @@ spec:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
app.kubernetes.io/component: training-workflow
|
||||
foxhunt/job-type: evaluate
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9094"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
nodeSelector:
|
||||
|
||||
Reference in New Issue
Block a user