Files
foxhunt/infra/k8s/monitoring/dashboards
jgrusewski 7384bd4c41 fix(dashboard): use real metric names in training cockpit
Replace 28 placeholder PromQL expressions with real metric names:
- Active Jobs: kube_job_status_active from kube-state-metrics
- Training Progress: foxhunt_training_epoch_loss, current_epoch, etc.
- GPU Resources: DCGM_FI_DEV_GPU_UTIL/FB_USED/GPU_TEMP/POWER_USAGE
- Model Quality: foxhunt_training_eval_accuracy/f1/precision/recall
- Checkpoints: foxhunt_training_checkpoint_saves/failures/duration/size
- Data Pipeline: foxhunt_training_batches_processed, data_load_seconds
- Errors: foxhunt_training_nan_detected/gradient_explosion/feature_errors
- Job History: kube_job_complete_time/start_time for duration

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 02:09:38 +01:00
..