jgrusewski 9dc4d2585f fix(infra): rewrite dashboards with verified metrics, fix KSM + internal DNS ...

Rewrite all 5 Grafana dashboards using only confirmed-existing Prometheus
metric names (1023 metrics inventoried). Dashboards now use hardcoded
datasource UIDs instead of unresolvable ${DS_*} template variables.

- Cockpit: 38 panels (service health, gRPC, Argo CI/CD, trading, metrics)
- Training: 33 panels (epochs, loss, Sharpe, Q-values, GPU, hyperopt)
- Trading: 5 rows (overview, latency, gRPC pipeline, data acq, backtesting)
- Infrastructure: 25 panels (cluster, nodes, workloads, storage, GPU, Prometheus)
- Observability: 22 panels (API gateway auth/security/routing, Loki, Tempo)

Fix kube-state-metrics: add part-of label for netpol, fix nodeSelector
(infra→platform), increase memory limit (128→256Mi for OOM), add K8s API
egress rules covering both 10.32.0.0/16 and 172.16.0.0/16 CIDR ranges.

Add training-pods pod-based scrape job for ephemeral Argo workflow pods
exposing metrics on port 9094.

Migrate all image references from localhost:30500 to internal DNS
(gitlab-registry.foxhunt.svc.cluster.local:5000) across 14 YAML files.

Fix import.sh ConfigMap name (grafana-dashboards-infra → infrastructure).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-05 18:55:06 +01:00

1.7 KiB

Raw Blame History

View Raw