jgrusewski
dba50fa3be
fix(dashboards): update datasource UIDs and folder annotations for Helm Prometheus
...
Update all dashboard JSON files to use Helm chart's Prometheus datasource
UID and add Grafana sidecar folder annotations for auto-provisioning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-14 11:34:14 +01:00
jgrusewski
ddf697641a
feat(monitoring): add per-service and platform dashboards, reorganize folder structure
...
- 7 application service dashboards (Services folder): API Gateway, Trading
Service, Backtesting, Data Acquisition, ML Training, Trading Agent, Web Gateway
- 9 platform dashboards (Platform folder): Prometheus, Grafana, Loki & Promtail,
Tempo, Redis, PostgreSQL, Argo Workflows, QuestDB, Cluster & Costs
- Cluster & Costs dashboard includes Scaleway node cost estimates (€/hr, €/month)
for platform, GPU (H100), and CI compile nodes
- Reorganized existing dashboards into 7 folders: CI, Infrastructure, Operations,
Platform, Services, Trading, Training (dropped "Foxhunt" prefix from titles)
- Rewrote import.sh: removed ConfigMap deployment, switched to API-only imports
with automatic folder creation and per-dashboard folder mapping
- All 24 dashboards deployed via Grafana API (Postgres-backed)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-14 02:30:52 +01:00
jgrusewski
2b77df7dda
fix(dashboards): correct metric prefixes, gRPC code filters, and service labels
...
- Replace api_gateway_ with api_ prefix to match actual metric exports (cockpit, observability)
- Change grpc_code!="OK" to grpc_code!="0" — gRPC uses numeric status codes (cockpit, trading)
- Update gRPC service filter to actual exported_service labels: trading-agent-service,
broker-gateway, backtesting-service, ml-training-service, data-acquisition-service (trading)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-05 19:02:32 +01:00
jgrusewski
9dc4d2585f
fix(infra): rewrite dashboards with verified metrics, fix KSM + internal DNS
...
Rewrite all 5 Grafana dashboards using only confirmed-existing Prometheus
metric names (1023 metrics inventoried). Dashboards now use hardcoded
datasource UIDs instead of unresolvable ${DS_*} template variables.
- Cockpit: 38 panels (service health, gRPC, Argo CI/CD, trading, metrics)
- Training: 33 panels (epochs, loss, Sharpe, Q-values, GPU, hyperopt)
- Trading: 5 rows (overview, latency, gRPC pipeline, data acq, backtesting)
- Infrastructure: 25 panels (cluster, nodes, workloads, storage, GPU, Prometheus)
- Observability: 22 panels (API gateway auth/security/routing, Loki, Tempo)
Fix kube-state-metrics: add part-of label for netpol, fix nodeSelector
(infra→platform), increase memory limit (128→256Mi for OOM), add K8s API
egress rules covering both 10.32.0.0/16 and 172.16.0.0/16 CIDR ranges.
Add training-pods pod-based scrape job for ephemeral Argo workflow pods
exposing metrics on port 9094.
Migrate all image references from localhost:30500 to internal DNS
(gitlab-registry.foxhunt.svc.cluster.local:5000) across 14 YAML files.
Fix import.sh ConfigMap name (grafana-dashboards-infra → infrastructure).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-05 18:55:06 +01:00
jgrusewski
b3c85d4f7c
fix(infra): resolve dashboard datasource UIDs, fix Grafana→Prometheus netpol
...
- Remove duplicate annotated-pods and gitlab-annotated-pods scrape jobs
(all targets already covered by dedicated jobs)
- Drop Argo controller from foxhunt-services job (dedicated HTTPS job exists)
- Resolve datasource variable placeholders to actual UIDs in dashboard JSON
- Add Ingress rule to Prometheus netpol allowing Grafana on port 9090
Targets: 49 → 18 (zero duplicates)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-05 17:43:25 +01:00
jgrusewski
26d2ac7a5c
feat(infra): redesign Grafana dashboards, consolidate to single Prometheus
...
Replace 11 overlapping legacy dashboards with 5 focused dashboards:
- Operations Cockpit (services, Argo CI/CD, resources, auth)
- Trading & ML Intelligence (signals, execution, risk, gRPC)
- Training & CI/CD (training curves, GPU, hyperopt, Argo pipelines)
- Infrastructure & Cluster (nodes, pods, storage, network)
- Observability & API Gateway (logs/Loki, traces/Tempo, API metrics)
Remove gitlab-prometheus-server dependency — single Prometheus instance
with all scrape configs including Argo Workflows (HTTPS + skip verify).
Update Grafana datasource and import.sh for new dashboard structure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-05 16:50:52 +01:00
