9dc4d2585f
Rewrite all 5 Grafana dashboards using only confirmed-existing Prometheus
metric names (1023 metrics inventoried). Dashboards now use hardcoded
datasource UIDs instead of unresolvable ${DS_*} template variables.
- Cockpit: 38 panels (service health, gRPC, Argo CI/CD, trading, metrics)
- Training: 33 panels (epochs, loss, Sharpe, Q-values, GPU, hyperopt)
- Trading: 5 rows (overview, latency, gRPC pipeline, data acq, backtesting)
- Infrastructure: 25 panels (cluster, nodes, workloads, storage, GPU, Prometheus)
- Observability: 22 panels (API gateway auth/security/routing, Loki, Tempo)
Fix kube-state-metrics: add part-of label for netpol, fix nodeSelector
(infra→platform), increase memory limit (128→256Mi for OOM), add K8s API
egress rules covering both 10.32.0.0/16 and 172.16.0.0/16 CIDR ranges.
Add training-pods pod-based scrape job for ephemeral Argo workflow pods
exposing metrics on port 9094.
Migrate all image references from localhost:30500 to internal DNS
(gitlab-registry.foxhunt.svc.cluster.local:5000) across 14 YAML files.
Fix import.sh ConfigMap name (grafana-dashboards-infra → infrastructure).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
61 lines
1.7 KiB
YAML
61 lines
1.7 KiB
YAML
# DaemonSet image pre-puller — keeps training images cached on GPU nodes
|
|
# Runs on ci-training (L40S) pool so training jobs skip the pull.
|
|
# Init containers pull :latest tags, then the main container sleeps forever.
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: image-prepuller
|
|
namespace: foxhunt
|
|
labels:
|
|
app: image-prepuller
|
|
app.kubernetes.io/part-of: foxhunt
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
app: image-prepuller
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: image-prepuller
|
|
spec:
|
|
affinity:
|
|
nodeAffinity:
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
nodeSelectorTerms:
|
|
- matchExpressions:
|
|
- key: k8s.scaleway.com/pool-name
|
|
operator: In
|
|
values:
|
|
- ci-training
|
|
tolerations:
|
|
- key: nvidia.com/gpu
|
|
operator: Exists
|
|
effect: NoSchedule
|
|
- key: node.cilium.io/agent-not-ready
|
|
operator: Exists
|
|
effect: NoSchedule
|
|
imagePullSecrets:
|
|
- name: gitlab-registry
|
|
initContainers:
|
|
- name: pull-training-runtime
|
|
image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
|
|
command: ["echo", "foxhunt-training-runtime image pulled"]
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 16Mi
|
|
limits:
|
|
cpu: 10m
|
|
memory: 16Mi
|
|
containers:
|
|
- name: pause
|
|
image: registry.k8s.io/pause:3.10
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 16Mi
|
|
limits:
|
|
cpu: 10m
|
|
memory: 16Mi
|