gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,782 Commits 13 Branches 0 Tags
ae704a7b7346da17039e4aeef8265f65c6df40ee
Commit Graph

9 Commits

Author SHA1 Message Date
jgrusewski
f3012611f0 feat(monitoring): migrate to kube-prometheus-stack Helm chart 
Replace standalone Prometheus/node-exporter/kube-state-metrics with
kube-prometheus-stack Helm chart. Add ServiceMonitor CRDs, PrometheusRule
CRDs (HFT + broker alerts), comprehensive network policies, and wire
AlertManager to Mattermost via incoming webhook.

- Delete old standalone prometheus.yaml, node-exporter.yaml, kube-state-metrics.yaml
- Add prometheus-stack-values.yaml (Helm values with Scaleway Kapsule tuning)
- Add service-monitors.yaml (foxhunt-services + dcgm-exporter ServiceMonitors)
- Add prometheus-rules-hft.yaml and prometheus-rules-broker.yaml (PrometheusRule CRDs)
- Rewrite prometheus network policy for operator-managed pods (DNS, kube-state-metrics,
  operator, alertmanager sidecar, monitoring namespace for DCGM)
- Add alertmanager network policy (Mattermost egress)
- Upgrade Grafana: AlertManager datasource, unified alerting, fix nodeSelector
- Fix Tempo: tolerations for GPU nodes, resource tuning

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-14 11:33:46 +01:00
jgrusewski
26d2ac7a5c feat(infra): redesign Grafana dashboards, consolidate to single Prometheus 
Replace 11 overlapping legacy dashboards with 5 focused dashboards:
- Operations Cockpit (services, Argo CI/CD, resources, auth)
- Trading & ML Intelligence (signals, execution, risk, gRPC)
- Training & CI/CD (training curves, GPU, hyperopt, Argo pipelines)
- Infrastructure & Cluster (nodes, pods, storage, network)
- Observability & API Gateway (logs/Loki, traces/Tempo, API metrics)

Remove gitlab-prometheus-server dependency — single Prometheus instance
with all scrape configs including Argo Workflows (HTTPS + skip verify).
Update Grafana datasource and import.sh for new dashboard structure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 16:50:52 +01:00
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab 
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 13:52:24 +01:00
jgrusewski
d501d53c9d chore(infra): remove dead Cockpit TF, CI-CD dashboards, stale Grafana configs 
- Delete infra/modules/cockpit/ and infra/live/production/cockpit/
  (Scaleway Cockpit replaced by self-hosted Grafana+Prometheus+Loki+Tempo)
- Delete CI-CD dashboards (foxhunt-ci-pipelines, gitlab-services) and
  grafana-dashboards-cicd ConfigMap from K8s
- Delete config/grafana/ — unreferenced old dashboards (15 files)
- Delete config/monitoring/grafana/ — unreferenced DQN staging dashboard
- Delete crates/ml/grafana/ — unreferenced ML performance dashboard
- Delete services/broker_gateway_service/grafana/ — unreferenced
- Delete .claude/agents/devops/ci-cd/ — GitHub Actions agent (we use GitLab CI)
- Fix grafana-values.yaml: dashboard folder GitLab → Foxhunt,
  hardcoded adminPassword → K8s secret (grafana-admin),
  gitlab-overview → node-exporter (correct name for gnetId 1860)
- Remove CI-CD group from import.sh ConfigMap groups and API fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 15:13:40 +01:00
jgrusewski
03fa993d54 chore(grafana): rotate admin password to random value 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 13:02:21 +01:00
jgrusewski
216240bcd5 feat(grafana): migrate from SQLite/PVC to PostgreSQL backend 
Grafana now stores users, preferences, and state in our existing
PostgreSQL instance (dedicated 'grafana' database). This eliminates
the 2Gi PVC dependency — all persistent state lives in Postgres,
dashboards in ConfigMaps, datasources in Helm values.

Also adds Grafana to Postgres NetworkPolicy ingress allowlist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 13:01:11 +01:00
jgrusewski
c608e62082 fix(grafana): restore full values file with sidecar, datasources, tolerations 
The Grafana PVC was lost during cleanup (orphaned PV deletion).
Dashboards survived (ConfigMap-based), but user accounts and
datasources were lost.

This commit captures all Helm values previously set via --set:
- Sidecar config with foldersFromFilesStructure for folder organization
- Loki + Tempo datasources alongside Prometheus
- Platform node toleration for gitlab taint

Also fixed CI/CD folder annotation (slash caused nested directory).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 12:33:42 +01:00
jgrusewski
17091500ed infra: move Grafana to platform pool 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 10:15:39 +01:00
Administrator
fa0e2a3809 infra: HTTPS wildcard cert, subdomain routing, CI/CD pipeline, Grafana 
- Wildcard LE cert for *.fxhnt.ai
- Subdomain routing: git/grafana/prometheus.fxhnt.ai
- CI/CD: Kaniko image builds, sccache, deploy stage
- Grafana with Prometheus datasource
- GitLab node upgraded to GP1-XS (16GB)
 2026-02-25 01:18:21 +00:00