jgrusewski
f3012611f0
feat(monitoring): migrate to kube-prometheus-stack Helm chart
...
Replace standalone Prometheus/node-exporter/kube-state-metrics with
kube-prometheus-stack Helm chart. Add ServiceMonitor CRDs, PrometheusRule
CRDs (HFT + broker alerts), comprehensive network policies, and wire
AlertManager to Mattermost via incoming webhook.
- Delete old standalone prometheus.yaml, node-exporter.yaml, kube-state-metrics.yaml
- Add prometheus-stack-values.yaml (Helm values with Scaleway Kapsule tuning)
- Add service-monitors.yaml (foxhunt-services + dcgm-exporter ServiceMonitors)
- Add prometheus-rules-hft.yaml and prometheus-rules-broker.yaml (PrometheusRule CRDs)
- Rewrite prometheus network policy for operator-managed pods (DNS, kube-state-metrics,
operator, alertmanager sidecar, monitoring namespace for DCGM)
- Add alertmanager network policy (Mattermost egress)
- Upgrade Grafana: AlertManager datasource, unified alerting, fix nodeSelector
- Fix Tempo: tolerations for GPU nodes, resource tuning
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-14 11:33:46 +01:00
jgrusewski
68b6aa8313
feat(infra): migrate container registry from SCW to internal GitLab
...
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.
Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml
Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules
DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform
Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP
Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-05 13:52:24 +01:00
jgrusewski
5fe3608d92
fix(fxt,infra): production hardening — OTLP telemetry, TUI fixes, K8s infra
...
- Remove opentelemetry-otlp internal-logs feature (OTLP feedback loop)
- Switch trace sampling from AlwaysOn to 10% ratio-based
- Add RUST_LOG filtering (opentelemetry/h2/tonic/hyper=warn) to all 8 services
- Wire per-service latency measurement via health check → proto metadata → TUI
- Replace Vec::remove(0) with VecDeque ring buffers (O(1) vs O(n))
- Add Arc<AtomicBool> connected_sent for first-connected detection across 12 streams
- Add MAX_RECONNECT_ATTEMPTS (10) uniformly to all stream spawners
- Change kill switch/circuit breaker fields to Option types with N/A display
- Wire data_cache to real download status stream, remove dead cluster_events
- Remove ServiceData::new() hardcoded stubs, add honest placeholders
- Fix nanos_to_hms zero/negative guard, total_records semantic fix
- Fix RwLock held across yield in broker_gateway stream_account_state
- Add break after yield Err in broker/trading stream generators
- Fix connected_at advancing per tick in stream_session_status
- Tempo: replace emptyDir with 10Gi PVC, bump memory to 512Mi/2Gi
- Remove Prometheus gitlab-annotated-pods duplicate scrape job
- Wire 6 new gRPC streaming adapters (risk, trading, ml, data-acquisition)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-04 10:39:56 +01:00
jgrusewski
a9edd4c0b3
infra: move monitoring stack (loki, tempo, kube-state-metrics, pushgateway) to platform pool
...
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-02 10:08:28 +01:00
jgrusewski
a36fce0b87
fix(infra): increase Tempo memory to 1Gi, RL runner CPU limits to 8000m
...
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-03-01 13:04:26 +01:00
jgrusewski
6c4482a5af
feat(monitoring): deploy Loki + Tempo + Promtail observability stack
...
- Loki (3.4.2): log aggregation on gitlab node, 7-day retention, TSDB storage
- Tempo (2.7.1): OTLP trace receiver on gitlab node, 7-day retention
- Promtail (3.4.2): DaemonSet log shipper with K8s pod discovery + RBAC
- Grafana datasources: Prometheus + Loki + Tempo with trace-to-log correlation
- Services: OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317 on ml-training,
trading, backtesting services
- CI deploy job: applies monitoring manifests alongside service deployments
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-02-28 21:32:01 +01:00
