Commit Graph

9 Commits

Author SHA1 Message Date
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:52:24 +01:00
jgrusewski
370752f253 feat(infra): expose Argo Workflows UI at ci.fxhnt.ai
Rename DNS record from argo→ci and update nginx proxy server_name
to match. Applied via terragrunt and kubectl.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
a2468846e1 feat(argo): add Argo Workflows infrastructure for training orchestration
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
cc209aec7e feat(infra): replace PVC binary distribution with MinIO S3 fetch
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.

Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
  aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 16:46:57 +01:00
jgrusewski
6b67f6193a infra: expose monitoring service via monitor.fxhnt.ai Tailscale proxy
Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 22:56:40 +01:00
jgrusewski
48316d6874 feat(infra): add api.fxhnt.ai DNS + gRPC reverse proxy for API Gateway
Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 22:41:55 +01:00
jgrusewski
3bdefe86d5 feat(monitoring): production dashboards, remove Cockpit, add Kapsule logs
- Remove Cockpit remote_write from GitLab Prometheus (all metrics local)
- Uninstall k8s-monitoring Helm (Alloy no longer pushes to Cockpit)
- Deploy standalone node-exporter DaemonSet + kube-state-metrics
- Add 10 extra Prometheus scrape configs: node-exporter, kube-state-metrics,
  kubelet, cadvisor, DCGM GPU, foxhunt services, Loki, Tempo, Promtail,
  prometheus.io annotated pods
- Provision 9 Grafana dashboards via sidecar (ConfigMap-based):
  overview, services, GPU/training, infrastructure, logs, CI/CD pipelines,
  cluster overview, GPU overview, GitLab services
- Expand Promtail to ingest kube-system + GitLab runner + node-level logs
- Add dashboard.fxhnt.ai DNS record + nginx routing to Grafana
- Update CI deploy job to apply new monitoring manifests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 22:20:32 +01:00
jgrusewski
d271315e7e infra(dns): add grafana + prometheus A records to Terraform
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:46:56 +01:00
jgrusewski
020e85af2b infra: add DNS module + rename gitlab.fxhnt.ai → git.fxhnt.ai
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:26:51 +01:00