Commit Graph

11 Commits

Author SHA1 Message Date
jgrusewski
a0bd744e5b fix(infra): remove stale DNS outputs, add chat/mail FQDNs
Remove prometheus_fqdn and monitor_fqdn outputs that referenced
deleted DNS records. Add chat_fqdn and mail_fqdn for Mattermost
and Stalwart services.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 14:47:04 +01:00
jgrusewski
9e8b9422cf feat(infra): deploy Mattermost on platform nodes with CI webhook egress
Add Mattermost Team Edition deployment (PVC, Service, NetworkPolicy),
DNS records (chat.fxhnt.ai, mail.fxhnt.ai), Tailscale nginx proxy
server block with WebSocket upgrade support, and Mattermost egress
rules to all 5 Argo workflow NetworkPolicies for exit handler webhooks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 09:20:32 +01:00
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:52:24 +01:00
jgrusewski
370752f253 feat(infra): expose Argo Workflows UI at ci.fxhnt.ai
Rename DNS record from argo→ci and update nginx proxy server_name
to match. Applied via terragrunt and kubectl.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
a2468846e1 feat(argo): add Argo Workflows infrastructure for training orchestration
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
cc209aec7e feat(infra): replace PVC binary distribution with MinIO S3 fetch
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.

Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
  aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 16:46:57 +01:00
jgrusewski
6b67f6193a infra: expose monitoring service via monitor.fxhnt.ai Tailscale proxy
Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 22:56:40 +01:00
jgrusewski
48316d6874 feat(infra): add api.fxhnt.ai DNS + gRPC reverse proxy for API Gateway
Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 22:41:55 +01:00
jgrusewski
3bdefe86d5 feat(monitoring): production dashboards, remove Cockpit, add Kapsule logs
- Remove Cockpit remote_write from GitLab Prometheus (all metrics local)
- Uninstall k8s-monitoring Helm (Alloy no longer pushes to Cockpit)
- Deploy standalone node-exporter DaemonSet + kube-state-metrics
- Add 10 extra Prometheus scrape configs: node-exporter, kube-state-metrics,
  kubelet, cadvisor, DCGM GPU, foxhunt services, Loki, Tempo, Promtail,
  prometheus.io annotated pods
- Provision 9 Grafana dashboards via sidecar (ConfigMap-based):
  overview, services, GPU/training, infrastructure, logs, CI/CD pipelines,
  cluster overview, GPU overview, GitLab services
- Expand Promtail to ingest kube-system + GitLab runner + node-level logs
- Add dashboard.fxhnt.ai DNS record + nginx routing to Grafana
- Update CI deploy job to apply new monitoring manifests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 22:20:32 +01:00
jgrusewski
d271315e7e infra(dns): add grafana + prometheus A records to Terraform
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:46:56 +01:00
jgrusewski
020e85af2b infra: add DNS module + rename gitlab.fxhnt.ai → git.fxhnt.ai
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:26:51 +01:00