Commit Graph

6 Commits

Author SHA1 Message Date
jgrusewski
2357337bbd fix(infra): move Argo logs to dedicated argo-logs bucket
Argo artifact logs were stored in foxhunt-training-results which is
meant for model checkpoints. Separate bucket makes logs findable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 09:33:11 +01:00
jgrusewski
23a122607d feat(infra): Argo log persistence, network policies, lightweight CI images
- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
  in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 12:00:46 +01:00
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:52:24 +01:00
jgrusewski
70f5f4ff52 feat(argo): add basic auth login for Argo UI
Add nginx basic auth (htpasswd) to ci.fxhnt.ai proxy block and
switch Argo to server auth mode. Login: jgrusewski / Welcome01.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
810e7172e8 feat(argo): switch to client auth mode with SA token login
Switch Argo server from --auth-mode=server (no auth) to
--auth-mode=client (K8s ServiceAccount token). Created
jgrusewski-argo-token secret bound to argo-workflows-server SA
with admin RBAC for full workflow management.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00
jgrusewski
a2468846e1 feat(argo): add Argo Workflows infrastructure for training orchestration
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:58:40 +01:00