- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add deleteDelayDuration: 600s to podGC on CPU workflow templates and
sensors (ci-pipeline, compile-deploy, build-image). GPU training
workflows keep immediate cleanup to avoid wasting expensive GPU time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add notify-result exit handler to ci-pipeline, compile-and-deploy, and
compile-and-train templates. Posts workflow outcome to Mattermost via
incoming webhook (secret: notification-webhook). Gracefully skips if
webhook URL is placeholder or missing (optional secretKeyRef).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image builds are handled by ci-pipeline, not by compile-and-train or
compile-and-deploy. Having them here caused network policy failures
(Kaniko can't reach registry) and securityContext conflicts (Kaniko
needs root). Also removed pod-level securityContext from
compile-and-train since ci-builder needs root for cargo on PVC.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Wire build-image endpoint into EventSource + Sensor for manual triggers
- Add rebuild-* DAG tasks to compile-and-deploy and compile-and-train
templates — Kaniko layer cache makes cached builds ~15-30s
- Fix foxhunt-runtime Dockerfile: rename ubuntu user instead of groupadd
(GID 1000 already exists in ubuntu:24.04)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PVCs may retain ownership from a previous pod UID (e.g. after
ci-builder image rebuild). Git 2.35.2+ refuses to operate without
safe.directory set. Applied to both persistent-checkout templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
