- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch sccache backend from S3 (MinIO) to persistent local storage.
Two 20Gi PVCs (sccache-cpu, sccache-cuda) on scw-bssd-retain eliminate
network I/O for cache reads/writes. Both compile steps always run on
the same node so RWO is sufficient.
Removes 13 S3 env vars per compile step, replaces with SCCACHE_DIR=/sccache.
Updates ci-pipeline-template, compile-and-train-template, kustomization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
build-ci-image used volumeClaimTemplates (PVC) shared between a
git-clone pod and a kaniko-build pod in a DAG. When called via
templateRef from ci-pipeline, the VCT wasn't inherited, causing
"volume 'workspace' not found" errors.
Fixed by merging into a single pod: init container (git clone) +
main container (kaniko build) sharing an emptyDir volume. No PVC
needed, works correctly when called via templateRef.
Also removed stale compile-training-template.yaml reference from
kustomization (file doesn't exist, template is inline in ci-pipeline).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three issues prevented services from being deployed after push:
1. No deploy step: pipeline compiled + uploaded binaries to MinIO but
never restarted service deployments. Added deploy-services step
(bitnami/kubectl) that runs rollout restart after compile-services.
2. compile-services couldn't schedule: targeted platform node (4 CPU,
6Gi allocatable) but requested 6Gi memory. Moved to ci-compile-cpu
(POP2-32C-128G) alongside compile-training — both fit simultaneously
(28/32 CPU, 48/128Gi).
3. No podGC: completed Argo pods held resources indefinitely, blocking
subsequent pipeline steps. Added podGC:OnPodCompletion.
Also: detect-changes now triggers service rebuild when CI pipeline
template or k8s service manifests change.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace compile-api-gateway + compile-web-gateway with single compile-api task
- Update detect-changes script: services/api/ instead of services/api_gateway/
- Remove web-gateway from change detection and compile targets
- Fix templateRef parameter resolution (inputs.parameters instead of workflow.parameters)
- Add SSH key setup and cluster-internal GitLab host for compile-service
- Use ci-builder image (has ssh) instead of ci-builder-cpu for service compilation
- Add compile-training and build-ci-image templates to kustomization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The default-deny-all NetworkPolicy blocks all egress for foxhunt-labeled
pods. Argo workflow pods need:
- Egress to MinIO (9000) for binary fetch and result upload
- Egress to K8s API (443/6443) for executor task result reporting
- Egress to Tempo (4317) for OTLP traces
- DNS already covered by allow-dns policy
Also fix PVC permission denied by setting fsGroup: 1000 to match the
foxhunt user (UID 1000) in the runtime images.
Smoke test fetch-binary step: Succeeded.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
