Prometheus was configured to scrape pods via prometheus.io/scrape
annotations, but all services and training jobs used gitlab.com/
prefix from legacy GitLab-managed Prometheus — causing Prometheus
to never discover any foxhunt pods. This resulted in stale/missing
metrics on the Grafana training dashboard.
12 files updated across services/, gpu-overlays/, training/, and
monitoring/ (node-exporter, dcgm-exporter cleanup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The S3→PVC migration had several issues causing deploy failures:
- Stale binary-writer pod from previous failed deploys blocked new ones
(terminated pods can't be updated via kubectl apply)
- Services were applied BEFORE binaries written to PVC, so first deploy
or empty PVC caused pods to crash (binary not found)
- GPU overlay files in services/ dir were auto-applied by kubectl apply,
referencing nonexistent S3/minio secrets and PVCs
- Training job template mounted foxhunt-binaries PVC but training runs
on ci-training node — RWO PVC is bound to foxhunt node
Fixes:
- Reorder deploy: write binaries → apply manifests → rollout restart
- Clean up stale writer pods before creating new ones
- Move GPU overlays to gpu-overlays/ (manual apply only), update to PVC
- Add training-binaries PVC for GPU node, best-effort population
- Training job template uses initContainer to copy from training PVC
- set -e for critical path, set +e for optional training binary copy
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
