- Rename node-dns-fix → node-bootstrap, fix nvidia gate race condition
(always create conf.d/ and write 50-registry.toml, don't gate on
99-nvidia.toml which doesn't exist on fresh autoscaled GPU nodes)
- Update busybox 1.36 → 1.37
- Fix fetch-binary: use PRIVATE-TOKEN/gitlab-pat (not DEPLOY-TOKEN)
- Fix upload-results: use gitlab-pat (gitlab-ci-token didn't exist)
- Add 'latest' rolling package version in both compile-services and
compile-training (re-upload after CalVer upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DaemonSet init container now:
- Detects GPU nodes (99-nvidia.toml in conf.d)
- Creates v3-compatible registry config drop-in
- Restarts containerd via chroot if config files changed
- Idempotent: skips restart if files already exist
Requires hostPID + privileged for chroot /proc/1/root.
On new nodes, first run creates files + restarts containerd
(kills pod), second run finds files unchanged (no restart).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Hyperopt: catch early stopping errors and return penalty metrics
instead of aborting entire run. Trials that stop early are scored
as poor (objective=-500) so optimizer avoids those configs.
- DaemonSet: detect GPU nodes (NVIDIA conf.d overlay) and create
v3-compatible registry config drop-in for containerd v2.1+.
Old grpc.v1.cri.registry path is silently ignored by containerd v2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule nodes use nameserver 127.0.53.53 which resolves external DNS
but returns NXDOMAIN for .svc.cluster.local. Containerd uses node DNS
for image pulls, causing ErrImagePull for cluster-internal registry refs.
Deploy node-dns-fix DaemonSet (kube-system) that:
1. Points /etc/resolv.conf to kube-dns ClusterIP (10.32.0.10)
2. Creates containerd certs.d config for HTTP GitLab registry
3. Watches for Kapsule node reconciliation resets (every 30s)
4. Restores original resolv.conf on SIGTERM
Also bump compile-services memory: 6Gi→10Gi limit (was OOM-killed at 5.7GB).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
