Root cause of DNS deadlock (2026-03-18, 2026-04-12): node-bootstrap
DaemonSet overwrote /etc/resolv.conf with ONLY nameserver 10.32.0.10
(cluster DNS). When nodes rebuild, CoreDNS can't pull its image because
containerd resolves via 10.32.0.10 which requires CoreDNS to be running.
Fix: resolv.conf now has DUAL nameservers:
nameserver 10.32.0.10 # primary: cluster DNS
nameserver 169.254.169.254 # fallback: Scaleway metadata DNS
Containerd tries kube-dns first (fast, resolves .svc.cluster.local),
falls back to VPC resolver when kube-dns is unreachable. Breaks the
chicken-and-egg permanently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rename node-dns-fix → node-bootstrap, fix nvidia gate race condition
(always create conf.d/ and write 50-registry.toml, don't gate on
99-nvidia.toml which doesn't exist on fresh autoscaled GPU nodes)
- Update busybox 1.36 → 1.37
- Fix fetch-binary: use PRIVATE-TOKEN/gitlab-pat (not DEPLOY-TOKEN)
- Fix upload-results: use gitlab-pat (gitlab-ci-token didn't exist)
- Add 'latest' rolling package version in both compile-services and
compile-training (re-upload after CalVer upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DaemonSet init container now:
- Detects GPU nodes (99-nvidia.toml in conf.d)
- Creates v3-compatible registry config drop-in
- Restarts containerd via chroot if config files changed
- Idempotent: skips restart if files already exist
Requires hostPID + privileged for chroot /proc/1/root.
On new nodes, first run creates files + restarts containerd
(kills pod), second run finds files unchanged (no restart).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Hyperopt: catch early stopping errors and return penalty metrics
instead of aborting entire run. Trials that stop early are scored
as poor (objective=-500) so optimizer avoids those configs.
- DaemonSet: detect GPU nodes (NVIDIA conf.d overlay) and create
v3-compatible registry config drop-in for containerd v2.1+.
Old grpc.v1.cri.registry path is silently ignored by containerd v2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule nodes use nameserver 127.0.53.53 which resolves external DNS
but returns NXDOMAIN for .svc.cluster.local. Containerd uses node DNS
for image pulls, causing ErrImagePull for cluster-internal registry refs.
Deploy node-dns-fix DaemonSet (kube-system) that:
1. Points /etc/resolv.conf to kube-dns ClusterIP (10.32.0.10)
2. Creates containerd certs.d config for HTTP GitLab registry
3. Watches for Kapsule node reconciliation resets (every 30s)
4. Restores original resolv.conf on SIGTERM
Also bump compile-services memory: 6Gi→10Gi limit (was OOM-killed at 5.7GB).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
