# DNS Bootstrap Prevention — Prevent CoreDNS chicken-and-egg deadlock # # ROOT CAUSE (2026-03-18 incident): # 1. Rolling replacement of ALL platform nodes simultaneously # 2. New nodes boot with /etc/resolv.conf → 10.32.0.10 (kube-dns only) # 3. CoreDNS pods need to pull images → need DNS → need CoreDNS # 4. Cilium BPF socket LB returns EPERM when kube-dns has no endpoints # 5. Complete cluster DNS deadlock — nothing can pull images # # PREVENTION: # 1. VPC DefaultRoutePropagation MUST be enabled (API, not Terraform yet) # 2. NEVER replace ALL nodes at once — one at a time, wait for CoreDNS # 3. Do NOT create CiliumClusterwideNetworkPolicy with egress rules # (enables default deny, makes the problem worse) # 4. bpf-lb-sock MUST remain "true" — disabling breaks ALL service routing # 5. Keep bastion enabled on VPC gateway for emergency SSH access # # RECOVERY PROCEDURE: # 1. SSH via bastion: ssh -o ProxyCommand="ssh -W %h:%p -p 61000 bastion@" root@ # 2. On the node: /usr/local/bin/crictl pull # 3. kubectl delete pods -l k8s-app=kube-dns --force # 4. CoreDNS starts from cached image → DNS recovers → cascade # # DATA PROTECTION: # All PVs set to Retain reclaim policy — data survives cluster deletion