ddfa57af11
- Added scaleway_vpc_public_gateway + DHCP + gateway_network to TF (was manually created, now codified with push_default_route=true) - Added scripts/safe-node-replace.sh — one-at-a-time with DNS verification - Added dns-bootstrap-policy.yaml — incident documentation + recovery procedure - Bastion enabled (port 61000) for emergency SSH access Prevention: NEVER replace all nodes at once. Use safe-node-replace.sh. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
26 lines
1.3 KiB
YAML
26 lines
1.3 KiB
YAML
# DNS Bootstrap Prevention — Prevent CoreDNS chicken-and-egg deadlock
|
|
#
|
|
# ROOT CAUSE (2026-03-18 incident):
|
|
# 1. Rolling replacement of ALL platform nodes simultaneously
|
|
# 2. New nodes boot with /etc/resolv.conf → 10.32.0.10 (kube-dns only)
|
|
# 3. CoreDNS pods need to pull images → need DNS → need CoreDNS
|
|
# 4. Cilium BPF socket LB returns EPERM when kube-dns has no endpoints
|
|
# 5. Complete cluster DNS deadlock — nothing can pull images
|
|
#
|
|
# PREVENTION:
|
|
# 1. VPC DefaultRoutePropagation MUST be enabled (API, not Terraform yet)
|
|
# 2. NEVER replace ALL nodes at once — one at a time, wait for CoreDNS
|
|
# 3. Do NOT create CiliumClusterwideNetworkPolicy with egress rules
|
|
# (enables default deny, makes the problem worse)
|
|
# 4. bpf-lb-sock MUST remain "true" — disabling breaks ALL service routing
|
|
# 5. Keep bastion enabled on VPC gateway for emergency SSH access
|
|
#
|
|
# RECOVERY PROCEDURE:
|
|
# 1. SSH via bastion: ssh -o ProxyCommand="ssh -W %h:%p -p 61000 bastion@<gw-public-ip>" root@<node-private-ip>
|
|
# 2. On the node: /usr/local/bin/crictl pull <coredns-image>
|
|
# 3. kubectl delete pods -l k8s-app=kube-dns --force
|
|
# 4. CoreDNS starts from cached image → DNS recovers → cascade
|
|
#
|
|
# DATA PROTECTION:
|
|
# All PVs set to Retain reclaim policy — data survives cluster deletion
|