diff --git a/docs/superpowers/plans/CONSOLIDATION-PROGRESS.md b/docs/superpowers/plans/CONSOLIDATION-PROGRESS.md index e97100e..259fcc0 100644 --- a/docs/superpowers/plans/CONSOLIDATION-PROGRESS.md +++ b/docs/superpowers/plans/CONSOLIDATION-PROGRESS.md @@ -75,10 +75,19 @@ Rollback until T11 = re-point DNS + un-suspend foxhunt. ## T8 PREP (2026-07-18) — DNS/ingress map + fixes (foxhunt still fully live) ### Gotchas hit this session -- **dagster regressed to 2/3** (tailscale sidecar CrashLoop): the dagster NP's public-HTTPS egress rule - `except: [10.32.0.0/16,...]` blocks the bizworx apiserver ClusterIP `10.32.0.1` → sidecar `context deadline - exceeded` reading its ts-state Secret. FIXED (06d12fd): added surgical `ipBlock 10.32.0.1/32:443` egress. - daemon+webserver were always healthy (sidecar-only). Now 3/3. (Cockpit's sidecar works because cockpit has NO NP.) +- **dagster tailscale sidecar CrashLoop (context deadline exceeded reaching kubernetes.default.svc).** TWO-STAGE + root cause — the FIRST fix (06d12fd, `10.32.0.1/32:443`) was WRONG and did not hold (regressed to 2/3): + * `kubernetes.default.svc` → ClusterIP `10.32.0.1:443`, BUT **kube-proxy DNATs it to the REAL apiserver + endpoint `172.16.0.11:6443` BEFORE the NetworkPolicy is evaluated.** The NP sees dest `172.16.0.11:6443`, + which is inside the NP's OTHER except entry `172.16.0.0/16` → blocked. So allowing the ClusterIP never matched. + * CORRECT FIX: `ipBlock 172.16.0.11/32` port **6443** (the real endpoint, post-DNAT). apiserver endpoint is + `kubectl -n default get endpoints kubernetes` = 172.16.0.11 (Kapsule fixed control-plane). + * SIDE EFFECT this exposed: when the tailscale sidecar is NotReady, the whole dagster pod is NotReady → + **dagster Service has NO endpoints → dagster.fxhnt.ai vhost would 502**. So this MUST be green before C4 DNS flip. + * daemon+webserver themselves were always healthy. Cockpit's sidecar works only because cockpit has NO NP. + * GENERAL LESSON for any pod with a tailscale sidecar + a restrictive egress NP on bizworx: must allow + egress to 172.16.0.11:6443 (apiserver). The cronjob/rebalancer NPs do NOT have tailscale sidecars so they're + unaffected — but if any future fund pod gets a TS sidecar under an NP, add this rule. - **foxhunt API ACL lapsed AGAIN**: devbox egress IP drifted `51.158.111.225`→`51.158.125.1`. Foxhunt Kapsule admin API (`*.api.k8s.fr-par.scw.cloud:6443`) is IP-allowlisted; bizworx keeps working (tailscale-cgnat range). FIX: `scw k8s acl add cluster-id=34a1e3c4… acls.0.ip=/32 acls.0.description=bizworx-devbox-vN` (APPENDS; diff --git a/infra/k8s/orchestration/dagster.yaml b/infra/k8s/orchestration/dagster.yaml index 5514740..ee7ae3d 100644 --- a/infra/k8s/orchestration/dagster.yaml +++ b/infra/k8s/orchestration/dagster.yaml @@ -196,7 +196,7 @@ spec: podSelector: { matchLabels: { app.kubernetes.io/name: dagster } } policyTypes: [Ingress, Egress] ingress: - - from: [{ podSelector: { matchLabels: { app.kubernetes.io/name: tailscale-gitlab-proxy } } }] # dagster.fxhnt.ai + - from: [{ podSelector: { matchLabels: { app.kubernetes.io/name: fxhnt-ingress } } }] # dagster.fxhnt.ai via the bizworx fxhnt-ingress proxy (was tailscale-gitlab-proxy on foxhunt) ports: [{ port: 3000, protocol: TCP }] egress: - to: [{ podSelector: { matchLabels: { app.kubernetes.io/name: postgres } } }] @@ -204,7 +204,7 @@ spec: - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: gitea } }, podSelector: { matchLabels: { app.kubernetes.io/name: gitea } } }] # git-sync clones from the bizworx gitea in ns gitea (cross-namespace) ports: [{ port: 2222, protocol: TCP }, { port: 22, protocol: TCP }] - ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }] - - to: [{ ipBlock: { cidr: 10.32.0.1/32 } }] # kube-apiserver ClusterIP: the tailscale sidecar reads/writes its ts-state Secret via kubernetes.default.svc (the public-HTTPS rule below excepts the service CIDR) - ports: [{ port: 443, protocol: TCP }] + - to: [{ ipBlock: { cidr: 172.16.0.11/32 } }] # kube-apiserver REAL endpoint (kube-proxy DNATs kubernetes.default.svc 10.32.0.1:443 -> this before NP eval, and 172.16.0.0/16 is excepted below). The tailscale sidecar reads/writes its ts-state Secret via the apiserver. + ports: [{ port: 6443, protocol: TCP }] - to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [10.32.0.0/16, 172.16.0.0/16] } }] # Binance/Databento + tailscale ports: [{ port: 443, protocol: TCP }, { port: 3478, protocol: UDP }, { port: 41641, protocol: UDP }]