fix(migrate): dagster NP apiserver egress (real endpoint) + ingress from fxhnt-ingress

Two corrections found while prepping P2:
1. The earlier 10.32.0.1/32 apiserver-egress fix was WRONG: kube-proxy
   DNATs kubernetes.default.svc (10.32.0.1:443) to the REAL apiserver
   endpoint 172.16.0.11:6443 BEFORE NetworkPolicy eval, and 172.16.0.0/16
   is in the public-HTTPS except list -> still blocked -> tailscale sidecar
   crashlooped again -> pod NotReady -> dagster Service had NO endpoints.
   Correct fix: allow egress to 172.16.0.11:6443 (post-DNAT dest). Now
   3/3 stable, endpoints populate. (apiserver endpoint from
   NAME         ENDPOINTS          AGE
kubernetes   172.16.0.11:6443   85d.)
2. dagster NP ingress referenced foxhunt-only label tailscale-gitlab-proxy;
   retargeted to app.kubernetes.io/name=fxhnt-ingress (the bizworx proxy
   built in P2) so dagster.fxhnt.ai can be fronted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-18 19:22:08 +00:00
parent 0d3fca6b0c
commit 9396f53669
2 changed files with 16 additions and 7 deletions

View File

@@ -75,10 +75,19 @@ Rollback until T11 = re-point DNS + un-suspend foxhunt.
## T8 PREP (2026-07-18) — DNS/ingress map + fixes (foxhunt still fully live)
### Gotchas hit this session
- **dagster regressed to 2/3** (tailscale sidecar CrashLoop): the dagster NP's public-HTTPS egress rule
`except: [10.32.0.0/16,...]` blocks the bizworx apiserver ClusterIP `10.32.0.1` → sidecar `context deadline
exceeded` reading its ts-state Secret. FIXED (06d12fd): added surgical `ipBlock 10.32.0.1/32:443` egress.
daemon+webserver were always healthy (sidecar-only). Now 3/3. (Cockpit's sidecar works because cockpit has NO NP.)
- **dagster tailscale sidecar CrashLoop (context deadline exceeded reaching kubernetes.default.svc).** TWO-STAGE
root cause — the FIRST fix (06d12fd, `10.32.0.1/32:443`) was WRONG and did not hold (regressed to 2/3):
* `kubernetes.default.svc` → ClusterIP `10.32.0.1:443`, BUT **kube-proxy DNATs it to the REAL apiserver
endpoint `172.16.0.11:6443` BEFORE the NetworkPolicy is evaluated.** The NP sees dest `172.16.0.11:6443`,
which is inside the NP's OTHER except entry `172.16.0.0/16` → blocked. So allowing the ClusterIP never matched.
* CORRECT FIX: `ipBlock 172.16.0.11/32` port **6443** (the real endpoint, post-DNAT). apiserver endpoint is
`kubectl -n default get endpoints kubernetes` = 172.16.0.11 (Kapsule fixed control-plane).
* SIDE EFFECT this exposed: when the tailscale sidecar is NotReady, the whole dagster pod is NotReady →
**dagster Service has NO endpoints → dagster.fxhnt.ai vhost would 502**. So this MUST be green before C4 DNS flip.
* daemon+webserver themselves were always healthy. Cockpit's sidecar works only because cockpit has NO NP.
* GENERAL LESSON for any pod with a tailscale sidecar + a restrictive egress NP on bizworx: must allow
egress to 172.16.0.11:6443 (apiserver). The cronjob/rebalancer NPs do NOT have tailscale sidecars so they're
unaffected — but if any future fund pod gets a TS sidecar under an NP, add this rule.
- **foxhunt API ACL lapsed AGAIN**: devbox egress IP drifted `51.158.111.225``51.158.125.1`. Foxhunt Kapsule
admin API (`*.api.k8s.fr-par.scw.cloud:6443`) is IP-allowlisted; bizworx keeps working (tailscale-cgnat range).
FIX: `scw k8s acl add cluster-id=34a1e3c4… acls.0.ip=<newIP>/32 acls.0.description=bizworx-devbox-vN` (APPENDS;

View File

@@ -196,7 +196,7 @@ spec:
podSelector: { matchLabels: { app.kubernetes.io/name: dagster } }
policyTypes: [Ingress, Egress]
ingress:
- from: [{ podSelector: { matchLabels: { app.kubernetes.io/name: tailscale-gitlab-proxy } } }] # dagster.fxhnt.ai
- from: [{ podSelector: { matchLabels: { app.kubernetes.io/name: fxhnt-ingress } } }] # dagster.fxhnt.ai via the bizworx fxhnt-ingress proxy (was tailscale-gitlab-proxy on foxhunt)
ports: [{ port: 3000, protocol: TCP }]
egress:
- to: [{ podSelector: { matchLabels: { app.kubernetes.io/name: postgres } } }]
@@ -204,7 +204,7 @@ spec:
- to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: gitea } }, podSelector: { matchLabels: { app.kubernetes.io/name: gitea } } }] # git-sync clones from the bizworx gitea in ns gitea (cross-namespace)
ports: [{ port: 2222, protocol: TCP }, { port: 22, protocol: TCP }]
- ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]
- to: [{ ipBlock: { cidr: 10.32.0.1/32 } }] # kube-apiserver ClusterIP: the tailscale sidecar reads/writes its ts-state Secret via kubernetes.default.svc (the public-HTTPS rule below excepts the service CIDR)
ports: [{ port: 443, protocol: TCP }]
- to: [{ ipBlock: { cidr: 172.16.0.11/32 } }] # kube-apiserver REAL endpoint (kube-proxy DNATs kubernetes.default.svc 10.32.0.1:443 -> this before NP eval, and 172.16.0.0/16 is excepted below). The tailscale sidecar reads/writes its ts-state Secret via the apiserver.
ports: [{ port: 6443, protocol: TCP }]
- to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [10.32.0.0/16, 172.16.0.0/16] } }] # Binance/Databento + tailscale
ports: [{ port: 443, protocol: TCP }, { port: 3478, protocol: UDP }, { port: 41641, protocol: UDP }]