fxhnt/docs/superpowers/plans/T8-CUTOVER-RUNBOOK.md

T8 — Cutover Runbook (foxhunt → bizworx)

Executable, verify-first, reversible. Expands Task 8 of the consolidation plan with the exact values discovered in T8-prep. foxhunt stays intact (suspended, NOT deleted) — rollback = re-point DNS + un-suspend foxhunt. Fund is ~1×/day; downtime OK. Run in a quiet window.

Contexts / constants (verified this session)

FOX=foxhunt-34a1e3c4-ac35-48c8-ab49-5f6ec4df32c1     # foxhunt Kapsule (retiring)
BIZ=bizworx-prod-kapsule-05e1dae3-8f35-4ad4-8fce-0f9a7691a760   # survivor

• foxhunt API is IP-allowlisted. If 6443 resets: curl -s ifconfig.me → compare to scw k8s acl list cluster-id=34a1e3c4-…; if drifted, scw k8s acl add cluster-id=34a1e3c4-… acls.0.ip=<newIP>/32 acls.0.description=bizworx-devbox-vN (APPENDS — verify all entries survive). bizworx never hits this (tailscale-cgnat range).
• DNS zone: Scaleway fxhnt.ai (project c293eb98). Records set with scw dns record set dns-zone=fxhnt.ai ....

The single most important safety fact

The fund pipeline never uses fxhnt.ai — git-sync = gitea-ssh.gitea.svc.cluster.local:22 (in-cluster), cockpit/dagster serve over ClusterIP + their own tailnet IP. fxhnt.ai is browser/human access only. → The fund keeps running through the DNS flip; the DNS cutover only affects the operator's browser access.

Current fxhnt.ai ingress (all → 100.95.225.27 = tailscale-gitlab-proxy pod, ns foxhunt, DIES at teardown)

host	fronts	cutover
git.fxhnt.ai	gitea-web.foxhunt:3000	→ bizworx gitea (ns gitea, mirrored)
dashboard/cockpit.fxhnt.ai	fxhnt-dashboard node → cockpit:8080	→ bizworx cockpit
dagster.fxhnt.ai	dagster.foxhunt:80	→ bizworx dagster
grafana / api / mail / chat / minio	legacy	DROP (operator confirmed grafana can go too)

Only git + dashboard/cockpit + dagster move. Everything else is dropped at Phase-2/teardown.

TLS: solved-path (no blocker)

Old proxy served a cert-manager wildcard *.fxhnt.ai (ClusterIssuer letsencrypt-prod, Scaleway DNS-01). bizworx already has the identical stack — cert-manager + scaleway-certmanager-webhook (1/1) + scaleway-api creds + letsencrypt-prod ClusterIssuer, already issuing real LE certs in ~10 ns. → Issuing *.fxhnt.ai on bizworx is just a Certificate resource.

---

PRE-CUTOVER (do these BEFORE the window; all non-destructive, foxhunt untouched)

P1. Issue the *.fxhnt.ai wildcard cert on bizworx (ns foxhunt) — needs the DNS-01 solver, so do it early

• Create infra/k8s/services/fxhnt-wildcard-cert.yaml: a cert-manager Certificate (dnsNames fxhnt.ai,*.fxhnt.ai; issuerRef ClusterIssuer letsencrypt-prod; secretName fxhnt-wildcard-tls; namespace foxhunt). Apply on $BIZ.
• VERIFY: kubectl --context $BIZ -n foxhunt get certificate fxhnt-wildcard -o wide → READY=True. (DNS-01 provisions a TXT on the fxhnt.ai zone; the same zone we control. Wait until True — can take minutes.)
• EVAL P1: wildcard cert READY on bizworx. foxhunt untouched. If it fails → debug the webhook, do NOT proceed.

P2. Decide + build the fxhnt.ai front on bizworx (the thing DNS will point at)

Two acceptable shapes — default = a stripped fund-only nginx proxy mirroring the foxhunt one (faithful, single tailnet node for DNS, wildcard TLS termination):

• Create infra/k8s/services/fxhnt-ingress-proxy.yaml on $BIZ ns foxhunt: nginx + tailscale sidecar (TS_HOSTNAME=fxhnt-ingress), wildcard fxhnt-wildcard-tls mounted at /etc/nginx/certs, vhosts: - git.fxhnt.ai → http://gitea-http.gitea.svc.cluster.local:3000 (CONFIRMED svc; headless/ClusterIP:None → use nginx resolver + variable upstream, or proxy_pass by name works via cluster DNS round-robin) - dashboard.fxhnt.ai cockpit.fxhnt.ai → http://fxhnt-dashboard.foxhunt.svc.cluster.local:80 (CONFIRMED) - dagster.fxhnt.ai → http://dagster.foxhunt.svc.cluster.local:80 (CONFIRMED) + socat sidecar for git.fxhnt.ai:22 SSH → gitea-ssh.gitea.svc.cluster.local:22 (CONFIRMED; human git over SSH).
• VERIFY (still pre-flip, over tailnet directly to the proxy's tailnet IP, Host header spoofed): curl -k --resolve dashboard.fxhnt.ai:443:<proxy-tailnet-ip> https://dashboard.fxhnt.ai/ -I → 200, and cert CN/SAN covers *.fxhnt.ai. Repeat for git + dagster.
• EVAL P2: all 3 fund vhosts serve 200 with a valid *.fxhnt.ai cert via the bizworx proxy — WITHOUT any DNS change yet. Record the proxy's tailnet IP for step C4. If any vhost fails → fix before the window.
• NOTE: this proxy points at the in-cluster Services (fxhnt-dashboard.foxhunt, dagster.foxhunt), so the "reclaim tailnet identity" rename of the pods is not required for browser access — the proxy uses ClusterIP, not the pod's tailnet name. (Identity rename becomes optional cosmetic; keep it out of the critical path.)

---

CUTOVER WINDOW (the only steps that touch live foxhunt — each is reversible)

C1. Freeze foxhunt writers (make its Postgres quiescent)

• Suspend every foxhunt fund CronJob: for cj in $(kubectl --context $FOX -n foxhunt get cronjobs -o name); do kubectl --context $FOX -n foxhunt patch $cj -p '{"spec":{"suspend":true}}'; done
• Scale down foxhunt writers so nothing writes to its DB during the delta dump: kubectl --context $FOX -n foxhunt scale deploy/dagster deploy/fxhnt-dashboard --replicas=0 (cockpit is read-only but scale it too to release the fxhnt-dashboard tailnet name; dagster daemon writes.)
• Scale foxhunt ib-gateway to 0 → releases the sole IBKR session so bizworx ib-gateway can hold it: kubectl --context $FOX -n foxhunt scale deploy/ib-gateway --replicas=0
• VERIFY C1: kubectl --context $FOX -n foxhunt get deploy all fund writers at 0/0; cronjobs SUSPEND=true.
• ROLLBACK: scale back up + un-suspend — foxhunt fully restored.

C2. Final delta Postgres dump/restore (frozen source → bizworx)

Re-run the proven Task-6 procedure on the now-frozen fxhnt DB (version-pinned tsdb 2.25.1 both sides).

• pg_dump -Fc the fxhnt DB from foxhunt postgres (port-forward or exec), pg_restore --no-owner --no-acl --clean --if-exists into the bizworx fxhnt DB (which currently holds the rehearsal copy → --clean refreshes it), wrapped in timescaledb_pre_restore() / post_restore().
• VERIFY C2: re-check the canonical row counts on bizworx match the (now-frozen) foxhunt source: forward_nav, backtest_summary, bybit_sleeve_ret, xsp_option_bars, exec_fills, ibkr_account_nav + SELECT max(t0) FROM forward_anchor matches. 37 tables / 3 hypertables.
• EVAL C2: every checked count matches the frozen source. If not → STOP, do NOT flip DNS (rollback = C1 up).

C3. Bring the bizworx fund fully live (still no DNS)

• kubectl --context $BIZ -n foxhunt scale deploy/dagster deploy/fxhnt-dashboard deploy/ib-gateway --replicas=1 (they were already deployed; ensure 1 replica). ib-gateway should now stay 1/1 (sole IBKR session).
• VERIFY C3: on bizworx — postgres 1/1, cockpit 2/2, dagster 3/3, ib-gateway 1/1 stable (no more crashloop now that foxhunt released the session). Confirm ib-gateway API port + one whatIf dry-run.
• EVAL C3: all bizworx fund pods healthy incl. ib-gateway stable. If ib-gateway still crashes → confirm foxhunt ib-gateway is really at 0; do NOT arm any rebalancer.

C4. Flip DNS (Scaleway fxhnt.ai zone → bizworx proxy tailnet IP)

TARGET IP (verified in EVAL P2): 100.93.141.11 (tailnet node fxhnt-ingress).

• scw dns record set dns-zone=fxhnt.ai name=dashboard type=A data=100.93.141.11 ttl=60
• same for name=cockpit, name=dagster, name=git. (TTL was 300; lower to 60 during cutover.) (Records currently point at 100.95.225.27 = the dying foxhunt tailscale-gitlab-proxy.)
• Leave grafana/api/mail/chat/minio records as-is for now (dropped at Phase-2; harmless dangling until then).
• VERIFY C4: from an external network (phone off-wifi is fine): dig +short dashboard.fxhnt.ai = new IP; open https://dashboard.fxhnt.ai in a browser → cockpit renders migrated data, valid TLS (LE *.fxhnt.ai). https://git.fxhnt.ai serves the bizworx gitea; git ls-remote ssh://git@git.fxhnt.ai/... works.
• ROLLBACK: scw dns record set ... data=100.95.225.27 (back to the still-live foxhunt proxy) + C1-up + un-suspend foxhunt cronjobs. Fully reversible until Phase-2 teardown.

C5. Arm the fund on bizworx (start conservative)

• Un-suspend ONLY the UCITS rebalancer on bizworx (already the armed one; US + bybit stay suspended): kubectl --context $BIZ -n foxhunt patch cronjob fxhnt-ucits-rebalancer -p '{"spec":{"suspend":false}}'
• VERIFY C5: its next scheduled run (09:00 UTC EU-hours) succeeds — a real fill or a clean gate WAIT, not a crash. Watch the Job pod: git-sync OK, DB reachable, ib-gateway reachable, recon-gate evaluated.
• EVAL C5: first armed run green. If it errors → re-suspend, investigate; foxhunt still intact for rollback.

---

POST-CUTOVER = Task 9 (verify) → Task 10 (observation window) → Task 11 (triage) → Task 12 (teardown)

• Rotate the pasted tailscale auth-key (was in transcript) once the bizworx pods are settled.
• DELETE NOTHING on foxhunt until T10 observation passes. foxhunt = suspended-but-intact through T12.