# T8 — Cutover Runbook (foxhunt → bizworx)

> Executable, verify-first, reversible. Expands Task 8 of the consolidation plan with the exact values
> discovered in T8-prep. **foxhunt stays intact (suspended, NOT deleted) — rollback = re-point DNS +
> un-suspend foxhunt.** Fund is ~1×/day; downtime OK. Run in a quiet window.

## Contexts / constants (verified this session)
```
FOX=foxhunt-34a1e3c4-ac35-48c8-ab49-5f6ec4df32c1     # foxhunt Kapsule (retiring)
BIZ=bizworx-prod-kapsule-05e1dae3-8f35-4ad4-8fce-0f9a7691a760   # survivor
```
- **foxhunt API is IP-allowlisted.** If `6443` resets: `curl -s ifconfig.me` → compare to
  `scw k8s acl list cluster-id=34a1e3c4-…`; if drifted, `scw k8s acl add cluster-id=34a1e3c4-…
  acls.0.ip=<newIP>/32 acls.0.description=bizworx-devbox-vN` (APPENDS — verify all entries survive).
  bizworx never hits this (tailscale-cgnat range).
- **DNS zone:** Scaleway `fxhnt.ai` (project c293eb98). Records set with `scw dns record set dns-zone=fxhnt.ai ...`.

## The single most important safety fact
The **fund pipeline never uses `fxhnt.ai`** — git-sync = `gitea-ssh.gitea.svc.cluster.local:22` (in-cluster),
cockpit/dagster serve over ClusterIP + their own tailnet IP. `fxhnt.ai` is **browser/human access only**.
→ The fund keeps running through the DNS flip; the DNS cutover only affects the operator's browser access.

## Current fxhnt.ai ingress (all → 100.95.225.27 = `tailscale-gitlab-proxy` pod, ns foxhunt, DIES at teardown)
| host | fronts | cutover |
|---|---|---|
| git.fxhnt.ai | gitea-web.foxhunt:3000 | → **bizworx gitea** (ns gitea, mirrored) |
| dashboard/cockpit.fxhnt.ai | fxhnt-dashboard node → cockpit:8080 | → **bizworx cockpit** |
| dagster.fxhnt.ai | dagster.foxhunt:80 | → **bizworx dagster** |
| grafana / api / mail / chat / minio | legacy | **DROP** (operator confirmed grafana can go too) |

Only **git + dashboard/cockpit + dagster** move. Everything else is dropped at Phase-2/teardown.

## TLS: solved-path (no blocker)
Old proxy served a cert-manager wildcard `*.fxhnt.ai` (ClusterIssuer `letsencrypt-prod`, Scaleway DNS-01).
**bizworx already has the identical stack** — cert-manager + `scaleway-certmanager-webhook` (1/1) + `scaleway-api`
creds + `letsencrypt-prod` ClusterIssuer, already issuing real LE certs in ~10 ns. → Issuing `*.fxhnt.ai` on
bizworx is just a `Certificate` resource.

---

## PRE-CUTOVER (do these BEFORE the window; all non-destructive, foxhunt untouched)

### P1. Issue the `*.fxhnt.ai` wildcard cert on bizworx (ns foxhunt) — needs the DNS-01 solver, so do it early
- [ ] Create `infra/k8s/services/fxhnt-wildcard-cert.yaml`: a cert-manager `Certificate` (dnsNames
      `fxhnt.ai`,`*.fxhnt.ai`; issuerRef ClusterIssuer `letsencrypt-prod`; secretName `fxhnt-wildcard-tls`;
      namespace `foxhunt`). Apply on `$BIZ`.
- [ ] **VERIFY:** `kubectl --context $BIZ -n foxhunt get certificate fxhnt-wildcard -o wide` → READY=True.
      (DNS-01 provisions a TXT on the fxhnt.ai zone; the same zone we control. Wait until True — can take minutes.)
- **EVAL P1:** wildcard cert READY on bizworx. foxhunt untouched. If it fails → debug the webhook, do NOT proceed.

### P2. Decide + build the fxhnt.ai front on bizworx (the thing DNS will point at)
Two acceptable shapes — **default = a stripped fund-only nginx proxy** mirroring the foxhunt one (faithful,
single tailnet node for DNS, wildcard TLS termination):
- [ ] Create `infra/k8s/services/fxhnt-ingress-proxy.yaml` on `$BIZ` ns foxhunt: nginx + tailscale sidecar
      (`TS_HOSTNAME=fxhnt-ingress`), wildcard `fxhnt-wildcard-tls` mounted at `/etc/nginx/certs`, vhosts:
      - `git.fxhnt.ai` → `http://gitea-http.gitea.svc.cluster.local:3000` (CONFIRMED svc; headless/ClusterIP:None
        → use nginx `resolver` + variable upstream, or proxy_pass by name works via cluster DNS round-robin)
      - `dashboard.fxhnt.ai cockpit.fxhnt.ai` → `http://fxhnt-dashboard.foxhunt.svc.cluster.local:80` (CONFIRMED)
      - `dagster.fxhnt.ai` → `http://dagster.foxhunt.svc.cluster.local:80` (CONFIRMED)
      + socat sidecar for `git.fxhnt.ai:22` SSH → `gitea-ssh.gitea.svc.cluster.local:22` (CONFIRMED; human git over SSH).
- [ ] **VERIFY** (still pre-flip, over tailnet directly to the proxy's tailnet IP, Host header spoofed):
      `curl -k --resolve dashboard.fxhnt.ai:443:<proxy-tailnet-ip> https://dashboard.fxhnt.ai/ -I` → 200,
      and cert CN/SAN covers `*.fxhnt.ai`. Repeat for git + dagster.
- **EVAL P2:** all 3 fund vhosts serve 200 with a valid `*.fxhnt.ai` cert via the bizworx proxy — WITHOUT any
      DNS change yet. Record the proxy's tailnet IP for step C4. If any vhost fails → fix before the window.
- NOTE: this proxy points at the in-cluster **Services** (`fxhnt-dashboard.foxhunt`, `dagster.foxhunt`), so the
      "reclaim tailnet identity" rename of the pods is **not required for browser access** — the proxy uses
      ClusterIP, not the pod's tailnet name. (Identity rename becomes optional cosmetic; keep it out of the
      critical path.)

---

## CUTOVER WINDOW (the only steps that touch live foxhunt — each is reversible)

### C1. Freeze foxhunt writers (make its Postgres quiescent)
- [ ] Suspend every foxhunt fund CronJob:
      `for cj in $(kubectl --context $FOX -n foxhunt get cronjobs -o name); do kubectl --context $FOX -n foxhunt patch $cj -p '{"spec":{"suspend":true}}'; done`
- [ ] Scale down foxhunt writers so nothing writes to its DB during the delta dump:
      `kubectl --context $FOX -n foxhunt scale deploy/dagster deploy/fxhnt-dashboard --replicas=0`
      (cockpit is read-only but scale it too to release the `fxhnt-dashboard` tailnet name; dagster daemon writes.)
- [ ] Scale foxhunt `ib-gateway` to 0 → **releases the sole IBKR session** so bizworx ib-gateway can hold it:
      `kubectl --context $FOX -n foxhunt scale deploy/ib-gateway --replicas=0`
- **VERIFY C1:** `kubectl --context $FOX -n foxhunt get deploy` all fund writers at 0/0; cronjobs SUSPEND=true.
- **ROLLBACK:** scale back up + un-suspend — foxhunt fully restored.

### C2. Final delta Postgres dump/restore (frozen source → bizworx)
Re-run the proven Task-6 procedure on the now-frozen `fxhnt` DB (version-pinned tsdb 2.25.1 both sides).
- [ ] `pg_dump -Fc` the `fxhnt` DB from foxhunt postgres (port-forward or exec), `pg_restore --no-owner
      --no-acl --clean --if-exists` into the bizworx `fxhnt` DB (which currently holds the rehearsal copy →
      `--clean` refreshes it), wrapped in `timescaledb_pre_restore()` / `post_restore()`.
- [ ] **VERIFY C2:** re-check the canonical row counts on bizworx match the (now-frozen) foxhunt source:
      `forward_nav, backtest_summary, bybit_sleeve_ret, xsp_option_bars, exec_fills, ibkr_account_nav`
      + `SELECT max(t0) FROM forward_anchor` matches. 37 tables / 3 hypertables.
- **EVAL C2:** every checked count matches the frozen source. If not → STOP, do NOT flip DNS (rollback = C1 up).

### C3. Bring the bizworx fund fully live (still no DNS)
- [ ] `kubectl --context $BIZ -n foxhunt scale deploy/dagster deploy/fxhnt-dashboard deploy/ib-gateway --replicas=1`
      (they were already deployed; ensure 1 replica). ib-gateway should now stay 1/1 (sole IBKR session).
- [ ] **VERIFY C3:** on bizworx — postgres 1/1, cockpit 2/2, dagster 3/3, **ib-gateway 1/1 stable** (no more
      crashloop now that foxhunt released the session). Confirm ib-gateway API port + one **whatIf** dry-run.
- **EVAL C3:** all bizworx fund pods healthy incl. ib-gateway stable. If ib-gateway still crashes → confirm
      foxhunt ib-gateway is really at 0; do NOT arm any rebalancer.

### C4. Flip DNS (Scaleway fxhnt.ai zone → bizworx proxy tailnet IP)
**TARGET IP (verified in EVAL P2): `100.93.141.11` (tailnet node `fxhnt-ingress`).**
- [ ] `scw dns record set dns-zone=fxhnt.ai name=dashboard type=A data=100.93.141.11 ttl=60`
- [ ] same for `name=cockpit`, `name=dagster`, `name=git`. (TTL was 300; lower to 60 during cutover.)
      (Records currently point at 100.95.225.27 = the dying foxhunt tailscale-gitlab-proxy.)
- [ ] Leave grafana/api/mail/chat/minio records as-is for now (dropped at Phase-2; harmless dangling until then).
- **VERIFY C4:** from an external network (phone off-wifi is fine): `dig +short dashboard.fxhnt.ai` = new IP;
      open `https://dashboard.fxhnt.ai` in a browser → cockpit renders migrated data, **valid TLS** (LE `*.fxhnt.ai`).
      `https://git.fxhnt.ai` serves the bizworx gitea; `git ls-remote ssh://git@git.fxhnt.ai/...` works.
- **ROLLBACK:** `scw dns record set ... data=100.95.225.27` (back to the still-live foxhunt proxy) + C1-up +
      un-suspend foxhunt cronjobs. Fully reversible until Phase-2 teardown.

### C5. Arm the fund on bizworx (start conservative)
- [ ] Un-suspend ONLY the UCITS rebalancer on bizworx (already the armed one; US + bybit stay suspended):
      `kubectl --context $BIZ -n foxhunt patch cronjob fxhnt-ucits-rebalancer -p '{"spec":{"suspend":false}}'`
- [ ] **VERIFY C5:** its next scheduled run (09:00 UTC EU-hours) succeeds — a real fill or a clean gate WAIT,
      not a crash. Watch the Job pod: git-sync OK, DB reachable, ib-gateway reachable, recon-gate evaluated.
- **EVAL C5:** first armed run green. If it errors → re-suspend, investigate; foxhunt still intact for rollback.

---

## POST-CUTOVER = Task 9 (verify) → Task 10 (observation window) → Task 11 (triage) → Task 12 (teardown)
- **Rotate the pasted tailscale auth-key** (was in transcript) once the bizworx pods are settled.
- **DELETE NOTHING** on foxhunt until T10 observation passes. foxhunt = suspended-but-intact through T12.
