# Foxhunt → Bizworx Consolidation — Implementation Plan

> **For the operator/agent:** execute **inline, with a checkpoint after every task** (this is live infra +
> real-money fund + secrets — NOT a subagent-driven code plan). Steps use `- [ ]` for tracking. Spec:
> `docs/superpowers/specs/2026-07-18-foxhunt-to-bizworx-consolidation-design.md`.

**Goal:** Move the fxhnt fund onto the bizworx Kapsule + consolidate to one gitea, verify it fully, THEN
retire the foxhunt Kapsule (cost saving).

**Architecture:** Build the fund on bizworx **alongside** the still-live foxhunt cluster, migrate the one
stateful thing (Postgres/Timescale), prove it works in parallel, cut over DNS, observe, and only then tear
down. Reuse bizworx's existing argo + observability + gitea.

**Tech stack:** Scaleway Kapsule (2 clusters, 2 projects, same org), kubectl (contexts `bizworx-prod-…` +
`foxhunt-…`), `scw` CLI, Terragrunt/Terraform (`infra/live/production`), argo (git-sync deploy), gitea,
TimescaleDB/Postgres, IBKR ib-gateway, tailscale.

## Global Constraints (apply to EVERY task)

- **VERIFY-FIRST, NOTHING DESTRUCTIVE UNTIL PROVEN.** No delete / teardown / `terraform destroy` / PVC
  removal / foxhunt scale-down happens before Phase 1 is fully verified AND the observation window (Task 10)
  passes. Tasks 1–9 are strictly additive; foxhunt stays fully live through Task 8, suspended-but-intact
  through teardown.
- **EVALUATE PER STEP.** Every task ends with an explicit `EVAL:` gate. If it fails → STOP, do not proceed.
- **ROLLBACK is always available** until Task 12: re-point DNS + un-suspend foxhunt CronJobs (foxhunt's
  Postgres still holds pre-cutover state).
- **Secrets NEVER through the transcript.** Create them with `! kubectl create secret …` / openbao /
  external-secrets. Never echo/commit secret values.
- **Real-money fund:** before arming any rebalancer on bizworx, a dry-run + recon-gate check must pass.
- **Two contexts, always explicit:** `FOX=foxhunt-34a1e3c4-ac35-48c8-ab49-5f6ec4df32c1`,
  `BIZ=bizworx-prod-kapsule-05e1dae3-8f35-4ad4-8fce-0f9a7691a760`. Never rely on current-context.
- Cockpit changes verified LOCALLY in a real browser + on prod, reading the numbers.
- Per-item operator sign-off before deleting any Phase-2 data.

---

## PHASE 1 — build + verify on bizworx (foxhunt stays live) — additive only

### Task 1: Prerequisites + target namespace

**Files:** none (cluster ops).
- [ ] Confirm both kube-contexts respond: `kubectl --context $BIZ get ns` and `kubectl --context $FOX get ns`.
- [ ] Pick + confirm a StorageClass on bizworx for the fund (foxhunt used `sbs-default`/`scw-bssd`; bizworx has
      `sbs-5k`,`sbs-default`): `kubectl --context $BIZ get storageclass`.
- [ ] Create the target namespace: `kubectl --context $BIZ create ns foxhunt`.
- [ ] Record the foxhunt Postgres + TimescaleDB versions to match on bizworx:
      `kubectl --context $FOX exec -n foxhunt deploy/postgres -- psql -U postgres -tAc "select version(); select extversion from pg_extension where extname='timescaledb';"`

**EVAL:** both contexts work, ns `foxhunt` exists on bizworx, TimescaleDB version noted. **No foxhunt change.**

### Task 2: Secrets on bizworx

**Files:** none (secrets via `!`/openbao — never committed).
- [ ] List the secrets the fund workloads consume on foxhunt (names only, NOT values):
      `kubectl --context $FOX get secrets -n foxhunt` → expect `db-credentials`, `databento-credentials`, an
      IBKR/ib-gateway secret, `scw-registry`, `argo-git-ssh-key`/`git-ssh-key`, tailscale, cockpit TLS.
- [ ] For each: recreate on bizworx sourced from openbao/external-secrets, or by hand with
      `! kubectl --context $BIZ create secret … -n foxhunt` (operator runs; values never in transcript).
- [ ] Verify presence: `kubectl --context $BIZ get secrets -n foxhunt` shows every required name.

**EVAL:** every fund secret exists on bizworx (by name). A missing secret wedges a workload later — do not
proceed until complete. **No foxhunt change.**

### Task 3: Consolidate gitea (mirror repos; DO NOT re-point DNS yet)

**Files:** none (gitea admin ops).
- [ ] Inventory both giteas' repos (names/sizes) to confirm scope:
      foxhunt gitea (via `git.fxhnt.ai` today) has `gitadmin/fxhnt`, `gitadmin/foxhunt` (+ any others — list them).
- [ ] On the **bizworx gitea**, create the `gitadmin` org/user + empty `fxhnt` and `foxhunt` repos.
- [ ] Mirror all refs/tags/LFS from foxhunt gitea → bizworx gitea (gitea migration UI/API, or
      `git clone --mirror` from `git.fxhnt.ai` then `git push --mirror` to the bizworx gitea endpoint via a
      temporary port-forward / tailnet address).
- [ ] Verify parity: for `fxhnt` and `foxhunt`, `git ls-remote` ref count + latest master/main sha match
      between the two giteas.

**EVAL:** bizworx gitea holds both repos with matching refs/tags. `git.fxhnt.ai` STILL points at foxhunt
(unchanged) — this task is a non-destructive copy. **No foxhunt change; both giteas intact.**

### Task 4: Re-target the k8s manifests (branch, dry-run only)

**Files:** `infra/k8s/**` (on this branch `chore/foxhunt-bizworx-consolidation`).
- [ ] Identify manifests that hardcode the foxhunt cluster/endpoints: gitea `gitea-sshd.foxhunt.svc…`
      (still valid — same ns name), image registry `rg.fr-par.scw.cloud/bizworx/…` (already bizworx — good),
      StorageClass names, any nodeSelector/affinity, the `git.fxhnt.ai` git-sync URL, ingress hosts.
- [ ] Change only what must change for bizworx (StorageClass if different; git-sync stays on `git.fxhnt.ai`
      since DNS will re-point; ns stays `foxhunt`). Commit to the branch.
- [ ] Dry-run against bizworx: `kubectl --context $BIZ apply -n foxhunt --dry-run=server -f <manifest>` for
      each (server-side dry-run catches admission/StorageClass errors).

**EVAL:** every fund manifest passes server-side dry-run on bizworx. **Nothing applied for real yet.**

### Task 5: Stand up Postgres/Timescale on bizworx (empty)

**Files:** `infra/k8s/databases/**`.
- [ ] Apply the Postgres/Timescale manifest to bizworx ns `foxhunt` (new PVC on the chosen StorageClass).
- [ ] Verify it comes up + the TimescaleDB extension version MATCHES foxhunt (from Task 1):
      `kubectl --context $BIZ exec -n foxhunt deploy/postgres -- psql -U postgres -tAc "select extversion from pg_extension where extname='timescaledb';"`

**EVAL:** bizworx Postgres is Running and its TimescaleDB version == foxhunt's. If mismatch → pin the image to
match BEFORE any restore (a version skew breaks hypertable restore).

### Task 6: DB migration REHEARSAL (non-destructive dump/restore)

**Files:** none (a rehearsal; foxhunt keeps running/serving).
- [ ] `pg_dump` the foxhunt `fxhnt` DB using the **hypertable-aware** path (call `timescaledb_pre_restore()`
      on the target before restore and `timescaledb_post_restore()` after; use `--no-owner` as needed). Stream
      foxhunt→bizworx (e.g. `kubectl exec … pg_dump … | kubectl exec -i … pg_restore …`) — read-only on foxhunt.
- [ ] Verify row counts match on the load-bearing tables:
      `forward_nav`, `backtest_summary`, `bybit_sleeve_ret`, `xsp_option_bars`, the feature/`*_features` tables,
      `exec_fills`, `ibkr_account_nav`. Compare `SELECT count(*)` foxhunt vs bizworx for each.

**EVAL:** every checked table's row count matches. This proves the migration procedure works BEFORE cutover.
foxhunt untouched (dump is read-only). (The real cutover in Task 8 re-runs this on the frozen DB.)

### Task 7: Stand up the fund on bizworx IN PARALLEL (no DNS yet) — the "alles werkt eerst" gate

**Files:** `infra/k8s/{jobs,gitea,monitoring→skip,...}`.
- [ ] Apply the fund workloads to bizworx ns `foxhunt`: dagster, ib-gateway, fxhnt-dashboard (cockpit),
      pointed at the bizworx Postgres (restored in Task 6) + git-sync from the **bizworx gitea** (use its
      tailnet/ClusterIP endpoint directly for now, since `git.fxhnt.ai` still points at foxhunt).
- [ ] Apply the CronJobs **all SUSPENDED** (`suspend: true`) initially.
- [ ] **ib-gateway:** verify it connects to IBKR paper on bizworx (no 2FA/IP-limit) + a `whatIf` order is
      accepted: exec the gateway health / a dry-run probe.
- [ ] **Cockpit:** port-forward the bizworx cockpit and verify in a **real browser** it renders the migrated
      data (NAVs, gates, holdings) — read the numbers; compare against the live foxhunt cockpit.
- [ ] **Rebalancer dry-run:** run `fxhnt-ucits-rebalancer` once manually as a dry-run Job on bizworx (EU hours)
      → it should plan/price the UCITS lines without error (or fill, on paper).
- [ ] **Dagster:** trigger the nightly asset graph once on bizworx → green.

**EVAL (the big one):** the ENTIRE fund works on bizworx in parallel with foxhunt still live — cockpit correct,
ib-gateway connected, rebalancer dry-run clean, dagster green, recon gate reconciles the migrated refs. **Only
if ALL pass** do we proceed to cutover. Nothing on foxhunt has changed.

---

## PHASE 1 — cutover (only after Task 7 passes)

### Task 8: Cutover window (downtime OK, flexible)

**Files:** DNS/tailnet + ingress config.
- [ ] Announce/pick a quiet window (fund is ~1×/day; any convenient time).
- [ ] Freeze foxhunt: `kubectl --context $FOX patch cronjob <each fund cronjob> -n foxhunt -p '{"spec":{"suspend":true}}'`
      and scale down foxhunt dagster/cockpit writers so its Postgres is quiescent.
- [ ] Final delta dump/restore foxhunt→bizworx Postgres (re-run Task 6's procedure on the now-frozen DB) so
      bizworx has the latest state. Re-verify the key row counts.
- [ ] Copy `unlock_calendar.json` (or rely on the DB-first loader), ib-gateway settings if not re-authed.
- [ ] Re-point DNS: `git.fxhnt.ai` and `dashboard.fxhnt.ai` → bizworx (tailnet node / ingress; ACME TLS for
      the dashboard). Fund git-sync now resolves the bizworx gitea.
- [ ] Un-suspend the fund CronJobs on **bizworx** (start with the UCITS rebalancer; leave US/bybit suspended
      as they already are).

**EVAL:** on bizworx via the real hostnames — cockpit at `dashboard.fxhnt.ai` renders correctly, `git.fxhnt.ai`
serves the repos + git-sync works, the armed rebalancer's next run succeeds. foxhunt is now suspended (NOT
deleted). **Rollback still trivial** (re-point DNS + un-suspend foxhunt).

### Task 9: Cockpit + prod verification post-cutover

**Files:** none.
- [ ] Verify cockpit LOCALLY in a real browser against the bizworx DB (dev-cockpit loop) AND on
      `dashboard.fxhnt.ai`; read the numbers (`feedback_verify_cockpit_locally_not_prod`).
- [ ] Confirm the reconciliation gate + forward tracks are intact (no spurious re-inception; deterministic
      recompute from the migrated DB anchor).

**EVAL:** prod cockpit + gate correct on bizworx. If anything is off → rollback (foxhunt still intact).

---

## PHASE 1 — observation (keep BOTH, delete NOTHING)

### Task 10: Parallel observation window

**Files:** none.
- [ ] Run the fund on bizworx for an agreed window (e.g. several trading days) with foxhunt suspended-but-intact.
- [ ] Each day, evaluate: dagster nightly green, the UCITS rebalancer run clean, recon gate healthy, cockpit
      numbers sane, no stray-holdings/venue surprises.

**EVAL:** N consecutive clean days on bizworx. **This is the gate that unlocks any teardown.** Until it passes,
nothing is deleted.

---

## PHASE 2 — triage the rest (per-item operator sign-off before any delete)

### Task 11: Keep/drop inventory of non-fund ns `foxhunt`

**Files:** produce `docs/superpowers/specs/…-phase2-triage.md` (the inventory).
- [ ] For every non-fund workload (0-replica legacy: api/backtesting-service/broker-gateway/
      data-acquisition-service/ml-training-service/questdb/loki/pushgateway; other: minio/redis/kanidm/
      mattermost/stalwart/netbird) and every orphan PVC (training-data 500Gi, gitlab 50Gi, feature-cache,
      fxhnt-backtest-data, multistrat-state, test-data, tempo, …): classify DROP / MIGRATE / KEEP-EXTERNAL.
- [ ] Operator signs off **per item** before its data is deleted (look before deleting).

**EVAL:** a signed-off inventory; no ambiguous items remain.

---

## PHASE 3 — teardown (ONLY after Task 10 + Task 11)

### Task 12: Retire the foxhunt Kapsule

**Files:** `infra/live/production/kapsule/**` (Terragrunt, project `c293eb98`), `infra/k8s/**`.
- [ ] Remove the foxhunt k8s manifests / argo apps.
- [ ] `terraform destroy` (via Terragrunt) the foxhunt Kapsule: control plane + `platform` (3× dev1_l) +
      `ci-compile-cpu` pools.
- [ ] Explicitly delete retain-class volumes that won't auto-release (`*-retain` PVCs), the LB, private
      network `4ac46c9e…`, and other foxhunt-project scw resources; drop the confirmed-dead PVCs from Task 11.
- [ ] Remove the temporary foxhunt API ACL entry for this box (`scw k8s acl` — the `bizworx-devbox-v4` entry),
      unless still useful.
- [ ] Update memory: foxhunt Kapsule retired; fund lives on bizworx; single gitea = bizworx.

**EVAL:** foxhunt Kapsule + nodes gone; fund still healthy on bizworx; scw monthly cost dropped by the
foxhunt control plane + 3 dev1_l nodes + its block storage. No wanted data lost.

---

## Self-review notes
- **Verify-first honored:** Tasks 1–7 are additive (foxhunt untouched); the only foxhunt change before teardown
  is *suspend* (Task 8), fully reversible. First delete is Task 12, gated behind Tasks 10 + 11.
- **Per-step eval honored:** every task has an `EVAL:` gate.
- **Secrets:** Task 2 uses `!`/openbao, never the transcript.
- **Rollback:** available through Task 11 (foxhunt intact-and-suspended).
