# Foxhunt → Bizworx Platform Consolidation — Design

**Date:** 2026-07-18
**Status:** DRAFT (awaiting operator review)
**Goal:** Retire the separate **foxhunt** Scaleway Kapsule cluster by moving the live fxhnt fund onto the
**bizworx** Kapsule cluster and consolidating to a **single gitea**, then destroying the foxhunt Kapsule —
eliminating a whole managed cluster + its nodes (cost saving).

> Written durably to git because this session suffered heavy context loss (a stale 2026-07-13 replay + a
> dev-machine move). Ground truth in this doc was verified live against both clusters on 2026-07-18.

---

## 1. Current state (verified live 2026-07-18)

Two Scaleway Kapsule clusters, **same org (`a55098a3`), different projects**, both `fr-par`, k8s 1.34.6, cilium:

| Platform | Kapsule | Project | Nodes | Role |
|---|---|---|---|---|
| **bizworx** (survivor) | `05e1dae3` | `96d0717d` | 3× gp1_xs (+large) | MSP platform (msp-portal, tacticalrmm, argocd, vaultwarden, gitea…) |
| **foxhunt** (to retire) | `34a1e3c4` | `c293eb98` | 3× dev1_l (+ci 0-4) | the fxhnt fund (ns `foxhunt`) + lots of legacy |

**Access note (resolved):** the foxhunt Kapsule API has an IP allowlist (`acl_available: true`). After the
dev-machine move, this box's egress `51.158.111.225` was not allowed → HTTP 000. Fixed by appending it to the
ACL (`scw k8s acl add … bizworx-devbox-v4`; existing tailscale-cgnat + admin entries preserved). kubeconfig
re-installed via `scw k8s kubeconfig install 34a1e3c4…`. Both contexts now in `~/.kube/config`.

**xps-ubnt** (tailnet `100.91.132.125`) = the OLD dev machine. k3s inactive; runs some Docker (a dev
`foxhunt-postgres` Timescale, a buildx builder, unrelated lexpdf/PDF tools). NOT the fund host — just where
the foxhunt kubeconfig still lived. Not part of this migration except as a source of the old kubeconfig.

### 1a. What actually runs in ns `foxhunt` (empirical)

**Live (READY=1) — the fund's real footprint:** `postgres`, `dagster`, `ib-gateway`, `fxhnt-dashboard`
(cockpit), `gitea`, `argo` (workflows + events + gitea push/deploy sensors), plus `minio`, `redis`,
`grafana`/`prometheus` (the last four are NOT fund dependencies — see below).

**Scaled to zero / dead (READY none) — legacy foxhunt, Phase-2 triage:** `api`, `backtesting-service`,
`broker-gateway`, `data-acquisition-service`, `ml-training-service` (old Rust microservice architecture),
`kanidm` (SSO), `mattermost`, `stalwart` (mail), `netbird-*`, `questdb`, `loki`, `pushgateway`.

### 1b. Stateful data — what migrates vs drops (empirical: mounted-by-running-pod)

| PVC | Size | Mounted by | Decision |
|---|---|---|---|
| `postgres-pvc` | 100Gi | postgres | **MIGRATE** (operational DB + Timescale warehouse = the SSOT) |
| `ib-gateway-settings` | 1Gi | ib-gateway | migrate OR re-auth fresh (trivial) |
| `fxhnt-surfer-data` | 5Gi | dagster | holds only `unlock_calendar.json` (DB-first + JSON fallback) → copy the file or let it rebuild |
| `gitea-shared-storage` | 5Gi | gitea | migrates via the gitea repo-migration (§3), not as a raw PVC |
| `feature-cache-pvc` | 100Gi | — (orphan) | DROP (rebuildable cache) |
| `fxhnt-backtest-data` | 20Gi | — (orphan) | DROP (B3 DuckDB warehouse, parked in 0a) |
| `questdb-pvc` | 10Gi | — (questdb dead) | DROP |
| `multistrat-state` | 1Gi | — (orphan) | DROP (vestigial state_file; forward state is DB-SSOT) |
| `minio-data-new` | 150Gi | minio | Phase-2 triage (NOT a fund dep — `grep` of fxhnt src/manifests finds no minio/s3/redis runtime use) |
| `training-data-pvc` | 500Gi | — | Phase-2 (decommissioned Rust ML) → DROP |
| `repo-data-gitlab-gitaly-0` | 50Gi | — | Phase-2 (legacy GitLab, replaced by gitea) → DROP |
| `kanidm/mattermost/stalwart/netbird/test-data/tempo/…` | ~120Gi | — | Phase-2 triage (drop unless a non-fund need surfaces) |

**Net fund data to physically migrate ≈ 100Gi (one Postgres), everything else is drop-or-rebuild.**
"fxhnt heeft minder nodig" — confirmed.

### 1c. Gitea reality

- `git.fxhnt.ai` → `100.95.225.27` (tailnet node `foxhunt-gitlab`) → the **foxhunt-cluster gitea** (repos
  `gitadmin/fxhnt`, `gitadmin/foxhunt`). Canonical today.
- **bizworx** already runs its own gitea (ns `gitea`, 50Gi, ClusterIP, no ingress).
- The fund's in-cluster git-sync clones from `gitea-sshd.foxhunt.svc.cluster.local:22`.

### 1d. IaC form

Two layers, both in the fxhnt repo: **Terragrunt/Terraform** (`infra/live/production/{kapsule,
public-gateway,block-storage}`, scw S3 tfstate backend) for the cloud infra (the Kapsule itself), and
**k8s manifests** (`infra/k8s/{argo,databases,gitea,jobs,monitoring,…}`) applied via **argo** (git-sync from
gitea; `reference_fxhnt_prod_deploy_mechanics`: `argo submit … -p mode=deploy`, NOT image-baked).

---

## 2. Target architecture (after Phase 1)

The fund runs in **ns `foxhunt` on the bizworx Kapsule** (name reused → minimal manifest churn):
- Workloads: `postgres` (Timescale), `dagster`, `ib-gateway`, `fxhnt-dashboard`, the rebalancer/backtest
  CronJobs, git-sync from the **bizworx gitea**.
- **Reuse bizworx's existing `argo` + observability** (do NOT migrate prometheus/grafana/loki/tempo).
- Single **bizworx gitea** holds `gitadmin/fxhnt` + `gitadmin/foxhunt` (archive); `git.fxhnt.ai` re-points to it.
- `dashboard.fxhnt.ai` ingress → bizworx (ACME TLS, per `reference_fxhnt_dashboard_proxy`).

Decisions locked with the operator (2026-07-18):
- **Survivor gitea = bizworx**; migrate both `fxhnt` + `foxhunt` (archive); keep hostname `git.fxhnt.ai`,
  re-point to bizworx (no remote-URL changes in repos/CI).
- **ib-gateway:** no 2FA, no IP allowlist on the IBKR side → re-auth = IBKR paper creds in a secret + reconnect.
- **Downtime:** flexible (fund runs ~once/day) — schedule a convenient quiet window; no tight SLA.
- Reuse bizworx argo + observability.

---

## 3. Phase 1 — migration mechanics

### 3.1 Prep bizworx (no downtime)
- Create ns `foxhunt` on bizworx; confirm a suitable StorageClass (bizworx uses `sbs-5k`/`sbs-default`;
  foxhunt used `sbs-default`/`scw-bssd`).
- Recreate secrets on bizworx (source via bizworx external-secrets + openbao where possible): `db-credentials`,
  `databento-credentials`, IBKR/IB-gateway creds, `scw-registry` (image pull), `argo-git-ssh-key` /
  `git-ssh-key`, tailscale auth. **Never** commit secrets; use `! kubectl`/openbao, never the transcript.
- Migrate gitea repos: create `gitadmin/fxhnt` + `gitadmin/foxhunt` on the bizworx gitea and
  `git push --mirror` (or gitea's built-in migration) all refs/LFS. Verify commit counts + tags match.
- Re-target the k8s manifests' cluster/namespace/gitea endpoint to bizworx; `kubectl apply --dry-run` on the
  bizworx `foxhunt` ns to validate before cutover.

### 3.2 Cutover (the downtime window)
1. **Freeze foxhunt:** `kubectl patch cronjob … suspend:true` for all fund CronJobs; scale dagster/cockpit
   writers down so Postgres is quiescent.
2. **Migrate the DB:** `pg_dump` the foxhunt Postgres (`fxhnt` DB, all schemas incl. TimescaleDB hypertables —
   use `pg_dump`-with-timescaledb guidance / `timescaledb_pre_restore`/`post_restore`) → `pg_restore` into the
   bizworx Postgres. Verify row counts on the key tables (`forward_nav`, `backtest_summary`, `bybit_sleeve_ret`,
   `xsp_option_bars`, the feature tables).
3. **Stand up the fund on bizworx:** apply the re-targeted manifests; bring up postgres (restored),
   dagster, cockpit, ib-gateway (re-auth the IBKR paper session).
4. **Re-point DNS:** `git.fxhnt.ai` and `dashboard.fxhnt.ai` → bizworx (tailnet node / ingress). Fund git-sync
   now resolves the bizworx gitea `gitea-sshd`.
5. **Copy the small bits:** `unlock_calendar.json` (or let it rebuild from the DB-first loader);
   ib-gateway-settings if not re-authing fresh.

### 3.3 Verify (before decommissioning anything)
- Cockpit renders real data — **locally in a real browser** (`reference_fxhnt_local_cockpit_dev_loop`) AND on
  `dashboard.fxhnt.ai`; read the numbers (`feedback_verify_cockpit_locally_not_prod`).
- A `fxhnt-ucits-rebalancer` **dry-run** on bizworx fills/plans correctly against IBKR (EU hours).
- Reconciliation gate + forward tracks intact (the migrated `forward_nav`/`backtest_summary` reconcile;
  no false re-inception — `forward_engine` recompute is deterministic from the DB anchor).
- Dagster nightly asset graph runs green on bizworx.
- Run **in parallel/observe** for a short window (foxhunt suspended, bizworx live) before teardown.

### 3.4 Rollback
Until the foxhunt Kapsule is destroyed (Phase 3), rollback = re-point DNS back + un-suspend foxhunt CronJobs
(its Postgres still holds the pre-cutover state). The foxhunt cluster is left intact-but-suspended through the
observation window precisely to keep this cheap.

---

## 4. Phase 2 — triage the rest of ns `foxhunt`

Produce a definitive **keep/drop** list for every non-fund service + orphan PVC, so the teardown is
accountable (nothing silently lost). Expected outcome (from §1a/1b, mostly DROP):
- **DROP (confirmed dead):** the Rust-era microservices (api/backtesting-service/broker-gateway/
  data-acquisition-service/ml-training-service), `questdb`, `loki`, `pushgateway`, `training-data-pvc` (500Gi),
  `repo-data-gitlab-gitaly-0` (50Gi), `feature-cache`, `fxhnt-backtest-data`, `multistrat-state`, `test-data`.
- **DECIDE (non-fund, may still be wanted somewhere):** `minio` (150Gi), `redis`, `kanidm` (SSO),
  `mattermost`, `stalwart` (mail), `netbird`. For each: drop, or migrate to bizworx / keep external. The
  operator confirms per item before its data is deleted (`feedback` — look before deleting).

## 5. Phase 3 — teardown (cost saving)

After Phase-1 verification holds and the Phase-2 list is resolved:
- Remove the foxhunt k8s manifests; delete the foxhunt Kapsule via **Terragrunt** (`infra/live/production/
  kapsule` for project `c293eb98`) — control plane + the 3 dev1_l nodes + the ci pool.
- Release orphaned block volumes (retain-class PVCs: `gitea-shared-storage`, `feature-cache` are
  `*-retain` → must be deleted explicitly), the load balancer, the private network, and the foxhunt-project
  scw resources.
- Remove the temporary foxhunt API ACL entry for this box (or keep if still useful).
- Update memory: mark the foxhunt Kapsule retired; the fund now lives on bizworx.

---

## 6. Risks / open items
- **TimescaleDB dump/restore** needs the hypertable-aware procedure (`timescaledb_pre_restore()` /
  `timescaledb_post_restore()`), not a naive `pg_dump`. Verify the extension version matches on bizworx.
- **ib-gateway** must re-establish its IBKR paper session on bizworx (no 2FA/IP-limit per operator, so
  expected simple — but confirm it connects + a whatIf order is accepted before arming the rebalancer).
- **Secrets provenance:** confirm every secret can be re-sourced on bizworx (openbao/external-secrets) before
  cutover — a missing secret wedges a workload.
- **Cross-project scw:** LB/DNS/tailnet re-pointing spans the two scw projects; confirm the tailnet ACLs +
  ingress + ACME cert issuance work from the bizworx side.
- **Deploy pipeline:** the fund deploys via argo git-sync from gitea (`mode=deploy`); confirm the bizworx argo
  + the migrated gitea + the eventsource/sensor wiring reproduce the deploy flow.

## 7. Success criteria
- The fxhnt fund runs entirely on the bizworx Kapsule (cockpit, dagster, ib-gateway, rebalancers) against the
  migrated Postgres, verified in a real browser + a live rebalancer dry-run.
- One gitea (bizworx) serves `git.fxhnt.ai` with both repos; fund git-sync + CI unchanged.
- The foxhunt Kapsule + its nodes are destroyed; no fund data or wanted service lost (Phase-2 list resolved).
- Monthly scw cost drops by the foxhunt cluster's control-plane + 3 dev1_l nodes + its block storage.

---
*Related memory:* `project_phase2_gitlab_to_gitea` (GitLab→Gitea + foxhunt→fxhnt IaC), `netbird-mesh`,
`reference_fxhnt_prod_deploy_mechanics`, `reference_fxhnt_dashboard_proxy`,
`reference_fxhnt_local_cockpit_dev_loop`, `reference_ci_pool_and_tfstate`, `infrastructure`.
