# Foxhunt→Bizworx Consolidation — Live Progress (resume anchor)

## 🚦 T8 CUTOVER EXECUTED (2026-07-18 ~20:15 UTC) — fxhnt.ai NOW SERVES FROM BIZWORX ✅
**The fund has cut over. foxhunt is FROZEN but INTACT (rollback available). DELETE NOTHING until T10 observation passes.**
- **C1 (freeze foxhunt):** suspended fund cronjobs (ucits-rebalancer, ucits-volume-ibkr, compare-measured-precompute)
  + scaled foxhunt dagster/fxhnt-dashboard/ib-gateway → 0/0. Infra cronjobs (postgres-backup/pvc-autoscaler/
  billing-scraper) LEFT RUNNING (they die at teardown, not now). PASS.
- **C2 (delta DB migrate):** pg_dump -Fc fxhnt (47.8M) from frozen foxhunt → restore into bizworx. GOTCHA:
  `pg_restore --clean --if-exists` over the existing rehearsal copy FAILED (dropped _timescaledb_internal schema
  mid-restore → 1136 errors, DB stuck restoring:on). FIX (rehearsal-proven path): DROP+CREATE empty fxhnt DB,
  then pre_restore → pg_restore (NO --clean) → post_restore = exit 0, 0 errors. EVAL C2 PASS: all counts match
  frozen source (forward_nav 106, backtest_summary 6, bybit_sleeve_ret 7023, xsp_option_bars 814373, exec_fills 9,
  ibkr_account_nav 4, forward_anchor 20; max_anchor_t0 2026-07-14, max_fwd_nav 2026-07-17; 37 tables/3 hypertables).
  NOTE: counts were IDENTICAL to the rehearsal (fund already suspended during parallel work → zero real delta).
  LESSON: NEVER pg_restore --clean over a live timescaledb DB; always restore into a fresh empty DB.
- **C3 (bizworx fund live):** scaled dagster/cockpit/ib-gateway→1 + rollout-restart cockpit&dagster (fresh DB
  conns). EVAL C3 PASS: cockpit serves real migrated data over the proxy (/ + /strategy/multistrat + /xsfunding
  = 200, shows PASS/WAIT/Sharpe/multistrat/xsfunding); ib-gateway 1/1 STABLE (~12min, foxhunt released IBKR
  session); dagster 3/3, ingress 3/3, postgres 1/1.
- **C4 (DNS flip):** Scaleway fxhnt.ai records dashboard/cockpit/dagster/git A → **100.93.141.11** (bizworx
  fxhnt-ingress), TTL 60 (was 100.95.225.27 foxhunt proxy). SYNTAX: `scw dns record set dns-zone=fxhnt.ai
  name=<n> type=A ttl=60 values.0=100.93.141.11` (values.0=, NOT data=). EVAL C4 PASS over REAL public DNS:
  dashboard/git/dagster.fxhnt.ai → 100.93.141.11, HTTP 200, tls_verify=0. Content confirmed: git=bizworx Gitea,
  dashboard=cockpit "Overview · fxhnt" w/ real data, SSH `git ls-remote ssh://git@git.fxhnt.ai/gitadmin/fxhnt.git`
  returns real refs (HEAD d66d1132) via socat path. (SSH hostkey-changed warning = correct, it's bizworx now.)
- **C5 (arm fund):** un-suspended ONLY fxhnt-ucits-rebalancer on bizworx (schedule 0 9 * * 1-5 = 09:00 UTC wkdays;
  next run tomorrow AM). US(ibkr)/bybit/vrp/factory/ucits-volume all stay SUSPENDED. EVAL C5 (first armed run
  green) is OBSERVABLE at next 09:00 UTC — deps verified (DB+ib-gateway+git-sync all live).
- **ROLLBACK (still trivial until T11):** `scw dns record set ... values.0=100.95.225.27` (back to foxhunt proxy)
  + `kubectl --context $FOX -n foxhunt scale deploy/dagster deploy/fxhnt-dashboard deploy/ib-gateway --replicas=1`
  + un-suspend foxhunt fund cronjobs. foxhunt Postgres still holds pre-cutover state.
- **NEXT:** T9 verify (cockpit LOCAL browser + prod, gate/forward-track intact) → T10 observation window (watch
  the first ucits run + a day) → T11 Phase-2 triage → T12 teardown. **ROTATE the pasted tailscale key.** DELETE NOTHING.


Branch `chore/foxhunt-bizworx-consolidation`. Real date 2026-07-18. Plan:
`docs/superpowers/plans/2026-07-18-foxhunt-to-bizworx-consolidation.md`. Contexts:
`FOX=foxhunt-34a1e3c4-…`, `BIZ=bizworx-prod-kapsule-05e1dae3-…` (both in ~/.kube/config).

## Done + verified
- **Access restored:** appended this box's egress `51.158.111.225/32` to the foxhunt Kapsule API ACL
  (`scw k8s acl`, entry `bizworx-devbox-v4`) + `scw k8s kubeconfig install` → foxhunt context works.
- **T1** ns `foxhunt` created on bizworx; StorageClass `sbs-default` matches; source = PG16.13/tsdb2.25.1.
- **T2** 6 fund secrets copied FOX→BIZ ns foxhunt (db-credentials, databento, tiingo-api, ibkr-credentials,
  argo-git-ssh-key, scw-registry). NOT copied: bybit-testnet (absent/suspended), tailscale-auth (deferred).
- **T3** gitea consolidated → bizworx gitea (ns gitea): `gitadmin/fxhnt` (8 heads/0 tags, master d66d113)
  + `gitadmin/foxhunt` (13 heads/105 tags, main 8213f29a5), refs verified identical. foxhunt (625MB) done
  via direct pack-copy (HTTP push + server-side index-pack both failed on size/OOM). `git.fxhnt.ai` STILL
  points at foxhunt (DNS re-point is T8). Caveat: foxhunt archive's gitea DB metadata may lag (disk-correct).
- **T4** manifests re-targeted (committed): git-sync `gitea-sshd.foxhunt`→`gitea-ssh.gitea.svc.cluster.local:22`;
  nodeSelector→`large`; ib-gateway pull-secret removed (public ghcr image). Server-side dry-run clean.
- **T5/T6** Postgres on bizworx PINNED to `timescale/timescaledb:2.25.1-pg16` (latest=2.28.3 broke catalog
  restore — REHEARSAL caught it). Rehearsal restore of `fxhnt` DB: exit 0, 0 errors, all row counts match
  (forward_nav 106, backtest_summary 6, bybit_sleeve_ret 7023, xsp_option_bars 814373, exec_fills 9,
  ibkr_account_nav 4), 37/37 tables, 3/3 hypertables. bizworx fxhnt DB currently holds this rehearsal copy.

## T7 IN PROGRESS (parallel verify — foxhunt still fully live)
- **7.1 DONE** — argo-git-ssh-key deploy-key registered read-only on bizworx gitadmin/fxhnt (id 2);
  verified `git ls-remote` via the key = 8 refs.
- **cockpit DONE** — `fxhnt-dashboard` deployed on bizworx; git-sync OK (cloned d66d113 from bizworx gitea);
  cockpit container Running+ready, renders migrated data (overview + /strategy/xsfunding + /strategy/multistrat
  all HTTP OK with real WAIT/PASS gates). Verified from INSIDE the container (port-forward blocked — see below).
- **BLOCKER / decision pending — tailscale:** dagster + cockpit pods have a tailscale sidecar needing
  `tailscale-auth` (not copied) → sidecar CreateContainerConfigError → pod phase Pending → port-forward
  refuses. Cockpit itself is fine. DECIDE: **A** fresh reusable tailscale auth-key (bizworx pods join as
  extra nodes alongside live foxhunt ones; removed at cutover) OR **B** disable sidecar for the parallel
  phase, verify via port-forward, enable tailscale at T8 cutover with real hostnames.

## T7 REMAINING
- ib-gateway → apply + verify IBKR paper connect (no 2FA/IP-limit per operator) + whatIf.
- dagster → apply (check its NetworkPolicy allows egress to ns gitea for git-sync — cross-ns) + trigger
  nightly asset graph green.
- CronJobs → apply all SUSPENDED; then one `fxhnt-ucits-rebalancer` dry-run (EU hours).
- NP note: fund NPs (dagster, cronjobs) allow egress to gitea in-ns; on bizworx gitea is ns `gitea` →
  may need namespaceSelector. (cockpit had no NP → git-sync worked.)

## HARD RULE (operator): verify-first, per-step eval, DELETE NOTHING until Phase-1 verified + observation
window (T10) passes. foxhunt stays live/intact through T8, suspended-but-intact until T12 teardown.
Rollback until T11 = re-point DNS + un-suspend foxhunt.

## T7 update (2026-07-18, later)
- tailscale-auth secret created on bizworx (operator gave a fresh reusable key — OPTION A; key was pasted in
  transcript, ROTATE after cutover). cockpit pod now 2/2 Running (cockpit + tailscale sidecar joined).
- ib-gateway deployed: IBC logs in to DU9600528 (Simulated Trading) — IBKR paper connect WORKS (no 2FA/IP).
  ⚠️ pod had 2 restarts during IBC startup churn — WATCH it stabilises to 1/1 ready; verify API port 4003/4004
  + a whatIf before arming any rebalancer.
- dagster was Pending on missing PVC `fxhnt-surfer-data` (dropped from migration; manifest still mounts it) →
  created empty 5Gi PVC (sbs-default); dagster now scheduling (git-sync init). unlock_calendar.json rebuilds.
- REMAINING T7: confirm ib-gateway 1/1 + whatIf; confirm dagster git-sync (cross-ns NP to gitea) + trigger
  nightly asset graph green; apply CronJobs SUSPENDED; one fxhnt-ucits-rebalancer dry-run (EU hours).
- Then T8 cutover (freeze foxhunt, delta restore, DNS re-point git.fxhnt.ai + dashboard.fxhnt.ai) → T9 verify
  → T10 observe → T11 triage → T12 teardown. foxhunt still fully live; delete NOTHING yet.

## T7 update 2 (2026-07-18)
- dagster FIXED + 3/3 Running: (1) created `dagster` DB on bizworx postgres (fresh dagster, not migrated);
  (2) dagster NP gitea-egress needed a cross-ns `namespaceSelector {kubernetes.io/metadata.name: gitea}`
  (committed in dagster.yaml) → git-sync from bizworx gitea now OK. webserver+daemon+tailscale all ready.
- ib-gateway: connects to DU9600528 (IBC login OK) but exits 0 after ~6min = IBKR single-session conflict with
  the STILL-LIVE foxhunt ib-gateway. Connect verified; stable run only after foxhunt-suspend (T8). NOT a bug.
- T7 CORE VERIFIED: postgres+cockpit+dagster all up on bizworx against migrated data.
- REMAINING T7: (a) apply the SAME cross-ns gitea-NP fix to every cronjob NP (jobs/fxhnt-*.yaml) BEFORE they
  run (else git-sync blocked like dagster was); (b) apply CronJobs SUSPENDED; (c) one ucits dry-run (needs
  ib-gateway sole-session → do at/after cutover, or briefly scale foxhunt ib-gateway to 0); (d) optional dagster
  asset-graph run. Then T8 cutover.
- ALSO for cutover: the tailscale bizworx pods joined as extra nodes (fresh key); at T8 suspend foxhunt so the
  real hostnames + IBKR session move cleanly. ROTATE the pasted tailscale key post-cutover.

## P1 EXECUTION (2026-07-18) — wildcard cert + the cross-project DNS-01 IAM fix
- Applied `infra/k8s/services/fxhnt-wildcard-cert.yaml` (Certificate fxhnt.ai + *.fxhnt.ai, ClusterIssuer
  letsencrypt-prod, secret fxhnt-wildcard-tls, ns foxhunt) on BIZ.
- **DNS-01 FAILED: `403 Forbidden: domain not found`.** Root cause: `fxhnt.ai` DNS zone lives in the **foxhunt
  project c293eb98**; the bizworx cert-manager key (`SCWW9261…`, app `bizworx-prod-runtime` 48418a71) has
  `DomainsDNSFullAccess` but SCOPED to the **bizworx project 96d0717d** only → can't touch fxhnt.ai's zone.
- **DEAD END (operator's first pick "move the domain to bizworx project"): Scaleway BLOCKS it.** fxhnt.ai is an
  EXTERNAL domain (`scw domain domain list` → IS EXTERNAL=true, REGISTRAR=EXTERNAL). `scw dns zone update
  … project-id=…` on the apex → **`Root zone can't be updated`**; on a scratch sub-zone → **`Subdomain not
  allowed`**; `scw domain domain` has NO move-project subcommand. Verified via a throwaway `_cutovertest.fxhnt.ai`
  sub-zone (created+move-attempt+deleted). So the domain CANNOT be reassigned to the bizworx project via API.
- **ACTUAL FIX (operator's 2nd pick, applied): add an IAM rule.** `scw iam rule create
  policy-id=8573ce25-… permission-set-names.0=DomainsDNSFullAccess project-ids.0=c293eb98-…` → policy
  `bizworx-prod-runtime-scoped` now has 2 rules (orig 6-perms@96d0717d untouched + new DNS@c293eb98). This lets
  the existing bizworx cert-manager key write the DNS-01 TXT on fxhnt.ai. Domain STAYS external in c293eb98;
  only IAM changed. NOTE for teardown: this rule references the foxhunt project — harmless to keep, or tighten
  later; if the foxhunt PROJECT (not just cluster) is ever deleted, the fxhnt.ai domain/zone must first move
  registrar-side or certs break. (Cluster teardown ≠ project deletion; the DNS zone + domain survive T12.)
- Forced cert-manager retry (deleted order+CR) after the IAM change → 403 GONE, challenges progressing normally
  (apex fxhnt.ai challenge = `valid`/"Successfully authorized domain"; wildcard `*.fxhnt.ai` challenge finishing
  its DNS-01 propagation). TXT is being written to the fxhnt.ai zone by cert-manager (verified via
  `scw dns record list dns-zone=fxhnt.ai | grep acme`). P1 is SUCCEEDING — cert will go READY shortly.
  If resuming and cert not yet READY: `kubectl --context $BIZ -n foxhunt get certificate fxhnt-wildcard -o wide`;
  if stuck >15min, `kubectl delete order,challenge,certificaterequest --all -n foxhunt` to force a clean retry
  (IAM rule is permanent so it'll succeed).

## P2 DRAFTED (2026-07-18) — fxhnt-ingress proxy (NOT applied yet; waiting on P1 cert READY)
- `infra/k8s/services/fxhnt-ingress-proxy.yaml` (committed ce1c978, draft): nginx (wildcard TLS via
  fxhnt-wildcard-tls) + socat (git SSH→gitea-ssh.gitea:22) + tailscale sidecar (TS_HOSTNAME=fxhnt-ingress).
  Fronts 3 fund vhosts BY CLUSTERIP: git→gitea-http.gitea:3000, dashboard/cockpit→fxhnt-dashboard.foxhunt:80,
  dagster→dagster.foxhunt:80. Has its own NP (egress to those svcs + apiserver 172.16.0.11:6443 for TS + DERP).
  Deps satisfied: tailscale-auth secret present; dagster NP ingress already allows fxhnt-ingress label.
## P1+P2 COMPLETE (2026-07-18) — bizworx fxhnt.ai front verified, NO DNS change yet ✅
- **P1 DONE:** fxhnt-wildcard cert READY (LE, SAN=*.fxhnt.ai + fxhnt.ai). IAM cross-project fix held.
- **P2 DONE:** fxhnt-ingress proxy applied + 3/3 Running. Tailnet identity = **`fxhnt-ingress` @ `100.93.141.11`**
  (clean, no -1 suffix). THIS IS THE C4 DNS TARGET.
- **EVAL P2 PASS (from devbox over the real tailnet → 100.93.141.11, the exact browser path post-flip):**
  dashboard / cockpit / dagster / git .fxhnt.ai ALL → **HTTP 200, tls=0 (cert chain valid)**. foxhunt still live.
- Fixes needed to get P2 green (all committed):
  1. **RBAC:** TS sidecar needs SA+Role+RoleBinding for get/create/update/patch on its ts-state Secret (added
     `tailscale-fxhnt-ingress` SA, mirrors tailscale-dagster). Without it: "missing get/update permission".
  2. **dagster NP ingress was stale on-cluster:** the 9396f53 edit (foxhunt→fxhnt-ingress label) was committed
     but the cluster still had `tailscale-gitlab-proxy` → dagster vhost hung. Re-applied dagster.yaml → fixed.
     LESSON: `git commit` ≠ `kubectl apply`; always re-apply after editing a live manifest.
  3. **gitea headless-service / CGNAT trap:** gitea-http & gitea-ssh are HEADLESS (ClusterIP:None) → DNS returns
     the gitea POD IP in 100.64.0.0/15, which the proxy's KERNEL-MODE tailscale sidecar (TS_USERSPACE=false)
     CANNOT reach (CGNAT overlap). FIX: added a non-headless **`gitea-clusterip`** Service (ns gitea, 10.32.x,
     selector app.kubernetes.io/instance=gitea+name=gitea, ports 3000+22); nginx+socat now target it. cockpit &
     dagster were fine because they have real 10.32.x ClusterIPs. (Kernel-mode TS reaches 10.32.x ClusterIPs but
     NOT 100.64.x pod IPs — verified both ways.)
- **Verify artifact:** `curl --resolve <host>:443:100.93.141.11 https://<host>/` from any tailnet node.
- PHASE-1 PREP COMPLETE. Ready to schedule the C1–C5 cutover window (see T8-CUTOVER-RUNBOOK.md). C4 DNS target
  = 100.93.141.11 for records dashboard/cockpit/dagster/git. foxhunt fully live; delete NOTHING.

## T8 PREP (2026-07-18) — DNS/ingress map + fixes (foxhunt still fully live)

### Gotchas hit this session
- **dagster tailscale sidecar CrashLoop (context deadline exceeded reaching kubernetes.default.svc).** TWO-STAGE
  root cause — the FIRST fix (06d12fd, `10.32.0.1/32:443`) was WRONG and did not hold (regressed to 2/3):
  * `kubernetes.default.svc` → ClusterIP `10.32.0.1:443`, BUT **kube-proxy DNATs it to the REAL apiserver
    endpoint `172.16.0.11:6443` BEFORE the NetworkPolicy is evaluated.** The NP sees dest `172.16.0.11:6443`,
    which is inside the NP's OTHER except entry `172.16.0.0/16` → blocked. So allowing the ClusterIP never matched.
  * CORRECT FIX: `ipBlock 172.16.0.11/32` port **6443** (the real endpoint, post-DNAT). apiserver endpoint is
    `kubectl -n default get endpoints kubernetes` = 172.16.0.11 (Kapsule fixed control-plane).
  * SIDE EFFECT this exposed: when the tailscale sidecar is NotReady, the whole dagster pod is NotReady →
    **dagster Service has NO endpoints → dagster.fxhnt.ai vhost would 502**. So this MUST be green before C4 DNS flip.
  * daemon+webserver themselves were always healthy. Cockpit's sidecar works only because cockpit has NO NP.
  * GENERAL LESSON for any pod with a tailscale sidecar + a restrictive egress NP on bizworx: must allow
    egress to 172.16.0.11:6443 (apiserver). The cronjob/rebalancer NPs do NOT have tailscale sidecars so they're
    unaffected — but if any future fund pod gets a TS sidecar under an NP, add this rule.
- **foxhunt API ACL lapsed AGAIN**: devbox egress IP drifted `51.158.111.225`→`51.158.125.1`. Foxhunt Kapsule
  admin API (`*.api.k8s.fr-par.scw.cloud:6443`) is IP-allowlisted; bizworx keeps working (tailscale-cgnat range).
  FIX: `scw k8s acl add cluster-id=34a1e3c4… acls.0.ip=<newIP>/32 acls.0.description=bizworx-devbox-vN` (APPENDS;
  verified all entries survive). Re-check `curl ifconfig.me` vs `scw k8s acl list` whenever foxhunt 6443 resets.

### THE fxhnt.ai DNS/ingress architecture (drives T8 DNS cutover)
- DNS zone = **Scaleway `fxhnt.ai`** (project c293eb98=foxhunt). ALL 13 A-records → **`100.95.225.27`** = tailnet
  node **`foxhunt-gitlab`** = pod **`tailscale-gitlab-proxy` (ns foxhunt, 4/4)** = nginx(vhosts)+2×socat+tailscale.
  **This pod DIES at foxhunt teardown (T12) → DNS MUST move off 100.95.225.27 regardless.**
- nginx vhost map (what each fxhnt.ai host fronts):
  - `git.fxhnt.ai` → gitea-web.foxhunt:3000  ✅FUND → **bizworx gitea (ns gitea), already mirrored**
  - `dashboard/cockpit.fxhnt.ai` → 100.81.150.18:80 (fxhnt-dashboard node→cockpit:8080)  ✅FUND → **bizworx cockpit**
  - `dagster.fxhnt.ai` → dagster.foxhunt:80  ✅FUND → **bizworx dagster**
  - grafana ❌ (operator: "grafana can go too" — drop, bizworx has own obs) · api(50051 legacy Rust gRPC) ❌
    · mail(stalwart)/chat(mattermost)/minio ❌legacy → all Phase-2 drop. ONLY git+dashboard/cockpit+dagster move.
- **CRITICAL SAFETY FACT:** the FUND PIPELINE never uses fxhnt.ai — git-sync=`gitea-ssh.gitea.svc.cluster.local:22`
  (in-cluster), cockpit/dagster serve over ClusterIP+own tailnet IP. fxhnt.ai is BROWSER/human access only.
  → the fund keeps running through the DNS flip; DNS cutover only affects operator browser access. Low-risk.
- Tailnet identities: live foxhunt `fxhnt-dashboard`=100.81.150.18, `fxhnt-dagster`=100.116.123.29. New bizworx
  cockpit joined as **`fxhnt-dashboard-1`=100.107.99.35** (`-1` because live one holds the name). Each bizworx pod
  serves TLS DIRECTLY via `tailscale serve` on its own MagicDNS cert (verified: cockpit `tailscale serve status`
  shows https://fxhnt-dashboard-1.tailff20e.ts.net → 127.0.0.1:8080). No bizworx proxy pod needed.

### DNS cutover DECISION (operator): "RENAME bizworx pods to take over the identity"
- At cutover: suspend/scale foxhunt fund pods → they RELEASE `fxhnt-dashboard`/`fxhnt-dagster` tailnet names →
  restart bizworx pods so they re-register WITHOUT the `-1` suffix (reclaim the names + get their IPs) → then
  repoint the Scaleway `fxhnt.ai` A-records (dashboard,cockpit,dagster,git) from the dying 100.95.225.27 to the
  reclaimed bizworx pod tailnet IPs. Records to change: `scw dns record` set on zone fxhnt.ai.
- TLS RESOLVED: old proxy served a cert-manager wildcard `*.fxhnt.ai` (ClusterIssuer letsencrypt-prod, Scaleway
  DNS-01 webhook). **bizworx already has the SAME stack** (cert-manager + scaleway-certmanager-webhook 1/1 +
  scaleway-api creds + letsencrypt-prod ClusterIssuer, already issuing LE certs in ~10 ns incl grafana-tls).
  → issuing `*.fxhnt.ai` on bizworx = just a Certificate resource. NO blocker.
- CHOSEN CUTOVER SHAPE (supersedes the naive pod-IP flip): build a **stripped fund-only nginx+tailscale proxy on
  bizworx** (`fxhnt-ingress`) fronting the 3 fund vhosts via ClusterIP (gitea-http.gitea:3000, fxhnt-dashboard
  .foxhunt:80, dagster.foxhunt:80) + socat SSH gitea-ssh.gitea:22, terminating the wildcard `*.fxhnt.ai` cert.
  Point the 4 fxhnt.ai A-records at THIS proxy's tailnet IP. Because it uses ClusterIP not pod tailnet names,
  the "reclaim fxhnt-dashboard identity" rename is NO LONGER on the critical path (optional cosmetic).
- **FULL EXECUTABLE RUNBOOK written: `docs/superpowers/plans/T8-CUTOVER-RUNBOOK.md`** (P1 wildcard cert →
  P2 build+verify proxy pre-flip → C1 freeze foxhunt → C2 delta dump/restore → C3 bizworx live+ib-gw sole session
  → C4 DNS flip → C5 arm ucits. Every step has VERIFY + ROLLBACK. Nothing destructive; foxhunt suspended-not-deleted.)
- Confirmed svc names for the proxy: gitea-http.gitea:3000 (headless), gitea-ssh.gitea:22 (headless),
  fxhnt-dashboard.foxhunt:80, dagster.foxhunt:80, ib-gateway.foxhunt:4002/4004.

## T7 update 3 (2026-07-18) — cronjobs
- Cross-ns gitea-NP fix applied to ALL 15 jobs/fxhnt-*.yaml (namespaceSelector kubernetes.io/metadata.name=gitea).
- 7 CronJobs applied to bizworx ns foxhunt, ALL suspend=true (verified): bybit-rebalancer,
  compare-measured-precompute, factory, ibkr-rebalancer, ucits-rebalancer, ucits-volume-ibkr, vrp-rebalancer.
  (real-money safety: none run parallel to the live foxhunt ucits-rebalancer.)
- The 9 one-time Jobs (opra-backfill/warehouse-migrate/paper-backfill/bybit-*-job/deribit) NOT applied (they
  run immediately); apply on demand post-cutover with the NP fix already in place.
- REMAINING before T8: one ucits dry-run (needs SOLE IBKR session — briefly scale foxhunt ib-gateway to 0,
  or do it as the first act of cutover) + optional dagster asset-graph run. Then T8 cutover.

## T9 VERIFY (2026-07-18) — cockpit + gate intact against MIGRATED bizworx DB ✅
- **Local cockpit** (`./scripts/dev-cockpit-local.sh`, current-context=bizworx → serves the MIGRATED DB read-only
  on http://127.0.0.1:8801). ENV GOTCHA fixed: the script's `uv run` needed `--extra web --extra pg` (uvicorn in
  `web`, psycopg in `pg`); a bare `uv run` fails ModuleNotFoundError. Patched dev-cockpit-local.sh.
- **NUMBERS CROSS-CHECKED (read the numbers, per feedback_verify_cockpit_locally_not_prod):**
  - /paper IBKR card = **$1,029,015** == ibkr_account_nav.nlv latest (2026-07-17 = 1029014.63). "not connected"
    ABSENT (the card that silently degraded in a prior session renders the REAL DU account here).
  - bybit Sh 2.11 == backtest_summary bybit_4edge 2.1067; unlock Sh 1.03 == 1.0258. All real, DB-backed.
  - /, /strategy/multistrat, /strategy/xsfunding, /paper all HTTP 200 with real content.
  - CAVEAT: Playwright MCP browser not connected this session → verified by rendering real HTML + reading
    numbers (not pixels). Local cockpit left running on :8801 for the operator to click through.
- **T9b forward tracks INTACT (no spurious re-inception from migration):** forward_anchor t0 dates are ORIGINAL
  historical inceptions (sixtyforty 2026-06-15, unlock 2026-06-20, positioning 2026-07-06, bybit_4edge/multistrat/
  xsfunding 2026-07-07, vrp/multistrat_exec 2026-07-14) — NOT today (2026-07-18). definition_hash/version chains +
  archived_at + record_mode all preserved. DB-anchor-SSOT survived byte-for-byte; recon gate will recompute correctly.
- **STILL PENDING:** T9 real-money confirm = watch the first armed fxhnt-ucits-rebalancer run at 09:00 UTC (tomorrow).
  Then T10 observe → T11 triage → T12 teardown. ROTATE the pasted tailscale key. DELETE NOTHING.

## T9 VISUAL BROWSER PASS (2026-07-18) — real Chrome via Playwright MCP ✅
- Playwright MCP now available (plugins reloaded); installed the `chrome` channel (Google Chrome 150 at
  /opt/google/chrome/chrome — the MCP hardcodes that path, bundled chromium alone isn't enough).
- Drove the LOCAL cockpit (http://127.0.0.1:8801, bizworx migrated DB) in a REAL browser — full render, all data:
  - Overview: Bybit 4-edge $832,534 +738% Sh 1.74 maxDD -13%, deploy $75k/75%, gate 11/21 WAIT; 4 edges
    (trend Sh0.72 / unlock Sh1.03 / xsfunding Sh3.62 / positioning Sh2.55); forward-tracks table w/ correct
    gates: sixtyforty 23/20 GO, unlock 26/14 PASS, multistrat 9/20 WAIT + real IBKR subtrack "real -0.77% ·
    9 trades · 5 holdings".
  - /paper: IBKR paper account **$1,029,015 · 1 strategy trading · 5 holdings · 0.9% behind plan** (the real
    DU account — NOT "not connected"). Bybit book $832,534.
  - Full-page screenshot saved (.playwright-mcp/cockpit-overview-bizworx.png). 1 non-material console error.
- Matches DB cross-checks exactly. T9 cockpit verification COMPLETE (visual + numeric).

## FOXHUNT SHUTDOWN (2026-07-18 ~20:55 UTC) — compute stopped, cluster+PVCs INTACT (not destroyed)
Operator decision after verifying migration complete: "shut down everything now, we can spin up if we need to
reboot." Verified first: (1) bizworx has ZERO runtime dep on foxhunt (its postgres/gitea/git-sync/registry are
all bizworx-local — the `postgres.foxhunt.svc` refs are bizworx's OWN ns foxhunt, different cluster); (2) the
pre-cutover state is durably backed up OFF-cluster: `~/Work/fxhnt-cutover-backup/fxhnt_precutover_2026-07-18.dump`
(48M, 805 objects, all tables verified via pg_restore --list in the bizworx pod); (3) fund live+serving on
bizworx (cockpit 2/2, dagster 3/3, ingress 3/3, ib-gateway 1/1; dashboard.fxhnt.ai HTTP 200).
- Scaled ALL foxhunt-ns deploys + statefulsets to 0 (incl postgres/gitea/tailscale-gitlab-proxy = the
  instant-rollback path — operator accepted dropping it). Also scaled cert-manager + tailscale-subnet-router to 0.
- Suspended ALL foxhunt cronjobs (fund + infra: postgres-backup/pvc-autoscaler/billing-scraper). Deleted leftover
  Completed/Error Job pods so they don't pin nodes.
- Pool `platform` (8b520ebb…): autoscaling=true, **min-size=0** (was 1). Can't force size=0 while autoscaling on;
  the Kapsule autoscaler drains nodes to 0 once only DaemonSets+kube-system remain (~10-15min scale-down delay).
  Monitoring the 3 dev1_l nodes → 0. `ci-compile-cpu` pool already 0.
- **ROLLBACK now = restore the .dump into a fresh pg (few min), NOT an instant DNS flip.** foxhunt cluster + ALL
  PVCs still EXIST (training-data 500Gi, minio 150Gi, postgres 100Gi, gitlab-gitaly 50Gi, gitea valkey, etc.) —
  scaling back up reschedules pods on data intact. terraform destroy (T12, irreversible) still DEFERRED.
- REVERSIBILITY: `scw k8s pool update 8b520ebb… min-size=1 size=1` + scale deploys back up → foxhunt returns.
- STILL PENDING: tomorrow 09:00 UTC bizworx ucits-rebalancer run (real-money confirm); then T12 terraform destroy
  (with operator confirm). ROTATE the pasted tailscale key.

## DATABENTO / .dbn DATA MIGRATION (2026-07-18 ~21:10 UTC) — operator caught this before teardown
Found databento .dbn data on foxhunt PVCs NOT covered by the Postgres migration. Split cleanly:
- **Fund-relevant (migrated):** `/data/surfer/*.dbn` — 22 futures .dbn (ES/NQ/GC/CL/NG/6E/ZC/ZS... universe,
  updated Jul 12-13) + *_state.json. READ BY the fund's tsmom/carry RESEARCH dagster assets
  (tsmom_research.py, carry_research.py, assets.py:247 read_front_arrays `$FXHNT_SURFER_DATA_DIR/<sym>.dbn`).
  bizworx had only an EMPTY fxhnt-surfer-data PVC → research would've failed. **MIGRATED**: tar (33M gz) foxhunt
  inspector → devbox → bizworx writer pod → extracted into fxhnt-surfer-data. Verified from the dagster pod's
  OWN view: 22 .dbn at /data/surfer/*.dbn + state JSONs. (Layout note: PVC mounted at /data on dagster,
  FXHNT_SURFER_DATA_DIR=/data/surfer → files live at PVC-root/surfer/.)
- **Archived Rust-ML (NOT fund-used): ~171GB .dbn** on training-data-pvc (59 files, 151GB: ES.FUT MBP10/trades
  quarterly) + test-data-pvc (21 files, 20GB). Operator decision (option 3): push to PRIVATE Scaleway Object
  Storage, NOT a bizworx PVC, and "cannot move 175GB over a tunnel" → **upload INTRA-SCALEWAY** (foxhunt pod →
  s3.fr-par.scw.cloud, both fr-par, no tunnel/devbox hop).
  - Created private bucket **`fxhnt-dbn-archive`** (fr-par, bizworx project 96d0717d, versioning off, ACL empty).
  - S3 creds → foxhunt secret `dbn-archive-s3` (from scw config, never printed).
  - `dbn-uploader` pod (rclone/rclone image) mounts training+test PVCs RO; `rclone copy --include "*.dbn*"
    --transfers 8` → scw:fxhnt-dbn-archive/{training,test}/. Connectivity tested (rclone lsd rc=0). 171.4GB / 80 files.
  - Upload running in background in-pod (logs /tmp/rclone-*.log, /tmp/rclone-done sentinel). Monitored.
- To restore archived data later: `rclone copy scw:fxhnt-dbn-archive/... <dest>` (creds in bizworx-prod-runtime
  key or scw config). Kept for the archived Rust codebase per [[project_rust_infra_decommissioned_phase1]].
- NOTE: I bumped platform pool min-size back to 1 (from 0) so the inspector/uploader pods can schedule; after the
  upload completes + surfer verified, set min-size=0 again to resume the drain. Cleanup pods: data-inspector, dbn-uploader.

## .dbn ARCHIVE — PIVOTED to block snapshots (rclone upload was a dead end) ✅
- The rclone S3 upload from the STARVED foxhunt cluster STALLED: dev1_l node egress + slow scw-bssd PVC read
  (20MB/s) → 76KiB/s, ETA 4 DAYS, byte count frozen. Killed it.
- **RIGHT SOLUTION: Scaleway block-volume SNAPSHOTS** (instant, storage-layer, no tunnel, restorable, cheap).
  The .dbn PVCs are scw block volumes → snapshotted directly:
  - `fxhnt-training-dbn-archive-2026-07-18` (snap 564a4ccb-ee69-4514-8a64-9932222a0334, 537GB vol = the 151GB
    training .dbn) — ParentVol cb327a05 (training-data-pvc).
  - `fxhnt-test-dbn-archive-2026-07-18` (snap 5ba0049f-799a-4694-b7a0-0192cf0cda11, 50GB = the 20GB test .dbn)
    — ParentVol dc144e62 (test-data-pvc).
  - BOTH `available` in ~20s, in the **bizworx project 96d0717d** (survive foxhunt cluster teardown), zone fr-par-2.
- **RESTORE:** `scw block volume create from-snapshot-id=<snap> zone=fr-par-2` → attach to a pod → read .dbn.
- Cleaned up: dbn-uploader + data-inspector pods deleted, dbn-archive-s3 secret deleted.
- CORRECTION (2026-07-18 later): there is NO stray junk. The earlier "132MB / 23 objects" were the FIRST partial
  `.dbn` transfers under `training/` — the resumed upload continues them (rclone skips already-present). Verified:
  bucket holds ONLY legit `training/*.dbn*` (singletest/ + speedtest/ test files were deleted during testing = 0
  objects). **DO NOT wipe/delete `fxhnt-dbn-archive` — it IS the active archive we're building.**
- Node drain RESUMED (platform pool min-size 1->0; no blocking pods). Autoscaler drops the last node to 0.

## STATE SUMMARY (2026-07-18 ~21:15 UTC)
- fxhnt.ai fund LIVE on bizworx (T8+T9 done, cockpit verified in real Chrome). foxhunt compute → 0 (draining).
- Data fully preserved: pg dump (~/Work/fxhnt-cutover-backup), surfer .dbn (migrated to bizworx PVC + verified
  from dagster pod), archived 171GB .dbn (2 block snapshots in bizworx project). foxhunt PVCs still exist too.
- PENDING: tomorrow 09:00 UTC ucits run (real-money confirm); then T12 terraform destroy (operator confirm) —
  which also removes the source PVCs, but snapshots + dump + bizworx-live cover everything. ROTATE tailscale key.
  Cleanup the stray fxhnt-dbn-archive bucket.

## .dbn S3 COPY (2026-07-18 ~21:53 UTC) — operator: also copy compressed .dbn.zst to S3, KEEP snapshots
The .dbn.zst are already compressed → S3 copy is a good browsable archive alongside the snapshots (belt+braces).
- Measured real S3 throughput: ~10-17 MB/s from these small VMs (dev1_l/gp1_xs) — NOT compression-limited, it's
  node network egress. Single-file test: 537MiB in 52s = 10MB/s, PVC read 217MB/s (fine). The earlier "0 B/s
  stall" was rclone stats not updating until a big multipart completed + too many concurrent 64M-chunk streams
  choking the node. FIX: --transfers 2 --s3-chunk-size 32M --s3-upload-concurrency 2 (node-friendly) → moving.
- Restore-snapshot-to-volume path abandoned (Kapsule CSI import of a raw scw volume = rabbit hole). Instead
  uploading directly from a foxhunt pod (dbn-uploader) that mounts training+test PVCs RO — data still live there.
- Upload RUNNING in background: rclone copy .dbn* → scw:fxhnt-dbn-archive/{training,test}/. 137GiB, ETA ~6-7h at
  ~6MB/s. Persistent Monitor watching for completion. Sentinel /tmp/rclone-done in the pod.
- KEEP THE SNAPSHOTS (operator) — validated safety net; only after S3 upload verified (object count + sizes) do
  we consider anything. Both archives coexist.
- foxhunt platform pool held at min-size=1 for the uploader; set min-size=0 to resume drain AFTER upload done.
- RESUME/CHECK: `kubectl --context $FOX -n foxhunt exec dbn-uploader -- sh -c 'cat /tmp/rclone-done; tail -1 /tmp/rclone-training.log'`
  and `... rclone size scw:fxhnt-dbn-archive`. If pod died: re-create it (manifest in git history commit 2438294-ish) — rclone copy is resumable/idempotent.

## .dbn S3 COPY COMPLETE + VERIFIED (2026-07-19 ~02:41 UTC) ✅
- Upload finished: **80 objects / 171.393 GiB** in `fxhnt-dbn-archive` (private, fr-par, bizworx project).
- VERIFIED byte-for-byte vs source PVCs: `rclone check --size-only --one-way`:
  - training: **0 differences, 59 matching files** ✅
  - test: **0 differences, 21 matching files** ✅
  - source .dbn* count = 80 = bucket object count = exact.
- **TWO independent verified archives now exist** (operator: keep snapshots): (1) S3 object storage
  `s3://fxhnt-dbn-archive/{training,test}/*.dbn.zst` (browsable, per-file restore), (2) block snapshots
  fxhnt-training/test-dbn-archive-2026-07-18 (564a4ccb / 5ba0049f, bizworx project). Both safe post-teardown.
- Cleaned up: dbn-uploader + s3-speedtest pods, dbn-archive-s3 secrets (both clusters). Node drain RESUMED
  (platform pool min-size 1->0; no blocking pods) → autoscaler drops the last dev1_l to 0. foxhunt compute → ~$0.
- REMAINING (next session / operator): tomorrow 09:00 UTC first armed ucits run (real-money confirm); T12
  terraform destroy foxhunt Kapsule (removes source PVCs — both .dbn archives + pg dump + bizworx-live cover it);
  rotate the pasted tailscale key. Restore .dbn: `rclone copy scw:fxhnt-dbn-archive/... <dest>` OR
  `scw block volume create from-snapshot.snapshot-id=<snap> perf-iops=5000 zone=fr-par-2`.

## SCHEDULING CONSISTENCY — ALL-DAGSTER K8sRunLauncher UPGRADE DONE (2026-07-19) ✅
Operator: "go all-Dagster properly + build the image right." Executed the 5-task plan
(docs/superpowers/plans/2026-07-19-production-scheduling-consistency.md). Result = consistent single-orchestrator
ETL, execution isolated in crons, nothing suspended-by-accident, VRP gone, documented.
- DIAGNOSIS (corrected): Dagster ETL was NEVER broken (23:30 ran SUCCESS, forward_nav fresh). The stale
  bybit_sleeve_ret was the SUSPENDED compare-measured cron (exists because the compute needs 6-8Gi, OOM-killed
  the 4Gi daemon). Fix = K8sRunLauncher, not naive daemon assets.
- BUILT: dagster-k8s dep added; image built via in-cluster **Kaniko** (devbox podman blocked by sandbox mount
  perms) → `fxhnt-cockpit:dagster-k8s-25e3d7c`, verified + retagged `:latest` (skopeo Job).
- RBAC: tailscale-dagster SA can create Jobs/pods/events. K8sRunLauncher configured (launched Jobs git-sync
  master, no PVC, fund secrets). New asset `bybit_book_precompute` (op_tags 6-8Gi) wraps `_persist_bybit_book`
  (moved to application/bybit_book_persist.py). MERGED branch→master (git-sync pulls master).
- VERIFIED (the money shot): triggered combined_book_forward_job → launched Job dagster-run-* → materialized all
  14 assets incl the heavy precompute → **daemon 0 restarts** → bybit_sleeve_ret + backtest_summary + forward_nav
  ALL fresh 2026-07-19. Run SUCCESS 6m54s.
- CRONS (final): DELETED compare-measured (→ asset) + vrp (shelved). ARMED factory + ucits-rebalancer +
  ucits-volume-ibkr. SUSPENDED bybit-rebalancer (no creds) + ibkr-rebalancer (PRIIPs). Manifests updated.
- Memory: [[reference_fxhnt_dagster_k8srunlauncher_scheduling]] created; VRP files + MEMORY.md updated.
- NOTE: the fund now runs off MASTER (28-commit consolidation branch merged 06156bd; +cron changes 1ecf398).