fxhnt/docs/superpowers/plans/CONSOLIDATION-PROGRESS.md

Foxhunt→Bizworx Consolidation — Live Progress (resume anchor)

🚦 T8 CUTOVER EXECUTED (2026-07-18 ~20:15 UTC) — fxhnt.ai NOW SERVES FROM BIZWORX ✅

The fund has cut over. foxhunt is FROZEN but INTACT (rollback available). DELETE NOTHING until T10 observation passes.

• C1 (freeze foxhunt): suspended fund cronjobs (ucits-rebalancer, ucits-volume-ibkr, compare-measured-precompute)
  • scaled foxhunt dagster/fxhnt-dashboard/ib-gateway → 0/0. Infra cronjobs (postgres-backup/pvc-autoscaler/ billing-scraper) LEFT RUNNING (they die at teardown, not now). PASS.
• C2 (delta DB migrate): pg_dump -Fc fxhnt (47.8M) from frozen foxhunt → restore into bizworx. GOTCHA: pg_restore --clean --if-exists over the existing rehearsal copy FAILED (dropped _timescaledb_internal schema mid-restore → 1136 errors, DB stuck restoring:on). FIX (rehearsal-proven path): DROP+CREATE empty fxhnt DB, then pre_restore → pg_restore (NO --clean) → post_restore = exit 0, 0 errors. EVAL C2 PASS: all counts match frozen source (forward_nav 106, backtest_summary 6, bybit_sleeve_ret 7023, xsp_option_bars 814373, exec_fills 9, ibkr_account_nav 4, forward_anchor 20; max_anchor_t0 2026-07-14, max_fwd_nav 2026-07-17; 37 tables/3 hypertables). NOTE: counts were IDENTICAL to the rehearsal (fund already suspended during parallel work → zero real delta). LESSON: NEVER pg_restore --clean over a live timescaledb DB; always restore into a fresh empty DB.
• C3 (bizworx fund live): scaled dagster/cockpit/ib-gateway→1 + rollout-restart cockpit&dagster (fresh DB conns). EVAL C3 PASS: cockpit serves real migrated data over the proxy (/ + /strategy/multistrat + /xsfunding = 200, shows PASS/WAIT/Sharpe/multistrat/xsfunding); ib-gateway 1/1 STABLE (~12min, foxhunt released IBKR session); dagster 3/3, ingress 3/3, postgres 1/1.
• C4 (DNS flip): Scaleway fxhnt.ai records dashboard/cockpit/dagster/git A → 100.93.141.11 (bizworx fxhnt-ingress), TTL 60 (was 100.95.225.27 foxhunt proxy). SYNTAX: scw dns record set dns-zone=fxhnt.ai name=<n> type=A ttl=60 values.0=100.93.141.11 (values.0=, NOT data=). EVAL C4 PASS over REAL public DNS: dashboard/git/dagster.fxhnt.ai → 100.93.141.11, HTTP 200, tls_verify=0. Content confirmed: git=bizworx Gitea, dashboard=cockpit "Overview · fxhnt" w/ real data, SSH git ls-remote ssh://git@git.fxhnt.ai/gitadmin/fxhnt.git returns real refs (HEAD d66d1132) via socat path. (SSH hostkey-changed warning = correct, it's bizworx now.)
• C5 (arm fund): un-suspended ONLY fxhnt-ucits-rebalancer on bizworx (schedule 0 9 * * 1-5 = 09:00 UTC wkdays; next run tomorrow AM). US(ibkr)/bybit/vrp/factory/ucits-volume all stay SUSPENDED. EVAL C5 (first armed run green) is OBSERVABLE at next 09:00 UTC — deps verified (DB+ib-gateway+git-sync all live).
• ROLLBACK (still trivial until T11): scw dns record set ... values.0=100.95.225.27 (back to foxhunt proxy)
  • kubectl --context $FOX -n foxhunt scale deploy/dagster deploy/fxhnt-dashboard deploy/ib-gateway --replicas=1
  • un-suspend foxhunt fund cronjobs. foxhunt Postgres still holds pre-cutover state.
• NEXT: T9 verify (cockpit LOCAL browser + prod, gate/forward-track intact) → T10 observation window (watch the first ucits run + a day) → T11 Phase-2 triage → T12 teardown. ROTATE the pasted tailscale key. DELETE NOTHING.

Branch chore/foxhunt-bizworx-consolidation. Real date 2026-07-18. Plan: docs/superpowers/plans/2026-07-18-foxhunt-to-bizworx-consolidation.md. Contexts: FOX=foxhunt-34a1e3c4-…, BIZ=bizworx-prod-kapsule-05e1dae3-… (both in ~/.kube/config).

Done + verified

• Access restored: appended this box's egress 51.158.111.225/32 to the foxhunt Kapsule API ACL (scw k8s acl, entry bizworx-devbox-v4) + scw k8s kubeconfig install → foxhunt context works.
• T1 ns foxhunt created on bizworx; StorageClass sbs-default matches; source = PG16.13/tsdb2.25.1.
• T2 6 fund secrets copied FOX→BIZ ns foxhunt (db-credentials, databento, tiingo-api, ibkr-credentials, argo-git-ssh-key, scw-registry). NOT copied: bybit-testnet (absent/suspended), tailscale-auth (deferred).
• T3 gitea consolidated → bizworx gitea (ns gitea): gitadmin/fxhnt (8 heads/0 tags, master d66d113)
  • gitadmin/foxhunt (13 heads/105 tags, main 8213f29a5), refs verified identical. foxhunt (625MB) done via direct pack-copy (HTTP push + server-side index-pack both failed on size/OOM). git.fxhnt.ai STILL points at foxhunt (DNS re-point is T8). Caveat: foxhunt archive's gitea DB metadata may lag (disk-correct).
• T4 manifests re-targeted (committed): git-sync gitea-sshd.foxhunt→gitea-ssh.gitea.svc.cluster.local:22; nodeSelector→large; ib-gateway pull-secret removed (public ghcr image). Server-side dry-run clean.
• T5/T6 Postgres on bizworx PINNED to timescale/timescaledb:2.25.1-pg16 (latest=2.28.3 broke catalog restore — REHEARSAL caught it). Rehearsal restore of fxhnt DB: exit 0, 0 errors, all row counts match (forward_nav 106, backtest_summary 6, bybit_sleeve_ret 7023, xsp_option_bars 814373, exec_fills 9, ibkr_account_nav 4), 37/37 tables, 3/3 hypertables. bizworx fxhnt DB currently holds this rehearsal copy.

T7 IN PROGRESS (parallel verify — foxhunt still fully live)

• 7.1 DONE — argo-git-ssh-key deploy-key registered read-only on bizworx gitadmin/fxhnt (id 2); verified git ls-remote via the key = 8 refs.
• cockpit DONE — fxhnt-dashboard deployed on bizworx; git-sync OK (cloned d66d113 from bizworx gitea); cockpit container Running+ready, renders migrated data (overview + /strategy/xsfunding + /strategy/multistrat all HTTP OK with real WAIT/PASS gates). Verified from INSIDE the container (port-forward blocked — see below).
• BLOCKER / decision pending — tailscale: dagster + cockpit pods have a tailscale sidecar needing tailscale-auth (not copied) → sidecar CreateContainerConfigError → pod phase Pending → port-forward refuses. Cockpit itself is fine. DECIDE: A fresh reusable tailscale auth-key (bizworx pods join as extra nodes alongside live foxhunt ones; removed at cutover) OR B disable sidecar for the parallel phase, verify via port-forward, enable tailscale at T8 cutover with real hostnames.

T7 REMAINING

• ib-gateway → apply + verify IBKR paper connect (no 2FA/IP-limit per operator) + whatIf.
• dagster → apply (check its NetworkPolicy allows egress to ns gitea for git-sync — cross-ns) + trigger nightly asset graph green.
• CronJobs → apply all SUSPENDED; then one fxhnt-ucits-rebalancer dry-run (EU hours).
• NP note: fund NPs (dagster, cronjobs) allow egress to gitea in-ns; on bizworx gitea is ns gitea → may need namespaceSelector. (cockpit had no NP → git-sync worked.)

HARD RULE (operator): verify-first, per-step eval, DELETE NOTHING until Phase-1 verified + observation

window (T10) passes. foxhunt stays live/intact through T8, suspended-but-intact until T12 teardown. Rollback until T11 = re-point DNS + un-suspend foxhunt.

T7 update (2026-07-18, later)

• tailscale-auth secret created on bizworx (operator gave a fresh reusable key — OPTION A; key was pasted in transcript, ROTATE after cutover). cockpit pod now 2/2 Running (cockpit + tailscale sidecar joined).
• ib-gateway deployed: IBC logs in to DU9600528 (Simulated Trading) — IBKR paper connect WORKS (no 2FA/IP). ⚠️ pod had 2 restarts during IBC startup churn — WATCH it stabilises to 1/1 ready; verify API port 4003/4004
  • a whatIf before arming any rebalancer.
• dagster was Pending on missing PVC fxhnt-surfer-data (dropped from migration; manifest still mounts it) → created empty 5Gi PVC (sbs-default); dagster now scheduling (git-sync init). unlock_calendar.json rebuilds.
• REMAINING T7: confirm ib-gateway 1/1 + whatIf; confirm dagster git-sync (cross-ns NP to gitea) + trigger nightly asset graph green; apply CronJobs SUSPENDED; one fxhnt-ucits-rebalancer dry-run (EU hours).
• Then T8 cutover (freeze foxhunt, delta restore, DNS re-point git.fxhnt.ai + dashboard.fxhnt.ai) → T9 verify → T10 observe → T11 triage → T12 teardown. foxhunt still fully live; delete NOTHING yet.

T7 update 2 (2026-07-18)

• dagster FIXED + 3/3 Running: (1) created dagster DB on bizworx postgres (fresh dagster, not migrated); (2) dagster NP gitea-egress needed a cross-ns namespaceSelector {kubernetes.io/metadata.name: gitea} (committed in dagster.yaml) → git-sync from bizworx gitea now OK. webserver+daemon+tailscale all ready.
• ib-gateway: connects to DU9600528 (IBC login OK) but exits 0 after ~6min = IBKR single-session conflict with the STILL-LIVE foxhunt ib-gateway. Connect verified; stable run only after foxhunt-suspend (T8). NOT a bug.
• T7 CORE VERIFIED: postgres+cockpit+dagster all up on bizworx against migrated data.
• REMAINING T7: (a) apply the SAME cross-ns gitea-NP fix to every cronjob NP (jobs/fxhnt-*.yaml) BEFORE they run (else git-sync blocked like dagster was); (b) apply CronJobs SUSPENDED; (c) one ucits dry-run (needs ib-gateway sole-session → do at/after cutover, or briefly scale foxhunt ib-gateway to 0); (d) optional dagster asset-graph run. Then T8 cutover.
• ALSO for cutover: the tailscale bizworx pods joined as extra nodes (fresh key); at T8 suspend foxhunt so the real hostnames + IBKR session move cleanly. ROTATE the pasted tailscale key post-cutover.

P1 EXECUTION (2026-07-18) — wildcard cert + the cross-project DNS-01 IAM fix

• Applied infra/k8s/services/fxhnt-wildcard-cert.yaml (Certificate fxhnt.ai + *.fxhnt.ai, ClusterIssuer letsencrypt-prod, secret fxhnt-wildcard-tls, ns foxhunt) on BIZ.
• DNS-01 FAILED: 403 Forbidden: domain not found. Root cause: fxhnt.ai DNS zone lives in the foxhunt project c293eb98; the bizworx cert-manager key (SCWW9261…, app bizworx-prod-runtime 48418a71) has DomainsDNSFullAccess but SCOPED to the bizworx project 96d0717d only → can't touch fxhnt.ai's zone.
• DEAD END (operator's first pick "move the domain to bizworx project"): Scaleway BLOCKS it. fxhnt.ai is an EXTERNAL domain (scw domain domain list → IS EXTERNAL=true, REGISTRAR=EXTERNAL). scw dns zone update … project-id=… on the apex → Root zone can't be updated; on a scratch sub-zone → Subdomain not allowed; scw domain domain has NO move-project subcommand. Verified via a throwaway _cutovertest.fxhnt.ai sub-zone (created+move-attempt+deleted). So the domain CANNOT be reassigned to the bizworx project via API.
• ACTUAL FIX (operator's 2nd pick, applied): add an IAM rule. scw iam rule create policy-id=8573ce25-… permission-set-names.0=DomainsDNSFullAccess project-ids.0=c293eb98-… → policy bizworx-prod-runtime-scoped now has 2 rules (orig 6-perms@96d0717d untouched + new DNS@c293eb98). This lets the existing bizworx cert-manager key write the DNS-01 TXT on fxhnt.ai. Domain STAYS external in c293eb98; only IAM changed. NOTE for teardown: this rule references the foxhunt project — harmless to keep, or tighten later; if the foxhunt PROJECT (not just cluster) is ever deleted, the fxhnt.ai domain/zone must first move registrar-side or certs break. (Cluster teardown ≠ project deletion; the DNS zone + domain survive T12.)
• Forced cert-manager retry (deleted order+CR) after the IAM change → 403 GONE, challenges progressing normally (apex fxhnt.ai challenge = valid/"Successfully authorized domain"; wildcard *.fxhnt.ai challenge finishing its DNS-01 propagation). TXT is being written to the fxhnt.ai zone by cert-manager (verified via scw dns record list dns-zone=fxhnt.ai | grep acme). P1 is SUCCEEDING — cert will go READY shortly. If resuming and cert not yet READY: kubectl --context $BIZ -n foxhunt get certificate fxhnt-wildcard -o wide; if stuck >15min, kubectl delete order,challenge,certificaterequest --all -n foxhunt to force a clean retry (IAM rule is permanent so it'll succeed).

P2 DRAFTED (2026-07-18) — fxhnt-ingress proxy (NOT applied yet; waiting on P1 cert READY)

• infra/k8s/services/fxhnt-ingress-proxy.yaml (committed ce1c978, draft): nginx (wildcard TLS via fxhnt-wildcard-tls) + socat (git SSH→gitea-ssh.gitea:22) + tailscale sidecar (TS_HOSTNAME=fxhnt-ingress). Fronts 3 fund vhosts BY CLUSTERIP: git→gitea-http.gitea:3000, dashboard/cockpit→fxhnt-dashboard.foxhunt:80, dagster→dagster.foxhunt:80. Has its own NP (egress to those svcs + apiserver 172.16.0.11:6443 for TS + DERP). Deps satisfied: tailscale-auth secret present; dagster NP ingress already allows fxhnt-ingress label.

P1+P2 COMPLETE (2026-07-18) — bizworx fxhnt.ai front verified, NO DNS change yet ✅

• P1 DONE: fxhnt-wildcard cert READY (LE, SAN=*.fxhnt.ai + fxhnt.ai). IAM cross-project fix held.
• P2 DONE: fxhnt-ingress proxy applied + 3/3 Running. Tailnet identity = fxhnt-ingress @ 100.93.141.11 (clean, no -1 suffix). THIS IS THE C4 DNS TARGET.
• EVAL P2 PASS (from devbox over the real tailnet → 100.93.141.11, the exact browser path post-flip): dashboard / cockpit / dagster / git .fxhnt.ai ALL → HTTP 200, tls=0 (cert chain valid). foxhunt still live.
• Fixes needed to get P2 green (all committed):
  1. RBAC: TS sidecar needs SA+Role+RoleBinding for get/create/update/patch on its ts-state Secret (added tailscale-fxhnt-ingress SA, mirrors tailscale-dagster). Without it: "missing get/update permission".
  2. dagster NP ingress was stale on-cluster: the 9396f53 edit (foxhunt→fxhnt-ingress label) was committed but the cluster still had tailscale-gitlab-proxy → dagster vhost hung. Re-applied dagster.yaml → fixed. LESSON: git commit ≠ kubectl apply; always re-apply after editing a live manifest.
  3. gitea headless-service / CGNAT trap: gitea-http & gitea-ssh are HEADLESS (ClusterIP:None) → DNS returns the gitea POD IP in 100.64.0.0/15, which the proxy's KERNEL-MODE tailscale sidecar (TS_USERSPACE=false) CANNOT reach (CGNAT overlap). FIX: added a non-headless gitea-clusterip Service (ns gitea, 10.32.x, selector app.kubernetes.io/instance=gitea+name=gitea, ports 3000+22); nginx+socat now target it. cockpit & dagster were fine because they have real 10.32.x ClusterIPs. (Kernel-mode TS reaches 10.32.x ClusterIPs but NOT 100.64.x pod IPs — verified both ways.)
• Verify artifact: curl --resolve <host>:443:100.93.141.11 https://<host>/ from any tailnet node.
• PHASE-1 PREP COMPLETE. Ready to schedule the C1–C5 cutover window (see T8-CUTOVER-RUNBOOK.md). C4 DNS target = 100.93.141.11 for records dashboard/cockpit/dagster/git. foxhunt fully live; delete NOTHING.

T8 PREP (2026-07-18) — DNS/ingress map + fixes (foxhunt still fully live)

Gotchas hit this session

• dagster tailscale sidecar CrashLoop (context deadline exceeded reaching kubernetes.default.svc). TWO-STAGE root cause — the FIRST fix (06d12fd, 10.32.0.1/32:443) was WRONG and did not hold (regressed to 2/3):
  • kubernetes.default.svc → ClusterIP 10.32.0.1:443, BUT kube-proxy DNATs it to the REAL apiserver endpoint 172.16.0.11:6443 BEFORE the NetworkPolicy is evaluated. The NP sees dest 172.16.0.11:6443, which is inside the NP's OTHER except entry 172.16.0.0/16 → blocked. So allowing the ClusterIP never matched.
  • CORRECT FIX: ipBlock 172.16.0.11/32 port 6443 (the real endpoint, post-DNAT). apiserver endpoint is kubectl -n default get endpoints kubernetes = 172.16.0.11 (Kapsule fixed control-plane).
  • SIDE EFFECT this exposed: when the tailscale sidecar is NotReady, the whole dagster pod is NotReady → dagster Service has NO endpoints → dagster.fxhnt.ai vhost would 502. So this MUST be green before C4 DNS flip.
  • daemon+webserver themselves were always healthy. Cockpit's sidecar works only because cockpit has NO NP.
  • GENERAL LESSON for any pod with a tailscale sidecar + a restrictive egress NP on bizworx: must allow egress to 172.16.0.11:6443 (apiserver). The cronjob/rebalancer NPs do NOT have tailscale sidecars so they're unaffected — but if any future fund pod gets a TS sidecar under an NP, add this rule.
• foxhunt API ACL lapsed AGAIN: devbox egress IP drifted 51.158.111.225→51.158.125.1. Foxhunt Kapsule admin API (*.api.k8s.fr-par.scw.cloud:6443) is IP-allowlisted; bizworx keeps working (tailscale-cgnat range). FIX: scw k8s acl add cluster-id=34a1e3c4… acls.0.ip=<newIP>/32 acls.0.description=bizworx-devbox-vN (APPENDS; verified all entries survive). Re-check curl ifconfig.me vs scw k8s acl list whenever foxhunt 6443 resets.

THE fxhnt.ai DNS/ingress architecture (drives T8 DNS cutover)

• DNS zone = Scaleway fxhnt.ai (project c293eb98=foxhunt). ALL 13 A-records → 100.95.225.27 = tailnet node foxhunt-gitlab = pod tailscale-gitlab-proxy (ns foxhunt, 4/4) = nginx(vhosts)+2×socat+tailscale. This pod DIES at foxhunt teardown (T12) → DNS MUST move off 100.95.225.27 regardless.
• nginx vhost map (what each fxhnt.ai host fronts):
  • git.fxhnt.ai → gitea-web.foxhunt:3000 ✅FUND → bizworx gitea (ns gitea), already mirrored
  • dashboard/cockpit.fxhnt.ai → 100.81.150.18:80 (fxhnt-dashboard node→cockpit:8080) ✅FUND → bizworx cockpit
  • dagster.fxhnt.ai → dagster.foxhunt:80 ✅FUND → bizworx dagster
  • grafana ❌ (operator: "grafana can go too" — drop, bizworx has own obs) · api(50051 legacy Rust gRPC) ❌ · mail(stalwart)/chat(mattermost)/minio ❌legacy → all Phase-2 drop. ONLY git+dashboard/cockpit+dagster move.
• CRITICAL SAFETY FACT: the FUND PIPELINE never uses fxhnt.ai — git-sync=gitea-ssh.gitea.svc.cluster.local:22 (in-cluster), cockpit/dagster serve over ClusterIP+own tailnet IP. fxhnt.ai is BROWSER/human access only. → the fund keeps running through the DNS flip; DNS cutover only affects operator browser access. Low-risk.
• Tailnet identities: live foxhunt fxhnt-dashboard=100.81.150.18, fxhnt-dagster=100.116.123.29. New bizworx cockpit joined as fxhnt-dashboard-1=100.107.99.35 (-1 because live one holds the name). Each bizworx pod serves TLS DIRECTLY via tailscale serve on its own MagicDNS cert (verified: cockpit tailscale serve status shows https://fxhnt-dashboard-1.tailff20e.ts.net → 127.0.0.1:8080). No bizworx proxy pod needed.

DNS cutover DECISION (operator): "RENAME bizworx pods to take over the identity"

• At cutover: suspend/scale foxhunt fund pods → they RELEASE fxhnt-dashboard/fxhnt-dagster tailnet names → restart bizworx pods so they re-register WITHOUT the -1 suffix (reclaim the names + get their IPs) → then repoint the Scaleway fxhnt.ai A-records (dashboard,cockpit,dagster,git) from the dying 100.95.225.27 to the reclaimed bizworx pod tailnet IPs. Records to change: scw dns record set on zone fxhnt.ai.
• TLS RESOLVED: old proxy served a cert-manager wildcard *.fxhnt.ai (ClusterIssuer letsencrypt-prod, Scaleway DNS-01 webhook). bizworx already has the SAME stack (cert-manager + scaleway-certmanager-webhook 1/1 + scaleway-api creds + letsencrypt-prod ClusterIssuer, already issuing LE certs in ~10 ns incl grafana-tls). → issuing *.fxhnt.ai on bizworx = just a Certificate resource. NO blocker.
• CHOSEN CUTOVER SHAPE (supersedes the naive pod-IP flip): build a stripped fund-only nginx+tailscale proxy on bizworx (fxhnt-ingress) fronting the 3 fund vhosts via ClusterIP (gitea-http.gitea:3000, fxhnt-dashboard .foxhunt:80, dagster.foxhunt:80) + socat SSH gitea-ssh.gitea:22, terminating the wildcard *.fxhnt.ai cert. Point the 4 fxhnt.ai A-records at THIS proxy's tailnet IP. Because it uses ClusterIP not pod tailnet names, the "reclaim fxhnt-dashboard identity" rename is NO LONGER on the critical path (optional cosmetic).
• FULL EXECUTABLE RUNBOOK written: docs/superpowers/plans/T8-CUTOVER-RUNBOOK.md (P1 wildcard cert → P2 build+verify proxy pre-flip → C1 freeze foxhunt → C2 delta dump/restore → C3 bizworx live+ib-gw sole session → C4 DNS flip → C5 arm ucits. Every step has VERIFY + ROLLBACK. Nothing destructive; foxhunt suspended-not-deleted.)
• Confirmed svc names for the proxy: gitea-http.gitea:3000 (headless), gitea-ssh.gitea:22 (headless), fxhnt-dashboard.foxhunt:80, dagster.foxhunt:80, ib-gateway.foxhunt:4002/4004.

T7 update 3 (2026-07-18) — cronjobs

• Cross-ns gitea-NP fix applied to ALL 15 jobs/fxhnt-*.yaml (namespaceSelector kubernetes.io/metadata.name=gitea).
• 7 CronJobs applied to bizworx ns foxhunt, ALL suspend=true (verified): bybit-rebalancer, compare-measured-precompute, factory, ibkr-rebalancer, ucits-rebalancer, ucits-volume-ibkr, vrp-rebalancer. (real-money safety: none run parallel to the live foxhunt ucits-rebalancer.)
• The 9 one-time Jobs (opra-backfill/warehouse-migrate/paper-backfill/bybit-*-job/deribit) NOT applied (they run immediately); apply on demand post-cutover with the NP fix already in place.
• REMAINING before T8: one ucits dry-run (needs SOLE IBKR session — briefly scale foxhunt ib-gateway to 0, or do it as the first act of cutover) + optional dagster asset-graph run. Then T8 cutover.

T9 VERIFY (2026-07-18) — cockpit + gate intact against MIGRATED bizworx DB ✅

• Local cockpit (./scripts/dev-cockpit-local.sh, current-context=bizworx → serves the MIGRATED DB read-only on http://127.0.0.1:8801). ENV GOTCHA fixed: the script's uv run needed --extra web --extra pg (uvicorn in web, psycopg in pg); a bare uv run fails ModuleNotFoundError. Patched dev-cockpit-local.sh.
• NUMBERS CROSS-CHECKED (read the numbers, per feedback_verify_cockpit_locally_not_prod):
  • /paper IBKR card = $1,029,015 == ibkr_account_nav.nlv latest (2026-07-17 = 1029014.63). "not connected" ABSENT (the card that silently degraded in a prior session renders the REAL DU account here).
  • bybit Sh 2.11 == backtest_summary bybit_4edge 2.1067; unlock Sh 1.03 == 1.0258. All real, DB-backed.
  • /, /strategy/multistrat, /strategy/xsfunding, /paper all HTTP 200 with real content.
  • CAVEAT: Playwright MCP browser not connected this session → verified by rendering real HTML + reading numbers (not pixels). Local cockpit left running on :8801 for the operator to click through.
• T9b forward tracks INTACT (no spurious re-inception from migration): forward_anchor t0 dates are ORIGINAL historical inceptions (sixtyforty 2026-06-15, unlock 2026-06-20, positioning 2026-07-06, bybit_4edge/multistrat/ xsfunding 2026-07-07, vrp/multistrat_exec 2026-07-14) — NOT today (2026-07-18). definition_hash/version chains + archived_at + record_mode all preserved. DB-anchor-SSOT survived byte-for-byte; recon gate will recompute correctly.
• STILL PENDING: T9 real-money confirm = watch the first armed fxhnt-ucits-rebalancer run at 09:00 UTC (tomorrow). Then T10 observe → T11 triage → T12 teardown. ROTATE the pasted tailscale key. DELETE NOTHING.

T9 VISUAL BROWSER PASS (2026-07-18) — real Chrome via Playwright MCP ✅

• Playwright MCP now available (plugins reloaded); installed the chrome channel (Google Chrome 150 at /opt/google/chrome/chrome — the MCP hardcodes that path, bundled chromium alone isn't enough).
• Drove the LOCAL cockpit (http://127.0.0.1:8801, bizworx migrated DB) in a REAL browser — full render, all data:
  • Overview: Bybit 4-edge $832,534 +738% Sh 1.74 maxDD -13%, deploy $75k/75%, gate 11/21 WAIT; 4 edges (trend Sh0.72 / unlock Sh1.03 / xsfunding Sh3.62 / positioning Sh2.55); forward-tracks table w/ correct gates: sixtyforty 23/20 GO, unlock 26/14 PASS, multistrat 9/20 WAIT + real IBKR subtrack "real -0.77% · 9 trades · 5 holdings".
  • /paper: IBKR paper account $1,029,015 · 1 strategy trading · 5 holdings · 0.9% behind plan (the real DU account — NOT "not connected"). Bybit book $832,534.
  • Full-page screenshot saved (.playwright-mcp/cockpit-overview-bizworx.png). 1 non-material console error.
• Matches DB cross-checks exactly. T9 cockpit verification COMPLETE (visual + numeric).