Pick arbitrary edges + weights + date range -> portfolio Sharpe/CAGR/maxDD +
correlation heatmap (full AND left-tail) + per-edge marginal-Sharpe, in a new
/paper/portfolio cockpit page. Engine already exists (combine_book/combine_returns/
compute_stats/marginal/correlation + materialized return series) -> glue + surface
only. Plotly via CDN as the new platform charting layer. Read-only, additive.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified the Argo Events CRD has an open schema -> trigger-level path-filter
negation can't be validated. Redesigned the mode-split to live in the
workflow (git diff --name-only) with a single msp-style sensor (lowest risk).
5 tasks: WorkflowTemplate build side, deploy step (verify tailscale route),
sensor + decide-mode, apply+webhook, e2e verify both modes + retire old file.
Every cluster apply gated behind server dry-run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator's principle: fxhnt and bizworx are different projects — use the
shared bizworx platform infra (Gitea, Argo engine, registry, node-pools)
but keep fxhnt-owned objects (WorkflowTemplate, build/deploy pods, secrets,
RBAC) in the foxhunt ns. Only the Sensor lives in argo-events (unavoidable,
kept minimal). No cross-ns RBAC, no fxhnt secrets in platform ns. Adds a
deployed-sha annotation for git-sync reproducibility.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reanimate the retired cockpit-build-deploy WorkflowTemplate with the mode
split (build vs deploy) driven by a path-filter sensor: dep/Dockerfile
changes -> Kaniko rebuild, code-only changes -> fast rollout (git-sync
re-pulls src). Reuses the existing bizworx Argo Events/Workflows/Kaniko
stack (one Gitea, verified). No ArgoCD app, no lock file. Deploy-step
route (tailscale scale-0->1 vs rollout-restart) flagged to verify.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Research brief (6-agent workflow): NL tax/fund/accounting landscape for
going live. Resolved structure given operator's existing tech-BV + holding
+ DGA-salary: arbeid taxed in the BV, dividend (cleanly taxed) to private,
passive fund capital in Box 3 -> removes the Box-1 reclassification risk.
EUR-functional fund-ledger design spec: append-only double-entry ledger in
the existing Postgres warehouse, IBKR Flex ingestion, FIFO, fund-only scope.
No conversion pipeline (EUR end-to-end). NOT tax advice.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: go all-Dagster properly + build the image right. Diagnosis
corrected: Dagster ETL was healthy all along (23:30 run SUCCESS, 14 assets,
forward_nav fresh); the stale data is the SUSPENDED compare-measured cron that
writes bybit_sleeve_ret. That cron exists because the compute needs 6-8Gi and
OOM-killed the 4Gi daemon — so the correct all-Dagster fix is K8sRunLauncher
(per-run Jobs with own memory), not daemon-run assets. Plan: add dagster-k8s,
build+push via podman (Argo/Kaniko pipeline is dead), add Job-launch RBAC,
configure K8sRunLauncher + per-asset memory, fold sleeve-ret/backtest-refs into
the DAG, materialize+verify, then delete compare-measured+VRP crons and arm
execution. Also commits the harmless working_directory=/code/src fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diagnosed why bybit went stale over the weekend: the Dagster code-server is
crash-looping (workspace.yaml working_directory=/app but git-sync code is at
/code/src -> no heartbeat -> 23:30 nightly never fires since Sat), AND all
fund crons are suspended from the cutover. Architecture is a deliberate
two-layer split (Dagster ETL @23:30 7-day + K8s cron execution) — operator
chose to keep it and make it consistent. Plan: fix the code-server dir,
catch-up materialize, arm compare-measured/factory/ucits-volume, delete the
dead VRP cron (shelved/falsified) + document. Real-money: only ucits trades.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The .dbn.zst S3 upload finished and verified byte-for-byte vs source
(rclone check --size-only: training 0 diffs/59 files, test 0 diffs/21 files;
80 objects total = source count). Two independent verified archives now exist
(S3 object storage + block snapshots, both in bizworx project, both survive
teardown). Cleaned up uploader/speedtest pods + s3 secrets; resumed the
foxhunt platform pool drain to 0. Compute heading to ~$0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The '132MB / 23 objects' were the first partial .dbn transfers, not garbage;
the resumed upload continues them. Verified singletest/+speedtest/ test files
already deleted (0 objects), bucket holds only legit training/*.dbn*. Marked
DO NOT wipe fxhnt-dbn-archive so it isn't mistaken for cleanable later.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator wants the compressed .dbn.zst copied to S3 too (browsable archive
alongside the block snapshots). Diagnosed the earlier stall: node egress
~10-17MB/s (not compression), and too many concurrent big-chunk streams
choked the dev1_l node. Retuned to --transfers 2 --s3-chunk-size 32M and it's
moving (137GiB, ETA ~6-7h). Uploading directly from a foxhunt pod mounting the
live PVCs (CSI import of restored volumes was a rabbit hole). Persistent
monitor watching completion. Snapshots kept as the validated safety net until
the S3 copy is verified. Pool held at 1 node for the uploader.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rclone S3 upload stalled on the starved foxhunt cluster (dev1_l egress +
slow bssd read -> 76KiB/s, ETA 4 days). Pivoted to Scaleway block-volume
snapshots: instant, storage-layer, restorable, no tunnel. Snapshotted both
.dbn PVCs (training 537GB, test 50GB) into the bizworx project so they survive
teardown; both available in ~20s. Cleaned up uploader/inspector pods + s3
secret; resumed the node drain to 0. Stray fxhnt-dbn-archive bucket (132MB
junk) noted for later cleanup. Surfer .dbn already migrated + verified.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator caught databento .dbn data not covered by the pg migration.
- Surfer 22 .dbn (fund tsmom/carry research universe) migrated into the
bizworx fxhnt-surfer-data PVC; verified from the dagster pod (/data/surfer/
*.dbn + state JSONs). Research assets would've failed on the empty PVC.
- 171GB archived Rust-ML .dbn (training+test PVCs, not fund-used) uploading
to a new PRIVATE Scaleway bucket fxhnt-dbn-archive via an in-cluster rclone
pod — intra-Scaleway (fr-par pod -> fr-par S3), no tunnel (can't move 175GB
over the tailscale hop). S3 creds in a foxhunt secret, never printed.
Bumped platform pool min-size 0->1 so the inspector/uploader can schedule;
resume the drain to 0 after the upload completes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator confirmed migration complete + accepted dropping the instant-rollback.
Verified bizworx has zero runtime dep on foxhunt, the pre-cutover dump is
backed up off-cluster (~/Work/fxhnt-cutover-backup, 805 objects), and the fund
is live on bizworx. Scaled all foxhunt workloads (incl postgres/gitea/proxy)
+ cert-manager/tailscale to 0, suspended all cronjobs, set platform pool
min-size=0 so the autoscaler drains the 3 dev1_l nodes to 0. Cluster + all
PVCs preserved (not destroyed) — reversible by scaling back up; terraform
destroy deferred to T12. Rollback is now dump-restore, not instant DNS flip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Installed the playwright chrome channel and drove the local cockpit (bizworx
migrated DB) in a real browser. Full render confirmed: overview forward-tracks
with correct gates (sixtyforty GO, unlock PASS, bybit/multistrat WAIT), 4-edge
book $832,534, and the /paper IBKR card showing the real DU account $1,029,015
(not the 'not connected' empty-state). Matches DB cross-checks. T9 complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Local cockpit (dev-cockpit-local.sh, bizworx context) serves the migrated DB
read-only; numbers cross-checked against the DB: /paper IBKR card $1,029,015
== ibkr_account_nav.nlv latest, bybit Sh 2.11 / unlock Sh 1.03 == backtest_summary.
The /paper IBKR card renders the real DU account (not the 'not connected'
empty-state that degraded in a prior session). Forward tracks intact: anchor
t0 dates are original historical inceptions, not re-incepted to today — DB-anchor
SSOT survived the migration; recon gate recomputes correctly.
Also fixed dev-cockpit-local.sh to pass --extra web --extra pg (bare uv run
failed ModuleNotFoundError: uvicorn/psycopg on a fresh env).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cutover complete, foxhunt frozen-but-intact (rollback available):
- C1 froze foxhunt fund writers (cronjobs suspended, dagster/cockpit/
ib-gateway scaled to 0); infra cronjobs left running.
- C2 final delta pg_dump/restore fxhnt -> bizworx. --clean over the existing
rehearsal copy broke the tsdb catalog; recovered via drop+recreate empty DB
then clean restore (exit 0, 0 errors). All row counts match frozen source
(identical to rehearsal = zero real delta). Lesson: never --clean over a
live timescaledb DB.
- C3 brought bizworx fund live: cockpit serves real migrated data, ib-gateway
1/1 stable (foxhunt released the IBKR session), dagster 3/3.
- C4 flipped Scaleway fxhnt.ai dashboard/cockpit/dagster/git A-records ->
100.93.141.11 (bizworx fxhnt-ingress). Verified over real public DNS:
HTTP 200 + valid cert; git=bizworx Gitea, dashboard=cockpit w/ data, SSH
ls-remote returns real refs.
- C5 armed only fxhnt-ucits-rebalancer (next run 09:00 UTC); rest suspended.
Rollback until T11 = flip DNS back to 100.95.225.27 + un-suspend/scale-up
foxhunt. Next: T9 verify, T10 observe, then triage/teardown. DELETE NOTHING.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fxhnt-ingress proxy (nginx wildcard-TLS + socat SSH + kernel-mode tailscale)
now 3/3 and serving dashboard/cockpit/dagster/git.fxhnt.ai. EVAL P2 PASS:
from the devbox over the tailnet -> proxy tailnet IP 100.93.141.11, all four
hosts return HTTP 200 with a valid *.fxhnt.ai LE cert chain — foxhunt still
fully live, zero DNS change.
Three fixes to get there:
- Added tailscale-fxhnt-ingress SA+Role+RoleBinding (sidecar needs ts-state
Secret RBAC; mirrors tailscale-dagster).
- Re-applied dagster.yaml (its NP ingress fxhnt-ingress edit was committed
but never applied on-cluster -> dagster vhost hung).
- Added non-headless gitea-clusterip Service (ns gitea, 10.32.x): the Helm
gitea-http/gitea-ssh are headless -> resolve to 100.64.x pod IPs the
kernel-mode TS sidecar can't reach (CGNAT overlap). nginx+socat retarget it.
C4 DNS target recorded as 100.93.141.11 in the runbook. Phase-1 prep complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
P1: fxhnt-wildcard Certificate (*.fxhnt.ai) on bizworx. DNS-01 initially
403'd because fxhnt.ai zone is in the foxhunt project (c293eb98) and the
bizworx cert-manager key's DomainsDNSFullAccess was scoped to the bizworx
project only. Domain-move is blocked by Scaleway (external apex 'Root zone
can't be updated'), so fixed via IAM: added a rule granting
DomainsDNSFullAccess scoped to c293eb98 on policy bizworx-prod-runtime-scoped
(scw iam rule create; original rule untouched). Challenge then progressed
past 403 to normal DNS propagation wait.
P2 (draft, not applied): fxhnt-ingress = nginx (wildcard TLS termination) +
socat (git SSH) + tailscale sidecar, fronting the 3 fund vhosts by ClusterIP
(git/dashboard+cockpit/dagster). Uses ClusterIP not pod IPs (bizworx TS
sidecars reach 10.32.x ClusterIPs but not 100.64 pod IPs — CGNAT overlap,
both verified). dagster NP ingress already retargeted to this proxy's label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two corrections found while prepping P2:
1. The earlier 10.32.0.1/32 apiserver-egress fix was WRONG: kube-proxy
DNATs kubernetes.default.svc (10.32.0.1:443) to the REAL apiserver
endpoint 172.16.0.11:6443 BEFORE NetworkPolicy eval, and 172.16.0.0/16
is in the public-HTTPS except list -> still blocked -> tailscale sidecar
crashlooped again -> pod NotReady -> dagster Service had NO endpoints.
Correct fix: allow egress to 172.16.0.11:6443 (post-DNAT dest). Now
3/3 stable, endpoints populate. (apiserver endpoint from
NAME ENDPOINTS AGE
kubernetes 172.16.0.11:6443 85d.)
2. dagster NP ingress referenced foxhunt-only label tailscale-gitlab-proxy;
retargeted to app.kubernetes.io/name=fxhnt-ingress (the bizworx proxy
built in P2) so dagster.fxhnt.ai can be fronted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mapped the full fxhnt.ai ingress: all 13 A-records -> 100.95.225.27
(tailscale-gitlab-proxy pod, ns foxhunt = nginx vhosts + socat + tailscale)
which DIES at teardown, so DNS MUST move. Only git + dashboard/cockpit +
dagster are fund-critical (operator: grafana can go too; api/mail/chat/minio
legacy). Fund pipeline never uses fxhnt.ai (in-cluster git-sync + ClusterIP)
-> DNS flip only affects browser access, low-risk.
TLS resolved: bizworx already has cert-manager + scaleway DNS-01 webhook +
letsencrypt-prod issuer -> *.fxhnt.ai wildcard is just a Certificate. Chosen
cutover = stripped fund-only nginx+tailscale proxy on bizworx fronting the 3
vhosts by ClusterIP; DNS points at its tailnet IP. Written full executable
runbook (P1-P2 pre-flight, C1-C5 window) with VERIFY+ROLLBACK per step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Probe-verified: IBKR historical TRADES volume works (no subscription); bid/ask
blocked (Error 354, subscription needed -> spread stays estimated). Fetch in
rebalancer context (netpol) -> ibkr_pit_volume DB -> daemon reads. Source flag
ucits_volume_source yahoo|ibkr. CronJob for the fetch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ingest UCITS LSE volume (CSPX.L/IDTM.L/IGLN.L/ICOM.L/DBMF.L), per-line UCITS
spreads, and a regime flag (ucits default | us). equity_allocation_inputs
sources each US sleeve's ADV+spread from the regime's instrument; returns stay
US. Switch to us regime after electing MiFID-II professional (~EUR500k).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Equity honest-reference analog for the IBKR US-ETF multistrat book: ingest real
Yahoo volume (PIT) for a data-driven dollar-ADV capacity model, charge real
costs (sqrt-impact on ADV + per-ETF spread + IBKR commission, not the 15bp
stub), produce a cost-netted book_series sizing series (>60d so it sizes),
merge into record_allocation. The growth vehicle above the crypto ~350k break.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deploy-the-fund knobs: single-row fund_config (capital, target_vol, kelly)
set via 'fxhnt fund-config', read by the nightly allocation with a settings
fallback. Per-sleeve crypto allocation (promote xsfunding/unlock via
promote_on_pass, demote bybit_4edge to a research reference; generalize the
honest-inputs provider to size any crypto sleeve). Marginal LOO gate KEPT +
a predetermined falsification gate (G1 survival / G2 deployment% / G3
anti-reactive) that accepts the honest verdict either way. target_vol 0.20.
Paper scope; execution wiring a follow-on.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Upgrade of the B2a allocation engine. Tasks: 1 inverse-vol (risk-parity) base weights, 2 capacity
ceiling from an honest capacity curve, 3 compute_allocation weights inverse-vol capped (not
equal), 4 honest-reference series+ceiling provider per deploy strategy (bybit_4edge book /
positioning sleeve / equity-default), 5 record_allocation sizes from the provider + caps at
capacity, 6 policy knobs (capacity_min_sharpe, equity_default_ceiling) + ceiling shown next to the
curve. Reuses forward-gating, cross-venue, funding-decay, killswitch, vol-target, $ conversion.
Self-review flags the int-vs-str series-key conversion at the provider seam. Scope stays paper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Focused upgrade (not a new system) of allocation_engine.compute_allocation: (1) equal-weight ->
inverse-vol (risk-parity) capped at max_strategy_weight; (2) a per-deploy-strategy capacity
ceiling from the honest-reference so a growing fund never over-scales a capacity-limited edge
(xsfunding breaks ~$350k). Everything else (forward-gating via deploy+promoted, cross-venue,
funding-decay, killswitch, vol-target, $ conversion) already exists and is reused.
Gate reality investigated: deploy-tier = only bybit_4edge + positioning (both WAIT post-re-warm);
strategy_allocation is empty because compute_allocation needs 60 forward days but the re-warm
reset tracks to ~8. So sizing basis moves to the honest-reference BACKTEST (stable, capacity-
adjusted), with the forward gate as the LIVE-DEPLOY switch only ("honest weights now, live per
gate"). Granularity: bybit_4edge needs a book-level capacity; positioning uses its sleeve curve;
multistrat (IBKR equity) gets a high default ceiling in v1 (real equity-ADV model = follow-on).
Self-critical section: Sharpe-4 not forward-proven (concentration cap + gating guard), equity
capacity stub, backtest-sizing-not-live caveat. Scope stays paper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Research/validation-scope design for a coin-level capacity-, cost-, and leverage-honest
backtest reference, after quantifying (2026-07-15) that the book's numbers are tail-inflated:
top-10 days drive 58% of unlock / 29% of positioning; unlock's Sharpe collapses 1.03->0.19
under a P98 upside haircut; the "levered" backtest's -11.8% maxDD < the naive -15.4% (a
re-weighting artefact, not leverage).
Locked decisions (from brainstorming): coin-level ADV/impact model (turnover data verified:
818 coins, 2021-2026, USD); participation_cap 3%; AUM curve {35k,100k,350k,1M,10M} to locate
the ceiling; risk-parity/inverse-vol weighting; ex-ante risk layer with an anti-reactive gate
(G4) — reactive de-sizing stays out (A/B-rejected 3x); candidate edge #4 (pump/anomaly
detection) falsification-tested through the same honest-cost pipeline. No live cockpit change
(production gate-replacement is a follow-on). Self-critical section names the 5 ways this could
still fool us (recompute fidelity, slippage-is-modeled-not-measured, survivorship, AUM
relevance, still-a-backtest).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consolidate /paper into ONE "Paper books" page (crypto simulated + IBKR real,
both clickable -> holdings detail); the holdings detail shows what it's holding
(assets + shares + value + %), recent trades (cost-to-trade %), value-over-time
(real vs plan) + per-strategy breakdown. Overview gets a plain "real trades"
sub-line + "behind plan" pill. HARD RULE: plain language, no dev jargon — a
single display_names map is the source of every rendered name; REAL PAPER vs
PAPER badges. Backtests page includes the IBKR tracks (D1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add D3 (overview EXEC badge + real-vs-sim divergence chip + account rollup),
D4 (/paper/ibkr account-level view: total NAV + all positions/fills +
per-strategy breakdown), a UX & information-architecture section (3-level
progressive disclosure, one divergence signal rendered consistently), and a
strategy-tagged data model so a second executor is a new ROW, not a redesign.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the IBKR paper strategies to cockpit parity with the Bybit book: their
0a backtest curve in /paper/sim (D1), and capture the REAL DU paper-account
state (fills -> exec_fills, positions, NAV) rendered as a cockpit book +
reconciled vs the recomputed multistrat sim (D2) — the honest pre-live
verification. Paper-only (no live/arm), Bybit untouched (no EU testnet keys),
faithful extension of the existing exec path (IbkrBroker already reads NLV+
positions; exec_fills exists but is empty).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
