41 Commits

Author SHA1 Message Date
0d8ce554a3 fix(dashboard): allow tailscale sidecar egress to kube-apiserver
The dashboard NetworkPolicy blocked egress to the kube-apiserver (kube-proxy
DNATs kubernetes.default.svc 10.32.0.1:443 -> 172.16.0.11:6443, and both
10.32/16 and 100.64/15 were excepted). On a FRESH sidecar start (e.g. after
the CI deploy's scale-0->1) tailscale must read its ts-state Secret via the
apiserver -> without the allow it crashlooped 'context deadline exceeded' ->
bad gateway on dashboard.fxhnt.ai. Adds the 6443->172.16.0.11/32 egress rule,
mirroring the dagster NP fix (06d12fd). Pre-existing gap, surfaced by the
first pipeline deploy that recreated the dashboard pod.
2026-07-21 12:27:00 +00:00
7da51863a1 Merge branch 'ci/fxhnt-pipeline': fxhnt CI/CD pipeline
Argo Events sensor (gitadmin/fxhnt master push) -> fxhnt-cockpit
WorkflowTemplate (foxhunt): git-diff picks build-vs-deploy, Kaniko builds
only on dep/Dockerfile changes, deploy rolls dashboard (scale 0->1) +
dagster. Both modes validated end-to-end live. Sensor lives in the bizworx
GitOps repo (bizworx-ops/infrastructure); RBAC + workflow + build/deploy
pods are foxhunt-owned (project boundary). Review-clean: injection-hardened
(env+hex-validate), least-privilege RBAC (no rolebinding-create), dagster
on :latest so dep-bumps propagate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 11:40:34 +00:00
05f6d624c0 fix(ci): M1 split tailscale RBAC out of app manifests, drop deployer rolebinding grant
The dashboard/dagster manifests bundled the tailscale-sidecar SA+Role+RoleBinding,
which forced the webhook-triggered fxhnt-ci-deployer SA to hold create/update/patch
on roles/rolebindings in its own namespace — a privilege-escalation primitive. Move
those 6 objects to infra/k8s/rbac/tailscale-sidecars.yaml, applied once out-of-band
(not by CI); trim the deployer Role to drop rbac.authorization.k8s.io entirely, and
drop serviceaccounts too since neither CI-applied manifest creates one anymore.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 11:32:51 +00:00
40c0b187a7 fix(ci): C1 dagster to :latest+Always-pull; I1 trap restores dashboard replicas
C1 (Critical, CI review): dagster webserver/daemon + K8sRunLauncher job_image
were pinned to a hardcoded tag (dagster-k8s-25e3d7c), so a mode=build dep bump
updated the dashboard (:latest) but left dagster + its run Jobs on stale deps
-> version skew in the nightly ETL. Now :latest with image_pull_policy Always.

I1 (Important): a trap scales the dashboard back to replicas=1 on ANY exit
path, so a failure between scale-0 and scale-1 can't strand the cockpit at 0.
2026-07-21 11:30:40 +00:00
ee077a2cc1 docs(ci): fix stale header comments (gitea-ssh host, submit usage) 2026-07-21 11:23:35 +00:00
e00bac067f ci(fxhnt): retire old cockpit-build-deploy + argo-deploy-cockpit.sh
Superseded by fxhnt-cockpit-workflowtemplate.yaml + fxhnt-build-sensor.yaml
+ fxhnt-ci-rbac.yaml (all validated end-to-end, both modes). The old files
targeted decommissioned infra (ci-compile-cpu pool, gitea-sshd.foxhunt,
MinIO artifacts) and no longer worked.
2026-07-21 11:23:01 +00:00
9d35df6bfc fix(ci): use plain param substitution for image tag (this Argo doesn't eval {{=...}})
The {{=sprig.trunc(...)}} expression wasn't evaluated by this Argo (v4.0.6) —
Kaniko received it literally and rejected it as an invalid tag. Use the plain
{{workflow.parameters.commit-sha}} substitution (which works everywhere else
in this template); the full 40-char sha is a valid Docker tag, no trunc needed.
2026-07-21 11:14:34 +00:00
da28c049f7 fix(ci): kaniko as args-list not sh -c (busybox choked on sprig expansion)
/busybox/sh -c parsed the {{=sprig.trunc(...)}} destination and errored on
the '(' -> syntax error, exit 2. Switched to Kaniko's entrypoint + a plain
YAML args list (the proven msp build-image pattern): Argo evaluates the
expression and passes it straight to Kaniko, no shell. Also pin executor
v1.23.2 (matches msp; no busybox needed).
2026-07-21 11:11:19 +00:00
6677f73a64 fix(ci): lower Kaniko resource requests to fit the large pool
Requested cpu:2/mem:4Gi was sized for the removed dedicated ci-compile-cpu
pool; the shared large nodes (3800m, ~1.3-1.6 CPU free) couldn't schedule it
(Insufficient cpu, autoscaler didn't add nodes). cpu:1/mem:3Gi request
(limit 3/10Gi) fits and still builds fine (Kaniko is I/O-bound here).
2026-07-21 11:07:11 +00:00
330bb56b52 fix(ci): deployer RBAC covers configmaps/serviceaccounts/roles/rolebindings
The dashboard+dagster manifests bundle a ConfigMap, ServiceAccount, and the
tailscale-sidecar Role/RoleBinding alongside the Deployment, so kubectl apply
of the whole file needs those types. RBAC-management is scoped to foxhunt
only (documented as a deliberate, namespace-local trade-off).
2026-07-21 10:51:03 +00:00
9f2b47a789 fix(ci): disable archiveLogs (no artifact repository in this Argo)
archiveLogs=true was inherited from the old MinIO-backed pipeline; this
Argo has no artifact repository configured, so log archival failed the
step. We don't need archived logs -> false.
2026-07-21 10:48:23 +00:00
80bf243023 fix(ci): use GIT_SSH_COMMAND absolute key path (Argo emissary has no HOME=/root)
The cp-key-to-~/.ssh idiom failed under Argo's executor because emissary
doesn't set HOME=/root, so git couldn't find the key -> clone exit 128.
GIT_SSH_COMMAND with the absolute key path is HOME-independent (verified
cloning under HOME=/nonexistent). Applied to all three git-clone steps.
2026-07-21 10:46:56 +00:00
6cf19ec379 fix(ci): correct gitea SSH host (gitea-sshd.foxhunt -> gitea-ssh.gitea)
The reanimated template used the OLD foxhunt-cluster gitea SSH service
(gitea-sshd.foxhunt), which doesn't exist on bizworx -> git clone failed
with exit 128. The real service is gitea-ssh.gitea.svc:22 (verified: the
argo-git-ssh-key deploy key resolves gitadmin/fxhnt HEAD there).
2026-07-21 10:41:12 +00:00
7d708b03b1 fix(ci): write decide-mode output to /workspace emptyDir not /tmp
Argo's executor reads a valueFrom.path output parameter from a mounted
volume, not the container rootfs -> /tmp/mode gave 'no such file or
directory'. /workspace is the shared emptyDir the step already mounts.
2026-07-21 10:39:15 +00:00
7318f77054 fix(ci): drop stale fr-par-2 zone pin from nodeSelectors
The reanimated template inherited topology.kubernetes.io/zone=fr-par-2 from
the old ci-compile-cpu pool, but all  nodes (and every fxhnt workload)
are in fr-par-1 -> pods were Unschedulable. pool-name=large alone is correct.
2026-07-21 10:38:05 +00:00
6d8bcc3ae7 ci(fxhnt): RBAC for the pipeline in foxhunt (SA + deploy role + cross-ns submit)
First real workflow run surfaced two gaps: (1) the WorkflowTemplate's SA
argo-workflow didn't exist in foxhunt (only in argo where msp builds);
(2) argo-events-sa couldn't submit workflows cross-ns into foxhunt. Adds
the SA, a least-privilege deployer Role (apply/scale/annotate/rollout the
dashboard+dagster deployments, exactly what kubectl-deploy runs), the Argo
workflowtaskresults role, and a foxhunt-scoped workflow-submitter binding.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 10:31:24 +00:00
0069b75d0e fix(ci): harden WorkflowTemplate against template injection
commit-sha (from the untrusted Gitea webhook body.after) was substituted
INLINE into shell scripts via {{workflow.parameters.commit-sha}} in all
three git-clone/decide-mode steps -> a malicious payload could inject shell
on a cluster-privileged pod. Now passed via ENV + hex-validated before use
(the build-image template's pattern), and mode is allowlist-checked. Closes
the background security review's Orchestrator Template Injection finding.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 10:20:26 +00:00
603aaaabff ci(fxhnt): single-trigger sensor + in-workflow git-diff mode decision 2026-07-21 10:14:48 +00:00
9b2b25433e ci(fxhnt): deploy step — keep scale 0->1 dance + deployed-sha annotation
Verified live: fxhnt-dashboard has its own dedicated tailscale sidecar
(TS_HOSTNAME=fxhnt-dashboard) and there is no Ingress/Gateway/VirtualService
in ns foxhunt routing dashboard.fxhnt.ai anywhere. fxhnt-ingress is an
unrelated nginx+tailscale pod (its own Deployment/Service, own tailscale
identity) for git/ssh routes, not the dashboard. The tailscale-session race
on `rollout restart` is real, so keep scale 0->1. Added the
fxhnt.io/deployed-sha annotation on both fxhnt-dashboard and dagster before
the rollouts, per the CI/CD plan Task 2.
2026-07-21 10:12:33 +00:00
6e9dfec067 ci(fxhnt): reanimate cockpit WorkflowTemplate (node pool large, sha7 tag) 2026-07-21 10:09:15 +00:00
c98b8d4b49 docs(ci): fxhnt CI/CD implementation plan (5 tasks, in-workflow mode decision)
Verified the Argo Events CRD has an open schema -> trigger-level path-filter
negation can't be validated. Redesigned the mode-split to live in the
workflow (git diff --name-only) with a single msp-style sensor (lowest risk).
5 tasks: WorkflowTemplate build side, deploy step (verify tailscale route),
sensor + decide-mode, apply+webhook, e2e verify both modes + retire old file.
Every cluster apply gated behind server dry-run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 10:07:03 +00:00
fc8b1b7849 docs(ci): sharpen CI/CD spec to shared-engine/separate-projects principle
Operator's principle: fxhnt and bizworx are different projects — use the
shared bizworx platform infra (Gitea, Argo engine, registry, node-pools)
but keep fxhnt-owned objects (WorkflowTemplate, build/deploy pods, secrets,
RBAC) in the foxhunt ns. Only the Sensor lives in argo-events (unavoidable,
kept minimal). No cross-ns RBAC, no fxhnt secrets in platform ns. Adds a
deployed-sha annotation for git-sync reproducibility.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 10:03:28 +00:00
5d15b0f801 docs(ci): fxhnt CI/CD pipeline design spec (git-sync-first, image-on-demand)
Reanimate the retired cockpit-build-deploy WorkflowTemplate with the mode
split (build vs deploy) driven by a path-filter sensor: dep/Dockerfile
changes -> Kaniko rebuild, code-only changes -> fast rollout (git-sync
re-pulls src). Reuses the existing bizworx Argo Events/Workflows/Kaniko
stack (one Gitea, verified). No ArgoCD app, no lock file. Deploy-step
route (tailscale scale-0->1 vs rollout-restart) flagged to verify.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 09:47:43 +00:00
bf99ce2bd1 docs(vrp): CronJob verified absent on cluster — removal fully complete
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 07:33:33 +00:00
2f957bb6a6 Merge branch 'chore/remove-vrp-code': full VRP strategy code removal
Removes all VRP (volatility-risk-premium) code — 7 modules, DVOL pipeline,
vrp_nav Dagster asset + OPRA-freeze helpers, XspOptionBarsRepo class, CLI
commands, CronJob, registry entries, 13 test files. KEEPS the OPRA data
(xsp_option_bars table + XspOptionBarRow model + 13yr .dbn) per constraint.
Full suite green (2019 passed, 2 skipped); whole-branch review clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 07:19:04 +00:00
aa85ffd3a3 docs(vrp): mark VRP removal complete + record outstanding CronJob deploy action
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 00:56:15 +00:00
baf6ddbe9e test(vrp): port archived-hide coverage to synthetic entry + honest test narratives (review fixes) 2026-07-21 00:53:51 +00:00
c7a40b70ec docs(vrp): scrub stale VRP comment/docstring references across cockpit + shared infra 2026-07-21 00:10:27 +00:00
92b0871050 chore(vrp): remove fxhnt-opra-backfill CronJob (backfill-xsp-opra command deleted)
The CronJob invoked `fxhnt backfill-xsp-opra` (removed in Task 3); with the
command gone it would crash-loop. Its only purpose was freezing OPRA marks
for VRP recompute. OPRA data already frozen in xsp_option_bars (KEPT).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 00:01:30 +00:00
adf3edb054 refactor(vrp): remove vrp/vrp_exec registry entries + _REF_ALIAS + _IBKR_ACCOUNT_SIDS/SIM_BOOKS literals
Task 6+7 (shipped together per plan so the key is never gone while a literal
still references it). Registry: 12->10 keys (vrp/vrp_exec removed). Literals:
_IBKR_ACCOUNT_SIDS + SIM_BOOKS + _REF_ALIAS drop vrp. Tests that asserted
vrp-present updated to assert vrp-ABSENT. Covering tests (forward_health,
dashboard_service, forward_definition, sim_curves, display_names) 39 green:
surviving strategies still WAIT-by-default (Constraint 2 upheld).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 00:01:02 +00:00
fc663215ef refactor(vrp): delete XspOptionBarsRepo class + _build_vrp migration builder (keep table + XspOptionBarRow) 2026-07-20 23:31:52 +00:00
5b5fbb0e0b refactor(vrp): delete vrp_nav Dagster asset + OPRA-freeze/backfill helper chain 2026-07-20 23:24:06 +00:00
eb6ecf8272 refactor(vrp): remove execute-vrp/ingest-dvol/vrp-eval/vrp-defined-risk-eval/backfill-xsp-opra CLI commands 2026-07-20 23:20:00 +00:00
b71cd6b3b0 refactor(vrp): delete DVOL pipeline (deribit_dvol, dvol_ingest, VolIndexClient port) + tests 2026-07-20 23:12:49 +00:00
59cc06e594 refactor(vrp): delete VRP application modules + black_scholes + their tests 2026-07-20 23:08:46 +00:00
a263b676de docs(vrp): final VRP code-removal plan (discovery-workflow derived)
10-task subagent-driven removal: delete leaves first (7 vrp modules +
black_scholes + DVOL pipeline + 12 tests), then callers (vrp_nav asset,
XspOptionBarsRepo class, registry keys, CLI cmds, migration builder),
then comment scrubs. KEEP xsp_option_bars table/model/data + OPRA .dbn
(Constraint 1). Verified against tree: test_xsp_option_bars_repo.py does
not exist; test_vrp_rebalancer_yaml already red (dangling manifest ref).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 23:05:21 +00:00
fb14ddc75d docs(fund): tax/fund structure brief + EUR fund-ledger design spec
Research brief (6-agent workflow): NL tax/fund/accounting landscape for
going live. Resolved structure given operator's existing tech-BV + holding
+ DGA-salary: arbeid taxed in the BV, dividend (cleanly taxed) to private,
passive fund capital in Box 3 -> removes the Box-1 reclassification risk.

EUR-functional fund-ledger design spec: append-only double-entry ledger in
the existing Postgres warehouse, IBKR Flex ingestion, FIFO, fund-only scope.
No conversion pipeline (EUR end-to-end). NOT tax advice.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 23:00:13 +00:00
8df02c2198 docs: VRP full-removal TODO for a fresh subagent-driven session (keep OPRA data) 2026-07-20 22:10:56 +00:00
39b313800d Merge: CBU0/CSBGU0 NLV fix + ib-gateway daily-restart fix
- fix(ucits): key the NLV stray-check on IBKR's reported symbol — the IEF UCITS
  line (ticker CBU0) reports holdings as canonical CSBGU0, so the legitimate
  Treasury book position was flagged stray + dropped from NLV. Adds
  UcitsListing.ibkr_symbol (probe-verified: only CBU0 diverges) + probe-ucits-symbols CLI.
- fix(ib-gateway): zero-pad AUTO_RESTART_TIME (8:00->08:00 AM) — IBC rejected the
  malformed time, leaving the daily auto-restart unconfigured -> ~16 restarts/day.
  Fixed + verified live (pod RESTARTS=0). Liveness failureThreshold 5->6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 22:10:01 +00:00
a16b341d90 fix(ib-gateway): zero-pad AUTO_RESTART_TIME (8:00->08:00 AM) — stops ~16 restarts/day
IBC's parser rejected '8:00 AM' ('Auto restart time setting must be hh:mm AM/PM'),
leaving the daily auto-restart UNCONFIGURED, so the gateway churned ~35 restarts over
2 days instead of one clean daily cycle. Zero-padded to '08:00 AM' — IBC now logs
'Auto restart time set to 08:00 AM' (verified live 2026-07-20, fresh pod RESTARTS=0).
08:00 UTC is before the 09:00 UTC UCITS rebalance window so the daily restart won't
collide with execution. Also bumped liveness failureThreshold 5->6 (180s) for the
reauth window.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 21:30:01 +00:00
9baf1f8115 fix(ucits): key NLV stray-check on IBKR-reported symbol (CBU0->CSBGU0)
Probe (probe-ucits-symbols, live 2026-07-20) confirmed only the IEF line diverges:
config ticker CBU0 resolves to IBKR canonical symbol CSBGU0, which is what
positions() reports the holding under. _expected_book_symbols keyed on the display
ticker, so the legitimate 55-share IEF/Treasury book position (reported CSBGU0) was
flagged 'outside the book' and dropped from NLV.

Add UcitsListing.ibkr_symbol (defaults to ticker; set CSBGU0 for IEF) and include it
in _expected_book_symbols so the stray-check matches what IBKR actually reports. A
genuine stray (IBIT) is still flagged. Also fixed 4 lint items on the touched lines.

11 dashboard tests green (new CSBGU0-not-stray test + IBIT-still-stray + no regression).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 21:26:18 +00:00
75 changed files with 2446 additions and 5934 deletions

View File

@@ -0,0 +1,119 @@
# fxhnt — Fund / Tax / Accounting Research Brief (2026-07-20)
**> This is researched context, NOT legal, tax, or financial advice.** The Dutch tax rules that govern this the most (Box 3 wealth taxation and the FGR fund regime) are in **active reform** as of mid-2026: Box 3's actual-return system is legislatively unsettled (Eerste Kamer voting paused) and the FGR definition was rewritten Jan 2025 with further consultation open. Every rate, threshold, and classification below must be confirmed with a licensed Dutch belastingadviseur / fiscaal jurist against the finalized Belastingdienst publications for your actual filing year before you rely on it. The single highest-stakes item — whether your systematic/algorithmic trading is Box 3 or Box 1 — is genuinely unsettled case law and needs a written advisor opinion (and possibly a vooroverleg with the Belastingdienst) *before* you scale real capital.
**> STATE NOTE (2026-07-20): the fund is STILL PAPER — no live book yet** (IBKR DU9600528 is a paper account; Bybit exec never armed). This is go-live planning from euro zero, and the Bybit ~1 Jul 2026 EEA change (§5.15) is **not** a live-position emergency — just a reason to open the right Bybit-EU (EUR-not-USDT) account before funding crypto.
**> RESOLVED STRUCTURE (2026-07-20) — supersedes §1's "start personal, then decide" and §3/§4's BV-from-zero modelling.** The operator surfaced that they **already have an IT/tech-BV + holding + a DGA-salary**. That collapses the whole Box-1-vs-Box-3 gamble into a clean, already-in-place structure:
> **Tech-BV = ALL the arbeid** (build/run the system, buy/develop technology) — already taxed there (Vpb 19% + the existing DGA loon, so *no extra €58k gebruikelijk-loon is needed*). → BV pays **dividend** (Vpb 19% + Box 2 24.5%, cleanly taxed) to private. → that al-belaste private money is invested in the **fund as passive capital → Box 3** (~6% forfait × 36% ≈ 2.2%/yr of assets, regardless of real return).
This **removes the Box-1 reclassification risk entirely** — the only real threat — because the danger was ever *arbeid + winst in one hand* (what sank GHARL:2024:4613). Here the labour is taxed in the BV and the private capital does nothing but *renderen*: exactly the "normaal vermogensbeheer" the daytrader cases won on. **No new entity, no fund wrapper, no AIFMD (own money). Dividend is an option the operator HAS, not a forced cost.** Two design guards remain (not risks): keep the fund genuinely passive → price any tech-BV→fund service **at arm's length**; and time the dividend for the Box 2 bracket. So §1's staged "personal → BV → fund" path and §3/§4's break-even math are **background theory** for this operator — the resolved route above is what applies. The advisor conversation shifts from *"which box?"* to *"how do I price the tech-BV→fund service so the fund stays demonstrably passive?"*
**> ACCOUNTING NOTE: EUR is the functional currency, no conversion pipeline.** IBKR is EUR-native and Bybit-EU is EUR too → there is no USD leg. §2's FX-revaluation subsystem (realized/unrealized-FX split, ECB-rate join, USDT proxy) is **downgraded from core to dormant insurance**: keep a `currency` + `fx_rate_to_eur` column per posting (default EUR/1.0) so a stray non-EUR line is representable, but build no conversion pipeline. Pure-EUR book = no FX P&L = cleaner books and cleaner tax.
---
# Decision brief: bookkeeping, profit extraction, and structure for an NL-resident agentic quant fund
## 1. Bottom line up front
**Start personal (Box 3), keep your own capital, and build the accounting system now — but get a written advisor opinion on the Box 1 reclassification risk before you go live with real money.** At ~$35k pilot scale, a BV's fixed costs (incorporation, annual accounts, Vpb filing, and especially the ~€58k gebruikelijk-loon obligation that an actively-run trading BV likely cannot escape) are disproportionate, and every pooled-fund wrapper (FGR/VBI/FBI/RAIF/QIAIF) is either closed off post-2025 reform or needs tens of millions in AUM to be economical. The one serious caveat: your setup — a continuously-running, code-driven, systematic strategy — sits squarely on the fact pattern (ECLI:NL:GHARL:2024:4613, the self-built crypto bot with a stable 7580% win rate) that Dutch courts pushed into **Box 1** (progressive rates to ~49.5% on actual profit) rather than Box 3, so "Box 3 is cheaper" is only true if Box 3 actually holds. **Move to a dedicated (ring-fenced) BV when either (a) your advisor judges Box 1 reclassification risk high enough that corporate-rate certainty is worth the overhead, or (b) profits are large and consistently retained/reinvested so Vpb-only compounding (19%) beats paying Box 3 tax on capital every year.** Move to a fund structure (de-minimis AIFM + FGR) only when external capital is a concrete, near-term plan — not before.
## 2. The accounting system (the part you can build)
**Recommendation: build the tax ledger inside your existing Postgres/Timescale warehouse, not in a plaintext tool (beancount/hledger).** You already run Postgres + Dagster with per-strategy tagging; a second, disconnected file-based ledger just creates a reconciliation surface between the warehouse and the books. beancount's one real advantage — a native FIFO/AVERAGE cost-basis engine — is a bounded amount of SQL to replicate, and neither tool has a Bybit importer (you build that either way). Keep beancount only as a *possible export target* if your accountant wants to inspect plain text.
### Schema (append-only double-entry)
- **`journal_entries`** (header): `entry_id, entry_date, description, source_venue, strategy_tag, source_ref` — immutable, corrections via reversing entries only (never UPDATE/DELETE).
- **`journal_postings`** (lines): `entry_id, account_id, side ('debit'|'credit'), amount, currency, fx_rate_to_eur, eur_amount` — with a DB-level CHECK/trigger enforcing that each entry's postings sum to zero **in EUR-converted terms**.
- **`accounts`**: per-venue sub-ledgers that each balance standalone — `IBKR-EUR-cash`, `IBKR-securities`, `Bybit-USD-cash`, `Bybit-crypto`, `Bybit-perp-margin` — plus **two distinct FX accounts per foreign balance**: `unrealized-FX-PnL` (hit at each period-end retranslation) and `realized-FX-PnL` (hit only on actual conversion/withdrawal). This split is what lets you resolve the accounting-vs-tax realization question later without re-deriving history.
- **`cost_basis_lots`** (per venue/asset): matched lots so realized gains come from actual lot-matching (FIFO — the NL de-facto default — or whatever your advisor confirms, applied *consistently* across both venues), not cash-flow deltas.
- **`fx_rates`**: daily ECB reference rates, keyed by value_date.
- **`account_balances`**: a materialized view for fast reads.
The **strategy_tag** on every posting makes P&L-by-strategy a `GROUP BY`, reusing your existing Dagster attribution — a filter, not a rebuild.
### Ingestion pipeline (Dagster assets, mirroring your nightly ETL pattern)
1. **IBKR Flex Web Service API** (not PDF scraping): a per-query token + query ID, 2-step SendRequest→GetStatement poll, ~1 req/sec limit, 23 pulls/day is safe. Pull trades, cash transactions, dividends, corporate actions, positions, NAV. Archive the annual Activity Statement as an audit doc even though you compute P&L independently. Note: as a W-8BEN non-US person you get a **1042-S**, never a Consolidated 1099, and there is no IBKR "NL tax statement."
2. **Bybit** transaction-history / Tax-API REST endpoints for fills, **funding fees**, and deposits/withdrawals. **Explicitly verify the Unified Trading Account API surfaces perp funding and mark-to-market events** — they are easy to miss and material to P&L.
3. Land raw payloads in an append-only `raw` schema (mirrors your DB-anchor-SSOT pattern), then a transform layer normalizes both venues into the canonical `journal_postings`.
### FX conversion & reconciliation
- **Standard rate: ECB daily reference rate** (`eurofxref-daily.xml`, ~16:00 CET; `eurofxref-hist.xml` for one-time backfill), applied at each transaction's value_date (IAS 21 transaction-date convention).
- **The ECB publishes no crypto crosses.** Two-step for Bybit: (1) treat USDT as a USD-proxy ~1:1 **and document that assumption explicitly** (or use Bybit's own USD-index for USDT legs), then (2) USD→EUR at the ECB rate for that value_date.
- **Box 3 peildatum (1 January)** crypto valuation: use the **exchange's own displayed value at 00:00 on 1 Jan** (crypto is venue-priced), and snapshot it via API.
- **Reconciliation target: DAC8/CARF.** From FY2026, in-scope CASPs report your crypto data straight to the Belastingdienst (first reporting Jan 2027). Your books must reconcile to what the venue reports. Given Bybit's current global entity is non-EU pre-migration, **assume no third-party feed and keep fully independent records** — this raises, not lowers, your self-reporting burden. (See §5 on the Bybit EU migration deadline.)
## 3. Tax-efficient profit extraction (ranked)
| Rank | Route | Box / rate implication | When it wins |
|---|---|---|---|
| **1** | **Stay personal, Box 3** | 36% on a *deemed* return (~6% "overige bezittingen" forfait for 2026 → effective ~2.2% of asset value/yr); tegenbewijsregeling lets you report actual return if lower | Pilot / own-capital, **if Box 3 classification holds** and actual returns exceed the forfait. Dramatically cheaper than any actual-profit regime in a strong year. |
| **2** | **BV, retain earnings** | Vpb only: 19% ≤€200k, 25.8% above — no Box 2 layer until you distribute | Once profits are large and **reinvested/compounded** inside the BV; deferral of the Box 2 layer is the compounding advantage. Also removes the Box 1/3 question entirely. |
| **3** | **BV, distribute as dividend** | Vpb (19/25.8%) **then** Box 2 (24.5% ≤~€68.8k / 31% above); combined effective ~3844% on distributed profit | Only when you actually need the cash personally *and* Box 1 reclassification would otherwise have applied (~49.5%) — you pay for certainty. |
| **4** | **DGA salary from a BV** | gebruikelijk loon ~€58k (2026), taxed as Box 1 wage income, progressive to ~49.5% | Not a "choice" — it's a *mandatory* cost of the BV route (an active trading BV is unlikely to qualify for the ~€5k limited-activity carve-out, and from 2026 needs an advisor-cosigned salary justification). Factor it into the BV break-even, don't treat it as extraction. |
**Key numbers to hold in mind:** Box 3 taxes *deemed* return on *capital*; the BV route taxes *actual* profit. So Box 3 is cheaper the more your real return exceeds ~6%, and the BV is cheaper mainly when Box 1 (~49.5% on actual) is the realistic alternative or when earnings are retained. Under the Box 3 **tegenbewijsregeling**, "actual return" includes **unrealized** mark-to-market gains and does **not** allow deduction of trading fees/funding (only actual Box-3 debt interest) — this is *not* how a trading P&L system naturally computes returns, so your ledger needs a parallel tax-basis calculation.
## 4. Fund-structure decision map
```
STAGE 0 — Personal, Box 3 own capital only, any size
trigger to advance ↓ external capital contemplated, OR Box 1 risk judged high, OR large retained profits
STAGE 1 — Dedicated BV (ring-fenced) own capital; new sister/subsidiary BV, NOT commingled with bizworx
trigger to advance ↓ first external euro is concrete & near-term
STAGE 2 — BV(manager) + FGR + AIFMD "light" registration
trigger to advance ↓ AUM approaches ~€2030m (dedicated-fund fixed costs become proportionate)
STAGE 3 — Licensed fund / RAIF (LU) or QIAIF (IE) institutional external capital, tens of €m+
```
**Regulatory gates:**
- **"Own money only" is a genuine AIFMD safe harbor** (Art. 4(1)(a); Recital 7 family-office carve-out). No license, no registration while you trade solely your own capital — *this covers a solo BV too*. Also outside MiFID II via the Art. 2(1)(d) own-account exemption (subject to a technical check that your order/message rate doesn't hit the "high-frequency algorithmic trading" threshold), and outside MiCA CASP authorization for own-account crypto.
- **The moment you accept one external euro**, AIF/AIFMD scope begins. Below **€100m AUM** (leveraged / redemption within 5yr) or **€500m** (unleveraged closed-end), you need only **AFM registration** (the "light"/de-minimis regime), *not* a full AIFM license — but that registration must be done **before** the first external euro, plus Wwft (AML) and Sanctiewet compliance and AFM/DNB reporting. Marketing to retail adds top-up rules (Wft 4:37p) and a <150-investor / €100k-minimum-ticket cap.
- **Ring-fence, don't commingle.** If/when you do incorporate, put trading in a **new dedicated BV**, not inside bizworx.nl — standard NL practice isolates a volatile, potentially loss-making book from an operating business's balance sheet, credit, and any regulatory status.
- **FGR is in flux and not fit-for-purpose at pilot scale.** The Jan-2025 reform abolished the open/closed test; a new fund must qualify directly under Wft-anchored rules and a further consultation (opened Dec 2025) is open. VBI is effectively closed to solo operators post-reform; FBI structurally disfavors active high-turnover trading; RAIF/QIAIF both *require* an external authorized AIFM and are uneconomical below ~€2030m. So Stage 2 is a *later* decision, timed to land after the FGR rules settle.
## 5. What to take to a Dutch tax advisor / notaris
**Classification (highest stakes):**
1. Given a fully agentic, continuously-running systematic strategy across IBKR UCITS ETFs + Bybit perps — is this **Box 3 or Box 1** (resultaat uit overige werkzaamheden / winst uit onderneming)? Reconcile the taxpayer-favorable *Gelderland/Noord-Holland* daytrading line against the *GHARL:2024:4613* bot line that landed in Box 1. **Is a vooroverleg / advance ruling advisable before scaling capital**, given the retroactive-assessment downside if reclassified after the fact?
**Entity & existing structure:**
2. Is **bizworx.nl a BV or an eenmanszaak?** This determines whether housing trading is a fresh incorporation or a restructuring (eenmanszaak→BV has geruisloze inbreng considerations; a fiscale eenheid question if a second BV is created).
3. If moving personally-held IBKR/Bybit positions into a BV — does that trigger a **deemed-disposal (afrekening)** for Box 3, and is any rollover/deferral available for a trading portfolio?
4. **gebruikelijk loon**: does running an autonomous/agentic system (software does most of the "work") qualify for the ~€5k limited-activity carve-out, or does the algorithm-design labor count against it?
**Accounting method & FX:**
5. **Cost-basis method** — FIFO or weighted-average, and must it be identical across IBKR ETF lots and Bybit crypto lots (goedkoopmansgebruik consistency)?
6. **Unrealized FX/crypto revaluation** — book to P&L each period (RJ122/IAS21) for internal reporting while the tax return recognizes only realized (goedkoopmansgebruik; the NLFiscaal unrealized-FX-loss-denial precedent)? Do I need two parallel FX-recognition tracks (book vs tax)?
7. **Functional-currency election** (EUR vs USD) — worth the 10-year lock-in, or is "EUR presentation + translate every Bybit leg at transaction-date ECB rate" sufficient without electing?
8. **USDT** — treated as USD-equivalent, or its own crypto-asset (own EUR cross) for both ongoing bookkeeping and the Box 3 peildatum?
**Instrument-specific (a genuine gap in the research):**
9. How are **perpetual-futures funding payments** characterized for Box 3 werkelijk-rendement — as income each period, or folded into position P&L? No authoritative Belastingdienst guidance was found.
10. Under werkelijk rendement, are Bybit **trading fees / funding paid / margin costs non-deductible** (as the tegenbewijsregeling mechanics suggest, unlike normal trading P&L)?
**Reform & reporting:**
11. Confirm the **finalized 2026 figures** at filing time (heffingsvrij vermogen — sources conflicted between ~€51,396 and ~€59,357 — forfait %, Box 2 thresholds).
12. Does the fund trigger a formal **boekhoudplicht** (BW 2:10 / 3:15i — 7-year retention, audit-trail format), and does the Postgres pipeline satisfy it?
13. **BTW/VAT** — confirm own-account securities/crypto trading is exempt, and re-check if a management fee is later introduced.
**Regulatory (Wft lawyer, not just belastingadviseur):**
14. Does the IBKR order/message rate meet MiFID II's **HFT** technical threshold (removes the own-account exemption)? Needs a real order-flow review.
15. **Bybit EU migration — operationally urgent:** the global platform closes EEA/NL access ~1 July 2026; you migrate to the Austria-MiCA "Bybit EU" entity (fresh KYC, Travel Rule proof-of-wallet for transfers >€1,000). **Bybit EU reportedly cannot offer USDT** (Tether has no MiCA EMT authorization), so USDT balances must convert to EUR or a MiCA stablecoin — **is that forced conversion a taxable disposal, and under which box?** Verify directly on Bybit's compliance pages before the deadline.
## 6. Immediate next build step
**Build the append-only double-entry ledger core + the IBKR Flex ingestion asset first — one Dagster asset landing IBKR Flex trades/cash/NAV into a `raw` schema, normalizing into `journal_entries`/`journal_postings` with the balanced-entry DB trigger, and joining an ECB `fx_rates` table by value_date to produce EUR-normalized postings.**
Why this one component first, over everything else:
- IBKR is your EUR-native, better-documented venue with a stable Flex Web Service API and an existing importer pattern to model on — lowest integration risk, so you validate the schema and the balanced-entry invariant against real data before adding Bybit's messier surface (funding fees, USDT proxy, migration churn).
- It exercises **every** load-bearing design decision at once: append-only immutability, the EUR-sum-to-zero constraint, the strategy_tag dimension, and the ECB-rate join — the FX-revaluation split and cost-basis lot tables extend cleanly on top once this spine is proven.
- It's a natural Dagster asset alongside your existing nightly ETL, reusing your monitoring/DB-anchor-SSOT patterns — no new infrastructure, and it turns tax season into a query rather than an annual export scramble regardless of which box you end up in.
Defer the Bybit importer, the cost-basis lot engine, and any beancount export until this spine is landing correct, reconciled EUR postings from IBKR.

View File

@@ -0,0 +1,29 @@
# VRP full removal — DONE (2026-07-20/21)
**Status: COMPLETE.** VRP code fully removed on branch `chore/remove-vrp-code` (9 commits over base a263b67),
full suite green (2018 passed, 2 skipped), whole-branch review clean (no Critical/Important). OPRA data KEPT.
## What was removed
- 7 VRP application modules (`vrp_book/eval/defined_risk_eval/exec_record/marking/pricing/research`) + `black_scholes.py`
- The DVOL pipeline (`deribit_dvol.py`, `dvol_ingest.py`, the `VolIndexClient` port) — DVOL was the IV leg of VRP
- The `vrp_nav` Dagster asset + its ~14 OPRA-freeze/backfill helper chain
- The `XspOptionBarsRepo` **class** + the `_build_vrp` migration builder + the `backfill-xsp-opra`/`ingest-dvol`/
`execute-vrp`/`vrp-eval`/`vrp-defined-risk-eval` CLI commands + the `fxhnt-opra-backfill` CronJob manifest
- The `vrp`/`vrp_exec` registry entries + `_REF_ALIAS`/`_IBKR_ACCOUNT_SIDS`/`SIM_BOOKS` literals
- 13 VRP test files; comment/docstring scrubs across 16 files; test updates (assert vrp-ABSENT / synthetic entries)
## What was KEPT (Constraint 1 — verified intact)
- `xsp_option_bars` **table** + `XspOptionBarRow` model (`cockpit_models.py:436`) + the 13-year OPRA `.dbn` data
- `ForwardNavRepo.xsp_freeze_max_date()` (`forward_nav.py:145`) — still reads the kept table
- The generic freeze-staleness health axis (`_FREEZE_SOURCES = {"opra-pit"}`) stays
- `deribit_funding` (carry edge) untouched
## CronJob cluster-resource — VERIFIED not present (no action needed)
Checked 2026-07-21 on `bizworx-prod-kapsule`: `kubectl get cronjob fxhnt-opra-backfill -n foxhunt` → NotFound,
0 hits cluster-wide. The manifest was never applied live (or already gone), so removing it from git was
sufficient. The 5 surviving foxhunt CronJobs (ucits/bybit/ibkr rebalancers, ucits-volume, factory) are all
non-VRP.
## Merged + pushed
Branch `chore/remove-vrp-code` merged to master (`2f957bb`) and pushed to origin. Live on prod after the next
normal git-sync + rollout-restart.

View File

@@ -0,0 +1,599 @@
# Positioning Per-Coin Gross Cap Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add an ex-ante per-coin gross-weight cap (0.07) to the `positioning` sleeve so no single coin dominates the book, then re-inception + backfill the forward anchor so the live gate keeps running against the capped construction.
**Architecture:** The cap is applied to the FORMING weights in `positioning_weights` (before returns), so it is strictly causal (no lookahead). It threads through the existing seams (`_book_breakdown``positioning_returns_from_store``_positioning_series``sleeve_returns_from_store`) as an optional `coin_gross_cap` param that defaults to `None` (OFF → byte-identical to today). It is switched ON at 0.07 only in `bybit_book_persist.py`. The registry `positioning` + `bybit_4edge` definition versions bump to re-inception; `inception_t0` pins the existing OOS start (2026-07-06) so the forward track backfills the window with the capped construction instead of resetting the gate to 0.
**Tech Stack:** Python 3.12, numpy-free pure math, pytest (`-n auto`), TimescaleFeatureStore (Postgres, table `bybit_features`).
## Global Constraints
- The cap is applied to WEIGHTS (in `positioning_weights`), NOT to returns/contributions — it must be ex-ante/causal (the cap decision cannot see the coin's realized return).
- `coin_gross_cap` default is `None` (OFF) at EVERY seam. With it OFF, every existing test and the whole backtest must be byte-identical to today. The OFF path must not add per-term division drift.
- After clipping `|w| ≤ coin_gross_cap`, RENORMALIZE the day's weights to unit gross (`Σ|w| = 1`), matching `positioning_weights`' existing unit-gross contract.
- The validated cap value is **0.07** (full-history sweep 1295d: Sharpe 2.60→2.69, maxDD 32.3%→29.9%, per-year Sharpe non-decreasing every year). Do NOT use 0.04 (falsified: Sharpe drops to 2.43).
- Re-inception uses the EXISTING pattern: bump `version` in `definition.params.version`, and set `inception_t0: {"date": "2026-07-06", "version": <new>}` (outside `params`, NOT hashed) so the anchor backdates and the forward window is recomputed, not reset. See `forward_definition.inception_override`.
- `positioning` is a CONSTITUENT of `bybit_4edge`; both registry entries' definition versions must bump together (the book's sleeve construction changed).
- Never touch real money / execution. This is a construction change to the modeled+forward track only.
- Run tests with `uv run --extra dev python -m pytest ...`. The positioning eval integration tests need no extra services (they use a fake store).
---
### Task 1: Add the ex-ante `coin_gross_cap` to `positioning_weights`
**Files:**
- Modify: `src/fxhnt/application/bybit_positioning_eval.py:78-104` (the `positioning_weights` function)
- Test: `tests/integration/test_bybit_positioning_eval.py`
**Interfaces:**
- Consumes: nothing new.
- Produces: `positioning_weights(ratio_by_day, *, direction="contrarian", coin_gross_cap: float | None = None) -> dict[int, dict[str, float]]`. When `coin_gross_cap` is a float, each day's weights are clipped to `|w| ≤ coin_gross_cap` then renormalized to unit gross. When `None`, output is byte-identical to today.
- [ ] **Step 1: Write the failing test**
Add to `tests/integration/test_bybit_positioning_eval.py`:
```python
def test_coin_gross_cap_clips_and_renormalizes() -> None:
# One coin dominates (retail extreme) -> uncapped weight far exceeds 0.07; cap must clip + renormalize.
ratio = {0: {"DOMUSDT": 0.95, "AUSDT": 0.50, "BUSDT": 0.50, "CUSDT": 0.05}}
uncapped = positioning_weights(ratio)[0]
assert max(abs(w) for w in uncapped.values()) > 0.07 # DOM dominates uncapped
capped = positioning_weights(ratio, coin_gross_cap=0.07)[0]
assert max(abs(w) for w in capped.values()) <= 0.07 + 1e-9 # no coin exceeds the cap
assert abs(sum(abs(w) for w in capped.values()) - 1.0) < 1e-9 # still unit gross
# sign structure preserved: DOM still short (over-long crowd), C still long (over-short)
assert capped["DOMUSDT"] < 0.0 and capped["CUSDT"] > 0.0
def test_coin_gross_cap_none_is_byte_identical() -> None:
ratio = {0: {"AAAUSDT": 0.9, "BBBUSDT": 0.5, "CCCUSDT": 0.1}}
assert positioning_weights(ratio, coin_gross_cap=None) == positioning_weights(ratio)
```
- [ ] **Step 2: Run test to verify it fails**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py::test_coin_gross_cap_clips_and_renormalizes -v`
Expected: FAIL with `TypeError: positioning_weights() got an unexpected keyword argument 'coin_gross_cap'`
- [ ] **Step 3: Implement the cap in `positioning_weights`**
Replace the body of `positioning_weights` (lines 91-104) so the signature gains `coin_gross_cap` and, after the existing unit-gross normalization, applies the clip+renormalize when the cap is set:
```python
def positioning_weights(ratio_by_day: dict[int, dict[str, float]], *,
direction: str = "contrarian",
coin_gross_cap: float | None = None) -> dict[int, dict[str, float]]:
"""`{day: {coin: w}}` market-neutral cross-sectional weights from the long/short positioning signal.
`direction="contrarian"` (DEFAULT) SHORTs the high-buyRatio coins (retail over-long → fade) and LONGs the
low-buyRatio coins (retail over-short). `direction="momentum"` is the EXACT SIGN-FLIP: LONG the over-long
/ SHORT the over-short (informed-flow continuation). Construction per day:
1. DEMEAN the per-day buyRatio across coins (so the book is dollar/market-neutral, Σw ≈ 0);
2. apply the DIRECTION sign — contrarian = demeaned (short the rich/over-long), momentum = +demeaned;
3. normalise to UNIT GROSS (Σ|w| = 1);
4. (OPTIONAL) if `coin_gross_cap` is set, CLIP each |w| to the cap and RENORMALISE to unit gross —
an EX-ANTE per-coin concentration limit (applied to the FORMING weight, so it is strictly causal:
it cannot see the coin's realized return). Validated at 0.07 (full-history sweep: Sharpe 2.60→2.69,
maxDD 32.3%29.9%, per-year non-decreasing) — caps the LABUSDT-class implosions (a 10.3% long leg
that did 59%/79.5% on 2026-07-06/07) without over-throttling the edge (0.04 falsified: Sharpe→2.43).
Because the gross is identical for both directions, `momentum` weights are the exact per-coin negation of
`contrarian`. A day with <2 coins, or zero gross after demeaning, yields no weights (skipped). Pure."""
if direction not in ("contrarian", "momentum"):
raise ValueError(f"direction must be 'contrarian' or 'momentum', got {direction!r}")
sign = -1.0 if direction == "contrarian" else 1.0
out: dict[int, dict[str, float]] = {}
for day, row in ratio_by_day.items():
if len(row) < 2:
continue
mean = st.mean(row.values())
raw = {c: sign * (v - mean) for c, v in row.items()}
gross = sum(abs(w) for w in raw.values())
if gross <= 0.0:
continue
w = {c: x / gross for c, x in raw.items()}
if coin_gross_cap is not None:
w = {c: max(-coin_gross_cap, min(coin_gross_cap, x)) for c, x in w.items()}
capped_gross = sum(abs(x) for x in w.values())
if capped_gross <= 0.0:
continue
w = {c: x / capped_gross for c, x in w.items()}
out[int(day)] = w
return out
```
- [ ] **Step 4: Run test to verify it passes**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py -v`
Expected: PASS (the two new tests + all existing positioning_weights tests still green)
- [ ] **Step 5: Commit**
```bash
git add src/fxhnt/application/bybit_positioning_eval.py tests/integration/test_bybit_positioning_eval.py
git commit -m "feat(positioning): ex-ante per-coin gross cap in positioning_weights (default off)"
```
---
### Task 2: Thread `coin_gross_cap` through `_book_breakdown` and `positioning_returns_from_store`
**Files:**
- Modify: `src/fxhnt/application/bybit_positioning_eval.py:140-166` (`_book_breakdown` signature + the `positioning_weights` call at line 177)
- Modify: `src/fxhnt/application/bybit_positioning_eval.py:287-303` (`positioning_returns_from_store` signature + its `_book_breakdown` call)
- Test: `tests/integration/test_bybit_positioning_eval.py`
**Interfaces:**
- Consumes: `positioning_weights(..., coin_gross_cap=...)` from Task 1.
- Produces:
- `_book_breakdown(store, *, universe, cost_bps, direction, capacity_capital=None, participation=0.10, turnover_window=30, tail_cap_k=None, book_vol_window=60, coin_gross_cap=None) -> dict`
- `positioning_returns_from_store(store, *, universe=None, cost_bps=5.5, direction="contrarian", capacity_capital=None, tail_cap_k=None, coin_gross_cap=None) -> dict[int, float]`
- [ ] **Step 1: Write the failing test**
Add to `tests/integration/test_bybit_positioning_eval.py` (reuse the module's existing fake store fixture — grep the file for the store used by `test_capacity_haircut_caps_gain_only_at_scale`, e.g. a `_FakeStore`/`_ReadOnlyStore` builder, and construct one where a single coin has an extreme long_ratio + a large adverse next-day return):
```python
def test_coin_gross_cap_reduces_single_coin_loss_in_book(positioning_store_with_shock):
# positioning_store_with_shock: one coin (retail over-short) craters next day; uncapped it dominates the
# book loss, capped its weight (and thus its loss contribution) is bounded.
store = positioning_store_with_shock
uncapped = positioning_returns_from_store(store, cost_bps=0.0)
capped = positioning_returns_from_store(store, cost_bps=0.0, coin_gross_cap=0.07)
shock_day = min(uncapped, key=lambda d: uncapped[d]) # the worst day uncapped
assert capped[shock_day] > uncapped[shock_day] # the cap made the worst day less bad
```
If the module has no reusable multi-coin shock fixture, build the store inline in the test from a `long_ratio` panel (one coin at ratio 0.95, three near 0.5) + a `close` panel where the over-short coin drops ~50% the next day, mirroring the `test_capacity_haircut_*` store construction in the same file.
- [ ] **Step 2: Run test to verify it fails**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py::test_coin_gross_cap_reduces_single_coin_loss_in_book -v`
Expected: FAIL with `TypeError: positioning_returns_from_store() got an unexpected keyword argument 'coin_gross_cap'`
- [ ] **Step 3: Thread the param through both functions**
In `_book_breakdown`, add `coin_gross_cap: float | None = None` to the signature (after `book_vol_window: int = 60`), and change the weight-forming call (currently line 177):
```python
weights_by_day = positioning_weights(_ratio_by_day(long_ratio), direction=direction,
coin_gross_cap=coin_gross_cap)
```
In `positioning_returns_from_store`, add `coin_gross_cap: float | None = None` to the signature (after `tail_cap_k`) and pass it through:
```python
return _book_breakdown(store, universe=universe, cost_bps=cost_bps, direction=direction,
capacity_capital=capacity_capital, tail_cap_k=tail_cap_k,
coin_gross_cap=coin_gross_cap)["net"]
```
Update the `positioning_returns_from_store` docstring's param note to add: `coin_gross_cap` (None default) applies the ex-ante per-coin concentration cap (see `positioning_weights`).
- [ ] **Step 4: Run test to verify it passes**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add src/fxhnt/application/bybit_positioning_eval.py tests/integration/test_bybit_positioning_eval.py
git commit -m "feat(positioning): thread coin_gross_cap through book_breakdown + returns_from_store"
```
---
### Task 3: Thread `coin_gross_cap` through the sleeve seam (`_positioning_series` + `sleeve_returns_from_store`)
**Files:**
- Modify: `src/fxhnt/application/bybit_book_eval.py:226-240` (`_positioning_series`)
- Modify: `src/fxhnt/application/bybit_book_eval.py:243-278` (`sleeve_returns_from_store` signature + the positioning branch)
- Test: `tests/integration/test_bybit_positioning_eval.py` (or `tests/integration/test_web_new_edges.py` if that is where sleeve_returns_from_store is exercised — grep first)
**Interfaces:**
- Consumes: `positioning_returns_from_store(..., coin_gross_cap=...)` from Task 2.
- Produces:
- `_positioning_series(store, *, universe, cost_bps, direction="contrarian", capacity_capital=None, tail_cap_k=None, coin_gross_cap=None) -> dict[int, float]`
- `sleeve_returns_from_store(store, edge, *, universe=None, cost_bps=0.0, unlock_events=None, stablecoin_spot_panel=None, capacity_capital=None, tail_cap_k=None, coin_gross_cap=None) -> dict[int, float]`. The `coin_gross_cap` is forwarded ONLY to the `positioning` branch (other edges ignore it).
- [ ] **Step 1: Write the failing test**
Add (same test file, reuse the store from Task 2's fixture/inline construction):
```python
def test_sleeve_returns_positioning_forwards_coin_gross_cap(positioning_store_with_shock):
from fxhnt.application.bybit_book_eval import sleeve_returns_from_store
store = positioning_store_with_shock
uncapped = sleeve_returns_from_store(store, "positioning", cost_bps=0.0)
capped = sleeve_returns_from_store(store, "positioning", cost_bps=0.0, coin_gross_cap=0.07)
shock_day = min(uncapped, key=lambda d: uncapped[d])
assert capped[shock_day] > uncapped[shock_day] # cap forwarded -> worst day improved
```
- [ ] **Step 2: Run test to verify it fails**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py::test_sleeve_returns_positioning_forwards_coin_gross_cap -v`
Expected: FAIL with `TypeError: sleeve_returns_from_store() got an unexpected keyword argument 'coin_gross_cap'`
- [ ] **Step 3: Thread through both functions**
In `_positioning_series` (line 226), add `coin_gross_cap: float | None = None` to the signature (after `tail_cap_k`) and forward it:
```python
return positioning_returns_from_store(store, universe=universe, cost_bps=cost_bps, direction=direction,
capacity_capital=capacity_capital, tail_cap_k=tail_cap_k,
coin_gross_cap=coin_gross_cap)
```
In `sleeve_returns_from_store` (line 243), add `coin_gross_cap: float | None = None` to the signature (after `tail_cap_k`), and change ONLY the positioning branch (line 276) to forward it:
```python
if edge == "positioning":
# ... existing comment about raw store + universe unchanged ...
return _positioning_series(store, universe=universe, cost_bps=cost_bps,
capacity_capital=capacity_capital, tail_cap_k=tail_cap_k,
coin_gross_cap=coin_gross_cap)
```
Leave the tstrend / unlock / stablecoin / xsfunding branches unchanged (they never receive `coin_gross_cap`).
- [ ] **Step 4: Run test to verify it passes**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add src/fxhnt/application/bybit_book_eval.py tests/integration/test_bybit_positioning_eval.py
git commit -m "feat(positioning): forward coin_gross_cap through sleeve_returns_from_store"
```
---
### Task 4: Turn the cap ON at 0.07 in the live precompute + add the constant
**Files:**
- Modify: `src/fxhnt/application/bybit_forward_track.py:47` (add `POSITIONING_COIN_GROSS_CAP` next to `POSITIONING_TAIL_CAP_K`)
- Modify: `src/fxhnt/application/bybit_book_persist.py:77-84` (pass the cap into `sleeve_returns_from_store`)
- Test: `tests/integration/test_bybit_paper_reconciliation.py` (or the persist test — grep for a test that calls `_persist_bybit_book` / `persist_bybit_sleeve_returns`; if none exists at unit level, assert the constant is wired via a focused test in `test_bybit_positioning_eval.py`)
**Interfaces:**
- Consumes: `sleeve_returns_from_store(..., coin_gross_cap=...)` from Task 3.
- Produces: `POSITIONING_COIN_GROSS_CAP = 0.07` in `bybit_forward_track.py`; the live `_persist_bybit_book` sleeve computation applies it to the positioning sleeve.
- [ ] **Step 1: Write the failing test**
Add to `tests/integration/test_bybit_positioning_eval.py`:
```python
def test_positioning_coin_gross_cap_constant_is_007() -> None:
from fxhnt.application.bybit_forward_track import POSITIONING_COIN_GROSS_CAP
assert POSITIONING_COIN_GROSS_CAP == 0.07
```
- [ ] **Step 2: Run test to verify it fails**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py::test_positioning_coin_gross_cap_constant_is_007 -v`
Expected: FAIL with `ImportError: cannot import name 'POSITIONING_COIN_GROSS_CAP'`
- [ ] **Step 3: Add the constant + wire it into the precompute**
In `src/fxhnt/application/bybit_forward_track.py`, directly after line 47 (`POSITIONING_TAIL_CAP_K = 4.0`):
```python
# Ex-ante per-coin gross-weight cap for the positioning sleeve — no single coin's |weight| may exceed this
# fraction of the book's unit gross. Validated 2026-07-20 over the full 1295-day history (Sharpe 2.60→2.69,
# maxDD 32.3%→29.9%, per-year Sharpe non-decreasing every year); caps LABUSDT-class single-coin implosions
# (a 10.3% long leg that did 59%/79.5% on 2026-07-06/07) without over-throttling (0.04 falsified → 2.43).
POSITIONING_COIN_GROSS_CAP = 0.07
```
In `src/fxhnt/application/bybit_book_persist.py`, update the import (line 78) and the sleeve computation (lines 80-83):
```python
from fxhnt.application.bybit_forward_track import POSITIONING_COIN_GROSS_CAP, POSITIONING_TAIL_CAP_K
sleeve_rets = {
s: sleeve_returns_from_store(store, s, universe=universe, cost_bps=cost_bps,
unlock_events=unlock_events,
capacity_capital=settings.paper_capital,
tail_cap_k=POSITIONING_TAIL_CAP_K,
coin_gross_cap=POSITIONING_COIN_GROSS_CAP)
for s in _DEFAULT_BYBIT_SLEEVES}
```
(`coin_gross_cap` is ignored by every sleeve except positioning — Task 3 only forwards it in the positioning branch — so passing it for all sleeves in the dict comprehension is correct and harmless.)
- [ ] **Step 4: Run test to verify it passes**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add src/fxhnt/application/bybit_forward_track.py src/fxhnt/application/bybit_book_persist.py tests/integration/test_bybit_positioning_eval.py
git commit -m "feat(positioning): enable coin_gross_cap=0.07 in the live bybit book precompute"
```
---
### Task 5: Bump definition versions to re-inception positioning + bybit_4edge (backdated anchor, no gate reset)
**Files:**
- Modify: `src/fxhnt/registry.py` (the `positioning` entry ~line 249-259 and the `bybit_4edge` entry ~line 198-211; also `bybit_4edge_levered` ~line 229-241 which shares the sleeve construction)
- Test: `tests/` — grep for `test_forward_definition` / a registry-consistency test; add an assertion that the bumped versions + inception_t0 are consistent
**Interfaces:**
- Consumes: the capped construction from Task 4 (the version bump signals the construction changed).
- Produces: `positioning`, `bybit_4edge`, `bybit_4edge_levered` definitions at `version: 3` with `inception_t0: {"date": "2026-07-06", "version": 3}`.
- [ ] **Step 1: Write the failing test**
Add `tests/unit/test_positioning_cap_reinception.py`:
```python
from fxhnt.registry import STRATEGY_REGISTRY
def test_positioning_and_book_bumped_to_v3_backdated():
for sid in ("positioning", "bybit_4edge", "bybit_4edge_levered"):
d = STRATEGY_REGISTRY[sid]["definition"]
assert d["params"]["version"] == 3, f"{sid} version not bumped to 3"
ov = d.get("inception_t0")
assert ov and ov["version"] == 3, f"{sid} inception_t0 not re-pinned to v3"
# backdated to the existing OOS start, NOT reset to today -> the forward window is recomputed, not lost
assert ov["date"] == "2026-07-06"
```
- [ ] **Step 2: Run test to verify it fails**
Run: `uv run --extra dev python -m pytest tests/unit/test_positioning_cap_reinception.py -v`
Expected: FAIL (versions are currently 2)
- [ ] **Step 3: Bump the three registry entries**
In each of the `positioning`, `bybit_4edge`, and `bybit_4edge_levered` entries in `src/fxhnt/registry.py`, change `"version": 2` to `"version": 3` inside `definition.params`, and change the `inception_t0` to `{"date": "2026-07-06", "version": 3}`. Note: `positioning`'s existing `inception_t0` date is `2026-07-06`; `bybit_4edge` / `bybit_4edge_levered` currently use `2026-07-07` — for this backfill set ALL three to `2026-07-06` (the positioning shock's first day) so the whole book re-warms over the same window. Add a comment on each:
```python
"inception_t0": {"date": "2026-07-06", "version": 3}, # v3: positioning coin_gross_cap=0.07 (2026-07-20 backfill)
```
- [ ] **Step 4: Run test to verify it passes**
Run: `uv run --extra dev python -m pytest tests/unit/test_positioning_cap_reinception.py tests/ -k "forward_definition or registry" -v`
Expected: PASS (the new test + any existing registry/definition consistency tests)
- [ ] **Step 5: Commit**
```bash
git add src/fxhnt/registry.py tests/unit/test_positioning_cap_reinception.py
git commit -m "feat(positioning): re-inception positioning+bybit_4edge v3 (coin_gross_cap, backdated 2026-07-06)"
```
---
### Task 6: Full-suite verification + lint/type gate
**Files:** none (verification task)
**Interfaces:** consumes all prior tasks.
- [ ] **Step 1: Run the full positioning + book + forward suite**
Run: `uv run --extra dev python -m pytest tests/ -k "positioning or book or forward or reconcil or bybit" -q`
Expected: all pass (no regression; the OFF-path byte-identity tests from Tasks 1-3 guarantee unchanged behavior everywhere the cap is not enabled).
- [ ] **Step 2: Lint + type-check the changed files**
Run: `uv run --extra dev ruff check src/fxhnt/application/bybit_positioning_eval.py src/fxhnt/application/bybit_book_eval.py src/fxhnt/application/bybit_book_persist.py src/fxhnt/application/bybit_forward_track.py src/fxhnt/registry.py`
Then: `uv run --extra dev --extra orchestration mypy src/fxhnt/application/bybit_positioning_eval.py src/fxhnt/application/bybit_book_eval.py`
Expected: no NEW errors on changed lines (pre-existing whole-file findings that are not in the diff are acceptable — confirm via `git diff --unified=0 ... | grep '^+' | awk 'length>120'` returning nothing).
- [ ] **Step 3: Commit any lint fixes** (only if Step 2 surfaced issues on changed lines)
```bash
git add -A
git commit -m "chore(positioning): lint/type fixes for coin_gross_cap"
```
---
## Post-merge deploy + backfill (OPERATOR steps — outside the coding plan, run after merge)
These are runtime/prod steps, NOT code, listed so the operator has the full picture. Do them after the branch is merged to master and pushed:
1. **Deploy the code:** `git push origin master` (the Dagster K8sRunLauncher git-syncs master HEAD; a code-only change needs no image rebuild). Rollout-restart the dagster daemon if it caches code: `kubectl --context=$BIZ -n foxhunt rollout restart deploy/dagster-daemon`.
2. **Backfill the anchor:** the re-inception is AUTOMATIC on the next `combined_book_daily` run (23:30 UTC) — the bumped version + backdated `inception_t0` cause `forward_engine` to reset the anchor to `t0=2026-07-06` and recompute the forward window with the capped construction. Do NOT run `migrate-forward-anchors` (re-inception is automatic — see `reference_fxhnt_prod_deploy_mechanics`).
3. **Refresh the backtest ref:** the `bybit_book_precompute` asset re-persists the `positioning` / `bybit_4edge` / `bybit_4edge_levered` `backtest_summary` from the capped `_persist_bybit_book` in the same nightly run, so the gate reconciles the forward track against the capped (capturable) reference.
4. **Verify next morning:** query the prod DB (port-forward svc/postgres) — `positioning` forward should re-show ~13 days (recomputed, capped), the 2026-07-06/07 daily returns should be materially less negative than 7.32%/6.89%, and the gate should read WAIT "building" (not reset to 0) or its real verdict.
---
## Self-Review
**1. Spec coverage:** cap in weights (T1) → threaded through book (T2) → sleeve seam (T3) → enabled live (T4) → re-inception+backfill (T5) → verified (T6). The "backfill not reset" constraint is implemented via `inception_t0` backdating in T5 + operator step 2. Covered.
**2. Placeholder scan:** every code step shows the exact code; the only deferred item is the shock-store fixture in T2/T3 which is explicitly instructed to reuse the existing `test_capacity_haircut_*` store construction in the same file (a real, named pattern, not a placeholder). Acceptable — the implementer has a concrete reference.
**3. Type consistency:** `coin_gross_cap: float | None = None` is identical across `positioning_weights`, `_book_breakdown`, `positioning_returns_from_store`, `_positioning_series`, `sleeve_returns_from_store`. Constant `POSITIONING_COIN_GROSS_CAP = 0.07` in `bybit_forward_track.py`, imported in `bybit_book_persist.py`. Versions all → 3, `inception_t0.date` all → "2026-07-06". Consistent.
---
### Task 7: Thread `coin_gross_cap` into the LIVE forward-nav path (Critical fix from final review)
**Why:** The final whole-branch review found the cap threads to the reconciliation REF (via `_persist_bybit_book`) but NOT to the live FORWARD NAV (built by `BybitFourEdgeStrategy`), so the gate would compare a capped ref against an uncapped forward and read a healthy book as "diverging below backtest" (reproduced: forward 25% vs ref 12.5% on a LABUSDT shock). Fail-safe (WAIT, never false-PASS) but defeats the branch's purpose. This threads the cap the same way `positioning_tail_cap_k` is already threaded.
**Files:**
- Modify: `src/fxhnt/application/bybit_forward_track.py` (`BybitFourEdgeStrategy.__init__` ~135-157, `advance` ~168-172)
- Modify: `src/fxhnt/adapters/orchestration/assets.py` (builders at 105-122, 128-143, 149-158, 164-169; nightly call sites 868-876, 916-924, 957-965)
- Modify: `src/fxhnt/adapters/orchestration/migration_builders.py` (builders at 106-117, 127-141, 148-156)
- Modify: `src/fxhnt/application/bybit_positioning_eval.py:280` (`positioning_weights_and_contributions` — the Important SSOT-attribution fix)
- Test: `tests/integration/test_positioning_forward_track.py` (+ the attribution invariant in `tests/integration/test_bybit_paper_reconciliation.py`)
**Interfaces:**
- Consumes: `POSITIONING_COIN_GROSS_CAP = 0.07` (Task 4), `sleeve_returns_from_store(..., coin_gross_cap=...)` (Task 3), `positioning_weights(..., coin_gross_cap=...)` (Task 1).
- Produces: `BybitFourEdgeStrategy(..., positioning_coin_gross_cap: float | None = None, ...)`; the four `assets.py` builders + three `migration_builders.py` builders gain a `positioning_coin_gross_cap` param forwarded to the strategy; the three nightly call sites pass `POSITIONING_COIN_GROSS_CAP`.
- [ ] **Step 1: Write the failing test**
Add to `tests/integration/test_positioning_forward_track.py` — a shock store where the forward NAV's worst day is materially less negative when the cap is threaded (mirror the shock-store construction already used in the positioning tests):
```python
def test_forward_nav_applies_coin_gross_cap() -> None:
"""The forward NAV (BybitFourEdgeStrategy store-read path) must apply the per-coin cap when given, so it
reconciles against the capped ref instead of diverging. Shock store: one over-short coin craters ~50%."""
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
from fxhnt.application.bybit_forward_track import BybitFourEdgeStrategy
_DAY = 86_400
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
px = {"LABUSDT": 100.0, "AAAUSDT": 100.0, "BBBUSDT": 100.0, "CCCUSDT": 100.0}
for d in range(4):
store.write_features("LABUSDT", [(d * _DAY, {"long_ratio": 0.05, "close": px["LABUSDT"]})])
store.write_features("AAAUSDT", [(d * _DAY, {"long_ratio": 0.50, "close": px["AAAUSDT"]})])
store.write_features("BBBUSDT", [(d * _DAY, {"long_ratio": 0.48, "close": px["BBBUSDT"]})])
store.write_features("CCCUSDT", [(d * _DAY, {"long_ratio": 0.52, "close": px["CCCUSDT"]})])
if d == 2:
px["LABUSDT"] *= 0.50
uncapped_rows, _ = BybitFourEdgeStrategy(store, sleeves=["positioning"]).advance(None, {})
capped_rows, _ = BybitFourEdgeStrategy(
store, sleeves=["positioning"], positioning_coin_gross_cap=0.07).advance(None, {})
store.close()
assert uncapped_rows and capped_rows, "forward strategy produced no rows — fixture broken"
u = dict(uncapped_rows); c = dict(capped_rows)
worst = min(u, key=lambda k: u[k])
assert c[worst] > u[worst], f"cap must reduce the forward worst day: uncapped={u[worst]}, capped={c[worst]}"
```
- [ ] **Step 2: Run test to verify it fails**
Run: `uv run --extra dev python -m pytest tests/integration/test_positioning_forward_track.py::test_forward_nav_applies_coin_gross_cap -v`
Expected: FAIL with `TypeError: BybitFourEdgeStrategy.__init__() got an unexpected keyword argument 'positioning_coin_gross_cap'`
- [ ] **Step 3: Add the param to `BybitFourEdgeStrategy` and thread it in `advance`**
In `__init__` (after `positioning_tail_cap_k: float | None = None,` on line 142) add:
```python
positioning_coin_gross_cap: float | None = None,
```
And in the body (after `self._positioning_tail_cap_k = positioning_tail_cap_k` line 154):
```python
self._positioning_coin_gross_cap = positioning_coin_gross_cap
```
In `advance`'s store-read `sleeve_returns_from_store(...)` call (lines 168-172), add the kwarg:
```python
series = sleeve_returns_from_store(
self._store, sleeve, universe=self._universe, cost_bps=self._cost_bps,
unlock_events=self._unlock_events, stablecoin_spot_panel=self._stablecoin_spot_panel,
capacity_capital=self._positioning_capacity_capital,
tail_cap_k=self._positioning_tail_cap_k,
coin_gross_cap=self._positioning_coin_gross_cap)
```
- [ ] **Step 4: Thread through every builder (parallel to `positioning_tail_cap_k`)**
In `src/fxhnt/adapters/orchestration/assets.py`, for EACH of the four builders (`build_bybit_4edge_forward_nav`, `build_bybit_4edge_levered_forward_nav`, and the two single-sleeve builders at 149 and 164): add a `positioning_coin_gross_cap=None` param to the signature (next to `positioning_tail_cap_k=None`) and forward `positioning_coin_gross_cap=positioning_coin_gross_cap` into the `BybitFourEdgeStrategy(...)` call. For the THREE nightly call sites (868-876, 916-924, 957-965) that already pass `positioning_tail_cap_k=POSITIONING_TAIL_CAP_K`: add `positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` (import it alongside `POSITIONING_TAIL_CAP_K`).
Do the identical change in `src/fxhnt/adapters/orchestration/migration_builders.py` for its three builders that pass `positioning_tail_cap_k=POSITIONING_TAIL_CAP_K` (106-117, 127-141, 148-156): add `positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` to each `BybitFourEdgeStrategy(...)` call, importing the constant.
- [ ] **Step 5: Fix the attribution SSOT (the Important finding)**
In `src/fxhnt/application/bybit_positioning_eval.py`, `positioning_weights_and_contributions` (line 263): add `coin_gross_cap: float | None = None` to its signature and forward it in the `positioning_weights(...)` call at line 280:
```python
weights_by_day = positioning_weights(_ratio_by_day(long_ratio), direction=direction,
coin_gross_cap=coin_gross_cap)
```
Then thread it through `sleeve_weights_and_contributions` in `src/fxhnt/application/bybit_paper_weights.py` (the caller at ~80-84) so the paper-book per-symbol weights match the capped sleeve return (preserving the `Σ weight·return == sleeve_returns_from_store` invariant documented at `bybit_paper_weights.py:9`). Default None everywhere.
- [ ] **Step 6: Run the tests + verify no regression**
Run: `uv run --extra dev python -m pytest tests/integration/test_positioning_forward_track.py tests/integration/test_bybit_paper_reconciliation.py tests/integration/test_bybit_positioning_eval.py -v`
Expected: all pass (new forward-nav cap test passes; existing reconciliation invariant test still passes with default-None on both sides).
- [ ] **Step 7: Commit**
```bash
git add -A
git commit -m "fix(positioning): thread coin_gross_cap into the LIVE forward-nav path + attribution (final-review Critical)"
```
---
### Task 8: Cap the LIVE paper-book POSITIONS + the report-ref path (Critical + Important from Task-7 review)
**Why:** The Task-7 review traced two more uncapped paths:
- **Critical:** the live paper-book position sizing (`combined_symbol_weights*` / `latest_raw_sleeve_weights``derive_and_persist_bybit_paper_book` → the nightly `bybit_paper_book` asset) never passes the positioning haircuts, so the fund's ACTUAL persisted positions for the positioning sleeve are sized uncapped even though the reconciled return is capped. It turns out capacity + tail_cap were ALSO never threaded here (pre-existing gap). Operator decision (2026-07-20): thread ALL THREE haircuts so executed positions match the reconciled returns (full SSOT).
- **Important:** `bybit_4edge_backtest_summary` + its caller `report_backtest_summary` pass `positioning_tail_cap_k` but no `positioning_coin_gross_cap` — a real, independently-callable uncapped ref path (masked in the scheduled flow only by the `_BYBIT_BOOK_SIDS` special-case).
**Files:**
- Modify: `src/fxhnt/application/bybit_paper_book.py``combined_symbol_weights` (~48), `combined_symbol_weights_and_returns` (~105), `latest_raw_sleeve_weights` (~80), and `derive_and_persist_bybit_paper_book` (the persist driver): add `positioning_capacity_capital`, `positioning_tail_cap_k`, `positioning_coin_gross_cap` (all `float | None = None`) and forward into every `sleeve_weights_and_contributions(...)` call.
- Modify: `src/fxhnt/adapters/orchestration/assets.py``build_bybit_paper_book` (~180): add the 3 params, forward into `derive_and_persist_bybit_paper_book`; nightly call site (~1105): pass `positioning_capacity_capital=s.paper_capital, positioning_tail_cap_k=POSITIONING_TAIL_CAP_K, positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` (import the two constants).
- Modify: `src/fxhnt/application/bybit_forward_track.py``bybit_4edge_backtest_summary` (~210): add `positioning_coin_gross_cap: float | None = None` next to `positioning_tail_cap_k`, forward into the `BybitFourEdgeStrategy(...)` call.
- Modify: `src/fxhnt/application/backtest_refs_report.py``report_backtest_summary` (~42): import `POSITIONING_COIN_GROSS_CAP`, pass `positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` into `bybit_4edge_backtest_summary(...)`.
- Test: `tests/integration/test_bybit_paper_reconciliation.py` (or the paper-book test file — grep for a test that calls `combined_symbol_weights_and_returns` / `derive_and_persist_bybit_paper_book`).
**Interfaces:**
- Consumes: `sleeve_weights_and_contributions(..., positioning_capacity_capital=None, positioning_tail_cap_k=None, positioning_coin_gross_cap=None)` (Task 7 added coin_gross_cap; VERIFY it also has capacity/tail params — if `sleeve_weights_and_contributions` does NOT yet accept capacity_capital/tail_cap_k, add them there too, forwarding into its `positioning_weights`/underlying call, same as coin_gross_cap).
- Produces: the 3 haircut params on `combined_symbol_weights`, `combined_symbol_weights_and_returns`, `latest_raw_sleeve_weights`, `derive_and_persist_bybit_paper_book`, `build_bybit_paper_book`; `positioning_coin_gross_cap` on `bybit_4edge_backtest_summary` + `report_backtest_summary`. All default None.
- [ ] **Step 1: Write the failing test**
Add a paper-book position test: on a LABUSDT-shock store, the positioning sleeve's persisted per-symbol weight for the crashing coin is smaller when `positioning_coin_gross_cap=0.07` is threaded through `combined_symbol_weights_and_returns` than without.
```python
def test_paper_book_positions_capped_by_coin_gross_cap() -> None:
"""The live paper-book per-symbol weights must honor the per-coin cap so executed positions match the
capped reconciled return (SSOT). On a store with one dominant positioning coin, the capped weight for that
coin is strictly smaller than uncapped."""
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
from fxhnt.application.bybit_paper_book import combined_symbol_weights_and_returns
_DAY = 86_400
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
for d in range(4):
store.write_features("LABUSDT", [(d * _DAY, {"long_ratio": 0.02, "close": 100.0})])
for i, sym in enumerate(("AAAUSDT", "BBBUSDT", "CCCUSDT", "DDDUSDT")):
store.write_features(sym, [(d * _DAY, {"long_ratio": 0.45 + 0.02 * i, "close": 100.0})])
w_unc, _r, _d = combined_symbol_weights_and_returns(store, sleeves=("positioning",))
w_cap, _r2, _d2 = combined_symbol_weights_and_returns(
store, sleeves=("positioning",), positioning_coin_gross_cap=0.07)
store.close()
assert w_unc.get("positioning") and w_cap.get("positioning"), "no positioning weights — fixture broken"
lab_unc = abs(w_unc["positioning"].get("LABUSDT", 0.0))
lab_cap = abs(w_cap["positioning"].get("LABUSDT", 0.0))
assert lab_unc > 0.0
assert lab_cap < lab_unc, f"cap must shrink the dominant coin's position: unc={lab_unc}, cap={lab_cap}"
```
- [ ] **Step 2: Run to verify it fails**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_paper_reconciliation.py::test_paper_book_positions_capped_by_coin_gross_cap -v`
Expected: FAIL with `TypeError: combined_symbol_weights_and_returns() got an unexpected keyword argument 'positioning_coin_gross_cap'`
- [ ] **Step 3: Thread the 3 haircuts through the paper-book weight functions**
For `combined_symbol_weights`, `combined_symbol_weights_and_returns`, and `latest_raw_sleeve_weights` in `bybit_paper_book.py`: add `positioning_capacity_capital: float | None = None`, `positioning_tail_cap_k: float | None = None`, `positioning_coin_gross_cap: float | None = None` to each signature, and forward all three into their `sleeve_weights_and_contributions(...)` call(s) (as `capacity_capital=`/`tail_cap_k=`/`positioning_coin_gross_cap=` per that function's actual param names — grep `def sleeve_weights_and_contributions` for the exact names). If `sleeve_weights_and_contributions` lacks capacity_capital/tail_cap_k params, add them there first (forwarding into its positioning path), then here.
- [ ] **Step 4: Thread through the persist driver + builder**
`derive_and_persist_bybit_paper_book` (bybit_paper_book.py): add the 3 params, forward into its `combined_symbol_weights_and_returns(...)` call. `build_bybit_paper_book` (assets.py:180): add the 3 params, forward into `derive_and_persist_bybit_paper_book(...)`. Nightly call site (assets.py:~1105): pass `positioning_capacity_capital=s.paper_capital, positioning_tail_cap_k=POSITIONING_TAIL_CAP_K, positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` (import both constants from `bybit_forward_track`).
- [ ] **Step 5: Fix the report-ref path (Important)**
`bybit_4edge_backtest_summary` (bybit_forward_track.py:~210): add `positioning_coin_gross_cap: float | None = None` next to `positioning_tail_cap_k`, forward into `BybitFourEdgeStrategy(..., positioning_coin_gross_cap=positioning_coin_gross_cap)`. `report_backtest_summary` (backtest_refs_report.py:~42): import `POSITIONING_COIN_GROSS_CAP`, pass `positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` into the `bybit_4edge_backtest_summary(...)` call.
- [ ] **Step 6: Run tests + no regression**
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_paper_reconciliation.py tests/integration/test_positioning_forward_track.py tests/unit/test_backtest_refs_report.py tests/integration/test_bybit_positioning_eval.py -v`
Expected: all pass. Then confirm no site was missed: `grep -c positioning_tail_cap_k` vs `grep -c positioning_coin_gross_cap` should MATCH in bybit_paper_book.py, assets.py, bybit_forward_track.py, backtest_refs_report.py.
- [ ] **Step 7: Commit**
```bash
git add -A
git commit -m "fix(positioning): cap live paper-book positions + report-ref path (Task-7-review Critical+Important)"
```

View File

@@ -0,0 +1,474 @@
# VRP Strategy Full Removal — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox tracking; delete leaves before callers so imports never break mid-removal.
**Derived from:** a discovery workflow (24 file classifications + 3 deep-traces) — `wf_ffbdc8b5-c02`. The
tricky shared-vs-VRP calls were resolved by trace: **DVOL pipeline = delete** (only VRP reads it),
**black_scholes.py = delete** (VRP-only), **surfer vendor = untouched** (docstring mention only),
**xsp_option_bars TABLE/MODEL/data = KEEP** (only the VRP-only `XspOptionBarsRepo` *class* is deleted).
## Goal
Fully delete the VRP (variance-risk-premium) strategy — the shelved/FALSIFIED equity-VRP
sleeve and its executed twin `vrp_exec` — from the fxhnt repo, leaving **zero orphaned code,
zero dead registry keys, and zero dangling imports**. The nightly Dagster pipeline, the cockpit
web app, live IBKR/UCITS execution, the Bybit paper book, and all four kept crypto edges must
remain fully functional and the pytest suite must be green at the end.
The removal spans four coupled clusters, all of which are VRP-exclusive per the deep-traces:
1. **VRP application modules**: `vrp_book`, `vrp_eval`, `vrp_defined_risk_eval`,
`vrp_exec_record`, `vrp_marking`, `vrp_pricing`, `vrp_research`.
2. **VRP option-math + IV plumbing**: `black_scholes.py` (VRP-only), the entire DVOL pipeline
(`deribit_dvol.py`, `dvol_ingest.py`, the `dvol` feature panel, the unused
`VolIndexClient.dvol()` port).
3. **VRP persistence/orchestration**: `xsp_option_bars.py` repo class, the `vrp_nav` Dagster
asset + its ~14 private OPRA-freeze/backfill helpers, the `backfill-xsp-opra` ingest path,
the `VrpStrategy` builder, and the `vrp`/`vrp_exec` registry entries.
4. **Comment/registry-string scrubs** across shared cockpit infrastructure that merely *names*
VRP as an illustrative example.
## Architecture
VRP was ARCHIVED 2026-07-15 (fails the LOO marginal-Sharpe gate; see
`pearl_fxhnt_diversifier_hunt_2026_07_15`). The two registry entries already carry
`archived: True`, and `vrp_nav` is already **de-wired** from `definitions.py`'s nightly asset
graph — so no live nightly asset materializes VRP today. What remains is dead weight plus two
active-but-purposeless CLI/CronJob ingest paths (`ingest-dvol`, `backfill-xsp-opra`) and a set
of stale registry keys/comment references.
Dependency direction (leaf → root), which dictates task order so imports never break mid-removal:
```
black_scholes.py ────────────► vrp_defined_risk_eval.py ──┐
deribit_dvol.py ─► dvol_ingest.py ─► (dvol panel) ────────┤
xsp_option_bars.py (repo class) ──► vrp_book.py ──────────┤──► cli.py commands
vrp_pricing.py / vrp_marking.py ──► vrp_book/eval/exec ───┤ migration_builders (_build_vrp)
└──► assets.py vrp_nav + helpers
registry.py vrp / vrp_exec
forward_ingest _REF_ALIAS
sim_curves / dashboard_service /
app.py / definitions.py (scrub)
```
Rule: delete leaf modules **and their tests together** first, then the callers that referenced
them (asset graph, registry, CLI, migration builder), then the pure comment-scrub edits last.
## DVOL and black_scholes — resolved decisions (per deep-traces)
- **DVOL pipeline → DELETE (do not keep).** Exhaustive grep confirms the only readers of the
`dvol` feature panel (`store.read_panel(..., "dvol")`) are `vrp_eval.py` and
`vrp_defined_risk_eval.py`. `VolIndexClient.dvol()` (`ports/market_data.py:22`) has **zero**
callers. DVOL is not a Dagster asset. DVOL is the Deribit IV index — a data source entirely
distinct from the OPRA `xsp_option_bars` data being kept; there is no keep-carve-out for it.
Delete `deribit_dvol.py`, `dvol_ingest.py`, the `ingest-dvol` CLI command, the
`VolIndexClient.dvol()` port method, and their tests. The `dvol` warehouse rows in
`bybit_features` are inert once nothing reads them — no migration needed; they can age out.
- **black_scholes.py → DELETE (VRP-only, not shared math).** The only importers are
`vrp_defined_risk_eval.py` and its own unit test `tests/unit/test_black_scholes.py`.
`vrp_pricing.py` defines a *separate* `bs_put` (different signature) and does **not** import
from `black_scholes.py`, so there is no hidden cross-file sharing. Delete both files.
- **`xsp_option_bars` TABLE + `XspOptionBarRow` model + OPRA `.dbn` → KEEP.** Only the
`XspOptionBarsRepo` *repo class* (`adapters/persistence/xsp_option_bars.py`) is VRP-only and
gets deleted. The ORM row model `XspOptionBarRow` in `cockpit_models.py`, the physical table,
the 13-year OPRA data, and `ForwardNavRepo.xsp_freeze_max_date()` (raw `SELECT max(date)`)
all stay untouched.
## Global Constraints
1. **KEEP the OPRA data.** Do **not** drop or migrate the `xsp_option_bars` table, do **not**
remove `XspOptionBarRow` from `cockpit_models.py`, do **not** delete OPRA `.dbn` files, and
do **not** touch `ForwardNavRepo.xsp_freeze_max_date()`. Only the VRP *ingest/consumer*
machinery (`XspOptionBarsRepo` class, `_backfill_month`/`_freeze_*` helpers, `backfill-xsp-opra`)
is removed. The freeze-staleness health axis (`_FREEZE_SOURCES = {"opra-pit"}`) stays generic.
2. **Real-money paths must err false-WAIT, never false-EXECUTE.** VRP execution
(`execute-vrp`/`vrp_exec_record`) is being deleted, not disabled — but during the removal no
edit may cause any *surviving* real-money path (multistrat/UCITS IBKR, Bybit paper) to
silently promote/execute on missing state. After removing the `vrp`/`vrp_exec` registry keys,
confirm the cockpit gate and `forward_ingest` promotion loop still default to WAIT for every
surviving strategy and never KeyError into a false-PASS.
3. **Full pytest suite green before finishing.** Run `uv run pytest -n auto` (or
`python -m pytest -n auto` per `pyproject.toml addopts = "-n auto"`) from the repo root and
confirm 0 failures / 0 errors before the final commit. Per-task commands below are the fast
local gate; the full-suite run in the final task is the release gate.
4. **The false VRP STALE health alarm must be gone.** After removal, verify **no `vrp` key
remains in `STRATEGY_REGISTRY`** (`vrp` and `vrp_exec` both removed) so the forward-health
freeze/staleness axis and `dashboard_service._active_ibkr_account_sids()` can no longer
surface a ghost VRP STALE row. Verification command in the final task.
### Pre-flight note (already-broken test)
`tests/unit/test_vrp_rebalancer_yaml.py` asserts the existence of
`infra/k8s/jobs/fxhnt-vrp-rebalancer.yaml`, which **no longer exists** in the tree. This test is
already failing/erroring at HEAD; it is deleted in Task 1 alongside the other VRP tests, which
also removes that pre-existing red.
---
## Task 1 — Delete VRP application modules + their tests (leaves first)
**Files to delete (whole file):**
- `src/fxhnt/application/vrp_book.py`
- `src/fxhnt/application/vrp_eval.py`
- `src/fxhnt/application/vrp_defined_risk_eval.py`
- `src/fxhnt/application/vrp_exec_record.py`
- `src/fxhnt/application/vrp_marking.py`
- `src/fxhnt/application/vrp_pricing.py`
- `src/fxhnt/application/vrp_research.py`
- `src/fxhnt/application/black_scholes.py`
- `tests/unit/test_vrp_book.py`
- `tests/unit/test_vrp_pricing.py`
- `tests/unit/test_vrp_exec_helpers.py`
- `tests/unit/test_vrp_exec_ladder_cap.py`
- `tests/unit/test_vrp_exec_record.py`
- `tests/unit/test_vrp_rebalancer_yaml.py` (also clears the pre-existing red — see pre-flight note)
- `tests/unit/test_registry_vrp.py`
- `tests/unit/test_black_scholes.py`
- `tests/integration/test_vrp_eval.py`
- `tests/integration/test_vrp_defined_risk_eval.py`
- `tests/integration/test_vrp_nav_asset.py`
- `tests/integration/test_cockpit_vrp_render.py`
**Rationale:** every one of these modules is imported only by other `vrp_*` modules or by the
`cli.py`/`assets.py`/`migration_builders.py` callers handled in later tasks. Deleting the tests
in the *same* task prevents collection-time `ImportError` on the deleted modules. Do NOT yet
touch `assets.py`, `cli.py`, `registry.py`, or `migration_builders.py` — those still reference
these modules only via **lazy (in-function) imports**, so the module tree still imports cleanly
after this task (the broken imports are inside function bodies not executed at import time). They
are removed in Tasks 35.
**Covering test command** (must import-clean and not collect the deleted files):
```
uv run pytest tests/unit tests/integration -q --co 2>&1 | tail -20
```
Expect: collection succeeds with the deleted VRP tests absent and no `ImportError`.
**Commit:** `refactor(vrp): delete VRP application modules + black_scholes + their tests`
---
## Task 2 — Delete the DVOL pipeline + its tests
**Files to delete (whole file):**
- `src/fxhnt/adapters/data/deribit_dvol.py`
- `src/fxhnt/application/dvol_ingest.py`
- `tests/integration/test_deribit_dvol.py`
- `tests/integration/test_dvol_ingest.py`
**Edit `src/fxhnt/ports/market_data.py`:**
- Remove the `VolIndexClient` Protocol (the `dvol()` method block, ~lines 2124). Confirmed
zero implementers and zero callers repo-wide; it is dead scaffolding.
**Do NOT touch:** `deribit_funding.py` and the `ingest-deribit-funding` CronJob/command —
that is the *funding* (cross-venue carry) edge, a separate kept data source that reads the
`funding`/`deribit_funding` features, never `dvol`.
**Rationale:** `ingest_dvol` and `DeribitDVOL` are consumed only by the `ingest-dvol` CLI
command (removed in Task 3) which fed only the two VRP evaluators (deleted in Task 1). The
`dvol` feature panel now has no readers.
**Covering test command:**
```
uv run pytest tests/integration -q --co 2>&1 | tail -20
grep -rn "VolIndexClient\|import.*deribit_dvol\|import.*dvol_ingest" src/ | grep -v "/cli.py"
```
Expect: no collection errors; the grep returns only the (still-present, removed next task)
`cli.py` lazy imports.
**Commit:** `refactor(vrp): delete DVOL pipeline (deribit_dvol, dvol_ingest, VolIndexClient port) + tests`
---
## Task 3 — Delete VRP + DVOL commands from `cli.py`
**File to edit:** `src/fxhnt/cli.py`
**Exact edits (delete each command block in full, including its in-function lazy imports):**
- Delete `@app.command("execute-vrp")` `execute_vrp` (lines ~439499), including the lazy
`from fxhnt.application.vrp_exec_record import plan_and_record_vrp`. **Keep** the module-level
and other commands' use of `ForwardNavRepo` — only remove this command's local import line if
it has one; the `ForwardNavRepo` module itself is reused by `execute-multistrat` and others.
- Delete `@app.command("ingest-dvol")` `ingest_dvol_cmd` (lines ~12171266), including the lazy
`from fxhnt.adapters.data.deribit_dvol import DeribitDVOL` and
`from fxhnt.application.dvol_ingest import ingest_dvol`.
- Delete `@app.command("vrp-eval")` `vrp_eval` (lines ~21932288), including the lazy
`from fxhnt.application.vrp_eval import (verify_vrp_edge, vrp_metrics_from_store, walk_forward_vrp)`.
- Delete `@app.command("vrp-defined-risk-eval")` `vrp_defined_risk_eval` (lines ~22912403),
including the lazy `from fxhnt.application.vrp_defined_risk_eval import (...)`.
- Delete the private helpers `_print_vrp_verify` (~24062446) and `_print_vrp_walk_forward`
(~24492469) — called only by the two eval commands above.
**Decision — `backfill-xsp-opra` (lines ~578623): DELETE this command too.** It calls
`_backfill_month` and `XspOptionBarsRepo`, both of which are deleted in Tasks 4/5, and its sole
purpose is freezing OPRA marks into `xsp_option_bars` **for VRP recompute**. Keeping the OPRA
*data* (Constraint 1) does not require keeping the *ingest path* — the data already exists and
the only consumer of freshly-frozen marks was VRP. Delete the `@app.command("backfill-xsp-opra")`
`backfill_xsp_opra` function and its `from fxhnt.adapters.orchestration.assets import _backfill_month`
+ `from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo` local imports. (If
the executor finds a documented non-VRP consumer of freshly-frozen OPRA marks, stop and flag —
none was found in the trace.)
**Covering test command:**
```
uv run python -c "import fxhnt.cli" # module must import clean
uv run fxhnt --help >/dev/null # Typer app builds with commands gone
grep -n "vrp\|dvol\|backfill-xsp\|VrpStrategy\|black_scholes\|XspOptionBarsRepo" src/fxhnt/cli.py
```
Expect: import + `--help` succeed; grep returns no hits.
**Commit:** `refactor(vrp): remove execute-vrp/ingest-dvol/vrp-eval/vrp-defined-risk-eval/backfill-xsp-opra CLI commands`
---
## Task 4 — Remove the `vrp_nav` asset + OPRA-freeze helpers from `assets.py`
**File to edit:** `src/fxhnt/adapters/orchestration/assets.py`
**Exact edits (delete code):**
- Delete the `@asset`-decorated `vrp_nav` function (lines ~686718), including its inline
imports of `DatabentoDataProvider`, `XspOptionBarsRepo`, `VrpStrategy` (~line 694).
- Delete the exclusive private helper chain (lines ~492683), all VRP-only and now callerless
after Tasks 1/3: `_iso_date`, `_forward_from_marks` (+ its inline
`fxhnt.application.vrp_pricing.parity_forward` import ~502), `_band_contracts`,
`_band_rows_from_marks`, `_day_marks`, `_xsp_forward_estimate`, `_freeze_xsp_slice`,
`_freeze_day`, `_prev_trading_day`, `_DEGENERATE_FORWARD_MARKERS`,
`_is_degenerate_forward_error`, `_freeze_latest_session` (+ its inline
`DatabentoRangeUnavailable` import ~614), `_rough_forward`, `_backfill_month`.
- Delete the trailing `# vrp_nav ARCHIVED 2026-07-15 — de-wired (fn kept below)` comment
(~line 1131) — the function no longer exists below.
- Reword the module docstring (~line 4): drop the trailing `/vrp` from the forward-track asset
enumeration.
**Do NOT touch:** `deribit_funding_bars` (~825856, unrelated Deribit perp *funding*, kept for
cross-venue-carry), and all other assets (`futures_bars`, `bybit_*`, `multistrat_*`,
`sixtyforty_nav`, `positioning_nav`, `xsfunding_nav`, `unlock_nav`, `cockpit_forward`, …). The
RETIRED-track comments at ~721 and ~748 do not mention vrp — leave as-is.
**Covering test command:**
```
uv run python -c "import fxhnt.adapters.orchestration.assets"
uv run python -c "import fxhnt.adapters.orchestration.definitions" # Dagster code-location loads
grep -n "vrp\|Vrp\|_backfill_month\|_freeze_\|XspOptionBarsRepo\|vrp_pricing" src/fxhnt/adapters/orchestration/assets.py
uv run pytest tests/integration/test_orchestration_definitions.py -q
```
Expect: both modules import clean, grep returns no hits, definitions test passes.
**Commit:** `refactor(vrp): delete vrp_nav Dagster asset + OPRA-freeze/backfill helper chain`
---
## Task 5 — Remove `XspOptionBarsRepo` class + the `_build_vrp` migration builder
**File to delete (whole file):**
- `src/fxhnt/adapters/persistence/xsp_option_bars.py` (the `XspOptionBarsRepo` repo class).
All callers (`vrp_book.py`, `vrp_exec_record.py`, `vrp_marking.py`, the `vrp_nav` asset, the
`backfill-xsp-opra` command) are gone after Tasks 1/3/4.
- **VERIFIED 2026-07-20: `tests/integration/test_xsp_option_bars_repo.py` does NOT exist in the tree** —
do not attempt to delete it. If a repo-class test surfaces under another name during removal, delete it;
otherwise there is no test file for this class.
**KEEP (Constraint 1):** `cockpit_models.XspOptionBarRow`, the `xsp_option_bars` table, and
`ForwardNavRepo.xsp_freeze_max_date()` — none of these import or depend on the repo class.
**File to edit:** `src/fxhnt/adapters/orchestration/migration_builders.py`
- Delete the `_build_vrp` closure (lines ~7176) and its `builders["vrp"] = ("vrp_state",
_build_vrp)` registration (~78), plus the preceding `# ---- vrp_nav: ...` comment banner (~71).
**Covering test command:**
```
uv run python -c "import fxhnt.adapters.orchestration.migration_builders"
grep -rn "XspOptionBarsRepo\|_build_vrp\|builders\[.vrp" src/
uv run pytest tests/ -q -k "migration or migrate" 2>&1 | tail -20
```
Expect: import clean, grep returns no hits (the `XspOptionBarRow` model in `cockpit_models.py`
is a different symbol and must still be present — verify separately:
`grep -n "class XspOptionBarRow" src/fxhnt/adapters/persistence/cockpit_models.py` returns 1 hit).
**Commit:** `refactor(vrp): delete XspOptionBarsRepo class + _build_vrp migration builder (keep table + XspOptionBarRow)`
---
## Task 6 — Remove `vrp`/`vrp_exec` registry keys + `forward_ingest` alias
**File to edit:** `src/fxhnt/registry.py`
- Delete the `"vrp"` `STRATEGY_REGISTRY` entry (lines ~161172) and its preceding comment
block (~157).
- Delete the `"vrp_exec"` `STRATEGY_REGISTRY` entry (lines ~177186) and its preceding comment
block (~173176).
- Keep the dict itself and all 9 non-VRP entries and the `FUND_INSTRUMENTS` import/re-export.
**File to edit:** `src/fxhnt/application/forward_ingest.py`
- Remove the `"vrp_exec": "vrp"` pair from `_REF_ALIAS` (line ~45), leaving
`{"multistrat_exec": "multistrat", "bybit_4edge_exec": "bybit_4edge"}`.
**Rationale + Constraint 4:** removing the `vrp` key is what actually kills the false VRP STALE
health alarm — with no `vrp` in `STRATEGY_REGISTRY`, the forward-health freeze/staleness loop
and `dashboard_service._active_ibkr_account_sids()` can no longer iterate a ghost VRP row.
(Note: `dashboard_service._IBKR_ACCOUNT_SIDS` hard-codes `"vrp"`; that literal is fixed in
Task 7 in the *same PR* so the window where the key is gone but the literal remains never ships.)
**Covering test command:**
```
uv run python -c "from fxhnt.registry import STRATEGY_REGISTRY; assert 'vrp' not in STRATEGY_REGISTRY and 'vrp_exec' not in STRATEGY_REGISTRY, STRATEGY_REGISTRY.keys(); print('registry clean')"
uv run pytest tests/unit/test_forward_definition.py tests/integration/test_forward_health.py tests/integration/test_dashboard_service.py -q 2>&1 | tail -20
```
Expect: assertion passes; forward/dashboard tests green (these confirm surviving strategies
still WAIT-by-default — Constraint 2).
**Commit:** `refactor(vrp): remove vrp/vrp_exec registry entries + forward_ingest _REF_ALIAS`
---
## Task 7 — Fix the one behavioral literal + `sim_curves` book pill (data edits)
These are the two non-comment edits that gate on `vrp` being a *present* registry key and would
misbehave once it is gone (Task 6). They ship together with — or immediately after — Task 6.
**File to edit:** `src/fxhnt/application/dashboard_service.py`
- Line ~30: change `_IBKR_ACCOUNT_SIDS = frozenset({"multistrat", "vrp"})` to
`frozenset({"multistrat"})`. **Why this is behavioral, not cosmetic:** `_active_ibkr_account_sids()`
computes `not STRATEGY_REGISTRY.get("vrp", {}).get("archived")`; once the `vrp` key is gone
this becomes `not None` = `True`, wrongly treating `vrp` as an ACTIVE ibkr-account sid and
rendering a ghost/broken VRP row on the IBKR account page. Removing the literal eliminates it.
**File to edit:** `src/fxhnt/application/sim_curves.py`
- Line ~15: remove the `"vrp"` string from the `SIM_BOOKS` tuple, leaving
`("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered")`. **Why:** once
the `vrp` registry entry is gone, a `vrp` Backtest-page pill would `KeyError`/500 at request
time in `app.py` (`registry_backtest_kind("vrp")`). The dispatch logic in `sim_returns_for`
is generic and unchanged.
**Covering test command:**
```
uv run pytest tests/unit/test_sim_curves.py tests/integration/test_dashboard_service.py tests/integration/test_ibkr_account_detail_web.py tests/integration/test_dashboard_ibkr_holdings.py tests/integration/test_paper_books_web.py tests/integration/test_ibkr_sim_web.py -q 2>&1 | tail -20
```
Expect: green; no VRP row/pill; surviving books render.
**Commit:** `fix(vrp): drop vrp from _IBKR_ACCOUNT_SIDS and SIM_BOOKS so ghost VRP row/pill can't render`
---
## Task 8 — Delete the now-purposeless VRP/OPRA CronJob manifest
**File to delete:**
- `infra/k8s/jobs/fxhnt-opra-backfill.yaml` — invokes `fxhnt backfill-xsp-opra` (deleted in
Task 3). With the command gone this CronJob would crash-loop; its only job was freezing OPRA
marks for VRP recompute. (The `fxhnt-vrp-rebalancer.yaml` manifest is already absent from the
tree — nothing to delete there; its dangling test is removed in Task 1.)
**Do NOT touch:** `fxhnt-deribit-carry-job.yaml` (funding/carry, not DVOL),
`fxhnt-bybit-*`, `fxhnt-ucits-rebalancer.yaml`, `fxhnt-ibkr-rebalancer.yaml`, and all other
manifests.
**Covering check:**
```
grep -rln "vrp\|ingest-dvol\|execute-vrp\|backfill-xsp-opra" infra/ # expect: no hits
```
**Commit:** `chore(vrp): remove fxhnt-opra-backfill CronJob (backfill-xsp-opra command deleted)`
---
## Task 9 — Comment/docstring scrubs (no logic change) — LAST
Pure prose edits that dangling-reference the deleted `vrp` strategy id. No imports, logic,
registry entries, or types change. Batched last because none affect importability.
**Files + edits (reword to drop the `vrp` example, keep meaning):**
- `src/fxhnt/adapters/orchestration/definitions.py` — lines ~1012: drop the
"vrp_nav ARCHIVED … KEPT in assets.py but DE-WIRED" paragraph (the "KEPT" claim is now false);
replace with a one-line "VRP removed entirely 2026-07-20" note or delete outright.
- `src/fxhnt/adapters/persistence/cockpit_models.py` — line ~46 (drop the `(… that would have
caught vrp)` parenthetical or generalize to "a dead strategy"); line ~228 (drop `/vrp` from
`(multistrat/vrp …)`).
- `src/fxhnt/adapters/persistence/forward_nav.py` — lines ~146147 (generalize "the vrp freeze
table" / "a vrp recompute is bounded by this max date" to opra-pit-anchored/strategy-generic
wording; the `xsp_freeze_max_date` query stays); line ~199 (drop the "shelved/FALSIFIED
vrp/vrp_exec" example from the `archived: True` registry note).
- `src/fxhnt/adapters/persistence/paper_repo.py` — line ~519 (drop `vrp` from the
`(multistrat/vrp …)` recompute-replay parenthetical).
- `src/fxhnt/adapters/web/app.py` — lines ~449, 453, 473, 482, 551, 567, 728, 752, 810: drop the
`/vrp` example from each `multistrat/vrp` phrase. No code change (the archived-filter
comprehension already yields one fewer element once `vrp` left `SIM_BOOKS`/registry).
- `src/fxhnt/adapters/web/templates/_macros.html` — line ~61 ("2-week vrp rot"): generalize.
- `src/fxhnt/adapters/web/templates/cockpit.html` — line ~78: drop `/vrp` from
"two IBKR real-account books (multistrat/vrp)".
- `src/fxhnt/application/dashboard_service.py` — comment-only lines ~27, 55, 89, 390, 416, 417:
drop/generalize the `vrp` examples (the code literal was already fixed in Task 7).
- `src/fxhnt/application/sim_curves.py` — lines ~2, 12, 22: drop `vrp` from the
`multistrat/vrp` recompute-replay examples (the tuple was already fixed in Task 7).
- `src/fxhnt/application/forward_health.py` — docstring lines ~3, 11, 13, 14, 15, 24, 31 and the
inline comment at ~147 ("for the vrp freeze axis"): generalize away from naming `vrp` as the
mechanism's reason for existing; the `_FREEZE_SOURCES = {"opra-pit"}` logic and
`xsp_freeze_max_date()` call are unchanged.
- `src/fxhnt/application/display_names.py` — lines ~2 and ~51: drop the `vrp` example id /
option-contract example from the illustrative prose.
- `src/fxhnt/application/em_asia_marginal_eval.py` — lines ~6, 19, 71: replace "VRP" in the
Stage-1 kill-gate precedent comments with a VRP-agnostic phrase ("prior failed diversifier
candidates"). No logic change.
- `src/fxhnt/config.py` — line ~215: drop `(… that would have caught vrp)` from the
`stale_after_business_days` comment; keep the setting + default.
- `src/fxhnt/ports/dashboard.py` — lines ~72 and ~190: drop the `vrp` example from the
`multistrat/vrp` IBKR-real-account docstrings. (Line ~34 is unrelated carry — leave.)
- `src/fxhnt/vendor/surfer/__init__.py` — lines ~1113: compress the `surfer_poc`/
`MomentumVrpStrategy` retirement note to a plain historical line without the VRP signpost.
**Do NOT edit** as part of scrubbing (already covered / must not change): the `xsp_option_bars`
query, `_FREEZE_SOURCES`, `XspOptionBarRow`, any settings field, or any Protocol field.
**Covering test command** (broad — scrubs must not have touched logic):
```
uv run python -c "import fxhnt.adapters.web.app, fxhnt.application.dashboard_service, fxhnt.application.forward_health, fxhnt.config, fxhnt.ports.dashboard, fxhnt.application.display_names, fxhnt.application.em_asia_marginal_eval, fxhnt.application.sim_curves"
uv run pytest tests/unit/test_display_names.py tests/integration/test_dashboard_service.py -q 2>&1 | tail -20
```
Expect: all import clean; tests green.
**Commit:** `docs(vrp): scrub stale VRP comment/docstring references across cockpit + shared infra`
---
## Task 10 — Full-suite green gate + removal verification (release gate)
**No file edits.** Final verification against all Global Constraints.
**Commands (all must pass):**
```
# (3) full suite green
uv run pytest -n auto 2>&1 | tail -30 # 0 failed, 0 errors
# (4) no vrp in registry — false STALE alarm gone
uv run python -c "from fxhnt.registry import STRATEGY_REGISTRY; assert not any(k.startswith('vrp') for k in STRATEGY_REGISTRY), list(STRATEGY_REGISTRY); print('no vrp in registry')"
# no orphaned VRP code / imports anywhere in src
grep -rn "import.*vrp_\|VrpStrategy\|deribit_dvol\|dvol_ingest\|black_scholes\|XspOptionBarsRepo\|VolIndexClient\|_backfill_month\|plan_and_record_vrp" src/ # expect: no hits
ls src/fxhnt/application/vrp_*.py 2>&1 # expect: No such file
ls src/fxhnt/application/black_scholes.py src/fxhnt/application/dvol_ingest.py src/fxhnt/adapters/data/deribit_dvol.py src/fxhnt/adapters/persistence/xsp_option_bars.py 2>&1 # expect: No such file (all four)
# (1) OPRA data kept — table model + freeze query still present
grep -n "class XspOptionBarRow" src/fxhnt/adapters/persistence/cockpit_models.py # expect: 1 hit
grep -n "def xsp_freeze_max_date" src/fxhnt/adapters/persistence/forward_nav.py # expect: 1 hit
# (2) Dagster code-location + cockpit app both import clean (no false-execute / no outage)
uv run python -c "import fxhnt.adapters.orchestration.definitions; import fxhnt.adapters.web.app; print('definitions + app import clean')"
# no residual vrp mentions in infra
grep -rln "vrp\|ingest-dvol\|execute-vrp\|backfill-xsp-opra" infra/ # expect: no hits
```
**Constraint 2 spot-check:** confirm `forward_ingest`'s promotion loop and the cockpit gate
default to WAIT for every surviving strategy (the recon-gate never false-PASSes) — covered by
`test_forward_health.py` / `test_dashboard_service.py` in the full run; if either regresses, the
removal touched a shared gate path and must be reverted to the last green task.
**Commit (only after everything above is green):** `chore(vrp): verify full VRP removal — suite green, registry clean, OPRA data intact`
---
## Post-removal note
After merge, update `docs/superpowers/VRP-REMOVAL-TODO.md` and project memory
(`pearl_fxhnt_diversifier_hunt_2026_07_15.md` already says "VRP now fully DELETED (shelved)") to
reflect that the code deletion — not just shelving — is complete, and that the OPRA
`xsp_option_bars` data was retained per Constraint 1.

View File

@@ -0,0 +1,269 @@
# fxhnt CI/CD Pipeline — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this
> plan task-by-task. Steps use checkbox tracking. This plan writes/edits YAML manifests and applies them to a
> LIVE cluster — every apply is gated behind a dry-run + explicit verify; a broken sensor/workflow must never
> silently swallow a push or false-deploy.
**Goal:** A push to `master` on `gitadmin/fxhnt` auto-builds (only when deps/Dockerfile change) and auto-deploys
the fxhnt cockpit+dagster daemons — no manual `kubectl rollout restart`.
**Architecture:** git-sync-first, image-on-demand. A minimal Sensor in `argo-events` hears the shared Gitea
webhook, path-filters the push to pick `mode=build` (Kaniko rebuild) vs `mode=deploy` (rollout only), and submits
the fxhnt-owned `fxhnt-cockpit` WorkflowTemplate into `foxhunt`.
**Tech Stack:** Argo Events v1.9.10 (Sensor), Argo Workflows (Kaniko build + kubectl deploy), Gitea webhook,
Scaleway registry, K8s (bizworx Kapsule).
## Global Constraints
- **Project boundary:** fxhnt-owned objects (WorkflowTemplate, build/deploy pods, secrets, RBAC) live in
`foxhunt`. ONLY the Sensor lives in `argo-events` (unavoidable). No fxhnt secrets in `argo`/`argo-events`; no
cross-namespace RBAC.
- **Node pool:** all build/deploy pods target `nodeSelector: k8s.scaleway.com/pool-name: large` (the
`ci-compile-cpu` pool is DECOMMISSIONED — only `large`/`sfs` exist). A pod pinned to `ci-compile-cpu` never
schedules.
- **Registry:** `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit`, tagged `:<sha7>` AND `:latest`. Secret
`scw-registry` (already in `foxhunt`).
- **Real-money safety:** the pipeline only builds/deploys the cockpit+dagster DAEMONS. It must NEVER touch the
execution CronJobs (ucits/bybit/ibkr rebalancers) or arm any executor.
- **No silent failures:** a Sensor that can't match, or a Workflow that errors, must surface loudly (a failed
push = a visible failed Workflow, never a swallowed no-op). Every manifest is `kubectl apply --dry-run=server`
validated before a real apply.
- **Reproducibility:** the deploy step annotates deployed Deployments with `fxhnt.io/deployed-sha=<sha7>`.
## File Structure
- `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` — the reanimated WorkflowTemplate (ns foxhunt). Supersedes
the old `infra/argo/cockpit-build-deploy.yaml` (which is renamed/removed in the final task).
- `infra/argo/fxhnt-build-sensor.yaml` — the new Sensor (ns argo-events).
- Gitea webhook on `gitadmin/fxhnt` — created via Gitea API/UI (not a repo file; documented in the plan).
---
## Task 1 — Reanimate + fix the WorkflowTemplate (build side)
**Files:**
- Create: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (start from `infra/argo/cockpit-build-deploy.yaml`)
**Interfaces:**
- Produces: a `WorkflowTemplate` named `fxhnt-cockpit` in ns `foxhunt` with params `commit-sha` (default HEAD)
and `mode` (default `build`; values `build`|`deploy`). The Sensor (Task 3) submits Workflows referencing it.
- [ ] **Step 1: Copy the old template as the new file, fix the node pool**
Copy `infra/argo/cockpit-build-deploy.yaml``infra/argo/fxhnt-cockpit-workflowtemplate.yaml`. In BOTH the
`kaniko-build` and `kubectl-deploy` templates, change every
`nodeSelector: { k8s.scaleway.com/pool-name: ci-compile-cpu, ... }` to
`nodeSelector: { k8s.scaleway.com/pool-name: large, topology.kubernetes.io/zone: fr-par-2 }`.
Grep to confirm no `ci-compile-cpu` remains: `grep -n ci-compile-cpu infra/argo/fxhnt-cockpit-workflowtemplate.yaml` → no hits.
- [ ] **Step 2: Add the immutable `:<sha7>` tag to the Kaniko destination**
In the `kaniko-build` container args, alongside the existing
`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`, ADD a second line:
`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:{{=sprig.trunc(7, workflow.parameters.commit-sha)}}`
(the `sprig.trunc` expression matches how `build-image` truncates the sha). Keep `--cache=true` and the
`--cache-repo`.
- [ ] **Step 3: Server-side dry-run validate the template**
Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt`
Expected: `workflowtemplate.argoproj.io/fxhnt-cockpit created (server dry run)` with no schema errors.
(If it errors on an unknown field, the Argo CRD version differs — STOP and report; do not force-apply.)
- [ ] **Step 4: Commit**
```bash
git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml
git commit -m "ci(fxhnt): reanimate cockpit WorkflowTemplate (node pool large, sha7 tag)"
```
---
## Task 2 — Fix the deploy step (verify the dashboard external route first)
**Files:**
- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (the `kubectl-deploy` template's script)
**Interfaces:**
- Consumes: the WorkflowTemplate from Task 1.
- Produces: a deploy step that refreshes the daemons correctly for whichever external route is live.
- [ ] **Step 1: Determine the live dashboard external route (decides restart vs scale-0→1)**
Run these and read the result:
```bash
kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
kubectl get ingress -n foxhunt 2>/dev/null
kubectl -n foxhunt describe svc fxhnt-ingress 2>/dev/null | grep -iE 'endpoints|selector'
```
Decide:
- If `dashboard.fxhnt.ai` is served THROUGH the tailscale sidecar (sidecar present AND no ingress route to the
dashboard) → KEEP the old `scale 0→1` dance (the tailscale session race is real).
- If it is served through `fxhnt-ingress` (an ingress/route points at the dashboard svc; the tailscale sidecar
is vestigial) → REPLACE the scale-dance with a plain `kubectl -n foxhunt rollout restart deploy/fxhnt-dashboard`.
Record the decision in the plan's progress ledger with the evidence. If ambiguous, KEEP `scale 0→1` (the
conservative choice that cannot cause the stale-proxy "down").
- [ ] **Step 2: Apply the deploy-step decision + add the deployed-sha annotation**
Edit the `kubectl-deploy` script. Regardless of the restart mechanism chosen, add BEFORE the rollout:
```
kubectl -n foxhunt annotate deploy/fxhnt-dashboard deploy/dagster \
fxhnt.io/deployed-sha="$(cd /workspace/src && git rev-parse --short HEAD)" --overwrite
```
Keep the existing `kubectl apply -f` of the three manifests (fxhnt-dashboard.yaml, dagster.yaml,
network-policies/fxhnt-cockpit.yaml) and the `rollout status` waits. Keep `rollout restart deploy/dagster`
(dagster has no proxy race).
- [ ] **Step 3: Dry-run validate again**
Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt`
Expected: server dry-run OK.
- [ ] **Step 4: Commit**
```bash
git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml
git commit -m "ci(fxhnt): deploy step — <restart-choice> + deployed-sha annotation"
```
---
## Task 3 — The Sensor (simple, single-trigger) — mode decided IN the workflow
**Design note (verified 2026-07-21):** the Argo Events Sensor CRD here has an OPEN schema (no strict property
validation — `kubectl explain sensor.spec.triggers.template.conditions` errors, and the only `conditions:` in
live sensors are in `status`, not trigger-level). Rather than gamble on a fragile sensor-side path-negation
filter to pick build-vs-deploy, we keep the SENSOR SIMPLE (one trigger, exactly like `build-msp-portal`) and
move the build-vs-deploy decision INTO the workflow, where a robust `git diff --name-only` between the previous
and pushed commit is trivial and testable. This is the more robust design, not a fallback.
**Files:**
- Create: `infra/argo/fxhnt-build-sensor.yaml` (ns argo-events)
- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (add a `decide-mode` step that git-diffs)
**Interfaces:**
- Consumes: the `gitea-webhook` eventsource (shared), the `fxhnt-cockpit` WorkflowTemplate (Task 1/2).
- Produces: a `Sensor` `build-fxhnt` (argo-events) that submits ONE Workflow per master push; the workflow
self-selects build vs deploy.
- [ ] **Step 1: Write the single-trigger Sensor (copy build-msp-portal structure exactly)**
`kubectl get sensor build-msp-portal -n argo-events -o yaml` and mirror it. ONE dependency `push-to-master`:
- `body.ref` == `refs/heads/master`
- `body.pusher.username` != `argocd-image-updater`
(NO `body.commits.#.modified` filter — the mode decision moved to the workflow.)
ONE trigger `submit`: submits a Workflow `generateName: build-fxhnt-`, `namespace: foxhunt`,
`workflowTemplateRef: fxhnt-cockpit`, injecting `commit-sha`=`body.after`. Do NOT set `mode` (the workflow
computes it). This is structurally identical to the proven msp sensors → lowest risk.
- [ ] **Step 2: Add a `decide-mode` step to the WorkflowTemplate**
In `fxhnt-cockpit-workflowtemplate.yaml`, change the DAG so the FIRST step (after git-clone) is `decide-mode`:
a small `alpine/git` script that runs, in the cloned repo at `commit-sha`:
```
CHANGED=$(git diff --name-only "${SHA}~1" "${SHA}" 2>/dev/null || echo "ALL")
if echo "$CHANGED" | grep -qE '^pyproject\.toml$|^infra/docker/fxhnt\.Dockerfile$'; then
echo build; else echo deploy; fi
```
Output the result (`build`/`deploy`) as an Argo output parameter `mode`. Then gate the `kaniko-build` task on
`{{tasks.decide-mode.outputs.parameters.mode}} == build`, and `kubectl-deploy` depends on
`build.Succeeded || build.Skipped` (unchanged). Edge case: if `${SHA}~1` doesn't exist (first commit / shallow),
default to `build` (safe — a rebuild is never wrong, only slower). The explicit `mode` PARAM stays on the
template (default `auto`) so a human can still force `mode=build`/`mode=deploy` on a manual `argo submit`;
`decide-mode` only runs when `mode=auto`.
- [ ] **Step 3: Dry-run validate both**
```bash
kubectl apply -f infra/argo/fxhnt-build-sensor.yaml --dry-run=server -n argo-events
kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt
```
Expected: both server-dry-run OK.
- [ ] **Step 4: Commit**
```bash
git add infra/argo/fxhnt-build-sensor.yaml infra/argo/fxhnt-cockpit-workflowtemplate.yaml
git commit -m "ci(fxhnt): single-trigger sensor + in-workflow git-diff mode decision"
```
---
## Task 4 — Apply to the cluster + wire the Gitea webhook
**Files:** none new — this task applies the manifests and creates the webhook.
- [ ] **Step 1: Apply the WorkflowTemplate + Sensor for real**
```bash
kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml -n foxhunt
kubectl apply -f infra/argo/fxhnt-build-sensor.yaml -n argo-events
kubectl get workflowtemplate fxhnt-cockpit -n foxhunt
kubectl get sensor build-fxhnt -n argo-events
```
Expected: both present. The sensor pod (`kubectl get pods -n argo-events | grep build-fxhnt`) reaches Running.
- [ ] **Step 2: Create the Gitea webhook on `gitadmin/fxhnt`**
The `gitea-webhook` eventsource endpoint is the same target the `bizworx-ops/infrastructure` repo already POSTs
to. Find it: `kubectl get eventsource gitea-webhook -n argo-events -o yaml` (the webhook `port`/`endpoint`) and
how the existing infra-repo webhook is configured (inspect via Gitea admin UI or API on an existing repo).
Create the equivalent webhook on `gitadmin/fxhnt`: push events, active, POST JSON to the eventsource URL. Record
the exact URL used in the ledger. Do NOT paste any webhook secret into the transcript.
- [ ] **Step 3: Verify the webhook reaches the sensor (dry, safe probe)**
Trigger a harmless test push (or use Gitea's "Test Delivery" webhook button) and confirm the sensor received it:
```bash
kubectl logs -n argo-events -l sensor-name=build-fxhnt --tail=50 | grep -iE 'triggered|push-to-master|dispatch'
```
Expected: the sensor logs the received push. If Gitea's test-delivery uses a non-master ref, the sensor
correctly does NOT fire — confirm it's reachable via the eventsource logs instead.
---
## Task 5 — End-to-end verification (both modes) + retire the old file
**Files:**
- Remove: `infra/argo/cockpit-build-deploy.yaml` (superseded)
- [ ] **Step 1: Verify mode=deploy end-to-end (code-only push)**
Make a trivial code-only commit on `master` (e.g. touch a comment in `src/`), push, and watch:
```bash
kubectl get wf -n foxhunt --sort-by=.metadata.creationTimestamp | tail -3
```
Expected: a `build-fxhnt-*` Workflow runs with `mode=deploy`, SKIPS the kaniko-build step, deploys, and the
dashboard stays UP (check `dashboard.fxhnt.ai` reachable post-deploy, and
`kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{.metadata.annotations.fxhnt\.io/deployed-sha}'`
shows the new sha). Total time ~1-2 min, no Kaniko build.
- [ ] **Step 2: Verify mode=build end-to-end (dep push)**
Make a no-op-but-real change to `pyproject.toml` (e.g. reorder a dependency, or bump a harmless comment), push,
and watch: the Workflow runs `mode=build`, Kaniko produces `fxhnt-cockpit:<sha7>`+`:latest`
(`kubectl run` a crane/skopeo check OR verify the workflow's kaniko step succeeded), then deploys. Confirm the
`:<sha7>` tag exists in the registry.
- [ ] **Step 3: Confirm no false trigger + real-money safety**
- Push a change touching ONLY `docs/` → confirm a `mode=deploy` Workflow runs (harmless) OR, if docs are
excluded, no Workflow — either is fine, but it must NOT trigger a build.
- Confirm NO execution CronJob was modified/triggered by any pipeline run:
`kubectl get cronjob -n foxhunt` (ucits/bybit/ibkr rebalancers unchanged, bybit/ibkr still Suspend=True).
- [ ] **Step 4: Retire the old file + commit**
```bash
git rm infra/argo/cockpit-build-deploy.yaml
git commit -m "ci(fxhnt): retire old cockpit-build-deploy (superseded by workflowtemplate + sensor)"
```
- [ ] **Step 5: Final verification summary**
Confirm all Global Constraints hold: sensor in argo-events only, workflow+pods in foxhunt, node pool `large`,
no cross-ns RBAC, no fxhnt secrets in platform ns, real-money CronJobs untouched. Record in the ledger.

View File

@@ -0,0 +1,85 @@
# EUR Fund Ledger — Design Spec (2026-07-20)
**Status:** design approved (scope + cost-basis decided with operator). Implementation plan follows separately.
**> NOT tax advice.** This is an engineering design for a bookkeeping system. The tax *treatment* it records
(Box 3 passive capital, FIFO cost basis, EUR-functional) reflects the operator's resolved structure and must be
confirmed by a Dutch belastingadviseur before it is relied on for a filing. See
`docs/superpowers/2026-07-20-fund-tax-accounting-research-brief.md` and memory
`project_fxhnt_fund_tax_accounting_research`.
## Goal
A tax-ready, append-only, double-entry **EUR ledger** for the fxhnt fund's **passive Box-3 investment book**,
living in the existing Postgres/Timescale warehouse, fed by an automated **IBKR Flex** ingestion Dagster asset.
Phase 1 books **the fund only** — the passive IBKR investment book. The tech-BV bookkeeping and the arm's-length
BV→fund service stay in the operator's existing BV administration (explicitly out of scope).
## Why this shape (context)
- The operator's structure separates **arbeid** (tech-BV, taxed there) from **kapitaal** (private fund, Box 3).
Phase 1 books only the kapitaal side → a clean Box-3 record with no BV entanglement.
- **EUR is the functional currency**, not just presentation. IBKR DU9600528 is EUR-native; if crypto ever goes
live it is Bybit-EU (EUR, not USDT). There is **no USD leg**, so there is **no conversion pipeline** — only a
dormant `currency` + `fx_rate_to_eur` seam (default EUR/1.0) so a stray non-EUR line is representable and
correct rather than silently mis-booked. A pure-EUR book has **no FX P&L** — only trading P&L.
- Reuses existing patterns: SQLAlchemy 2.0 ORM (`Base`/`Mapped`, Postgres+SQLite), a `*Repo` with `migrate()`,
a Dagster asset alongside the nightly ETL. No new infrastructure.
## Global Constraints
1. **EUR-functional.** Every posting stored in EUR. `currency` defaults `"EUR"`, `fx_rate_to_eur` defaults
`1.0`. Build NO conversion pipeline — the FX columns are dormant insurance only.
2. **Append-only.** `journal_entries` / `journal_postings` are immutable. Corrections are reversing entries
(never UPDATE/DELETE of a posted row). Ingestion is idempotent (re-running a Flex pull must not double-book).
3. **Balanced in EUR.** A DB-level trigger (or repo-enforced invariant on both Postgres and SQLite) rejects any
entry whose postings do not sum to zero in `eur_amount`.
4. **FIFO** cost basis, applied consistently. Realized gains come from lot-matching, not cash-flow deltas.
5. **Scope = fund only.** No tech-BV, dividend, or arm's-length-service postings in Phase 1.
6. **Runs on paper now.** The IBKR Flex asset must run against the *paper* account (DU9600528) — the schema and
the EUR-sum-to-zero invariant are validated on paper data before any real capital exists.
7. **Same test discipline as the codebase:** models work on SQLite (unit tests) and Postgres (prod); full suite
green before finishing.
## Schema (SQLAlchemy 2.0 ORM, in a new `ledger` module)
- **`accounts`** — chart of accounts. `account_id` (str PK, e.g. `IBKR-EUR-cash`, `IBKR-securities`,
`contributed-capital`, `trading-pnl`, `fees`), `kind` (asset|liability|equity|income|expense), `venue`,
`currency` (default EUR). Per-venue sub-ledgers each balance standalone.
- **`journal_entries`** — header. `entry_id` (str PK), `entry_date` (date), `description`, `source_venue`,
`strategy_tag` (nullable — the Dagster attribution dimension), `source_ref` (the idempotency key: the IBKR
Flex trade/cash id, so re-ingest is a no-op). Immutable.
- **`journal_postings`** — lines. `posting_id` (PK), `entry_id` (FK), `account_id` (FK), `side` (debit|credit),
`amount`, `currency` (default EUR), `fx_rate_to_eur` (default 1.0), `eur_amount`. Invariant: per `entry_id`,
Σ `eur_amount`(debit) == Σ `eur_amount`(credit).
- **`cost_basis_lots`** — `lot_id` (PK), `account_id`, `asset`, `open_date`, `qty_open`, `qty_remaining`,
`unit_cost_eur`, and on close a link to the realizing entry. FIFO consumption.
- **`fx_rates`** — `value_date` + `pair``rate`. **Dormant** — populated only if a non-EUR asset appears.
Present so the seam exists; no ingestion job in Phase 1.
## IBKR Flex ingestion (new adapter surface + Dagster asset)
The existing `IbkrBroker` is a live `ib_insync` socket connection — NOT the Flex API. Flex is a separate
**Flex Web Service**: a per-query token + query id, a 2-step `SendRequest``GetStatement` poll (~1 req/sec),
returning an XML statement. Add a read-only `IbkrFlexClient` (new file) that fetches and parses:
trades, cash transactions, dividends, corporate actions, positions, NAV.
- Land raw XML/rows in an append-only `raw` schema (mirrors the DB-anchor-SSOT pattern), then a transform
normalizes into `journal_entries`/`journal_postings` (idempotent on `source_ref`).
- One Dagster asset (`fund_ledger_ingest` or similar) alongside the nightly ETL; reuses monitoring patterns.
- W-8BEN note: as a non-US person the account gets a 1042-S, never a Consolidated 1099 — irrelevant to the
EUR ledger math, but archive the annual Activity Statement as an audit doc.
## Out of scope (Phase 1) — deliberately deferred
- Bybit ingestion (no live crypto book; Bybit-EU is EUR anyway → add later, same schema).
- The tech-BV → fund arm's-length service postings, dividend, capital-contribution accounting (stays in BV admin).
- The Box-3 *werkelijk-rendement* parallel tax-basis calc (unrealized + fee-disallow) — only needed if the
tegenbewijsregeling is elected; add when the advisor confirms it's worth it.
- Any FX conversion pipeline (there is no non-EUR leg).
## First build step
The EUR-functional ledger core (models + repo + the EUR-sum-to-zero invariant) **plus** the IBKR Flex ingestion
asset — validated on the paper account — before anything else. This exercises every load-bearing decision:
append-only immutability, the EUR-balanced invariant, the `strategy_tag` dimension, FIFO lots.

View File

@@ -0,0 +1,134 @@
# fxhnt CI/CD Pipeline on bizworx — Design Spec (2026-07-21)
**Status:** design approved (approach + build-trigger decided with operator). Implementation plan follows.
## Goal
Restore automated build+deploy for fxhnt on the bizworx Kapsule so a push to `master` on
`gitadmin/fxhnt` (Gitea) lands live without a manual `kubectl rollout restart` — the exact gap that left the
cockpit on 2.5-day-old code (the CBU0/CSBGU0 fix). **Git-sync-first, image-on-demand:** code changes trigger a
fast rollout (git-sync re-pulls src); only dependency/Dockerfile changes trigger a Kaniko image rebuild.
## Why this shape
The operator's insight: **code changes often, the image changes rarely.** git-sync already pulls fresh `src/`
on every (re)start, so a code-only push needs only a rollout — not a 5-minute Kaniko build. A rebuild is needed
only when `pyproject.toml` / `fxhnt.Dockerfile` change. This is exactly the `mode=build` vs `mode=deploy` split
the retired `infra/argo/cockpit-build-deploy.yaml` already encoded — we reanimate and fix it, rather than adopt
the generic `build-image` template (which lacks the split and fxhnt's deploy specifics).
## Governing principle: shared engine, separate projects
fxhnt and bizworx are **different projects**. fxhnt should *use* bizworx's heavy shared platform infra but not
*pollute* the platform namespaces with fxhnt-specific objects or secrets:
- **SHARE (bizworx platform, use as-is):** the one Gitea (webhook source), the Argo Events eventbus +
`gitea-webhook` eventsource, the Argo Workflows controller (the build *engine*), external-secrets/openbao,
the `large` node-pool, the Scaleway registry.
- **SEPARATE (fxhnt-owned, live in `foxhunt`):** the `WorkflowTemplate`, all build+deploy pods, fxhnt's
secrets (`argo-git-ssh-key`, `scw-registry` — already in `foxhunt`), and fxhnt's RBAC.
- **The ONE unavoidable object in a platform ns:** the Sensor. A Sensor must live in `argo-events` to hear the
shared eventbus. Keep its footprint minimal — it only matches fxhnt pushes and *submits* a Workflow that runs
in `foxhunt`. No fxhnt secrets, no fxhnt logic beyond the trigger, live in `argo-events`.
This is NOT the old "everything in foxhunt because the secrets were there" (path of least resistance) — it's the
same placement for the *right* reason: respecting the project boundary. It also means **no cross-namespace RBAC**
(the deploy runs in `foxhunt`, where the app + its permissions already are) and **no fxhnt secrets replicated
into `argo`**.
## Verified current state (all components already exist on bizworx)
- **One Gitea** (ns `gitea`): `git.fxhnt.ai` and `gitea-http.gitea.svc` are the SAME instance; it holds
`gitadmin/fxhnt` (code) alongside `bizworx-ops/infrastructure` (the msp GitOps monorepo). No second Gitea.
- **Argo Events** (`gitea-webhook` eventsource, `/push`, repo-agnostic) + **Argo Workflows** (Kaniko) — already
driving `build-msp-*` via path-filtered sensors. Reusable.
- **Registry:** fxhnt's image is `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit` (Scaleway) — DIFFERENT from msp's
`registry.bizworx.nl/bizworx-ops/`. Secret `scw-registry` (ns foxhunt) holds the push creds.
- **Dockerfile EXISTS:** `infra/docker/fxhnt.Dockerfile` (python:3.11-slim, `fxhnt serve` :8080). Reusable as-is.
- **SAs exist:** `argo-workflow` (ns argo), `argo-events-sa` (ns argo-events); `argo-git-ssh-key` (ns foxhunt).
- **What's broken in the old pipeline:** (1) `nodeSelector: ci-compile-cpu` — that pool is DECOMMISSIONED
(only `large` + `sfs` pools exist now) → build/deploy pods won't schedule; (2) the Sensor that triggered it
is gone; (3) no webhook on `gitadmin/fxhnt` → the eventsource never hears fxhnt pushes.
- **No fxhnt ArgoCD app** — fxhnt manifests are applied by the workflow's `kubectl apply`, not GitOps-synced.
We keep it that way (see Non-goals).
## Architecture
```
git push → gitadmin/fxhnt (master) [SHARED: Gitea]
│ Gitea webhook ──────────────► argo-events `gitea-webhook` eventsource [SHARED: platform eventbus]
Sensor `build-fxhnt` (ns argo-events — the ONE fxhnt object in a platform ns, kept minimal)
│ filters: ref==refs/heads/master, pusher != argocd-image-updater (SINGLE trigger, like the msp sensors)
▼ submits ONE Workflow into ns `foxhunt`, injecting commit-sha ─────── [FXHNT-OWNED from here down]
Argo Workflow (WorkflowTemplate `fxhnt-cockpit`, ns foxhunt, SA argo-workflow, node-pool `large`) — DAG:
├─ decide-mode: git-clone(sha) → `git diff --name-only sha~1 sha` →
│ touched pyproject.toml OR infra/docker/fxhnt.Dockerfile → mode=build ; else mode=deploy
│ (mode decided IN the workflow, not a fragile sensor filter — the CRD schema is open, so a
│ sensor-side path-negation can't be validated; a git-diff step is robust + testable. No lock file
│ exists → those two paths are the complete rebuild-trigger set.)
├─ mode=build: Kaniko → rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:<sha7>+:latest
└─ (build skipped when mode=deploy)
→ deploy step (in foxhunt, no cross-ns RBAC): kubectl apply manifests + refresh daemons (git-sync re-pulls src)
```
## Components to build/fix
### 1. WorkflowTemplate `fxhnt-cockpit` (reanimate `infra/argo/cockpit-build-deploy.yaml`, ns `foxhunt`)
- Fix `nodeSelector` `ci-compile-cpu``large` (both build + deploy templates — `ci-compile-cpu` pool is gone;
only `large`/`sfs` exist).
- Keep the `mode` DAG: `build` (Kaniko rebuild + deploy) vs `deploy` (skip build, deploy only).
- Keep the Kaniko dest `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit` but ADD an immutable `:<sha7>` tag next to
`:latest` (matches the msp pattern). Cache stays on.
- Clone via the in-cluster Gitea using `argo-git-ssh-key`; push via `scw-registry`. Both already exist in
`foxhunt` — the workflow runs there, so they resolve locally (no replication into a platform ns).
- **Reproducibility despite git-sync:** the deploy step stamps the deployed commit sha onto the daemon
Deployments as an annotation (`fxhnt.io/deployed-sha=<sha7>`), so "which commit is this pod running" is always
answerable — closing the one real gap of git-sync-first without adopting full image-based deploys.
### 2. Sensor `build-fxhnt` (new, model on `build-msp-portal`)
- Dependency: `gitea-webhook` push, filter `body.ref == refs/heads/master` AND
`body.pusher.username != argocd-image-updater` (loop-guard; harmless here since no image-updater writes fxhnt,
but kept for parity/safety).
- **Mode decision via path-filter on `body.commits.#.modified`:** if any modified path matches
`pyproject.toml` | `infra/docker/fxhnt.Dockerfile` → submit workflow with `mode=build`; else `mode=deploy`.
(No lock file exists, so those two paths are the complete rebuild-trigger set.) Mechanism: two triggers with
complementary data-filters (one matching the dep-paths → build, one matching their negation → deploy), or one
trigger whose param is computed — the plan picks whichever Argo Events supports cleanly.
- Inject `commit-sha` from `body.after`.
### 3. Gitea webhook on `gitadmin/fxhnt`
- Add a repo (or org) webhook POSTing push events to the `gitea-webhook` eventsource endpoint (same target the
infrastructure repo already uses). This is the one genuinely-missing wire.
### 4. The deploy step — VERIFY, don't assume (open design point)
The old deploy used **`scale 0→1` for fxhnt-dashboard, NOT `rollout restart`**, because the tailscale sidecar
served `dashboard.fxhnt.ai` and a rolling restart overlapped two tailscale sessions → external proxy read
"down". BUT: the dashboard now also has a `fxhnt-ingress`, and a manual `rollout restart` on 2026-07-21
succeeded cleanly. **The plan must verify which external route is live** before choosing:
- If `dashboard.fxhnt.ai` still resolves through the tailscale sidecar → keep `scale 0→1` (the race is real).
- If it now resolves through `fxhnt-ingress` (tailscale sidecar vestigial) → a plain `rollout restart` is
simpler and correct; drop the scale-dance.
Dagster has no external-proxy race → `rollout restart deploy/dagster` either way.
## Non-goals (YAGNI) — deliberately NOT doing, with the reason
- **No ArgoCD GitOps app for fxhnt.** Chosen over declarative GitOps: it would add a second source of truth for
the pod's src (ArgoCD-synced manifests vs git-sync'd code) and needs a git.fxhnt.ai repo registration. The
imperative `kubectl apply` from the foxhunt workflow fits git-sync-first and keeps manifests in the fxhnt repo.
- **No registry unification.** fxhnt stays on Scaleway `rg.fr-par.scw.cloud/bizworx` (not the shared
`registry.bizworx.nl/bizworx-ops`). Migrating would mean re-pointing the dashboard + reconciling old images
for zero pipeline benefit — a separate, riskier task.
- **No CI workloads in a platform namespace.** Only the Sensor lives in `argo-events` (unavoidable). The
WorkflowTemplate + all build/deploy pods run in `foxhunt` — no fxhnt secrets in `argo`, no cross-ns RBAC.
- **No image-updater for fxhnt.** The sensor writes the `:sha7`+`:latest` tags; the deploy step points the
daemons at the fresh image. No git-write-back loop needed.
- **No change to the execution CronJobs** (ucits/bybit/ibkr rebalancers) — they git-sync per run already.
## Success criteria
1. A code-only push to `master` → cockpit reflects it within ~12 min WITHOUT a manual restart, NO Kaniko build.
2. A `pyproject.toml`/Dockerfile push → a Kaniko rebuild produces `fxhnt-cockpit:<sha7>`+`:latest`, then deploys.
3. The dashboard external route (`dashboard.fxhnt.ai`) stays UP across a deploy (no stale-proxy "down").
4. No build-loop, no false trigger on unrelated pushes.
5. Real-money safety unaffected — the pipeline only builds/deploys the cockpit+dagster daemons, never arms exec.

View File

@@ -1,135 +0,0 @@
# Argo WorkflowTemplate — build + deploy the fxhnt cockpit, fully in-cluster.
#
# Reuses cluster primitives: the `argo-workflow` ServiceAccount, the `argo-git-ssh-key` deploy key (clone
# the fxhnt repo over in-cluster Gitea SSH, gitea-sshd:22), and the `scw-registry` dockerconfig (push to
# the external Scaleway Container Registry rg.fr-par.scw.cloud/bizworx). Build runs in-cluster.
# (Phase 2B: migrated off the in-cluster GitLab shell + registry.)
#
# Submit: argo submit -n foxhunt --from=wftmpl/fxhnt-cockpit -p commit-sha=<sha> --watch
# (or use scripts/argo-deploy-cockpit.sh)
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: fxhnt-cockpit
namespace: foxhunt
labels: { app.kubernetes.io/name: fxhnt-cockpit, app.kubernetes.io/part-of: foxhunt }
spec:
serviceAccountName: argo-workflow
entrypoint: pipeline
archiveLogs: true # archive build/deploy logs to MinIO (argo-logs bucket). Reachable now that the
# pods carry app.kubernetes.io/part-of: argo-workflows (MinIO ingress netpol allow).
activeDeadlineSeconds: 1800
ttlStrategy: { secondsAfterCompletion: 3600 }
podGC: { strategy: OnPodCompletion }
arguments:
parameters:
- { name: commit-sha, value: HEAD }
- { name: mode, value: build } # "build" = kaniko rebuild + deploy; "deploy" = skip build, just apply + rollout restart (git-sync re-pulls code). The sensor routes src-only pushes to "deploy".
templates:
- name: pipeline
dag:
tasks:
# Skip the ~5min kaniko build for code-only pushes (mode=deploy) — git-sync re-pulls src/ on the
# rollout restart, so only dep/Dockerfile changes need a new image (mode=build).
- { name: build, template: kaniko-build, when: "{{workflow.parameters.mode}} == build" }
- { name: deploy, template: kubectl-deploy, depends: "build.Succeeded || build.Skipped" }
# --- build: Kaniko builds infra/docker/fxhnt.Dockerfile -> internal registry ---
- name: kaniko-build
metadata:
labels: { app.kubernetes.io/part-of: argo-workflows } # MinIO ingress allows part-of in {foxhunt,foxhunt-ci,argo-workflows} → log archival works. NOT 'foxhunt' (that adds argo-base-egress which blocks registry:5000/git:2222).
nodeSelector: { k8s.scaleway.com/pool-name: ci-compile-cpu, topology.kubernetes.io/zone: fr-par-2 }
tolerations: [{ effect: NoSchedule, key: node.cilium.io/agent-not-ready, operator: Exists }]
initContainers:
- name: git-clone
image: alpine/git:latest
command: [/bin/sh, -c]
args:
- |
set -ex
mkdir -p /root/.ssh
cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
chmod 600 /root/.ssh/id_ed25519
printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > /root/.ssh/config
git clone --no-checkout --filter=blob:none \
"ssh://git@gitea-sshd.foxhunt.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
cd /workspace/src && git checkout "{{workflow.parameters.commit-sha}}"
echo "checked out $(git rev-parse --short HEAD)"
volumeMounts:
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
- { name: workspace, mountPath: /workspace }
container:
image: gcr.io/kaniko-project/executor:debug
command: [/busybox/sh, -c]
args:
- |
/kaniko/executor \
--context=/workspace/src \
--dockerfile=/workspace/src/infra/docker/fxhnt.Dockerfile \
--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest \
--cache=true \
--cache-repo=rg.fr-par.scw.cloud/bizworx/cache \
--snapshot-mode=redo
env:
- { name: DOCKER_CONFIG, value: /kaniko/.docker }
resources: { requests: { cpu: "2", memory: 4Gi }, limits: { cpu: "6", memory: 12Gi } }
volumeMounts:
- { name: registry-auth, mountPath: /kaniko/.docker, readOnly: true }
- { name: workspace, mountPath: /workspace }
volumes:
- { name: workspace, emptyDir: {} }
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
- name: registry-auth
secret:
secretName: scw-registry
items: [{ key: .dockerconfigjson, path: config.json }]
# --- deploy: apply manifests from the repo + roll the dashboard to the fresh image ---
- name: kubectl-deploy
metadata:
labels: { app.kubernetes.io/part-of: argo-workflows } # MinIO ingress allow → log archival works (see kaniko-build note)
nodeSelector: { k8s.scaleway.com/pool-name: platform, topology.kubernetes.io/zone: fr-par-2 }
serviceAccountName: argo-workflow
initContainers:
- name: git-clone
image: alpine/git:latest
command: [/bin/sh, -c]
args:
- |
set -ex
mkdir -p /root/.ssh
cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
chmod 600 /root/.ssh/id_ed25519
printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > /root/.ssh/config
git clone --no-checkout --filter=blob:none \
"ssh://git@gitea-sshd.foxhunt.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
cd /workspace/src && git checkout "{{workflow.parameters.commit-sha}}"
volumeMounts:
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
- { name: workspace, mountPath: /workspace }
container:
image: alpine/k8s:1.31.0 # Alpine-based (has /bin/sh + kubectl). NOT registry.k8s.io/kubectl — that's distroless (no shell) and the deploy runs a shell script. bitnami/kubectl was removed from Docker Hub (2025).
command: [/bin/sh, -c]
args:
- |
set -e
cd /workspace/src
kubectl apply -f infra/k8s/services/fxhnt-dashboard.yaml
kubectl apply -f infra/k8s/orchestration/dagster.yaml
kubectl apply -f infra/k8s/network-policies/fxhnt-cockpit.yaml
# Dashboard: scale 0 -> 1, NOT `rollout restart`. Its tailscale sidecar serves dashboard.fxhnt.ai;
# a rolling restart briefly overlaps two tailscale sessions on the same identity -> session race ->
# the external proxy link goes stale and the cockpit reads "down" post-deploy. Scaling to 0 cleanly
# terminates the session before a fresh, non-overlapping one comes up; the git-sync initContainer
# re-pulls src the same way, so code still updates. See reference_fxhnt_dashboard_proxy.
kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=0
kubectl -n foxhunt wait --for=delete pod -l app.kubernetes.io/name=fxhnt-dashboard --timeout=90s || true
kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=1
kubectl -n foxhunt rollout restart deploy/dagster # dagster has no external-proxy race
kubectl -n foxhunt rollout status deploy/fxhnt-dashboard --timeout=180s
kubectl -n foxhunt rollout status deploy/dagster --timeout=300s
volumeMounts:
- { name: workspace, mountPath: /workspace }
volumes:
- { name: workspace, emptyDir: {} }
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }

View File

@@ -0,0 +1,57 @@
# Argo Events Sensor — hears the shared Gitea webhook for pushes to gitadmin/fxhnt master, and submits the
# fxhnt-cockpit WorkflowTemplate (ns foxhunt). Structurally identical to build-msp-portal (single dependency,
# single trigger) — the build-vs-deploy decision is made INSIDE the workflow via `git diff` (decide-mode step in
# fxhnt-cockpit-workflowtemplate.yaml), not via a sensor-side path filter. See
# docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md Task 3 design note: the Sensor CRD here has an open
# schema, so trigger-level path-filter negation can't be reliably validated — keeping the sensor simple is the
# robust choice.
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: build-fxhnt
namespace: argo-events
labels: { app.kubernetes.io/instance: build-pipeline }
spec:
dependencies:
- name: push-to-master
eventSourceName: gitea-webhook
eventName: push
filters:
dataLogicalOperator: and
data:
- path: body.ref
type: string
comparator: "="
value: ["refs/heads/master"]
- path: body.pusher.username
type: string
comparator: "!="
value: ["argocd-image-updater"]
template:
serviceAccountName: argo-events-sa
triggers:
- template:
name: submit-build
argoWorkflow:
operation: submit
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: build-fxhnt-
namespace: foxhunt
spec:
workflowTemplateRef:
name: fxhnt-cockpit
arguments:
parameters:
- name: commit-sha
value: ""
- name: mode
value: auto
parameters:
- src:
dependencyName: push-to-master
dataKey: body.after
dest: spec.arguments.parameters.0.value

View File

@@ -0,0 +1,124 @@
# RBAC for the fxhnt CI pipeline — all fxhnt-owned, in the foxhunt namespace (project boundary).
#
# Fills the two gaps the first real workflow run surfaced (2026-07-21):
# 1. The fxhnt-cockpit WorkflowTemplate references SA `argo-workflow`, which existed in `argo` (where msp
# builds) but NOT in `foxhunt` (where fxhnt's project boundary places its build/deploy pods).
# 2. The build-fxhnt Sensor (argo-events-sa) submits Workflows into `foxhunt`; the bizworx `workflow-submitter`
# Role only granted that in `argo`, so a cross-namespace submit into foxhunt was forbidden.
#
# Least privilege: the workflow SA gets exactly what Argo needs to run pods + what the kubectl-deploy step does
# (apply/scale/annotate/rollout-status the two daemon Deployments and their Services/NetworkPolicies), nothing more.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: argo-workflow
namespace: foxhunt
labels: { app.kubernetes.io/part-of: fxhnt-ci }
---
# Argo needs the workflow SA to write task results (the minimum the workflow-controller requires per step).
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: fxhnt-argo-workflow
namespace: foxhunt
labels: { app.kubernetes.io/part-of: fxhnt-ci }
rules:
- apiGroups: ["argoproj.io"]
resources: ["workflowtaskresults"]
verbs: ["create", "patch"]
---
# The kubectl-deploy step's exact actions on the foxhunt daemons — scoped to what it runs, no wildcards.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: fxhnt-ci-deployer
namespace: foxhunt
labels: { app.kubernetes.io/part-of: fxhnt-ci }
rules:
# apply + scale + annotate + rollout-status the daemon Deployments (dashboard, dagster).
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["apps"]
resources: ["deployments/scale"]
verbs: ["get", "update", "patch"]
- apiGroups: ["apps"]
resources: ["replicasets"]
verbs: ["get", "list", "watch"] # rollout status reads the RS
# apply the Services the manifests contain, and `wait --for=delete pod` + rollout status read pods.
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
# the fxhnt-dashboard + dagster manifests also bundle ConfigMaps (the tailscale sidecar serve-config) —
# kubectl apply of the whole file must be able to read/apply them. NOTE (CI review M1): the tailscale
# ServiceAccount + Role + RoleBinding objects were MOVED out of these app manifests into
# infra/k8s/rbac/tailscale-sidecars.yaml (applied once, out-of-band, by a human) specifically so this
# deployer SA does not need roles/rolebindings/serviceaccounts create-update-patch — that combination on a
# webhook-triggered SA is a privilege-escalation primitive. Neither manifest applied by CI creates a
# ServiceAccount anymore, so `serviceaccounts` is dropped from this grant too.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
# the manifests apply NetworkPolicies (fxhnt-cockpit.yaml + dagster's).
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: fxhnt-argo-workflow
namespace: foxhunt
labels: { app.kubernetes.io/part-of: fxhnt-ci }
subjects:
- { kind: ServiceAccount, name: argo-workflow, namespace: foxhunt }
roleRef:
kind: Role
name: fxhnt-argo-workflow
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: fxhnt-ci-deployer
namespace: foxhunt
labels: { app.kubernetes.io/part-of: fxhnt-ci }
subjects:
- { kind: ServiceAccount, name: argo-workflow, namespace: foxhunt }
roleRef:
kind: Role
name: fxhnt-ci-deployer
apiGroup: rbac.authorization.k8s.io
---
# Cross-namespace: let the shared argo-events-sa submit Workflows into foxhunt (the sensor lives in argo-events
# but its Workflow runs here). Mirrors the bizworx `workflow-submitter` Role, scoped to foxhunt.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: fxhnt-workflow-submitter
namespace: foxhunt
labels: { app.kubernetes.io/part-of: fxhnt-ci }
rules:
- apiGroups: ["argoproj.io"]
resources: ["workflows"]
verbs: ["create", "get", "list", "watch"]
- apiGroups: ["argoproj.io"]
resources: ["workflowtemplates"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: argo-events-sa-fxhnt-submitter
namespace: foxhunt
labels: { app.kubernetes.io/part-of: fxhnt-ci }
subjects:
- { kind: ServiceAccount, name: argo-events-sa, namespace: argo-events }
roleRef:
kind: Role
name: fxhnt-workflow-submitter
apiGroup: rbac.authorization.k8s.io

View File

@@ -0,0 +1,208 @@
# Argo WorkflowTemplate — build + deploy the fxhnt cockpit, fully in-cluster.
#
# Reuses cluster primitives: the `argo-workflow` ServiceAccount (foxhunt, see fxhnt-ci-rbac.yaml), the
# `argo-git-ssh-key` deploy key (clone the fxhnt repo over in-cluster Gitea SSH, gitea-ssh.gitea.svc:22), and
# the `scw-registry` dockerconfig (push to the external Scaleway Container Registry rg.fr-par.scw.cloud/bizworx).
#
# Submit: argo submit -n foxhunt --from=wftmpl/fxhnt-cockpit -p commit-sha=<sha> -p mode=<auto|build|deploy> --watch
# Normally auto-triggered: a push to gitadmin/fxhnt master -> the build-fxhnt Sensor (argo-events) submits this.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: fxhnt-cockpit
namespace: foxhunt
labels: { app.kubernetes.io/name: fxhnt-cockpit, app.kubernetes.io/part-of: foxhunt }
spec:
serviceAccountName: argo-workflow
entrypoint: pipeline
archiveLogs: false # archive build/deploy logs to MinIO (argo-logs bucket). Reachable now that the
# pods carry app.kubernetes.io/part-of: argo-workflows (MinIO ingress netpol allow).
activeDeadlineSeconds: 1800
ttlStrategy: { secondsAfterCompletion: 3600 }
podGC: { strategy: OnPodCompletion }
arguments:
parameters:
- { name: commit-sha, value: HEAD }
- { name: mode, value: auto } # "auto" = decide-mode git-diffs commit-sha~1..commit-sha to pick build/deploy; "build" = force kaniko rebuild + deploy; "deploy" = force skip build, just apply + rollout restart (git-sync re-pulls code). A human can still force build/deploy via `argo submit -p mode=...`.
templates:
- name: pipeline
dag:
tasks:
# decide-mode: when mode=auto, git-diffs the pushed commit against its parent to pick build vs deploy
# (dep/Dockerfile changes -> build; anything else, or a missing parent, -> build/deploy per script logic
# below). When mode is forced (build|deploy), decide-mode just echoes it back unchanged.
- { name: decide-mode, template: decide-mode }
# Skip the ~5min kaniko build for code-only pushes (mode=deploy) — git-sync re-pulls src/ on the
# rollout restart, so only dep/Dockerfile changes need a new image (mode=build).
- { name: build, template: kaniko-build, depends: decide-mode, when: "{{tasks.decide-mode.outputs.parameters.mode}} == build" }
- { name: deploy, template: kubectl-deploy, depends: "decide-mode && (build.Succeeded || build.Skipped)" }
# --- decide-mode: resolve "auto" to build|deploy via git diff of the pushed commit vs its parent ---
- name: decide-mode
metadata:
labels: { app.kubernetes.io/part-of: argo-workflows }
nodeSelector: { k8s.scaleway.com/pool-name: large }
tolerations: [{ effect: NoSchedule, key: node.cilium.io/agent-not-ready, operator: Exists }]
outputs:
parameters:
- name: mode
valueFrom: { path: /workspace/mode }
container:
image: alpine/git:latest
command: [/bin/sh, -c]
# SECURITY: commit-sha/mode arrive via ENV (not inline {{...}} substitution) so an untrusted webhook
# payload cannot inject shell. commit-sha is validated hex before use; mode is checked against an allowlist.
env:
- { name: COMMIT_SHA, value: "{{workflow.parameters.commit-sha}}" }
- { name: MODE, value: "{{workflow.parameters.mode}}" }
# Absolute key path via GIT_SSH_COMMAND — Argo's emissary executor does not set HOME=/root, so the
# old `cp key -> ~/.ssh` idiom left git unable to find the key (clone failed with exit 128). This is
# HOME-independent and verified to clone under HOME=/nonexistent.
- { name: GIT_SSH_COMMAND, value: "ssh -i /etc/git-ssh/ssh-privatekey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" }
args:
- |
set -e
case "$COMMIT_SHA" in *[!0-9a-fA-F]*|"") echo "refusing non-hex commit-sha" >&2; exit 1 ;; esac
case "$MODE" in auto|build|deploy) : ;; *) echo "refusing unknown mode: $MODE" >&2; exit 1 ;; esac
git clone --filter=blob:none \
"ssh://git@gitea-ssh.gitea.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
cd /workspace/src && git checkout "$COMMIT_SHA"
if [ "$MODE" != "auto" ]; then
printf '%s' "$MODE" > /workspace/mode
else
RESOLVED="$(git rev-parse "$COMMIT_SHA")"
if CHANGED=$(git diff --name-only "${RESOLVED}~1" "${RESOLVED}" 2>/dev/null); then
# diff succeeded (parent exists): build only if a dep/Dockerfile file changed, else deploy.
if echo "$CHANGED" | grep -qE '^pyproject\.toml$|^infra/docker/fxhnt\.Dockerfile$'; then
printf 'build' > /workspace/mode
else
printf 'deploy' > /workspace/mode
fi
else
# diff failed (no parent -- first commit / shallow history): default to build (safe, never wrong).
printf 'build' > /workspace/mode
fi
fi
cat /workspace/mode
volumeMounts:
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
- { name: workspace, mountPath: /workspace }
volumes:
- { name: workspace, emptyDir: {} }
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
# --- build: Kaniko builds infra/docker/fxhnt.Dockerfile -> internal registry ---
- name: kaniko-build
metadata:
labels: { app.kubernetes.io/part-of: argo-workflows } # MinIO ingress allows part-of in {foxhunt,foxhunt-ci,argo-workflows} → log archival works. NOT 'foxhunt' (that adds argo-base-egress which blocks registry:5000/git:2222).
nodeSelector: { k8s.scaleway.com/pool-name: large }
tolerations: [{ effect: NoSchedule, key: node.cilium.io/agent-not-ready, operator: Exists }]
initContainers:
- name: git-clone
image: alpine/git:latest
command: [/bin/sh, -c]
# SECURITY: commit-sha via ENV + hex-validated (see decide-mode note) — no inline shell substitution.
env:
- { name: COMMIT_SHA, value: "{{workflow.parameters.commit-sha}}" }
# Absolute key path via GIT_SSH_COMMAND — HOME-independent (Argo's emissary doesn't set HOME=/root).
- { name: GIT_SSH_COMMAND, value: "ssh -i /etc/git-ssh/ssh-privatekey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" }
args:
- |
set -ex
case "$COMMIT_SHA" in *[!0-9a-fA-F]*|"") echo "refusing non-hex commit-sha" >&2; exit 1 ;; esac
git clone --no-checkout --filter=blob:none \
"ssh://git@gitea-ssh.gitea.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
cd /workspace/src && git checkout "$COMMIT_SHA"
echo "checked out $(git rev-parse --short HEAD)"
volumeMounts:
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
- { name: workspace, mountPath: /workspace }
container:
image: gcr.io/kaniko-project/executor:v1.23.2
# Kaniko's own entrypoint + a YAML args LIST (the msp build-image pattern) — NOT `sh -c` with one big
# string. A shell wrapper made busybox try to parse the `{{=sprig.trunc(...)}}` expansion and choke on
# the `(` (syntax error). As a plain args list, Argo evaluates the expression and hands it straight to
# Kaniko, no shell in between.
args:
- --context=/workspace/src
- --dockerfile=/workspace/src/infra/docker/fxhnt.Dockerfile
- --destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
- --destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:{{workflow.parameters.commit-sha}}
- --cache=true
- --cache-repo=rg.fr-par.scw.cloud/bizworx/cache
- --snapshot-mode=redo
env:
- { name: DOCKER_CONFIG, value: /kaniko/.docker }
resources: { requests: { cpu: "1", memory: 3Gi }, limits: { cpu: "3", memory: 10Gi } }
volumeMounts:
- { name: registry-auth, mountPath: /kaniko/.docker, readOnly: true }
- { name: workspace, mountPath: /workspace }
volumes:
- { name: workspace, emptyDir: {} }
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
- name: registry-auth
secret:
secretName: scw-registry
items: [{ key: .dockerconfigjson, path: config.json }]
# --- deploy: apply manifests from the repo + roll the dashboard to the fresh image ---
- name: kubectl-deploy
metadata:
labels: { app.kubernetes.io/part-of: argo-workflows } # MinIO ingress allow → log archival works (see kaniko-build note)
nodeSelector: { k8s.scaleway.com/pool-name: large }
serviceAccountName: argo-workflow
initContainers:
- name: git-clone
image: alpine/git:latest
command: [/bin/sh, -c]
# SECURITY: commit-sha via ENV + hex-validated — no inline shell substitution.
env:
- { name: COMMIT_SHA, value: "{{workflow.parameters.commit-sha}}" }
# Absolute key path via GIT_SSH_COMMAND — HOME-independent (Argo's emissary doesn't set HOME=/root).
- { name: GIT_SSH_COMMAND, value: "ssh -i /etc/git-ssh/ssh-privatekey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" }
args:
- |
set -ex
case "$COMMIT_SHA" in *[!0-9a-fA-F]*|"") echo "refusing non-hex commit-sha" >&2; exit 1 ;; esac
git clone --no-checkout --filter=blob:none \
"ssh://git@gitea-ssh.gitea.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
cd /workspace/src && git checkout "$COMMIT_SHA"
volumeMounts:
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
- { name: workspace, mountPath: /workspace }
container:
image: alpine/k8s:1.31.0 # Alpine-based (has /bin/sh + kubectl). NOT registry.k8s.io/kubectl — that's distroless (no shell) and the deploy runs a shell script. bitnami/kubectl was removed from Docker Hub (2025).
command: [/bin/sh, -c]
args:
- |
set -e
cd /workspace/src
kubectl apply -f infra/k8s/services/fxhnt-dashboard.yaml
kubectl apply -f infra/k8s/orchestration/dagster.yaml
kubectl apply -f infra/k8s/network-policies/fxhnt-cockpit.yaml
kubectl -n foxhunt annotate deploy/fxhnt-dashboard deploy/dagster \
fxhnt.io/deployed-sha="$(cd /workspace/src && git rev-parse --short HEAD)" --overwrite
# Dashboard: scale 0 -> 1, NOT `rollout restart`. Its tailscale sidecar serves dashboard.fxhnt.ai;
# a rolling restart briefly overlaps two tailscale sessions on the same identity -> session race ->
# the external proxy link goes stale and the cockpit reads "down" post-deploy. Scaling to 0 cleanly
# terminates the session before a fresh, non-overlapping one comes up; the git-sync initContainer
# re-pulls src the same way, so code still updates. See reference_fxhnt_dashboard_proxy.
#
# SAFETY (CI review I1): once we scale to 0, a trap guarantees the deployment is scaled BACK to 1 on
# ANY exit path — so a failure between scale-0 and scale-1 (transient apiserver error, throttle) can
# never strand the production dashboard at 0 replicas. The trap is best-effort (its own failure is
# swallowed) and cleared on success.
trap 'kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=1 || true' EXIT
kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=0
kubectl -n foxhunt wait --for=delete pod -l app.kubernetes.io/name=fxhnt-dashboard --timeout=90s || true
kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=1
trap - EXIT # scaled back up cleanly; no longer need the safety net
kubectl -n foxhunt rollout restart deploy/dagster # dagster has no external-proxy race
kubectl -n foxhunt rollout status deploy/fxhnt-dashboard --timeout=180s
kubectl -n foxhunt rollout status deploy/dagster --timeout=300s
volumeMounts:
- { name: workspace, mountPath: /workspace }
volumes:
- { name: workspace, emptyDir: {} }
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }

View File

@@ -1,99 +0,0 @@
# ============================================================================================
# ONE-TIME MANUAL APPLY ONLY. Do NOT add this file to any automated/bulk `kubectl apply -f infra/k8s/jobs/`
# path or CI pipeline. It is a `kind: Job` (not a CronJob), so there is no `suspend` flag to gate it —
# submitting this manifest starts the backfill immediately. Apply it by hand when ready:
# kubectl apply -f infra/k8s/jobs/fxhnt-opra-backfill.yaml
# and delete it (`kubectl delete job/fxhnt-opra-backfill -n foxhunt`) before ever re-applying.
# ============================================================================================
#
# One-time historical freeze: `fxhnt backfill-xsp-opra --start 2013-04-01 --max-cost 25.0`, from 2013-04-01
# (XSP's mini-SPX option listing era) through today, freezing the WIDE DTE[1,45]/moneyness[0.70,1.20] put
# slice + near-ATM calls into `xsp_option_bars`. BATCHED per MONTH (`_backfill_month`): one range `definition`
# + one range `ohlcv-1d` fetch per month, then every trading day processed LOCALLY — ~2 OPRA calls/month vs
# the old ~6-8/day (~1h vs ~45h over 2013->). `--max-cost 25.0` is the per-MONTH cost cap (the guard stays
# ACTIVE, never disabled; a month over budget is logged + skipped, never a silent partial-freeze).
#
# Idempotent/resumable: `upsert_bars` merges on (osi_symbol, date), so RE-APPLYING this exact manifest after
# an interruption is safe (already-frozen days are silently re-written, not duplicated) — it just re-spends
# the OPRA cost on the days already done. To resume WITHOUT re-paying for completed months, read the last
# "reached YYYY-MM" line from `kubectl logs`, delete the Job, and re-apply with `--start` bumped to that
# month before resubmitting.
apiVersion: batch/v1
kind: Job
metadata:
name: fxhnt-opra-backfill
namespace: foxhunt
labels: { app.kubernetes.io/name: fxhnt-opra-backfill, app.kubernetes.io/part-of: foxhunt }
spec:
backoffLimit: 2
ttlSecondsAfterFinished: 604800 # 7d — keep the finished Job around for log inspection before GC
template:
metadata:
labels:
app.kubernetes.io/name: fxhnt-opra-backfill # egress NetworkPolicy selects this
app.kubernetes.io/part-of: foxhunt # gitea ingress (git-sync) + base egress
spec:
restartPolicy: Never
imagePullSecrets:
- name: scw-registry
initContainers:
- name: git-sync
image: alpine/git:latest
command: [/bin/sh, -c]
args:
- |
mkdir -p /root/.ssh && cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519 && chmod 600 /root/.ssh/id_ed25519
printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > /root/.ssh/config
rm -rf /code/src
git clone --depth 1 --branch master --filter=blob:none --sparse \
"ssh://git@gitea-ssh.gitea.svc.cluster.local:22/gitadmin/fxhnt.git" /tmp/repo
cd /tmp/repo && git sparse-checkout set src && [ -d /tmp/repo/src ]
cp -a /tmp/repo/src /code/src && echo "git-sync OK: $(git rev-parse --short HEAD)"
resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { cpu: 500m, memory: 256Mi } }
volumeMounts:
- { name: code, mountPath: /code }
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
containers:
- name: backfill
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
command: ["/bin/sh", "-c"]
args:
- |
pip install --quiet --no-cache-dir databento==0.79.0
fxhnt backfill-xsp-opra --start 2013-04-01 --max-cost 25.0
env:
- { name: PYTHONPATH, value: /code/src }
- { name: FXHNT_OPERATIONAL_DSN, value: "postgresql+psycopg://foxhunt@postgres.foxhunt.svc.cluster.local:5432/fxhnt" }
- name: PGPASSWORD
valueFrom: { secretKeyRef: { name: db-credentials, key: password } }
- name: DATABENTO_API_KEY
valueFrom: { secretKeyRef: { name: databento-credentials, key: api-key } }
- { name: PYTHONUNBUFFERED, value: "1" } # stream monthly progress logs live
resources:
requests: { memory: "512Mi", cpu: "250m" }
limits: { memory: "1Gi", cpu: "1" }
volumeMounts:
- { name: code, mountPath: /code, readOnly: true }
volumes:
- { name: code, emptyDir: {} }
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
---
# Egress: DNS + Postgres (5432, write xsp_option_bars) + gitea-sshd (22, git-sync) + external 443 (OPRA via
# Databento historical API). No IBKR/ib-gateway needed — this Job never places an order.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: fxhnt-opra-backfill
namespace: foxhunt
labels: { app.kubernetes.io/part-of: foxhunt }
spec:
podSelector: { matchLabels: { app.kubernetes.io/name: fxhnt-opra-backfill } }
policyTypes: [Egress]
egress:
- ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]
- to: [{ podSelector: { matchLabels: { app.kubernetes.io/name: postgres } } }]
ports: [{ port: 5432, protocol: TCP }]
- to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: gitea } }, podSelector: { matchLabels: { app.kubernetes.io/name: gitea } } }]
ports: [{ port: 2222, protocol: TCP }, { port: 22, protocol: TCP }]
- to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16] } }] # OPRA 443
ports: [{ port: 443, protocol: TCP }]

View File

@@ -28,8 +28,10 @@ spec:
- ports: # DNS
- { port: 53, protocol: UDP }
- { port: 53, protocol: TCP }
- to: # tailscale sidecar -> coordination/DERP (public internet)
- ipBlock: { cidr: 0.0.0.0/0, except: [10.32.0.0/16, 100.64.0.0/15] }
- to: [{ ipBlock: { cidr: 172.16.0.11/32 } }] # kube-apiserver REAL endpoint (kube-proxy DNATs kubernetes.default.svc 10.32.0.1:443 -> this before NP eval). The tailscale sidecar reads/writes its ts-state Secret via the apiserver — WITHOUT this it crashloops "context deadline exceeded" on a fresh start (e.g. after the deploy scale-0->1), causing a bad gateway. Mirrors the dagster NP fix (06d12fd).
ports: [{ port: 6443, protocol: TCP }]
- to: # tailscale sidecar -> coordination/DERP (public internet). except now includes 172.16.0.0/16 so the apiserver rule above is the only cluster-internal path.
- ipBlock: { cidr: 0.0.0.0/0, except: [10.32.0.0/16, 100.64.0.0/15, 172.16.0.0/16] }
ports:
- { port: 443, protocol: TCP }
- { port: 3478, protocol: UDP }

View File

@@ -40,8 +40,8 @@ data:
config:
service_account_name: tailscale-dagster
job_namespace: foxhunt
job_image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s-25e3d7c
image_pull_policy: IfNotPresent
job_image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
image_pull_policy: Always # :latest -> Always-pull so a dep-bump build reaches launched run Jobs (CI review C1)
image_pull_secrets:
- name: scw-registry
load_incluster_config: true
@@ -115,36 +115,6 @@ data:
}
}
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: tailscale-dagster, namespace: foxhunt }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: tailscale-dagster, namespace: foxhunt }
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch", "list", "watch"]
# K8sRunLauncher: the daemon launches each Dagster run as its OWN K8s Job (own memory) so the heavy
# bybit_book_precompute asset (6-8Gi) never OOMs the 4Gi daemon. Needs to manage Jobs + read their pods/logs.
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "get", "list", "watch", "delete"]
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: tailscale-dagster, namespace: foxhunt }
subjects:
- { kind: ServiceAccount, name: tailscale-dagster, namespace: foxhunt }
roleRef: { kind: Role, name: tailscale-dagster, apiGroup: rbac.authorization.k8s.io }
---
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -201,7 +171,7 @@ spec:
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
containers:
- name: webserver
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s-25e3d7c
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
command: ["dagster-webserver", "-h", "0.0.0.0", "-p", "3000",
"-w", "/dagster-home/workspace.yaml"]
env: &dagster_env
@@ -229,7 +199,7 @@ spec:
- { name: code, mountPath: /code, readOnly: true } # git-sync'd src (PYTHONPATH=/code/src)
resources: { requests: { cpu: 100m, memory: 512Mi }, limits: { cpu: "1", memory: 1Gi } }
- name: daemon
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s-25e3d7c
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
command: ["dagster-daemon", "run", "-w", "/dagster-home/workspace.yaml"]
env: *dagster_env
volumeMounts:

View File

@@ -0,0 +1,72 @@
# Tailscale sidecar RBAC (ServiceAccount + Role + RoleBinding) for the fxhnt-dashboard and dagster
# Deployments. Moved out of infra/k8s/services/fxhnt-dashboard.yaml and infra/k8s/orchestration/dagster.yaml
# (CI review M1): those app manifests are applied every deploy by the fxhnt-ci-deployer ServiceAccount via
# the kubectl-deploy Argo template, and having RBAC objects bundled in there meant the deployer SA needed
# create/update/patch on roles/rolebindings in its own namespace — a privilege-escalation primitive on a
# webhook-triggered SA. This file is STATIC and applied ONCE, out-of-band, by a human:
#
# kubectl apply -f infra/k8s/rbac/tailscale-sidecars.yaml
#
# It is NOT applied by the CI pipeline. The Deployments still reference these ServiceAccounts via
# `serviceAccountName: tailscale-dashboard` / `tailscale-dagster` — that reference resolves because the
# SAs are managed here, separately, not because the app manifests create them anymore.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tailscale-dashboard
namespace: foxhunt
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tailscale-dashboard
namespace: foxhunt
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tailscale-dashboard
namespace: foxhunt
subjects:
- kind: ServiceAccount
name: tailscale-dashboard
namespace: foxhunt
roleRef:
kind: Role
name: tailscale-dashboard
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: tailscale-dagster, namespace: foxhunt }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: tailscale-dagster, namespace: foxhunt }
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch", "list", "watch"]
# K8sRunLauncher: the daemon launches each Dagster run as its OWN K8s Job (own memory) so the heavy
# bybit_book_precompute asset (6-8Gi) never OOMs the 4Gi daemon. Needs to manage Jobs + read their pods/logs.
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "get", "list", "watch", "delete"]
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: tailscale-dagster, namespace: foxhunt }
subjects:
- { kind: ServiceAccount, name: tailscale-dagster, namespace: foxhunt }
roleRef: { kind: Role, name: tailscale-dagster, apiGroup: rbac.authorization.k8s.io }

View File

@@ -112,33 +112,3 @@ metadata:
spec:
selector: { app.kubernetes.io/name: fxhnt-dashboard }
ports: [{ port: 80, targetPort: 8080, name: http }]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tailscale-dashboard
namespace: foxhunt
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tailscale-dashboard
namespace: foxhunt
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "update", "patch", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tailscale-dashboard
namespace: foxhunt
subjects:
- kind: ServiceAccount
name: tailscale-dashboard
namespace: foxhunt
roleRef:
kind: Role
name: tailscale-dashboard
apiGroup: rbac.authorization.k8s.io

View File

@@ -76,8 +76,11 @@ spec:
# full re-login each time). 08:00 UTC is clear of the 14:35 rebalancer + 23:30 nightly windows.
- name: TIME_ZONE
value: "Etc/UTC"
# MUST be zero-padded hh:mm (IBC's parser rejects "8:00 AM" with "Auto restart time setting must be
# hh:mm AM or hh:mm PM" — the rejection left the daily auto-restart UNCONFIGURED, so the gateway
# churned ~16 restarts/day instead of one clean daily cycle. Verified in the pod log 2026-07-20.)
- name: AUTO_RESTART_TIME
value: "8:00 AM"
value: "08:00 AM"
# Persist jts.ini + IBC config across pod restarts (a fresh empty volume just does a clean first-run
# login) so settings survive a deploy/reschedule instead of being rebuilt each time.
- name: TWS_SETTINGS_PATH
@@ -105,7 +108,9 @@ spec:
port: 4002
initialDelaySeconds: 120
periodSeconds: 30
failureThreshold: 5
# 6×30s = 180s tolerance: the daily auto-restart + IBKR reauth window drops port 4002 for up to a
# couple of minutes; too tight a threshold would k8s-kill the pod mid-reauth and compound the churn.
failureThreshold: 6
volumeMounts:
- name: tws-settings
mountPath: /home/ibgateway/tws_settings

View File

@@ -1,24 +0,0 @@
#!/usr/bin/env bash
# Build + deploy the fxhnt cockpit via Argo Workflows (in-cluster Kaniko build, no laptop push).
#
# Prereqs (one-time): the `fxhnt-cockpit` WorkflowTemplate is registered
# (kubectl -n foxhunt apply -f infra/argo/cockpit-build-deploy.yaml)
# and this repo is pushed to the in-cluster GitLab as root/fxhnt.git.
#
# Usage:
# ./scripts/argo-deploy-cockpit.sh # build+deploy HEAD of root/fxhnt master
# ./scripts/argo-deploy-cockpit.sh <commit-sha> # a specific commit
set -euo pipefail
SHA="${1:-HEAD}"
NS="foxhunt"
# Resolve HEAD to the real master sha so the in-cluster clone checks out exactly what was pushed.
if [ "$SHA" = "HEAD" ]; then
SHA="$(git rev-parse HEAD)"
fi
echo "Submitting fxhnt-cockpit build+deploy for ${SHA} ..."
argo submit -n "$NS" --from=wftmpl/fxhnt-cockpit \
-p commit-sha="$SHA" \
--watch

View File

@@ -1,134 +0,0 @@
"""Deribit DVOL adapter — the DVOL annualised IMPLIED-vol index (the IV leg of the VRP edge).
Why this signal: Deribit's DVOL is a BTC/ETH 30-day implied-volatility index (annualised, in vol points, e.g.
3383 for BTC) computed off the live options order book — the market's forward-looking expected vol. Paired
with the REALIZED vol of the underlying it is the volatility risk premium (VRP): IV systematically exceeds
RV, so SELLING variance earns the spread, but with fat left tails when a vol spike makes RV >> IV. The
backtest SIGNAL uses DVOL (IV) + close-to-close RV; live execution would be short Bybit option straddles
(a documented cross-venue caveat — the DVOL index is Deribit's, the harvest is Bybit's options).
Endpoint (HTTP is INJECTED so tests run with no network):
GET /api/v2/public/get_volatility_index_data?currency=BTC&start_timestamp=<ms>&end_timestamp=<ms>
&resolution=86400
-> result.data[] of rows [ts_ms, open, high, low, close], oldest-first. close (idx 4) = the DVOL value.
resolution=86400 (seconds) = daily, so there is one row per UTC day. The API caps ~1000 rows/request, so
we paginate BACKWARD via end_timestamp for the full ~5.3y of history (BTC/ETH DVOL begins 2021-03-24).
`dvol_history(currency, *, start_ms=None) -> {epoch_day: dvol_close}` paginates BACKWARD via end_timestamp
(mirrors BybitAccountRatio.long_ratio_history's shape) down to a floor (default ≈ 2021-03, just below the
DVOL inception 2021-03-24), stopping on the first empty page or once the oldest row falls below the floor.
epoch_day =
timestamp_ms // 86_400_000.
CLOCK SEAM (mirrors DeribitFunding): the wall clock is INJECTED (`clock: Callable[[], float] = time.time`).
The first page anchors its end_timestamp at `now`; a bare `time.time()` in the loop would make the backward
walk depend on the calendar date the test runs on — the determinism bug this seam exists to prevent. Tests
pass a fixed clock so the walk is reproducible.
"""
from __future__ import annotations
import datetime as _dt
import json
import time
import urllib.request
from collections.abc import Callable
from typing import Any
_API = "https://www.deribit.com/api/v2/public/get_volatility_index_data"
_PAGE_LIMIT = 1000 # Deribit caps ~1000 rows/request on the volatility-index endpoint
_DAY_MS = 86_400_000
_CLOSE_IX = 4 # row layout: [ts_ms, open, high, low, close]
# Backstop floor for the backward pagination: ≈ 2021-03-01 UTC in ms. The Deribit DVOL index for BTC and ETH
# actually begins 2021-03-24 (verified against the live get_volatility_index_data endpoint); 2021-03-01 is the
# natural lower bound just below inception at which the backward walk terminates. (The previous 2023-09 floor
# silently discarded ~890 days — nearly half — of valid DVOL history, halving the VRP study's non-overlapping
# hold count; this floor recovers the full ~5.3y back to inception.)
DERIBIT_DVOL_FLOOR_MS = int(_dt.datetime(2021, 3, 1, tzinfo=_dt.UTC).timestamp() * 1000)
def _http_get_json(url: str, tries: int = 4) -> Any:
"""Default JSON fetcher (urllib, retry+backoff) — mirrors the other free-data adapters' `_get`."""
for a in range(tries):
try:
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=40) as r:
return json.loads(r.read())
except Exception: # noqa: BLE001 — transient HTTP; retry then re-raise
if a == tries - 1:
raise
time.sleep(2 * (a + 1))
raise RuntimeError("unreachable")
def _with_backoff(fetch: Callable[[str], Any], *, sleep: Callable[[float], None],
base: float, tries: int) -> Callable[[str], Any]:
"""Wrap `fetch(url)->dict` with exponential backoff on ANY transient exception (the Deribit endpoint has
no Bybit-style retCode; a 429/5xx surfaces as a raised exception). Retries `base·2**attempt` up to
`tries`; the final attempt re-raises. `sleep` is injectable (tests pass a no-op recorder)."""
def wrapped(url: str) -> Any:
for attempt in range(tries):
try:
return fetch(url)
except Exception: # noqa: BLE001 — transient HTTP; backoff then re-raise on exhaustion
if attempt == tries - 1:
raise
sleep(base * (2 ** attempt))
raise RuntimeError("unreachable") # pragma: no cover — loop always returns or raises
return wrapped
class DeribitDVOL:
def __init__(self, *, fetch: Callable[[str], Any] = _http_get_json,
sleep: Callable[[float], None] = time.sleep,
clock: Callable[[], float] = time.time,
backoff_base: float = 0.5, backoff_tries: int = 5) -> None:
# fetch(url) -> parsed JSON dict (injected for tests). Wrapped with exponential backoff so a transient
# 429/5xx from Deribit is retried instead of failing — `sleep` is the (injectable) backoff sleeper
# too, so tests passing a no-op sleep never actually wait.
self._fetch = _with_backoff(fetch, sleep=sleep, base=backoff_base, tries=backoff_tries)
self._sleep = sleep # throttle between pages in production; no-op-able in tests
# `clock` is the wall-clock seam (epoch seconds; defaults to time.time). The first page anchors its
# end_timestamp at `now`; injecting a FIXED clock makes the backward walk deterministic so tests do
# not depend on the calendar date they run on (mirrors DeribitFunding — a bare time.time() in the loop
# is the determinism bug this seam exists to prevent).
self._clock = clock
def dvol_history(self, currency: str, *, start_ms: int | None = None) -> dict[int, float]:
"""{epoch_day: dvol_close} for `currency` (BTC/ETH), where dvol_close = the daily DVOL index value
(annualised implied vol, vol points). resolution=86400 → one row/day, no aggregation.
Paginates BACKWARD via end_timestamp: fetch the newest page, then set end_timestamp = oldest row's
ts 1 and loop, stopping on the first empty page or when the oldest row falls below `start_ms`
(default DERIBIT_DVOL_FLOOR_MS ≈ 2021-03). Last-write-wins on a duplicate (currency, day) — there is
at most one row per day, so this only guards page-boundary repeats. The infinite-loop guard breaks if
the cursor stops advancing."""
floor = DERIBIT_DVOL_FLOOR_MS if start_ms is None else int(start_ms)
daily: dict[int, float] = {}
seen: set[int] = set() # guard against duplicate ts across page boundaries
end_ms: int | None = None # first page: no cursor (newest)
while True:
now_ms = int(self._clock() * 1000)
url = (f"{_API}?currency={currency}&start_timestamp={floor}"
f"&end_timestamp={end_ms if end_ms is not None else now_ms}&resolution=86400")
data = self._fetch(url)
rows = data.get("result", {}).get("data", [])
if not rows:
break
oldest_ms = end_ms if end_ms is not None else None
for row in rows:
ts_ms = int(row[0])
if oldest_ms is None or ts_ms < oldest_ms:
oldest_ms = ts_ms
if ts_ms < floor or ts_ms in seen:
continue
seen.add(ts_ms)
daily[ts_ms // _DAY_MS] = float(row[_CLOSE_IX])
if oldest_ms is None or oldest_ms <= floor:
break # walked back to the floor → done
next_end = oldest_ms - 1
if end_ms is not None and next_end >= end_ms:
break # cursor not advancing → avoid an infinite loop
end_ms = next_end
self._sleep(0.1) # gentle throttle; tests inject a no-op sleep
return daily

View File

@@ -2,7 +2,7 @@
Key assets: `futures_bars` (Databento front-month arrays), `bybit_warehouse_refresh` (nightly Bybit
`bybit_features` update), the per-edge forward-nav tracks (xsfunding/unlock/positioning/bybit_4edge/
sixtyforty/multistrat/vrp), and `cockpit_forward` (recompute the forward state from the DB anchors and
sixtyforty/multistrat), and `cockpit_forward` (recompute the forward state from the DB anchors and
upsert into the operational cockpit DB). The DB is the single SSOT — no JSON state files.
NOTE: this module intentionally omits `from __future__ import annotations` because
@@ -290,7 +290,7 @@ def _persist_track_backtest_ref(context: AssetExecutionContext, name: str, build
"""Persist the reconciliation-gate backtest reference for an individual forward track, routed through the
ONE `backtest_ref` dispatcher (fxhnt.application.backtest_refs) so this inline nightly path, the
`backtest-refs --all` CLI (report-kind), and the B3 runner all share the same dispatch. For
`recompute-replay` strategies (multistrat/vrp/crypto_tstrend/stablecoin/unlock) this is bit-identical to
`recompute-replay` strategies (multistrat/crypto_tstrend/stablecoin/unlock) this is bit-identical to
the prior inline behaviour: `build_strategy().advance(None, {})` = the full inception series at the EXACT
basis the forward books (e.g. crypto_tstrend's forward is 15%-book-vol-targeted, not the raw TrendRunner
sleeve). For a `sleeve` strategy it would resolve a feature store instead, since `sleeve_returns_from_store`
@@ -302,9 +302,9 @@ def _persist_track_backtest_ref(context: AssetExecutionContext, name: str, build
BEST-EFFORT: a ref-persist failure logs and returns — it must NEVER fail the forward step (the ref is a
gate input, not the track itself). Without it the gate WITHHOLDS PASS (no ref to reconcile).
For `recompute-replay` strategies (multistrat/vrp/crypto_tstrend/stablecoin/unlock) this ALSO persists
For `recompute-replay` strategies (multistrat/crypto_tstrend/stablecoin/unlock) this ALSO persists
the full inception series into `sim_curve_ret` (Task 1/D1) — the precomputed curve the cockpit's
`/paper/sim` reads for its IBKR books (multistrat/vrp), mirroring the Bybit book's `bybit_sleeve_ret`
`/paper/sim` reads for its IBKR books (multistrat), mirroring the Bybit book's `bybit_sleeve_ret`
precompute. One replay call feeds BOTH writes (the reconciliation ref and the sim curve), so the sim's
curve is always the SAME basis the gate reconciles the forward track against."""
import datetime as _dt
@@ -489,235 +489,6 @@ def multistrat_levered_nav(context: AssetExecutionContext) -> dict: # type: ign
)
def _iso_date(s: str): # type: ignore[no-untyped-def]
import datetime as _dt
return _dt.date.fromisoformat(s)
def _forward_from_marks(chain, marks: dict, as_of: str) -> float: # type: ignore[no-untyped-def]
"""Put-call-parity forward from a PRE-FETCHED single-day marks dict `{osi_symbol: close}`. Pure (no OPRA)
— shared by the per-day fetch path (`_xsp_forward_estimate`) and the batched backfill (in-memory marks),
so both derive the forward identically. Nearest-to-30-DTE expiry, ATM common strike by min distance to
the mid; ties over a SORTED sequence (deterministic across PYTHONHASHSEED)."""
from fxhnt.application.vrp_pricing import parity_forward
near = [c for c in chain if 20 <= (c.expiry - _iso_date(as_of)).days <= 45]
if not near:
raise RuntimeError(f"no XSP chain for {as_of} — cannot estimate forward")
exp = min(sorted({c.expiry for c in near}), key=lambda e: abs((e - _iso_date(as_of)).days - 30))
puts = {c.strike: c for c in near if c.expiry == exp and c.right == "P"}
calls = {c.strike: c for c in near if c.expiry == exp and c.right == "C"}
common = sorted(set(puts) & set(calls))
if not common:
raise RuntimeError(f"no matched call/put strike for {as_of} forward")
mid = common[len(common) // 2]
for k in sorted(common, key=lambda s: abs(s - mid)):
p, c = marks.get(puts[k].osi_symbol), marks.get(calls[k].osi_symbol)
if p is not None and c is not None:
return parity_forward(call_atm=c, put_atm=p, k_atm=k)
raise RuntimeError(f"no ATM call/put pair with marks for {as_of}")
def _band_contracts(chain, as_of: str, forward: float): # type: ignore[no-untyped-def]
"""The daily freeze band: WIDE put band (DTE[1,45], moneyness[0.70,1.20] — wider than the ~30-DTE ENTRY
window so a spread's legs stay frozen through their whole held life + market moves, no zombie positions)
plus near-ATM CALLS (for the parity forward off frozen data). Pure filter."""
d = _iso_date(as_of)
puts = [c for c in chain if c.right == "P" and 1 <= (c.expiry - d).days <= 45
and 0.70 <= c.strike / forward <= 1.20]
calls = [c for c in chain if c.right == "C" and 20 <= (c.expiry - d).days <= 45
and 0.96 <= c.strike / forward <= 1.04]
return puts + calls
def _band_rows_from_marks(chain, marks: dict, as_of: str, forward: float) -> list: # type: ignore[type-arg,no-untyped-def]
"""Band rows for `as_of` from a PRE-FETCHED single-day marks dict `{osi_symbol: close}`. Pure — shared by
the per-day path and the batched backfill."""
return [dict(osi_symbol=c.osi_symbol, date=as_of, expiry=c.expiry.isoformat(),
strike=c.strike, right=c.right, close=marks[c.osi_symbol])
for c in _band_contracts(chain, as_of, forward) if c.osi_symbol in marks]
def _day_marks(dbn, osis, as_of: str) -> dict: # type: ignore[type-arg,no-untyped-def]
"""Fetch `osis` for a single day and flatten `{osi:{date:close}}` -> `{osi: close}` (dropping missing)."""
return {osi: m[as_of] for osi, m in dbn.fetch_option_bars(osis, as_of, as_of).items() if as_of in m}
def _xsp_forward_estimate(dbn, as_of: str, chain=None) -> float: # type: ignore[no-untyped-def]
"""Per-day parity forward: fetch the near-ATM expiry's marks, then `_forward_from_marks`. `chain` may be
passed in to avoid a redundant OPRA definition fetch (see `_freeze_day`)."""
full = chain if chain is not None else dbn.fetch_option_chain("XSP.OPT", as_of)
near = [c for c in full if 20 <= (c.expiry - _iso_date(as_of)).days <= 45]
if not near:
raise RuntimeError(f"no XSP chain for {as_of} — cannot estimate forward")
exp = min(sorted({c.expiry for c in near}), key=lambda e: abs((e - _iso_date(as_of)).days - 30))
atm_osis = [c.osi_symbol for c in near if c.expiry == exp]
return _forward_from_marks(full, _day_marks(dbn, atm_osis, as_of), as_of)
def _freeze_xsp_slice(dbn, bars, *, as_of: str, forward: float, at, chain=None) -> None: # type: ignore[no-untyped-def]
"""Per-day freeze: fetch the band's marks, then upsert `_band_rows_from_marks`. `chain` may be passed in
to avoid a redundant OPRA definition fetch (see `_freeze_day`)."""
full = chain if chain is not None else dbn.fetch_option_chain("XSP.OPT", as_of)
band = _band_contracts(full, as_of, forward)
marks = _day_marks(dbn, [c.osi_symbol for c in band], as_of)
bars.upsert_bars(_band_rows_from_marks(full, marks, as_of, forward), at)
def _freeze_day(dbn, bars, *, as_of: str, at) -> float: # type: ignore[no-untyped-def]
"""Fetch the XSP option chain ONCE for `as_of`, derive the parity forward from it, and freeze the band
from the SAME chain. Avoids the double `fetch_option_chain` (definition) call — `_xsp_forward_estimate`
and `_freeze_xsp_slice` each used to fetch it independently, doubling OPRA definition cost on every
nightly `vrp_nav` run and the 2013-> backfill. Returns the forward."""
chain = dbn.fetch_option_chain("XSP.OPT", as_of)
fwd = _xsp_forward_estimate(dbn, as_of, chain=chain)
_freeze_xsp_slice(dbn, bars, as_of=as_of, forward=fwd, at=at, chain=chain)
return fwd
def _prev_trading_day(d): # type: ignore[no-untyped-def]
"""One trading day BEFORE `d`, skipping Sat/Sun (isoweekday 6/7). A market holiday is not a fixed calendar
rule, so it is absorbed by one more step-back in `_freeze_latest_session`'s bound rather than skipped here."""
import datetime as _dt
d -= _dt.timedelta(days=1)
while d.isoweekday() >= 6: # Sat=6, Sun=7 — never a trading session
d -= _dt.timedelta(days=1)
return d
# Message fragments raised by `_xsp_forward_estimate`/`_forward_from_marks` when a PUBLISHED session is too
# degenerate to form the parity forward (no near-DTE chain / no matched call-put strike / no ATM pair with
# marks). Matched narrowly so `_freeze_latest_session` steps back on a broken session but never swallows an
# unrelated RuntimeError (e.g. a DB write failure inside the freeze).
_DEGENERATE_FORWARD_MARKERS = ("cannot estimate forward", "matched call/put strike", "ATM call/put pair")
def _is_degenerate_forward_error(e: Exception) -> bool:
msg = str(e)
return any(marker in msg for marker in _DEGENERATE_FORWARD_MARKERS)
def _freeze_latest_session(dbn, bars, *, at, max_steps: int = 5) -> str: # type: ignore[no-untyped-def]
"""Freeze the LATEST COMPLETE, USABLE OPRA session (robust, timing-independent). The candidate starts at
OPRA's available-END date (`dbn.last_available_option_date()`); today's session is still settling until
after the close, so freezing it fails with the normalized `DatabentoRangeUnavailable` (the
`data_schema_not_fully_available` 422 OR the C1a empty-window clamp). A PUBLISHED-but-DEGENERATE session
instead surfaces as a forward-estimation `RuntimeError` ("no XSP chain … cannot estimate forward" / "no
matched call/put strike …" / "no ATM call/put pair …"). Either signal — unavailable OR degenerate — steps
the candidate back ONE trading day (skip Sat/Sun) and retries, BOUNDED to `max_steps` (a market holiday or
one broken session just costs one more step). The FIRST candidate that freezes is the target; the ISO date
frozen is RETURNED (vrp_nav logs it). Unrelated `RuntimeError`s (not forward-estimation) propagate.
If no usable session is found within the bound, raise `DatabentoRangeUnavailable` — the C1b resilience
wrapper in `vrp_nav` catches it → recompute off the frozen table, and the health axis keeps vrp STALE
(correct: we genuinely could not advance the freeze). This is LOUD by design; NEVER a silent no-op, and a
bad/bar-less date is NEVER returned — only the last COMPLETE, non-degenerate session's marks are frozen."""
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
start = dbn.last_available_option_date()
candidate = start
last_err: Exception | None = None
for _ in range(max_steps + 1): # the start candidate + up to `max_steps` bounded step-backs
iso = candidate.isoformat()
try:
_freeze_day(dbn, bars, as_of=iso, at=at)
return iso
except DatabentoRangeUnavailable as e: # still-settling / not-yet-available session → step back
last_err = e
candidate = _prev_trading_day(candidate)
except RuntimeError as e: # published-but-degenerate session (no derivable forward)
if not _is_degenerate_forward_error(e):
raise # an unrelated RuntimeError — do NOT silently swallow it
last_err = e
candidate = _prev_trading_day(candidate)
raise DatabentoRangeUnavailable(
f"no complete, non-degenerate OPRA session within {max_steps} trading days back "
f"from {start.isoformat()}") from last_err
def _rough_forward(chain, as_of: str): # type: ignore[no-untyped-def]
"""Coarse forward = median strike of the ~30-DTE expiry (NO marks needed) — used ONLY to bound the bulk
monthly ohlcv fetch to a wide strike band. The precise per-day forward is always recomputed from real
marks via `_forward_from_marks`. XSP lists strikes densely+symmetrically around ATM, so the median strike
is a good ATM proxy for bounding."""
d = _iso_date(as_of)
near = [c for c in chain if 20 <= (c.expiry - d).days <= 45]
if not near:
return None
exp = min(sorted({c.expiry for c in near}), key=lambda e: abs((e - d).days - 30))
strikes = sorted({c.strike for c in near if c.expiry == exp})
return strikes[len(strikes) // 2] if strikes else None
def _backfill_month(dbn, bars_repo, *, month_start: str, month_end: str, at) -> tuple: # type: ignore[type-arg,no-untyped-def]
"""BATCHED one-month freeze: ONE range definition fetch + ONE range ohlcv fetch (bulk), then process each
trading day LOCALLY (parity forward + band + upsert). ~2 OPRA calls/month vs ~6-8/day — the per-day
pattern was ~26k API round-trips over 2013->, ~45h; batched is ~2 calls/month * ~160 months, ~1h.
Returns (n_ok, n_fail) days. A day with no arb-free ATM pair (sparse early-XSP data) is counted failed,
never aborts the month."""
import datetime as _dt
chain = dbn.fetch_option_chain_range("XSP.OPT", month_start, month_end)
if not chain:
return (0, 0)
rough = _rough_forward(chain, month_start) or _rough_forward(chain, month_end)
if rough is None:
return (0, 0)
ms, me = _iso_date(month_start), _iso_date(month_end)
# bound the bulk fetch: any expiry a day this month could reference at DTE<=45, and a strike band WIDE
# enough (+-50% of the month's median) to cover intra-month moves + the precise [0.70,1.20] daily band
# even in a crash month (COVID Mar-2020 XSP fell ~34% intramonth).
band = [c for c in chain
if (ms + _dt.timedelta(days=1)) <= c.expiry <= (me + _dt.timedelta(days=45))
and 0.50 <= c.strike / rough <= 1.50]
if not band:
return (0, 0)
bars = dbn.fetch_option_bars([c.osi_symbol for c in band], month_start, month_end) # bulk range, chunked
days = sorted({dd for m in bars.values() for dd in m})
n_ok = n_fail = 0
for iso in days:
day_marks = {osi: m[iso] for osi, m in bars.items() if iso in m}
try:
fwd = _forward_from_marks(band, day_marks, iso)
bars_repo.upsert_bars(_band_rows_from_marks(band, day_marks, iso, fwd), at)
n_ok += 1
except Exception: # a sparse/degenerate day (no ATM pair with marks) — skip, don't abort the month
n_fail += 1
return (n_ok, n_fail)
@asset
def vrp_nav(context: AssetExecutionContext) -> dict: # type: ignore[type-arg]
"""Paper-forward defined-risk XSP put-credit-spread VRP on REAL OPRA data. Freezes today's DTE/moneyness
put slice (plus near-ATM calls, for the parity forward) PIT into xsp_option_bars, then recomputes off the
FROZEN table (spec §A) so the record is stable against any OPRA revision. persist_backtest_ref=True
publishes the basis-matched real ref for the gate."""
import datetime as _dt
from fxhnt.adapters.data.databento import DatabentoDataProvider
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
from fxhnt.application.vrp_book import VrpStrategy
from fxhnt.config import get_settings
s = get_settings()
bars = XspOptionBarsRepo(s.operational_dsn)
bars.migrate()
dbn = DatabentoDataProvider(max_cost_usd=s.databento.max_cost_usd)
at = _dt.datetime.now(_dt.UTC).replace(tzinfo=None)
try:
# Freeze the LATEST COMPLETE OPRA session via a bounded step-back — robust and timing-independent, so
# the freeze advances deterministically regardless of when the job runs (never freezes today's still-
# settling partial session's marks). Returns the ISO date actually frozen.
session = _freeze_latest_session(dbn, bars, at=at)
context.log.info(f"vrp_nav: froze OPRA session {session}")
except Exception as e: # noqa: BLE001 — freeze is best-effort; the tracker recomputes off the FROZEN table
# A bad-data day, no complete session within the step-back bound (DatabentoRangeUnavailable), or a
# cost-cap trip must NOT kill vrp_nav: it would zero out the forward anchor AND skip the downstream
# gate. Log + continue — the health axis keeps vrp STALE (loud), which is the correct signal.
context.log.warning(f"vrp_nav: freeze step failed (recomputing off the frozen table): {e}")
return _run_paper_tracker(
context, "vrp_nav", lambda: VrpStrategy(bars), persist_backtest_ref=True,
)
# RETIRED 2026-06-20 (cockpit cleanup): the live tracks funding_nav (absolute-hurdle, superseded by
# cross-sectional xsfunding), gd_nav / multistrat_nav (futures/TradFi premia — falsified this session,
# scale-gated, no deployable edge), and poc_nav (old crypto residual-mom+VRP PoC, superseded by the two
@@ -1128,7 +899,7 @@ def bybit_paper_book(context: AssetExecutionContext) -> dict: # type: ignore[ty
@asset(deps=[sixtyforty_nav, xsfunding_nav, unlock_nav, multistrat_nav, bybit_4edge_nav,
bybit_4edge_levered_nav, positioning_nav]) # vrp_nav ARCHIVED 2026-07-15 — de-wired (fn kept below)
bybit_4edge_levered_nav, positioning_nav])
def cockpit_forward(context: AssetExecutionContext) -> None:
"""Evaluate + persist the forward gate from the DB record the engine already wrote (all_summaries +
nav_history — see `forward_ingest.ingest_forward_state`), then auto-promote any `promote_on_pass`
@@ -1142,7 +913,7 @@ def cockpit_forward(context: AssetExecutionContext) -> None:
dsn = get_settings().operational_dsn
ingest_forward_state(dsn)
# Runtime health axis (SEPARATE from the gate): surface ANY track that stopped updating LOUDLY — a red
# cockpit badge + an ERROR log the FIRST night, so a stale recompute (the 2-week vrp rot) can never sit as
# cockpit badge + an ERROR log the FIRST night, so a stale recompute (a dead strategy's silent rot) can never sit as
# a quiet WAIT again. evaluate_health logs at ERROR with the unhealthy list; also surface the count here.
unhealthy = [v for v in evaluate_health(dsn) if not v.ok]
if unhealthy:

View File

@@ -7,9 +7,8 @@ retired here — crypto is 100% Bybit now (`bybit_warehouse_refresh` + the Bybit
`bybit_4edge` deploy book below) — was RETIRED (Phase 0b Task 7d); sixtyforty_nav/multistrat_nav
are their OWN independent live assets below, unaffected by that retirement.)
(vrp_nav ARCHIVED/FALSIFIED 2026-07-15the VRP sleeve is a shelved dead-end; its asset function is KEPT
in assets.py but DE-WIRED from the nightly here so it no longer materializes. See
project_fxhnt_diversifier_hunt_2026_07_15.)
(VRP removed entirely 2026-07-20see project_fxhnt_diversifier_hunt_2026_07_15 for the
falsification that led to the removal.)
NOTE: no `from __future__ import annotations` — Dagster 1.12 runtime type-hint inspection."""
from dagster import DefaultScheduleStatus, Definitions, ScheduleDefinition, define_asset_job

View File

@@ -68,15 +68,6 @@ def track_builders(settings: Settings, data_dir: str) -> TrackBuilders:
# builders here are gone with them. crypto_tstrend the BYBIT BOOK SLEEVE is unaffected — that lives
# under `bybit_4edge`'s own builder below, not here.)
# ---- vrp_nav: defined-risk XSP put-spread VRP on the frozen OPRA PIT table (recompute) ----------------
def _build_vrp() -> Any:
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
from fxhnt.application.vrp_book import VrpStrategy
return VrpStrategy(XspOptionBarsRepo(settings.operational_dsn))
builders["vrp"] = ("vrp_state", _build_vrp)
# ---- unlock_nav: single-sleeve unlock edge on the liquid universe (Phase 0b Task 2: own Bybit track, ----
# ---- the SAME single-sleeve construction as xsfunding/positioning, mirrored here; carries the unlock ----
# ---- calendar the sleeve needs, unlike xsfunding/positioning) --------------------------------------------

View File

@@ -43,7 +43,8 @@ class StrategyHealthRow(CockpitBase):
"""Runtime HEALTH verdict per track — a SEPARATE axis from the gate (a track can be legitimately
WAIT/building AND healthy; STALE/NO_REF/BROKEN is orthogonal). Written each nightly run by
`forward_health.evaluate_health`; the cockpit reads it to render a LOUD red badge so a live/gated track
that stops updating surfaces on day 1 (the anti-silent-rot backbone that would have caught vrp). One row
that stops updating surfaces on day 1 (the anti-silent-rot backbone that would have caught a dead
strategy's silent staleness). One row
per strategy, replaced each run (a track that recovers clears its stale row). Brand-new table →
`create_all` (ForwardNavRepo.migrate) creates it; no ALTER, Postgres-safe."""
__tablename__ = "strategy_health"
@@ -225,7 +226,7 @@ class BybitSleeveRetRow(CockpitBase):
class SimCurveRetRow(CockpitBase):
"""One recompute-replay strategy's (multistrat/vrp — the IBKR paper books the Backtest page's sim reads)
"""One recompute-replay strategy's (multistrat — the IBKR paper books the Backtest page's sim reads)
daily honest-cost return for one run_date. Mirrors `BybitSleeveRetRow`'s shape/purpose for the Bybit-book
sim: written nightly by `_persist_track_backtest_ref` (the `fxhnt-backtest-refs` Job) from the SAME
inception rows `backtest_ref`'s recompute-replay branch derives the reconciliation-gate ref from, so

View File

@@ -143,9 +143,9 @@ class ForwardNavRepo:
return list(s.scalars(select(StrategyHealthRow).order_by(StrategyHealthRow.strategy_id)))
def xsp_freeze_max_date(self) -> str | None:
"""The latest date present in the vrp freeze table (`xsp_option_bars`), ISO yyyy-mm-dd, or None when
the table is empty. The freeze-staleness axis: a vrp recompute is bounded by this max date, so if the
freeze stops advancing the track is STALE even though its forward 'updates'."""
"""The latest date present in the opra-pit-anchored freeze table (`xsp_option_bars`), ISO yyyy-mm-dd,
or None when the table is empty. The freeze-staleness axis: a frozen-input recompute is bounded by
this max date, so if the freeze stops advancing the track is STALE even though its forward 'updates'."""
from sqlalchemy import func
with Session(self._engine) as s:
d = s.scalar(select(func.max(XspOptionBarRow.date)))
@@ -196,7 +196,7 @@ class ForwardNavRepo:
def registry(self) -> list[StrategyRegistryRow]:
"""The cockpit-visible registry rows. SINGLE SOURCE for the hide: an ARCHIVED sleeve (its
`STRATEGY_REGISTRY` entry carries `archived: True` — e.g. the shelved/FALSIFIED vrp/vrp_exec) is
`STRATEGY_REGISTRY` entry carries `archived: True` — e.g. a shelved/FALSIFIED strategy) is
excluded here, so EVERY caller (fleet overview, per-strategy gate specs, `detail()`'s reg lookup)
drops it automatically. The DB row is still seeded (code + data kept) — it is only hidden from the UI."""
with Session(self._engine) as s:

View File

@@ -515,8 +515,8 @@ class PaperRepo:
# ---- sim_curve_ret: the PRECOMPUTED recompute-replay (IBKR) book curve /paper/sim reads ------------
def upsert_sim_curve_ret(self, strategy_id: str, run_date: str, ret: float, *, at: dt.datetime) -> None:
"""Idempotent per (strategy_id, run_date): write/overwrite that recompute-replay book's (multistrat/
vrp) per-day honest-cost return. `_persist_track_backtest_ref` writes the full inception series here
"""Idempotent per (strategy_id, run_date): write/overwrite that recompute-replay book's (multistrat)
per-day honest-cost return. `_persist_track_backtest_ref` writes the full inception series here
from the SAME rows it derives the reconciliation-gate ref from, so /paper/sim NEVER re-runs an IBKR
replay on a web request — mirrors `upsert_bybit_sleeve_ret`'s pattern for the Bybit-book sim."""
rd = dt.date.fromisoformat(run_date)

View File

@@ -1,119 +0,0 @@
"""PIT store for frozen XSP option daily bars — the deterministic substrate the VRP recompute reads. OPRA is
hit only when FILLING this table (asset/backfill); VrpStrategy never calls OPRA. Same freeze discipline as the
equity Yahoo-PIT snapshot."""
from __future__ import annotations
import datetime as dt
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from fxhnt.adapters.persistence.cockpit_models import CockpitBase, XspOptionBarRow
from fxhnt.domain.models import OptionContract
class XspOptionBarsRepo:
def __init__(self, dsn: str) -> None:
kw = {} if dsn.startswith("postgresql") else {"connect_args": {"check_same_thread": False}}
self._engine = create_engine(dsn, **kw)
self._pre: dict | None = None # in-memory replay cache (None = not preloaded; per-day DB path)
def migrate(self) -> None:
CockpitBase.metadata.create_all(self._engine)
def preload(self, after: str | None = None) -> None:
"""Bulk-load every frozen bar for days strictly after `after` (all days if None) into an in-memory
index, in ONE query. A full VrpStrategy replay then serves chain_asof/bars_for/trading_days from
memory instead of a fresh Session + round-trip per day — the N+1 that made a 1600-day replay issue
thousands of queries. The live single-day exec path (plan_and_record_vrp) never calls this, so it
keeps its cheap per-day DB queries unchanged. The cache is a faithful mirror of the covered range:
every cached method falls back to the DB for any date OUTSIDE that range, so it can never
under-return. Idempotent (re-call to re-scope)."""
lo = dt.date.fromisoformat(after) if after else None
marks: dict[str, dict[str, float]] = {}
rows_by_date: dict[str, list[tuple[str, dt.date, float, str]]] = {}
with Session(self._engine) as s:
q = select(XspOptionBarRow.osi_symbol, XspOptionBarRow.date, XspOptionBarRow.expiry,
XspOptionBarRow.strike, XspOptionBarRow.right, XspOptionBarRow.close)
if lo is not None:
q = q.where(XspOptionBarRow.date > lo)
for osi, date, expiry, strike, right, close in s.execute(q):
di = date.isoformat()
marks.setdefault(di, {})[osi] = close
rows_by_date.setdefault(di, []).append((osi, expiry, strike, right))
self._pre = {"after": after, "marks": marks, "rows": rows_by_date, "days": sorted(rows_by_date)}
def _pre_covers(self, date_iso: str) -> bool:
"""True iff `date_iso` is inside the preloaded range (strictly after the preload bound)."""
pre = self._pre
return pre is not None and (pre["after"] is None or date_iso > pre["after"])
def upsert_bars(self, rows: list[dict], at: dt.datetime) -> None:
with Session(self._engine) as s:
for r in rows:
obj = XspOptionBarRow(
osi_symbol=r["osi_symbol"], date=dt.date.fromisoformat(r["date"]),
expiry=dt.date.fromisoformat(r["expiry"]), strike=float(r["strike"]),
right=r["right"], close=float(r["close"]), src_updated_at=at)
s.merge(obj) # merge = insert-or-overwrite on the (osi_symbol, date) PK
s.commit()
def bars_for(self, osi_symbols: list[str], start: str, end: str) -> dict[str, dict[str, float]]:
if not osi_symbols:
return {}
# Preloaded fast path — serve from memory when the whole [start, end] range is inside the cache
# (start > preload bound => the entire range is, since start <= end). Iso date strings compare
# lexically like dt.date, so the range test matches the DB WHERE below.
if self._pre is not None and self._pre_covers(start):
want = set(osi_symbols)
out: dict[str, dict[str, float]] = {}
if start == end: # the replay's single-day case (hot path)
m = self._pre["marks"].get(start, {})
return {osi: {start: m[osi]} for osi in want if osi in m}
for di, m in self._pre["marks"].items():
if start <= di <= end:
for osi in want:
if osi in m:
out.setdefault(osi, {})[di] = m[osi]
return out
lo, hi = dt.date.fromisoformat(start), dt.date.fromisoformat(end)
out: dict[str, dict[str, float]] = {}
with Session(self._engine) as s:
q = select(XspOptionBarRow).where(
XspOptionBarRow.osi_symbol.in_(osi_symbols),
XspOptionBarRow.date >= lo, XspOptionBarRow.date <= hi)
for row in s.scalars(q):
out.setdefault(row.osi_symbol, {})[row.date.isoformat()] = row.close
return out
def trading_days(self, after: str | None) -> list[str]:
"""Ascending distinct dates present in the table, strictly after `after` (or all if None)."""
# Preloaded fast path — the cache holds every day > its bound, so it can answer any query whose
# `after` is at or beyond that bound (a full preload, after=None, answers everything).
pre = self._pre
if pre is not None and (pre["after"] is None or (after is not None and after >= pre["after"])):
return [d for d in pre["days"] if after is None or d > after]
from sqlalchemy import distinct
with Session(self._engine) as s:
q = select(distinct(XspOptionBarRow.date)).order_by(XspOptionBarRow.date)
days = [d.isoformat() for d in s.scalars(q)]
return [d for d in days if after is None or d > after]
def chain_asof(self, date: str, dte_lo: int, dte_hi: int, right: str = "P") -> list[OptionContract]:
d = dt.date.fromisoformat(date)
exp_lo, exp_hi = d + dt.timedelta(days=dte_lo), d + dt.timedelta(days=dte_hi)
# Preloaded fast path — same filter (date == d, right, expiry in [exp_lo, exp_hi]) + strike order.
if self._pre is not None and self._pre_covers(date):
hits = [(osi, exp, strike, r) for (osi, exp, strike, r) in self._pre["rows"].get(date, [])
if r == right and exp_lo <= exp <= exp_hi]
hits.sort(key=lambda t: t[2]) # deterministic strike order (matches ORDER BY)
return [OptionContract(osi_symbol=osi, expiry=exp, strike=strike, right=r)
for (osi, exp, strike, r) in hits]
with Session(self._engine) as s:
q = select(XspOptionBarRow).where(
XspOptionBarRow.date == d, XspOptionBarRow.right == right,
XspOptionBarRow.expiry >= exp_lo, XspOptionBarRow.expiry <= exp_hi
).order_by(XspOptionBarRow.strike) # deterministic strike order (no ORDER BY = arbitrary
# row order on Postgres — the ATM pairing below relies on stable iteration)
return [OptionContract(osi_symbol=r.osi_symbol, expiry=r.expiry, strike=r.strike, right=r.right)
for r in s.scalars(q)]

View File

@@ -446,12 +446,13 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
# The books the sim can configure. bybit_4edge = the precomputed 4-edge Bybit backtest (naive eq-wt —
# NOT the overlay, since the A/B proved naive is best OOS), sized from bybit_sleeve_ret. bybit_4edge_levered
# = the observe-only levered shadow. multistrat/vrp (Task 1/D1) = the IBKR paper strategies' precomputed
# = the observe-only levered shadow. multistrat (Task 1/D1) = the IBKR paper strategies' precomputed
# recompute-replay curve (sim_curve_ret) — generalizes the sim beyond the Bybit-only venue. Order = render
# order of the book-switch pills: the deploy book leads (and is the default).
# Exclude ARCHIVED books (their STRATEGY_REGISTRY entry carries `archived: True` — e.g. the
# shelved/FALSIFIED vrp): they never appear as a Backtest book pill, and a `?book=vrp` request falls
# back to `_DEFAULT_BOOK` (so the folded "Live forward" section can't surface an archived sleeve either).
# Exclude ARCHIVED books (their STRATEGY_REGISTRY entry carries `archived: True` — e.g. a
# shelved/FALSIFIED strategy): they never appear as a Backtest book pill, and a request for an
# archived book falls back to `_DEFAULT_BOOK` (so the folded "Live forward" section can't surface an
# archived sleeve either).
_SIM_BOOKS = tuple(b for b in SIM_BOOKS if not STRATEGY_REGISTRY.get(b, {}).get("archived"))
_DEFAULT_BOOK = "bybit_4edge"
# Plain-language pill labels for the sim's book switch — NEVER the raw registry id. Sourced from the
@@ -470,7 +471,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
_sim_returns_cache: dict[str, tuple[float, dict[str, dict[int, float]]]] = {}
def _ibkr_sim_curve(book: str) -> dict[str, dict[int, float]]:
"""The precomputed recompute-replay (IBKR) book curve for `book` (multistrat/vrp), as
"""The precomputed recompute-replay (IBKR) book curve for `book` (multistrat), as
`{book: {epoch_day: ret}}` — read from `sim_curve_ret` (written nightly by
`_persist_track_backtest_ref`/the `fxhnt-backtest-refs` Job from the SAME inception rows the
reconciliation-gate ref is derived from). NEVER recomputes the replay on this request path."""
@@ -479,7 +480,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
def _sim_returns(book: str = _DEFAULT_BOOK) -> dict[str, dict[int, float]]:
"""Cached per-sleeve returns for `book`, dispatched by the registry's `backtest.kind` (Task 1/D1):
`report`-kind Bybit books (bybit_4edge / bybit_4edge_levered) read the precomputed bybit_sleeve_ret
table; `recompute-replay`-kind IBKR books (multistrat / vrp) read the precomputed sim_curve_ret
table; `recompute-replay`-kind IBKR books (multistrat) read the precomputed sim_curve_ret
table via `_ibkr_sim_curve`. NEVER recomputes any edge on the request path — both sources are
nightly-precomputed. Memoized in-process with a 300s TTL, keyed by book (returns change only nightly)."""
now = time.monotonic()
@@ -548,7 +549,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
def _measured_pending_context(book: str) -> dict[str, Any]:
"""Graceful degrade when the precomputed curve is ABSENT: a short 'precomputing — check back after the
nightly job' note (NOT a flat slider, never a 500). The live forward track still renders for bybit_4edge
and for the IBKR recompute-replay books (multistrat/vrp) — each has its own real forward track."""
and for the IBKR recompute-replay books (multistrat) — each has its own real forward track."""
is_ibkr = registry_backtest_kind(book) == "recompute-replay"
return {
"measured_pending": True,
@@ -564,7 +565,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
def _ibkr_measured_context(book: str, *, capital: float, start: str | None = None,
end: str | None = None) -> dict[str, Any] | None:
"""The IBKR recompute-replay books' (multistrat/vrp) Backtest view: compound the PRECOMPUTED per-day
"""The IBKR recompute-replay books' (multistrat) Backtest view: compound the PRECOMPUTED per-day
honest-cost return series (`_sim_returns(book)`, written nightly from the SAME replay basis the
reconciliation gate reconciles the forward track against) into an equity curve via the pure
`simulate_naive_eqwt` engine — no per-coin cost model here (that is the Bybit-only measured-cost path;
@@ -725,7 +726,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
capital = get_settings().paper_capital
# Pre-fill AND bound the period inputs with the book's full precomputed date range, so the date fields
# are never blank ("dates not set") and the picker can't go off the data. Bybit books read their range
# from the per-coin measured cache; IBKR recompute-replay books (multistrat/vrp, Task 1/D1) read it
# from the per-coin measured cache; IBKR recompute-replay books (multistrat, Task 1/D1) read it
# from their own precomputed sim_curve_ret series (`_sim_returns`).
if registry_backtest_kind(book) == "recompute-replay":
_days = sorted(_sim_returns(book).get(book) or {})
@@ -749,7 +750,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
end: str | None = None, book: str = _DEFAULT_BOOK) -> HTMLResponse:
"""The honest Backtest: book + capital + period -> the precomputed curve. Capital is parsed leniently
(a cleared/comma'd/odd capital must not 422). Bybit books (report-kind) render the per-coin MEASURED-
cost cache scaled by capital + windowed by period; IBKR recompute-replay books (multistrat/vrp, Task
cost cache scaled by capital + windowed by period; IBKR recompute-replay books (multistrat, Task
1/D1) compound their own precomputed honest-cost return series the same way. Absent precompute ->
a graceful 'precomputing' note (never the flat slider, never a 500)."""
book = book if book in _SIM_BOOKS else _DEFAULT_BOOK
@@ -807,7 +808,7 @@ def create_app_from_settings() -> FastAPI:
repo.migrate()
paper_repo = PaperRepo(settings.operational_dsn)
paper_repo.migrate()
# The REAL IBKR paper-account section on the multistrat/vrp detail pages (D2.2/2.3) — a light
# The REAL IBKR paper-account section on the multistrat detail pages (D2.2/2.3) — a light
# schema-ensure migrate (create_all + a guarded additive ALTER), same shape as repo/paper_repo above.
ibkr_account_repo = IbkrAccountRepo(settings.operational_dsn)
ibkr_account_repo.migrate()

View File

@@ -57,8 +57,8 @@
{% macro rollup_plan_pill(rollup) %}{% if rollup and rollup.n_executing > 0 %}<span class="b {{ 'funded' if rollup.agg_divergence >= 0 else 'decaying' }} small"><span class="dot"></span>{{ '%.1f'|format((rollup.agg_divergence|abs) * 100) }}% {{ 'ahead of plan' if rollup.agg_divergence >= 0 else 'behind plan' }}</span>{% endif %}{% endmacro %}
{# ---- health_badge: the LOUD runtime-health axis (SEPARATE from the gate). A track that is STALE / NO_REF /
BROKEN shows a red badge — visually impossible to confuse with a healthy WAIT — so a stale recompute (the
2-week vrp rot) surfaces on day 1. HEALTHY renders nothing. #}
BROKEN shows a red badge — visually impossible to confuse with a healthy WAIT — so a stale recompute
(a strategy silently rotting for weeks) surfaces on day 1. HEALTHY renders nothing. #}
{% macro health_badge(f) %}{% if f.health and f.health != 'HEALTHY' %}<span class="badge-health {{ f.health }}" style="background:#c0392b;color:#fff;font-size:9px;font-weight:600;border-radius:3px;padding:1px 5px;margin-left:4px;letter-spacing:.03em" title="{{ f.health_reason }}">{% if f.health == 'STALE' %}STALE {{ f.health_stale_days }}d{% elif f.health == 'NO_REF' %}NO REF{% else %}{{ f.health }}{% endif %}</span>{% endif %}{% endmacro %}
{# ---- track_row: one compact edge row. `record` is the meaningful number (backfill/backtest ref, else

View File

@@ -75,7 +75,7 @@
{# ---- D3 (Task 4): the REAL IBKR paper-account "real trades" sub-row — reuses the exec-twin's
`.exrow`/`.exec-tag` style, but plain-language content (real return, click-through to holdings,
the shared `plan_pill`, honest cost-to-trade). Replaces the generic exec-twin row below for the
two IBKR real-account books (multistrat/vrp); the crypto book's own exec-twin row is unaffected. #}
IBKR real-account book (multistrat); the crypto book's own exec-twin row is unaffected. #}
{# The sub-row BORROWS the parent table's 6 columns, so each cell is self-labeled inline (like the
exec-twin row below) — the real return in particular must read as a return, not sit naked under the
"allocated" header. data-label carries the true meaning for the mobile card reflow, not the borrowed

View File

@@ -1,59 +0,0 @@
"""A tiny, dependency-light Black-Scholes European option pricer (r=0, no dividends — crypto convention).
Used by the DEFINED-RISK VRP evaluator to price the legs of a short-straddle-plus-protective-wings spread
from the DVOL implied-vol index (we have NO historical option chains, so we synthesize prices via BS from the
ATM IV — the standard vol-backtest method; the modelling caveats are documented in `vrp_defined_risk_eval`).
Conventions
-----------
* European options, r = 0 (crypto has no risk-free carry to speak of; forward ≈ spot), no dividends.
* `sigma` is the ANNUALISED vol (a fraction, e.g. 0.60 for 60%); `T` is the time to expiry in YEARS.
* The standard-normal CDF uses `math.erf` (no external dependency) — exact to float precision. (scipy's
`norm.cdf` agrees to ~1e-15; we deliberately avoid the import so the pricer is pure-stdlib.)
The only formulas:
d1 = (ln(S/K) + 0.5·σ²·T) / (σ·√T) (r=0)
d2 = d1 σ·√T
call = S·N(d1) K·N(d2)
put = K·N(d2) S·N(d1) (= call S + K, put-call parity at r=0)
Degenerate inputs (T≤0 or σ≤0) collapse to intrinsic value max(SK,0) / max(KS,0), which is the correct
zero-time / zero-vol limit and keeps the pricer total (never raises on an expiring/zero-vol leg)."""
from __future__ import annotations
import math
def _norm_cdf(x: float) -> float:
"""Standard-normal CDF via the error function: N(x) = 0.5·(1 + erf(x/√2)). Pure stdlib, no scipy."""
return 0.5 * (1.0 + math.erf(float(x) / math.sqrt(2.0)))
def _d1_d2(s: float, k: float, t: float, sigma: float) -> tuple[float, float]:
"""The Black-Scholes d1, d2 (r=0). Assumes s,k>0 and t,sigma>0 (callers handle the degenerate cases)."""
vol_sqrt_t = sigma * math.sqrt(t)
d1 = (math.log(s / k) + 0.5 * sigma * sigma * t) / vol_sqrt_t
d2 = d1 - vol_sqrt_t
return d1, d2
def bs_call(*, s: float, k: float, t: float, sigma: float) -> float:
"""Black-Scholes European CALL price (r=0). Degenerate T≤0 or σ≤0 → intrinsic max(SK, 0). Pure."""
s, k, t, sigma = float(s), float(k), float(t), float(sigma)
if s <= 0.0 or k <= 0.0:
return 0.0
if t <= 0.0 or sigma <= 0.0:
return max(s - k, 0.0)
d1, d2 = _d1_d2(s, k, t, sigma)
return s * _norm_cdf(d1) - k * _norm_cdf(d2)
def bs_put(*, s: float, k: float, t: float, sigma: float) -> float:
"""Black-Scholes European PUT price (r=0). Degenerate T≤0 or σ≤0 → intrinsic max(KS, 0). Pure."""
s, k, t, sigma = float(s), float(k), float(t), float(sigma)
if s <= 0.0 or k <= 0.0:
return 0.0
if t <= 0.0 or sigma <= 0.0:
return max(k - s, 0.0)
d1, d2 = _d1_d2(s, k, t, sigma)
return k * _norm_cdf(-d2) - s * _norm_cdf(-d1)

View File

@@ -27,7 +27,7 @@ from fxhnt.registry import STRATEGY_REGISTRY
# tagged under these MODELED sids — see `multistrat_exec_record.plan_and_record`'s
# `persist_execution(..., strategy_id="multistrat", ...)`). Their detail page gets the holdings/recent-
# trades/account-nav/recon section (D2.2/2.3); every other strategy simply doesn't have that data.
_IBKR_ACCOUNT_SIDS = frozenset({"multistrat", "vrp"})
_IBKR_ACCOUNT_SIDS = frozenset({"multistrat"})
def _expected_book_symbols() -> frozenset[str]:
@@ -38,7 +38,13 @@ def _expected_book_symbols() -> frozenset[str]:
from fxhnt.domain.strategies.multistrat import FUND_INSTRUMENTS
syms = set(FUND_INSTRUMENTS)
try:
syms |= {listing.ticker for listing in get_settings().ucits.map.values()}
# Include BOTH the cosmetic ticker AND the symbol IBKR actually REPORTS the holding under
# (`ibkr_symbol`, which diverges for the IEF line: ticker CBU0 -> reported CSBGU0, probe 2026-07-20).
# account_snapshot reads `p.contract.symbol`, so a legitimate book holding IBKR reports as CSBGU0 must
# be in this set or it is wrongly flagged a stray and dropped from the book's NLV.
for listing in get_settings().ucits.map.values():
syms.add(listing.ticker)
syms.add(listing.ibkr_symbol)
except Exception: # noqa: BLE001 — config-less test path: the US instruments alone still guard the check
pass
return frozenset(syms)
@@ -46,7 +52,7 @@ def _expected_book_symbols() -> frozenset[str]:
def _active_ibkr_account_sids() -> list[str]:
"""The IBKR-account sids that are still cockpit-visible — `_IBKR_ACCOUNT_SIDS` minus any ARCHIVED entry
(`STRATEGY_REGISTRY` `archived: True`, e.g. shelved/FALSIFIED vrp). Sorted for a deterministic render
(`STRATEGY_REGISTRY` `archived: True`, e.g. a shelved/FALSIFIED strategy). Sorted for a deterministic render
order. Every account-wide aggregation (the per-strategy breakdown, the recent-fills roll-up) iterates
THIS, so an archived book never surfaces a row/fill on the IBKR account page even though its fills are
still captured in the repo."""
@@ -80,7 +86,7 @@ class DashboardService:
def __init__(self, repo: ForwardNavRepo, ibkr_repo: IbkrAccountRepo | None = None) -> None:
self._repo = repo
# Optional: absent in most callers/tests (the ordinary crypto/registry tracks never touch it); when
# given, powers the IBKR real-account section on multistrat/vrp's detail pages (D2.2/2.3).
# given, powers the IBKR real-account section on multistrat's detail pages (D2.2/2.3).
self._ibkr_repo = ibkr_repo
def _safe_allocs(self) -> dict:
@@ -152,10 +158,11 @@ class DashboardService:
b = backtests.get(reg.strategy_id) # most strategies have no backtest verdict -> None
meta = STRATEGY_REGISTRY.get(reg.strategy_id, {}) # honest not-live flag lives in the Python registry
if meta.get("archived"): # defense-in-depth: `registry()` already excludes archived rows (the
continue # authoritative filter), so an archived sleeve (vrp/vrp_exec) never reaches here.
continue # authoritative filter), so an archived sleeve never reaches here.
min_days = _gate_min_days(reg.gate_spec) # warming = fewer forward days than the gate needs
out.append(FleetRow(
strategy_id=reg.strategy_id, display_name=display_name(reg.strategy_id, reg.display_name), sleeve=reg.sleeve,
strategy_id=reg.strategy_id, sleeve=reg.sleeve,
display_name=display_name(reg.strategy_id, reg.display_name),
days=s.days if s else 0,
total_return_pct=100.0 * s.total_return if s else 0.0,
sharpe=s.sharpe if s else 0.0, maxdd=s.maxdd if s else 0.0,
@@ -380,7 +387,7 @@ class DashboardService:
# silently carried in the account NLV — surfaced as a warning in the holdings view.
"in_book": sym in expected}
for sym, qty in positions.items()]
# GROSS exposure denominator: a defined-risk book (e.g. vrp put-credit-spreads = short put + long
# GROSS exposure denominator: a defined-risk book (e.g. a put-credit-spread book = short put + long
# wing) holds mixed-sign legs, so a signed total nets toward the small net credit and makes "% of
# book" blow past 100% / go negative. % of GROSS keeps every row's signed value but a sane [0,100]
# share that sums to ~100.
@@ -406,8 +413,8 @@ class DashboardService:
"stray_holdings": stray_holdings}
def detail(self, strategy_id: str) -> StrategyDetail | None:
# An ARCHIVED sleeve (STRATEGY_REGISTRY `archived: True` — e.g. shelved/FALSIFIED vrp/vrp_exec) must
# never render its detail page: a direct hit to `/strategy/vrp` returns None so the route 404s. This
# An ARCHIVED sleeve (STRATEGY_REGISTRY `archived: True` — e.g. a shelved/FALSIFIED strategy) must
# never render its detail page: a direct hit to its `/strategy/<id>` returns None so the route 404s. This
# is belt-and-suspenders — `registry()` below already excludes archived rows (single source) — but it
# keeps the hide explicit and independent of the repo filter.
if STRATEGY_REGISTRY.get(strategy_id, {}).get("archived"):
@@ -428,7 +435,8 @@ class DashboardService:
periods = {p: aggregate(dates, rets, p) for p in PERIODS} # raw daily/weekly/monthly, no normalization
pair = self._exec_pair(strategy_id, summaries)
return StrategyDetail(
strategy_id=strategy_id, display_name=display_name(strategy_id, reg.display_name), sleeve=reg.sleeve, venue=reg.venue,
strategy_id=strategy_id, sleeve=reg.sleeve, venue=reg.venue,
display_name=display_name(strategy_id, reg.display_name),
days=s.days if s else 0, total_return_pct=100.0 * s.total_return if s else 0.0,
sharpe=s.sharpe if s else 0.0, maxdd=s.maxdd if s else 0.0,
gate_status=s.gate_status if s else "WAIT",

View File

@@ -1,5 +1,5 @@
"""display_names.py — the SINGLE internal-id -> human-readable name map used by EVERY cockpit view (the
plain-language global constraint: no raw registry id — `multistrat`/`vrp`/`bybit_4edge` — ever reaches an
plain-language global constraint: no raw registry id — `multistrat`/`bybit_4edge` — ever reaches an
end user). Absorbs the interim `_SIM_BOOK_LABELS` local map that lived in `app.py` (Task 1) for the
sim/Backtest book pills — those two overrides now live here as the SSOT, and `app.py` reads through this
module instead of its own copy.
@@ -48,7 +48,7 @@ def display_name(strategy_id: str, fallback: str | None = None) -> str:
def asset_description(symbol: str) -> str:
"""Plain 'what it is' description for an asset symbol (e.g. SPY -> 'US stocks'). An UNMAPPED symbol (a
future vrp option contract, say) falls back to a plain generic 'other' rather than echoing a raw/odd
future option contract, say) falls back to a plain generic 'other' rather than echoing a raw/odd
ticker string — the symbol itself is still shown in its own column, so nothing is lost. Never blank,
never raises."""
return ASSET_DESCRIPTIONS.get(symbol, "other")

View File

@@ -1,46 +0,0 @@
"""Ingest the Deribit DVOL implied-vol index into the Bybit-namespaced feature store (table='bybit_features')
as feature 'dvol' for the perp symbols BTCUSDT / ETHUSDT.
The DVOL index is currency-keyed (BTC/ETH); the warehouse panels are symbol-keyed (the VRP eval reads `dvol`
alongside the BTCUSDT/ETHUSDT `close`), so the ingest maps currency -> "<CCY>USDT" and writes feature='dvol'
at ts = epoch_day*86400. ONLY 'dvol' is written — close/funding are the kline/funding ingests' job (the PK is
(feature, symbol, ts) so distinct features never collide). Idempotent (upsert overwrites in place, so re-runs
are safe). Per-currency skip-on-empty: a currency that returns no rows contributes nothing and never aborts.
"""
from __future__ import annotations
from typing import Any, Protocol
# Deribit DVOL currency -> the Bybit USDT-perp symbol the warehouse panels key on.
_CCY_TO_SYMBOL = {"BTC": "BTCUSDT", "ETH": "ETHUSDT"}
class _Deribit(Protocol):
def dvol_history(self, currency: str, *, start_ms: int | None = None) -> dict[int, float]: ...
def ingest_dvol(store: Any, deribit: _Deribit, currencies: tuple[str, ...] = ("BTC", "ETH"), *,
start_ms: int | None = None) -> dict[str, Any]:
"""Fetch + write the daily DVOL index for each `currency` into `store` (feature='dvol',
ts=epoch_day*86400, symbol="<CCY>USDT").
Returns {"symbols": N, "day_rows": M, "oldest_day": D|None}: N = currencies that produced >=1 day-row,
M = total day-rows written, D = the earliest epoch_day across all currencies (None if nothing ingested).
Per-currency skip-on-empty (a currency with no DVOL history contributes nothing, never aborts). Writes
are bulk (one transaction); the upsert is idempotent so re-running is safe and cheap."""
items: list[tuple[str, list[tuple[int, dict[str, float]]]]] = []
oldest_day: int | None = None
day_rows = 0
for ccy in currencies:
symbol = _CCY_TO_SYMBOL.get(ccy.upper(), f"{ccy.upper()}USDT")
daily = deribit.dvol_history(ccy, start_ms=start_ms)
if not daily:
continue
rows = [(day * 86_400, {"dvol": float(val)}) for day, val in sorted(daily.items())]
items.append((symbol, rows))
day_rows += len(rows)
run_oldest = min(daily)
oldest_day = run_oldest if oldest_day is None else min(oldest_day, run_oldest)
if items:
store.write_features_bulk(items)
return {"symbols": len(items), "day_rows": day_rows, "oldest_day": oldest_day}

View File

@@ -3,8 +3,8 @@ LOO marginal-Sharpe gate, run BEFORE any forward tracker is wired. This is the g
reconciliation gate on two EM-beta ETFs would PASS trivially (the forward NAV just tracks the backtest NAV)
and prove NOTHING about edge. The real question is decision-theoretic and answerable now from ~8-10y of
history: does adding CNYA.L / NDIA.L to the fund's ETF book RAISE the book's risk-adjusted return after
costs, or is it a co-crashing beta sleeve that fails the marginal gate — the SAME bar that killed VRP and
the token-unlock diversifier (see pearl_fxhnt_diversifier_hunt_2026_07_15).
costs, or is it a co-crashing beta sleeve that fails the marginal gate — the SAME bar that killed prior failed
diversifier candidates, including the token-unlock diversifier (see pearl_fxhnt_diversifier_hunt_2026_07_15).
PASS bar (predetermined, not chosen after seeing the numbers): marginal Sharpe > 0 after cost.
> 0 -> proceed to Stage 2 (wire the `em_asia` registry sleeve + reconciliation forward gate).
@@ -15,8 +15,8 @@ overlapping dates — CNYA.L ~10y, NDIA.L ~8y. Per-name verdicts (not a cross-na
windows are acceptable.
Two marginal measures are reported per name:
1. `marginal_sharpe` (SSOT weight-blend, domain/diversification/marginal.py) — DIRECTLY comparable to the
prior VRP / unlock verdicts, which used this exact primitive.
1. `marginal_sharpe` (SSOT weight-blend, domain/diversification/marginal.py) — DIRECTLY comparable to
prior failed diversifier candidates' verdicts (e.g. unlock), which used this exact primitive.
2. In-book delta-Sharpe — `book_series(FUND + candidate) - book_series(FUND)` through the FULL adaptive
machinery (vol-norm -> trust -> correlation de-risk), i.e. the candidate's effect where it would
actually live. The truer measure; if the two disagree, that disagreement is itself signal.
@@ -68,7 +68,7 @@ class CandidateVerdict:
window_start: str
window_end: str
n_days: int
marginal_sharpe_blend: float # SSOT weight-blend, cost-inclusive (comparable to VRP/unlock)
marginal_sharpe_blend: float # SSOT weight-blend, cost-inclusive (comparable to prior candidates/unlock)
delta_sharpe_inbook: float # book_series(FUND+cand) - book_series(FUND), cost-inclusive
corr_to_book: float
crash_drawdowns: dict[str, float] = field(default_factory=dict)

View File

@@ -9,7 +9,7 @@ module does NO new compute, only comparison, plus a read of the strategy-tagged
cost-to-trade inputs (median slippage, total fees).
`has_exec` is the gate: when the `<sid>_exec` track has no persisted forward_summary row yet (a real account
that hasn't run, or an exec twin that's suspended — e.g. `vrp_exec` today), there is nothing to reconcile
that hasn't run, or an exec twin that's suspended pending a fund decision), there is nothing to reconcile
against — every other field defaults to zero rather than computing on unavailable data."""
from __future__ import annotations

View File

@@ -1,19 +1,21 @@
"""Runtime staleness/health axis — the anti-silent-rot backbone.
WHY: vrp was silently broken for 2 weeks because a data failure degraded QUIETLY — it recomputed off a
frozen-but-stale `xsp_option_bars` table, its forward "updated" every night, and only a log line marked the
rot. The gate never noticed (a stale track just sits as a quiet WAIT). This module adds a HEALTH axis that is
WHY: a dead/archived strategy was once silently broken for 2 weeks because a data failure degraded
QUIETLY — it recomputed off a frozen-but-stale `xsp_option_bars` table, its forward "updated" every night,
and only a log line marked the rot. The gate never noticed (a stale track just sits as a quiet WAIT). This
module adds a HEALTH axis that is
SEPARATE from the gate: a track can be legitimately WAIT/building AND healthy; STALE/NO_REF/BROKEN is
orthogonal. Any live/gated track that stops updating surfaces LOUDLY on day 1 (red cockpit badge + ERROR log),
not a footnote two weeks later.
The health-check line is `record_mode`, which faithfully encodes "is this track EXPECTED to advance nightly":
- `recompute` tracks (sixtyforty/xsfunding/unlock/multistrat/vrp/bybit_4edge/…): the engine recomputes from
- `recompute` tracks (sixtyforty/xsfunding/unlock/multistrat/bybit_4edge/…): the engine recomputes from
fresh source data EVERY run, so a non-advancing `as_of` (or a frozen freeze table) is a real STALE fault.
vrp lives here → the incident is caught.
- `observed` tracks (the executed twins multistrat_exec/vrp_exec/bybit_4edge_exec): advance ONLY when a
human/executor books a fill, so they are legitimately DORMANT when execution is suspended (vrp_exec has no
IBKR options permission yet). Staleness/no-ref do NOT apply — they must not false-alarm. This is the
A `recompute` track lives here → an incident like the one above is caught.
- `observed` tracks (the executed twins multistrat_exec/bybit_4edge_exec): advance ONLY when a
human/executor books a fill, so they are legitimately DORMANT when execution is suspended (an executed
twin can lack IBKR options permission, for example). Staleness/no-ref do NOT apply — they must not
false-alarm. This is the
"intentionally-dormant reference_only" carve-out. (BROKEN — a health-compute error — still applies, since
a track whose health we cannot even determine is not healthy.)
@@ -21,14 +23,14 @@ States (worst wins, precedence BROKEN > STALE > NO_REF > HEALTHY):
- HEALTHY — advancing within threshold, or dormant-by-design, or a GENUINELY new recompute track (its active
`forward_anchor.t0` is within the threshold, so "no data yet" is expected, not rot).
- STALE(n) — a recompute track's source `as_of` OR its freeze table's max date is > threshold business days
behind the run date. The freeze axis catches the exact vrp failure mode: forward "updates" yet the frozen
input table is stale. Also STALE (not the HEALTHY "no data yet" carve-out): a recompute track with ZERO
behind the run date. The freeze axis catches the exact failure mode described above: forward "updates"
yet the frozen input table is stale. Also STALE (not the HEALTHY "no data yet" carve-out): a recompute track with ZERO
forward data whose anchor `t0` is OLDER than the threshold (broken-from-inception — never advanced since
it was created), or which has no active anchor at all (should have one; a live track with no anchor is a
real signal, never silently HEALTHY).
- NO_REF — a reconciliation-gated recompute track with NO resolvable backtest reference: the gate can NEVER
reconcile (it silently degrades to the min-window/clean-exec checks only). The M6 static invariant that
missed vrp, made a real runtime check.
missed the incident described above, made a real runtime check.
- BROKEN — the PER-TRACK health computation raised (e.g. a malformed registry entry for that strategy):
surfaced for that track only, not swallowed. This does NOT cover the repo-wide reads `evaluate_health`
does once up front (`migrate`/`all_summaries`/`all_backtest_summaries`/`xsp_freeze_max_date`) — those are
@@ -144,7 +146,7 @@ def evaluate_health(dsn: str, at: dt.datetime | None = None) -> list[HealthVerdi
summaries = {s.strategy_id: s for s in repo.all_summaries()}
backtest_sids = {b.strategy_id for b in repo.all_backtest_summaries()}
ref_alias = CockpitBacktestRefProvider._REF_ALIAS # the SAME alias the gate uses to resolve its ref
freeze_max = repo.xsp_freeze_max_date() # one query for the vrp freeze axis
freeze_max = repo.xsp_freeze_max_date() # one query for the frozen-input freeze axis
verdicts: list[HealthVerdict] = []
for sid, meta in STRATEGY_REGISTRY.items():

View File

@@ -42,7 +42,7 @@ class CockpitBacktestRefProvider:
# A forward track whose reconciliation reference is ANOTHER track's backtest (the executed multistrat
# track reconciles against the MODELED multistrat's basis-matched reference — spec C1 §4: execution decay
# shows as the executed track diverging from the modeled book's expectation).
_REF_ALIAS = {"multistrat_exec": "multistrat", "vrp_exec": "vrp", "bybit_4edge_exec": "bybit_4edge"}
_REF_ALIAS = {"multistrat_exec": "multistrat", "bybit_4edge_exec": "bybit_4edge"}
def __init__(self, repo: ForwardNavRepo) -> None:
self._cagr = {b.strategy_id: float(b.cagr) for b in repo.all_backtest_summaries()}

View File

@@ -1,5 +1,5 @@
"""Dispatch a sim book's honest-cost curve by its registry backtest.kind — bybit books read the precomputed
bybit_sleeve_ret table; recompute-replay books (multistrat/vrp, the IBKR paper strategies) read their own
bybit_sleeve_ret table; recompute-replay books (multistrat, the IBKR paper strategies) read their own
precomputed curve. The sim NEVER recomputes an edge on the request path (both sources are nightly-precomputed:
bybit_sleeve_ret by `_persist_bybit_book`, the IBKR curve by `_persist_track_backtest_ref`)."""
from __future__ import annotations
@@ -9,17 +9,17 @@ from collections.abc import Callable
from fxhnt.application.backtest_refs import registry_backtest_kind
# The Backtest page's book pills, in render order. bybit_4edge/bybit_4edge_levered are the existing Bybit
# books (report-kind, precomputed bybit_sleeve_ret). multistrat/vrp are the IBKR paper strategies
# books (report-kind, precomputed bybit_sleeve_ret). multistrat is the IBKR paper strategy
# (recompute-replay-kind) added by Task 1/D1 — their curve is a precomputed per-day return series read the
# same cheap way (see `sim_returns_for`'s dispatch below).
SIM_BOOKS: tuple[str, ...] = ("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered", "vrp")
SIM_BOOKS: tuple[str, ...] = ("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered")
def sim_returns_for(book: str, *, bybit_returns: Callable[[], dict[str, dict[int, float]]],
ibkr_curve: Callable[[str], dict[str, dict[int, float]]]
) -> dict[str, dict[int, float]]:
"""`book`'s per-sleeve honest-cost return series — `{sleeve: {epoch_day: ret}}` — dispatched by the
registry's `backtest.kind` for `book`: `recompute-replay` (multistrat/vrp) reads the precomputed IBKR
registry's `backtest.kind` for `book`: `recompute-replay` (multistrat) reads the precomputed IBKR
curve via `ibkr_curve(book)`; everything else (the `report`-kind Bybit books) reads the precomputed
`bybit_sleeve_ret` table via `bybit_returns()` — the existing path, unchanged."""
kind = registry_backtest_kind(book)

View File

@@ -1,167 +0,0 @@
"""The defined-risk XSP put-credit-spread VRP sleeve (recompute mode). Recomputes the daily book ENTIRELY off
frozen `xsp_option_bars` rows — put-call-parity forward -> IV inversion -> ~20Δ short strike, long a fixed
width below -> smooth MODEL-mark daily P&L per dollar of defined-risk margin (the daily value is rebuilt from
the day's forward + ATM IV via the shared `vrp_marking` valuation — the same one the live exec twin uses for
its profit-take signal — NOT a difference of two individually-noisy raw leg quotes). Never calls OPRA. Mirrors
how MultiStratStrategy produces both the full-history ref (advance(None,{})) and the nightly forward window."""
from __future__ import annotations
import datetime as dt
from typing import Any
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
from fxhnt.application import vrp_marking, vrp_pricing
class VrpStrategy:
def __init__(self, bars: XspOptionBarsRepo, *, dte_lo: int = 20, dte_hi: int = 45, dte_target: int = 30,
short_delta: float = 0.20, width_points: float = 5.0, ladder: int = 5,
profit_take: float = 0.5, entry_weekday: int = 0) -> None:
self._bars = bars
self._dte_lo, self._dte_hi, self._dte_target = dte_lo, dte_hi, dte_target
self._short_delta, self._width, self._ladder = short_delta, width_points, ladder
self._profit_take, self._entry_weekday = profit_take, entry_weekday
self.margin = width_points * 100.0
@staticmethod
def _dte(day: str, expiry: dt.date) -> int:
return (expiry - dt.date.fromisoformat(day)).days
def _select_spread(self, day: str) -> dict | None:
"""Pick the ~30-DTE, ~20Δ short put + long a width below, priced from frozen marks on `day`."""
chain = self._bars.chain_asof(day, self._dte_lo, self._dte_hi)
if not chain:
return None
# nearest expiry to dte_target; ties broken by lexicographic date order (deterministic across
# processes) rather than `set` iteration order, which is PYTHONHASHSEED-salted.
expiries = sorted(dict.fromkeys(c.expiry for c in chain))
exp = min(expiries, key=lambda e: abs(self._dte(day, e) - self._dte_target))
puts = sorted((c for c in chain if c.expiry == exp), key=lambda c: c.strike)
tau = max(self._dte(day, exp), 1) / 365.0
marks = self._bars.bars_for([c.osi_symbol for c in puts], day, day)
priced = [(c, marks.get(c.osi_symbol, {}).get(day)) for c in puts]
priced = [(c, p) for c, p in priced if p is not None and p > 0.0]
if len(priced) < 2:
return None
forward, atm_iv = vrp_marking.forward_and_atm_iv(self._bars, day, exp, tau)
if forward is None:
# FALLBACK: no common strike with both a priced put AND a priced call for this expiry
# (degenerate/missing data) — approximate the forward with the highest surviving put strike,
# the closest available proxy to ATM.
forward = priced[-1][0].strike
# choose short strike whose |delta| is nearest short_delta
best = None
for c, price in priced:
iv = vrp_pricing.implied_vol_put(price, forward, c.strike, tau)
if iv is None:
continue
dlt = vrp_pricing.put_delta(forward, c.strike, iv, tau)
score = abs(dlt - self._short_delta)
if best is None or score < best[0]:
best = (score, c, price, iv)
if best is None:
return None
_, short_c, short_price, short_iv = best
long_strike = short_c.strike - self._width
long_c = next((c for c, _ in priced if abs(c.strike - long_strike) < 1e-6), None)
long_price = None
if long_c is not None:
long_price = self._bars.bars_for([long_c.osi_symbol], day, day).get(long_c.osi_symbol, {}).get(day)
if long_c is None or long_price is None:
return None
credit = short_price - long_price
if credit <= 0.0:
return None
# seed the mark IV with a clean ATM read (fall back to the short-leg IV when the ATM inversion failed);
# `_spread_value` re-derives F+IV each day but carries this on a degenerate day.
return dict(entry=day, expiry=exp.isoformat(), short_osi=short_c.osi_symbol, long_osi=long_c.osi_symbol,
short_strike=short_c.strike, long_strike=long_c.strike, credit=credit,
last_iv=(atm_iv if atm_iv is not None else short_iv))
def _spread_value(self, sp: dict, day: str, day_cache: dict | None = None) -> float | None:
"""MODEL value (cost to close) of the open spread on `day`, via the shared `vrp_marking` valuation the
live exec twin also uses (single source): smooth Black-Scholes from the day's forward + ATM IV, clipped
to the defined-risk bound [0, width]. None only when the expiry chain has aged out (caller closes — no
zombie). `day_cache` memoizes (F, IV) across spreads sharing an expiry in one day's pass."""
return vrp_marking.model_spread_value(self._bars, sp, day, self._width, day_cache)
def _settlement_value(self, sp: dict, day: str, day_cache: dict | None = None) -> float | None:
"""Terminal settlement value clip(K_short F, 0, width) from `day`'s forward — the shared
`vrp_marking.settlement_value`, realized on the close day (DTE≤1) instead of the theta model mark."""
return vrp_marking.settlement_value(self._bars, sp, day, self._width, day_cache)
def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]:
opens: list[dict] = list(extra.get("open_spreads", []))
rows: list[tuple[str, float]] = []
# Replay is read-heavy over every trading day after `last_date`; preload that whole slice in ONE
# query so the per-day chain_asof/bars_for below serve from memory instead of a round-trip each
# (the N+1 that made a full ~1600-day recompute issue thousands of queries). Scoped to `last_date`
# so an incremental from-anchor recompute only loads the days it will actually touch.
preload = getattr(self._bars, "preload", None)
if callable(preload):
preload(after=last_date)
for day in self._bars.trading_days(last_date):
# 1) mark open spreads -> daily defined-risk return
pnl = 0.0
marked = 0 # spreads that produced a mark today (pre-closure)
still: list[dict] = []
day_cache: dict = {} # memoize (F, IV) per (day, expiry) across rungs
for sp in opens:
val = self._spread_value(sp, day, day_cache)
if val is None:
# the expiry's chain has aged/moved out of the frozen slice (no clean forward) -- CLOSE
# rather than zombie-hold: don't re-append to `still` and don't count it in `marked`, so a
# spread whose marks stop no longer permanently occupies a ladder slot.
#
# INTENTIONAL ref-vs-live ASYMMETRY (documented, not a bug): this pure-model backtest's
# ONLY mark IS the model value, so model==None => the rung has no mark at all => drop it.
# The live exec twin (`vrp_exec_record.plan_vrp_ladder`) legitimately DIFFERS on the same
# degenerate slice — a slice where a rung's own put legs price (real `_mark_spread` actual
# mark) but no ATM call/put pair prices anywhere (model==None): the twin HOLDS such a rung
# and manages it on the real fill mark until DTE<=1 or the model recovers, because it has
# strictly MORE information (an actual close price) than this recompute has. Making the two
# consistent would mean either giving the backtest a raw-leg fallback (re-introducing the
# exact mark noise this whole fix removed) or making the twin discard a real fill mark it
# holds — both regressions. The asymmetry is correct; the twin never has LESS info.
continue
marked += 1
# 2) management: 50% profit (on the smooth model_val) or expiry-1. On the expiry close day,
# realize the terminal SETTLEMENT (intrinsic in-the-moneyness capped at width) rather than the
# theta-model mark -- the put-credit-spread's true P&L at expiry.
expiry = dt.date.fromisoformat(sp["expiry"])
closing = val <= (1.0 - self._profit_take) * sp["credit"] or self._dte(day, expiry) <= 1
if self._dte(day, expiry) <= 1:
settle = self._settlement_value(sp, day, day_cache)
if settle is not None:
val = settle
prev = sp.get("last_val", sp["credit"]) # baseline = entry-day model value (see entry below)
# DEFINED-RISK BOUND: value is clipped to [0, width] (in `_spread_value`) or is the settlement
# intrinsic clip(K_shortF, 0, width). The daily increments telescope to a per-spread cumulative
# P&L of (entry_model_value value)·100 dollars; the ABSOLUTE defined-risk bound [width·100,
# +width·100] holds unconditionally, and on a clean BS surface (entry model value == credit) it
# tightens to exactly [(widthcredit)·100, +credit·100]. No day (nor the total) breaches it.
# option marks are per-share; x100 = the contract multiplier, converting the point-change
# into DOLLARS so the numerator matches self.margin's dollar basis (width_points*100) --
# short-vol: value falling = profit.
pnl += (prev - val) * 100.0
sp["last_val"] = val
if closing:
continue # closed (drop from book)
still.append(sp)
opens = still
# denominator = spreads MARKED this day (before closure), matching pnl's numerator basis —
# NOT the post-closure survivor count, which would asymmetrically inflate the return on
# closure days.
rows.append((day, pnl / (max(1, marked) * self.margin)))
# 3) weekly entry. Baseline the P&L at the MODEL value on the entry day (not the raw
# short_closelong_close credit): every daily return is then the change in the SMOOTH model value,
# so the first mark day can't inject the noisy far-OTM leg marks that the raw credit carries. The
# raw `credit` is still the profit-take threshold and the defined-risk reference; on a clean
# (BS-consistent) surface the entry model value equals the credit, so the bound is unchanged.
if dt.date.fromisoformat(day).weekday() == self._entry_weekday and len(opens) < self._ladder:
sp = self._select_spread(day)
if sp is not None:
entry_val = self._spread_value(sp, day, day_cache)
sp["last_val"] = entry_val if entry_val is not None else sp["credit"]
opens.append(sp)
return rows, {"open_spreads": opens}

View File

@@ -1,430 +0,0 @@
"""READ-ONLY, TAIL-HONEST evaluator for the DEFINED-RISK VRP — the *deployable* form of the vol-risk premium.
Why this module exists (the naked VRP failed the tail)
------------------------------------------------------
`vrp_eval.py` confirmed a REAL gross VRP edge (a short-variance swap: Sharpe ~1, OOS positive) but it FAILS
the deploy gate because the NAKED short-vol payoff has an UNBOUNDED left tail — a vol spike drove the modelled
maxDD to ~74% and realistic OPTION costs (50100bp) ate the rest. Selling naked variance is uninsurable.
The textbook fix is to CAP the tail: sell the at-the-money straddle (collect the premium) but BUY out-of-the-
money WINGS (a protective strangle) so the maximum loss is BOUNDED — an iron-condor / straddle-with-wings.
This module prices that spread from DVOL via Black-Scholes and re-runs the SAME tail-aware harness as the
naked evaluator, so the two are directly comparable. The whole point is the maxDD: the wings should keep the
defined-risk curve well above the 40% floor (ideally < 20%) where the naked 74% failed.
The construction (defined-risk short-vol, per asset, rolling `swap_days`=30, entered day t)
------------------------------------------------------------------------------------------
* Spot S = close_t. ATM IV = DVOL_t/100 (annualised, CAUSAL — known at entry). T = swap_days/365. r=0.
* SELL the ATM straddle: 1 call + 1 put struck at K≈S, priced BS(S, S, T, IV) → COLLECT this premium.
* BUY OTM wings (a strangle) at K_up = S·(1+w), K_dn = S·(1w), where `w` = wing width. `w` may be given
directly as a fraction (`wing_width`) OR derived from the ATM std-move (`wing_in_atm_std`·IV·√T) — the
latter is the natural "N sigma" placement. PAY the wing premium BS at the wing strikes.
* NET PREMIUM (the defined-risk credit) = straddle_premium strangle_premium (per 1 unit of spot notional,
expressed as a FRACTION of S so it composes with the daily-return ladder).
* AT EXPIRY (t+swap_days), realized move m = close_{t+swap_days}/S 1. The position P&L (as a fraction of S):
pnl = net_premium |m| (the short straddle pays the realized |move|)
+ max(|m| w, 0) (the wings refund everything BEYOND the wing distance w)
so the loss is BOUNDED below at net_premium w (= (w net_premium), the DEFINED max loss). A huge
move can NEVER lose more than the wing distance net of the credit — that is the entire thesis.
Skew (crypto puts are richer) — the honesty caveat
--------------------------------------------------
We have only the ATM DVOL, not a full smile. Crypto put wings trade at a HIGHER IV than ATM (put skew), so a
short-condor pays MORE for its put wing than flat-ATM pricing implies. We model this with a `put_skew_vol`
bump (in vol points, default +5): the put legs (short ATM put + long put wing) are priced at IV + bump/100.
The long put wing costs more (reduces our credit, conservative) AND the short ATM put collects more (raises
credit). The NET sign of a uniform put-IV bump on a vertical is small; the bigger honesty point is the
OPPOSITE choice: pricing the put wing at FLAT ATM IV (skew bump = 0) UNDER-prices the protection we buy →
OVER-states the credit → the modelled result is then mildly OPTIMISTIC. We therefore default to a non-zero
put-skew bump and DOCUMENT that bump=0 is the optimistic bound. Either way the price is a BS approximation
from a single ATM vol, not a real chain — a modelling caveat the deploy decision must weigh alongside the
cross-venue caveat (signal = Deribit DVOL; execution = Bybit options).
Reuse
-----
The book ladder, vol-target sizing, cost model, verify / walk-forward / look-ahead / correlation-vs-4-edges /
per-year / worst-day / maxDD / tail-aware PASS-FAIL gate are REUSED from `vrp_eval`: the `_defined_risk_book`
context manager temporarily binds `vrp_eval._book_short_var` to the capped-spread ladder (with the wing params
closed over), so the entire harness runs verbatim on the defined-risk daily P&L instead of the variance-swap
P&L — the defined-risk verdict is computed by the exact same judge as the naked one.
READ-ONLY: no store.write_* — the live paper_* tables are never touched.
"""
from __future__ import annotations
import contextlib
import math
import statistics as st
from typing import Any
from fxhnt.application import vrp_eval
from fxhnt.application.black_scholes import bs_call, bs_put
from fxhnt.application.vrp_eval import (
_DIRECTIONS,
_VRP_SYMBOLS,
_daily_close_returns,
_FeatureStore,
)
_DEFAULT_SWAP_DAYS = vrp_eval._DEFAULT_SWAP_DAYS
_DEFAULT_TARGET_VOL = vrp_eval._DEFAULT_TARGET_VOL
_DEFAULT_WING_WIDTH = 0.15 # default wing distance: ±15% of spot (a defined-risk condor width)
_DEFAULT_PUT_SKEW_VOL = 5.0 # crypto put-skew bump (vol points) added to the put-leg IV; 0 = optimistic
_N_LEGS = 4 # legs per defined-risk condor: sell call, sell put, buy call-wing, buy put-wing
# --- 0. the TURNOVER / COST model — the amortized daily roll (NOT full-ladder-daily) -------------
#
# THE BUG (now fixed): the reused naked harness sizes the curve with `vrp_eval._sized_series`, whose cost model
# is `haircut = k · cost_bps/1e4` charged EVERY day. That is the NAKED short-variance model — a delta-hedged
# straddle re-established/re-hedged daily, so the WHOLE k-leverage book turns over each day. It is WRONG for the
# defined-risk CONDOR LADDER, which holds each 4-leg spread ~`swap_days` (=30) days to EXPIRY. In that ladder:
#
# * exactly ONE new spread is OPENED per day (its n_legs legs are traded — the entry round-trip);
# * exactly ONE spread EXPIRES per day — options held to expiry SETTLE, there is NO closing trade/cost;
# * so the daily TRADED notional is (n_legs / swap_days) of the gross book's option notional — NOT the whole
# ladder re-traded. Charging the full ladder every day over-charges turnover by swap_days/n_legs (= 30/4 ≈
# 7.5x), which is what made the defined-risk curve falsely collapse (5.96 @ 5.5bp, 86 @ 50bp).
#
# Equivalently: charge each spread its entry leg cost ONCE (n_legs · cost_bps · leg_notional, held to expiry, no
# exit), then amortize across the swap's life — one spread opens per day, so the per-day haircut on the sized
# (leverage-k) book is k · (n_legs / swap_days) · cost_bps/1e4. The realized per-year drag at leverage 1 is then
# n_legs · cost_bps · (365/swap_days) ≈ 24%/yr at 50bp, ~2-3%/yr at 5.5bp — proportionate, not destroying.
def _defined_risk_daily_cost(*, k: float, cost_bps: float, swap_days: int = _DEFAULT_SWAP_DAYS) -> float:
"""The AMORTIZED daily-roll haircut for the defined-risk condor ladder, as a per-day return drag on the
vol-targeted (leverage-`k`) book: `k · (n_legs / swap_days) · cost_bps/1e4`.
One spread (n_legs legs) is OPENED per day and one EXPIRES (settles — no closing trade), so the daily
traded notional is (n_legs / swap_days) of the gross book, not the whole ladder. cost_bps is applied to the
OPTION leg notional, consistent with the premium/P&L scaling (both fractions of spot ⇒ the leg notional that
the credit is collected on). `swap_days<=0` ⇒ 0. Pure."""
if not cost_bps or swap_days <= 0:
return 0.0
return float(k) * (_N_LEGS / float(swap_days)) * (float(cost_bps) / 1e4)
def _defined_risk_sized_series(raw: dict[int, float], *, k: float, cost_bps: float,
swap_days: int = _DEFAULT_SWAP_DAYS) -> dict[int, float]:
"""`{day: net_ret}` = k·raw the AMORTIZED daily-roll haircut. Mirrors `vrp_eval._sized_series` but replaces
its full-ladder-daily cost with the defined-risk daily roll (`_defined_risk_daily_cost`). Pure."""
haircut = _defined_risk_daily_cost(k=k, cost_bps=cost_bps, swap_days=swap_days)
out: dict[int, float] = {}
for d in sorted(raw):
v = k * raw[d] - haircut
if math.isfinite(v):
out[int(d)] = float(v)
return out
# --- 1. the defined-risk spread: credit at entry + bounded payoff at expiry ----------------------
def defined_risk_credit(*, s: float, iv: float, t: float, wing_width: float,
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL) -> float:
"""The NET PREMIUM collected at entry for the defined-risk short-vol spread, as a FRACTION of spot S.
credit = [ short ATM straddle ] [ long OTM strangle wings ]
= ( call(S,S) + put(S,S) ) ( call(S, S·(1+w)) + put(S, S·(1w)) ) (all ÷ S)
The PUT legs (short ATM put + long put wing) are priced at IV + put_skew_vol/100 (crypto put skew); the
CALL legs at the flat ATM IV. `iv` is the annualised ATM vol fraction (DVOL/100); `t` years; `w`=wing_width
a positive fraction of S. Returned per 1 unit of spot notional (÷S) so it composes with the return ladder.
A WIDER wing → CHEAPER wing → LARGER credit (but a larger max loss); the naked limit is w→∞ (no wing cost,
no cap). Pure."""
iv = float(iv)
put_iv = iv + float(put_skew_vol) / 100.0
k_up = s * (1.0 + wing_width)
k_dn = s * (1.0 - wing_width)
short_straddle = bs_call(s=s, k=s, t=t, sigma=iv) + bs_put(s=s, k=s, t=t, sigma=put_iv)
long_wings = bs_call(s=s, k=k_up, t=t, sigma=iv) + bs_put(s=s, k=k_dn, t=t, sigma=put_iv)
return (short_straddle - long_wings) / s
def defined_risk_payoff(*, credit: float, realized_move: float, wing_width: float,
direction: str = "short_vol") -> float:
"""The defined-risk short-vol P&L at expiry, as a FRACTION of spot. `realized_move` m = S_T/S 1.
pnl = credit |m| + max(|m| w, 0) (short_vol)
The short straddle pays the realized |move|; the long wings REFUND everything beyond the wing distance w,
so the loss is BOUNDED below at credit w. A calm expiry (|m| ≤ ... ) keeps (most of) the credit; a huge
move loses AT MOST w credit (the DEFINED risk), never more. `direction="long_vol"` negates the whole
structure (buy the straddle, sell the wings — capped GAIN, the mirror). Pure."""
if direction not in _DIRECTIONS:
raise ValueError(f"direction must be one of {_DIRECTIONS}, got {direction!r}")
m = abs(float(realized_move))
short_vol_pnl = float(credit) - m + max(m - float(wing_width), 0.0)
sign = 1.0 if direction == "short_vol" else -1.0
return sign * short_vol_pnl
def defined_risk_max_loss(*, credit: float, wing_width: float) -> float:
"""The DEFINED (bounded) worst-case loss of the short-vol spread, as a fraction of spot: w credit
(a positive number = the most we can lose). This is the whole point — the naked short-straddle's loss is
unbounded; the wings cap it here. Pure."""
return float(wing_width) - float(credit)
# --- 2. the per-asset rolling ladder of defined-risk spreads ------------------------------------
def _asset_defined_risk_ladder(cser: dict[int, float], dser: dict[int, float], *, direction: str,
swap_days: int, wing_width: float, put_skew_vol: float,
wing_in_atm_std: float | None) -> dict[int, float]:
"""The rolling defined-risk-spread LADDER for ONE asset → `{day: daily_pnl}` (UNSIZED, fraction of spot).
For every entry day s with a known close S_s and DVOL_s (causal): price the spread credit at entry, look
forward `swap_days` to the expiry close, compute the bounded expiry P&L from the realized move, and SPREAD
that single expiry P&L over the spread's life (÷ its length) — accruing to each day it covers, exactly like
the variance-swap ladder. The day-t book P&L is the AVERAGE of all in-flight spreads' per-day accruals. If
`wing_in_atm_std` is set, the wing distance is `wing_in_atm_std · IV · √T` (an N-sigma placement) instead
of the fixed `wing_width`. CAUSAL: credit + wing placement use entry-day IV; the realized move is strictly
forward of entry. Pure."""
rets = _daily_close_returns(cser)
if not rets:
return {}
ret_days = sorted(rets)
closes = cser
pnl_sum: dict[int, float] = {}
pnl_cnt: dict[int, int] = {}
t_years = swap_days / 365.0
for idx, s_day in enumerate(ret_days[:-1] if len(ret_days) > 1 else ret_days):
window_days = ret_days[idx + 1: idx + 1 + swap_days]
if not window_days:
continue
spot = closes.get(s_day)
dvol_entry = dser.get(s_day)
expiry_close = closes.get(window_days[-1])
if spot is None or dvol_entry is None or expiry_close is None:
continue
if not (math.isfinite(spot) and spot > 0 and math.isfinite(expiry_close)):
continue
iv = float(dvol_entry) / 100.0
w = wing_in_atm_std * iv * math.sqrt(t_years) if wing_in_atm_std is not None else wing_width
if not (0.0 < w < 1.0):
w = min(max(w, 1e-6), 0.999) # keep wing strikes strictly between 0 and 2·S
credit = defined_risk_credit(s=spot, iv=iv, t=t_years, wing_width=w, put_skew_vol=put_skew_vol)
realized_move = expiry_close / spot - 1.0
payoff = defined_risk_payoff(credit=credit, realized_move=realized_move, wing_width=w,
direction=direction)
per_day = payoff / len(window_days)
for d in window_days:
pnl_sum[d] = pnl_sum.get(d, 0.0) + per_day
pnl_cnt[d] = pnl_cnt.get(d, 0) + 1
return {d: pnl_sum[d] / pnl_cnt[d] for d in pnl_sum}
def _book_defined_risk(store: _FeatureStore, *, direction: str, swap_days: int = _DEFAULT_SWAP_DAYS,
wing_width: float = _DEFAULT_WING_WIDTH, put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
wing_in_atm_std: float | None = None) -> dict[int, float]:
"""`{day: raw_defined_risk_ret}` — the BTC+ETH EQUAL-WEIGHT (mean over assets present) raw, UNSIZED (k=1)
rolling defined-risk-spread daily return. CAUSAL + READ-ONLY (pure off panels). Empty when no overlapping
dvol+close data. Mirrors `vrp_eval._book_short_var` but uses the BS-priced capped spread ladder."""
close = store.read_panel(list(_VRP_SYMBOLS), "close")
dvol = store.read_panel(list(_VRP_SYMBOLS), "dvol")
close = {s: v for s, v in close.items() if v}
dvol = {s: v for s, v in dvol.items() if v}
if not close or not dvol:
return {}
per_day: dict[int, list[float]] = {}
for sym in _VRP_SYMBOLS:
cser = close.get(sym, {})
dser = dvol.get(sym, {})
if not cser or not dser:
continue
ladder = _asset_defined_risk_ladder(cser, dser, direction=direction, swap_days=swap_days,
wing_width=wing_width, put_skew_vol=put_skew_vol,
wing_in_atm_std=wing_in_atm_std)
for d, v in ladder.items():
if math.isfinite(v):
per_day.setdefault(int(d), []).append(v)
return {d: st.mean(vals) for d, vals in per_day.items() if vals}
# --- 3. reuse the naked harness by swapping in the defined-risk raw book ------------------------
#
# vrp_eval's verify / walk-forward / metrics all build their raw series by calling `_book_short_var`. We run
# them against the DEFINED-RISK book by temporarily binding `vrp_eval._book_short_var` to our capped builder
# (with the wing params closed over) for the duration of the call. This REUSES the entire tail-aware harness
# — vol-target sizing, cost grid, per-year, worst-day, maxDD, corr-vs-4-edges, look-ahead audit, PASS/FAIL
# gate — verbatim, so the defined-risk verdict is computed by the exact same judge as the naked one.
@contextlib.contextmanager
def _defined_risk_book(*, wing_width: float, put_skew_vol: float, wing_in_atm_std: float | None,
swap_days: int = _DEFAULT_SWAP_DAYS):
"""Bind `vrp_eval._book_short_var` (+ leaked control) to the defined-risk builders AND `vrp_eval._sized_series`
to the defined-risk AMORTIZED-ROLL cost model, for the duration of the `with` block. The cost rebind is the
turnover FIX: the naked harness's `_sized_series` charges the full ladder every day (k·cost_bps), which
over-charges a held-to-expiry 30-day condor ladder by ~swap_days/n_legs; the defined-risk series charges only
the one-spread-per-day roll (`_defined_risk_sized_series`). Restores the originals on exit (even on error)."""
orig = vrp_eval._book_short_var
orig_leaked = vrp_eval._book_short_var_leaked
orig_sized = vrp_eval._sized_series
def _capped(store, *, direction, swap_days=swap_days, _leak=False):
if _leak:
return _book_defined_risk_leaked(store, direction=direction, swap_days=swap_days,
wing_width=wing_width, put_skew_vol=put_skew_vol,
wing_in_atm_std=wing_in_atm_std)
return _book_defined_risk(store, direction=direction, swap_days=swap_days, wing_width=wing_width,
put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std)
def _capped_leaked(store, *, direction):
return _book_defined_risk_leaked(store, direction=direction, wing_width=wing_width,
put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std)
def _capped_sized(raw, *, k, cost_bps):
return _defined_risk_sized_series(raw, k=k, cost_bps=cost_bps, swap_days=swap_days)
vrp_eval._book_short_var = _capped
vrp_eval._book_short_var_leaked = _capped_leaked
vrp_eval._sized_series = _capped_sized
try:
yield
finally:
vrp_eval._book_short_var = orig
vrp_eval._book_short_var_leaked = orig_leaked
vrp_eval._sized_series = orig_sized
def _book_defined_risk_leaked(store: _FeatureStore, *, direction: str, swap_days: int = _DEFAULT_SWAP_DAYS,
wing_width: float = _DEFAULT_WING_WIDTH,
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
wing_in_atm_std: float | None = None) -> dict[int, float]:
"""NEGATIVE-CONTROL defined-risk book: prices each spread's credit/wings off the EXPIRY-day IV (knowable
only at maturity), a look-ahead leak the causality audit must catch. Mirrors the variance-swap leak."""
close = store.read_panel(list(_VRP_SYMBOLS), "close")
dvol = store.read_panel(list(_VRP_SYMBOLS), "dvol")
close = {s: v for s, v in close.items() if v}
dvol = {s: v for s, v in dvol.items() if v}
if not close or not dvol:
return {}
t_years = swap_days / 365.0
per_day: dict[int, list[float]] = {}
for sym in _VRP_SYMBOLS:
cser = close.get(sym, {})
dser = dvol.get(sym, {})
if not cser or not dser:
continue
rets = _daily_close_returns(cser)
if not rets:
continue
ret_days = sorted(rets)
for idx, s_day in enumerate(ret_days[:-1] if len(ret_days) > 1 else ret_days):
window_days = ret_days[idx + 1: idx + 1 + swap_days]
if not window_days:
continue
spot = cser.get(s_day)
expiry_close = cser.get(window_days[-1])
dvol_leak = dser.get(window_days[-1]) # leak: IV dated at expiry (look-ahead)
if spot is None or expiry_close is None or dvol_leak is None:
continue
if not (math.isfinite(spot) and spot > 0 and math.isfinite(expiry_close)):
continue
iv = float(dvol_leak) / 100.0
w = (wing_in_atm_std * iv * math.sqrt(t_years)) if wing_in_atm_std is not None else wing_width
if not (0.0 < w < 1.0):
w = min(max(w, 1e-6), 0.999)
credit = defined_risk_credit(s=spot, iv=iv, t=t_years, wing_width=w, put_skew_vol=put_skew_vol)
payoff = defined_risk_payoff(credit=credit, realized_move=expiry_close / spot - 1.0,
wing_width=w, direction=direction)
per_day_pnl = payoff / len(window_days)
for d in window_days:
per_day.setdefault(int(d), []).append(per_day_pnl)
# average over in-flight spreads, mirroring the causal ladder's per-day mean over assets+swaps
return {d: st.mean(vals) for d, vals in per_day.items() if vals}
# --- 4. public entry points (mirror vrp_eval, with the wing params) ----------------------------
def defined_risk_returns_from_store(store: _FeatureStore, *, cost_bps: float = 5.5,
direction: str = "short_vol",
target_ann_vol: float = _DEFAULT_TARGET_VOL,
wing_width: float = _DEFAULT_WING_WIDTH,
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
wing_in_atm_std: float | None = None) -> dict[int, float]:
"""READ-ONLY per-day DEFINED-RISK VRP return series `{epoch_day: net_ret}`. BTC+ETH equal-weight capped
short-vol spread, vol-targeted, net of cost. Writes NOTHING."""
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
return vrp_eval.vrp_returns_from_store(store, cost_bps=cost_bps, direction=direction,
target_ann_vol=target_ann_vol)
def defined_risk_metrics_from_store(store: _FeatureStore, *, cost_bps: float = 5.5,
direction: str = "short_vol",
target_ann_vol: float = _DEFAULT_TARGET_VOL,
wing_width: float = _DEFAULT_WING_WIDTH,
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
wing_in_atm_std: float | None = None) -> dict[str, Any]:
"""READ-ONLY DEFINED-RISK VRP metrics (the standard curve metrics + tail diagnostics). READ-ONLY."""
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
m = vrp_eval.vrp_metrics_from_store(store, cost_bps=cost_bps, direction=direction,
target_ann_vol=target_ann_vol)
m["wing_width"] = wing_width
m["put_skew_vol"] = put_skew_vol
return m
def verify_defined_risk_edge(store: _FeatureStore, *, cost_bps: float = 5.5,
cost_grid: tuple[float, ...] = (5.5, 11.0, 22.0, 50.0, 100.0),
direction: str = "short_vol",
target_ann_vol: float = _DEFAULT_TARGET_VOL,
wing_width: float = _DEFAULT_WING_WIDTH,
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
wing_in_atm_std: float | None = None,
universe: set[str] | None = None,
unlock_events: list[dict[str, Any]] | None = None) -> dict[str, Any]:
"""READ-ONLY adversarial verification of the DEFINED-RISK VRP (both directions) — the naked `verify`
harness fed the capped spread book (cost grid incl realistic option costs, per-year, worst-day, maxDD,
corr-vs-4-edges). Adds `wing_width`/`put_skew_vol` to the report. READ-ONLY."""
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
rep = vrp_eval.verify_vrp_edge(store, cost_bps=cost_bps, cost_grid=cost_grid, direction=direction,
target_ann_vol=target_ann_vol, universe=universe,
unlock_events=unlock_events)
if rep.get("available"):
rep["wing_width"] = wing_width
rep["put_skew_vol"] = put_skew_vol
rep["construction_note"] = ("DEFINED-RISK: short ATM straddle + long OTM wings (BS-priced from DVOL); "
"max loss BOUNDED at wing_width credit")
return rep
def look_ahead_audit_defined_risk(store: _FeatureStore, *, direction: str = "short_vol",
_leak: bool = False, wing_width: float = _DEFAULT_WING_WIDTH,
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
wing_in_atm_std: float | None = None) -> dict[str, Any]:
"""READ-ONLY strict-causality audit of the DEFINED-RISK book: prove each spread prices its credit/wings off
the ENTRY-day IV, never the expiry IV. Reuses `vrp_eval.look_ahead_audit_vrp` against the capped builder.
Returns its `{causal, leak_days, n_days_compared}` shape."""
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
return vrp_eval.look_ahead_audit_vrp(store, direction=direction, _leak=_leak)
def look_ahead_audit_defined_risk_ok(store: _FeatureStore, *, direction: str = "short_vol",
_leak: bool = False, **kw: Any) -> bool:
"""Convenience boolean: True iff the defined-risk credit/wing pricing is strictly causal (no look-ahead)."""
return bool(look_ahead_audit_defined_risk(store, direction=direction, _leak=_leak, **kw)["causal"])
def walk_forward_defined_risk(store: _FeatureStore, *, cost_bps: float = 5.5, window_days: int = 180,
train_frac: float = 0.6, target_ann_vol: float = _DEFAULT_TARGET_VOL,
max_dd_floor: float = -0.40, wing_width: float = _DEFAULT_WING_WIDTH,
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
wing_in_atm_std: float | None = None,
universe: set[str] | None = None) -> dict[str, Any]:
"""READ-ONLY, TAIL-AWARE OOS / WALK-FORWARD validation of the DEFINED-RISK VRP — the DEPLOY GATE — via the
naked `walk_forward_vrp` harness fed the capped book. The verdict is the SAME tail-aware PASS/FAIL: OOS
Sharpe>0, positive in >70% of windows, maxDD (full AND OOS) above the floor, look-ahead clean. The wings
are meant to make the `tail_survivable` check PASS where the naked book failed (74%). READ-ONLY."""
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
rep = vrp_eval.walk_forward_vrp(store, cost_bps=cost_bps, window_days=window_days,
train_frac=train_frac, target_ann_vol=target_ann_vol,
max_dd_floor=max_dd_floor, universe=universe)
if rep.get("available"):
rep["wing_width"] = wing_width
rep["put_skew_vol"] = put_skew_vol
rep["construction_note"] = ("DEFINED-RISK: short ATM straddle + long OTM wings (BS-priced from DVOL); "
"max loss BOUNDED at wing_width credit (caps the naked 74% tail)")
return rep

View File

@@ -1,554 +0,0 @@
"""READ-ONLY, TAIL-HONEST evaluator for the VRP (volatility-risk-premium) edge — a genuinely DIFFERENT edge
type (a vol premium, uncorrelated with the directional/flow edges already in the book).
Economic thesis
---------------
Deribit's DVOL is a 30-day annualised IMPLIED-vol index off the BTC/ETH options book — the market's expected
vol. It systematically exceeds the REALIZED vol of the underlying (the volatility risk premium), so SELLING
variance ("short-vol") earns the spread. The catch: the payoff has a fat LEFT tail — when a vol spike makes
RV >> IV, the short-vol position takes a large loss that can wipe out many days of premium. VRP looks great
until a spike; this evaluator is built to make the tail VISIBLE and to judge the edge NET of it.
The VRP return — a PROPER ROLLING VARIANCE SWAP (per asset, per day — CAUSAL)
----------------------------------------------------------------------------
The old construction paid a SINGLE day's `ret_t²` as the realized variance — a wildly noisy/fat-tailed RV
estimate whose spikes blew through the vol-target (the walk-forward showed Sharpe 40 / 96% per 180d window,
which is a CONSTRUCTION ARTIFACT, not a verdict). We replace it with the textbook variance swap.
A SHORT variance swap entered at day t with maturity `swap_days` (default 30):
* STRIKE (implied annual variance, CAUSAL — known at t): K_t = (DVOL_t / 100)²
DVOL_t is the annualised implied vol in vol POINTS (e.g. 50 → (0.50)² = 0.25 annual variance).
* REALIZED annual variance over (t, t+swap_days]: RV²_t = (365/swap_days)·Σ_{i=t+1..t+swap_days} ret_i²
the annualised realized variance from daily close-to-close returns over the swap's life.
* SHORT-VAR SWAP PAYOFF (per unit variance notional): K_t RV²_t
POSITIVE when implied>realized (the premium harvested), NEGATIVE on a vol-spike window (the tail).
Daily series via a rolling LADDER (`_book_short_var`)
----------------------------------------------------
Each day the book holds `swap_days` overlapping short-variance swaps — one ENTERED on each of the prior
`swap_days` days. On day t exactly ONE swap (the one entered at tswap_days) MATURES; its realized window
payoff `K_{tswap_days} RV²_{tswap_days}` is recognised, SPREAD over the swap's life (÷swap_days) so the
ladder produces ONE smooth daily P&L per day rather than a single raw `ret²`. Because RV² is a swap_days-window
AVERAGE of daily ret², the daily series is far LESS noisy than the old single-day proxy — a properly
vol-targeted curve must NOT lose ~everything in 180d. (Mathematically the ladder's day-t P&L equals the
average daily accrual of all in-flight swaps; the maturing-increment form is the simplest exactly-equivalent
expression.) `direction="long_vol"` negates (buy variance). `k` is the SIZING constant (below).
The book is BTC + ETH EQUAL-WEIGHT (`_book_short_var`), then vol-TARGETED.
Sizing (vol-target — matters, a naive short-variance has meaningless scale)
---------------------------------------------------------------------------
The raw average short-var series has an arbitrary scale (variance units). We VOL-TARGET it: pick `k` so the
full-sample annualised vol of `k · raw` equals `target_ann_vol` (default 0.15 = 15%/yr), making Sharpe/DD
comparable to the other edges. `k` is reported as the (in-sample) sizing constant and the implied
`avg_leverage`/exposure. CAVEAT: `k` here is full-sample in-sample (a known overfit-of-scale — the walk-
forward sizes `k` on TRAIN only). Costs scale with the leverage (a short-straddle is re-established/re-hedged
daily, so the per-day cost ≈ leverage × cost_bps).
READ-ONLY + cross-venue caveat
------------------------------
No store.write_* — the live paper_* tables are never touched. The SIGNAL is Deribit's DVOL; live EXECUTION
would be short Bybit option straddles (delta-hedged) — a cross-venue caveat the deploy decision must weigh.
"""
from __future__ import annotations
import datetime as dt
import math
import statistics as st
from typing import Any, Protocol
from fxhnt.application.bybit_book_eval import sleeve_returns_from_store
from fxhnt.application.bybit_carry_revalidate import _curve_metrics
_PERIODS_PER_YEAR = 365
_SQRT_YEAR = math.sqrt(_PERIODS_PER_YEAR)
_EXISTING_EDGES = ("tstrend", "unlock", "xsfunding", "positioning")
_VRP_SYMBOLS = ("BTCUSDT", "ETHUSDT") # the assets with a Deribit DVOL index (BTC/ETH)
_DIRECTIONS = ("short_vol", "long_vol")
_DEFAULT_TARGET_VOL = 0.15 # 15%/yr vol-target (sane, comparable to the other edges)
_DEFAULT_SWAP_DAYS = 30 # variance-swap maturity (DVOL is a 30-day implied-vol index)
class _FeatureStore(Protocol):
"""The panel reads the VRP pipeline needs (TimescaleFeatureStore satisfies this). READ-ONLY."""
def crypto_close_panel(self, *, suffix: str = "USDT") -> dict[str, dict[int, float]]: ...
def read_panel(self, symbols: list[str], feature: str) -> dict[str, dict[int, float]]: ...
def _epoch_day_to_iso(day: int) -> str:
return (dt.date(1970, 1, 1) + dt.timedelta(days=int(day))).isoformat()
def _year_of_day(day: int) -> str:
return str((dt.date(1970, 1, 1) + dt.timedelta(days=int(day))).year)
# --- 1. the VRP return formulas ----------------------------------------------------------------
def short_var_return(*, dvol_prev: float, ret: float, k: float = 1.0,
direction: str = "short_vol") -> float:
"""The per-asset, per-day SINGLE-DAY short-variance increment:
short_var_ret = k * [ (dvol_prev/√365/100)² ret² ]
`dvol_prev` is the IV (DVOL index, vol points) known at the START of the day; `ret` is the close-to-close
return realized OVER the day. POSITIVE when calm (implied daily variance > realized), big-NEGATIVE on a
vol spike. `direction="long_vol"` negates (buy variance). Pure.
NOTE: this single-day proxy is the OLD noisy construction — kept only as the per-day accrual building
block / the look-ahead audit's negative control. The book series now uses the rolling variance swap
(`variance_swap_payoff` + the ladder in `_book_short_var`)."""
if direction not in _DIRECTIONS:
raise ValueError(f"direction must be one of {_DIRECTIONS}, got {direction!r}")
implied_daily_var = (float(dvol_prev) / _SQRT_YEAR / 100.0) ** 2
realized_daily_var = float(ret) ** 2
sign = 1.0 if direction == "short_vol" else -1.0
return float(k) * sign * (implied_daily_var - realized_daily_var)
def variance_swap_payoff(*, dvol_entry: float, fwd_rets: list[float], swap_days: int = _DEFAULT_SWAP_DAYS,
k: float = 1.0, direction: str = "short_vol") -> float:
"""The textbook SHORT variance-swap payoff (per unit variance notional) over one swap's life:
payoff = k * sign * [ K RV² ]
K = (dvol_entry/100)² (annualised IMPLIED variance, strike)
RV² = (365/swap_days) · Σ_{i=1..swap_days} fwd_rets[i]² (annualised REALIZED variance, the window)
`dvol_entry` is the DVOL (vol points, annualised) known at swap ENTRY (causal). `fwd_rets` is the list of
the `swap_days` daily close-to-close returns realized OVER the swap's life (forward of entry); fewer than
`swap_days` returns annualises by however many are present (a partial/truncated final swap). POSITIVE when
implied>realized (the premium), big-NEGATIVE on a vol-spike window (the tail). `direction="long_vol"`
negates (buy variance). Pure."""
if direction not in _DIRECTIONS:
raise ValueError(f"direction must be one of {_DIRECTIONS}, got {direction!r}")
n = len(fwd_rets)
if n == 0:
return 0.0
strike = (float(dvol_entry) / 100.0) ** 2
realized = (_PERIODS_PER_YEAR / n) * sum(float(r) ** 2 for r in fwd_rets)
sign = 1.0 if direction == "short_vol" else -1.0
return float(k) * sign * (strike - realized)
# --- 2. the BTC+ETH equal-weight raw book + vol-target sizing ----------------------------------
def _daily_close_returns(cser: dict[int, float]) -> dict[int, float]:
"""`{day: close-to-close ret}` for the days with a finite, non-zero prior close. Pure."""
days = sorted(cser)
out: dict[int, float] = {}
for i in range(1, len(days)):
d0, d1 = days[i - 1], days[i]
p0, p1 = cser[d0], cser[d1]
if p0 and math.isfinite(p0) and math.isfinite(p1):
out[int(d1)] = p1 / p0 - 1.0
return out
def _asset_swap_ladder(cser: dict[int, float], dser: dict[int, float], *, direction: str,
swap_days: int, leak: bool = False) -> dict[int, float]:
"""The rolling variance-swap LADDER for ONE asset → `{day: daily_swap_pnl}` (UNSIZED, per unit notional).
For every entry day s with a known DVOL_s (causal), open a short-variance swap whose strike is
K_s=(DVOL_s/100)² and whose realized window is the next `swap_days` daily returns (forward of s). The
swap's window payoff `K_s RV²_s` is SPREAD over the swap's life (÷ its realized length) and accrued to
each day it covers; the day-t book P&L is the AVERAGE of the per-day accruals of all swaps in flight on t
(a smooth `swap_days`-window-averaged quantity, NOT a single raw ret²). `leak=True` is the negative
control: it dates the strike by the END of the realized window (DVOL_{s+swap_days}, knowable only at
maturity) — a look-ahead the audit must catch. Pure."""
rets = _daily_close_returns(cser)
if not rets:
return {}
ret_days = sorted(rets)
pnl_sum: dict[int, float] = {}
pnl_cnt: dict[int, int] = {}
for idx, s in enumerate(ret_days[:-1] if len(ret_days) > 1 else ret_days):
# forward window of up to swap_days realized daily returns AFTER entry day s
window_days = ret_days[idx + 1: idx + 1 + swap_days]
if not window_days:
continue
fwd = [rets[d] for d in window_days]
strike_day = window_days[-1] if leak else s # leak: DVOL dated at maturity (look-ahead)
dvol_entry = dser.get(strike_day)
if dvol_entry is None:
continue
payoff = variance_swap_payoff(dvol_entry=dvol_entry, fwd_rets=fwd,
swap_days=swap_days, direction=direction)
per_day = payoff / len(fwd) # spread the window payoff over the swap's life
for d in window_days: # accrue to each day the swap covers
pnl_sum[d] = pnl_sum.get(d, 0.0) + per_day
pnl_cnt[d] = pnl_cnt.get(d, 0) + 1
return {d: pnl_sum[d] / pnl_cnt[d] for d in pnl_sum} # average over in-flight swaps that day
def _book_short_var(store: _FeatureStore, *, direction: str,
swap_days: int = _DEFAULT_SWAP_DAYS, _leak: bool = False,
) -> dict[int, float]:
"""`{day: raw_short_var_ret}` — the BTC+ETH EQUAL-WEIGHT (mean over the assets present that day) raw,
UNSIZED (k=1) ROLLING VARIANCE-SWAP daily return (the `_asset_swap_ladder` per asset). CAUSAL: each swap's
strike DVOL is known at entry and its realized window is strictly FORWARD of entry. READ-ONLY. Pure off
panels. `_leak=True` builds the look-ahead negative control. Empty when no overlapping dvol+close data."""
close = store.read_panel(list(_VRP_SYMBOLS), "close")
dvol = store.read_panel(list(_VRP_SYMBOLS), "dvol")
close = {s: v for s, v in close.items() if v}
dvol = {s: v for s, v in dvol.items() if v}
if not close or not dvol:
return {}
per_day: dict[int, list[float]] = {}
for sym in _VRP_SYMBOLS:
cser = close.get(sym, {})
dser = dvol.get(sym, {})
if not cser or not dser:
continue
ladder = _asset_swap_ladder(cser, dser, direction=direction, swap_days=swap_days, leak=_leak)
for d, v in ladder.items():
if math.isfinite(v):
per_day.setdefault(int(d), []).append(v)
return {d: st.mean(vals) for d, vals in per_day.items() if vals}
def _vol_target_k(raw: dict[int, float], *, target_ann_vol: float,
day_lo: int | None = None, day_hi: int | None = None) -> float:
"""The sizing constant `k` so the annualised vol of `k·raw` over [day_lo, day_hi] == `target_ann_vol`.
`k = target_ann_vol / (pstdev(raw) · √365)`; 0 when the raw series has <2 obs or zero vol. Pure."""
rets = [raw[d] for d in sorted(raw)
if (day_lo is None or d >= day_lo) and (day_hi is None or d <= day_hi)]
if len(rets) < 2:
return 0.0
sd = st.pstdev(rets)
if sd <= 0.0:
return 0.0
return float(target_ann_vol) / (sd * _SQRT_YEAR)
def _sized_series(raw: dict[int, float], *, k: float, cost_bps: float) -> dict[int, float]:
"""`{day: net_ret}` = k·raw per-day cost. A short-straddle is re-established/re-hedged daily, so the
per-day cost ≈ leverage(k) × cost_bps (a constant daily haircut on the sized notional). Pure."""
haircut = k * (cost_bps / 1e4) if cost_bps else 0.0
out: dict[int, float] = {}
for d in sorted(raw):
v = k * raw[d] - haircut
if math.isfinite(v):
out[int(d)] = float(v)
return out
def vrp_returns_from_store(store: _FeatureStore, *, cost_bps: float = 5.5,
direction: str = "short_vol",
target_ann_vol: float = _DEFAULT_TARGET_VOL) -> dict[int, float]:
"""READ-ONLY per-day VRP RETURN SERIES `{epoch_day: net_ret}` — the book-level seam an N-th sleeve would
consume. BTC+ETH equal-weight short-variance (or long-vol negation), vol-TARGETED to `target_ann_vol`
(full-sample k), net of the per-day cost. Writes NOTHING."""
raw = _book_short_var(store, direction=direction)
if len(raw) < 2:
return {}
k = _vol_target_k(raw, target_ann_vol=target_ann_vol)
return _sized_series(raw, k=k, cost_bps=cost_bps)
def _metrics_from_series(series: dict[int, float], *, k: float = 0.0) -> dict[str, Any]:
"""`_curve_metrics` over a `{day: ret}` series, PLUS the tail diagnostics: `worst_day` (the single most
negative day — the vol-spike loss), `sum_ret`, and `avg_leverage` (= k, the vol-target exposure)."""
daily = [(d, series[d]) for d in sorted(series)]
meta = {"symbols": len(_VRP_SYMBOLS),
"first_day": _epoch_day_to_iso(daily[0][0]) if daily else None,
"last_day": _epoch_day_to_iso(daily[-1][0]) if daily else None}
m = _curve_metrics(daily, days_meta=meta)
m["sum_ret"] = sum(series.values())
m["worst_day"] = min(series.values()) if series else 0.0
m["avg_leverage"] = float(k)
return m
def vrp_metrics_from_store(store: _FeatureStore, *, cost_bps: float = 5.5,
direction: str = "short_vol",
target_ann_vol: float = _DEFAULT_TARGET_VOL) -> dict[str, Any]:
"""READ-ONLY VRP metrics (the standard `_curve_metrics` shape + tail diagnostics `worst_day`/`max_dd` +
`avg_leverage`). `available=False` (+ `reason`) when there is no overlapping dvol+close data / too few
days. READ-ONLY."""
raw = _book_short_var(store, direction=direction)
if len(raw) < 2:
return {"cagr": 0.0, "sharpe": 0.0, "max_dd": 0.0, "total_return": 0.0, "ann_return": 0.0,
"days": 0, "symbols": 0, "first_day": None, "last_day": None,
"worst_day": 0.0, "avg_leverage": 0.0, "available": False,
"reason": "VRP curve has <2 days (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)"}
k = _vol_target_k(raw, target_ann_vol=target_ann_vol)
series = _sized_series(raw, k=k, cost_bps=cost_bps)
m = _metrics_from_series(series, k=k)
m["available"] = True
return m
# --- 3. correlation vs the existing edges ------------------------------------------------------
def _correlation(a_series: dict[int, float], b_series: dict[int, float]) -> float | None:
"""Pearson correlation over the two series' COMMON days; None if <2 common days or zero variance. Pure."""
common = sorted(set(a_series) & set(b_series))
if len(common) < 2:
return None
a = [a_series[d] for d in common]
b = [b_series[d] for d in common]
sa, sb = st.pstdev(a), st.pstdev(b)
if sa <= 0.0 or sb <= 0.0:
return None
ma, mb = st.mean(a), st.mean(b)
cov = sum((x - ma) * (y - mb) for x, y in zip(a, b, strict=True)) / len(common)
return cov / (sa * sb)
def _existing_edge_series(store: _FeatureStore, *, universe: set[str] | None, cost_bps: float,
unlock_events: list[dict[str, Any]] | None,
) -> dict[str, dict[int, float]]:
"""The per-day return series for each EXISTING edge (tstrend/unlock/xsfunding/positioning) via the live
sleeve runners. unlock needs the injected calendar; absent → empty. READ-ONLY."""
out: dict[str, dict[int, float]] = {}
for edge in _EXISTING_EDGES:
out[edge] = sleeve_returns_from_store(
store, edge, universe=universe, cost_bps=cost_bps, unlock_events=unlock_events)
return out
# --- 4. adversarial verification ---------------------------------------------------------------
def verify_vrp_edge(store: _FeatureStore, *, cost_bps: float = 5.5,
cost_grid: tuple[float, ...] = (5.5, 11.0, 22.0, 50.0, 100.0),
direction: str = "short_vol",
target_ann_vol: float = _DEFAULT_TARGET_VOL,
universe: set[str] | None = None,
unlock_events: list[dict[str, Any]] | None = None) -> dict[str, Any]:
"""READ-ONLY adversarial verification of the VRP edge in BOTH directions (short_vol / long_vol).
Per direction:
* `net_cost[dir][bps]` — `_curve_metrics` (+ `sum_ret`/`worst_day`/`avg_leverage`) net of cost at EACH
bps in `cost_grid`. The grid spans the cheap perp-style 5.5/11/22 AND the REALISTIC OPTION costs
50/100 bp (selling options isn't 5.5bp), so the cost sensitivity is honest. Net is RE-RUN per cost.
* `worst_day[dir]` — the single most-negative day (the vol-spike tail loss);
* `max_dd[dir]` — the maxDD at the headline cost (the tail);
* `per_year[dir]` — `{YEAR: {sharpe, sum_ret, max_dd, days}}` net at `cost_bps` (all-one-year ⇒ overfit);
* `corr_edges[dir]` — `{tstrend, unlock, xsfunding, positioning: corr|None}` of the (net @ `cost_bps`)
series with EACH existing edge (the additive/uncorrelated check vs the whole 4-edge book);
* `avg_leverage[dir]` — the vol-target exposure constant k.
The k (vol-target) is the FULL-SAMPLE constant (the walk-forward re-sizes on TRAIN). `available=False`
(+ reason) when there is no data. READ-ONLY."""
grid = tuple(sorted(set(cost_grid) | {cost_bps}))
edge_series = _existing_edge_series(store, universe=universe, cost_bps=cost_bps,
unlock_events=unlock_events)
net_cost: dict[str, dict[float, dict[str, Any]]] = {}
per_year: dict[str, dict[str, dict[str, Any]]] = {}
worst_day: dict[str, float] = {}
max_dd: dict[str, float] = {}
avg_leverage: dict[str, float] = {}
corr_edges: dict[str, dict[str, float | None]] = {}
n_days = 0
for d in _DIRECTIONS:
raw = _book_short_var(store, direction=d)
if len(raw) < 2:
continue
k = _vol_target_k(raw, target_ann_vol=target_ann_vol)
avg_leverage[d] = k
net_cost[d] = {}
net_at_headline: dict[int, float] = {}
for bps in grid:
ser = _sized_series(raw, k=k, cost_bps=bps)
net_cost[d][bps] = _metrics_from_series(ser, k=k)
if bps == cost_bps:
net_at_headline = ser
if not net_at_headline:
net_at_headline = _sized_series(raw, k=k, cost_bps=cost_bps)
n_days = max(n_days, len(net_at_headline))
worst_day[d] = min(net_at_headline.values()) if net_at_headline else 0.0
max_dd[d] = _metrics_from_series(net_at_headline, k=k)["max_dd"]
by_year: dict[str, dict[int, float]] = {}
for day, r in net_at_headline.items():
by_year.setdefault(_year_of_day(day), {})[day] = r
per_year[d] = {}
for yr, ser in sorted(by_year.items()):
ym = _metrics_from_series(ser, k=k)
per_year[d][yr] = {"sharpe": ym["sharpe"], "sum_ret": ym["sum_ret"],
"max_dd": ym["max_dd"], "days": ym["days"]}
corr_edges[d] = {edge: _correlation(net_at_headline, edge_series.get(edge, {}))
for edge in _EXISTING_EDGES}
if n_days < 2:
return {"available": False,
"reason": "VRP curve has <2 days (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)"}
return {"available": True, "cost_bps": cost_bps, "cost_grid": list(grid),
"net_cost": net_cost, "per_year": per_year, "worst_day": worst_day, "max_dd": max_dd,
"avg_leverage": avg_leverage, "corr_edges": corr_edges,
"cross_venue_note": "SIGNAL = Deribit DVOL; EXECUTION would be short Bybit option straddles"}
# --- 5. look-ahead audit -----------------------------------------------------------------------
def _book_short_var_leaked(store: _FeatureStore, *, direction: str) -> dict[int, float]:
"""The NEGATIVE-CONTROL book: dates each swap's STRIKE DVOL at the END of its realized window (knowable
only at maturity, not at entry) — a look-ahead leak. Used ONLY by the audit to prove it bites. Pure."""
return _book_short_var(store, direction=direction, _leak=True)
def look_ahead_audit_vrp(store: _FeatureStore, *, direction: str = "short_vol",
_leak: bool = False) -> dict[str, Any]:
"""Assert the variance-swap strike→realized-window mapping is STRICTLY CAUSAL: a swap's strike DVOL must
be the value known at ENTRY, never one dated at the end of the realized window (knowable only at maturity).
Builds two books from the SAME days: the CAUSAL book (strike = DVOL at entry — the live mapping) and a
LEAKED book (strike = DVOL at the window END — a look-ahead negative control). The candidate is the causal
book unless `_leak=True` forces the leaked one. `leak_days` counts days where the candidate's return
equals the leaked book's — the live causal mapping uses the entry-day DVOL, which differs from the
window-end DVOL whenever DVOL varies, so a clean (causal) candidate is DISTINCT from the leaked book on
EVERY overlapping day, while a leaked candidate is IDENTICAL to it on every day. `leak_days` counts the
days the candidate equals the leaked book; `causal` iff `leak_days == 0` (no day silently uses the
look-ahead strike). A leaked candidate matches on all days → not causal. READ-ONLY."""
causal = _book_short_var(store, direction=direction)
leaked = _book_short_var_leaked(store, direction=direction)
candidate = leaked if _leak else causal
common = sorted(set(candidate) & set(leaked))
leak_days = sum(1 for d in common if abs(candidate[d] - leaked[d]) <= 1e-15)
return {"causal": bool(common) and leak_days == 0,
"leak_days": leak_days, "n_days_compared": len(common)}
# --- 6. OOS / WALK-FORWARD validation (the TAIL-AWARE DEPLOY GATE) ------------------------------
def _sharpe(series: dict[int, float]) -> float:
rets = [series[d] for d in sorted(series)]
if len(rets) < 2:
return 0.0
sd = st.pstdev(rets)
return (st.mean(rets) / sd * _SQRT_YEAR) if sd > 0 else 0.0
def _max_dd(series: dict[int, float]) -> float:
equity, peak, mdd = 1.0, 1.0, 0.0
for d in sorted(series):
equity *= (1.0 + series[d])
peak = max(peak, equity)
if peak > 0:
mdd = min(mdd, equity / peak - 1.0)
return mdd
def _rolling_windows(net: dict[int, float], *, window_days: int) -> dict[str, Any]:
"""Consecutive non-overlapping `window_days` windows; per window Sharpe + total + maxDD + days, and the
fraction of windows with positive total return. A real edge is positive in MOST windows, not one regime."""
if not net:
return {"n_windows": 0, "fraction_positive": 0.0, "windows": []}
days = sorted(net)
lo, hi = days[0], days[-1]
windows: list[dict[str, Any]] = []
start = lo
while start <= hi:
end = start + window_days - 1
ser = {d: net[d] for d in days if start <= d <= end}
if len(ser) >= 2:
total = math.prod(1.0 + ser[d] for d in sorted(ser)) - 1.0
windows.append({"start": _epoch_day_to_iso(start), "end": _epoch_day_to_iso(min(end, hi)),
"sharpe": _sharpe(ser), "total_return": total, "max_dd": _max_dd(ser),
"days": len(ser)})
start = end + 1
n = len(windows)
frac = (sum(1 for w in windows if w["total_return"] > 0.0) / n) if n else 0.0
return {"n_windows": n, "fraction_positive": frac, "windows": windows}
def _train_holdout(store: _FeatureStore, *, cost_bps: float, target_ann_vol: float,
train_frac: float = 0.6) -> dict[str, Any]:
"""The KEY OOS check, TAIL-AWARE. Split time into TRAIN (first `train_frac`) and TEST (the rest). On
TRAIN: pick the winning DIRECTION (short_vol vs long_vol by TRAIN Sharpe) AND size k on TRAIN ONLY (so
the vol-target scale is not fit on the held-out data). Apply THAT direction + THAT k to TEST. Report the
chosen direction, TRAIN Sharpe, TEST (OOS) Sharpe, and the TEST maxDD (the OOS tail)."""
raws = {d: _book_short_var(store, direction=d) for d in _DIRECTIONS}
days = sorted(set().union(*[set(r) for r in raws.values()]))
if len(days) < 4:
return {"train_direction": None, "train_sharpe": 0.0, "test_sharpe": 0.0,
"test_max_dd": 0.0, "train_days": 0, "test_days": 0, "available": False}
split_idx = max(1, min(len(days) - 1, int(len(days) * train_frac)))
train_hi = days[split_idx - 1]
test_lo = days[split_idx]
best_dir, best_sharpe, best_k = "short_vol", -math.inf, 0.0
for d in _DIRECTIONS:
raw = raws[d]
k = _vol_target_k(raw, target_ann_vol=target_ann_vol, day_hi=train_hi)
tr = _sized_series({day: raw[day] for day in raw if day <= train_hi}, k=k, cost_bps=cost_bps)
s = _sharpe(tr)
if s > best_sharpe:
best_dir, best_sharpe, best_k = d, s, k
raw = raws[best_dir]
test = _sized_series({day: raw[day] for day in raw if day >= test_lo}, k=best_k, cost_bps=cost_bps)
return {"train_direction": best_dir, "train_sharpe": best_sharpe, "test_sharpe": _sharpe(test),
"test_max_dd": _max_dd(test), "test_total_return": (math.prod(
1.0 + test[d] for d in sorted(test)) - 1.0) if test else 0.0,
"train_days": split_idx, "test_days": len(days) - split_idx,
"split_at": _epoch_day_to_iso(test_lo), "available": True}
def walk_forward_vrp(store: _FeatureStore, *, cost_bps: float = 5.5, window_days: int = 180,
train_frac: float = 0.6, target_ann_vol: float = _DEFAULT_TARGET_VOL,
max_dd_floor: float = -0.40, universe: set[str] | None = None) -> dict[str, Any]:
"""READ-ONLY, TAIL-AWARE OOS / WALK-FORWARD validation of the VRP edge — the DEPLOY GATE.
Computes (all off the same return + `_curve_metrics`-style accounting as the in-sample verify):
1. `train_holdout` — TRAIN picks the DIRECTION and sizes k (TRAIN-only); applied to the never-looked-at
TEST. Reports `train_sharpe`, `test_sharpe` (OOS), `test_max_dd` (the OOS tail);
2. `rolling_windows` — consecutive non-overlapping windows (per-window Sharpe/total/maxDD) +
`fraction_positive` (a real edge is positive in MOST windows, not one calm regime);
3. `look_ahead` — the strict-causality audit;
4. `full` — the full-sample metrics at the TRAIN-chosen direction (for the headline + the full maxDD);
5. `verdict` — a TAIL-AWARE PASS/FAIL. Deployable ONLY if:
(a) OOS (TEST) Sharpe > 0 at the (realistic-option) cost,
(b) positive in >1 rolling window (fraction_positive > 0.5) AND >70% positive,
(c) the TAIL is SURVIVABLE: full-sample maxDD AND OOS-test maxDD are both above `max_dd_floor`
(default 40%) — VRP looks great until a spike, so a catastrophic DD fails even if Sharpe>0,
(d) look-ahead clean.
A clean "no / too-tail-heavy" is a VALID outcome (FAIL). READ-ONLY; `available=False` (+ reason) when
there is no dvol+close data / too few days."""
raw_probe = _book_short_var(store, direction="short_vol")
if len(raw_probe) < 4:
return {"available": False,
"reason": "VRP curve has <4 usable days for a walk-forward split "
"(run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)"}
train_holdout = _train_holdout(store, cost_bps=cost_bps, target_ann_vol=target_ann_vol,
train_frac=train_frac)
direction = train_holdout["train_direction"] or "short_vol"
raw = _book_short_var(store, direction=direction)
k_full = _vol_target_k(raw, target_ann_vol=target_ann_vol)
full_net = _sized_series(raw, k=k_full, cost_bps=cost_bps)
full = _metrics_from_series(full_net, k=k_full)
rolling = _rolling_windows(full_net, window_days=window_days)
audit = look_ahead_audit_vrp(store, direction=direction)
# --- TAIL-AWARE PASS/FAIL verdict ---------------------------------------------------------
oos_pos = train_holdout["available"] and train_holdout["test_sharpe"] > 0.0
rw_pos = rolling["fraction_positive"] > 0.70
tail_survivable = (full["max_dd"] > max_dd_floor
and train_holdout.get("test_max_dd", 0.0) > max_dd_floor)
causal = audit["causal"]
checks = {
"oos_direction_positive": oos_pos,
"rolling_windows_positive": rw_pos,
"tail_survivable": tail_survivable,
"look_ahead_clean": causal,
}
deployable = all(checks.values())
return {"available": True, "cost_bps": cost_bps, "window_days": window_days, "direction": direction,
"target_ann_vol": target_ann_vol, "max_dd_floor": max_dd_floor,
"avg_leverage": k_full, "full": full, "rolling_windows": rolling,
"train_holdout": train_holdout, "look_ahead": audit,
"verdict": {"deployable": deployable, "checks": checks},
"cross_venue_note": "SIGNAL = Deribit DVOL; EXECUTION would be short Bybit option straddles"}

View File

@@ -1,331 +0,0 @@
"""Executed VRP twin (sub-project C-style): place the atomic XSP put-credit-spread on IBKR and record the
executed-vs-modeled reconciliation row. SUSPENDED until the account has options permission (no live OPRA in
CI either, so the runtime path is not testable here).
Maintains a LADDER of open rungs in the `vrp_exec_pos` book-state (mirrors `VrpStrategy.advance`'s ladder
discipline in `vrp_book.py`, not a single overwritten position): each run marks every open rung against
today's frozen slice, closes any rung that hit 50% profit-take or DTE<=1 (placing the reverse combo), and
opens AT MOST ONE new rung — sized at `envelope / ladder`, never the full envelope — when there is a free
ladder slot on the weekly entry day. The 50%-profit-take DECISION uses the SAME smooth `vrp_marking` model
value the backtest uses (single source) — not the raw shortlong leg difference — so a noisy far-OTM long-leg
quote cannot spuriously trip a live close; realized P&L and close order limits stay on the actual fill-basis
mark. This bounds total open notional at ~envelope regardless of how many
rungs are simultaneously open; armed weekly, it can never stack N spreads at full size.
`build_combo_legs`/`build_close_legs`, `refuse_if_live`, `plan_vrp_ladder`, `rung_notional`,
`compute_spread_exec_return`/`compute_ladder_exec_return` and `build_pos_state` are pure and unit-tested.
`plan_and_record_vrp` mirrors `multistrat_exec_record.plan_and_record`'s guard structure but sizes a LADDER
of defined-risk spreads (not a continuous weight vector) — there is no B2a allocation for VRP yet, so new
entries only happen when `paper_envelope > 0`; existing rungs still mark/close on a de-funded envelope
(they cannot be silently stranded), matching the convention elsewhere that a de-funded envelope stops new
risk but does not orphan positions."""
from __future__ import annotations
import datetime as dt
from typing import Any
def build_combo_legs(*, short_osi: str, long_osi: str, short_price: float, long_price: float,
qty: int) -> tuple[list[tuple[str, str, int]], float]:
"""Defined-risk put-credit-spread OPEN: SELL the short put, BUY the wing. Net limit is negative (a
credit) — see `IbkrBroker.place_combo` for the parent-order sign convention this feeds."""
legs = [(short_osi, "SELL", qty), (long_osi, "BUY", qty)]
net = -(short_price - long_price) # credit received -> negative limit
return legs, round(net, 2)
def build_close_legs(*, short_osi: str, long_osi: str, qty: int,
mark_today: float) -> tuple[list[tuple[str, str, int]], float]:
"""Reverse of `build_combo_legs` to FLATTEN a held rung: BUY back the short (was sold), SELL the long
(was bought). `net_limit` is POSITIVE — the debit paid to close, equal to `mark_today` (`_mark_spread`'s
short-minus-long convention)."""
legs = [(short_osi, "BUY", qty), (long_osi, "SELL", qty)]
return legs, round(float(mark_today), 2)
def refuse_if_live(paper_envelope: float, account_is_paper: bool) -> None:
"""The real-capital exposure guard, factored out pure so it is testable without a live broker/repo: a
PAPER-validation envelope (`paper_envelope > 0`) NEVER runs against a non-paper account. Must be the
FIRST thing `plan_and_record_vrp` does — before any repo/broker/OPRA call — so a refusal places (and
reads) exactly nothing."""
if paper_envelope > 0.0 and not account_is_paper:
raise ValueError("paper-envelope refused: account is not paper (real-capital exposure guard)")
def _mark_spread(bars: Any, spread: dict[str, Any], day: str) -> float | None:
"""ACTUAL cost to close `spread` on `day` from the frozen slice (short - long), or None if either leg has
no mark today (aged/moved out of the frozen band). This is the real fill-basis mark — used for the
executed-vs-modeled realized P&L and as the close order's net limit (the debit truly paid). The
profit-take DECISION does NOT use this (it uses the smooth `vrp_marking` model mark) — a noisy far-OTM
long-leg quote must not be able to spuriously trip a live close."""
marks = bars.bars_for([spread["short_osi"], spread["long_osi"]], day, day)
short_val = marks.get(spread["short_osi"], {}).get(day)
long_val = marks.get(spread["long_osi"], {}).get(day)
if short_val is None or long_val is None:
return None
return short_val - long_val
def rung_notional(envelope: float, ladder: int) -> float:
"""The per-rung dollar target: the envelope split evenly across the ladder's slots. Sizing every new
entry at this (instead of the full envelope) bounds total open notional at ~envelope regardless of how
many rungs are simultaneously open."""
return envelope / ladder if ladder > 0 else 0.0
def plan_vrp_ladder(open_spreads: list[dict[str, Any]], exec_marks: dict[str, float | None],
signal_marks: dict[str, float | None], today: str, *,
ladder: int, profit_take: float = 0.5, entry_weekday: int = 0
) -> tuple[list[tuple[dict[str, Any], float]], bool, int]:
"""Pure ladder-discipline decision (mirrors `VrpStrategy.advance`'s close-then-open management, for the
single-day-per-run exec cadence). Both mark dicts are pre-fetched by the caller (keyed by `short_osi`) so
this stays I/O-free:
- `exec_marks` — the ACTUAL shortlong fill-basis mark (`_mark_spread`); the close order's net limit
and the HOLD gate (a rung with no actual price cannot be closed).
- `signal_marks` — the SMOOTH `vrp_marking` model value (same valuation the backtest uses); the ONLY
input to the 50%-profit-take decision, so a noisy far-OTM long-leg quote can't trip
a spurious close.
Returns:
to_close — [(spread, exec_mark)] rungs to close THIS run: 50%-profit-take
(signal <= (1-profit_take)*credit) or DTE<=1. A rung with NO actual mark today (aged out of
the frozen band) is a HOLD, not a close — it cannot be closed without a price (mirrors
`compute_spread_exec_return`'s None-not-phantom-zero convention). The paired value is the
ACTUAL mark (the real debit to close), NOT the model signal.
may_open — True iff today is the weekly entry weekday AND the ladder has a free slot once this run's
closes are applied (len(open_spreads) - len(to_close) < ladder).
open_slots — 0 or 1 (a once-daily run opens at most one new rung, mirroring `VrpStrategy.advance`)."""
to_close: list[tuple[dict[str, Any], float]] = []
for sp in open_spreads:
exec_mark = exec_marks.get(sp["short_osi"])
if exec_mark is None:
continue # no actual price -> cannot close (HOLD)
expiry = dt.date.fromisoformat(sp["expiry"])
dte = (expiry - dt.date.fromisoformat(today)).days
signal = signal_marks.get(sp["short_osi"])
# INTENTIONAL ref-vs-live ASYMMETRY (documented, not a bug): on a degenerate slice where this rung's
# own put legs price (so `exec_mark` is a REAL actual close mark) but no ATM call/put pair prices at any
# expiry (so `signal` / `vrp_marking.model_spread_value` == None), the pure-model backtest
# (`vrp_book.advance`) DROPS the rung — its only mark IS the model value. The live twin instead HOLDS:
# `take_profit` is False when `signal is None` (never profit-take on a missing model signal — a noisy or
# absent model must NOT trip a live close), so the rung stays managed on its real fill mark until DTE<=1
# or the model recovers. Correct: the twin has strictly MORE information than the recompute (a real
# actual mark), so it need not discard a live position the way the info-poorer backtest must.
take_profit = signal is not None and signal <= (1.0 - profit_take) * sp["credit"]
if take_profit or dte <= 1:
to_close.append((sp, exec_mark)) # close at the ACTUAL mark, decide on the model signal
remaining = len(open_spreads) - len(to_close)
is_entry_day = dt.date.fromisoformat(today).weekday() == entry_weekday
may_open = is_entry_day and remaining < ladder
return to_close, may_open, (1 if may_open else 0)
def compute_spread_exec_return(prior_val: float, mark_today: float | None, qty: int,
envelope: float) -> float | None:
"""Envelope-basis executed return of ONE rung since its last mark: the mark-to-market P&L, as a fraction
of the shared ladder envelope. Short-vol: value falling = profit. The $100/contract multiplier converts
the frozen per-share option marks into real dollars. None on a de-funded envelope or when today's freeze
is missing a mark for the held legs (a HOLD, not a phantom zero)."""
if envelope <= 0.0 or mark_today is None:
return None
pnl = (prior_val - mark_today) * 100.0 * qty
return pnl / envelope
def compute_ladder_exec_return(open_spreads_prior: list[dict[str, Any]], marks_today: dict[str, float | None],
envelope_prior: float) -> float | None:
"""Sum `compute_spread_exec_return` across every rung held since the last run (mirrors
`multistrat_exec_record.compute_exec_return`'s book-level pattern, one rung at a time). None on
inception (no prior rungs), a de-funded prior envelope, or when NOT ONE rung produced a mark today (an
all-HOLD run — not a phantom zero); a rung with no mark today simply contributes 0 to the sum, same as
`VrpStrategy.advance`'s per-day loop skipping unmarked spreads."""
if not open_spreads_prior or envelope_prior <= 0.0:
return None
total = 0.0
any_marked = False
for sp in open_spreads_prior:
mark = marks_today.get(sp["short_osi"])
r = compute_spread_exec_return(float(sp.get("last_val", sp["credit"])), mark, int(sp.get("qty", 0)),
envelope_prior)
if r is not None:
total += r
any_marked = True
return total if any_marked else None
def build_pos_state(open_spreads: list[dict[str, Any]], envelope: float, venue: str,
today: str) -> dict[str, Any]:
return {"open_spreads": [dict(sp) for sp in open_spreads], "envelope": float(envelope), "venue": str(venue),
"last_run_date": today}
class _VrpExecBookingStrategy:
"""A trivial observed ForwardStrategy for run_track: books today's pre-computed executed return (or
nothing on inception/no-mark). Mirrors `multistrat_exec_record._ExecBookingStrategy`."""
def __init__(self, exec_return: float | None) -> None:
self._r = exec_return
def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]:
if self._r is None:
return [], extra
return [(str(extra.get("_today")), float(self._r))], extra
def _record(repo: Any, exec_return: float | None, *, open_spreads: list[dict[str, Any]], envelope: float,
venue: str, today: str, at: dt.datetime) -> None:
from fxhnt.application.forward_engine import run_track
run_track(repo, "vrp_exec", _VrpExecBookingStrategy(exec_return), today=today, at=at)
repo.set_book_state("vrp_exec_pos", build_pos_state(open_spreads, envelope, venue, today), at=at)
def plan_and_record_vrp(repo: Any, *, dbn: Any, broker: Any, nlv: float, today: str, at: dt.datetime,
paper_envelope: float, account_is_paper: bool, venue: str, execute: bool) -> dict[str, Any]:
"""Mark every PRIOR held rung (if any) against today's frozen slice, apply the ladder discipline (close
rungs at 50% profit / DTE<=1, cap entries at `ladder`, size a new rung at `envelope/ladder`), and — when
`execute` — place the closing/opening combos and record the observed `vrp_exec` return.
Guards (mirrors `multistrat_exec_record.plan_and_record` guard-for-guard):
1. `refuse_if_live` — paper-envelope on a non-paper account raises BEFORE any repo/broker/OPRA call
(zero orders placed, zero rows read/written).
2. per-day idempotency — a `vrp_exec_pos` book-state already stamped `today` is a no-op re-run.
3. a `broker.place_combo` failure (open OR close) becomes a structured entry in `errors` (never a
crash); a failed CLOSE re-adds the rung UNCHANGED to the ladder (mirrors the bybit per-order gap-row
+ kill-switch discipline: a failed order must not silently advance the position basis). The OPEN is
gated on the ACTUAL post-close ladder occupancy (not the pre-execution `may_open` plan), so a failed
close that leaves the ladder full skips the new entry instead of breaching the cap.
4. recording is skipped entirely when `execute` is False (dry-run plans only, same as multistrat)."""
refuse_if_live(paper_envelope, account_is_paper)
pos_prior = repo.get_book_state("vrp_exec_pos")
if pos_prior.get("last_run_date") == today:
return {"noop": True, "reason": f"already ran on {today}", "exec_return": None, "placed": 0,
"closed": 0, "spread": None}
open_spreads_prior: list[dict[str, Any]] = list(pos_prior.get("open_spreads", []))
envelope_prior = float(pos_prior.get("envelope", 0.0))
envelope = min(float(paper_envelope), float(nlv)) if paper_envelope > 0.0 else 0.0
from fxhnt.adapters.orchestration.assets import _freeze_latest_session
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
from fxhnt.config import get_settings
dsn = getattr(repo, "_dsn", None) or get_settings().operational_dsn
bars = XspOptionBarsRepo(dsn)
bars.migrate()
# TWO dates, deliberately distinct: OPRA publishes on a ~1-session lag, so `today`'s slice is NEVER
# available intraday — freezing `today` crashes a live run with DatabentoRangeUnavailable before it ever
# reaches the broker. Freeze/mark against the last COMPLETE OPRA session (`freeze_date`, T-1, via the same
# robust bounded step-back the nightly `vrp_nav` uses); use `today` (calendar) for all date-LOGIC — DTE,
# entry-weekday, the idempotency key, `run_track` booking and the pos-state stamp. Only the DATA date lags.
freeze_date = _freeze_latest_session(dbn, bars, at=at) # last complete OPRA session (data lag), ISO date
from fxhnt.application.vrp_book import VrpStrategy
from fxhnt.registry import STRATEGY_REGISTRY
ladder = int(STRATEGY_REGISTRY["vrp"]["definition"]["params"]["ladder"])
strat = VrpStrategy(bars, ladder=ladder)
from fxhnt.application import vrp_marking
# ACTUAL fill-basis marks (shortlong) for the realized exec P&L and the close order net limits — read at
# the frozen data session, not `today` (whose slice OPRA hasn't published yet -> would come back None).
marks_today = {sp["short_osi"]: _mark_spread(bars, sp, freeze_date) for sp in open_spreads_prior}
exec_return = compute_ladder_exec_return(open_spreads_prior, marks_today, envelope_prior)
# SMOOTH model marks (the SAME `vrp_marking` valuation the backtest uses) drive the 50%-profit-take
# decision, so a noisy far-OTM long-leg quote can't spuriously trip a live close. Memoized per expiry.
signal_cache: dict[tuple[str, str], tuple[float | None, float | None]] = {}
signal_marks = {sp["short_osi"]: vrp_marking.model_spread_value(bars, sp, freeze_date, strat.margin / 100.0,
signal_cache)
for sp in open_spreads_prior}
to_close, may_open, _open_slots = plan_vrp_ladder(
open_spreads_prior, marks_today, signal_marks, today, ladder=ladder,
profit_take=strat._profit_take, entry_weekday=strat._entry_weekday) # noqa: SLF001 — composition
# root reaches into the sim's private ladder-management params so the exec twin and the backtest
# never drift apart.
close_osi = {sp["short_osi"] for sp, _mark in to_close}
kept: list[dict[str, Any]] = []
for sp in open_spreads_prior:
if sp["short_osi"] in close_osi:
continue
m = marks_today.get(sp["short_osi"])
kept.append(dict(sp, last_val=m) if m is not None else sp)
result: dict[str, Any] = {"exec_return": exec_return, "placed": 0, "closed": 0, "to_close": len(to_close),
"errors": [], "notes": [], "spread": None}
candidate: dict[str, Any] | None = None
qty_new = 0
legs_new: list[tuple[str, str, int]] | None = None
net_new: float | None = None
if envelope <= 0.0:
result["notes"].append("no envelope (paper-envelope off, no B2a allocation for vrp_exec yet) — no new rungs")
elif may_open:
# select + price the new rung off the FROZEN data session (freeze_date), not `today` (unpublished).
candidate = strat._select_spread(freeze_date) # noqa: SLF001 — composition root reaches into the selector
if candidate is None:
result["notes"].append("no candidate spread selected for today")
else:
marks = bars.bars_for([candidate["short_osi"], candidate["long_osi"]], freeze_date, freeze_date)
short_price = marks.get(candidate["short_osi"], {}).get(freeze_date)
long_price = marks.get(candidate["long_osi"], {}).get(freeze_date)
if short_price is None or long_price is None:
raise RuntimeError(f"vrp_exec: selected spread has no frozen mark for {freeze_date}")
rung_target = rung_notional(envelope, ladder)
qty_new = int(rung_target // strat.margin)
if qty_new < 1:
result["notes"].append(
f"rung target ${rung_target:,.0f} (envelope/{ladder}) < margin ${strat.margin:,.0f}/contract"
" — no new rung")
candidate = None
else:
legs_new, net_new = build_combo_legs(
short_osi=candidate["short_osi"], long_osi=candidate["long_osi"],
short_price=short_price, long_price=long_price, qty=qty_new)
result.update(spread=candidate, legs=legs_new, net=net_new, qty=qty_new)
else:
result["notes"].append("ladder full or off entry day — no new rung")
if not execute:
return result
open_spreads_next = list(kept)
for sp, mark in to_close:
close_legs, close_net = build_close_legs(short_osi=sp["short_osi"], long_osi=sp["long_osi"],
qty=int(sp["qty"]), mark_today=mark)
try:
broker.place_combo(close_legs, close_net)
result["closed"] += 1
except Exception as e: # a failed CLOSE must not silently drop the rung -- re-add it UNCHANGED.
result["errors"].append(f"close {sp['short_osi']}: {type(e).__name__}: {str(e)[:120]}")
open_spreads_next.append(sp)
if candidate is not None and legs_new is not None and net_new is not None:
# Gate the OPEN on the ACTUAL post-close occupancy, not the pre-execution plan: `may_open` above
# was computed from `to_close` before any order was placed. If a CLOSE above failed, its rung was
# re-added to `open_spreads_next` unchanged, so the ladder may still be at/over cap here even though
# `may_open` was True. Re-checking `len(open_spreads_next) < ladder` at this point makes a failed
# close naturally back-pressure new entries instead of breaching the cap (and, since DTE only
# decreases, prevents the un-closable rung from re-freeing a phantom slot every subsequent run).
if len(open_spreads_next) >= ladder:
result["errors"].append(
f"open skipped: ladder full after close failures ({len(open_spreads_next)}/{ladder} occupied)")
result["spread"] = None
else:
try:
order_id = broker.place_combo(legs_new, net_new)
result["placed"] = 1
result["order_id"] = order_id
opened = dict(candidate, qty=qty_new, last_val=candidate["credit"])
open_spreads_next.append(opened)
result["spread"] = opened
except Exception as e: # a failed OPEN just means no new rung this run -- ladder stays as-is.
result["errors"].append(f"open: {type(e).__name__}: {str(e)[:120]}")
result["spread"] = None
_record(repo, exec_return, open_spreads=open_spreads_next, envelope=envelope, venue=venue, today=today, at=at)
result["open_spreads"] = len(open_spreads_next)
return result

View File

@@ -1,116 +0,0 @@
"""SINGLE SOURCE of the VRP defined-risk spread's MODEL mark — the smooth Black-Scholes value used as the
50%-profit-take management SIGNAL by BOTH the backtest (`vrp_book.VrpStrategy`) and the live exec twin
(`vrp_exec_record`). Rebuilt each day from the put-call-parity forward + a robust ATM implied vol (a single
CLEAN near-the-money read), then clipped to the defined-risk bound [0, width] — never the difference of two
individually-noisy raw OPRA leg quotes (a ~$1 spike in a far-OTM long leg swung that raw difference 50-100%
and could spuriously trip a profit-take close).
Only the management SIGNAL uses this model value; realized/entry P&L on real fills stays mark-to-actual.
The forward is derived over a WIDE DTE window and, because the parity forward is spot (common across
expiries to first order under the r≈0 forward measure), falls back to the globally cleanest ATM call/put pair
when the spread's OWN expiry has no frozen calls — the daily freeze band freezes near-ATM calls only for
DTE∈[20,45], so a spread aged below 20 DTE has puts (band DTE∈[1,45]) but no calls at its expiry. Without
this fallback such a spread could not be marked and would be spuriously closed."""
from __future__ import annotations
import datetime as dt
from typing import Any
from fxhnt.application import vrp_pricing
# Wide DTE window for MARKING an already-open spread (any age, incl. below the selection dte_lo). Large
# enough to cover the longest listed expiry; callers filter to the relevant expiry.
MARK_DTE_HI = 3650
def forward_and_atm_iv(bars: Any, day: str, exp: dt.date, tau: float) -> tuple[float | None, float | None]:
"""(forward F, ATM IV) for expiry `exp` on `day`, derived purely from the frozen slice.
forward = put-call parity at the ATM strike (smallest |C P|, ties → lowest strike), preferring the
spread's OWN expiry and falling back to the globally cleanest ATM pair across all frozen expiries (spot is
common). Rejects an implausible forward. ATM IV = `implied_vol_put` inverted from `exp`'s put nearest the
forward (puts are frozen at any age). Returns (None, None) when no clean ATM pair exists; (F, None) when
the forward is clean but IV inversion fails (the caller carries the last good IV)."""
puts = bars.chain_asof(day, 0, MARK_DTE_HI, right="P")
calls = bars.chain_asof(day, 0, MARK_DTE_HI, right="C")
if not puts or not calls:
return None, None
put_marks = bars.bars_for([c.osi_symbol for c in puts], day, day)
call_marks = bars.bars_for([c.osi_symbol for c in calls], day, day)
# priced ATM candidates keyed by (expiry, strike): both a positive put AND call mark at the same strike.
put_by = {(c.expiry, c.strike): p for c in puts
if (p := put_marks.get(c.osi_symbol, {}).get(day)) is not None and p > 0.0}
call_by = {(c.expiry, c.strike): p for c in calls
if (p := call_marks.get(c.osi_symbol, {}).get(day)) is not None and p > 0.0}
common = set(put_by) & set(call_by)
# prefer the spread's own expiry (entry-time behaviour, preserved); else any expiry (spot is common).
same_exp = [ek for ek in common if ek[0] == exp]
candidates = same_exp or list(common)
if not candidates:
return None, None
e_atm, k_atm = min(candidates, key=lambda ek: (abs(call_by[ek] - put_by[ek]), ek[1], ek[0]))
forward = vrp_pricing.parity_forward(call_atm=call_by[(e_atm, k_atm)], put_atm=put_by[(e_atm, k_atm)],
k_atm=k_atm)
if not (0.2 * k_atm < forward < 5.0 * k_atm): # implausible parity forward -> never mark off it
return None, None
# ATM IV from the spread's OWN expiry put nearest the forward (present at any age).
exp_puts = {k: p for (e, k), p in put_by.items() if e == exp}
if not exp_puts:
return forward, None
k_iv = min(exp_puts, key=lambda k: abs(k - forward))
iv = vrp_pricing.implied_vol_put(exp_puts[k_iv], forward, k_iv, tau)
return forward, iv
def model_spread_value(bars: Any, spread: dict[str, Any], day: str, width: float,
day_cache: dict[tuple[str, str], tuple[float | None, float | None]] | None = None
) -> float | None:
"""Smooth MODEL value (cost to close) of `spread` on `day`, clipped to the defined-risk bound [0, width].
Carries the last good IV on `spread['last_iv']`. Returns None when the strikes are unknown or the expiry
has no clean forward (caller CLOSES — no zombie). `day_cache` (optional) memoizes (F, IV) per
(day, expiry) so spreads sharing an expiry in one pass don't re-query the chain."""
ks, kl = spread.get("short_strike"), spread.get("long_strike")
if ks is None or kl is None:
return None # unknown strikes -> no model signal (safe: no close)
exp = dt.date.fromisoformat(spread["expiry"])
tau = max((exp - dt.date.fromisoformat(day)).days, 1) / 365.0 # tau > 0 guard
key = (day, spread["expiry"])
if day_cache is not None and key in day_cache:
forward, iv = day_cache[key]
else:
forward, iv = forward_and_atm_iv(bars, day, exp, tau)
if day_cache is not None:
day_cache[key] = (forward, iv)
if forward is None:
return None # expiry chain gone -> close (not zombie)
if iv is None:
iv = spread.get("last_iv") # carry the last good IV on a degenerate mark day
if iv is None:
return None
spread["last_iv"] = iv
model = vrp_pricing.bs_put(forward, ks, iv, tau) - vrp_pricing.bs_put(forward, kl, iv, tau)
return min(max(model, 0.0), width) # hard defined-risk clip [0, width]
def settlement_value(bars: Any, spread: dict[str, Any], day: str, width: float,
day_cache: dict[tuple[str, str], tuple[float | None, float | None]] | None = None
) -> float | None:
"""Terminal settlement value from `day`'s forward: the put-credit-spread's loss is the short strike's
in-the-moneyness capped at the width — clip(K_short F, 0, width). Realized on the close day (DTE≤1)
instead of the theta model mark."""
ks = spread.get("short_strike")
if ks is None:
return None
exp = dt.date.fromisoformat(spread["expiry"])
tau = max((exp - dt.date.fromisoformat(day)).days, 1) / 365.0
key = (day, spread["expiry"])
if day_cache is not None and key in day_cache:
forward, _iv = day_cache[key]
else:
forward, _iv = forward_and_atm_iv(bars, day, exp, tau)
if day_cache is not None:
day_cache[key] = (forward, _iv)
if forward is None:
return None
return min(max(ks - forward, 0.0), width)

View File

@@ -1,57 +0,0 @@
"""Pure BlackScholes helpers for the defined-risk XSP VRP sleeve — priced under the FORWARD measure so
the underlying comes from put-call parity on the option chain (no external index feed; OPRA is options-only).
No I/O: every input is a real OPRA mark or a frozen bar. r is folded into `disc = e^{-rT}` and the forward F."""
from __future__ import annotations
import math
_SQRT2 = math.sqrt(2.0)
def _norm_cdf(x: float) -> float:
return 0.5 * (1.0 + math.erf(x / _SQRT2))
def bs_put(F: float, K: float, sigma: float, tau: float, disc: float = 1.0) -> float:
"""European put under the forward measure: disc*(K*N(-d2) - F*N(-d1)). disc = e^{-rT} (1.0 for r≈0)."""
if tau <= 0.0 or sigma <= 0.0 or F <= 0.0 or K <= 0.0:
return max(0.0, disc * (K - F)) # intrinsic (degenerate day)
vol = sigma * math.sqrt(tau)
d1 = (math.log(F / K) + 0.5 * sigma * sigma * tau) / vol
d2 = d1 - vol
return disc * (K * _norm_cdf(-d2) - F * _norm_cdf(-d1))
def put_delta(F: float, K: float, sigma: float, tau: float) -> float:
"""|Δ| of the put under the forward measure = N(-d1) in [0,1]. Used for ~20-delta strike selection."""
if tau <= 0.0 or sigma <= 0.0 or F <= 0.0 or K <= 0.0:
return 1.0 if K > F else 0.0
vol = sigma * math.sqrt(tau)
d1 = (math.log(F / K) + 0.5 * sigma * sigma * tau) / vol
return _norm_cdf(-d1)
def implied_vol_put(price: float, F: float, K: float, tau: float, disc: float = 1.0) -> float | None:
"""Invert bs_put for sigma via bisection. Returns None when `price` is below intrinsic / above the bound
(no arb-free vol) — the caller skips that contract for the day."""
if tau <= 0.0 or price <= 0.0:
return None
intrinsic = max(0.0, disc * (K - F))
upper_bound = disc * K # put <= disc*K (sigma -> inf)
if price < intrinsic - 1e-9 or price > upper_bound + 1e-9:
return None
lo, hi = 1e-4, 5.0
if bs_put(F, K, hi, tau, disc) < price: # unreachable even at 500% vol
return None
for _ in range(80):
mid = 0.5 * (lo + hi)
if bs_put(F, K, mid, tau, disc) < price:
lo = mid
else:
hi = mid
return 0.5 * (lo + hi)
def parity_forward(call_atm: float, put_atm: float, k_atm: float, disc: float = 1.0) -> float:
"""Implied forward from put-call parity: C - P = disc*(F - K) => F = K + (C - P)/disc."""
return k_atm + (call_atm - put_atm) / disc

View File

@@ -1,116 +0,0 @@
"""Equity-index Volatility Risk Premium (VRP) — a STRUCTURAL risk premium (not momentum): implied vol
exceeds subsequent realized vol on average because option sellers are paid to bear crash risk. Short vol
harvests IV RV. The catch (and why the deflated Sharpe matters): VRP returns are negatively skewed and
fat-tailed (vol spikes = big losses), so a raw Sharpe overstates it — the skew/kurtosis-adjusted deflated
Sharpe is the honest measure. Built on XSP (mini-SPX) options, 2013-2026, internal put-call-parity (vrp3).
"""
from __future__ import annotations
import datetime as dt
import glob
import math
import os
from dataclasses import dataclass
import numpy as np
from fxhnt.domain.validation import deflated_sharpe_ratio, excess_kurtosis, sharpe_ratio, skewness
_DAY_NS = 86_400 * 10**9
def _to_epoch_day(yymmdd: str) -> int:
y, m, d = 2000 + int(yymmdd[:2]), int(yymmdd[2:4]), int(yymmdd[4:6])
return (dt.date(y, m, d) - dt.date(1970, 1, 1)).days
def _load_xsp(xsp_dir: str): # type: ignore[no-untyped-def]
import databento as db
import pandas as pd
rows = []
for p in sorted(glob.glob(os.path.join(xsp_dir, "*.dbn"))):
try:
df = db.DBNStore.from_file(p).to_df().reset_index()
except Exception:
continue
if df.empty or "symbol" not in df.columns:
continue
df = df[df["close"] > 0]
rows.append(df[["ts_event", "symbol", "close"]])
d = pd.concat(rows, ignore_index=True)
s = d["symbol"].astype(str).str.replace(" ", "", regex=False) # XSP240119C00450000
d["right"] = s.str[-9]
d["strike"] = s.str[-8:].astype(float) / 1000.0
d["expiry"] = s.str[-15:-9]
d["day"] = d["ts_event"].astype("int64") // _DAY_NS
return d
@dataclass
class VrpResult:
n_periods: int
mean_vrp: float
sharpe: float
skew: float
excess_kurt: float
deflated_sharpe: float # skew/kurtosis-adjusted (the honest measure)
dsr_pvalue: float
is_sharpe: float
oos_sharpe: float
max_drawdown: float
def evaluate(xsp_dir: str, *, holding_days: int = 21, n_trials: int = 5, oos_fraction: float = 0.40) -> VrpResult:
import pandas as pd
d = _load_xsp(xsp_dir)
d["exp_day"] = d["expiry"].map(_to_epoch_day)
d["ttm"] = d["exp_day"] - d["day"]
d = d[(d["ttm"] >= 20) & (d["ttm"] <= 45)]
forward, iv = {}, {}
for day, g in d.groupby("day"):
exp = g.iloc[(g["ttm"] - 30).abs().argsort()].iloc[0]["exp_day"]
ge = g[g["exp_day"] == exp]
calls = ge[ge["right"] == "C"].groupby("strike")["close"].mean()
puts = ge[ge["right"] == "P"].groupby("strike")["close"].mean()
common = np.array(sorted(set(calls.index) & set(puts.index)))
if len(common) < 5:
continue
rough = common[np.abs(calls[common].values - puts[common].values).argmin()]
near = common[np.argsort(np.abs(common - rough))[:5]]
f = float(np.median([k + float(calls[k]) - float(puts[k]) for k in near]))
katm = common[np.abs(common - f).argmin()]
straddle = float(calls[katm] + puts[katm])
t_yrs = float(ge["ttm"].iloc[0]) / 365.0
if f <= 0 or straddle <= 0 or t_yrs <= 0:
continue
forward[int(day)] = f
iv[int(day)] = straddle / (0.8 * f * math.sqrt(t_yrs)) # ATM-straddle implied-vol approximation
days = np.array(sorted(forward))
fwd = np.array([forward[x] for x in days])
iv_arr = np.array([iv[x] for x in days])
logret = np.zeros(len(days))
logret[1:] = np.clip(np.log(fwd[1:] / fwd[:-1]), -0.25, 0.25)
# NON-OVERLAPPING short-vol periods: enter, hold `holding_days`, P&L proxy = IV_entry RV_realized
returns = []
step = holding_days
for t in range(0, len(days) - step, step):
rv = float(np.std(logret[t + 1:t + 1 + step]) * math.sqrt(252))
returns.append(iv_arr[t] - rv)
r = np.array(returns)
if len(r) < 8:
raise ValueError("not enough VRP periods")
sk, ek = skewness(r), excess_kurtosis(r)
dsr = deflated_sharpe_ratio(sharpe_ratio(r), n_trials, 0.0, sk, ek, len(r))
split = int((1.0 - oos_fraction) * len(r))
eq = np.cumprod(1.0 + r / 100.0) # scale vol-points to a rough equity curve
mdd = float((eq / np.maximum.accumulate(eq) - 1.0).min())
return VrpResult(
n_periods=len(r), mean_vrp=float(r.mean()), sharpe=sharpe_ratio(r), skew=sk, excess_kurt=ek,
deflated_sharpe=1.0 - dsr.pvalue, dsr_pvalue=dsr.pvalue,
is_sharpe=sharpe_ratio(r[:split]), oos_sharpe=sharpe_ratio(r[split:]), max_drawdown=mdd)

View File

@@ -436,69 +436,6 @@ def execute_bybit(
f"| exec_return {out.get('exec_return')}")
@app.command("execute-vrp")
def execute_vrp(
do_execute: bool = typer.Option(False, "--execute", help="place combo orders (default: dry-run plan only)"),
paper_envelope: float = typer.Option(
0.0, "--paper-envelope",
help="PAPER-only validation envelope in $ (there is no B2a allocation for VRP yet, so this is the "
"ONLY way this track sizes a new ladder rung; 0 = no new rungs, existing rungs still mark/close)."),
) -> None:
"""Reconcile + record the EXECUTED XSP put-credit-spread VRP twin on IBKR (`mleg` combo), maintaining a
ladder of open rungs (mirrors `VrpStrategy.advance`'s discipline: close a rung at 50% profit or DTE<=1,
cap entries at `ladder`, size each new rung at envelope/ladder — never the full envelope). SUSPENDED: the
IBKR account currently lacks options trading permission and CI has no live OPRA, so this path is coded
but not runnable here — dry-run by default even once armed. Same paper-envelope refusal guard as
`execute-multistrat`/`execute-bybit`. Records the observed `vrp_exec` return."""
import datetime as _dt
from fxhnt.adapters.persistence.forward_nav import ForwardNavRepo
from fxhnt.application.vrp_exec_record import plan_and_record_vrp
settings = get_settings()
d = settings.databento
dbn = DatabentoDataProvider(max_cost_usd=d.max_cost_usd, equity_dataset=d.equity_dataset,
future_dataset=d.future_dataset, default_start=d.default_start)
repo = ForwardNavRepo(settings.operational_dsn)
repo.migrate()
today = _dt.datetime.now(_dt.UTC).strftime("%Y-%m-%d")
at = _dt.datetime.now(_dt.UTC).replace(tzinfo=None)
broker = IbkrBroker(settings.ibkr.host, settings.ibkr.port, settings.ibkr.client_id, settings.ibkr.timeout)
paper_env = _resolve_paper_envelope(paper_envelope, settings.paper_validation.envelope)
try:
with broker as brk:
acct = brk.account_state()
venue = _exec_venue_string(brk.name, acct.is_paper)
out = plan_and_record_vrp(
repo, dbn=dbn, broker=brk, nlv=acct.nlv, today=today, at=at,
paper_envelope=paper_env, account_is_paper=acct.is_paper, venue=venue, execute=do_execute)
except ValueError as e: # a deliberate safety refusal (e.g. paper-envelope on a non-paper account) —
typer.echo(f"paper-envelope refused: {e}") # distinct from a connectivity/data error
raise typer.Exit(1) from e
except Exception as e: # connection / OPRA / runtime
typer.echo(f"broker error: {type(e).__name__}: {str(e)[:100]}")
typer.echo(f" reachable? IBKR ({settings.ibkr.host}:{settings.ibkr.port}) "
f"— creds/gateway + options permission set? OPRA reachable?")
raise typer.Exit(1) from e
if out.get("noop"):
typer.echo(f"-> no-op: {out.get('reason')}")
return
typer.echo(f"to_close={out.get('to_close', 0)} closed={out.get('closed', 0)}")
spread = out.get("spread")
if spread:
typer.echo(f"spread: SELL {spread['short_osi']} / BUY {spread['long_osi']} qty={out.get('qty')} "
f"net ${out.get('net'):+.2f}")
for n in out.get("notes", []):
typer.echo(f" - {n}")
for err in out.get("errors", []):
typer.echo(f" order error: {err}")
typer.echo(f"-> exec_return {out.get('exec_return')} | placed {out.get('placed', 0)} "
f"| open_spreads {out.get('open_spreads')}")
@app.command("migrate-cockpit")
def migrate_cockpit() -> None:
"""Create the cockpit tables (+ TimescaleDB hypertable on Postgres) and seed the strategy registry."""
@@ -575,54 +512,6 @@ def unlock_calendar_import(
f"unlock_events @ {dsn.split('@')[-1]} (table now has {len(repo.read_events())})")
@app.command("backfill-xsp-opra")
def backfill_xsp_opra(
start: str = typer.Option(..., "--start", help="ISO date to begin the backfill from (e.g. 2013-04-01)"),
max_cost: float = typer.Option(
25.0, "--max-cost",
help="per-MONTH OPRA cost cap in $ (BATCHED: one range definition + one range ohlcv fetch per month, "
"so the guard is per-month, not per-day)."),
) -> None:
"""One-time historical freeze of the XSP OPRA slice from --start through today into `xsp_option_bars`,
BATCHED per month: one range `definition` + one range `ohlcv-1d` fetch per month, then every trading day
processed LOCALLY (parity forward + band + upsert) — ~2 OPRA calls/month vs the old ~6-8/day (~1h vs ~45h
over 2013->). Resumable/idempotent (`upsert_bars` merges on (osi_symbol, date)) — safe to re-run or resume
from a later --start. A bad month is logged + skipped, never aborts. Manual, one-time; NOT the nightly
asset graph. The per-fetch databento cost guard stays active (raise --max-cost, never disable it)."""
import datetime as _dt
from fxhnt.adapters.orchestration.assets import _backfill_month
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
settings = get_settings()
d = settings.databento
dbn = DatabentoDataProvider(max_cost_usd=max_cost, equity_dataset=d.equity_dataset,
future_dataset=d.future_dataset, default_start=d.default_start)
bars = XspOptionBarsRepo(settings.operational_dsn)
bars.migrate()
at = _dt.datetime.now(_dt.UTC).replace(tzinfo=None)
start_d = _dt.date.fromisoformat(start)
today = _dt.datetime.now(_dt.UTC).date()
cur = start_d.replace(day=1)
tot_ok = tot_fail = 0
while cur <= today:
nxt = (cur + _dt.timedelta(days=32)).replace(day=1) # first of the next month
m_start = max(cur, start_d)
m_end = min(nxt - _dt.timedelta(days=1), today)
try:
ok, fail = _backfill_month(dbn, bars, month_start=m_start.isoformat(),
month_end=m_end.isoformat(), at=at)
tot_ok += ok
tot_fail += fail
typer.echo(f"backfill-xsp-opra: {cur:%Y-%m} done ({ok} ok, {fail} fail) — {tot_ok} ok total")
except Exception as e: # a bad month (cost/network) — log + continue, never abort the whole backfill
typer.echo(f"backfill-xsp-opra: {cur:%Y-%m} FAILED ({type(e).__name__}: {str(e)[:120]}) — skipped")
cur = nxt
typer.echo(f"-> backfill-xsp-opra complete: {tot_ok} days frozen, {tot_fail} failed, "
f"{start}..{today.isoformat()}")
@app.command("backfill-etf-volume")
def backfill_etf_volume() -> None:
"""One-time/resumable historical freeze of each `FUND_INSTRUMENTS` ETF's raw Yahoo daily share volume
@@ -1214,58 +1103,6 @@ def bybit_ingest_klines(
f"oldest date {oldest_date}")
@app.command("ingest-dvol")
def ingest_dvol_cmd(
currencies: str = typer.Option(
"BTC,ETH", "--currencies",
help="comma-separated Deribit DVOL currencies to ingest (default BTC,ETH — the assets with a DVOL "
"index). Each is written as feature 'dvol' under '<CCY>USDT' in bybit_features."),
from_month: str = typer.Option(
"", "--from", help="floor for the backward pagination, YYYY-MM (default: the 2021-03 backstop, just "
"below DVOL inception 2021-03-24)."),
) -> None:
"""Ingest the Deribit DVOL annualised IMPLIED-vol index (the IV leg of the VRP edge) into the
Bybit-namespaced TimescaleDB table `bybit_features` as feature 'dvol', keyed by the perp symbol
(BTC -> BTCUSDT, ETH -> ETHUSDT). resolution=86400 -> one value/day. ONLY 'dvol' is written
(close/funding are the other ingests' job); idempotent/resumable. The VRP eval reads this `dvol` panel
alongside the kline `close` panel. No live paper_* tables are touched."""
import datetime as _dt
from fxhnt.adapters.data.deribit_dvol import DeribitDVOL
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
from fxhnt.application.dvol_ingest import ingest_dvol
settings = get_settings()
deribit = DeribitDVOL()
start_ms: int | None = None
if from_month:
try:
y, m = (int(x) for x in from_month.split("-"))
start_ms = int(_dt.datetime(y, m, 1, tzinfo=_dt.UTC).timestamp() * 1000)
except (ValueError, TypeError) as e:
typer.echo(f"error: --from must be YYYY-MM (got {from_month!r})", err=True)
raise typer.Exit(1) from e
ccys = tuple(c.strip().upper() for c in currencies.split(",") if c.strip())
if not ccys:
typer.echo("error: empty --currencies", err=True)
raise typer.Exit(1)
typer.echo(f"ingest-dvol: {ccys}, floor={from_month or '2021-03 (default)'} "
f"-> bybit_features feature='dvol' @ {settings.operational_dsn.split('@')[-1]}")
store = TimescaleFeatureStore(settings.operational_dsn, table="bybit_features")
try:
summary = ingest_dvol(store, deribit, currencies=ccys, start_ms=start_ms)
finally:
store.close()
oldest = summary["oldest_day"]
oldest_date = ((_dt.date(1970, 1, 1) + _dt.timedelta(days=oldest)).isoformat()
if oldest is not None else "n/a")
typer.echo(f"\ningest-dvol done: {summary['symbols']} currencies, "
f"{summary['day_rows']} day-rows, oldest date {oldest_date}")
@app.command("ingest-deribit-funding")
def ingest_deribit_funding_cmd(
instruments: str = typer.Option(
@@ -2189,303 +2026,6 @@ def bybit_positioning_eval(
typer.echo("\nNOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched. Memory-bounded: only the "
"(liquid, with --liquid) coins are READ — the full panels are never loaded.")
@app.command("vrp-eval")
def vrp_eval(
direction: str = typer.Option(
"short_vol", "--direction",
help="short_vol (DEFAULT: SELL variance — harvest the IV-over-RV premium; fat LEFT tail on vol "
"spikes) or long_vol (BUY variance — the negation). Both are RE-RUN so the cost subtracts "
"correctly in each."),
cost_bps: float = typer.Option(
5.5, "--cost-bps",
help="Per-day cost (bps, scaled by the vol-target leverage). NOTE: selling OPTIONS is NOT 5.5bp — "
"--verify shows a 50/100bp option-cost sensitivity."),
target_ann_vol: float = typer.Option(
0.15, "--target-vol", help="Vol-target the VRP book to this annual vol (default 0.15 = 15%/yr)."),
verify: bool = typer.Option(
False, "--verify",
help="ADVERSARIAL verification: BOTH directions' net-of-cost metrics across a cost grid INCLUDING "
"realistic OPTION costs (5.5/11/22/50/100 bps), the single WORST DAY (the vol-spike tail), "
"maxDD, per-year Sharpe+maxDD, AND the correlation with EACH existing edge "
"(tstrend/unlock/xsfunding/positioning)."),
walk_forward: bool = typer.Option(
False, "--walk-forward",
help="OOS / WALK-FORWARD validation (the TAIL-AWARE DEPLOY GATE): a TRAIN/TEST split that picks the "
"direction AND sizes k on TRAIN then scores OOS, rolling non-overlapping windows, a look-ahead "
"audit, and a PASS/FAIL verdict that FAILS a catastrophic-tail book even when Sharpe>0."),
window_days: int = typer.Option(
180, "--window-days", help="Width (days) of each rolling non-overlapping OOS window in --walk-forward."),
cost_grid: str = typer.Option(
"5.5,11,22,50,100", "--cost-grid",
help="comma-separated cost levels (bps) for --verify (default spans cheap perp-style + realistic "
"option costs)."),
) -> None:
"""READ-ONLY, TAIL-HONEST: evaluate the VRP (volatility-risk-premium) edge — a vol premium, a genuinely
DIFFERENT edge type from the directional/flow edges already in the book.
SIGNAL = Deribit DVOL (implied vol) vs the BTC/ETH close-to-close REALIZED vol. short-vol (default)
harvests the IV-over-RV spread via a ROLLING VARIANCE SWAP: strike K=(DVOL/100)² (implied annual variance,
causal) vs RV²=(365/swap_days)·Σ ret² over the swap's life; a daily ladder of overlapping 30-day swaps
gives a SMOOTH daily series, BTC+ETH equal-weight, vol-targeted to --target-vol. POSITIVE on calm windows,
big-NEGATIVE on vol-spike windows (the fat left tail). EXECUTION would be short Bybit option straddles
(a cross-venue caveat vs the Deribit signal).
Prints CAGR/Sharpe/maxDD + the single WORST DAY (the vol spike) + the vol-target leverage. --verify adds
the option-cost sensitivity, per-year tails, and the correlation with each existing edge. --walk-forward
is the tail-aware deploy gate (a clean "no / too-tail-heavy" is a VALID outcome). Writes NOTHING."""
if direction not in ("short_vol", "long_vol"):
raise typer.BadParameter("must be 'short_vol' or 'long_vol'", param_hint="--direction")
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
from fxhnt.application.vrp_eval import (
verify_vrp_edge,
vrp_metrics_from_store,
walk_forward_vrp,
)
settings = get_settings()
store = TimescaleFeatureStore(settings.operational_dsn, table="bybit_features")
if walk_forward:
try:
rep = walk_forward_vrp(store, cost_bps=cost_bps, window_days=window_days,
target_ann_vol=target_ann_vol)
finally:
store.close()
_print_vrp_walk_forward(rep, cost_bps=cost_bps)
return
if verify:
grid = tuple(float(x) for x in cost_grid.split(",") if x.strip())
try:
rep = verify_vrp_edge(store, cost_bps=cost_bps, cost_grid=grid, target_ann_vol=target_ann_vol)
finally:
store.close()
_print_vrp_verify(rep, cost_bps=cost_bps)
return
try:
m = vrp_metrics_from_store(store, cost_bps=cost_bps, direction=direction,
target_ann_vol=target_ann_vol)
finally:
store.close()
typer.echo("\n=== VRP (volatility risk premium) — TAIL-HONEST, READ-ONLY ===")
typer.echo(f"direction: {direction.upper()} ({'SELL variance — harvest' if direction == 'short_vol' else 'BUY variance — negation'})")
typer.echo("signal: Deribit DVOL (IV) vs BTC/ETH realized vol | book: BTC+ETH equal-weight, vol-targeted")
typer.echo(f"construction: rolling 30d variance swap K=(DVOL/100)^2 vs RV^2=(365/n)*sum(ret^2) (daily "
f"ladder) | target vol={target_ann_vol*100:.0f}%/yr")
if m.get("available"):
typer.echo(f" CAGR={m['cagr']*100:7.1f}% Sharpe={m['sharpe']:6.2f} maxDD={m['max_dd']*100:7.1f}% "
f" total={m['total_return']*100:7.1f}% days={m['days']} span={m['first_day']}..{m['last_day']}")
typer.echo(f" WORST DAY (vol-spike tail loss): {m['worst_day']*100:+.2f}% | avg leverage (k): {m['avg_leverage']:.2f}")
else:
typer.echo(f" N/A — {m.get('reason', '')}")
typer.echo(" (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)")
typer.echo("\nCROSS-VENUE CAVEAT: the SIGNAL is Deribit's DVOL; live EXECUTION would be short Bybit option "
"straddles (delta-hedged).")
typer.echo("NOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched.")
@app.command("vrp-defined-risk-eval")
def vrp_defined_risk_eval(
direction: str = typer.Option(
"short_vol", "--direction",
help="short_vol (DEFAULT: short ATM straddle + long OTM wings — harvest the IV-over-RV premium with a "
"BOUNDED tail) or long_vol (the negation)."),
cost_bps: float = typer.Option(
5.5, "--cost-bps",
help="Per-day cost (bps, scaled by the vol-target leverage). Selling OPTIONS is NOT 5.5bp — use "
"--cost-bps 50 (or --verify) for the realistic option-cost deploy gate."),
target_ann_vol: float = typer.Option(
0.15, "--target-vol", help="Vol-target the defined-risk book to this annual vol (default 0.15)."),
wing_width: float = typer.Option(
0.15, "--wing-width",
help="OTM wing distance as a FRACTION of spot (default 0.15 = ±15%). The max loss is BOUNDED at "
"wing_width credit. Wider wing => cheaper protection => larger credit but larger max loss "
"(naked limit as wing_width -> 1)."),
wing_in_atm_std: float = typer.Option(
0.0, "--wing-in-atm-std",
help="If > 0, place the wings at this many ATM std-moves (wing = N·IV·sqrt(T)) instead of the fixed "
"--wing-width. 0 (default) uses --wing-width."),
put_skew_vol: float = typer.Option(
5.0, "--put-skew-vol",
help="Crypto put-skew bump (vol POINTS) added to the put-leg IV (default +5). 0 = flat-ATM = under-"
"prices the put wing => mildly OPTIMISTIC (documented caveat)."),
verify: bool = typer.Option(
False, "--verify",
help="ADVERSARIAL verification: both directions net-of-cost across the cost grid (incl 50/100bp option "
"costs), worst-day, maxDD, per-year, and corr with each existing edge."),
walk_forward: bool = typer.Option(
False, "--walk-forward",
help="OOS / WALK-FORWARD validation (the TAIL-AWARE DEPLOY GATE): TRAIN picks direction+k, scored OOS, "
"rolling windows, look-ahead audit, PASS/FAIL verdict. The wings are meant to PASS the tail "
"check the naked VRP (74% maxDD) failed."),
window_days: int = typer.Option(
180, "--window-days", help="Width (days) of each rolling non-overlapping OOS window in --walk-forward."),
cost_grid: str = typer.Option(
"5.5,11,22,50,100", "--cost-grid",
help="comma-separated cost levels (bps) for --verify (cheap perp-style + realistic option costs)."),
) -> None:
"""READ-ONLY, TAIL-HONEST: evaluate the DEFINED-RISK VRP — the deployable form of the vol-risk premium.
The naked short-variance VRP has a real gross edge but FAILS the deploy gate on an UNBOUNDED left tail
(modelled maxDD ~74% on vol spikes + realistic option costs). This caps it: SELL the ATM straddle but BUY
OTM wings (an iron condor) so the max loss is BOUNDED at wing_width credit. The legs are priced from DVOL
via BLACK-SCHOLES (we have no historical option chains — a documented BS + put-skew approximation, so the
modelled credit is an approximation, not a real chain). Re-runs the SAME tail-aware harness as the naked
evaluator (vol-target, cost grid, per-year, worst-day, maxDD, corr-vs-4-edges, look-ahead, PASS/FAIL) so
the two are directly comparable — the headline is the maxDD the wings buy back. Writes NOTHING."""
if direction not in ("short_vol", "long_vol"):
raise typer.BadParameter("must be 'short_vol' or 'long_vol'", param_hint="--direction")
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
from fxhnt.application.vrp_defined_risk_eval import (
defined_risk_metrics_from_store,
verify_defined_risk_edge,
walk_forward_defined_risk,
)
settings = get_settings()
store = TimescaleFeatureStore(settings.operational_dsn, table="bybit_features")
wstd = wing_in_atm_std if wing_in_atm_std > 0.0 else None
if walk_forward:
try:
rep = walk_forward_defined_risk(store, cost_bps=cost_bps, window_days=window_days,
target_ann_vol=target_ann_vol, wing_width=wing_width,
put_skew_vol=put_skew_vol, wing_in_atm_std=wstd)
finally:
store.close()
_print_vrp_walk_forward(rep, cost_bps=cost_bps)
typer.echo(f"\nDEFINED-RISK: wing_width=±{wing_width*100:.0f}% put_skew_vol=+{put_skew_vol:.1f}vp "
"(short ATM straddle + long OTM wings; max loss BOUNDED at wing_width credit)")
typer.echo("MODELLING CAVEAT: legs are BLACK-SCHOLES-priced from the ATM DVOL (no real option chain); "
"put_skew_vol=0 under-prices the put wing => optimistic.")
return
if verify:
grid = tuple(float(x) for x in cost_grid.split(",") if x.strip())
try:
rep = verify_defined_risk_edge(store, cost_bps=cost_bps, cost_grid=grid,
target_ann_vol=target_ann_vol, wing_width=wing_width,
put_skew_vol=put_skew_vol, wing_in_atm_std=wstd)
finally:
store.close()
_print_vrp_verify(rep, cost_bps=cost_bps)
typer.echo(f"\nDEFINED-RISK: wing_width=±{wing_width*100:.0f}% put_skew_vol=+{put_skew_vol:.1f}vp "
"(BS-priced from DVOL; max loss BOUNDED at wing_width credit)")
return
try:
m = defined_risk_metrics_from_store(store, cost_bps=cost_bps, direction=direction,
target_ann_vol=target_ann_vol, wing_width=wing_width,
put_skew_vol=put_skew_vol, wing_in_atm_std=wstd)
finally:
store.close()
typer.echo("\n=== DEFINED-RISK VRP (short straddle + protective wings) — TAIL-HONEST, READ-ONLY ===")
typer.echo(f"direction: {direction.upper()} | wing_width: ±{wing_width*100:.0f}% | put_skew_vol: "
f"+{put_skew_vol:.1f}vp | target vol={target_ann_vol*100:.0f}%/yr")
typer.echo("construction: SELL ATM straddle, BUY OTM wings (BS-priced from DVOL); max loss BOUNDED at "
"wing_width credit")
if m.get("available"):
typer.echo(f" CAGR={m['cagr']*100:7.1f}% Sharpe={m['sharpe']:6.2f} maxDD={m['max_dd']*100:7.1f}% "
f" total={m['total_return']*100:7.1f}% days={m['days']} span={m['first_day']}..{m['last_day']}")
typer.echo(f" WORST DAY (capped vol-spike loss): {m['worst_day']*100:+.2f}% | avg leverage (k): {m['avg_leverage']:.2f}")
else:
typer.echo(f" N/A — {m.get('reason', '')}")
typer.echo(" (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)")
typer.echo("\nMODELLING CAVEAT: legs are BLACK-SCHOLES-priced from the ATM DVOL (no real option chain); a "
"put-skew bump models richer crypto puts (put_skew_vol=0 under-prices the put wing => optimistic).")
typer.echo("CROSS-VENUE CAVEAT: SIGNAL = Deribit DVOL; live EXECUTION = short Bybit option condors.")
typer.echo("NOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched.")
def _print_vrp_verify(rep: dict, *, cost_bps: float) -> None:
"""Pretty-print the adversarial VRP verification report (both directions, tail-honest)."""
typer.echo("\n=== VRP (volatility risk premium) — ADVERSARIAL VERIFICATION (TAIL-HONEST, READ-ONLY) ===")
typer.echo("signal: Deribit DVOL (IV) vs BTC/ETH realized vol | book: BTC+ETH equal-weight, vol-targeted")
if not rep.get("available"):
typer.echo(f" N/A — {rep.get('reason', '')}")
typer.echo(" (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)")
return
typer.echo("\n-- NET-OF-COST, BOTH DIRECTIONS (incl REALISTIC OPTION costs 50/100bp) --")
for d in ("short_vol", "long_vol"):
typer.echo(f" [{d.upper()}] avg leverage (k)={rep['avg_leverage'].get(d, 0.0):.2f}")
for bps in rep["cost_grid"]:
m = rep["net_cost"][d][bps]
typer.echo(f" cost={bps:6.1f}bps Sharpe={m['sharpe']:6.2f} CAGR={m['cagr']*100:7.1f}% "
f"maxDD={m['max_dd']*100:7.1f}% total={m['total_return']*100:8.1f}% days={m['days']}")
typer.echo("\n-- THE TAIL (single worst day = the vol spike; maxDD @ headline cost) --")
for d in ("short_vol", "long_vol"):
typer.echo(f" [{d.upper()}] WORST DAY={rep['worst_day'][d]*100:+.2f}% maxDD={rep['max_dd'][d]*100:+.1f}%")
typer.echo("\n-- PER-YEAR Sharpe + maxDD (net @ headline cost; all-from-one-year => overfit) --")
for d in ("short_vol", "long_vol"):
years = rep["per_year"][d]
ys = " ".join(f"{yr}:Sh={y['sharpe']:+5.2f}/DD={y['max_dd']*100:+4.0f}%(n={y['days']})"
for yr, y in sorted(years.items()))
typer.echo(f" [{d.upper()}] {ys}")
typer.echo(f"\n-- CORRELATION with EACH existing edge (net @ {cost_bps:.1f}bps; ~0 => additive) --")
for d in ("short_vol", "long_vol"):
corrs = rep["corr_edges"][d]
parts = [f"{edge}={'N/A' if corrs.get(edge) is None else f'{corrs[edge]:+.3f}'}"
for edge in ("tstrend", "unlock", "xsfunding", "positioning")]
typer.echo(f" [{d.upper()}] " + " ".join(parts))
typer.echo("\nVERDICT GUIDE: a deployable VRP edge stays Sharpe>0 at REALISTIC OPTION cost (50/100bp), is "
"positive in MORE THAN ONE year, has a SURVIVABLE tail (maxDD not catastrophic), AND is "
"uncorrelated with all four existing edges. VRP looks great until a spike — judge it NET of "
"the worst day.")
typer.echo(f"\nCROSS-VENUE CAVEAT: {rep.get('cross_venue_note', '')}")
typer.echo("NOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched.")
def _print_vrp_walk_forward(rep: dict, *, cost_bps: float) -> None:
"""Pretty-print the tail-aware OOS / walk-forward validation report (the deploy gate)."""
typer.echo("\n=== VRP — OOS / WALK-FORWARD VALIDATION (TAIL-AWARE DEPLOY GATE, READ-ONLY) ===")
typer.echo(f"signal: Deribit DVOL (IV) vs BTC/ETH realized vol | cost: {cost_bps:.1f}bps")
if not rep.get("available"):
typer.echo(f" N/A — {rep.get('reason', '')}")
typer.echo(" (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)")
return
th = rep["train_holdout"]
rw = rep["rolling_windows"]
full = rep["full"]
typer.echo(f" train-chosen direction: {rep['direction'].upper()} | vol-target={rep['target_ann_vol']*100:.0f}%/yr "
f"| avg leverage (k)={rep['avg_leverage']:.2f} | maxDD floor={rep['max_dd_floor']*100:.0f}%")
typer.echo(f" FULL-sample: Sharpe={full['sharpe']:6.2f} maxDD={full['max_dd']*100:+.1f}% "
f"worst day={full['worst_day']*100:+.2f}% total={full['total_return']*100:+.1f}%")
typer.echo("\n-- TRAIN/TEST (OOS) split: direction+k chosen on TRAIN, scored on held-out TEST --")
typer.echo(f" train Sharpe={th['train_sharpe']:+6.2f} -> TEST(OOS) Sharpe={th['test_sharpe']:+6.2f} "
f"TEST maxDD={th.get('test_max_dd', 0.0)*100:+.1f}% (split@{th.get('split_at', 'n/a')})")
typer.echo(f"\n-- ROLLING non-overlapping {rep['window_days']}d windows ({rw['n_windows']}); "
f"fraction positive={rw['fraction_positive']*100:.0f}% --")
for w in rw["windows"]:
typer.echo(f" {w['start']}..{w['end']} Sharpe={w['sharpe']:+6.2f} total={w['total_return']*100:+7.1f}% "
f"maxDD={w['max_dd']*100:+5.1f}% (n={w['days']})")
typer.echo(f"\n-- LOOK-AHEAD audit: causal={rep['look_ahead']['causal']} "
f"(leak_days={rep['look_ahead']['leak_days']}) --")
v = rep["verdict"]
typer.echo("\n-- VERDICT (tail-aware) --")
for name, ok in v["checks"].items():
typer.echo(f" [{'PASS' if ok else 'FAIL'}] {name}")
typer.echo(f"\n ==> {'PASS — DEPLOYABLE' if v['deployable'] else 'FAIL — NOT deployable'} "
f"(a clean 'no / too-tail-heavy' is a valid outcome)")
typer.echo(f"\nCROSS-VENUE CAVEAT: {rep.get('cross_venue_note', '')}")
typer.echo("NOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched.")
def _print_oi_deleverage_verify(rep: dict, *, scope: str, direction_note: str, cost_bps: float) -> None:
"""Pretty-print the adversarial OI-deleveraging verification report (both directions)."""
typer.echo("\n=== Bybit OI-deleveraging reversion — ADVERSARIAL VERIFICATION (READ-ONLY) ===")

View File

@@ -114,6 +114,12 @@ class UcitsListing(BaseModel):
isin: str # unambiguous security id (e.g. "IE00B5BMR087") — the real resolution key
exchange: str = "LSEETF" # the LSE ETF venue: LSEETF + ISIN + USD resolves the line (SMART+ISIN = 0)
currency: str = "USD"
# The symbol IBKR REPORTS a HOLDING of this line under (`contract.symbol` from `positions()`), which can
# diverge from the cosmetic `ticker`. Probe-verified live 2026-07-20 (`probe-ucits-symbols`): only IEF's
# `CBU0` diverges → IBKR reports it as `CSBGU0`; the other 5 lines report under their ticker. The cockpit's
# NLV stray-check (`_expected_book_symbols`) keys on this, so a legitimate book position is not mislabeled
# a stray + dropped from NLV. Defaults to `ticker` when they match (the common case).
ibkr_symbol: str = ""
# Yahoo LSE daily-volume ticker (ADV input for the UCITS-regime capacity/cost model). Defaults to
# f"{ticker}.L" when left empty; set explicitly if Yahoo's symbol diverges from the display ticker.
yahoo_ticker: str = ""
@@ -122,9 +128,11 @@ class UcitsListing(BaseModel):
half_spread_bps: float = 10.0
@model_validator(mode="after")
def _default_yahoo_ticker(self) -> "UcitsListing":
def _default_derived_fields(self) -> UcitsListing:
if not self.yahoo_ticker:
self.yahoo_ticker = f"{self.ticker}.L"
if not self.ibkr_symbol: # the common case: IBKR reports the holding under the ticker
self.ibkr_symbol = self.ticker
return self
@@ -149,6 +157,7 @@ class UcitsSettings(BaseSettings):
map: dict[str, UcitsListing] = Field(default_factory=lambda: {
"SPY": UcitsListing(ticker="CSPX", isin="IE00B5BMR087", half_spread_bps=1.5),
"IEF": UcitsListing(ticker="CBU0", isin="IE00B3VWN518", yahoo_ticker="CBU0.L", # Acc line, ~$12M ADV
ibkr_symbol="CSBGU0", # IBKR reports the HOLDING under CSBGU0 (probe 2026-07-20)
half_spread_bps=2.0),
"GLD": UcitsListing(ticker="IGLN", isin="IE00B4ND3602", # USD line = IGLN (SGLN is the GBP line)
half_spread_bps=2.0),
@@ -203,7 +212,8 @@ class Settings(BaseSettings):
paper_enabled: bool = True # toggle the daily-rebalance paper-book persistence hook (T7)
# Staleness/health axis: a recompute track whose source data (as_of) OR freeze table hasn't advanced in
# more than this many BUSINESS days reads as STALE (a red cockpit badge + ERROR log). Default 3 allows
# the normal ~1-business-day ingest lag; the anti-silent-rot threshold that would have caught vrp.
# the normal ~1-business-day ingest lag; the anti-silent-rot threshold that would have caught a
# strategy silently rotting for weeks.
stale_after_business_days: int = 3
cost_bps_per_turnover: float = 10.0 # round-trip cost model (bps of traded notional)

View File

@@ -68,8 +68,8 @@ class FleetRow:
exec_twin_sid: str = ""
exec_return_pct: float | None = None
exec_divergence_pct: float | None = None
# D3 (Task 4) — the REAL IBKR paper-account "real trades" signal: "EXEC" only for the two IBKR real-
# account books (multistrat/vrp) once real fills have been captured (`exec_vs_sim(...).has_exec`); ""
# D3 (Task 4) — the REAL IBKR paper-account "real trades" signal: "EXEC" only for the IBKR real-
# account book (multistrat) once real fills have been captured (`exec_vs_sim(...).has_exec`); ""
# (with `recon=None`) for every other strategy AND for an IBKR book with no captured data yet — the
# cockpit's Overview sub-row only ever renders for a genuinely-executing row, never a fabricated one.
exec_status: str = ""
@@ -187,7 +187,7 @@ class StrategyDetail:
exec_venue: str | None = None
# D2.2/2.3 (Task 3) — the REAL IBKR paper-account section: what the book is holding right now, its
# recent trades, and the account's raw NLV series (for the value-over-time overlay curve). Empty for
# every strategy that isn't one of the IBKR real-account books (multistrat/vrp) or when no account data
# every strategy that isn't the IBKR real-account book (multistrat) or when no account data
# has been captured yet — the template renders "no trades recorded yet", never a 500.
holdings: list[dict[str, Any]] = field(default_factory=list)
# Held symbols outside the book's allowed universe (US + UCITS) — a legacy/stray position (e.g. an IBIT

View File

@@ -18,12 +18,6 @@ class CryptoPerpBarClient(Protocol):
...
class VolIndexClient(Protocol):
def dvol(self, currency: str) -> dict[int, float]:
"""{ epoch_day: implied-vol-index } for BTC/ETH (Deribit DVOL)."""
...
class FundamentalsClient(Protocol):
def metrics(self, symbol: str) -> dict[str, float]:
"""Latest daily valuation metrics: {'peRatio','pbRatio','marketCap', ...} (most recent available)."""

View File

@@ -154,36 +154,6 @@ STRATEGY_REGISTRY: dict[str, dict[str, Any]] = {
# track) — FDUSD/USDP aren't listed on Bybit, doesn't port to the venue. `StableReversionRunner` itself
# (stablecoin_runner.py) stays — it is still live via `bybit_stablecoin_eval.py` / `bybit_edges_eval.py`
# / `bybit_book_eval.py` (Bybit-native stablecoin evaluation, a separate consumer of the runner core).
# Defined-risk equity VRP (XSP put-credit-spreads) on REAL OPRA option data — the cross-asset diversifier
# (co-crashes systemic risk-off, so NOT a systemic hedge; diversifies idiosyncratic crypto crashes).
# record_mode recompute off the frozen xsp_option_bars PIT table. promote_on_pass False: real-capital-
# adjacent, so promotion to deploy (CVaR-sized capital) is a deliberate human step after it reconciles.
"vrp": {
# ARCHIVED/FALSIFIED 2026-07-15 — shelved dead-end; code + 13yr OPRA data kept; see project_fxhnt_diversifier_hunt_2026_07_15
"archived": True,
"display_name": "Options income (put spreads)", "sleeve": "equity-vol",
"venue": "ibkr-xsp",
"tier": "research", "execution": "paper", "promote_on_pass": False,
"gate_spec": {"gate_type": "reconciliation", "min_forward_days": 21},
"definition": {"params": {"instrument": "XSP", "structure": "put-credit-spread",
"dte_target": 30, "short_delta": 0.20, "width_points": 5.0,
"ladder": 5, "source": "opra-pit"},
"version": 1, "record_mode": "recompute", "backtest": {"kind": "recompute-replay"}},
},
# EXECUTED VRP twin (sub-project C-style): the real-fill reconciliation for `vrp` on IBKR XSP `mleg`
# combos — SUSPENDED (the IBKR account has no options permission yet), coded ahead of that grant so the
# cockpit twin + gate wiring is ready. record_mode observed (booked from `vrp_exec_record`, not
# recomputed).
"vrp_exec": {
# ARCHIVED/FALSIFIED 2026-07-15 — shelved dead-end; code + 13yr OPRA data kept; see project_fxhnt_diversifier_hunt_2026_07_15
"archived": True,
"display_name": "Equity VRP (EXECUTED, IBKR XSP)", "sleeve": "equity-vol",
"venue": "ibkr-xsp",
"tier": "research", "execution": "paper",
"gate_spec": {"gate_type": "reconciliation", "min_forward_days": 21},
"definition": {"params": {"source": "ibkr-xsp-fills", "models": "vrp"},
"version": 1, "record_mode": "observed", "backtest": {"kind": "model-ref"}},
},
# Forward PAPER TRACK for the 4-edge Bybit book — the OUT-OF-SAMPLE record on the venue we'll actually
# trade. Records the NAIVE equal-weight 4-sleeve book (tstrend/unlock/xsfunding/positioning) on
# `bybit_features` (the live risk overlay over-levers on Bybit; that is a separate re-tune, deliberately

View File

@@ -8,9 +8,8 @@ ONLY edits applied (no algorithm / math / constant changes):
* data root made configurable via `$FXHNT_SURFER_DATA_DIR` (default `/data/surfer`):
`pit_sweep.load()` reads `$FXHNT_SURFER_DATA_DIR/crypto_pit/*.npz`.
(The `surfer_poc` module — a value-returning `paper_step` refactor of `paper()` for the fxhnt
ForwardTracker's tail-managed VRP sleeve — was RETIRED with its dormant `MomentumVrpStrategy` caller;
Phase 0b Task 7c.)
(The `surfer_poc` module — a value-returning `paper_step` refactor of `paper()` for a since-retired
ForwardTracker sleeve — was RETIRED along with its dormant caller; Phase 0b Task 7c.)
torch is imported at module level (used in `signal_sweep.sharpe_t`/`validate`) and is kept — this is a
faithful port, not a numpy reimplementation.

View File

@@ -1,26 +0,0 @@
import datetime as dt
from fastapi.testclient import TestClient
from sqlalchemy.orm import Session
from fxhnt.adapters.persistence.cockpit_models import ForwardSummaryRow
from fxhnt.adapters.persistence.forward_nav import ForwardNavRepo
from fxhnt.adapters.web.app import create_app
def test_archived_vrp_detail_is_hidden(tmp_path):
"""vrp is ARCHIVED/FALSIFIED (shelved 2026-07-15): even with a persisted forward_summary row in the DB,
a direct `/strategy/vrp` hit must 404 (detail() returns None) and the page must never leak the VRP label
or its "Options income (put spreads)" alias. The code + OPRA data are kept; only the cockpit hides it."""
repo = ForwardNavRepo(f"sqlite:///{tmp_path/'c.db'}")
repo.migrate()
at = dt.datetime(2026, 7, 12)
with Session(repo._engine) as s:
s.add(ForwardSummaryRow(strategy_id="vrp", as_of="2026-07-12", days=8, nav=1.004,
total_return=0.004, sharpe=0.9, maxdd=-0.03, gate_status="WAIT",
gate_reason="building 8/21", src_updated_at=at))
s.commit()
r = TestClient(create_app(repo)).get("/strategy/vrp")
assert r.status_code == 404
assert "Equity VRP" not in r.text
assert "Options income" not in r.text

View File

@@ -9,6 +9,7 @@ from fxhnt.application.dashboard_service import DashboardService
from fxhnt.application.forward_models import BacktestSummary
from fxhnt.application.forward_models import ForwardNavRow as Row
from fxhnt.application.forward_models import ForwardSummary
from fxhnt.registry import STRATEGY_REGISTRY
def _seeded() -> ForwardNavRepo:
@@ -92,38 +93,31 @@ def test_fleet_no_backtest_renders_blank() -> None:
assert unlock.bt_as_of == ""
def test_archived_sleeves_hidden_everywhere() -> None:
"""The `archived: True` entries (vrp/vrp_exec — FALSIFIED/shelved 2026-07-15) are KEPT in the Python
registry (code + OPRA data preserved, entry not deleted) AND still seeded into the DB registry table by
`_seed_registry` — but `ForwardNavRepo.registry()` (the SINGLE authoritative filter) excludes them, so
every registry-driven render path drops them: the fleet Overview, per-strategy gate specs, and
`detail()`. Proven here at the repo, fleet, and detail levels."""
from fxhnt.adapters.persistence.cockpit_models import StrategyRegistryRow
from fxhnt.registry import STRATEGY_REGISTRY
from sqlalchemy import select as _select
from sqlalchemy.orm import Session as _Session
# entries KEPT in the Python registry, flagged archived (not deleted)
assert STRATEGY_REGISTRY["vrp"].get("archived") is True
assert STRATEGY_REGISTRY["vrp_exec"].get("archived") is True
repo = _seeded()
# the DB row IS still seeded (code + data kept) — a raw table read finds it...
with _Session(repo._engine) as s:
raw_ids = {r.strategy_id for r in s.scalars(_select(StrategyRegistryRow))}
assert {"vrp", "vrp_exec"} <= raw_ids, "archived rows must still be seeded in the DB (not pruned)"
# ...but the cockpit-facing registry() EXCLUDES them (the authoritative hide).
reg_ids = {r.strategy_id for r in repo.registry()}
assert "vrp" not in reg_ids and "vrp_exec" not in reg_ids
assert "unlock" in reg_ids # a non-archived track is still visible (the filter is targeted)
# fleet Overview drops them; a non-archived track still renders
fleet_ids = {f.strategy_id for f in DashboardService(repo).fleet()}
assert "vrp" not in fleet_ids and "vrp_exec" not in fleet_ids
assert "unlock" in fleet_ids
# a direct /strategy/vrp hit returns None (route 404s), but a live track still resolves
def test_archived_sleeves_hidden_everywhere(monkeypatch) -> None:
# A SYNTHETIC archived registry entry (`STRATEGY_REGISTRY` `archived: True` — the pattern
# `test_paper_books_web.py` uses for its synthetic `archivedstrat`/`newstrat`, in place of the deleted
# vrp-based test) must be excluded from every cockpit surface `ForwardNavRepo.registry()` feeds:
# `DashboardService.fleet()` and `.detail()`. Monkeypatch BEFORE `migrate()` so `_seed_registry()` (which
# seeds the DB registry table FROM `STRATEGY_REGISTRY` and prunes anything not present there) picks up
# the synthetic entry.
monkeypatch.setitem(STRATEGY_REGISTRY, "archivedstrat",
{"archived": True, "display_name": "Options income (put spreads)",
"sleeve": "equity-vol", "gate_spec": {}})
repo = _seeded() # seeds registry AFTER the monkeypatch above, so archivedstrat lands in the DB row set
svc = DashboardService(repo)
assert svc.detail("vrp") is None
assert svc.detail("vrp_exec") is None
# registry() itself: the archived row never comes back, a non-archived row (unlock) does.
reg_ids = {r.strategy_id for r in repo.registry()}
assert "archivedstrat" not in reg_ids
assert "unlock" in reg_ids
# fleet(): archived sleeve excluded; a live sleeve still shows.
fleet = svc.fleet()
assert all(f.strategy_id != "archivedstrat" for f in fleet)
assert any(f.strategy_id == "unlock" for f in fleet)
# detail(): direct hit on the archived id returns None (404 at the route layer); a live id still resolves.
assert svc.detail("archivedstrat") is None
assert svc.detail("unlock") is not None

View File

@@ -1,152 +0,0 @@
"""DeribitDVOL adapter — the Deribit DVOL annualised implied-vol index (the IV leg of the VRP edge).
ALL HTTP is injected/mocked — no network in these tests. The adapter takes a `fetch(url)->dict` callable
(the JSON endpoint), so the tests drive it with canned paginated responses and assert: the {epoch_day:
dvol_close} mapping (close = row index 4), backward pagination via end_timestamp to a floor, the floor stop,
the empty-page stop, dedupe across page boundaries, and rate-limit/transient backoff.
"""
from __future__ import annotations
import datetime as _dt
from fxhnt.adapters.data.deribit_dvol import DERIBIT_DVOL_FLOOR_MS, DeribitDVOL
_DAY_MS = 86_400_000
# DETERMINISM SEAM: pin the adapter's wall clock to a fixed recent instant so the backward walk anchors at a
# constant `now` regardless of the calendar date the suite runs on (mirrors test_deribit_funding). If the
# adapter ever reverts to a bare time.time() in the loop, the clock-determinism guard below fails.
_NOW_MS = 1780315200_000 # noon 2026-06-01 UTC, in ms
def _FIXED_CLOCK() -> float: # noqa: N802 — constant-like injectable clock
return _NOW_MS / 1000.0
def _dvol_row(ts_ms: int, close: float) -> list:
# Deribit volatility-index rows: [ts_ms, open, high, low, close]. close (idx 4) is the DVOL value.
return [ts_ms, close - 1.0, close + 1.0, close - 2.0, close]
class _PagedDVOL:
"""Mock of /public/get_volatility_index_data: serves rows oldest->newest within the requested
[start_timestamp, end_timestamp] window, honoring the end_timestamp cursor like Deribit (returns up to
`limit` rows ending at end_timestamp, oldest-first). Walking end_timestamp back past the oldest row
returns an empty list (stops the loop)."""
def __init__(self, rows: list[list], *, limit: int = 1000) -> None:
self._rows = sorted(rows, key=lambda r: r[0])
self._limit = limit
self.calls: list[tuple[int | None, int | None]] = []
def fetch(self, url: str) -> dict:
start = _parse_qs_int(url, "start_timestamp")
end = _parse_qs_int(url, "end_timestamp")
self.calls.append((start, end))
cand = [r for r in self._rows
if (start is None or r[0] >= start) and (end is None or r[0] <= end)]
# newest `limit` rows in the window (Deribit caps 1000/request, returns the most recent of the window)
page = cand[-self._limit:]
return {"result": {"data": page}}
def _parse_qs_int(url: str, key: str) -> int | None:
from urllib.parse import parse_qs, urlparse
q = parse_qs(urlparse(url).query)
return int(q[key][0]) if key in q else None
def test_dvol_history_maps_epoch_day_to_close_and_paginates_backward():
rows = [
_dvol_row(100 * _DAY_MS, 60.0),
_dvol_row(99 * _DAY_MS, 55.0),
_dvol_row(98 * _DAY_MS, 48.0),
_dvol_row(97 * _DAY_MS, 52.0),
]
paged = _PagedDVOL(rows, limit=2)
dv = DeribitDVOL(fetch=paged.fetch, sleep=lambda _: None)
daily = dv.dvol_history("BTC", start_ms=0)
assert daily == {100: 60.0, 99: 55.0, 98: 48.0, 97: 52.0}
assert len(paged.calls) >= 2 # it paginated and walked end_timestamp backward
def test_dvol_history_url_carries_currency_and_resolution():
seen: list[str] = []
def fetch(url: str) -> dict:
seen.append(url)
return {"result": {"data": []}}
dv = DeribitDVOL(fetch=fetch, sleep=lambda _: None)
dv.dvol_history("ETH")
assert "get_volatility_index_data" in seen[0]
assert "currency=ETH" in seen[0]
assert "resolution=86400" in seen[0]
def test_dvol_history_empty_page_stops_loop():
paged = _PagedDVOL([], limit=1000)
dv = DeribitDVOL(fetch=paged.fetch, sleep=lambda _: None)
assert dv.dvol_history("BTC") == {}
assert len(paged.calls) == 1
def test_dvol_history_respects_start_ms_floor():
rows = [_dvol_row(10 * _DAY_MS, 50.0), _dvol_row(20 * _DAY_MS, 60.0)]
paged = _PagedDVOL(rows, limit=1000)
dv = DeribitDVOL(fetch=paged.fetch, sleep=lambda _: None)
daily = dv.dvol_history("BTC", start_ms=15 * _DAY_MS)
assert 20 in daily and 10 not in daily # only the row at/after the floor survives
def test_default_floor_is_2021_03():
# The floor sits just below the true BTC/ETH DVOL inception (2021-03-24); the old 2023-09 floor silently
# discarded ~890 days of valid history.
import datetime as dt
assert int(dt.datetime(2021, 3, 1, tzinfo=dt.UTC).timestamp() * 1000) == DERIBIT_DVOL_FLOOR_MS
def test_dvol_history_dedups_across_page_boundaries():
rows = [_dvol_row(d * _DAY_MS, 40.0 + d) for d in range(50, 55)]
paged = _PagedDVOL(rows, limit=2)
dv = DeribitDVOL(fetch=paged.fetch, sleep=lambda _: None)
daily = dv.dvol_history("BTC", start_ms=0)
assert set(daily) == {50, 51, 52, 53, 54}
def test_dvol_history_is_clock_deterministic():
# The first page's end_timestamp anchors at the INJECTED clock, not the wall clock: the URL of the first
# request is identical across runs when a fixed clock is supplied — proving no bare time.time() leaks into
# the loop (the determinism bug the clock seam prevents).
seen_a: list[str] = []
seen_b: list[str] = []
DeribitDVOL(fetch=lambda u: (seen_a.append(u), {"result": {"data": []}})[1],
sleep=lambda _: None, clock=_FIXED_CLOCK).dvol_history("BTC")
DeribitDVOL(fetch=lambda u: (seen_b.append(u), {"result": {"data": []}})[1],
sleep=lambda _: None, clock=_FIXED_CLOCK).dvol_history("BTC")
assert seen_a == seen_b
assert f"end_timestamp={_NOW_MS}" in seen_a[0] # anchored at the fixed clock, not today
assert int(_FIXED_CLOCK()) == 1780315200 # frozen anchor, never tracks today
def test_default_floor_matches_2021_03():
assert int(_dt.datetime(2021, 3, 1, tzinfo=_dt.UTC).timestamp() * 1000) == DERIBIT_DVOL_FLOOR_MS
def test_dvol_history_retries_on_transient_error_then_succeeds():
d = 30 * _DAY_MS
real = _PagedDVOL([_dvol_row(d, 70.0)], limit=1000)
state = {"n": 0}
def flaky(url: str) -> dict:
state["n"] += 1
if state["n"] <= 2:
raise RuntimeError("transient 502")
return real.fetch(url)
waited: list[float] = []
dv = DeribitDVOL(fetch=flaky, sleep=waited.append, backoff_base=0.5, backoff_tries=5)
daily = dv.dvol_history("BTC", start_ms=0)
assert abs(daily[30] - 70.0) < 1e-12
assert state["n"] >= 3
assert waited[:2] == [0.5, 1.0] # exponential backoff (no real sleep)

View File

@@ -1,79 +0,0 @@
"""ingest_dvol — write the Deribit DVOL index into bybit_features as feature 'dvol' for BTCUSDT/ETHUSDT.
The DVOL index is currency-keyed (BTC/ETH); the warehouse panels are symbol-keyed (BTCUSDT/ETHUSDT), so the
ingest maps currency -> symbol and writes feature='dvol' at ts = epoch_day*86400. ONLY 'dvol' is written
(close is the kline ingest's job). Idempotent/resumable. No network — the adapter is a stub.
"""
from __future__ import annotations
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
from fxhnt.application.dvol_ingest import ingest_dvol
_DAY = 86_400
class _StubDeribit:
"""Stub DeribitDVOL: serves a canned {epoch_day: dvol} per currency, records the calls."""
def __init__(self, by_currency: dict[str, dict[int, float]]) -> None:
self._by_currency = by_currency
self.calls: list[str] = []
def dvol_history(self, currency: str, *, start_ms: int | None = None) -> dict[int, float]:
self.calls.append(currency)
return dict(self._by_currency.get(currency, {}))
def test_ingest_writes_dvol_for_btc_and_eth() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
deribit = _StubDeribit({"BTC": {100: 55.0, 101: 60.0}, "ETH": {100: 70.0, 101: 72.0}})
summary = ingest_dvol(store, deribit)
# BTCUSDT/ETHUSDT now carry the 'dvol' feature at the right timestamps.
btc = store.read_panel(["BTCUSDT"], "dvol")["BTCUSDT"]
eth = store.read_panel(["ETHUSDT"], "dvol")["ETHUSDT"]
store.close()
assert btc == {100: 55.0, 101: 60.0}
assert eth == {100: 70.0, 101: 72.0}
assert summary["symbols"] == 2
assert summary["day_rows"] == 4
assert summary["oldest_day"] == 100
assert set(deribit.calls) == {"BTC", "ETH"}
def test_ingest_maps_currency_to_usdt_symbol() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
deribit = _StubDeribit({"BTC": {1: 50.0}})
ingest_dvol(store, deribit, currencies=("BTC",))
# The row lands under BTCUSDT at ts = epoch_day*86400, not under "BTC".
rows = store.read_features("BTCUSDT", features=["dvol"])
store.close()
assert rows == [(1 * _DAY, {"dvol": 50.0})]
def test_ingest_skips_currency_with_no_data() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
deribit = _StubDeribit({"BTC": {1: 50.0}}) # ETH returns nothing
summary = ingest_dvol(store, deribit)
store.close()
assert summary["symbols"] == 1 # only BTC produced rows; ETH skipped (never aborts)
def test_ingest_is_idempotent() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
deribit = _StubDeribit({"BTC": {1: 50.0}, "ETH": {1: 60.0}})
ingest_dvol(store, deribit)
ingest_dvol(store, deribit) # re-run overwrites in place, no duplicate rows
btc = store.read_features("BTCUSDT", features=["dvol"])
store.close()
assert btc == [(1 * _DAY, {"dvol": 50.0})]
def test_ingest_only_writes_dvol_feature() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
deribit = _StubDeribit({"BTC": {1: 50.0}})
ingest_dvol(store, deribit, currencies=("BTC",))
rows = store.read_features("BTCUSDT")
store.close()
# exactly one feature written, and it's 'dvol' (close/funding are the other ingests' job).
feats = {name for _, feats in rows for name in feats}
assert feats == {"dvol"}

View File

@@ -5,8 +5,11 @@ from __future__ import annotations
import datetime as dt
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from fxhnt.adapters.persistence.cockpit_models import CockpitBase, XspOptionBarRow
from fxhnt.adapters.persistence.forward_nav import ForwardNavRepo
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
from fxhnt.application.forward_health import (
BROKEN,
HEALTHY,
@@ -97,61 +100,88 @@ def _seed_summary(repo: ForwardNavRepo, sid: str, as_of: str, at: dt.datetime) -
gate_status="WAIT", gate_reason="building", at=at)
def test_evaluate_health_persists_and_catches_stale_freeze(tmp_path):
def _seed_freeze_bars(dsn: str, rows: list[dict], at: dt.datetime) -> None:
"""Insert rows directly into the kept `xsp_option_bars` table (no VRP repo class involved) — mirrors
what the retired XspOptionBarsRepo.migrate()/upsert_bars() used to do, for the freeze-staleness axis."""
kw = {} if dsn.startswith("postgresql") else {"connect_args": {"check_same_thread": False}}
engine = create_engine(dsn, **kw)
CockpitBase.metadata.create_all(engine)
with Session(engine) as s:
for r in rows:
s.merge(XspOptionBarRow(
osi_symbol=r["osi_symbol"], date=dt.date.fromisoformat(r["date"]),
expiry=dt.date.fromisoformat(r["expiry"]), strike=float(r["strike"]),
right=r["right"], close=float(r["close"]), src_updated_at=at))
s.commit()
def _freeze_track_entry() -> dict:
"""A synthetic opra-pit-sourced registry entry — VRP was the only real track ever wired to the freeze
axis (`_FREEZE_SOURCES = {"opra-pit"}`), and it is now fully deleted, but the axis itself is strategy-
agnostic and must stay covered generically (per-track monkeypatch, mirrors test_forward_definition.py's
`t_ov` pattern)."""
return {"display_name": "Freeze-axis test track", "sleeve": "test",
"gate_spec": {"gate_type": "reconciliation"},
"definition": {"params": {"source": "opra-pit"}, "record_mode": "recompute"}}
def test_evaluate_health_persists_and_catches_stale_freeze(tmp_path, monkeypatch):
from fxhnt.registry import STRATEGY_REGISTRY
monkeypatch.setitem(STRATEGY_REGISTRY, "t_freeze", _freeze_track_entry())
dsn = f"sqlite:///{tmp_path}/health.db"
at = dt.datetime(2026, 7, 14, 23, 30)
repo = ForwardNavRepo(dsn)
repo.migrate()
# vrp: fresh forward summary but a STALE frozen table (the incident) → must read STALE.
_seed_summary(repo, "vrp", "2026-07-13", at)
# t_freeze: fresh forward summary but a STALE frozen table (the incident) → must read STALE.
_seed_summary(repo, "t_freeze", "2026-07-13", at)
# sixtyforty: recompute, fresh, absolute gate → HEALTHY.
_seed_summary(repo, "sixtyforty", "2026-07-13", at)
# unlock: fresh forward data (isolates the NO_REF path below from the separate no-anchor/no-data STALE
# check — this test seeds no forward_anchor rows at all, so without real forward data unlock would hit
# that check first).
_seed_summary(repo, "unlock", "2026-07-13", at)
# seed the vrp freeze table with a max date weeks behind the run date.
xsp = XspOptionBarsRepo(dsn)
xsp.migrate()
xsp.upsert_bars([{"osi_symbol": "XSP 260620P00470000", "date": "2026-06-15", "expiry": "2026-07-17",
"strike": 470.0, "right": "P", "close": 1.0}], at)
# seed the freeze table with a max date weeks behind the run date.
_seed_freeze_bars(dsn, [{"osi_symbol": "XSP 260620P00470000", "date": "2026-06-15",
"expiry": "2026-07-17", "strike": 470.0, "right": "P", "close": 1.0}], at)
verdicts = {v.strategy_id: v for v in evaluate_health(dsn, at=at)}
assert verdicts["vrp"].health == STALE and verdicts["vrp"].stale_days > 3
assert "freeze table" in verdicts["vrp"].reason
assert verdicts["t_freeze"].health == STALE and verdicts["t_freeze"].stale_days > 3
assert "freeze table" in verdicts["t_freeze"].reason
assert verdicts["sixtyforty"].health == HEALTHY
# unlock is a reconciliation recompute track with no backtest ref seeded → NO_REF (M6 invariant).
assert verdicts["unlock"].health == NO_REF
# persisted to strategy_health, readable by the dashboard.
persisted = {h.strategy_id: h for h in repo.all_health()}
assert persisted["vrp"].health == STALE and persisted["vrp"].stale_days > 3
assert persisted["t_freeze"].health == STALE and persisted["t_freeze"].stale_days > 3
def test_evaluate_health_replaces_snapshot_on_recovery(tmp_path):
def test_evaluate_health_replaces_snapshot_on_recovery(tmp_path, monkeypatch):
"""A track that recovers must CLEAR its old STALE row (replace-all semantics)."""
from fxhnt.application.forward_models import BacktestSummary
from fxhnt.registry import STRATEGY_REGISTRY
monkeypatch.setitem(STRATEGY_REGISTRY, "t_freeze", _freeze_track_entry())
dsn = f"sqlite:///{tmp_path}/rec.db"
repo = ForwardNavRepo(dsn)
repo.migrate()
xsp = XspOptionBarsRepo(dsn)
xsp.migrate()
# run 1: freeze stale → vrp STALE
# run 1: freeze stale → t_freeze STALE
at1 = dt.datetime(2026, 7, 14, 23, 30)
# a backtest ref so a recovered vrp resolves to HEALTHY (not NO_REF) — isolates the STALE→clear behavior.
repo.upsert_backtest_summary(BacktestSummary(strategy_id="vrp", as_of="2026-06-01", cagr=0.2, ann_vol=0.1,
sharpe=1.0, max_drawdown=-0.05, passed=True, dsr=0.9, is_sharpe=1.0,
oos_sharpe=0.9, pvalue=0.01), at1)
_seed_summary(repo, "vrp", "2026-07-13", at1)
xsp.upsert_bars([{"osi_symbol": "X", "date": "2026-06-15", "expiry": "2026-07-17", "strike": 470.0,
"right": "P", "close": 1.0}], at1)
assert {v.strategy_id: v for v in evaluate_health(dsn, at=at1)}["vrp"].health == STALE
# run 2: freeze advances to yesterday → vrp HEALTHY, the old STALE row cleared
xsp.upsert_bars([{"osi_symbol": "Y", "date": "2026-07-13", "expiry": "2026-07-17", "strike": 470.0,
"right": "P", "close": 1.0}], at1)
_seed_summary(repo, "vrp", "2026-07-13", at1)
assert {v.strategy_id: v for v in evaluate_health(dsn, at=at1)}["vrp"].health == HEALTHY
assert {h.strategy_id: h for h in repo.all_health()}["vrp"].health == HEALTHY
# a backtest ref so a recovered t_freeze resolves to HEALTHY (not NO_REF) — isolates the STALE→clear
# behavior.
repo.upsert_backtest_summary(BacktestSummary(strategy_id="t_freeze", as_of="2026-06-01", cagr=0.2,
ann_vol=0.1, sharpe=1.0, max_drawdown=-0.05, passed=True, dsr=0.9,
is_sharpe=1.0, oos_sharpe=0.9, pvalue=0.01), at1)
_seed_summary(repo, "t_freeze", "2026-07-13", at1)
_seed_freeze_bars(dsn, [{"osi_symbol": "X", "date": "2026-06-15", "expiry": "2026-07-17",
"strike": 470.0, "right": "P", "close": 1.0}], at1)
assert {v.strategy_id: v for v in evaluate_health(dsn, at=at1)}["t_freeze"].health == STALE
# run 2: freeze advances to yesterday → t_freeze HEALTHY, the old STALE row cleared
_seed_freeze_bars(dsn, [{"osi_symbol": "Y", "date": "2026-07-13", "expiry": "2026-07-17",
"strike": 470.0, "right": "P", "close": 1.0}], at1)
_seed_summary(repo, "t_freeze", "2026-07-13", at1)
assert {v.strategy_id: v for v in evaluate_health(dsn, at=at1)}["t_freeze"].health == HEALTHY
assert {h.strategy_id: h for h in repo.all_health()}["t_freeze"].health == HEALTHY
# ------------------------------------------------------------------ LOUD cockpit badge (rendering)

View File

@@ -192,3 +192,24 @@ def test_clean_account_holdings_show_no_stray_warning() -> None:
# SPY + IEF are both book instruments -> no "outside the book" warning renders.
html = _client(_seed_forward(), _seed_ibkr()).get("/strategy/multistrat").text
assert "outside the book" not in html
def test_ibkr_reported_ucits_symbol_csbgu0_is_not_flagged_stray() -> None:
# The IEF UCITS line's config ticker is CBU0, but IBKR REPORTS the holding under its canonical symbol
# CSBGU0 (probe-verified 2026-07-20). A legitimate CSBGU0 book position must NOT be flagged "outside the
# book" / dropped from NLV — the expected set keys on `ibkr_symbol`, not just the display ticker. A genuine
# stray (IBIT) in the SAME account is still flagged.
fwd = _seed_forward()
ibkr = IbkrAccountRepo("sqlite://")
ibkr.migrate()
at = dt.datetime(2026, 7, 20, 15, 0)
ibkr.replace_snapshot("2026-07-20", nlv=1_000_000.0, cash=900_000.0, gross=100_000.0,
positions={"CSPX": 13.0, "CSBGU0": 55.0, "IBIT": 699.0}, at=at)
ibkr.record_fills(strategy_id="multistrat", rebalance_id="r1",
fills=[FillDTO(symbol="CSBGU0", side="BOT", qty=55.0, price=153.8, fee=1.0,
status="Filled", shortfall_bps=0.0)], at=at)
html = _client(fwd, ibkr).get("/strategy/multistrat").text
banner = re.search(r"Position outside the book:[^<]*", html)
assert banner is not None # IBIT is still a genuine stray
assert "IBIT" in banner.group(0)
assert "CSBGU0" not in banner.group(0) # the IEF book line is NOT flagged stray

View File

@@ -4,20 +4,24 @@ from the PRECOMPUTED `sim_curve_ret` table (never recomputed on the request path
measured-cost path's shape (curve + metrics + Live forward), with plain-language pill labels (never the raw
registry id "multistrat").
NOTE: vrp was ARCHIVED/FALSIFIED (shelved 2026-07-15) — its `STRATEGY_REGISTRY` entry carries
`archived: True`, so it is EXCLUDED from the Backtest book pills (`_SIM_BOOKS`) and a `?book=vrp` request
falls back to the default book. The recompute-replay machinery itself still supports vrp (code + OPRA data
kept); this test file now asserts vrp is HIDDEN from the sim UI (see the archived-book tests below).
NOTE: vrp has been FULLY REMOVED from the codebase (shelved 2026-07-15, code+data deleted in the
`chore/remove-vrp-code` cleanup) — there is no longer a `STRATEGY_REGISTRY` entry, `_SIM_BOOKS` pill, or
recompute-replay support for it at all. This test file uses a synthetic non-existent book id
(`no-such-book`) as an honest stand-in to probe the SAME fallback behavior an archived/unknown book must
still get: never selectable, never rendered, always a graceful fallback to the default book (see the
archived-book tests below).
Asserts:
* /paper/sim shows the multistrat pill labelled in plain language, not the raw id, and NO vrp pill;
* /paper/sim shows the multistrat pill labelled in plain language, not the raw id, and NO stray pill for an
unknown book;
* /paper/sim/run?book=multistrat renders a real curve + metrics from the precomputed sim_curve_ret series;
* capital scales the curve linearly (same contract as the Bybit measured path);
* the request path never recomputes the replay (a tripwire on ForwardStrategy.advance would fire if it did —
proven implicitly: the seeded repo carries no build_strategy at all, so a recompute is structurally
impossible; the curve still renders from sim_curve_ret alone);
* the Live forward section renders for multistrat (its own forward track), not just bybit_4edge;
* an archived book (vrp) is not selectable — the request falls back to the default book, never rendering vrp;
* an unknown/archived book id is not selectable — the request falls back to the default book, never
rendering that id;
* absent precompute -> the graceful 'precomputing' note (never a 500, never the Bybit-specific caption).
NO network — in-memory SQLite repos, seeded directly.
"""
@@ -47,7 +51,7 @@ def _seed_ibkr_curve(repo: PaperRepo, book: str, *, start: dt.date = dt.date(202
repo.upsert_sim_curve_ret(book, d, ret, at=at)
def _seeded_paper_repo(*, books: tuple[str, ...] = ("multistrat", "vrp")) -> PaperRepo:
def _seeded_paper_repo(*, books: tuple[str, ...] = ("multistrat", "no-such-book")) -> PaperRepo:
repo = PaperRepo("sqlite://")
repo.migrate()
for b in books:
@@ -89,16 +93,14 @@ def _sim_data(html: str) -> dict:
return json.loads(m.group(1))
def test_sim_page_shows_multistrat_pill_and_hides_archived_vrp() -> None:
def test_sim_page_shows_multistrat_pill_and_hides_unknown_book() -> None:
c = _client(_seeded_paper_repo())
html = c.get("/paper/sim").text
assert "ETF Portfolio" in html
# vrp is ARCHIVED — neither its pill label nor its raw id may appear anywhere on the Backtest page.
assert "Options income (put spreads)" not in html
assert "Equity VRP" not in html
assert 'data-book="vrp"' not in html and "book=vrp" not in html
# a book id absent from `_SIM_BOOKS` (unknown or archived) must never get a pill on the Backtest page.
assert 'data-book="no-such-book"' not in html and "book=no-such-book" not in html
# the raw registry id must never leak as pill TEXT (it is fine as a query-string value).
assert ">multistrat<" not in html and ">vrp<" not in html
assert ">multistrat<" not in html and ">no-such-book<" not in html
def test_multistrat_book_is_selectable_and_lazy_loads() -> None:
@@ -121,14 +123,14 @@ def test_multistrat_run_renders_curve_and_metrics_from_precomputed_sim_curve_ret
assert 80000 < data["equity"][0] < 120000 # curve starts near the configured capital
def test_archived_vrp_run_falls_back_and_never_renders_vrp() -> None:
"""An archived book (vrp) is not in `_SIM_BOOKS`, so `/paper/sim/run?book=vrp` falls back to the default
book (bybit_4edge) — it renders that book's view (or the graceful pending note), never vrp's curve/label."""
def test_unknown_book_run_falls_back_and_never_renders_it() -> None:
"""A book id absent from `_SIM_BOOKS` (unknown or archived) makes `/paper/sim/run?book=no-such-book`
fall back to the default book (bybit_4edge) — it renders that book's view (or the graceful pending
note), never the unknown id's curve/label."""
c = _client(_seeded_paper_repo())
r = c.get("/paper/sim/run", params={"book": "vrp", "capital": 100000})
r = c.get("/paper/sim/run", params={"book": "no-such-book", "capital": 100000})
assert r.status_code == 200 # graceful fallback, never a 500
assert "Options income (put spreads)" not in r.text and "Equity VRP" not in r.text
assert 'data-book="vrp"' not in r.text # the effective book is the default, not vrp
assert 'data-book="no-such-book"' not in r.text # the effective book is the default, not the unknown id
def test_ibkr_capital_scales_the_curve_linearly() -> None:
@@ -180,12 +182,12 @@ def test_ibkr_missing_precompute_shows_pending_note_not_bybit_caption() -> None:
assert 'data-cost-mode="flat"' not in r.text
def test_sim_book_switch_covers_visible_books_and_excludes_archived_vrp() -> None:
def test_sim_book_switch_covers_visible_books_and_excludes_unknown_book() -> None:
c = _client(_seeded_paper_repo())
# the three VISIBLE books each render their own page...
for book in ("bybit_4edge", "bybit_4edge_levered", "multistrat"):
r = c.get("/paper/sim", params={"book": book})
assert r.status_code == 200 and f'data-book="{book}"' in r.text
# ...but the ARCHIVED vrp is not selectable: it falls back to the default book (200, never vrp).
r = c.get("/paper/sim", params={"book": "vrp"})
assert r.status_code == 200 and 'data-book="vrp"' not in r.text
# ...but an unknown/archived book id is not selectable: it falls back to the default book (200, never it).
r = c.get("/paper/sim", params={"book": "no-such-book"})
assert r.status_code == 200 and 'data-book="no-such-book"' not in r.text

View File

@@ -7,9 +7,10 @@ one row per strategy trading on the shared real account — a dimmed "not starte
registered-but-not-yet-executing strategy (a synthetic `newstrat` stands in for that here), and the table
SCALES BY ROW (proven by a synthetic second executing strategy, not a redesign).
NOTE: vrp was ARCHIVED/FALSIFIED (shelved 2026-07-15) — it is EXCLUDED from the account breakdown
(`_active_ibkr_account_sids`), so it never renders a row / its "Options income (put spreads)" label even
though its fills stay captured in the repo. These tests now prove vrp is ABSENT wherever it used to show.
NOTE: an ARCHIVED registry entry is EXCLUDED from the account breakdown (`_active_ibkr_account_sids`), so it
never renders a row even though its fills stay captured in the repo (VRP was the original real-world case —
ARCHIVED/FALSIFIED, shelved 2026-07-15 — but the code itself has since been fully removed; the exclusion
mechanism is proven here with a synthetic archived entry instead).
Plain-language HARD RULE (Task 3/4 precedent): no raw registry id / `bps` / "vs sim" / "divergence" may ever
appear in the rendered `/paper` landing — asserted against the VISIBLE text.
@@ -31,6 +32,7 @@ from fxhnt.application.dashboard_service import DashboardService
from fxhnt.application.exec_capture import FillDTO
from fxhnt.application.forward_models import ForwardSummary
from fxhnt.ports.live_price import FakeLivePrice
from fxhnt.registry import STRATEGY_REGISTRY
def _visible_text(html: str) -> str:
@@ -89,16 +91,20 @@ def _seed_ibkr(*, extra_fills_for: tuple[str, ...] = ()) -> IbkrAccountRepo:
def test_ibkr_account_view_one_executing_row_and_dimmed_not_started_row(monkeypatch) -> None:
# vrp is ARCHIVED -> excluded from the account breakdown. A synthetic non-archived `newstrat` (registered
# against the account but with NO captured fills) stands in as the dimmed "not started yet" row, so the
# one-executing / one-dimmed shape is still covered — and vrp is proven ABSENT.
# A synthetic `archivedstrat` (registry entry with archived=True) is EXCLUDED from the account
# breakdown. A synthetic non-archived `newstrat` (registered against the account but with NO captured
# fills) stands in as the dimmed "not started yet" row, so the one-executing / one-dimmed shape is still
# covered — and the archived book is proven ABSENT.
monkeypatch.setitem(STRATEGY_REGISTRY, "archivedstrat",
{"archived": True, "display_name": "Options income (put spreads)",
"sleeve": "equity-vol", "gate_spec": {}})
monkeypatch.setattr(dashboard_service, "_IBKR_ACCOUNT_SIDS",
frozenset({"multistrat", "vrp", "newstrat"}))
frozenset({"multistrat", "archivedstrat", "newstrat"}))
svc = DashboardService(_seed_forward(), _seed_ibkr()) # only multistrat has fills; newstrat does not
view = svc.ibkr_account_view()
assert view is not None
ids = {r.strategy_id for r in view.per_strategy}
assert ids == {"multistrat", "newstrat"} # archived vrp is excluded
assert ids == {"multistrat", "newstrat"} # archived entry is excluded
multistrat = next(r for r in view.per_strategy if r.strategy_id == "multistrat")
newstrat = next(r for r in view.per_strategy if r.strategy_id == "newstrat")
assert multistrat.executing is True
@@ -117,16 +123,19 @@ def test_ibkr_account_view_one_executing_row_and_dimmed_not_started_row(monkeypa
def test_ibkr_account_view_scale_test_adds_new_executing_row_no_redesign(monkeypatch) -> None:
# Seed a SYNTHETIC third strategy (not in STRATEGY_REGISTRY) trading on the same account, and register
# it against the account via the module constant — proves the breakdown scales by ROW, not a redesign.
monkeypatch.setitem(STRATEGY_REGISTRY, "archivedstrat",
{"archived": True, "display_name": "Options income (put spreads)",
"sleeve": "equity-vol", "gate_spec": {}})
monkeypatch.setattr(dashboard_service, "_IBKR_ACCOUNT_SIDS",
frozenset({"multistrat", "vrp", "newstrat"}))
frozenset({"multistrat", "archivedstrat", "newstrat"}))
fwd = _seed_forward(extra_exec_sids=("newstrat",))
ibkr = _seed_ibkr(extra_fills_for=("newstrat",))
svc = DashboardService(fwd, ibkr)
view = svc.ibkr_account_view()
assert view is not None
# archived vrp is excluded; the visible set is the two live account books (multistrat + synthetic newstrat)
# archived entry is excluded; the visible set is the two live account books (multistrat + synthetic newstrat)
assert len(view.per_strategy) == 2
assert all(r.strategy_id != "vrp" for r in view.per_strategy)
assert all(r.strategy_id != "archivedstrat" for r in view.per_strategy)
executing_ids = {r.strategy_id for r in view.per_strategy if r.executing}
assert executing_ids == {"multistrat", "newstrat"}
newstrat = next(r for r in view.per_strategy if r.strategy_id == "newstrat")

View File

@@ -1,7 +1,7 @@
"""The nightly `_persist_track_backtest_ref` step (fxhnt-backtest-refs Job) persists a recompute-replay
strategy's (multistrat/vrp — the IBKR paper books) FULL inception return series into the `sim_curve_ret`
table, so /paper/sim reads a precomputed IBKR curve instead of ever re-running `ForwardStrategy.advance` on a
web request (Task 1/D1). Asserts:
strategy's (multistrat/multistrat_levered — the IBKR paper books) FULL inception return series into the
`sim_curve_ret` table, so /paper/sim reads a precomputed IBKR curve instead of ever re-running
`ForwardStrategy.advance` on a web request (Task 1/D1). Asserts:
* the series persisted is the SAME rows the reconciliation-gate `backtest_summary` ref is derived from
(one replay call, two writes);
* `PaperRepo.sim_curve_returns(strategy_id)` reads it back as {epoch_day: ret};
@@ -54,15 +54,15 @@ def test_persist_track_backtest_ref_also_writes_the_reconciliation_ref(tmp_path,
dsn = f"sqlite:///{tmp_path / 'sim_curve2.db'}"
monkeypatch.setattr("fxhnt.config.get_settings", lambda: Settings(operational_dsn=dsn))
assets._persist_track_backtest_ref(build_op_context(), "vrp_nav", lambda: _Replay())
assets._persist_track_backtest_ref(build_op_context(), "multistrat_levered_nav", lambda: _Replay())
fwd = ForwardNavRepo(dsn)
fwd.migrate()
assert any(b.strategy_id == "vrp" for b in fwd.all_backtest_summaries())
assert any(b.strategy_id == "multistrat_levered" for b in fwd.all_backtest_summaries())
repo = PaperRepo(dsn)
repo.migrate()
assert repo.sim_curve_returns("vrp") # non-empty
assert repo.sim_curve_returns("multistrat_levered") # non-empty
def test_persist_track_backtest_ref_idempotent_on_rerun(tmp_path, monkeypatch):

View File

@@ -1,452 +0,0 @@
"""READ-ONLY, TAIL-HONEST evaluator for the DEFINED-RISK VRP — the deployable form of the vol-risk premium.
The naked short-variance VRP has a real gross edge but FAILS the deploy gate on an UNBOUNDED left tail
(modelled maxDD ~74%). The fix: sell the ATM straddle but BUY OTM wings so the max loss is BOUNDED (an iron
condor). The legs are priced from DVOL via Black-Scholes (we have no option chains — documented caveat).
Tests assert:
* the BOUNDED MAX LOSS — a huge move loses ≤ the defined max (wing_width credit), NOT unbounded;
* a calm expiry keeps (most of) the credit; the wing cost reduces the credit vs a naked straddle;
* the capped curve has a DRAMATICALLY shallower maxDD than the naked variance swap on the SAME spike fixture
(the wings work — this is the whole point);
* verify is cost-monotone (incl 50/100bp option costs), per-year, worst-day, maxDD, corr-vs-4-edges;
* walk-forward OOS + look-ahead-clean; tail-survivable PASSES;
* READ-ONLY (a write-tripwire never trips);
* the CLI smoke prints metrics + verify + walk-forward (in-memory store, NO network).
"""
from __future__ import annotations
import math
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
from fxhnt.application import vrp_eval
from fxhnt.application.vrp_defined_risk_eval import (
_N_LEGS,
_defined_risk_daily_cost,
_defined_risk_sized_series,
defined_risk_credit,
defined_risk_max_loss,
defined_risk_metrics_from_store,
defined_risk_payoff,
defined_risk_returns_from_store,
look_ahead_audit_defined_risk_ok,
verify_defined_risk_edge,
walk_forward_defined_risk,
)
_DAY = 86_400
_START = 19_358 # 2023-01-01 epoch day
class _WriteTripwireStore:
"""Read-only proxy: forwards reads, but ANY write/persist call trips an assertion."""
_FORBIDDEN = frozenset({
"write_features", "write_features_bulk", "upsert_feature_rows", "upsert_membership",
"replace_positions", "upsert_nav", "upsert_sleeve_ret", "replace_shadow_positions",
"replace_trades", "_create_schema", "_bulk_upsert",
})
def __init__(self, inner: TimescaleFeatureStore) -> None:
object.__setattr__(self, "_inner", inner)
def __getattr__(self, name: str):
if name in _WriteTripwireStore._FORBIDDEN:
raise AssertionError(f"READ-ONLY violation: evaluator called write method {name!r}")
return getattr(self._inner, name)
# --- 1. the defined-risk spread: credit + BOUNDED payoff ---------------------------------------
def test_credit_is_positive_and_a_fraction_of_spot() -> None:
# selling the ATM straddle collects more than the OTM wings cost -> a positive credit, sane in magnitude.
c = defined_risk_credit(s=30_000.0, iv=0.60, t=30.0 / 365.0, wing_width=0.15)
assert 0.0 < c < 0.20 # a fraction of spot, smaller than the wing distance
def test_wing_cost_reduces_credit_vs_naked_straddle() -> None:
# the naked straddle credit (no wings) = w -> infinity limit; buying wings (finite w) costs premium, so the
# defined-risk credit is STRICTLY LESS than the naked straddle premium. A WIDER wing is cheaper -> larger
# credit (approaching the naked premium).
s, iv, t = 30_000.0, 0.60, 30.0 / 365.0
narrow = defined_risk_credit(s=s, iv=iv, t=t, wing_width=0.10)
wide = defined_risk_credit(s=s, iv=iv, t=t, wing_width=0.30)
naked = defined_risk_credit(s=s, iv=iv, t=t, wing_width=0.95) # ~no wing cost -> ~naked straddle premium
assert narrow < wide < naked
def test_max_loss_is_bounded_the_core_property() -> None:
# THE CORE PROPERTY: a catastrophic move loses AT MOST the defined max (wing_width credit), NOT unbounded.
credit = defined_risk_credit(s=30_000.0, iv=0.60, t=30.0 / 365.0, wing_width=0.15)
defined_max = defined_risk_max_loss(credit=credit, wing_width=0.15)
# a 10%, 50%, 90%, 99% crash all lose the SAME bounded amount once past the wing.
for move in (-0.50, -0.90, -0.99, +0.95):
pnl = defined_risk_payoff(credit=credit, realized_move=move, wing_width=0.15)
assert pnl >= -defined_max - 1e-12 # never worse than the defined max loss
assert math.isclose(pnl, -defined_max, abs_tol=1e-9) # past the wing -> exactly the cap
def test_naked_straddle_loss_is_unbounded_but_capped_is_not() -> None:
# contrast: with NO wings the loss grows with the move (unbounded); with wings it plateaus at the cap.
credit = 0.05
capped_small = defined_risk_payoff(credit=credit, realized_move=-0.20, wing_width=0.15)
capped_huge = defined_risk_payoff(credit=credit, realized_move=-0.80, wing_width=0.15)
assert math.isclose(capped_small, capped_huge, abs_tol=1e-12) # capped: same loss past the wing
# the implicit naked short straddle (credit |m|) keeps getting worse.
naked_small = credit - 0.20
naked_huge = credit - 0.80
assert naked_huge < naked_small # naked: unbounded
def test_calm_expiry_keeps_the_credit() -> None:
# a small realized move (well inside the wings) -> pnl ≈ credit |m|, still positive when |m| < credit.
credit = 0.05
pnl = defined_risk_payoff(credit=credit, realized_move=0.01, wing_width=0.15)
assert math.isclose(pnl, credit - 0.01, abs_tol=1e-12)
assert pnl > 0.0
def test_long_vol_direction_negates() -> None:
credit = 0.04
short = defined_risk_payoff(credit=credit, realized_move=-0.30, wing_width=0.12, direction="short_vol")
long_ = defined_risk_payoff(credit=credit, realized_move=-0.30, wing_width=0.12, direction="long_vol")
assert math.isclose(long_, -short, rel_tol=1e-12)
# --- 2. fixtures (mirror the naked evaluator's, with a tradeable defined-risk premium) ----------
def _seed_calm_premium(store: TimescaleFeatureStore, *, days: int = 400, dvol: float = 60.0,
daily_move: float = 0.004) -> None:
"""Calm: IV high (dvol=60), tiny realized oscillation -> the defined-risk condor harvests its credit."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
sign = 1.0 if i % 2 == 0 else -1.0
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol})])
px[s] *= (1.0 + sign * daily_move)
def _seed_with_spike(store: TimescaleFeatureStore, *, days: int = 400, spike_at: int = 200,
crash: float = 0.45) -> None:
"""Calm condor premium most days, but ONE catastrophic crash that would blow up a NAKED short straddle;
the wings cap the defined-risk loss at the spread width. Used for the naked-vs-defined tail comparison."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": 50.0})])
if i == spike_at:
px[s] *= (1.0 - crash)
else:
px[s] *= (1.0 + (0.003 if i % 2 == 0 else -0.003))
def _seed_tail_survivable(store: TimescaleFeatureStore, *, days: int = 700) -> None:
"""A defined-risk premium that survives its tails: calm harvest, a moderate spike every ~90 days. DVOL
varies day to day (so the causality audit can distinguish prior-day from same-day IV)."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
dvol = 60.0 + 8.0 * math.sin(i * 0.11)
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol, "turnover": 5e8})])
if i % 90 == 45:
px[s] *= (1.0 - 0.06)
else:
px[s] *= (1.0 + (0.004 if i % 2 == 0 else -0.004))
def _seed_varying_dvol(store: TimescaleFeatureStore, *, days: int = 120) -> None:
"""Premium fixture with monotone time-varying DVOL so the causal and leaked credit-pricing mappings are
distinguishable on every day (the look-ahead audit can only bite when entry-day and expiry IV differ)."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
dvol = 50.0 + 0.137 * i
sign = 1.0 if i % 2 == 0 else -1.0
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol})])
px[s] *= (1.0 + sign * 0.004)
# --- 3. the book series: harvest, sizing, read-only --------------------------------------------
def test_defined_risk_returns_positive_on_calm_premium() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
series = defined_risk_returns_from_store(store, cost_bps=0.0, direction="short_vol")
store.close()
assert len(series) > 2
assert sum(series.values()) > 0.0 # the condor credit is harvested gross
def test_defined_risk_metrics_is_read_only() -> None:
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(inner)
store = _WriteTripwireStore(inner)
m = defined_risk_metrics_from_store(store, cost_bps=5.5)
inner.close()
assert m["available"] is True
assert m["wing_width"] > 0.0
def test_defined_risk_metrics_reports_tail() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
m = defined_risk_metrics_from_store(store, cost_bps=0.0)
store.close()
assert m["available"] is True
assert m["max_dd"] <= 0.0
assert "worst_day" in m and "avg_leverage" in m
# --- 4. THE TAIL COMPARISON: defined-risk maxDD << naked maxDD on the SAME spike ---------------
def test_defined_risk_maxdd_dramatically_shallower_than_naked_on_spike() -> None:
"""THE WHOLE POINT. On the SAME catastrophic-spike fixture, the defined-risk (wings) curve must have a
DRAMATICALLY shallower maxDD than the naked variance-swap curve — the wings cap the tail the naked book
cannot. Both are vol-targeted+costed by the identical harness; only the construction differs."""
from fxhnt.application.vrp_eval import vrp_metrics_from_store
naked_store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_with_spike(naked_store)
naked = vrp_metrics_from_store(naked_store, cost_bps=5.5, direction="short_vol")
naked_store.close()
capped_store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_with_spike(capped_store)
capped = defined_risk_metrics_from_store(capped_store, cost_bps=5.5, direction="short_vol",
wing_width=0.15)
capped_store.close()
assert naked["available"] and capped["available"]
# the defined-risk drawdown is materially shallower than the naked one (the wings work).
assert capped["max_dd"] > naked["max_dd"]
assert capped["max_dd"] > 1.3 * naked["max_dd"] # at least ~30% shallower (maxDD is negative)
# --- 4b. THE TURNOVER / COST-ACCOUNTING MODEL (the amortized daily roll, not full-ladder-daily) -
def test_daily_cost_is_one_spreads_legs_not_the_whole_ladder() -> None:
"""THE CORE TURNOVER FIX. A 30-day-roll condor ladder opens exactly ONE new spread per day (n_legs legs)
and lets one expire (settles — NO closing trade). So the daily traded notional is (n_legs / swap_days) of
the gross book, NOT the whole k-leverage ladder re-traded every day. The defined-risk daily haircut must be
k · (n_legs / swap_days) · cost_bps/1e4 — far smaller than the naked daily-rehedge haircut k · cost_bps/1e4."""
k, cost_bps, swap_days = 3.0, 50.0, 30
dr = _defined_risk_daily_cost(k=k, cost_bps=cost_bps, swap_days=swap_days)
expected = k * (_N_LEGS / swap_days) * (cost_bps / 1e4)
assert math.isclose(dr, expected, rel_tol=1e-12)
# it is exactly (n_legs / swap_days) of the naked full-ladder-daily haircut.
naked_full_ladder_daily = k * (cost_bps / 1e4)
assert math.isclose(dr, naked_full_ladder_daily * (_N_LEGS / swap_days), rel_tol=1e-12)
# with 4 legs over a 30-day roll that is ~7.5x cheaper than charging the whole ladder daily.
assert dr < naked_full_ladder_daily / 7.0
def test_old_full_ladder_daily_overcharging_fails_the_turnover_assertion() -> None:
"""REGRESSION GUARD. The OLD behavior charged the FULL ladder every day (k · cost_bps/1e4 — the naked
daily-rehedge model). On a 30-day-roll 4-leg condor that over-charges turnover by swap_days/n_legs = 7.5x;
the corrected daily roll must be strictly, materially smaller — proving the bug is fixed, not re-introduced."""
k, cost_bps, swap_days = 3.0, 50.0, 30
old_full_ladder_daily = k * (cost_bps / 1e4) # the buggy model
corrected = _defined_risk_daily_cost(k=k, cost_bps=cost_bps, swap_days=swap_days)
over_charge_factor = old_full_ladder_daily / corrected
assert math.isclose(over_charge_factor, swap_days / _N_LEGS, rel_tol=1e-9)
assert over_charge_factor > 7.0 # ~30x/4legs ≈ 7.5x over-charge
def test_defined_risk_sized_series_uses_amortized_roll_cost() -> None:
"""The defined-risk sized series subtracts the AMORTIZED daily-roll haircut from each k·raw day — NOT the
naked full-ladder haircut the reused vrp_eval._sized_series applies."""
raw = {1: 0.001, 2: -0.002, 3: 0.0015}
k, cost_bps, swap_days = 2.0, 50.0, 30
dr = _defined_risk_sized_series(raw, k=k, cost_bps=cost_bps, swap_days=swap_days)
haircut = _defined_risk_daily_cost(k=k, cost_bps=cost_bps, swap_days=swap_days)
for d in raw:
assert math.isclose(dr[d], k * raw[d] - haircut, rel_tol=1e-12, abs_tol=1e-15)
# and it is strictly less punitive than the naked full-ladder series on every day (cost-only difference).
naked = vrp_eval._sized_series(raw, k=k, cost_bps=cost_bps)
for d in raw:
assert dr[d] > naked[d]
def test_per_year_cost_drag_is_proportionate_not_strategy_destroying() -> None:
"""SANITY on the realized cost drag. The per-YEAR cost of the amortized roll is
n_legs · cost_bps · (365/swap_days) · 1 (per unit notional, leverage 1). At 50bp on a 4-leg 30-day roll
that is 4·0.005·(365/30) ≈ 24%/yr at leverage 1 — a meaningful but NON-catastrophic drag, NOT the ~7.5x
inflated number the old full-ladder-daily model implied (which would be ~180%/yr)."""
cost_bps, swap_days = 50.0, 30
per_year_drag = _N_LEGS * (cost_bps / 1e4) * 365 / swap_days
assert 0.20 < per_year_drag < 0.30 # ~24%/yr at leverage 1 — bounded
# at the cheap 5.5bp the per-year drag is tiny (~2-3%/yr at leverage 1).
cheap = _N_LEGS * (5.5 / 1e4) * 365 / swap_days
assert cheap < 0.04
def test_cost_sensitivity_is_proportionate_5p5_close_to_gross_50_bounded() -> None:
"""END-TO-END SANITY: the corrected cost makes the strategy COST-PROPORTIONATE, not cost-destroyed. On the
tail-survivable fixture: at 5.5bp the net Sharpe is CLOSE to gross (small drag, NO sign flip); at 50bp the
drag is meaningful but BOUNDED — Sharpe stays in the same ballpark and maxDD nowhere near 100%."""
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_survivable(store)
gross = defined_risk_metrics_from_store(store, cost_bps=0.0, direction="short_vol")
cheap = defined_risk_metrics_from_store(store, cost_bps=5.5, direction="short_vol")
pricey = defined_risk_metrics_from_store(store, cost_bps=50.0, direction="short_vol")
store.close()
assert gross["available"] and cheap["available"] and pricey["available"]
assert gross["sharpe"] > 0.0
# 5.5bp: a SMALL drag, no sign flip, close to gross.
assert cheap["sharpe"] > 0.0
assert cheap["sharpe"] > 0.80 * gross["sharpe"]
# 50bp: meaningful but proportionate — still a sizeable fraction of gross, NOT a catastrophic collapse.
assert pricey["sharpe"] > 0.40 * gross["sharpe"]
# the tail stays survivable (the wings cap it) — nowhere near 100%.
assert pricey["max_dd"] > -0.40
# --- 5. adversarial verify ---------------------------------------------------------------------
def test_verify_cost_monotone_incl_option_costs() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
rep = verify_defined_risk_edge(store, cost_bps=5.5, cost_grid=(5.5, 11.0, 22.0, 50.0, 100.0))
store.close()
sv = rep["net_cost"]["short_vol"]
grid = sorted(sv)
for a, b in zip(grid, grid[1:], strict=False):
assert sv[a]["sharpe"] >= sv[b]["sharpe"]
assert 50.0 in sv and 100.0 in sv
def test_verify_reports_correlation_with_all_four_edges() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
rep = verify_defined_risk_edge(store, cost_bps=5.5)
store.close()
for d in ("short_vol", "long_vol"):
for edge in ("tstrend", "unlock", "xsfunding", "positioning"):
assert edge in rep["corr_edges"][d]
v = rep["corr_edges"][d][edge]
assert v is None or (-1.0 - 1e-9 <= v <= 1.0 + 1e-9)
def test_verify_is_read_only() -> None:
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(inner)
store = _WriteTripwireStore(inner)
rep = verify_defined_risk_edge(store, cost_bps=5.5)
inner.close()
assert rep["available"] is True
assert rep["wing_width"] > 0.0
# --- 6. look-ahead audit -----------------------------------------------------------------------
def test_look_ahead_clean_on_causal_signal() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_varying_dvol(store, days=120)
ok = look_ahead_audit_defined_risk_ok(store)
store.close()
assert ok is True
def test_look_ahead_audit_catches_leaked_mapping() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_varying_dvol(store, days=120)
ok = look_ahead_audit_defined_risk_ok(store, _leak=True)
store.close()
assert ok is False
# --- 7. walk-forward / OOS deploy gate ---------------------------------------------------------
def test_walk_forward_runs_and_reports_verdict() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_survivable(store)
rep = walk_forward_defined_risk(store, cost_bps=5.5, window_days=180)
store.close()
assert rep["available"] is True
assert "deployable" in rep["verdict"]
assert rep["look_ahead"]["causal"] is True
assert rep["wing_width"] > 0.0
def test_walk_forward_tail_survivable_passes_tail_check() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_survivable(store)
rep = walk_forward_defined_risk(store, cost_bps=5.5, window_days=180)
store.close()
# the wings make the tail survivable (the defining property of the defined-risk form).
assert rep["verdict"]["checks"]["tail_survivable"] is True
def test_walk_forward_is_read_only() -> None:
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_survivable(inner)
store = _WriteTripwireStore(inner)
rep = walk_forward_defined_risk(store, cost_bps=5.5)
inner.close()
assert rep["available"] is True
# --- 8. CLI ------------------------------------------------------------------------------------
def test_cli_defined_risk_prints_metrics(monkeypatch) -> None:
from typer.testing import CliRunner
import fxhnt.cli as cli
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
monkeypatch.setattr(
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
lambda *_a, **_k: store, raising=False)
result = CliRunner().invoke(cli.app, ["vrp-defined-risk-eval"])
store.close()
assert result.exit_code == 0, result.output
out = result.output.lower()
assert "defined" in out or "wing" in out
assert "maxdd" in out or "max dd" in out
assert "worst" in out
def test_cli_defined_risk_verify(monkeypatch) -> None:
from typer.testing import CliRunner
import fxhnt.cli as cli
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_with_spike(store, days=400)
monkeypatch.setattr(
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
lambda *_a, **_k: store, raising=False)
result = CliRunner().invoke(cli.app, ["vrp-defined-risk-eval", "--verify"])
store.close()
assert result.exit_code == 0, result.output
out = result.output.lower()
assert "short" in out and "long" in out
assert "100" in out
assert "tstrend" in out and "positioning" in out
def test_cli_defined_risk_walk_forward(monkeypatch) -> None:
from typer.testing import CliRunner
import fxhnt.cli as cli
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_survivable(store)
monkeypatch.setattr(
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
lambda *_a, **_k: store, raising=False)
result = CliRunner().invoke(cli.app, ["vrp-defined-risk-eval", "--walk-forward", "--cost-bps", "50"])
store.close()
assert result.exit_code == 0, result.output
out = result.output.lower()
assert "walk-forward" in out or "walk forward" in out
assert "verdict" in out
assert "pass" in out or "fail" in out

View File

@@ -1,493 +0,0 @@
"""READ-ONLY, TAIL-HONEST evaluator for the NEW VRP (volatility-risk-premium) edge.
Thesis: Deribit DVOL (implied vol) systematically exceeds realized vol, so SELLING variance (short-vol)
earns the spread — but with fat LEFT tails on vol spikes (RV >> IV). The signal/backtest uses DVOL (IV) +
close-to-close RV; live execution would be short Bybit option straddles (a cross-venue caveat).
VRP return (per asset, per day, CAUSAL):
short_var_ret_t = k * [ (DVOL_{t-1}/sqrt(365)/100)^2 - ret_t^2 ]
DVOL_{t-1} is the IV known at the START of day t (no look-ahead); ret_t = close-to-close (t-1 -> t). The book
is BTC+ETH equal-weight, vol-targeted to a sane annual vol (k). direction="short_vol" harvests; "long_vol" is
the negation.
Tests assert:
* the VRP return is POSITIVE on a calm-IV-high fixture and BIG-NEGATIVE on a spike fixture (the tail shows);
* sizing (vol-target) scales the series; direction flips the sign;
* the evaluator is READ-ONLY (a write-tripwire never trips);
* verify is cost-monotone (incl 50/100bp option costs), per-year, reports worst-day + maxDD, and the
correlation vs ALL FOUR existing edges (tstrend/unlock/xsfunding/positioning);
* walk-forward train/test OOS + look-ahead-clean; a tail-survivable fixture PASSES, a tail-dominated one
FAILS;
* the CLI smoke prints metrics + verify + walk-forward (mocked in-memory store, NO network).
"""
from __future__ import annotations
import math
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
from fxhnt.application.vrp_eval import (
look_ahead_audit_vrp,
short_var_return,
variance_swap_payoff,
verify_vrp_edge,
vrp_metrics_from_store,
vrp_returns_from_store,
walk_forward_vrp,
)
_DAY = 86_400
_START = 19_358 # 2023-01-01 epoch day
class _WriteTripwireStore:
"""Read-only proxy: forwards reads, but ANY write/persist call trips an assertion."""
_FORBIDDEN = frozenset({
"write_features", "write_features_bulk", "upsert_feature_rows", "upsert_membership",
"replace_positions", "upsert_nav", "upsert_sleeve_ret", "replace_shadow_positions",
"replace_trades", "_create_schema", "_bulk_upsert",
})
def __init__(self, inner: TimescaleFeatureStore) -> None:
object.__setattr__(self, "_inner", inner)
def __getattr__(self, name: str):
if name in _WriteTripwireStore._FORBIDDEN:
raise AssertionError(f"READ-ONLY violation: evaluator called write method {name!r}")
return getattr(self._inner, name)
# --- 1. short_var_return (the formula) ---------------------------------------------------------
def test_short_var_positive_when_calm_iv_high() -> None:
# IV (DVOL) = 80 vol pts; realized daily move tiny -> RV << IV -> short-vol HARVESTS (positive).
r = short_var_return(dvol_prev=80.0, ret=0.001)
assert r > 0.0
def test_short_var_big_negative_on_spike() -> None:
# IV = 40 vol pts (calm implied), but realized move is a -20% spike -> RV >> IV -> big NEGATIVE (the tail).
calm = short_var_return(dvol_prev=40.0, ret=0.005)
spike = short_var_return(dvol_prev=40.0, ret=-0.20)
assert spike < 0.0
assert spike < calm
# the spike loss dwarfs a calm-day gain in magnitude (fat left tail).
assert abs(spike) > 5.0 * abs(calm)
def test_short_var_implied_daily_variance_is_iv_squared() -> None:
# On a zero-realized-move day the harvest equals the implied daily variance (DVOL/sqrt(365)/100)^2.
iv_daily = 60.0 / math.sqrt(365) / 100.0
r = short_var_return(dvol_prev=60.0, ret=0.0)
assert math.isclose(r, iv_daily ** 2, rel_tol=1e-12)
def test_long_vol_is_negation_of_short_vol() -> None:
assert math.isclose(short_var_return(dvol_prev=50.0, ret=0.03, direction="long_vol"),
-short_var_return(dvol_prev=50.0, ret=0.03, direction="short_vol"),
rel_tol=1e-12)
# --- 1b. variance_swap_payoff (the PROPER variance swap: implied strike vs realized-WINDOW variance) ----
def test_variance_swap_payoff_positive_when_calm_window() -> None:
# IV (DVOL) = 60 vol pts (strike K=0.36 annual var); realized window is tiny ±0.2% daily moves -> RV << K
# -> short-var swap HARVESTS the premium (positive).
fwd = [0.002 if i % 2 == 0 else -0.002 for i in range(30)]
assert variance_swap_payoff(dvol_entry=60.0, fwd_rets=fwd, swap_days=30) > 0.0
def test_variance_swap_payoff_negative_on_spike_window() -> None:
# IV = 40 vol pts (K=0.16); the realized window contains a -25% crash -> RV >> K -> NEGATIVE (the tail).
fwd = [0.001] * 29 + [-0.25]
assert variance_swap_payoff(dvol_entry=40.0, fwd_rets=fwd, swap_days=30) < 0.0
def test_variance_swap_payoff_annualization_zero_when_rv_equals_iv() -> None:
# Constant-vol synthetic: pick a daily move whose annualised realized variance EXACTLY equals the strike
# K=(DVOL/100)^2 -> the short-var swap payoff is ~0 (the annualisation 365/swap_days is correct).
dvol = 50.0
k = (dvol / 100.0) ** 2 # annual implied variance = 0.25
daily_var = k / 365.0 # per-day variance so (365/n)*n*daily_var == k
move = math.sqrt(daily_var)
fwd = [move if i % 2 == 0 else -move for i in range(30)]
payoff = variance_swap_payoff(dvol_entry=dvol, fwd_rets=fwd, swap_days=30)
assert abs(payoff) < 1e-12
def test_variance_swap_payoff_long_is_negation() -> None:
fwd = [0.01, -0.02, 0.015] * 10
assert math.isclose(variance_swap_payoff(dvol_entry=55.0, fwd_rets=fwd, direction="long_vol"),
-variance_swap_payoff(dvol_entry=55.0, fwd_rets=fwd, direction="short_vol"),
rel_tol=1e-12)
def test_variance_swap_payoff_empty_window_is_zero() -> None:
assert variance_swap_payoff(dvol_entry=60.0, fwd_rets=[]) == 0.0
# --- 2. vrp_returns_from_store / sizing --------------------------------------------------------
def _seed_calm_premium(store: TimescaleFeatureStore, *, days: int = 260, dvol: float = 60.0,
daily_move: float = 0.004) -> None:
"""Seed BTC+ETH so IV (dvol=60) comfortably exceeds RV (a tiny ±daily_move oscillation): short-vol earns
a steady premium. close oscillates so RV is small + bounded; dvol flat-high. Spans ~9 months."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
sign = 1.0 if i % 2 == 0 else -1.0
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol})])
px[s] *= (1.0 + sign * daily_move)
def test_vrp_returns_positive_on_calm_premium() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
series = vrp_returns_from_store(store, cost_bps=0.0, direction="short_vol")
store.close()
assert len(series) > 2
assert sum(series.values()) > 0.0 # the premium is harvested gross
def test_vrp_sizing_targets_annual_vol() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
s15 = vrp_returns_from_store(store, cost_bps=0.0, target_ann_vol=0.15)
s30 = vrp_returns_from_store(store, cost_bps=0.0, target_ann_vol=0.30)
store.close()
import statistics as st
v15 = st.pstdev(s15.values()) * math.sqrt(365)
v30 = st.pstdev(s30.values()) * math.sqrt(365)
# vol-target hits (roughly) the requested annual vol, and doubling the target ~doubles realized vol.
assert abs(v15 - 0.15) < 0.03
assert v30 > 1.8 * v15
def test_vrp_constant_premium_series_is_low_noise_and_sane() -> None:
"""THE KEY FIX. A constant-premium synthetic (IV steadily > RV every day) must give a SMOOTH, LOW-noise,
SANE-Sharpe daily series — NOT the 40 Sharpe / blow-up the old single-day-ret^2 proxy produced. The
rolling variance-swap ladder averages RV over a 30-day window, so the day-to-day series is far less noisy
than a raw ret^2. We assert the GROSS (pre-vol-target) ladder series has a high, positive Sharpe and tiny
relative dispersion — i.e. it is well-behaved enough that vol-targeting it can't lose ~everything."""
import statistics as st
from fxhnt.application.vrp_eval import _book_short_var
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store, days=400, dvol=60.0, daily_move=0.004)
raw = _book_short_var(store, direction="short_vol")
store.close()
vals = [raw[d] for d in sorted(raw)]
assert len(vals) > 30
mean, sd = st.mean(vals), st.pstdev(vals)
assert mean > 0.0 # a steady harvested premium
gross_sharpe = mean / sd * math.sqrt(365)
# the ladder series is sane: a strongly POSITIVE Sharpe (the old single-day proxy gave -40 / blew up).
assert gross_sharpe > 3.0
# and genuinely low-noise: the daily dispersion is small relative to the mean (a smooth window-averaged
# quantity, not a spiky raw ret^2).
assert sd < mean # coefficient of variation < 1 (the smoothing fix)
def test_vrp_vol_targeted_constant_premium_does_not_blow_up_drawdown() -> None:
"""The corrected + vol-targeted constant-premium curve must be SANE: no catastrophic drawdown. The old
proxy's single-day spikes blew through the vol-target to 96%/180d; the window-averaged ladder must not."""
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store, days=400, dvol=60.0, daily_move=0.004)
m = vrp_metrics_from_store(store, cost_bps=5.5, target_ann_vol=0.15)
store.close()
assert m["available"] is True
assert m["sharpe"] > 1.0 # a real, sane edge on the calm fixture
assert m["max_dd"] > -0.40 # NOT a 96% blow-up
def test_vrp_direction_flips_sign() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
short = vrp_returns_from_store(store, cost_bps=0.0, direction="short_vol")
long_ = vrp_returns_from_store(store, cost_bps=0.0, direction="long_vol")
store.close()
assert sum(short.values()) > 0.0 > sum(long_.values())
def test_vrp_metrics_na_when_no_dvol() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
store.write_features("BTCUSDT", [(0, {"close": 30_000.0})]) # close but no dvol
m = vrp_metrics_from_store(store)
store.close()
assert m["available"] is False
assert "reason" in m and m["reason"]
def test_vrp_metrics_is_read_only() -> None:
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(inner)
store = _WriteTripwireStore(inner)
m = vrp_metrics_from_store(store, cost_bps=5.5)
inner.close()
assert m["available"] is True
def test_vrp_metrics_reports_tail() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
m = vrp_metrics_from_store(store, cost_bps=0.0)
store.close()
assert m["available"] is True
assert m["max_dd"] <= 0.0 # maxDD is a (non-positive) drawdown
assert "worst_day" in m # the single worst vol-spike day loss
assert "avg_leverage" in m # exposure from the vol-target
# --- 3. tail handling: a spike fixture must show the loss ---------------------------------------
def _seed_with_spike(store: TimescaleFeatureStore, *, days: int = 260, spike_at: int = 130) -> None:
"""Calm short-vol premium most days, but ONE catastrophic vol-spike day (a -35% move with calm prior IV)
that drives RV >> IV -> a huge negative VRP return on that day (the tail the backtest must capture)."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": 55.0})])
if i == spike_at:
px[s] *= (1.0 - 0.35) # crash: realized vol explodes past implied
else:
px[s] *= (1.0 + (0.003 if i % 2 == 0 else -0.003))
def test_spike_day_is_the_worst_day_and_dominates_maxdd() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_with_spike(store)
m = vrp_metrics_from_store(store, cost_bps=0.0)
series = vrp_returns_from_store(store, cost_bps=0.0)
store.close()
worst = min(series.values())
assert worst < 0.0
assert math.isclose(m["worst_day"], worst, rel_tol=1e-9)
# the spike produces a material drawdown — the tail is captured, not smoothed away.
assert m["max_dd"] < -0.05
# --- 4. adversarial verify ---------------------------------------------------------------------
def test_verify_cost_monotone_incl_option_costs() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
rep = verify_vrp_edge(store, cost_bps=5.5, cost_grid=(5.5, 11.0, 22.0, 50.0, 100.0))
store.close()
sv = rep["net_cost"]["short_vol"]
grid = sorted(sv)
for a, b in zip(grid, grid[1:], strict=False):
assert sv[a]["sharpe"] >= sv[b]["sharpe"] # higher cost -> lower Sharpe
assert 50.0 in sv and 100.0 in sv # realistic option-cost sensitivity shown
def test_verify_reports_worst_day_and_per_year() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_with_spike(store, days=400) # spans 2023 and 2024
rep = verify_vrp_edge(store, cost_bps=5.5)
store.close()
assert rep["available"] is True
assert rep["worst_day"]["short_vol"] < 0.0
py = rep["per_year"]["short_vol"]
assert len(py) >= 2
def test_verify_reports_correlation_with_all_four_edges() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
rep = verify_vrp_edge(store, cost_bps=5.5)
store.close()
for d in ("short_vol", "long_vol"):
corrs = rep["corr_edges"][d]
for edge in ("tstrend", "unlock", "xsfunding", "positioning"):
assert edge in corrs
v = corrs[edge]
assert v is None or (-1.0 - 1e-9 <= v <= 1.0 + 1e-9)
def test_verify_is_read_only() -> None:
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(inner)
store = _WriteTripwireStore(inner)
rep = verify_vrp_edge(store, cost_bps=5.5)
inner.close()
assert rep["available"] is True
# --- 5. look-ahead audit -----------------------------------------------------------------------
def _seed_varying_dvol(store: TimescaleFeatureStore, *, days: int = 120) -> None:
"""A premium fixture with NON-PERIODIC TIME-VARYING DVOL so the causal (strike=DVOL at swap entry) and
leaked (strike=DVOL at the window END) mappings are genuinely distinguishable on EVERY day — the
look-ahead audit can only bite when the entry-day and window-end IV differ. A slow irrational-step ramp
guarantees no two days share a DVOL value (no coincidental entry/window-end collision)."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
dvol = 50.0 + 0.137 * i # strictly monotone, distinct every day (no collisions)
sign = 1.0 if i % 2 == 0 else -1.0
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol})])
px[s] *= (1.0 + sign * 0.004)
def test_look_ahead_audit_passes_on_causal_signal() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_varying_dvol(store, days=120)
rep = look_ahead_audit_vrp(store)
store.close()
assert rep["causal"] is True
assert rep["leak_days"] == 0
def test_look_ahead_audit_catches_a_deliberately_leaked_mapping() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_varying_dvol(store, days=120)
rep = look_ahead_audit_vrp(store, _leak=True) # pair the SAME-day DVOL with the day's return
store.close()
assert rep["causal"] is False
assert rep["leak_days"] > 0
# --- 6. walk-forward / OOS deploy gate ---------------------------------------------------------
def _seed_tail_survivable(store: TimescaleFeatureStore, *, days: int = 700) -> None:
"""A short-vol premium that SURVIVES its occasional tails: calm harvest most days (RV well below the
varying IV), a moderate spike every ~90 days whose loss is recouped within the next window. Positive in
most windows + OOS, drawdown bounded above the 40% floor. DVOL varies day to day (so the causality audit
can distinguish the prior-day from the same-day IV)."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
dvol = 60.0 + 8.0 * math.sin(i * 0.11) # non-periodic-on-30d drift in 52..68 (no collisions)
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol, "turnover": 5e8})])
if i % 90 == 45:
px[s] *= (1.0 - 0.06) # moderate, recoverable spike
else:
px[s] *= (1.0 + (0.004 if i % 2 == 0 else -0.004))
def _seed_tail_dominated(store: TimescaleFeatureStore, *, days: int = 700) -> None:
"""A short-vol book the gate must FAIL: thin implied premium (low DVOL), but FREQUENT, LARGE spikes whose
realized variance dwarfs the premium. After vol-targeting the rare-but-huge spike days produce a
CATASTROPHIC drawdown (worse than the 40% floor) — VRP looks fine on the calm days and then a cluster of
spikes wipes it out. DVOL varies day to day (so the causality audit can distinguish prior vs same day)."""
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
for i in range(days):
d = _START + i
dvol = 25.0 + 4.0 * math.sin(i * 0.11) # thin implied vol ~21..29, non-periodic-on-30d
for s in ("BTCUSDT", "ETHUSDT"):
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol, "turnover": 5e8})])
# spikes cluster late (after the train window) so the short-vol harvest that "worked" on TRAIN
# is wiped in TEST -> OOS negative AND catastrophic drawdown.
late_cluster = i > int(days * 0.6) and (i % 9 == 0)
if late_cluster:
px[s] *= (1.0 - 0.22) # frequent, large, late, unrecouped spikes
else:
px[s] *= (1.0 + (0.0008 if i % 2 == 0 else -0.0008))
def test_walk_forward_pass_on_tail_survivable() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_survivable(store)
rep = walk_forward_vrp(store, cost_bps=5.5, window_days=180)
store.close()
v = rep["verdict"]
assert v["deployable"] is True
assert v["checks"]["oos_direction_positive"] is True
assert v["checks"]["tail_survivable"] is True
assert v["checks"]["look_ahead_clean"] is True
def test_walk_forward_fail_on_tail_dominated() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_dominated(store)
rep = walk_forward_vrp(store, cost_bps=5.5, window_days=180)
store.close()
v = rep["verdict"]
assert v["deployable"] is False
# it fails because the tail is NOT survivable (and/or OOS is negative).
assert v["checks"]["tail_survivable"] is False or v["checks"]["oos_direction_positive"] is False
def test_walk_forward_at_realistic_option_costs_judges_net_of_tail() -> None:
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_dominated(store)
# at a realistic OPTION cost (selling options isn't 5.5bp) the dominated book is plainly undeployable.
rep = walk_forward_vrp(store, cost_bps=50.0, window_days=180)
store.close()
assert rep["verdict"]["deployable"] is False
def test_walk_forward_is_read_only() -> None:
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_survivable(inner)
store = _WriteTripwireStore(inner)
rep = walk_forward_vrp(store, cost_bps=5.5)
inner.close()
assert rep["available"] is True
# --- 7. CLI ------------------------------------------------------------------------------------
def test_cli_vrp_eval_prints_metrics(monkeypatch) -> None:
from typer.testing import CliRunner
import fxhnt.cli as cli
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_calm_premium(store)
monkeypatch.setattr(
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
lambda *_a, **_k: store, raising=False)
result = CliRunner().invoke(cli.app, ["vrp-eval"])
store.close()
assert result.exit_code == 0, result.output
out = result.output.lower()
assert "vrp" in out or "volatility risk premium" in out
assert "maxdd" in out or "max dd" in out
assert "worst" in out
def test_cli_vrp_eval_verify_prints_diagnostics(monkeypatch) -> None:
from typer.testing import CliRunner
import fxhnt.cli as cli
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_with_spike(store, days=400)
monkeypatch.setattr(
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
lambda *_a, **_k: store, raising=False)
result = CliRunner().invoke(cli.app, ["vrp-eval", "--verify"])
store.close()
assert result.exit_code == 0, result.output
out = result.output.lower()
assert "short" in out and "long" in out
assert "worst" in out
assert "100" in out # the 100bp option-cost column
assert "tstrend" in out and "positioning" in out
def test_cli_vrp_eval_walk_forward_prints_verdict(monkeypatch) -> None:
from typer.testing import CliRunner
import fxhnt.cli as cli
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
_seed_tail_survivable(store)
monkeypatch.setattr(
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
lambda *_a, **_k: store, raising=False)
result = CliRunner().invoke(cli.app, ["vrp-eval", "--walk-forward"])
store.close()
assert result.exit_code == 0, result.output
out = result.output.lower()
assert "walk-forward" in out or "walk forward" in out
assert "verdict" in out
assert "pass" in out or "fail" in out

View File

@@ -1,291 +0,0 @@
import datetime as dt
from fxhnt.adapters.orchestration.assets import (
_freeze_day,
_freeze_latest_session,
_freeze_xsp_slice,
_is_degenerate_forward_error,
)
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
def test_vrp_nav_survives_freeze_failure(tmp_path, monkeypatch):
"""C1b: a freeze failure (bad-data day / still-unavailable OPRA window / cost-cap trip) must NOT kill
vrp_nav — it logs a WARNING and CONTINUES to `_run_paper_tracker`, which recomputes off the FROZEN
xsp_option_bars table. Guarantees vrp always produces a forward row/anchor/ref off whatever IS frozen and
never skips the downstream gate (`cockpit_forward`)."""
from dagster import build_op_context
import fxhnt.adapters.orchestration.assets as assets
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
from fxhnt.config import Settings
monkeypatch.setattr("fxhnt.config.get_settings",
lambda: Settings(operational_dsn=f"sqlite:///{tmp_path / 'op.db'}"))
def _boom(*a, **k): # the freeze entry point exhausts its bounded step-back → catchable range error
raise DatabentoRangeUnavailable("no complete OPRA session within the step-back bound")
monkeypatch.setattr(assets, "_freeze_latest_session", _boom)
ran = {"tracker": False}
def _fake_tracker(context, name, build_strategy, *, persist_backtest_ref=False):
ran["tracker"] = True
return {"forward_days": 0, "last_date": None, "reinception": False}
monkeypatch.setattr(assets, "_run_paper_tracker", _fake_tracker)
out = assets.vrp_nav(build_op_context())
assert isinstance(out, dict) # returned a dict, did NOT raise out of the asset
assert ran["tracker"] # freeze died, but the tracker STILL recomputed off the frozen table
class _FakeDbn:
def __init__(self):
self.chain_calls = 0
def fetch_option_chain(self, parent, as_of):
from fxhnt.domain.models import OptionContract
self.chain_calls += 1
puts = [OptionContract(f"XSP240816P00{k}000", dt.date(2024, 8, 16), float(k), "P")
for k in range(440, 475, 5)]
calls = [OptionContract(f"XSP240816C00{k}000", dt.date(2024, 8, 16), float(k), "C")
for k in range(460, 481, 5)]
return puts + calls
def fetch_option_bars(self, osi, start, end):
return {s: {start: 1.0} for s in osi}
def test_freeze_xsp_slice_writes_puts_and_atm_calls(tmp_path):
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'o.db'}")
bars.migrate()
_freeze_xsp_slice(_FakeDbn(), bars, as_of="2024-07-20", forward=470.0, at=dt.datetime(2024, 7, 20))
puts = bars.chain_asof("2024-07-20", 20, 45, right="P")
calls = bars.chain_asof("2024-07-20", 20, 45, right="C")
# puts moneyness [0.70,1.20]*470 = [329,564] (wide band so held legs stay frozen) -> all seeded puts qualify
assert {c.strike for c in puts} == {float(k) for k in range(440, 475, 5)}
# calls moneyness [0.96,1.04]*470 = [451.2,488.8] -> 460..480 qualify (so parity has a common strike)
assert {c.strike for c in calls} == {460.0, 465.0, 470.0, 475.0, 480.0}
class _RangeFakeDbn:
"""Fake for the BATCHED backfill: one range-definition + one range-ohlcv call per month, marks across days."""
def __init__(self):
from fxhnt.domain.models import OptionContract
exp = dt.date(2024, 8, 5)
self.puts = [OptionContract(f"XSP240805P00{k}000", exp, float(k), "P") for k in range(455, 481, 5)]
self.calls = [OptionContract(f"XSP240805C00{k}000", exp, float(k), "C") for k in range(460, 481, 5)]
self.days = ["2024-07-01", "2024-07-02", "2024-07-03"]
self.chain_range_calls = 0
self.bar_calls = 0
def fetch_option_chain_range(self, parent, start, end):
self.chain_range_calls += 1
return self.puts + self.calls
def fetch_option_bars(self, osis, start, end):
self.bar_calls += 1
return {osi: {d: 1.0 for d in self.days} for osi in osis}
def test_backfill_month_batched_processes_days_locally(tmp_path):
"""_backfill_month must do ONE range-chain + ONE range-ohlcv fetch for the whole month, then process each
trading day LOCALLY (parity forward + band + upsert) — the batched win over ~6-8 API calls/day."""
from fxhnt.adapters.orchestration.assets import _backfill_month
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'b.db'}")
bars.migrate()
fake = _RangeFakeDbn()
ok, fail = _backfill_month(fake, bars, month_start="2024-07-01", month_end="2024-07-31",
at=dt.datetime(2024, 7, 1))
assert fake.chain_range_calls == 1 and fake.bar_calls == 1 # ONE bulk fetch each, not per-day
assert ok == 3 and fail == 0 # 3 trading days processed locally
puts = bars.chain_asof("2024-07-01", 20, 45, right="P")
assert puts # band frozen from the shared bulk data
got = bars.bars_for([puts[0].osi_symbol], "2024-07-01", "2024-07-03")
assert len(next(iter(got.values()))) == 3 # marks for all 3 days landed
def test_freeze_day_fetches_chain_once(tmp_path):
"""The nightly asset + backfill call _freeze_day, which must fetch the option chain (definition) exactly
ONCE per day and share it between the parity-forward estimate and the band freeze — the old code fetched
it twice (in _xsp_forward_estimate AND _freeze_xsp_slice), doubling OPRA definition cost."""
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'o.db'}")
bars.migrate()
fake = _FakeDbn()
fwd = _freeze_day(fake, bars, as_of="2024-07-20", at=dt.datetime(2024, 7, 20))
assert fake.chain_calls == 1 # ONE chain fetch/day, not two
assert 440.0 <= fwd <= 480.0 # a sane parity forward derived from the shared chain
assert bars.chain_asof("2024-07-20", 20, 45, right="P") # the band was frozen off that single chain
class _StepBackDbn:
"""Fake provider for the freeze step-back: `last_available_option_date()` returns a partial (still-settling)
today; `fetch_option_chain` raises DatabentoRangeUnavailable for any as_of NOT in `good_dates` (models a
partial session's `data_schema_not_fully_available`, already normalized by the provider). Records the
candidate order attempted so tests can assert weekend-skipping and step order."""
def __init__(self, available_date: dt.date, good_dates: set[str]) -> None:
self._available = available_date
self._good = good_dates
self.attempts: list[str] = []
def last_available_option_date(self) -> dt.date:
return self._available
def fetch_option_chain(self, parent: str, as_of: str):
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
from fxhnt.domain.models import OptionContract
self.attempts.append(as_of)
if as_of not in self._good:
raise DatabentoRangeUnavailable(f"session {as_of} not fully available")
d = dt.date.fromisoformat(as_of)
exp = d + dt.timedelta(days=30) # ~30-DTE expiry relative to the candidate
puts = [OptionContract(f"XSP{as_of}P{k}", exp, float(k), "P") for k in range(440, 475, 5)]
calls = [OptionContract(f"XSP{as_of}C{k}", exp, float(k), "C") for k in range(460, 481, 5)]
return puts + calls
def fetch_option_bars(self, osis, start, end):
return {s: {start: 1.0} for s in osis}
def test_freeze_latest_session_steps_back_past_partial_today(tmp_path):
"""Robust freeze: today's OPRA session is partial (raises), so the entry point steps back ONE trading day
to yesterday's COMPLETE session, freezes it, and RETURNS that date. Deterministic regardless of run time."""
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'o.db'}")
bars.migrate()
# Wed 2026-07-15 (today, partial) → Tue 2026-07-14 (complete)
fake = _StepBackDbn(dt.date(2026, 7, 15), good_dates={"2026-07-14"})
frozen = _freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15))
assert frozen == "2026-07-14" # landed on the last COMPLETE session
assert fake.attempts == ["2026-07-15", "2026-07-14"] # tried today first, then stepped back one day
assert bars.chain_asof("2026-07-14", 20, 45, right="P") # the band was frozen for that session
def test_freeze_latest_session_skips_weekend(tmp_path):
"""The step-back is by TRADING day: a Monday candidate steps to the prior FRIDAY, never Sat/Sun."""
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'w.db'}")
bars.migrate()
# Mon 2026-07-13 (partial) → skip Sun 12 / Sat 11 → Fri 2026-07-10 (complete)
fake = _StepBackDbn(dt.date(2026, 7, 13), good_dates={"2026-07-10"})
frozen = _freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 13))
assert frozen == "2026-07-10"
assert fake.attempts == ["2026-07-13", "2026-07-10"] # Sat 11 / Sun 12 skipped entirely
def test_freeze_latest_session_raises_when_no_complete_session_in_bound(tmp_path):
"""No complete session within the bounded step-back → raise DatabentoRangeUnavailable (LOUD). The C1b
wrapper in vrp_nav then recomputes off frozen and the health axis keeps vrp STALE — never a silent no-op."""
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'n.db'}")
bars.migrate()
fake = _StepBackDbn(dt.date(2026, 7, 15), good_dates=set()) # every candidate fails
try:
_freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15), max_steps=5)
raise AssertionError("expected DatabentoRangeUnavailable when the bound is exhausted")
except DatabentoRangeUnavailable:
pass
assert len(fake.attempts) == 6 # start candidate + 5 bounded step-backs
# --- MINOR-2: step back on a PUBLISHED-but-DEGENERATE session (not only an unavailable one) ---------------
class _DegenerateDbn:
"""A published session whose chain cannot form a parity forward — `fetch_option_chain` RETURNS contracts
(no DatabentoRangeUnavailable), but for a DEGENERATE date the only expiry is far outside the [20,45]-DTE
forward window, so `_xsp_forward_estimate` raises `RuntimeError('no XSP chain … cannot estimate forward')`.
A GOOD date returns a proper ~30-DTE chain with a matched ATM call/put pair. Models a broken published
session that must be stepped over, not crashed on."""
def __init__(self, available_date: dt.date, good_dates: set[str]) -> None:
self._available = available_date
self._good = good_dates
self.attempts: list[str] = []
def last_available_option_date(self) -> dt.date:
return self._available
def fetch_option_chain(self, _parent: str, as_of: str):
from fxhnt.domain.models import OptionContract
self.attempts.append(as_of)
d = dt.date.fromisoformat(as_of)
if as_of in self._good:
exp = d + dt.timedelta(days=30) # in [20,45] -> forward derivable
puts = [OptionContract(f"XSP{as_of}P{k}", exp, float(k), "P") for k in range(440, 475, 5)]
calls = [OptionContract(f"XSP{as_of}C{k}", exp, float(k), "C") for k in range(460, 481, 5)]
return puts + calls
exp = d + dt.timedelta(days=100) # OUTSIDE [20,45] -> `near` empty -> RuntimeError
return [OptionContract(f"XSP{as_of}P{k}", exp, float(k), "P") for k in range(440, 475, 5)]
def fetch_option_bars(self, osis, start, _end):
return {s: {start: 1.0} for s in osis}
def test_freeze_latest_session_steps_back_past_degenerate_published_session(tmp_path):
"""A PUBLISHED but degenerate today (forward not derivable -> RuntimeError, NOT DatabentoRangeUnavailable)
must step back to the prior USABLE session, not hard-crash the caller."""
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'deg.db'}")
bars.migrate()
fake = _DegenerateDbn(dt.date(2026, 7, 15), good_dates={"2026-07-14"}) # Wed degenerate -> Tue usable
frozen = _freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15))
assert frozen == "2026-07-14"
assert fake.attempts == ["2026-07-15", "2026-07-14"] # tried the broken published session, then stepped back
assert bars.chain_asof("2026-07-14", 20, 45, right="P") # the usable session was frozen
def test_freeze_latest_session_raises_when_all_degenerate_in_bound(tmp_path):
"""Every candidate within the bound is a degenerate published session -> raise DatabentoRangeUnavailable
(LOUD), never silently return a bar-less date."""
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'alldeg.db'}")
bars.migrate()
fake = _DegenerateDbn(dt.date(2026, 7, 15), good_dates=set()) # every session degenerate
try:
_freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15), max_steps=5)
raise AssertionError("expected DatabentoRangeUnavailable when every candidate is degenerate")
except DatabentoRangeUnavailable:
pass
assert len(fake.attempts) == 6 # start candidate + 5 bounded step-backs
class _UnrelatedErrorDbn:
"""A session whose chain is fine but whose bar fetch fails with an UNRELATED RuntimeError (e.g. a DB
outage). `_freeze_latest_session` must NOT swallow this as a degenerate-session step-back — it must
propagate, so a real infra failure is never masked as 'no session available'."""
def __init__(self) -> None:
self.attempts: list[str] = []
def last_available_option_date(self) -> dt.date:
return dt.date(2026, 7, 15)
def fetch_option_chain(self, _parent: str, as_of: str):
from fxhnt.domain.models import OptionContract
self.attempts.append(as_of)
exp = dt.date.fromisoformat(as_of) + dt.timedelta(days=30)
puts = [OptionContract(f"XSP{as_of}P{k}", exp, float(k), "P") for k in range(440, 475, 5)]
calls = [OptionContract(f"XSP{as_of}C{k}", exp, float(k), "C") for k in range(460, 481, 5)]
return puts + calls
def fetch_option_bars(self, _osis, _start, _end):
raise RuntimeError("db connection lost") # unrelated to forward estimation
def test_freeze_latest_session_propagates_unrelated_runtime_error(tmp_path):
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'unrel.db'}")
bars.migrate()
fake = _UnrelatedErrorDbn()
try:
_freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15))
raise AssertionError("expected the unrelated RuntimeError to propagate")
except RuntimeError as e:
assert "db connection lost" in str(e) # propagated, not swallowed as a step-back
assert fake.attempts == ["2026-07-15"] # did NOT step back on an unrelated error
def test_is_degenerate_forward_error_matches_forward_messages_only():
# the three forward-estimation messages `_xsp_forward_estimate`/`_forward_from_marks` raise -> step-back.
assert _is_degenerate_forward_error(RuntimeError("no XSP chain for 2026-07-15 — cannot estimate forward"))
assert _is_degenerate_forward_error(RuntimeError("no matched call/put strike for 2026-07-15 forward"))
assert _is_degenerate_forward_error(RuntimeError("no ATM call/put pair with marks for 2026-07-15"))
# anything else (e.g. a DB/infra error) is NOT a step-back trigger.
assert not _is_degenerate_forward_error(RuntimeError("db connection lost"))
assert not _is_degenerate_forward_error(ValueError("no XSP chain")) # wrong type is irrelevant here; msg-only

View File

@@ -1,80 +0,0 @@
"""Black-Scholes pricer (r=0, crypto convention) — known-value + invariant tests.
The pricer is used to synthesize the legs of the defined-risk VRP spread from DVOL (we have no historical
option chains). These tests pin the standard BS identities so the spread pricing is trustworthy:
* the erf-based normal CDF is correct (matches scipy / known values);
* an ATM call ≈ 0.4·S·σ·√T (the textbook small-T approximation);
* put-call parity at r=0: call put = S K;
* prices are monotone increasing in IV (vega > 0) and bounded by intrinsic from below;
* degenerate (T=0 / σ=0) legs collapse to intrinsic value.
"""
from __future__ import annotations
import math
from fxhnt.application.black_scholes import _norm_cdf, bs_call, bs_put
def test_norm_cdf_known_values() -> None:
assert math.isclose(_norm_cdf(0.0), 0.5, abs_tol=1e-12)
# N(1.96) ≈ 0.975, N(-1.96) ≈ 0.025 (the textbook two-sided 95% points).
assert math.isclose(_norm_cdf(1.96), 0.9750021, abs_tol=1e-6)
assert math.isclose(_norm_cdf(-1.96), 0.0249979, abs_tol=1e-6)
# symmetry: N(x) + N(-x) = 1.
assert math.isclose(_norm_cdf(0.73) + _norm_cdf(-0.73), 1.0, abs_tol=1e-12)
def test_norm_cdf_matches_scipy_if_available() -> None:
try:
from scipy.stats import norm
except ImportError:
return
for x in (-2.5, -1.0, -0.1, 0.0, 0.3, 1.5, 3.0):
assert math.isclose(_norm_cdf(x), float(norm.cdf(x)), abs_tol=1e-12)
def test_atm_call_approx_zero_point_four_s_sigma_sqrt_t() -> None:
# The standard small-T ATM approximation: C_atm ≈ 0.3989·S·σ·√T ≈ 0.4·S·σ·√T.
s, sigma, t = 30_000.0, 0.60, 30.0 / 365.0
approx = 0.3989 * s * sigma * math.sqrt(t)
price = bs_call(s=s, k=s, t=t, sigma=sigma)
assert math.isclose(price, approx, rel_tol=0.02) # within 2% of the textbook approximation
def test_put_call_parity_r_zero() -> None:
# At r=0 (and no dividends): call put = S K, for any K.
s, t, sigma = 2_000.0, 45.0 / 365.0, 0.75
for k in (1_800.0, 2_000.0, 2_300.0):
c = bs_call(s=s, k=k, t=t, sigma=sigma)
p = bs_put(s=s, k=k, t=t, sigma=sigma)
assert math.isclose(c - p, s - k, abs_tol=1e-6)
def test_prices_monotone_increasing_in_iv() -> None:
s, k, t = 30_000.0, 30_000.0, 30.0 / 365.0
lo = bs_call(s=s, k=k, t=t, sigma=0.30)
hi = bs_call(s=s, k=k, t=t, sigma=0.90)
assert hi > lo > 0.0
plo = bs_put(s=s, k=k, t=t, sigma=0.30)
phi = bs_put(s=s, k=k, t=t, sigma=0.90)
assert phi > plo > 0.0
def test_price_bounded_below_by_intrinsic() -> None:
s, t, sigma = 30_000.0, 30.0 / 365.0, 0.60
# an ITM call (K below S) is worth at least its intrinsic value S K.
c = bs_call(s=s, k=27_000.0, t=t, sigma=sigma)
assert c >= (s - 27_000.0)
# an ITM put (K above S) is worth at least K S.
p = bs_put(s=s, k=33_000.0, t=t, sigma=sigma)
assert p >= (33_000.0 - s)
def test_degenerate_collapses_to_intrinsic() -> None:
# T=0: call = max(SK,0), put = max(KS,0).
assert bs_call(s=100.0, k=90.0, t=0.0, sigma=0.5) == 10.0
assert bs_call(s=100.0, k=110.0, t=0.0, sigma=0.5) == 0.0
assert bs_put(s=100.0, k=110.0, t=0.0, sigma=0.5) == 10.0
# σ=0: same intrinsic limit (a zero-vol option is just its forward intrinsic).
assert bs_call(s=100.0, k=90.0, t=0.5, sigma=0.0) == 10.0
assert bs_put(s=100.0, k=110.0, t=0.5, sigma=0.0) == 10.0

View File

@@ -10,9 +10,9 @@ from fxhnt.registry import STRATEGY_REGISTRY
def test_every_registry_strategy_has_a_display_name() -> None:
for sid, entry in STRATEGY_REGISTRY.items():
# multistrat/vrp are deliberately overridden with a friendlier pill label (see the next test) — every
# multistrat is deliberately overridden with a friendlier pill label (see the next test) — every
# other registry entry must resolve to its own registry `display_name`.
if sid not in ("multistrat", "vrp"):
if sid != "multistrat":
assert display_name(sid) == entry["display_name"]
assert display_name(sid) != sid # never the bare raw id
@@ -20,7 +20,6 @@ def test_every_registry_strategy_has_a_display_name() -> None:
def test_sim_book_overrides_are_plain_language() -> None:
# The interim local pill labels from app.py's `_SIM_BOOK_LABELS` (Task 1), absorbed here.
assert display_name("multistrat") == "ETF Portfolio"
assert display_name("vrp") == "Options income (put spreads)"
def test_unknown_id_falls_back_to_itself_never_raises() -> None:

View File

@@ -26,12 +26,12 @@ def test_every_active_track_declares_a_definition():
def test_observed_mode_is_exactly_the_executed_twins():
# Pin is deliberately exhaustive: whenever a new track adopts record_mode="observed", update this set
# (C2 Task 3 added bybit_4edge_exec; Task 7 added vrp_exec) so the change is reviewed, not silently
# absorbed. `xsfunding` moved OFF observed (Phase 0b venue consolidation: standalone Binance
# `XsFundingForward` -> its own Bybit single-sleeve `recompute` track, mirroring `positioning`) — the
# remaining observed tracks are the OBSERVED-fills executed twins (fills are observed, not recomputable).
# (C2 Task 3 added bybit_4edge_exec) so the change is reviewed, not silently absorbed. `xsfunding` moved
# OFF observed (Phase 0b venue consolidation: standalone Binance `XsFundingForward` -> its own Bybit
# single-sleeve `recompute` track, mirroring `positioning`) — the remaining observed tracks are the
# OBSERVED-fills executed twins (fills are observed, not recomputable).
observed = {sid for sid in STRATEGY_REGISTRY if track_definition(sid)[2] == "observed"}
assert observed == {"multistrat_exec", "bybit_4edge_exec", "vrp_exec"}
assert observed == {"multistrat_exec", "bybit_4edge_exec"}
def test_inception_override_returns_date_when_version_matches(monkeypatch):

View File

@@ -1,20 +0,0 @@
from fxhnt.registry import STRATEGY_REGISTRY
def test_vrp_registry_entry_shape():
e = STRATEGY_REGISTRY["vrp"]
assert e["venue"] == "ibkr-xsp"
assert e["sleeve"] == "equity-vol"
assert e["tier"] == "research"
assert e["execution"] == "paper"
assert e["gate_spec"] == {"gate_type": "reconciliation", "min_forward_days": 21}
assert e["definition"]["record_mode"] == "recompute"
assert e["promote_on_pass"] is False
def test_vrp_exec_twin_is_observed():
from fxhnt.registry import STRATEGY_REGISTRY
e = STRATEGY_REGISTRY["vrp_exec"]
assert e["venue"] == "ibkr-xsp"
assert e["definition"]["record_mode"] == "observed"
assert e["gate_spec"]["gate_type"] == "reconciliation"

View File

@@ -15,11 +15,6 @@ def test_ibkr_recompute_replay_book_reads_ibkr_curve():
assert out == {"multistrat": {0: 0.0, 1: 0.02}}
def test_vrp_recompute_replay_book_reads_ibkr_curve():
out = sim_returns_for("vrp", bybit_returns=_bybit, ibkr_curve=_ibkr)
assert out == {"vrp": {0: 0.0, 1: 0.02}}
def test_sim_books_includes_ibkr():
assert "multistrat" in SIM_BOOKS and "bybit_4edge" in SIM_BOOKS
assert SIM_BOOKS == ("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered", "vrp")
assert SIM_BOOKS == ("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered")

View File

@@ -1,320 +0,0 @@
import datetime as dt
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
from fxhnt.application import vrp_book, vrp_marking, vrp_pricing
from fxhnt.application.allocation_engine import cvar_daily
from fxhnt.application.vrp_book import VrpStrategy
def _seed(tmp_path, days, contracts):
"""contracts: list of (osi, expiry, strike, right, {iso_date: close}). Returns a repo."""
r = XspOptionBarsRepo(f"sqlite:///{tmp_path/'v.db'}")
r.migrate()
at = dt.datetime(2024, 1, 1)
rows = []
for osi, expiry, strike, right, closes in contracts:
for d, c in closes.items():
rows.append(dict(osi_symbol=osi, date=d, expiry=expiry, strike=strike, right=right, close=c))
r.upsert_bars(rows, at)
return r
def _put_mark(k: float, day_index: int) -> float:
"""Realistic put mark: OTM puts (low strike) are cheaper than near-ATM puts (high strike, richer
extrinsic value), rising monotonically with strike toward the forward (470) -- e.g. strike 440 ~= 0.15,
465 ~= 0.90, 470 ~= 1.05. Decays ~3%/day (theta): since it's a PERCENTAGE decay, the higher-dollar
near-ATM leg loses more in absolute terms per day than the cheaper far-OTM leg, which is exactly the
source of a short put-credit-spread's daily theta P&L (spread value = short - long shrinks over time)."""
base = 0.15 + 0.03 * (k - 440.0)
return base * (0.97 ** day_index)
def _straight_chain(start="2024-07-01", n=10, expiry="2024-07-31", forward=470.0):
"""A flat-vol synthetic put chain across strikes (OTM puts cheaper, correctly rising toward the
forward), decaying ~3%/day, plus an ATM call tracking the ATM put 1:1 so parity_forward(K=470) recovers
~470 every day. A short-higher-strike/long-lower-strike credit spread's credit is always positive here
since put price increases monotonically with strike (short leg is worth more than the long leg)."""
days = [(dt.date.fromisoformat(start) + dt.timedelta(days=i)).isoformat() for i in range(n)]
contracts = []
for k in range(440, 475, 5): # put strikes below/at forward
osi = f"XSP{expiry.replace('-','')[2:]}P{k*1000:08d}"
closes = {d: _put_mark(float(k), i) for i, d in enumerate(days)}
contracts.append((osi, expiry, float(k), "P", closes))
# ATM call tracks the ATM (470) put mark exactly so parity_forward(K=470) recovers ~470 every day.
contracts.append((f"XSP{expiry.replace('-','')[2:]}C{470*1000:08d}", expiry, 470.0, "C",
{d: _put_mark(470.0, i) for i, d in enumerate(days)}))
return days, contracts
def test_advance_full_history_is_deterministic(tmp_path):
days, contracts = _straight_chain()
r = _seed(tmp_path, days, contracts)
# a spread must actually open on the entry day -- proves the fixture's marks are realistic (positive
# credit), not vacuously producing an empty series (see module docstring history: inverted put pricing
# used to make every credit <= 0 and _select_spread always return None).
opened = VrpStrategy(r, entry_weekday=0)._select_spread(days[0])
assert opened is not None and opened["credit"] > 0.0
a, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
b, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
assert a == b # same frozen input -> identical series
assert all(isinstance(d, str) and isinstance(x, float) for d, x in a)
assert any(abs(x) > 1e-12 for _, x in a) # a real, non-empty return stream -- not an empty ladder
def test_defined_risk_daily_loss_is_bounded(tmp_path):
# a crash day: the short put mark spikes; the spread loss per day cannot exceed width/margin = 1.0
days, contracts = _straight_chain(n=6)
# overwrite the last day's marks with a crash spike on all puts
crash = days[-1]
for _osi, _expiry, _strike, right, closes in contracts:
if right == "P":
closes[crash] = closes[crash] + 50.0
r = _seed(tmp_path, days, contracts)
series, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
assert series, "series should not be empty"
# a spread must be open and marked before the crash day, so the bound below is tested against a REAL
# marked position (theta-decay P&L), not an empty ladder that trivially satisfies min() >= -1.0.
assert any(abs(x) > 1e-12 for d, x in series if d != crash)
assert min(x for _, x in series) >= -1.0 - 1e-9 # defined-risk: daily loss per margin bounded
assert max(abs(x) for _, x in series) < 5.0 # sane magnitude
def test_atm_pair_selected_by_min_abs_c_minus_p(tmp_path, monkeypatch):
"""Multiple call strikes exist on the same grid as the puts. The true ATM by min|C-P| is 470 (put and
call both mark 1.0 -> diff=0); 460 has a much larger |C-P| (20.0) that would imply a forward ~10 points
off if it were wrongly picked instead. Spy on `parity_forward` to assert the pairing selects k_atm=470
-- proving selection by min|C-P| over sorted common strikes, not the old minimal-strike-distance
cross-product (which ties at diff==0 for every same-strike pair and silently depended on whatever
order `chain_asof` happened to return rows in)."""
expiry, day = "2024-08-16", "2024-07-15"
contracts = [
("P440", expiry, 440.0, "P", {day: 3.0}),
("P450", expiry, 450.0, "P", {day: 2.0}),
("P460", expiry, 460.0, "P", {day: 2.0}),
("P470", expiry, 470.0, "P", {day: 1.0}),
("C460", expiry, 460.0, "C", {day: 22.0}), # |C-P| = 20 at 460 -- would give forward 480 if picked
("C470", expiry, 470.0, "C", {day: 1.0}), # |C-P| = 0 at 470 -- the true ATM
]
r = _seed(tmp_path, [day], contracts)
strat = VrpStrategy(r)
seen: list[dict] = []
real_parity_forward = vrp_pricing.parity_forward
def spy(**kwargs):
seen.append(kwargs)
return real_parity_forward(**kwargs)
monkeypatch.setattr(vrp_book.vrp_pricing, "parity_forward", spy)
strat._select_spread(day)
assert seen, "parity_forward should have been called"
assert seen[0]["k_atm"] == 470.0
assert seen[0]["call_atm"] == 1.0 and seen[0]["put_atm"] == 1.0
def test_advance_deterministic_with_multiple_common_call_strikes(tmp_path):
"""Guard against the old minimal-strike-distance ATM pairing in the full advance() loop: add a second
call strike (460) that shares a strike with a put but whose mark is far off (|C-P|=20 vs 0 at the true
ATM, 470) -- if it were wrongly picked as ATM, the forward and downstream short-strike selection would
differ. Two fresh `advance(None, {})` runs must stay byte-identical."""
days, contracts = _straight_chain(n=10)
# a second call strike (460) whose |C-P| is always 20 (way off the true ATM's ~0) -- tracks the actual
# decaying 460 put mark each day via `_put_mark` so the offset holds on every day, not just day 0.
contracts.append(("XSP240731C00460000", "2024-07-31", 460.0, "C",
{d: _put_mark(460.0, i) + 20.0 for i, d in enumerate(days)}))
r = _seed(tmp_path, days, contracts)
a, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
b, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
assert a == b
assert all(isinstance(d, str) and isinstance(x, float) for d, x in a)
assert any(abs(x) > 1e-12 for _, x in a) # canary: a spread actually opened (not a vacuous empty series)
def test_advance_closes_spread_on_missing_marks_not_zombie(tmp_path):
"""A held spread whose legs' marks disappear (e.g. aged/moved out of the frozen slice) must be CLOSED,
not zombie-held forever via the `val is None -> still.append(sp)` branch -- otherwise it permanently
occupies a ladder slot and starves new entries. Truncate all put marks after day 3 so the spread
entered on day 0 (a Monday) has no mark from day 4 onward; assert it is dropped from `open_spreads`
(not carried forward) and the series stays finite."""
days, contracts = _straight_chain(n=10)
stale_after = days[3] # put marks stop appearing after this day
new_contracts = []
for osi, expiry, strike, right, closes in contracts:
if right == "P":
closes = {d: c for d, c in closes.items() if d <= stale_after}
new_contracts.append((osi, expiry, strike, right, closes))
r = _seed(tmp_path, days, new_contracts)
series, extra = VrpStrategy(r, entry_weekday=0).advance(None, {})
assert series, "series should not be empty"
# a spread must actually have OPENED and been marked while the puts were still fresh (days 1-3), so
# dropping it below proves a real held position was closed -- not that an empty ladder stayed empty.
assert any(abs(x) > 1e-12 for d, x in series if d <= stale_after)
assert all(isinstance(x, float) and x == x and abs(x) != float("inf") for _, x in series)
assert extra["open_spreads"] == [] # closed on missing marks, not zombied
def _bs_chain(days, expiry, forward, sigma, strikes):
"""A Black-Scholes-CONSISTENT put+call chain: put mark = bs_put(F, K, sigma, tau), call mark via
put-call parity C = P + (F - K). Parity recovers F exactly and `implied_vol_put` recovers `sigma` at
every strike -> a clean, smooth vol surface where the MODEL marks reproduce the marks themselves (so a
spread's entry credit equals its model value at entry: no synthetic model-vs-raw basis jump). `forward`
may be a float (constant) or a dict {iso_day: F} for a moving-spot path."""
exp_d = dt.date.fromisoformat(expiry)
fwd_of = (lambda _d: forward) if not isinstance(forward, dict) else (lambda d: forward[d])
contracts = []
for k in strikes:
pcloses, ccloses = {}, {}
for d in days:
tau = max((exp_d - dt.date.fromisoformat(d)).days, 1) / 365.0
f = fwd_of(d)
p = vrp_pricing.bs_put(f, float(k), sigma, tau)
c = p + (f - float(k)) # parity: C - P = F - K
if p > 0.0:
pcloses[d] = p
if c > 0.0:
ccloses[d] = c
tag = expiry.replace("-", "")[2:]
contracts.append((f"XSP{tag}P{int(k * 1000):08d}", expiry, float(k), "P", pcloses))
contracts.append((f"XSP{tag}C{int(k * 1000):08d}", expiry, float(k), "C", ccloses))
return contracts
def _weekdays(start, n):
return [(dt.date.fromisoformat(start) + dt.timedelta(days=i)).isoformat() for i in range(n)]
def test_advance_dollar_basis_is_dollars_over_margin(tmp_path):
"""Regression for the FIX-2 100x-too-small bug, adapted to MODEL marking: `pnl`'s numerator must be in
DOLLARS (per-share model value x100 = the contract multiplier) to match `self.margin`'s dollar basis
(width_points*100). The booked daily return must equal the MODEL value change / width -- NOT that change
/ (width*100) (the old points/dollars basis, 100x smaller).
One spread (ladder=1) opens on day0 of a BS-consistent chain; on day1 the smooth model value has moved by
theta. Assert the booked return equals (credit - model_val_day1)/width and is ~100x the pre-fix basis."""
days = _weekdays("2024-07-01", 2) # Mon, Tue
contracts = _bs_chain(days, "2024-07-31", forward=470.0, sigma=0.20, strikes=range(380, 481, 5))
r = _seed(tmp_path, days, contracts)
strat = VrpStrategy(r, entry_weekday=0, ladder=1)
sp = strat._select_spread(days[0])
assert sp is not None and sp["credit"] > 0.0
# the model value on day1 (independent instance; _spread_value mutates last_iv, so pass a copy)
val1 = VrpStrategy(r, entry_weekday=0, ladder=1)._spread_value(dict(sp), days[1])
assert val1 is not None
series, _ = strat.advance(None, {})
by_day = dict(series)
assert days[1] in by_day
width = strat._width # default 5.0
expected = (sp["credit"] - val1) / width # (Δpts * 100) / (width * 100) = Δpts / width
old_buggy = expected / 100.0 # pre-fix basis: Δpts / (width * 100)
assert abs(by_day[days[1]] - expected) < 1e-9
assert abs(by_day[days[1]] - old_buggy) > 50 * abs(old_buggy) # ~100x larger than the old bug
def test_model_value_bounded_and_smooth_despite_noisy_single_leg(tmp_path):
"""(a) The MODEL spread value is CLIPPED to [0, width] and SMOOTH: it is rebuilt from the day's ATM
forward + ATM IV via Black-Scholes, so corrupting ONE leg's raw OPRA mark by a huge spike (the exact
mark-noise that made the OLD short_close-long_close difference swing 50-100%) does NOT move it -- the ATM
parity/IV are read off the clean at-the-money strike, not the corrupted leg."""
days = _weekdays("2024-07-01", 5)
strikes = list(range(380, 481, 5))
contracts = _bs_chain(days, "2024-08-05", forward=470.0, sigma=0.20, strikes=strikes) # ~35 DTE
strat = VrpStrategy(_seed(tmp_path, days, contracts), entry_weekday=0)
sp = strat._select_spread(days[0])
assert sp is not None
clean_vals = [strat._spread_value(dict(sp), d) for d in days]
for v in clean_vals:
assert v is not None and 0.0 <= v <= strat._width + 1e-12 # hard defined-risk clip
# corrupt ONE non-ATM leg's raw put mark by a +50 spike on day2 (would blow up a raw short-long diff).
long_osi = sp["long_osi"]
l_exp = dt.date.fromisoformat(sp["expiry"])
corrupt = [dict(osi_symbol=long_osi, date=days[2], expiry=l_exp.isoformat(),
strike=sp["long_strike"], right="P", close=999.0)]
r2 = _seed(tmp_path, days, contracts)
r2.upsert_bars(corrupt, dt.datetime(2024, 1, 1))
strat2 = VrpStrategy(r2, entry_weekday=0)
noisy_vals = [strat2._spread_value(dict(sp), d) for d in days]
for v in noisy_vals:
assert v is not None and 0.0 <= v <= strat2._width + 1e-12
# the model value is UNCHANGED by the corrupted leg (parity/IV come from the clean ATM strike).
for cv, nv in zip(clean_vals, noisy_vals, strict=True):
assert abs(cv - nv) < 1e-9
# and the clean path is smooth: no day-over-day jump anywhere near the width.
for a, b in zip(clean_vals, clean_vals[1:], strict=False):
assert abs(a - b) < 0.25 * strat._width
def test_settlement_value_is_clipped_intrinsic(tmp_path):
"""(d) Terminal settlement = clip(K_short - F, 0, width): OTM -> 0, partially ITM -> the in-the-moneyness,
deep ITM -> the width cap. Derived from that day's parity forward (spot), never unbounded."""
day, expiry = "2024-07-30", "2024-07-31" # DTE = 1
short_strike, long_strike, width = 460.0, 455.0, 5.0
for forward, expected in [(470.0, 0.0), (458.0, 2.0), (450.0, 5.0)]:
contracts = _bs_chain([day], expiry, forward=forward, sigma=0.20, strikes=range(400, 501, 5))
strat = VrpStrategy(_seed(tmp_path, [day], contracts), width_points=width)
sp = dict(short_strike=short_strike, long_strike=long_strike, expiry=expiry)
settle = strat._settlement_value(sp, day)
assert settle is not None and abs(settle - expected) < 1e-6
def test_per_spread_cumulative_pnl_bounded_to_defined_risk_max_loss(tmp_path):
"""(b) Per-spread cumulative P&L is bounded to the defined-risk envelope [-(width-credit)*100,
+credit*100] dollars. A spot that crashes deep below both strikes by expiry drives the spread to its MAX
LOSS: with ladder=1 the summed daily returns * margin == the spread's total dollar P&L, which must sit at
(not below) the lower bound -(width-credit)*100."""
days = _weekdays("2024-07-01", 31) # Mon entry, ~30 DTE, held to expiry
expiry = "2024-07-31"
# forward glides 470 -> 400 (deep below any 20Δ short strike & its 5-wide long) by expiry.
fwd = {d: 470.0 - 70.0 * (i / (len(days) - 1)) for i, d in enumerate(days)}
contracts = _bs_chain(days, expiry, forward=fwd, sigma=0.20, strikes=range(360, 481, 5))
strat = VrpStrategy(_seed(tmp_path, days, contracts), entry_weekday=0, ladder=1)
sp0 = strat._select_spread(days[0])
assert sp0 is not None
credit, width = sp0["credit"], strat._width
series, _ = strat.advance(None, {})
total_dollars = sum(x for _, x in series) * strat.margin # ladder=1 -> return = pnl/margin
lo, hi = -(width - credit) * 100.0, credit * 100.0
assert lo - 1e-6 <= total_dollars <= hi + 1e-6 # inside the defined-risk envelope
assert total_dollars < 0.0 # a crash to deep-ITM is a real loss
assert abs(total_dollars - lo) < 1.0 # and it is AT the max-loss bound
def test_no_daily_return_exceeds_plausible_bound_on_clean_vol_path(tmp_path):
"""(c) On a synthetic CLEAN constant-vol path the whole ladder's daily returns are tiny (theta decay),
never the >15%/day mark-noise artifacts (single days of -50%, +59.6%) the raw-diff marks produced."""
days = _weekdays("2024-06-03", 30) # several Monday entries in the [20,45] DTE window
contracts = _bs_chain(days, "2024-07-15", forward=470.0, sigma=0.20, strikes=range(360, 481, 5))
strat = VrpStrategy(_seed(tmp_path, days, contracts), entry_weekday=0)
series, _ = strat.advance(None, {})
assert series and any(abs(x) > 1e-12 for _, x in series) # a real, non-empty return stream
assert max(abs(x) for _, x in series) < 0.10 # no |daily return| > 10%
def test_backtest_and_shared_marking_are_single_source(tmp_path):
"""The backtest (`VrpStrategy._spread_value`) and the live exec twin (`vrp_exec_record`) both value the
spread through the SAME `vrp_marking.model_spread_value` — no drift between the ref and the live
profit-take signal. Assert the strategy's mark equals the shared helper's on the same frozen slice."""
days = _weekdays("2024-07-01", 4)
contracts = _bs_chain(days, "2024-08-05", forward=470.0, sigma=0.20, strikes=range(380, 481, 5))
repo = _seed(tmp_path, days, contracts)
strat = VrpStrategy(repo, entry_weekday=0)
sp = strat._select_spread(days[0])
assert sp is not None
for d in days:
via_strategy = strat._spread_value(dict(sp), d)
via_shared = vrp_marking.model_spread_value(repo, dict(sp), d, strat._width)
assert via_strategy is not None and via_shared is not None
assert abs(via_strategy - via_shared) < 1e-12
def test_fat_left_tail_triggers_cvar_floor():
# A VRP-shaped series: small positive credit-decay most days (net drift ~0), plus occasional deep
# defined-risk losses. cvar_daily averages the worst ceil(0.10*n) returns, so the tail must actually
# be filled with losses: n=50 -> 5 tail slots -> 5 genuine losses. Such a fat-LEFT series has
# cvar_daily/std ABOVE the Gaussian ES-10% multiplier (1.755), which is exactly what makes the merged
# CVaR floor de-lever it (K_cvar < K_vol) where a Gaussian book would be a no-op.
series = [0.06] * 45 + [-0.5] * 5
returns = {str(i): x for i, x in enumerate(series)}
mean = sum(series) / len(series)
std = (sum((x - mean) ** 2 for x in series) / len(series)) ** 0.5
ratio = cvar_daily(returns, 0.10) / std
assert ratio > 1.755 # fatter than Gaussian ES-10% -> CVaR floor bites

View File

@@ -1,80 +0,0 @@
import pytest
from fxhnt.application.vrp_exec_record import (
build_pos_state,
compute_ladder_exec_return,
compute_spread_exec_return,
)
def test_compute_spread_exec_return_envelope_zero_is_none():
assert compute_spread_exec_return(prior_val=1.0, mark_today=0.8, qty=2, envelope=0.0) is None
def test_compute_spread_exec_return_no_mark_today_is_hold_not_phantom_zero():
assert compute_spread_exec_return(prior_val=1.0, mark_today=None, qty=2, envelope=200_000.0) is None
def test_compute_spread_exec_return_winning_spread_is_positive_credit_decay():
# short-vol: credit decayed from 1.00 -> 0.60 = profit. qty=2, $100/contract multiplier.
r = compute_spread_exec_return(prior_val=1.00, mark_today=0.60, qty=2, envelope=200_000.0)
assert r == pytest.approx((1.00 - 0.60) * 100.0 * 2 / 200_000.0)
assert r > 0.0
def test_compute_spread_exec_return_losing_spread_is_negative_and_bounded_by_width():
# a defined-risk put-credit-spread's worst mark is bounded by the wing width ($5 here = $500/contract);
# the value rising toward that cap is a loss, correctly signed negative.
r = compute_spread_exec_return(prior_val=1.00, mark_today=4.50, qty=2, envelope=200_000.0)
assert r == pytest.approx((1.00 - 4.50) * 100.0 * 2 / 200_000.0)
assert r < 0.0
def test_compute_ladder_exec_return_inception_no_prior_rungs_is_none():
assert compute_ladder_exec_return([], {}, 200_000.0) is None
def test_compute_ladder_exec_return_definanced_envelope_is_none():
prior = [{"short_osi": "S1", "long_osi": "L1", "credit": 1.0, "last_val": 1.0, "qty": 2}]
assert compute_ladder_exec_return(prior, {"S1": 0.5}, 0.0) is None
def test_compute_ladder_exec_return_sums_only_marked_rungs():
prior = [
{"short_osi": "S1", "long_osi": "L1", "credit": 1.0, "last_val": 1.0, "qty": 2},
{"short_osi": "S2", "long_osi": "L2", "credit": 1.0, "last_val": 1.0, "qty": 2},
]
marks = {"S1": 0.60, "S2": None} # S2 has no mark today -> HOLD, contributes 0 (not a phantom zero)
r = compute_ladder_exec_return(prior, marks, 200_000.0)
assert r == pytest.approx((1.00 - 0.60) * 100.0 * 2 / 200_000.0) # only S1's marked pnl
def test_compute_ladder_exec_return_all_holds_is_none():
prior = [{"short_osi": "S1", "long_osi": "L1", "credit": 1.0, "last_val": 1.0, "qty": 2}]
assert compute_ladder_exec_return(prior, {"S1": None}, 200_000.0) is None
def test_build_pos_state_round_trip():
opens = [{"entry": "2026-07-06", "expiry": "2026-08-15", "short_osi": "S1", "long_osi": "L1",
"credit": 1.0, "last_val": 0.6, "qty": 2}]
st = build_pos_state(opens, envelope=200_000.0, venue="ibkr-paper", today="2026-07-13")
assert st["open_spreads"] == opens
assert st["envelope"] == 200_000.0
assert st["venue"] == "ibkr-paper"
assert st["last_run_date"] == "2026-07-13"
def test_build_pos_state_defensively_copies_rungs():
opens = [{"entry": "2026-07-06", "expiry": "2026-08-15", "short_osi": "S1", "long_osi": "L1",
"credit": 1.0, "last_val": 0.6, "qty": 2}]
st = build_pos_state(opens, envelope=200_000.0, venue="ibkr-paper", today="2026-07-13")
opens.append({"short_osi": "S2"}) # mutate the caller's list after the call
opens[0]["last_val"] = 999.0 # mutate a rung dict in the caller's list
assert len(st["open_spreads"]) == 1
assert st["open_spreads"][0]["last_val"] == 0.6
def test_build_pos_state_empty_ladder():
st = build_pos_state([], envelope=0.0, venue="", today="2026-07-13")
assert st["open_spreads"] == []
assert st["envelope"] == 0.0

View File

@@ -1,245 +0,0 @@
"""Composition test for the ladder-cap-breach-under-close-failure defect: `plan_and_record_vrp` used to
decide whether to open a new rung from `plan_vrp_ladder`'s PRE-EXECUTION plan (`may_open`), computed before
any order was placed. If a planned CLOSE then failed at the broker (e.g. an illiquid near-expiry contract
rejected), the closing rung was re-added to `open_spreads_next` UNCHANGED, but the OPEN still went ahead
because it was gated on the stale `may_open` plan -- breaching the ladder cap (6 open rungs instead of 5).
Worse, since DTE only decreases, the un-closable rung would be re-planned-for-close (and re-free a phantom
slot) every subsequent run, growing the ladder unbounded behind the broker-reject.
This test drives the REAL `plan_and_record_vrp` composition: the real `ForwardNavRepo`/`XspOptionBarsRepo`/
`VrpStrategy` against a throwaway sqlite file, a fake OPRA `dbn` source, and a fake broker whose CLOSE raises
but whose OPEN would otherwise succeed -- then asserts the post-run ladder never exceeds its cap."""
import datetime as dt
from fxhnt.adapters.persistence.forward_nav import ForwardNavRepo
from fxhnt.application.vrp_exec_record import plan_and_record_vrp
from fxhnt.application.vrp_pricing import bs_put
from fxhnt.domain.models import OptionContract
LADDER = 5 # matches STRATEGY_REGISTRY["vrp"]["definition"]["params"]["ladder"]
TODAY = "2026-07-13" # a Monday -> VrpStrategy's default entry_weekday=0
NEAR_EXPIRY = (dt.date.fromisoformat(TODAY) + dt.timedelta(days=1)).isoformat() # DTE=1 -> planned close
NEW_EXPIRY = (dt.date.fromisoformat(TODAY) + dt.timedelta(days=30)).isoformat() # ~30 DTE -> new candidate
_FORWARD = 470.0
_TAU = 30 / 365.0
_SIGMA = 0.20 # flat-vol BS put marks -> monotonic in strike (real credit, real ~20-delta pick)
class _TestRepo(ForwardNavRepo):
"""The real forward-nav repo (backing `run_track`'s anchor/book-state/record machinery), pointed at a
throwaway sqlite file and stamped with `_dsn` so `plan_and_record_vrp` never falls back to
`get_settings().operational_dsn` (no live Postgres in this test)."""
def __init__(self, dsn: str) -> None:
super().__init__(dsn)
self._dsn = dsn
class _FakeDbn:
"""OPRA stand-in: a DTE=1 spread (the rung about to be closed) plus a ~30-DTE put ladder + ATM call (the
new-candidate universe). The 30-DTE puts are priced off real Black-Scholes (`bs_put`, flat 20% vol) so
they are monotonically increasing in strike -- a genuine (K_short, K_short-5) vertical has POSITIVE
credit and a real ~20-delta short strike, unlike a hand-waved linear-in-strike toy price."""
def last_available_option_date(self) -> dt.date:
# OPRA's available-end == TODAY here (this fake's slice IS complete for TODAY), so
# `_freeze_latest_session` freezes TODAY on its first candidate -> freeze_date == today.
return dt.date.fromisoformat(TODAY)
def fetch_option_chain(self, _parent: str, _as_of: str) -> list[OptionContract]:
contracts = [
OptionContract("XSPNEARP00460000", dt.date.fromisoformat(NEAR_EXPIRY), 460.0, "P"),
OptionContract("XSPNEARP00455000", dt.date.fromisoformat(NEAR_EXPIRY), 455.0, "P"),
OptionContract("XSPNEWC00470000", dt.date.fromisoformat(NEW_EXPIRY), 470.0, "C"),
]
for k in range(420, 475, 5):
contracts.append(OptionContract(f"XSPNEWP{k*1000:08d}", dt.date.fromisoformat(NEW_EXPIRY), float(k), "P"))
return contracts
def fetch_option_bars(self, osi_symbols: list[str], start: str, _end: str) -> dict[str, dict[str, float]]:
atm_put = round(bs_put(_FORWARD, _FORWARD, _SIGMA, _TAU), 4) # K==F -> C==P at the forward (parity)
marks = {"XSPNEARP00460000": 1.0, "XSPNEARP00455000": 0.5, "XSPNEWC00470000": atm_put}
for k in range(420, 475, 5):
marks[f"XSPNEWP{k*1000:08d}"] = round(bs_put(_FORWARD, float(k), _SIGMA, _TAU), 4)
return {osi: {start: marks[osi]} for osi in osi_symbols if osi in marks}
class _CapFakeBroker:
"""Distinguishes CLOSE vs OPEN by `build_close_legs`/`build_combo_legs`'s documented sign convention
(close BUYS the short back first; open SELLs the short first): the CLOSE always raises (an illiquid
near-expiry contract rejected by the broker); an OPEN would otherwise succeed."""
def __init__(self) -> None:
self.close_calls = 0
self.open_calls = 0
def place_combo(self, legs: list[tuple[str, str, int]], _net_limit: float) -> str:
first_action = legs[0][1]
if first_action == "BUY":
self.close_calls += 1
raise RuntimeError("broker reject: illiquid near-expiry contract")
self.open_calls += 1
return "ORDER-OPEN-1"
def _held_rung(i: int) -> dict:
"""A held rung far from expiry/profit-take, with an osi that is never frozen -> `_mark_spread` returns
None -> plan_vrp_ladder HOLDs it (not a close), regardless of its (irrelevant) credit/expiry values."""
return {"entry": "2026-06-01", "expiry": "2026-12-01", "short_osi": f"HELD-S{i}", "long_osi": f"HELD-L{i}",
"credit": 1.0, "last_val": 1.0, "qty": 1}
def _near_expiry_rung() -> dict:
return {"entry": "2026-07-06", "expiry": NEAR_EXPIRY, "short_osi": "XSPNEARP00460000",
"long_osi": "XSPNEARP00455000", "credit": 1.0, "last_val": 1.0, "qty": 1}
def _seed_full_ladder(repo: ForwardNavRepo, at: dt.datetime) -> None:
open_spreads = [_near_expiry_rung()] + [_held_rung(i) for i in range(4)]
assert len(open_spreads) == LADDER # ladder starts FULL (5/5)
repo.set_book_state("vrp_exec_pos", {"open_spreads": open_spreads, "envelope": 1_000_000.0,
"venue": "ibkr-xsp", "last_run_date": "2026-07-06"}, at=at)
def test_ladder_cap_not_breached_when_close_fails(tmp_path):
dsn = f"sqlite:///{tmp_path / 'ladder_cap.db'}"
repo = _TestRepo(dsn)
repo.migrate()
at = dt.datetime(2026, 7, 13)
_seed_full_ladder(repo, at)
broker = _CapFakeBroker()
result = plan_and_record_vrp(
repo, dbn=_FakeDbn(), broker=broker, nlv=1_000_000.0, today=TODAY, at=at,
paper_envelope=1_000_000.0, account_is_paper=True, venue="ibkr-xsp", execute=True)
# Sanity: the planner DID plan to close the near-expiry rung (reproducing the exact setup the defect
# requires -- may_open was True from the pre-execution plan, since remaining < ladder once this run's
# close is applied).
assert result["to_close"] == 1
# The CLOSE was attempted and failed at the broker.
assert broker.close_calls == 1
# THE FIX: the OPEN must be gated on actual post-close occupancy. The failed close re-added its rung
# unchanged, so the ladder is still full (5/5) -- the new rung must NOT be opened.
assert broker.open_calls == 0, "OPEN must not be placed when a failed close leaves the ladder at cap"
# The persisted book-state must never exceed the ladder cap.
persisted = repo.get_book_state("vrp_exec_pos")
assert len(persisted["open_spreads"]) == LADDER
assert result["open_spreads"] == LADDER
# The skip must be visible in the result, not silently dropped.
assert any("ladder full" in e for e in result["errors"])
assert result["spread"] is None
def test_ladder_opens_normally_when_close_succeeds(tmp_path):
"""Companion sanity check: with a broker that never rejects, the same setup DOES free a slot and open
the new rung -- proving the fix only back-pressures on an actual close FAILURE, not on every close."""
class _AlwaysSucceedsBroker:
def __init__(self) -> None:
self.close_calls = 0
self.open_calls = 0
def place_combo(self, legs, _net_limit):
if legs[0][1] == "BUY":
self.close_calls += 1
else:
self.open_calls += 1
return "ORDER-OK"
dsn = f"sqlite:///{tmp_path / 'ladder_ok.db'}"
repo = _TestRepo(dsn)
repo.migrate()
at = dt.datetime(2026, 7, 13)
_seed_full_ladder(repo, at)
broker = _AlwaysSucceedsBroker()
result = plan_and_record_vrp(
repo, dbn=_FakeDbn(), broker=broker, nlv=1_000_000.0, today=TODAY, at=at,
paper_envelope=1_000_000.0, account_is_paper=True, venue="ibkr-xsp", execute=True)
assert broker.close_calls == 1
assert broker.open_calls == 1
assert result["open_spreads"] == LADDER # one closed, one opened -> still 5, not 6
# --- OPRA publish-lag: freeze the last COMPLETE session (T-1), not today's unpublished slice --------------
_FREEZE_DATE = "2026-07-10" # _prev_trading_day("2026-07-13" Monday) -> Friday (T-1 complete session)
_LAG_EXPIRY = "2026-08-19" # ~40 DTE from the frozen session -> in the put band + calls band
class _LagDbn:
"""OPRA publishes on a ~1-session lag: TODAY's slice is still settling, so a freeze for TODAY raises
`DatabentoRangeUnavailable` (exactly the live 07:42 UTC crash), while the prior COMPLETE session freezes
fine. `last_available_option_date` reports TODAY (the naive available-end); `_freeze_latest_session` must
step back one trading day to _FREEZE_DATE. A put ladder + ATM call priced off flat-vol Black-Scholes."""
def __init__(self) -> None:
self.chain_calls: list[str] = []
def last_available_option_date(self) -> dt.date:
return dt.date.fromisoformat(TODAY)
def fetch_option_chain(self, _parent: str, as_of: str) -> list[OptionContract]:
self.chain_calls.append(as_of)
if as_of == TODAY: # today's session is not published yet -> unavailable
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
raise DatabentoRangeUnavailable(f"requested start {as_of} >= available end {_FREEZE_DATE}")
contracts = [OptionContract("XSPLAGC00470000", dt.date.fromisoformat(_LAG_EXPIRY), 470.0, "C")]
for k in range(420, 475, 5):
contracts.append(OptionContract(f"XSPLAGP{k*1000:08d}", dt.date.fromisoformat(_LAG_EXPIRY),
float(k), "P"))
return contracts
def fetch_option_bars(self, osi_symbols: list[str], start: str, _end: str) -> dict[str, dict[str, float]]:
tau = 40 / 365.0
atm_put = round(bs_put(470.0, 470.0, 0.20, tau), 4)
marks = {"XSPLAGC00470000": atm_put}
for k in range(420, 475, 5):
marks[f"XSPLAGP{k*1000:08d}"] = round(bs_put(470.0, float(k), 0.20, tau), 4)
return {osi: {start: marks[osi]} for osi in osi_symbols if osi in marks}
class _NoopBroker:
def place_combo(self, _legs, _net): # never called: held rung, no new entry (envelope 0)
raise AssertionError("broker must not be called in the lag test")
def test_freeze_uses_last_complete_session_not_today(tmp_path):
"""The live production bug: `plan_and_record_vrp` froze `today`, whose OPRA slice is NEVER published
intraday -> DatabentoRangeUnavailable before the run reached the broker. FIX: freeze the last COMPLETE
session (T-1) via `_freeze_latest_session`, but keep `today` for DTE / booking. Assert the run does not
crash, freezes T-1 (marks present at _FREEZE_DATE, absent at TODAY), and still marks + records under today."""
dsn = f"sqlite:///{tmp_path / 'lag.db'}"
repo = _TestRepo(dsn)
repo.migrate()
at = dt.datetime(2026, 7, 13)
# a prior held rung whose legs are real band contracts (frozen at T-1) -> it marks -> exec_return records.
held = {"entry": "2026-06-15", "expiry": _LAG_EXPIRY, "short_osi": "XSPLAGP00450000",
"long_osi": "XSPLAGP00445000", "credit": 1.0, "last_val": 1.0, "qty": 1}
repo.set_book_state("vrp_exec_pos", {"open_spreads": [held], "envelope": 1_000_000.0,
"venue": "ibkr-xsp", "last_run_date": "2026-07-06"}, at=at)
dbn = _LagDbn()
result = plan_and_record_vrp(
repo, dbn=dbn, broker=_NoopBroker(), nlv=1_000_000.0, today=TODAY, at=at,
paper_envelope=0.0, account_is_paper=True, venue="ibkr-xsp", execute=True) # envelope 0 -> no new entry
# TODAY was attempted (raised) and the run stepped back to the last complete session.
assert TODAY in dbn.chain_calls
assert _FREEZE_DATE in dbn.chain_calls
# marks were frozen at T-1, NOT at today (today's slice is unpublished).
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
bars = XspOptionBarsRepo(dsn)
assert bars.bars_for(["XSPLAGP00450000"], _FREEZE_DATE, _FREEZE_DATE) # frozen at the complete session
assert not bars.bars_for(["XSPLAGP00450000"], TODAY, TODAY) # nothing under today
# the prior rung still marked and the exec return was recorded under `today` (booking/calendar date).
assert result["exec_return"] is not None
persisted = repo.get_book_state("vrp_exec_pos")
assert persisted["last_run_date"] == TODAY
assert len(persisted["open_spreads"]) == 1 # held (not closed, not stacked)

View File

@@ -1,136 +0,0 @@
import pytest
from fxhnt.application.vrp_exec_record import (
build_close_legs,
build_combo_legs,
plan_vrp_ladder,
refuse_if_live,
rung_notional,
)
def test_build_combo_legs_is_defined_risk_credit():
legs, net = build_combo_legs(short_osi="XSP..P00470000", long_osi="XSP..P00465000",
short_price=1.20, long_price=0.40, qty=1)
assert ("XSP..P00470000", "SELL", 1) in legs
assert ("XSP..P00465000", "BUY", 1) in legs
assert net == pytest.approx(-0.80) # negative limit = net credit received
def test_build_close_legs_reverses_open_legs_with_positive_debit():
legs, net = build_close_legs(short_osi="XSP..P00470000", long_osi="XSP..P00465000", qty=1, mark_today=0.30)
assert ("XSP..P00470000", "BUY", 1) in legs # BUY back the short
assert ("XSP..P00465000", "SELL", 1) in legs # SELL the long
assert net == pytest.approx(0.30) # positive limit = debit paid to close
def _rung(*, short_osi="S1", long_osi="L1", credit=1.0, qty=2, expiry="2026-09-01"):
return {"entry": "2026-07-06", "expiry": expiry, "short_osi": short_osi, "long_osi": long_osi,
"credit": credit, "last_val": credit, "qty": qty}
def test_plan_vrp_ladder_no_stacking_when_ladder_is_full():
# 5 rungs already open, none hitting profit-take/DTE -> no closes -> ladder stays full -> may NOT open.
opens = [_rung(short_osi=f"S{i}") for i in range(5)]
marks = {sp["short_osi"]: 0.9 for sp in opens} # well above the 50% profit-take line, DTE far out
to_close, may_open, slots = plan_vrp_ladder(opens, marks, marks, "2026-07-06", ladder=5, entry_weekday=0)
assert to_close == []
assert may_open is False
assert slots == 0
def test_plan_vrp_ladder_closes_rung_at_dte_le_1():
sp = _rung(expiry="2026-07-07") # DTE = 1 on "2026-07-06"
# signal above the profit line: the close is driven by DTE<=1, not profit-take; paired with the ACTUAL mark.
to_close, _may_open, _slots = plan_vrp_ladder([sp], {"S1": 0.9}, {"S1": 0.9}, "2026-07-06", ladder=5,
entry_weekday=0)
assert len(to_close) == 1
assert to_close[0][0] is sp
assert to_close[0][1] == 0.9 # paired value is the ACTUAL mark (the debit to close)
def test_plan_vrp_ladder_closes_rung_at_profit_take_on_model_signal():
sp = _rung(credit=1.0)
# signal_mark <= (1 - profit_take) * credit == 0.50 -> closes; paired with the actual mark (here also 0.40).
to_close, _may_open, _slots = plan_vrp_ladder([sp], {"S1": 0.40}, {"S1": 0.40}, "2026-07-06", ladder=5,
profit_take=0.5, entry_weekday=0)
assert len(to_close) == 1
def test_plan_vrp_ladder_noisy_actual_mark_does_not_trip_profit_take():
"""THE FIX (MINOR-1): a noisy far-OTM long-leg spike drags the RAW shortlong actual mark down under the
50%-profit line, but the SMOOTH model signal stays above it — so no spurious close. The close decision must
use the model signal, not the actual mark."""
sp = _rung(credit=1.0)
actual = {"S1": 0.30} # raw shortlong, dragged low by a long-leg quote spike -> would false-trip
signal = {"S1": 0.90} # smooth model value, well above the 0.50 profit line
to_close, _may_open, _slots = plan_vrp_ladder([sp], actual, signal, "2026-07-06", ladder=5,
profit_take=0.5, entry_weekday=0)
assert to_close == [] # model signal governs -> no spurious profit-take close
# and when the MODEL genuinely shows 50% profit, it DOES close (at the actual mark).
to_close2, _m, _s = plan_vrp_ladder([sp], {"S1": 0.55}, {"S1": 0.45}, "2026-07-06", ladder=5,
profit_take=0.5, entry_weekday=0)
assert len(to_close2) == 1 and to_close2[0][1] == 0.55
def test_plan_vrp_ladder_holds_rung_with_no_actual_mark_today():
sp = _rung()
# no ACTUAL mark -> HOLD (cannot close without a real price), regardless of the model signal.
to_close, _may_open, _slots = plan_vrp_ladder([sp], {"S1": None}, {"S1": 0.10}, "2026-07-06", ladder=5,
entry_weekday=0)
assert to_close == []
def test_plan_vrp_ladder_may_open_on_entry_day_with_free_slot():
to_close, may_open, slots = plan_vrp_ladder([], {}, {}, "2026-07-06", ladder=5, entry_weekday=0) # Monday
assert to_close == []
assert may_open is True
assert slots == 1
def test_plan_vrp_ladder_no_open_off_entry_day():
to_close, may_open, slots = plan_vrp_ladder([], {}, {}, "2026-07-07", ladder=5, entry_weekday=0) # Tuesday
assert may_open is False
assert slots == 0
def test_plan_vrp_ladder_open_frees_up_when_a_close_makes_room():
# ladder full (5/5), but one rung closes this run (DTE<=1) -> a slot opens for a new entry.
opens = [_rung(short_osi=f"S{i}", expiry="2026-07-07") if i == 0 else _rung(short_osi=f"S{i}")
for i in range(5)]
marks = {sp["short_osi"]: 0.9 for sp in opens}
to_close, may_open, slots = plan_vrp_ladder(opens, marks, marks, "2026-07-06", ladder=5, entry_weekday=0)
assert len(to_close) == 1
assert may_open is True
assert slots == 1
def test_rung_notional_splits_envelope_across_ladder_and_bounds_total():
per_rung = rung_notional(1_000_000.0, 5)
assert per_rung == pytest.approx(200_000.0)
assert per_rung * 5 == pytest.approx(1_000_000.0) # total open notional bounded at ~envelope
class _RaisingBroker:
"""A fake broker whose place_combo must NEVER be called by a refused (non-paper, real-capital) run."""
def __init__(self) -> None:
self.calls = 0
def place_combo(self, legs, net_limit): # noqa: ANN001 - test double
self.calls += 1
raise AssertionError("place_combo must not be called when the paper-envelope guard refuses")
def test_refuse_if_live_blocks_non_paper_account_and_places_zero_orders():
broker = _RaisingBroker()
with pytest.raises(ValueError, match="refused"):
refuse_if_live(paper_envelope=1_000_000, account_is_paper=False)
assert broker.calls == 0
def test_refuse_if_live_allows_paper_account():
refuse_if_live(paper_envelope=1_000_000, account_is_paper=True) # must not raise
def test_refuse_if_live_allows_zero_envelope_on_any_account():
refuse_if_live(paper_envelope=0.0, account_is_paper=False) # off = no real-capital exposure guard

View File

@@ -1,43 +0,0 @@
import math
import pytest
from fxhnt.application.vrp_pricing import bs_put, implied_vol_put, put_delta, parity_forward
def test_bs_put_atm_positive_and_bounded():
# ATM put (F=K) has value between 0 and K; ~0.4*F*sigma*sqrt(tau) for small tau
p = bs_put(F=100.0, K=100.0, sigma=0.20, tau=30 / 365)
assert 0.0 < p < 100.0
assert p == pytest.approx(0.4 * 100.0 * 0.20 * math.sqrt(30 / 365), rel=0.05)
def test_bs_put_monotone_in_strike():
lo = bs_put(F=100.0, K=90.0, sigma=0.2, tau=0.1)
hi = bs_put(F=100.0, K=110.0, sigma=0.2, tau=0.1)
assert hi > lo # higher strike put is worth more
def test_implied_vol_round_trip():
price = bs_put(F=100.0, K=95.0, sigma=0.27, tau=45 / 365)
iv = implied_vol_put(price=price, F=100.0, K=95.0, tau=45 / 365)
assert iv == pytest.approx(0.27, abs=1e-4)
def test_implied_vol_returns_none_on_arbitrage():
# a price below intrinsic (K-F, discounted) has no arb-free IV
assert implied_vol_put(price=0.0001, F=100.0, K=130.0, tau=0.1) is None
def test_put_delta_otm_near_target():
# ~20-delta OTM put sits below spot; delta magnitude in (0,0.5)
d = put_delta(F=100.0, K=94.0, sigma=0.20, tau=30 / 365)
assert 0.0 < d < 0.5
def test_parity_forward_recovers_forward():
F = 100.0
tau = 0.1
sigma = 0.2
# build synthetic ATM call/put at K=100 consistent with forward F via parity: C - P = F - K
p = bs_put(F=F, K=100.0, sigma=sigma, tau=tau)
c = p + (F - 100.0)
assert parity_forward(call_atm=c, put_atm=p, k_atm=100.0) == pytest.approx(F, abs=1e-6)

View File

@@ -1,28 +0,0 @@
"""Task 7: the VRP CronJob YAML must parse and ship SUSPENDED — it places live-adjacent combo orders and the
IBKR account has no options permission yet, so it must not be armable by an accidental merge."""
from __future__ import annotations
from pathlib import Path
import yaml
_REPO_ROOT = Path(__file__).resolve().parents[2]
_CRONJOB_PATH = _REPO_ROOT / "infra" / "k8s" / "jobs" / "fxhnt-vrp-rebalancer.yaml"
_BACKFILL_PATH = _REPO_ROOT / "infra" / "k8s" / "jobs" / "fxhnt-opra-backfill.yaml"
def test_vrp_rebalancer_cronjob_suspended():
docs = list(yaml.safe_load_all(_CRONJOB_PATH.read_text()))
cronjobs = [d for d in docs if d and d.get("kind") == "CronJob"]
assert len(cronjobs) == 1
cj = cronjobs[0]
assert cj["metadata"]["name"] == "fxhnt-vrp-rebalancer"
assert cj["spec"]["suspend"] is True
def test_opra_backfill_yaml_is_valid_and_is_a_bare_job():
docs = [d for d in yaml.safe_load_all(_BACKFILL_PATH.read_text()) if d]
jobs = [d for d in docs if d.get("kind") == "Job"]
assert len(jobs) == 1
assert jobs[0]["metadata"]["name"] == "fxhnt-opra-backfill"
assert "suspend" not in jobs[0]["spec"] # bare Jobs have no suspend gate — manual apply only

View File

@@ -1,109 +0,0 @@
import datetime as dt
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
def _repo(tmp_path):
r = XspOptionBarsRepo(f"sqlite:///{tmp_path/'o.db'}")
r.migrate()
return r
def _bar(osi, date, expiry, strike, right, close):
return dict(osi_symbol=osi, date=date, expiry=expiry, strike=strike, right=right, close=close)
def test_upsert_and_bars_for(tmp_path):
r = _repo(tmp_path)
at = dt.datetime(2024, 7, 15)
r.upsert_bars([_bar("XSP240816P00470000", "2024-07-15", "2024-08-16", 470.0, "P", 1.20),
_bar("XSP240816P00470000", "2024-07-16", "2024-08-16", 470.0, "P", 1.35)], at)
assert r.bars_for(["XSP240816P00470000"], "2024-07-15", "2024-07-16") == {
"XSP240816P00470000": {"2024-07-15": 1.20, "2024-07-16": 1.35}}
def test_upsert_is_idempotent(tmp_path):
r = _repo(tmp_path)
at = dt.datetime(2024, 7, 15)
r.upsert_bars([_bar("A", "2024-07-15", "2024-08-16", 470.0, "P", 1.20)], at)
r.upsert_bars([_bar("A", "2024-07-15", "2024-08-16", 470.0, "P", 1.25)], at) # same PK -> overwrite
assert r.bars_for(["A"], "2024-07-15", "2024-07-15") == {"A": {"2024-07-15": 1.25}}
def test_chain_asof_filters_by_dte_window(tmp_path):
r = _repo(tmp_path)
at = dt.datetime(2024, 7, 15)
r.upsert_bars([
_bar("P30", "2024-07-15", "2024-08-14", 470.0, "P", 1.2), # 30 DTE -> in [20,45]
_bar("P10", "2024-07-15", "2024-07-25", 470.0, "P", 0.4), # 10 DTE -> out
_bar("P60", "2024-07-15", "2024-09-13", 470.0, "P", 2.0), # 60 DTE -> out
], at)
got = {c.osi_symbol for c in r.chain_asof("2024-07-15", 20, 45)}
assert got == {"P30"}
# ---- preload (replay N+1 -> one-query in-memory mirror) correctness ---------------------------------
_DAYS = ["2024-07-15", "2024-07-16", "2024-07-17", "2024-07-18"]
def _multi_day_rows():
rows = []
for di, day in enumerate(_DAYS):
for exp in ["2024-08-05", "2024-08-16", "2024-09-13"]:
for strike in [460.0, 470.0, 480.0]:
for right in ("P", "C"):
osi = f"XSP-{exp}-{int(strike)}-{right}"
rows.append(_bar(osi, day, exp, strike, right, round(1.0 + di * 0.1 + strike * 0.001, 4)))
return rows
def _load(tmp_path, name, rows):
r = XspOptionBarsRepo(f"sqlite:///{tmp_path / name}")
r.migrate()
r.upsert_bars(rows, dt.datetime(2024, 1, 1))
return r
def _chain(r, day, lo, hi, right):
return [(c.osi_symbol, c.expiry, c.strike, c.right) for c in r.chain_asof(day, lo, hi, right)]
def test_preload_full_mirror_matches_db_exactly(tmp_path):
"""The preloaded (in-memory) path must return results IDENTICAL to the per-day DB path — same
contracts, same strike order, same marks, same date filtering — for every query shape the replay uses."""
rows = _multi_day_rows()
db = _load(tmp_path, "db.db", rows) # never preloaded -> DB path
ca = _load(tmp_path, "ca.db", rows)
ca.preload() # full mirror (after=None)
assert ca.trading_days(None) == db.trading_days(None)
assert ca.trading_days("2024-07-16") == db.trading_days("2024-07-16")
for day in _DAYS:
for lo, hi in ((20, 45), (0, 60)):
for right in ("P", "C"):
assert _chain(ca, day, lo, hi, right) == _chain(db, day, lo, hi, right)
osis = [rows[0]["osi_symbol"], rows[20]["osi_symbol"], "MISSING-OSI"]
assert ca.bars_for(osis, "2024-07-16", "2024-07-16") == db.bars_for(osis, "2024-07-16", "2024-07-16")
assert ca.bars_for(osis, "2024-07-15", "2024-07-18") == db.bars_for(osis, "2024-07-15", "2024-07-18")
assert ca.bars_for([], "2024-07-15", "2024-07-15") == db.bars_for([], "2024-07-15", "2024-07-15")
def test_preload_scoped_covers_range_and_falls_back_outside(tmp_path):
"""A scoped preload (after=X) mirrors days > X and, critically, FALLS BACK to the DB for any date at
or before X — so it can never under-return an out-of-range query."""
rows = _multi_day_rows()
db = _load(tmp_path, "db2.db", rows)
sc = _load(tmp_path, "sc.db", rows)
sc.preload(after="2024-07-16") # covers 07-17, 07-18 in memory; 07-15/07-16 -> DB fallback
# covered days: served from cache, identical to DB
for day in ("2024-07-17", "2024-07-18"):
assert _chain(sc, day, 20, 45, "P") == _chain(db, day, 20, 45, "P")
osis = [rows[0]["osi_symbol"]]
assert sc.bars_for(osis, "2024-07-17", "2024-07-17") == db.bars_for(osis, "2024-07-17", "2024-07-17")
assert sc.trading_days("2024-07-16") == db.trading_days("2024-07-16")
# out-of-range (<= bound): must fall back to DB and still be correct, NOT return empty
assert _chain(sc, "2024-07-15", 20, 45, "P") == _chain(db, "2024-07-15", 20, 45, "P")
assert _chain(sc, "2024-07-15", 20, 45, "P") # non-empty proof the fallback fired
assert sc.bars_for(osis, "2024-07-15", "2024-07-15") == db.bars_for(osis, "2024-07-15", "2024-07-15")
assert sc.trading_days(None) == db.trading_days(None) # after=None under scoped preload -> DB fallback, all days