Compare commits
41 Commits
fix/ucits-
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
| 0d8ce554a3 | |||
| 7da51863a1 | |||
| 05f6d624c0 | |||
| 40c0b187a7 | |||
| ee077a2cc1 | |||
| e00bac067f | |||
| 9d35df6bfc | |||
| da28c049f7 | |||
| 6677f73a64 | |||
| 330bb56b52 | |||
| 9f2b47a789 | |||
| 80bf243023 | |||
| 6cf19ec379 | |||
| 7d708b03b1 | |||
| 7318f77054 | |||
| 6d8bcc3ae7 | |||
| 0069b75d0e | |||
| 603aaaabff | |||
| 9b2b25433e | |||
| 6e9dfec067 | |||
| c98b8d4b49 | |||
| fc8b1b7849 | |||
| 5d15b0f801 | |||
| bf99ce2bd1 | |||
| 2f957bb6a6 | |||
| aa85ffd3a3 | |||
| baf6ddbe9e | |||
| c7a40b70ec | |||
| 92b0871050 | |||
| adf3edb054 | |||
| fc663215ef | |||
| 5b5fbb0e0b | |||
| eb6ecf8272 | |||
| b71cd6b3b0 | |||
| 59cc06e594 | |||
| a263b676de | |||
| fb14ddc75d | |||
| 8df02c2198 | |||
| 39b313800d | |||
| a16b341d90 | |||
| 9baf1f8115 |
@@ -0,0 +1,119 @@
|
||||
# fxhnt — Fund / Tax / Accounting Research Brief (2026-07-20)
|
||||
|
||||
**> This is researched context, NOT legal, tax, or financial advice.** The Dutch tax rules that govern this the most (Box 3 wealth taxation and the FGR fund regime) are in **active reform** as of mid-2026: Box 3's actual-return system is legislatively unsettled (Eerste Kamer voting paused) and the FGR definition was rewritten Jan 2025 with further consultation open. Every rate, threshold, and classification below must be confirmed with a licensed Dutch belastingadviseur / fiscaal jurist against the finalized Belastingdienst publications for your actual filing year before you rely on it. The single highest-stakes item — whether your systematic/algorithmic trading is Box 3 or Box 1 — is genuinely unsettled case law and needs a written advisor opinion (and possibly a vooroverleg with the Belastingdienst) *before* you scale real capital.
|
||||
|
||||
**> STATE NOTE (2026-07-20): the fund is STILL PAPER — no live book yet** (IBKR DU9600528 is a paper account; Bybit exec never armed). This is go-live planning from euro zero, and the Bybit ~1 Jul 2026 EEA change (§5.15) is **not** a live-position emergency — just a reason to open the right Bybit-EU (EUR-not-USDT) account before funding crypto.
|
||||
|
||||
**> RESOLVED STRUCTURE (2026-07-20) — supersedes §1's "start personal, then decide" and §3/§4's BV-from-zero modelling.** The operator surfaced that they **already have an IT/tech-BV + holding + a DGA-salary**. That collapses the whole Box-1-vs-Box-3 gamble into a clean, already-in-place structure:
|
||||
|
||||
> **Tech-BV = ALL the arbeid** (build/run the system, buy/develop technology) — already taxed there (Vpb 19% + the existing DGA loon, so *no extra €58k gebruikelijk-loon is needed*). → BV pays **dividend** (Vpb 19% + Box 2 24.5%, cleanly taxed) to private. → that al-belaste private money is invested in the **fund as passive capital → Box 3** (~6% forfait × 36% ≈ 2.2%/yr of assets, regardless of real return).
|
||||
|
||||
This **removes the Box-1 reclassification risk entirely** — the only real threat — because the danger was ever *arbeid + winst in one hand* (what sank GHARL:2024:4613). Here the labour is taxed in the BV and the private capital does nothing but *renderen*: exactly the "normaal vermogensbeheer" the daytrader cases won on. **No new entity, no fund wrapper, no AIFMD (own money). Dividend is an option the operator HAS, not a forced cost.** Two design guards remain (not risks): keep the fund genuinely passive → price any tech-BV→fund service **at arm's length**; and time the dividend for the Box 2 bracket. So §1's staged "personal → BV → fund" path and §3/§4's break-even math are **background theory** for this operator — the resolved route above is what applies. The advisor conversation shifts from *"which box?"* to *"how do I price the tech-BV→fund service so the fund stays demonstrably passive?"*
|
||||
|
||||
**> ACCOUNTING NOTE: EUR is the functional currency, no conversion pipeline.** IBKR is EUR-native and Bybit-EU is EUR too → there is no USD leg. §2's FX-revaluation subsystem (realized/unrealized-FX split, ECB-rate join, USDT proxy) is **downgraded from core to dormant insurance**: keep a `currency` + `fx_rate_to_eur` column per posting (default EUR/1.0) so a stray non-EUR line is representable, but build no conversion pipeline. Pure-EUR book = no FX P&L = cleaner books and cleaner tax.
|
||||
|
||||
---
|
||||
|
||||
# Decision brief: bookkeeping, profit extraction, and structure for an NL-resident agentic quant fund
|
||||
|
||||
## 1. Bottom line up front
|
||||
|
||||
**Start personal (Box 3), keep your own capital, and build the accounting system now — but get a written advisor opinion on the Box 1 reclassification risk before you go live with real money.** At ~$35k pilot scale, a BV's fixed costs (incorporation, annual accounts, Vpb filing, and especially the ~€58k gebruikelijk-loon obligation that an actively-run trading BV likely cannot escape) are disproportionate, and every pooled-fund wrapper (FGR/VBI/FBI/RAIF/QIAIF) is either closed off post-2025 reform or needs tens of millions in AUM to be economical. The one serious caveat: your setup — a continuously-running, code-driven, systematic strategy — sits squarely on the fact pattern (ECLI:NL:GHARL:2024:4613, the self-built crypto bot with a stable 75–80% win rate) that Dutch courts pushed into **Box 1** (progressive rates to ~49.5% on actual profit) rather than Box 3, so "Box 3 is cheaper" is only true if Box 3 actually holds. **Move to a dedicated (ring-fenced) BV when either (a) your advisor judges Box 1 reclassification risk high enough that corporate-rate certainty is worth the overhead, or (b) profits are large and consistently retained/reinvested so Vpb-only compounding (19%) beats paying Box 3 tax on capital every year.** Move to a fund structure (de-minimis AIFM + FGR) only when external capital is a concrete, near-term plan — not before.
|
||||
|
||||
## 2. The accounting system (the part you can build)
|
||||
|
||||
**Recommendation: build the tax ledger inside your existing Postgres/Timescale warehouse, not in a plaintext tool (beancount/hledger).** You already run Postgres + Dagster with per-strategy tagging; a second, disconnected file-based ledger just creates a reconciliation surface between the warehouse and the books. beancount's one real advantage — a native FIFO/AVERAGE cost-basis engine — is a bounded amount of SQL to replicate, and neither tool has a Bybit importer (you build that either way). Keep beancount only as a *possible export target* if your accountant wants to inspect plain text.
|
||||
|
||||
### Schema (append-only double-entry)
|
||||
|
||||
- **`journal_entries`** (header): `entry_id, entry_date, description, source_venue, strategy_tag, source_ref` — immutable, corrections via reversing entries only (never UPDATE/DELETE).
|
||||
- **`journal_postings`** (lines): `entry_id, account_id, side ('debit'|'credit'), amount, currency, fx_rate_to_eur, eur_amount` — with a DB-level CHECK/trigger enforcing that each entry's postings sum to zero **in EUR-converted terms**.
|
||||
- **`accounts`**: per-venue sub-ledgers that each balance standalone — `IBKR-EUR-cash`, `IBKR-securities`, `Bybit-USD-cash`, `Bybit-crypto`, `Bybit-perp-margin` — plus **two distinct FX accounts per foreign balance**: `unrealized-FX-PnL` (hit at each period-end retranslation) and `realized-FX-PnL` (hit only on actual conversion/withdrawal). This split is what lets you resolve the accounting-vs-tax realization question later without re-deriving history.
|
||||
- **`cost_basis_lots`** (per venue/asset): matched lots so realized gains come from actual lot-matching (FIFO — the NL de-facto default — or whatever your advisor confirms, applied *consistently* across both venues), not cash-flow deltas.
|
||||
- **`fx_rates`**: daily ECB reference rates, keyed by value_date.
|
||||
- **`account_balances`**: a materialized view for fast reads.
|
||||
|
||||
The **strategy_tag** on every posting makes P&L-by-strategy a `GROUP BY`, reusing your existing Dagster attribution — a filter, not a rebuild.
|
||||
|
||||
### Ingestion pipeline (Dagster assets, mirroring your nightly ETL pattern)
|
||||
|
||||
1. **IBKR Flex Web Service API** (not PDF scraping): a per-query token + query ID, 2-step SendRequest→GetStatement poll, ~1 req/sec limit, 2–3 pulls/day is safe. Pull trades, cash transactions, dividends, corporate actions, positions, NAV. Archive the annual Activity Statement as an audit doc even though you compute P&L independently. Note: as a W-8BEN non-US person you get a **1042-S**, never a Consolidated 1099, and there is no IBKR "NL tax statement."
|
||||
2. **Bybit** transaction-history / Tax-API REST endpoints for fills, **funding fees**, and deposits/withdrawals. **Explicitly verify the Unified Trading Account API surfaces perp funding and mark-to-market events** — they are easy to miss and material to P&L.
|
||||
3. Land raw payloads in an append-only `raw` schema (mirrors your DB-anchor-SSOT pattern), then a transform layer normalizes both venues into the canonical `journal_postings`.
|
||||
|
||||
### FX conversion & reconciliation
|
||||
|
||||
- **Standard rate: ECB daily reference rate** (`eurofxref-daily.xml`, ~16:00 CET; `eurofxref-hist.xml` for one-time backfill), applied at each transaction's value_date (IAS 21 transaction-date convention).
|
||||
- **The ECB publishes no crypto crosses.** Two-step for Bybit: (1) treat USDT as a USD-proxy ~1:1 **and document that assumption explicitly** (or use Bybit's own USD-index for USDT legs), then (2) USD→EUR at the ECB rate for that value_date.
|
||||
- **Box 3 peildatum (1 January)** crypto valuation: use the **exchange's own displayed value at 00:00 on 1 Jan** (crypto is venue-priced), and snapshot it via API.
|
||||
- **Reconciliation target: DAC8/CARF.** From FY2026, in-scope CASPs report your crypto data straight to the Belastingdienst (first reporting Jan 2027). Your books must reconcile to what the venue reports. Given Bybit's current global entity is non-EU pre-migration, **assume no third-party feed and keep fully independent records** — this raises, not lowers, your self-reporting burden. (See §5 on the Bybit EU migration deadline.)
|
||||
|
||||
## 3. Tax-efficient profit extraction (ranked)
|
||||
|
||||
| Rank | Route | Box / rate implication | When it wins |
|
||||
|---|---|---|---|
|
||||
| **1** | **Stay personal, Box 3** | 36% on a *deemed* return (~6% "overige bezittingen" forfait for 2026 → effective ~2.2% of asset value/yr); tegenbewijsregeling lets you report actual return if lower | Pilot / own-capital, **if Box 3 classification holds** and actual returns exceed the forfait. Dramatically cheaper than any actual-profit regime in a strong year. |
|
||||
| **2** | **BV, retain earnings** | Vpb only: 19% ≤€200k, 25.8% above — no Box 2 layer until you distribute | Once profits are large and **reinvested/compounded** inside the BV; deferral of the Box 2 layer is the compounding advantage. Also removes the Box 1/3 question entirely. |
|
||||
| **3** | **BV, distribute as dividend** | Vpb (19/25.8%) **then** Box 2 (24.5% ≤~€68.8k / 31% above); combined effective ~38–44% on distributed profit | Only when you actually need the cash personally *and* Box 1 reclassification would otherwise have applied (~49.5%) — you pay for certainty. |
|
||||
| **4** | **DGA salary from a BV** | gebruikelijk loon ~€58k (2026), taxed as Box 1 wage income, progressive to ~49.5% | Not a "choice" — it's a *mandatory* cost of the BV route (an active trading BV is unlikely to qualify for the ~€5k limited-activity carve-out, and from 2026 needs an advisor-cosigned salary justification). Factor it into the BV break-even, don't treat it as extraction. |
|
||||
|
||||
**Key numbers to hold in mind:** Box 3 taxes *deemed* return on *capital*; the BV route taxes *actual* profit. So Box 3 is cheaper the more your real return exceeds ~6%, and the BV is cheaper mainly when Box 1 (~49.5% on actual) is the realistic alternative or when earnings are retained. Under the Box 3 **tegenbewijsregeling**, "actual return" includes **unrealized** mark-to-market gains and does **not** allow deduction of trading fees/funding (only actual Box-3 debt interest) — this is *not* how a trading P&L system naturally computes returns, so your ledger needs a parallel tax-basis calculation.
|
||||
|
||||
## 4. Fund-structure decision map
|
||||
|
||||
```
|
||||
STAGE 0 — Personal, Box 3 own capital only, any size
|
||||
trigger to advance ↓ external capital contemplated, OR Box 1 risk judged high, OR large retained profits
|
||||
STAGE 1 — Dedicated BV (ring-fenced) own capital; new sister/subsidiary BV, NOT commingled with bizworx
|
||||
trigger to advance ↓ first external euro is concrete & near-term
|
||||
STAGE 2 — BV(manager) + FGR + AIFMD "light" registration
|
||||
trigger to advance ↓ AUM approaches ~€20–30m (dedicated-fund fixed costs become proportionate)
|
||||
STAGE 3 — Licensed fund / RAIF (LU) or QIAIF (IE) institutional external capital, tens of €m+
|
||||
```
|
||||
|
||||
**Regulatory gates:**
|
||||
|
||||
- **"Own money only" is a genuine AIFMD safe harbor** (Art. 4(1)(a); Recital 7 family-office carve-out). No license, no registration while you trade solely your own capital — *this covers a solo BV too*. Also outside MiFID II via the Art. 2(1)(d) own-account exemption (subject to a technical check that your order/message rate doesn't hit the "high-frequency algorithmic trading" threshold), and outside MiCA CASP authorization for own-account crypto.
|
||||
- **The moment you accept one external euro**, AIF/AIFMD scope begins. Below **€100m AUM** (leveraged / redemption within 5yr) or **€500m** (unleveraged closed-end), you need only **AFM registration** (the "light"/de-minimis regime), *not* a full AIFM license — but that registration must be done **before** the first external euro, plus Wwft (AML) and Sanctiewet compliance and AFM/DNB reporting. Marketing to retail adds top-up rules (Wft 4:37p) and a <150-investor / €100k-minimum-ticket cap.
|
||||
- **Ring-fence, don't commingle.** If/when you do incorporate, put trading in a **new dedicated BV**, not inside bizworx.nl — standard NL practice isolates a volatile, potentially loss-making book from an operating business's balance sheet, credit, and any regulatory status.
|
||||
- **FGR is in flux and not fit-for-purpose at pilot scale.** The Jan-2025 reform abolished the open/closed test; a new fund must qualify directly under Wft-anchored rules and a further consultation (opened Dec 2025) is open. VBI is effectively closed to solo operators post-reform; FBI structurally disfavors active high-turnover trading; RAIF/QIAIF both *require* an external authorized AIFM and are uneconomical below ~€20–30m. So Stage 2 is a *later* decision, timed to land after the FGR rules settle.
|
||||
|
||||
## 5. What to take to a Dutch tax advisor / notaris
|
||||
|
||||
**Classification (highest stakes):**
|
||||
1. Given a fully agentic, continuously-running systematic strategy across IBKR UCITS ETFs + Bybit perps — is this **Box 3 or Box 1** (resultaat uit overige werkzaamheden / winst uit onderneming)? Reconcile the taxpayer-favorable *Gelderland/Noord-Holland* daytrading line against the *GHARL:2024:4613* bot line that landed in Box 1. **Is a vooroverleg / advance ruling advisable before scaling capital**, given the retroactive-assessment downside if reclassified after the fact?
|
||||
|
||||
**Entity & existing structure:**
|
||||
2. Is **bizworx.nl a BV or an eenmanszaak?** This determines whether housing trading is a fresh incorporation or a restructuring (eenmanszaak→BV has geruisloze inbreng considerations; a fiscale eenheid question if a second BV is created).
|
||||
3. If moving personally-held IBKR/Bybit positions into a BV — does that trigger a **deemed-disposal (afrekening)** for Box 3, and is any rollover/deferral available for a trading portfolio?
|
||||
4. **gebruikelijk loon**: does running an autonomous/agentic system (software does most of the "work") qualify for the ~€5k limited-activity carve-out, or does the algorithm-design labor count against it?
|
||||
|
||||
**Accounting method & FX:**
|
||||
5. **Cost-basis method** — FIFO or weighted-average, and must it be identical across IBKR ETF lots and Bybit crypto lots (goedkoopmansgebruik consistency)?
|
||||
6. **Unrealized FX/crypto revaluation** — book to P&L each period (RJ122/IAS21) for internal reporting while the tax return recognizes only realized (goedkoopmansgebruik; the NLFiscaal unrealized-FX-loss-denial precedent)? Do I need two parallel FX-recognition tracks (book vs tax)?
|
||||
7. **Functional-currency election** (EUR vs USD) — worth the 10-year lock-in, or is "EUR presentation + translate every Bybit leg at transaction-date ECB rate" sufficient without electing?
|
||||
8. **USDT** — treated as USD-equivalent, or its own crypto-asset (own EUR cross) for both ongoing bookkeeping and the Box 3 peildatum?
|
||||
|
||||
**Instrument-specific (a genuine gap in the research):**
|
||||
9. How are **perpetual-futures funding payments** characterized for Box 3 werkelijk-rendement — as income each period, or folded into position P&L? No authoritative Belastingdienst guidance was found.
|
||||
10. Under werkelijk rendement, are Bybit **trading fees / funding paid / margin costs non-deductible** (as the tegenbewijsregeling mechanics suggest, unlike normal trading P&L)?
|
||||
|
||||
**Reform & reporting:**
|
||||
11. Confirm the **finalized 2026 figures** at filing time (heffingsvrij vermogen — sources conflicted between ~€51,396 and ~€59,357 — forfait %, Box 2 thresholds).
|
||||
12. Does the fund trigger a formal **boekhoudplicht** (BW 2:10 / 3:15i — 7-year retention, audit-trail format), and does the Postgres pipeline satisfy it?
|
||||
13. **BTW/VAT** — confirm own-account securities/crypto trading is exempt, and re-check if a management fee is later introduced.
|
||||
|
||||
**Regulatory (Wft lawyer, not just belastingadviseur):**
|
||||
14. Does the IBKR order/message rate meet MiFID II's **HFT** technical threshold (removes the own-account exemption)? Needs a real order-flow review.
|
||||
15. **Bybit EU migration — operationally urgent:** the global platform closes EEA/NL access ~1 July 2026; you migrate to the Austria-MiCA "Bybit EU" entity (fresh KYC, Travel Rule proof-of-wallet for transfers >€1,000). **Bybit EU reportedly cannot offer USDT** (Tether has no MiCA EMT authorization), so USDT balances must convert to EUR or a MiCA stablecoin — **is that forced conversion a taxable disposal, and under which box?** Verify directly on Bybit's compliance pages before the deadline.
|
||||
|
||||
## 6. Immediate next build step
|
||||
|
||||
**Build the append-only double-entry ledger core + the IBKR Flex ingestion asset first — one Dagster asset landing IBKR Flex trades/cash/NAV into a `raw` schema, normalizing into `journal_entries`/`journal_postings` with the balanced-entry DB trigger, and joining an ECB `fx_rates` table by value_date to produce EUR-normalized postings.**
|
||||
|
||||
Why this one component first, over everything else:
|
||||
- IBKR is your EUR-native, better-documented venue with a stable Flex Web Service API and an existing importer pattern to model on — lowest integration risk, so you validate the schema and the balanced-entry invariant against real data before adding Bybit's messier surface (funding fees, USDT proxy, migration churn).
|
||||
- It exercises **every** load-bearing design decision at once: append-only immutability, the EUR-sum-to-zero constraint, the strategy_tag dimension, and the ECB-rate join — the FX-revaluation split and cost-basis lot tables extend cleanly on top once this spine is proven.
|
||||
- It's a natural Dagster asset alongside your existing nightly ETL, reusing your monitoring/DB-anchor-SSOT patterns — no new infrastructure, and it turns tax season into a query rather than an annual export scramble regardless of which box you end up in.
|
||||
|
||||
Defer the Bybit importer, the cost-basis lot engine, and any beancount export until this spine is landing correct, reconciled EUR postings from IBKR.
|
||||
29
docs/superpowers/VRP-REMOVAL-TODO.md
Normal file
29
docs/superpowers/VRP-REMOVAL-TODO.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# VRP full removal — DONE (2026-07-20/21)
|
||||
|
||||
**Status: COMPLETE.** VRP code fully removed on branch `chore/remove-vrp-code` (9 commits over base a263b67),
|
||||
full suite green (2018 passed, 2 skipped), whole-branch review clean (no Critical/Important). OPRA data KEPT.
|
||||
|
||||
## What was removed
|
||||
- 7 VRP application modules (`vrp_book/eval/defined_risk_eval/exec_record/marking/pricing/research`) + `black_scholes.py`
|
||||
- The DVOL pipeline (`deribit_dvol.py`, `dvol_ingest.py`, the `VolIndexClient` port) — DVOL was the IV leg of VRP
|
||||
- The `vrp_nav` Dagster asset + its ~14 OPRA-freeze/backfill helper chain
|
||||
- The `XspOptionBarsRepo` **class** + the `_build_vrp` migration builder + the `backfill-xsp-opra`/`ingest-dvol`/
|
||||
`execute-vrp`/`vrp-eval`/`vrp-defined-risk-eval` CLI commands + the `fxhnt-opra-backfill` CronJob manifest
|
||||
- The `vrp`/`vrp_exec` registry entries + `_REF_ALIAS`/`_IBKR_ACCOUNT_SIDS`/`SIM_BOOKS` literals
|
||||
- 13 VRP test files; comment/docstring scrubs across 16 files; test updates (assert vrp-ABSENT / synthetic entries)
|
||||
|
||||
## What was KEPT (Constraint 1 — verified intact)
|
||||
- `xsp_option_bars` **table** + `XspOptionBarRow` model (`cockpit_models.py:436`) + the 13-year OPRA `.dbn` data
|
||||
- `ForwardNavRepo.xsp_freeze_max_date()` (`forward_nav.py:145`) — still reads the kept table
|
||||
- The generic freeze-staleness health axis (`_FREEZE_SOURCES = {"opra-pit"}`) stays
|
||||
- `deribit_funding` (carry edge) untouched
|
||||
|
||||
## CronJob cluster-resource — VERIFIED not present (no action needed)
|
||||
Checked 2026-07-21 on `bizworx-prod-kapsule`: `kubectl get cronjob fxhnt-opra-backfill -n foxhunt` → NotFound,
|
||||
0 hits cluster-wide. The manifest was never applied live (or already gone), so removing it from git was
|
||||
sufficient. The 5 surviving foxhunt CronJobs (ucits/bybit/ibkr rebalancers, ucits-volume, factory) are all
|
||||
non-VRP.
|
||||
|
||||
## Merged + pushed
|
||||
Branch `chore/remove-vrp-code` merged to master (`2f957bb`) and pushed to origin. Live on prod after the next
|
||||
normal git-sync + rollout-restart.
|
||||
@@ -0,0 +1,599 @@
|
||||
# Positioning Per-Coin Gross Cap Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add an ex-ante per-coin gross-weight cap (0.07) to the `positioning` sleeve so no single coin dominates the book, then re-inception + backfill the forward anchor so the live gate keeps running against the capped construction.
|
||||
|
||||
**Architecture:** The cap is applied to the FORMING weights in `positioning_weights` (before returns), so it is strictly causal (no lookahead). It threads through the existing seams (`_book_breakdown` → `positioning_returns_from_store` → `_positioning_series` → `sleeve_returns_from_store`) as an optional `coin_gross_cap` param that defaults to `None` (OFF → byte-identical to today). It is switched ON at 0.07 only in `bybit_book_persist.py`. The registry `positioning` + `bybit_4edge` definition versions bump to re-inception; `inception_t0` pins the existing OOS start (2026-07-06) so the forward track backfills the window with the capped construction instead of resetting the gate to 0.
|
||||
|
||||
**Tech Stack:** Python 3.12, numpy-free pure math, pytest (`-n auto`), TimescaleFeatureStore (Postgres, table `bybit_features`).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- The cap is applied to WEIGHTS (in `positioning_weights`), NOT to returns/contributions — it must be ex-ante/causal (the cap decision cannot see the coin's realized return).
|
||||
- `coin_gross_cap` default is `None` (OFF) at EVERY seam. With it OFF, every existing test and the whole backtest must be byte-identical to today. The OFF path must not add per-term division drift.
|
||||
- After clipping `|w| ≤ coin_gross_cap`, RENORMALIZE the day's weights to unit gross (`Σ|w| = 1`), matching `positioning_weights`' existing unit-gross contract.
|
||||
- The validated cap value is **0.07** (full-history sweep 1295d: Sharpe 2.60→2.69, maxDD −32.3%→−29.9%, per-year Sharpe non-decreasing every year). Do NOT use 0.04 (falsified: Sharpe drops to 2.43).
|
||||
- Re-inception uses the EXISTING pattern: bump `version` in `definition.params.version`, and set `inception_t0: {"date": "2026-07-06", "version": <new>}` (outside `params`, NOT hashed) so the anchor backdates and the forward window is recomputed, not reset. See `forward_definition.inception_override`.
|
||||
- `positioning` is a CONSTITUENT of `bybit_4edge`; both registry entries' definition versions must bump together (the book's sleeve construction changed).
|
||||
- Never touch real money / execution. This is a construction change to the modeled+forward track only.
|
||||
- Run tests with `uv run --extra dev python -m pytest ...`. The positioning eval integration tests need no extra services (they use a fake store).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add the ex-ante `coin_gross_cap` to `positioning_weights`
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fxhnt/application/bybit_positioning_eval.py:78-104` (the `positioning_weights` function)
|
||||
- Test: `tests/integration/test_bybit_positioning_eval.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: nothing new.
|
||||
- Produces: `positioning_weights(ratio_by_day, *, direction="contrarian", coin_gross_cap: float | None = None) -> dict[int, dict[str, float]]`. When `coin_gross_cap` is a float, each day's weights are clipped to `|w| ≤ coin_gross_cap` then renormalized to unit gross. When `None`, output is byte-identical to today.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `tests/integration/test_bybit_positioning_eval.py`:
|
||||
|
||||
```python
|
||||
def test_coin_gross_cap_clips_and_renormalizes() -> None:
|
||||
# One coin dominates (retail extreme) -> uncapped weight far exceeds 0.07; cap must clip + renormalize.
|
||||
ratio = {0: {"DOMUSDT": 0.95, "AUSDT": 0.50, "BUSDT": 0.50, "CUSDT": 0.05}}
|
||||
uncapped = positioning_weights(ratio)[0]
|
||||
assert max(abs(w) for w in uncapped.values()) > 0.07 # DOM dominates uncapped
|
||||
capped = positioning_weights(ratio, coin_gross_cap=0.07)[0]
|
||||
assert max(abs(w) for w in capped.values()) <= 0.07 + 1e-9 # no coin exceeds the cap
|
||||
assert abs(sum(abs(w) for w in capped.values()) - 1.0) < 1e-9 # still unit gross
|
||||
# sign structure preserved: DOM still short (over-long crowd), C still long (over-short)
|
||||
assert capped["DOMUSDT"] < 0.0 and capped["CUSDT"] > 0.0
|
||||
|
||||
|
||||
def test_coin_gross_cap_none_is_byte_identical() -> None:
|
||||
ratio = {0: {"AAAUSDT": 0.9, "BBBUSDT": 0.5, "CCCUSDT": 0.1}}
|
||||
assert positioning_weights(ratio, coin_gross_cap=None) == positioning_weights(ratio)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py::test_coin_gross_cap_clips_and_renormalizes -v`
|
||||
Expected: FAIL with `TypeError: positioning_weights() got an unexpected keyword argument 'coin_gross_cap'`
|
||||
|
||||
- [ ] **Step 3: Implement the cap in `positioning_weights`**
|
||||
|
||||
Replace the body of `positioning_weights` (lines 91-104) so the signature gains `coin_gross_cap` and, after the existing unit-gross normalization, applies the clip+renormalize when the cap is set:
|
||||
|
||||
```python
|
||||
def positioning_weights(ratio_by_day: dict[int, dict[str, float]], *,
|
||||
direction: str = "contrarian",
|
||||
coin_gross_cap: float | None = None) -> dict[int, dict[str, float]]:
|
||||
"""`{day: {coin: w}}` market-neutral cross-sectional weights from the long/short positioning signal.
|
||||
|
||||
`direction="contrarian"` (DEFAULT) SHORTs the high-buyRatio coins (retail over-long → fade) and LONGs the
|
||||
low-buyRatio coins (retail over-short). `direction="momentum"` is the EXACT SIGN-FLIP: LONG the over-long
|
||||
/ SHORT the over-short (informed-flow continuation). Construction per day:
|
||||
1. DEMEAN the per-day buyRatio across coins (so the book is dollar/market-neutral, Σw ≈ 0);
|
||||
2. apply the DIRECTION sign — contrarian = −demeaned (short the rich/over-long), momentum = +demeaned;
|
||||
3. normalise to UNIT GROSS (Σ|w| = 1);
|
||||
4. (OPTIONAL) if `coin_gross_cap` is set, CLIP each |w| to the cap and RENORMALISE to unit gross —
|
||||
an EX-ANTE per-coin concentration limit (applied to the FORMING weight, so it is strictly causal:
|
||||
it cannot see the coin's realized return). Validated at 0.07 (full-history sweep: Sharpe 2.60→2.69,
|
||||
maxDD −32.3%→−29.9%, per-year non-decreasing) — caps the LABUSDT-class implosions (a 10.3% long leg
|
||||
that did −59%/−79.5% on 2026-07-06/07) without over-throttling the edge (0.04 falsified: Sharpe→2.43).
|
||||
|
||||
Because the gross is identical for both directions, `momentum` weights are the exact per-coin negation of
|
||||
`contrarian`. A day with <2 coins, or zero gross after demeaning, yields no weights (skipped). Pure."""
|
||||
if direction not in ("contrarian", "momentum"):
|
||||
raise ValueError(f"direction must be 'contrarian' or 'momentum', got {direction!r}")
|
||||
sign = -1.0 if direction == "contrarian" else 1.0
|
||||
out: dict[int, dict[str, float]] = {}
|
||||
for day, row in ratio_by_day.items():
|
||||
if len(row) < 2:
|
||||
continue
|
||||
mean = st.mean(row.values())
|
||||
raw = {c: sign * (v - mean) for c, v in row.items()}
|
||||
gross = sum(abs(w) for w in raw.values())
|
||||
if gross <= 0.0:
|
||||
continue
|
||||
w = {c: x / gross for c, x in raw.items()}
|
||||
if coin_gross_cap is not None:
|
||||
w = {c: max(-coin_gross_cap, min(coin_gross_cap, x)) for c, x in w.items()}
|
||||
capped_gross = sum(abs(x) for x in w.values())
|
||||
if capped_gross <= 0.0:
|
||||
continue
|
||||
w = {c: x / capped_gross for c, x in w.items()}
|
||||
out[int(day)] = w
|
||||
return out
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py -v`
|
||||
Expected: PASS (the two new tests + all existing positioning_weights tests still green)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fxhnt/application/bybit_positioning_eval.py tests/integration/test_bybit_positioning_eval.py
|
||||
git commit -m "feat(positioning): ex-ante per-coin gross cap in positioning_weights (default off)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Thread `coin_gross_cap` through `_book_breakdown` and `positioning_returns_from_store`
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fxhnt/application/bybit_positioning_eval.py:140-166` (`_book_breakdown` signature + the `positioning_weights` call at line 177)
|
||||
- Modify: `src/fxhnt/application/bybit_positioning_eval.py:287-303` (`positioning_returns_from_store` signature + its `_book_breakdown` call)
|
||||
- Test: `tests/integration/test_bybit_positioning_eval.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `positioning_weights(..., coin_gross_cap=...)` from Task 1.
|
||||
- Produces:
|
||||
- `_book_breakdown(store, *, universe, cost_bps, direction, capacity_capital=None, participation=0.10, turnover_window=30, tail_cap_k=None, book_vol_window=60, coin_gross_cap=None) -> dict`
|
||||
- `positioning_returns_from_store(store, *, universe=None, cost_bps=5.5, direction="contrarian", capacity_capital=None, tail_cap_k=None, coin_gross_cap=None) -> dict[int, float]`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `tests/integration/test_bybit_positioning_eval.py` (reuse the module's existing fake store fixture — grep the file for the store used by `test_capacity_haircut_caps_gain_only_at_scale`, e.g. a `_FakeStore`/`_ReadOnlyStore` builder, and construct one where a single coin has an extreme long_ratio + a large adverse next-day return):
|
||||
|
||||
```python
|
||||
def test_coin_gross_cap_reduces_single_coin_loss_in_book(positioning_store_with_shock):
|
||||
# positioning_store_with_shock: one coin (retail over-short) craters next day; uncapped it dominates the
|
||||
# book loss, capped its weight (and thus its loss contribution) is bounded.
|
||||
store = positioning_store_with_shock
|
||||
uncapped = positioning_returns_from_store(store, cost_bps=0.0)
|
||||
capped = positioning_returns_from_store(store, cost_bps=0.0, coin_gross_cap=0.07)
|
||||
shock_day = min(uncapped, key=lambda d: uncapped[d]) # the worst day uncapped
|
||||
assert capped[shock_day] > uncapped[shock_day] # the cap made the worst day less bad
|
||||
```
|
||||
|
||||
If the module has no reusable multi-coin shock fixture, build the store inline in the test from a `long_ratio` panel (one coin at ratio 0.95, three near 0.5) + a `close` panel where the over-short coin drops ~50% the next day, mirroring the `test_capacity_haircut_*` store construction in the same file.
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py::test_coin_gross_cap_reduces_single_coin_loss_in_book -v`
|
||||
Expected: FAIL with `TypeError: positioning_returns_from_store() got an unexpected keyword argument 'coin_gross_cap'`
|
||||
|
||||
- [ ] **Step 3: Thread the param through both functions**
|
||||
|
||||
In `_book_breakdown`, add `coin_gross_cap: float | None = None` to the signature (after `book_vol_window: int = 60`), and change the weight-forming call (currently line 177):
|
||||
|
||||
```python
|
||||
weights_by_day = positioning_weights(_ratio_by_day(long_ratio), direction=direction,
|
||||
coin_gross_cap=coin_gross_cap)
|
||||
```
|
||||
|
||||
In `positioning_returns_from_store`, add `coin_gross_cap: float | None = None` to the signature (after `tail_cap_k`) and pass it through:
|
||||
|
||||
```python
|
||||
return _book_breakdown(store, universe=universe, cost_bps=cost_bps, direction=direction,
|
||||
capacity_capital=capacity_capital, tail_cap_k=tail_cap_k,
|
||||
coin_gross_cap=coin_gross_cap)["net"]
|
||||
```
|
||||
|
||||
Update the `positioning_returns_from_store` docstring's param note to add: `coin_gross_cap` (None default) applies the ex-ante per-coin concentration cap (see `positioning_weights`).
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fxhnt/application/bybit_positioning_eval.py tests/integration/test_bybit_positioning_eval.py
|
||||
git commit -m "feat(positioning): thread coin_gross_cap through book_breakdown + returns_from_store"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Thread `coin_gross_cap` through the sleeve seam (`_positioning_series` + `sleeve_returns_from_store`)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fxhnt/application/bybit_book_eval.py:226-240` (`_positioning_series`)
|
||||
- Modify: `src/fxhnt/application/bybit_book_eval.py:243-278` (`sleeve_returns_from_store` signature + the positioning branch)
|
||||
- Test: `tests/integration/test_bybit_positioning_eval.py` (or `tests/integration/test_web_new_edges.py` if that is where sleeve_returns_from_store is exercised — grep first)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `positioning_returns_from_store(..., coin_gross_cap=...)` from Task 2.
|
||||
- Produces:
|
||||
- `_positioning_series(store, *, universe, cost_bps, direction="contrarian", capacity_capital=None, tail_cap_k=None, coin_gross_cap=None) -> dict[int, float]`
|
||||
- `sleeve_returns_from_store(store, edge, *, universe=None, cost_bps=0.0, unlock_events=None, stablecoin_spot_panel=None, capacity_capital=None, tail_cap_k=None, coin_gross_cap=None) -> dict[int, float]`. The `coin_gross_cap` is forwarded ONLY to the `positioning` branch (other edges ignore it).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add (same test file, reuse the store from Task 2's fixture/inline construction):
|
||||
|
||||
```python
|
||||
def test_sleeve_returns_positioning_forwards_coin_gross_cap(positioning_store_with_shock):
|
||||
from fxhnt.application.bybit_book_eval import sleeve_returns_from_store
|
||||
store = positioning_store_with_shock
|
||||
uncapped = sleeve_returns_from_store(store, "positioning", cost_bps=0.0)
|
||||
capped = sleeve_returns_from_store(store, "positioning", cost_bps=0.0, coin_gross_cap=0.07)
|
||||
shock_day = min(uncapped, key=lambda d: uncapped[d])
|
||||
assert capped[shock_day] > uncapped[shock_day] # cap forwarded -> worst day improved
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py::test_sleeve_returns_positioning_forwards_coin_gross_cap -v`
|
||||
Expected: FAIL with `TypeError: sleeve_returns_from_store() got an unexpected keyword argument 'coin_gross_cap'`
|
||||
|
||||
- [ ] **Step 3: Thread through both functions**
|
||||
|
||||
In `_positioning_series` (line 226), add `coin_gross_cap: float | None = None` to the signature (after `tail_cap_k`) and forward it:
|
||||
|
||||
```python
|
||||
return positioning_returns_from_store(store, universe=universe, cost_bps=cost_bps, direction=direction,
|
||||
capacity_capital=capacity_capital, tail_cap_k=tail_cap_k,
|
||||
coin_gross_cap=coin_gross_cap)
|
||||
```
|
||||
|
||||
In `sleeve_returns_from_store` (line 243), add `coin_gross_cap: float | None = None` to the signature (after `tail_cap_k`), and change ONLY the positioning branch (line 276) to forward it:
|
||||
|
||||
```python
|
||||
if edge == "positioning":
|
||||
# ... existing comment about raw store + universe unchanged ...
|
||||
return _positioning_series(store, universe=universe, cost_bps=cost_bps,
|
||||
capacity_capital=capacity_capital, tail_cap_k=tail_cap_k,
|
||||
coin_gross_cap=coin_gross_cap)
|
||||
```
|
||||
|
||||
Leave the tstrend / unlock / stablecoin / xsfunding branches unchanged (they never receive `coin_gross_cap`).
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fxhnt/application/bybit_book_eval.py tests/integration/test_bybit_positioning_eval.py
|
||||
git commit -m "feat(positioning): forward coin_gross_cap through sleeve_returns_from_store"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Turn the cap ON at 0.07 in the live precompute + add the constant
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fxhnt/application/bybit_forward_track.py:47` (add `POSITIONING_COIN_GROSS_CAP` next to `POSITIONING_TAIL_CAP_K`)
|
||||
- Modify: `src/fxhnt/application/bybit_book_persist.py:77-84` (pass the cap into `sleeve_returns_from_store`)
|
||||
- Test: `tests/integration/test_bybit_paper_reconciliation.py` (or the persist test — grep for a test that calls `_persist_bybit_book` / `persist_bybit_sleeve_returns`; if none exists at unit level, assert the constant is wired via a focused test in `test_bybit_positioning_eval.py`)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `sleeve_returns_from_store(..., coin_gross_cap=...)` from Task 3.
|
||||
- Produces: `POSITIONING_COIN_GROSS_CAP = 0.07` in `bybit_forward_track.py`; the live `_persist_bybit_book` sleeve computation applies it to the positioning sleeve.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `tests/integration/test_bybit_positioning_eval.py`:
|
||||
|
||||
```python
|
||||
def test_positioning_coin_gross_cap_constant_is_007() -> None:
|
||||
from fxhnt.application.bybit_forward_track import POSITIONING_COIN_GROSS_CAP
|
||||
assert POSITIONING_COIN_GROSS_CAP == 0.07
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py::test_positioning_coin_gross_cap_constant_is_007 -v`
|
||||
Expected: FAIL with `ImportError: cannot import name 'POSITIONING_COIN_GROSS_CAP'`
|
||||
|
||||
- [ ] **Step 3: Add the constant + wire it into the precompute**
|
||||
|
||||
In `src/fxhnt/application/bybit_forward_track.py`, directly after line 47 (`POSITIONING_TAIL_CAP_K = 4.0`):
|
||||
|
||||
```python
|
||||
# Ex-ante per-coin gross-weight cap for the positioning sleeve — no single coin's |weight| may exceed this
|
||||
# fraction of the book's unit gross. Validated 2026-07-20 over the full 1295-day history (Sharpe 2.60→2.69,
|
||||
# maxDD −32.3%→−29.9%, per-year Sharpe non-decreasing every year); caps LABUSDT-class single-coin implosions
|
||||
# (a 10.3% long leg that did −59%/−79.5% on 2026-07-06/07) without over-throttling (0.04 falsified → 2.43).
|
||||
POSITIONING_COIN_GROSS_CAP = 0.07
|
||||
```
|
||||
|
||||
In `src/fxhnt/application/bybit_book_persist.py`, update the import (line 78) and the sleeve computation (lines 80-83):
|
||||
|
||||
```python
|
||||
from fxhnt.application.bybit_forward_track import POSITIONING_COIN_GROSS_CAP, POSITIONING_TAIL_CAP_K
|
||||
sleeve_rets = {
|
||||
s: sleeve_returns_from_store(store, s, universe=universe, cost_bps=cost_bps,
|
||||
unlock_events=unlock_events,
|
||||
capacity_capital=settings.paper_capital,
|
||||
tail_cap_k=POSITIONING_TAIL_CAP_K,
|
||||
coin_gross_cap=POSITIONING_COIN_GROSS_CAP)
|
||||
for s in _DEFAULT_BYBIT_SLEEVES}
|
||||
```
|
||||
|
||||
(`coin_gross_cap` is ignored by every sleeve except positioning — Task 3 only forwards it in the positioning branch — so passing it for all sleeves in the dict comprehension is correct and harmless.)
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_positioning_eval.py -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fxhnt/application/bybit_forward_track.py src/fxhnt/application/bybit_book_persist.py tests/integration/test_bybit_positioning_eval.py
|
||||
git commit -m "feat(positioning): enable coin_gross_cap=0.07 in the live bybit book precompute"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Bump definition versions to re-inception positioning + bybit_4edge (backdated anchor, no gate reset)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fxhnt/registry.py` (the `positioning` entry ~line 249-259 and the `bybit_4edge` entry ~line 198-211; also `bybit_4edge_levered` ~line 229-241 which shares the sleeve construction)
|
||||
- Test: `tests/` — grep for `test_forward_definition` / a registry-consistency test; add an assertion that the bumped versions + inception_t0 are consistent
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the capped construction from Task 4 (the version bump signals the construction changed).
|
||||
- Produces: `positioning`, `bybit_4edge`, `bybit_4edge_levered` definitions at `version: 3` with `inception_t0: {"date": "2026-07-06", "version": 3}`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add `tests/unit/test_positioning_cap_reinception.py`:
|
||||
|
||||
```python
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
|
||||
|
||||
def test_positioning_and_book_bumped_to_v3_backdated():
|
||||
for sid in ("positioning", "bybit_4edge", "bybit_4edge_levered"):
|
||||
d = STRATEGY_REGISTRY[sid]["definition"]
|
||||
assert d["params"]["version"] == 3, f"{sid} version not bumped to 3"
|
||||
ov = d.get("inception_t0")
|
||||
assert ov and ov["version"] == 3, f"{sid} inception_t0 not re-pinned to v3"
|
||||
# backdated to the existing OOS start, NOT reset to today -> the forward window is recomputed, not lost
|
||||
assert ov["date"] == "2026-07-06"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/unit/test_positioning_cap_reinception.py -v`
|
||||
Expected: FAIL (versions are currently 2)
|
||||
|
||||
- [ ] **Step 3: Bump the three registry entries**
|
||||
|
||||
In each of the `positioning`, `bybit_4edge`, and `bybit_4edge_levered` entries in `src/fxhnt/registry.py`, change `"version": 2` to `"version": 3` inside `definition.params`, and change the `inception_t0` to `{"date": "2026-07-06", "version": 3}`. Note: `positioning`'s existing `inception_t0` date is `2026-07-06`; `bybit_4edge` / `bybit_4edge_levered` currently use `2026-07-07` — for this backfill set ALL three to `2026-07-06` (the positioning shock's first day) so the whole book re-warms over the same window. Add a comment on each:
|
||||
|
||||
```python
|
||||
"inception_t0": {"date": "2026-07-06", "version": 3}, # v3: positioning coin_gross_cap=0.07 (2026-07-20 backfill)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/unit/test_positioning_cap_reinception.py tests/ -k "forward_definition or registry" -v`
|
||||
Expected: PASS (the new test + any existing registry/definition consistency tests)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/fxhnt/registry.py tests/unit/test_positioning_cap_reinception.py
|
||||
git commit -m "feat(positioning): re-inception positioning+bybit_4edge v3 (coin_gross_cap, backdated 2026-07-06)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Full-suite verification + lint/type gate
|
||||
|
||||
**Files:** none (verification task)
|
||||
|
||||
**Interfaces:** consumes all prior tasks.
|
||||
|
||||
- [ ] **Step 1: Run the full positioning + book + forward suite**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/ -k "positioning or book or forward or reconcil or bybit" -q`
|
||||
Expected: all pass (no regression; the OFF-path byte-identity tests from Tasks 1-3 guarantee unchanged behavior everywhere the cap is not enabled).
|
||||
|
||||
- [ ] **Step 2: Lint + type-check the changed files**
|
||||
|
||||
Run: `uv run --extra dev ruff check src/fxhnt/application/bybit_positioning_eval.py src/fxhnt/application/bybit_book_eval.py src/fxhnt/application/bybit_book_persist.py src/fxhnt/application/bybit_forward_track.py src/fxhnt/registry.py`
|
||||
Then: `uv run --extra dev --extra orchestration mypy src/fxhnt/application/bybit_positioning_eval.py src/fxhnt/application/bybit_book_eval.py`
|
||||
Expected: no NEW errors on changed lines (pre-existing whole-file findings that are not in the diff are acceptable — confirm via `git diff --unified=0 ... | grep '^+' | awk 'length>120'` returning nothing).
|
||||
|
||||
- [ ] **Step 3: Commit any lint fixes** (only if Step 2 surfaced issues on changed lines)
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "chore(positioning): lint/type fixes for coin_gross_cap"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Post-merge deploy + backfill (OPERATOR steps — outside the coding plan, run after merge)
|
||||
|
||||
These are runtime/prod steps, NOT code, listed so the operator has the full picture. Do them after the branch is merged to master and pushed:
|
||||
|
||||
1. **Deploy the code:** `git push origin master` (the Dagster K8sRunLauncher git-syncs master HEAD; a code-only change needs no image rebuild). Rollout-restart the dagster daemon if it caches code: `kubectl --context=$BIZ -n foxhunt rollout restart deploy/dagster-daemon`.
|
||||
2. **Backfill the anchor:** the re-inception is AUTOMATIC on the next `combined_book_daily` run (23:30 UTC) — the bumped version + backdated `inception_t0` cause `forward_engine` to reset the anchor to `t0=2026-07-06` and recompute the forward window with the capped construction. Do NOT run `migrate-forward-anchors` (re-inception is automatic — see `reference_fxhnt_prod_deploy_mechanics`).
|
||||
3. **Refresh the backtest ref:** the `bybit_book_precompute` asset re-persists the `positioning` / `bybit_4edge` / `bybit_4edge_levered` `backtest_summary` from the capped `_persist_bybit_book` in the same nightly run, so the gate reconciles the forward track against the capped (capturable) reference.
|
||||
4. **Verify next morning:** query the prod DB (port-forward svc/postgres) — `positioning` forward should re-show ~13 days (recomputed, capped), the 2026-07-06/07 daily returns should be materially less negative than −7.32%/−6.89%, and the gate should read WAIT "building" (not reset to 0) or its real verdict.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**1. Spec coverage:** cap in weights (T1) → threaded through book (T2) → sleeve seam (T3) → enabled live (T4) → re-inception+backfill (T5) → verified (T6). The "backfill not reset" constraint is implemented via `inception_t0` backdating in T5 + operator step 2. Covered.
|
||||
|
||||
**2. Placeholder scan:** every code step shows the exact code; the only deferred item is the shock-store fixture in T2/T3 which is explicitly instructed to reuse the existing `test_capacity_haircut_*` store construction in the same file (a real, named pattern, not a placeholder). Acceptable — the implementer has a concrete reference.
|
||||
|
||||
**3. Type consistency:** `coin_gross_cap: float | None = None` is identical across `positioning_weights`, `_book_breakdown`, `positioning_returns_from_store`, `_positioning_series`, `sleeve_returns_from_store`. Constant `POSITIONING_COIN_GROSS_CAP = 0.07` in `bybit_forward_track.py`, imported in `bybit_book_persist.py`. Versions all → 3, `inception_t0.date` all → "2026-07-06". Consistent.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Thread `coin_gross_cap` into the LIVE forward-nav path (Critical fix from final review)
|
||||
|
||||
**Why:** The final whole-branch review found the cap threads to the reconciliation REF (via `_persist_bybit_book`) but NOT to the live FORWARD NAV (built by `BybitFourEdgeStrategy`), so the gate would compare a capped ref against an uncapped forward and read a healthy book as "diverging below backtest" (reproduced: forward −25% vs ref −12.5% on a LABUSDT shock). Fail-safe (WAIT, never false-PASS) but defeats the branch's purpose. This threads the cap the same way `positioning_tail_cap_k` is already threaded.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fxhnt/application/bybit_forward_track.py` (`BybitFourEdgeStrategy.__init__` ~135-157, `advance` ~168-172)
|
||||
- Modify: `src/fxhnt/adapters/orchestration/assets.py` (builders at 105-122, 128-143, 149-158, 164-169; nightly call sites 868-876, 916-924, 957-965)
|
||||
- Modify: `src/fxhnt/adapters/orchestration/migration_builders.py` (builders at 106-117, 127-141, 148-156)
|
||||
- Modify: `src/fxhnt/application/bybit_positioning_eval.py:280` (`positioning_weights_and_contributions` — the Important SSOT-attribution fix)
|
||||
- Test: `tests/integration/test_positioning_forward_track.py` (+ the attribution invariant in `tests/integration/test_bybit_paper_reconciliation.py`)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `POSITIONING_COIN_GROSS_CAP = 0.07` (Task 4), `sleeve_returns_from_store(..., coin_gross_cap=...)` (Task 3), `positioning_weights(..., coin_gross_cap=...)` (Task 1).
|
||||
- Produces: `BybitFourEdgeStrategy(..., positioning_coin_gross_cap: float | None = None, ...)`; the four `assets.py` builders + three `migration_builders.py` builders gain a `positioning_coin_gross_cap` param forwarded to the strategy; the three nightly call sites pass `POSITIONING_COIN_GROSS_CAP`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `tests/integration/test_positioning_forward_track.py` — a shock store where the forward NAV's worst day is materially less negative when the cap is threaded (mirror the shock-store construction already used in the positioning tests):
|
||||
|
||||
```python
|
||||
def test_forward_nav_applies_coin_gross_cap() -> None:
|
||||
"""The forward NAV (BybitFourEdgeStrategy store-read path) must apply the per-coin cap when given, so it
|
||||
reconciles against the capped ref instead of diverging. Shock store: one over-short coin craters ~50%."""
|
||||
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
|
||||
from fxhnt.application.bybit_forward_track import BybitFourEdgeStrategy
|
||||
|
||||
_DAY = 86_400
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
px = {"LABUSDT": 100.0, "AAAUSDT": 100.0, "BBBUSDT": 100.0, "CCCUSDT": 100.0}
|
||||
for d in range(4):
|
||||
store.write_features("LABUSDT", [(d * _DAY, {"long_ratio": 0.05, "close": px["LABUSDT"]})])
|
||||
store.write_features("AAAUSDT", [(d * _DAY, {"long_ratio": 0.50, "close": px["AAAUSDT"]})])
|
||||
store.write_features("BBBUSDT", [(d * _DAY, {"long_ratio": 0.48, "close": px["BBBUSDT"]})])
|
||||
store.write_features("CCCUSDT", [(d * _DAY, {"long_ratio": 0.52, "close": px["CCCUSDT"]})])
|
||||
if d == 2:
|
||||
px["LABUSDT"] *= 0.50
|
||||
uncapped_rows, _ = BybitFourEdgeStrategy(store, sleeves=["positioning"]).advance(None, {})
|
||||
capped_rows, _ = BybitFourEdgeStrategy(
|
||||
store, sleeves=["positioning"], positioning_coin_gross_cap=0.07).advance(None, {})
|
||||
store.close()
|
||||
assert uncapped_rows and capped_rows, "forward strategy produced no rows — fixture broken"
|
||||
u = dict(uncapped_rows); c = dict(capped_rows)
|
||||
worst = min(u, key=lambda k: u[k])
|
||||
assert c[worst] > u[worst], f"cap must reduce the forward worst day: uncapped={u[worst]}, capped={c[worst]}"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_positioning_forward_track.py::test_forward_nav_applies_coin_gross_cap -v`
|
||||
Expected: FAIL with `TypeError: BybitFourEdgeStrategy.__init__() got an unexpected keyword argument 'positioning_coin_gross_cap'`
|
||||
|
||||
- [ ] **Step 3: Add the param to `BybitFourEdgeStrategy` and thread it in `advance`**
|
||||
|
||||
In `__init__` (after `positioning_tail_cap_k: float | None = None,` on line 142) add:
|
||||
```python
|
||||
positioning_coin_gross_cap: float | None = None,
|
||||
```
|
||||
And in the body (after `self._positioning_tail_cap_k = positioning_tail_cap_k` line 154):
|
||||
```python
|
||||
self._positioning_coin_gross_cap = positioning_coin_gross_cap
|
||||
```
|
||||
In `advance`'s store-read `sleeve_returns_from_store(...)` call (lines 168-172), add the kwarg:
|
||||
```python
|
||||
series = sleeve_returns_from_store(
|
||||
self._store, sleeve, universe=self._universe, cost_bps=self._cost_bps,
|
||||
unlock_events=self._unlock_events, stablecoin_spot_panel=self._stablecoin_spot_panel,
|
||||
capacity_capital=self._positioning_capacity_capital,
|
||||
tail_cap_k=self._positioning_tail_cap_k,
|
||||
coin_gross_cap=self._positioning_coin_gross_cap)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Thread through every builder (parallel to `positioning_tail_cap_k`)**
|
||||
|
||||
In `src/fxhnt/adapters/orchestration/assets.py`, for EACH of the four builders (`build_bybit_4edge_forward_nav`, `build_bybit_4edge_levered_forward_nav`, and the two single-sleeve builders at 149 and 164): add a `positioning_coin_gross_cap=None` param to the signature (next to `positioning_tail_cap_k=None`) and forward `positioning_coin_gross_cap=positioning_coin_gross_cap` into the `BybitFourEdgeStrategy(...)` call. For the THREE nightly call sites (868-876, 916-924, 957-965) that already pass `positioning_tail_cap_k=POSITIONING_TAIL_CAP_K`: add `positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` (import it alongside `POSITIONING_TAIL_CAP_K`).
|
||||
|
||||
Do the identical change in `src/fxhnt/adapters/orchestration/migration_builders.py` for its three builders that pass `positioning_tail_cap_k=POSITIONING_TAIL_CAP_K` (106-117, 127-141, 148-156): add `positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` to each `BybitFourEdgeStrategy(...)` call, importing the constant.
|
||||
|
||||
- [ ] **Step 5: Fix the attribution SSOT (the Important finding)**
|
||||
|
||||
In `src/fxhnt/application/bybit_positioning_eval.py`, `positioning_weights_and_contributions` (line 263): add `coin_gross_cap: float | None = None` to its signature and forward it in the `positioning_weights(...)` call at line 280:
|
||||
```python
|
||||
weights_by_day = positioning_weights(_ratio_by_day(long_ratio), direction=direction,
|
||||
coin_gross_cap=coin_gross_cap)
|
||||
```
|
||||
Then thread it through `sleeve_weights_and_contributions` in `src/fxhnt/application/bybit_paper_weights.py` (the caller at ~80-84) so the paper-book per-symbol weights match the capped sleeve return (preserving the `Σ weight·return == sleeve_returns_from_store` invariant documented at `bybit_paper_weights.py:9`). Default None everywhere.
|
||||
|
||||
- [ ] **Step 6: Run the tests + verify no regression**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_positioning_forward_track.py tests/integration/test_bybit_paper_reconciliation.py tests/integration/test_bybit_positioning_eval.py -v`
|
||||
Expected: all pass (new forward-nav cap test passes; existing reconciliation invariant test still passes with default-None on both sides).
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix(positioning): thread coin_gross_cap into the LIVE forward-nav path + attribution (final-review Critical)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Cap the LIVE paper-book POSITIONS + the report-ref path (Critical + Important from Task-7 review)
|
||||
|
||||
**Why:** The Task-7 review traced two more uncapped paths:
|
||||
- **Critical:** the live paper-book position sizing (`combined_symbol_weights*` / `latest_raw_sleeve_weights` → `derive_and_persist_bybit_paper_book` → the nightly `bybit_paper_book` asset) never passes the positioning haircuts, so the fund's ACTUAL persisted positions for the positioning sleeve are sized uncapped even though the reconciled return is capped. It turns out capacity + tail_cap were ALSO never threaded here (pre-existing gap). Operator decision (2026-07-20): thread ALL THREE haircuts so executed positions match the reconciled returns (full SSOT).
|
||||
- **Important:** `bybit_4edge_backtest_summary` + its caller `report_backtest_summary` pass `positioning_tail_cap_k` but no `positioning_coin_gross_cap` — a real, independently-callable uncapped ref path (masked in the scheduled flow only by the `_BYBIT_BOOK_SIDS` special-case).
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/fxhnt/application/bybit_paper_book.py` — `combined_symbol_weights` (~48), `combined_symbol_weights_and_returns` (~105), `latest_raw_sleeve_weights` (~80), and `derive_and_persist_bybit_paper_book` (the persist driver): add `positioning_capacity_capital`, `positioning_tail_cap_k`, `positioning_coin_gross_cap` (all `float | None = None`) and forward into every `sleeve_weights_and_contributions(...)` call.
|
||||
- Modify: `src/fxhnt/adapters/orchestration/assets.py` — `build_bybit_paper_book` (~180): add the 3 params, forward into `derive_and_persist_bybit_paper_book`; nightly call site (~1105): pass `positioning_capacity_capital=s.paper_capital, positioning_tail_cap_k=POSITIONING_TAIL_CAP_K, positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` (import the two constants).
|
||||
- Modify: `src/fxhnt/application/bybit_forward_track.py` — `bybit_4edge_backtest_summary` (~210): add `positioning_coin_gross_cap: float | None = None` next to `positioning_tail_cap_k`, forward into the `BybitFourEdgeStrategy(...)` call.
|
||||
- Modify: `src/fxhnt/application/backtest_refs_report.py` — `report_backtest_summary` (~42): import `POSITIONING_COIN_GROSS_CAP`, pass `positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` into `bybit_4edge_backtest_summary(...)`.
|
||||
- Test: `tests/integration/test_bybit_paper_reconciliation.py` (or the paper-book test file — grep for a test that calls `combined_symbol_weights_and_returns` / `derive_and_persist_bybit_paper_book`).
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `sleeve_weights_and_contributions(..., positioning_capacity_capital=None, positioning_tail_cap_k=None, positioning_coin_gross_cap=None)` (Task 7 added coin_gross_cap; VERIFY it also has capacity/tail params — if `sleeve_weights_and_contributions` does NOT yet accept capacity_capital/tail_cap_k, add them there too, forwarding into its `positioning_weights`/underlying call, same as coin_gross_cap).
|
||||
- Produces: the 3 haircut params on `combined_symbol_weights`, `combined_symbol_weights_and_returns`, `latest_raw_sleeve_weights`, `derive_and_persist_bybit_paper_book`, `build_bybit_paper_book`; `positioning_coin_gross_cap` on `bybit_4edge_backtest_summary` + `report_backtest_summary`. All default None.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add a paper-book position test: on a LABUSDT-shock store, the positioning sleeve's persisted per-symbol weight for the crashing coin is smaller when `positioning_coin_gross_cap=0.07` is threaded through `combined_symbol_weights_and_returns` than without.
|
||||
|
||||
```python
|
||||
def test_paper_book_positions_capped_by_coin_gross_cap() -> None:
|
||||
"""The live paper-book per-symbol weights must honor the per-coin cap so executed positions match the
|
||||
capped reconciled return (SSOT). On a store with one dominant positioning coin, the capped weight for that
|
||||
coin is strictly smaller than uncapped."""
|
||||
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
|
||||
from fxhnt.application.bybit_paper_book import combined_symbol_weights_and_returns
|
||||
_DAY = 86_400
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
for d in range(4):
|
||||
store.write_features("LABUSDT", [(d * _DAY, {"long_ratio": 0.02, "close": 100.0})])
|
||||
for i, sym in enumerate(("AAAUSDT", "BBBUSDT", "CCCUSDT", "DDDUSDT")):
|
||||
store.write_features(sym, [(d * _DAY, {"long_ratio": 0.45 + 0.02 * i, "close": 100.0})])
|
||||
w_unc, _r, _d = combined_symbol_weights_and_returns(store, sleeves=("positioning",))
|
||||
w_cap, _r2, _d2 = combined_symbol_weights_and_returns(
|
||||
store, sleeves=("positioning",), positioning_coin_gross_cap=0.07)
|
||||
store.close()
|
||||
assert w_unc.get("positioning") and w_cap.get("positioning"), "no positioning weights — fixture broken"
|
||||
lab_unc = abs(w_unc["positioning"].get("LABUSDT", 0.0))
|
||||
lab_cap = abs(w_cap["positioning"].get("LABUSDT", 0.0))
|
||||
assert lab_unc > 0.0
|
||||
assert lab_cap < lab_unc, f"cap must shrink the dominant coin's position: unc={lab_unc}, cap={lab_cap}"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_paper_reconciliation.py::test_paper_book_positions_capped_by_coin_gross_cap -v`
|
||||
Expected: FAIL with `TypeError: combined_symbol_weights_and_returns() got an unexpected keyword argument 'positioning_coin_gross_cap'`
|
||||
|
||||
- [ ] **Step 3: Thread the 3 haircuts through the paper-book weight functions**
|
||||
|
||||
For `combined_symbol_weights`, `combined_symbol_weights_and_returns`, and `latest_raw_sleeve_weights` in `bybit_paper_book.py`: add `positioning_capacity_capital: float | None = None`, `positioning_tail_cap_k: float | None = None`, `positioning_coin_gross_cap: float | None = None` to each signature, and forward all three into their `sleeve_weights_and_contributions(...)` call(s) (as `capacity_capital=`/`tail_cap_k=`/`positioning_coin_gross_cap=` per that function's actual param names — grep `def sleeve_weights_and_contributions` for the exact names). If `sleeve_weights_and_contributions` lacks capacity_capital/tail_cap_k params, add them there first (forwarding into its positioning path), then here.
|
||||
|
||||
- [ ] **Step 4: Thread through the persist driver + builder**
|
||||
|
||||
`derive_and_persist_bybit_paper_book` (bybit_paper_book.py): add the 3 params, forward into its `combined_symbol_weights_and_returns(...)` call. `build_bybit_paper_book` (assets.py:180): add the 3 params, forward into `derive_and_persist_bybit_paper_book(...)`. Nightly call site (assets.py:~1105): pass `positioning_capacity_capital=s.paper_capital, positioning_tail_cap_k=POSITIONING_TAIL_CAP_K, positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` (import both constants from `bybit_forward_track`).
|
||||
|
||||
- [ ] **Step 5: Fix the report-ref path (Important)**
|
||||
|
||||
`bybit_4edge_backtest_summary` (bybit_forward_track.py:~210): add `positioning_coin_gross_cap: float | None = None` next to `positioning_tail_cap_k`, forward into `BybitFourEdgeStrategy(..., positioning_coin_gross_cap=positioning_coin_gross_cap)`. `report_backtest_summary` (backtest_refs_report.py:~42): import `POSITIONING_COIN_GROSS_CAP`, pass `positioning_coin_gross_cap=POSITIONING_COIN_GROSS_CAP` into the `bybit_4edge_backtest_summary(...)` call.
|
||||
|
||||
- [ ] **Step 6: Run tests + no regression**
|
||||
|
||||
Run: `uv run --extra dev python -m pytest tests/integration/test_bybit_paper_reconciliation.py tests/integration/test_positioning_forward_track.py tests/unit/test_backtest_refs_report.py tests/integration/test_bybit_positioning_eval.py -v`
|
||||
Expected: all pass. Then confirm no site was missed: `grep -c positioning_tail_cap_k` vs `grep -c positioning_coin_gross_cap` should MATCH in bybit_paper_book.py, assets.py, bybit_forward_track.py, backtest_refs_report.py.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix(positioning): cap live paper-book positions + report-ref path (Task-7-review Critical+Important)"
|
||||
```
|
||||
474
docs/superpowers/plans/2026-07-20-vrp-code-removal.md
Normal file
474
docs/superpowers/plans/2026-07-20-vrp-code-removal.md
Normal file
@@ -0,0 +1,474 @@
|
||||
# VRP Strategy Full Removal — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox tracking; delete leaves before callers so imports never break mid-removal.
|
||||
|
||||
**Derived from:** a discovery workflow (24 file classifications + 3 deep-traces) — `wf_ffbdc8b5-c02`. The
|
||||
tricky shared-vs-VRP calls were resolved by trace: **DVOL pipeline = delete** (only VRP reads it),
|
||||
**black_scholes.py = delete** (VRP-only), **surfer vendor = untouched** (docstring mention only),
|
||||
**xsp_option_bars TABLE/MODEL/data = KEEP** (only the VRP-only `XspOptionBarsRepo` *class* is deleted).
|
||||
|
||||
## Goal
|
||||
|
||||
Fully delete the VRP (variance-risk-premium) strategy — the shelved/FALSIFIED equity-VRP
|
||||
sleeve and its executed twin `vrp_exec` — from the fxhnt repo, leaving **zero orphaned code,
|
||||
zero dead registry keys, and zero dangling imports**. The nightly Dagster pipeline, the cockpit
|
||||
web app, live IBKR/UCITS execution, the Bybit paper book, and all four kept crypto edges must
|
||||
remain fully functional and the pytest suite must be green at the end.
|
||||
|
||||
The removal spans four coupled clusters, all of which are VRP-exclusive per the deep-traces:
|
||||
|
||||
1. **VRP application modules**: `vrp_book`, `vrp_eval`, `vrp_defined_risk_eval`,
|
||||
`vrp_exec_record`, `vrp_marking`, `vrp_pricing`, `vrp_research`.
|
||||
2. **VRP option-math + IV plumbing**: `black_scholes.py` (VRP-only), the entire DVOL pipeline
|
||||
(`deribit_dvol.py`, `dvol_ingest.py`, the `dvol` feature panel, the unused
|
||||
`VolIndexClient.dvol()` port).
|
||||
3. **VRP persistence/orchestration**: `xsp_option_bars.py` repo class, the `vrp_nav` Dagster
|
||||
asset + its ~14 private OPRA-freeze/backfill helpers, the `backfill-xsp-opra` ingest path,
|
||||
the `VrpStrategy` builder, and the `vrp`/`vrp_exec` registry entries.
|
||||
4. **Comment/registry-string scrubs** across shared cockpit infrastructure that merely *names*
|
||||
VRP as an illustrative example.
|
||||
|
||||
## Architecture
|
||||
|
||||
VRP was ARCHIVED 2026-07-15 (fails the LOO marginal-Sharpe gate; see
|
||||
`pearl_fxhnt_diversifier_hunt_2026_07_15`). The two registry entries already carry
|
||||
`archived: True`, and `vrp_nav` is already **de-wired** from `definitions.py`'s nightly asset
|
||||
graph — so no live nightly asset materializes VRP today. What remains is dead weight plus two
|
||||
active-but-purposeless CLI/CronJob ingest paths (`ingest-dvol`, `backfill-xsp-opra`) and a set
|
||||
of stale registry keys/comment references.
|
||||
|
||||
Dependency direction (leaf → root), which dictates task order so imports never break mid-removal:
|
||||
|
||||
```
|
||||
black_scholes.py ────────────► vrp_defined_risk_eval.py ──┐
|
||||
deribit_dvol.py ─► dvol_ingest.py ─► (dvol panel) ────────┤
|
||||
xsp_option_bars.py (repo class) ──► vrp_book.py ──────────┤──► cli.py commands
|
||||
vrp_pricing.py / vrp_marking.py ──► vrp_book/eval/exec ───┤ migration_builders (_build_vrp)
|
||||
└──► assets.py vrp_nav + helpers
|
||||
registry.py vrp / vrp_exec
|
||||
forward_ingest _REF_ALIAS
|
||||
sim_curves / dashboard_service /
|
||||
app.py / definitions.py (scrub)
|
||||
```
|
||||
|
||||
Rule: delete leaf modules **and their tests together** first, then the callers that referenced
|
||||
them (asset graph, registry, CLI, migration builder), then the pure comment-scrub edits last.
|
||||
|
||||
## DVOL and black_scholes — resolved decisions (per deep-traces)
|
||||
|
||||
- **DVOL pipeline → DELETE (do not keep).** Exhaustive grep confirms the only readers of the
|
||||
`dvol` feature panel (`store.read_panel(..., "dvol")`) are `vrp_eval.py` and
|
||||
`vrp_defined_risk_eval.py`. `VolIndexClient.dvol()` (`ports/market_data.py:22`) has **zero**
|
||||
callers. DVOL is not a Dagster asset. DVOL is the Deribit IV index — a data source entirely
|
||||
distinct from the OPRA `xsp_option_bars` data being kept; there is no keep-carve-out for it.
|
||||
Delete `deribit_dvol.py`, `dvol_ingest.py`, the `ingest-dvol` CLI command, the
|
||||
`VolIndexClient.dvol()` port method, and their tests. The `dvol` warehouse rows in
|
||||
`bybit_features` are inert once nothing reads them — no migration needed; they can age out.
|
||||
|
||||
- **black_scholes.py → DELETE (VRP-only, not shared math).** The only importers are
|
||||
`vrp_defined_risk_eval.py` and its own unit test `tests/unit/test_black_scholes.py`.
|
||||
`vrp_pricing.py` defines a *separate* `bs_put` (different signature) and does **not** import
|
||||
from `black_scholes.py`, so there is no hidden cross-file sharing. Delete both files.
|
||||
|
||||
- **`xsp_option_bars` TABLE + `XspOptionBarRow` model + OPRA `.dbn` → KEEP.** Only the
|
||||
`XspOptionBarsRepo` *repo class* (`adapters/persistence/xsp_option_bars.py`) is VRP-only and
|
||||
gets deleted. The ORM row model `XspOptionBarRow` in `cockpit_models.py`, the physical table,
|
||||
the 13-year OPRA data, and `ForwardNavRepo.xsp_freeze_max_date()` (raw `SELECT max(date)`)
|
||||
all stay untouched.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
1. **KEEP the OPRA data.** Do **not** drop or migrate the `xsp_option_bars` table, do **not**
|
||||
remove `XspOptionBarRow` from `cockpit_models.py`, do **not** delete OPRA `.dbn` files, and
|
||||
do **not** touch `ForwardNavRepo.xsp_freeze_max_date()`. Only the VRP *ingest/consumer*
|
||||
machinery (`XspOptionBarsRepo` class, `_backfill_month`/`_freeze_*` helpers, `backfill-xsp-opra`)
|
||||
is removed. The freeze-staleness health axis (`_FREEZE_SOURCES = {"opra-pit"}`) stays generic.
|
||||
2. **Real-money paths must err false-WAIT, never false-EXECUTE.** VRP execution
|
||||
(`execute-vrp`/`vrp_exec_record`) is being deleted, not disabled — but during the removal no
|
||||
edit may cause any *surviving* real-money path (multistrat/UCITS IBKR, Bybit paper) to
|
||||
silently promote/execute on missing state. After removing the `vrp`/`vrp_exec` registry keys,
|
||||
confirm the cockpit gate and `forward_ingest` promotion loop still default to WAIT for every
|
||||
surviving strategy and never KeyError into a false-PASS.
|
||||
3. **Full pytest suite green before finishing.** Run `uv run pytest -n auto` (or
|
||||
`python -m pytest -n auto` per `pyproject.toml addopts = "-n auto"`) from the repo root and
|
||||
confirm 0 failures / 0 errors before the final commit. Per-task commands below are the fast
|
||||
local gate; the full-suite run in the final task is the release gate.
|
||||
4. **The false VRP STALE health alarm must be gone.** After removal, verify **no `vrp` key
|
||||
remains in `STRATEGY_REGISTRY`** (`vrp` and `vrp_exec` both removed) so the forward-health
|
||||
freeze/staleness axis and `dashboard_service._active_ibkr_account_sids()` can no longer
|
||||
surface a ghost VRP STALE row. Verification command in the final task.
|
||||
|
||||
### Pre-flight note (already-broken test)
|
||||
|
||||
`tests/unit/test_vrp_rebalancer_yaml.py` asserts the existence of
|
||||
`infra/k8s/jobs/fxhnt-vrp-rebalancer.yaml`, which **no longer exists** in the tree. This test is
|
||||
already failing/erroring at HEAD; it is deleted in Task 1 alongside the other VRP tests, which
|
||||
also removes that pre-existing red.
|
||||
|
||||
---
|
||||
|
||||
## Task 1 — Delete VRP application modules + their tests (leaves first)
|
||||
|
||||
**Files to delete (whole file):**
|
||||
- `src/fxhnt/application/vrp_book.py`
|
||||
- `src/fxhnt/application/vrp_eval.py`
|
||||
- `src/fxhnt/application/vrp_defined_risk_eval.py`
|
||||
- `src/fxhnt/application/vrp_exec_record.py`
|
||||
- `src/fxhnt/application/vrp_marking.py`
|
||||
- `src/fxhnt/application/vrp_pricing.py`
|
||||
- `src/fxhnt/application/vrp_research.py`
|
||||
- `src/fxhnt/application/black_scholes.py`
|
||||
- `tests/unit/test_vrp_book.py`
|
||||
- `tests/unit/test_vrp_pricing.py`
|
||||
- `tests/unit/test_vrp_exec_helpers.py`
|
||||
- `tests/unit/test_vrp_exec_ladder_cap.py`
|
||||
- `tests/unit/test_vrp_exec_record.py`
|
||||
- `tests/unit/test_vrp_rebalancer_yaml.py` (also clears the pre-existing red — see pre-flight note)
|
||||
- `tests/unit/test_registry_vrp.py`
|
||||
- `tests/unit/test_black_scholes.py`
|
||||
- `tests/integration/test_vrp_eval.py`
|
||||
- `tests/integration/test_vrp_defined_risk_eval.py`
|
||||
- `tests/integration/test_vrp_nav_asset.py`
|
||||
- `tests/integration/test_cockpit_vrp_render.py`
|
||||
|
||||
**Rationale:** every one of these modules is imported only by other `vrp_*` modules or by the
|
||||
`cli.py`/`assets.py`/`migration_builders.py` callers handled in later tasks. Deleting the tests
|
||||
in the *same* task prevents collection-time `ImportError` on the deleted modules. Do NOT yet
|
||||
touch `assets.py`, `cli.py`, `registry.py`, or `migration_builders.py` — those still reference
|
||||
these modules only via **lazy (in-function) imports**, so the module tree still imports cleanly
|
||||
after this task (the broken imports are inside function bodies not executed at import time). They
|
||||
are removed in Tasks 3–5.
|
||||
|
||||
**Covering test command** (must import-clean and not collect the deleted files):
|
||||
```
|
||||
uv run pytest tests/unit tests/integration -q --co 2>&1 | tail -20
|
||||
```
|
||||
Expect: collection succeeds with the deleted VRP tests absent and no `ImportError`.
|
||||
|
||||
**Commit:** `refactor(vrp): delete VRP application modules + black_scholes + their tests`
|
||||
|
||||
---
|
||||
|
||||
## Task 2 — Delete the DVOL pipeline + its tests
|
||||
|
||||
**Files to delete (whole file):**
|
||||
- `src/fxhnt/adapters/data/deribit_dvol.py`
|
||||
- `src/fxhnt/application/dvol_ingest.py`
|
||||
- `tests/integration/test_deribit_dvol.py`
|
||||
- `tests/integration/test_dvol_ingest.py`
|
||||
|
||||
**Edit `src/fxhnt/ports/market_data.py`:**
|
||||
- Remove the `VolIndexClient` Protocol (the `dvol()` method block, ~lines 21–24). Confirmed
|
||||
zero implementers and zero callers repo-wide; it is dead scaffolding.
|
||||
|
||||
**Do NOT touch:** `deribit_funding.py` and the `ingest-deribit-funding` CronJob/command —
|
||||
that is the *funding* (cross-venue carry) edge, a separate kept data source that reads the
|
||||
`funding`/`deribit_funding` features, never `dvol`.
|
||||
|
||||
**Rationale:** `ingest_dvol` and `DeribitDVOL` are consumed only by the `ingest-dvol` CLI
|
||||
command (removed in Task 3) which fed only the two VRP evaluators (deleted in Task 1). The
|
||||
`dvol` feature panel now has no readers.
|
||||
|
||||
**Covering test command:**
|
||||
```
|
||||
uv run pytest tests/integration -q --co 2>&1 | tail -20
|
||||
grep -rn "VolIndexClient\|import.*deribit_dvol\|import.*dvol_ingest" src/ | grep -v "/cli.py"
|
||||
```
|
||||
Expect: no collection errors; the grep returns only the (still-present, removed next task)
|
||||
`cli.py` lazy imports.
|
||||
|
||||
**Commit:** `refactor(vrp): delete DVOL pipeline (deribit_dvol, dvol_ingest, VolIndexClient port) + tests`
|
||||
|
||||
---
|
||||
|
||||
## Task 3 — Delete VRP + DVOL commands from `cli.py`
|
||||
|
||||
**File to edit:** `src/fxhnt/cli.py`
|
||||
|
||||
**Exact edits (delete each command block in full, including its in-function lazy imports):**
|
||||
- Delete `@app.command("execute-vrp")` `execute_vrp` (lines ~439–499), including the lazy
|
||||
`from fxhnt.application.vrp_exec_record import plan_and_record_vrp`. **Keep** the module-level
|
||||
and other commands' use of `ForwardNavRepo` — only remove this command's local import line if
|
||||
it has one; the `ForwardNavRepo` module itself is reused by `execute-multistrat` and others.
|
||||
- Delete `@app.command("ingest-dvol")` `ingest_dvol_cmd` (lines ~1217–1266), including the lazy
|
||||
`from fxhnt.adapters.data.deribit_dvol import DeribitDVOL` and
|
||||
`from fxhnt.application.dvol_ingest import ingest_dvol`.
|
||||
- Delete `@app.command("vrp-eval")` `vrp_eval` (lines ~2193–2288), including the lazy
|
||||
`from fxhnt.application.vrp_eval import (verify_vrp_edge, vrp_metrics_from_store, walk_forward_vrp)`.
|
||||
- Delete `@app.command("vrp-defined-risk-eval")` `vrp_defined_risk_eval` (lines ~2291–2403),
|
||||
including the lazy `from fxhnt.application.vrp_defined_risk_eval import (...)`.
|
||||
- Delete the private helpers `_print_vrp_verify` (~2406–2446) and `_print_vrp_walk_forward`
|
||||
(~2449–2469) — called only by the two eval commands above.
|
||||
|
||||
**Decision — `backfill-xsp-opra` (lines ~578–623): DELETE this command too.** It calls
|
||||
`_backfill_month` and `XspOptionBarsRepo`, both of which are deleted in Tasks 4/5, and its sole
|
||||
purpose is freezing OPRA marks into `xsp_option_bars` **for VRP recompute**. Keeping the OPRA
|
||||
*data* (Constraint 1) does not require keeping the *ingest path* — the data already exists and
|
||||
the only consumer of freshly-frozen marks was VRP. Delete the `@app.command("backfill-xsp-opra")`
|
||||
`backfill_xsp_opra` function and its `from fxhnt.adapters.orchestration.assets import _backfill_month`
|
||||
+ `from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo` local imports. (If
|
||||
the executor finds a documented non-VRP consumer of freshly-frozen OPRA marks, stop and flag —
|
||||
none was found in the trace.)
|
||||
|
||||
**Covering test command:**
|
||||
```
|
||||
uv run python -c "import fxhnt.cli" # module must import clean
|
||||
uv run fxhnt --help >/dev/null # Typer app builds with commands gone
|
||||
grep -n "vrp\|dvol\|backfill-xsp\|VrpStrategy\|black_scholes\|XspOptionBarsRepo" src/fxhnt/cli.py
|
||||
```
|
||||
Expect: import + `--help` succeed; grep returns no hits.
|
||||
|
||||
**Commit:** `refactor(vrp): remove execute-vrp/ingest-dvol/vrp-eval/vrp-defined-risk-eval/backfill-xsp-opra CLI commands`
|
||||
|
||||
---
|
||||
|
||||
## Task 4 — Remove the `vrp_nav` asset + OPRA-freeze helpers from `assets.py`
|
||||
|
||||
**File to edit:** `src/fxhnt/adapters/orchestration/assets.py`
|
||||
|
||||
**Exact edits (delete code):**
|
||||
- Delete the `@asset`-decorated `vrp_nav` function (lines ~686–718), including its inline
|
||||
imports of `DatabentoDataProvider`, `XspOptionBarsRepo`, `VrpStrategy` (~line 694).
|
||||
- Delete the exclusive private helper chain (lines ~492–683), all VRP-only and now callerless
|
||||
after Tasks 1/3: `_iso_date`, `_forward_from_marks` (+ its inline
|
||||
`fxhnt.application.vrp_pricing.parity_forward` import ~502), `_band_contracts`,
|
||||
`_band_rows_from_marks`, `_day_marks`, `_xsp_forward_estimate`, `_freeze_xsp_slice`,
|
||||
`_freeze_day`, `_prev_trading_day`, `_DEGENERATE_FORWARD_MARKERS`,
|
||||
`_is_degenerate_forward_error`, `_freeze_latest_session` (+ its inline
|
||||
`DatabentoRangeUnavailable` import ~614), `_rough_forward`, `_backfill_month`.
|
||||
- Delete the trailing `# vrp_nav ARCHIVED 2026-07-15 — de-wired (fn kept below)` comment
|
||||
(~line 1131) — the function no longer exists below.
|
||||
- Reword the module docstring (~line 4): drop the trailing `/vrp` from the forward-track asset
|
||||
enumeration.
|
||||
|
||||
**Do NOT touch:** `deribit_funding_bars` (~825–856, unrelated Deribit perp *funding*, kept for
|
||||
cross-venue-carry), and all other assets (`futures_bars`, `bybit_*`, `multistrat_*`,
|
||||
`sixtyforty_nav`, `positioning_nav`, `xsfunding_nav`, `unlock_nav`, `cockpit_forward`, …). The
|
||||
RETIRED-track comments at ~721 and ~748 do not mention vrp — leave as-is.
|
||||
|
||||
**Covering test command:**
|
||||
```
|
||||
uv run python -c "import fxhnt.adapters.orchestration.assets"
|
||||
uv run python -c "import fxhnt.adapters.orchestration.definitions" # Dagster code-location loads
|
||||
grep -n "vrp\|Vrp\|_backfill_month\|_freeze_\|XspOptionBarsRepo\|vrp_pricing" src/fxhnt/adapters/orchestration/assets.py
|
||||
uv run pytest tests/integration/test_orchestration_definitions.py -q
|
||||
```
|
||||
Expect: both modules import clean, grep returns no hits, definitions test passes.
|
||||
|
||||
**Commit:** `refactor(vrp): delete vrp_nav Dagster asset + OPRA-freeze/backfill helper chain`
|
||||
|
||||
---
|
||||
|
||||
## Task 5 — Remove `XspOptionBarsRepo` class + the `_build_vrp` migration builder
|
||||
|
||||
**File to delete (whole file):**
|
||||
- `src/fxhnt/adapters/persistence/xsp_option_bars.py` (the `XspOptionBarsRepo` repo class).
|
||||
All callers (`vrp_book.py`, `vrp_exec_record.py`, `vrp_marking.py`, the `vrp_nav` asset, the
|
||||
`backfill-xsp-opra` command) are gone after Tasks 1/3/4.
|
||||
- **VERIFIED 2026-07-20: `tests/integration/test_xsp_option_bars_repo.py` does NOT exist in the tree** —
|
||||
do not attempt to delete it. If a repo-class test surfaces under another name during removal, delete it;
|
||||
otherwise there is no test file for this class.
|
||||
|
||||
**KEEP (Constraint 1):** `cockpit_models.XspOptionBarRow`, the `xsp_option_bars` table, and
|
||||
`ForwardNavRepo.xsp_freeze_max_date()` — none of these import or depend on the repo class.
|
||||
|
||||
**File to edit:** `src/fxhnt/adapters/orchestration/migration_builders.py`
|
||||
- Delete the `_build_vrp` closure (lines ~71–76) and its `builders["vrp"] = ("vrp_state",
|
||||
_build_vrp)` registration (~78), plus the preceding `# ---- vrp_nav: ...` comment banner (~71).
|
||||
|
||||
**Covering test command:**
|
||||
```
|
||||
uv run python -c "import fxhnt.adapters.orchestration.migration_builders"
|
||||
grep -rn "XspOptionBarsRepo\|_build_vrp\|builders\[.vrp" src/
|
||||
uv run pytest tests/ -q -k "migration or migrate" 2>&1 | tail -20
|
||||
```
|
||||
Expect: import clean, grep returns no hits (the `XspOptionBarRow` model in `cockpit_models.py`
|
||||
is a different symbol and must still be present — verify separately:
|
||||
`grep -n "class XspOptionBarRow" src/fxhnt/adapters/persistence/cockpit_models.py` returns 1 hit).
|
||||
|
||||
**Commit:** `refactor(vrp): delete XspOptionBarsRepo class + _build_vrp migration builder (keep table + XspOptionBarRow)`
|
||||
|
||||
---
|
||||
|
||||
## Task 6 — Remove `vrp`/`vrp_exec` registry keys + `forward_ingest` alias
|
||||
|
||||
**File to edit:** `src/fxhnt/registry.py`
|
||||
- Delete the `"vrp"` `STRATEGY_REGISTRY` entry (lines ~161–172) and its preceding comment
|
||||
block (~157).
|
||||
- Delete the `"vrp_exec"` `STRATEGY_REGISTRY` entry (lines ~177–186) and its preceding comment
|
||||
block (~173–176).
|
||||
- Keep the dict itself and all 9 non-VRP entries and the `FUND_INSTRUMENTS` import/re-export.
|
||||
|
||||
**File to edit:** `src/fxhnt/application/forward_ingest.py`
|
||||
- Remove the `"vrp_exec": "vrp"` pair from `_REF_ALIAS` (line ~45), leaving
|
||||
`{"multistrat_exec": "multistrat", "bybit_4edge_exec": "bybit_4edge"}`.
|
||||
|
||||
**Rationale + Constraint 4:** removing the `vrp` key is what actually kills the false VRP STALE
|
||||
health alarm — with no `vrp` in `STRATEGY_REGISTRY`, the forward-health freeze/staleness loop
|
||||
and `dashboard_service._active_ibkr_account_sids()` can no longer iterate a ghost VRP row.
|
||||
(Note: `dashboard_service._IBKR_ACCOUNT_SIDS` hard-codes `"vrp"`; that literal is fixed in
|
||||
Task 7 in the *same PR* so the window where the key is gone but the literal remains never ships.)
|
||||
|
||||
**Covering test command:**
|
||||
```
|
||||
uv run python -c "from fxhnt.registry import STRATEGY_REGISTRY; assert 'vrp' not in STRATEGY_REGISTRY and 'vrp_exec' not in STRATEGY_REGISTRY, STRATEGY_REGISTRY.keys(); print('registry clean')"
|
||||
uv run pytest tests/unit/test_forward_definition.py tests/integration/test_forward_health.py tests/integration/test_dashboard_service.py -q 2>&1 | tail -20
|
||||
```
|
||||
Expect: assertion passes; forward/dashboard tests green (these confirm surviving strategies
|
||||
still WAIT-by-default — Constraint 2).
|
||||
|
||||
**Commit:** `refactor(vrp): remove vrp/vrp_exec registry entries + forward_ingest _REF_ALIAS`
|
||||
|
||||
---
|
||||
|
||||
## Task 7 — Fix the one behavioral literal + `sim_curves` book pill (data edits)
|
||||
|
||||
These are the two non-comment edits that gate on `vrp` being a *present* registry key and would
|
||||
misbehave once it is gone (Task 6). They ship together with — or immediately after — Task 6.
|
||||
|
||||
**File to edit:** `src/fxhnt/application/dashboard_service.py`
|
||||
- Line ~30: change `_IBKR_ACCOUNT_SIDS = frozenset({"multistrat", "vrp"})` to
|
||||
`frozenset({"multistrat"})`. **Why this is behavioral, not cosmetic:** `_active_ibkr_account_sids()`
|
||||
computes `not STRATEGY_REGISTRY.get("vrp", {}).get("archived")`; once the `vrp` key is gone
|
||||
this becomes `not None` = `True`, wrongly treating `vrp` as an ACTIVE ibkr-account sid and
|
||||
rendering a ghost/broken VRP row on the IBKR account page. Removing the literal eliminates it.
|
||||
|
||||
**File to edit:** `src/fxhnt/application/sim_curves.py`
|
||||
- Line ~15: remove the `"vrp"` string from the `SIM_BOOKS` tuple, leaving
|
||||
`("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered")`. **Why:** once
|
||||
the `vrp` registry entry is gone, a `vrp` Backtest-page pill would `KeyError`/500 at request
|
||||
time in `app.py` (`registry_backtest_kind("vrp")`). The dispatch logic in `sim_returns_for`
|
||||
is generic and unchanged.
|
||||
|
||||
**Covering test command:**
|
||||
```
|
||||
uv run pytest tests/unit/test_sim_curves.py tests/integration/test_dashboard_service.py tests/integration/test_ibkr_account_detail_web.py tests/integration/test_dashboard_ibkr_holdings.py tests/integration/test_paper_books_web.py tests/integration/test_ibkr_sim_web.py -q 2>&1 | tail -20
|
||||
```
|
||||
Expect: green; no VRP row/pill; surviving books render.
|
||||
|
||||
**Commit:** `fix(vrp): drop vrp from _IBKR_ACCOUNT_SIDS and SIM_BOOKS so ghost VRP row/pill can't render`
|
||||
|
||||
---
|
||||
|
||||
## Task 8 — Delete the now-purposeless VRP/OPRA CronJob manifest
|
||||
|
||||
**File to delete:**
|
||||
- `infra/k8s/jobs/fxhnt-opra-backfill.yaml` — invokes `fxhnt backfill-xsp-opra` (deleted in
|
||||
Task 3). With the command gone this CronJob would crash-loop; its only job was freezing OPRA
|
||||
marks for VRP recompute. (The `fxhnt-vrp-rebalancer.yaml` manifest is already absent from the
|
||||
tree — nothing to delete there; its dangling test is removed in Task 1.)
|
||||
|
||||
**Do NOT touch:** `fxhnt-deribit-carry-job.yaml` (funding/carry, not DVOL),
|
||||
`fxhnt-bybit-*`, `fxhnt-ucits-rebalancer.yaml`, `fxhnt-ibkr-rebalancer.yaml`, and all other
|
||||
manifests.
|
||||
|
||||
**Covering check:**
|
||||
```
|
||||
grep -rln "vrp\|ingest-dvol\|execute-vrp\|backfill-xsp-opra" infra/ # expect: no hits
|
||||
```
|
||||
|
||||
**Commit:** `chore(vrp): remove fxhnt-opra-backfill CronJob (backfill-xsp-opra command deleted)`
|
||||
|
||||
---
|
||||
|
||||
## Task 9 — Comment/docstring scrubs (no logic change) — LAST
|
||||
|
||||
Pure prose edits that dangling-reference the deleted `vrp` strategy id. No imports, logic,
|
||||
registry entries, or types change. Batched last because none affect importability.
|
||||
|
||||
**Files + edits (reword to drop the `vrp` example, keep meaning):**
|
||||
- `src/fxhnt/adapters/orchestration/definitions.py` — lines ~10–12: drop the
|
||||
"vrp_nav ARCHIVED … KEPT in assets.py but DE-WIRED" paragraph (the "KEPT" claim is now false);
|
||||
replace with a one-line "VRP removed entirely 2026-07-20" note or delete outright.
|
||||
- `src/fxhnt/adapters/persistence/cockpit_models.py` — line ~46 (drop the `(… that would have
|
||||
caught vrp)` parenthetical or generalize to "a dead strategy"); line ~228 (drop `/vrp` from
|
||||
`(multistrat/vrp …)`).
|
||||
- `src/fxhnt/adapters/persistence/forward_nav.py` — lines ~146–147 (generalize "the vrp freeze
|
||||
table" / "a vrp recompute is bounded by this max date" to opra-pit-anchored/strategy-generic
|
||||
wording; the `xsp_freeze_max_date` query stays); line ~199 (drop the "shelved/FALSIFIED
|
||||
vrp/vrp_exec" example from the `archived: True` registry note).
|
||||
- `src/fxhnt/adapters/persistence/paper_repo.py` — line ~519 (drop `vrp` from the
|
||||
`(multistrat/vrp …)` recompute-replay parenthetical).
|
||||
- `src/fxhnt/adapters/web/app.py` — lines ~449, 453, 473, 482, 551, 567, 728, 752, 810: drop the
|
||||
`/vrp` example from each `multistrat/vrp` phrase. No code change (the archived-filter
|
||||
comprehension already yields one fewer element once `vrp` left `SIM_BOOKS`/registry).
|
||||
- `src/fxhnt/adapters/web/templates/_macros.html` — line ~61 ("2-week vrp rot"): generalize.
|
||||
- `src/fxhnt/adapters/web/templates/cockpit.html` — line ~78: drop `/vrp` from
|
||||
"two IBKR real-account books (multistrat/vrp)".
|
||||
- `src/fxhnt/application/dashboard_service.py` — comment-only lines ~27, 55, 89, 390, 416, 417:
|
||||
drop/generalize the `vrp` examples (the code literal was already fixed in Task 7).
|
||||
- `src/fxhnt/application/sim_curves.py` — lines ~2, 12, 22: drop `vrp` from the
|
||||
`multistrat/vrp` recompute-replay examples (the tuple was already fixed in Task 7).
|
||||
- `src/fxhnt/application/forward_health.py` — docstring lines ~3, 11, 13, 14, 15, 24, 31 and the
|
||||
inline comment at ~147 ("for the vrp freeze axis"): generalize away from naming `vrp` as the
|
||||
mechanism's reason for existing; the `_FREEZE_SOURCES = {"opra-pit"}` logic and
|
||||
`xsp_freeze_max_date()` call are unchanged.
|
||||
- `src/fxhnt/application/display_names.py` — lines ~2 and ~51: drop the `vrp` example id /
|
||||
option-contract example from the illustrative prose.
|
||||
- `src/fxhnt/application/em_asia_marginal_eval.py` — lines ~6, 19, 71: replace "VRP" in the
|
||||
Stage-1 kill-gate precedent comments with a VRP-agnostic phrase ("prior failed diversifier
|
||||
candidates"). No logic change.
|
||||
- `src/fxhnt/config.py` — line ~215: drop `(… that would have caught vrp)` from the
|
||||
`stale_after_business_days` comment; keep the setting + default.
|
||||
- `src/fxhnt/ports/dashboard.py` — lines ~72 and ~190: drop the `vrp` example from the
|
||||
`multistrat/vrp` IBKR-real-account docstrings. (Line ~34 is unrelated carry — leave.)
|
||||
- `src/fxhnt/vendor/surfer/__init__.py` — lines ~11–13: compress the `surfer_poc`/
|
||||
`MomentumVrpStrategy` retirement note to a plain historical line without the VRP signpost.
|
||||
|
||||
**Do NOT edit** as part of scrubbing (already covered / must not change): the `xsp_option_bars`
|
||||
query, `_FREEZE_SOURCES`, `XspOptionBarRow`, any settings field, or any Protocol field.
|
||||
|
||||
**Covering test command** (broad — scrubs must not have touched logic):
|
||||
```
|
||||
uv run python -c "import fxhnt.adapters.web.app, fxhnt.application.dashboard_service, fxhnt.application.forward_health, fxhnt.config, fxhnt.ports.dashboard, fxhnt.application.display_names, fxhnt.application.em_asia_marginal_eval, fxhnt.application.sim_curves"
|
||||
uv run pytest tests/unit/test_display_names.py tests/integration/test_dashboard_service.py -q 2>&1 | tail -20
|
||||
```
|
||||
Expect: all import clean; tests green.
|
||||
|
||||
**Commit:** `docs(vrp): scrub stale VRP comment/docstring references across cockpit + shared infra`
|
||||
|
||||
---
|
||||
|
||||
## Task 10 — Full-suite green gate + removal verification (release gate)
|
||||
|
||||
**No file edits.** Final verification against all Global Constraints.
|
||||
|
||||
**Commands (all must pass):**
|
||||
```
|
||||
# (3) full suite green
|
||||
uv run pytest -n auto 2>&1 | tail -30 # 0 failed, 0 errors
|
||||
|
||||
# (4) no vrp in registry — false STALE alarm gone
|
||||
uv run python -c "from fxhnt.registry import STRATEGY_REGISTRY; assert not any(k.startswith('vrp') for k in STRATEGY_REGISTRY), list(STRATEGY_REGISTRY); print('no vrp in registry')"
|
||||
|
||||
# no orphaned VRP code / imports anywhere in src
|
||||
grep -rn "import.*vrp_\|VrpStrategy\|deribit_dvol\|dvol_ingest\|black_scholes\|XspOptionBarsRepo\|VolIndexClient\|_backfill_month\|plan_and_record_vrp" src/ # expect: no hits
|
||||
ls src/fxhnt/application/vrp_*.py 2>&1 # expect: No such file
|
||||
ls src/fxhnt/application/black_scholes.py src/fxhnt/application/dvol_ingest.py src/fxhnt/adapters/data/deribit_dvol.py src/fxhnt/adapters/persistence/xsp_option_bars.py 2>&1 # expect: No such file (all four)
|
||||
|
||||
# (1) OPRA data kept — table model + freeze query still present
|
||||
grep -n "class XspOptionBarRow" src/fxhnt/adapters/persistence/cockpit_models.py # expect: 1 hit
|
||||
grep -n "def xsp_freeze_max_date" src/fxhnt/adapters/persistence/forward_nav.py # expect: 1 hit
|
||||
|
||||
# (2) Dagster code-location + cockpit app both import clean (no false-execute / no outage)
|
||||
uv run python -c "import fxhnt.adapters.orchestration.definitions; import fxhnt.adapters.web.app; print('definitions + app import clean')"
|
||||
|
||||
# no residual vrp mentions in infra
|
||||
grep -rln "vrp\|ingest-dvol\|execute-vrp\|backfill-xsp-opra" infra/ # expect: no hits
|
||||
```
|
||||
|
||||
**Constraint 2 spot-check:** confirm `forward_ingest`'s promotion loop and the cockpit gate
|
||||
default to WAIT for every surviving strategy (the recon-gate never false-PASSes) — covered by
|
||||
`test_forward_health.py` / `test_dashboard_service.py` in the full run; if either regresses, the
|
||||
removal touched a shared gate path and must be reverted to the last green task.
|
||||
|
||||
**Commit (only after everything above is green):** `chore(vrp): verify full VRP removal — suite green, registry clean, OPRA data intact`
|
||||
|
||||
---
|
||||
|
||||
## Post-removal note
|
||||
|
||||
After merge, update `docs/superpowers/VRP-REMOVAL-TODO.md` and project memory
|
||||
(`pearl_fxhnt_diversifier_hunt_2026_07_15.md` already says "VRP now fully DELETED (shelved)") to
|
||||
reflect that the code deletion — not just shelving — is complete, and that the OPRA
|
||||
`xsp_option_bars` data was retained per Constraint 1.
|
||||
269
docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md
Normal file
269
docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# fxhnt CI/CD Pipeline — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this
|
||||
> plan task-by-task. Steps use checkbox tracking. This plan writes/edits YAML manifests and applies them to a
|
||||
> LIVE cluster — every apply is gated behind a dry-run + explicit verify; a broken sensor/workflow must never
|
||||
> silently swallow a push or false-deploy.
|
||||
|
||||
**Goal:** A push to `master` on `gitadmin/fxhnt` auto-builds (only when deps/Dockerfile change) and auto-deploys
|
||||
the fxhnt cockpit+dagster daemons — no manual `kubectl rollout restart`.
|
||||
|
||||
**Architecture:** git-sync-first, image-on-demand. A minimal Sensor in `argo-events` hears the shared Gitea
|
||||
webhook, path-filters the push to pick `mode=build` (Kaniko rebuild) vs `mode=deploy` (rollout only), and submits
|
||||
the fxhnt-owned `fxhnt-cockpit` WorkflowTemplate into `foxhunt`.
|
||||
|
||||
**Tech Stack:** Argo Events v1.9.10 (Sensor), Argo Workflows (Kaniko build + kubectl deploy), Gitea webhook,
|
||||
Scaleway registry, K8s (bizworx Kapsule).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **Project boundary:** fxhnt-owned objects (WorkflowTemplate, build/deploy pods, secrets, RBAC) live in
|
||||
`foxhunt`. ONLY the Sensor lives in `argo-events` (unavoidable). No fxhnt secrets in `argo`/`argo-events`; no
|
||||
cross-namespace RBAC.
|
||||
- **Node pool:** all build/deploy pods target `nodeSelector: k8s.scaleway.com/pool-name: large` (the
|
||||
`ci-compile-cpu` pool is DECOMMISSIONED — only `large`/`sfs` exist). A pod pinned to `ci-compile-cpu` never
|
||||
schedules.
|
||||
- **Registry:** `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit`, tagged `:<sha7>` AND `:latest`. Secret
|
||||
`scw-registry` (already in `foxhunt`).
|
||||
- **Real-money safety:** the pipeline only builds/deploys the cockpit+dagster DAEMONS. It must NEVER touch the
|
||||
execution CronJobs (ucits/bybit/ibkr rebalancers) or arm any executor.
|
||||
- **No silent failures:** a Sensor that can't match, or a Workflow that errors, must surface loudly (a failed
|
||||
push = a visible failed Workflow, never a swallowed no-op). Every manifest is `kubectl apply --dry-run=server`
|
||||
validated before a real apply.
|
||||
- **Reproducibility:** the deploy step annotates deployed Deployments with `fxhnt.io/deployed-sha=<sha7>`.
|
||||
|
||||
## File Structure
|
||||
|
||||
- `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` — the reanimated WorkflowTemplate (ns foxhunt). Supersedes
|
||||
the old `infra/argo/cockpit-build-deploy.yaml` (which is renamed/removed in the final task).
|
||||
- `infra/argo/fxhnt-build-sensor.yaml` — the new Sensor (ns argo-events).
|
||||
- Gitea webhook on `gitadmin/fxhnt` — created via Gitea API/UI (not a repo file; documented in the plan).
|
||||
|
||||
---
|
||||
|
||||
## Task 1 — Reanimate + fix the WorkflowTemplate (build side)
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (start from `infra/argo/cockpit-build-deploy.yaml`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: a `WorkflowTemplate` named `fxhnt-cockpit` in ns `foxhunt` with params `commit-sha` (default HEAD)
|
||||
and `mode` (default `build`; values `build`|`deploy`). The Sensor (Task 3) submits Workflows referencing it.
|
||||
|
||||
- [ ] **Step 1: Copy the old template as the new file, fix the node pool**
|
||||
|
||||
Copy `infra/argo/cockpit-build-deploy.yaml` → `infra/argo/fxhnt-cockpit-workflowtemplate.yaml`. In BOTH the
|
||||
`kaniko-build` and `kubectl-deploy` templates, change every
|
||||
`nodeSelector: { k8s.scaleway.com/pool-name: ci-compile-cpu, ... }` to
|
||||
`nodeSelector: { k8s.scaleway.com/pool-name: large, topology.kubernetes.io/zone: fr-par-2 }`.
|
||||
Grep to confirm no `ci-compile-cpu` remains: `grep -n ci-compile-cpu infra/argo/fxhnt-cockpit-workflowtemplate.yaml` → no hits.
|
||||
|
||||
- [ ] **Step 2: Add the immutable `:<sha7>` tag to the Kaniko destination**
|
||||
|
||||
In the `kaniko-build` container args, alongside the existing
|
||||
`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`, ADD a second line:
|
||||
`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:{{=sprig.trunc(7, workflow.parameters.commit-sha)}}`
|
||||
(the `sprig.trunc` expression matches how `build-image` truncates the sha). Keep `--cache=true` and the
|
||||
`--cache-repo`.
|
||||
|
||||
- [ ] **Step 3: Server-side dry-run validate the template**
|
||||
|
||||
Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt`
|
||||
Expected: `workflowtemplate.argoproj.io/fxhnt-cockpit created (server dry run)` with no schema errors.
|
||||
(If it errors on an unknown field, the Argo CRD version differs — STOP and report; do not force-apply.)
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml
|
||||
git commit -m "ci(fxhnt): reanimate cockpit WorkflowTemplate (node pool large, sha7 tag)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2 — Fix the deploy step (verify the dashboard external route first)
|
||||
|
||||
**Files:**
|
||||
- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (the `kubectl-deploy` template's script)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the WorkflowTemplate from Task 1.
|
||||
- Produces: a deploy step that refreshes the daemons correctly for whichever external route is live.
|
||||
|
||||
- [ ] **Step 1: Determine the live dashboard external route (decides restart vs scale-0→1)**
|
||||
|
||||
Run these and read the result:
|
||||
```bash
|
||||
kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
|
||||
kubectl get ingress -n foxhunt 2>/dev/null
|
||||
kubectl -n foxhunt describe svc fxhnt-ingress 2>/dev/null | grep -iE 'endpoints|selector'
|
||||
```
|
||||
Decide:
|
||||
- If `dashboard.fxhnt.ai` is served THROUGH the tailscale sidecar (sidecar present AND no ingress route to the
|
||||
dashboard) → KEEP the old `scale 0→1` dance (the tailscale session race is real).
|
||||
- If it is served through `fxhnt-ingress` (an ingress/route points at the dashboard svc; the tailscale sidecar
|
||||
is vestigial) → REPLACE the scale-dance with a plain `kubectl -n foxhunt rollout restart deploy/fxhnt-dashboard`.
|
||||
|
||||
Record the decision in the plan's progress ledger with the evidence. If ambiguous, KEEP `scale 0→1` (the
|
||||
conservative choice that cannot cause the stale-proxy "down").
|
||||
|
||||
- [ ] **Step 2: Apply the deploy-step decision + add the deployed-sha annotation**
|
||||
|
||||
Edit the `kubectl-deploy` script. Regardless of the restart mechanism chosen, add BEFORE the rollout:
|
||||
```
|
||||
kubectl -n foxhunt annotate deploy/fxhnt-dashboard deploy/dagster \
|
||||
fxhnt.io/deployed-sha="$(cd /workspace/src && git rev-parse --short HEAD)" --overwrite
|
||||
```
|
||||
Keep the existing `kubectl apply -f` of the three manifests (fxhnt-dashboard.yaml, dagster.yaml,
|
||||
network-policies/fxhnt-cockpit.yaml) and the `rollout status` waits. Keep `rollout restart deploy/dagster`
|
||||
(dagster has no proxy race).
|
||||
|
||||
- [ ] **Step 3: Dry-run validate again**
|
||||
|
||||
Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt`
|
||||
Expected: server dry-run OK.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml
|
||||
git commit -m "ci(fxhnt): deploy step — <restart-choice> + deployed-sha annotation"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3 — The Sensor (simple, single-trigger) — mode decided IN the workflow
|
||||
|
||||
**Design note (verified 2026-07-21):** the Argo Events Sensor CRD here has an OPEN schema (no strict property
|
||||
validation — `kubectl explain sensor.spec.triggers.template.conditions` errors, and the only `conditions:` in
|
||||
live sensors are in `status`, not trigger-level). Rather than gamble on a fragile sensor-side path-negation
|
||||
filter to pick build-vs-deploy, we keep the SENSOR SIMPLE (one trigger, exactly like `build-msp-portal`) and
|
||||
move the build-vs-deploy decision INTO the workflow, where a robust `git diff --name-only` between the previous
|
||||
and pushed commit is trivial and testable. This is the more robust design, not a fallback.
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/argo/fxhnt-build-sensor.yaml` (ns argo-events)
|
||||
- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (add a `decide-mode` step that git-diffs)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the `gitea-webhook` eventsource (shared), the `fxhnt-cockpit` WorkflowTemplate (Task 1/2).
|
||||
- Produces: a `Sensor` `build-fxhnt` (argo-events) that submits ONE Workflow per master push; the workflow
|
||||
self-selects build vs deploy.
|
||||
|
||||
- [ ] **Step 1: Write the single-trigger Sensor (copy build-msp-portal structure exactly)**
|
||||
|
||||
`kubectl get sensor build-msp-portal -n argo-events -o yaml` and mirror it. ONE dependency `push-to-master`:
|
||||
- `body.ref` == `refs/heads/master`
|
||||
- `body.pusher.username` != `argocd-image-updater`
|
||||
(NO `body.commits.#.modified` filter — the mode decision moved to the workflow.)
|
||||
ONE trigger `submit`: submits a Workflow `generateName: build-fxhnt-`, `namespace: foxhunt`,
|
||||
`workflowTemplateRef: fxhnt-cockpit`, injecting `commit-sha`=`body.after`. Do NOT set `mode` (the workflow
|
||||
computes it). This is structurally identical to the proven msp sensors → lowest risk.
|
||||
|
||||
- [ ] **Step 2: Add a `decide-mode` step to the WorkflowTemplate**
|
||||
|
||||
In `fxhnt-cockpit-workflowtemplate.yaml`, change the DAG so the FIRST step (after git-clone) is `decide-mode`:
|
||||
a small `alpine/git` script that runs, in the cloned repo at `commit-sha`:
|
||||
```
|
||||
CHANGED=$(git diff --name-only "${SHA}~1" "${SHA}" 2>/dev/null || echo "ALL")
|
||||
if echo "$CHANGED" | grep -qE '^pyproject\.toml$|^infra/docker/fxhnt\.Dockerfile$'; then
|
||||
echo build; else echo deploy; fi
|
||||
```
|
||||
Output the result (`build`/`deploy`) as an Argo output parameter `mode`. Then gate the `kaniko-build` task on
|
||||
`{{tasks.decide-mode.outputs.parameters.mode}} == build`, and `kubectl-deploy` depends on
|
||||
`build.Succeeded || build.Skipped` (unchanged). Edge case: if `${SHA}~1` doesn't exist (first commit / shallow),
|
||||
default to `build` (safe — a rebuild is never wrong, only slower). The explicit `mode` PARAM stays on the
|
||||
template (default `auto`) so a human can still force `mode=build`/`mode=deploy` on a manual `argo submit`;
|
||||
`decide-mode` only runs when `mode=auto`.
|
||||
|
||||
- [ ] **Step 3: Dry-run validate both**
|
||||
|
||||
```bash
|
||||
kubectl apply -f infra/argo/fxhnt-build-sensor.yaml --dry-run=server -n argo-events
|
||||
kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt
|
||||
```
|
||||
Expected: both server-dry-run OK.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/argo/fxhnt-build-sensor.yaml infra/argo/fxhnt-cockpit-workflowtemplate.yaml
|
||||
git commit -m "ci(fxhnt): single-trigger sensor + in-workflow git-diff mode decision"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4 — Apply to the cluster + wire the Gitea webhook
|
||||
|
||||
**Files:** none new — this task applies the manifests and creates the webhook.
|
||||
|
||||
- [ ] **Step 1: Apply the WorkflowTemplate + Sensor for real**
|
||||
|
||||
```bash
|
||||
kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml -n foxhunt
|
||||
kubectl apply -f infra/argo/fxhnt-build-sensor.yaml -n argo-events
|
||||
kubectl get workflowtemplate fxhnt-cockpit -n foxhunt
|
||||
kubectl get sensor build-fxhnt -n argo-events
|
||||
```
|
||||
Expected: both present. The sensor pod (`kubectl get pods -n argo-events | grep build-fxhnt`) reaches Running.
|
||||
|
||||
- [ ] **Step 2: Create the Gitea webhook on `gitadmin/fxhnt`**
|
||||
|
||||
The `gitea-webhook` eventsource endpoint is the same target the `bizworx-ops/infrastructure` repo already POSTs
|
||||
to. Find it: `kubectl get eventsource gitea-webhook -n argo-events -o yaml` (the webhook `port`/`endpoint`) and
|
||||
how the existing infra-repo webhook is configured (inspect via Gitea admin UI or API on an existing repo).
|
||||
Create the equivalent webhook on `gitadmin/fxhnt`: push events, active, POST JSON to the eventsource URL. Record
|
||||
the exact URL used in the ledger. Do NOT paste any webhook secret into the transcript.
|
||||
|
||||
- [ ] **Step 3: Verify the webhook reaches the sensor (dry, safe probe)**
|
||||
|
||||
Trigger a harmless test push (or use Gitea's "Test Delivery" webhook button) and confirm the sensor received it:
|
||||
```bash
|
||||
kubectl logs -n argo-events -l sensor-name=build-fxhnt --tail=50 | grep -iE 'triggered|push-to-master|dispatch'
|
||||
```
|
||||
Expected: the sensor logs the received push. If Gitea's test-delivery uses a non-master ref, the sensor
|
||||
correctly does NOT fire — confirm it's reachable via the eventsource logs instead.
|
||||
|
||||
---
|
||||
|
||||
## Task 5 — End-to-end verification (both modes) + retire the old file
|
||||
|
||||
**Files:**
|
||||
- Remove: `infra/argo/cockpit-build-deploy.yaml` (superseded)
|
||||
|
||||
- [ ] **Step 1: Verify mode=deploy end-to-end (code-only push)**
|
||||
|
||||
Make a trivial code-only commit on `master` (e.g. touch a comment in `src/`), push, and watch:
|
||||
```bash
|
||||
kubectl get wf -n foxhunt --sort-by=.metadata.creationTimestamp | tail -3
|
||||
```
|
||||
Expected: a `build-fxhnt-*` Workflow runs with `mode=deploy`, SKIPS the kaniko-build step, deploys, and the
|
||||
dashboard stays UP (check `dashboard.fxhnt.ai` reachable post-deploy, and
|
||||
`kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{.metadata.annotations.fxhnt\.io/deployed-sha}'`
|
||||
shows the new sha). Total time ~1-2 min, no Kaniko build.
|
||||
|
||||
- [ ] **Step 2: Verify mode=build end-to-end (dep push)**
|
||||
|
||||
Make a no-op-but-real change to `pyproject.toml` (e.g. reorder a dependency, or bump a harmless comment), push,
|
||||
and watch: the Workflow runs `mode=build`, Kaniko produces `fxhnt-cockpit:<sha7>`+`:latest`
|
||||
(`kubectl run` a crane/skopeo check OR verify the workflow's kaniko step succeeded), then deploys. Confirm the
|
||||
`:<sha7>` tag exists in the registry.
|
||||
|
||||
- [ ] **Step 3: Confirm no false trigger + real-money safety**
|
||||
|
||||
- Push a change touching ONLY `docs/` → confirm a `mode=deploy` Workflow runs (harmless) OR, if docs are
|
||||
excluded, no Workflow — either is fine, but it must NOT trigger a build.
|
||||
- Confirm NO execution CronJob was modified/triggered by any pipeline run:
|
||||
`kubectl get cronjob -n foxhunt` (ucits/bybit/ibkr rebalancers unchanged, bybit/ibkr still Suspend=True).
|
||||
|
||||
- [ ] **Step 4: Retire the old file + commit**
|
||||
|
||||
```bash
|
||||
git rm infra/argo/cockpit-build-deploy.yaml
|
||||
git commit -m "ci(fxhnt): retire old cockpit-build-deploy (superseded by workflowtemplate + sensor)"
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Final verification summary**
|
||||
|
||||
Confirm all Global Constraints hold: sensor in argo-events only, workflow+pods in foxhunt, node pool `large`,
|
||||
no cross-ns RBAC, no fxhnt secrets in platform ns, real-money CronJobs untouched. Record in the ledger.
|
||||
85
docs/superpowers/specs/2026-07-20-eur-fund-ledger-design.md
Normal file
85
docs/superpowers/specs/2026-07-20-eur-fund-ledger-design.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# EUR Fund Ledger — Design Spec (2026-07-20)
|
||||
|
||||
**Status:** design approved (scope + cost-basis decided with operator). Implementation plan follows separately.
|
||||
|
||||
**> NOT tax advice.** This is an engineering design for a bookkeeping system. The tax *treatment* it records
|
||||
(Box 3 passive capital, FIFO cost basis, EUR-functional) reflects the operator's resolved structure and must be
|
||||
confirmed by a Dutch belastingadviseur before it is relied on for a filing. See
|
||||
`docs/superpowers/2026-07-20-fund-tax-accounting-research-brief.md` and memory
|
||||
`project_fxhnt_fund_tax_accounting_research`.
|
||||
|
||||
## Goal
|
||||
|
||||
A tax-ready, append-only, double-entry **EUR ledger** for the fxhnt fund's **passive Box-3 investment book**,
|
||||
living in the existing Postgres/Timescale warehouse, fed by an automated **IBKR Flex** ingestion Dagster asset.
|
||||
Phase 1 books **the fund only** — the passive IBKR investment book. The tech-BV bookkeeping and the arm's-length
|
||||
BV→fund service stay in the operator's existing BV administration (explicitly out of scope).
|
||||
|
||||
## Why this shape (context)
|
||||
|
||||
- The operator's structure separates **arbeid** (tech-BV, taxed there) from **kapitaal** (private fund, Box 3).
|
||||
Phase 1 books only the kapitaal side → a clean Box-3 record with no BV entanglement.
|
||||
- **EUR is the functional currency**, not just presentation. IBKR DU9600528 is EUR-native; if crypto ever goes
|
||||
live it is Bybit-EU (EUR, not USDT). There is **no USD leg**, so there is **no conversion pipeline** — only a
|
||||
dormant `currency` + `fx_rate_to_eur` seam (default EUR/1.0) so a stray non-EUR line is representable and
|
||||
correct rather than silently mis-booked. A pure-EUR book has **no FX P&L** — only trading P&L.
|
||||
- Reuses existing patterns: SQLAlchemy 2.0 ORM (`Base`/`Mapped`, Postgres+SQLite), a `*Repo` with `migrate()`,
|
||||
a Dagster asset alongside the nightly ETL. No new infrastructure.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
1. **EUR-functional.** Every posting stored in EUR. `currency` defaults `"EUR"`, `fx_rate_to_eur` defaults
|
||||
`1.0`. Build NO conversion pipeline — the FX columns are dormant insurance only.
|
||||
2. **Append-only.** `journal_entries` / `journal_postings` are immutable. Corrections are reversing entries
|
||||
(never UPDATE/DELETE of a posted row). Ingestion is idempotent (re-running a Flex pull must not double-book).
|
||||
3. **Balanced in EUR.** A DB-level trigger (or repo-enforced invariant on both Postgres and SQLite) rejects any
|
||||
entry whose postings do not sum to zero in `eur_amount`.
|
||||
4. **FIFO** cost basis, applied consistently. Realized gains come from lot-matching, not cash-flow deltas.
|
||||
5. **Scope = fund only.** No tech-BV, dividend, or arm's-length-service postings in Phase 1.
|
||||
6. **Runs on paper now.** The IBKR Flex asset must run against the *paper* account (DU9600528) — the schema and
|
||||
the EUR-sum-to-zero invariant are validated on paper data before any real capital exists.
|
||||
7. **Same test discipline as the codebase:** models work on SQLite (unit tests) and Postgres (prod); full suite
|
||||
green before finishing.
|
||||
|
||||
## Schema (SQLAlchemy 2.0 ORM, in a new `ledger` module)
|
||||
|
||||
- **`accounts`** — chart of accounts. `account_id` (str PK, e.g. `IBKR-EUR-cash`, `IBKR-securities`,
|
||||
`contributed-capital`, `trading-pnl`, `fees`), `kind` (asset|liability|equity|income|expense), `venue`,
|
||||
`currency` (default EUR). Per-venue sub-ledgers each balance standalone.
|
||||
- **`journal_entries`** — header. `entry_id` (str PK), `entry_date` (date), `description`, `source_venue`,
|
||||
`strategy_tag` (nullable — the Dagster attribution dimension), `source_ref` (the idempotency key: the IBKR
|
||||
Flex trade/cash id, so re-ingest is a no-op). Immutable.
|
||||
- **`journal_postings`** — lines. `posting_id` (PK), `entry_id` (FK), `account_id` (FK), `side` (debit|credit),
|
||||
`amount`, `currency` (default EUR), `fx_rate_to_eur` (default 1.0), `eur_amount`. Invariant: per `entry_id`,
|
||||
Σ `eur_amount`(debit) == Σ `eur_amount`(credit).
|
||||
- **`cost_basis_lots`** — `lot_id` (PK), `account_id`, `asset`, `open_date`, `qty_open`, `qty_remaining`,
|
||||
`unit_cost_eur`, and on close a link to the realizing entry. FIFO consumption.
|
||||
- **`fx_rates`** — `value_date` + `pair` → `rate`. **Dormant** — populated only if a non-EUR asset appears.
|
||||
Present so the seam exists; no ingestion job in Phase 1.
|
||||
|
||||
## IBKR Flex ingestion (new adapter surface + Dagster asset)
|
||||
|
||||
The existing `IbkrBroker` is a live `ib_insync` socket connection — NOT the Flex API. Flex is a separate
|
||||
**Flex Web Service**: a per-query token + query id, a 2-step `SendRequest` → `GetStatement` poll (~1 req/sec),
|
||||
returning an XML statement. Add a read-only `IbkrFlexClient` (new file) that fetches and parses:
|
||||
trades, cash transactions, dividends, corporate actions, positions, NAV.
|
||||
|
||||
- Land raw XML/rows in an append-only `raw` schema (mirrors the DB-anchor-SSOT pattern), then a transform
|
||||
normalizes into `journal_entries`/`journal_postings` (idempotent on `source_ref`).
|
||||
- One Dagster asset (`fund_ledger_ingest` or similar) alongside the nightly ETL; reuses monitoring patterns.
|
||||
- W-8BEN note: as a non-US person the account gets a 1042-S, never a Consolidated 1099 — irrelevant to the
|
||||
EUR ledger math, but archive the annual Activity Statement as an audit doc.
|
||||
|
||||
## Out of scope (Phase 1) — deliberately deferred
|
||||
|
||||
- Bybit ingestion (no live crypto book; Bybit-EU is EUR anyway → add later, same schema).
|
||||
- The tech-BV → fund arm's-length service postings, dividend, capital-contribution accounting (stays in BV admin).
|
||||
- The Box-3 *werkelijk-rendement* parallel tax-basis calc (unrealized + fee-disallow) — only needed if the
|
||||
tegenbewijsregeling is elected; add when the advisor confirms it's worth it.
|
||||
- Any FX conversion pipeline (there is no non-EUR leg).
|
||||
|
||||
## First build step
|
||||
|
||||
The EUR-functional ledger core (models + repo + the EUR-sum-to-zero invariant) **plus** the IBKR Flex ingestion
|
||||
asset — validated on the paper account — before anything else. This exercises every load-bearing decision:
|
||||
append-only immutability, the EUR-balanced invariant, the `strategy_tag` dimension, FIFO lots.
|
||||
134
docs/superpowers/specs/2026-07-21-fxhnt-cicd-pipeline-design.md
Normal file
134
docs/superpowers/specs/2026-07-21-fxhnt-cicd-pipeline-design.md
Normal file
@@ -0,0 +1,134 @@
|
||||
# fxhnt CI/CD Pipeline on bizworx — Design Spec (2026-07-21)
|
||||
|
||||
**Status:** design approved (approach + build-trigger decided with operator). Implementation plan follows.
|
||||
|
||||
## Goal
|
||||
|
||||
Restore automated build+deploy for fxhnt on the bizworx Kapsule so a push to `master` on
|
||||
`gitadmin/fxhnt` (Gitea) lands live without a manual `kubectl rollout restart` — the exact gap that left the
|
||||
cockpit on 2.5-day-old code (the CBU0/CSBGU0 fix). **Git-sync-first, image-on-demand:** code changes trigger a
|
||||
fast rollout (git-sync re-pulls src); only dependency/Dockerfile changes trigger a Kaniko image rebuild.
|
||||
|
||||
## Why this shape
|
||||
|
||||
The operator's insight: **code changes often, the image changes rarely.** git-sync already pulls fresh `src/`
|
||||
on every (re)start, so a code-only push needs only a rollout — not a 5-minute Kaniko build. A rebuild is needed
|
||||
only when `pyproject.toml` / `fxhnt.Dockerfile` change. This is exactly the `mode=build` vs `mode=deploy` split
|
||||
the retired `infra/argo/cockpit-build-deploy.yaml` already encoded — we reanimate and fix it, rather than adopt
|
||||
the generic `build-image` template (which lacks the split and fxhnt's deploy specifics).
|
||||
|
||||
## Governing principle: shared engine, separate projects
|
||||
|
||||
fxhnt and bizworx are **different projects**. fxhnt should *use* bizworx's heavy shared platform infra but not
|
||||
*pollute* the platform namespaces with fxhnt-specific objects or secrets:
|
||||
|
||||
- **SHARE (bizworx platform, use as-is):** the one Gitea (webhook source), the Argo Events eventbus +
|
||||
`gitea-webhook` eventsource, the Argo Workflows controller (the build *engine*), external-secrets/openbao,
|
||||
the `large` node-pool, the Scaleway registry.
|
||||
- **SEPARATE (fxhnt-owned, live in `foxhunt`):** the `WorkflowTemplate`, all build+deploy pods, fxhnt's
|
||||
secrets (`argo-git-ssh-key`, `scw-registry` — already in `foxhunt`), and fxhnt's RBAC.
|
||||
- **The ONE unavoidable object in a platform ns:** the Sensor. A Sensor must live in `argo-events` to hear the
|
||||
shared eventbus. Keep its footprint minimal — it only matches fxhnt pushes and *submits* a Workflow that runs
|
||||
in `foxhunt`. No fxhnt secrets, no fxhnt logic beyond the trigger, live in `argo-events`.
|
||||
|
||||
This is NOT the old "everything in foxhunt because the secrets were there" (path of least resistance) — it's the
|
||||
same placement for the *right* reason: respecting the project boundary. It also means **no cross-namespace RBAC**
|
||||
(the deploy runs in `foxhunt`, where the app + its permissions already are) and **no fxhnt secrets replicated
|
||||
into `argo`**.
|
||||
|
||||
## Verified current state (all components already exist on bizworx)
|
||||
|
||||
- **One Gitea** (ns `gitea`): `git.fxhnt.ai` and `gitea-http.gitea.svc` are the SAME instance; it holds
|
||||
`gitadmin/fxhnt` (code) alongside `bizworx-ops/infrastructure` (the msp GitOps monorepo). No second Gitea.
|
||||
- **Argo Events** (`gitea-webhook` eventsource, `/push`, repo-agnostic) + **Argo Workflows** (Kaniko) — already
|
||||
driving `build-msp-*` via path-filtered sensors. Reusable.
|
||||
- **Registry:** fxhnt's image is `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit` (Scaleway) — DIFFERENT from msp's
|
||||
`registry.bizworx.nl/bizworx-ops/`. Secret `scw-registry` (ns foxhunt) holds the push creds.
|
||||
- **Dockerfile EXISTS:** `infra/docker/fxhnt.Dockerfile` (python:3.11-slim, `fxhnt serve` :8080). Reusable as-is.
|
||||
- **SAs exist:** `argo-workflow` (ns argo), `argo-events-sa` (ns argo-events); `argo-git-ssh-key` (ns foxhunt).
|
||||
- **What's broken in the old pipeline:** (1) `nodeSelector: ci-compile-cpu` — that pool is DECOMMISSIONED
|
||||
(only `large` + `sfs` pools exist now) → build/deploy pods won't schedule; (2) the Sensor that triggered it
|
||||
is gone; (3) no webhook on `gitadmin/fxhnt` → the eventsource never hears fxhnt pushes.
|
||||
- **No fxhnt ArgoCD app** — fxhnt manifests are applied by the workflow's `kubectl apply`, not GitOps-synced.
|
||||
We keep it that way (see Non-goals).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
git push → gitadmin/fxhnt (master) [SHARED: Gitea]
|
||||
│ Gitea webhook ──────────────► argo-events `gitea-webhook` eventsource [SHARED: platform eventbus]
|
||||
▼
|
||||
Sensor `build-fxhnt` (ns argo-events — the ONE fxhnt object in a platform ns, kept minimal)
|
||||
│ filters: ref==refs/heads/master, pusher != argocd-image-updater (SINGLE trigger, like the msp sensors)
|
||||
▼ submits ONE Workflow into ns `foxhunt`, injecting commit-sha ─────── [FXHNT-OWNED from here down]
|
||||
Argo Workflow (WorkflowTemplate `fxhnt-cockpit`, ns foxhunt, SA argo-workflow, node-pool `large`) — DAG:
|
||||
├─ decide-mode: git-clone(sha) → `git diff --name-only sha~1 sha` →
|
||||
│ touched pyproject.toml OR infra/docker/fxhnt.Dockerfile → mode=build ; else mode=deploy
|
||||
│ (mode decided IN the workflow, not a fragile sensor filter — the CRD schema is open, so a
|
||||
│ sensor-side path-negation can't be validated; a git-diff step is robust + testable. No lock file
|
||||
│ exists → those two paths are the complete rebuild-trigger set.)
|
||||
├─ mode=build: Kaniko → rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:<sha7>+:latest
|
||||
└─ (build skipped when mode=deploy)
|
||||
→ deploy step (in foxhunt, no cross-ns RBAC): kubectl apply manifests + refresh daemons (git-sync re-pulls src)
|
||||
```
|
||||
|
||||
## Components to build/fix
|
||||
|
||||
### 1. WorkflowTemplate `fxhnt-cockpit` (reanimate `infra/argo/cockpit-build-deploy.yaml`, ns `foxhunt`)
|
||||
- Fix `nodeSelector` `ci-compile-cpu` → `large` (both build + deploy templates — `ci-compile-cpu` pool is gone;
|
||||
only `large`/`sfs` exist).
|
||||
- Keep the `mode` DAG: `build` (Kaniko rebuild + deploy) vs `deploy` (skip build, deploy only).
|
||||
- Keep the Kaniko dest `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit` but ADD an immutable `:<sha7>` tag next to
|
||||
`:latest` (matches the msp pattern). Cache stays on.
|
||||
- Clone via the in-cluster Gitea using `argo-git-ssh-key`; push via `scw-registry`. Both already exist in
|
||||
`foxhunt` — the workflow runs there, so they resolve locally (no replication into a platform ns).
|
||||
- **Reproducibility despite git-sync:** the deploy step stamps the deployed commit sha onto the daemon
|
||||
Deployments as an annotation (`fxhnt.io/deployed-sha=<sha7>`), so "which commit is this pod running" is always
|
||||
answerable — closing the one real gap of git-sync-first without adopting full image-based deploys.
|
||||
|
||||
### 2. Sensor `build-fxhnt` (new, model on `build-msp-portal`)
|
||||
- Dependency: `gitea-webhook` push, filter `body.ref == refs/heads/master` AND
|
||||
`body.pusher.username != argocd-image-updater` (loop-guard; harmless here since no image-updater writes fxhnt,
|
||||
but kept for parity/safety).
|
||||
- **Mode decision via path-filter on `body.commits.#.modified`:** if any modified path matches
|
||||
`pyproject.toml` | `infra/docker/fxhnt.Dockerfile` → submit workflow with `mode=build`; else `mode=deploy`.
|
||||
(No lock file exists, so those two paths are the complete rebuild-trigger set.) Mechanism: two triggers with
|
||||
complementary data-filters (one matching the dep-paths → build, one matching their negation → deploy), or one
|
||||
trigger whose param is computed — the plan picks whichever Argo Events supports cleanly.
|
||||
- Inject `commit-sha` from `body.after`.
|
||||
|
||||
### 3. Gitea webhook on `gitadmin/fxhnt`
|
||||
- Add a repo (or org) webhook POSTing push events to the `gitea-webhook` eventsource endpoint (same target the
|
||||
infrastructure repo already uses). This is the one genuinely-missing wire.
|
||||
|
||||
### 4. The deploy step — VERIFY, don't assume (open design point)
|
||||
The old deploy used **`scale 0→1` for fxhnt-dashboard, NOT `rollout restart`**, because the tailscale sidecar
|
||||
served `dashboard.fxhnt.ai` and a rolling restart overlapped two tailscale sessions → external proxy read
|
||||
"down". BUT: the dashboard now also has a `fxhnt-ingress`, and a manual `rollout restart` on 2026-07-21
|
||||
succeeded cleanly. **The plan must verify which external route is live** before choosing:
|
||||
- If `dashboard.fxhnt.ai` still resolves through the tailscale sidecar → keep `scale 0→1` (the race is real).
|
||||
- If it now resolves through `fxhnt-ingress` (tailscale sidecar vestigial) → a plain `rollout restart` is
|
||||
simpler and correct; drop the scale-dance.
|
||||
Dagster has no external-proxy race → `rollout restart deploy/dagster` either way.
|
||||
|
||||
## Non-goals (YAGNI) — deliberately NOT doing, with the reason
|
||||
|
||||
- **No ArgoCD GitOps app for fxhnt.** Chosen over declarative GitOps: it would add a second source of truth for
|
||||
the pod's src (ArgoCD-synced manifests vs git-sync'd code) and needs a git.fxhnt.ai repo registration. The
|
||||
imperative `kubectl apply` from the foxhunt workflow fits git-sync-first and keeps manifests in the fxhnt repo.
|
||||
- **No registry unification.** fxhnt stays on Scaleway `rg.fr-par.scw.cloud/bizworx` (not the shared
|
||||
`registry.bizworx.nl/bizworx-ops`). Migrating would mean re-pointing the dashboard + reconciling old images
|
||||
for zero pipeline benefit — a separate, riskier task.
|
||||
- **No CI workloads in a platform namespace.** Only the Sensor lives in `argo-events` (unavoidable). The
|
||||
WorkflowTemplate + all build/deploy pods run in `foxhunt` — no fxhnt secrets in `argo`, no cross-ns RBAC.
|
||||
- **No image-updater for fxhnt.** The sensor writes the `:sha7`+`:latest` tags; the deploy step points the
|
||||
daemons at the fresh image. No git-write-back loop needed.
|
||||
- **No change to the execution CronJobs** (ucits/bybit/ibkr rebalancers) — they git-sync per run already.
|
||||
|
||||
## Success criteria
|
||||
|
||||
1. A code-only push to `master` → cockpit reflects it within ~1–2 min WITHOUT a manual restart, NO Kaniko build.
|
||||
2. A `pyproject.toml`/Dockerfile push → a Kaniko rebuild produces `fxhnt-cockpit:<sha7>`+`:latest`, then deploys.
|
||||
3. The dashboard external route (`dashboard.fxhnt.ai`) stays UP across a deploy (no stale-proxy "down").
|
||||
4. No build-loop, no false trigger on unrelated pushes.
|
||||
5. Real-money safety unaffected — the pipeline only builds/deploys the cockpit+dagster daemons, never arms exec.
|
||||
@@ -1,135 +0,0 @@
|
||||
# Argo WorkflowTemplate — build + deploy the fxhnt cockpit, fully in-cluster.
|
||||
#
|
||||
# Reuses cluster primitives: the `argo-workflow` ServiceAccount, the `argo-git-ssh-key` deploy key (clone
|
||||
# the fxhnt repo over in-cluster Gitea SSH, gitea-sshd:22), and the `scw-registry` dockerconfig (push to
|
||||
# the external Scaleway Container Registry rg.fr-par.scw.cloud/bizworx). Build runs in-cluster.
|
||||
# (Phase 2B: migrated off the in-cluster GitLab shell + registry.)
|
||||
#
|
||||
# Submit: argo submit -n foxhunt --from=wftmpl/fxhnt-cockpit -p commit-sha=<sha> --watch
|
||||
# (or use scripts/argo-deploy-cockpit.sh)
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: WorkflowTemplate
|
||||
metadata:
|
||||
name: fxhnt-cockpit
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/name: fxhnt-cockpit, app.kubernetes.io/part-of: foxhunt }
|
||||
spec:
|
||||
serviceAccountName: argo-workflow
|
||||
entrypoint: pipeline
|
||||
archiveLogs: true # archive build/deploy logs to MinIO (argo-logs bucket). Reachable now that the
|
||||
# pods carry app.kubernetes.io/part-of: argo-workflows (MinIO ingress netpol allow).
|
||||
activeDeadlineSeconds: 1800
|
||||
ttlStrategy: { secondsAfterCompletion: 3600 }
|
||||
podGC: { strategy: OnPodCompletion }
|
||||
arguments:
|
||||
parameters:
|
||||
- { name: commit-sha, value: HEAD }
|
||||
- { name: mode, value: build } # "build" = kaniko rebuild + deploy; "deploy" = skip build, just apply + rollout restart (git-sync re-pulls code). The sensor routes src-only pushes to "deploy".
|
||||
templates:
|
||||
- name: pipeline
|
||||
dag:
|
||||
tasks:
|
||||
# Skip the ~5min kaniko build for code-only pushes (mode=deploy) — git-sync re-pulls src/ on the
|
||||
# rollout restart, so only dep/Dockerfile changes need a new image (mode=build).
|
||||
- { name: build, template: kaniko-build, when: "{{workflow.parameters.mode}} == build" }
|
||||
- { name: deploy, template: kubectl-deploy, depends: "build.Succeeded || build.Skipped" }
|
||||
|
||||
# --- build: Kaniko builds infra/docker/fxhnt.Dockerfile -> internal registry ---
|
||||
- name: kaniko-build
|
||||
metadata:
|
||||
labels: { app.kubernetes.io/part-of: argo-workflows } # MinIO ingress allows part-of in {foxhunt,foxhunt-ci,argo-workflows} → log archival works. NOT 'foxhunt' (that adds argo-base-egress which blocks registry:5000/git:2222).
|
||||
nodeSelector: { k8s.scaleway.com/pool-name: ci-compile-cpu, topology.kubernetes.io/zone: fr-par-2 }
|
||||
tolerations: [{ effect: NoSchedule, key: node.cilium.io/agent-not-ready, operator: Exists }]
|
||||
initContainers:
|
||||
- name: git-clone
|
||||
image: alpine/git:latest
|
||||
command: [/bin/sh, -c]
|
||||
args:
|
||||
- |
|
||||
set -ex
|
||||
mkdir -p /root/.ssh
|
||||
cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
|
||||
chmod 600 /root/.ssh/id_ed25519
|
||||
printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > /root/.ssh/config
|
||||
git clone --no-checkout --filter=blob:none \
|
||||
"ssh://git@gitea-sshd.foxhunt.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
|
||||
cd /workspace/src && git checkout "{{workflow.parameters.commit-sha}}"
|
||||
echo "checked out $(git rev-parse --short HEAD)"
|
||||
volumeMounts:
|
||||
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
container:
|
||||
image: gcr.io/kaniko-project/executor:debug
|
||||
command: [/busybox/sh, -c]
|
||||
args:
|
||||
- |
|
||||
/kaniko/executor \
|
||||
--context=/workspace/src \
|
||||
--dockerfile=/workspace/src/infra/docker/fxhnt.Dockerfile \
|
||||
--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest \
|
||||
--cache=true \
|
||||
--cache-repo=rg.fr-par.scw.cloud/bizworx/cache \
|
||||
--snapshot-mode=redo
|
||||
env:
|
||||
- { name: DOCKER_CONFIG, value: /kaniko/.docker }
|
||||
resources: { requests: { cpu: "2", memory: 4Gi }, limits: { cpu: "6", memory: 12Gi } }
|
||||
volumeMounts:
|
||||
- { name: registry-auth, mountPath: /kaniko/.docker, readOnly: true }
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
volumes:
|
||||
- { name: workspace, emptyDir: {} }
|
||||
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
|
||||
- name: registry-auth
|
||||
secret:
|
||||
secretName: scw-registry
|
||||
items: [{ key: .dockerconfigjson, path: config.json }]
|
||||
|
||||
# --- deploy: apply manifests from the repo + roll the dashboard to the fresh image ---
|
||||
- name: kubectl-deploy
|
||||
metadata:
|
||||
labels: { app.kubernetes.io/part-of: argo-workflows } # MinIO ingress allow → log archival works (see kaniko-build note)
|
||||
nodeSelector: { k8s.scaleway.com/pool-name: platform, topology.kubernetes.io/zone: fr-par-2 }
|
||||
serviceAccountName: argo-workflow
|
||||
initContainers:
|
||||
- name: git-clone
|
||||
image: alpine/git:latest
|
||||
command: [/bin/sh, -c]
|
||||
args:
|
||||
- |
|
||||
set -ex
|
||||
mkdir -p /root/.ssh
|
||||
cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
|
||||
chmod 600 /root/.ssh/id_ed25519
|
||||
printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > /root/.ssh/config
|
||||
git clone --no-checkout --filter=blob:none \
|
||||
"ssh://git@gitea-sshd.foxhunt.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
|
||||
cd /workspace/src && git checkout "{{workflow.parameters.commit-sha}}"
|
||||
volumeMounts:
|
||||
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
container:
|
||||
image: alpine/k8s:1.31.0 # Alpine-based (has /bin/sh + kubectl). NOT registry.k8s.io/kubectl — that's distroless (no shell) and the deploy runs a shell script. bitnami/kubectl was removed from Docker Hub (2025).
|
||||
command: [/bin/sh, -c]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
cd /workspace/src
|
||||
kubectl apply -f infra/k8s/services/fxhnt-dashboard.yaml
|
||||
kubectl apply -f infra/k8s/orchestration/dagster.yaml
|
||||
kubectl apply -f infra/k8s/network-policies/fxhnt-cockpit.yaml
|
||||
# Dashboard: scale 0 -> 1, NOT `rollout restart`. Its tailscale sidecar serves dashboard.fxhnt.ai;
|
||||
# a rolling restart briefly overlaps two tailscale sessions on the same identity -> session race ->
|
||||
# the external proxy link goes stale and the cockpit reads "down" post-deploy. Scaling to 0 cleanly
|
||||
# terminates the session before a fresh, non-overlapping one comes up; the git-sync initContainer
|
||||
# re-pulls src the same way, so code still updates. See reference_fxhnt_dashboard_proxy.
|
||||
kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=0
|
||||
kubectl -n foxhunt wait --for=delete pod -l app.kubernetes.io/name=fxhnt-dashboard --timeout=90s || true
|
||||
kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=1
|
||||
kubectl -n foxhunt rollout restart deploy/dagster # dagster has no external-proxy race
|
||||
kubectl -n foxhunt rollout status deploy/fxhnt-dashboard --timeout=180s
|
||||
kubectl -n foxhunt rollout status deploy/dagster --timeout=300s
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
volumes:
|
||||
- { name: workspace, emptyDir: {} }
|
||||
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
|
||||
57
infra/argo/fxhnt-build-sensor.yaml
Normal file
57
infra/argo/fxhnt-build-sensor.yaml
Normal file
@@ -0,0 +1,57 @@
|
||||
# Argo Events Sensor — hears the shared Gitea webhook for pushes to gitadmin/fxhnt master, and submits the
|
||||
# fxhnt-cockpit WorkflowTemplate (ns foxhunt). Structurally identical to build-msp-portal (single dependency,
|
||||
# single trigger) — the build-vs-deploy decision is made INSIDE the workflow via `git diff` (decide-mode step in
|
||||
# fxhnt-cockpit-workflowtemplate.yaml), not via a sensor-side path filter. See
|
||||
# docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md Task 3 design note: the Sensor CRD here has an open
|
||||
# schema, so trigger-level path-filter negation can't be reliably validated — keeping the sensor simple is the
|
||||
# robust choice.
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Sensor
|
||||
metadata:
|
||||
name: build-fxhnt
|
||||
namespace: argo-events
|
||||
labels: { app.kubernetes.io/instance: build-pipeline }
|
||||
spec:
|
||||
dependencies:
|
||||
- name: push-to-master
|
||||
eventSourceName: gitea-webhook
|
||||
eventName: push
|
||||
filters:
|
||||
dataLogicalOperator: and
|
||||
data:
|
||||
- path: body.ref
|
||||
type: string
|
||||
comparator: "="
|
||||
value: ["refs/heads/master"]
|
||||
- path: body.pusher.username
|
||||
type: string
|
||||
comparator: "!="
|
||||
value: ["argocd-image-updater"]
|
||||
template:
|
||||
serviceAccountName: argo-events-sa
|
||||
triggers:
|
||||
- template:
|
||||
name: submit-build
|
||||
argoWorkflow:
|
||||
operation: submit
|
||||
source:
|
||||
resource:
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Workflow
|
||||
metadata:
|
||||
generateName: build-fxhnt-
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
workflowTemplateRef:
|
||||
name: fxhnt-cockpit
|
||||
arguments:
|
||||
parameters:
|
||||
- name: commit-sha
|
||||
value: ""
|
||||
- name: mode
|
||||
value: auto
|
||||
parameters:
|
||||
- src:
|
||||
dependencyName: push-to-master
|
||||
dataKey: body.after
|
||||
dest: spec.arguments.parameters.0.value
|
||||
124
infra/argo/fxhnt-ci-rbac.yaml
Normal file
124
infra/argo/fxhnt-ci-rbac.yaml
Normal file
@@ -0,0 +1,124 @@
|
||||
# RBAC for the fxhnt CI pipeline — all fxhnt-owned, in the foxhunt namespace (project boundary).
|
||||
#
|
||||
# Fills the two gaps the first real workflow run surfaced (2026-07-21):
|
||||
# 1. The fxhnt-cockpit WorkflowTemplate references SA `argo-workflow`, which existed in `argo` (where msp
|
||||
# builds) but NOT in `foxhunt` (where fxhnt's project boundary places its build/deploy pods).
|
||||
# 2. The build-fxhnt Sensor (argo-events-sa) submits Workflows into `foxhunt`; the bizworx `workflow-submitter`
|
||||
# Role only granted that in `argo`, so a cross-namespace submit into foxhunt was forbidden.
|
||||
#
|
||||
# Least privilege: the workflow SA gets exactly what Argo needs to run pods + what the kubectl-deploy step does
|
||||
# (apply/scale/annotate/rollout-status the two daemon Deployments and their Services/NetworkPolicies), nothing more.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: argo-workflow
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/part-of: fxhnt-ci }
|
||||
---
|
||||
# Argo needs the workflow SA to write task results (the minimum the workflow-controller requires per step).
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: fxhnt-argo-workflow
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/part-of: fxhnt-ci }
|
||||
rules:
|
||||
- apiGroups: ["argoproj.io"]
|
||||
resources: ["workflowtaskresults"]
|
||||
verbs: ["create", "patch"]
|
||||
---
|
||||
# The kubectl-deploy step's exact actions on the foxhunt daemons — scoped to what it runs, no wildcards.
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: fxhnt-ci-deployer
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/part-of: fxhnt-ci }
|
||||
rules:
|
||||
# apply + scale + annotate + rollout-status the daemon Deployments (dashboard, dagster).
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["deployments"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "patch"]
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["deployments/scale"]
|
||||
verbs: ["get", "update", "patch"]
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["replicasets"]
|
||||
verbs: ["get", "list", "watch"] # rollout status reads the RS
|
||||
# apply the Services the manifests contain, and `wait --for=delete pod` + rollout status read pods.
|
||||
- apiGroups: [""]
|
||||
resources: ["services"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "patch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
# the fxhnt-dashboard + dagster manifests also bundle ConfigMaps (the tailscale sidecar serve-config) —
|
||||
# kubectl apply of the whole file must be able to read/apply them. NOTE (CI review M1): the tailscale
|
||||
# ServiceAccount + Role + RoleBinding objects were MOVED out of these app manifests into
|
||||
# infra/k8s/rbac/tailscale-sidecars.yaml (applied once, out-of-band, by a human) specifically so this
|
||||
# deployer SA does not need roles/rolebindings/serviceaccounts create-update-patch — that combination on a
|
||||
# webhook-triggered SA is a privilege-escalation primitive. Neither manifest applied by CI creates a
|
||||
# ServiceAccount anymore, so `serviceaccounts` is dropped from this grant too.
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "patch"]
|
||||
# the manifests apply NetworkPolicies (fxhnt-cockpit.yaml + dagster's).
|
||||
- apiGroups: ["networking.k8s.io"]
|
||||
resources: ["networkpolicies"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "patch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: fxhnt-argo-workflow
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/part-of: fxhnt-ci }
|
||||
subjects:
|
||||
- { kind: ServiceAccount, name: argo-workflow, namespace: foxhunt }
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: fxhnt-argo-workflow
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: fxhnt-ci-deployer
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/part-of: fxhnt-ci }
|
||||
subjects:
|
||||
- { kind: ServiceAccount, name: argo-workflow, namespace: foxhunt }
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: fxhnt-ci-deployer
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
# Cross-namespace: let the shared argo-events-sa submit Workflows into foxhunt (the sensor lives in argo-events
|
||||
# but its Workflow runs here). Mirrors the bizworx `workflow-submitter` Role, scoped to foxhunt.
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: fxhnt-workflow-submitter
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/part-of: fxhnt-ci }
|
||||
rules:
|
||||
- apiGroups: ["argoproj.io"]
|
||||
resources: ["workflows"]
|
||||
verbs: ["create", "get", "list", "watch"]
|
||||
- apiGroups: ["argoproj.io"]
|
||||
resources: ["workflowtemplates"]
|
||||
verbs: ["get", "list"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: argo-events-sa-fxhnt-submitter
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/part-of: fxhnt-ci }
|
||||
subjects:
|
||||
- { kind: ServiceAccount, name: argo-events-sa, namespace: argo-events }
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: fxhnt-workflow-submitter
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
208
infra/argo/fxhnt-cockpit-workflowtemplate.yaml
Normal file
208
infra/argo/fxhnt-cockpit-workflowtemplate.yaml
Normal file
@@ -0,0 +1,208 @@
|
||||
# Argo WorkflowTemplate — build + deploy the fxhnt cockpit, fully in-cluster.
|
||||
#
|
||||
# Reuses cluster primitives: the `argo-workflow` ServiceAccount (foxhunt, see fxhnt-ci-rbac.yaml), the
|
||||
# `argo-git-ssh-key` deploy key (clone the fxhnt repo over in-cluster Gitea SSH, gitea-ssh.gitea.svc:22), and
|
||||
# the `scw-registry` dockerconfig (push to the external Scaleway Container Registry rg.fr-par.scw.cloud/bizworx).
|
||||
#
|
||||
# Submit: argo submit -n foxhunt --from=wftmpl/fxhnt-cockpit -p commit-sha=<sha> -p mode=<auto|build|deploy> --watch
|
||||
# Normally auto-triggered: a push to gitadmin/fxhnt master -> the build-fxhnt Sensor (argo-events) submits this.
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: WorkflowTemplate
|
||||
metadata:
|
||||
name: fxhnt-cockpit
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/name: fxhnt-cockpit, app.kubernetes.io/part-of: foxhunt }
|
||||
spec:
|
||||
serviceAccountName: argo-workflow
|
||||
entrypoint: pipeline
|
||||
archiveLogs: false # archive build/deploy logs to MinIO (argo-logs bucket). Reachable now that the
|
||||
# pods carry app.kubernetes.io/part-of: argo-workflows (MinIO ingress netpol allow).
|
||||
activeDeadlineSeconds: 1800
|
||||
ttlStrategy: { secondsAfterCompletion: 3600 }
|
||||
podGC: { strategy: OnPodCompletion }
|
||||
arguments:
|
||||
parameters:
|
||||
- { name: commit-sha, value: HEAD }
|
||||
- { name: mode, value: auto } # "auto" = decide-mode git-diffs commit-sha~1..commit-sha to pick build/deploy; "build" = force kaniko rebuild + deploy; "deploy" = force skip build, just apply + rollout restart (git-sync re-pulls code). A human can still force build/deploy via `argo submit -p mode=...`.
|
||||
templates:
|
||||
- name: pipeline
|
||||
dag:
|
||||
tasks:
|
||||
# decide-mode: when mode=auto, git-diffs the pushed commit against its parent to pick build vs deploy
|
||||
# (dep/Dockerfile changes -> build; anything else, or a missing parent, -> build/deploy per script logic
|
||||
# below). When mode is forced (build|deploy), decide-mode just echoes it back unchanged.
|
||||
- { name: decide-mode, template: decide-mode }
|
||||
# Skip the ~5min kaniko build for code-only pushes (mode=deploy) — git-sync re-pulls src/ on the
|
||||
# rollout restart, so only dep/Dockerfile changes need a new image (mode=build).
|
||||
- { name: build, template: kaniko-build, depends: decide-mode, when: "{{tasks.decide-mode.outputs.parameters.mode}} == build" }
|
||||
- { name: deploy, template: kubectl-deploy, depends: "decide-mode && (build.Succeeded || build.Skipped)" }
|
||||
|
||||
# --- decide-mode: resolve "auto" to build|deploy via git diff of the pushed commit vs its parent ---
|
||||
- name: decide-mode
|
||||
metadata:
|
||||
labels: { app.kubernetes.io/part-of: argo-workflows }
|
||||
nodeSelector: { k8s.scaleway.com/pool-name: large }
|
||||
tolerations: [{ effect: NoSchedule, key: node.cilium.io/agent-not-ready, operator: Exists }]
|
||||
outputs:
|
||||
parameters:
|
||||
- name: mode
|
||||
valueFrom: { path: /workspace/mode }
|
||||
container:
|
||||
image: alpine/git:latest
|
||||
command: [/bin/sh, -c]
|
||||
# SECURITY: commit-sha/mode arrive via ENV (not inline {{...}} substitution) so an untrusted webhook
|
||||
# payload cannot inject shell. commit-sha is validated hex before use; mode is checked against an allowlist.
|
||||
env:
|
||||
- { name: COMMIT_SHA, value: "{{workflow.parameters.commit-sha}}" }
|
||||
- { name: MODE, value: "{{workflow.parameters.mode}}" }
|
||||
# Absolute key path via GIT_SSH_COMMAND — Argo's emissary executor does not set HOME=/root, so the
|
||||
# old `cp key -> ~/.ssh` idiom left git unable to find the key (clone failed with exit 128). This is
|
||||
# HOME-independent and verified to clone under HOME=/nonexistent.
|
||||
- { name: GIT_SSH_COMMAND, value: "ssh -i /etc/git-ssh/ssh-privatekey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" }
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
case "$COMMIT_SHA" in *[!0-9a-fA-F]*|"") echo "refusing non-hex commit-sha" >&2; exit 1 ;; esac
|
||||
case "$MODE" in auto|build|deploy) : ;; *) echo "refusing unknown mode: $MODE" >&2; exit 1 ;; esac
|
||||
git clone --filter=blob:none \
|
||||
"ssh://git@gitea-ssh.gitea.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
|
||||
cd /workspace/src && git checkout "$COMMIT_SHA"
|
||||
|
||||
if [ "$MODE" != "auto" ]; then
|
||||
printf '%s' "$MODE" > /workspace/mode
|
||||
else
|
||||
RESOLVED="$(git rev-parse "$COMMIT_SHA")"
|
||||
if CHANGED=$(git diff --name-only "${RESOLVED}~1" "${RESOLVED}" 2>/dev/null); then
|
||||
# diff succeeded (parent exists): build only if a dep/Dockerfile file changed, else deploy.
|
||||
if echo "$CHANGED" | grep -qE '^pyproject\.toml$|^infra/docker/fxhnt\.Dockerfile$'; then
|
||||
printf 'build' > /workspace/mode
|
||||
else
|
||||
printf 'deploy' > /workspace/mode
|
||||
fi
|
||||
else
|
||||
# diff failed (no parent -- first commit / shallow history): default to build (safe, never wrong).
|
||||
printf 'build' > /workspace/mode
|
||||
fi
|
||||
fi
|
||||
cat /workspace/mode
|
||||
volumeMounts:
|
||||
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
volumes:
|
||||
- { name: workspace, emptyDir: {} }
|
||||
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
|
||||
|
||||
# --- build: Kaniko builds infra/docker/fxhnt.Dockerfile -> internal registry ---
|
||||
- name: kaniko-build
|
||||
metadata:
|
||||
labels: { app.kubernetes.io/part-of: argo-workflows } # MinIO ingress allows part-of in {foxhunt,foxhunt-ci,argo-workflows} → log archival works. NOT 'foxhunt' (that adds argo-base-egress which blocks registry:5000/git:2222).
|
||||
nodeSelector: { k8s.scaleway.com/pool-name: large }
|
||||
tolerations: [{ effect: NoSchedule, key: node.cilium.io/agent-not-ready, operator: Exists }]
|
||||
initContainers:
|
||||
- name: git-clone
|
||||
image: alpine/git:latest
|
||||
command: [/bin/sh, -c]
|
||||
# SECURITY: commit-sha via ENV + hex-validated (see decide-mode note) — no inline shell substitution.
|
||||
env:
|
||||
- { name: COMMIT_SHA, value: "{{workflow.parameters.commit-sha}}" }
|
||||
# Absolute key path via GIT_SSH_COMMAND — HOME-independent (Argo's emissary doesn't set HOME=/root).
|
||||
- { name: GIT_SSH_COMMAND, value: "ssh -i /etc/git-ssh/ssh-privatekey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" }
|
||||
args:
|
||||
- |
|
||||
set -ex
|
||||
case "$COMMIT_SHA" in *[!0-9a-fA-F]*|"") echo "refusing non-hex commit-sha" >&2; exit 1 ;; esac
|
||||
git clone --no-checkout --filter=blob:none \
|
||||
"ssh://git@gitea-ssh.gitea.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
|
||||
cd /workspace/src && git checkout "$COMMIT_SHA"
|
||||
echo "checked out $(git rev-parse --short HEAD)"
|
||||
volumeMounts:
|
||||
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
container:
|
||||
image: gcr.io/kaniko-project/executor:v1.23.2
|
||||
# Kaniko's own entrypoint + a YAML args LIST (the msp build-image pattern) — NOT `sh -c` with one big
|
||||
# string. A shell wrapper made busybox try to parse the `{{=sprig.trunc(...)}}` expansion and choke on
|
||||
# the `(` (syntax error). As a plain args list, Argo evaluates the expression and hands it straight to
|
||||
# Kaniko, no shell in between.
|
||||
args:
|
||||
- --context=/workspace/src
|
||||
- --dockerfile=/workspace/src/infra/docker/fxhnt.Dockerfile
|
||||
- --destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
|
||||
- --destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:{{workflow.parameters.commit-sha}}
|
||||
- --cache=true
|
||||
- --cache-repo=rg.fr-par.scw.cloud/bizworx/cache
|
||||
- --snapshot-mode=redo
|
||||
env:
|
||||
- { name: DOCKER_CONFIG, value: /kaniko/.docker }
|
||||
resources: { requests: { cpu: "1", memory: 3Gi }, limits: { cpu: "3", memory: 10Gi } }
|
||||
volumeMounts:
|
||||
- { name: registry-auth, mountPath: /kaniko/.docker, readOnly: true }
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
volumes:
|
||||
- { name: workspace, emptyDir: {} }
|
||||
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
|
||||
- name: registry-auth
|
||||
secret:
|
||||
secretName: scw-registry
|
||||
items: [{ key: .dockerconfigjson, path: config.json }]
|
||||
|
||||
# --- deploy: apply manifests from the repo + roll the dashboard to the fresh image ---
|
||||
- name: kubectl-deploy
|
||||
metadata:
|
||||
labels: { app.kubernetes.io/part-of: argo-workflows } # MinIO ingress allow → log archival works (see kaniko-build note)
|
||||
nodeSelector: { k8s.scaleway.com/pool-name: large }
|
||||
serviceAccountName: argo-workflow
|
||||
initContainers:
|
||||
- name: git-clone
|
||||
image: alpine/git:latest
|
||||
command: [/bin/sh, -c]
|
||||
# SECURITY: commit-sha via ENV + hex-validated — no inline shell substitution.
|
||||
env:
|
||||
- { name: COMMIT_SHA, value: "{{workflow.parameters.commit-sha}}" }
|
||||
# Absolute key path via GIT_SSH_COMMAND — HOME-independent (Argo's emissary doesn't set HOME=/root).
|
||||
- { name: GIT_SSH_COMMAND, value: "ssh -i /etc/git-ssh/ssh-privatekey -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" }
|
||||
args:
|
||||
- |
|
||||
set -ex
|
||||
case "$COMMIT_SHA" in *[!0-9a-fA-F]*|"") echo "refusing non-hex commit-sha" >&2; exit 1 ;; esac
|
||||
git clone --no-checkout --filter=blob:none \
|
||||
"ssh://git@gitea-ssh.gitea.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
|
||||
cd /workspace/src && git checkout "$COMMIT_SHA"
|
||||
volumeMounts:
|
||||
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
container:
|
||||
image: alpine/k8s:1.31.0 # Alpine-based (has /bin/sh + kubectl). NOT registry.k8s.io/kubectl — that's distroless (no shell) and the deploy runs a shell script. bitnami/kubectl was removed from Docker Hub (2025).
|
||||
command: [/bin/sh, -c]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
cd /workspace/src
|
||||
kubectl apply -f infra/k8s/services/fxhnt-dashboard.yaml
|
||||
kubectl apply -f infra/k8s/orchestration/dagster.yaml
|
||||
kubectl apply -f infra/k8s/network-policies/fxhnt-cockpit.yaml
|
||||
kubectl -n foxhunt annotate deploy/fxhnt-dashboard deploy/dagster \
|
||||
fxhnt.io/deployed-sha="$(cd /workspace/src && git rev-parse --short HEAD)" --overwrite
|
||||
# Dashboard: scale 0 -> 1, NOT `rollout restart`. Its tailscale sidecar serves dashboard.fxhnt.ai;
|
||||
# a rolling restart briefly overlaps two tailscale sessions on the same identity -> session race ->
|
||||
# the external proxy link goes stale and the cockpit reads "down" post-deploy. Scaling to 0 cleanly
|
||||
# terminates the session before a fresh, non-overlapping one comes up; the git-sync initContainer
|
||||
# re-pulls src the same way, so code still updates. See reference_fxhnt_dashboard_proxy.
|
||||
#
|
||||
# SAFETY (CI review I1): once we scale to 0, a trap guarantees the deployment is scaled BACK to 1 on
|
||||
# ANY exit path — so a failure between scale-0 and scale-1 (transient apiserver error, throttle) can
|
||||
# never strand the production dashboard at 0 replicas. The trap is best-effort (its own failure is
|
||||
# swallowed) and cleared on success.
|
||||
trap 'kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=1 || true' EXIT
|
||||
kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=0
|
||||
kubectl -n foxhunt wait --for=delete pod -l app.kubernetes.io/name=fxhnt-dashboard --timeout=90s || true
|
||||
kubectl -n foxhunt scale deploy/fxhnt-dashboard --replicas=1
|
||||
trap - EXIT # scaled back up cleanly; no longer need the safety net
|
||||
kubectl -n foxhunt rollout restart deploy/dagster # dagster has no external-proxy race
|
||||
kubectl -n foxhunt rollout status deploy/fxhnt-dashboard --timeout=180s
|
||||
kubectl -n foxhunt rollout status deploy/dagster --timeout=300s
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
volumes:
|
||||
- { name: workspace, emptyDir: {} }
|
||||
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
|
||||
@@ -1,99 +0,0 @@
|
||||
# ============================================================================================
|
||||
# ONE-TIME MANUAL APPLY ONLY. Do NOT add this file to any automated/bulk `kubectl apply -f infra/k8s/jobs/`
|
||||
# path or CI pipeline. It is a `kind: Job` (not a CronJob), so there is no `suspend` flag to gate it —
|
||||
# submitting this manifest starts the backfill immediately. Apply it by hand when ready:
|
||||
# kubectl apply -f infra/k8s/jobs/fxhnt-opra-backfill.yaml
|
||||
# and delete it (`kubectl delete job/fxhnt-opra-backfill -n foxhunt`) before ever re-applying.
|
||||
# ============================================================================================
|
||||
#
|
||||
# One-time historical freeze: `fxhnt backfill-xsp-opra --start 2013-04-01 --max-cost 25.0`, from 2013-04-01
|
||||
# (XSP's mini-SPX option listing era) through today, freezing the WIDE DTE[1,45]/moneyness[0.70,1.20] put
|
||||
# slice + near-ATM calls into `xsp_option_bars`. BATCHED per MONTH (`_backfill_month`): one range `definition`
|
||||
# + one range `ohlcv-1d` fetch per month, then every trading day processed LOCALLY — ~2 OPRA calls/month vs
|
||||
# the old ~6-8/day (~1h vs ~45h over 2013->). `--max-cost 25.0` is the per-MONTH cost cap (the guard stays
|
||||
# ACTIVE, never disabled; a month over budget is logged + skipped, never a silent partial-freeze).
|
||||
#
|
||||
# Idempotent/resumable: `upsert_bars` merges on (osi_symbol, date), so RE-APPLYING this exact manifest after
|
||||
# an interruption is safe (already-frozen days are silently re-written, not duplicated) — it just re-spends
|
||||
# the OPRA cost on the days already done. To resume WITHOUT re-paying for completed months, read the last
|
||||
# "reached YYYY-MM" line from `kubectl logs`, delete the Job, and re-apply with `--start` bumped to that
|
||||
# month before resubmitting.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: fxhnt-opra-backfill
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/name: fxhnt-opra-backfill, app.kubernetes.io/part-of: foxhunt }
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
ttlSecondsAfterFinished: 604800 # 7d — keep the finished Job around for log inspection before GC
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: fxhnt-opra-backfill # egress NetworkPolicy selects this
|
||||
app.kubernetes.io/part-of: foxhunt # gitea ingress (git-sync) + base egress
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
imagePullSecrets:
|
||||
- name: scw-registry
|
||||
initContainers:
|
||||
- name: git-sync
|
||||
image: alpine/git:latest
|
||||
command: [/bin/sh, -c]
|
||||
args:
|
||||
- |
|
||||
mkdir -p /root/.ssh && cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519 && chmod 600 /root/.ssh/id_ed25519
|
||||
printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > /root/.ssh/config
|
||||
rm -rf /code/src
|
||||
git clone --depth 1 --branch master --filter=blob:none --sparse \
|
||||
"ssh://git@gitea-ssh.gitea.svc.cluster.local:22/gitadmin/fxhnt.git" /tmp/repo
|
||||
cd /tmp/repo && git sparse-checkout set src && [ -d /tmp/repo/src ]
|
||||
cp -a /tmp/repo/src /code/src && echo "git-sync OK: $(git rev-parse --short HEAD)"
|
||||
resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { cpu: 500m, memory: 256Mi } }
|
||||
volumeMounts:
|
||||
- { name: code, mountPath: /code }
|
||||
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
|
||||
containers:
|
||||
- name: backfill
|
||||
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
pip install --quiet --no-cache-dir databento==0.79.0
|
||||
fxhnt backfill-xsp-opra --start 2013-04-01 --max-cost 25.0
|
||||
env:
|
||||
- { name: PYTHONPATH, value: /code/src }
|
||||
- { name: FXHNT_OPERATIONAL_DSN, value: "postgresql+psycopg://foxhunt@postgres.foxhunt.svc.cluster.local:5432/fxhnt" }
|
||||
- name: PGPASSWORD
|
||||
valueFrom: { secretKeyRef: { name: db-credentials, key: password } }
|
||||
- name: DATABENTO_API_KEY
|
||||
valueFrom: { secretKeyRef: { name: databento-credentials, key: api-key } }
|
||||
- { name: PYTHONUNBUFFERED, value: "1" } # stream monthly progress logs live
|
||||
resources:
|
||||
requests: { memory: "512Mi", cpu: "250m" }
|
||||
limits: { memory: "1Gi", cpu: "1" }
|
||||
volumeMounts:
|
||||
- { name: code, mountPath: /code, readOnly: true }
|
||||
volumes:
|
||||
- { name: code, emptyDir: {} }
|
||||
- { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
|
||||
---
|
||||
# Egress: DNS + Postgres (5432, write xsp_option_bars) + gitea-sshd (22, git-sync) + external 443 (OPRA via
|
||||
# Databento historical API). No IBKR/ib-gateway needed — this Job never places an order.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: fxhnt-opra-backfill
|
||||
namespace: foxhunt
|
||||
labels: { app.kubernetes.io/part-of: foxhunt }
|
||||
spec:
|
||||
podSelector: { matchLabels: { app.kubernetes.io/name: fxhnt-opra-backfill } }
|
||||
policyTypes: [Egress]
|
||||
egress:
|
||||
- ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]
|
||||
- to: [{ podSelector: { matchLabels: { app.kubernetes.io/name: postgres } } }]
|
||||
ports: [{ port: 5432, protocol: TCP }]
|
||||
- to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: gitea } }, podSelector: { matchLabels: { app.kubernetes.io/name: gitea } } }]
|
||||
ports: [{ port: 2222, protocol: TCP }, { port: 22, protocol: TCP }]
|
||||
- to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16] } }] # OPRA 443
|
||||
ports: [{ port: 443, protocol: TCP }]
|
||||
@@ -28,8 +28,10 @@ spec:
|
||||
- ports: # DNS
|
||||
- { port: 53, protocol: UDP }
|
||||
- { port: 53, protocol: TCP }
|
||||
- to: # tailscale sidecar -> coordination/DERP (public internet)
|
||||
- ipBlock: { cidr: 0.0.0.0/0, except: [10.32.0.0/16, 100.64.0.0/15] }
|
||||
- to: [{ ipBlock: { cidr: 172.16.0.11/32 } }] # kube-apiserver REAL endpoint (kube-proxy DNATs kubernetes.default.svc 10.32.0.1:443 -> this before NP eval). The tailscale sidecar reads/writes its ts-state Secret via the apiserver — WITHOUT this it crashloops "context deadline exceeded" on a fresh start (e.g. after the deploy scale-0->1), causing a bad gateway. Mirrors the dagster NP fix (06d12fd).
|
||||
ports: [{ port: 6443, protocol: TCP }]
|
||||
- to: # tailscale sidecar -> coordination/DERP (public internet). except now includes 172.16.0.0/16 so the apiserver rule above is the only cluster-internal path.
|
||||
- ipBlock: { cidr: 0.0.0.0/0, except: [10.32.0.0/16, 100.64.0.0/15, 172.16.0.0/16] }
|
||||
ports:
|
||||
- { port: 443, protocol: TCP }
|
||||
- { port: 3478, protocol: UDP }
|
||||
|
||||
@@ -40,8 +40,8 @@ data:
|
||||
config:
|
||||
service_account_name: tailscale-dagster
|
||||
job_namespace: foxhunt
|
||||
job_image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s-25e3d7c
|
||||
image_pull_policy: IfNotPresent
|
||||
job_image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
|
||||
image_pull_policy: Always # :latest -> Always-pull so a dep-bump build reaches launched run Jobs (CI review C1)
|
||||
image_pull_secrets:
|
||||
- name: scw-registry
|
||||
load_incluster_config: true
|
||||
@@ -115,36 +115,6 @@ data:
|
||||
}
|
||||
}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata: { name: tailscale-dagster, namespace: foxhunt }
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata: { name: tailscale-dagster, namespace: foxhunt }
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["create", "get", "update", "patch", "list", "watch"]
|
||||
# K8sRunLauncher: the daemon launches each Dagster run as its OWN K8s Job (own memory) so the heavy
|
||||
# bybit_book_precompute asset (6-8Gi) never OOMs the 4Gi daemon. Needs to manage Jobs + read their pods/logs.
|
||||
- apiGroups: ["batch"]
|
||||
resources: ["jobs"]
|
||||
verbs: ["create", "get", "list", "watch", "delete"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "pods/log"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["events"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata: { name: tailscale-dagster, namespace: foxhunt }
|
||||
subjects:
|
||||
- { kind: ServiceAccount, name: tailscale-dagster, namespace: foxhunt }
|
||||
roleRef: { kind: Role, name: tailscale-dagster, apiGroup: rbac.authorization.k8s.io }
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
@@ -201,7 +171,7 @@ spec:
|
||||
- { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }
|
||||
containers:
|
||||
- name: webserver
|
||||
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s-25e3d7c
|
||||
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
|
||||
command: ["dagster-webserver", "-h", "0.0.0.0", "-p", "3000",
|
||||
"-w", "/dagster-home/workspace.yaml"]
|
||||
env: &dagster_env
|
||||
@@ -229,7 +199,7 @@ spec:
|
||||
- { name: code, mountPath: /code, readOnly: true } # git-sync'd src (PYTHONPATH=/code/src)
|
||||
resources: { requests: { cpu: 100m, memory: 512Mi }, limits: { cpu: "1", memory: 1Gi } }
|
||||
- name: daemon
|
||||
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:dagster-k8s-25e3d7c
|
||||
image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
|
||||
command: ["dagster-daemon", "run", "-w", "/dagster-home/workspace.yaml"]
|
||||
env: *dagster_env
|
||||
volumeMounts:
|
||||
|
||||
72
infra/k8s/rbac/tailscale-sidecars.yaml
Normal file
72
infra/k8s/rbac/tailscale-sidecars.yaml
Normal file
@@ -0,0 +1,72 @@
|
||||
# Tailscale sidecar RBAC (ServiceAccount + Role + RoleBinding) for the fxhnt-dashboard and dagster
|
||||
# Deployments. Moved out of infra/k8s/services/fxhnt-dashboard.yaml and infra/k8s/orchestration/dagster.yaml
|
||||
# (CI review M1): those app manifests are applied every deploy by the fxhnt-ci-deployer ServiceAccount via
|
||||
# the kubectl-deploy Argo template, and having RBAC objects bundled in there meant the deployer SA needed
|
||||
# create/update/patch on roles/rolebindings in its own namespace — a privilege-escalation primitive on a
|
||||
# webhook-triggered SA. This file is STATIC and applied ONCE, out-of-band, by a human:
|
||||
#
|
||||
# kubectl apply -f infra/k8s/rbac/tailscale-sidecars.yaml
|
||||
#
|
||||
# It is NOT applied by the CI pipeline. The Deployments still reference these ServiceAccounts via
|
||||
# `serviceAccountName: tailscale-dashboard` / `tailscale-dagster` — that reference resolves because the
|
||||
# SAs are managed here, separately, not because the app manifests create them anymore.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: tailscale-dashboard
|
||||
namespace: foxhunt
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: tailscale-dashboard
|
||||
namespace: foxhunt
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["create", "get", "update", "patch", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: tailscale-dashboard
|
||||
namespace: foxhunt
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: tailscale-dashboard
|
||||
namespace: foxhunt
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: tailscale-dashboard
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata: { name: tailscale-dagster, namespace: foxhunt }
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata: { name: tailscale-dagster, namespace: foxhunt }
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["create", "get", "update", "patch", "list", "watch"]
|
||||
# K8sRunLauncher: the daemon launches each Dagster run as its OWN K8s Job (own memory) so the heavy
|
||||
# bybit_book_precompute asset (6-8Gi) never OOMs the 4Gi daemon. Needs to manage Jobs + read their pods/logs.
|
||||
- apiGroups: ["batch"]
|
||||
resources: ["jobs"]
|
||||
verbs: ["create", "get", "list", "watch", "delete"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "pods/log"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["events"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata: { name: tailscale-dagster, namespace: foxhunt }
|
||||
subjects:
|
||||
- { kind: ServiceAccount, name: tailscale-dagster, namespace: foxhunt }
|
||||
roleRef: { kind: Role, name: tailscale-dagster, apiGroup: rbac.authorization.k8s.io }
|
||||
@@ -112,33 +112,3 @@ metadata:
|
||||
spec:
|
||||
selector: { app.kubernetes.io/name: fxhnt-dashboard }
|
||||
ports: [{ port: 80, targetPort: 8080, name: http }]
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: tailscale-dashboard
|
||||
namespace: foxhunt
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: tailscale-dashboard
|
||||
namespace: foxhunt
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["create", "get", "update", "patch", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: tailscale-dashboard
|
||||
namespace: foxhunt
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: tailscale-dashboard
|
||||
namespace: foxhunt
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: tailscale-dashboard
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
|
||||
@@ -76,8 +76,11 @@ spec:
|
||||
# full re-login each time). 08:00 UTC is clear of the 14:35 rebalancer + 23:30 nightly windows.
|
||||
- name: TIME_ZONE
|
||||
value: "Etc/UTC"
|
||||
# MUST be zero-padded hh:mm (IBC's parser rejects "8:00 AM" with "Auto restart time setting must be
|
||||
# hh:mm AM or hh:mm PM" — the rejection left the daily auto-restart UNCONFIGURED, so the gateway
|
||||
# churned ~16 restarts/day instead of one clean daily cycle. Verified in the pod log 2026-07-20.)
|
||||
- name: AUTO_RESTART_TIME
|
||||
value: "8:00 AM"
|
||||
value: "08:00 AM"
|
||||
# Persist jts.ini + IBC config across pod restarts (a fresh empty volume just does a clean first-run
|
||||
# login) so settings survive a deploy/reschedule instead of being rebuilt each time.
|
||||
- name: TWS_SETTINGS_PATH
|
||||
@@ -105,7 +108,9 @@ spec:
|
||||
port: 4002
|
||||
initialDelaySeconds: 120
|
||||
periodSeconds: 30
|
||||
failureThreshold: 5
|
||||
# 6×30s = 180s tolerance: the daily auto-restart + IBKR reauth window drops port 4002 for up to a
|
||||
# couple of minutes; too tight a threshold would k8s-kill the pod mid-reauth and compound the churn.
|
||||
failureThreshold: 6
|
||||
volumeMounts:
|
||||
- name: tws-settings
|
||||
mountPath: /home/ibgateway/tws_settings
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build + deploy the fxhnt cockpit via Argo Workflows (in-cluster Kaniko build, no laptop push).
|
||||
#
|
||||
# Prereqs (one-time): the `fxhnt-cockpit` WorkflowTemplate is registered
|
||||
# (kubectl -n foxhunt apply -f infra/argo/cockpit-build-deploy.yaml)
|
||||
# and this repo is pushed to the in-cluster GitLab as root/fxhnt.git.
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/argo-deploy-cockpit.sh # build+deploy HEAD of root/fxhnt master
|
||||
# ./scripts/argo-deploy-cockpit.sh <commit-sha> # a specific commit
|
||||
set -euo pipefail
|
||||
|
||||
SHA="${1:-HEAD}"
|
||||
NS="foxhunt"
|
||||
|
||||
# Resolve HEAD to the real master sha so the in-cluster clone checks out exactly what was pushed.
|
||||
if [ "$SHA" = "HEAD" ]; then
|
||||
SHA="$(git rev-parse HEAD)"
|
||||
fi
|
||||
|
||||
echo "Submitting fxhnt-cockpit build+deploy for ${SHA} ..."
|
||||
argo submit -n "$NS" --from=wftmpl/fxhnt-cockpit \
|
||||
-p commit-sha="$SHA" \
|
||||
--watch
|
||||
@@ -1,134 +0,0 @@
|
||||
"""Deribit DVOL adapter — the DVOL annualised IMPLIED-vol index (the IV leg of the VRP edge).
|
||||
|
||||
Why this signal: Deribit's DVOL is a BTC/ETH 30-day implied-volatility index (annualised, in vol points, e.g.
|
||||
33–83 for BTC) computed off the live options order book — the market's forward-looking expected vol. Paired
|
||||
with the REALIZED vol of the underlying it is the volatility risk premium (VRP): IV systematically exceeds
|
||||
RV, so SELLING variance earns the spread, but with fat left tails when a vol spike makes RV >> IV. The
|
||||
backtest SIGNAL uses DVOL (IV) + close-to-close RV; live execution would be short Bybit option straddles
|
||||
(a documented cross-venue caveat — the DVOL index is Deribit's, the harvest is Bybit's options).
|
||||
|
||||
Endpoint (HTTP is INJECTED so tests run with no network):
|
||||
GET /api/v2/public/get_volatility_index_data?currency=BTC&start_timestamp=<ms>&end_timestamp=<ms>
|
||||
&resolution=86400
|
||||
-> result.data[] of rows [ts_ms, open, high, low, close], oldest-first. close (idx 4) = the DVOL value.
|
||||
resolution=86400 (seconds) = daily, so there is one row per UTC day. The API caps ~1000 rows/request, so
|
||||
we paginate BACKWARD via end_timestamp for the full ~5.3y of history (BTC/ETH DVOL begins 2021-03-24).
|
||||
|
||||
`dvol_history(currency, *, start_ms=None) -> {epoch_day: dvol_close}` paginates BACKWARD via end_timestamp
|
||||
(mirrors BybitAccountRatio.long_ratio_history's shape) down to a floor (default ≈ 2021-03, just below the
|
||||
DVOL inception 2021-03-24), stopping on the first empty page or once the oldest row falls below the floor.
|
||||
epoch_day =
|
||||
timestamp_ms // 86_400_000.
|
||||
|
||||
CLOCK SEAM (mirrors DeribitFunding): the wall clock is INJECTED (`clock: Callable[[], float] = time.time`).
|
||||
The first page anchors its end_timestamp at `now`; a bare `time.time()` in the loop would make the backward
|
||||
walk depend on the calendar date the test runs on — the determinism bug this seam exists to prevent. Tests
|
||||
pass a fixed clock so the walk is reproducible.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
import json
|
||||
import time
|
||||
import urllib.request
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
|
||||
_API = "https://www.deribit.com/api/v2/public/get_volatility_index_data"
|
||||
_PAGE_LIMIT = 1000 # Deribit caps ~1000 rows/request on the volatility-index endpoint
|
||||
_DAY_MS = 86_400_000
|
||||
_CLOSE_IX = 4 # row layout: [ts_ms, open, high, low, close]
|
||||
|
||||
# Backstop floor for the backward pagination: ≈ 2021-03-01 UTC in ms. The Deribit DVOL index for BTC and ETH
|
||||
# actually begins 2021-03-24 (verified against the live get_volatility_index_data endpoint); 2021-03-01 is the
|
||||
# natural lower bound just below inception at which the backward walk terminates. (The previous 2023-09 floor
|
||||
# silently discarded ~890 days — nearly half — of valid DVOL history, halving the VRP study's non-overlapping
|
||||
# hold count; this floor recovers the full ~5.3y back to inception.)
|
||||
DERIBIT_DVOL_FLOOR_MS = int(_dt.datetime(2021, 3, 1, tzinfo=_dt.UTC).timestamp() * 1000)
|
||||
|
||||
|
||||
def _http_get_json(url: str, tries: int = 4) -> Any:
|
||||
"""Default JSON fetcher (urllib, retry+backoff) — mirrors the other free-data adapters' `_get`."""
|
||||
for a in range(tries):
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
|
||||
with urllib.request.urlopen(req, timeout=40) as r:
|
||||
return json.loads(r.read())
|
||||
except Exception: # noqa: BLE001 — transient HTTP; retry then re-raise
|
||||
if a == tries - 1:
|
||||
raise
|
||||
time.sleep(2 * (a + 1))
|
||||
raise RuntimeError("unreachable")
|
||||
|
||||
|
||||
def _with_backoff(fetch: Callable[[str], Any], *, sleep: Callable[[float], None],
|
||||
base: float, tries: int) -> Callable[[str], Any]:
|
||||
"""Wrap `fetch(url)->dict` with exponential backoff on ANY transient exception (the Deribit endpoint has
|
||||
no Bybit-style retCode; a 429/5xx surfaces as a raised exception). Retries `base·2**attempt` up to
|
||||
`tries`; the final attempt re-raises. `sleep` is injectable (tests pass a no-op recorder)."""
|
||||
def wrapped(url: str) -> Any:
|
||||
for attempt in range(tries):
|
||||
try:
|
||||
return fetch(url)
|
||||
except Exception: # noqa: BLE001 — transient HTTP; backoff then re-raise on exhaustion
|
||||
if attempt == tries - 1:
|
||||
raise
|
||||
sleep(base * (2 ** attempt))
|
||||
raise RuntimeError("unreachable") # pragma: no cover — loop always returns or raises
|
||||
return wrapped
|
||||
|
||||
|
||||
class DeribitDVOL:
|
||||
def __init__(self, *, fetch: Callable[[str], Any] = _http_get_json,
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
clock: Callable[[], float] = time.time,
|
||||
backoff_base: float = 0.5, backoff_tries: int = 5) -> None:
|
||||
# fetch(url) -> parsed JSON dict (injected for tests). Wrapped with exponential backoff so a transient
|
||||
# 429/5xx from Deribit is retried instead of failing — `sleep` is the (injectable) backoff sleeper
|
||||
# too, so tests passing a no-op sleep never actually wait.
|
||||
self._fetch = _with_backoff(fetch, sleep=sleep, base=backoff_base, tries=backoff_tries)
|
||||
self._sleep = sleep # throttle between pages in production; no-op-able in tests
|
||||
# `clock` is the wall-clock seam (epoch seconds; defaults to time.time). The first page anchors its
|
||||
# end_timestamp at `now`; injecting a FIXED clock makes the backward walk deterministic so tests do
|
||||
# not depend on the calendar date they run on (mirrors DeribitFunding — a bare time.time() in the loop
|
||||
# is the determinism bug this seam exists to prevent).
|
||||
self._clock = clock
|
||||
|
||||
def dvol_history(self, currency: str, *, start_ms: int | None = None) -> dict[int, float]:
|
||||
"""{epoch_day: dvol_close} for `currency` (BTC/ETH), where dvol_close = the daily DVOL index value
|
||||
(annualised implied vol, vol points). resolution=86400 → one row/day, no aggregation.
|
||||
|
||||
Paginates BACKWARD via end_timestamp: fetch the newest page, then set end_timestamp = oldest row's
|
||||
ts − 1 and loop, stopping on the first empty page or when the oldest row falls below `start_ms`
|
||||
(default DERIBIT_DVOL_FLOOR_MS ≈ 2021-03). Last-write-wins on a duplicate (currency, day) — there is
|
||||
at most one row per day, so this only guards page-boundary repeats. The infinite-loop guard breaks if
|
||||
the cursor stops advancing."""
|
||||
floor = DERIBIT_DVOL_FLOOR_MS if start_ms is None else int(start_ms)
|
||||
daily: dict[int, float] = {}
|
||||
seen: set[int] = set() # guard against duplicate ts across page boundaries
|
||||
end_ms: int | None = None # first page: no cursor (newest)
|
||||
while True:
|
||||
now_ms = int(self._clock() * 1000)
|
||||
url = (f"{_API}?currency={currency}&start_timestamp={floor}"
|
||||
f"&end_timestamp={end_ms if end_ms is not None else now_ms}&resolution=86400")
|
||||
data = self._fetch(url)
|
||||
rows = data.get("result", {}).get("data", [])
|
||||
if not rows:
|
||||
break
|
||||
oldest_ms = end_ms if end_ms is not None else None
|
||||
for row in rows:
|
||||
ts_ms = int(row[0])
|
||||
if oldest_ms is None or ts_ms < oldest_ms:
|
||||
oldest_ms = ts_ms
|
||||
if ts_ms < floor or ts_ms in seen:
|
||||
continue
|
||||
seen.add(ts_ms)
|
||||
daily[ts_ms // _DAY_MS] = float(row[_CLOSE_IX])
|
||||
if oldest_ms is None or oldest_ms <= floor:
|
||||
break # walked back to the floor → done
|
||||
next_end = oldest_ms - 1
|
||||
if end_ms is not None and next_end >= end_ms:
|
||||
break # cursor not advancing → avoid an infinite loop
|
||||
end_ms = next_end
|
||||
self._sleep(0.1) # gentle throttle; tests inject a no-op sleep
|
||||
return daily
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Key assets: `futures_bars` (Databento front-month arrays), `bybit_warehouse_refresh` (nightly Bybit
|
||||
`bybit_features` update), the per-edge forward-nav tracks (xsfunding/unlock/positioning/bybit_4edge/
|
||||
sixtyforty/multistrat/vrp), and `cockpit_forward` (recompute the forward state from the DB anchors and
|
||||
sixtyforty/multistrat), and `cockpit_forward` (recompute the forward state from the DB anchors and
|
||||
upsert into the operational cockpit DB). The DB is the single SSOT — no JSON state files.
|
||||
|
||||
NOTE: this module intentionally omits `from __future__ import annotations` because
|
||||
@@ -290,7 +290,7 @@ def _persist_track_backtest_ref(context: AssetExecutionContext, name: str, build
|
||||
"""Persist the reconciliation-gate backtest reference for an individual forward track, routed through the
|
||||
ONE `backtest_ref` dispatcher (fxhnt.application.backtest_refs) so this inline nightly path, the
|
||||
`backtest-refs --all` CLI (report-kind), and the B3 runner all share the same dispatch. For
|
||||
`recompute-replay` strategies (multistrat/vrp/crypto_tstrend/stablecoin/unlock) this is bit-identical to
|
||||
`recompute-replay` strategies (multistrat/crypto_tstrend/stablecoin/unlock) this is bit-identical to
|
||||
the prior inline behaviour: `build_strategy().advance(None, {})` = the full inception series at the EXACT
|
||||
basis the forward books (e.g. crypto_tstrend's forward is 15%-book-vol-targeted, not the raw TrendRunner
|
||||
sleeve). For a `sleeve` strategy it would resolve a feature store instead, since `sleeve_returns_from_store`
|
||||
@@ -302,9 +302,9 @@ def _persist_track_backtest_ref(context: AssetExecutionContext, name: str, build
|
||||
BEST-EFFORT: a ref-persist failure logs and returns — it must NEVER fail the forward step (the ref is a
|
||||
gate input, not the track itself). Without it the gate WITHHOLDS PASS (no ref to reconcile).
|
||||
|
||||
For `recompute-replay` strategies (multistrat/vrp/crypto_tstrend/stablecoin/unlock) this ALSO persists
|
||||
For `recompute-replay` strategies (multistrat/crypto_tstrend/stablecoin/unlock) this ALSO persists
|
||||
the full inception series into `sim_curve_ret` (Task 1/D1) — the precomputed curve the cockpit's
|
||||
`/paper/sim` reads for its IBKR books (multistrat/vrp), mirroring the Bybit book's `bybit_sleeve_ret`
|
||||
`/paper/sim` reads for its IBKR books (multistrat), mirroring the Bybit book's `bybit_sleeve_ret`
|
||||
precompute. One replay call feeds BOTH writes (the reconciliation ref and the sim curve), so the sim's
|
||||
curve is always the SAME basis the gate reconciles the forward track against."""
|
||||
import datetime as _dt
|
||||
@@ -489,235 +489,6 @@ def multistrat_levered_nav(context: AssetExecutionContext) -> dict: # type: ign
|
||||
)
|
||||
|
||||
|
||||
def _iso_date(s: str): # type: ignore[no-untyped-def]
|
||||
import datetime as _dt
|
||||
return _dt.date.fromisoformat(s)
|
||||
|
||||
|
||||
def _forward_from_marks(chain, marks: dict, as_of: str) -> float: # type: ignore[no-untyped-def]
|
||||
"""Put-call-parity forward from a PRE-FETCHED single-day marks dict `{osi_symbol: close}`. Pure (no OPRA)
|
||||
— shared by the per-day fetch path (`_xsp_forward_estimate`) and the batched backfill (in-memory marks),
|
||||
so both derive the forward identically. Nearest-to-30-DTE expiry, ATM common strike by min distance to
|
||||
the mid; ties over a SORTED sequence (deterministic across PYTHONHASHSEED)."""
|
||||
from fxhnt.application.vrp_pricing import parity_forward
|
||||
near = [c for c in chain if 20 <= (c.expiry - _iso_date(as_of)).days <= 45]
|
||||
if not near:
|
||||
raise RuntimeError(f"no XSP chain for {as_of} — cannot estimate forward")
|
||||
exp = min(sorted({c.expiry for c in near}), key=lambda e: abs((e - _iso_date(as_of)).days - 30))
|
||||
puts = {c.strike: c for c in near if c.expiry == exp and c.right == "P"}
|
||||
calls = {c.strike: c for c in near if c.expiry == exp and c.right == "C"}
|
||||
common = sorted(set(puts) & set(calls))
|
||||
if not common:
|
||||
raise RuntimeError(f"no matched call/put strike for {as_of} forward")
|
||||
mid = common[len(common) // 2]
|
||||
for k in sorted(common, key=lambda s: abs(s - mid)):
|
||||
p, c = marks.get(puts[k].osi_symbol), marks.get(calls[k].osi_symbol)
|
||||
if p is not None and c is not None:
|
||||
return parity_forward(call_atm=c, put_atm=p, k_atm=k)
|
||||
raise RuntimeError(f"no ATM call/put pair with marks for {as_of}")
|
||||
|
||||
|
||||
def _band_contracts(chain, as_of: str, forward: float): # type: ignore[no-untyped-def]
|
||||
"""The daily freeze band: WIDE put band (DTE[1,45], moneyness[0.70,1.20] — wider than the ~30-DTE ENTRY
|
||||
window so a spread's legs stay frozen through their whole held life + market moves, no zombie positions)
|
||||
plus near-ATM CALLS (for the parity forward off frozen data). Pure filter."""
|
||||
d = _iso_date(as_of)
|
||||
puts = [c for c in chain if c.right == "P" and 1 <= (c.expiry - d).days <= 45
|
||||
and 0.70 <= c.strike / forward <= 1.20]
|
||||
calls = [c for c in chain if c.right == "C" and 20 <= (c.expiry - d).days <= 45
|
||||
and 0.96 <= c.strike / forward <= 1.04]
|
||||
return puts + calls
|
||||
|
||||
|
||||
def _band_rows_from_marks(chain, marks: dict, as_of: str, forward: float) -> list: # type: ignore[type-arg,no-untyped-def]
|
||||
"""Band rows for `as_of` from a PRE-FETCHED single-day marks dict `{osi_symbol: close}`. Pure — shared by
|
||||
the per-day path and the batched backfill."""
|
||||
return [dict(osi_symbol=c.osi_symbol, date=as_of, expiry=c.expiry.isoformat(),
|
||||
strike=c.strike, right=c.right, close=marks[c.osi_symbol])
|
||||
for c in _band_contracts(chain, as_of, forward) if c.osi_symbol in marks]
|
||||
|
||||
|
||||
def _day_marks(dbn, osis, as_of: str) -> dict: # type: ignore[type-arg,no-untyped-def]
|
||||
"""Fetch `osis` for a single day and flatten `{osi:{date:close}}` -> `{osi: close}` (dropping missing)."""
|
||||
return {osi: m[as_of] for osi, m in dbn.fetch_option_bars(osis, as_of, as_of).items() if as_of in m}
|
||||
|
||||
|
||||
def _xsp_forward_estimate(dbn, as_of: str, chain=None) -> float: # type: ignore[no-untyped-def]
|
||||
"""Per-day parity forward: fetch the near-ATM expiry's marks, then `_forward_from_marks`. `chain` may be
|
||||
passed in to avoid a redundant OPRA definition fetch (see `_freeze_day`)."""
|
||||
full = chain if chain is not None else dbn.fetch_option_chain("XSP.OPT", as_of)
|
||||
near = [c for c in full if 20 <= (c.expiry - _iso_date(as_of)).days <= 45]
|
||||
if not near:
|
||||
raise RuntimeError(f"no XSP chain for {as_of} — cannot estimate forward")
|
||||
exp = min(sorted({c.expiry for c in near}), key=lambda e: abs((e - _iso_date(as_of)).days - 30))
|
||||
atm_osis = [c.osi_symbol for c in near if c.expiry == exp]
|
||||
return _forward_from_marks(full, _day_marks(dbn, atm_osis, as_of), as_of)
|
||||
|
||||
|
||||
def _freeze_xsp_slice(dbn, bars, *, as_of: str, forward: float, at, chain=None) -> None: # type: ignore[no-untyped-def]
|
||||
"""Per-day freeze: fetch the band's marks, then upsert `_band_rows_from_marks`. `chain` may be passed in
|
||||
to avoid a redundant OPRA definition fetch (see `_freeze_day`)."""
|
||||
full = chain if chain is not None else dbn.fetch_option_chain("XSP.OPT", as_of)
|
||||
band = _band_contracts(full, as_of, forward)
|
||||
marks = _day_marks(dbn, [c.osi_symbol for c in band], as_of)
|
||||
bars.upsert_bars(_band_rows_from_marks(full, marks, as_of, forward), at)
|
||||
|
||||
|
||||
def _freeze_day(dbn, bars, *, as_of: str, at) -> float: # type: ignore[no-untyped-def]
|
||||
"""Fetch the XSP option chain ONCE for `as_of`, derive the parity forward from it, and freeze the band
|
||||
from the SAME chain. Avoids the double `fetch_option_chain` (definition) call — `_xsp_forward_estimate`
|
||||
and `_freeze_xsp_slice` each used to fetch it independently, doubling OPRA definition cost on every
|
||||
nightly `vrp_nav` run and the 2013-> backfill. Returns the forward."""
|
||||
chain = dbn.fetch_option_chain("XSP.OPT", as_of)
|
||||
fwd = _xsp_forward_estimate(dbn, as_of, chain=chain)
|
||||
_freeze_xsp_slice(dbn, bars, as_of=as_of, forward=fwd, at=at, chain=chain)
|
||||
return fwd
|
||||
|
||||
|
||||
def _prev_trading_day(d): # type: ignore[no-untyped-def]
|
||||
"""One trading day BEFORE `d`, skipping Sat/Sun (isoweekday 6/7). A market holiday is not a fixed calendar
|
||||
rule, so it is absorbed by one more step-back in `_freeze_latest_session`'s bound rather than skipped here."""
|
||||
import datetime as _dt
|
||||
d -= _dt.timedelta(days=1)
|
||||
while d.isoweekday() >= 6: # Sat=6, Sun=7 — never a trading session
|
||||
d -= _dt.timedelta(days=1)
|
||||
return d
|
||||
|
||||
|
||||
# Message fragments raised by `_xsp_forward_estimate`/`_forward_from_marks` when a PUBLISHED session is too
|
||||
# degenerate to form the parity forward (no near-DTE chain / no matched call-put strike / no ATM pair with
|
||||
# marks). Matched narrowly so `_freeze_latest_session` steps back on a broken session but never swallows an
|
||||
# unrelated RuntimeError (e.g. a DB write failure inside the freeze).
|
||||
_DEGENERATE_FORWARD_MARKERS = ("cannot estimate forward", "matched call/put strike", "ATM call/put pair")
|
||||
|
||||
|
||||
def _is_degenerate_forward_error(e: Exception) -> bool:
|
||||
msg = str(e)
|
||||
return any(marker in msg for marker in _DEGENERATE_FORWARD_MARKERS)
|
||||
|
||||
|
||||
def _freeze_latest_session(dbn, bars, *, at, max_steps: int = 5) -> str: # type: ignore[no-untyped-def]
|
||||
"""Freeze the LATEST COMPLETE, USABLE OPRA session (robust, timing-independent). The candidate starts at
|
||||
OPRA's available-END date (`dbn.last_available_option_date()`); today's session is still settling until
|
||||
after the close, so freezing it fails with the normalized `DatabentoRangeUnavailable` (the
|
||||
`data_schema_not_fully_available` 422 OR the C1a empty-window clamp). A PUBLISHED-but-DEGENERATE session
|
||||
instead surfaces as a forward-estimation `RuntimeError` ("no XSP chain … cannot estimate forward" / "no
|
||||
matched call/put strike …" / "no ATM call/put pair …"). Either signal — unavailable OR degenerate — steps
|
||||
the candidate back ONE trading day (skip Sat/Sun) and retries, BOUNDED to `max_steps` (a market holiday or
|
||||
one broken session just costs one more step). The FIRST candidate that freezes is the target; the ISO date
|
||||
frozen is RETURNED (vrp_nav logs it). Unrelated `RuntimeError`s (not forward-estimation) propagate.
|
||||
|
||||
If no usable session is found within the bound, raise `DatabentoRangeUnavailable` — the C1b resilience
|
||||
wrapper in `vrp_nav` catches it → recompute off the frozen table, and the health axis keeps vrp STALE
|
||||
(correct: we genuinely could not advance the freeze). This is LOUD by design; NEVER a silent no-op, and a
|
||||
bad/bar-less date is NEVER returned — only the last COMPLETE, non-degenerate session's marks are frozen."""
|
||||
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
|
||||
start = dbn.last_available_option_date()
|
||||
candidate = start
|
||||
last_err: Exception | None = None
|
||||
for _ in range(max_steps + 1): # the start candidate + up to `max_steps` bounded step-backs
|
||||
iso = candidate.isoformat()
|
||||
try:
|
||||
_freeze_day(dbn, bars, as_of=iso, at=at)
|
||||
return iso
|
||||
except DatabentoRangeUnavailable as e: # still-settling / not-yet-available session → step back
|
||||
last_err = e
|
||||
candidate = _prev_trading_day(candidate)
|
||||
except RuntimeError as e: # published-but-degenerate session (no derivable forward)
|
||||
if not _is_degenerate_forward_error(e):
|
||||
raise # an unrelated RuntimeError — do NOT silently swallow it
|
||||
last_err = e
|
||||
candidate = _prev_trading_day(candidate)
|
||||
raise DatabentoRangeUnavailable(
|
||||
f"no complete, non-degenerate OPRA session within {max_steps} trading days back "
|
||||
f"from {start.isoformat()}") from last_err
|
||||
|
||||
|
||||
def _rough_forward(chain, as_of: str): # type: ignore[no-untyped-def]
|
||||
"""Coarse forward = median strike of the ~30-DTE expiry (NO marks needed) — used ONLY to bound the bulk
|
||||
monthly ohlcv fetch to a wide strike band. The precise per-day forward is always recomputed from real
|
||||
marks via `_forward_from_marks`. XSP lists strikes densely+symmetrically around ATM, so the median strike
|
||||
is a good ATM proxy for bounding."""
|
||||
d = _iso_date(as_of)
|
||||
near = [c for c in chain if 20 <= (c.expiry - d).days <= 45]
|
||||
if not near:
|
||||
return None
|
||||
exp = min(sorted({c.expiry for c in near}), key=lambda e: abs((e - d).days - 30))
|
||||
strikes = sorted({c.strike for c in near if c.expiry == exp})
|
||||
return strikes[len(strikes) // 2] if strikes else None
|
||||
|
||||
|
||||
def _backfill_month(dbn, bars_repo, *, month_start: str, month_end: str, at) -> tuple: # type: ignore[type-arg,no-untyped-def]
|
||||
"""BATCHED one-month freeze: ONE range definition fetch + ONE range ohlcv fetch (bulk), then process each
|
||||
trading day LOCALLY (parity forward + band + upsert). ~2 OPRA calls/month vs ~6-8/day — the per-day
|
||||
pattern was ~26k API round-trips over 2013->, ~45h; batched is ~2 calls/month * ~160 months, ~1h.
|
||||
Returns (n_ok, n_fail) days. A day with no arb-free ATM pair (sparse early-XSP data) is counted failed,
|
||||
never aborts the month."""
|
||||
import datetime as _dt
|
||||
chain = dbn.fetch_option_chain_range("XSP.OPT", month_start, month_end)
|
||||
if not chain:
|
||||
return (0, 0)
|
||||
rough = _rough_forward(chain, month_start) or _rough_forward(chain, month_end)
|
||||
if rough is None:
|
||||
return (0, 0)
|
||||
ms, me = _iso_date(month_start), _iso_date(month_end)
|
||||
# bound the bulk fetch: any expiry a day this month could reference at DTE<=45, and a strike band WIDE
|
||||
# enough (+-50% of the month's median) to cover intra-month moves + the precise [0.70,1.20] daily band
|
||||
# even in a crash month (COVID Mar-2020 XSP fell ~34% intramonth).
|
||||
band = [c for c in chain
|
||||
if (ms + _dt.timedelta(days=1)) <= c.expiry <= (me + _dt.timedelta(days=45))
|
||||
and 0.50 <= c.strike / rough <= 1.50]
|
||||
if not band:
|
||||
return (0, 0)
|
||||
bars = dbn.fetch_option_bars([c.osi_symbol for c in band], month_start, month_end) # bulk range, chunked
|
||||
days = sorted({dd for m in bars.values() for dd in m})
|
||||
n_ok = n_fail = 0
|
||||
for iso in days:
|
||||
day_marks = {osi: m[iso] for osi, m in bars.items() if iso in m}
|
||||
try:
|
||||
fwd = _forward_from_marks(band, day_marks, iso)
|
||||
bars_repo.upsert_bars(_band_rows_from_marks(band, day_marks, iso, fwd), at)
|
||||
n_ok += 1
|
||||
except Exception: # a sparse/degenerate day (no ATM pair with marks) — skip, don't abort the month
|
||||
n_fail += 1
|
||||
return (n_ok, n_fail)
|
||||
|
||||
|
||||
@asset
|
||||
def vrp_nav(context: AssetExecutionContext) -> dict: # type: ignore[type-arg]
|
||||
"""Paper-forward defined-risk XSP put-credit-spread VRP on REAL OPRA data. Freezes today's DTE/moneyness
|
||||
put slice (plus near-ATM calls, for the parity forward) PIT into xsp_option_bars, then recomputes off the
|
||||
FROZEN table (spec §A) so the record is stable against any OPRA revision. persist_backtest_ref=True
|
||||
publishes the basis-matched real ref for the gate."""
|
||||
import datetime as _dt
|
||||
|
||||
from fxhnt.adapters.data.databento import DatabentoDataProvider
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
from fxhnt.application.vrp_book import VrpStrategy
|
||||
from fxhnt.config import get_settings
|
||||
|
||||
s = get_settings()
|
||||
bars = XspOptionBarsRepo(s.operational_dsn)
|
||||
bars.migrate()
|
||||
dbn = DatabentoDataProvider(max_cost_usd=s.databento.max_cost_usd)
|
||||
at = _dt.datetime.now(_dt.UTC).replace(tzinfo=None)
|
||||
try:
|
||||
# Freeze the LATEST COMPLETE OPRA session via a bounded step-back — robust and timing-independent, so
|
||||
# the freeze advances deterministically regardless of when the job runs (never freezes today's still-
|
||||
# settling partial session's marks). Returns the ISO date actually frozen.
|
||||
session = _freeze_latest_session(dbn, bars, at=at)
|
||||
context.log.info(f"vrp_nav: froze OPRA session {session}")
|
||||
except Exception as e: # noqa: BLE001 — freeze is best-effort; the tracker recomputes off the FROZEN table
|
||||
# A bad-data day, no complete session within the step-back bound (DatabentoRangeUnavailable), or a
|
||||
# cost-cap trip must NOT kill vrp_nav: it would zero out the forward anchor AND skip the downstream
|
||||
# gate. Log + continue — the health axis keeps vrp STALE (loud), which is the correct signal.
|
||||
context.log.warning(f"vrp_nav: freeze step failed (recomputing off the frozen table): {e}")
|
||||
|
||||
return _run_paper_tracker(
|
||||
context, "vrp_nav", lambda: VrpStrategy(bars), persist_backtest_ref=True,
|
||||
)
|
||||
|
||||
|
||||
# RETIRED 2026-06-20 (cockpit cleanup): the live tracks funding_nav (absolute-hurdle, superseded by
|
||||
# cross-sectional xsfunding), gd_nav / multistrat_nav (futures/TradFi premia — falsified this session,
|
||||
# scale-gated, no deployable edge), and poc_nav (old crypto residual-mom+VRP PoC, superseded by the two
|
||||
@@ -1128,7 +899,7 @@ def bybit_paper_book(context: AssetExecutionContext) -> dict: # type: ignore[ty
|
||||
|
||||
|
||||
@asset(deps=[sixtyforty_nav, xsfunding_nav, unlock_nav, multistrat_nav, bybit_4edge_nav,
|
||||
bybit_4edge_levered_nav, positioning_nav]) # vrp_nav ARCHIVED 2026-07-15 — de-wired (fn kept below)
|
||||
bybit_4edge_levered_nav, positioning_nav])
|
||||
def cockpit_forward(context: AssetExecutionContext) -> None:
|
||||
"""Evaluate + persist the forward gate from the DB record the engine already wrote (all_summaries +
|
||||
nav_history — see `forward_ingest.ingest_forward_state`), then auto-promote any `promote_on_pass`
|
||||
@@ -1142,7 +913,7 @@ def cockpit_forward(context: AssetExecutionContext) -> None:
|
||||
dsn = get_settings().operational_dsn
|
||||
ingest_forward_state(dsn)
|
||||
# Runtime health axis (SEPARATE from the gate): surface ANY track that stopped updating LOUDLY — a red
|
||||
# cockpit badge + an ERROR log the FIRST night, so a stale recompute (the 2-week vrp rot) can never sit as
|
||||
# cockpit badge + an ERROR log the FIRST night, so a stale recompute (a dead strategy's silent rot) can never sit as
|
||||
# a quiet WAIT again. evaluate_health logs at ERROR with the unhealthy list; also surface the count here.
|
||||
unhealthy = [v for v in evaluate_health(dsn) if not v.ok]
|
||||
if unhealthy:
|
||||
|
||||
@@ -7,9 +7,8 @@ retired here — crypto is 100% Bybit now (`bybit_warehouse_refresh` + the Bybit
|
||||
`bybit_4edge` deploy book below) — was RETIRED (Phase 0b Task 7d); sixtyforty_nav/multistrat_nav
|
||||
are their OWN independent live assets below, unaffected by that retirement.)
|
||||
|
||||
(vrp_nav ARCHIVED/FALSIFIED 2026-07-15 — the VRP sleeve is a shelved dead-end; its asset function is KEPT
|
||||
in assets.py but DE-WIRED from the nightly here so it no longer materializes. See
|
||||
project_fxhnt_diversifier_hunt_2026_07_15.)
|
||||
(VRP removed entirely 2026-07-20 — see project_fxhnt_diversifier_hunt_2026_07_15 for the
|
||||
falsification that led to the removal.)
|
||||
|
||||
NOTE: no `from __future__ import annotations` — Dagster 1.12 runtime type-hint inspection."""
|
||||
from dagster import DefaultScheduleStatus, Definitions, ScheduleDefinition, define_asset_job
|
||||
|
||||
@@ -68,15 +68,6 @@ def track_builders(settings: Settings, data_dir: str) -> TrackBuilders:
|
||||
# builders here are gone with them. crypto_tstrend the BYBIT BOOK SLEEVE is unaffected — that lives
|
||||
# under `bybit_4edge`'s own builder below, not here.)
|
||||
|
||||
# ---- vrp_nav: defined-risk XSP put-spread VRP on the frozen OPRA PIT table (recompute) ----------------
|
||||
def _build_vrp() -> Any:
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
from fxhnt.application.vrp_book import VrpStrategy
|
||||
|
||||
return VrpStrategy(XspOptionBarsRepo(settings.operational_dsn))
|
||||
|
||||
builders["vrp"] = ("vrp_state", _build_vrp)
|
||||
|
||||
# ---- unlock_nav: single-sleeve unlock edge on the liquid universe (Phase 0b Task 2: own Bybit track, ----
|
||||
# ---- the SAME single-sleeve construction as xsfunding/positioning, mirrored here; carries the unlock ----
|
||||
# ---- calendar the sleeve needs, unlike xsfunding/positioning) --------------------------------------------
|
||||
|
||||
@@ -43,7 +43,8 @@ class StrategyHealthRow(CockpitBase):
|
||||
"""Runtime HEALTH verdict per track — a SEPARATE axis from the gate (a track can be legitimately
|
||||
WAIT/building AND healthy; STALE/NO_REF/BROKEN is orthogonal). Written each nightly run by
|
||||
`forward_health.evaluate_health`; the cockpit reads it to render a LOUD red badge so a live/gated track
|
||||
that stops updating surfaces on day 1 (the anti-silent-rot backbone that would have caught vrp). One row
|
||||
that stops updating surfaces on day 1 (the anti-silent-rot backbone that would have caught a dead
|
||||
strategy's silent staleness). One row
|
||||
per strategy, replaced each run (a track that recovers clears its stale row). Brand-new table →
|
||||
`create_all` (ForwardNavRepo.migrate) creates it; no ALTER, Postgres-safe."""
|
||||
__tablename__ = "strategy_health"
|
||||
@@ -225,7 +226,7 @@ class BybitSleeveRetRow(CockpitBase):
|
||||
|
||||
|
||||
class SimCurveRetRow(CockpitBase):
|
||||
"""One recompute-replay strategy's (multistrat/vrp — the IBKR paper books the Backtest page's sim reads)
|
||||
"""One recompute-replay strategy's (multistrat — the IBKR paper books the Backtest page's sim reads)
|
||||
daily honest-cost return for one run_date. Mirrors `BybitSleeveRetRow`'s shape/purpose for the Bybit-book
|
||||
sim: written nightly by `_persist_track_backtest_ref` (the `fxhnt-backtest-refs` Job) from the SAME
|
||||
inception rows `backtest_ref`'s recompute-replay branch derives the reconciliation-gate ref from, so
|
||||
|
||||
@@ -143,9 +143,9 @@ class ForwardNavRepo:
|
||||
return list(s.scalars(select(StrategyHealthRow).order_by(StrategyHealthRow.strategy_id)))
|
||||
|
||||
def xsp_freeze_max_date(self) -> str | None:
|
||||
"""The latest date present in the vrp freeze table (`xsp_option_bars`), ISO yyyy-mm-dd, or None when
|
||||
the table is empty. The freeze-staleness axis: a vrp recompute is bounded by this max date, so if the
|
||||
freeze stops advancing the track is STALE even though its forward 'updates'."""
|
||||
"""The latest date present in the opra-pit-anchored freeze table (`xsp_option_bars`), ISO yyyy-mm-dd,
|
||||
or None when the table is empty. The freeze-staleness axis: a frozen-input recompute is bounded by
|
||||
this max date, so if the freeze stops advancing the track is STALE even though its forward 'updates'."""
|
||||
from sqlalchemy import func
|
||||
with Session(self._engine) as s:
|
||||
d = s.scalar(select(func.max(XspOptionBarRow.date)))
|
||||
@@ -196,7 +196,7 @@ class ForwardNavRepo:
|
||||
|
||||
def registry(self) -> list[StrategyRegistryRow]:
|
||||
"""The cockpit-visible registry rows. SINGLE SOURCE for the hide: an ARCHIVED sleeve (its
|
||||
`STRATEGY_REGISTRY` entry carries `archived: True` — e.g. the shelved/FALSIFIED vrp/vrp_exec) is
|
||||
`STRATEGY_REGISTRY` entry carries `archived: True` — e.g. a shelved/FALSIFIED strategy) is
|
||||
excluded here, so EVERY caller (fleet overview, per-strategy gate specs, `detail()`'s reg lookup)
|
||||
drops it automatically. The DB row is still seeded (code + data kept) — it is only hidden from the UI."""
|
||||
with Session(self._engine) as s:
|
||||
|
||||
@@ -515,8 +515,8 @@ class PaperRepo:
|
||||
|
||||
# ---- sim_curve_ret: the PRECOMPUTED recompute-replay (IBKR) book curve /paper/sim reads ------------
|
||||
def upsert_sim_curve_ret(self, strategy_id: str, run_date: str, ret: float, *, at: dt.datetime) -> None:
|
||||
"""Idempotent per (strategy_id, run_date): write/overwrite that recompute-replay book's (multistrat/
|
||||
vrp) per-day honest-cost return. `_persist_track_backtest_ref` writes the full inception series here
|
||||
"""Idempotent per (strategy_id, run_date): write/overwrite that recompute-replay book's (multistrat)
|
||||
per-day honest-cost return. `_persist_track_backtest_ref` writes the full inception series here
|
||||
from the SAME rows it derives the reconciliation-gate ref from, so /paper/sim NEVER re-runs an IBKR
|
||||
replay on a web request — mirrors `upsert_bybit_sleeve_ret`'s pattern for the Bybit-book sim."""
|
||||
rd = dt.date.fromisoformat(run_date)
|
||||
|
||||
@@ -1,119 +0,0 @@
|
||||
"""PIT store for frozen XSP option daily bars — the deterministic substrate the VRP recompute reads. OPRA is
|
||||
hit only when FILLING this table (asset/backfill); VrpStrategy never calls OPRA. Same freeze discipline as the
|
||||
equity Yahoo-PIT snapshot."""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
|
||||
from sqlalchemy import create_engine, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from fxhnt.adapters.persistence.cockpit_models import CockpitBase, XspOptionBarRow
|
||||
from fxhnt.domain.models import OptionContract
|
||||
|
||||
|
||||
class XspOptionBarsRepo:
|
||||
def __init__(self, dsn: str) -> None:
|
||||
kw = {} if dsn.startswith("postgresql") else {"connect_args": {"check_same_thread": False}}
|
||||
self._engine = create_engine(dsn, **kw)
|
||||
self._pre: dict | None = None # in-memory replay cache (None = not preloaded; per-day DB path)
|
||||
|
||||
def migrate(self) -> None:
|
||||
CockpitBase.metadata.create_all(self._engine)
|
||||
|
||||
def preload(self, after: str | None = None) -> None:
|
||||
"""Bulk-load every frozen bar for days strictly after `after` (all days if None) into an in-memory
|
||||
index, in ONE query. A full VrpStrategy replay then serves chain_asof/bars_for/trading_days from
|
||||
memory instead of a fresh Session + round-trip per day — the N+1 that made a 1600-day replay issue
|
||||
thousands of queries. The live single-day exec path (plan_and_record_vrp) never calls this, so it
|
||||
keeps its cheap per-day DB queries unchanged. The cache is a faithful mirror of the covered range:
|
||||
every cached method falls back to the DB for any date OUTSIDE that range, so it can never
|
||||
under-return. Idempotent (re-call to re-scope)."""
|
||||
lo = dt.date.fromisoformat(after) if after else None
|
||||
marks: dict[str, dict[str, float]] = {}
|
||||
rows_by_date: dict[str, list[tuple[str, dt.date, float, str]]] = {}
|
||||
with Session(self._engine) as s:
|
||||
q = select(XspOptionBarRow.osi_symbol, XspOptionBarRow.date, XspOptionBarRow.expiry,
|
||||
XspOptionBarRow.strike, XspOptionBarRow.right, XspOptionBarRow.close)
|
||||
if lo is not None:
|
||||
q = q.where(XspOptionBarRow.date > lo)
|
||||
for osi, date, expiry, strike, right, close in s.execute(q):
|
||||
di = date.isoformat()
|
||||
marks.setdefault(di, {})[osi] = close
|
||||
rows_by_date.setdefault(di, []).append((osi, expiry, strike, right))
|
||||
self._pre = {"after": after, "marks": marks, "rows": rows_by_date, "days": sorted(rows_by_date)}
|
||||
|
||||
def _pre_covers(self, date_iso: str) -> bool:
|
||||
"""True iff `date_iso` is inside the preloaded range (strictly after the preload bound)."""
|
||||
pre = self._pre
|
||||
return pre is not None and (pre["after"] is None or date_iso > pre["after"])
|
||||
|
||||
def upsert_bars(self, rows: list[dict], at: dt.datetime) -> None:
|
||||
with Session(self._engine) as s:
|
||||
for r in rows:
|
||||
obj = XspOptionBarRow(
|
||||
osi_symbol=r["osi_symbol"], date=dt.date.fromisoformat(r["date"]),
|
||||
expiry=dt.date.fromisoformat(r["expiry"]), strike=float(r["strike"]),
|
||||
right=r["right"], close=float(r["close"]), src_updated_at=at)
|
||||
s.merge(obj) # merge = insert-or-overwrite on the (osi_symbol, date) PK
|
||||
s.commit()
|
||||
|
||||
def bars_for(self, osi_symbols: list[str], start: str, end: str) -> dict[str, dict[str, float]]:
|
||||
if not osi_symbols:
|
||||
return {}
|
||||
# Preloaded fast path — serve from memory when the whole [start, end] range is inside the cache
|
||||
# (start > preload bound => the entire range is, since start <= end). Iso date strings compare
|
||||
# lexically like dt.date, so the range test matches the DB WHERE below.
|
||||
if self._pre is not None and self._pre_covers(start):
|
||||
want = set(osi_symbols)
|
||||
out: dict[str, dict[str, float]] = {}
|
||||
if start == end: # the replay's single-day case (hot path)
|
||||
m = self._pre["marks"].get(start, {})
|
||||
return {osi: {start: m[osi]} for osi in want if osi in m}
|
||||
for di, m in self._pre["marks"].items():
|
||||
if start <= di <= end:
|
||||
for osi in want:
|
||||
if osi in m:
|
||||
out.setdefault(osi, {})[di] = m[osi]
|
||||
return out
|
||||
lo, hi = dt.date.fromisoformat(start), dt.date.fromisoformat(end)
|
||||
out: dict[str, dict[str, float]] = {}
|
||||
with Session(self._engine) as s:
|
||||
q = select(XspOptionBarRow).where(
|
||||
XspOptionBarRow.osi_symbol.in_(osi_symbols),
|
||||
XspOptionBarRow.date >= lo, XspOptionBarRow.date <= hi)
|
||||
for row in s.scalars(q):
|
||||
out.setdefault(row.osi_symbol, {})[row.date.isoformat()] = row.close
|
||||
return out
|
||||
|
||||
def trading_days(self, after: str | None) -> list[str]:
|
||||
"""Ascending distinct dates present in the table, strictly after `after` (or all if None)."""
|
||||
# Preloaded fast path — the cache holds every day > its bound, so it can answer any query whose
|
||||
# `after` is at or beyond that bound (a full preload, after=None, answers everything).
|
||||
pre = self._pre
|
||||
if pre is not None and (pre["after"] is None or (after is not None and after >= pre["after"])):
|
||||
return [d for d in pre["days"] if after is None or d > after]
|
||||
from sqlalchemy import distinct
|
||||
with Session(self._engine) as s:
|
||||
q = select(distinct(XspOptionBarRow.date)).order_by(XspOptionBarRow.date)
|
||||
days = [d.isoformat() for d in s.scalars(q)]
|
||||
return [d for d in days if after is None or d > after]
|
||||
|
||||
def chain_asof(self, date: str, dte_lo: int, dte_hi: int, right: str = "P") -> list[OptionContract]:
|
||||
d = dt.date.fromisoformat(date)
|
||||
exp_lo, exp_hi = d + dt.timedelta(days=dte_lo), d + dt.timedelta(days=dte_hi)
|
||||
# Preloaded fast path — same filter (date == d, right, expiry in [exp_lo, exp_hi]) + strike order.
|
||||
if self._pre is not None and self._pre_covers(date):
|
||||
hits = [(osi, exp, strike, r) for (osi, exp, strike, r) in self._pre["rows"].get(date, [])
|
||||
if r == right and exp_lo <= exp <= exp_hi]
|
||||
hits.sort(key=lambda t: t[2]) # deterministic strike order (matches ORDER BY)
|
||||
return [OptionContract(osi_symbol=osi, expiry=exp, strike=strike, right=r)
|
||||
for (osi, exp, strike, r) in hits]
|
||||
with Session(self._engine) as s:
|
||||
q = select(XspOptionBarRow).where(
|
||||
XspOptionBarRow.date == d, XspOptionBarRow.right == right,
|
||||
XspOptionBarRow.expiry >= exp_lo, XspOptionBarRow.expiry <= exp_hi
|
||||
).order_by(XspOptionBarRow.strike) # deterministic strike order (no ORDER BY = arbitrary
|
||||
# row order on Postgres — the ATM pairing below relies on stable iteration)
|
||||
return [OptionContract(osi_symbol=r.osi_symbol, expiry=r.expiry, strike=r.strike, right=r.right)
|
||||
for r in s.scalars(q)]
|
||||
@@ -446,12 +446,13 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
|
||||
|
||||
# The books the sim can configure. bybit_4edge = the precomputed 4-edge Bybit backtest (naive eq-wt —
|
||||
# NOT the overlay, since the A/B proved naive is best OOS), sized from bybit_sleeve_ret. bybit_4edge_levered
|
||||
# = the observe-only levered shadow. multistrat/vrp (Task 1/D1) = the IBKR paper strategies' precomputed
|
||||
# = the observe-only levered shadow. multistrat (Task 1/D1) = the IBKR paper strategies' precomputed
|
||||
# recompute-replay curve (sim_curve_ret) — generalizes the sim beyond the Bybit-only venue. Order = render
|
||||
# order of the book-switch pills: the deploy book leads (and is the default).
|
||||
# Exclude ARCHIVED books (their STRATEGY_REGISTRY entry carries `archived: True` — e.g. the
|
||||
# shelved/FALSIFIED vrp): they never appear as a Backtest book pill, and a `?book=vrp` request falls
|
||||
# back to `_DEFAULT_BOOK` (so the folded "Live forward" section can't surface an archived sleeve either).
|
||||
# Exclude ARCHIVED books (their STRATEGY_REGISTRY entry carries `archived: True` — e.g. a
|
||||
# shelved/FALSIFIED strategy): they never appear as a Backtest book pill, and a request for an
|
||||
# archived book falls back to `_DEFAULT_BOOK` (so the folded "Live forward" section can't surface an
|
||||
# archived sleeve either).
|
||||
_SIM_BOOKS = tuple(b for b in SIM_BOOKS if not STRATEGY_REGISTRY.get(b, {}).get("archived"))
|
||||
_DEFAULT_BOOK = "bybit_4edge"
|
||||
# Plain-language pill labels for the sim's book switch — NEVER the raw registry id. Sourced from the
|
||||
@@ -470,7 +471,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
|
||||
_sim_returns_cache: dict[str, tuple[float, dict[str, dict[int, float]]]] = {}
|
||||
|
||||
def _ibkr_sim_curve(book: str) -> dict[str, dict[int, float]]:
|
||||
"""The precomputed recompute-replay (IBKR) book curve for `book` (multistrat/vrp), as
|
||||
"""The precomputed recompute-replay (IBKR) book curve for `book` (multistrat), as
|
||||
`{book: {epoch_day: ret}}` — read from `sim_curve_ret` (written nightly by
|
||||
`_persist_track_backtest_ref`/the `fxhnt-backtest-refs` Job from the SAME inception rows the
|
||||
reconciliation-gate ref is derived from). NEVER recomputes the replay on this request path."""
|
||||
@@ -479,7 +480,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
|
||||
def _sim_returns(book: str = _DEFAULT_BOOK) -> dict[str, dict[int, float]]:
|
||||
"""Cached per-sleeve returns for `book`, dispatched by the registry's `backtest.kind` (Task 1/D1):
|
||||
`report`-kind Bybit books (bybit_4edge / bybit_4edge_levered) read the precomputed bybit_sleeve_ret
|
||||
table; `recompute-replay`-kind IBKR books (multistrat / vrp) read the precomputed sim_curve_ret
|
||||
table; `recompute-replay`-kind IBKR books (multistrat) read the precomputed sim_curve_ret
|
||||
table via `_ibkr_sim_curve`. NEVER recomputes any edge on the request path — both sources are
|
||||
nightly-precomputed. Memoized in-process with a 300s TTL, keyed by book (returns change only nightly)."""
|
||||
now = time.monotonic()
|
||||
@@ -548,7 +549,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
|
||||
def _measured_pending_context(book: str) -> dict[str, Any]:
|
||||
"""Graceful degrade when the precomputed curve is ABSENT: a short 'precomputing — check back after the
|
||||
nightly job' note (NOT a flat slider, never a 500). The live forward track still renders for bybit_4edge
|
||||
and for the IBKR recompute-replay books (multistrat/vrp) — each has its own real forward track."""
|
||||
and for the IBKR recompute-replay books (multistrat) — each has its own real forward track."""
|
||||
is_ibkr = registry_backtest_kind(book) == "recompute-replay"
|
||||
return {
|
||||
"measured_pending": True,
|
||||
@@ -564,7 +565,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
|
||||
|
||||
def _ibkr_measured_context(book: str, *, capital: float, start: str | None = None,
|
||||
end: str | None = None) -> dict[str, Any] | None:
|
||||
"""The IBKR recompute-replay books' (multistrat/vrp) Backtest view: compound the PRECOMPUTED per-day
|
||||
"""The IBKR recompute-replay books' (multistrat) Backtest view: compound the PRECOMPUTED per-day
|
||||
honest-cost return series (`_sim_returns(book)`, written nightly from the SAME replay basis the
|
||||
reconciliation gate reconciles the forward track against) into an equity curve via the pure
|
||||
`simulate_naive_eqwt` engine — no per-coin cost model here (that is the Bybit-only measured-cost path;
|
||||
@@ -725,7 +726,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
|
||||
capital = get_settings().paper_capital
|
||||
# Pre-fill AND bound the period inputs with the book's full precomputed date range, so the date fields
|
||||
# are never blank ("dates not set") and the picker can't go off the data. Bybit books read their range
|
||||
# from the per-coin measured cache; IBKR recompute-replay books (multistrat/vrp, Task 1/D1) read it
|
||||
# from the per-coin measured cache; IBKR recompute-replay books (multistrat, Task 1/D1) read it
|
||||
# from their own precomputed sim_curve_ret series (`_sim_returns`).
|
||||
if registry_backtest_kind(book) == "recompute-replay":
|
||||
_days = sorted(_sim_returns(book).get(book) or {})
|
||||
@@ -749,7 +750,7 @@ def create_app(repo: ForwardNavRepo, svc: DashboardReadModel | None = None,
|
||||
end: str | None = None, book: str = _DEFAULT_BOOK) -> HTMLResponse:
|
||||
"""The honest Backtest: book + capital + period -> the precomputed curve. Capital is parsed leniently
|
||||
(a cleared/comma'd/odd capital must not 422). Bybit books (report-kind) render the per-coin MEASURED-
|
||||
cost cache scaled by capital + windowed by period; IBKR recompute-replay books (multistrat/vrp, Task
|
||||
cost cache scaled by capital + windowed by period; IBKR recompute-replay books (multistrat, Task
|
||||
1/D1) compound their own precomputed honest-cost return series the same way. Absent precompute ->
|
||||
a graceful 'precomputing' note (never the flat slider, never a 500)."""
|
||||
book = book if book in _SIM_BOOKS else _DEFAULT_BOOK
|
||||
@@ -807,7 +808,7 @@ def create_app_from_settings() -> FastAPI:
|
||||
repo.migrate()
|
||||
paper_repo = PaperRepo(settings.operational_dsn)
|
||||
paper_repo.migrate()
|
||||
# The REAL IBKR paper-account section on the multistrat/vrp detail pages (D2.2/2.3) — a light
|
||||
# The REAL IBKR paper-account section on the multistrat detail pages (D2.2/2.3) — a light
|
||||
# schema-ensure migrate (create_all + a guarded additive ALTER), same shape as repo/paper_repo above.
|
||||
ibkr_account_repo = IbkrAccountRepo(settings.operational_dsn)
|
||||
ibkr_account_repo.migrate()
|
||||
|
||||
@@ -57,8 +57,8 @@
|
||||
{% macro rollup_plan_pill(rollup) %}{% if rollup and rollup.n_executing > 0 %}<span class="b {{ 'funded' if rollup.agg_divergence >= 0 else 'decaying' }} small"><span class="dot"></span>{{ '%.1f'|format((rollup.agg_divergence|abs) * 100) }}% {{ 'ahead of plan' if rollup.agg_divergence >= 0 else 'behind plan' }}</span>{% endif %}{% endmacro %}
|
||||
|
||||
{# ---- health_badge: the LOUD runtime-health axis (SEPARATE from the gate). A track that is STALE / NO_REF /
|
||||
BROKEN shows a red badge — visually impossible to confuse with a healthy WAIT — so a stale recompute (the
|
||||
2-week vrp rot) surfaces on day 1. HEALTHY renders nothing. #}
|
||||
BROKEN shows a red badge — visually impossible to confuse with a healthy WAIT — so a stale recompute
|
||||
(a strategy silently rotting for weeks) surfaces on day 1. HEALTHY renders nothing. #}
|
||||
{% macro health_badge(f) %}{% if f.health and f.health != 'HEALTHY' %}<span class="badge-health {{ f.health }}" style="background:#c0392b;color:#fff;font-size:9px;font-weight:600;border-radius:3px;padding:1px 5px;margin-left:4px;letter-spacing:.03em" title="{{ f.health_reason }}">{% if f.health == 'STALE' %}STALE {{ f.health_stale_days }}d{% elif f.health == 'NO_REF' %}NO REF{% else %}{{ f.health }}{% endif %}</span>{% endif %}{% endmacro %}
|
||||
|
||||
{# ---- track_row: one compact edge row. `record` is the meaningful number (backfill/backtest ref, else
|
||||
|
||||
@@ -75,7 +75,7 @@
|
||||
{# ---- D3 (Task 4): the REAL IBKR paper-account "real trades" sub-row — reuses the exec-twin's
|
||||
`.exrow`/`.exec-tag` style, but plain-language content (real return, click-through to holdings,
|
||||
the shared `plan_pill`, honest cost-to-trade). Replaces the generic exec-twin row below for the
|
||||
two IBKR real-account books (multistrat/vrp); the crypto book's own exec-twin row is unaffected. #}
|
||||
IBKR real-account book (multistrat); the crypto book's own exec-twin row is unaffected. #}
|
||||
{# The sub-row BORROWS the parent table's 6 columns, so each cell is self-labeled inline (like the
|
||||
exec-twin row below) — the real return in particular must read as a return, not sit naked under the
|
||||
"allocated" header. data-label carries the true meaning for the mobile card reflow, not the borrowed
|
||||
|
||||
@@ -1,59 +0,0 @@
|
||||
"""A tiny, dependency-light Black-Scholes European option pricer (r=0, no dividends — crypto convention).
|
||||
|
||||
Used by the DEFINED-RISK VRP evaluator to price the legs of a short-straddle-plus-protective-wings spread
|
||||
from the DVOL implied-vol index (we have NO historical option chains, so we synthesize prices via BS from the
|
||||
ATM IV — the standard vol-backtest method; the modelling caveats are documented in `vrp_defined_risk_eval`).
|
||||
|
||||
Conventions
|
||||
-----------
|
||||
* European options, r = 0 (crypto has no risk-free carry to speak of; forward ≈ spot), no dividends.
|
||||
* `sigma` is the ANNUALISED vol (a fraction, e.g. 0.60 for 60%); `T` is the time to expiry in YEARS.
|
||||
* The standard-normal CDF uses `math.erf` (no external dependency) — exact to float precision. (scipy's
|
||||
`norm.cdf` agrees to ~1e-15; we deliberately avoid the import so the pricer is pure-stdlib.)
|
||||
|
||||
The only formulas:
|
||||
d1 = (ln(S/K) + 0.5·σ²·T) / (σ·√T) (r=0)
|
||||
d2 = d1 − σ·√T
|
||||
call = S·N(d1) − K·N(d2)
|
||||
put = K·N(−d2) − S·N(−d1) (= call − S + K, put-call parity at r=0)
|
||||
|
||||
Degenerate inputs (T≤0 or σ≤0) collapse to intrinsic value max(S−K,0) / max(K−S,0), which is the correct
|
||||
zero-time / zero-vol limit and keeps the pricer total (never raises on an expiring/zero-vol leg)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
|
||||
def _norm_cdf(x: float) -> float:
|
||||
"""Standard-normal CDF via the error function: N(x) = 0.5·(1 + erf(x/√2)). Pure stdlib, no scipy."""
|
||||
return 0.5 * (1.0 + math.erf(float(x) / math.sqrt(2.0)))
|
||||
|
||||
|
||||
def _d1_d2(s: float, k: float, t: float, sigma: float) -> tuple[float, float]:
|
||||
"""The Black-Scholes d1, d2 (r=0). Assumes s,k>0 and t,sigma>0 (callers handle the degenerate cases)."""
|
||||
vol_sqrt_t = sigma * math.sqrt(t)
|
||||
d1 = (math.log(s / k) + 0.5 * sigma * sigma * t) / vol_sqrt_t
|
||||
d2 = d1 - vol_sqrt_t
|
||||
return d1, d2
|
||||
|
||||
|
||||
def bs_call(*, s: float, k: float, t: float, sigma: float) -> float:
|
||||
"""Black-Scholes European CALL price (r=0). Degenerate T≤0 or σ≤0 → intrinsic max(S−K, 0). Pure."""
|
||||
s, k, t, sigma = float(s), float(k), float(t), float(sigma)
|
||||
if s <= 0.0 or k <= 0.0:
|
||||
return 0.0
|
||||
if t <= 0.0 or sigma <= 0.0:
|
||||
return max(s - k, 0.0)
|
||||
d1, d2 = _d1_d2(s, k, t, sigma)
|
||||
return s * _norm_cdf(d1) - k * _norm_cdf(d2)
|
||||
|
||||
|
||||
def bs_put(*, s: float, k: float, t: float, sigma: float) -> float:
|
||||
"""Black-Scholes European PUT price (r=0). Degenerate T≤0 or σ≤0 → intrinsic max(K−S, 0). Pure."""
|
||||
s, k, t, sigma = float(s), float(k), float(t), float(sigma)
|
||||
if s <= 0.0 or k <= 0.0:
|
||||
return 0.0
|
||||
if t <= 0.0 or sigma <= 0.0:
|
||||
return max(k - s, 0.0)
|
||||
d1, d2 = _d1_d2(s, k, t, sigma)
|
||||
return k * _norm_cdf(-d2) - s * _norm_cdf(-d1)
|
||||
@@ -27,7 +27,7 @@ from fxhnt.registry import STRATEGY_REGISTRY
|
||||
# tagged under these MODELED sids — see `multistrat_exec_record.plan_and_record`'s
|
||||
# `persist_execution(..., strategy_id="multistrat", ...)`). Their detail page gets the holdings/recent-
|
||||
# trades/account-nav/recon section (D2.2/2.3); every other strategy simply doesn't have that data.
|
||||
_IBKR_ACCOUNT_SIDS = frozenset({"multistrat", "vrp"})
|
||||
_IBKR_ACCOUNT_SIDS = frozenset({"multistrat"})
|
||||
|
||||
|
||||
def _expected_book_symbols() -> frozenset[str]:
|
||||
@@ -38,7 +38,13 @@ def _expected_book_symbols() -> frozenset[str]:
|
||||
from fxhnt.domain.strategies.multistrat import FUND_INSTRUMENTS
|
||||
syms = set(FUND_INSTRUMENTS)
|
||||
try:
|
||||
syms |= {listing.ticker for listing in get_settings().ucits.map.values()}
|
||||
# Include BOTH the cosmetic ticker AND the symbol IBKR actually REPORTS the holding under
|
||||
# (`ibkr_symbol`, which diverges for the IEF line: ticker CBU0 -> reported CSBGU0, probe 2026-07-20).
|
||||
# account_snapshot reads `p.contract.symbol`, so a legitimate book holding IBKR reports as CSBGU0 must
|
||||
# be in this set or it is wrongly flagged a stray and dropped from the book's NLV.
|
||||
for listing in get_settings().ucits.map.values():
|
||||
syms.add(listing.ticker)
|
||||
syms.add(listing.ibkr_symbol)
|
||||
except Exception: # noqa: BLE001 — config-less test path: the US instruments alone still guard the check
|
||||
pass
|
||||
return frozenset(syms)
|
||||
@@ -46,7 +52,7 @@ def _expected_book_symbols() -> frozenset[str]:
|
||||
|
||||
def _active_ibkr_account_sids() -> list[str]:
|
||||
"""The IBKR-account sids that are still cockpit-visible — `_IBKR_ACCOUNT_SIDS` minus any ARCHIVED entry
|
||||
(`STRATEGY_REGISTRY` `archived: True`, e.g. shelved/FALSIFIED vrp). Sorted for a deterministic render
|
||||
(`STRATEGY_REGISTRY` `archived: True`, e.g. a shelved/FALSIFIED strategy). Sorted for a deterministic render
|
||||
order. Every account-wide aggregation (the per-strategy breakdown, the recent-fills roll-up) iterates
|
||||
THIS, so an archived book never surfaces a row/fill on the IBKR account page even though its fills are
|
||||
still captured in the repo."""
|
||||
@@ -80,7 +86,7 @@ class DashboardService:
|
||||
def __init__(self, repo: ForwardNavRepo, ibkr_repo: IbkrAccountRepo | None = None) -> None:
|
||||
self._repo = repo
|
||||
# Optional: absent in most callers/tests (the ordinary crypto/registry tracks never touch it); when
|
||||
# given, powers the IBKR real-account section on multistrat/vrp's detail pages (D2.2/2.3).
|
||||
# given, powers the IBKR real-account section on multistrat's detail pages (D2.2/2.3).
|
||||
self._ibkr_repo = ibkr_repo
|
||||
|
||||
def _safe_allocs(self) -> dict:
|
||||
@@ -152,10 +158,11 @@ class DashboardService:
|
||||
b = backtests.get(reg.strategy_id) # most strategies have no backtest verdict -> None
|
||||
meta = STRATEGY_REGISTRY.get(reg.strategy_id, {}) # honest not-live flag lives in the Python registry
|
||||
if meta.get("archived"): # defense-in-depth: `registry()` already excludes archived rows (the
|
||||
continue # authoritative filter), so an archived sleeve (vrp/vrp_exec) never reaches here.
|
||||
continue # authoritative filter), so an archived sleeve never reaches here.
|
||||
min_days = _gate_min_days(reg.gate_spec) # warming = fewer forward days than the gate needs
|
||||
out.append(FleetRow(
|
||||
strategy_id=reg.strategy_id, display_name=display_name(reg.strategy_id, reg.display_name), sleeve=reg.sleeve,
|
||||
strategy_id=reg.strategy_id, sleeve=reg.sleeve,
|
||||
display_name=display_name(reg.strategy_id, reg.display_name),
|
||||
days=s.days if s else 0,
|
||||
total_return_pct=100.0 * s.total_return if s else 0.0,
|
||||
sharpe=s.sharpe if s else 0.0, maxdd=s.maxdd if s else 0.0,
|
||||
@@ -380,7 +387,7 @@ class DashboardService:
|
||||
# silently carried in the account NLV — surfaced as a warning in the holdings view.
|
||||
"in_book": sym in expected}
|
||||
for sym, qty in positions.items()]
|
||||
# GROSS exposure denominator: a defined-risk book (e.g. vrp put-credit-spreads = short put + long
|
||||
# GROSS exposure denominator: a defined-risk book (e.g. a put-credit-spread book = short put + long
|
||||
# wing) holds mixed-sign legs, so a signed total nets toward the small net credit and makes "% of
|
||||
# book" blow past 100% / go negative. % of GROSS keeps every row's signed value but a sane [0,100]
|
||||
# share that sums to ~100.
|
||||
@@ -406,8 +413,8 @@ class DashboardService:
|
||||
"stray_holdings": stray_holdings}
|
||||
|
||||
def detail(self, strategy_id: str) -> StrategyDetail | None:
|
||||
# An ARCHIVED sleeve (STRATEGY_REGISTRY `archived: True` — e.g. shelved/FALSIFIED vrp/vrp_exec) must
|
||||
# never render its detail page: a direct hit to `/strategy/vrp` returns None so the route 404s. This
|
||||
# An ARCHIVED sleeve (STRATEGY_REGISTRY `archived: True` — e.g. a shelved/FALSIFIED strategy) must
|
||||
# never render its detail page: a direct hit to its `/strategy/<id>` returns None so the route 404s. This
|
||||
# is belt-and-suspenders — `registry()` below already excludes archived rows (single source) — but it
|
||||
# keeps the hide explicit and independent of the repo filter.
|
||||
if STRATEGY_REGISTRY.get(strategy_id, {}).get("archived"):
|
||||
@@ -428,7 +435,8 @@ class DashboardService:
|
||||
periods = {p: aggregate(dates, rets, p) for p in PERIODS} # raw daily/weekly/monthly, no normalization
|
||||
pair = self._exec_pair(strategy_id, summaries)
|
||||
return StrategyDetail(
|
||||
strategy_id=strategy_id, display_name=display_name(strategy_id, reg.display_name), sleeve=reg.sleeve, venue=reg.venue,
|
||||
strategy_id=strategy_id, sleeve=reg.sleeve, venue=reg.venue,
|
||||
display_name=display_name(strategy_id, reg.display_name),
|
||||
days=s.days if s else 0, total_return_pct=100.0 * s.total_return if s else 0.0,
|
||||
sharpe=s.sharpe if s else 0.0, maxdd=s.maxdd if s else 0.0,
|
||||
gate_status=s.gate_status if s else "WAIT",
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""display_names.py — the SINGLE internal-id -> human-readable name map used by EVERY cockpit view (the
|
||||
plain-language global constraint: no raw registry id — `multistrat`/`vrp`/`bybit_4edge` — ever reaches an
|
||||
plain-language global constraint: no raw registry id — `multistrat`/`bybit_4edge` — ever reaches an
|
||||
end user). Absorbs the interim `_SIM_BOOK_LABELS` local map that lived in `app.py` (Task 1) for the
|
||||
sim/Backtest book pills — those two overrides now live here as the SSOT, and `app.py` reads through this
|
||||
module instead of its own copy.
|
||||
@@ -48,7 +48,7 @@ def display_name(strategy_id: str, fallback: str | None = None) -> str:
|
||||
|
||||
def asset_description(symbol: str) -> str:
|
||||
"""Plain 'what it is' description for an asset symbol (e.g. SPY -> 'US stocks'). An UNMAPPED symbol (a
|
||||
future vrp option contract, say) falls back to a plain generic 'other' rather than echoing a raw/odd
|
||||
future option contract, say) falls back to a plain generic 'other' rather than echoing a raw/odd
|
||||
ticker string — the symbol itself is still shown in its own column, so nothing is lost. Never blank,
|
||||
never raises."""
|
||||
return ASSET_DESCRIPTIONS.get(symbol, "other")
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
"""Ingest the Deribit DVOL implied-vol index into the Bybit-namespaced feature store (table='bybit_features')
|
||||
as feature 'dvol' for the perp symbols BTCUSDT / ETHUSDT.
|
||||
|
||||
The DVOL index is currency-keyed (BTC/ETH); the warehouse panels are symbol-keyed (the VRP eval reads `dvol`
|
||||
alongside the BTCUSDT/ETHUSDT `close`), so the ingest maps currency -> "<CCY>USDT" and writes feature='dvol'
|
||||
at ts = epoch_day*86400. ONLY 'dvol' is written — close/funding are the kline/funding ingests' job (the PK is
|
||||
(feature, symbol, ts) so distinct features never collide). Idempotent (upsert overwrites in place, so re-runs
|
||||
are safe). Per-currency skip-on-empty: a currency that returns no rows contributes nothing and never aborts.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Protocol
|
||||
|
||||
# Deribit DVOL currency -> the Bybit USDT-perp symbol the warehouse panels key on.
|
||||
_CCY_TO_SYMBOL = {"BTC": "BTCUSDT", "ETH": "ETHUSDT"}
|
||||
|
||||
|
||||
class _Deribit(Protocol):
|
||||
def dvol_history(self, currency: str, *, start_ms: int | None = None) -> dict[int, float]: ...
|
||||
|
||||
|
||||
def ingest_dvol(store: Any, deribit: _Deribit, currencies: tuple[str, ...] = ("BTC", "ETH"), *,
|
||||
start_ms: int | None = None) -> dict[str, Any]:
|
||||
"""Fetch + write the daily DVOL index for each `currency` into `store` (feature='dvol',
|
||||
ts=epoch_day*86400, symbol="<CCY>USDT").
|
||||
|
||||
Returns {"symbols": N, "day_rows": M, "oldest_day": D|None}: N = currencies that produced >=1 day-row,
|
||||
M = total day-rows written, D = the earliest epoch_day across all currencies (None if nothing ingested).
|
||||
Per-currency skip-on-empty (a currency with no DVOL history contributes nothing, never aborts). Writes
|
||||
are bulk (one transaction); the upsert is idempotent so re-running is safe and cheap."""
|
||||
items: list[tuple[str, list[tuple[int, dict[str, float]]]]] = []
|
||||
oldest_day: int | None = None
|
||||
day_rows = 0
|
||||
for ccy in currencies:
|
||||
symbol = _CCY_TO_SYMBOL.get(ccy.upper(), f"{ccy.upper()}USDT")
|
||||
daily = deribit.dvol_history(ccy, start_ms=start_ms)
|
||||
if not daily:
|
||||
continue
|
||||
rows = [(day * 86_400, {"dvol": float(val)}) for day, val in sorted(daily.items())]
|
||||
items.append((symbol, rows))
|
||||
day_rows += len(rows)
|
||||
run_oldest = min(daily)
|
||||
oldest_day = run_oldest if oldest_day is None else min(oldest_day, run_oldest)
|
||||
if items:
|
||||
store.write_features_bulk(items)
|
||||
return {"symbols": len(items), "day_rows": day_rows, "oldest_day": oldest_day}
|
||||
@@ -3,8 +3,8 @@ LOO marginal-Sharpe gate, run BEFORE any forward tracker is wired. This is the g
|
||||
reconciliation gate on two EM-beta ETFs would PASS trivially (the forward NAV just tracks the backtest NAV)
|
||||
and prove NOTHING about edge. The real question is decision-theoretic and answerable now from ~8-10y of
|
||||
history: does adding CNYA.L / NDIA.L to the fund's ETF book RAISE the book's risk-adjusted return after
|
||||
costs, or is it a co-crashing beta sleeve that fails the marginal gate — the SAME bar that killed VRP and
|
||||
the token-unlock diversifier (see pearl_fxhnt_diversifier_hunt_2026_07_15).
|
||||
costs, or is it a co-crashing beta sleeve that fails the marginal gate — the SAME bar that killed prior failed
|
||||
diversifier candidates, including the token-unlock diversifier (see pearl_fxhnt_diversifier_hunt_2026_07_15).
|
||||
|
||||
PASS bar (predetermined, not chosen after seeing the numbers): marginal Sharpe > 0 after cost.
|
||||
> 0 -> proceed to Stage 2 (wire the `em_asia` registry sleeve + reconciliation forward gate).
|
||||
@@ -15,8 +15,8 @@ overlapping dates — CNYA.L ~10y, NDIA.L ~8y. Per-name verdicts (not a cross-na
|
||||
windows are acceptable.
|
||||
|
||||
Two marginal measures are reported per name:
|
||||
1. `marginal_sharpe` (SSOT weight-blend, domain/diversification/marginal.py) — DIRECTLY comparable to the
|
||||
prior VRP / unlock verdicts, which used this exact primitive.
|
||||
1. `marginal_sharpe` (SSOT weight-blend, domain/diversification/marginal.py) — DIRECTLY comparable to
|
||||
prior failed diversifier candidates' verdicts (e.g. unlock), which used this exact primitive.
|
||||
2. In-book delta-Sharpe — `book_series(FUND + candidate) - book_series(FUND)` through the FULL adaptive
|
||||
machinery (vol-norm -> trust -> correlation de-risk), i.e. the candidate's effect where it would
|
||||
actually live. The truer measure; if the two disagree, that disagreement is itself signal.
|
||||
@@ -68,7 +68,7 @@ class CandidateVerdict:
|
||||
window_start: str
|
||||
window_end: str
|
||||
n_days: int
|
||||
marginal_sharpe_blend: float # SSOT weight-blend, cost-inclusive (comparable to VRP/unlock)
|
||||
marginal_sharpe_blend: float # SSOT weight-blend, cost-inclusive (comparable to prior candidates/unlock)
|
||||
delta_sharpe_inbook: float # book_series(FUND+cand) - book_series(FUND), cost-inclusive
|
||||
corr_to_book: float
|
||||
crash_drawdowns: dict[str, float] = field(default_factory=dict)
|
||||
|
||||
@@ -9,7 +9,7 @@ module does NO new compute, only comparison, plus a read of the strategy-tagged
|
||||
cost-to-trade inputs (median slippage, total fees).
|
||||
|
||||
`has_exec` is the gate: when the `<sid>_exec` track has no persisted forward_summary row yet (a real account
|
||||
that hasn't run, or an exec twin that's suspended — e.g. `vrp_exec` today), there is nothing to reconcile
|
||||
that hasn't run, or an exec twin that's suspended pending a fund decision), there is nothing to reconcile
|
||||
against — every other field defaults to zero rather than computing on unavailable data."""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -1,19 +1,21 @@
|
||||
"""Runtime staleness/health axis — the anti-silent-rot backbone.
|
||||
|
||||
WHY: vrp was silently broken for 2 weeks because a data failure degraded QUIETLY — it recomputed off a
|
||||
frozen-but-stale `xsp_option_bars` table, its forward "updated" every night, and only a log line marked the
|
||||
rot. The gate never noticed (a stale track just sits as a quiet WAIT). This module adds a HEALTH axis that is
|
||||
WHY: a dead/archived strategy was once silently broken for 2 weeks because a data failure degraded
|
||||
QUIETLY — it recomputed off a frozen-but-stale `xsp_option_bars` table, its forward "updated" every night,
|
||||
and only a log line marked the rot. The gate never noticed (a stale track just sits as a quiet WAIT). This
|
||||
module adds a HEALTH axis that is
|
||||
SEPARATE from the gate: a track can be legitimately WAIT/building AND healthy; STALE/NO_REF/BROKEN is
|
||||
orthogonal. Any live/gated track that stops updating surfaces LOUDLY on day 1 (red cockpit badge + ERROR log),
|
||||
not a footnote two weeks later.
|
||||
|
||||
The health-check line is `record_mode`, which faithfully encodes "is this track EXPECTED to advance nightly":
|
||||
- `recompute` tracks (sixtyforty/xsfunding/unlock/multistrat/vrp/bybit_4edge/…): the engine recomputes from
|
||||
- `recompute` tracks (sixtyforty/xsfunding/unlock/multistrat/bybit_4edge/…): the engine recomputes from
|
||||
fresh source data EVERY run, so a non-advancing `as_of` (or a frozen freeze table) is a real STALE fault.
|
||||
vrp lives here → the incident is caught.
|
||||
- `observed` tracks (the executed twins multistrat_exec/vrp_exec/bybit_4edge_exec): advance ONLY when a
|
||||
human/executor books a fill, so they are legitimately DORMANT when execution is suspended (vrp_exec has no
|
||||
IBKR options permission yet). Staleness/no-ref do NOT apply — they must not false-alarm. This is the
|
||||
A `recompute` track lives here → an incident like the one above is caught.
|
||||
- `observed` tracks (the executed twins multistrat_exec/bybit_4edge_exec): advance ONLY when a
|
||||
human/executor books a fill, so they are legitimately DORMANT when execution is suspended (an executed
|
||||
twin can lack IBKR options permission, for example). Staleness/no-ref do NOT apply — they must not
|
||||
false-alarm. This is the
|
||||
"intentionally-dormant reference_only" carve-out. (BROKEN — a health-compute error — still applies, since
|
||||
a track whose health we cannot even determine is not healthy.)
|
||||
|
||||
@@ -21,14 +23,14 @@ States (worst wins, precedence BROKEN > STALE > NO_REF > HEALTHY):
|
||||
- HEALTHY — advancing within threshold, or dormant-by-design, or a GENUINELY new recompute track (its active
|
||||
`forward_anchor.t0` is within the threshold, so "no data yet" is expected, not rot).
|
||||
- STALE(n) — a recompute track's source `as_of` OR its freeze table's max date is > threshold business days
|
||||
behind the run date. The freeze axis catches the exact vrp failure mode: forward "updates" yet the frozen
|
||||
input table is stale. Also STALE (not the HEALTHY "no data yet" carve-out): a recompute track with ZERO
|
||||
behind the run date. The freeze axis catches the exact failure mode described above: forward "updates"
|
||||
yet the frozen input table is stale. Also STALE (not the HEALTHY "no data yet" carve-out): a recompute track with ZERO
|
||||
forward data whose anchor `t0` is OLDER than the threshold (broken-from-inception — never advanced since
|
||||
it was created), or which has no active anchor at all (should have one; a live track with no anchor is a
|
||||
real signal, never silently HEALTHY).
|
||||
- NO_REF — a reconciliation-gated recompute track with NO resolvable backtest reference: the gate can NEVER
|
||||
reconcile (it silently degrades to the min-window/clean-exec checks only). The M6 static invariant that
|
||||
missed vrp, made a real runtime check.
|
||||
missed the incident described above, made a real runtime check.
|
||||
- BROKEN — the PER-TRACK health computation raised (e.g. a malformed registry entry for that strategy):
|
||||
surfaced for that track only, not swallowed. This does NOT cover the repo-wide reads `evaluate_health`
|
||||
does once up front (`migrate`/`all_summaries`/`all_backtest_summaries`/`xsp_freeze_max_date`) — those are
|
||||
@@ -144,7 +146,7 @@ def evaluate_health(dsn: str, at: dt.datetime | None = None) -> list[HealthVerdi
|
||||
summaries = {s.strategy_id: s for s in repo.all_summaries()}
|
||||
backtest_sids = {b.strategy_id for b in repo.all_backtest_summaries()}
|
||||
ref_alias = CockpitBacktestRefProvider._REF_ALIAS # the SAME alias the gate uses to resolve its ref
|
||||
freeze_max = repo.xsp_freeze_max_date() # one query for the vrp freeze axis
|
||||
freeze_max = repo.xsp_freeze_max_date() # one query for the frozen-input freeze axis
|
||||
|
||||
verdicts: list[HealthVerdict] = []
|
||||
for sid, meta in STRATEGY_REGISTRY.items():
|
||||
|
||||
@@ -42,7 +42,7 @@ class CockpitBacktestRefProvider:
|
||||
# A forward track whose reconciliation reference is ANOTHER track's backtest (the executed multistrat
|
||||
# track reconciles against the MODELED multistrat's basis-matched reference — spec C1 §4: execution decay
|
||||
# shows as the executed track diverging from the modeled book's expectation).
|
||||
_REF_ALIAS = {"multistrat_exec": "multistrat", "vrp_exec": "vrp", "bybit_4edge_exec": "bybit_4edge"}
|
||||
_REF_ALIAS = {"multistrat_exec": "multistrat", "bybit_4edge_exec": "bybit_4edge"}
|
||||
|
||||
def __init__(self, repo: ForwardNavRepo) -> None:
|
||||
self._cagr = {b.strategy_id: float(b.cagr) for b in repo.all_backtest_summaries()}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""Dispatch a sim book's honest-cost curve by its registry backtest.kind — bybit books read the precomputed
|
||||
bybit_sleeve_ret table; recompute-replay books (multistrat/vrp, the IBKR paper strategies) read their own
|
||||
bybit_sleeve_ret table; recompute-replay books (multistrat, the IBKR paper strategies) read their own
|
||||
precomputed curve. The sim NEVER recomputes an edge on the request path (both sources are nightly-precomputed:
|
||||
bybit_sleeve_ret by `_persist_bybit_book`, the IBKR curve by `_persist_track_backtest_ref`)."""
|
||||
from __future__ import annotations
|
||||
@@ -9,17 +9,17 @@ from collections.abc import Callable
|
||||
from fxhnt.application.backtest_refs import registry_backtest_kind
|
||||
|
||||
# The Backtest page's book pills, in render order. bybit_4edge/bybit_4edge_levered are the existing Bybit
|
||||
# books (report-kind, precomputed bybit_sleeve_ret). multistrat/vrp are the IBKR paper strategies
|
||||
# books (report-kind, precomputed bybit_sleeve_ret). multistrat is the IBKR paper strategy
|
||||
# (recompute-replay-kind) added by Task 1/D1 — their curve is a precomputed per-day return series read the
|
||||
# same cheap way (see `sim_returns_for`'s dispatch below).
|
||||
SIM_BOOKS: tuple[str, ...] = ("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered", "vrp")
|
||||
SIM_BOOKS: tuple[str, ...] = ("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered")
|
||||
|
||||
|
||||
def sim_returns_for(book: str, *, bybit_returns: Callable[[], dict[str, dict[int, float]]],
|
||||
ibkr_curve: Callable[[str], dict[str, dict[int, float]]]
|
||||
) -> dict[str, dict[int, float]]:
|
||||
"""`book`'s per-sleeve honest-cost return series — `{sleeve: {epoch_day: ret}}` — dispatched by the
|
||||
registry's `backtest.kind` for `book`: `recompute-replay` (multistrat/vrp) reads the precomputed IBKR
|
||||
registry's `backtest.kind` for `book`: `recompute-replay` (multistrat) reads the precomputed IBKR
|
||||
curve via `ibkr_curve(book)`; everything else (the `report`-kind Bybit books) reads the precomputed
|
||||
`bybit_sleeve_ret` table via `bybit_returns()` — the existing path, unchanged."""
|
||||
kind = registry_backtest_kind(book)
|
||||
|
||||
@@ -1,167 +0,0 @@
|
||||
"""The defined-risk XSP put-credit-spread VRP sleeve (recompute mode). Recomputes the daily book ENTIRELY off
|
||||
frozen `xsp_option_bars` rows — put-call-parity forward -> IV inversion -> ~20Δ short strike, long a fixed
|
||||
width below -> smooth MODEL-mark daily P&L per dollar of defined-risk margin (the daily value is rebuilt from
|
||||
the day's forward + ATM IV via the shared `vrp_marking` valuation — the same one the live exec twin uses for
|
||||
its profit-take signal — NOT a difference of two individually-noisy raw leg quotes). Never calls OPRA. Mirrors
|
||||
how MultiStratStrategy produces both the full-history ref (advance(None,{})) and the nightly forward window."""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
from typing import Any
|
||||
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
from fxhnt.application import vrp_marking, vrp_pricing
|
||||
|
||||
|
||||
class VrpStrategy:
|
||||
def __init__(self, bars: XspOptionBarsRepo, *, dte_lo: int = 20, dte_hi: int = 45, dte_target: int = 30,
|
||||
short_delta: float = 0.20, width_points: float = 5.0, ladder: int = 5,
|
||||
profit_take: float = 0.5, entry_weekday: int = 0) -> None:
|
||||
self._bars = bars
|
||||
self._dte_lo, self._dte_hi, self._dte_target = dte_lo, dte_hi, dte_target
|
||||
self._short_delta, self._width, self._ladder = short_delta, width_points, ladder
|
||||
self._profit_take, self._entry_weekday = profit_take, entry_weekday
|
||||
self.margin = width_points * 100.0
|
||||
|
||||
@staticmethod
|
||||
def _dte(day: str, expiry: dt.date) -> int:
|
||||
return (expiry - dt.date.fromisoformat(day)).days
|
||||
|
||||
def _select_spread(self, day: str) -> dict | None:
|
||||
"""Pick the ~30-DTE, ~20Δ short put + long a width below, priced from frozen marks on `day`."""
|
||||
chain = self._bars.chain_asof(day, self._dte_lo, self._dte_hi)
|
||||
if not chain:
|
||||
return None
|
||||
# nearest expiry to dte_target; ties broken by lexicographic date order (deterministic across
|
||||
# processes) rather than `set` iteration order, which is PYTHONHASHSEED-salted.
|
||||
expiries = sorted(dict.fromkeys(c.expiry for c in chain))
|
||||
exp = min(expiries, key=lambda e: abs(self._dte(day, e) - self._dte_target))
|
||||
puts = sorted((c for c in chain if c.expiry == exp), key=lambda c: c.strike)
|
||||
tau = max(self._dte(day, exp), 1) / 365.0
|
||||
marks = self._bars.bars_for([c.osi_symbol for c in puts], day, day)
|
||||
priced = [(c, marks.get(c.osi_symbol, {}).get(day)) for c in puts]
|
||||
priced = [(c, p) for c, p in priced if p is not None and p > 0.0]
|
||||
if len(priced) < 2:
|
||||
return None
|
||||
forward, atm_iv = vrp_marking.forward_and_atm_iv(self._bars, day, exp, tau)
|
||||
if forward is None:
|
||||
# FALLBACK: no common strike with both a priced put AND a priced call for this expiry
|
||||
# (degenerate/missing data) — approximate the forward with the highest surviving put strike,
|
||||
# the closest available proxy to ATM.
|
||||
forward = priced[-1][0].strike
|
||||
# choose short strike whose |delta| is nearest short_delta
|
||||
best = None
|
||||
for c, price in priced:
|
||||
iv = vrp_pricing.implied_vol_put(price, forward, c.strike, tau)
|
||||
if iv is None:
|
||||
continue
|
||||
dlt = vrp_pricing.put_delta(forward, c.strike, iv, tau)
|
||||
score = abs(dlt - self._short_delta)
|
||||
if best is None or score < best[0]:
|
||||
best = (score, c, price, iv)
|
||||
if best is None:
|
||||
return None
|
||||
_, short_c, short_price, short_iv = best
|
||||
long_strike = short_c.strike - self._width
|
||||
long_c = next((c for c, _ in priced if abs(c.strike - long_strike) < 1e-6), None)
|
||||
long_price = None
|
||||
if long_c is not None:
|
||||
long_price = self._bars.bars_for([long_c.osi_symbol], day, day).get(long_c.osi_symbol, {}).get(day)
|
||||
if long_c is None or long_price is None:
|
||||
return None
|
||||
credit = short_price - long_price
|
||||
if credit <= 0.0:
|
||||
return None
|
||||
# seed the mark IV with a clean ATM read (fall back to the short-leg IV when the ATM inversion failed);
|
||||
# `_spread_value` re-derives F+IV each day but carries this on a degenerate day.
|
||||
return dict(entry=day, expiry=exp.isoformat(), short_osi=short_c.osi_symbol, long_osi=long_c.osi_symbol,
|
||||
short_strike=short_c.strike, long_strike=long_c.strike, credit=credit,
|
||||
last_iv=(atm_iv if atm_iv is not None else short_iv))
|
||||
|
||||
def _spread_value(self, sp: dict, day: str, day_cache: dict | None = None) -> float | None:
|
||||
"""MODEL value (cost to close) of the open spread on `day`, via the shared `vrp_marking` valuation the
|
||||
live exec twin also uses (single source): smooth Black-Scholes from the day's forward + ATM IV, clipped
|
||||
to the defined-risk bound [0, width]. None only when the expiry chain has aged out (caller closes — no
|
||||
zombie). `day_cache` memoizes (F, IV) across spreads sharing an expiry in one day's pass."""
|
||||
return vrp_marking.model_spread_value(self._bars, sp, day, self._width, day_cache)
|
||||
|
||||
def _settlement_value(self, sp: dict, day: str, day_cache: dict | None = None) -> float | None:
|
||||
"""Terminal settlement value clip(K_short − F, 0, width) from `day`'s forward — the shared
|
||||
`vrp_marking.settlement_value`, realized on the close day (DTE≤1) instead of the theta model mark."""
|
||||
return vrp_marking.settlement_value(self._bars, sp, day, self._width, day_cache)
|
||||
|
||||
def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]:
|
||||
opens: list[dict] = list(extra.get("open_spreads", []))
|
||||
rows: list[tuple[str, float]] = []
|
||||
# Replay is read-heavy over every trading day after `last_date`; preload that whole slice in ONE
|
||||
# query so the per-day chain_asof/bars_for below serve from memory instead of a round-trip each
|
||||
# (the N+1 that made a full ~1600-day recompute issue thousands of queries). Scoped to `last_date`
|
||||
# so an incremental from-anchor recompute only loads the days it will actually touch.
|
||||
preload = getattr(self._bars, "preload", None)
|
||||
if callable(preload):
|
||||
preload(after=last_date)
|
||||
for day in self._bars.trading_days(last_date):
|
||||
# 1) mark open spreads -> daily defined-risk return
|
||||
pnl = 0.0
|
||||
marked = 0 # spreads that produced a mark today (pre-closure)
|
||||
still: list[dict] = []
|
||||
day_cache: dict = {} # memoize (F, IV) per (day, expiry) across rungs
|
||||
for sp in opens:
|
||||
val = self._spread_value(sp, day, day_cache)
|
||||
if val is None:
|
||||
# the expiry's chain has aged/moved out of the frozen slice (no clean forward) -- CLOSE
|
||||
# rather than zombie-hold: don't re-append to `still` and don't count it in `marked`, so a
|
||||
# spread whose marks stop no longer permanently occupies a ladder slot.
|
||||
#
|
||||
# INTENTIONAL ref-vs-live ASYMMETRY (documented, not a bug): this pure-model backtest's
|
||||
# ONLY mark IS the model value, so model==None => the rung has no mark at all => drop it.
|
||||
# The live exec twin (`vrp_exec_record.plan_vrp_ladder`) legitimately DIFFERS on the same
|
||||
# degenerate slice — a slice where a rung's own put legs price (real `_mark_spread` actual
|
||||
# mark) but no ATM call/put pair prices anywhere (model==None): the twin HOLDS such a rung
|
||||
# and manages it on the real fill mark until DTE<=1 or the model recovers, because it has
|
||||
# strictly MORE information (an actual close price) than this recompute has. Making the two
|
||||
# consistent would mean either giving the backtest a raw-leg fallback (re-introducing the
|
||||
# exact mark noise this whole fix removed) or making the twin discard a real fill mark it
|
||||
# holds — both regressions. The asymmetry is correct; the twin never has LESS info.
|
||||
continue
|
||||
marked += 1
|
||||
# 2) management: 50% profit (on the smooth model_val) or expiry-1. On the expiry close day,
|
||||
# realize the terminal SETTLEMENT (intrinsic in-the-moneyness capped at width) rather than the
|
||||
# theta-model mark -- the put-credit-spread's true P&L at expiry.
|
||||
expiry = dt.date.fromisoformat(sp["expiry"])
|
||||
closing = val <= (1.0 - self._profit_take) * sp["credit"] or self._dte(day, expiry) <= 1
|
||||
if self._dte(day, expiry) <= 1:
|
||||
settle = self._settlement_value(sp, day, day_cache)
|
||||
if settle is not None:
|
||||
val = settle
|
||||
prev = sp.get("last_val", sp["credit"]) # baseline = entry-day model value (see entry below)
|
||||
# DEFINED-RISK BOUND: value is clipped to [0, width] (in `_spread_value`) or is the settlement
|
||||
# intrinsic clip(K_short−F, 0, width). The daily increments telescope to a per-spread cumulative
|
||||
# P&L of (entry_model_value − value)·100 dollars; the ABSOLUTE defined-risk bound [−width·100,
|
||||
# +width·100] holds unconditionally, and on a clean BS surface (entry model value == credit) it
|
||||
# tightens to exactly [−(width−credit)·100, +credit·100]. No day (nor the total) breaches it.
|
||||
# option marks are per-share; x100 = the contract multiplier, converting the point-change
|
||||
# into DOLLARS so the numerator matches self.margin's dollar basis (width_points*100) --
|
||||
# short-vol: value falling = profit.
|
||||
pnl += (prev - val) * 100.0
|
||||
sp["last_val"] = val
|
||||
if closing:
|
||||
continue # closed (drop from book)
|
||||
still.append(sp)
|
||||
opens = still
|
||||
# denominator = spreads MARKED this day (before closure), matching pnl's numerator basis —
|
||||
# NOT the post-closure survivor count, which would asymmetrically inflate the return on
|
||||
# closure days.
|
||||
rows.append((day, pnl / (max(1, marked) * self.margin)))
|
||||
# 3) weekly entry. Baseline the P&L at the MODEL value on the entry day (not the raw
|
||||
# short_close−long_close credit): every daily return is then the change in the SMOOTH model value,
|
||||
# so the first mark day can't inject the noisy far-OTM leg marks that the raw credit carries. The
|
||||
# raw `credit` is still the profit-take threshold and the defined-risk reference; on a clean
|
||||
# (BS-consistent) surface the entry model value equals the credit, so the bound is unchanged.
|
||||
if dt.date.fromisoformat(day).weekday() == self._entry_weekday and len(opens) < self._ladder:
|
||||
sp = self._select_spread(day)
|
||||
if sp is not None:
|
||||
entry_val = self._spread_value(sp, day, day_cache)
|
||||
sp["last_val"] = entry_val if entry_val is not None else sp["credit"]
|
||||
opens.append(sp)
|
||||
return rows, {"open_spreads": opens}
|
||||
@@ -1,430 +0,0 @@
|
||||
"""READ-ONLY, TAIL-HONEST evaluator for the DEFINED-RISK VRP — the *deployable* form of the vol-risk premium.
|
||||
|
||||
Why this module exists (the naked VRP failed the tail)
|
||||
------------------------------------------------------
|
||||
`vrp_eval.py` confirmed a REAL gross VRP edge (a short-variance swap: Sharpe ~1, OOS positive) but it FAILS
|
||||
the deploy gate because the NAKED short-vol payoff has an UNBOUNDED left tail — a vol spike drove the modelled
|
||||
maxDD to ~−74% and realistic OPTION costs (50–100bp) ate the rest. Selling naked variance is uninsurable.
|
||||
|
||||
The textbook fix is to CAP the tail: sell the at-the-money straddle (collect the premium) but BUY out-of-the-
|
||||
money WINGS (a protective strangle) so the maximum loss is BOUNDED — an iron-condor / straddle-with-wings.
|
||||
This module prices that spread from DVOL via Black-Scholes and re-runs the SAME tail-aware harness as the
|
||||
naked evaluator, so the two are directly comparable. The whole point is the maxDD: the wings should keep the
|
||||
defined-risk curve well above the −40% floor (ideally < −20%) where the naked −74% failed.
|
||||
|
||||
The construction (defined-risk short-vol, per asset, rolling `swap_days`=30, entered day t)
|
||||
------------------------------------------------------------------------------------------
|
||||
* Spot S = close_t. ATM IV = DVOL_t/100 (annualised, CAUSAL — known at entry). T = swap_days/365. r=0.
|
||||
* SELL the ATM straddle: 1 call + 1 put struck at K≈S, priced BS(S, S, T, IV) → COLLECT this premium.
|
||||
* BUY OTM wings (a strangle) at K_up = S·(1+w), K_dn = S·(1−w), where `w` = wing width. `w` may be given
|
||||
directly as a fraction (`wing_width`) OR derived from the ATM std-move (`wing_in_atm_std`·IV·√T) — the
|
||||
latter is the natural "N sigma" placement. PAY the wing premium BS at the wing strikes.
|
||||
* NET PREMIUM (the defined-risk credit) = straddle_premium − strangle_premium (per 1 unit of spot notional,
|
||||
expressed as a FRACTION of S so it composes with the daily-return ladder).
|
||||
* AT EXPIRY (t+swap_days), realized move m = close_{t+swap_days}/S − 1. The position P&L (as a fraction of S):
|
||||
pnl = net_premium − |m| (the short straddle pays the realized |move|)
|
||||
+ max(|m| − w, 0) (the wings refund everything BEYOND the wing distance w)
|
||||
so the loss is BOUNDED below at net_premium − w (= −(w − net_premium), the DEFINED max loss). A huge
|
||||
move can NEVER lose more than the wing distance net of the credit — that is the entire thesis.
|
||||
|
||||
Skew (crypto puts are richer) — the honesty caveat
|
||||
--------------------------------------------------
|
||||
We have only the ATM DVOL, not a full smile. Crypto put wings trade at a HIGHER IV than ATM (put skew), so a
|
||||
short-condor pays MORE for its put wing than flat-ATM pricing implies. We model this with a `put_skew_vol`
|
||||
bump (in vol points, default +5): the put legs (short ATM put + long put wing) are priced at IV + bump/100.
|
||||
The long put wing costs more (reduces our credit, conservative) AND the short ATM put collects more (raises
|
||||
credit). The NET sign of a uniform put-IV bump on a vertical is small; the bigger honesty point is the
|
||||
OPPOSITE choice: pricing the put wing at FLAT ATM IV (skew bump = 0) UNDER-prices the protection we buy →
|
||||
OVER-states the credit → the modelled result is then mildly OPTIMISTIC. We therefore default to a non-zero
|
||||
put-skew bump and DOCUMENT that bump=0 is the optimistic bound. Either way the price is a BS approximation
|
||||
from a single ATM vol, not a real chain — a modelling caveat the deploy decision must weigh alongside the
|
||||
cross-venue caveat (signal = Deribit DVOL; execution = Bybit options).
|
||||
|
||||
Reuse
|
||||
-----
|
||||
The book ladder, vol-target sizing, cost model, verify / walk-forward / look-ahead / correlation-vs-4-edges /
|
||||
per-year / worst-day / maxDD / tail-aware PASS-FAIL gate are REUSED from `vrp_eval`: the `_defined_risk_book`
|
||||
context manager temporarily binds `vrp_eval._book_short_var` to the capped-spread ladder (with the wing params
|
||||
closed over), so the entire harness runs verbatim on the defined-risk daily P&L instead of the variance-swap
|
||||
P&L — the defined-risk verdict is computed by the exact same judge as the naked one.
|
||||
|
||||
READ-ONLY: no store.write_* — the live paper_* tables are never touched.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import math
|
||||
import statistics as st
|
||||
from typing import Any
|
||||
|
||||
from fxhnt.application import vrp_eval
|
||||
from fxhnt.application.black_scholes import bs_call, bs_put
|
||||
from fxhnt.application.vrp_eval import (
|
||||
_DIRECTIONS,
|
||||
_VRP_SYMBOLS,
|
||||
_daily_close_returns,
|
||||
_FeatureStore,
|
||||
)
|
||||
|
||||
_DEFAULT_SWAP_DAYS = vrp_eval._DEFAULT_SWAP_DAYS
|
||||
_DEFAULT_TARGET_VOL = vrp_eval._DEFAULT_TARGET_VOL
|
||||
_DEFAULT_WING_WIDTH = 0.15 # default wing distance: ±15% of spot (a defined-risk condor width)
|
||||
_DEFAULT_PUT_SKEW_VOL = 5.0 # crypto put-skew bump (vol points) added to the put-leg IV; 0 = optimistic
|
||||
_N_LEGS = 4 # legs per defined-risk condor: sell call, sell put, buy call-wing, buy put-wing
|
||||
|
||||
|
||||
# --- 0. the TURNOVER / COST model — the amortized daily roll (NOT full-ladder-daily) -------------
|
||||
#
|
||||
# THE BUG (now fixed): the reused naked harness sizes the curve with `vrp_eval._sized_series`, whose cost model
|
||||
# is `haircut = k · cost_bps/1e4` charged EVERY day. That is the NAKED short-variance model — a delta-hedged
|
||||
# straddle re-established/re-hedged daily, so the WHOLE k-leverage book turns over each day. It is WRONG for the
|
||||
# defined-risk CONDOR LADDER, which holds each 4-leg spread ~`swap_days` (=30) days to EXPIRY. In that ladder:
|
||||
#
|
||||
# * exactly ONE new spread is OPENED per day (its n_legs legs are traded — the entry round-trip);
|
||||
# * exactly ONE spread EXPIRES per day — options held to expiry SETTLE, there is NO closing trade/cost;
|
||||
# * so the daily TRADED notional is (n_legs / swap_days) of the gross book's option notional — NOT the whole
|
||||
# ladder re-traded. Charging the full ladder every day over-charges turnover by swap_days/n_legs (= 30/4 ≈
|
||||
# 7.5x), which is what made the defined-risk curve falsely collapse (−5.96 @ 5.5bp, −86 @ 50bp).
|
||||
#
|
||||
# Equivalently: charge each spread its entry leg cost ONCE (n_legs · cost_bps · leg_notional, held to expiry, no
|
||||
# exit), then amortize across the swap's life — one spread opens per day, so the per-day haircut on the sized
|
||||
# (leverage-k) book is k · (n_legs / swap_days) · cost_bps/1e4. The realized per-year drag at leverage 1 is then
|
||||
# n_legs · cost_bps · (365/swap_days) ≈ 24%/yr at 50bp, ~2-3%/yr at 5.5bp — proportionate, not destroying.
|
||||
|
||||
|
||||
def _defined_risk_daily_cost(*, k: float, cost_bps: float, swap_days: int = _DEFAULT_SWAP_DAYS) -> float:
|
||||
"""The AMORTIZED daily-roll haircut for the defined-risk condor ladder, as a per-day return drag on the
|
||||
vol-targeted (leverage-`k`) book: `k · (n_legs / swap_days) · cost_bps/1e4`.
|
||||
|
||||
One spread (n_legs legs) is OPENED per day and one EXPIRES (settles — no closing trade), so the daily
|
||||
traded notional is (n_legs / swap_days) of the gross book, not the whole ladder. cost_bps is applied to the
|
||||
OPTION leg notional, consistent with the premium/P&L scaling (both fractions of spot ⇒ the leg notional that
|
||||
the credit is collected on). `swap_days<=0` ⇒ 0. Pure."""
|
||||
if not cost_bps or swap_days <= 0:
|
||||
return 0.0
|
||||
return float(k) * (_N_LEGS / float(swap_days)) * (float(cost_bps) / 1e4)
|
||||
|
||||
|
||||
def _defined_risk_sized_series(raw: dict[int, float], *, k: float, cost_bps: float,
|
||||
swap_days: int = _DEFAULT_SWAP_DAYS) -> dict[int, float]:
|
||||
"""`{day: net_ret}` = k·raw − the AMORTIZED daily-roll haircut. Mirrors `vrp_eval._sized_series` but replaces
|
||||
its full-ladder-daily cost with the defined-risk daily roll (`_defined_risk_daily_cost`). Pure."""
|
||||
haircut = _defined_risk_daily_cost(k=k, cost_bps=cost_bps, swap_days=swap_days)
|
||||
out: dict[int, float] = {}
|
||||
for d in sorted(raw):
|
||||
v = k * raw[d] - haircut
|
||||
if math.isfinite(v):
|
||||
out[int(d)] = float(v)
|
||||
return out
|
||||
|
||||
|
||||
# --- 1. the defined-risk spread: credit at entry + bounded payoff at expiry ----------------------
|
||||
|
||||
def defined_risk_credit(*, s: float, iv: float, t: float, wing_width: float,
|
||||
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL) -> float:
|
||||
"""The NET PREMIUM collected at entry for the defined-risk short-vol spread, as a FRACTION of spot S.
|
||||
|
||||
credit = [ short ATM straddle ] − [ long OTM strangle wings ]
|
||||
= ( call(S,S) + put(S,S) ) − ( call(S, S·(1+w)) + put(S, S·(1−w)) ) (all ÷ S)
|
||||
|
||||
The PUT legs (short ATM put + long put wing) are priced at IV + put_skew_vol/100 (crypto put skew); the
|
||||
CALL legs at the flat ATM IV. `iv` is the annualised ATM vol fraction (DVOL/100); `t` years; `w`=wing_width
|
||||
a positive fraction of S. Returned per 1 unit of spot notional (÷S) so it composes with the return ladder.
|
||||
A WIDER wing → CHEAPER wing → LARGER credit (but a larger max loss); the naked limit is w→∞ (no wing cost,
|
||||
no cap). Pure."""
|
||||
iv = float(iv)
|
||||
put_iv = iv + float(put_skew_vol) / 100.0
|
||||
k_up = s * (1.0 + wing_width)
|
||||
k_dn = s * (1.0 - wing_width)
|
||||
short_straddle = bs_call(s=s, k=s, t=t, sigma=iv) + bs_put(s=s, k=s, t=t, sigma=put_iv)
|
||||
long_wings = bs_call(s=s, k=k_up, t=t, sigma=iv) + bs_put(s=s, k=k_dn, t=t, sigma=put_iv)
|
||||
return (short_straddle - long_wings) / s
|
||||
|
||||
|
||||
def defined_risk_payoff(*, credit: float, realized_move: float, wing_width: float,
|
||||
direction: str = "short_vol") -> float:
|
||||
"""The defined-risk short-vol P&L at expiry, as a FRACTION of spot. `realized_move` m = S_T/S − 1.
|
||||
|
||||
pnl = credit − |m| + max(|m| − w, 0) (short_vol)
|
||||
|
||||
The short straddle pays the realized |move|; the long wings REFUND everything beyond the wing distance w,
|
||||
so the loss is BOUNDED below at credit − w. A calm expiry (|m| ≤ ... ) keeps (most of) the credit; a huge
|
||||
move loses AT MOST w − credit (the DEFINED risk), never more. `direction="long_vol"` negates the whole
|
||||
structure (buy the straddle, sell the wings — capped GAIN, the mirror). Pure."""
|
||||
if direction not in _DIRECTIONS:
|
||||
raise ValueError(f"direction must be one of {_DIRECTIONS}, got {direction!r}")
|
||||
m = abs(float(realized_move))
|
||||
short_vol_pnl = float(credit) - m + max(m - float(wing_width), 0.0)
|
||||
sign = 1.0 if direction == "short_vol" else -1.0
|
||||
return sign * short_vol_pnl
|
||||
|
||||
|
||||
def defined_risk_max_loss(*, credit: float, wing_width: float) -> float:
|
||||
"""The DEFINED (bounded) worst-case loss of the short-vol spread, as a fraction of spot: w − credit
|
||||
(a positive number = the most we can lose). This is the whole point — the naked short-straddle's loss is
|
||||
unbounded; the wings cap it here. Pure."""
|
||||
return float(wing_width) - float(credit)
|
||||
|
||||
|
||||
# --- 2. the per-asset rolling ladder of defined-risk spreads ------------------------------------
|
||||
|
||||
def _asset_defined_risk_ladder(cser: dict[int, float], dser: dict[int, float], *, direction: str,
|
||||
swap_days: int, wing_width: float, put_skew_vol: float,
|
||||
wing_in_atm_std: float | None) -> dict[int, float]:
|
||||
"""The rolling defined-risk-spread LADDER for ONE asset → `{day: daily_pnl}` (UNSIZED, fraction of spot).
|
||||
|
||||
For every entry day s with a known close S_s and DVOL_s (causal): price the spread credit at entry, look
|
||||
forward `swap_days` to the expiry close, compute the bounded expiry P&L from the realized move, and SPREAD
|
||||
that single expiry P&L over the spread's life (÷ its length) — accruing to each day it covers, exactly like
|
||||
the variance-swap ladder. The day-t book P&L is the AVERAGE of all in-flight spreads' per-day accruals. If
|
||||
`wing_in_atm_std` is set, the wing distance is `wing_in_atm_std · IV · √T` (an N-sigma placement) instead
|
||||
of the fixed `wing_width`. CAUSAL: credit + wing placement use entry-day IV; the realized move is strictly
|
||||
forward of entry. Pure."""
|
||||
rets = _daily_close_returns(cser)
|
||||
if not rets:
|
||||
return {}
|
||||
ret_days = sorted(rets)
|
||||
closes = cser
|
||||
pnl_sum: dict[int, float] = {}
|
||||
pnl_cnt: dict[int, int] = {}
|
||||
t_years = swap_days / 365.0
|
||||
for idx, s_day in enumerate(ret_days[:-1] if len(ret_days) > 1 else ret_days):
|
||||
window_days = ret_days[idx + 1: idx + 1 + swap_days]
|
||||
if not window_days:
|
||||
continue
|
||||
spot = closes.get(s_day)
|
||||
dvol_entry = dser.get(s_day)
|
||||
expiry_close = closes.get(window_days[-1])
|
||||
if spot is None or dvol_entry is None or expiry_close is None:
|
||||
continue
|
||||
if not (math.isfinite(spot) and spot > 0 and math.isfinite(expiry_close)):
|
||||
continue
|
||||
iv = float(dvol_entry) / 100.0
|
||||
w = wing_in_atm_std * iv * math.sqrt(t_years) if wing_in_atm_std is not None else wing_width
|
||||
if not (0.0 < w < 1.0):
|
||||
w = min(max(w, 1e-6), 0.999) # keep wing strikes strictly between 0 and 2·S
|
||||
credit = defined_risk_credit(s=spot, iv=iv, t=t_years, wing_width=w, put_skew_vol=put_skew_vol)
|
||||
realized_move = expiry_close / spot - 1.0
|
||||
payoff = defined_risk_payoff(credit=credit, realized_move=realized_move, wing_width=w,
|
||||
direction=direction)
|
||||
per_day = payoff / len(window_days)
|
||||
for d in window_days:
|
||||
pnl_sum[d] = pnl_sum.get(d, 0.0) + per_day
|
||||
pnl_cnt[d] = pnl_cnt.get(d, 0) + 1
|
||||
return {d: pnl_sum[d] / pnl_cnt[d] for d in pnl_sum}
|
||||
|
||||
|
||||
def _book_defined_risk(store: _FeatureStore, *, direction: str, swap_days: int = _DEFAULT_SWAP_DAYS,
|
||||
wing_width: float = _DEFAULT_WING_WIDTH, put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
|
||||
wing_in_atm_std: float | None = None) -> dict[int, float]:
|
||||
"""`{day: raw_defined_risk_ret}` — the BTC+ETH EQUAL-WEIGHT (mean over assets present) raw, UNSIZED (k=1)
|
||||
rolling defined-risk-spread daily return. CAUSAL + READ-ONLY (pure off panels). Empty when no overlapping
|
||||
dvol+close data. Mirrors `vrp_eval._book_short_var` but uses the BS-priced capped spread ladder."""
|
||||
close = store.read_panel(list(_VRP_SYMBOLS), "close")
|
||||
dvol = store.read_panel(list(_VRP_SYMBOLS), "dvol")
|
||||
close = {s: v for s, v in close.items() if v}
|
||||
dvol = {s: v for s, v in dvol.items() if v}
|
||||
if not close or not dvol:
|
||||
return {}
|
||||
|
||||
per_day: dict[int, list[float]] = {}
|
||||
for sym in _VRP_SYMBOLS:
|
||||
cser = close.get(sym, {})
|
||||
dser = dvol.get(sym, {})
|
||||
if not cser or not dser:
|
||||
continue
|
||||
ladder = _asset_defined_risk_ladder(cser, dser, direction=direction, swap_days=swap_days,
|
||||
wing_width=wing_width, put_skew_vol=put_skew_vol,
|
||||
wing_in_atm_std=wing_in_atm_std)
|
||||
for d, v in ladder.items():
|
||||
if math.isfinite(v):
|
||||
per_day.setdefault(int(d), []).append(v)
|
||||
return {d: st.mean(vals) for d, vals in per_day.items() if vals}
|
||||
|
||||
|
||||
# --- 3. reuse the naked harness by swapping in the defined-risk raw book ------------------------
|
||||
#
|
||||
# vrp_eval's verify / walk-forward / metrics all build their raw series by calling `_book_short_var`. We run
|
||||
# them against the DEFINED-RISK book by temporarily binding `vrp_eval._book_short_var` to our capped builder
|
||||
# (with the wing params closed over) for the duration of the call. This REUSES the entire tail-aware harness
|
||||
# — vol-target sizing, cost grid, per-year, worst-day, maxDD, corr-vs-4-edges, look-ahead audit, PASS/FAIL
|
||||
# gate — verbatim, so the defined-risk verdict is computed by the exact same judge as the naked one.
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _defined_risk_book(*, wing_width: float, put_skew_vol: float, wing_in_atm_std: float | None,
|
||||
swap_days: int = _DEFAULT_SWAP_DAYS):
|
||||
"""Bind `vrp_eval._book_short_var` (+ leaked control) to the defined-risk builders AND `vrp_eval._sized_series`
|
||||
to the defined-risk AMORTIZED-ROLL cost model, for the duration of the `with` block. The cost rebind is the
|
||||
turnover FIX: the naked harness's `_sized_series` charges the full ladder every day (k·cost_bps), which
|
||||
over-charges a held-to-expiry 30-day condor ladder by ~swap_days/n_legs; the defined-risk series charges only
|
||||
the one-spread-per-day roll (`_defined_risk_sized_series`). Restores the originals on exit (even on error)."""
|
||||
orig = vrp_eval._book_short_var
|
||||
orig_leaked = vrp_eval._book_short_var_leaked
|
||||
orig_sized = vrp_eval._sized_series
|
||||
|
||||
def _capped(store, *, direction, swap_days=swap_days, _leak=False):
|
||||
if _leak:
|
||||
return _book_defined_risk_leaked(store, direction=direction, swap_days=swap_days,
|
||||
wing_width=wing_width, put_skew_vol=put_skew_vol,
|
||||
wing_in_atm_std=wing_in_atm_std)
|
||||
return _book_defined_risk(store, direction=direction, swap_days=swap_days, wing_width=wing_width,
|
||||
put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std)
|
||||
|
||||
def _capped_leaked(store, *, direction):
|
||||
return _book_defined_risk_leaked(store, direction=direction, wing_width=wing_width,
|
||||
put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std)
|
||||
|
||||
def _capped_sized(raw, *, k, cost_bps):
|
||||
return _defined_risk_sized_series(raw, k=k, cost_bps=cost_bps, swap_days=swap_days)
|
||||
|
||||
vrp_eval._book_short_var = _capped
|
||||
vrp_eval._book_short_var_leaked = _capped_leaked
|
||||
vrp_eval._sized_series = _capped_sized
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
vrp_eval._book_short_var = orig
|
||||
vrp_eval._book_short_var_leaked = orig_leaked
|
||||
vrp_eval._sized_series = orig_sized
|
||||
|
||||
|
||||
def _book_defined_risk_leaked(store: _FeatureStore, *, direction: str, swap_days: int = _DEFAULT_SWAP_DAYS,
|
||||
wing_width: float = _DEFAULT_WING_WIDTH,
|
||||
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
|
||||
wing_in_atm_std: float | None = None) -> dict[int, float]:
|
||||
"""NEGATIVE-CONTROL defined-risk book: prices each spread's credit/wings off the EXPIRY-day IV (knowable
|
||||
only at maturity), a look-ahead leak the causality audit must catch. Mirrors the variance-swap leak."""
|
||||
close = store.read_panel(list(_VRP_SYMBOLS), "close")
|
||||
dvol = store.read_panel(list(_VRP_SYMBOLS), "dvol")
|
||||
close = {s: v for s, v in close.items() if v}
|
||||
dvol = {s: v for s, v in dvol.items() if v}
|
||||
if not close or not dvol:
|
||||
return {}
|
||||
t_years = swap_days / 365.0
|
||||
per_day: dict[int, list[float]] = {}
|
||||
for sym in _VRP_SYMBOLS:
|
||||
cser = close.get(sym, {})
|
||||
dser = dvol.get(sym, {})
|
||||
if not cser or not dser:
|
||||
continue
|
||||
rets = _daily_close_returns(cser)
|
||||
if not rets:
|
||||
continue
|
||||
ret_days = sorted(rets)
|
||||
for idx, s_day in enumerate(ret_days[:-1] if len(ret_days) > 1 else ret_days):
|
||||
window_days = ret_days[idx + 1: idx + 1 + swap_days]
|
||||
if not window_days:
|
||||
continue
|
||||
spot = cser.get(s_day)
|
||||
expiry_close = cser.get(window_days[-1])
|
||||
dvol_leak = dser.get(window_days[-1]) # leak: IV dated at expiry (look-ahead)
|
||||
if spot is None or expiry_close is None or dvol_leak is None:
|
||||
continue
|
||||
if not (math.isfinite(spot) and spot > 0 and math.isfinite(expiry_close)):
|
||||
continue
|
||||
iv = float(dvol_leak) / 100.0
|
||||
w = (wing_in_atm_std * iv * math.sqrt(t_years)) if wing_in_atm_std is not None else wing_width
|
||||
if not (0.0 < w < 1.0):
|
||||
w = min(max(w, 1e-6), 0.999)
|
||||
credit = defined_risk_credit(s=spot, iv=iv, t=t_years, wing_width=w, put_skew_vol=put_skew_vol)
|
||||
payoff = defined_risk_payoff(credit=credit, realized_move=expiry_close / spot - 1.0,
|
||||
wing_width=w, direction=direction)
|
||||
per_day_pnl = payoff / len(window_days)
|
||||
for d in window_days:
|
||||
per_day.setdefault(int(d), []).append(per_day_pnl)
|
||||
# average over in-flight spreads, mirroring the causal ladder's per-day mean over assets+swaps
|
||||
return {d: st.mean(vals) for d, vals in per_day.items() if vals}
|
||||
|
||||
|
||||
# --- 4. public entry points (mirror vrp_eval, with the wing params) ----------------------------
|
||||
|
||||
def defined_risk_returns_from_store(store: _FeatureStore, *, cost_bps: float = 5.5,
|
||||
direction: str = "short_vol",
|
||||
target_ann_vol: float = _DEFAULT_TARGET_VOL,
|
||||
wing_width: float = _DEFAULT_WING_WIDTH,
|
||||
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
|
||||
wing_in_atm_std: float | None = None) -> dict[int, float]:
|
||||
"""READ-ONLY per-day DEFINED-RISK VRP return series `{epoch_day: net_ret}`. BTC+ETH equal-weight capped
|
||||
short-vol spread, vol-targeted, net of cost. Writes NOTHING."""
|
||||
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
|
||||
return vrp_eval.vrp_returns_from_store(store, cost_bps=cost_bps, direction=direction,
|
||||
target_ann_vol=target_ann_vol)
|
||||
|
||||
|
||||
def defined_risk_metrics_from_store(store: _FeatureStore, *, cost_bps: float = 5.5,
|
||||
direction: str = "short_vol",
|
||||
target_ann_vol: float = _DEFAULT_TARGET_VOL,
|
||||
wing_width: float = _DEFAULT_WING_WIDTH,
|
||||
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
|
||||
wing_in_atm_std: float | None = None) -> dict[str, Any]:
|
||||
"""READ-ONLY DEFINED-RISK VRP metrics (the standard curve metrics + tail diagnostics). READ-ONLY."""
|
||||
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
|
||||
m = vrp_eval.vrp_metrics_from_store(store, cost_bps=cost_bps, direction=direction,
|
||||
target_ann_vol=target_ann_vol)
|
||||
m["wing_width"] = wing_width
|
||||
m["put_skew_vol"] = put_skew_vol
|
||||
return m
|
||||
|
||||
|
||||
def verify_defined_risk_edge(store: _FeatureStore, *, cost_bps: float = 5.5,
|
||||
cost_grid: tuple[float, ...] = (5.5, 11.0, 22.0, 50.0, 100.0),
|
||||
direction: str = "short_vol",
|
||||
target_ann_vol: float = _DEFAULT_TARGET_VOL,
|
||||
wing_width: float = _DEFAULT_WING_WIDTH,
|
||||
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
|
||||
wing_in_atm_std: float | None = None,
|
||||
universe: set[str] | None = None,
|
||||
unlock_events: list[dict[str, Any]] | None = None) -> dict[str, Any]:
|
||||
"""READ-ONLY adversarial verification of the DEFINED-RISK VRP (both directions) — the naked `verify`
|
||||
harness fed the capped spread book (cost grid incl realistic option costs, per-year, worst-day, maxDD,
|
||||
corr-vs-4-edges). Adds `wing_width`/`put_skew_vol` to the report. READ-ONLY."""
|
||||
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
|
||||
rep = vrp_eval.verify_vrp_edge(store, cost_bps=cost_bps, cost_grid=cost_grid, direction=direction,
|
||||
target_ann_vol=target_ann_vol, universe=universe,
|
||||
unlock_events=unlock_events)
|
||||
if rep.get("available"):
|
||||
rep["wing_width"] = wing_width
|
||||
rep["put_skew_vol"] = put_skew_vol
|
||||
rep["construction_note"] = ("DEFINED-RISK: short ATM straddle + long OTM wings (BS-priced from DVOL); "
|
||||
"max loss BOUNDED at wing_width − credit")
|
||||
return rep
|
||||
|
||||
|
||||
def look_ahead_audit_defined_risk(store: _FeatureStore, *, direction: str = "short_vol",
|
||||
_leak: bool = False, wing_width: float = _DEFAULT_WING_WIDTH,
|
||||
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
|
||||
wing_in_atm_std: float | None = None) -> dict[str, Any]:
|
||||
"""READ-ONLY strict-causality audit of the DEFINED-RISK book: prove each spread prices its credit/wings off
|
||||
the ENTRY-day IV, never the expiry IV. Reuses `vrp_eval.look_ahead_audit_vrp` against the capped builder.
|
||||
Returns its `{causal, leak_days, n_days_compared}` shape."""
|
||||
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
|
||||
return vrp_eval.look_ahead_audit_vrp(store, direction=direction, _leak=_leak)
|
||||
|
||||
|
||||
def look_ahead_audit_defined_risk_ok(store: _FeatureStore, *, direction: str = "short_vol",
|
||||
_leak: bool = False, **kw: Any) -> bool:
|
||||
"""Convenience boolean: True iff the defined-risk credit/wing pricing is strictly causal (no look-ahead)."""
|
||||
return bool(look_ahead_audit_defined_risk(store, direction=direction, _leak=_leak, **kw)["causal"])
|
||||
|
||||
|
||||
def walk_forward_defined_risk(store: _FeatureStore, *, cost_bps: float = 5.5, window_days: int = 180,
|
||||
train_frac: float = 0.6, target_ann_vol: float = _DEFAULT_TARGET_VOL,
|
||||
max_dd_floor: float = -0.40, wing_width: float = _DEFAULT_WING_WIDTH,
|
||||
put_skew_vol: float = _DEFAULT_PUT_SKEW_VOL,
|
||||
wing_in_atm_std: float | None = None,
|
||||
universe: set[str] | None = None) -> dict[str, Any]:
|
||||
"""READ-ONLY, TAIL-AWARE OOS / WALK-FORWARD validation of the DEFINED-RISK VRP — the DEPLOY GATE — via the
|
||||
naked `walk_forward_vrp` harness fed the capped book. The verdict is the SAME tail-aware PASS/FAIL: OOS
|
||||
Sharpe>0, positive in >70% of windows, maxDD (full AND OOS) above the floor, look-ahead clean. The wings
|
||||
are meant to make the `tail_survivable` check PASS where the naked book failed (−74%). READ-ONLY."""
|
||||
with _defined_risk_book(wing_width=wing_width, put_skew_vol=put_skew_vol, wing_in_atm_std=wing_in_atm_std):
|
||||
rep = vrp_eval.walk_forward_vrp(store, cost_bps=cost_bps, window_days=window_days,
|
||||
train_frac=train_frac, target_ann_vol=target_ann_vol,
|
||||
max_dd_floor=max_dd_floor, universe=universe)
|
||||
if rep.get("available"):
|
||||
rep["wing_width"] = wing_width
|
||||
rep["put_skew_vol"] = put_skew_vol
|
||||
rep["construction_note"] = ("DEFINED-RISK: short ATM straddle + long OTM wings (BS-priced from DVOL); "
|
||||
"max loss BOUNDED at wing_width − credit (caps the naked −74% tail)")
|
||||
return rep
|
||||
@@ -1,554 +0,0 @@
|
||||
"""READ-ONLY, TAIL-HONEST evaluator for the VRP (volatility-risk-premium) edge — a genuinely DIFFERENT edge
|
||||
type (a vol premium, uncorrelated with the directional/flow edges already in the book).
|
||||
|
||||
Economic thesis
|
||||
---------------
|
||||
Deribit's DVOL is a 30-day annualised IMPLIED-vol index off the BTC/ETH options book — the market's expected
|
||||
vol. It systematically exceeds the REALIZED vol of the underlying (the volatility risk premium), so SELLING
|
||||
variance ("short-vol") earns the spread. The catch: the payoff has a fat LEFT tail — when a vol spike makes
|
||||
RV >> IV, the short-vol position takes a large loss that can wipe out many days of premium. VRP looks great
|
||||
until a spike; this evaluator is built to make the tail VISIBLE and to judge the edge NET of it.
|
||||
|
||||
The VRP return — a PROPER ROLLING VARIANCE SWAP (per asset, per day — CAUSAL)
|
||||
----------------------------------------------------------------------------
|
||||
The old construction paid a SINGLE day's `ret_t²` as the realized variance — a wildly noisy/fat-tailed RV
|
||||
estimate whose spikes blew through the vol-target (the walk-forward showed Sharpe −40 / −96% per 180d window,
|
||||
which is a CONSTRUCTION ARTIFACT, not a verdict). We replace it with the textbook variance swap.
|
||||
|
||||
A SHORT variance swap entered at day t with maturity `swap_days` (default 30):
|
||||
|
||||
* STRIKE (implied annual variance, CAUSAL — known at t): K_t = (DVOL_t / 100)²
|
||||
DVOL_t is the annualised implied vol in vol POINTS (e.g. 50 → (0.50)² = 0.25 annual variance).
|
||||
* REALIZED annual variance over (t, t+swap_days]: RV²_t = (365/swap_days)·Σ_{i=t+1..t+swap_days} ret_i²
|
||||
the annualised realized variance from daily close-to-close returns over the swap's life.
|
||||
* SHORT-VAR SWAP PAYOFF (per unit variance notional): K_t − RV²_t
|
||||
POSITIVE when implied>realized (the premium harvested), NEGATIVE on a vol-spike window (the tail).
|
||||
|
||||
Daily series via a rolling LADDER (`_book_short_var`)
|
||||
----------------------------------------------------
|
||||
Each day the book holds `swap_days` overlapping short-variance swaps — one ENTERED on each of the prior
|
||||
`swap_days` days. On day t exactly ONE swap (the one entered at t−swap_days) MATURES; its realized window
|
||||
payoff `K_{t−swap_days} − RV²_{t−swap_days}` is recognised, SPREAD over the swap's life (÷swap_days) so the
|
||||
ladder produces ONE smooth daily P&L per day rather than a single raw `ret²`. Because RV² is a swap_days-window
|
||||
AVERAGE of daily ret², the daily series is far LESS noisy than the old single-day proxy — a properly
|
||||
vol-targeted curve must NOT lose ~everything in 180d. (Mathematically the ladder's day-t P&L equals the
|
||||
average daily accrual of all in-flight swaps; the maturing-increment form is the simplest exactly-equivalent
|
||||
expression.) `direction="long_vol"` negates (buy variance). `k` is the SIZING constant (below).
|
||||
|
||||
The book is BTC + ETH EQUAL-WEIGHT (`_book_short_var`), then vol-TARGETED.
|
||||
|
||||
Sizing (vol-target — matters, a naive short-variance has meaningless scale)
|
||||
---------------------------------------------------------------------------
|
||||
The raw average short-var series has an arbitrary scale (variance units). We VOL-TARGET it: pick `k` so the
|
||||
full-sample annualised vol of `k · raw` equals `target_ann_vol` (default 0.15 = 15%/yr), making Sharpe/DD
|
||||
comparable to the other edges. `k` is reported as the (in-sample) sizing constant and the implied
|
||||
`avg_leverage`/exposure. CAVEAT: `k` here is full-sample in-sample (a known overfit-of-scale — the walk-
|
||||
forward sizes `k` on TRAIN only). Costs scale with the leverage (a short-straddle is re-established/re-hedged
|
||||
daily, so the per-day cost ≈ leverage × cost_bps).
|
||||
|
||||
READ-ONLY + cross-venue caveat
|
||||
------------------------------
|
||||
No store.write_* — the live paper_* tables are never touched. The SIGNAL is Deribit's DVOL; live EXECUTION
|
||||
would be short Bybit option straddles (delta-hedged) — a cross-venue caveat the deploy decision must weigh.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import math
|
||||
import statistics as st
|
||||
from typing import Any, Protocol
|
||||
|
||||
from fxhnt.application.bybit_book_eval import sleeve_returns_from_store
|
||||
from fxhnt.application.bybit_carry_revalidate import _curve_metrics
|
||||
|
||||
_PERIODS_PER_YEAR = 365
|
||||
_SQRT_YEAR = math.sqrt(_PERIODS_PER_YEAR)
|
||||
_EXISTING_EDGES = ("tstrend", "unlock", "xsfunding", "positioning")
|
||||
_VRP_SYMBOLS = ("BTCUSDT", "ETHUSDT") # the assets with a Deribit DVOL index (BTC/ETH)
|
||||
_DIRECTIONS = ("short_vol", "long_vol")
|
||||
_DEFAULT_TARGET_VOL = 0.15 # 15%/yr vol-target (sane, comparable to the other edges)
|
||||
_DEFAULT_SWAP_DAYS = 30 # variance-swap maturity (DVOL is a 30-day implied-vol index)
|
||||
|
||||
|
||||
class _FeatureStore(Protocol):
|
||||
"""The panel reads the VRP pipeline needs (TimescaleFeatureStore satisfies this). READ-ONLY."""
|
||||
|
||||
def crypto_close_panel(self, *, suffix: str = "USDT") -> dict[str, dict[int, float]]: ...
|
||||
def read_panel(self, symbols: list[str], feature: str) -> dict[str, dict[int, float]]: ...
|
||||
|
||||
|
||||
def _epoch_day_to_iso(day: int) -> str:
|
||||
return (dt.date(1970, 1, 1) + dt.timedelta(days=int(day))).isoformat()
|
||||
|
||||
|
||||
def _year_of_day(day: int) -> str:
|
||||
return str((dt.date(1970, 1, 1) + dt.timedelta(days=int(day))).year)
|
||||
|
||||
|
||||
# --- 1. the VRP return formulas ----------------------------------------------------------------
|
||||
|
||||
def short_var_return(*, dvol_prev: float, ret: float, k: float = 1.0,
|
||||
direction: str = "short_vol") -> float:
|
||||
"""The per-asset, per-day SINGLE-DAY short-variance increment:
|
||||
|
||||
short_var_ret = k * [ (dvol_prev/√365/100)² − ret² ]
|
||||
|
||||
`dvol_prev` is the IV (DVOL index, vol points) known at the START of the day; `ret` is the close-to-close
|
||||
return realized OVER the day. POSITIVE when calm (implied daily variance > realized), big-NEGATIVE on a
|
||||
vol spike. `direction="long_vol"` negates (buy variance). Pure.
|
||||
|
||||
NOTE: this single-day proxy is the OLD noisy construction — kept only as the per-day accrual building
|
||||
block / the look-ahead audit's negative control. The book series now uses the rolling variance swap
|
||||
(`variance_swap_payoff` + the ladder in `_book_short_var`)."""
|
||||
if direction not in _DIRECTIONS:
|
||||
raise ValueError(f"direction must be one of {_DIRECTIONS}, got {direction!r}")
|
||||
implied_daily_var = (float(dvol_prev) / _SQRT_YEAR / 100.0) ** 2
|
||||
realized_daily_var = float(ret) ** 2
|
||||
sign = 1.0 if direction == "short_vol" else -1.0
|
||||
return float(k) * sign * (implied_daily_var - realized_daily_var)
|
||||
|
||||
|
||||
def variance_swap_payoff(*, dvol_entry: float, fwd_rets: list[float], swap_days: int = _DEFAULT_SWAP_DAYS,
|
||||
k: float = 1.0, direction: str = "short_vol") -> float:
|
||||
"""The textbook SHORT variance-swap payoff (per unit variance notional) over one swap's life:
|
||||
|
||||
payoff = k * sign * [ K − RV² ]
|
||||
K = (dvol_entry/100)² (annualised IMPLIED variance, strike)
|
||||
RV² = (365/swap_days) · Σ_{i=1..swap_days} fwd_rets[i]² (annualised REALIZED variance, the window)
|
||||
|
||||
`dvol_entry` is the DVOL (vol points, annualised) known at swap ENTRY (causal). `fwd_rets` is the list of
|
||||
the `swap_days` daily close-to-close returns realized OVER the swap's life (forward of entry); fewer than
|
||||
`swap_days` returns annualises by however many are present (a partial/truncated final swap). POSITIVE when
|
||||
implied>realized (the premium), big-NEGATIVE on a vol-spike window (the tail). `direction="long_vol"`
|
||||
negates (buy variance). Pure."""
|
||||
if direction not in _DIRECTIONS:
|
||||
raise ValueError(f"direction must be one of {_DIRECTIONS}, got {direction!r}")
|
||||
n = len(fwd_rets)
|
||||
if n == 0:
|
||||
return 0.0
|
||||
strike = (float(dvol_entry) / 100.0) ** 2
|
||||
realized = (_PERIODS_PER_YEAR / n) * sum(float(r) ** 2 for r in fwd_rets)
|
||||
sign = 1.0 if direction == "short_vol" else -1.0
|
||||
return float(k) * sign * (strike - realized)
|
||||
|
||||
|
||||
# --- 2. the BTC+ETH equal-weight raw book + vol-target sizing ----------------------------------
|
||||
|
||||
def _daily_close_returns(cser: dict[int, float]) -> dict[int, float]:
|
||||
"""`{day: close-to-close ret}` for the days with a finite, non-zero prior close. Pure."""
|
||||
days = sorted(cser)
|
||||
out: dict[int, float] = {}
|
||||
for i in range(1, len(days)):
|
||||
d0, d1 = days[i - 1], days[i]
|
||||
p0, p1 = cser[d0], cser[d1]
|
||||
if p0 and math.isfinite(p0) and math.isfinite(p1):
|
||||
out[int(d1)] = p1 / p0 - 1.0
|
||||
return out
|
||||
|
||||
|
||||
def _asset_swap_ladder(cser: dict[int, float], dser: dict[int, float], *, direction: str,
|
||||
swap_days: int, leak: bool = False) -> dict[int, float]:
|
||||
"""The rolling variance-swap LADDER for ONE asset → `{day: daily_swap_pnl}` (UNSIZED, per unit notional).
|
||||
|
||||
For every entry day s with a known DVOL_s (causal), open a short-variance swap whose strike is
|
||||
K_s=(DVOL_s/100)² and whose realized window is the next `swap_days` daily returns (forward of s). The
|
||||
swap's window payoff `K_s − RV²_s` is SPREAD over the swap's life (÷ its realized length) and accrued to
|
||||
each day it covers; the day-t book P&L is the AVERAGE of the per-day accruals of all swaps in flight on t
|
||||
(a smooth `swap_days`-window-averaged quantity, NOT a single raw ret²). `leak=True` is the negative
|
||||
control: it dates the strike by the END of the realized window (DVOL_{s+swap_days}, knowable only at
|
||||
maturity) — a look-ahead the audit must catch. Pure."""
|
||||
rets = _daily_close_returns(cser)
|
||||
if not rets:
|
||||
return {}
|
||||
ret_days = sorted(rets)
|
||||
pnl_sum: dict[int, float] = {}
|
||||
pnl_cnt: dict[int, int] = {}
|
||||
for idx, s in enumerate(ret_days[:-1] if len(ret_days) > 1 else ret_days):
|
||||
# forward window of up to swap_days realized daily returns AFTER entry day s
|
||||
window_days = ret_days[idx + 1: idx + 1 + swap_days]
|
||||
if not window_days:
|
||||
continue
|
||||
fwd = [rets[d] for d in window_days]
|
||||
strike_day = window_days[-1] if leak else s # leak: DVOL dated at maturity (look-ahead)
|
||||
dvol_entry = dser.get(strike_day)
|
||||
if dvol_entry is None:
|
||||
continue
|
||||
payoff = variance_swap_payoff(dvol_entry=dvol_entry, fwd_rets=fwd,
|
||||
swap_days=swap_days, direction=direction)
|
||||
per_day = payoff / len(fwd) # spread the window payoff over the swap's life
|
||||
for d in window_days: # accrue to each day the swap covers
|
||||
pnl_sum[d] = pnl_sum.get(d, 0.0) + per_day
|
||||
pnl_cnt[d] = pnl_cnt.get(d, 0) + 1
|
||||
return {d: pnl_sum[d] / pnl_cnt[d] for d in pnl_sum} # average over in-flight swaps that day
|
||||
|
||||
|
||||
def _book_short_var(store: _FeatureStore, *, direction: str,
|
||||
swap_days: int = _DEFAULT_SWAP_DAYS, _leak: bool = False,
|
||||
) -> dict[int, float]:
|
||||
"""`{day: raw_short_var_ret}` — the BTC+ETH EQUAL-WEIGHT (mean over the assets present that day) raw,
|
||||
UNSIZED (k=1) ROLLING VARIANCE-SWAP daily return (the `_asset_swap_ladder` per asset). CAUSAL: each swap's
|
||||
strike DVOL is known at entry and its realized window is strictly FORWARD of entry. READ-ONLY. Pure off
|
||||
panels. `_leak=True` builds the look-ahead negative control. Empty when no overlapping dvol+close data."""
|
||||
close = store.read_panel(list(_VRP_SYMBOLS), "close")
|
||||
dvol = store.read_panel(list(_VRP_SYMBOLS), "dvol")
|
||||
close = {s: v for s, v in close.items() if v}
|
||||
dvol = {s: v for s, v in dvol.items() if v}
|
||||
if not close or not dvol:
|
||||
return {}
|
||||
|
||||
per_day: dict[int, list[float]] = {}
|
||||
for sym in _VRP_SYMBOLS:
|
||||
cser = close.get(sym, {})
|
||||
dser = dvol.get(sym, {})
|
||||
if not cser or not dser:
|
||||
continue
|
||||
ladder = _asset_swap_ladder(cser, dser, direction=direction, swap_days=swap_days, leak=_leak)
|
||||
for d, v in ladder.items():
|
||||
if math.isfinite(v):
|
||||
per_day.setdefault(int(d), []).append(v)
|
||||
return {d: st.mean(vals) for d, vals in per_day.items() if vals}
|
||||
|
||||
|
||||
def _vol_target_k(raw: dict[int, float], *, target_ann_vol: float,
|
||||
day_lo: int | None = None, day_hi: int | None = None) -> float:
|
||||
"""The sizing constant `k` so the annualised vol of `k·raw` over [day_lo, day_hi] == `target_ann_vol`.
|
||||
`k = target_ann_vol / (pstdev(raw) · √365)`; 0 when the raw series has <2 obs or zero vol. Pure."""
|
||||
rets = [raw[d] for d in sorted(raw)
|
||||
if (day_lo is None or d >= day_lo) and (day_hi is None or d <= day_hi)]
|
||||
if len(rets) < 2:
|
||||
return 0.0
|
||||
sd = st.pstdev(rets)
|
||||
if sd <= 0.0:
|
||||
return 0.0
|
||||
return float(target_ann_vol) / (sd * _SQRT_YEAR)
|
||||
|
||||
|
||||
def _sized_series(raw: dict[int, float], *, k: float, cost_bps: float) -> dict[int, float]:
|
||||
"""`{day: net_ret}` = k·raw − per-day cost. A short-straddle is re-established/re-hedged daily, so the
|
||||
per-day cost ≈ leverage(k) × cost_bps (a constant daily haircut on the sized notional). Pure."""
|
||||
haircut = k * (cost_bps / 1e4) if cost_bps else 0.0
|
||||
out: dict[int, float] = {}
|
||||
for d in sorted(raw):
|
||||
v = k * raw[d] - haircut
|
||||
if math.isfinite(v):
|
||||
out[int(d)] = float(v)
|
||||
return out
|
||||
|
||||
|
||||
def vrp_returns_from_store(store: _FeatureStore, *, cost_bps: float = 5.5,
|
||||
direction: str = "short_vol",
|
||||
target_ann_vol: float = _DEFAULT_TARGET_VOL) -> dict[int, float]:
|
||||
"""READ-ONLY per-day VRP RETURN SERIES `{epoch_day: net_ret}` — the book-level seam an N-th sleeve would
|
||||
consume. BTC+ETH equal-weight short-variance (or long-vol negation), vol-TARGETED to `target_ann_vol`
|
||||
(full-sample k), net of the per-day cost. Writes NOTHING."""
|
||||
raw = _book_short_var(store, direction=direction)
|
||||
if len(raw) < 2:
|
||||
return {}
|
||||
k = _vol_target_k(raw, target_ann_vol=target_ann_vol)
|
||||
return _sized_series(raw, k=k, cost_bps=cost_bps)
|
||||
|
||||
|
||||
def _metrics_from_series(series: dict[int, float], *, k: float = 0.0) -> dict[str, Any]:
|
||||
"""`_curve_metrics` over a `{day: ret}` series, PLUS the tail diagnostics: `worst_day` (the single most
|
||||
negative day — the vol-spike loss), `sum_ret`, and `avg_leverage` (= k, the vol-target exposure)."""
|
||||
daily = [(d, series[d]) for d in sorted(series)]
|
||||
meta = {"symbols": len(_VRP_SYMBOLS),
|
||||
"first_day": _epoch_day_to_iso(daily[0][0]) if daily else None,
|
||||
"last_day": _epoch_day_to_iso(daily[-1][0]) if daily else None}
|
||||
m = _curve_metrics(daily, days_meta=meta)
|
||||
m["sum_ret"] = sum(series.values())
|
||||
m["worst_day"] = min(series.values()) if series else 0.0
|
||||
m["avg_leverage"] = float(k)
|
||||
return m
|
||||
|
||||
|
||||
def vrp_metrics_from_store(store: _FeatureStore, *, cost_bps: float = 5.5,
|
||||
direction: str = "short_vol",
|
||||
target_ann_vol: float = _DEFAULT_TARGET_VOL) -> dict[str, Any]:
|
||||
"""READ-ONLY VRP metrics (the standard `_curve_metrics` shape + tail diagnostics `worst_day`/`max_dd` +
|
||||
`avg_leverage`). `available=False` (+ `reason`) when there is no overlapping dvol+close data / too few
|
||||
days. READ-ONLY."""
|
||||
raw = _book_short_var(store, direction=direction)
|
||||
if len(raw) < 2:
|
||||
return {"cagr": 0.0, "sharpe": 0.0, "max_dd": 0.0, "total_return": 0.0, "ann_return": 0.0,
|
||||
"days": 0, "symbols": 0, "first_day": None, "last_day": None,
|
||||
"worst_day": 0.0, "avg_leverage": 0.0, "available": False,
|
||||
"reason": "VRP curve has <2 days (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)"}
|
||||
k = _vol_target_k(raw, target_ann_vol=target_ann_vol)
|
||||
series = _sized_series(raw, k=k, cost_bps=cost_bps)
|
||||
m = _metrics_from_series(series, k=k)
|
||||
m["available"] = True
|
||||
return m
|
||||
|
||||
|
||||
# --- 3. correlation vs the existing edges ------------------------------------------------------
|
||||
|
||||
def _correlation(a_series: dict[int, float], b_series: dict[int, float]) -> float | None:
|
||||
"""Pearson correlation over the two series' COMMON days; None if <2 common days or zero variance. Pure."""
|
||||
common = sorted(set(a_series) & set(b_series))
|
||||
if len(common) < 2:
|
||||
return None
|
||||
a = [a_series[d] for d in common]
|
||||
b = [b_series[d] for d in common]
|
||||
sa, sb = st.pstdev(a), st.pstdev(b)
|
||||
if sa <= 0.0 or sb <= 0.0:
|
||||
return None
|
||||
ma, mb = st.mean(a), st.mean(b)
|
||||
cov = sum((x - ma) * (y - mb) for x, y in zip(a, b, strict=True)) / len(common)
|
||||
return cov / (sa * sb)
|
||||
|
||||
|
||||
def _existing_edge_series(store: _FeatureStore, *, universe: set[str] | None, cost_bps: float,
|
||||
unlock_events: list[dict[str, Any]] | None,
|
||||
) -> dict[str, dict[int, float]]:
|
||||
"""The per-day return series for each EXISTING edge (tstrend/unlock/xsfunding/positioning) via the live
|
||||
sleeve runners. unlock needs the injected calendar; absent → empty. READ-ONLY."""
|
||||
out: dict[str, dict[int, float]] = {}
|
||||
for edge in _EXISTING_EDGES:
|
||||
out[edge] = sleeve_returns_from_store(
|
||||
store, edge, universe=universe, cost_bps=cost_bps, unlock_events=unlock_events)
|
||||
return out
|
||||
|
||||
|
||||
# --- 4. adversarial verification ---------------------------------------------------------------
|
||||
|
||||
def verify_vrp_edge(store: _FeatureStore, *, cost_bps: float = 5.5,
|
||||
cost_grid: tuple[float, ...] = (5.5, 11.0, 22.0, 50.0, 100.0),
|
||||
direction: str = "short_vol",
|
||||
target_ann_vol: float = _DEFAULT_TARGET_VOL,
|
||||
universe: set[str] | None = None,
|
||||
unlock_events: list[dict[str, Any]] | None = None) -> dict[str, Any]:
|
||||
"""READ-ONLY adversarial verification of the VRP edge in BOTH directions (short_vol / long_vol).
|
||||
|
||||
Per direction:
|
||||
* `net_cost[dir][bps]` — `_curve_metrics` (+ `sum_ret`/`worst_day`/`avg_leverage`) net of cost at EACH
|
||||
bps in `cost_grid`. The grid spans the cheap perp-style 5.5/11/22 AND the REALISTIC OPTION costs
|
||||
50/100 bp (selling options isn't 5.5bp), so the cost sensitivity is honest. Net is RE-RUN per cost.
|
||||
* `worst_day[dir]` — the single most-negative day (the vol-spike tail loss);
|
||||
* `max_dd[dir]` — the maxDD at the headline cost (the tail);
|
||||
* `per_year[dir]` — `{YEAR: {sharpe, sum_ret, max_dd, days}}` net at `cost_bps` (all-one-year ⇒ overfit);
|
||||
* `corr_edges[dir]` — `{tstrend, unlock, xsfunding, positioning: corr|None}` of the (net @ `cost_bps`)
|
||||
series with EACH existing edge (the additive/uncorrelated check vs the whole 4-edge book);
|
||||
* `avg_leverage[dir]` — the vol-target exposure constant k.
|
||||
|
||||
The k (vol-target) is the FULL-SAMPLE constant (the walk-forward re-sizes on TRAIN). `available=False`
|
||||
(+ reason) when there is no data. READ-ONLY."""
|
||||
grid = tuple(sorted(set(cost_grid) | {cost_bps}))
|
||||
edge_series = _existing_edge_series(store, universe=universe, cost_bps=cost_bps,
|
||||
unlock_events=unlock_events)
|
||||
|
||||
net_cost: dict[str, dict[float, dict[str, Any]]] = {}
|
||||
per_year: dict[str, dict[str, dict[str, Any]]] = {}
|
||||
worst_day: dict[str, float] = {}
|
||||
max_dd: dict[str, float] = {}
|
||||
avg_leverage: dict[str, float] = {}
|
||||
corr_edges: dict[str, dict[str, float | None]] = {}
|
||||
|
||||
n_days = 0
|
||||
for d in _DIRECTIONS:
|
||||
raw = _book_short_var(store, direction=d)
|
||||
if len(raw) < 2:
|
||||
continue
|
||||
k = _vol_target_k(raw, target_ann_vol=target_ann_vol)
|
||||
avg_leverage[d] = k
|
||||
|
||||
net_cost[d] = {}
|
||||
net_at_headline: dict[int, float] = {}
|
||||
for bps in grid:
|
||||
ser = _sized_series(raw, k=k, cost_bps=bps)
|
||||
net_cost[d][bps] = _metrics_from_series(ser, k=k)
|
||||
if bps == cost_bps:
|
||||
net_at_headline = ser
|
||||
if not net_at_headline:
|
||||
net_at_headline = _sized_series(raw, k=k, cost_bps=cost_bps)
|
||||
n_days = max(n_days, len(net_at_headline))
|
||||
worst_day[d] = min(net_at_headline.values()) if net_at_headline else 0.0
|
||||
max_dd[d] = _metrics_from_series(net_at_headline, k=k)["max_dd"]
|
||||
|
||||
by_year: dict[str, dict[int, float]] = {}
|
||||
for day, r in net_at_headline.items():
|
||||
by_year.setdefault(_year_of_day(day), {})[day] = r
|
||||
per_year[d] = {}
|
||||
for yr, ser in sorted(by_year.items()):
|
||||
ym = _metrics_from_series(ser, k=k)
|
||||
per_year[d][yr] = {"sharpe": ym["sharpe"], "sum_ret": ym["sum_ret"],
|
||||
"max_dd": ym["max_dd"], "days": ym["days"]}
|
||||
|
||||
corr_edges[d] = {edge: _correlation(net_at_headline, edge_series.get(edge, {}))
|
||||
for edge in _EXISTING_EDGES}
|
||||
|
||||
if n_days < 2:
|
||||
return {"available": False,
|
||||
"reason": "VRP curve has <2 days (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)"}
|
||||
|
||||
return {"available": True, "cost_bps": cost_bps, "cost_grid": list(grid),
|
||||
"net_cost": net_cost, "per_year": per_year, "worst_day": worst_day, "max_dd": max_dd,
|
||||
"avg_leverage": avg_leverage, "corr_edges": corr_edges,
|
||||
"cross_venue_note": "SIGNAL = Deribit DVOL; EXECUTION would be short Bybit option straddles"}
|
||||
|
||||
|
||||
# --- 5. look-ahead audit -----------------------------------------------------------------------
|
||||
|
||||
def _book_short_var_leaked(store: _FeatureStore, *, direction: str) -> dict[int, float]:
|
||||
"""The NEGATIVE-CONTROL book: dates each swap's STRIKE DVOL at the END of its realized window (knowable
|
||||
only at maturity, not at entry) — a look-ahead leak. Used ONLY by the audit to prove it bites. Pure."""
|
||||
return _book_short_var(store, direction=direction, _leak=True)
|
||||
|
||||
|
||||
def look_ahead_audit_vrp(store: _FeatureStore, *, direction: str = "short_vol",
|
||||
_leak: bool = False) -> dict[str, Any]:
|
||||
"""Assert the variance-swap strike→realized-window mapping is STRICTLY CAUSAL: a swap's strike DVOL must
|
||||
be the value known at ENTRY, never one dated at the end of the realized window (knowable only at maturity).
|
||||
|
||||
Builds two books from the SAME days: the CAUSAL book (strike = DVOL at entry — the live mapping) and a
|
||||
LEAKED book (strike = DVOL at the window END — a look-ahead negative control). The candidate is the causal
|
||||
book unless `_leak=True` forces the leaked one. `leak_days` counts days where the candidate's return
|
||||
equals the leaked book's — the live causal mapping uses the entry-day DVOL, which differs from the
|
||||
window-end DVOL whenever DVOL varies, so a clean (causal) candidate is DISTINCT from the leaked book on
|
||||
EVERY overlapping day, while a leaked candidate is IDENTICAL to it on every day. `leak_days` counts the
|
||||
days the candidate equals the leaked book; `causal` iff `leak_days == 0` (no day silently uses the
|
||||
look-ahead strike). A leaked candidate matches on all days → not causal. READ-ONLY."""
|
||||
causal = _book_short_var(store, direction=direction)
|
||||
leaked = _book_short_var_leaked(store, direction=direction)
|
||||
candidate = leaked if _leak else causal
|
||||
common = sorted(set(candidate) & set(leaked))
|
||||
leak_days = sum(1 for d in common if abs(candidate[d] - leaked[d]) <= 1e-15)
|
||||
return {"causal": bool(common) and leak_days == 0,
|
||||
"leak_days": leak_days, "n_days_compared": len(common)}
|
||||
|
||||
|
||||
# --- 6. OOS / WALK-FORWARD validation (the TAIL-AWARE DEPLOY GATE) ------------------------------
|
||||
|
||||
def _sharpe(series: dict[int, float]) -> float:
|
||||
rets = [series[d] for d in sorted(series)]
|
||||
if len(rets) < 2:
|
||||
return 0.0
|
||||
sd = st.pstdev(rets)
|
||||
return (st.mean(rets) / sd * _SQRT_YEAR) if sd > 0 else 0.0
|
||||
|
||||
|
||||
def _max_dd(series: dict[int, float]) -> float:
|
||||
equity, peak, mdd = 1.0, 1.0, 0.0
|
||||
for d in sorted(series):
|
||||
equity *= (1.0 + series[d])
|
||||
peak = max(peak, equity)
|
||||
if peak > 0:
|
||||
mdd = min(mdd, equity / peak - 1.0)
|
||||
return mdd
|
||||
|
||||
|
||||
def _rolling_windows(net: dict[int, float], *, window_days: int) -> dict[str, Any]:
|
||||
"""Consecutive non-overlapping `window_days` windows; per window Sharpe + total + maxDD + days, and the
|
||||
fraction of windows with positive total return. A real edge is positive in MOST windows, not one regime."""
|
||||
if not net:
|
||||
return {"n_windows": 0, "fraction_positive": 0.0, "windows": []}
|
||||
days = sorted(net)
|
||||
lo, hi = days[0], days[-1]
|
||||
windows: list[dict[str, Any]] = []
|
||||
start = lo
|
||||
while start <= hi:
|
||||
end = start + window_days - 1
|
||||
ser = {d: net[d] for d in days if start <= d <= end}
|
||||
if len(ser) >= 2:
|
||||
total = math.prod(1.0 + ser[d] for d in sorted(ser)) - 1.0
|
||||
windows.append({"start": _epoch_day_to_iso(start), "end": _epoch_day_to_iso(min(end, hi)),
|
||||
"sharpe": _sharpe(ser), "total_return": total, "max_dd": _max_dd(ser),
|
||||
"days": len(ser)})
|
||||
start = end + 1
|
||||
n = len(windows)
|
||||
frac = (sum(1 for w in windows if w["total_return"] > 0.0) / n) if n else 0.0
|
||||
return {"n_windows": n, "fraction_positive": frac, "windows": windows}
|
||||
|
||||
|
||||
def _train_holdout(store: _FeatureStore, *, cost_bps: float, target_ann_vol: float,
|
||||
train_frac: float = 0.6) -> dict[str, Any]:
|
||||
"""The KEY OOS check, TAIL-AWARE. Split time into TRAIN (first `train_frac`) and TEST (the rest). On
|
||||
TRAIN: pick the winning DIRECTION (short_vol vs long_vol by TRAIN Sharpe) AND size k on TRAIN ONLY (so
|
||||
the vol-target scale is not fit on the held-out data). Apply THAT direction + THAT k to TEST. Report the
|
||||
chosen direction, TRAIN Sharpe, TEST (OOS) Sharpe, and the TEST maxDD (the OOS tail)."""
|
||||
raws = {d: _book_short_var(store, direction=d) for d in _DIRECTIONS}
|
||||
days = sorted(set().union(*[set(r) for r in raws.values()]))
|
||||
if len(days) < 4:
|
||||
return {"train_direction": None, "train_sharpe": 0.0, "test_sharpe": 0.0,
|
||||
"test_max_dd": 0.0, "train_days": 0, "test_days": 0, "available": False}
|
||||
split_idx = max(1, min(len(days) - 1, int(len(days) * train_frac)))
|
||||
train_hi = days[split_idx - 1]
|
||||
test_lo = days[split_idx]
|
||||
|
||||
best_dir, best_sharpe, best_k = "short_vol", -math.inf, 0.0
|
||||
for d in _DIRECTIONS:
|
||||
raw = raws[d]
|
||||
k = _vol_target_k(raw, target_ann_vol=target_ann_vol, day_hi=train_hi)
|
||||
tr = _sized_series({day: raw[day] for day in raw if day <= train_hi}, k=k, cost_bps=cost_bps)
|
||||
s = _sharpe(tr)
|
||||
if s > best_sharpe:
|
||||
best_dir, best_sharpe, best_k = d, s, k
|
||||
|
||||
raw = raws[best_dir]
|
||||
test = _sized_series({day: raw[day] for day in raw if day >= test_lo}, k=best_k, cost_bps=cost_bps)
|
||||
return {"train_direction": best_dir, "train_sharpe": best_sharpe, "test_sharpe": _sharpe(test),
|
||||
"test_max_dd": _max_dd(test), "test_total_return": (math.prod(
|
||||
1.0 + test[d] for d in sorted(test)) - 1.0) if test else 0.0,
|
||||
"train_days": split_idx, "test_days": len(days) - split_idx,
|
||||
"split_at": _epoch_day_to_iso(test_lo), "available": True}
|
||||
|
||||
|
||||
def walk_forward_vrp(store: _FeatureStore, *, cost_bps: float = 5.5, window_days: int = 180,
|
||||
train_frac: float = 0.6, target_ann_vol: float = _DEFAULT_TARGET_VOL,
|
||||
max_dd_floor: float = -0.40, universe: set[str] | None = None) -> dict[str, Any]:
|
||||
"""READ-ONLY, TAIL-AWARE OOS / WALK-FORWARD validation of the VRP edge — the DEPLOY GATE.
|
||||
|
||||
Computes (all off the same return + `_curve_metrics`-style accounting as the in-sample verify):
|
||||
1. `train_holdout` — TRAIN picks the DIRECTION and sizes k (TRAIN-only); applied to the never-looked-at
|
||||
TEST. Reports `train_sharpe`, `test_sharpe` (OOS), `test_max_dd` (the OOS tail);
|
||||
2. `rolling_windows` — consecutive non-overlapping windows (per-window Sharpe/total/maxDD) +
|
||||
`fraction_positive` (a real edge is positive in MOST windows, not one calm regime);
|
||||
3. `look_ahead` — the strict-causality audit;
|
||||
4. `full` — the full-sample metrics at the TRAIN-chosen direction (for the headline + the full maxDD);
|
||||
5. `verdict` — a TAIL-AWARE PASS/FAIL. Deployable ONLY if:
|
||||
(a) OOS (TEST) Sharpe > 0 at the (realistic-option) cost,
|
||||
(b) positive in >1 rolling window (fraction_positive > 0.5) AND >70% positive,
|
||||
(c) the TAIL is SURVIVABLE: full-sample maxDD AND OOS-test maxDD are both above `max_dd_floor`
|
||||
(default −40%) — VRP looks great until a spike, so a catastrophic DD fails even if Sharpe>0,
|
||||
(d) look-ahead clean.
|
||||
|
||||
A clean "no / too-tail-heavy" is a VALID outcome (FAIL). READ-ONLY; `available=False` (+ reason) when
|
||||
there is no dvol+close data / too few days."""
|
||||
raw_probe = _book_short_var(store, direction="short_vol")
|
||||
if len(raw_probe) < 4:
|
||||
return {"available": False,
|
||||
"reason": "VRP curve has <4 usable days for a walk-forward split "
|
||||
"(run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)"}
|
||||
|
||||
train_holdout = _train_holdout(store, cost_bps=cost_bps, target_ann_vol=target_ann_vol,
|
||||
train_frac=train_frac)
|
||||
direction = train_holdout["train_direction"] or "short_vol"
|
||||
|
||||
raw = _book_short_var(store, direction=direction)
|
||||
k_full = _vol_target_k(raw, target_ann_vol=target_ann_vol)
|
||||
full_net = _sized_series(raw, k=k_full, cost_bps=cost_bps)
|
||||
full = _metrics_from_series(full_net, k=k_full)
|
||||
|
||||
rolling = _rolling_windows(full_net, window_days=window_days)
|
||||
audit = look_ahead_audit_vrp(store, direction=direction)
|
||||
|
||||
# --- TAIL-AWARE PASS/FAIL verdict ---------------------------------------------------------
|
||||
oos_pos = train_holdout["available"] and train_holdout["test_sharpe"] > 0.0
|
||||
rw_pos = rolling["fraction_positive"] > 0.70
|
||||
tail_survivable = (full["max_dd"] > max_dd_floor
|
||||
and train_holdout.get("test_max_dd", 0.0) > max_dd_floor)
|
||||
causal = audit["causal"]
|
||||
checks = {
|
||||
"oos_direction_positive": oos_pos,
|
||||
"rolling_windows_positive": rw_pos,
|
||||
"tail_survivable": tail_survivable,
|
||||
"look_ahead_clean": causal,
|
||||
}
|
||||
deployable = all(checks.values())
|
||||
|
||||
return {"available": True, "cost_bps": cost_bps, "window_days": window_days, "direction": direction,
|
||||
"target_ann_vol": target_ann_vol, "max_dd_floor": max_dd_floor,
|
||||
"avg_leverage": k_full, "full": full, "rolling_windows": rolling,
|
||||
"train_holdout": train_holdout, "look_ahead": audit,
|
||||
"verdict": {"deployable": deployable, "checks": checks},
|
||||
"cross_venue_note": "SIGNAL = Deribit DVOL; EXECUTION would be short Bybit option straddles"}
|
||||
@@ -1,331 +0,0 @@
|
||||
"""Executed VRP twin (sub-project C-style): place the atomic XSP put-credit-spread on IBKR and record the
|
||||
executed-vs-modeled reconciliation row. SUSPENDED until the account has options permission (no live OPRA in
|
||||
CI either, so the runtime path is not testable here).
|
||||
|
||||
Maintains a LADDER of open rungs in the `vrp_exec_pos` book-state (mirrors `VrpStrategy.advance`'s ladder
|
||||
discipline in `vrp_book.py`, not a single overwritten position): each run marks every open rung against
|
||||
today's frozen slice, closes any rung that hit 50% profit-take or DTE<=1 (placing the reverse combo), and
|
||||
opens AT MOST ONE new rung — sized at `envelope / ladder`, never the full envelope — when there is a free
|
||||
ladder slot on the weekly entry day. The 50%-profit-take DECISION uses the SAME smooth `vrp_marking` model
|
||||
value the backtest uses (single source) — not the raw short−long leg difference — so a noisy far-OTM long-leg
|
||||
quote cannot spuriously trip a live close; realized P&L and close order limits stay on the actual fill-basis
|
||||
mark. This bounds total open notional at ~envelope regardless of how many
|
||||
rungs are simultaneously open; armed weekly, it can never stack N spreads at full size.
|
||||
|
||||
`build_combo_legs`/`build_close_legs`, `refuse_if_live`, `plan_vrp_ladder`, `rung_notional`,
|
||||
`compute_spread_exec_return`/`compute_ladder_exec_return` and `build_pos_state` are pure and unit-tested.
|
||||
`plan_and_record_vrp` mirrors `multistrat_exec_record.plan_and_record`'s guard structure but sizes a LADDER
|
||||
of defined-risk spreads (not a continuous weight vector) — there is no B2a allocation for VRP yet, so new
|
||||
entries only happen when `paper_envelope > 0`; existing rungs still mark/close on a de-funded envelope
|
||||
(they cannot be silently stranded), matching the convention elsewhere that a de-funded envelope stops new
|
||||
risk but does not orphan positions."""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
from typing import Any
|
||||
|
||||
|
||||
def build_combo_legs(*, short_osi: str, long_osi: str, short_price: float, long_price: float,
|
||||
qty: int) -> tuple[list[tuple[str, str, int]], float]:
|
||||
"""Defined-risk put-credit-spread OPEN: SELL the short put, BUY the wing. Net limit is negative (a
|
||||
credit) — see `IbkrBroker.place_combo` for the parent-order sign convention this feeds."""
|
||||
legs = [(short_osi, "SELL", qty), (long_osi, "BUY", qty)]
|
||||
net = -(short_price - long_price) # credit received -> negative limit
|
||||
return legs, round(net, 2)
|
||||
|
||||
|
||||
def build_close_legs(*, short_osi: str, long_osi: str, qty: int,
|
||||
mark_today: float) -> tuple[list[tuple[str, str, int]], float]:
|
||||
"""Reverse of `build_combo_legs` to FLATTEN a held rung: BUY back the short (was sold), SELL the long
|
||||
(was bought). `net_limit` is POSITIVE — the debit paid to close, equal to `mark_today` (`_mark_spread`'s
|
||||
short-minus-long convention)."""
|
||||
legs = [(short_osi, "BUY", qty), (long_osi, "SELL", qty)]
|
||||
return legs, round(float(mark_today), 2)
|
||||
|
||||
|
||||
def refuse_if_live(paper_envelope: float, account_is_paper: bool) -> None:
|
||||
"""The real-capital exposure guard, factored out pure so it is testable without a live broker/repo: a
|
||||
PAPER-validation envelope (`paper_envelope > 0`) NEVER runs against a non-paper account. Must be the
|
||||
FIRST thing `plan_and_record_vrp` does — before any repo/broker/OPRA call — so a refusal places (and
|
||||
reads) exactly nothing."""
|
||||
if paper_envelope > 0.0 and not account_is_paper:
|
||||
raise ValueError("paper-envelope refused: account is not paper (real-capital exposure guard)")
|
||||
|
||||
|
||||
def _mark_spread(bars: Any, spread: dict[str, Any], day: str) -> float | None:
|
||||
"""ACTUAL cost to close `spread` on `day` from the frozen slice (short - long), or None if either leg has
|
||||
no mark today (aged/moved out of the frozen band). This is the real fill-basis mark — used for the
|
||||
executed-vs-modeled realized P&L and as the close order's net limit (the debit truly paid). The
|
||||
profit-take DECISION does NOT use this (it uses the smooth `vrp_marking` model mark) — a noisy far-OTM
|
||||
long-leg quote must not be able to spuriously trip a live close."""
|
||||
marks = bars.bars_for([spread["short_osi"], spread["long_osi"]], day, day)
|
||||
short_val = marks.get(spread["short_osi"], {}).get(day)
|
||||
long_val = marks.get(spread["long_osi"], {}).get(day)
|
||||
if short_val is None or long_val is None:
|
||||
return None
|
||||
return short_val - long_val
|
||||
|
||||
|
||||
def rung_notional(envelope: float, ladder: int) -> float:
|
||||
"""The per-rung dollar target: the envelope split evenly across the ladder's slots. Sizing every new
|
||||
entry at this (instead of the full envelope) bounds total open notional at ~envelope regardless of how
|
||||
many rungs are simultaneously open."""
|
||||
return envelope / ladder if ladder > 0 else 0.0
|
||||
|
||||
|
||||
def plan_vrp_ladder(open_spreads: list[dict[str, Any]], exec_marks: dict[str, float | None],
|
||||
signal_marks: dict[str, float | None], today: str, *,
|
||||
ladder: int, profit_take: float = 0.5, entry_weekday: int = 0
|
||||
) -> tuple[list[tuple[dict[str, Any], float]], bool, int]:
|
||||
"""Pure ladder-discipline decision (mirrors `VrpStrategy.advance`'s close-then-open management, for the
|
||||
single-day-per-run exec cadence). Both mark dicts are pre-fetched by the caller (keyed by `short_osi`) so
|
||||
this stays I/O-free:
|
||||
- `exec_marks` — the ACTUAL short−long fill-basis mark (`_mark_spread`); the close order's net limit
|
||||
and the HOLD gate (a rung with no actual price cannot be closed).
|
||||
- `signal_marks` — the SMOOTH `vrp_marking` model value (same valuation the backtest uses); the ONLY
|
||||
input to the 50%-profit-take decision, so a noisy far-OTM long-leg quote can't trip
|
||||
a spurious close.
|
||||
|
||||
Returns:
|
||||
to_close — [(spread, exec_mark)] rungs to close THIS run: 50%-profit-take
|
||||
(signal <= (1-profit_take)*credit) or DTE<=1. A rung with NO actual mark today (aged out of
|
||||
the frozen band) is a HOLD, not a close — it cannot be closed without a price (mirrors
|
||||
`compute_spread_exec_return`'s None-not-phantom-zero convention). The paired value is the
|
||||
ACTUAL mark (the real debit to close), NOT the model signal.
|
||||
may_open — True iff today is the weekly entry weekday AND the ladder has a free slot once this run's
|
||||
closes are applied (len(open_spreads) - len(to_close) < ladder).
|
||||
open_slots — 0 or 1 (a once-daily run opens at most one new rung, mirroring `VrpStrategy.advance`)."""
|
||||
to_close: list[tuple[dict[str, Any], float]] = []
|
||||
for sp in open_spreads:
|
||||
exec_mark = exec_marks.get(sp["short_osi"])
|
||||
if exec_mark is None:
|
||||
continue # no actual price -> cannot close (HOLD)
|
||||
expiry = dt.date.fromisoformat(sp["expiry"])
|
||||
dte = (expiry - dt.date.fromisoformat(today)).days
|
||||
signal = signal_marks.get(sp["short_osi"])
|
||||
# INTENTIONAL ref-vs-live ASYMMETRY (documented, not a bug): on a degenerate slice where this rung's
|
||||
# own put legs price (so `exec_mark` is a REAL actual close mark) but no ATM call/put pair prices at any
|
||||
# expiry (so `signal` / `vrp_marking.model_spread_value` == None), the pure-model backtest
|
||||
# (`vrp_book.advance`) DROPS the rung — its only mark IS the model value. The live twin instead HOLDS:
|
||||
# `take_profit` is False when `signal is None` (never profit-take on a missing model signal — a noisy or
|
||||
# absent model must NOT trip a live close), so the rung stays managed on its real fill mark until DTE<=1
|
||||
# or the model recovers. Correct: the twin has strictly MORE information than the recompute (a real
|
||||
# actual mark), so it need not discard a live position the way the info-poorer backtest must.
|
||||
take_profit = signal is not None and signal <= (1.0 - profit_take) * sp["credit"]
|
||||
if take_profit or dte <= 1:
|
||||
to_close.append((sp, exec_mark)) # close at the ACTUAL mark, decide on the model signal
|
||||
remaining = len(open_spreads) - len(to_close)
|
||||
is_entry_day = dt.date.fromisoformat(today).weekday() == entry_weekday
|
||||
may_open = is_entry_day and remaining < ladder
|
||||
return to_close, may_open, (1 if may_open else 0)
|
||||
|
||||
|
||||
def compute_spread_exec_return(prior_val: float, mark_today: float | None, qty: int,
|
||||
envelope: float) -> float | None:
|
||||
"""Envelope-basis executed return of ONE rung since its last mark: the mark-to-market P&L, as a fraction
|
||||
of the shared ladder envelope. Short-vol: value falling = profit. The $100/contract multiplier converts
|
||||
the frozen per-share option marks into real dollars. None on a de-funded envelope or when today's freeze
|
||||
is missing a mark for the held legs (a HOLD, not a phantom zero)."""
|
||||
if envelope <= 0.0 or mark_today is None:
|
||||
return None
|
||||
pnl = (prior_val - mark_today) * 100.0 * qty
|
||||
return pnl / envelope
|
||||
|
||||
|
||||
def compute_ladder_exec_return(open_spreads_prior: list[dict[str, Any]], marks_today: dict[str, float | None],
|
||||
envelope_prior: float) -> float | None:
|
||||
"""Sum `compute_spread_exec_return` across every rung held since the last run (mirrors
|
||||
`multistrat_exec_record.compute_exec_return`'s book-level pattern, one rung at a time). None on
|
||||
inception (no prior rungs), a de-funded prior envelope, or when NOT ONE rung produced a mark today (an
|
||||
all-HOLD run — not a phantom zero); a rung with no mark today simply contributes 0 to the sum, same as
|
||||
`VrpStrategy.advance`'s per-day loop skipping unmarked spreads."""
|
||||
if not open_spreads_prior or envelope_prior <= 0.0:
|
||||
return None
|
||||
total = 0.0
|
||||
any_marked = False
|
||||
for sp in open_spreads_prior:
|
||||
mark = marks_today.get(sp["short_osi"])
|
||||
r = compute_spread_exec_return(float(sp.get("last_val", sp["credit"])), mark, int(sp.get("qty", 0)),
|
||||
envelope_prior)
|
||||
if r is not None:
|
||||
total += r
|
||||
any_marked = True
|
||||
return total if any_marked else None
|
||||
|
||||
|
||||
def build_pos_state(open_spreads: list[dict[str, Any]], envelope: float, venue: str,
|
||||
today: str) -> dict[str, Any]:
|
||||
return {"open_spreads": [dict(sp) for sp in open_spreads], "envelope": float(envelope), "venue": str(venue),
|
||||
"last_run_date": today}
|
||||
|
||||
|
||||
class _VrpExecBookingStrategy:
|
||||
"""A trivial observed ForwardStrategy for run_track: books today's pre-computed executed return (or
|
||||
nothing on inception/no-mark). Mirrors `multistrat_exec_record._ExecBookingStrategy`."""
|
||||
def __init__(self, exec_return: float | None) -> None:
|
||||
self._r = exec_return
|
||||
|
||||
def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]:
|
||||
if self._r is None:
|
||||
return [], extra
|
||||
return [(str(extra.get("_today")), float(self._r))], extra
|
||||
|
||||
|
||||
def _record(repo: Any, exec_return: float | None, *, open_spreads: list[dict[str, Any]], envelope: float,
|
||||
venue: str, today: str, at: dt.datetime) -> None:
|
||||
from fxhnt.application.forward_engine import run_track
|
||||
run_track(repo, "vrp_exec", _VrpExecBookingStrategy(exec_return), today=today, at=at)
|
||||
repo.set_book_state("vrp_exec_pos", build_pos_state(open_spreads, envelope, venue, today), at=at)
|
||||
|
||||
|
||||
def plan_and_record_vrp(repo: Any, *, dbn: Any, broker: Any, nlv: float, today: str, at: dt.datetime,
|
||||
paper_envelope: float, account_is_paper: bool, venue: str, execute: bool) -> dict[str, Any]:
|
||||
"""Mark every PRIOR held rung (if any) against today's frozen slice, apply the ladder discipline (close
|
||||
rungs at 50% profit / DTE<=1, cap entries at `ladder`, size a new rung at `envelope/ladder`), and — when
|
||||
`execute` — place the closing/opening combos and record the observed `vrp_exec` return.
|
||||
|
||||
Guards (mirrors `multistrat_exec_record.plan_and_record` guard-for-guard):
|
||||
1. `refuse_if_live` — paper-envelope on a non-paper account raises BEFORE any repo/broker/OPRA call
|
||||
(zero orders placed, zero rows read/written).
|
||||
2. per-day idempotency — a `vrp_exec_pos` book-state already stamped `today` is a no-op re-run.
|
||||
3. a `broker.place_combo` failure (open OR close) becomes a structured entry in `errors` (never a
|
||||
crash); a failed CLOSE re-adds the rung UNCHANGED to the ladder (mirrors the bybit per-order gap-row
|
||||
+ kill-switch discipline: a failed order must not silently advance the position basis). The OPEN is
|
||||
gated on the ACTUAL post-close ladder occupancy (not the pre-execution `may_open` plan), so a failed
|
||||
close that leaves the ladder full skips the new entry instead of breaching the cap.
|
||||
4. recording is skipped entirely when `execute` is False (dry-run plans only, same as multistrat)."""
|
||||
refuse_if_live(paper_envelope, account_is_paper)
|
||||
|
||||
pos_prior = repo.get_book_state("vrp_exec_pos")
|
||||
if pos_prior.get("last_run_date") == today:
|
||||
return {"noop": True, "reason": f"already ran on {today}", "exec_return": None, "placed": 0,
|
||||
"closed": 0, "spread": None}
|
||||
|
||||
open_spreads_prior: list[dict[str, Any]] = list(pos_prior.get("open_spreads", []))
|
||||
envelope_prior = float(pos_prior.get("envelope", 0.0))
|
||||
envelope = min(float(paper_envelope), float(nlv)) if paper_envelope > 0.0 else 0.0
|
||||
|
||||
from fxhnt.adapters.orchestration.assets import _freeze_latest_session
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
from fxhnt.config import get_settings
|
||||
|
||||
dsn = getattr(repo, "_dsn", None) or get_settings().operational_dsn
|
||||
bars = XspOptionBarsRepo(dsn)
|
||||
bars.migrate()
|
||||
# TWO dates, deliberately distinct: OPRA publishes on a ~1-session lag, so `today`'s slice is NEVER
|
||||
# available intraday — freezing `today` crashes a live run with DatabentoRangeUnavailable before it ever
|
||||
# reaches the broker. Freeze/mark against the last COMPLETE OPRA session (`freeze_date`, T-1, via the same
|
||||
# robust bounded step-back the nightly `vrp_nav` uses); use `today` (calendar) for all date-LOGIC — DTE,
|
||||
# entry-weekday, the idempotency key, `run_track` booking and the pos-state stamp. Only the DATA date lags.
|
||||
freeze_date = _freeze_latest_session(dbn, bars, at=at) # last complete OPRA session (data lag), ISO date
|
||||
|
||||
from fxhnt.application.vrp_book import VrpStrategy
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
|
||||
ladder = int(STRATEGY_REGISTRY["vrp"]["definition"]["params"]["ladder"])
|
||||
strat = VrpStrategy(bars, ladder=ladder)
|
||||
|
||||
from fxhnt.application import vrp_marking
|
||||
|
||||
# ACTUAL fill-basis marks (short−long) for the realized exec P&L and the close order net limits — read at
|
||||
# the frozen data session, not `today` (whose slice OPRA hasn't published yet -> would come back None).
|
||||
marks_today = {sp["short_osi"]: _mark_spread(bars, sp, freeze_date) for sp in open_spreads_prior}
|
||||
exec_return = compute_ladder_exec_return(open_spreads_prior, marks_today, envelope_prior)
|
||||
|
||||
# SMOOTH model marks (the SAME `vrp_marking` valuation the backtest uses) drive the 50%-profit-take
|
||||
# decision, so a noisy far-OTM long-leg quote can't spuriously trip a live close. Memoized per expiry.
|
||||
signal_cache: dict[tuple[str, str], tuple[float | None, float | None]] = {}
|
||||
signal_marks = {sp["short_osi"]: vrp_marking.model_spread_value(bars, sp, freeze_date, strat.margin / 100.0,
|
||||
signal_cache)
|
||||
for sp in open_spreads_prior}
|
||||
|
||||
to_close, may_open, _open_slots = plan_vrp_ladder(
|
||||
open_spreads_prior, marks_today, signal_marks, today, ladder=ladder,
|
||||
profit_take=strat._profit_take, entry_weekday=strat._entry_weekday) # noqa: SLF001 — composition
|
||||
# root reaches into the sim's private ladder-management params so the exec twin and the backtest
|
||||
# never drift apart.
|
||||
|
||||
close_osi = {sp["short_osi"] for sp, _mark in to_close}
|
||||
kept: list[dict[str, Any]] = []
|
||||
for sp in open_spreads_prior:
|
||||
if sp["short_osi"] in close_osi:
|
||||
continue
|
||||
m = marks_today.get(sp["short_osi"])
|
||||
kept.append(dict(sp, last_val=m) if m is not None else sp)
|
||||
|
||||
result: dict[str, Any] = {"exec_return": exec_return, "placed": 0, "closed": 0, "to_close": len(to_close),
|
||||
"errors": [], "notes": [], "spread": None}
|
||||
|
||||
candidate: dict[str, Any] | None = None
|
||||
qty_new = 0
|
||||
legs_new: list[tuple[str, str, int]] | None = None
|
||||
net_new: float | None = None
|
||||
if envelope <= 0.0:
|
||||
result["notes"].append("no envelope (paper-envelope off, no B2a allocation for vrp_exec yet) — no new rungs")
|
||||
elif may_open:
|
||||
# select + price the new rung off the FROZEN data session (freeze_date), not `today` (unpublished).
|
||||
candidate = strat._select_spread(freeze_date) # noqa: SLF001 — composition root reaches into the selector
|
||||
if candidate is None:
|
||||
result["notes"].append("no candidate spread selected for today")
|
||||
else:
|
||||
marks = bars.bars_for([candidate["short_osi"], candidate["long_osi"]], freeze_date, freeze_date)
|
||||
short_price = marks.get(candidate["short_osi"], {}).get(freeze_date)
|
||||
long_price = marks.get(candidate["long_osi"], {}).get(freeze_date)
|
||||
if short_price is None or long_price is None:
|
||||
raise RuntimeError(f"vrp_exec: selected spread has no frozen mark for {freeze_date}")
|
||||
rung_target = rung_notional(envelope, ladder)
|
||||
qty_new = int(rung_target // strat.margin)
|
||||
if qty_new < 1:
|
||||
result["notes"].append(
|
||||
f"rung target ${rung_target:,.0f} (envelope/{ladder}) < margin ${strat.margin:,.0f}/contract"
|
||||
" — no new rung")
|
||||
candidate = None
|
||||
else:
|
||||
legs_new, net_new = build_combo_legs(
|
||||
short_osi=candidate["short_osi"], long_osi=candidate["long_osi"],
|
||||
short_price=short_price, long_price=long_price, qty=qty_new)
|
||||
result.update(spread=candidate, legs=legs_new, net=net_new, qty=qty_new)
|
||||
else:
|
||||
result["notes"].append("ladder full or off entry day — no new rung")
|
||||
|
||||
if not execute:
|
||||
return result
|
||||
|
||||
open_spreads_next = list(kept)
|
||||
|
||||
for sp, mark in to_close:
|
||||
close_legs, close_net = build_close_legs(short_osi=sp["short_osi"], long_osi=sp["long_osi"],
|
||||
qty=int(sp["qty"]), mark_today=mark)
|
||||
try:
|
||||
broker.place_combo(close_legs, close_net)
|
||||
result["closed"] += 1
|
||||
except Exception as e: # a failed CLOSE must not silently drop the rung -- re-add it UNCHANGED.
|
||||
result["errors"].append(f"close {sp['short_osi']}: {type(e).__name__}: {str(e)[:120]}")
|
||||
open_spreads_next.append(sp)
|
||||
|
||||
if candidate is not None and legs_new is not None and net_new is not None:
|
||||
# Gate the OPEN on the ACTUAL post-close occupancy, not the pre-execution plan: `may_open` above
|
||||
# was computed from `to_close` before any order was placed. If a CLOSE above failed, its rung was
|
||||
# re-added to `open_spreads_next` unchanged, so the ladder may still be at/over cap here even though
|
||||
# `may_open` was True. Re-checking `len(open_spreads_next) < ladder` at this point makes a failed
|
||||
# close naturally back-pressure new entries instead of breaching the cap (and, since DTE only
|
||||
# decreases, prevents the un-closable rung from re-freeing a phantom slot every subsequent run).
|
||||
if len(open_spreads_next) >= ladder:
|
||||
result["errors"].append(
|
||||
f"open skipped: ladder full after close failures ({len(open_spreads_next)}/{ladder} occupied)")
|
||||
result["spread"] = None
|
||||
else:
|
||||
try:
|
||||
order_id = broker.place_combo(legs_new, net_new)
|
||||
result["placed"] = 1
|
||||
result["order_id"] = order_id
|
||||
opened = dict(candidate, qty=qty_new, last_val=candidate["credit"])
|
||||
open_spreads_next.append(opened)
|
||||
result["spread"] = opened
|
||||
except Exception as e: # a failed OPEN just means no new rung this run -- ladder stays as-is.
|
||||
result["errors"].append(f"open: {type(e).__name__}: {str(e)[:120]}")
|
||||
result["spread"] = None
|
||||
|
||||
_record(repo, exec_return, open_spreads=open_spreads_next, envelope=envelope, venue=venue, today=today, at=at)
|
||||
result["open_spreads"] = len(open_spreads_next)
|
||||
return result
|
||||
@@ -1,116 +0,0 @@
|
||||
"""SINGLE SOURCE of the VRP defined-risk spread's MODEL mark — the smooth Black-Scholes value used as the
|
||||
50%-profit-take management SIGNAL by BOTH the backtest (`vrp_book.VrpStrategy`) and the live exec twin
|
||||
(`vrp_exec_record`). Rebuilt each day from the put-call-parity forward + a robust ATM implied vol (a single
|
||||
CLEAN near-the-money read), then clipped to the defined-risk bound [0, width] — never the difference of two
|
||||
individually-noisy raw OPRA leg quotes (a ~$1 spike in a far-OTM long leg swung that raw difference 50-100%
|
||||
and could spuriously trip a profit-take close).
|
||||
|
||||
Only the management SIGNAL uses this model value; realized/entry P&L on real fills stays mark-to-actual.
|
||||
|
||||
The forward is derived over a WIDE DTE window and, because the parity forward is spot (common across
|
||||
expiries to first order under the r≈0 forward measure), falls back to the globally cleanest ATM call/put pair
|
||||
when the spread's OWN expiry has no frozen calls — the daily freeze band freezes near-ATM calls only for
|
||||
DTE∈[20,45], so a spread aged below 20 DTE has puts (band DTE∈[1,45]) but no calls at its expiry. Without
|
||||
this fallback such a spread could not be marked and would be spuriously closed."""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
from typing import Any
|
||||
|
||||
from fxhnt.application import vrp_pricing
|
||||
|
||||
# Wide DTE window for MARKING an already-open spread (any age, incl. below the selection dte_lo). Large
|
||||
# enough to cover the longest listed expiry; callers filter to the relevant expiry.
|
||||
MARK_DTE_HI = 3650
|
||||
|
||||
|
||||
def forward_and_atm_iv(bars: Any, day: str, exp: dt.date, tau: float) -> tuple[float | None, float | None]:
|
||||
"""(forward F, ATM IV) for expiry `exp` on `day`, derived purely from the frozen slice.
|
||||
|
||||
forward = put-call parity at the ATM strike (smallest |C − P|, ties → lowest strike), preferring the
|
||||
spread's OWN expiry and falling back to the globally cleanest ATM pair across all frozen expiries (spot is
|
||||
common). Rejects an implausible forward. ATM IV = `implied_vol_put` inverted from `exp`'s put nearest the
|
||||
forward (puts are frozen at any age). Returns (None, None) when no clean ATM pair exists; (F, None) when
|
||||
the forward is clean but IV inversion fails (the caller carries the last good IV)."""
|
||||
puts = bars.chain_asof(day, 0, MARK_DTE_HI, right="P")
|
||||
calls = bars.chain_asof(day, 0, MARK_DTE_HI, right="C")
|
||||
if not puts or not calls:
|
||||
return None, None
|
||||
put_marks = bars.bars_for([c.osi_symbol for c in puts], day, day)
|
||||
call_marks = bars.bars_for([c.osi_symbol for c in calls], day, day)
|
||||
# priced ATM candidates keyed by (expiry, strike): both a positive put AND call mark at the same strike.
|
||||
put_by = {(c.expiry, c.strike): p for c in puts
|
||||
if (p := put_marks.get(c.osi_symbol, {}).get(day)) is not None and p > 0.0}
|
||||
call_by = {(c.expiry, c.strike): p for c in calls
|
||||
if (p := call_marks.get(c.osi_symbol, {}).get(day)) is not None and p > 0.0}
|
||||
common = set(put_by) & set(call_by)
|
||||
# prefer the spread's own expiry (entry-time behaviour, preserved); else any expiry (spot is common).
|
||||
same_exp = [ek for ek in common if ek[0] == exp]
|
||||
candidates = same_exp or list(common)
|
||||
if not candidates:
|
||||
return None, None
|
||||
e_atm, k_atm = min(candidates, key=lambda ek: (abs(call_by[ek] - put_by[ek]), ek[1], ek[0]))
|
||||
forward = vrp_pricing.parity_forward(call_atm=call_by[(e_atm, k_atm)], put_atm=put_by[(e_atm, k_atm)],
|
||||
k_atm=k_atm)
|
||||
if not (0.2 * k_atm < forward < 5.0 * k_atm): # implausible parity forward -> never mark off it
|
||||
return None, None
|
||||
# ATM IV from the spread's OWN expiry put nearest the forward (present at any age).
|
||||
exp_puts = {k: p for (e, k), p in put_by.items() if e == exp}
|
||||
if not exp_puts:
|
||||
return forward, None
|
||||
k_iv = min(exp_puts, key=lambda k: abs(k - forward))
|
||||
iv = vrp_pricing.implied_vol_put(exp_puts[k_iv], forward, k_iv, tau)
|
||||
return forward, iv
|
||||
|
||||
|
||||
def model_spread_value(bars: Any, spread: dict[str, Any], day: str, width: float,
|
||||
day_cache: dict[tuple[str, str], tuple[float | None, float | None]] | None = None
|
||||
) -> float | None:
|
||||
"""Smooth MODEL value (cost to close) of `spread` on `day`, clipped to the defined-risk bound [0, width].
|
||||
Carries the last good IV on `spread['last_iv']`. Returns None when the strikes are unknown or the expiry
|
||||
has no clean forward (caller CLOSES — no zombie). `day_cache` (optional) memoizes (F, IV) per
|
||||
(day, expiry) so spreads sharing an expiry in one pass don't re-query the chain."""
|
||||
ks, kl = spread.get("short_strike"), spread.get("long_strike")
|
||||
if ks is None or kl is None:
|
||||
return None # unknown strikes -> no model signal (safe: no close)
|
||||
exp = dt.date.fromisoformat(spread["expiry"])
|
||||
tau = max((exp - dt.date.fromisoformat(day)).days, 1) / 365.0 # tau > 0 guard
|
||||
key = (day, spread["expiry"])
|
||||
if day_cache is not None and key in day_cache:
|
||||
forward, iv = day_cache[key]
|
||||
else:
|
||||
forward, iv = forward_and_atm_iv(bars, day, exp, tau)
|
||||
if day_cache is not None:
|
||||
day_cache[key] = (forward, iv)
|
||||
if forward is None:
|
||||
return None # expiry chain gone -> close (not zombie)
|
||||
if iv is None:
|
||||
iv = spread.get("last_iv") # carry the last good IV on a degenerate mark day
|
||||
if iv is None:
|
||||
return None
|
||||
spread["last_iv"] = iv
|
||||
model = vrp_pricing.bs_put(forward, ks, iv, tau) - vrp_pricing.bs_put(forward, kl, iv, tau)
|
||||
return min(max(model, 0.0), width) # hard defined-risk clip [0, width]
|
||||
|
||||
|
||||
def settlement_value(bars: Any, spread: dict[str, Any], day: str, width: float,
|
||||
day_cache: dict[tuple[str, str], tuple[float | None, float | None]] | None = None
|
||||
) -> float | None:
|
||||
"""Terminal settlement value from `day`'s forward: the put-credit-spread's loss is the short strike's
|
||||
in-the-moneyness capped at the width — clip(K_short − F, 0, width). Realized on the close day (DTE≤1)
|
||||
instead of the theta model mark."""
|
||||
ks = spread.get("short_strike")
|
||||
if ks is None:
|
||||
return None
|
||||
exp = dt.date.fromisoformat(spread["expiry"])
|
||||
tau = max((exp - dt.date.fromisoformat(day)).days, 1) / 365.0
|
||||
key = (day, spread["expiry"])
|
||||
if day_cache is not None and key in day_cache:
|
||||
forward, _iv = day_cache[key]
|
||||
else:
|
||||
forward, _iv = forward_and_atm_iv(bars, day, exp, tau)
|
||||
if day_cache is not None:
|
||||
day_cache[key] = (forward, _iv)
|
||||
if forward is None:
|
||||
return None
|
||||
return min(max(ks - forward, 0.0), width)
|
||||
@@ -1,57 +0,0 @@
|
||||
"""Pure Black–Scholes helpers for the defined-risk XSP VRP sleeve — priced under the FORWARD measure so
|
||||
the underlying comes from put-call parity on the option chain (no external index feed; OPRA is options-only).
|
||||
No I/O: every input is a real OPRA mark or a frozen bar. r is folded into `disc = e^{-rT}` and the forward F."""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
_SQRT2 = math.sqrt(2.0)
|
||||
|
||||
|
||||
def _norm_cdf(x: float) -> float:
|
||||
return 0.5 * (1.0 + math.erf(x / _SQRT2))
|
||||
|
||||
|
||||
def bs_put(F: float, K: float, sigma: float, tau: float, disc: float = 1.0) -> float:
|
||||
"""European put under the forward measure: disc*(K*N(-d2) - F*N(-d1)). disc = e^{-rT} (1.0 for r≈0)."""
|
||||
if tau <= 0.0 or sigma <= 0.0 or F <= 0.0 or K <= 0.0:
|
||||
return max(0.0, disc * (K - F)) # intrinsic (degenerate day)
|
||||
vol = sigma * math.sqrt(tau)
|
||||
d1 = (math.log(F / K) + 0.5 * sigma * sigma * tau) / vol
|
||||
d2 = d1 - vol
|
||||
return disc * (K * _norm_cdf(-d2) - F * _norm_cdf(-d1))
|
||||
|
||||
|
||||
def put_delta(F: float, K: float, sigma: float, tau: float) -> float:
|
||||
"""|Δ| of the put under the forward measure = N(-d1) in [0,1]. Used for ~20-delta strike selection."""
|
||||
if tau <= 0.0 or sigma <= 0.0 or F <= 0.0 or K <= 0.0:
|
||||
return 1.0 if K > F else 0.0
|
||||
vol = sigma * math.sqrt(tau)
|
||||
d1 = (math.log(F / K) + 0.5 * sigma * sigma * tau) / vol
|
||||
return _norm_cdf(-d1)
|
||||
|
||||
|
||||
def implied_vol_put(price: float, F: float, K: float, tau: float, disc: float = 1.0) -> float | None:
|
||||
"""Invert bs_put for sigma via bisection. Returns None when `price` is below intrinsic / above the bound
|
||||
(no arb-free vol) — the caller skips that contract for the day."""
|
||||
if tau <= 0.0 or price <= 0.0:
|
||||
return None
|
||||
intrinsic = max(0.0, disc * (K - F))
|
||||
upper_bound = disc * K # put <= disc*K (sigma -> inf)
|
||||
if price < intrinsic - 1e-9 or price > upper_bound + 1e-9:
|
||||
return None
|
||||
lo, hi = 1e-4, 5.0
|
||||
if bs_put(F, K, hi, tau, disc) < price: # unreachable even at 500% vol
|
||||
return None
|
||||
for _ in range(80):
|
||||
mid = 0.5 * (lo + hi)
|
||||
if bs_put(F, K, mid, tau, disc) < price:
|
||||
lo = mid
|
||||
else:
|
||||
hi = mid
|
||||
return 0.5 * (lo + hi)
|
||||
|
||||
|
||||
def parity_forward(call_atm: float, put_atm: float, k_atm: float, disc: float = 1.0) -> float:
|
||||
"""Implied forward from put-call parity: C - P = disc*(F - K) => F = K + (C - P)/disc."""
|
||||
return k_atm + (call_atm - put_atm) / disc
|
||||
@@ -1,116 +0,0 @@
|
||||
"""Equity-index Volatility Risk Premium (VRP) — a STRUCTURAL risk premium (not momentum): implied vol
|
||||
exceeds subsequent realized vol on average because option sellers are paid to bear crash risk. Short vol
|
||||
harvests IV − RV. The catch (and why the deflated Sharpe matters): VRP returns are negatively skewed and
|
||||
fat-tailed (vol spikes = big losses), so a raw Sharpe overstates it — the skew/kurtosis-adjusted deflated
|
||||
Sharpe is the honest measure. Built on XSP (mini-SPX) options, 2013-2026, internal put-call-parity (vrp3).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import glob
|
||||
import math
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
from fxhnt.domain.validation import deflated_sharpe_ratio, excess_kurtosis, sharpe_ratio, skewness
|
||||
|
||||
_DAY_NS = 86_400 * 10**9
|
||||
|
||||
|
||||
def _to_epoch_day(yymmdd: str) -> int:
|
||||
y, m, d = 2000 + int(yymmdd[:2]), int(yymmdd[2:4]), int(yymmdd[4:6])
|
||||
return (dt.date(y, m, d) - dt.date(1970, 1, 1)).days
|
||||
|
||||
|
||||
def _load_xsp(xsp_dir: str): # type: ignore[no-untyped-def]
|
||||
import databento as db
|
||||
import pandas as pd
|
||||
|
||||
rows = []
|
||||
for p in sorted(glob.glob(os.path.join(xsp_dir, "*.dbn"))):
|
||||
try:
|
||||
df = db.DBNStore.from_file(p).to_df().reset_index()
|
||||
except Exception:
|
||||
continue
|
||||
if df.empty or "symbol" not in df.columns:
|
||||
continue
|
||||
df = df[df["close"] > 0]
|
||||
rows.append(df[["ts_event", "symbol", "close"]])
|
||||
d = pd.concat(rows, ignore_index=True)
|
||||
s = d["symbol"].astype(str).str.replace(" ", "", regex=False) # XSP240119C00450000
|
||||
d["right"] = s.str[-9]
|
||||
d["strike"] = s.str[-8:].astype(float) / 1000.0
|
||||
d["expiry"] = s.str[-15:-9]
|
||||
d["day"] = d["ts_event"].astype("int64") // _DAY_NS
|
||||
return d
|
||||
|
||||
|
||||
@dataclass
|
||||
class VrpResult:
|
||||
n_periods: int
|
||||
mean_vrp: float
|
||||
sharpe: float
|
||||
skew: float
|
||||
excess_kurt: float
|
||||
deflated_sharpe: float # skew/kurtosis-adjusted (the honest measure)
|
||||
dsr_pvalue: float
|
||||
is_sharpe: float
|
||||
oos_sharpe: float
|
||||
max_drawdown: float
|
||||
|
||||
|
||||
def evaluate(xsp_dir: str, *, holding_days: int = 21, n_trials: int = 5, oos_fraction: float = 0.40) -> VrpResult:
|
||||
import pandas as pd
|
||||
|
||||
d = _load_xsp(xsp_dir)
|
||||
d["exp_day"] = d["expiry"].map(_to_epoch_day)
|
||||
d["ttm"] = d["exp_day"] - d["day"]
|
||||
d = d[(d["ttm"] >= 20) & (d["ttm"] <= 45)]
|
||||
|
||||
forward, iv = {}, {}
|
||||
for day, g in d.groupby("day"):
|
||||
exp = g.iloc[(g["ttm"] - 30).abs().argsort()].iloc[0]["exp_day"]
|
||||
ge = g[g["exp_day"] == exp]
|
||||
calls = ge[ge["right"] == "C"].groupby("strike")["close"].mean()
|
||||
puts = ge[ge["right"] == "P"].groupby("strike")["close"].mean()
|
||||
common = np.array(sorted(set(calls.index) & set(puts.index)))
|
||||
if len(common) < 5:
|
||||
continue
|
||||
rough = common[np.abs(calls[common].values - puts[common].values).argmin()]
|
||||
near = common[np.argsort(np.abs(common - rough))[:5]]
|
||||
f = float(np.median([k + float(calls[k]) - float(puts[k]) for k in near]))
|
||||
katm = common[np.abs(common - f).argmin()]
|
||||
straddle = float(calls[katm] + puts[katm])
|
||||
t_yrs = float(ge["ttm"].iloc[0]) / 365.0
|
||||
if f <= 0 or straddle <= 0 or t_yrs <= 0:
|
||||
continue
|
||||
forward[int(day)] = f
|
||||
iv[int(day)] = straddle / (0.8 * f * math.sqrt(t_yrs)) # ATM-straddle implied-vol approximation
|
||||
|
||||
days = np.array(sorted(forward))
|
||||
fwd = np.array([forward[x] for x in days])
|
||||
iv_arr = np.array([iv[x] for x in days])
|
||||
logret = np.zeros(len(days))
|
||||
logret[1:] = np.clip(np.log(fwd[1:] / fwd[:-1]), -0.25, 0.25)
|
||||
|
||||
# NON-OVERLAPPING short-vol periods: enter, hold `holding_days`, P&L proxy = IV_entry − RV_realized
|
||||
returns = []
|
||||
step = holding_days
|
||||
for t in range(0, len(days) - step, step):
|
||||
rv = float(np.std(logret[t + 1:t + 1 + step]) * math.sqrt(252))
|
||||
returns.append(iv_arr[t] - rv)
|
||||
r = np.array(returns)
|
||||
if len(r) < 8:
|
||||
raise ValueError("not enough VRP periods")
|
||||
|
||||
sk, ek = skewness(r), excess_kurtosis(r)
|
||||
dsr = deflated_sharpe_ratio(sharpe_ratio(r), n_trials, 0.0, sk, ek, len(r))
|
||||
split = int((1.0 - oos_fraction) * len(r))
|
||||
eq = np.cumprod(1.0 + r / 100.0) # scale vol-points to a rough equity curve
|
||||
mdd = float((eq / np.maximum.accumulate(eq) - 1.0).min())
|
||||
return VrpResult(
|
||||
n_periods=len(r), mean_vrp=float(r.mean()), sharpe=sharpe_ratio(r), skew=sk, excess_kurt=ek,
|
||||
deflated_sharpe=1.0 - dsr.pvalue, dsr_pvalue=dsr.pvalue,
|
||||
is_sharpe=sharpe_ratio(r[:split]), oos_sharpe=sharpe_ratio(r[split:]), max_drawdown=mdd)
|
||||
460
src/fxhnt/cli.py
460
src/fxhnt/cli.py
@@ -436,69 +436,6 @@ def execute_bybit(
|
||||
f"| exec_return {out.get('exec_return')}")
|
||||
|
||||
|
||||
@app.command("execute-vrp")
|
||||
def execute_vrp(
|
||||
do_execute: bool = typer.Option(False, "--execute", help="place combo orders (default: dry-run plan only)"),
|
||||
paper_envelope: float = typer.Option(
|
||||
0.0, "--paper-envelope",
|
||||
help="PAPER-only validation envelope in $ (there is no B2a allocation for VRP yet, so this is the "
|
||||
"ONLY way this track sizes a new ladder rung; 0 = no new rungs, existing rungs still mark/close)."),
|
||||
) -> None:
|
||||
"""Reconcile + record the EXECUTED XSP put-credit-spread VRP twin on IBKR (`mleg` combo), maintaining a
|
||||
ladder of open rungs (mirrors `VrpStrategy.advance`'s discipline: close a rung at 50% profit or DTE<=1,
|
||||
cap entries at `ladder`, size each new rung at envelope/ladder — never the full envelope). SUSPENDED: the
|
||||
IBKR account currently lacks options trading permission and CI has no live OPRA, so this path is coded
|
||||
but not runnable here — dry-run by default even once armed. Same paper-envelope refusal guard as
|
||||
`execute-multistrat`/`execute-bybit`. Records the observed `vrp_exec` return."""
|
||||
import datetime as _dt
|
||||
|
||||
from fxhnt.adapters.persistence.forward_nav import ForwardNavRepo
|
||||
from fxhnt.application.vrp_exec_record import plan_and_record_vrp
|
||||
|
||||
settings = get_settings()
|
||||
d = settings.databento
|
||||
dbn = DatabentoDataProvider(max_cost_usd=d.max_cost_usd, equity_dataset=d.equity_dataset,
|
||||
future_dataset=d.future_dataset, default_start=d.default_start)
|
||||
|
||||
repo = ForwardNavRepo(settings.operational_dsn)
|
||||
repo.migrate()
|
||||
today = _dt.datetime.now(_dt.UTC).strftime("%Y-%m-%d")
|
||||
at = _dt.datetime.now(_dt.UTC).replace(tzinfo=None)
|
||||
|
||||
broker = IbkrBroker(settings.ibkr.host, settings.ibkr.port, settings.ibkr.client_id, settings.ibkr.timeout)
|
||||
paper_env = _resolve_paper_envelope(paper_envelope, settings.paper_validation.envelope)
|
||||
try:
|
||||
with broker as brk:
|
||||
acct = brk.account_state()
|
||||
venue = _exec_venue_string(brk.name, acct.is_paper)
|
||||
out = plan_and_record_vrp(
|
||||
repo, dbn=dbn, broker=brk, nlv=acct.nlv, today=today, at=at,
|
||||
paper_envelope=paper_env, account_is_paper=acct.is_paper, venue=venue, execute=do_execute)
|
||||
except ValueError as e: # a deliberate safety refusal (e.g. paper-envelope on a non-paper account) —
|
||||
typer.echo(f"paper-envelope refused: {e}") # distinct from a connectivity/data error
|
||||
raise typer.Exit(1) from e
|
||||
except Exception as e: # connection / OPRA / runtime
|
||||
typer.echo(f"broker error: {type(e).__name__}: {str(e)[:100]}")
|
||||
typer.echo(f" reachable? IBKR ({settings.ibkr.host}:{settings.ibkr.port}) "
|
||||
f"— creds/gateway + options permission set? OPRA reachable?")
|
||||
raise typer.Exit(1) from e
|
||||
|
||||
if out.get("noop"):
|
||||
typer.echo(f"-> no-op: {out.get('reason')}")
|
||||
return
|
||||
typer.echo(f"to_close={out.get('to_close', 0)} closed={out.get('closed', 0)}")
|
||||
spread = out.get("spread")
|
||||
if spread:
|
||||
typer.echo(f"spread: SELL {spread['short_osi']} / BUY {spread['long_osi']} qty={out.get('qty')} "
|
||||
f"net ${out.get('net'):+.2f}")
|
||||
for n in out.get("notes", []):
|
||||
typer.echo(f" - {n}")
|
||||
for err in out.get("errors", []):
|
||||
typer.echo(f" order error: {err}")
|
||||
typer.echo(f"-> exec_return {out.get('exec_return')} | placed {out.get('placed', 0)} "
|
||||
f"| open_spreads {out.get('open_spreads')}")
|
||||
|
||||
|
||||
@app.command("migrate-cockpit")
|
||||
def migrate_cockpit() -> None:
|
||||
"""Create the cockpit tables (+ TimescaleDB hypertable on Postgres) and seed the strategy registry."""
|
||||
@@ -575,54 +512,6 @@ def unlock_calendar_import(
|
||||
f"unlock_events @ {dsn.split('@')[-1]} (table now has {len(repo.read_events())})")
|
||||
|
||||
|
||||
@app.command("backfill-xsp-opra")
|
||||
def backfill_xsp_opra(
|
||||
start: str = typer.Option(..., "--start", help="ISO date to begin the backfill from (e.g. 2013-04-01)"),
|
||||
max_cost: float = typer.Option(
|
||||
25.0, "--max-cost",
|
||||
help="per-MONTH OPRA cost cap in $ (BATCHED: one range definition + one range ohlcv fetch per month, "
|
||||
"so the guard is per-month, not per-day)."),
|
||||
) -> None:
|
||||
"""One-time historical freeze of the XSP OPRA slice from --start through today into `xsp_option_bars`,
|
||||
BATCHED per month: one range `definition` + one range `ohlcv-1d` fetch per month, then every trading day
|
||||
processed LOCALLY (parity forward + band + upsert) — ~2 OPRA calls/month vs the old ~6-8/day (~1h vs ~45h
|
||||
over 2013->). Resumable/idempotent (`upsert_bars` merges on (osi_symbol, date)) — safe to re-run or resume
|
||||
from a later --start. A bad month is logged + skipped, never aborts. Manual, one-time; NOT the nightly
|
||||
asset graph. The per-fetch databento cost guard stays active (raise --max-cost, never disable it)."""
|
||||
import datetime as _dt
|
||||
|
||||
from fxhnt.adapters.orchestration.assets import _backfill_month
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
|
||||
settings = get_settings()
|
||||
d = settings.databento
|
||||
dbn = DatabentoDataProvider(max_cost_usd=max_cost, equity_dataset=d.equity_dataset,
|
||||
future_dataset=d.future_dataset, default_start=d.default_start)
|
||||
bars = XspOptionBarsRepo(settings.operational_dsn)
|
||||
bars.migrate()
|
||||
at = _dt.datetime.now(_dt.UTC).replace(tzinfo=None)
|
||||
|
||||
start_d = _dt.date.fromisoformat(start)
|
||||
today = _dt.datetime.now(_dt.UTC).date()
|
||||
cur = start_d.replace(day=1)
|
||||
tot_ok = tot_fail = 0
|
||||
while cur <= today:
|
||||
nxt = (cur + _dt.timedelta(days=32)).replace(day=1) # first of the next month
|
||||
m_start = max(cur, start_d)
|
||||
m_end = min(nxt - _dt.timedelta(days=1), today)
|
||||
try:
|
||||
ok, fail = _backfill_month(dbn, bars, month_start=m_start.isoformat(),
|
||||
month_end=m_end.isoformat(), at=at)
|
||||
tot_ok += ok
|
||||
tot_fail += fail
|
||||
typer.echo(f"backfill-xsp-opra: {cur:%Y-%m} done ({ok} ok, {fail} fail) — {tot_ok} ok total")
|
||||
except Exception as e: # a bad month (cost/network) — log + continue, never abort the whole backfill
|
||||
typer.echo(f"backfill-xsp-opra: {cur:%Y-%m} FAILED ({type(e).__name__}: {str(e)[:120]}) — skipped")
|
||||
cur = nxt
|
||||
typer.echo(f"-> backfill-xsp-opra complete: {tot_ok} days frozen, {tot_fail} failed, "
|
||||
f"{start}..{today.isoformat()}")
|
||||
|
||||
|
||||
@app.command("backfill-etf-volume")
|
||||
def backfill_etf_volume() -> None:
|
||||
"""One-time/resumable historical freeze of each `FUND_INSTRUMENTS` ETF's raw Yahoo daily share volume
|
||||
@@ -1214,58 +1103,6 @@ def bybit_ingest_klines(
|
||||
f"oldest date {oldest_date}")
|
||||
|
||||
|
||||
@app.command("ingest-dvol")
|
||||
def ingest_dvol_cmd(
|
||||
currencies: str = typer.Option(
|
||||
"BTC,ETH", "--currencies",
|
||||
help="comma-separated Deribit DVOL currencies to ingest (default BTC,ETH — the assets with a DVOL "
|
||||
"index). Each is written as feature 'dvol' under '<CCY>USDT' in bybit_features."),
|
||||
from_month: str = typer.Option(
|
||||
"", "--from", help="floor for the backward pagination, YYYY-MM (default: the 2021-03 backstop, just "
|
||||
"below DVOL inception 2021-03-24)."),
|
||||
) -> None:
|
||||
"""Ingest the Deribit DVOL annualised IMPLIED-vol index (the IV leg of the VRP edge) into the
|
||||
Bybit-namespaced TimescaleDB table `bybit_features` as feature 'dvol', keyed by the perp symbol
|
||||
(BTC -> BTCUSDT, ETH -> ETHUSDT). resolution=86400 -> one value/day. ONLY 'dvol' is written
|
||||
(close/funding are the other ingests' job); idempotent/resumable. The VRP eval reads this `dvol` panel
|
||||
alongside the kline `close` panel. No live paper_* tables are touched."""
|
||||
import datetime as _dt
|
||||
|
||||
from fxhnt.adapters.data.deribit_dvol import DeribitDVOL
|
||||
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
|
||||
from fxhnt.application.dvol_ingest import ingest_dvol
|
||||
|
||||
settings = get_settings()
|
||||
deribit = DeribitDVOL()
|
||||
|
||||
start_ms: int | None = None
|
||||
if from_month:
|
||||
try:
|
||||
y, m = (int(x) for x in from_month.split("-"))
|
||||
start_ms = int(_dt.datetime(y, m, 1, tzinfo=_dt.UTC).timestamp() * 1000)
|
||||
except (ValueError, TypeError) as e:
|
||||
typer.echo(f"error: --from must be YYYY-MM (got {from_month!r})", err=True)
|
||||
raise typer.Exit(1) from e
|
||||
|
||||
ccys = tuple(c.strip().upper() for c in currencies.split(",") if c.strip())
|
||||
if not ccys:
|
||||
typer.echo("error: empty --currencies", err=True)
|
||||
raise typer.Exit(1)
|
||||
typer.echo(f"ingest-dvol: {ccys}, floor={from_month or '2021-03 (default)'} "
|
||||
f"-> bybit_features feature='dvol' @ {settings.operational_dsn.split('@')[-1]}")
|
||||
|
||||
store = TimescaleFeatureStore(settings.operational_dsn, table="bybit_features")
|
||||
try:
|
||||
summary = ingest_dvol(store, deribit, currencies=ccys, start_ms=start_ms)
|
||||
finally:
|
||||
store.close()
|
||||
oldest = summary["oldest_day"]
|
||||
oldest_date = ((_dt.date(1970, 1, 1) + _dt.timedelta(days=oldest)).isoformat()
|
||||
if oldest is not None else "n/a")
|
||||
typer.echo(f"\ningest-dvol done: {summary['symbols']} currencies, "
|
||||
f"{summary['day_rows']} day-rows, oldest date {oldest_date}")
|
||||
|
||||
|
||||
@app.command("ingest-deribit-funding")
|
||||
def ingest_deribit_funding_cmd(
|
||||
instruments: str = typer.Option(
|
||||
@@ -2189,303 +2026,6 @@ def bybit_positioning_eval(
|
||||
typer.echo("\nNOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched. Memory-bounded: only the "
|
||||
"(liquid, with --liquid) coins are READ — the full panels are never loaded.")
|
||||
|
||||
|
||||
@app.command("vrp-eval")
|
||||
def vrp_eval(
|
||||
direction: str = typer.Option(
|
||||
"short_vol", "--direction",
|
||||
help="short_vol (DEFAULT: SELL variance — harvest the IV-over-RV premium; fat LEFT tail on vol "
|
||||
"spikes) or long_vol (BUY variance — the negation). Both are RE-RUN so the cost subtracts "
|
||||
"correctly in each."),
|
||||
cost_bps: float = typer.Option(
|
||||
5.5, "--cost-bps",
|
||||
help="Per-day cost (bps, scaled by the vol-target leverage). NOTE: selling OPTIONS is NOT 5.5bp — "
|
||||
"--verify shows a 50/100bp option-cost sensitivity."),
|
||||
target_ann_vol: float = typer.Option(
|
||||
0.15, "--target-vol", help="Vol-target the VRP book to this annual vol (default 0.15 = 15%/yr)."),
|
||||
verify: bool = typer.Option(
|
||||
False, "--verify",
|
||||
help="ADVERSARIAL verification: BOTH directions' net-of-cost metrics across a cost grid INCLUDING "
|
||||
"realistic OPTION costs (5.5/11/22/50/100 bps), the single WORST DAY (the vol-spike tail), "
|
||||
"maxDD, per-year Sharpe+maxDD, AND the correlation with EACH existing edge "
|
||||
"(tstrend/unlock/xsfunding/positioning)."),
|
||||
walk_forward: bool = typer.Option(
|
||||
False, "--walk-forward",
|
||||
help="OOS / WALK-FORWARD validation (the TAIL-AWARE DEPLOY GATE): a TRAIN/TEST split that picks the "
|
||||
"direction AND sizes k on TRAIN then scores OOS, rolling non-overlapping windows, a look-ahead "
|
||||
"audit, and a PASS/FAIL verdict that FAILS a catastrophic-tail book even when Sharpe>0."),
|
||||
window_days: int = typer.Option(
|
||||
180, "--window-days", help="Width (days) of each rolling non-overlapping OOS window in --walk-forward."),
|
||||
cost_grid: str = typer.Option(
|
||||
"5.5,11,22,50,100", "--cost-grid",
|
||||
help="comma-separated cost levels (bps) for --verify (default spans cheap perp-style + realistic "
|
||||
"option costs)."),
|
||||
) -> None:
|
||||
"""READ-ONLY, TAIL-HONEST: evaluate the VRP (volatility-risk-premium) edge — a vol premium, a genuinely
|
||||
DIFFERENT edge type from the directional/flow edges already in the book.
|
||||
|
||||
SIGNAL = Deribit DVOL (implied vol) vs the BTC/ETH close-to-close REALIZED vol. short-vol (default)
|
||||
harvests the IV-over-RV spread via a ROLLING VARIANCE SWAP: strike K=(DVOL/100)² (implied annual variance,
|
||||
causal) vs RV²=(365/swap_days)·Σ ret² over the swap's life; a daily ladder of overlapping 30-day swaps
|
||||
gives a SMOOTH daily series, BTC+ETH equal-weight, vol-targeted to --target-vol. POSITIVE on calm windows,
|
||||
big-NEGATIVE on vol-spike windows (the fat left tail). EXECUTION would be short Bybit option straddles
|
||||
(a cross-venue caveat vs the Deribit signal).
|
||||
|
||||
Prints CAGR/Sharpe/maxDD + the single WORST DAY (the vol spike) + the vol-target leverage. --verify adds
|
||||
the option-cost sensitivity, per-year tails, and the correlation with each existing edge. --walk-forward
|
||||
is the tail-aware deploy gate (a clean "no / too-tail-heavy" is a VALID outcome). Writes NOTHING."""
|
||||
if direction not in ("short_vol", "long_vol"):
|
||||
raise typer.BadParameter("must be 'short_vol' or 'long_vol'", param_hint="--direction")
|
||||
|
||||
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
|
||||
from fxhnt.application.vrp_eval import (
|
||||
verify_vrp_edge,
|
||||
vrp_metrics_from_store,
|
||||
walk_forward_vrp,
|
||||
)
|
||||
|
||||
settings = get_settings()
|
||||
store = TimescaleFeatureStore(settings.operational_dsn, table="bybit_features")
|
||||
|
||||
if walk_forward:
|
||||
try:
|
||||
rep = walk_forward_vrp(store, cost_bps=cost_bps, window_days=window_days,
|
||||
target_ann_vol=target_ann_vol)
|
||||
finally:
|
||||
store.close()
|
||||
_print_vrp_walk_forward(rep, cost_bps=cost_bps)
|
||||
return
|
||||
|
||||
if verify:
|
||||
grid = tuple(float(x) for x in cost_grid.split(",") if x.strip())
|
||||
try:
|
||||
rep = verify_vrp_edge(store, cost_bps=cost_bps, cost_grid=grid, target_ann_vol=target_ann_vol)
|
||||
finally:
|
||||
store.close()
|
||||
_print_vrp_verify(rep, cost_bps=cost_bps)
|
||||
return
|
||||
|
||||
try:
|
||||
m = vrp_metrics_from_store(store, cost_bps=cost_bps, direction=direction,
|
||||
target_ann_vol=target_ann_vol)
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
typer.echo("\n=== VRP (volatility risk premium) — TAIL-HONEST, READ-ONLY ===")
|
||||
typer.echo(f"direction: {direction.upper()} ({'SELL variance — harvest' if direction == 'short_vol' else 'BUY variance — negation'})")
|
||||
typer.echo("signal: Deribit DVOL (IV) vs BTC/ETH realized vol | book: BTC+ETH equal-weight, vol-targeted")
|
||||
typer.echo(f"construction: rolling 30d variance swap K=(DVOL/100)^2 vs RV^2=(365/n)*sum(ret^2) (daily "
|
||||
f"ladder) | target vol={target_ann_vol*100:.0f}%/yr")
|
||||
if m.get("available"):
|
||||
typer.echo(f" CAGR={m['cagr']*100:7.1f}% Sharpe={m['sharpe']:6.2f} maxDD={m['max_dd']*100:7.1f}% "
|
||||
f" total={m['total_return']*100:7.1f}% days={m['days']} span={m['first_day']}..{m['last_day']}")
|
||||
typer.echo(f" WORST DAY (vol-spike tail loss): {m['worst_day']*100:+.2f}% | avg leverage (k): {m['avg_leverage']:.2f}")
|
||||
else:
|
||||
typer.echo(f" N/A — {m.get('reason', '')}")
|
||||
typer.echo(" (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)")
|
||||
typer.echo("\nCROSS-VENUE CAVEAT: the SIGNAL is Deribit's DVOL; live EXECUTION would be short Bybit option "
|
||||
"straddles (delta-hedged).")
|
||||
typer.echo("NOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched.")
|
||||
|
||||
|
||||
@app.command("vrp-defined-risk-eval")
|
||||
def vrp_defined_risk_eval(
|
||||
direction: str = typer.Option(
|
||||
"short_vol", "--direction",
|
||||
help="short_vol (DEFAULT: short ATM straddle + long OTM wings — harvest the IV-over-RV premium with a "
|
||||
"BOUNDED tail) or long_vol (the negation)."),
|
||||
cost_bps: float = typer.Option(
|
||||
5.5, "--cost-bps",
|
||||
help="Per-day cost (bps, scaled by the vol-target leverage). Selling OPTIONS is NOT 5.5bp — use "
|
||||
"--cost-bps 50 (or --verify) for the realistic option-cost deploy gate."),
|
||||
target_ann_vol: float = typer.Option(
|
||||
0.15, "--target-vol", help="Vol-target the defined-risk book to this annual vol (default 0.15)."),
|
||||
wing_width: float = typer.Option(
|
||||
0.15, "--wing-width",
|
||||
help="OTM wing distance as a FRACTION of spot (default 0.15 = ±15%). The max loss is BOUNDED at "
|
||||
"wing_width − credit. Wider wing => cheaper protection => larger credit but larger max loss "
|
||||
"(naked limit as wing_width -> 1)."),
|
||||
wing_in_atm_std: float = typer.Option(
|
||||
0.0, "--wing-in-atm-std",
|
||||
help="If > 0, place the wings at this many ATM std-moves (wing = N·IV·sqrt(T)) instead of the fixed "
|
||||
"--wing-width. 0 (default) uses --wing-width."),
|
||||
put_skew_vol: float = typer.Option(
|
||||
5.0, "--put-skew-vol",
|
||||
help="Crypto put-skew bump (vol POINTS) added to the put-leg IV (default +5). 0 = flat-ATM = under-"
|
||||
"prices the put wing => mildly OPTIMISTIC (documented caveat)."),
|
||||
verify: bool = typer.Option(
|
||||
False, "--verify",
|
||||
help="ADVERSARIAL verification: both directions net-of-cost across the cost grid (incl 50/100bp option "
|
||||
"costs), worst-day, maxDD, per-year, and corr with each existing edge."),
|
||||
walk_forward: bool = typer.Option(
|
||||
False, "--walk-forward",
|
||||
help="OOS / WALK-FORWARD validation (the TAIL-AWARE DEPLOY GATE): TRAIN picks direction+k, scored OOS, "
|
||||
"rolling windows, look-ahead audit, PASS/FAIL verdict. The wings are meant to PASS the tail "
|
||||
"check the naked VRP (−74% maxDD) failed."),
|
||||
window_days: int = typer.Option(
|
||||
180, "--window-days", help="Width (days) of each rolling non-overlapping OOS window in --walk-forward."),
|
||||
cost_grid: str = typer.Option(
|
||||
"5.5,11,22,50,100", "--cost-grid",
|
||||
help="comma-separated cost levels (bps) for --verify (cheap perp-style + realistic option costs)."),
|
||||
) -> None:
|
||||
"""READ-ONLY, TAIL-HONEST: evaluate the DEFINED-RISK VRP — the deployable form of the vol-risk premium.
|
||||
|
||||
The naked short-variance VRP has a real gross edge but FAILS the deploy gate on an UNBOUNDED left tail
|
||||
(modelled maxDD ~−74% on vol spikes + realistic option costs). This caps it: SELL the ATM straddle but BUY
|
||||
OTM wings (an iron condor) so the max loss is BOUNDED at wing_width − credit. The legs are priced from DVOL
|
||||
via BLACK-SCHOLES (we have no historical option chains — a documented BS + put-skew approximation, so the
|
||||
modelled credit is an approximation, not a real chain). Re-runs the SAME tail-aware harness as the naked
|
||||
evaluator (vol-target, cost grid, per-year, worst-day, maxDD, corr-vs-4-edges, look-ahead, PASS/FAIL) so
|
||||
the two are directly comparable — the headline is the maxDD the wings buy back. Writes NOTHING."""
|
||||
if direction not in ("short_vol", "long_vol"):
|
||||
raise typer.BadParameter("must be 'short_vol' or 'long_vol'", param_hint="--direction")
|
||||
|
||||
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
|
||||
from fxhnt.application.vrp_defined_risk_eval import (
|
||||
defined_risk_metrics_from_store,
|
||||
verify_defined_risk_edge,
|
||||
walk_forward_defined_risk,
|
||||
)
|
||||
|
||||
settings = get_settings()
|
||||
store = TimescaleFeatureStore(settings.operational_dsn, table="bybit_features")
|
||||
wstd = wing_in_atm_std if wing_in_atm_std > 0.0 else None
|
||||
|
||||
if walk_forward:
|
||||
try:
|
||||
rep = walk_forward_defined_risk(store, cost_bps=cost_bps, window_days=window_days,
|
||||
target_ann_vol=target_ann_vol, wing_width=wing_width,
|
||||
put_skew_vol=put_skew_vol, wing_in_atm_std=wstd)
|
||||
finally:
|
||||
store.close()
|
||||
_print_vrp_walk_forward(rep, cost_bps=cost_bps)
|
||||
typer.echo(f"\nDEFINED-RISK: wing_width=±{wing_width*100:.0f}% put_skew_vol=+{put_skew_vol:.1f}vp "
|
||||
"(short ATM straddle + long OTM wings; max loss BOUNDED at wing_width − credit)")
|
||||
typer.echo("MODELLING CAVEAT: legs are BLACK-SCHOLES-priced from the ATM DVOL (no real option chain); "
|
||||
"put_skew_vol=0 under-prices the put wing => optimistic.")
|
||||
return
|
||||
|
||||
if verify:
|
||||
grid = tuple(float(x) for x in cost_grid.split(",") if x.strip())
|
||||
try:
|
||||
rep = verify_defined_risk_edge(store, cost_bps=cost_bps, cost_grid=grid,
|
||||
target_ann_vol=target_ann_vol, wing_width=wing_width,
|
||||
put_skew_vol=put_skew_vol, wing_in_atm_std=wstd)
|
||||
finally:
|
||||
store.close()
|
||||
_print_vrp_verify(rep, cost_bps=cost_bps)
|
||||
typer.echo(f"\nDEFINED-RISK: wing_width=±{wing_width*100:.0f}% put_skew_vol=+{put_skew_vol:.1f}vp "
|
||||
"(BS-priced from DVOL; max loss BOUNDED at wing_width − credit)")
|
||||
return
|
||||
|
||||
try:
|
||||
m = defined_risk_metrics_from_store(store, cost_bps=cost_bps, direction=direction,
|
||||
target_ann_vol=target_ann_vol, wing_width=wing_width,
|
||||
put_skew_vol=put_skew_vol, wing_in_atm_std=wstd)
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
typer.echo("\n=== DEFINED-RISK VRP (short straddle + protective wings) — TAIL-HONEST, READ-ONLY ===")
|
||||
typer.echo(f"direction: {direction.upper()} | wing_width: ±{wing_width*100:.0f}% | put_skew_vol: "
|
||||
f"+{put_skew_vol:.1f}vp | target vol={target_ann_vol*100:.0f}%/yr")
|
||||
typer.echo("construction: SELL ATM straddle, BUY OTM wings (BS-priced from DVOL); max loss BOUNDED at "
|
||||
"wing_width − credit")
|
||||
if m.get("available"):
|
||||
typer.echo(f" CAGR={m['cagr']*100:7.1f}% Sharpe={m['sharpe']:6.2f} maxDD={m['max_dd']*100:7.1f}% "
|
||||
f" total={m['total_return']*100:7.1f}% days={m['days']} span={m['first_day']}..{m['last_day']}")
|
||||
typer.echo(f" WORST DAY (capped vol-spike loss): {m['worst_day']*100:+.2f}% | avg leverage (k): {m['avg_leverage']:.2f}")
|
||||
else:
|
||||
typer.echo(f" N/A — {m.get('reason', '')}")
|
||||
typer.echo(" (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)")
|
||||
typer.echo("\nMODELLING CAVEAT: legs are BLACK-SCHOLES-priced from the ATM DVOL (no real option chain); a "
|
||||
"put-skew bump models richer crypto puts (put_skew_vol=0 under-prices the put wing => optimistic).")
|
||||
typer.echo("CROSS-VENUE CAVEAT: SIGNAL = Deribit DVOL; live EXECUTION = short Bybit option condors.")
|
||||
typer.echo("NOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched.")
|
||||
|
||||
|
||||
def _print_vrp_verify(rep: dict, *, cost_bps: float) -> None:
|
||||
"""Pretty-print the adversarial VRP verification report (both directions, tail-honest)."""
|
||||
typer.echo("\n=== VRP (volatility risk premium) — ADVERSARIAL VERIFICATION (TAIL-HONEST, READ-ONLY) ===")
|
||||
typer.echo("signal: Deribit DVOL (IV) vs BTC/ETH realized vol | book: BTC+ETH equal-weight, vol-targeted")
|
||||
if not rep.get("available"):
|
||||
typer.echo(f" N/A — {rep.get('reason', '')}")
|
||||
typer.echo(" (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)")
|
||||
return
|
||||
|
||||
typer.echo("\n-- NET-OF-COST, BOTH DIRECTIONS (incl REALISTIC OPTION costs 50/100bp) --")
|
||||
for d in ("short_vol", "long_vol"):
|
||||
typer.echo(f" [{d.upper()}] avg leverage (k)={rep['avg_leverage'].get(d, 0.0):.2f}")
|
||||
for bps in rep["cost_grid"]:
|
||||
m = rep["net_cost"][d][bps]
|
||||
typer.echo(f" cost={bps:6.1f}bps Sharpe={m['sharpe']:6.2f} CAGR={m['cagr']*100:7.1f}% "
|
||||
f"maxDD={m['max_dd']*100:7.1f}% total={m['total_return']*100:8.1f}% days={m['days']}")
|
||||
|
||||
typer.echo("\n-- THE TAIL (single worst day = the vol spike; maxDD @ headline cost) --")
|
||||
for d in ("short_vol", "long_vol"):
|
||||
typer.echo(f" [{d.upper()}] WORST DAY={rep['worst_day'][d]*100:+.2f}% maxDD={rep['max_dd'][d]*100:+.1f}%")
|
||||
|
||||
typer.echo("\n-- PER-YEAR Sharpe + maxDD (net @ headline cost; all-from-one-year => overfit) --")
|
||||
for d in ("short_vol", "long_vol"):
|
||||
years = rep["per_year"][d]
|
||||
ys = " ".join(f"{yr}:Sh={y['sharpe']:+5.2f}/DD={y['max_dd']*100:+4.0f}%(n={y['days']})"
|
||||
for yr, y in sorted(years.items()))
|
||||
typer.echo(f" [{d.upper()}] {ys}")
|
||||
|
||||
typer.echo(f"\n-- CORRELATION with EACH existing edge (net @ {cost_bps:.1f}bps; ~0 => additive) --")
|
||||
for d in ("short_vol", "long_vol"):
|
||||
corrs = rep["corr_edges"][d]
|
||||
parts = [f"{edge}={'N/A' if corrs.get(edge) is None else f'{corrs[edge]:+.3f}'}"
|
||||
for edge in ("tstrend", "unlock", "xsfunding", "positioning")]
|
||||
typer.echo(f" [{d.upper()}] " + " ".join(parts))
|
||||
|
||||
typer.echo("\nVERDICT GUIDE: a deployable VRP edge stays Sharpe>0 at REALISTIC OPTION cost (50/100bp), is "
|
||||
"positive in MORE THAN ONE year, has a SURVIVABLE tail (maxDD not catastrophic), AND is "
|
||||
"uncorrelated with all four existing edges. VRP looks great until a spike — judge it NET of "
|
||||
"the worst day.")
|
||||
typer.echo(f"\nCROSS-VENUE CAVEAT: {rep.get('cross_venue_note', '')}")
|
||||
typer.echo("NOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched.")
|
||||
|
||||
|
||||
def _print_vrp_walk_forward(rep: dict, *, cost_bps: float) -> None:
|
||||
"""Pretty-print the tail-aware OOS / walk-forward validation report (the deploy gate)."""
|
||||
typer.echo("\n=== VRP — OOS / WALK-FORWARD VALIDATION (TAIL-AWARE DEPLOY GATE, READ-ONLY) ===")
|
||||
typer.echo(f"signal: Deribit DVOL (IV) vs BTC/ETH realized vol | cost: {cost_bps:.1f}bps")
|
||||
if not rep.get("available"):
|
||||
typer.echo(f" N/A — {rep.get('reason', '')}")
|
||||
typer.echo(" (run `fxhnt ingest-dvol` + `fxhnt bybit-ingest-klines` first)")
|
||||
return
|
||||
|
||||
th = rep["train_holdout"]
|
||||
rw = rep["rolling_windows"]
|
||||
full = rep["full"]
|
||||
typer.echo(f" train-chosen direction: {rep['direction'].upper()} | vol-target={rep['target_ann_vol']*100:.0f}%/yr "
|
||||
f"| avg leverage (k)={rep['avg_leverage']:.2f} | maxDD floor={rep['max_dd_floor']*100:.0f}%")
|
||||
typer.echo(f" FULL-sample: Sharpe={full['sharpe']:6.2f} maxDD={full['max_dd']*100:+.1f}% "
|
||||
f"worst day={full['worst_day']*100:+.2f}% total={full['total_return']*100:+.1f}%")
|
||||
|
||||
typer.echo("\n-- TRAIN/TEST (OOS) split: direction+k chosen on TRAIN, scored on held-out TEST --")
|
||||
typer.echo(f" train Sharpe={th['train_sharpe']:+6.2f} -> TEST(OOS) Sharpe={th['test_sharpe']:+6.2f} "
|
||||
f"TEST maxDD={th.get('test_max_dd', 0.0)*100:+.1f}% (split@{th.get('split_at', 'n/a')})")
|
||||
|
||||
typer.echo(f"\n-- ROLLING non-overlapping {rep['window_days']}d windows ({rw['n_windows']}); "
|
||||
f"fraction positive={rw['fraction_positive']*100:.0f}% --")
|
||||
for w in rw["windows"]:
|
||||
typer.echo(f" {w['start']}..{w['end']} Sharpe={w['sharpe']:+6.2f} total={w['total_return']*100:+7.1f}% "
|
||||
f"maxDD={w['max_dd']*100:+5.1f}% (n={w['days']})")
|
||||
|
||||
typer.echo(f"\n-- LOOK-AHEAD audit: causal={rep['look_ahead']['causal']} "
|
||||
f"(leak_days={rep['look_ahead']['leak_days']}) --")
|
||||
|
||||
v = rep["verdict"]
|
||||
typer.echo("\n-- VERDICT (tail-aware) --")
|
||||
for name, ok in v["checks"].items():
|
||||
typer.echo(f" [{'PASS' if ok else 'FAIL'}] {name}")
|
||||
typer.echo(f"\n ==> {'PASS — DEPLOYABLE' if v['deployable'] else 'FAIL — NOT deployable'} "
|
||||
f"(a clean 'no / too-tail-heavy' is a valid outcome)")
|
||||
typer.echo(f"\nCROSS-VENUE CAVEAT: {rep.get('cross_venue_note', '')}")
|
||||
typer.echo("NOTE: writes NOTHING — paper_nav/paper_sleeve_ret untouched.")
|
||||
|
||||
|
||||
def _print_oi_deleverage_verify(rep: dict, *, scope: str, direction_note: str, cost_bps: float) -> None:
|
||||
"""Pretty-print the adversarial OI-deleveraging verification report (both directions)."""
|
||||
typer.echo("\n=== Bybit OI-deleveraging reversion — ADVERSARIAL VERIFICATION (READ-ONLY) ===")
|
||||
|
||||
@@ -114,6 +114,12 @@ class UcitsListing(BaseModel):
|
||||
isin: str # unambiguous security id (e.g. "IE00B5BMR087") — the real resolution key
|
||||
exchange: str = "LSEETF" # the LSE ETF venue: LSEETF + ISIN + USD resolves the line (SMART+ISIN = 0)
|
||||
currency: str = "USD"
|
||||
# The symbol IBKR REPORTS a HOLDING of this line under (`contract.symbol` from `positions()`), which can
|
||||
# diverge from the cosmetic `ticker`. Probe-verified live 2026-07-20 (`probe-ucits-symbols`): only IEF's
|
||||
# `CBU0` diverges → IBKR reports it as `CSBGU0`; the other 5 lines report under their ticker. The cockpit's
|
||||
# NLV stray-check (`_expected_book_symbols`) keys on this, so a legitimate book position is not mislabeled
|
||||
# a stray + dropped from NLV. Defaults to `ticker` when they match (the common case).
|
||||
ibkr_symbol: str = ""
|
||||
# Yahoo LSE daily-volume ticker (ADV input for the UCITS-regime capacity/cost model). Defaults to
|
||||
# f"{ticker}.L" when left empty; set explicitly if Yahoo's symbol diverges from the display ticker.
|
||||
yahoo_ticker: str = ""
|
||||
@@ -122,9 +128,11 @@ class UcitsListing(BaseModel):
|
||||
half_spread_bps: float = 10.0
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _default_yahoo_ticker(self) -> "UcitsListing":
|
||||
def _default_derived_fields(self) -> UcitsListing:
|
||||
if not self.yahoo_ticker:
|
||||
self.yahoo_ticker = f"{self.ticker}.L"
|
||||
if not self.ibkr_symbol: # the common case: IBKR reports the holding under the ticker
|
||||
self.ibkr_symbol = self.ticker
|
||||
return self
|
||||
|
||||
|
||||
@@ -149,6 +157,7 @@ class UcitsSettings(BaseSettings):
|
||||
map: dict[str, UcitsListing] = Field(default_factory=lambda: {
|
||||
"SPY": UcitsListing(ticker="CSPX", isin="IE00B5BMR087", half_spread_bps=1.5),
|
||||
"IEF": UcitsListing(ticker="CBU0", isin="IE00B3VWN518", yahoo_ticker="CBU0.L", # Acc line, ~$12M ADV
|
||||
ibkr_symbol="CSBGU0", # IBKR reports the HOLDING under CSBGU0 (probe 2026-07-20)
|
||||
half_spread_bps=2.0),
|
||||
"GLD": UcitsListing(ticker="IGLN", isin="IE00B4ND3602", # USD line = IGLN (SGLN is the GBP line)
|
||||
half_spread_bps=2.0),
|
||||
@@ -203,7 +212,8 @@ class Settings(BaseSettings):
|
||||
paper_enabled: bool = True # toggle the daily-rebalance paper-book persistence hook (T7)
|
||||
# Staleness/health axis: a recompute track whose source data (as_of) OR freeze table hasn't advanced in
|
||||
# more than this many BUSINESS days reads as STALE (a red cockpit badge + ERROR log). Default 3 allows
|
||||
# the normal ~1-business-day ingest lag; the anti-silent-rot threshold that would have caught vrp.
|
||||
# the normal ~1-business-day ingest lag; the anti-silent-rot threshold that would have caught a
|
||||
# strategy silently rotting for weeks.
|
||||
stale_after_business_days: int = 3
|
||||
|
||||
cost_bps_per_turnover: float = 10.0 # round-trip cost model (bps of traded notional)
|
||||
|
||||
@@ -68,8 +68,8 @@ class FleetRow:
|
||||
exec_twin_sid: str = ""
|
||||
exec_return_pct: float | None = None
|
||||
exec_divergence_pct: float | None = None
|
||||
# D3 (Task 4) — the REAL IBKR paper-account "real trades" signal: "EXEC" only for the two IBKR real-
|
||||
# account books (multistrat/vrp) once real fills have been captured (`exec_vs_sim(...).has_exec`); ""
|
||||
# D3 (Task 4) — the REAL IBKR paper-account "real trades" signal: "EXEC" only for the IBKR real-
|
||||
# account book (multistrat) once real fills have been captured (`exec_vs_sim(...).has_exec`); ""
|
||||
# (with `recon=None`) for every other strategy AND for an IBKR book with no captured data yet — the
|
||||
# cockpit's Overview sub-row only ever renders for a genuinely-executing row, never a fabricated one.
|
||||
exec_status: str = ""
|
||||
@@ -187,7 +187,7 @@ class StrategyDetail:
|
||||
exec_venue: str | None = None
|
||||
# D2.2/2.3 (Task 3) — the REAL IBKR paper-account section: what the book is holding right now, its
|
||||
# recent trades, and the account's raw NLV series (for the value-over-time overlay curve). Empty for
|
||||
# every strategy that isn't one of the IBKR real-account books (multistrat/vrp) or when no account data
|
||||
# every strategy that isn't the IBKR real-account book (multistrat) or when no account data
|
||||
# has been captured yet — the template renders "no trades recorded yet", never a 500.
|
||||
holdings: list[dict[str, Any]] = field(default_factory=list)
|
||||
# Held symbols outside the book's allowed universe (US + UCITS) — a legacy/stray position (e.g. an IBIT
|
||||
|
||||
@@ -18,12 +18,6 @@ class CryptoPerpBarClient(Protocol):
|
||||
...
|
||||
|
||||
|
||||
class VolIndexClient(Protocol):
|
||||
def dvol(self, currency: str) -> dict[int, float]:
|
||||
"""{ epoch_day: implied-vol-index } for BTC/ETH (Deribit DVOL)."""
|
||||
...
|
||||
|
||||
|
||||
class FundamentalsClient(Protocol):
|
||||
def metrics(self, symbol: str) -> dict[str, float]:
|
||||
"""Latest daily valuation metrics: {'peRatio','pbRatio','marketCap', ...} (most recent available)."""
|
||||
|
||||
@@ -154,36 +154,6 @@ STRATEGY_REGISTRY: dict[str, dict[str, Any]] = {
|
||||
# track) — FDUSD/USDP aren't listed on Bybit, doesn't port to the venue. `StableReversionRunner` itself
|
||||
# (stablecoin_runner.py) stays — it is still live via `bybit_stablecoin_eval.py` / `bybit_edges_eval.py`
|
||||
# / `bybit_book_eval.py` (Bybit-native stablecoin evaluation, a separate consumer of the runner core).
|
||||
# Defined-risk equity VRP (XSP put-credit-spreads) on REAL OPRA option data — the cross-asset diversifier
|
||||
# (co-crashes systemic risk-off, so NOT a systemic hedge; diversifies idiosyncratic crypto crashes).
|
||||
# record_mode recompute off the frozen xsp_option_bars PIT table. promote_on_pass False: real-capital-
|
||||
# adjacent, so promotion to deploy (CVaR-sized capital) is a deliberate human step after it reconciles.
|
||||
"vrp": {
|
||||
# ARCHIVED/FALSIFIED 2026-07-15 — shelved dead-end; code + 13yr OPRA data kept; see project_fxhnt_diversifier_hunt_2026_07_15
|
||||
"archived": True,
|
||||
"display_name": "Options income (put spreads)", "sleeve": "equity-vol",
|
||||
"venue": "ibkr-xsp",
|
||||
"tier": "research", "execution": "paper", "promote_on_pass": False,
|
||||
"gate_spec": {"gate_type": "reconciliation", "min_forward_days": 21},
|
||||
"definition": {"params": {"instrument": "XSP", "structure": "put-credit-spread",
|
||||
"dte_target": 30, "short_delta": 0.20, "width_points": 5.0,
|
||||
"ladder": 5, "source": "opra-pit"},
|
||||
"version": 1, "record_mode": "recompute", "backtest": {"kind": "recompute-replay"}},
|
||||
},
|
||||
# EXECUTED VRP twin (sub-project C-style): the real-fill reconciliation for `vrp` on IBKR XSP `mleg`
|
||||
# combos — SUSPENDED (the IBKR account has no options permission yet), coded ahead of that grant so the
|
||||
# cockpit twin + gate wiring is ready. record_mode observed (booked from `vrp_exec_record`, not
|
||||
# recomputed).
|
||||
"vrp_exec": {
|
||||
# ARCHIVED/FALSIFIED 2026-07-15 — shelved dead-end; code + 13yr OPRA data kept; see project_fxhnt_diversifier_hunt_2026_07_15
|
||||
"archived": True,
|
||||
"display_name": "Equity VRP (EXECUTED, IBKR XSP)", "sleeve": "equity-vol",
|
||||
"venue": "ibkr-xsp",
|
||||
"tier": "research", "execution": "paper",
|
||||
"gate_spec": {"gate_type": "reconciliation", "min_forward_days": 21},
|
||||
"definition": {"params": {"source": "ibkr-xsp-fills", "models": "vrp"},
|
||||
"version": 1, "record_mode": "observed", "backtest": {"kind": "model-ref"}},
|
||||
},
|
||||
# Forward PAPER TRACK for the 4-edge Bybit book — the OUT-OF-SAMPLE record on the venue we'll actually
|
||||
# trade. Records the NAIVE equal-weight 4-sleeve book (tstrend/unlock/xsfunding/positioning) on
|
||||
# `bybit_features` (the live risk overlay over-levers on Bybit; that is a separate re-tune, deliberately
|
||||
|
||||
5
src/fxhnt/vendor/surfer/__init__.py
vendored
5
src/fxhnt/vendor/surfer/__init__.py
vendored
@@ -8,9 +8,8 @@ ONLY edits applied (no algorithm / math / constant changes):
|
||||
* data root made configurable via `$FXHNT_SURFER_DATA_DIR` (default `/data/surfer`):
|
||||
`pit_sweep.load()` reads `$FXHNT_SURFER_DATA_DIR/crypto_pit/*.npz`.
|
||||
|
||||
(The `surfer_poc` module — a value-returning `paper_step` refactor of `paper()` for the fxhnt
|
||||
ForwardTracker's tail-managed VRP sleeve — was RETIRED with its dormant `MomentumVrpStrategy` caller;
|
||||
Phase 0b Task 7c.)
|
||||
(The `surfer_poc` module — a value-returning `paper_step` refactor of `paper()` for a since-retired
|
||||
ForwardTracker sleeve — was RETIRED along with its dormant caller; Phase 0b Task 7c.)
|
||||
|
||||
torch is imported at module level (used in `signal_sweep.sharpe_t`/`validate`) and is kept — this is a
|
||||
faithful port, not a numpy reimplementation.
|
||||
|
||||
@@ -1,26 +0,0 @@
|
||||
import datetime as dt
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from fxhnt.adapters.persistence.cockpit_models import ForwardSummaryRow
|
||||
from fxhnt.adapters.persistence.forward_nav import ForwardNavRepo
|
||||
from fxhnt.adapters.web.app import create_app
|
||||
|
||||
|
||||
def test_archived_vrp_detail_is_hidden(tmp_path):
|
||||
"""vrp is ARCHIVED/FALSIFIED (shelved 2026-07-15): even with a persisted forward_summary row in the DB,
|
||||
a direct `/strategy/vrp` hit must 404 (detail() returns None) and the page must never leak the VRP label
|
||||
or its "Options income (put spreads)" alias. The code + OPRA data are kept; only the cockpit hides it."""
|
||||
repo = ForwardNavRepo(f"sqlite:///{tmp_path/'c.db'}")
|
||||
repo.migrate()
|
||||
at = dt.datetime(2026, 7, 12)
|
||||
with Session(repo._engine) as s:
|
||||
s.add(ForwardSummaryRow(strategy_id="vrp", as_of="2026-07-12", days=8, nav=1.004,
|
||||
total_return=0.004, sharpe=0.9, maxdd=-0.03, gate_status="WAIT",
|
||||
gate_reason="building 8/21", src_updated_at=at))
|
||||
s.commit()
|
||||
r = TestClient(create_app(repo)).get("/strategy/vrp")
|
||||
assert r.status_code == 404
|
||||
assert "Equity VRP" not in r.text
|
||||
assert "Options income" not in r.text
|
||||
@@ -9,6 +9,7 @@ from fxhnt.application.dashboard_service import DashboardService
|
||||
from fxhnt.application.forward_models import BacktestSummary
|
||||
from fxhnt.application.forward_models import ForwardNavRow as Row
|
||||
from fxhnt.application.forward_models import ForwardSummary
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
|
||||
|
||||
def _seeded() -> ForwardNavRepo:
|
||||
@@ -92,38 +93,31 @@ def test_fleet_no_backtest_renders_blank() -> None:
|
||||
assert unlock.bt_as_of == ""
|
||||
|
||||
|
||||
def test_archived_sleeves_hidden_everywhere() -> None:
|
||||
"""The `archived: True` entries (vrp/vrp_exec — FALSIFIED/shelved 2026-07-15) are KEPT in the Python
|
||||
registry (code + OPRA data preserved, entry not deleted) AND still seeded into the DB registry table by
|
||||
`_seed_registry` — but `ForwardNavRepo.registry()` (the SINGLE authoritative filter) excludes them, so
|
||||
every registry-driven render path drops them: the fleet Overview, per-strategy gate specs, and
|
||||
`detail()`. Proven here at the repo, fleet, and detail levels."""
|
||||
from fxhnt.adapters.persistence.cockpit_models import StrategyRegistryRow
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
from sqlalchemy import select as _select
|
||||
from sqlalchemy.orm import Session as _Session
|
||||
|
||||
# entries KEPT in the Python registry, flagged archived (not deleted)
|
||||
assert STRATEGY_REGISTRY["vrp"].get("archived") is True
|
||||
assert STRATEGY_REGISTRY["vrp_exec"].get("archived") is True
|
||||
|
||||
repo = _seeded()
|
||||
# the DB row IS still seeded (code + data kept) — a raw table read finds it...
|
||||
with _Session(repo._engine) as s:
|
||||
raw_ids = {r.strategy_id for r in s.scalars(_select(StrategyRegistryRow))}
|
||||
assert {"vrp", "vrp_exec"} <= raw_ids, "archived rows must still be seeded in the DB (not pruned)"
|
||||
# ...but the cockpit-facing registry() EXCLUDES them (the authoritative hide).
|
||||
reg_ids = {r.strategy_id for r in repo.registry()}
|
||||
assert "vrp" not in reg_ids and "vrp_exec" not in reg_ids
|
||||
assert "unlock" in reg_ids # a non-archived track is still visible (the filter is targeted)
|
||||
|
||||
# fleet Overview drops them; a non-archived track still renders
|
||||
fleet_ids = {f.strategy_id for f in DashboardService(repo).fleet()}
|
||||
assert "vrp" not in fleet_ids and "vrp_exec" not in fleet_ids
|
||||
assert "unlock" in fleet_ids
|
||||
|
||||
# a direct /strategy/vrp hit returns None (route 404s), but a live track still resolves
|
||||
def test_archived_sleeves_hidden_everywhere(monkeypatch) -> None:
|
||||
# A SYNTHETIC archived registry entry (`STRATEGY_REGISTRY` `archived: True` — the pattern
|
||||
# `test_paper_books_web.py` uses for its synthetic `archivedstrat`/`newstrat`, in place of the deleted
|
||||
# vrp-based test) must be excluded from every cockpit surface `ForwardNavRepo.registry()` feeds:
|
||||
# `DashboardService.fleet()` and `.detail()`. Monkeypatch BEFORE `migrate()` so `_seed_registry()` (which
|
||||
# seeds the DB registry table FROM `STRATEGY_REGISTRY` and prunes anything not present there) picks up
|
||||
# the synthetic entry.
|
||||
monkeypatch.setitem(STRATEGY_REGISTRY, "archivedstrat",
|
||||
{"archived": True, "display_name": "Options income (put spreads)",
|
||||
"sleeve": "equity-vol", "gate_spec": {}})
|
||||
repo = _seeded() # seeds registry AFTER the monkeypatch above, so archivedstrat lands in the DB row set
|
||||
svc = DashboardService(repo)
|
||||
assert svc.detail("vrp") is None
|
||||
assert svc.detail("vrp_exec") is None
|
||||
|
||||
# registry() itself: the archived row never comes back, a non-archived row (unlock) does.
|
||||
reg_ids = {r.strategy_id for r in repo.registry()}
|
||||
assert "archivedstrat" not in reg_ids
|
||||
assert "unlock" in reg_ids
|
||||
|
||||
# fleet(): archived sleeve excluded; a live sleeve still shows.
|
||||
fleet = svc.fleet()
|
||||
assert all(f.strategy_id != "archivedstrat" for f in fleet)
|
||||
assert any(f.strategy_id == "unlock" for f in fleet)
|
||||
|
||||
# detail(): direct hit on the archived id returns None (404 at the route layer); a live id still resolves.
|
||||
assert svc.detail("archivedstrat") is None
|
||||
assert svc.detail("unlock") is not None
|
||||
|
||||
|
||||
|
||||
@@ -1,152 +0,0 @@
|
||||
"""DeribitDVOL adapter — the Deribit DVOL annualised implied-vol index (the IV leg of the VRP edge).
|
||||
|
||||
ALL HTTP is injected/mocked — no network in these tests. The adapter takes a `fetch(url)->dict` callable
|
||||
(the JSON endpoint), so the tests drive it with canned paginated responses and assert: the {epoch_day:
|
||||
dvol_close} mapping (close = row index 4), backward pagination via end_timestamp to a floor, the floor stop,
|
||||
the empty-page stop, dedupe across page boundaries, and rate-limit/transient backoff.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
|
||||
from fxhnt.adapters.data.deribit_dvol import DERIBIT_DVOL_FLOOR_MS, DeribitDVOL
|
||||
|
||||
_DAY_MS = 86_400_000
|
||||
|
||||
# DETERMINISM SEAM: pin the adapter's wall clock to a fixed recent instant so the backward walk anchors at a
|
||||
# constant `now` regardless of the calendar date the suite runs on (mirrors test_deribit_funding). If the
|
||||
# adapter ever reverts to a bare time.time() in the loop, the clock-determinism guard below fails.
|
||||
_NOW_MS = 1780315200_000 # noon 2026-06-01 UTC, in ms
|
||||
|
||||
|
||||
def _FIXED_CLOCK() -> float: # noqa: N802 — constant-like injectable clock
|
||||
return _NOW_MS / 1000.0
|
||||
|
||||
|
||||
def _dvol_row(ts_ms: int, close: float) -> list:
|
||||
# Deribit volatility-index rows: [ts_ms, open, high, low, close]. close (idx 4) is the DVOL value.
|
||||
return [ts_ms, close - 1.0, close + 1.0, close - 2.0, close]
|
||||
|
||||
|
||||
class _PagedDVOL:
|
||||
"""Mock of /public/get_volatility_index_data: serves rows oldest->newest within the requested
|
||||
[start_timestamp, end_timestamp] window, honoring the end_timestamp cursor like Deribit (returns up to
|
||||
`limit` rows ending at end_timestamp, oldest-first). Walking end_timestamp back past the oldest row
|
||||
returns an empty list (stops the loop)."""
|
||||
|
||||
def __init__(self, rows: list[list], *, limit: int = 1000) -> None:
|
||||
self._rows = sorted(rows, key=lambda r: r[0])
|
||||
self._limit = limit
|
||||
self.calls: list[tuple[int | None, int | None]] = []
|
||||
|
||||
def fetch(self, url: str) -> dict:
|
||||
start = _parse_qs_int(url, "start_timestamp")
|
||||
end = _parse_qs_int(url, "end_timestamp")
|
||||
self.calls.append((start, end))
|
||||
cand = [r for r in self._rows
|
||||
if (start is None or r[0] >= start) and (end is None or r[0] <= end)]
|
||||
# newest `limit` rows in the window (Deribit caps 1000/request, returns the most recent of the window)
|
||||
page = cand[-self._limit:]
|
||||
return {"result": {"data": page}}
|
||||
|
||||
|
||||
def _parse_qs_int(url: str, key: str) -> int | None:
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
q = parse_qs(urlparse(url).query)
|
||||
return int(q[key][0]) if key in q else None
|
||||
|
||||
|
||||
def test_dvol_history_maps_epoch_day_to_close_and_paginates_backward():
|
||||
rows = [
|
||||
_dvol_row(100 * _DAY_MS, 60.0),
|
||||
_dvol_row(99 * _DAY_MS, 55.0),
|
||||
_dvol_row(98 * _DAY_MS, 48.0),
|
||||
_dvol_row(97 * _DAY_MS, 52.0),
|
||||
]
|
||||
paged = _PagedDVOL(rows, limit=2)
|
||||
dv = DeribitDVOL(fetch=paged.fetch, sleep=lambda _: None)
|
||||
daily = dv.dvol_history("BTC", start_ms=0)
|
||||
assert daily == {100: 60.0, 99: 55.0, 98: 48.0, 97: 52.0}
|
||||
assert len(paged.calls) >= 2 # it paginated and walked end_timestamp backward
|
||||
|
||||
|
||||
def test_dvol_history_url_carries_currency_and_resolution():
|
||||
seen: list[str] = []
|
||||
|
||||
def fetch(url: str) -> dict:
|
||||
seen.append(url)
|
||||
return {"result": {"data": []}}
|
||||
|
||||
dv = DeribitDVOL(fetch=fetch, sleep=lambda _: None)
|
||||
dv.dvol_history("ETH")
|
||||
assert "get_volatility_index_data" in seen[0]
|
||||
assert "currency=ETH" in seen[0]
|
||||
assert "resolution=86400" in seen[0]
|
||||
|
||||
|
||||
def test_dvol_history_empty_page_stops_loop():
|
||||
paged = _PagedDVOL([], limit=1000)
|
||||
dv = DeribitDVOL(fetch=paged.fetch, sleep=lambda _: None)
|
||||
assert dv.dvol_history("BTC") == {}
|
||||
assert len(paged.calls) == 1
|
||||
|
||||
|
||||
def test_dvol_history_respects_start_ms_floor():
|
||||
rows = [_dvol_row(10 * _DAY_MS, 50.0), _dvol_row(20 * _DAY_MS, 60.0)]
|
||||
paged = _PagedDVOL(rows, limit=1000)
|
||||
dv = DeribitDVOL(fetch=paged.fetch, sleep=lambda _: None)
|
||||
daily = dv.dvol_history("BTC", start_ms=15 * _DAY_MS)
|
||||
assert 20 in daily and 10 not in daily # only the row at/after the floor survives
|
||||
|
||||
|
||||
def test_default_floor_is_2021_03():
|
||||
# The floor sits just below the true BTC/ETH DVOL inception (2021-03-24); the old 2023-09 floor silently
|
||||
# discarded ~890 days of valid history.
|
||||
import datetime as dt
|
||||
assert int(dt.datetime(2021, 3, 1, tzinfo=dt.UTC).timestamp() * 1000) == DERIBIT_DVOL_FLOOR_MS
|
||||
|
||||
|
||||
def test_dvol_history_dedups_across_page_boundaries():
|
||||
rows = [_dvol_row(d * _DAY_MS, 40.0 + d) for d in range(50, 55)]
|
||||
paged = _PagedDVOL(rows, limit=2)
|
||||
dv = DeribitDVOL(fetch=paged.fetch, sleep=lambda _: None)
|
||||
daily = dv.dvol_history("BTC", start_ms=0)
|
||||
assert set(daily) == {50, 51, 52, 53, 54}
|
||||
|
||||
|
||||
def test_dvol_history_is_clock_deterministic():
|
||||
# The first page's end_timestamp anchors at the INJECTED clock, not the wall clock: the URL of the first
|
||||
# request is identical across runs when a fixed clock is supplied — proving no bare time.time() leaks into
|
||||
# the loop (the determinism bug the clock seam prevents).
|
||||
seen_a: list[str] = []
|
||||
seen_b: list[str] = []
|
||||
DeribitDVOL(fetch=lambda u: (seen_a.append(u), {"result": {"data": []}})[1],
|
||||
sleep=lambda _: None, clock=_FIXED_CLOCK).dvol_history("BTC")
|
||||
DeribitDVOL(fetch=lambda u: (seen_b.append(u), {"result": {"data": []}})[1],
|
||||
sleep=lambda _: None, clock=_FIXED_CLOCK).dvol_history("BTC")
|
||||
assert seen_a == seen_b
|
||||
assert f"end_timestamp={_NOW_MS}" in seen_a[0] # anchored at the fixed clock, not today
|
||||
assert int(_FIXED_CLOCK()) == 1780315200 # frozen anchor, never tracks today
|
||||
|
||||
|
||||
def test_default_floor_matches_2021_03():
|
||||
assert int(_dt.datetime(2021, 3, 1, tzinfo=_dt.UTC).timestamp() * 1000) == DERIBIT_DVOL_FLOOR_MS
|
||||
|
||||
|
||||
def test_dvol_history_retries_on_transient_error_then_succeeds():
|
||||
d = 30 * _DAY_MS
|
||||
real = _PagedDVOL([_dvol_row(d, 70.0)], limit=1000)
|
||||
state = {"n": 0}
|
||||
|
||||
def flaky(url: str) -> dict:
|
||||
state["n"] += 1
|
||||
if state["n"] <= 2:
|
||||
raise RuntimeError("transient 502")
|
||||
return real.fetch(url)
|
||||
|
||||
waited: list[float] = []
|
||||
dv = DeribitDVOL(fetch=flaky, sleep=waited.append, backoff_base=0.5, backoff_tries=5)
|
||||
daily = dv.dvol_history("BTC", start_ms=0)
|
||||
assert abs(daily[30] - 70.0) < 1e-12
|
||||
assert state["n"] >= 3
|
||||
assert waited[:2] == [0.5, 1.0] # exponential backoff (no real sleep)
|
||||
@@ -1,79 +0,0 @@
|
||||
"""ingest_dvol — write the Deribit DVOL index into bybit_features as feature 'dvol' for BTCUSDT/ETHUSDT.
|
||||
|
||||
The DVOL index is currency-keyed (BTC/ETH); the warehouse panels are symbol-keyed (BTCUSDT/ETHUSDT), so the
|
||||
ingest maps currency -> symbol and writes feature='dvol' at ts = epoch_day*86400. ONLY 'dvol' is written
|
||||
(close is the kline ingest's job). Idempotent/resumable. No network — the adapter is a stub.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
|
||||
from fxhnt.application.dvol_ingest import ingest_dvol
|
||||
|
||||
_DAY = 86_400
|
||||
|
||||
|
||||
class _StubDeribit:
|
||||
"""Stub DeribitDVOL: serves a canned {epoch_day: dvol} per currency, records the calls."""
|
||||
|
||||
def __init__(self, by_currency: dict[str, dict[int, float]]) -> None:
|
||||
self._by_currency = by_currency
|
||||
self.calls: list[str] = []
|
||||
|
||||
def dvol_history(self, currency: str, *, start_ms: int | None = None) -> dict[int, float]:
|
||||
self.calls.append(currency)
|
||||
return dict(self._by_currency.get(currency, {}))
|
||||
|
||||
|
||||
def test_ingest_writes_dvol_for_btc_and_eth() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
deribit = _StubDeribit({"BTC": {100: 55.0, 101: 60.0}, "ETH": {100: 70.0, 101: 72.0}})
|
||||
summary = ingest_dvol(store, deribit)
|
||||
# BTCUSDT/ETHUSDT now carry the 'dvol' feature at the right timestamps.
|
||||
btc = store.read_panel(["BTCUSDT"], "dvol")["BTCUSDT"]
|
||||
eth = store.read_panel(["ETHUSDT"], "dvol")["ETHUSDT"]
|
||||
store.close()
|
||||
assert btc == {100: 55.0, 101: 60.0}
|
||||
assert eth == {100: 70.0, 101: 72.0}
|
||||
assert summary["symbols"] == 2
|
||||
assert summary["day_rows"] == 4
|
||||
assert summary["oldest_day"] == 100
|
||||
assert set(deribit.calls) == {"BTC", "ETH"}
|
||||
|
||||
|
||||
def test_ingest_maps_currency_to_usdt_symbol() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
deribit = _StubDeribit({"BTC": {1: 50.0}})
|
||||
ingest_dvol(store, deribit, currencies=("BTC",))
|
||||
# The row lands under BTCUSDT at ts = epoch_day*86400, not under "BTC".
|
||||
rows = store.read_features("BTCUSDT", features=["dvol"])
|
||||
store.close()
|
||||
assert rows == [(1 * _DAY, {"dvol": 50.0})]
|
||||
|
||||
|
||||
def test_ingest_skips_currency_with_no_data() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
deribit = _StubDeribit({"BTC": {1: 50.0}}) # ETH returns nothing
|
||||
summary = ingest_dvol(store, deribit)
|
||||
store.close()
|
||||
assert summary["symbols"] == 1 # only BTC produced rows; ETH skipped (never aborts)
|
||||
|
||||
|
||||
def test_ingest_is_idempotent() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
deribit = _StubDeribit({"BTC": {1: 50.0}, "ETH": {1: 60.0}})
|
||||
ingest_dvol(store, deribit)
|
||||
ingest_dvol(store, deribit) # re-run overwrites in place, no duplicate rows
|
||||
btc = store.read_features("BTCUSDT", features=["dvol"])
|
||||
store.close()
|
||||
assert btc == [(1 * _DAY, {"dvol": 50.0})]
|
||||
|
||||
|
||||
def test_ingest_only_writes_dvol_feature() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
deribit = _StubDeribit({"BTC": {1: 50.0}})
|
||||
ingest_dvol(store, deribit, currencies=("BTC",))
|
||||
rows = store.read_features("BTCUSDT")
|
||||
store.close()
|
||||
# exactly one feature written, and it's 'dvol' (close/funding are the other ingests' job).
|
||||
feats = {name for _, feats in rows for name in feats}
|
||||
assert feats == {"dvol"}
|
||||
@@ -5,8 +5,11 @@ from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from fxhnt.adapters.persistence.cockpit_models import CockpitBase, XspOptionBarRow
|
||||
from fxhnt.adapters.persistence.forward_nav import ForwardNavRepo
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
from fxhnt.application.forward_health import (
|
||||
BROKEN,
|
||||
HEALTHY,
|
||||
@@ -97,61 +100,88 @@ def _seed_summary(repo: ForwardNavRepo, sid: str, as_of: str, at: dt.datetime) -
|
||||
gate_status="WAIT", gate_reason="building", at=at)
|
||||
|
||||
|
||||
def test_evaluate_health_persists_and_catches_stale_freeze(tmp_path):
|
||||
def _seed_freeze_bars(dsn: str, rows: list[dict], at: dt.datetime) -> None:
|
||||
"""Insert rows directly into the kept `xsp_option_bars` table (no VRP repo class involved) — mirrors
|
||||
what the retired XspOptionBarsRepo.migrate()/upsert_bars() used to do, for the freeze-staleness axis."""
|
||||
kw = {} if dsn.startswith("postgresql") else {"connect_args": {"check_same_thread": False}}
|
||||
engine = create_engine(dsn, **kw)
|
||||
CockpitBase.metadata.create_all(engine)
|
||||
with Session(engine) as s:
|
||||
for r in rows:
|
||||
s.merge(XspOptionBarRow(
|
||||
osi_symbol=r["osi_symbol"], date=dt.date.fromisoformat(r["date"]),
|
||||
expiry=dt.date.fromisoformat(r["expiry"]), strike=float(r["strike"]),
|
||||
right=r["right"], close=float(r["close"]), src_updated_at=at))
|
||||
s.commit()
|
||||
|
||||
|
||||
def _freeze_track_entry() -> dict:
|
||||
"""A synthetic opra-pit-sourced registry entry — VRP was the only real track ever wired to the freeze
|
||||
axis (`_FREEZE_SOURCES = {"opra-pit"}`), and it is now fully deleted, but the axis itself is strategy-
|
||||
agnostic and must stay covered generically (per-track monkeypatch, mirrors test_forward_definition.py's
|
||||
`t_ov` pattern)."""
|
||||
return {"display_name": "Freeze-axis test track", "sleeve": "test",
|
||||
"gate_spec": {"gate_type": "reconciliation"},
|
||||
"definition": {"params": {"source": "opra-pit"}, "record_mode": "recompute"}}
|
||||
|
||||
|
||||
def test_evaluate_health_persists_and_catches_stale_freeze(tmp_path, monkeypatch):
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
|
||||
monkeypatch.setitem(STRATEGY_REGISTRY, "t_freeze", _freeze_track_entry())
|
||||
dsn = f"sqlite:///{tmp_path}/health.db"
|
||||
at = dt.datetime(2026, 7, 14, 23, 30)
|
||||
repo = ForwardNavRepo(dsn)
|
||||
repo.migrate()
|
||||
# vrp: fresh forward summary but a STALE frozen table (the incident) → must read STALE.
|
||||
_seed_summary(repo, "vrp", "2026-07-13", at)
|
||||
# t_freeze: fresh forward summary but a STALE frozen table (the incident) → must read STALE.
|
||||
_seed_summary(repo, "t_freeze", "2026-07-13", at)
|
||||
# sixtyforty: recompute, fresh, absolute gate → HEALTHY.
|
||||
_seed_summary(repo, "sixtyforty", "2026-07-13", at)
|
||||
# unlock: fresh forward data (isolates the NO_REF path below from the separate no-anchor/no-data STALE
|
||||
# check — this test seeds no forward_anchor rows at all, so without real forward data unlock would hit
|
||||
# that check first).
|
||||
_seed_summary(repo, "unlock", "2026-07-13", at)
|
||||
# seed the vrp freeze table with a max date weeks behind the run date.
|
||||
xsp = XspOptionBarsRepo(dsn)
|
||||
xsp.migrate()
|
||||
xsp.upsert_bars([{"osi_symbol": "XSP 260620P00470000", "date": "2026-06-15", "expiry": "2026-07-17",
|
||||
"strike": 470.0, "right": "P", "close": 1.0}], at)
|
||||
# seed the freeze table with a max date weeks behind the run date.
|
||||
_seed_freeze_bars(dsn, [{"osi_symbol": "XSP 260620P00470000", "date": "2026-06-15",
|
||||
"expiry": "2026-07-17", "strike": 470.0, "right": "P", "close": 1.0}], at)
|
||||
|
||||
verdicts = {v.strategy_id: v for v in evaluate_health(dsn, at=at)}
|
||||
assert verdicts["vrp"].health == STALE and verdicts["vrp"].stale_days > 3
|
||||
assert "freeze table" in verdicts["vrp"].reason
|
||||
assert verdicts["t_freeze"].health == STALE and verdicts["t_freeze"].stale_days > 3
|
||||
assert "freeze table" in verdicts["t_freeze"].reason
|
||||
assert verdicts["sixtyforty"].health == HEALTHY
|
||||
# unlock is a reconciliation recompute track with no backtest ref seeded → NO_REF (M6 invariant).
|
||||
assert verdicts["unlock"].health == NO_REF
|
||||
# persisted to strategy_health, readable by the dashboard.
|
||||
persisted = {h.strategy_id: h for h in repo.all_health()}
|
||||
assert persisted["vrp"].health == STALE and persisted["vrp"].stale_days > 3
|
||||
assert persisted["t_freeze"].health == STALE and persisted["t_freeze"].stale_days > 3
|
||||
|
||||
|
||||
def test_evaluate_health_replaces_snapshot_on_recovery(tmp_path):
|
||||
def test_evaluate_health_replaces_snapshot_on_recovery(tmp_path, monkeypatch):
|
||||
"""A track that recovers must CLEAR its old STALE row (replace-all semantics)."""
|
||||
from fxhnt.application.forward_models import BacktestSummary
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
|
||||
monkeypatch.setitem(STRATEGY_REGISTRY, "t_freeze", _freeze_track_entry())
|
||||
dsn = f"sqlite:///{tmp_path}/rec.db"
|
||||
repo = ForwardNavRepo(dsn)
|
||||
repo.migrate()
|
||||
xsp = XspOptionBarsRepo(dsn)
|
||||
xsp.migrate()
|
||||
# run 1: freeze stale → vrp STALE
|
||||
# run 1: freeze stale → t_freeze STALE
|
||||
at1 = dt.datetime(2026, 7, 14, 23, 30)
|
||||
# a backtest ref so a recovered vrp resolves to HEALTHY (not NO_REF) — isolates the STALE→clear behavior.
|
||||
repo.upsert_backtest_summary(BacktestSummary(strategy_id="vrp", as_of="2026-06-01", cagr=0.2, ann_vol=0.1,
|
||||
sharpe=1.0, max_drawdown=-0.05, passed=True, dsr=0.9, is_sharpe=1.0,
|
||||
oos_sharpe=0.9, pvalue=0.01), at1)
|
||||
_seed_summary(repo, "vrp", "2026-07-13", at1)
|
||||
xsp.upsert_bars([{"osi_symbol": "X", "date": "2026-06-15", "expiry": "2026-07-17", "strike": 470.0,
|
||||
"right": "P", "close": 1.0}], at1)
|
||||
assert {v.strategy_id: v for v in evaluate_health(dsn, at=at1)}["vrp"].health == STALE
|
||||
# run 2: freeze advances to yesterday → vrp HEALTHY, the old STALE row cleared
|
||||
xsp.upsert_bars([{"osi_symbol": "Y", "date": "2026-07-13", "expiry": "2026-07-17", "strike": 470.0,
|
||||
"right": "P", "close": 1.0}], at1)
|
||||
_seed_summary(repo, "vrp", "2026-07-13", at1)
|
||||
assert {v.strategy_id: v for v in evaluate_health(dsn, at=at1)}["vrp"].health == HEALTHY
|
||||
assert {h.strategy_id: h for h in repo.all_health()}["vrp"].health == HEALTHY
|
||||
# a backtest ref so a recovered t_freeze resolves to HEALTHY (not NO_REF) — isolates the STALE→clear
|
||||
# behavior.
|
||||
repo.upsert_backtest_summary(BacktestSummary(strategy_id="t_freeze", as_of="2026-06-01", cagr=0.2,
|
||||
ann_vol=0.1, sharpe=1.0, max_drawdown=-0.05, passed=True, dsr=0.9,
|
||||
is_sharpe=1.0, oos_sharpe=0.9, pvalue=0.01), at1)
|
||||
_seed_summary(repo, "t_freeze", "2026-07-13", at1)
|
||||
_seed_freeze_bars(dsn, [{"osi_symbol": "X", "date": "2026-06-15", "expiry": "2026-07-17",
|
||||
"strike": 470.0, "right": "P", "close": 1.0}], at1)
|
||||
assert {v.strategy_id: v for v in evaluate_health(dsn, at=at1)}["t_freeze"].health == STALE
|
||||
# run 2: freeze advances to yesterday → t_freeze HEALTHY, the old STALE row cleared
|
||||
_seed_freeze_bars(dsn, [{"osi_symbol": "Y", "date": "2026-07-13", "expiry": "2026-07-17",
|
||||
"strike": 470.0, "right": "P", "close": 1.0}], at1)
|
||||
_seed_summary(repo, "t_freeze", "2026-07-13", at1)
|
||||
assert {v.strategy_id: v for v in evaluate_health(dsn, at=at1)}["t_freeze"].health == HEALTHY
|
||||
assert {h.strategy_id: h for h in repo.all_health()}["t_freeze"].health == HEALTHY
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ LOUD cockpit badge (rendering)
|
||||
|
||||
@@ -192,3 +192,24 @@ def test_clean_account_holdings_show_no_stray_warning() -> None:
|
||||
# SPY + IEF are both book instruments -> no "outside the book" warning renders.
|
||||
html = _client(_seed_forward(), _seed_ibkr()).get("/strategy/multistrat").text
|
||||
assert "outside the book" not in html
|
||||
|
||||
|
||||
def test_ibkr_reported_ucits_symbol_csbgu0_is_not_flagged_stray() -> None:
|
||||
# The IEF UCITS line's config ticker is CBU0, but IBKR REPORTS the holding under its canonical symbol
|
||||
# CSBGU0 (probe-verified 2026-07-20). A legitimate CSBGU0 book position must NOT be flagged "outside the
|
||||
# book" / dropped from NLV — the expected set keys on `ibkr_symbol`, not just the display ticker. A genuine
|
||||
# stray (IBIT) in the SAME account is still flagged.
|
||||
fwd = _seed_forward()
|
||||
ibkr = IbkrAccountRepo("sqlite://")
|
||||
ibkr.migrate()
|
||||
at = dt.datetime(2026, 7, 20, 15, 0)
|
||||
ibkr.replace_snapshot("2026-07-20", nlv=1_000_000.0, cash=900_000.0, gross=100_000.0,
|
||||
positions={"CSPX": 13.0, "CSBGU0": 55.0, "IBIT": 699.0}, at=at)
|
||||
ibkr.record_fills(strategy_id="multistrat", rebalance_id="r1",
|
||||
fills=[FillDTO(symbol="CSBGU0", side="BOT", qty=55.0, price=153.8, fee=1.0,
|
||||
status="Filled", shortfall_bps=0.0)], at=at)
|
||||
html = _client(fwd, ibkr).get("/strategy/multistrat").text
|
||||
banner = re.search(r"Position outside the book:[^<]*", html)
|
||||
assert banner is not None # IBIT is still a genuine stray
|
||||
assert "IBIT" in banner.group(0)
|
||||
assert "CSBGU0" not in banner.group(0) # the IEF book line is NOT flagged stray
|
||||
|
||||
@@ -4,20 +4,24 @@ from the PRECOMPUTED `sim_curve_ret` table (never recomputed on the request path
|
||||
measured-cost path's shape (curve + metrics + Live forward), with plain-language pill labels (never the raw
|
||||
registry id "multistrat").
|
||||
|
||||
NOTE: vrp was ARCHIVED/FALSIFIED (shelved 2026-07-15) — its `STRATEGY_REGISTRY` entry carries
|
||||
`archived: True`, so it is EXCLUDED from the Backtest book pills (`_SIM_BOOKS`) and a `?book=vrp` request
|
||||
falls back to the default book. The recompute-replay machinery itself still supports vrp (code + OPRA data
|
||||
kept); this test file now asserts vrp is HIDDEN from the sim UI (see the archived-book tests below).
|
||||
NOTE: vrp has been FULLY REMOVED from the codebase (shelved 2026-07-15, code+data deleted in the
|
||||
`chore/remove-vrp-code` cleanup) — there is no longer a `STRATEGY_REGISTRY` entry, `_SIM_BOOKS` pill, or
|
||||
recompute-replay support for it at all. This test file uses a synthetic non-existent book id
|
||||
(`no-such-book`) as an honest stand-in to probe the SAME fallback behavior an archived/unknown book must
|
||||
still get: never selectable, never rendered, always a graceful fallback to the default book (see the
|
||||
archived-book tests below).
|
||||
|
||||
Asserts:
|
||||
* /paper/sim shows the multistrat pill labelled in plain language, not the raw id, and NO vrp pill;
|
||||
* /paper/sim shows the multistrat pill labelled in plain language, not the raw id, and NO stray pill for an
|
||||
unknown book;
|
||||
* /paper/sim/run?book=multistrat renders a real curve + metrics from the precomputed sim_curve_ret series;
|
||||
* capital scales the curve linearly (same contract as the Bybit measured path);
|
||||
* the request path never recomputes the replay (a tripwire on ForwardStrategy.advance would fire if it did —
|
||||
proven implicitly: the seeded repo carries no build_strategy at all, so a recompute is structurally
|
||||
impossible; the curve still renders from sim_curve_ret alone);
|
||||
* the Live forward section renders for multistrat (its own forward track), not just bybit_4edge;
|
||||
* an archived book (vrp) is not selectable — the request falls back to the default book, never rendering vrp;
|
||||
* an unknown/archived book id is not selectable — the request falls back to the default book, never
|
||||
rendering that id;
|
||||
* absent precompute -> the graceful 'precomputing' note (never a 500, never the Bybit-specific caption).
|
||||
NO network — in-memory SQLite repos, seeded directly.
|
||||
"""
|
||||
@@ -47,7 +51,7 @@ def _seed_ibkr_curve(repo: PaperRepo, book: str, *, start: dt.date = dt.date(202
|
||||
repo.upsert_sim_curve_ret(book, d, ret, at=at)
|
||||
|
||||
|
||||
def _seeded_paper_repo(*, books: tuple[str, ...] = ("multistrat", "vrp")) -> PaperRepo:
|
||||
def _seeded_paper_repo(*, books: tuple[str, ...] = ("multistrat", "no-such-book")) -> PaperRepo:
|
||||
repo = PaperRepo("sqlite://")
|
||||
repo.migrate()
|
||||
for b in books:
|
||||
@@ -89,16 +93,14 @@ def _sim_data(html: str) -> dict:
|
||||
return json.loads(m.group(1))
|
||||
|
||||
|
||||
def test_sim_page_shows_multistrat_pill_and_hides_archived_vrp() -> None:
|
||||
def test_sim_page_shows_multistrat_pill_and_hides_unknown_book() -> None:
|
||||
c = _client(_seeded_paper_repo())
|
||||
html = c.get("/paper/sim").text
|
||||
assert "ETF Portfolio" in html
|
||||
# vrp is ARCHIVED — neither its pill label nor its raw id may appear anywhere on the Backtest page.
|
||||
assert "Options income (put spreads)" not in html
|
||||
assert "Equity VRP" not in html
|
||||
assert 'data-book="vrp"' not in html and "book=vrp" not in html
|
||||
# a book id absent from `_SIM_BOOKS` (unknown or archived) must never get a pill on the Backtest page.
|
||||
assert 'data-book="no-such-book"' not in html and "book=no-such-book" not in html
|
||||
# the raw registry id must never leak as pill TEXT (it is fine as a query-string value).
|
||||
assert ">multistrat<" not in html and ">vrp<" not in html
|
||||
assert ">multistrat<" not in html and ">no-such-book<" not in html
|
||||
|
||||
|
||||
def test_multistrat_book_is_selectable_and_lazy_loads() -> None:
|
||||
@@ -121,14 +123,14 @@ def test_multistrat_run_renders_curve_and_metrics_from_precomputed_sim_curve_ret
|
||||
assert 80000 < data["equity"][0] < 120000 # curve starts near the configured capital
|
||||
|
||||
|
||||
def test_archived_vrp_run_falls_back_and_never_renders_vrp() -> None:
|
||||
"""An archived book (vrp) is not in `_SIM_BOOKS`, so `/paper/sim/run?book=vrp` falls back to the default
|
||||
book (bybit_4edge) — it renders that book's view (or the graceful pending note), never vrp's curve/label."""
|
||||
def test_unknown_book_run_falls_back_and_never_renders_it() -> None:
|
||||
"""A book id absent from `_SIM_BOOKS` (unknown or archived) makes `/paper/sim/run?book=no-such-book`
|
||||
fall back to the default book (bybit_4edge) — it renders that book's view (or the graceful pending
|
||||
note), never the unknown id's curve/label."""
|
||||
c = _client(_seeded_paper_repo())
|
||||
r = c.get("/paper/sim/run", params={"book": "vrp", "capital": 100000})
|
||||
r = c.get("/paper/sim/run", params={"book": "no-such-book", "capital": 100000})
|
||||
assert r.status_code == 200 # graceful fallback, never a 500
|
||||
assert "Options income (put spreads)" not in r.text and "Equity VRP" not in r.text
|
||||
assert 'data-book="vrp"' not in r.text # the effective book is the default, not vrp
|
||||
assert 'data-book="no-such-book"' not in r.text # the effective book is the default, not the unknown id
|
||||
|
||||
|
||||
def test_ibkr_capital_scales_the_curve_linearly() -> None:
|
||||
@@ -180,12 +182,12 @@ def test_ibkr_missing_precompute_shows_pending_note_not_bybit_caption() -> None:
|
||||
assert 'data-cost-mode="flat"' not in r.text
|
||||
|
||||
|
||||
def test_sim_book_switch_covers_visible_books_and_excludes_archived_vrp() -> None:
|
||||
def test_sim_book_switch_covers_visible_books_and_excludes_unknown_book() -> None:
|
||||
c = _client(_seeded_paper_repo())
|
||||
# the three VISIBLE books each render their own page...
|
||||
for book in ("bybit_4edge", "bybit_4edge_levered", "multistrat"):
|
||||
r = c.get("/paper/sim", params={"book": book})
|
||||
assert r.status_code == 200 and f'data-book="{book}"' in r.text
|
||||
# ...but the ARCHIVED vrp is not selectable: it falls back to the default book (200, never vrp).
|
||||
r = c.get("/paper/sim", params={"book": "vrp"})
|
||||
assert r.status_code == 200 and 'data-book="vrp"' not in r.text
|
||||
# ...but an unknown/archived book id is not selectable: it falls back to the default book (200, never it).
|
||||
r = c.get("/paper/sim", params={"book": "no-such-book"})
|
||||
assert r.status_code == 200 and 'data-book="no-such-book"' not in r.text
|
||||
|
||||
@@ -7,9 +7,10 @@ one row per strategy trading on the shared real account — a dimmed "not starte
|
||||
registered-but-not-yet-executing strategy (a synthetic `newstrat` stands in for that here), and the table
|
||||
SCALES BY ROW (proven by a synthetic second executing strategy, not a redesign).
|
||||
|
||||
NOTE: vrp was ARCHIVED/FALSIFIED (shelved 2026-07-15) — it is EXCLUDED from the account breakdown
|
||||
(`_active_ibkr_account_sids`), so it never renders a row / its "Options income (put spreads)" label even
|
||||
though its fills stay captured in the repo. These tests now prove vrp is ABSENT wherever it used to show.
|
||||
NOTE: an ARCHIVED registry entry is EXCLUDED from the account breakdown (`_active_ibkr_account_sids`), so it
|
||||
never renders a row even though its fills stay captured in the repo (VRP was the original real-world case —
|
||||
ARCHIVED/FALSIFIED, shelved 2026-07-15 — but the code itself has since been fully removed; the exclusion
|
||||
mechanism is proven here with a synthetic archived entry instead).
|
||||
|
||||
Plain-language HARD RULE (Task 3/4 precedent): no raw registry id / `bps` / "vs sim" / "divergence" may ever
|
||||
appear in the rendered `/paper` landing — asserted against the VISIBLE text.
|
||||
@@ -31,6 +32,7 @@ from fxhnt.application.dashboard_service import DashboardService
|
||||
from fxhnt.application.exec_capture import FillDTO
|
||||
from fxhnt.application.forward_models import ForwardSummary
|
||||
from fxhnt.ports.live_price import FakeLivePrice
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
|
||||
|
||||
def _visible_text(html: str) -> str:
|
||||
@@ -89,16 +91,20 @@ def _seed_ibkr(*, extra_fills_for: tuple[str, ...] = ()) -> IbkrAccountRepo:
|
||||
|
||||
|
||||
def test_ibkr_account_view_one_executing_row_and_dimmed_not_started_row(monkeypatch) -> None:
|
||||
# vrp is ARCHIVED -> excluded from the account breakdown. A synthetic non-archived `newstrat` (registered
|
||||
# against the account but with NO captured fills) stands in as the dimmed "not started yet" row, so the
|
||||
# one-executing / one-dimmed shape is still covered — and vrp is proven ABSENT.
|
||||
# A synthetic `archivedstrat` (registry entry with archived=True) is EXCLUDED from the account
|
||||
# breakdown. A synthetic non-archived `newstrat` (registered against the account but with NO captured
|
||||
# fills) stands in as the dimmed "not started yet" row, so the one-executing / one-dimmed shape is still
|
||||
# covered — and the archived book is proven ABSENT.
|
||||
monkeypatch.setitem(STRATEGY_REGISTRY, "archivedstrat",
|
||||
{"archived": True, "display_name": "Options income (put spreads)",
|
||||
"sleeve": "equity-vol", "gate_spec": {}})
|
||||
monkeypatch.setattr(dashboard_service, "_IBKR_ACCOUNT_SIDS",
|
||||
frozenset({"multistrat", "vrp", "newstrat"}))
|
||||
frozenset({"multistrat", "archivedstrat", "newstrat"}))
|
||||
svc = DashboardService(_seed_forward(), _seed_ibkr()) # only multistrat has fills; newstrat does not
|
||||
view = svc.ibkr_account_view()
|
||||
assert view is not None
|
||||
ids = {r.strategy_id for r in view.per_strategy}
|
||||
assert ids == {"multistrat", "newstrat"} # archived vrp is excluded
|
||||
assert ids == {"multistrat", "newstrat"} # archived entry is excluded
|
||||
multistrat = next(r for r in view.per_strategy if r.strategy_id == "multistrat")
|
||||
newstrat = next(r for r in view.per_strategy if r.strategy_id == "newstrat")
|
||||
assert multistrat.executing is True
|
||||
@@ -117,16 +123,19 @@ def test_ibkr_account_view_one_executing_row_and_dimmed_not_started_row(monkeypa
|
||||
def test_ibkr_account_view_scale_test_adds_new_executing_row_no_redesign(monkeypatch) -> None:
|
||||
# Seed a SYNTHETIC third strategy (not in STRATEGY_REGISTRY) trading on the same account, and register
|
||||
# it against the account via the module constant — proves the breakdown scales by ROW, not a redesign.
|
||||
monkeypatch.setitem(STRATEGY_REGISTRY, "archivedstrat",
|
||||
{"archived": True, "display_name": "Options income (put spreads)",
|
||||
"sleeve": "equity-vol", "gate_spec": {}})
|
||||
monkeypatch.setattr(dashboard_service, "_IBKR_ACCOUNT_SIDS",
|
||||
frozenset({"multistrat", "vrp", "newstrat"}))
|
||||
frozenset({"multistrat", "archivedstrat", "newstrat"}))
|
||||
fwd = _seed_forward(extra_exec_sids=("newstrat",))
|
||||
ibkr = _seed_ibkr(extra_fills_for=("newstrat",))
|
||||
svc = DashboardService(fwd, ibkr)
|
||||
view = svc.ibkr_account_view()
|
||||
assert view is not None
|
||||
# archived vrp is excluded; the visible set is the two live account books (multistrat + synthetic newstrat)
|
||||
# archived entry is excluded; the visible set is the two live account books (multistrat + synthetic newstrat)
|
||||
assert len(view.per_strategy) == 2
|
||||
assert all(r.strategy_id != "vrp" for r in view.per_strategy)
|
||||
assert all(r.strategy_id != "archivedstrat" for r in view.per_strategy)
|
||||
executing_ids = {r.strategy_id for r in view.per_strategy if r.executing}
|
||||
assert executing_ids == {"multistrat", "newstrat"}
|
||||
newstrat = next(r for r in view.per_strategy if r.strategy_id == "newstrat")
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""The nightly `_persist_track_backtest_ref` step (fxhnt-backtest-refs Job) persists a recompute-replay
|
||||
strategy's (multistrat/vrp — the IBKR paper books) FULL inception return series into the `sim_curve_ret`
|
||||
table, so /paper/sim reads a precomputed IBKR curve instead of ever re-running `ForwardStrategy.advance` on a
|
||||
web request (Task 1/D1). Asserts:
|
||||
strategy's (multistrat/multistrat_levered — the IBKR paper books) FULL inception return series into the
|
||||
`sim_curve_ret` table, so /paper/sim reads a precomputed IBKR curve instead of ever re-running
|
||||
`ForwardStrategy.advance` on a web request (Task 1/D1). Asserts:
|
||||
* the series persisted is the SAME rows the reconciliation-gate `backtest_summary` ref is derived from
|
||||
(one replay call, two writes);
|
||||
* `PaperRepo.sim_curve_returns(strategy_id)` reads it back as {epoch_day: ret};
|
||||
@@ -54,15 +54,15 @@ def test_persist_track_backtest_ref_also_writes_the_reconciliation_ref(tmp_path,
|
||||
dsn = f"sqlite:///{tmp_path / 'sim_curve2.db'}"
|
||||
monkeypatch.setattr("fxhnt.config.get_settings", lambda: Settings(operational_dsn=dsn))
|
||||
|
||||
assets._persist_track_backtest_ref(build_op_context(), "vrp_nav", lambda: _Replay())
|
||||
assets._persist_track_backtest_ref(build_op_context(), "multistrat_levered_nav", lambda: _Replay())
|
||||
|
||||
fwd = ForwardNavRepo(dsn)
|
||||
fwd.migrate()
|
||||
assert any(b.strategy_id == "vrp" for b in fwd.all_backtest_summaries())
|
||||
assert any(b.strategy_id == "multistrat_levered" for b in fwd.all_backtest_summaries())
|
||||
|
||||
repo = PaperRepo(dsn)
|
||||
repo.migrate()
|
||||
assert repo.sim_curve_returns("vrp") # non-empty
|
||||
assert repo.sim_curve_returns("multistrat_levered") # non-empty
|
||||
|
||||
|
||||
def test_persist_track_backtest_ref_idempotent_on_rerun(tmp_path, monkeypatch):
|
||||
|
||||
@@ -1,452 +0,0 @@
|
||||
"""READ-ONLY, TAIL-HONEST evaluator for the DEFINED-RISK VRP — the deployable form of the vol-risk premium.
|
||||
|
||||
The naked short-variance VRP has a real gross edge but FAILS the deploy gate on an UNBOUNDED left tail
|
||||
(modelled maxDD ~−74%). The fix: sell the ATM straddle but BUY OTM wings so the max loss is BOUNDED (an iron
|
||||
condor). The legs are priced from DVOL via Black-Scholes (we have no option chains — documented caveat).
|
||||
|
||||
Tests assert:
|
||||
* the BOUNDED MAX LOSS — a huge move loses ≤ the defined max (wing_width − credit), NOT unbounded;
|
||||
* a calm expiry keeps (most of) the credit; the wing cost reduces the credit vs a naked straddle;
|
||||
* the capped curve has a DRAMATICALLY shallower maxDD than the naked variance swap on the SAME spike fixture
|
||||
(the wings work — this is the whole point);
|
||||
* verify is cost-monotone (incl 50/100bp option costs), per-year, worst-day, maxDD, corr-vs-4-edges;
|
||||
* walk-forward OOS + look-ahead-clean; tail-survivable PASSES;
|
||||
* READ-ONLY (a write-tripwire never trips);
|
||||
* the CLI smoke prints metrics + verify + walk-forward (in-memory store, NO network).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
|
||||
from fxhnt.application import vrp_eval
|
||||
from fxhnt.application.vrp_defined_risk_eval import (
|
||||
_N_LEGS,
|
||||
_defined_risk_daily_cost,
|
||||
_defined_risk_sized_series,
|
||||
defined_risk_credit,
|
||||
defined_risk_max_loss,
|
||||
defined_risk_metrics_from_store,
|
||||
defined_risk_payoff,
|
||||
defined_risk_returns_from_store,
|
||||
look_ahead_audit_defined_risk_ok,
|
||||
verify_defined_risk_edge,
|
||||
walk_forward_defined_risk,
|
||||
)
|
||||
|
||||
_DAY = 86_400
|
||||
_START = 19_358 # 2023-01-01 epoch day
|
||||
|
||||
|
||||
class _WriteTripwireStore:
|
||||
"""Read-only proxy: forwards reads, but ANY write/persist call trips an assertion."""
|
||||
|
||||
_FORBIDDEN = frozenset({
|
||||
"write_features", "write_features_bulk", "upsert_feature_rows", "upsert_membership",
|
||||
"replace_positions", "upsert_nav", "upsert_sleeve_ret", "replace_shadow_positions",
|
||||
"replace_trades", "_create_schema", "_bulk_upsert",
|
||||
})
|
||||
|
||||
def __init__(self, inner: TimescaleFeatureStore) -> None:
|
||||
object.__setattr__(self, "_inner", inner)
|
||||
|
||||
def __getattr__(self, name: str):
|
||||
if name in _WriteTripwireStore._FORBIDDEN:
|
||||
raise AssertionError(f"READ-ONLY violation: evaluator called write method {name!r}")
|
||||
return getattr(self._inner, name)
|
||||
|
||||
|
||||
# --- 1. the defined-risk spread: credit + BOUNDED payoff ---------------------------------------
|
||||
|
||||
def test_credit_is_positive_and_a_fraction_of_spot() -> None:
|
||||
# selling the ATM straddle collects more than the OTM wings cost -> a positive credit, sane in magnitude.
|
||||
c = defined_risk_credit(s=30_000.0, iv=0.60, t=30.0 / 365.0, wing_width=0.15)
|
||||
assert 0.0 < c < 0.20 # a fraction of spot, smaller than the wing distance
|
||||
|
||||
|
||||
def test_wing_cost_reduces_credit_vs_naked_straddle() -> None:
|
||||
# the naked straddle credit (no wings) = w -> infinity limit; buying wings (finite w) costs premium, so the
|
||||
# defined-risk credit is STRICTLY LESS than the naked straddle premium. A WIDER wing is cheaper -> larger
|
||||
# credit (approaching the naked premium).
|
||||
s, iv, t = 30_000.0, 0.60, 30.0 / 365.0
|
||||
narrow = defined_risk_credit(s=s, iv=iv, t=t, wing_width=0.10)
|
||||
wide = defined_risk_credit(s=s, iv=iv, t=t, wing_width=0.30)
|
||||
naked = defined_risk_credit(s=s, iv=iv, t=t, wing_width=0.95) # ~no wing cost -> ~naked straddle premium
|
||||
assert narrow < wide < naked
|
||||
|
||||
|
||||
def test_max_loss_is_bounded_the_core_property() -> None:
|
||||
# THE CORE PROPERTY: a catastrophic move loses AT MOST the defined max (wing_width − credit), NOT unbounded.
|
||||
credit = defined_risk_credit(s=30_000.0, iv=0.60, t=30.0 / 365.0, wing_width=0.15)
|
||||
defined_max = defined_risk_max_loss(credit=credit, wing_width=0.15)
|
||||
# a 10%, 50%, 90%, 99% crash all lose the SAME bounded amount once past the wing.
|
||||
for move in (-0.50, -0.90, -0.99, +0.95):
|
||||
pnl = defined_risk_payoff(credit=credit, realized_move=move, wing_width=0.15)
|
||||
assert pnl >= -defined_max - 1e-12 # never worse than the defined max loss
|
||||
assert math.isclose(pnl, -defined_max, abs_tol=1e-9) # past the wing -> exactly the cap
|
||||
|
||||
|
||||
def test_naked_straddle_loss_is_unbounded_but_capped_is_not() -> None:
|
||||
# contrast: with NO wings the loss grows with the move (unbounded); with wings it plateaus at the cap.
|
||||
credit = 0.05
|
||||
capped_small = defined_risk_payoff(credit=credit, realized_move=-0.20, wing_width=0.15)
|
||||
capped_huge = defined_risk_payoff(credit=credit, realized_move=-0.80, wing_width=0.15)
|
||||
assert math.isclose(capped_small, capped_huge, abs_tol=1e-12) # capped: same loss past the wing
|
||||
# the implicit naked short straddle (credit − |m|) keeps getting worse.
|
||||
naked_small = credit - 0.20
|
||||
naked_huge = credit - 0.80
|
||||
assert naked_huge < naked_small # naked: unbounded
|
||||
|
||||
|
||||
def test_calm_expiry_keeps_the_credit() -> None:
|
||||
# a small realized move (well inside the wings) -> pnl ≈ credit − |m|, still positive when |m| < credit.
|
||||
credit = 0.05
|
||||
pnl = defined_risk_payoff(credit=credit, realized_move=0.01, wing_width=0.15)
|
||||
assert math.isclose(pnl, credit - 0.01, abs_tol=1e-12)
|
||||
assert pnl > 0.0
|
||||
|
||||
|
||||
def test_long_vol_direction_negates() -> None:
|
||||
credit = 0.04
|
||||
short = defined_risk_payoff(credit=credit, realized_move=-0.30, wing_width=0.12, direction="short_vol")
|
||||
long_ = defined_risk_payoff(credit=credit, realized_move=-0.30, wing_width=0.12, direction="long_vol")
|
||||
assert math.isclose(long_, -short, rel_tol=1e-12)
|
||||
|
||||
|
||||
# --- 2. fixtures (mirror the naked evaluator's, with a tradeable defined-risk premium) ----------
|
||||
|
||||
def _seed_calm_premium(store: TimescaleFeatureStore, *, days: int = 400, dvol: float = 60.0,
|
||||
daily_move: float = 0.004) -> None:
|
||||
"""Calm: IV high (dvol=60), tiny realized oscillation -> the defined-risk condor harvests its credit."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
sign = 1.0 if i % 2 == 0 else -1.0
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol})])
|
||||
px[s] *= (1.0 + sign * daily_move)
|
||||
|
||||
|
||||
def _seed_with_spike(store: TimescaleFeatureStore, *, days: int = 400, spike_at: int = 200,
|
||||
crash: float = 0.45) -> None:
|
||||
"""Calm condor premium most days, but ONE catastrophic crash that would blow up a NAKED short straddle;
|
||||
the wings cap the defined-risk loss at the spread width. Used for the naked-vs-defined tail comparison."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": 50.0})])
|
||||
if i == spike_at:
|
||||
px[s] *= (1.0 - crash)
|
||||
else:
|
||||
px[s] *= (1.0 + (0.003 if i % 2 == 0 else -0.003))
|
||||
|
||||
|
||||
def _seed_tail_survivable(store: TimescaleFeatureStore, *, days: int = 700) -> None:
|
||||
"""A defined-risk premium that survives its tails: calm harvest, a moderate spike every ~90 days. DVOL
|
||||
varies day to day (so the causality audit can distinguish prior-day from same-day IV)."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
dvol = 60.0 + 8.0 * math.sin(i * 0.11)
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol, "turnover": 5e8})])
|
||||
if i % 90 == 45:
|
||||
px[s] *= (1.0 - 0.06)
|
||||
else:
|
||||
px[s] *= (1.0 + (0.004 if i % 2 == 0 else -0.004))
|
||||
|
||||
|
||||
def _seed_varying_dvol(store: TimescaleFeatureStore, *, days: int = 120) -> None:
|
||||
"""Premium fixture with monotone time-varying DVOL so the causal and leaked credit-pricing mappings are
|
||||
distinguishable on every day (the look-ahead audit can only bite when entry-day and expiry IV differ)."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
dvol = 50.0 + 0.137 * i
|
||||
sign = 1.0 if i % 2 == 0 else -1.0
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol})])
|
||||
px[s] *= (1.0 + sign * 0.004)
|
||||
|
||||
|
||||
# --- 3. the book series: harvest, sizing, read-only --------------------------------------------
|
||||
|
||||
def test_defined_risk_returns_positive_on_calm_premium() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
series = defined_risk_returns_from_store(store, cost_bps=0.0, direction="short_vol")
|
||||
store.close()
|
||||
assert len(series) > 2
|
||||
assert sum(series.values()) > 0.0 # the condor credit is harvested gross
|
||||
|
||||
|
||||
def test_defined_risk_metrics_is_read_only() -> None:
|
||||
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(inner)
|
||||
store = _WriteTripwireStore(inner)
|
||||
m = defined_risk_metrics_from_store(store, cost_bps=5.5)
|
||||
inner.close()
|
||||
assert m["available"] is True
|
||||
assert m["wing_width"] > 0.0
|
||||
|
||||
|
||||
def test_defined_risk_metrics_reports_tail() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
m = defined_risk_metrics_from_store(store, cost_bps=0.0)
|
||||
store.close()
|
||||
assert m["available"] is True
|
||||
assert m["max_dd"] <= 0.0
|
||||
assert "worst_day" in m and "avg_leverage" in m
|
||||
|
||||
|
||||
# --- 4. THE TAIL COMPARISON: defined-risk maxDD << naked maxDD on the SAME spike ---------------
|
||||
|
||||
def test_defined_risk_maxdd_dramatically_shallower_than_naked_on_spike() -> None:
|
||||
"""THE WHOLE POINT. On the SAME catastrophic-spike fixture, the defined-risk (wings) curve must have a
|
||||
DRAMATICALLY shallower maxDD than the naked variance-swap curve — the wings cap the tail the naked book
|
||||
cannot. Both are vol-targeted+costed by the identical harness; only the construction differs."""
|
||||
from fxhnt.application.vrp_eval import vrp_metrics_from_store
|
||||
|
||||
naked_store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_with_spike(naked_store)
|
||||
naked = vrp_metrics_from_store(naked_store, cost_bps=5.5, direction="short_vol")
|
||||
naked_store.close()
|
||||
|
||||
capped_store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_with_spike(capped_store)
|
||||
capped = defined_risk_metrics_from_store(capped_store, cost_bps=5.5, direction="short_vol",
|
||||
wing_width=0.15)
|
||||
capped_store.close()
|
||||
|
||||
assert naked["available"] and capped["available"]
|
||||
# the defined-risk drawdown is materially shallower than the naked one (the wings work).
|
||||
assert capped["max_dd"] > naked["max_dd"]
|
||||
assert capped["max_dd"] > 1.3 * naked["max_dd"] # at least ~30% shallower (maxDD is negative)
|
||||
|
||||
|
||||
# --- 4b. THE TURNOVER / COST-ACCOUNTING MODEL (the amortized daily roll, not full-ladder-daily) -
|
||||
|
||||
def test_daily_cost_is_one_spreads_legs_not_the_whole_ladder() -> None:
|
||||
"""THE CORE TURNOVER FIX. A 30-day-roll condor ladder opens exactly ONE new spread per day (n_legs legs)
|
||||
and lets one expire (settles — NO closing trade). So the daily traded notional is (n_legs / swap_days) of
|
||||
the gross book, NOT the whole k-leverage ladder re-traded every day. The defined-risk daily haircut must be
|
||||
k · (n_legs / swap_days) · cost_bps/1e4 — far smaller than the naked daily-rehedge haircut k · cost_bps/1e4."""
|
||||
k, cost_bps, swap_days = 3.0, 50.0, 30
|
||||
dr = _defined_risk_daily_cost(k=k, cost_bps=cost_bps, swap_days=swap_days)
|
||||
expected = k * (_N_LEGS / swap_days) * (cost_bps / 1e4)
|
||||
assert math.isclose(dr, expected, rel_tol=1e-12)
|
||||
# it is exactly (n_legs / swap_days) of the naked full-ladder-daily haircut.
|
||||
naked_full_ladder_daily = k * (cost_bps / 1e4)
|
||||
assert math.isclose(dr, naked_full_ladder_daily * (_N_LEGS / swap_days), rel_tol=1e-12)
|
||||
# with 4 legs over a 30-day roll that is ~7.5x cheaper than charging the whole ladder daily.
|
||||
assert dr < naked_full_ladder_daily / 7.0
|
||||
|
||||
|
||||
def test_old_full_ladder_daily_overcharging_fails_the_turnover_assertion() -> None:
|
||||
"""REGRESSION GUARD. The OLD behavior charged the FULL ladder every day (k · cost_bps/1e4 — the naked
|
||||
daily-rehedge model). On a 30-day-roll 4-leg condor that over-charges turnover by swap_days/n_legs = 7.5x;
|
||||
the corrected daily roll must be strictly, materially smaller — proving the bug is fixed, not re-introduced."""
|
||||
k, cost_bps, swap_days = 3.0, 50.0, 30
|
||||
old_full_ladder_daily = k * (cost_bps / 1e4) # the buggy model
|
||||
corrected = _defined_risk_daily_cost(k=k, cost_bps=cost_bps, swap_days=swap_days)
|
||||
over_charge_factor = old_full_ladder_daily / corrected
|
||||
assert math.isclose(over_charge_factor, swap_days / _N_LEGS, rel_tol=1e-9)
|
||||
assert over_charge_factor > 7.0 # ~30x/4legs ≈ 7.5x over-charge
|
||||
|
||||
|
||||
def test_defined_risk_sized_series_uses_amortized_roll_cost() -> None:
|
||||
"""The defined-risk sized series subtracts the AMORTIZED daily-roll haircut from each k·raw day — NOT the
|
||||
naked full-ladder haircut the reused vrp_eval._sized_series applies."""
|
||||
raw = {1: 0.001, 2: -0.002, 3: 0.0015}
|
||||
k, cost_bps, swap_days = 2.0, 50.0, 30
|
||||
dr = _defined_risk_sized_series(raw, k=k, cost_bps=cost_bps, swap_days=swap_days)
|
||||
haircut = _defined_risk_daily_cost(k=k, cost_bps=cost_bps, swap_days=swap_days)
|
||||
for d in raw:
|
||||
assert math.isclose(dr[d], k * raw[d] - haircut, rel_tol=1e-12, abs_tol=1e-15)
|
||||
# and it is strictly less punitive than the naked full-ladder series on every day (cost-only difference).
|
||||
naked = vrp_eval._sized_series(raw, k=k, cost_bps=cost_bps)
|
||||
for d in raw:
|
||||
assert dr[d] > naked[d]
|
||||
|
||||
|
||||
def test_per_year_cost_drag_is_proportionate_not_strategy_destroying() -> None:
|
||||
"""SANITY on the realized cost drag. The per-YEAR cost of the amortized roll is
|
||||
n_legs · cost_bps · (365/swap_days) · 1 (per unit notional, leverage 1). At 50bp on a 4-leg 30-day roll
|
||||
that is 4·0.005·(365/30) ≈ 24%/yr at leverage 1 — a meaningful but NON-catastrophic drag, NOT the ~7.5x
|
||||
inflated number the old full-ladder-daily model implied (which would be ~180%/yr)."""
|
||||
cost_bps, swap_days = 50.0, 30
|
||||
per_year_drag = _N_LEGS * (cost_bps / 1e4) * 365 / swap_days
|
||||
assert 0.20 < per_year_drag < 0.30 # ~24%/yr at leverage 1 — bounded
|
||||
# at the cheap 5.5bp the per-year drag is tiny (~2-3%/yr at leverage 1).
|
||||
cheap = _N_LEGS * (5.5 / 1e4) * 365 / swap_days
|
||||
assert cheap < 0.04
|
||||
|
||||
|
||||
def test_cost_sensitivity_is_proportionate_5p5_close_to_gross_50_bounded() -> None:
|
||||
"""END-TO-END SANITY: the corrected cost makes the strategy COST-PROPORTIONATE, not cost-destroyed. On the
|
||||
tail-survivable fixture: at 5.5bp the net Sharpe is CLOSE to gross (small drag, NO sign flip); at 50bp the
|
||||
drag is meaningful but BOUNDED — Sharpe stays in the same ballpark and maxDD nowhere near −100%."""
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_survivable(store)
|
||||
gross = defined_risk_metrics_from_store(store, cost_bps=0.0, direction="short_vol")
|
||||
cheap = defined_risk_metrics_from_store(store, cost_bps=5.5, direction="short_vol")
|
||||
pricey = defined_risk_metrics_from_store(store, cost_bps=50.0, direction="short_vol")
|
||||
store.close()
|
||||
assert gross["available"] and cheap["available"] and pricey["available"]
|
||||
assert gross["sharpe"] > 0.0
|
||||
# 5.5bp: a SMALL drag, no sign flip, close to gross.
|
||||
assert cheap["sharpe"] > 0.0
|
||||
assert cheap["sharpe"] > 0.80 * gross["sharpe"]
|
||||
# 50bp: meaningful but proportionate — still a sizeable fraction of gross, NOT a catastrophic collapse.
|
||||
assert pricey["sharpe"] > 0.40 * gross["sharpe"]
|
||||
# the tail stays survivable (the wings cap it) — nowhere near −100%.
|
||||
assert pricey["max_dd"] > -0.40
|
||||
|
||||
|
||||
# --- 5. adversarial verify ---------------------------------------------------------------------
|
||||
|
||||
def test_verify_cost_monotone_incl_option_costs() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
rep = verify_defined_risk_edge(store, cost_bps=5.5, cost_grid=(5.5, 11.0, 22.0, 50.0, 100.0))
|
||||
store.close()
|
||||
sv = rep["net_cost"]["short_vol"]
|
||||
grid = sorted(sv)
|
||||
for a, b in zip(grid, grid[1:], strict=False):
|
||||
assert sv[a]["sharpe"] >= sv[b]["sharpe"]
|
||||
assert 50.0 in sv and 100.0 in sv
|
||||
|
||||
|
||||
def test_verify_reports_correlation_with_all_four_edges() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
rep = verify_defined_risk_edge(store, cost_bps=5.5)
|
||||
store.close()
|
||||
for d in ("short_vol", "long_vol"):
|
||||
for edge in ("tstrend", "unlock", "xsfunding", "positioning"):
|
||||
assert edge in rep["corr_edges"][d]
|
||||
v = rep["corr_edges"][d][edge]
|
||||
assert v is None or (-1.0 - 1e-9 <= v <= 1.0 + 1e-9)
|
||||
|
||||
|
||||
def test_verify_is_read_only() -> None:
|
||||
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(inner)
|
||||
store = _WriteTripwireStore(inner)
|
||||
rep = verify_defined_risk_edge(store, cost_bps=5.5)
|
||||
inner.close()
|
||||
assert rep["available"] is True
|
||||
assert rep["wing_width"] > 0.0
|
||||
|
||||
|
||||
# --- 6. look-ahead audit -----------------------------------------------------------------------
|
||||
|
||||
def test_look_ahead_clean_on_causal_signal() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_varying_dvol(store, days=120)
|
||||
ok = look_ahead_audit_defined_risk_ok(store)
|
||||
store.close()
|
||||
assert ok is True
|
||||
|
||||
|
||||
def test_look_ahead_audit_catches_leaked_mapping() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_varying_dvol(store, days=120)
|
||||
ok = look_ahead_audit_defined_risk_ok(store, _leak=True)
|
||||
store.close()
|
||||
assert ok is False
|
||||
|
||||
|
||||
# --- 7. walk-forward / OOS deploy gate ---------------------------------------------------------
|
||||
|
||||
def test_walk_forward_runs_and_reports_verdict() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_survivable(store)
|
||||
rep = walk_forward_defined_risk(store, cost_bps=5.5, window_days=180)
|
||||
store.close()
|
||||
assert rep["available"] is True
|
||||
assert "deployable" in rep["verdict"]
|
||||
assert rep["look_ahead"]["causal"] is True
|
||||
assert rep["wing_width"] > 0.0
|
||||
|
||||
|
||||
def test_walk_forward_tail_survivable_passes_tail_check() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_survivable(store)
|
||||
rep = walk_forward_defined_risk(store, cost_bps=5.5, window_days=180)
|
||||
store.close()
|
||||
# the wings make the tail survivable (the defining property of the defined-risk form).
|
||||
assert rep["verdict"]["checks"]["tail_survivable"] is True
|
||||
|
||||
|
||||
def test_walk_forward_is_read_only() -> None:
|
||||
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_survivable(inner)
|
||||
store = _WriteTripwireStore(inner)
|
||||
rep = walk_forward_defined_risk(store, cost_bps=5.5)
|
||||
inner.close()
|
||||
assert rep["available"] is True
|
||||
|
||||
|
||||
# --- 8. CLI ------------------------------------------------------------------------------------
|
||||
|
||||
def test_cli_defined_risk_prints_metrics(monkeypatch) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
import fxhnt.cli as cli
|
||||
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
monkeypatch.setattr(
|
||||
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
|
||||
lambda *_a, **_k: store, raising=False)
|
||||
|
||||
result = CliRunner().invoke(cli.app, ["vrp-defined-risk-eval"])
|
||||
store.close()
|
||||
assert result.exit_code == 0, result.output
|
||||
out = result.output.lower()
|
||||
assert "defined" in out or "wing" in out
|
||||
assert "maxdd" in out or "max dd" in out
|
||||
assert "worst" in out
|
||||
|
||||
|
||||
def test_cli_defined_risk_verify(monkeypatch) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
import fxhnt.cli as cli
|
||||
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_with_spike(store, days=400)
|
||||
monkeypatch.setattr(
|
||||
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
|
||||
lambda *_a, **_k: store, raising=False)
|
||||
|
||||
result = CliRunner().invoke(cli.app, ["vrp-defined-risk-eval", "--verify"])
|
||||
store.close()
|
||||
assert result.exit_code == 0, result.output
|
||||
out = result.output.lower()
|
||||
assert "short" in out and "long" in out
|
||||
assert "100" in out
|
||||
assert "tstrend" in out and "positioning" in out
|
||||
|
||||
|
||||
def test_cli_defined_risk_walk_forward(monkeypatch) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
import fxhnt.cli as cli
|
||||
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_survivable(store)
|
||||
monkeypatch.setattr(
|
||||
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
|
||||
lambda *_a, **_k: store, raising=False)
|
||||
|
||||
result = CliRunner().invoke(cli.app, ["vrp-defined-risk-eval", "--walk-forward", "--cost-bps", "50"])
|
||||
store.close()
|
||||
assert result.exit_code == 0, result.output
|
||||
out = result.output.lower()
|
||||
assert "walk-forward" in out or "walk forward" in out
|
||||
assert "verdict" in out
|
||||
assert "pass" in out or "fail" in out
|
||||
@@ -1,493 +0,0 @@
|
||||
"""READ-ONLY, TAIL-HONEST evaluator for the NEW VRP (volatility-risk-premium) edge.
|
||||
|
||||
Thesis: Deribit DVOL (implied vol) systematically exceeds realized vol, so SELLING variance (short-vol)
|
||||
earns the spread — but with fat LEFT tails on vol spikes (RV >> IV). The signal/backtest uses DVOL (IV) +
|
||||
close-to-close RV; live execution would be short Bybit option straddles (a cross-venue caveat).
|
||||
|
||||
VRP return (per asset, per day, CAUSAL):
|
||||
short_var_ret_t = k * [ (DVOL_{t-1}/sqrt(365)/100)^2 - ret_t^2 ]
|
||||
DVOL_{t-1} is the IV known at the START of day t (no look-ahead); ret_t = close-to-close (t-1 -> t). The book
|
||||
is BTC+ETH equal-weight, vol-targeted to a sane annual vol (k). direction="short_vol" harvests; "long_vol" is
|
||||
the negation.
|
||||
|
||||
Tests assert:
|
||||
* the VRP return is POSITIVE on a calm-IV-high fixture and BIG-NEGATIVE on a spike fixture (the tail shows);
|
||||
* sizing (vol-target) scales the series; direction flips the sign;
|
||||
* the evaluator is READ-ONLY (a write-tripwire never trips);
|
||||
* verify is cost-monotone (incl 50/100bp option costs), per-year, reports worst-day + maxDD, and the
|
||||
correlation vs ALL FOUR existing edges (tstrend/unlock/xsfunding/positioning);
|
||||
* walk-forward train/test OOS + look-ahead-clean; a tail-survivable fixture PASSES, a tail-dominated one
|
||||
FAILS;
|
||||
* the CLI smoke prints metrics + verify + walk-forward (mocked in-memory store, NO network).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
from fxhnt.adapters.warehouse.timescale_feature_store import TimescaleFeatureStore
|
||||
from fxhnt.application.vrp_eval import (
|
||||
look_ahead_audit_vrp,
|
||||
short_var_return,
|
||||
variance_swap_payoff,
|
||||
verify_vrp_edge,
|
||||
vrp_metrics_from_store,
|
||||
vrp_returns_from_store,
|
||||
walk_forward_vrp,
|
||||
)
|
||||
|
||||
_DAY = 86_400
|
||||
_START = 19_358 # 2023-01-01 epoch day
|
||||
|
||||
|
||||
class _WriteTripwireStore:
|
||||
"""Read-only proxy: forwards reads, but ANY write/persist call trips an assertion."""
|
||||
|
||||
_FORBIDDEN = frozenset({
|
||||
"write_features", "write_features_bulk", "upsert_feature_rows", "upsert_membership",
|
||||
"replace_positions", "upsert_nav", "upsert_sleeve_ret", "replace_shadow_positions",
|
||||
"replace_trades", "_create_schema", "_bulk_upsert",
|
||||
})
|
||||
|
||||
def __init__(self, inner: TimescaleFeatureStore) -> None:
|
||||
object.__setattr__(self, "_inner", inner)
|
||||
|
||||
def __getattr__(self, name: str):
|
||||
if name in _WriteTripwireStore._FORBIDDEN:
|
||||
raise AssertionError(f"READ-ONLY violation: evaluator called write method {name!r}")
|
||||
return getattr(self._inner, name)
|
||||
|
||||
|
||||
# --- 1. short_var_return (the formula) ---------------------------------------------------------
|
||||
|
||||
def test_short_var_positive_when_calm_iv_high() -> None:
|
||||
# IV (DVOL) = 80 vol pts; realized daily move tiny -> RV << IV -> short-vol HARVESTS (positive).
|
||||
r = short_var_return(dvol_prev=80.0, ret=0.001)
|
||||
assert r > 0.0
|
||||
|
||||
|
||||
def test_short_var_big_negative_on_spike() -> None:
|
||||
# IV = 40 vol pts (calm implied), but realized move is a -20% spike -> RV >> IV -> big NEGATIVE (the tail).
|
||||
calm = short_var_return(dvol_prev=40.0, ret=0.005)
|
||||
spike = short_var_return(dvol_prev=40.0, ret=-0.20)
|
||||
assert spike < 0.0
|
||||
assert spike < calm
|
||||
# the spike loss dwarfs a calm-day gain in magnitude (fat left tail).
|
||||
assert abs(spike) > 5.0 * abs(calm)
|
||||
|
||||
|
||||
def test_short_var_implied_daily_variance_is_iv_squared() -> None:
|
||||
# On a zero-realized-move day the harvest equals the implied daily variance (DVOL/sqrt(365)/100)^2.
|
||||
iv_daily = 60.0 / math.sqrt(365) / 100.0
|
||||
r = short_var_return(dvol_prev=60.0, ret=0.0)
|
||||
assert math.isclose(r, iv_daily ** 2, rel_tol=1e-12)
|
||||
|
||||
|
||||
def test_long_vol_is_negation_of_short_vol() -> None:
|
||||
assert math.isclose(short_var_return(dvol_prev=50.0, ret=0.03, direction="long_vol"),
|
||||
-short_var_return(dvol_prev=50.0, ret=0.03, direction="short_vol"),
|
||||
rel_tol=1e-12)
|
||||
|
||||
|
||||
# --- 1b. variance_swap_payoff (the PROPER variance swap: implied strike vs realized-WINDOW variance) ----
|
||||
|
||||
def test_variance_swap_payoff_positive_when_calm_window() -> None:
|
||||
# IV (DVOL) = 60 vol pts (strike K=0.36 annual var); realized window is tiny ±0.2% daily moves -> RV << K
|
||||
# -> short-var swap HARVESTS the premium (positive).
|
||||
fwd = [0.002 if i % 2 == 0 else -0.002 for i in range(30)]
|
||||
assert variance_swap_payoff(dvol_entry=60.0, fwd_rets=fwd, swap_days=30) > 0.0
|
||||
|
||||
|
||||
def test_variance_swap_payoff_negative_on_spike_window() -> None:
|
||||
# IV = 40 vol pts (K=0.16); the realized window contains a -25% crash -> RV >> K -> NEGATIVE (the tail).
|
||||
fwd = [0.001] * 29 + [-0.25]
|
||||
assert variance_swap_payoff(dvol_entry=40.0, fwd_rets=fwd, swap_days=30) < 0.0
|
||||
|
||||
|
||||
def test_variance_swap_payoff_annualization_zero_when_rv_equals_iv() -> None:
|
||||
# Constant-vol synthetic: pick a daily move whose annualised realized variance EXACTLY equals the strike
|
||||
# K=(DVOL/100)^2 -> the short-var swap payoff is ~0 (the annualisation 365/swap_days is correct).
|
||||
dvol = 50.0
|
||||
k = (dvol / 100.0) ** 2 # annual implied variance = 0.25
|
||||
daily_var = k / 365.0 # per-day variance so (365/n)*n*daily_var == k
|
||||
move = math.sqrt(daily_var)
|
||||
fwd = [move if i % 2 == 0 else -move for i in range(30)]
|
||||
payoff = variance_swap_payoff(dvol_entry=dvol, fwd_rets=fwd, swap_days=30)
|
||||
assert abs(payoff) < 1e-12
|
||||
|
||||
|
||||
def test_variance_swap_payoff_long_is_negation() -> None:
|
||||
fwd = [0.01, -0.02, 0.015] * 10
|
||||
assert math.isclose(variance_swap_payoff(dvol_entry=55.0, fwd_rets=fwd, direction="long_vol"),
|
||||
-variance_swap_payoff(dvol_entry=55.0, fwd_rets=fwd, direction="short_vol"),
|
||||
rel_tol=1e-12)
|
||||
|
||||
|
||||
def test_variance_swap_payoff_empty_window_is_zero() -> None:
|
||||
assert variance_swap_payoff(dvol_entry=60.0, fwd_rets=[]) == 0.0
|
||||
|
||||
|
||||
# --- 2. vrp_returns_from_store / sizing --------------------------------------------------------
|
||||
|
||||
def _seed_calm_premium(store: TimescaleFeatureStore, *, days: int = 260, dvol: float = 60.0,
|
||||
daily_move: float = 0.004) -> None:
|
||||
"""Seed BTC+ETH so IV (dvol=60) comfortably exceeds RV (a tiny ±daily_move oscillation): short-vol earns
|
||||
a steady premium. close oscillates so RV is small + bounded; dvol flat-high. Spans ~9 months."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
sign = 1.0 if i % 2 == 0 else -1.0
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol})])
|
||||
px[s] *= (1.0 + sign * daily_move)
|
||||
|
||||
|
||||
def test_vrp_returns_positive_on_calm_premium() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
series = vrp_returns_from_store(store, cost_bps=0.0, direction="short_vol")
|
||||
store.close()
|
||||
assert len(series) > 2
|
||||
assert sum(series.values()) > 0.0 # the premium is harvested gross
|
||||
|
||||
|
||||
def test_vrp_sizing_targets_annual_vol() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
s15 = vrp_returns_from_store(store, cost_bps=0.0, target_ann_vol=0.15)
|
||||
s30 = vrp_returns_from_store(store, cost_bps=0.0, target_ann_vol=0.30)
|
||||
store.close()
|
||||
import statistics as st
|
||||
v15 = st.pstdev(s15.values()) * math.sqrt(365)
|
||||
v30 = st.pstdev(s30.values()) * math.sqrt(365)
|
||||
# vol-target hits (roughly) the requested annual vol, and doubling the target ~doubles realized vol.
|
||||
assert abs(v15 - 0.15) < 0.03
|
||||
assert v30 > 1.8 * v15
|
||||
|
||||
|
||||
def test_vrp_constant_premium_series_is_low_noise_and_sane() -> None:
|
||||
"""THE KEY FIX. A constant-premium synthetic (IV steadily > RV every day) must give a SMOOTH, LOW-noise,
|
||||
SANE-Sharpe daily series — NOT the −40 Sharpe / blow-up the old single-day-ret^2 proxy produced. The
|
||||
rolling variance-swap ladder averages RV over a 30-day window, so the day-to-day series is far less noisy
|
||||
than a raw ret^2. We assert the GROSS (pre-vol-target) ladder series has a high, positive Sharpe and tiny
|
||||
relative dispersion — i.e. it is well-behaved enough that vol-targeting it can't lose ~everything."""
|
||||
import statistics as st
|
||||
|
||||
from fxhnt.application.vrp_eval import _book_short_var
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store, days=400, dvol=60.0, daily_move=0.004)
|
||||
raw = _book_short_var(store, direction="short_vol")
|
||||
store.close()
|
||||
vals = [raw[d] for d in sorted(raw)]
|
||||
assert len(vals) > 30
|
||||
mean, sd = st.mean(vals), st.pstdev(vals)
|
||||
assert mean > 0.0 # a steady harvested premium
|
||||
gross_sharpe = mean / sd * math.sqrt(365)
|
||||
# the ladder series is sane: a strongly POSITIVE Sharpe (the old single-day proxy gave -40 / blew up).
|
||||
assert gross_sharpe > 3.0
|
||||
# and genuinely low-noise: the daily dispersion is small relative to the mean (a smooth window-averaged
|
||||
# quantity, not a spiky raw ret^2).
|
||||
assert sd < mean # coefficient of variation < 1 (the smoothing fix)
|
||||
|
||||
|
||||
def test_vrp_vol_targeted_constant_premium_does_not_blow_up_drawdown() -> None:
|
||||
"""The corrected + vol-targeted constant-premium curve must be SANE: no catastrophic drawdown. The old
|
||||
proxy's single-day spikes blew through the vol-target to −96%/180d; the window-averaged ladder must not."""
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store, days=400, dvol=60.0, daily_move=0.004)
|
||||
m = vrp_metrics_from_store(store, cost_bps=5.5, target_ann_vol=0.15)
|
||||
store.close()
|
||||
assert m["available"] is True
|
||||
assert m["sharpe"] > 1.0 # a real, sane edge on the calm fixture
|
||||
assert m["max_dd"] > -0.40 # NOT a −96% blow-up
|
||||
|
||||
|
||||
def test_vrp_direction_flips_sign() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
short = vrp_returns_from_store(store, cost_bps=0.0, direction="short_vol")
|
||||
long_ = vrp_returns_from_store(store, cost_bps=0.0, direction="long_vol")
|
||||
store.close()
|
||||
assert sum(short.values()) > 0.0 > sum(long_.values())
|
||||
|
||||
|
||||
def test_vrp_metrics_na_when_no_dvol() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
store.write_features("BTCUSDT", [(0, {"close": 30_000.0})]) # close but no dvol
|
||||
m = vrp_metrics_from_store(store)
|
||||
store.close()
|
||||
assert m["available"] is False
|
||||
assert "reason" in m and m["reason"]
|
||||
|
||||
|
||||
def test_vrp_metrics_is_read_only() -> None:
|
||||
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(inner)
|
||||
store = _WriteTripwireStore(inner)
|
||||
m = vrp_metrics_from_store(store, cost_bps=5.5)
|
||||
inner.close()
|
||||
assert m["available"] is True
|
||||
|
||||
|
||||
def test_vrp_metrics_reports_tail() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
m = vrp_metrics_from_store(store, cost_bps=0.0)
|
||||
store.close()
|
||||
assert m["available"] is True
|
||||
assert m["max_dd"] <= 0.0 # maxDD is a (non-positive) drawdown
|
||||
assert "worst_day" in m # the single worst vol-spike day loss
|
||||
assert "avg_leverage" in m # exposure from the vol-target
|
||||
|
||||
|
||||
# --- 3. tail handling: a spike fixture must show the loss ---------------------------------------
|
||||
|
||||
def _seed_with_spike(store: TimescaleFeatureStore, *, days: int = 260, spike_at: int = 130) -> None:
|
||||
"""Calm short-vol premium most days, but ONE catastrophic vol-spike day (a -35% move with calm prior IV)
|
||||
that drives RV >> IV -> a huge negative VRP return on that day (the tail the backtest must capture)."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": 55.0})])
|
||||
if i == spike_at:
|
||||
px[s] *= (1.0 - 0.35) # crash: realized vol explodes past implied
|
||||
else:
|
||||
px[s] *= (1.0 + (0.003 if i % 2 == 0 else -0.003))
|
||||
|
||||
|
||||
def test_spike_day_is_the_worst_day_and_dominates_maxdd() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_with_spike(store)
|
||||
m = vrp_metrics_from_store(store, cost_bps=0.0)
|
||||
series = vrp_returns_from_store(store, cost_bps=0.0)
|
||||
store.close()
|
||||
worst = min(series.values())
|
||||
assert worst < 0.0
|
||||
assert math.isclose(m["worst_day"], worst, rel_tol=1e-9)
|
||||
# the spike produces a material drawdown — the tail is captured, not smoothed away.
|
||||
assert m["max_dd"] < -0.05
|
||||
|
||||
|
||||
# --- 4. adversarial verify ---------------------------------------------------------------------
|
||||
|
||||
def test_verify_cost_monotone_incl_option_costs() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
rep = verify_vrp_edge(store, cost_bps=5.5, cost_grid=(5.5, 11.0, 22.0, 50.0, 100.0))
|
||||
store.close()
|
||||
sv = rep["net_cost"]["short_vol"]
|
||||
grid = sorted(sv)
|
||||
for a, b in zip(grid, grid[1:], strict=False):
|
||||
assert sv[a]["sharpe"] >= sv[b]["sharpe"] # higher cost -> lower Sharpe
|
||||
assert 50.0 in sv and 100.0 in sv # realistic option-cost sensitivity shown
|
||||
|
||||
|
||||
def test_verify_reports_worst_day_and_per_year() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_with_spike(store, days=400) # spans 2023 and 2024
|
||||
rep = verify_vrp_edge(store, cost_bps=5.5)
|
||||
store.close()
|
||||
assert rep["available"] is True
|
||||
assert rep["worst_day"]["short_vol"] < 0.0
|
||||
py = rep["per_year"]["short_vol"]
|
||||
assert len(py) >= 2
|
||||
|
||||
|
||||
def test_verify_reports_correlation_with_all_four_edges() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
rep = verify_vrp_edge(store, cost_bps=5.5)
|
||||
store.close()
|
||||
for d in ("short_vol", "long_vol"):
|
||||
corrs = rep["corr_edges"][d]
|
||||
for edge in ("tstrend", "unlock", "xsfunding", "positioning"):
|
||||
assert edge in corrs
|
||||
v = corrs[edge]
|
||||
assert v is None or (-1.0 - 1e-9 <= v <= 1.0 + 1e-9)
|
||||
|
||||
|
||||
def test_verify_is_read_only() -> None:
|
||||
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(inner)
|
||||
store = _WriteTripwireStore(inner)
|
||||
rep = verify_vrp_edge(store, cost_bps=5.5)
|
||||
inner.close()
|
||||
assert rep["available"] is True
|
||||
|
||||
|
||||
# --- 5. look-ahead audit -----------------------------------------------------------------------
|
||||
|
||||
def _seed_varying_dvol(store: TimescaleFeatureStore, *, days: int = 120) -> None:
|
||||
"""A premium fixture with NON-PERIODIC TIME-VARYING DVOL so the causal (strike=DVOL at swap entry) and
|
||||
leaked (strike=DVOL at the window END) mappings are genuinely distinguishable on EVERY day — the
|
||||
look-ahead audit can only bite when the entry-day and window-end IV differ. A slow irrational-step ramp
|
||||
guarantees no two days share a DVOL value (no coincidental entry/window-end collision)."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
dvol = 50.0 + 0.137 * i # strictly monotone, distinct every day (no collisions)
|
||||
sign = 1.0 if i % 2 == 0 else -1.0
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol})])
|
||||
px[s] *= (1.0 + sign * 0.004)
|
||||
|
||||
|
||||
def test_look_ahead_audit_passes_on_causal_signal() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_varying_dvol(store, days=120)
|
||||
rep = look_ahead_audit_vrp(store)
|
||||
store.close()
|
||||
assert rep["causal"] is True
|
||||
assert rep["leak_days"] == 0
|
||||
|
||||
|
||||
def test_look_ahead_audit_catches_a_deliberately_leaked_mapping() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_varying_dvol(store, days=120)
|
||||
rep = look_ahead_audit_vrp(store, _leak=True) # pair the SAME-day DVOL with the day's return
|
||||
store.close()
|
||||
assert rep["causal"] is False
|
||||
assert rep["leak_days"] > 0
|
||||
|
||||
|
||||
# --- 6. walk-forward / OOS deploy gate ---------------------------------------------------------
|
||||
|
||||
def _seed_tail_survivable(store: TimescaleFeatureStore, *, days: int = 700) -> None:
|
||||
"""A short-vol premium that SURVIVES its occasional tails: calm harvest most days (RV well below the
|
||||
varying IV), a moderate spike every ~90 days whose loss is recouped within the next window. Positive in
|
||||
most windows + OOS, drawdown bounded above the −40% floor. DVOL varies day to day (so the causality audit
|
||||
can distinguish the prior-day from the same-day IV)."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
dvol = 60.0 + 8.0 * math.sin(i * 0.11) # non-periodic-on-30d drift in 52..68 (no collisions)
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol, "turnover": 5e8})])
|
||||
if i % 90 == 45:
|
||||
px[s] *= (1.0 - 0.06) # moderate, recoverable spike
|
||||
else:
|
||||
px[s] *= (1.0 + (0.004 if i % 2 == 0 else -0.004))
|
||||
|
||||
|
||||
def _seed_tail_dominated(store: TimescaleFeatureStore, *, days: int = 700) -> None:
|
||||
"""A short-vol book the gate must FAIL: thin implied premium (low DVOL), but FREQUENT, LARGE spikes whose
|
||||
realized variance dwarfs the premium. After vol-targeting the rare-but-huge spike days produce a
|
||||
CATASTROPHIC drawdown (worse than the −40% floor) — VRP looks fine on the calm days and then a cluster of
|
||||
spikes wipes it out. DVOL varies day to day (so the causality audit can distinguish prior vs same day)."""
|
||||
px = {"BTCUSDT": 30_000.0, "ETHUSDT": 2_000.0}
|
||||
for i in range(days):
|
||||
d = _START + i
|
||||
dvol = 25.0 + 4.0 * math.sin(i * 0.11) # thin implied vol ~21..29, non-periodic-on-30d
|
||||
for s in ("BTCUSDT", "ETHUSDT"):
|
||||
store.write_features(s, [(d * _DAY, {"close": px[s], "dvol": dvol, "turnover": 5e8})])
|
||||
# spikes cluster late (after the train window) so the short-vol harvest that "worked" on TRAIN
|
||||
# is wiped in TEST -> OOS negative AND catastrophic drawdown.
|
||||
late_cluster = i > int(days * 0.6) and (i % 9 == 0)
|
||||
if late_cluster:
|
||||
px[s] *= (1.0 - 0.22) # frequent, large, late, unrecouped spikes
|
||||
else:
|
||||
px[s] *= (1.0 + (0.0008 if i % 2 == 0 else -0.0008))
|
||||
|
||||
|
||||
def test_walk_forward_pass_on_tail_survivable() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_survivable(store)
|
||||
rep = walk_forward_vrp(store, cost_bps=5.5, window_days=180)
|
||||
store.close()
|
||||
v = rep["verdict"]
|
||||
assert v["deployable"] is True
|
||||
assert v["checks"]["oos_direction_positive"] is True
|
||||
assert v["checks"]["tail_survivable"] is True
|
||||
assert v["checks"]["look_ahead_clean"] is True
|
||||
|
||||
|
||||
def test_walk_forward_fail_on_tail_dominated() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_dominated(store)
|
||||
rep = walk_forward_vrp(store, cost_bps=5.5, window_days=180)
|
||||
store.close()
|
||||
v = rep["verdict"]
|
||||
assert v["deployable"] is False
|
||||
# it fails because the tail is NOT survivable (and/or OOS is negative).
|
||||
assert v["checks"]["tail_survivable"] is False or v["checks"]["oos_direction_positive"] is False
|
||||
|
||||
|
||||
def test_walk_forward_at_realistic_option_costs_judges_net_of_tail() -> None:
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_dominated(store)
|
||||
# at a realistic OPTION cost (selling options isn't 5.5bp) the dominated book is plainly undeployable.
|
||||
rep = walk_forward_vrp(store, cost_bps=50.0, window_days=180)
|
||||
store.close()
|
||||
assert rep["verdict"]["deployable"] is False
|
||||
|
||||
|
||||
def test_walk_forward_is_read_only() -> None:
|
||||
inner = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_survivable(inner)
|
||||
store = _WriteTripwireStore(inner)
|
||||
rep = walk_forward_vrp(store, cost_bps=5.5)
|
||||
inner.close()
|
||||
assert rep["available"] is True
|
||||
|
||||
|
||||
# --- 7. CLI ------------------------------------------------------------------------------------
|
||||
|
||||
def test_cli_vrp_eval_prints_metrics(monkeypatch) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
import fxhnt.cli as cli
|
||||
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_calm_premium(store)
|
||||
monkeypatch.setattr(
|
||||
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
|
||||
lambda *_a, **_k: store, raising=False)
|
||||
|
||||
result = CliRunner().invoke(cli.app, ["vrp-eval"])
|
||||
store.close()
|
||||
assert result.exit_code == 0, result.output
|
||||
out = result.output.lower()
|
||||
assert "vrp" in out or "volatility risk premium" in out
|
||||
assert "maxdd" in out or "max dd" in out
|
||||
assert "worst" in out
|
||||
|
||||
|
||||
def test_cli_vrp_eval_verify_prints_diagnostics(monkeypatch) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
import fxhnt.cli as cli
|
||||
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_with_spike(store, days=400)
|
||||
monkeypatch.setattr(
|
||||
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
|
||||
lambda *_a, **_k: store, raising=False)
|
||||
|
||||
result = CliRunner().invoke(cli.app, ["vrp-eval", "--verify"])
|
||||
store.close()
|
||||
assert result.exit_code == 0, result.output
|
||||
out = result.output.lower()
|
||||
assert "short" in out and "long" in out
|
||||
assert "worst" in out
|
||||
assert "100" in out # the 100bp option-cost column
|
||||
assert "tstrend" in out and "positioning" in out
|
||||
|
||||
|
||||
def test_cli_vrp_eval_walk_forward_prints_verdict(monkeypatch) -> None:
|
||||
from typer.testing import CliRunner
|
||||
|
||||
import fxhnt.cli as cli
|
||||
|
||||
store = TimescaleFeatureStore("sqlite://", table="bybit_features")
|
||||
_seed_tail_survivable(store)
|
||||
monkeypatch.setattr(
|
||||
"fxhnt.adapters.warehouse.timescale_feature_store.TimescaleFeatureStore",
|
||||
lambda *_a, **_k: store, raising=False)
|
||||
|
||||
result = CliRunner().invoke(cli.app, ["vrp-eval", "--walk-forward"])
|
||||
store.close()
|
||||
assert result.exit_code == 0, result.output
|
||||
out = result.output.lower()
|
||||
assert "walk-forward" in out or "walk forward" in out
|
||||
assert "verdict" in out
|
||||
assert "pass" in out or "fail" in out
|
||||
@@ -1,291 +0,0 @@
|
||||
import datetime as dt
|
||||
|
||||
from fxhnt.adapters.orchestration.assets import (
|
||||
_freeze_day,
|
||||
_freeze_latest_session,
|
||||
_freeze_xsp_slice,
|
||||
_is_degenerate_forward_error,
|
||||
)
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
|
||||
|
||||
def test_vrp_nav_survives_freeze_failure(tmp_path, monkeypatch):
|
||||
"""C1b: a freeze failure (bad-data day / still-unavailable OPRA window / cost-cap trip) must NOT kill
|
||||
vrp_nav — it logs a WARNING and CONTINUES to `_run_paper_tracker`, which recomputes off the FROZEN
|
||||
xsp_option_bars table. Guarantees vrp always produces a forward row/anchor/ref off whatever IS frozen and
|
||||
never skips the downstream gate (`cockpit_forward`)."""
|
||||
from dagster import build_op_context
|
||||
|
||||
import fxhnt.adapters.orchestration.assets as assets
|
||||
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
|
||||
from fxhnt.config import Settings
|
||||
|
||||
monkeypatch.setattr("fxhnt.config.get_settings",
|
||||
lambda: Settings(operational_dsn=f"sqlite:///{tmp_path / 'op.db'}"))
|
||||
|
||||
def _boom(*a, **k): # the freeze entry point exhausts its bounded step-back → catchable range error
|
||||
raise DatabentoRangeUnavailable("no complete OPRA session within the step-back bound")
|
||||
monkeypatch.setattr(assets, "_freeze_latest_session", _boom)
|
||||
|
||||
ran = {"tracker": False}
|
||||
|
||||
def _fake_tracker(context, name, build_strategy, *, persist_backtest_ref=False):
|
||||
ran["tracker"] = True
|
||||
return {"forward_days": 0, "last_date": None, "reinception": False}
|
||||
monkeypatch.setattr(assets, "_run_paper_tracker", _fake_tracker)
|
||||
|
||||
out = assets.vrp_nav(build_op_context())
|
||||
assert isinstance(out, dict) # returned a dict, did NOT raise out of the asset
|
||||
assert ran["tracker"] # freeze died, but the tracker STILL recomputed off the frozen table
|
||||
|
||||
|
||||
class _FakeDbn:
|
||||
def __init__(self):
|
||||
self.chain_calls = 0
|
||||
|
||||
def fetch_option_chain(self, parent, as_of):
|
||||
from fxhnt.domain.models import OptionContract
|
||||
self.chain_calls += 1
|
||||
puts = [OptionContract(f"XSP240816P00{k}000", dt.date(2024, 8, 16), float(k), "P")
|
||||
for k in range(440, 475, 5)]
|
||||
calls = [OptionContract(f"XSP240816C00{k}000", dt.date(2024, 8, 16), float(k), "C")
|
||||
for k in range(460, 481, 5)]
|
||||
return puts + calls
|
||||
|
||||
def fetch_option_bars(self, osi, start, end):
|
||||
return {s: {start: 1.0} for s in osi}
|
||||
|
||||
|
||||
def test_freeze_xsp_slice_writes_puts_and_atm_calls(tmp_path):
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'o.db'}")
|
||||
bars.migrate()
|
||||
_freeze_xsp_slice(_FakeDbn(), bars, as_of="2024-07-20", forward=470.0, at=dt.datetime(2024, 7, 20))
|
||||
puts = bars.chain_asof("2024-07-20", 20, 45, right="P")
|
||||
calls = bars.chain_asof("2024-07-20", 20, 45, right="C")
|
||||
# puts moneyness [0.70,1.20]*470 = [329,564] (wide band so held legs stay frozen) -> all seeded puts qualify
|
||||
assert {c.strike for c in puts} == {float(k) for k in range(440, 475, 5)}
|
||||
# calls moneyness [0.96,1.04]*470 = [451.2,488.8] -> 460..480 qualify (so parity has a common strike)
|
||||
assert {c.strike for c in calls} == {460.0, 465.0, 470.0, 475.0, 480.0}
|
||||
|
||||
|
||||
class _RangeFakeDbn:
|
||||
"""Fake for the BATCHED backfill: one range-definition + one range-ohlcv call per month, marks across days."""
|
||||
def __init__(self):
|
||||
from fxhnt.domain.models import OptionContract
|
||||
exp = dt.date(2024, 8, 5)
|
||||
self.puts = [OptionContract(f"XSP240805P00{k}000", exp, float(k), "P") for k in range(455, 481, 5)]
|
||||
self.calls = [OptionContract(f"XSP240805C00{k}000", exp, float(k), "C") for k in range(460, 481, 5)]
|
||||
self.days = ["2024-07-01", "2024-07-02", "2024-07-03"]
|
||||
self.chain_range_calls = 0
|
||||
self.bar_calls = 0
|
||||
|
||||
def fetch_option_chain_range(self, parent, start, end):
|
||||
self.chain_range_calls += 1
|
||||
return self.puts + self.calls
|
||||
|
||||
def fetch_option_bars(self, osis, start, end):
|
||||
self.bar_calls += 1
|
||||
return {osi: {d: 1.0 for d in self.days} for osi in osis}
|
||||
|
||||
|
||||
def test_backfill_month_batched_processes_days_locally(tmp_path):
|
||||
"""_backfill_month must do ONE range-chain + ONE range-ohlcv fetch for the whole month, then process each
|
||||
trading day LOCALLY (parity forward + band + upsert) — the batched win over ~6-8 API calls/day."""
|
||||
from fxhnt.adapters.orchestration.assets import _backfill_month
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'b.db'}")
|
||||
bars.migrate()
|
||||
fake = _RangeFakeDbn()
|
||||
ok, fail = _backfill_month(fake, bars, month_start="2024-07-01", month_end="2024-07-31",
|
||||
at=dt.datetime(2024, 7, 1))
|
||||
assert fake.chain_range_calls == 1 and fake.bar_calls == 1 # ONE bulk fetch each, not per-day
|
||||
assert ok == 3 and fail == 0 # 3 trading days processed locally
|
||||
puts = bars.chain_asof("2024-07-01", 20, 45, right="P")
|
||||
assert puts # band frozen from the shared bulk data
|
||||
got = bars.bars_for([puts[0].osi_symbol], "2024-07-01", "2024-07-03")
|
||||
assert len(next(iter(got.values()))) == 3 # marks for all 3 days landed
|
||||
|
||||
|
||||
def test_freeze_day_fetches_chain_once(tmp_path):
|
||||
"""The nightly asset + backfill call _freeze_day, which must fetch the option chain (definition) exactly
|
||||
ONCE per day and share it between the parity-forward estimate and the band freeze — the old code fetched
|
||||
it twice (in _xsp_forward_estimate AND _freeze_xsp_slice), doubling OPRA definition cost."""
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'o.db'}")
|
||||
bars.migrate()
|
||||
fake = _FakeDbn()
|
||||
fwd = _freeze_day(fake, bars, as_of="2024-07-20", at=dt.datetime(2024, 7, 20))
|
||||
assert fake.chain_calls == 1 # ONE chain fetch/day, not two
|
||||
assert 440.0 <= fwd <= 480.0 # a sane parity forward derived from the shared chain
|
||||
assert bars.chain_asof("2024-07-20", 20, 45, right="P") # the band was frozen off that single chain
|
||||
|
||||
|
||||
class _StepBackDbn:
|
||||
"""Fake provider for the freeze step-back: `last_available_option_date()` returns a partial (still-settling)
|
||||
today; `fetch_option_chain` raises DatabentoRangeUnavailable for any as_of NOT in `good_dates` (models a
|
||||
partial session's `data_schema_not_fully_available`, already normalized by the provider). Records the
|
||||
candidate order attempted so tests can assert weekend-skipping and step order."""
|
||||
def __init__(self, available_date: dt.date, good_dates: set[str]) -> None:
|
||||
self._available = available_date
|
||||
self._good = good_dates
|
||||
self.attempts: list[str] = []
|
||||
|
||||
def last_available_option_date(self) -> dt.date:
|
||||
return self._available
|
||||
|
||||
def fetch_option_chain(self, parent: str, as_of: str):
|
||||
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
|
||||
from fxhnt.domain.models import OptionContract
|
||||
self.attempts.append(as_of)
|
||||
if as_of not in self._good:
|
||||
raise DatabentoRangeUnavailable(f"session {as_of} not fully available")
|
||||
d = dt.date.fromisoformat(as_of)
|
||||
exp = d + dt.timedelta(days=30) # ~30-DTE expiry relative to the candidate
|
||||
puts = [OptionContract(f"XSP{as_of}P{k}", exp, float(k), "P") for k in range(440, 475, 5)]
|
||||
calls = [OptionContract(f"XSP{as_of}C{k}", exp, float(k), "C") for k in range(460, 481, 5)]
|
||||
return puts + calls
|
||||
|
||||
def fetch_option_bars(self, osis, start, end):
|
||||
return {s: {start: 1.0} for s in osis}
|
||||
|
||||
|
||||
def test_freeze_latest_session_steps_back_past_partial_today(tmp_path):
|
||||
"""Robust freeze: today's OPRA session is partial (raises), so the entry point steps back ONE trading day
|
||||
to yesterday's COMPLETE session, freezes it, and RETURNS that date. Deterministic regardless of run time."""
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'o.db'}")
|
||||
bars.migrate()
|
||||
# Wed 2026-07-15 (today, partial) → Tue 2026-07-14 (complete)
|
||||
fake = _StepBackDbn(dt.date(2026, 7, 15), good_dates={"2026-07-14"})
|
||||
frozen = _freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15))
|
||||
assert frozen == "2026-07-14" # landed on the last COMPLETE session
|
||||
assert fake.attempts == ["2026-07-15", "2026-07-14"] # tried today first, then stepped back one day
|
||||
assert bars.chain_asof("2026-07-14", 20, 45, right="P") # the band was frozen for that session
|
||||
|
||||
|
||||
def test_freeze_latest_session_skips_weekend(tmp_path):
|
||||
"""The step-back is by TRADING day: a Monday candidate steps to the prior FRIDAY, never Sat/Sun."""
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'w.db'}")
|
||||
bars.migrate()
|
||||
# Mon 2026-07-13 (partial) → skip Sun 12 / Sat 11 → Fri 2026-07-10 (complete)
|
||||
fake = _StepBackDbn(dt.date(2026, 7, 13), good_dates={"2026-07-10"})
|
||||
frozen = _freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 13))
|
||||
assert frozen == "2026-07-10"
|
||||
assert fake.attempts == ["2026-07-13", "2026-07-10"] # Sat 11 / Sun 12 skipped entirely
|
||||
|
||||
|
||||
def test_freeze_latest_session_raises_when_no_complete_session_in_bound(tmp_path):
|
||||
"""No complete session within the bounded step-back → raise DatabentoRangeUnavailable (LOUD). The C1b
|
||||
wrapper in vrp_nav then recomputes off frozen and the health axis keeps vrp STALE — never a silent no-op."""
|
||||
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
|
||||
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'n.db'}")
|
||||
bars.migrate()
|
||||
fake = _StepBackDbn(dt.date(2026, 7, 15), good_dates=set()) # every candidate fails
|
||||
try:
|
||||
_freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15), max_steps=5)
|
||||
raise AssertionError("expected DatabentoRangeUnavailable when the bound is exhausted")
|
||||
except DatabentoRangeUnavailable:
|
||||
pass
|
||||
assert len(fake.attempts) == 6 # start candidate + 5 bounded step-backs
|
||||
|
||||
|
||||
# --- MINOR-2: step back on a PUBLISHED-but-DEGENERATE session (not only an unavailable one) ---------------
|
||||
|
||||
class _DegenerateDbn:
|
||||
"""A published session whose chain cannot form a parity forward — `fetch_option_chain` RETURNS contracts
|
||||
(no DatabentoRangeUnavailable), but for a DEGENERATE date the only expiry is far outside the [20,45]-DTE
|
||||
forward window, so `_xsp_forward_estimate` raises `RuntimeError('no XSP chain … cannot estimate forward')`.
|
||||
A GOOD date returns a proper ~30-DTE chain with a matched ATM call/put pair. Models a broken published
|
||||
session that must be stepped over, not crashed on."""
|
||||
def __init__(self, available_date: dt.date, good_dates: set[str]) -> None:
|
||||
self._available = available_date
|
||||
self._good = good_dates
|
||||
self.attempts: list[str] = []
|
||||
|
||||
def last_available_option_date(self) -> dt.date:
|
||||
return self._available
|
||||
|
||||
def fetch_option_chain(self, _parent: str, as_of: str):
|
||||
from fxhnt.domain.models import OptionContract
|
||||
self.attempts.append(as_of)
|
||||
d = dt.date.fromisoformat(as_of)
|
||||
if as_of in self._good:
|
||||
exp = d + dt.timedelta(days=30) # in [20,45] -> forward derivable
|
||||
puts = [OptionContract(f"XSP{as_of}P{k}", exp, float(k), "P") for k in range(440, 475, 5)]
|
||||
calls = [OptionContract(f"XSP{as_of}C{k}", exp, float(k), "C") for k in range(460, 481, 5)]
|
||||
return puts + calls
|
||||
exp = d + dt.timedelta(days=100) # OUTSIDE [20,45] -> `near` empty -> RuntimeError
|
||||
return [OptionContract(f"XSP{as_of}P{k}", exp, float(k), "P") for k in range(440, 475, 5)]
|
||||
|
||||
def fetch_option_bars(self, osis, start, _end):
|
||||
return {s: {start: 1.0} for s in osis}
|
||||
|
||||
|
||||
def test_freeze_latest_session_steps_back_past_degenerate_published_session(tmp_path):
|
||||
"""A PUBLISHED but degenerate today (forward not derivable -> RuntimeError, NOT DatabentoRangeUnavailable)
|
||||
must step back to the prior USABLE session, not hard-crash the caller."""
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'deg.db'}")
|
||||
bars.migrate()
|
||||
fake = _DegenerateDbn(dt.date(2026, 7, 15), good_dates={"2026-07-14"}) # Wed degenerate -> Tue usable
|
||||
frozen = _freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15))
|
||||
assert frozen == "2026-07-14"
|
||||
assert fake.attempts == ["2026-07-15", "2026-07-14"] # tried the broken published session, then stepped back
|
||||
assert bars.chain_asof("2026-07-14", 20, 45, right="P") # the usable session was frozen
|
||||
|
||||
|
||||
def test_freeze_latest_session_raises_when_all_degenerate_in_bound(tmp_path):
|
||||
"""Every candidate within the bound is a degenerate published session -> raise DatabentoRangeUnavailable
|
||||
(LOUD), never silently return a bar-less date."""
|
||||
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
|
||||
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'alldeg.db'}")
|
||||
bars.migrate()
|
||||
fake = _DegenerateDbn(dt.date(2026, 7, 15), good_dates=set()) # every session degenerate
|
||||
try:
|
||||
_freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15), max_steps=5)
|
||||
raise AssertionError("expected DatabentoRangeUnavailable when every candidate is degenerate")
|
||||
except DatabentoRangeUnavailable:
|
||||
pass
|
||||
assert len(fake.attempts) == 6 # start candidate + 5 bounded step-backs
|
||||
|
||||
|
||||
class _UnrelatedErrorDbn:
|
||||
"""A session whose chain is fine but whose bar fetch fails with an UNRELATED RuntimeError (e.g. a DB
|
||||
outage). `_freeze_latest_session` must NOT swallow this as a degenerate-session step-back — it must
|
||||
propagate, so a real infra failure is never masked as 'no session available'."""
|
||||
def __init__(self) -> None:
|
||||
self.attempts: list[str] = []
|
||||
|
||||
def last_available_option_date(self) -> dt.date:
|
||||
return dt.date(2026, 7, 15)
|
||||
|
||||
def fetch_option_chain(self, _parent: str, as_of: str):
|
||||
from fxhnt.domain.models import OptionContract
|
||||
self.attempts.append(as_of)
|
||||
exp = dt.date.fromisoformat(as_of) + dt.timedelta(days=30)
|
||||
puts = [OptionContract(f"XSP{as_of}P{k}", exp, float(k), "P") for k in range(440, 475, 5)]
|
||||
calls = [OptionContract(f"XSP{as_of}C{k}", exp, float(k), "C") for k in range(460, 481, 5)]
|
||||
return puts + calls
|
||||
|
||||
def fetch_option_bars(self, _osis, _start, _end):
|
||||
raise RuntimeError("db connection lost") # unrelated to forward estimation
|
||||
|
||||
|
||||
def test_freeze_latest_session_propagates_unrelated_runtime_error(tmp_path):
|
||||
bars = XspOptionBarsRepo(f"sqlite:///{tmp_path/'unrel.db'}")
|
||||
bars.migrate()
|
||||
fake = _UnrelatedErrorDbn()
|
||||
try:
|
||||
_freeze_latest_session(fake, bars, at=dt.datetime(2026, 7, 15))
|
||||
raise AssertionError("expected the unrelated RuntimeError to propagate")
|
||||
except RuntimeError as e:
|
||||
assert "db connection lost" in str(e) # propagated, not swallowed as a step-back
|
||||
assert fake.attempts == ["2026-07-15"] # did NOT step back on an unrelated error
|
||||
|
||||
|
||||
def test_is_degenerate_forward_error_matches_forward_messages_only():
|
||||
# the three forward-estimation messages `_xsp_forward_estimate`/`_forward_from_marks` raise -> step-back.
|
||||
assert _is_degenerate_forward_error(RuntimeError("no XSP chain for 2026-07-15 — cannot estimate forward"))
|
||||
assert _is_degenerate_forward_error(RuntimeError("no matched call/put strike for 2026-07-15 forward"))
|
||||
assert _is_degenerate_forward_error(RuntimeError("no ATM call/put pair with marks for 2026-07-15"))
|
||||
# anything else (e.g. a DB/infra error) is NOT a step-back trigger.
|
||||
assert not _is_degenerate_forward_error(RuntimeError("db connection lost"))
|
||||
assert not _is_degenerate_forward_error(ValueError("no XSP chain")) # wrong type is irrelevant here; msg-only
|
||||
@@ -1,80 +0,0 @@
|
||||
"""Black-Scholes pricer (r=0, crypto convention) — known-value + invariant tests.
|
||||
|
||||
The pricer is used to synthesize the legs of the defined-risk VRP spread from DVOL (we have no historical
|
||||
option chains). These tests pin the standard BS identities so the spread pricing is trustworthy:
|
||||
* the erf-based normal CDF is correct (matches scipy / known values);
|
||||
* an ATM call ≈ 0.4·S·σ·√T (the textbook small-T approximation);
|
||||
* put-call parity at r=0: call − put = S − K;
|
||||
* prices are monotone increasing in IV (vega > 0) and bounded by intrinsic from below;
|
||||
* degenerate (T=0 / σ=0) legs collapse to intrinsic value.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
from fxhnt.application.black_scholes import _norm_cdf, bs_call, bs_put
|
||||
|
||||
|
||||
def test_norm_cdf_known_values() -> None:
|
||||
assert math.isclose(_norm_cdf(0.0), 0.5, abs_tol=1e-12)
|
||||
# N(1.96) ≈ 0.975, N(-1.96) ≈ 0.025 (the textbook two-sided 95% points).
|
||||
assert math.isclose(_norm_cdf(1.96), 0.9750021, abs_tol=1e-6)
|
||||
assert math.isclose(_norm_cdf(-1.96), 0.0249979, abs_tol=1e-6)
|
||||
# symmetry: N(x) + N(-x) = 1.
|
||||
assert math.isclose(_norm_cdf(0.73) + _norm_cdf(-0.73), 1.0, abs_tol=1e-12)
|
||||
|
||||
|
||||
def test_norm_cdf_matches_scipy_if_available() -> None:
|
||||
try:
|
||||
from scipy.stats import norm
|
||||
except ImportError:
|
||||
return
|
||||
for x in (-2.5, -1.0, -0.1, 0.0, 0.3, 1.5, 3.0):
|
||||
assert math.isclose(_norm_cdf(x), float(norm.cdf(x)), abs_tol=1e-12)
|
||||
|
||||
|
||||
def test_atm_call_approx_zero_point_four_s_sigma_sqrt_t() -> None:
|
||||
# The standard small-T ATM approximation: C_atm ≈ 0.3989·S·σ·√T ≈ 0.4·S·σ·√T.
|
||||
s, sigma, t = 30_000.0, 0.60, 30.0 / 365.0
|
||||
approx = 0.3989 * s * sigma * math.sqrt(t)
|
||||
price = bs_call(s=s, k=s, t=t, sigma=sigma)
|
||||
assert math.isclose(price, approx, rel_tol=0.02) # within 2% of the textbook approximation
|
||||
|
||||
|
||||
def test_put_call_parity_r_zero() -> None:
|
||||
# At r=0 (and no dividends): call − put = S − K, for any K.
|
||||
s, t, sigma = 2_000.0, 45.0 / 365.0, 0.75
|
||||
for k in (1_800.0, 2_000.0, 2_300.0):
|
||||
c = bs_call(s=s, k=k, t=t, sigma=sigma)
|
||||
p = bs_put(s=s, k=k, t=t, sigma=sigma)
|
||||
assert math.isclose(c - p, s - k, abs_tol=1e-6)
|
||||
|
||||
|
||||
def test_prices_monotone_increasing_in_iv() -> None:
|
||||
s, k, t = 30_000.0, 30_000.0, 30.0 / 365.0
|
||||
lo = bs_call(s=s, k=k, t=t, sigma=0.30)
|
||||
hi = bs_call(s=s, k=k, t=t, sigma=0.90)
|
||||
assert hi > lo > 0.0
|
||||
plo = bs_put(s=s, k=k, t=t, sigma=0.30)
|
||||
phi = bs_put(s=s, k=k, t=t, sigma=0.90)
|
||||
assert phi > plo > 0.0
|
||||
|
||||
|
||||
def test_price_bounded_below_by_intrinsic() -> None:
|
||||
s, t, sigma = 30_000.0, 30.0 / 365.0, 0.60
|
||||
# an ITM call (K below S) is worth at least its intrinsic value S − K.
|
||||
c = bs_call(s=s, k=27_000.0, t=t, sigma=sigma)
|
||||
assert c >= (s - 27_000.0)
|
||||
# an ITM put (K above S) is worth at least K − S.
|
||||
p = bs_put(s=s, k=33_000.0, t=t, sigma=sigma)
|
||||
assert p >= (33_000.0 - s)
|
||||
|
||||
|
||||
def test_degenerate_collapses_to_intrinsic() -> None:
|
||||
# T=0: call = max(S−K,0), put = max(K−S,0).
|
||||
assert bs_call(s=100.0, k=90.0, t=0.0, sigma=0.5) == 10.0
|
||||
assert bs_call(s=100.0, k=110.0, t=0.0, sigma=0.5) == 0.0
|
||||
assert bs_put(s=100.0, k=110.0, t=0.0, sigma=0.5) == 10.0
|
||||
# σ=0: same intrinsic limit (a zero-vol option is just its forward intrinsic).
|
||||
assert bs_call(s=100.0, k=90.0, t=0.5, sigma=0.0) == 10.0
|
||||
assert bs_put(s=100.0, k=110.0, t=0.5, sigma=0.0) == 10.0
|
||||
@@ -10,9 +10,9 @@ from fxhnt.registry import STRATEGY_REGISTRY
|
||||
|
||||
def test_every_registry_strategy_has_a_display_name() -> None:
|
||||
for sid, entry in STRATEGY_REGISTRY.items():
|
||||
# multistrat/vrp are deliberately overridden with a friendlier pill label (see the next test) — every
|
||||
# multistrat is deliberately overridden with a friendlier pill label (see the next test) — every
|
||||
# other registry entry must resolve to its own registry `display_name`.
|
||||
if sid not in ("multistrat", "vrp"):
|
||||
if sid != "multistrat":
|
||||
assert display_name(sid) == entry["display_name"]
|
||||
assert display_name(sid) != sid # never the bare raw id
|
||||
|
||||
@@ -20,7 +20,6 @@ def test_every_registry_strategy_has_a_display_name() -> None:
|
||||
def test_sim_book_overrides_are_plain_language() -> None:
|
||||
# The interim local pill labels from app.py's `_SIM_BOOK_LABELS` (Task 1), absorbed here.
|
||||
assert display_name("multistrat") == "ETF Portfolio"
|
||||
assert display_name("vrp") == "Options income (put spreads)"
|
||||
|
||||
|
||||
def test_unknown_id_falls_back_to_itself_never_raises() -> None:
|
||||
|
||||
@@ -26,12 +26,12 @@ def test_every_active_track_declares_a_definition():
|
||||
|
||||
def test_observed_mode_is_exactly_the_executed_twins():
|
||||
# Pin is deliberately exhaustive: whenever a new track adopts record_mode="observed", update this set
|
||||
# (C2 Task 3 added bybit_4edge_exec; Task 7 added vrp_exec) so the change is reviewed, not silently
|
||||
# absorbed. `xsfunding` moved OFF observed (Phase 0b venue consolidation: standalone Binance
|
||||
# `XsFundingForward` -> its own Bybit single-sleeve `recompute` track, mirroring `positioning`) — the
|
||||
# remaining observed tracks are the OBSERVED-fills executed twins (fills are observed, not recomputable).
|
||||
# (C2 Task 3 added bybit_4edge_exec) so the change is reviewed, not silently absorbed. `xsfunding` moved
|
||||
# OFF observed (Phase 0b venue consolidation: standalone Binance `XsFundingForward` -> its own Bybit
|
||||
# single-sleeve `recompute` track, mirroring `positioning`) — the remaining observed tracks are the
|
||||
# OBSERVED-fills executed twins (fills are observed, not recomputable).
|
||||
observed = {sid for sid in STRATEGY_REGISTRY if track_definition(sid)[2] == "observed"}
|
||||
assert observed == {"multistrat_exec", "bybit_4edge_exec", "vrp_exec"}
|
||||
assert observed == {"multistrat_exec", "bybit_4edge_exec"}
|
||||
|
||||
|
||||
def test_inception_override_returns_date_when_version_matches(monkeypatch):
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
|
||||
|
||||
def test_vrp_registry_entry_shape():
|
||||
e = STRATEGY_REGISTRY["vrp"]
|
||||
assert e["venue"] == "ibkr-xsp"
|
||||
assert e["sleeve"] == "equity-vol"
|
||||
assert e["tier"] == "research"
|
||||
assert e["execution"] == "paper"
|
||||
assert e["gate_spec"] == {"gate_type": "reconciliation", "min_forward_days": 21}
|
||||
assert e["definition"]["record_mode"] == "recompute"
|
||||
assert e["promote_on_pass"] is False
|
||||
|
||||
|
||||
def test_vrp_exec_twin_is_observed():
|
||||
from fxhnt.registry import STRATEGY_REGISTRY
|
||||
e = STRATEGY_REGISTRY["vrp_exec"]
|
||||
assert e["venue"] == "ibkr-xsp"
|
||||
assert e["definition"]["record_mode"] == "observed"
|
||||
assert e["gate_spec"]["gate_type"] == "reconciliation"
|
||||
@@ -15,11 +15,6 @@ def test_ibkr_recompute_replay_book_reads_ibkr_curve():
|
||||
assert out == {"multistrat": {0: 0.0, 1: 0.02}}
|
||||
|
||||
|
||||
def test_vrp_recompute_replay_book_reads_ibkr_curve():
|
||||
out = sim_returns_for("vrp", bybit_returns=_bybit, ibkr_curve=_ibkr)
|
||||
assert out == {"vrp": {0: 0.0, 1: 0.02}}
|
||||
|
||||
|
||||
def test_sim_books_includes_ibkr():
|
||||
assert "multistrat" in SIM_BOOKS and "bybit_4edge" in SIM_BOOKS
|
||||
assert SIM_BOOKS == ("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered", "vrp")
|
||||
assert SIM_BOOKS == ("bybit_4edge", "bybit_4edge_levered", "multistrat", "multistrat_levered")
|
||||
|
||||
@@ -1,320 +0,0 @@
|
||||
import datetime as dt
|
||||
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
from fxhnt.application import vrp_book, vrp_marking, vrp_pricing
|
||||
from fxhnt.application.allocation_engine import cvar_daily
|
||||
from fxhnt.application.vrp_book import VrpStrategy
|
||||
|
||||
|
||||
def _seed(tmp_path, days, contracts):
|
||||
"""contracts: list of (osi, expiry, strike, right, {iso_date: close}). Returns a repo."""
|
||||
r = XspOptionBarsRepo(f"sqlite:///{tmp_path/'v.db'}")
|
||||
r.migrate()
|
||||
at = dt.datetime(2024, 1, 1)
|
||||
rows = []
|
||||
for osi, expiry, strike, right, closes in contracts:
|
||||
for d, c in closes.items():
|
||||
rows.append(dict(osi_symbol=osi, date=d, expiry=expiry, strike=strike, right=right, close=c))
|
||||
r.upsert_bars(rows, at)
|
||||
return r
|
||||
|
||||
|
||||
def _put_mark(k: float, day_index: int) -> float:
|
||||
"""Realistic put mark: OTM puts (low strike) are cheaper than near-ATM puts (high strike, richer
|
||||
extrinsic value), rising monotonically with strike toward the forward (470) -- e.g. strike 440 ~= 0.15,
|
||||
465 ~= 0.90, 470 ~= 1.05. Decays ~3%/day (theta): since it's a PERCENTAGE decay, the higher-dollar
|
||||
near-ATM leg loses more in absolute terms per day than the cheaper far-OTM leg, which is exactly the
|
||||
source of a short put-credit-spread's daily theta P&L (spread value = short - long shrinks over time)."""
|
||||
base = 0.15 + 0.03 * (k - 440.0)
|
||||
return base * (0.97 ** day_index)
|
||||
|
||||
|
||||
def _straight_chain(start="2024-07-01", n=10, expiry="2024-07-31", forward=470.0):
|
||||
"""A flat-vol synthetic put chain across strikes (OTM puts cheaper, correctly rising toward the
|
||||
forward), decaying ~3%/day, plus an ATM call tracking the ATM put 1:1 so parity_forward(K=470) recovers
|
||||
~470 every day. A short-higher-strike/long-lower-strike credit spread's credit is always positive here
|
||||
since put price increases monotonically with strike (short leg is worth more than the long leg)."""
|
||||
days = [(dt.date.fromisoformat(start) + dt.timedelta(days=i)).isoformat() for i in range(n)]
|
||||
contracts = []
|
||||
for k in range(440, 475, 5): # put strikes below/at forward
|
||||
osi = f"XSP{expiry.replace('-','')[2:]}P{k*1000:08d}"
|
||||
closes = {d: _put_mark(float(k), i) for i, d in enumerate(days)}
|
||||
contracts.append((osi, expiry, float(k), "P", closes))
|
||||
# ATM call tracks the ATM (470) put mark exactly so parity_forward(K=470) recovers ~470 every day.
|
||||
contracts.append((f"XSP{expiry.replace('-','')[2:]}C{470*1000:08d}", expiry, 470.0, "C",
|
||||
{d: _put_mark(470.0, i) for i, d in enumerate(days)}))
|
||||
return days, contracts
|
||||
|
||||
|
||||
def test_advance_full_history_is_deterministic(tmp_path):
|
||||
days, contracts = _straight_chain()
|
||||
r = _seed(tmp_path, days, contracts)
|
||||
# a spread must actually open on the entry day -- proves the fixture's marks are realistic (positive
|
||||
# credit), not vacuously producing an empty series (see module docstring history: inverted put pricing
|
||||
# used to make every credit <= 0 and _select_spread always return None).
|
||||
opened = VrpStrategy(r, entry_weekday=0)._select_spread(days[0])
|
||||
assert opened is not None and opened["credit"] > 0.0
|
||||
a, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
|
||||
b, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
|
||||
assert a == b # same frozen input -> identical series
|
||||
assert all(isinstance(d, str) and isinstance(x, float) for d, x in a)
|
||||
assert any(abs(x) > 1e-12 for _, x in a) # a real, non-empty return stream -- not an empty ladder
|
||||
|
||||
|
||||
def test_defined_risk_daily_loss_is_bounded(tmp_path):
|
||||
# a crash day: the short put mark spikes; the spread loss per day cannot exceed width/margin = 1.0
|
||||
days, contracts = _straight_chain(n=6)
|
||||
# overwrite the last day's marks with a crash spike on all puts
|
||||
crash = days[-1]
|
||||
for _osi, _expiry, _strike, right, closes in contracts:
|
||||
if right == "P":
|
||||
closes[crash] = closes[crash] + 50.0
|
||||
r = _seed(tmp_path, days, contracts)
|
||||
series, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
|
||||
assert series, "series should not be empty"
|
||||
# a spread must be open and marked before the crash day, so the bound below is tested against a REAL
|
||||
# marked position (theta-decay P&L), not an empty ladder that trivially satisfies min() >= -1.0.
|
||||
assert any(abs(x) > 1e-12 for d, x in series if d != crash)
|
||||
assert min(x for _, x in series) >= -1.0 - 1e-9 # defined-risk: daily loss per margin bounded
|
||||
assert max(abs(x) for _, x in series) < 5.0 # sane magnitude
|
||||
|
||||
|
||||
def test_atm_pair_selected_by_min_abs_c_minus_p(tmp_path, monkeypatch):
|
||||
"""Multiple call strikes exist on the same grid as the puts. The true ATM by min|C-P| is 470 (put and
|
||||
call both mark 1.0 -> diff=0); 460 has a much larger |C-P| (20.0) that would imply a forward ~10 points
|
||||
off if it were wrongly picked instead. Spy on `parity_forward` to assert the pairing selects k_atm=470
|
||||
-- proving selection by min|C-P| over sorted common strikes, not the old minimal-strike-distance
|
||||
cross-product (which ties at diff==0 for every same-strike pair and silently depended on whatever
|
||||
order `chain_asof` happened to return rows in)."""
|
||||
expiry, day = "2024-08-16", "2024-07-15"
|
||||
contracts = [
|
||||
("P440", expiry, 440.0, "P", {day: 3.0}),
|
||||
("P450", expiry, 450.0, "P", {day: 2.0}),
|
||||
("P460", expiry, 460.0, "P", {day: 2.0}),
|
||||
("P470", expiry, 470.0, "P", {day: 1.0}),
|
||||
("C460", expiry, 460.0, "C", {day: 22.0}), # |C-P| = 20 at 460 -- would give forward 480 if picked
|
||||
("C470", expiry, 470.0, "C", {day: 1.0}), # |C-P| = 0 at 470 -- the true ATM
|
||||
]
|
||||
r = _seed(tmp_path, [day], contracts)
|
||||
strat = VrpStrategy(r)
|
||||
seen: list[dict] = []
|
||||
real_parity_forward = vrp_pricing.parity_forward
|
||||
|
||||
def spy(**kwargs):
|
||||
seen.append(kwargs)
|
||||
return real_parity_forward(**kwargs)
|
||||
|
||||
monkeypatch.setattr(vrp_book.vrp_pricing, "parity_forward", spy)
|
||||
strat._select_spread(day)
|
||||
assert seen, "parity_forward should have been called"
|
||||
assert seen[0]["k_atm"] == 470.0
|
||||
assert seen[0]["call_atm"] == 1.0 and seen[0]["put_atm"] == 1.0
|
||||
|
||||
|
||||
def test_advance_deterministic_with_multiple_common_call_strikes(tmp_path):
|
||||
"""Guard against the old minimal-strike-distance ATM pairing in the full advance() loop: add a second
|
||||
call strike (460) that shares a strike with a put but whose mark is far off (|C-P|=20 vs 0 at the true
|
||||
ATM, 470) -- if it were wrongly picked as ATM, the forward and downstream short-strike selection would
|
||||
differ. Two fresh `advance(None, {})` runs must stay byte-identical."""
|
||||
days, contracts = _straight_chain(n=10)
|
||||
# a second call strike (460) whose |C-P| is always 20 (way off the true ATM's ~0) -- tracks the actual
|
||||
# decaying 460 put mark each day via `_put_mark` so the offset holds on every day, not just day 0.
|
||||
contracts.append(("XSP240731C00460000", "2024-07-31", 460.0, "C",
|
||||
{d: _put_mark(460.0, i) + 20.0 for i, d in enumerate(days)}))
|
||||
r = _seed(tmp_path, days, contracts)
|
||||
a, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
|
||||
b, _ = VrpStrategy(r, entry_weekday=0).advance(None, {})
|
||||
assert a == b
|
||||
assert all(isinstance(d, str) and isinstance(x, float) for d, x in a)
|
||||
assert any(abs(x) > 1e-12 for _, x in a) # canary: a spread actually opened (not a vacuous empty series)
|
||||
|
||||
|
||||
def test_advance_closes_spread_on_missing_marks_not_zombie(tmp_path):
|
||||
"""A held spread whose legs' marks disappear (e.g. aged/moved out of the frozen slice) must be CLOSED,
|
||||
not zombie-held forever via the `val is None -> still.append(sp)` branch -- otherwise it permanently
|
||||
occupies a ladder slot and starves new entries. Truncate all put marks after day 3 so the spread
|
||||
entered on day 0 (a Monday) has no mark from day 4 onward; assert it is dropped from `open_spreads`
|
||||
(not carried forward) and the series stays finite."""
|
||||
days, contracts = _straight_chain(n=10)
|
||||
stale_after = days[3] # put marks stop appearing after this day
|
||||
new_contracts = []
|
||||
for osi, expiry, strike, right, closes in contracts:
|
||||
if right == "P":
|
||||
closes = {d: c for d, c in closes.items() if d <= stale_after}
|
||||
new_contracts.append((osi, expiry, strike, right, closes))
|
||||
r = _seed(tmp_path, days, new_contracts)
|
||||
series, extra = VrpStrategy(r, entry_weekday=0).advance(None, {})
|
||||
assert series, "series should not be empty"
|
||||
# a spread must actually have OPENED and been marked while the puts were still fresh (days 1-3), so
|
||||
# dropping it below proves a real held position was closed -- not that an empty ladder stayed empty.
|
||||
assert any(abs(x) > 1e-12 for d, x in series if d <= stale_after)
|
||||
assert all(isinstance(x, float) and x == x and abs(x) != float("inf") for _, x in series)
|
||||
assert extra["open_spreads"] == [] # closed on missing marks, not zombied
|
||||
|
||||
|
||||
def _bs_chain(days, expiry, forward, sigma, strikes):
|
||||
"""A Black-Scholes-CONSISTENT put+call chain: put mark = bs_put(F, K, sigma, tau), call mark via
|
||||
put-call parity C = P + (F - K). Parity recovers F exactly and `implied_vol_put` recovers `sigma` at
|
||||
every strike -> a clean, smooth vol surface where the MODEL marks reproduce the marks themselves (so a
|
||||
spread's entry credit equals its model value at entry: no synthetic model-vs-raw basis jump). `forward`
|
||||
may be a float (constant) or a dict {iso_day: F} for a moving-spot path."""
|
||||
exp_d = dt.date.fromisoformat(expiry)
|
||||
fwd_of = (lambda _d: forward) if not isinstance(forward, dict) else (lambda d: forward[d])
|
||||
contracts = []
|
||||
for k in strikes:
|
||||
pcloses, ccloses = {}, {}
|
||||
for d in days:
|
||||
tau = max((exp_d - dt.date.fromisoformat(d)).days, 1) / 365.0
|
||||
f = fwd_of(d)
|
||||
p = vrp_pricing.bs_put(f, float(k), sigma, tau)
|
||||
c = p + (f - float(k)) # parity: C - P = F - K
|
||||
if p > 0.0:
|
||||
pcloses[d] = p
|
||||
if c > 0.0:
|
||||
ccloses[d] = c
|
||||
tag = expiry.replace("-", "")[2:]
|
||||
contracts.append((f"XSP{tag}P{int(k * 1000):08d}", expiry, float(k), "P", pcloses))
|
||||
contracts.append((f"XSP{tag}C{int(k * 1000):08d}", expiry, float(k), "C", ccloses))
|
||||
return contracts
|
||||
|
||||
|
||||
def _weekdays(start, n):
|
||||
return [(dt.date.fromisoformat(start) + dt.timedelta(days=i)).isoformat() for i in range(n)]
|
||||
|
||||
|
||||
def test_advance_dollar_basis_is_dollars_over_margin(tmp_path):
|
||||
"""Regression for the FIX-2 100x-too-small bug, adapted to MODEL marking: `pnl`'s numerator must be in
|
||||
DOLLARS (per-share model value x100 = the contract multiplier) to match `self.margin`'s dollar basis
|
||||
(width_points*100). The booked daily return must equal the MODEL value change / width -- NOT that change
|
||||
/ (width*100) (the old points/dollars basis, 100x smaller).
|
||||
|
||||
One spread (ladder=1) opens on day0 of a BS-consistent chain; on day1 the smooth model value has moved by
|
||||
theta. Assert the booked return equals (credit - model_val_day1)/width and is ~100x the pre-fix basis."""
|
||||
days = _weekdays("2024-07-01", 2) # Mon, Tue
|
||||
contracts = _bs_chain(days, "2024-07-31", forward=470.0, sigma=0.20, strikes=range(380, 481, 5))
|
||||
r = _seed(tmp_path, days, contracts)
|
||||
strat = VrpStrategy(r, entry_weekday=0, ladder=1)
|
||||
sp = strat._select_spread(days[0])
|
||||
assert sp is not None and sp["credit"] > 0.0
|
||||
# the model value on day1 (independent instance; _spread_value mutates last_iv, so pass a copy)
|
||||
val1 = VrpStrategy(r, entry_weekday=0, ladder=1)._spread_value(dict(sp), days[1])
|
||||
assert val1 is not None
|
||||
series, _ = strat.advance(None, {})
|
||||
by_day = dict(series)
|
||||
assert days[1] in by_day
|
||||
width = strat._width # default 5.0
|
||||
expected = (sp["credit"] - val1) / width # (Δpts * 100) / (width * 100) = Δpts / width
|
||||
old_buggy = expected / 100.0 # pre-fix basis: Δpts / (width * 100)
|
||||
assert abs(by_day[days[1]] - expected) < 1e-9
|
||||
assert abs(by_day[days[1]] - old_buggy) > 50 * abs(old_buggy) # ~100x larger than the old bug
|
||||
|
||||
|
||||
def test_model_value_bounded_and_smooth_despite_noisy_single_leg(tmp_path):
|
||||
"""(a) The MODEL spread value is CLIPPED to [0, width] and SMOOTH: it is rebuilt from the day's ATM
|
||||
forward + ATM IV via Black-Scholes, so corrupting ONE leg's raw OPRA mark by a huge spike (the exact
|
||||
mark-noise that made the OLD short_close-long_close difference swing 50-100%) does NOT move it -- the ATM
|
||||
parity/IV are read off the clean at-the-money strike, not the corrupted leg."""
|
||||
days = _weekdays("2024-07-01", 5)
|
||||
strikes = list(range(380, 481, 5))
|
||||
contracts = _bs_chain(days, "2024-08-05", forward=470.0, sigma=0.20, strikes=strikes) # ~35 DTE
|
||||
strat = VrpStrategy(_seed(tmp_path, days, contracts), entry_weekday=0)
|
||||
sp = strat._select_spread(days[0])
|
||||
assert sp is not None
|
||||
clean_vals = [strat._spread_value(dict(sp), d) for d in days]
|
||||
for v in clean_vals:
|
||||
assert v is not None and 0.0 <= v <= strat._width + 1e-12 # hard defined-risk clip
|
||||
|
||||
# corrupt ONE non-ATM leg's raw put mark by a +50 spike on day2 (would blow up a raw short-long diff).
|
||||
long_osi = sp["long_osi"]
|
||||
l_exp = dt.date.fromisoformat(sp["expiry"])
|
||||
corrupt = [dict(osi_symbol=long_osi, date=days[2], expiry=l_exp.isoformat(),
|
||||
strike=sp["long_strike"], right="P", close=999.0)]
|
||||
r2 = _seed(tmp_path, days, contracts)
|
||||
r2.upsert_bars(corrupt, dt.datetime(2024, 1, 1))
|
||||
strat2 = VrpStrategy(r2, entry_weekday=0)
|
||||
noisy_vals = [strat2._spread_value(dict(sp), d) for d in days]
|
||||
for v in noisy_vals:
|
||||
assert v is not None and 0.0 <= v <= strat2._width + 1e-12
|
||||
# the model value is UNCHANGED by the corrupted leg (parity/IV come from the clean ATM strike).
|
||||
for cv, nv in zip(clean_vals, noisy_vals, strict=True):
|
||||
assert abs(cv - nv) < 1e-9
|
||||
# and the clean path is smooth: no day-over-day jump anywhere near the width.
|
||||
for a, b in zip(clean_vals, clean_vals[1:], strict=False):
|
||||
assert abs(a - b) < 0.25 * strat._width
|
||||
|
||||
|
||||
def test_settlement_value_is_clipped_intrinsic(tmp_path):
|
||||
"""(d) Terminal settlement = clip(K_short - F, 0, width): OTM -> 0, partially ITM -> the in-the-moneyness,
|
||||
deep ITM -> the width cap. Derived from that day's parity forward (spot), never unbounded."""
|
||||
day, expiry = "2024-07-30", "2024-07-31" # DTE = 1
|
||||
short_strike, long_strike, width = 460.0, 455.0, 5.0
|
||||
for forward, expected in [(470.0, 0.0), (458.0, 2.0), (450.0, 5.0)]:
|
||||
contracts = _bs_chain([day], expiry, forward=forward, sigma=0.20, strikes=range(400, 501, 5))
|
||||
strat = VrpStrategy(_seed(tmp_path, [day], contracts), width_points=width)
|
||||
sp = dict(short_strike=short_strike, long_strike=long_strike, expiry=expiry)
|
||||
settle = strat._settlement_value(sp, day)
|
||||
assert settle is not None and abs(settle - expected) < 1e-6
|
||||
|
||||
|
||||
def test_per_spread_cumulative_pnl_bounded_to_defined_risk_max_loss(tmp_path):
|
||||
"""(b) Per-spread cumulative P&L is bounded to the defined-risk envelope [-(width-credit)*100,
|
||||
+credit*100] dollars. A spot that crashes deep below both strikes by expiry drives the spread to its MAX
|
||||
LOSS: with ladder=1 the summed daily returns * margin == the spread's total dollar P&L, which must sit at
|
||||
(not below) the lower bound -(width-credit)*100."""
|
||||
days = _weekdays("2024-07-01", 31) # Mon entry, ~30 DTE, held to expiry
|
||||
expiry = "2024-07-31"
|
||||
# forward glides 470 -> 400 (deep below any 20Δ short strike & its 5-wide long) by expiry.
|
||||
fwd = {d: 470.0 - 70.0 * (i / (len(days) - 1)) for i, d in enumerate(days)}
|
||||
contracts = _bs_chain(days, expiry, forward=fwd, sigma=0.20, strikes=range(360, 481, 5))
|
||||
strat = VrpStrategy(_seed(tmp_path, days, contracts), entry_weekday=0, ladder=1)
|
||||
sp0 = strat._select_spread(days[0])
|
||||
assert sp0 is not None
|
||||
credit, width = sp0["credit"], strat._width
|
||||
series, _ = strat.advance(None, {})
|
||||
total_dollars = sum(x for _, x in series) * strat.margin # ladder=1 -> return = pnl/margin
|
||||
lo, hi = -(width - credit) * 100.0, credit * 100.0
|
||||
assert lo - 1e-6 <= total_dollars <= hi + 1e-6 # inside the defined-risk envelope
|
||||
assert total_dollars < 0.0 # a crash to deep-ITM is a real loss
|
||||
assert abs(total_dollars - lo) < 1.0 # and it is AT the max-loss bound
|
||||
|
||||
|
||||
def test_no_daily_return_exceeds_plausible_bound_on_clean_vol_path(tmp_path):
|
||||
"""(c) On a synthetic CLEAN constant-vol path the whole ladder's daily returns are tiny (theta decay),
|
||||
never the >15%/day mark-noise artifacts (single days of -50%, +59.6%) the raw-diff marks produced."""
|
||||
days = _weekdays("2024-06-03", 30) # several Monday entries in the [20,45] DTE window
|
||||
contracts = _bs_chain(days, "2024-07-15", forward=470.0, sigma=0.20, strikes=range(360, 481, 5))
|
||||
strat = VrpStrategy(_seed(tmp_path, days, contracts), entry_weekday=0)
|
||||
series, _ = strat.advance(None, {})
|
||||
assert series and any(abs(x) > 1e-12 for _, x in series) # a real, non-empty return stream
|
||||
assert max(abs(x) for _, x in series) < 0.10 # no |daily return| > 10%
|
||||
|
||||
|
||||
def test_backtest_and_shared_marking_are_single_source(tmp_path):
|
||||
"""The backtest (`VrpStrategy._spread_value`) and the live exec twin (`vrp_exec_record`) both value the
|
||||
spread through the SAME `vrp_marking.model_spread_value` — no drift between the ref and the live
|
||||
profit-take signal. Assert the strategy's mark equals the shared helper's on the same frozen slice."""
|
||||
days = _weekdays("2024-07-01", 4)
|
||||
contracts = _bs_chain(days, "2024-08-05", forward=470.0, sigma=0.20, strikes=range(380, 481, 5))
|
||||
repo = _seed(tmp_path, days, contracts)
|
||||
strat = VrpStrategy(repo, entry_weekday=0)
|
||||
sp = strat._select_spread(days[0])
|
||||
assert sp is not None
|
||||
for d in days:
|
||||
via_strategy = strat._spread_value(dict(sp), d)
|
||||
via_shared = vrp_marking.model_spread_value(repo, dict(sp), d, strat._width)
|
||||
assert via_strategy is not None and via_shared is not None
|
||||
assert abs(via_strategy - via_shared) < 1e-12
|
||||
|
||||
|
||||
def test_fat_left_tail_triggers_cvar_floor():
|
||||
# A VRP-shaped series: small positive credit-decay most days (net drift ~0), plus occasional deep
|
||||
# defined-risk losses. cvar_daily averages the worst ceil(0.10*n) returns, so the tail must actually
|
||||
# be filled with losses: n=50 -> 5 tail slots -> 5 genuine losses. Such a fat-LEFT series has
|
||||
# cvar_daily/std ABOVE the Gaussian ES-10% multiplier (1.755), which is exactly what makes the merged
|
||||
# CVaR floor de-lever it (K_cvar < K_vol) where a Gaussian book would be a no-op.
|
||||
series = [0.06] * 45 + [-0.5] * 5
|
||||
returns = {str(i): x for i, x in enumerate(series)}
|
||||
mean = sum(series) / len(series)
|
||||
std = (sum((x - mean) ** 2 for x in series) / len(series)) ** 0.5
|
||||
ratio = cvar_daily(returns, 0.10) / std
|
||||
assert ratio > 1.755 # fatter than Gaussian ES-10% -> CVaR floor bites
|
||||
@@ -1,80 +0,0 @@
|
||||
import pytest
|
||||
|
||||
from fxhnt.application.vrp_exec_record import (
|
||||
build_pos_state,
|
||||
compute_ladder_exec_return,
|
||||
compute_spread_exec_return,
|
||||
)
|
||||
|
||||
|
||||
def test_compute_spread_exec_return_envelope_zero_is_none():
|
||||
assert compute_spread_exec_return(prior_val=1.0, mark_today=0.8, qty=2, envelope=0.0) is None
|
||||
|
||||
|
||||
def test_compute_spread_exec_return_no_mark_today_is_hold_not_phantom_zero():
|
||||
assert compute_spread_exec_return(prior_val=1.0, mark_today=None, qty=2, envelope=200_000.0) is None
|
||||
|
||||
|
||||
def test_compute_spread_exec_return_winning_spread_is_positive_credit_decay():
|
||||
# short-vol: credit decayed from 1.00 -> 0.60 = profit. qty=2, $100/contract multiplier.
|
||||
r = compute_spread_exec_return(prior_val=1.00, mark_today=0.60, qty=2, envelope=200_000.0)
|
||||
assert r == pytest.approx((1.00 - 0.60) * 100.0 * 2 / 200_000.0)
|
||||
assert r > 0.0
|
||||
|
||||
|
||||
def test_compute_spread_exec_return_losing_spread_is_negative_and_bounded_by_width():
|
||||
# a defined-risk put-credit-spread's worst mark is bounded by the wing width ($5 here = $500/contract);
|
||||
# the value rising toward that cap is a loss, correctly signed negative.
|
||||
r = compute_spread_exec_return(prior_val=1.00, mark_today=4.50, qty=2, envelope=200_000.0)
|
||||
assert r == pytest.approx((1.00 - 4.50) * 100.0 * 2 / 200_000.0)
|
||||
assert r < 0.0
|
||||
|
||||
|
||||
def test_compute_ladder_exec_return_inception_no_prior_rungs_is_none():
|
||||
assert compute_ladder_exec_return([], {}, 200_000.0) is None
|
||||
|
||||
|
||||
def test_compute_ladder_exec_return_definanced_envelope_is_none():
|
||||
prior = [{"short_osi": "S1", "long_osi": "L1", "credit": 1.0, "last_val": 1.0, "qty": 2}]
|
||||
assert compute_ladder_exec_return(prior, {"S1": 0.5}, 0.0) is None
|
||||
|
||||
|
||||
def test_compute_ladder_exec_return_sums_only_marked_rungs():
|
||||
prior = [
|
||||
{"short_osi": "S1", "long_osi": "L1", "credit": 1.0, "last_val": 1.0, "qty": 2},
|
||||
{"short_osi": "S2", "long_osi": "L2", "credit": 1.0, "last_val": 1.0, "qty": 2},
|
||||
]
|
||||
marks = {"S1": 0.60, "S2": None} # S2 has no mark today -> HOLD, contributes 0 (not a phantom zero)
|
||||
r = compute_ladder_exec_return(prior, marks, 200_000.0)
|
||||
assert r == pytest.approx((1.00 - 0.60) * 100.0 * 2 / 200_000.0) # only S1's marked pnl
|
||||
|
||||
|
||||
def test_compute_ladder_exec_return_all_holds_is_none():
|
||||
prior = [{"short_osi": "S1", "long_osi": "L1", "credit": 1.0, "last_val": 1.0, "qty": 2}]
|
||||
assert compute_ladder_exec_return(prior, {"S1": None}, 200_000.0) is None
|
||||
|
||||
|
||||
def test_build_pos_state_round_trip():
|
||||
opens = [{"entry": "2026-07-06", "expiry": "2026-08-15", "short_osi": "S1", "long_osi": "L1",
|
||||
"credit": 1.0, "last_val": 0.6, "qty": 2}]
|
||||
st = build_pos_state(opens, envelope=200_000.0, venue="ibkr-paper", today="2026-07-13")
|
||||
assert st["open_spreads"] == opens
|
||||
assert st["envelope"] == 200_000.0
|
||||
assert st["venue"] == "ibkr-paper"
|
||||
assert st["last_run_date"] == "2026-07-13"
|
||||
|
||||
|
||||
def test_build_pos_state_defensively_copies_rungs():
|
||||
opens = [{"entry": "2026-07-06", "expiry": "2026-08-15", "short_osi": "S1", "long_osi": "L1",
|
||||
"credit": 1.0, "last_val": 0.6, "qty": 2}]
|
||||
st = build_pos_state(opens, envelope=200_000.0, venue="ibkr-paper", today="2026-07-13")
|
||||
opens.append({"short_osi": "S2"}) # mutate the caller's list after the call
|
||||
opens[0]["last_val"] = 999.0 # mutate a rung dict in the caller's list
|
||||
assert len(st["open_spreads"]) == 1
|
||||
assert st["open_spreads"][0]["last_val"] == 0.6
|
||||
|
||||
|
||||
def test_build_pos_state_empty_ladder():
|
||||
st = build_pos_state([], envelope=0.0, venue="", today="2026-07-13")
|
||||
assert st["open_spreads"] == []
|
||||
assert st["envelope"] == 0.0
|
||||
@@ -1,245 +0,0 @@
|
||||
"""Composition test for the ladder-cap-breach-under-close-failure defect: `plan_and_record_vrp` used to
|
||||
decide whether to open a new rung from `plan_vrp_ladder`'s PRE-EXECUTION plan (`may_open`), computed before
|
||||
any order was placed. If a planned CLOSE then failed at the broker (e.g. an illiquid near-expiry contract
|
||||
rejected), the closing rung was re-added to `open_spreads_next` UNCHANGED, but the OPEN still went ahead
|
||||
because it was gated on the stale `may_open` plan -- breaching the ladder cap (6 open rungs instead of 5).
|
||||
Worse, since DTE only decreases, the un-closable rung would be re-planned-for-close (and re-free a phantom
|
||||
slot) every subsequent run, growing the ladder unbounded behind the broker-reject.
|
||||
|
||||
This test drives the REAL `plan_and_record_vrp` composition: the real `ForwardNavRepo`/`XspOptionBarsRepo`/
|
||||
`VrpStrategy` against a throwaway sqlite file, a fake OPRA `dbn` source, and a fake broker whose CLOSE raises
|
||||
but whose OPEN would otherwise succeed -- then asserts the post-run ladder never exceeds its cap."""
|
||||
import datetime as dt
|
||||
|
||||
from fxhnt.adapters.persistence.forward_nav import ForwardNavRepo
|
||||
from fxhnt.application.vrp_exec_record import plan_and_record_vrp
|
||||
from fxhnt.application.vrp_pricing import bs_put
|
||||
from fxhnt.domain.models import OptionContract
|
||||
|
||||
LADDER = 5 # matches STRATEGY_REGISTRY["vrp"]["definition"]["params"]["ladder"]
|
||||
TODAY = "2026-07-13" # a Monday -> VrpStrategy's default entry_weekday=0
|
||||
NEAR_EXPIRY = (dt.date.fromisoformat(TODAY) + dt.timedelta(days=1)).isoformat() # DTE=1 -> planned close
|
||||
NEW_EXPIRY = (dt.date.fromisoformat(TODAY) + dt.timedelta(days=30)).isoformat() # ~30 DTE -> new candidate
|
||||
_FORWARD = 470.0
|
||||
_TAU = 30 / 365.0
|
||||
_SIGMA = 0.20 # flat-vol BS put marks -> monotonic in strike (real credit, real ~20-delta pick)
|
||||
|
||||
|
||||
class _TestRepo(ForwardNavRepo):
|
||||
"""The real forward-nav repo (backing `run_track`'s anchor/book-state/record machinery), pointed at a
|
||||
throwaway sqlite file and stamped with `_dsn` so `plan_and_record_vrp` never falls back to
|
||||
`get_settings().operational_dsn` (no live Postgres in this test)."""
|
||||
def __init__(self, dsn: str) -> None:
|
||||
super().__init__(dsn)
|
||||
self._dsn = dsn
|
||||
|
||||
|
||||
class _FakeDbn:
|
||||
"""OPRA stand-in: a DTE=1 spread (the rung about to be closed) plus a ~30-DTE put ladder + ATM call (the
|
||||
new-candidate universe). The 30-DTE puts are priced off real Black-Scholes (`bs_put`, flat 20% vol) so
|
||||
they are monotonically increasing in strike -- a genuine (K_short, K_short-5) vertical has POSITIVE
|
||||
credit and a real ~20-delta short strike, unlike a hand-waved linear-in-strike toy price."""
|
||||
|
||||
def last_available_option_date(self) -> dt.date:
|
||||
# OPRA's available-end == TODAY here (this fake's slice IS complete for TODAY), so
|
||||
# `_freeze_latest_session` freezes TODAY on its first candidate -> freeze_date == today.
|
||||
return dt.date.fromisoformat(TODAY)
|
||||
|
||||
def fetch_option_chain(self, _parent: str, _as_of: str) -> list[OptionContract]:
|
||||
contracts = [
|
||||
OptionContract("XSPNEARP00460000", dt.date.fromisoformat(NEAR_EXPIRY), 460.0, "P"),
|
||||
OptionContract("XSPNEARP00455000", dt.date.fromisoformat(NEAR_EXPIRY), 455.0, "P"),
|
||||
OptionContract("XSPNEWC00470000", dt.date.fromisoformat(NEW_EXPIRY), 470.0, "C"),
|
||||
]
|
||||
for k in range(420, 475, 5):
|
||||
contracts.append(OptionContract(f"XSPNEWP{k*1000:08d}", dt.date.fromisoformat(NEW_EXPIRY), float(k), "P"))
|
||||
return contracts
|
||||
|
||||
def fetch_option_bars(self, osi_symbols: list[str], start: str, _end: str) -> dict[str, dict[str, float]]:
|
||||
atm_put = round(bs_put(_FORWARD, _FORWARD, _SIGMA, _TAU), 4) # K==F -> C==P at the forward (parity)
|
||||
marks = {"XSPNEARP00460000": 1.0, "XSPNEARP00455000": 0.5, "XSPNEWC00470000": atm_put}
|
||||
for k in range(420, 475, 5):
|
||||
marks[f"XSPNEWP{k*1000:08d}"] = round(bs_put(_FORWARD, float(k), _SIGMA, _TAU), 4)
|
||||
return {osi: {start: marks[osi]} for osi in osi_symbols if osi in marks}
|
||||
|
||||
|
||||
class _CapFakeBroker:
|
||||
"""Distinguishes CLOSE vs OPEN by `build_close_legs`/`build_combo_legs`'s documented sign convention
|
||||
(close BUYS the short back first; open SELLs the short first): the CLOSE always raises (an illiquid
|
||||
near-expiry contract rejected by the broker); an OPEN would otherwise succeed."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.close_calls = 0
|
||||
self.open_calls = 0
|
||||
|
||||
def place_combo(self, legs: list[tuple[str, str, int]], _net_limit: float) -> str:
|
||||
first_action = legs[0][1]
|
||||
if first_action == "BUY":
|
||||
self.close_calls += 1
|
||||
raise RuntimeError("broker reject: illiquid near-expiry contract")
|
||||
self.open_calls += 1
|
||||
return "ORDER-OPEN-1"
|
||||
|
||||
|
||||
def _held_rung(i: int) -> dict:
|
||||
"""A held rung far from expiry/profit-take, with an osi that is never frozen -> `_mark_spread` returns
|
||||
None -> plan_vrp_ladder HOLDs it (not a close), regardless of its (irrelevant) credit/expiry values."""
|
||||
return {"entry": "2026-06-01", "expiry": "2026-12-01", "short_osi": f"HELD-S{i}", "long_osi": f"HELD-L{i}",
|
||||
"credit": 1.0, "last_val": 1.0, "qty": 1}
|
||||
|
||||
|
||||
def _near_expiry_rung() -> dict:
|
||||
return {"entry": "2026-07-06", "expiry": NEAR_EXPIRY, "short_osi": "XSPNEARP00460000",
|
||||
"long_osi": "XSPNEARP00455000", "credit": 1.0, "last_val": 1.0, "qty": 1}
|
||||
|
||||
|
||||
def _seed_full_ladder(repo: ForwardNavRepo, at: dt.datetime) -> None:
|
||||
open_spreads = [_near_expiry_rung()] + [_held_rung(i) for i in range(4)]
|
||||
assert len(open_spreads) == LADDER # ladder starts FULL (5/5)
|
||||
repo.set_book_state("vrp_exec_pos", {"open_spreads": open_spreads, "envelope": 1_000_000.0,
|
||||
"venue": "ibkr-xsp", "last_run_date": "2026-07-06"}, at=at)
|
||||
|
||||
|
||||
def test_ladder_cap_not_breached_when_close_fails(tmp_path):
|
||||
dsn = f"sqlite:///{tmp_path / 'ladder_cap.db'}"
|
||||
repo = _TestRepo(dsn)
|
||||
repo.migrate()
|
||||
at = dt.datetime(2026, 7, 13)
|
||||
_seed_full_ladder(repo, at)
|
||||
broker = _CapFakeBroker()
|
||||
|
||||
result = plan_and_record_vrp(
|
||||
repo, dbn=_FakeDbn(), broker=broker, nlv=1_000_000.0, today=TODAY, at=at,
|
||||
paper_envelope=1_000_000.0, account_is_paper=True, venue="ibkr-xsp", execute=True)
|
||||
|
||||
# Sanity: the planner DID plan to close the near-expiry rung (reproducing the exact setup the defect
|
||||
# requires -- may_open was True from the pre-execution plan, since remaining < ladder once this run's
|
||||
# close is applied).
|
||||
assert result["to_close"] == 1
|
||||
|
||||
# The CLOSE was attempted and failed at the broker.
|
||||
assert broker.close_calls == 1
|
||||
|
||||
# THE FIX: the OPEN must be gated on actual post-close occupancy. The failed close re-added its rung
|
||||
# unchanged, so the ladder is still full (5/5) -- the new rung must NOT be opened.
|
||||
assert broker.open_calls == 0, "OPEN must not be placed when a failed close leaves the ladder at cap"
|
||||
|
||||
# The persisted book-state must never exceed the ladder cap.
|
||||
persisted = repo.get_book_state("vrp_exec_pos")
|
||||
assert len(persisted["open_spreads"]) == LADDER
|
||||
assert result["open_spreads"] == LADDER
|
||||
|
||||
# The skip must be visible in the result, not silently dropped.
|
||||
assert any("ladder full" in e for e in result["errors"])
|
||||
assert result["spread"] is None
|
||||
|
||||
|
||||
def test_ladder_opens_normally_when_close_succeeds(tmp_path):
|
||||
"""Companion sanity check: with a broker that never rejects, the same setup DOES free a slot and open
|
||||
the new rung -- proving the fix only back-pressures on an actual close FAILURE, not on every close."""
|
||||
class _AlwaysSucceedsBroker:
|
||||
def __init__(self) -> None:
|
||||
self.close_calls = 0
|
||||
self.open_calls = 0
|
||||
|
||||
def place_combo(self, legs, _net_limit):
|
||||
if legs[0][1] == "BUY":
|
||||
self.close_calls += 1
|
||||
else:
|
||||
self.open_calls += 1
|
||||
return "ORDER-OK"
|
||||
|
||||
dsn = f"sqlite:///{tmp_path / 'ladder_ok.db'}"
|
||||
repo = _TestRepo(dsn)
|
||||
repo.migrate()
|
||||
at = dt.datetime(2026, 7, 13)
|
||||
_seed_full_ladder(repo, at)
|
||||
broker = _AlwaysSucceedsBroker()
|
||||
|
||||
result = plan_and_record_vrp(
|
||||
repo, dbn=_FakeDbn(), broker=broker, nlv=1_000_000.0, today=TODAY, at=at,
|
||||
paper_envelope=1_000_000.0, account_is_paper=True, venue="ibkr-xsp", execute=True)
|
||||
|
||||
assert broker.close_calls == 1
|
||||
assert broker.open_calls == 1
|
||||
assert result["open_spreads"] == LADDER # one closed, one opened -> still 5, not 6
|
||||
|
||||
|
||||
# --- OPRA publish-lag: freeze the last COMPLETE session (T-1), not today's unpublished slice --------------
|
||||
|
||||
_FREEZE_DATE = "2026-07-10" # _prev_trading_day("2026-07-13" Monday) -> Friday (T-1 complete session)
|
||||
_LAG_EXPIRY = "2026-08-19" # ~40 DTE from the frozen session -> in the put band + calls band
|
||||
|
||||
|
||||
class _LagDbn:
|
||||
"""OPRA publishes on a ~1-session lag: TODAY's slice is still settling, so a freeze for TODAY raises
|
||||
`DatabentoRangeUnavailable` (exactly the live 07:42 UTC crash), while the prior COMPLETE session freezes
|
||||
fine. `last_available_option_date` reports TODAY (the naive available-end); `_freeze_latest_session` must
|
||||
step back one trading day to _FREEZE_DATE. A put ladder + ATM call priced off flat-vol Black-Scholes."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.chain_calls: list[str] = []
|
||||
|
||||
def last_available_option_date(self) -> dt.date:
|
||||
return dt.date.fromisoformat(TODAY)
|
||||
|
||||
def fetch_option_chain(self, _parent: str, as_of: str) -> list[OptionContract]:
|
||||
self.chain_calls.append(as_of)
|
||||
if as_of == TODAY: # today's session is not published yet -> unavailable
|
||||
from fxhnt.adapters.data.databento import DatabentoRangeUnavailable
|
||||
raise DatabentoRangeUnavailable(f"requested start {as_of} >= available end {_FREEZE_DATE}")
|
||||
contracts = [OptionContract("XSPLAGC00470000", dt.date.fromisoformat(_LAG_EXPIRY), 470.0, "C")]
|
||||
for k in range(420, 475, 5):
|
||||
contracts.append(OptionContract(f"XSPLAGP{k*1000:08d}", dt.date.fromisoformat(_LAG_EXPIRY),
|
||||
float(k), "P"))
|
||||
return contracts
|
||||
|
||||
def fetch_option_bars(self, osi_symbols: list[str], start: str, _end: str) -> dict[str, dict[str, float]]:
|
||||
tau = 40 / 365.0
|
||||
atm_put = round(bs_put(470.0, 470.0, 0.20, tau), 4)
|
||||
marks = {"XSPLAGC00470000": atm_put}
|
||||
for k in range(420, 475, 5):
|
||||
marks[f"XSPLAGP{k*1000:08d}"] = round(bs_put(470.0, float(k), 0.20, tau), 4)
|
||||
return {osi: {start: marks[osi]} for osi in osi_symbols if osi in marks}
|
||||
|
||||
|
||||
class _NoopBroker:
|
||||
def place_combo(self, _legs, _net): # never called: held rung, no new entry (envelope 0)
|
||||
raise AssertionError("broker must not be called in the lag test")
|
||||
|
||||
|
||||
def test_freeze_uses_last_complete_session_not_today(tmp_path):
|
||||
"""The live production bug: `plan_and_record_vrp` froze `today`, whose OPRA slice is NEVER published
|
||||
intraday -> DatabentoRangeUnavailable before the run reached the broker. FIX: freeze the last COMPLETE
|
||||
session (T-1) via `_freeze_latest_session`, but keep `today` for DTE / booking. Assert the run does not
|
||||
crash, freezes T-1 (marks present at _FREEZE_DATE, absent at TODAY), and still marks + records under today."""
|
||||
dsn = f"sqlite:///{tmp_path / 'lag.db'}"
|
||||
repo = _TestRepo(dsn)
|
||||
repo.migrate()
|
||||
at = dt.datetime(2026, 7, 13)
|
||||
# a prior held rung whose legs are real band contracts (frozen at T-1) -> it marks -> exec_return records.
|
||||
held = {"entry": "2026-06-15", "expiry": _LAG_EXPIRY, "short_osi": "XSPLAGP00450000",
|
||||
"long_osi": "XSPLAGP00445000", "credit": 1.0, "last_val": 1.0, "qty": 1}
|
||||
repo.set_book_state("vrp_exec_pos", {"open_spreads": [held], "envelope": 1_000_000.0,
|
||||
"venue": "ibkr-xsp", "last_run_date": "2026-07-06"}, at=at)
|
||||
dbn = _LagDbn()
|
||||
|
||||
result = plan_and_record_vrp(
|
||||
repo, dbn=dbn, broker=_NoopBroker(), nlv=1_000_000.0, today=TODAY, at=at,
|
||||
paper_envelope=0.0, account_is_paper=True, venue="ibkr-xsp", execute=True) # envelope 0 -> no new entry
|
||||
|
||||
# TODAY was attempted (raised) and the run stepped back to the last complete session.
|
||||
assert TODAY in dbn.chain_calls
|
||||
assert _FREEZE_DATE in dbn.chain_calls
|
||||
|
||||
# marks were frozen at T-1, NOT at today (today's slice is unpublished).
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
bars = XspOptionBarsRepo(dsn)
|
||||
assert bars.bars_for(["XSPLAGP00450000"], _FREEZE_DATE, _FREEZE_DATE) # frozen at the complete session
|
||||
assert not bars.bars_for(["XSPLAGP00450000"], TODAY, TODAY) # nothing under today
|
||||
|
||||
# the prior rung still marked and the exec return was recorded under `today` (booking/calendar date).
|
||||
assert result["exec_return"] is not None
|
||||
persisted = repo.get_book_state("vrp_exec_pos")
|
||||
assert persisted["last_run_date"] == TODAY
|
||||
assert len(persisted["open_spreads"]) == 1 # held (not closed, not stacked)
|
||||
@@ -1,136 +0,0 @@
|
||||
import pytest
|
||||
|
||||
from fxhnt.application.vrp_exec_record import (
|
||||
build_close_legs,
|
||||
build_combo_legs,
|
||||
plan_vrp_ladder,
|
||||
refuse_if_live,
|
||||
rung_notional,
|
||||
)
|
||||
|
||||
|
||||
def test_build_combo_legs_is_defined_risk_credit():
|
||||
legs, net = build_combo_legs(short_osi="XSP..P00470000", long_osi="XSP..P00465000",
|
||||
short_price=1.20, long_price=0.40, qty=1)
|
||||
assert ("XSP..P00470000", "SELL", 1) in legs
|
||||
assert ("XSP..P00465000", "BUY", 1) in legs
|
||||
assert net == pytest.approx(-0.80) # negative limit = net credit received
|
||||
|
||||
|
||||
def test_build_close_legs_reverses_open_legs_with_positive_debit():
|
||||
legs, net = build_close_legs(short_osi="XSP..P00470000", long_osi="XSP..P00465000", qty=1, mark_today=0.30)
|
||||
assert ("XSP..P00470000", "BUY", 1) in legs # BUY back the short
|
||||
assert ("XSP..P00465000", "SELL", 1) in legs # SELL the long
|
||||
assert net == pytest.approx(0.30) # positive limit = debit paid to close
|
||||
|
||||
|
||||
def _rung(*, short_osi="S1", long_osi="L1", credit=1.0, qty=2, expiry="2026-09-01"):
|
||||
return {"entry": "2026-07-06", "expiry": expiry, "short_osi": short_osi, "long_osi": long_osi,
|
||||
"credit": credit, "last_val": credit, "qty": qty}
|
||||
|
||||
|
||||
def test_plan_vrp_ladder_no_stacking_when_ladder_is_full():
|
||||
# 5 rungs already open, none hitting profit-take/DTE -> no closes -> ladder stays full -> may NOT open.
|
||||
opens = [_rung(short_osi=f"S{i}") for i in range(5)]
|
||||
marks = {sp["short_osi"]: 0.9 for sp in opens} # well above the 50% profit-take line, DTE far out
|
||||
to_close, may_open, slots = plan_vrp_ladder(opens, marks, marks, "2026-07-06", ladder=5, entry_weekday=0)
|
||||
assert to_close == []
|
||||
assert may_open is False
|
||||
assert slots == 0
|
||||
|
||||
|
||||
def test_plan_vrp_ladder_closes_rung_at_dte_le_1():
|
||||
sp = _rung(expiry="2026-07-07") # DTE = 1 on "2026-07-06"
|
||||
# signal above the profit line: the close is driven by DTE<=1, not profit-take; paired with the ACTUAL mark.
|
||||
to_close, _may_open, _slots = plan_vrp_ladder([sp], {"S1": 0.9}, {"S1": 0.9}, "2026-07-06", ladder=5,
|
||||
entry_weekday=0)
|
||||
assert len(to_close) == 1
|
||||
assert to_close[0][0] is sp
|
||||
assert to_close[0][1] == 0.9 # paired value is the ACTUAL mark (the debit to close)
|
||||
|
||||
|
||||
def test_plan_vrp_ladder_closes_rung_at_profit_take_on_model_signal():
|
||||
sp = _rung(credit=1.0)
|
||||
# signal_mark <= (1 - profit_take) * credit == 0.50 -> closes; paired with the actual mark (here also 0.40).
|
||||
to_close, _may_open, _slots = plan_vrp_ladder([sp], {"S1": 0.40}, {"S1": 0.40}, "2026-07-06", ladder=5,
|
||||
profit_take=0.5, entry_weekday=0)
|
||||
assert len(to_close) == 1
|
||||
|
||||
|
||||
def test_plan_vrp_ladder_noisy_actual_mark_does_not_trip_profit_take():
|
||||
"""THE FIX (MINOR-1): a noisy far-OTM long-leg spike drags the RAW short−long actual mark down under the
|
||||
50%-profit line, but the SMOOTH model signal stays above it — so no spurious close. The close decision must
|
||||
use the model signal, not the actual mark."""
|
||||
sp = _rung(credit=1.0)
|
||||
actual = {"S1": 0.30} # raw short−long, dragged low by a long-leg quote spike -> would false-trip
|
||||
signal = {"S1": 0.90} # smooth model value, well above the 0.50 profit line
|
||||
to_close, _may_open, _slots = plan_vrp_ladder([sp], actual, signal, "2026-07-06", ladder=5,
|
||||
profit_take=0.5, entry_weekday=0)
|
||||
assert to_close == [] # model signal governs -> no spurious profit-take close
|
||||
# and when the MODEL genuinely shows 50% profit, it DOES close (at the actual mark).
|
||||
to_close2, _m, _s = plan_vrp_ladder([sp], {"S1": 0.55}, {"S1": 0.45}, "2026-07-06", ladder=5,
|
||||
profit_take=0.5, entry_weekday=0)
|
||||
assert len(to_close2) == 1 and to_close2[0][1] == 0.55
|
||||
|
||||
|
||||
def test_plan_vrp_ladder_holds_rung_with_no_actual_mark_today():
|
||||
sp = _rung()
|
||||
# no ACTUAL mark -> HOLD (cannot close without a real price), regardless of the model signal.
|
||||
to_close, _may_open, _slots = plan_vrp_ladder([sp], {"S1": None}, {"S1": 0.10}, "2026-07-06", ladder=5,
|
||||
entry_weekday=0)
|
||||
assert to_close == []
|
||||
|
||||
|
||||
def test_plan_vrp_ladder_may_open_on_entry_day_with_free_slot():
|
||||
to_close, may_open, slots = plan_vrp_ladder([], {}, {}, "2026-07-06", ladder=5, entry_weekday=0) # Monday
|
||||
assert to_close == []
|
||||
assert may_open is True
|
||||
assert slots == 1
|
||||
|
||||
|
||||
def test_plan_vrp_ladder_no_open_off_entry_day():
|
||||
to_close, may_open, slots = plan_vrp_ladder([], {}, {}, "2026-07-07", ladder=5, entry_weekday=0) # Tuesday
|
||||
assert may_open is False
|
||||
assert slots == 0
|
||||
|
||||
|
||||
def test_plan_vrp_ladder_open_frees_up_when_a_close_makes_room():
|
||||
# ladder full (5/5), but one rung closes this run (DTE<=1) -> a slot opens for a new entry.
|
||||
opens = [_rung(short_osi=f"S{i}", expiry="2026-07-07") if i == 0 else _rung(short_osi=f"S{i}")
|
||||
for i in range(5)]
|
||||
marks = {sp["short_osi"]: 0.9 for sp in opens}
|
||||
to_close, may_open, slots = plan_vrp_ladder(opens, marks, marks, "2026-07-06", ladder=5, entry_weekday=0)
|
||||
assert len(to_close) == 1
|
||||
assert may_open is True
|
||||
assert slots == 1
|
||||
|
||||
|
||||
def test_rung_notional_splits_envelope_across_ladder_and_bounds_total():
|
||||
per_rung = rung_notional(1_000_000.0, 5)
|
||||
assert per_rung == pytest.approx(200_000.0)
|
||||
assert per_rung * 5 == pytest.approx(1_000_000.0) # total open notional bounded at ~envelope
|
||||
|
||||
|
||||
class _RaisingBroker:
|
||||
"""A fake broker whose place_combo must NEVER be called by a refused (non-paper, real-capital) run."""
|
||||
def __init__(self) -> None:
|
||||
self.calls = 0
|
||||
|
||||
def place_combo(self, legs, net_limit): # noqa: ANN001 - test double
|
||||
self.calls += 1
|
||||
raise AssertionError("place_combo must not be called when the paper-envelope guard refuses")
|
||||
|
||||
|
||||
def test_refuse_if_live_blocks_non_paper_account_and_places_zero_orders():
|
||||
broker = _RaisingBroker()
|
||||
with pytest.raises(ValueError, match="refused"):
|
||||
refuse_if_live(paper_envelope=1_000_000, account_is_paper=False)
|
||||
assert broker.calls == 0
|
||||
|
||||
|
||||
def test_refuse_if_live_allows_paper_account():
|
||||
refuse_if_live(paper_envelope=1_000_000, account_is_paper=True) # must not raise
|
||||
|
||||
|
||||
def test_refuse_if_live_allows_zero_envelope_on_any_account():
|
||||
refuse_if_live(paper_envelope=0.0, account_is_paper=False) # off = no real-capital exposure guard
|
||||
@@ -1,43 +0,0 @@
|
||||
import math
|
||||
import pytest
|
||||
from fxhnt.application.vrp_pricing import bs_put, implied_vol_put, put_delta, parity_forward
|
||||
|
||||
|
||||
def test_bs_put_atm_positive_and_bounded():
|
||||
# ATM put (F=K) has value between 0 and K; ~0.4*F*sigma*sqrt(tau) for small tau
|
||||
p = bs_put(F=100.0, K=100.0, sigma=0.20, tau=30 / 365)
|
||||
assert 0.0 < p < 100.0
|
||||
assert p == pytest.approx(0.4 * 100.0 * 0.20 * math.sqrt(30 / 365), rel=0.05)
|
||||
|
||||
|
||||
def test_bs_put_monotone_in_strike():
|
||||
lo = bs_put(F=100.0, K=90.0, sigma=0.2, tau=0.1)
|
||||
hi = bs_put(F=100.0, K=110.0, sigma=0.2, tau=0.1)
|
||||
assert hi > lo # higher strike put is worth more
|
||||
|
||||
|
||||
def test_implied_vol_round_trip():
|
||||
price = bs_put(F=100.0, K=95.0, sigma=0.27, tau=45 / 365)
|
||||
iv = implied_vol_put(price=price, F=100.0, K=95.0, tau=45 / 365)
|
||||
assert iv == pytest.approx(0.27, abs=1e-4)
|
||||
|
||||
|
||||
def test_implied_vol_returns_none_on_arbitrage():
|
||||
# a price below intrinsic (K-F, discounted) has no arb-free IV
|
||||
assert implied_vol_put(price=0.0001, F=100.0, K=130.0, tau=0.1) is None
|
||||
|
||||
|
||||
def test_put_delta_otm_near_target():
|
||||
# ~20-delta OTM put sits below spot; delta magnitude in (0,0.5)
|
||||
d = put_delta(F=100.0, K=94.0, sigma=0.20, tau=30 / 365)
|
||||
assert 0.0 < d < 0.5
|
||||
|
||||
|
||||
def test_parity_forward_recovers_forward():
|
||||
F = 100.0
|
||||
tau = 0.1
|
||||
sigma = 0.2
|
||||
# build synthetic ATM call/put at K=100 consistent with forward F via parity: C - P = F - K
|
||||
p = bs_put(F=F, K=100.0, sigma=sigma, tau=tau)
|
||||
c = p + (F - 100.0)
|
||||
assert parity_forward(call_atm=c, put_atm=p, k_atm=100.0) == pytest.approx(F, abs=1e-6)
|
||||
@@ -1,28 +0,0 @@
|
||||
"""Task 7: the VRP CronJob YAML must parse and ship SUSPENDED — it places live-adjacent combo orders and the
|
||||
IBKR account has no options permission yet, so it must not be armable by an accidental merge."""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
_CRONJOB_PATH = _REPO_ROOT / "infra" / "k8s" / "jobs" / "fxhnt-vrp-rebalancer.yaml"
|
||||
_BACKFILL_PATH = _REPO_ROOT / "infra" / "k8s" / "jobs" / "fxhnt-opra-backfill.yaml"
|
||||
|
||||
|
||||
def test_vrp_rebalancer_cronjob_suspended():
|
||||
docs = list(yaml.safe_load_all(_CRONJOB_PATH.read_text()))
|
||||
cronjobs = [d for d in docs if d and d.get("kind") == "CronJob"]
|
||||
assert len(cronjobs) == 1
|
||||
cj = cronjobs[0]
|
||||
assert cj["metadata"]["name"] == "fxhnt-vrp-rebalancer"
|
||||
assert cj["spec"]["suspend"] is True
|
||||
|
||||
|
||||
def test_opra_backfill_yaml_is_valid_and_is_a_bare_job():
|
||||
docs = [d for d in yaml.safe_load_all(_BACKFILL_PATH.read_text()) if d]
|
||||
jobs = [d for d in docs if d.get("kind") == "Job"]
|
||||
assert len(jobs) == 1
|
||||
assert jobs[0]["metadata"]["name"] == "fxhnt-opra-backfill"
|
||||
assert "suspend" not in jobs[0]["spec"] # bare Jobs have no suspend gate — manual apply only
|
||||
@@ -1,109 +0,0 @@
|
||||
import datetime as dt
|
||||
from fxhnt.adapters.persistence.xsp_option_bars import XspOptionBarsRepo
|
||||
|
||||
|
||||
def _repo(tmp_path):
|
||||
r = XspOptionBarsRepo(f"sqlite:///{tmp_path/'o.db'}")
|
||||
r.migrate()
|
||||
return r
|
||||
|
||||
|
||||
def _bar(osi, date, expiry, strike, right, close):
|
||||
return dict(osi_symbol=osi, date=date, expiry=expiry, strike=strike, right=right, close=close)
|
||||
|
||||
|
||||
def test_upsert_and_bars_for(tmp_path):
|
||||
r = _repo(tmp_path)
|
||||
at = dt.datetime(2024, 7, 15)
|
||||
r.upsert_bars([_bar("XSP240816P00470000", "2024-07-15", "2024-08-16", 470.0, "P", 1.20),
|
||||
_bar("XSP240816P00470000", "2024-07-16", "2024-08-16", 470.0, "P", 1.35)], at)
|
||||
assert r.bars_for(["XSP240816P00470000"], "2024-07-15", "2024-07-16") == {
|
||||
"XSP240816P00470000": {"2024-07-15": 1.20, "2024-07-16": 1.35}}
|
||||
|
||||
|
||||
def test_upsert_is_idempotent(tmp_path):
|
||||
r = _repo(tmp_path)
|
||||
at = dt.datetime(2024, 7, 15)
|
||||
r.upsert_bars([_bar("A", "2024-07-15", "2024-08-16", 470.0, "P", 1.20)], at)
|
||||
r.upsert_bars([_bar("A", "2024-07-15", "2024-08-16", 470.0, "P", 1.25)], at) # same PK -> overwrite
|
||||
assert r.bars_for(["A"], "2024-07-15", "2024-07-15") == {"A": {"2024-07-15": 1.25}}
|
||||
|
||||
|
||||
def test_chain_asof_filters_by_dte_window(tmp_path):
|
||||
r = _repo(tmp_path)
|
||||
at = dt.datetime(2024, 7, 15)
|
||||
r.upsert_bars([
|
||||
_bar("P30", "2024-07-15", "2024-08-14", 470.0, "P", 1.2), # 30 DTE -> in [20,45]
|
||||
_bar("P10", "2024-07-15", "2024-07-25", 470.0, "P", 0.4), # 10 DTE -> out
|
||||
_bar("P60", "2024-07-15", "2024-09-13", 470.0, "P", 2.0), # 60 DTE -> out
|
||||
], at)
|
||||
got = {c.osi_symbol for c in r.chain_asof("2024-07-15", 20, 45)}
|
||||
assert got == {"P30"}
|
||||
|
||||
|
||||
# ---- preload (replay N+1 -> one-query in-memory mirror) correctness ---------------------------------
|
||||
_DAYS = ["2024-07-15", "2024-07-16", "2024-07-17", "2024-07-18"]
|
||||
|
||||
|
||||
def _multi_day_rows():
|
||||
rows = []
|
||||
for di, day in enumerate(_DAYS):
|
||||
for exp in ["2024-08-05", "2024-08-16", "2024-09-13"]:
|
||||
for strike in [460.0, 470.0, 480.0]:
|
||||
for right in ("P", "C"):
|
||||
osi = f"XSP-{exp}-{int(strike)}-{right}"
|
||||
rows.append(_bar(osi, day, exp, strike, right, round(1.0 + di * 0.1 + strike * 0.001, 4)))
|
||||
return rows
|
||||
|
||||
|
||||
def _load(tmp_path, name, rows):
|
||||
r = XspOptionBarsRepo(f"sqlite:///{tmp_path / name}")
|
||||
r.migrate()
|
||||
r.upsert_bars(rows, dt.datetime(2024, 1, 1))
|
||||
return r
|
||||
|
||||
|
||||
def _chain(r, day, lo, hi, right):
|
||||
return [(c.osi_symbol, c.expiry, c.strike, c.right) for c in r.chain_asof(day, lo, hi, right)]
|
||||
|
||||
|
||||
def test_preload_full_mirror_matches_db_exactly(tmp_path):
|
||||
"""The preloaded (in-memory) path must return results IDENTICAL to the per-day DB path — same
|
||||
contracts, same strike order, same marks, same date filtering — for every query shape the replay uses."""
|
||||
rows = _multi_day_rows()
|
||||
db = _load(tmp_path, "db.db", rows) # never preloaded -> DB path
|
||||
ca = _load(tmp_path, "ca.db", rows)
|
||||
ca.preload() # full mirror (after=None)
|
||||
|
||||
assert ca.trading_days(None) == db.trading_days(None)
|
||||
assert ca.trading_days("2024-07-16") == db.trading_days("2024-07-16")
|
||||
for day in _DAYS:
|
||||
for lo, hi in ((20, 45), (0, 60)):
|
||||
for right in ("P", "C"):
|
||||
assert _chain(ca, day, lo, hi, right) == _chain(db, day, lo, hi, right)
|
||||
osis = [rows[0]["osi_symbol"], rows[20]["osi_symbol"], "MISSING-OSI"]
|
||||
assert ca.bars_for(osis, "2024-07-16", "2024-07-16") == db.bars_for(osis, "2024-07-16", "2024-07-16")
|
||||
assert ca.bars_for(osis, "2024-07-15", "2024-07-18") == db.bars_for(osis, "2024-07-15", "2024-07-18")
|
||||
assert ca.bars_for([], "2024-07-15", "2024-07-15") == db.bars_for([], "2024-07-15", "2024-07-15")
|
||||
|
||||
|
||||
def test_preload_scoped_covers_range_and_falls_back_outside(tmp_path):
|
||||
"""A scoped preload (after=X) mirrors days > X and, critically, FALLS BACK to the DB for any date at
|
||||
or before X — so it can never under-return an out-of-range query."""
|
||||
rows = _multi_day_rows()
|
||||
db = _load(tmp_path, "db2.db", rows)
|
||||
sc = _load(tmp_path, "sc.db", rows)
|
||||
sc.preload(after="2024-07-16") # covers 07-17, 07-18 in memory; 07-15/07-16 -> DB fallback
|
||||
|
||||
# covered days: served from cache, identical to DB
|
||||
for day in ("2024-07-17", "2024-07-18"):
|
||||
assert _chain(sc, day, 20, 45, "P") == _chain(db, day, 20, 45, "P")
|
||||
osis = [rows[0]["osi_symbol"]]
|
||||
assert sc.bars_for(osis, "2024-07-17", "2024-07-17") == db.bars_for(osis, "2024-07-17", "2024-07-17")
|
||||
assert sc.trading_days("2024-07-16") == db.trading_days("2024-07-16")
|
||||
|
||||
# out-of-range (<= bound): must fall back to DB and still be correct, NOT return empty
|
||||
assert _chain(sc, "2024-07-15", 20, 45, "P") == _chain(db, "2024-07-15", 20, 45, "P")
|
||||
assert _chain(sc, "2024-07-15", 20, 45, "P") # non-empty proof the fallback fired
|
||||
assert sc.bars_for(osis, "2024-07-15", "2024-07-15") == db.bars_for(osis, "2024-07-15", "2024-07-15")
|
||||
assert sc.trading_days(None) == db.trading_days(None) # after=None under scoped preload -> DB fallback, all days
|
||||
Reference in New Issue
Block a user