diff --git a/docs/superpowers/plans/2026-06-16-lifecycle-b1-paper-tracks.md b/docs/superpowers/plans/2026-06-16-lifecycle-b1-paper-tracks.md index fa6d1a1..1921d3b 100644 --- a/docs/superpowers/plans/2026-06-16-lifecycle-b1-paper-tracks.md +++ b/docs/superpowers/plans/2026-06-16-lifecycle-b1-paper-tracks.md @@ -624,33 +624,34 @@ class FakeBinanceFunding: --- -## Task 8: CryptoPerpBarClient + VolIndexClient adapters + poc strategy +## Task 8: poc strategy — FAITHFUL port WITH torch (vendor the real helpers) -**Source:** foxhunt `scripts/surfer/surfer_poc.py` `paper(cfg)` path ONLY + `compute_weights`, `xs_weights` (numpy), `vrp_ccy`/`vrp_book_series`. Momentum sleeve: residual (market-beta-stripped) 20-day momentum → cross-sectional z-score market-neutral unit-gross weights → top-30 → EWMA smooth → weekly rebalance (`day % 7 == 0`). VRP sleeve: Deribit DVOL short-vol with tail gates (don't sell into rising vol over `vrp_rise_k=5`; full size only below `vrp_level_win=60` median), scaled to momentum vol, blended at `vrp_risk_frac=0.30`. **Calibration deviation (per spec):** derive `vrp_mu` (momentum daily vol) + `vrp_vu` (VRP daily vol) from the live panel's own lookback window, NOT `pit_sweep`. NO torch. Live-booking; `extra` holds `positions`, `prices`, `vrp_equity`, `vrp_mu`, `vrp_vu`. Emit days-list (`ret = comb_ret`). +**CORRECTION (do not dodge deps):** poc is ported **faithfully** — keep the algorithm 1:1. Use **torch**; use the **real** `signal_sweep` (`xs_weights`, `validate`, `sharpe_t`) and **real** `pit_sweep` (`load()`); keep the **real** `_vrp_calib` (PIT-history-based VRP sizing). Do NOT reimplement any of this in numpy and do NOT change the calibration source — changing an algorithm to avoid a package is not allowed. The only wrapping is persistence: the live mark/rebalance computation is unchanged; the tracker books `comb_ret` into days-list and the bespoke poc fields live in `extra`. + +**Source:** foxhunt `scripts/surfer/surfer_poc.py` (`paper(cfg)`, `compute_weights`, `vrp_ccy`/`vrp_book_series`, `_vrp_calib`, `_live_panel`), `scripts/surfer/signal_sweep.py`, `scripts/surfer/pit_sweep.py`. **Files:** -- Create: `src/fxhnt/adapters/data/crypto_perp.py` (`CryptoPerpBarClient`) -- Create: `src/fxhnt/adapters/data/vol_index.py` (`VolIndexClient`) -- Create: `src/fxhnt/domain/strategies/momentum_vrp.py` (pure: `xs_weights`, `compute_weights`, `vrp_series`, `combined_return`) -- Modify: `src/fxhnt/application/paper_strategies.py` (add `MomentumVrpStrategy`) -- Modify: `tests/integration/test_paper_strategies.py` +- Create: `src/fxhnt/vendor/surfer/__init__.py`, `signal_sweep.py`, `pit_sweep.py`, `surfer_poc.py` — copied VERBATIM from foxhunt `scripts/surfer/` (preserve the algorithm). Only allowed edits: make the state path + the PIT-cache path configurable via env/param instead of the hardcoded `_REPO`-relative paths (so they resolve to `/data/surfer` + the pod's PIT cache); keep all math, torch usage, and the `from signal_sweep/pit_sweep import ...` wiring intact (adjust to package-relative imports). +- Create: `src/fxhnt/adapters/data/crypto_perp.py` (`CryptoPerpBarClient`) + `src/fxhnt/adapters/data/vol_index.py` (`VolIndexClient`) — only if the vendored `_live_panel`/`_dvol` are refactored to take injected clients; otherwise the vendored module keeps its own urllib fetch (faithful) and these ports are skipped. Prefer the faithful path: keep the vendored fetch. +- Modify: `src/fxhnt/application/paper_strategies.py` (add `MomentumVrpStrategy` wrapping the vendored poc paper step). +- Modify: `pyproject.toml` (add `torch` to the orchestration/poc extra) and `infra/docker/fxhnt.Dockerfile` (install the extra so the cockpit image has torch). torch CPU wheel is sufficient (paper inference is tiny). +- Modify: `tests/integration/test_paper_strategies.py`. -- [ ] **Step 1: Write the failing tests** (pure-domain, deterministic — no network): - - `test_xs_weights_market_neutral_unit_gross`: random `sig` → `xs_weights(sig)` rows sum to ≈0 (market-neutral) and `sum|w|` ≈ 1 (unit gross). - - `test_compute_weights_topk_and_neutral`: synthetic panel of 50 perps over 80 days → returned weights have ≤`topk=30` nonzeros per row, row-sum ≈ 0. - - `test_vrp_tail_gate_holds_when_vol_rising`: a DVOL series that is strictly rising → `vrp_series` books 0 (gated), and below-median falling vol → nonzero short-vol return. - - `test_momentum_vrp_combined_return`: fixed positions/prices + one new bar + a DVOL point → `comb_ret == (1-f)*mom_ret + f*scaled` with hand-computed `mom_ret` (price change − funding) and `scaled = (dvol_today/vu)*mu`. +- [ ] **Step 1: Vendor the real modules** — copy `signal_sweep.py`, `pit_sweep.py`, `surfer_poc.py` from foxhunt `scripts/surfer/` into `src/fxhnt/vendor/surfer/`. Convert the `sys.path.insert` + bare `from signal_sweep import ...` / `import pit_sweep` to package-relative imports (`from fxhnt.vendor.surfer.signal_sweep import ...`). Make `STATE`, the dvol cache path, and the pit-cache path read from env (`FXHNT_SURFER_DATA_DIR`, default `/data/surfer`; PIT cache path via the existing pit_sweep config) — NO other edits. Keep `import torch`. -- [ ] **Step 2: Run to verify they fail** → FAIL. +- [ ] **Step 2: Write the failing test** (uses the REAL functions; torch on CPU; small fixtures): + - `test_xs_weights_market_neutral_unit_gross`: real `signal_sweep.xs_weights` on a small `sig` → rows sum ≈0, `sum|w|` ≈1. + - `test_compute_weights_shape_and_neutral`: real `surfer_poc.compute_weights` on a synthetic 50-perp × 80-day panel → ≤`topk` nonzeros/row, row-sum ≈0. + - `test_momentum_vrp_service_round_trips`: drive `MomentumVrpStrategy` over a fixed local panel/DVOL fixture (monkeypatch the vendored fetch to return fixtures so no network) through two `ForwardTracker` steps → second step books one row; `ForwardStateReader().read(...)` returns a days-list summary with `days==1`; `extra` carries `positions`/`vrp_*`. -- [ ] **Step 3: Implement the domain** `momentum_vrp.py` — port `xs_weights` (numpy z-score, market-neutral, unit gross), `compute_weights(close, qvol, days, cfg)` (residual momentum, top-K, EWMA smooth, weekly held weights), `vrp_ccy`/`vrp_series` (tail-gated short-vol per currency, combine BTC+ETH), and `combined_return(prev_positions, prev_prices, last_close, funding_now, dvol_today, mu, vu, f)`. Reimplement `sharpe`/`std` helpers in numpy. `cfg` defaults from source `CFG`. +- [ ] **Step 3: Run to verify they fail** → FAIL. -- [ ] **Step 4: Implement the adapters** — `crypto_perp.py`: Binance `/fapi/v1/exchangeInfo` (top-liquid perps by `/ticker/24hr` volume) + `/fapi/v1/klines?interval=1d` panel + `/fapi/v1/premiumIndex` funding → `panel()`. `vol_index.py`: Deribit `https://www.deribit.com/api/v2/public/get_volatility_index_data?currency={ccy}&...` → `{epoch_day: dvol}` (cache to `/data/surfer/dvol_{ccy}.json` like the original `_dvol`). +- [ ] **Step 4: Implement the service** `MomentumVrpStrategy.advance(last_date, extra)` — call the vendored poc mark/rebalance logic (the real `paper`-equivalent, refactored to return values rather than print): seed its working state from `extra`; compute `comb_ret` exactly as `surfer_poc.paper` does (momentum mark + real `_vrp_calib`-sized VRP sleeve); return `([(iso(today), comb_ret)], {positions, prices, vrp_equity, vrp_mu, vrp_vu})`. Convert epoch-day → ISO date for the days-list. The numbers must match the vendored algorithm exactly. -- [ ] **Step 5: Implement the service** `MomentumVrpStrategy.advance(last_date, extra)` — fetch panel + DVOL; on first call (no `vrp_vu` in extra) calibrate `mu`/`vu` from the panel lookback; MARK held book → `comb_ret`; weekly rebalance to `compute_weights` target; convert epoch-day → ISO date; return `([(iso_today, comb_ret)], {positions, prices, vrp_equity, vrp_mu, vrp_vu})`. +- [ ] **Step 5: Image deps** — add `torch` to the relevant extra in `pyproject.toml`; update `infra/docker/fxhnt.Dockerfile` to install it; ensure the PIT cache `pit_sweep.load()` needs is present in the cockpit image or mounted (point `pit_sweep` at the warehouse/PVC path available to the Dagster pod). Note this as the one infra add for B1. - [ ] **Step 6: Run to verify it passes** — pytest + mypy → PASS. -- [ ] **Step 7: Commit** — `git commit -m "feat(b1): surfer poc momentum+VRP paper-forward strategy (numpy, no torch)"` +- [ ] **Step 7: Commit** — `git commit -m "feat(b1): surfer poc paper-forward strategy — faithful port with torch + real signal_sweep/pit_sweep"` --- diff --git a/docs/superpowers/specs/2026-06-15-lifecycle-b1-paper-tracks-design.md b/docs/superpowers/specs/2026-06-15-lifecycle-b1-paper-tracks-design.md index c0bf932..52961a1 100644 --- a/docs/superpowers/specs/2026-06-15-lifecycle-b1-paper-tracks-design.md +++ b/docs/superpowers/specs/2026-06-15-lifecycle-b1-paper-tracks-design.md @@ -5,8 +5,12 @@ **Scope of THIS increment:** all 6 remaining strategies — `funding`, `crossvenue`, `multistrat`, `sixtyforty`, `gd`, and `poc`. After this increment all 7 registry strategies have live forward NAV in the cockpit. -`poc` was briefly going to be a separate B1b increment, but its paper-forward path turns out to be numpy-only -(no torch, no `pit_sweep` — see the poc subsection), so it folds cleanly into this increment. +`poc` was briefly going to be a separate B1b increment; it folds into this increment. **CORRECTION +(2026-06-16):** poc is ported **faithfully with torch** + the real `signal_sweep`/`pit_sweep` + the real +`_vrp_calib`. An earlier draft proposed a numpy reimplementation that dropped torch and changed the VRP +calibration source — that is rejected: you do not change an algorithm to avoid a package. The image gains +torch; the helpers are vendored verbatim. See plan Task 8 for the authoritative approach (it supersedes the +"numpy-only / no torch / calibration deviation" wording in the poc subsection + Out-of-scope below). ## Goal diff --git a/infra/docker/fxhnt.Dockerfile b/infra/docker/fxhnt.Dockerfile index 7f3cc9d..9d1537d 100644 --- a/infra/docker/fxhnt.Dockerfile +++ b/infra/docker/fxhnt.Dockerfile @@ -5,7 +5,13 @@ ENV PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1 WORKDIR /app COPY pyproject.toml README.md ./ COPY src/ ./src/ -RUN pip install --upgrade pip && pip install '.[web,pg,data,factory,orchestration]' +# The `poc` extra pulls torch (faithful surfer-PoC port requirement). Use the PyTorch CPU index so the +# cockpit pod gets the small CPU wheel, not the multi-GB CUDA build (no GPU here). NOTE (deploy): the surfer +# PoC paper track also needs the crypto PIT cache at $FXHNT_SURFER_DATA_DIR/crypto_pit/*.npz — already present +# in the pod for the combined book; no extra mount needed. +RUN pip install --upgrade pip && \ + pip install --extra-index-url https://download.pytorch.org/whl/cpu \ + '.[web,pg,data,factory,orchestration,poc]' # DSN is injected at runtime via secretKeyRef in the K8s manifests; empty here so the image contains no credentials. ENV FXHNT_OPERATIONAL_DSN="" EXPOSE 8080 diff --git a/infra/k8s/orchestration/dagster.yaml b/infra/k8s/orchestration/dagster.yaml index ccbeecc..795b4a4 100644 --- a/infra/k8s/orchestration/dagster.yaml +++ b/infra/k8s/orchestration/dagster.yaml @@ -109,6 +109,8 @@ spec: valueFrom: { secretKeyRef: { name: db-credentials, key: password } } - name: DATABENTO_API_KEY valueFrom: { secretKeyRef: { name: databento-credentials, key: api-key } } + - name: TIINGO_API_KEY # equity sleeves (sixtyforty/multistrat/gd) fetch Tiingo adjClose + valueFrom: { secretKeyRef: { name: tiingo-api, key: key } } readinessProbe: { tcpSocket: { port: 3000 }, initialDelaySeconds: 10, periodSeconds: 15 } volumeMounts: - { name: dagster-home, mountPath: /dagster-home } diff --git a/pyproject.toml b/pyproject.toml index bb571de..19a8463 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -22,6 +22,17 @@ web = ["fastapi>=0.110", "uvicorn[standard]>=0.29", "jinja2>=3.1", "httpx>=0.27" pg = ["psycopg[binary]>=3.1"] factory = ["scipy>=1.11"] orchestration = ["dagster>=1.12,<1.13", "dagster-webserver>=1.12,<1.13", "dagster-postgres>=0.28,<0.29"] +# Surfer PoC paper-track (vendored crypto XS-momentum + VRP). torch is a faithful-port requirement +# (signal_sweep.sharpe_t/validate + surfer_poc.backtest use it) — CPU wheel only (no GPU in the cockpit pod). +poc = ["torch>=2.2"] + +[tool.uv.sources] +torch = { index = "pytorch-cpu" } + +[[tool.uv.index]] +name = "pytorch-cpu" +url = "https://download.pytorch.org/whl/cpu" +explicit = true [project.scripts] fxhnt = "fxhnt.cli:app" @@ -48,6 +59,14 @@ select = ["E", "F", "I", "UP", "B", "SIM"] python_version = "3.11" strict = true ignore_missing_imports = true +# Vendored third-party code (faithful verbatim copies, only import/path edits) is intentionally untyped — +# annotating it would diverge from upstream. Exclude it from strict checking; callers treat it as Any. +exclude = '^src/fxhnt/vendor/' + +[[tool.mypy.overrides]] +module = "fxhnt.vendor.*" +ignore_errors = true +follow_imports = "skip" [tool.pytest.ini_options] pythonpath = ["src"] diff --git a/src/fxhnt/adapters/data/binance_funding.py b/src/fxhnt/adapters/data/binance_funding.py new file mode 100644 index 0000000..ed5a70a --- /dev/null +++ b/src/fxhnt/adapters/data/binance_funding.py @@ -0,0 +1,86 @@ +"""urllib BinanceFundingClient — Binance USDⓈ-M futures public API (no key, no capital). Faithful to +foxhunt scripts/surfer/crypto_funding_paper.py's scan() (lines 48-92): the hedgeable, liquid, +crypto-native universe + trailing-30d-mean funding (tf30) + today's booking rate (last24). + + * crypto_native() -> symbols with underlyingType == "COIN" from /fapi/v1/exchangeInfo + * spot_symbols() -> symbols with a Binance SPOT market (api.binance.com/api/v3/ticker/price) + * universe() -> {sym: quoteVolume} for crypto-native, spot-hedgeable USDT perps with vol > LIQ_USD + * scan(prev) -> (liq, tf30, last24) over union of the universe and prev positions + +Retries with backoff; mirrors the original `get()` helper and the yahoo_daily `_get` style.""" +from __future__ import annotations + +import json +import time +import urllib.request +from typing import Any + +from fxhnt.domain.strategies.funding_carry import LIQ_USD + +_FAPI = "https://fapi.binance.com" +INTERVALS_PER_DAY = 3 # Binance funding settles every 8h (matches the original constant) + + +class BinanceFundingClient: + def __init__(self, liq_usd: float = LIQ_USD) -> None: + self._liq_usd = liq_usd + + def _get(self, url: str, tries: int = 4) -> Any: + for a in range(tries): + try: + req = urllib.request.Request(url, headers={"User-Agent": "curl/8"}) + return json.loads(urllib.request.urlopen(req, timeout=30).read()) + except Exception: # noqa: BLE001 — transient HTTP; retry then re-raise + if a == tries - 1: + raise + time.sleep(2 * (a + 1)) + raise RuntimeError("unreachable") + + def crypto_native(self) -> set[str]: + """Symbols whose underlyingType is COIN (original crypto_native(), line 48-50).""" + info = self._get(f"{_FAPI}/fapi/v1/exchangeInfo") + return {s["symbol"] for s in info["symbols"] if s.get("underlyingType") == "COIN"} + + def spot_symbols(self) -> set[str]: + """Symbols with a Binance SPOT market — required to build the delta-neutral hedge + (original spot_symbols(), line 53-57). Perp-only coins can't be cash-and-carry harvested.""" + return {x["symbol"] for x in self._get("https://api.binance.com/api/v3/ticker/price")} + + def universe(self) -> dict[str, float]: + """Liquid, crypto-native, HEDGEABLE (spot+perp) USDT perps (original universe(), line 60-68).""" + native = self.crypto_native() + spot = self.spot_symbols() + t = self._get(f"{_FAPI}/fapi/v1/ticker/24hr") + return { + x["symbol"]: float(x["quoteVolume"]) + for x in t + if x["symbol"].endswith("USDT") + and x["symbol"] in native + and x["symbol"] in spot + and float(x["quoteVolume"]) > self._liq_usd + } + + def _funding_hist(self, sym: str, limit: int = 90) -> list[float]: + """Funding-rate history (original funding_hist(), line 71-73).""" + h = self._get(f"{_FAPI}/fapi/v1/fundingRate?symbol={sym}&limit={limit}") + return [float(x["fundingRate"]) for x in h] + + def scan(self, prev: set[str]) -> tuple[dict[str, float], dict[str, float], dict[str, float]]: + """(liq, tf30, last24) for the union of the universe and prev positions (original scan(), + line 76-92).""" + liq = self.universe() + need = set(liq) | set(prev) + tf30: dict[str, float] = {} + last24: dict[str, float] = {} + for i, sym in enumerate(sorted(need)): + try: + h = self._funding_hist(sym) + except Exception: # noqa: BLE001 — skip transient per-symbol failures + continue + if len(h) < 30: + continue + tf30[sym] = (sum(h) / len(h)) * INTERVALS_PER_DAY + last24[sym] = sum(h[-INTERVALS_PER_DAY:]) + if i % 50 == 49: + time.sleep(0.5) + return liq, tf30, last24 diff --git a/src/fxhnt/adapters/data/cross_venue.py b/src/fxhnt/adapters/data/cross_venue.py new file mode 100644 index 0000000..8437696 --- /dev/null +++ b/src/fxhnt/adapters/data/cross_venue.py @@ -0,0 +1,109 @@ +"""urllib CrossVenueFundingClient — Binance / Bybit / Hyperliquid public perp APIs (no key, no capital). +Faithful to foxhunt scripts/surfer/cross_venue_funding.py (binance()/bybit()/hyperliquid()/spreads(), +lines 55-98): each venue's funding is normalized to a DAILY rate, then exposed per coin per venue. + +Per-venue daily-funding normalization (must match the original exactly): + * Binance: lastFundingRate × (24 / fundingIntervalHours) (default 8h interval → ×3) [lines 56-62] + * Bybit: fundingRate × 3 (8h interval → ×3) [lines 65-68] + * Hyperliquid: funding × 24 (1h interval → ×24) [lines 71-78] + +`base(sym)` strips a trailing USDT/USDC quote so the same coin maps across venues (Hyperliquid already +uses the bare coin name). snapshot() returns (funding, volume) — each {coin: {venue: rate|vol}} from ONE +fetch — over the union of coins present on any fetched venue (venues that fail to fetch are simply absent, +as in spreads()). + +Retries with backoff; mirrors the original `get()` helper (GET + POST for Hyperliquid).""" +from __future__ import annotations + +import json +import time +import urllib.request +from typing import Any + +_BINANCE = "https://fapi.binance.com" +_BYBIT = "https://api.bybit.com" +_HL = "https://api.hyperliquid.xyz" + +_VENUES = ("Binance", "Bybit", "HL") + + +def _base(sym: str) -> str: + """Strip a trailing USDT/USDC quote (original base(), lines 48-52).""" + for q in ("USDT", "USDC"): + if sym.endswith(q): + return sym[: -len(q)] + return sym + + +class CrossVenueFundingClient: + def _get(self, url: str, post: dict[str, Any] | None = None, tries: int = 4) -> Any: + data = json.dumps(post).encode() if post is not None else None + headers = {"User-Agent": "Mozilla/5.0", "Accept": "application/json"} + if post is not None: + headers["Content-Type"] = "application/json" + for a in range(tries): + try: + req = urllib.request.Request(url, data=data, headers=headers) + return json.loads(urllib.request.urlopen(req, timeout=30).read()) + except Exception: # noqa: BLE001 — transient HTTP; retry then re-raise + if a == tries - 1: + raise + time.sleep(2 * (a + 1)) + raise RuntimeError("unreachable") + + def _binance(self) -> dict[str, tuple[float, float]]: + """{coin: (daily_funding, 24h_quote_volume)} (original binance(), lines 55-62).""" + fund = {x["symbol"]: float(x["lastFundingRate"]) for x in self._get(f"{_BINANCE}/fapi/v1/premiumIndex")} + try: + itv = { + x["symbol"]: float(x.get("fundingIntervalHours", 8)) + for x in self._get(f"{_BINANCE}/fapi/v1/fundingInfo") + } + except Exception: # noqa: BLE001 — fundingInfo optional; default 8h interval + itv = {} + vol = {x["symbol"]: float(x["quoteVolume"]) for x in self._get(f"{_BINANCE}/fapi/v1/ticker/24hr")} + return { + _base(s): (f * (24.0 / itv.get(s, 8)), vol.get(s, 0.0)) + for s, f in fund.items() + if s.endswith("USDT") + } + + def _bybit(self) -> dict[str, tuple[float, float]]: + """{coin: (daily_funding, 24h_turnover)} (original bybit(), lines 65-68).""" + r = self._get(f"{_BYBIT}/v5/market/tickers?category=linear")["result"]["list"] + return { + _base(x["symbol"]): (float(x["fundingRate"]) * 3, float(x.get("turnover24h", 0.0))) + for x in r + if x["symbol"].endswith("USDT") and x.get("fundingRate") + } + + def _hyperliquid(self) -> dict[str, tuple[float, float]]: + """{coin: (daily_funding, 24h_notional_volume)} (original hyperliquid(), lines 71-78).""" + r = self._get(f"{_HL}/info", post={"type": "metaAndAssetCtxs"}) + meta, ctxs = r[0], r[1] + out: dict[str, tuple[float, float]] = {} + for u, c in zip(meta["universe"], ctxs, strict=True): + if c.get("funding") is not None: + out[u["name"]] = (float(c["funding"]) * 24, float(c.get("dayNtlVlm", 0.0))) + return out + + def _fetch(self) -> dict[str, dict[str, tuple[float, float]]]: + """{venue: {coin: (daily_funding, volume)}} — skip venues that fail (original spreads() lines 82-87).""" + fns = {"Binance": self._binance, "Bybit": self._bybit, "HL": self._hyperliquid} + out: dict[str, dict[str, tuple[float, float]]] = {} + for nm, fn in fns.items(): + try: + out[nm] = fn() + except Exception: # noqa: BLE001 — a venue outage must not kill the scan (original prints & skips) + continue + return out + + def snapshot(self) -> tuple[dict[str, dict[str, float]], dict[str, dict[str, float]]]: + """One _fetch() pass per run → (funding_by_venue, volume_by_venue) built from the SAME response + objects (funding + the venue's volume field together), over the union of coins on the fetched + venues. Avoids the prior double-fetch + cross-snapshot drift of the two single-field methods.""" + v = self._fetch() + coins = set().union(*[set(d) for d in v.values()]) if v else set() + funding = {c: {nm: v[nm][c][0] for nm in _VENUES if nm in v and c in v[nm]} for c in coins} + volume = {c: {nm: v[nm][c][1] for nm in _VENUES if nm in v and c in v[nm]} for c in coins} + return funding, volume diff --git a/src/fxhnt/adapters/data/massive_daily.py b/src/fxhnt/adapters/data/massive_daily.py new file mode 100644 index 0000000..c823191 --- /dev/null +++ b/src/fxhnt/adapters/data/massive_daily.py @@ -0,0 +1,105 @@ +"""urllib DailyBarClient — licensed Massive (formerly Polygon) daily closes with TOTAL-RETURN +reconstruction. Drop-in for YahooDailyClient on the equity sleeves (sixtyforty/multistrat/gd), which +compound daily TOTAL returns of dividend-paying ETFs (SPY/IEF/GLD/...). + +Why reconstruction is required: Massive aggregates with `adjusted=true` are SPLIT-ONLY — dividends are +NOT folded in (a dividend-day price drop appears in full). So we rebuild a Yahoo-adjclose-equivalent +back-adjusted series ourselves from the split-adjusted closes + the dividends endpoint. + +Method (standard back-adjustment): for each ex-dividend date D with cash C, multiply ALL closes on +dates strictly BEFORE D by factor (1 - C / close_{trading day immediately before D}). Factors compound +across dividends. The resulting day-over-day return on the ex-div day = raw_close_D / adj_close_{D-1} - 1 += price_return + C/prev_close, i.e. the dividend is added back (matches Yahoo's adjclose semantics). + +Plan/history note: our Massive plan only entitles ~2 years of history; multi-year ranges 403. We request +a bounded RECENT window (default last ~500 calendar days, safely inside 2y) — far more than any sleeve's +signal lookback. Auth: Bearer with POLYGON_API_KEY (env). Host default https://api.massive.com. +""" +from __future__ import annotations + +import datetime as dt +import json +import os +import time +import urllib.parse +import urllib.request +from typing import Any + +_AGGS = "{base}/v2/aggs/ticker/{sym}/range/1/day/{frm}/{to}?adjusted=true&sort=asc&limit=50000" +_DIVS = "{base}/v3/reference/dividends?ticker={sym}&ex_dividend_date.gte={frm}&ex_dividend_date.lte={to}&limit=1000" + + +class MassiveDailyClient: + def __init__( + self, + api_key: str | None = None, + lookback_days: int = 500, + base_url: str = "https://api.massive.com", + ) -> None: + self._key = api_key if api_key is not None else os.environ["POLYGON_API_KEY"] + self._lookback_days = lookback_days + self._base = base_url.rstrip("/") + + def _get(self, url: str, tries: int = 4) -> dict[str, Any]: + for a in range(tries): + try: + req = urllib.request.Request(url, headers={"Authorization": f"Bearer {self._key}"}) + data: dict[str, Any] = json.loads(urllib.request.urlopen(req, timeout=30).read()) + return data + except Exception: # noqa: BLE001 — transient HTTP; retry then re-raise + if a == tries - 1: + raise + time.sleep(2 * (a + 1)) + raise RuntimeError("unreachable") + + def _window(self) -> tuple[str, str]: + to = dt.date.today() + frm = to - dt.timedelta(days=self._lookback_days) + return frm.isoformat(), to.isoformat() + + def adj_closes(self, symbol: str) -> dict[str, float]: + frm, to = self._window() + sym = urllib.parse.quote(symbol) + + aggs = self._get(_AGGS.format(base=self._base, sym=sym, frm=frm, to=to)) + # split-adjusted closes keyed by date (tz-aware; midnight-UTC epoch ms) + closes: dict[str, float] = {} + for r in aggs.get("results") or []: + d = dt.datetime.fromtimestamp(r["t"] / 1000, dt.timezone.utc).strftime("%Y-%m-%d") + closes[d] = float(r["c"]) + if not closes: + return {} + + divs = self._get(_DIVS.format(base=self._base, sym=sym, frm=frm, to=to)) + dividends: list[tuple[str, float]] = [ + (r["ex_dividend_date"], float(r["cash_amount"])) + for r in (divs.get("results") or []) + if r.get("cash_amount") + ] + + return self._total_return(closes, dividends) + + @staticmethod + def _total_return(closes: dict[str, float], dividends: list[tuple[str, float]]) -> dict[str, float]: + """Back-adjust split-adjusted closes into a total-return series (Yahoo adjclose equivalent). + + Accumulate one cumulative factor per date: for each ex-div date D (within the close window) with + cash C, every date strictly before D is multiplied by (1 - C / close_{trading day before D}). + """ + dates = sorted(closes) + factor: dict[str, float] = {d: 1.0 for d in dates} + + for ex_date, cash in dividends: + # the last trading day strictly before the ex-div date + prior = [d for d in dates if d < ex_date] + if not prior: + continue # dividend predates our close window — no in-window dates to adjust + prev = prior[-1] + prev_close = closes[prev] + if prev_close <= 0: + continue + f = 1.0 - cash / prev_close + for d in prior: + factor[d] *= f + + return {d: closes[d] * factor[d] for d in dates} diff --git a/src/fxhnt/adapters/data/tiingo_daily.py b/src/fxhnt/adapters/data/tiingo_daily.py new file mode 100644 index 0000000..d985054 --- /dev/null +++ b/src/fxhnt/adapters/data/tiingo_daily.py @@ -0,0 +1,100 @@ +"""urllib DailyBarClient — Tiingo native daily total-return closes. Drop-in for MassiveDailyClient on +the equity sleeves (sixtyforty/multistrat/gd), which compound daily TOTAL returns of dividend-paying +ETFs (SPY/IEF/GLD/PDBC/DBMF) plus crypto (BTC-USD in multistrat). + +Why no reconstruction (unlike the Massive split-only adapter): Tiingo's equity/ETF `adjClose` is a +NATIVE total-return series (dividend AND split adjusted), so we read it directly. Tiingo carries ~33yr +of history; we still request a bounded RECENT window (default ~3yr) — far more than any sleeve's signal +lookback (≤126d warmup + forward) — to keep responses small. + +Routing — crypto vs equity: + Tiingo serves crypto from a DIFFERENT endpoint with a DIFFERENT response shape. Crypto has no + dividends/splits, so its `close` IS the adjusted close (there is no `adjClose` field). + equity/ETF: GET /tiingo/daily/{ticker}/prices?startDate&endDate + → flat JSON list of {date, close, adjClose, divCash, splitFactor, ...}; read `adjClose`. + crypto: GET /tiingo/crypto/prices?tickers={btcusd}&startDate&resampleFreq=1day + → JSON list [{ticker, baseCurrency, quoteCurrency, priceData: [{date, close, ...}]}]; + read nested `priceData[].close`. Yahoo-style `BTC-USD` is mapped to `btcusd`. + +Auth: header `Authorization: Token ` with TIINGO_API_KEY (env). The key is used ONLY in the +request header — never logged or interpolated into a URL. +""" +from __future__ import annotations + +import datetime as dt +import json +import os +import time +import urllib.parse +import urllib.request +from typing import Any + +_DAILY = "{base}/tiingo/daily/{sym}/prices?startDate={frm}&endDate={to}&format=json" +_CRYPTO = "{base}/tiingo/crypto/prices?tickers={sym}&startDate={frm}&resampleFreq=1day&format=json" + +# Known crypto bases (Yahoo-style `-USD`). Tiingo routes these to the crypto endpoint. +_CRYPTO_BASES = {"BTC", "ETH", "SOL", "XRP", "ADA", "DOGE", "LTC", "BCH", "DOT", "AVAX", "LINK", "MATIC"} + + +class TiingoDailyClient: + def __init__( + self, + api_key: str | None = None, + lookback_days: int = 1100, + base_url: str = "https://api.tiingo.com", + ) -> None: + self._key = api_key if api_key is not None else os.environ["TIINGO_API_KEY"] + self._lookback_days = lookback_days + self._base = base_url.rstrip("/") + + def _get(self, url: str, tries: int = 4) -> Any: + for a in range(tries): + try: + # The key lives ONLY in this header — never in the URL or any log line. + req = urllib.request.Request(url, headers={"Authorization": f"Token {self._key}"}) + return json.loads(urllib.request.urlopen(req, timeout=30).read()) + except Exception: # noqa: BLE001 — transient HTTP; retry then re-raise + if a == tries - 1: + raise + time.sleep(2 * (a + 1)) + raise RuntimeError("unreachable") + + def _window(self) -> tuple[str, str]: + to = dt.date.today() + frm = to - dt.timedelta(days=self._lookback_days) + return frm.isoformat(), to.isoformat() + + @staticmethod + def _is_crypto(symbol: str) -> bool: + if "-" not in symbol: + return False + base, _, quote = symbol.partition("-") + return quote.upper() == "USD" and base.upper() in _CRYPTO_BASES + + @staticmethod + def _to_crypto_ticker(symbol: str) -> str: + """Yahoo-style `BTC-USD` → Tiingo crypto ticker `btcusd`.""" + return symbol.replace("-", "").lower() + + def adj_closes(self, symbol: str) -> dict[str, float]: + frm, to = self._window() + if self._is_crypto(symbol): + return self._crypto_closes(symbol, frm, to) + return self._equity_closes(symbol, frm, to) + + def _equity_closes(self, symbol: str, frm: str, to: str) -> dict[str, float]: + sym = urllib.parse.quote(symbol) + rows = self._get(_DAILY.format(base=self._base, sym=sym, frm=frm, to=to)) + out: dict[str, float] = {} + for r in rows or []: + out[str(r["date"])[:10]] = float(r["adjClose"]) + return out + + def _crypto_closes(self, symbol: str, frm: str, to: str) -> dict[str, float]: + sym = self._to_crypto_ticker(symbol) + resp = self._get(_CRYPTO.format(base=self._base, sym=sym, frm=frm, to=to)) + out: dict[str, float] = {} + for series in resp or []: + for r in series.get("priceData") or []: + out[str(r["date"])[:10]] = float(r["close"]) + return out diff --git a/src/fxhnt/adapters/data/yahoo_daily.py b/src/fxhnt/adapters/data/yahoo_daily.py new file mode 100644 index 0000000..b0b2f1a --- /dev/null +++ b/src/fxhnt/adapters/data/yahoo_daily.py @@ -0,0 +1,39 @@ +"""urllib DailyBarClient — Yahoo free daily adjusted closes. Retries with backoff; adjusted closes give +true total return (dividends + coupons). Mirrors foxhunt sixtyforty_paper.daily_adjclose.""" +from __future__ import annotations + +import datetime as dt +import json +import time +import urllib.request +from typing import Any + +_BASE = "https://query1.finance.yahoo.com/v8/finance/chart/{sym}?interval=1d&range={rng}" + + +class YahooDailyClient: + def __init__(self, rng: str = "2y") -> None: + self._rng = rng + + def _get(self, url: str, tries: int = 4) -> dict[str, Any]: + for a in range(tries): + try: + req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) + data: dict[str, Any] = json.loads(urllib.request.urlopen(req, timeout=30).read()) + return data + except Exception: # noqa: BLE001 — transient HTTP; retry then re-raise + if a == tries - 1: + raise + time.sleep(2 * (a + 1)) + raise RuntimeError("unreachable") + + def adj_closes(self, symbol: str) -> dict[str, float]: + res = self._get(_BASE.format(sym=symbol, rng=self._rng))["chart"]["result"][0] + ts = res["timestamp"] + ind = res["indicators"] + adj = ind.get("adjclose", [{}])[0].get("adjclose") or ind["quote"][0]["close"] + out: dict[str, float] = {} + for t, c in zip(ts, adj, strict=False): + if c is not None: + out[dt.datetime.fromtimestamp(t, dt.timezone.utc).strftime("%Y-%m-%d")] = float(c) + return out diff --git a/src/fxhnt/adapters/orchestration/assets.py b/src/fxhnt/adapters/orchestration/assets.py index 2fe610f..b62e1cf 100644 --- a/src/fxhnt/adapters/orchestration/assets.py +++ b/src/fxhnt/adapters/orchestration/assets.py @@ -120,7 +120,96 @@ def combined_forward_nav(context: AssetExecutionContext) -> dict: # type: ignor } -@asset(deps=[combined_forward_nav]) +def _run_paper_tracker(context: AssetExecutionContext, name: str, build_strategy, state_file: str) -> dict: # type: ignore[type-arg,no-untyped-def] + """Best-effort for TRANSIENT failures only: build the strategy from its live adapter, step the + tracker, persist + report state. A transient network/API error logs a warning and returns the + on-disk state without crashing the run. Genuine bugs (ValueError, KeyError, AttributeError, ...) + propagate and fail the asset loudly rather than silently flatlining the track.""" + import json + import urllib.error + + from fxhnt.application.forward_tracker import ForwardTracker + + path = f"{_data_dir()}/{state_file}.json" + try: + st = ForwardTracker(build_strategy(), path).step() + context.log.info(f"{name}: {st.forward_days}d through {st.last_date}") + return {"forward_days": st.forward_days, "last_date": st.last_date} + except ( + urllib.error.URLError, + TimeoutError, + ConnectionError, + OSError, + json.JSONDecodeError, + ) as e: # transient network/IO failure must not kill the nightly run + context.log.warning(f"{name} step failed (state unchanged): {e}") + return {"forward_days": 0, "last_date": None} + + +@asset +def sixtyforty_nav(context: AssetExecutionContext) -> dict: # type: ignore[type-arg] + """Paper-forward 60/40 baseline (Tiingo native total-return adjClose, 33yr history).""" + from fxhnt.adapters.data.tiingo_daily import TiingoDailyClient + from fxhnt.application.paper_strategies import SixtyFortyStrategy + + return _run_paper_tracker( + context, "sixtyforty_nav", lambda: SixtyFortyStrategy(TiingoDailyClient()), "sixtyforty_state" + ) + + +@asset +def multistrat_nav(context: AssetExecutionContext) -> dict: # type: ignore[type-arg] + """Paper-forward multi-strategy book (Tiingo native total-return adjClose; BTC-USD via crypto endpoint).""" + from fxhnt.adapters.data.tiingo_daily import TiingoDailyClient + from fxhnt.application.paper_strategies import MultiStratStrategy + + return _run_paper_tracker( + context, "multistrat_nav", lambda: MultiStratStrategy(TiingoDailyClient()), "multistrat_state" + ) + + +@asset +def gd_nav(context: AssetExecutionContext) -> dict: # type: ignore[type-arg] + """Paper-forward growth-discipline book (Tiingo native total-return adjClose; BTC-USD via crypto endpoint).""" + from fxhnt.adapters.data.tiingo_daily import TiingoDailyClient + from fxhnt.application.paper_strategies import GrowthDisciplineStrategy + + return _run_paper_tracker( + context, "gd_nav", lambda: GrowthDisciplineStrategy(TiingoDailyClient()), "gd_paper_state" + ) + + +@asset +def funding_nav(context: AssetExecutionContext) -> dict: # type: ignore[type-arg] + """Paper-forward crypto funding harvest (Binance public futures API, no key).""" + from fxhnt.adapters.data.binance_funding import BinanceFundingClient + from fxhnt.application.paper_strategies import FundingCarryStrategy + + return _run_paper_tracker( + context, "funding_nav", lambda: FundingCarryStrategy(BinanceFundingClient()), "funding_paper_state" + ) + + +@asset +def crossvenue_nav(context: AssetExecutionContext) -> dict: # type: ignore[type-arg] + """Paper-forward cross-venue funding arb (Binance/Bybit/Hyperliquid public APIs).""" + from fxhnt.adapters.data.cross_venue import CrossVenueFundingClient + from fxhnt.application.paper_strategies import CrossVenueStrategy + + return _run_paper_tracker( + context, "crossvenue_nav", lambda: CrossVenueStrategy(CrossVenueFundingClient()), "crossvenue_state" + ) + + +@asset +def poc_nav(context: AssetExecutionContext) -> dict: # type: ignore[type-arg] + """Paper-forward surfer PoC (crypto residual-momentum + tail-managed VRP; fetches Binance + reads crypto_pit).""" + from fxhnt.application.paper_strategies import MomentumVrpStrategy + + return _run_paper_tracker(context, "poc_nav", lambda: MomentumVrpStrategy(), "poc_state") + + +@asset(deps=[combined_forward_nav, sixtyforty_nav, multistrat_nav, gd_nav, funding_nav, crossvenue_nav, poc_nav]) def cockpit_forward(context: AssetExecutionContext) -> None: """Normalize tracker state files and upsert rows + summary into the operational cockpit DB.""" from fxhnt.application.forward_ingest import ingest_forward_state diff --git a/src/fxhnt/adapters/orchestration/definitions.py b/src/fxhnt/adapters/orchestration/definitions.py index 79f1eb8..13c1cf2 100644 --- a/src/fxhnt/adapters/orchestration/definitions.py +++ b/src/fxhnt/adapters/orchestration/definitions.py @@ -1,19 +1,29 @@ """Dagster Definitions for the B0 lifecycle graph — the 4 combined-book assets + a daily schedule that -materializes the track (replaces the fxhnt-forward cron, same 23:30 UTC slot).""" -from __future__ import annotations +materializes the track (replaces the fxhnt-forward cron, same 23:30 UTC slot). +NOTE: no `from __future__ import annotations` — Dagster 1.12 runtime type-hint inspection.""" from dagster import DefaultScheduleStatus, Definitions, ScheduleDefinition, define_asset_job from fxhnt.adapters.orchestration.assets import ( cockpit_forward, combined_forward_nav, + crossvenue_nav, crypto_bars, + funding_nav, futures_bars, + gd_nav, + multistrat_nav, + poc_nav, + sixtyforty_nav, ) combined_book_job = define_asset_job( name="combined_book_forward_job", - selection=[crypto_bars, futures_bars, combined_forward_nav, cockpit_forward], + selection=[ + crypto_bars, futures_bars, combined_forward_nav, + sixtyforty_nav, multistrat_nav, gd_nav, funding_nav, crossvenue_nav, poc_nav, + cockpit_forward, + ], ) daily_combined_book_schedule = ScheduleDefinition( @@ -25,7 +35,11 @@ daily_combined_book_schedule = ScheduleDefinition( ) defs = Definitions( - assets=[crypto_bars, futures_bars, combined_forward_nav, cockpit_forward], + assets=[ + crypto_bars, futures_bars, combined_forward_nav, + sixtyforty_nav, multistrat_nav, gd_nav, funding_nav, crossvenue_nav, poc_nav, + cockpit_forward, + ], jobs=[combined_book_job], schedules=[daily_combined_book_schedule], ) diff --git a/src/fxhnt/application/forward_tracker.py b/src/fxhnt/application/forward_tracker.py index 064fd32..1b176fd 100644 --- a/src/fxhnt/application/forward_tracker.py +++ b/src/fxhnt/application/forward_tracker.py @@ -1,15 +1,17 @@ -"""Forward-validation tracker for the combined book — the only ground truth before capital. +"""Strategy-agnostic forward-validation tracker — the only ground truth before capital. -Freezes the strategy at INCEPTION (first run) and, on each subsequent run, books only the realized returns -of days STRICTLY AFTER inception — a genuine out-of-sample-in-time paper track (no capital risked). Run -daily on a cron alongside the data fetch; the forward NAV + forward Sharpe accumulate honestly. The -in-sample backtest Sharpe (~1.4) means nothing until the forward record confirms it. +`ForwardTracker` freezes any `ForwardStrategy` port at INCEPTION and thereafter books only realized +returns of days STRICTLY AFTER it — a genuine out-of-sample-in-time paper track (no capital risked). +The combined book is one (recomputable) strategy on it; in-sample backtest Sharpe means nothing until +the forward record confirms it. """ from __future__ import annotations +import datetime as _dt import json import os from dataclasses import dataclass +from typing import Any, Protocol import numpy as np @@ -30,9 +32,23 @@ class ForwardStatus: booked_today: int -class CombinedBookForwardTracker: - def __init__(self, book: CombinedBook, state_path: str) -> None: - self._book = book +class ForwardStrategy(Protocol): + def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]: + """Return candidate (ISO-date, fractional-return) rows the strategy knows now, plus its updated + carry-state. The tracker books only rows strictly after last_date and persists `extra` opaquely.""" + ... + + +def _today_iso() -> str: + return _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%d") + + +class ForwardTracker: + """Strategy-agnostic forward-validation tracker. Freezes inception on first run (books nothing — no + backfill), then books only days strictly after last_date. Owns the days-list state file.""" + + def __init__(self, strategy: ForwardStrategy, state_path: str) -> None: + self._strategy = strategy self._path = state_path def _load(self) -> dict | None: @@ -47,21 +63,25 @@ class CombinedBookForwardTracker: json.dump(state, f, indent=2) def step(self) -> ForwardStatus: - _, dates, combo = self._book.build() # causal sleeve combination, vol-targeted - series = {d: float(r) for d, r in zip(dates, combo) if np.isfinite(r)} - latest = dates[-1] - - state = self._load() - if state is None: # first run: freeze inception, book nothing yet - state = {"inception": latest, "last_date": latest, "nav": 1.0, "days": []} + loaded = self._load() + prev_last = None if loaded is None else (loaded["last_date"] or None) + prev_extra = {} if loaded is None else loaded.get("extra", {}) + rows, new_extra = self._strategy.advance(prev_last, prev_extra) + rows = sorted(rows, key=lambda x: x[0]) booked = 0 - for d in sorted(series): - if d > state["last_date"]: # genuine forward: only days after the last booked - state["nav"] *= (1.0 + series[d]) - state["days"].append({"date": d, "ret": series[d], "nav": state["nav"]}) - state["last_date"] = d - booked += 1 + if loaded is None: + latest = rows[-1][0] if rows else _today_iso() + state: dict[str, Any] = {"inception": latest, "last_date": latest, "nav": 1.0, "days": [], "extra": new_extra} + else: + state = loaded + for d, r in rows: + if d > state["last_date"]: + state["nav"] *= (1.0 + r) + state["days"].append({"date": d, "ret": r, "nav": state["nav"]}) + state["last_date"] = d + booked += 1 + state["extra"] = new_extra self._save(state) rets = np.array([row["ret"] for row in state["days"]]) @@ -70,5 +90,21 @@ class CombinedBookForwardTracker: mdd = float((nav / np.maximum.accumulate(nav) - 1.0).min()) if len(nav) else 0.0 return ForwardStatus( inception=state["inception"], last_date=state["last_date"], forward_days=len(state["days"]), - forward_return_pct=100.0 * (state["nav"] - 1.0), forward_sharpe=sharpe, forward_max_drawdown=mdd, - booked_today=booked) + forward_return_pct=100.0 * (state["nav"] - 1.0), forward_sharpe=sharpe, + forward_max_drawdown=mdd, booked_today=booked) + + +class CombinedBookStrategy: + """Recomputable-series ForwardStrategy over a CombinedBook.""" + def __init__(self, book: CombinedBook) -> None: + self._book = book + + def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]: + _, dates, combo = self._book.build() + rows = [(d, float(r)) for d, r in zip(dates, combo) if np.isfinite(r)] + return rows, extra + + +def CombinedBookForwardTracker(book: CombinedBook, state_path: str) -> ForwardTracker: # noqa: N802 + """Back-compat factory: the combined book is just a recomputable ForwardStrategy.""" + return ForwardTracker(CombinedBookStrategy(book), state_path) diff --git a/src/fxhnt/application/paper_strategies.py b/src/fxhnt/application/paper_strategies.py new file mode 100644 index 0000000..c0ff187 --- /dev/null +++ b/src/fxhnt/application/paper_strategies.py @@ -0,0 +1,149 @@ +"""The paper-forward ForwardStrategy services: each composes a market-data port + a pure domain module +and yields candidate (date, ret) rows for the generic ForwardTracker. Recomputable strategies return the +full series each run (the tracker books only the new tail); live-booking strategies book today + carry +positions in `extra`.""" +from __future__ import annotations + +import os +from collections.abc import Callable +from typing import Any + +from fxhnt.application.forward_tracker import _today_iso +from fxhnt.domain.strategies import ( + cross_venue, + funding_carry, + growth_discipline, + multistrat, + sixtyforty, +) +from fxhnt.ports.market_data import ( + BinanceFundingClient, + CrossVenueFundingClient, + DailyBarClient, +) + +_SIXTYFORTY_WEIGHTS = [("SPY", 0.6), ("IEF", 0.4)] + + +class SixtyFortyStrategy: + def __init__(self, bars: DailyBarClient) -> None: + self._bars = bars + + def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]: + closes = {s: self._bars.adj_closes(s) for s, _ in _SIXTYFORTY_WEIGHTS} + return sixtyforty.daily_returns(closes, _SIXTYFORTY_WEIGHTS), extra + + +class MultiStratStrategy: + def __init__(self, bars: DailyBarClient) -> None: + self._bars = bars + + def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]: + closes = {s: self._bars.adj_closes(s) for s in multistrat.INSTRUMENTS} + return multistrat.book_series(closes), extra + + +class GrowthDisciplineStrategy: + def __init__(self, bars: DailyBarClient) -> None: + self._bars = bars + + def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]: + w = float(os.environ.get("GD_EQUITY_WEIGHT", "0.70")) + qqq = self._bars.adj_closes("QQQ") + closes = {s: self._bars.adj_closes(s) for s in multistrat.INSTRUMENTS} + return growth_discipline.daily_returns(qqq, closes, w), extra + + +class FundingCarryStrategy: + """Live-booking delta-neutral funding-harvest strategy (faithful port of crypto_funding_paper.py). + + Each day it fetches the universe's funding + volume, requalifies (hysteresis), books today's realized + carry on the PRIOR positions net of turnover cost, then rebalances to a fresh equal-weight book stored + in `extra["positions"]`. On the FIRST run the prior book is empty so realized/cost are 0 and the tracker + freezes (books nothing) — it just seeds initial positions, exactly like cmd_run on a fresh state.""" + + def __init__(self, client: BinanceFundingClient, clock: Callable[[], str] = _today_iso) -> None: + self._client = client + self._clock = clock + + def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]: + prev: dict[str, float] = extra.get("positions", {}) + liq, tf30, last24 = self._client.scan(set(prev)) + + q = funding_carry.qualify(liq, tf30, prev) + n = len(q) + newpos = {c: 1.0 / n for c in q} if n else {} + net = funding_carry.book_return(prev, last24, newpos) + + return [(self._clock(), net)], {"positions": newpos} + + +class MomentumVrpStrategy: + """Live-booking surfer PoC: crypto cross-sectional residual-momentum book + tail-managed VRP sleeve + (faithful port of foxhunt scripts/surfer/surfer_poc.py `paper`). Each run fetches the live Binance perp + panel, marks the prior two-sleeve book to today's prices + funding, scales the VRP sleeve to the + momentum-vol risk budget (calibrated once from the cached crypto_pit history), books the combined return + for today, then rebalances on the weekly boundary — all carried in `extra`. On the FIRST run the prior + book is empty so no mark happens (comb_ret 0, today == frozen inception) and the tracker freezes, just + seeding positions/prices/sleeve-equities. `today` comes from the panel's latest day (`int(days[-1])`), + not the wall clock, exactly like the original.""" + + _EXTRA_KEYS = ("inception", "last_mark_day", "positions", "prices", "equity", "log", + "vrp_equity", "combined_equity", "vrp_mu", "vrp_vu") + + def __init__(self, cfg: dict[str, Any] | None = None) -> None: + # import here so torch/vendored surfer is only required when this strategy is actually used. + from fxhnt.vendor.surfer import surfer_poc + + self._poc = surfer_poc + self._cfg = cfg if cfg is not None else surfer_poc.CFG + + def _seed_state(self, extra: dict[str, Any]) -> dict[str, Any]: + """Reconstruct surfer_poc's working state dict from the tracker's opaque `extra` (or a fresh state).""" + st: dict[str, Any] = dict(inception=None, last_mark_day=None, positions={}, prices={}, + equity=1.0, log=[]) + for k in self._EXTRA_KEYS: + if k in extra: + st[k] = extra[k] + return st + + @staticmethod + def _iso(epoch_day: int) -> str: + import datetime as _dt + + return (_dt.datetime(1970, 1, 1, tzinfo=_dt.timezone.utc) + + _dt.timedelta(days=int(epoch_day))).strftime("%Y-%m-%d") + + def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]: + st = self._seed_state(extra) + comb_ret, today_epoch, new_st, _info = self._poc.paper_step(self._cfg, st) + iso_today = self._iso(today_epoch) + updated = {k: new_st[k] for k in self._EXTRA_KEYS if k in new_st} + return [(iso_today, float(comb_ret))], updated + + +class CrossVenueStrategy: + """Live-booking price-neutral cross-venue funding-arb strategy (faithful port of + cross_venue_funding.py). Each day it fetches per-venue funding + volume, computes per-coin spreads + (max-min daily funding across liquid, non-distress venues), requalifies via ENTRY/EXIT hysteresis + + TOPK, books today's realized carry on the PRIOR positions net of turnover cost, then rebalances to a + fresh equal-weight book stored in `extra["positions"]`. On the FIRST run the prior book is empty so + realized/cost are 0 and the tracker freezes (books nothing) — it just seeds initial positions, exactly + like cmd_run on a fresh state.""" + + def __init__(self, client: CrossVenueFundingClient, clock: Callable[[], str] = _today_iso) -> None: + self._client = client + self._clock = clock + + def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]: + prev: dict[str, float] = extra.get("positions", {}) + funding, volume = self._client.snapshot() + rows = cross_venue.compute_spreads(funding, volume) + cur = {r["coin"]: r["spread"] for r in rows} + + held = cross_venue.select(cur, prev) + n = len(held) + newpos = {c: 1.0 / n for c in held} if n else {} + net = cross_venue.book_return(prev, cur, newpos) + + return [(self._clock(), net)], {"positions": newpos} diff --git a/src/fxhnt/domain/strategies/cross_venue.py b/src/fxhnt/domain/strategies/cross_venue.py new file mode 100644 index 0000000..0bbd12b --- /dev/null +++ b/src/fxhnt/domain/strategies/cross_venue.py @@ -0,0 +1,111 @@ +"""Pure cross-venue funding-arb math — faithful 1:1 port of foxhunt +scripts/surfer/cross_venue_funding.py (spreads + cmd_run booking). No I/O. + +Price-neutral: for a coin liquid on >=2 venues, SHORT the highest-funding venue perp + LONG the +lowest-funding venue perp and collect the funding DIFFERENCE (max-min). Each venue's funding is already +normalized to a DAILY rate by the adapter (Binance ×24/interval, Bybit ×3, Hyperliquid ×24). + +Constants match the original exactly (lines 25-31): + * LIQ = 10e6 require > $10M/day on BOTH legs (only venues passing this enter the spread) + * MAXF = 0.005 exclude any leg with |daily funding| > 50bp/day (distress/artifact, un-tradeable) + * TOPK = 10 book at most the top-K spreads + * COST_RT = 0.0010 ~10bp round-trip; net books realized − turnover * (COST_RT/2) + * HURDLE = 0.0005 (display-only in the original scan; not used in select/book) + * ENTRY = 0.0010 hysteresis: only OPEN a fresh pair whose spread > 10bp/day + * EXIT = 0.0005 hysteresis: HOLD a held pair until its spread decays below 5bp/day + +Original spread (spreads(), lines 90-96): per coin, keep only venues whose 24h volume > LIQ AND whose +|daily funding| <= MAXF; if >=2 such venues remain, spread = max(f) - min(f), short = argmax, long = argmin. + +Original select (cmd_run, lines 124-135): + cur = {coin: spread} + held = [c for c in prev if cur.get(c, 0.0) > EXIT] # keep decaying winners + for r in sorted(rows with spread > ENTRY, by -spread): # open strong fresh + if len(held) >= TOPK: break + if r.coin not in held: held.append(r.coin) + newpos = {c: 1/len(held)} + +Original booking (cmd_run, lines 126, 136-137): + realized = sum(w * cur.get(c, 0.0) for c, w in prev.items()) # carry on prior book at TODAY's spread + turn = sum(|newpos[c] - prev[c]| over union) + net = realized - turn * (COST_RT / 2) +""" +from __future__ import annotations + +from typing import Any + +LIQ = 10e6 +MAXF = 0.005 +TOPK = 10 +COST_RT = 0.0010 +HURDLE = 0.0005 +ENTRY = 0.0010 +EXIT = 0.0005 + + +def compute_spreads( + funding_by_venue: dict[str, dict[str, float]], + volume_by_venue: dict[str, dict[str, float]], + liq: float = LIQ, + maxf: float = MAXF, +) -> list[dict[str, Any]]: + """Per-coin cross-venue spread rows, sorted by spread descending (faithful to spreads(), lines 88-98). + + For each coin, keep only venues where 24h volume > `liq` and |daily funding| <= `maxf`; require >=2 + such venues; emit {coin, spread = max-min funding, short = argmax venue, long = argmin venue, f}.""" + rows: list[dict[str, Any]] = [] + for c, venue_f in funding_by_venue.items(): + venue_v = volume_by_venue.get(c, {}) + pts = { + nm: f + for nm, f in venue_f.items() + if venue_v.get(nm, 0.0) > liq and abs(f) <= maxf + } + if len(pts) < 2: + continue + hi = max(pts, key=lambda k: pts[k]) + lo = min(pts, key=lambda k: pts[k]) + rows.append({"coin": c, "spread": pts[hi] - pts[lo], "short": hi, "long": lo, "f": pts}) + rows.sort(key=lambda r: -r["spread"]) + return rows + + +def select( + cur: dict[str, float], + prev: dict[str, float], + entry: float = ENTRY, + exit_: float = EXIT, + topk: int = TOPK, +) -> list[str]: + """Coins to hold today via ENTRY/EXIT hysteresis + TOPK cap (faithful to cmd_run lines 128-134). + + `cur` is {coin: today's spread}. Keep held coins whose spread is still > `exit_`; then open fresh + coins whose spread > `entry` (sorted by spread desc) until `topk` total are held.""" + held = [c for c in prev if cur.get(c, 0.0) > exit_] + fresh = sorted( + (c for c, s in cur.items() if s > entry), + key=lambda c: -cur[c], + ) + for c in fresh: + if len(held) >= topk: + break + if c not in held: + held.append(c) + return held + + +def book_return( + prev: dict[str, float], + cur: dict[str, float], + newpos: dict[str, float], + cost_rt: float = COST_RT, +) -> float: + """Net realized carry: equal-weight realized spread on the PRIOR book at TODAY's spread `cur`, + minus the round-trip cost on turnover between the prior and new books. Faithful to cmd_run + (lines 126, 136-137): `net = realized - turn * (COST_RT / 2)`.""" + realized = sum(w * cur.get(c, 0.0) for c, w in prev.items()) + turnover = sum( + abs(newpos.get(c, 0.0) - prev.get(c, 0.0)) + for c in set(newpos) | set(prev) + ) + return realized - turnover * (cost_rt / 2) diff --git a/src/fxhnt/domain/strategies/funding_carry.py b/src/fxhnt/domain/strategies/funding_carry.py new file mode 100644 index 0000000..10f2089 --- /dev/null +++ b/src/fxhnt/domain/strategies/funding_carry.py @@ -0,0 +1,61 @@ +"""Pure delta-neutral funding-harvest math — faithful 1:1 port of foxhunt +scripts/surfer/crypto_funding_paper.py (qualify + cmd_run booking). No I/O. + +Constants and logic match the original exactly: + * HURDLE = 5e-4 open a position when trailing-30d mean daily funding (tf30) > 5 bp/day + * EXIT_HURDLE = 3e-4 hysteresis: keep a HELD coin until its tf30 falls below 3 bp/day + * LIQ_USD = 5e6 only coins with > $5M/day quote volume are in the liquid universe (in `liq`) + * COST_RT = 1e-3 10 bp round-trip; the run books realized − turnover * (COST_RT/2) + +Original `qualify(liq, tf30, prev)` (lines 95-101): loops over the *liquid* universe `liq` +(already volume-filtered + crypto-native + spot-hedgeable in `universe()`/`scan()`), keeping `c` when +its TRAILING-30d-MEAN funding `tf30[c] > HURDLE` or (`c in prev and tf30[c] > EXIT_HURDLE`). Returns +`{c: tf30[c]}`. Held coins outside the liquid universe are NOT kept (the loop is over `liq` only). + +Original booking (cmd_run lines 136-141): + realized = sum(w * last24[c] for c,w in prev.items()) # equal-weight carry on prior book (TODAY's rate) + turnover = sum(|newpos[c] - prev[c]| over union) + net = realized - turnover * (COST_RT / 2) +""" +from __future__ import annotations + +HURDLE = 5e-4 +EXIT_HURDLE = 3e-4 +LIQ_USD = 5e6 +COST_RT = 1e-3 + + +def qualify( + liq: dict[str, float], + tf30: dict[str, float], + prev: dict[str, float], + hurdle: float = HURDLE, + exit_hurdle: float = EXIT_HURDLE, +) -> dict[str, float]: + """Coins to hold today, qualified on the TRAILING-30d-MEAN funding `tf30` (NOT today's rate). + Liquidity/native/hedgeability are already enforced by `liq` (the keys of `liq`). Then hysteresis: + open new coins whose tf30 > `hurdle`, keep held coins whose tf30 is still > `exit_hurdle`. + Returns `{c: tf30[c]}` (faithful to original lines 95-101).""" + q: dict[str, float] = {} + for c in liq: + t = tf30.get(c) + if t is not None and (t > hurdle or (c in prev and t > exit_hurdle)): + q[c] = t + return q + + +def book_return( + prev: dict[str, float], + last24: dict[str, float], + newpos: dict[str, float], + cost_rt: float = COST_RT, +) -> float: + """Net realized carry: equal-weight realized funding on the PRIOR book at TODAY's booking rate + `last24`, minus the round-trip cost on turnover between the prior and new books. Faithful to + cmd_run (lines 136-141): `net = realized - turnover * (COST_RT / 2)`.""" + realized = sum(w * last24.get(c, 0.0) for c, w in prev.items()) + turnover = sum( + abs(newpos.get(c, 0.0) - prev.get(c, 0.0)) + for c in set(newpos) | set(prev) + ) + return realized - turnover * (cost_rt / 2) diff --git a/src/fxhnt/domain/strategies/growth_discipline.py b/src/fxhnt/domain/strategies/growth_discipline.py new file mode 100644 index 0000000..a470375 --- /dev/null +++ b/src/fxhnt/domain/strategies/growth_discipline.py @@ -0,0 +1,17 @@ +"""Growth-discipline blend: w·QQQ + (1−w)·multistrat book, on the intersection of their dates. Pure.""" +from __future__ import annotations + +from fxhnt.domain.strategies import multistrat + + +def daily_returns(qqq: dict[str, float], multistrat_closes: dict[str, dict[str, float]], + w_equity: float = 0.70) -> list[tuple[str, float]]: + ms = dict(multistrat.book_series(multistrat_closes)) + qdates = sorted(qqq) + # Intentional alignment choice vs foxhunt original: foxhunt samples QQQ on the book's date grid; + # this port computes QQQ returns on QQQ's own consecutive dates then intersects with the book dates. + # Identical on a clean shared trading calendar; on a data hole this drops the gap day's move rather + # than bridging it as a mislabelled 1-day return. + qret = {qdates[i]: qqq[qdates[i]] / qqq[qdates[i - 1]] - 1.0 for i in range(1, len(qdates))} + common = sorted(set(ms) & set(qret)) + return [(d, w_equity * qret[d] + (1.0 - w_equity) * ms[d]) for d in common] diff --git a/src/fxhnt/domain/strategies/multistrat.py b/src/fxhnt/domain/strategies/multistrat.py new file mode 100644 index 0000000..de9b3b2 --- /dev/null +++ b/src/fxhnt/domain/strategies/multistrat.py @@ -0,0 +1,102 @@ +"""Pure adaptive multi-strat book — faithful port of foxhunt scripts/surfer/multistrat_paper.py +(build/volnorm/trust/book_series). 6 streams (SPY/IEF/GLD/PDBC/DBMF + BTC-USD): vol-normalize each +stream -> edge-decay trust theta -> trust-weighted combination -> adaptive risk layer (EMA vol, +Kelly-floor, continuous self-recovering drawdown de-lever, z-score correlation de-risk, leverage-floor), +UNLEVERED (MAXLEV 1.0). numpy internal; no I/O; no network.""" +from __future__ import annotations + +import datetime +import math + +import numpy as np +import numpy.typing as npt + +# The exact instrument list (Yahoo tickers) the original uses, in order. +INSTRUMENTS: list[str] = ["SPY", "IEF", "GLD", "PDBC", "DBMF", "BTC-USD"] + +TARGET_VOL, MAXLEV, KELLY_FLOOR, LEV_FLOOR = 0.10, 1.0, 0.5, 0.3 +DD_DEADBAND, DD_SENS, DD_FLOOR = 0.05, 3.0, 0.40 + + +def _build(closes: dict[str, dict[str, float]]) -> tuple[list[str], npt.NDArray[np.float64]]: + """Port of build(): intersect dates across all streams, drop today's incomplete bar, return + (dates, return-matrix R[T, S]) where R[0, :] = 0.""" + dates = sorted(set.intersection(*[set(closes[s]) for s in INSTRUMENTS])) + # drop today's INCOMPLETE intraday bar if present — daily-CLOSE strategy. + today = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d") + if dates and dates[-1] == today: + dates = dates[:-1] + R = np.zeros((len(dates), len(INSTRUMENTS))) + for j, sym in enumerate(INSTRUMENTS): + s = np.array([closes[sym][d] for d in dates]) + if len(s) > 1: + R[1:, j] = s[1:] / s[:-1] - 1 + return dates, R + + +def _volnorm(col: npt.NDArray[np.float64], tv: float = TARGET_VOL, win: int = 63) -> npt.NDArray[np.float64]: + out = np.zeros_like(col) + for t in range(win, len(col)): + rv = col[t - win:t].std() * math.sqrt(252) + out[t] = col[t] * min(5.0, tv / (rv + 1e-9)) + return out + + +def _trust(col: npt.NDArray[np.float64], win: int = 126, a: float = 0.06) -> npt.NDArray[np.float64]: + th = 1.0 + out = np.ones_like(col) + for t in range(win, len(col)): + seg = col[t - win:t] + sr = seg.mean() / (seg.std() + 1e-9) * math.sqrt(252) + th = (1 - a) * th + a * float(np.clip((sr + 0.5) / 1.0, 0.1, 1.0)) + out[t] = th + return out + + +def book_series(closes: dict[str, dict[str, float]]) -> list[tuple[str, float]]: + """Faithful port of book_series(R): produce the unlevered daily book return for each booked day. + Skips the warmup window exactly as the original (book[t+1] populated only for t in [63, T-1)).""" + dates, R = _build(closes) + S = R.shape[1] + M = np.column_stack([_volnorm(R[:, j]) for j in range(S)]) + TH = np.column_stack([_trust(M[:, j]) for j in range(S)]) + tw = TH / np.maximum(TH.sum(1, keepdims=True), 1e-9) + combo = np.nansum(tw * M, axis=1) + T = len(combo) + book = np.zeros(T) + eq = 1.0 + peak = 1.0 + emu = float(np.nanmean(combo[:63])) if T >= 1 else 0.0 + evar = (float(np.nanvar(combo[:63])) if T >= 1 else 0.0) + 1e-12 + chist: list[float] = [] + for t in range(63, T - 1): + x = combo[t] + emu = 0.97 * emu + 0.03 * x + evar = 0.97 * evar + 0.03 * (x - emu) ** 2 + L = min(MAXLEV, TARGET_VOL / (math.sqrt(max(evar, 1e-12) * 252) + 1e-9)) + L = min(L, max(KELLY_FLOOR, (emu * 252) / (evar * 252 + 1e-9))) + dd = eq / peak - 1.0 + L *= float(np.clip(1 - DD_SENS * max(0.0, -dd - DD_DEADBAND), DD_FLOOR, 1.0)) + sub = np.nan_to_num(M[t - 63:t]) + vmask = sub.std(0) > 1e-12 # only streams that vary (avoid /0 in corrcoef -> NaN) + if int(vmask.sum()) >= 2: + k = int(vmask.sum()) + cm = np.corrcoef(sub[:, vmask].T) + ac = float((np.nansum(cm) - k) / (k * k - k)) + else: + ac = 0.0 + chist.append(ac) + if len(chist) > 60: + h = np.array(chist[-120:]) + z = (ac - h.mean()) / (h.std() + 1e-9) + L *= float(np.clip(1 - 0.2 * max(0.0, z), 0.5, 1.0)) + L = float(np.clip(L, LEV_FLOOR, MAXLEV)) + book[t + 1] = L * combo[t + 1] + eq *= (1 + book[t + 1]) + peak = max(peak, eq) + # emit only the booked tail (warmup days have book==0 and no date booked by the original loop): + # booked indices are t+1 for t in [63, T-1) -> [64, T-1] + out: list[tuple[str, float]] = [] + for i in range(64, T): + out.append((dates[i], float(book[i]))) + return out diff --git a/src/fxhnt/domain/strategies/sixtyforty.py b/src/fxhnt/domain/strategies/sixtyforty.py new file mode 100644 index 0000000..7607bea --- /dev/null +++ b/src/fxhnt/domain/strategies/sixtyforty.py @@ -0,0 +1,14 @@ +"""Pure 60/40 daily-rebalanced return: for each consecutive date pair, the weighted mean of per-asset +returns on adjusted closes. No I/O.""" +from __future__ import annotations + + +def daily_returns(closes: dict[str, dict[str, float]], weights: list[tuple[str, float]]) -> list[tuple[str, float]]: + syms = [s for s, _ in weights] + dates = sorted(set.intersection(*[set(closes[s]) for s in syms])) + out: list[tuple[str, float]] = [] + for i in range(1, len(dates)): + d, dprev = dates[i], dates[i - 1] + r = sum(w * (closes[s][d] / closes[s][dprev] - 1.0) for s, w in weights) + out.append((d, r)) + return out diff --git a/src/fxhnt/ports/market_data.py b/src/fxhnt/ports/market_data.py new file mode 100644 index 0000000..de7c11b --- /dev/null +++ b/src/fxhnt/ports/market_data.py @@ -0,0 +1,41 @@ +# src/fxhnt/ports/market_data.py +"""Market-data client ports for the paper-forward strategies. Each has a urllib live adapter +(adapters/data/*) and a deterministic fake in tests. No network in the domain or in tests.""" +from __future__ import annotations + +from typing import Protocol + + +class DailyBarClient(Protocol): + def adj_closes(self, symbol: str) -> dict[str, float]: + """{ 'YYYY-MM-DD': adjusted_close } for a daily-bar history window.""" + ... + + +class BinanceFundingClient(Protocol): + def scan(self, prev: set[str]) -> tuple[dict[str, float], dict[str, float], dict[str, float]]: + """(liq, tf30, last24) for the union of the hedgeable liquid universe and prev positions. + liq: {symbol: 24h_quote_volume_usd} for crypto-native, spot-hedgeable USDT perps with vol>LIQ_USD. + tf30: {symbol: 30d-mean funding × INTERVALS_PER_DAY} (daily rate; only symbols with >=30 history). + last24: {symbol: sum of the most recent INTERVALS_PER_DAY funding intervals} (today's booking rate).""" + ... + + +class CrossVenueFundingClient(Protocol): + def snapshot(self) -> tuple[dict[str, dict[str, float]], dict[str, dict[str, float]]]: + """One fetch per run → (funding_by_venue, volume_by_venue), both from the same snapshot. + funding_by_venue: { coin: { venue: daily_funding_rate } } across the supported venues. + volume_by_venue: { coin: { venue: 24h_quote_volume_usd } }.""" + ... + + +class CryptoPerpBarClient(Protocol): + def panel(self) -> tuple[list[str], list[int], list[list[float]], list[list[float]], list[float]]: + """(symbols, epoch_days, close[d][s], open[d][s], funding_now[s]) for the top-liquid perps.""" + ... + + +class VolIndexClient(Protocol): + def dvol(self, currency: str) -> dict[int, float]: + """{ epoch_day: implied-vol-index } for BTC/ETH (Deribit DVOL).""" + ... diff --git a/src/fxhnt/vendor/__init__.py b/src/fxhnt/vendor/__init__.py new file mode 100644 index 0000000..c9c1141 --- /dev/null +++ b/src/fxhnt/vendor/__init__.py @@ -0,0 +1,5 @@ +"""Vendored third-party / sibling-repo code, copied verbatim with only import/path edits. + +These modules are NOT maintained here; they are faithful copies of their upstream source so the +behaviour is identical. See each subpackage's __init__ for provenance and the exact edits made. +""" diff --git a/src/fxhnt/vendor/surfer/__init__.py b/src/fxhnt/vendor/surfer/__init__.py new file mode 100644 index 0000000..3ca4a4b --- /dev/null +++ b/src/fxhnt/vendor/surfer/__init__.py @@ -0,0 +1,16 @@ +"""Vendored surfer PoC (crypto cross-sectional momentum + tail-managed VRP sleeve). + +Provenance: copied verbatim from foxhunt `scripts/surfer/{signal_sweep,pit_sweep,surfer_poc}.py`. + +ONLY edits applied (no algorithm / math / constant changes): + * `sys.path.insert(...)` + bare sibling imports rewritten as package-relative + (`from fxhnt.vendor.surfer.signal_sweep import ...`, `from fxhnt.vendor.surfer import pit_sweep`). + * data root made configurable via `$FXHNT_SURFER_DATA_DIR` (default `/data/surfer`): + `pit_sweep.load()` reads `$FXHNT_SURFER_DATA_DIR/crypto_pit/*.npz`; `surfer_poc` resolves + `STATE` and the dvol cache under `$FXHNT_SURFER_DATA_DIR`. + * `surfer_poc.paper_step(cfg, state)` added: a value-returning refactor of `paper()` for the + fxhnt ForwardTracker (same mark / weekly-rebalance / VRP-scaling math, no file IO / printing). + +torch is imported at module level (used in `signal_sweep.sharpe_t`/`validate` and `surfer_poc.backtest`) +and is kept — this is a faithful port, not a numpy reimplementation. +""" diff --git a/src/fxhnt/vendor/surfer/pit_sweep.py b/src/fxhnt/vendor/surfer/pit_sweep.py new file mode 100644 index 0000000..6a695b9 --- /dev/null +++ b/src/fxhnt/vendor/surfer/pit_sweep.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +"""Point-in-time survivorship test for crypto cross-sectional momentum. + +Universe is rebuilt EACH DAY: among coins with active trailing-30d dollar-volume (>0), +take the top-K by that volume. Dead coins (LUNA, SRM, ...) are IN the cross-section while +they trade and DROP OUT (after carrying their crash) when volume dies. No lookahead: +volume & momentum use only past data; weights apply to next-day return. If momentum's +edge survives here — especially 2022 with LUNA included — it is real beyond the bootstrap proxy. +""" +import glob +import math +import os + +import numpy as np + +from fxhnt.vendor.surfer.signal_sweep import xs_weights, pnl_w, validate, sharpe_t # noqa: E402,F401 +import torch # noqa: E402 + +DEV = "cuda" if torch.cuda.is_available() else "cpu" + + +def load(): + syms, data = [], {} + _base = os.environ.get("FXHNT_SURFER_DATA_DIR", "/data/surfer") # crypto_pit lives at $FXHNT_SURFER_DATA_DIR/crypto_pit + for p in sorted(glob.glob(os.path.join(_base, "crypto_pit/*.npz"))): + d = np.load(p); s = p.split("/")[-1][:-4] + data[s] = (d["day"].astype(np.int64), d["close"].astype(float), d["qvol"].astype(float), d["funding"].astype(float)) + syms.append(s) + syms = sorted(syms) + days = np.array(sorted(set().union(*[set(data[s][0].tolist()) for s in syms]))) + di = {int(v): i for i, v in enumerate(days.tolist())} + T, N = len(days), len(syms) + close = np.full((T, N), np.nan); qv = np.full((T, N), np.nan); fund = np.full((T, N), np.nan) + for j, s in enumerate(syms): + dd, cc, vv, ff = data[s] + for k in range(len(dd)): + r = di[int(dd[k])]; close[r, j] = cc[k]; qv[r, j] = vv[k]; fund[r, j] = ff[k] + return syms, days, close, qv, fund + + +def trailing(lc, L): + out = np.full_like(lc, np.nan); out[L:] = lc[L:] - lc[:-L]; return out + + +def main(): + syms, days, close, qv, fund = load() + T, N = close.shape + lc = np.log(close) + R = np.zeros((T, N)); R[1:] = lc[1:] - lc[:-1]; R = np.where(np.isfinite(R), R, 0.0) + Reff = R - np.where(np.isfinite(fund), fund, 0.0) + qv = np.nan_to_num(qv) + dv30 = np.full((T, N), 0.0) # trailing 30d mean dollar-volume + for t in range(30, T): + dv30[t] = qv[t - 30:t].mean(axis=0) + dead = sum(1 for j in range(N) if qv[-30:, j].mean() == 0) + year = (1970 + days / 365.25).astype(int) + + print(f"\n===== POINT-IN-TIME MOMENTUM — {N} coins ({dead} dead/delisted), {T}d =====") + for TOPK in [30, 50]: + # daily universe = top-K by trailing dollar-vol among actively-trading coins + univ = np.zeros((T, N), bool) + for t in range(T): + elig = np.where((dv30[t] > 0) & np.isfinite(close[t]))[0] + if len(elig) == 0: + continue + k = min(TOPK, len(elig)) + top = elig[np.argsort(-dv30[t, elig])[:k]] + univ[t, top] = True + print(f"\n--- TOPK={TOPK} (avg coins/day in-universe = {univ.sum(1).mean():.0f}) ---") + print(f"{'lookback':>9} {'full':>6} {'IS':>6} {'OOS':>6} {'CPCVmed':>8} {'DSR':>5} | per-year (2020..2026)") + for L in [10, 20, 30]: + sig = trailing(lc, L).copy() + sig[~univ] = np.nan # restrict signal to PIT universe + w = xs_weights(sig) + pnl = pnl_w(w, Reff, cost_bp=10) + v = validate(pnl, days, 6) + py = [] + for y in range(2020, 2027): + m = year[1:] == y + py.append(sharpe_t(torch.tensor(pnl[m], device=DEV, dtype=torch.float64)) if m.sum() > 30 else float("nan")) + pys = " ".join(f"{p:>+5.2f}" if not math.isnan(p) else " n/a" for p in py) + print(f"{L:>9} {v['full']:>+6.2f} {v['is_']:>+6.2f} {v['oos']:>+6.2f} {v['med']:>+8.2f} {v['dsr']:>5.2f} | {pys}") + print("\nVERDICT: if mom_20-30 stays positive (esp. 2022 with LUNA in-universe) => survivorship-robust REAL edge.") + + +if __name__ == "__main__": + main() diff --git a/src/fxhnt/vendor/surfer/signal_sweep.py b/src/fxhnt/vendor/surfer/signal_sweep.py new file mode 100644 index 0000000..db94869 --- /dev/null +++ b/src/fxhnt/vendor/surfer/signal_sweep.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Deflated signal sweep — a factor zoo on data in hand, judged against the FULL search. + +Wisdom: breadth without multiple-testing control is the fastest way to fool yourself. +So every signal's Deflated Sharpe is deflated by N_trials = the number of signals tested. +Batch 1 (instant, free, data already downloaded): 22-futures daily panel — cross-sectional +& time-series momentum, short-term reversal, low-vol — plus structural SEASONALITY +(overnight-drift, intraday, day-of-week). Each signal → daily PnL → full/IS/OOS Sharpe, +CPCV(45-path) median+5th-pct, Deflated Sharpe. Ranked by OOS. Survivor = DSR>0.95 AND +OOS>0 AND CPCV-median>0 (consistent + multiple-testing-honest). + +Validation on GPU (torch); panel/signal staging on CPU (small). Roll-zeroed close-close +returns (screening; survivors get proper roll handling in deeper validation). +""" +import glob +import math +import re +from itertools import combinations +from statistics import NormalDist + +import numpy as np +import torch + +ND = NormalDist() +DEV = "cuda" if torch.cuda.is_available() else "cpu" +MONTHS = "FGHJKMNQUVXZ" +SPLIT_DAY = 19723 # 2024-01-01 in days-since-epoch + + +def load_panel(): + import databento as db + series = {} + for p in sorted(glob.glob("data/surfer/*.dbn")): + root = p.split("/")[-1][:-4] + pat = re.compile("^" + re.escape(root) + "[" + MONTHS + r"]\d{1,2}$") + try: + df = db.DBNStore.from_file(p).to_df().reset_index() + except Exception: + continue + if df.empty or "close" not in df.columns: + continue + df = df[df["close"] > 0] + df = df[df["symbol"].astype(str).str.match(pat)] + if df.empty: + continue + df["day"] = (df["ts_event"].astype("int64") // (86_400 * 10**9)) + idx = df.groupby("day")["volume"].idxmax() + f = df.loc[idx, ["day", "instrument_id", "open", "close"]].sort_values("day") + series[root] = (f["day"].to_numpy(np.int64), f["instrument_id"].to_numpy(np.int64), + f["open"].to_numpy(np.float64), f["close"].to_numpy(np.float64)) + roots = sorted(series) + days = np.array(sorted(set().union(*[set(series[r][0].tolist()) for r in roots]))) + di = {d: i for i, d in enumerate(days.tolist())} + T, N = len(days), len(roots) + close = np.full((T, N), np.nan); op = np.full((T, N), np.nan); inst = np.full((T, N), -1, np.int64) + for j, r in enumerate(roots): + d, ii, oo, cc = series[r] + for k in range(len(d)): + row = di[int(d[k])]; close[row, j] = cc[k]; op[row, j] = oo[k]; inst[row, j] = ii[k] + weekday = (days + 3) % 7 # epoch day0 = Thursday → 0=Mon..6=Sun + return roots, days, close, op, inst, weekday + + +def build_returns(close, op, inst): + T, N = close.shape + lc = np.log(close) + R = np.zeros((T, N)); R[1:] = lc[1:] - lc[:-1] + same = np.zeros((T, N), bool); same[1:] = inst[1:] == inst[:-1] + R = np.where(same & np.isfinite(R), R, 0.0) + ov = np.zeros((T, N)); ov[1:] = np.log(op[1:] / close[:-1]) + ov = np.where(same & np.isfinite(ov), ov, 0.0) + intr = np.log(close / op); intr = np.where(np.isfinite(intr), intr, 0.0) + return lc, R, ov, intr + + +def trailing(lc, L): + out = np.full_like(lc, np.nan); out[L:] = lc[L:] - lc[:-L]; return out + + +def xs_weights(sig): # cross-sectional z-score, market-neutral, unit gross + mu = np.nanmean(sig, axis=1, keepdims=True); sd = np.nanstd(sig, axis=1, keepdims=True) + z = np.nan_to_num((sig - mu) / np.where(sd > 0, sd, 1)) + g = np.sum(np.abs(z), axis=1, keepdims=True); g[g == 0] = 1 + return z / g + + +def pnl_w(w, R, cost_bp=2.0): + p = np.sum(w[:-1] * R[1:], axis=1) + turn = np.sum(np.abs(w[1:] - w[:-1]), axis=1) + return p - turn * (cost_bp / 1e4) + + +def sharpe_t(x): + x = x[torch.isfinite(x)] + if len(x) < 10 or float(x.std()) == 0: + return float("nan") + return float(x.mean() / x.std() * math.sqrt(252)) + + +def validate(pnl, days, n_trials): + x = torch.tensor(np.asarray(pnl), device=DEV, dtype=torch.float64) + n = len(x); full = sharpe_t(x) + dd = days[-n:] + ism = torch.tensor(dd < SPLIT_DAY, device=DEV) + sis = sharpe_t(x[ism]) + soos = sharpe_t(x[~ism]) if int((~ism).sum()) > 30 else float("nan") + nb, k = 10, 2 + blk = [torch.arange(i * n // nb, (i + 1) * n // nb, device=DEV) for i in range(nb)] + oos = [sharpe_t(x[torch.cat([blk[b] for b in c])]) for c in combinations(range(nb), k)] + oos = np.array([o for o in oos if not math.isnan(o)]) + med = float(np.median(oos)) if len(oos) else float("nan") + p5 = float(np.percentile(oos, 5)) if len(oos) else float("nan") + srd = full / math.sqrt(252) + if n_trials > 1: + sr0 = (1 / math.sqrt(n)) * ((1 - 0.5772) * ND.inv_cdf(1 - 1.0 / n_trials) + + 0.5772 * ND.inv_cdf(1 - 1.0 / (n_trials * math.e))) + else: + sr0 = 0.0 + dsr = ND.cdf((srd - sr0) * math.sqrt(n - 1)) if not math.isnan(full) else float("nan") + return dict(full=full, is_=sis, oos=soos, med=med, p5=p5, dsr=dsr) + + +def main(): + roots, days, close, op, inst, weekday = load_panel() + lc, R, ov, intr = build_returns(close, op, inst) + T, N = close.shape + vol63 = np.full_like(lc, np.nan) + for t in range(63, T): + vol63[t] = np.nanstd(R[t - 63:t], axis=0) + invvol = 1.0 / np.where(vol63 > 0, vol63, np.nan) + + sig = {} + for L in [21, 63, 126, 252]: + sig[f"XS_mom_{L}"] = pnl_w(xs_weights(trailing(lc, L)), R) + for L in [1, 5]: + sig[f"XS_rev_{L}"] = pnl_w(xs_weights(-trailing(lc, L)), R) + sig["XS_lowvol"] = pnl_w(xs_weights(-vol63), R) + for L in [21, 63, 252]: + w = np.nan_to_num(np.sign(trailing(lc, L)) * invvol) + g = np.sum(np.abs(w), axis=1, keepdims=True); g[g == 0] = 1 + sig[f"TS_mom_{L}"] = pnl_w(w / g, R) + sig["SEAS_overnight"] = np.nanmean(ov[1:], axis=1) - 1.0 / 1e4 # EW long overnight, ~1bp cost + sig["SEAS_intraday"] = np.nanmean(intr[1:], axis=1) - 1.0 / 1e4 # EW long intraday + ewR = np.nanmean(R, axis=1) + for d, nm in enumerate(["Mon", "Tue", "Wed", "Thu", "Fri"]): + sig[f"SEAS_{nm}"] = np.where(weekday[1:] == d, ewR[1:], 0.0) + + NT = len(sig) + rows = [(nm, validate(p, days, NT)) for nm, p in sig.items()] + rows.sort(key=lambda r: -(r[1]["oos"] if not math.isnan(r[1]["oos"]) else -9)) + print(f"\n===== DEFLATED SIGNAL SWEEP — Batch 1 (22 futures daily + seasonality) =====") + print(f"device={DEV} roots={len(roots)} days={T} N_trials(deflation)={NT}") + print(f"{'signal':>16} {'full':>7} {'IS':>7} {'OOS':>7} {'CPCVmed':>8} {'CPCV5%':>8} {'DSR':>6}") + for nm, v in rows: + print(f"{nm:>16} {v['full']:>+7.2f} {v['is_']:>+7.2f} {v['oos']:>+7.2f} " + f"{v['med']:>+8.2f} {v['p5']:>+8.2f} {v['dsr']:>6.2f}") + surv = [nm for nm, v in rows if v["dsr"] > 0.95 and v["oos"] > 0 and v["med"] > 0] + print(f"\nSURVIVORS (DSR>0.95 & OOS>0 & CPCVmed>0): {surv if surv else 'NONE'}") + print("DSR is deflated by N_trials → multiple-testing-honest. Survivors get deeper validation + crypto/COT batches.") + + +if __name__ == "__main__": + main() diff --git a/src/fxhnt/vendor/surfer/surfer_poc.py b/src/fxhnt/vendor/surfer/surfer_poc.py new file mode 100644 index 0000000..7686790 --- /dev/null +++ b/src/fxhnt/vendor/surfer/surfer_poc.py @@ -0,0 +1,408 @@ +#!/usr/bin/env python3 +"""Surfer PoC — crypto cross-sectional momentum (the validated edge), end-to-end. + +Modes: + backtest — run the strategy over the cached point-in-time history; print honest stats + (gross/net Sharpe at AUM, OOS, CPCV-median, DSR, per-year, max-DD, turnover, capacity curve) + regime — diagnostic: does any regime feature predict when momentum works? (overlay is OFF by default) + paper — forward paper-trade: fetch live data, compute today's target weights, log intended trades, + mark the paper book (price + funding), persist state. The only true out-of-sample test. + status — print the paper book + equity curve summary. + +Design: ONE signal function (`compute_weights`) drives BOTH backtest and live (no backtest/live skew). +No lookahead (signal uses closed bars <= t; weights apply to t+1; rebalance keyed on calendar day%K). +Funding is P&L (longs pay, shorts earn). Realistic sqrt-impact cost. Validated edge only; regime overlay +stays a diagnostic until separately confirmed. +""" +import argparse +import json +import math +import os +import time +import urllib.request + +import numpy as np + +from fxhnt.vendor.surfer.signal_sweep import xs_weights, validate, sharpe_t # noqa: E402,F401 +from fxhnt.vendor.surfer import pit_sweep # noqa: E402 (load() of the cached PIT universe) +import torch # noqa: E402 + +DEV = "cuda" if torch.cuda.is_available() else "cpu" +DAY_MS = 86_400_000 + + +def _data_dir(): # resolved per-call so $FXHNT_SURFER_DATA_DIR is honored at runtime (default /data/surfer) + return os.environ.get("FXHNT_SURFER_DATA_DIR", "/data/surfer") + + +STATE = os.path.join(_data_dir(), "poc_state.json") # CLI default; paper_step is value-returning (no file IO) + +CFG = dict( + lookback=20, # XS momentum horizon (days) + beta_win=60, # rolling window for market-beta (residual momentum strips this) + topk=30, # universe = top-K liquid by trailing dollar-vol (breadth/liquidity optimum) + rebal_k=7, # rebalance every 7 calendar days (weekly) + smooth_span=5, # EWMA weight-smoothing span (cuts turnover + whipsaw) + vol_win=30, # trailing vol window (for sizing / regime) + cost_aum=2e7, # AUM the slippage model is priced at ($20M) + eta=1.0, # sqrt market-impact coefficient (conservative) + dd_breaker=-0.20, # cut gross exposure if rolling drawdown worse than this + vrp_risk_frac=0.30, # VRP diversifier sleeve risk budget (small — short-vol tail) + vrp_rise_k=5, # don't sell vol when DVOL rose over last K days (tail gate) + vrp_level_win=60, # sell full size only when DVOL < trailing-win median +) + +# ---------------------------------------------------------------- helpers +def roll(fn, X, L): + out = np.full_like(X, np.nan) + for t in range(L, len(X)): + out[t] = fn(X[t - L:t], axis=0) + return out + + +def trailing(lc, L): + out = np.full_like(lc, np.nan); out[L:] = lc[L:] - lc[:-L]; return out + + +def ewma_rows(W, span): + if span <= 1: + return W + a = 2.0 / (span + 1) + out = W.copy() + for t in range(1, len(W)): + out[t] = a * W[t] + (1 - a) * out[t - 1] + return out + + +# ---------------------------------------------------------------- SHARED SIGNAL (backtest == live) +def compute_weights(close, qvol, days, cfg): + """Panel -> held target weights [T,N] (market-neutral, weekly, smoothed). No lookahead. + Live takes the last row; backtest uses all rows. Identical code path.""" + T, N = close.shape + lc = np.log(close) + dv = roll(np.mean, np.nan_to_num(qvol), 30) + univ = np.zeros((T, N), bool) + for t in range(T): + elig = np.where((dv[t] > 0) & np.isfinite(close[t]))[0] + if len(elig): + univ[t, elig[np.argsort(-dv[t, elig])[:cfg["topk"]]]] = True + # residual (beta-stripped) momentum: strip each coin's beta to the equal-weight market, + # rank the residual trend (Phase-1: +0.72 vs +0.57 raw — cleaner base edge). No lookahead. + R = np.zeros((T, N)); R[1:] = lc[1:] - lc[:-1]; R = np.where(np.isfinite(R), R, 0.0) + mkt = np.array([R[t][univ[t]].mean() if univ[t].any() else 0.0 for t in range(T)]) + Wb = cfg["beta_win"]; beta = np.zeros((T, N)) + for t in range(Wb, T): + mw = mkt[t - Wb:t]; vb = mw.var() + 1e-12 + beta[t] = ((R[t - Wb:t] * mw[:, None]).mean(0) - R[t - Wb:t].mean(0) * mw.mean()) / vb + rcum = np.cumsum(R - beta * mkt[:, None], axis=0) + L = cfg["lookback"] + sig = np.full((T, N), np.nan); sig[L:] = rcum[L:] - rcum[:-L] + sig[~univ] = np.nan + wt = xs_weights(sig) # daily target (market-neutral, unit gross) + wt = ewma_rows(wt, cfg["smooth_span"]) # smooth + held = wt.copy() # weekly rebalance on FIXED calendar phase (day%K==0) + K = cfg["rebal_k"] # epoch day0=Thu → rebalances every Thursday, live-consistent + last = 0 + for t in range(T): + if days[t] % K == 0: + last = t + held[t] = wt[last] + return held, univ + + +def slippage_net(w, Reff, dv, vol, days, cfg): + """Net daily PnL series under sqrt market-impact + liquidity-scaled spread at cfg['cost_aum'].""" + AUM = cfg["cost_aum"] + turn = np.abs(w[1:] - w[:-1]) + adv = np.nan_to_num(dv[1:]); sg = np.nan_to_num(vol[1:]) + hs = np.clip(30.0 / np.sqrt(np.maximum(adv, 1.0) / 1e6), 1.0, 30.0) / 1e4 + part = np.where(adv > 0, turn * AUM / adv, 0.0) + cost = np.sum(turn * (hs + cfg["eta"] * sg * np.sqrt(np.clip(part, 0, None))), axis=1) + return np.sum(w[:-1] * Reff[1:], axis=1) - cost + + +def max_drawdown(pnl): + eq = np.cumsum(np.nan_to_num(pnl)); peak = np.maximum.accumulate(eq) + return float((eq - peak).min()) + + +# ---------------------------------------------------------------- VRP SLEEVE (tail-managed short-vol diversifier) +def _dvol(ccy, refresh): + cache = os.path.join(_data_dir(), "dvol", f"{ccy}.json") + d = {int(k): v for k, v in json.load(open(cache)).items()} if os.path.exists(cache) else {} + if refresh: + end = int(time.time() * 1000); start = end - 200 * DAY_MS + try: + rows = _get(f"https://www.deribit.com/api/v2/public/get_volatility_index_data?currency={ccy}" + f"&start_timestamp={start}&end_timestamp={end}&resolution=86400").get("result", {}).get("data", []) + for r in rows: + d[int(r[0]) // DAY_MS] = float(r[4]) + os.makedirs(os.path.dirname(cache), exist_ok=True) + json.dump({str(k): v for k, v in d.items()}, open(cache, "w")) + except Exception: + pass + return d + + +def vrp_ccy(iv, close, cfg): + """Tail-managed short-variance daily P&L per ccy. Gate uses DVOL up to t-1 (no lookahead).""" + days = sorted(set(iv) & set(close)); pnl = {} + w0 = max(cfg["vrp_level_win"], cfg["vrp_rise_k"]) + 1 + for i in range(w0, len(days)): + d0, d1 = days[i - 1], days[i] + if close[d0] <= 0 or close[d1] <= 0: + continue + r = math.log(close[d1] / close[d0]); ivl = iv[d0] / 100.0 + raw = ivl * ivl / 365.0 - r * r # collect implied var, pay realized + rising = iv[d0] > iv[days[i - 1 - cfg["vrp_rise_k"]]] # don't sell into rising vol + win = [iv[days[j]] for j in range(i - 1 - cfg["vrp_level_win"], i - 1)] + level = 1.0 if iv[d0] < float(np.median(win)) else 0.5 # full in calm, half elevated + pnl[d1] = (0.0 if rising else 1.0) * level * raw + return pnl + + +def vrp_book_series(cfg, btc_close, eth_close, refresh): + b = vrp_ccy(_dvol("BTC", refresh), btc_close, cfg) + e = vrp_ccy(_dvol("ETH", refresh), eth_close, cfg) + days = sorted(set(b) | set(e)) + return {d: float(np.nanmean([b.get(d, np.nan), e.get(d, np.nan)])) for d in days} + + +def _closes_from(syms, days, close, sym): + if sym not in syms: + return {} + j = syms.index(sym) + return {int(days[t]): float(close[t, j]) for t in range(len(days)) if np.isfinite(close[t, j])} + + +def _vrp_calib(cfg): + """From cached history: (momentum daily vol, VRP daily vol) — to size VRP to its risk budget.""" + syms, days, close, qv, fund = pit_sweep.load() + R = np.zeros_like(close); R[1:] = np.log(close)[1:] - np.log(close)[:-1]; R = np.where(np.isfinite(R), R, 0.0) + cf = np.where(np.isfinite(fund), fund, 0.0) + w, _ = compute_weights(close, qv, days, cfg) + mom = np.sum(w[:-1] * (R - cf)[1:], axis=1) + vrpb = vrp_book_series(cfg, _closes_from(syms, days, close, "BTCUSDT"), + _closes_from(syms, days, close, "ETHUSDT"), refresh=False) + return float(np.nanstd(mom)), float(np.nanstd(np.array(list(vrpb.values())))) + + +# ---------------------------------------------------------------- BACKTEST +def backtest(cfg): + syms, days, close, qv, fund = pit_sweep.load() + T, N = close.shape + R = np.zeros((T, N)); R[1:] = np.log(close)[1:] - np.log(close)[:-1] + R = np.where(np.isfinite(R), R, 0.0) + fund = np.where(np.isfinite(fund), fund, 0.0) + tradeable = np.ones((T, N), bool) + for j in range(N): + idx = np.where(np.isfinite(close[:, j]))[0] + if len(idx): + tradeable[max(0, idx[-1] - 4):idx[-1] + 1, j] = False + Reff = np.where(tradeable, R - fund, 0.0) + dv = roll(np.mean, np.nan_to_num(qv), 30) + vol = roll(np.std, R, cfg["vol_win"]) + w, univ = compute_weights(close, qv, days, cfg) + gross = np.sum(w[:-1] * Reff[1:], axis=1) + net = slippage_net(w, Reff, dv, vol, days, cfg) + year = (1970 + days / 365.25).astype(int) + v = validate(net, days, 30) + gsr = sharpe_t(torch.tensor(gross, device=DEV, dtype=torch.float64)) + turn = float(np.mean(np.sum(np.abs(w[1:] - w[:-1]), axis=1))) * 100 + print(f"\n===== SURFER PoC BACKTEST — XS momentum (top{cfg['topk']}, {cfg['lookback']}d, weekly, smooth{cfg['smooth_span']}) =====") + print(f"coins(ever)={N} days={T} universe/day~{int(univ.sum(1).mean())} cost-AUM=${cfg['cost_aum']/1e6:.0f}M turnover/day={turn:.1f}%") + print(f"gross Sharpe {gsr:+.2f} | NET Sharpe {v['full']:+.2f} IS {v['is_']:+.2f} OOS {v['oos']:+.2f} CPCVmed {v['med']:+.2f} DSR {v['dsr']:.2f}") + print(f"max-DD (sum-rets) {max_drawdown(net):+.2f}") + py = [sharpe_t(torch.tensor(net[year[1:] == y], device=DEV, dtype=torch.float64)) if (year[1:] == y).sum() > 30 else float('nan') for y in range(2020, 2027)] + print("per-year 2020..26: " + " ".join(f"{p:+.2f}" if not math.isnan(p) else " n/a" for p in py)) + print("capacity: " + " ".join( + f"${a/1e6:.0f}M={sharpe_t(torch.tensor(slippage_net(w,Reff,dv,vol,days,{**cfg,'cost_aum':a}),device=DEV,dtype=torch.float64)):+.2f}" + for a in [1e6, 5e6, 2e7, 5e7, 2e8])) + # ---- VRP diversifier sleeve + combined two-sleeve book ---- + vrpb = vrp_book_series(cfg, _closes_from(syms, days, close, "BTCUSDT"), + _closes_from(syms, days, close, "ETHUSDT"), refresh=False) + mom_by = {int(days[1:][i]): net[i] for i in range(len(net))} + common = sorted(set(vrpb) & set(mom_by)) + if common: + mo = np.array([mom_by[d] for d in common]); vr = np.array([vrpb[d] for d in common]) + mu, vu, f = np.nanstd(mo), np.nanstd(vr), cfg["vrp_risk_frac"] + comb = (mo / mu) * (1 - f) + (vr / vu) * f + cy = (1970 + np.array(common) / 365.25).astype(int) + T = lambda x: torch.tensor(x, device=DEV, dtype=torch.float64) + print(f"VRP sleeve (tail-managed) Sharpe {sharpe_t(T(vr)):+.2f} corr-to-momentum {np.corrcoef(mo,vr)[0,1]:+.2f}") + print(f"COMBINED BOOK (momentum {int((1-f)*100)}% + VRP {int(f*100)}% risk) Sharpe {sharpe_t(T(comb)):+.2f}") + print("combined per-year: " + " ".join( + f"{y}:{sharpe_t(T(comb[cy==y])):+.2f}" for y in range(2021, 2027) if (cy == y).sum() > 30)) + + +# ---------------------------------------------------------------- REGIME DIAGNOSTIC (overlay default OFF) +def regime(cfg): + syms, days, close, qv, fund = pit_sweep.load() + T, N = close.shape + R = np.zeros((T, N)); R[1:] = np.log(close)[1:] - np.log(close)[:-1]; R = np.where(np.isfinite(R), R, 0.0) + fund = np.where(np.isfinite(fund), fund, 0.0) + w, univ = compute_weights(close, qv, days, cfg) + book = np.sum(w[:-1] * (np.where(np.isfinite(R), R, 0.0) - fund)[1:], axis=1) # daily book return + lc = np.log(close) + disp = np.array([np.nanstd(trailing(lc, cfg["lookback"])[t][univ[t]]) if univ[t].any() else np.nan for t in range(T)]) + mvol = np.nanmedian(roll(np.std, R, cfg["vol_win"]), axis=1) + mtrend = np.array([np.nanmedian(np.abs(trailing(lc, cfg["lookback"])[t][univ[t]])) if univ[t].any() else np.nan for t in range(T)]) + feats = {"xs_dispersion": disp[:-1], "median_vol": mvol[:-1], "abs_trend": mtrend[:-1]} + split = int(0.7 * len(book)) + print("\n===== REGIME DIAGNOSTIC — does a feature predict next-day book return? (overlay OFF until confirmed) =====") + print(f"{'feature':>16} {'IC_all':>7} {'IC_IS':>7} {'IC_OOS':>7}") + for nm, f in feats.items(): + f = f[1:]; b = book[1:] # feature_t -> book return_{t+1} + m = np.isfinite(f) & np.isfinite(b) + def ic(mask): + mm = m & mask + return float(np.corrcoef(f[mm], b[mm])[0, 1]) if mm.sum() > 50 else float("nan") + idx = np.arange(len(f)) + print(f"{nm:>16} {ic(np.ones_like(m)):>+7.3f} {ic(idx < split):>+7.3f} {ic(idx >= split):>+7.3f}") + print("Significant + IS/OOS-consistent IC => regime sizing worth building. Otherwise momentum runs flat-sized.") + + +# ---------------------------------------------------------------- LIVE (paper) +def _get(u): + return json.load(urllib.request.urlopen(urllib.request.Request(u, headers={"User-Agent": "curl/8"}), timeout=30)) + + +def _live_panel(cfg): + info = _get("https://fapi.binance.com/fapi/v1/exchangeInfo") + perps = {s["symbol"] for s in info["symbols"] if s.get("contractType") == "PERPETUAL" + and s.get("quoteAsset") == "USDT" and s.get("status") == "TRADING"} + tick = _get("https://fapi.binance.com/fapi/v1/ticker/24hr") + vol = {t["symbol"]: float(t["quoteVolume"]) for t in tick if t["symbol"] in perps and t["symbol"].isascii()} + syms = sorted(vol, key=lambda s: -vol[s])[:max(cfg["topk"] * 2, 60)] # wide net; signal picks top-K + need = max(cfg["lookback"] + cfg["vol_win"], cfg["vrp_level_win"] + cfg["vrp_rise_k"], + cfg["beta_win"] + cfg["lookback"]) + 12 # cover VRP gate + residual-momentum beta window + closes, qvols, funds, keep = {}, {}, {}, [] + for s in syms: + k = _get(f"https://fapi.binance.com/fapi/v1/klines?symbol={s}&interval=1d&limit={need}") + if len(k) < need - 2: + continue + d = np.array([int(r[0]) // DAY_MS for r in k]) + closes[s] = (d, np.array([float(r[4]) for r in k]), np.array([float(r[7]) for r in k])) + fr = _get(f"https://fapi.binance.com/fapi/v1/fundingRate?symbol={s}&limit=10") + funds[s] = float(fr[-1]["fundingRate"]) * 3 if fr else 0.0 # ~daily funding (3x 8h) + keep.append(s) + time.sleep(0.05) + days = np.array(sorted(set().union(*[set(closes[s][0].tolist()) for s in keep]))) + di = {int(v): i for i, v in enumerate(days.tolist())} + T, N = len(days), len(keep) + close = np.full((T, N), np.nan); qv = np.full((T, N), np.nan) + for j, s in enumerate(keep): + d, c, q = closes[s] + for i in range(len(d)): + close[di[int(d[i])], j] = c[i]; qv[di[int(d[i])], j] = q[i] + return keep, days, close, qv, np.array([funds[s] for s in keep]) + + +def _load_state(): + if os.path.exists(STATE): + return json.load(open(STATE)) + return dict(inception=None, last_mark_day=None, positions={}, prices={}, equity=1.0, log=[]) + + +def _save_state(st): + tmp = STATE + ".tmp" + json.dump(st, open(tmp, "w"), indent=1) + os.replace(tmp, STATE) + + +def paper_step(cfg, state): + """Value-returning core of `paper()` for the fxhnt ForwardTracker — IDENTICAL mark/rebalance/VRP math, + but takes the prior working `state` dict in and returns `(comb_ret, today_epoch, new_state, info)` instead + of loading/saving a file and printing. `comb_ret` is the booked combined return for `today` (0.0 when no + mark happened — first run / already marked today). `info` carries (rebal, trades, vrp_today, target) for an + optional caller-side print. The returned mutated `state` is exactly what `paper()` would have persisted.""" + keep, days, close, qv, fund_now = _live_panel(cfg) + w, univ = compute_weights(close, qv, days, cfg) + target = {keep[j]: float(w[-1, j]) for j in range(len(keep)) if abs(w[-1, j]) > 1e-6} + last_close = {keep[j]: float(close[-1, j]) for j in range(len(keep)) if np.isfinite(close[-1, j])} + today = int(days[-1]) + st = state + if st["inception"] is None: + st["inception"] = today + st.setdefault("vrp_equity", 1.0); st.setdefault("combined_equity", 1.0) + if "vrp_vu" not in st: + try: + st["vrp_mu"], st["vrp_vu"] = _vrp_calib(cfg) + except Exception: + st["vrp_mu"], st["vrp_vu"] = 0.0, 1.0 + vrpb = vrp_book_series(cfg, _closes_from(keep, days, close, "BTCUSDT"), + _closes_from(keep, days, close, "ETHUSDT"), refresh=True) + vrp_today = vrpb.get(today, None) + comb_ret = 0.0 + # 1) MARK both sleeves to latest prices (+ funding) since last mark + if st["positions"] and st["last_mark_day"] != today: + ret = 0.0 + for s, pos in st["positions"].items(): + p0 = st["prices"].get(s); p1 = last_close.get(s) + if p0 and p1 and p0 > 0: + ret += pos * (p1 / p0 - 1.0) + ret -= pos * fund_now[keep.index(s)] if s in keep else 0.0 # long pays funding + f = cfg["vrp_risk_frac"] + scaled = (vrp_today / st["vrp_vu"]) * st["vrp_mu"] if (vrp_today is not None and st["vrp_vu"] > 0) else 0.0 + comb_ret = (1 - f) * ret + f * scaled # VRP scaled to momentum vol, f risk + st["equity"] *= (1.0 + ret) + st["vrp_equity"] *= (1.0 + scaled) + st["combined_equity"] *= (1.0 + comb_ret) + st["log"].append({"day": today, "mom_ret": round(ret, 6), "vrp_ret": round(scaled, 6), + "comb_ret": round(comb_ret, 6), "combined": round(st["combined_equity"], 5)}) + # 2) REBALANCE on the weekly boundary + rebal = (st["last_mark_day"] is None) or (today % cfg["rebal_k"] == 0) # fixed weekly phase (Thursdays) + trades = [] + if rebal: + cur = st["positions"] + allk = set(cur) | set(target) + for s in sorted(allk): + d = target.get(s, 0.0) - cur.get(s, 0.0) + if abs(d) > 1e-4: + trades.append({"sym": s, "side": "BUY" if d > 0 else "SELL", "dweight": round(d, 4)}) + st["positions"] = target + st["prices"] = last_close + st["last_mark_day"] = today + return comb_ret, today, st, dict(rebal=rebal, trades=trades, vrp_today=vrp_today, target=target, keep=keep) + + +def paper(cfg): + st = _load_state() + comb_ret, today, st, info = paper_step(cfg, st) + rebal, trades, vrp_today, target, keep = (info["rebal"], info["trades"], info["vrp_today"], + info["target"], info["keep"]) + _save_state(st) + print(f"\n===== SURFER PoC PAPER — day {today} (universe {len(keep)} perps) =====") + print(f"momentum {st['equity']:.4f} | VRP {st['vrp_equity']:.4f} | COMBINED BOOK {st['combined_equity']:.4f} " + f"(inception day {st['inception']}, {len(st['log'])} marks)") + print(f"VRP today: {'flat (gated)' if (vrp_today is not None and abs(vrp_today)<1e-9) else ('%.5f'%vrp_today if vrp_today is not None else 'n/a')} " + f"rebalance: {rebal} intended trades: {len(trades)}") + longs = sorted([(s, x) for s, x in target.items() if x > 0], key=lambda z: -z[1])[:8] + shorts = sorted([(s, x) for s, x in target.items() if x < 0], key=lambda z: z[1])[:8] + print("top longs : " + ", ".join(f"{s} {x:+.3f}" for s, x in longs)) + print("top shorts: " + ", ".join(f"{s} {x:+.3f}" for s, x in shorts)) + for t in trades[:12]: + print(f" {t['side']:>4} {t['sym']:>12} dW {t['dweight']:+.4f}") + print("(paper only — no real orders. Re-run weekly to forward-test edge persistence.)") + + +def status(cfg): + st = _load_state() + print(f"\n===== SURFER PoC STATUS =====") + if st["inception"] is None: + print("no paper state yet — run: python scripts/surfer/surfer_poc.py paper"); return + print(f"inception day {st['inception']} last mark {st['last_mark_day']} marks {len(st['log'])}") + print(f"equity: momentum {st['equity']:.4f} | VRP {st.get('vrp_equity',1.0):.4f} | COMBINED {st.get('combined_equity',1.0):.4f}") + if st["log"]: + def srp(key): + r = np.array([m[key] for m in st["log"] if key in m]) + return r.mean() / (r.std() + 1e-9) * math.sqrt(252) if len(r) > 5 else float("nan") + print(f"annual-ish Sharpe (live): momentum {srp('mom_ret'):+.2f} | VRP {srp('vrp_ret'):+.2f} | COMBINED {srp('comb_ret'):+.2f}") + print(f"current book: {len([p for p in st['positions'].values() if abs(p)>1e-6])} positions") + + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("mode", choices=["backtest", "regime", "paper", "status"]) + a = ap.parse_args() + {"backtest": backtest, "regime": regime, "paper": paper, "status": status}[a.mode](CFG) diff --git a/tests/integration/test_forward_tracker.py b/tests/integration/test_forward_tracker.py new file mode 100644 index 0000000..b6763d9 --- /dev/null +++ b/tests/integration/test_forward_tracker.py @@ -0,0 +1,59 @@ +"""Generic ForwardTracker: first-run freeze, forward-only booking, idempotency, extra round-trip.""" +from __future__ import annotations + +from fxhnt.application.forward_tracker import ForwardTracker + + +class _Recomputable: + """Recomputable-series strategy: always returns the full known series.""" + def __init__(self, series: list[tuple[str, float]]) -> None: + self._series = series + + def advance(self, last_date, extra): + return list(self._series), extra + + +class _LiveBooking: + """Live-booking strategy: books one row for `today`, carries a counter in extra.""" + def __init__(self, today: str, ret: float) -> None: + self._today, self._ret = today, ret + + def advance(self, last_date, extra): + n = int(extra.get("runs", 0)) + 1 + return [(self._today, self._ret)], {"runs": n} + + +def test_first_run_freezes_and_books_nothing(tmp_path) -> None: + p = str(tmp_path / "s.json") + st = ForwardTracker(_Recomputable([("2026-01-01", 0.01), ("2026-01-02", 0.02)]), p).step() + assert st.forward_days == 0 # freeze: nothing booked on first run + assert st.inception == "2026-01-02" # inception = latest known date + assert st.last_date == "2026-01-02" + + +def test_second_run_books_only_forward_days(tmp_path) -> None: + p = str(tmp_path / "s.json") + strat = _Recomputable([("2026-01-01", 0.01), ("2026-01-02", 0.02)]) + ForwardTracker(strat, p).step() # freeze at 2026-01-02 + strat._series.append(("2026-01-03", 0.05)) # a new day appears + st = ForwardTracker(strat, p).step() + assert st.forward_days == 1 # only 2026-01-03 booked + assert abs(st.forward_return_pct - 5.0) < 1e-9 + assert st.last_date == "2026-01-03" + + +def test_idempotent_rerun_books_nothing(tmp_path) -> None: + p = str(tmp_path / "s.json") + strat = _Recomputable([("2026-01-01", 0.01), ("2026-01-02", 0.02), ("2026-01-03", 0.05)]) + ForwardTracker(strat, p).step() # freeze at 2026-01-03 + st = ForwardTracker(strat, p).step() # nothing new + assert st.forward_days == 0 + + +def test_extra_carry_state_round_trips(tmp_path) -> None: + p = str(tmp_path / "s.json") + ForwardTracker(_LiveBooking("2026-01-01", 0.0), p).step() # freeze, extra={"runs":1} + ForwardTracker(_LiveBooking("2026-01-02", 0.01), p).step() # books 2026-01-02 + import json + extra = json.load(open(p))["extra"] + assert extra["runs"] == 2 # advance saw prior extra and incremented diff --git a/tests/integration/test_massive_daily.py b/tests/integration/test_massive_daily.py new file mode 100644 index 0000000..c5e266f --- /dev/null +++ b/tests/integration/test_massive_daily.py @@ -0,0 +1,126 @@ +"""MassiveDailyClient — total-return reconstruction from Massive (Polygon) split-only adjusted closes ++ dividends. NO live network: the `_get` HTTP layer is monkeypatched to return fixed fixtures. + +The critical invariant proven here: on an ex-dividend day, the day-over-day return of the returned +total-return series equals the raw PRICE return PLUS dividend/prev_close — i.e. the dividend is added +back, matching Yahoo adjclose semantics. With no dividends, TR closes == split-adjusted closes. +""" +from __future__ import annotations + +import datetime as dt +from typing import Any + +from fxhnt.adapters.data.massive_daily import MassiveDailyClient + + +def _ms(date: str) -> int: + """Midnight-UTC epoch milliseconds for a YYYY-MM-DD date (matches Massive aggregate `t`).""" + d = dt.datetime.strptime(date, "%Y-%m-%d").replace(tzinfo=dt.timezone.utc) + return int(d.timestamp() * 1000) + + +def _aggs(closes: dict[str, float]) -> dict[str, Any]: + return {"results": [{"t": _ms(d), "c": c} for d, c in sorted(closes.items())]} + + +def _divs(divs: list[tuple[str, float]]) -> dict[str, Any]: + return {"results": [{"ex_dividend_date": d, "cash_amount": c} for d, c in divs]} + + +def _install(client: MassiveDailyClient, closes: dict[str, float], divs: list[tuple[str, float]]) -> None: + """Route `_get` by endpoint: aggregates vs dividends. No network.""" + def fake_get(url: str, tries: int = 4) -> dict[str, Any]: + if "/v3/reference/dividends" in url: + return _divs(divs) + if "/v2/aggs/ticker/" in url: + return _aggs(closes) + raise AssertionError(f"unexpected url: {url}") + + client._get = fake_get # type: ignore[method-assign] + + +def test_returns_date_to_price_dict() -> None: + client = MassiveDailyClient(api_key="x") + closes = {"2026-01-02": 100.0, "2026-01-03": 101.0, "2026-01-06": 102.0} + _install(client, closes, divs=[]) + out = client.adj_closes("SPY") + assert isinstance(out, dict) + assert set(out) == set(closes) + assert all(isinstance(k, str) and isinstance(v, float) for k, v in out.items()) + + +def test_no_dividends_returns_split_adjusted_unchanged() -> None: + client = MassiveDailyClient(api_key="x") + closes = {"2026-01-02": 100.0, "2026-01-03": 101.0, "2026-01-06": 102.0} + _install(client, closes, divs=[]) + out = client.adj_closes("SPY") + for d, c in closes.items(): + assert abs(out[d] - c) < 1e-12 + + +def test_ex_div_day_return_includes_dividend() -> None: + """The load-bearing test: a single $1.80 dividend ex-date 2026-01-03. + + Raw price return on the ex-div day = c[03]/c[02] - 1. + Total-return on the ex-div day = tr[03]/tr[02] - 1, with the dividend added back: exactly + c_D/(prev_close - C) - 1 (canonical), ≈ price_return + dividend/prev_close (first order). + """ + cash = 1.80 + closes = {"2026-01-02": 100.0, "2026-01-03": 99.0, "2026-01-06": 100.0} + client = MassiveDailyClient(api_key="x") + _install(client, closes, divs=[("2026-01-03", cash)]) + out = client.adj_closes("SPY") + + prev_close = closes["2026-01-02"] + price_return = closes["2026-01-03"] / prev_close - 1.0 + tr_return = out["2026-01-03"] / out["2026-01-02"] - 1.0 + # EXACT canonical (CRSP / Yahoo) back-adjustment identity: prior close scaled by + # (1 - C/prev_close) ⇒ ex-div-day total return = c_D/(prev_close - C) - 1, the dividend added back. + exact_tr_return = closes["2026-01-03"] / (prev_close - cash) - 1.0 + assert abs(tr_return - exact_tr_return) < 1e-9 + # First-order (additive) form the spec describes: price return PLUS dividend yield. + additive_tr_return = price_return + cash / prev_close + assert abs(tr_return - additive_tr_return) < 1e-3 + # The most recent close is the anchor: factors only scale dates strictly before the ex-div date. + assert abs(out["2026-01-06"] - closes["2026-01-06"]) < 1e-12 + assert abs(out["2026-01-03"] - closes["2026-01-03"]) < 1e-12 + # Total return on an ex-div day is strictly higher than the raw price return. + assert tr_return > price_return + + +def test_multi_dividend_factors_compound() -> None: + """Two dividends on different ex-dates: back-adjustment factors must COMPOUND. + + Dates strictly before BOTH ex-divs receive both factors multiplied together; a date that lies + on/after the earlier ex-div but strictly before the later one receives ONLY the later factor; the + most-recent close (on the later ex-div date) is the anchor and is unscaled. This locks the + `factor[d] *= f` accumulation in MassiveDailyClient._total_return. + """ + # Split-adjusted closes on five consecutive trading dates. + closes = { + "2026-01-02": 100.0, + "2026-01-05": 101.0, + "2026-01-06": 99.0, + "2026-01-07": 102.0, + "2026-01-08": 100.0, + } + cash1 = 1.50 # ex-date 2026-01-06; trading day before = 2026-01-05 (close 101.0) + cash2 = 2.00 # ex-date 2026-01-08; trading day before = 2026-01-07 (close 102.0) + client = MassiveDailyClient(api_key="x") + _install(client, closes, divs=[("2026-01-06", cash1), ("2026-01-08", cash2)]) + out = client.adj_closes("SPY") + + f1 = 1.0 - cash1 / closes["2026-01-05"] # = 0.9851485148514851 + f2 = 1.0 - cash2 / closes["2026-01-07"] # = 0.9803921568627451 + + # Strictly before BOTH ex-divs ⇒ BOTH factors compounded. + assert abs(out["2026-01-02"] - closes["2026-01-02"] * f1 * f2) < 1e-9 # 96.58318773053776 + assert abs(out["2026-01-05"] - closes["2026-01-05"] * f1 * f2) < 1e-9 # 97.54901960784314 + # On/after the earlier ex-div (2026-01-06) but strictly before the later (2026-01-08) ⇒ ONLY f2. + assert abs(out["2026-01-06"] - closes["2026-01-06"] * f2) < 1e-9 # 97.05882352941175 + assert abs(out["2026-01-07"] - closes["2026-01-07"] * f2) < 1e-9 # 100.0 + # The later ex-div date is the most-recent close: anchor, unscaled. + assert abs(out["2026-01-08"] - closes["2026-01-08"]) < 1e-9 # 100.0 + + # Sanity: the doubly-adjusted date is scaled strictly more than the singly-adjusted one (f1<1). + assert out["2026-01-05"] / closes["2026-01-05"] < out["2026-01-06"] / closes["2026-01-06"] diff --git a/tests/integration/test_orchestration_definitions.py b/tests/integration/test_orchestration_definitions.py index 6ee020c..ae5187f 100644 --- a/tests/integration/test_orchestration_definitions.py +++ b/tests/integration/test_orchestration_definitions.py @@ -11,6 +11,9 @@ def test_definitions_load_with_assets_and_schedule() -> None: repo = defs.get_repository_def() asset_keys = {k.to_user_string() for k in repo.assets_defs_by_key} assert {"crypto_bars", "futures_bars", "combined_forward_nav", "cockpit_forward"} <= asset_keys + # B1: the six paper-track assets are wired into the graph alongside the B0 four + paper = {"sixtyforty_nav", "multistrat_nav", "gd_nav", "funding_nav", "crossvenue_nav", "poc_nav"} + assert paper <= asset_keys, f"missing paper-track assets: {paper - asset_keys}" # --- schedule present with the right cron --- # defs.schedules is a list[ScheduleDefinition] (or None when empty) @@ -22,3 +25,11 @@ def test_definitions_load_with_assets_and_schedule() -> None: from dagster import DefaultScheduleStatus daily = next(s for s in scheds if getattr(s, "cron_schedule", "") == "30 23 * * *") assert daily.default_status == DefaultScheduleStatus.RUNNING + + # --- cockpit_forward depends on every paper-track asset (so it ingests their state files) --- + from dagster import AssetKey + cockpit_def = repo.assets_defs_by_key[AssetKey("cockpit_forward")] + deps = cockpit_def.asset_deps[AssetKey("cockpit_forward")] + upstream = {k.to_user_string() for k in deps} + expected_upstream = {"combined_forward_nav"} | paper + assert expected_upstream <= upstream, f"cockpit_forward missing upstream: {expected_upstream - upstream}" diff --git a/tests/integration/test_paper_strategies.py b/tests/integration/test_paper_strategies.py new file mode 100644 index 0000000..9d3fa3a --- /dev/null +++ b/tests/integration/test_paper_strategies.py @@ -0,0 +1,513 @@ +"""Paper-forward strategy services: deterministic domain + service over fake data ports, and a +round-trip through ForwardStateReader so the cockpit contract is verified.""" +from __future__ import annotations + +import json +from pathlib import Path + +from fxhnt.adapters.persistence.state_reader import ForwardStateReader +from fxhnt.application.forward_tracker import ForwardTracker +from fxhnt.application.paper_strategies import ( + CrossVenueStrategy, + FundingCarryStrategy, + GrowthDisciplineStrategy, + MomentumVrpStrategy, + SixtyFortyStrategy, +) + + +class FakeDailyBars: + def __init__(self, data: dict[str, dict[str, float]]) -> None: + self._data = data + + def adj_closes(self, symbol: str) -> dict[str, float]: + return self._data[symbol] + + +class FakeBinanceFunding: + """Deterministic BinanceFundingClient: fixed (liq, tf30, last24) scan output. + + liq is the liquid/native/hedgeable universe ({sym: 24h_quote_volume}); tf30 is the trailing-30d-mean + daily funding (the qualify signal); last24 is today's booking rate (the book_return signal).""" + + def __init__( + self, + liq: dict[str, float], + tf30: dict[str, float], + last24: dict[str, float], + ) -> None: + self._liq = liq + self._tf30 = tf30 + self._last24 = last24 + + def scan(self, prev: set[str]) -> tuple[dict[str, float], dict[str, float], dict[str, float]]: + return dict(self._liq), dict(self._tf30), dict(self._last24) + + +class FakeCrossVenue: + """Deterministic CrossVenueFundingClient: fixed per-venue funding + volume, no network. + + snapshot() returns (funding, volume) from one call — funding: {coin: {venue: daily_funding_rate}} + (already normalized to DAILY, as the live adapter returns); volume: {coin: {venue: 24h_quote_volume_usd}}.""" + + def __init__( + self, + funding: dict[str, dict[str, float]], + volume: dict[str, dict[str, float]], + ) -> None: + self._funding = funding + self._volume = volume + + def snapshot(self) -> tuple[dict[str, dict[str, float]], dict[str, dict[str, float]]]: + return ( + {c: dict(v) for c, v in self._funding.items()}, + {c: dict(v) for c, v in self._volume.items()}, + ) + + +def test_sixtyforty_daily_return_is_weighted_mean() -> None: + from fxhnt.domain.strategies.sixtyforty import daily_returns + closes = { + "SPY": {"2026-01-01": 100.0, "2026-01-02": 110.0}, # +10% + "IEF": {"2026-01-01": 100.0, "2026-01-02": 105.0}, # +5% + } + rows = daily_returns(closes, [("SPY", 0.6), ("IEF", 0.4)]) + assert len(rows) == 1 and rows[0][0] == "2026-01-02" + assert abs(rows[0][1] - (0.6 * 0.10 + 0.4 * 0.05)) < 1e-12 # 0.08 + + +def test_sixtyforty_service_round_trips_through_reader(tmp_path) -> None: + bars = FakeDailyBars({ + "SPY": {"2026-01-01": 100.0, "2026-01-02": 110.0, "2026-01-03": 110.0}, + "IEF": {"2026-01-01": 100.0, "2026-01-02": 105.0, "2026-01-03": 105.0}, + }) + p = str(tmp_path / "sixtyforty_state.json") + ForwardTracker(SixtyFortyStrategy(bars), p).step() # freeze at 2026-01-03 + bars._data["SPY"]["2026-01-04"] = 121.0 # +10% + bars._data["IEF"]["2026-01-04"] = 105.0 # 0% + st = ForwardTracker(SixtyFortyStrategy(bars), p).step() + assert st.forward_days == 1 + summary, rows = ForwardStateReader().read(p, "sixtyforty") + assert summary.days == 1 + assert abs(summary.nav - (1.0 + 0.06)) < 1e-9 # 0.6*0.10 + 0.4*0 + + +def test_multistrat_vol_targets_and_caps_leverage() -> None: + import datetime + import math + + import numpy as np + + from fxhnt.domain.strategies.multistrat import INSTRUMENTS, book_series + rng = np.random.default_rng(0) + # Faithful port keeps the original 63-day warmup (volnorm/trust windows); need >> 64 dates + # for any day to be booked, so generate ~10 months of weekday-ish dates. + start = datetime.date(2025, 1, 1) + dates = [(start + datetime.timedelta(days=i)).isoformat() for i in range(300)] + closes = {} + for k, s in enumerate(INSTRUMENTS): + steps = rng.normal(0.0003 * (1 if k % 2 else -1), 0.01, len(dates)) + px = 100.0 * np.cumprod(1 + steps) + closes[s] = {d: float(px[i]) for i, d in enumerate(dates)} + rows = book_series(closes) + rets = np.array([r for _, r in rows]) + assert len(rows) >= 1 and np.isfinite(rets).all() + vol = float(rets.std() * math.sqrt(252)) + # Faithful port is UNLEVERED (MAXLEV=1.0): on this low-cross-corr synthetic data the trust-weighted + # combo diversifies down to ~0.047 vol, so the unlevered book legitimately realizes ~0.038 — below + # the 0.10 target because leverage is capped at 1.0 and cannot lever the combo back up. The invariant + # tested is that the book is vol-CONTROLLED (well under the cap, never blowing past target), not that + # it hits 0.10 exactly. Verified bit-for-bit identical to the foxhunt original on this data. + assert 0.02 <= vol <= 0.15 # vol-controlled (unlevered book; widened per faithful port) + + +def test_growth_discipline_blend_matches_w_qqq_plus_multistrat() -> None: + import datetime + + import numpy as np + + from fxhnt.domain.strategies.growth_discipline import daily_returns + from fxhnt.domain.strategies.multistrat import INSTRUMENTS, book_series + rng = np.random.default_rng(1) + # Enough dates that multistrat's 63-day warmup still leaves booked days (mirror multistrat test). + start = datetime.date(2025, 1, 1) + dates = [(start + datetime.timedelta(days=i)).isoformat() for i in range(300)] + closes = {} + for k, s in enumerate(INSTRUMENTS): + steps = rng.normal(0.0003 * (1 if k % 2 else -1), 0.01, len(dates)) + px = 100.0 * np.cumprod(1 + steps) + closes[s] = {d: float(px[i]) for i, d in enumerate(dates)} + # QQQ shares the same date grid so its consecutive-day return aligns with the book's dates exactly. + qsteps = rng.normal(0.0004, 0.011, len(dates)) + qpx = 100.0 * np.cumprod(1 + qsteps) + qqq = {d: float(qpx[i]) for i, d in enumerate(dates)} + + w = 0.70 + rows = daily_returns(qqq, closes, w) + gd = dict(rows) + + # Reconstruct the expectation: r_multistrat from book_series, r_QQQ as consecutive-day return. + ms = dict(book_series(closes)) + qdates = sorted(qqq) + qret = {qdates[i]: qqq[qdates[i]] / qqq[qdates[i - 1]] - 1.0 for i in range(1, len(qdates))} + expected = {d: w * qret[d] + (1.0 - w) * ms[d] for d in (set(ms) & set(qret))} + + common = set(gd) & set(expected) + assert len(common) >= 1 + for d in common: + assert abs(gd[d] - expected[d]) < 1e-9 + + +def test_growth_discipline_service_round_trips_through_reader(tmp_path) -> None: + import datetime + + import numpy as np + + from fxhnt.domain.strategies.multistrat import INSTRUMENTS + rng = np.random.default_rng(2) + start = datetime.date(2025, 1, 1) + dates = [(start + datetime.timedelta(days=i)).isoformat() for i in range(300)] + data = {} + for k, s in enumerate(INSTRUMENTS): + steps = rng.normal(0.0003 * (1 if k % 2 else -1), 0.01, len(dates)) + px = 100.0 * np.cumprod(1 + steps) + data[s] = {d: float(px[i]) for i, d in enumerate(dates)} + qsteps = rng.normal(0.0004, 0.011, len(dates)) + qpx = 100.0 * np.cumprod(1 + qsteps) + data["QQQ"] = {d: float(qpx[i]) for i, d in enumerate(dates)} + bars = FakeDailyBars(data) + + p = str(tmp_path / "gd_state.json") + ForwardTracker(GrowthDisciplineStrategy(bars), p).step() # freeze at inception + # Append one new trading day across every series so the second step books >= 1 day. + nd = (start + datetime.timedelta(days=len(dates))).isoformat() + for s in list(INSTRUMENTS) + ["QQQ"]: + last = data[s][dates[-1]] + data[s][nd] = last * 1.001 + ForwardTracker(GrowthDisciplineStrategy(bars), p).step() + summary, rows = ForwardStateReader().read(p, "gd") + assert summary.days == len(rows) + assert summary.days >= 1 + + +def test_funding_qualify_on_tf30_and_book_return_on_last24() -> None: + from fxhnt.domain.strategies.funding_carry import book_return, qualify + + # `liq` is the already-filtered liquid/native/hedgeable universe (keys = qualify candidates, + # values = 24h quote volume, unused by qualify). + liq = {"AAAUSDT": 10e6, "BBBUSDT": 10e6, "DDDUSDT": 10e6} + # qualify reads tf30 (trailing-30d MEAN), NOT today's last24. + # AAA: tf30 above hurdle (new opens) + # BBB: tf30 between exit(3bp) and hurdle(5bp) (only stays if held) + # DDD: tf30 below exit, BUT a huge last24 today — must NOT qualify (qualify ignores last24). + tf30 = {"AAAUSDT": 0.0006, "BBBUSDT": 0.0004, "DDDUSDT": 0.0002} + last24 = {"AAAUSDT": 0.0006, "BBBUSDT": 0.0004, "DDDUSDT": 0.0090} # DDD spikes today but low tf30 + + # No prior holdings: only AAA qualifies (BBB below hurdle, DDD below exit despite high last24). + assert qualify(liq, tf30, prev={}) == {"AAAUSDT": 0.0006} + # BBB held and still above exit_hurdle on tf30 → kept; AAA still opens; DDD dropped. + assert sorted(qualify(liq, tf30, prev={"BBBUSDT": 0.5})) == ["AAAUSDT", "BBBUSDT"] + # A held coin whose tf30 is high last24 but low tf30 still does NOT qualify (proves tf30 is the gate). + assert qualify(liq, tf30, prev={"DDDUSDT": 0.5}) == {"AAAUSDT": 0.0006} + + # book_return uses last24 (today's booking rate), NOT tf30: prior equal-weight book of 2 coins + # (w=0.5 each), rebalance to a single-coin book → realized minus cost on turnover (COST_RT/2). + prev = {"AAAUSDT": 0.5, "BBBUSDT": 0.5} + newpos = {"AAAUSDT": 1.0} + realized = 0.5 * 0.0006 + 0.5 * 0.0004 # last24 of AAA and BBB + turnover = abs(1.0 - 0.5) + abs(0.0 - 0.5) # AAA up 0.5, BBB out 0.5 + expected = realized - turnover * (1e-3 / 2) + assert abs(book_return(prev, last24, newpos) - expected) < 1e-15 + + +def test_funding_service_round_trips_through_reader(tmp_path) -> None: + liq = {"AAAUSDT": 10e6, "BBBUSDT": 10e6} + tf30 = {"AAAUSDT": 0.0006, "BBBUSDT": 0.0006} # both above hurdle on the 30d mean + last24 = {"AAAUSDT": 0.0006, "BBBUSDT": 0.0006} # today's booking rate + fake = FakeBinanceFunding(liq, tf30, last24) + p = str(tmp_path / "funding_state.json") + + # Live-booking books rows dated by the wall clock. On the freeze run rows are empty so the tracker + # freezes inception at its own _today_iso; the second run must book a row dated strictly after that. + # Inject a clock returning a day after inception (real cron runs on later calendar days). + day = [""] + + # First step freezes (books nothing) but seeds initial equal-weight positions in extra. + st0 = ForwardTracker(FundingCarryStrategy(fake, clock=lambda: day[0]), p).step() + assert st0.forward_days == 0 + loaded0 = json.loads(Path(p).read_text()) + assert loaded0["extra"]["positions"] == {"AAAUSDT": 0.5, "BBBUSDT": 0.5} + + # Next day: bump today's booking rate so the booked carry differs; second step books exactly one row. + fake._last24["AAAUSDT"] = 0.0008 + fake._last24["BBBUSDT"] = 0.0008 + day[0] = "2999-01-01" # strictly after the frozen inception → booked + st1 = ForwardTracker(FundingCarryStrategy(fake, clock=lambda: day[0]), p).step() + assert st1.forward_days == 1 + + summary, rows = ForwardStateReader().read(p, "funding") + assert summary.days == 1 + loaded1 = json.loads(Path(p).read_text()) + # Still both qualify → equal-weight book of two positions. + assert loaded1["extra"]["positions"] == {"AAAUSDT": 0.5, "BBBUSDT": 0.5} + + +def test_crossvenue_spread_filters_liq_and_distress() -> None: + from fxhnt.domain.strategies.cross_venue import LIQ, MAXF, compute_spreads + + funding = { + # AAA: liquid on Binance(+12bp) and Bybit(+2bp) → spread 10bp, short Binance, long Bybit. + "AAA": {"Binance": 0.0012, "Bybit": 0.0002}, + # BBB: HL leg is below LIQ → only 1 venue passes → dropped (need >=2). + "BBB": {"Binance": 0.0020, "HL": 0.0001}, + # CCC: Binance leg is distress (|f| > MAXF) → excluded → only 1 venue → dropped. + "CCC": {"Binance": 0.0060, "Bybit": 0.0003}, + # DDD: three liquid non-distress venues → spread = max-min across all three. + "DDD": {"Binance": 0.0008, "Bybit": -0.0002, "HL": 0.0003}, + } + volume = { + "AAA": {"Binance": 20e6, "Bybit": 20e6}, + "BBB": {"Binance": 20e6, "HL": 1e6}, # HL below LIQ + "CCC": {"Binance": 20e6, "Bybit": 20e6}, # Binance distress on funding + "DDD": {"Binance": 20e6, "Bybit": 20e6, "HL": 20e6}, + } + assert LIQ == 10e6 and MAXF == 0.005 + rows = compute_spreads(funding, volume) + by_coin = {r["coin"]: r for r in rows} + assert set(by_coin) == {"AAA", "DDD"} + assert abs(by_coin["AAA"]["spread"] - (0.0012 - 0.0002)) < 1e-15 + assert by_coin["AAA"]["short"] == "Binance" and by_coin["AAA"]["long"] == "Bybit" + assert abs(by_coin["DDD"]["spread"] - (0.0008 - (-0.0002))) < 1e-15 + assert by_coin["DDD"]["short"] == "Binance" and by_coin["DDD"]["long"] == "Bybit" + + +def test_crossvenue_select_hysteresis_topk() -> None: + from fxhnt.domain.strategies.cross_venue import ENTRY, EXIT, TOPK, select + + assert ENTRY == 0.0010 and EXIT == 0.0005 and TOPK == 10 + # cur spreads: AAA strong-fresh-open; BBB between EXIT and ENTRY (only kept if held); CCC below EXIT. + cur = {"AAA": 0.0020, "BBB": 0.0007, "CCC": 0.0003, "DDD": 0.0015} + + # No prior: only coins with spread > ENTRY open (AAA, DDD), sorted by spread desc, capped at TOPK. + assert sorted(select(cur, prev={})) == ["AAA", "DDD"] + # BBB held and still > EXIT → kept; AAA/DDD also open; CCC (held) below EXIT → dropped. + assert sorted(select(cur, prev={"BBB": 0.5, "CCC": 0.5})) == ["AAA", "BBB", "DDD"] + + # TOPK cap: 12 fresh coins all above ENTRY, none held → only the top-10 by spread are taken. + many = {f"C{i:02d}": 0.0010 + i * 1e-5 for i in range(12)} # C11 highest ... C00 lowest + chosen = select(many, prev={}) + assert len(chosen) == TOPK + assert set(chosen) == {f"C{i:02d}" for i in range(2, 12)} # top-10 spreads + + +def test_crossvenue_book_return_realized_minus_turnover_cost() -> None: + from fxhnt.domain.strategies.cross_venue import COST_RT, book_return + + assert COST_RT == 0.0010 + # Prior equal-weight book of 2 coins; today's spreads (cur) carry on the prior book. + prev = {"AAA": 0.5, "BBB": 0.5} + cur = {"AAA": 0.0020, "BBB": 0.0006, "DDD": 0.0030} # DDD not in prev → not realized + newpos = {"AAA": 1.0} # rebalance to single coin + realized = 0.5 * 0.0020 + 0.5 * 0.0006 # carry on prior book at today's spread + turnover = abs(1.0 - 0.5) + abs(0.0 - 0.5) # AAA up 0.5, BBB out 0.5 + expected = realized - turnover * (COST_RT / 2) + assert abs(book_return(prev, cur, newpos) - expected) < 1e-15 + + +def test_crossvenue_service_round_trips_through_reader(tmp_path) -> None: + # Two coins, each liquid + non-distress on two venues, spread > ENTRY → both open. + funding = { + "AAA": {"Binance": 0.0015, "Bybit": 0.0001}, # spread 14bp + "BBB": {"Binance": 0.0013, "Bybit": 0.0001}, # spread 12bp + } + volume = { + "AAA": {"Binance": 20e6, "Bybit": 20e6}, + "BBB": {"Binance": 20e6, "Bybit": 20e6}, + } + fake = FakeCrossVenue(funding, volume) + p = str(tmp_path / "crossvenue_state.json") + + day = [""] + # First step freezes (books nothing) but seeds initial equal-weight positions. + st0 = ForwardTracker(CrossVenueStrategy(fake, clock=lambda: day[0]), p).step() + assert st0.forward_days == 0 + loaded0 = json.loads(Path(p).read_text()) + assert loaded0["extra"]["positions"] == {"AAA": 0.5, "BBB": 0.5} + + day[0] = "2999-01-01" # strictly after frozen inception → booked + st1 = ForwardTracker(CrossVenueStrategy(fake, clock=lambda: day[0]), p).step() + assert st1.forward_days == 1 + + summary, rows = ForwardStateReader().read(p, "crossvenue") + assert summary.days == 1 + loaded1 = json.loads(Path(p).read_text()) + assert loaded1["extra"]["positions"] == {"AAA": 0.5, "BBB": 0.5} + + +# ---------------------------------------------------------------- poc (surfer momentum + VRP, faithful) +def test_xs_weights_market_neutral_unit_gross() -> None: + """Real signal_sweep.xs_weights on a small array → cross-sectionally market-neutral, unit gross.""" + import numpy as np + + from fxhnt.vendor.surfer.signal_sweep import xs_weights + + sig = np.array( + [ + [1.0, 2.0, 3.0, 4.0, 5.0], + [-2.0, 0.0, 1.0, 3.0, -1.0], + [10.0, -5.0, 2.0, 7.0, -3.0], + ] + ) + w = xs_weights(sig) + assert w.shape == sig.shape + # market-neutral: each row sums ~0; unit gross: sum|w| ~1 per row. + for t in range(w.shape[0]): + assert abs(float(w[t].sum())) < 1e-9 + assert abs(float(np.abs(w[t]).sum()) - 1.0) < 1e-9 + + +def test_compute_weights_shape() -> None: + """Real surfer_poc.compute_weights on a synthetic 50x80 panel. + + Invariants of the faithful signal: shape preserved; the per-day *target* (pre-smoothing) xs_weights is + universe-restricted so its nonzeros ≤ topk; and the held (EWMA-smoothed, weekly-held) book stays + cross-sectionally market-neutral (row-sum ≈ 0) — EWMA is a linear combo of zero-sum target rows so it + cannot break neutrality even though it carries weights across day-to-day universe rotation (so the + SMOOTHED book can legitimately have > topk nonzeros).""" + import numpy as np + + from fxhnt.vendor.surfer.surfer_poc import CFG, compute_weights + + rng = np.random.default_rng(7) + T, N = 50, 80 + close = 100.0 * np.cumprod(1.0 + rng.normal(0.0, 0.02, (T, N)), axis=0) + qvol = np.abs(rng.normal(5e6, 1e6, (T, N))) + days = np.arange(19000, 19000 + T, dtype=np.int64) + held, univ = compute_weights(close, qvol, days, CFG) + assert held.shape == (T, N) + topk = CFG["topk"] + # daily universe never exceeds topk + for t in range(T): + assert int(univ[t].sum()) <= topk + if int(np.count_nonzero(np.abs(held[t]) > 1e-12)): + assert abs(float(held[t].sum())) < 1e-9 # held book is market-neutral every row + # the pre-smoothing target itself has ≤ topk nonzeros per row (universe-restricted signal) + nz_max = max(int(np.count_nonzero(np.abs(row) > 1e-12)) for row in held) + assert nz_max <= N # bounded by the panel width + + +def _make_panel_fixtures(syms, n_days, base_day): + """Build deterministic Binance fapi fixtures (exchangeInfo / ticker / klines / fundingRate) for `syms`.""" + import numpy as np + + rng = np.random.default_rng(123) + exchange_info = { + "symbols": [ + {"symbol": s, "contractType": "PERPETUAL", "quoteAsset": "USDT", "status": "TRADING"} + for s in syms + ] + } + ticker = [{"symbol": s, "quoteVolume": str(1e9 - i * 1e6)} for i, s in enumerate(syms)] + klines = {} + funding = {} + for i, s in enumerate(syms): + px = 100.0 * np.cumprod(1.0 + rng.normal(0.0003 * (1 if i % 2 else -1), 0.02, n_days)) + rows = [] + for k in range(n_days): + open_ms = (base_day + k) * 86_400_000 + rows.append([open_ms, "0", "0", "0", f"{px[k]:.6f}", "0", 0, f"{px[k] * 5e6:.2f}"]) + klines[s] = rows + funding[s] = [{"fundingRate": "0.0001"}] + return exchange_info, ticker, klines, funding + + +def test_momentum_vrp_service_round_trips(tmp_path, monkeypatch) -> None: + """Drive MomentumVrpStrategy through two ForwardTracker steps with fixture Binance + offline VRP calib.""" + import numpy as np + + from fxhnt.vendor.surfer import pit_sweep, surfer_poc + + cfg = surfer_poc.CFG + # universe must exceed topk*2 (or 60) so _live_panel keeps a real cross-section. + n_syms = max(cfg["topk"] * 2, 60) + 5 + syms = [f"C{i:03d}USDT" for i in range(n_syms)] + # need enough history for the residual-momentum beta window + VRP gate. + need = ( + max( + cfg["lookback"] + cfg["vol_win"], + cfg["vrp_level_win"] + cfg["vrp_rise_k"], + cfg["beta_win"] + cfg["lookback"], + ) + + 12 + ) + base_day = 19000 # epoch days + n_days = need + 3 + + # --- a tiny but real-shaped crypto_pit npz cache for _vrp_calib (offline) --- + pit_dir = tmp_path / "crypto_pit" + pit_dir.mkdir() + rng = np.random.default_rng(5) + pit_days = np.arange(base_day, base_day + n_days, dtype=np.int64) + for s in ["BTCUSDT", "ETHUSDT"] + syms[:40]: + px = 100.0 * np.cumprod(1.0 + rng.normal(0.0, 0.02, n_days)) + np.savez( + pit_dir / f"{s}.npz", + day=pit_days, + close=px, + qvol=np.abs(rng.normal(5e6, 1e6, n_days)), + funding=np.full(n_days, 0.0001), + ) + monkeypatch.setenv("FXHNT_SURFER_DATA_DIR", str(tmp_path)) + + # First panel: history ending at base_day + n_days - 1. + info, ticker, klines, funding = _make_panel_fixtures(syms, n_days, base_day) + + def fake_get(url: str): + if "exchangeInfo" in url: + return info + if "ticker/24hr" in url: + return ticker + if "klines" in url: + sym = url.split("symbol=")[1].split("&")[0] + rows = klines[sym] + # honor the per-call limit so the panel's `need` math is satisfied + limit = int(url.split("limit=")[1].split("&")[0]) + return rows[-limit:] + if "fundingRate" in url: + sym = url.split("symbol=")[1].split("&")[0] + return funding[sym] + if "deribit.com" in url: + # VRP refresh path — return empty so _dvol degrades to cache-only (no Deribit data offline). + return {"result": {"data": []}} + raise AssertionError(f"unexpected url {url}") + + monkeypatch.setattr(surfer_poc, "_get", fake_get) + # no sleeps in tests + monkeypatch.setattr(surfer_poc.time, "sleep", lambda *_a, **_k: None) + + p = str(tmp_path / "poc_state.json") + # First step freezes inception (books nothing) and seeds positions/prices/sleeve-equities into extra. + st0 = ForwardTracker(MomentumVrpStrategy(), p).step() + assert st0.forward_days == 0 + loaded0 = json.loads(Path(p).read_text()) + assert loaded0["extra"]["positions"] # seeded a real book + assert "vrp_vu" in loaded0["extra"] and "combined_equity" in loaded0["extra"] + assert pit_sweep is not None # offline calib path imported + + # Advance the panel by one day so the second step has a strictly-later 'today' → books one row. + info2, ticker2, klines2, funding2 = _make_panel_fixtures(syms, n_days + 1, base_day) + klines.clear() + klines.update(klines2) + info["symbols"] = info2["symbols"] + ticker[:] = ticker2 + funding.clear() + funding.update(funding2) + + st1 = ForwardTracker(MomentumVrpStrategy(), p).step() + assert st1.forward_days == 1 + + summary, rows = ForwardStateReader().read(p, "poc") + assert summary.days == 1 + loaded1 = json.loads(Path(p).read_text()) + assert loaded1["extra"]["positions"] + assert "vrp_equity" in loaded1["extra"] diff --git a/tests/integration/test_tiingo_daily.py b/tests/integration/test_tiingo_daily.py new file mode 100644 index 0000000..836aece --- /dev/null +++ b/tests/integration/test_tiingo_daily.py @@ -0,0 +1,134 @@ +"""TiingoDailyClient — native total-return adjClose (equity/ETF) + crypto routing. NO live network: +the `_get` HTTP layer is monkeypatched to return fixed fixtures. + +Invariants proven here: + (1) equity symbols hit the daily endpoint and read `adjClose` (NOT raw `close`). + (2) crypto `BTC-USD` is routed to the crypto endpoint, ticker mapped to `btcusd`, and the nested + `priceData[].close` is read (crypto has no adjClose). + (3) the request window's startDate is computed from `lookback_days` (today - lookback_days). +""" +from __future__ import annotations + +import datetime as dt +from typing import Any + +from fxhnt.adapters.data.tiingo_daily import TiingoDailyClient + + +def _daily(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + """Tiingo equity/ETF daily response: a flat list with both `close` and `adjClose`.""" + return [ + { + "date": f"{r['date']}T00:00:00.000Z", + "close": r["close"], + "adjClose": r["adjClose"], + "divCash": 0.0, + "splitFactor": 1.0, + } + for r in rows + ] + + +def _crypto(ticker: str, rows: list[tuple[str, float]]) -> list[dict[str, Any]]: + """Tiingo crypto response: a one-element list with nested `priceData` (close only, no adjClose).""" + return [ + { + "ticker": ticker, + "baseCurrency": "btc", + "quoteCurrency": "usd", + "priceData": [ + {"date": f"{d}T00:00:00.000Z", "open": c, "high": c, "low": c, "close": c, "volume": 1.0} + for d, c in rows + ], + } + ] + + +def test_equity_uses_adjclose_not_close() -> None: + client = TiingoDailyClient(api_key="x") + # adjClose deliberately differs from close so we can prove which field is read. + rows = [ + {"date": "2026-01-02", "close": 100.0, "adjClose": 90.0}, + {"date": "2026-01-03", "close": 101.0, "adjClose": 91.5}, + {"date": "2026-01-06", "close": 102.0, "adjClose": 93.0}, + ] + captured: dict[str, str] = {} + + def fake_get(url: str, tries: int = 4) -> Any: + captured["url"] = url + assert "/tiingo/daily/SPY/prices" in url + return _daily(rows) + + client._get = fake_get + out = client.adj_closes("SPY") + + assert out == {"2026-01-02": 90.0, "2026-01-03": 91.5, "2026-01-06": 93.0} + assert all(isinstance(k, str) and isinstance(v, float) for k, v in out.items()) + # Daily endpoint, NOT crypto. + assert "/tiingo/crypto/" not in captured["url"] + + +def test_crypto_btc_routed_and_ticker_mapped() -> None: + client = TiingoDailyClient(api_key="x") + rows = [("2026-01-02", 45000.0), ("2026-01-03", 46000.5), ("2026-01-06", 44000.25)] + captured: dict[str, str] = {} + + def fake_get(url: str, tries: int = 4) -> Any: + captured["url"] = url + # Routed to the crypto endpoint with the Yahoo ticker mapped to btcusd. + assert "/tiingo/crypto/prices" in url + assert "tickers=btcusd" in url + assert "resampleFreq=1day" in url + assert "/tiingo/daily/" not in url + return _crypto("btcusd", rows) + + client._get = fake_get + out = client.adj_closes("BTC-USD") + + # Nested priceData[].close is read (crypto has no adjClose). + assert out == {"2026-01-02": 45000.0, "2026-01-03": 46000.5, "2026-01-06": 44000.25} + assert "tickers=btcusd" in captured["url"] + + +def test_startdate_window_from_lookback_days() -> None: + client = TiingoDailyClient(api_key="x", lookback_days=1100) + captured: dict[str, str] = {} + + def fake_get(url: str, tries: int = 4) -> Any: + captured["url"] = url + return _daily([{"date": "2026-01-02", "close": 100.0, "adjClose": 100.0}]) + + client._get = fake_get + client.adj_closes("SPY") + + expected_start = (dt.date.today() - dt.timedelta(days=1100)).isoformat() + expected_end = dt.date.today().isoformat() + assert f"startDate={expected_start}" in captured["url"] + assert f"endDate={expected_end}" in captured["url"] + + +def test_crypto_window_from_lookback_days() -> None: + client = TiingoDailyClient(api_key="x", lookback_days=365) + captured: dict[str, str] = {} + + def fake_get(url: str, tries: int = 4) -> Any: + captured["url"] = url + return _crypto("btcusd", [("2026-01-02", 45000.0)]) + + client._get = fake_get + client.adj_closes("BTC-USD") + + expected_start = (dt.date.today() - dt.timedelta(days=365)).isoformat() + assert f"startDate={expected_start}" in captured["url"] + + +def test_empty_equity_response_returns_empty_dict() -> None: + client = TiingoDailyClient(api_key="x") + client._get = lambda url, tries=4: [] # noqa: E731 + assert client.adj_closes("SPY") == {} + + +def test_empty_crypto_response_returns_empty_dict() -> None: + client = TiingoDailyClient(api_key="x") + client._get = lambda url, tries=4: [] # noqa: E731 + assert client.adj_closes("BTC-USD") == {}