# Funding-carry tail: data-acquisition & ingestion design (research output)

**Date:** 2026-06-22
**Status:** Design / feasibility (from the `carry-data-acquisition-research` workflow: 5 parallel research agents
→ adversarial fact-check → synthesis). NOT yet built. Gated on a decision to bring carry into the replay.
**Problem:** xsfunding (delta-neutral funding carry) shows a fake Sharpe (15 on 1yr, 1.80 on 6yr) because its
real tail is invisible to our data: daily close marks + survivorship-biased (only currently-listed Binance
USDT-perps). Deep 2022 losses were intraday squeezes, DELISTED coins (LUNA), ADL, and counterparty (FTX).

## 1. Source: free `data.binance.vision` covers almost everything
- **Public, unauthenticated, LISTABLE S3 bucket** (XML listing: `https://s3-ap-northeast-1.amazonaws.com/
  data.binance.vision?delimiter=/&prefix=<P>&marker=<K>`; download `https://data.binance.vision/<KEY>`;
  `.CHECKSUM` siblings). **CONFIRMED survivorship-free**: delisted symbols retained permanently — LUNAUSDT /
  FTTUSDT / SRMUSDT klines + fundingRate present (LUNA death-week 1m + funding confirmed). **Never use REST
  `exchangeInfo`** (not survivorship-free; LUNA fully purged).
- Date floors (CONFIRMED): UM klines/funding from 2020-01, UM metrics from 2020-09, spot from 2017-08 → the
  ~6.4yr window covers 2022-05 LUNA + 2022-11 FTX.
- **Liquidation tape is the ONLY gap** → free UM `liquidationSnapshot` was **DELETED by Binance** (404 now).
  Only source is a paid third-party capture (Tardis.dev, from 2020-01) of the websocket `forceOrder` stream —
  and it's **fundamentally lossy** (one largest liq/symbol/1000ms → under-reports cascades = the tail). Treat
  as a lower bound; do NOT buy a vendor for funding/klines/universe (all free).

## 2. Survivorship-free universe (the highest-ROI piece)
1. Enumerate the **full ever-listed set from the klines prefix** (price = SSOT), not exchangeInfo: list
   `data/futures/um/monthly/klines/` CommonPrefixes (~**876** symbols today, single page; code for
   `IsTruncated`/`marker` anyway).
2. Per symbol, derive `[first_trade_ts, last_trade_ts]` from file presence; PIT universe on D = `{sym: first ≤
   D ≤ last}`.
3. Four traps that silently re-introduce survivorship bias: **rename/redenomination** (LUNAUSDT→1000LUNCUSDT,
   1000BONK/PEPE/SHIB/FLOKI — chain-link returns across the rename or you drop the 99.9% crash), **BUSD→USDT/
   USDC quote migration** (41 dead *BUSD + 39 *USDC overlaps), **re-listings** (disjoint intervals — emit
   multiple per symbol), **spot-leg survivorship** (apply to `data/spot/` too; reconcile funding vs klines
   symbol sets — carry needs both legs).
4. Cross-check interval ends vs the Binance delisting-announcement feed (audit layer only) to tag voluntary
   delist / forced settlement / rename + anchor the qualitative tail events.

## 3. Minimal-sufficient feature set (a daily book stores daily *descriptors* of the intraday path)
Already have: `close` (perp), `spot_close`, `funding`. **Add (load-bearing in bold):**
- **`worst_basis`** = min over the day of cumulative `(spot_1m_ret − perp_1m_ret)` (basis MAE) — the single
  number that converts "always-holding fiction" → "would I survive the day" (a delta-neutral pos is
  liquidated by the *spread* under finite margin). From 1m spot + perp klines. **Free.**
- **delisted-coin inclusion + `delisted` marker** (ticker_membership `end_date` at delist) + **funding on
  delisted symbols**. **Free.** (Phase-0 alone collapses most of the fake Sharpe — LUNA-class losses are
  currently DELETED from the panel.)
- `basis_range`, `oi_eod`/`premium_eod` (free diagnostics); `liq_intensity`/`liq_flag` (**paid, lossy**).

## 4. Volume + effort
- Raw 1m (throwaway): **tens of GB zipped / ~100–180 GB unzipped** (perp-1m + spot-1m, ever-listed incl.
  delisted; prefer monthly files, ~30× fewer requests). funding: **<100 MB total** (acquire first). 
- Reduced (stored in DuckDB): **~0.1 GB** net (~1000:1 compression — one daily MAE replaces 1440 bars × 2).
- Effort ≈ **5–6 engineering days** (aggregator + delisted-aware S3 walker + single-writer Dagster asset +
  backtest consumption). Pilot ~30 symbols incl. LUNA/2022-Q2 before the full backfill.

## 5. Storage + single-writer-safe ingestion (zero migration)
New features are just new `feature` values in the existing long-format store; delist via existing
`upsert_membership`. **Fan-out compute, fan-in write:** N parallel workers (NO DuckDB handle) download +
aggregate per symbol-shard → emit feature rows → a SINGLE serialized writer commits to DuckDB. (This is the
workflow-shaped parallel work.)

## 6. Modeling the carry tail in the (still daily) book
- **worst-basis marks:** when the day's `worst_basis` breached an intraday margin/liq threshold for a held
  symbol, realize the basis loss / margin-stop haircut instead of the benign close.
- **liq/ADL haircut:** on `liq_flag` (or free fallback: `metrics` OI-collapse + one-sided taker), apply a
  parametric slippage/ADL haircut to the short-perp leg (× a stress multiplier, since liq data is a lower bound).
- **delisted-coin loss:** PIT membership keeps a dying coin in-universe through its death → it pays its carry
  loss then exits (force-settle at last/settlement mark; respect the reduce-only pre-suspension window).

## HONEST residual limit (never ingested)
Counterparty/exchange failure (FTX), withdrawal/settlement freezes, ADL per-position socialization, collateral
de-peg = credit/operational events with **no market-data artifact at any vendor**. → a **scenario haircut**
(probabilistic venue-collateral loss, position-locked-N-days), anchored to the announcement audit — never
claimed as backtested. The improved backtest **narrows but never closes** the gap; **live `xsfunding_nav`
stays ground truth.**

## Phased plan (cheapest meaningful improvement first)
- **Phase 0 — survivorship-free universe** (~free, ~1–2d, highest ROI): enumerate 876 prefixes → intervals →
  rename-stitch + relisting-gap map → backfill delisted close+funding (+spot leg) → set membership end_date.
  *Falsification gate: re-run xsfunding on the survivorship-free universe; confirm SR drops.*
- **Phase 1 — intraday `worst_basis` marks** (free, ~2–3d): 1m→MAE aggregator (pure + tests), pilot 30 sym
  incl. LUNA 2022-Q2, confirm the intraday tail reappears, then full backfill via fan-out/single-writer.
- **Phase 2 — free liq/ADL proxy** (free, ~1–2d): `metrics` OI-collapse + one-sided-taker → parametric haircut.
- **Phase 3 — paid Tardis liq tape** (only if Phases 0–2 leave a material gap): validate free first-of-month
  LUNA-era samples; yearly/Business billing needed for 2020–2026 (monthly billing only grants a few months).
- **Phase 4 — scenario layer** (always, never ingested): counterparty/de-peg/ADL haircuts; live nav = truth.

## UNCERTAIN (flagged by the adversarial verifier)
- Symbol count **876** (live single page) supersedes 922/815; volume keyed to <876 → revise upward.
- Tardis "~$599/mo" is a single third-party blog (order-of-magnitude only); CONFIRMED from Tardis: $300 min
  order, survivorship guarantee, forceOrder from 2020-01-07, billing-interval = history-depth. Funding start
  likely ~2020-05 (not 2020-02). Kaiko/Amberdata/CoinAPI/CCData pricing = indicative only.

Files to extend if built: `adapters/warehouse/duckdb_feature_store.py`, `ports/warehouse.py`,
`application/{funding_ingest,spot_ingest,xsfunding_strategy,xsfunding_replay}.py`, `adapters/orchestration/assets.py`.
