"""STEP 1 of the "spread-as-a-strat" research — a reusable survivorship-free LIQUIDITY PANEL + a
FALSIFICATION test of the STATIC ILLIQUIDITY PREMIUM, NET of real measured costs.

FALSIFICATION-FIRST. The classic cross-asset "illiquidity premium" (illiquid assets out-earn liquid ones as
compensation for transaction-cost risk) is a strong prior in equities. This module asks whether it survives
in the survivorship-free Bybit perp universe ONCE you pay the spread you are supposedly harvesting. A clean
"the premium dies net of cost" (FALSIFY) is the EXPECTED, VALUABLE outcome — it RULES the static premium OUT
as a tradeable crypto edge (the cost trap), which is exactly what falsification-first research is for. CONFIRM
is reserved for the rare case where the NET-OOS economics genuinely hold AND it is not a rebrand of a live
edge.

THE LIQUIDITY PANEL (reusable — later steps build on it)
--------------------------------------------------------
`build_liquidity_panel(store, universe=...)` returns, per coin per day from `bybit_features` (survivorship
-free, the full universe including coins for the days they traded then delisted):
  * CS spread (bps) — a TRAILING-window Corwin–Schultz (2012) effective spread (reuses
    `bybit_liquidity_sweep.corwin_schultz_spread_bps`: median of the daily estimates, negatives clamped to 0).
    The TRANSACTION-COST measure of illiquidity.
  * Amihud illiquidity = |daily return| / daily dollar-volume — the PRICE-IMPACT measure of illiquidity
    (complements the spread). Daily dollar-volume = the `turnover` feature.
  * close-to-close daily return per coin (for forming portfolios).
Structure: `{epoch_day: {coin: (spread_bps, amihud, ret)}}`. READ-ONLY, memory-bounded (a `universe` restricts
the reads to `read_panel(universe, feature)` = a `WHERE symbol IN (...)` read), deterministic, pure.

THE STATIC-PREMIUM FALSIFICATION (`static_illiquidity_premium`)
--------------------------------------------------------------
Cross-sectional sort with NO lookahead — rank the eligible universe on info up to day t, hold the t+1 return:
  * each day rank by trailing CS spread (and SEPARATELY by Amihud) into QUINTILES (equal-weight within);
  * the long-short = Q5 (most illiquid) − Q1 (most liquid); the DIRECTION is DATA-DRIVEN (the sign is chosen
    IN-SAMPLE on the TRAIN gross mean, so a negative "illiquidity premium" = a liquidity premium is reported
    with its own sign, not assumed);
  * GROSS daily long-short return; NET = GROSS − the MEASURED per-coin spread cost on the rebalance turnover
    (reuses `bybit_liquidity_sweep.coin_cost_bps` = Bybit's published taker fee + half the coin's measured CS
    spread). Holding/trading illiquid coins costs the spread you are harvesting — this is the whole point.
  * metrics: annualised Sharpe + total return GROSS and NET; OOS/walk-forward (TRAIN/TEST split, report TEST);
    NON-OVERLAPPING t-stat (the 1-day-held long-short series is already non-overlapping); maxDD; turnover; avg
    cost drag; correlation of the NET daily series to each live edge (tstrend/unlock/xsfunding/positioning).
  * VERDICT: CONFIRM only if NET-OOS Sharpe is meaningfully positive AND the non-overlapping t-stat holds AND
    it is not a rebrand of a live edge (low |corr|). Otherwise FALSIFY with the numbers. The EXPECTED outcome
    is FALSIFY: a gross premium may exist but NET <= 0 — the cost trap.

REUSE: the sweep's `corwin_schultz_spread_bps` + `coin_cost_bps` (the measured, parameter-free cost) and the
`bybit_book_eval._bounded_store` memory seam. READ-ONLY — no `store.write_*`, no `PaperRepo` (a WriteTripwire
test proves it). NO network — external inputs are injected and the panel is pure.
"""
from __future__ import annotations

import math
import statistics as st
from typing import Any, Protocol

from fxhnt.application.bybit_liquidity_sweep import coin_cost_bps, corwin_schultz_spread_bps

_PERIODS_PER_YEAR = 365

# Trailing window (days) for the rolling Corwin–Schultz spread — long enough to median out the per-pair CS
# noise, short enough to track a regime change in a coin's liquidity. A research knob, not a tuned parameter.
_CS_WINDOW = 30

# Number of cross-sectional buckets. Quintiles is the canonical illiquidity-sort granularity.
_N_QUINTILES = 5

# Adoption bars for the CONFIRM verdict (a premium must clear ALL of them; else FALSIFY).
_SHARPE_BAR = 0.5      # NET-OOS (TEST) Sharpe must be at least this to be "meaningfully positive".
_T_BAR = 2.0           # |non-overlapping t-stat| of the NET daily long-short must hold.
_CORR_BAR = 0.6        # max |corr| to any live edge below this = not a rebrand; at/above = a rebrand.


class _FeatureStore(Protocol):
    def crypto_close_panel(self, *, suffix: str = "USDT") -> dict[str, dict[int, float]]: ...
    def read_panel(self, symbols: list[str], feature: str) -> dict[str, dict[int, float]]: ...


# --- THE LIQUIDITY PANEL -------------------------------------------------------------------------

def build_liquidity_panel(store: _FeatureStore, *, universe: set[str] | None = None,
                          suffix: str = "USDT", cs_window: int = _CS_WINDOW,
                          amihud_scale: float = 1e9) -> dict[int, dict[str, tuple[float, float | None, float]]]:
    """Build `{epoch_day: {coin: (spread_bps, amihud, ret)}}` from `bybit_features`, survivorship-free.

    Per coin per day d (with a prior trading day dp):
      * ret = close[d]/close[dp] − 1 — the close-to-close return realised OVER day d.
      * spread_bps = trailing Corwin–Schultz over the coin's HIGH/LOW for the days in (d-cs_window, d] — uses
        ONLY days <= d (no lookahead). The transaction-cost illiquidity measure.
      * amihud = |ret| / dollar_volume[d] × `amihud_scale` (the `turnover` feature is dollar-volume). The
        price-impact illiquidity measure. `None` when dollar-volume is missing/0 (uncomputable, not faked).
    A coin contributes ONLY for the days it actually traded (the survivorship-free shape: a coin that delists
    is simply absent from later days). READ-ONLY; memory-bounded when a `universe` is given (reads go through
    `read_panel(universe, feature)`). `amihud_scale` only rescales the Amihud level for readability — it is a
    positive monotone transform, so it never changes any cross-sectional RANK or verdict."""
    if universe is not None:
        symbols = sorted(s for s in universe if s.endswith(suffix))
        if not symbols:
            return {}
        close_panel = {s: v for s, v in store.read_panel(symbols, "close").items() if v}
    else:
        close_panel = store.crypto_close_panel(suffix=suffix)
        symbols = sorted(close_panel)
    if not close_panel:
        return {}

    high_panel = store.read_panel(symbols, "high")
    low_panel = store.read_panel(symbols, "low")
    turn_panel = store.read_panel(symbols, "turnover")

    panel: dict[int, dict[str, tuple[float, float | None, float]]] = {}
    for sym in symbols:
        closes = close_panel.get(sym, {})
        if len(closes) < 2:
            continue
        highs = high_panel.get(sym, {})
        lows = low_panel.get(sym, {})
        turns = turn_panel.get(sym, {})
        days = sorted(closes)
        for i in range(1, len(days)):
            d, dp = days[i], days[i - 1]
            c, cp = closes[d], closes[dp]
            if cp <= 0.0 or c <= 0.0:
                continue
            ret = c / cp - 1.0
            if not math.isfinite(ret):
                continue
            # trailing CS spread on the last <= cs_window days with both high & low present, up to & incl. d.
            win = [x for x in days if x <= d and x in highs and x in lows][-cs_window:]
            spread = corwin_schultz_spread_bps([highs[x] for x in win], [lows[x] for x in win])
            dv = turns.get(d, 0.0)
            amihud = (abs(ret) / dv * amihud_scale) if (dv and dv > 0.0) else None
            panel.setdefault(d, {})[sym] = (spread, amihud, ret)
    return panel


# --- metrics (annualised Sharpe / total return / maxDD / non-overlapping t-stat / Pearson) -------

def _sharpe(rets: list[float]) -> float:
    if len(rets) < 2:
        return 0.0
    sd = st.pstdev(rets)
    return (st.mean(rets) / sd * math.sqrt(_PERIODS_PER_YEAR)) if sd > 0 else 0.0


def _total_return(rets: list[float]) -> float:
    eq = 1.0
    for r in rets:
        eq *= (1.0 + r)
    return eq - 1.0


def _max_dd(rets: list[float]) -> float:
    eq = peak = 1.0
    dd = 0.0
    for r in rets:
        eq *= (1.0 + r)
        peak = max(peak, eq)
        if peak > 0:
            dd = min(dd, eq / peak - 1.0)
    return dd


def _t_stat(rets: list[float]) -> float:
    """Non-overlapping t-stat of the daily long-short series. The 1-day-held long-short return on day t uses
    a fresh formation at t-1 and a fresh realisation at t, so the daily series is NON-OVERLAPPING — the plain
    mean/(sd/sqrt(n)) is the honest t-stat (no overlap inflation)."""
    n = len(rets)
    if n < 2:
        return 0.0
    sd = st.stdev(rets)   # sample stdev for the t-stat
    return (st.mean(rets) / (sd / math.sqrt(n))) if sd > 0 else 0.0


def _pearson(a: dict[int, float], b: dict[int, float]) -> float:
    days = sorted(set(a) & set(b))
    if len(days) < 3:
        return 0.0
    xs = [a[d] for d in days]
    ys = [b[d] for d in days]
    mx, my = st.mean(xs), st.mean(ys)
    num = sum((x - mx) * (y - my) for x, y in zip(xs, ys, strict=True))
    dx = math.sqrt(sum((x - mx) ** 2 for x in xs))
    dy = math.sqrt(sum((y - my) ** 2 for y in ys))
    return (num / (dx * dy)) if dx > 0 and dy > 0 else 0.0


def _split_day(days: list[int], train_frac: float) -> int:
    if len(days) < 4:
        return (days[-1] + 1) if days else 1
    idx = max(1, min(len(days) - 1, int(len(days) * train_frac)))
    return days[idx]


# --- the cross-sectional long-short (one sort key) -----------------------------------------------

def _quintile_long_short(panel: dict[int, dict[str, tuple[float, float | None, float]]], *,
                         metric_idx: int, taker_fee_bps: float, n_buckets: int,
                         ) -> tuple[dict[int, float], dict[int, float], dict[int, float], float, float]:
    """Build the per-day GROSS-raw (Q5−Q1) long-short series, the per-day rebalance COST series, and the per-
    day turnover, for ONE sort key (`metric_idx` 0=spread, 1=amihud).

    NO LOOKAHEAD: at each formation day t we rank the eligible coins by their metric AT t and HOLD the return
    realised at the NEXT panel day t+1. Eligible = a coin present at both t and t+1, with a finite metric at t
    and a finite return at t+1. Quintile membership is equal-weight (+1/|Q5| long, −1/|Q1| short).

    The COST reuses `coin_cost_bps` (Bybit taker fee + half the coin's MEASURED CS spread at t) on the day's
    |Δweight| turnover — the measured spread you pay to rebalance into/out of illiquid coins. Returns
    (gross_raw_series, cost_series, turnover_series, turnover_total, cost_drag_bps)."""
    days = sorted(panel)
    gross_raw: dict[int, float] = {}
    cost_series: dict[int, float] = {}
    turnover_series: dict[int, float] = {}
    prev_w: dict[str, float] = {}
    turnover_total = 0.0
    cost_bps_weighted = 0.0
    traded_w = 0.0
    for i in range(len(days) - 1):
        t, t1 = days[i], days[i + 1]
        row_t, row_t1 = panel[t], panel[t1]
        elig = []
        for coin, (sp, am, _ret) in row_t.items():
            metric = (sp, am)[metric_idx]
            if metric is None or not math.isfinite(metric):
                continue
            if coin not in row_t1:
                continue
            r1 = row_t1[coin][2]
            if not math.isfinite(r1):
                continue
            elig.append((metric, coin, r1))
        if len(elig) < n_buckets:
            # not enough names to form quintiles → no position; weights decay to flat (close everything).
            new_w: dict[str, float] = {}
        else:
            elig.sort(key=lambda x: x[0])
            q = len(elig) // n_buckets
            q1 = elig[:q]                 # most LIQUID (low metric)
            q5 = elig[-q:]                # most ILLIQUID (high metric)
            q1_ret = st.mean(r for (_m, _c, r) in q1)
            q5_ret = st.mean(r for (_m, _c, r) in q5)
            gross_raw[t1] = q5_ret - q1_ret   # illiquid − liquid (raw; sign chosen later, in-sample)
            new_w = {}
            for (_m, c, _r) in q5:
                new_w[c] = new_w.get(c, 0.0) + 1.0 / len(q5)
            for (_m, c, _r) in q1:
                new_w[c] = new_w.get(c, 0.0) - 1.0 / len(q1)
        # rebalance cost on |Δweight| at the coin's MEASURED cost (fee + half its CS spread at t).
        day_cost = 0.0
        day_turnover = 0.0
        for coin in set(new_w) | set(prev_w):
            dw = abs(new_w.get(coin, 0.0) - prev_w.get(coin, 0.0))
            if dw <= 0.0:
                continue
            sp_t = row_t.get(coin, (0.0, None, 0.0))[0]
            cbps = coin_cost_bps(effective_spread_bps=sp_t, taker_fee_bps=taker_fee_bps)
            day_cost += dw * cbps / 1e4
            day_turnover += dw
            turnover_total += dw
            cost_bps_weighted += dw * cbps
            traded_w += dw
        if t1 in gross_raw:
            cost_series[t1] = day_cost
            turnover_series[t1] = day_turnover
        prev_w = new_w
    cost_drag_bps = (cost_bps_weighted / traded_w) if traded_w > 0 else 0.0
    return gross_raw, cost_series, turnover_series, turnover_total, cost_drag_bps


def _evaluate_sort(panel: dict[int, dict[str, tuple[float, float | None, float]]], *, metric_idx: int,
                   train_frac: float, taker_fee_bps: float, n_buckets: int,
                   edge_series: dict[str, dict[int, float]] | None) -> dict[str, Any]:
    """The full GROSS/NET/OOS/t-stat/corr/verdict block for ONE sort key."""
    gross_raw, cost_series, _turn_series, turnover_total, cost_drag_bps = _quintile_long_short(
        panel, metric_idx=metric_idx, taker_fee_bps=taker_fee_bps, n_buckets=n_buckets)
    days = sorted(gross_raw)
    if len(days) < 4:
        return {"available": False, "verdict": "FALSIFY", "net_test_sharpe": 0.0, "gross_full_sharpe": 0.0,
                "t_stat": 0.0, "direction_sign": 1, "cost_drag_bps": cost_drag_bps,
                "turnover": turnover_total, "corr_to_edges": {}, "net_series": {},
                "reason": "too few cross-sectional days to form a long-short"}

    test_lo = _split_day(days, train_frac)
    train_days = [d for d in days if d < test_lo]
    # DATA-DRIVEN direction: sign of the TRAIN gross mean (in-sample), applied to BOTH train & test (no peek).
    train_mean = st.mean(gross_raw[d] for d in train_days) if train_days else st.mean(gross_raw.values())
    sign = 1 if train_mean >= 0 else -1

    gross = {d: sign * gross_raw[d] for d in days}
    net = {d: sign * gross_raw[d] - cost_series.get(d, 0.0) for d in days}

    net_test = [net[d] for d in days if d >= test_lo]
    gross_all = [gross[d] for d in days]
    net_all = [net[d] for d in days]

    net_test_sharpe = _sharpe(net_test)
    gross_full_sharpe = _sharpe(gross_all)
    t_stat = _t_stat(net_test)

    corr_to_edges: dict[str, float] = {}
    if edge_series:
        for name, series in edge_series.items():
            corr_to_edges[name] = _pearson(net, series)
    max_abs_corr = max((abs(c) for c in corr_to_edges.values()), default=0.0)

    confirm = (net_test_sharpe >= _SHARPE_BAR and abs(t_stat) >= _T_BAR
               and (not corr_to_edges or max_abs_corr < _CORR_BAR))
    verdict = "CONFIRM" if confirm else "FALSIFY"

    return {
        "available": True,
        "verdict": verdict,
        "direction_sign": sign,
        "gross_full_sharpe": gross_full_sharpe,
        "net_full_sharpe": _sharpe(net_all),
        "net_test_sharpe": net_test_sharpe,
        "gross_total_return": _total_return(gross_all),
        "net_total_return": _total_return(net_all),
        "net_test_total_return": _total_return(net_test),
        "t_stat": t_stat,
        "max_dd": _max_dd(net_all),
        "turnover": turnover_total,
        "cost_drag_bps": cost_drag_bps,
        "corr_to_edges": corr_to_edges,
        "max_abs_corr": max_abs_corr,
        "n_days": len(days),
        "net_series": net,
        "gross_series": gross,
    }


def static_illiquidity_premium(panel: dict[int, dict[str, tuple[float, float | None, float]]], *,
                               train_frac: float = 0.6, taker_fee_bps: float = 5.5,
                               n_buckets: int = _N_QUINTILES,
                               edge_series: dict[str, dict[int, float]] | None = None) -> dict[str, Any]:
    """Falsify the STATIC ILLIQUIDITY PREMIUM on the liquidity `panel`, NET of measured cost.

    For each sort key (CS spread, then Amihud): form the daily Q5−Q1 long-short with NO lookahead (sort on
    metric at t, hold the t+1 return), pick the DIRECTION in-sample on the TRAIN gross mean, charge the
    MEASURED per-coin spread cost (`coin_cost_bps`) on the rebalance turnover, and report GROSS+NET Sharpe /
    total return, the OOS (TEST) NET Sharpe, the NON-OVERLAPPING t-stat, maxDD, turnover, avg cost drag, and
    the correlation of the NET daily series to each live edge in `edge_series`. VERDICT per sort key: CONFIRM
    only if NET-OOS Sharpe is meaningfully positive AND the t-stat holds AND it is not a rebrand (low |corr|);
    else FALSIFY. The EXPECTED outcome is FALSIFY — a gross premium eroded to <= 0 by the spread it harvests
    (the cost trap). Returns `{available, spread{...}, amihud{...}}`. Pure; writes nothing."""
    if not panel:
        return {"available": False, "reason": "empty liquidity panel"}
    return {
        "available": True,
        "spread": _evaluate_sort(panel, metric_idx=0, train_frac=train_frac, taker_fee_bps=taker_fee_bps,
                                 n_buckets=n_buckets, edge_series=edge_series),
        "amihud": _evaluate_sort(panel, metric_idx=1, train_frac=train_frac, taker_fee_bps=taker_fee_bps,
                                 n_buckets=n_buckets, edge_series=edge_series),
    }


# =================================================================================================
# STEP 2 — SPREAD-DETERIORATION SHORTS (the "abandonment" edge), NET of REAL entry + exit cost.
# =================================================================================================
#
# WHY (and what Step 1 taught us). Step 1 FALSIFIED the STATIC illiquidity premium: the GROSS premium is real
# but 100% COST-TRAPPED — you cannot *hold* illiquid coins because the ~spread you must cross dwarfs the move
# (it fails LARGE). Step 2 ESCAPES the trap with a different thesis: do not hold illiquid coins for a premium —
# SHORT coins AS THEIR LIQUIDITY STARTS DETERIORATING, while they are STILL TRADEABLE (cheap entry), and
# capture the large ABANDONMENT move down. This is conceptually adjacent to the unlock-shorts edge (dilution →
# abandonment) that already cleared the gauntlet — here it is liquidity → abandonment. FALSIFICATION-FIRST: a
# clean CONFIRM or FALSIFY are both valuable; the verdict tracks the NET economics, never a hard-coded answer.
#
# THE SIGNAL (no lookahead). Per coin at formation day t, using ONLY days <= t: spread WIDENING — the z-score
# of the coin's CS spread vs its own trailing mean/std (a regime change in its OWN liquidity, not a cross-
# sectional level). Optionally combined with Amihud RISING (volume drying) into an "abandonment" composite
# (the mean of the two z-scores). Higher = faster deterioration.
#
# THE TRADEABLE-AT-ENTRY FILTER (the cost-escape — the whole point). Only short coins that are STILL liquid
# enough to ENTER at t: entry-day CS spread BELOW a cap AND dollar-volume ABOVE a floor. We short BEFORE the
# spread blows out; a coin that is already illiquid at entry is EXCLUDED (you would pay Step 1's cost trap to
# get in). Both thresholds are research knobs (reported, not tuned).
#
# THE PORTFOLIO (no lookahead, non-overlapping). At each formation day t we SHORT the top-decile most-
# deteriorating, tradeable-at-entry coins (equal-weight) and go LONG the matched liquid universe mean (market-
# neutral, so it is a clean long/short, not a market bet). We HOLD T panel-days and re-form on a NON-
# OVERLAPPING T-day grid (formations at days[0], days[T], days[2T], ...), so the period series is genuinely
# non-overlapping → an honest t-stat. The DIRECTION is FIXED by the thesis (short the deteriorating) — it is
# NOT data-driven; a negative result is a FALSIFY of the abandonment-short thesis, not a sign to flip.
#
# THE REAL COST (the honest part). Each leg pays the per-day MEASURED spread at ENTRY (spread at t) AND at
# EXIT (spread at t+T) + the published taker fee per leg (`coin_cost_bps`). Because the spread WIDENS during
# the hold, the EXIT is costlier than the ENTRY — the abandonment move must clear BOTH legs NET. We use the
# panel's ACTUAL per-day spread at t and at t+T, never a static one.
#
# VERDICT. CONFIRM only if NET-OOS (TEST) Sharpe >= 0.5 AND |non-overlapping t| >= 2 AND max|corr-to-edge| <
# 0.6 (NOT a rebrand of a live edge — especially unlock) AND it passes LARGE+FREQUENT+CONCENTRATED (the
# abandonment move clears the round-trip spread; enough events/yr to compound; concentrated in sizeable
# names). Else FALSIFY with the numbers. READ-ONLY; pure; network-free.

# Trailing window (days) for the per-coin spread/Amihud deterioration z-score. Long enough to define a stable
# baseline, short enough to react to a regime change in the coin's own liquidity.
_DET_WINDOW = 20
# Minimum trailing observations before a deterioration z-score is computed (no score on too-short history).
_DET_MIN_OBS = 10
# Cross-sectional bucket count for the SHORT basket — decile (top 10% most-deteriorating) is the canonical
# "tail" granularity (mirrors the unlock-shorts concentration).
_N_DECILES = 10
# Tradeable-at-entry caps (the cost-escape). A coin must be BELOW this CS spread AND ABOVE this dollar-volume
# at entry to be shortable — i.e. still liquid enough to get in cheaply. Research knobs, reported not tuned.
_MAX_ENTRY_SPREAD_BPS = 60.0
_MIN_ENTRY_DOLLAR_VOL = 5_000_000.0
# LARGE+FREQUENT+CONCENTRATED bars for the CONFIRM verdict (on TOP of Sharpe/t/corr).
_MIN_EVENTS_PER_YEAR = 12.0     # FREQUENT: enough shorted names per year to compound.
_MIN_NAMES_PER_PERIOD = 1.0     # CONCENTRATED: at least one sizeable name actually traded per period.

# A deterioration-panel row carries the Step-1 triple PLUS dollar-volume (needed by the tradeable-at-entry
# floor): (spread_bps, amihud, ret, dollar_vol). dollar_vol/amihud are None when uncomputable (not faked).
_DetRow = tuple[float, float | None, float, float | None]


def build_deterioration_panel(store: _FeatureStore, *, universe: set[str] | None = None,
                              suffix: str = "USDT", cs_window: int = _CS_WINDOW,
                              amihud_scale: float = 1e9) -> dict[int, dict[str, _DetRow]]:
    """Build `{epoch_day: {coin: (spread_bps, amihud, ret, dollar_vol)}}` — the Step-1 liquidity panel PLUS
    per-day dollar-volume (the `turnover` feature), which the tradeable-at-entry floor needs.

    REUSES `build_liquidity_panel` verbatim for the spread/Amihud/return triple (no re-implementation), then
    attaches each coin's dollar-volume on each day from the same survivorship-free `turnover` read. READ-ONLY,
    memory-bounded when a `universe` is given. `dollar_vol` is `None` when missing/0 (uncomputable, not faked
    — such a coin then fails the dollar-volume floor and is never shorted)."""
    base = build_liquidity_panel(store, universe=universe, suffix=suffix, cs_window=cs_window,
                                 amihud_scale=amihud_scale)
    if not base:
        return {}
    symbols = sorted({c for row in base.values() for c in row})
    turn_panel = store.read_panel(symbols, "turnover")
    out: dict[int, dict[str, _DetRow]] = {}
    for d, row in base.items():
        for coin, (sp, am, ret) in row.items():
            dv = turn_panel.get(coin, {}).get(d)
            out.setdefault(d, {})[coin] = (sp, am, ret, (dv if (dv and dv > 0.0) else None))
    return out


def _ann_sharpe(rets: list[float], periods_per_year: float) -> float:
    """Annualised Sharpe of a series of PERIOD returns (each return spans `hold_days` days, so the annualiser
    is sqrt(365/hold_days), NOT the daily sqrt(365))."""
    if len(rets) < 2:
        return 0.0
    sd = st.pstdev(rets)
    return (st.mean(rets) / sd * math.sqrt(periods_per_year)) if sd > 0 else 0.0


def _zscore_last(window: list[float]) -> float | None:
    """z-score of the LAST element of `window` vs the window's mean/std (population). `None` when the window
    is too short or has zero dispersion (a flat series has no deterioration signal). Uses only the window
    given — the caller passes a trailing window ending at the formation day, so there is no lookahead."""
    if len(window) < 2:
        return None
    mu = st.mean(window)
    sd = st.pstdev(window)
    if sd <= 0.0:
        return None
    z = (window[-1] - mu) / sd
    return z if math.isfinite(z) else None


def deterioration_scores(panel: dict[int, dict[str, _DetRow]], *, det_window: int = _DET_WINDOW,
                         min_obs: int = _DET_MIN_OBS, use_amihud: bool = True,
                         ) -> dict[int, dict[str, float]]:
    """The per-coin per-day DETERIORATION score `{day: {coin: score}}`, higher = faster deterioration.

    For each coin we walk its OWN days in order and, at each day d, take the trailing `det_window` spread
    observations ENDING AT d (so only days <= d — NO lookahead) and compute the z-score of the spread at d
    vs that window. With `use_amihud`, the same trailing z-score of the Amihud illiquidity (volume drying) is
    averaged in when available → the "abandonment" composite. A coin with a flat / too-short spread history
    has no score that day (it is not deteriorating). Pure; deterministic."""
    coin_days: dict[str, dict[int, tuple[float, float | None]]] = {}
    for d, row in panel.items():
        for c, (sp, am, _ret, _dv) in row.items():
            coin_days.setdefault(c, {})[d] = (sp, am)
    scores: dict[int, dict[str, float]] = {}
    for c, ser in coin_days.items():
        days = sorted(ser)
        sp_hist: list[float] = []
        am_hist: list[float | None] = []
        for d in days:
            sp, am = ser[d]
            sp_hist.append(sp)
            am_hist.append(am)
            win_sp = [x for x in sp_hist[-det_window:] if math.isfinite(x)]
            if len(win_sp) < min_obs:
                continue
            z_sp = _zscore_last(win_sp)
            if z_sp is None:
                continue
            comps = [z_sp]
            if use_amihud:
                win_am = [x for x in am_hist[-det_window:] if x is not None and math.isfinite(x)]
                if len(win_am) >= min_obs:
                    z_am = _zscore_last(win_am)
                    if z_am is not None:
                        comps.append(z_am)
            score = sum(comps) / len(comps)
            if math.isfinite(score):
                scores.setdefault(d, {})[c] = score
    return scores


def tradeable_at_entry(panel: dict[int, dict[str, _DetRow]], t: int, hold_slice: list[int], *,
                       max_entry_spread_bps: float = _MAX_ENTRY_SPREAD_BPS,
                       min_entry_dollar_vol: float = _MIN_ENTRY_DOLLAR_VOL) -> list[str]:
    """The TRADEABLE-AT-ENTRY liquid universe at formation day `t` (the cost-escape). A coin qualifies only if
    its entry-day CS spread is BELOW the cap AND its entry-day dollar-volume is ABOVE the floor AND it is
    present through the entire `hold_slice` (so entry/exit spread + every hold-day return exist). A coin whose
    spread has ALREADY blown out (or whose volume has dried up) at entry is EXCLUDED — shorting it would pay
    Step 1's cost trap. Pure; deterministic."""
    out: list[str] = []
    for c, (sp, _am, _ret, dv) in panel[t].items():
        if sp is None or not math.isfinite(sp) or sp > max_entry_spread_bps:
            continue
        if dv is None or dv < min_entry_dollar_vol:
            continue
        if not all(c in panel[d] for d in hold_slice):
            continue
        out.append(c)
    return out


def pick_shorts(liq: list[str], scores_at_t: dict[str, float], n_buckets: int = _N_DECILES) -> list[str]:
    """The SHORT basket = the top-(1/`n_buckets`) most-deteriorating coins among the tradeable-at-entry
    universe `liq`, by deterioration score at t. Empty when fewer than `n_buckets` coins carry a score (cannot
    form the tail). Pure; deterministic (stable sort → reproducible tie-breaks)."""
    scored = [(scores_at_t[c], c) for c in liq if c in scores_at_t and math.isfinite(scores_at_t[c])]
    if len(scored) < n_buckets:
        return []
    scored.sort(key=lambda x: x[0])
    k = max(1, len(scored) // n_buckets)
    return [c for _s, c in scored[-k:]]


def deterioration_shorts(panel: dict[int, dict[str, _DetRow]], *, hold_days: int,
                         det_window: int = _DET_WINDOW, min_obs: int = _DET_MIN_OBS,
                         use_amihud: bool = True, max_entry_spread_bps: float = _MAX_ENTRY_SPREAD_BPS,
                         min_entry_dollar_vol: float = _MIN_ENTRY_DOLLAR_VOL, n_buckets: int = _N_DECILES,
                         taker_fee_bps: float = 5.5, train_frac: float = 0.6,
                         edge_series: dict[str, dict[int, float]] | None = None) -> dict[str, Any]:
    """Run the spread-DETERIORATION SHORT strategy on the deterioration `panel` for ONE hold horizon, NET of
    the REAL entry+exit spread cost. SHORT the top-decile most-deteriorating, tradeable-at-entry coins, LONG
    the matched liquid-universe mean (market-neutral), hold `hold_days`, re-form on a NON-OVERLAPPING grid.

    NO LOOKAHEAD: the deterioration score at t uses only days <= t; the held return is compounded over the
    NEXT `hold_days` panel-days (t+1..t+T). REAL COST: each leg pays the MEASURED spread at ENTRY (t) AND at
    EXIT (t+T) + taker fee — the spread widens during the hold, so the exit is costlier. The DIRECTION is
    FIXED (short the deteriorating) — a negative result FALSIFIES the thesis, it is not flipped. Returns the
    GROSS/NET Sharpe + total return, OOS (TEST) NET Sharpe, non-overlapping t-stat, maxDD, turnover, avg
    entry-vs-exit cost drag, #names/period, corr of the NET period series to each live edge, and the verdict.
    Pure; writes nothing."""
    days = sorted(panel)
    ppy = _PERIODS_PER_YEAR / hold_days
    unavail = {"available": False, "verdict": "FALSIFY", "hold_days": hold_days,
               "net_test_sharpe": 0.0, "gross_full_sharpe": 0.0, "t_stat": 0.0, "max_dd": 0.0,
               "turnover": 0.0, "avg_entry_drag_bps": 0.0, "avg_exit_drag_bps": 0.0,
               "avg_names_per_period": 0.0, "events_per_year": 0.0, "n_periods": 0,
               "corr_to_edges": {}, "max_abs_corr": 0.0, "net_series": {},
               "reason": "too few panel days for the hold horizon"}
    if len(days) <= hold_days:
        return unavail

    scores = deterioration_scores(panel, det_window=det_window, min_obs=min_obs, use_amihud=use_amihud)

    gross: dict[int, float] = {}
    cost_series: dict[int, float] = {}
    net: dict[int, float] = {}
    entry_bps_period: list[float] = []
    exit_bps_period: list[float] = []
    names_per_period: list[int] = []
    windows: list[tuple[int, int]] = []   # (formation_day, exit_day) for accepted periods
    hold_slices: list[list[int]] = []
    total_short_names = 0

    for i in range(0, len(days) - hold_days, hold_days):
        t = days[i]
        ex = days[i + hold_days]
        hold_slice = days[i + 1:i + hold_days + 1]
        sc = scores.get(t, {})

        # tradeable-at-entry LIQUID universe (the cost-escape) + the top-decile SHORT basket.
        liq = tradeable_at_entry(panel, t, hold_slice, max_entry_spread_bps=max_entry_spread_bps,
                                 min_entry_dollar_vol=min_entry_dollar_vol)
        if len(liq) < n_buckets:
            continue
        shorts = pick_shorts(liq, sc, n_buckets=n_buckets)
        if not shorts:
            continue

        def _fwd(c: str, _slice: list[int] = hold_slice) -> float | None:
            r = 1.0
            for d in _slice:
                rr = panel[d][c][2]
                if not math.isfinite(rr):
                    return None
                r *= (1.0 + rr)
            return r - 1.0

        long_rets = [r for r in (_fwd(c) for c in liq) if r is not None]
        short_rets = [r for r in (_fwd(c) for c in shorts) if r is not None]
        if not long_rets or not short_rets:
            continue
        # market-neutral: LONG the liquid-universe mean, SHORT the deteriorating decile.
        g = st.mean(long_rets) - st.mean(short_rets)

        # REAL cost: round-trip per leg = entry spread (t) + exit spread (t+T) + fee per leg.
        def _leg_cost(cset: list[str], _t: int = t, _ex: int = ex) -> tuple[float, float]:
            e = x = 0.0
            for c in cset:
                e += coin_cost_bps(effective_spread_bps=panel[_t][c][0], taker_fee_bps=taker_fee_bps)
                x += coin_cost_bps(effective_spread_bps=panel[_ex][c][0], taker_fee_bps=taker_fee_bps)
            n = len(cset)
            return (e / n, x / n) if n else (0.0, 0.0)

        se, sx = _leg_cost(shorts)   # short leg avg entry / exit cost (bps)
        le, lx = _leg_cost(liq)      # long leg avg entry / exit cost (bps)
        period_entry_bps = se + le
        period_exit_bps = sx + lx
        period_cost = (period_entry_bps + period_exit_bps) / 1e4

        gross[ex] = g
        cost_series[ex] = period_cost
        net[ex] = g - period_cost
        entry_bps_period.append(period_entry_bps)
        exit_bps_period.append(period_exit_bps)
        names_per_period.append(len(shorts))
        total_short_names += len(shorts)
        windows.append((t, ex))
        hold_slices.append(hold_slice)

    period_days = sorted(net)
    if len(period_days) < 4:
        return {**unavail, "reason": "too few non-overlapping periods to evaluate"}

    test_lo = _split_day(period_days, train_frac)
    net_all = [net[d] for d in period_days]
    gross_all = [gross[d] for d in period_days]
    net_test = [net[d] for d in period_days if d >= test_lo]
    gross_test = [gross[d] for d in period_days if d >= test_lo]
    cost_test = [cost_series[d] for d in period_days if d >= test_lo]

    net_test_sharpe = _ann_sharpe(net_test, ppy)
    gross_full_sharpe = _ann_sharpe(gross_all, ppy)
    t_stat = _t_stat(net_test)

    # correlation to the live edges — aggregate each edge to the SAME non-overlapping period windows (compound
    # the edge's daily return over each hold window) so it is an apples-to-apples period-to-period comparison.
    corr_to_edges: dict[str, float] = {}
    if edge_series:
        for name, daily in edge_series.items():
            edge_periods: dict[int, float] = {}
            for (_t, ex), hslice in zip(windows, hold_slices, strict=True):
                r = 1.0
                seen = False
                for d in hslice:
                    if d in daily and math.isfinite(daily[d]):
                        r *= (1.0 + daily[d])
                        seen = True
                if seen:
                    edge_periods[ex] = r - 1.0
            corr_to_edges[name] = _pearson(net, edge_periods)
    max_abs_corr = max((abs(c) for c in corr_to_edges.values()), default=0.0)

    avg_entry_drag = st.mean(entry_bps_period) if entry_bps_period else 0.0
    avg_exit_drag = st.mean(exit_bps_period) if exit_bps_period else 0.0
    avg_names = st.mean(names_per_period) if names_per_period else 0.0
    span_years = (period_days[-1] - period_days[0]) / _PERIODS_PER_YEAR if len(period_days) >= 2 else 0.0
    events_per_year = (total_short_names / span_years) if span_years > 0 else 0.0

    # LARGE+FREQUENT+CONCENTRATED (on TOP of Sharpe/t/corr).
    large = st.mean(gross_test) > st.mean(cost_test) if (gross_test and cost_test) else False
    frequent = events_per_year >= _MIN_EVENTS_PER_YEAR
    concentrated = avg_names >= _MIN_NAMES_PER_PERIOD
    passes_lfc = bool(large and frequent and concentrated)

    confirm = (net_test_sharpe >= _SHARPE_BAR and abs(t_stat) >= _T_BAR
               and (not corr_to_edges or max_abs_corr < _CORR_BAR) and passes_lfc)
    verdict = "CONFIRM" if confirm else "FALSIFY"

    return {
        "available": True,
        "verdict": verdict,
        "hold_days": hold_days,
        "gross_full_sharpe": gross_full_sharpe,
        "net_full_sharpe": _ann_sharpe(net_all, ppy),
        "net_test_sharpe": net_test_sharpe,
        "gross_total_return": _total_return(gross_all),
        "net_total_return": _total_return(net_all),
        "net_test_total_return": _total_return(net_test),
        "t_stat": t_stat,
        "max_dd": _max_dd(net_all),
        "turnover": float(2 * len(period_days)),   # a full round-trip (enter+exit) per non-overlapping period.
        "avg_entry_drag_bps": avg_entry_drag,
        "avg_exit_drag_bps": avg_exit_drag,
        "avg_names_per_period": avg_names,
        "events_per_year": events_per_year,
        "n_periods": len(period_days),
        "corr_to_edges": corr_to_edges,
        "max_abs_corr": max_abs_corr,
        "large": large,
        "frequent": frequent,
        "concentrated": concentrated,
        "passes_lfc": passes_lfc,
        "net_series": net,
        "gross_series": gross,
    }


def spread_deterioration_sweep(panel: dict[int, dict[str, _DetRow]], *,
                               holds: tuple[int, ...] = (5, 10, 20),
                               det_window: int = _DET_WINDOW, min_obs: int = _DET_MIN_OBS,
                               use_amihud: bool = True,
                               max_entry_spread_bps: float = _MAX_ENTRY_SPREAD_BPS,
                               min_entry_dollar_vol: float = _MIN_ENTRY_DOLLAR_VOL,
                               n_buckets: int = _N_DECILES, taker_fee_bps: float = 5.5,
                               train_frac: float = 0.6,
                               edge_series: dict[str, dict[int, float]] | None = None) -> dict[str, Any]:
    """Run `deterioration_shorts` across the hold horizons `holds` (default {5,10,20}) and return the per-T
    reports `{available, by_T:{T: report}}`. The EXPECTED outcome is FALSIFY (the abandonment move does not
    clear the round-trip spread, or it is a rebrand of unlock); a CONFIRM on any T that clears ALL bars is the
    rare, valuable positive. Pure; writes nothing."""
    if not panel:
        return {"available": False, "reason": "empty deterioration panel", "by_T": {}}
    by_t = {
        T: deterioration_shorts(panel, hold_days=T, det_window=det_window, min_obs=min_obs,
                                use_amihud=use_amihud, max_entry_spread_bps=max_entry_spread_bps,
                                min_entry_dollar_vol=min_entry_dollar_vol, n_buckets=n_buckets,
                                taker_fee_bps=taker_fee_bps, train_frac=train_frac, edge_series=edge_series)
        for T in holds
    }
    return {"available": True, "by_T": by_t}
