"""STEP 3 (final) of the "spread-as-a-strat" research — the SPREAD-AWARE EXECUTION OVERLAY. Use the MEASURED
spread to AVOID cost (not to bet on it), and measure whether it lifts the EXISTING Bybit 4-edge book's NET
Sharpe. READ-ONLY, memory-bounded, network-free, pure.

WHY (what Steps 1-2 proved)
---------------------------
Both DIRECTIONAL spread ideas were FALSIFIED: the STATIC illiquidity premium (Step 1) is a cost trap on
entry/holding, and the spread-DETERIORATION shorts (Step 2) are a cost trap on the WIDENED exit. The illiquid
tail is cost-cursed BOTH ways. Conclusion: the spread is a COST signal, not alpha. The only remaining value is
NON-DIRECTIONAL — use the spread to REDUCE the cost of the edges we already run. Step 3 tests exactly that,
honestly, as a SHADOW eval: it NEVER changes the live book's returns.

THE TWO OVERLAY MECHANISMS (both use the spread to AVOID cost, neither bets on it)
---------------------------------------------------------------------------------
Run the Bybit 4-edge per-coin book (`bybit_paper_weights.sleeve_weights_and_contributions`, combined eq-wt —
the SAME per-symbol weights as the live book) three overlaid ways and compare NET (measured per-coin cost) to
the baseline:

  (A) BASELINE — the current book: naive eq-wt over the liquid universe, per-coin weights as-is.

  (B) SPREAD-AWARE SIZING — scale each coin's target weight by 1/(1+k·spread_bps) then RENORMALISE to the same
      GROSS exposure (Σ|w| preserved). Trade low-spread coins bigger, high-spread coins smaller — a CONTINUOUS
      version of the $10M hard cut. The per-coin contribution scales by the SAME factor as its weight (sizing
      a position scales both its exposure and its P&L), so it is a faithful re-sizing, not a re-pricing. Sweep
      k. NO LOOKAHEAD: the factor uses ONLY the coin's measured spread (a cost property), never a future
      return.

  (C) NO-TRADE BANDS — only rebalance coin i when its target moves more than a COST-PROPORTIONAL band
      `band_i = λ · cost_bps_i / 1e4`; otherwise HOLD the stale weight (skip the trade). Higher-cost coins get
      WIDER bands (you trade them less often). The realised return marks the HELD (possibly stale) weight, so
      the gross is honest — skipping a trade keeps a slightly stale portfolio, it is not free. A coin the book
      drops to zero is always closed (you cannot mark a position the book has zeroed). Sweep λ.

  (B+C) COMBINED — size first, then band the sized target.

OOS DISCIPLINE. Each mechanism's hyper-parameter (k, λ) is SELECTED on the TRAIN split (best TRAIN NET Sharpe)
and the held-out TEST NET Sharpe is reported — selection on train, verdict on test. Baseline is k=0, λ=0
(always trade), which reduces the engine EXACTLY to `bybit_liquidity_sweep.per_coin_book_returns`.

THE HONESTY + BREADTH CHECK
---------------------------
The live Bybit book is ALREADY liquidity-bounded ($10M / liquid-61), so it is already cost-optimised — the
overlay's lift on it may be MODEST. We therefore run the SAME overlay on a BROADER universe (`min_dollar_vol`
lowered to $1M and to 0/broad) where it should help MORE. Comparing the lift across universes quantifies how
much cost the $10M bound ALREADY saves and tells us whether the overlay is worth wiring into the live book or
whether the bound already captures the gain.

VERDICT (per universe). best of {sizing, band, combined} vs baseline:
  * CONFIRM   — NET-OOS Sharpe lift >= +0.10 (worth wiring into the live book).
  * MARGINAL  — small/no Sharpe lift but a clear cost-drag / turnover reduction (the $10M bound likely already
                captures most of it).
  * NO-HELP   — the overlay does not improve NET and gives no cost benefit.

REUSE: `bybit_paper_weights.sleeve_weights_and_contributions` (live per-symbol weights), `bybit_book_eval.
_bounded_store` (memory seam), `bybit_liquidity_sweep.{_coin_spread_bps, coin_cost_bps, _metrics, _split_day,
_window}` (the MEASURED cost = Bybit 5.5bp taker + half the Corwin–Schultz spread, + the OOS split). READ-ONLY:
no PaperRepo, no `store.write_*` (a WriteTripwire test proves it). MEMORY-BOUNDED by `universe`.
"""
from __future__ import annotations

import statistics as st
from typing import Any, Protocol

from fxhnt.application.bybit_book_eval import _DEFAULT_BYBIT_SLEEVES, _bounded_store
from fxhnt.application.bybit_liquidity_sweep import (
    _coin_spread_bps,
    _metrics,
    _split_day,
    _window,
    coin_cost_bps,
)
from fxhnt.application.bybit_paper_weights import sleeve_weights_and_contributions

_TAKER_FEE_BPS = 5.5

# Default sizing strengths to sweep. k multiplies the spread (in bps): a coin with a 50bp spread is halved at
# k=0.02, quartered at k=0.06. 0 = sizing OFF. Research knobs, reported not tuned.
_K_GRID: tuple[float, ...] = (0.005, 0.02, 0.05, 0.1, 0.2)
# Default no-trade-band strengths to sweep. band_i = λ · cost_bps_i / 1e4 (weight units). 0 = band OFF.
_LAM_GRID: tuple[float, ...] = (0.5, 1.0, 2.0, 4.0, 8.0)

# The universes for the breadth check: the live $10M (liquid-61) default, a broad $1M, and 0 = the full
# survivorship-free universe. The overlay's lift SHOULD grow as the universe gets broader (more thin coins).
_DEFAULT_UNIVERSES: tuple[float, ...] = (10_000_000.0, 1_000_000.0, 0.0)
_LIVE_BOUND = 10_000_000.0

# Adoption bar: the best overlay must lift NET-OOS Sharpe by at least this to CONFIRM (worth wiring in).
_SHARPE_LIFT_BAR = 0.10
# A "clear" cost benefit (for the MARGINAL verdict) — at least this avg cost-drag reduction (bps) with the
# Sharpe not materially worse.
_COST_DRAG_BAR = 1.0
_NOT_WORSE = -0.05


class _FeatureStore(Protocol):
    def crypto_close_panel(self, *, suffix: str = "USDT") -> dict[str, dict[int, float]]: ...
    def read_panel(self, symbols: list[str], feature: str) -> dict[str, dict[int, float]]: ...


# --- the per-coin (weight, contribution) panel — the SAME book as the live edges ----------------

def per_coin_weight_panel(store: _FeatureStore, *, universe: set[str] | None,
                          sleeves: list[str] | None = None,
                          unlock_events: list[dict[str, Any]] | None = None,
                          taker_fee_bps: float = _TAKER_FEE_BPS,
                          ) -> tuple[dict[int, dict[str, tuple[float, float]]], dict[str, float],
                                     dict[str, float]]:
    """`({day: {coin: (book_weight, book_contribution)}}, cost_by_coin, spread_by_coin)` for the Bybit 4-edge
    book, combining the sleeves EQ-WT exactly as `per_coin_book_returns` does (so it reconciles with the live
    edges, no drift). `book_weight = Σ_sleeve w_sleeve/n_sleeves`; `book_contribution = Σ_sleeve (w/n)·ret`.

    `cost_by_coin` / `spread_by_coin` are the MEASURED per-coin cost (Bybit taker fee + half the Corwin–Schultz
    spread) and the measured CS spread (bps). READ-ONLY; memory-bounded by `universe` (the wrapped store reads
    `WHERE symbol IN (...)`). Empty when no sleeve has per-coin weights."""
    candidate = list(sleeves) if sleeves is not None else list(_DEFAULT_BYBIT_SLEEVES)
    n_sleeves = len(candidate)
    bounded = _bounded_store(store, universe)

    per_sleeve: dict[str, dict[int, dict[str, tuple[float, float]]]] = {}
    for sleeve in candidate:
        w = sleeve_weights_and_contributions(bounded, sleeve, universe=None, unlock_events=unlock_events)
        if w:
            per_sleeve[sleeve] = w
    if not per_sleeve:
        return {}, {}, {}

    all_days = sorted({d for w in per_sleeve.values() for d in w})
    daily: dict[int, dict[str, tuple[float, float]]] = {}
    coins_seen: set[str] = set()
    for d in all_days:
        row: dict[str, tuple[float, float]] = {}
        for by_day in per_sleeve.values():
            for coin, (w, r) in by_day.get(d, {}).items():
                pw, pc = row.get(coin, (0.0, 0.0))
                row[coin] = (pw + w / n_sleeves, pc + (w / n_sleeves) * r)
                coins_seen.add(coin)
        daily[d] = row

    spread_by_coin = _coin_spread_bps(store, universe, sorted(coins_seen))
    cost_by_coin = {c: coin_cost_bps(effective_spread_bps=spread_by_coin.get(c, 0.0),
                                     taker_fee_bps=taker_fee_bps) for c in coins_seen}
    return daily, cost_by_coin, spread_by_coin


# --- mechanism (B): spread-aware sizing (pure) --------------------------------------------------

def sized_weights(weights: dict[str, float], spread_by_coin: dict[str, float], k: float) -> dict[str, float]:
    """Scale each coin's weight by `1/(1+k·spread_bps)` then RENORMALISE so the gross exposure Σ|w| is
    UNCHANGED — downweight high-spread coins, redistribute their budget onto low-spread coins, same total risk.
    Pure; no lookahead (uses only the coin's measured spread). k<=0 → unchanged (sizing OFF). Direction (sign)
    of every weight is preserved (the factor is strictly positive)."""
    if k <= 0.0:
        return dict(weights)
    sized = {c: w / (1.0 + k * max(0.0, spread_by_coin.get(c, 0.0))) for c, w in weights.items()}
    gross_target = sum(abs(w) for w in weights.values())
    gross_sized = sum(abs(w) for w in sized.values())
    if gross_sized > 0.0 and gross_target > 0.0:
        scale = gross_target / gross_sized
        sized = {c: w * scale for c, w in sized.items()}
    return sized


# --- the overlay engine (sizing + no-trade band) ------------------------------------------------

def _overlay_series(daily: dict[int, dict[str, tuple[float, float]]], cost_by_coin: dict[str, float],
                    spread_by_coin: dict[str, float], *, k: float, lam: float,
                    ) -> tuple[dict[int, float], dict[int, float], float, float]:
    """Walk the days applying spread-aware SIZING (strength `k`) then the COST-PROPORTIONAL NO-TRADE BAND
    (strength `lam`), marking the HELD (possibly stale) weights. Returns (gross_series, net_series,
    turnover_total, cost_drag_bps).

    k=0 and lam=0 reduce this EXACTLY to the naive eq-wt book (always trade to target; gross = Σ contribution;
    net = gross − Σ|Δw|·cost), i.e. `per_coin_book_returns`. The band marks `held·ρ` where `ρ = contribution/
    weight` is the coin's per-unit book return that day; a coin with zero target weight is always closed (no ρ
    to mark). Pure; sequential forward (no lookahead)."""
    days = sorted(daily)
    gross: dict[int, float] = {}
    net: dict[int, float] = {}
    turnover_total = 0.0
    cost_bps_weighted = 0.0
    traded_w = 0.0
    prev_held: dict[str, float] = {}

    for d in days:
        row = daily[d]
        base_w = {c: wc[0] for c, wc in row.items()}
        base_c = {c: wc[1] for c, wc in row.items()}
        if k > 0.0:
            tgt_w = sized_weights(base_w, spread_by_coin, k)
            # contribution scales by the SAME per-coin factor as the weight (sizing scales exposure AND P&L).
            tgt_c = {c: (base_c[c] * (tgt_w[c] / base_w[c]) if base_w[c] != 0.0 else 0.0) for c in base_w}
        else:
            tgt_w, tgt_c = base_w, base_c

        new_held: dict[str, float] = {}
        day_real = 0.0
        day_cost = 0.0
        for coin in set(tgt_w) | set(prev_held):
            w_t = tgt_w.get(coin, 0.0)
            h_prev = prev_held.get(coin, 0.0)
            if w_t == 0.0:
                h = 0.0          # the book has no position today (or the coin is gone) — close it.
                realized = 0.0
            else:
                rho = tgt_c.get(coin, 0.0) / w_t
                band = lam * cost_by_coin.get(coin, 0.0) / 1e4
                h = h_prev if abs(w_t - h_prev) < band else w_t
                realized = h * rho
            dw = abs(h - h_prev)
            if dw > 0.0:
                cbps = cost_by_coin.get(coin, 0.0)
                day_cost += dw * cbps / 1e4
                turnover_total += dw
                cost_bps_weighted += dw * cbps
                traded_w += dw
            if h != 0.0:
                new_held[coin] = h
            day_real += realized
        gross[d] = day_real
        net[d] = day_real - day_cost
        prev_held = new_held

    cost_drag_bps = (cost_bps_weighted / traded_w) if traded_w > 0.0 else 0.0
    return gross, net, turnover_total, cost_drag_bps


def evaluate_overlay(daily: dict[int, dict[str, tuple[float, float]]], cost_by_coin: dict[str, float],
                     spread_by_coin: dict[str, float], *, k: float, lam: float,
                     train_frac: float = 0.6) -> dict[str, Any]:
    """One (k, lam) variant's NET metrics with an OOS split: NET TRAIN/TEST/FULL Sharpe + total return + maxDD
    + turnover + avg cost-drag. The TRAIN Sharpe is the SELECTION criterion; the TEST Sharpe is the verdict."""
    gross, net, turnover, cost_drag = _overlay_series(daily, cost_by_coin, spread_by_coin, k=k, lam=lam)
    days = sorted(net)
    test_lo = _split_day(days, train_frac)
    net_full = _metrics(net)
    net_train = _metrics(_window(net, day_hi=test_lo - 1))
    net_test = _metrics(_window(net, day_lo=test_lo))
    return {
        "k": k, "lam": lam,
        "net_train_sharpe": net_train["sharpe"],
        "net_test_sharpe": net_test["sharpe"],
        "net_full_sharpe": net_full["sharpe"],
        "net_test_total_return": net_test["total_return"],
        "net_full_total_return": net_full["total_return"],
        "max_dd": net_full["max_dd"],
        "turnover": turnover,
        "cost_drag_bps": cost_drag,
        "net_series": net,
        "gross_series": gross,
    }


def _select_best(daily: dict[int, dict[str, tuple[float, float]]], cost_by_coin: dict[str, float],
                 spread_by_coin: dict[str, float], *, grid: list[tuple[float, float]],
                 train_frac: float) -> dict[str, Any]:
    """The grid point with the best TRAIN NET Sharpe (selection on train), reported with its full metrics
    (incl. the held-out TEST NET Sharpe)."""
    best: dict[str, Any] | None = None
    for (k, lam) in grid:
        r = evaluate_overlay(daily, cost_by_coin, spread_by_coin, k=k, lam=lam, train_frac=train_frac)
        if best is None or r["net_train_sharpe"] > best["net_train_sharpe"]:
            best = r
    assert best is not None
    return best


# --- the per-universe verdict -------------------------------------------------------------------

def _overlay_verdict(baseline: dict[str, Any], sizing: dict[str, Any], band: dict[str, Any],
                     combined: dict[str, Any]) -> dict[str, Any]:
    """CONFIRM / MARGINAL / NO-HELP for ONE universe. Best of {sizing, band, combined} vs the baseline on the
    held-out TEST NET Sharpe. CONFIRM if the Sharpe lift clears the bar; MARGINAL if there is a clear
    cost-drag reduction without a Sharpe penalty; NO-HELP otherwise."""
    candidates = {"sizing": sizing, "band": band, "combined": combined}
    best_name = max(candidates, key=lambda n: candidates[n]["net_test_sharpe"])
    best = candidates[best_name]
    d_sharpe = best["net_test_sharpe"] - baseline["net_test_sharpe"]
    d_cost = baseline["cost_drag_bps"] - best["cost_drag_bps"]
    d_turn = baseline["turnover"] - best["turnover"]

    if d_sharpe >= _SHARPE_LIFT_BAR:
        verdict = "CONFIRM"
    elif d_sharpe >= 0.0 or (d_cost >= _COST_DRAG_BAR and d_sharpe >= _NOT_WORSE):
        verdict = "MARGINAL"
    else:
        verdict = "NO-HELP"
    return {
        "verdict": verdict,
        "best_variant": best_name,
        "best_k": best["k"],
        "best_lam": best["lam"],
        "d_sharpe": d_sharpe,
        "d_cost_drag_bps": d_cost,
        "d_turnover": d_turn,
        "best_net_test_sharpe": best["net_test_sharpe"],
        "baseline_net_test_sharpe": baseline["net_test_sharpe"],
    }


# --- one-universe eval --------------------------------------------------------------------------

def spread_overlay_eval(store: _FeatureStore, *, universe: set[str] | None,
                        k_grid: tuple[float, ...] = _K_GRID, lam_grid: tuple[float, ...] = _LAM_GRID,
                        train_frac: float = 0.6, taker_fee_bps: float = _TAKER_FEE_BPS,
                        sleeves: list[str] | None = None,
                        unlock_events: list[dict[str, Any]] | None = None) -> dict[str, Any]:
    """Run the Bybit 4-edge book on ONE `universe` four ways — baseline, spread-aware sizing (sweep k),
    no-trade band (sweep λ), combined (sweep k×λ) — and report each variant's NET-OOS metrics + the per-universe
    verdict. Hyper-parameters are SELECTED on TRAIN, the verdict uses the held-out TEST. READ-ONLY; pure."""
    daily, cost_by_coin, spread_by_coin = per_coin_weight_panel(
        store, universe=universe, sleeves=sleeves, unlock_events=unlock_events, taker_fee_bps=taker_fee_bps)
    if not daily or len(daily) < 4:
        return {"available": False, "reason": "no usable per-coin book (run the Phase-A/B Bybit ingests first)"}

    baseline = evaluate_overlay(daily, cost_by_coin, spread_by_coin, k=0.0, lam=0.0, train_frac=train_frac)
    sizing = _select_best(daily, cost_by_coin, spread_by_coin,
                          grid=[(k, 0.0) for k in k_grid], train_frac=train_frac)
    band = _select_best(daily, cost_by_coin, spread_by_coin,
                        grid=[(0.0, lam) for lam in lam_grid], train_frac=train_frac)
    combined = _select_best(daily, cost_by_coin, spread_by_coin,
                            grid=[(k, lam) for k in k_grid for lam in lam_grid], train_frac=train_frac)
    verdict = _overlay_verdict(baseline, sizing, band, combined)

    spreads = [spread_by_coin[c] for c in spread_by_coin]
    return {
        "available": True,
        "n_coins": len(cost_by_coin),
        "median_spread_bps": float(st.median(spreads)) if spreads else 0.0,
        "baseline": baseline,
        "sizing": sizing,
        "band": band,
        "combined": combined,
        "verdict": verdict,
    }


# --- the breadth sweep (variant × universe) -----------------------------------------------------

def spread_overlay_test(store: _FeatureStore, *, min_dollar_vols: tuple[float, ...] = _DEFAULT_UNIVERSES,
                        k_grid: tuple[float, ...] = _K_GRID, lam_grid: tuple[float, ...] = _LAM_GRID,
                        train_frac: float = 0.6, taker_fee_bps: float = _TAKER_FEE_BPS,
                        lookback_days: int = 90, sleeves: list[str] | None = None,
                        unlock_events: list[dict[str, Any]] | None = None) -> dict[str, Any]:
    """The BREADTH check: run `spread_overlay_eval` on each universe in `min_dollar_vols` (0 = broad/all) and
    return the variant×universe table. The overlay's lift SHOULD grow as the universe broadens (more thin
    coins) — quantifying how much cost the live $10M bound ALREADY saves and whether the overlay is worth
    wiring into the (already liquidity-bounded) live book. READ-ONLY; memory-bounded; writes NOTHING."""
    from fxhnt.application.bybit_liquidity import liquid_universe

    rows: list[dict[str, Any]] = []
    for bound in min_dollar_vols:
        universe = None if bound <= 0.0 else liquid_universe(store, min_dollar_vol=bound,
                                                             lookback_days=lookback_days)
        if bound > 0.0 and not universe:
            continue
        rep = spread_overlay_eval(store, universe=universe, k_grid=k_grid, lam_grid=lam_grid,
                                  train_frac=train_frac, taker_fee_bps=taker_fee_bps, sleeves=sleeves,
                                  unlock_events=unlock_events)
        if not rep.get("available"):
            continue
        rep["bound"] = bound
        rep["universe_label"] = "broad/all" if bound <= 0.0 else f"${bound/1e6:.0f}M"
        rep["is_live_bound"] = (bound == _LIVE_BOUND)
        rows.append(rep)

    if not rows:
        return {"available": False, "train_frac": train_frac, "taker_fee_bps": taker_fee_bps, "table": [],
                "reason": "no universe produced a usable per-coin book (run the Phase-A/B Bybit ingests first)"}
    return {"available": True, "train_frac": train_frac, "taker_fee_bps": taker_fee_bps, "table": rows}
