"""WALK-FORWARD validation of the DEPLOY-BOOK question Step 3 only hinted at: is a BROAD universe + the
spread-aware execution overlay an UPGRADE over the current liquid-61 ($10M) Bybit book? READ-ONLY, SHADOW
eval (never mutates the live book), network-free, memory-bounded, pure given the store.

WHY (the lead + the risk)
-------------------------
Step 3 (`spread_overlay`) found, on ONE OOS split with TRAIN-tuned k/λ, that a broad universe + the overlay
BEAT the live book: $10M liquid-61 NET-OOS Sharpe 1.04 / cost-drag 67.8bp vs $1M broad+overlay 1.30 /
28.7bp. ONE split with selected knobs is exactly where overfitting hides. This module stress-tests that lead
with WALK-FORWARD folds so we learn whether the 1.30 is a real, repeatable edge or a one-split fluke — and,
critically, whether the broad book can actually hold deployable capital (the $10M bound exists for a reason).

THE WALK-FORWARD DESIGN (the rigor Step 3 lacked)
-------------------------------------------------
EXPANDING-WINDOW folds over the full day axis. The axis (sorted union of every candidate universe's days) is
cut into `n_folds + 1` contiguous index blocks; fold i TRAINS on every day up to the end of block i and TESTS
on the disjoint next block. Per fold:
  * SELECTION (on TRAIN only): pick BOTH the overlay knobs (k, λ) AND the universe breadth (`min_dollar_vol`)
    that maximise the broad book's TRAIN NET Sharpe. No future day touches the choice.
  * VERDICT (on the held-out TEST only): evaluate that frozen selection NET on the disjoint test block.
Test blocks are disjoint and every fold's train strictly precedes its test (train_hi < test_lo) → no leakage.
We aggregate the OOS test results across folds (mean NET Sharpe + dispersion + how often broad beats live),
which is what kills the single-split overfit.

HEAD-TO-HEAD (same test windows)
--------------------------------
On EVERY fold's TEST window we compare, NET of the MEASURED per-coin cost (Bybit 5.5bp taker + half the
Corwin–Schultz spread):
  (A) LIVE book  — liquid-61 ($10M `min_dollar_vol`), naive eq-wt per-coin (k=0, λ=0). The current deploy.
  (B) BROAD+overlay — the per-fold-train-selected broader universe + spread-aware overlay (k*, λ*).
Both report NET Sharpe / total return / cost-drag / maxDD / turnover, plus Δ(B−A), per fold and aggregated.

CAPACITY (the $10M bound exists for a reason)
---------------------------------------------
A higher-Sharpe book that can only hold $200k is NOT an upgrade for a fund that wants to scale. We estimate
each book's DEPLOYABLE CAPITAL from the BOTTLENECK coin: to deploy capital C the book must put |w_i|·C into
coin i, which can absorb at most `participation_frac · ADV_i` (a fraction of the coin's median daily
dollar-volume). So C ≤ participation_frac·ADV_i / |w_i| for every held coin, and the deployable capital is the
MIN over coins of that bound — the thinnest-relative-to-weight name. The overlay DOWNWEIGHTS thin coins (lower
|w_i|), which RAISES their bound and so lifts the book's capacity; capacity therefore reflects the overlay. The
capacity RATIO (B/A) is invariant to `participation_frac` (it cancels), so the verdict's capacity test does not
depend on that assumption. We also report the gross-weighted absorbable notional `Σ participation_frac·ADV_i·
|w_i|` (the brief's sketch) as a supplementary figure.

VERDICT
-------
  * UPGRADE          — aggregate walk-forward NET Sharpe lift >= +0.20, broad beats live in a majority (>=60%)
                       of folds, AND capacity is acceptable (B >= 50% of A's deployable capital). Wire it in.
  * CAPACITY-TRADEOFF— same Sharpe win, but B's deployable capital is materially below A's. A choice, not a
                       free win.
  * OVERFIT          — a positive but small/inconsistent lift: the 1.30 shrinks toward liquid-61 under
                       walk-forward (Step 3 was a one-split fluke). Keep liquid-61.
  * NO               — broad+overlay is no better on average. Keep liquid-61.

REUSE: `spread_overlay.{per_coin_weight_panel, _overlay_series, sized_weights}` (the overlay engine + the
live per-coin weights), `bybit_liquidity_sweep.{_metrics, _window}` (metrics + windowing),
`bybit_liquidity.liquid_universe` (the `--min-dollar-vol` path the live book uses). READ-ONLY: only panel
READS; no PaperRepo, no `store.write_*` (a WriteTripwire test proves it).
"""
from __future__ import annotations

import math
import statistics as st
from typing import Any, Protocol

from fxhnt.application.bybit_liquidity_sweep import _metrics, _window
from fxhnt.application.spread_overlay import (
    _K_GRID,
    _LAM_GRID,
    _overlay_series,
    per_coin_weight_panel,
    sized_weights,
)

_TAKER_FEE_BPS = 5.5
_LIVE_BOUND = 10_000_000.0
# The broader universes to consider for (B). Each is selected-against on TRAIN; the per-fold winner is tested.
_DEFAULT_BROAD_BOUNDS: tuple[float, ...] = (5_000_000.0, 1_000_000.0, 500_000.0)
# Grids INCLUDE the no-overlay point (0.0) so "broad without overlay" is an honest candidate the train can pick.
_K_GRID_WF: tuple[float, ...] = (0.0,) + _K_GRID
_LAM_GRID_WF: tuple[float, ...] = (0.0,) + _LAM_GRID

# A coin can absorb this fraction of its median daily dollar-volume without undue impact (capacity assumption).
# The capacity RATIO (B/A) is invariant to this value, so the verdict does not depend on it.
_PARTICIPATION_FRAC = 0.10

# Verdict bars.
_SHARPE_UPGRADE_BAR = 0.20      # mean OOS NET Sharpe lift (B−A) to call broad+overlay meaningfully better.
_WIN_FRAC_BAR = 0.60           # fraction of folds B must beat A to count as consistent (not a one-fold fluke).
_CAPACITY_FLOOR_FRAC = 0.50    # B's deployable capital must be >= this × A's to be "acceptable".


class _FeatureStore(Protocol):
    def crypto_close_panel(self, *, suffix: str = "USDT") -> dict[str, dict[int, float]]: ...
    def read_panel(self, symbols: list[str], feature: str) -> dict[str, dict[int, float]]: ...


# --- walk-forward fold construction (no leakage) ------------------------------------------------

def walk_forward_folds(days: list[int], n_folds: int) -> list[dict[str, int]]:
    """EXPANDING-WINDOW walk-forward folds over the sorted day axis. The axis is cut into `n_folds + 1`
    contiguous index blocks; fold i TRAINS on days up to the end of block i (`train_hi`, inclusive) and TESTS
    on the disjoint next block `[test_lo, test_hi]` (inclusive). Guarantees: every fold's train strictly
    precedes its test (`train_hi < test_lo`) and the test blocks are disjoint (no leakage). Returns
    `[{fold, train_hi, test_lo, test_hi}]`. Empty when there are too few days for `n_folds + 1` non-empty
    blocks of >= 2 days each."""
    uniq = sorted(set(days))
    n = len(uniq)
    if n_folds < 1 or n < 2 * (n_folds + 1):
        return []
    block = n // (n_folds + 1)
    folds: list[dict[str, int]] = []
    for i in range(n_folds):
        train_end = block * (i + 1)              # index one-past the end of the training span
        test_end = block * (i + 2) if i < n_folds - 1 else n
        if train_end < 1 or test_end - train_end < 2:
            continue
        folds.append({
            "fold": i + 1,
            "train_hi": uniq[train_end - 1],
            "test_lo": uniq[train_end],
            "test_hi": uniq[test_end - 1],
        })
    return folds


# --- one (universe, k, λ) evaluated on a [day_lo, day_hi] window ---------------------------------

def _eval_window(daily: dict[int, dict[str, tuple[float, float]]], cost_by_coin: dict[str, float],
                 spread_by_coin: dict[str, float], *, k: float, lam: float,
                 day_lo: int | None, day_hi: int | None) -> dict[str, Any]:
    """NET metrics for one overlay variant restricted to a day window. The overlay walk runs on ONLY the
    window's days (held state starts fresh at the window edge — correct for a disjoint OOS test block, and
    leakage-free for train selection). Returns Sharpe/total_return/max_dd + window turnover + cost-drag + the
    NET series + day count."""
    sub = _window(daily, day_lo=day_lo, day_hi=day_hi)  # type: ignore[arg-type]
    gross, net, turnover, cost_drag = _overlay_series(sub, cost_by_coin, spread_by_coin, k=k, lam=lam)
    m = _metrics(net)
    return {"sharpe": m["sharpe"], "total_return": m["total_return"], "max_dd": m["max_dd"],
            "turnover": turnover, "cost_drag_bps": cost_drag, "net_series": net, "days": m["days"]}


# --- capacity (bottleneck deployable capital) ---------------------------------------------------

def _median_dollar_vol(store: _FeatureStore, coins: list[str], day_lo: int, day_hi: int) -> dict[str, float]:
    """Median daily dollar-volume (turnover) per coin over `[day_lo, day_hi]`. Memory-bounded: reads only the
    held coins' turnover panel. A coin with no turnover rows in the window → absent (treated as unassessable)."""
    if not coins:
        return {}
    panel = store.read_panel(sorted(coins), "turnover")
    out: dict[str, float] = {}
    for c in coins:
        vals = [v for d, v in panel.get(c, {}).items() if day_lo <= d <= day_hi]
        if vals:
            out[c] = float(st.median(vals))
    return out


def book_capacity(store: _FeatureStore, daily: dict[int, dict[str, tuple[float, float]]],
                  spread_by_coin: dict[str, float], *, k: float, day_lo: int, day_hi: int,
                  participation_frac: float = _PARTICIPATION_FRAC) -> dict[str, Any]:
    """Estimate the book's DEPLOYABLE CAPITAL over `[day_lo, day_hi]` AFTER overlay sizing (strength `k`).

    Each coin's deployed weight is the mean |overlay-sized weight| over the window. To deploy capital C the
    book needs |w_i|·C in coin i, which can absorb at most `participation_frac · ADV_i` (ADV_i = median daily
    dollar-volume). So C <= participation_frac·ADV_i / |w_i| per coin; the BOTTLENECK deployable capital is the
    MIN over held coins (the thinnest relative to its weight). The overlay downweighting thin coins (smaller
    |w_i|) RAISES their bound → lifts capacity, so the estimate reflects the overlay. Also returns the
    gross-weighted absorbable notional `Σ participation_frac·ADV_i·|w_i|` (the brief's sketch) and the weight
    Herfindahl (concentration). Coins with no ADV data are skipped (unassessable, not zeroed). READ-ONLY."""
    test_days = [d for d in daily if day_lo <= d <= day_hi]
    if not test_days:
        return {"bottleneck_capital": 0.0, "weighted_absorbable": 0.0, "n_coins": 0, "hhi": 0.0}
    acc: dict[str, float] = {}
    for d in test_days:
        base_w = {c: wc[0] for c, wc in daily[d].items()}
        sized = sized_weights(base_w, spread_by_coin, k)
        for c, w in sized.items():
            acc[c] = acc.get(c, 0.0) + abs(w)
    n = len(test_days)
    mean_w = {c: acc[c] / n for c in acc if acc[c] > 0.0}
    if not mean_w:
        return {"bottleneck_capital": 0.0, "weighted_absorbable": 0.0, "n_coins": 0, "hhi": 0.0}
    adv = _median_dollar_vol(store, list(mean_w), day_lo, day_hi)
    bounds = [participation_frac * adv[c] / mean_w[c] for c in mean_w if adv.get(c, 0.0) > 0.0]
    bottleneck = min(bounds) if bounds else 0.0
    weighted = sum(participation_frac * adv.get(c, 0.0) * mean_w[c] for c in mean_w)
    tot = sum(mean_w.values())
    hhi = sum((w / tot) ** 2 for w in mean_w.values()) if tot > 0 else 0.0
    return {"bottleneck_capital": bottleneck, "weighted_absorbable": weighted,
            "n_coins": len(mean_w), "hhi": hhi}


# --- the verdict (pure, testable) ---------------------------------------------------------------

def decide_verdict(*, mean_a: float, mean_b: float, win_frac: float, cap_a: float, cap_b: float) -> dict[str, Any]:
    """UPGRADE / CAPACITY-TRADEOFF / OVERFIT / NO from the aggregated walk-forward economics. Pure.

    Δ = mean_b − mean_a (aggregate OOS NET Sharpe lift). `win_frac` = fraction of folds B beat A. Capacity
    ratio = cap_b/cap_a (deployable capital). NO if no average lift; UPGRADE if the lift is meaningful
    (>= +0.20) AND consistent (>= 60% of folds) AND capacity acceptable (>= 50% of live); CAPACITY-TRADEOFF if
    the Sharpe win holds but capacity is materially lower; OVERFIT for a positive but small/inconsistent lift
    (the single-split edge shrank toward liquid-61 under walk-forward)."""
    d = mean_b - mean_a
    if cap_a > 0.0:
        cap_ratio = cap_b / cap_a
    elif cap_b > 0.0:
        cap_ratio = math.inf
    else:
        cap_ratio = 1.0

    if d <= 0.0:
        verdict = "NO"
    elif d >= _SHARPE_UPGRADE_BAR and win_frac >= _WIN_FRAC_BAR:
        verdict = "UPGRADE" if cap_ratio >= _CAPACITY_FLOOR_FRAC else "CAPACITY-TRADEOFF"
    else:
        verdict = "OVERFIT"

    return {"verdict": verdict, "d_sharpe": d, "win_frac": win_frac,
            "capacity_ratio": cap_ratio, "mean_a": mean_a, "mean_b": mean_b,
            "cap_a": cap_a, "cap_b": cap_b}


# --- correlation (robustness) -------------------------------------------------------------------

def _pearson(a: dict[int, float], b: dict[int, float]) -> float:
    days = sorted(set(a) & set(b))
    if len(days) < 2:
        return 0.0
    xs = [a[d] for d in days]
    ys = [b[d] for d in days]
    mx, my = st.mean(xs), st.mean(ys)
    num = sum((x - mx) * (y - my) for x, y in zip(xs, ys, strict=True))
    dx = math.sqrt(sum((x - mx) ** 2 for x in xs))
    dy = math.sqrt(sum((y - my) ** 2 for y in ys))
    return (num / (dx * dy)) if dx > 0 and dy > 0 else 0.0


# --- the driver ---------------------------------------------------------------------------------

def broaden_overlay_validate(store: _FeatureStore, *, live_bound: float = _LIVE_BOUND,
                             broad_bounds: tuple[float, ...] = _DEFAULT_BROAD_BOUNDS,
                             n_folds: int = 4, k_grid: tuple[float, ...] = _K_GRID_WF,
                             lam_grid: tuple[float, ...] = _LAM_GRID_WF,
                             participation_frac: float = _PARTICIPATION_FRAC,
                             lookback_days: int = 90, taker_fee_bps: float = _TAKER_FEE_BPS,
                             sleeves: list[str] | None = None,
                             unlock_events: list[dict[str, Any]] | None = None) -> dict[str, Any]:
    """Walk-forward head-to-head: is BROAD + overlay an UPGRADE over the live liquid-61 ($10M) book? READ-ONLY,
    SHADOW eval. Builds the live and each broad per-coin panel once, cuts the day axis into expanding-window
    folds, and for each fold SELECTS the broad book's (breadth, k, λ) on TRAIN then evaluates both books NET on
    the disjoint TEST block. Aggregates OOS test Sharpe across folds, estimates each book's deployable capital,
    correlates the two NET series, and returns the verdict. Writes NOTHING."""
    from fxhnt.application.bybit_liquidity import liquid_universe

    # 1) build the live ($10M) panel + each broad panel ONCE (memory-bounded by each universe).
    live_universe = liquid_universe(store, min_dollar_vol=live_bound, lookback_days=lookback_days)
    live_daily, live_cost, live_spread = per_coin_weight_panel(
        store, universe=live_universe, sleeves=sleeves, unlock_events=unlock_events,
        taker_fee_bps=taker_fee_bps)
    if not live_daily or len(live_daily) < 4:
        return {"available": False, "reason": "no usable liquid-61 ($10M) per-coin book "
                "(run the Phase-A/B Bybit ingests first)"}

    broad_panels: dict[float, dict[str, Any]] = {}
    for bound in broad_bounds:
        universe = None if bound <= 0.0 else liquid_universe(store, min_dollar_vol=bound,
                                                             lookback_days=lookback_days)
        if bound > 0.0 and not universe:
            continue
        daily, cost, spread = per_coin_weight_panel(
            store, universe=universe, sleeves=sleeves, unlock_events=unlock_events,
            taker_fee_bps=taker_fee_bps)
        if daily and len(daily) >= 4:
            broad_panels[bound] = {"daily": daily, "cost": cost, "spread": spread}
    if not broad_panels:
        return {"available": False, "reason": "no broad universe produced a usable per-coin book"}

    # 2) fold the day axis (union of all candidate universes' days).
    all_days = set(live_daily)
    for p in broad_panels.values():
        all_days |= set(p["daily"])
    folds = walk_forward_folds(sorted(all_days), n_folds)
    if not folds:
        return {"available": False, "reason": f"too few days for {n_folds} walk-forward folds"}

    # 3) per fold: select broad (breadth, k, λ) on TRAIN, evaluate both books on the disjoint TEST block.
    fold_rows: list[dict[str, Any]] = []
    a_test_concat: dict[int, float] = {}
    b_test_concat: dict[int, float] = {}
    for f in folds:
        train_hi, test_lo, test_hi = f["train_hi"], f["test_lo"], f["test_hi"]

        # (A) LIVE: liquid-61 naive eq-wt (k=0, λ=0), tested on the held-out block.
        a = _eval_window(live_daily, live_cost, live_spread, k=0.0, lam=0.0,
                         day_lo=test_lo, day_hi=test_hi)
        cap_a = book_capacity(store, live_daily, live_spread, k=0.0, day_lo=test_lo, day_hi=test_hi,
                              participation_frac=participation_frac)

        # (B) BROAD+overlay: select (breadth, k, λ) by best TRAIN Sharpe (no leakage), test on the block.
        best_sel: dict[str, Any] | None = None
        for bound, p in broad_panels.items():
            for k in k_grid:
                for lam in lam_grid:
                    tr = _eval_window(p["daily"], p["cost"], p["spread"], k=k, lam=lam,
                                      day_lo=None, day_hi=train_hi)
                    if tr["days"] < 2:
                        continue
                    if best_sel is None or tr["sharpe"] > best_sel["train_sharpe"]:
                        best_sel = {"bound": bound, "k": k, "lam": lam, "train_sharpe": tr["sharpe"]}
        if best_sel is None:
            continue
        bp = broad_panels[best_sel["bound"]]
        b = _eval_window(bp["daily"], bp["cost"], bp["spread"], k=best_sel["k"], lam=best_sel["lam"],
                         day_lo=test_lo, day_hi=test_hi)
        cap_b = book_capacity(store, bp["daily"], bp["spread"], k=best_sel["k"], day_lo=test_lo,
                              day_hi=test_hi, participation_frac=participation_frac)

        a_test_concat.update(a["net_series"])
        b_test_concat.update(b["net_series"])
        fold_rows.append({
            "fold": f["fold"], "train_hi": train_hi, "test_lo": test_lo, "test_hi": test_hi,
            "selected_bound": best_sel["bound"], "selected_k": best_sel["k"], "selected_lam": best_sel["lam"],
            "live": {kk: a[kk] for kk in ("sharpe", "total_return", "max_dd", "turnover", "cost_drag_bps", "days")},
            "broad": {kk: b[kk] for kk in ("sharpe", "total_return", "max_dd", "turnover", "cost_drag_bps", "days")},
            "d_sharpe": b["sharpe"] - a["sharpe"],
            "cap_live": cap_a["bottleneck_capital"], "cap_broad": cap_b["bottleneck_capital"],
            "cap_live_detail": cap_a, "cap_broad_detail": cap_b,
        })

    if not fold_rows:
        return {"available": False, "reason": "no fold produced a usable head-to-head"}

    # 4) aggregate across folds.
    a_sharpes = [r["live"]["sharpe"] for r in fold_rows]
    b_sharpes = [r["broad"]["sharpe"] for r in fold_rows]
    mean_a, mean_b = st.mean(a_sharpes), st.mean(b_sharpes)
    std_a = st.pstdev(a_sharpes) if len(a_sharpes) > 1 else 0.0
    std_b = st.pstdev(b_sharpes) if len(b_sharpes) > 1 else 0.0
    win_frac = sum(1 for r in fold_rows if r["d_sharpe"] > 0.0) / len(fold_rows)
    # capacity comparison: aggregate (median across folds — robust to a single thin-coin spike).
    cap_a_agg = float(st.median([r["cap_live"] for r in fold_rows]))
    cap_b_agg = float(st.median([r["cap_broad"] for r in fold_rows]))

    verdict = decide_verdict(mean_a=mean_a, mean_b=mean_b, win_frac=win_frac,
                             cap_a=cap_a_agg, cap_b=cap_b_agg)

    # 5) robustness: correlation of the two OOS NET series + concentration (LARGE+FREQUENT+CONCENTRATED).
    corr = _pearson(a_test_concat, b_test_concat)
    hhis = [r["cap_broad_detail"]["hhi"] for r in fold_rows if r["cap_broad_detail"]["n_coins"] > 0]
    broad_hhi = float(st.mean(hhis)) if hhis else 0.0

    return {
        "available": True,
        "n_folds": len(fold_rows),
        "live_bound": live_bound,
        "broad_bounds": list(broad_panels),
        "participation_frac": participation_frac,
        "folds": fold_rows,
        "aggregate": {
            "mean_live_sharpe": mean_a, "mean_broad_sharpe": mean_b,
            "std_live_sharpe": std_a, "std_broad_sharpe": std_b,
            "d_sharpe": mean_b - mean_a, "win_frac": win_frac,
            "cap_live": cap_a_agg, "cap_broad": cap_b_agg,
            "capacity_ratio": verdict["capacity_ratio"],
            "corr_broad_to_live": corr, "broad_hhi": broad_hhi,
        },
        "verdict": verdict,
    }
