# Paper book: marginal-gate + cleanup — Implementation Plan (Phase 2F-A3e)

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or
> superpowers:executing-plans. TDD. Steps use checkbox (`- [ ]`).

**Goal:** Add leave-one-out marginal-Sharpe edge pruning (`marginal_gate`) to the paper book's sizing (both
backfill + snapshot, live==replay), and clear two logged follow-ups: dead `equal_weights` and the
`_regime_lookback` zero-fill bias.

**Architecture:** A pure `marginal_gate_active(returns, active)` helper in `application/paper_book.py` that
reuses `book_allocator.align_edges` + `_marginal_prune`; it prunes the active set before
`isv_adaptive_weights` in both `paper_risk_overlay` and `paper_book_returns`. `_regime_lookback` becomes
NaN-aware. `equal_weights` + its test removed. Read-only/additive; no new tables.

**Tech Stack:** Python 3.12, pytest. **Repo:** `~/Work/fxhnt`, branch `feat/paper-marginal-gate`.
**Spec:** `docs/superpowers/specs/2026-06-22-paper-marginal-gate-cleanup-design.md`.

**Verified context:**
- `book_allocator.align_edges(series_by_edge) -> (dates, {edge: np.ndarray})` (union axis, 0-fill flat days).
- `book_allocator._marginal_prune(window: {edge: np.ndarray}, max_sleeve_weight) -> {edge: np.ndarray}`
  (leave-one-out: bench a sleeve only when removing it raises trailing book Sharpe; shrinks to ≥1 sleeve).
- `paper_book.py` has: `isv_adaptive_weights(returns_by_sleeve, active)`, `_regime_lookback(series_by_sleeve,
  active)` (currently `np.zeros` fill + mean), `paper_risk_overlay` (weights=isv over active),
  `paper_book_returns` (per-date `active=[s for s,r in before_d.items() if r]`; `w_held=isv(before_d,active)`),
  constants `_VOL_LOOKBACK=60`, `_PHI=0.5`, `_L_MIN=10`, `_L_MAX=60`.
- `equal_weights` has NO production caller (only `tests/unit/test_equal_weights.py`). The
  `test_*_equal_weights_active_sleeves_size_positions` tests in backfill/snapshot assert cold-start sizing and
  do NOT call the function — leave them.

---

## Task 1: `marginal_gate_active` helper

**Files:** Modify `src/fxhnt/application/paper_book.py`; Test `tests/unit/test_marginal_gate_active.py`

- [ ] **Step 1: Failing tests**
```python
# tests/unit/test_marginal_gate_active.py
from fxhnt.application.paper_book import marginal_gate_active

def _series(vals, start=1):
    return {start + i: v for i, v in enumerate(vals)}

def test_benches_dead_non_diversifying_sleeve():
    good = _series([0.02, 0.01, 0.03, 0.00] * 15)     # positive-mean, real Sharpe
    dead = _series([-0.02, -0.01, -0.03, 0.00] * 15)  # negative-mean, drags the book
    kept = marginal_gate_active({"good": good, "dead": dead}, ["good", "dead"])
    assert "good" in kept and "dead" not in kept

def test_keeps_both_diversifying_sleeves():
    a = _series([0.03, 0.01] * 30)                    # both positive-mean, anti-phase -> diversify
    b = _series([0.01, 0.03] * 30)
    kept = marginal_gate_active({"a": a, "b": b}, ["a", "b"])
    assert set(kept) == {"a", "b"}

def test_cold_sleeve_passes_through_and_no_pruning_under_two_history():
    good = _series([0.02, 0.01] * 30)
    kept = marginal_gate_active({"good": good, "new": {1: 0.01}}, ["good", "new"])  # only 1 has-history
    assert kept == ["good", "new"]                    # <2 has-history -> unchanged, order preserved

def test_never_empty_and_preserves_input_order():
    a = _series([0.02, 0.01] * 30); b = _series([0.015, 0.02] * 30)
    kept = marginal_gate_active({"a": a, "b": b}, ["a", "b"])
    assert kept and kept == [s for s in ["a", "b"] if s in kept]
```
- [ ] **Step 2:** Run → FAIL. (If the exact Sharpe ordering in test 1/2 differs from intent, adjust the
  numeric series to make `dead` clearly book-Sharpe-lowering / `a`,`b` clearly diversifying — preserve the
  assertion intent, report any change.)
- [ ] **Step 3:** Implement in `paper_book.py` (after `isv_adaptive_weights`; add the import at top with the
  other imports):
```python
from fxhnt.application.book_allocator import align_edges as _align_edges, _marginal_prune
```
```python
def marginal_gate_active(returns_by_sleeve: dict[str, dict[int, float]], active_sleeves: list[str], *,
                         vol_lookback: int = _VOL_LOOKBACK, max_sleeve_weight: float = _PHI) -> list[str]:
    """Leave-one-out marginal-Sharpe edge gate (reuses book_allocator._marginal_prune): bench a has-history
    sleeve only when REMOVING it raises the trailing book Sharpe (keeps anticorrelated hedges, benches dead /
    non-diversifying sleeves). Cold sleeves (<2 returns) pass through untouched; <2 has-history → unchanged.
    Causal over the trailing `vol_lookback`. Returns the kept active sleeves in input order. Pure."""
    has = [s for s in active_sleeves if len(returns_by_sleeve.get(s, {})) >= 2]
    cold = [s for s in active_sleeves if s not in has]
    if len(has) < 2:
        return list(active_sleeves)
    _, arrs = _align_edges({s: returns_by_sleeve[s] for s in has})
    window = {s: a[max(0, a.size - vol_lookback):] for s, a in arrs.items()}
    kept_has = set(_marginal_prune(window, max_sleeve_weight))
    return [s for s in active_sleeves if s in cold or s in kept_has]
```
- [ ] **Step 4:** Run → PASS (`uv run pytest tests/unit/test_marginal_gate_active.py -q`).
- [ ] **Step 5:** Commit `git commit -m "feat(paper): marginal_gate_active (leave-one-out edge pruning)"`

---

## Task 2: wire marginal_gate into `paper_risk_overlay` + `paper_book_returns`

**Files:** Modify `src/fxhnt/application/paper_book.py`; Test extend `tests/unit/test_paper_risk_overlay.py`

- [ ] **Step 1: Failing test** (append):
```python
def test_overlay_drops_dead_sleeve_from_weights():
    from fxhnt.application.paper_book import paper_risk_overlay
    good = {i + 1: v for i, v in enumerate([0.02, 0.01, 0.03, 0.00] * 15)}
    dead = {i + 1: v for i, v in enumerate([-0.02, -0.01, -0.03, 0.00] * 15)}
    w, _lev, _killed = paper_risk_overlay({"good": good, "dead": dead}, ["good", "dead"])
    assert "good" in w and w.get("dead", 0.0) == 0.0   # dead benched -> no weight
```
- [ ] **Step 2:** Run → FAIL (dead currently gets inverse-vol weight).
- [ ] **Step 3:** In `paper_risk_overlay`, prune before weighting:
```python
    active = marginal_gate_active(returns_by_sleeve, active_sleeves)
    weights = isv_adaptive_weights(returns_by_sleeve, active)
```
  (replace the existing `weights = isv_adaptive_weights(returns_by_sleeve, active_sleeves)` line; everything
  else in the fn unchanged.)
  In `paper_book_returns`, inside the loop, prune the per-date active set before `isv_adaptive_weights`:
```python
        active = [s for s, r in before_d.items() if r]
        active = marginal_gate_active(before_d, active)
        w_held = isv_adaptive_weights(before_d, active)
```
- [ ] **Step 4:** Run → PASS (`uv run pytest tests/unit/test_paper_risk_overlay.py -q`).
- [ ] **Step 5:** Regression `uv run pytest tests/unit/test_isv_adaptive_weights.py tests/unit/test_vol_target_leverage.py tests/unit/test_drawdown_killswitch.py -q`.
- [ ] **Step 6:** Commit `git commit -m "feat(paper): apply marginal_gate in risk overlay + book-return replay"`

---

## Task 3: `_regime_lookback` NaN-aware

**Files:** Modify `src/fxhnt/application/paper_book.py`; Test `tests/unit/test_regime_lookback.py`

- [ ] **Step 1: Failing/guard test**
```python
# tests/unit/test_regime_lookback.py
from fxhnt.application.paper_book import _regime_lookback

def test_late_joining_sleeve_does_not_zero_bias_regime():
    # 'a' alone has a rising-vol regime over 60 days -> L_t < 60.
    calm = [0.005, -0.005] * 25; hot = [0.05, -0.05] * 5
    a = {i + 1: v for i, v in enumerate(calm + hot)}
    assert _regime_lookback({"a": a}, ["a"]) < 60
    # 'b' joins only for the last 10 days. Zero-filling its first 50 days would distort the pooled
    # mean's vol; NaN-aware ignores the missing cells, so the regime is still detected (L_t < 60).
    b = {i + 1: (0.005 if i % 2 else -0.005) for i in range(50, 60)}
    assert _regime_lookback({"a": a, "b": b}, ["a", "b"]) < 60

def test_all_calm_holds_l_max():
    flat = {i + 1: (0.01 if i % 2 else -0.01) for i in range(60)}
    assert _regime_lookback({"a": flat}, ["a"]) == 60
```
- [ ] **Step 2:** Run → the late-joining test may already pass or fail depending on the zero-fill distortion;
  run to see. Either way implement Step 3 (the NaN-aware version is the correct intent).
- [ ] **Step 3:** Replace the matrix build + mean in `_regime_lookback` (keep the rest — the `< 2*_L_MIN`
  early return, the `sp <= 1e-12` guard, the final clamp):
```python
    idx = {d: i for i, d in enumerate(days)}
    mat = np.full((len(active), len(days)), np.nan)
    for r, s in enumerate(active):
        for d, v in series_by_sleeve.get(s, {}).items():
            mat[r, idx[d]] = v
    # NaN-aware mean across active sleeves so a sleeve missing a day does NOT get a spurious 0
    # (which would depress the pooled vol and bias the regime ratio toward L_max).
    with np.errstate(invalid="ignore"):
        rep = np.nanmean(mat, axis=0)
    rep = np.nan_to_num(rep, nan=0.0)   # a day with NO active sleeve -> neutral 0 (rare; union axis)
    recent, prior = rep[-_L_MIN:], rep[-2 * _L_MIN:-_L_MIN]
    sp = float(np.std(prior))
    if sp <= 1e-12:
        return _L_MAX
    rho = float(np.std(recent)) / sp
    return int(round(min(_L_MAX, max(_L_MIN, _L_MAX / max(1.0, rho)))))
```
- [ ] **Step 4:** Run → PASS (`uv run pytest tests/unit/test_regime_lookback.py tests/unit/test_isv_adaptive_weights.py -q`; the existing A3c regime test inside the isv suite must stay green).
- [ ] **Step 5:** Commit `git commit -m "fix(paper): _regime_lookback NaN-aware (drop zero-fill bias)"`

---

## Task 4: remove dead `equal_weights`

**Files:** Modify `src/fxhnt/application/paper_book.py`; Delete `tests/unit/test_equal_weights.py`

- [ ] **Step 1:** Confirm no production caller: `grep -rn "equal_weights" src/` shows only the `def
  equal_weights` line (and `marginal_gate_active`/others unrelated). If any `src/` file calls it, STOP and
  report (do not remove).
- [ ] **Step 2:** Remove the `equal_weights` function from `src/fxhnt/application/paper_book.py` and
  `git rm tests/unit/test_equal_weights.py`.
- [ ] **Step 3:** Run the paper suite: `uv run pytest tests/unit/ tests/integration/ -k paper -q` → all green
  (the backfill/snapshot `*_equal_weights_*` cold-start tests still pass; they don't call the function).
- [ ] **Step 4:** Commit `git commit -m "refactor(paper): remove dead equal_weights helper + its test"`

---

## Task 5: Full suite + finish + deploy + re-backfill + verify

- [ ] **Step 1:** `cd ~/Work/fxhnt && uv run pytest -q` → all green.
- [ ] **Step 2:** `git status --short | grep -i '\.dbn'` must print nothing.
- [ ] **Step 3:** **superpowers:finishing-a-development-branch** — merge `feat/paper-marginal-gate` → master,
  push (auto-deploys via Gitea→Argo). Confirm cockpit workflow Succeeded + dashboard/dagster pods rolled.
- [ ] **Step 4:** Re-run backfill in-cluster:
  `kubectl exec deploy/dagster -c daemon -n foxhunt -- fxhnt paper-backfill --days 365`.
- [ ] **Step 5: VERIFY** — book still populated (gate didn't empty it) + whether any sleeve gets benched:
```bash
kubectl exec deploy/dagster -c daemon -n foxhunt -- python -c "
from sqlalchemy import create_engine, text
from fxhnt.config import get_settings
e=create_engine(get_settings().operational_dsn)
with e.connect() as c:
    print('positions per sleeve:', list(c.execute(text('select sleeve,count(*) n,count(distinct run_date) d from paper_positions group by sleeve order by sleeve'))))
    print('shadow sleeves (composition always present):', list(c.execute(text('select sleeve,count(distinct run_date) d from paper_shadow_positions group by sleeve order by sleeve'))))
    print('dates where a shadow sleeve had NO throttled position (benched-or-killed):', list(c.execute(text('select s.sleeve, count(*) from (select distinct run_date,sleeve from paper_shadow_positions) s left join (select distinct run_date,sleeve from paper_positions) p on s.run_date=p.run_date and s.sleeve=p.sleeve where p.sleeve is null group by s.sleeve order by s.sleeve'))))
    print('nav span:', list(c.execute(text('select round(min(equity)::numeric,0),round(max(equity)::numeric,0),count(*) from paper_nav'))))
"
```
  Confirm the book is non-empty; compare benched/killed dates per sleeve (the marginal gate may now bench a
  sleeve on some dates where the throttled book is otherwise non-flat). Load `https://dashboard.fxhnt.ai/paper`
  + `/paper/replay` and report what renders + whether any sleeve gets benched.

---

## Self-review notes
- `marginal_gate_active` reuses the trusted `_marginal_prune` (leave-one-out, keeps hedges) — not a Sharpe floor.
- Same helper in `paper_risk_overlay` + `paper_book_returns` → live `/paper` == `/paper/replay`.
- Cold sleeves pass through; `<2` has-history → unchanged; never empties the book (`_marginal_prune` ≥1).
- Causal: prunes over strictly-prior `returns_by_sleeve` / `before_d`.
- `_regime_lookback` NaN-aware: missing cell ≠ spurious 0; `L_t` still bounded `[L_min, L_max]`.
- `equal_weights` removal verified to have no production caller first.
- NEVER `git add -A`; `*.dbn` gitignored — verify nothing stray staged before each commit.

## Acceptance criteria (from spec)
- `marginal_gate_active`: benches dead/non-diversifying, keeps diversifying, cold pass-through, <2-history
  unchanged, never empty. ✅ T1
- applied in overlay + book-return replay; dead sleeve dropped from overlay weights. ✅ T2
- `_regime_lookback` NaN-aware; A3c regime tests green. ✅ T3
- `equal_weights` + its test removed; suite green. ✅ T4
- re-backfill → book non-empty; /paper + /paper/replay render; benched sleeves reported. ✅ T5
- read-only/additive; no `*.dbn`. ✅ T5
