# IBKR UCITS Volume Source Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Checkbox (`- [ ]`) steps.

**Goal:** Source the UCITS regime's ADV from **IBKR real daily volume** (`reqHistoricalData TRADES` — the authoritative on-exchange volume of the exact line we execute) instead of Yahoo's coarse LSE feed. Fetch runs in the IBKR-network-reachable rebalancer context → DB; the daemon/nightly reads the DB.

**Architecture:** A parallel `ibkr_pit_volume` PIT table (first-seen-frozen, mirrors `yahoo_pit_volume`); an `IbkrBroker.historical_daily_volume` method (reqHistoricalData TRADES); a `backfill-ucits-volume-ibkr` CLI (rebalancer context); a `ucits_volume_source` config flag (`yahoo` default | `ibkr`) that `equity_allocation_inputs` honors; a fetch CronJob. Spread stays the estimated per-line constant (real IBKR bid/ask needs a market-data subscription — Error 354, a documented follow-on).

**Tech Stack:** Python 3.12, SQLAlchemy 2.0, `ib_async` (runtime-installed in the rebalancer image), typer, pytest, k8s CronJob.

## Global Constraints

- **Probe-verified (2026-07-16):** IBKR `reqHistoricalData(contract, "", "N D", "1 day", "TRADES", useRTH=True, formatDate=1)` returns real daily volume for the UCITS lines (CSPX/CBU0/ICOM/DBMF all resolved via LSEETF+ISIN; DBMF ~16 shares/day confirms genuine illiquidity). `reqMktData` bid/ask = NaN / Error 354 (no market-data subscription) → **spread stays estimated; do NOT attempt real-time spread here.**
- **Network reality:** only pods labeled `app: multistrat-rebalancer` (or trading-service/broker-gateway) can reach `ib-gateway:4004` (NetworkPolicy). The daemon CANNOT. So the IBKR fetch MUST run in the rebalancer context (a CronJob), write to the DB; the daemon reads the DB. `ib_async` is a runtime `pip install ib_async==2.1.0` (extra `ibkr = ["ib-async>=2.0"]`), NOT in the base image.
- **PIT discipline:** `ibkr_pit_volume` is first-seen-frozen (mirror `snapshot_yahoo_volumes` — insert only (symbol,date) not present).
- **Byte-identity of `ucits_volume_source="yahoo"` (the default):** `equity_allocation_inputs` must be byte-identical to today when the flag is `yahoo`. The IBKR read path only activates on `ibkr`.
- Store IBKR volume keyed by the UCITS `yahoo_ticker` string (e.g. `CSPX.L`) so the equity provider's existing per-sleeve key mapping is reused unchanged — the source flag only switches WHICH table is read.
- Commit trailer: `Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>`.

---

### Task 1: IBKR volume storage + broker fetch method + config flag

**Files:**
- Modify: `src/fxhnt/adapters/persistence/cockpit_models.py` (`IbkrPitVolumeRow`)
- Modify: `src/fxhnt/adapters/persistence/forward_nav.py` (`snapshot_ibkr_volumes` / `ibkr_pit_volumes`)
- Modify: `src/fxhnt/adapters/broker/ibkr.py` (`historical_daily_volume`)
- Modify: `src/fxhnt/config.py` (`AllocationSettings.ucits_volume_source`)
- Test: `tests/unit/test_ibkr_volume_pit.py`

**Interfaces:**
- `IbkrPitVolumeRow` (`ibkr_pit_volume`, PK (symbol,date), volume Float, first_seen_at) — rides `create_all`.
- `ForwardNavRepo.snapshot_ibkr_volumes(symbol, series, at)` / `.ibkr_pit_volumes(symbol) -> dict[str,float]` (mirror the yahoo methods exactly, first-seen-frozen).
- `IbkrBroker.historical_daily_volume(*, ticker, isin, exchange="LSEETF", currency="USD", days=90) -> dict[str,float]`: resolve via `_resolve_contract`, `reqHistoricalData(contract, "", f"{days} D", "1 day", "TRADES", useRTH=True, formatDate=1)`, return `{bar.date.isoformat() or str(bar.date): float(bar.volume)}` for bars with `volume not in (None, -1)`. Empty dict on resolve-fail/no-data (log + return {}, never raise into the caller's loop).
- `AllocationSettings.ucits_volume_source: str = "yahoo"` (yahoo | ibkr), env `FXHNT_ALLOCATION_UCITS_VOLUME_SOURCE`, comment: which store the UCITS-regime ADV reads from; `ibkr` = the real IBKR on-exchange volume (fetched by the `ucits-volume-ibkr` CronJob), `yahoo` = the Yahoo LSE feed (default until the CronJob is running).

- [ ] **Step 1: Failing tests** — `ibkr_pit_volume` roundtrip + first-seen-frozen (mirror the yahoo volume test); `ucits_volume_source` default "yahoo" + env override "ibkr". (The broker method needs a live gateway — do NOT unit-test the fetch; test the storage + config only. Note in the test file why the broker fetch is integration-only.)
- [ ] **Step 2: Run — expect FAIL.**
- [ ] **Step 3: Implement** all four (mirror the yahoo volume + the existing `market_price` reqHistoricalData pattern for the broker method; `bar.date` from ib_async is a `datetime.date` — format to ISO `YYYY-MM-DD`).
- [ ] **Step 4: Run — expect PASS.**
- [ ] **Step 5: Commit** (`feat(equity): IBKR pit-volume store + historical_daily_volume + ucits_volume_source flag`).

---

### Task 2: `backfill-ucits-volume-ibkr` CLI + source-flag read

**Files:**
- Modify: `src/fxhnt/cli.py` (`backfill-ucits-volume-ibkr`)
- Modify: `src/fxhnt/application/equity_allocation_inputs.py` (read ibkr vs yahoo by flag)
- Test: `tests/unit/test_equity_allocation_inputs.py` (extend)

- [ ] **Step 1: Failing tests** — `equity_allocation_inputs(..., regime="ucits", volume_source="ibkr")` reads `ibkr_pit_volumes(ucits_ticker)`; `volume_source="yahoo"` (default) reads `yahoo_pit_volumes` (byte-identical to today, regression). Seed one repo with DIFFERENT ibkr vs yahoo volume for a UCITS ticker → the ceiling differs by source (proves the switch reads the right table).
- [ ] **Step 2: Run — expect FAIL.**
- [ ] **Step 3: Implement:**
  - `equity_allocation_inputs` gains `volume_source: str = "yahoo"` (default from `settings.allocation.ucits_volume_source` at the `record_allocation` call). In the ucits branch, `vol_read = repo.ibkr_pit_volumes if volume_source == "ibkr" else repo.yahoo_pit_volumes`; `ucits_volumes = {sym: vol_read(resolved_map[sym].yahoo_ticker) ...}`. The us branch is unchanged (always yahoo US volume — Databento is the future US source, out of scope). `record_allocation` threads `volume_source=get_settings().allocation.ucits_volume_source`.
  - `cli.py` `backfill-ucits-volume-ibkr`: `pip`-installed `ib_async` is assumed present (rebalancer context). For each `settings.ucits.map` listing, `IbkrBroker(host, port, client_id).historical_daily_volume(ticker=listing.ticker, isin=listing.isin, exchange=listing.exchange, currency=listing.currency, days=<opt, default 90>)` → `repo.snapshot_ibkr_volumes(listing.yahoo_ticker, series, at)`. Per-line try/except (a thin/unresolvable line logs + continues). Echo per-line day counts. Use `get_settings().ibkr` for host/port/client_id.
- [ ] **Step 4: Run — expect PASS** (+ full unit suite).
- [ ] **Step 5: Commit** (`feat(equity): backfill-ucits-volume-ibkr CLI + source-flag read in equity provider`).

---

### Task 3: fetch CronJob (infra) + full green — CONTROLLER-run (not a subagent task)
- [ ] Add `infra/k8s/.../ucits-volume-ibkr-cronjob.yaml`: label `app: multistrat-rebalancer`, image `fxhnt-cockpit:latest`, `imagePullSecrets: scw-registry`, env (FXHNT_OPERATIONAL_DSN + PGPASSWORD from the same secret the rebalancer uses, FXHNT_IBKR_HOST=ib-gateway, PORT=4004, CLIENT_ID=27), command `pip install --quiet ib_async==2.1.0 && fxhnt backfill-ucits-volume-ibkr`. Schedule daily before the ucits-rebalancer (e.g. `30 8 * * 1-5`). Clone the ucits-rebalancer's spec for the DSN/secret wiring.
- [ ] `.venv/bin/python3 -m pytest tests/unit -q` + `tests/integration -k "equity or allocation or ibkr or yahoo"` → green.

## Post-implementation (controller-run, in-cluster)
Deploy (git-sync) + migrate-cockpit (creates `ibkr_pit_volume`). Run the fetch as a one-off Job (rebalancer context) → confirm `ibkr_pit_volume` populates with real volume (DBMF ~16 shares/day). Flip `FXHNT_ALLOCATION_UCITS_VOLUME_SOURCE=ibkr` and re-run the equity gate → the book sizes on the real IBKR ADV (partial: liquid lines + DBMF scaled to ~0). Contrast the IBKR-sourced ceiling vs the Yahoo-sourced one. NOTE: the fetch MUST run labeled `app: multistrat-rebalancer` (netpol); the daemon cannot reach the gateway.

## Self-Review
- Byte-identity: `ucits_volume_source="yahoo"` (default) unchanged; ibkr path additive.
- Network-correct: fetch in rebalancer context → DB → daemon reads DB (no IBKR in the daemon).
- Spread deferred honestly (subscription follow-on; Error 354 documented).
- PIT first-seen-frozen for ibkr_pit_volume.
