# fxhnt edge-pipeline containerization (combined book) — design

**Date:** 2026-06-14
**Status:** Design approved (scope = edge only), ready for implementation plan
**Parent:** extends `2026-06-14-fxhnt-cockpit-dashboard-design.md` (Milestone 3 open follow-up #1)

## Problem

The cockpit CronJob is currently ingest-only: it reads tracker `*_state.json` from a PVC but does not produce
them. The fetch + track steps for the combined book still run on the laptop (foxhunt `scripts/surfer/`), so
the in-cluster pipeline is not actually laptop-independent. We will containerize the **edge strategy only** —
the combined book (crypto cross-sectional momentum 83% + futures trend hedge 17%, the one validated edge) —
so its full daily loop runs in the cluster: **fetch → forward-track → ingest → dashboard**.

## Scope

- **In:** port the two data fetchers into fxhnt as a first-class `fxhnt fetch` command, so the existing
  fxhnt-native combined-book tracker (`combined_book.py` + `forward_tracker.py` + `fxhnt forward-track`) runs
  end-to-end in-cluster; update the cockpit CronJob to run the full loop.
- **Out (deferred, fold in later via the registry pattern):** the funding-harvest tracker and the other
  foxhunt/surfer prototypes; multi-page deep crypto history (only the forward window is needed).
- **Out (still user-gated):** the live deploy itself (image push + `kubectl apply` + DNS re-point) — that is
  the cockpit plan's Task 17, run on explicit go.

## What exists (verified 2026-06-14)

| Piece | Reality |
|---|---|
| Crypto fetcher | foxhunt `scripts/surfer/fetch_crypto_pit.py` — stdlib + numpy, **no API key**, hits Binance `fapi` public endpoints; writes `data/surfer/crypto_pit/<SYM>.npz` with keys `day` (ms//86_400_000), `close`, `qvol`, `funding`. **Skips existing files** (incremental for new symbols only). |
| Futures fetcher | foxhunt `scripts/surfer/fetch_daily.py` — Databento (`GLBX.MDP3`, ohlcv-1d, parent symbology), `DATABENTO_API_KEY` from env, hard cost cap + per-root `get_cost` gate, clamps `END` to dataset range; writes `data/surfer/<ROOT>.dbn`. Overwrites each run. |
| Crypto consumer | fxhnt `crypto_momentum_research.load_crypto_panel(crypto_dir)` reads `<crypto_dir>/*.npz` needing keys `day` + `close`. |
| Futures consumer | fxhnt `adapters/data/dbn_local.LocalDbnLoader.load(f"{dir}/{sym}.dbn", sym)`. |
| Book + tracker | fxhnt `CombinedBook(data_dir, futures, crypto_subdir="crypto_pit")`, `CombinedBookForwardTracker(book, state).step()`, CLI `fxhnt forward-track --data-dir --state`. Already fxhnt-native. |

## Design

### Two ported fetch adapters (fxhnt, network injected for testability)

- `src/fxhnt/adapters/data/crypto_fetch.py` — `fetch_crypto_pit(data_dir, *, top_n=120, get=<default urllib JSON GET>, sleep=<default>) -> tuple[int,int]`.
  - Writes `<data_dir>/crypto_pit/<SYM>.npz` with keys `day, close, qvol, funding` (drop-in for `load_crypto_panel`).
  - **Behavioral change vs the original: REFRESH, not skip.** Daily forward tracking needs fresh days, so the
    ported version overwrites. It uses a **single klines page** (`limit=1500`, no deep `endTime` pagination):
    ~1500 daily bars is far more than the combined book's 30-day momentum lookback needs, and one request per
    symbol keeps a daily refresh cheap (free Binance, ~155 requests). Funding is fetched best-effort (one page).
  - The HTTP GET is an injected callable (default: real `urllib.request` JSON GET) so tests run offline with
    canned `exchangeInfo` / `ticker` / `klines` responses. Symbols with `< 200` bars are skipped (as before).
  - Survivorship-free universe = top-N by 24h quote-volume ∪ the known-dead perp list (ported verbatim).

- `src/fxhnt/adapters/data/futures_fetch.py` — `fetch_futures(data_dir, *, roots, start, end, max_cost_usd, per_root_cap_usd, client=<default databento Historical>) -> tuple[int,int,float]`.
  - Ports `fetch_daily.py` faithfully: per-root `get_cost` gate, cumulative cap, `END` clamp to dataset range,
    retry loop, write `<data_dir>/<ROOT>.dbn`. The Databento client is injected (default reads
    `DATABENTO_API_KEY` from env) so tests exercise the cost-gate / cap / skip logic with a fake client (no
    network, no spend). Returns `(records, roots_ok, spent_usd)`.

### CLI

- `fxhnt fetch --data-dir <dir> [--crypto/--no-crypto] [--futures/--no-futures]` — runs the selected fetchers
  into `data_dir`. Futures needs `DATABENTO_API_KEY` in env; if absent and `--futures`, it errors clearly.
  Roots/start/cost-cap come from `Settings.databento` (already exists) with sane defaults.

### CronJob (full loop, futures weekly for cost control)

The cockpit CronJob command becomes (one image, all fxhnt-native):

```sh
fxhnt migrate-cockpit
fxhnt fetch --data-dir /data/surfer --crypto --no-futures        # crypto daily, free
[ "$(date -u +%u)" = "7" ] && fxhnt fetch --data-dir /data/surfer --no-crypto --futures   # futures Sundays only (~$25/mo)
fxhnt forward-track --data-dir /data/surfer --state /data/surfer/fxhnt_combined_forward.json
fxhnt ingest-forward --state-dir /data/surfer
```

`DATABENTO_API_KEY` is injected from `foxhunt-secrets`. The PVC persists `crypto_pit/`, `<root>.dbn`, and the
forward state JSON between runs. Schedule stays `30 23 * * *` (matches the retired laptop cron). The combined
book books a day only when both sleeves have data through that date (futures lag on non-Sundays just delays the
trend sleeve's newest point; the forward NAV advances when the overlap extends — acceptable for a slow hedge).

## Error handling

- Crypto fetch: a symbol that fails to fetch or has `< 200` bars is skipped (logged), never aborts the run.
- Futures fetch: a root over the per-root cap or that would breach the cumulative cap is skipped (logged); the
  cost gate runs **before** any download (no surprise spend) — this is a hard-won invariant, preserve it.
- `fxhnt fetch --futures` with no `DATABENTO_API_KEY` exits non-zero with a clear message (no silent skip).

## Testing (TDD)

- `crypto_fetch`: offline with an injected `get` returning canned JSON — universe selection, npz keys/shape
  written, `< 200`-bar skip, refresh-overwrites-existing.
- `futures_fetch`: injected fake Databento client — per-root cost gate, cumulative cap stop, `END` clamp, the
  `to_file` write path, returns `(records, ok, spent)`. No network.
- `fxhnt fetch` CLI smoke with both fetchers monkeypatched (no network): writes into a temp data-dir; the
  missing-key error path for `--futures`.

## Milestones

- **M4 — Edge fetch port:** `crypto_fetch` + `futures_fetch` adapters + `fxhnt fetch` CLI (TDD, offline).
- **M5 — CronJob full loop:** update `infra/k8s/jobs/fxhnt-forward-cronjob.yaml` to fetch → track → ingest;
  server-dry-run validate. (Live deploy remains the user-gated Task 17.)
