# Warehouse → Postgres consolidation — Design (A15)

**Date:** 2026-06-23
**Status:** Design (for review)
**Repo:** fxhnt

## Why
The warehouse is a single **DuckDB file on an RWO block volume** → single-writer, pinned to ONE node (the
8GB DEV1-L platform node `dagster` runs on). All warehouse compute is trapped there → batch jobs OOM (worst_basis
ingest, the carry+4-sleeve backfill), pod churn, serialized reads. The data is **small** (~5–10M rows, hundreds
of MB) — this is a *topology* problem, not a scale problem. Fix: move the feature store to the **networked
Postgres we already run**, so compute decouples from the storage node and batch can run anywhere (incl. a
bigger node). ClickHouse is overkill at this size; revisit only if we commit to tick-scale data.

## Target architecture
- **Feature store = TimescaleDB** (we ALREADY run `timescale/timescaledb:latest-pg16`, TimescaleDB 2.25.1, same
  instance as the operational DB `FXHNT_OPERATIONAL_DSN`; `forward_nav` is already a hypertable). Networked →
  cockpit, sim, backfill, ingest, dagster all connect over the network from any node/pod. **Zero new infra.**
- The `features` table is a **TimescaleDB hypertable with columnar compression** → near-DuckDB analytical speed
  (chunk exclusion on time-range + compressed columnar segments by feature/symbol) AND networked AND smaller on
  disk than the DuckDB file. This RESOLVES the row-store scan-speed caveat.
- **No RWO warehouse PVC, no podAffinity** for batch → batch jobs run on a bigger / scale-to-zero node pool.
- DuckDB kept behind a flag as a **fallback / cold backup** during/after cutover (reversible).

## The interface to preserve (drop-in)
`DuckDbFeatureStore` (`src/fxhnt/adapters/warehouse/duckdb_feature_store.py`) — long format
`features(symbol VARCHAR, ts BIGINT, feature VARCHAR, value DOUBLE)` + `ticker_membership`. Methods used across
19 sites: `write_features`, `write_features_bulk`, `read_features`, `read_panel(symbols, feature)`,
`crypto_close_panel`, `crypto_funding_panel`, `crypto_spot_panel`, `upsert_membership` (+ membership reads).
A `PostgresFeatureStore` must implement the SAME methods identically (same return shapes, same `suffix`/keying,
`ts//86400` → epoch_day).

## Phases
### 1. `PostgresFeatureStore` + factory
- New `src/fxhnt/adapters/warehouse/postgres_feature_store.py` — same interface, backed by Postgres tables
  `features(symbol text, ts bigint, feature text, value double precision)` (PK `(feature, symbol, ts)`) +
  `ticker_membership`. Index `(feature, symbol, ts)` for panel reads + `(symbol, ts)` for point reads.
  Writes = upsert (`INSERT ... ON CONFLICT (feature,symbol,ts) DO UPDATE`), preserving the scoped-idempotent
  semantics of `write_features` (re-running a day overwrites, doesn't duplicate).
- `get_feature_store(settings)` factory in one place; `FXHNT_FEATURE_STORE=postgres|duckdb` setting (default
  duckdb until cutover). Tests: parity — same fixture written to both stores yields identical
  `crypto_*_panel`/`read_panel`/`read_features` output.

### 2. Data migration (idempotent, one-shot)
- A CLI `fxhnt warehouse-migrate-pg`: read all `features` + `ticker_membership` from the DuckDB file, bulk-upsert
  into Postgres (chunked, memory-bounded — stream by symbol/feature, don't load all in RAM). Idempotent
  (ON CONFLICT). Verify: row counts + a few panel checksums match DuckDB.

### 3. Cutover
- Replace the 19 direct `DuckDbFeatureStore(...)` instantiations with `get_feature_store(settings)`.
- Flip `FXHNT_FEATURE_STORE=postgres`. Cockpit/sim/backfill/ingest now read/write Postgres.
- Keep DuckDB as cold backup (the flag flips back if needed).

### 4. Unpin batch compute (the payoff)
- Remove the warehouse RWO PVC mount + `podAffinity: dagster` from the backfill/ingest Jobs (they now reach the
  warehouse over Postgres). Give batch Jobs adequate memory.
- Add a **scale-to-zero batch node pool** (POP2-4C-16G — same type as `ci-compile-cpu`, in stock) via the
  Kapsule terragrunt (`infra/live/production/kapsule`); batch Jobs target it (nodeSelector + toleration),
  scaled 0 when idle (≈free). Platform pool stays DEV1-L. Re-run the carry+4-sleeve backfill + the Bybit/worst_basis
  ingests there without OOM.

## Schema notes
- One `features` table for all features (close/funding/high/low/spot_close + futures), as today — new features
  need no migration. Bybit data lands as additional symbols/features (or a `venue` column if we want to keep
  Binance/Bybit separate — decide at Bybit ingest time; default: prefix/suffix the symbol or add `venue`).
- Postgres disk: hundreds of MB of features — trivial for the existing Postgres PVC.

## Risks / rollback
- Parity bug → caught by Phase-1 parity tests + Phase-2 checksum verify.
- Cutover → reversible via `FXHNT_FEATURE_STORE` flag (DuckDB retained).
- Panel-read latency: Postgres indexed panel load ~1–3s vs DuckDB ~0.35s — acceptable (the sim's bottleneck is
  the overlay compute, not the load); if it ever matters, add a read cache or revisit Parquet/DuckDB-on-S3.

## Acceptance criteria
- `PostgresFeatureStore` parity-tested against `DuckDbFeatureStore` (identical panel/read output on a fixture).
- Migration copies all data; row-count + checksum verification passes.
- Cutover via flag; full suite green with `FXHNT_FEATURE_STORE=postgres`.
- A batch job (the carry backfill) runs to completion on the new batch pool **without the warehouse PVC** and
  without OOM.
- No `*.dbn` staged; DuckDB retained as fallback.
