# fxhnt Cockpit — Postgres-backed in-cluster dashboard

**Date:** 2026-06-14
**Status:** Design approved, ready for implementation plan
**Author:** brainstormed with Jeroen

## Problem

The research program now runs a fleet of no-capital paper-forward trackers (combined book, crypto
funding-harvest, crossvenue, multistrat, 60/40, gd, poc) plus a strategy factory (research runs,
regime-tagged specialists, a global trial count). Their state is scattered across `*_state.json` files on a
laptop (xps-ubnt) and a local SQLite/Postgres. To make a go/no-go call the user currently has to *ask*. We
want a single Postgres-backed cockpit to *follow* — forward NAV, Sharpe, drawdown, and Phase-1 gate status
per strategy — hosted in the existing Scaleway cluster, behind Tailscale, at `dashboard.fxhnt.ai`,
independent of the laptop being on.

## Goals

- One read-only cockpit that shows the **whole paper fleet** + the **factory population** + recent
  **research runs**, refreshed automatically.
- **Single source of truth = the in-cluster TimescaleDB** (`postgres.foxhunt.svc.cluster.local:5432`,
  `timescale/timescaledb:latest-pg16`, already running). The dashboard reads only the DB.
- **Fully in-cluster, laptop-independent**: a daily k8s CronJob fetches data, runs each tracker, and ingests
  normalized rows into the cluster DB. A Deployment serves the dashboard; a per-service Tailscale proxy
  exposes `dashboard.fxhnt.ai` over the tailnet with TLS.
- Clean hexagonal fit inside fxhnt (new ports + adapters), DRY with existing domain/persistence, TDD.

## Non-goals

- No in-app authentication — Tailscale membership is the identity boundary (private to the tailnet).
- No capital, no live order routing. This is a *following* surface over paper-forward tracks.
- No migration of GB of market data: the fetch **regenerates** `.dbn`/`.npz` each run, so the PVC is
  scratch; only the small tracker `*_state.json` files persist between runs.
- Not replacing Grafana (`grafana.fxhnt.ai`) — that stays for ops/infra metrics. This is the product cockpit.

## Existing infrastructure (verified 2026-06-14)

| Asset | State |
|---|---|
| Cluster | 3-node Scaleway `scw-foxhunt-platform`, k8s 1.34, ns `foxhunt`/`monitoring`/`tailscale`/`cert-manager` |
| In-cluster DB | `postgres.foxhunt.svc.cluster.local:5432`, TimescaleDB pg16, running 63d; `redis` also present |
| Exposure pattern | **per-service Tailscale-proxy Deployment** (e.g. `tailscale-gitlab-proxy`), NOT k8s ingress |
| `dashboard.fxhnt.ai` | DNS name reserved (→ tailnet), no live route yet |
| Broken | `tailscale-subnet-router` in CrashLoopBackOff (17k restarts) — use per-service proxy instead |
| Deploy patterns to mirror | `infra/k8s/services/*.yaml` (e.g. `web-dashboard.yaml`), `infra/docker/` Kaniko push |
| fxhnt persistence | SQLAlchemy 2.0 ORM (`research_runs`, `strategies`, `factory_meta`, `seen_candidates`) over one `operational_dsn`; SQLite↔Postgres same models |
| Logo | `~/Work/foxhunt/fxhnt.png` |
| Trackers (fleet) | `data/surfer/*_state.json`: `fxhnt_combined_forward`, `funding_paper_state`, `crossvenue_state`, `multistrat_state`, `gd_paper_state`, `poc_state`, `sixtyforty_state` |

## Architecture

Hexagonal, all inside `fxhnt`:

```
domain/                          (unchanged — pure strategy/validation logic)
ports/
  dashboard.py                   read-model contract (DashboardReadModel)
  forward_sink.py                ingest contract (write ForwardNavRow[])
application/
  dashboard_service.py           queries -> view-models (fleet, detail, factory)
  forward_ingest.py              normalize tracker state -> DB rows (the ETL)
  gate.py                        Phase-1 go/no-go logic (per strategy gate_spec)
adapters/
  persistence/forward_nav.py     TimescaleDB hypertable repo (SQLAlchemy)
  persistence/state_reader.py    parse each *_state.json -> normalized ForwardNavRow
  web/app.py                     FastAPI composition root
  web/templates/*.html           Jinja2 + HTMX (auto-refresh)
  web/charts.py                  inline-SVG sparklines / NAV curves (no JS libs)
cli.py                           new sub-commands: ingest-forward, serve, migrate-cockpit
```

The **state_reader is the normalization seam**: the one place that understands the heterogeneous
`*_state.json` shapes and maps them to a single `ForwardNavRow`. Trackers keep writing their own JSON state
(their working state); ingest is the ETL into the DB. New strategy = teach the reader its shape + one
registry row.

## Data model (cluster TimescaleDB = SSOT)

- **`forward_nav`** — TimescaleDB hypertable, append-only daily rows:
  `date DATE, strategy_id TEXT, ret DOUBLE, nav DOUBLE, sharpe DOUBLE, maxdd DOUBLE,
   src_updated_at TIMESTAMPTZ`. Primary key `(strategy_id, date)`; **idempotent upsert** so re-ingest is
  safe. `create_hypertable('forward_nav', by_range('date'))`.
- **`strategy_registry`** — `strategy_id TEXT PK, display_name, sleeve, venue, capital_base DOUBLE,
   gate_spec JSON, enabled BOOL`. Maps the technical tracker → what the user sees + its gate thresholds.
- **`research_runs`, `strategies`, `factory_meta`, `seen_candidates`** — existing `sql_models.py`; created
  via `Base.metadata.create_all` against the cluster DSN. The factory view reads these.
- **Migration** = idempotent init step (a `fxhnt migrate-cockpit` command / k8s Job): `create_all` +
  `create_hypertable(..., if_not_exists => TRUE)` + seed `strategy_registry` from a checked-in registry.
  Hypertable DDL is guarded so the same code runs against SQLite in tests (plain table, no hypertable).

## Data flow (fully in-cluster)

```
k8s CronJob (foxhunt ns, daily 23:30 UTC)              in-cluster TimescaleDB
 1. fetch crypto (free, Binance) + futures (Sundays, Databento)   forward_nav      <-- upsert
 2. run each registered tracker -> updates *_state.json on PVC    strategy_registry
 3. fxhnt ingest-forward --state-dir /data/surfer --dsn $DSN  ----^
fxhnt-dashboard Deployment -- reads DB --> Service --> tailscale-proxy --> dashboard.fxhnt.ai (TLS, tailnet)
```

- Tracker commands live in a **registry** (extensible: add a strategy = add a registry entry + teach the
  reader its state shape). The CronJob iterates the registry: run command -> ingest.
- The PVC is scratch for regenerated market data and persists the small `*_state.json` between runs.
- Secrets (`DATABENTO_API_KEY`, any Binance keys) come from the existing `foxhunt-secrets`.
- v1 ships the fxhnt-owned pipeline (combined book + funding-harvest) in the CronJob; the remaining
  foxhunt/surfer scripts fold into the registry one at a time afterward (the ingest seam is universal, so
  the cockpit shows them as soon as their state files land on the PVC).

## Dashboard (FastAPI + HTMX, read-only, branded with fxhnt.png)

- `/` cockpit: fleet table (days, ret%, Sharpe, maxDD, gate badge) + NAV sparklines; HTMX polls for refresh.
- `/strategy/{id}`: full NAV curve (inline SVG), per-day rows, gate detail.
- `/factory`: strategy population by status/regime + global trial count + recent research runs.
- `/healthz`: liveness for k8s probes.
- One process, no Node/npm. Served via `fxhnt serve` (uvicorn). Logo bundled as a static asset.

## Gate logic

`application/gate.py` computes a per-strategy verdict (`WAIT` / `GO` / `NO_GO`) from the forward stats and
the strategy's `gate_spec` (thresholds: min forward days, forward net return/APR consistency band vs the
backtest, decay-toward-cost check). Pure function, unit-tested. The cockpit renders it as a badge; it never
acts on it.

## Error handling

- Ingest is idempotent and per-row best-effort: a malformed/unknown state file is logged and skipped, never
  aborts the batch (one bad tracker must not blind the whole cockpit).
- Dashboard degrades gracefully: a strategy with no `forward_nav` rows shows "no data yet", not an error.
- DB unreachable -> `/healthz` fails (k8s restarts) and pages show a clear banner, not a stack trace.
- CronJob fetch failures (e.g. Databento range lag) are non-fatal for the ingest of already-updated trackers.

## Testing (TDD, fxhnt convention)

- `state_reader` normalization — fixtures of each `*_state.json` shape -> expected `ForwardNavRow[]`.
- `forward_ingest` — upsert idempotency (re-ingest same state = no dupes, updated values applied).
- `dashboard_service` — queries against an in-memory SQLite seeded with known rows.
- `gate` — table-driven cases (wait/go/no-go) across thresholds.
- web routes — FastAPI `TestClient` smoke (200s, key fields present).
- Hypertable-specific DDL behind a Postgres guard (skipped on SQLite test runs).

## Milestones (for the implementation plan)

- **M1 — Data spine:** `forward_nav` + `strategy_registry` schema, `migrate-cockpit`, `state_reader`,
  `forward_ingest`, `ingest-forward` CLI. Tested locally against the cluster DB over Tailscale.
- **M2 — Cockpit app:** `dashboard.py` port, `dashboard_service`, `gate`, `web/` adapter (FastAPI + HTMX +
  SVG charts) + branding, `serve` CLI.
- **M3 — In-cluster deploy:** `infra/docker/fxhnt.Dockerfile` (+ Kaniko push), `fxhnt-forward-cronjob.yaml`
  (fetch + trackers + ingest, PVC, secrets), `fxhnt-dashboard.yaml` (Deployment + Service +
  per-service Tailscale proxy for `dashboard.fxhnt.ai`). **Cutover:** once the in-cluster CronJob is
  verified writing to the cluster DB, retire the laptop cron (`fxhnt_forward_cron.sh`) so the pipeline runs
  in exactly one place.

## Open follow-ups (post-v1, not in this spec)

- Fold the remaining foxhunt/surfer trackers into the CronJob registry.
- Optional: surface the same data in Grafana via the TimescaleDB datasource for ops-style alerting.
- Capital-relative sizing / honest capacity once a tracker clears its Phase-1 gate.
