feat(infra): re-enable prometheus+grafana+alertmanager, expose grafana.fxhnt.ai + dagster.fxhnt.ai (Phase 2E)

- scaled prometheus-stack operator/kube-state-metrics + grafana to 1; operator brought
  prometheus + alertmanager up. node-exporters already running. loki/tempo/dcgm stay 0.
- grafana-netpol: admit tailscale proxy -> grafana:3000 (fixes the 502)
- dagster.yaml netpol: + ingress proxy -> :3000; nginx dagster.fxhnt.ai block (websocket)
- dns module: + dagster A record (imported live; plan=No changes — no out-of-band drift)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitignore

@@ -9,3 +9,11 @@ __pycache__/
 .env
 build/
 dist/
+
+# terragrunt-generated
+**/.terraform/
+**/backend.tf
+**/provider.tf
+*.tfstate
+*.tfstate.*
+**/.terragrunt-cache/

docs/superpowers/plans/2026-06-17-equity-backtest-b3b.md

@@ -0,0 +1,792 @@
+# B3b — Walk-Forward Equity-Factor Backtest Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Backtest the 3 price-only equity-factor constructions (`long`/`ls`/`tilt`, momentum + low-vol) over the full survivorship-free price history in the dedicated backtest warehouse, producing a gauntlet GO/NO_GO verdict per construction.
+
+**Architecture:** Pure domain layer (PIT universe reconstruction + rebalance-date math + extracted factor-window helpers) consumed by an application-layer walk-forward runner that reads the DuckDB warehouse (never live Tiingo), books daily returns per construction across monthly rebalances, then runs each construction's daily-return series through the existing `compute_stats` + gauntlet `evaluate`. A CLI command runs it against the backtest warehouse and writes a JSON report.
+
+**Tech Stack:** Python 3.12, DuckDB warehouse (`DuckDbFeatureStore`), numpy, pydantic, Typer CLI, pytest (`.venv/bin/python -m pytest`), strict mypy. All tests use a synthetic on-tmp-path warehouse — no live network.
+
+**Reference spec:** `docs/superpowers/specs/2026-06-17-equity-backtest-design.md`.
+
+**House rules:** TDD (failing test first). Frozen dataclasses / pure functions in `domain/`. Fakes in tests, never live network. Strict mypy must stay clean (no NEW errors). Commit after each task. Do NOT push. Trailer on every commit:
+```
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
+```
+
+---
+
+## File Structure
+
+**New files:**
+- `src/fxhnt/domain/equity_backtest.py` — pure: `universe_asof`, `month_end_rebalance_dates`, `daily_returns_for_weights`.
+- `src/fxhnt/application/equity_backtest_runner.py` — walk-forward orchestration over the warehouse → per-construction daily return series → verdict → report dict.
+- `tests/unit/test_equity_backtest_domain.py` — domain unit tests.
+- `tests/integration/test_equity_backtest_runner.py` — runner tests on a synthetic warehouse.
+
+**Modified files:**
+- `src/fxhnt/domain/strategies/equity_factor.py` — add two pure helpers (`momentum_12_1_from_closes`, `realized_vol_from_closes`) extracted from the live path.
+- `src/fxhnt/application/equity_factor_strategy.py` — refactor `compute_factor_scores` to call the two new domain helpers (behaviour-preserving; DRY + single source of truth).
+- `src/fxhnt/cli.py` — add `backtest-equity` command.
+- `tests/unit/test_equity_factor.py` (or the existing equity_factor domain test) — tests for the two new helpers.
+
+---
+
+## Reuse contract (exact signatures — do not re-derive)
+
+```python
+# src/fxhnt/domain/strategies/equity_factor.py
+def momentum_score(mom_12_1: list[float | None]) -> list[float]      # = robust_z(mom_12_1)
+def lowvol_score(vols: list[float | None]) -> list[float]            # robust_z of negated vols
+def composite_price(momentum_z: list[float], lowvol_z: list[float]) -> list[float]
+def construction_weights(scores: list[float], construction: str, quantile: float = 0.2) -> list[float]
+def book_return(prev_weights: dict[str, float], prev_prices: dict[str, float],
+                cur_prices: dict[str, float], borrow_daily: float = 0.0) -> float
+
+# src/fxhnt/domain/backtest.py
+class BacktestStats(BaseModel): cagr: float; ann_vol: float; sharpe: float; max_drawdown: float; n_obs: int
+def compute_stats(returns: np.ndarray, periods_per_year: int = 252) -> BacktestStats
+
+# src/fxhnt/domain/gauntlet/core.py
+class Verdict(BaseModel): passed: bool; dsr: float; is_sharpe: float; oos_sharpe: float; n_trials: int; reasons: list[str]; pvalue: float
+def evaluate(is_returns, oos_returns, n_trials, sr_variance, *, dsr_min=0.95,
+             oos_min_sharpe=0.0, max_is_oos_decay=0.50, has_economic_rationale=None) -> Verdict
+
+# src/fxhnt/adapters/warehouse/duckdb_feature_store.py
+class DuckDbFeatureStore:
+    def __init__(self, path: str) -> None
+    def read_panel(self, symbols: list[str], feature: str) -> dict[str, dict[int, float]]   # {symbol: {epoch_day: value}}
+    def read_membership(self) -> list[tuple[str, str, str, str]]                              # (symbol, exchange, start_date, end_date)
+    def write_features_bulk(self, items: list[tuple[str, list[tuple[int, dict[str, float]]]]]) -> None
+    def upsert_membership(self, rows: list[tuple[str, str, str, str]]) -> None
+
+# src/fxhnt/config.py
+Settings.backtest_warehouse_path: str            # env FXHNT_BACKTEST_WAREHOUSE_PATH, default /backtest-data/warehouse.duckdb
+Settings.cost_bps_per_turnover: float = 10.0
+Settings.gauntlet: GauntletSettings              # dsr_min .95, oos_min_sharpe 0.0, max_is_oos_decay .50, oos_fraction .40
+```
+
+**Warehouse `ts` convention:** stored as epoch-seconds at 00:00 UTC; `read_panel` returns keys as **epoch-days** (`ts // 86400`). Convert an ISO date to epoch-day with the helper in Task 2 (`_epoch_day`). Stored features: `"adjclose"`, `"close"`, `"volume"`.
+
+**Cost model (spec default):** 15 bps per unit turnover (overrides the engine's 10.0 default; pass explicitly). OOS holdout 40%. n = 150 names. DSR n_trials = 3 (the three constructions).
+
+---
+
+## Task 1: Extract pure factor-window helpers (DRY the live path)
+
+**Why:** The backtest scorer and the live `compute_factor_scores` must compute 12-1 momentum and trailing realized vol from a close series **identically**. Today that window logic is inline in `equity_factor_strategy.py`. Extract it into the domain so there is one source of truth.
+
+**Files:**
+- Modify: `src/fxhnt/domain/strategies/equity_factor.py`
+- Modify: `src/fxhnt/application/equity_factor_strategy.py`
+- Test: `tests/unit/test_equity_factor.py` (append; if absent, create)
+
+- [ ] **Step 1: Read the existing live logic.** Open `src/fxhnt/application/equity_factor_strategy.py` and locate (a) the inline 12-1 momentum computation inside `compute_factor_scores` and (b) `_realized_vol`. Note the EXACT window constants used (lookback length for momentum, the 1-month skip, the vol window, min-history thresholds). These exact numbers are the source of truth — preserve them verbatim.
+
+- [ ] **Step 2: Write failing tests** in `tests/unit/test_equity_factor.py`:
+
+```python
+from fxhnt.domain.strategies.equity_factor import (
+    momentum_12_1_from_closes,
+    realized_vol_from_closes,
+)
+
+
+def _ramp(n: int, daily: float = 0.001) -> list[float]:
+    """Strictly increasing close series: 100 * (1+daily)**i."""
+    return [100.0 * (1.0 + daily) ** i for i in range(n)]
+
+
+def test_momentum_12_1_positive_for_uptrend():
+    # An up-trending series of sufficient length yields positive 12-1 momentum.
+    closes = _ramp(300)
+    m = momentum_12_1_from_closes(closes)
+    assert m is not None and m > 0.0
+
+
+def test_momentum_12_1_none_when_insufficient_history():
+    assert momentum_12_1_from_closes(_ramp(50)) is None
+
+
+def test_realized_vol_zero_for_constant_series():
+    v = realized_vol_from_closes([100.0] * 120)
+    assert v is not None and v == 0.0
+
+
+def test_realized_vol_positive_for_wiggly_series():
+    closes = [100.0 * (1.0 + (0.02 if i % 2 == 0 else -0.02)) for i in range(120)]
+    v = realized_vol_from_closes(closes)
+    assert v is not None and v > 0.0
+
+
+def test_realized_vol_none_when_insufficient_history():
+    assert realized_vol_from_closes([100.0] * 5) is None
+```
+
+- [ ] **Step 3: Run, verify failure.** `.venv/bin/python -m pytest tests/unit/test_equity_factor.py -k "momentum_12_1 or realized_vol" -v` → FAIL (ImportError).
+
+- [ ] **Step 4: Add the helpers** to `src/fxhnt/domain/strategies/equity_factor.py`. Move the EXACT window math from `equity_factor_strategy.py` (the numbers you read in Step 1 — do not change them). Signatures:
+
+```python
+def momentum_12_1_from_closes(closes: list[float]) -> float | None:
+    """12-1 month momentum from an ascending daily-adjusted-close series.
+    Returns the lookback-window return EXCLUDING the most recent ~1 month, or
+    None if there is insufficient history. Window lengths copied verbatim from
+    the prior inline logic in equity_factor_strategy.compute_factor_scores."""
+    ...
+
+
+def realized_vol_from_closes(closes: list[float]) -> float | None:
+    """Trailing realized volatility from an ascending close series (the prior
+    _realized_vol logic, moved verbatim). None if insufficient history."""
+    ...
+```
+
+- [ ] **Step 5: Run new tests.** `.venv/bin/python -m pytest tests/unit/test_equity_factor.py -k "momentum_12_1 or realized_vol" -v` → PASS.
+
+- [ ] **Step 6: Refactor the live path.** In `src/fxhnt/application/equity_factor_strategy.py`, replace the inline momentum computation and the `_realized_vol` body with calls to the new domain helpers (delete the now-dead inline code / make `_realized_vol` delegate, or remove it and call the domain helper directly). Behaviour MUST be identical.
+
+- [ ] **Step 7: Run the full equity-factor suite.** `.venv/bin/python -m pytest tests/ -k equity_factor -v` → all PASS (the existing live tests prove the refactor preserved behaviour). Then `.venv/bin/python -m pytest -q` → all PASS.
+
+- [ ] **Step 8: mypy.** `.venv/bin/python -m mypy src/fxhnt/domain/strategies/equity_factor.py src/fxhnt/application/equity_factor_strategy.py` → no new errors.
+
+- [ ] **Step 9: Commit.**
+```bash
+git add src/fxhnt/domain/strategies/equity_factor.py src/fxhnt/application/equity_factor_strategy.py tests/unit/test_equity_factor.py
+git commit -m "refactor(equity-factor): extract pure momentum_12_1/realized_vol window helpers (DRY live + backtest)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 2: PIT universe + rebalance-date domain
+
+**Files:**
+- Create: `src/fxhnt/domain/equity_backtest.py`
+- Test: `tests/unit/test_equity_backtest_domain.py`
+
+- [ ] **Step 1: Write failing tests** in `tests/unit/test_equity_backtest_domain.py`:
+
+```python
+from fxhnt.domain.equity_backtest import (
+    universe_asof,
+    month_end_rebalance_dates,
+    _epoch_day,
+)
+
+MEMBERS = [
+    ("LIVE", "NASDAQ", "2000-01-01", "2026-06-01"),   # active across all asof dates
+    ("DEAD", "NYSE", "1998-01-01", "2005-03-15"),     # delisted 2005
+    ("IPO", "NASDAQ", "2010-06-01", "2026-06-01"),    # IPO 2010
+]
+
+
+def test_universe_asof_excludes_not_yet_listed_and_delisted():
+    # At 2008-01-01: LIVE active, DEAD already delisted, IPO not yet listed.
+    dollar_vol = {"LIVE": 1e9, "DEAD": 1e9, "IPO": 1e9}
+    assert universe_asof(MEMBERS, dollar_vol, "2008-01-01", n=10) == ["LIVE"]
+
+
+def test_universe_asof_includes_delisted_while_alive():
+    # At 2003-01-01: LIVE + DEAD both alive (survivorship-free correctness).
+    dollar_vol = {"LIVE": 2e9, "DEAD": 1e9}
+    assert universe_asof(MEMBERS, dollar_vol, "2003-01-01", n=10) == ["LIVE", "DEAD"]
+
+
+def test_universe_asof_ranks_by_dollar_volume_desc_and_takes_top_n():
+    members = [(s, "NYSE", "2000-01-01", "2026-06-01") for s in ("A", "B", "C")]
+    dollar_vol = {"A": 10.0, "B": 30.0, "C": 20.0}
+    assert universe_asof(members, dollar_vol, "2020-01-01", n=2) == ["B", "C"]
+
+
+def test_universe_asof_drops_zero_or_missing_dollar_volume():
+    members = [(s, "NYSE", "2000-01-01", "2026-06-01") for s in ("A", "B")]
+    assert universe_asof(members, {"A": 5.0, "B": 0.0}, "2020-01-01", n=10) == ["A"]
+
+
+def test_month_end_rebalance_dates_one_per_month():
+    dates = ["2020-01-02", "2020-01-31", "2020-02-03", "2020-02-27", "2020-03-02"]
+    # last available trading date within each calendar month
+    assert month_end_rebalance_dates(dates) == ["2020-01-31", "2020-02-27", "2020-03-02"]
+
+
+def test_epoch_day_roundtrip():
+    # 1970-01-01 is epoch-day 0; 1970-01-02 is 1.
+    assert _epoch_day("1970-01-01") == 0
+    assert _epoch_day("1970-01-02") == 1
+```
+
+- [ ] **Step 2: Run, verify failure.** `.venv/bin/python -m pytest tests/unit/test_equity_backtest_domain.py -v` → FAIL (ImportError).
+
+- [ ] **Step 3: Implement** `src/fxhnt/domain/equity_backtest.py`:
+
+```python
+"""Pure walk-forward equity-factor backtest domain: point-in-time universe
+reconstruction and rebalance-date math. No I/O, no live network."""
+from __future__ import annotations
+
+import datetime as dt
+
+_SECONDS_PER_DAY = 86_400
+
+
+def _epoch_day(iso_date: str) -> int:
+    """ISO 'YYYY-MM-DD' -> epoch-day (matches DuckDbFeatureStore.read_panel keys)."""
+    d = dt.date.fromisoformat(iso_date)
+    return (d - dt.date(1970, 1, 1)).days
+
+
+def universe_asof(
+    members: list[tuple[str, str, str, str]],
+    dollar_vol: dict[str, float],
+    asof: str,
+    n: int,
+) -> list[str]:
+    """Point-in-time top-N universe at `asof` (ISO date).
+
+    A name is eligible iff it was listed at `asof` (start_date <= asof <= end_date)
+    AND has strictly positive trailing dollar-volume. Eligible names are ranked by
+    `dollar_vol` descending; the top `n` symbols are returned. Including names that
+    were alive at `asof` but later delisted is what makes this survivorship-free.
+    """
+    eligible = [
+        sym
+        for (sym, _exch, start, end) in members
+        if start <= asof <= end and dollar_vol.get(sym, 0.0) > 0.0
+    ]
+    eligible.sort(key=lambda s: dollar_vol[s], reverse=True)
+    return eligible[:n]
+
+
+def month_end_rebalance_dates(dates: list[str]) -> list[str]:
+    """Given ascending ISO trading dates, return the last available trading date
+    within each calendar (year, month) — the monthly rebalance schedule."""
+    last_by_month: dict[tuple[int, int], str] = {}
+    for d in dates:
+        y, m, _ = d.split("-")
+        last_by_month[(int(y), int(m))] = d
+    return [last_by_month[k] for k in sorted(last_by_month)]
+```
+
+- [ ] **Step 4: Run tests.** `.venv/bin/python -m pytest tests/unit/test_equity_backtest_domain.py -v` → PASS.
+
+- [ ] **Step 5: mypy.** `.venv/bin/python -m mypy src/fxhnt/domain/equity_backtest.py` → no errors.
+
+- [ ] **Step 6: Commit.**
+```bash
+git add src/fxhnt/domain/equity_backtest.py tests/unit/test_equity_backtest_domain.py
+git commit -m "feat(b3b): PIT universe_asof + month-end rebalance-date domain (survivorship-free)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 3: Daily-returns-for-weights domain
+
+**Why:** Between rebalances we hold fixed weights and accrue the daily book return. This is pure given the per-day price panel.
+
+**Files:**
+- Modify: `src/fxhnt/domain/equity_backtest.py`
+- Test: `tests/unit/test_equity_backtest_domain.py` (append)
+
+- [ ] **Step 1: Append failing tests:**
+
+```python
+from fxhnt.domain.equity_backtest import daily_returns_for_weights
+
+
+def test_daily_returns_long_only_matches_weighted_price_change():
+    weights = {"A": 0.5, "B": 0.5}
+    # day0 -> day1: A +10%, B 0%  => 0.5*0.10 + 0.5*0.0 = 0.05
+    closes = {
+        "A": {0: 100.0, 1: 110.0},
+        "B": {0: 50.0, 1: 50.0},
+    }
+    rets = daily_returns_for_weights(weights, closes, [0, 1], borrow_daily=0.0)
+    assert rets == [0.05]
+
+
+def test_daily_returns_applies_borrow_on_short_notional():
+    weights = {"A": -1.0}                       # 100% short
+    closes = {"A": {0: 100.0, 1: 100.0}}        # flat price
+    rets = daily_returns_for_weights(weights, closes, [0, 1], borrow_daily=0.001)
+    assert rets == [-0.001]                      # only the borrow cost
+
+
+def test_daily_returns_skips_day_with_missing_price():
+    weights = {"A": 1.0}
+    closes = {"A": {0: 100.0, 2: 121.0}}        # day 1 missing
+    # day0->1 unusable (no price@1); day1->2 unusable (no price@1). No usable steps.
+    rets = daily_returns_for_weights(weights, closes, [0, 1, 2], borrow_daily=0.0)
+    assert rets == []
+```
+
+- [ ] **Step 2: Run, verify failure.** `.venv/bin/python -m pytest tests/unit/test_equity_backtest_domain.py -k daily_returns -v` → FAIL.
+
+- [ ] **Step 3: Implement** (append to `src/fxhnt/domain/equity_backtest.py`):
+
+```python
+from fxhnt.domain.strategies.equity_factor import book_return
+
+
+def daily_returns_for_weights(
+    weights: dict[str, float],
+    closes: dict[str, dict[int, float]],
+    days: list[int],
+    borrow_daily: float,
+) -> list[float]:
+    """Daily book returns of fixed `weights` over consecutive `days` (epoch-days,
+    ascending). For each adjacent (prev, cur) pair, both prices must exist for a
+    symbol to contribute; a step with no usable price pair is dropped (not 0.0)."""
+    out: list[float] = []
+    for prev, cur in zip(days, days[1:]):
+        prev_p = {s: closes[s][prev] for s in weights if s in closes and prev in closes[s] and cur in closes[s]}
+        cur_p = {s: closes[s][cur] for s in prev_p}
+        if not prev_p:
+            continue
+        w = {s: weights[s] for s in prev_p}
+        out.append(book_return(w, prev_p, cur_p, borrow_daily=borrow_daily))
+    return out
+```
+
+- [ ] **Step 4: Run.** `.venv/bin/python -m pytest tests/unit/test_equity_backtest_domain.py -k daily_returns -v` → PASS.
+
+- [ ] **Step 5: mypy.** `.venv/bin/python -m mypy src/fxhnt/domain/equity_backtest.py` → no errors.
+
+- [ ] **Step 6: Commit.**
+```bash
+git add src/fxhnt/domain/equity_backtest.py tests/unit/test_equity_backtest_domain.py
+git commit -m "feat(b3b): daily_returns_for_weights (fixed-weight holding-period book returns)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 4: Walk-forward runner (warehouse → per-construction daily return series)
+
+**Files:**
+- Create: `src/fxhnt/application/equity_backtest_runner.py`
+- Test: `tests/integration/test_equity_backtest_runner.py`
+
+This is the integration core. The runner reads the warehouse once, builds the per-construction daily-return series, and returns them plus diagnostics. Verdict + report come in Task 5.
+
+- [ ] **Step 1: Write the synthetic-warehouse fixture + failing test** in `tests/integration/test_equity_backtest_runner.py`:
+
+```python
+import datetime as dt
+
+import numpy as np
+
+from fxhnt.adapters.warehouse.duckdb_feature_store import DuckDbFeatureStore
+from fxhnt.application.equity_backtest_runner import EquityBacktestRunner
+
+_SPD = 86_400
+
+
+def _ts(iso: str) -> int:
+    d = dt.date.fromisoformat(iso)
+    return (d - dt.date(1970, 1, 1)).days * _SPD
+
+
+def _build_warehouse(path: str) -> DuckDbFeatureStore:
+    """3 names over ~2 years of daily bars. AAA strong uptrend, BBB mild,
+    CCC downtrend — gives a non-degenerate cross-section for long/ls/tilt."""
+    store = DuckDbFeatureStore(path)
+    start = dt.date(2022, 1, 3)
+    n = 520  # ~2 trading years
+    drifts = {"AAA": 0.0015, "BBB": 0.0003, "CCC": -0.0010}
+    items = []
+    members = []
+    for sym, dr in drifts.items():
+        rows = []
+        px = 100.0
+        for i in range(n):
+            day = start + dt.timedelta(days=i)
+            px *= (1.0 + dr)
+            ts = (day - dt.date(1970, 1, 1)).days * _SPD
+            rows.append((ts, {"adjclose": px, "close": px, "volume": 1_000_000.0}))
+        items.append((sym, rows))
+        members.append((sym, "NYSE", "2000-01-01", "2026-06-01"))
+    store.write_features_bulk(items)
+    store.upsert_membership(members)
+    return store
+
+
+def test_runner_produces_three_construction_series(tmp_path):
+    store = _build_warehouse(str(tmp_path / "wh.duckdb"))
+    runner = EquityBacktestRunner(store, n=10, cost_bps_per_turnover=15.0,
+                                  borrow_annual=0.0, momentum_min_history=252)
+    result = runner.run()
+    store.close()
+    assert set(result.returns_by_construction) == {"long", "ls", "tilt"}
+    # Each construction yields a non-trivial daily return series.
+    for series in result.returns_by_construction.values():
+        assert len(series) > 50
+        assert np.isfinite(series).all()
+    # The long book (holding the uptrending AAA) should be net positive over the window.
+    assert float(np.prod(1.0 + result.returns_by_construction["long"]) - 1.0) > 0.0
+
+
+def test_runner_is_deterministic(tmp_path):
+    store = _build_warehouse(str(tmp_path / "wh.duckdb"))
+    runner = EquityBacktestRunner(store, n=10, cost_bps_per_turnover=15.0,
+                                  borrow_annual=0.0, momentum_min_history=252)
+    a = runner.run().returns_by_construction["ls"]
+    b = runner.run().returns_by_construction["ls"]
+    store.close()
+    assert np.array_equal(a, b)
+```
+
+- [ ] **Step 2: Run, verify failure.** `.venv/bin/python -m pytest tests/integration/test_equity_backtest_runner.py -v` → FAIL (ImportError).
+
+- [ ] **Step 3: Implement** `src/fxhnt/application/equity_backtest_runner.py`:
+
+```python
+"""Walk-forward equity-factor backtest runner. Reads the dedicated DuckDB
+backtest warehouse (never live Tiingo), reconstructs the point-in-time universe
+each month, scores names with the SAME domain factor math as the live sleeves,
+holds weights between monthly rebalances, and produces a daily net-return series
+per construction."""
+from __future__ import annotations
+
+import datetime as dt
+from dataclasses import dataclass
+
+import numpy as np
+
+from fxhnt.adapters.warehouse.duckdb_feature_store import DuckDbFeatureStore
+from fxhnt.domain.equity_backtest import (
+    _epoch_day,
+    daily_returns_for_weights,
+    month_end_rebalance_dates,
+    universe_asof,
+)
+from fxhnt.domain.strategies.equity_factor import (
+    composite_price,
+    construction_weights,
+    lowvol_score,
+    momentum_12_1_from_closes,
+    momentum_score,
+    realized_vol_from_closes,
+)
+
+_CONSTRUCTIONS = ("long", "ls", "tilt")
+_DOLLAR_VOL_WINDOW_DAYS = 63   # ~3 trading months trailing, for liquidity ranking
+_TRADING_DAYS_YEAR = 252
+
+
+@dataclass(frozen=True)
+class BacktestRunResult:
+    returns_by_construction: dict[str, np.ndarray]   # construction -> daily net returns
+    rebalance_dates: list[str]
+    n_names_avg: float                               # mean PIT universe size per rebalance
+
+
+def _iso(epoch_day: int) -> str:
+    return (dt.date(1970, 1, 1) + dt.timedelta(days=epoch_day)).isoformat()
+
+
+class EquityBacktestRunner:
+    def __init__(
+        self,
+        store: DuckDbFeatureStore,
+        *,
+        n: int = 150,
+        cost_bps_per_turnover: float = 15.0,
+        borrow_annual: float = 0.0,
+        momentum_min_history: int = 252,
+    ) -> None:
+        self._store = store
+        self._n = n
+        self._cost = cost_bps_per_turnover / 1e4
+        self._borrow_daily = borrow_annual / _TRADING_DAYS_YEAR
+        self._mom_min = momentum_min_history
+
+    def run(self) -> BacktestRunResult:
+        members = self._store.read_membership()
+        syms = [m[0] for m in members]
+        adj = self._store.read_panel(syms, "adjclose")     # {sym: {epoch_day: px}}
+        close = self._store.read_panel(syms, "close")
+        vol = self._store.read_panel(syms, "volume")
+
+        all_days = sorted({d for s in adj.values() for d in s})
+        all_iso = [_iso(d) for d in all_days]
+        rebals = month_end_rebalance_dates(all_iso)
+
+        series: dict[str, list[float]] = {c: [] for c in _CONSTRUCTIONS}
+        prev_weights: dict[str, dict[str, float]] = {c: {} for c in _CONSTRUCTIONS}
+        names_seen = 0
+        n_rebals = 0
+
+        for k, rebal in enumerate(rebals):
+            rebal_day = _epoch_day(rebal)
+            # liquidity: trailing dollar-volume (close*volume) over the window ending rebal
+            lo = rebal_day - _DOLLAR_VOL_WINDOW_DAYS
+            dollar_vol: dict[str, float] = {}
+            for s in syms:
+                cl, vl = close.get(s, {}), vol.get(s, {})
+                acc = [cl[d] * vl[d] for d in cl if lo <= d <= rebal_day and d in vl]
+                if acc:
+                    dollar_vol[s] = float(np.mean(acc))
+            uni = universe_asof(members, dollar_vol, rebal, self._n)
+
+            # PIT scores: momentum + low-vol from closes up to and including rebal_day
+            mom: list[float | None] = []
+            vols: list[float | None] = []
+            kept: list[str] = []
+            prices_now: dict[str, float] = {}
+            for s in uni:
+                hist = [adj[s][d] for d in sorted(adj[s]) if d <= rebal_day]
+                m = momentum_12_1_from_closes(hist)
+                v = realized_vol_from_closes(hist)
+                if m is None or v is None or rebal_day not in adj[s]:
+                    continue
+                kept.append(s)
+                mom.append(m)
+                vols.append(v)
+                prices_now[s] = adj[s][rebal_day]
+            names_seen += len(kept)
+            n_rebals += 1
+            if len(kept) < 2:
+                continue
+
+            scores = composite_price(momentum_score(mom), lowvol_score(vols))
+
+            # holding-period trading days: (this rebal, next rebal]
+            end_day = _epoch_day(rebals[k + 1]) if k + 1 < len(rebals) else all_days[-1]
+            hold_days = [d for d in all_days if rebal_day <= d <= end_day]
+
+            for c in _CONSTRUCTIONS:
+                w_list = construction_weights(scores, c)
+                weights = {kept[i]: w_list[i] for i in range(len(kept)) if w_list[i] != 0.0}
+                # turnover cost at rebalance (vs prior book), charged on the first held day
+                turnover = _turnover(prev_weights[c], weights)
+                rets = daily_returns_for_weights(weights, adj, hold_days, self._borrow_daily)
+                if rets:
+                    rets[0] -= turnover * self._cost
+                series[c].extend(rets)
+                prev_weights[c] = weights
+
+        return BacktestRunResult(
+            returns_by_construction={c: np.asarray(series[c], dtype=float) for c in _CONSTRUCTIONS},
+            rebalance_dates=rebals,
+            n_names_avg=(names_seen / n_rebals) if n_rebals else 0.0,
+        )
+
+
+def _turnover(prev: dict[str, float], cur: dict[str, float]) -> float:
+    """Sum of absolute weight changes across the union of names (one-way turnover)."""
+    names = set(prev) | set(cur)
+    return float(sum(abs(cur.get(s, 0.0) - prev.get(s, 0.0)) for s in names))
+```
+
+- [ ] **Step 4: Run.** `.venv/bin/python -m pytest tests/integration/test_equity_backtest_runner.py -v` → PASS. If the "long > 0" assertion is flaky on the synthetic data, inspect the realized series; do NOT weaken the determinism test.
+
+- [ ] **Step 5: mypy.** `.venv/bin/python -m mypy src/fxhnt/application/equity_backtest_runner.py` → no errors.
+
+- [ ] **Step 6: Commit.**
+```bash
+git add src/fxhnt/application/equity_backtest_runner.py tests/integration/test_equity_backtest_runner.py
+git commit -m "feat(b3b): walk-forward runner — PIT universe + monthly rebalance + per-construction daily returns
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 5: Verdict + JSON report
+
+**Files:**
+- Modify: `src/fxhnt/application/equity_backtest_runner.py` (add `evaluate_constructions`)
+- Test: `tests/integration/test_equity_backtest_runner.py` (append)
+
+- [ ] **Step 1: Append failing test:**
+
+```python
+from fxhnt.application.equity_backtest_runner import evaluate_constructions
+
+
+def test_evaluate_constructions_emits_stats_and_verdict_per_construction(tmp_path):
+    store = _build_warehouse(str(tmp_path / "wh.duckdb"))
+    runner = EquityBacktestRunner(store, n=10, cost_bps_per_turnover=15.0,
+                                  borrow_annual=0.0, momentum_min_history=252)
+    result = runner.run()
+    store.close()
+    report = evaluate_constructions(result, oos_fraction=0.40, dsr_min=0.95,
+                                    oos_min_sharpe=0.0, max_is_oos_decay=0.50)
+    assert set(report["constructions"]) == {"long", "ls", "tilt"}
+    for c, block in report["constructions"].items():
+        assert {"stats", "verdict"} <= set(block)
+        assert {"sharpe", "cagr", "max_drawdown", "n_obs"} <= set(block["stats"])
+        assert {"passed", "dsr", "is_sharpe", "oos_sharpe", "n_trials", "reasons"} <= set(block["verdict"])
+        assert block["verdict"]["n_trials"] == 3        # the three constructions
+    assert report["n_names_avg"] >= 0.0
+    assert isinstance(report["rebalance_dates"], list)
+```
+
+- [ ] **Step 2: Run, verify failure.** `.venv/bin/python -m pytest tests/integration/test_equity_backtest_runner.py -k evaluate_constructions -v` → FAIL.
+
+- [ ] **Step 3: Implement** (append to `src/fxhnt/application/equity_backtest_runner.py`):
+
+```python
+from typing import Any
+
+from fxhnt.domain.backtest import compute_stats
+from fxhnt.domain.gauntlet.core import evaluate
+
+
+def evaluate_constructions(
+    result: BacktestRunResult,
+    *,
+    oos_fraction: float = 0.40,
+    dsr_min: float = 0.95,
+    oos_min_sharpe: float = 0.0,
+    max_is_oos_decay: float = 0.50,
+) -> dict[str, Any]:
+    """Stack each construction's daily returns, split IS/OOS, run the gauntlet.
+    n_trials = the number of constructions tested (multiple-testing correction)."""
+    n_trials = len(result.returns_by_construction)
+    out: dict[str, Any] = {
+        "constructions": {},
+        "rebalance_dates": result.rebalance_dates,
+        "n_names_avg": result.n_names_avg,
+        "settings": {
+            "oos_fraction": oos_fraction, "dsr_min": dsr_min,
+            "oos_min_sharpe": oos_min_sharpe, "max_is_oos_decay": max_is_oos_decay,
+            "n_trials": n_trials,
+        },
+    }
+    for c, r in result.returns_by_construction.items():
+        stats = compute_stats(r)
+        split = int((1.0 - oos_fraction) * len(r))
+        is_r, oos_r = r[:split], r[split:]
+        # sr_variance ~ Var of the per-trial Sharpe estimates; with one realized series
+        # per construction we use the standard small-sample proxy 1/n_obs (0 if too short).
+        sr_variance = (1.0 / len(r)) if len(r) > 1 else 0.0
+        verdict = evaluate(
+            is_r, oos_r, n_trials=n_trials, sr_variance=sr_variance,
+            dsr_min=dsr_min, oos_min_sharpe=oos_min_sharpe,
+            max_is_oos_decay=max_is_oos_decay,
+            has_economic_rationale=True,  # momentum + low-vol are documented risk premia
+        )
+        out["constructions"][c] = {
+            "stats": stats.model_dump(),
+            "verdict": verdict.model_dump(),
+        }
+    return out
+```
+
+- [ ] **Step 4: Run.** `.venv/bin/python -m pytest tests/integration/test_equity_backtest_runner.py -v` → all PASS.
+
+- [ ] **Step 5: mypy.** `.venv/bin/python -m mypy src/fxhnt/application/equity_backtest_runner.py` → no errors.
+
+- [ ] **Step 6: Commit.**
+```bash
+git add src/fxhnt/application/equity_backtest_runner.py tests/integration/test_equity_backtest_runner.py
+git commit -m "feat(b3b): gauntlet verdict + JSON report per construction (DSR/OOS, n_trials=3)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 6: CLI command `backtest-equity`
+
+**Files:**
+- Modify: `src/fxhnt/cli.py`
+- Test: `tests/integration/test_equity_backtest_runner.py` (append a CLI smoke test) OR `tests/unit/test_cli.py` if one exists — follow the pattern used by the existing `ingest-historical` command test.
+
+- [ ] **Step 1: Read** `src/fxhnt/cli.py` to match the Typer style of the existing `ingest-historical` command (how it builds `Settings`, `DuckDbFeatureStore(settings.backtest_warehouse_path)`, and echoes a summary line).
+
+- [ ] **Step 2: Write a failing CLI test** mirroring the existing CLI-test pattern. Build a synthetic warehouse at a tmp path, point `FXHNT_BACKTEST_WAREHOUSE_PATH` at it (monkeypatch env or pass `--warehouse`), invoke the command via Typer's `CliRunner`, assert exit code 0 and that a report JSON file is written with the three constructions. Example (adapt to the house CLI-test idiom found in Step 1):
+
+```python
+import json
+from typer.testing import CliRunner
+from fxhnt.cli import app
+
+def test_cli_backtest_equity_writes_report(tmp_path, monkeypatch):
+    wh = str(tmp_path / "wh.duckdb")
+    _build_warehouse(wh).close()
+    out = tmp_path / "report.json"
+    monkeypatch.setenv("FXHNT_BACKTEST_WAREHOUSE_PATH", wh)
+    res = CliRunner().invoke(app, ["backtest-equity", "--n", "10", "--out", str(out)])
+    assert res.exit_code == 0, res.output
+    report = json.loads(out.read_text())
+    assert set(report["constructions"]) == {"long", "ls", "tilt"}
+```
+
+- [ ] **Step 3: Run, verify failure.** `.venv/bin/python -m pytest -k backtest_equity -v` → FAIL.
+
+- [ ] **Step 4: Implement** the command in `src/fxhnt/cli.py`:
+
+```python
+@app.command("backtest-equity")
+def backtest_equity(
+    n: int = typer.Option(150, help="PIT universe size (top-N by trailing dollar-volume)."),
+    cost_bps: float = typer.Option(15.0, help="Cost in bps per unit turnover."),
+    borrow_annual: float = typer.Option(0.0, help="Annual borrow cost on short notional."),
+    out: str = typer.Option("/backtest-data/equity_backtest_report.json", help="Report JSON path."),
+) -> None:
+    """Walk-forward backtest of the 3 price-only equity-factor constructions against
+    the dedicated survivorship-free backtest warehouse → gauntlet verdict + JSON report."""
+    import json
+    from fxhnt.config import Settings
+    from fxhnt.adapters.warehouse.duckdb_feature_store import DuckDbFeatureStore
+    from fxhnt.application.equity_backtest_runner import (
+        EquityBacktestRunner, evaluate_constructions,
+    )
+
+    s = Settings()
+    store = DuckDbFeatureStore(s.backtest_warehouse_path)
+    try:
+        result = EquityBacktestRunner(
+            store, n=n, cost_bps_per_turnover=cost_bps, borrow_annual=borrow_annual,
+        ).run()
+        report = evaluate_constructions(
+            result,
+            oos_fraction=s.gauntlet.oos_fraction, dsr_min=s.gauntlet.dsr_min,
+            oos_min_sharpe=s.gauntlet.oos_min_sharpe, max_is_oos_decay=s.gauntlet.max_is_oos_decay,
+        )
+    finally:
+        store.close()
+    with open(out, "w") as fh:
+        json.dump(report, fh, indent=2)
+    for c, block in report["constructions"].items():
+        v = block["verdict"]
+        typer.echo(f"backtest-equity {c}: passed={v['passed']} dsr={v['dsr']:.3f} "
+                   f"is_sharpe={v['is_sharpe']:.2f} oos_sharpe={v['oos_sharpe']:.2f} "
+                   f"sharpe={block['stats']['sharpe']:.2f} -> {out}")
+```
+
+If the CLI test passes `--out` as the report path but the command reads the warehouse from env, ensure the command honours `FXHNT_BACKTEST_WAREHOUSE_PATH` (it does, via `Settings()`); if the house pattern threads `--warehouse` instead, add that option to match `ingest-historical`.
+
+- [ ] **Step 5: Run.** `.venv/bin/python -m pytest -k backtest_equity -v` → PASS, then full `.venv/bin/python -m pytest -q` → all PASS.
+
+- [ ] **Step 6: mypy.** `.venv/bin/python -m mypy src/fxhnt/cli.py` → no new errors.
+
+- [ ] **Step 7: Commit.**
+```bash
+git add src/fxhnt/cli.py tests/
+git commit -m "feat(b3b): fxhnt backtest-equity CLI — run walk-forward verdict + write report JSON
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Known caveat — real-data memory (flag, do not fix now)
+The Task-4 runner calls `read_panel` over **all** member symbols up front (3 features). That is fine for the synthetic tests and for a few-thousand-name universe, but the full survivorship-free set (~22k names × ~30 yr) would not fit in memory as Python dicts. Before the *real* run (after the ingest completes), add a pre-filter: read `catalog()` first, keep only names with enough history and non-trivial dollar-volume, and/or chunk the panel read by rebalance window. Build + test now on synthetic data; this optimization is a deliberate follow-up, logged here so it is not a silent OOM later (cf. the B3a ingest OOM).
+
+## Out of scope (do NOT build)
+- Fundamentals factors (value/quality) — gated on the Tiingo add-on; kept in the domain unused.
+- Cockpit `backtest_summary` wiring — optional follow-up after we see a real verdict.
+- Nightly automation — the backtest is run on demand.
+- Live execution.
+
+## Final review (after all tasks)
+Dispatch a final code reviewer over the whole B3b diff, then run the full suite + mypy once more. Do NOT push; report the commit range and a summary so we can run the real backtest against the warehouse once the ingest completes.

docs/superpowers/plans/2026-06-18-cross-sectional-funding.md

@@ -0,0 +1,694 @@
+# Cross-Sectional Crypto Funding-Dispersion — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Backtest (survivorship-free, PIT) a cross-sectional crypto funding-dispersion strategy across a matrix of constructions × liquidity floors, through the gauntlet, to find a real uncrowded small-capital edge — then surface the verdict.
+
+**Architecture:** Pure domain (PIT eligibility + cross-sectional funding score + 3 construction modes + signed delta-neutral carry book-return) consumed by a walk-forward runner that reads the existing survivorship-free `crypto_pit` npz panel, producing a per-(mode × floor) daily return series, each scored by the existing gauntlet (DSR/OOS). A CLI runs it and prints the verdict matrix.
+
+**Tech Stack:** Python 3.12, numpy, the `crypto_pit` npz panel (foxhunt `data/surfer/crypto_pit/`, 180 symbols incl. 25 delisted, 2019–2026), the gauntlet (`compute_stats`/`evaluate`), Typer CLI, pytest (`.venv/bin/python -m pytest`), strict mypy.
+
+**Spec:** `docs/superpowers/specs/2026-06-18-cross-sectional-funding-design.md`.
+
+**House rules:** TDD (failing test first). Pure functions in `domain/`. Synthetic in-memory panels in tests — NO live network, `.npz` loaded WITHOUT `allow_pickle`. Strict mypy (no NEW errors). Commit per task, do NOT push. Trailer:
+```
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
+```
+**Crypto convention:** daily series, **365 periods/year** (pass `periods_per_year=365` to `compute_stats`).
+
+**Scope:** data adapter → domain → runner → gauntlet verdict matrix → CLI. Paper-forward wiring of the *winning* construction is a deliberate follow-on AFTER we see the verdict (don't wire a possibly-falsified strategy into the cockpit) — NOT in this plan.
+
+---
+
+## Reuse contract (exact — verified this session)
+```python
+# src/fxhnt/domain/strategies/equity_factor.py
+def robust_z(values: list[float | None], k: float = 3.0) -> list[float]   # median/MAD winsorized z, missing→0.0
+
+# src/fxhnt/domain/backtest.py
+class BacktestStats(BaseModel): cagr: float; ann_vol: float; sharpe: float; max_drawdown: float; n_obs: int
+def compute_stats(returns: np.ndarray, periods_per_year: int = 252) -> BacktestStats
+
+# src/fxhnt/domain/gauntlet/core.py
+def evaluate(is_returns, oos_returns, n_trials, sr_variance, *, dsr_min=0.95, oos_min_sharpe=0.0,
+             max_is_oos_decay=0.50, has_economic_rationale=None) -> Verdict
+def per_period_sharpe(returns: np.ndarray) -> float    # per discovery.py; if the real name differs, use what discovery.py uses
+class Verdict(BaseModel): passed; dsr; is_sharpe; oos_sharpe; n_trials; reasons; pvalue
+
+# src/fxhnt/config.py — Settings.gauntlet has oos_fraction/dsr_min/oos_min_sharpe/max_is_oos_decay
+```
+**npz panel format** (verified): each `data/surfer/crypto_pit/<SYMBOL>.npz` has parallel arrays `day` (int epoch-day), `close`, `qvol` (daily USD quote-volume), `funding` (daily-summed 8h funding rate). Symbol = filename stem.
+
+---
+
+## File Structure
+- Create `src/fxhnt/adapters/data/crypto_pit_panel.py` — npz loader → in-memory panel.
+- Create `src/fxhnt/domain/cross_sectional_funding.py` — pure: `eligible_asof`, `funding_score`, `construction_weights`, `book_return_xs`, `_epoch_day`.
+- Create `src/fxhnt/application/funding_backtest_runner.py` — walk-forward runner + `evaluate_funding_constructions`.
+- Modify `src/fxhnt/config.py` — add `crypto_pit_dir`.
+- Modify `src/fxhnt/cli.py` — add `backtest-funding`.
+- Tests: `tests/unit/test_cross_sectional_funding.py`, `tests/integration/test_crypto_pit_panel.py`, `tests/integration/test_funding_backtest_runner.py`, `tests/integration/test_backtest_funding_cli.py`.
+
+---
+
+## Task 1: crypto_pit npz panel adapter
+
+**Files:** Create `src/fxhnt/adapters/data/crypto_pit_panel.py`; Modify `src/fxhnt/config.py`; Test `tests/integration/test_crypto_pit_panel.py`.
+
+- [ ] **Step 1: config** — add to `Settings` in `src/fxhnt/config.py` (mirror the existing `backtest_warehouse_path` field):
+```python
+    crypto_pit_dir: str = Field(default=str(_DATA_DIR / "crypto_pit"))   # FXHNT_CRYPTO_PIT_DIR; survivorship-free npz panel
+```
+(use the same `_DATA_DIR`/`Field` idiom already in the file.)
+
+- [ ] **Step 2: failing test** `tests/integration/test_crypto_pit_panel.py`:
+```python
+import numpy as np
+from fxhnt.adapters.data.crypto_pit_panel import CryptoPitPanelSource
+
+
+def _write(tmp_path, sym, days, close, qvol, funding):
+    np.savez(tmp_path / f"{sym}.npz",
+             day=np.array(days, dtype=np.int64), close=np.array(close, dtype=float),
+             qvol=np.array(qvol, dtype=float), funding=np.array(funding, dtype=float))
+
+
+def test_loads_panel_from_npz(tmp_path):
+    _write(tmp_path, "AAA", [100, 101], [1.0, 1.1], [2e6, 2e6], [0.001, 0.0012])
+    _write(tmp_path, "DEAD", [100], [5.0], [1e6], [0.003])
+    src = CryptoPitPanelSource(str(tmp_path))
+    panel = src.panel()
+    assert set(panel) == {"AAA", "DEAD"}
+    assert panel["AAA"][100] == (1.0, 2e6, 0.001)        # (close, qvol, funding)
+    assert panel["DEAD"][100] == (5.0, 1e6, 0.003)
+    assert sorted(src.all_days()) == [100, 101]           # union of all symbols' days
+
+
+def test_panel_ignores_non_npz_and_is_deterministic(tmp_path):
+    _write(tmp_path, "BBB", [100], [1.0], [3e6], [0.0])
+    (tmp_path / "notes.txt").write_text("ignore me")
+    a = CryptoPitPanelSource(str(tmp_path)).panel()
+    b = CryptoPitPanelSource(str(tmp_path)).panel()
+    assert a == b and set(a) == {"BBB"}
+```
+
+- [ ] **Step 3: run → FAIL** `.venv/bin/python -m pytest tests/integration/test_crypto_pit_panel.py -v`
+
+- [ ] **Step 4: implement** `src/fxhnt/adapters/data/crypto_pit_panel.py`:
+```python
+"""Loader for the survivorship-free `crypto_pit` npz panel (Binance USDT-perp daily
+close/quote-volume/funding incl. delisted coins). Read-only; no live network."""
+from __future__ import annotations
+
+import glob
+import os
+
+import numpy as np
+
+
+class CryptoPitPanelSource:
+    def __init__(self, pit_dir: str) -> None:
+        self._dir = pit_dir
+
+    def panel(self) -> dict[str, dict[int, tuple[float, float, float]]]:
+        """{symbol: {epoch_day: (close, qvol, funding)}}. Deterministic (sorted files)."""
+        out: dict[str, dict[int, tuple[float, float, float]]] = {}
+        for path in sorted(glob.glob(os.path.join(self._dir, "*.npz"))):
+            sym = os.path.splitext(os.path.basename(path))[0]
+            with np.load(path) as z:   # numeric arrays only; no allow_pickle
+                days = z["day"].astype(np.int64)
+                close = z["close"].astype(float)
+                qvol = z["qvol"].astype(float)
+                funding = z["funding"].astype(float)
+            out[sym] = {int(d): (float(c), float(q), float(f))
+                        for d, c, q, f in zip(days, close, qvol, funding)}
+        return out
+
+    def all_days(self) -> list[int]:
+        return sorted({d for s in self.panel().values() for d in s})
+```
+
+- [ ] **Step 5: run → PASS**, then `.venv/bin/python -m mypy src/fxhnt/adapters/data/crypto_pit_panel.py src/fxhnt/config.py` → no new errors.
+
+- [ ] **Step 6: commit**
+```bash
+git add src/fxhnt/adapters/data/crypto_pit_panel.py src/fxhnt/config.py tests/integration/test_crypto_pit_panel.py
+git commit -m "feat(xsfunding): crypto_pit npz panel adapter (survivorship-free, read-only)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 2: domain — PIT eligibility + cross-sectional funding score
+
+**Files:** Create `src/fxhnt/domain/cross_sectional_funding.py`; Test `tests/unit/test_cross_sectional_funding.py`.
+
+- [ ] **Step 1: failing tests** `tests/unit/test_cross_sectional_funding.py`:
+```python
+from fxhnt.domain.cross_sectional_funding import eligible_asof, funding_score, _epoch_day
+
+# panel: {symbol: {epoch_day: (close, qvol, funding)}}
+def _coin(start, n, qvol, funding, close=1.0):
+    return {start + i: (close, qvol, funding) for i in range(n)}
+
+
+def test_eligible_filters_by_liquidity_and_history_pit():
+    panel = {
+        "LIQ":   _coin(100, 40, 5e6, 0.001),   # liquid, long history
+        "THIN":  _coin(100, 40, 1e5, 0.001),   # too illiquid
+        "NEW":   _coin(135, 40, 5e6, 0.001),   # not enough history before asof
+    }
+    elig = eligible_asof(panel, asof_day=139, min_qvol=1e6, min_history=20, vol_window=30)
+    assert elig == ["LIQ"]                       # THIN below floor, NEW lacks 20d history by 139
+
+
+def test_funding_score_ranks_high_funding_higher():
+    # three liquid coins, different trailing funding; score = robust_z of trailing-mean funding
+    panel = {
+        "HI":  _coin(100, 30, 5e6, 0.003),
+        "MID": _coin(100, 30, 5e6, 0.001),
+        "LO":  _coin(100, 30, 5e6, -0.001),
+    }
+    elig = ["HI", "MID", "LO"]
+    sc = funding_score(panel, elig, asof_day=129, lookback_days=7)
+    assert sc["HI"] > sc["MID"] > sc["LO"]
+
+
+def test_funding_score_is_pit_no_lookahead():
+    # LATE: funding low early, high late. At an EARLY asof it must NOT rank above a steady HI.
+    panel = {
+        "HI":   {100 + i: (1.0, 5e6, 0.002) for i in range(60)},
+        "LATE": {100 + i: (1.0, 5e6, (0.0001 if i < 30 else 0.01)) for i in range(60)},
+    }
+    early = funding_score(panel, ["HI", "LATE"], asof_day=120, lookback_days=7)   # day120: LATE still in low regime
+    assert early["HI"] > early["LATE"]
+    late = funding_score(panel, ["HI", "LATE"], asof_day=159, lookback_days=7)    # day159: LATE now high
+    assert late["LATE"] > late["HI"]
+
+
+def test_epoch_day_roundtrip():
+    assert _epoch_day("1970-01-02") == 1
+```
+
+- [ ] **Step 2: run → FAIL**
+
+- [ ] **Step 3: implement** `src/fxhnt/domain/cross_sectional_funding.py` (part 1):
+```python
+"""Pure cross-sectional crypto funding-dispersion domain: PIT eligibility, the
+cross-sectional funding score, construction weights, and signed delta-neutral
+carry book-return. No I/O. Panel = {symbol: {epoch_day: (close, qvol, funding)}}."""
+from __future__ import annotations
+
+import datetime as dt
+
+from fxhnt.domain.strategies.equity_factor import robust_z
+
+Panel = dict[str, dict[int, tuple[float, float, float]]]
+
+
+def _epoch_day(iso_date: str) -> int:
+    return (dt.date.fromisoformat(iso_date) - dt.date(1970, 1, 1)).days
+
+
+def eligible_asof(panel: Panel, asof_day: int, *, min_qvol: float,
+                  min_history: int, vol_window: int = 30) -> list[str]:
+    """Coins tradeable at `asof_day` (PIT): have >= `min_history` bars on/before asof,
+    AND trailing-`vol_window` mean qvol >= `min_qvol`. Uses ONLY data <= asof_day."""
+    out: list[str] = []
+    lo = asof_day - vol_window
+    for sym, series in panel.items():
+        hist = [d for d in series if d <= asof_day]
+        if len(hist) < min_history:
+            continue
+        vols = [series[d][1] for d in hist if d > lo]
+        if vols and (sum(vols) / len(vols)) >= min_qvol:
+            out.append(sym)
+    out.sort()
+    return out
+
+
+def funding_score(panel: Panel, eligible: list[str], asof_day: int,
+                  lookback_days: int = 7) -> dict[str, float]:
+    """Cross-sectional carry score: robust_z (across `eligible`) of each coin's
+    trailing-`lookback_days` mean funding (<= asof_day). Short window keeps it
+    responsive (avoids stale-level bias). Returns {symbol: score} aligned to `eligible`."""
+    lo = asof_day - lookback_days
+    tf: list[float | None] = []
+    for sym in eligible:
+        series = panel[sym]
+        vals = [series[d][2] for d in series if lo < d <= asof_day]
+        tf.append(sum(vals) / len(vals) if vals else None)
+    z = robust_z(tf)
+    return {eligible[i]: z[i] for i in range(len(eligible))}
+```
+
+- [ ] **Step 4: run → PASS** `.venv/bin/python -m pytest tests/unit/test_cross_sectional_funding.py -v`
+
+- [ ] **Step 5: mypy** on the new module → no errors.
+
+- [ ] **Step 6: commit**
+```bash
+git add src/fxhnt/domain/cross_sectional_funding.py tests/unit/test_cross_sectional_funding.py
+git commit -m "feat(xsfunding): PIT eligibility + cross-sectional funding score (robust_z of trailing funding)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 3: domain — construction weights (3 modes) + signed carry book-return
+
+**Files:** Modify `src/fxhnt/domain/cross_sectional_funding.py`; Test `tests/unit/test_cross_sectional_funding.py` (append).
+
+- [ ] **Step 1: append failing tests:**
+```python
+from fxhnt.domain.cross_sectional_funding import construction_weights, book_return_xs
+
+# scores aligned to a symbol list
+SC = {"A": 2.0, "B": 1.0, "C": 0.0, "D": -1.0, "E": -2.0}   # A richest carry, E poorest
+
+
+def test_long_tilt_longs_top_quantile_sums_to_one():
+    w = construction_weights(SC, "long_tilt", quantile=0.4)   # top 40% of 5 = top 2 = A,B
+    assert set(k for k, v in w.items() if v != 0) == {"A", "B"}
+    assert all(v > 0 for v in w.values() if v != 0)
+    assert abs(sum(w.values()) - 1.0) < 1e-9
+
+
+def test_market_neutral_longs_top_shorts_bottom_sums_to_zero():
+    w = construction_weights(SC, "market_neutral", quantile=0.4)   # long A,B ; short D,E
+    assert w["A"] > 0 and w["B"] > 0 and w["D"] < 0 and w["E"] < 0
+    assert abs(sum(w.values())) < 1e-9          # dollar-neutral
+    assert abs(sum(abs(v) for v in w.values()) - 2.0) < 1e-9   # gross 2.0
+
+
+def test_executable_shorts_only_borrowable_names():
+    # only A,B,C are "borrowable"; market_neutral would short D,E but executable can't
+    w = construction_weights(SC, "executable", quantile=0.4, borrowable={"A", "B", "C"})
+    assert w["A"] > 0 and w["B"] > 0           # long side intact
+    assert w.get("D", 0.0) == 0.0 and w.get("E", 0.0) == 0.0   # un-borrowable shorts dropped
+
+
+def test_book_return_signed_carry_minus_cost():
+    # long A (collect +funding), short E (collect -funding => profit when E funding negative)
+    prev = {"A": 0.5, "E": -0.5}
+    funding_today = {"A": 0.002, "E": -0.002}
+    new = {"A": 0.5, "E": -0.5}                # no turnover
+    cost = {"A": 0.001, "E": 0.001}
+    r = book_return_xs(prev, funding_today, new, cost)
+    # realized = 0.5*0.002 + (-0.5)*(-0.002) = 0.001 + 0.001 = 0.002 ; no turnover cost
+    assert abs(r - 0.002) < 1e-12
+
+
+def test_book_return_charges_turnover():
+    prev = {"A": 1.0}
+    new = {"B": 1.0}                            # full switch: turnover |0-1|_A + |1-0|_B = 2.0
+    cost = {"A": 0.001, "B": 0.001}
+    r = book_return_xs(prev, {"A": 0.0}, new, cost)
+    # realized 0 ; cost = (|1|*0.001 + |1|*0.001)/2 = 0.001
+    assert abs(r - (-0.001)) < 1e-12
+```
+
+- [ ] **Step 2: run → FAIL**
+
+- [ ] **Step 3: implement** (append to `src/fxhnt/domain/cross_sectional_funding.py`):
+```python
+def _quantile_cut(scores: dict[str, float], quantile: float) -> tuple[list[str], list[str]]:
+    """Return (top, bottom) symbol lists at the given quantile (by score desc)."""
+    ranked = sorted(scores, key=lambda s: (-scores[s], s))
+    k = max(1, int(quantile * len(ranked)))
+    return ranked[:k], ranked[-k:]
+
+
+def construction_weights(scores: dict[str, float], mode: str, *, quantile: float = 0.2,
+                         borrowable: set[str] | None = None) -> dict[str, float]:
+    """Map cross-sectional funding scores -> delta-neutral weights.
+    long_tilt: equal-weight long the top quantile, sum=1.
+    market_neutral: long top + short bottom, dollar-neutral (sum=0, gross=2).
+    executable: long top (sum=1) + short bottom ONLY for names in `borrowable`."""
+    top, bot = _quantile_cut(scores, quantile)
+    w: dict[str, float] = {}
+    if mode == "long_tilt":
+        for s in top:
+            w[s] = 1.0 / len(top)
+    elif mode == "market_neutral":
+        for s in top:
+            w[s] = 0.5 / len(top)
+        for s in bot:
+            w[s] = w.get(s, 0.0) - 0.5 / len(bot)
+    elif mode == "executable":
+        for s in top:
+            w[s] = 1.0 / len(top)
+        shorts = [s for s in bot if borrowable is None or s in borrowable]
+        for s in shorts:
+            w[s] = w.get(s, 0.0) - 1.0 / len(shorts) if shorts else w.get(s, 0.0)
+    else:
+        raise ValueError(f"unknown construction mode: {mode}")
+    return {s: v for s, v in w.items() if v != 0.0}
+
+
+def book_return_xs(prev_weights: dict[str, float], funding_today: dict[str, float],
+                   new_weights: dict[str, float], coin_cost_rt: dict[str, float]) -> float:
+    """Signed delta-neutral carry on the PRIOR book at today's funding, minus round-trip
+    cost on turnover. realized = sum(w_i * funding_i); a short (w<0) on a negative-funding
+    coin earns w*funding>0. cost = sum(|Δw_i| * cost_i)/2 over the union."""
+    realized = sum(w * funding_today.get(s, 0.0) for s, w in prev_weights.items())
+    cost = sum(abs(new_weights.get(s, 0.0) - prev_weights.get(s, 0.0)) * coin_cost_rt.get(s, 0.0)
+               for s in set(new_weights) | set(prev_weights)) / 2.0
+    return realized - cost
+```
+
+- [ ] **Step 4: run → PASS**
+
+- [ ] **Step 5: mypy** → no errors.
+
+- [ ] **Step 6: commit**
+```bash
+git add src/fxhnt/domain/cross_sectional_funding.py tests/unit/test_cross_sectional_funding.py
+git commit -m "feat(xsfunding): construction weights (long_tilt/market_neutral/executable) + signed carry book-return
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 4: walk-forward backtest runner
+
+**Files:** Create `src/fxhnt/application/funding_backtest_runner.py`; Test `tests/integration/test_funding_backtest_runner.py`.
+
+- [ ] **Step 1: failing test** (synthetic panel where high-funding coins persist → long_tilt should be net positive):
+```python
+import numpy as np
+from fxhnt.application.funding_backtest_runner import FundingBacktestRunner
+
+
+def _panel():
+    # 4 coins, 200 days; HI/MID positive funding, LO/NEG low/negative; all liquid
+    panel = {}
+    fund = {"HI": 0.0015, "MID": 0.0008, "LO": 0.0001, "NEG": -0.0010}
+    for sym, f in fund.items():
+        panel[sym] = {1000 + i: (1.0, 5e6, f) for i in range(200)}
+    return panel
+
+
+def test_runner_produces_series_per_mode():
+    runner = FundingBacktestRunner(_panel(), min_qvol=1e6, min_history=20,
+                                   lookback_days=7, quantile=0.5, cost_bps=8.0, slip_coef=0.0)
+    res = runner.run()
+    assert set(res.returns_by_mode) == {"long_tilt", "market_neutral", "executable"}
+    for series in res.returns_by_mode.values():
+        assert len(series) > 50 and np.isfinite(series).all()
+    # long_tilt holds the high-funding coins -> positive cumulative carry
+    assert float(np.sum(res.returns_by_mode["long_tilt"])) > 0.0
+
+
+def test_runner_deterministic():
+    r1 = FundingBacktestRunner(_panel(), min_qvol=1e6, min_history=20, lookback_days=7,
+                               quantile=0.5, cost_bps=8.0, slip_coef=0.0).run().returns_by_mode["long_tilt"]
+    r2 = FundingBacktestRunner(_panel(), min_qvol=1e6, min_history=20, lookback_days=7,
+                               quantile=0.5, cost_bps=8.0, slip_coef=0.0).run().returns_by_mode["long_tilt"]
+    assert np.array_equal(r1, r2)
+```
+
+- [ ] **Step 2: run → FAIL**
+
+- [ ] **Step 3: implement** `src/fxhnt/application/funding_backtest_runner.py`:
+```python
+"""Walk-forward cross-sectional funding-dispersion backtest over the crypto_pit panel.
+Daily rebalance, PIT universe, signed delta-neutral carry minus a liquidity-scaled cost.
+Produces a daily net-return series per construction mode."""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import numpy as np
+
+from fxhnt.domain.cross_sectional_funding import (
+    Panel, book_return_xs, construction_weights, eligible_asof, funding_score,
+)
+
+_MODES = ("long_tilt", "market_neutral", "executable")
+
+
+@dataclass(frozen=True)
+class FundingRunResult:
+    returns_by_mode: dict[str, np.ndarray]
+    n_names_avg: float
+    trading_days: int
+
+
+class FundingBacktestRunner:
+    def __init__(self, panel: Panel, *, min_qvol: float = 1e6, min_history: int = 30,
+                 lookback_days: int = 7, quantile: float = 0.2, cost_bps: float = 8.0,
+                 slip_coef: float = 0.0005, slip_cap_bps: float = 50.0,
+                 liq_ref: float = 1e6, borrowable_qvol: float = 2e7) -> None:
+        self._p = panel
+        self._minq, self._minh, self._lb, self._q = min_qvol, min_history, lookback_days, quantile
+        self._cost = cost_bps / 1e4
+        self._slip_coef, self._slip_cap = slip_coef, slip_cap_bps / 1e4
+        self._liq_ref, self._borrow_q = liq_ref, borrowable_qvol
+
+    def _coin_cost(self, qvol: float) -> float:
+        """Round-trip cost: taker fee both legs + liquidity-scaled slippage (thin=more)."""
+        slip = min(self._slip_cap, self._slip_coef * (self._liq_ref / qvol)) if qvol > 0 else self._slip_cap
+        return self._cost + slip
+
+    def run(self) -> FundingRunResult:
+        all_days = sorted({d for s in self._p.values() for d in s})
+        series: dict[str, list[float]] = {m: [] for m in _MODES}
+        prev_w: dict[str, dict[str, float]] = {m: {} for m in _MODES}
+        names_seen = 0
+        n_rebals = 0
+        for i in range(len(all_days) - 1):
+            d, nxt = all_days[i], all_days[i + 1]
+            elig = eligible_asof(self._p, d, min_qvol=self._minq, min_history=self._minh)
+            if len(elig) < 4:                       # need a cross-section to rank
+                continue
+            scores = funding_score(self._p, elig, d, lookback_days=self._lb)
+            borrowable = {s for s in elig if self._p[s].get(d, (0, 0, 0))[1] >= self._borrow_q}
+            funding_next = {s: self._p[s][nxt][2] for s in elig if nxt in self._p[s]}
+            cost = {s: self._coin_cost(self._p[s][d][1]) for s in elig}
+            names_seen += len(elig)
+            n_rebals += 1
+            for m in _MODES:
+                w = construction_weights(scores, m, quantile=self._q, borrowable=borrowable)
+                # carry the NEW book earns over [d, nxt] at next-day funding, minus rebalance cost vs prior book
+                series[m].append(_book(w, prev_w[m], funding_next, cost))
+                prev_w[m] = w
+        return FundingRunResult(
+            returns_by_mode={m: np.asarray(series[m], dtype=float) for m in _MODES},
+            n_names_avg=(names_seen / n_rebals) if n_rebals else 0.0,
+            trading_days=len(all_days),
+        )
+
+
+def _book(w: dict[str, float], prev_w: dict[str, float],
+          funding_next: dict[str, float], cost: dict[str, float]) -> float:
+    """Carry the NEW book `w` earns over the next day (sum w_i * funding_next_i) minus the
+    round-trip rebalance cost charged on turnover between the prior book and the new one."""
+    realized = sum(wt * funding_next.get(s, 0.0) for s, wt in w.items())
+    turnover = sum(abs(w.get(s, 0.0) - prev_w.get(s, 0.0)) * cost.get(s, 0.0)
+                   for s in set(w) | set(prev_w)) / 2.0
+    return realized - turnover
+```
+(`book_return_xs` stays as the domain primitive exercised by the Task-3 unit tests; the runner uses this direct, clearer `_book`.)
+
+- [ ] **Step 4: run → PASS.** If `long_tilt` isn't positive, debug by printing the realized series; do NOT weaken the determinism test. Adjust the synthetic fixture only if the quantile/360-day math genuinely needs it (report why).
+
+- [ ] **Step 5: mypy** → no errors.
+
+- [ ] **Step 6: commit**
+```bash
+git add src/fxhnt/application/funding_backtest_runner.py tests/integration/test_funding_backtest_runner.py
+git commit -m "feat(xsfunding): walk-forward runner — PIT universe + per-mode daily carry series + liquidity-scaled cost
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 5: gauntlet verdict matrix (modes × liquidity floors)
+
+**Files:** Modify `src/fxhnt/application/funding_backtest_runner.py` (add `evaluate_funding_matrix`); Test `tests/integration/test_funding_backtest_runner.py` (append).
+
+- [ ] **Step 1: append failing test:**
+```python
+from fxhnt.application.funding_backtest_runner import evaluate_funding_matrix
+
+
+def test_evaluate_matrix_runs_all_cells(tmp_path):
+    panel = _panel()
+    report = evaluate_funding_matrix(panel, floors=[1e6, 5e6], lookback_days=7, quantile=0.5,
+                                     cost_bps=8.0, slip_coef=0.0, oos_fraction=0.4)
+    # 3 modes x 2 floors = 6 cells
+    assert len(report["cells"]) == 6
+    for key, cell in report["cells"].items():
+        assert {"mode", "floor", "stats", "verdict"} <= set(cell)
+        assert {"sharpe", "cagr", "max_drawdown", "n_obs"} <= set(cell["stats"])
+        assert {"passed", "dsr", "is_sharpe", "oos_sharpe", "n_trials"} <= set(cell["verdict"])
+        assert cell["verdict"]["n_trials"] == 6        # multiple-testing across the whole matrix
+    import json
+    json.dumps(report)                                  # JSON-serializable (no NaN/np types)
+```
+
+- [ ] **Step 2: run → FAIL**
+
+- [ ] **Step 3: implement** (append to `src/fxhnt/application/funding_backtest_runner.py`):
+```python
+from typing import Any
+
+from fxhnt.domain.backtest import compute_stats
+from fxhnt.domain.gauntlet.core import evaluate, per_period_sharpe
+
+_PERIODS_PER_YEAR = 365   # crypto trades daily, all year
+
+
+def evaluate_funding_matrix(panel: Panel, *, floors: list[float], lookback_days: int = 7,
+                            quantile: float = 0.2, cost_bps: float = 8.0, slip_coef: float = 0.0005,
+                            oos_fraction: float = 0.40, dsr_min: float = 0.95,
+                            oos_min_sharpe: float = 0.0, max_is_oos_decay: float = 0.50) -> dict[str, Any]:
+    """Run the runner at each liquidity floor, gauntlet every (mode, floor) cell.
+    n_trials = total cells tested (multiple-testing correction)."""
+    runs = {f: FundingBacktestRunner(panel, min_qvol=f, lookback_days=lookback_days, quantile=quantile,
+                                     cost_bps=cost_bps, slip_coef=slip_coef).run() for f in floors}
+    series_by_cell = {(m, f): runs[f].returns_by_mode[m] for f in floors for m in _MODES}
+    n_trials = len(series_by_cell)
+    # cross-cell sr_variance (Bailey/Lopez de Prado), matching discovery.py convention
+    is_sharpes = [per_period_sharpe(r[:int((1.0 - oos_fraction) * len(r))])
+                  for r in series_by_cell.values() if len(r) > 3]
+    sr_variance = float(np.var(is_sharpes)) if len(is_sharpes) > 1 else 0.0
+    out: dict[str, Any] = {"cells": {}, "n_trials": n_trials,
+                           "settings": {"floors": floors, "lookback_days": lookback_days,
+                                        "quantile": quantile, "cost_bps": cost_bps,
+                                        "slip_coef": slip_coef, "oos_fraction": oos_fraction}}
+    for (m, f), r in series_by_cell.items():
+        stats = compute_stats(r, periods_per_year=_PERIODS_PER_YEAR)
+        key = f"{m}@{f:.0e}"
+        if len(r) < 3:
+            out["cells"][key] = {"mode": m, "floor": float(f), "stats": stats.model_dump(),
+                                 "verdict": {"passed": False, "dsr": 0.0, "is_sharpe": 0.0,
+                                             "oos_sharpe": 0.0, "n_trials": n_trials,
+                                             "reasons": ["insufficient data"], "pvalue": 1.0}}
+            continue
+        split = int((1.0 - oos_fraction) * len(r))
+        v = evaluate(r[:split], r[split:], n_trials=n_trials, sr_variance=sr_variance,
+                     dsr_min=dsr_min, oos_min_sharpe=oos_min_sharpe, max_is_oos_decay=max_is_oos_decay,
+                     has_economic_rationale=True)   # funding carry is a documented structural premium
+        out["cells"][key] = {"mode": m, "floor": float(f),
+                             "stats": stats.model_dump(), "verdict": v.model_dump()}
+    return out
+```
+
+- [ ] **Step 4: run → PASS.** Verify `per_period_sharpe` is the real gauntlet helper name (check `domain/gauntlet/core.py` + how `application/discovery.py` computes `sr_variance`); if it differs, use the real one. Confirm `model_dump()` is JSON-native (no numpy scalars); if `compute_stats`/`evaluate` leak `np.float64`, coerce with `float(...)`.
+
+- [ ] **Step 5: mypy** → no errors; full `.venv/bin/python -m pytest -q` → all PASS.
+
+- [ ] **Step 6: commit**
+```bash
+git add src/fxhnt/application/funding_backtest_runner.py tests/integration/test_funding_backtest_runner.py
+git commit -m "feat(xsfunding): gauntlet verdict matrix (modes x floors, crypto 365d, cross-cell DSR n_trials)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 6: CLI `backtest-funding` + run the real verdict
+
+**Files:** Modify `src/fxhnt/cli.py`; Test `tests/integration/test_backtest_funding_cli.py`.
+
+- [ ] **Step 1: read** `src/fxhnt/cli.py` `backtest-equity` for the house idiom (`get_settings()`, function-local imports, `--out` json write, echo loop).
+
+- [ ] **Step 2: failing CLI test** (mirror `test_backtest_equity_cli.py`): build a synthetic crypto_pit dir of npz (reuse the `_panel()` shape, write each coin to `<sym>.npz`), set `FXHNT_CRYPTO_PIT_DIR` to it (monkeypatch + `cli.get_settings.cache_clear()`), invoke `backtest-funding --floors 1e6 --out <path>`, assert exit 0 and the report JSON has cells for the 3 modes.
+```python
+import json
+import numpy as np
+from typer.testing import CliRunner
+from fxhnt.cli import app
+
+
+def _write_pit(d):
+    fund = {"HI": 0.0015, "MID": 0.0008, "LO": 0.0001, "NEG": -0.0010}
+    for sym, f in fund.items():
+        days = np.array([1000 + i for i in range(200)], dtype=np.int64)
+        np.savez(d / f"{sym}.npz", day=days, close=np.ones(200), qvol=np.full(200, 5e6),
+                 funding=np.full(200, f))
+
+
+def test_cli_backtest_funding_writes_report(tmp_path, monkeypatch):
+    pit = tmp_path / "pit"; pit.mkdir(); _write_pit(pit)
+    out = tmp_path / "funding_report.json"
+    monkeypatch.setenv("FXHNT_CRYPTO_PIT_DIR", str(pit))
+    from fxhnt import cli as climod
+    climod.get_settings.cache_clear()
+    res = CliRunner().invoke(app, ["backtest-funding", "--floors", "1e6", "--quantile", "0.5",
+                                   "--cost-bps", "8", "--out", str(out)])
+    assert res.exit_code == 0, res.output
+    report = json.loads(out.read_text())
+    modes = {c["mode"] for c in report["cells"].values()}
+    assert modes == {"long_tilt", "market_neutral", "executable"}
+```
+
+- [ ] **Step 3: run → FAIL**
+
+- [ ] **Step 4: implement** the command in `src/fxhnt/cli.py`:
+```python
+@app.command("backtest-funding")
+def backtest_funding(
+    floors: list[float] = typer.Option([1e6, 5e6, 2e7], "--floors", help="Liquidity floors ($/day) to sweep."),
+    lookback_days: int = typer.Option(7, "--lookback-days", help="Trailing window for the funding score."),
+    quantile: float = typer.Option(0.2, "--quantile", help="Top/bottom quantile for construction."),
+    cost_bps: float = typer.Option(8.0, "--cost-bps", help="Taker round-trip cost (both legs), bps."),
+    slip_coef: float = typer.Option(0.0005, "--slip-coef", help="Liquidity-scaled slippage coefficient."),
+    out: str = typer.Option("/backtest-data/funding_backtest_report.json", "--out", help="Report JSON path."),
+) -> None:
+    """Cross-sectional crypto funding-dispersion backtest: verdict matrix (modes x liquidity floors)
+    over the survivorship-free crypto_pit panel."""
+    import json
+    from fxhnt.adapters.data.crypto_pit_panel import CryptoPitPanelSource
+    from fxhnt.application.funding_backtest_runner import evaluate_funding_matrix
+
+    s = get_settings()
+    panel = CryptoPitPanelSource(s.crypto_pit_dir).panel()
+    report = evaluate_funding_matrix(panel, floors=floors, lookback_days=lookback_days, quantile=quantile,
+                                     cost_bps=cost_bps, slip_coef=slip_coef,
+                                     oos_fraction=s.gauntlet.oos_fraction, dsr_min=s.gauntlet.dsr_min,
+                                     oos_min_sharpe=s.gauntlet.oos_min_sharpe,
+                                     max_is_oos_decay=s.gauntlet.max_is_oos_decay)
+    with open(out, "w") as fh:
+        json.dump(report, fh, indent=2)
+    for key, c in sorted(report["cells"].items()):
+        v, st = c["verdict"], c["stats"]
+        typer.echo(f"backtest-funding {key}: passed={v['passed']} dsr={v['dsr']:.3f} "
+                   f"sharpe={st['sharpe']:.2f} cagr={st['cagr']*100:.1f}% maxDD={st['max_drawdown']*100:.1f}% -> {out}")
+```
+(Match the real `get_settings()` idiom + function-local imports from `backtest-equity`; there's no store to close here.)
+
+- [ ] **Step 5: run → PASS**, full `.venv/bin/python -m pytest -q` → all PASS, mypy on cli.py → no NEW errors.
+
+- [ ] **Step 6: commit**
+```bash
+git add src/fxhnt/cli.py tests/integration/test_backtest_funding_cli.py
+git commit -m "feat(xsfunding): backtest-funding CLI — verdict matrix over crypto_pit panel
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+- [ ] **Step 7: RUN THE REAL VERDICT** (controller does this, not a subagent): point at the on-disk panel and run the matrix:
+```bash
+FXHNT_CRYPTO_PIT_DIR=/home/jgrusewski/Work/foxhunt/data/surfer/crypto_pit \
+  .venv/bin/fxhnt backtest-funding --floors 1e6 --floors 5e6 --floors 2e7 \
+  --quantile 0.2 --cost-bps 8 --out /tmp/funding_report.json
+```
+Report the verdict matrix; apply the pre-registered kill criteria (DSR<0.5 / edge only sub-$1M-ADV / corr>0.6 to existing funding+momentum).
+
+---
+
+## Known caveats (flag, do not fix here)
+- Delisted-ticker roster = the 35 famous blow-ups in the panel (survivorship-free for those, not an exhaustive census). Documented; future `data.binance.vision` hardening.
+- Daily-summed funding only (no 8h). Daily rebalance.
+- Cost model is a liquidity-scaled approximation, not order-book-exact — deliberately conservative.
+
+## Final review (after Task 6)
+Final code review over the whole diff; full suite + mypy. Then interpret the verdict and decide on the paper-forward follow-on. Do NOT push.

docs/superpowers/plans/2026-06-19-edge-discovery-sprint.md

@@ -0,0 +1,30 @@
+# Edge-Discovery Sprint Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use `- [ ]`.
+
+**Goal:** Widen the book with more genuinely-uncorrelated edges (+ raise return). Build + gauntlet a batch of candidates, keep the survivors that are uncorrelated to the existing 4 edges.
+
+**Discipline (non-negotiable):** every candidate → PIT survivorship-free backtest → gauntlet (DSR/OOS, crypto periods_per_year=365) → PRE-REGISTERED kill: net DSR<0.5, OR edge only in uninvestable thin names, OR |correlation| > 0.6 to the existing book (multi-premia / crypto-momentum / funding / xsfunding). A sleeve earns a slot ONLY if uncorrelated. Report each survivor's correlation + return contribution.
+
+**Tech:** Python 3.12, numpy; reuse the `crypto_pit` panel + the cross-sectional machinery (`eligible_asof`, `construction_weights`, `robust_z`, `realized_vol_from_closes`, `daily_returns_for_weights`, the gauntlet `compute_stats`/`evaluate`/`per_period_sharpe`). pytest, strict mypy, NO live network in tests, commit per task (do NOT push), trailer `Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>`.
+
+## Candidates
+**Stream A — crypto, data-in-hand (`crypto_pit` panel = {sym:{day:(close[perp], qvol, funding)}}):**
+1. Crypto **TS momentum/trend** (directional — return engine).
+2. XS **short-term reversal** (market-neutral).
+3. XS **low-volatility** (market-neutral).
+4. XS **illiquidity/volume** (market-neutral; cost-fragile — flag).
+
+**Stream B — non-crypto diversifier (liquid multi-asset ETFs, data-in-hand via the multistrat source):**
+5. **Cross-asset trend** (managed-futures style on SPY/IEF/GLD/PDBC/DBMF/+) — crisis-alpha tail-hedge.
+
+## Build order (max reuse first; discovery is emergent — drop a candidate if the gauntlet kills it)
+1. **Crypto XS price-factor domain** — `xs_reversal_score`, `xs_lowvol_score`, `xs_illiquidity_score` (pure, on the panel; reuse robust_z + realized_vol_from_closes).
+2. **CryptoFactorBacktestRunner** — PIT cross-sectional, pluggable score fn, books PRICE returns (reuse `daily_returns_for_weights` on the close panel) − liquidity-scaled cost; `evaluate_factor_matrix` (gauntlet per construction × factor). 
+3. **RUN** the 3 XS factors (long_tilt + market_neutral) → verdicts (controller).
+4. **TS-trend domain + runner** — per-asset trend signal (multi-horizon TSMOM, vol-scaled) → directional book; reusable for crypto + ETF panels.
+5. **RUN crypto TS-trend** on crypto_pit → verdict (controller).
+6. **ETF panel adapter** (reuse multistrat's ETF source) + **RUN cross-asset TS-trend** → verdict (controller).
+7. **Correlation check + synthesis** — generate the existing edges' return series, correlate each survivor, report the widened book + which earn a slot.
+
+Each build task: TDD, full suite + mypy green, commit. Each RUN task: controller runs locally on data-in-hand, applies kill criteria, reports the verdict honestly (expect several to die — that's the loop working).

docs/superpowers/plans/2026-06-19-funding-basis-modeling.md

@@ -0,0 +1,360 @@
+# Funding Backtest — Spot-Perp Basis Modeling — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use `- [ ]`.
+
+**Goal:** Extend the cross-sectional funding backtest to model the spot-perp BASIS P&L (the dominant missing risk), turning the funding-only Sharpe into the real net-of-basis number.
+
+**Architecture:** Add a SPOT price leg (fetched from Binance spot klines) alongside the panel's existing PERP `close`, guarded against the delisted-ticker-reuse trap (clip spot to each coin's perp-alive window + reject |basis|>30% decoupling days). The runner's per-coin return becomes the true delta-neutral cash-and-carry `(spot_ret − perp_ret + funding)`, with funding-only fallback where spot is unavailable. Re-run the verdict matrix.
+
+**Tech Stack:** Python 3.12, numpy, Binance spot klines (`api/v3/klines`), the existing crypto_pit panel + funding runner/gauntlet, pytest, strict mypy.
+
+**Spike (done):** `close`=PERP; spot is the missing leg; spot obtainable incl. delisted BUT ticker-reuse trap (spot LUNAUSDT→LUNA2.0) MUST be guarded; alt Δbasis vol is 1.6–5.5× funding vol → basis materially changes the Sharpe.
+
+**House rules:** TDD, pure functions in domain, NO live network in tests (fake clients / synthetic panels), strict mypy (no NEW errors), commit per task (do NOT push), trailer `Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>`. Crypto 365/yr.
+
+**Backward-compat:** when no spot panel is supplied, the runner must behave EXACTLY as today (funding-only) — all existing tests stay green.
+
+---
+
+## Reuse contract
+```python
+# domain/cross_sectional_funding.py
+Panel = dict[str, dict[int, tuple[float, float, float]]]   # {sym:{day:(perp_close, qvol, funding)}}
+# application/funding_backtest_runner.py
+class FundingBacktestRunner: __init__(panel, *, min_qvol, min_history, lookback_days, quantile, cost_bps, slip_coef, ...)
+  .run() -> FundingRunResult(returns_by_mode, n_names_avg, panel_days, n_rebalances)
+def evaluate_funding_matrix(panel, *, floors, ...) -> dict
+# adapters/data/crypto_pit_panel.py : CryptoPitPanelSource(pit_dir).panel()
+```
+New type: `SpotPanel = dict[str, dict[int, float]]` ({sym:{epoch_day:spot_close}}).
+
+---
+
+## File Structure
+- Create `src/fxhnt/adapters/data/binance_spot_history.py` — `BinanceSpotHistory` (fetch spot daily close, paginated).
+- Create `src/fxhnt/application/basis_panel_builder.py` — orchestrate: perp panel + spot fetch → cleaned `SpotPanel`; pure guard `clean_spot_leg`.
+- Modify `src/fxhnt/domain/cross_sectional_funding.py` — add pure `clean_spot_leg` + `SpotPanel` alias.
+- Modify `src/fxhnt/application/funding_backtest_runner.py` — `_book` generalized to a per-coin return dict; runner accepts `spot: SpotPanel | None`; `evaluate_funding_matrix` accepts/threads spot.
+- Modify `src/fxhnt/adapters/data/crypto_pit_panel.py` — add `spot_panel(spot_dir)` loader (reads the spot npz sidecar).
+- Modify `src/fxhnt/cli.py` — `build-basis-panel` command + `backtest-funding --spot-dir`.
+- Tests under `tests/unit/` + `tests/integration/`.
+
+---
+
+## Task 1: spot-history adapter + the ticker-reuse guard (pure)
+
+**Files:** Create `src/fxhnt/adapters/data/binance_spot_history.py`; Modify `src/fxhnt/domain/cross_sectional_funding.py`; Tests `tests/unit/test_cross_sectional_funding.py` (append), `tests/integration/test_binance_spot_history.py`.
+
+- [ ] **Step 1: pure guard test** (append to `tests/unit/test_cross_sectional_funding.py`):
+```python
+from fxhnt.domain.cross_sectional_funding import clean_spot_leg
+
+
+def test_clean_spot_clips_to_perp_window_and_rejects_relist():
+    perp_days = [100, 101, 102, 103]                       # perp alive days 100-103
+    perp_close = {100: 1.00, 101: 1.00, 102: 1.00, 103: 1.00}
+    # spot has a pre-window day (99), in-window clean days, a relist-jump day (102 → basis 50x), and a post-window day (200)
+    spot_raw = {99: 1.0, 100: 1.001, 101: 0.999, 102: 50.0, 103: 1.000, 200: 8.87}
+    clean = clean_spot_leg(perp_days, perp_close, spot_raw, max_basis=0.30)
+    assert set(clean) == {100, 101, 103}                  # 99/200 out of window; 102 rejected (|basis|>30%)
+    assert clean[100] == 1.001
+
+
+def test_clean_spot_empty_when_no_overlap():
+    assert clean_spot_leg([100, 101], {100: 1.0, 101: 1.0}, {200: 5.0}, max_basis=0.30) == {}
+```
+
+- [ ] **Step 2: run → FAIL.**
+
+- [ ] **Step 3: implement** `clean_spot_leg` (append to `src/fxhnt/domain/cross_sectional_funding.py`):
+```python
+SpotPanel = dict[str, dict[int, float]]
+
+
+def clean_spot_leg(perp_days: list[int], perp_close: dict[int, float],
+                   spot_raw: dict[int, float], *, max_basis: float = 0.30) -> dict[int, float]:
+    """Cleaned spot leg for one coin: keep spot ONLY on days within the perp's alive window
+    [min(perp_days), max(perp_days)] AND where |basis|=|(perp-spot)/spot| <= max_basis.
+    Drops pre-listing/post-delisting spot bars and ticker-reuse/relist decoupling artifacts
+    (e.g. spot LUNAUSDT becoming LUNA 2.0). Days dropped here fall back to funding-only downstream."""
+    if not perp_days:
+        return {}
+    lo, hi = min(perp_days), max(perp_days)
+    out: dict[int, float] = {}
+    for d, sp in spot_raw.items():
+        if d < lo or d > hi or sp <= 0:
+            continue
+        pc = perp_close.get(d)
+        if pc is None or pc <= 0:
+            continue
+        if abs((pc - sp) / sp) <= max_basis:
+            out[d] = sp
+    return out
+```
+
+- [ ] **Step 4: spot adapter test** `tests/integration/test_binance_spot_history.py` (fake HTTP, no live net) — mirror how `binance_funding.py` is tested if a pattern exists; otherwise inject a fake fetch:
+```python
+from fxhnt.adapters.data.binance_spot_history import BinanceSpotHistory
+
+
+def test_spot_history_parses_klines(monkeypatch):
+    # kline row: [openTime, open, high, low, close, vol, closeTime, quoteVol, ...]; close=index 4, openTime ms
+    rows = [[100 * 86400000, "1", "1", "1", "1.5", "0", 0, "0"],
+            [101 * 86400000, "1", "1", "1", "1.7", "0", 0, "0"]]
+    h = BinanceSpotHistory()
+    monkeypatch.setattr(h, "_get", lambda url: rows)   # one page then empty
+    calls = {"n": 0}
+    def fake_get(url):
+        calls["n"] += 1
+        return rows if calls["n"] == 1 else []
+    monkeypatch.setattr(h, "_get", fake_get)
+    out = h.daily_close("AAAUSDT")
+    assert out == {100: 1.5, 101: 1.7}                  # epoch-DAY keys, close values
+```
+
+- [ ] **Step 5: implement** `src/fxhnt/adapters/data/binance_spot_history.py`:
+```python
+"""Binance SPOT daily-close history (the long leg of the cash-and-carry), paginated.
+Returns {epoch_day: close}. Live network in production; tests inject a fake `_get`."""
+from __future__ import annotations
+
+import json
+import time
+import urllib.request
+from typing import Any
+
+_SPOT = "https://api.binance.com"
+
+
+class BinanceSpotHistory:
+    def _get(self, url: str, tries: int = 4) -> Any:
+        last: Exception | None = None
+        for i in range(tries):
+            try:
+                with urllib.request.urlopen(url, timeout=30) as r:
+                    return json.loads(r.read())
+            except Exception as e:   # network/HTTP — retry with backoff
+                last = e
+                time.sleep(0.5 * (i + 1))
+        raise RuntimeError(f"spot fetch failed: {url}: {last}")
+
+    def daily_close(self, symbol: str, start_ms: int = 0) -> dict[int, float]:
+        """Full daily spot-close history {epoch_day: close}, forward-paginated (limit 1000)."""
+        out: dict[int, float] = {}
+        st = start_ms
+        while True:
+            url = (f"{_SPOT}/api/v3/klines?symbol={symbol}&interval=1d"
+                   f"&startTime={st}&limit=1000")
+            rows = self._get(url)
+            if not rows:
+                break
+            for r in rows:
+                out[int(r[0]) // 86_400_000] = float(r[4])
+            nxt = int(rows[-1][0]) + 86_400_000
+            if nxt <= st or len(rows) < 1000:
+                st = nxt
+                if len(rows) < 1000:
+                    break
+            st = nxt
+        return out
+```
+(NOTE: keep the pagination simple + correct — advance `st` past the last row's openTime; stop when a page returns <1000 rows or empty. The test's fake returns one full-looking page then empty; ensure the loop terminates. Adjust the loop so it terminates cleanly for both the fake and real Binance; the key invariant: every row's `openTime_ms // 86_400_000` is the epoch-day key and `r[4]` is the close.)
+
+- [ ] **Step 6: run both test files → PASS;** mypy on the 2 files → no errors; full suite → PASS.
+
+- [ ] **Step 7: commit**
+```bash
+git add src/fxhnt/adapters/data/binance_spot_history.py src/fxhnt/domain/cross_sectional_funding.py tests/unit/test_cross_sectional_funding.py tests/integration/test_binance_spot_history.py
+git commit -m "feat(xsfunding): spot-history adapter + ticker-reuse guard clean_spot_leg (basis prep)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 2: basis-panel builder + CLI
+
+**Files:** Create `src/fxhnt/application/basis_panel_builder.py`; Modify `src/fxhnt/adapters/data/crypto_pit_panel.py` (add `spot_panel` loader); Modify `src/fxhnt/cli.py` (`build-basis-panel`); Tests `tests/integration/test_basis_panel_builder.py`.
+
+- [ ] **Step 1: failing test** `tests/integration/test_basis_panel_builder.py`:
+```python
+import numpy as np
+from fxhnt.application.basis_panel_builder import build_spot_panel
+
+
+class _FakeSpot:
+    def __init__(self, data): self._d = data
+    def daily_close(self, symbol, start_ms=0): return dict(self._d.get(symbol, {}))
+
+
+def test_build_spot_panel_applies_guard():
+    # perp panel: AAA alive days 100-103; spot has an in-window relist jump on 102 + a post-window 200
+    panel = {"AAA": {100: (1.0, 5e6, 0.001), 101: (1.0, 5e6, 0.001),
+                     102: (1.0, 5e6, 0.001), 103: (1.0, 5e6, 0.001)}}
+    spot = _FakeSpot({"AAA": {100: 1.001, 101: 0.999, 102: 50.0, 103: 1.0, 200: 8.87}})
+    sp = build_spot_panel(panel, spot, max_basis=0.30)
+    assert set(sp["AAA"]) == {100, 101, 103}            # 102 rejected, 200 out of window
+```
+
+- [ ] **Step 2: run → FAIL.**
+
+- [ ] **Step 3: implement** `src/fxhnt/application/basis_panel_builder.py`:
+```python
+"""Build the cleaned SPOT leg for the funding backtest: fetch spot history per panel symbol,
+apply the perp-window + |basis| guard (clean_spot_leg), return a SpotPanel."""
+from __future__ import annotations
+
+from fxhnt.domain.cross_sectional_funding import Panel, SpotPanel, clean_spot_leg
+
+
+class _SpotClient:                       # structural typing hint
+    def daily_close(self, symbol: str, start_ms: int = 0) -> dict[int, float]: ...
+
+
+def build_spot_panel(panel: Panel, spot_client: _SpotClient, *, max_basis: float = 0.30) -> SpotPanel:
+    out: SpotPanel = {}
+    for sym, series in panel.items():
+        perp_days = sorted(series)
+        perp_close = {d: series[d][0] for d in perp_days}
+        try:
+            raw = spot_client.daily_close(sym)
+        except Exception:
+            continue                     # spot unavailable for this coin -> funding-only downstream
+        cleaned = clean_spot_leg(perp_days, perp_close, raw, max_basis=max_basis)
+        if cleaned:
+            out[sym] = cleaned
+    return out
+```
+
+- [ ] **Step 4: spot-panel npz persistence** — add to `CryptoPitPanelSource` (or a sibling) a way to WRITE the spot panel to npz + a `spot_panel(spot_dir)` loader, so the CLI can persist + the backtest can reload without re-fetching. Add to `src/fxhnt/adapters/data/crypto_pit_panel.py`:
+```python
+    @staticmethod
+    def write_spot_panel(spot: dict[str, dict[int, float]], spot_dir: str) -> None:
+        import os
+        os.makedirs(spot_dir, exist_ok=True)
+        for sym, series in spot.items():
+            days = sorted(series)
+            np.savez(os.path.join(spot_dir, f"{sym}.npz"),
+                     day=np.array(days, dtype=np.int64),
+                     spot=np.array([series[d] for d in days], dtype=float))
+
+    @staticmethod
+    def load_spot_panel(spot_dir: str) -> dict[str, dict[int, float]]:
+        import glob, os
+        out: dict[str, dict[int, float]] = {}
+        for path in sorted(glob.glob(os.path.join(spot_dir, "*.npz"))):
+            sym = os.path.splitext(os.path.basename(path))[0]
+            with np.load(path) as z:
+                out[sym] = {int(d): float(s) for d, s in zip(z["day"].astype(np.int64), z["spot"].astype(float))}
+        return out
+```
+Add a round-trip test (write then load == original).
+
+- [ ] **Step 5: CLI `build-basis-panel`** in `src/fxhnt/cli.py` (function-local imports, `get_settings()` idiom):
+```python
+@app.command("build-basis-panel")
+def build_basis_panel(
+    spot_dir: str = typer.Option(..., "--spot-dir", help="Output dir for the cleaned spot-leg npz panel."),
+    max_basis: float = typer.Option(0.30, "--max-basis", help="Reject |basis| above this (decoupling/relist guard)."),
+) -> None:
+    """Fetch + clean the SPOT leg (perp-window + |basis| guard) for every coin in the crypto_pit panel,
+    writing a spot npz panel for the basis-aware funding backtest."""
+    from fxhnt.adapters.data.crypto_pit_panel import CryptoPitPanelSource
+    from fxhnt.adapters.data.binance_spot_history import BinanceSpotHistory
+    from fxhnt.application.basis_panel_builder import build_spot_panel
+    s = get_settings()
+    panel = CryptoPitPanelSource(s.crypto_pit_dir).panel()
+    spot = build_spot_panel(panel, BinanceSpotHistory(), max_basis=max_basis)
+    CryptoPitPanelSource.write_spot_panel(spot, spot_dir)
+    typer.echo(f"build-basis-panel: wrote spot leg for {len(spot)}/{len(panel)} coins -> {spot_dir}")
+```
+
+- [ ] **Step 6: run tests → PASS;** mypy → no new errors; full suite → PASS.
+
+- [ ] **Step 7: commit**
+```bash
+git add src/fxhnt/application/basis_panel_builder.py src/fxhnt/adapters/data/crypto_pit_panel.py src/fxhnt/cli.py tests/integration/test_basis_panel_builder.py
+git commit -m "feat(xsfunding): basis-panel builder + build-basis-panel CLI + spot-panel npz persistence
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 3: runner — true delta-neutral return (spot+perp+funding) with funding-only fallback
+
+**Files:** Modify `src/fxhnt/application/funding_backtest_runner.py`; Modify `src/fxhnt/cli.py` (`backtest-funding --spot-dir`); Test `tests/integration/test_funding_backtest_runner.py` (append).
+
+- [ ] **Step 1: failing test:**
+```python
+def test_runner_basis_changes_return_vs_funding_only():
+    # one coin held; spot moves AGAINST the perp (basis widens) -> net return < funding-only
+    panel = {}
+    fund = {"HI": 0.002, "B": 0.001, "C": 0.0005, "D": -0.001}
+    for sym, f in fund.items():
+        panel[sym] = {1000 + i: (100.0, 5e6, f) for i in range(60)}   # perp flat at 100
+    # spot for HI declines 100 -> 99 over the window (long-spot leg loses -> basis P&L negative)
+    spot = {"HI": {1000 + i: 100.0 - 0.02 * i for i in range(60)}}
+    from fxhnt.application.funding_backtest_runner import FundingBacktestRunner
+    fo = FundingBacktestRunner(panel, min_qvol=1e6, min_history=20, lookback_days=7, quantile=0.5,
+                               cost_bps=0.0, slip_coef=0.0).run().returns_by_mode["long_tilt"]
+    bm = FundingBacktestRunner(panel, spot=spot, min_qvol=1e6, min_history=20, lookback_days=7,
+                               quantile=0.5, cost_bps=0.0, slip_coef=0.0).run().returns_by_mode["long_tilt"]
+    import numpy as np
+    assert float(np.sum(bm)) < float(np.sum(fo))        # basis loss on the long-spot leg reduces return
+    # coins without spot fall back to funding-only (no crash)
+    assert np.isfinite(bm).all()
+```
+
+- [ ] **Step 2: run → FAIL.**
+
+- [ ] **Step 3: implement** — generalize `_book` to take a per-coin return dict; runner accepts `spot`:
+  - Change `_book(w, prev_w, unit_ret, cost)` → `realized = Σ w·unit_ret[s]`, turnover cost unchanged (rename `funding_next` param to `unit_ret`).
+  - `FundingBacktestRunner.__init__` gains `spot: SpotPanel | None = None` (store `self._spot`).
+  - In `run()`, per rebalance, build `unit_ret` per eligible coin:
+    ```python
+    unit_ret: dict[str, float] = {}
+    for s in elig:
+        f = self._p[s][nxt][2] if nxt in self._p[s] else 0.0
+        if self._spot is not None and s in self._spot and d in self._spot[s] and nxt in self._spot[s] \
+           and d in self._p[s] and nxt in self._p[s] and self._p[s][d][0] > 0 and self._spot[s][d] > 0:
+            spot_ret = self._spot[s][nxt] / self._spot[s][d] - 1.0
+            perp_ret = self._p[s][nxt][0] / self._p[s][d][0] - 1.0
+            unit_ret[s] = (spot_ret - perp_ret) + f      # delta-neutral cash-and-carry
+        else:
+            unit_ret[s] = f                               # funding-only fallback
+    ```
+    then `series[m].append(_book(w, prev_w[m], unit_ret, cost))`.
+  - `evaluate_funding_matrix(panel, *, spot=None, floors, ...)` threads `spot` into each `FundingBacktestRunner(..., spot=spot)`. Add a `"basis_modeled": spot is not None` flag to the report dict + update the `funding_only` caveat text to say basis IS modeled when spot is provided.
+  - Backward-compat: `spot=None` → `unit_ret = funding-only` → identical to current behavior (existing tests pass).
+
+- [ ] **Step 4: run** the new test + the full runner file → PASS. Existing funding-only tests MUST stay green (spot defaults None). mypy → no errors. Full suite → PASS.
+
+- [ ] **Step 5: CLI wiring** — `backtest-funding` gains `--spot-dir` (optional); when given, load the spot panel and pass `spot=` to `evaluate_funding_matrix`; echo whether basis is modeled. Update the caveat echo to reflect basis-modeled vs funding-only.
+
+- [ ] **Step 6: commit**
+```bash
+git add src/fxhnt/application/funding_backtest_runner.py src/fxhnt/cli.py tests/integration/test_funding_backtest_runner.py
+git commit -m "feat(xsfunding): model spot-perp basis P&L in runner (true delta-neutral return; funding-only fallback)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 4: build the real spot panel + re-run the net-of-basis verdict (controller)
+- [ ] Build the spot panel against the real universe (network, ~180 coins):
+```bash
+FXHNT_CRYPTO_PIT_DIR=/home/jgrusewski/Work/foxhunt/data/surfer/crypto_pit \
+  .venv/bin/fxhnt build-basis-panel --spot-dir /tmp/crypto_pit_spot --max-basis 0.30
+```
+- [ ] Re-run the matrix WITH basis:
+```bash
+FXHNT_CRYPTO_PIT_DIR=/home/jgrusewski/Work/foxhunt/data/surfer/crypto_pit \
+  .venv/bin/fxhnt backtest-funding --floors 1e6 --floors 5e6 --floors 2e7 \
+  --quantile 0.2 --cost-bps 8 --spot-dir /tmp/crypto_pit_spot --out /tmp/funding_basis_report.json
+```
+- [ ] Report the net-of-basis verdict vs the funding-only number; apply the pre-registered kill criteria (DSR<0.5 / edge only sub-$1M / corr>0.6 to existing books). Honest verdict.
+
+## Final review (after Task 3)
+Final code review over the whole basis diff; full suite + mypy. Then interpret the net-of-basis verdict and decide next (paper-forward / cockpit wiring / prune).

docs/superpowers/plans/2026-06-19-xsfunding-paper-track.md

@@ -0,0 +1,273 @@
+# xsfunding Paper-Forward Track Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use `- [ ]`.
+
+**Goal:** Wire the validated `executable` cross-sectional funding construction as a live daily paper-forward cockpit track (`xsfunding`) that experiences the REAL net-of-everything return (funding + basis + the tail the backtest had to guard out), with its backtest verdict surfaced.
+
+**Architecture:** A live-booking `ForwardStrategy` (mirrors the existing eqfactor/funding tracks) that each day fetches live universe + trailing funding + perp & spot prices, books the prior delta-neutral book's realized return using the SAME `cross_sectional_funding` domain math the backtest validated, rebalances `executable` weights, persists state in `extra`. Registered in `STRATEGY_REGISTRY` + a Dagster asset in the daily combined-book job + cockpit, backtest verdict in `backtest_summary`.
+
+**Tech Stack:** Python 3.12, `BinanceFundingClient` + Binance spot/perp tickers, the `cross_sectional_funding` domain, `ForwardTracker`/`ForwardStrategy`, Dagster, cockpit pg. pytest, strict mypy.
+
+**House rules:** TDD, NO live network in tests (fake clients), strict mypy (no NEW errors), commit per task (do NOT push until the deploy task), trailer `Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>`.
+
+---
+
+## Reuse contract (READ these first)
+- `src/fxhnt/application/forward_tracker.py` — `ForwardStrategy` Protocol `advance(last_date: str|None, extra: dict) -> tuple[list[tuple[str,float]], dict]`; `ForwardTracker(strategy, state_file).step()`. Study the LIVE-BOOKING pattern (positions+marks in `extra`).
+- `src/fxhnt/application/paper_strategies.py` — the existing `FundingCarryStrategy` live track (the closest analog: live crypto booking). Mirror its shape.
+- `src/fxhnt/application/equity_factor_strategy.py` — `EquityFactorLong.advance` live-booking (positions/prices/last_rebal in extra, monthly rebal, book_return on prior book). Mirror this structure.
+- `src/fxhnt/adapters/data/binance_funding.py` — `BinanceFundingClient`: `universe() -> {sym: quoteVolume}`, `scan(prev) -> (liq, tf30, last24)` (liq=liquid universe vol, tf30=trailing-30d-mean funding, last24=today's funding). 
+- `src/fxhnt/domain/cross_sectional_funding.py` — `funding_score`-style scoring + `construction_weights(scores, "executable", quantile, borrowable)` + the signed-carry concept.
+- `src/fxhnt/registry.py`, `src/fxhnt/adapters/orchestration/assets.py` (eqfactor_long_nav, cockpit_forward), `definitions.py` (combined_book_job + Definitions) — the wiring pattern (see the eqfactor entries).
+- `src/fxhnt/application/backtest_ingest.py` + `adapters/persistence/forward_nav.py` `upsert_backtest_summary` — the verdict-surface path.
+
+---
+
+## File Structure
+- Create `src/fxhnt/adapters/data/binance_xsfunding_live.py` — `BinanceXsFundingLive`: per-coin live `{sym: (trailing_funding, today_funding, perp_price, spot_price, qvol)}` for the universe.
+- Create `src/fxhnt/application/xsfunding_strategy.py` — `XsFundingForward` live-booking ForwardStrategy (executable construction, delta-neutral funding+basis book).
+- Modify `src/fxhnt/registry.py` — add `xsfunding` entry.
+- Modify `src/fxhnt/adapters/orchestration/assets.py` + `definitions.py` — `xsfunding_nav` asset in the daily job + Definitions + cockpit_forward dep.
+- Tests: `tests/integration/test_xsfunding_strategy.py`, `tests/integration/test_binance_xsfunding_live.py`.
+
+---
+
+## Task 1: live data source
+
+**Files:** Create `src/fxhnt/adapters/data/binance_xsfunding_live.py`; Test `tests/integration/test_binance_xsfunding_live.py`.
+
+- [ ] **Step 1** — READ `binance_funding.py` to confirm `BinanceFundingClient.scan`/`universe` signatures + how it does `_get`. The live source COMPOSES it for universe+funding, and adds perp+spot current prices via two bulk ticker calls: perp `fapi/v1/ticker/price` (all perps), spot `api/v3/ticker/price` (all spot). 
+
+- [ ] **Step 2** — failing test `tests/integration/test_binance_xsfunding_live.py` (inject fakes; NO live net):
+```python
+from fxhnt.adapters.data.binance_xsfunding_live import BinanceXsFundingLive
+
+
+class _FakeFunding:
+    def scan(self, prev):
+        liq = {"AAAUSDT": 5e6, "BBBUSDT": 3e6}
+        tf30 = {"AAAUSDT": 0.001, "BBBUSDT": 0.0003}
+        last24 = {"AAAUSDT": 0.0012, "BBBUSDT": 0.0004}
+        return liq, tf30, last24
+
+
+def test_live_source_merges_funding_and_prices(monkeypatch):
+    src = BinanceXsFundingLive(funding=_FakeFunding())
+    monkeypatch.setattr(src, "_perp_prices", lambda: {"AAAUSDT": 100.0, "BBBUSDT": 10.0})
+    monkeypatch.setattr(src, "_spot_prices", lambda: {"AAAUSDT": 99.5, "BBBUSDT": 10.1})
+    snap = src.snapshot(prev={})
+    assert snap["AAAUSDT"] == (0.001, 0.0012, 100.0, 99.5, 5e6)   # (tf, today_funding, perp, spot, qvol)
+    assert snap["BBBUSDT"][2] == 10.0 and snap["BBBUSDT"][3] == 10.1
+```
+
+- [ ] **Step 3** — run → FAIL.
+
+- [ ] **Step 4** — implement `src/fxhnt/adapters/data/binance_xsfunding_live.py`:
+```python
+"""Live per-coin snapshot for the cross-sectional funding paper track: trailing funding,
+today's funding, perp price, spot price, quote-volume — for the liquid universe."""
+from __future__ import annotations
+
+import json
+import urllib.request
+from typing import Any
+
+from fxhnt.adapters.data.binance_funding import BinanceFundingClient
+
+_FAPI = "https://fapi.binance.com"
+_SPOT = "https://api.binance.com"
+
+
+class BinanceXsFundingLive:
+    def __init__(self, funding: Any | None = None) -> None:
+        self._f = funding if funding is not None else BinanceFundingClient()
+
+    def _get(self, url: str) -> Any:
+        with urllib.request.urlopen(url, timeout=30) as r:
+            return json.loads(r.read())
+
+    def _perp_prices(self) -> dict[str, float]:
+        return {x["symbol"]: float(x["price"]) for x in self._get(f"{_FAPI}/fapi/v1/ticker/price")}
+
+    def _spot_prices(self) -> dict[str, float]:
+        return {x["symbol"]: float(x["price"]) for x in self._get(f"{_SPOT}/api/v3/ticker/price")}
+
+    def snapshot(self, prev: dict[str, float]) -> dict[str, tuple[float, float, float, float, float]]:
+        """{sym: (trailing_funding, today_funding, perp_price, spot_price, qvol)} over the liquid
+        universe (+ any prev-held coin still scannable). Coins lacking a spot price are omitted
+        (funding-only handling happens upstream if needed)."""
+        liq, tf30, last24 = self._f.scan(prev)
+        perp, spot = self._perp_prices(), self._spot_prices()
+        out: dict[str, tuple[float, float, float, float, float]] = {}
+        for s in liq:
+            if s in perp and s in spot and perp[s] > 0 and spot[s] > 0:
+                out[s] = (tf30.get(s, 0.0), last24.get(s, 0.0), perp[s], spot[s], liq[s])
+        return out
+```
+
+- [ ] **Step 5** — run → PASS; mypy → no errors; full suite → PASS.
+
+- [ ] **Step 6** — commit:
+```bash
+git add src/fxhnt/adapters/data/binance_xsfunding_live.py tests/integration/test_binance_xsfunding_live.py
+git commit -m "feat(xsfunding): live per-coin snapshot (funding + perp + spot + qvol)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 2: XsFundingForward live-booking strategy
+
+**Files:** Create `src/fxhnt/application/xsfunding_strategy.py`; Test `tests/integration/test_xsfunding_strategy.py`.
+
+- [ ] **Step 1** — READ `EquityFactorLong.advance` + `ForwardTracker` to mirror the live-booking contract exactly (extra carries positions/marks/last_date; advance books prior book then rebalances; returns `(rows, extra)` where rows = `[(date, realized_return)]`).
+
+- [ ] **Step 2** — failing test `tests/integration/test_xsfunding_strategy.py`:
+```python
+from fxhnt.application.xsfunding_strategy import XsFundingForward
+
+
+class _FakeLive:
+    def __init__(self, seq): self._seq = seq; self._i = -1
+    def snapshot(self, prev):
+        self._i += 1
+        return self._seq[self._i]
+
+
+def _snap(funding):   # 5 coins, perp=spot=100 (no basis move on day 1), given trailing funding
+    return {s: (f, f, 100.0, 100.0, 5e6) for s, f in funding.items()}
+
+
+def test_first_advance_inception_no_booking():
+    live = _FakeLive([_snap({"A": 0.002, "B": 0.001, "C": 0.0, "D": -0.001, "E": -0.002})])
+    strat = XsFundingForward(live, quantile=0.4, clock=lambda: "2026-06-19")
+    rows, extra = strat.advance(None, {})
+    assert rows == []                          # inception: no prior book to book
+    assert extra["positions"]                  # executable weights set (top-carry long)
+
+
+def test_second_advance_books_delta_neutral_return():
+    day1 = _snap({"A": 0.002, "B": 0.001, "C": 0.0, "D": -0.001, "E": -0.002})
+    # day2: A's perp falls to 99 (short-perp gains) and spot flat -> basis P&L positive; funding accrues
+    day2 = {"A": (0.002, 0.002, 99.0, 100.0, 5e6), "B": (0.001, 0.001, 100.0, 100.0, 5e6),
+            "C": (0.0, 0.0, 100.0, 100.0, 5e6), "D": (-0.001, -0.001, 100.0, 100.0, 5e6),
+            "E": (-0.002, -0.002, 100.0, 100.0, 5e6)}
+    live = _FakeLive([day1, day2])
+    strat = XsFundingForward(live, quantile=0.4, clock=lambda: "2026-06-20")
+    _, extra = strat.advance(None, {})         # inception (day1)
+    rows, extra = strat.advance("2026-06-19", extra)
+    assert len(rows) == 1 and rows[0][0] == "2026-06-20"
+    assert rows[0][1] > 0.0                     # A long-carry: short-perp gain (99/100) + funding
+```
+
+- [ ] **Step 3** — run → FAIL.
+
+- [ ] **Step 4** — implement `src/fxhnt/application/xsfunding_strategy.py`. Use the cross-sectional scoring (robust_z of trailing funding across the snapshot coins) + `construction_weights(scores, "executable", quantile, borrowable)` (borrowable = high-qvol coins) + book the prior book's delta-neutral return `Σ w·[(spot/spot_prev−1) − (perp/perp_prev−1) + today_funding] − turnover·cost`. State (`extra`): `{positions: {sym: w}, perp: {sym: px}, spot: {sym: px}, last_date}`. On inception (no prior positions) return `[]`. Reuse `robust_z` from equity_factor (or `funding_score` adapted to the snapshot). Daily rebalance. cost = a flat `cost_bps` (default 8bp) on turnover (live track doesn't need the liquidity-slip model — keep simple + documented).
+```python
+"""Live-booking cross-sectional funding-dispersion paper track (executable construction).
+Books the REAL delta-neutral return (funding + spot-perp basis) forward each day — so the
+paper record experiences the basis tail the backtest had to guard out. Mirrors the
+equity-factor live tracks' ForwardStrategy contract."""
+from __future__ import annotations
+
+from typing import Any, Callable
+
+from fxhnt.domain.strategies.equity_factor import robust_z
+from fxhnt.domain.cross_sectional_funding import construction_weights
+
+
+def _today() -> str:
+    import datetime as dt
+    return dt.date.today().isoformat()
+
+
+class XsFundingForward:
+    def __init__(self, live: Any, *, quantile: float = 0.2, cost_bps: float = 8.0,
+                 borrowable_qvol: float = 2e7, clock: Callable[[], str] = _today) -> None:
+        self._live, self._q, self._cost = live, quantile, cost_bps / 1e4
+        self._bq, self._clock = borrowable_qvol, clock
+
+    def advance(self, last_date: str | None, extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]:
+        today = self._clock()
+        prev_pos: dict[str, float] = extra.get("positions", {})
+        prev_perp: dict[str, float] = extra.get("perp", {})
+        prev_spot: dict[str, float] = extra.get("spot", {})
+        snap = self._live.snapshot(prev_pos)            # {sym: (tf, today_funding, perp, spot, qvol)}
+        syms = sorted(snap)
+        # book prior book's realized delta-neutral return
+        rows: list[tuple[str, float]] = []
+        if prev_pos:
+            realized = 0.0
+            for s, w in prev_pos.items():
+                if s in snap and s in prev_perp and s in prev_spot and prev_perp[s] > 0 and prev_spot[s] > 0:
+                    _, tf_today, perp, spot, _ = snap[s]
+                    spot_ret = spot / prev_spot[s] - 1.0
+                    perp_ret = perp / prev_perp[s] - 1.0
+                    realized += w * ((spot_ret - perp_ret) + tf_today)
+            # turnover cost vs the NEW book computed below — compute weights first
+            # (we charge cost on the rebalance into the new book)
+            ...
+        # new executable weights from trailing funding cross-section
+        tf = [snap[s][0] for s in syms]
+        scores = {syms[i]: z for i, z in enumerate(robust_z(tf))}
+        borrowable = {s for s in syms if snap[s][4] >= self._bq}
+        new_w = construction_weights(scores, "executable", quantile=self._q, borrowable=borrowable)
+        if prev_pos:
+            turnover = sum(abs(new_w.get(s, 0.0) - prev_pos.get(s, 0.0)) for s in set(new_w) | set(prev_pos))
+            realized -= turnover * self._cost / 2.0
+            rows.append((today, realized))
+        extra = {"positions": new_w,
+                 "perp": {s: snap[s][2] for s in new_w},
+                 "spot": {s: snap[s][3] for s in new_w},
+                 "last_date": today}
+        return rows, extra
+```
+(Clean up the `...` placeholder — fold the turnover cost after weights are computed, as shown in the bottom block; the top `if prev_pos:` block computes only the carry+basis realized, then after weights the turnover cost is subtracted and the row appended. Restructure so it's linear and the two unit tests pass.)
+
+- [ ] **Step 5** — run → PASS; mypy → no errors; full suite → PASS.
+
+- [ ] **Step 6** — commit:
+```bash
+git add src/fxhnt/application/xsfunding_strategy.py tests/integration/test_xsfunding_strategy.py
+git commit -m "feat(xsfunding): live-booking forward strategy (delta-neutral funding+basis, executable)
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 3: registry + Dagster wiring + verdict surface
+
+**Files:** Modify `src/fxhnt/registry.py`, `src/fxhnt/adapters/orchestration/assets.py`, `definitions.py`; Test `tests/integration/test_orchestration_definitions.py` (update count + xsfunding present).
+
+- [ ] **Step 1** — add to `STRATEGY_REGISTRY` (mirror an eqfactor entry):
+```python
+    "xsfunding": {
+        "display_name": "Crypto funding — cross-sectional", "sleeve": "crypto-funding",
+        "venue": "binance-perp", "state_file": "xsfunding_state",
+        "gate_spec": {"min_days": 60, "min_total_return": 0.0, "min_sharpe": 0.5},
+    },
+```
+
+- [ ] **Step 2** — add a Dagster asset `xsfunding_nav` in `assets.py` (mirror `eqfactor_long_nav`): construct `XsFundingForward(BinanceXsFundingLive())` wrapped in `ForwardTracker(..., f"{_data_dir()}/xsfunding_state.json").step()`, log forward_days. Add `xsfunding_nav` to `cockpit_forward`'s deps. Register in `definitions.py` (import + combined_book_job selection + Definitions assets list). Update `test_orchestration_definitions.py` (asset count +1, assert `xsfunding_nav` present).
+
+- [ ] **Step 3** — run the orchestration test + full suite → PASS; mypy → no new errors; confirm `defs` loads (`python -c "from fxhnt.adapters.orchestration.definitions import defs"`).
+
+- [ ] **Step 4** — commit:
+```bash
+git add src/fxhnt/registry.py src/fxhnt/adapters/orchestration/assets.py src/fxhnt/adapters/orchestration/definitions.py tests/integration/test_orchestration_definitions.py
+git commit -m "feat(xsfunding): register xsfunding paper track + Dagster asset in daily book
+
+Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
+```
+
+---
+
+## Task 4: deploy + surface verdict (controller)
+- [ ] Push fxhnt master; rebuild cockpit image via Argo `fxhnt-cockpit` at HEAD; wait for Succeeded.
+- [ ] Persist the xsfunding backtest verdict (executable, net-of-basis) to `backtest_summary` keyed `xsfunding` — extend `backtest_ingest._CONSTRUCTION_TO_SID` mapping or run a one-off upsert from the basis report's `executable@5e6` cell (sharpe ~6.3, but record with the honest caveat). Simplest: a tiny one-off that reads `/tmp/funding_basis_report.json` executable cell → `upsert_backtest_summary(BacktestSummary(strategy_id="xsfunding", ...))` against the cockpit pg (in-cluster Job or local+port-forward). 
+- [ ] Verify: cockpit fleet shows `xsfunding` (gate=WAIT initially); the daily job materializes `xsfunding_nav`; backtest verdict visible. Trigger one manual materialization to seed inception.
+
+## Final review (after Task 3) + honest note
+Final review over the whole diff; full suite + mypy. The paper track will accumulate the REAL net-of-everything record (incl. the basis tail the backtest guarded out + execution reality) — gate=WAIT until weeks of forward data. Do NOT oversell the backtest Sharpe in the cockpit; the surfaced verdict carries the funding-research-consistent ~3-5 deployable expectation.

docs/superpowers/plans/2026-06-21-crypto-tstrend-sleeve.md

@@ -0,0 +1,582 @@
+# Crypto TS-Trend Paper-Track Sleeve — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add a `crypto_tstrend` live paper-track sleeve (long/flat crypto TSMOM) that books daily paper NAV via the nightly Dagster combined-book job, with a configurable portfolio vol-target overlay and a binary Page-Hinkley edge-decay kill, gated `WAIT` until forward-proven.
+
+**Architecture:** Recomputable-series `ForwardStrategy` (mirrors `CombinedBookStrategy`) that reuses the validated `TrendRunner.run()` unchanged on the existing `crypto_pit` panel, then applies a causal portfolio vol-target overlay + a `PageHinkleyDecay` binary kill incrementally over new days (state carried in the tracker's `extra`). No new data adapter.
+
+**Tech Stack:** Python 3, pytest, numpy, Dagster 1.12, fxhnt (`TrendRunner`, `ForwardTracker`, `CryptoPitPanelSource`, `STRATEGY_REGISTRY`).
+
+**Spec:** `docs/superpowers/specs/2026-06-21-crypto-tstrend-sleeve-design.md`
+
+---
+
+## File Structure
+
+- **Create** `src/fxhnt/domain/edge_decay.py` — `PageHinkleyDecay` (pure, reusable mean-shift detector).
+- **Create** `src/fxhnt/application/ts_trend_strategy.py` — `TsTrendForward` (recomputable ForwardStrategy + overlay + kill).
+- **Modify** `src/fxhnt/registry.py` — add `crypto_tstrend` to `STRATEGY_REGISTRY`.
+- **Modify** `src/fxhnt/adapters/orchestration/assets.py` — add `crypto_tstrend_nav` asset; add it to `cockpit_forward` deps.
+- **Modify** `src/fxhnt/adapters/orchestration/definitions.py` — import + add to `combined_book_job.selection` + `defs.assets`.
+- **Create** `tests/unit/test_edge_decay.py`, `tests/integration/test_tstrend_strategy.py`, `tests/unit/test_registry_tstrend.py`, `tests/integration/test_tstrend_wiring.py`.
+
+---
+
+## Task 0: Branch
+
+- [ ] **Step 1: Create a feature branch** (fxhnt is on `master`; never commit feature work to the default branch)
+
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git checkout -b feat/crypto-tstrend-sleeve
+```
+
+- [ ] **Step 2: Commit the spec** (written during brainstorming, currently uncommitted)
+
+```bash
+git add docs/superpowers/specs/2026-06-21-crypto-tstrend-sleeve-design.md docs/superpowers/plans/2026-06-21-crypto-tstrend-sleeve.md
+git commit -m "docs: crypto TS-trend sleeve spec + implementation plan"
+```
+
+---
+
+## Task 1: PageHinkleyDecay edge-decay detector
+
+**Files:**
+- Create: `src/fxhnt/domain/edge_decay.py`
+- Test: `tests/unit/test_edge_decay.py`
+
+- [ ] **Step 1: Write the failing tests**
+
+```python
+# tests/unit/test_edge_decay.py
+import json
+
+import numpy as np
+
+from fxhnt.domain.edge_decay import PageHinkleyDecay
+
+
+def test_stationary_positive_never_kills():
+    ph = PageHinkleyDecay(lam=0.05, reenter_days=10)
+    rng = np.random.default_rng(0)
+    alive = [ph.update(0.002 + 0.0005 * float(rng.standard_normal())) for _ in range(500)]
+    assert all(alive)
+    assert not ph.killed
+
+
+def test_mean_flip_kills():
+    ph = PageHinkleyDecay(lam=0.05, reenter_days=10)
+    for _ in range(200):
+        ph.update(0.002)
+    killed_at = None
+    for i in range(200):
+        if not ph.update(-0.01):
+            killed_at = i
+            break
+    assert killed_at is not None
+    assert ph.killed
+
+
+def test_rearm_after_hysteresis():
+    ph = PageHinkleyDecay(lam=0.01, reenter_days=5)
+    for _ in range(50):
+        ph.update(0.001)
+    for _ in range(50):
+        if not ph.update(-0.02):
+            break
+    assert ph.killed
+    states = [ph.update(0.0) for _ in range(5)]
+    assert states[-1] is True
+    assert not ph.killed
+
+
+def test_to_from_dict_roundtrip():
+    ph = PageHinkleyDecay(lam=0.05)
+    for _ in range(20):
+        ph.update(0.001)
+    d = json.loads(json.dumps(ph.to_dict()))
+    ph2 = PageHinkleyDecay.from_dict(d)
+    assert ph2.to_dict() == ph.to_dict()
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest tests/unit/test_edge_decay.py -v`
+Expected: FAIL — `ModuleNotFoundError: No module named 'fxhnt.domain.edge_decay'`
+
+- [ ] **Step 3: Implement `PageHinkleyDecay`**
+
+```python
+# src/fxhnt/domain/edge_decay.py
+"""Page-Hinkley mean-SHIFT edge-decay detector with binary kill + hysteresis. Pure, no I/O.
+
+Detects a sustained DOWNWARD shift in a return stream's mean (slow-bleed edge death a drawdown
+trigger misses). Shift-form (not level — level froze during normal operation in the RL context;
+see pearl_edge_decay_detector_phase1_validated_train_also_decays). Consumes the SHADOW (unkilled)
+return so it can detect recovery and re-arm while booked exposure is flat.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass
+class PageHinkleyDecay:
+    delta: float = 0.0        # deadband below the running mean before drift accumulates
+    lam: float = 0.05         # cumulative-deviation threshold to KILL (pre-registered, conservative)
+    reenter_days: int = 10    # consecutive non-negative shadow returns required to re-arm
+    n: int = 0
+    mean: float = 0.0
+    m_t: float = 0.0
+    m_max: float = 0.0
+    killed: bool = False
+    recover: int = 0
+
+    def update(self, ret: float) -> bool:
+        """Feed one realized SHADOW daily return; return True if ALIVE (book), False if KILLED (flat)."""
+        if self.killed:
+            self.recover = self.recover + 1 if ret >= 0.0 else 0
+            if self.recover >= self.reenter_days:
+                self.n, self.mean, self.m_t, self.m_max = 0, 0.0, 0.0, 0.0
+                self.killed, self.recover = False, 0
+            return not self.killed
+        self.n += 1
+        self.mean += (ret - self.mean) / self.n
+        self.m_t += (ret - self.mean + self.delta)
+        if self.m_t > self.m_max:
+            self.m_max = self.m_t
+        if (self.m_max - self.m_t) > self.lam:
+            self.killed, self.recover = True, 0
+            return False
+        return True
+
+    def to_dict(self) -> dict:
+        return {"delta": self.delta, "lam": self.lam, "reenter_days": self.reenter_days,
+                "n": self.n, "mean": self.mean, "m_t": self.m_t, "m_max": self.m_max,
+                "killed": self.killed, "recover": self.recover}
+
+    @classmethod
+    def from_dict(cls, d: dict) -> "PageHinkleyDecay":
+        return cls(**d)
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest tests/unit/test_edge_decay.py -v`
+Expected: PASS (4 passed)
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/fxhnt/domain/edge_decay.py tests/unit/test_edge_decay.py
+git commit -m "feat(domain): PageHinkleyDecay binary edge-decay detector"
+```
+
+---
+
+## Task 2: TsTrendForward strategy
+
+**Files:**
+- Create: `src/fxhnt/application/ts_trend_strategy.py`
+- Test: `tests/integration/test_tstrend_strategy.py`
+
+- [ ] **Step 1: Write the failing tests**
+
+```python
+# tests/integration/test_tstrend_strategy.py
+import json
+import math
+
+import numpy as np
+
+from fxhnt.application.forward_tracker import ForwardTracker
+from fxhnt.application.trend_runner import TrendRunner
+from fxhnt.application.ts_trend_strategy import TsTrendForward, _iso
+
+
+def _coin(n, start_day, drift, seed):
+    rng = np.random.default_rng(seed)
+    rets = drift + 0.01 * rng.standard_normal(n)
+    close = 100.0 * np.cumprod(1.0 + rets)
+    return {int(start_day + i): float(close[i]) for i in range(n)}
+
+
+def _panel(n=160):
+    return {f"C{j}": _coin(n, 0, 0.001, seed=j) for j in range(6)}
+
+
+def _trend_returns(panel):
+    return TrendRunner(panel, lookbacks=(20, 60, 120), vol_window=30, target_vol=0.10,
+                       gross_cap=1.0, long_short=False, cost_bps=10.0,
+                       periods_per_year=365).run()
+
+
+def test_inception_books_nothing_and_warms_state(tmp_path):
+    panel = _panel()
+    strat = TsTrendForward(lambda: panel, book_target_vol=0.15)
+    path = str(tmp_path / "st.json")
+    status = ForwardTracker(strat, path).step()
+    assert status.forward_days == 0                         # inception books nothing
+    state = json.load(open(path))
+    assert state["extra"]["ph"]["killed"] is False          # fresh, alive (history can't pre-kill)
+    assert len(state["extra"]["vol_buf"]) > 0               # overlay warm-started
+    assert state["inception"] == state["last_date"]
+
+
+def test_subsequent_books_lev_times_raw_no_drift(tmp_path):
+    panel1 = _panel()
+    holder = {"p": panel1}
+    # decay_lam high → never kills in this test; isolates the overlay + no-drift property
+    strat = TsTrendForward(lambda: holder["p"], book_target_vol=0.15, lev_cap=3.0, decay_lam=1e9)
+    path = str(tmp_path / "st.json")
+    ForwardTracker(strat, path).step()                      # inception
+    vb = json.load(open(path))["extra"]["vol_buf"]
+    # extend the panel by exactly one day per coin
+    last = max(next(iter(panel1.values())).keys())
+    panel2 = {s: dict(series) for s, series in panel1.items()}
+    for s in panel2:
+        panel2[s][last + 1] = panel2[s][last] * 1.002
+    holder["p"] = panel2
+    status = ForwardTracker(strat, path).step()
+    assert status.booked_today == 1
+    res = _trend_returns(panel2)
+    raw_last = float(res.returns[-1])
+    date_last = _iso(int(res.dates[-1]))
+    tv = float(np.std(vb))
+    lev = min(3.0, (0.15 / math.sqrt(365)) / tv) if tv > 1e-12 else 1.0
+    booked = next(d["ret"] for d in json.load(open(path))["days"] if d["date"] == date_last)
+    assert booked == lev * raw_last                         # booked == lev*raw AND raw == TrendRunner (no drift)
+
+
+def test_killed_state_flattens_booking(tmp_path):
+    panel1 = _panel()
+    holder = {"p": panel1}
+    strat = TsTrendForward(lambda: holder["p"], book_target_vol=0.15)
+    path = str(tmp_path / "st.json")
+    ForwardTracker(strat, path).step()                      # inception
+    # force the carried PH into a killed, non-recovering state
+    state = json.load(open(path))
+    state["extra"]["ph"]["killed"] = True
+    state["extra"]["ph"]["recover"] = 0
+    json.dump(state, open(path, "w"))
+    last = max(next(iter(panel1.values())).keys())
+    panel2 = {s: dict(series) for s, series in panel1.items()}
+    for s in panel2:
+        panel2[s][last + 1] = panel2[s][last] * 0.99        # a down day (keeps it killed)
+    holder["p"] = panel2
+    status = ForwardTracker(strat, path).step()
+    assert status.booked_today == 1
+    state2 = json.load(open(path))
+    assert state2["days"][-1]["ret"] == 0.0                 # flattened while killed
+    assert len(state2["extra"]["vol_buf"]) > 0              # shadow buffer still advances
+
+
+def test_extra_json_roundtrips(tmp_path):
+    panel = _panel()
+    strat = TsTrendForward(lambda: panel, book_target_vol=0.15)
+    path = str(tmp_path / "st.json")
+    ForwardTracker(strat, path).step()
+    state = json.load(open(path))
+    json.dumps(state["extra"])                              # must not raise
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest tests/integration/test_tstrend_strategy.py -v`
+Expected: FAIL — `ModuleNotFoundError: No module named 'fxhnt.application.ts_trend_strategy'`
+
+- [ ] **Step 3: Implement `TsTrendForward`**
+
+```python
+# src/fxhnt/application/ts_trend_strategy.py
+"""Live-booking crypto TS-trend (long/flat TSMOM) paper track — recomputable series.
+
+Reuses the validated TrendRunner unchanged on the existing crypto_pit panel, then applies a CAUSAL
+portfolio vol-target overlay (the configurable book-vol knob) and a binary PageHinkleyDecay kill
+incrementally over new days, carrying overlay-buffer + detector state in the tracker's `extra`.
+Mirrors the ForwardStrategy contract: advance(last_date, extra) -> (rows, extra)."""
+from __future__ import annotations
+
+import datetime as dt
+import math
+from typing import Any, Callable
+
+import numpy as np
+
+from fxhnt.application.trend_runner import TrendRunner
+from fxhnt.domain.edge_decay import PageHinkleyDecay
+
+_EPOCH = dt.date(1970, 1, 1)
+
+
+def _today() -> str:
+    return dt.date.today().isoformat()
+
+
+def _iso(epoch_day: int) -> str:
+    return (_EPOCH + dt.timedelta(days=int(epoch_day))).isoformat()
+
+
+class TsTrendForward:
+    def __init__(self, load_panel: Callable[[], dict[str, dict[int, float]]], *,
+                 lookbacks: tuple[int, ...] = (20, 60, 120), vol_window: int = 30,
+                 internal_target_vol: float = 0.10, gross_cap: float = 1.0,
+                 cost_bps: float = 10.0, periods_per_year: int = 365,
+                 book_target_vol: float = 0.15, lev_cap: float = 3.0, vol_buf_window: int = 30,
+                 decay_lam: float = 0.05, decay_reenter_days: int = 10,
+                 clock: Callable[[], str] = _today) -> None:
+        self._load_panel = load_panel
+        self._lb = tuple(lookbacks)
+        self._vw = vol_window
+        self._itv = internal_target_vol
+        self._gcap = gross_cap
+        self._cost = cost_bps
+        self._ppy = periods_per_year
+        self._btv = book_target_vol
+        self._levcap = lev_cap
+        self._bufw = vol_buf_window
+        self._lam = decay_lam
+        self._reenter = decay_reenter_days
+        self._clock = clock
+
+    def _raw_series(self) -> tuple[list[int], list[float]]:
+        panel = self._load_panel()
+        res = TrendRunner(panel, lookbacks=self._lb, vol_window=self._vw, target_vol=self._itv,
+                          gross_cap=self._gcap, long_short=False, cost_bps=self._cost,
+                          periods_per_year=self._ppy).run()
+        return [int(d) for d in res.dates], [float(r) for r in res.returns]
+
+    def advance(self, last_date: str | None,
+                extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]:
+        dates, raw = self._raw_series()
+
+        if not extra:                                       # INCEPTION: warm overlay, fresh PH, book nothing
+            vol_buf = raw[-self._bufw:] if raw else []
+            ph = PageHinkleyDecay(lam=self._lam, reenter_days=self._reenter)
+            rows = [(_iso(d), r) for d, r in zip(dates, raw)]   # tracker freezes inception at latest, books none
+            new_extra = {"ph": ph.to_dict(), "vol_buf": vol_buf,
+                         "last_day": (int(dates[-1]) if dates else None)}
+            return rows, new_extra
+
+        ph = PageHinkleyDecay.from_dict(extra["ph"])
+        vol_buf = [float(x) for x in extra.get("vol_buf", [])]
+        cutoff = last_date or ""
+        btv_daily = self._btv / math.sqrt(self._ppy)
+        rows: list[tuple[str, float]] = []
+        for d, r in zip(dates, raw):
+            iso = _iso(d)
+            if iso <= cutoff:                               # already booked / pre-inception
+                continue
+            tv = float(np.std(vol_buf)) if len(vol_buf) >= 2 else 0.0
+            lev = min(self._levcap, btv_daily / tv) if tv > 1e-12 else 1.0
+            alive = ph.update(r)                            # PH consumes shadow raw return
+            rows.append((iso, lev * r if alive else 0.0))
+            vol_buf.append(r)                               # shadow buffer always advances
+            if len(vol_buf) > self._bufw:
+                vol_buf = vol_buf[-self._bufw:]
+        new_extra = {"ph": ph.to_dict(), "vol_buf": vol_buf,
+                     "last_day": (int(dates[-1]) if dates else extra.get("last_day"))}
+        return rows, new_extra
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest tests/integration/test_tstrend_strategy.py -v`
+Expected: PASS (5 passed)
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/fxhnt/application/ts_trend_strategy.py tests/integration/test_tstrend_strategy.py
+git commit -m "feat(application): TsTrendForward live paper-track strategy (overlay + decay kill)"
+```
+
+---
+
+## Task 3: Registry entry
+
+**Files:**
+- Modify: `src/fxhnt/registry.py` (inside the `STRATEGY_REGISTRY` dict)
+- Test: `tests/unit/test_registry_tstrend.py`
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+# tests/unit/test_registry_tstrend.py
+from fxhnt.registry import STRATEGY_REGISTRY
+
+
+def test_crypto_tstrend_registered():
+    e = STRATEGY_REGISTRY["crypto_tstrend"]
+    assert e["state_file"] == "crypto_tstrend_state"
+    assert e["sleeve"] == "crypto-trend"
+    assert e["venue"] == "binance-perp"
+    assert e["gate_spec"] == {"min_days": 60, "min_total_return": 0.0, "min_sharpe": 0.5}
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest tests/unit/test_registry_tstrend.py -v`
+Expected: FAIL — `KeyError: 'crypto_tstrend'`
+
+- [ ] **Step 3: Add the registry entry**
+
+In `src/fxhnt/registry.py`, add this entry inside the `STRATEGY_REGISTRY` dict, immediately after the `"unlock"` entry:
+
+```python
+    "crypto_tstrend": {
+        "display_name": "Crypto TS-momentum (long/flat daily trend, 20/60/120d)",
+        "sleeve": "crypto-trend",
+        "venue": "binance-perp", "state_file": "crypto_tstrend_state",
+        "gate_spec": {"min_days": 60, "min_total_return": 0.0, "min_sharpe": 0.5},
+    },
+```
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest tests/unit/test_registry_tstrend.py -v`
+Expected: PASS
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/fxhnt/registry.py tests/unit/test_registry_tstrend.py
+git commit -m "feat(registry): register crypto_tstrend sleeve (gate WAIT 60d)"
+```
+
+---
+
+## Task 4: Dagster wiring
+
+**Files:**
+- Modify: `src/fxhnt/adapters/orchestration/assets.py` (add `crypto_tstrend_nav`; add to `cockpit_forward` deps)
+- Modify: `src/fxhnt/adapters/orchestration/definitions.py` (import + job selection + `defs.assets`)
+- Test: `tests/integration/test_tstrend_wiring.py`
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+# tests/integration/test_tstrend_wiring.py
+def test_asset_importable():
+    from fxhnt.adapters.orchestration.assets import crypto_tstrend_nav
+    assert crypto_tstrend_nav is not None
+
+
+def test_definitions_load_and_include_tstrend():
+    # Importing `defs` constructs Definitions(...), which validates the asset graph — a missing
+    # crypto_tstrend_nav (referenced by cockpit_forward's deps) would raise here.
+    from fxhnt.adapters.orchestration.assets import crypto_tstrend_nav
+    from fxhnt.adapters.orchestration.definitions import combined_book_job, defs
+    assert crypto_tstrend_nav in defs.assets          # in the Definitions assets list
+    assert combined_book_job is not None              # job built without error
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest tests/integration/test_tstrend_wiring.py -v`
+Expected: FAIL — `ImportError: cannot import name 'crypto_tstrend_nav'`
+
+- [ ] **Step 3: Add the asset in `assets.py`**
+
+Add this asset definition immediately after the `xsfunding_nav` asset (around line 188):
+
+```python
+@asset(deps=[crypto_bars])   # dep on crypto_bars so the crypto_pit panel is fresh before we read it
+def crypto_tstrend_nav(context: AssetExecutionContext) -> dict:
+    """Paper-forward crypto TS-momentum (long/flat daily TSMOM on the survivorship-free crypto_pit panel)."""
+    from fxhnt.adapters.data.crypto_pit_panel import CryptoPitPanelSource
+    from fxhnt.application.ts_trend_strategy import TsTrendForward
+    s = get_settings()
+
+    def load_panel() -> dict:
+        raw = CryptoPitPanelSource(s.crypto_pit_dir).panel()    # {sym: {day: (close, qvol, funding)}}
+        return {sym: {d: v[0] for d, v in series.items()} for sym, series in raw.items()}
+
+    return _run_paper_tracker(
+        context, "crypto_tstrend_nav",
+        lambda: TsTrendForward(load_panel, book_target_vol=0.15),
+        "crypto_tstrend_state",
+    )
+```
+
+- [ ] **Step 4: Add `crypto_tstrend_nav` to the `cockpit_forward` deps**
+
+In `assets.py` change the `cockpit_forward` decorator (line ~207) from:
+
+```python
+@asset(deps=[combined_forward_nav, sixtyforty_nav, xsfunding_nav, unlock_nav])
+```
+
+to:
+
+```python
+@asset(deps=[combined_forward_nav, sixtyforty_nav, xsfunding_nav, unlock_nav, crypto_tstrend_nav])
+```
+
+- [ ] **Step 5: Wire it in `definitions.py`**
+
+Add `crypto_tstrend_nav,` to the import block from `fxhnt.adapters.orchestration.assets`:
+
+```python
+from fxhnt.adapters.orchestration.assets import (
+    cockpit_forward,
+    combined_forward_nav,
+    crypto_bars,
+    crypto_tstrend_nav,
+    futures_bars,
+    sixtyforty_nav,
+    unlock_nav,
+    xsfunding_nav,
+)
+```
+
+Add `crypto_tstrend_nav,` to BOTH the `combined_book_job` selection and the `defs` assets list (same line as `xsfunding_nav, unlock_nav,`):
+
+```python
+        xsfunding_nav, unlock_nav, crypto_tstrend_nav,
+```
+
+- [ ] **Step 6: Run tests to verify they pass**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest tests/integration/test_tstrend_wiring.py -v`
+Expected: PASS (2 passed)
+
+- [ ] **Step 7: Commit**
+
+```bash
+git add src/fxhnt/adapters/orchestration/assets.py src/fxhnt/adapters/orchestration/definitions.py tests/integration/test_tstrend_wiring.py
+git commit -m "feat(orchestration): wire crypto_tstrend_nav into nightly combined-book job + cockpit"
+```
+
+---
+
+## Task 5: Full verification
+
+- [ ] **Step 1: Run the full test suite**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m pytest -q`
+Expected: all green (the ~467 existing + new tests)
+
+- [ ] **Step 2: Type-check**
+
+Run: `cd /home/jgrusewski/Work/fxhnt && python -m mypy src/fxhnt`
+Expected: clean (no new errors)
+
+- [ ] **Step 3: Final commit (if any fixups were needed)**
+
+```bash
+git add -A
+git commit -m "chore: crypto_tstrend sleeve — green tests + mypy clean"
+```
+
+---
+
+## Notes for the executor
+
+- **Honest falsification:** the `PageHinkleyDecay` unit tests assert no-kill on a *synthetic stationary-positive* series (the defensible no-false-positive guarantee). On the REAL validated series it is expected and CORRECT for PH to kill during the 2022 crypto bear and re-arm in 2023 — that is the feature working, not a bug; do not add a hard assertion that it never kills on real history.
+- **No-drift is the load-bearing property:** `TsTrendForward` calls `TrendRunner.run()` directly, so the live signal IS the validated backtest signal. `test_subsequent_books_lev_times_raw_no_drift` guards this — keep it.
+- **Do not wire the tracker `kill_dd`** for this sleeve (would double-kill with the PH detector). The PH decay is the sleeve's edge-death guard.
+- **`s.crypto_pit_dir`** is the settings field already used by `xsfunding`/momentum assets; if it is absent in `get_settings()`, use the same path expression those assets use (verify against `assets.py` line ~196 / 202).
+```

docs/superpowers/plans/2026-06-21-observability-2e.md

@@ -0,0 +1,216 @@
+# Re-enable Observability + Expose dagster.fxhnt.ai — Implementation Plan (Phase 2E)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> Additive (scale-up + wiring); the proxy edit is gated on `*.fxhnt.ai` staying healthy. Steps use
+> checkbox (`- [ ]`) syntax.
+
+**Goal:** Turn the scaled-to-zero Prometheus+Grafana core back on and expose `grafana.fxhnt.ai` +
+`dagster.fxhnt.ai` over the tailnet proxy.
+
+**Architecture:** `kubectl scale` the operator/kube-state-metrics/grafana back to 1 (operator brings
+prometheus up; alertmanager kept at 0 via its CR); admit the tailscale proxy to grafana + dagster via
+netpols; add a dagster nginx block + DNS record.
+
+**Tech Stack:** kube-prometheus-stack (operator), Grafana, Dagster, nginx tailscale proxy, Scaleway DNS.
+
+**Repo:** `~/Work/fxhnt` (IaC home). **Spec:** `docs/superpowers/specs/2026-06-21-observability-2e-design.md`.
+**Branch:** `chore/observability-2e` (already created).
+
+---
+
+## Task 1: Re-enable core metrics (prometheus + alertmanager + grafana + kube-state-metrics)
+
+- [ ] **Step 1: Scale up the normal deployments**
+```bash
+kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=1
+```
+Expected: 3 deployments scaled. The operator then reconciles the Prometheus **and Alertmanager** CRs
+(both `replicas=1`) → brings both statefulsets up automatically (no manual scale needed for them).
+
+- [ ] **Step 2: Wait for the operator to bring prometheus + alertmanager up + pods ready**
+```bash
+i=0; until kubectl get sts prometheus-prometheus-stack-kube-prom-prometheus -n foxhunt -o jsonpath='{.status.readyReplicas}' 2>/dev/null | grep -q 1 || [ $i -ge 36 ]; do sleep 5; i=$((i+1)); done
+kubectl get pods -n foxhunt | grep -iE "prometheus-stack-kube-prom-prometheus|alertmanager|kube-state-metrics|grafana|kube-prom-operator" | grep -v node-exporter | awk '{print $1,$3}'
+```
+Expected: `prometheus-...-0`, `alertmanager-...-0`, `grafana`, `kube-state-metrics`, `kube-prom-operator`
+pods Running. (If alertmanager stays at 0, its CR replicas may be 0 → `kubectl patch alertmanager
+prometheus-stack-kube-prom-alertmanager -n foxhunt --type merge -p '{"spec":{"replicas":1}}'`.)
+
+---
+
+## Task 2: Expose grafana.fxhnt.ai (admit the proxy)
+
+**Files:** Create `infra/k8s/monitoring/grafana-netpol.yaml`
+
+- [ ] **Step 1: Add a netpol admitting the tailscale proxy → grafana:3000 (grafana pod port)**
+
+First find grafana's pod port:
+```bash
+kubectl get deploy grafana -n foxhunt -o jsonpath='{.spec.template.spec.containers[0].ports[*].containerPort}{"\n"}'
+```
+(Grafana listens on 3000 in-pod; svc maps 80→3000.) Create `infra/k8s/monitoring/grafana-netpol.yaml`:
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: grafana-proxy-ingress
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: grafana
+  policyTypes: [Ingress]
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tailscale-gitlab-proxy
+      ports:
+        - { port: 3000, protocol: TCP }
+```
+(If Step-1 shows a different containerPort, use that.)
+
+- [ ] **Step 2: Apply + verify grafana.fxhnt.ai**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+kubectl apply -f infra/k8s/monitoring/grafana-netpol.yaml 2>&1 | tail -1
+curl -sk -m 12 -o /dev/null -w "grafana.fxhnt.ai: %{http_code}\n" https://grafana.fxhnt.ai/login 2>&1
+```
+Expected: `200` (Grafana login). **If 502/000:** grafana pod not ready yet, or the proxy can't reach it —
+check the grafana label matches the netpol `podSelector` and the proxy nginx grafana block points at
+`grafana.foxhunt.svc:80`.
+
+- [ ] **Step 3: Verify Prometheus datasource works (grafana → prometheus in-cluster)**
+```bash
+GP=$(kubectl get secret grafana-admin -n foxhunt -o jsonpath='{.data.admin-password}'|base64 -d 2>/dev/null)
+curl -sk -u "admin:$GP" "https://grafana.fxhnt.ai/api/datasources" 2>/dev/null | python3 -c "import sys,json;[print(d['name'],d['type'],d['url']) for d in json.load(sys.stdin)]" 2>/dev/null || echo "(check datasource manually in UI)"
+```
+Expected: a Prometheus datasource listed. **If grafana can't reach prometheus**, add an egress allow
+(grafana → prometheus:9090) — only if a datasource health check fails.
+
+---
+
+## Task 3: Expose dagster.fxhnt.ai
+
+**Files:** Modify `infra/k8s/orchestration/dagster.yaml` (its NetworkPolicy), `infra/k8s/gitlab/tailscale-proxy.yaml`
+
+- [ ] **Step 1: Add an Ingress rule to the dagster NetworkPolicy (in dagster.yaml)**
+
+The dagster NetworkPolicy (in `dagster.yaml`) is currently `policyTypes: [Egress]` only. Add `Ingress`
+and a rule admitting the proxy → the webserver pod port `3000`:
+```yaml
+  policyTypes: [Ingress, Egress]
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tailscale-gitlab-proxy
+      ports:
+        - { port: 3000, protocol: TCP }
+```
+(Keep the existing `egress:` block. The dagster webserver listens on pod port 3000; svc maps 80→3000.)
+
+- [ ] **Step 2: Add the dagster nginx server block (with WebSocket upgrade)**
+
+In `infra/k8s/gitlab/tailscale-proxy.yaml`, after the Grafana server block, add:
+```
+    # Dagster — dagster.fxhnt.ai
+    server {
+        listen 443 ssl;
+        server_name dagster.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        location / {
+            proxy_pass http://dagster.foxhunt.svc.cluster.local:80;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 3600s;
+        }
+    }
+```
+
+- [ ] **Step 3: Create the Scaleway DNS A record `dagster → 100.95.225.27`**
+```bash
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+scw dns record add fxhnt.ai name=dagster type=A data=100.95.225.27 ttl=300 2>&1 | tail -2
+```
+Expected: record created. (Mirror via the dns terragrunt module later if you want it IaC-managed — out of scope here.)
+
+- [ ] **Step 4: Apply the netpol + proxy, restart proxy, GATE on platform health**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+kubectl apply -f infra/k8s/orchestration/dagster.yaml 2>&1 | grep -iE "networkpolicy|configured" | tail -2
+kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml >/dev/null 2>&1
+kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
+kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s 2>&1 | tail -1
+# Cilium can hold a stale endpoint after rollout — force-recreate the proxy pod (proven pattern)
+kubectl delete pod -n foxhunt -l app.kubernetes.io/name=tailscale-gitlab-proxy --force --grace-period=0 2>&1 | tail -1
+kubectl wait --for=condition=ready pod -n foxhunt -l app.kubernetes.io/name=tailscale-gitlab-proxy --timeout=120s 2>&1 | tail -1
+for h in git dashboard grafana dagster; do curl -sk -m 12 -o /dev/null -w "  $h.fxhnt.ai: %{http_code}\n" https://$h.fxhnt.ai/ 2>&1; done
+```
+Expected: `git`/`dashboard` 200 (not regressed); `grafana` 200; `dagster` 200/302. **STOP + revert proxy if
+git/dashboard go 000.**
+
+---
+
+## Task 4: Commit + push + merge
+
+- [ ] **Step 1: Commit fxhnt + push branch**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add infra/k8s/monitoring/grafana-netpol.yaml infra/k8s/orchestration/dagster.yaml infra/k8s/gitlab/tailscale-proxy.yaml
+git commit -m "feat(infra): re-enable prometheus+grafana, expose grafana.fxhnt.ai + dagster.fxhnt.ai (Phase 2E)
+
+- scaled prometheus-stack operator/kube-state-metrics + grafana back to 1; operator brings prometheus + alertmanager up
+- grafana-netpol: admit tailscale proxy -> grafana:3000 (was 502)
+- dagster.yaml netpol: + ingress proxy -> :3000; nginx dagster.fxhnt.ai block (ws); DNS A record
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/observability-2e 2>&1 | tail -2
+```
+
+- [ ] **Step 2: Merge → master + push**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git checkout master && git merge --no-ff chore/observability-2e -m "Merge: re-enable observability + expose grafana/dagster (Phase 2E)" 2>&1 | tail -2
+git branch -d chore/observability-2e
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1
+```
+
+---
+
+## Task 5: Final verification
+```bash
+echo "=== pods (metrics core + alertmanager up; loki/tempo absent) ==="
+kubectl get pods -n foxhunt | grep -iE "prometheus|alertmanager|grafana|kube-state" | grep -v node-exporter | awk '{print $1,$3}'
+echo "=== UIs ==="
+GP=$(kubectl get secret grafana-admin -n foxhunt -o jsonpath='{.data.admin-password}'|base64 -d 2>/dev/null)
+curl -sk -m 12 -o /dev/null -w "grafana login: %{http_code}\n" https://grafana.fxhnt.ai/login
+curl -sk -m 12 -o /dev/null -w "dagster: %{http_code}\n" https://dagster.fxhnt.ai/
+echo "=== prometheus targets healthy (via grafana datasource proxy) ==="
+curl -sk -u "admin:$GP" "https://grafana.fxhnt.ai/api/datasources" 2>/dev/null | grep -o '"type":"prometheus"' | head -1
+```
+Expected: prometheus/grafana/kube-state pods Running; grafana 200; dagster 200/302; prometheus datasource present.
+
+---
+
+## Rollback
+- Proxy: `git checkout infra/k8s/gitlab/tailscale-proxy.yaml` + apply + restart.
+- Metrics: `kubectl scale deploy prometheus-stack-kube-prom-operator prometheus-stack-kube-state-metrics grafana -n foxhunt --replicas=0` (back to dormant).
+
+## Acceptance criteria (from spec)
+- prometheus/kube-state-metrics/operator/alertmanager/grafana Running; loki/tempo stay 0. ✅ T1, T5
+- grafana.fxhnt.ai 200 + prometheus datasource. ✅ T2, T5
+- dagster.fxhnt.ai 200 (UI). ✅ T3, T5
+- git/dashboard still 200. ✅ T3, T5
+- committed + pushed; DNS record created. ✅ T3, T4

docs/superpowers/plans/2026-06-21-stablecoin-rotation-sleeve.md

@@ -0,0 +1,421 @@
+# Stablecoin Peg-Reversion (Treasury-Rotation) Sleeve — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Add a `stablecoin_rotation` live paper-track sleeve — a daily-rebalanced SHORT-rich stablecoin peg-reversion book — booking paper NAV nightly via Dagster, gated WAIT until forward-proven.
+
+**Architecture:** Recomputable-series `ForwardStrategy` (mirrors `crypto_tstrend`): each night recompute the daily short-rich book from fresh Binance spot daily closes (`BinanceSpotHistory`), apply a `PageHinkleyDecay` kill, book only new days. Short-rich only (long-cheap = death-spiral trap, excluded).
+
+**Tech Stack:** Python 3, pytest, numpy, Dagster 1.12, fxhnt (`ForwardTracker`, `BinanceSpotHistory`, `PageHinkleyDecay`, `STRATEGY_REGISTRY`, `_run_paper_tracker`).
+
+**Spec:** `docs/superpowers/specs/2026-06-21-stablecoin-rotation-sleeve-design.md`
+
+---
+
+## File Structure
+- **Create** `src/fxhnt/domain/stablecoin_reversion.py` — `short_rich_weights` (pure).
+- **Create** `src/fxhnt/application/stablecoin_runner.py` — `StableReversionRunner` + `StableRunResult`.
+- **Create** `src/fxhnt/application/stablecoin_strategy.py` — `StableRotationForward`.
+- **Modify** `src/fxhnt/registry.py` — add `stablecoin_rotation`.
+- **Modify** `src/fxhnt/adapters/orchestration/assets.py` — add `stablecoin_rotation_nav`; add to `cockpit_forward` deps.
+- **Modify** `src/fxhnt/adapters/orchestration/definitions.py` — import + job selection + `defs.assets`.
+- **Modify** `tests/integration/test_orchestration_definitions.py` — asset count 8→9 + assertions.
+- **Create** `tests/unit/test_stablecoin_reversion.py`, `tests/integration/test_stablecoin_strategy.py`, `tests/integration/test_stablecoin_wiring.py`.
+
+---
+
+## Task 0: Branch
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/fxhnt && git checkout -b feat/stablecoin-rotation-sleeve
+git add docs/superpowers/specs/2026-06-21-stablecoin-rotation-sleeve-design.md docs/superpowers/plans/2026-06-21-stablecoin-rotation-sleeve.md
+git commit -m "docs: stablecoin-rotation sleeve spec + implementation plan
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 1: Domain weights + runner
+
+**Files:** Create `src/fxhnt/domain/stablecoin_reversion.py`, `src/fxhnt/application/stablecoin_runner.py`; Tests `tests/unit/test_stablecoin_reversion.py`.
+
+- [ ] **Step 1: Write failing tests** — `tests/unit/test_stablecoin_reversion.py`:
+```python
+import numpy as np
+
+from fxhnt.application.stablecoin_runner import StableReversionRunner
+from fxhnt.domain.stablecoin_reversion import short_rich_weights
+
+
+def test_shorts_only_rich_equal_weight():
+    w = short_rich_weights({"A": 0.006, "B": 0.001, "C": 0.008}, thresh=0.005)
+    assert set(w) == {"A", "C"}
+    assert all(v == -0.5 for v in w.values())          # equal-weight short, gross 1
+
+
+def test_never_longs_cheap_and_empty_when_none_rich():
+    assert short_rich_weights({"A": -0.02, "B": -0.006}, thresh=0.005) == {}   # death-spiral guard
+    assert short_rich_weights({"A": 0.001}, thresh=0.005) == {}
+
+
+def test_runner_books_short_reversion_profit():
+    # S0 rich (1.01) on day 0, reverts to 1.0 on day 1 -> short profits
+    panel = {"S0": {0: 1.01, 1: 1.00}, "S1": {0: 1.00, 1: 1.00}}
+    res = StableReversionRunner(panel, thresh_bp=50.0, cost_bps=4.0).run()
+    assert res.dates == [1]
+    # realized = (-1)*(1.0/1.01 - 1) - turnover(=1)*cost/2 ; cost=4bp
+    expected = -1.0 * (1.00 / 1.01 - 1.0) - 1.0 * (4.0 / 1e4) / 2.0
+    assert abs(res.returns[0] - expected) < 1e-12
+```
+
+- [ ] **Step 2: Run, verify FAIL** — `cd /home/jgrusewski/Work/fxhnt && .venv/bin/python -m pytest tests/unit/test_stablecoin_reversion.py -v` (ModuleNotFoundError).
+
+- [ ] **Step 3a: Implement** `src/fxhnt/domain/stablecoin_reversion.py`:
+```python
+"""Stablecoin peg-reversion weights. Pure, no I/O.
+
+SHORT-rich only: short every stablecoin trading above peg beyond a threshold (mint/redeem arb pulls it
+back). Long-cheap is NEVER taken — buying a depegging stable is the death-spiral trap (UST -> 0)."""
+from __future__ import annotations
+
+
+def short_rich_weights(dev: dict[str, float], thresh: float) -> dict[str, float]:
+    """dev = {sym: close-1.0} as-of a day. Equal-weight SHORT (-1/k each) on names with dev > thresh
+    (rich), gross 1. Returns {} if no name is rich. Cheap names (dev < 0) are never taken."""
+    rich = [s for s, d in dev.items() if d > thresh]
+    if not rich:
+        return {}
+    w = -1.0 / len(rich)
+    return {s: w for s in rich}
+```
+
+- [ ] **Step 3b: Implement** `src/fxhnt/application/stablecoin_runner.py`:
+```python
+"""Daily-rebalanced SHORT-rich stablecoin peg-reversion backtest over a spot-close panel.
+
+panel = {symbol: {epoch_day: close}}. Each day d->nxt: short the rich names (short_rich_weights),
+book realized daily return minus turnover cost. Flat on days with no rich name."""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import numpy as np
+
+from fxhnt.domain.stablecoin_reversion import short_rich_weights
+
+
+@dataclass(frozen=True)
+class StableRunResult:
+    returns: np.ndarray
+    dates: list[int]          # epoch-day each booked return is realized on (nxt)
+    n_active_avg: float
+
+
+class StableReversionRunner:
+    def __init__(self, panel: dict[str, dict[int, float]], *, thresh_bp: float = 50.0,
+                 cost_bps: float = 4.0, periods_per_year: int = 365) -> None:
+        self._p = panel
+        self._th = thresh_bp / 1e4
+        self._cost = cost_bps / 1e4
+        self._ppy = periods_per_year
+
+    def run(self) -> StableRunResult:
+        all_days = sorted({d for s in self._p.values() for d in s})
+        rets: list[float] = []
+        dates: list[int] = []
+        prev: dict[str, float] = {}
+        active = 0
+        n = 0
+        for i in range(len(all_days) - 1):
+            d, nxt = all_days[i], all_days[i + 1]
+            dev = {s: series[d] - 1.0 for s, series in self._p.items()
+                   if d in series and series[d] > 0}
+            w = short_rich_weights(dev, self._th)
+            realized = 0.0
+            for s, wt in w.items():
+                series = self._p[s]
+                if nxt in series and series[d] > 0 and series[nxt] > 0:
+                    realized += wt * (series[nxt] / series[d] - 1.0)
+            turnover = sum(abs(w.get(s, 0.0) - prev.get(s, 0.0)) for s in set(w) | set(prev))
+            rets.append(realized - turnover * self._cost / 2.0)
+            dates.append(int(nxt))
+            active += len(w)
+            n += 1
+            prev = w
+        return StableRunResult(np.asarray(rets, dtype=float), dates,
+                               (active / n) if n else 0.0)
+```
+
+- [ ] **Step 4: Run, verify PASS** (3 passed).
+- [ ] **Step 5: Commit**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add src/fxhnt/domain/stablecoin_reversion.py src/fxhnt/application/stablecoin_runner.py tests/unit/test_stablecoin_reversion.py
+git commit -m "feat(domain): stablecoin short-rich reversion weights + daily-book runner
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 2: StableRotationForward strategy
+
+**Files:** Create `src/fxhnt/application/stablecoin_strategy.py`; Test `tests/integration/test_stablecoin_strategy.py`.
+
+- [ ] **Step 1: Write failing tests** — `tests/integration/test_stablecoin_strategy.py`:
+```python
+import json
+
+from fxhnt.application.forward_tracker import ForwardTracker
+from fxhnt.application.stablecoin_runner import StableReversionRunner
+from fxhnt.application.stablecoin_strategy import StableRotationForward, _iso
+
+
+def _panel():
+    # S0 trades rich (1.01) only on day 159 (the last day) -> no nxt yet at inception
+    s0 = {d: 1.00 for d in range(160)}
+    s0[159] = 1.01
+    return {"S0": s0, "S1": {d: 1.00 for d in range(160)}}
+
+
+def test_inception_books_nothing(tmp_path):
+    strat = StableRotationForward(lambda: _panel())
+    path = str(tmp_path / "st.json")
+    status = ForwardTracker(strat, path).step()
+    assert status.forward_days == 0
+    state = json.load(open(path))
+    assert state["extra"]["ph"]["killed"] is False
+
+
+def test_subsequent_books_no_drift(tmp_path):
+    p1 = _panel()
+    holder = {"p": p1}
+    strat = StableRotationForward(lambda: holder["p"], decay_lam=1e9)   # never kill
+    path = str(tmp_path / "st.json")
+    ForwardTracker(strat, path).step()                                  # inception, last_date=day159
+    p2 = {s: dict(series) for s, series in p1.items()}
+    for s in p2:
+        p2[s][160] = 1.00                                              # S0 reverts 1.01->1.00 on day160
+    holder["p"] = p2
+    status = ForwardTracker(strat, path).step()
+    assert status.booked_today == 1
+    res = StableReversionRunner(p2, thresh_bp=50.0, cost_bps=4.0).run()
+    booked = next(d["ret"] for d in json.load(open(path))["days"] if d["date"] == _iso(int(res.dates[-1])))
+    assert booked == float(res.returns[-1])                            # no drift
+    assert booked > 0                                                  # short of a reverting rich stable profits
+
+
+def test_killed_state_flattens(tmp_path):
+    p1 = _panel()
+    holder = {"p": p1}
+    strat = StableRotationForward(lambda: holder["p"])
+    path = str(tmp_path / "st.json")
+    ForwardTracker(strat, path).step()
+    state = json.load(open(path)); state["extra"]["ph"]["killed"] = True; state["extra"]["ph"]["recover"] = 0
+    json.dump(state, open(path, "w"))
+    p2 = {s: dict(series) for s, series in p1.items()}
+    for s in p2:
+        p2[s][160] = 1.00
+    holder["p"] = p2
+    ForwardTracker(strat, path).step()
+    assert json.load(open(path))["days"][-1]["ret"] == 0.0
+
+
+def test_extra_json_roundtrips(tmp_path):
+    strat = StableRotationForward(lambda: _panel())
+    path = str(tmp_path / "st.json")
+    ForwardTracker(strat, path).step()
+    json.dumps(json.load(open(path))["extra"])
+```
+
+- [ ] **Step 2: Run, verify FAIL** (ModuleNotFoundError).
+
+- [ ] **Step 3: Implement** `src/fxhnt/application/stablecoin_strategy.py`:
+```python
+"""Live-booking stablecoin peg-reversion (short-rich) paper track — recomputable series.
+
+Reuses StableReversionRunner on a fresh Binance-spot daily-close panel; applies a PageHinkleyDecay kill
+incrementally over new days, carrying detector state in the tracker's `extra`. Short-rich only.
+Mirrors the ForwardStrategy contract: advance(last_date, extra) -> (rows, extra)."""
+from __future__ import annotations
+
+import datetime as dt
+from typing import Any, Callable
+
+from fxhnt.application.stablecoin_runner import StableReversionRunner
+from fxhnt.domain.edge_decay import PageHinkleyDecay
+
+_EPOCH = dt.date(1970, 1, 1)
+
+
+def _iso(epoch_day: int) -> str:
+    return (_EPOCH + dt.timedelta(days=int(epoch_day))).isoformat()
+
+
+class StableRotationForward:
+    def __init__(self, load_panel: Callable[[], dict[str, dict[int, float]]], *,
+                 thresh_bp: float = 50.0, cost_bps: float = 4.0, periods_per_year: int = 365,
+                 decay_lam: float = 0.20, decay_reenter_days: int = 10) -> None:
+        self._load_panel = load_panel
+        self._th = thresh_bp
+        self._cost = cost_bps
+        self._ppy = periods_per_year
+        self._lam = decay_lam
+        self._reenter = decay_reenter_days
+
+    def _series(self) -> tuple[list[int], list[float]]:
+        res = StableReversionRunner(self._load_panel(), thresh_bp=self._th, cost_bps=self._cost,
+                                    periods_per_year=self._ppy).run()
+        return [int(d) for d in res.dates], [float(r) for r in res.returns]
+
+    def advance(self, last_date: str | None,
+                extra: dict[str, Any]) -> tuple[list[tuple[str, float]], dict[str, Any]]:
+        dates, raw = self._series()
+        if not extra:                                       # inception: fresh PH, book nothing
+            ph = PageHinkleyDecay(lam=self._lam, reenter_days=self._reenter)
+            rows = [(_iso(d), r) for d, r in zip(dates, raw)]
+            return rows, {"ph": ph.to_dict()}
+        ph = PageHinkleyDecay.from_dict(extra["ph"])
+        cutoff = last_date or ""
+        rows: list[tuple[str, float]] = []
+        for d, r in zip(dates, raw):
+            iso = _iso(d)
+            if iso <= cutoff:
+                continue
+            alive = ph.update(r)
+            rows.append((iso, r if alive else 0.0))
+        return rows, {"ph": ph.to_dict()}
+```
+
+- [ ] **Step 4: Run, verify PASS** (5 passed).
+- [ ] **Step 5: Commit**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add src/fxhnt/application/stablecoin_strategy.py tests/integration/test_stablecoin_strategy.py
+git commit -m "feat(application): StableRotationForward live paper-track strategy (short-rich + decay kill)
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 3: Registry entry
+
+**Files:** Modify `src/fxhnt/registry.py`; Test `tests/unit/test_stablecoin_registry.py`.
+
+- [ ] **Step 1: Failing test** — `tests/unit/test_stablecoin_registry.py`:
+```python
+from fxhnt.registry import STRATEGY_REGISTRY
+
+
+def test_stablecoin_rotation_registered():
+    e = STRATEGY_REGISTRY["stablecoin_rotation"]
+    assert e["state_file"] == "stablecoin_rotation_state"
+    assert e["sleeve"] == "crypto-stablecoin"
+    assert e["venue"] == "binance-spot"
+    assert e["gate_spec"] == {"min_days": 60, "min_total_return": 0.0, "min_sharpe": 0.5}
+```
+
+- [ ] **Step 2: Run, verify FAIL** (KeyError).
+
+- [ ] **Step 3:** First READ `src/fxhnt/registry.py`. Add this entry inside `STRATEGY_REGISTRY`, immediately after the `"crypto_tstrend"` entry (match indentation):
+```python
+    "stablecoin_rotation": {
+        "display_name": "Stablecoin peg-reversion (short-rich treasury rotation)",
+        "sleeve": "crypto-stablecoin",
+        "venue": "binance-spot", "state_file": "stablecoin_rotation_state",
+        "gate_spec": {"min_days": 60, "min_total_return": 0.0, "min_sharpe": 0.5},
+    },
+```
+
+- [ ] **Step 4: Run, verify PASS.**
+- [ ] **Step 5: Commit**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add src/fxhnt/registry.py tests/unit/test_stablecoin_registry.py
+git commit -m "feat(registry): register stablecoin_rotation sleeve (gate WAIT 60d)
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 4: Dagster wiring
+
+**Files:** Modify `assets.py`, `definitions.py`, `tests/integration/test_orchestration_definitions.py`; Create `tests/integration/test_stablecoin_wiring.py`.
+
+- [ ] **Step 1: Failing test** — `tests/integration/test_stablecoin_wiring.py`:
+```python
+def test_asset_importable():
+    from fxhnt.adapters.orchestration.assets import stablecoin_rotation_nav
+    assert stablecoin_rotation_nav is not None
+
+
+def test_definitions_include_stablecoin():
+    from fxhnt.adapters.orchestration.assets import stablecoin_rotation_nav
+    from fxhnt.adapters.orchestration.definitions import combined_book_job, defs
+    assert stablecoin_rotation_nav in defs.assets
+    assert combined_book_job is not None
+```
+
+- [ ] **Step 2: Run, verify FAIL** (ImportError).
+
+- [ ] **Step 3:** Add the asset in `assets.py` immediately after `crypto_tstrend_nav`. Match the file's conventions (local imports, `get_settings` if used by siblings — check how `crypto_tstrend_nav`/`unlock_nav` import it and mirror exactly):
+```python
+@asset
+def stablecoin_rotation_nav(context: AssetExecutionContext) -> dict:
+    """Paper-forward stablecoin peg-reversion (short-rich daily treasury rotation, Binance spot)."""
+    from fxhnt.adapters.data.binance_spot_history import BinanceSpotHistory
+    from fxhnt.application.stablecoin_strategy import StableRotationForward
+
+    def load_panel() -> dict:
+        h = BinanceSpotHistory()
+        return {s: h.daily_close(s) for s in ("FDUSDUSDT", "TUSDUSDT", "USDPUSDT")}
+
+    return _run_paper_tracker(
+        context, "stablecoin_rotation_nav",
+        lambda: StableRotationForward(load_panel),
+        "stablecoin_rotation_state",
+    )
+```
+NOTE: no `crypto_bars` dep — this asset fetches its own spot data. If `_run_paper_tracker`/`AssetExecutionContext` aren't already in scope at this point in the file, they are (used by `crypto_tstrend_nav`/`xsfunding_nav` just above).
+
+- [ ] **Step 4:** Add `stablecoin_rotation_nav` to the `cockpit_forward` deps list in `assets.py`:
+`@asset(deps=[combined_forward_nav, sixtyforty_nav, xsfunding_nav, unlock_nav, crypto_tstrend_nav, stablecoin_rotation_nav])`
+
+- [ ] **Step 5:** In `definitions.py` add `stablecoin_rotation_nav,` to: the import block from assets (alphabetical), the `combined_book_job` `selection=[...]`, and the `defs = Definitions(assets=[...])` list.
+
+- [ ] **Step 6:** Update `tests/integration/test_orchestration_definitions.py`:
+  - change `assert len(asset_keys) == 8` to `== 9` and add `stablecoin_rotation_nav` to the comment list.
+  - after the `crypto_tstrend_nav` presence assert, add: `assert "stablecoin_rotation_nav" in asset_keys, f"missing stablecoin_rotation_nav: {asset_keys}"`
+  - add `"stablecoin_rotation_nav"` to the `expected_upstream` set for `cockpit_forward`.
+
+- [ ] **Step 7: Run** `cd /home/jgrusewski/Work/fxhnt && .venv/bin/python -m pytest tests/integration/test_stablecoin_wiring.py tests/integration/test_orchestration_definitions.py -v` — all pass.
+
+- [ ] **Step 8: Commit**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add src/fxhnt/adapters/orchestration/assets.py src/fxhnt/adapters/orchestration/definitions.py tests/integration/test_stablecoin_wiring.py tests/integration/test_orchestration_definitions.py
+git commit -m "feat(orchestration): wire stablecoin_rotation_nav into nightly job + cockpit
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 5: Full verification
+
+- [ ] **Step 1:** `cd /home/jgrusewski/Work/fxhnt && .venv/bin/python -m pytest -q` → all green.
+- [ ] **Step 2:** `cd /home/jgrusewski/Work/fxhnt && .venv/bin/python -m mypy src/fxhnt` → "Success".
+- [ ] **Step 3:** If fixups were needed, commit them.
+
+---
+
+## Notes for the executor
+- Use `.venv/bin/python` for pytest/mypy (`python` is not on PATH).
+- **No-drift is load-bearing:** `StableRotationForward` calls `StableReversionRunner` directly — keep `test_subsequent_books_no_drift`.
+- **Short-rich only** — `short_rich_weights` must NEVER return a positive (long) weight; `test_never_longs_cheap...` guards the death-spiral exclusion.
+- Mirror `crypto_tstrend_nav`'s exact import style for `get_settings`/`_run_paper_tracker` if the asset needs them (this one doesn't use `get_settings` — it hardcodes the universe; fine).
+- mypy is strict (`disallow_any_generics`): annotate `dict`/`list` with type args. The `load_panel() -> dict` in the asset mirrors the sibling `crypto_tstrend_nav` which uses `-> dict` with the file's `# type: ignore[type-arg]` convention — match whatever the sibling does.
+```

docs/superpowers/specs/2026-06-18-cross-sectional-funding-design.md

@@ -0,0 +1,56 @@
+# Cross-Sectional Crypto Funding-Dispersion Strategy — Design
+
+**Status:** design approved 2026-06-18 (brainstorm). Data-feasibility spike PASSED (verdict A).
+
+## Why
+Our two real crypto edges (XS momentum, funding carry) live in the less-efficient crypto market. The *existing* `funding` track is a **time-series** carry book (hold every liquid coin whose trailing-30d funding clears a 5 bp/day hurdle). This strategy targets the **cross-sectional dispersion** instead: rank coins by funding *relative to peers*, long the richest-carry / short the poorest. It hedges out the "all funding moves together" common factor (more market-neutral) and lives in the **illiquid tail** that large funds are structurally locked out of — a genuine small-capital edge. Institutions crowd the BTC/ETH basis (now ETF-ified, decayed); the tail dispersion is comparatively open. Target: a modest (0.5–1.0), uncorrelated, capacity-limited edge that *adds to the book-of-books* — not a standalone money-printer.
+
+## Data (feasibility-confirmed, on disk, survivorship-free)
+- **Source:** `foxhunt/data/surfer/crypto_pit/*.npz` — 180 Binance USDT-perp symbols, each with `day` (epoch-day), `close`, `qvol` (daily USD quote-volume), `funding` (daily-summed 8h funding rate). Span 2019-09 → 2026-06. **Survivorship-free**: includes 25 delisted coins (LUNA/FTT/SRM/…), funding present right up to each delisting.
+- fxhnt reads it through a **configurable path** (`FXHNT_CRYPTO_PIT_DIR`, default the foxhunt path for local runs). A thin `CryptoPitPanelSource` adapter loads the npz set into an in-memory panel (`{symbol: {epoch_day: (close, qvol, funding)}}`); `.npz` loaded WITHOUT `allow_pickle` (numeric arrays only).
+- **Cadence is DAILY** (the panel's funding is daily-summed; 8h is not in this panel — out of scope).
+- **Documented caveats:** (1) the delisted-ticker roster is the 35 hard-coded famous blow-ups in `fetch_crypto_pit.py` — survivorship-free for those, but not an exhaustive dead-coin census (future hardening via `data.binance.vision` monthly archives; noted, not in scope). (2) No separate mark-vs-spot basis column — `close` provides returns; the funding rate itself is the carry signal, so this suffices.
+
+## Architecture (mirrors the proven B3b pattern: pure domain + PIT runner + gauntlet)
+### Domain (pure, no I/O) — `domain/strategies/cross_sectional_funding.py`
+- `funding_scores(panel_asof) -> {symbol: score}`: cross-sectional signal at date D from data ≤ D — a blend of **funding z-score across the eligible peers** (robust z, median/MAD) + **funding momentum** (recent trailing funding vs longer trailing). Blending avoids stale-level bias (a coin that paid in the past but not now ranks lower).
+- `construction_weights(scores, mode, quantile=0.2) -> {symbol: weight}` for **three modes (all tested)**:
+  - `long_tilt` — long-carry only: top-quantile by score, positive weights summing to 1 (short perp + long spot — executable at any size, no borrow).
+  - `market_neutral` — top-quantile long + bottom-quantile short, weights sum to 0 (the short leg = long perp + short spot, needs coin-borrow; modelled in backtest, borrow-gated live).
+  - `executable` — long side full + short side ONLY where borrow is plausible (majors); the realistically-deployable subset.
+- `book_return(prev_weights, funding_today, new_weights, cost)`: delta-neutral carry — realized = Σ wᵢ·fundingᵢ (signed: long collects +funding, short collects −funding) − turnover·cost. Generalizes the existing `funding_carry.book_return` to signed cross-sectional weights.
+
+### PIT universe — `domain/cross_sectional_funding` (pure)
+- `eligible_asof(panel, asof, *, min_qvol, min_history)`: coins ALIVE at D (have bars in a trailing window ending D) with trailing dollar-volume ≥ `min_qvol` (the liquidity floor) and enough history to score. Price/volume presence = the tradeability signal (PIT, no membership table needed). No future data.
+
+### Application — `application/funding_backtest_runner.py`
+- Walk-forward: for each daily rebalance D → `eligible_asof` (PIT) → `funding_scores` (≤ D) → for each mode, `construction_weights` → hold to D+1 → accrue `book_return` (funding carry − cost) → per-(mode × floor) daily return series.
+- **The output is a verdict matrix:** `{long_tilt, market_neutral, executable} × {$1M, $5M, $20M floor}` (deep-tail target = $1M; the sweep shows where the edge lives vs where it's uninvestable).
+- `evaluate_funding_constructions(...)` → `compute_stats` + IS/OOS split + gauntlet `evaluate` per cell. **`n_trials` = number of matrix cells tested** (multiple-testing correction — we ARE testing several variants, so DSR must deflate for it).
+
+### Cost model (the crux — deep tail = thin coins)
+- Per-rebalance turnover × round-trip cost, where cost = taker fee (both perp + spot legs) **+ liquidity-scaled slippage** (slippage rises as a coin's `qvol` falls — the tail pays more). Parameterised; the backtest sweeps a conservative cost so we see the edge net of realistic tail friction. The primary way this edge dies is costs eating it on tiny coins — the model must be brutal and the result reported net.
+
+### Pre-registered kill criteria (decide before looking)
+A matrix cell is KILLED if any: net **DSR < 0.5**; OR the edge concentrates only in sub-$1M-ADV names (uninvestable even for us); OR **correlation > 0.6** to the existing `funding` track and/or crypto momentum (not a real diversifier). Surviving cell(s) → paper-forward.
+
+### Paper-forward + cockpit
+- The winning **executable** construction becomes a `ForwardStrategy` cockpit track (new registry entry `xsfunding`), gate=WAIT until forward data confirms, with its backtest verdict surfaced via the `backtest_summary` table we already wired.
+
+## Reuse
+`crypto_pit` npz (existing data), the gauntlet (`compute_stats`/`evaluate`), the B3b PIT-runner + `evaluate_constructions` pattern, `robust_z` (from `equity_factor`), `ForwardTracker` + cockpit `backtest_summary` surface. New: the npz panel adapter, the cross-sectional-funding domain, the funding backtest runner, a CLI `backtest-funding`.
+
+## Testing (same discipline as B3b — no live network)
+- Synthetic in-memory panel (a few coins incl. one that "delists" mid-history, one always-illiquid, one high-dispersion) for all unit/integration tests.
+- **The decisive PIT test:** a coin whose funding is low early / high late is NOT long-ranked early (no look-ahead).
+- Cost-sensitivity test (edge must survive a realistic tail-slippage cost). Gauntlet on a known return series. Determinism.
+
+## Out of scope
+- Live execution / order routing (paper-forward only).
+- 8h-cadence backtest (panel is daily-summed).
+- Exhaustive delisted-coin census (documented caveat; future `data.binance.vision` hardening).
+- Mark-vs-spot basis modelling (funding signal + close suffice).
+- Colocation (not needed for daily carry; available later if ever warranted).
+
+## Honest expectation
+A modest, capacity-limited, uncorrelated carry edge in the tail — or a clean falsification if costs eat it. Either outcome is a win: a new diversifier for the book, or days (not months) spent proving it isn't one. The answer is weeks (backtest now → paper-forward), not years.

docs/superpowers/specs/2026-06-21-crypto-tstrend-sleeve-design.md

@@ -0,0 +1,237 @@
+# Crypto TS-Trend Paper-Track Sleeve — Design (rev. 2)
+
+**Date:** 2026-06-21
+**Status:** Design (pending approval) — rev. 2 after critical review against fxhnt source
+**Repo:** fxhnt (Python agentic fund)
+
+## Motivation
+
+The "speed up profits" exploration (2026-06-20/21) falsified the high-frequency direction twice (crypto
+intraday cross-sectional momentum net SR −21→−0.67; cascade-reversion losers keep losing = continuation;
+see `pearl_crypto_intraday_no_edge_faster_is_feetrap`). The crypto edges are daily-or-slower and
+flow-based. The one validated, uncorrelated, **not-yet-deployed** return engine is crypto time-series
+trend (TSMOM, long/flat):
+
+- Re-verified 2026-06-21 by running the actual `TrendRunner` on the real survivorship-free `crypto_pit`
+  panel (180 coins, 2472 days): `long_short=False` → annualized Sharpe **+1.25** (memory ~1.23 ✓), in
+  **8.2 s** wall-clock (full recompute — cheap enough to run nightly).
+- corr ~0 to the funding edge → adds return at diversified risk. It is a **directional crypto-beta-timing**
+  engine (carries crypto beta when "on"), NOT market-neutral alpha. Intended: the book's directional sleeve.
+
+Missing piece = the **live paper-forward track** so it accumulates a real forward record and graduates
+through the gate like `xsfunding` / `unlock`.
+
+## Goal
+
+Add `crypto_tstrend` as a live paper-track sleeve that books daily paper NAV from inception via the
+nightly Dagster combined-book job and surfaces in the cockpit with `gate=WAIT` (60 days, Sharpe ≥ 0.5)
+until forward-proven. No capital until the gate clears.
+
+## Key design decisions (after critical review)
+
+1. **Recomputable-series pattern, NOT the xsfunding snapshot/position-carry pattern.** Trend's return is
+   just the realized price return of held positions, which `TrendRunner` already computes from a price
+   panel. So the sleeve mirrors `CombinedBookStrategy` (forward_tracker.py:123): each night it recomputes
+   the full series and returns rows; the tracker books only days strictly after `last_date`. This
+   **reuses `TrendRunner.run()` unchanged** → zero live/backtest drift, no bespoke `advance` math.
+2. **Reuse the existing crypto_pit panel — no new data adapter.** `crypto_bars` (assets.py:43) already
+   REFRESHes `crypto_pit` nightly (survivorship-free top-120-by-volume ∪ dead) and `CryptoPitPanelSource`
+   (crypto_pit_panel.py) reads it: `{sym: {day: (close, qvol, funding)}}`. The sleeve reads close-only
+   from it. The earlier scorecard worry ("trend needs a rolling-price-history live source, more plumbing")
+   is wrong — the source already exists.
+3. **Static (configurable) vol-target, NOT adaptive ISV re-sizing.** Own ML/RL research: continuous
+   trust-scaling on a single noisy edge whipsaws (Bongaerts FAJ 2020) and re-introduces the RL
+   train→eval-collapse pathology. The legitimate adaptive-control home is the book-of-books continuous
+   edge-decay layer — a SEPARATE later spec. This sleeve gets a SIMPLE BINARY decay kill only.
+4. **`target_vol` semantics made precise.** `TrendRunner` sizes **per-asset** (`p = sig·target_vol_daily/rv`)
+   then caps GROSS at 1.0 → book vol is emergent (~86%), NOT equal to `target_vol`. The configurable
+   book-vol knob is a SEPARATE **causal portfolio vol-target overlay** applied on top (this is the sizer
+   whose −15%/−23%/−30% maxDD at 10/15/20% the 2026-06-21 probe measured). TrendRunner's internal
+   per-asset `target_vol` stays at its validated default; the overlay normalizes book vol to the target.
+
+## Architecture
+
+Reuses generic forward-tracker / registry / Dagster-schedule / ingest / cockpit infra. New code = one
+domain helper (decay detector) + one strategy + wiring.
+
+### 1. Edge-decay detector (NEW, reusable)
+
+**File:** `src/fxhnt/domain/edge_decay.py` — **Class:** `PageHinkleyDecay`
+
+Canonical Page-Hinkley **mean-shift** detector with binary kill + hysteresis (per
+`pearl_edge_decay_detection_is_a_missing_abstraction_layer`; shift-form, not level — level froze in the
+RL context per `pearl_edge_decay_detector_phase1_validated_train_also_decays`).
+
+```python
+@dataclass
+class PageHinkleyDecay:
+    delta: float = 0.0        # deadband: larger tolerates more downward deviation (LESS sensitive)
+    lam: float = 0.20         # cumulative-deviation threshold to KILL (calibrated to ~0.5% daily std)
+    reenter_days: int = 10    # consecutive non-negative SHADOW days to re-arm (hysteresis)
+    # mutable state (all JSON scalars): mean, n, m_t, ph_min, killed, recover
+    def update(self, ret: float) -> bool:
+        """Feed one realized SHADOW daily return; return True if ALIVE, False if KILLED.
+        ALIVE→KILLED when m_t - min(m_t) > lam (sustained downward mean shift).
+        KILLED→ALIVE only after `reenter_days` consecutive ret >= 0 (then reset PH accumulators)."""
+    def to_dict(self) -> dict: ...
+    @classmethod
+    def from_dict(cls, d: dict) -> "PageHinkleyDecay": ...
+```
+
+Pre-registered: `lam=0.20, delta=0.0, reenter_days=10`. `lam=0.20` is calibrated to the strategy's
+~0.5%/day return std (the PH statistic is a near-driftless random walk with ~0.5% steps; `lam=0.05`
+≈ only 10 adverse steps → false-kills a healthy edge ~71%/yr, `lam=0.20` ≈ 38 cumulative adverse
+steps → ~92% healthy-edge survival/yr while still catching genuine sustained decay). Falsification:
+must NOT kill on a clean stationary-positive series; must KILL on a synthetic mean-flip series.
+
+### 2. Live-booking strategy (NEW)
+
+**File:** `src/fxhnt/application/ts_trend_strategy.py` — **Class:** `TsTrendForward` (implements `ForwardStrategy`)
+
+```python
+class TsTrendForward:
+    def __init__(self, load_panel: Callable[[], dict[str, dict[int, float]]], *,
+                 lookbacks=(20, 60, 120), vol_window=30, internal_target_vol=0.10,
+                 gross_cap=1.0, cost_bps=10.0, periods_per_year=365,
+                 book_target_vol=0.15, lev_cap=3.0, vol_buf_window=30,
+                 decay_lam=0.20, decay_reenter_days=10) -> None: ...
+
+    def advance(self, last_date, extra) -> tuple[list[tuple[str, float]], dict]:
+        """Recomputable series + causal overlay + binary decay kill.
+
+        1. panel = load_panel()  # {sym:{day:close}} (close extracted from CryptoPitPanelSource)
+        2. res = TrendRunner(panel, lookbacks, vol_window, internal_target_vol, gross_cap,
+                             long_short=False, cost_bps, periods_per_year).run()
+           → raw_rets[], dates[]  (dates = epoch-day each return is REALIZED on; ~8s)
+        3. INCEPTION (extra empty / last_date None):
+             - warm vol_buf = trailing `vol_buf_window` raw_rets from history (overlay warm-start ONLY)
+             - PH = fresh ALIVE  (history decay must NOT pre-kill the forward track)
+             - return ALL rows [(iso(d), raw) ...] so tracker freezes inception at latest date
+               (tracker books NONE on inception); persist extra {ph, vol_buf}
+        4. SUBSEQUENT — for each d in dates with iso(d) > last_date, in chronological order:
+             tv  = std(vol_buf) (daily); lev = clip(book_target_vol/sqrt(ppy) / max(tv,eps), 0, lev_cap)
+             alive = ph.update(raw_d)            # PH consumes SHADOW raw return (continuous)
+             booked = lev*raw_d if alive else 0.0
+             rows.append((iso(d), booked)); vol_buf.append(raw_d) [trim]
+           persist extra {ph: ph.to_dict(), vol_buf}; return ONLY new rows
+        """
+```
+
+**Shadow semantics** (mirror the tracker's `kill_dd` shadow): the overlay buffer and PH always consume
+the *raw* (unkilled) return, so the kill can detect recovery and re-arm even while booked exposure is 0.
+**Causality:** `lev` uses the trailing buffer of returns *before* `d`; `raw_d` is realized over `[d, nxt]`
+by TrendRunner. No lookahead. **Path-stability:** PH + vol_buf are carried in `extra` and advanced only
+over NEW days, so panel top-N churn / historical revision cannot retro-change a booked kill decision (the
+tracker also books only strictly-new days, so past NAV is sticky regardless).
+
+### 3. Registry entry
+
+**File:** `src/fxhnt/registry.py`
+```python
+"crypto_tstrend": {
+    "display_name": "Crypto TS-momentum (long/flat daily trend, 20/60/120d)",
+    "sleeve": "crypto-trend", "venue": "binance-perp",
+    "state_file": "crypto_tstrend_state",
+    "gate_spec": {"min_days": 60, "min_total_return": 0.0, "min_sharpe": 0.5},
+},
+```
+
+### 4. Dagster wiring
+
+**File:** `src/fxhnt/adapters/orchestration/assets.py`
+```python
+@asset(deps=[crypto_bars])     # MUST dep on crypto_bars so the panel is fresh before we read it
+def crypto_tstrend_nav(context: AssetExecutionContext) -> dict:
+    from fxhnt.adapters.data.crypto_pit_panel import CryptoPitPanelSource
+    from fxhnt.application.ts_trend_strategy import TsTrendForward
+    s = get_settings()
+    def load_panel():
+        raw = CryptoPitPanelSource(s.crypto_pit_dir).panel()   # {sym:{day:(close,qvol,funding)}}
+        return {sym: {d: v[0] for d, v in series.items()} for sym, series in raw.items()}
+    return _run_paper_tracker(
+        context, "crypto_tstrend_nav",
+        lambda: TsTrendForward(load_panel, book_target_vol=0.15),
+        "crypto_tstrend_state",
+    )
+```
+
+**File:** `src/fxhnt/adapters/orchestration/definitions.py` — add `crypto_tstrend_nav` to the
+`combined_book_job` selection.
+
+**File:** `assets.py` — **add `crypto_tstrend_nav` to the explicit `cockpit_forward` dep list**
+(currently `@asset(deps=[combined_forward_nav, sixtyforty_nav, xsfunding_nav, unlock_nav])`, line ~207),
+else cockpit may run before the track materializes / omit it. **(latent-bug catch from review.)**
+
+`book_target_vol` (and `decay_lam`/`decay_reenter_days`) are set at this composition root → re-tunable
+with no strategy-code change.
+
+### 5. Drawdown kill-switch note (corrected)
+
+`_run_paper_tracker` (assets.py:127) builds `ForwardTracker(strategy, path).step()` and **does NOT pass
+`kill_dd`** — so per-sleeve paper tracks (xsfunding, unlock, and this one) have **no** tracker drawdown
+kill. The strategy-level `PageHinkleyDecay` is therefore this sleeve's **primary** (and only) edge-death
+guard, not a complement. We deliberately leave the tracker `kill_dd` OFF to avoid a double-kill
+interaction; it remains available if a level-trigger is wanted later.
+
+### 6. Tests (mirror xsfunding pattern + new)
+
+- `tests/unit/test_edge_decay.py` — `PageHinkleyDecay`: stays ALIVE on stationary-positive series; KILLS
+  on positive→negative mean-flip; RE-ARMS only after `reenter_days` non-negative days; `to_dict`/`from_dict`
+  round-trip; does NOT kill on the validated long/flat backtest return series (falsification).
+- `tests/integration/test_tstrend_strategy.py` — fake `load_panel` (synthetic uptrend/downtrend panel):
+  - inception → returns rows, tracker books none, extra warm-started (vol_buf full, PH alive);
+  - subsequent day → booked == `lev*raw` (hand-computed), `lev` from carried buffer, causal;
+  - **no-drift**: the raw return for a new day equals `TrendRunner.run()`'s tail value for that date;
+  - decay kill → booked flattens to 0, vol_buf still advances (shadow), re-arms after hysteresis;
+  - `extra` JSON round-trips through `json.dumps`/`loads`.
+- `tests/integration/test_orchestration_assets.py` — extend: `crypto_tstrend_nav` materializes a valid
+  days-list state file from a synthetic panel; asset is reachable after `crypto_bars`.
+- `tests/unit/test_registry.py` — extend: `crypto_tstrend` present with required fields.
+
+## Data flow
+
+```
+nightly 23:30 UTC combined_book_job
+  → crypto_bars (REFRESH crypto_pit panel)
+  → crypto_tstrend_nav  [deps=crypto_bars]
+      → TsTrendForward(load_panel, book_target_vol=0.15)
+          → TrendRunner.run(long_short=False, ppy=365)   (~8s) → raw_rets, dates
+          → per new day: causal vol-overlay (lev) · PH shadow kill → booked
+      → ForwardTracker.step()  → crypto_tstrend_state.json (days-list NAV)
+  → cockpit_forward [deps += crypto_tstrend_nav]  → cockpit DB (gate=WAIT until 60d & Sharpe≥0.5)
+```
+
+## Error handling
+
+- Transient (network/disk): handled by `_run_paper_tracker` (logged, state unchanged, job not blocked).
+- Genuine bugs (ValueError/KeyError/AttributeError): propagate and fail loudly.
+- Empty panel / all symbols too short: TrendRunner returns empty `dates` → strategy books nothing that
+  day (no crash). At inception with empty history: freeze at `_today_iso()`, empty extra warmed lazily.
+- Insufficient vol_buf (shouldn't happen post-warm-start): `lev` falls back to 1.0 (eps-guarded std).
+
+## Risks / honest caveats
+
+- **Directional beta, not alpha** — long crypto when trends are up; goes flat in bears but bled in 2022.
+  corr~0 to funding is the diversification claim, not market-neutrality.
+- **Backtest → deployable haircut** — Sharpe ~1.25 backtest → realistically ~0.8–1.0 deployable; the
+  forward track is the truth-teller (same role xsfunding's forward track plays for tail-basis).
+- **Crash co-movement** — in a crypto deleveraging cascade funding (short-convex) and trend (de-risks but
+  lags fast cascades) can both hurt before trend flattens. Book-of-books sizing (later) handles this; this
+  sleeve alone does not.
+- **Recompute redundancy** — full 8s recompute nightly is wasteful but correct and drift-free; acceptable
+  at this cadence. If the panel grows large enough to matter, factor a single-day position helper later.
+
+## Acceptance criteria
+
+- `crypto_tstrend` registered; nightly job materializes `crypto_tstrend_state.json` (days-list shape);
+  `cockpit_forward` depends on it and surfaces it with `gate=WAIT`, NAV booking from inception.
+- **No drift**: a test asserts the strategy's raw per-day return equals `TrendRunner.run()`'s value for
+  the same date on a synthetic panel.
+- `PageHinkleyDecay`: no-kill on the validated window, kills on a synthetic mean-flip, re-arms with
+  hysteresis, state persists through `extra` JSON.
+- Overlay is causal (uses only trailing returns) and `book_target_vol` controls realized book vol;
+  the −15/−23/−30% maxDD at 10/15/20% targets reproduces the 2026-06-21 probe.
+- All new + existing tests green; mypy clean.
+- `book_target_vol`, `decay_lam`, `decay_reenter_days` adjustable at the asset composition root without
+  touching strategy code.
+```

docs/superpowers/specs/2026-06-21-observability-2e-design.md

@@ -0,0 +1,68 @@
+# Re-enable Observability (Prometheus + Grafana) + Expose dagster.fxhnt.ai — Design (Phase 2E)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** fxhnt (IaC home as of 2D)
+**Part of:** Phase 2 platform consolidation ([[project_phase2_gitlab_to_gitea]]). 2A–2D DONE. 2E is the
+optional observability follow-up.
+
+## Motivation
+
+GitLab's removal (2C) freed ~5.5 GiB. The `grafana` + `prometheus-stack` (kube-prometheus-stack) Helm
+releases are already installed but **scaled to zero** (only node-exporters run). Re-enable the *core
+metrics* slice to monitor the cluster + fund, and expose Grafana and the Dagster UI over the tailnet
+proxy so they're reachable like `git`/`dashboard`/`minio`.
+
+## Scope — re-enable (core metrics only)
+
+ON: `prometheus` (via operator), `prometheus-stack-kube-prom-operator`,
+`prometheus-stack-kube-state-metrics`, `alertmanager`, `grafana`, node-exporters (already running).
+OFF (stay at 0): loki, tempo, alloy, promtail, pushgateway, dcgm-exporter (GPU, dead).
+
+Re-enable by scaling the operator + kube-state-metrics + grafana deployments back to 1; the operator then
+reconciles the Prometheus **and Alertmanager** CRs (both `replicas=1`) → brings those statefulsets up. The
+configured prometheus-rules (hft/broker/storage) fire into Alertmanager.
+
+**Note — alert notification routing is a follow-up:** Alertmanager will run + collect firing alerts, but
+*where* they get sent (Mattermost / email via Stalwart / webhook) needs a receiver config + channel choice.
+2E gets Alertmanager running with rules active (visible in Grafana unified-alerting + the in-cluster
+Alertmanager UI); wiring a notification receiver is a small follow-up.
+
+## Scope — expose via tailnet proxy (`infra/k8s/gitlab/tailscale-proxy.yaml`)
+
+1. **`grafana.fxhnt.ai`** — svc (`grafana:80`) + nginx block + DNS A record already exist. Missing piece:
+   a NetworkPolicy admitting the `tailscale-gitlab-proxy` pod → `grafana:80` (same fix as minio/gitea;
+   that's why grafana was 502/unreachable). Grafana Prometheus datasource already configured
+   (`grafana-datasources.yaml`); admin from the `grafana-admin` secret.
+2. **`dagster.fxhnt.ai`** — NEW:
+   - nginx server block `dagster.fxhnt.ai` → `dagster.foxhunt.svc.cluster.local:80` (Dagster webserver),
+     with websocket headers (Dagster UI uses WebSockets for live run updates).
+   - Scaleway DNS A record `dagster → 100.95.225.27` (live proxy node IP).
+   - dagster NetworkPolicy admitting the `tailscale-gitlab-proxy` → `:80` (dagster webserver container port).
+
+## Data flow
+
+Browser (tailnet) → `grafana.fxhnt.ai`/`dagster.fxhnt.ai` → nginx (tailscale-gitlab-proxy) → svc →
+pod. Grafana → Prometheus (in-cluster) → scrapes node-exporters + kube-state-metrics + service-monitors.
+
+## Risks / safety
+
+- **Resource:** ~1.5–2 GiB for prometheus+grafana+kube-state-metrics+alertmanager. Headroom exists (GitLab freed 5.5 GiB).
+- **Proxy edit:** adding two server blocks + restarting the proxy — verify `git`/`dashboard` stay 200
+  after (the proxy fronts everything). nginx resolves upstreams at boot, so the new upstreams
+  (grafana/dagster svcs) must exist before restart — they do.
+- **netpol pattern:** admit-the-proxy is the proven pattern (postgres/gitea/minio). Low risk.
+- **No data destroyed** — purely additive (scale-up + wiring).
+- **Dagster auth:** the Dagster UI has no built-in auth; it's only reachable over the private tailnet
+  (same as the other `*.fxhnt.ai` services) — acceptable for a single-maintainer private platform.
+
+## Out of scope
+Logs (Loki), traces (Tempo), GPU metrics (dcgm) — the heavier options not chosen. Alert notification
+receiver/routing config (Mattermost/email/webhook) — a follow-up after Alertmanager is up.
+
+## Acceptance criteria
+- `prometheus`, `kube-state-metrics`, operator, `alertmanager`, `grafana` pods Running; loki/tempo stay 0.
+- `grafana.fxhnt.ai` returns 200 (login page); a dashboard shows live pod CPU/RAM; Prometheus targets up.
+- `dagster.fxhnt.ai` returns 200 (Dagster run UI loads, live updates work).
+- `git.fxhnt.ai` + `dashboard.fxhnt.ai` still 200 (proxy not regressed).
+- Changes committed to fxhnt (values + proxy + netpols) + pushed; DNS record created.

docs/superpowers/specs/2026-06-21-stablecoin-rotation-sleeve-design.md

@@ -0,0 +1,156 @@
+# Stablecoin Peg-Reversion (Treasury-Rotation) Paper-Track Sleeve — Design
+
+**Date:** 2026-06-21
+**Status:** Design (pending approval)
+**Repo:** fxhnt
+
+## Motivation
+
+The crypto-FX dislocation hunt found a validated, bounded, uncorrelated edge: **short-rich stablecoin
+peg-reversion**. When a stablecoin trades *above* $1, mint/redeem arbitrage (forced flow) pulls it back
+to parity; selling the rich stable and rebuying on reversion harvests that. Validation (foxhunt probes
+`scripts/surfer/crypto_stablecoin_*.py`):
+
+- **Spike** (daily): edge grows with dislocation size (+11→+23→+33 bp as threshold 25→50bp), concentrated
+  in less-arbed stables; small <10bp gaps die at cost (HFT-arbed), big >50bp gaps survive even 10bp cost.
+- **Hardened** (non-overlapping, UST included, slippage-scaled, long/short split): **SHORT-rich** = mean
+  +24.7 bp/trade, hit 68-70%, worst only −148bp, SR~4, positive 5/6 years, robust to all filters. Only 8%
+  of events on BTC-crash days → idiosyncratic/uncorrelated. **LONG-cheap = death-spiral trap** (UST −75%,
+  stops gap through) → EXCLUDED.
+- **Execution** (1h, next-bar-open fills + spread): NOT latency-gated — rich deviations persist for hours
+  (TUSD 90% still rich at +8h), so next-open fills + 2bp spread still net +18-36bp, hit 85-93%. Live book:
+  spreads 0.1-1bp, deep enough for small capital. USDC too tight (never rich); DAI/BUSD delisted (empty book).
+
+It's a small, uncorrelated **yield overlay on stablecoin cash** (capacity = cash balance), not a return
+engine. Deployed as a treasury rotation it needs no borrow.
+
+## Goal
+
+Add `stablecoin_rotation` as a live paper-track sleeve that books a **daily-rebalanced short-rich
+stablecoin reversion** return from inception via the nightly Dagster job, surfacing in the cockpit with
+`gate=WAIT` (60 days, Sharpe ≥ 0.5). No capital until forward-proven.
+
+## Non-Goals
+
+- **No LONG-cheap leg** — death-spiral trap (UST→0); excluded by design.
+- **No intraday execution** in this track — the daily-rebalance book is the honest conservative proxy
+  (the daily hardened test confirmed +25bp/SR~4); a future intraday executor could capture more, separately.
+- No order/treasury-execution bridge here (deferred until gate=GO, analogous to `xsfunding-orders`).
+- No new adaptive sizing — fixed equal-risk-per-rich-name; the edge is bounded so no continuous controller.
+
+## Key design decisions
+
+1. **Daily-rebalanced short-rich book** (mirrors `xsfunding`'s daily-book + `crypto_tstrend`'s
+   recomputable-series wiring). Each day: short (w<0) every stablecoin currently trading rich beyond a
+   threshold, equal-weight; book the realized daily return minus turnover cost. Flat on days with no rich
+   name (honest — it's episodic).
+2. **Universe = live-tradeable, deviation-prone liquid stables**: `FDUSDUSDT, TUSDUSDT, USDPUSDT`
+   (configurable). USDC excluded (reverts in ~2h, never rich); DAI/BUSD excluded (delisted, empty books).
+3. **Recomputable series, reuse the existing spot source.** `BinanceSpotHistory.daily_close(sym)` returns
+   `{epoch_day: close}` per pair (test-injectable). The strategy recomputes the full daily book each night;
+   the `ForwardTracker` books only days strictly after `last_date` (same pattern as `crypto_tstrend`).
+4. **PageHinkleyDecay kill reused** (`lam=0.20`) as cheap insurance — bounded edge, but consistent with
+   the book's edge-decay discipline; configurable.
+
+## Architecture (mirrors crypto_tstrend; generic tracker/registry/cockpit reused)
+
+### 1. Domain (NEW) — `src/fxhnt/domain/stablecoin_reversion.py`
+
+```python
+def short_rich_weights(dev: dict[str, float], thresh: float) -> dict[str, float]:
+    """dev = {sym: close-1.0} as-of a day. Return equal-weight SHORT weights on names with dev>thresh
+    (rich), summing to gross 1 (e.g. -1/k each over k rich names); {} if none rich. Long-cheap is NEVER
+    taken (death-spiral trap)."""
+```
+
+### 2. Runner (NEW) — `src/fxhnt/application/stablecoin_runner.py`
+
+```python
+@dataclass(frozen=True)
+class StableRunResult:
+    returns: np.ndarray
+    dates: list[int]          # epoch-day each daily book return is realized on
+    n_active_avg: float
+
+class StableReversionRunner:
+    def __init__(self, panel: dict[str, dict[int, float]], *, thresh_bp: float = 50.0,
+                 cost_bps: float = 4.0, periods_per_year: int = 365) -> None: ...
+    def run(self) -> StableRunResult:
+        """Walk the union of days; each day d→nxt: dev_i = close_i[d]-1; w = short_rich_weights(dev,thresh);
+        realized = Σ w_i·(close_i[nxt]/close_i[d]-1) − turnover(w,prev)·cost; book on nxt."""
+```
+
+### 3. Strategy (NEW) — `src/fxhnt/application/stablecoin_strategy.py`
+
+```python
+class StableRotationForward:   # ForwardStrategy: advance(last_date, extra) -> (rows, extra)
+    def __init__(self, load_panel: Callable[[], dict[str, dict[int, float]]], *,
+                 thresh_bp=50.0, cost_bps=4.0, periods_per_year=365,
+                 decay_lam=0.20, decay_reenter_days=10) -> None: ...
+    # recomputable: run StableReversionRunner on fresh panel → (dates, returns);
+    # inception → warm PH fresh/alive, book nothing; subsequent → for new days apply PH kill
+    #   (booked = ret if alive else 0.0), carry ph-state in extra. (No vol overlay — bounded edge.)
+```
+
+The Dagster asset builds `load_panel` from `BinanceSpotHistory().daily_close` over the universe:
+```python
+def load_panel():
+    h = BinanceSpotHistory()
+    return {s: h.daily_close(s) for s in ("FDUSDUSDT", "TUSDUSDT", "USDPUSDT")}
+```
+
+### 4. Registry — `src/fxhnt/registry.py`
+```python
+"stablecoin_rotation": {
+    "display_name": "Stablecoin peg-reversion (short-rich treasury rotation)",
+    "sleeve": "crypto-stablecoin", "venue": "binance-spot",
+    "state_file": "stablecoin_rotation_state",
+    "gate_spec": {"min_days": 60, "min_total_return": 0.0, "min_sharpe": 0.5},
+},
+```
+
+### 5. Dagster wiring — `assets.py` + `definitions.py`
+- New `@asset stablecoin_rotation_nav` via `_run_paper_tracker(...)` (no `crypto_bars` dep — it fetches its
+  own spot data; best-effort transient handling already in `_run_paper_tracker`).
+- Add to `cockpit_forward` deps, the `combined_book_job` selection, and `defs.assets`.
+- Update `test_orchestration_definitions.py` asset count (8 → 9).
+
+### 6. Tests (mirror crypto_tstrend)
+- `tests/unit/test_stablecoin_reversion.py` — `short_rich_weights`: shorts only rich names, equal-weight
+  gross 1, empty when none rich, NEVER longs a cheap name.
+- `tests/integration/test_stablecoin_strategy.py` — fake `load_panel`: inception books nothing; a rich
+  name → next-day booked return matches hand-computed `Σw·r − cost`; no-drift vs `StableReversionRunner`;
+  PH kill flattens + carries; `extra` JSON round-trips.
+- `tests/integration/test_stablecoin_wiring.py` — asset importable; `defs` includes it.
+
+## Data flow
+```
+nightly job → stablecoin_rotation_nav
+  → StableRotationForward(load_panel = BinanceSpotHistory.daily_close over {FDUSD,TUSD,USDP})
+      → StableReversionRunner.run() → daily short-rich book returns + dates
+      → per new day: PH kill → booked
+  → ForwardTracker.step() → stablecoin_rotation_state.json
+  → cockpit_forward → cockpit DB (gate=WAIT until 60d & Sharpe≥0.5)
+```
+
+## Error handling
+- Network/transient: handled by `_run_paper_tracker` (logged, state unchanged, job not blocked).
+- A pair returning empty/short history: skipped that run (excluded from the day's universe); book 0 if none.
+- Genuine bugs propagate.
+
+## Risks / honest caveats
+- **Small capacity / episodic** — ~modest trades/yr, flat on no-rich days; a yield overlay, not an engine.
+- **Daily proxy understates** the intraday edge (conservative — fine for a paper track).
+- **Stress-depeg book thinning** unmodeled (calm-state spreads used); the rich side is the calmer side, but
+  a real cascade can widen spreads — the forward track is the truth-teller.
+- **Cost assumption** `cost_bps=4` (≈2bp spread rt + slippage buffer); the forward record validates it.
+
+## Acceptance criteria
+- `stablecoin_rotation` registered; nightly job materializes `stablecoin_rotation_state.json` (days-list);
+  `cockpit_forward` depends on it; cockpit shows it `gate=WAIT`, booking from inception.
+- `short_rich_weights` never longs a cheap name (death-spiral guard); shorts only rich, equal-weight.
+- No-drift: strategy's daily return == `StableReversionRunner` value for the same date (test).
+- PH kill flattens + re-arms; `extra` JSON-safe.
+- All new + existing tests green (definitions count 8→9); mypy clean.
+- `thresh_bp`, `cost_bps`, `decay_lam` adjustable at the asset composition root.
+```

infra/argo/cockpit-build-deploy.yaml

@@ -1,10 +1,9 @@
 # Argo WorkflowTemplate — build + deploy the fxhnt cockpit, fully in-cluster.
 #
-# Mirrors foxhunt's proven `build-ci-image` (Kaniko) + `compile-and-deploy` (kubectl) patterns, reusing the
-# same cluster primitives: the `argo-workflow` ServiceAccount, the `argo-git-ssh-key` deploy key (clone over
-# the in-cluster gitlab-shell), and the `gitlab-registry` dockerconfig (push to the internal registry). The
-# build runs in-cluster so it reaches the registry + token endpoint natively — no laptop, no private-CA trust,
-# nothing public.
+# Reuses cluster primitives: the `argo-workflow` ServiceAccount, the `argo-git-ssh-key` deploy key (clone
+# the fxhnt repo over in-cluster Gitea SSH, gitea-sshd:22), and the `scw-registry` dockerconfig (push to
+# the external Scaleway Container Registry rg.fr-par.scw.cloud/bizworx). Build runs in-cluster.
+# (Phase 2B: migrated off the in-cluster GitLab shell + registry.)
 #
 # Submit:  argo submit -n foxhunt --from=wftmpl/fxhnt-cockpit -p commit-sha=<sha> --watch
 # (or use scripts/argo-deploy-cockpit.sh)
@@ -50,7 +49,7 @@ spec:
               chmod 600 /root/.ssh/id_ed25519
               printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
               git clone --no-checkout --filter=blob:none \
-                "ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/fxhnt.git" /workspace/src
+                "ssh://git@gitea-sshd.foxhunt.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
               cd /workspace/src && git checkout "{{workflow.parameters.commit-sha}}"
               echo "checked out $(git rev-parse --short HEAD)"
           volumeMounts:
@@ -64,11 +63,9 @@ spec:
             /kaniko/executor \
               --context=/workspace/src \
               --dockerfile=/workspace/src/infra/docker/fxhnt.Dockerfile \
-              --destination=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/fxhnt-cockpit:latest \
-              --insecure-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --skip-tls-verify-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
+              --destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest \
               --cache=true \
-              --cache-repo=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/cache \
+              --cache-repo=rg.fr-par.scw.cloud/bizworx/cache \
               --snapshot-mode=redo
         env:
           - { name: DOCKER_CONFIG, value: /kaniko/.docker }
@@ -81,7 +78,7 @@ spec:
         - { name: git-ssh-key, secret: { secretName: argo-git-ssh-key, defaultMode: 256 } }
         - name: registry-auth
           secret:
-            secretName: gitlab-registry
+            secretName: scw-registry
             items: [{ key: .dockerconfigjson, path: config.json }]
 
     # --- deploy: apply manifests from the repo + roll the dashboard to the fresh image ---
@@ -102,7 +99,7 @@ spec:
               chmod 600 /root/.ssh/id_ed25519
               printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
               git clone --no-checkout --filter=blob:none \
-                "ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/fxhnt.git" /workspace/src
+                "ssh://git@gitea-sshd.foxhunt.svc.cluster.local:22/gitadmin/fxhnt.git" /workspace/src
               cd /workspace/src && git checkout "{{workflow.parameters.commit-sha}}"
           volumeMounts:
             - { name: git-ssh-key, mountPath: /etc/git-ssh, readOnly: true }

infra/k8s/argo/archive-rbac.yaml

@@ -0,0 +1,34 @@
+# RBAC for Argo workflow archive — grants controller and server access
+# to the PostgreSQL archive secret and MinIO credentials.
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: argo-archive-secrets
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: argo-workflows
+rules:
+- apiGroups: [""]
+  resources: ["secrets"]
+  resourceNames: ["argo-postgres-config", "minio-credentials"]
+  verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: argo-archive-secrets
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: argo-workflows
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: argo-archive-secrets
+subjects:
+- kind: ServiceAccount
+  name: argo-workflows-workflow-controller
+  namespace: foxhunt
+- kind: ServiceAccount
+  name: argo-workflows-server
+  namespace: foxhunt

infra/k8s/argo/argo-workflow-netpol.yaml

@@ -0,0 +1,518 @@
+# Base egress for ALL Argo workflow pods: DNS, K8s API, MinIO (logging), Mattermost (notifications).
+# Applies to every pod with app.kubernetes.io/part-of: foxhunt.
+# Component-specific policies below add extra rules (GitLab SSH, external HTTPS, etc.).
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: argo-base-egress
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/part-of: foxhunt
+  policyTypes:
+    - Egress
+  egress:
+    # DNS
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Kubernetes API — Argo executor reports task results
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 10.32.0.0/16
+        - ipBlock:
+            cidr: 172.16.0.4/32
+    # MinIO — log archival (Argo wait sidecar)
+    - ports:
+        - port: 9000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+    # Mattermost — exit handler webhook notifications
+    - ports:
+        - port: 8065
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: mattermost
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: argo-training-workflow
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/component: training-workflow
+  policyTypes:
+    - Egress
+  egress:
+    # DNS — required for hostname resolution (minio, gitlab, etc.)
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Kubernetes API — Argo executor reports task results
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 10.32.0.0/16
+        - ipBlock:
+            cidr: 172.16.0.4/32
+    # HTTPS (external registries, etc.)
+    - ports:
+        - port: 443
+          protocol: TCP
+    # GitLab SSH — git clone
+    - ports:
+        - port: 2222
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 100.90.76.85/32
+        - podSelector:
+            matchLabels:
+              app: gitlab-shell
+    # GitLab API — binary fetch from packages
+    - ports:
+        - port: 8181
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app: webservice
+    # MinIO — log archival + result upload
+    - ports:
+        - port: 9000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+    # Pushgateway — Prometheus metrics
+    - ports:
+        - port: 9091
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: pushgateway
+    # Tempo — OTLP traces
+    - ports:
+        - port: 4317
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tempo
+    # Mattermost — exit handler webhook notifications
+    - ports:
+        - port: 8065
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: mattermost
+    # Container registry — image pulls
+    - ports:
+        - port: 5000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app: registry
+---
+# Compile workflow pods: need GitLab SSH, MinIO, k8s API, external HTTPS (cargo deps)
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: argo-compile-workflow
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/component: compile
+  policyTypes:
+    - Egress
+  egress:
+    # DNS — resolve external hostnames (crates.io, github.com)
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Kubernetes API — Argo executor
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 10.32.0.0/16
+        - ipBlock:
+            cidr: 172.16.0.4/32
+    # External HTTPS — cargo fetches crates.io, github.com (candle git dep)
+    - ports:
+        - port: 443
+          protocol: TCP
+    # GitLab SSH — git clone (Tailscale IP + cluster service)
+    - ports:
+        - port: 2222
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 100.90.76.85/32
+        - podSelector:
+            matchLabels:
+              app: gitlab-shell
+    # MinIO — binary upload
+    - ports:
+        - port: 9000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+    # Mattermost — exit handler webhook notifications
+    - ports:
+        - port: 8065
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: mattermost
+---
+# Compile-and-deploy workflow pods: need GitLab HTTP (tagging/releases), MinIO,
+# k8s API (deploy rollouts + Argo executor), DNS, GitLab SSH, external HTTPS.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: argo-compile-and-deploy-workflow
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/component: compile-and-deploy
+  policyTypes:
+    - Egress
+  egress:
+    # DNS
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Kubernetes API — Argo executor + kubectl rollout
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 10.32.0.0/16
+        - ipBlock:
+            cidr: 172.16.0.4/32
+    # External HTTPS — cargo fetches, kubectl download
+    - ports:
+        - port: 443
+          protocol: TCP
+    # GitLab SSH — git clone
+    - ports:
+        - port: 2222
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 100.90.76.85/32
+        - podSelector:
+            matchLabels:
+              app: gitlab-shell
+    # GitLab webservice — CalVer tagging, releases, package uploads
+    - ports:
+        - port: 8181
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app: webservice
+    # MinIO — binary upload + log archival
+    - ports:
+        - port: 9000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+    # Mattermost — exit handler webhook notifications
+    - ports:
+        - port: 8065
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: mattermost
+---
+# Compile-and-train workflow pods: compile step needs GitLab SSH + external HTTPS + DNS,
+# training steps need GitLab HTTP (package fetch/upload), MinIO, Tempo, k8s API.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: argo-train-workflow
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/component: train
+  policyTypes:
+    - Egress
+  egress:
+    # DNS
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Kubernetes API — Argo executor
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 10.32.0.0/16
+        - ipBlock:
+            cidr: 172.16.0.4/32
+    # External HTTPS — cargo fetches crates.io, github.com (candle git dep)
+    - ports:
+        - port: 443
+          protocol: TCP
+    # GitLab SSH — git clone
+    - ports:
+        - port: 2222
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 100.90.76.85/32
+        - podSelector:
+            matchLabels:
+              app: gitlab-shell
+    # GitLab webservice — package upload/download
+    - ports:
+        - port: 8181
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app: webservice
+    # MinIO — binary upload
+    - ports:
+        - port: 9000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+    # Tempo — OTLP traces
+    - ports:
+        - port: 4317
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tempo
+    # Mattermost — exit handler webhook notifications
+    - ports:
+        - port: 8065
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: mattermost
+---
+# CI pipeline pods (detect-changes + compile-service inline templates)
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: argo-ci-pipeline
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/component: ci-pipeline
+  policyTypes:
+    - Egress
+  egress:
+    # DNS — resolve external hostnames
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Kubernetes API — Argo executor
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 10.32.0.0/16
+        - ipBlock:
+            cidr: 172.16.0.4/32
+    # External HTTP/HTTPS — Kaniko base image pulls, apt-get, cargo fetches
+    - ports:
+        - port: 80
+          protocol: TCP
+        - port: 443
+          protocol: TCP
+    # GitLab SSH — git clone (compile-service uses ssh://git@git.fxhnt.ai)
+    - ports:
+        - port: 2222
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 100.90.76.85/32
+        - podSelector:
+            matchLabels:
+              app: gitlab-shell
+    # MinIO — binary upload from compile-service
+    - ports:
+        - port: 9000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+    # GitLab registry — Kaniko image push
+    - ports:
+        - port: 5000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app: registry
+    # GitLab webservice — JWT auth for registry tokens
+    - ports:
+        - port: 8181
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app: webservice
+    # Mattermost — exit handler webhook notifications
+    - ports:
+        - port: 8065
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: mattermost
+---
+# ── sanitizer-test + nsys-test + smoke-test workflows: same egress as ci-pipeline ──
+# (git fetch over SSH, MinIO log archive + nsys-rep upload, registry image pull,
+# external HTTPS for mc download)
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: argo-sanitizer-nsys-workflow
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchExpressions:
+      - key: app.kubernetes.io/component
+        operator: In
+        values:
+          - sanitizer-test
+          - nsys-test
+          - smoke-test
+  policyTypes:
+    - Egress
+  egress:
+    # DNS
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Kubernetes API
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 10.32.0.0/16
+        - ipBlock:
+            cidr: 172.16.0.4/32
+    # External HTTPS — mc binary download from dl.min.io
+    - ports:
+        - port: 80
+          protocol: TCP
+        - port: 443
+          protocol: TCP
+    # GitLab SSH — git fetch
+    - ports:
+        - port: 2222
+          protocol: TCP
+      to:
+        - ipBlock:
+            cidr: 100.90.76.85/32
+        - podSelector:
+            matchLabels:
+              app: gitlab-shell
+    # MinIO — log archive + nsys profile upload
+    - ports:
+        - port: 9000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+    # GitLab registry — ci-builder image pull
+    - ports:
+        - port: 5000
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app: registry
+    # GitLab webservice — JWT auth for registry tokens
+    - ports:
+        - port: 8181
+          protocol: TCP
+      to:
+        - podSelector:
+            matchLabels:
+              app: webservice

infra/k8s/argo/ci-deploy-rbac.yaml

@@ -0,0 +1,47 @@
+# RBAC for CI pipeline — deploy step + self-apply Argo templates
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ci-deploy
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+rules:
+  # apps: roll + apply Deployments. watch is required by `kubectl rollout status`.
+  - apiGroups: ["apps"]
+    resources: [deployments]
+    verbs: [get, list, watch, patch]
+  - apiGroups: ["argoproj.io"]
+    resources: [workflowtemplates, eventsources, sensors, eventbus]
+    verbs: [get, list, create, update, patch]
+  # core: services/configmaps, plus serviceaccounts (tailscale-dashboard, dagster) and the forward-track PVC
+  # — added so the fxhnt-cockpit deploy can `kubectl apply` its full manifest set without a partial-apply failure.
+  - apiGroups: [""]
+    resources: [services, configmaps, serviceaccounts, persistentvolumeclaims]
+    verbs: [get, list, create, update, patch]
+  - apiGroups: ["networking.k8s.io"]
+    resources: [networkpolicies]
+    verbs: [get, list, create, update, patch]
+  # batch: the fxhnt-forward CronJob
+  - apiGroups: ["batch"]
+    resources: [cronjobs]
+    verbs: [get, list, create, update, patch]
+  - apiGroups: ["rbac.authorization.k8s.io"]
+    resources: [roles, rolebindings]
+    verbs: [get, list, create, update, patch, bind, escalate]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ci-deploy
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ci-deploy
+subjects:
+  - kind: ServiceAccount
+    name: argo-workflow
+    namespace: foxhunt

infra/k8s/argo/databento-download-template.yaml

@@ -0,0 +1,135 @@
+# Databento data download via streaming API (get_range).
+#
+# Parameterized Argo WorkflowTemplate — replaces the hardcoded K8s Job.
+# Binary pre-uploaded to /data/bin/download_baseline on training-data-pvc.
+# Universe configs loaded from configmap databento-universe-all.
+#
+# Usage:
+#   argo submit -n foxhunt --from=wftmpl/databento-download
+#   argo submit -n foxhunt --from=wftmpl/databento-download -p schema=trades -p output-dir=/data/futures-baseline-trades
+#   ./scripts/argo-download.sh mbp-10 --parallel 9 --pool ci-compile-cpu
+#
+# Databento rate limits: 100 concurrent connections per IP, 100 req/s.
+---
+apiVersion: argoproj.io/v1alpha1
+kind: WorkflowTemplate
+metadata:
+  name: databento-download
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: databento-download
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  entrypoint: download
+  serviceAccountName: argo-workflow
+  podMetadata:
+    labels:
+      app.kubernetes.io/part-of: foxhunt
+      app.kubernetes.io/component: databento-download
+  securityContext:
+    fsGroup: 1000
+  ttlStrategy:
+    secondsAfterCompletion: 3600
+  activeDeadlineSeconds: 86400  # 24 hours
+
+  arguments:
+    parameters:
+      - name: schema
+        value: mbp-10
+        description: "Databento schema: ohlcv-1m, ohlcv-1s, trades, mbp-10"
+      - name: output-dir
+        value: /data/futures-baseline-mbp10
+        description: "Output directory on training-data PVC"
+      - name: parallel
+        value: "9"
+        description: "Number of parallel quarterly downloads (max ~100 per Databento rate limit)"
+      - name: node-pool
+        value: ci-compile-cpu
+        description: "K8s node pool: ci-compile-cpu (32c/64G) or platform (4c/8G)"
+
+  volumes:
+    - name: training-data
+      persistentVolumeClaim:
+        claimName: training-data-pvc
+    - name: universe-configs
+      configMap:
+        name: databento-universe-all
+
+  templates:
+    - name: download
+      inputs:
+        parameters:
+          - name: schema
+          - name: output-dir
+          - name: parallel
+          - name: node-pool
+      nodeSelector:
+        k8s.scaleway.com/pool-name: "{{inputs.parameters.node-pool}}"
+        topology.kubernetes.io/zone: fr-par-2
+      container:
+        image: ubuntu:24.04
+        command: ["/bin/bash", "-c"]
+        env:
+          - name: RUST_LOG
+            value: info
+          - name: DATABENTO_API_KEY
+            valueFrom:
+              secretKeyRef:
+                name: databento-credentials
+                key: api-key
+        resources:
+          requests:
+            cpu: 500m
+            memory: 512Mi
+          limits:
+            cpu: "2"
+            memory: 4Gi
+        volumeMounts:
+          - name: training-data
+            mountPath: /data
+          - name: universe-configs
+            mountPath: /configs
+        args:
+          - |
+            set -e
+            apt-get update -qq > /dev/null 2>&1 && apt-get install -y -qq ca-certificates > /dev/null 2>&1
+
+            BINARY=/data/bin/download_baseline
+            chmod +x "$BINARY"
+
+            SCHEMA="{{inputs.parameters.schema}}"
+            OUTPUT_DIR="{{inputs.parameters.output-dir}}"
+            PARALLEL="{{inputs.parameters.parallel}}"
+
+            # Map schema name to configmap key
+            CONFIG="/configs/${SCHEMA}.toml"
+            if [ ! -f "$CONFIG" ]; then
+              echo "ERROR: Config not found: $CONFIG"
+              echo "Available configs:"
+              ls /configs/
+              exit 1
+            fi
+
+            mkdir -p "$OUTPUT_DIR"
+
+            echo "=== Databento Download ==="
+            echo "Schema:    $SCHEMA"
+            echo "Config:    $CONFIG"
+            echo "Output:    $OUTPUT_DIR"
+            echo "Parallel:  $PARALLEL"
+            echo "Node:      $(hostname), CPUs: $(nproc), RAM: $(free -g | awk '/Mem:/{print $2}')G"
+            echo ""
+
+            $BINARY \
+              --universe-config "$CONFIG" \
+              --output-dir "$OUTPUT_DIR" \
+              --parallel "$PARALLEL" \
+              --yes
+
+            echo ""
+            echo "=== Download complete ==="
+            echo "Files:"
+            find "$OUTPUT_DIR" -name '*.dbn.zst' -exec ls -lh {} \;
+            echo ""
+            du -sh "$OUTPUT_DIR"
+            df -h /data

infra/k8s/argo/dns-bootstrap-policy.yaml

@@ -0,0 +1,25 @@
+# DNS Bootstrap Prevention — Prevent CoreDNS chicken-and-egg deadlock
+#
+# ROOT CAUSE (2026-03-18 incident):
+# 1. Rolling replacement of ALL platform nodes simultaneously
+# 2. New nodes boot with /etc/resolv.conf → 10.32.0.10 (kube-dns only)
+# 3. CoreDNS pods need to pull images → need DNS → need CoreDNS
+# 4. Cilium BPF socket LB returns EPERM when kube-dns has no endpoints
+# 5. Complete cluster DNS deadlock — nothing can pull images
+#
+# PREVENTION:
+# 1. VPC DefaultRoutePropagation MUST be enabled (API, not Terraform yet)
+# 2. NEVER replace ALL nodes at once — one at a time, wait for CoreDNS
+# 3. Do NOT create CiliumClusterwideNetworkPolicy with egress rules
+#    (enables default deny, makes the problem worse)
+# 4. bpf-lb-sock MUST remain "true" — disabling breaks ALL service routing
+# 5. Keep bastion enabled on VPC gateway for emergency SSH access
+#
+# RECOVERY PROCEDURE:
+# 1. SSH via bastion: ssh -o ProxyCommand="ssh -W %h:%p -p 61000 bastion@<gw-public-ip>" root@<node-private-ip>
+# 2. On the node: /usr/local/bin/crictl pull <coredns-image>
+# 3. kubectl delete pods -l k8s-app=kube-dns --force
+# 4. CoreDNS starts from cached image → DNS recovers → cascade
+#
+# DATA PROTECTION:
+# All PVs set to Retain reclaim policy — data survives cluster deletion

infra/k8s/argo/events/eventbus.yaml

@@ -0,0 +1,10 @@
+apiVersion: argoproj.io/v1alpha1
+kind: EventBus
+metadata:
+  name: default
+  namespace: foxhunt
+spec:
+  nats:
+    native:
+      replicas: 3
+      auth: token

infra/k8s/argo/events/eventsource-svc.yaml

@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: gitlab-push-eventsource-svc
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: gitlab-push-eventsource
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  type: ClusterIP
+  ports:
+    - port: 12000
+      targetPort: 12000
+      protocol: TCP
+  selector:
+    eventsource-name: gitlab-push

infra/k8s/argo/events/gitea-deploy-sensor.yaml

@@ -0,0 +1,56 @@
+# On push to fxhnt main/master in Gitea, submit the fxhnt-cockpit build+deploy workflow (Phase 2B).
+# Gitea push payload: { "ref": "refs/heads/main", "after": "<new-sha>", ... }
+apiVersion: argoproj.io/v1alpha1
+kind: Sensor
+metadata:
+  name: gitea-deploy
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: gitea-deploy
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  template:
+    # pod intentionally NOT part-of:foxhunt (avoids default-deny-all; matches working sensors)
+    serviceAccountName: argo-workflow
+    nodeSelector:
+      k8s.scaleway.com/pool-name: platform
+  dependencies:
+    - name: gitea-push-dep
+      eventSourceName: gitea-push
+      eventName: push
+      filters:
+        data:
+          - path: body.ref
+            type: string
+            value:
+              - "refs/heads/main"
+              - "refs/heads/master"
+  triggers:
+    - template:
+        name: deploy-cockpit
+        conditions: gitea-push-dep
+        argoWorkflow:
+          operation: submit
+          source:
+            resource:
+              apiVersion: argoproj.io/v1alpha1
+              kind: Workflow
+              metadata:
+                generateName: fxhnt-cockpit-
+                namespace: foxhunt
+              spec:
+                serviceAccountName: argo-workflow
+                podGC:
+                  strategy: OnPodCompletion
+                ttlStrategy:
+                  secondsAfterCompletion: 3600
+                workflowTemplateRef:
+                  name: fxhnt-cockpit
+                arguments:
+                  parameters:
+                    - name: commit-sha
+          parameters:
+            - src:
+                dependencyName: gitea-push-dep
+                dataKey: body.after
+              dest: spec.arguments.parameters.0.value

infra/k8s/argo/events/gitea-push-eventsource.yaml

@@ -0,0 +1,28 @@
+# Gitea push webhook → Argo Events. Replaces the old gitlab-push eventsource (Phase 2B).
+# Gitea posts to gitea-push-eventsource-svc:12000/push on push to the fxhnt repo.
+apiVersion: argoproj.io/v1alpha1
+kind: EventSource
+metadata:
+  name: gitea-push
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: gitea-push
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  template:
+    # NOTE: pod is intentionally NOT part-of:foxhunt — that label triggers default-deny-all. The working
+    # workflow-trigger/gitlab-push eventsource pods have no part-of, so they stay default-allow.
+    metadata:
+      labels:
+        app.kubernetes.io/name: gitea-push
+    nodeSelector:
+      k8s.scaleway.com/pool-name: platform
+  service:
+    ports:
+      - port: 12000
+        targetPort: 12000
+  webhook:
+    push:
+      port: "12000"
+      endpoint: /push
+      method: POST

infra/k8s/argo/events/rbac.yaml

@@ -0,0 +1,29 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: argo-sensor-submit
+  namespace: foxhunt
+rules:
+  - apiGroups: ["argoproj.io"]
+    resources: ["workflows"]
+    verbs: ["create", "get", "list", "watch"]
+  - apiGroups: ["argoproj.io"]
+    resources: ["workflowtemplates"]
+    verbs: ["get", "list"]
+  - apiGroups: ["argoproj.io"]
+    resources: ["workflowtaskresults"]
+    verbs: ["create", "patch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: argo-sensor-submit
+  namespace: foxhunt
+subjects:
+  - kind: ServiceAccount
+    name: argo-workflow
+    namespace: foxhunt
+roleRef:
+  kind: Role
+  name: argo-sensor-submit
+  apiGroup: rbac.authorization.k8s.io

infra/k8s/argo/events/workflow-trigger-eventsource.yaml

@@ -0,0 +1,51 @@
+# Manual workflow trigger EventSource.
+#
+# Webhook endpoints for triggering compile/deploy, training, and image builds.
+# Each endpoint accepts a JSON POST body with workflow-specific parameters.
+#
+# Endpoints (all on port 12001):
+#   POST /compile-deploy    — {"commit_sha": "HEAD"}
+#   POST /train/dqn         — {"binary_tag": "latest"}
+#   POST /train/ppo         — {"binary_tag": "latest"}
+#   POST /train/supervised  — {"model": "tft", "binary_tag": "latest"}
+#   POST /build-image       — {"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}
+#
+# Example:
+#   curl -X POST http://workflow-trigger-eventsource-svc.foxhunt:12001/build-image \
+#     -H 'Content-Type: application/json' \
+#     -d '{"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}'
+---
+apiVersion: argoproj.io/v1alpha1
+kind: EventSource
+metadata:
+  name: workflow-trigger
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: workflow-trigger
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  eventBusName: default
+  template:
+    nodeSelector:
+      k8s.scaleway.com/pool-name: platform
+  webhook:
+    compile-deploy:
+      port: "12001"
+      endpoint: /compile-deploy
+      method: POST
+    train-dqn:
+      port: "12001"
+      endpoint: /train/dqn
+      method: POST
+    train-ppo:
+      port: "12001"
+      endpoint: /train/ppo
+      method: POST
+    train-supervised:
+      port: "12001"
+      endpoint: /train/supervised
+      method: POST
+    build-image:
+      port: "12001"
+      endpoint: /build-image
+      method: POST

infra/k8s/argo/events/workflow-trigger-sensor.yaml

@@ -0,0 +1,217 @@
+# Sensor for manual workflow triggers.
+#
+# Listens on workflow-trigger EventSource and submits the appropriate workflow
+# when a webhook endpoint is hit. Each trigger maps JSON body fields to
+# workflow parameters.
+#
+# Trigger reference:
+#   compile-deploy    — POST /compile-deploy    {"commit_sha": "..."}
+#   train-dqn         — POST /train/dqn         {"binary_tag": "..."}
+#   train-ppo         — POST /train/ppo         {"binary_tag": "..."}
+#   train-supervised  — POST /train/supervised  {"model": "...", "binary_tag": "..."}
+#   build-image       — POST /build-image       {"dockerfile": "...", "image_name": "...", "commit_sha": "..."}
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Sensor
+metadata:
+  name: workflow-trigger
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: workflow-trigger
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  template:
+    serviceAccountName: argo-workflow
+    nodeSelector:
+      k8s.scaleway.com/pool-name: platform
+  eventBusName: default
+
+  dependencies:
+    - name: compile-deploy-dep
+      eventSourceName: workflow-trigger
+      eventName: compile-deploy
+    - name: train-dqn-dep
+      eventSourceName: workflow-trigger
+      eventName: train-dqn
+    - name: train-ppo-dep
+      eventSourceName: workflow-trigger
+      eventName: train-ppo
+    - name: train-supervised-dep
+      eventSourceName: workflow-trigger
+      eventName: train-supervised
+    - name: build-image-dep
+      eventSourceName: workflow-trigger
+      eventName: build-image
+
+  triggers:
+    # ── Compile + Deploy ──
+    # POST /compile-deploy {"commit_sha": "abc1234"}
+    # Omit service_packages to build all 7 services (default).
+    - template:
+        name: compile-deploy
+        conditions: compile-deploy-dep
+        argoWorkflow:
+          operation: submit
+          source:
+            resource:
+              apiVersion: argoproj.io/v1alpha1
+              kind: Workflow
+              metadata:
+                generateName: compile-deploy-
+                namespace: foxhunt
+              spec:
+                serviceAccountName: argo-workflow
+                podGC:
+                  strategy: OnPodCompletion
+                ttlStrategy:
+                  secondsAfterCompletion: 3600
+                workflowTemplateRef:
+                  name: compile-and-deploy
+                arguments:
+                  parameters:
+                    - name: commit-sha
+          parameters:
+            - src:
+                dependencyName: compile-deploy-dep
+                dataKey: body.commit_sha
+              dest: spec.arguments.parameters.0.value
+
+    # ── Train DQN ──
+    # POST /train/dqn {"binary_tag": "latest"}
+    - template:
+        name: train-dqn
+        conditions: train-dqn-dep
+        argoWorkflow:
+          operation: submit
+          source:
+            resource:
+              apiVersion: argoproj.io/v1alpha1
+              kind: Workflow
+              metadata:
+                generateName: train-dqn-
+                namespace: foxhunt
+              spec:
+                serviceAccountName: argo-workflow
+                podGC:
+                  strategy: OnPodCompletion
+                ttlStrategy:
+                  secondsAfterCompletion: 3600
+                workflowTemplateRef:
+                  name: train-dqn
+                arguments:
+                  parameters:
+                    - name: binary-tag
+          parameters:
+            - src:
+                dependencyName: train-dqn-dep
+                dataKey: body.binary_tag
+              dest: spec.arguments.parameters.0.value
+
+    # ── Train PPO ──
+    # POST /train/ppo {"binary_tag": "latest"}
+    - template:
+        name: train-ppo
+        conditions: train-ppo-dep
+        argoWorkflow:
+          operation: submit
+          source:
+            resource:
+              apiVersion: argoproj.io/v1alpha1
+              kind: Workflow
+              metadata:
+                generateName: train-ppo-
+                namespace: foxhunt
+              spec:
+                serviceAccountName: argo-workflow
+                podGC:
+                  strategy: OnPodCompletion
+                ttlStrategy:
+                  secondsAfterCompletion: 3600
+                workflowTemplateRef:
+                  name: train-ppo
+                arguments:
+                  parameters:
+                    - name: binary-tag
+          parameters:
+            - src:
+                dependencyName: train-ppo-dep
+                dataKey: body.binary_tag
+              dest: spec.arguments.parameters.0.value
+
+    # ── Train Supervised ──
+    # POST /train/supervised {"model": "tft", "binary_tag": "latest"}
+    # model: one of tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
+    - template:
+        name: train-supervised
+        conditions: train-supervised-dep
+        argoWorkflow:
+          operation: submit
+          source:
+            resource:
+              apiVersion: argoproj.io/v1alpha1
+              kind: Workflow
+              metadata:
+                generateName: train-supervised-
+                namespace: foxhunt
+              spec:
+                serviceAccountName: argo-workflow
+                podGC:
+                  strategy: OnPodCompletion
+                ttlStrategy:
+                  secondsAfterCompletion: 3600
+                workflowTemplateRef:
+                  name: train-supervised
+                arguments:
+                  parameters:
+                    - name: model
+                    - name: binary-tag
+          parameters:
+            - src:
+                dependencyName: train-supervised-dep
+                dataKey: body.model
+              dest: spec.arguments.parameters.0.value
+            - src:
+                dependencyName: train-supervised-dep
+                dataKey: body.binary_tag
+              dest: spec.arguments.parameters.1.value
+
+    # ── Build Image ──
+    # POST /build-image {"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}
+    - template:
+        name: build-image
+        conditions: build-image-dep
+        argoWorkflow:
+          operation: submit
+          source:
+            resource:
+              apiVersion: argoproj.io/v1alpha1
+              kind: Workflow
+              metadata:
+                generateName: build-image-
+                namespace: foxhunt
+              spec:
+                serviceAccountName: argo-workflow
+                podGC:
+                  strategy: OnPodCompletion
+                ttlStrategy:
+                  secondsAfterCompletion: 3600
+                workflowTemplateRef:
+                  name: build-ci-image
+                arguments:
+                  parameters:
+                    - name: commit-sha
+                    - name: dockerfile
+                    - name: image-name
+          parameters:
+            - src:
+                dependencyName: build-image-dep
+                dataKey: body.commit_sha
+              dest: spec.arguments.parameters.0.value
+            - src:
+                dependencyName: build-image-dep
+                dataKey: body.dockerfile
+              dest: spec.arguments.parameters.1.value
+            - src:
+                dependencyName: build-image-dep
+                dataKey: body.image_name
+              dest: spec.arguments.parameters.2.value

infra/k8s/argo/events/workflow-trigger-svc.yaml

@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: workflow-trigger-eventsource-svc
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: workflow-trigger
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  type: ClusterIP
+  ports:
+    - port: 12001
+      targetPort: 12001
+      protocol: TCP
+  selector:
+    eventsource-name: workflow-trigger

infra/k8s/argo/feature-cache-pvc.yaml

@@ -0,0 +1,18 @@
+# Dedicated PVC for precomputed .fxcache files.
+# Survives workflow completions (unlike workspace volumeClaimTemplate).
+# Precompute writes here, all training steps read from here.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: feature-cache-pvc
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: feature-cache
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: scw-bssd-retain
+  resources:
+    requests:
+      storage: 10Gi

infra/k8s/argo/kustomization.yaml

@@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+# Rust train/CI/cache templates removed 2026-06-21 (decommission-rust-infra).
+# Remaining: platform RBAC + network policy used by the active fxhnt-cockpit deploys.
+resources:
+  - argo-workflow-netpol.yaml
+  - ci-deploy-rbac.yaml
+  - archive-rbac.yaml

infra/k8s/argo/notification-webhook.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: notification-webhook
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+type: Opaque
+stringData:
+  # Mattermost incoming webhook URL — update after Mattermost is deployed
+  webhook-url: "http://mattermost.foxhunt.svc.cluster.local:8065/hooks/PLACEHOLDER"

infra/k8s/argo/values.yaml

@@ -0,0 +1,80 @@
+# Argo Workflows Helm values — foxhunt cluster
+# Chart: argo/argo-workflows v1.0.2 (Argo v4.0.2)
+# Install: helm upgrade argo-workflows argo/argo-workflows -n foxhunt -f infra/k8s/argo/values.yaml
+
+controller:
+  nodeSelector:
+    k8s.scaleway.com/pool-name: platform
+  resources:
+    requests:
+      cpu: 50m
+      memory: 128Mi
+    limits:
+      cpu: 200m
+      memory: 256Mi
+  # Only watch foxhunt namespace (not cluster-wide)
+  workflowNamespaces:
+    - foxhunt
+  # Workflow archive — persists workflow state + logs to PostgreSQL after pod GC
+  persistence:
+    archive: true
+    postgresql:
+      host: postgres.foxhunt.svc.cluster.local
+      port: 5432
+      database: argo_archive
+      tableName: argo_workflows
+      userNameSecret:
+        name: argo-postgres-config
+        key: username
+      passwordSecret:
+        name: argo-postgres-config
+        key: password
+      ssl: false
+      sslMode: disable
+
+server:
+  nodeSelector:
+    k8s.scaleway.com/pool-name: platform
+  resources:
+    requests:
+      cpu: 25m
+      memory: 64Mi
+    limits:
+      cpu: 100m
+      memory: 128Mi
+  ingress:
+    enabled: false
+  extraArgs:
+    - --auth-mode=server
+    - --secure=false
+
+# Default artifact repository — MinIO (in-cluster)
+artifactRepository:
+  archiveLogs: true
+  s3:
+    endpoint: minio.foxhunt.svc.cluster.local:9000
+    insecure: true
+    bucket: argo-logs
+    accessKeySecret:
+      name: minio-credentials
+      key: access-key
+    secretKeySecret:
+      name: minio-credentials
+      key: secret-key
+
+executor:
+  resources:
+    requests:
+      cpu: 10m
+      memory: 32Mi
+    limits:
+      cpu: 100m
+      memory: 64Mi
+
+workflow:
+  serviceAccount:
+    create: true
+    name: argo-workflow
+
+crds:
+  install: true

infra/k8s/cert-manager/fxhnt-acme.yaml

@@ -0,0 +1,72 @@
+# ACME (Let's Encrypt) wildcard cert for *.fxhnt.ai via cert-manager + Scaleway DNS-01.
+# Replaces the prior MANUAL, unrenewed LE cert that expired 2026-05-26 (the tailscale-proxy
+# fell back to the GitLab self-signed cert). DNS-01 is required: the services are tailnet-only
+# (*.fxhnt.ai -> 100.x Tailscale CGNAT) so HTTP-01 is unreachable, and wildcards need DNS-01.
+#
+# Solver creds: cert-manager/scaleway-dns-credentials (SCW_ACCESS_KEY/SCW_SECRET_KEY, copied from
+# foxhunt/scaleway-credentials). Webhook: scaleway-certmanager-webhook (helm, cert-manager ns).
+# The issued secret gitlab-tls-cert is mounted by infra/k8s/gitlab/tailscale-proxy.yaml.
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-staging
+spec:
+  acme:
+    email: jeroen@grusewski.nl
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-staging-account-key
+    solvers:
+      - dns01:
+          webhook:
+            groupName: acme.scaleway.com
+            solverName: scaleway
+            config:
+              accessKeySecretRef:
+                name: scaleway-dns-credentials
+                key: SCW_ACCESS_KEY
+              secretKeySecretRef:
+                name: scaleway-dns-credentials
+                key: SCW_SECRET_KEY
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+spec:
+  acme:
+    email: jeroen@grusewski.nl
+    server: https://acme-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-prod-account-key
+    solvers:
+      - dns01:
+          webhook:
+            groupName: acme.scaleway.com
+            solverName: scaleway
+            config:
+              accessKeySecretRef:
+                name: scaleway-dns-credentials
+                key: SCW_ACCESS_KEY
+              secretKeySecretRef:
+                name: scaleway-dns-credentials
+                key: SCW_SECRET_KEY
+---
+# Wildcard cert for the tailscale-proxy. issuerRef flips staging->prod after staging validates;
+# secretName flips to gitlab-tls-cert (the secret the proxy mounts) for the prod issuance.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: fxhnt-wildcard
+  namespace: foxhunt
+spec:
+  secretName: gitlab-tls-cert          # takes over the secret the tailscale-proxy mounts
+  privateKey:
+    rotationPolicy: Always             # old manual key had a mismatching algorithm; regenerate on issue/renew
+  dnsNames:
+    - fxhnt.ai
+    - "*.fxhnt.ai"
+  issuerRef:
+    name: letsencrypt-prod
+    kind: ClusterIssuer

infra/k8s/databases/postgres-backup.yaml

@@ -0,0 +1,249 @@
+# Postgres backup — daily pg_dump to MinIO S3
+# Uses initContainer (postgres:16-alpine) for pg_dump, then main container
+# (python:3.12-alpine, already cached) uploads to MinIO via S3 API.
+# Retains 7 daily + 4 weekly backups.
+# Apply: kubectl apply -f postgres-backup.yaml
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: postgres-backup-script
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: postgres-backup
+    app.kubernetes.io/part-of: foxhunt
+data:
+  upload.py: |
+    #!/usr/bin/env python3
+    """Upload pg_dump backup to MinIO, prune old backups."""
+    import hashlib, hmac, os, sys, glob, urllib.request, urllib.error
+    from datetime import datetime, timezone
+
+    ENDPOINT = os.environ["MINIO_ENDPOINT"]  # http://minio.foxhunt...:9000
+    ACCESS_KEY = os.environ["MINIO_ACCESS_KEY"]
+    SECRET_KEY = os.environ["MINIO_SECRET_KEY"]
+    BUCKET = "foxhunt-backups"
+    BACKUP_DIR = "/backup"
+
+    def s3_sign(method, path, headers, payload_hash):
+        """AWS Signature V4 for MinIO."""
+        now = datetime.now(timezone.utc)
+        date_stamp = now.strftime("%Y%m%d")
+        amz_date = now.strftime("%Y%m%dT%H%M%SZ")
+        region = "us-east-1"
+        service = "s3"
+        scope = f"{date_stamp}/{region}/{service}/aws4_request"
+
+        headers["x-amz-date"] = amz_date
+        headers["x-amz-content-sha256"] = payload_hash
+
+        signed_headers = ";".join(sorted(headers.keys()))
+        canonical_headers = "".join(f"{k}:{headers[k]}\n" for k in sorted(headers.keys()))
+        canonical_request = f"{method}\n{path}\n\n{canonical_headers}\n{signed_headers}\n{payload_hash}"
+
+        string_to_sign = f"AWS4-HMAC-SHA256\n{amz_date}\n{scope}\n{hashlib.sha256(canonical_request.encode()).hexdigest()}"
+
+        def sign(key, msg):
+            return hmac.new(key, msg.encode(), hashlib.sha256).digest()
+
+        signing_key = sign(sign(sign(sign(
+            f"AWS4{SECRET_KEY}".encode(), date_stamp), region), service), "aws4_request")
+        signature = hmac.new(signing_key, string_to_sign.encode(), hashlib.sha256).hexdigest()
+
+        headers["Authorization"] = (
+            f"AWS4-HMAC-SHA256 Credential={ACCESS_KEY}/{scope}, "
+            f"SignedHeaders={signed_headers}, Signature={signature}"
+        )
+        return headers
+
+    def s3_put(key, data):
+        path = f"/{BUCKET}/{key}"
+        url = f"{ENDPOINT}{path}"
+        payload_hash = hashlib.sha256(data).hexdigest()
+        host = ENDPOINT.split("//")[1]
+        headers = {"host": host, "content-length": str(len(data))}
+        headers = s3_sign("PUT", path, headers, payload_hash)
+        req = urllib.request.Request(url, data=data, method="PUT", headers=headers)
+        urllib.request.urlopen(req, timeout=120)
+
+    def s3_list(prefix):
+        path = f"/{BUCKET}?prefix={prefix}&list-type=2"
+        url = f"{ENDPOINT}{path}"
+        host = ENDPOINT.split("//")[1]
+        headers = {"host": host}
+        headers = s3_sign("GET", f"/{BUCKET}", headers, hashlib.sha256(b"").hexdigest())
+        req = urllib.request.Request(f"{url}", method="GET", headers=headers)
+        try:
+            resp = urllib.request.urlopen(req, timeout=30).read().decode()
+            import re
+            return sorted(re.findall(r"<Key>([^<]+)</Key>", resp))
+        except Exception:
+            return []
+
+    def s3_delete(key):
+        path = f"/{BUCKET}/{key}"
+        url = f"{ENDPOINT}{path}"
+        host = ENDPOINT.split("//")[1]
+        headers = {"host": host}
+        headers = s3_sign("DELETE", path, headers, hashlib.sha256(b"").hexdigest())
+        req = urllib.request.Request(url, method="DELETE", headers=headers)
+        urllib.request.urlopen(req, timeout=30)
+
+    def s3_ensure_bucket():
+        path = f"/{BUCKET}"
+        url = f"{ENDPOINT}{path}"
+        host = ENDPOINT.split("//")[1]
+        headers = {"host": host}
+        headers = s3_sign("PUT", path, headers, hashlib.sha256(b"").hexdigest())
+        req = urllib.request.Request(url, method="PUT", headers=headers)
+        try:
+            urllib.request.urlopen(req, timeout=30)
+        except urllib.error.HTTPError as e:
+            if e.code not in (409, 200):  # 409 = bucket exists
+                raise
+
+    # Find the backup file
+    files = glob.glob(f"{BACKUP_DIR}/*.sql.gz")
+    if not files:
+        print("ERROR: No backup file found in /backup/")
+        sys.exit(1)
+
+    backup_file = files[0]
+    filename = os.path.basename(backup_file)
+    size_mb = os.path.getsize(backup_file) / (1024 * 1024)
+    print(f"Uploading {filename} ({size_mb:.1f} MB)")
+
+    s3_ensure_bucket()
+
+    with open(backup_file, "rb") as f:
+        data = f.read()
+
+    # Upload daily
+    s3_put(f"daily/{filename}", data)
+    print(f"  Uploaded daily/{filename}")
+
+    # On Sundays, also upload weekly
+    if datetime.now(timezone.utc).isoweekday() == 7:
+        s3_put(f"weekly/{filename}", data)
+        print(f"  Uploaded weekly/{filename}")
+
+    # Prune old dailies (keep 7)
+    dailies = s3_list("daily/")
+    if len(dailies) > 7:
+        for old in dailies[:len(dailies) - 7]:
+            s3_delete(old)
+            print(f"  Pruned {old}")
+
+    # Prune old weeklies (keep 4)
+    weeklies = s3_list("weekly/")
+    if len(weeklies) > 4:
+        for old in weeklies[:len(weeklies) - 4]:
+            s3_delete(old)
+            print(f"  Pruned {old}")
+
+    print("=== Backup upload complete ===")
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: postgres-backup
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: postgres-backup
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  schedule: "30 2 * * *"
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 600
+      template:
+        metadata:
+          labels:
+            app: postgres-backup
+            app.kubernetes.io/part-of: foxhunt
+        spec:
+          nodeSelector:
+            k8s.scaleway.com/pool-name: platform
+          restartPolicy: OnFailure
+          # Step 1: pg_dump in initContainer → shared /backup volume
+          initContainers:
+            - name: pg-dump
+              image: postgres:16-alpine
+              command:
+                - sh
+                - -c
+                - |
+                  TIMESTAMP=$(date -u +%Y%m%d-%H%M%S)
+                  echo "=== Postgres Backup ${TIMESTAMP} ==="
+                  PGPASSWORD="${POSTGRES_PASSWORD}" pg_dumpall \
+                    -h "${POSTGRES_HOST}" \
+                    -p 5432 \
+                    -U "${POSTGRES_USER}" \
+                    --clean --if-exists \
+                    | gzip > "/backup/foxhunt-${TIMESTAMP}.sql.gz"
+                  ls -lh /backup/
+                  echo "Dump complete"
+              env:
+                - name: POSTGRES_HOST
+                  value: "postgres.foxhunt.svc.cluster.local"
+                - name: POSTGRES_USER
+                  value: "foxhunt"
+                - name: POSTGRES_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: db-credentials
+                      key: password
+              volumeMounts:
+                - name: backup
+                  mountPath: /backup
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 128Mi
+                limits:
+                  cpu: 500m
+                  memory: 512Mi
+          # Step 2: Upload to MinIO in main container (python cached on nodes)
+          containers:
+            - name: upload
+              image: python:3.12-alpine
+              command: ["python3", "-u", "/scripts/upload.py"]
+              env:
+                - name: MINIO_ENDPOINT
+                  value: "http://minio.foxhunt.svc.cluster.local:9000"
+                - name: MINIO_ACCESS_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: minio-credentials
+                      key: access-key
+                - name: MINIO_SECRET_KEY
+                  valueFrom:
+                    secretKeyRef:
+                      name: minio-credentials
+                      key: secret-key
+              volumeMounts:
+                - name: backup
+                  mountPath: /backup
+                - name: script
+                  mountPath: /scripts
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 64Mi
+                limits:
+                  cpu: 100m
+                  memory: 128Mi
+          volumes:
+            - name: backup
+              emptyDir:
+                sizeLimit: 2Gi
+            - name: script
+              configMap:
+                name: postgres-backup-script
+                items:
+                  - key: upload.py
+                    path: upload.py
+                defaultMode: 0755

infra/k8s/databases/postgres.yaml

@@ -0,0 +1,96 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: postgres-pvc
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: postgres
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 20Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: postgres
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: postgres
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: postgres
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: postgres
+        app.kubernetes.io/part-of: foxhunt
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: infra
+      containers:
+        - name: postgres
+          image: timescale/timescaledb:latest-pg16
+          ports:
+            - containerPort: 5432
+              name: postgres
+          args:
+            - -c
+            - max_connections=100
+          env:
+            - name: POSTGRES_DB
+              value: foxhunt
+            - name: POSTGRES_USER
+              value: foxhunt
+            - name: POSTGRES_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: db-credentials
+                  key: password
+          volumeMounts:
+            - name: postgres-data
+              mountPath: /var/lib/postgresql/data
+              subPath: pgdata
+          readinessProbe:
+            exec:
+              command:
+                - pg_isready
+                - -U
+                - foxhunt
+            initialDelaySeconds: 10
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: 500m
+              memory: 1Gi
+      volumes:
+        - name: postgres-data
+          persistentVolumeClaim:
+            claimName: postgres-pvc
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: postgres
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: postgres
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    app.kubernetes.io/name: postgres
+  ports:
+    - port: 5432
+      targetPort: 5432
+      name: postgres

infra/k8s/databases/questdb.yaml

@@ -0,0 +1,95 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: questdb-pvc
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: questdb
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: questdb
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: questdb
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: questdb
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: questdb
+        app.kubernetes.io/part-of: foxhunt
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: platform
+      containers:
+        - name: questdb
+          image: questdb/questdb:8.2.3
+          ports:
+            - containerPort: 9009
+              name: ilp
+            - containerPort: 8812
+              name: pg
+            - containerPort: 9003
+              name: metrics
+          env:
+            - name: QDB_PG_ENABLED
+              value: "true"
+            - name: QDB_METRICS_ENABLED
+              value: "TRUE"
+          volumeMounts:
+            - name: questdb-data
+              mountPath: /var/lib/questdb
+          readinessProbe:
+            httpGet:
+              path: /exec?query=SELECT%201
+              port: 9003
+            initialDelaySeconds: 10
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: 500m
+              memory: 1Gi
+      volumes:
+        - name: questdb-data
+          persistentVolumeClaim:
+            claimName: questdb-pvc
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: questdb
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: questdb
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    app.kubernetes.io/name: questdb
+  ports:
+    - port: 9009
+      targetPort: 9009
+      name: ilp
+    - port: 8812
+      targetPort: 8812
+      name: pg
+    - port: 9003
+      targetPort: 9003
+      name: metrics

infra/k8s/databases/redis.yaml

@@ -0,0 +1,63 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: redis
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: redis
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: redis
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: redis
+        app.kubernetes.io/part-of: foxhunt
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: infra
+      containers:
+        - name: redis
+          image: redis:7-alpine
+          command:
+            - redis-server
+            - --maxmemory
+            - 512mb
+            - --maxmemory-policy
+            - allkeys-lru
+          ports:
+            - containerPort: 6379
+              name: redis
+          readinessProbe:
+            exec:
+              command:
+                - redis-cli
+                - ping
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: 250m
+              memory: 640Mi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: redis
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    app.kubernetes.io/name: redis
+  ports:
+    - port: 6379
+      targetPort: 6379
+      name: redis

infra/k8s/dns/node-local-dns.yaml

@@ -0,0 +1,182 @@
+# Node Bootstrap DaemonSet for Kapsule
+#
+# Two problems on Kapsule nodes:
+# 1. Nodes have nameserver 127.0.53.53 which can't resolve .svc.cluster.local
+# 2. Containerd defaults to HTTPS for non-localhost registries
+#
+# Fixes:
+# 1. Point /etc/resolv.conf to kube-dns ClusterIP (10.32.0.10)
+# 2. Create /etc/containerd/certs.d/ config for HTTP GitLab registry
+# 3. On GPU nodes (containerd v3 with conf.d/), add registry drop-in
+#
+# Init container sets up containerd registry config (one-time).
+# Main container manages resolv.conf (watches for Kapsule resets).
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: node-bootstrap
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: node-bootstrap
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: node-bootstrap
+  updateStrategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 1
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: node-bootstrap
+    spec:
+      hostNetwork: true
+      dnsPolicy: Default
+      priorityClassName: system-node-critical
+      tolerations:
+        - operator: Exists
+      hostPID: true
+      initContainers:
+        # Configure containerd to use HTTP for the GitLab registry.
+        # On GPU nodes (NVIDIA overlay), also creates a v3-compatible conf.d drop-in
+        # and restarts containerd so it picks up the new config.
+        - name: setup-registry
+          image: alpine:3.19
+          securityContext:
+            privileged: true
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -e
+              NEEDS_RESTART=false
+
+              # 1. Create per-host containerd registry config (HTTP for GitLab)
+              REGISTRY_DIR="/host-etc/containerd/certs.d/gitlab-registry.foxhunt.svc.cluster.local:5000"
+              mkdir -p "$REGISTRY_DIR"
+              DESIRED='server = "http://gitlab-registry.foxhunt.svc.cluster.local:5000"
+
+              [host."http://gitlab-registry.foxhunt.svc.cluster.local:5000"]
+                capabilities = ["pull", "resolve"]'
+              DESIRED=$(printf '%s' "$DESIRED" | sed 's/^              //')
+              if [ ! -f "$REGISTRY_DIR/hosts.toml" ] || [ "$(cat "$REGISTRY_DIR/hosts.toml")" != "$DESIRED" ]; then
+                printf '%s\n' "$DESIRED" > "$REGISTRY_DIR/hosts.toml"
+                echo "Created hosts.toml for HTTP gitlab-registry"
+                NEEDS_RESTART=true
+              else
+                echo "hosts.toml already up to date"
+              fi
+
+              # 2. Ensure conf.d/ exists and always write the v3 registry drop-in.
+              #    containerd 2.x may ignore the v2-style grpc.v1.cri.registry.config_path
+              #    even when it's set in config.toml. The v1 images drop-in is the reliable path.
+              #    Create conf.d/ if missing (base config already has imports = ["conf.d/*.toml"]).
+              CONF_D="/host-etc/containerd/conf.d"
+              mkdir -p "$CONF_D"
+              if [ -d "$CONF_D" ]; then
+                DROPIN='version = 3
+
+              [plugins."io.containerd.cri.v1.images".registry]
+                config_path = "/etc/containerd/certs.d"'
+                DROPIN=$(printf '%s' "$DROPIN" | sed 's/^              //')
+                if [ ! -f "$CONF_D/50-registry.toml" ] || [ "$(cat "$CONF_D/50-registry.toml")" != "$DROPIN" ]; then
+                  printf '%s\n' "$DROPIN" > "$CONF_D/50-registry.toml"
+                  echo "GPU node: created v3 registry config drop-in"
+                  NEEDS_RESTART=true
+                else
+                  echo "GPU node: v3 drop-in already up to date"
+                fi
+              fi
+
+              # 3. Restart containerd only if we changed config files.
+              #    Uses /proc/1/root chroot to access host systemctl (requires hostPID + privileged).
+              #    On a new node this will kill the DaemonSet pod; kubelet recreates it,
+              #    second run finds files unchanged (NEEDS_RESTART=false), proceeds normally.
+              if [ "$NEEDS_RESTART" = "true" ]; then
+                echo "Restarting containerd to pick up config changes..."
+                chroot /proc/1/root systemctl restart containerd || \
+                  echo "WARNING: containerd restart failed (node may need manual restart)"
+                echo "containerd restart issued"
+              fi
+          resources:
+            requests:
+              cpu: 10m
+              memory: 8Mi
+            limits:
+              cpu: 50m
+              memory: 16Mi
+          volumeMounts:
+            - name: host-etc
+              mountPath: /host-etc
+      containers:
+        - name: resolv-manager
+          image: busybox:1.37
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -e
+              RESOLV="/host-etc/resolv.conf"
+              BACKUP="/host-etc/resolv.conf.pre-kube-dns"
+              KUBE_DNS="10.32.0.10"
+
+              restore() {
+                if [ -f "$BACKUP" ]; then
+                  cp "$BACKUP" "$RESOLV"
+                  echo "$(date): Restored original resolv.conf"
+                fi
+                exit 0
+              }
+              trap restore TERM INT
+
+              # Save original
+              if [ ! -f "$BACKUP" ]; then
+                cp "$RESOLV" "$BACKUP"
+                echo "Saved original: $(cat "$BACKUP")"
+              fi
+
+              apply() {
+                # CRITICAL: keep VPC upstream resolver as FALLBACK after kube-dns.
+                # If resolv.conf ONLY has 10.32.0.10, node DNS is dead when CoreDNS
+                # is down — containerd can't pull images, CoreDNS can't start,
+                # entire cluster deadlocks (incident 2026-03-18, 2026-04-12).
+                # Scaleway VPC gateway pushes 172.16.0.13 as DNS via DHCP.
+                UPSTREAM=""
+                if [ -f "$BACKUP" ]; then
+                  UPSTREAM=$(grep "^nameserver" "$BACKUP" | grep -v "127.0.53.53\|$KUBE_DNS" | head -1 | awk '{print $2}')
+                fi
+                if [ -z "$UPSTREAM" ]; then
+                  UPSTREAM="169.254.169.254"  # Scaleway metadata DNS fallback
+                fi
+                printf 'nameserver %s\nnameserver %s\nsearch foxhunt.svc.cluster.local svc.cluster.local cluster.local\noptions ndots:5 timeout:2 attempts:3\n' "$KUBE_DNS" "$UPSTREAM" > "$RESOLV"
+                echo "$(date): resolv.conf → $KUBE_DNS + $UPSTREAM (fallback)"
+              }
+
+              # Initial apply
+              apply
+
+              # Watch for resets
+              while true; do
+                sleep 30 &
+                wait $!
+                if ! grep -q "$KUBE_DNS" "$RESOLV" 2>/dev/null; then
+                  echo "$(date): resolv.conf was reset, re-applying"
+                  apply
+                fi
+              done
+          resources:
+            requests:
+              cpu: 5m
+              memory: 8Mi
+            limits:
+              cpu: 10m
+              memory: 16Mi
+          volumeMounts:
+            - name: host-etc
+              mountPath: /host-etc
+      terminationGracePeriodSeconds: 10
+      volumes:
+        - name: host-etc
+          hostPath:
+            path: /etc
+            type: Directory

infra/k8s/gitea/clusterip-services.yaml

@@ -0,0 +1,40 @@
+# The gitea chart's gitea-http/gitea-ssh services are HEADLESS (clusterIP: None), so they resolve to the
+# pod IP (100.64.x.x). That range OVERLAPS the Tailscale CGNAT range (100.64.0.0/10), so the tailscale
+# sidecar in the tailscale-gitlab-proxy pod swallows traffic to the pod IP. These normal ClusterIP services
+# give Gitea a service IP in the service CIDR (10.32.x.x, outside the tailscale range) — the proxy nginx +
+# socat target these instead. (GitLab worked because its webservice svc was a normal ClusterIP.)
+apiVersion: v1
+kind: Service
+metadata:
+  name: gitea-web
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: gitea
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: gitea
+    app.kubernetes.io/instance: gitea
+  ports:
+    - name: http
+      port: 3000
+      targetPort: 3000
+      protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: gitea-sshd
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: gitea
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: gitea
+    app.kubernetes.io/instance: gitea
+  ports:
+    - name: ssh
+      port: 22
+      targetPort: 2222   # rootless gitea listens on 2222 internally (can't bind privileged 22)
+      protocol: TCP

infra/k8s/gitea/networkpolicy.yaml

@@ -0,0 +1,58 @@
+# Gitea NetworkPolicy (foxhunt ns has default-deny-all). Egress: postgres + DNS + in-cluster webhook
+# target + general HTTPS. Ingress: from platform/argo pods + the tailscale proxy → :3000 (web) and :22 (ssh).
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: gitea
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    # web (3000) + ssh (22) from platform + argo-workflow pods (cockpit build clones over HTTP)
+    - from:
+        - podSelector:
+            matchExpressions:
+              - key: app.kubernetes.io/part-of
+                operator: In
+                values: ["foxhunt", "argo-workflows"]
+      ports:
+        - { port: 3000, protocol: TCP }
+        - { port: 2222, protocol: TCP }
+    # web + ssh from the tailscale proxy (nginx + socat) that fronts git.fxhnt.ai
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tailscale-gitlab-proxy
+      ports:
+        - { port: 3000, protocol: TCP }
+        - { port: 2222, protocol: TCP }
+  egress:
+    # PostgreSQL
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: postgres
+      ports:
+        - { port: 5432, protocol: TCP }
+    # DNS
+    - ports:
+        - { port: 53, protocol: UDP }
+        - { port: 53, protocol: TCP }
+    # Argo Events webhook target (gitea push hook → gitea-push eventsource :12000). The eventsource pod is
+    # not part-of:foxhunt (stays default-allow), so allow egress to any in-namespace pod on 12000.
+    - to:
+        - podSelector: {}
+      ports:
+        - { port: 12000, protocol: TCP }
+    # general HTTPS (avatars, external clones) — external only, exclude cluster CIDRs
+    - to:
+        - ipBlock:
+            cidr: 0.0.0.0/0
+            except: ["10.32.0.0/16", "172.16.0.0/16"]
+      ports:
+        - { port: 443, protocol: TCP }

infra/k8s/gitea/values.yaml

@@ -0,0 +1,73 @@
+# Gitea — lightweight git host replacing GitLab (Phase 2B). External Postgres (existing in-cluster
+# `postgres`), no bundled DB/redis/memcached, Actions off (Argo does CI). Internal-only until cutover.
+replicaCount: 1
+image:
+  rootless: true
+
+# RWO PVC (sbs-default-retain) → only one pod can mount it. RollingUpdate deadlocks (new pod can't
+# attach while old holds it). Recreate terminates old before starting new.
+strategy:
+  type: Recreate
+
+# Disable all bundled subcharts — reuse the existing in-cluster postgres, no redis/memcached.
+postgresql:
+  enabled: false
+postgresql-ha:
+  enabled: false
+redis-cluster:
+  enabled: false
+redis:
+  enabled: false
+# chart 12.x replaced redis with valkey — disable both, gitea uses embedded queue/cache/session
+valkey-cluster:
+  enabled: false
+valkey:
+  enabled: false
+
+persistence:
+  enabled: true
+  size: 5Gi
+  storageClass: sbs-default-retain
+
+resources:
+  requests: { cpu: 100m, memory: 256Mi }
+  limits:   { cpu: "1",  memory: 2Gi }   # 2Gi headroom: indexing the 876MB foxhunt pack OOMed at 512Mi
+
+service:
+  http: { type: ClusterIP, port: 3000 }
+  ssh:  { type: ClusterIP, port: 22 }
+
+gitea:
+  admin:
+    existingSecret: gitea-admin
+  config:
+    actions:
+      ENABLED: "false"        # Argo does CI/CD; Gitea Actions off (subchart removed in chart 12.x)
+    server:
+      ROOT_URL: https://git.fxhnt.ai/
+      DOMAIN: git.fxhnt.ai
+      SSH_DOMAIN: git.fxhnt.ai
+      SSH_PORT: "22"            # clean git@git.fxhnt.ai clone URLs; :2222 retired (Task 5).
+      DISABLE_SSH: "false"
+    database:
+      DB_TYPE: postgres
+      HOST: postgres.foxhunt.svc.cluster.local:5432
+      NAME: gitea
+      USER: gitea
+    service:
+      DISABLE_REGISTRATION: "true"
+    webhook:
+      # allow delivery to in-cluster targets (Argo Events eventsource on the 10.32.x service CIDR);
+      # default blocks private hosts. Argo auto-deploy webhook needs this.
+      ALLOWED_HOST_LIST: "private"
+    # No redis/memcached — use embedded adapters
+    cache:
+      ADAPTER: memory
+    session:
+      PROVIDER: db
+    queue:
+      TYPE: level
+  additionalConfigFromEnvs:
+    - name: GITEA__database__PASSWD
+      valueFrom:
+        secretKeyRef: { name: gitea-db, key: password }

infra/k8s/gitlab/grafana-values.yaml

@@ -0,0 +1,120 @@
+# Grafana — Foxhunt monitoring dashboard
+# Accessed via Tailscale proxy on port 3000
+
+replicas: 1
+
+nodeSelector:
+  k8s.scaleway.com/pool-name: platform
+
+resources:
+  requests:
+    cpu: 50m
+    memory: 128Mi
+  limits:
+    cpu: 200m
+    memory: 256Mi
+
+persistence:
+  enabled: false
+
+admin:
+  existingSecret: grafana-admin
+  userKey: admin-user
+  passwordKey: admin-password
+
+envFromSecrets:
+  - name: db-credentials
+
+grafana.ini:
+  server:
+    root_url: https://grafana.fxhnt.ai
+  unified_alerting:
+    enabled: true
+  alerting:
+    enabled: false
+  database:
+    type: postgres
+    host: postgres:5432
+    name: grafana
+    user: foxhunt
+    password: $__env{password}
+    ssl_mode: disable
+
+datasources:
+  datasources.yaml:
+    apiVersion: 1
+    datasources:
+      - name: Prometheus
+        type: prometheus
+        url: http://prometheus-stack-kube-prom-prometheus.foxhunt.svc.cluster.local:9090
+        access: proxy
+        isDefault: true
+      - name: Alertmanager
+        type: alertmanager
+        uid: alertmanager
+        access: proxy
+        url: http://prometheus-stack-kube-prom-alertmanager.foxhunt.svc.cluster.local:9093
+        isDefault: false
+        jsonData:
+          implementation: prometheus
+          handleGrafanaManagedAlerts: true
+      - name: Loki
+        type: loki
+        access: proxy
+        url: http://loki.foxhunt.svc.cluster.local:3100
+        isDefault: false
+        jsonData:
+          derivedFields:
+            - datasourceUid: tempo
+              matcherRegex: '"trace_id":"(\w+)"'
+              name: TraceID
+              url: "$${__value.raw}"
+      - name: Tempo
+        type: tempo
+        uid: tempo
+        access: proxy
+        url: http://tempo.foxhunt.svc.cluster.local:3200
+        isDefault: false
+        jsonData:
+          tracesToLogsV2:
+            datasourceUid: loki
+            filterByTraceID: true
+          nodeGraph:
+            enabled: true
+          serviceMap:
+            datasourceUid: prometheus
+
+sidecar:
+  dashboards:
+    enabled: true
+    folder: /tmp/dashboards
+    folderAnnotation: grafana_folder
+    label: grafana_dashboard
+    labelValue: "1"
+    searchNamespace: foxhunt
+    provider:
+      foldersFromFilesStructure: true
+  datasources:
+    enabled: false
+    label: grafana_datasource
+    labelValue: ""
+
+dashboardProviders:
+  dashboardproviders.yaml:
+    apiVersion: 1
+    providers:
+      - name: default
+        orgId: 1
+        folder: Foxhunt
+        type: file
+        disableDeletion: false
+        editable: true
+        options:
+          path: /var/lib/grafana/dashboards/default
+
+dashboards:
+  default:
+    node-exporter:
+      gnetId: 1860
+      revision: 37
+      datasource: Prometheus

infra/k8s/gitlab/tailscale-proxy.yaml

@@ -0,0 +1,347 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: tailscale-gitlab
+  namespace: foxhunt
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: tailscale-gitlab
+  namespace: foxhunt
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["create", "get", "update", "patch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: tailscale-gitlab
+  namespace: foxhunt
+subjects:
+  - kind: ServiceAccount
+    name: tailscale-gitlab
+    namespace: foxhunt
+roleRef:
+  kind: Role
+  name: tailscale-gitlab
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: tailscale-gitlab-proxy
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: tailscale-gitlab-proxy
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: tailscale-gitlab-proxy
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: tailscale-gitlab-proxy
+    spec:
+      serviceAccountName: tailscale-gitlab
+      nodeSelector:
+        k8s.scaleway.com/pool-name: platform
+      containers:
+        - name: nginx
+          image: nginx:alpine
+          ports:
+            - containerPort: 80
+            - containerPort: 443
+          volumeMounts:
+            - name: nginx-conf
+              mountPath: /etc/nginx/conf.d
+            - name: tls-certs
+              mountPath: /etc/nginx/certs
+              readOnly: true
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 64Mi
+        - name: ssh-proxy
+          image: alpine/socat:latest
+          args:
+            - "TCP-LISTEN:22,fork,reuseaddr,sndbuf=1048576,rcvbuf=1048576"
+            - "TCP:gitea-sshd.foxhunt.svc.cluster.local:22,sndbuf=1048576,rcvbuf=1048576"
+          ports:
+            - containerPort: 22
+          resources:
+            requests:
+              cpu: 25m
+              memory: 64Mi
+            limits:
+              cpu: 200m
+              memory: 128Mi
+        - name: imap-proxy
+          image: alpine/socat:latest
+          args:
+            - "TCP-LISTEN:993,fork,reuseaddr"
+            - "TCP:stalwart.foxhunt.svc.cluster.local:993"
+          ports:
+            - containerPort: 993
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              cpu: 50m
+              memory: 32Mi
+        - name: tailscale
+          image: ghcr.io/tailscale/tailscale:latest
+          env:
+            - name: TS_AUTHKEY
+              valueFrom:
+                secretKeyRef:
+                  name: tailscale-auth
+                  key: TS_AUTHKEY
+            - name: TS_KUBE_SECRET
+              value: tailscale-gitlab-state
+            - name: TS_USERSPACE
+              value: "false"
+            - name: TS_HOSTNAME
+              value: "foxhunt-gitlab"
+            - name: TS_ACCEPT_DNS
+              value: "false"
+            - name: TS_EXTRA_ARGS
+              value: ""
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 64Mi
+          securityContext:
+            capabilities:
+              add:
+                - NET_ADMIN
+                - NET_RAW
+      volumes:
+        - name: nginx-conf
+          configMap:
+            name: tailscale-gitlab-nginx
+        - name: tls-certs
+          secret:
+            secretName: gitlab-tls-cert
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: tailscale-gitlab-nginx
+  namespace: foxhunt
+data:
+  default.conf: |
+    # HTTP → HTTPS redirect for all subdomains
+    server {
+        listen 80;
+        server_name *.fxhnt.ai;
+        return 301 https://$host$request_uri;
+    }
+
+    # Gitea — git.fxhnt.ai (replaced GitLab, Phase 2B cutover)
+    server {
+        listen 443 ssl;
+        server_name git.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        client_max_body_size 0;
+
+        location / {
+            proxy_pass http://gitea-web.foxhunt.svc.cluster.local:3000;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+
+            proxy_buffering off;
+            proxy_request_buffering off;
+            proxy_http_version 1.1;
+            proxy_read_timeout 3600s;
+            proxy_send_timeout 3600s;
+            proxy_connect_timeout 300s;
+        }
+    }
+
+    # Grafana — grafana.fxhnt.ai
+    server {
+        listen 443 ssl;
+        server_name grafana.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        location / {
+            proxy_pass http://grafana.foxhunt.svc.cluster.local:80;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+        }
+    }
+
+    # Dagster — dagster.fxhnt.ai
+    server {
+        listen 443 ssl;
+        server_name dagster.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        location / {
+            proxy_pass http://dagster.foxhunt.svc.cluster.local:80;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 3600s;
+        }
+    }
+
+    # fxhnt cockpit — dashboard.fxhnt.ai
+    # Proxies to the cockpit's Tailscale node (peer-to-peer over the tailnet) rather than the cluster Service:
+    # the pod CIDR 100.64.0.0/15 overlaps Tailscale CGNAT, so this kernel-mode proxy can't reach platform-pool
+    # pods via ClusterIP, but it CAN reach another tailnet node. IP is stable while the cockpit's TS state
+    # secret (fxhnt-dashboard-ts-state) persists. (Was web-dashboard.foxhunt.svc — replaced by the fxhnt cockpit.)
+    server {
+        listen 443 ssl;
+        server_name dashboard.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        location / {
+            proxy_pass http://100.81.150.18:80;   # fxhnt-dashboard tailnet node (TCP:80 -> cockpit :8080)
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+        }
+    }
+
+    # API Gateway (gRPC-Web + gRPC) — api.fxhnt.ai
+    # Browser dashboard uses gRPC-web (HTTP/1.1 POST), so proxy_pass not grpc_pass
+    server {
+        listen 443 ssl;
+        http2 on;
+        server_name api.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        location / {
+            proxy_pass http://api.foxhunt.svc.cluster.local:50051;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_read_timeout 3600s;
+            proxy_send_timeout 3600s;
+            proxy_connect_timeout 30s;
+        }
+    }
+
+    # Stalwart Admin — mail.fxhnt.ai
+    server {
+        listen 443 ssl;
+        server_name mail.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        location / {
+            proxy_pass https://stalwart.foxhunt.svc.cluster.local:443;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_ssl_verify off;
+        }
+    }
+
+    # Mattermost — chat.fxhnt.ai
+    server {
+        listen 443 ssl;
+        server_name chat.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        client_max_body_size 50m;
+
+        location / {
+            proxy_pass http://mattermost.foxhunt.svc.cluster.local:8065;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 3600s;
+            proxy_send_timeout 3600s;
+        }
+    }
+
+    # MinIO S3 API — minio.fxhnt.ai
+    server {
+        listen 443 ssl;
+        server_name minio.fxhnt.ai;
+
+        ssl_certificate /etc/nginx/certs/tls.crt;
+        ssl_certificate_key /etc/nginx/certs/tls.key;
+        ssl_protocols TLSv1.2 TLSv1.3;
+
+        client_max_body_size 0;
+
+        location / {
+            proxy_pass http://minio.foxhunt.svc.cluster.local:9000;
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto https;
+
+            proxy_buffering off;
+            proxy_request_buffering off;
+            proxy_http_version 1.1;
+            proxy_read_timeout 3600s;
+            proxy_send_timeout 3600s;
+            proxy_connect_timeout 300s;
+
+            # MinIO uses self-signed TLS internally
+            proxy_ssl_verify off;
+        }
+    }
+

infra/k8s/jobs/fxhnt-backtest-ingest-job.yaml

@@ -26,10 +26,10 @@ spec:
       nodeSelector:
         k8s.scaleway.com/pool-name: platform
       imagePullSecrets:
-        - name: gitlab-registry
+        - name: scw-registry
       containers:
         - name: ingest
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/fxhnt-cockpit:latest
+          image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
           # Full pull (no --limit). For a small-batch validation run, use:
           #   args: ["ingest-historical", "--limit", "100"]
           command: ["fxhnt"]
@@ -37,8 +37,16 @@ spec:
           env:
             - name: FXHNT_BACKTEST_WAREHOUSE_PATH
               value: /backtest-data/warehouse.duckdb
+            - name: PYTHONUNBUFFERED         # stream ingest progress logs live (no stdout buffering)
+              value: "1"
             - name: TIINGO_API_KEY
               valueFrom: { secretKeyRef: { name: tiingo-api, key: key } }
+          # Bounded memory: the ingest now caps its in-flight window (~2x workers), so peak RSS is small and
+          # flat. The request+limit make the scheduler place this on a node with headroom and prevent it from
+          # evicting neighbours (an earlier unbounded run hit ~2.3Gi and was evicted off a shared 6.4Gi node).
+          resources:
+            requests: { memory: "768Mi", cpu: "250m" }
+            limits: { memory: "1536Mi" }
           volumeMounts:
             - { name: backtest-data, mountPath: /backtest-data }
       volumes:

infra/k8s/jobs/fxhnt-backtest-verdict-job.yaml

@@ -0,0 +1,67 @@
+# B3b-verdict — run the survivorship-free walk-forward equity-factor backtest against the dedicated
+# backtest warehouse and PUBLISH the per-construction gauntlet verdict into the cockpit DB
+# (backtest_summary table, surfaced in the dashboard). Run DELIBERATELY/occasionally (not nightly) —
+# delete + re-apply to refresh after a new ingest or a methodology change.
+#
+# PIT (point-in-time) universe selection by default (no peak-liquidity lookahead); reads ONLY the
+# dedicated warehouse (never live Tiingo). Memory: the run peaks ~4.3Gi (103M-row monthly aggregate +
+# union adjclose panel), so request 3Gi / limit 5Gi.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: fxhnt-backtest-verdict
+  namespace: foxhunt
+  labels: { app.kubernetes.io/name: fxhnt-backtest-verdict, app.kubernetes.io/part-of: foxhunt }
+spec:
+  backoffLimit: 1
+  ttlSecondsAfterFinished: 604800   # 7d — keep for log inspection
+  template:
+    metadata:
+      labels: { app.kubernetes.io/part-of: foxhunt, app.kubernetes.io/name: fxhnt-backtest-verdict }
+    spec:
+      restartPolicy: Never
+      nodeSelector:
+        k8s.scaleway.com/pool-name: platform
+      imagePullSecrets:
+        - name: scw-registry
+      containers:
+        - name: backtest
+          image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest
+          command: ["fxhnt"]
+          args: ["backtest-equity", "--pit", "--n", "150", "--trailing-window-months", "12",
+                 "--cost-bps", "15", "--delisting-return", "-0.30", "--persist-cockpit",
+                 "--out", "/backtest-data/equity_backtest_report.json"]
+          env:
+            - name: PYTHONUNBUFFERED
+              value: "1"
+            - name: FXHNT_BACKTEST_WAREHOUSE_PATH
+              value: /backtest-data/warehouse.duckdb
+            - name: FXHNT_OPERATIONAL_DSN
+              value: postgresql+psycopg://foxhunt@postgres.foxhunt.svc.cluster.local:5432/fxhnt
+            - name: PGPASSWORD
+              valueFrom: { secretKeyRef: { name: db-credentials, key: password } }
+          resources:
+            requests: { memory: "3Gi", cpu: "500m" }
+            limits: { memory: "5Gi" }
+          volumeMounts:
+            - { name: backtest-data, mountPath: /backtest-data }
+      volumes:
+        - name: backtest-data
+          persistentVolumeClaim: { claimName: fxhnt-backtest-data }
+---
+# Egress for the verdict Job: DNS + internal Postgres (5432). foxhunt has default-deny-all egress and
+# the shared argo-base-egress only opens 443 — this Job talks to the cockpit Postgres on 5432 (internal),
+# so it needs its own rule. No external egress needed (reads the local warehouse, writes internal pg).
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: fxhnt-backtest-verdict
+  namespace: foxhunt
+  labels: { app.kubernetes.io/part-of: foxhunt }
+spec:
+  podSelector: { matchLabels: { app.kubernetes.io/name: fxhnt-backtest-verdict } }
+  policyTypes: [Egress]
+  egress:
+    - ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]
+    - ports: [{ port: 5432, protocol: TCP }]
+      to: [{ podSelector: { matchLabels: { app.kubernetes.io/name: postgres } } }]   # identity-based: postgres pod is in the pod CIDR (100.64.x), not the service CIDR

infra/k8s/jobs/fxhnt-factory-cronjob.yaml

@@ -20,10 +20,10 @@ spec:
           nodeSelector:
             k8s.scaleway.com/pool-name: platform
           imagePullSecrets:
-            - name: gitlab-registry
+            - name: scw-registry
           containers:
             - name: factory
-              image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/fxhnt-cockpit:latest   # set by CI / Kaniko push
+              image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest   # set by CI / Kaniko push
               command: ["sh", "-c"]
               args:
                 - |

infra/k8s/jobs/fxhnt-forward-cronjob.yaml

@@ -31,10 +31,10 @@ spec:
           nodeSelector:
             k8s.scaleway.com/pool-name: platform
           imagePullSecrets:
-            - name: gitlab-registry
+            - name: scw-registry
           containers:
             - name: ingest
-              image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/fxhnt-cockpit:latest   # set by CI / Kaniko push
+              image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest   # set by CI / Kaniko push
               command: ["sh", "-c"]
               args:
                 - |

infra/k8s/jobs/migrate.yaml

@@ -0,0 +1,34 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: foxhunt-migrate
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+    app.kubernetes.io/component: migration
+spec:
+  backoffLimit: 3
+  template:
+    spec:
+      restartPolicy: Never
+      imagePullSecrets:
+        - name: gitlab-registry
+      containers:
+        - name: migrate
+          image: rg.fr-par.scw.cloud/foxhunt/api:latest
+          command: ["sqlx", "migrate", "run"]
+          env:
+            - name: DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: db-credentials
+                  key: password
+            - name: DATABASE_URL
+              value: "postgresql://foxhunt:$(DB_PASSWORD)@postgres:5432/foxhunt"
+          resources:
+            requests:
+              memory: "64Mi"
+              cpu: "50m"
+            limits:
+              memory: "256Mi"
+              cpu: "200m"

infra/k8s/mattermost/mattermost.yaml

@@ -0,0 +1,121 @@
+# Mattermost Team Edition — self-hosted Slack alternative for foxhunt.
+# Runs on platform nodes. PostgreSQL is the shared foxhunt instance.
+# Access: https://chat.fxhnt.ai (Tailscale only)
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: mattermost-data
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: mattermost
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: scw-bssd
+  resources:
+    requests:
+      storage: 5Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mattermost
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: mattermost
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: mattermost
+  strategy:
+    type: Recreate  # RWO PVC
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mattermost
+        app.kubernetes.io/part-of: foxhunt
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: platform
+      containers:
+        - name: mattermost
+          image: mattermost/mattermost-team-edition:10.5
+          ports:
+            - containerPort: 8065
+              name: http
+          env:
+            - name: DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: db-credentials
+                  key: password
+            - name: MM_SQLSETTINGS_DRIVERNAME
+              value: postgres
+            - name: MM_SQLSETTINGS_DATASOURCE
+              value: "postgres://foxhunt:$(DB_PASSWORD)@postgres.foxhunt.svc.cluster.local:5432/mattermost?sslmode=disable"
+            - name: MM_SERVICESETTINGS_SITEURL
+              value: "https://chat.fxhnt.ai"
+            - name: MM_SERVICESETTINGS_LISTENADDRESS
+              value: ":8065"
+            - name: MM_SERVICESETTINGS_ENABLEINCOMINGWEBHOOKS
+              value: "true"
+            - name: MM_SERVICESETTINGS_ENABLEPOSTUSERNAMEOVERRIDE
+              value: "true"
+            - name: MM_SERVICESETTINGS_ENABLEPOSTICONOVERRIDE
+              value: "true"
+            - name: MM_LOGSETTINGS_ENABLECONSOLE
+              value: "true"
+            - name: MM_LOGSETTINGS_CONSOLELEVEL
+              value: INFO
+            - name: MM_LOGSETTINGS_ENABLEFILE
+              value: "false"
+            - name: MM_FILESETTINGS_DRIVERNAME
+              value: local
+            - name: MM_FILESETTINGS_DIRECTORY
+              value: /mattermost/data
+            # Email via Stalwart — uncomment after Stalwart is deployed:
+            # - name: MM_EMAILSETTINGS_SMTPSERVER
+            #   value: stalwart.foxhunt.svc.cluster.local
+            # - name: MM_EMAILSETTINGS_SMTPPORT
+            #   value: "25"
+            # - name: MM_EMAILSETTINGS_SENDEMAILNOTIFICATIONS
+            #   value: "true"
+          volumeMounts:
+            - name: mattermost-data
+              mountPath: /mattermost/data
+          readinessProbe:
+            httpGet:
+              path: /api/v4/system/ping
+              port: 8065
+            initialDelaySeconds: 15
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: "1"
+              memory: 1Gi
+      volumes:
+        - name: mattermost-data
+          persistentVolumeClaim:
+            claimName: mattermost-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: mattermost
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: mattermost
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    app.kubernetes.io/name: mattermost
+  ports:
+    - port: 8065
+      targetPort: 8065
+      name: http

infra/k8s/minio/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - minio.yaml

infra/k8s/minio/minio.yaml

@@ -0,0 +1,188 @@
+# MinIO — in-cluster S3 for compiled binaries and trained models
+# Replaces Scaleway public S3 to keep proprietary code/IP in-cluster
+# Plain HTTP only — internal cluster traffic, no TLS needed
+apiVersion: v1
+kind: Secret
+metadata:
+  name: minio-credentials
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: minio
+    app.kubernetes.io/part-of: foxhunt
+type: Opaque
+stringData:
+  root-user: foxhunt-minio
+  root-password: CHANGE_ME_BEFORE_DEPLOY
+  # rclone-compatible keys (same values, different key names for initContainers)
+  access-key: foxhunt-minio
+  secret-key: CHANGE_ME_BEFORE_DEPLOY
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: minio-data-new
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: minio
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: scw-bssd
+  resources:
+    requests:
+      storage: 150Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: minio
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: minio
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate  # single-node MinIO — no rolling update
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: minio
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: minio
+        app.kubernetes.io/part-of: foxhunt
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: platform
+      containers:
+        - name: minio
+          image: minio/minio:latest
+          args: ["server", "/data", "--console-address", ":9001"]
+          ports:
+            - containerPort: 9000
+              name: api
+            - containerPort: 9001
+              name: console
+          env:
+            - name: MINIO_ROOT_USER
+              valueFrom:
+                secretKeyRef:
+                  name: minio-credentials
+                  key: root-user
+            - name: MINIO_ROOT_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: minio-credentials
+                  key: root-password
+          volumeMounts:
+            - name: data
+              mountPath: /data
+          readinessProbe:
+            httpGet:
+              path: /minio/health/live
+              port: 9000
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /minio/health/live
+              port: 9000
+            initialDelaySeconds: 10
+            periodSeconds: 15
+            failureThreshold: 5
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: "1"
+              memory: 2Gi
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: minio-data-new
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: minio
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: minio
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    app.kubernetes.io/name: minio
+  ports:
+    - port: 9000
+      targetPort: 9000
+      name: api
+    - port: 9001
+      targetPort: 9001
+      name: console
+---
+# Create buckets on first deploy
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: minio-init-buckets
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: minio
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  backoffLimit: 5
+  ttlSecondsAfterFinished: 300
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: minio-init
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: platform
+      restartPolicy: OnFailure
+      containers:
+        - name: mc
+          image: minio/mc:latest
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -e
+              echo "Waiting for MinIO to be ready..."
+              until mc alias set foxhunt http://minio.foxhunt.svc.cluster.local:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" 2>/dev/null; do
+                echo "MinIO not ready, retrying in 3s..."
+                sleep 3
+              done
+              echo "Creating buckets..."
+              mc mb --ignore-existing foxhunt/foxhunt-binaries
+              mc mb --ignore-existing foxhunt/foxhunt-models
+              mc mb --ignore-existing foxhunt/foxhunt-training-results
+              mc mb --ignore-existing foxhunt/argo-logs
+              mc mb --ignore-existing foxhunt/foxhunt-sccache
+              mc mb --ignore-existing foxhunt/foxhunt-training-data
+              # Plan 5 Task 3 (A.4.1): nsys profile artefacts bucket.
+              # Stores .nsys-rep files keyed by commit SHA under
+              # profiles/<short-sha>/profile-seed*-fold*-*.nsys-rep.
+              mc mb --ignore-existing foxhunt/foxhunt-training-artifacts
+              echo "Buckets created:"
+              mc ls foxhunt/
+          env:
+            - name: MINIO_ROOT_USER
+              valueFrom:
+                secretKeyRef:
+                  name: minio-credentials
+                  key: root-user
+            - name: MINIO_ROOT_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: minio-credentials
+                  key: root-password
+          resources:
+            requests:
+              cpu: 50m
+              memory: 32Mi
+            limits:
+              cpu: 200m
+              memory: 64Mi

infra/k8s/monitoring/APPLY-ORDER.md

@@ -0,0 +1,104 @@
+# Cockpit + NAT Gateway — Apply Order
+
+## Prerequisites
+- `scw` CLI authenticated
+- `terragrunt` installed
+- `kubectl` access to foxhunt cluster
+- `helm` installed
+
+## Phase 1: Cockpit (metrics backend)
+
+1. Apply Cockpit Terraform module:
+   ```bash
+   cd infra/live/production/cockpit
+   terragrunt apply
+   ```
+
+2. Note the outputs:
+   ```bash
+   terragrunt output metrics_push_url
+   terragrunt output -raw push_token
+   terragrunt output grafana_url
+   terragrunt output -raw grafana_password
+   ```
+
+3. Create the K8s secret for the push token:
+   ```bash
+   kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
+   kubectl -n monitoring create secret generic cockpit-push-token \
+     --from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
+   ```
+
+4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`:
+   - Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output
+
+5. Deploy DCGM Exporter:
+   ```bash
+   kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
+   ```
+
+6. Deploy Grafana Alloy (k8s-monitoring):
+   ```bash
+   helm repo add grafana https://grafana.github.io/helm-charts
+   helm repo update
+   helm install k8s-monitoring grafana/k8s-monitoring \
+     --namespace monitoring \
+     -f infra/k8s/monitoring/alloy-values.yaml
+   ```
+
+7. Verify metrics flow:
+   - Open Cockpit Grafana (URL from step 2)
+   - Login with foxhunt-admin / (password from step 2)
+   - Check "Kubernetes / Compute Resources / Cluster" dashboard
+
+## Phase 2: GitLab Prometheus remote_write
+
+1. Update placeholders in `infra/k8s/gitlab/values.yaml`:
+   - Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN`
+
+2. Upgrade GitLab Helm release:
+   ```bash
+   helm upgrade gitlab gitlab/gitlab \
+     --namespace foxhunt \
+     -f infra/k8s/gitlab/values.yaml
+   ```
+
+## Phase 3: NAT Gateway
+
+1. Apply Public Gateway module:
+   ```bash
+   cd infra/live/production/public-gateway
+   terragrunt apply
+   ```
+   This creates the gateway and attaches it to the private network.
+   Existing nodes still have public IPs — nothing breaks yet.
+
+2. Apply Kapsule pool changes (disables public IPs):
+   ```bash
+   cd infra/live/production/kapsule
+   terragrunt apply
+   ```
+   **WARNING**: This may recreate nodes. For zero-downtime:
+
+3. Zero-downtime node migration for services pool:
+   ```bash
+   # Temporarily allow 2 nodes
+   scw k8s pool update <services-pool-id> max-size=2 size=2
+   # Wait for new node (private IP) to be Ready
+   kubectl get nodes -w
+   # Drain old node
+   kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
+   # Scale back
+   scw k8s pool update <services-pool-id> max-size=1 size=1
+   ```
+
+4. Repeat for gitlab pool (if not autoscaling, same process).
+
+5. Autoscaling pools (ci-training, gpu-dev):
+   - No action needed — next scale-up automatically uses private IP.
+
+6. Verify Tailscale still works:
+   ```bash
+   tailscale status  # should show foxhunt-kapsule and foxhunt-gitlab nodes
+   curl -k https://git.fxhnt.ai  # should still resolve via Tailscale
+   ```

infra/k8s/monitoring/alloy-values.yaml

@@ -0,0 +1,88 @@
+# Grafana k8s-monitoring Helm chart values (v1.x format)
+# Chart: grafana/k8s-monitoring --version 1.6.50
+# Deploys: Alloy (DaemonSet) + node-exporter + kube-state-metrics
+#
+# Install:
+#   helm repo add grafana https://grafana.github.io/helm-charts
+#   PUSH_TOKEN=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
+#   helm install k8s-monitoring grafana/k8s-monitoring \
+#     --namespace monitoring --version 1.6.50 \
+#     -f infra/k8s/monitoring/alloy-values.yaml \
+#     --set "externalServices.prometheus.extraHeaders.X-Token=${PUSH_TOKEN}"
+
+cluster:
+  name: foxhunt
+
+externalServices:
+  prometheus:
+    host: "https://d8cd4c5d-6d3b-4258-9f92-2ad0eaa36ba0.metrics.cockpit.fr-par.scw.cloud"
+    writeEndpoint: "/api/v1/push"
+    authMode: none
+    # X-Token header passed via --set to keep secret out of git
+  # Loki host required by chart validation even with logs disabled
+  loki:
+    host: "https://noop.localhost"
+    authMode: none
+
+metrics:
+  enabled: true
+  cost:
+    enabled: false
+  node-exporter:
+    enabled: true
+  kube-state-metrics:
+    enabled: true
+  kubelet:
+    enabled: true
+  cadvisor:
+    enabled: true
+  apiserver:
+    enabled: true
+
+# Alloy DaemonSet — runs on every node
+alloy-metrics:
+  alloy:
+    resources:
+      requests:
+        cpu: 50m
+        memory: 128Mi
+      limits:
+        cpu: 200m
+        memory: 256Mi
+
+# node-exporter — host metrics (CPU, memory, disk, network)
+prometheus-node-exporter:
+  resources:
+    requests:
+      cpu: 25m
+      memory: 32Mi
+    limits:
+      cpu: 100m
+      memory: 64Mi
+
+# kube-state-metrics — K8s object state
+kube-state-metrics:
+  resources:
+    requests:
+      cpu: 25m
+      memory: 64Mi
+    limits:
+      cpu: 100m
+      memory: 128Mi
+  # Scheduled on gitlab node (GP1-XS 16GB) — services (DEV1-M 4GB) too small
+
+# Disable components we don't need
+logs:
+  enabled: false
+  cluster_events:
+    enabled: false
+traces:
+  enabled: false
+opencost:
+  enabled: false
+
+# Disable log/event collectors
+alloy-logs:
+  enabled: false
+alloy-events:
+  enabled: false

infra/k8s/monitoring/dashboards/foxhunt-ci-logs.json

@@ -0,0 +1,809 @@
+{
+  "description": "Loki-powered CI pipeline log viewer \u2014 test gate, clippy, builds, errors",
+  "editable": true,
+  "graphTooltip": 2,
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "title": "Pipeline Overview",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "red",
+            "mode": "fixed"
+          },
+          "mappings": [],
+          "noValue": "0",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 0,
+        "y": 1
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "sum"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=\"ERROR\"} [$__range]))",
+          "queryType": "instant",
+          "refId": "A"
+        }
+      ],
+      "title": "Errors",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "yellow",
+            "mode": "fixed"
+          },
+          "mappings": [],
+          "noValue": "0",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 4,
+        "y": 1
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "sum"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=\"WARN\"} [$__range]))",
+          "queryType": "instant",
+          "refId": "A"
+        }
+      ],
+      "title": "Warnings",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "blue",
+            "mode": "fixed"
+          },
+          "mappings": [],
+          "noValue": "0"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 8,
+        "y": 1
+      },
+      "id": 4,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "sum"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__range]))",
+          "queryType": "instant",
+          "refId": "A"
+        }
+      ],
+      "title": "Total Log Lines",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisLabel": "lines/interval",
+            "drawStyle": "bars",
+            "fillOpacity": 80,
+            "lineWidth": 0,
+            "pointSize": 5,
+            "showPoints": "never",
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "ERROR"
+            },
+            "properties": [
+              {
+                "id": "color",
+                "value": {
+                  "fixedColor": "red",
+                  "mode": "fixed"
+                }
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "WARN"
+            },
+            "properties": [
+              {
+                "id": "color",
+                "value": {
+                  "fixedColor": "yellow",
+                  "mode": "fixed"
+                }
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "INFO"
+            },
+            "properties": [
+              {
+                "id": "color",
+                "value": {
+                  "fixedColor": "green",
+                  "mode": "fixed"
+                }
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "DEBUG"
+            },
+            "properties": [
+              {
+                "id": "color",
+                "value": {
+                  "fixedColor": "blue",
+                  "mode": "fixed"
+                }
+              }
+            ]
+          }
+        ]
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 12,
+        "y": 1
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (level) (count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__interval]))",
+          "legendFormat": "{{level}}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Volume by Level",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "drawStyle": "bars",
+            "fillOpacity": 60,
+            "lineWidth": 0,
+            "showPoints": "never",
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 12,
+        "x": 0,
+        "y": 5
+      },
+      "id": 6,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "right"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (container) (count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__interval]))",
+          "legendFormat": "{{container}}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Volume by Container",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 9
+      },
+      "id": 10,
+      "title": "Full Pipeline Log Stream",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 14,
+        "w": 24,
+        "x": 0,
+        "y": 10
+      },
+      "id": 11,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "{namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=~\"$level\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg ${search:pipe}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "Pipeline Logs \u2014 $pipeline",
+      "type": "logs"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 24
+      },
+      "id": 20,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 12,
+            "w": 24,
+            "x": 0,
+            "y": 25
+          },
+          "id": 21,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": false,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"ci-pipeline-.*\", container=\"main\"} |~ \"(clippy|cargo test|warning\\\\[|error\\\\[|FAILED|test result|running \\\\d+ test|failures:)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "Clippy + Test Output",
+          "type": "logs"
+        }
+      ],
+      "title": "Test Gate (clippy + cargo test)",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "id": 30,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 24,
+            "x": 0,
+            "y": 26
+          },
+          "id": 31,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": false,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"ci-pipeline-.*\", container=\"redis\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "Redis Sidecar Logs",
+          "type": "logs"
+        }
+      ],
+      "title": "Redis Sidecar",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 26
+      },
+      "id": 40,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 12,
+            "w": 24,
+            "x": 0,
+            "y": 27
+          },
+          "id": 41,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": true,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} |~ \"(?i)(error|failed|failure|panic|fatal|SIGABRT|SIGSEGV|thread .* panicked)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "All Errors & Failures",
+          "type": "logs"
+        }
+      ],
+      "title": "Errors & Failures",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 27
+      },
+      "id": 50,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 10,
+            "w": 12,
+            "x": 0,
+            "y": 28
+          },
+          "id": 51,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": false,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"(ci-pipeline|compile-and-deploy|compile-and-train)-.*\", container=\"main\"} |~ \"(Compiling|Finished|warning|error|Downloading|Building|cargo build|rclone)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "Compile Logs",
+          "type": "logs"
+        },
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 10,
+            "w": 12,
+            "x": 12,
+            "y": 28
+          },
+          "id": 52,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": false,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"(ci-pipeline|compile-and-deploy)-.*\", container=\"main\"} |~ \"(kubectl|apply|rollout|deploy|configured|unchanged|created|image|kaniko)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "Deploy & Argo Template Logs",
+          "type": "logs"
+        }
+      ],
+      "title": "Build & Deploy",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 28
+      },
+      "id": 60,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 12,
+            "w": 24,
+            "x": 0,
+            "y": 29
+          },
+          "id": 61,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": true,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"gpu-test-.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "GPU Test Logs (DQN/PPO/TFT + Supervised)",
+          "type": "logs"
+        }
+      ],
+      "title": "GPU Test Pipeline",
+      "type": "row"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "foxhunt",
+    "ci",
+    "loki",
+    "logs"
+  ],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": true,
+          "text": "ci-pipeline",
+          "value": "ci-pipeline"
+        },
+        "label": "Pipeline",
+        "name": "pipeline",
+        "options": [],
+        "query": "ci-pipeline,compile-and-deploy,compile-and-train,gpu-test,gpu-diag,gpu-walltime,build-ci-image,build-training-runtime,evaluate",
+        "type": "custom"
+      },
+      {
+        "allValue": ".*",
+        "current": {
+          "selected": true,
+          "text": "All",
+          "value": "$__all"
+        },
+        "includeAll": true,
+        "label": "Level",
+        "multi": true,
+        "name": "level",
+        "options": [
+          {
+            "selected": true,
+            "text": "All",
+            "value": "$__all"
+          },
+          {
+            "selected": false,
+            "text": "ERROR",
+            "value": "ERROR"
+          },
+          {
+            "selected": false,
+            "text": "WARN",
+            "value": "WARN"
+          },
+          {
+            "selected": false,
+            "text": "INFO",
+            "value": "INFO"
+          },
+          {
+            "selected": false,
+            "text": "DEBUG",
+            "value": "DEBUG"
+          },
+          {
+            "selected": false,
+            "text": "TRACE",
+            "value": "TRACE"
+          }
+        ],
+        "query": "ERROR,WARN,INFO,DEBUG,TRACE",
+        "type": "custom"
+      },
+      {
+        "current": {
+          "selected": false,
+          "text": "",
+          "value": ""
+        },
+        "label": "Search",
+        "name": "search",
+        "type": "textbox"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "CI Pipeline Logs",
+  "uid": "foxhunt-ci-logs",
+  "__inputs": [
+    {
+      "name": "DS_PROMETHEUS",
+      "type": "datasource",
+      "pluginId": "prometheus"
+    },
+    {
+      "name": "DS_LOKI",
+      "type": "datasource",
+      "pluginId": "loki"
+    },
+    {
+      "name": "DS_TEMPO",
+      "type": "datasource",
+      "pluginId": "${DS_TEMPO}"
+    }
+  ]
+}

infra/k8s/monitoring/dashboards/foxhunt-training-logs.json

@@ -0,0 +1,835 @@
+{
+  "description": "Loki-powered training log viewer \u2014 hyperopt trials, epoch progress, GPU status, errors",
+  "editable": true,
+  "graphTooltip": 2,
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "title": "Training Overview",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "red",
+            "mode": "fixed"
+          },
+          "mappings": [],
+          "noValue": "0",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 0,
+        "y": 1
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "reduceOptions": {
+          "calcs": [
+            "sum"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\", level=\"ERROR\"} [$__range]))",
+          "queryType": "instant",
+          "refId": "A"
+        }
+      ],
+      "title": "Errors",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "yellow",
+            "mode": "fixed"
+          },
+          "mappings": [],
+          "noValue": "0",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 4,
+        "y": 1
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "reduceOptions": {
+          "calcs": [
+            "sum"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\", level=\"WARN\"} [$__range]))",
+          "queryType": "instant",
+          "refId": "A"
+        }
+      ],
+      "title": "Warnings",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "fixedColor": "blue",
+            "mode": "fixed"
+          },
+          "mappings": [],
+          "noValue": "0"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 4,
+        "x": 8,
+        "y": 1
+      },
+      "id": 4,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "reduceOptions": {
+          "calcs": [
+            "sum"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__range]))",
+          "queryType": "instant",
+          "refId": "A"
+        }
+      ],
+      "title": "Total Log Lines",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisLabel": "lines/interval",
+            "drawStyle": "bars",
+            "fillOpacity": 80,
+            "lineWidth": 0,
+            "showPoints": "never",
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "ERROR"
+            },
+            "properties": [
+              {
+                "id": "color",
+                "value": {
+                  "fixedColor": "red",
+                  "mode": "fixed"
+                }
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "WARN"
+            },
+            "properties": [
+              {
+                "id": "color",
+                "value": {
+                  "fixedColor": "yellow",
+                  "mode": "fixed"
+                }
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "INFO"
+            },
+            "properties": [
+              {
+                "id": "color",
+                "value": {
+                  "fixedColor": "green",
+                  "mode": "fixed"
+                }
+              }
+            ]
+          }
+        ]
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 12,
+        "x": 12,
+        "y": 1
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (level) (count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__interval]))",
+          "legendFormat": "{{level}}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Volume by Level",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "drawStyle": "bars",
+            "fillOpacity": 60,
+            "lineWidth": 0,
+            "showPoints": "never",
+            "stacking": {
+              "mode": "normal"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 12,
+        "x": 0,
+        "y": 5
+      },
+      "id": 6,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "right"
+        },
+        "tooltip": {
+          "mode": "multi"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (pod) (count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__interval]))",
+          "legendFormat": "{{pod}}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Volume by Pod",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 9
+      },
+      "id": 10,
+      "title": "Full Training Log Stream",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 14,
+        "w": 24,
+        "x": 0,
+        "y": 10
+      },
+      "id": 11,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "editorMode": "code",
+          "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\", level=~\"$level\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg ${search:pipe}",
+          "queryType": "range",
+          "refId": "A"
+        }
+      ],
+      "title": "Training Logs \u2014 $job",
+      "type": "logs"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 24
+      },
+      "id": 20,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 12,
+            "w": 24,
+            "x": 0,
+            "y": 25
+          },
+          "id": 21,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": true,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(epoch|loss|reward|sharpe|accuracy|val_loss|train_loss|best_trial|trial|fold|window|learning_rate|lr=|MaxDD|win.?rate|drawdown)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "Epoch / Loss / Reward / Sharpe",
+          "type": "logs"
+        }
+      ],
+      "title": "Epoch Progress & Loss",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "id": 30,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 12,
+            "w": 24,
+            "x": 0,
+            "y": 26
+          },
+          "id": 31,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": true,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"(hyperopt|train)-.*\"} |~ \"(?i)(trial|best|cost|param|learning_rate|hidden_dim|batch_size|buffer_size|gamma|epsilon|clip|PSO|iteration|objective|optimal)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "Hyperopt Trial Results",
+          "type": "logs"
+        }
+      ],
+      "title": "Hyperopt Trials",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 26
+      },
+      "id": 40,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 12,
+            "w": 24,
+            "x": 0,
+            "y": 27
+          },
+          "id": 41,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": true,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(GPU|CUDA|VRAM|cuda|gpu|vram|OOM|out.of.memory|nvrtc|tensor.core|BF16|bf16|SM|occupancy|kernel|device|nvidia)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "GPU / CUDA / VRAM Logs",
+          "type": "logs"
+        }
+      ],
+      "title": "GPU / CUDA / VRAM",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 27
+      },
+      "id": 50,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 10,
+            "w": 24,
+            "x": 0,
+            "y": 28
+          },
+          "id": 51,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": true,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(walk.?forward|fold|backtest|window|train_start|train_end|val_start|val_end|test_start|test_end|embargo|purge|bars|features)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "Walk-Forward / Fold / Backtest",
+          "type": "logs"
+        }
+      ],
+      "title": "Walk-Forward Windows",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 28
+      },
+      "id": 60,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 12,
+            "w": 24,
+            "x": 0,
+            "y": 29
+          },
+          "id": 61,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": true,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(error|failed|failure|panic|fatal|SIGABRT|SIGSEGV|thread .* panicked|OOM|NaN|inf|diverge)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "All Training Errors & Failures",
+          "type": "logs"
+        }
+      ],
+      "title": "Errors & Failures",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 29
+      },
+      "id": 70,
+      "panels": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "gridPos": {
+            "h": 10,
+            "w": 24,
+            "x": 0,
+            "y": 30
+          },
+          "id": 71,
+          "options": {
+            "dedupStrategy": "none",
+            "enableLogDetails": true,
+            "showLabels": true,
+            "showTime": true,
+            "sortOrder": "Descending",
+            "wrapLogMessage": true
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "loki",
+                "uid": "${DS_LOKI}"
+              },
+              "editorMode": "code",
+              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(loading|loaded|dbn|ofi|features|bars|MBP.?10|OHLCV|symbol|normalize|norm_stats|state_dim|rclone|S3|download)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+              "queryType": "range",
+              "refId": "A"
+            }
+          ],
+          "title": "Data Loading / OFI / DBN / Features",
+          "type": "logs"
+        }
+      ],
+      "title": "Data Loading & OFI",
+      "type": "row"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "foxhunt",
+    "training",
+    "loki",
+    "logs",
+    "gpu"
+  ],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": true,
+          "text": "All Training",
+          "value": "(train|hyperopt|compile-and-train)"
+        },
+        "label": "Job Type",
+        "name": "job",
+        "options": [
+          {
+            "selected": true,
+            "text": "All Training",
+            "value": "(train|hyperopt|compile-and-train)"
+          },
+          {
+            "selected": false,
+            "text": "DQN",
+            "value": "train-dqn"
+          },
+          {
+            "selected": false,
+            "text": "PPO",
+            "value": "train-ppo"
+          },
+          {
+            "selected": false,
+            "text": "Supervised",
+            "value": "train-supervised"
+          },
+          {
+            "selected": false,
+            "text": "Hyperopt",
+            "value": "hyperopt"
+          },
+          {
+            "selected": false,
+            "text": "Compile+Train",
+            "value": "compile-and-train"
+          }
+        ],
+        "query": "All Training : (train|hyperopt|compile-and-train), DQN : train-dqn, PPO : train-ppo, Supervised : train-supervised, Hyperopt : hyperopt, Compile+Train : compile-and-train",
+        "type": "custom"
+      },
+      {
+        "allValue": ".*",
+        "current": {
+          "selected": true,
+          "text": "All",
+          "value": "$__all"
+        },
+        "includeAll": true,
+        "label": "Level",
+        "multi": true,
+        "name": "level",
+        "options": [
+          {
+            "selected": true,
+            "text": "All",
+            "value": "$__all"
+          },
+          {
+            "selected": false,
+            "text": "ERROR",
+            "value": "ERROR"
+          },
+          {
+            "selected": false,
+            "text": "WARN",
+            "value": "WARN"
+          },
+          {
+            "selected": false,
+            "text": "INFO",
+            "value": "INFO"
+          },
+          {
+            "selected": false,
+            "text": "DEBUG",
+            "value": "DEBUG"
+          },
+          {
+            "selected": false,
+            "text": "TRACE",
+            "value": "TRACE"
+          }
+        ],
+        "query": "ERROR,WARN,INFO,DEBUG,TRACE",
+        "type": "custom"
+      },
+      {
+        "current": {
+          "selected": false,
+          "text": "",
+          "value": ""
+        },
+        "label": "Search",
+        "name": "search",
+        "type": "textbox"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-12h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "Training Pipeline Logs",
+  "uid": "foxhunt-training-logs",
+  "__inputs": [
+    {
+      "name": "DS_PROMETHEUS",
+      "type": "datasource",
+      "pluginId": "prometheus"
+    },
+    {
+      "name": "DS_LOKI",
+      "type": "datasource",
+      "pluginId": "loki"
+    },
+    {
+      "name": "DS_TEMPO",
+      "type": "datasource",
+      "pluginId": "${DS_TEMPO}"
+    }
+  ]
+}

infra/k8s/monitoring/dashboards/import.sh

@@ -0,0 +1,212 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Import all Grafana dashboards via API (Postgres-backed, no ConfigMaps).
+#
+# Dashboards use ${DS_PROMETHEUS}, ${DS_LOKI}, ${DS_TEMPO} variables.
+# This script auto-detects datasource UIDs, resolves variables, and
+# imports via the Grafana API. Dashboards are stored in Postgres.
+#
+# Required environment variables:
+#   GRAFANA_PASS - Password for admin user
+#
+# Optional overrides:
+#   GRAFANA_URL       - Grafana base URL (default: https://grafana.fxhnt.ai)
+#   GRAFANA_USER      - Grafana user (default: admin)
+#   GRAFANA_PROM_UID  - Override Prometheus datasource UID
+#   GRAFANA_LOKI_UID  - Override Loki datasource UID
+#   GRAFANA_TEMPO_UID - Override Tempo datasource UID
+#
+# Usage:
+#   GRAFANA_PASS="xxx" ./import.sh
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+GRAFANA_USER="${GRAFANA_USER:-admin}"
+GRAFANA_URL="${GRAFANA_URL:-https://grafana.fxhnt.ai}"
+GRAFANA_URL="${GRAFANA_URL%/}"
+
+if [[ -z "${GRAFANA_PASS:-}" ]]; then
+  echo "ERROR: GRAFANA_PASS is not set" >&2
+  exit 1
+fi
+
+# --- Datasource auto-detection ---
+
+detect_datasource_uid() {
+  local ds_type="$1"
+  curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" "${GRAFANA_URL}/api/datasources" 2>/dev/null | \
+    python3 -c "
+import sys, json
+for ds in json.load(sys.stdin):
+    if ds.get('type') == '${ds_type}':
+        print(ds['uid'])
+        break
+" 2>/dev/null || true
+}
+
+echo "Detecting datasource UIDs..."
+PROM_UID="${GRAFANA_PROM_UID:-$(detect_datasource_uid prometheus)}"
+LOKI_UID="${GRAFANA_LOKI_UID:-$(detect_datasource_uid loki)}"
+TEMPO_UID="${GRAFANA_TEMPO_UID:-$(detect_datasource_uid tempo)}"
+
+echo "  Prometheus: ${PROM_UID:-NOT FOUND}"
+echo "  Loki:       ${LOKI_UID:-NOT FOUND}"
+echo "  Tempo:      ${TEMPO_UID:-NOT FOUND}"
+
+if [[ -z "$PROM_UID" ]]; then
+  echo "ERROR: Could not detect Prometheus datasource. Set GRAFANA_PROM_UID." >&2
+  exit 1
+fi
+
+# --- Resolve datasource variables ---
+
+resolve_dashboard() {
+  local file="$1"
+  python3 -c "
+import sys, json
+
+with open('${file}') as f:
+    d = json.load(f)
+
+d.pop('__inputs', None)
+d.pop('__requires', None)
+
+raw = json.dumps(d)
+raw = raw.replace('\${DS_PROMETHEUS}', '${PROM_UID}')
+raw = raw.replace('\${DS_LOKI}', '${LOKI_UID}')
+raw = raw.replace('\${DS_TEMPO}', '${TEMPO_UID}')
+
+print(raw)
+"
+}
+
+# --- Ensure folders exist ---
+
+ensure_folder() {
+  local title="$1"
+  local uid
+  uid=$(curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" "${GRAFANA_URL}/api/folders" 2>/dev/null | \
+    python3 -c "
+import sys, json
+for f in json.load(sys.stdin):
+    if f['title'] == '${title}':
+        print(f['uid'])
+        break
+" 2>/dev/null || true)
+
+  if [[ -z "$uid" ]]; then
+    uid=$(curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
+      -H "Content-Type: application/json" \
+      -d "{\"title\": \"${title}\"}" \
+      "${GRAFANA_URL}/api/folders" 2>/dev/null | \
+      python3 -c "import sys,json; print(json.load(sys.stdin)['uid'])" 2>/dev/null)
+    echo "  Created folder: ${title} (${uid})"
+  fi
+  echo "$uid"
+}
+
+echo ""
+echo "Ensuring folders..."
+FOLDER_CI=$(ensure_folder "CI")
+FOLDER_INFRA=$(ensure_folder "Infrastructure")
+FOLDER_OPS=$(ensure_folder "Operations")
+FOLDER_TRADING=$(ensure_folder "Trading")
+FOLDER_TRAINING=$(ensure_folder "Training")
+FOLDER_SERVICES=$(ensure_folder "Services")
+FOLDER_PLATFORM=$(ensure_folder "Platform")
+
+# --- Import a single dashboard to a folder ---
+
+import_dashboard() {
+  local file="$1"
+  local folder_uid="$2"
+  local name
+  name="$(basename "$file" .json)"
+
+  local resolved
+  resolved=$(resolve_dashboard "$file")
+
+  local response
+  response=$(curl -s -w "\n%{http_code}" \
+    -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
+    -H "Content-Type: application/json" \
+    -d "{\"dashboard\": ${resolved}, \"folderUid\": \"${folder_uid}\", \"overwrite\": true}" \
+    "${GRAFANA_URL}/api/dashboards/db")
+
+  local http_code
+  http_code=$(echo "$response" | tail -1)
+
+  if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then
+    echo "  OK   ${name}"
+  else
+    local body
+    body=$(echo "$response" | sed '$d')
+    echo "  FAIL ${name} (${http_code}): ${body}" >&2
+    return 1
+  fi
+}
+
+# --- Dashboard → Folder mapping ---
+
+declare -A DASHBOARD_FOLDERS=(
+  # Original dashboards
+  [foxhunt-cockpit]="${FOLDER_OPS}"
+  [foxhunt-trading]="${FOLDER_TRADING}"
+  [foxhunt-training]="${FOLDER_TRAINING}"
+  [foxhunt-infrastructure]="${FOLDER_INFRA}"
+  [foxhunt-observability]="${FOLDER_INFRA}"
+  [foxhunt-ci-logs]="${FOLDER_CI}"
+  [foxhunt-training-logs]="${FOLDER_TRAINING}"
+  [foxhunt-training-deep-dive]="${FOLDER_TRAINING}"
+  # Application services
+  [svc-api-gateway]="${FOLDER_SERVICES}"
+  [svc-trading-service]="${FOLDER_SERVICES}"
+  [svc-backtesting]="${FOLDER_SERVICES}"
+  [svc-data-acquisition]="${FOLDER_SERVICES}"
+  [svc-ml-training]="${FOLDER_SERVICES}"
+  [svc-trading-agent]="${FOLDER_SERVICES}"
+  [svc-web-gateway]="${FOLDER_SERVICES}"
+  # Platform services
+  [platform-prometheus]="${FOLDER_PLATFORM}"
+  [platform-grafana]="${FOLDER_PLATFORM}"
+  [platform-loki]="${FOLDER_PLATFORM}"
+  [platform-tempo]="${FOLDER_PLATFORM}"
+  [platform-redis]="${FOLDER_PLATFORM}"
+  [platform-postgres]="${FOLDER_PLATFORM}"
+  [platform-argo]="${FOLDER_PLATFORM}"
+  [platform-questdb]="${FOLDER_PLATFORM}"
+  [platform-costs]="${FOLDER_PLATFORM}"
+  [platform-scaleway]="${FOLDER_PLATFORM}"
+)
+
+# --- Deploy all dashboards ---
+
+echo ""
+echo "Importing dashboards..."
+
+failed=0
+total=0
+
+for dashboard in "${SCRIPT_DIR}"/*.json; do
+  [[ -f "$dashboard" ]] || continue
+  name="$(basename "$dashboard" .json)"
+  folder_uid="${DASHBOARD_FOLDERS[$name]:-}"
+
+  if [[ -z "$folder_uid" ]]; then
+    echo "  SKIP ${name} (no folder mapping)"
+    continue
+  fi
+
+  total=$((total + 1))
+  if ! import_dashboard "$dashboard" "$folder_uid"; then
+    failed=$((failed + 1))
+  fi
+done
+
+echo ""
+if [[ $failed -eq 0 ]]; then
+  echo "All ${total} dashboards deployed successfully."
+else
+  echo "ERROR: ${failed}/${total} dashboard(s) failed." >&2
+  exit 1
+fi

infra/k8s/monitoring/dashboards/platform-grafana.json

@@ -0,0 +1,802 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "panels": [],
+      "title": "Grafana Health",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 0,
+        "y": 1
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"grafana.*\", phase=\"Running\"}",
+          "instant": true,
+          "legendFormat": "Running",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Pod Status",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 3
+              },
+              {
+                "color": "red",
+                "value": 10
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 3,
+        "y": 1
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"grafana.*\"})",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Restarts",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.7
+              },
+              {
+                "color": "red",
+                "value": 0.9
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 6,
+        "y": 1
+      },
+      "id": 4,
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[$interval]))",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.7
+              },
+              {
+                "color": "red",
+                "value": 0.9
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 9,
+        "y": 1
+      },
+      "id": 5,
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"grafana.*\", resource=\"memory\"})",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 12,
+        "y": 1
+      },
+      "id": 6,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"})",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Memory (bytes)",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 15,
+        "y": 1
+      },
+      "id": 7,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Cores",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 1
+      },
+      "id": 8,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Network RX",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 1
+      },
+      "id": 9,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Network TX",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 5
+      },
+      "id": 10,
+      "panels": [],
+      "title": "Resources",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 6
+      },
+      "id": 11,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[$interval])",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 6
+      },
+      "id": 12,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 6
+      },
+      "id": 13,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[$interval])",
+          "legendFormat": "RX {{pod}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[$interval])",
+          "legendFormat": "TX {{pod}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Network I/O",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 14
+      },
+      "id": 14,
+      "panels": [],
+      "title": "Logs",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 10,
+        "w": 24,
+        "x": 0,
+        "y": 15
+      },
+      "id": 15,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"grafana.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Stream",
+      "type": "logs"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "id": 16,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"grafana.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Errors & Warnings",
+      "type": "logs"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "platform",
+    "grafana"
+  ],
+  "templating": {
+    "list": [
+      {
+        "auto": false,
+        "current": {
+          "text": "5m",
+          "value": "5m"
+        },
+        "name": "interval",
+        "options": [
+          {
+            "selected": false,
+            "text": "1m",
+            "value": "1m"
+          },
+          {
+            "selected": true,
+            "text": "5m",
+            "value": "5m"
+          },
+          {
+            "selected": false,
+            "text": "15m",
+            "value": "15m"
+          },
+          {
+            "selected": false,
+            "text": "30m",
+            "value": "30m"
+          },
+          {
+            "selected": false,
+            "text": "1h",
+            "value": "1h"
+          }
+        ],
+        "query": "1m,5m,15m,30m,1h",
+        "type": "interval"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "Grafana",
+  "uid": "plat-grafana"
+}

infra/k8s/monitoring/dashboards/platform-prometheus.json

@@ -0,0 +1,946 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "panels": [],
+      "title": "Prometheus Health",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 0,
+        "y": 1
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "count(up{job!=\"\"} == 1)",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Targets Up",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 3,
+        "y": 1
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "count(up{job!=\"\"} == 0) or vector(0)",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Targets Down",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 6,
+        "y": 1
+      },
+      "id": 4,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "prometheus_tsdb_head_series",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Head Series",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 9,
+        "y": 1
+      },
+      "id": 5,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(prometheus_tsdb_head_samples_appended_total[5m])",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Ingestion Rate",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 12,
+        "y": 1
+      },
+      "id": 6,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "prometheus_tsdb_wal_storage_size_bytes",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "WAL Size",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 15,
+        "y": 1
+      },
+      "id": 7,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "prometheus_tsdb_head_chunks",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Head Chunks",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 1
+      },
+      "id": 8,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "prometheus_tsdb_compactions_total",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Compactions",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 1
+      },
+      "id": 9,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "prometheus_tsdb_storage_blocks_bytes",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Storage Size",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 5
+      },
+      "id": 10,
+      "panels": [],
+      "title": "Resources",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 6
+      },
+      "id": 11,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"prometheus.*\", container!=\"\"}[$interval])",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 6
+      },
+      "id": 12,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"prometheus.*\", container!=\"\"}",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 6
+      },
+      "id": 13,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"prometheus.*\"}[$interval])",
+          "legendFormat": "RX {{pod}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"prometheus.*\"}[$interval])",
+          "legendFormat": "TX {{pod}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Network I/O",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 14
+      },
+      "id": 14,
+      "panels": [],
+      "title": "Scrape Performance",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 15
+      },
+      "id": 15,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "scrape_duration_seconds",
+          "legendFormat": "{{job}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Scrape Duration",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 15
+      },
+      "id": 16,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(prometheus_tsdb_head_samples_appended_total[$interval])",
+          "legendFormat": "samples/s",
+          "refId": "A"
+        }
+      ],
+      "title": "Samples Ingested",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 23
+      },
+      "id": 17,
+      "panels": [],
+      "title": "Target Status",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {},
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 24
+      },
+      "id": 18,
+      "options": {
+        "showHeader": true,
+        "sortBy": []
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "up",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Target Health",
+      "type": "table"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 32
+      },
+      "id": 19,
+      "panels": [],
+      "title": "Logs",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 10,
+        "w": 24,
+        "x": 0,
+        "y": 33
+      },
+      "id": 20,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"prometheus.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Stream",
+      "type": "logs"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 43
+      },
+      "id": 21,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"prometheus.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Errors & Warnings",
+      "type": "logs"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "platform",
+    "prometheus"
+  ],
+  "templating": {
+    "list": [
+      {
+        "auto": false,
+        "current": {
+          "text": "5m",
+          "value": "5m"
+        },
+        "name": "interval",
+        "options": [
+          {
+            "selected": false,
+            "text": "1m",
+            "value": "1m"
+          },
+          {
+            "selected": true,
+            "text": "5m",
+            "value": "5m"
+          },
+          {
+            "selected": false,
+            "text": "15m",
+            "value": "15m"
+          },
+          {
+            "selected": false,
+            "text": "30m",
+            "value": "30m"
+          },
+          {
+            "selected": false,
+            "text": "1h",
+            "value": "1h"
+          }
+        ],
+        "query": "1m,5m,15m,30m,1h",
+        "type": "interval"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "Prometheus",
+  "uid": "plat-prometheus"
+}

infra/k8s/monitoring/dashboards/platform-questdb.json

@@ -0,0 +1,802 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "panels": [],
+      "title": "QuestDB Health",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 0,
+        "y": 1
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"questdb.*\", phase=\"Running\"}",
+          "instant": true,
+          "legendFormat": "Running",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Pod Status",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 3
+              },
+              {
+                "color": "red",
+                "value": 10
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 3,
+        "y": 1
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"questdb.*\"})",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Restarts",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.7
+              },
+              {
+                "color": "red",
+                "value": 0.9
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 6,
+        "y": 1
+      },
+      "id": 4,
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[$interval]))",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.7
+              },
+              {
+                "color": "red",
+                "value": 0.9
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 9,
+        "y": 1
+      },
+      "id": 5,
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"questdb.*\", resource=\"memory\"})",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 12,
+        "y": 1
+      },
+      "id": 6,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"})",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Memory (bytes)",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 15,
+        "y": 1
+      },
+      "id": 7,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Cores",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 1
+      },
+      "id": 8,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Network RX",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 1
+      },
+      "id": 9,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Network TX",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 5
+      },
+      "id": 10,
+      "panels": [],
+      "title": "Resources",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 6
+      },
+      "id": 11,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[$interval])",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 6
+      },
+      "id": 12,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 6
+      },
+      "id": 13,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[$interval])",
+          "legendFormat": "RX {{pod}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[$interval])",
+          "legendFormat": "TX {{pod}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Network I/O",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 14
+      },
+      "id": 14,
+      "panels": [],
+      "title": "Logs",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 10,
+        "w": 24,
+        "x": 0,
+        "y": 15
+      },
+      "id": 15,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"questdb.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Stream",
+      "type": "logs"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "id": 16,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"questdb.*\"} | json | level =~ \"ERROR|WARN|FATAL\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Errors & Warnings",
+      "type": "logs"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "platform",
+    "questdb"
+  ],
+  "templating": {
+    "list": [
+      {
+        "auto": false,
+        "current": {
+          "text": "5m",
+          "value": "5m"
+        },
+        "name": "interval",
+        "options": [
+          {
+            "selected": false,
+            "text": "1m",
+            "value": "1m"
+          },
+          {
+            "selected": true,
+            "text": "5m",
+            "value": "5m"
+          },
+          {
+            "selected": false,
+            "text": "15m",
+            "value": "15m"
+          },
+          {
+            "selected": false,
+            "text": "30m",
+            "value": "30m"
+          },
+          {
+            "selected": false,
+            "text": "1h",
+            "value": "1h"
+          }
+        ],
+        "query": "1m,5m,15m,30m,1h",
+        "type": "interval"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "QuestDB",
+  "uid": "plat-questdb"
+}

infra/k8s/monitoring/dashboards/platform-tempo.json

@@ -0,0 +1,802 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "panels": [],
+      "title": "Tempo Health",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 0,
+        "y": 1
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", phase=\"Running\"}",
+          "instant": true,
+          "legendFormat": "Running",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Pod Status",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 3
+              },
+              {
+                "color": "red",
+                "value": 10
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 3,
+        "y": 1
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"})",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Restarts",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.7
+              },
+              {
+                "color": "red",
+                "value": 0.9
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 6,
+        "y": 1
+      },
+      "id": 4,
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[$interval]))",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.7
+              },
+              {
+                "color": "red",
+                "value": 0.9
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 9,
+        "y": 1
+      },
+      "id": 5,
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", resource=\"memory\"})",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 12,
+        "y": 1
+      },
+      "id": 6,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"})",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Memory (bytes)",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 15,
+        "y": 1
+      },
+      "id": 7,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Cores",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 1
+      },
+      "id": 8,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Network RX",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 1
+      },
+      "id": 9,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Network TX",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 5
+      },
+      "id": 10,
+      "panels": [],
+      "title": "Resources",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 6
+      },
+      "id": 11,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[$interval])",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 6
+      },
+      "id": 12,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 6
+      },
+      "id": 13,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[$interval])",
+          "legendFormat": "RX {{pod}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[$interval])",
+          "legendFormat": "TX {{pod}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Network I/O",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 14
+      },
+      "id": 14,
+      "panels": [],
+      "title": "Logs",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 10,
+        "w": 24,
+        "x": 0,
+        "y": 15
+      },
+      "id": 15,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Stream",
+      "type": "logs"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 25
+      },
+      "id": 16,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Errors & Warnings",
+      "type": "logs"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "platform",
+    "${DS_TEMPO}"
+  ],
+  "templating": {
+    "list": [
+      {
+        "auto": false,
+        "current": {
+          "text": "5m",
+          "value": "5m"
+        },
+        "name": "interval",
+        "options": [
+          {
+            "selected": false,
+            "text": "1m",
+            "value": "1m"
+          },
+          {
+            "selected": true,
+            "text": "5m",
+            "value": "5m"
+          },
+          {
+            "selected": false,
+            "text": "15m",
+            "value": "15m"
+          },
+          {
+            "selected": false,
+            "text": "30m",
+            "value": "30m"
+          },
+          {
+            "selected": false,
+            "text": "1h",
+            "value": "1h"
+          }
+        ],
+        "query": "1m,5m,15m,30m,1h",
+        "type": "interval"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "Tempo",
+  "uid": "plat-${DS_TEMPO}"
+}

infra/k8s/monitoring/dashboards/svc-api-gateway.json

@@ -0,0 +1,963 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "panels": [],
+      "title": "Service Health",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": null
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 0,
+        "y": 1
+      },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"api.*\", phase=\"Running\"}",
+          "instant": true,
+          "legendFormat": "Running",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Pod Status",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 3
+              },
+              {
+                "color": "red",
+                "value": 10
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 3,
+        "y": 1
+      },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"api.*\"})",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Restarts",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.7
+              },
+              {
+                "color": "red",
+                "value": 0.9
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 6,
+        "y": 1
+      },
+      "id": 4,
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[$interval]))",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.7
+              },
+              {
+                "color": "red",
+                "value": 0.9
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 9,
+        "y": 1
+      },
+      "id": 5,
+      "options": {
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"api.*\", resource=\"memory\"})",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 12,
+        "y": 1
+      },
+      "id": 6,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"})",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Memory (bytes)",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 15,
+        "y": 1
+      },
+      "id": 7,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Cores",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 18,
+        "y": 1
+      },
+      "id": 8,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Network RX",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 4,
+        "w": 3,
+        "x": 21,
+        "y": 1
+      },
+      "id": 9,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ]
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[5m]))",
+          "instant": true,
+          "legendFormat": "",
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Network TX",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 5
+      },
+      "id": 10,
+      "panels": [],
+      "title": "Resources",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 6
+      },
+      "id": 11,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[$interval])",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 6
+      },
+      "id": 12,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}",
+          "legendFormat": "{{container}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Memory",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 6
+      },
+      "id": 13,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[$interval])",
+          "legendFormat": "RX {{pod}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[$interval])",
+          "legendFormat": "TX {{pod}}",
+          "refId": "B"
+        }
+      ],
+      "title": "Network I/O",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 14
+      },
+      "id": 14,
+      "panels": [],
+      "title": "gRPC Metrics",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 15
+      },
+      "id": 15,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(grpc_server_handled_total{grpc_service=~\"foxhunt.*\", pod=~\"api.*\"}[$interval])",
+          "legendFormat": "{{grpc_method}}",
+          "refId": "A"
+        }
+      ],
+      "title": "gRPC Request Rate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 15
+      },
+      "id": 16,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.99, rate(grpc_server_handling_seconds_bucket{pod=~\"api.*\"}[$interval]))",
+          "legendFormat": "p99",
+          "refId": "A"
+        }
+      ],
+      "title": "gRPC Latency (p99)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 1,
+            "stacking": {
+              "mode": "none"
+            }
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 15
+      },
+      "id": 17,
+      "options": {
+        "legend": {
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rate(grpc_server_handled_total{grpc_code!=\"OK\", pod=~\"api.*\"}[$interval])",
+          "legendFormat": "{{grpc_code}}",
+          "refId": "A"
+        }
+      ],
+      "title": "gRPC Errors",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 23
+      },
+      "id": 18,
+      "panels": [],
+      "title": "Logs",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 10,
+        "w": 24,
+        "x": 0,
+        "y": 24
+      },
+      "id": 19,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"api.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Log Stream",
+      "type": "logs"
+    },
+    {
+      "datasource": {
+        "type": "loki",
+        "uid": "${DS_LOKI}"
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 34
+      },
+      "id": 20,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": false,
+        "showCommonLabels": false,
+        "showLabels": true,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "loki",
+            "uid": "${DS_LOKI}"
+          },
+          "expr": "{namespace=\"foxhunt\", pod=~\"api.*\"} | json | level =~ \"ERROR|WARN|FATAL\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
+          "refId": "A"
+        }
+      ],
+      "title": "Errors & Warnings",
+      "type": "logs"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": [
+    "service",
+    "api"
+  ],
+  "templating": {
+    "list": [
+      {
+        "auto": false,
+        "current": {
+          "text": "5m",
+          "value": "5m"
+        },
+        "name": "interval",
+        "options": [
+          {
+            "selected": false,
+            "text": "1m",
+            "value": "1m"
+          },
+          {
+            "selected": true,
+            "text": "5m",
+            "value": "5m"
+          },
+          {
+            "selected": false,
+            "text": "15m",
+            "value": "15m"
+          },
+          {
+            "selected": false,
+            "text": "30m",
+            "value": "30m"
+          },
+          {
+            "selected": false,
+            "text": "1h",
+            "value": "1h"
+          }
+        ],
+        "query": "1m,5m,15m,30m,1h",
+        "type": "interval"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "browser",
+  "title": "API Gateway",
+  "uid": "svc-api-gateway"
+}

infra/k8s/monitoring/dcgm-exporter.yaml

@@ -0,0 +1,125 @@
+# NVIDIA DCGM Exporter — GPU metrics for Cockpit
+# Runs on all GPU node pools (ci-training, gpu-dev)
+# Alloy auto-discovers and scrapes these pods via annotations
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: monitoring
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: dcgm-counters
+  namespace: monitoring
+data:
+  counters.csv: |
+    # Basic GPU metrics (no DCP/profiling — avoids SYS_ADMIN requirement)
+    dcgm_sm_clock,     gauge, SM clock frequency (in MHz).
+    dcgm_memory_clock, gauge, Memory clock frequency (in MHz).
+    dcgm_memory_temp,  gauge, Memory temperature (in C).
+    dcgm_gpu_temp,     gauge, GPU temperature (in C).
+    dcgm_power_usage,              gauge, Power draw (in W).
+    dcgm_total_energy_consumption, counter, Total energy consumption since boot (in mJ).
+    dcgm_gpu_utilization,      gauge, GPU utilization (in %).
+    dcgm_mem_copy_utilization, gauge, Memory utilization (in %).
+    dcgm_enc_utilization,      gauge, Encoder utilization (in %).
+    dcgm_dec_utilization,      gauge, Decoder utilization (in %).
+    dcgm_xid_errors,           gauge, Value of the last XID error encountered.
+    dcgm_fb_free, gauge, Framebuffer memory free (in MiB).
+    dcgm_fb_used, gauge, Framebuffer memory used (in MiB).
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: dcgm-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: dcgm-exporter
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: dcgm-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: dcgm-exporter
+      annotations:
+        prometheus.io/scrape: "true"
+        prometheus.io/port: "9400"
+        prometheus.io/path: "/metrics"
+    spec:
+      # Only schedule on nodes that have GPUs
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                  - key: nvidia.com/gpu.present
+                    operator: In
+                    values:
+                      - "true"
+      # NOTE: runtimeClassName: nvidia removed — Scaleway Kapsule GPU pools use
+      # the nvidia container runtime as the default containerd handler. The
+      # RuntimeClass CRD is only created by the full NVIDIA GPU Operator, not
+      # the device plugin alone, so requiring it prevents pod scheduling.
+      tolerations:
+        # Tolerate GPU taints so the exporter can run on GPU nodes
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: dcgm-exporter
+          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
+          ports:
+            - name: metrics
+              containerPort: 9400
+          args:
+            - "-f"
+            - "/etc/dcgm-exporter/custom/counters.csv"
+          env:
+            - name: DCGM_EXPORTER_KUBERNETES
+              value: "true"
+          resources:
+            requests:
+              cpu: 50m
+              memory: 256Mi
+            limits:
+              cpu: 200m
+              memory: 1Gi
+          securityContext:
+            runAsNonRoot: false
+            runAsUser: 0
+          volumeMounts:
+            - name: device-metrics
+              mountPath: /var/lib/dcgm
+            - name: custom-counters
+              mountPath: /etc/dcgm-exporter/custom
+              readOnly: true
+      volumes:
+        - name: device-metrics
+          emptyDir: {}
+        - name: custom-counters
+          configMap:
+            name: dcgm-counters
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: dcgm-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: dcgm-exporter
+  annotations:
+    prometheus.io/scrape: "true"
+    prometheus.io/port: "9400"
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: dcgm-exporter
+  ports:
+    - name: metrics
+      port: 9400
+      targetPort: 9400

infra/k8s/monitoring/grafana-datasources.yaml

@@ -0,0 +1,57 @@
+# Grafana datasource provisioning for Loki, Tempo, and QuestDB
+# Apply via: kubectl apply -f grafana-datasources.yaml && kubectl rollout restart deploy/grafana -n foxhunt
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-datasources-extra
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: grafana
+    grafana_datasource: "1"
+data:
+  loki-tempo.yaml: |
+    apiVersion: 1
+    datasources:
+      - name: Loki
+        type: loki
+        access: proxy
+        url: http://loki.foxhunt.svc.cluster.local:3100
+        isDefault: false
+        jsonData:
+          derivedFields:
+            - datasourceUid: tempo
+              matcherRegex: '"trace_id":"(\w+)"'
+              name: TraceID
+              url: "$${__value.raw}"
+      - name: Tempo
+        type: tempo
+        uid: tempo
+        access: proxy
+        url: http://tempo.foxhunt.svc.cluster.local:3200
+        isDefault: false
+        jsonData:
+          tracesToLogsV2:
+            datasourceUid: loki
+            filterByTraceID: true
+          nodeGraph:
+            enabled: true
+          serviceMap:
+            datasourceUid: prometheus
+  questdb.yaml: |
+    apiVersion: 1
+    datasources:
+      - name: QuestDB
+        type: postgres
+        uid: questdb
+        access: proxy
+        url: questdb.foxhunt.svc.cluster.local:8812
+        user: admin
+        database: qdb
+        isDefault: false
+        jsonData:
+          sslmode: disable
+          maxOpenConns: 5
+          postgresVersion: 1200
+          timescaledb: false
+        secureJsonData:
+          password: quest

infra/k8s/monitoring/grafana-netpol.yaml

@@ -0,0 +1,19 @@
+# Admit the tailscale proxy to Grafana so grafana.fxhnt.ai works (default-deny-all otherwise blocks it —
+# this was the 502). Same admit-the-proxy pattern as postgres/gitea/minio. Grafana pod listens on 3000.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: grafana-proxy-ingress
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: grafana
+  policyTypes: [Ingress]
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tailscale-gitlab-proxy
+      ports:
+        - { port: 3000, protocol: TCP }

infra/k8s/monitoring/loki.yaml

@@ -0,0 +1,122 @@
+# Grafana Loki — lightweight log aggregation (single-binary mode)
+# Runs on gitlab node pool alongside Grafana and Prometheus
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: loki-config
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: loki
+data:
+  loki.yaml: |
+    auth_enabled: false
+    server:
+      http_listen_port: 3100
+      grpc_listen_port: 9096
+    common:
+      path_prefix: /loki
+      storage:
+        filesystem:
+          chunks_directory: /loki/chunks
+          rules_directory: /loki/rules
+      replication_factor: 1
+      ring:
+        kvstore:
+          store: inmemory
+    schema_config:
+      configs:
+        - from: "2024-01-01"
+          store: tsdb
+          object_store: filesystem
+          schema: v13
+          index:
+            prefix: index_
+            period: 24h
+    limits_config:
+      retention_period: 168h  # 7 days
+      max_query_series: 500
+    compactor:
+      working_directory: /loki/compactor
+      compaction_interval: 10m
+      retention_enabled: true
+      delete_request_store: filesystem
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: loki
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: loki
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: loki
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: loki
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: infra
+      containers:
+        - name: loki
+          image: grafana/loki:3.4.2
+          args: ["-config.file=/etc/loki/loki.yaml"]
+          ports:
+            - containerPort: 3100
+              name: http
+            - containerPort: 9096
+              name: grpc
+          volumeMounts:
+            - name: config
+              mountPath: /etc/loki
+            - name: data
+              mountPath: /loki
+          resources:
+            requests:
+              cpu: 100m
+              memory: 128Mi
+            limits:
+              cpu: 500m
+              memory: 512Mi
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: 3100
+            initialDelaySeconds: 10
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /ready
+              port: 3100
+            initialDelaySeconds: 30
+            periodSeconds: 15
+      volumes:
+        - name: config
+          configMap:
+            name: loki-config
+        - name: data
+          emptyDir:
+            sizeLimit: 5Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: loki
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: loki
+spec:
+  selector:
+    app.kubernetes.io/name: loki
+  ports:
+    - port: 3100
+      targetPort: 3100
+      name: http
+    - port: 9096
+      targetPort: 9096
+      name: grpc

infra/k8s/monitoring/prometheus-rules-broker.yaml

@@ -0,0 +1,169 @@
+# Broker Gateway alert rules — discovered automatically by Prometheus Operator
+# Apply: kubectl apply -f prometheus-rules-broker.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: foxhunt-broker-alerts
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+    role: alert-rules
+spec:
+  groups:
+    - name: broker_gateway_fix_session
+      interval: 30s
+      rules:
+        # FIX session disconnected >60s
+        - alert: BrokerGatewayFIXSessionDisconnected
+          expr: broker_gateway_fix_session_status == 0
+          for: 60s
+          labels:
+            severity: critical
+            service: broker_gateway_service
+            component: fix_session
+            impact: trading_halted
+          annotations:
+            summary: "FIX session {{ $labels.session_id }} DISCONNECTED"
+            description: "FIX session down >60s. All order routing halted."
+            action: "Check FIX engine logs, verify network to AMP Futures gateway"
+
+        # Order latency P95 > 100ms
+        - alert: BrokerGatewayHighOrderLatency
+          expr: |
+            histogram_quantile(0.95,
+              sum by(le, order_type) (rate(broker_gateway_order_latency_seconds_bucket[5m]))
+            ) * 1000 > 100
+          for: 5m
+          labels:
+            severity: warning
+            service: broker_gateway_service
+            component: order_routing
+            impact: performance_degraded
+          annotations:
+            summary: "Order latency P95 > 100ms for {{ $labels.order_type }}"
+
+        # Error rate > 5%
+        - alert: BrokerGatewayHighErrorRate
+          expr: |
+            (
+              sum by(severity) (rate(broker_gateway_error_total[5m]))
+              /
+              (sum(rate(broker_gateway_orders_submitted_total[5m])) + 0.001)
+            ) > 0.05
+          for: 5m
+          labels:
+            severity: warning
+            service: broker_gateway_service
+            component: error_handling
+            impact: reliability_degraded
+          annotations:
+            summary: "Error rate {{ $value | humanizePercentage }} ({{ $labels.severity }})"
+
+        # No orders in 10m during market hours
+        - alert: BrokerGatewayNoOrderActivity
+          expr: |
+            (time() - broker_gateway_last_order_time) > 600
+            and
+            (hour() >= 9 and hour() < 16)
+          for: 1m
+          labels:
+            severity: warning
+            service: broker_gateway_service
+            component: order_flow
+            impact: trading_inactive
+          annotations:
+            summary: "No order activity for 10+ minutes during market hours"
+
+        # Position mismatch > $10K
+        - alert: BrokerGatewayPositionMismatch
+          expr: |
+            abs(
+              broker_gateway_position_value_usd
+              -
+              on(symbol, account_id) group_left()
+              trading_service_position_value_usd
+            ) > 10000
+          for: 2m
+          labels:
+            severity: critical
+            service: broker_gateway_service
+            component: position_management
+            impact: data_integrity
+          annotations:
+            summary: "Position mismatch ${{ $value | humanize }} on {{ $labels.symbol }}"
+            action: "HALT trading on affected symbol, reconcile with broker"
+
+        # FIX sequence number gap
+        - alert: BrokerGatewayFIXSequenceGap
+          expr: rate(broker_gateway_sequence_number_gap_total[5m]) > 0
+          for: 1m
+          labels:
+            severity: warning
+            service: broker_gateway_service
+            component: fix_session
+            impact: message_loss
+          annotations:
+            summary: "FIX sequence gap on {{ $labels.session_id }} ({{ $value | humanize }} gaps/sec)"
+
+    - name: broker_gateway_performance
+      interval: 60s
+      rules:
+        # High FIX heartbeat RTT
+        - alert: BrokerGatewayHighFIXHeartbeatRTT
+          expr: broker_gateway_fix_heartbeat_rtt_ms > 50
+          for: 5m
+          labels:
+            severity: warning
+            service: broker_gateway_service
+            component: fix_session
+            impact: network_latency
+          annotations:
+            summary: "FIX heartbeat RTT {{ $value }}ms on {{ $labels.session_id }}"
+
+        # High order rejection rate per symbol
+        - alert: BrokerGatewayHighOrderRejectionRate
+          expr: |
+            (
+              sum by(symbol) (rate(broker_gateway_orders_rejected_total[5m]))
+              /
+              (sum by(symbol) (rate(broker_gateway_orders_submitted_total[5m])) + 0.001)
+            ) > 0.10
+          for: 5m
+          labels:
+            severity: warning
+            service: broker_gateway_service
+            component: order_validation
+            impact: order_flow_disrupted
+          annotations:
+            summary: "Order rejection rate {{ $value | humanizePercentage }} for {{ $labels.symbol }}"
+
+        # Slow database queries
+        - alert: BrokerGatewaySlowDatabaseQueries
+          expr: |
+            histogram_quantile(0.95,
+              sum by(le) (rate(database_query_duration_seconds_bucket{service="broker_gateway_service"}[5m]))
+            ) > 0.100
+          for: 5m
+          labels:
+            severity: warning
+            service: broker_gateway_service
+            component: database
+            impact: performance_degraded
+          annotations:
+            summary: "Database query P95 > 100ms"
+
+        # High margin usage
+        - alert: BrokerGatewayHighMarginUsage
+          expr: |
+            (
+              broker_gateway_margin_used_usd
+              / (broker_gateway_cash_balance_usd + 0.001)
+            ) > 0.80
+          for: 5m
+          labels:
+            severity: warning
+            service: broker_gateway_service
+            component: risk_management
+            impact: margin_pressure
+          annotations:
+            summary: "Margin usage {{ $value | humanizePercentage }} for {{ $labels.account_id }}"

infra/k8s/monitoring/prometheus-rules-hft.yaml

@@ -0,0 +1,328 @@
+# HFT alert rules — discovered automatically by Prometheus Operator
+# Apply: kubectl apply -f prometheus-rules-hft.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: foxhunt-hft-alerts
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+    role: alert-rules
+spec:
+  groups:
+    # ═══════════════════════════════
+    # CRITICAL HFT ALERTS
+    # ═══════════════════════════════
+    - name: hft_critical
+      rules:
+        - alert: TradingServiceDown
+          expr: up{job="foxhunt-services", service="trading-service"} == 0
+          for: 5s
+          labels:
+            severity: critical
+            component: trading_engine
+            impact: trading_halt
+          annotations:
+            summary: "Trading Service is DOWN"
+            description: "Trading Service has been down for >5s. All trading operations halted."
+            action: "Immediate intervention required"
+
+        - alert: DataAcquisitionDown
+          expr: up{job="foxhunt-services", service="data-acquisition-service"} == 0
+          for: 10s
+          labels:
+            severity: critical
+            component: market_data
+            impact: blind_trading
+          annotations:
+            summary: "Data Acquisition Service is DOWN"
+            description: "Market data feed down >10s. Trading without data is dangerous."
+            action: "Stop all trading immediately"
+
+        - alert: BrokerGatewayDown
+          expr: up{job="foxhunt-services", service="broker-gateway-service"} == 0
+          for: 5s
+          labels:
+            severity: critical
+            component: risk_management
+            impact: uncontrolled_risk
+          annotations:
+            summary: "Broker Gateway (risk controls) is DOWN"
+            description: "Broker gateway down >5s. Order routing and risk controls offline."
+            action: "Emergency trading halt"
+
+        - alert: TradingLatencyHigh
+          expr: histogram_quantile(0.95, foxhunt_trading_latency_seconds_bucket{operation="order_placement"}) > 0.001
+          for: 30s
+          labels:
+            severity: critical
+            component: trading_engine
+            impact: competitive_disadvantage
+          annotations:
+            summary: "Trading latency P95 > 1ms"
+            description: "95th percentile order placement latency is {{ $value }}s. HFT advantage compromised."
+            action: "Investigate performance bottlenecks"
+
+        - alert: PositionLimitBreach
+          expr: foxhunt_position_size_usd / foxhunt_position_limit_usd > 0.95
+          for: 0s
+          labels:
+            severity: critical
+            component: risk_management
+            impact: regulatory_breach
+          annotations:
+            summary: "Position limit nearly breached ({{ $value | humanizePercentage }})"
+            description: "Current position at {{ $value | humanizePercentage }} of limit."
+            action: "Reduce positions immediately"
+
+        - alert: LossLimitBreach
+          expr: foxhunt_daily_pnl_usd < foxhunt_loss_limit_usd
+          for: 0s
+          labels:
+            severity: critical
+            component: risk_management
+            impact: financial_loss
+          annotations:
+            summary: "Daily loss limit breached"
+            description: "Daily P&L ${{ $value }} breaches loss limit. Trading must halt."
+            action: "Emergency trading halt"
+
+    # ═══════════════════════════════
+    # HIGH PRIORITY ALERTS
+    # ═══════════════════════════════
+    - name: hft_high
+      rules:
+        - alert: DatabaseConnectionHigh
+          expr: foxhunt_database_connections_active / foxhunt_database_connections_max > 0.85
+          for: 1m
+          labels:
+            severity: high
+            component: database
+            impact: performance_degradation
+          annotations:
+            summary: "Database connection pool at {{ $value | humanizePercentage }}"
+
+        - alert: MemoryUsageHigh
+          expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
+          for: 2m
+          labels:
+            severity: high
+            component: system
+            impact: performance_degradation
+          annotations:
+            summary: "Memory usage {{ $value | humanizePercentage }} on {{ $labels.instance }}"
+
+        - alert: CPUUsageHigh
+          expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
+          for: 5m
+          labels:
+            severity: high
+            component: system
+            impact: performance_degradation
+          annotations:
+            summary: "CPU usage {{ $value | printf \"%.1f\" }}% on {{ $labels.instance }}"
+
+        - alert: DiskUsageHigh
+          expr: (1 - (node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"})) > 0.85
+          for: 5m
+          labels:
+            severity: high
+            component: system
+            impact: storage_failure
+          annotations:
+            summary: "Disk usage {{ $value | humanizePercentage }} on {{ $labels.instance }} {{ $labels.mountpoint }}"
+
+        - alert: OrderRejectionRateHigh
+          expr: rate(foxhunt_orders_rejected_total[5m]) / rate(foxhunt_orders_total[5m]) > 0.05
+          for: 2m
+          labels:
+            severity: high
+            component: trading_engine
+            impact: trading_inefficiency
+          annotations:
+            summary: "Order rejection rate {{ $value | humanizePercentage }}"
+
+        - alert: MarketDataLag
+          expr: foxhunt_market_data_lag_seconds > 0.1
+          for: 1m
+          labels:
+            severity: high
+            component: market_data
+            impact: stale_data
+          annotations:
+            summary: "Market data lag {{ $value }}s — trading on stale data"
+
+        - alert: DrawdownHigh
+          expr: foxhunt_drawdown_percent > 0.05
+          for: 15m
+          labels:
+            severity: high
+            component: risk_management
+            impact: financial_risk
+          annotations:
+            summary: "Portfolio drawdown {{ $value | humanizePercentage }}"
+
+    # ═══════════════════════════════
+    # MEDIUM / WARNING ALERTS
+    # ═══════════════════════════════
+    - name: hft_medium
+      rules:
+        - alert: ServiceRestarted
+          expr: increase(process_start_time_seconds[10m]) > 0
+          for: 0s
+          labels:
+            severity: medium
+            component: service
+            impact: disruption
+          annotations:
+            summary: "Service {{ $labels.job }} restarted on {{ $labels.instance }}"
+
+        - alert: NetworkLatencyHigh
+          expr: foxhunt_network_latency_seconds > 0.01
+          for: 5m
+          labels:
+            severity: medium
+            component: network
+            impact: performance_impact
+          annotations:
+            summary: "Network latency {{ $value }}s"
+
+        - alert: BackupFailed
+          expr: time() - foxhunt_last_backup_timestamp > 86400
+          for: 1h
+          labels:
+            severity: medium
+            component: backup
+            impact: data_risk
+          annotations:
+            summary: "Last backup {{ $value | humanizeDuration }} ago"
+
+        - alert: SSLCertificateExpiringSoon
+          expr: (ssl_certificate_expiry_timestamp - time()) / 86400 < 30
+          for: 1h
+          labels:
+            severity: medium
+            component: security
+            impact: service_disruption
+          annotations:
+            summary: "SSL cert expires in {{ $value }} days for {{ $labels.instance }}"
+
+    # ═══════════════════════════════
+    # BUSINESS LOGIC ALERTS
+    # ═══════════════════════════════
+    - name: hft_business
+      rules:
+        - alert: TradingVolumeAnomalyHigh
+          expr: rate(foxhunt_trades_total[5m]) > (avg_over_time(rate(foxhunt_trades_total[5m])[1h:5m]) * 3)
+          for: 2m
+          labels:
+            severity: medium
+            component: trading_engine
+            impact: business_anomaly
+          annotations:
+            summary: "Trading volume 3x hourly average ({{ $value }} trades/sec)"
+
+        - alert: TradingVolumeAnomalyLow
+          expr: rate(foxhunt_trades_total[5m]) < (avg_over_time(rate(foxhunt_trades_total[5m])[1h:5m]) * 0.1)
+          for: 10m
+          labels:
+            severity: medium
+            component: trading_engine
+            impact: business_anomaly
+          annotations:
+            summary: "Trading volume 10% of hourly average ({{ $value }} trades/sec)"
+
+        - alert: PnLVolatilityHigh
+          expr: stddev_over_time(foxhunt_pnl_usd[1h]) > 10000
+          for: 30m
+          labels:
+            severity: medium
+            component: risk_management
+            impact: financial_risk
+          annotations:
+            summary: "P&L std dev ${{ $value }} over 1h"
+
+    # ═══════════════════════════════
+    # INFRASTRUCTURE ALERTS
+    # ═══════════════════════════════
+    - name: hft_infrastructure
+      rules:
+        - alert: RedisConnectionFailed
+          expr: redis_connected_clients{job="redis"} == 0
+          for: 30s
+          labels:
+            severity: critical
+            component: redis
+            impact: cache_failure
+          annotations:
+            summary: "Redis has 0 connected clients"
+
+        - alert: PostgreSQLDown
+          expr: pg_up{job="postgres"} == 0
+          for: 30s
+          labels:
+            severity: critical
+            component: postgresql
+            impact: data_unavailable
+          annotations:
+            summary: "PostgreSQL is not responding"
+
+    # ═══════════════════════════════
+    # SECURITY ALERTS
+    # ═══════════════════════════════
+    - name: hft_security
+      rules:
+        - alert: AuthenticationFailuresHigh
+          expr: rate(foxhunt_auth_failures_total[5m]) > 5
+          for: 2m
+          labels:
+            severity: high
+            component: security
+            impact: security_breach
+          annotations:
+            summary: "{{ $value }} auth failures/sec"
+
+        - alert: UnauthorizedAccessAttempts
+          expr: rate(foxhunt_unauthorized_requests_total[5m]) > 1
+          for: 1m
+          labels:
+            severity: critical
+            component: security
+            impact: security_breach
+          annotations:
+            summary: "{{ $value }} unauthorized requests/sec from {{ $labels.source_ip }}"
+
+        - alert: APIRateLimitExceeded
+          expr: rate(foxhunt_api_rate_limit_exceeded_total[5m]) > 0.1
+          for: 5m
+          labels:
+            severity: medium
+            component: api
+            impact: service_degradation
+          annotations:
+            summary: "API rate limits exceeded {{ $value }}/sec"
+
+    # ═══════════════════════════════
+    # MONITORING HEALTH
+    # ═══════════════════════════════
+    - name: hft_monitoring
+      rules:
+        - alert: PrometheusTargetDown
+          expr: up == 0
+          for: 2m
+          labels:
+            severity: medium
+            component: monitoring
+            impact: observability_loss
+          annotations:
+            summary: "{{ $labels.job }} target {{ $labels.instance }} down >2m"
+
+        - alert: HighAlertRate
+          expr: rate(prometheus_notifications_total[5m]) > 10
+          for: 10m
+          labels:
+            severity: medium
+            component: monitoring
+            impact: alert_fatigue
+          annotations:
+            summary: "{{ $value }} alerts/sec — check for alert storm"

infra/k8s/monitoring/prometheus-rules-storage.yaml

@@ -0,0 +1,56 @@
+# Storage alert rules — volume usage warnings before auto-expand kicks in
+# Apply: kubectl apply -f prometheus-rules-storage.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: foxhunt-storage-alerts
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+    role: alert-rules
+spec:
+  groups:
+    - name: storage_volume_usage
+      interval: 60s
+      rules:
+        # PVC usage > 75% — warning (auto-expand triggers at 85%)
+        - alert: PVCUsageHigh
+          expr: |
+            100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
+              / kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 75
+          for: 5m
+          labels:
+            severity: warning
+            component: storage
+          annotations:
+            summary: "PVC {{ $labels.persistentvolumeclaim }} at {{ $value | printf \"%.0f\" }}% usage"
+            description: "Volume {{ $labels.persistentvolumeclaim }} is {{ $value | printf \"%.0f\" }}% full. Auto-expand triggers at 85%."
+
+        # PVC usage > 90% — critical (auto-expand should have already acted)
+        - alert: PVCUsageCritical
+          expr: |
+            100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
+              / kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 90
+          for: 2m
+          labels:
+            severity: critical
+            component: storage
+          annotations:
+            summary: "PVC {{ $labels.persistentvolumeclaim }} CRITICAL at {{ $value | printf \"%.0f\" }}%"
+            description: "Volume {{ $labels.persistentvolumeclaim }} is {{ $value | printf \"%.0f\" }}% full despite auto-expand. Investigate immediately."
+            action: "Check pvc-autoscaler CronJob logs. Manual expand: kubectl patch pvc {{ $labels.persistentvolumeclaim }} -n foxhunt --type merge -p '{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"XXGi\"}}}}'"
+
+        # PVC approaching max auto-expand limit (>80Gi of 100Gi max)
+        - alert: PVCApproachingMaxSize
+          expr: |
+            kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 80 * 1024 * 1024 * 1024
+            and
+            100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
+              / kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 70
+          for: 10m
+          labels:
+            severity: warning
+            component: storage
+          annotations:
+            summary: "PVC {{ $labels.persistentvolumeclaim }} is large and growing"
+            description: "Volume is >80Gi and >70% full. Auto-expand max is 100Gi. Consider manual intervention or data cleanup."

infra/k8s/monitoring/prometheus-stack-values.yaml

@@ -0,0 +1,400 @@
+# kube-prometheus-stack Helm values — Foxhunt production monitoring
+#
+# Install:
+#   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+#   helm repo update
+#   helm install prometheus-stack prometheus-community/kube-prometheus-stack \
+#     --namespace foxhunt \
+#     -f infra/k8s/monitoring/prometheus-stack-values.yaml
+#
+# Upgrade:
+#   helm upgrade prometheus-stack prometheus-community/kube-prometheus-stack \
+#     --namespace foxhunt \
+#     -f infra/k8s/monitoring/prometheus-stack-values.yaml
+#
+# Prerequisites:
+#   kubectl -n foxhunt create secret generic alertmanager-slack \
+#     --from-literal=slack-webhook-url='https://hooks.slack.com/services/...'
+#
+# After install, delete old standalone resources:
+#   kubectl -n foxhunt delete deploy prometheus
+#   kubectl -n foxhunt delete svc prometheus
+#   kubectl -n foxhunt delete cm prometheus-config
+#   kubectl -n foxhunt delete pvc prometheus-data
+#   kubectl -n foxhunt delete sa prometheus
+#   kubectl delete clusterrole foxhunt-prometheus
+#   kubectl delete clusterrolebinding foxhunt-prometheus
+#   kubectl -n foxhunt delete deploy node-exporter
+#   kubectl -n foxhunt delete ds node-exporter  # if DaemonSet
+#   kubectl -n foxhunt delete deploy kube-state-metrics
+
+# ─────────────────────────────────────────────
+# Prometheus Operator
+# ─────────────────────────────────────────────
+prometheusOperator:
+  enabled: true
+  nodeSelector:
+    k8s.scaleway.com/pool-name: platform
+  resources:
+    requests:
+      cpu: 50m
+      memory: 64Mi
+    limits:
+      cpu: 200m
+      memory: 128Mi
+
+# ─────────────────────────────────────────────
+# Prometheus
+# ─────────────────────────────────────────────
+prometheus:
+  prometheusSpec:
+    nodeSelector:
+      k8s.scaleway.com/pool-name: platform
+    podMetadata:
+      labels:
+        app.kubernetes.io/part-of: foxhunt
+    retention: 14d
+    retentionSize: 8GB
+    replicas: 1
+    resources:
+      requests:
+        cpu: 100m
+        memory: 256Mi
+      limits:
+        cpu: 500m
+        memory: 512Mi
+    storageSpec:
+      volumeClaimTemplate:
+        spec:
+          storageClassName: scw-bssd
+          accessModes: ["ReadWriteOnce"]
+          resources:
+            requests:
+              storage: 10Gi
+
+    # Discover ALL ServiceMonitors/PodMonitors/PrometheusRules in the cluster,
+    # not just those matching Helm release labels.
+    serviceMonitorSelectorNilUsesHelmValues: false
+    podMonitorSelectorNilUsesHelmValues: false
+    ruleSelectorNilUsesHelmValues: false
+
+    # Scrape configs for targets that don't fit ServiceMonitor/PodMonitor CRDs
+    additionalScrapeConfigs:
+      # Argo Workflows controller (HTTPS with self-signed cert)
+      - job_name: argo-workflows
+        scheme: https
+        tls_config:
+          insecure_skip_verify: true
+        static_configs:
+          - targets: ['argo-workflows-workflow-controller-metrics.foxhunt.svc.cluster.local:9090']
+        relabel_configs:
+          - target_label: service
+            replacement: argo-workflows
+
+      # Training workflow pods (ephemeral Argo pods on port 9094, Running only)
+      - job_name: training-pods
+        kubernetes_sd_configs:
+          - role: pod
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
+            action: keep
+            regex: training-workflow|compile-and-train
+          - source_labels: [__meta_kubernetes_pod_phase]
+            action: keep
+            regex: Running
+          - source_labels: [__meta_kubernetes_pod_ip]
+            target_label: __address__
+            replacement: ${1}:9094
+          - source_labels: [__meta_kubernetes_pod_name]
+            target_label: pod
+          - source_labels: [__meta_kubernetes_pod_label_workflows_argoproj_io_workflow]
+            target_label: workflow
+
+      # QuestDB (ML metrics store)
+      - job_name: questdb
+        metrics_path: /metrics
+        static_configs:
+          - targets: ['questdb.foxhunt.svc.cluster.local:9003']
+        relabel_configs:
+          - target_label: service
+            replacement: questdb
+
+      # Pushgateway (training job push metrics)
+      - job_name: pushgateway
+        honor_labels: true
+        static_configs:
+          - targets: ['pushgateway.foxhunt.svc.cluster.local:9091']
+
+      # Loki metrics
+      - job_name: loki
+        static_configs:
+          - targets: ['loki.foxhunt.svc.cluster.local:3100']
+
+      # Tempo metrics
+      - job_name: tempo
+        static_configs:
+          - targets: ['tempo.foxhunt.svc.cluster.local:3200']
+
+# ─────────────────────────────────────────────
+# AlertManager
+# ─────────────────────────────────────────────
+alertmanager:
+  alertmanagerSpec:
+    nodeSelector:
+      k8s.scaleway.com/pool-name: platform
+    replicas: 1
+    resources:
+      requests:
+        cpu: 25m
+        memory: 32Mi
+      limits:
+        cpu: 100m
+        memory: 64Mi
+    # Mount K8s Secrets into AlertManager at /etc/alertmanager/secrets/<name>/
+    secrets:
+      - alertmanager-mattermost
+
+  config:
+    global:
+      # Mattermost incoming webhook (Slack-compatible API).
+      # Reads URL from K8s Secret: alertmanager-mattermost/webhook-url
+      # Create with:
+      #   kubectl -n foxhunt create secret generic alertmanager-mattermost \
+      #     --from-literal=webhook-url='http://mattermost.foxhunt.svc.cluster.local:8065/hooks/<HOOK_ID>'
+      slack_api_url_file: /etc/alertmanager/secrets/alertmanager-mattermost/webhook-url
+
+    route:
+      receiver: default
+      group_by: ['alertname', 'severity', 'service']
+      group_wait: 5s
+      group_interval: 30s
+      repeat_interval: 4h
+
+      routes:
+        # CRITICAL trading/risk → immediate Slack + PagerDuty
+        - matchers:
+            - severity="critical"
+            - component=~"trading_engine|risk_management|trading|risk"
+          receiver: critical-trading
+          group_wait: 0s
+          group_interval: 5s
+          repeat_interval: 5m
+          continue: true
+
+        # CRITICAL infrastructure → Slack critical channel
+        - matchers:
+            - severity="critical"
+          receiver: critical-infra
+          group_wait: 0s
+          group_interval: 10s
+          repeat_interval: 15m
+
+        # HIGH priority → Slack alerts channel
+        - matchers:
+            - severity="high"
+          receiver: high-priority
+          group_wait: 10s
+          group_interval: 1m
+          repeat_interval: 30m
+
+        # WARNING / MEDIUM → Slack digest
+        - matchers:
+            - severity=~"warning|WARNING|medium"
+          receiver: warning-alerts
+          group_wait: 30s
+          group_interval: 5m
+          repeat_interval: 4h
+
+    receivers:
+      - name: default
+        slack_configs:
+          - channel: '#foxhunt-alerts'
+            title: '{{ .GroupLabels.alertname }}'
+            text: >-
+              {{ range .Alerts }}*{{ .Labels.severity | toUpper }}*: {{ .Annotations.summary }}
+              {{ end }}
+            send_resolved: true
+
+      - name: critical-trading
+        slack_configs:
+          - channel: '#foxhunt-critical'
+            title: 'CRITICAL: {{ .GroupLabels.alertname }}'
+            text: >-
+              {{ range .Alerts }}
+              *Alert*: {{ .Annotations.summary }}
+              *Impact*: {{ .Annotations.impact }}
+              *Action*: {{ .Annotations.action }}
+              {{ end }}
+            color: 'danger'
+            send_resolved: true
+
+      - name: critical-infra
+        slack_configs:
+          - channel: '#foxhunt-critical'
+            title: 'INFRA CRITICAL: {{ .GroupLabels.alertname }}'
+            text: >-
+              {{ range .Alerts }}{{ .Annotations.summary }}
+              {{ end }}
+            color: 'danger'
+            send_resolved: true
+
+      - name: high-priority
+        slack_configs:
+          - channel: '#foxhunt-alerts'
+            title: 'HIGH: {{ .GroupLabels.alertname }}'
+            text: >-
+              {{ range .Alerts }}{{ .Annotations.summary }}
+              {{ end }}
+            color: 'warning'
+            send_resolved: true
+
+      - name: warning-alerts
+        slack_configs:
+          - channel: '#foxhunt-alerts'
+            title: 'WARNING: {{ .GroupLabels.alertname }}'
+            text: >-
+              {{ range .Alerts }}{{ .Annotations.summary }}
+              {{ end }}
+            color: '#ff9500'
+            send_resolved: true
+
+    inhibit_rules:
+      # Trading engine down → suppress all trading alerts
+      - source_matchers:
+          - alertname="TradingEngineDown"
+        target_matchers:
+          - component=~"trading_engine|trading"
+        equal: ['service']
+
+      # Node down → suppress service-level alerts on that node
+      - source_matchers:
+          - alertname=~"NodeDown|KubeNodeNotReady"
+        target_matchers:
+          - severity=~"high|warning|medium"
+        equal: ['instance']
+
+      # PostgreSQL down → suppress database alerts
+      - source_matchers:
+          - alertname="PostgreSQLDown"
+        target_matchers:
+          - component="database"
+
+# ─────────────────────────────────────────────
+# ServiceMonitors
+# ─────────────────────────────────────────────
+additionalServiceMonitors:
+  # All foxhunt microservices (7 services, all expose named port "metrics")
+  - name: foxhunt-services
+    selector:
+      matchLabels:
+        app.kubernetes.io/part-of: foxhunt
+    namespaceSelector:
+      matchNames: [foxhunt]
+    endpoints:
+      - port: metrics
+        interval: 15s
+
+  # DCGM Exporter (GPU metrics from kube-system)
+  - name: dcgm-exporter
+    selector:
+      matchExpressions:
+        - key: app.kubernetes.io/name
+          operator: In
+          values: [dcgm-exporter]
+    namespaceSelector:
+      matchNames: [kube-system, monitoring, foxhunt]
+    endpoints:
+      - port: metrics
+        interval: 15s
+
+# ─────────────────────────────────────────────
+# Grafana (disabled — managed separately via grafana Helm chart)
+# ─────────────────────────────────────────────
+grafana:
+  enabled: false
+
+# ─────────────────────────────────────────────
+# Node Exporter (replaces standalone node-exporter.yaml)
+# ─────────────────────────────────────────────
+prometheus-node-exporter:
+  tolerations:
+    - effect: NoSchedule
+      operator: Exists
+  resources:
+    requests:
+      cpu: 10m
+      memory: 16Mi
+    limits:
+      cpu: 50m
+      memory: 32Mi
+
+# ─────────────────────────────────────────────
+# kube-state-metrics (replaces standalone kube-state-metrics.yaml)
+# ─────────────────────────────────────────────
+kube-state-metrics:
+  nodeSelector:
+    k8s.scaleway.com/pool-name: platform
+  resources:
+    requests:
+      cpu: 10m
+      memory: 32Mi
+    limits:
+      cpu: 50m
+      memory: 64Mi
+
+# ─────────────────────────────────────────────
+# Default rules (Kubernetes cluster health alerts included out-of-box)
+# ─────────────────────────────────────────────
+defaultRules:
+  create: true
+  rules:
+    alertmanager: true
+    etcd: false            # managed by Scaleway Kapsule
+    configReloaders: true
+    general: true
+    k8sContainerCpuUsageSecondsTotal: true
+    k8sContainerMemoryCache: true
+    k8sContainerMemoryRss: true
+    k8sContainerMemorySwap: true
+    k8sContainerMemoryWorkingSetBytes: true
+    k8sPodOwner: true
+    kubeApiserverAvailability: false  # managed by Scaleway
+    kubeApiserverBurnrate: false
+    kubeApiserverHistogram: false
+    kubeApiserverSlos: false
+    kubeControllerManager: false      # managed by Scaleway
+    kubelet: false              # kubelet metrics not accessible on Kapsule
+    kubeProxy: false                  # managed by Scaleway
+    kubePrometheusGeneral: true
+    kubePrometheusNodeRecording: true
+    kubernetesAbsent: true
+    kubernetesApps: true
+    kubernetesResources: true
+    kubernetesStorage: true
+    kubernetesSystem: true
+    kubeScheduler: false              # managed by Scaleway
+    kubeStateMetrics: true
+    network: true
+    node: true
+    nodeExporterAlerting: true
+    nodeExporterRecording: true
+    prometheus: true
+    prometheusOperator: true
+
+# ─────────────────────────────────────────────
+# Disable components managed by Scaleway Kapsule
+# ─────────────────────────────────────────────
+kubeApiServer:
+  enabled: false
+kubeControllerManager:
+  enabled: false
+kubeScheduler:
+  enabled: false
+kubeProxy:
+  enabled: false
+kubelet:
+  enabled: false               # kubelet metrics not accessible on Kapsule
+kubeEtcd:
+  enabled: false

infra/k8s/monitoring/prometheus-values.yaml

@@ -0,0 +1,172 @@
+# Prometheus override values for GitLab Helm chart
+# Removes Cockpit remote_write, adds comprehensive cluster scrape configs
+#
+# Apply with:
+#   helm upgrade gitlab gitlab/gitlab -n foxhunt --reuse-values \
+#     -f infra/k8s/monitoring/prometheus-values.yaml
+prometheus:
+  server:
+    # Remove Cockpit remote_write — all metrics stay local
+    remoteWrite: []
+
+  # Extra scrape configs for full cluster observability
+  # NOTE: this is at prometheus.extraScrapeConfigs (chart root), NOT prometheus.server.extraScrapeConfigs
+  extraScrapeConfigs: |
+      # --- Host metrics from node-exporter ---
+      - job_name: node-exporter
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
+            action: keep
+            regex: node-exporter
+          - source_labels: [__meta_kubernetes_endpoints_name]
+            action: keep
+            regex: node-exporter
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+
+      # --- Kubernetes object state ---
+      - job_name: kube-state-metrics
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
+            action: keep
+            regex: kube-state-metrics
+          - source_labels: [__meta_kubernetes_endpoint_port_name]
+            action: keep
+            regex: http
+
+      # --- Kubelet metrics (container resource usage) ---
+      - job_name: kubelet
+        scheme: https
+        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+        tls_config:
+          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+          insecure_skip_verify: true
+        kubernetes_sd_configs:
+          - role: node
+        relabel_configs:
+          - target_label: __address__
+            replacement: kubernetes.default.svc:443
+          - source_labels: [__meta_kubernetes_node_name]
+            regex: (.+)
+            target_label: __metrics_path__
+            replacement: /api/v1/nodes/$1/proxy/metrics
+
+      # --- cAdvisor (container CPU, memory, network, filesystem) ---
+      - job_name: cadvisor
+        scheme: https
+        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+        tls_config:
+          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+          insecure_skip_verify: true
+        kubernetes_sd_configs:
+          - role: node
+        relabel_configs:
+          - target_label: __address__
+            replacement: kubernetes.default.svc:443
+          - source_labels: [__meta_kubernetes_node_name]
+            regex: (.+)
+            target_label: __metrics_path__
+            replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
+        metric_relabel_configs:
+          - source_labels: [namespace]
+            action: keep
+            regex: (foxhunt|monitoring|kube-system)
+
+      # --- GPU metrics from NVIDIA DCGM Exporter ---
+      - job_name: dcgm-exporter
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [kube-system, monitoring]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_name]
+            action: keep
+            regex: .*dcgm.*
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+
+      # --- Foxhunt services (app metrics ports) ---
+      - job_name: foxhunt-services
+        kubernetes_sd_configs:
+          - role: endpoints
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_part_of]
+            action: keep
+            regex: foxhunt
+          - source_labels: [__meta_kubernetes_endpoint_port_name]
+            action: keep
+            regex: metrics
+          - source_labels: [__meta_kubernetes_service_name]
+            target_label: service
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+
+      # --- General prometheus.io annotation-based scraping ---
+      - job_name: prometheus-annotated-pods
+        kubernetes_sd_configs:
+          - role: pod
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+            action: keep
+            regex: "true"
+          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
+            action: replace
+            regex: (https?)
+            target_label: __scheme__
+          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
+            action: replace
+            regex: (.+)
+            target_label: __metrics_path__
+          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
+            action: replace
+            regex: ([^:]+)(?::\d+)?;(\d+)
+            replacement: $1:$2
+            target_label: __address__
+          - action: labelmap
+            regex: __meta_kubernetes_pod_label_(.+)
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+          - source_labels: [__meta_kubernetes_pod_name]
+            target_label: pod
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+
+      # --- Loki metrics ---
+      - job_name: loki
+        static_configs:
+          - targets: ['loki.foxhunt.svc.cluster.local:3100']
+
+      # --- Tempo metrics ---
+      - job_name: tempo
+        static_configs:
+          - targets: ['tempo.foxhunt.svc.cluster.local:3200']
+
+      # --- Promtail metrics ---
+      - job_name: promtail
+        kubernetes_sd_configs:
+          - role: pod
+            namespaces:
+              names: [foxhunt]
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
+            action: keep
+            regex: promtail
+          - source_labels: [__meta_kubernetes_pod_ip]
+            target_label: __address__
+            replacement: $1:9080
+
+      # --- Pushgateway (training job metrics) ---
+      - job_name: pushgateway
+        honor_labels: true
+        static_configs:
+          - targets: ['pushgateway.foxhunt.svc.cluster.local:9091']

infra/k8s/monitoring/promtail.yaml

@@ -0,0 +1,265 @@
+# Promtail — ships container logs to Loki (DaemonSet on all nodes)
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: promtail
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: promtail
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: promtail
+  labels:
+    app.kubernetes.io/name: promtail
+rules:
+  - apiGroups: [""]
+    resources: ["nodes", "nodes/proxy", "services", "endpoints", "pods"]
+    verbs: ["get", "watch", "list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: promtail
+  labels:
+    app.kubernetes.io/name: promtail
+subjects:
+  - kind: ServiceAccount
+    name: promtail
+    namespace: foxhunt
+roleRef:
+  kind: ClusterRole
+  name: promtail
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: promtail-config
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: promtail
+data:
+  promtail.yaml: |
+    server:
+      http_listen_port: 9080
+    positions:
+      filename: /run/promtail/positions.yaml
+    clients:
+      - url: http://loki.foxhunt.svc.cluster.local:3100/loki/api/v1/push
+    scrape_configs:
+      # Foxhunt application pods (structured Rust logs)
+      - job_name: foxhunt-pods
+        kubernetes_sd_configs:
+          - role: pod
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_namespace]
+            action: keep
+            regex: foxhunt
+          # Drop noisy infra pods (gitlab internals, prometheus, grafana)
+          - source_labels: [__meta_kubernetes_pod_label_app]
+            action: drop
+            regex: (gitlab|prometheus|grafana|promtail)
+          - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
+            action: drop
+            regex: (promtail|node-exporter|kube-state-metrics)
+          # Map to containerd log path
+          - source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
+            separator: /
+            target_label: __path__
+            replacement: /var/log/pods/*$1/*.log
+          # Standard K8s metadata labels
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+          - source_labels: [__meta_kubernetes_pod_name]
+            target_label: pod
+          - source_labels: [__meta_kubernetes_pod_container_name]
+            target_label: container
+          - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
+            target_label: app
+          - source_labels: [__meta_kubernetes_pod_label_foxhunt_job__type]
+            target_label: job_type
+          - source_labels: [__meta_kubernetes_pod_label_foxhunt_model]
+            target_label: model
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+        pipeline_stages:
+          - json:
+              expressions:
+                level: level
+                message: fields.message
+                target: target
+                timestamp: timestamp
+          - labels:
+              level:
+              target:
+          - match:
+              selector: '{level=""}'
+              stages:
+                - regex:
+                    expression: '(?P<level>TRACE|DEBUG|INFO|WARN|ERROR)'
+                - labels:
+                    level:
+
+      # Kapsule system pods (kube-system, kube-node-lease, kube-public)
+      - job_name: kapsule-system
+        kubernetes_sd_configs:
+          - role: pod
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_namespace]
+            action: keep
+            regex: (kube-system|kube-node-lease|kube-public)
+          # Map to containerd log path
+          - source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
+            separator: /
+            target_label: __path__
+            replacement: /var/log/pods/*$1/*.log
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+          - source_labels: [__meta_kubernetes_pod_name]
+            target_label: pod
+          - source_labels: [__meta_kubernetes_pod_container_name]
+            target_label: container
+          - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
+            target_label: app
+          - source_labels: [__meta_kubernetes_pod_label_app]
+            target_label: kube_app
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+        pipeline_stages:
+          - match:
+              selector: '{namespace="kube-system"}'
+              stages:
+                - regex:
+                    expression: '(?P<level>TRACE|DEBUG|INFO|WARN|ERROR|warning|error|fatal)'
+                - labels:
+                    level:
+
+      # GitLab CI runner logs (foxhunt namespace, runner pods)
+      - job_name: gitlab-runners
+        kubernetes_sd_configs:
+          - role: pod
+        relabel_configs:
+          - source_labels: [__meta_kubernetes_namespace]
+            action: keep
+            regex: foxhunt
+          - source_labels: [__meta_kubernetes_pod_label_app]
+            action: keep
+            regex: gitlab-runner
+          - source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
+            separator: /
+            target_label: __path__
+            replacement: /var/log/pods/*$1/*.log
+          - source_labels: [__meta_kubernetes_namespace]
+            target_label: namespace
+          - source_labels: [__meta_kubernetes_pod_name]
+            target_label: pod
+          - source_labels: [__meta_kubernetes_pod_container_name]
+            target_label: container
+          - replacement: gitlab-runner
+            target_label: app
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            target_label: node
+        pipeline_stages:
+          - regex:
+              expression: '(?P<level>TRACE|DEBUG|INFO|WARN|ERROR|WARNING)'
+          - labels:
+              level:
+
+      # Kubernetes node-level logs (kubelet, containerd)
+      - job_name: kapsule-node-logs
+        static_configs:
+          - targets: [localhost]
+            labels:
+              job: kapsule-nodes
+              __path__: /var/log/syslog
+          - targets: [localhost]
+            labels:
+              job: kapsule-nodes
+              __path__: /var/log/kern.log
+          - targets: [localhost]
+            labels:
+              job: kapsule-nodes
+              __path__: /var/log/kubelet.log
+        relabel_configs:
+          - source_labels: [__path__]
+            regex: .*/(.+)\.log
+            target_label: log_type
+        pipeline_stages:
+          - regex:
+              expression: '(?P<level>TRACE|DEBUG|INFO|WARN|ERROR|warning|error|fatal)'
+          - labels:
+              level:
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: promtail
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: promtail
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: promtail
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: promtail
+    spec:
+      serviceAccountName: promtail
+      tolerations:
+        - effect: NoSchedule
+          operator: Exists
+      containers:
+        - name: promtail
+          image: grafana/promtail:3.4.2
+          args: ["-config.file=/etc/promtail/promtail.yaml"]
+          ports:
+            - containerPort: 9080
+              name: http
+          volumeMounts:
+            - name: config
+              mountPath: /etc/promtail
+            - name: varlog
+              mountPath: /var/log
+              readOnly: true
+            - name: containers
+              mountPath: /var/lib/docker/containers
+              readOnly: true
+            - name: pods
+              mountPath: /var/log/pods
+              readOnly: true
+            - name: run
+              mountPath: /run/promtail
+          env:
+            - name: HOSTNAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 200m
+              memory: 256Mi
+          securityContext:
+            readOnlyRootFilesystem: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: config
+          configMap:
+            name: promtail-config
+        - name: varlog
+          hostPath:
+            path: /var/log
+        - name: containers
+          hostPath:
+            path: /var/lib/docker/containers
+        - name: pods
+          hostPath:
+            path: /var/log/pods
+        - name: run
+          emptyDir: {}

infra/k8s/monitoring/pushgateway.yaml

@@ -0,0 +1,61 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: pushgateway
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: pushgateway
+    app.kubernetes.io/part-of: monitoring
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: pushgateway
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: pushgateway
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: infra
+      containers:
+        - name: pushgateway
+          image: prom/pushgateway:v1.11.0
+          args:
+            - "--persistence.interval=5m"
+          ports:
+            - containerPort: 9091
+              name: http
+          resources:
+            requests:
+              cpu: 50m
+              memory: 32Mi
+            limits:
+              cpu: 200m
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /-/healthy
+              port: 9091
+            initialDelaySeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /-/ready
+              port: 9091
+            initialDelaySeconds: 5
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: pushgateway
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: pushgateway
+spec:
+  type: ClusterIP
+  ports:
+    - port: 9091
+      targetPort: 9091
+      name: http
+  selector:
+    app.kubernetes.io/name: pushgateway

infra/k8s/monitoring/pvc-autoscaler.yaml

@@ -0,0 +1,250 @@
+# PVC Auto-Expand — expands PVCs automatically when usage exceeds 85%
+# Uses K8s API to list PVCs and exec df inside pods to check usage.
+# Runs every 15 minutes. Apply: kubectl apply -f pvc-autoscaler.yaml
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: pvc-autoscaler
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: pvc-autoscaler
+    app.kubernetes.io/part-of: foxhunt
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: pvc-autoscaler
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: pvc-autoscaler
+    app.kubernetes.io/part-of: foxhunt
+rules:
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["get", "list", "patch"]
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list"]
+  - apiGroups: ["apps"]
+    resources: ["deployments", "statefulsets"]
+    verbs: ["get", "list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: pvc-autoscaler-nodes
+  labels:
+    app.kubernetes.io/name: pvc-autoscaler
+    app.kubernetes.io/part-of: foxhunt
+rules:
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["list"]
+  - apiGroups: [""]
+    resources: ["nodes/proxy"]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: pvc-autoscaler
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: pvc-autoscaler
+    app.kubernetes.io/part-of: foxhunt
+subjects:
+  - kind: ServiceAccount
+    name: pvc-autoscaler
+    namespace: foxhunt
+roleRef:
+  kind: Role
+  name: pvc-autoscaler
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: pvc-autoscaler-nodes
+  labels:
+    app.kubernetes.io/name: pvc-autoscaler
+    app.kubernetes.io/part-of: foxhunt
+subjects:
+  - kind: ServiceAccount
+    name: pvc-autoscaler
+    namespace: foxhunt
+roleRef:
+  kind: ClusterRole
+  name: pvc-autoscaler-nodes
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: pvc-autoscaler-script
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: pvc-autoscaler
+    app.kubernetes.io/part-of: foxhunt
+data:
+  autoscale.py: |
+    #!/usr/bin/env python3
+    """PVC auto-expand via kubelet stats API (no Prometheus dependency)."""
+    import json, os, sys, ssl, urllib.request
+    from datetime import datetime, timezone
+
+    THRESHOLD = int(os.environ.get("PVC_USAGE_THRESHOLD", "85"))
+    GROW_PERCENT = int(os.environ.get("PVC_GROW_PERCENT", "50"))
+    MAX_SIZE_GI = int(os.environ.get("PVC_MAX_SIZE_GI", "100"))
+    NAMESPACE = os.environ.get("NAMESPACE", "foxhunt")
+
+    TOKEN_PATH = "/var/run/secrets/kubernetes.io/serviceaccount/token"
+    CA_PATH = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+    K8S_API = "https://kubernetes.default.svc"
+
+    def k8s(path, method="GET", data=None):
+        token = open(TOKEN_PATH).read().strip()
+        ctx = ssl.create_default_context(cafile=CA_PATH)
+        body = json.dumps(data).encode() if data else None
+        req = urllib.request.Request(f"{K8S_API}{path}", data=body, method=method)
+        req.add_header("Authorization", f"Bearer {token}")
+        if data:
+            req.add_header("Content-Type", "application/merge-patch+json")
+        return json.loads(urllib.request.urlopen(req, context=ctx, timeout=30).read())
+
+    now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+    print(f"=== PVC Auto-Expand {now} ===")
+    print(f"Threshold: {THRESHOLD}% | Grow: +{GROW_PERCENT}% | Max: {MAX_SIZE_GI}Gi")
+    sys.stdout.flush()
+
+    # Step 1: Get all nodes and their kubelet stats via K8s proxy API
+    nodes = k8s("/api/v1/nodes")
+    pvc_usage = {}  # pvc_name -> (used_bytes, capacity_bytes)
+
+    for node in nodes["items"]:
+        node_name = node["metadata"]["name"]
+        try:
+            stats = k8s(f"/api/v1/nodes/{node_name}/proxy/stats/summary")
+            for pod in stats.get("pods", []):
+                if pod.get("podRef", {}).get("namespace") != NAMESPACE:
+                    continue
+                for vol in pod.get("volume", []):
+                    pvc_ref = vol.get("pvcRef")
+                    if not pvc_ref:
+                        continue
+                    pvc_name = pvc_ref["name"]
+                    used = vol.get("usedBytes", 0)
+                    cap = vol.get("capacityBytes", 1)
+                    pvc_usage[pvc_name] = (used, cap)
+        except Exception as e:
+            print(f"  WARN: could not get stats from node {node_name}: {e}")
+    sys.stdout.flush()
+
+    if not pvc_usage:
+        print("  No PVC stats found")
+        sys.exit(0)
+
+    # Step 2: Check each PVC and expand if needed
+    expanded = 0
+    for pvc_name, (used, cap) in sorted(pvc_usage.items()):
+        pct = 100 * used / cap if cap > 0 else 0
+        used_gi = used / (1024**3)
+        cap_gi = cap / (1024**3)
+        print(f"  {pvc_name}: {pct:.1f}% ({used_gi:.1f}G / {cap_gi:.1f}G)")
+
+        if pct >= THRESHOLD:
+            print(f"  >> {pvc_name} exceeds {THRESHOLD}%")
+            try:
+                pvc_obj = k8s(f"/api/v1/namespaces/{NAMESPACE}/persistentvolumeclaims/{pvc_name}")
+                sc = pvc_obj.get("spec", {}).get("storageClassName", "")
+                # Only expand if StorageClass supports it
+                if sc:
+                    try:
+                        sc_obj = k8s(f"/apis/storage.k8s.io/v1/storageclasses/{sc}")
+                        if not sc_obj.get("allowVolumeExpansion"):
+                            print(f"     SKIP: StorageClass {sc} does not allow expansion")
+                            continue
+                    except Exception:
+                        pass
+
+                status_storage = pvc_obj.get("status", {}).get("capacity", {}).get("storage", "0Gi")
+                current_gi = int(status_storage.replace("Gi", ""))
+                spec_storage = pvc_obj["spec"]["resources"]["requests"]["storage"]
+                spec_gi = int(spec_storage.replace("Gi", ""))
+
+                new_gi = current_gi + max(current_gi * GROW_PERCENT // 100, 1)
+                if new_gi > MAX_SIZE_GI:
+                    print(f"     SKIP: target {new_gi}Gi > max {MAX_SIZE_GI}Gi")
+                    continue
+                if spec_gi >= new_gi:
+                    print(f"     SKIP: spec={spec_gi}Gi >= target={new_gi}Gi (resize pending)")
+                    continue
+
+                patch = {"spec": {"resources": {"requests": {"storage": f"{new_gi}Gi"}}}}
+                k8s(f"/api/v1/namespaces/{NAMESPACE}/persistentvolumeclaims/{pvc_name}",
+                    method="PATCH", data=patch)
+                print(f"     EXPANDED: {current_gi}Gi -> {new_gi}Gi")
+                expanded += 1
+            except Exception as e:
+                print(f"     FAILED: {e}", file=sys.stderr)
+        sys.stdout.flush()
+
+    print(f"=== Done ({expanded} expanded) ===")
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: pvc-autoscaler
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: pvc-autoscaler
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  schedule: "*/15 * * * *"
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 120
+      template:
+        metadata:
+          labels:
+            app: pvc-autoscaler
+            app.kubernetes.io/part-of: foxhunt
+        spec:
+          nodeSelector:
+            k8s.scaleway.com/pool-name: platform
+          serviceAccountName: pvc-autoscaler
+          restartPolicy: OnFailure
+          containers:
+            - name: autoscaler
+              image: python:3.12-alpine
+              command: ["python3", "-u", "/scripts/autoscale.py"]
+              env:
+                - name: PVC_USAGE_THRESHOLD
+                  value: "85"
+                - name: PVC_GROW_PERCENT
+                  value: "50"
+                - name: PVC_MAX_SIZE_GI
+                  value: "100"
+                - name: NAMESPACE
+                  value: "foxhunt"
+              volumeMounts:
+                - name: script
+                  mountPath: /scripts
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 64Mi
+                limits:
+                  cpu: 100m
+                  memory: 128Mi
+          volumes:
+            - name: script
+              configMap:
+                name: pvc-autoscaler-script
+                items:
+                  - key: autoscale.py
+                    path: autoscale.py
+                defaultMode: 0755

infra/k8s/monitoring/scaleway-billing.yaml

@@ -0,0 +1,239 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: scaleway-billing
+  namespace: foxhunt
+type: Opaque
+stringData:
+  SCW_SECRET_KEY: "84fa878b-81c2-401a-8aef-22996d30e9ea"
+  SCW_ORG_ID: "a55098a3-2cf6-4f73-b011-fff5a299ed94"
+  SCW_PROJECT_ID: "c293eb98-228d-427d-9b16-f0941f3f2adb"
+  SCW_CLUSTER_ID: "34a1e3c4-ac35-48c8-ab49-5f6ec4df32c1"
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: scaleway-billing-script
+  namespace: foxhunt
+data:
+  scrape.py: |
+    #!/usr/bin/env python3
+    """Scaleway infrastructure scraper → Prometheus pushgateway."""
+    import json, os, sys, urllib.request, urllib.error, re
+
+    def sanitize(s):
+        """Sanitize label values for Prometheus text format."""
+        return str(s).replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
+
+    PUSHGATEWAY = os.environ.get("PUSHGATEWAY_URL", "http://pushgateway.foxhunt.svc.cluster.local:9091")
+    SECRET = os.environ["SCW_SECRET_KEY"]
+    ORG = os.environ["SCW_ORG_ID"]
+    PROJECT = os.environ["SCW_PROJECT_ID"]
+    CLUSTER = os.environ["SCW_CLUSTER_ID"]
+
+    SCW_BILLING  = "https://api.scaleway.com/billing/v2beta1"
+    SCW_INSTANCE = "https://api.scaleway.com/instance/v1/zones/fr-par-2"
+    SCW_K8S      = "https://api.scaleway.com/k8s/v1/regions/fr-par"
+    SCW_BLOCK    = "https://api.scaleway.com/block/v1alpha1/zones/fr-par-2"
+    SCW_IPAM     = "https://api.scaleway.com/ipam/v1/regions/fr-par"
+
+    def fetch(url):
+        req = urllib.request.Request(url)
+        req.add_header("X-Auth-Token", SECRET)
+        try:
+            return json.loads(urllib.request.urlopen(req, timeout=30).read())
+        except Exception as e:
+            print(f"WARN: {url}: {e}", file=sys.stderr)
+            return {}
+
+    # ── Fetch all data (dynamic — auto-discovers any infra changes) ──
+    billing   = fetch(f"{SCW_BILLING}/consumptions?organization_id={ORG}")
+    instances = fetch(f"{SCW_INSTANCE}/servers?project={PROJECT}&per_page=50")
+    pools     = fetch(f"{SCW_K8S}/clusters/{CLUSTER}/pools")
+    volumes   = fetch(f"{SCW_BLOCK}/volumes?project_id={PROJECT}&per_page=100")
+    ips       = fetch(f"{SCW_IPAM}/ips?project_id={PROJECT}&is_ipv6=false&per_page=100")
+
+    L = []
+
+    # ── Billing ──
+    total = 0
+    cats = {}
+    products = {}
+    for c in billing.get("consumptions", []):
+        cat = c.get("category_name", "other").lower().replace(" ", "_").replace("&", "and")
+        product = c.get("product_name", "unknown").lower().replace(" ", "_").replace("-", "_")
+        sku = c.get("sku", "")
+        val = c.get("value", {})
+        amount = int(val.get("units", 0) or 0) + int(val.get("nanos", 0) or 0) / 1e9
+        total += amount
+        cats[cat] = cats.get(cat, 0) + amount
+        products[(cat, product, sku)] = products.get((cat, product, sku), 0) + amount
+
+    L.append("# HELP scaleway_billing_total_eur Total current month billing in EUR")
+    L.append("# TYPE scaleway_billing_total_eur gauge")
+    L.append(f"scaleway_billing_total_eur {total:.4f}")
+    L.append("# HELP scaleway_billing_consumption_eur Billing by category in EUR")
+    L.append("# TYPE scaleway_billing_consumption_eur gauge")
+    for cat, amt in sorted(cats.items(), key=lambda x: -x[1]):
+        L.append(f'scaleway_billing_consumption_eur{{category="{sanitize(cat)}"}} {amt:.4f}')
+    L.append("# HELP scaleway_billing_product_eur Billing by product in EUR")
+    L.append("# TYPE scaleway_billing_product_eur gauge")
+    for (cat, product, sku), amt in sorted(products.items(), key=lambda x: -x[1]):
+        if amt >= 0.01:
+            L.append(f'scaleway_billing_product_eur{{category="{sanitize(cat)}",product="{sanitize(product)}",sku="{sanitize(sku)}"}} {amt:.4f}')
+
+    # ── Instances (dynamic — auto-discovers new/removed instances) ──
+    L.append("# HELP scaleway_instance_info Instance metadata")
+    L.append("# TYPE scaleway_instance_info gauge")
+    L.append("# HELP scaleway_instance_running Instance running state (1=running)")
+    L.append("# TYPE scaleway_instance_running gauge")
+    type_counts = {}
+    for s in instances.get("servers", []):
+        name = s.get("name", "?")
+        itype = s.get("commercial_type", "?")
+        state = s.get("state", "?")
+        tags = dict(t.split("=", 1) for t in s.get("tags", []) if "=" in t)
+        pool = tags.get("pool-name", "unknown")
+        L.append(f'scaleway_instance_info{{name="{sanitize(name)}",type="{sanitize(itype)}",pool="{sanitize(pool)}",state="{sanitize(state)}"}} 1')
+        L.append(f'scaleway_instance_running{{name="{sanitize(name)}",type="{sanitize(itype)}",pool="{sanitize(pool)}"}} {1 if state == "running" else 0}')
+        type_counts[itype] = type_counts.get(itype, 0) + 1
+    L.append("# HELP scaleway_instances_total Total instances by type")
+    L.append("# TYPE scaleway_instances_total gauge")
+    for t, c in type_counts.items():
+        L.append(f'scaleway_instances_total{{type="{sanitize(t)}"}} {c}')
+
+    # ── K8s Pools (dynamic — auto-discovers pool changes/autoscaler config) ──
+    L.append("# HELP scaleway_pool_size Current pool node count")
+    L.append("# TYPE scaleway_pool_size gauge")
+    L.append("# HELP scaleway_pool_min Minimum pool size")
+    L.append("# TYPE scaleway_pool_min gauge")
+    L.append("# HELP scaleway_pool_max Maximum pool size")
+    L.append("# TYPE scaleway_pool_max gauge")
+    L.append("# HELP scaleway_pool_autoscaling Pool autoscaling enabled")
+    L.append("# TYPE scaleway_pool_autoscaling gauge")
+    L.append("# HELP scaleway_pool_root_volume_bytes Pool root volume size")
+    L.append("# TYPE scaleway_pool_root_volume_bytes gauge")
+    for p in pools.get("pools", []):
+        name = p.get("name", "?")
+        ntype = p.get("node_type", "?")
+        status = p.get("status", "?")
+        lb = f'pool="{sanitize(name)}",type="{sanitize(ntype)}",status="{sanitize(status)}"'
+        L.append(f"scaleway_pool_size{{{lb}}} {p.get('size', 0)}")
+        L.append(f"scaleway_pool_min{{{lb}}} {p.get('min_size', 0)}")
+        L.append(f"scaleway_pool_max{{{lb}}} {p.get('max_size', 0)}")
+        L.append(f"scaleway_pool_autoscaling{{{lb}}} {1 if p.get('autoscaling') else 0}")
+        L.append(f"scaleway_pool_root_volume_bytes{{{lb}}} {p.get('root_volume_size', 0)}")
+
+    # ── Block Storage (dynamic — auto-discovers volume additions/removals) ──
+    L.append("# HELP scaleway_volume_size_bytes Volume size in bytes")
+    L.append("# TYPE scaleway_volume_size_bytes gauge")
+    L.append("# HELP scaleway_volume_attached Volume attachment state (1=in_use, 0=available)")
+    L.append("# TYPE scaleway_volume_attached gauge")
+    total_bytes = attached_bytes = detached_bytes = 0
+    vol_count = attached_count = detached_count = 0
+    for v in volumes.get("volumes", []):
+        vid = v.get("id", "?")[:36]
+        name = v.get("name", "?")[:60]
+        size = v.get("size", 0)
+        state = v.get("status", "unknown")
+        attached = 1 if state == "in_use" else 0
+        total_bytes += size; vol_count += 1
+        if attached:
+            attached_bytes += size; attached_count += 1
+        else:
+            detached_bytes += size; detached_count += 1
+        L.append(f'scaleway_volume_size_bytes{{id="{sanitize(vid)}",name="{sanitize(name)}",state="{sanitize(state)}"}} {size}')
+        L.append(f'scaleway_volume_attached{{id="{sanitize(vid)}",name="{sanitize(name)}"}} {attached}')
+    L.append("# HELP scaleway_volumes_total_bytes Total block storage bytes")
+    L.append("# TYPE scaleway_volumes_total_bytes gauge")
+    L.append(f"scaleway_volumes_total_bytes {total_bytes}")
+    L.append("# HELP scaleway_volumes_attached_bytes Attached volume bytes")
+    L.append("# TYPE scaleway_volumes_attached_bytes gauge")
+    L.append(f"scaleway_volumes_attached_bytes {attached_bytes}")
+    L.append("# HELP scaleway_volumes_detached_bytes Detached (wasted) volume bytes")
+    L.append("# TYPE scaleway_volumes_detached_bytes gauge")
+    L.append(f"scaleway_volumes_detached_bytes {detached_bytes}")
+    L.append("# HELP scaleway_volumes_count Volume counts")
+    L.append("# TYPE scaleway_volumes_count gauge")
+    L.append(f'scaleway_volumes_count{{state="total"}} {vol_count}')
+    L.append(f'scaleway_volumes_count{{state="attached"}} {attached_count}')
+    L.append(f'scaleway_volumes_count{{state="detached"}} {detached_count}')
+
+    # ── IPs (dynamic) ──
+    L.append("# HELP scaleway_ips_total Total allocated IPs")
+    L.append("# TYPE scaleway_ips_total gauge")
+    L.append(f'scaleway_ips_total {len(ips.get("ips", []))}')
+
+    # ── Push to pushgateway ──
+    metrics = "\n".join(L) + "\n"
+    count = sum(1 for line in L if line.startswith("scaleway_"))
+    print(f"Generated {count} metrics ({len(metrics)} bytes)", file=sys.stderr)
+
+    # Validate: no empty lines between metrics (pushgateway is strict)
+    metrics = re.sub(r'\n{2,}', '\n', metrics)
+
+    url = f"{PUSHGATEWAY}/metrics/job/scaleway_billing"
+    req = urllib.request.Request(url, data=metrics.encode(), method="PUT")
+    req.add_header("Content-Type", "text/plain; version=0.0.4")
+    req.add_header("Connection", "close")
+    try:
+        resp = urllib.request.urlopen(req, timeout=30)
+        print(f"OK: pushed {count} metrics (HTTP {resp.status})")
+    except Exception as e:
+        print(f"WARN: PUT failed ({e}), trying POST...", file=sys.stderr)
+        req2 = urllib.request.Request(url, data=metrics.encode(), method="POST")
+        req2.add_header("Content-Type", "text/plain; version=0.0.4")
+        req2.add_header("Connection", "close")
+        try:
+            resp = urllib.request.urlopen(req2, timeout=30)
+            print(f"OK: pushed {count} metrics via POST (HTTP {resp.status})")
+        except Exception as e2:
+            print(f"ERROR: push failed: PUT={e}, POST={e2}", file=sys.stderr)
+            sys.exit(1)
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: scaleway-billing-scraper
+  namespace: foxhunt
+spec:
+  schedule: "0 * * * *"
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 120
+      template:
+        metadata:
+          labels:
+            app: scaleway-billing
+        spec:
+          nodeSelector:
+            k8s.scaleway.com/pool-name: platform
+          restartPolicy: OnFailure
+          containers:
+            - name: scraper
+              image: python:3.12-alpine
+              command: ["python3", "/scripts/scrape.py"]
+              envFrom:
+                - secretRef:
+                    name: scaleway-billing
+              volumeMounts:
+                - name: script
+                  mountPath: /scripts
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 32Mi
+                limits:
+                  cpu: 100m
+                  memory: 64Mi
+          volumes:
+            - name: script
+              configMap:
+                name: scaleway-billing-script
+                items:
+                  - key: scrape.py
+                    path: scrape.py
+                defaultMode: 0755

infra/k8s/monitoring/service-monitors.yaml

@@ -0,0 +1,43 @@
+# ServiceMonitors for Foxhunt services and GPU metrics
+# Applied separately from the Helm chart (additionalServiceMonitors not rendered by chart)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: foxhunt-services
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/part-of: foxhunt
+  namespaceSelector:
+    matchNames: [foxhunt]
+  endpoints:
+    - port: metrics
+      interval: 15s
+      # Override job label to "foxhunt-services" (matching old standalone config)
+      # so all existing dashboard queries continue to work
+      relabelings:
+        - targetLabel: job
+          replacement: foxhunt-services
+---
+# DCGM Exporter (GPU metrics from nvidia-gpu-operator DaemonSet)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: dcgm-exporter
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  selector:
+    matchExpressions:
+      - key: app.kubernetes.io/name
+        operator: In
+        values: [dcgm-exporter]
+  namespaceSelector:
+    matchNames: [kube-system, monitoring, foxhunt]
+  endpoints:
+    - port: metrics
+      interval: 15s

infra/k8s/monitoring/tempo.yaml

@@ -0,0 +1,139 @@
+# Grafana Tempo — lightweight distributed tracing (single-binary mode)
+# Receives OTLP traces from services, queryable via Grafana
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: tempo-config
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: tempo
+data:
+  tempo.yaml: |
+    server:
+      http_listen_port: 3200
+    distributor:
+      receivers:
+        otlp:
+          protocols:
+            grpc:
+              endpoint: "0.0.0.0:4317"
+            http:
+              endpoint: "0.0.0.0:4318"
+    storage:
+      trace:
+        backend: local
+        local:
+          path: /var/tempo/traces
+        wal:
+          path: /var/tempo/wal
+    compactor:
+      compaction:
+        block_retention: 168h  # 7 days
+    metrics_generator:
+      registry:
+        external_labels:
+          source: tempo
+          cluster: foxhunt
+      storage:
+        path: /var/tempo/generator/wal
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: tempo
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: tempo
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: tempo
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: tempo
+    spec:
+      nodeSelector:
+        k8s.scaleway.com/pool-name: platform
+      securityContext:
+        fsGroup: 10001
+        runAsUser: 10001
+        runAsNonRoot: true
+      containers:
+        - name: tempo
+          image: grafana/tempo:2.7.1
+          args: ["-config.file=/etc/tempo/tempo.yaml"]
+          ports:
+            - containerPort: 3200
+              name: http
+            - containerPort: 4317
+              name: otlp-grpc
+            - containerPort: 4318
+              name: otlp-http
+          volumeMounts:
+            - name: config
+              mountPath: /etc/tempo
+            - name: data
+              mountPath: /var/tempo
+          resources:
+            requests:
+              cpu: 100m
+              memory: 512Mi
+            limits:
+              cpu: 500m
+              memory: 2Gi
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: 3200
+            initialDelaySeconds: 10
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /ready
+              port: 3200
+            initialDelaySeconds: 30
+            periodSeconds: 15
+      volumes:
+        - name: config
+          configMap:
+            name: tempo-config
+        - name: data
+          persistentVolumeClaim:
+            claimName: tempo-data
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: tempo-data
+  namespace: foxhunt
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: scw-bssd
+  resources:
+    requests:
+      storage: 10Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: tempo
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: tempo
+spec:
+  selector:
+    app.kubernetes.io/name: tempo
+  ports:
+    - port: 3200
+      targetPort: 3200
+      name: http
+    - port: 4317
+      targetPort: 4317
+      name: otlp-grpc
+    - port: 4318
+      targetPort: 4318
+      name: otlp-http

infra/k8s/namespace.yaml

@@ -0,0 +1,7 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+    app.kubernetes.io/managed-by: kubectl

infra/k8s/network-policies/alertmanager.yaml

@@ -0,0 +1,64 @@
+# AlertManager network policy
+# Ingress: Prometheus (alert delivery), Grafana (alert UI queries)
+# Egress: Mattermost webhooks (cluster-internal), DNS
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: alertmanager
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: alertmanager
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    # Prometheus pushes alerts and scrapes metrics (9093 API, 8080 config-reloader)
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: prometheus
+      ports:
+        - protocol: TCP
+          port: 9093
+        - protocol: TCP
+          port: 8080
+    # Grafana queries AlertManager API for alert state/silences
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: grafana
+      ports:
+        - protocol: TCP
+          port: 9093
+  egress:
+    # DNS resolution
+    - to:
+        - namespaceSelector: {}
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+    # Mattermost incoming webhook (cluster-internal)
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: mattermost
+      ports:
+        - protocol: TCP
+          port: 8065
+    # External HTTPS (PagerDuty, email, future integrations)
+    - to:
+        - ipBlock:
+            cidr: 0.0.0.0/0
+            except:
+              - 10.0.0.0/8
+              - 172.16.0.0/12
+              - 192.168.0.0/16
+      ports:
+        - protocol: TCP
+          port: 443

infra/k8s/network-policies/allow-dns.yaml

@@ -0,0 +1,26 @@
+# Allow all foxhunt + CI pods to reach CoreDNS (kube-system) for name resolution.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-dns
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchExpressions:
+      - key: app.kubernetes.io/part-of
+        operator: In
+        values: [foxhunt, foxhunt-ci]
+  policyTypes:
+    - Egress
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53

infra/k8s/network-policies/allow-monitoring.yaml

@@ -0,0 +1,76 @@
+# Allow Prometheus to scrape metrics ports on all foxhunt + CI pods.
+# Also allow all foxhunt + CI pods to push traces to Tempo.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-monitoring-scrape
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  podSelector:
+    matchExpressions:
+      - key: app.kubernetes.io/part-of
+        operator: In
+        values: ["foxhunt", "foxhunt-ci"]
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    - from:
+        # External monitoring namespace (if any)
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: monitoring
+        # GitLab Prometheus in foxhunt namespace
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: prometheus
+      ports:
+        - protocol: TCP
+          port: 8080
+        - protocol: TCP
+          port: 9003
+        - protocol: TCP
+          port: 9091
+        - protocol: TCP
+          port: 9092
+        - protocol: TCP
+          port: 9093
+        - protocol: TCP
+          port: 9094
+        - protocol: TCP
+          port: 9095
+        - protocol: TCP
+          port: 9096
+        - protocol: TCP
+          port: 9097
+        - protocol: TCP
+          port: 9098
+  egress:
+    # Tempo OTLP gRPC
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: foxhunt
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tempo
+      ports:
+        - protocol: TCP
+          port: 4317
+    # Kubernetes API (for kube-state-metrics list/watch)
+    - to:
+        - ipBlock:
+            cidr: 10.32.0.0/16
+      ports:
+        - protocol: TCP
+          port: 443
+        - protocol: TCP
+          port: 6443
+    - to:
+        - ipBlock:
+            cidr: 172.16.0.0/16
+      ports:
+        - protocol: TCP
+          port: 6443

infra/k8s/network-policies/api.yaml

@@ -0,0 +1,110 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: api
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: api
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    # Tailscale ingress proxy
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: tailscale-gitlab-proxy
+      ports:
+        - protocol: TCP
+          port: 50051
+  egress:
+    # Postgres
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: postgres
+      ports:
+        - protocol: TCP
+          port: 5432
+    # Redis
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: redis
+      ports:
+        - protocol: TCP
+          port: 6379
+    # Downstream gRPC services
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: trading-service
+      ports:
+        - protocol: TCP
+          port: 50051
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: backtesting-service
+      ports:
+        - protocol: TCP
+          port: 50053
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: ml-training-service
+      ports:
+        - protocol: TCP
+          port: 50053
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: trading-agent-service
+      ports:
+        - protocol: TCP
+          port: 50055
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: broker-gateway
+      ports:
+        - protocol: TCP
+          port: 50056
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: data-acquisition-service
+      ports:
+        - protocol: TCP
+          port: 50057
+    # Prometheus (metric scraping from monitoring handler)
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: prometheus
+      ports:
+        - protocol: TCP
+          port: 9090
+    # Minio (initContainer binary fetch)
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+      ports:
+        - protocol: TCP
+          port: 9000
+    # Kubernetes API (pod listing for Pods cockpit)
+    - to:
+        - ipBlock:
+            cidr: 10.32.0.0/12
+      ports:
+        - protocol: TCP
+          port: 443
+    - to:
+        - ipBlock:
+            cidr: 172.16.0.0/16
+      ports:
+        - protocol: TCP
+          port: 6443

infra/k8s/network-policies/backtesting-service.yaml

@@ -0,0 +1,42 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: backtesting-service
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: backtesting-service
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: api
+      ports:
+        - protocol: TCP
+          port: 50053
+  egress:
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: postgres
+      ports:
+        - protocol: TCP
+          port: 5432
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: redis
+      ports:
+        - protocol: TCP
+          port: 6379
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+      ports:
+        - protocol: TCP
+          port: 9000

infra/k8s/network-policies/broker-gateway.yaml

@@ -0,0 +1,52 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: broker-gateway
+  namespace: foxhunt
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: broker-gateway
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: api
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: trading-service
+      ports:
+        - protocol: TCP
+          port: 50056
+  egress:
+    - to:
+        - podSelector:
+            matchLabels:
+              app: ib-gateway
+      ports:
+        - protocol: TCP
+          port: 4002
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: postgres
+      ports:
+        - protocol: TCP
+          port: 5432
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: redis
+      ports:
+        - protocol: TCP
+          port: 6379
+    - to:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: minio
+      ports:
+        - protocol: TCP
+          port: 9000