docs(ci): fxhnt CI/CD pipeline design spec (git-sync-first, image-on-demand)

Reanimate the retired cockpit-build-deploy WorkflowTemplate with the mode
split (build vs deploy) driven by a path-filter sensor: dep/Dockerfile
changes -> Kaniko rebuild, code-only changes -> fast rollout (git-sync
re-pulls src). Reuses the existing bizworx Argo Events/Workflows/Kaniko
stack (one Gitea, verified). No ArgoCD app, no lock file. Deploy-step
route (tailscale scale-0->1 vs rollout-restart) flagged to verify.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-21 09:47:43 +00:00
parent bf99ce2bd1
commit 5d15b0f801

View File

@@ -0,0 +1,105 @@
# fxhnt CI/CD Pipeline on bizworx — Design Spec (2026-07-21)
**Status:** design approved (approach + build-trigger decided with operator). Implementation plan follows.
## Goal
Restore automated build+deploy for fxhnt on the bizworx Kapsule so a push to `master` on
`gitadmin/fxhnt` (Gitea) lands live without a manual `kubectl rollout restart` — the exact gap that left the
cockpit on 2.5-day-old code (the CBU0/CSBGU0 fix). **Git-sync-first, image-on-demand:** code changes trigger a
fast rollout (git-sync re-pulls src); only dependency/Dockerfile changes trigger a Kaniko image rebuild.
## Why this shape
The operator's insight: **code changes often, the image changes rarely.** git-sync already pulls fresh `src/`
on every (re)start, so a code-only push needs only a rollout — not a 5-minute Kaniko build. A rebuild is needed
only when `pyproject.toml` / `fxhnt.Dockerfile` / lock change. This is exactly the `mode=build` vs `mode=deploy`
split the retired `infra/argo/cockpit-build-deploy.yaml` already encoded — we reanimate and fix it, rather than
adopt the generic `build-image` template (which lacks the split and fxhnt's deploy specifics).
## Verified current state (all components already exist on bizworx)
- **One Gitea** (ns `gitea`): `git.fxhnt.ai` and `gitea-http.gitea.svc` are the SAME instance; it holds
`gitadmin/fxhnt` (code) alongside `bizworx-ops/infrastructure` (the msp GitOps monorepo). No second Gitea.
- **Argo Events** (`gitea-webhook` eventsource, `/push`, repo-agnostic) + **Argo Workflows** (Kaniko) — already
driving `build-msp-*` via path-filtered sensors. Reusable.
- **Registry:** fxhnt's image is `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit` (Scaleway) — DIFFERENT from msp's
`registry.bizworx.nl/bizworx-ops/`. Secret `scw-registry` (ns foxhunt) holds the push creds.
- **Dockerfile EXISTS:** `infra/docker/fxhnt.Dockerfile` (python:3.11-slim, `fxhnt serve` :8080). Reusable as-is.
- **SAs exist:** `argo-workflow` (ns argo), `argo-events-sa` (ns argo-events); `argo-git-ssh-key` (ns foxhunt).
- **What's broken in the old pipeline:** (1) `nodeSelector: ci-compile-cpu` — that pool is DECOMMISSIONED
(only `large` + `sfs` pools exist now) → build/deploy pods won't schedule; (2) the Sensor that triggered it
is gone; (3) no webhook on `gitadmin/fxhnt` → the eventsource never hears fxhnt pushes.
- **No fxhnt ArgoCD app** — fxhnt manifests are applied by the workflow's `kubectl apply`, not GitOps-synced.
We keep it that way (see Non-goals).
## Architecture
```
git push → gitadmin/fxhnt (master)
│ Gitea webhook ──────────────► argo-events `gitea-webhook` eventsource (/push)
Sensor `build-fxhnt` — filters: ref==refs/heads/master, pusher != argocd-image-updater
│ path-filter on the pushed commits' modified files decides the mode:
│ • touched pyproject.toml OR infra/docker/fxhnt.Dockerfile → param mode=build
│ • otherwise (src/, infra/k8s/, config) → param mode=deploy
│ (NB: the repo has NO lock file — the Dockerfile `pip install`s straight from pyproject.toml,
│ so pyproject.toml + the Dockerfile are the ONLY dependency sources that require a rebuild.)
Argo Workflow (WorkflowTemplate `fxhnt-cockpit`, reanimated) — DAG:
├─ mode=build: git-clone(sha) → Kaniko build → rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:<sha7>+:latest
└─ (build skipped when mode=deploy)
→ deploy step: kubectl apply manifests + refresh the daemons (git-sync re-pulls src)
```
## Components to build/fix
### 1. WorkflowTemplate `fxhnt-cockpit` (reanimate `infra/argo/cockpit-build-deploy.yaml`)
- Fix `nodeSelector` `ci-compile-cpu``large` (both build + deploy templates).
- Keep the `mode` DAG: `build` (Kaniko rebuild + deploy) vs `deploy` (skip build, deploy only).
- Keep the Kaniko dest `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit` but ADD an immutable `:<sha7>` tag next to
`:latest` (reproducibility — image == commit; matches the msp pattern). Cache stays on.
- Clone via the in-cluster Gitea (`ssh://…gitea…/gitadmin/fxhnt.git`) using `argo-git-ssh-key`; push via
`scw-registry`. Verify both secrets resolve from the namespace the workflow runs in.
### 2. Sensor `build-fxhnt` (new, model on `build-msp-portal`)
- Dependency: `gitea-webhook` push, filter `body.ref == refs/heads/master` AND
`body.pusher.username != argocd-image-updater` (loop-guard; harmless here since no image-updater writes fxhnt,
but kept for parity/safety).
- **Mode decision via path-filter on `body.commits.#.modified`:** if any modified path matches
`pyproject.toml` | `infra/docker/fxhnt.Dockerfile` → submit workflow with `mode=build`; else `mode=deploy`.
(No lock file exists, so those two paths are the complete rebuild-trigger set.) Mechanism: two triggers with
complementary data-filters (one matching the dep-paths → build, one matching their negation → deploy), or one
trigger whose param is computed — the plan picks whichever Argo Events supports cleanly.
- Inject `commit-sha` from `body.after`.
### 3. Gitea webhook on `gitadmin/fxhnt`
- Add a repo (or org) webhook POSTing push events to the `gitea-webhook` eventsource endpoint (same target the
infrastructure repo already uses). This is the one genuinely-missing wire.
### 4. The deploy step — VERIFY, don't assume (open design point)
The old deploy used **`scale 0→1` for fxhnt-dashboard, NOT `rollout restart`**, because the tailscale sidecar
served `dashboard.fxhnt.ai` and a rolling restart overlapped two tailscale sessions → external proxy read
"down". BUT: the dashboard now also has a `fxhnt-ingress`, and a manual `rollout restart` on 2026-07-21
succeeded cleanly. **The plan must verify which external route is live** before choosing:
- If `dashboard.fxhnt.ai` still resolves through the tailscale sidecar → keep `scale 0→1` (the race is real).
- If it now resolves through `fxhnt-ingress` (tailscale sidecar vestigial) → a plain `rollout restart` is
simpler and correct; drop the scale-dance.
Dagster has no external-proxy race → `rollout restart deploy/dagster` either way.
## Non-goals (YAGNI)
- **No ArgoCD GitOps app for fxhnt.** That would force fxhnt's manifests into the bizworx monorepo. git-sync +
the mode-split deliver auto-deploy while keeping manifests in the fxhnt repo. A declarative ArgoCD migration
is a separate, later decision.
- **No image-updater for fxhnt.** The sensor writes the `:sha7`+`:latest` tags; the deploy step points the
daemons at the fresh image. No git-write-back loop needed.
- **No change to the execution CronJobs** (ucits/bybit/ibkr rebalancers) — they git-sync per run already.
## Success criteria
1. A code-only push to `master` → cockpit reflects it within ~12 min WITHOUT a manual restart, NO Kaniko build.
2. A `pyproject.toml`/Dockerfile push → a Kaniko rebuild produces `fxhnt-cockpit:<sha7>`+`:latest`, then deploys.
3. The dashboard external route (`dashboard.fxhnt.ai`) stays UP across a deploy (no stale-proxy "down").
4. No build-loop, no false trigger on unrelated pushes.
5. Real-money safety unaffected — the pipeline only builds/deploys the cockpit+dagster daemons, never arms exec.