docs(ci): fxhnt CI/CD pipeline design spec (git-sync-first, image-on-demand)
Reanimate the retired cockpit-build-deploy WorkflowTemplate with the mode split (build vs deploy) driven by a path-filter sensor: dep/Dockerfile changes -> Kaniko rebuild, code-only changes -> fast rollout (git-sync re-pulls src). Reuses the existing bizworx Argo Events/Workflows/Kaniko stack (one Gitea, verified). No ArgoCD app, no lock file. Deploy-step route (tailscale scale-0->1 vs rollout-restart) flagged to verify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
105
docs/superpowers/specs/2026-07-21-fxhnt-cicd-pipeline-design.md
Normal file
105
docs/superpowers/specs/2026-07-21-fxhnt-cicd-pipeline-design.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# fxhnt CI/CD Pipeline on bizworx — Design Spec (2026-07-21)
|
||||
|
||||
**Status:** design approved (approach + build-trigger decided with operator). Implementation plan follows.
|
||||
|
||||
## Goal
|
||||
|
||||
Restore automated build+deploy for fxhnt on the bizworx Kapsule so a push to `master` on
|
||||
`gitadmin/fxhnt` (Gitea) lands live without a manual `kubectl rollout restart` — the exact gap that left the
|
||||
cockpit on 2.5-day-old code (the CBU0/CSBGU0 fix). **Git-sync-first, image-on-demand:** code changes trigger a
|
||||
fast rollout (git-sync re-pulls src); only dependency/Dockerfile changes trigger a Kaniko image rebuild.
|
||||
|
||||
## Why this shape
|
||||
|
||||
The operator's insight: **code changes often, the image changes rarely.** git-sync already pulls fresh `src/`
|
||||
on every (re)start, so a code-only push needs only a rollout — not a 5-minute Kaniko build. A rebuild is needed
|
||||
only when `pyproject.toml` / `fxhnt.Dockerfile` / lock change. This is exactly the `mode=build` vs `mode=deploy`
|
||||
split the retired `infra/argo/cockpit-build-deploy.yaml` already encoded — we reanimate and fix it, rather than
|
||||
adopt the generic `build-image` template (which lacks the split and fxhnt's deploy specifics).
|
||||
|
||||
## Verified current state (all components already exist on bizworx)
|
||||
|
||||
- **One Gitea** (ns `gitea`): `git.fxhnt.ai` and `gitea-http.gitea.svc` are the SAME instance; it holds
|
||||
`gitadmin/fxhnt` (code) alongside `bizworx-ops/infrastructure` (the msp GitOps monorepo). No second Gitea.
|
||||
- **Argo Events** (`gitea-webhook` eventsource, `/push`, repo-agnostic) + **Argo Workflows** (Kaniko) — already
|
||||
driving `build-msp-*` via path-filtered sensors. Reusable.
|
||||
- **Registry:** fxhnt's image is `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit` (Scaleway) — DIFFERENT from msp's
|
||||
`registry.bizworx.nl/bizworx-ops/`. Secret `scw-registry` (ns foxhunt) holds the push creds.
|
||||
- **Dockerfile EXISTS:** `infra/docker/fxhnt.Dockerfile` (python:3.11-slim, `fxhnt serve` :8080). Reusable as-is.
|
||||
- **SAs exist:** `argo-workflow` (ns argo), `argo-events-sa` (ns argo-events); `argo-git-ssh-key` (ns foxhunt).
|
||||
- **What's broken in the old pipeline:** (1) `nodeSelector: ci-compile-cpu` — that pool is DECOMMISSIONED
|
||||
(only `large` + `sfs` pools exist now) → build/deploy pods won't schedule; (2) the Sensor that triggered it
|
||||
is gone; (3) no webhook on `gitadmin/fxhnt` → the eventsource never hears fxhnt pushes.
|
||||
- **No fxhnt ArgoCD app** — fxhnt manifests are applied by the workflow's `kubectl apply`, not GitOps-synced.
|
||||
We keep it that way (see Non-goals).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
git push → gitadmin/fxhnt (master)
|
||||
│ Gitea webhook ──────────────► argo-events `gitea-webhook` eventsource (/push)
|
||||
▼
|
||||
Sensor `build-fxhnt` — filters: ref==refs/heads/master, pusher != argocd-image-updater
|
||||
│ path-filter on the pushed commits' modified files decides the mode:
|
||||
│ • touched pyproject.toml OR infra/docker/fxhnt.Dockerfile → param mode=build
|
||||
│ • otherwise (src/, infra/k8s/, config) → param mode=deploy
|
||||
│ (NB: the repo has NO lock file — the Dockerfile `pip install`s straight from pyproject.toml,
|
||||
│ so pyproject.toml + the Dockerfile are the ONLY dependency sources that require a rebuild.)
|
||||
▼
|
||||
Argo Workflow (WorkflowTemplate `fxhnt-cockpit`, reanimated) — DAG:
|
||||
├─ mode=build: git-clone(sha) → Kaniko build → rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:<sha7>+:latest
|
||||
└─ (build skipped when mode=deploy)
|
||||
→ deploy step: kubectl apply manifests + refresh the daemons (git-sync re-pulls src)
|
||||
```
|
||||
|
||||
## Components to build/fix
|
||||
|
||||
### 1. WorkflowTemplate `fxhnt-cockpit` (reanimate `infra/argo/cockpit-build-deploy.yaml`)
|
||||
- Fix `nodeSelector` `ci-compile-cpu` → `large` (both build + deploy templates).
|
||||
- Keep the `mode` DAG: `build` (Kaniko rebuild + deploy) vs `deploy` (skip build, deploy only).
|
||||
- Keep the Kaniko dest `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit` but ADD an immutable `:<sha7>` tag next to
|
||||
`:latest` (reproducibility — image == commit; matches the msp pattern). Cache stays on.
|
||||
- Clone via the in-cluster Gitea (`ssh://…gitea…/gitadmin/fxhnt.git`) using `argo-git-ssh-key`; push via
|
||||
`scw-registry`. Verify both secrets resolve from the namespace the workflow runs in.
|
||||
|
||||
### 2. Sensor `build-fxhnt` (new, model on `build-msp-portal`)
|
||||
- Dependency: `gitea-webhook` push, filter `body.ref == refs/heads/master` AND
|
||||
`body.pusher.username != argocd-image-updater` (loop-guard; harmless here since no image-updater writes fxhnt,
|
||||
but kept for parity/safety).
|
||||
- **Mode decision via path-filter on `body.commits.#.modified`:** if any modified path matches
|
||||
`pyproject.toml` | `infra/docker/fxhnt.Dockerfile` → submit workflow with `mode=build`; else `mode=deploy`.
|
||||
(No lock file exists, so those two paths are the complete rebuild-trigger set.) Mechanism: two triggers with
|
||||
complementary data-filters (one matching the dep-paths → build, one matching their negation → deploy), or one
|
||||
trigger whose param is computed — the plan picks whichever Argo Events supports cleanly.
|
||||
- Inject `commit-sha` from `body.after`.
|
||||
|
||||
### 3. Gitea webhook on `gitadmin/fxhnt`
|
||||
- Add a repo (or org) webhook POSTing push events to the `gitea-webhook` eventsource endpoint (same target the
|
||||
infrastructure repo already uses). This is the one genuinely-missing wire.
|
||||
|
||||
### 4. The deploy step — VERIFY, don't assume (open design point)
|
||||
The old deploy used **`scale 0→1` for fxhnt-dashboard, NOT `rollout restart`**, because the tailscale sidecar
|
||||
served `dashboard.fxhnt.ai` and a rolling restart overlapped two tailscale sessions → external proxy read
|
||||
"down". BUT: the dashboard now also has a `fxhnt-ingress`, and a manual `rollout restart` on 2026-07-21
|
||||
succeeded cleanly. **The plan must verify which external route is live** before choosing:
|
||||
- If `dashboard.fxhnt.ai` still resolves through the tailscale sidecar → keep `scale 0→1` (the race is real).
|
||||
- If it now resolves through `fxhnt-ingress` (tailscale sidecar vestigial) → a plain `rollout restart` is
|
||||
simpler and correct; drop the scale-dance.
|
||||
Dagster has no external-proxy race → `rollout restart deploy/dagster` either way.
|
||||
|
||||
## Non-goals (YAGNI)
|
||||
|
||||
- **No ArgoCD GitOps app for fxhnt.** That would force fxhnt's manifests into the bizworx monorepo. git-sync +
|
||||
the mode-split deliver auto-deploy while keeping manifests in the fxhnt repo. A declarative ArgoCD migration
|
||||
is a separate, later decision.
|
||||
- **No image-updater for fxhnt.** The sensor writes the `:sha7`+`:latest` tags; the deploy step points the
|
||||
daemons at the fresh image. No git-write-back loop needed.
|
||||
- **No change to the execution CronJobs** (ucits/bybit/ibkr rebalancers) — they git-sync per run already.
|
||||
|
||||
## Success criteria
|
||||
|
||||
1. A code-only push to `master` → cockpit reflects it within ~1–2 min WITHOUT a manual restart, NO Kaniko build.
|
||||
2. A `pyproject.toml`/Dockerfile push → a Kaniko rebuild produces `fxhnt-cockpit:<sha7>`+`:latest`, then deploys.
|
||||
3. The dashboard external route (`dashboard.fxhnt.ai`) stays UP across a deploy (no stale-proxy "down").
|
||||
4. No build-loop, no false trigger on unrelated pushes.
|
||||
5. Real-money safety unaffected — the pipeline only builds/deploys the cockpit+dagster daemons, never arms exec.
|
||||
Reference in New Issue
Block a user