docs(ci): fxhnt CI/CD implementation plan (5 tasks, in-workflow mode decision)
Verified the Argo Events CRD has an open schema -> trigger-level path-filter negation can't be validated. Redesigned the mode-split to live in the workflow (git diff --name-only) with a single msp-style sensor (lowest risk). 5 tasks: WorkflowTemplate build side, deploy step (verify tailscale route), sensor + decide-mode, apply+webhook, e2e verify both modes + retire old file. Every cluster apply gated behind server dry-run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
269
docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md
Normal file
269
docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# fxhnt CI/CD Pipeline — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this
|
||||
> plan task-by-task. Steps use checkbox tracking. This plan writes/edits YAML manifests and applies them to a
|
||||
> LIVE cluster — every apply is gated behind a dry-run + explicit verify; a broken sensor/workflow must never
|
||||
> silently swallow a push or false-deploy.
|
||||
|
||||
**Goal:** A push to `master` on `gitadmin/fxhnt` auto-builds (only when deps/Dockerfile change) and auto-deploys
|
||||
the fxhnt cockpit+dagster daemons — no manual `kubectl rollout restart`.
|
||||
|
||||
**Architecture:** git-sync-first, image-on-demand. A minimal Sensor in `argo-events` hears the shared Gitea
|
||||
webhook, path-filters the push to pick `mode=build` (Kaniko rebuild) vs `mode=deploy` (rollout only), and submits
|
||||
the fxhnt-owned `fxhnt-cockpit` WorkflowTemplate into `foxhunt`.
|
||||
|
||||
**Tech Stack:** Argo Events v1.9.10 (Sensor), Argo Workflows (Kaniko build + kubectl deploy), Gitea webhook,
|
||||
Scaleway registry, K8s (bizworx Kapsule).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **Project boundary:** fxhnt-owned objects (WorkflowTemplate, build/deploy pods, secrets, RBAC) live in
|
||||
`foxhunt`. ONLY the Sensor lives in `argo-events` (unavoidable). No fxhnt secrets in `argo`/`argo-events`; no
|
||||
cross-namespace RBAC.
|
||||
- **Node pool:** all build/deploy pods target `nodeSelector: k8s.scaleway.com/pool-name: large` (the
|
||||
`ci-compile-cpu` pool is DECOMMISSIONED — only `large`/`sfs` exist). A pod pinned to `ci-compile-cpu` never
|
||||
schedules.
|
||||
- **Registry:** `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit`, tagged `:<sha7>` AND `:latest`. Secret
|
||||
`scw-registry` (already in `foxhunt`).
|
||||
- **Real-money safety:** the pipeline only builds/deploys the cockpit+dagster DAEMONS. It must NEVER touch the
|
||||
execution CronJobs (ucits/bybit/ibkr rebalancers) or arm any executor.
|
||||
- **No silent failures:** a Sensor that can't match, or a Workflow that errors, must surface loudly (a failed
|
||||
push = a visible failed Workflow, never a swallowed no-op). Every manifest is `kubectl apply --dry-run=server`
|
||||
validated before a real apply.
|
||||
- **Reproducibility:** the deploy step annotates deployed Deployments with `fxhnt.io/deployed-sha=<sha7>`.
|
||||
|
||||
## File Structure
|
||||
|
||||
- `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` — the reanimated WorkflowTemplate (ns foxhunt). Supersedes
|
||||
the old `infra/argo/cockpit-build-deploy.yaml` (which is renamed/removed in the final task).
|
||||
- `infra/argo/fxhnt-build-sensor.yaml` — the new Sensor (ns argo-events).
|
||||
- Gitea webhook on `gitadmin/fxhnt` — created via Gitea API/UI (not a repo file; documented in the plan).
|
||||
|
||||
---
|
||||
|
||||
## Task 1 — Reanimate + fix the WorkflowTemplate (build side)
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (start from `infra/argo/cockpit-build-deploy.yaml`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: a `WorkflowTemplate` named `fxhnt-cockpit` in ns `foxhunt` with params `commit-sha` (default HEAD)
|
||||
and `mode` (default `build`; values `build`|`deploy`). The Sensor (Task 3) submits Workflows referencing it.
|
||||
|
||||
- [ ] **Step 1: Copy the old template as the new file, fix the node pool**
|
||||
|
||||
Copy `infra/argo/cockpit-build-deploy.yaml` → `infra/argo/fxhnt-cockpit-workflowtemplate.yaml`. In BOTH the
|
||||
`kaniko-build` and `kubectl-deploy` templates, change every
|
||||
`nodeSelector: { k8s.scaleway.com/pool-name: ci-compile-cpu, ... }` to
|
||||
`nodeSelector: { k8s.scaleway.com/pool-name: large, topology.kubernetes.io/zone: fr-par-2 }`.
|
||||
Grep to confirm no `ci-compile-cpu` remains: `grep -n ci-compile-cpu infra/argo/fxhnt-cockpit-workflowtemplate.yaml` → no hits.
|
||||
|
||||
- [ ] **Step 2: Add the immutable `:<sha7>` tag to the Kaniko destination**
|
||||
|
||||
In the `kaniko-build` container args, alongside the existing
|
||||
`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`, ADD a second line:
|
||||
`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:{{=sprig.trunc(7, workflow.parameters.commit-sha)}}`
|
||||
(the `sprig.trunc` expression matches how `build-image` truncates the sha). Keep `--cache=true` and the
|
||||
`--cache-repo`.
|
||||
|
||||
- [ ] **Step 3: Server-side dry-run validate the template**
|
||||
|
||||
Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt`
|
||||
Expected: `workflowtemplate.argoproj.io/fxhnt-cockpit created (server dry run)` with no schema errors.
|
||||
(If it errors on an unknown field, the Argo CRD version differs — STOP and report; do not force-apply.)
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml
|
||||
git commit -m "ci(fxhnt): reanimate cockpit WorkflowTemplate (node pool large, sha7 tag)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2 — Fix the deploy step (verify the dashboard external route first)
|
||||
|
||||
**Files:**
|
||||
- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (the `kubectl-deploy` template's script)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the WorkflowTemplate from Task 1.
|
||||
- Produces: a deploy step that refreshes the daemons correctly for whichever external route is live.
|
||||
|
||||
- [ ] **Step 1: Determine the live dashboard external route (decides restart vs scale-0→1)**
|
||||
|
||||
Run these and read the result:
|
||||
```bash
|
||||
kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
|
||||
kubectl get ingress -n foxhunt 2>/dev/null
|
||||
kubectl -n foxhunt describe svc fxhnt-ingress 2>/dev/null | grep -iE 'endpoints|selector'
|
||||
```
|
||||
Decide:
|
||||
- If `dashboard.fxhnt.ai` is served THROUGH the tailscale sidecar (sidecar present AND no ingress route to the
|
||||
dashboard) → KEEP the old `scale 0→1` dance (the tailscale session race is real).
|
||||
- If it is served through `fxhnt-ingress` (an ingress/route points at the dashboard svc; the tailscale sidecar
|
||||
is vestigial) → REPLACE the scale-dance with a plain `kubectl -n foxhunt rollout restart deploy/fxhnt-dashboard`.
|
||||
|
||||
Record the decision in the plan's progress ledger with the evidence. If ambiguous, KEEP `scale 0→1` (the
|
||||
conservative choice that cannot cause the stale-proxy "down").
|
||||
|
||||
- [ ] **Step 2: Apply the deploy-step decision + add the deployed-sha annotation**
|
||||
|
||||
Edit the `kubectl-deploy` script. Regardless of the restart mechanism chosen, add BEFORE the rollout:
|
||||
```
|
||||
kubectl -n foxhunt annotate deploy/fxhnt-dashboard deploy/dagster \
|
||||
fxhnt.io/deployed-sha="$(cd /workspace/src && git rev-parse --short HEAD)" --overwrite
|
||||
```
|
||||
Keep the existing `kubectl apply -f` of the three manifests (fxhnt-dashboard.yaml, dagster.yaml,
|
||||
network-policies/fxhnt-cockpit.yaml) and the `rollout status` waits. Keep `rollout restart deploy/dagster`
|
||||
(dagster has no proxy race).
|
||||
|
||||
- [ ] **Step 3: Dry-run validate again**
|
||||
|
||||
Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt`
|
||||
Expected: server dry-run OK.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml
|
||||
git commit -m "ci(fxhnt): deploy step — <restart-choice> + deployed-sha annotation"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3 — The Sensor (simple, single-trigger) — mode decided IN the workflow
|
||||
|
||||
**Design note (verified 2026-07-21):** the Argo Events Sensor CRD here has an OPEN schema (no strict property
|
||||
validation — `kubectl explain sensor.spec.triggers.template.conditions` errors, and the only `conditions:` in
|
||||
live sensors are in `status`, not trigger-level). Rather than gamble on a fragile sensor-side path-negation
|
||||
filter to pick build-vs-deploy, we keep the SENSOR SIMPLE (one trigger, exactly like `build-msp-portal`) and
|
||||
move the build-vs-deploy decision INTO the workflow, where a robust `git diff --name-only` between the previous
|
||||
and pushed commit is trivial and testable. This is the more robust design, not a fallback.
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/argo/fxhnt-build-sensor.yaml` (ns argo-events)
|
||||
- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (add a `decide-mode` step that git-diffs)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the `gitea-webhook` eventsource (shared), the `fxhnt-cockpit` WorkflowTemplate (Task 1/2).
|
||||
- Produces: a `Sensor` `build-fxhnt` (argo-events) that submits ONE Workflow per master push; the workflow
|
||||
self-selects build vs deploy.
|
||||
|
||||
- [ ] **Step 1: Write the single-trigger Sensor (copy build-msp-portal structure exactly)**
|
||||
|
||||
`kubectl get sensor build-msp-portal -n argo-events -o yaml` and mirror it. ONE dependency `push-to-master`:
|
||||
- `body.ref` == `refs/heads/master`
|
||||
- `body.pusher.username` != `argocd-image-updater`
|
||||
(NO `body.commits.#.modified` filter — the mode decision moved to the workflow.)
|
||||
ONE trigger `submit`: submits a Workflow `generateName: build-fxhnt-`, `namespace: foxhunt`,
|
||||
`workflowTemplateRef: fxhnt-cockpit`, injecting `commit-sha`=`body.after`. Do NOT set `mode` (the workflow
|
||||
computes it). This is structurally identical to the proven msp sensors → lowest risk.
|
||||
|
||||
- [ ] **Step 2: Add a `decide-mode` step to the WorkflowTemplate**
|
||||
|
||||
In `fxhnt-cockpit-workflowtemplate.yaml`, change the DAG so the FIRST step (after git-clone) is `decide-mode`:
|
||||
a small `alpine/git` script that runs, in the cloned repo at `commit-sha`:
|
||||
```
|
||||
CHANGED=$(git diff --name-only "${SHA}~1" "${SHA}" 2>/dev/null || echo "ALL")
|
||||
if echo "$CHANGED" | grep -qE '^pyproject\.toml$|^infra/docker/fxhnt\.Dockerfile$'; then
|
||||
echo build; else echo deploy; fi
|
||||
```
|
||||
Output the result (`build`/`deploy`) as an Argo output parameter `mode`. Then gate the `kaniko-build` task on
|
||||
`{{tasks.decide-mode.outputs.parameters.mode}} == build`, and `kubectl-deploy` depends on
|
||||
`build.Succeeded || build.Skipped` (unchanged). Edge case: if `${SHA}~1` doesn't exist (first commit / shallow),
|
||||
default to `build` (safe — a rebuild is never wrong, only slower). The explicit `mode` PARAM stays on the
|
||||
template (default `auto`) so a human can still force `mode=build`/`mode=deploy` on a manual `argo submit`;
|
||||
`decide-mode` only runs when `mode=auto`.
|
||||
|
||||
- [ ] **Step 3: Dry-run validate both**
|
||||
|
||||
```bash
|
||||
kubectl apply -f infra/argo/fxhnt-build-sensor.yaml --dry-run=server -n argo-events
|
||||
kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt
|
||||
```
|
||||
Expected: both server-dry-run OK.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/argo/fxhnt-build-sensor.yaml infra/argo/fxhnt-cockpit-workflowtemplate.yaml
|
||||
git commit -m "ci(fxhnt): single-trigger sensor + in-workflow git-diff mode decision"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4 — Apply to the cluster + wire the Gitea webhook
|
||||
|
||||
**Files:** none new — this task applies the manifests and creates the webhook.
|
||||
|
||||
- [ ] **Step 1: Apply the WorkflowTemplate + Sensor for real**
|
||||
|
||||
```bash
|
||||
kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml -n foxhunt
|
||||
kubectl apply -f infra/argo/fxhnt-build-sensor.yaml -n argo-events
|
||||
kubectl get workflowtemplate fxhnt-cockpit -n foxhunt
|
||||
kubectl get sensor build-fxhnt -n argo-events
|
||||
```
|
||||
Expected: both present. The sensor pod (`kubectl get pods -n argo-events | grep build-fxhnt`) reaches Running.
|
||||
|
||||
- [ ] **Step 2: Create the Gitea webhook on `gitadmin/fxhnt`**
|
||||
|
||||
The `gitea-webhook` eventsource endpoint is the same target the `bizworx-ops/infrastructure` repo already POSTs
|
||||
to. Find it: `kubectl get eventsource gitea-webhook -n argo-events -o yaml` (the webhook `port`/`endpoint`) and
|
||||
how the existing infra-repo webhook is configured (inspect via Gitea admin UI or API on an existing repo).
|
||||
Create the equivalent webhook on `gitadmin/fxhnt`: push events, active, POST JSON to the eventsource URL. Record
|
||||
the exact URL used in the ledger. Do NOT paste any webhook secret into the transcript.
|
||||
|
||||
- [ ] **Step 3: Verify the webhook reaches the sensor (dry, safe probe)**
|
||||
|
||||
Trigger a harmless test push (or use Gitea's "Test Delivery" webhook button) and confirm the sensor received it:
|
||||
```bash
|
||||
kubectl logs -n argo-events -l sensor-name=build-fxhnt --tail=50 | grep -iE 'triggered|push-to-master|dispatch'
|
||||
```
|
||||
Expected: the sensor logs the received push. If Gitea's test-delivery uses a non-master ref, the sensor
|
||||
correctly does NOT fire — confirm it's reachable via the eventsource logs instead.
|
||||
|
||||
---
|
||||
|
||||
## Task 5 — End-to-end verification (both modes) + retire the old file
|
||||
|
||||
**Files:**
|
||||
- Remove: `infra/argo/cockpit-build-deploy.yaml` (superseded)
|
||||
|
||||
- [ ] **Step 1: Verify mode=deploy end-to-end (code-only push)**
|
||||
|
||||
Make a trivial code-only commit on `master` (e.g. touch a comment in `src/`), push, and watch:
|
||||
```bash
|
||||
kubectl get wf -n foxhunt --sort-by=.metadata.creationTimestamp | tail -3
|
||||
```
|
||||
Expected: a `build-fxhnt-*` Workflow runs with `mode=deploy`, SKIPS the kaniko-build step, deploys, and the
|
||||
dashboard stays UP (check `dashboard.fxhnt.ai` reachable post-deploy, and
|
||||
`kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{.metadata.annotations.fxhnt\.io/deployed-sha}'`
|
||||
shows the new sha). Total time ~1-2 min, no Kaniko build.
|
||||
|
||||
- [ ] **Step 2: Verify mode=build end-to-end (dep push)**
|
||||
|
||||
Make a no-op-but-real change to `pyproject.toml` (e.g. reorder a dependency, or bump a harmless comment), push,
|
||||
and watch: the Workflow runs `mode=build`, Kaniko produces `fxhnt-cockpit:<sha7>`+`:latest`
|
||||
(`kubectl run` a crane/skopeo check OR verify the workflow's kaniko step succeeded), then deploys. Confirm the
|
||||
`:<sha7>` tag exists in the registry.
|
||||
|
||||
- [ ] **Step 3: Confirm no false trigger + real-money safety**
|
||||
|
||||
- Push a change touching ONLY `docs/` → confirm a `mode=deploy` Workflow runs (harmless) OR, if docs are
|
||||
excluded, no Workflow — either is fine, but it must NOT trigger a build.
|
||||
- Confirm NO execution CronJob was modified/triggered by any pipeline run:
|
||||
`kubectl get cronjob -n foxhunt` (ucits/bybit/ibkr rebalancers unchanged, bybit/ibkr still Suspend=True).
|
||||
|
||||
- [ ] **Step 4: Retire the old file + commit**
|
||||
|
||||
```bash
|
||||
git rm infra/argo/cockpit-build-deploy.yaml
|
||||
git commit -m "ci(fxhnt): retire old cockpit-build-deploy (superseded by workflowtemplate + sensor)"
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Final verification summary**
|
||||
|
||||
Confirm all Global Constraints hold: sensor in argo-events only, workflow+pods in foxhunt, node pool `large`,
|
||||
no cross-ns RBAC, no fxhnt secrets in platform ns, real-money CronJobs untouched. Record in the ledger.
|
||||
@@ -59,15 +59,15 @@ git push → gitadmin/fxhnt (master) [SHARED: Gitea]
|
||||
│ Gitea webhook ──────────────► argo-events `gitea-webhook` eventsource [SHARED: platform eventbus]
|
||||
▼
|
||||
Sensor `build-fxhnt` (ns argo-events — the ONE fxhnt object in a platform ns, kept minimal)
|
||||
│ filters: ref==refs/heads/master, pusher != argocd-image-updater
|
||||
│ path-filter on the pushed commits' modified files decides the mode:
|
||||
│ • touched pyproject.toml OR infra/docker/fxhnt.Dockerfile → param mode=build
|
||||
│ • otherwise (src/, infra/k8s/, config) → param mode=deploy
|
||||
│ (NB: the repo has NO lock file — the Dockerfile `pip install`s straight from pyproject.toml,
|
||||
│ so pyproject.toml + the Dockerfile are the ONLY dependency sources that require a rebuild.)
|
||||
▼ submits a Workflow into ns `foxhunt` ─────────────────────────────── [FXHNT-OWNED from here down]
|
||||
│ filters: ref==refs/heads/master, pusher != argocd-image-updater (SINGLE trigger, like the msp sensors)
|
||||
▼ submits ONE Workflow into ns `foxhunt`, injecting commit-sha ─────── [FXHNT-OWNED from here down]
|
||||
Argo Workflow (WorkflowTemplate `fxhnt-cockpit`, ns foxhunt, SA argo-workflow, node-pool `large`) — DAG:
|
||||
├─ mode=build: git-clone(sha) → Kaniko → rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:<sha7>+:latest
|
||||
├─ decide-mode: git-clone(sha) → `git diff --name-only sha~1 sha` →
|
||||
│ touched pyproject.toml OR infra/docker/fxhnt.Dockerfile → mode=build ; else mode=deploy
|
||||
│ (mode decided IN the workflow, not a fragile sensor filter — the CRD schema is open, so a
|
||||
│ sensor-side path-negation can't be validated; a git-diff step is robust + testable. No lock file
|
||||
│ exists → those two paths are the complete rebuild-trigger set.)
|
||||
├─ mode=build: Kaniko → rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:<sha7>+:latest
|
||||
└─ (build skipped when mode=deploy)
|
||||
→ deploy step (in foxhunt, no cross-ns RBAC): kubectl apply manifests + refresh daemons (git-sync re-pulls src)
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user