diff --git a/docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md b/docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md new file mode 100644 index 0000000..908cec5 --- /dev/null +++ b/docs/superpowers/plans/2026-07-21-fxhnt-cicd-pipeline.md @@ -0,0 +1,269 @@ +# fxhnt CI/CD Pipeline — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this +> plan task-by-task. Steps use checkbox tracking. This plan writes/edits YAML manifests and applies them to a +> LIVE cluster — every apply is gated behind a dry-run + explicit verify; a broken sensor/workflow must never +> silently swallow a push or false-deploy. + +**Goal:** A push to `master` on `gitadmin/fxhnt` auto-builds (only when deps/Dockerfile change) and auto-deploys +the fxhnt cockpit+dagster daemons — no manual `kubectl rollout restart`. + +**Architecture:** git-sync-first, image-on-demand. A minimal Sensor in `argo-events` hears the shared Gitea +webhook, path-filters the push to pick `mode=build` (Kaniko rebuild) vs `mode=deploy` (rollout only), and submits +the fxhnt-owned `fxhnt-cockpit` WorkflowTemplate into `foxhunt`. + +**Tech Stack:** Argo Events v1.9.10 (Sensor), Argo Workflows (Kaniko build + kubectl deploy), Gitea webhook, +Scaleway registry, K8s (bizworx Kapsule). + +## Global Constraints + +- **Project boundary:** fxhnt-owned objects (WorkflowTemplate, build/deploy pods, secrets, RBAC) live in + `foxhunt`. ONLY the Sensor lives in `argo-events` (unavoidable). No fxhnt secrets in `argo`/`argo-events`; no + cross-namespace RBAC. +- **Node pool:** all build/deploy pods target `nodeSelector: k8s.scaleway.com/pool-name: large` (the + `ci-compile-cpu` pool is DECOMMISSIONED — only `large`/`sfs` exist). A pod pinned to `ci-compile-cpu` never + schedules. +- **Registry:** `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit`, tagged `:` AND `:latest`. Secret + `scw-registry` (already in `foxhunt`). +- **Real-money safety:** the pipeline only builds/deploys the cockpit+dagster DAEMONS. It must NEVER touch the + execution CronJobs (ucits/bybit/ibkr rebalancers) or arm any executor. +- **No silent failures:** a Sensor that can't match, or a Workflow that errors, must surface loudly (a failed + push = a visible failed Workflow, never a swallowed no-op). Every manifest is `kubectl apply --dry-run=server` + validated before a real apply. +- **Reproducibility:** the deploy step annotates deployed Deployments with `fxhnt.io/deployed-sha=`. + +## File Structure + +- `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` — the reanimated WorkflowTemplate (ns foxhunt). Supersedes + the old `infra/argo/cockpit-build-deploy.yaml` (which is renamed/removed in the final task). +- `infra/argo/fxhnt-build-sensor.yaml` — the new Sensor (ns argo-events). +- Gitea webhook on `gitadmin/fxhnt` — created via Gitea API/UI (not a repo file; documented in the plan). + +--- + +## Task 1 — Reanimate + fix the WorkflowTemplate (build side) + +**Files:** +- Create: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (start from `infra/argo/cockpit-build-deploy.yaml`) + +**Interfaces:** +- Produces: a `WorkflowTemplate` named `fxhnt-cockpit` in ns `foxhunt` with params `commit-sha` (default HEAD) + and `mode` (default `build`; values `build`|`deploy`). The Sensor (Task 3) submits Workflows referencing it. + +- [ ] **Step 1: Copy the old template as the new file, fix the node pool** + +Copy `infra/argo/cockpit-build-deploy.yaml` → `infra/argo/fxhnt-cockpit-workflowtemplate.yaml`. In BOTH the +`kaniko-build` and `kubectl-deploy` templates, change every +`nodeSelector: { k8s.scaleway.com/pool-name: ci-compile-cpu, ... }` to +`nodeSelector: { k8s.scaleway.com/pool-name: large, topology.kubernetes.io/zone: fr-par-2 }`. +Grep to confirm no `ci-compile-cpu` remains: `grep -n ci-compile-cpu infra/argo/fxhnt-cockpit-workflowtemplate.yaml` → no hits. + +- [ ] **Step 2: Add the immutable `:` tag to the Kaniko destination** + +In the `kaniko-build` container args, alongside the existing +`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`, ADD a second line: +`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:{{=sprig.trunc(7, workflow.parameters.commit-sha)}}` +(the `sprig.trunc` expression matches how `build-image` truncates the sha). Keep `--cache=true` and the +`--cache-repo`. + +- [ ] **Step 3: Server-side dry-run validate the template** + +Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt` +Expected: `workflowtemplate.argoproj.io/fxhnt-cockpit created (server dry run)` with no schema errors. +(If it errors on an unknown field, the Argo CRD version differs — STOP and report; do not force-apply.) + +- [ ] **Step 4: Commit** + +```bash +git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml +git commit -m "ci(fxhnt): reanimate cockpit WorkflowTemplate (node pool large, sha7 tag)" +``` + +--- + +## Task 2 — Fix the deploy step (verify the dashboard external route first) + +**Files:** +- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (the `kubectl-deploy` template's script) + +**Interfaces:** +- Consumes: the WorkflowTemplate from Task 1. +- Produces: a deploy step that refreshes the daemons correctly for whichever external route is live. + +- [ ] **Step 1: Determine the live dashboard external route (decides restart vs scale-0→1)** + +Run these and read the result: +```bash +kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}' +kubectl get ingress -n foxhunt 2>/dev/null +kubectl -n foxhunt describe svc fxhnt-ingress 2>/dev/null | grep -iE 'endpoints|selector' +``` +Decide: +- If `dashboard.fxhnt.ai` is served THROUGH the tailscale sidecar (sidecar present AND no ingress route to the + dashboard) → KEEP the old `scale 0→1` dance (the tailscale session race is real). +- If it is served through `fxhnt-ingress` (an ingress/route points at the dashboard svc; the tailscale sidecar + is vestigial) → REPLACE the scale-dance with a plain `kubectl -n foxhunt rollout restart deploy/fxhnt-dashboard`. + +Record the decision in the plan's progress ledger with the evidence. If ambiguous, KEEP `scale 0→1` (the +conservative choice that cannot cause the stale-proxy "down"). + +- [ ] **Step 2: Apply the deploy-step decision + add the deployed-sha annotation** + +Edit the `kubectl-deploy` script. Regardless of the restart mechanism chosen, add BEFORE the rollout: +``` +kubectl -n foxhunt annotate deploy/fxhnt-dashboard deploy/dagster \ + fxhnt.io/deployed-sha="$(cd /workspace/src && git rev-parse --short HEAD)" --overwrite +``` +Keep the existing `kubectl apply -f` of the three manifests (fxhnt-dashboard.yaml, dagster.yaml, +network-policies/fxhnt-cockpit.yaml) and the `rollout status` waits. Keep `rollout restart deploy/dagster` +(dagster has no proxy race). + +- [ ] **Step 3: Dry-run validate again** + +Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt` +Expected: server dry-run OK. + +- [ ] **Step 4: Commit** + +```bash +git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml +git commit -m "ci(fxhnt): deploy step — + deployed-sha annotation" +``` + +--- + +## Task 3 — The Sensor (simple, single-trigger) — mode decided IN the workflow + +**Design note (verified 2026-07-21):** the Argo Events Sensor CRD here has an OPEN schema (no strict property +validation — `kubectl explain sensor.spec.triggers.template.conditions` errors, and the only `conditions:` in +live sensors are in `status`, not trigger-level). Rather than gamble on a fragile sensor-side path-negation +filter to pick build-vs-deploy, we keep the SENSOR SIMPLE (one trigger, exactly like `build-msp-portal`) and +move the build-vs-deploy decision INTO the workflow, where a robust `git diff --name-only` between the previous +and pushed commit is trivial and testable. This is the more robust design, not a fallback. + +**Files:** +- Create: `infra/argo/fxhnt-build-sensor.yaml` (ns argo-events) +- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (add a `decide-mode` step that git-diffs) + +**Interfaces:** +- Consumes: the `gitea-webhook` eventsource (shared), the `fxhnt-cockpit` WorkflowTemplate (Task 1/2). +- Produces: a `Sensor` `build-fxhnt` (argo-events) that submits ONE Workflow per master push; the workflow + self-selects build vs deploy. + +- [ ] **Step 1: Write the single-trigger Sensor (copy build-msp-portal structure exactly)** + +`kubectl get sensor build-msp-portal -n argo-events -o yaml` and mirror it. ONE dependency `push-to-master`: +- `body.ref` == `refs/heads/master` +- `body.pusher.username` != `argocd-image-updater` +(NO `body.commits.#.modified` filter — the mode decision moved to the workflow.) +ONE trigger `submit`: submits a Workflow `generateName: build-fxhnt-`, `namespace: foxhunt`, +`workflowTemplateRef: fxhnt-cockpit`, injecting `commit-sha`=`body.after`. Do NOT set `mode` (the workflow +computes it). This is structurally identical to the proven msp sensors → lowest risk. + +- [ ] **Step 2: Add a `decide-mode` step to the WorkflowTemplate** + +In `fxhnt-cockpit-workflowtemplate.yaml`, change the DAG so the FIRST step (after git-clone) is `decide-mode`: +a small `alpine/git` script that runs, in the cloned repo at `commit-sha`: +``` +CHANGED=$(git diff --name-only "${SHA}~1" "${SHA}" 2>/dev/null || echo "ALL") +if echo "$CHANGED" | grep -qE '^pyproject\.toml$|^infra/docker/fxhnt\.Dockerfile$'; then + echo build; else echo deploy; fi +``` +Output the result (`build`/`deploy`) as an Argo output parameter `mode`. Then gate the `kaniko-build` task on +`{{tasks.decide-mode.outputs.parameters.mode}} == build`, and `kubectl-deploy` depends on +`build.Succeeded || build.Skipped` (unchanged). Edge case: if `${SHA}~1` doesn't exist (first commit / shallow), +default to `build` (safe — a rebuild is never wrong, only slower). The explicit `mode` PARAM stays on the +template (default `auto`) so a human can still force `mode=build`/`mode=deploy` on a manual `argo submit`; +`decide-mode` only runs when `mode=auto`. + +- [ ] **Step 3: Dry-run validate both** + +```bash +kubectl apply -f infra/argo/fxhnt-build-sensor.yaml --dry-run=server -n argo-events +kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt +``` +Expected: both server-dry-run OK. + +- [ ] **Step 4: Commit** + +```bash +git add infra/argo/fxhnt-build-sensor.yaml infra/argo/fxhnt-cockpit-workflowtemplate.yaml +git commit -m "ci(fxhnt): single-trigger sensor + in-workflow git-diff mode decision" +``` + +--- + +## Task 4 — Apply to the cluster + wire the Gitea webhook + +**Files:** none new — this task applies the manifests and creates the webhook. + +- [ ] **Step 1: Apply the WorkflowTemplate + Sensor for real** + +```bash +kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml -n foxhunt +kubectl apply -f infra/argo/fxhnt-build-sensor.yaml -n argo-events +kubectl get workflowtemplate fxhnt-cockpit -n foxhunt +kubectl get sensor build-fxhnt -n argo-events +``` +Expected: both present. The sensor pod (`kubectl get pods -n argo-events | grep build-fxhnt`) reaches Running. + +- [ ] **Step 2: Create the Gitea webhook on `gitadmin/fxhnt`** + +The `gitea-webhook` eventsource endpoint is the same target the `bizworx-ops/infrastructure` repo already POSTs +to. Find it: `kubectl get eventsource gitea-webhook -n argo-events -o yaml` (the webhook `port`/`endpoint`) and +how the existing infra-repo webhook is configured (inspect via Gitea admin UI or API on an existing repo). +Create the equivalent webhook on `gitadmin/fxhnt`: push events, active, POST JSON to the eventsource URL. Record +the exact URL used in the ledger. Do NOT paste any webhook secret into the transcript. + +- [ ] **Step 3: Verify the webhook reaches the sensor (dry, safe probe)** + +Trigger a harmless test push (or use Gitea's "Test Delivery" webhook button) and confirm the sensor received it: +```bash +kubectl logs -n argo-events -l sensor-name=build-fxhnt --tail=50 | grep -iE 'triggered|push-to-master|dispatch' +``` +Expected: the sensor logs the received push. If Gitea's test-delivery uses a non-master ref, the sensor +correctly does NOT fire — confirm it's reachable via the eventsource logs instead. + +--- + +## Task 5 — End-to-end verification (both modes) + retire the old file + +**Files:** +- Remove: `infra/argo/cockpit-build-deploy.yaml` (superseded) + +- [ ] **Step 1: Verify mode=deploy end-to-end (code-only push)** + +Make a trivial code-only commit on `master` (e.g. touch a comment in `src/`), push, and watch: +```bash +kubectl get wf -n foxhunt --sort-by=.metadata.creationTimestamp | tail -3 +``` +Expected: a `build-fxhnt-*` Workflow runs with `mode=deploy`, SKIPS the kaniko-build step, deploys, and the +dashboard stays UP (check `dashboard.fxhnt.ai` reachable post-deploy, and +`kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{.metadata.annotations.fxhnt\.io/deployed-sha}'` +shows the new sha). Total time ~1-2 min, no Kaniko build. + +- [ ] **Step 2: Verify mode=build end-to-end (dep push)** + +Make a no-op-but-real change to `pyproject.toml` (e.g. reorder a dependency, or bump a harmless comment), push, +and watch: the Workflow runs `mode=build`, Kaniko produces `fxhnt-cockpit:`+`:latest` +(`kubectl run` a crane/skopeo check OR verify the workflow's kaniko step succeeded), then deploys. Confirm the +`:` tag exists in the registry. + +- [ ] **Step 3: Confirm no false trigger + real-money safety** + +- Push a change touching ONLY `docs/` → confirm a `mode=deploy` Workflow runs (harmless) OR, if docs are + excluded, no Workflow — either is fine, but it must NOT trigger a build. +- Confirm NO execution CronJob was modified/triggered by any pipeline run: + `kubectl get cronjob -n foxhunt` (ucits/bybit/ibkr rebalancers unchanged, bybit/ibkr still Suspend=True). + +- [ ] **Step 4: Retire the old file + commit** + +```bash +git rm infra/argo/cockpit-build-deploy.yaml +git commit -m "ci(fxhnt): retire old cockpit-build-deploy (superseded by workflowtemplate + sensor)" +``` + +- [ ] **Step 5: Final verification summary** + +Confirm all Global Constraints hold: sensor in argo-events only, workflow+pods in foxhunt, node pool `large`, +no cross-ns RBAC, no fxhnt secrets in platform ns, real-money CronJobs untouched. Record in the ledger. diff --git a/docs/superpowers/specs/2026-07-21-fxhnt-cicd-pipeline-design.md b/docs/superpowers/specs/2026-07-21-fxhnt-cicd-pipeline-design.md index 81aeb37..d86a963 100644 --- a/docs/superpowers/specs/2026-07-21-fxhnt-cicd-pipeline-design.md +++ b/docs/superpowers/specs/2026-07-21-fxhnt-cicd-pipeline-design.md @@ -59,15 +59,15 @@ git push → gitadmin/fxhnt (master) [SHARED: Gitea] │ Gitea webhook ──────────────► argo-events `gitea-webhook` eventsource [SHARED: platform eventbus] ▼ Sensor `build-fxhnt` (ns argo-events — the ONE fxhnt object in a platform ns, kept minimal) - │ filters: ref==refs/heads/master, pusher != argocd-image-updater - │ path-filter on the pushed commits' modified files decides the mode: - │ • touched pyproject.toml OR infra/docker/fxhnt.Dockerfile → param mode=build - │ • otherwise (src/, infra/k8s/, config) → param mode=deploy - │ (NB: the repo has NO lock file — the Dockerfile `pip install`s straight from pyproject.toml, - │ so pyproject.toml + the Dockerfile are the ONLY dependency sources that require a rebuild.) - ▼ submits a Workflow into ns `foxhunt` ─────────────────────────────── [FXHNT-OWNED from here down] + │ filters: ref==refs/heads/master, pusher != argocd-image-updater (SINGLE trigger, like the msp sensors) + ▼ submits ONE Workflow into ns `foxhunt`, injecting commit-sha ─────── [FXHNT-OWNED from here down] Argo Workflow (WorkflowTemplate `fxhnt-cockpit`, ns foxhunt, SA argo-workflow, node-pool `large`) — DAG: - ├─ mode=build: git-clone(sha) → Kaniko → rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:+:latest + ├─ decide-mode: git-clone(sha) → `git diff --name-only sha~1 sha` → + │ touched pyproject.toml OR infra/docker/fxhnt.Dockerfile → mode=build ; else mode=deploy + │ (mode decided IN the workflow, not a fragile sensor filter — the CRD schema is open, so a + │ sensor-side path-negation can't be validated; a git-diff step is robust + testable. No lock file + │ exists → those two paths are the complete rebuild-trigger set.) + ├─ mode=build: Kaniko → rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:+:latest └─ (build skipped when mode=deploy) → deploy step (in foxhunt, no cross-ns RBAC): kubectl apply manifests + refresh daemons (git-sync re-pulls src) ```