# fxhnt CI/CD Pipeline — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this
> plan task-by-task. Steps use checkbox tracking. This plan writes/edits YAML manifests and applies them to a
> LIVE cluster — every apply is gated behind a dry-run + explicit verify; a broken sensor/workflow must never
> silently swallow a push or false-deploy.

**Goal:** A push to `master` on `gitadmin/fxhnt` auto-builds (only when deps/Dockerfile change) and auto-deploys
the fxhnt cockpit+dagster daemons — no manual `kubectl rollout restart`.

**Architecture:** git-sync-first, image-on-demand. A minimal Sensor in `argo-events` hears the shared Gitea
webhook, path-filters the push to pick `mode=build` (Kaniko rebuild) vs `mode=deploy` (rollout only), and submits
the fxhnt-owned `fxhnt-cockpit` WorkflowTemplate into `foxhunt`.

**Tech Stack:** Argo Events v1.9.10 (Sensor), Argo Workflows (Kaniko build + kubectl deploy), Gitea webhook,
Scaleway registry, K8s (bizworx Kapsule).

## Global Constraints

- **Project boundary:** fxhnt-owned objects (WorkflowTemplate, build/deploy pods, secrets, RBAC) live in
  `foxhunt`. ONLY the Sensor lives in `argo-events` (unavoidable). No fxhnt secrets in `argo`/`argo-events`; no
  cross-namespace RBAC.
- **Node pool:** all build/deploy pods target `nodeSelector: k8s.scaleway.com/pool-name: large` (the
  `ci-compile-cpu` pool is DECOMMISSIONED — only `large`/`sfs` exist). A pod pinned to `ci-compile-cpu` never
  schedules.
- **Registry:** `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit`, tagged `:<sha7>` AND `:latest`. Secret
  `scw-registry` (already in `foxhunt`).
- **Real-money safety:** the pipeline only builds/deploys the cockpit+dagster DAEMONS. It must NEVER touch the
  execution CronJobs (ucits/bybit/ibkr rebalancers) or arm any executor.
- **No silent failures:** a Sensor that can't match, or a Workflow that errors, must surface loudly (a failed
  push = a visible failed Workflow, never a swallowed no-op). Every manifest is `kubectl apply --dry-run=server`
  validated before a real apply.
- **Reproducibility:** the deploy step annotates deployed Deployments with `fxhnt.io/deployed-sha=<sha7>`.

## File Structure

- `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` — the reanimated WorkflowTemplate (ns foxhunt). Supersedes
  the old `infra/argo/cockpit-build-deploy.yaml` (which is renamed/removed in the final task).
- `infra/argo/fxhnt-build-sensor.yaml` — the new Sensor (ns argo-events).
- Gitea webhook on `gitadmin/fxhnt` — created via Gitea API/UI (not a repo file; documented in the plan).

---

## Task 1 — Reanimate + fix the WorkflowTemplate (build side)

**Files:**
- Create: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (start from `infra/argo/cockpit-build-deploy.yaml`)

**Interfaces:**
- Produces: a `WorkflowTemplate` named `fxhnt-cockpit` in ns `foxhunt` with params `commit-sha` (default HEAD)
  and `mode` (default `build`; values `build`|`deploy`). The Sensor (Task 3) submits Workflows referencing it.

- [ ] **Step 1: Copy the old template as the new file, fix the node pool**

Copy `infra/argo/cockpit-build-deploy.yaml` → `infra/argo/fxhnt-cockpit-workflowtemplate.yaml`. In BOTH the
`kaniko-build` and `kubectl-deploy` templates, change every
`nodeSelector: { k8s.scaleway.com/pool-name: ci-compile-cpu, ... }` to
`nodeSelector: { k8s.scaleway.com/pool-name: large, topology.kubernetes.io/zone: fr-par-2 }`.
Grep to confirm no `ci-compile-cpu` remains: `grep -n ci-compile-cpu infra/argo/fxhnt-cockpit-workflowtemplate.yaml` → no hits.

- [ ] **Step 2: Add the immutable `:<sha7>` tag to the Kaniko destination**

In the `kaniko-build` container args, alongside the existing
`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`, ADD a second line:
`--destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:{{=sprig.trunc(7, workflow.parameters.commit-sha)}}`
(the `sprig.trunc` expression matches how `build-image` truncates the sha). Keep `--cache=true` and the
`--cache-repo`.

- [ ] **Step 3: Server-side dry-run validate the template**

Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt`
Expected: `workflowtemplate.argoproj.io/fxhnt-cockpit created (server dry run)` with no schema errors.
(If it errors on an unknown field, the Argo CRD version differs — STOP and report; do not force-apply.)

- [ ] **Step 4: Commit**

```bash
git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml
git commit -m "ci(fxhnt): reanimate cockpit WorkflowTemplate (node pool large, sha7 tag)"
```

---

## Task 2 — Fix the deploy step (verify the dashboard external route first)

**Files:**
- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (the `kubectl-deploy` template's script)

**Interfaces:**
- Consumes: the WorkflowTemplate from Task 1.
- Produces: a deploy step that refreshes the daemons correctly for whichever external route is live.

- [ ] **Step 1: Determine the live dashboard external route (decides restart vs scale-0→1)**

Run these and read the result:
```bash
kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
kubectl get ingress -n foxhunt 2>/dev/null
kubectl -n foxhunt describe svc fxhnt-ingress 2>/dev/null | grep -iE 'endpoints|selector'
```
Decide:
- If `dashboard.fxhnt.ai` is served THROUGH the tailscale sidecar (sidecar present AND no ingress route to the
  dashboard) → KEEP the old `scale 0→1` dance (the tailscale session race is real).
- If it is served through `fxhnt-ingress` (an ingress/route points at the dashboard svc; the tailscale sidecar
  is vestigial) → REPLACE the scale-dance with a plain `kubectl -n foxhunt rollout restart deploy/fxhnt-dashboard`.

Record the decision in the plan's progress ledger with the evidence. If ambiguous, KEEP `scale 0→1` (the
conservative choice that cannot cause the stale-proxy "down").

- [ ] **Step 2: Apply the deploy-step decision + add the deployed-sha annotation**

Edit the `kubectl-deploy` script. Regardless of the restart mechanism chosen, add BEFORE the rollout:
```
kubectl -n foxhunt annotate deploy/fxhnt-dashboard deploy/dagster \
  fxhnt.io/deployed-sha="$(cd /workspace/src && git rev-parse --short HEAD)" --overwrite
```
Keep the existing `kubectl apply -f` of the three manifests (fxhnt-dashboard.yaml, dagster.yaml,
network-policies/fxhnt-cockpit.yaml) and the `rollout status` waits. Keep `rollout restart deploy/dagster`
(dagster has no proxy race).

- [ ] **Step 3: Dry-run validate again**

Run: `kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt`
Expected: server dry-run OK.

- [ ] **Step 4: Commit**

```bash
git add infra/argo/fxhnt-cockpit-workflowtemplate.yaml
git commit -m "ci(fxhnt): deploy step — <restart-choice> + deployed-sha annotation"
```

---

## Task 3 — The Sensor (simple, single-trigger) — mode decided IN the workflow

**Design note (verified 2026-07-21):** the Argo Events Sensor CRD here has an OPEN schema (no strict property
validation — `kubectl explain sensor.spec.triggers.template.conditions` errors, and the only `conditions:` in
live sensors are in `status`, not trigger-level). Rather than gamble on a fragile sensor-side path-negation
filter to pick build-vs-deploy, we keep the SENSOR SIMPLE (one trigger, exactly like `build-msp-portal`) and
move the build-vs-deploy decision INTO the workflow, where a robust `git diff --name-only` between the previous
and pushed commit is trivial and testable. This is the more robust design, not a fallback.

**Files:**
- Create: `infra/argo/fxhnt-build-sensor.yaml` (ns argo-events)
- Modify: `infra/argo/fxhnt-cockpit-workflowtemplate.yaml` (add a `decide-mode` step that git-diffs)

**Interfaces:**
- Consumes: the `gitea-webhook` eventsource (shared), the `fxhnt-cockpit` WorkflowTemplate (Task 1/2).
- Produces: a `Sensor` `build-fxhnt` (argo-events) that submits ONE Workflow per master push; the workflow
  self-selects build vs deploy.

- [ ] **Step 1: Write the single-trigger Sensor (copy build-msp-portal structure exactly)**

`kubectl get sensor build-msp-portal -n argo-events -o yaml` and mirror it. ONE dependency `push-to-master`:
- `body.ref` == `refs/heads/master`
- `body.pusher.username` != `argocd-image-updater`
(NO `body.commits.#.modified` filter — the mode decision moved to the workflow.)
ONE trigger `submit`: submits a Workflow `generateName: build-fxhnt-`, `namespace: foxhunt`,
`workflowTemplateRef: fxhnt-cockpit`, injecting `commit-sha`=`body.after`. Do NOT set `mode` (the workflow
computes it). This is structurally identical to the proven msp sensors → lowest risk.

- [ ] **Step 2: Add a `decide-mode` step to the WorkflowTemplate**

In `fxhnt-cockpit-workflowtemplate.yaml`, change the DAG so the FIRST step (after git-clone) is `decide-mode`:
a small `alpine/git` script that runs, in the cloned repo at `commit-sha`:
```
CHANGED=$(git diff --name-only "${SHA}~1" "${SHA}" 2>/dev/null || echo "ALL")
if echo "$CHANGED" | grep -qE '^pyproject\.toml$|^infra/docker/fxhnt\.Dockerfile$'; then
  echo build; else echo deploy; fi
```
Output the result (`build`/`deploy`) as an Argo output parameter `mode`. Then gate the `kaniko-build` task on
`{{tasks.decide-mode.outputs.parameters.mode}} == build`, and `kubectl-deploy` depends on
`build.Succeeded || build.Skipped` (unchanged). Edge case: if `${SHA}~1` doesn't exist (first commit / shallow),
default to `build` (safe — a rebuild is never wrong, only slower). The explicit `mode` PARAM stays on the
template (default `auto`) so a human can still force `mode=build`/`mode=deploy` on a manual `argo submit`;
`decide-mode` only runs when `mode=auto`.

- [ ] **Step 3: Dry-run validate both**

```bash
kubectl apply -f infra/argo/fxhnt-build-sensor.yaml --dry-run=server -n argo-events
kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml --dry-run=server -n foxhunt
```
Expected: both server-dry-run OK.

- [ ] **Step 4: Commit**

```bash
git add infra/argo/fxhnt-build-sensor.yaml infra/argo/fxhnt-cockpit-workflowtemplate.yaml
git commit -m "ci(fxhnt): single-trigger sensor + in-workflow git-diff mode decision"
```

---

## Task 4 — Apply to the cluster + wire the Gitea webhook

**Files:** none new — this task applies the manifests and creates the webhook.

- [ ] **Step 1: Apply the WorkflowTemplate + Sensor for real**

```bash
kubectl apply -f infra/argo/fxhnt-cockpit-workflowtemplate.yaml -n foxhunt
kubectl apply -f infra/argo/fxhnt-build-sensor.yaml -n argo-events
kubectl get workflowtemplate fxhnt-cockpit -n foxhunt
kubectl get sensor build-fxhnt -n argo-events
```
Expected: both present. The sensor pod (`kubectl get pods -n argo-events | grep build-fxhnt`) reaches Running.

- [ ] **Step 2: Create the Gitea webhook on `gitadmin/fxhnt`**

The `gitea-webhook` eventsource endpoint is the same target the `bizworx-ops/infrastructure` repo already POSTs
to. Find it: `kubectl get eventsource gitea-webhook -n argo-events -o yaml` (the webhook `port`/`endpoint`) and
how the existing infra-repo webhook is configured (inspect via Gitea admin UI or API on an existing repo).
Create the equivalent webhook on `gitadmin/fxhnt`: push events, active, POST JSON to the eventsource URL. Record
the exact URL used in the ledger. Do NOT paste any webhook secret into the transcript.

- [ ] **Step 3: Verify the webhook reaches the sensor (dry, safe probe)**

Trigger a harmless test push (or use Gitea's "Test Delivery" webhook button) and confirm the sensor received it:
```bash
kubectl logs -n argo-events -l sensor-name=build-fxhnt --tail=50 | grep -iE 'triggered|push-to-master|dispatch'
```
Expected: the sensor logs the received push. If Gitea's test-delivery uses a non-master ref, the sensor
correctly does NOT fire — confirm it's reachable via the eventsource logs instead.

---

## Task 5 — End-to-end verification (both modes) + retire the old file

**Files:**
- Remove: `infra/argo/cockpit-build-deploy.yaml` (superseded)

- [ ] **Step 1: Verify mode=deploy end-to-end (code-only push)**

Make a trivial code-only commit on `master` (e.g. touch a comment in `src/`), push, and watch:
```bash
kubectl get wf -n foxhunt --sort-by=.metadata.creationTimestamp | tail -3
```
Expected: a `build-fxhnt-*` Workflow runs with `mode=deploy`, SKIPS the kaniko-build step, deploys, and the
dashboard stays UP (check `dashboard.fxhnt.ai` reachable post-deploy, and
`kubectl get deploy fxhnt-dashboard -n foxhunt -o jsonpath='{.metadata.annotations.fxhnt\.io/deployed-sha}'`
shows the new sha). Total time ~1-2 min, no Kaniko build.

- [ ] **Step 2: Verify mode=build end-to-end (dep push)**

Make a no-op-but-real change to `pyproject.toml` (e.g. reorder a dependency, or bump a harmless comment), push,
and watch: the Workflow runs `mode=build`, Kaniko produces `fxhnt-cockpit:<sha7>`+`:latest`
(`kubectl run` a crane/skopeo check OR verify the workflow's kaniko step succeeded), then deploys. Confirm the
`:<sha7>` tag exists in the registry.

- [ ] **Step 3: Confirm no false trigger + real-money safety**

- Push a change touching ONLY `docs/` → confirm a `mode=deploy` Workflow runs (harmless) OR, if docs are
  excluded, no Workflow — either is fine, but it must NOT trigger a build.
- Confirm NO execution CronJob was modified/triggered by any pipeline run:
  `kubectl get cronjob -n foxhunt` (ucits/bybit/ibkr rebalancers unchanged, bybit/ibkr still Suspend=True).

- [ ] **Step 4: Retire the old file + commit**

```bash
git rm infra/argo/cockpit-build-deploy.yaml
git commit -m "ci(fxhnt): retire old cockpit-build-deploy (superseded by workflowtemplate + sensor)"
```

- [ ] **Step 5: Final verification summary**

Confirm all Global Constraints hold: sensor in argo-events only, workflow+pods in foxhunt, node pool `large`,
no cross-ns RBAC, no fxhnt secrets in platform ns, real-money CronJobs untouched. Record in the ledger.
