Merge: relocate platform IaC to fxhnt (Phase 2D)

Platform terraform (live+modules) + k8s manifests moved to fxhnt (now the IaC home).
Rust-only remnants kept: docker build images, gpu-taint.yaml, Rust scripts. foxhunt
being archived; history retained. Live platform IaC now in the fxhnt repo.

docs/superpowers/plans/2026-06-21-decommission-gitlab.md

@@ -0,0 +1,191 @@
+# Decommission GitLab (compute) — Implementation Plan (Phase 2C)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> **Do NOT run subagent-driven** — irreversible helm uninstall + a proxy edit that can take down all
+> `*.fxhnt.ai` if mis-ordered; each step needs a human-confirmed gate. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Remove GitLab's compute (2 helm releases, dead Argo events, the `:5050` proxy block, the values
+manifest) while retaining all GitLab data as a cold backup.
+
+**Architecture:** Proxy `:5050` block removed FIRST (defuses the nginx-startup landmine), then helm
+uninstall, then dead-event + manifest cleanup. gitaly PVC + GitLab buckets + secrets are left untouched.
+
+**Tech Stack:** helm, kubectl, nginx+socat Tailscale proxy, Argo Events.
+
+**Spec:** `docs/superpowers/specs/2026-06-21-decommission-gitlab-design.md`
+
+---
+
+## Task 0: Branch
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/decommission-gitlab
+```
+
+---
+
+## Task 1: Pre-flight — confirm no live GitLab consumers + baseline
+
+- [ ] **Step 1: Confirm nothing pulls the GitLab image / uses the GitLab backend anymore**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+echo "pods on gitlab-registry image (expect none):"
+kubectl get pods -n foxhunt -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' 2>/dev/null | grep -c "gitlab-registry.foxhunt.svc.cluster.local:5000" | xargs echo
+echo "git remotes (expect git.fxhnt.ai = Gitea, not :2222/root):"
+git remote get-url origin; (cd /home/jgrusewski/Work/fxhnt && git remote get-url origin)
+```
+Expected: `0` pods on the GitLab image; both remotes `ssh://git@git.fxhnt.ai/gitadmin/...`. **STOP if any
+pod still references the GitLab registry.**
+
+- [ ] **Step 2: Baseline RAM + pod count**
+```bash
+kubectl top pods -n foxhunt 2>/dev/null | grep -i gitlab | awk '{s+=$3} END{print NR" gitlab pods, "s" Mi"}'
+```
+Expected: ~12 pods, ~4.6 GiB (this is what gets reclaimed).
+
+---
+
+## Task 2: Remove the `:5050` registry block from the proxy (FIRST — safety-critical)
+
+**Files:** Modify `infra/k8s/gitlab/tailscale-proxy.yaml`
+
+- [ ] **Step 1: Delete `containerPort: 5050`**
+
+In `infra/k8s/gitlab/tailscale-proxy.yaml`, remove the line:
+```
+            - containerPort: 5050
+```
+(leave `- containerPort: 443`).
+
+- [ ] **Step 2: Delete the entire `# Container Registry — git.fxhnt.ai:5050` server block**
+
+Remove the whole block (the `# Container Registry — git.fxhnt.ai:5050` comment + its `server { listen 5050
+ssl; ... }`, i.e. the `proxy_pass http://gitlab-registry.foxhunt.svc.cluster.local:5000;` server). Leave
+the `git.fxhnt.ai:443` (Gitea) block and all other server blocks intact.
+
+- [ ] **Step 3: Confirm no GitLab upstream remains in the proxy**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+grep -nE "gitlab-registry|gitlab-webservice|gitlab-gitlab-shell|listen 5050" infra/k8s/gitlab/tailscale-proxy.yaml || echo "OK: no gitlab upstreams left"
+```
+Expected: `OK: no gitlab upstreams left`.
+
+- [ ] **Step 4: Apply + restart proxy**
+```bash
+kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml
+kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
+kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s
+```
+
+- [ ] **Step 5: GATE — proxy still serves all `*.fxhnt.ai`**
+```bash
+for h in git dashboard grafana minio; do curl -sk -m 10 -o /dev/null -w "$h.fxhnt.ai: %{http_code}\n" https://$h.fxhnt.ai/ 2>&1; done
+curl -sk -m 10 https://git.fxhnt.ai/api/healthz 2>/dev/null | grep -o '"status": "pass"'
+```
+Expected: `git`/`dashboard` 200 (others may 200/302/401 — just not 000). `"status": "pass"` for Gitea.
+**STOP + revert (`git checkout` the file + re-apply) if git/dashboard go 000.**
+
+---
+
+## Task 3: Helm-uninstall GitLab + agent
+
+- [ ] **Step 1: Uninstall both releases**
+```bash
+helm uninstall gitlab -n foxhunt 2>&1 | tail -1
+helm uninstall foxhunt-agent -n foxhunt 2>&1 | tail -1
+```
+Expected: both `uninstalled`.
+
+- [ ] **Step 2: Wait for GitLab pods to terminate + confirm RAM reclaimed**
+```bash
+i=0; until [ -z "$(kubectl get pods -n foxhunt 2>/dev/null | grep -iE 'gitlab|gitaly' | grep -v Terminating)" ] || [ $i -ge 24 ]; do sleep 5; i=$((i+1)); done
+echo "remaining gitlab pods:"; kubectl get pods -n foxhunt 2>/dev/null | grep -iE "gitlab|gitaly" || echo "  (none)"
+```
+Expected: no GitLab pods (or only `Terminating` briefly). The `gitaly` PVC remains (cold backup).
+
+---
+
+## Task 4: Delete dead Argo Events
+
+- [ ] **Step 1: Delete the GitLab CI eventsource + sensor**
+```bash
+kubectl delete eventsource gitlab-push -n foxhunt 2>&1 | tail -1
+kubectl delete sensor ci-pipeline-trigger -n foxhunt 2>&1 | tail -1
+echo "=== remaining events (gitea-push + gitea-deploy must stay) ==="
+kubectl get eventsource,sensor -n foxhunt 2>/dev/null
+```
+Expected: both deleted; `gitea-push` eventsource + `gitea-deploy` sensor still present (+ workflow-trigger).
+
+---
+
+## Task 5: Verify cold backup intact + remove values manifest + commit
+
+- [ ] **Step 1: Confirm the cold backup is untouched**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+echo "gitaly PVC (must remain):"; kubectl get pvc -n foxhunt 2>/dev/null | grep gitaly || echo "  MISSING!"
+echo "gitlab secrets (sample, must remain):"; kubectl get secret -n foxhunt 2>/dev/null | grep -c gitlab | xargs echo "  gitlab-* secrets:"
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+echo "Scaleway gitlab buckets (must remain):"; scw object bucket list 2>/dev/null | grep -i gitlab
+```
+Expected: gitaly PVC present; gitlab secrets count > 0; both Scaleway gitlab buckets present. **STOP if the
+gitaly PVC is gone** (helm uninstall should not have removed a StatefulSet PVC — investigate).
+
+- [ ] **Step 2: Remove the GitLab Helm values manifest**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git rm infra/k8s/gitlab/values.yaml 2>&1 | tail -1
+ls infra/k8s/gitlab/   # tailscale-proxy.yaml must remain
+```
+Expected: `values.yaml` removed; `tailscale-proxy.yaml` still listed.
+
+- [ ] **Step 3: Commit**
+```bash
+git add -A infra/k8s/gitlab/
+git commit -m "chore(infra): decommission GitLab compute — keep data as cold backup (Phase 2C)
+
+helm uninstall gitlab + foxhunt-agent (~4.6GiB RAM / 12 pods freed); removed :5050
+registry block from the (now Gitea) proxy; deleted dead gitlab-push/ci-pipeline-trigger
+Argo events; removed gitlab values.yaml. RETAINED (cold backup): gitaly PVC,
+gitlab MinIO/Scaleway buckets, gitlab-* secrets. tailscale-proxy.yaml kept (serves Gitea).
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 6: Final verification
+
+- [ ] **Step 1: Releases gone + platform healthy + auto-deploy still works**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+echo "=== helm (no gitlab/agent) ==="; helm list -n foxhunt | grep -iE "gitlab" || echo "  (none — removed)"
+echo "=== platform health ==="
+curl -sk -m 10 -o /dev/null -w "git.fxhnt.ai: %{http_code}\n" https://git.fxhnt.ai/api/healthz
+curl -sk -m 10 -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
+echo "=== auto-deploy still wired (push test) ==="
+BEFORE=$(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)
+( cd /home/jgrusewski/Work/fxhnt && git commit --allow-empty -m "test: post-GitLab-removal auto-deploy" -q && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1 )
+i=0; until [ "$(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)" -gt "$BEFORE" ] || [ $i -ge 12 ]; do sleep 5; i=$((i+1)); done
+echo "cockpit wfs: $BEFORE -> $(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)"
+```
+Expected: no gitlab helm releases; `git`/`dashboard` 200; cockpit wf count incremented (webhook→Argo intact).
+
+---
+
+## Rollback
+- **Proxy (Task 2):** `git checkout infra/k8s/gitlab/tailscale-proxy.yaml` + `kubectl apply` + rollout restart.
+- **GitLab (Task 3):** data is retained (gitaly PVC + buckets + secrets) — `helm install gitlab gitlab/gitlab
+  --version 9.9.3 -f <values>` would re-attach the gitaly PVC. (Primary going forward is Gitea/Scaleway.)
+
+---
+
+## Acceptance criteria (from spec)
+- No `gitlab`/`foxhunt-agent` helm releases; GitLab pods gone (~4.6 GiB reclaimed). ✅ Task 3, Task 6
+- `git.fxhnt.ai` + `dashboard.fxhnt.ai` 200; push auto-deploys via Gitea→Argo. ✅ Task 6
+- `gitlab-push`/`ci-pipeline-trigger` deleted; `gitea-push`/`gitea-deploy` kept. ✅ Task 4
+- `:5050` block gone; proxy healthy. ✅ Task 2
+- Cold backup intact: gitaly PVC + GitLab buckets + `gitlab-*` secrets. ✅ Task 5
+- `values.yaml` removed; `tailscale-proxy.yaml` kept; committed. ✅ Task 5

docs/superpowers/plans/2026-06-21-gitea-replace-gitlab.md

@@ -0,0 +1,472 @@
+# Gitea Replaces GitLab + Cockpit Registry to Scaleway — Implementation Plan (Phase 2B)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> **Do NOT run subagent-driven** — this touches live git access, the container registry, and the cockpit
+> deploy; each cutover step needs a human-confirmed gate. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Stand up Gitea (chart + existing Postgres) as the git host behind the unchanged `git.fxhnt.ai`,
+migrate both repos, and move the cockpit image to Scaleway Container Registry — so GitLab can be removed in 2C.
+
+**Architecture:** Gitea comes up internal-only; repos migrate + validate over a port-forward; the cockpit
+image moves to Scaleway registry and is build+rollout tested; THEN `git.fxhnt.ai` (HTTPS + SSH `:2222`)
+cuts over to Gitea in one switch; finally a Gitea webhook → Argo Events auto-deploys the cockpit. GitLab
+stays running (unaddressed) as the rollback net until 2C.
+
+**Tech Stack:** Gitea Helm chart (`gitea-charts/gitea`), existing in-cluster PostgreSQL, nginx+socat
+Tailscale proxy, Scaleway Container Registry, Argo Events, kaniko.
+
+**Spec:** `docs/superpowers/specs/2026-06-21-gitea-replace-gitlab-design.md`
+
+**TWO REPOS:** infra changes → `foxhunt` (`~/Work/foxhunt`); cockpit build/deploy → `fxhnt`
+(`~/Work/fxhnt`). Each gets its own branch + commits.
+
+---
+
+## Shared env
+```bash
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
+```
+
+---
+
+## Task 0: Branches
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/gitea-replace-gitlab
+cd /home/jgrusewski/Work/fxhnt   && git checkout -b chore/gitea-cockpit-registry
+```
+
+---
+
+## Task 1: Postgres `gitea` DB + secrets
+
+**Files:** none committed (cluster secrets). Record secret names only.
+
+- [ ] **Step 1: Read existing postgres superuser creds**
+```bash
+PGU=$(kubectl get deploy postgres -n foxhunt -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="POSTGRES_USER")].value}')
+PGPOD=$(kubectl get pod -n foxhunt -l app=postgres -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || kubectl get pod -n foxhunt | grep '^postgres-' | grep -v backup | awk '{print $1}' | head -1)
+echo "postgres user=$PGU pod=$PGPOD"
+```
+Expected: a username + a running pod name. (If `POSTGRES_USER` is via secretRef not value, read it from the referenced secret instead.)
+
+- [ ] **Step 2: Generate a Gitea DB password + create role/db**
+```bash
+GITEA_DB_PW=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
+kubectl exec -n foxhunt "$PGPOD" -- psql -U "$PGU" -v ON_ERROR_STOP=1 -c \
+  "CREATE ROLE gitea LOGIN PASSWORD '$GITEA_DB_PW';" -c \
+  "CREATE DATABASE gitea OWNER gitea ENCODING 'UTF8';" 2>&1 | grep -iE "CREATE|already exists|Error"
+echo "$GITEA_DB_PW" # capture for the next step
+```
+Expected: `CREATE ROLE` + `CREATE DATABASE` (or `already exists` if re-run).
+
+- [ ] **Step 3: Create the k8s secrets Gitea needs** (DB creds + admin)
+```bash
+GITEA_ADMIN_PW=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
+kubectl create secret generic gitea-db -n foxhunt \
+  --from-literal=password="$GITEA_DB_PW" --dry-run=client -o yaml | kubectl apply -f -
+kubectl create secret generic gitea-admin -n foxhunt \
+  --from-literal=username=gitadmin --from-literal=password="$GITEA_ADMIN_PW" \
+  --from-literal=email=jeroen@bizworx.nl --dry-run=client -o yaml | kubectl apply -f -
+echo "ADMIN PW (save in your password manager): $GITEA_ADMIN_PW"
+```
+Expected: both secrets `created`/`configured`. **Save the admin password** — needed for Gitea login.
+
+---
+
+## Task 2: Gitea Helm values + install (internal-only)
+
+**Files:** Create `infra/k8s/gitea/values.yaml` (foxhunt repo).
+
+- [ ] **Step 1: Add the Gitea Helm repo**
+```bash
+helm repo add gitea-charts https://dl.gitea.com/charts/ && helm repo update gitea-charts
+helm search repo gitea-charts/gitea --versions | head -3
+```
+Expected: lists chart versions (use the latest stable in Step 3).
+
+- [ ] **Step 2: Write `infra/k8s/gitea/values.yaml`**
+```yaml
+# Gitea — lightweight git host replacing GitLab (Phase 2B). External Postgres (existing in-cluster
+# `postgres`), no bundled DB/redis/memcached, Actions off (Argo does CI). Internal-only until cutover.
+replicaCount: 1
+image:
+  rootless: true
+
+# Disable all bundled subcharts — reuse the existing in-cluster postgres
+postgresql:
+  enabled: false
+postgresql-ha:
+  enabled: false
+redis-cluster:
+  enabled: false
+redis:
+  enabled: false
+
+persistence:
+  enabled: true
+  size: 5Gi
+  storageClass: sbs-default-retain
+
+resources:
+  requests: { cpu: 100m, memory: 128Mi }
+  limits:   { cpu: "1",  memory: 512Mi }
+
+service:
+  http: { type: ClusterIP, port: 3000 }
+  ssh:  { type: ClusterIP, port: 22 }
+
+actions:
+  enabled: false
+
+gitea:
+  admin:
+    existingSecret: gitea-admin
+  config:
+    server:
+      ROOT_URL: https://git.fxhnt.ai/
+      DOMAIN: git.fxhnt.ai
+      SSH_DOMAIN: git.fxhnt.ai
+      SSH_PORT: "22"            # clean git@git.fxhnt.ai clone URLs. Port 22 is free (nothing host-SSHes
+                               # the git node); :2222 retired, local remotes updated in Task 5 Step 2b.
+      DISABLE_SSH: "false"
+    database:
+      DB_TYPE: postgres
+      HOST: postgres.foxhunt.svc.cluster.local:5432
+      NAME: gitea
+      USER: gitea
+    service:
+      DISABLE_REGISTRATION: "true"
+    cache:
+      ADAPTER: memory
+  additionalConfigFromEnvs:
+    - name: GITEA__database__PASSWD
+      valueFrom:
+        secretKeyRef: { name: gitea-db, key: password }
+```
+
+- [ ] **Step 3: Install (internal-only — no public hostname yet)**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+helm install gitea gitea-charts/gitea -n foxhunt -f infra/k8s/gitea/values.yaml --wait --timeout 5m 2>&1 | tail -5
+```
+Expected: `STATUS: deployed`. (If `--wait` times out, continue; verify pod in Step 4.)
+
+- [ ] **Step 4: Verify Gitea is healthy**
+```bash
+kubectl get pods -n foxhunt | grep gitea
+kubectl run gitea-probe --rm -i --restart=Never -n foxhunt --image=curlimages/curl -- \
+  curl -s http://gitea-http.foxhunt.svc.cluster.local:3000/api/healthz
+```
+Expected: a JSON health blob with `"status": "pass"`. Pod `Running`.
+
+- [ ] **Step 5: Commit values (foxhunt repo)**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/gitea/values.yaml
+git commit -m "feat(infra): Gitea Helm values (external postgres, internal-only) — Phase 2B"
+```
+
+---
+
+## Task 3: Migrate both repos via port-forward + validate
+
+**Files:** none committed (git data operations).
+
+- [ ] **Step 1: Port-forward Gitea + create the repos via API**
+```bash
+kubectl port-forward -n foxhunt svc/gitea-http 3000:3000 >/tmp/gitea-pf.log 2>&1 & echo $! >/tmp/gitea-pf.pid
+sleep 3
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+for r in fxhnt foxhunt; do
+  curl -s -u "$GA:$GP" -X POST http://localhost:3000/api/v1/user/repos \
+    -H 'Content-Type: application/json' -d "{\"name\":\"$r\",\"private\":true}" -o /dev/null -w "$r: %{http_code}\n"
+done
+```
+Expected: `fxhnt: 201` and `foxhunt: 201`.
+
+- [ ] **Step 2: Mirror-push both repos from local clones**
+```bash
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+( cd /home/jgrusewski/Work/fxhnt   && git push --mirror "http://$GA:$GP@localhost:3000/$GA/fxhnt.git" 2>&1 | tail -2 )
+( cd /home/jgrusewski/Work/foxhunt && git push --mirror "http://$GA:$GP@localhost:3000/$GA/foxhunt.git" 2>&1 | tail -2 )
+```
+Expected: both report refs pushed (no error). foxhunt is 876 MB — may take a minute.
+
+- [ ] **Step 3: VALIDATE — clone back + diff top SHAs vs GitLab**
+```bash
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+for r in fxhnt foxhunt; do
+  gitea_sha=$(git ls-remote "http://$GA:$GP@localhost:3000/$GA/$r.git" HEAD | awk '{print $1}')
+  gitlab_sha=$(cd /home/jgrusewski/Work/$r && git ls-remote origin HEAD | awk '{print $1}')
+  [ "$gitea_sha" = "$gitlab_sha" ] && echo "$r: MATCH ($gitea_sha)" || echo "$r: MISMATCH gitea=$gitea_sha gitlab=$gitlab_sha"
+done
+```
+Expected: `fxhnt: MATCH` and `foxhunt: MATCH`. **STOP if any MISMATCH.**
+
+- [ ] **Step 4: Mark foxhunt archived (read-only)**
+```bash
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+curl -s -u "$GA:$GP" -X PATCH "http://localhost:3000/api/v1/repos/$GA/foxhunt" \
+  -H 'Content-Type: application/json' -d '{"archived":true}' -o /dev/null -w "archive foxhunt: %{http_code}\n"
+kill $(cat /tmp/gitea-pf.pid) 2>/dev/null
+```
+Expected: `archive foxhunt: 200`.
+
+---
+
+## Task 4: Move cockpit image to Scaleway Container Registry
+
+**Files:** Modify `~/Work/fxhnt/infra/argo/cockpit-build-deploy.yaml`,
+`~/Work/fxhnt/infra/k8s/orchestration/dagster.yaml` (+ dashboard manifest if separate).
+
+- [ ] **Step 1: Create the Scaleway registry pull/push secret**
+```bash
+kubectl create secret docker-registry scw-registry -n foxhunt \
+  --docker-server=rg.fr-par.scw.cloud \
+  --docker-username=nologin --docker-password="$SCW_SECRET_KEY" \
+  --dry-run=client -o yaml | kubectl apply -f -
+```
+Expected: secret `created`/`configured`. (Scaleway registry auth: username `nologin`, password = a
+Scaleway secret key.)
+
+- [ ] **Step 2: Repoint kaniko build → Scaleway registry**
+
+In `~/Work/fxhnt/infra/argo/cockpit-build-deploy.yaml`, change the kaniko args:
+```
+              --destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest \
+              --cache-repo=rg.fr-par.scw.cloud/bizworx/cache \
+```
+Remove the `--insecure-registry=…gitlab-registry…:5000` and `--skip-tls-verify-registry=…` lines
+(Scaleway is TLS). Mount the `scw-registry` dockerconfig for kaniko (replace the `gitlab-registry`
+dockerconfig volume): set the kaniko `DOCKER_CONFIG`/volume to the `scw-registry` secret at
+`/kaniko/.docker/config.json`.
+
+- [ ] **Step 3: Repoint the deploy image refs**
+
+In `~/Work/fxhnt/infra/k8s/orchestration/dagster.yaml` (and the dashboard manifest), change every
+`image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/fxhnt-cockpit:latest` →
+`image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`, and add under each pod spec:
+```yaml
+      imagePullSecrets:
+        - name: scw-registry
+```
+
+- [ ] **Step 4: Run one cockpit build → Scaleway + verify the image exists**
+```bash
+cd /home/jgrusewski/Work/fxhnt && ./scripts/argo-deploy-cockpit.sh 2>&1 | tail -8
+# verify the pushed image
+scw registry image list namespace-id=6561a90a-f4ba-4f44-b43f-be6e27cbdfca 2>/dev/null | grep -i fxhnt-cockpit || echo "(verify via Scaleway console)"
+```
+Expected: workflow `Succeeded`; `fxhnt-cockpit` image listed in the `bizworx` namespace.
+
+- [ ] **Step 5: Verify a rollout pulls from Scaleway**
+```bash
+kubectl rollout restart deploy/dagster deploy/fxhnt-dashboard -n foxhunt 2>/dev/null
+kubectl rollout status deploy/dagster -n foxhunt --timeout=180s
+kubectl get pods -n foxhunt -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | grep -i "rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit" | head
+```
+Expected: rollout succeeds; pods now reference the `rg.fr-par.scw.cloud/bizworx/...` image (no `ImagePullBackOff`).
+
+- [ ] **Step 6: Commit (fxhnt repo)**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add infra/argo/cockpit-build-deploy.yaml infra/k8s/orchestration/dagster.yaml
+git commit -m "feat(infra): cockpit image -> Scaleway Container Registry (Phase 2B)"
+```
+
+---
+
+## Task 5: Cut `git.fxhnt.ai` + SSH `:2222` over to Gitea
+
+**Files:** Modify `infra/k8s/gitlab/tailscale-proxy.yaml` (foxhunt repo). **CHECKPOINT before applying.**
+
+- [ ] **Step 1: Repoint nginx `git.fxhnt.ai:443` → Gitea**
+
+In `infra/k8s/gitlab/tailscale-proxy.yaml`, in the `server_name git.fxhnt.ai;` (`listen 443`) block,
+change:
+```
+            proxy_pass http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181;
+```
+to:
+```
+            proxy_pass http://gitea-http.foxhunt.svc.cluster.local:3000;
+```
+Delete the entire `listen 5050 ssl; server_name git.fxhnt.ai;` registry block (Scaleway registry is
+external — not proxied).
+
+- [ ] **Step 2: Repoint the socat SSH proxy to Gitea SSH on `:22` (drop `:2222`)**
+
+Nothing host-SSHes the git node, so use the standard port `22` and retire `:2222` entirely (clean
+`git@git.fxhnt.ai` URLs). In the `ssh-proxy` container args, change the listen port to `22` and the target
+to Gitea:
+```
+            - "TCP-LISTEN:22,fork,reuseaddr,sndbuf=1048576,rcvbuf=1048576"
+            - "TCP:gitea-ssh.foxhunt.svc.cluster.local:22,sndbuf=1048576,rcvbuf=1048576"
+```
+Update the container `ports:` entry from `containerPort: 2222` → `containerPort: 22`. Ensure the Tailscale
+node/proxy advertises port `22` (update the tailnet sidecar's exposed ports / `serve` config: `22`,
+replacing `2222`).
+
+- [ ] **Step 2b: Update local git remotes from `:2222` → default (port 22)**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git remote set-url origin ssh://git@git.fxhnt.ai/gitadmin/foxhunt.git
+cd /home/jgrusewski/Work/fxhnt   && git remote set-url origin ssh://git@git.fxhnt.ai/gitadmin/fxhnt.git
+cd /home/jgrusewski/Work/foxhunt && git remote -v | grep fetch
+```
+Expected: remotes now show `ssh://git@git.fxhnt.ai/...` (no `:2222`).
+
+- [ ] **Step 3: Apply + restart the proxy**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml
+kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
+kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s
+```
+Expected: rollout complete.
+
+- [ ] **Step 4: GATE — verify `git.fxhnt.ai` serves Gitea**
+```bash
+curl -sk https://git.fxhnt.ai/api/healthz | head -c 200; echo
+echo "--- SSH on :22 (git@git.fxhnt.ai) ---"; git ls-remote ssh://git@git.fxhnt.ai/gitadmin/fxhnt.git HEAD 2>&1 | head -1
+```
+Expected: Gitea healthz JSON; the `:22` ssh ls-remote returns the HEAD sha (proves Gitea web + SSH on the
+canonical name). **STOP + rollback (revert this file, re-apply) if either fails.**
+
+- [ ] **Step 5: Commit (foxhunt repo)**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/gitlab/tailscale-proxy.yaml
+git commit -m "feat(infra): cut git.fxhnt.ai + SSH :2222 over to Gitea; drop :5050 registry (Phase 2B)"
+```
+
+---
+
+## Task 6: Wire Gitea push webhook → Argo Events auto-deploy
+
+**Files:** Create `infra/k8s/argo/events/gitea-push-eventsource.yaml` +
+`infra/k8s/argo/events/gitea-deploy-sensor.yaml` (foxhunt repo).
+
+- [ ] **Step 1: Create the webhook eventsource**
+
+Create `infra/k8s/argo/events/gitea-push-eventsource.yaml`:
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: EventSource
+metadata:
+  name: gitea-push
+  namespace: foxhunt
+spec:
+  service:
+    ports:
+      - port: 12000
+        targetPort: 12000
+  webhook:
+    fxhnt-push:
+      port: "12000"
+      endpoint: /push
+      method: POST
+```
+
+- [ ] **Step 2: Create the sensor that submits the cockpit workflow**
+
+Create `infra/k8s/argo/events/gitea-deploy-sensor.yaml`:
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Sensor
+metadata:
+  name: gitea-deploy
+  namespace: foxhunt
+spec:
+  dependencies:
+    - name: push
+      eventSourceName: gitea-push
+      eventName: fxhnt-push
+      filters:
+        data:
+          - path: body.ref
+            type: string
+            value: ["refs/heads/main", "refs/heads/master"]
+  triggers:
+    - template:
+        name: submit-cockpit
+        argoWorkflow:
+          operation: submit
+          source:
+            resource:
+              apiVersion: argoproj.io/v1alpha1
+              kind: Workflow
+              metadata:
+                generateName: fxhnt-cockpit-
+                namespace: foxhunt
+              spec:
+                workflowTemplateRef:
+                  name: fxhnt-cockpit
+```
+
+- [ ] **Step 3: Apply + register the webhook in Gitea**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+kubectl apply -f infra/k8s/argo/events/gitea-push-eventsource.yaml -f infra/k8s/argo/events/gitea-deploy-sensor.yaml
+kubectl get pods -n foxhunt | grep -iE "gitea-push|gitea-deploy"
+# register webhook (in-cluster URL) on the fxhnt repo
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+kubectl port-forward -n foxhunt svc/gitea-http 3000:3000 >/tmp/gitea-pf.log 2>&1 & echo $! >/tmp/gitea-pf.pid; sleep 3
+curl -s -u "$GA:$GP" -X POST "http://localhost:3000/api/v1/repos/$GA/fxhnt/hooks" -H 'Content-Type: application/json' -d '{
+  "type":"gitea","active":true,"events":["push"],
+  "config":{"url":"http://gitea-push-eventsource-svc.foxhunt.svc.cluster.local:12000/push","content_type":"json"}
+}' -o /dev/null -w "hook: %{http_code}\n"
+kill $(cat /tmp/gitea-pf.pid) 2>/dev/null
+```
+Expected: eventsource + sensor pods `Running`; `hook: 201`.
+
+- [ ] **Step 4: GATE — test push triggers a deploy**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git commit --allow-empty -m "test: trigger Gitea->Argo cockpit deploy"
+git push origin main   # origin already = git.fxhnt.ai (now Gitea)
+sleep 10
+kubectl get wf -n foxhunt --sort-by=.metadata.creationTimestamp 2>/dev/null | grep fxhnt-cockpit | tail -2
+```
+Expected: a new `fxhnt-cockpit-*` workflow appears (triggered by the push). **If none fires**, check the
+eventsource pod logs; fallback is the manual `./scripts/argo-deploy-cockpit.sh` (still works).
+
+- [ ] **Step 5: Commit (foxhunt repo)**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/argo/events/gitea-push-eventsource.yaml infra/k8s/argo/events/gitea-deploy-sensor.yaml
+git commit -m "feat(infra): Gitea push webhook -> Argo Events cockpit auto-deploy (Phase 2B)"
+```
+
+---
+
+## Task 7: Final verification
+
+- [ ] **Step 1: Footprint + health summary**
+```bash
+echo "=== Gitea (new) ==="; kubectl top pods -n foxhunt 2>/dev/null | grep -i gitea
+echo "=== cockpit healthy on Scaleway image ==="; curl -sk -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
+echo "=== git.fxhnt.ai serves Gitea ==="; curl -sk https://git.fxhnt.ai/api/healthz | head -c 80; echo
+echo "=== GitLab still running (fallback for 2C) ==="; kubectl get pods -n foxhunt | grep -c gitlab | xargs echo "gitlab pods:"
+```
+Expected: Gitea ≤~300 MiB; dashboard 200; git.fxhnt.ai healthz pass; GitLab pods still present.
+
+---
+
+## Rollback
+- **Cutover (Task 5):** `git checkout infra/k8s/gitlab/tailscale-proxy.yaml` + `kubectl apply` + rollout
+  restart → `git.fxhnt.ai`/SSH back to GitLab.
+- **Registry (Task 4):** revert the fxhnt manifests + rollout restart → pods pull the GitLab image again
+  (still present until 2C).
+- **Gitea:** `helm uninstall gitea -n foxhunt`; drop the `gitea` DB. GitLab was never touched.
+
+---
+
+## Acceptance criteria (from spec)
+- Gitea running on existing postgres, healthz pass, ≤~300 MiB. ✅ Task 2, Task 7
+- Both repos in Gitea, SHAs match GitLab, foxhunt archived. ✅ Task 3
+- Cockpit builds to + deploys from Scaleway registry; dashboard 200. ✅ Task 4, Task 7
+- `git.fxhnt.ai` (HTTPS + SSH :2222) serves Gitea; remotes unchanged. ✅ Task 5
+- Push to fxhnt main auto-triggers a cockpit deploy. ✅ Task 6
+- GitLab still running as fallback. ✅ Task 7
+```

docs/superpowers/plans/2026-06-21-relocate-iac-to-fxhnt.md

@@ -0,0 +1,206 @@
+# Relocate Platform IaC foxhunt → fxhnt + Archive foxhunt — Implementation Plan (Phase 2D)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> Spans TWO repos + a production Terraform state gate; the `terragrunt plan = No changes` check is the
+> safety pin. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Move active platform IaC (terraform `live`+`modules`, platform k8s manifests) from `foxhunt`
+into `fxhnt`, preserving TF state keys (no migration), then archive `foxhunt`.
+
+**Architecture:** Copy into fxhnt preserving the `live/production/<module>` structure → state keys stay
+identical → `terragrunt plan = No changes` proves it. Then `git rm` from foxhunt, commit/push both,
+archive foxhunt. Cluster untouched throughout (source-of-truth move only).
+
+**Tech Stack:** terragrunt + OpenTofu (Scaleway s3 backend), kubectl manifests, git, Gitea.
+
+**Repos:** source `~/Work/foxhunt`, dest `~/Work/fxhnt`. **Spec:**
+`docs/superpowers/specs/2026-06-21-relocate-iac-to-fxhnt-design.md`
+
+---
+
+## Shared env (terragrunt steps)
+```bash
+export TG_TF_PATH=tofu TERRAGRUNT_TFPATH=tofu
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
+export AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY" AWS_REGION=fr-par
+```
+
+---
+
+## Task 0: Branches
+```bash
+cd /home/jgrusewski/Work/fxhnt   && git checkout master && git checkout -b chore/relocate-iac
+cd /home/jgrusewski/Work/foxhunt && git checkout main   && git checkout -b chore/relocate-iac-remove
+```
+
+---
+
+## Task 1: Collision pre-flight (must be empty)
+```bash
+cd /home/jgrusewski/Work/foxhunt
+for d in k8s/network-policies k8s/services k8s/jobs k8s/storage; do
+  echo "--- $d ---"; comm -12 <(ls infra/$d 2>/dev/null|sort) <(ls ~/Work/fxhnt/infra/$d 2>/dev/null|sort)
+done
+```
+Expected: no filenames printed. **STOP + resolve if any collision appears.**
+
+---
+
+## Task 2: Copy Terraform (live + modules) into fxhnt, structure preserved
+
+- [ ] **Step 1: rsync source files (exclude generated/cached, keep .terraform.lock.hcl)**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+rsync -a --exclude '.terraform/' --exclude 'backend.tf' --exclude 'provider.tf' --exclude '*.tfstate*' \
+  infra/live infra/modules /home/jgrusewski/Work/fxhnt/infra/
+find /home/jgrusewski/Work/fxhnt/infra/live /home/jgrusewski/Work/fxhnt/infra/modules -name '*.hcl' -o -name '*.tf' | sort
+```
+Expected: `fxhnt/infra/live/production/{root.hcl,kapsule,public-gateway,dns,block-storage}` +
+`fxhnt/infra/modules/{kapsule,public-gateway,dns,block-storage}` present.
+
+- [ ] **Step 2: Ensure fxhnt .gitignore covers terragrunt-generated files**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+grep -qE '^backend\.tf' .gitignore || printf '\n# terragrunt-generated\n**/.terraform/\n**/backend.tf\n**/provider.tf\n*.tfstate\n*.tfstate.*\n' >> .gitignore
+tail -7 .gitignore
+```
+
+- [ ] **Step 3: GATE — terragrunt plan = No changes for all 4 modules (from fxhnt)**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+for m in kapsule public-gateway dns block-storage; do
+  echo "=== $m ==="
+  ( cd infra/live/production/$m && terragrunt init -reconfigure -input=false -no-color >/dev/null 2>&1 && \
+    terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head -1 )
+done
+```
+Expected: every module prints `No changes.` — proves state keys are unchanged (no migration). **STOP if any
+module shows a plan diff or Error** (structure not preserved → do not proceed).
+
+---
+
+## Task 3: Copy platform k8s manifests + the one platform script into fxhnt
+
+- [ ] **Step 1: rsync platform k8s dirs (everything except gpu-taint.yaml) + namespace.yaml**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+for d in argo cert-manager databases dns gitea gitlab jobs mattermost minio monitoring network-policies secrets services stalwart tailscale training storage; do
+  rsync -a "infra/k8s/$d" /home/jgrusewski/Work/fxhnt/infra/k8s/
+done
+cp infra/k8s/namespace.yaml /home/jgrusewski/Work/fxhnt/infra/k8s/
+mkdir -p /home/jgrusewski/Work/fxhnt/infra/scripts && cp infra/scripts/generate-minio-tls.sh /home/jgrusewski/Work/fxhnt/infra/scripts/
+echo "=== fxhnt infra/k8s now ==="; ls /home/jgrusewski/Work/fxhnt/infra/k8s/
+```
+Expected: fxhnt/infra/k8s/ now contains the platform dirs (merged with fxhnt's existing
+jobs/network-policies/services/storage/orchestration) + namespace.yaml. `gpu-taint.yaml` NOT copied.
+
+- [ ] **Step 2: Sanity — no remaining hardcoded foxhunt-repo paths in moved manifests**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+grep -rIn "Work/foxhunt\|foxhunt/infra\|root/foxhunt" infra/k8s/ infra/live/ infra/modules/ 2>/dev/null | grep -v "foxhunt.svc.cluster.local\|namespace: foxhunt\|name: foxhunt\|cluster_name\|foxhunt-tfstate\|foxhunt-gitlab\|foxhunt-training\|foxhunt-backups\|foxhunt-sccache\|foxhunt-binaries\|foxhunt-agent" | head
+echo "(above should be empty — 'foxhunt' as the k8s namespace / SCW resource names is expected + correct)"
+```
+Expected: empty (the namespace `foxhunt` + Scaleway resource names legitimately stay).
+
+---
+
+## Task 4: Verify cluster unaffected + cockpit still deploys from fxhnt
+
+- [ ] **Step 1: terragrunt still clean from fxhnt + platform health**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+curl -sk -m 10 -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
+curl -sk -m 10 https://git.fxhnt.ai/api/healthz 2>/dev/null | grep -o '"status": "pass"'
+```
+Expected: dashboard 200; gitea pass. (No cluster change was made — this confirms nothing regressed.)
+
+---
+
+## Task 5: Commit fxhnt (additions) + push
+
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add infra/ .gitignore
+git commit -m "feat(infra): adopt platform IaC from foxhunt — terraform live+modules + k8s manifests (Phase 2D)
+
+fxhnt is now the single home for platform IaC. Terraform state keys unchanged
+(live/production structure preserved) — terragrunt plan = No changes, no migration.
+Includes k8s platform manifests (cert-manager, databases, dns, gitea, minio, monitoring,
+network-policies, tailscale, stalwart, mattermost, services, storage, jobs, training,
+secrets, argo, gitlab proxy) + generate-minio-tls.sh.
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/relocate-iac 2>&1 | tail -2
+```
+
+---
+
+## Task 6: Remove the moved paths from foxhunt + commit + push
+
+- [ ] **Step 1: git rm the relocated IaC from foxhunt (keep gpu-taint.yaml, docker, Rust scripts)**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git rm -r -q infra/live infra/modules
+for d in argo cert-manager databases dns gitea gitlab jobs mattermost minio monitoring network-policies secrets services stalwart tailscale training storage; do git rm -r -q "infra/k8s/$d"; done
+git rm -q infra/k8s/namespace.yaml infra/scripts/generate-minio-tls.sh
+echo "=== foxhunt infra/ remnant (Rust-only) ==="; find infra -maxdepth 2 | sort | head -30
+```
+Expected: `infra/k8s/gpu-taint.yaml` + `infra/docker/*` + `infra/scripts/{build-and-push,harden,smoke-test,train,...}` remain; live/modules + platform k8s gone.
+
+- [ ] **Step 2: Commit + push foxhunt**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git commit -q -m "chore(infra): relocate platform IaC to fxhnt (Phase 2D)
+
+Platform terraform (live+modules) + k8s manifests moved to the fxhnt repo (now the IaC
+home). Rust-only remnants kept here (docker build images, gpu-taint, Rust scripts).
+foxhunt is being archived; history retained. See fxhnt for live platform IaC."
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/relocate-iac-remove 2>&1 | tail -2
+```
+
+---
+
+## Task 7: Merge both branches, then archive foxhunt
+
+- [ ] **Step 1: Merge fxhnt branch → master + push**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git checkout master && git merge --no-ff chore/relocate-iac -m "Merge: adopt platform IaC from foxhunt (Phase 2D)" 2>&1 | tail -2
+git branch -d chore/relocate-iac
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1
+```
+
+- [ ] **Step 2: Merge foxhunt branch → main + push**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git checkout main && git merge --no-ff chore/relocate-iac-remove -m "Merge: relocate platform IaC to fxhnt (Phase 2D)" 2>&1 | tail -2
+git branch -d chore/relocate-iac-remove
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin main 2>&1 | tail -1
+```
+
+- [ ] **Step 3: Final re-verify terragrunt from fxhnt (post-merge) + archive foxhunt in Gitea**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+( cd infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:" | head -1 )
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+curl -sk -u "$GA:$GP" -X PATCH "https://git.fxhnt.ai/api/v1/repos/$GA/foxhunt" -H 'Content-Type: application/json' -d '{"archived":true}' -o /dev/null -w "archive foxhunt: %{http_code}\n"
+```
+Expected: kapsule `No changes`; `archive foxhunt: 200`.
+
+---
+
+## Rollback
+- Before Task 6 (foxhunt rm): just delete `fxhnt/infra/{live,modules}` + the copied k8s dirs; nothing lost
+  (foxhunt still authoritative).
+- After: both repos retain history; un-archive foxhunt + `git revert` if needed. TF state never moved.
+
+## Acceptance criteria (from spec)
+- `fxhnt/infra/{live,modules}` present; `terragrunt plan = No changes` all 4 modules from fxhnt. ✅ T2, T7
+- Platform k8s manifests + generate-minio-tls.sh in fxhnt; no collisions lost. ✅ T1, T3
+- Moved paths `git rm`'d from foxhunt; both repos committed + pushed. ✅ T5, T6, T7
+- dashboard 200 + cockpit deploy unaffected. ✅ T4
+- foxhunt archived in Gitea; Rust code + history intact. ✅ T7

docs/superpowers/plans/2026-06-21-tfstate-to-scaleway.md

@@ -0,0 +1,264 @@
+# Migrate Terraform State to Scaleway Object Storage — Implementation Plan (Phase 2A)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> **Do NOT run subagent-driven** — state migration is irreversible-adjacent and each module needs a
+> human-confirmed `plan = No changes` gate. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Move the 4 production modules' Terraform state from the self-hosted GitLab http backend to
+external Scaleway Object Storage (bucket `foxhunt-tfstate`), with native S3 lockfile locking.
+
+**Architecture:** Back up all 4 states from GitLab first; flip the single `remote_state` block in
+`infra/live/production/root.hcl` from `http`→`s3`; migrate each module in-place with `terragrunt init
+-migrate-state`; gate each on `terragrunt plan = No changes`. GitLab state is left intact as a fallback
+(deleted later in 2C).
+
+**Tech Stack:** terragrunt v0.77.20 + OpenTofu v1.11.5, Scaleway Object Storage (S3-compatible), AWS s3
+backend.
+
+**Spec:** `docs/superpowers/specs/2026-06-21-tfstate-to-scaleway-design.md`
+
+---
+
+## Shared env (export in EVERY shell that runs terragrunt during this plan)
+The s3 backend reads **`AWS_*`** env vars; the Scaleway provider reads **`SCW_*`**; reading the OLD
+GitLab state during `-migrate-state` needs **`TF_HTTP_*`**. Export all three sets:
+```bash
+cd /home/jgrusewski/Work/foxhunt
+export TG_TF_PATH=tofu TERRAGRUNT_TFPATH=tofu
+# Scaleway provider creds
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
+# s3 backend creds (SAME Scaleway keys, AWS-style names)
+export AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY"
+export AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY"
+export AWS_REGION=fr-par
+# OLD GitLab http backend creds (needed to READ existing state during migrate)
+export TF_HTTP_USERNAME=root
+export TF_HTTP_PASSWORD=$(kubectl get secret gitlab-pat -n foxhunt -o jsonpath='{.data.token}'|base64 -d)
+echo "env: SCW=${SCW_ACCESS_KEY:0:6}… AWS=${AWS_ACCESS_KEY_ID:0:6}… PAT=${TF_HTTP_PASSWORD:0:4}…"
+```
+
+---
+
+## Task 0: Branch
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/tfstate-to-scaleway
+```
+
+---
+
+## Task 1: Verify prerequisites + back up all 4 GitLab states
+
+**Files:** none (read-only + /tmp backups)
+
+- [ ] **Step 1: Confirm the target bucket exists + is empty-ish**
+```bash
+scw object bucket list 2>/dev/null | grep foxhunt-tfstate || echo "MISSING — create: scw object bucket create name=foxhunt-tfstate region=fr-par"
+```
+Expected: a line containing `foxhunt-tfstate`. If MISSING, run the create shown, then re-check.
+
+- [ ] **Step 2: Confirm tooling versions**
+```bash
+tofu version | head -1; terragrunt --version | head -1
+```
+Expected: OpenTofu `v1.11.x` (≥1.10 for `use_lockfile`), terragrunt `v0.77.x`.
+
+- [ ] **Step 3: Back up each module's CURRENT (GitLab) state** — run with shared env exported
+```bash
+for m in kapsule public-gateway dns block-storage; do
+  ( cd infra/live/production/$m && terragrunt state pull > /tmp/tfstate-backup-$m.tfstate 2>/dev/null )
+  echo "$m: $(python3 -c "import json;d=json.load(open('/tmp/tfstate-backup-$m.tfstate'));print('serial',d.get('serial'),'resources',len(d.get('resources',[])))" 2>/dev/null || echo 'EMPTY/ERROR')"
+done
+```
+Expected: each module prints a non-zero `resources` count (e.g. kapsule ~5+, dns/public-gateway/block-storage ≥1). **STOP if any backup is EMPTY/ERROR** — that module's GitLab state isn't readable; do not migrate it blind.
+
+- [ ] **Step 4: Record current state serials for post-migration comparison**
+```bash
+grep -H '"serial"' /tmp/tfstate-backup-*.tfstate 2>/dev/null || for m in kapsule public-gateway dns block-storage; do echo "$m $(python3 -c "import json;print(json.load(open('/tmp/tfstate-backup-$m.tfstate'))['serial'])")"; done
+```
+Expected: a serial number per module. Keep this output — the migrated state should have the same resource set.
+
+---
+
+## Task 2: Switch the backend in root.hcl (http → s3)
+
+**Files:** Modify `infra/live/production/root.hcl`
+
+- [ ] **Step 1: Replace the `remote_state` block**
+
+In `infra/live/production/root.hcl`, replace the entire existing `remote_state { ... }` block (the `backend = "http"`
+GitLab block) with:
+```hcl
+# Remote state in Scaleway Object Storage (bucket foxhunt-tfstate, fr-par) — migrated off GitLab 2026-06-21.
+# s3 backend reads AWS_* env vars (set to the Scaleway access/secret keys); SCW_* drives the provider.
+remote_state {
+  backend = "s3"
+  generate = {
+    path      = "backend.tf"
+    if_exists = "overwrite"
+  }
+  config = {
+    bucket = "foxhunt-tfstate"
+    key    = "${path_relative_to_include()}/terraform.tfstate"
+    region = "fr-par"
+
+    endpoints = {
+      s3 = "https://s3.fr-par.scw.cloud"
+    }
+
+    # Scaleway S3-compat: skip AWS-specific preflight calls
+    skip_credentials_validation = true
+    skip_region_validation      = true
+    skip_requesting_account_id  = true
+    skip_metadata_api_check     = true
+
+    # OpenTofu-native lock (no DynamoDB)
+    use_lockfile = true
+  }
+}
+```
+Leave the `generate "provider"` block and `inputs`/`locals` untouched.
+
+- [ ] **Step 2: Sanity-check the file parses**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+grep -A2 'backend = "s3"' infra/live/production/root.hcl && grep -c 'backend = "http"' infra/live/production/root.hcl
+```
+Expected: shows the `s3` backend line; the `backend = "http"` count is `0`.
+
+---
+
+## Task 3: Migrate `kapsule` (canary) + lock smoke test
+
+**Files:** none (state operation)
+
+- [ ] **Step 1: Migrate state (with shared env exported)**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule
+terragrunt init -migrate-state -force-copy -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "Successfully configured|migrat|Error|s3" | head
+```
+Expected: a line like `Terraform has been successfully migrated to the "s3" backend!` (or `Successfully
+configured the backend "s3"`). **If it errors on checksum** (Scaleway quirk), add `skip_s3_checksum = true`
+to the `config` block in `infra/live/production/root.hcl` and re-run this step.
+
+- [ ] **Step 2: GATE — plan must show No changes**
+```bash
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head
+```
+Expected: `No changes. Your infrastructure matches the configuration.`
+**STOP if it shows any add/change/destroy** — the migrated state doesn't match reality. Revert (Task 7
+rollback) and investigate before continuing.
+
+- [ ] **Step 3: Confirm the state object landed in the bucket**
+```bash
+scw object bucket list 2>/dev/null >/dev/null  # ensure scw configured
+aws --endpoint-url https://s3.fr-par.scw.cloud s3 ls s3://foxhunt-tfstate/kapsule/ 2>/dev/null || \
+  echo "(aws cli n/a — verify via: mc ls on a scw alias, or scw object)"
+```
+Expected: `terraform.tfstate` listed under `kapsule/`. (If `aws` CLI is absent, this is best-effort;
+the `plan = No changes` in Step 2 already proves the s3 backend is the live source.)
+
+- [ ] **Step 4: Lock smoke test** — confirm `use_lockfile` works on Scaleway
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule
+# A refresh acquires + releases the lock; success + no leftover .tflock proves locking works
+terragrunt apply -refresh-only -auto-approve -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "Apply complete|Error|lock" | head
+aws --endpoint-url https://s3.fr-par.scw.cloud s3 ls s3://foxhunt-tfstate/kapsule/ 2>/dev/null | grep -i tflock && echo "WARN: orphan lockfile" || echo "OK: no orphan lockfile"
+```
+Expected: `Apply complete!` and `OK: no orphan lockfile`. **If locking errors**, fall back to
+`use_lockfile = false` (single-maintainer is safe) — document the change in the commit.
+
+---
+
+## Task 4: Migrate the remaining 3 modules (`public-gateway`, `dns`, `block-storage`)
+
+**Files:** none (state operations). Same procedure as Task 3 Steps 1-2, per module, each gated.
+
+- [ ] **Step 1: Migrate + gate `public-gateway`**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/public-gateway
+terragrunt init -migrate-state -force-copy -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "migrat|successfully|Error" | head
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head
+```
+Expected: migration success line, then `No changes`. **STOP on any drift.**
+
+- [ ] **Step 2: Migrate + gate `dns`**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/dns
+terragrunt init -migrate-state -force-copy -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "migrat|successfully|Error" | head
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head
+```
+Expected: migration success line, then `No changes`. **STOP on any drift.**
+
+- [ ] **Step 3: Migrate + gate `block-storage`**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/block-storage
+terragrunt init -migrate-state -force-copy -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "migrat|successfully|Error" | head
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head
+```
+Expected: migration success line, then `No changes`. **STOP on any drift.**
+
+---
+
+## Task 5: Final verification + commit
+
+**Files:** Modify (commit) `infra/live/production/root.hcl`; also commit generated `backend.tf` only if not gitignored.
+
+- [ ] **Step 1: Re-plan all 4 modules from a clean shell** (proves the s3 backend is authoritative)
+```bash
+cd /home/jgrusewski/Work/foxhunt
+for m in kapsule public-gateway dns block-storage; do
+  echo "=== $m ==="
+  ( cd infra/live/production/$m && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head -1 )
+done
+```
+Expected: every module prints `No changes.`
+
+- [ ] **Step 2: Confirm all 4 state objects exist in the bucket**
+```bash
+for m in kapsule public-gateway dns block-storage; do
+  aws --endpoint-url https://s3.fr-par.scw.cloud s3 ls s3://foxhunt-tfstate/$m/ 2>/dev/null | grep -q terraform.tfstate && echo "$m: state present" || echo "$m: NOT FOUND (verify manually)"
+done
+```
+Expected: `state present` for all 4 (or manual confirmation if `aws` CLI absent — `plan = No changes` is the real proof).
+
+- [ ] **Step 3: Verify GitLab state still intact (fallback preserved)** — should NOT be deleted in 2A
+```bash
+echo "GitLab TF state is intentionally left in place as a 2A fallback; deleted in Phase 2C."
+```
+
+- [ ] **Step 4: Commit**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git add infra/live/production/root.hcl
+# backend.tf is terragrunt-generated; add only if tracked (usually gitignored)
+git status --porcelain infra/live/production/*/backend.tf 2>/dev/null
+git commit -m "chore(infra): migrate Terraform state GitLab -> Scaleway Object Storage (Phase 2A)
+
+All 4 modules (kapsule/public-gateway/dns/block-storage) migrated to s3 backend
+(bucket foxhunt-tfstate, fr-par, use_lockfile). plan=No changes verified per module.
+GitLab http state left intact as fallback (removed in 2C).
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Rollback (if any module's post-migration plan shows drift)
+1. Restore the `http` backend block in `infra/live/production/root.hcl` (git checkout the file).
+2. Re-attach GitLab state: in each affected module, `terragrunt init -migrate-state -force-copy` (copies
+   back s3→http) OR `terragrunt init -reconfigure` (the GitLab state was never deleted).
+3. If state is corrupted, restore from `/tmp/tfstate-backup-<module>.tfstate` via `terragrunt state push`.
+4. The GitLab copy is untouched throughout 2A, so reverting is always possible.
+
+---
+
+## Acceptance criteria (from spec)
+- `infra/live/production/root.hcl` uses the Scaleway s3 backend; committed. ✅ Task 2 + Task 5.4
+- All 4 modules: `terragrunt plan = No changes` on the new backend. ✅ Task 3.2, Task 4, Task 5.1
+- State objects at `foxhunt-tfstate/<module>/terraform.tfstate`. ✅ Task 3.3, Task 5.2
+- Lock smoke test passes. ✅ Task 3.4
+- GitLab http state intact (fallback). ✅ Task 5.3

docs/superpowers/specs/2026-06-21-decommission-gitlab-design.md

@@ -0,0 +1,69 @@
+# Decommission GitLab (compute) — Design (Phase 2C)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** foxhunt
+**Part of:** Phase 2 ([[project_phase2_gitlab_to_gitea]]). 2A (TF state→Scaleway) + 2B (Gitea + Scaleway
+registry + Argo webhook) DONE. This is **2C**. 2D (IaC relocation foxhunt→fxhnt) follows.
+
+## Motivation
+
+GitLab is fully replaced: git→Gitea (`git.fxhnt.ai`, SHA-verified), Terraform state→Scaleway Object
+Storage, container images→Scaleway Container Registry, CI/CD→Argo (Gitea webhook). It has **zero live
+consumers** but still runs 12 pods / ~4.6 GiB RAM. 2C removes GitLab's compute and retains its data as a
+cold backup (per the data-safety preference — migrated copies exist and are verified, but keep the
+originals through a grace period).
+
+## Scope — remove (compute only)
+
+1. **Helm uninstall** (foxhunt ns): `gitlab` (chart gitlab-9.9.3, rev 25) + `foxhunt-agent`
+   (gitlab-agent). Frees ~4.6 GiB RAM / 12 pods.
+2. **Dead Argo Events**: `kubectl delete eventsource gitlab-push` + `sensor ci-pipeline-trigger`
+   (Rust-era GitLab CI triggers; the gitea-push/gitea-deploy pair replaced them in 2B).
+3. **nginx proxy `:5050` registry block** + `containerPort: 5050` in `infra/k8s/gitlab/tailscale-proxy.yaml`
+   (the GitLab container registry face; images are on Scaleway now).
+4. **Repo manifest** `infra/k8s/gitlab/values.yaml` (GitLab Helm values — chart no longer deployed).
+
+## 🛑 PRESERVE (cold backup — delete in a later grace-period pass)
+- **gitaly PVC** `repo-data-gitlab-gitaly-0` (50 GiB, old git repos). StatefulSet PVCs survive
+  `helm uninstall` by default — no action needed to keep it.
+- **MinIO buckets**: `foxhunt-gitlab-artifacts` (holds the superseded TF-state fallback),
+  `foxhunt-gitlab-packages`, `foxhunt-gitlab-registry`.
+- **Scaleway buckets**: `foxhunt-gitlab-artifacts`, `foxhunt-gitlab-registry`.
+- **All `gitlab-*` secrets** (e.g. `gitlab-pat`, `gitlab-s3-credentials`, `gitlab-tls-cert`).
+- **`infra/k8s/gitlab/tailscale-proxy.yaml`** — KEEP: it is now the **Gitea** proxy (serves
+  `git.fxhnt.ai` + all `*.fxhnt.ai`). Only its `:5050` block is removed.
+
+## Execution order (safety-critical)
+
+1. **Proxy first:** remove the `:5050` server block + `containerPort: 5050` from `tailscale-proxy.yaml`;
+   `kubectl apply` + rollout restart; **verify `git.fxhnt.ai` + `dashboard.fxhnt.ai` still 200**.
+   *Rationale:* nginx resolves `proxy_pass` upstream hostnames at startup. If GitLab is uninstalled while
+   the `:5050` block still references `gitlab-registry.foxhunt.svc`, nginx fails to start on its next
+   restart → **all `*.fxhnt.ai` go down**. Removing the dead upstream first defuses this.
+2. **helm uninstall** `gitlab` then `foxhunt-agent`.
+3. **Delete dead Argo Events** (`gitlab-push`, `ci-pipeline-trigger`).
+4. **Remove `values.yaml`** from the repo; commit.
+5. **Verify** (see acceptance).
+
+## Risks / safety
+
+- **Proxy outage** if `:5050` not removed before uninstall → mitigated by step 1 + the 200-check gate.
+- **Irreversible**: helm uninstall. Mitigated: all data retained (gitaly PVC + buckets + secrets), so a
+  reinstall could re-attach. Migrated copies (Gitea/Scaleway) are the primary going forward.
+- **Shared resource**: confirm the `gitlab` release owns only GitLab objects (it does — namespaced release).
+  The `postgres`/`minio`/`tailscale-proxy` are separate and stay.
+- **No `.dbn`/market data** in any GitLab bucket (git artifacts/registry/packages only) — unaffected.
+
+## Out of scope
+- Deleting the retained GitLab data (gitaly PVC + buckets + secrets) — a later grace-period pass.
+- 2D IaC relocation foxhunt→fxhnt (separate spec); re-archiving the foxhunt repo happens there.
+
+## Acceptance criteria
+- `helm list -n foxhunt` shows no `gitlab` / `foxhunt-agent`; GitLab pods gone (~4.6 GiB RAM reclaimed).
+- `git.fxhnt.ai` + `dashboard.fxhnt.ai` return 200; a push to fxhnt master still auto-deploys via
+  Gitea→Argo.
+- `gitlab-push` eventsource + `ci-pipeline-trigger` sensor deleted; `gitea-push`/`gitea-deploy` still present.
+- `:5050` block gone from the proxy; proxy healthy.
+- **Cold backup intact:** gitaly PVC + all GitLab MinIO/Scaleway buckets + `gitlab-*` secrets still present.
+- `infra/k8s/gitlab/values.yaml` removed; `tailscale-proxy.yaml` kept; committed.

docs/superpowers/specs/2026-06-21-gitea-replace-gitlab-design.md

@@ -0,0 +1,114 @@
+# Stand up Gitea + migrate repos + move cockpit registry — Design (Phase 2B)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** foxhunt (IaC current home; relocation to fxhnt is Phase 2D)
+**Part of:** Phase 2 (GitLab → Gitea consolidation, [[project_phase2_gitlab_to_gitea]]). 2A (TF state →
+Scaleway) DONE. This is **2B**. 2C (GitLab removal) and 2D (IaC relocation) follow.
+
+## Motivation
+
+Self-hosted GitLab uses ~6.5 GiB RAM across 17 pods (webservice 1.96 GiB + 3× sidekiq ~3 GiB + gitaly +
+registry + prometheus + …). The active work is the Python `fxhnt` fund; GitLab is Rust-era overkill.
+Replace it with **Gitea** (~150–250 MiB, chart + existing Postgres) for git hosting, and move the
+**cockpit container image** to **Scaleway Container Registry** so GitLab can be removed in 2C without
+breaking the live cockpit deploy.
+
+## Decisions (from brainstorming)
+
+- **Gitea install:** `gitea/gitea` Helm chart. Subcharts `postgresql`/`redis-cluster`/`redis`/`memcached`
+  **disabled**. Built-in **Actions disabled** (Argo does CI/CD). Cache = memory (single replica).
+- **Database:** reuse the **existing in-cluster `postgres`** (svc `postgres:5432`, already has a backup
+  cronjob). Create a dedicated `gitea` database + role.
+- **Repos:** migrate **both** — `fxhnt` (active) and `foxhunt` (876 MB, flagged **archived/read-only**) —
+  via `git push --mirror` from existing local clones.
+- **Canonical hostname stays `git.fxhnt.ai`** (no permanent `gitea.fxhnt.ai`, no remote-URL churn).
+  Gitea comes up internal-only; validate over a port-forward; then cut `git.fxhnt.ai` (HTTPS) + SSH over
+  to Gitea in one switch. **SSH on `:22`** (nothing host-SSHes the git node, so the standard port is free);
+  clone URLs become the clean `git@git.fxhnt.ai` and `:2222` is retired (local remotes updated once).
+  GitLab keeps running (unaddressed) as the 2C-removal fallback.
+- **Container registry:** cockpit image → **Scaleway Container Registry** (`rg.fr-par.scw.cloud/bizworx/`,
+  namespace exists). Retire the in-cluster GitLab registry. External/managed, no registry pod.
+- **Argo integration:** Gitea push webhook → Argo Events webhook eventsource → sensor → submit
+  `wftmpl/fxhnt-cockpit`. Replaces the old `gitlab-push-eventsource`.
+
+## Components
+
+1. **`infra/k8s/gitea/` (new):** Helm values `values.yaml` + a thin install (release `gitea`, ns
+   `foxhunt`). Storage: 1 PVC `sbs-default-retain` ~5 Gi (foxhunt 876 MB + fxhnt + headroom). Resources:
+   requests 128 Mi/100m, limits 512 Mi. Admin user + secrets from a k8s secret (NOT committed).
+   Gitea config: `server.SSH_PORT`/`SSH_LISTEN_PORT`, `server.ROOT_URL=https://git.fxhnt.ai/`,
+   `server.DISABLE_SSH=false`, DB type `postgres` → `postgres:5432/gitea`.
+2. **Postgres prep:** one-time `psql` on the existing `postgres` pod — `CREATE ROLE gitea …; CREATE
+   DATABASE gitea OWNER gitea;`. Credentials in a k8s secret consumed by the Gitea release.
+3. **Exposure:**
+   - HTTPS: extend the nginx `*.fxhnt.ai` proxy (`infra/k8s/gitlab/tailscale-proxy.yaml`) — at cutover,
+     repoint the `git.fxhnt.ai` server block `proxy_pass` → `gitea-http.foxhunt.svc:3000`.
+   - SSH: expose Gitea SSH on `:22` via the Tailscale proxy (socat) at cutover; retire `:2222`; update the
+     two local remotes to `ssh://git@git.fxhnt.ai/...`.
+   - Registry `:5050` server block: removed (Scaleway registry is external, not proxied here).
+4. **Registry migration:**
+   - `fxhnt/infra/argo/cockpit-build-deploy.yaml`: kaniko `--destination` + `--cache-repo` →
+     `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest` (+ cache repo); drop the `--insecure-registry`/
+     `--skip-tls-verify` GitLab flags (Scaleway is TLS); auth via a Scaleway registry dockerconfig secret.
+   - `fxhnt/infra/k8s/orchestration/dagster.yaml` + dashboard: image ref →
+     `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`; `imagePullSecrets` → Scaleway registry secret.
+   - Create the Scaleway registry pull/push secret (dockerconfigjson) in `foxhunt` ns.
+5. **Argo Events wiring:** `infra/k8s/argo/events/` — a Gitea webhook eventsource + sensor that submits
+   `fxhnt-cockpit` on push to `fxhnt` default branch (reuses the surviving eventbus + workflow-trigger
+   machinery). Webhook secret in a k8s secret; configured in Gitea repo settings.
+6. **Deploy script:** `fxhnt/scripts/argo-deploy-cockpit.sh` + the `fxhnt-cockpit` wftmpl clone URL →
+   Gitea (`git.fxhnt.ai`). After cutover the remote name is unchanged, so this is a clone-source confirm.
+
+## Data flow
+
+Dev pushes to `git.fxhnt.ai` (Gitea) → webhook → Argo Events sensor → `fxhnt-cockpit` workflow → kaniko
+builds + pushes to Scaleway registry → `kubectl apply` + rollout → dagster/dashboard pull the new image
+from Scaleway. Terraform unaffected (state external since 2A).
+
+## Migration & cutover sequence (each step verified)
+
+1. Create `gitea` DB on existing postgres + secrets.
+2. `helm install gitea` (internal svc only). Verify `/api/healthz` 200, admin login via port-forward.
+3. Port-forward Gitea; `git push --mirror` `fxhnt` and `foxhunt` to it; mark `foxhunt` archived.
+4. **Validate (pre-cutover):** clone both repos back from the port-forward, diff top commit SHAs vs
+   GitLab — must match.
+5. **Registry migration:** create Scaleway registry secret; update kaniko build (+cache) + deploy image
+   refs; run one cockpit build → confirm image in Scaleway registry + a test rollout pulls it OK.
+6. **Cutover:** repoint nginx `git.fxhnt.ai` → Gitea + SSH `:2222` → Gitea; remove the `:5050` block.
+   Verify `git.fxhnt.ai` serves Gitea (web + clone + SSH push).
+7. **Wire Argo Events:** add Gitea webhook → sensor; test push to `fxhnt` main fires a cockpit deploy.
+
+## Safety / reversibility
+
+- GitLab stays fully running throughout 2B (just unaddressed after cutover) — rollback = repoint nginx/SSH
+  back to GitLab. Removal is **2C** only.
+- The Scaleway registry change is additive until the deploy refs flip; the GitLab registry image stays as
+  fallback until 2C.
+- All repo data validated (SHA diff) before the public name moves.
+- Secrets (DB creds, admin, webhook, registry dockerconfig) are k8s secrets — **never committed**.
+
+## Risks
+
+- **Registry auth:** Scaleway registry needs a valid API-key dockerconfig for both kaniko push and pod
+  pull. Mitigation: test the build+rollout (step 5) before cutover.
+- **Gitea↔Postgres coupling:** Gitea shares the existing `postgres`. If that instance is later removed,
+  Gitea breaks. Acceptable (postgres is a kept platform service with backups); noted for 2C/2D.
+- **SSH port collision at cutover:** GitLab + Gitea can't both own `:2222` on the Tailscale proxy. Cutover
+  flips it atomically; brief push unavailability during the switch (single maintainer — fine).
+- **Webhook reachability:** Gitea (in-cluster) must reach the Argo Events webhook svc in-cluster — same
+  ns, no Tailscale hop needed.
+
+## Out of scope
+
+GitLab helm uninstall + GitLab buckets/state cleanup (**2C**); `git mv` IaC foxhunt → fxhnt (**2D**).
+
+## Acceptance criteria
+
+- Gitea running (1 pod, existing postgres), `/api/healthz` 200, ~≤300 MiB.
+- Both repos in Gitea; clone SHAs match GitLab; `foxhunt` archived.
+- Cockpit image builds to + deploys from Scaleway Container Registry; `dashboard.fxhnt.ai` 200 on the
+  Scaleway-sourced image.
+- `git.fxhnt.ai` (HTTPS + SSH `:2222`) serves Gitea; git remotes unchanged.
+- Push to `fxhnt` main auto-triggers a cockpit deploy via Argo Events.
+- GitLab still running as fallback (removed in 2C).

docs/superpowers/specs/2026-06-21-relocate-iac-to-fxhnt-design.md

@@ -0,0 +1,85 @@
+# Relocate Platform IaC foxhunt → fxhnt + Archive foxhunt — Design (Phase 2D)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repos:** foxhunt (source), fxhnt (destination)
+**Part of:** Phase 2 ([[project_phase2_gitlab_to_gitea]]). 2A/2B/2C DONE. This is the final sub-project, 2D.
+
+## Motivation
+
+The active work is the Python `fxhnt` fund; `foxhunt` is the dormant Rust system that still happens to
+hold the platform IaC. Consolidate the active platform IaC into `fxhnt` so it is the single home, then
+archive `foxhunt` (keeping its Rust code + history as a read-only snapshot).
+
+## Decision (keystone) — no Terraform state migration
+
+State keys are `${path_relative_to_include()}/terraform.tfstate` (relative to `root.hcl`). Moving
+`infra/live/` + `infra/modules/` into `fxhnt/infra/` **with byte-identical internal structure**
+(`fxhnt/infra/live/production/root.hcl`, `.../<module>/terragrunt.hcl`, module `source = "../../../modules/<m>"`)
+keeps every state key unchanged (`kapsule/`, `public-gateway/`, `dns/`, `block-storage/`). **No state
+migration** — proven by `terragrunt plan = No changes` from the new location. Move method: **plain copy**
+(git history stays in the archived foxhunt).
+
+## Scope — MOVE to `fxhnt/infra/` (active platform)
+
+- **Terraform:** `infra/live/` (terragrunt) + `infra/modules/` (kapsule, public-gateway, dns,
+  block-storage) — structure preserved exactly.
+- **k8s platform manifests** (`infra/k8s/`): `cert-manager`, `databases`, `dns`, `gitea`, `mattermost`,
+  `minio`, `monitoring`, `network-policies`, `secrets`, `services`, `stalwart`, `tailscale`, `storage`,
+  `argo` (surviving gitea events + rbac + netpol + values + databento-download-template + feature-cache-pvc
+  + kustomization), `gitlab` (only `tailscale-proxy.yaml` + `grafana-values.yaml`), `jobs`, `training`
+  (the kept `.dbn` data-acquisition tooling), `namespace.yaml`.
+- **Platform scripts:** the deploy/ops scripts that `kubectl apply`/terragrunt the above (enumerated in the
+  plan after a content scan). `fxhnt/scripts/argo-deploy-cockpit.sh` already lives in fxhnt.
+
+## Scope — STAYS in `foxhunt` (archived, Rust-dead)
+
+Rust source (`crates/`, `services/`, `bin/`, `proto/`, `migrations/`, etc.), Rust scripts (`argo-alpha-*`,
+`argo-compile-deploy`, `argo-sanitizer`, `argo-nsys`, `argo-precompute`, `argo-lob-sweep`, `argo-smoke`,
+`argo-download`), Rust docker build images (ci-builder/training-runtime), `infra/k8s/gpu-taint.yaml`, and
+all foxhunt git history.
+
+## Merge strategy (fxhnt already has `infra/{argo,docker,k8s/{jobs,network-policies,services,storage,orchestration}}`)
+
+Same-named dirs merge **file-by-file**. Pre-flight: scan for filename collisions between foxhunt's and
+fxhnt's `infra/k8s/{network-policies,services,jobs,storage}` (+ `docker`); resolve any collision explicitly
+before copying (expected: none — different resources). Foxhunt's `infra/k8s/argo` lands at
+`fxhnt/infra/k8s/argo` (fxhnt's existing argo is `fxhnt/infra/argo` — different path, no collision).
+
+## Execution order (verify-gated)
+
+1. **Collision pre-flight** — list overlaps; resolve/confirm none.
+2. **Copy Terraform** (`live/` + `modules/`) into `fxhnt/infra/` preserving structure; **`terragrunt
+   init -reconfigure && plan` from `fxhnt/infra/live/production/<each module>` → must be `No changes`**
+   (the no-migration proof). STOP on any diff.
+3. **Copy platform k8s manifests + scripts** into `fxhnt/infra/`; update any in-script `infra/...` paths.
+4. **Verify** manifests still apply + a cockpit deploy still works from fxhnt (it already does).
+5. **`git rm` the moved paths from foxhunt**; commit both repos; push.
+6. **Archive `foxhunt`** in Gitea (read-only) once 2-5 verified.
+
+## Safety / reversibility
+
+- Plain copy first; foxhunt files removed only after fxhnt copy + terragrunt-No-changes verified.
+- TF state untouched (same keys); the Scaleway backend doesn't care which repo runs terragrunt.
+- foxhunt archive (not delete) — full history + Rust code remain restorable.
+- No live workload changes — this is a source-of-truth move, not a cluster change. The running cluster is
+  unaffected (manifests are already applied; terraform state already external).
+
+## Risks
+- **Path drift** breaking TF state → mitigated by the structure-preserve rule + the `No changes` gate.
+- **Merge collision** silently overwriting a fxhnt file → mitigated by the collision pre-flight.
+- **A moved script with a stale `infra/` path** → grep + fix in step 3; verify in step 4.
+- **Premature archive** (foxhunt still referenced) → archive last, after all verification.
+
+## Out of scope
+- Deleting GitLab cold-backup data (separate grace-period pass).
+- Pruning the Rust codebase itself (foxhunt is archived as-is).
+- Phase 2E observability (Grafana/Prometheus).
+
+## Acceptance criteria
+- `fxhnt/infra/{live,modules}` present; `terragrunt plan` = `No changes` for all 4 modules from fxhnt
+  (state keys unchanged — no migration).
+- Platform k8s manifests + scripts present in `fxhnt/infra/`; no filename collisions lost.
+- Moved paths `git rm`'d from foxhunt; both repos committed + pushed to Gitea.
+- Cockpit deploy + `dashboard.fxhnt.ai` 200 still work (cluster unaffected).
+- `foxhunt` repo archived in Gitea; Rust code + history intact.

docs/superpowers/specs/2026-06-21-tfstate-to-scaleway-design.md

@@ -0,0 +1,94 @@
+# Migrate Terraform State to Scaleway Object Storage — Design (Phase 2A)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** foxhunt (IaC current home; relocation to fxhnt is Phase 2D)
+**Part of:** Phase 2 (GitLab → Gitea consolidation). Sub-projects: **2A state migration (this)**, 2B Gitea
+stand-up + repo migration + Argo wiring, 2C GitLab decommission, 2D IaC relocation into fxhnt.
+
+## Motivation
+
+Terraform state for the whole production cluster currently lives in the **self-hosted GitLab** http
+backend (project ID=1), whose object data sits in **in-cluster MinIO**. That is a chicken-and-egg
+fragility: the state describing the cluster lives *inside* the cluster it manages. GitLab is also slated
+for decommission (Phase 2C, replaced by lightweight Gitea). Moving state to **external Scaleway Object
+Storage** decouples it from both GitLab and the cluster — a prerequisite for removing GitLab and a
+durability win on its own (state survives cluster loss).
+
+## Decision (keystone)
+
+**Target backend: Scaleway Object Storage (native S3), bucket `foxhunt-tfstate`** (already exists,
+fr-par, created ~3 months ago but never wired). Reuse it (no rename — avoids a needless create; the
+`foxhunt-` prefix is cosmetic and harmless). Locking via **OpenTofu native S3 lockfile** (`use_lockfile`,
+OpenTofu v1.11.5 present) — no DynamoDB.
+
+## Scope — what changes
+
+1. **`infra/live/production/root.hcl`** — replace the single `remote_state` block: `backend = "http"` (GitLab) →
+   `backend = "s3"` (Scaleway). New config:
+   ```hcl
+   remote_state {
+     backend = "s3"
+     generate = { path = "backend.tf", if_exists = "overwrite" }
+     config = {
+       bucket = "foxhunt-tfstate"
+       key    = "${path_relative_to_include()}/terraform.tfstate"
+       region = "fr-par"
+       endpoints = { s3 = "https://s3.fr-par.scw.cloud" }
+       skip_credentials_validation = true
+       skip_region_validation      = true
+       skip_requesting_account_id  = true
+       skip_metadata_api_check     = true
+       use_lockfile                = true
+     }
+   }
+   ```
+   Credentials: Scaleway access/secret keys exported as `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`
+   (the s3 backend reads AWS-style env vars) from the existing `scaleway-credentials` k8s secret.
+2. **State migration, per module, sequential** (`kapsule`, `public-gateway`, `dns`, `block-storage`):
+   - Back up first: `terragrunt state pull > /tmp/tfstate-backup-<module>.tfstate` (from GitLab, BEFORE
+     changing the backend for that module).
+   - `terragrunt init -migrate-state -force-copy` → copies GitLab state → Scaleway S3.
+   - `terragrunt plan` → **must report `No changes`** before proceeding to the next module.
+3. **Lock smoke test** (once, on one module): confirm `use_lockfile` works against Scaleway — a lock
+   object is created during apply/plan-with-lock and released after (verify no orphan `.tflock` left).
+
+## Migration order & gates
+
+`kapsule` → `public-gateway` → `dns` → `block-storage`. After EACH: backup taken, `-migrate-state`
+succeeded, `plan = No changes` confirmed. **STOP and revert if any module's post-migration plan shows
+drift** (revert = restore `http` backend in root.hcl + `terragrunt init` to re-attach GitLab state; the
+GitLab copy is never deleted in 2A).
+
+## Safety / reversibility
+
+- **Old GitLab state is NOT deleted in 2A** — it stays as a live fallback until GitLab is removed in 2C.
+  So the entire migration is reversible by flipping `root.hcl` back to the http backend.
+- `/tmp` state backups taken before each module (belt-and-suspenders; never committed — contains
+  resource IDs, not secrets, but treat as sensitive).
+- One module at a time with a `plan = No changes` gate prevents a bad backend config from cascading.
+- `endpoints.s3` (not deprecated top-level `endpoint`) + the four `skip_*` flags are required for the
+  AWS s3 backend to talk to Scaleway's S3-compatible API without AWS-specific preflight calls.
+
+## Risks
+
+- **Scaleway S3 quirks:** some provider/backend versions need `skip_s3_checksum = true` for Scaleway. If
+  `-migrate-state` errors on checksum, add it. (Documented as a known fallback in the plan.)
+- **Lock semantics:** Scaleway Object Storage must honor conditional-write for `use_lockfile`. The lock
+  smoke test catches this; fallback is to run with locking disabled (single maintainer) if unsupported.
+- **Credential confusion:** the s3 backend reads `AWS_*` env vars, NOT `SCW_*`. Plan must export both
+  (SCW_* for the provider, AWS_* for the backend) — a common foot-gun.
+
+## Out of scope (later sub-projects)
+
+Gitea stand-up + repo migration + Argo wiring (2B); GitLab helm removal + GitLab state/bucket cleanup
+(2C); `git mv` of IaC from foxhunt → fxhnt + path updates (2D). Files stay in foxhunt for 2A; only the
+backend changes.
+
+## Acceptance criteria
+
+- `infra/live/production/root.hcl` uses the Scaleway s3 backend; committed.
+- All 4 modules migrated: each shows `terragrunt plan = No changes` against the new backend.
+- State objects present in `foxhunt-tfstate` at `<module>/terraform.tfstate` (verified via `scw`/`mc`).
+- Lock smoke test passes (lock acquired + released, no orphan lockfile).
+- GitLab http state still intact (untouched fallback) — not yet deleted.

infra/k8s/argo/README-deps-cache.md

@@ -1,161 +0,0 @@
-# Deps Cache Image — Cold-start Compile Acceleration
-
-This README documents the **`ci-builder-cpu-with-deps:nightly`** image
-and the Argo plumbing that uses it. Created as part of the CI cold-start
-optimization sweep (April 2026).
-
-## What it is
-
-A Docker image that carries a pre-built `target/` directory at
-`/cargo-target-prebuilt/` containing the workspace's third-party
-dependency rlibs. It's pulled by the `compile-services` step of
-`compile-and-deploy-template.yaml` as an **initContainer** that rsyncs
-its payload into the `cargo-target-cpu` PVC.
-
-## Why it exists
-
-On a fresh node (autoscaler scaled up, fresh PVC, or after a PVC purge),
-`cargo build --release` would otherwise spend 3-5 minutes compiling
-~1500 third-party crates before touching workspace member code. With
-the deps cache image, the cold-start compile path is:
-
-1. **initContainer pulls image** from in-cluster registry (~30-60s)
-2. **rsync prebuilt rlibs into PVC** (~30-60s, ~6-8 GB transfer over local FS)
-3. **cargo build delta-compiles only workspace member code** (~1-2 min)
-
-Net cold-start saving: **2-4 min per compile pod**.
-
-## Files
-
-| File | Role |
-|------|------|
-| `infra/docker/Dockerfile.ci-deps-cache` | Builds the deps-cache image (multi-stage). Stage 1 atop ci-builder-cpu, runs `cargo build --release --workspace`. Stage 2 thin Ubuntu + rsync layer carrying just `/cargo-target-prebuilt/`. |
-| `infra/k8s/argo/refresh-deps-cache-template.yaml` | WorkflowTemplate that drives a Kaniko build of the Dockerfile + push to GitLab registry. Includes a CronWorkflow that runs nightly at 03:00 UTC (suspended by default). |
-| `infra/k8s/argo/compile-and-deploy-template.yaml` | The `compile-services` step has a `seed-deps-cache` initContainer that rsyncs `/cargo-target-prebuilt/` into the PVC, gated by a stamp file. |
-
-## How to refresh manually
-
-```bash
-# Trigger a one-shot rebuild against current main HEAD.
-argo submit --from workflowtemplate/refresh-deps-cache -n foxhunt
-```
-
-To target a specific commit or bump the cache version (forces re-seed
-on every compile pod):
-
-```bash
-argo submit --from workflowtemplate/refresh-deps-cache -n foxhunt \
-  -p commit-sha=$(git rev-parse main) \
-  -p deps-version=2 \
-  -p image-tag=nightly
-```
-
-## Enabling the nightly cron
-
-The CronWorkflow ships **suspended** so you can validate manually first.
-To enable:
-
-```bash
-kubectl -n foxhunt patch cronworkflow refresh-deps-cache-nightly \
-  --type=merge -p '{"spec":{"suspend":false}}'
-```
-
-To disable (return to manual-only):
-
-```bash
-kubectl -n foxhunt patch cronworkflow refresh-deps-cache-nightly \
-  --type=merge -p '{"spec":{"suspend":true}}'
-```
-
-## Bumping `DEPS_VERSION`
-
-The compile-services initContainer skips re-seeding when
-`/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp` exists on the PVC.
-Bump `DEPS_VERSION` to force every running compile pod to re-seed
-its PVC from a fresh image. Bump in two places (must match):
-
-1. `refresh-deps-cache-template.yaml` workflow `deps-version` parameter
-   default value
-2. `compile-and-deploy-template.yaml` initContainer `DEPS_VERSION` env var
-
-Bump when:
-- Cargo.lock has churned significantly (most prebuilt rlibs
-  fingerprint-mismatch cargo's freshness check)
-- The Rust toolchain version changes (e.g. 1.89 -> 1.90)
-- A workspace member is renamed or split (rare)
-
-You do **not** need to bump for routine commits — cargo's own
-fingerprint check will discard stale rlibs and rebuild as needed.
-
-## Debugging
-
-**Symptom: compile-services is still slow on a fresh PVC.**
-
-Check the seed-deps-cache initContainer logs first:
-
-```bash
-argo logs -n foxhunt <workflow-name> -c seed-deps-cache
-```
-
-Look for:
-- `WARN: /cargo-target-prebuilt missing in image` — the image was built
-  but the snapshot dir is empty. The inner `cargo build --workspace`
-  failed in the deps-cache image build. Check the
-  `refresh-deps-cache` workflow logs.
-- `PVC already seeded for deps v1 (stamp present), skipping.` —
-  expected steady-state on warm pods.
-- `=== Seeding cargo-target-cpu PVC from prebuilt deps cache ===` +
-  `=== Seed complete ===` — first run on this PVC, working as designed.
-
-**Symptom: cargo recompiles many third-party deps anyway despite the
-seed completing.**
-
-Cargo's fingerprint check is sensitive to: rustc version, the entire
-target.rustflags list, `RUSTFLAGS` env var, source code mtimes (on
-build.rs), feature flags. The commit-SHA used to build the deps cache
-image must match the live compile workflow's flag set. The most common
-cause of full-rebuild is a `RUSTFLAGS` mismatch between the deps-cache
-image build and the compile pod (e.g. mold vs wild linker swap).
-
-To diagnose, look at the cargo build output: lines like
-`Compiling tokio v1.x.x` for crates that were obviously in the cache
-indicate fingerprint mismatch.
-
-**Symptom: PVC fills up faster than before.**
-
-Each seed adds ~6-8 GB to the PVC. The compile-services step has a
-prune-at-25GB guard that wipes `release/` and `debug/` if usage
-exceeds 25GB — when that fires, the next run re-seeds (stamp file is
-in PVC root, survives the prune). To raise the threshold, edit the
-`TARGET_SIZE_MB -gt 25000` check in compile-and-deploy-template.yaml.
-
-## Assumptions that break the cache
-
-- **rustc toolchain version mismatch** between deps-cache image and
-  compile pod — cargo will full-rebuild. Resolution: rebuild the
-  deps-cache image (refresh-deps-cache).
-- **Different `RUSTFLAGS` / `target.<triple>.rustflags`** between
-  image build and live compile (e.g. switching CPU target features)
-  — full rebuild. Resolution: rebuild deps-cache image after the
-  rustflags change is committed.
-- **Cargo.lock SHA divergence** > a few weeks old — most rlibs no
-  longer match. Resolution: rebuild deps-cache image (refresh nightly,
-  or trigger manually).
-- **Linker swap (e.g. mold -> wild)** — link-output is cached at the
-  fingerprint level; changing linker doesn't trigger rebuild but may
-  produce different final binaries. Not a cache invalidation, just a
-  link-time difference.
-
-## When to delete this entirely
-
-If the underlying compile time drops to <30s in cold-start (e.g. via a
-much faster registry or sccache covering 100% of deps), the
-deps-cache complexity is no longer worth maintaining. To remove:
-
-1. Remove the `seed-deps-cache` initContainer block from
-   `compile-and-deploy-template.yaml` (the compile script tolerates
-   an unseeded PVC — cargo just compiles deps from scratch).
-2. Delete `infra/docker/Dockerfile.ci-deps-cache` and
-   `infra/k8s/argo/refresh-deps-cache-template.yaml`.
-3. Delete the published image:
-   `crane delete gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:nightly`.

infra/k8s/argo/alpha-cv-template.yaml

@@ -1,495 +0,0 @@
-# Alpha walk-forward CV workflow.
-#
-# Trains the Mamba2+stacker alpha logits cache on the full 9Q fxcache,
-# then runs `alpha_baseline` 9 times across disjoint 1.9M-bar windows
-# (one per quarter). Each fold trains its own execution-policy DQN
-# from scratch on the front of the window and backtests on the back
-# (purged train/eval split via `--train-frac`).
-#
-# Sequential by design: 9 folds chain in the DAG so a single L40S node
-# stays warm across the run. With --decision-stride 200 and --horizon
-# 1200, each fold's DQN training is ~5-10 min on L40S; total wall ~60-90
-# min for stacker + 9 folds.
-#
-# DAG:
-#   ensure-binary ──> ensure-fxcache ──> stacker-train ──> fold-0 ──> ... ──> fold-8
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/alpha-cv \
-#     -p commit-sha=HEAD -p git-branch=main -p decision-stride=200
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: alpha-cv
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: alpha-cv
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: pipeline
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 28800
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: main
-      - name: cuda-compute-cap
-        value: "89"
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: symbol
-        value: ES.FUT
-      # Match the fxcache build params (cluster's 9Q fxcache was built
-      # with these on 2026-05-16; see commit e7ce4395e).
-      - name: imbalance-bar-threshold
-        value: "20.0"
-      - name: imbalance-bar-ewma-alpha
-        value: "0.1"
-      - name: volume-bar-size
-        value: "100"
-      - name: data-source
-        value: "mbp10"
-      # Stacker training params (mirror local 2Q smoke).
-      - name: stacker-horizon
-        value: "6000"
-      - name: stacker-seq-len
-        value: "32"
-      - name: stacker-hidden-dim
-        value: "64"
-      - name: stacker-state-dim
-        value: "16"
-      - name: stacker-epochs
-        value: "5"
-      - name: stacker-batch-size
-        value: "128"
-      - name: stacker-lr
-        value: "3e-3"
-      - name: stacker-train-frac
-        value: "0.8"
-      # CV / alpha_baseline params — defaults match the validated 2Q
-      # config that produced +1.78 Sharpe at quarter-tick cost.
-      - name: decision-stride
-        value: "200"
-      - name: fold-window
-        value: "1900000"
-      - name: train-frac
-        value: "0.6"
-      - name: window-k
-        value: "16"
-      - name: horizon
-        value: "1200"
-      - name: n-train-par
-        value: "25"
-      - name: n-train-episodes
-        value: "8000"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  templates:
-    # ── DAG ──
-    - name: pipeline
-      dag:
-        tasks:
-          - name: ensure-binary
-            template: ensure-binary
-          - name: stacker-train
-            template: stacker-train
-            dependencies: [ensure-binary]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: fold-0
-            template: alpha-cv-fold
-            dependencies: [stacker-train]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "0"
-                - name: offset
-                  value: "0"
-          - name: fold-1
-            template: alpha-cv-fold
-            dependencies: [fold-0]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "1"
-                - name: offset
-                  value: "1900000"
-          - name: fold-2
-            template: alpha-cv-fold
-            dependencies: [fold-1]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "2"
-                - name: offset
-                  value: "3800000"
-          - name: fold-3
-            template: alpha-cv-fold
-            dependencies: [fold-2]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "3"
-                - name: offset
-                  value: "5700000"
-          - name: fold-4
-            template: alpha-cv-fold
-            dependencies: [fold-3]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "4"
-                - name: offset
-                  value: "7600000"
-          - name: fold-5
-            template: alpha-cv-fold
-            dependencies: [fold-4]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "5"
-                - name: offset
-                  value: "9500000"
-          - name: fold-6
-            template: alpha-cv-fold
-            dependencies: [fold-5]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "6"
-                - name: offset
-                  value: "11400000"
-          - name: fold-7
-            template: alpha-cv-fold
-            dependencies: [fold-6]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "7"
-                - name: offset
-                  value: "13300000"
-          - name: fold-8
-            template: alpha-cv-fold
-            dependencies: [fold-7]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "8"
-                - name: offset
-                  value: "15200000"
-
-    # ── ensure-binary: compile alpha_train_stacker + alpha_baseline ──
-    - name: ensure-binary
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /cargo-target/sccache
-          - name: SCCACHE_CACHE_SIZE
-            value: "40G"
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"; git fetch origin; SHA=$(git rev-parse "origin/$BRANCH"); cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"
-                git checkout "origin/$BRANCH"; SHA=$(git rev-parse HEAD); cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            BINARIES="alpha_train_stacker alpha_baseline"
-            ALL_CACHED=true
-            for bin in $BINARIES; do
-              [ ! -x "$BIN_DIR/$bin" ] && ALL_CACHED=false && break
-            done
-
-            if [ "$ALL_CACHED" = "true" ]; then
-              echo "=== Cache HIT ==="
-              ls -lh "$BIN_DIR/"
-              echo "$SHORT_SHA" > /tmp/sha
-              exit 0
-            fi
-            echo "=== Cache MISS: compiling alpha binaries for $SHORT_SHA ==="
-
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"; git fetch origin
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$SHA" ]; then
-                git checkout --force "$SHA"; git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"; git checkout "$SHA"
-            fi
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            # alpha_train_stacker lives in ml-alpha, alpha_baseline in ml.
-            echo "Building alpha_train_stacker (ml-alpha)..."
-            cargo build --release -p ml-alpha --example alpha_train_stacker
-            echo "Building alpha_baseline (ml)..."
-            cargo build --release -p ml --features ml/cuda --example alpha_baseline
-
-            mkdir -p "$BIN_DIR"
-            cp "$CARGO_TARGET_DIR/release/examples/alpha_train_stacker" "$BIN_DIR/"
-            cp "$CARGO_TARGET_DIR/release/examples/alpha_baseline" "$BIN_DIR/"
-            strip "$BIN_DIR/"*
-            # alpha_baseline reads config/ml/alpha_fill_coeffs.json as a
-            # required --fill-coeffs input. The source tree's copy travels
-            # with the binaries so the GPU fold pods don't need to clone.
-            cp "$BUILD/config/ml/alpha_fill_coeffs.json" "$BIN_DIR/" || true
-            ls -lh "$BIN_DIR/"
-
-            cd /data/bin
-            ls -1t | tail -n +6 | while read -r old; do
-              echo "Pruning old cache: $old"; rm -rf "$old"
-            done
-            echo "$SHORT_SHA" > /tmp/sha
-
-    # ── stacker-train: train Mamba2+stacker on the 9Q fxcache ──
-    - name: stacker-train
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-        resources:
-          requests:
-            cpu: "8"
-            memory: 32Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "16"
-            memory: 80Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BIN="/data/bin/$SHA/alpha_train_stacker"
-            FXCACHE=$(ls -t /feature-cache/*.fxcache | head -1)
-            ALPHA_OUT="/feature-cache/alpha_logits_cache_9q.bin"
-            echo "Using fxcache: $FXCACHE"
-            echo "Output: $ALPHA_OUT"
-
-            "$BIN" \
-              --fxcache-path "$FXCACHE" \
-              --horizon {{workflow.parameters.stacker-horizon}} \
-              --seq-len {{workflow.parameters.stacker-seq-len}} \
-              --hidden-dim {{workflow.parameters.stacker-hidden-dim}} \
-              --state-dim {{workflow.parameters.stacker-state-dim}} \
-              --epochs {{workflow.parameters.stacker-epochs}} \
-              --batch-size {{workflow.parameters.stacker-batch-size}} \
-              --lr {{workflow.parameters.stacker-lr}} \
-              --train-frac {{workflow.parameters.stacker-train-frac}} \
-              --alpha-cache-out "$ALPHA_OUT"
-            ls -lh "$ALPHA_OUT"
-
-    # ── alpha-cv-fold: run alpha_baseline on one window ──
-    - name: alpha-cv-fold
-      inputs:
-        parameters:
-          - name: sha
-          - name: fold-idx
-          - name: offset
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-        resources:
-          requests:
-            cpu: "8"
-            memory: 32Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "16"
-            memory: 80Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            FOLD={{inputs.parameters.fold-idx}}
-            OFFSET={{inputs.parameters.offset}}
-            BIN="/data/bin/$SHA/alpha_baseline"
-            FXCACHE=$(ls -t /feature-cache/*.fxcache | head -1)
-            ALPHA="/feature-cache/alpha_logits_cache_9q.bin"
-            # alpha_fill_coeffs.json travels with the source tree; the
-            # binary needs it next to itself or via explicit --fill-coeffs.
-            # The ci-builder image bakes it under /opt/foxhunt/config/ml.
-            FILL="/data/bin/$SHA/alpha_fill_coeffs.json"
-            if [ ! -f "$FILL" ]; then
-              # Fall back to copying from the cloned source on the
-              # cargo-target PVC mounted in ensure-binary; we don't have
-              # that PVC here, so embed via fxcache PVC pre-stage.
-              FILL="/feature-cache/alpha_fill_coeffs.json"
-            fi
-            OUT="/feature-cache/cv_results_9fold/fold_${FOLD}.json"
-            mkdir -p /feature-cache/cv_results_9fold
-            echo "Fold $FOLD: offset=$OFFSET out=$OUT"
-
-            "$BIN" \
-              --fxcache-path "$FXCACHE" \
-              --alpha-cache "$ALPHA" \
-              --fill-coeffs "$FILL" \
-              --data-start-offset $OFFSET \
-              --max-snapshots {{workflow.parameters.fold-window}} \
-              --train-frac {{workflow.parameters.train-frac}} \
-              --window-k {{workflow.parameters.window-k}} \
-              --decision-stride {{workflow.parameters.decision-stride}} \
-              --horizon {{workflow.parameters.horizon}} \
-              --n-train-par {{workflow.parameters.n-train-par}} \
-              --n-train-episodes {{workflow.parameters.n-train-episodes}} \
-              --out-path "$OUT"
-            ls -lh "$OUT"

infra/k8s/argo/alpha-perception-template.yaml

@@ -1,484 +0,0 @@
-# Alpha perception trainer workflow.
-#
-# Stacked Mamba2 -> CfC -> heads supervised pretrain on MBP-10 from the
-# training-data PVC. Emits `alpha_train_summary.json` to MinIO-backed
-# feature-cache PVC with per-horizon val AUC for monitoring. Runs on
-# a single L40S; ~30-90 min wall depending on
-# n-train-seqs.
-#
-# DAG:
-#   ensure-binary ──> train
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/alpha-perception \
-#     -p commit-sha=HEAD -p git-branch=ml-alpha-phase-a
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: alpha-perception
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: alpha-perception
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: pipeline
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 14400
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: ml-alpha-phase-a
-      - name: cuda-compute-cap
-        value: "89"
-      - name: gpu-pool
-        value: ci-training-l40s
-      # Trainer hyperparameters (mirror the validated smoke config).
-      - name: epochs
-        value: "5"
-      - name: multi-resolution
-        value: "1:32"
-      - name: mamba2-state-dim
-        value: "16"
-      - name: lr-cfc
-        value: "3e-3"
-      - name: lr-mamba2
-        value: "1e-3"
-      - name: n-train-seqs
-        value: "8000"
-      - name: n-val-seqs
-        value: "1000"
-      - name: seed
-        value: "16962"
-      - name: batch-size
-        value: "1"
-      - name: auto-horizon-weights
-        value: "false"
-      - name: instrument-mode
-        value: "all"
-      - name: early-stop-metric
-        value: "mean_auc"
-      - name: early-stop-patience
-        value: "5"
-      - name: cv-fold
-        value: "0"
-      - name: cv-n-folds
-        value: "1"
-      - name: cv-train-window
-        value: "0"
-      - name: smoothness-base-lambda
-        value: "0.0"
-      - name: kernel-step-trace-enable
-        value: "false"
-      - name: kernel-step-trace-path
-        value: ""
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  templates:
-    # ── DAG ──
-    #
-    # check-cache (tiny alpine pod on platform pool, ~3 sec) inspects
-    # the training-data PVC for /data/bin/$SHA/alpha_train. If present
-    # (cache hit), ensure-binary is SKIPPED via `when:` — the
-    # ~4.8GB ci-builder image pull + sccache compile cycle are avoided
-    # entirely. train still runs, sourcing the binary path from
-    # check-cache's SHA output.
-    #
-    # warmup-gpu fires unconditionally in parallel — triggers the L40S
-    # autoscaler so the node is warm by the time train needs it,
-    # whether or not we compile.
-    - name: pipeline
-      dag:
-        tasks:
-          - name: check-cache
-            template: check-cache
-          - name: ensure-binary
-            template: ensure-binary
-            dependencies: [check-cache]
-            when: "{{tasks.check-cache.outputs.parameters.cache}} == miss"
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.check-cache.outputs.parameters.sha}}"
-          - name: warmup-gpu
-            template: warmup-gpu
-          - name: train
-            template: train
-            dependencies: [check-cache, ensure-binary]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.check-cache.outputs.parameters.sha}}"
-
-    # ── check-cache: probe the training-data PVC for an existing binary ──
-    #
-    # Runs on the platform pool (always up, no autoscaler delay). Tiny
-    # alpine pod, ~3 sec end-to-end. Emits two output parameters:
-    #   - sha   = the short SHA used for binary cache keying
-    #   - cache = "hit" if /data/bin/$SHA/alpha_train exists else "miss"
-    #
-    # Resolves `commit-sha=HEAD` by reading /tmp/head-sha from a
-    # ConfigMap... actually no, simpler: the submission script
-    # (scripts/argo-alpha-perception.sh) pre-resolves HEAD via local
-    # git so commit-sha is always a real SHA in the workflow. This
-    # pod just trims to short SHA and stats the binary.
-    - name: check-cache
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-          - name: cache
-            valueFrom:
-              path: /tmp/cache
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: alpine:3.20
-        command: ["/bin/sh", "-c"]
-        resources:
-          requests:
-            cpu: "50m"
-            memory: 32Mi
-          limits:
-            cpu: "100m"
-            memory: 64Mi
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            if [ "$SHA" = "HEAD" ]; then
-              echo "ERROR: commit-sha cannot be HEAD inside the workflow."
-              echo "       Resolve via the submission script (scripts/argo-alpha-perception.sh)."
-              exit 1
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "$SHORT_SHA" > /tmp/sha
-
-            BIN="/data/bin/$SHORT_SHA/alpha_train"
-            if [ -x "$BIN" ]; then
-              SIZE=$(stat -c %s "$BIN")
-              echo "Cache HIT: $BIN ($SIZE bytes)"
-              echo "hit" > /tmp/cache
-            else
-              echo "Cache MISS: $BIN not present, ensure-binary will compile"
-              ls -lh "/data/bin/" 2>/dev/null | head -10 || echo "  (no /data/bin/ directory yet)"
-              echo "miss" > /tmp/cache
-            fi
-
-    # ── ensure-binary: compile `alpha_train` example via sccache ──
-    # Only runs when check-cache returned cache=miss. Outputs the SHA
-    # for consistency, but the DAG sources SHA from check-cache directly.
-    - name: ensure-binary
-      inputs:
-        parameters:
-          - name: sha
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /cargo-target/sccache
-          - name: SCCACHE_CACHE_SIZE
-            value: "40G"
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          - name: CUDA_COMPUTE_CAP
-            value: "{{workflow.parameters.cuda-compute-cap}}"
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"; git fetch origin; SHA=$(git rev-parse "origin/$BRANCH"); cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"
-                git checkout "origin/$BRANCH"; SHA=$(git rev-parse HEAD); cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            if [ -x "$BIN_DIR/alpha_train" ]; then
-              echo "=== Cache HIT: $BIN_DIR/alpha_train ==="
-              ls -lh "$BIN_DIR/"
-              echo "$SHORT_SHA" > /tmp/sha
-              exit 0
-            fi
-            echo "=== Cache MISS: compiling alpha_train for $SHORT_SHA ==="
-
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"; git fetch origin
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$SHA" ]; then
-                git checkout --force "$SHA"; git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"; git checkout "$SHA"
-            fi
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-
-            FEATURES_FLAG=""
-            if [ "{{workflow.parameters.kernel-step-trace-enable}}" = "true" ]; then
-              FEATURES_FLAG="--features kernel-step-trace"
-              echo "Building with --features kernel-step-trace"
-              # Bust cache: feature-on binary differs from feature-off at same SHA.
-              if [ -x "$BIN_DIR/alpha_train" ]; then
-                echo "Removing cached binary (feature change requires rebuild)"
-                rm -f "$BIN_DIR/alpha_train"
-              fi
-            fi
-
-            echo "Building alpha_train (ml-alpha)..."
-            cargo build --release $FEATURES_FLAG -p ml-alpha --example alpha_train
-
-            mkdir -p "$BIN_DIR"
-            cp "$CARGO_TARGET_DIR/release/examples/alpha_train" "$BIN_DIR/"
-            strip "$BIN_DIR/alpha_train"
-            ls -lh "$BIN_DIR/"
-
-            # Prune old cache (keep last 5)
-            cd /data/bin
-            ls -1t | tail -n +6 | while read -r old; do
-              echo "Pruning old cache: $old"; rm -rf "$old"
-            done
-            echo "$SHORT_SHA" > /tmp/sha
-
-    # ── warmup-gpu: trigger L40S autoscaler scale-up in parallel ──
-    #
-    # Schedules a tiny CPU-only pod on the gpu-pool's nodeSelector.
-    # Kubernetes sees an unschedulable pod (no node currently in the
-    # pool), autoscaler scales the pool 0 → 1. The pod exits the
-    # instant it lands; the node enters scaledown-grace.
-    #
-    # Cluster autoscaler config (see infra/modules/kapsule/main.tf):
-    #   scale_down_delay_after_add = "10m"  -- 10m before any scaledown consideration
-    #   scale_down_unneeded_time   = "10m"  -- 10m empty before action
-    # Effective grace window is up to ~20m, which always spans
-    # ensure-binary compile (sccache-hit ~10s through cold ~15m).
-    # train lands on a hot node with zero provisioning latency.
-    #
-    # No GPU resource request — that would compete with `train` for
-    # the single GPU and serialise the steps. The nodeSelector +
-    # nvidia.com/gpu toleration are enough to force placement on the
-    # right pool.
-    - name: warmup-gpu
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: alpine:3.20
-        command: ["/bin/sh", "-c"]
-        resources:
-          requests:
-            cpu: "50m"
-            memory: 32Mi
-          limits:
-            cpu: "100m"
-            memory: 64Mi
-        args:
-          - |
-            echo "GPU warmup pod scheduled on $(hostname) — autoscaler scale-up triggered, node now in scaledown-grace window."
-
-    # ── train: run alpha_train on L40S ──
-    - name: train
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CUDA_COMPUTE_CAP
-            value: "{{workflow.parameters.cuda-compute-cap}}"
-        resources:
-          # L40S-1-48G has 8 vCPU (7800m allocatable, after kubelet
-          # overhead) and ~91Gi memory. Request <7800m so the pod
-          # fits alongside the system daemonsets (cilium, csi-node,
-          # nvidia drivers/dcgm/feature-discovery) on the same node.
-          requests:
-            cpu: "6"
-            memory: 16Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "7"
-            memory: 64Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BIN="/data/bin/$SHA/alpha_train"
-            MBP10_DIR="/data/futures-baseline-mbp10/ES.FUT"
-            PREDECODED_DIR="/feature-cache/predecoded"
-            OUT_DIR="/feature-cache/alpha-perception-runs/$SHA"
-            mkdir -p "$OUT_DIR" "$PREDECODED_DIR"
-
-            if [ ! -x "$BIN" ]; then
-              echo "ERROR: binary not found at $BIN"
-              ls -lh /data/bin/ || true
-              exit 1
-            fi
-            if [ ! -d "$MBP10_DIR" ]; then
-              echo "ERROR: MBP-10 data directory not found at $MBP10_DIR"
-              ls -lh /data/ || true
-              exit 1
-            fi
-            echo "Running alpha_train (stacked Mamba2 -> CfC -> heads)"
-            echo "  binary:        $BIN"
-            echo "  mbp10:         $MBP10_DIR ($(find "$MBP10_DIR" -name '*.dbn.zst' | wc -l) files)"
-            echo "  predecoded:    $PREDECODED_DIR"
-            echo "  out:           $OUT_DIR"
-
-            EXTRA_FLAGS=""
-            if [ "{{workflow.parameters.auto-horizon-weights}}" = "true" ]; then
-              EXTRA_FLAGS="$EXTRA_FLAGS --auto-horizon-weights"
-            fi
-            if [ -n "{{workflow.parameters.instrument-mode}}" ]; then
-              EXTRA_FLAGS="$EXTRA_FLAGS --instrument-mode {{workflow.parameters.instrument-mode}}"
-            fi
-
-            TRACE_FLAG=""
-            if [ -n "{{workflow.parameters.kernel-step-trace-path}}" ]; then
-              TRACE_FLAG="--kernel-step-trace {{workflow.parameters.kernel-step-trace-path}}"
-            fi
-
-            "$BIN" \
-              --mbp10-data-dir   "$MBP10_DIR" \
-              --predecoded-dir   "$PREDECODED_DIR" \
-              --out              "$OUT_DIR" \
-              --epochs           {{workflow.parameters.epochs}} \
-              --multi-resolution "{{workflow.parameters.multi-resolution}}" \
-              --mamba2-state-dim {{workflow.parameters.mamba2-state-dim}} \
-              --lr-cfc           {{workflow.parameters.lr-cfc}} \
-              --lr-mamba2        {{workflow.parameters.lr-mamba2}} \
-              --n-train-seqs     {{workflow.parameters.n-train-seqs}} \
-              --n-val-seqs       {{workflow.parameters.n-val-seqs}} \
-              --seed             {{workflow.parameters.seed}} \
-              --batch-size       {{workflow.parameters.batch-size}} \
-              --early-stop-metric   {{workflow.parameters.early-stop-metric}} \
-              --early-stop-patience {{workflow.parameters.early-stop-patience}} \
-              --cv-fold             {{workflow.parameters.cv-fold}} \
-              --cv-n-folds          {{workflow.parameters.cv-n-folds}} \
-              --cv-train-window     {{workflow.parameters.cv-train-window}} \
-              --smoothness-base-lambda {{workflow.parameters.smoothness-base-lambda}} \
-              ${TRACE_FLAG} \
-              $EXTRA_FLAGS
-
-            echo "=== Training complete ==="
-            ls -lh "$OUT_DIR/"
-            echo "--- alpha_train_summary.json ---"
-            cat "$OUT_DIR/alpha_train_summary.json"

infra/k8s/argo/alpha-rl-template.yaml

@@ -1,207 +0,0 @@
-# alpha-rl: single-pod compile + train on GPU.
-#
-# Compiles alpha_rl_train incrementally on the GPU node (~3s warm,
-# ~90s cold) using the cargo-target-cuda PVC, then runs training
-# immediately. No separate compile node, no binary transfer, no
-# fxcache step. Predecoded MBP-10 sidecars live on feature-cache PVC.
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/alpha-rl \
-#     -p git-branch=ml-alpha-phase-a
-#   argo submit -n foxhunt --from=wftmpl/alpha-rl \
-#     -p git-branch=ml-alpha-phase-a -p n-steps=50000 -p n-backtests=128
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: alpha-rl
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: alpha-rl
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: compile-and-train
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 14400
-
-  arguments:
-    parameters:
-      - name: git-branch
-        value: ml-alpha-phase-a
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: n-steps
-        value: "50000"
-      - name: n-backtests
-        value: "128"
-      - name: seed
-        value: "16962"
-      - name: seq-len
-        value: "32"
-      - name: per-capacity
-        value: "32768"
-      - name: instrument-mode
-        value: "all"
-      - name: log-every
-        value: "5000"
-      - name: fold-idx
-        value: "0"
-      - name: n-folds
-        value: "1"
-      - name: n-eval-steps
-        value: "0"
-      - name: nsys-profile
-        value: "false"
-      - name: use-multi-head-policy
-        value: "0"
-      - name: band-enabled
-        value: "0"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  templates:
-    - name: compile-and-train
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CARGO_INCREMENTAL
-            value: "1"
-          - name: FOXHUNT_USE_MULTI_HEAD_POLICY
-            value: "{{workflow.parameters.use-multi-head-policy}}"
-          - name: FOXHUNT_BAND_ENABLED
-            value: "{{workflow.parameters.band-enabled}}"
-        resources:
-          requests:
-            cpu: "4"
-            memory: 16Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "7"
-            memory: 40Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-        args:
-          - |
-            set -e
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-
-            # SSH
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            nvidia-smi
-
-            # Git
-            BRANCH="{{workflow.parameters.git-branch}}"
-            BUILD="/cargo-target/src"
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              git checkout --force "origin/$BRANCH"
-              git clean -fd
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git checkout "origin/$BRANCH"
-            fi
-            SHA=$(git rev-parse --short=9 HEAD)
-            echo "=== Branch: $BRANCH  SHA: $SHA ==="
-
-            # Auto-detect GPU compute capability for all crates' build.rs
-            export CUDA_COMPUTE_CAP=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1 | tr -d '.')
-            echo "=== GPU arch: sm_${CUDA_COMPUTE_CAP} ==="
-
-            # Clear stale build artifacts when arch changes
-            rm -rf ${CARGO_TARGET_DIR}/build/ml-alpha-* ${CARGO_TARGET_DIR}/build/ml-core-*
-
-            # Compile (~90s cold, ~3s warm after first run)
-            echo "=== Compile ==="
-            time cargo build --release -p ml-alpha --example alpha_rl_train 2>&1
-
-            # Train
-            OUT="/feature-cache/alpha-rl-runs/$SHA/fold{{workflow.parameters.fold-idx}}"
-            mkdir -p "$OUT"
-            echo "=== Train on $(nvidia-smi --query-gpu=name --format=csv,noheader) ==="
-            TRAIN_BIN="${CARGO_TARGET_DIR}/release/examples/alpha_rl_train"
-            if [ "{{workflow.parameters.nsys-profile}}" = "true" ]; then
-              TRAIN_CMD="nsys profile -o $OUT/nsys_trace --stats=true --force-overwrite=true $TRAIN_BIN"
-            else
-              TRAIN_CMD="stdbuf -oL $TRAIN_BIN"
-            fi
-            $TRAIN_CMD \
-              --mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT \
-              --predecoded-dir /feature-cache/predecoded \
-              --out "$OUT" \
-              --n-steps {{workflow.parameters.n-steps}} \
-              --seq-len {{workflow.parameters.seq-len}} \
-              --n-backtests {{workflow.parameters.n-backtests}} \
-              --per-capacity {{workflow.parameters.per-capacity}} \
-              --seed {{workflow.parameters.seed}} \
-              --instrument-mode "{{workflow.parameters.instrument-mode}}" \
-              --fold-idx {{workflow.parameters.fold-idx}} \
-              --n-folds {{workflow.parameters.n-folds}} \
-              --n-eval-steps {{workflow.parameters.n-eval-steps}} \
-              --log-every {{workflow.parameters.log-every}}
-
-            echo "=== Complete: $OUT ==="
-            ls -lh "$OUT/"
-            if [ -f "$OUT/alpha_rl_train_summary.json" ]; then
-              cat "$OUT/alpha_rl_train_summary.json"
-            fi

infra/k8s/argo/archive-rbac.yaml

@@ -1,34 +0,0 @@
-# RBAC for Argo workflow archive — grants controller and server access
-# to the PostgreSQL archive secret and MinIO credentials.
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: argo-archive-secrets
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: argo-workflows
-rules:
-- apiGroups: [""]
-  resources: ["secrets"]
-  resourceNames: ["argo-postgres-config", "minio-credentials"]
-  verbs: ["get"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: argo-archive-secrets
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: argo-workflows
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: argo-archive-secrets
-subjects:
-- kind: ServiceAccount
-  name: argo-workflows-workflow-controller
-  namespace: foxhunt
-- kind: ServiceAccount
-  name: argo-workflows-server
-  namespace: foxhunt

infra/k8s/argo/argo-workflow-netpol.yaml

@@ -1,518 +0,0 @@
-# Base egress for ALL Argo workflow pods: DNS, K8s API, MinIO (logging), Mattermost (notifications).
-# Applies to every pod with app.kubernetes.io/part-of: foxhunt.
-# Component-specific policies below add extra rules (GitLab SSH, external HTTPS, etc.).
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-base-egress
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/part-of: foxhunt
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor reports task results
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # MinIO — log archival (Argo wait sidecar)
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-training-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: training-workflow
-  policyTypes:
-    - Egress
-  egress:
-    # DNS — required for hostname resolution (minio, gitlab, etc.)
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor reports task results
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # HTTPS (external registries, etc.)
-    - ports:
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # GitLab API — binary fetch from packages
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # MinIO — log archival + result upload
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Pushgateway — Prometheus metrics
-    - ports:
-        - port: 9091
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: pushgateway
-    # Tempo — OTLP traces
-    - ports:
-        - port: 4317
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: tempo
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
-    # Container registry — image pulls
-    - ports:
-        - port: 5000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: registry
----
-# Compile workflow pods: need GitLab SSH, MinIO, k8s API, external HTTPS (cargo deps)
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-compile-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: compile
-  policyTypes:
-    - Egress
-  egress:
-    # DNS — resolve external hostnames (crates.io, github.com)
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTPS — cargo fetches crates.io, github.com (candle git dep)
-    - ports:
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone (Tailscale IP + cluster service)
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # MinIO — binary upload
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-# Compile-and-deploy workflow pods: need GitLab HTTP (tagging/releases), MinIO,
-# k8s API (deploy rollouts + Argo executor), DNS, GitLab SSH, external HTTPS.
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-compile-and-deploy-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: compile-and-deploy
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor + kubectl rollout
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTPS — cargo fetches, kubectl download
-    - ports:
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # GitLab webservice — CalVer tagging, releases, package uploads
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # MinIO — binary upload + log archival
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-# Compile-and-train workflow pods: compile step needs GitLab SSH + external HTTPS + DNS,
-# training steps need GitLab HTTP (package fetch/upload), MinIO, Tempo, k8s API.
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-train-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: train
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTPS — cargo fetches crates.io, github.com (candle git dep)
-    - ports:
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # GitLab webservice — package upload/download
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # MinIO — binary upload
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Tempo — OTLP traces
-    - ports:
-        - port: 4317
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: tempo
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-# CI pipeline pods (detect-changes + compile-service inline templates)
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-ci-pipeline
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: ci-pipeline
-  policyTypes:
-    - Egress
-  egress:
-    # DNS — resolve external hostnames
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTP/HTTPS — Kaniko base image pulls, apt-get, cargo fetches
-    - ports:
-        - port: 80
-          protocol: TCP
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone (compile-service uses ssh://git@git.fxhnt.ai)
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # MinIO — binary upload from compile-service
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # GitLab registry — Kaniko image push
-    - ports:
-        - port: 5000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: registry
-    # GitLab webservice — JWT auth for registry tokens
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-# ── sanitizer-test + nsys-test + smoke-test workflows: same egress as ci-pipeline ──
-# (git fetch over SSH, MinIO log archive + nsys-rep upload, registry image pull,
-# external HTTPS for mc download)
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-sanitizer-nsys-workflow
-  namespace: foxhunt
-spec:
-  podSelector:
-    matchExpressions:
-      - key: app.kubernetes.io/component
-        operator: In
-        values:
-          - sanitizer-test
-          - nsys-test
-          - smoke-test
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTPS — mc binary download from dl.min.io
-    - ports:
-        - port: 80
-          protocol: TCP
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git fetch
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # MinIO — log archive + nsys profile upload
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # GitLab registry — ci-builder image pull
-    - ports:
-        - port: 5000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: registry
-    # GitLab webservice — JWT auth for registry tokens
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice

infra/k8s/argo/build-ci-image-template.yaml

@@ -1,121 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: build-ci-image
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: build-ci-image
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  activeDeadlineSeconds: 3600
-  serviceAccountName: argo-workflow
-  entrypoint: build
-  podMetadata:
-    labels:
-      app.kubernetes.io/component: ci-pipeline
-      app.kubernetes.io/part-of: foxhunt
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: dockerfile
-        value: Dockerfile.ci-builder
-      - name: image-name
-        value: ci-builder
-
-  templates:
-    # Single-pod build: init container clones repo, kaniko builds image.
-    # Uses emptyDir — no PVC needed, works when called via templateRef.
-    - name: build
-      inputs:
-        parameters:
-          - name: commit-sha
-            value: "{{workflow.parameters.commit-sha}}"
-          - name: dockerfile
-            value: "{{workflow.parameters.dockerfile}}"
-          - name: image-name
-            value: "{{workflow.parameters.image-name}}"
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      volumes:
-        - name: workspace
-          emptyDir: {}
-        - name: git-ssh-key
-          secret:
-            secretName: argo-git-ssh-key
-            defaultMode: 256
-        - name: registry-auth
-          secret:
-            secretName: gitlab-registry
-            items:
-              - key: .dockerconfigjson
-                path: config.json
-      initContainers:
-        - name: git-clone
-          image: alpine/git:latest
-          command: ["/bin/sh", "-c"]
-          resources:
-            requests:
-              cpu: 200m
-              memory: 256Mi
-            limits:
-              cpu: "1"
-              memory: 512Mi
-          volumeMounts:
-            - name: git-ssh-key
-              mountPath: /etc/git-ssh
-              readOnly: true
-            - name: workspace
-              mountPath: /workspace
-          args:
-            - |
-              set -ex
-              mkdir -p /root/.ssh
-              cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
-              chmod 600 /root/.ssh/id_ed25519
-              printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
-              chmod 600 /root/.ssh/config
-
-              SHA="{{inputs.parameters.commit-sha}}"
-              REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-              git clone --no-checkout --filter=blob:none "$REPO" /workspace/src
-              cd /workspace/src
-              git checkout "$SHA"
-              echo "Checked out $(git rev-parse --short HEAD)"
-      container:
-        image: gcr.io/kaniko-project/executor:debug
-        command: ["/busybox/sh", "-c"]
-        env:
-          - name: DOCKER_CONFIG
-            value: /kaniko/.docker
-        resources:
-          requests:
-            cpu: "4"
-            memory: 8Gi
-          limits:
-            cpu: "8"
-            memory: 16Gi
-        volumeMounts:
-          - name: registry-auth
-            mountPath: /kaniko/.docker
-            readOnly: true
-          - name: workspace
-            mountPath: /workspace
-        args:
-          - |
-            /kaniko/executor \
-              --context=/workspace/src \
-              --dockerfile=/workspace/src/infra/docker/{{inputs.parameters.dockerfile}} \
-              --destination=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/{{inputs.parameters.image-name}}:latest \
-              --insecure-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --skip-tls-verify-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --cache=true \
-              --cache-repo=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/cache \
-              --snapshot-mode=redo

infra/k8s/argo/cargo-target-cuda-test-pvc.yaml

@@ -1,16 +0,0 @@
-# infra/k8s/argo/cargo-target-cuda-test-pvc.yaml
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: cargo-target-cuda-test
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: cargo-target
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 30Gi

infra/k8s/argo/cargo-target-pvcs.yaml

@@ -1,43 +0,0 @@
-# Persistent cargo target directories for incremental compilation.
-#
-# Why: sccache can't cache workspace rlib crates (109 non-cacheable per build).
-# Persisting target/ lets cargo's incremental compilation skip unchanged crates,
-# reducing typical CI builds from ~20 min (full rebuild) to ~2-3 min.
-#
-# Two separate PVCs because compile-services (no cuda feature) and
-# compile-training (cuda feature) produce incompatible artifacts.
-#
-# NOTE: Only one workflow should use each PVC at a time. ci-pipeline and
-# compile-and-train must not run their compile steps concurrently.
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: cargo-target-cpu
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: cargo-target
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 30Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: cargo-target-cuda
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: cargo-target
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 30Gi

infra/k8s/argo/ci-deploy-rbac.yaml

@@ -1,47 +0,0 @@
-# RBAC for CI pipeline — deploy step + self-apply Argo templates
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: ci-deploy
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-rules:
-  # apps: roll + apply Deployments. watch is required by `kubectl rollout status`.
-  - apiGroups: ["apps"]
-    resources: [deployments]
-    verbs: [get, list, watch, patch]
-  - apiGroups: ["argoproj.io"]
-    resources: [workflowtemplates, eventsources, sensors, eventbus]
-    verbs: [get, list, create, update, patch]
-  # core: services/configmaps, plus serviceaccounts (tailscale-dashboard, dagster) and the forward-track PVC
-  # — added so the fxhnt-cockpit deploy can `kubectl apply` its full manifest set without a partial-apply failure.
-  - apiGroups: [""]
-    resources: [services, configmaps, serviceaccounts, persistentvolumeclaims]
-    verbs: [get, list, create, update, patch]
-  - apiGroups: ["networking.k8s.io"]
-    resources: [networkpolicies]
-    verbs: [get, list, create, update, patch]
-  # batch: the fxhnt-forward CronJob
-  - apiGroups: ["batch"]
-    resources: [cronjobs]
-    verbs: [get, list, create, update, patch]
-  - apiGroups: ["rbac.authorization.k8s.io"]
-    resources: [roles, rolebindings]
-    verbs: [get, list, create, update, patch, bind, escalate]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: ci-deploy
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: ci-deploy
-subjects:
-  - kind: ServiceAccount
-    name: argo-workflow
-    namespace: foxhunt

infra/k8s/argo/ci-pipeline-template.yaml

@@ -1,685 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: ci-pipeline
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: ci-pipeline
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  activeDeadlineSeconds: 7200
-  serviceAccountName: argo-workflow
-  entrypoint: pipeline
-  onExit: notify-result
-  podMetadata:
-    labels:
-      app.kubernetes.io/component: ci-pipeline
-      app.kubernetes.io/part-of: foxhunt
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: registry-auth
-      secret:
-        secretName: gitlab-registry
-        optional: true
-        items:
-          - key: .dockerconfigjson
-            path: config.json
-    - name: cargo-target-cpu
-      persistentVolumeClaim:
-        claimName: cargo-target-cpu
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: commits-json
-        value: "[]"
-  templates:
-    # ── DAG: orchestrate pipeline ──
-    # Policy (user directive): push-triggered runs are limited to infrastructure
-    # sync work — docker image rebuilds + argo-template apply + terragrunt apply.
-    # These run automatically when the relevant paths change:
-    #   - rebuild-ci-builder* / rebuild-runtime / rebuild-training-runtime: gated on
-    #     `detect-changes.docker-images == true` (infra/docker/)
-    #   - apply-argo-templates: gated on `detect-changes.needs-argo-templates` (infra/k8s/argo/)
-    #   - terragrunt-apply:     gated on `detect-changes.needs-infra` (infra/live/, infra/modules/)
-    # All other tasks (test-gate, build-web-dashboard, gpu-test) are `when: "false"` —
-    # they run via manual script invocation (`argo submit` / `./scripts/argo-*.sh`)
-    # or the manual `/compile-deploy` webhook. Training + compile/deploy workflows
-    # are separate templates, always manual.
-    - name: pipeline
-      dag:
-        tasks:
-          - name: detect-changes
-            template: detect-changes
-
-          - name: build-web-dashboard
-            depends: "detect-changes"
-            template: build-web-dashboard
-            # Policy: push-triggered runs are docker image rebuilds only.
-            # Dashboard builds run via manual `argo submit` (scripts).
-            when: "false"
-
-          - name: rebuild-ci-builder
-            depends: "detect-changes"
-            templateRef:
-              name: build-ci-image
-              template: build
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: dockerfile
-                  value: Dockerfile.ci-builder
-                - name: image-name
-                  value: ci-builder
-            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
-
-          - name: rebuild-ci-builder-cpu
-            depends: "detect-changes"
-            templateRef:
-              name: build-ci-image
-              template: build
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: dockerfile
-                  value: Dockerfile.ci-builder-cpu
-                - name: image-name
-                  value: ci-builder-cpu
-            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
-
-          - name: rebuild-runtime
-            depends: "detect-changes"
-            templateRef:
-              name: build-ci-image
-              template: build
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: dockerfile
-                  value: Dockerfile.foxhunt-runtime
-                - name: image-name
-                  value: foxhunt-runtime
-            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
-
-          - name: rebuild-training-runtime
-            depends: "detect-changes"
-            templateRef:
-              name: build-ci-image
-              template: build
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: dockerfile
-                  value: Dockerfile.foxhunt-training-runtime
-                - name: image-name
-                  value: foxhunt-training-runtime
-            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
-
-          - name: test-gate
-            depends: "detect-changes && (rebuild-ci-builder-cpu.Succeeded || rebuild-ci-builder-cpu.Skipped)"
-            template: test-gate
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-            # Policy: push-triggered runs are docker image rebuilds only.
-            # Cargo test + clippy quality gate runs via manual `argo submit`
-            # (scripts/argo-test.sh or CI-driven `argo submit --from=wftmpl/ci-pipeline`).
-            when: "false"
-
-          - name: apply-argo-templates
-            depends: "detect-changes.Succeeded && (rebuild-ci-builder-cpu.Succeeded || rebuild-ci-builder-cpu.Skipped)"
-            template: apply-argo-templates
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-            when: "{{tasks.detect-changes.outputs.parameters.needs-argo-templates}} == true"
-
-          - name: terragrunt-apply
-            depends: "detect-changes.Succeeded && (rebuild-ci-builder-cpu.Succeeded || rebuild-ci-builder-cpu.Skipped)"
-            template: terragrunt-apply
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-            when: "{{tasks.detect-changes.outputs.parameters.needs-infra}} == true"
-
-          # GPU tests: manual only (argo submit --from=wftmpl/gpu-test-pipeline).
-          # Auto-trigger disabled — H100 nodes are expensive, run on demand.
-          - name: gpu-test
-            depends: "detect-changes"
-            template: submit-gpu-test
-            when: "false"
-
-    # ── detect-changes: classify changed files to gate downstream steps ──
-    - name: detect-changes
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: COMMITS_JSON
-            value: "{{workflow.parameters.commits-json}}"
-        resources:
-          requests:
-            cpu: 100m
-            memory: 128Mi
-          limits:
-            cpu: 500m
-            memory: 256Mi
-        args:
-          - |
-            set -e
-            cat <<'SCRIPT' > /tmp/detect.sh
-            #!/bin/sh
-            set -e
-
-            # Extract changed file paths from webhook JSON using grep+sed (no jq dependency).
-            CHANGED_FILES=$(echo "$COMMITS_JSON" \
-              | grep -oE '"(added|modified|removed)":\[[^]]*\]' \
-              | sed 's/"added"://;s/"modified"://;s/"removed"://' \
-              | tr ',' '\n' | tr -d '[]"' | sed '/^$/d' | sort -u \
-              || echo "")
-
-            if [ -z "$CHANGED_FILES" ]; then
-              echo "No changed files detected — triggering full rebuild"
-              CHANGED_FILES="Cargo.toml"
-            fi
-
-            echo "=== Changed files ==="
-            echo "$CHANGED_FILES"
-            echo "===================="
-
-            # --- Helper: check if any changed file matches a set of path prefixes ---
-            check_paths() {
-              local patterns="$1"
-              for file in $CHANGED_FILES; do
-                for pattern in $patterns; do
-                  case "$file" in
-                    ${pattern}*) echo "true"; return ;;
-                  esac
-                done
-              done
-              echo "false"
-            }
-
-            # --- Boolean gates for downstream DAG tasks ---
-            NEEDS_CODE=$(check_paths "Cargo.toml Cargo.lock crates/ services/ bin/fxt/")
-            NEEDS_DASHBOARD=$(check_paths "web-dashboard/")
-            DOCKER_IMAGES=$(check_paths "infra/docker/")
-            NEEDS_INFRA=$(check_paths "infra/live/ infra/modules/")
-            NEEDS_ARGO_TEMPLATES=$(check_paths "infra/k8s/argo/")
-            ML_CHANGED=$(check_paths "crates/ml crates/ml-dqn crates/ml-core crates/ml-regime crates/ml-features crates/ml-supervised crates/ml-ppo crates/ml-ensemble crates/ml-hyperopt crates/ml-labeling config/training config/gpu")
-
-            echo "=== Build decisions ==="
-            echo "needs-code:        $NEEDS_CODE"
-            echo "needs-dashboard:   $NEEDS_DASHBOARD"
-            echo "docker-images:     $DOCKER_IMAGES"
-            echo "needs-infra:       $NEEDS_INFRA"
-            echo "needs-argo:        $NEEDS_ARGO_TEMPLATES"
-            echo "ml-changed:        $ML_CHANGED"
-            echo "======================"
-
-            mkdir -p /tmp/outputs
-            echo -n "$NEEDS_CODE"        > /tmp/outputs/needs-code
-            echo -n "$NEEDS_DASHBOARD"   > /tmp/outputs/needs-dashboard
-            echo -n "$DOCKER_IMAGES"     > /tmp/outputs/docker-images
-            echo -n "$NEEDS_INFRA"       > /tmp/outputs/needs-infra
-            echo -n "$NEEDS_ARGO_TEMPLATES" > /tmp/outputs/needs-argo-templates
-            echo -n "$ML_CHANGED"          > /tmp/outputs/ml-changed
-            SCRIPT
-            chmod +x /tmp/detect.sh
-            /tmp/detect.sh
-      outputs:
-        parameters:
-          - name: needs-code
-            valueFrom:
-              path: /tmp/outputs/needs-code
-          - name: needs-dashboard
-            valueFrom:
-              path: /tmp/outputs/needs-dashboard
-          - name: docker-images
-            valueFrom:
-              path: /tmp/outputs/docker-images
-          - name: needs-infra
-            valueFrom:
-              path: /tmp/outputs/needs-infra
-          - name: needs-argo-templates
-            valueFrom:
-              path: /tmp/outputs/needs-argo-templates
-          - name: ml-changed
-            valueFrom:
-              path: /tmp/outputs/ml-changed
-
-    # ── test-gate: clippy + cargo test quality gate ──
-    # Uses ci-builder (with CUDA toolkit) because cudarc 0.19 requires nvcc at build time.
-    - name: test-gate
-      inputs:
-        parameters:
-          - name: commit-sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: k8s.scaleway.com/pool-name
-          operator: Equal
-          value: ci-compile-cpu
-          effect: NoSchedule
-      sidecars:
-        - name: redis
-          image: redis:7-alpine
-          command: ["redis-server", "--save", "", "--appendonly", "no"]
-          ports:
-            - containerPort: 6379
-          resources:
-            requests:
-              cpu: 100m
-              memory: 64Mi
-            limits:
-              cpu: 500m
-              memory: 128Mi
-          readinessProbe:
-            tcpSocket:
-              port: 6379
-            initialDelaySeconds: 1
-            periodSeconds: 1
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: REDIS_URL
-            value: "redis://localhost:6379"
-        resources:
-          requests:
-            cpu: "14"
-            memory: 16Gi
-          limits:
-            cpu: "30"
-            memory: 32Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cpu
-            mountPath: /cargo-target
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.commit-sha}}"
-
-            # SSH setup
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            WORKSPACE="/cargo-target/src"
-
-            git config --global --add safe.directory "$WORKSPACE"
-
-            if [ -d "$WORKSPACE/.git" ]; then
-              cd "$WORKSPACE"
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" = "$SHA" ]; then
-                echo "=== Already at $SHA ==="
-              else
-                git fetch origin
-                git checkout --force "$SHA"
-                git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$WORKSPACE"
-              cd "$WORKSPACE"
-              git checkout "$SHA"
-            fi
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-
-            echo "=== Waiting for Redis sidecar ==="
-            for i in 1 2 3 4 5; do
-              if printf 'PING\r\n' | nc -w1 127.0.0.1 6379 2>/dev/null | grep -q PONG; then
-                echo "Redis ready"
-                break
-              fi
-              sleep 1
-            done
-
-            echo "=== Running clippy (lib targets) ==="
-            cargo clippy --workspace --lib -- -D warnings 2>&1 | tee /cargo-target/clippy.log
-
-            echo "=== Running tests (lib only, no integration) ==="
-            set +e
-            cargo test --workspace --lib 2>&1 | tee /cargo-target/test-output.log
-            TEST_EXIT=$?
-            set -e
-            if [ "$TEST_EXIT" -ne 0 ]; then
-              echo "=== TEST FAILURES ==="
-              grep -A5 'FAILED\|panicked\|test result: FAILED\|error\[' /cargo-target/test-output.log || true
-              exit "$TEST_EXIT"
-            fi
-
-            echo "=== Test gate passed ==="
-
-    # ── build-web-dashboard: npm build + upload to MinIO ──
-    - name: build-web-dashboard
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: node:22-alpine
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: MINIO_ACCESS_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: access-key
-          - name: MINIO_SECRET_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: secret-key
-        resources:
-          requests:
-            cpu: "1"
-            memory: 1Gi
-          limits:
-            cpu: "2"
-            memory: 2Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-
-            # Git clone
-            apk add --no-cache git openssh rclone
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            WORKSPACE="/tmp/workspace"
-            git clone --no-checkout --filter=blob:none "$REPO" "$WORKSPACE"
-            cd "$WORKSPACE"
-            git checkout "$SHA"
-
-            cd "$WORKSPACE/web-dashboard"
-            echo "=== Building web dashboard ==="
-            npm ci
-            npm run build
-
-            echo "=== Uploading to MinIO ==="
-            rclone copy dist/ :s3:foxhunt-binaries/web-dashboard/ \
-              --s3-provider=Minio \
-              --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
-              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
-              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
-              --s3-no-check-bucket
-
-            echo "=== Web dashboard build + upload done ==="
-
-    # ── terragrunt-apply: apply infra changes on main push ──
-    - name: terragrunt-apply
-      inputs:
-        parameters:
-          - name: commit-sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      serviceAccountName: argo-workflow
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu:latest
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: GITLAB_TOKEN
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-          - name: TF_HTTP_USERNAME
-            value: root
-          - name: TF_HTTP_PASSWORD
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-          - name: SCW_ACCESS_KEY
-            valueFrom:
-              secretKeyRef:
-                name: scaleway-credentials
-                key: access-key
-          - name: SCW_SECRET_KEY
-            valueFrom:
-              secretKeyRef:
-                name: scaleway-credentials
-                key: secret-key
-          - name: SCW_DEFAULT_PROJECT_ID
-            valueFrom:
-              secretKeyRef:
-                name: scaleway-credentials
-                key: project-id
-          - name: GITLAB_TF_STATE_URL
-            value: "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-        resources:
-          requests:
-            cpu: 200m
-            memory: 256Mi
-          limits:
-            cpu: "1"
-            memory: 512Mi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-        args:
-          - |
-            set -e
-
-            # Clone repo
-            mkdir -p /root/.ssh
-            cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
-            chmod 600 /root/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
-            chmod 600 /root/.ssh/config
-
-            SHA="{{inputs.parameters.commit-sha}}"
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            git clone --no-checkout --filter=blob:none "$REPO" /workspace/src
-            cd /workspace/src
-            git checkout "$SHA"
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            tofu --version
-            terragrunt --version
-
-            # Apply each terragrunt module
-            for module in block-storage dns public-gateway kapsule; do
-              MODULE_DIR="infra/live/production/${module}"
-              if [ -d "$MODULE_DIR" ]; then
-                echo "=== Terragrunt plan: ${module} ==="
-                cd "/workspace/src/${MODULE_DIR}"
-                terragrunt init --non-interactive -reconfigure
-                OUTPUT=$(terragrunt plan --non-interactive -detailed-exitcode 2>&1) || EXITCODE=$?
-                EXITCODE=${EXITCODE:-0}
-
-                if [ "$EXITCODE" -eq 2 ]; then
-                  echo "=== Terragrunt plan output: ${module} ==="
-                  echo "$OUTPUT"
-                  echo "=== Terragrunt apply: ${module} ==="
-                  terragrunt apply --non-interactive -auto-approve
-                elif [ "$EXITCODE" -eq 0 ]; then
-                  echo "=== No changes for ${module} ==="
-                else
-                  echo "=== ERROR planning ${module} ==="
-                  echo "$OUTPUT"
-                  exit 1
-                fi
-                cd /workspace/src
-              fi
-            done
-
-            echo "=== Terragrunt apply complete ==="
-
-    # ── apply-argo-templates: self-apply WorkflowTemplates, EventSources, Sensors ──
-    - name: apply-argo-templates
-      inputs:
-        parameters:
-          - name: commit-sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      serviceAccountName: argo-workflow
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu:latest
-        command: ["/bin/sh", "-c"]
-        resources:
-          requests:
-            cpu: 100m
-            memory: 128Mi
-          limits:
-            cpu: 500m
-            memory: 256Mi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-        args:
-          - |
-            set -e
-
-            # Clone repo at commit
-            mkdir -p /root/.ssh
-            cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
-            chmod 600 /root/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
-            chmod 600 /root/.ssh/config
-
-            SHA="{{inputs.parameters.commit-sha}}"
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            git clone --no-checkout --filter=blob:none "$REPO" /workspace/src
-            cd /workspace/src
-            git checkout "$SHA"
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            # Apply all Argo WorkflowTemplates
-            for f in infra/k8s/argo/*-template.yaml; do
-              if [ -f "$f" ]; then
-                echo "=== Applying $(basename $f) ==="
-                kubectl -n foxhunt apply -f "$f"
-              fi
-            done
-
-            # Apply EventSources and Sensors
-            for f in infra/k8s/argo/events/*.yaml; do
-              if [ -f "$f" ]; then
-                echo "=== Applying $(basename $f) ==="
-                kubectl -n foxhunt apply -f "$f"
-              fi
-            done
-
-            # Apply RBAC and NetworkPolicies
-            kubectl -n foxhunt apply -f infra/k8s/argo/ci-deploy-rbac.yaml
-            kubectl -n foxhunt apply -f infra/k8s/argo/argo-workflow-netpol.yaml
-
-            echo "=== Argo templates applied ==="
-
-    # ── submit-gpu-test: launch GPU test workflow as a child workflow ──
-    - name: submit-gpu-test
-      # Argo executor (main + wait) needs 256Mi to track large child workflow status JSON.
-      # Default 64Mi causes OOMKilled when child workflow has many parameters.
-      podSpecPatch: '{"containers":[{"name":"main","resources":{"requests":{"memory":"128Mi"},"limits":{"memory":"256Mi"}}},{"name":"wait","resources":{"requests":{"memory":"128Mi"},"limits":{"memory":"256Mi"}}}]}'
-      resource:
-        action: create
-        setOwnerReference: true
-        successCondition: status.phase == Succeeded
-        failureCondition: status.phase in (Failed, Error)
-        manifest: |
-          apiVersion: argoproj.io/v1alpha1
-          kind: Workflow
-          metadata:
-            generateName: gpu-test-on-push-
-            namespace: foxhunt
-          spec:
-            workflowTemplateRef:
-              name: gpu-test-pipeline
-            arguments:
-              parameters:
-                - name: commit-ref
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: models
-                  value: "dqn,ppo,tft"
-
-    # ── notify-result: post workflow outcome to Mattermost ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"
-              exit 0
-            fi
-
-            if [ "$STATUS" = "Succeeded" ]; then
-              EMOJI=":white_check_mark:"
-            else
-              EMOJI=":x:"
-            fi
-
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **${NAME}** — ${STATUS} ({{workflow.duration}}s)\"}"
-
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/compile-and-deploy-template.yaml

@@ -1,614 +0,0 @@
-# Compile-and-Deploy: manual workflow for building and deploying service binaries.
-#
-# DAG:
-#   create-tag ──> compile-services ──> upload-release ──> deploy-services
-#
-# Image builds handled by ci-pipeline, not here.
-# Training compilation lives in compile-and-train-template.yaml.
----
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: compile-and-deploy
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: compile-and-deploy
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: pipeline
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: compile-and-deploy
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 3600  # 1 hour (compile + deploy)
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: service-packages
-        value: "api trading-service backtesting-service trading-agent-service broker-gateway data-acquisition-service"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target-cpu
-      persistentVolumeClaim:
-        claimName: cargo-target-cpu
-    - name: sccache
-      persistentVolumeClaim:
-        claimName: sccache-cpu
-
-  templates:
-    - name: pipeline
-      dag:
-        tasks:
-          - name: create-tag
-            template: create-tag
-          - name: compile-services
-            depends: "create-tag"
-            template: compile-services
-            arguments:
-              parameters:
-                - name: tag
-                  value: "{{tasks.create-tag.outputs.parameters.tag}}"
-                - name: service-packages
-                  value: "{{workflow.parameters.service-packages}}"
-          - name: upload-release
-            depends: "compile-services"
-            template: upload-release
-            arguments:
-              parameters:
-                - name: tag
-                  value: "{{tasks.create-tag.outputs.parameters.tag}}"
-          - name: deploy-services
-            depends: "upload-release"
-            template: deploy-services
-            arguments:
-              parameters:
-                - name: tag
-                  value: "{{tasks.create-tag.outputs.parameters.tag}}"
-                - name: deploy-list
-                  value: "{{workflow.parameters.service-packages}}"
-
-    # ── create-tag: CalVer auto-tag on code changes ──
-    - name: create-tag
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            cpu: 200m
-            memory: 128Mi
-        args:
-          - |
-            set -e
-            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-            PROJECT_ID=1
-            SHA="{{workflow.parameters.commit-sha}}"
-
-            # Compute CalVer prefix: vYYYY.MM
-            PREFIX="v$(date +%Y.%m)"
-
-            # Query existing tags for this month
-            TAGS=$(curl -sf \
-              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?search=${PREFIX}" \
-              || echo "[]")
-
-            # Parse highest N from vYYYY.MM.N tags
-            LAST_N=$(echo "$TAGS" | grep -oP "\"name\":\"${PREFIX}\.\K[0-9]+" | sort -n | tail -1)
-            if [ -z "$LAST_N" ]; then
-              NEXT_N=1
-            else
-              NEXT_N=$((LAST_N + 1))
-            fi
-
-            TAG="${PREFIX}.${NEXT_N}"
-            echo "Creating tag: ${TAG} at ${SHA}"
-
-            # Create the tag (tolerate failure if tag already exists)
-            HTTP_CODE=$(curl -s -o /tmp/tag_response -w "%{http_code}" -X POST \
-              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              -d "tag_name=${TAG}" \
-              -d "ref=${SHA}" \
-              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags")
-
-            RESULT=$(cat /tmp/tag_response 2>/dev/null || echo "{}")
-
-            if [ "$HTTP_CODE" = "201" ]; then
-              echo "Tag ${TAG} created successfully"
-            elif [ "$HTTP_CODE" = "400" ] && echo "$RESULT" | grep -q "already exists"; then
-              echo "Tag ${TAG} already exists — reusing"
-            else
-              echo "WARN: Tag creation returned HTTP ${HTTP_CODE}: ${RESULT}"
-              echo "Falling back to dev tag"
-              TAG="dev-$(echo $SHA | cut -c1-8)"
-            fi
-
-            mkdir -p /tmp/outputs
-            echo -n "$TAG" > /tmp/outputs/tag
-      outputs:
-        parameters:
-          - name: tag
-            valueFrom:
-              path: /tmp/outputs/tag
-
-    # ── compile-services: selective per-binary build, incremental on local RWO PVC ──
-    - name: compile-services
-      metadata:
-        labels:
-          app.kubernetes.io/component: compile
-      inputs:
-        parameters:
-          - name: tag
-          - name: service-packages
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      # ── seed-deps-cache: rsync prebuilt third-party rlibs into the PVC ──
-      # On a true cold PVC (e.g. fresh node, first run, or after a purge),
-      # this saves 2-4 min that would otherwise be spent compiling the
-      # ~1500 third-party crates in the workspace dep graph. The image is
-      # rebuilt nightly by refresh-deps-cache (see
-      # refresh-deps-cache-template.yaml). On a warm PVC, the stamp file
-      # short-circuits the whole step in ~50ms.
-      #
-      # Why rsync --ignore-existing instead of cp/overwrite: a warmer PVC
-      # may already have NEWER artifacts from a prior compile; we don't
-      # want to clobber them with potentially-stale prebuilt rlibs. Cargo
-      # will discard rlibs whose fingerprint mismatches anyway.
-      #
-      # If the deps-cache image isn't published yet (first deploy of this
-      # template), set imagePullPolicy: IfNotPresent + the initContainer
-      # is non-blocking-on-failure (`|| true`) so the compile-services
-      # step still runs without the cache prelude.
-      initContainers:
-        - name: seed-deps-cache
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:nightly
-          imagePullPolicy: IfNotPresent
-          env:
-            - name: DEPS_VERSION
-              value: "1"
-          resources:
-            requests:
-              cpu: 500m
-              memory: 512Mi
-            limits:
-              cpu: "2"
-              memory: 1Gi
-          volumeMounts:
-            - name: cargo-target-cpu
-              mountPath: /cargo-target
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              STAMP="/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp"
-              if [ -f "$STAMP" ]; then
-                echo "PVC already seeded for deps v${DEPS_VERSION} (stamp present), skipping."
-                exit 0
-              fi
-              if [ ! -d /cargo-target-prebuilt ]; then
-                echo "WARN: /cargo-target-prebuilt missing in image — image not yet published?"
-                echo "      Compile will run without prebuilt deps cache (slower cold-start)."
-                exit 0
-              fi
-              echo "=== Seeding cargo-target-cpu PVC from prebuilt deps cache (v${DEPS_VERSION}) ==="
-              du -sh /cargo-target-prebuilt 2>/dev/null || true
-              # --ignore-existing: never clobber newer artifacts already on PVC.
-              # -a: preserve mtimes, perms, links — cargo's freshness check needs accurate mtimes.
-              rsync -a --ignore-existing /cargo-target-prebuilt/ /cargo-target/
-              touch "$STAMP"
-              echo "=== Seed complete ==="
-              du -sh /cargo-target/release 2>/dev/null || true
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu:latest
-        imagePullPolicy: Always
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          # sccache: rustc wrapper that content-hashes compile inputs and reuses
-          # object output across pods/commits via the sccache-cpu PVC. Survives
-          # cargo's incremental-cache invalidation from git checkout mtime touches.
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /sccache-cache
-          - name: SCCACHE_CACHE_SIZE
-            value: 20G
-          # Required: workspace `.cargo/config.toml` sets [build] incremental = true,
-          # which forces incremental compilation even on --release.  sccache cannot
-          # cache incremental rustc output (documented limitation).  Override here
-          # so sccache actually catches.  Local dev keeps incremental via config.toml.
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          # Match the cgroup cpu limit below (limits.cpu: "30").
-          # Without this, cargo asks num_cpus::get_physical() — the host's
-          # whole CPU count — and over-subscribes vs the cgroup throttle,
-          # producing scheduling waste under load.
-          - name: CARGO_BUILD_JOBS
-            value: "30"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 16Gi
-          limits:
-            cpu: "30"
-            memory: 32Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cpu
-            mountPath: /cargo-target
-          - name: sccache
-            mountPath: /sccache-cache
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            SERVICE_PKGS="{{inputs.parameters.service-packages}}"
-
-            # SSH setup
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            WORKSPACE="/cargo-target/src"
-
-            # PVC may be owned by a different UID from a previous run
-            git config --global --add safe.directory "$WORKSPACE"
-
-            # Persistent checkout on PVC — only changed files get new mtimes,
-            # so cargo skips recompiling unchanged workspace crates.
-            if [ -d "$WORKSPACE/.git" ]; then
-              cd "$WORKSPACE"
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" = "$SHA" ]; then
-                echo "=== Already at $SHA, skipping checkout ==="
-              else
-                echo "=== Updating checkout: $(echo $CURRENT | cut -c1-8) -> $(echo $SHA | cut -c1-8) ==="
-                git fetch origin
-                git checkout --force "$SHA"
-                git clean -fd
-              fi
-            else
-              echo "=== Initial clone (first run) ==="
-              git clone --filter=blob:none "$REPO" "$WORKSPACE"
-              cd "$WORKSPACE"
-              git checkout "$SHA"
-            fi
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            # Ensure cargo home registry is on PVC
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-
-            export FOXHUNT_BUILD_VERSION="{{inputs.parameters.tag}}"
-
-            # Prune build artifacts if PVC exceeds 25GB (prevents unbounded growth)
-            TARGET_SIZE_MB=$(du -sm "$CARGO_TARGET_DIR" 2>/dev/null | cut -f1 || echo 0)
-            echo "PVC usage: ${TARGET_SIZE_MB}MB"
-            if [ "$TARGET_SIZE_MB" -gt 25000 ]; then
-              echo "PVC exceeds 25GB limit, pruning build artifacts..."
-              rm -rf "$CARGO_TARGET_DIR/release" "$CARGO_TARGET_DIR/debug"
-            fi
-
-            # Guard: empty package list would build entire workspace
-            if [ -z "$SERVICE_PKGS" ]; then
-              echo "ERROR: service-packages is empty, refusing to build entire workspace"
-              exit 1
-            fi
-
-            # Build only the affected service packages (incremental via persistent target dir)
-            CARGO_ARGS=""
-            for pkg in $SERVICE_PKGS; do
-              CARGO_ARGS="$CARGO_ARGS -p $pkg"
-            done
-
-            # CI-only linker swap: prefer wild over mold. wild is a Rust-native
-            # linker, typically 10-30% faster than mold on release-LTO links;
-            # we have ~5 service binaries each doing release-LTO, so this is
-            # a meaningful 30-90 sec wall-time saving. Both linkers ship in
-            # Dockerfile.ci-builder-cpu — to revert, drop this sed and rebuild.
-            # Sed is idempotent (no-op if wild already substituted from a prior
-            # run on the same PVC checkout) and surgical (single line in
-            # .cargo/config.toml). If wild is missing for any reason, the build
-            # falls back to mold once we revert this hunk.
-            if command -v wild >/dev/null 2>&1; then
-              echo "=== Swapping linker mold -> wild for this CI run ==="
-              sed -i 's|-fuse-ld=mold|-fuse-ld=wild|' .cargo/config.toml
-              grep -n 'fuse-ld' .cargo/config.toml || true
-            else
-              echo "WARN: wild not on PATH, sticking with mold"
-            fi
-
-            echo "=== Building service binaries: $SERVICE_PKGS (incremental) ==="
-            # --locked: skip Cargo.lock resolver work (~5-15s saved) and fail fast
-            # if the lock has drifted (catches accidental Cargo.toml edits without
-            # a re-resolve commit).
-            cargo build --locked --release $CARGO_ARGS
-
-            # Collect built binaries
-            mkdir -p "$WORKSPACE/bin/services"
-            for pkg in $SERVICE_PKGS; do
-              bin_name=$(echo "$pkg" | tr '-' '_')
-              cp "$CARGO_TARGET_DIR/release/$pkg" "$WORKSPACE/bin/services/" 2>/dev/null \
-                || cp "$CARGO_TARGET_DIR/release/$bin_name" "$WORKSPACE/bin/services/" 2>/dev/null \
-                || { echo "Binary not found for $pkg"; ls "$CARGO_TARGET_DIR/release/"; exit 1; }
-            done
-            strip "$WORKSPACE/bin/services/"*
-
-            echo "=== Service binaries ==="
-            ls -lh "$WORKSPACE/bin/services/"
-
-            echo "=== Uploading service binaries to GitLab packages ==="
-            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-            TAG="${FOXHUNT_BUILD_VERSION}"
-            for bin in "$WORKSPACE/bin/services/"*; do
-              BIN_NAME=$(basename "$bin")
-              echo "Uploading ${BIN_NAME} (${TAG})..."
-              curl -f --upload-file "$bin" \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-services/${TAG}/${BIN_NAME}"
-            done
-
-            # Update 'latest' per-file (preserves unbuilt binaries from prior runs)
-            echo "=== Updating 'latest' package ==="
-            LATEST_PKG=$(curl -sf -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              "${GITLAB}/api/v4/projects/1/packages?package_name=foxhunt-services&package_version=latest" \
-              | grep -oP '"id":\K[0-9]+' | head -1)
-            for bin in "$WORKSPACE/bin/services/"*; do
-              BIN_NAME=$(basename "$bin")
-              # Delete existing file by name before uploading replacement
-              if [ -n "$LATEST_PKG" ]; then
-                FILE_ID=$(curl -sf -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                  "${GITLAB}/api/v4/projects/1/packages/${LATEST_PKG}/package_files" \
-                  | grep -oP "\"id\":([0-9]+),\"package_id\":${LATEST_PKG}[^}]*\"file_name\":\"${BIN_NAME}\"" \
-                  | grep -oP '"id":\K[0-9]+' | head -1)
-                if [ -n "$FILE_ID" ]; then
-                  curl -sf -X DELETE -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                    "${GITLAB}/api/v4/projects/1/packages/${LATEST_PKG}/package_files/${FILE_ID}" || true
-                fi
-              fi
-              curl -f --upload-file "$bin" \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-services/latest/${BIN_NAME}" || true
-            done
-
-            echo "=== sccache stats ==="
-            sccache --show-stats || true
-
-            echo "=== Service compile + upload done ($SERVICE_PKGS) ==="
-
-    # ── upload-release: create GitLab Release with package links ──
-    - name: upload-release
-      inputs:
-        parameters:
-          - name: tag
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            cpu: 200m
-            memory: 128Mi
-        args:
-          - |
-            set -e
-            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-            PROJECT_ID=1
-            TAG="{{inputs.parameters.tag}}"
-
-            echo "=== Creating GitLab Release ${TAG} ==="
-
-            # Get commits since previous tag for release notes
-            PREV_TAG=$(curl -sf \
-              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?per_page=2" \
-              | grep -oP '"name":"\K[^"]+' | sed -n '2p')
-
-            if [ -n "$PREV_TAG" ]; then
-              COMMITS=$(curl -sf \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/compare?from=${PREV_TAG}&to=${TAG}" \
-                | grep -oP '"title":"\K[^"]+' | head -20 \
-                | sed 's/^/- /' || echo "- Release ${TAG}")
-              DESCRIPTION="## Changes since ${PREV_TAG}\n\n${COMMITS}"
-            else
-              DESCRIPTION="## Initial release\n\nFirst CalVer release."
-            fi
-
-            # Create the release
-            RESPONSE=$(curl -sf -X POST \
-              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              -H "Content-Type: application/json" \
-              -d "{
-                \"tag_name\": \"${TAG}\",
-                \"name\": \"${TAG}\",
-                \"description\": \"$(printf '%s' "$DESCRIPTION" | sed 's/"/\\"/g')\"
-              }" \
-              "${GITLAB}/api/v4/projects/${PROJECT_ID}/releases") || {
-              echo "WARN: Release creation failed (may already exist)"
-            }
-
-            echo "Release ${TAG} created"
-            echo "$RESPONSE" | head -5
-
-            # Add package links as release assets
-            for pkg_name in foxhunt-services foxhunt-training; do
-              PKG_CHECK=$(curl -sf \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB}/api/v4/projects/${PROJECT_ID}/packages?package_name=${pkg_name}&package_version=${TAG}" \
-                || echo "[]")
-
-              if echo "$PKG_CHECK" | grep -q "$TAG"; then
-                curl -sf -X POST \
-                  -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                  -H "Content-Type: application/json" \
-                  -d "{
-                    \"name\": \"${pkg_name}\",
-                    \"url\": \"${GITLAB}/-/packages?type=generic&search=${pkg_name}&version=${TAG}\",
-                    \"link_type\": \"package\"
-                  }" \
-                  "${GITLAB}/api/v4/projects/${PROJECT_ID}/releases/${TAG}/assets/links" || true
-                echo "Linked ${pkg_name} package to release"
-              fi
-            done
-
-            echo "=== Release ${TAG} complete ==="
-
-    # ── deploy-services: selective rolling restart for affected deployments ──
-    - name: deploy-services
-      inputs:
-        parameters:
-          - name: tag
-          - name: deploy-list
-      serviceAccountName: argo-workflow
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        resources:
-          requests:
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            cpu: 200m
-            memory: 128Mi
-        args:
-          - |
-            set -e
-            if ! command -v kubectl >/dev/null 2>&1; then
-              echo "=== Installing kubectl ==="
-              curl -sLo /tmp/kubectl "https://dl.k8s.io/release/v1.31.4/bin/linux/amd64/kubectl"
-              chmod +x /tmp/kubectl
-              export PATH="/tmp:$PATH"
-            fi
-
-            TAG="{{inputs.parameters.tag}}"
-            DEPLOY_LIST="{{inputs.parameters.deploy-list}}"
-
-            if [ -z "$DEPLOY_LIST" ]; then
-              echo "=== No services to deploy ==="
-              exit 0
-            fi
-
-            echo "=== Deploying release ${TAG}: $DEPLOY_LIST ==="
-
-            for svc in $DEPLOY_LIST; do
-              echo "Patching $svc with FOXHUNT_RELEASE=${TAG}..."
-              kubectl -n foxhunt patch deployment "$svc" -p "{
-                \"spec\":{\"template\":{
-                  \"metadata\":{\"annotations\":{\"foxhunt.io/release\":\"${TAG}\"}},
-                  \"spec\":{\"initContainers\":[{
-                    \"name\":\"fetch-binary\",
-                    \"env\":[{\"name\":\"FOXHUNT_RELEASE\",\"value\":\"${TAG}\"}]
-                  }]}
-                }}
-              }" || echo "WARN: $svc patch failed (may not exist)"
-            done
-
-            echo "=== Waiting for rollouts ==="
-            for svc in $DEPLOY_LIST; do
-              kubectl -n foxhunt rollout status deployment "$svc" --timeout=120s || echo "WARN: $svc rollout timeout"
-            done
-
-            echo "=== Deploy ${TAG} complete ($DEPLOY_LIST) ==="
-
-    # ── notify-result: post workflow outcome to Mattermost ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"
-              exit 0
-            fi
-
-            if [ "$STATUS" = "Succeeded" ]; then
-              EMOJI=":white_check_mark:"
-            else
-              EMOJI=":x:"
-            fi
-
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **${NAME}** — ${STATUS} ({{workflow.duration}}s)\"}"
-
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/databento-download-template.yaml

@@ -1,135 +0,0 @@
-# Databento data download via streaming API (get_range).
-#
-# Parameterized Argo WorkflowTemplate — replaces the hardcoded K8s Job.
-# Binary pre-uploaded to /data/bin/download_baseline on training-data-pvc.
-# Universe configs loaded from configmap databento-universe-all.
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/databento-download
-#   argo submit -n foxhunt --from=wftmpl/databento-download -p schema=trades -p output-dir=/data/futures-baseline-trades
-#   ./scripts/argo-download.sh mbp-10 --parallel 9 --pool ci-compile-cpu
-#
-# Databento rate limits: 100 concurrent connections per IP, 100 req/s.
----
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: databento-download
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: databento-download
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: download
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: databento-download
-  securityContext:
-    fsGroup: 1000
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 86400  # 24 hours
-
-  arguments:
-    parameters:
-      - name: schema
-        value: mbp-10
-        description: "Databento schema: ohlcv-1m, ohlcv-1s, trades, mbp-10"
-      - name: output-dir
-        value: /data/futures-baseline-mbp10
-        description: "Output directory on training-data PVC"
-      - name: parallel
-        value: "9"
-        description: "Number of parallel quarterly downloads (max ~100 per Databento rate limit)"
-      - name: node-pool
-        value: ci-compile-cpu
-        description: "K8s node pool: ci-compile-cpu (32c/64G) or platform (4c/8G)"
-
-  volumes:
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: universe-configs
-      configMap:
-        name: databento-universe-all
-
-  templates:
-    - name: download
-      inputs:
-        parameters:
-          - name: schema
-          - name: output-dir
-          - name: parallel
-          - name: node-pool
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{inputs.parameters.node-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: ubuntu:24.04
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: DATABENTO_API_KEY
-            valueFrom:
-              secretKeyRef:
-                name: databento-credentials
-                key: api-key
-        resources:
-          requests:
-            cpu: 500m
-            memory: 512Mi
-          limits:
-            cpu: "2"
-            memory: 4Gi
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-          - name: universe-configs
-            mountPath: /configs
-        args:
-          - |
-            set -e
-            apt-get update -qq > /dev/null 2>&1 && apt-get install -y -qq ca-certificates > /dev/null 2>&1
-
-            BINARY=/data/bin/download_baseline
-            chmod +x "$BINARY"
-
-            SCHEMA="{{inputs.parameters.schema}}"
-            OUTPUT_DIR="{{inputs.parameters.output-dir}}"
-            PARALLEL="{{inputs.parameters.parallel}}"
-
-            # Map schema name to configmap key
-            CONFIG="/configs/${SCHEMA}.toml"
-            if [ ! -f "$CONFIG" ]; then
-              echo "ERROR: Config not found: $CONFIG"
-              echo "Available configs:"
-              ls /configs/
-              exit 1
-            fi
-
-            mkdir -p "$OUTPUT_DIR"
-
-            echo "=== Databento Download ==="
-            echo "Schema:    $SCHEMA"
-            echo "Config:    $CONFIG"
-            echo "Output:    $OUTPUT_DIR"
-            echo "Parallel:  $PARALLEL"
-            echo "Node:      $(hostname), CPUs: $(nproc), RAM: $(free -g | awk '/Mem:/{print $2}')G"
-            echo ""
-
-            $BINARY \
-              --universe-config "$CONFIG" \
-              --output-dir "$OUTPUT_DIR" \
-              --parallel "$PARALLEL" \
-              --yes
-
-            echo ""
-            echo "=== Download complete ==="
-            echo "Files:"
-            find "$OUTPUT_DIR" -name '*.dbn.zst' -exec ls -lh {} \;
-            echo ""
-            du -sh "$OUTPUT_DIR"
-            df -h /data

infra/k8s/argo/dns-bootstrap-policy.yaml

@@ -1,25 +0,0 @@
-# DNS Bootstrap Prevention — Prevent CoreDNS chicken-and-egg deadlock
-#
-# ROOT CAUSE (2026-03-18 incident):
-# 1. Rolling replacement of ALL platform nodes simultaneously
-# 2. New nodes boot with /etc/resolv.conf → 10.32.0.10 (kube-dns only)
-# 3. CoreDNS pods need to pull images → need DNS → need CoreDNS
-# 4. Cilium BPF socket LB returns EPERM when kube-dns has no endpoints
-# 5. Complete cluster DNS deadlock — nothing can pull images
-#
-# PREVENTION:
-# 1. VPC DefaultRoutePropagation MUST be enabled (API, not Terraform yet)
-# 2. NEVER replace ALL nodes at once — one at a time, wait for CoreDNS
-# 3. Do NOT create CiliumClusterwideNetworkPolicy with egress rules
-#    (enables default deny, makes the problem worse)
-# 4. bpf-lb-sock MUST remain "true" — disabling breaks ALL service routing
-# 5. Keep bastion enabled on VPC gateway for emergency SSH access
-#
-# RECOVERY PROCEDURE:
-# 1. SSH via bastion: ssh -o ProxyCommand="ssh -W %h:%p -p 61000 bastion@<gw-public-ip>" root@<node-private-ip>
-# 2. On the node: /usr/local/bin/crictl pull <coredns-image>
-# 3. kubectl delete pods -l k8s-app=kube-dns --force
-# 4. CoreDNS starts from cached image → DNS recovers → cascade
-#
-# DATA PROTECTION:
-# All PVs set to Retain reclaim policy — data survives cluster deletion

infra/k8s/argo/events/auto-compile-configmap.yaml

@@ -1,12 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: auto-compile-config
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: auto-compile-config
-    app.kubernetes.io/part-of: foxhunt
-data:
-  # Set to "true" to enable auto-compile on push to main.
-  # kubectl -n foxhunt patch configmap auto-compile-config -p '{"data":{"enabled":"true"}}'
-  enabled: "false"

infra/k8s/argo/events/ci-pipeline-sensor.yaml

@@ -1,70 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Sensor
-metadata:
-  name: ci-pipeline-trigger
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: ci-pipeline-trigger
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  template:
-    serviceAccountName: argo-workflow
-    nodeSelector:
-      k8s.scaleway.com/pool-name: platform
-  eventBusName: default
-  dependencies:
-    - name: gitlab-push-dep
-      eventSourceName: gitlab-push
-      eventName: gitlab-push
-      filters:
-        data:
-          - path: body.ref
-            type: string
-            value:
-              - "refs/heads/main"
-  # Policy (user directive): push-triggered runs are infra-sync only.
-  # The ci-pipeline DAG runs three conditional task groups on push:
-  #   - docker image rebuilds    (gated on infra/docker/ changes)
-  #   - apply-argo-templates     (gated on infra/k8s/argo/ changes)
-  #   - terragrunt-apply         (gated on infra/live/ or infra/modules/ changes)
-  # Non-infra tasks (test-gate, build-web-dashboard, gpu-test) have `when: "false"`
-  # in the WorkflowTemplate. Training + compile/deploy are separate templates,
-  # always manual.
-  #
-  # All non-push workflows (cargo tests, training, compile/deploy, GPU tests)
-  # run via scripts/manual `argo submit`.
-
-  triggers:
-    - template:
-        name: ci-pipeline
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: ci-pipeline-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: ci-pipeline
-                arguments:
-                  parameters:
-                    - name: commit-sha
-                    - name: commits-json
-          parameters:
-            - src:
-                dependencyName: gitlab-push-dep
-                dataKey: body.checkout_sha
-              dest: spec.arguments.parameters.0.value
-            - src:
-                dependencyName: gitlab-push-dep
-                dataKey: body.commits
-                dataTemplate: "{{ toJson .Input }}"
-              dest: spec.arguments.parameters.1.value

infra/k8s/argo/events/eventbus.yaml

@@ -1,10 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: EventBus
-metadata:
-  name: default
-  namespace: foxhunt
-spec:
-  nats:
-    native:
-      replicas: 3
-      auth: token

infra/k8s/argo/events/eventsource-svc.yaml

@@ -1,16 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: gitlab-push-eventsource-svc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gitlab-push-eventsource
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  type: ClusterIP
-  ports:
-    - port: 12000
-      targetPort: 12000
-      protocol: TCP
-  selector:
-    eventsource-name: gitlab-push

infra/k8s/argo/events/gitlab-push-eventsource.yaml

@@ -1,15 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: EventSource
-metadata:
-  name: gitlab-push
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gitlab-push-eventsource
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  eventBusName: default
-  webhook:
-    gitlab-push:
-      port: "12000"
-      endpoint: /push
-      method: POST

infra/k8s/argo/events/rbac.yaml

@@ -1,29 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: argo-sensor-submit
-  namespace: foxhunt
-rules:
-  - apiGroups: ["argoproj.io"]
-    resources: ["workflows"]
-    verbs: ["create", "get", "list", "watch"]
-  - apiGroups: ["argoproj.io"]
-    resources: ["workflowtemplates"]
-    verbs: ["get", "list"]
-  - apiGroups: ["argoproj.io"]
-    resources: ["workflowtaskresults"]
-    verbs: ["create", "patch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: argo-sensor-submit
-  namespace: foxhunt
-subjects:
-  - kind: ServiceAccount
-    name: argo-workflow
-    namespace: foxhunt
-roleRef:
-  kind: Role
-  name: argo-sensor-submit
-  apiGroup: rbac.authorization.k8s.io

infra/k8s/argo/events/workflow-trigger-eventsource.yaml

@@ -1,51 +0,0 @@
-# Manual workflow trigger EventSource.
-#
-# Webhook endpoints for triggering compile/deploy, training, and image builds.
-# Each endpoint accepts a JSON POST body with workflow-specific parameters.
-#
-# Endpoints (all on port 12001):
-#   POST /compile-deploy    — {"commit_sha": "HEAD"}
-#   POST /train/dqn         — {"binary_tag": "latest"}
-#   POST /train/ppo         — {"binary_tag": "latest"}
-#   POST /train/supervised  — {"model": "tft", "binary_tag": "latest"}
-#   POST /build-image       — {"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}
-#
-# Example:
-#   curl -X POST http://workflow-trigger-eventsource-svc.foxhunt:12001/build-image \
-#     -H 'Content-Type: application/json' \
-#     -d '{"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}'
----
-apiVersion: argoproj.io/v1alpha1
-kind: EventSource
-metadata:
-  name: workflow-trigger
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: workflow-trigger
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  eventBusName: default
-  template:
-    nodeSelector:
-      k8s.scaleway.com/pool-name: platform
-  webhook:
-    compile-deploy:
-      port: "12001"
-      endpoint: /compile-deploy
-      method: POST
-    train-dqn:
-      port: "12001"
-      endpoint: /train/dqn
-      method: POST
-    train-ppo:
-      port: "12001"
-      endpoint: /train/ppo
-      method: POST
-    train-supervised:
-      port: "12001"
-      endpoint: /train/supervised
-      method: POST
-    build-image:
-      port: "12001"
-      endpoint: /build-image
-      method: POST

infra/k8s/argo/events/workflow-trigger-sensor.yaml

@@ -1,217 +0,0 @@
-# Sensor for manual workflow triggers.
-#
-# Listens on workflow-trigger EventSource and submits the appropriate workflow
-# when a webhook endpoint is hit. Each trigger maps JSON body fields to
-# workflow parameters.
-#
-# Trigger reference:
-#   compile-deploy    — POST /compile-deploy    {"commit_sha": "..."}
-#   train-dqn         — POST /train/dqn         {"binary_tag": "..."}
-#   train-ppo         — POST /train/ppo         {"binary_tag": "..."}
-#   train-supervised  — POST /train/supervised  {"model": "...", "binary_tag": "..."}
-#   build-image       — POST /build-image       {"dockerfile": "...", "image_name": "...", "commit_sha": "..."}
----
-apiVersion: argoproj.io/v1alpha1
-kind: Sensor
-metadata:
-  name: workflow-trigger
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: workflow-trigger
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  template:
-    serviceAccountName: argo-workflow
-    nodeSelector:
-      k8s.scaleway.com/pool-name: platform
-  eventBusName: default
-
-  dependencies:
-    - name: compile-deploy-dep
-      eventSourceName: workflow-trigger
-      eventName: compile-deploy
-    - name: train-dqn-dep
-      eventSourceName: workflow-trigger
-      eventName: train-dqn
-    - name: train-ppo-dep
-      eventSourceName: workflow-trigger
-      eventName: train-ppo
-    - name: train-supervised-dep
-      eventSourceName: workflow-trigger
-      eventName: train-supervised
-    - name: build-image-dep
-      eventSourceName: workflow-trigger
-      eventName: build-image
-
-  triggers:
-    # ── Compile + Deploy ──
-    # POST /compile-deploy {"commit_sha": "abc1234"}
-    # Omit service_packages to build all 7 services (default).
-    - template:
-        name: compile-deploy
-        conditions: compile-deploy-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: compile-deploy-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: compile-and-deploy
-                arguments:
-                  parameters:
-                    - name: commit-sha
-          parameters:
-            - src:
-                dependencyName: compile-deploy-dep
-                dataKey: body.commit_sha
-              dest: spec.arguments.parameters.0.value
-
-    # ── Train DQN ──
-    # POST /train/dqn {"binary_tag": "latest"}
-    - template:
-        name: train-dqn
-        conditions: train-dqn-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: train-dqn-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: train-dqn
-                arguments:
-                  parameters:
-                    - name: binary-tag
-          parameters:
-            - src:
-                dependencyName: train-dqn-dep
-                dataKey: body.binary_tag
-              dest: spec.arguments.parameters.0.value
-
-    # ── Train PPO ──
-    # POST /train/ppo {"binary_tag": "latest"}
-    - template:
-        name: train-ppo
-        conditions: train-ppo-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: train-ppo-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: train-ppo
-                arguments:
-                  parameters:
-                    - name: binary-tag
-          parameters:
-            - src:
-                dependencyName: train-ppo-dep
-                dataKey: body.binary_tag
-              dest: spec.arguments.parameters.0.value
-
-    # ── Train Supervised ──
-    # POST /train/supervised {"model": "tft", "binary_tag": "latest"}
-    # model: one of tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
-    - template:
-        name: train-supervised
-        conditions: train-supervised-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: train-supervised-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: train-supervised
-                arguments:
-                  parameters:
-                    - name: model
-                    - name: binary-tag
-          parameters:
-            - src:
-                dependencyName: train-supervised-dep
-                dataKey: body.model
-              dest: spec.arguments.parameters.0.value
-            - src:
-                dependencyName: train-supervised-dep
-                dataKey: body.binary_tag
-              dest: spec.arguments.parameters.1.value
-
-    # ── Build Image ──
-    # POST /build-image {"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}
-    - template:
-        name: build-image
-        conditions: build-image-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: build-image-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: build-ci-image
-                arguments:
-                  parameters:
-                    - name: commit-sha
-                    - name: dockerfile
-                    - name: image-name
-          parameters:
-            - src:
-                dependencyName: build-image-dep
-                dataKey: body.commit_sha
-              dest: spec.arguments.parameters.0.value
-            - src:
-                dependencyName: build-image-dep
-                dataKey: body.dockerfile
-              dest: spec.arguments.parameters.1.value
-            - src:
-                dependencyName: build-image-dep
-                dataKey: body.image_name
-              dest: spec.arguments.parameters.2.value

infra/k8s/argo/events/workflow-trigger-svc.yaml

@@ -1,16 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: workflow-trigger-eventsource-svc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: workflow-trigger
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  type: ClusterIP
-  ports:
-    - port: 12001
-      targetPort: 12001
-      protocol: TCP
-  selector:
-    eventsource-name: workflow-trigger

infra/k8s/argo/feature-cache-pvc.yaml

@@ -1,18 +0,0 @@
-# Dedicated PVC for precomputed .fxcache files.
-# Survives workflow completions (unlike workspace volumeClaimTemplate).
-# Precompute writes here, all training steps read from here.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: feature-cache-pvc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: feature-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 10Gi

infra/k8s/argo/gpu-test-network-policy.yaml

@@ -1,93 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-gpu-test-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: gpu-test
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API + internal services (service CIDR)
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # HTTPS egress (crates.io, etc.)
-    - ports:
-        - port: 443
-          protocol: TCP
-    # Git SSH (GitLab)
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # GitLab webservice (git-http)
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # MinIO (artifact/log storage)
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Pushgateway (Prometheus metrics)
-    - ports:
-        - port: 9091
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: pushgateway
-    # OTLP (Tempo)
-    - ports:
-        - port: 4317
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: tempo
-    # Mattermost (notifications)
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
-    # GitLab container registry
-    - ports:
-        - port: 5000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: registry

infra/k8s/argo/gpu-test-nightly-cron.yaml

@@ -1,25 +0,0 @@
-# infra/k8s/argo/gpu-test-nightly-cron.yaml
-apiVersion: argoproj.io/v1alpha1
-kind: CronWorkflow
-metadata:
-  name: gpu-test-nightly
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gpu-test-nightly
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  schedule: "0 2 * * *"
-  timezone: "UTC"
-  suspend: true  # Enable when ready: kubectl patch cronworkflow gpu-test-nightly -n foxhunt -p '{"spec":{"suspend":false}}'
-  concurrencyPolicy: Replace
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 5
-  workflowSpec:
-    workflowTemplateRef:
-      name: gpu-test-pipeline
-    arguments:
-      parameters:
-        - name: models
-          value: "dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion"
-        - name: commit-ref
-          value: "main"

infra/k8s/argo/gpu-test-pipeline-template.yaml

@@ -1,529 +0,0 @@
-# infra/k8s/argo/gpu-test-pipeline-template.yaml
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: gpu-test-pipeline
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gpu-test-pipeline
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: pipeline
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: gpu-test
-  securityContext:
-    fsGroup: 0
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 7200
-  arguments:
-    parameters:
-      - name: commit-ref
-        value: HEAD
-      - name: models
-        value: "dqn,ppo,tft"
-      - name: test-scope
-        value: all
-      - name: gpu-pool
-        value: ci-training-h100
-      - name: cuda-compute-cap
-        value: "90"
-      - name: notify
-        value: "true"
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda-test
-    - name: test-data
-      persistentVolumeClaim:
-        claimName: test-data-pvc
-        readOnly: true
-
-  templates:
-    # ── pipeline: DAG entrypoint ──
-    # compile-and-test runs on GPU node (RWO PVC can't be shared cross-node).
-    # gpu-warmup ensures H100 is scaled up before compile starts.
-    - name: pipeline
-      dag:
-        tasks:
-          - name: gpu-warmup
-            template: gpu-warmup
-          - name: compile-and-test
-            template: compile-and-test
-            dependencies: [gpu-warmup]
-          - name: perf-benchmark
-            template: perf-benchmark
-            dependencies: [compile-and-test]
-
-    # ── gpu-warmup: trigger H100 autoscale ──
-    # Requests GPU to force autoscaler to provision node, then releases it.
-    - name: gpu-warmup
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: alpine:3.21
-        command: ["/bin/sh", "-c"]
-        args:
-          - |
-            echo "GPU warmup: triggering node autoscale..."
-            nvidia-smi --query-gpu=name,memory.total --format=csv,noheader || true
-            echo "GPU node ready, releasing for compile-and-test"
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: 200m
-            memory: 128Mi
-
-    # ── compile-and-test: compile + run GPU tests in single H100 pod ──
-    - name: compile-and-test
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      outputs:
-        parameters:
-          - name: results
-            valueFrom:
-              path: /tmp/outputs/results
-              default: "unknown:FAIL"
-          - name: failures
-            valueFrom:
-              path: /tmp/outputs/failures
-              default: "1"
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: TEST_DATA_DIR
-            value: /data/test-data
-          - name: RUST_LOG
-            value: info
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
-          - name: CARGO_PROFILE_TEST_OPT_LEVEL
-            value: "2"
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: test-data
-            mountPath: /data/test-data
-            readOnly: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        args:
-          - |
-            set -e
-
-            REF="{{workflow.parameters.commit-ref}}"
-            MODELS="{{workflow.parameters.models}}"
-            SCOPE="{{workflow.parameters.test-scope}}"
-
-            # --- SSH setup (same as compile-and-train) ---
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            # --- Persistent checkout on PVC ---
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              echo "=== Fetching latest refs ==="
-              git fetch origin
-              # Resolve REF after fetch so we always get the latest commit.
-              # Try origin/$REF (branch), then $REF directly (tag or SHA).
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                TARGET=$(git rev-parse "origin/$REF")
-              else
-                TARGET=$(git rev-parse --verify "$REF" 2>/dev/null || echo "$REF")
-              fi
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" = "$TARGET" ]; then
-                echo "=== Already at $REF ($TARGET) ==="
-              else
-                echo "=== Updating checkout to $REF ($TARGET) ==="
-                git checkout --force --detach "$TARGET"
-                git clean -fd
-              fi
-            else
-              echo "=== Initial clone ==="
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                git checkout --force --detach "origin/$REF"
-              else
-                git checkout "$REF"
-              fi
-            fi
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            # --- PTX cache invalidation ---
-            # Purge stale cached PTX if any CUDA kernel source changed since last run.
-            bash scripts/ptx-cache-invalidate.sh "${CARGO_TARGET_DIR}/.ptx_cache"
-
-            # --- PVC size guard ---
-            TARGET_SIZE_MB=$(du -sm "$CARGO_TARGET_DIR" 2>/dev/null | cut -f1 || echo 0)
-            echo "PVC usage: ${TARGET_SIZE_MB}MB"
-            if [ "$TARGET_SIZE_MB" -gt 25000 ]; then
-              echo "PVC exceeds 25GB, pruning..."
-              rm -rf "$CARGO_TARGET_DIR/release" "$CARGO_TARGET_DIR/debug"
-            fi
-
-            # --- Compile ---
-            echo "=== Compiling test binaries (--features cuda) ==="
-            cargo test -p ml -p ml-dqn -p ml-core --features cuda --no-run 2>&1 | tee /cargo-target/gpu-compile.log
-            echo "=== Compilation done ==="
-
-            # --- Expand "all" ---
-            if [ "$MODELS" = "all" ]; then
-              MODELS="dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion"
-            fi
-
-            # --- Test runner ---
-            RESULTS=""
-            FAILURES=0
-
-            # Reset CUDA context between test binaries to prevent cuBLAS
-            # CUBLAS_STATUS_NOT_INITIALIZED cascades. Each test binary creates
-            # and destroys hundreds of cuBLAS handles; without a reset, the
-            # driver fails to re-init for the next binary.
-            gpu_context_drain() {
-              nvidia-smi -rgc >/dev/null 2>&1 || true
-              sleep 1
-            }
-
-            run_tests() {
-              local NAME="$1"; shift
-              echo ""
-              echo "========================================"
-              echo "  TEST: $NAME"
-              echo "========================================"
-              # --nocapture is a test-binary flag, must come after --
-              # If args already contain --, append after it; otherwise add -- first
-              local HAS_SEP=false
-              for arg in "$@"; do
-                [ "$arg" = "--" ] && HAS_SEP=true && break
-              done
-              set +e
-              if $HAS_SEP; then
-                "$@" --nocapture 2>&1
-              else
-                "$@" -- --nocapture 2>&1
-              fi
-              EXIT=$?
-              set -e
-              if [ $EXIT -eq 0 ]; then
-                RESULTS="${RESULTS}${NAME}:PASS\n"
-              else
-                RESULTS="${RESULTS}${NAME}:FAIL\n"
-                FAILURES=$((FAILURES + 1))
-              fi
-              # Drain CUDA context after each GPU test suite
-              gpu_context_drain
-            }
-
-            # --- Core tests (always run) ---
-            # --test-threads=1 for all GPU lib tests: prevents concurrent cuBLAS
-            # handle creation that causes CUBLAS_STATUS_NOT_INITIALIZED cascades.
-            run_tests "core-lib" cargo test -p ml-core --features cuda --lib -- --test-threads=1
-            run_tests "bayesian" cargo test -p ml --features cuda --test bayesian_changepoint_test -- --test-threads=1
-
-            # --- Per-model tests ---
-            IFS=',' read -ra MODEL_LIST <<< "$MODELS"
-            for MODEL in "${MODEL_LIST[@]}"; do
-              case "$MODEL" in
-                dqn)
-                  if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
-                    # --test-threads=1: GPU lib tests must run serially — each test
-                    # creates a cuBLAS handle via Device::new_cuda(0). Under parallel
-                    # execution, concurrent cuBLAS init races cause
-                    # CUBLAS_STATUS_NOT_INITIALIZED failures (51 test cascade).
-                    run_tests "dqn-lib" cargo test -p ml-dqn --features cuda --lib -- --test-threads=1
-                    run_tests "dqn-ml-lib" cargo test -p ml --features cuda --lib -- --test-threads=1 dqn
-                  fi
-                  if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
-                    # --test-threads=1 for ALL GPU integration tests: parallel execution
-                    # corrupts the CUDA primary context (cuDevicePrimaryCtxRetain fails
-                    # when multiple threads race on context init/teardown).
-                    run_tests "dqn-smoke" cargo test -p ml --features cuda --test smoke_test_real_data -- --test-threads=1
-                    # Run each pipeline test in its own cargo test process.
-                    # CUDA Graph capture corrupts the async memory pool, making
-                    # cuMemAllocAsync fail with CUDA_ERROR_INVALID_VALUE in
-                    # subsequent DQNTrainer instances within the same process.
-                    run_tests "dqn-pipeline-train" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_trains_on_es_fut -- --test-threads=1 --exact
-                    run_tests "dqn-pipeline-loss" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_loss_decreases -- --test-threads=1 --exact
-                    run_tests "dqn-pipeline-ckpt" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_checkpoint_save_load -- --test-threads=1 --exact
-                    run_tests "dqn-pipeline-qval" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_q_value_predictions -- --test-threads=1 --exact
-                    run_tests "dqn-pipeline-eps" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_epsilon_greedy -- --test-threads=1 --exact
-                    run_tests "dqn-smoke-train" cargo test -p ml --features cuda --test dqn_training_smoke_test -- --test-threads=1
-                    run_tests "dqn-early-stop" cargo test -p ml --features cuda --test dqn_early_stopping_termination_test -- --test-threads=1
-                    run_tests "dqn-collapse" cargo test -p ml --features cuda --test dqn_action_collapse_fix_test -- --test-threads=1
-                  fi
-                  ;;
-                ppo)
-                  if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
-                    run_tests "ppo-lib" cargo test -p ml --features cuda --lib -- --test-threads=1 ppo
-                  fi
-                  if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
-                    run_tests "ppo-barrier" cargo test -p ml --features cuda --test barrier_optimization_test -- --test-threads=1
-                  fi
-                  ;;
-                *)
-                  # Supervised models (TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion)
-                  # Map model names to Rust module names where they differ
-                  LIB_FILTER="$MODEL"
-                  [ "$MODEL" = "tggn" ] && LIB_FILTER="tgnn"
-                  if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
-                    run_tests "${MODEL}-lib" cargo test -p ml --features cuda --lib -- --test-threads=1 "$LIB_FILTER"
-                  fi
-                  if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
-                    run_tests "${MODEL}-gpu" cargo test -p ml --features cuda --test supervised_gpu_smoke_test -- --test-threads=1 "test_${MODEL}_gpu_smoke"
-                    # Also run model-specific integration tests if they exist
-                    if cargo test -p ml --features cuda --test "${MODEL}_integration" --no-run 2>/dev/null; then
-                      run_tests "${MODEL}-integ" cargo test -p ml --features cuda --test "${MODEL}_integration" -- --test-threads=1
-                    fi
-                  fi
-                  ;;
-              esac
-            done
-
-            # --- Summary ---
-            echo ""
-            echo "========================================"
-            echo "  GPU TEST RESULTS"
-            echo "========================================"
-            printf "$RESULTS" | while IFS=: read -r name status; do
-              [ -z "$name" ] && continue
-              printf "  %-25s %s\n" "$name" "$status"
-            done
-            echo "========================================"
-            echo "  Total failures: $FAILURES"
-            echo "========================================"
-
-            # Write results for notification step
-            mkdir -p /tmp/outputs
-            printf "$RESULTS" > /tmp/outputs/results
-            echo "$FAILURES" > /tmp/outputs/failures
-
-            [ "$FAILURES" -gt 0 ] && exit 1 || exit 0
-
-    # ── perf-benchmark: DQN epoch/s on 3Q data (performance regression guard) ──
-    # Runs after tests pass. Trains DQN on 3Q of ES.FUT data and reports epoch time.
-    # Uses the same binary compiled by compile-and-test (shared cargo-target PVC).
-    # Fails the pipeline if epoch time exceeds 500ms (regression threshold for H100).
-    - name: perf-benchmark
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      outputs:
-        parameters:
-          - name: epoch-ms
-            valueFrom:
-              path: /tmp/outputs/epoch-ms
-              default: "9999"
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
-        volumeMounts:
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: test-data
-            mountPath: /data/test-data
-            readOnly: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 16Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 32Gi
-        args:
-          - |
-            set -e
-            cd /cargo-target/src
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            echo "========================================"
-            echo "  PERF BENCHMARK: DQN epoch/s (3Q ES.FUT)"
-            echo "========================================"
-
-            # Run 5 epochs × 100 steps on 3Q data (--train-months 3 fits in 1 quarter)
-            # Use --step-months 3 to avoid walk-forward window splits
-            OUTPUT=$(cargo run --release --example train_baseline_rl -p ml -- \
-              --model dqn \
-              --data-dir /data/test-data/ohlcv \
-              --mbp10-data-dir /data/test-data/mbp10 \
-              --trades-data-dir /data/test-data/trades \
-              --symbol ES.FUT \
-              --epochs 5 \
-              --train-months 3 --val-months 1 --test-months 1 --step-months 3 \
-              2>&1)
-
-            # Extract epoch times (skip epoch 1 which includes init)
-            EPOCH_TIMES=$(echo "$OUTPUT" | grep "phase breakdown" | grep -v "Epoch 1/" | \
-              sed 's/.*total=\([0-9]*\)ms.*/\1/' | head -4)
-
-            if [ -z "$EPOCH_TIMES" ]; then
-              echo "ERROR: No phase breakdown output found"
-              echo "$OUTPUT" | tail -20
-              mkdir -p /tmp/outputs
-              echo "9999" > /tmp/outputs/epoch-ms
-              exit 1
-            fi
-
-            # Compute average epoch time (epochs 2-5)
-            SUM=0
-            COUNT=0
-            for T in $EPOCH_TIMES; do
-              SUM=$((SUM + T))
-              COUNT=$((COUNT + 1))
-            done
-            AVG=$((SUM / COUNT))
-
-            echo ""
-            echo "  Epoch times (ms, excl. epoch 1): $EPOCH_TIMES"
-            echo "  Average: ${AVG}ms"
-            echo ""
-
-            mkdir -p /tmp/outputs
-            echo "$AVG" > /tmp/outputs/epoch-ms
-
-            # Regression guard: fail if avg epoch > 500ms on H100
-            THRESHOLD=500
-            if [ "$AVG" -gt "$THRESHOLD" ]; then
-              echo "PERF REGRESSION: ${AVG}ms > ${THRESHOLD}ms threshold"
-              echo "========================================"
-              exit 1
-            fi
-
-            echo "  PASS: ${AVG}ms <= ${THRESHOLD}ms threshold"
-            echo "========================================"
-
-    # ── notify-result: post test outcome to Mattermost (onExit) ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            NOTIFY="{{workflow.parameters.notify}}"
-            if [ "$NOTIFY" != "true" ]; then
-              echo "Notifications disabled, skipping"
-              exit 0
-            fi
-
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"
-              exit 0
-            fi
-
-            if [ "$STATUS" = "Succeeded" ]; then
-              EMOJI=":white_check_mark:"
-            else
-              EMOJI=":x:"
-            fi
-
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **GPU Tests** ${NAME} — ${STATUS} ({{workflow.duration}}s)\"}"
-
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/kustomization.yaml

@@ -1,15 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - sccache-pvcs.yaml
-  - ci-pipeline-template.yaml
-  - compile-and-deploy-template.yaml
-  - train-template.yaml
-  - build-ci-image-template.yaml
-  - sanitizer-test-template.yaml
-  - nsys-test-template.yaml
-  - smoke-test-template.yaml
-  - refresh-deps-cache-template.yaml
-  - argo-workflow-netpol.yaml
-  - ci-deploy-rbac.yaml
-  - archive-rbac.yaml

infra/k8s/argo/lob-backtest-sweep-template.yaml

@@ -1,459 +0,0 @@
-# Real-LOB backtest sweep workflow.
-#
-# Fans out a parameter grid across N parallel GPU pods, each running
-# `fxt-backtest run` against the same MBP-10 data + checkpoint with one
-# cell's worth of overrides. A final CPU pod runs `fxt-backtest aggregate`
-# to produce aggregate.parquet + pareto_frontier.json at the sweep root.
-#
-# The `# __SWEEP_CELLS__` marker on the dag.tasks line is replaced by
-# scripts/argo-lob-sweep.sh with N generated WorkflowTask stanzas before
-# submission. Same convention as train-multi-seed-template.yaml's
-# # __MATRIX_TASKS__ marker.
-#
-# DAG:
-#   ensure-binary ──> [N parallel run-cell-<i> tasks on ci-training-l40s,
-#                       each with its own outputDir on the shared PVC]
-#                     │
-#                     └──> aggregate (CPU node, runs fxt-backtest aggregate
-#                                     against the sweep root)
-#
-# Per `feedback_default_to_l40s_pool.md` (2026-05-09): defaults to
-# ci-training-l40s + sm_89. Override via --gpu-pool ci-training-h100
-# for sm_90 / 80 GB.
-#
-# Usage:
-#   ./scripts/argo-lob-sweep.sh --grid config/ml/sweep_decision_stride_example.yaml
-#   ./scripts/argo-lob-sweep.sh --grid <path> --sha abc1234 --gpu-pool ci-training-h100
-#   ./scripts/argo-lob-sweep.sh --grid <path> --watch
-#   ./scripts/argo-lob-sweep.sh --grid <path> --dry-run > /tmp/wf.yaml
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: lob-backtest-sweep
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: lob-backtest-sweep
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train          # reuses argo-train-workflow NetworkPolicy egress (port 2222 to gitlab-shell)
-spec:
-  entrypoint: sweep-matrix
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  # LOB sweep cells are much faster than training (~minutes vs hours);
-  # 2h walltime is generous even for N=128 cells.
-  activeDeadlineSeconds: 7200
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: main
-      - name: cuda-compute-cap
-        value: "89"
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: data-root
-        value: /mnt/training-data/futures-baseline/ES.FUT
-      - name: predecoded-dir
-        value: /mnt/training-data/futures-baseline/ES.FUT
-      - name: checkpoint
-        value: ""               # empty = trunk runs with --seed (random init);
-                                # set to a path on the data PVC for trained weights
-      - name: sweep-root
-        value: /mnt/training-data/sweeps/lob-backtest
-      - name: sweep-tag
-        value: default          # subdirectory: <sweep-root>/<sweep-tag>/
-      # Operational knobs. argo-lob-sweep.sh sets these per-grid.
-      - name: n-parallel
-        value: "1"
-      - name: latency-ns
-        value: "100000000"
-      - name: target-annual-vol-units
-        value: "50.0"
-      - name: annualisation-factor
-        value: "825.0"
-      - name: max-lots
-        value: "5"
-      - name: max-events
-        value: "0"
-      # Per-step kernel-state JSONL trace path. Empty (default) = disabled:
-      # ensure-binary compiles fxt-backtest WITHOUT --features kernel-step-trace
-      # and run-cell omits the --kernel-step-trace flag. When set to a
-      # non-empty path, ensure-binary rebuilds with the feature enabled
-      # and run-cell passes --kernel-step-trace <path> through to the
-      # binary. Trace is written to that absolute path on the pod (must
-      # resolve to a mounted PVC — usually /feature-cache or /mnt/training-data).
-      - name: kernel-step-trace
-        value: ""
-
-  volumes:
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 0400
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  templates:
-    # ── DAG entrypoint ──────────────────────────────────────────────
-    - name: sweep-matrix
-      dag:
-        tasks:
-          - name: ensure-binary
-            template: ensure-binary
-          # __SWEEP_CELLS__ — replaced by argo-lob-sweep.sh
-          - name: aggregate
-            template: aggregate
-            dependencies: [ensure-binary]    # plus all run-cell-* via shell-injected list
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-
-    # ── ensure-binary: cache-or-compile fxt-backtest by commit SHA ─
-    # Same shape as train-multi-seed-template.yaml's ensure-binary but
-    # for the fxt-backtest binary only.
-    - name: ensure-binary
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /cargo-target/sccache
-          - name: SCCACHE_CACHE_SIZE
-            value: "40G"
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          - name: FOXHUNT_CUDA_ARCH
-            value: "sm_{{workflow.parameters.cuda-compute-cap}}"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /mnt/training-data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            KERNEL_TRACE="{{workflow.parameters.kernel-step-trace}}"
-
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"; git fetch origin
-                SHA=$(git rev-parse "origin/$BRANCH"); cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"
-                cd "$BUILD"; git checkout "origin/$BRANCH"
-                SHA=$(git rev-parse HEAD); cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            # Feature variant: when kernel-step-trace is enabled, build
-            # with the Cargo feature and cache under a distinct subdir
-            # so the default + diagnostic binaries don't clobber each other.
-            FEATURE_FLAGS=""
-            VARIANT="default"
-            if [ -n "$KERNEL_TRACE" ]; then
-              FEATURE_FLAGS="--features kernel-step-trace"
-              VARIANT="kstrace"
-            fi
-            BIN_DIR="/mnt/training-data/bin/$SHORT_SHA-$VARIANT"
-            mkdir -p "$BIN_DIR"
-            if [ -x "$BIN_DIR/fxt-backtest" ]; then
-              echo "=== Cache HIT: fxt-backtest ($VARIANT) present in $BIN_DIR ==="
-              ls -lh "$BIN_DIR/fxt-backtest"
-              echo "$SHORT_SHA-$VARIANT" > /tmp/sha
-              exit 0
-            fi
-
-            echo "=== Cache MISS: compiling fxt-backtest ($VARIANT) for $SHORT_SHA ==="
-            if [ -d "$BUILD/.git" ]; then
-              # Same pattern as alpha-perception-template.yaml: prior
-              # cargo build mutates Cargo.lock (and sometimes other
-              # generated files); --force checkout overwrites them,
-              # `git clean -fd` drops untracked. Without this, the
-              # second SHA's checkout fails on dirty working tree.
-              cd "$BUILD"; git fetch origin
-              git checkout --force "$SHA"; git clean -fd
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"; git checkout "$SHA"
-            fi
-
-            cargo build -p fxt-backtest --release $FEATURE_FLAGS
-            cp "$CARGO_TARGET_DIR/release/fxt-backtest" "$BIN_DIR/fxt-backtest"
-            echo "$SHORT_SHA-$VARIANT" > /tmp/sha
-            ls -lh "$BIN_DIR/fxt-backtest"
-
-    # ── run-cell: one sweep cell against a GPU pod ─────────────────
-    # Inputs: cell name + every Run arg that varies across the grid.
-    # The shell-rendered DAG tasks supply these per-cell.
-    - name: run-cell
-      inputs:
-        parameters:
-          - name: sha
-          - name: cell-name
-          - name: decision-stride
-            value: "4"
-          - name: latency-ns
-            value: "{{workflow.parameters.latency-ns}}"
-          - name: target-annual-vol-units
-            value: "{{workflow.parameters.target-annual-vol-units}}"
-          - name: annualisation-factor
-            value: "{{workflow.parameters.annualisation-factor}}"
-          - name: max-lots
-            value: "{{workflow.parameters.max-lots}}"
-          - name: max-events
-            value: "{{workflow.parameters.max-events}}"
-          - name: seed
-            value: "0xC0FFEE"
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        resources:
-          requests:
-            cpu: "4"
-            memory: 16Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "8"
-            memory: 32Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /mnt/training-data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            CELL="{{inputs.parameters.cell-name}}"
-            BIN="/mnt/training-data/bin/$SHA/fxt-backtest"
-            CKPT="{{workflow.parameters.checkpoint}}"
-            KERNEL_TRACE="{{workflow.parameters.kernel-step-trace}}"
-            OUT="{{workflow.parameters.sweep-root}}/{{workflow.parameters.sweep-tag}}/$CELL"
-            mkdir -p "$OUT"
-            echo "=== sweep cell $CELL on $(hostname) → $OUT ==="
-
-            CKPT_FLAG=""
-            if [ -n "$CKPT" ]; then
-              CKPT_FLAG="--checkpoint $CKPT"
-            fi
-            KERNEL_TRACE_FLAG=""
-            if [ -n "$KERNEL_TRACE" ]; then
-              # Per-cell trace path: append cell name so concurrent run-cell
-              # pods don't clobber each other's JSONL files. Operators get
-              # one trace per cell; aggregate by reading <root>/<sweep>/<cell>/
-              # at analysis time.
-              KERNEL_TRACE_FLAG="--kernel-step-trace $OUT/kernel_step_trace.jsonl"
-            fi
-
-            "$BIN" run \
-              --data "{{workflow.parameters.data-root}}" \
-              --predecoded-dir "{{workflow.parameters.predecoded-dir}}" \
-              --n-parallel "{{workflow.parameters.n-parallel}}" \
-              --decision-stride "{{inputs.parameters.decision-stride}}" \
-              --latency-ns "{{inputs.parameters.latency-ns}}" \
-              --target-annual-vol-units "{{inputs.parameters.target-annual-vol-units}}" \
-              --annualisation-factor "{{inputs.parameters.annualisation-factor}}" \
-              --max-lots "{{inputs.parameters.max-lots}}" \
-              --max-events "{{inputs.parameters.max-events}}" \
-              --seed "{{inputs.parameters.seed}}" \
-              $CKPT_FLAG \
-              $KERNEL_TRACE_FLAG \
-              --out "$OUT"
-
-            echo "=== cell $CELL done ==="
-            ls -lh "$OUT"
-
-    # ── run-sweep: P6 batched flow. Invokes `fxt-backtest sweep` ───
-    # against the FULL grid YAML inside a single GPU pod. When the YAML
-    # carries sim_variants, each cell expands into n_parallel=variants.len()
-    # backtests sharing one forward pass. argo-lob-sweep.sh emits ONE
-    # run-sweep task (instead of N fan-out run-cell tasks) when --batched
-    # is set or sim_variants is detected in the YAML.
-    - name: run-sweep
-      inputs:
-        parameters:
-          - name: sha
-          - name: grid-yaml-b64        # base64-encoded grid YAML, written to a file in-pod
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        resources:
-          requests:
-            cpu: "4"
-            memory: 16Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "8"
-            memory: 32Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /mnt/training-data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BIN="/mnt/training-data/bin/$SHA/fxt-backtest"
-            OUT="{{workflow.parameters.sweep-root}}/{{workflow.parameters.sweep-tag}}"
-            mkdir -p "$OUT"
-            # Write the grid YAML to a known path. base64 in/out keeps Argo
-            # parameter encoding stable across YAML special characters
-            # (curly braces from data_template + sim_variants).
-            GRID_YAML=/tmp/sweep-grid.yaml
-            echo "{{inputs.parameters.grid-yaml-b64}}" | base64 -d > "$GRID_YAML"
-            echo "=== running fxt-backtest sweep on $(hostname) → $OUT ==="
-            echo "=== grid yaml: ==="
-            head -20 "$GRID_YAML"
-            echo "=== ... ==="
-
-            "$BIN" sweep --grid "$GRID_YAML" --out "$OUT"
-
-            echo "=== sweep complete ==="
-            ls -lh "$OUT"
-
-    # ── aggregate: runs fxt-backtest aggregate on the GPU pool. ───
-    # The binary is dynamically linked against libcuda.so.1, which the
-    # ci-compile-cpu pool's host doesn't expose; the aggregate workload
-    # itself is CPU-only (~seconds) but the binary needs the driver libs.
-    # Reusing the GPU pool is cheap (single fast pod) and avoids needing
-    # a separate CPU-only binary.
-    - name: aggregate
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        resources:
-          requests:
-            cpu: "2"
-            memory: 4Gi
-            # Request a GPU only to make the Scaleway L40S device plugin
-            # mount libcuda.so.1 into the container — aggregate logic is
-            # CPU-only (~seconds). Without the request, the binary's
-            # dynamic loader fails before main(). Cheapest fix; the GPU
-            # is held for ~5s of CPU work, then released.
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "4"
-            memory: 8Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /mnt/training-data
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BIN="/mnt/training-data/bin/$SHA/fxt-backtest"
-            SWEEP_DIR="{{workflow.parameters.sweep-root}}/{{workflow.parameters.sweep-tag}}"
-            echo "=== aggregate sweep at $SWEEP_DIR ==="
-            "$BIN" aggregate "$SWEEP_DIR"
-            ls -lh "$SWEEP_DIR"
-
-    # ── notify-result: exit hook (placeholder; real impl emits to Slack/MinIO) ─
-    - name: notify-result
-      container:
-        image: alpine:3.20
-        command: ["sh", "-c"]
-        args:
-          - |
-            echo "lob-backtest-sweep workflow {{workflow.name}} finished with status {{workflow.status}}"
-            echo "results under {{workflow.parameters.sweep-root}}/{{workflow.parameters.sweep-tag}}"

infra/k8s/argo/notification-webhook.yaml

@@ -1,11 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-  name: notification-webhook
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-type: Opaque
-stringData:
-  # Mattermost incoming webhook URL — update after Mattermost is deployed
-  webhook-url: "http://mattermost.foxhunt.svc.cluster.local:8065/hooks/PLACEHOLDER"

infra/k8s/argo/nsys-test-template.yaml

@@ -1,318 +0,0 @@
-# infra/k8s/argo/nsys-test-template.yaml
-#
-# One-shot Nsight Systems profiling run on L40S. Wraps a smoke test under
-# `nsys profile` and uploads the .nsys-rep to MinIO at
-# foxhunt-training-results/profiles/smoke/<short-sha>/.
-#
-# Mirrors the nsys integration in train-multi-seed-template (Plan 5 Task 3,
-# A.4.1) but for smoke tests. Use this for rapid kernel-timing iteration
-# (per-step bottleneck hunts, SOL %, occupancy diffs) without the full
-# multi-seed Argo cycle.
-#
-# Why L40S not local: laptop RTX 3050 Ti has 4 GB VRAM; nsys on L40S (48 GB)
-# avoids any contention and runs at near-native speed (typical ~5-10%
-# slowdown vs uninstrumented).
-#
-# Usage:
-#   argo submit --watch -n foxhunt nsys-test-template.yaml \
-#     -p commit-ref=main \
-#     -p test-name=performance::test_real_data_single_epoch
-#
-# Or via the wrapper script: scripts/argo-nsys.sh
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: nsys-test
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: nsys-test
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: nsys-run
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: nsys-test
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 7200
-  activeDeadlineSeconds: 3600  # nsys overhead is small — 1h cap is generous
-  arguments:
-    parameters:
-      - name: commit-ref
-        value: HEAD
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: cuda-compute-cap
-        value: "89"
-      - name: test-name
-        value: "performance::test_real_data_single_epoch"
-      # nsys capture options. Defaults capture the full kernel timeline.
-      # GPU metrics counters (--gpu-metrics-devices) require elevated CUDA
-      # performance-counter privileges (NVGPUCTRPERM) which the container
-      # lacks; CUDA/NVTX/osrt traces alone still give per-kernel timing and
-      # call-graph context — sufficient for the bottleneck-hunt use case.
-      # Override via --extra-args if running on a node with relaxed perms.
-      - name: nsys-extra-args
-        value: "--trace=cuda,nvtx,osrt"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda-test
-    # Use the same training-data-pvc as production training (full 27 months
-    # of OHLCV + MBP-10 + trades) instead of the smaller test-data-pvc.
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-        readOnly: true
-
-  templates:
-    - name: nsys-run
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_TEST_DATA
-            value: /data/futures-baseline
-          - name: TEST_DATA_DIR
-            value: /data/futures-baseline
-          # MBP-10 + trades are mandatory per feedback_mbp10_mandatory.md
-          # (OFI features are part of the production state vector).
-          - name: FOXHUNT_MBP10_DATA
-            value: /data/futures-baseline-mbp10
-          - name: FOXHUNT_TRADES_DATA
-            value: /data/futures-baseline-trades
-          - name: RUST_LOG
-            value: info
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
-          # MinIO upload — same secret + optional pattern as train-multi-seed.
-          # Both refs are `optional: true` so env mount succeeds on clusters
-          # that do not pre-create `minio-credentials` (upload then warn-fails).
-          - name: MINIO_ACCESS_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: access-key
-                optional: true
-          - name: MINIO_SECRET_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: secret-key
-                optional: true
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        args:
-          - |
-            set -euo pipefail
-
-            REF="{{workflow.parameters.commit-ref}}"
-            TEST_NAME="{{workflow.parameters.test-name}}"
-            EXTRA_ARGS="{{workflow.parameters.nsys-extra-args}}"
-
-            echo "==================================="
-            echo "  nsys profile L40S run"
-            echo "==================================="
-            echo "  Ref:      $REF"
-            echo "  Test:     $TEST_NAME"
-            echo "  Extra:    $EXTRA_ARGS"
-            echo "  Pool:     {{workflow.parameters.gpu-pool}}"
-            echo "==================================="
-
-            # --- SSH setup ---
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            # --- Persistent checkout on PVC ---
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                TARGET=$(git rev-parse "origin/$REF")
-              else
-                TARGET=$(git rev-parse --verify "$REF" 2>/dev/null || echo "$REF")
-              fi
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$TARGET" ]; then
-                echo "=== Updating checkout to $REF ($TARGET) ==="
-                git checkout --force --detach "$TARGET"
-                git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                git checkout --force --detach "origin/$REF"
-              else
-                git checkout "$REF"
-              fi
-            fi
-            SHA=$(git rev-parse --short HEAD)
-            echo "Checked out $SHA"
-
-            export PATH="${CARGO_HOME}/bin:/usr/local/cuda/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            bash scripts/ptx-cache-invalidate.sh "${CARGO_TARGET_DIR}/.ptx_cache"
-
-            # --- Verify nsys ---
-            NSYS_BIN=$(which nsys 2>/dev/null || echo /usr/local/cuda/bin/nsys)
-            if [ ! -x "$NSYS_BIN" ]; then
-              echo "ERROR: nsys not found at $NSYS_BIN"
-              exit 2
-            fi
-            echo "=== nsys version ==="
-            "$NSYS_BIN" --version | head -3
-
-            # --- Compile test binary + train_baseline_rl ---
-            echo "=== Compiling (--release --no-run + train_baseline_rl) ==="
-            cargo test -p ml --release --lib --no-run 2>&1 | tee /cargo-target/nsys-compile.log
-            cargo build -p ml --release --example train_baseline_rl 2>&1 | tee -a /cargo-target/nsys-compile.log
-
-            # Honour CARGO_TARGET_DIR — outputs go to ${CARGO_TARGET_DIR}/release/deps,
-            # NOT the repo-relative target/. (CI workflow sets CARGO_TARGET_DIR=/cargo-target.)
-            # Subshell + || true wraps the head -1 SIGPIPE so pipefail doesn't fire.
-            TARGET_DIR="${CARGO_TARGET_DIR:-target}"
-            TEST_BIN=$( (ls -t "$TARGET_DIR/release/deps/ml-"* 2>/dev/null \
-              | grep -vE '\.(d|rcgu|rmeta|rlib|so|json)$' \
-              | head -1) || true )
-            if [ -z "$TEST_BIN" ] || [ ! -x "$TEST_BIN" ]; then
-              echo "ERROR: could not locate compiled test binary in $TARGET_DIR/release/deps/"
-              ls -la "$TARGET_DIR/release/deps/" 2>&1 | head -20
-              exit 3
-            fi
-            echo "=== Test binary: $TEST_BIN ==="
-
-            # grep -q + pipefail = SIGPIPE on early exit. Capture once, grep
-            # without -q so the consumer reads all input.
-            TEST_LIST=$("$TEST_BIN" --list 2>&1 || true)
-            if ! echo "$TEST_LIST" | grep -F "$TEST_NAME" > /dev/null; then
-              echo "ERROR: test '$TEST_NAME' not found"
-              echo "$TEST_LIST" | grep smoke_tests | (head -30 || true)
-              exit 4
-            fi
-
-            # --- Run under nsys profile ---
-            mkdir -p /tmp/nsys-output
-            POD_NAME="${HOSTNAME:-pod}"
-            NSYS_OUT="/tmp/nsys-output/profile-${POD_NAME}.nsys-rep"
-
-            echo ""
-            echo "==================================="
-            echo "  Launching nsys profile..."
-            echo "==================================="
-            START_TS=$(date +%s)
-            set +e
-            "$NSYS_BIN" profile \
-              --output="$NSYS_OUT" \
-              --force-overwrite=true \
-              --stats=true \
-              ${EXTRA_ARGS} \
-              -- "$TEST_BIN" "$TEST_NAME" --ignored --nocapture
-            EXIT_CODE=$?
-            set -e
-            END_TS=$(date +%s)
-            ELAPSED=$((END_TS - START_TS))
-
-            echo ""
-            echo "==================================="
-            echo "  nsys run complete"
-            echo "==================================="
-            echo "  Exit code:  $EXIT_CODE"
-            echo "  Elapsed:    ${ELAPSED}s"
-            ls -la "$NSYS_OUT" 2>/dev/null && \
-              echo "  Profile:    $(du -h "$NSYS_OUT" | cut -f1)"
-            echo ""
-
-            # --- Upload to MinIO (same pattern as train-multi-seed) ---
-            if [ -f "$NSYS_OUT" ]; then
-              echo "=== Uploading nsys profile to MinIO ==="
-              MC_BIN=$(which mc 2>/dev/null || echo "")
-              if [ -z "$MC_BIN" ]; then
-                MC_BIN=/tmp/mc
-                curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/mc -o "$MC_BIN" || {
-                  echo "WARN: failed to download mc — skipping upload"
-                  echo "Profile available locally at: $NSYS_OUT (PVC: cargo-target)"
-                  exit 0
-                }
-                chmod +x "$MC_BIN"
-              fi
-              "$MC_BIN" alias set foxhunt http://minio.foxhunt.svc.cluster.local:9000 \
-                "${MINIO_ACCESS_KEY:-}" "${MINIO_SECRET_KEY:-}" 2>/dev/null || true
-              # Smoke profiles go to a separate prefix to keep the
-              # production training profile bucket clean.
-              UPLOAD_PATH="foxhunt/foxhunt-training-results/profiles/smoke/${SHA}/profile-${POD_NAME}.nsys-rep"
-              if "$MC_BIN" cp "$NSYS_OUT" "$UPLOAD_PATH"; then
-                echo "=== nsys profile uploaded to ${UPLOAD_PATH} ==="
-                echo ""
-                echo "Download with:"
-                echo "  mc cp ${UPLOAD_PATH} ./profile.nsys-rep"
-                echo "Open with: nsys-ui profile.nsys-rep"
-              else
-                echo "WARN: nsys upload failed — profile remains on PVC at $NSYS_OUT"
-              fi
-            fi
-
-            if [ "$EXIT_CODE" -ne 0 ]; then
-              echo ""
-              echo "FAIL: test exited with code $EXIT_CODE"
-              exit "$EXIT_CODE"
-            fi
-            echo ""
-            echo "PASS: nsys profile captured, test passed."

infra/k8s/argo/refresh-deps-cache-template.yaml

@@ -1,181 +0,0 @@
-# refresh-deps-cache: rebuild ci-builder-cpu-with-deps:nightly image
-#
-# Triggered:
-#   - nightly via CronWorkflow (see CronWorkflow at the bottom of this file)
-#   - manually via:  argo submit --from workflowtemplate/refresh-deps-cache -n foxhunt
-#
-# What it does: clones the repo at HEAD of main, then runs Kaniko to build
-# infra/docker/Dockerfile.ci-deps-cache and pushes to
-# gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:nightly.
-#
-# The compile-and-deploy workflow's seed-deps-cache initContainer pulls
-# this image and rsyncs its /cargo-target-prebuilt/ into the cargo-target-cpu PVC.
----
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: refresh-deps-cache
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: refresh-deps-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  activeDeadlineSeconds: 7200  # 2 hours; full workspace cargo build is slow
-  serviceAccountName: argo-workflow
-  entrypoint: build
-  podMetadata:
-    labels:
-      # Reuse ci-pipeline label so the pod inherits argo-ci-pipeline netpol egress
-      # clone, gitlab-registry:5000 for image push, crates.io HTTPS for cargo
-      # NetworkPolicy clone — same egress targets, no functional difference.
-      app.kubernetes.io/component: ci-pipeline
-      app.kubernetes.io/part-of: foxhunt
-  ttlStrategy:
-    secondsAfterCompletion: 7200
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: deps-version
-        value: "1"
-      - name: image-tag
-        value: nightly
-
-  templates:
-    - name: build
-      inputs:
-        parameters:
-          - name: commit-sha
-            value: "{{workflow.parameters.commit-sha}}"
-          - name: deps-version
-            value: "{{workflow.parameters.deps-version}}"
-          - name: image-tag
-            value: "{{workflow.parameters.image-tag}}"
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      volumes:
-        - name: workspace
-          emptyDir:
-            sizeLimit: 30Gi  # full workspace + target/release ~10GB
-        - name: git-ssh-key
-          secret:
-            secretName: argo-git-ssh-key
-            defaultMode: 256
-        - name: registry-auth
-          secret:
-            secretName: gitlab-registry
-            items:
-              - key: .dockerconfigjson
-                path: config.json
-      initContainers:
-        - name: git-clone
-          image: alpine/git:latest
-          command: ["/bin/sh", "-c"]
-          resources:
-            requests:
-              cpu: 200m
-              memory: 256Mi
-            limits:
-              cpu: "1"
-              memory: 512Mi
-          volumeMounts:
-            - name: git-ssh-key
-              mountPath: /etc/git-ssh
-              readOnly: true
-            - name: workspace
-              mountPath: /workspace
-          args:
-            - |
-              set -ex
-              mkdir -p /root/.ssh
-              cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
-              chmod 600 /root/.ssh/id_ed25519
-              printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
-              chmod 600 /root/.ssh/config
-
-              SHA="{{inputs.parameters.commit-sha}}"
-              REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-              git clone --no-checkout --filter=blob:none "$REPO" /workspace/src
-              cd /workspace/src
-              git checkout "$SHA"
-              echo "Checked out $(git rev-parse --short HEAD)"
-      container:
-        image: gcr.io/kaniko-project/executor:debug
-        command: ["/busybox/sh", "-c"]
-        env:
-          - name: DOCKER_CONFIG
-            value: /kaniko/.docker
-        # Larger than build-ci-image because the inner cargo build is heavy.
-        resources:
-          requests:
-            cpu: "8"
-            memory: 16Gi
-          limits:
-            cpu: "16"
-            memory: 32Gi
-        volumeMounts:
-          - name: registry-auth
-            mountPath: /kaniko/.docker
-            readOnly: true
-          - name: workspace
-            mountPath: /workspace
-        args:
-          - |
-            /kaniko/executor \
-              --context=/workspace/src \
-              --dockerfile=/workspace/src/infra/docker/Dockerfile.ci-deps-cache \
-              --destination=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:{{inputs.parameters.image-tag}} \
-              --build-arg=DEPS_VERSION={{inputs.parameters.deps-version}} \
-              --build-arg=BASE_TAG=latest \
-              --insecure-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --skip-tls-verify-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --cache=true \
-              --cache-repo=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/cache \
-              --snapshot-mode=redo
-
----
-# Nightly cron — rebuilds the deps cache image at 03:00 UTC.
-# Suspended by default; enable with:
-#   kubectl -n foxhunt patch cronworkflow refresh-deps-cache-nightly \
-#     -p '{"spec":{"suspend":false}}' --type=merge
-apiVersion: argoproj.io/v1alpha1
-kind: CronWorkflow
-metadata:
-  name: refresh-deps-cache-nightly
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: refresh-deps-cache-nightly
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  schedules:
-    - "0 3 * * *"
-  timezone: UTC
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 2
-  failedJobsHistoryLimit: 3
-  suspend: true
-  workflowSpec:
-    entrypoint: trigger
-    serviceAccountName: argo-workflow
-    ttlStrategy:
-      secondsAfterCompletion: 7200
-    templates:
-      - name: trigger
-        steps:
-          - - name: refresh
-              templateRef:
-                name: refresh-deps-cache
-                template: build
-              arguments:
-                parameters:
-                  - name: commit-sha
-                    value: HEAD
-                  - name: deps-version
-                    value: "1"
-                  - name: image-tag
-                    value: nightly

infra/k8s/argo/sanitizer-test-template.yaml

@@ -1,311 +0,0 @@
-# infra/k8s/argo/sanitizer-test-template.yaml
-#
-# One-shot compute-sanitizer run on L40S. Wraps a smoke test under
-# `compute-sanitizer --tool memcheck` to validate no CUDA memory errors
-# (out-of-bounds, leaks, sync violations, race conditions).
-#
-# Why L40S not local: the laptop RTX 3050 Ti (4 GB VRAM) cannot fit
-# compute-sanitizer's instrumentation metadata alongside the production
-# training workload — the sanitizer falls back to "didn't track the
-# launch" with 60k+ internal-allocation errors. L40S (48 GB) has ample
-# headroom for memcheck's 2-3× shadow-memory overhead.
-#
-# Usage:
-#   argo submit --watch -n foxhunt sanitizer-test-template.yaml \
-#     -p commit-ref=main \
-#     -p test-name=iqn_quantile_monotonicity::iqn_multi_quantile_heads_produce_monotonic_estimates \
-#     -p sanitizer-tool=memcheck
-#
-# Or via the wrapper script: scripts/argo-sanitizer.sh
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: sanitizer-test
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: sanitizer-test
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: sanitizer-run
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: sanitizer-test
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 7200
-  activeDeadlineSeconds: 7200  # 2h cap — sanitizer is slow but bounded
-  arguments:
-    parameters:
-      - name: commit-ref
-        value: HEAD
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: cuda-compute-cap
-        value: "89"  # L40S Ada Lovelace
-      - name: test-name
-        value: "iqn_quantile_monotonicity::iqn_multi_quantile_heads_produce_monotonic_estimates"
-      - name: sanitizer-tool
-        value: memcheck  # memcheck | racecheck | synccheck | initcheck
-      - name: sanitizer-extra-args
-        value: ""
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda-test
-    # Use the same training-data-pvc as production training (full 27 months
-    # of OHLCV + MBP-10 + trades) instead of the smaller test-data-pvc which
-    # only has 3-4 months of MBP-10 — walk-forward needs ≥10 months.
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-        readOnly: true
-
-  templates:
-    - name: sanitizer-run
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_TEST_DATA
-            value: /data/futures-baseline
-          - name: TEST_DATA_DIR
-            value: /data/futures-baseline
-          # MBP-10 + trades are mandatory per feedback_mbp10_mandatory.md
-          # (OFI features are part of the production state vector).
-          - name: FOXHUNT_MBP10_DATA
-            value: /data/futures-baseline-mbp10
-          - name: FOXHUNT_TRADES_DATA
-            value: /data/futures-baseline-trades
-          - name: RUST_LOG
-            value: info
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        args:
-          - |
-            set -euo pipefail
-
-            REF="{{workflow.parameters.commit-ref}}"
-            TEST_NAME="{{workflow.parameters.test-name}}"
-            SANITIZER_TOOL="{{workflow.parameters.sanitizer-tool}}"
-            EXTRA_ARGS="{{workflow.parameters.sanitizer-extra-args}}"
-
-            echo "==================================="
-            echo "  compute-sanitizer L40S run"
-            echo "==================================="
-            echo "  Ref:      $REF"
-            echo "  Test:     $TEST_NAME"
-            echo "  Tool:     $SANITIZER_TOOL"
-            echo "  Pool:     {{workflow.parameters.gpu-pool}}"
-            echo "  Compute:  {{workflow.parameters.cuda-compute-cap}}"
-            echo "==================================="
-
-            # --- SSH setup (same as gpu-test-pipeline) ---
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            # --- Persistent checkout on PVC ---
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                TARGET=$(git rev-parse "origin/$REF")
-              else
-                TARGET=$(git rev-parse --verify "$REF" 2>/dev/null || echo "$REF")
-              fi
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$TARGET" ]; then
-                echo "=== Updating checkout to $REF ($TARGET) ==="
-                git checkout --force --detach "$TARGET"
-                git clean -fd
-              fi
-            else
-              echo "=== Initial clone ==="
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                git checkout --force --detach "origin/$REF"
-              else
-                git checkout "$REF"
-              fi
-            fi
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            export PATH="${CARGO_HOME}/bin:/usr/local/cuda/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            # --- PTX cache invalidation ---
-            bash scripts/ptx-cache-invalidate.sh "${CARGO_TARGET_DIR}/.ptx_cache"
-
-            # --- Verify compute-sanitizer is available ---
-            if ! command -v compute-sanitizer >/dev/null 2>&1; then
-              echo "ERROR: compute-sanitizer not found in PATH. CUDA toolkit incomplete."
-              exit 2
-            fi
-            echo "=== compute-sanitizer version ==="
-            compute-sanitizer --version | head -3
-
-            # --- Compile test binary + train_baseline_rl example ---
-            # train_baseline_rl is needed for multi_fold_convergence test which
-            # spawns it as a subprocess via `cargo run --example`. Pre-building
-            # avoids cargo doing it inside the sanitizer-instrumented run.
-            echo "=== Compiling test binary + train_baseline_rl (--release) ==="
-            cargo test -p ml --release --lib --no-run 2>&1 | tee /cargo-target/sanitizer-compile.log
-            cargo build -p ml --release --example train_baseline_rl 2>&1 | tee -a /cargo-target/sanitizer-compile.log
-
-            # Locate the freshest ml-* test binary
-            # Honour CARGO_TARGET_DIR — outputs go to ${CARGO_TARGET_DIR}/release/deps,
-            # NOT the repo-relative target/. (CI workflow sets CARGO_TARGET_DIR=/cargo-target.)
-            # Subshell + || true wraps the head -1 SIGPIPE so pipefail doesn't fire.
-            TARGET_DIR="${CARGO_TARGET_DIR:-target}"
-            TEST_BIN=$( (ls -t "$TARGET_DIR/release/deps/ml-"* 2>/dev/null \
-              | grep -vE '\.(d|rcgu|rmeta|rlib|so|json)$' \
-              | head -1) || true )
-            if [ -z "$TEST_BIN" ] || [ ! -x "$TEST_BIN" ]; then
-              echo "ERROR: could not locate compiled test binary in $TARGET_DIR/release/deps/"
-              ls -la "$TARGET_DIR/release/deps/" 2>&1 | head -20
-              exit 3
-            fi
-            echo "=== Test binary: $TEST_BIN ==="
-
-            # --- Verify the test exists in the binary ---
-            # grep -q + pipefail = SIGPIPE on early exit. Capture once, grep
-            # without -q so the consumer reads all input. Same below for nsys.
-            TEST_LIST=$("$TEST_BIN" --list 2>&1 || true)
-            if ! echo "$TEST_LIST" | grep -F "$TEST_NAME" > /dev/null; then
-              echo "ERROR: test '$TEST_NAME' not found in binary."
-              echo "Available smoke tests:"
-              echo "$TEST_LIST" | grep smoke_tests | (head -30 || true)
-              exit 4
-            fi
-
-            # --- Run under compute-sanitizer ---
-            mkdir -p /tmp/sanitizer-output
-            LOG_FILE=/tmp/sanitizer-output/${SANITIZER_TOOL}.log
-
-            echo ""
-            echo "==================================="
-            echo "  Launching compute-sanitizer..."
-            echo "==================================="
-            START_TS=$(date +%s)
-            # --target-processes all so multi_fold_convergence's spawned
-            # `train_baseline_rl` subprocess is also instrumented (the test
-            # itself does no GPU work — the child binary does).
-            set +e
-            compute-sanitizer \
-              --tool "$SANITIZER_TOOL" \
-              --target-processes all \
-              --launch-timeout 1200 \
-              --error-exitcode 99 \
-              --print-limit 200 \
-              --log-file "$LOG_FILE" \
-              ${EXTRA_ARGS} \
-              -- "$TEST_BIN" "$TEST_NAME" --ignored --nocapture
-            EXIT_CODE=$?
-            set -e
-            END_TS=$(date +%s)
-            ELAPSED=$((END_TS - START_TS))
-
-            echo ""
-            echo "==================================="
-            echo "  Sanitizer run complete"
-            echo "==================================="
-            echo "  Exit code:  $EXIT_CODE"
-            echo "  Elapsed:    ${ELAPSED}s"
-            echo "  Log:        $LOG_FILE ($(wc -l <"$LOG_FILE") lines)"
-            echo ""
-
-            # --- Triage report ---
-            echo "=== Sanitizer error summary ==="
-            grep -E "ERROR SUMMARY|Internal Sanitizer Error" "$LOG_FILE" | tail -5 || echo "  (no summary line)"
-            echo ""
-
-            # Categorise errors. compute-sanitizer prints "ERROR SUMMARY: N errors" at the
-            # end. "Internal Sanitizer Error" is sanitizer-internal (allocation failure,
-            # tracking gap) — NOT a real bug in the application code.
-            INTERNAL_ERRORS=$(grep -c "Internal Sanitizer Error" "$LOG_FILE" || true)
-            REAL_ERROR_LINE=$(grep "ERROR SUMMARY:" "$LOG_FILE" | tail -1 || echo "")
-            REAL_ERRORS=$(echo "$REAL_ERROR_LINE" | grep -oE "[0-9]+ errors" | head -1 | grep -oE "[0-9]+" || echo "0")
-
-            echo "=== Triage ==="
-            echo "  Internal Sanitizer Errors (instrumentation gaps): $INTERNAL_ERRORS"
-            echo "  Reported errors total:                            $REAL_ERRORS"
-            echo "  Real (non-internal) errors:                       $((REAL_ERRORS - INTERNAL_ERRORS))"
-            echo ""
-
-            # Show first 50 real (non-internal) errors with context
-            echo "=== First non-internal error excerpts ==="
-            grep -vE "Internal Sanitizer Error|^=========\s*$" "$LOG_FILE" \
-              | grep -E "^=========" \
-              | head -50 || echo "  (none)"
-
-            # Final pass/fail
-            if [ "$EXIT_CODE" -eq 99 ]; then
-              echo ""
-              echo "FAIL: compute-sanitizer detected real memory errors (--error-exitcode 99 fired)."
-              exit 1
-            elif [ "$EXIT_CODE" -ne 0 ]; then
-              echo ""
-              echo "FAIL: test process exited with code $EXIT_CODE (test failure or sanitizer abort)."
-              exit "$EXIT_CODE"
-            else
-              echo ""
-              echo "PASS: zero real memory errors detected, test passed."
-            fi

infra/k8s/argo/sccache-pvcs.yaml

@@ -1,32 +0,0 @@
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: sccache-cpu
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: sccache
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 20Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: sccache-cuda
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: sccache
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 20Gi

infra/k8s/argo/smoke-test-template.yaml

@@ -1,308 +0,0 @@
-# infra/k8s/argo/smoke-test-template.yaml
-#
-# Plain smoke-test runner on L40S — no compute-sanitizer, no nsys profile,
-# no MinIO artefact upload. Mirrors the compile + checkout + GPU + data-mount
-# layout of nsys-test-template.yaml but executes the test binary directly.
-#
-# Use this for fast multi-fold / single-epoch validation runs against the
-# full training-data PVC (27 months) when local laptop data is too short
-# (laptop only has 1 quarter of MBP-10/trades).
-#
-# Why a separate template: the nsys/sanitizer wrappers add overhead and
-# extract artefacts not needed for a straight "did the test pass on the
-# real dataset" gate. Keeping the smoke variant lean keeps the iteration
-# cycle short (no .nsys-rep upload, no sanitizer instrumentation slowdown).
-#
-# Usage:
-#   argo submit --watch -n foxhunt --from=wftmpl/smoke-test \
-#     -p commit-ref=main \
-#     -p test-name=multi_fold_convergence::test_multi_fold_convergence
-#
-# Or via the wrapper script: scripts/argo-smoke.sh
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: smoke-test
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: smoke-test
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: smoke-run
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: smoke-test
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 7200
-  activeDeadlineSeconds: 3600
-  arguments:
-    parameters:
-      - name: commit-ref
-        value: HEAD
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: cuda-compute-cap
-        value: "89"
-      - name: test-name
-        value: "multi_fold_convergence::test_multi_fold_convergence"
-      # Wipe `/cargo-target/release` for the ml + ml-dqn crates before
-      # compile so the smoke binary is built from a known-clean state. The
-      # PVC's persistent target dir accumulates rmeta/object artefacts
-      # across probes; file deletions (e.g., `regime_conditional.rs` in
-      # ff00af68a) can leave dangling references that perturb downstream
-      # codegen between bisect runs. Defaults to "false" — only set when
-      # bisecting suspected build-cache contamination.
-      - name: clean-cache
-        value: "false"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda-test
-    # Same training-data-pvc as production training (full 27 months of
-    # OHLCV + MBP-10 + trades) — laptop's 1-quarter MBP-10 truncates the
-    # fxcache below the smoke's 10-month minimum.
-    #
-    # RW (not readOnly) so the smoke can populate `/data/bin/$SHORT_SHA/`
-    # with the train binaries it just built, letting the next train run's
-    # `ensure-binary` cache check (train-multi-seed-template.yaml:235-246)
-    # hit and skip the 6-min compile. Read-side (fxcache + raw market data)
-    # is unchanged.
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-
-  templates:
-    - name: smoke-run
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_TEST_DATA
-            value: /data/futures-baseline
-          - name: TEST_DATA_DIR
-            value: /data/futures-baseline
-          # MBP-10 + trades are mandatory per feedback_mbp10_mandatory.md
-          # (OFI features are part of the production state vector).
-          - name: FOXHUNT_MBP10_DATA
-            value: /data/futures-baseline-mbp10
-          - name: FOXHUNT_TRADES_DATA
-            value: /data/futures-baseline-trades
-          - name: RUST_LOG
-            value: info
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-            # RW so the post-PASS cache-population step can write to
-            # /data/bin/$SHORT_SHA/. Read-side paths (fxcache, MBP-10,
-            # trades) are unaffected.
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        args:
-          - |
-            set -euo pipefail
-
-            REF="{{workflow.parameters.commit-ref}}"
-            TEST_NAME="{{workflow.parameters.test-name}}"
-
-            echo "==================================="
-            echo "  Plain smoke L40S run"
-            echo "==================================="
-            echo "  Ref:      $REF"
-            echo "  Test:     $TEST_NAME"
-            echo "  Pool:     {{workflow.parameters.gpu-pool}}"
-            echo "==================================="
-
-            # --- SSH setup ---
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            # --- Persistent checkout on PVC ---
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                TARGET=$(git rev-parse "origin/$REF")
-              else
-                TARGET=$(git rev-parse --verify "$REF" 2>/dev/null || echo "$REF")
-              fi
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$TARGET" ]; then
-                echo "=== Updating checkout to $REF ($TARGET) ==="
-                git checkout --force --detach "$TARGET"
-                git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                git checkout --force --detach "origin/$REF"
-              else
-                git checkout "$REF"
-              fi
-            fi
-            SHA=$(git rev-parse --short HEAD)
-            echo "Checked out $SHA"
-
-            export PATH="${CARGO_HOME}/bin:/usr/local/cuda/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            bash scripts/ptx-cache-invalidate.sh "${CARGO_TARGET_DIR}/.ptx_cache"
-
-            # --- Optional clean: ml + ml-dqn before compile ---
-            CLEAN_CACHE="{{workflow.parameters.clean-cache}}"
-            if [ "$CLEAN_CACHE" = "true" ]; then
-              echo "=== clean-cache=true — wiping ml + ml-dqn build artefacts ==="
-              cargo clean -p ml --release 2>&1 | tee /cargo-target/smoke-clean.log
-              cargo clean -p ml-dqn --release 2>&1 | tee -a /cargo-target/smoke-clean.log
-              echo "=== Cleaned. Forced fresh compile of ml/ml-dqn (deps stay cached) ==="
-            fi
-
-            # --- Compile test binary + the 3 train binaries ---
-            #
-            # The 3 examples mirror exactly what `ensure-binary` produces
-            # (see train-multi-seed-template.yaml line ~268). Compiling
-            # them here lets the post-PASS step populate the
-            # `/data/bin/$SHORT_SHA/` cache so the next train run at the
-            # same SHA skips its own ensure-binary compile.
-            echo "=== Compiling (--release --no-run + train binaries) ==="
-            cargo test -p ml --release --lib --no-run 2>&1 | tee /cargo-target/smoke-compile.log
-            cargo build -p ml --release \
-              --example train_baseline_rl \
-              --example evaluate_baseline \
-              --example precompute_features \
-              2>&1 | tee -a /cargo-target/smoke-compile.log
-
-            # Honour CARGO_TARGET_DIR — outputs go to ${CARGO_TARGET_DIR}/release/deps,
-            # NOT the repo-relative target/. Subshell + || true wraps the head -1
-            # SIGPIPE so pipefail doesn't fire.
-            TARGET_DIR="${CARGO_TARGET_DIR:-target}"
-            TEST_BIN=$( (ls -t "$TARGET_DIR/release/deps/ml-"* 2>/dev/null \
-              | grep -vE '\.(d|rcgu|rmeta|rlib|so|json)$' \
-              | head -1) || true )
-            if [ -z "$TEST_BIN" ] || [ ! -x "$TEST_BIN" ]; then
-              echo "ERROR: could not locate compiled test binary in $TARGET_DIR/release/deps/"
-              ls -la "$TARGET_DIR/release/deps/" 2>&1 | head -20
-              exit 3
-            fi
-            echo "=== Test binary: $TEST_BIN ==="
-
-            # grep -q + pipefail = SIGPIPE on early exit. Capture once, grep
-            # without -q so the consumer reads all input.
-            TEST_LIST=$("$TEST_BIN" --list 2>&1 || true)
-            if ! echo "$TEST_LIST" | grep -F "$TEST_NAME" > /dev/null; then
-              echo "ERROR: test '$TEST_NAME' not found"
-              echo "$TEST_LIST" | grep smoke_tests | (head -30 || true)
-              exit 4
-            fi
-
-            # --- Run the test ---
-            echo ""
-            echo "==================================="
-            echo "  Launching test..."
-            echo "==================================="
-            START_TS=$(date +%s)
-            set +e
-            "$TEST_BIN" "$TEST_NAME" --ignored --nocapture
-            EXIT_CODE=$?
-            set -e
-            END_TS=$(date +%s)
-            ELAPSED=$((END_TS - START_TS))
-
-            echo ""
-            echo "==================================="
-            echo "  Smoke run complete"
-            echo "==================================="
-            echo "  Exit code:  $EXIT_CODE"
-            echo "  Elapsed:    ${ELAPSED}s"
-            echo "  Commit:     $SHA"
-            echo "  Test:       $TEST_NAME"
-            echo "==================================="
-
-            if [ "$EXIT_CODE" -ne 0 ]; then
-              echo ""
-              echo "FAIL: test exited with code $EXIT_CODE"
-              exit "$EXIT_CODE"
-            fi
-            echo ""
-            echo "PASS: smoke completed."
-
-            # --- Populate /data/bin/$SHORT_SHA/ for ensure-binary cache ---
-            # On PASS only — broken binaries must not be cached. Mirrors
-            # train-multi-seed-template.yaml:270-274 exactly: same
-            # destination layout, same strip step. Idempotent: writes are
-            # to a SHA-keyed dir so concurrent smokes at different SHAs
-            # don't collide; same-SHA writes are deterministic compile
-            # output and overwrite-safe.
-            FULL_SHA=$(git rev-parse HEAD)
-            SHORT_SHA=$(echo "$FULL_SHA" | cut -c1-9)
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            BINARIES="train_baseline_rl evaluate_baseline precompute_features"
-            echo ""
-            echo "=== Populating ensure-binary cache at $BIN_DIR ==="
-            mkdir -p "$BIN_DIR"
-            for bin in $BINARIES; do
-              src="${CARGO_TARGET_DIR}/release/examples/$bin"
-              if [ ! -x "$src" ]; then
-                echo "WARN: $src not found (cache pop skipped for $bin)"
-                continue
-              fi
-              cp "$src" "$BIN_DIR/"
-              strip "$BIN_DIR/$bin" 2>/dev/null || true
-            done
-            ls -lh "$BIN_DIR/"
-            echo "=== Cache populated; next train@$SHORT_SHA hits ensure-binary cache ==="

infra/k8s/argo/train-multi-seed-template.yaml

@@ -1,605 +0,0 @@
-# Multi-seed training workflow — Plan 5 Task 5 Phase B (one-job-per-seed).
-#
-# Renders an Argo DAG that fans out N seeds into N parallel `train-single` task
-# instances. Each task receives `seed` via inputs.parameters and runs a
-# walk-forward training that internally sweeps all K folds via the binary's
-# `--max-folds K` arg. Per-job runtime is K× longer than the original (seed,
-# fold) matrix but fanout drops from N*K to N — a better fit for the L40S pool
-# (5-GPU capacity vs 30 jobs queueing) and a simpler binary contract
-# (`train_baseline_rl` is a multi-fold walk-forward executor; it does NOT
-# accept `--fold K`).
-#
-# The `# __MATRIX_TASKS__` marker on the dag.tasks line is replaced by
-# scripts/argo-train.sh with N generated WorkflowTask stanzas before
-# submission. This avoids hand-writing the matrix and keeps the template
-# human-readable.
-#
-# Usage:
-#   ./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --tag plan5-final
-#
-# DAG (per task):
-#   ensure-binary ──┐
-#   gpu-warmup ─────┼──> ensure-fxcache ──> [N parallel train-single tasks,
-#                                              one per seed, each runs all K folds]
-#                                              │
-#                                              └──> aggregate (manual via
-#                                                   scripts/gather-multi-seed-metrics.sh)
----
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: train-multi-seed
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: train-multi-seed
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: multi-seed-matrix
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  # Multi-seed runs: N jobs in parallel (one per seed), each running all K folds
-  # in walk-forward sequence. Allow 12h walltime — 6-fold runs are ~6× longer
-  # than the original per-(seed,fold) jobs but easily fit in 12h on L40S.
-  activeDeadlineSeconds: 43200
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: main
-      # Default sm_89 / ci-training-l40s as of 2026-05-14 per
-      # `feedback_default_to_l40s_pool.md`. argo-train.sh's
-      # CUDA_COMPUTE_CAP derivation passes "89" by default to match;
-      # explicit --gpu-pool ci-training-h100 overrides both back to
-      # sm_90 + H100.
-      - name: cuda-compute-cap
-        value: "89"
-      - name: model
-        value: dqn
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: hyperopt-trials
-        value: "0"  # Multi-seed runs typically skip hyperopt (re-use baseline params)
-      - name: hyperopt-epochs
-        value: "8"
-      - name: train-epochs
-        value: "50"
-      - name: symbol
-        value: ES.FUT
-      - name: initial-capital
-        value: "35000"
-      - name: tx-cost-bps
-        value: "0.1"
-      - name: tick-size
-        value: "0.25"
-      - name: spread-ticks
-        value: "1.0"
-      - name: sanitizer
-        value: "none"
-      - name: multi-seed
-        value: "5"
-      - name: folds
-        value: "6"
-      # Plan 5 Task 3 (A.4.1): nsys profile harness toggle. When "true", each
-      # train-single job wraps the training binary under `nsys profile` and
-      # uploads the resulting .nsys-rep to MinIO bucket
-      # foxhunt-training-artifacts/profiles/<short-sha>/. Default off.
-      - name: profile
-        value: "false"
-      # Bar formation params — passed to BOTH precompute_features (when
-      # building fxcache) AND train_baseline_rl (when looking up fxcache).
-      # MUST match between the two for fxcache HIT (cache key includes them
-      # as of 2026-05-09 architectural fix). Defaults match dqn-production.toml.
-      #
-      # imbalance-bar-threshold default: 20.0 (set 2026-05-10). At threshold=0.5
-      # the sampler produced 209M bars from 209M trade ticks (1:1 ratio,
-      # essentially per-tick) on workflow f5wnd, near-OOM in feature extraction
-      # (54Gi/56Gi limit). To match the volume-bar density baseline (~5.74M
-      # bars at 100 contracts), threshold ~20 is the right scale. Override via
-      # argo-train.sh --imbalance-bar-threshold for resolution-sweep experiments.
-      - name: imbalance-bar-threshold
-        value: "20.0"
-      - name: imbalance-bar-ewma-alpha
-        value: "0.1"
-      # Volume bar size (contracts/bar). Used when data-source != "mbp10".
-      # Default 100 matches DEFAULT_VOLUME_BAR_SIZE. Cache key includes this.
-      - name: volume-bar-size
-        value: "100"
-      # Data source mode: "mbp10" (imbalance bars from MBP-10) or "ohlcv"
-      # (volume bars from trades). Threaded into both precompute_features and
-      # the trainer so cache keys align.
-      - name: data-source
-        value: "mbp10"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  volumeClaimTemplates:
-    - metadata:
-        name: workspace
-      spec:
-        accessModes: ["ReadWriteOnce"]
-        storageClassName: scw-bssd
-        resources:
-          requests:
-            storage: 5Gi
-
-  templates:
-    # ── DAG: fan out to N train-single tasks (one per seed) ──
-    # The `# __MATRIX_TASKS__` marker is replaced by argo-train.sh with the
-    # generated per-seed WorkflowTask stanzas. Each task sweeps all K folds
-    # via the binary's `--max-folds {{workflow.parameters.folds}}` argument.
-    # The marker MUST stay on its own line for the awk substitution to work.
-    - name: multi-seed-matrix
-      dag:
-        tasks:
-          - name: ensure-binary
-            template: ensure-binary
-          - name: gpu-warmup
-            template: gpu-warmup
-          - name: ensure-fxcache
-            template: ensure-fxcache
-            dependencies: [ensure-binary]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          # __MATRIX_TASKS__
-
-    # ── ensure-binary: cache-or-compile training binaries by commit SHA ──
-    # (Identical to train-template.yaml. Kept inline rather than templated
-    # via wftmpl-cross-ref to avoid Argo's reluctance to chase template
-    # references across WorkflowTemplates at submission time.)
-    - name: ensure-binary
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CARGO_INCREMENTAL
-            value: "1"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            MODEL="{{workflow.parameters.model}}"
-
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"; git fetch origin
-                SHA=$(git rev-parse "origin/$BRANCH"); cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"
-                cd "$BUILD"; git checkout "origin/$BRANCH"
-                SHA=$(git rev-parse HEAD); cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            case "$MODEL" in
-              alpha-rl) BINARIES="alpha_rl_train" ;;
-              dqn|ppo) BINARIES="train_baseline_rl evaluate_baseline precompute_features" ;;
-              *)       BINARIES="train_baseline_supervised evaluate_supervised precompute_features" ;;
-            esac
-
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            ALL_CACHED=true
-            for bin in $BINARIES; do
-              if [ ! -x "$BIN_DIR/$bin" ]; then ALL_CACHED=false; break; fi
-            done
-
-            if [ "$ALL_CACHED" = "true" ]; then
-              echo "=== Cache HIT: all binaries present in $BIN_DIR ==="
-              ls -lh "$BIN_DIR/"
-              echo "$SHORT_SHA" > /tmp/sha
-              exit 0
-            fi
-
-            echo "=== Cache MISS: compiling binaries for $SHORT_SHA ==="
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"; git fetch origin
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$SHA" ]; then
-                git checkout --force "$SHA"; git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"; git checkout "$SHA"
-            fi
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            if [ "$MODEL" = "alpha-rl" ]; then
-              cargo build --release -p ml-alpha --example alpha_rl_train
-            else
-              ML_EXAMPLE_ARGS=""
-              for ex in $BINARIES; do
-                ML_EXAMPLE_ARGS="$ML_EXAMPLE_ARGS --example $ex"
-              done
-              cargo build --release -p ml --features ml/cuda $ML_EXAMPLE_ARGS
-            fi
-
-            mkdir -p "$BIN_DIR"
-            for bin in $BINARIES; do
-              cp "$CARGO_TARGET_DIR/release/examples/$bin" "$BIN_DIR/"
-            done
-            strip "$BIN_DIR/"*
-            echo "$SHORT_SHA" > /tmp/sha
-
-    # ── gpu-warmup: trigger GPU node autoscale during compile ──
-    - name: gpu-warmup
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: busybox:1.37
-        command: ["/bin/sh", "-c"]
-        args:
-          - |
-            echo "GPU warmup: triggering node autoscale..."
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: 200m
-            memory: 128Mi
-
-    # ── ensure-fxcache: precompute feature cache (shared across all seeds/folds) ──
-    - name: ensure-fxcache
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: ubuntu:24.04
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-        resources:
-          requests:
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            cpu: "28"
-            memory: 96Gi
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-            readOnly: false
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BINARY="/data/bin/$SHA/precompute_features"
-            if [ ! -x "$BINARY" ]; then
-              echo "ERROR: precompute_features not found at $BINARY"; exit 1
-            fi
-            export RAYON_NUM_THREADS=20
-            if $BINARY \
-              --data-dir /data/futures-baseline \
-              --mbp10-data-dir /data/futures-baseline-mbp10 \
-              --trades-data-dir /data/futures-baseline-trades \
-              --output-dir /feature-cache \
-              --symbol {{workflow.parameters.symbol}} \
-              --data-source {{workflow.parameters.data-source}} \
-              --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-              --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-              --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-              --yes; then
-              echo "=== Feature cache ready ==="
-            else
-              echo "=== Cache stale or missing — regenerating ==="
-              # NOTE: don't blanket-rm /feature-cache/*.fxcache anymore. With
-              # bar-params now in the cache key, multiple valid caches can
-              # coexist (different threshold experiments). The early-exit
-              # check above already validates the specific key — fall through
-              # to regen only the one we need.
-              $BINARY \
-                --data-dir /data/futures-baseline \
-                --mbp10-data-dir /data/futures-baseline-mbp10 \
-                --trades-data-dir /data/futures-baseline-trades \
-                --output-dir /feature-cache \
-                --symbol {{workflow.parameters.symbol}} \
-                --data-source {{workflow.parameters.data-source}} \
-                --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-                --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-                --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-                --yes
-            fi
-
-    # ── train-single: one seed, all folds (training instance) ──
-    # Invoked once per matrix entry. Reads SEED from inputs.parameters and
-    # forwards it via `--seed`; the binary's `--max-folds K` arg drives the
-    # walk-forward sweep over all K folds inside this single process.
-    # Plan 5 Task 5 Phase B pivot: was per-(seed, fold) on the failed deploy
-    # because train_baseline_rl does not accept `--fold N` (it is a multi-fold
-    # executor, not a single-fold one). One-job-per-seed matches the binary's
-    # actual contract and the L40S pool's capacity.
-    - name: train-single
-      inputs:
-        parameters:
-          - name: seed
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: IfNotPresent
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_FEATURE_CACHE_DIR
-            value: /feature-cache
-          - name: SEED
-            value: "{{inputs.parameters.seed}}"
-          # Plan 5 Task 3 (A.4.1): MinIO creds for the optional `mc cp` of
-          # the .nsys-rep artefact at the end of train-single. Both refs are
-          # `optional: true` so the env mount succeeds on clusters that do
-          # not pre-create `minio-credentials` (the upload then warn-fails
-          # gracefully — training itself doesn't depend on these).
-          - name: MINIO_ACCESS_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: access-key
-                optional: true
-          - name: MINIO_SECRET_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: secret-key
-                optional: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-            readOnly: true
-        args:
-          - |
-            set -e
-            # Resolve SHA via the binary cache directory layout the
-            # ensure-binary task established. multi-seed-matrix passes the
-            # SHA implicitly via the workflow-scoped /data/bin tree.
-            SHA=$(ls -1t /data/bin | head -1)
-            export PATH="/data/bin/$SHA:$PATH"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-            nvidia-smi
-
-            MODEL="{{workflow.parameters.model}}"
-            case "$MODEL" in
-              alpha-rl) BINARY=alpha_rl_train ;;
-              dqn|ppo) BINARY=train_baseline_rl ;;
-              *)       BINARY=train_baseline_supervised ;;
-            esac
-
-            mkdir -p /workspace/output
-
-            # Plan 5 Task 3 (A.4.1): optional nsys profile wrapper.
-            # When workflow.parameters.profile=="true", wrap the training
-            # binary under `nsys profile --capture-range=cudaProfilerApi`.
-            # The .nsys-rep is uploaded to MinIO bucket
-            # foxhunt-training-artifacts/profiles/<short-sha>/ in the
-            # post-training upload block below.
-            PROFILE="{{workflow.parameters.profile}}"
-            POD_NAME="${HOSTNAME:-pod}"
-            NSYS_OUT="/workspace/output/profile-${POD_NAME}.nsys-rep"
-            NSYS_PREFIX=""
-            if [ "$PROFILE" = "true" ]; then
-              NSYS_BIN=$(which nsys 2>/dev/null || echo "/usr/local/cuda/bin/nsys")
-              if [ -x "$NSYS_BIN" ]; then
-                NSYS_PREFIX="$NSYS_BIN profile --capture-range=cudaProfilerApi --output=${NSYS_OUT} --force-overwrite=true"
-                echo "=== nsys profile enabled: ${NSYS_OUT} ==="
-              else
-                echo "WARN: --profile requested but nsys not found at $NSYS_BIN — running without"
-              fi
-            fi
-
-            echo "=== Training: $MODEL seed=$SEED folds={{workflow.parameters.folds}} ==="
-            if [ "$MODEL" = "alpha-rl" ]; then
-              stdbuf -oL $NSYS_PREFIX ${BINARY} \
-                --mbp10-data-dir /data/futures-baseline-mbp10/{{workflow.parameters.symbol}} \
-                --predecoded-dir /feature-cache/predecoded \
-                --out /workspace/output \
-                --n-steps 50000 \
-                --n-backtests 16 \
-                --seed "$SEED" \
-                --instrument-mode all
-            else
-              stdbuf -oL $NSYS_PREFIX ${BINARY} \
-                --model "$MODEL" \
-                --symbol {{workflow.parameters.symbol}} \
-                --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
-                --tick-size {{workflow.parameters.tick-size}} \
-                --spread-ticks {{workflow.parameters.spread-ticks}} \
-                --initial-capital {{workflow.parameters.initial-capital}} \
-                --data-dir /data/futures-baseline \
-                --mbp10-data-dir /data/futures-baseline-mbp10 \
-                --trades-data-dir /data/futures-baseline-trades \
-                --output-dir /workspace/output \
-                --epochs {{workflow.parameters.train-epochs}} \
-                --seed "$SEED" \
-                --max-folds {{workflow.parameters.folds}} \
-                --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-                --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-                --volume-bar-size {{workflow.parameters.volume-bar-size}}
-            fi
-
-            echo "=== Training complete: seed=$SEED folds={{workflow.parameters.folds}} ==="
-
-            # Plan 5 Task 3 (A.4.1): upload .nsys-rep to MinIO if profile run.
-            # mc is fetched on-demand (~25 MB single static binary) — the
-            # ci-builder training image does not bundle it. Bucket:
-            # foxhunt-training-artifacts/profiles/<short-sha>/.
-            if [ "$PROFILE" = "true" ] && [ -f "$NSYS_OUT" ]; then
-              echo "=== Uploading nsys profile to MinIO ==="
-              MC_BIN=$(which mc 2>/dev/null || echo "")
-              if [ -z "$MC_BIN" ]; then
-                MC_BIN=/tmp/mc
-                curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/mc -o "$MC_BIN" || {
-                  echo "WARN: failed to download mc — skipping upload"
-                  exit 0
-                }
-                chmod +x "$MC_BIN"
-              fi
-              "$MC_BIN" alias set foxhunt http://minio.foxhunt.svc.cluster.local:9000 \
-                "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY" 2>/dev/null || true
-              "$MC_BIN" cp "$NSYS_OUT" \
-                "foxhunt/foxhunt-training-artifacts/profiles/$SHA/profile-seed${SEED}-${POD_NAME}.nsys-rep" || \
-                echo "WARN: nsys upload failed for seed=$SEED"
-              echo "=== nsys profile upload complete ==="
-            fi
-
-    # ── notify-result: post workflow outcome to Mattermost (onExit) ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"; exit 0
-            fi
-            EMOJI=":x:"
-            [ "$STATUS" = "Succeeded" ] && EMOJI=":white_check_mark:"
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **${NAME}** (multi-seed) — ${STATUS}\"}"
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/train-template.yaml

@@ -1,838 +0,0 @@
-# Unified training workflow — replaces all model-specific training templates.
-#
-# Key innovation: binary caching by commit SHA on PVC.
-# Check /data/bin/$SHA first; compile only on cache miss.
-#
-# DAG:
-#   ensure-binary ──┐
-#   gpu-warmup ─────┼──> ensure-fxcache ──> hyperopt ──> train-best ──> evaluate ──> upload-results
-#
-# When hyperopt-trials=0 (the `scripts/argo-precompute.sh` path),
-# gpu-warmup / hyperopt / train-best / evaluate all skip and the
-# workflow runs ensure-binary → ensure-fxcache only — no GPU node
-# gets provisioned for nothing.
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/train
-#   argo submit -n foxhunt --from=wftmpl/train -p model=ppo -p train-epochs=100
-#   argo submit -n foxhunt --from=wftmpl/train -p hyperopt-trials=0   # fxcache-only path
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: train
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: train
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: pipeline
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 28800  # 8 hours
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: main
-      - name: cuda-compute-cap
-        value: "90"
-      - name: model
-        value: dqn
-      # gpu-pool default: ci-training-l40s (set 2026-05-14 per
-      # `feedback_default_to_l40s_pool.md` — SP-chain training has been
-      # standardising on L40S since 2026-05-09; the prior ci-training-h100
-      # default required every SP-run invocation to pass an explicit
-      # `--gpu-pool ci-training-l40s` override. H100 remains opt-in via
-      # `--gpu-pool ci-training-h100` for runs that genuinely need
-      # 80 GB VRAM or sm_90 features). Compute-cap derivation in
-      # `argo-train.sh` matches this default to sm_89 (Ada Lovelace).
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: hyperopt-trials
-        value: "20"
-      - name: hyperopt-epochs
-        value: "8"
-      - name: train-epochs
-        value: "50"
-      - name: symbol
-        value: ES.FUT
-      - name: initial-capital
-        value: "35000"
-      - name: tx-cost-bps
-        value: "0.1"
-      - name: tick-size
-        value: "0.25"
-      - name: spread-ticks
-        value: "1.0"
-      - name: sanitizer
-        value: "none"  # "none", "memcheck", "racecheck", "synccheck"
-      # Bar formation params — passed to BOTH precompute_features (when
-      # building fxcache) AND train_baseline_rl (when looking up fxcache).
-      # MUST match between the two for fxcache HIT (cache key includes them
-      # as of 2026-05-09 architectural fix).
-      #
-      # imbalance-bar-threshold default: 20.0 (set 2026-05-10). At threshold=0.5
-      # the sampler produced 209M bars from 209M trade ticks (1:1 ratio,
-      # essentially per-tick) on workflow f5wnd, near-OOM in feature extraction.
-      # threshold=20 produces ~5-6M bars matching the volume-bar density baseline.
-      - name: imbalance-bar-threshold
-        value: "20.0"
-      - name: imbalance-bar-ewma-alpha
-        value: "0.1"
-      - name: volume-bar-size
-        value: "100"
-      - name: data-source
-        value: "mbp10"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  volumeClaimTemplates:
-    - metadata:
-        name: workspace
-      spec:
-        accessModes: ["ReadWriteOnce"]
-        storageClassName: scw-bssd
-        resources:
-          requests:
-            storage: 5Gi
-
-  templates:
-    # ── DAG: orchestrate all steps ──
-    - name: pipeline
-      dag:
-        tasks:
-          - name: ensure-binary
-            template: ensure-binary
-          - name: gpu-warmup
-            template: gpu-warmup
-            # gpu-warmup pre-provisions an L40S node so hyperopt /
-            # train-best don't pay autoscaler latency at start. When
-            # hyperopt-trials==0 (precompute-only path used by
-            # `scripts/argo-precompute.sh`), both downstream consumers
-            # are skipped and the GPU node would sit idle until the
-            # workflow ends. Skip the warmup in that case to avoid
-            # provisioning expensive L40S capacity for nothing.
-            when: "{{workflow.parameters.hyperopt-trials}} != 0"
-          - name: ensure-fxcache
-            template: ensure-fxcache
-            dependencies: [ensure-binary]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: hyperopt
-            template: hyperopt
-            dependencies: [ensure-binary, gpu-warmup, ensure-fxcache]
-            when: "{{workflow.parameters.hyperopt-trials}} != 0"
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: train-best
-            template: train-best
-            dependencies: [ensure-binary, gpu-warmup, ensure-fxcache, hyperopt]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: evaluate
-            template: evaluate
-            dependencies: [train-best]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: upload-results
-            template: upload-results
-            dependencies: [evaluate]
-
-    # ── ensure-binary: cache-or-compile training binaries by commit SHA ──
-    - name: ensure-binary
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          # sccache: per-crate compile-output cache. Disk-local on the same
-          # cargo-target-cuda PVC so no network cost on hit; survives across
-          # pods and commits (cargo's incremental cache is invalidated by
-          # git checkout mtime touches — sccache is content-hash keyed so it
-          # isn't). Binary assumed present in ci-builder image.
-          #
-          # CARGO_INCREMENTAL=0 is required to make sccache effective:
-          # the project's Cargo.toml / .cargo/config.toml set
-          # `incremental = true`, which emits per-query save-analysis
-          # artifacts that sccache does not cache. Turning incremental off
-          # lets rustc emit pure object output that sccache hashes
-          # uniformly — same source + same args → cache hit.
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /cargo-target/sccache
-          - name: SCCACHE_CACHE_SIZE
-            value: "40G"
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            MODEL="{{workflow.parameters.model}}"
-
-            # ── SSH setup ──
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-
-            git config --global --add safe.directory "$BUILD"
-
-            # ── Resolve SHA ──
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"
-                git fetch origin
-                SHA=$(git rev-parse "origin/$BRANCH")
-                cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"
-                cd "$BUILD"
-                git checkout "origin/$BRANCH"
-                SHA=$(git rev-parse HEAD)
-                cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            # ── Derive needed binaries from model ──
-            case "$MODEL" in
-              dqn|ppo)
-                BINARIES="hyperopt_baseline_rl train_baseline_rl evaluate_baseline precompute_features"
-                ;;
-              *)
-                BINARIES="hyperopt_baseline_supervised train_baseline_supervised evaluate_supervised precompute_features"
-                ;;
-            esac
-
-            # ── Cache check ──
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            ALL_CACHED=true
-            for bin in $BINARIES; do
-              if [ ! -x "$BIN_DIR/$bin" ]; then
-                ALL_CACHED=false
-                break
-              fi
-            done
-
-            if [ "$ALL_CACHED" = "true" ]; then
-              echo "=== Cache HIT: all binaries present in $BIN_DIR ==="
-              ls -lh "$BIN_DIR/"
-              echo "$SHORT_SHA" > /tmp/sha
-              exit 0
-            fi
-
-            echo "=== Cache MISS: compiling binaries for $SHORT_SHA ==="
-
-            # ── Clone / checkout ──
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$SHA" ]; then
-                echo "Updating checkout: $(echo $CURRENT | cut -c1-8) -> $SHORT_SHA"
-                git checkout --force "$SHA"
-                git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git checkout "$SHA"
-            fi
-
-            # ── Build ──
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            ML_EXAMPLE_ARGS=""
-            for ex in $BINARIES; do
-              ML_EXAMPLE_ARGS="$ML_EXAMPLE_ARGS --example $ex"
-            done
-
-            echo "Building: $BINARIES"
-            echo "  CUDA_COMPUTE_CAP=$CUDA_COMPUTE_CAP"
-            cargo build --release -p ml --features ml/cuda $ML_EXAMPLE_ARGS
-
-            # ── Install to cache dir ──
-            mkdir -p "$BIN_DIR"
-            for bin in $BINARIES; do
-              cp "$CARGO_TARGET_DIR/release/examples/$bin" "$BIN_DIR/"
-            done
-            strip "$BIN_DIR/"*
-
-            echo "=== Cached binaries ==="
-            ls -lh "$BIN_DIR/"
-
-            # ── Prune old SHAs: keep last 5 ──
-            cd /data/bin
-            ls -1t | tail -n +6 | while read -r old; do
-              echo "Pruning old cache: $old"
-              rm -rf "$old"
-            done
-
-            echo "$SHORT_SHA" > /tmp/sha
-
-    # ── ensure-fxcache: precompute feature cache if needed ──
-    #
-    # Pinned to the high-memory ci-compile-cpu-hm pool (POP2-HM-32C-256G,
-    # 32 vCPU + 256GB RAM, min_size=0). The 9-quarter precompute_features
-    # peaks ~50-60GB during the post-OFI alpha_trades conversion which
-    # OOM-killed the standard 64GB ci-compile-cpu pool twice on
-    # 2026-05-16 (workflows train-wq8b8 + train-2l6p4 both died at
-    # exitCode 137 right after "OFI computed"). The HM pool autoscales
-    # to zero when idle so this pin only costs anything during an
-    # actual fxcache rebuild.
-    - name: ensure-fxcache
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu-hm
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: ubuntu:24.04
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-        resources:
-          requests:
-            cpu: "4"
-            memory: 16Gi
-          limits:
-            cpu: "28"
-            memory: 200Gi
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BINARY="/data/bin/$SHA/precompute_features"
-
-            if [ ! -x "$BINARY" ]; then
-              echo "ERROR: precompute_features not found at $BINARY"
-              exit 1
-            fi
-
-            # fxcache has built-in version validation (FXCACHE_VERSION in header).
-            # The precompute binary checks existing cache — if version matches,
-            # it skips regeneration. If version mismatches, it fails and we
-            # delete + regenerate. No unconditional rm — cache is reused when valid.
-            echo "=== Running precompute_features (SHA: $SHA) ==="
-            if $BINARY \
-              --data-dir /data/futures-baseline \
-              --mbp10-data-dir /data/futures-baseline-mbp10 \
-              --trades-data-dir /data/futures-baseline-trades \
-              --output-dir /feature-cache \
-              --symbol {{workflow.parameters.symbol}} \
-              --data-source {{workflow.parameters.data-source}} \
-              --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-              --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-              --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-              --yes; then
-              echo "=== Feature cache ready ==="
-            else
-              echo "=== Cache stale or missing — regenerating ==="
-              # NOTE: don't blanket-rm /feature-cache/*.fxcache anymore. With
-              # bar-params now in the cache key, multiple valid caches can
-              # coexist (different threshold experiments).
-              $BINARY \
-                --data-dir /data/futures-baseline \
-                --mbp10-data-dir /data/futures-baseline-mbp10 \
-                --trades-data-dir /data/futures-baseline-trades \
-                --output-dir /feature-cache \
-                --symbol {{workflow.parameters.symbol}} \
-                --data-source {{workflow.parameters.data-source}} \
-                --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-                --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-                --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-                --yes
-              echo "=== Feature cache regenerated ==="
-            fi
-
-    # ── gpu-warmup: trigger GPU node autoscale during compile ──
-    - name: gpu-warmup
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: busybox:1.37
-        command: ["/bin/sh", "-c"]
-        args:
-          - |
-            echo "GPU warmup: triggering node autoscale..."
-            echo "GPU node scheduled, exiting to free resources"
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: 200m
-            memory: 128Mi
-
-    # ── hyperopt: PSO/TPE hyperparameter optimization on GPU ──
-    - name: hyperopt
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: IfNotPresent
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_FEATURE_CACHE_DIR
-            value: /feature-cache
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            export PATH="/data/bin/$SHA:$PATH"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-            nvidia-smi
-
-            mkdir -p /workspace/output/hyperopt
-
-            MODEL="{{workflow.parameters.model}}"
-            case "$MODEL" in
-              dqn|ppo) BINARY=hyperopt_baseline_rl ;;
-              *)       BINARY=hyperopt_baseline_supervised ;;
-            esac
-
-            echo "=== Running hyperopt: $MODEL ($BINARY) ==="
-            echo "  Trials: {{workflow.parameters.hyperopt-trials}}, Epochs: {{workflow.parameters.hyperopt-epochs}}"
-
-            ${BINARY} \
-              --model "$MODEL" \
-              --phase fast \
-              --trials {{workflow.parameters.hyperopt-trials}} \
-              --epochs {{workflow.parameters.hyperopt-epochs}} \
-              --symbol {{workflow.parameters.symbol}} \
-              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
-              --tick-size {{workflow.parameters.tick-size}} \
-              --spread-ticks {{workflow.parameters.spread-ticks}} \
-              --initial-capital {{workflow.parameters.initial-capital}} \
-              --data-dir /data/futures-baseline \
-              --mbp10-data-dir /data/futures-baseline-mbp10 \
-              --trades-data-dir /data/futures-baseline-trades \
-              --base-dir /workspace/output/hyperopt \
-              --output /workspace/output/${MODEL}_hyperopt_results.json
-
-            echo "=== Hyperopt complete ==="
-            cat /workspace/output/${MODEL}_hyperopt_results.json 2>/dev/null || echo "No results file"
-
-    # ── train-best: full training with best hyperparams ──
-    - name: train-best
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: IfNotPresent
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_FEATURE_CACHE_DIR
-            value: /feature-cache
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            export PATH="/data/bin/$SHA:$PATH"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-            nvidia-smi
-
-            MODEL="{{workflow.parameters.model}}"
-            case "$MODEL" in
-              dqn|ppo) BINARY=train_baseline_rl ;;
-              *)       BINARY=train_baseline_supervised ;;
-            esac
-
-            HYPEROPT_FLAG=""
-            if [ -f "/workspace/output/${MODEL}_hyperopt_results.json" ]; then
-              HYPEROPT_FLAG="--hyperopt-params /workspace/output/${MODEL}_hyperopt_results.json"
-              echo "  Using hyperopt results: ${MODEL}_hyperopt_results.json"
-            else
-              echo "  No hyperopt results — training with default hyperparams"
-            fi
-
-            echo "=== Training: $MODEL ({{workflow.parameters.train-epochs}} epochs) ==="
-
-            # compute-sanitizer / nsys: optional GPU debugging/profiling
-            SANITIZER="{{workflow.parameters.sanitizer}}"
-            SANITIZER_PREFIX=""
-            if [ "$SANITIZER" = "nsys" ]; then
-              NSYS_BIN=$(which nsys 2>/dev/null || echo "/usr/local/cuda/bin/nsys")
-              if [ -x "$NSYS_BIN" ]; then
-                NSYS_OUT="/feature-cache/nsys_$(date +%Y%m%d_%H%M%S)"
-                mkdir -p /feature-cache
-                SANITIZER_PREFIX="$NSYS_BIN profile -o $NSYS_OUT --cuda-graph-trace=node --stats=true --show-output=true -f true --duration=60"
-                echo "  nsys profiling enabled — output: ${NSYS_OUT}.nsys-rep (persistent PVC)"
-                echo "  --cuda-graph-trace=node: per-kernel timing inside parent graph"
-                echo "  --cudabacktrace=kernel: kernel source attribution"
-                echo "  --capture-range=cudaProfilerApi: use cudaProfilerStart/Stop to limit capture"
-                echo "  WARNING: ~2x slower — use with 1-2 epochs only"
-              else
-                echo "  WARNING: nsys not found at $NSYS_BIN — running without"
-              fi
-            elif [ "$SANITIZER" != "none" ] && [ -n "$SANITIZER" ]; then
-              SANITIZER_BIN=$(which compute-sanitizer 2>/dev/null || echo "/usr/local/cuda/bin/compute-sanitizer")
-              if [ -x "$SANITIZER_BIN" ]; then
-                SANITIZER_PREFIX="$SANITIZER_BIN --tool $SANITIZER --print-limit 20 --error-exitcode 1"
-                echo "  compute-sanitizer enabled: --tool $SANITIZER"
-                echo "  WARNING: 10-100x slower — use with 1-2 epochs only"
-              else
-                echo "  WARNING: compute-sanitizer not found at $SANITIZER_BIN — running without"
-              fi
-            fi
-
-            stdbuf -oL $SANITIZER_PREFIX ${BINARY} \
-              --model "$MODEL" \
-              --symbol {{workflow.parameters.symbol}} \
-              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
-              --tick-size {{workflow.parameters.tick-size}} \
-              --spread-ticks {{workflow.parameters.spread-ticks}} \
-              --initial-capital {{workflow.parameters.initial-capital}} \
-              --data-dir /data/futures-baseline \
-              --mbp10-data-dir /data/futures-baseline-mbp10 \
-              --trades-data-dir /data/futures-baseline-trades \
-              --output-dir /workspace/output \
-              --epochs {{workflow.parameters.train-epochs}} \
-              --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-              --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-              --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-              $HYPEROPT_FLAG
-
-            echo "=== Training complete ==="
-
-    # ── evaluate: run evaluation on trained model ──
-    - name: evaluate
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: IfNotPresent
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "2"
-            memory: 16Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            export PATH="/data/bin/$SHA:$PATH"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-
-            MODEL="{{workflow.parameters.model}}"
-            case "$MODEL" in
-              dqn|ppo) BINARY=evaluate_baseline ;;
-              *)       BINARY=evaluate_supervised ;;
-            esac
-
-            echo "=== Evaluating: $MODEL ==="
-            # Args must match evaluate_baseline's Args struct — see crates/ml/examples/evaluate_baseline.rs
-            # `--models-dir` (not --checkpoint-dir), `--output` is a FILE path (not --output-dir).
-            mkdir -p /workspace/output/eval
-            ${BINARY} \
-              --model "$MODEL" \
-              --symbol {{workflow.parameters.symbol}} \
-              --data-dir /data/futures-baseline \
-              --models-dir /workspace/output \
-              --output /workspace/output/eval/evaluation_report.json \
-              --initial-capital {{workflow.parameters.initial-capital}} \
-              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
-              --tick-size {{workflow.parameters.tick-size}} \
-              --spread-ticks {{workflow.parameters.spread-ticks}} || {
-              echo "WARN: Evaluation failed, continuing"
-            }
-
-            echo "=== Evaluation complete ==="
-
-    # ── upload-results: push artifacts to GitLab packages ──
-    - name: upload-results
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: 100m
-            memory: 128Mi
-          limits:
-            cpu: 500m
-            memory: 256Mi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-        args:
-          - |
-            set -e
-            MODEL="{{workflow.parameters.model}}"
-            SYMBOL="{{workflow.parameters.symbol}}"
-            TIMESTAMP=$(date +%Y%m%d-%H%M%S)
-            GITLAB_API="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181/api/v4"
-            PKG_NAME="foxhunt-training-results"
-            PKG_VERSION="${MODEL}-${SYMBOL}-${TIMESTAMP}"
-
-            echo "Uploading training artifacts: ${PKG_NAME}/${PKG_VERSION}"
-
-            UPLOADED=0
-            find /workspace/output -type f | while read -r file; do
-              REL_PATH="${file#/workspace/output/}"
-              SAFE_NAME=$(echo "$REL_PATH" | tr '/' '--')
-              echo "  Uploading ${REL_PATH} as ${SAFE_NAME}..."
-              curl -f --upload-file "$file" \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB_API}/projects/1/packages/generic/${PKG_NAME}/${PKG_VERSION}/${SAFE_NAME}" && \
-                UPLOADED=$((UPLOADED + 1)) || \
-                echo "  WARN: Failed to upload ${REL_PATH}"
-            done
-
-            echo "=== Upload complete (${UPLOADED} files) ==="
-            echo "Package: ${PKG_NAME}/${PKG_VERSION}"
-
-    # ── notify-result: post workflow outcome to Mattermost (onExit) ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"
-              exit 0
-            fi
-
-            if [ "$STATUS" = "Succeeded" ]; then
-              EMOJI=":white_check_mark:"
-            else
-              EMOJI=":x:"
-            fi
-
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **${NAME}** — ${STATUS} ({{workflow.duration}}s)\"}"
-
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/values.yaml

@@ -1,80 +0,0 @@
-# Argo Workflows Helm values — foxhunt cluster
-# Chart: argo/argo-workflows v1.0.2 (Argo v4.0.2)
-# Install: helm upgrade argo-workflows argo/argo-workflows -n foxhunt -f infra/k8s/argo/values.yaml
-
-controller:
-  nodeSelector:
-    k8s.scaleway.com/pool-name: platform
-  resources:
-    requests:
-      cpu: 50m
-      memory: 128Mi
-    limits:
-      cpu: 200m
-      memory: 256Mi
-  # Only watch foxhunt namespace (not cluster-wide)
-  workflowNamespaces:
-    - foxhunt
-  # Workflow archive — persists workflow state + logs to PostgreSQL after pod GC
-  persistence:
-    archive: true
-    postgresql:
-      host: postgres.foxhunt.svc.cluster.local
-      port: 5432
-      database: argo_archive
-      tableName: argo_workflows
-      userNameSecret:
-        name: argo-postgres-config
-        key: username
-      passwordSecret:
-        name: argo-postgres-config
-        key: password
-      ssl: false
-      sslMode: disable
-
-server:
-  nodeSelector:
-    k8s.scaleway.com/pool-name: platform
-  resources:
-    requests:
-      cpu: 25m
-      memory: 64Mi
-    limits:
-      cpu: 100m
-      memory: 128Mi
-  ingress:
-    enabled: false
-  extraArgs:
-    - --auth-mode=server
-    - --secure=false
-
-# Default artifact repository — MinIO (in-cluster)
-artifactRepository:
-  archiveLogs: true
-  s3:
-    endpoint: minio.foxhunt.svc.cluster.local:9000
-    insecure: true
-    bucket: argo-logs
-    accessKeySecret:
-      name: minio-credentials
-      key: access-key
-    secretKeySecret:
-      name: minio-credentials
-      key: secret-key
-
-executor:
-  resources:
-    requests:
-      cpu: 10m
-      memory: 32Mi
-    limits:
-      cpu: 100m
-      memory: 64Mi
-
-workflow:
-  serviceAccount:
-    create: true
-    name: argo-workflow
-
-crds:
-  install: true

infra/k8s/cert-manager/fxhnt-acme.yaml

@@ -1,72 +0,0 @@
-# ACME (Let's Encrypt) wildcard cert for *.fxhnt.ai via cert-manager + Scaleway DNS-01.
-# Replaces the prior MANUAL, unrenewed LE cert that expired 2026-05-26 (the tailscale-proxy
-# fell back to the GitLab self-signed cert). DNS-01 is required: the services are tailnet-only
-# (*.fxhnt.ai -> 100.x Tailscale CGNAT) so HTTP-01 is unreachable, and wildcards need DNS-01.
-#
-# Solver creds: cert-manager/scaleway-dns-credentials (SCW_ACCESS_KEY/SCW_SECRET_KEY, copied from
-# foxhunt/scaleway-credentials). Webhook: scaleway-certmanager-webhook (helm, cert-manager ns).
-# The issued secret gitlab-tls-cert is mounted by infra/k8s/gitlab/tailscale-proxy.yaml.
----
-apiVersion: cert-manager.io/v1
-kind: ClusterIssuer
-metadata:
-  name: letsencrypt-staging
-spec:
-  acme:
-    email: jeroen@grusewski.nl
-    server: https://acme-staging-v02.api.letsencrypt.org/directory
-    privateKeySecretRef:
-      name: letsencrypt-staging-account-key
-    solvers:
-      - dns01:
-          webhook:
-            groupName: acme.scaleway.com
-            solverName: scaleway
-            config:
-              accessKeySecretRef:
-                name: scaleway-dns-credentials
-                key: SCW_ACCESS_KEY
-              secretKeySecretRef:
-                name: scaleway-dns-credentials
-                key: SCW_SECRET_KEY
----
-apiVersion: cert-manager.io/v1
-kind: ClusterIssuer
-metadata:
-  name: letsencrypt-prod
-spec:
-  acme:
-    email: jeroen@grusewski.nl
-    server: https://acme-v02.api.letsencrypt.org/directory
-    privateKeySecretRef:
-      name: letsencrypt-prod-account-key
-    solvers:
-      - dns01:
-          webhook:
-            groupName: acme.scaleway.com
-            solverName: scaleway
-            config:
-              accessKeySecretRef:
-                name: scaleway-dns-credentials
-                key: SCW_ACCESS_KEY
-              secretKeySecretRef:
-                name: scaleway-dns-credentials
-                key: SCW_SECRET_KEY
----
-# Wildcard cert for the tailscale-proxy. issuerRef flips staging->prod after staging validates;
-# secretName flips to gitlab-tls-cert (the secret the proxy mounts) for the prod issuance.
-apiVersion: cert-manager.io/v1
-kind: Certificate
-metadata:
-  name: fxhnt-wildcard
-  namespace: foxhunt
-spec:
-  secretName: gitlab-tls-cert          # takes over the secret the tailscale-proxy mounts
-  privateKey:
-    rotationPolicy: Always             # old manual key had a mismatching algorithm; regenerate on issue/renew
-  dnsNames:
-    - fxhnt.ai
-    - "*.fxhnt.ai"
-  issuerRef:
-    name: letsencrypt-prod
-    kind: ClusterIssuer

infra/k8s/databases/postgres-backup.yaml

@@ -1,249 +0,0 @@
-# Postgres backup — daily pg_dump to MinIO S3
-# Uses initContainer (postgres:16-alpine) for pg_dump, then main container
-# (python:3.12-alpine, already cached) uploads to MinIO via S3 API.
-# Retains 7 daily + 4 weekly backups.
-# Apply: kubectl apply -f postgres-backup.yaml
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: postgres-backup-script
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres-backup
-    app.kubernetes.io/part-of: foxhunt
-data:
-  upload.py: |
-    #!/usr/bin/env python3
-    """Upload pg_dump backup to MinIO, prune old backups."""
-    import hashlib, hmac, os, sys, glob, urllib.request, urllib.error
-    from datetime import datetime, timezone
-
-    ENDPOINT = os.environ["MINIO_ENDPOINT"]  # http://minio.foxhunt...:9000
-    ACCESS_KEY = os.environ["MINIO_ACCESS_KEY"]
-    SECRET_KEY = os.environ["MINIO_SECRET_KEY"]
-    BUCKET = "foxhunt-backups"
-    BACKUP_DIR = "/backup"
-
-    def s3_sign(method, path, headers, payload_hash):
-        """AWS Signature V4 for MinIO."""
-        now = datetime.now(timezone.utc)
-        date_stamp = now.strftime("%Y%m%d")
-        amz_date = now.strftime("%Y%m%dT%H%M%SZ")
-        region = "us-east-1"
-        service = "s3"
-        scope = f"{date_stamp}/{region}/{service}/aws4_request"
-
-        headers["x-amz-date"] = amz_date
-        headers["x-amz-content-sha256"] = payload_hash
-
-        signed_headers = ";".join(sorted(headers.keys()))
-        canonical_headers = "".join(f"{k}:{headers[k]}\n" for k in sorted(headers.keys()))
-        canonical_request = f"{method}\n{path}\n\n{canonical_headers}\n{signed_headers}\n{payload_hash}"
-
-        string_to_sign = f"AWS4-HMAC-SHA256\n{amz_date}\n{scope}\n{hashlib.sha256(canonical_request.encode()).hexdigest()}"
-
-        def sign(key, msg):
-            return hmac.new(key, msg.encode(), hashlib.sha256).digest()
-
-        signing_key = sign(sign(sign(sign(
-            f"AWS4{SECRET_KEY}".encode(), date_stamp), region), service), "aws4_request")
-        signature = hmac.new(signing_key, string_to_sign.encode(), hashlib.sha256).hexdigest()
-
-        headers["Authorization"] = (
-            f"AWS4-HMAC-SHA256 Credential={ACCESS_KEY}/{scope}, "
-            f"SignedHeaders={signed_headers}, Signature={signature}"
-        )
-        return headers
-
-    def s3_put(key, data):
-        path = f"/{BUCKET}/{key}"
-        url = f"{ENDPOINT}{path}"
-        payload_hash = hashlib.sha256(data).hexdigest()
-        host = ENDPOINT.split("//")[1]
-        headers = {"host": host, "content-length": str(len(data))}
-        headers = s3_sign("PUT", path, headers, payload_hash)
-        req = urllib.request.Request(url, data=data, method="PUT", headers=headers)
-        urllib.request.urlopen(req, timeout=120)
-
-    def s3_list(prefix):
-        path = f"/{BUCKET}?prefix={prefix}&list-type=2"
-        url = f"{ENDPOINT}{path}"
-        host = ENDPOINT.split("//")[1]
-        headers = {"host": host}
-        headers = s3_sign("GET", f"/{BUCKET}", headers, hashlib.sha256(b"").hexdigest())
-        req = urllib.request.Request(f"{url}", method="GET", headers=headers)
-        try:
-            resp = urllib.request.urlopen(req, timeout=30).read().decode()
-            import re
-            return sorted(re.findall(r"<Key>([^<]+)</Key>", resp))
-        except Exception:
-            return []
-
-    def s3_delete(key):
-        path = f"/{BUCKET}/{key}"
-        url = f"{ENDPOINT}{path}"
-        host = ENDPOINT.split("//")[1]
-        headers = {"host": host}
-        headers = s3_sign("DELETE", path, headers, hashlib.sha256(b"").hexdigest())
-        req = urllib.request.Request(url, method="DELETE", headers=headers)
-        urllib.request.urlopen(req, timeout=30)
-
-    def s3_ensure_bucket():
-        path = f"/{BUCKET}"
-        url = f"{ENDPOINT}{path}"
-        host = ENDPOINT.split("//")[1]
-        headers = {"host": host}
-        headers = s3_sign("PUT", path, headers, hashlib.sha256(b"").hexdigest())
-        req = urllib.request.Request(url, method="PUT", headers=headers)
-        try:
-            urllib.request.urlopen(req, timeout=30)
-        except urllib.error.HTTPError as e:
-            if e.code not in (409, 200):  # 409 = bucket exists
-                raise
-
-    # Find the backup file
-    files = glob.glob(f"{BACKUP_DIR}/*.sql.gz")
-    if not files:
-        print("ERROR: No backup file found in /backup/")
-        sys.exit(1)
-
-    backup_file = files[0]
-    filename = os.path.basename(backup_file)
-    size_mb = os.path.getsize(backup_file) / (1024 * 1024)
-    print(f"Uploading {filename} ({size_mb:.1f} MB)")
-
-    s3_ensure_bucket()
-
-    with open(backup_file, "rb") as f:
-        data = f.read()
-
-    # Upload daily
-    s3_put(f"daily/{filename}", data)
-    print(f"  Uploaded daily/{filename}")
-
-    # On Sundays, also upload weekly
-    if datetime.now(timezone.utc).isoweekday() == 7:
-        s3_put(f"weekly/{filename}", data)
-        print(f"  Uploaded weekly/{filename}")
-
-    # Prune old dailies (keep 7)
-    dailies = s3_list("daily/")
-    if len(dailies) > 7:
-        for old in dailies[:len(dailies) - 7]:
-            s3_delete(old)
-            print(f"  Pruned {old}")
-
-    # Prune old weeklies (keep 4)
-    weeklies = s3_list("weekly/")
-    if len(weeklies) > 4:
-        for old in weeklies[:len(weeklies) - 4]:
-            s3_delete(old)
-            print(f"  Pruned {old}")
-
-    print("=== Backup upload complete ===")
----
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: postgres-backup
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres-backup
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  schedule: "30 2 * * *"
-  successfulJobsHistoryLimit: 1
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    spec:
-      backoffLimit: 2
-      activeDeadlineSeconds: 600
-      template:
-        metadata:
-          labels:
-            app: postgres-backup
-            app.kubernetes.io/part-of: foxhunt
-        spec:
-          nodeSelector:
-            k8s.scaleway.com/pool-name: platform
-          restartPolicy: OnFailure
-          # Step 1: pg_dump in initContainer → shared /backup volume
-          initContainers:
-            - name: pg-dump
-              image: postgres:16-alpine
-              command:
-                - sh
-                - -c
-                - |
-                  TIMESTAMP=$(date -u +%Y%m%d-%H%M%S)
-                  echo "=== Postgres Backup ${TIMESTAMP} ==="
-                  PGPASSWORD="${POSTGRES_PASSWORD}" pg_dumpall \
-                    -h "${POSTGRES_HOST}" \
-                    -p 5432 \
-                    -U "${POSTGRES_USER}" \
-                    --clean --if-exists \
-                    | gzip > "/backup/foxhunt-${TIMESTAMP}.sql.gz"
-                  ls -lh /backup/
-                  echo "Dump complete"
-              env:
-                - name: POSTGRES_HOST
-                  value: "postgres.foxhunt.svc.cluster.local"
-                - name: POSTGRES_USER
-                  value: "foxhunt"
-                - name: POSTGRES_PASSWORD
-                  valueFrom:
-                    secretKeyRef:
-                      name: db-credentials
-                      key: password
-              volumeMounts:
-                - name: backup
-                  mountPath: /backup
-              resources:
-                requests:
-                  cpu: 50m
-                  memory: 128Mi
-                limits:
-                  cpu: 500m
-                  memory: 512Mi
-          # Step 2: Upload to MinIO in main container (python cached on nodes)
-          containers:
-            - name: upload
-              image: python:3.12-alpine
-              command: ["python3", "-u", "/scripts/upload.py"]
-              env:
-                - name: MINIO_ENDPOINT
-                  value: "http://minio.foxhunt.svc.cluster.local:9000"
-                - name: MINIO_ACCESS_KEY
-                  valueFrom:
-                    secretKeyRef:
-                      name: minio-credentials
-                      key: access-key
-                - name: MINIO_SECRET_KEY
-                  valueFrom:
-                    secretKeyRef:
-                      name: minio-credentials
-                      key: secret-key
-              volumeMounts:
-                - name: backup
-                  mountPath: /backup
-                - name: script
-                  mountPath: /scripts
-              resources:
-                requests:
-                  cpu: 10m
-                  memory: 64Mi
-                limits:
-                  cpu: 100m
-                  memory: 128Mi
-          volumes:
-            - name: backup
-              emptyDir:
-                sizeLimit: 2Gi
-            - name: script
-              configMap:
-                name: postgres-backup-script
-                items:
-                  - key: upload.py
-                    path: upload.py
-                defaultMode: 0755

infra/k8s/databases/postgres.yaml

@@ -1,96 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: postgres-pvc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 20Gi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: postgres
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: postgres
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: postgres
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: infra
-      containers:
-        - name: postgres
-          image: timescale/timescaledb:latest-pg16
-          ports:
-            - containerPort: 5432
-              name: postgres
-          args:
-            - -c
-            - max_connections=100
-          env:
-            - name: POSTGRES_DB
-              value: foxhunt
-            - name: POSTGRES_USER
-              value: foxhunt
-            - name: POSTGRES_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-          volumeMounts:
-            - name: postgres-data
-              mountPath: /var/lib/postgresql/data
-              subPath: pgdata
-          readinessProbe:
-            exec:
-              command:
-                - pg_isready
-                - -U
-                - foxhunt
-            initialDelaySeconds: 10
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              cpu: 500m
-              memory: 1Gi
-      volumes:
-        - name: postgres-data
-          persistentVolumeClaim:
-            claimName: postgres-pvc
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: postgres
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: postgres
-  ports:
-    - port: 5432
-      targetPort: 5432
-      name: postgres

infra/k8s/databases/questdb.yaml

@@ -1,95 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: questdb-pvc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: questdb
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 10Gi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: questdb
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: questdb
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: questdb
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: questdb
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      containers:
-        - name: questdb
-          image: questdb/questdb:8.2.3
-          ports:
-            - containerPort: 9009
-              name: ilp
-            - containerPort: 8812
-              name: pg
-            - containerPort: 9003
-              name: metrics
-          env:
-            - name: QDB_PG_ENABLED
-              value: "true"
-            - name: QDB_METRICS_ENABLED
-              value: "TRUE"
-          volumeMounts:
-            - name: questdb-data
-              mountPath: /var/lib/questdb
-          readinessProbe:
-            httpGet:
-              path: /exec?query=SELECT%201
-              port: 9003
-            initialDelaySeconds: 10
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              cpu: 500m
-              memory: 1Gi
-      volumes:
-        - name: questdb-data
-          persistentVolumeClaim:
-            claimName: questdb-pvc
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: questdb
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: questdb
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: questdb
-  ports:
-    - port: 9009
-      targetPort: 9009
-      name: ilp
-    - port: 8812
-      targetPort: 8812
-      name: pg
-    - port: 9003
-      targetPort: 9003
-      name: metrics

infra/k8s/databases/redis.yaml

@@ -1,63 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: redis
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: redis
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: redis
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: redis
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: infra
-      containers:
-        - name: redis
-          image: redis:7-alpine
-          command:
-            - redis-server
-            - --maxmemory
-            - 512mb
-            - --maxmemory-policy
-            - allkeys-lru
-          ports:
-            - containerPort: 6379
-              name: redis
-          readinessProbe:
-            exec:
-              command:
-                - redis-cli
-                - ping
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              cpu: 250m
-              memory: 640Mi
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: redis
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: redis
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: redis
-  ports:
-    - port: 6379
-      targetPort: 6379
-      name: redis

infra/k8s/dns/node-local-dns.yaml

@@ -1,182 +0,0 @@
-# Node Bootstrap DaemonSet for Kapsule
-#
-# Two problems on Kapsule nodes:
-# 1. Nodes have nameserver 127.0.53.53 which can't resolve .svc.cluster.local
-# 2. Containerd defaults to HTTPS for non-localhost registries
-#
-# Fixes:
-# 1. Point /etc/resolv.conf to kube-dns ClusterIP (10.32.0.10)
-# 2. Create /etc/containerd/certs.d/ config for HTTP GitLab registry
-# 3. On GPU nodes (containerd v3 with conf.d/), add registry drop-in
-#
-# Init container sets up containerd registry config (one-time).
-# Main container manages resolv.conf (watches for Kapsule resets).
----
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: node-bootstrap
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: node-bootstrap
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: node-bootstrap
-  updateStrategy:
-    type: RollingUpdate
-    rollingUpdate:
-      maxUnavailable: 1
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: node-bootstrap
-    spec:
-      hostNetwork: true
-      dnsPolicy: Default
-      priorityClassName: system-node-critical
-      tolerations:
-        - operator: Exists
-      hostPID: true
-      initContainers:
-        # Configure containerd to use HTTP for the GitLab registry.
-        # On GPU nodes (NVIDIA overlay), also creates a v3-compatible conf.d drop-in
-        # and restarts containerd so it picks up the new config.
-        - name: setup-registry
-          image: alpine:3.19
-          securityContext:
-            privileged: true
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              NEEDS_RESTART=false
-
-              # 1. Create per-host containerd registry config (HTTP for GitLab)
-              REGISTRY_DIR="/host-etc/containerd/certs.d/gitlab-registry.foxhunt.svc.cluster.local:5000"
-              mkdir -p "$REGISTRY_DIR"
-              DESIRED='server = "http://gitlab-registry.foxhunt.svc.cluster.local:5000"
-
-              [host."http://gitlab-registry.foxhunt.svc.cluster.local:5000"]
-                capabilities = ["pull", "resolve"]'
-              DESIRED=$(printf '%s' "$DESIRED" | sed 's/^              //')
-              if [ ! -f "$REGISTRY_DIR/hosts.toml" ] || [ "$(cat "$REGISTRY_DIR/hosts.toml")" != "$DESIRED" ]; then
-                printf '%s\n' "$DESIRED" > "$REGISTRY_DIR/hosts.toml"
-                echo "Created hosts.toml for HTTP gitlab-registry"
-                NEEDS_RESTART=true
-              else
-                echo "hosts.toml already up to date"
-              fi
-
-              # 2. Ensure conf.d/ exists and always write the v3 registry drop-in.
-              #    containerd 2.x may ignore the v2-style grpc.v1.cri.registry.config_path
-              #    even when it's set in config.toml. The v1 images drop-in is the reliable path.
-              #    Create conf.d/ if missing (base config already has imports = ["conf.d/*.toml"]).
-              CONF_D="/host-etc/containerd/conf.d"
-              mkdir -p "$CONF_D"
-              if [ -d "$CONF_D" ]; then
-                DROPIN='version = 3
-
-              [plugins."io.containerd.cri.v1.images".registry]
-                config_path = "/etc/containerd/certs.d"'
-                DROPIN=$(printf '%s' "$DROPIN" | sed 's/^              //')
-                if [ ! -f "$CONF_D/50-registry.toml" ] || [ "$(cat "$CONF_D/50-registry.toml")" != "$DROPIN" ]; then
-                  printf '%s\n' "$DROPIN" > "$CONF_D/50-registry.toml"
-                  echo "GPU node: created v3 registry config drop-in"
-                  NEEDS_RESTART=true
-                else
-                  echo "GPU node: v3 drop-in already up to date"
-                fi
-              fi
-
-              # 3. Restart containerd only if we changed config files.
-              #    Uses /proc/1/root chroot to access host systemctl (requires hostPID + privileged).
-              #    On a new node this will kill the DaemonSet pod; kubelet recreates it,
-              #    second run finds files unchanged (NEEDS_RESTART=false), proceeds normally.
-              if [ "$NEEDS_RESTART" = "true" ]; then
-                echo "Restarting containerd to pick up config changes..."
-                chroot /proc/1/root systemctl restart containerd || \
-                  echo "WARNING: containerd restart failed (node may need manual restart)"
-                echo "containerd restart issued"
-              fi
-          resources:
-            requests:
-              cpu: 10m
-              memory: 8Mi
-            limits:
-              cpu: 50m
-              memory: 16Mi
-          volumeMounts:
-            - name: host-etc
-              mountPath: /host-etc
-      containers:
-        - name: resolv-manager
-          image: busybox:1.37
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              RESOLV="/host-etc/resolv.conf"
-              BACKUP="/host-etc/resolv.conf.pre-kube-dns"
-              KUBE_DNS="10.32.0.10"
-
-              restore() {
-                if [ -f "$BACKUP" ]; then
-                  cp "$BACKUP" "$RESOLV"
-                  echo "$(date): Restored original resolv.conf"
-                fi
-                exit 0
-              }
-              trap restore TERM INT
-
-              # Save original
-              if [ ! -f "$BACKUP" ]; then
-                cp "$RESOLV" "$BACKUP"
-                echo "Saved original: $(cat "$BACKUP")"
-              fi
-
-              apply() {
-                # CRITICAL: keep VPC upstream resolver as FALLBACK after kube-dns.
-                # If resolv.conf ONLY has 10.32.0.10, node DNS is dead when CoreDNS
-                # is down — containerd can't pull images, CoreDNS can't start,
-                # entire cluster deadlocks (incident 2026-03-18, 2026-04-12).
-                # Scaleway VPC gateway pushes 172.16.0.13 as DNS via DHCP.
-                UPSTREAM=""
-                if [ -f "$BACKUP" ]; then
-                  UPSTREAM=$(grep "^nameserver" "$BACKUP" | grep -v "127.0.53.53\|$KUBE_DNS" | head -1 | awk '{print $2}')
-                fi
-                if [ -z "$UPSTREAM" ]; then
-                  UPSTREAM="169.254.169.254"  # Scaleway metadata DNS fallback
-                fi
-                printf 'nameserver %s\nnameserver %s\nsearch foxhunt.svc.cluster.local svc.cluster.local cluster.local\noptions ndots:5 timeout:2 attempts:3\n' "$KUBE_DNS" "$UPSTREAM" > "$RESOLV"
-                echo "$(date): resolv.conf → $KUBE_DNS + $UPSTREAM (fallback)"
-              }
-
-              # Initial apply
-              apply
-
-              # Watch for resets
-              while true; do
-                sleep 30 &
-                wait $!
-                if ! grep -q "$KUBE_DNS" "$RESOLV" 2>/dev/null; then
-                  echo "$(date): resolv.conf was reset, re-applying"
-                  apply
-                fi
-              done
-          resources:
-            requests:
-              cpu: 5m
-              memory: 8Mi
-            limits:
-              cpu: 10m
-              memory: 16Mi
-          volumeMounts:
-            - name: host-etc
-              mountPath: /host-etc
-      terminationGracePeriodSeconds: 10
-      volumes:
-        - name: host-etc
-          hostPath:
-            path: /etc
-            type: Directory

infra/k8s/gitlab/grafana-values.yaml

@@ -1,120 +0,0 @@
-# Grafana — Foxhunt monitoring dashboard
-# Accessed via Tailscale proxy on port 3000
-
-replicas: 1
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: platform
-
-resources:
-  requests:
-    cpu: 50m
-    memory: 128Mi
-  limits:
-    cpu: 200m
-    memory: 256Mi
-
-persistence:
-  enabled: false
-
-admin:
-  existingSecret: grafana-admin
-  userKey: admin-user
-  passwordKey: admin-password
-
-envFromSecrets:
-  - name: db-credentials
-
-grafana.ini:
-  server:
-    root_url: https://grafana.fxhnt.ai
-  unified_alerting:
-    enabled: true
-  alerting:
-    enabled: false
-  database:
-    type: postgres
-    host: postgres:5432
-    name: grafana
-    user: foxhunt
-    password: $__env{password}
-    ssl_mode: disable
-
-datasources:
-  datasources.yaml:
-    apiVersion: 1
-    datasources:
-      - name: Prometheus
-        type: prometheus
-        url: http://prometheus-stack-kube-prom-prometheus.foxhunt.svc.cluster.local:9090
-        access: proxy
-        isDefault: true
-      - name: Alertmanager
-        type: alertmanager
-        uid: alertmanager
-        access: proxy
-        url: http://prometheus-stack-kube-prom-alertmanager.foxhunt.svc.cluster.local:9093
-        isDefault: false
-        jsonData:
-          implementation: prometheus
-          handleGrafanaManagedAlerts: true
-      - name: Loki
-        type: loki
-        access: proxy
-        url: http://loki.foxhunt.svc.cluster.local:3100
-        isDefault: false
-        jsonData:
-          derivedFields:
-            - datasourceUid: tempo
-              matcherRegex: '"trace_id":"(\w+)"'
-              name: TraceID
-              url: "$${__value.raw}"
-      - name: Tempo
-        type: tempo
-        uid: tempo
-        access: proxy
-        url: http://tempo.foxhunt.svc.cluster.local:3200
-        isDefault: false
-        jsonData:
-          tracesToLogsV2:
-            datasourceUid: loki
-            filterByTraceID: true
-          nodeGraph:
-            enabled: true
-          serviceMap:
-            datasourceUid: prometheus
-
-sidecar:
-  dashboards:
-    enabled: true
-    folder: /tmp/dashboards
-    folderAnnotation: grafana_folder
-    label: grafana_dashboard
-    labelValue: "1"
-    searchNamespace: foxhunt
-    provider:
-      foldersFromFilesStructure: true
-  datasources:
-    enabled: false
-    label: grafana_datasource
-    labelValue: ""
-
-dashboardProviders:
-  dashboardproviders.yaml:
-    apiVersion: 1
-    providers:
-      - name: default
-        orgId: 1
-        folder: Foxhunt
-        type: file
-        disableDeletion: false
-        editable: true
-        options:
-          path: /var/lib/grafana/dashboards/default
-
-dashboards:
-  default:
-    node-exporter:
-      gnetId: 1860
-      revision: 37
-      datasource: Prometheus

infra/k8s/gitlab/pat-rotation.yaml

@@ -1,253 +0,0 @@
-# GitLab PAT auto-rotation CronJob
-# Rotates the personal access token monthly, storing the new token in a K8s Secret.
-# Uses GitLab's atomic rotate API: creates new token + revokes old in one call.
-#
-# The rotation runs on the 1st of every month. Additionally, a weekly expiry check
-# runs every Monday and logs a warning if the token expires within 14 days, giving
-# time to investigate rotation failures before CI breaks.
-#
-# Initial setup:
-#   kubectl -n foxhunt create secret generic gitlab-pat \
-#     --from-literal=token=glpat-<initial-token>
-#
-# Manual rotation:
-#   kubectl -n foxhunt create job pat-rotate-manual --from=cronjob/gitlab-pat-rotation
-#
-# Manual expiry check:
-#   kubectl -n foxhunt create job pat-check-manual --from=cronjob/gitlab-pat-expiry-check
-#
-# Sync local token:
-#   export GITLAB_TOKEN=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: gitlab-pat-rotator
-  namespace: foxhunt
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: gitlab-pat-rotator
-  namespace: foxhunt
-rules:
-  - apiGroups: [""]
-    resources: ["secrets"]
-    resourceNames: ["gitlab-pat"]
-    verbs: ["get", "patch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: gitlab-pat-rotator
-  namespace: foxhunt
-subjects:
-  - kind: ServiceAccount
-    name: gitlab-pat-rotator
-    namespace: foxhunt
-roleRef:
-  kind: Role
-  name: gitlab-pat-rotator
-  apiGroup: rbac.authorization.k8s.io
----
-# Monthly rotation — creates new token, revokes old, updates K8s secret
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: gitlab-pat-rotation
-  namespace: foxhunt
-spec:
-  schedule: "0 3 1 * *"  # 1st of every month at 03:00 UTC
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 5
-  concurrencyPolicy: Forbid
-  jobTemplate:
-    spec:
-      backoffLimit: 3
-      activeDeadlineSeconds: 180
-      template:
-        metadata:
-          labels:
-            app: gitlab-pat-rotation
-        spec:
-          serviceAccountName: gitlab-pat-rotator
-          restartPolicy: Never
-          nodeSelector:
-            k8s.scaleway.com/pool-name: platform
-          containers:
-            - name: rotate
-              image: alpine:3.21
-              command: ["/bin/sh", "-c"]
-              args:
-                - |
-                  set -e
-
-                  apk add --no-cache -q curl jq
-
-                  CURRENT_PAT=$(cat /etc/gitlab-pat/token)
-                  GITLAB_URL="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-
-                  # 1. Verify current token is valid
-                  TOKEN_INFO=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
-                    -H "PRIVATE-TOKEN: ${CURRENT_PAT}") || {
-                    echo "CRITICAL: Current PAT is invalid or GitLab unreachable."
-                    echo "Manual recovery required: create a new PAT in GitLab UI and run:"
-                    echo "  kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<new-pat> --dry-run=client -o yaml | kubectl apply -f -"
-                    exit 1
-                  }
-                  OLD_EXPIRY=$(echo "$TOKEN_INFO" | jq -r '.expires_at')
-                  TOKEN_NAME=$(echo "$TOKEN_INFO" | jq -r '.name')
-                  echo "Current PAT '${TOKEN_NAME}' expires: ${OLD_EXPIRY}"
-
-                  # 2. Rotate: new expiry = 1 year from now
-                  EXPIRES=$(date -d "@$(($(date +%s) + 31536000))" +%Y-%m-%d)
-                  RESPONSE=$(curl -sf -X POST \
-                    "${GITLAB_URL}/api/v4/personal_access_tokens/self/rotate" \
-                    -H "PRIVATE-TOKEN: ${CURRENT_PAT}" \
-                    -d "expires_at=${EXPIRES}") || {
-                    echo "ERROR: Rotation API call failed. Token may still be valid."
-                    exit 1
-                  }
-
-                  NEW_PAT=$(echo "$RESPONSE" | jq -r '.token')
-                  if [ "$NEW_PAT" = "null" ] || [ -z "$NEW_PAT" ]; then
-                    echo "ERROR: No token in rotation response: $RESPONSE"
-                    exit 1
-                  fi
-
-                  echo "PAT rotated successfully. New expiry: ${EXPIRES}"
-
-                  # 3. Update K8s secret (old token is already revoked at this point)
-                  K8S_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
-                  K8S_CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-                  K8S_NS=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
-                  K8S_API="https://kubernetes.default.svc"
-
-                  NEW_PAT_B64=$(printf '%s' "$NEW_PAT" | base64)
-
-                  for attempt in 1 2 3 4 5; do
-                    if curl -sf -X PATCH \
-                      "${K8S_API}/api/v1/namespaces/${K8S_NS}/secrets/gitlab-pat" \
-                      --cacert "$K8S_CA" \
-                      -H "Authorization: Bearer ${K8S_TOKEN}" \
-                      -H "Content-Type: application/strategic-merge-patch+json" \
-                      -d "{\"data\":{\"token\":\"${NEW_PAT_B64}\"}}"; then
-                      echo ""
-                      echo "K8s secret updated on attempt ${attempt}."
-
-                      # 4. Verify the new token works
-                      VERIFY=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
-                        -H "PRIVATE-TOKEN: ${NEW_PAT}" | jq -r '.expires_at') || true
-                      if [ "$VERIFY" = "$EXPIRES" ]; then
-                        echo "Verification passed: new token valid until ${VERIFY}"
-                      else
-                        echo "WARNING: Verification returned unexpected expiry: ${VERIFY}"
-                      fi
-                      exit 0
-                    fi
-                    echo "Secret update attempt ${attempt} failed, retrying in 3s..."
-                    sleep 3
-                  done
-
-                  # If we get here, rotation succeeded but secret update failed.
-                  # Print token so it can be recovered from job logs.
-                  echo "CRITICAL: Secret update failed after 5 attempts!"
-                  echo "New PAT (recover manually): ${NEW_PAT}"
-                  echo "Run: kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<above> --dry-run=client -o yaml | kubectl apply -f -"
-                  exit 1
-              volumeMounts:
-                - name: gitlab-pat
-                  mountPath: /etc/gitlab-pat
-                  readOnly: true
-              resources:
-                requests:
-                  cpu: 50m
-                  memory: 32Mi
-                limits:
-                  cpu: 200m
-                  memory: 64Mi
-          volumes:
-            - name: gitlab-pat
-              secret:
-                secretName: gitlab-pat
----
-# Weekly expiry check — warns if token expires within 14 days
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: gitlab-pat-expiry-check
-  namespace: foxhunt
-spec:
-  schedule: "0 8 * * 1"  # Every Monday at 08:00 UTC
-  successfulJobsHistoryLimit: 1
-  failedJobsHistoryLimit: 3
-  concurrencyPolicy: Forbid
-  jobTemplate:
-    spec:
-      backoffLimit: 1
-      activeDeadlineSeconds: 60
-      template:
-        metadata:
-          labels:
-            app: gitlab-pat-expiry-check
-        spec:
-          restartPolicy: Never
-          nodeSelector:
-            k8s.scaleway.com/pool-name: platform
-          containers:
-            - name: check
-              image: alpine:3.21
-              command: ["/bin/sh", "-c"]
-              args:
-                - |
-                  set -e
-
-                  apk add --no-cache -q curl jq
-
-                  CURRENT_PAT=$(cat /etc/gitlab-pat/token)
-                  GITLAB_URL="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-
-                  TOKEN_INFO=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
-                    -H "PRIVATE-TOKEN: ${CURRENT_PAT}") || {
-                    echo "CRITICAL: PAT is INVALID. CI will fail!"
-                    echo "Create a new PAT in GitLab UI and run:"
-                    echo "  kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<new-pat> --dry-run=client -o yaml | kubectl apply -f -"
-                    exit 1
-                  }
-
-                  EXPIRY=$(echo "$TOKEN_INFO" | jq -r '.expires_at')
-                  TOKEN_NAME=$(echo "$TOKEN_INFO" | jq -r '.name')
-                  SCOPES=$(echo "$TOKEN_INFO" | jq -r '.scopes | join(",")')
-
-                  # Calculate days until expiry
-                  EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
-                  NOW_EPOCH=$(date +%s)
-                  DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
-
-                  echo "PAT '${TOKEN_NAME}': expires=${EXPIRY} scopes=${SCOPES} days_left=${DAYS_LEFT}"
-
-                  if [ "$DAYS_LEFT" -lt 0 ]; then
-                    echo "CRITICAL: PAT has EXPIRED ${DAYS_LEFT} days ago!"
-                    exit 1
-                  elif [ "$DAYS_LEFT" -lt 14 ]; then
-                    echo "WARNING: PAT expires in ${DAYS_LEFT} days. Rotation may have failed."
-                    echo "Manual rotation: kubectl -n foxhunt create job pat-rotate-manual --from=cronjob/gitlab-pat-rotation"
-                    exit 1
-                  else
-                    echo "OK: PAT valid for ${DAYS_LEFT} more days."
-                  fi
-              volumeMounts:
-                - name: gitlab-pat
-                  mountPath: /etc/gitlab-pat
-                  readOnly: true
-              resources:
-                requests:
-                  cpu: 10m
-                  memory: 32Mi
-                limits:
-                  cpu: 100m
-                  memory: 64Mi
-          volumes:
-            - name: gitlab-pat
-              secret:
-                secretName: gitlab-pat

infra/k8s/gitlab/postgres-init.yaml

@@ -1,63 +0,0 @@
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: gitlab-postgres-init
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gitlab-postgres-init
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  ttlSecondsAfterFinished: 300
-  backoffLimit: 3
-  template:
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: infra
-      restartPolicy: Never
-      containers:
-        - name: init
-          image: timescale/timescaledb:latest-pg16
-          command:
-            - bash
-            - -c
-            - |
-              set -euo pipefail
-              export PGPASSWORD="$POSTGRES_PASSWORD"
-
-              until pg_isready -h postgres -U foxhunt; do
-                echo "Waiting for postgres..."
-                sleep 2
-              done
-
-              echo "Creating gitlab user..."
-              psql -h postgres -U foxhunt -d foxhunt -tc \
-                "SELECT 1 FROM pg_roles WHERE rolname='gitlab'" | grep -q 1 || \
-                psql -h postgres -U foxhunt -d foxhunt -c \
-                  "CREATE ROLE gitlab WITH LOGIN PASSWORD '${GITLAB_DB_PASSWORD}'"
-
-              echo "Creating gitlab database..."
-              psql -h postgres -U foxhunt -tc \
-                "SELECT 1 FROM pg_database WHERE datname='gitlab'" | grep -q 1 || \
-                psql -h postgres -U foxhunt -c \
-                  "CREATE DATABASE gitlab OWNER gitlab"
-
-              echo "Creating extensions..."
-              psql -h postgres -U foxhunt -d gitlab -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
-              psql -h postgres -U foxhunt -d gitlab -c "CREATE EXTENSION IF NOT EXISTS btree_gist;"
-
-              echo "Granting permissions..."
-              psql -h postgres -U foxhunt -d gitlab -c "GRANT ALL PRIVILEGES ON DATABASE gitlab TO gitlab;"
-              psql -h postgres -U foxhunt -d gitlab -c "GRANT ALL ON SCHEMA public TO gitlab;"
-
-              echo "GitLab database initialized successfully."
-          env:
-            - name: POSTGRES_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: GITLAB_DB_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: gitlab-secrets
-                  key: db-password

infra/k8s/gitlab/runner-h100-values.yaml

@@ -1,73 +0,0 @@
-# GitLab Runner — H100 GPU training workloads (hyperopt, walk-forward)
-# Runs on ci-training-h100 pool (H100 PCIe 1x80GB)
-# SXM pools have zero Scaleway quota — use PCIe until quota is granted.
-
-gitlabUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-# runnerToken set via --set at install time
-
-replicas: 1
-
-# Reuse the existing gitlab-runner SA (has pods/secrets/configmaps RBAC)
-rbac:
-  create: false
-serviceAccount:
-  create: false
-  name: gitlab-runner
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: infra
-
-runners:
-  cloneUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-  config: |
-    [[runners]]
-      clone_url = "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-      tag_list = ["kapsule", "h100"]
-      [runners.kubernetes]
-        namespace = "foxhunt"
-        service_account = "gitlab-runner"
-        image = "rust:1.89-slim"
-        privileged = false
-        node_selector_overwrite_allowed = ".*"
-        cpu_request_overwrite_max_allowed = "24000m"
-        cpu_limit_overwrite_max_allowed = "24000m"
-        memory_request_overwrite_max_allowed = "200Gi"
-        memory_limit_overwrite_max_allowed = "200Gi"
-        poll_timeout = 600
-        runtime_class_name = "nvidia"
-        pod_annotations_overwrite_allowed = ".*"
-        # Default resources for H100 training
-        cpu_request = "2000m"
-        cpu_limit = "3800m"
-        memory_request = "4Gi"
-        memory_limit = "8Gi"
-        helper_cpu_request = "100m"
-        helper_cpu_limit = "500m"
-        helper_memory_request = "128Mi"
-        helper_memory_limit = "512Mi"
-        image_pull_secrets = ["gitlab-registry"]
-      [runners.kubernetes.node_selector]
-        "k8s.scaleway.com/pool-name" = "ci-training-h100"
-      [runners.kubernetes.node_tolerations]
-        "nvidia.com/gpu" = "NoSchedule"
-        "node.cilium.io/agent-not-ready" = "NoSchedule"
-      [runners.kubernetes.pod_labels]
-        "app.kubernetes.io/part-of" = "foxhunt-ci"
-      # Request GPU via K8s scheduler so only one training pod runs per GPU
-      [[runners.kubernetes.pod_spec]]
-        name = "build"
-        patch_type = "strategic"
-        patch = '{"containers":[{"name":"build","resources":{"requests":{"nvidia.com/gpu":"1"},"limits":{"nvidia.com/gpu":"1"}}}]}'
-      # H100 PCIe PVCs (separate from L40S to avoid RWO conflicts)
-      [[runners.kubernetes.volumes.pvc]]
-        name = "training-data-h100-pvc"
-        mount_path = "/mnt/training-data"
-        read_only = true
-      [[runners.kubernetes.volumes.pvc]]
-        name = "sccache-h100-pvc"
-        mount_path = "/mnt/sccache"
-        read_only = false
-
-  tags: "kapsule,h100"
-
-concurrent: 2

infra/k8s/gitlab/runner-h100x2-values.yaml

@@ -1,69 +0,0 @@
-# GitLab Runner — 2×H100 GPU training workloads (multi-GPU hyperopt)
-# Runs on ci-training-h100x2 pool (2× H100 PCIe: 160GB total VRAM)
-# Multi-GPU auto-detected by MultiGpuConfig::detect() in ml crate.
-
-gitlabUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-
-replicas: 1
-
-rbac:
-  create: false
-serviceAccount:
-  create: false
-  name: gitlab-runner
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: infra
-
-runners:
-  cloneUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-  config: |
-    [[runners]]
-      clone_url = "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-      tag_list = ["kapsule", "h100x2"]
-      [runners.kubernetes]
-        namespace = "foxhunt"
-        service_account = "gitlab-runner"
-        image = "rust:1.89-slim"
-        privileged = false
-        node_selector_overwrite_allowed = ".*"
-        cpu_request_overwrite_max_allowed = "16000m"
-        cpu_limit_overwrite_max_allowed = "16000m"
-        memory_request_overwrite_max_allowed = "192Gi"
-        memory_limit_overwrite_max_allowed = "192Gi"
-        poll_timeout = 600
-        runtime_class_name = "nvidia"
-        pod_annotations_overwrite_allowed = ".*"
-        cpu_request = "4000m"
-        cpu_limit = "8000m"
-        memory_request = "8Gi"
-        memory_limit = "16Gi"
-        helper_cpu_request = "100m"
-        helper_cpu_limit = "500m"
-        helper_memory_request = "128Mi"
-        helper_memory_limit = "512Mi"
-        image_pull_secrets = ["gitlab-registry"]
-      [runners.kubernetes.node_selector]
-        "k8s.scaleway.com/pool-name" = "ci-training-h100x2"
-      [runners.kubernetes.node_tolerations]
-        "nvidia.com/gpu" = "NoSchedule"
-        "node.cilium.io/agent-not-ready" = "NoSchedule"
-      [runners.kubernetes.pod_labels]
-        "app.kubernetes.io/part-of" = "foxhunt-ci"
-      # Request both GPUs via K8s scheduler for exclusive access
-      [[runners.kubernetes.pod_spec]]
-        name = "build"
-        patch_type = "strategic"
-        patch = '{"containers":[{"name":"build","resources":{"requests":{"nvidia.com/gpu":"2"},"limits":{"nvidia.com/gpu":"2"}}}]}'
-      [[runners.kubernetes.volumes.pvc]]
-        name = "training-data-h100x2-pvc"
-        mount_path = "/mnt/training-data"
-        read_only = true
-      [[runners.kubernetes.volumes.pvc]]
-        name = "sccache-h100x2-pvc"
-        mount_path = "/mnt/sccache"
-        read_only = false
-
-  tags: "kapsule,h100x2"
-
-concurrent: 2

infra/k8s/gitlab/runner-rl-values.yaml

@@ -1,76 +0,0 @@
-# GitLab Runner — RL training workloads (DQN, PPO)
-# Runs on ci-training pool (L40S: 48GB VRAM)
-# Mounts separate PVCs to avoid RWO conflicts with main runner
-
-gitlabUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-# runnerToken set via --set at install time
-
-replicas: 1
-
-# Reuse the existing gitlab-runner SA (has pods/secrets/configmaps RBAC)
-rbac:
-  create: false
-serviceAccount:
-  create: false
-  name: gitlab-runner
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: infra
-
-runners:
-  cloneUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-  config: |
-    [[runners]]
-      clone_url = "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-      tag_list = ["kapsule", "l40s"]
-      [runners.kubernetes]
-        namespace = "foxhunt"
-        service_account = "gitlab-runner"
-        image = "rust:1.89-slim"
-        privileged = false
-        node_selector_overwrite_allowed = ".*"
-        # Hyperopt RL jobs need 6000m CPU + 20Gi mem (PSO parallel trials)
-        cpu_request_overwrite_max_allowed = "8000m"
-        cpu_limit_overwrite_max_allowed = "8000m"
-        memory_request_overwrite_max_allowed = "48Gi"
-        memory_limit_overwrite_max_allowed = "48Gi"
-        poll_timeout = 600
-        # All RL runner jobs target GPU nodes → set nvidia runtime globally
-        runtime_class_name = "nvidia"
-        # Allow CI jobs to set pod annotations (training jobs expose Prometheus metrics)
-        pod_annotations_overwrite_allowed = ".*"
-        # Default resources for RL training on L40S
-        cpu_request = "2000m"
-        cpu_limit = "3800m"
-        memory_request = "4Gi"
-        memory_limit = "8Gi"
-        helper_cpu_request = "100m"
-        helper_cpu_limit = "500m"
-        helper_memory_request = "128Mi"
-        helper_memory_limit = "512Mi"
-        image_pull_secrets = ["gitlab-registry"]
-      [runners.kubernetes.node_selector]
-        "k8s.scaleway.com/pool-name" = "ci-training"
-      [runners.kubernetes.node_tolerations]
-        "nvidia.com/gpu" = "NoSchedule"
-        "node.cilium.io/agent-not-ready" = "NoSchedule"
-      [runners.kubernetes.pod_labels]
-        "app.kubernetes.io/part-of" = "foxhunt-ci"
-      # Request GPU via K8s scheduler so only one training pod runs per GPU
-      [[runners.kubernetes.pod_spec]]
-        name = "build"
-        patch_type = "strategic"
-        patch = '{"containers":[{"name":"build","resources":{"requests":{"nvidia.com/gpu":"1"},"limits":{"nvidia.com/gpu":"1"}}}]}'
-      # Training PVCs — separate from main runner to avoid RWO Multi-Attach errors
-      [[runners.kubernetes.volumes.pvc]]
-        name = "training-data-l4-pvc"
-        mount_path = "/mnt/training-data"
-        read_only = true
-      [[runners.kubernetes.volumes.pvc]]
-        name = "sccache-l4-pvc"
-        mount_path = "/mnt/sccache"
-        read_only = false
-
-  tags: "kapsule,l40s"
-
-concurrent: 2

infra/k8s/gitlab/runner-values.yaml

@@ -1,86 +0,0 @@
-# GitLab Runner — Kubernetes executor
-# Runner manager pod lives on gitlab node pool
-# Default: build pods spawn on ci-compile-cpu pool (POP2-32C-128G)
-# CI jobs override node_selector for their target pool
-
-gitlabUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-# runnerToken set via --set at install time
-
-replicas: 1
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: platform
-
-runners:
-  # Override clone URL to internal service (pods can't reach Tailscale IPs)
-  cloneUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-  config: |
-    [[runners]]
-      clone_url = "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-      tag_list = ["kapsule", "rust", "docker", "gpu"]
-      [runners.kubernetes]
-        namespace = "foxhunt"
-        image = "rust:1.89-slim"
-        privileged = false
-        # Allow CI jobs to override node selector via KUBERNETES_NODE_SELECTOR_* vars
-        # Format: KUBERNETES_NODE_SELECTOR_<LABEL>: "key=value"
-        node_selector_overwrite_allowed = ".*"
-        # Allow CI jobs to override resources via KUBERNETES_*_{REQUEST,LIMIT} vars
-        # Values are maximums (not regexes) — training can request up to 80Gi
-        cpu_request_overwrite_max_allowed = "24000m"
-        cpu_limit_overwrite_max_allowed = "24000m"
-        memory_request_overwrite_max_allowed = "80Gi"
-        memory_limit_overwrite_max_allowed = "80Gi"
-        # Scale-to-zero pools need ~3-5 min to provision; default 180s times out
-        poll_timeout = 600
-        # Allow CI jobs to override runtimeClassName via KUBERNETES_RUNTIME_CLASS_NAME
-        # GPU jobs set "nvidia"; compile/kaniko jobs leave unset (default runc)
-        runtime_class_name_overwrite_allowed = ".*"
-        # Allow CI jobs to set pod annotations (training jobs expose Prometheus metrics)
-        pod_annotations_overwrite_allowed = ".*"
-        # Default resource limits for ci-compile-cpu (POP2-32C-128G)
-        cpu_request = "3500m"
-        cpu_limit = "7800m"
-        memory_request = "12Gi"
-        memory_limit = "28Gi"
-        helper_cpu_request = "100m"
-        helper_cpu_limit = "500m"
-        helper_memory_request = "128Mi"
-        helper_memory_limit = "512Mi"
-        image_pull_secrets = ["gitlab-registry"]
-      # Sub-tables must come AFTER all scalar values (TOML rule)
-      [runners.kubernetes.node_selector]
-        "k8s.scaleway.com/pool-name" = "ci-compile-cpu"
-      [runners.kubernetes.node_tolerations]
-        "nvidia.com/gpu" = "NoSchedule"
-        # Cilium CNI agent takes ~30s to initialize on fresh scale-from-zero nodes
-        "node.cilium.io/agent-not-ready" = "NoSchedule"
-      [runners.kubernetes.pod_labels]
-        "app.kubernetes.io/part-of" = "foxhunt-ci"
-      # Mount training data PVC (Databento futures .dbn.zst files)
-      # Path must NOT be under /data — Redis image's WORKDIR is /data and entrypoint chowns it
-      [[runners.kubernetes.volumes.pvc]]
-        name = "training-data-pvc"
-        mount_path = "/mnt/training-data"
-        read_only = true
-
-  # Runner tags for job matching
-  tags: "kapsule,rust,docker,gpu"
-
-# Concurrency — 10 allows all 9 Kaniko builds + test to run in parallel
-concurrent: 10
-
-# RBAC for runner to spawn pods
-rbac:
-  create: true
-  rules:
-    - apiGroups: [""]
-      resources: ["pods", "pods/exec", "pods/log", "secrets", "configmaps"]
-      verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
-    - apiGroups: [""]
-      resources: ["pods/attach"]
-      verbs: ["create", "get"]
-
-serviceAccount:
-  create: true
-  name: gitlab-runner

infra/k8s/gitlab/tailscale-proxy.yaml

@@ -1,352 +0,0 @@
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: tailscale-gitlab
-  namespace: foxhunt
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: tailscale-gitlab
-  namespace: foxhunt
-rules:
-  - apiGroups: [""]
-    resources: ["secrets"]
-    verbs: ["create", "get", "update", "patch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: tailscale-gitlab
-  namespace: foxhunt
-subjects:
-  - kind: ServiceAccount
-    name: tailscale-gitlab
-    namespace: foxhunt
-roleRef:
-  kind: Role
-  name: tailscale-gitlab
-  apiGroup: rbac.authorization.k8s.io
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: tailscale-gitlab-proxy
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: tailscale-gitlab-proxy
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: tailscale-gitlab-proxy
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: tailscale-gitlab-proxy
-    spec:
-      serviceAccountName: tailscale-gitlab
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      containers:
-        - name: nginx
-          image: nginx:alpine
-          ports:
-            - containerPort: 80
-            - containerPort: 443
-            - containerPort: 5050
-          volumeMounts:
-            - name: nginx-conf
-              mountPath: /etc/nginx/conf.d
-            - name: tls-certs
-              mountPath: /etc/nginx/certs
-              readOnly: true
-          resources:
-            requests:
-              cpu: 25m
-              memory: 32Mi
-            limits:
-              cpu: 100m
-              memory: 64Mi
-        - name: ssh-proxy
-          image: alpine/socat:latest
-          args:
-            - "TCP-LISTEN:2222,fork,reuseaddr,sndbuf=1048576,rcvbuf=1048576"
-            - "TCP:gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222,sndbuf=1048576,rcvbuf=1048576"
-          ports:
-            - containerPort: 2222
-          resources:
-            requests:
-              cpu: 25m
-              memory: 64Mi
-            limits:
-              cpu: 200m
-              memory: 128Mi
-        - name: imap-proxy
-          image: alpine/socat:latest
-          args:
-            - "TCP-LISTEN:993,fork,reuseaddr"
-            - "TCP:stalwart.foxhunt.svc.cluster.local:993"
-          ports:
-            - containerPort: 993
-          resources:
-            requests:
-              cpu: 10m
-              memory: 16Mi
-            limits:
-              cpu: 50m
-              memory: 32Mi
-        - name: tailscale
-          image: ghcr.io/tailscale/tailscale:latest
-          env:
-            - name: TS_AUTHKEY
-              valueFrom:
-                secretKeyRef:
-                  name: tailscale-auth
-                  key: TS_AUTHKEY
-            - name: TS_KUBE_SECRET
-              value: tailscale-gitlab-state
-            - name: TS_USERSPACE
-              value: "false"
-            - name: TS_HOSTNAME
-              value: "foxhunt-gitlab"
-            - name: TS_ACCEPT_DNS
-              value: "false"
-            - name: TS_EXTRA_ARGS
-              value: ""
-          resources:
-            requests:
-              cpu: 25m
-              memory: 32Mi
-            limits:
-              cpu: 100m
-              memory: 64Mi
-          securityContext:
-            capabilities:
-              add:
-                - NET_ADMIN
-                - NET_RAW
-      volumes:
-        - name: nginx-conf
-          configMap:
-            name: tailscale-gitlab-nginx
-        - name: tls-certs
-          secret:
-            secretName: gitlab-tls-cert
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: tailscale-gitlab-nginx
-  namespace: foxhunt
-data:
-  default.conf: |
-    # HTTP → HTTPS redirect for all subdomains
-    server {
-        listen 80;
-        server_name *.fxhnt.ai;
-        return 301 https://$host$request_uri;
-    }
-
-    # GitLab — git.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name git.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        client_max_body_size 0;
-
-        location / {
-            proxy_pass http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-
-            proxy_buffering off;
-            proxy_request_buffering off;
-            proxy_http_version 1.1;
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-            proxy_connect_timeout 300s;
-        }
-    }
-
-    # Container Registry — git.fxhnt.ai:5050
-    server {
-        listen 5050 ssl;
-        server_name git.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        client_max_body_size 0;
-
-        location / {
-            proxy_pass http://gitlab-registry.foxhunt.svc.cluster.local:5000;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-
-            proxy_buffering off;
-            proxy_request_buffering off;
-            proxy_http_version 1.1;
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-        }
-    }
-
-    # Grafana — grafana.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name grafana.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        location / {
-            proxy_pass http://grafana.foxhunt.svc.cluster.local:80;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection "upgrade";
-        }
-    }
-
-    # fxhnt cockpit — dashboard.fxhnt.ai
-    # Proxies to the cockpit's Tailscale node (peer-to-peer over the tailnet) rather than the cluster Service:
-    # the pod CIDR 100.64.0.0/15 overlaps Tailscale CGNAT, so this kernel-mode proxy can't reach platform-pool
-    # pods via ClusterIP, but it CAN reach another tailnet node. IP is stable while the cockpit's TS state
-    # secret (fxhnt-dashboard-ts-state) persists. (Was web-dashboard.foxhunt.svc — replaced by the fxhnt cockpit.)
-    server {
-        listen 443 ssl;
-        server_name dashboard.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        location / {
-            proxy_pass http://100.81.150.18:80;   # fxhnt-dashboard tailnet node (TCP:80 -> cockpit :8080)
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection "upgrade";
-        }
-    }
-
-    # API Gateway (gRPC-Web + gRPC) — api.fxhnt.ai
-    # Browser dashboard uses gRPC-web (HTTP/1.1 POST), so proxy_pass not grpc_pass
-    server {
-        listen 443 ssl;
-        http2 on;
-        server_name api.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        location / {
-            proxy_pass http://api.foxhunt.svc.cluster.local:50051;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-            proxy_connect_timeout 30s;
-        }
-    }
-
-    # Stalwart Admin — mail.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name mail.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        location / {
-            proxy_pass https://stalwart.foxhunt.svc.cluster.local:443;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_ssl_verify off;
-        }
-    }
-
-    # Mattermost — chat.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name chat.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        client_max_body_size 50m;
-
-        location / {
-            proxy_pass http://mattermost.foxhunt.svc.cluster.local:8065;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection "upgrade";
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-        }
-    }
-
-    # MinIO S3 API — minio.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name minio.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        client_max_body_size 0;
-
-        location / {
-            proxy_pass http://minio.foxhunt.svc.cluster.local:9000;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-
-            proxy_buffering off;
-            proxy_request_buffering off;
-            proxy_http_version 1.1;
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-            proxy_connect_timeout 300s;
-
-            # MinIO uses self-signed TLS internally
-            proxy_ssl_verify off;
-        }
-    }
-

infra/k8s/gitlab/values.yaml

@@ -1,206 +0,0 @@
-# GitLab CE - Foxhunt deployment
-# Runs on 'infra' node pool alongside platform services
-# External Postgres + Redis on same pool
-
-global:
-  edition: ce
-  kas:
-    enabled: false
-  hosts:
-    domain: fxhnt.ai
-    gitlab:
-      name: git.fxhnt.ai
-      https: true
-    registry:
-      name: git.fxhnt.ai
-    https: true
-  ingress:
-    enabled: false
-    configureCertmanager: false
-  shell:
-    port: 2222
-  psql:
-    host: postgres.foxhunt.svc.cluster.local
-    port: 5432
-    database: gitlab
-    username: gitlab
-    password:
-      secret: gitlab-secrets
-      key: db-password
-  redis:
-    host: redis.foxhunt.svc.cluster.local
-    port: 6379
-    auth:
-      enabled: false
-  minio:
-    enabled: false
-  registry:
-    bucket: foxhunt-gitlab-registry
-  appConfig:
-    lfs:
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    artifacts:
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    uploads:
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    packages:
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    # Terraform/OpenTofu state object storage. MUST be present — without it the
-    # Terraform::StateUploader has no object store and every TF-state API call 403s
-    # ("Object Storage is not enabled for Terraform::StateUploader"). State files already
-    # live in foxhunt-gitlab-artifacts (6b/86/<sha256(project_id)>/<state>/<ver>.tfstate).
-    terraformState:
-      enabled: true
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    gitlab_kas:
-      enabled: false
-
-# Disable bundled databases — use external
-postgresql:
-  install: false
-redis:
-  install: false
-
-# Disable components we don't need
-installCertmanager: false
-certmanager:
-  installCRDs: false
-nginx-ingress:
-  enabled: false
-prometheus:
-  install: true
-  rbac:
-    create: true
-  alertmanager:
-    enabled: false
-  server:
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    persistentVolume:
-      enabled: false
-    resources:
-      requests:
-        cpu: 100m
-        memory: 512Mi
-      limits:
-        cpu: 500m
-        memory: 1Gi
-gitlab-runner:
-  install: false
-
-# Registry — S3-backed, NodePort for kubelet image pulls
-# Token auth disabled (cluster-internal only, secured by network policies)
-# Pods pull via gitlab-registry.foxhunt.svc.cluster.local:5000 (internal DNS)
-registry:
-  replicaCount: 1
-  hpa:
-    minReplicas: 1
-    maxReplicas: 1
-  service:
-    type: NodePort
-    nodePort: 30500
-  storage:
-    secret: gitlab-s3-credentials
-    key: registry
-    redirect:
-      disable: true
-  nodeSelector:
-    k8s.scaleway.com/pool-name: infra
-
-# GitLab core components — all on infra node pool
-gitlab:
-  webservice:
-    replicaCount: 2
-    hpa:
-      minReplicas: 2
-      maxReplicas: 2
-    workerProcesses: 2
-    workhorse:
-      extraArgs: "-apiLimit 0 -apiQueueLimit 0"
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    resources:
-      requests:
-        cpu: 500m
-        memory: 2Gi
-      limits:
-        cpu: 2000m
-        memory: 4Gi
-    extraEnv:
-      REDIS_URL: "redis://redis.foxhunt.svc.cluster.local:6379/8"
-
-  sidekiq:
-    replicas: 1
-    hpa:
-      minReplicas: 1
-      maxReplicas: 1
-    concurrency: 10
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    resources:
-      requests:
-        cpu: 250m
-        memory: 1Gi
-      limits:
-        cpu: 1000m
-        memory: 2Gi
-    extraEnv:
-      REDIS_URL: "redis://redis.foxhunt.svc.cluster.local:6379/8"
-
-  gitaly:
-    persistence:
-      enabled: true
-      size: 50Gi
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    resources:
-      requests:
-        cpu: 200m
-        memory: 512Mi
-      limits:
-        cpu: 1000m
-        memory: 2Gi
-
-  gitlab-shell:
-    replicaCount: 1
-    hpa:
-      minReplicas: 1
-      maxReplicas: 1
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    service:
-      type: NodePort
-      nodePort: 32222
-
-  toolbox:
-    backups:
-      objectStorage:
-        config:
-          secret: gitlab-s3-credentials
-          key: connection
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-
-  kas:
-    enabled: false
-    hpa:
-      minReplicas: 0
-      maxReplicas: 0
-
-  gitlab-exporter:
-    enabled: false

infra/k8s/gpu-overlays/ml-training-service-gpu.yaml

@@ -1,169 +0,0 @@
-# GPU-enabled overlay for ml-training-service
-# Apply manually: kubectl apply -f infra/k8s/gpu-overlays/ml-training-service-gpu.yaml
-# Revert to CPU: kubectl apply -f infra/k8s/services/ml-training-service.yaml
-#
-# Binary fetched from MinIO at pod startup — works on any node pool.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: ml-training-service
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: ml-training-service
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  strategy:
-    type: RollingUpdate
-    rollingUpdate:
-      maxSurge: 0
-      maxUnavailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: ml-training-service
-  template:
-    metadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "9094"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: ml-training-service
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      serviceAccountName: ml-training-service
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 1000
-        runAsGroup: 1000
-        fsGroup: 1000
-        seccompProfile:
-          type: RuntimeDefault
-      nodeSelector:
-        k8s.scaleway.com/pool-name: gpu-inference
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      imagePullSecrets:
-        - name: gitlab-registry
-      initContainers:
-        - name: fetch-binary
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              rclone copyto \
-                ":s3:foxhunt-binaries/services/ml-training-service" \
-                "/binaries/ml-training-service" \
-                --s3-provider=Minio \
-                --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
-                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
-                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
-                --s3-no-check-bucket
-              chmod +x /binaries/ml-training-service
-              echo "Fetched ml-training-service ($(stat -c%s /binaries/ml-training-service) bytes)"
-          env:
-            - name: MINIO_ACCESS_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: access-key
-            - name: MINIO_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: secret-key
-          volumeMounts:
-            - name: binaries
-              mountPath: /binaries
-          resources:
-            requests:
-              cpu: 100m
-              memory: 64Mi
-            limits:
-              cpu: 500m
-              memory: 128Mi
-      containers:
-        - name: ml-training-service
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-          command: ["/binaries/ml-training-service", "serve"]
-          ports:
-            - containerPort: 50053
-              name: grpc
-            - containerPort: 9094
-              name: metrics
-          env:
-            - name: DATABASE_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: DATABASE_URL
-              value: "postgresql://foxhunt:$(DATABASE_PASSWORD)@postgres:5432/foxhunt"
-            - name: REDIS_URL
-              value: "redis://redis:6379"
-            - name: JWT_SECRET
-              valueFrom:
-                secretKeyRef:
-                  name: jwt-secret
-                  key: secret
-            - name: JWT_ISSUER
-              value: foxhunt-api
-            - name: JWT_AUDIENCE
-              value: foxhunt-services
-            - name: S3_ENDPOINT
-              value: "http://minio.foxhunt.svc.cluster.local:9000"
-            - name: S3_BUCKET
-              value: foxhunt-models
-            - name: ENABLE_GPU
-              value: "true"
-            - name: RUST_LOG
-              value: info
-            - name: OTEL_EXPORTER_OTLP_ENDPOINT
-              value: "http://tempo.foxhunt.svc.cluster.local:4317"
-          volumeMounts:
-            - name: binaries
-              mountPath: /binaries
-              readOnly: true
-            - name: tls-certs
-              mountPath: /app/certs/ml-training-service
-              readOnly: true
-            - name: tmp
-              mountPath: /tmp
-          readinessProbe:
-            tcpSocket:
-              port: 50053
-            initialDelaySeconds: 15
-            periodSeconds: 10
-          livenessProbe:
-            tcpSocket:
-              port: 50053
-            initialDelaySeconds: 30
-            periodSeconds: 15
-            failureThreshold: 5
-          resources:
-            requests:
-              nvidia.com/gpu: "1"
-              cpu: "1"
-              memory: 2Gi
-            limits:
-              nvidia.com/gpu: "1"
-              cpu: "4"
-              memory: 8Gi
-      volumes:
-        - name: binaries
-          emptyDir:
-            sizeLimit: 200Mi
-        - name: tls-certs
-          secret:
-            secretName: ml-training-tls
-        - name: tmp
-          emptyDir:
-            sizeLimit: 50Mi

infra/k8s/gpu-overlays/trading-service-gpu.yaml

@@ -1,156 +0,0 @@
-# GPU-enabled overlay for trading-service
-# Apply manually: kubectl apply -f infra/k8s/gpu-overlays/trading-service-gpu.yaml
-# Revert to CPU: kubectl apply -f infra/k8s/services/trading-service.yaml
-#
-# Binary fetched from MinIO at pod startup — works on any node pool.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: trading-service
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: trading-service
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  strategy:
-    type: RollingUpdate
-    rollingUpdate:
-      maxSurge: 0
-      maxUnavailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: trading-service
-  template:
-    metadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "9092"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: trading-service
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 1000
-        runAsGroup: 1000
-        fsGroup: 1000
-        seccompProfile:
-          type: RuntimeDefault
-      nodeSelector:
-        k8s.scaleway.com/pool-name: l40s
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      imagePullSecrets:
-        - name: gitlab-registry
-      initContainers:
-        - name: fetch-binary
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              rclone copyto \
-                ":s3:foxhunt-binaries/services/trading-service" \
-                "/binaries/trading-service" \
-                --s3-provider=Minio \
-                --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
-                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
-                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
-                --s3-no-check-bucket
-              chmod +x /binaries/trading-service
-              echo "Fetched trading-service ($(stat -c%s /binaries/trading-service) bytes)"
-          env:
-            - name: MINIO_ACCESS_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: access-key
-            - name: MINIO_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: secret-key
-          volumeMounts:
-            - name: binaries
-              mountPath: /binaries
-          resources:
-            requests:
-              cpu: 100m
-              memory: 64Mi
-            limits:
-              cpu: 500m
-              memory: 128Mi
-      containers:
-        - name: trading-service
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-          command: ["/binaries/trading-service"]
-          ports:
-            - containerPort: 50051
-              name: grpc
-            - containerPort: 9092
-              name: metrics
-          env:
-            - name: DATABASE_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: DATABASE_URL
-              value: "postgresql://foxhunt:$(DATABASE_PASSWORD)@postgres:5432/foxhunt"
-            - name: REDIS_URL
-              value: "redis://redis:6379"
-            - name: JWT_SECRET
-              valueFrom:
-                secretKeyRef:
-                  name: jwt-secret
-                  key: secret
-            - name: JWT_ISSUER
-              value: foxhunt-api
-            - name: JWT_AUDIENCE
-              value: foxhunt-services
-            - name: QUESTDB_ILP_HOST
-              value: "questdb:9009"
-            - name: GRPC_PORT
-              value: "50051"
-            - name: ENABLE_GPU_INFERENCE
-              value: "true"
-            - name: RUST_LOG
-              value: info
-          volumeMounts:
-            - name: binaries
-              mountPath: /binaries
-              readOnly: true
-            - name: tmp
-              mountPath: /tmp
-          readinessProbe:
-            exec:
-              command:
-                - grpc_health_probe
-                - -addr=localhost:50051
-            initialDelaySeconds: 10
-            periodSeconds: 10
-          resources:
-            requests:
-              nvidia.com/gpu: "1"
-              cpu: "1"
-              memory: 2Gi
-            limits:
-              nvidia.com/gpu: "1"
-              cpu: "4"
-              memory: 8Gi
-      volumes:
-        - name: binaries
-          emptyDir:
-            sizeLimit: 200Mi
-        - name: tmp
-          emptyDir:
-            sizeLimit: 50Mi

infra/k8s/jobs/migrate.yaml

@@ -1,34 +0,0 @@
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: foxhunt-migrate
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: migration
-spec:
-  backoffLimit: 3
-  template:
-    spec:
-      restartPolicy: Never
-      imagePullSecrets:
-        - name: gitlab-registry
-      containers:
-        - name: migrate
-          image: rg.fr-par.scw.cloud/foxhunt/api:latest
-          command: ["sqlx", "migrate", "run"]
-          env:
-            - name: DB_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: DATABASE_URL
-              value: "postgresql://foxhunt:$(DB_PASSWORD)@postgres:5432/foxhunt"
-          resources:
-            requests:
-              memory: "64Mi"
-              cpu: "50m"
-            limits:
-              memory: "256Mi"
-              cpu: "200m"

infra/k8s/mattermost/mattermost.yaml

@@ -1,121 +0,0 @@
-# Mattermost Team Edition — self-hosted Slack alternative for foxhunt.
-# Runs on platform nodes. PostgreSQL is the shared foxhunt instance.
-# Access: https://chat.fxhnt.ai (Tailscale only)
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: mattermost-data
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: mattermost
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd
-  resources:
-    requests:
-      storage: 5Gi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: mattermost
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: mattermost
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: mattermost
-  strategy:
-    type: Recreate  # RWO PVC
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: mattermost
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      containers:
-        - name: mattermost
-          image: mattermost/mattermost-team-edition:10.5
-          ports:
-            - containerPort: 8065
-              name: http
-          env:
-            - name: DB_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: MM_SQLSETTINGS_DRIVERNAME
-              value: postgres
-            - name: MM_SQLSETTINGS_DATASOURCE
-              value: "postgres://foxhunt:$(DB_PASSWORD)@postgres.foxhunt.svc.cluster.local:5432/mattermost?sslmode=disable"
-            - name: MM_SERVICESETTINGS_SITEURL
-              value: "https://chat.fxhnt.ai"
-            - name: MM_SERVICESETTINGS_LISTENADDRESS
-              value: ":8065"
-            - name: MM_SERVICESETTINGS_ENABLEINCOMINGWEBHOOKS
-              value: "true"
-            - name: MM_SERVICESETTINGS_ENABLEPOSTUSERNAMEOVERRIDE
-              value: "true"
-            - name: MM_SERVICESETTINGS_ENABLEPOSTICONOVERRIDE
-              value: "true"
-            - name: MM_LOGSETTINGS_ENABLECONSOLE
-              value: "true"
-            - name: MM_LOGSETTINGS_CONSOLELEVEL
-              value: INFO
-            - name: MM_LOGSETTINGS_ENABLEFILE
-              value: "false"
-            - name: MM_FILESETTINGS_DRIVERNAME
-              value: local
-            - name: MM_FILESETTINGS_DIRECTORY
-              value: /mattermost/data
-            # Email via Stalwart — uncomment after Stalwart is deployed:
-            # - name: MM_EMAILSETTINGS_SMTPSERVER
-            #   value: stalwart.foxhunt.svc.cluster.local
-            # - name: MM_EMAILSETTINGS_SMTPPORT
-            #   value: "25"
-            # - name: MM_EMAILSETTINGS_SENDEMAILNOTIFICATIONS
-            #   value: "true"
-          volumeMounts:
-            - name: mattermost-data
-              mountPath: /mattermost/data
-          readinessProbe:
-            httpGet:
-              path: /api/v4/system/ping
-              port: 8065
-            initialDelaySeconds: 15
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              cpu: "1"
-              memory: 1Gi
-      volumes:
-        - name: mattermost-data
-          persistentVolumeClaim:
-            claimName: mattermost-data
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: mattermost
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: mattermost
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: mattermost
-  ports:
-    - port: 8065
-      targetPort: 8065
-      name: http

infra/k8s/minio/kustomization.yaml

@@ -1,4 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - minio.yaml

infra/k8s/minio/minio.yaml

@@ -1,188 +0,0 @@
-# MinIO — in-cluster S3 for compiled binaries and trained models
-# Replaces Scaleway public S3 to keep proprietary code/IP in-cluster
-# Plain HTTP only — internal cluster traffic, no TLS needed
-apiVersion: v1
-kind: Secret
-metadata:
-  name: minio-credentials
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-type: Opaque
-stringData:
-  root-user: foxhunt-minio
-  root-password: CHANGE_ME_BEFORE_DEPLOY
-  # rclone-compatible keys (same values, different key names for initContainers)
-  access-key: foxhunt-minio
-  secret-key: CHANGE_ME_BEFORE_DEPLOY
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: minio-data-new
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: scw-bssd
-  resources:
-    requests:
-      storage: 150Gi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: minio
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate  # single-node MinIO — no rolling update
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: minio
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: minio
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      containers:
-        - name: minio
-          image: minio/minio:latest
-          args: ["server", "/data", "--console-address", ":9001"]
-          ports:
-            - containerPort: 9000
-              name: api
-            - containerPort: 9001
-              name: console
-          env:
-            - name: MINIO_ROOT_USER
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: root-user
-            - name: MINIO_ROOT_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: root-password
-          volumeMounts:
-            - name: data
-              mountPath: /data
-          readinessProbe:
-            httpGet:
-              path: /minio/health/live
-              port: 9000
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          livenessProbe:
-            httpGet:
-              path: /minio/health/live
-              port: 9000
-            initialDelaySeconds: 10
-            periodSeconds: 15
-            failureThreshold: 5
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              cpu: "1"
-              memory: 2Gi
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: minio-data-new
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: minio
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: minio
-  ports:
-    - port: 9000
-      targetPort: 9000
-      name: api
-    - port: 9001
-      targetPort: 9001
-      name: console
----
-# Create buckets on first deploy
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: minio-init-buckets
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  backoffLimit: 5
-  ttlSecondsAfterFinished: 300
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: minio-init
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      restartPolicy: OnFailure
-      containers:
-        - name: mc
-          image: minio/mc:latest
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              echo "Waiting for MinIO to be ready..."
-              until mc alias set foxhunt http://minio.foxhunt.svc.cluster.local:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" 2>/dev/null; do
-                echo "MinIO not ready, retrying in 3s..."
-                sleep 3
-              done
-              echo "Creating buckets..."
-              mc mb --ignore-existing foxhunt/foxhunt-binaries
-              mc mb --ignore-existing foxhunt/foxhunt-models
-              mc mb --ignore-existing foxhunt/foxhunt-training-results
-              mc mb --ignore-existing foxhunt/argo-logs
-              mc mb --ignore-existing foxhunt/foxhunt-sccache
-              mc mb --ignore-existing foxhunt/foxhunt-training-data
-              # Plan 5 Task 3 (A.4.1): nsys profile artefacts bucket.
-              # Stores .nsys-rep files keyed by commit SHA under
-              # profiles/<short-sha>/profile-seed*-fold*-*.nsys-rep.
-              mc mb --ignore-existing foxhunt/foxhunt-training-artifacts
-              echo "Buckets created:"
-              mc ls foxhunt/
-          env:
-            - name: MINIO_ROOT_USER
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: root-user
-            - name: MINIO_ROOT_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: root-password
-          resources:
-            requests:
-              cpu: 50m
-              memory: 32Mi
-            limits:
-              cpu: 200m
-              memory: 64Mi

infra/k8s/monitoring/APPLY-ORDER.md

@@ -1,104 +0,0 @@
-# Cockpit + NAT Gateway — Apply Order
-
-## Prerequisites
-- `scw` CLI authenticated
-- `terragrunt` installed
-- `kubectl` access to foxhunt cluster
-- `helm` installed
-
-## Phase 1: Cockpit (metrics backend)
-
-1. Apply Cockpit Terraform module:
-   ```bash
-   cd infra/live/production/cockpit
-   terragrunt apply
-   ```
-
-2. Note the outputs:
-   ```bash
-   terragrunt output metrics_push_url
-   terragrunt output -raw push_token
-   terragrunt output grafana_url
-   terragrunt output -raw grafana_password
-   ```
-
-3. Create the K8s secret for the push token:
-   ```bash
-   kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
-   kubectl -n monitoring create secret generic cockpit-push-token \
-     --from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
-   ```
-
-4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`:
-   - Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output
-
-5. Deploy DCGM Exporter:
-   ```bash
-   kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
-   ```
-
-6. Deploy Grafana Alloy (k8s-monitoring):
-   ```bash
-   helm repo add grafana https://grafana.github.io/helm-charts
-   helm repo update
-   helm install k8s-monitoring grafana/k8s-monitoring \
-     --namespace monitoring \
-     -f infra/k8s/monitoring/alloy-values.yaml
-   ```
-
-7. Verify metrics flow:
-   - Open Cockpit Grafana (URL from step 2)
-   - Login with foxhunt-admin / (password from step 2)
-   - Check "Kubernetes / Compute Resources / Cluster" dashboard
-
-## Phase 2: GitLab Prometheus remote_write
-
-1. Update placeholders in `infra/k8s/gitlab/values.yaml`:
-   - Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN`
-
-2. Upgrade GitLab Helm release:
-   ```bash
-   helm upgrade gitlab gitlab/gitlab \
-     --namespace foxhunt \
-     -f infra/k8s/gitlab/values.yaml
-   ```
-
-## Phase 3: NAT Gateway
-
-1. Apply Public Gateway module:
-   ```bash
-   cd infra/live/production/public-gateway
-   terragrunt apply
-   ```
-   This creates the gateway and attaches it to the private network.
-   Existing nodes still have public IPs — nothing breaks yet.
-
-2. Apply Kapsule pool changes (disables public IPs):
-   ```bash
-   cd infra/live/production/kapsule
-   terragrunt apply
-   ```
-   **WARNING**: This may recreate nodes. For zero-downtime:
-
-3. Zero-downtime node migration for services pool:
-   ```bash
-   # Temporarily allow 2 nodes
-   scw k8s pool update <services-pool-id> max-size=2 size=2
-   # Wait for new node (private IP) to be Ready
-   kubectl get nodes -w
-   # Drain old node
-   kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
-   # Scale back
-   scw k8s pool update <services-pool-id> max-size=1 size=1
-   ```
-
-4. Repeat for gitlab pool (if not autoscaling, same process).
-
-5. Autoscaling pools (ci-training, gpu-dev):
-   - No action needed — next scale-up automatically uses private IP.
-
-6. Verify Tailscale still works:
-   ```bash
-   tailscale status  # should show foxhunt-kapsule and foxhunt-gitlab nodes
-   curl -k https://git.fxhnt.ai  # should still resolve via Tailscale
-   ```

infra/k8s/monitoring/alloy-values.yaml

@@ -1,88 +0,0 @@
-# Grafana k8s-monitoring Helm chart values (v1.x format)
-# Chart: grafana/k8s-monitoring --version 1.6.50
-# Deploys: Alloy (DaemonSet) + node-exporter + kube-state-metrics
-#
-# Install:
-#   helm repo add grafana https://grafana.github.io/helm-charts
-#   PUSH_TOKEN=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
-#   helm install k8s-monitoring grafana/k8s-monitoring \
-#     --namespace monitoring --version 1.6.50 \
-#     -f infra/k8s/monitoring/alloy-values.yaml \
-#     --set "externalServices.prometheus.extraHeaders.X-Token=${PUSH_TOKEN}"
-
-cluster:
-  name: foxhunt
-
-externalServices:
-  prometheus:
-    host: "https://d8cd4c5d-6d3b-4258-9f92-2ad0eaa36ba0.metrics.cockpit.fr-par.scw.cloud"
-    writeEndpoint: "/api/v1/push"
-    authMode: none
-    # X-Token header passed via --set to keep secret out of git
-  # Loki host required by chart validation even with logs disabled
-  loki:
-    host: "https://noop.localhost"
-    authMode: none
-
-metrics:
-  enabled: true
-  cost:
-    enabled: false
-  node-exporter:
-    enabled: true
-  kube-state-metrics:
-    enabled: true
-  kubelet:
-    enabled: true
-  cadvisor:
-    enabled: true
-  apiserver:
-    enabled: true
-
-# Alloy DaemonSet — runs on every node
-alloy-metrics:
-  alloy:
-    resources:
-      requests:
-        cpu: 50m
-        memory: 128Mi
-      limits:
-        cpu: 200m
-        memory: 256Mi
-
-# node-exporter — host metrics (CPU, memory, disk, network)
-prometheus-node-exporter:
-  resources:
-    requests:
-      cpu: 25m
-      memory: 32Mi
-    limits:
-      cpu: 100m
-      memory: 64Mi
-
-# kube-state-metrics — K8s object state
-kube-state-metrics:
-  resources:
-    requests:
-      cpu: 25m
-      memory: 64Mi
-    limits:
-      cpu: 100m
-      memory: 128Mi
-  # Scheduled on gitlab node (GP1-XS 16GB) — services (DEV1-M 4GB) too small
-
-# Disable components we don't need
-logs:
-  enabled: false
-  cluster_events:
-    enabled: false
-traces:
-  enabled: false
-opencost:
-  enabled: false
-
-# Disable log/event collectors
-alloy-logs:
-  enabled: false
-alloy-events:
-  enabled: false

infra/k8s/monitoring/dashboards/foxhunt-ci-logs.json

@@ -1,809 +0,0 @@
-{
-  "description": "Loki-powered CI pipeline log viewer \u2014 test gate, clippy, builds, errors",
-  "editable": true,
-  "graphTooltip": 2,
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "title": "Pipeline Overview",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "red",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0",
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "orientation": "auto",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=\"ERROR\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "yellow",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0",
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 4,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "orientation": "auto",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=\"WARN\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Warnings",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "blue",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 8,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "orientation": "auto",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Total Log Lines",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisLabel": "lines/interval",
-            "drawStyle": "bars",
-            "fillOpacity": 80,
-            "lineWidth": 0,
-            "pointSize": 5,
-            "showPoints": "never",
-            "stacking": {
-              "mode": "normal"
-            }
-          }
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "ERROR"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "red",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "WARN"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "yellow",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "INFO"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "green",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "DEBUG"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "blue",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 6,
-        "w": 12,
-        "x": 12,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum by (level) (count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__interval]))",
-          "legendFormat": "{{level}}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Volume by Level",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "drawStyle": "bars",
-            "fillOpacity": 60,
-            "lineWidth": 0,
-            "showPoints": "never",
-            "stacking": {
-              "mode": "normal"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 12,
-        "x": 0,
-        "y": 5
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum by (container) (count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__interval]))",
-          "legendFormat": "{{container}}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Volume by Container",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 9
-      },
-      "id": 10,
-      "title": "Full Pipeline Log Stream",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 14,
-        "w": 24,
-        "x": 0,
-        "y": 10
-      },
-      "id": 11,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "{namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=~\"$level\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg ${search:pipe}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Pipeline Logs \u2014 $pipeline",
-      "type": "logs"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "id": 20,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 25
-          },
-          "id": 21,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": false,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"ci-pipeline-.*\", container=\"main\"} |~ \"(clippy|cargo test|warning\\\\[|error\\\\[|FAILED|test result|running \\\\d+ test|failures:)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Clippy + Test Output",
-          "type": "logs"
-        }
-      ],
-      "title": "Test Gate (clippy + cargo test)",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 30,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 24,
-            "x": 0,
-            "y": 26
-          },
-          "id": 31,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": false,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"ci-pipeline-.*\", container=\"redis\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Redis Sidecar Logs",
-          "type": "logs"
-        }
-      ],
-      "title": "Redis Sidecar",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 26
-      },
-      "id": 40,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 27
-          },
-          "id": 41,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} |~ \"(?i)(error|failed|failure|panic|fatal|SIGABRT|SIGSEGV|thread .* panicked)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "All Errors & Failures",
-          "type": "logs"
-        }
-      ],
-      "title": "Errors & Failures",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 27
-      },
-      "id": 50,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 12,
-            "x": 0,
-            "y": 28
-          },
-          "id": 51,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": false,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"(ci-pipeline|compile-and-deploy|compile-and-train)-.*\", container=\"main\"} |~ \"(Compiling|Finished|warning|error|Downloading|Building|cargo build|rclone)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Compile Logs",
-          "type": "logs"
-        },
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 12,
-            "x": 12,
-            "y": 28
-          },
-          "id": 52,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": false,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"(ci-pipeline|compile-and-deploy)-.*\", container=\"main\"} |~ \"(kubectl|apply|rollout|deploy|configured|unchanged|created|image|kaniko)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Deploy & Argo Template Logs",
-          "type": "logs"
-        }
-      ],
-      "title": "Build & Deploy",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 28
-      },
-      "id": 60,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 29
-          },
-          "id": 61,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"gpu-test-.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "GPU Test Logs (DQN/PPO/TFT + Supervised)",
-          "type": "logs"
-        }
-      ],
-      "title": "GPU Test Pipeline",
-      "type": "row"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "foxhunt",
-    "ci",
-    "loki",
-    "logs"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": true,
-          "text": "ci-pipeline",
-          "value": "ci-pipeline"
-        },
-        "label": "Pipeline",
-        "name": "pipeline",
-        "options": [],
-        "query": "ci-pipeline,compile-and-deploy,compile-and-train,gpu-test,gpu-diag,gpu-walltime,build-ci-image,build-training-runtime,evaluate",
-        "type": "custom"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "selected": true,
-          "text": "All",
-          "value": "$__all"
-        },
-        "includeAll": true,
-        "label": "Level",
-        "multi": true,
-        "name": "level",
-        "options": [
-          {
-            "selected": true,
-            "text": "All",
-            "value": "$__all"
-          },
-          {
-            "selected": false,
-            "text": "ERROR",
-            "value": "ERROR"
-          },
-          {
-            "selected": false,
-            "text": "WARN",
-            "value": "WARN"
-          },
-          {
-            "selected": false,
-            "text": "INFO",
-            "value": "INFO"
-          },
-          {
-            "selected": false,
-            "text": "DEBUG",
-            "value": "DEBUG"
-          },
-          {
-            "selected": false,
-            "text": "TRACE",
-            "value": "TRACE"
-          }
-        ],
-        "query": "ERROR,WARN,INFO,DEBUG,TRACE",
-        "type": "custom"
-      },
-      {
-        "current": {
-          "selected": false,
-          "text": "",
-          "value": ""
-        },
-        "label": "Search",
-        "name": "search",
-        "type": "textbox"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-6h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "browser",
-  "title": "CI Pipeline Logs",
-  "uid": "foxhunt-ci-logs",
-  "__inputs": [
-    {
-      "name": "DS_PROMETHEUS",
-      "type": "datasource",
-      "pluginId": "prometheus"
-    },
-    {
-      "name": "DS_LOKI",
-      "type": "datasource",
-      "pluginId": "loki"
-    },
-    {
-      "name": "DS_TEMPO",
-      "type": "datasource",
-      "pluginId": "${DS_TEMPO}"
-    }
-  ]
-}

infra/k8s/monitoring/dashboards/foxhunt-training-logs.json

@@ -1,835 +0,0 @@
-{
-  "description": "Loki-powered training log viewer \u2014 hyperopt trials, epoch progress, GPU status, errors",
-  "editable": true,
-  "graphTooltip": 2,
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "title": "Training Overview",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "red",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0",
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\", level=\"ERROR\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "yellow",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0",
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 4,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\", level=\"WARN\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Warnings",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "blue",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 8,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Total Log Lines",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisLabel": "lines/interval",
-            "drawStyle": "bars",
-            "fillOpacity": 80,
-            "lineWidth": 0,
-            "showPoints": "never",
-            "stacking": {
-              "mode": "normal"
-            }
-          }
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "ERROR"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "red",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "WARN"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "yellow",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "INFO"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "green",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 6,
-        "w": 12,
-        "x": 12,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum by (level) (count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__interval]))",
-          "legendFormat": "{{level}}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Volume by Level",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "drawStyle": "bars",
-            "fillOpacity": 60,
-            "lineWidth": 0,
-            "showPoints": "never",
-            "stacking": {
-              "mode": "normal"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 12,
-        "x": 0,
-        "y": 5
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum by (pod) (count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__interval]))",
-          "legendFormat": "{{pod}}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Volume by Pod",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 9
-      },
-      "id": 10,
-      "title": "Full Training Log Stream",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 14,
-        "w": 24,
-        "x": 0,
-        "y": 10
-      },
-      "id": 11,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\", level=~\"$level\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg ${search:pipe}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Training Logs \u2014 $job",
-      "type": "logs"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "id": 20,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 25
-          },
-          "id": 21,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(epoch|loss|reward|sharpe|accuracy|val_loss|train_loss|best_trial|trial|fold|window|learning_rate|lr=|MaxDD|win.?rate|drawdown)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Epoch / Loss / Reward / Sharpe",
-          "type": "logs"
-        }
-      ],
-      "title": "Epoch Progress & Loss",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 30,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 26
-          },
-          "id": 31,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"(hyperopt|train)-.*\"} |~ \"(?i)(trial|best|cost|param|learning_rate|hidden_dim|batch_size|buffer_size|gamma|epsilon|clip|PSO|iteration|objective|optimal)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Hyperopt Trial Results",
-          "type": "logs"
-        }
-      ],
-      "title": "Hyperopt Trials",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 26
-      },
-      "id": 40,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 27
-          },
-          "id": 41,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(GPU|CUDA|VRAM|cuda|gpu|vram|OOM|out.of.memory|nvrtc|tensor.core|BF16|bf16|SM|occupancy|kernel|device|nvidia)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "GPU / CUDA / VRAM Logs",
-          "type": "logs"
-        }
-      ],
-      "title": "GPU / CUDA / VRAM",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 27
-      },
-      "id": 50,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 24,
-            "x": 0,
-            "y": 28
-          },
-          "id": 51,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(walk.?forward|fold|backtest|window|train_start|train_end|val_start|val_end|test_start|test_end|embargo|purge|bars|features)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Walk-Forward / Fold / Backtest",
-          "type": "logs"
-        }
-      ],
-      "title": "Walk-Forward Windows",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 28
-      },
-      "id": 60,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 29
-          },
-          "id": 61,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(error|failed|failure|panic|fatal|SIGABRT|SIGSEGV|thread .* panicked|OOM|NaN|inf|diverge)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "All Training Errors & Failures",
-          "type": "logs"
-        }
-      ],
-      "title": "Errors & Failures",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 29
-      },
-      "id": 70,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 24,
-            "x": 0,
-            "y": 30
-          },
-          "id": 71,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(loading|loaded|dbn|ofi|features|bars|MBP.?10|OHLCV|symbol|normalize|norm_stats|state_dim|rclone|S3|download)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Data Loading / OFI / DBN / Features",
-          "type": "logs"
-        }
-      ],
-      "title": "Data Loading & OFI",
-      "type": "row"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "foxhunt",
-    "training",
-    "loki",
-    "logs",
-    "gpu"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": true,
-          "text": "All Training",
-          "value": "(train|hyperopt|compile-and-train)"
-        },
-        "label": "Job Type",
-        "name": "job",
-        "options": [
-          {
-            "selected": true,
-            "text": "All Training",
-            "value": "(train|hyperopt|compile-and-train)"
-          },
-          {
-            "selected": false,
-            "text": "DQN",
-            "value": "train-dqn"
-          },
-          {
-            "selected": false,
-            "text": "PPO",
-            "value": "train-ppo"
-          },
-          {
-            "selected": false,
-            "text": "Supervised",
-            "value": "train-supervised"
-          },
-          {
-            "selected": false,
-            "text": "Hyperopt",
-            "value": "hyperopt"
-          },
-          {
-            "selected": false,
-            "text": "Compile+Train",
-            "value": "compile-and-train"
-          }
-        ],
-        "query": "All Training : (train|hyperopt|compile-and-train), DQN : train-dqn, PPO : train-ppo, Supervised : train-supervised, Hyperopt : hyperopt, Compile+Train : compile-and-train",
-        "type": "custom"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "selected": true,
-          "text": "All",
-          "value": "$__all"
-        },
-        "includeAll": true,
-        "label": "Level",
-        "multi": true,
-        "name": "level",
-        "options": [
-          {
-            "selected": true,
-            "text": "All",
-            "value": "$__all"
-          },
-          {
-            "selected": false,
-            "text": "ERROR",
-            "value": "ERROR"
-          },
-          {
-            "selected": false,
-            "text": "WARN",
-            "value": "WARN"
-          },
-          {
-            "selected": false,
-            "text": "INFO",
-            "value": "INFO"
-          },
-          {
-            "selected": false,
-            "text": "DEBUG",
-            "value": "DEBUG"
-          },
-          {
-            "selected": false,
-            "text": "TRACE",
-            "value": "TRACE"
-          }
-        ],
-        "query": "ERROR,WARN,INFO,DEBUG,TRACE",
-        "type": "custom"
-      },
-      {
-        "current": {
-          "selected": false,
-          "text": "",
-          "value": ""
-        },
-        "label": "Search",
-        "name": "search",
-        "type": "textbox"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-12h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "browser",
-  "title": "Training Pipeline Logs",
-  "uid": "foxhunt-training-logs",
-  "__inputs": [
-    {
-      "name": "DS_PROMETHEUS",
-      "type": "datasource",
-      "pluginId": "prometheus"
-    },
-    {
-      "name": "DS_LOKI",
-      "type": "datasource",
-      "pluginId": "loki"
-    },
-    {
-      "name": "DS_TEMPO",
-      "type": "datasource",
-      "pluginId": "${DS_TEMPO}"
-    }
-  ]
-}

infra/k8s/monitoring/dashboards/import.sh

@@ -1,212 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-# Import all Grafana dashboards via API (Postgres-backed, no ConfigMaps).
-#
-# Dashboards use ${DS_PROMETHEUS}, ${DS_LOKI}, ${DS_TEMPO} variables.
-# This script auto-detects datasource UIDs, resolves variables, and
-# imports via the Grafana API. Dashboards are stored in Postgres.
-#
-# Required environment variables:
-#   GRAFANA_PASS - Password for admin user
-#
-# Optional overrides:
-#   GRAFANA_URL       - Grafana base URL (default: https://grafana.fxhnt.ai)
-#   GRAFANA_USER      - Grafana user (default: admin)
-#   GRAFANA_PROM_UID  - Override Prometheus datasource UID
-#   GRAFANA_LOKI_UID  - Override Loki datasource UID
-#   GRAFANA_TEMPO_UID - Override Tempo datasource UID
-#
-# Usage:
-#   GRAFANA_PASS="xxx" ./import.sh
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-GRAFANA_USER="${GRAFANA_USER:-admin}"
-GRAFANA_URL="${GRAFANA_URL:-https://grafana.fxhnt.ai}"
-GRAFANA_URL="${GRAFANA_URL%/}"
-
-if [[ -z "${GRAFANA_PASS:-}" ]]; then
-  echo "ERROR: GRAFANA_PASS is not set" >&2
-  exit 1
-fi
-
-# --- Datasource auto-detection ---
-
-detect_datasource_uid() {
-  local ds_type="$1"
-  curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" "${GRAFANA_URL}/api/datasources" 2>/dev/null | \
-    python3 -c "
-import sys, json
-for ds in json.load(sys.stdin):
-    if ds.get('type') == '${ds_type}':
-        print(ds['uid'])
-        break
-" 2>/dev/null || true
-}
-
-echo "Detecting datasource UIDs..."
-PROM_UID="${GRAFANA_PROM_UID:-$(detect_datasource_uid prometheus)}"
-LOKI_UID="${GRAFANA_LOKI_UID:-$(detect_datasource_uid loki)}"
-TEMPO_UID="${GRAFANA_TEMPO_UID:-$(detect_datasource_uid tempo)}"
-
-echo "  Prometheus: ${PROM_UID:-NOT FOUND}"
-echo "  Loki:       ${LOKI_UID:-NOT FOUND}"
-echo "  Tempo:      ${TEMPO_UID:-NOT FOUND}"
-
-if [[ -z "$PROM_UID" ]]; then
-  echo "ERROR: Could not detect Prometheus datasource. Set GRAFANA_PROM_UID." >&2
-  exit 1
-fi
-
-# --- Resolve datasource variables ---
-
-resolve_dashboard() {
-  local file="$1"
-  python3 -c "
-import sys, json
-
-with open('${file}') as f:
-    d = json.load(f)
-
-d.pop('__inputs', None)
-d.pop('__requires', None)
-
-raw = json.dumps(d)
-raw = raw.replace('\${DS_PROMETHEUS}', '${PROM_UID}')
-raw = raw.replace('\${DS_LOKI}', '${LOKI_UID}')
-raw = raw.replace('\${DS_TEMPO}', '${TEMPO_UID}')
-
-print(raw)
-"
-}
-
-# --- Ensure folders exist ---
-
-ensure_folder() {
-  local title="$1"
-  local uid
-  uid=$(curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" "${GRAFANA_URL}/api/folders" 2>/dev/null | \
-    python3 -c "
-import sys, json
-for f in json.load(sys.stdin):
-    if f['title'] == '${title}':
-        print(f['uid'])
-        break
-" 2>/dev/null || true)
-
-  if [[ -z "$uid" ]]; then
-    uid=$(curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
-      -H "Content-Type: application/json" \
-      -d "{\"title\": \"${title}\"}" \
-      "${GRAFANA_URL}/api/folders" 2>/dev/null | \
-      python3 -c "import sys,json; print(json.load(sys.stdin)['uid'])" 2>/dev/null)
-    echo "  Created folder: ${title} (${uid})"
-  fi
-  echo "$uid"
-}
-
-echo ""
-echo "Ensuring folders..."
-FOLDER_CI=$(ensure_folder "CI")
-FOLDER_INFRA=$(ensure_folder "Infrastructure")
-FOLDER_OPS=$(ensure_folder "Operations")
-FOLDER_TRADING=$(ensure_folder "Trading")
-FOLDER_TRAINING=$(ensure_folder "Training")
-FOLDER_SERVICES=$(ensure_folder "Services")
-FOLDER_PLATFORM=$(ensure_folder "Platform")
-
-# --- Import a single dashboard to a folder ---
-
-import_dashboard() {
-  local file="$1"
-  local folder_uid="$2"
-  local name
-  name="$(basename "$file" .json)"
-
-  local resolved
-  resolved=$(resolve_dashboard "$file")
-
-  local response
-  response=$(curl -s -w "\n%{http_code}" \
-    -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
-    -H "Content-Type: application/json" \
-    -d "{\"dashboard\": ${resolved}, \"folderUid\": \"${folder_uid}\", \"overwrite\": true}" \
-    "${GRAFANA_URL}/api/dashboards/db")
-
-  local http_code
-  http_code=$(echo "$response" | tail -1)
-
-  if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then
-    echo "  OK   ${name}"
-  else
-    local body
-    body=$(echo "$response" | sed '$d')
-    echo "  FAIL ${name} (${http_code}): ${body}" >&2
-    return 1
-  fi
-}
-
-# --- Dashboard → Folder mapping ---
-
-declare -A DASHBOARD_FOLDERS=(
-  # Original dashboards
-  [foxhunt-cockpit]="${FOLDER_OPS}"
-  [foxhunt-trading]="${FOLDER_TRADING}"
-  [foxhunt-training]="${FOLDER_TRAINING}"
-  [foxhunt-infrastructure]="${FOLDER_INFRA}"
-  [foxhunt-observability]="${FOLDER_INFRA}"
-  [foxhunt-ci-logs]="${FOLDER_CI}"
-  [foxhunt-training-logs]="${FOLDER_TRAINING}"
-  [foxhunt-training-deep-dive]="${FOLDER_TRAINING}"
-  # Application services
-  [svc-api-gateway]="${FOLDER_SERVICES}"
-  [svc-trading-service]="${FOLDER_SERVICES}"
-  [svc-backtesting]="${FOLDER_SERVICES}"
-  [svc-data-acquisition]="${FOLDER_SERVICES}"
-  [svc-ml-training]="${FOLDER_SERVICES}"
-  [svc-trading-agent]="${FOLDER_SERVICES}"
-  [svc-web-gateway]="${FOLDER_SERVICES}"
-  # Platform services
-  [platform-prometheus]="${FOLDER_PLATFORM}"
-  [platform-grafana]="${FOLDER_PLATFORM}"
-  [platform-loki]="${FOLDER_PLATFORM}"
-  [platform-tempo]="${FOLDER_PLATFORM}"
-  [platform-redis]="${FOLDER_PLATFORM}"
-  [platform-postgres]="${FOLDER_PLATFORM}"
-  [platform-argo]="${FOLDER_PLATFORM}"
-  [platform-questdb]="${FOLDER_PLATFORM}"
-  [platform-costs]="${FOLDER_PLATFORM}"
-  [platform-scaleway]="${FOLDER_PLATFORM}"
-)
-
-# --- Deploy all dashboards ---
-
-echo ""
-echo "Importing dashboards..."
-
-failed=0
-total=0
-
-for dashboard in "${SCRIPT_DIR}"/*.json; do
-  [[ -f "$dashboard" ]] || continue
-  name="$(basename "$dashboard" .json)"
-  folder_uid="${DASHBOARD_FOLDERS[$name]:-}"
-
-  if [[ -z "$folder_uid" ]]; then
-    echo "  SKIP ${name} (no folder mapping)"
-    continue
-  fi
-
-  total=$((total + 1))
-  if ! import_dashboard "$dashboard" "$folder_uid"; then
-    failed=$((failed + 1))
-  fi
-done
-
-echo ""
-if [[ $failed -eq 0 ]]; then
-  echo "All ${total} dashboards deployed successfully."
-else
-  echo "ERROR: ${failed}/${total} dashboard(s) failed." >&2
-  exit 1
-fi

infra/k8s/monitoring/dashboards/platform-grafana.json

@@ -1,802 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "Grafana Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "red",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"grafana.*\", phase=\"Running\"}",
-          "instant": true,
-          "legendFormat": "Running",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Pod Status",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 3
-              },
-              {
-                "color": "red",
-                "value": 10
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"grafana.*\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Restarts",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[$interval]))",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"grafana.*\", resource=\"memory\"})",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Memory (bytes)",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Cores",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network RX",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network TX",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"grafana.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 16,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"grafana.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "platform",
-    "grafana"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "Grafana",
-  "uid": "plat-grafana"
-}

infra/k8s/monitoring/dashboards/platform-prometheus.json

@@ -1,946 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "Prometheus Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "count(up{job!=\"\"} == 1)",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Targets Up",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "count(up{job!=\"\"} == 0) or vector(0)",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Targets Down",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_head_series",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Head Series",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(prometheus_tsdb_head_samples_appended_total[5m])",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Ingestion Rate",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_wal_storage_size_bytes",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "WAL Size",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_head_chunks",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Head Chunks",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_compactions_total",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Compactions",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_storage_blocks_bytes",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Storage Size",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"prometheus.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"prometheus.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"prometheus.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"prometheus.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "Scrape Performance",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "s"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "scrape_duration_seconds",
-          "legendFormat": "{{job}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Scrape Duration",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 15
-      },
-      "id": 16,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(prometheus_tsdb_head_samples_appended_total[$interval])",
-          "legendFormat": "samples/s",
-          "refId": "A"
-        }
-      ],
-      "title": "Samples Ingested",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 23
-      },
-      "id": 17,
-      "panels": [],
-      "title": "Target Status",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {},
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "id": 18,
-      "options": {
-        "showHeader": true,
-        "sortBy": []
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "up",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Target Health",
-      "type": "table"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 32
-      },
-      "id": 19,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 33
-      },
-      "id": 20,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"prometheus.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 43
-      },
-      "id": 21,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"prometheus.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "platform",
-    "prometheus"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "Prometheus",
-  "uid": "plat-prometheus"
-}

infra/k8s/monitoring/dashboards/platform-questdb.json

@@ -1,802 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "QuestDB Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "red",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"questdb.*\", phase=\"Running\"}",
-          "instant": true,
-          "legendFormat": "Running",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Pod Status",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 3
-              },
-              {
-                "color": "red",
-                "value": 10
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"questdb.*\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Restarts",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[$interval]))",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"questdb.*\", resource=\"memory\"})",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Memory (bytes)",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Cores",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network RX",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network TX",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"questdb.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 16,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"questdb.*\"} | json | level =~ \"ERROR|WARN|FATAL\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "platform",
-    "questdb"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "QuestDB",
-  "uid": "plat-questdb"
-}

infra/k8s/monitoring/dashboards/platform-tempo.json

@@ -1,802 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "Tempo Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "red",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", phase=\"Running\"}",
-          "instant": true,
-          "legendFormat": "Running",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Pod Status",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 3
-              },
-              {
-                "color": "red",
-                "value": 10
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Restarts",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[$interval]))",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", resource=\"memory\"})",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Memory (bytes)",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Cores",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network RX",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network TX",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 16,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "platform",
-    "${DS_TEMPO}"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "Tempo",
-  "uid": "plat-${DS_TEMPO}"
-}

infra/k8s/monitoring/dashboards/svc-api-gateway.json

@@ -1,963 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "Service Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "red",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"api.*\", phase=\"Running\"}",
-          "instant": true,
-          "legendFormat": "Running",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Pod Status",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 3
-              },
-              {
-                "color": "red",
-                "value": 10
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"api.*\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Restarts",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[$interval]))",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"api.*\", resource=\"memory\"})",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Memory (bytes)",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Cores",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network RX",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network TX",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "gRPC Metrics",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(grpc_server_handled_total{grpc_service=~\"foxhunt.*\", pod=~\"api.*\"}[$interval])",
-          "legendFormat": "{{grpc_method}}",
-          "refId": "A"
-        }
-      ],
-      "title": "gRPC Request Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "s"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 15
-      },
-      "id": 16,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, rate(grpc_server_handling_seconds_bucket{pod=~\"api.*\"}[$interval]))",
-          "legendFormat": "p99",
-          "refId": "A"
-        }
-      ],
-      "title": "gRPC Latency (p99)",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 15
-      },
-      "id": 17,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(grpc_server_handled_total{grpc_code!=\"OK\", pod=~\"api.*\"}[$interval])",
-          "legendFormat": "{{grpc_code}}",
-          "refId": "A"
-        }
-      ],
-      "title": "gRPC Errors",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 23
-      },
-      "id": 18,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "id": 19,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"api.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 34
-      },
-      "id": 20,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"api.*\"} | json | level =~ \"ERROR|WARN|FATAL\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "service",
-    "api"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "API Gateway",
-  "uid": "svc-api-gateway"
-}

infra/k8s/monitoring/dcgm-exporter.yaml

@@ -1,125 +0,0 @@
-# NVIDIA DCGM Exporter — GPU metrics for Cockpit
-# Runs on all GPU node pools (ci-training, gpu-dev)
-# Alloy auto-discovers and scrapes these pods via annotations
----
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: monitoring
-  labels:
-    app.kubernetes.io/part-of: foxhunt
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: dcgm-counters
-  namespace: monitoring
-data:
-  counters.csv: |
-    # Basic GPU metrics (no DCP/profiling — avoids SYS_ADMIN requirement)
-    dcgm_sm_clock,     gauge, SM clock frequency (in MHz).
-    dcgm_memory_clock, gauge, Memory clock frequency (in MHz).
-    dcgm_memory_temp,  gauge, Memory temperature (in C).
-    dcgm_gpu_temp,     gauge, GPU temperature (in C).
-    dcgm_power_usage,              gauge, Power draw (in W).
-    dcgm_total_energy_consumption, counter, Total energy consumption since boot (in mJ).
-    dcgm_gpu_utilization,      gauge, GPU utilization (in %).
-    dcgm_mem_copy_utilization, gauge, Memory utilization (in %).
-    dcgm_enc_utilization,      gauge, Encoder utilization (in %).
-    dcgm_dec_utilization,      gauge, Decoder utilization (in %).
-    dcgm_xid_errors,           gauge, Value of the last XID error encountered.
-    dcgm_fb_free, gauge, Framebuffer memory free (in MiB).
-    dcgm_fb_used, gauge, Framebuffer memory used (in MiB).
----
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: dcgm-exporter
-  namespace: monitoring
-  labels:
-    app.kubernetes.io/name: dcgm-exporter
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: dcgm-exporter
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: dcgm-exporter
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "9400"
-        prometheus.io/path: "/metrics"
-    spec:
-      # Only schedule on nodes that have GPUs
-      affinity:
-        nodeAffinity:
-          requiredDuringSchedulingIgnoredDuringExecution:
-            nodeSelectorTerms:
-              - matchExpressions:
-                  - key: nvidia.com/gpu.present
-                    operator: In
-                    values:
-                      - "true"
-      # NOTE: runtimeClassName: nvidia removed — Scaleway Kapsule GPU pools use
-      # the nvidia container runtime as the default containerd handler. The
-      # RuntimeClass CRD is only created by the full NVIDIA GPU Operator, not
-      # the device plugin alone, so requiring it prevents pod scheduling.
-      tolerations:
-        # Tolerate GPU taints so the exporter can run on GPU nodes
-        - operator: Exists
-          effect: NoSchedule
-      containers:
-        - name: dcgm-exporter
-          image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
-          ports:
-            - name: metrics
-              containerPort: 9400
-          args:
-            - "-f"
-            - "/etc/dcgm-exporter/custom/counters.csv"
-          env:
-            - name: DCGM_EXPORTER_KUBERNETES
-              value: "true"
-          resources:
-            requests:
-              cpu: 50m
-              memory: 256Mi
-            limits:
-              cpu: 200m
-              memory: 1Gi
-          securityContext:
-            runAsNonRoot: false
-            runAsUser: 0
-          volumeMounts:
-            - name: device-metrics
-              mountPath: /var/lib/dcgm
-            - name: custom-counters
-              mountPath: /etc/dcgm-exporter/custom
-              readOnly: true
-      volumes:
-        - name: device-metrics
-          emptyDir: {}
-        - name: custom-counters
-          configMap:
-            name: dcgm-counters
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: dcgm-exporter
-  namespace: monitoring
-  labels:
-    app.kubernetes.io/name: dcgm-exporter
-  annotations:
-    prometheus.io/scrape: "true"
-    prometheus.io/port: "9400"
-spec:
-  type: ClusterIP
-  selector:
-    app.kubernetes.io/name: dcgm-exporter
-  ports:
-    - name: metrics
-      port: 9400
-      targetPort: 9400

infra/k8s/monitoring/grafana-datasources.yaml

@@ -1,57 +0,0 @@
-# Grafana datasource provisioning for Loki, Tempo, and QuestDB
-# Apply via: kubectl apply -f grafana-datasources.yaml && kubectl rollout restart deploy/grafana -n foxhunt
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-datasources-extra
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: grafana
-    grafana_datasource: "1"
-data:
-  loki-tempo.yaml: |
-    apiVersion: 1
-    datasources:
-      - name: Loki
-        type: loki
-        access: proxy
-        url: http://loki.foxhunt.svc.cluster.local:3100
-        isDefault: false
-        jsonData:
-          derivedFields:
-            - datasourceUid: tempo
-              matcherRegex: '"trace_id":"(\w+)"'
-              name: TraceID
-              url: "$${__value.raw}"
-      - name: Tempo
-        type: tempo
-        uid: tempo
-        access: proxy
-        url: http://tempo.foxhunt.svc.cluster.local:3200
-        isDefault: false
-        jsonData:
-          tracesToLogsV2:
-            datasourceUid: loki
-            filterByTraceID: true
-          nodeGraph:
-            enabled: true
-          serviceMap:
-            datasourceUid: prometheus
-  questdb.yaml: |
-    apiVersion: 1
-    datasources:
-      - name: QuestDB
-        type: postgres
-        uid: questdb
-        access: proxy
-        url: questdb.foxhunt.svc.cluster.local:8812
-        user: admin
-        database: qdb
-        isDefault: false
-        jsonData:
-          sslmode: disable
-          maxOpenConns: 5
-          postgresVersion: 1200
-          timescaledb: false
-        secureJsonData:
-          password: quest

infra/k8s/monitoring/loki.yaml

@@ -1,122 +0,0 @@
-# Grafana Loki — lightweight log aggregation (single-binary mode)
-# Runs on gitlab node pool alongside Grafana and Prometheus
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: loki-config
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: loki
-data:
-  loki.yaml: |
-    auth_enabled: false
-    server:
-      http_listen_port: 3100
-      grpc_listen_port: 9096
-    common:
-      path_prefix: /loki
-      storage:
-        filesystem:
-          chunks_directory: /loki/chunks
-          rules_directory: /loki/rules
-      replication_factor: 1
-      ring:
-        kvstore:
-          store: inmemory
-    schema_config:
-      configs:
-        - from: "2024-01-01"
-          store: tsdb
-          object_store: filesystem
-          schema: v13
-          index:
-            prefix: index_
-            period: 24h
-    limits_config:
-      retention_period: 168h  # 7 days
-      max_query_series: 500
-    compactor:
-      working_directory: /loki/compactor
-      compaction_interval: 10m
-      retention_enabled: true
-      delete_request_store: filesystem
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: loki
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: loki
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: loki
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: loki
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: infra
-      containers:
-        - name: loki
-          image: grafana/loki:3.4.2
-          args: ["-config.file=/etc/loki/loki.yaml"]
-          ports:
-            - containerPort: 3100
-              name: http
-            - containerPort: 9096
-              name: grpc
-          volumeMounts:
-            - name: config
-              mountPath: /etc/loki
-            - name: data
-              mountPath: /loki
-          resources:
-            requests:
-              cpu: 100m
-              memory: 128Mi
-            limits:
-              cpu: 500m
-              memory: 512Mi
-          readinessProbe:
-            httpGet:
-              path: /ready
-              port: 3100
-            initialDelaySeconds: 10
-            periodSeconds: 10
-          livenessProbe:
-            httpGet:
-              path: /ready
-              port: 3100
-            initialDelaySeconds: 30
-            periodSeconds: 15
-      volumes:
-        - name: config
-          configMap:
-            name: loki-config
-        - name: data
-          emptyDir:
-            sizeLimit: 5Gi
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: loki
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: loki
-spec:
-  selector:
-    app.kubernetes.io/name: loki
-  ports:
-    - port: 3100
-      targetPort: 3100
-      name: http
-    - port: 9096
-      targetPort: 9096
-      name: grpc

infra/k8s/monitoring/prometheus-rules-broker.yaml

@@ -1,169 +0,0 @@
-# Broker Gateway alert rules — discovered automatically by Prometheus Operator
-# Apply: kubectl apply -f prometheus-rules-broker.yaml
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: foxhunt-broker-alerts
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-    role: alert-rules
-spec:
-  groups:
-    - name: broker_gateway_fix_session
-      interval: 30s
-      rules:
-        # FIX session disconnected >60s
-        - alert: BrokerGatewayFIXSessionDisconnected
-          expr: broker_gateway_fix_session_status == 0
-          for: 60s
-          labels:
-            severity: critical
-            service: broker_gateway_service
-            component: fix_session
-            impact: trading_halted
-          annotations:
-            summary: "FIX session {{ $labels.session_id }} DISCONNECTED"
-            description: "FIX session down >60s. All order routing halted."
-            action: "Check FIX engine logs, verify network to AMP Futures gateway"
-
-        # Order latency P95 > 100ms
-        - alert: BrokerGatewayHighOrderLatency
-          expr: |
-            histogram_quantile(0.95,
-              sum by(le, order_type) (rate(broker_gateway_order_latency_seconds_bucket[5m]))
-            ) * 1000 > 100
-          for: 5m
-          labels:
-            severity: warning
-            service: broker_gateway_service
-            component: order_routing
-            impact: performance_degraded
-          annotations:
-            summary: "Order latency P95 > 100ms for {{ $labels.order_type }}"
-
-        # Error rate > 5%
-        - alert: BrokerGatewayHighErrorRate
-          expr: |
-            (
-              sum by(severity) (rate(broker_gateway_error_total[5m]))
-              /
-              (sum(rate(broker_gateway_orders_submitted_total[5m])) + 0.001)
-            ) > 0.05
-          for: 5m
-          labels:
-            severity: warning
-            service: broker_gateway_service
-            component: error_handling
-            impact: reliability_degraded
-          annotations:
-            summary: "Error rate {{ $value | humanizePercentage }} ({{ $labels.severity }})"
-
-        # No orders in 10m during market hours
-        - alert: BrokerGatewayNoOrderActivity
-          expr: |
-            (time() - broker_gateway_last_order_time) > 600
-            and
-            (hour() >= 9 and hour() < 16)
-          for: 1m
-          labels:
-            severity: warning
-            service: broker_gateway_service
-            component: order_flow
-            impact: trading_inactive
-          annotations:
-            summary: "No order activity for 10+ minutes during market hours"
-
-        # Position mismatch > $10K
-        - alert: BrokerGatewayPositionMismatch
-          expr: |
-            abs(
-              broker_gateway_position_value_usd
-              -
-              on(symbol, account_id) group_left()
-              trading_service_position_value_usd
-            ) > 10000
-          for: 2m
-          labels:
-            severity: critical
-            service: broker_gateway_service
-            component: position_management
-            impact: data_integrity
-          annotations:
-            summary: "Position mismatch ${{ $value | humanize }} on {{ $labels.symbol }}"
-            action: "HALT trading on affected symbol, reconcile with broker"
-
-        # FIX sequence number gap
-        - alert: BrokerGatewayFIXSequenceGap
-          expr: rate(broker_gateway_sequence_number_gap_total[5m]) > 0
-          for: 1m
-          labels:
-            severity: warning
-            service: broker_gateway_service
-            component: fix_session
-            impact: message_loss
-          annotations:
-            summary: "FIX sequence gap on {{ $labels.session_id }} ({{ $value | humanize }} gaps/sec)"
-
-    - name: broker_gateway_performance
-      interval: 60s
-      rules:
-        # High FIX heartbeat RTT
-        - alert: BrokerGatewayHighFIXHeartbeatRTT
-          expr: broker_gateway_fix_heartbeat_rtt_ms > 50
-          for: 5m
-          labels:
-            severity: warning
-            service: broker_gateway_service
-            component: fix_session
-            impact: network_latency
-          annotations:
-            summary: "FIX heartbeat RTT {{ $value }}ms on {{ $labels.session_id }}"
-
-        # High order rejection rate per symbol
-        - alert: BrokerGatewayHighOrderRejectionRate
-          expr: |
-            (
-              sum by(symbol) (rate(broker_gateway_orders_rejected_total[5m]))
-              /
-              (sum by(symbol) (rate(broker_gateway_orders_submitted_total[5m])) + 0.001)
-            ) > 0.10
-          for: 5m
-          labels:
-            severity: warning
-            service: broker_gateway_service
-            component: order_validation
-            impact: order_flow_disrupted
-          annotations:
-            summary: "Order rejection rate {{ $value | humanizePercentage }} for {{ $labels.symbol }}"
-
-        # Slow database queries
-        - alert: BrokerGatewaySlowDatabaseQueries
-          expr: |
-            histogram_quantile(0.95,
-              sum by(le) (rate(database_query_duration_seconds_bucket{service="broker_gateway_service"}[5m]))
-            ) > 0.100
-          for: 5m
-          labels:
-            severity: warning
-            service: broker_gateway_service
-            component: database
-            impact: performance_degraded
-          annotations:
-            summary: "Database query P95 > 100ms"
-
-        # High margin usage
-        - alert: BrokerGatewayHighMarginUsage
-          expr: |
-            (
-              broker_gateway_margin_used_usd
-              / (broker_gateway_cash_balance_usd + 0.001)
-            ) > 0.80
-          for: 5m
-          labels:
-            severity: warning
-            service: broker_gateway_service
-            component: risk_management
-            impact: margin_pressure
-          annotations:
-            summary: "Margin usage {{ $value | humanizePercentage }} for {{ $labels.account_id }}"

infra/k8s/monitoring/prometheus-rules-hft.yaml

@@ -1,328 +0,0 @@
-# HFT alert rules — discovered automatically by Prometheus Operator
-# Apply: kubectl apply -f prometheus-rules-hft.yaml
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: foxhunt-hft-alerts
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-    role: alert-rules
-spec:
-  groups:
-    # ═══════════════════════════════
-    # CRITICAL HFT ALERTS
-    # ═══════════════════════════════
-    - name: hft_critical
-      rules:
-        - alert: TradingServiceDown
-          expr: up{job="foxhunt-services", service="trading-service"} == 0
-          for: 5s
-          labels:
-            severity: critical
-            component: trading_engine
-            impact: trading_halt
-          annotations:
-            summary: "Trading Service is DOWN"
-            description: "Trading Service has been down for >5s. All trading operations halted."
-            action: "Immediate intervention required"
-
-        - alert: DataAcquisitionDown
-          expr: up{job="foxhunt-services", service="data-acquisition-service"} == 0
-          for: 10s
-          labels:
-            severity: critical
-            component: market_data
-            impact: blind_trading
-          annotations:
-            summary: "Data Acquisition Service is DOWN"
-            description: "Market data feed down >10s. Trading without data is dangerous."
-            action: "Stop all trading immediately"
-
-        - alert: BrokerGatewayDown
-          expr: up{job="foxhunt-services", service="broker-gateway-service"} == 0
-          for: 5s
-          labels:
-            severity: critical
-            component: risk_management
-            impact: uncontrolled_risk
-          annotations:
-            summary: "Broker Gateway (risk controls) is DOWN"
-            description: "Broker gateway down >5s. Order routing and risk controls offline."
-            action: "Emergency trading halt"
-
-        - alert: TradingLatencyHigh
-          expr: histogram_quantile(0.95, foxhunt_trading_latency_seconds_bucket{operation="order_placement"}) > 0.001
-          for: 30s
-          labels:
-            severity: critical
-            component: trading_engine
-            impact: competitive_disadvantage
-          annotations:
-            summary: "Trading latency P95 > 1ms"
-            description: "95th percentile order placement latency is {{ $value }}s. HFT advantage compromised."
-            action: "Investigate performance bottlenecks"
-
-        - alert: PositionLimitBreach
-          expr: foxhunt_position_size_usd / foxhunt_position_limit_usd > 0.95
-          for: 0s
-          labels:
-            severity: critical
-            component: risk_management
-            impact: regulatory_breach
-          annotations:
-            summary: "Position limit nearly breached ({{ $value | humanizePercentage }})"
-            description: "Current position at {{ $value | humanizePercentage }} of limit."
-            action: "Reduce positions immediately"
-
-        - alert: LossLimitBreach
-          expr: foxhunt_daily_pnl_usd < foxhunt_loss_limit_usd
-          for: 0s
-          labels:
-            severity: critical
-            component: risk_management
-            impact: financial_loss
-          annotations:
-            summary: "Daily loss limit breached"
-            description: "Daily P&L ${{ $value }} breaches loss limit. Trading must halt."
-            action: "Emergency trading halt"
-
-    # ═══════════════════════════════
-    # HIGH PRIORITY ALERTS
-    # ═══════════════════════════════
-    - name: hft_high
-      rules:
-        - alert: DatabaseConnectionHigh
-          expr: foxhunt_database_connections_active / foxhunt_database_connections_max > 0.85
-          for: 1m
-          labels:
-            severity: high
-            component: database
-            impact: performance_degradation
-          annotations:
-            summary: "Database connection pool at {{ $value | humanizePercentage }}"
-
-        - alert: MemoryUsageHigh
-          expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
-          for: 2m
-          labels:
-            severity: high
-            component: system
-            impact: performance_degradation
-          annotations:
-            summary: "Memory usage {{ $value | humanizePercentage }} on {{ $labels.instance }}"
-
-        - alert: CPUUsageHigh
-          expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
-          for: 5m
-          labels:
-            severity: high
-            component: system
-            impact: performance_degradation
-          annotations:
-            summary: "CPU usage {{ $value | printf \"%.1f\" }}% on {{ $labels.instance }}"
-
-        - alert: DiskUsageHigh
-          expr: (1 - (node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"})) > 0.85
-          for: 5m
-          labels:
-            severity: high
-            component: system
-            impact: storage_failure
-          annotations:
-            summary: "Disk usage {{ $value | humanizePercentage }} on {{ $labels.instance }} {{ $labels.mountpoint }}"
-
-        - alert: OrderRejectionRateHigh
-          expr: rate(foxhunt_orders_rejected_total[5m]) / rate(foxhunt_orders_total[5m]) > 0.05
-          for: 2m
-          labels:
-            severity: high
-            component: trading_engine
-            impact: trading_inefficiency
-          annotations:
-            summary: "Order rejection rate {{ $value | humanizePercentage }}"
-
-        - alert: MarketDataLag
-          expr: foxhunt_market_data_lag_seconds > 0.1
-          for: 1m
-          labels:
-            severity: high
-            component: market_data
-            impact: stale_data
-          annotations:
-            summary: "Market data lag {{ $value }}s — trading on stale data"
-
-        - alert: DrawdownHigh
-          expr: foxhunt_drawdown_percent > 0.05
-          for: 15m
-          labels:
-            severity: high
-            component: risk_management
-            impact: financial_risk
-          annotations:
-            summary: "Portfolio drawdown {{ $value | humanizePercentage }}"
-
-    # ═══════════════════════════════
-    # MEDIUM / WARNING ALERTS
-    # ═══════════════════════════════
-    - name: hft_medium
-      rules:
-        - alert: ServiceRestarted
-          expr: increase(process_start_time_seconds[10m]) > 0
-          for: 0s
-          labels:
-            severity: medium
-            component: service
-            impact: disruption
-          annotations:
-            summary: "Service {{ $labels.job }} restarted on {{ $labels.instance }}"
-
-        - alert: NetworkLatencyHigh
-          expr: foxhunt_network_latency_seconds > 0.01
-          for: 5m
-          labels:
-            severity: medium
-            component: network
-            impact: performance_impact
-          annotations:
-            summary: "Network latency {{ $value }}s"
-
-        - alert: BackupFailed
-          expr: time() - foxhunt_last_backup_timestamp > 86400
-          for: 1h
-          labels:
-            severity: medium
-            component: backup
-            impact: data_risk
-          annotations:
-            summary: "Last backup {{ $value | humanizeDuration }} ago"
-
-        - alert: SSLCertificateExpiringSoon
-          expr: (ssl_certificate_expiry_timestamp - time()) / 86400 < 30
-          for: 1h
-          labels:
-            severity: medium
-            component: security
-            impact: service_disruption
-          annotations:
-            summary: "SSL cert expires in {{ $value }} days for {{ $labels.instance }}"
-
-    # ═══════════════════════════════
-    # BUSINESS LOGIC ALERTS
-    # ═══════════════════════════════
-    - name: hft_business
-      rules:
-        - alert: TradingVolumeAnomalyHigh
-          expr: rate(foxhunt_trades_total[5m]) > (avg_over_time(rate(foxhunt_trades_total[5m])[1h:5m]) * 3)
-          for: 2m
-          labels:
-            severity: medium
-            component: trading_engine
-            impact: business_anomaly
-          annotations:
-            summary: "Trading volume 3x hourly average ({{ $value }} trades/sec)"
-
-        - alert: TradingVolumeAnomalyLow
-          expr: rate(foxhunt_trades_total[5m]) < (avg_over_time(rate(foxhunt_trades_total[5m])[1h:5m]) * 0.1)
-          for: 10m
-          labels:
-            severity: medium
-            component: trading_engine
-            impact: business_anomaly
-          annotations:
-            summary: "Trading volume 10% of hourly average ({{ $value }} trades/sec)"
-
-        - alert: PnLVolatilityHigh
-          expr: stddev_over_time(foxhunt_pnl_usd[1h]) > 10000
-          for: 30m
-          labels:
-            severity: medium
-            component: risk_management
-            impact: financial_risk
-          annotations:
-            summary: "P&L std dev ${{ $value }} over 1h"
-
-    # ═══════════════════════════════
-    # INFRASTRUCTURE ALERTS
-    # ═══════════════════════════════
-    - name: hft_infrastructure
-      rules:
-        - alert: RedisConnectionFailed
-          expr: redis_connected_clients{job="redis"} == 0
-          for: 30s
-          labels:
-            severity: critical
-            component: redis
-            impact: cache_failure
-          annotations:
-            summary: "Redis has 0 connected clients"
-
-        - alert: PostgreSQLDown
-          expr: pg_up{job="postgres"} == 0
-          for: 30s
-          labels:
-            severity: critical
-            component: postgresql
-            impact: data_unavailable
-          annotations:
-            summary: "PostgreSQL is not responding"
-
-    # ═══════════════════════════════
-    # SECURITY ALERTS
-    # ═══════════════════════════════
-    - name: hft_security
-      rules:
-        - alert: AuthenticationFailuresHigh
-          expr: rate(foxhunt_auth_failures_total[5m]) > 5
-          for: 2m
-          labels:
-            severity: high
-            component: security
-            impact: security_breach
-          annotations:
-            summary: "{{ $value }} auth failures/sec"
-
-        - alert: UnauthorizedAccessAttempts
-          expr: rate(foxhunt_unauthorized_requests_total[5m]) > 1
-          for: 1m
-          labels:
-            severity: critical
-            component: security
-            impact: security_breach
-          annotations:
-            summary: "{{ $value }} unauthorized requests/sec from {{ $labels.source_ip }}"
-
-        - alert: APIRateLimitExceeded
-          expr: rate(foxhunt_api_rate_limit_exceeded_total[5m]) > 0.1
-          for: 5m
-          labels:
-            severity: medium
-            component: api
-            impact: service_degradation
-          annotations:
-            summary: "API rate limits exceeded {{ $value }}/sec"
-
-    # ═══════════════════════════════
-    # MONITORING HEALTH
-    # ═══════════════════════════════
-    - name: hft_monitoring
-      rules:
-        - alert: PrometheusTargetDown
-          expr: up == 0
-          for: 2m
-          labels:
-            severity: medium
-            component: monitoring
-            impact: observability_loss
-          annotations:
-            summary: "{{ $labels.job }} target {{ $labels.instance }} down >2m"
-
-        - alert: HighAlertRate
-          expr: rate(prometheus_notifications_total[5m]) > 10
-          for: 10m
-          labels:
-            severity: medium
-            component: monitoring
-            impact: alert_fatigue
-          annotations:
-            summary: "{{ $value }} alerts/sec — check for alert storm"