Two bugs in gpu-test-pipeline-template.yaml:
1. /bin/sh doesn't support bash-isms (<<<, read -ra, arrays) used in
the test runner script. Switch to /bin/bash.
2. --nocapture was appended as a cargo argument instead of a test binary
argument. Now detects whether -- separator exists in args and places
--nocapture correctly after it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: default-deny-all blocks egress from gpu-test pods to MinIO,
causing the Argo wait sidecar to hang on log upload. Fix: add egress
policy for gpu-test component (DNS, git, MinIO, Mattermost, registry).
Revert archiveLogs: false — logs should be stored.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The wait sidecar blocks indefinitely trying to save logs to MinIO which
has no running pods. Disable log archiving on all 3 templates since test
output is captured in Argo's pod logs already.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates step failed because the service account lacked
permissions to manage configmaps (auto-compile-configmap.yaml) and
networkpolicies (argo-workflow-netpol.yaml) in the foxhunt namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
busybox doesn't support Argo's emissary executor protocol, causing the
wait sidecar to hang indefinitely and hold the GPU allocation. Switch to
foxhunt-runtime which has proper /bin/sh and lets the sidecar detect
container exit. Also adds nvidia-smi check during warmup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove #[ignore] from 3 Redis tests and use REDIS_URL env var from CI.
Add Redis 7 sidecar to test-gate pod with readiness probe + nc wait loop.
Tests gracefully skip if Redis unavailable (local dev without Docker).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test-gate script runs under /bin/sh (dash), which doesn't support
${PIPESTATUS[0]}. This caused 'Bad substitution' (exit 2) on EVERY CI
run regardless of test results.
Fix: redirect cargo test to file, capture $?, then cat for log output.
This is POSIX-compatible and correctly captures the test exit code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stalwart's --init only runs when config.toml is absent, but ours
is mounted from a ConfigMap so no admin account was ever created.
Use the fallback-admin config with %{env:ADMIN_SECRET}% referencing
a stalwart-admin K8s secret. This survives pod restarts without
depending on RocksDB-stored credentials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hyperopt binary was crashing with "trials (5) must be greater than
n_initial (5)" when --trials 0 was passed. Root cause: the bump logic
set trials = n_initial instead of max(5, n_initial + 1).
Fix: both hyperopt_baseline_rl and hyperopt_baseline_supervised now clamp
trials to max(5, n_initial + 1). Also add `when` condition to the
compile-and-train DAG so hyperopt step is skipped when trials=0, and
train-best gracefully handles missing hyperopt results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test-gate pod gets cleaned up immediately on failure, losing all
stdout. Pipe cargo test through tee and grep for FAILED/panicked
output before exiting, so failure logs are preserved in Argo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CI test-gate used `--all-targets` which includes test targets where
700+ workspace lint violations accumulated (to_string on &str, assert!
on Result, shadow_unrelated, etc.). Switched to `--lib` for clippy —
library code is what matters for production safety. Test code is still
validated by `cargo test --workspace --lib`.
Also fixed:
- config: allow expect_used in test module
- ctrader-openapi: relaxed lint profile (proto-generated code)
- ctrader-openapi: constant assertion in dispatch test
- testing/load: removed [[test]] targets for non-existent files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stalwart v0.15.5 requires %{file:path}% macros to read cert/key
from disk — bare paths were treated as literal content, causing
"No certificates found" errors.
Added HTTPS egress to NetworkPolicy so Stalwart can download the
webadmin bundle and spam/geo databases from GitHub/CDN on boot.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove prometheus_fqdn and monitor_fqdn outputs that referenced
deleted DNS records. Add chat_fqdn and mail_fqdn for Mattermost
and Stalwart services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image renamed from stalwartlabs/mail-server to stalwartlabs/stalwart.
Volume mounts updated from /opt/stalwart-mail/ to /opt/stalwart/
to match the actual container filesystem layout.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Stalwart all-in-one mail server: SMTP (cluster-internal relay for
Mattermost), IMAP + HTTPS admin via Tailscale proxy. RocksDB storage,
self-signed TLS (stalwart-tls secret). No public MX, no outbound SMTP.
SCW TEM relay noted in NetworkPolicy comments for future use.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Mattermost Team Edition deployment (PVC, Service, NetworkPolicy),
DNS records (chat.fxhnt.ai, mail.fxhnt.ai), Tailscale nginx proxy
server block with WebSocket upgrade support, and Mattermost egress
rules to all 5 Argo workflow NetworkPolicies for exit handler webhooks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add auto-compile-config ConfigMap with enabled=false. Document the
manual compile-deploy webhook trigger and toggle pattern in the
ci-pipeline sensor. Auto-compile can be enabled per-push by adding
a second trigger to the sensor or via manual webhook.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add notify-result exit handler to ci-pipeline, compile-and-deploy, and
compile-and-train templates. Posts workflow outcome to Mattermost via
incoming webhook (secret: notification-webhook). Gracefully skips if
webhook URL is placeholder or missing (optional secretKeyRef).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add clippy + cargo test quality gate to CI pipeline DAG. The test-gate
runs on ci-compile-cpu nodes with persistent cargo cache PVC for fast
incremental builds. Also simplifies detect-changes by removing 5 unused
output parameters (service-packages, training-examples, deploy-list,
needs-services, needs-training) and ~170 lines of dead shell logic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
POP2-32C-128G is decommissioned. Sensor pods (ci-pipeline-trigger,
workflow-trigger) now target the platform node pool.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Without fsGroup, workspace PVC created by root-running compile step
is not writable by other steps. fsGroup=0 ensures all pods in the
workflow can write to the shared workspace volume.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- apply-argo-templates step now also applies argo-workflow-netpol.yaml
so NetworkPolicy changes auto-deploy on push
- Deleted stale compile-training WorkflowTemplate from cluster
(superseded by compile-and-train)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image builds are handled by ci-pipeline, not by compile-and-train or
compile-and-deploy. Having them here caused network policy failures
(Kaniko can't reach registry) and securityContext conflicts (Kaniko
needs root). Also removed pod-level securityContext from
compile-and-train since ci-builder needs root for cargo on PVC.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Wire build-image endpoint into EventSource + Sensor for manual triggers
- Add rebuild-* DAG tasks to compile-and-deploy and compile-and-train
templates — Kaniko layer cache makes cached builds ~15-30s
- Fix foxhunt-runtime Dockerfile: rename ubuntu user instead of groupadd
(GID 1000 already exists in ubuntu:24.04)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The compile-and-train workflow uses component label 'compile-and-train'
which wasn't covered by existing network policies. Pods were blocked by
default-deny-all from reaching GitLab SSH (git clone) and HTTP (package
upload/download). Combines compile egress (SSH, external HTTPS, DNS) with
training egress (Tempo OTLP, GitLab HTTP).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PVCs may retain ownership from a previous pod UID (e.g. after
ci-builder image rebuild). Git 2.35.2+ refuses to operate without
safe.directory set. Applied to both persistent-checkout templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The cargo-target PVC retains the source checkout between workflow runs.
When the container UID differs from the PVC owner, git refuses to operate
with "dubious ownership" error. Adding safe.directory resolves this.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
No cudnn feature flags remain after 6ba52425 — the devel headers
and libs were dead weight (~1.5 GB).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training data is already on the PVC — the rclone sync step was
wasting 2-3 minutes per job. Also fixes --output-dir → --output
to match the actual CLI flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: `option_env!("FOXHUNT_BUILD_VERSION")` in common/build_info.rs
was a compile-time macro tracked by cargo fingerprints (Rust 1.80+). Every
pipeline run with a new tag invalidated `common` → cascading rebuild of all
38 dependent workspace crates, even when zero source files changed.
Fix: replace `option_env!()` with `std::env::var()` (runtime LazyLock). Cargo
no longer tracks the version env var, so `common` only recompiles when its
source actually changes.
Also: skip git checkout when HEAD already matches target SHA (zero mtime
changes), and drop `-x` from git clean to preserve gitignored files.
Expected: ~3.7min → <1min for unchanged-crate rebuilds on warm PVC.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep gpu-warmup only in compile-and-train-template where it runs
parallel with the CPU compile step. The other templates don't need
it — CI pipeline doesn't always train, and training-workflow already
has the GPU node available from the triggering event.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
git clone gives all files the current timestamp, causing cargo to
recompile every workspace crate even when source is unchanged. Switch
to persistent checkout on the PVC: git fetch + git checkout --force
only updates files that actually changed between commits, preserving
mtimes on unchanged files so cargo correctly skips them.
- compile-services: /cargo-target/src persistent checkout
- compile-training: /cargo-target/src persistent checkout
- compile-and-train: /cargo-target/src persistent checkout + CARGO_HOME
- First run: full clone (fallback), subsequent: fetch + checkout
Expected: ~40 crate recompile → only changed crates (~1-2 min)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With 37+ workspace crates, cargo's incremental fingerprinting already
skips unchanged crates — sccache adds per-rustc wrapping overhead for
no benefit. The real speedup is CARGO_HOME on PVC (eliminates crate
re-downloads every build).
Kept: CARGO_HOME=/cargo-target/cargo-home on PVC
Removed: RUSTC_WRAPPER=sccache, SCCACHE_DIR, CARGO_INCREMENTAL=0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both compile-services and compile-training were doing full rebuilds
every run because:
1. No RUSTC_WRAPPER=sccache — sccache was installed but never used
2. No persistent CARGO_HOME — crates.io registry re-downloaded each time
3. Incremental compilation doesn't work across git clones (mtime-based)
Fix: enable sccache (content-hash based, clone-agnostic), persist
CARGO_HOME on the PVC, disable incremental (conflicts with sccache).
Expected: first run populates cache, subsequent runs ~2-3min vs ~8min.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates failed because ci-deploy role was missing
permissions for eventbus (argoproj.io), services (core), and
roles/rolebindings (rbac) needed to apply events/*.yaml files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo forbids mixing `dependencies` and `depends` in the same DAG.
Converted all tasks to `depends` with explicit state conditions so
infra steps can wait for image rebuilds while still running when
rebuilds are skipped.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Uses Argo `depends` conditions so apply-argo-templates and
terragrunt-apply wait for rebuild-ci-builder-cpu to finish (or skip)
before running, ensuring they always get the latest image.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates needs git (clone repo) + kubectl (apply manifests).
ci-builder-cpu has git but lacked kubectl — added it. foxhunt-runtime
has kubectl but no git and runs as non-root, so can't be used here.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pipeline now auto-applies WorkflowTemplates, EventSources, Sensors,
and RBAC when infra/k8s/argo/ files change — no more manual kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add terragrunt-apply step to Argo CI pipeline (plan+apply on main push
when infra/live/ or infra/modules/ change)
- Bake OpenTofu 1.9.0 + Terragrunt 0.77.12 into ci-builder-cpu image
with SHA256 checksum verification
- Remove 3 ghost node pools (foxhunt, gitlab, h100-sxm8) from kapsule
module to match Scaleway reality
- Make terragrunt.hcl single source of truth (remove variable defaults)
- Fix GitLab TF state lock methods (POST/DELETE for HTTP backend)
- Harden PAT rotation: more retries, verification step, recovery docs
- Add weekly PAT expiry check CronJob (warns 14 days before expiry)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace fragile `git fetch --depth=1 origin "$SHA"` (fails with short SHAs)
with `git clone --no-checkout --filter=blob:none` + `git checkout "$SHA"`
across all 3 Argo templates (5 clone blocks total)
- Upgrade CI compile pool from POP2-32C-128G to POP2-HC-32C-64G
(higher clock EPYC cores, 28% cheaper at €0.851/hr vs €1.18/hr)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
