wild (https://github.com/davidlattimore/wild) is a Rust-native linker,
typically 10-30% faster than mold on large release-LTO links. We have
~5 service binaries each doing release-LTO link, so the saving is
meaningful: ~30-90 sec wall-time on a fresh compile.
Approach (CI-only swap, zero local-dev impact):
1. Dockerfile.ci-builder-cpu installs wild 0.8.0 alongside mold (both
linkers present; revert path is trivial).
2. .cargo/config.toml keeps `-fuse-ld=mold` as the file default so
`cargo build` works locally without requiring wild on PATH.
3. compile-and-deploy-template.yaml's compile-services script does an
in-place sed substitution `mold -> wild` immediately before
`cargo build`, gated on `command -v wild` so a missing binary
silently falls back to mold instead of failing the build.
Why sed-in-script over RUSTFLAGS env var: RUSTFLAGS env var REPLACES
the entire target.<triple>.rustflags array (per cargo docs precedence:
env > target > build, mutually exclusive — they do NOT merge), which
would silently drop our existing -Wl,-z,relro/--as-needed/target-cpu
flags. Sed swap edits one token while preserving everything else.
Validation:
- python3 yaml.safe_load_all parses the template
- cargo check --workspace --release --locked succeeds locally (still
using mold per the unchanged config.toml default)
- Dockerfile syntax visually verified; wild tarball URL confirmed live
via curl https://api.github.com/repos/davidlattimore/wild/releases/latest
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rclone (~50 MB installed) is not used by any Argo step that runs in
ci-builder-cpu. The only Argo step that does use rclone
(build-web-dashboard) runs in node:22-alpine and apk-installs rclone
itself. Removing here saves an unused ~50 MB layer in every
ci-builder-cpu pull on every CI compile pod cold-start.
Tools deliberately KEPT despite the original task suggestion to remove
them:
- terragrunt + tofu — used by terragrunt-apply step in
ci-pipeline-template.yaml (runs in ci-builder-cpu)
- kubectl — used by apply-argo-templates step in
ci-pipeline-template.yaml (runs in ci-builder-cpu)
Removing those would break those pipeline steps.
Expected impact: ~50 MB image size reduction, ~3-7 sec cold-pull
savings per compile pod. Smaller than the original ~250 MB estimate
because terragrunt/tofu/kubectl had to stay.
Validation: visually inspected Dockerfile syntax (no docker daemon
available locally). Grepped infra/k8s/argo/*.yaml for rclone usage —
only build-web-dashboard (node:22-alpine) references it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cuda-nvcc-12-4 + cuda-cudart-dev-12-4 (~400MB) added to the CPU builder.
Reverts compile-and-deploy back to ci-builder-cpu (was briefly switched
to full ci-builder which is 6GB+ and unnecessary).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
apply-argo-templates needs git (clone repo) + kubectl (apply manifests).
ci-builder-cpu has git but lacked kubectl — added it. foxhunt-runtime
has kubectl but no git and runs as non-root, so can't be used here.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add terragrunt-apply step to Argo CI pipeline (plan+apply on main push
when infra/live/ or infra/modules/ change)
- Bake OpenTofu 1.9.0 + Terragrunt 0.77.12 into ci-builder-cpu image
with SHA256 checksum verification
- Remove 3 ghost node pools (foxhunt, gitlab, h100-sxm8) from kapsule
module to match Scaleway reality
- Make terragrunt.hcl single source of truth (remove variable defaults)
- Fix GitLab TF state lock methods (POST/DELETE for HTTP backend)
- Harden PAT rotation: more retries, verification step, recovery docs
- Add weekly PAT expiry check CronJob (warns 14 days before expiry)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align all 4 Docker images with local dev environment:
- ci-builder: CUDA 12.4.1/Ubuntu 22.04 → 12.9.1/Ubuntu 24.04
- ci-builder-cpu: Debian bookworm → Ubuntu 24.04 (+ explicit rustup)
- foxhunt-runtime: Debian bookworm → Ubuntu 24.04
- foxhunt-training-runtime: CUDA 12.6.3 → 12.9.1, nvrtc 12-6 → 12-9
All images now have glibc 2.39, matching the local build machine.
Binaries compiled locally or in CI will run in any of these containers
without GLIBC_2.3x version mismatch errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both Dockerfiles already install mold v2.35.1 but the registry images
are stale builds without it. This change triggers rebuild-ci-builder
and rebuild-ci-builder-cpu pipeline steps via detect-changes.
.cargo/config.toml uses -fuse-ld=mold — without mold in the image,
linking falls back to the system default (slower).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Based on rust:1.89-slim-bookworm (~2-3GB vs ~8GB CUDA devel).
Same toolchain: mold 2.35, protoc 28.3, sccache 0.10, clang, lld.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
