build(ci): install wild linker, swap from mold in CI compile step
wild (https://github.com/davidlattimore/wild) is a Rust-native linker, typically 10-30% faster than mold on large release-LTO links. We have ~5 service binaries each doing release-LTO link, so the saving is meaningful: ~30-90 sec wall-time on a fresh compile. Approach (CI-only swap, zero local-dev impact): 1. Dockerfile.ci-builder-cpu installs wild 0.8.0 alongside mold (both linkers present; revert path is trivial). 2. .cargo/config.toml keeps `-fuse-ld=mold` as the file default so `cargo build` works locally without requiring wild on PATH. 3. compile-and-deploy-template.yaml's compile-services script does an in-place sed substitution `mold -> wild` immediately before `cargo build`, gated on `command -v wild` so a missing binary silently falls back to mold instead of failing the build. Why sed-in-script over RUSTFLAGS env var: RUSTFLAGS env var REPLACES the entire target.<triple>.rustflags array (per cargo docs precedence: env > target > build, mutually exclusive — they do NOT merge), which would silently drop our existing -Wl,-z,relro/--as-needed/target-cpu flags. Sed swap edits one token while preserving everything else. Validation: - python3 yaml.safe_load_all parses the template - cargo check --workspace --release --locked succeeds locally (still using mold per the unchanged config.toml default) - Dockerfile syntax visually verified; wild tarball URL confirmed live via curl https://api.github.com/repos/davidlattimore/wild/releases/latest Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -32,6 +32,11 @@ progress.when = "auto"
|
||||
[target.x86_64-unknown-linux-gnu]
|
||||
linker = "clang"
|
||||
rustflags = [
|
||||
# mold is the default for local + CI. CI optionally swaps to `wild`
|
||||
# via an in-script sed before `cargo build` (see compile-and-deploy
|
||||
# WorkflowTemplate); both linkers are installed in
|
||||
# Dockerfile.ci-builder-cpu. Keeping mold as the file default means
|
||||
# local dev works out-of-the-box with the more widely packaged linker.
|
||||
"-C", "link-arg=-fuse-ld=mold",
|
||||
"-C", "link-arg=-Wl,-z,relro,-z,now",
|
||||
"-C", "link-arg=-Wl,--as-needed",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# CUDA-free CI builder image for Foxhunt service compilation
|
||||
# ~2-3GB vs ~8GB for the CUDA devel variant (Dockerfile.ci-builder)
|
||||
# Contains: Rust 1.89, protoc, sccache, OpenTofu, Terragrunt, kubectl, git, OpenSSL, clang, mold, lld, make,
|
||||
# Contains: Rust 1.89, protoc, sccache, OpenTofu, Terragrunt, kubectl, git, OpenSSL, clang, mold, wild, lld, make,
|
||||
# CUDA nvcc/cudart-dev/cublas-dev (for cudarc .cubin precompilation in ml-* crate builds)
|
||||
# Base: Ubuntu 24.04 (glibc 2.39) — matches local dev and training runtime
|
||||
# Build: docker build -f infra/docker/Dockerfile.ci-builder-cpu -t foxhunt-ci-builder-cpu .
|
||||
@@ -40,6 +40,16 @@ RUN curl -fsSL https://github.com/rui314/mold/releases/download/v2.35.1/mold-2.3
|
||||
| tar xz -C /usr/local --strip-components=1 \
|
||||
&& chmod +x /usr/local/bin/mold
|
||||
|
||||
# wild linker — Rust-native, typically 10-30% faster than mold on link.
|
||||
# Drop-in replacement; selected per-build via RUSTFLAGS=-Clink-arg=-fuse-ld=wild
|
||||
# (CI sets that override). Local dev keeps mold via .cargo/config.toml.
|
||||
# Single static musl binary; mold stays as fallback if wild ever regresses.
|
||||
RUN curl -fsSL https://github.com/wild-linker/wild/releases/download/0.8.0/wild-linker-0.8.0-x86_64-unknown-linux-musl.tar.gz \
|
||||
| tar xz -C /tmp \
|
||||
&& mv /tmp/wild-linker-0.8.0-x86_64-unknown-linux-musl/wild /usr/local/bin/wild \
|
||||
&& chmod +x /usr/local/bin/wild \
|
||||
&& rm -rf /tmp/wild-linker-0.8.0-x86_64-unknown-linux-musl
|
||||
|
||||
# protoc 28.3 (proto3 optional support, matches local dev)
|
||||
RUN curl -fsSL https://github.com/protocolbuffers/protobuf/releases/download/v28.3/protoc-28.3-linux-x86_64.zip -o /tmp/protoc.zip \
|
||||
&& unzip -o /tmp/protoc.zip -d /usr/local bin/protoc 'include/*' \
|
||||
@@ -80,4 +90,4 @@ RUN curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu24
|
||||
ENV PATH="/usr/local/cuda-12.6/bin:${PATH}"
|
||||
|
||||
# Verify
|
||||
RUN rustc --version && cargo --version && git --version && protoc --version && sccache --version && make --version && mold --version && tofu --version && terragrunt --version && kubectl version --client && nvcc --version
|
||||
RUN rustc --version && cargo --version && git --version && protoc --version && sccache --version && make --version && mold --version && wild --version && tofu --version && terragrunt --version && kubectl version --client && nvcc --version
|
||||
|
||||
@@ -288,6 +288,23 @@ spec:
|
||||
CARGO_ARGS="$CARGO_ARGS -p $pkg"
|
||||
done
|
||||
|
||||
# CI-only linker swap: prefer wild over mold. wild is a Rust-native
|
||||
# linker, typically 10-30% faster than mold on release-LTO links;
|
||||
# we have ~5 service binaries each doing release-LTO, so this is
|
||||
# a meaningful 30-90 sec wall-time saving. Both linkers ship in
|
||||
# Dockerfile.ci-builder-cpu — to revert, drop this sed and rebuild.
|
||||
# Sed is idempotent (no-op if wild already substituted from a prior
|
||||
# run on the same PVC checkout) and surgical (single line in
|
||||
# .cargo/config.toml). If wild is missing for any reason, the build
|
||||
# falls back to mold once we revert this hunk.
|
||||
if command -v wild >/dev/null 2>&1; then
|
||||
echo "=== Swapping linker mold -> wild for this CI run ==="
|
||||
sed -i 's|-fuse-ld=mold|-fuse-ld=wild|' .cargo/config.toml
|
||||
grep -n 'fuse-ld' .cargo/config.toml || true
|
||||
else
|
||||
echo "WARN: wild not on PATH, sticking with mold"
|
||||
fi
|
||||
|
||||
echo "=== Building service binaries: $SERVICE_PKGS (incremental) ==="
|
||||
# --locked: skip Cargo.lock resolver work (~5-15s saved) and fail fast
|
||||
# if the lock has drifted (catches accidental Cargo.toml edits without
|
||||
|
||||
Reference in New Issue
Block a user