dbae70e811
The biggest unexplored win against CI cold-start: ship a Docker image
that carries a fully-built /cargo-target-prebuilt/ snapshot of the
workspace's third-party dependency rlibs, and rsync it into the
cargo-target-cpu PVC as the first step of compile-services.
Cold-start compile path before this commit (fresh node, fresh PVC):
1. cargo resolves Cargo.lock ~10s
2. cargo refreshes registry index ~30s
3. cargo compiles ~1500 dep crates ~3-5 min
4. cargo compiles workspace members ~1-2 min
Total cold compile: ~5-8 min
After this commit:
1. seed-deps-cache initContainer pull image ~30-60s (in-cluster registry)
2. rsync /cargo-target-prebuilt/ into PVC ~30-60s (~6-8 GB local)
3. cargo delta-compiles workspace members ~1-2 min
Total cold compile: ~2-4 min
Net saving: 2-4 min per cold-cache compile pod.
Steady-state (warm PVC, stamp file present): the seed initContainer
exits in ~50ms, no rsync.
Files:
- infra/docker/Dockerfile.ci-deps-cache:
Multi-stage. Stage 1 atop ci-builder-cpu, runs
`cargo build --locked --release --workspace || true` (|| true so
transient errors don't kill the cache build), then snapshots
target/release/{deps,.fingerprint,build} into /cargo-target-prebuilt.
Stage 2 thin Ubuntu + rsync, COPY's the snapshot. Default CMD just
prints the contents — actual entrypoint set by initContainer spec.
Used the "build everything" approach over cargo-chef-style stubbing
because it has zero special-case logic for build.rs / examples /
weird workspace shapes; image is ~1-2 GB bigger as a result, but
pulls fast from in-cluster registry.
- infra/k8s/argo/refresh-deps-cache-template.yaml:
WorkflowTemplate that drives a Kaniko build of the Dockerfile and
pushes to gitlab-registry/.../ci-builder-cpu-with-deps:nightly.
Includes a CronWorkflow that runs nightly at 03:00 UTC, suspended
by default (enable manually after first successful run validation).
Mirrors the existing build-ci-image-template.yaml pattern.
- infra/k8s/argo/compile-and-deploy-template.yaml:
Adds a `seed-deps-cache` initContainer to compile-services. Uses
rsync --ignore-existing (don't clobber newer artifacts the PVC may
already have from a prior build) and a stamp file
/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp to short-circuit
re-seeding on warm pods. Tolerates a missing
/cargo-target-prebuilt dir (image not yet published) — emits a
WARN and continues without the seed.
- infra/k8s/argo/kustomization.yaml:
Registers the new template so kustomize/argo deploys it.
- infra/k8s/argo/README-deps-cache.md:
How to refresh manually, enable the cron, bump DEPS_VERSION,
debug a slow seed, and remove this when no longer needed.
Validation:
- python3 yaml.safe_load_all parses both compile-and-deploy and
refresh-deps-cache templates (1 + 2 docs respectively)
- cargo check --workspace --release --locked passes locally
- Dockerfile syntax visually inspected; the multi-stage COPY pattern
matches existing infra/docker/Dockerfile.* conventions
- Argo template structure mirrors build-ci-image-template.yaml which
is known-working in this cluster
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
89 lines
4.4 KiB
Docker
89 lines
4.4 KiB
Docker
# Pre-built workspace third-party dependency cache.
|
|
#
|
|
# Built nightly by the refresh-deps-cache Argo WorkflowTemplate
|
|
# (infra/k8s/argo/refresh-deps-cache-template.yaml). Compile pods consume
|
|
# this image as an initContainer that rsyncs /cargo-target-prebuilt/ into
|
|
# the cargo-target-cpu PVC, dramatically cutting cold-cache compile time.
|
|
#
|
|
# Cold compile timing on a new node (PVC empty):
|
|
# without this image: 3-5 min compiling third-party deps
|
|
# with this image: ~30-60s rsync (in-cluster registry, fast layer)
|
|
# + delta-compile of workspace members only
|
|
#
|
|
# Cache invalidation: bumping DEPS_VERSION (build-arg) forces every
|
|
# compile pod to re-seed its PVC from this image's snapshot. Bump when:
|
|
# - Cargo.lock has churned significantly (most dep rlibs invalid)
|
|
# - rustc/toolchain version changes
|
|
# - workspace member structure changes meaningfully
|
|
# Otherwise leave DEPS_VERSION alone; cargo's own fingerprinting will
|
|
# discard stale rlibs and rebuild as needed.
|
|
#
|
|
# Stage 1: build atop ci-builder-cpu (same toolchain, same cuda).
|
|
# Stage 2: copy the snapshot into a thin layer for the runtime.
|
|
ARG BASE_TAG=latest
|
|
FROM gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu:${BASE_TAG} AS builder
|
|
|
|
ARG DEPS_VERSION=1
|
|
ENV DEPS_VERSION=$DEPS_VERSION
|
|
ENV CARGO_TARGET_DIR=/workspace/target
|
|
# Prebuild MUST mirror the CI compile step's env so generated rlib
|
|
# fingerprints match what the live build expects:
|
|
ENV CARGO_INCREMENTAL=0
|
|
ENV SQLX_OFFLINE=true
|
|
# (We deliberately do NOT set RUSTC_WRAPPER=sccache here — sccache cache
|
|
# state lives on a different PVC; populating its cache from a transient
|
|
# image would be useless. The deps-cache image contains rlibs directly.)
|
|
|
|
WORKDIR /workspace
|
|
|
|
# Copy the entire workspace into the build stage. We rely on .dockerignore
|
|
# to skip target/, .git, etc. — see /workspace/.dockerignore in the repo.
|
|
# This is the "cargo-chef" approach inverted: instead of stubbing
|
|
# workspace members and building only deps, we build EVERYTHING (deps +
|
|
# workspace member rlibs). The downside is a slightly bigger image; the
|
|
# upside is zero special-case logic for build.rs files, examples, or
|
|
# weird workspace shapes.
|
|
COPY . .
|
|
|
|
# Build the entire workspace in release mode. This populates target/
|
|
# with: third-party crate rlibs (the expensive part), proc-macro .so
|
|
# files, build.rs outputs, and workspace member rlibs at this SHA.
|
|
# `|| true` so a transient compile error doesn't kill the cache build —
|
|
# we still get partial deps populated. The compile pod tolerates
|
|
# missing rlibs (cargo just rebuilds them).
|
|
RUN cargo build --locked --release --workspace || true
|
|
|
|
# Snapshot just the directories cargo actually re-uses. We exclude:
|
|
# - target/release/build (build script outputs; cargo regenerates per-SHA)
|
|
# - target/release/incremental (per-build incremental, plus we set
|
|
# CARGO_INCREMENTAL=0 so this should be empty anyway)
|
|
# - target/release/examples / tests (not needed by the service compile)
|
|
# We DO ship:
|
|
# - target/release/deps (the big one — third-party + workspace rlibs)
|
|
# - target/release/.fingerprint (mandatory; cargo's freshness oracle)
|
|
# - target/release/{*.rmeta,*.d} (top-level workspace artifacts)
|
|
RUN mkdir -p /cargo-target-prebuilt/release \
|
|
&& cp -a /workspace/target/release/deps /cargo-target-prebuilt/release/ 2>/dev/null || true \
|
|
&& cp -a /workspace/target/release/.fingerprint /cargo-target-prebuilt/release/ 2>/dev/null || true \
|
|
&& cp -a /workspace/target/release/build /cargo-target-prebuilt/release/ 2>/dev/null || true \
|
|
&& du -sh /cargo-target-prebuilt /cargo-target-prebuilt/release/* 2>/dev/null || true \
|
|
&& echo "$DEPS_VERSION" > /cargo-target-prebuilt/.deps_version
|
|
|
|
# ── Stage 2: thin runtime that just carries the snapshot + rsync ──
|
|
# rsync is needed by the initContainer in compile-and-deploy-template.yaml
|
|
# to copy this snapshot into the PVC with --ignore-existing.
|
|
FROM ubuntu:24.04 AS runtime
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
RUN apt-get update \
|
|
&& apt-get install -y --no-install-recommends rsync ca-certificates \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
ARG DEPS_VERSION=1
|
|
ENV DEPS_VERSION=$DEPS_VERSION
|
|
|
|
COPY --from=builder /cargo-target-prebuilt /cargo-target-prebuilt
|
|
|
|
# Default command prints what's in the image; the real entrypoint is
|
|
# overridden by the initContainer spec in compile-and-deploy-template.yaml.
|
|
CMD ["sh", "-c", "echo 'foxhunt ci-builder-cpu-with-deps v'$DEPS_VERSION; du -sh /cargo-target-prebuilt"]
|