The biggest unexplored win against CI cold-start: ship a Docker image
that carries a fully-built /cargo-target-prebuilt/ snapshot of the
workspace's third-party dependency rlibs, and rsync it into the
cargo-target-cpu PVC as the first step of compile-services.
Cold-start compile path before this commit (fresh node, fresh PVC):
1. cargo resolves Cargo.lock ~10s
2. cargo refreshes registry index ~30s
3. cargo compiles ~1500 dep crates ~3-5 min
4. cargo compiles workspace members ~1-2 min
Total cold compile: ~5-8 min
After this commit:
1. seed-deps-cache initContainer pull image ~30-60s (in-cluster registry)
2. rsync /cargo-target-prebuilt/ into PVC ~30-60s (~6-8 GB local)
3. cargo delta-compiles workspace members ~1-2 min
Total cold compile: ~2-4 min
Net saving: 2-4 min per cold-cache compile pod.
Steady-state (warm PVC, stamp file present): the seed initContainer
exits in ~50ms, no rsync.
Files:
- infra/docker/Dockerfile.ci-deps-cache:
Multi-stage. Stage 1 atop ci-builder-cpu, runs
`cargo build --locked --release --workspace || true` (|| true so
transient errors don't kill the cache build), then snapshots
target/release/{deps,.fingerprint,build} into /cargo-target-prebuilt.
Stage 2 thin Ubuntu + rsync, COPY's the snapshot. Default CMD just
prints the contents — actual entrypoint set by initContainer spec.
Used the "build everything" approach over cargo-chef-style stubbing
because it has zero special-case logic for build.rs / examples /
weird workspace shapes; image is ~1-2 GB bigger as a result, but
pulls fast from in-cluster registry.
- infra/k8s/argo/refresh-deps-cache-template.yaml:
WorkflowTemplate that drives a Kaniko build of the Dockerfile and
pushes to gitlab-registry/.../ci-builder-cpu-with-deps:nightly.
Includes a CronWorkflow that runs nightly at 03:00 UTC, suspended
by default (enable manually after first successful run validation).
Mirrors the existing build-ci-image-template.yaml pattern.
- infra/k8s/argo/compile-and-deploy-template.yaml:
Adds a `seed-deps-cache` initContainer to compile-services. Uses
rsync --ignore-existing (don't clobber newer artifacts the PVC may
already have from a prior build) and a stamp file
/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp to short-circuit
re-seeding on warm pods. Tolerates a missing
/cargo-target-prebuilt dir (image not yet published) — emits a
WARN and continues without the seed.
- infra/k8s/argo/kustomization.yaml:
Registers the new template so kustomize/argo deploys it.
- infra/k8s/argo/README-deps-cache.md:
How to refresh manually, enable the cron, bump DEPS_VERSION,
debug a slow seed, and remove this when no longer needed.
Validation:
- python3 yaml.safe_load_all parses both compile-and-deploy and
refresh-deps-cache templates (1 + 2 docs respectively)
- cargo check --workspace --release --locked passes locally
- Dockerfile syntax visually inspected; the multi-stage COPY pattern
matches existing infra/docker/Dockerfile.* conventions
- Argo template structure mirrors build-ci-image-template.yaml which
is known-working in this cluster
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dbae70e811