# Deps Cache Image — Cold-start Compile Acceleration

This README documents the **`ci-builder-cpu-with-deps:nightly`** image
and the Argo plumbing that uses it. Created as part of the CI cold-start
optimization sweep (April 2026).

## What it is

A Docker image that carries a pre-built `target/` directory at
`/cargo-target-prebuilt/` containing the workspace's third-party
dependency rlibs. It's pulled by the `compile-services` step of
`compile-and-deploy-template.yaml` as an **initContainer** that rsyncs
its payload into the `cargo-target-cpu` PVC.

## Why it exists

On a fresh node (autoscaler scaled up, fresh PVC, or after a PVC purge),
`cargo build --release` would otherwise spend 3-5 minutes compiling
~1500 third-party crates before touching workspace member code. With
the deps cache image, the cold-start compile path is:

1. **initContainer pulls image** from in-cluster registry (~30-60s)
2. **rsync prebuilt rlibs into PVC** (~30-60s, ~6-8 GB transfer over local FS)
3. **cargo build delta-compiles only workspace member code** (~1-2 min)

Net cold-start saving: **2-4 min per compile pod**.

## Files

| File | Role |
|------|------|
| `infra/docker/Dockerfile.ci-deps-cache` | Builds the deps-cache image (multi-stage). Stage 1 atop ci-builder-cpu, runs `cargo build --release --workspace`. Stage 2 thin Ubuntu + rsync layer carrying just `/cargo-target-prebuilt/`. |
| `infra/k8s/argo/refresh-deps-cache-template.yaml` | WorkflowTemplate that drives a Kaniko build of the Dockerfile + push to GitLab registry. Includes a CronWorkflow that runs nightly at 03:00 UTC (suspended by default). |
| `infra/k8s/argo/compile-and-deploy-template.yaml` | The `compile-services` step has a `seed-deps-cache` initContainer that rsyncs `/cargo-target-prebuilt/` into the PVC, gated by a stamp file. |

## How to refresh manually

```bash
# Trigger a one-shot rebuild against current main HEAD.
argo submit --from workflowtemplate/refresh-deps-cache -n foxhunt
```

To target a specific commit or bump the cache version (forces re-seed
on every compile pod):

```bash
argo submit --from workflowtemplate/refresh-deps-cache -n foxhunt \
  -p commit-sha=$(git rev-parse main) \
  -p deps-version=2 \
  -p image-tag=nightly
```

## Enabling the nightly cron

The CronWorkflow ships **suspended** so you can validate manually first.
To enable:

```bash
kubectl -n foxhunt patch cronworkflow refresh-deps-cache-nightly \
  --type=merge -p '{"spec":{"suspend":false}}'
```

To disable (return to manual-only):

```bash
kubectl -n foxhunt patch cronworkflow refresh-deps-cache-nightly \
  --type=merge -p '{"spec":{"suspend":true}}'
```

## Bumping `DEPS_VERSION`

The compile-services initContainer skips re-seeding when
`/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp` exists on the PVC.
Bump `DEPS_VERSION` to force every running compile pod to re-seed
its PVC from a fresh image. Bump in two places (must match):

1. `refresh-deps-cache-template.yaml` workflow `deps-version` parameter
   default value
2. `compile-and-deploy-template.yaml` initContainer `DEPS_VERSION` env var

Bump when:
- Cargo.lock has churned significantly (most prebuilt rlibs
  fingerprint-mismatch cargo's freshness check)
- The Rust toolchain version changes (e.g. 1.89 -> 1.90)
- A workspace member is renamed or split (rare)

You do **not** need to bump for routine commits — cargo's own
fingerprint check will discard stale rlibs and rebuild as needed.

## Debugging

**Symptom: compile-services is still slow on a fresh PVC.**

Check the seed-deps-cache initContainer logs first:

```bash
argo logs -n foxhunt <workflow-name> -c seed-deps-cache
```

Look for:
- `WARN: /cargo-target-prebuilt missing in image` — the image was built
  but the snapshot dir is empty. The inner `cargo build --workspace`
  failed in the deps-cache image build. Check the
  `refresh-deps-cache` workflow logs.
- `PVC already seeded for deps v1 (stamp present), skipping.` —
  expected steady-state on warm pods.
- `=== Seeding cargo-target-cpu PVC from prebuilt deps cache ===` +
  `=== Seed complete ===` — first run on this PVC, working as designed.

**Symptom: cargo recompiles many third-party deps anyway despite the
seed completing.**

Cargo's fingerprint check is sensitive to: rustc version, the entire
target.rustflags list, `RUSTFLAGS` env var, source code mtimes (on
build.rs), feature flags. The commit-SHA used to build the deps cache
image must match the live compile workflow's flag set. The most common
cause of full-rebuild is a `RUSTFLAGS` mismatch between the deps-cache
image build and the compile pod (e.g. mold vs wild linker swap).

To diagnose, look at the cargo build output: lines like
`Compiling tokio v1.x.x` for crates that were obviously in the cache
indicate fingerprint mismatch.

**Symptom: PVC fills up faster than before.**

Each seed adds ~6-8 GB to the PVC. The compile-services step has a
prune-at-25GB guard that wipes `release/` and `debug/` if usage
exceeds 25GB — when that fires, the next run re-seeds (stamp file is
in PVC root, survives the prune). To raise the threshold, edit the
`TARGET_SIZE_MB -gt 25000` check in compile-and-deploy-template.yaml.

## Assumptions that break the cache

- **rustc toolchain version mismatch** between deps-cache image and
  compile pod — cargo will full-rebuild. Resolution: rebuild the
  deps-cache image (refresh-deps-cache).
- **Different `RUSTFLAGS` / `target.<triple>.rustflags`** between
  image build and live compile (e.g. switching CPU target features)
  — full rebuild. Resolution: rebuild deps-cache image after the
  rustflags change is committed.
- **Cargo.lock SHA divergence** > a few weeks old — most rlibs no
  longer match. Resolution: rebuild deps-cache image (refresh nightly,
  or trigger manually).
- **Linker swap (e.g. mold -> wild)** — link-output is cached at the
  fingerprint level; changing linker doesn't trigger rebuild but may
  produce different final binaries. Not a cache invalidation, just a
  link-time difference.

## When to delete this entirely

If the underlying compile time drops to <30s in cold-start (e.g. via a
much faster registry or sccache covering 100% of deps), the
deps-cache complexity is no longer worth maintaining. To remove:

1. Remove the `seed-deps-cache` initContainer block from
   `compile-and-deploy-template.yaml` (the compile script tolerates
   an unseeded PVC — cargo just compiles deps from scratch).
2. Delete `infra/docker/Dockerfile.ci-deps-cache` and
   `infra/k8s/argo/refresh-deps-cache-template.yaml`.
3. Delete the published image:
   `crane delete gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:nightly`.
