foxhunt/docs/plans/2026-03-02-binary-cache-consolidation-design.md

Binary Cache PVC Consolidation Design

Goal

Replace 8 per-service binary-cache PVCs + S3 round-trip with a single shared PVC written by CI via kubectl cp.

Current Architecture

CI compile → rclone upload to MinIO S3 → 8 service pods each with initContainer + rclone S3 download + per-service 1Gi PVC cache → run binary.

Problems: 8 initContainers, 8 PVCs, 8 S3 downloads, rclone CA cert management in every pod, PVC migration headaches on node pool changes.

Proposed Architecture

CI compile → kubectl cp to temporary writer pod → single shared foxhunt-binaries PVC (2Gi) → all services mount read-only.

Shared PVC

• Name: foxhunt-binaries
• Size: 2Gi (8 binaries × ~200MB stripped = ~1.6GB)
• Access: ReadWriteOnce (scw-bssd) — all services on same foxhunt node can mount RO
• Location: foxhunt node pool

Deploy Flow

1. CI compiles all 8 service binaries on ci-compile-cpu (unchanged)
2. CI creates temporary binary-writer pod on foxhunt node with PVC mounted RW
3. CI runs kubectl cp build-out/services/ foxhunt/binary-writer:/binaries/
4. CI deletes writer pod
5. CI runs kubectl rollout restart for all service deployments
6. Services start, read binary directly from /binaries/ mount — no initContainer

Per-Service YAML Changes

Remove: initContainer (fetch-binary), binary-cache volume, minio-ca volume, binaries emptyDir Add: foxhunt-binaries PVC volume mounted at /binaries/ readOnly

CI Pipeline Changes

Remove: rclone S3 upload of service binaries, MANIFEST.json Add: kubectl run writer pod → kubectl cp → kubectl delete pod Keep: rclone upload for training binaries + models (different nodes, GPU pipeline)

Training Pipeline

Unchanged. Training jobs run on GPU nodes (L40S), not foxhunt. They continue using S3 for binary fetch and model upload.

Rollback

Copy older binaries via kubectl cp, then rollout restart. CI artifacts preserve binaries by commit SHA if needed.

Cleanup

• Delete 8 per-service PVCs (binary-cache-*)
• Delete infra/k8s/storage/binary-cache-pvcs.yaml
• Remove S3 service binary upload from .gitlab-ci.yml
• Remove minio-credentials and minio-ca-cert references from service pods

Cost Impact

• Remove: 8 × 1Gi PVCs (€0.16/mo)
• Add: 1 × 2Gi PVC (€0.04/mo)
• Net: -€0.12/mo (negligible, but complexity reduction is significant)