From d35f4e0565509c0abc8d1212e8b80183b02c5249 Mon Sep 17 00:00:00 2001
From: jgrusewski <jgrusewski@localhost>
Date: Mon, 2 Mar 2026 10:58:33 +0100
Subject: [PATCH] =?UTF-8?q?docs:=20binary=20cache=20PVC=20consolidation=20?=
 =?UTF-8?q?design=20(8=20PVCs=20=E2=86=92=201=20shared)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 ...03-02-binary-cache-consolidation-design.md | 63 +++++++++++++++++++
 1 file changed, 63 insertions(+)
 create mode 100644 docs/plans/2026-03-02-binary-cache-consolidation-design.md

diff --git a/docs/plans/2026-03-02-binary-cache-consolidation-design.md b/docs/plans/2026-03-02-binary-cache-consolidation-design.md
new file mode 100644
index 000000000..5528b554b
--- /dev/null
+++ b/docs/plans/2026-03-02-binary-cache-consolidation-design.md
@@ -0,0 +1,63 @@
+# Binary Cache PVC Consolidation Design
+
+## Goal
+
+Replace 8 per-service binary-cache PVCs + S3 round-trip with a single shared PVC written by CI via `kubectl cp`.
+
+## Current Architecture
+
+CI compile → rclone upload to MinIO S3 → 8 service pods each with initContainer + rclone S3 download + per-service 1Gi PVC cache → run binary.
+
+**Problems**: 8 initContainers, 8 PVCs, 8 S3 downloads, rclone CA cert management in every pod, PVC migration headaches on node pool changes.
+
+## Proposed Architecture
+
+CI compile → `kubectl cp` to temporary writer pod → single shared `foxhunt-binaries` PVC (2Gi) → all services mount read-only.
+
+### Shared PVC
+
+- **Name**: `foxhunt-binaries`
+- **Size**: 2Gi (8 binaries × ~200MB stripped = ~1.6GB)
+- **Access**: ReadWriteOnce (`scw-bssd`) — all services on same foxhunt node can mount RO
+- **Location**: foxhunt node pool
+
+### Deploy Flow
+
+1. CI compiles all 8 service binaries on ci-compile-cpu (unchanged)
+2. CI creates temporary `binary-writer` pod on foxhunt node with PVC mounted RW
+3. CI runs `kubectl cp build-out/services/ foxhunt/binary-writer:/binaries/`
+4. CI deletes writer pod
+5. CI runs `kubectl rollout restart` for all service deployments
+6. Services start, read binary directly from `/binaries/` mount — no initContainer
+
+### Per-Service YAML Changes
+
+**Remove**: initContainer (fetch-binary), binary-cache volume, minio-ca volume, binaries emptyDir
+**Add**: `foxhunt-binaries` PVC volume mounted at `/binaries/` readOnly
+
+### CI Pipeline Changes
+
+**Remove**: rclone S3 upload of service binaries, MANIFEST.json
+**Add**: kubectl run writer pod → kubectl cp → kubectl delete pod
+**Keep**: rclone upload for training binaries + models (different nodes, GPU pipeline)
+
+### Training Pipeline
+
+Unchanged. Training jobs run on GPU nodes (L40S), not foxhunt. They continue using S3 for binary fetch and model upload.
+
+### Rollback
+
+Copy older binaries via `kubectl cp`, then rollout restart. CI artifacts preserve binaries by commit SHA if needed.
+
+### Cleanup
+
+- Delete 8 per-service PVCs (`binary-cache-*`)
+- Delete `infra/k8s/storage/binary-cache-pvcs.yaml`
+- Remove S3 service binary upload from `.gitlab-ci.yml`
+- Remove minio-credentials and minio-ca-cert references from service pods
+
+## Cost Impact
+
+- Remove: 8 × 1Gi PVCs (€0.16/mo)
+- Add: 1 × 2Gi PVC (€0.04/mo)
+- Net: -€0.12/mo (negligible, but complexity reduction is significant)