# Binary Cache PVC Consolidation Design

## Goal

Replace 8 per-service binary-cache PVCs + S3 round-trip with a single shared PVC written by CI via `kubectl cp`.

## Current Architecture

CI compile → rclone upload to MinIO S3 → 8 service pods each with initContainer + rclone S3 download + per-service 1Gi PVC cache → run binary.

**Problems**: 8 initContainers, 8 PVCs, 8 S3 downloads, rclone CA cert management in every pod, PVC migration headaches on node pool changes.

## Proposed Architecture

CI compile → `kubectl cp` to temporary writer pod → single shared `foxhunt-binaries` PVC (2Gi) → all services mount read-only.

### Shared PVC

- **Name**: `foxhunt-binaries`
- **Size**: 2Gi (8 binaries × ~200MB stripped = ~1.6GB)
- **Access**: ReadWriteOnce (`scw-bssd`) — all services on same foxhunt node can mount RO
- **Location**: foxhunt node pool

### Deploy Flow

1. CI compiles all 8 service binaries on ci-compile-cpu (unchanged)
2. CI creates temporary `binary-writer` pod on foxhunt node with PVC mounted RW
3. CI runs `kubectl cp build-out/services/ foxhunt/binary-writer:/binaries/`
4. CI deletes writer pod
5. CI runs `kubectl rollout restart` for all service deployments
6. Services start, read binary directly from `/binaries/` mount — no initContainer

### Per-Service YAML Changes

**Remove**: initContainer (fetch-binary), binary-cache volume, minio-ca volume, binaries emptyDir
**Add**: `foxhunt-binaries` PVC volume mounted at `/binaries/` readOnly

### CI Pipeline Changes

**Remove**: rclone S3 upload of service binaries, MANIFEST.json
**Add**: kubectl run writer pod → kubectl cp → kubectl delete pod
**Keep**: rclone upload for training binaries + models (different nodes, GPU pipeline)

### Training Pipeline

Unchanged. Training jobs run on GPU nodes (L40S), not foxhunt. They continue using S3 for binary fetch and model upload.

### Rollback

Copy older binaries via `kubectl cp`, then rollout restart. CI artifacts preserve binaries by commit SHA if needed.

### Cleanup

- Delete 8 per-service PVCs (`binary-cache-*`)
- Delete `infra/k8s/storage/binary-cache-pvcs.yaml`
- Remove S3 service binary upload from `.gitlab-ci.yml`
- Remove minio-credentials and minio-ca-cert references from service pods

## Cost Impact

- Remove: 8 × 1Gi PVCs (€0.16/mo)
- Add: 1 × 2Gi PVC (€0.04/mo)
- Net: -€0.12/mo (negligible, but complexity reduction is significant)