64 lines
2.4 KiB
Markdown
64 lines
2.4 KiB
Markdown
# Binary Cache PVC Consolidation Design
|
||
|
||
## Goal
|
||
|
||
Replace 8 per-service binary-cache PVCs + S3 round-trip with a single shared PVC written by CI via `kubectl cp`.
|
||
|
||
## Current Architecture
|
||
|
||
CI compile → rclone upload to MinIO S3 → 8 service pods each with initContainer + rclone S3 download + per-service 1Gi PVC cache → run binary.
|
||
|
||
**Problems**: 8 initContainers, 8 PVCs, 8 S3 downloads, rclone CA cert management in every pod, PVC migration headaches on node pool changes.
|
||
|
||
## Proposed Architecture
|
||
|
||
CI compile → `kubectl cp` to temporary writer pod → single shared `foxhunt-binaries` PVC (2Gi) → all services mount read-only.
|
||
|
||
### Shared PVC
|
||
|
||
- **Name**: `foxhunt-binaries`
|
||
- **Size**: 2Gi (8 binaries × ~200MB stripped = ~1.6GB)
|
||
- **Access**: ReadWriteOnce (`scw-bssd`) — all services on same foxhunt node can mount RO
|
||
- **Location**: foxhunt node pool
|
||
|
||
### Deploy Flow
|
||
|
||
1. CI compiles all 8 service binaries on ci-compile-cpu (unchanged)
|
||
2. CI creates temporary `binary-writer` pod on foxhunt node with PVC mounted RW
|
||
3. CI runs `kubectl cp build-out/services/ foxhunt/binary-writer:/binaries/`
|
||
4. CI deletes writer pod
|
||
5. CI runs `kubectl rollout restart` for all service deployments
|
||
6. Services start, read binary directly from `/binaries/` mount — no initContainer
|
||
|
||
### Per-Service YAML Changes
|
||
|
||
**Remove**: initContainer (fetch-binary), binary-cache volume, minio-ca volume, binaries emptyDir
|
||
**Add**: `foxhunt-binaries` PVC volume mounted at `/binaries/` readOnly
|
||
|
||
### CI Pipeline Changes
|
||
|
||
**Remove**: rclone S3 upload of service binaries, MANIFEST.json
|
||
**Add**: kubectl run writer pod → kubectl cp → kubectl delete pod
|
||
**Keep**: rclone upload for training binaries + models (different nodes, GPU pipeline)
|
||
|
||
### Training Pipeline
|
||
|
||
Unchanged. Training jobs run on GPU nodes (L40S), not foxhunt. They continue using S3 for binary fetch and model upload.
|
||
|
||
### Rollback
|
||
|
||
Copy older binaries via `kubectl cp`, then rollout restart. CI artifacts preserve binaries by commit SHA if needed.
|
||
|
||
### Cleanup
|
||
|
||
- Delete 8 per-service PVCs (`binary-cache-*`)
|
||
- Delete `infra/k8s/storage/binary-cache-pvcs.yaml`
|
||
- Remove S3 service binary upload from `.gitlab-ci.yml`
|
||
- Remove minio-credentials and minio-ca-cert references from service pods
|
||
|
||
## Cost Impact
|
||
|
||
- Remove: 8 × 1Gi PVCs (€0.16/mo)
|
||
- Add: 1 × 2Gi PVC (€0.04/mo)
|
||
- Net: -€0.12/mo (negligible, but complexity reduction is significant)
|