diff --git a/docs/plans/2026-03-02-binary-cache-consolidation-implementation.md b/docs/plans/2026-03-02-binary-cache-consolidation-implementation.md new file mode 100644 index 000000000..7d7d9f411 --- /dev/null +++ b/docs/plans/2026-03-02-binary-cache-consolidation-implementation.md @@ -0,0 +1,715 @@ +# Binary Cache PVC Consolidation Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Replace 8 per-service binary-cache PVCs + S3 round-trip with a single shared `foxhunt-binaries` PVC, eliminating initContainers, S3 uploads, and 7 unnecessary PVCs. + +**Architecture:** CI compile jobs produce artifacts. Deploy job creates a temporary writer pod on the foxhunt node, `kubectl cp`s binaries + static assets onto the shared PVC, deletes the writer pod, then rollout-restarts all services. Services mount the shared PVC read-only at `/binaries/` — no initContainer, no S3. + +**Tech Stack:** Kubernetes PVC (scw-bssd RWO), GitLab CI artifacts, kubectl cp, busybox writer pod + +**Design doc:** `docs/plans/2026-03-02-binary-cache-consolidation-design.md` + +--- + +## Reference: Services and Binary Names + +| Service YAML | Binary Name | PVC to Delete | +|---|---|---| +| `api-gateway.yaml` | `api_gateway` | `binary-cache-api-gateway` | +| `trading-service.yaml` | `trading_service` | `binary-cache-trading-service` | +| `broker-gateway.yaml` | `broker_gateway_service` | `binary-cache-broker-gateway` | +| `ml-training-service.yaml` | `ml_training_service` | `binary-cache-ml-training-service` | +| `backtesting-service.yaml` | `backtesting_service` | `binary-cache-backtesting-service` | +| `trading-agent-service.yaml` | `trading_agent_service` | `binary-cache-trading-agent-service` | +| `data-acquisition-service.yaml` | `data_acquisition_service` | `binary-cache-data-acquisition-service` | +| `web-gateway.yaml` | `web-gateway` | `binary-cache-web-gateway` | + +**Not modified:** `ib-gateway.yaml` (third-party Docker image), `*-gpu.yaml` overlays (different node pools, keep S3) + +**Special cases:** +- `ml-training-service.yaml`: keeps `minio-ca` volume in the container (S3 for model storage, not binary fetch) +- `web-gateway.yaml`: initContainer also fetched static assets from S3 — these go into shared PVC at `/binaries/static/` + +--- + +### Task 1: Create shared PVC YAML + +**Files:** +- Create: `infra/k8s/storage/foxhunt-binaries-pvc.yaml` + +**Step 1: Create the PVC manifest** + +```yaml +# Shared binary PVC — all foxhunt services mount read-only +# CI deploy job writes via temporary binary-writer pod (kubectl cp) +# RWO on scw-bssd: all pods on the same foxhunt node can mount +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: foxhunt-binaries + namespace: foxhunt + labels: + app.kubernetes.io/name: foxhunt-binaries + app.kubernetes.io/part-of: foxhunt +spec: + accessModes: [ReadWriteOnce] + storageClassName: scw-bssd + resources: + requests: + storage: 2Gi +``` + +**Step 2: Verify YAML is valid** + +Run: `kubectl apply --dry-run=client -f infra/k8s/storage/foxhunt-binaries-pvc.yaml` +Expected: `persistentvolumeclaim/foxhunt-binaries created (dry run)` + +**Step 3: Commit** + +```bash +git add infra/k8s/storage/foxhunt-binaries-pvc.yaml +git commit -m "feat(infra): add shared foxhunt-binaries PVC (2Gi)" +``` + +--- + +### Task 2: Modify compile-services CI job + +**Files:** +- Modify: `.gitlab-ci.yml` (lines 409-458, `compile-services` job) + +**Step 1: Add CI artifacts, remove S3 upload** + +In the `compile-services` job, replace the S3 upload section (lines 445-458) with CI artifacts: + +Remove this block (lines 445-458): +```yaml + # Upload stripped binaries to MinIO (in-cluster S3) + # Trust MinIO CA — rclone (Go) uses system CAs, not RCLONE_CA_CERT for S3. + # Container runs non-root, so merge system + MinIO CA into writable /tmp bundle. + - cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-bundle.crt 2>/dev/null || true + - cat /etc/ssl/minio-ca/ca.crt >> /tmp/ca-bundle.crt 2>/dev/null || true + - export SSL_CERT_FILE=/tmp/ca-bundle.crt + - export RCLONE_S3_PROVIDER=Minio RCLONE_S3_ENDPOINT=https://minio.foxhunt.svc.cluster.local:9000 RCLONE_S3_REGION=us-east-1 + - export RCLONE_S3_ACCESS_KEY_ID=$MINIO_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY + - | + for bin in trading_service api_gateway broker_gateway_service ml_training_service backtesting_service trading_agent_service data_acquisition_service web-gateway; do + rclone copyto build-out/services/$bin :s3:foxhunt-binaries/latest/services/$bin + done + - rclone copy build-out/services/ :s3:foxhunt-binaries/archive/${CI_COMMIT_SHA}/services/ + - echo "Service binaries uploaded to MinIO" +``` + +Replace with: +```yaml + - echo "Service binaries ready as CI artifacts" +``` + +Add artifacts section at the end of the job: +```yaml + artifacts: + paths: + - build-out/services/ + expire_in: 1 day +``` + +**Step 2: Verify CI syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"` +Expected: `Valid YAML` + +**Step 3: Commit** + +```bash +git add .gitlab-ci.yml +git commit -m "feat(ci): replace S3 upload with CI artifacts in compile-services" +``` + +--- + +### Task 3: Modify build-web-dashboard CI job + +**Files:** +- Modify: `.gitlab-ci.yml` (lines 529-563, `build-web-dashboard` job) + +**Step 1: Add CI artifacts, remove S3 upload** + +In the `build-web-dashboard` job, remove the rclone/S3 section (lines 547-563) and add CI artifacts. + +Remove the `before_script` block (lines 548-550, rclone install): +```yaml + before_script: + - apt-get update && apt-get install -y --no-install-recommends curl unzip ca-certificates + - curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip + - unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ && rm /tmp/rclone.zip +``` + +In the `script` section, remove the rclone upload lines (555-563): +```yaml + # Trust MinIO CA — rclone (Go) uses system CAs, not RCLONE_CA_CERT for S3. + # Container runs non-root, so merge system + MinIO CA into writable /tmp bundle. + - cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-bundle.crt 2>/dev/null || true + - cat /etc/ssl/minio-ca/ca.crt >> /tmp/ca-bundle.crt 2>/dev/null || true + - export SSL_CERT_FILE=/tmp/ca-bundle.crt + - export RCLONE_S3_PROVIDER=Minio RCLONE_S3_ENDPOINT=https://minio.foxhunt.svc.cluster.local:9000 RCLONE_S3_REGION=us-east-1 + - export RCLONE_S3_ACCESS_KEY_ID=$MINIO_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY + - rclone sync dist/ :s3:foxhunt-binaries/latest/web-dashboard/dist/ + - echo "Web dashboard assets uploaded to MinIO" +``` + +Replace with: +```yaml + - echo "Web dashboard assets ready as CI artifacts" +``` + +Add artifacts section: +```yaml + artifacts: + paths: + - web-dashboard/dist/ + expire_in: 1 day +``` + +**Step 2: Verify CI syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"` +Expected: `Valid YAML` + +**Step 3: Commit** + +```bash +git add .gitlab-ci.yml +git commit -m "feat(ci): replace S3 upload with CI artifacts in build-web-dashboard" +``` + +--- + +### Task 4: Remove write-manifest CI job + +**Files:** +- Modify: `.gitlab-ci.yml` (lines 565-597, `write-manifest` job) + +**Step 1: Delete the entire write-manifest job** + +Remove lines 565-597 (the entire `write-manifest` job block including the section header comment). MANIFEST.json was S3-specific tracking — no longer needed. + +**Step 2: Verify CI syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"` +Expected: `Valid YAML` + +**Step 3: Commit** + +```bash +git add .gitlab-ci.yml +git commit -m "chore(ci): remove write-manifest job (S3 tracking no longer needed)" +``` + +--- + +### Task 5: Rewrite deploy CI job + +**Files:** +- Modify: `.gitlab-ci.yml` (deploy job, currently starting around line 1460) + +**Step 1: Update job dependencies** + +Change the `needs` section from: +```yaml + needs: + - job: write-manifest + optional: true +``` + +To: +```yaml + needs: + - job: compile-services + optional: true + artifacts: true + - job: build-web-dashboard + optional: true + artifacts: true +``` + +**Step 2: Replace service deploy section** + +Find the section that applies binary-cache-pvcs.yaml and does rollout restarts (around line 1595-1613). Replace: + +```yaml + # Apply service deployments and binary cache PVCs (idempotent — picks up any YAML changes) + # Skip training/job-template.yaml (placeholder names, not directly applicable) + # Skip storage/training-data-pv.yaml (PVC spec immutable after creation) + - kubectl apply -f infra/k8s/services/ -f infra/k8s/storage/binary-cache-pvcs.yaml + # Rolling restart triggers initContainers to fetch latest binaries from MinIO + - | + for svc in trading-service api-gateway broker-gateway \ + ml-training-service backtesting-service \ + trading-agent-service data-acquisition-service web-gateway; do + kubectl -n foxhunt rollout restart deployment/$svc + echo "Restarted $svc" + done + - | + for svc in trading-service api-gateway broker-gateway \ + ml-training-service backtesting-service \ + trading-agent-service data-acquisition-service web-gateway; do + kubectl -n foxhunt rollout status deployment/$svc --timeout=300s + done + - echo "All services deployed with MinIO binaries" +``` + +With: + +```yaml + # Apply shared binaries PVC (idempotent) + - kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml + # Apply service deployments (picks up any YAML changes) + # Skip training/job-template.yaml (placeholder names, not directly applicable) + # Skip storage/training-data-pv.yaml (PVC spec immutable after creation) + - kubectl apply -f infra/k8s/services/ + # Copy new binaries to shared PVC (if compile ran) + - | + if [ -d "build-out/services" ]; then + echo "Deploying new binaries via kubectl cp..." + # Create temporary writer pod on foxhunt node + kubectl apply -f - <<'WRITER_EOF' + apiVersion: v1 + kind: Pod + metadata: + name: binary-writer + namespace: foxhunt + labels: + app.kubernetes.io/name: binary-writer + spec: + nodeSelector: + k8s.scaleway.com/pool-name: foxhunt + securityContext: + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + containers: + - name: writer + image: busybox:1.36 + command: ["sleep", "600"] + volumeMounts: + - name: binaries + mountPath: /binaries + volumes: + - name: binaries + persistentVolumeClaim: + claimName: foxhunt-binaries + restartPolicy: Never + WRITER_EOF + kubectl wait --for=condition=Ready pod/binary-writer -n foxhunt --timeout=120s + # Copy service binaries + for bin in trading_service api_gateway broker_gateway_service \ + ml_training_service backtesting_service \ + trading_agent_service data_acquisition_service web-gateway; do + kubectl cp build-out/services/$bin foxhunt/binary-writer:/binaries/$bin + echo " Copied $bin ($(stat -c%s build-out/services/$bin) bytes)" + done + # Copy web dashboard static assets (if build-web-dashboard ran) + if [ -d "web-dashboard/dist" ]; then + kubectl exec -n foxhunt binary-writer -- mkdir -p /binaries/static + tar -C web-dashboard/dist -cf - . | kubectl exec -i -n foxhunt binary-writer -- tar -C /binaries/static -xf - + echo " Copied web-dashboard static assets" + fi + # Make binaries executable + for bin in trading_service api_gateway broker_gateway_service \ + ml_training_service backtesting_service \ + trading_agent_service data_acquisition_service web-gateway; do + kubectl exec -n foxhunt binary-writer -- chmod +x /binaries/$bin + done + # Cleanup writer pod + kubectl delete pod binary-writer -n foxhunt --wait=false + echo "Binaries deployed to shared PVC" + else + echo "No new binaries — compile-services did not run, PVC has existing binaries" + fi + # Rolling restart all services (they read from shared PVC, no initContainer needed) + - | + for svc in trading-service api-gateway broker-gateway \ + ml-training-service backtesting-service \ + trading-agent-service data-acquisition-service web-gateway; do + kubectl -n foxhunt rollout restart deployment/$svc + echo "Restarted $svc" + done + - | + for svc in trading-service api-gateway broker-gateway \ + ml-training-service backtesting-service \ + trading-agent-service data-acquisition-service web-gateway; do + kubectl -n foxhunt rollout status deployment/$svc --timeout=300s + done + - echo "All services deployed" +``` + +Also update the comment at line 1507: +```yaml + # S3 binary share: services use generic runtime image + initContainer to fetch binaries from S3. + # Deploy = apply K8s manifests (picks up YAML changes) + rollout restart (fetch new binaries). +``` +Change to: +```yaml + # Shared PVC binary deploy: CI copies binaries to foxhunt-binaries PVC via writer pod. + # Services mount the PVC read-only — no initContainer, no S3. +``` + +**Step 3: Verify CI syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"` +Expected: `Valid YAML` + +**Step 4: Commit** + +```bash +git add .gitlab-ci.yml +git commit -m "feat(ci): deploy binaries via kubectl cp to shared PVC instead of S3" +``` + +--- + +### Task 6: Modify all 8 service YAMLs + +**Files:** +- Modify: `infra/k8s/services/api-gateway.yaml` +- Modify: `infra/k8s/services/trading-service.yaml` +- Modify: `infra/k8s/services/broker-gateway.yaml` +- Modify: `infra/k8s/services/ml-training-service.yaml` +- Modify: `infra/k8s/services/backtesting-service.yaml` +- Modify: `infra/k8s/services/trading-agent-service.yaml` +- Modify: `infra/k8s/services/data-acquisition-service.yaml` +- Modify: `infra/k8s/services/web-gateway.yaml` + +**Step 1: For each of the 8 services, apply these changes:** + +**A) Remove the entire `initContainers` section** (the `fetch-binary` initContainer with rclone S3 download, ~57 lines per service) + +**B) Replace the `volumes` section.** Remove `binary-cache` (per-service PVC) and `minio-ca` (ConfigMap). Change `binaries` from `emptyDir` to shared PVC. + +Before (standard service — api-gateway, trading-service, broker-gateway, backtesting-service, trading-agent-service, data-acquisition-service): +```yaml + volumes: + - name: binaries + emptyDir: + sizeLimit: 200Mi + - name: binary-cache + persistentVolumeClaim: + claimName: binary-cache- + - name: minio-ca + configMap: + name: minio-ca-cert + - name: tmp + emptyDir: + sizeLimit: 50Mi +``` + +After: +```yaml + volumes: + - name: binaries + persistentVolumeClaim: + claimName: foxhunt-binaries + readOnly: true + - name: tmp + emptyDir: + sizeLimit: 50Mi +``` + +**Special case — ml-training-service:** Keep `minio-ca` (used by container for S3 model storage) and keep `tls-certs`: +```yaml + volumes: + - name: binaries + persistentVolumeClaim: + claimName: foxhunt-binaries + readOnly: true + - name: tls-certs + secret: + secretName: ml-training-tls + - name: minio-ca + configMap: + name: minio-ca-cert + - name: tmp + emptyDir: + sizeLimit: 50Mi +``` + +**Special case — web-gateway:** Same as standard (minio-ca removed — static assets are on the PVC now). + +**C) No changes to container section** — containers already mount `/binaries` readOnly and `/tmp`. The `command` field still points to `/binaries/`. + +**Step 2: Verify all YAMLs** + +Run: `for f in infra/k8s/services/*.yaml; do kubectl apply --dry-run=client -f "$f" 2>&1 | head -5; done` +Expected: Each file shows `deployment.apps/ created (dry run)` and `service/ created (dry run)` + +**Step 3: Commit** + +```bash +git add infra/k8s/services/api-gateway.yaml \ + infra/k8s/services/trading-service.yaml \ + infra/k8s/services/broker-gateway.yaml \ + infra/k8s/services/ml-training-service.yaml \ + infra/k8s/services/backtesting-service.yaml \ + infra/k8s/services/trading-agent-service.yaml \ + infra/k8s/services/data-acquisition-service.yaml \ + infra/k8s/services/web-gateway.yaml +git commit -m "feat(infra): remove initContainers, mount shared foxhunt-binaries PVC" +``` + +--- + +### Task 7: Apply shared PVC and seed initial binaries + +**Prerequisite:** Tasks 1-6 completed (code changes done locally). This task operates on the live cluster. + +**Step 1: Create the shared PVC** + +Run: `kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml` +Expected: `persistentvolumeclaim/foxhunt-binaries created` + +Verify: `kubectl get pvc foxhunt-binaries -n foxhunt` +Expected: STATUS=Bound (or Pending briefly, then Bound) + +**Step 2: Seed binaries from current running pods** + +The current pods have working binaries in their `/binaries/` emptyDir (fetched from S3). Copy them to the new shared PVC via a writer pod. + +```bash +# Create writer pod +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: binary-writer + namespace: foxhunt + labels: + app.kubernetes.io/name: binary-writer +spec: + nodeSelector: + k8s.scaleway.com/pool-name: foxhunt + securityContext: + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + containers: + - name: writer + image: busybox:1.36 + command: ["sleep", "600"] + volumeMounts: + - name: binaries + mountPath: /binaries + volumes: + - name: binaries + persistentVolumeClaim: + claimName: foxhunt-binaries + restartPolicy: Never +EOF + +# Wait for writer pod +kubectl wait --for=condition=Ready pod/binary-writer -n foxhunt --timeout=120s +``` + +**Step 3: Copy binaries from running service pods to writer pod** + +```bash +# For each service, copy the binary from the running pod to the writer pod +for svc_bin in "api-gateway:api_gateway" \ + "trading-service:trading_service" \ + "broker-gateway:broker_gateway_service" \ + "ml-training-service:ml_training_service" \ + "backtesting-service:backtesting_service" \ + "trading-agent-service:trading_agent_service" \ + "data-acquisition-service:data_acquisition_service" \ + "web-gateway:web-gateway"; do + svc="${svc_bin%%:*}" + bin="${svc_bin##*:}" + POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=$svc -o jsonpath='{.items[0].metadata.name}') + if [ -n "$POD" ]; then + kubectl cp foxhunt/$POD:/binaries/$bin /tmp/seed-$bin + kubectl cp /tmp/seed-$bin foxhunt/binary-writer:/binaries/$bin + rm /tmp/seed-$bin + echo "Seeded $bin from $POD" + else + echo "WARNING: no pod for $svc" + fi +done +``` + +**Step 4: Copy web-gateway static assets** + +```bash +WG_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=web-gateway -o jsonpath='{.items[0].metadata.name}') +kubectl exec -n foxhunt binary-writer -- mkdir -p /binaries/static +kubectl exec -n foxhunt $WG_POD -- tar -C /binaries/static -cf - . 2>/dev/null | kubectl exec -i -n foxhunt binary-writer -- tar -C /binaries/static -xf - +echo "Seeded web-gateway static assets" +``` + +**Step 5: Make binaries executable and verify** + +```bash +for bin in trading_service api_gateway broker_gateway_service \ + ml_training_service backtesting_service \ + trading_agent_service data_acquisition_service web-gateway; do + kubectl exec -n foxhunt binary-writer -- chmod +x /binaries/$bin +done + +# Verify all binaries present +kubectl exec -n foxhunt binary-writer -- ls -lh /binaries/ +kubectl exec -n foxhunt binary-writer -- ls /binaries/static/ | head -5 +``` + +Expected: 8 binaries (~150-250MB each) + `static/` directory + +**Step 6: Delete writer pod** + +Run: `kubectl delete pod binary-writer -n foxhunt` + +--- + +### Task 8: Apply modified service YAMLs and verify + +**Step 1: Apply all service YAMLs** + +Run: `kubectl apply -f infra/k8s/services/` +Expected: 8 deployments configured, services unchanged + +**Step 2: Rolling restart all services** + +```bash +for svc in trading-service api-gateway broker-gateway \ + ml-training-service backtesting-service \ + trading-agent-service data-acquisition-service web-gateway; do + kubectl -n foxhunt rollout restart deployment/$svc +done +``` + +**Step 3: Wait for rollout** + +```bash +for svc in trading-service api-gateway broker-gateway \ + ml-training-service backtesting-service \ + trading-agent-service data-acquisition-service web-gateway; do + kubectl -n foxhunt rollout status deployment/$svc --timeout=300s +done +``` + +Expected: All 8 deployments successfully rolled out + +**Step 4: Verify pods have no initContainers** + +```bash +kubectl get pods -n foxhunt -l app.kubernetes.io/part-of=foxhunt -o custom-columns='NAME:.metadata.name,INIT:.spec.initContainers[*].name,STATUS:.status.phase' | grep -v ib-gateway +``` + +Expected: INIT column is empty for all service pods (no initContainers) + +**Step 5: Verify binaries are accessible** + +```bash +# Spot-check: api-gateway can read its binary +API_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=api-gateway -o jsonpath='{.items[0].metadata.name}') +kubectl exec -n foxhunt $API_POD -- ls -l /binaries/api_gateway +``` + +Expected: Binary exists and is executable + +--- + +### Task 9: Delete old PVCs and cleanup files + +**Step 1: Delete old per-service PVCs from cluster** + +```bash +for pvc in binary-cache-trading-service binary-cache-api-gateway \ + binary-cache-broker-gateway binary-cache-ml-training-service \ + binary-cache-backtesting-service binary-cache-trading-agent-service \ + binary-cache-data-acquisition-service binary-cache-web-gateway; do + kubectl delete pvc $pvc -n foxhunt --ignore-not-found +done +``` + +Expected: 8 PVCs deleted + +**Step 2: Verify only foxhunt-binaries PVC remains for services** + +Run: `kubectl get pvc -n foxhunt -l app.kubernetes.io/name=binary-cache` +Expected: No resources found + +Run: `kubectl get pvc foxhunt-binaries -n foxhunt` +Expected: STATUS=Bound + +**Step 3: Delete old PVC manifest file** + +Run: `rm infra/k8s/storage/binary-cache-pvcs.yaml` + +**Step 4: Commit cleanup** + +```bash +git add -A infra/k8s/storage/binary-cache-pvcs.yaml +git commit -m "chore(infra): delete 8 per-service binary-cache PVCs (replaced by foxhunt-binaries)" +``` + +--- + +### Task 10: End-to-end verification + +**Step 1: Verify all pods running** + +Run: `kubectl get pods -n foxhunt -l app.kubernetes.io/part-of=foxhunt -o wide` +Expected: All pods Running, no restarts, no initContainers + +**Step 2: Verify gRPC health for key services** + +```bash +API_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=api-gateway -o jsonpath='{.items[0].metadata.name}') +kubectl exec -n foxhunt $API_POD -- grpc_health_probe -addr=localhost:50051 +``` + +Expected: `status: SERVING` + +**Step 3: Verify web-gateway serves static assets** + +```bash +WG_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=web-gateway -o jsonpath='{.items[0].metadata.name}') +kubectl exec -n foxhunt $WG_POD -- ls /binaries/static/index.html +``` + +Expected: File exists + +**Step 4: Verify PVC usage** + +Run: `kubectl get pvc -n foxhunt | grep -E 'foxhunt-binaries|binary-cache'` +Expected: Only `foxhunt-binaries` (Bound, 2Gi) + +**Step 5: Final commit with all changes** + +```bash +git add -A +git commit -m "feat(infra): consolidate 8 binary-cache PVCs into single shared foxhunt-binaries PVC + +Eliminates S3 round-trip for service binaries. CI deploys via kubectl cp +to a temporary writer pod. Services mount shared PVC read-only. +Training pipeline (GPU nodes) unchanged — still uses S3." +``` + +**Step 6: Push to GitLab** + +Run: `git push origin main` + +--- + +## Summary of Changes + +| What | Before | After | +|---|---|---| +| Binary source | S3 (MinIO) via initContainer | Shared PVC via kubectl cp | +| PVCs | 8 × 1Gi per-service | 1 × 2Gi shared | +| initContainers | 8 (fetch-binary) | 0 | +| S3 uploads (services) | compile-services + build-web-dashboard | None | +| CI artifact transfer | None | compile-services + build-web-dashboard | +| Deploy mechanism | rollout restart → initContainer fetches from S3 | kubectl cp to PVC → rollout restart | +| Training pipeline | S3 (unchanged) | S3 (unchanged) | +| GPU overlays | S3 (unchanged) | S3 (unchanged) | +| Cost | 8Gi PVCs (~€0.16/mo) | 2Gi PVC (~€0.04/mo) | diff --git a/infra/k8s/storage/binary-cache-pvcs.yaml b/infra/k8s/storage/binary-cache-pvcs.yaml deleted file mode 100644 index c12a5fad1..000000000 --- a/infra/k8s/storage/binary-cache-pvcs.yaml +++ /dev/null @@ -1,124 +0,0 @@ -# Per-service binary cache PVCs — fallback when S3 is unavailable -# Each service gets its own 1Gi PVC (ReadWriteOnce, scw-bssd) -# Scaleway minimum block volume is 1Gi — cannot request less -# Total: 8 × 1Gi = 8Gi, ~€0.16/month ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: binary-cache-trading-service - namespace: foxhunt - labels: - app.kubernetes.io/name: binary-cache - app.kubernetes.io/part-of: foxhunt -spec: - accessModes: [ReadWriteOnce] - storageClassName: scw-bssd - resources: - requests: - storage: 1Gi ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: binary-cache-api-gateway - namespace: foxhunt - labels: - app.kubernetes.io/name: binary-cache - app.kubernetes.io/part-of: foxhunt -spec: - accessModes: [ReadWriteOnce] - storageClassName: scw-bssd - resources: - requests: - storage: 1Gi ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: binary-cache-broker-gateway - namespace: foxhunt - labels: - app.kubernetes.io/name: binary-cache - app.kubernetes.io/part-of: foxhunt -spec: - accessModes: [ReadWriteOnce] - storageClassName: scw-bssd - resources: - requests: - storage: 1Gi ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: binary-cache-ml-training-service - namespace: foxhunt - labels: - app.kubernetes.io/name: binary-cache - app.kubernetes.io/part-of: foxhunt -spec: - accessModes: [ReadWriteOnce] - storageClassName: scw-bssd - resources: - requests: - storage: 1Gi ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: binary-cache-backtesting-service - namespace: foxhunt - labels: - app.kubernetes.io/name: binary-cache - app.kubernetes.io/part-of: foxhunt -spec: - accessModes: [ReadWriteOnce] - storageClassName: scw-bssd - resources: - requests: - storage: 1Gi ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: binary-cache-trading-agent-service - namespace: foxhunt - labels: - app.kubernetes.io/name: binary-cache - app.kubernetes.io/part-of: foxhunt -spec: - accessModes: [ReadWriteOnce] - storageClassName: scw-bssd - resources: - requests: - storage: 1Gi ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: binary-cache-data-acquisition-service - namespace: foxhunt - labels: - app.kubernetes.io/name: binary-cache - app.kubernetes.io/part-of: foxhunt -spec: - accessModes: [ReadWriteOnce] - storageClassName: scw-bssd - resources: - requests: - storage: 1Gi ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: binary-cache-web-gateway - namespace: foxhunt - labels: - app.kubernetes.io/name: binary-cache - app.kubernetes.io/part-of: foxhunt -spec: - accessModes: [ReadWriteOnce] - storageClassName: scw-bssd - resources: - requests: - storage: 1Gi