foxhunt/docs/plans/2026-03-02-binary-cache-consolidation-implementation.md

Binary Cache PVC Consolidation Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Replace 8 per-service binary-cache PVCs + S3 round-trip with a single shared foxhunt-binaries PVC, eliminating initContainers, S3 uploads, and 7 unnecessary PVCs.

Architecture: CI compile jobs produce artifacts. Deploy job creates a temporary writer pod on the foxhunt node, kubectl cps binaries + static assets onto the shared PVC, deletes the writer pod, then rollout-restarts all services. Services mount the shared PVC read-only at /binaries/ — no initContainer, no S3.

Tech Stack: Kubernetes PVC (scw-bssd RWO), GitLab CI artifacts, kubectl cp, busybox writer pod

Design doc: docs/plans/2026-03-02-binary-cache-consolidation-design.md

---

Reference: Services and Binary Names

Service YAML	Binary Name	PVC to Delete
api-gateway.yaml	api_gateway	binary-cache-api-gateway
trading-service.yaml	trading_service	binary-cache-trading-service
broker-gateway.yaml	broker_gateway_service	binary-cache-broker-gateway
ml-training-service.yaml	ml_training_service	binary-cache-ml-training-service
backtesting-service.yaml	backtesting_service	binary-cache-backtesting-service
trading-agent-service.yaml	trading_agent_service	binary-cache-trading-agent-service
data-acquisition-service.yaml	data_acquisition_service	binary-cache-data-acquisition-service
web-gateway.yaml	web-gateway	binary-cache-web-gateway

Not modified: ib-gateway.yaml (third-party Docker image), *-gpu.yaml overlays (different node pools, keep S3)

Special cases:

• ml-training-service.yaml: keeps minio-ca volume in the container (S3 for model storage, not binary fetch)
• web-gateway.yaml: initContainer also fetched static assets from S3 — these go into shared PVC at /binaries/static/

---

Task 1: Create shared PVC YAML

Files:

• Create: infra/k8s/storage/foxhunt-binaries-pvc.yaml

Step 1: Create the PVC manifest

# Shared binary PVC — all foxhunt services mount read-only
# CI deploy job writes via temporary binary-writer pod (kubectl cp)
# RWO on scw-bssd: all pods on the same foxhunt node can mount
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: foxhunt-binaries
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: foxhunt-binaries
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 2Gi

Step 2: Verify YAML is valid

Run: kubectl apply --dry-run=client -f infra/k8s/storage/foxhunt-binaries-pvc.yaml Expected: persistentvolumeclaim/foxhunt-binaries created (dry run)

Step 3: Commit

git add infra/k8s/storage/foxhunt-binaries-pvc.yaml
git commit -m "feat(infra): add shared foxhunt-binaries PVC (2Gi)"

---

Task 2: Modify compile-services CI job

Files:

• Modify: .gitlab-ci.yml (lines 409-458, compile-services job)

Step 1: Add CI artifacts, remove S3 upload

In the compile-services job, replace the S3 upload section (lines 445-458) with CI artifacts:

Remove this block (lines 445-458):

    # Upload stripped binaries to MinIO (in-cluster S3)
    # Trust MinIO CA — rclone (Go) uses system CAs, not RCLONE_CA_CERT for S3.
    # Container runs non-root, so merge system + MinIO CA into writable /tmp bundle.
    - cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-bundle.crt 2>/dev/null || true
    - cat /etc/ssl/minio-ca/ca.crt >> /tmp/ca-bundle.crt 2>/dev/null || true
    - export SSL_CERT_FILE=/tmp/ca-bundle.crt
    - export RCLONE_S3_PROVIDER=Minio RCLONE_S3_ENDPOINT=https://minio.foxhunt.svc.cluster.local:9000 RCLONE_S3_REGION=us-east-1
    - export RCLONE_S3_ACCESS_KEY_ID=$MINIO_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY
    - |
      for bin in trading_service api_gateway broker_gateway_service ml_training_service backtesting_service trading_agent_service data_acquisition_service web-gateway; do
        rclone copyto build-out/services/$bin :s3:foxhunt-binaries/latest/services/$bin
      done
    - rclone copy build-out/services/ :s3:foxhunt-binaries/archive/${CI_COMMIT_SHA}/services/
    - echo "Service binaries uploaded to MinIO"

Replace with:

    - echo "Service binaries ready as CI artifacts"

Add artifacts section at the end of the job:

  artifacts:
    paths:
      - build-out/services/
    expire_in: 1 day

Step 2: Verify CI syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML" Expected: Valid YAML

Step 3: Commit

git add .gitlab-ci.yml
git commit -m "feat(ci): replace S3 upload with CI artifacts in compile-services"

---

Task 3: Modify build-web-dashboard CI job

Files:

• Modify: .gitlab-ci.yml (lines 529-563, build-web-dashboard job)

Step 1: Add CI artifacts, remove S3 upload

In the build-web-dashboard job, remove the rclone/S3 section (lines 547-563) and add CI artifacts.

Remove the before_script block (lines 548-550, rclone install):

  before_script:
    - apt-get update && apt-get install -y --no-install-recommends curl unzip ca-certificates
    - curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip
    - unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ && rm /tmp/rclone.zip

In the script section, remove the rclone upload lines (555-563):

    # Trust MinIO CA — rclone (Go) uses system CAs, not RCLONE_CA_CERT for S3.
    # Container runs non-root, so merge system + MinIO CA into writable /tmp bundle.
    - cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-bundle.crt 2>/dev/null || true
    - cat /etc/ssl/minio-ca/ca.crt >> /tmp/ca-bundle.crt 2>/dev/null || true
    - export SSL_CERT_FILE=/tmp/ca-bundle.crt
    - export RCLONE_S3_PROVIDER=Minio RCLONE_S3_ENDPOINT=https://minio.foxhunt.svc.cluster.local:9000 RCLONE_S3_REGION=us-east-1
    - export RCLONE_S3_ACCESS_KEY_ID=$MINIO_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY
    - rclone sync dist/ :s3:foxhunt-binaries/latest/web-dashboard/dist/
    - echo "Web dashboard assets uploaded to MinIO"

Replace with:

    - echo "Web dashboard assets ready as CI artifacts"

Add artifacts section:

  artifacts:
    paths:
      - web-dashboard/dist/
    expire_in: 1 day

Step 2: Verify CI syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML" Expected: Valid YAML

Step 3: Commit

git add .gitlab-ci.yml
git commit -m "feat(ci): replace S3 upload with CI artifacts in build-web-dashboard"

---

Task 4: Remove write-manifest CI job

Files:

• Modify: .gitlab-ci.yml (lines 565-597, write-manifest job)

Step 1: Delete the entire write-manifest job

Remove lines 565-597 (the entire write-manifest job block including the section header comment). MANIFEST.json was S3-specific tracking — no longer needed.

Step 2: Verify CI syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML" Expected: Valid YAML

Step 3: Commit

git add .gitlab-ci.yml
git commit -m "chore(ci): remove write-manifest job (S3 tracking no longer needed)"

---

Task 5: Rewrite deploy CI job

Files:

• Modify: .gitlab-ci.yml (deploy job, currently starting around line 1460)

Step 1: Update job dependencies

Change the needs section from:

  needs:
    - job: write-manifest
      optional: true

To:

  needs:
    - job: compile-services
      optional: true
      artifacts: true
    - job: build-web-dashboard
      optional: true
      artifacts: true

Step 2: Replace service deploy section

Find the section that applies binary-cache-pvcs.yaml and does rollout restarts (around line 1595-1613). Replace:

    # Apply service deployments and binary cache PVCs (idempotent — picks up any YAML changes)
    # Skip training/job-template.yaml (placeholder names, not directly applicable)
    # Skip storage/training-data-pv.yaml (PVC spec immutable after creation)
    - kubectl apply -f infra/k8s/services/ -f infra/k8s/storage/binary-cache-pvcs.yaml
    # Rolling restart triggers initContainers to fetch latest binaries from MinIO
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout restart deployment/$svc
        echo "Restarted $svc"
      done
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
      done
    - echo "All services deployed with MinIO binaries"

With:

    # Apply shared binaries PVC (idempotent)
    - kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml
    # Apply service deployments (picks up any YAML changes)
    # Skip training/job-template.yaml (placeholder names, not directly applicable)
    # Skip storage/training-data-pv.yaml (PVC spec immutable after creation)
    - kubectl apply -f infra/k8s/services/
    # Copy new binaries to shared PVC (if compile ran)
    - |
      if [ -d "build-out/services" ]; then
        echo "Deploying new binaries via kubectl cp..."
        # Create temporary writer pod on foxhunt node
        kubectl apply -f - <<'WRITER_EOF'
      apiVersion: v1
      kind: Pod
      metadata:
        name: binary-writer
        namespace: foxhunt
        labels:
          app.kubernetes.io/name: binary-writer
      spec:
        nodeSelector:
          k8s.scaleway.com/pool-name: foxhunt
        securityContext:
          runAsUser: 1000
          runAsGroup: 1000
          fsGroup: 1000
        containers:
          - name: writer
            image: busybox:1.36
            command: ["sleep", "600"]
            volumeMounts:
              - name: binaries
                mountPath: /binaries
        volumes:
          - name: binaries
            persistentVolumeClaim:
              claimName: foxhunt-binaries
        restartPolicy: Never
      WRITER_EOF
        kubectl wait --for=condition=Ready pod/binary-writer -n foxhunt --timeout=120s
        # Copy service binaries
        for bin in trading_service api_gateway broker_gateway_service \
                   ml_training_service backtesting_service \
                   trading_agent_service data_acquisition_service web-gateway; do
          kubectl cp build-out/services/$bin foxhunt/binary-writer:/binaries/$bin
          echo "  Copied $bin ($(stat -c%s build-out/services/$bin) bytes)"
        done
        # Copy web dashboard static assets (if build-web-dashboard ran)
        if [ -d "web-dashboard/dist" ]; then
          kubectl exec -n foxhunt binary-writer -- mkdir -p /binaries/static
          tar -C web-dashboard/dist -cf - . | kubectl exec -i -n foxhunt binary-writer -- tar -C /binaries/static -xf -
          echo "  Copied web-dashboard static assets"
        fi
        # Make binaries executable
        for bin in trading_service api_gateway broker_gateway_service \
                   ml_training_service backtesting_service \
                   trading_agent_service data_acquisition_service web-gateway; do
          kubectl exec -n foxhunt binary-writer -- chmod +x /binaries/$bin
        done
        # Cleanup writer pod
        kubectl delete pod binary-writer -n foxhunt --wait=false
        echo "Binaries deployed to shared PVC"
      else
        echo "No new binaries — compile-services did not run, PVC has existing binaries"
      fi
    # Rolling restart all services (they read from shared PVC, no initContainer needed)
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout restart deployment/$svc
        echo "Restarted $svc"
      done
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
      done
    - echo "All services deployed"

Also update the comment at line 1507:

    # S3 binary share: services use generic runtime image + initContainer to fetch binaries from S3.
    # Deploy = apply K8s manifests (picks up YAML changes) + rollout restart (fetch new binaries).

Change to:

    # Shared PVC binary deploy: CI copies binaries to foxhunt-binaries PVC via writer pod.
    # Services mount the PVC read-only — no initContainer, no S3.

Step 3: Verify CI syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML" Expected: Valid YAML

Step 4: Commit

git add .gitlab-ci.yml
git commit -m "feat(ci): deploy binaries via kubectl cp to shared PVC instead of S3"

---

Task 6: Modify all 8 service YAMLs

Files:

• Modify: infra/k8s/services/api-gateway.yaml
• Modify: infra/k8s/services/trading-service.yaml
• Modify: infra/k8s/services/broker-gateway.yaml
• Modify: infra/k8s/services/ml-training-service.yaml
• Modify: infra/k8s/services/backtesting-service.yaml
• Modify: infra/k8s/services/trading-agent-service.yaml
• Modify: infra/k8s/services/data-acquisition-service.yaml
• Modify: infra/k8s/services/web-gateway.yaml

Step 1: For each of the 8 services, apply these changes:

A) Remove the entire initContainers section (the fetch-binary initContainer with rclone S3 download, ~57 lines per service)

B) Replace the volumes section. Remove binary-cache (per-service PVC) and minio-ca (ConfigMap). Change binaries from emptyDir to shared PVC.

Before (standard service — api-gateway, trading-service, broker-gateway, backtesting-service, trading-agent-service, data-acquisition-service):

      volumes:
        - name: binaries
          emptyDir:
            sizeLimit: 200Mi
        - name: binary-cache
          persistentVolumeClaim:
            claimName: binary-cache-<service-name>
        - name: minio-ca
          configMap:
            name: minio-ca-cert
        - name: tmp
          emptyDir:
            sizeLimit: 50Mi

After:

      volumes:
        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true
        - name: tmp
          emptyDir:
            sizeLimit: 50Mi

Special case — ml-training-service: Keep minio-ca (used by container for S3 model storage) and keep tls-certs:

      volumes:
        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true
        - name: tls-certs
          secret:
            secretName: ml-training-tls
        - name: minio-ca
          configMap:
            name: minio-ca-cert
        - name: tmp
          emptyDir:
            sizeLimit: 50Mi

Special case — web-gateway: Same as standard (minio-ca removed — static assets are on the PVC now).

C) No changes to container section — containers already mount /binaries readOnly and /tmp. The command field still points to /binaries/<binary_name>.

Step 2: Verify all YAMLs

Run: for f in infra/k8s/services/*.yaml; do kubectl apply --dry-run=client -f "$f" 2>&1 | head -5; done Expected: Each file shows deployment.apps/<name> created (dry run) and service/<name> created (dry run)

Step 3: Commit

git add infra/k8s/services/api-gateway.yaml \
        infra/k8s/services/trading-service.yaml \
        infra/k8s/services/broker-gateway.yaml \
        infra/k8s/services/ml-training-service.yaml \
        infra/k8s/services/backtesting-service.yaml \
        infra/k8s/services/trading-agent-service.yaml \
        infra/k8s/services/data-acquisition-service.yaml \
        infra/k8s/services/web-gateway.yaml
git commit -m "feat(infra): remove initContainers, mount shared foxhunt-binaries PVC"

---

Task 7: Apply shared PVC and seed initial binaries

Prerequisite: Tasks 1-6 completed (code changes done locally). This task operates on the live cluster.

Step 1: Create the shared PVC

Run: kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml Expected: persistentvolumeclaim/foxhunt-binaries created

Verify: kubectl get pvc foxhunt-binaries -n foxhunt Expected: STATUS=Bound (or Pending briefly, then Bound)

Step 2: Seed binaries from current running pods

The current pods have working binaries in their /binaries/ emptyDir (fetched from S3). Copy them to the new shared PVC via a writer pod.

# Create writer pod
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: binary-writer
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-writer
spec:
  nodeSelector:
    k8s.scaleway.com/pool-name: foxhunt
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  containers:
    - name: writer
      image: busybox:1.36
      command: ["sleep", "600"]
      volumeMounts:
        - name: binaries
          mountPath: /binaries
  volumes:
    - name: binaries
      persistentVolumeClaim:
        claimName: foxhunt-binaries
  restartPolicy: Never
EOF

# Wait for writer pod
kubectl wait --for=condition=Ready pod/binary-writer -n foxhunt --timeout=120s

Step 3: Copy binaries from running service pods to writer pod

# For each service, copy the binary from the running pod to the writer pod
for svc_bin in "api-gateway:api_gateway" \
               "trading-service:trading_service" \
               "broker-gateway:broker_gateway_service" \
               "ml-training-service:ml_training_service" \
               "backtesting-service:backtesting_service" \
               "trading-agent-service:trading_agent_service" \
               "data-acquisition-service:data_acquisition_service" \
               "web-gateway:web-gateway"; do
  svc="${svc_bin%%:*}"
  bin="${svc_bin##*:}"
  POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=$svc -o jsonpath='{.items[0].metadata.name}')
  if [ -n "$POD" ]; then
    kubectl cp foxhunt/$POD:/binaries/$bin /tmp/seed-$bin
    kubectl cp /tmp/seed-$bin foxhunt/binary-writer:/binaries/$bin
    rm /tmp/seed-$bin
    echo "Seeded $bin from $POD"
  else
    echo "WARNING: no pod for $svc"
  fi
done

Step 4: Copy web-gateway static assets

WG_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=web-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n foxhunt binary-writer -- mkdir -p /binaries/static
kubectl exec -n foxhunt $WG_POD -- tar -C /binaries/static -cf - . 2>/dev/null | kubectl exec -i -n foxhunt binary-writer -- tar -C /binaries/static -xf -
echo "Seeded web-gateway static assets"

Step 5: Make binaries executable and verify

for bin in trading_service api_gateway broker_gateway_service \
           ml_training_service backtesting_service \
           trading_agent_service data_acquisition_service web-gateway; do
  kubectl exec -n foxhunt binary-writer -- chmod +x /binaries/$bin
done

# Verify all binaries present
kubectl exec -n foxhunt binary-writer -- ls -lh /binaries/
kubectl exec -n foxhunt binary-writer -- ls /binaries/static/ | head -5

Expected: 8 binaries (~150-250MB each) + static/ directory

Step 6: Delete writer pod

Run: kubectl delete pod binary-writer -n foxhunt

---

Task 8: Apply modified service YAMLs and verify

Step 1: Apply all service YAMLs

Run: kubectl apply -f infra/k8s/services/ Expected: 8 deployments configured, services unchanged

Step 2: Rolling restart all services

for svc in trading-service api-gateway broker-gateway \
           ml-training-service backtesting-service \
           trading-agent-service data-acquisition-service web-gateway; do
  kubectl -n foxhunt rollout restart deployment/$svc
done

Step 3: Wait for rollout

for svc in trading-service api-gateway broker-gateway \
           ml-training-service backtesting-service \
           trading-agent-service data-acquisition-service web-gateway; do
  kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
done

Expected: All 8 deployments successfully rolled out

Step 4: Verify pods have no initContainers

kubectl get pods -n foxhunt -l app.kubernetes.io/part-of=foxhunt -o custom-columns='NAME:.metadata.name,INIT:.spec.initContainers[*].name,STATUS:.status.phase' | grep -v ib-gateway

Expected: INIT column is empty for all service pods (no initContainers)

Step 5: Verify binaries are accessible

# Spot-check: api-gateway can read its binary
API_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=api-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n foxhunt $API_POD -- ls -l /binaries/api_gateway

Expected: Binary exists and is executable

---

Task 9: Delete old PVCs and cleanup files

Step 1: Delete old per-service PVCs from cluster

for pvc in binary-cache-trading-service binary-cache-api-gateway \
           binary-cache-broker-gateway binary-cache-ml-training-service \
           binary-cache-backtesting-service binary-cache-trading-agent-service \
           binary-cache-data-acquisition-service binary-cache-web-gateway; do
  kubectl delete pvc $pvc -n foxhunt --ignore-not-found
done

Expected: 8 PVCs deleted

Step 2: Verify only foxhunt-binaries PVC remains for services

Run: kubectl get pvc -n foxhunt -l app.kubernetes.io/name=binary-cache Expected: No resources found

Run: kubectl get pvc foxhunt-binaries -n foxhunt Expected: STATUS=Bound

Step 3: Delete old PVC manifest file

Run: rm infra/k8s/storage/binary-cache-pvcs.yaml

Step 4: Commit cleanup

git add -A infra/k8s/storage/binary-cache-pvcs.yaml
git commit -m "chore(infra): delete 8 per-service binary-cache PVCs (replaced by foxhunt-binaries)"

---

Task 10: End-to-end verification

Step 1: Verify all pods running

Run: kubectl get pods -n foxhunt -l app.kubernetes.io/part-of=foxhunt -o wide Expected: All pods Running, no restarts, no initContainers

Step 2: Verify gRPC health for key services

API_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=api-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n foxhunt $API_POD -- grpc_health_probe -addr=localhost:50051

Expected: status: SERVING

Step 3: Verify web-gateway serves static assets

WG_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=web-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n foxhunt $WG_POD -- ls /binaries/static/index.html

Expected: File exists

Step 4: Verify PVC usage

Run: kubectl get pvc -n foxhunt | grep -E 'foxhunt-binaries|binary-cache' Expected: Only foxhunt-binaries (Bound, 2Gi)

Step 5: Final commit with all changes

git add -A
git commit -m "feat(infra): consolidate 8 binary-cache PVCs into single shared foxhunt-binaries PVC

Eliminates S3 round-trip for service binaries. CI deploys via kubectl cp
to a temporary writer pod. Services mount shared PVC read-only.
Training pipeline (GPU nodes) unchanged — still uses S3."

Step 6: Push to GitLab

Run: git push origin main

---

Summary of Changes

What	Before	After
Binary source	S3 (MinIO) via initContainer	Shared PVC via kubectl cp
PVCs	8 × 1Gi per-service	1 × 2Gi shared
initContainers	8 (fetch-binary)	0
S3 uploads (services)	compile-services + build-web-dashboard	None
CI artifact transfer	None	compile-services + build-web-dashboard
Deploy mechanism	rollout restart → initContainer fetches from S3	kubectl cp to PVC → rollout restart
Training pipeline	S3 (unchanged)	S3 (unchanged)
GPU overlays	S3 (unchanged)	S3 (unchanged)
Cost	8Gi PVCs (~€0.16/mo)	2Gi PVC (~€0.04/mo)