foxhunt/docs/plans/2026-03-02-binary-cache-consolidation-implementation.md

# Binary Cache PVC Consolidation Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Replace 8 per-service binary-cache PVCs + S3 round-trip with a single shared `foxhunt-binaries` PVC, eliminating initContainers, S3 uploads, and 7 unnecessary PVCs.

**Architecture:** CI compile jobs produce artifacts. Deploy job creates a temporary writer pod on the foxhunt node, `kubectl cp`s binaries + static assets onto the shared PVC, deletes the writer pod, then rollout-restarts all services. Services mount the shared PVC read-only at `/binaries/` — no initContainer, no S3.

**Tech Stack:** Kubernetes PVC (scw-bssd RWO), GitLab CI artifacts, kubectl cp, busybox writer pod

**Design doc:** `docs/plans/2026-03-02-binary-cache-consolidation-design.md`

---

## Reference: Services and Binary Names

| Service YAML | Binary Name | PVC to Delete |
|---|---|---|
| `api-gateway.yaml` | `api_gateway` | `binary-cache-api-gateway` |
| `trading-service.yaml` | `trading_service` | `binary-cache-trading-service` |
| `broker-gateway.yaml` | `broker_gateway_service` | `binary-cache-broker-gateway` |
| `ml-training-service.yaml` | `ml_training_service` | `binary-cache-ml-training-service` |
| `backtesting-service.yaml` | `backtesting_service` | `binary-cache-backtesting-service` |
| `trading-agent-service.yaml` | `trading_agent_service` | `binary-cache-trading-agent-service` |
| `data-acquisition-service.yaml` | `data_acquisition_service` | `binary-cache-data-acquisition-service` |
| `web-gateway.yaml` | `web-gateway` | `binary-cache-web-gateway` |

**Not modified:** `ib-gateway.yaml` (third-party Docker image), `*-gpu.yaml` overlays (different node pools, keep S3)

**Special cases:**
- `ml-training-service.yaml`: keeps `minio-ca` volume in the container (S3 for model storage, not binary fetch)
- `web-gateway.yaml`: initContainer also fetched static assets from S3 — these go into shared PVC at `/binaries/static/`

---

### Task 1: Create shared PVC YAML

**Files:**
- Create: `infra/k8s/storage/foxhunt-binaries-pvc.yaml`

**Step 1: Create the PVC manifest**

```yaml
# Shared binary PVC — all foxhunt services mount read-only
# CI deploy job writes via temporary binary-writer pod (kubectl cp)
# RWO on scw-bssd: all pods on the same foxhunt node can mount
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: foxhunt-binaries
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: foxhunt-binaries
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 2Gi
```

**Step 2: Verify YAML is valid**

Run: `kubectl apply --dry-run=client -f infra/k8s/storage/foxhunt-binaries-pvc.yaml`
Expected: `persistentvolumeclaim/foxhunt-binaries created (dry run)`

**Step 3: Commit**

```bash
git add infra/k8s/storage/foxhunt-binaries-pvc.yaml
git commit -m "feat(infra): add shared foxhunt-binaries PVC (2Gi)"
```

---

### Task 2: Modify compile-services CI job

**Files:**
- Modify: `.gitlab-ci.yml` (lines 409-458, `compile-services` job)

**Step 1: Add CI artifacts, remove S3 upload**

In the `compile-services` job, replace the S3 upload section (lines 445-458) with CI artifacts:

Remove this block (lines 445-458):
```yaml
    # Upload stripped binaries to MinIO (in-cluster S3)
    # Trust MinIO CA — rclone (Go) uses system CAs, not RCLONE_CA_CERT for S3.
    # Container runs non-root, so merge system + MinIO CA into writable /tmp bundle.
    - cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-bundle.crt 2>/dev/null || true
    - cat /etc/ssl/minio-ca/ca.crt >> /tmp/ca-bundle.crt 2>/dev/null || true
    - export SSL_CERT_FILE=/tmp/ca-bundle.crt
    - export RCLONE_S3_PROVIDER=Minio RCLONE_S3_ENDPOINT=https://minio.foxhunt.svc.cluster.local:9000 RCLONE_S3_REGION=us-east-1
    - export RCLONE_S3_ACCESS_KEY_ID=$MINIO_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY
    - |
      for bin in trading_service api_gateway broker_gateway_service ml_training_service backtesting_service trading_agent_service data_acquisition_service web-gateway; do
        rclone copyto build-out/services/$bin :s3:foxhunt-binaries/latest/services/$bin
      done
    - rclone copy build-out/services/ :s3:foxhunt-binaries/archive/${CI_COMMIT_SHA}/services/
    - echo "Service binaries uploaded to MinIO"
```

Replace with:
```yaml
    - echo "Service binaries ready as CI artifacts"
```

Add artifacts section at the end of the job:
```yaml
  artifacts:
    paths:
      - build-out/services/
    expire_in: 1 day
```

**Step 2: Verify CI syntax**

Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"`
Expected: `Valid YAML`

**Step 3: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "feat(ci): replace S3 upload with CI artifacts in compile-services"
```

---

### Task 3: Modify build-web-dashboard CI job

**Files:**
- Modify: `.gitlab-ci.yml` (lines 529-563, `build-web-dashboard` job)

**Step 1: Add CI artifacts, remove S3 upload**

In the `build-web-dashboard` job, remove the rclone/S3 section (lines 547-563) and add CI artifacts.

Remove the `before_script` block (lines 548-550, rclone install):
```yaml
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends curl unzip ca-certificates
    - curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip
    - unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ && rm /tmp/rclone.zip
```

In the `script` section, remove the rclone upload lines (555-563):
```yaml
    # Trust MinIO CA — rclone (Go) uses system CAs, not RCLONE_CA_CERT for S3.
    # Container runs non-root, so merge system + MinIO CA into writable /tmp bundle.
    - cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-bundle.crt 2>/dev/null || true
    - cat /etc/ssl/minio-ca/ca.crt >> /tmp/ca-bundle.crt 2>/dev/null || true
    - export SSL_CERT_FILE=/tmp/ca-bundle.crt
    - export RCLONE_S3_PROVIDER=Minio RCLONE_S3_ENDPOINT=https://minio.foxhunt.svc.cluster.local:9000 RCLONE_S3_REGION=us-east-1
    - export RCLONE_S3_ACCESS_KEY_ID=$MINIO_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY
    - rclone sync dist/ :s3:foxhunt-binaries/latest/web-dashboard/dist/
    - echo "Web dashboard assets uploaded to MinIO"
```

Replace with:
```yaml
    - echo "Web dashboard assets ready as CI artifacts"
```

Add artifacts section:
```yaml
  artifacts:
    paths:
      - web-dashboard/dist/
    expire_in: 1 day
```

**Step 2: Verify CI syntax**

Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"`
Expected: `Valid YAML`

**Step 3: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "feat(ci): replace S3 upload with CI artifacts in build-web-dashboard"
```

---

### Task 4: Remove write-manifest CI job

**Files:**
- Modify: `.gitlab-ci.yml` (lines 565-597, `write-manifest` job)

**Step 1: Delete the entire write-manifest job**

Remove lines 565-597 (the entire `write-manifest` job block including the section header comment). MANIFEST.json was S3-specific tracking — no longer needed.

**Step 2: Verify CI syntax**

Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"`
Expected: `Valid YAML`

**Step 3: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "chore(ci): remove write-manifest job (S3 tracking no longer needed)"
```

---

### Task 5: Rewrite deploy CI job

**Files:**
- Modify: `.gitlab-ci.yml` (deploy job, currently starting around line 1460)

**Step 1: Update job dependencies**

Change the `needs` section from:
```yaml
  needs:
    - job: write-manifest
      optional: true
```

To:
```yaml
  needs:
    - job: compile-services
      optional: true
      artifacts: true
    - job: build-web-dashboard
      optional: true
      artifacts: true
```

**Step 2: Replace service deploy section**

Find the section that applies binary-cache-pvcs.yaml and does rollout restarts (around line 1595-1613). Replace:

```yaml
    # Apply service deployments and binary cache PVCs (idempotent — picks up any YAML changes)
    # Skip training/job-template.yaml (placeholder names, not directly applicable)
    # Skip storage/training-data-pv.yaml (PVC spec immutable after creation)
    - kubectl apply -f infra/k8s/services/ -f infra/k8s/storage/binary-cache-pvcs.yaml
    # Rolling restart triggers initContainers to fetch latest binaries from MinIO
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout restart deployment/$svc
        echo "Restarted $svc"
      done
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
      done
    - echo "All services deployed with MinIO binaries"
```

With:

```yaml
    # Apply shared binaries PVC (idempotent)
    - kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml
    # Apply service deployments (picks up any YAML changes)
    # Skip training/job-template.yaml (placeholder names, not directly applicable)
    # Skip storage/training-data-pv.yaml (PVC spec immutable after creation)
    - kubectl apply -f infra/k8s/services/
    # Copy new binaries to shared PVC (if compile ran)
    - |
      if [ -d "build-out/services" ]; then
        echo "Deploying new binaries via kubectl cp..."
        # Create temporary writer pod on foxhunt node
        kubectl apply -f - <<'WRITER_EOF'
      apiVersion: v1
      kind: Pod
      metadata:
        name: binary-writer
        namespace: foxhunt
        labels:
          app.kubernetes.io/name: binary-writer
      spec:
        nodeSelector:
          k8s.scaleway.com/pool-name: foxhunt
        securityContext:
          runAsUser: 1000
          runAsGroup: 1000
          fsGroup: 1000
        containers:
          - name: writer
            image: busybox:1.36
            command: ["sleep", "600"]
            volumeMounts:
              - name: binaries
                mountPath: /binaries
        volumes:
          - name: binaries
            persistentVolumeClaim:
              claimName: foxhunt-binaries
        restartPolicy: Never
      WRITER_EOF
        kubectl wait --for=condition=Ready pod/binary-writer -n foxhunt --timeout=120s
        # Copy service binaries
        for bin in trading_service api_gateway broker_gateway_service \
                   ml_training_service backtesting_service \
                   trading_agent_service data_acquisition_service web-gateway; do
          kubectl cp build-out/services/$bin foxhunt/binary-writer:/binaries/$bin
          echo "  Copied $bin ($(stat -c%s build-out/services/$bin) bytes)"
        done
        # Copy web dashboard static assets (if build-web-dashboard ran)
        if [ -d "web-dashboard/dist" ]; then
          kubectl exec -n foxhunt binary-writer -- mkdir -p /binaries/static
          tar -C web-dashboard/dist -cf - . | kubectl exec -i -n foxhunt binary-writer -- tar -C /binaries/static -xf -
          echo "  Copied web-dashboard static assets"
        fi
        # Make binaries executable
        for bin in trading_service api_gateway broker_gateway_service \
                   ml_training_service backtesting_service \
                   trading_agent_service data_acquisition_service web-gateway; do
          kubectl exec -n foxhunt binary-writer -- chmod +x /binaries/$bin
        done
        # Cleanup writer pod
        kubectl delete pod binary-writer -n foxhunt --wait=false
        echo "Binaries deployed to shared PVC"
      else
        echo "No new binaries — compile-services did not run, PVC has existing binaries"
      fi
    # Rolling restart all services (they read from shared PVC, no initContainer needed)
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout restart deployment/$svc
        echo "Restarted $svc"
      done
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
      done
    - echo "All services deployed"
```

Also update the comment at line 1507:
```yaml
    # S3 binary share: services use generic runtime image + initContainer to fetch binaries from S3.
    # Deploy = apply K8s manifests (picks up YAML changes) + rollout restart (fetch new binaries).
```
Change to:
```yaml
    # Shared PVC binary deploy: CI copies binaries to foxhunt-binaries PVC via writer pod.
    # Services mount the PVC read-only — no initContainer, no S3.
```

**Step 3: Verify CI syntax**

Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"`
Expected: `Valid YAML`

**Step 4: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "feat(ci): deploy binaries via kubectl cp to shared PVC instead of S3"
```

---

### Task 6: Modify all 8 service YAMLs

**Files:**
- Modify: `infra/k8s/services/api-gateway.yaml`
- Modify: `infra/k8s/services/trading-service.yaml`
- Modify: `infra/k8s/services/broker-gateway.yaml`
- Modify: `infra/k8s/services/ml-training-service.yaml`
- Modify: `infra/k8s/services/backtesting-service.yaml`
- Modify: `infra/k8s/services/trading-agent-service.yaml`
- Modify: `infra/k8s/services/data-acquisition-service.yaml`
- Modify: `infra/k8s/services/web-gateway.yaml`

**Step 1: For each of the 8 services, apply these changes:**

**A) Remove the entire `initContainers` section** (the `fetch-binary` initContainer with rclone S3 download, ~57 lines per service)

**B) Replace the `volumes` section.** Remove `binary-cache` (per-service PVC) and `minio-ca` (ConfigMap). Change `binaries` from `emptyDir` to shared PVC.

Before (standard service — api-gateway, trading-service, broker-gateway, backtesting-service, trading-agent-service, data-acquisition-service):
```yaml
      volumes:
        - name: binaries
          emptyDir:
            sizeLimit: 200Mi
        - name: binary-cache
          persistentVolumeClaim:
            claimName: binary-cache-<service-name>
        - name: minio-ca
          configMap:
            name: minio-ca-cert
        - name: tmp
          emptyDir:
            sizeLimit: 50Mi
```

After:
```yaml
      volumes:
        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true
        - name: tmp
          emptyDir:
            sizeLimit: 50Mi
```

**Special case — ml-training-service:** Keep `minio-ca` (used by container for S3 model storage) and keep `tls-certs`:
```yaml
      volumes:
        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true
        - name: tls-certs
          secret:
            secretName: ml-training-tls
        - name: minio-ca
          configMap:
            name: minio-ca-cert
        - name: tmp
          emptyDir:
            sizeLimit: 50Mi
```

**Special case — web-gateway:** Same as standard (minio-ca removed — static assets are on the PVC now).

**C) No changes to container section** — containers already mount `/binaries` readOnly and `/tmp`. The `command` field still points to `/binaries/<binary_name>`.

**Step 2: Verify all YAMLs**

Run: `for f in infra/k8s/services/*.yaml; do kubectl apply --dry-run=client -f "$f" 2>&1 | head -5; done`
Expected: Each file shows `deployment.apps/<name> created (dry run)` and `service/<name> created (dry run)`

**Step 3: Commit**

```bash
git add infra/k8s/services/api-gateway.yaml \
        infra/k8s/services/trading-service.yaml \
        infra/k8s/services/broker-gateway.yaml \
        infra/k8s/services/ml-training-service.yaml \
        infra/k8s/services/backtesting-service.yaml \
        infra/k8s/services/trading-agent-service.yaml \
        infra/k8s/services/data-acquisition-service.yaml \
        infra/k8s/services/web-gateway.yaml
git commit -m "feat(infra): remove initContainers, mount shared foxhunt-binaries PVC"
```

---

### Task 7: Apply shared PVC and seed initial binaries

**Prerequisite:** Tasks 1-6 completed (code changes done locally). This task operates on the live cluster.

**Step 1: Create the shared PVC**

Run: `kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml`
Expected: `persistentvolumeclaim/foxhunt-binaries created`

Verify: `kubectl get pvc foxhunt-binaries -n foxhunt`
Expected: STATUS=Bound (or Pending briefly, then Bound)

**Step 2: Seed binaries from current running pods**

The current pods have working binaries in their `/binaries/` emptyDir (fetched from S3). Copy them to the new shared PVC via a writer pod.

```bash
# Create writer pod
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: binary-writer
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-writer
spec:
  nodeSelector:
    k8s.scaleway.com/pool-name: foxhunt
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
  containers:
    - name: writer
      image: busybox:1.36
      command: ["sleep", "600"]
      volumeMounts:
        - name: binaries
          mountPath: /binaries
  volumes:
    - name: binaries
      persistentVolumeClaim:
        claimName: foxhunt-binaries
  restartPolicy: Never
EOF

# Wait for writer pod
kubectl wait --for=condition=Ready pod/binary-writer -n foxhunt --timeout=120s
```

**Step 3: Copy binaries from running service pods to writer pod**

```bash
# For each service, copy the binary from the running pod to the writer pod
for svc_bin in "api-gateway:api_gateway" \
               "trading-service:trading_service" \
               "broker-gateway:broker_gateway_service" \
               "ml-training-service:ml_training_service" \
               "backtesting-service:backtesting_service" \
               "trading-agent-service:trading_agent_service" \
               "data-acquisition-service:data_acquisition_service" \
               "web-gateway:web-gateway"; do
  svc="${svc_bin%%:*}"
  bin="${svc_bin##*:}"
  POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=$svc -o jsonpath='{.items[0].metadata.name}')
  if [ -n "$POD" ]; then
    kubectl cp foxhunt/$POD:/binaries/$bin /tmp/seed-$bin
    kubectl cp /tmp/seed-$bin foxhunt/binary-writer:/binaries/$bin
    rm /tmp/seed-$bin
    echo "Seeded $bin from $POD"
  else
    echo "WARNING: no pod for $svc"
  fi
done
```

**Step 4: Copy web-gateway static assets**

```bash
WG_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=web-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n foxhunt binary-writer -- mkdir -p /binaries/static
kubectl exec -n foxhunt $WG_POD -- tar -C /binaries/static -cf - . 2>/dev/null | kubectl exec -i -n foxhunt binary-writer -- tar -C /binaries/static -xf -
echo "Seeded web-gateway static assets"
```

**Step 5: Make binaries executable and verify**

```bash
for bin in trading_service api_gateway broker_gateway_service \
           ml_training_service backtesting_service \
           trading_agent_service data_acquisition_service web-gateway; do
  kubectl exec -n foxhunt binary-writer -- chmod +x /binaries/$bin
done

# Verify all binaries present
kubectl exec -n foxhunt binary-writer -- ls -lh /binaries/
kubectl exec -n foxhunt binary-writer -- ls /binaries/static/ | head -5
```

Expected: 8 binaries (~150-250MB each) + `static/` directory

**Step 6: Delete writer pod**

Run: `kubectl delete pod binary-writer -n foxhunt`

---

### Task 8: Apply modified service YAMLs and verify

**Step 1: Apply all service YAMLs**

Run: `kubectl apply -f infra/k8s/services/`
Expected: 8 deployments configured, services unchanged

**Step 2: Rolling restart all services**

```bash
for svc in trading-service api-gateway broker-gateway \
           ml-training-service backtesting-service \
           trading-agent-service data-acquisition-service web-gateway; do
  kubectl -n foxhunt rollout restart deployment/$svc
done
```

**Step 3: Wait for rollout**

```bash
for svc in trading-service api-gateway broker-gateway \
           ml-training-service backtesting-service \
           trading-agent-service data-acquisition-service web-gateway; do
  kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
done
```

Expected: All 8 deployments successfully rolled out

**Step 4: Verify pods have no initContainers**

```bash
kubectl get pods -n foxhunt -l app.kubernetes.io/part-of=foxhunt -o custom-columns='NAME:.metadata.name,INIT:.spec.initContainers[*].name,STATUS:.status.phase' | grep -v ib-gateway
```

Expected: INIT column is empty for all service pods (no initContainers)

**Step 5: Verify binaries are accessible**

```bash
# Spot-check: api-gateway can read its binary
API_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=api-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n foxhunt $API_POD -- ls -l /binaries/api_gateway
```

Expected: Binary exists and is executable

---

### Task 9: Delete old PVCs and cleanup files

**Step 1: Delete old per-service PVCs from cluster**

```bash
for pvc in binary-cache-trading-service binary-cache-api-gateway \
           binary-cache-broker-gateway binary-cache-ml-training-service \
           binary-cache-backtesting-service binary-cache-trading-agent-service \
           binary-cache-data-acquisition-service binary-cache-web-gateway; do
  kubectl delete pvc $pvc -n foxhunt --ignore-not-found
done
```

Expected: 8 PVCs deleted

**Step 2: Verify only foxhunt-binaries PVC remains for services**

Run: `kubectl get pvc -n foxhunt -l app.kubernetes.io/name=binary-cache`
Expected: No resources found

Run: `kubectl get pvc foxhunt-binaries -n foxhunt`
Expected: STATUS=Bound

**Step 3: Delete old PVC manifest file**

Run: `rm infra/k8s/storage/binary-cache-pvcs.yaml`

**Step 4: Commit cleanup**

```bash
git add -A infra/k8s/storage/binary-cache-pvcs.yaml
git commit -m "chore(infra): delete 8 per-service binary-cache PVCs (replaced by foxhunt-binaries)"
```

---

### Task 10: End-to-end verification

**Step 1: Verify all pods running**

Run: `kubectl get pods -n foxhunt -l app.kubernetes.io/part-of=foxhunt -o wide`
Expected: All pods Running, no restarts, no initContainers

**Step 2: Verify gRPC health for key services**

```bash
API_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=api-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n foxhunt $API_POD -- grpc_health_probe -addr=localhost:50051
```

Expected: `status: SERVING`

**Step 3: Verify web-gateway serves static assets**

```bash
WG_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=web-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n foxhunt $WG_POD -- ls /binaries/static/index.html
```

Expected: File exists

**Step 4: Verify PVC usage**

Run: `kubectl get pvc -n foxhunt | grep -E 'foxhunt-binaries|binary-cache'`
Expected: Only `foxhunt-binaries` (Bound, 2Gi)

**Step 5: Final commit with all changes**

```bash
git add -A
git commit -m "feat(infra): consolidate 8 binary-cache PVCs into single shared foxhunt-binaries PVC

Eliminates S3 round-trip for service binaries. CI deploys via kubectl cp
to a temporary writer pod. Services mount shared PVC read-only.
Training pipeline (GPU nodes) unchanged — still uses S3."
```

**Step 6: Push to GitLab**

Run: `git push origin main`

---

## Summary of Changes

| What | Before | After |
|---|---|---|
| Binary source | S3 (MinIO) via initContainer | Shared PVC via kubectl cp |
| PVCs | 8 × 1Gi per-service | 1 × 2Gi shared |
| initContainers | 8 (fetch-binary) | 0 |
| S3 uploads (services) | compile-services + build-web-dashboard | None |
| CI artifact transfer | None | compile-services + build-web-dashboard |
| Deploy mechanism | rollout restart → initContainer fetches from S3 | kubectl cp to PVC → rollout restart |
| Training pipeline | S3 (unchanged) | S3 (unchanged) |
| GPU overlays | S3 (unchanged) | S3 (unchanged) |
| Cost | 8Gi PVCs (~€0.16/mo) | 2Gi PVC (~€0.04/mo) |