foxhunt/docs/plans/2026-03-04-minio-binary-distribution.md

MinIO Binary Distribution Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Replace PVC-based binary distribution with MinIO S3 fetch, eliminating the L40S autoscale-for-PVC-writes problem.

Architecture: Deploy job uploads compiled binaries to MinIO via aws s3 cp. Service pods and training jobs fetch their binary at startup via rclone initContainer into an emptyDir. Two binary PVCs are deleted.

Tech Stack: MinIO (in-cluster S3), rclone (in runtime images), aws-cli (in infra-runner)

---

Task 1: Replace binary-writer pods with MinIO upload in deploy job

Files:

• Modify: .gitlab-ci.yml:1564-1681 (deploy job binary section)

Step 1: Delete PVC apply lines and entire binary-writer block

In .gitlab-ci.yml, find and delete these two PVC apply lines (around line 1565-1566):

    - kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml
    - kubectl apply -f infra/k8s/training/training-binaries-pvc.yaml

Then delete the entire block from # ── Step 1: Copy binaries to PVCs through echo "=== Binary deployment complete ===" (lines ~1567-1681). This removes all binary-writer/training-binary-writer pod logic.

Step 2: Add MinIO upload block

Insert in place of the deleted block (same indentation level as other - | script blocks in the deploy job):

    # ── Step 1: Upload binaries to MinIO (replaces PVC binary-writer pods) ──
    - |
      if [ -d "build-out/services" ] || [ -d "build-out/training" ] || [ -d "web-dashboard/dist" ]; then
        set -e
        echo "=== Uploading binaries to MinIO ==="
        export AWS_ACCESS_KEY_ID="${MINIO_ACCESS_KEY}"
        export AWS_SECRET_ACCESS_KEY="${MINIO_SECRET_KEY}"
        S3="aws s3 --endpoint-url https://minio.foxhunt.svc.cluster.local:9000 --no-verify-ssl"

        if [ -d "build-out/services" ]; then
          $S3 cp build-out/services/ s3://foxhunt-binaries/services/ --recursive
          echo "Service binaries uploaded to MinIO"
        fi
        if [ -d "build-out/training" ]; then
          $S3 cp build-out/training/ s3://foxhunt-binaries/training/ --recursive
          echo "Training binaries uploaded to MinIO"
        fi
        if [ -d "web-dashboard/dist" ]; then
          $S3 cp web-dashboard/dist/ s3://foxhunt-binaries/static/ --recursive
          echo "Web dashboard assets uploaded to MinIO"
        fi
        echo "=== MinIO upload complete ==="
      else
        echo "No new binaries — compile jobs did not run"
      fi

Step 3: Verify YAML syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo OK Expected: OK

Step 4: Commit

git add .gitlab-ci.yml
git commit -m "refactor(ci): replace PVC binary-writers with MinIO upload

Deploy job now uploads binaries to MinIO via aws-cli (already in
infra-runner image). Eliminates binary-writer and training-binary-writer
pods, removing the L40S autoscale trigger for PVC writes."

---

Task 2: Update trading-service.yaml — PVC to initContainer+emptyDir

Files:

• Modify: infra/k8s/services/trading-service.yaml

Step 1: Add initContainer block

Insert after nodeSelector: / before containers: (between lines 39 and 40):

      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              rclone copyto \
                ":s3:foxhunt-binaries/services/trading_service" \
                "/binaries/trading_service" \
                --s3-provider=Minio \
                --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
                --s3-no-check-bucket \
                --no-check-certificate
              chmod +x /binaries/trading_service
              echo "Fetched trading_service ($(stat -c%s /binaries/trading_service) bytes)"
          env:
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi

Step 2: Replace PVC volume with emptyDir

Replace:

        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true

With:

        - name: binaries
          emptyDir:
            sizeLimit: 200Mi

Step 3: Remove readOnly from main container's binaries volumeMount

The main container volumeMounts has readOnly: true on the binaries mount. The initContainer needs write access. Change:

            - name: binaries
              mountPath: /binaries
              readOnly: true

To:

            - name: binaries
              mountPath: /binaries
              readOnly: true

Actually — the initContainer writes, the main container reads. emptyDir supports this: the initContainer writes, then terminates, then the main container mounts the same emptyDir read-only. Keep readOnly: true on the main container's mount. The initContainer's mount does NOT have readOnly: true.

Step 4: Verify YAML

Run: python3 -c "import yaml; list(yaml.safe_load_all(open('infra/k8s/services/trading-service.yaml')))" && echo OK Expected: OK

---

Task 3: Update api-gateway.yaml

Files:

• Modify: infra/k8s/services/api-gateway.yaml

Same pattern as Task 2. Binary name: api_gateway

Step 1: Add initContainer (between nodeSelector: and containers:, after line 40)

Same initContainer as Task 2 but with binary name api_gateway:

• rclone copyto ":s3:foxhunt-binaries/services/api_gateway" "/binaries/api_gateway"
• chmod +x /binaries/api_gateway
• echo "Fetched api_gateway ..."

Step 2: Replace PVC volume with emptyDir (lines 122-126)

Replace:

        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true

With:

        - name: binaries
          emptyDir:
            sizeLimit: 200Mi

---

Task 4: Update broker-gateway.yaml

Files:

• Modify: infra/k8s/services/broker-gateway.yaml

Same pattern. Binary name: broker_gateway_service

Step 1: Add initContainer (between nodeSelector: line 39 and containers: line 40)

Binary: broker_gateway_service

Step 2: Replace PVC volume with emptyDir (lines 116-120)

---

Task 5: Update ml-training-service.yaml

Files:

• Modify: infra/k8s/services/ml-training-service.yaml

Binary name: ml_training_service. This service already has minio-credentials env vars and minio-ca volume.

Step 1: Add initContainer (between nodeSelector: line 82 and containers: line 83)

Same pattern. Binary: ml_training_service. Command includes "serve" arg — that's on the main container, not the initContainer.

Step 2: Replace PVC volume with emptyDir (lines 166-170)

Replace:

        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true

With:

        - name: binaries
          emptyDir:
            sizeLimit: 200Mi

---

Task 6: Update backtesting-service.yaml

Files:

• Modify: infra/k8s/services/backtesting-service.yaml

Binary name: backtesting_service

Step 1: Add initContainer (between nodeSelector: line 39 and containers: line 40) Step 2: Replace PVC volume with emptyDir (lines 105-109)

---

Task 7: Update trading-agent-service.yaml

Files:

• Modify: infra/k8s/services/trading-agent-service.yaml

Binary name: trading_agent_service

Step 1: Add initContainer (between nodeSelector: line 39 and containers: line 40) Step 2: Replace PVC volume with emptyDir (lines 103-107)

---

Task 8: Update data-acquisition-service.yaml

Files:

• Modify: infra/k8s/services/data-acquisition-service.yaml

Binary name: data_acquisition_service

Step 1: Add initContainer (between nodeSelector: line 39 and containers: line 40) Step 2: Replace PVC volume with emptyDir (lines 96-100)

---

Task 9: Update web-gateway.yaml (special — needs static assets too)

Files:

• Modify: infra/k8s/services/web-gateway.yaml

Binary name: web-gateway. Additionally fetches static dashboard assets.

Step 1: Add initContainer with dual fetch

Insert between nodeSelector: line 39 and containers: line 40:

      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              RCLONE_FLAGS="--s3-provider=Minio --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 --s3-access-key-id=${MINIO_ACCESS_KEY} --s3-secret-access-key=${MINIO_SECRET_KEY} --s3-no-check-bucket --no-check-certificate"
              rclone copyto ":s3:foxhunt-binaries/services/web-gateway" "/binaries/web-gateway" $RCLONE_FLAGS
              chmod +x /binaries/web-gateway
              echo "Fetched web-gateway ($(stat -c%s /binaries/web-gateway) bytes)"
              rclone copy ":s3:foxhunt-binaries/static/" "/binaries/static/" $RCLONE_FLAGS
              echo "Fetched static assets"
          env:
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 256Mi

Note: sizeLimit on the emptyDir should be 500Mi for web-gateway (binary + static assets).

Step 2: Replace PVC volume with emptyDir (lines 102-106)

Replace:

        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true

With:

        - name: binaries
          emptyDir:
            sizeLimit: 500Mi

---

Task 10: Commit all service yaml changes

git add infra/k8s/services/
git commit -m "refactor(k8s): replace PVC mounts with MinIO rclone initContainers

All 8 service deployments now fetch their binary from MinIO at pod
startup via rclone initContainer into emptyDir. Eliminates dependency
on foxhunt-binaries PVC (RWO node affinity constraint).

web-gateway additionally fetches static dashboard assets."

---

Task 11: Update training job-template.yaml

Files:

• Modify: infra/k8s/training/job-template.yaml

Step 1: Replace existing fetch-binary initContainer

The current initContainer (lines 41-73) copies from training-binaries PVC. Replace entire initContainer with:

      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              rclone copyto \
                ":s3:foxhunt-binaries/training/$(TRAINING_BINARY)" \
                "/binaries/$(TRAINING_BINARY)" \
                --s3-provider=Minio \
                --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
                --s3-no-check-bucket \
                --no-check-certificate
              chmod +x "/binaries/$(TRAINING_BINARY)"
              echo "Fetched $(TRAINING_BINARY) ($(stat -c%s /binaries/$(TRAINING_BINARY)) bytes)"
          env:
            - name: TRAINING_BINARY
              value: train_baseline_supervised
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi

Step 2: Remove training-binaries-src volume

Delete from volumes section (lines 122-127):

        - name: training-binaries-src
          persistentVolumeClaim:
            claimName: training-binaries
            readOnly: true

Step 3: Verify YAML

Run: python3 -c "import yaml; list(yaml.safe_load_all(open('infra/k8s/training/job-template.yaml')))" && echo OK Expected: OK

Step 4: Commit

git add infra/k8s/training/job-template.yaml
git commit -m "refactor(k8s): training job fetches binary from MinIO

Replace PVC-based training binary distribution with MinIO rclone
initContainer. Training jobs no longer need training-binaries PVC,
eliminating L40S GPU node autoscale just for binary distribution."

---

Task 12: Delete PVC manifests

Files:

• Delete: infra/k8s/storage/foxhunt-binaries-pvc.yaml
• Delete: infra/k8s/training/training-binaries-pvc.yaml

Step 1: Delete the files

rm infra/k8s/storage/foxhunt-binaries-pvc.yaml
rm infra/k8s/training/training-binaries-pvc.yaml

Step 2: Verify no remaining references to deleted PVCs

Run: grep -r "foxhunt-binaries" infra/ .gitlab-ci.yml --include='*.yaml' --include='*.yml' Expected: Only MinIO bucket references (s3://foxhunt-binaries/), no PVC references.

Run: grep -r "training-binaries" infra/ .gitlab-ci.yml --include='*.yaml' --include='*.yml' Expected: No results (the PVC name is gone).

Step 3: Commit

git add -A infra/k8s/storage/foxhunt-binaries-pvc.yaml infra/k8s/training/training-binaries-pvc.yaml
git commit -m "chore(k8s): delete foxhunt-binaries and training-binaries PVCs

These PVCs are replaced by MinIO-based binary distribution.
Service pods and training jobs now fetch binaries from MinIO
via rclone initContainers.

NOTE: Run 'kubectl delete pvc foxhunt-binaries training-binaries -n foxhunt'
on the cluster to reclaim the block storage volumes."

---

Task 13: Final verification

Step 1: Grep for any remaining PVC binary references

Run: grep -rn "claimName: foxhunt-binaries\|claimName: training-binaries" infra/ .gitlab-ci.yml Expected: No results.

Step 2: Grep for binary-writer references

Run: grep -rn "binary-writer\|binary_writer" .gitlab-ci.yml Expected: No results.

Step 3: Verify all service yamls have initContainers

Run: grep -l "fetch-binary" infra/k8s/services/*.yaml | wc -l Expected: 8

Step 4: Verify training job has MinIO fetch

Run: grep "foxhunt-binaries/training" infra/k8s/training/job-template.yaml Expected: Match on the rclone line.

---

Summary of Changes

File	Action	Lines ~Δ
.gitlab-ci.yml	Replace binary-writer pods with aws s3 cp	-100/+20
infra/k8s/services/trading-service.yaml	Add initContainer, PVC→emptyDir	+30/-4
infra/k8s/services/api-gateway.yaml	Add initContainer, PVC→emptyDir	+30/-4
infra/k8s/services/broker-gateway.yaml	Add initContainer, PVC→emptyDir	+30/-4
infra/k8s/services/ml-training-service.yaml	Add initContainer, PVC→emptyDir	+30/-4
infra/k8s/services/backtesting-service.yaml	Add initContainer, PVC→emptyDir	+30/-4
infra/k8s/services/trading-agent-service.yaml	Add initContainer, PVC→emptyDir	+30/-4
infra/k8s/services/data-acquisition-service.yaml	Add initContainer, PVC→emptyDir	+30/-4
infra/k8s/services/web-gateway.yaml	Add initContainer (binary+static), PVC→emptyDir	+35/-4
infra/k8s/training/job-template.yaml	Replace PVC initContainer with MinIO	+30/-20
infra/k8s/storage/foxhunt-binaries-pvc.yaml	Delete	-19
infra/k8s/training/training-binaries-pvc.yaml	Delete	-19

Total: 12 files, ~+275/-210 lines, 0 image rebuilds

Post-Deploy Cluster Cleanup

After deploying this change, manually delete the orphaned PVCs:

kubectl delete pvc foxhunt-binaries training-binaries -n foxhunt

Add MINIO_ACCESS_KEY and MINIO_SECRET_KEY as CI/CD variables in GitLab (Settings → CI/CD → Variables), or source them from the cluster secret during the deploy job.