foxhunt/docs/plans/2026-03-04-minio-binary-distribution-design.md

MinIO Binary Distribution — Eliminate PVC Binary Writers

Date: 2026-03-04 Status: Approved Scope: Replace PVC-based binary distribution with MinIO S3 fetch pattern

Problem

The current deploy cycle uses kubectl cp via temporary busybox writer pods to copy compiled binaries onto RWO PVCs. Two PVCs exist:

• foxhunt-binaries (foxhunt node) — service binaries
• training-binaries (ci-training node) — training binaries

The training PVC writer triggers L40S GPU autoscale from zero just to receive a binary copy — expensive and slow. RWO PVCs also create node-affinity constraints that prevent cross-pool flexibility.

Solution

Use MinIO (already deployed on platform node) as the binary distribution layer.

Flow:

compile jobs → GitLab artifacts (source of truth)
                    ↓
              deploy job: aws s3 cp → MinIO foxhunt-binaries bucket
                    ↓
              service pods: rclone initContainer → emptyDir
              training jobs: rclone initContainer → emptyDir
              external GPUs: GitLab artifacts API

Key insight: Zero image rebuilds. aws-cli is already in infra-runner, rclone is already in both runtime images.

Changes

1. Deploy Job (.gitlab-ci.yml)

Delete: ~115 lines of binary-writer pod logic (lines 1567-1681) Add: ~20 lines of aws s3 cp to MinIO

# Upload binaries to MinIO (replaces binary-writer pods)
- |
  if [ -d "build-out/services" ] || [ -d "build-out/training" ]; then
    export AWS_ACCESS_KEY_ID="$MINIO_ACCESS_KEY"
    export AWS_SECRET_ACCESS_KEY="$MINIO_SECRET_KEY"
    S3="aws s3 --endpoint-url https://minio.foxhunt.svc.cluster.local:9000 --no-verify-ssl"

    if [ -d "build-out/services" ]; then
      $S3 cp build-out/services/ s3://foxhunt-binaries/services/ --recursive
      echo "Service binaries → MinIO"
    fi
    if [ -d "build-out/training" ]; then
      $S3 cp build-out/training/ s3://foxhunt-binaries/training/ --recursive
      echo "Training binaries → MinIO"
    fi
    if [ -d "web-dashboard/dist" ]; then
      $S3 cp web-dashboard/dist/ s3://foxhunt-binaries/static/ --recursive
      echo "Web dashboard assets → MinIO"
    fi
  fi

Also remove:

• kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml
• kubectl apply -f infra/k8s/training/training-binaries-pvc.yaml

Add MinIO credentials as CI variables (MINIO_ACCESS_KEY, MINIO_SECRET_KEY) or source from existing minio-credentials secret.

2. Service Deployments (8 files in infra/k8s/services/)

For each service, replace PVC volume mount with initContainer + emptyDir:

spec:
  template:
    spec:
      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              rclone copyto \
                ":s3:foxhunt-binaries/services/${SERVICE_NAME}" \
                "/binaries/${SERVICE_NAME}" \
                --s3-provider=Minio \
                --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
                --s3-no-check-bucket \
                --no-check-certificate
              chmod +x "/binaries/${SERVICE_NAME}"
              echo "Fetched ${SERVICE_NAME} ($(stat -c%s /binaries/${SERVICE_NAME}) bytes)"
          env:
            - name: SERVICE_NAME
              value: trading_service  # per-service value
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
      # Also fetch web-dashboard static assets for web-gateway
      volumes:
        - name: binaries
          emptyDir:
            sizeLimit: 200Mi  # single stripped binary ~20-50MB

Remove from each service:

# DELETE these lines:
volumes:
  - name: binaries
    persistentVolumeClaim:
      claimName: foxhunt-binaries
      readOnly: true

web-gateway additionally needs static assets fetched:

# Extra step in web-gateway initContainer:
rclone copy ":s3:foxhunt-binaries/static/" "/binaries/static/" \
  --s3-provider=Minio ...

3. Training Job Template (infra/k8s/training/job-template.yaml)

Replace the PVC-based fetch-binary initContainer with MinIO rclone:

initContainers:
  - name: fetch-binary
    image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
    command: ["/bin/sh", "-c"]
    args:
      - |
        rclone copyto \
          ":s3:foxhunt-binaries/training/${TRAINING_BINARY}" \
          "/binaries/${TRAINING_BINARY}" \
          --s3-provider=Minio \
          --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
          --s3-access-key-id=${MINIO_ACCESS_KEY} \
          --s3-secret-access-key=${MINIO_SECRET_KEY} \
          --s3-no-check-bucket \
          --no-check-certificate
        chmod +x "/binaries/${TRAINING_BINARY}"
    env:
      - name: TRAINING_BINARY
        value: train_baseline_supervised
      - name: MINIO_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: minio-credentials
            key: access-key
      - name: MINIO_SECRET_KEY
        valueFrom:
          secretKeyRef:
            name: minio-credentials
            key: secret-key
    volumeMounts:
      - name: binaries
        mountPath: /binaries

Remove training-binaries-src volume (PVC mount).

4. Files to Delete

• infra/k8s/storage/foxhunt-binaries-pvc.yaml
• infra/k8s/training/training-binaries-pvc.yaml

5. Files Unchanged

• training-data-pvc.yaml — keep as-is
• sccache-pvc.yaml — keep
• minio-data-new (in minio.yaml) — keep
• CI training jobs (.train-rl-base, .train-validate-base) — already use GL artifacts via needs:

Service-to-Binary Mapping

Service Deployment	Binary Name	Pool
trading-service	trading_service	foxhunt
api-gateway	api_gateway	foxhunt
broker-gateway	broker_gateway_service	foxhunt
ml-training-service	ml_training_service	foxhunt
backtesting-service	backtesting_service	foxhunt
trading-agent-service	trading_agent_service	foxhunt
data-acquisition-service	data_acquisition_service	foxhunt
web-gateway	web-gateway	foxhunt

MinIO Bucket Layout

foxhunt-binaries/
├── services/
│   ├── trading_service
│   ├── api_gateway
│   ├── broker_gateway_service
│   ├── ml_training_service
│   ├── backtesting_service
│   ├── trading_agent_service
│   ├── data_acquisition_service
│   └── web-gateway
├── training/
│   ├── train_baseline_rl
│   ├── train_baseline_supervised
│   ├── evaluate_baseline
│   ├── evaluate_supervised
│   ├── hyperopt_baseline_rl
│   ├── hyperopt_baseline_supervised
│   └── training_uploader
└── static/
    └── (web-dashboard dist files)

External GPU Access

External GPUs download training binaries from GitLab artifacts API:

curl --header "PRIVATE-TOKEN: $GL_TOKEN" \
  "https://<gitlab>/api/v4/projects/<id>/jobs/artifacts/main/download?job=compile-training" \
  -o artifacts.zip

Or from MinIO via Tailscale (if MinIO is exposed).

Rollback

If MinIO is down during deploy, the deploy job fails fast (aws s3 cp fails). Service pods with existing emptyDir continue running. New pods fail to start (initContainer fails). MinIO recovery restores normal operation.

Previous binaries remain in MinIO until overwritten — no expiry, no garbage.

Net Effect

• Deleted: ~115 lines of binary-writer pod logic
• Deleted: 2 PVC manifests
• Added: ~20 lines of aws s3 cp in deploy job
• Modified: 8 service yamls + 1 job template (PVC → initContainer+emptyDir)
• Image rebuilds: 0
• L40S autoscale for PVC writes: eliminated