foxhunt/docs/plans/2026-03-04-minio-binary-distribution-design.md

# MinIO Binary Distribution — Eliminate PVC Binary Writers

**Date:** 2026-03-04
**Status:** Approved
**Scope:** Replace PVC-based binary distribution with MinIO S3 fetch pattern

## Problem

The current deploy cycle uses `kubectl cp` via temporary busybox writer pods to copy
compiled binaries onto RWO PVCs. Two PVCs exist:

- `foxhunt-binaries` (foxhunt node) — service binaries
- `training-binaries` (ci-training node) — training binaries

The training PVC writer triggers L40S GPU autoscale from zero just to receive a binary
copy — expensive and slow. RWO PVCs also create node-affinity constraints that prevent
cross-pool flexibility.

## Solution

Use MinIO (already deployed on `platform` node) as the binary distribution layer.

**Flow:**
```
compile jobs → GitLab artifacts (source of truth)
                    ↓
              deploy job: aws s3 cp → MinIO foxhunt-binaries bucket
                    ↓
              service pods: rclone initContainer → emptyDir
              training jobs: rclone initContainer → emptyDir
              external GPUs: GitLab artifacts API
```

**Key insight:** Zero image rebuilds. `aws-cli` is already in infra-runner, `rclone`
is already in both runtime images.

## Changes

### 1. Deploy Job (`.gitlab-ci.yml`)

**Delete:** ~115 lines of binary-writer pod logic (lines 1567-1681)
**Add:** ~20 lines of `aws s3 cp` to MinIO

```yaml
# Upload binaries to MinIO (replaces binary-writer pods)
- |
  if [ -d "build-out/services" ] || [ -d "build-out/training" ]; then
    export AWS_ACCESS_KEY_ID="$MINIO_ACCESS_KEY"
    export AWS_SECRET_ACCESS_KEY="$MINIO_SECRET_KEY"
    S3="aws s3 --endpoint-url https://minio.foxhunt.svc.cluster.local:9000 --no-verify-ssl"

    if [ -d "build-out/services" ]; then
      $S3 cp build-out/services/ s3://foxhunt-binaries/services/ --recursive
      echo "Service binaries → MinIO"
    fi
    if [ -d "build-out/training" ]; then
      $S3 cp build-out/training/ s3://foxhunt-binaries/training/ --recursive
      echo "Training binaries → MinIO"
    fi
    if [ -d "web-dashboard/dist" ]; then
      $S3 cp web-dashboard/dist/ s3://foxhunt-binaries/static/ --recursive
      echo "Web dashboard assets → MinIO"
    fi
  fi
```

Also remove:
- `kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml`
- `kubectl apply -f infra/k8s/training/training-binaries-pvc.yaml`

Add MinIO credentials as CI variables (`MINIO_ACCESS_KEY`, `MINIO_SECRET_KEY`) or
source from existing `minio-credentials` secret.

### 2. Service Deployments (8 files in `infra/k8s/services/`)

For each service, replace PVC volume mount with initContainer + emptyDir:

```yaml
spec:
  template:
    spec:
      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              rclone copyto \
                ":s3:foxhunt-binaries/services/${SERVICE_NAME}" \
                "/binaries/${SERVICE_NAME}" \
                --s3-provider=Minio \
                --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
                --s3-no-check-bucket \
                --no-check-certificate
              chmod +x "/binaries/${SERVICE_NAME}"
              echo "Fetched ${SERVICE_NAME} ($(stat -c%s /binaries/${SERVICE_NAME}) bytes)"
          env:
            - name: SERVICE_NAME
              value: trading_service  # per-service value
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
      # Also fetch web-dashboard static assets for web-gateway
      volumes:
        - name: binaries
          emptyDir:
            sizeLimit: 200Mi  # single stripped binary ~20-50MB
```

Remove from each service:
```yaml
# DELETE these lines:
volumes:
  - name: binaries
    persistentVolumeClaim:
      claimName: foxhunt-binaries
      readOnly: true
```

**web-gateway** additionally needs static assets fetched:
```yaml
# Extra step in web-gateway initContainer:
rclone copy ":s3:foxhunt-binaries/static/" "/binaries/static/" \
  --s3-provider=Minio ...
```

### 3. Training Job Template (`infra/k8s/training/job-template.yaml`)

Replace the PVC-based fetch-binary initContainer with MinIO rclone:

```yaml
initContainers:
  - name: fetch-binary
    image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
    command: ["/bin/sh", "-c"]
    args:
      - |
        rclone copyto \
          ":s3:foxhunt-binaries/training/${TRAINING_BINARY}" \
          "/binaries/${TRAINING_BINARY}" \
          --s3-provider=Minio \
          --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
          --s3-access-key-id=${MINIO_ACCESS_KEY} \
          --s3-secret-access-key=${MINIO_SECRET_KEY} \
          --s3-no-check-bucket \
          --no-check-certificate
        chmod +x "/binaries/${TRAINING_BINARY}"
    env:
      - name: TRAINING_BINARY
        value: train_baseline_supervised
      - name: MINIO_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: minio-credentials
            key: access-key
      - name: MINIO_SECRET_KEY
        valueFrom:
          secretKeyRef:
            name: minio-credentials
            key: secret-key
    volumeMounts:
      - name: binaries
        mountPath: /binaries
```

Remove `training-binaries-src` volume (PVC mount).

### 4. Files to Delete

- `infra/k8s/storage/foxhunt-binaries-pvc.yaml`
- `infra/k8s/training/training-binaries-pvc.yaml`

### 5. Files Unchanged

- `training-data-pvc.yaml` — keep as-is
- `sccache-pvc.yaml` — keep
- `minio-data-new` (in minio.yaml) — keep
- CI training jobs (`.train-rl-base`, `.train-validate-base`) — already use GL artifacts via `needs:`

## Service-to-Binary Mapping

| Service Deployment | Binary Name | Pool |
|--------------------|-------------|------|
| trading-service | trading_service | foxhunt |
| api-gateway | api_gateway | foxhunt |
| broker-gateway | broker_gateway_service | foxhunt |
| ml-training-service | ml_training_service | foxhunt |
| backtesting-service | backtesting_service | foxhunt |
| trading-agent-service | trading_agent_service | foxhunt |
| data-acquisition-service | data_acquisition_service | foxhunt |
| web-gateway | web-gateway | foxhunt |

## MinIO Bucket Layout

```
foxhunt-binaries/
├── services/
│   ├── trading_service
│   ├── api_gateway
│   ├── broker_gateway_service
│   ├── ml_training_service
│   ├── backtesting_service
│   ├── trading_agent_service
│   ├── data_acquisition_service
│   └── web-gateway
├── training/
│   ├── train_baseline_rl
│   ├── train_baseline_supervised
│   ├── evaluate_baseline
│   ├── evaluate_supervised
│   ├── hyperopt_baseline_rl
│   ├── hyperopt_baseline_supervised
│   └── training_uploader
└── static/
    └── (web-dashboard dist files)
```

## External GPU Access

External GPUs download training binaries from GitLab artifacts API:
```bash
curl --header "PRIVATE-TOKEN: $GL_TOKEN" \
  "https://<gitlab>/api/v4/projects/<id>/jobs/artifacts/main/download?job=compile-training" \
  -o artifacts.zip
```

Or from MinIO via Tailscale (if MinIO is exposed).

## Rollback

If MinIO is down during deploy, the deploy job fails fast (aws s3 cp fails).
Service pods with existing emptyDir continue running. New pods fail to start
(initContainer fails). MinIO recovery restores normal operation.

Previous binaries remain in MinIO until overwritten — no expiry, no garbage.

## Net Effect

- **Deleted:** ~115 lines of binary-writer pod logic
- **Deleted:** 2 PVC manifests
- **Added:** ~20 lines of aws s3 cp in deploy job
- **Modified:** 8 service yamls + 1 job template (PVC → initContainer+emptyDir)
- **Image rebuilds:** 0
- **L40S autoscale for PVC writes:** eliminated