# MinIO Binary Distribution Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Replace PVC-based binary distribution with MinIO S3 fetch, eliminating the L40S autoscale-for-PVC-writes problem.

**Architecture:** Deploy job uploads compiled binaries to MinIO via `aws s3 cp`. Service pods and training jobs fetch their binary at startup via rclone initContainer into an emptyDir. Two binary PVCs are deleted.

**Tech Stack:** MinIO (in-cluster S3), rclone (in runtime images), aws-cli (in infra-runner)

---

### Task 1: Replace binary-writer pods with MinIO upload in deploy job

**Files:**
- Modify: `.gitlab-ci.yml:1564-1681` (deploy job binary section)

**Step 1: Delete PVC apply lines and entire binary-writer block**

In `.gitlab-ci.yml`, find and delete these two PVC apply lines (around line 1565-1566):
```yaml
    - kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml
    - kubectl apply -f infra/k8s/training/training-binaries-pvc.yaml
```

Then delete the entire block from `# ── Step 1: Copy binaries to PVCs` through `echo "=== Binary deployment complete ==="` (lines ~1567-1681). This removes all binary-writer/training-binary-writer pod logic.

**Step 2: Add MinIO upload block**

Insert in place of the deleted block (same indentation level as other `- |` script blocks in the deploy job):

```yaml
    # ── Step 1: Upload binaries to MinIO (replaces PVC binary-writer pods) ──
    - |
      if [ -d "build-out/services" ] || [ -d "build-out/training" ] || [ -d "web-dashboard/dist" ]; then
        set -e
        echo "=== Uploading binaries to MinIO ==="
        export AWS_ACCESS_KEY_ID="${MINIO_ACCESS_KEY}"
        export AWS_SECRET_ACCESS_KEY="${MINIO_SECRET_KEY}"
        S3="aws s3 --endpoint-url https://minio.foxhunt.svc.cluster.local:9000 --no-verify-ssl"

        if [ -d "build-out/services" ]; then
          $S3 cp build-out/services/ s3://foxhunt-binaries/services/ --recursive
          echo "Service binaries uploaded to MinIO"
        fi
        if [ -d "build-out/training" ]; then
          $S3 cp build-out/training/ s3://foxhunt-binaries/training/ --recursive
          echo "Training binaries uploaded to MinIO"
        fi
        if [ -d "web-dashboard/dist" ]; then
          $S3 cp web-dashboard/dist/ s3://foxhunt-binaries/static/ --recursive
          echo "Web dashboard assets uploaded to MinIO"
        fi
        echo "=== MinIO upload complete ==="
      else
        echo "No new binaries — compile jobs did not run"
      fi
```

**Step 3: Verify YAML syntax**

Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo OK`
Expected: `OK`

**Step 4: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "refactor(ci): replace PVC binary-writers with MinIO upload

Deploy job now uploads binaries to MinIO via aws-cli (already in
infra-runner image). Eliminates binary-writer and training-binary-writer
pods, removing the L40S autoscale trigger for PVC writes."
```

---

### Task 2: Update trading-service.yaml — PVC to initContainer+emptyDir

**Files:**
- Modify: `infra/k8s/services/trading-service.yaml`

**Step 1: Add initContainer block**

Insert after `nodeSelector:` / before `containers:` (between lines 39 and 40):

```yaml
      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              rclone copyto \
                ":s3:foxhunt-binaries/services/trading_service" \
                "/binaries/trading_service" \
                --s3-provider=Minio \
                --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
                --s3-no-check-bucket \
                --no-check-certificate
              chmod +x /binaries/trading_service
              echo "Fetched trading_service ($(stat -c%s /binaries/trading_service) bytes)"
          env:
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
```

**Step 2: Replace PVC volume with emptyDir**

Replace:
```yaml
        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true
```

With:
```yaml
        - name: binaries
          emptyDir:
            sizeLimit: 200Mi
```

**Step 3: Remove readOnly from main container's binaries volumeMount**

The main container `volumeMounts` has `readOnly: true` on the binaries mount. The initContainer needs write access. Change:
```yaml
            - name: binaries
              mountPath: /binaries
              readOnly: true
```
To:
```yaml
            - name: binaries
              mountPath: /binaries
              readOnly: true
```

Actually — the initContainer writes, the main container reads. emptyDir supports this: the initContainer writes, then terminates, then the main container mounts the same emptyDir read-only. Keep `readOnly: true` on the main container's mount. The initContainer's mount does NOT have `readOnly: true`.

**Step 4: Verify YAML**

Run: `python3 -c "import yaml; list(yaml.safe_load_all(open('infra/k8s/services/trading-service.yaml')))" && echo OK`
Expected: `OK`

---

### Task 3: Update api-gateway.yaml

**Files:**
- Modify: `infra/k8s/services/api-gateway.yaml`

Same pattern as Task 2. Binary name: `api_gateway`

**Step 1: Add initContainer** (between `nodeSelector:` and `containers:`, after line 40)

Same initContainer as Task 2 but with binary name `api_gateway`:
- `rclone copyto ":s3:foxhunt-binaries/services/api_gateway" "/binaries/api_gateway"`
- `chmod +x /binaries/api_gateway`
- `echo "Fetched api_gateway ..."`

**Step 2: Replace PVC volume with emptyDir** (lines 122-126)

Replace:
```yaml
        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true
```
With:
```yaml
        - name: binaries
          emptyDir:
            sizeLimit: 200Mi
```

---

### Task 4: Update broker-gateway.yaml

**Files:**
- Modify: `infra/k8s/services/broker-gateway.yaml`

Same pattern. Binary name: `broker_gateway_service`

**Step 1: Add initContainer** (between `nodeSelector:` line 39 and `containers:` line 40)

Binary: `broker_gateway_service`

**Step 2: Replace PVC volume with emptyDir** (lines 116-120)

---

### Task 5: Update ml-training-service.yaml

**Files:**
- Modify: `infra/k8s/services/ml-training-service.yaml`

Binary name: `ml_training_service`. This service already has `minio-credentials` env vars and `minio-ca` volume.

**Step 1: Add initContainer** (between `nodeSelector:` line 82 and `containers:` line 83)

Same pattern. Binary: `ml_training_service`. Command includes `"serve"` arg — that's on the main container, not the initContainer.

**Step 2: Replace PVC volume with emptyDir** (lines 166-170)

Replace:
```yaml
        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true
```
With:
```yaml
        - name: binaries
          emptyDir:
            sizeLimit: 200Mi
```

---

### Task 6: Update backtesting-service.yaml

**Files:**
- Modify: `infra/k8s/services/backtesting-service.yaml`

Binary name: `backtesting_service`

**Step 1: Add initContainer** (between `nodeSelector:` line 39 and `containers:` line 40)
**Step 2: Replace PVC volume with emptyDir** (lines 105-109)

---

### Task 7: Update trading-agent-service.yaml

**Files:**
- Modify: `infra/k8s/services/trading-agent-service.yaml`

Binary name: `trading_agent_service`

**Step 1: Add initContainer** (between `nodeSelector:` line 39 and `containers:` line 40)
**Step 2: Replace PVC volume with emptyDir** (lines 103-107)

---

### Task 8: Update data-acquisition-service.yaml

**Files:**
- Modify: `infra/k8s/services/data-acquisition-service.yaml`

Binary name: `data_acquisition_service`

**Step 1: Add initContainer** (between `nodeSelector:` line 39 and `containers:` line 40)
**Step 2: Replace PVC volume with emptyDir** (lines 96-100)

---

### Task 9: Update web-gateway.yaml (special — needs static assets too)

**Files:**
- Modify: `infra/k8s/services/web-gateway.yaml`

Binary name: `web-gateway`. Additionally fetches static dashboard assets.

**Step 1: Add initContainer with dual fetch**

Insert between `nodeSelector:` line 39 and `containers:` line 40:

```yaml
      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              RCLONE_FLAGS="--s3-provider=Minio --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 --s3-access-key-id=${MINIO_ACCESS_KEY} --s3-secret-access-key=${MINIO_SECRET_KEY} --s3-no-check-bucket --no-check-certificate"
              rclone copyto ":s3:foxhunt-binaries/services/web-gateway" "/binaries/web-gateway" $RCLONE_FLAGS
              chmod +x /binaries/web-gateway
              echo "Fetched web-gateway ($(stat -c%s /binaries/web-gateway) bytes)"
              rclone copy ":s3:foxhunt-binaries/static/" "/binaries/static/" $RCLONE_FLAGS
              echo "Fetched static assets"
          env:
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 256Mi
```

Note: `sizeLimit` on the emptyDir should be `500Mi` for web-gateway (binary + static assets).

**Step 2: Replace PVC volume with emptyDir** (lines 102-106)

Replace:
```yaml
        - name: binaries
          persistentVolumeClaim:
            claimName: foxhunt-binaries
            readOnly: true
```
With:
```yaml
        - name: binaries
          emptyDir:
            sizeLimit: 500Mi
```

---

### Task 10: Commit all service yaml changes

```bash
git add infra/k8s/services/
git commit -m "refactor(k8s): replace PVC mounts with MinIO rclone initContainers

All 8 service deployments now fetch their binary from MinIO at pod
startup via rclone initContainer into emptyDir. Eliminates dependency
on foxhunt-binaries PVC (RWO node affinity constraint).

web-gateway additionally fetches static dashboard assets."
```

---

### Task 11: Update training job-template.yaml

**Files:**
- Modify: `infra/k8s/training/job-template.yaml`

**Step 1: Replace existing fetch-binary initContainer**

The current initContainer (lines 41-73) copies from `training-binaries` PVC. Replace entire initContainer with:

```yaml
      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              rclone copyto \
                ":s3:foxhunt-binaries/training/$(TRAINING_BINARY)" \
                "/binaries/$(TRAINING_BINARY)" \
                --s3-provider=Minio \
                --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
                --s3-no-check-bucket \
                --no-check-certificate
              chmod +x "/binaries/$(TRAINING_BINARY)"
              echo "Fetched $(TRAINING_BINARY) ($(stat -c%s /binaries/$(TRAINING_BINARY)) bytes)"
          env:
            - name: TRAINING_BINARY
              value: train_baseline_supervised
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: access-key
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
```

**Step 2: Remove training-binaries-src volume**

Delete from volumes section (lines 122-127):
```yaml
        - name: training-binaries-src
          persistentVolumeClaim:
            claimName: training-binaries
            readOnly: true
```

**Step 3: Verify YAML**

Run: `python3 -c "import yaml; list(yaml.safe_load_all(open('infra/k8s/training/job-template.yaml')))" && echo OK`
Expected: `OK`

**Step 4: Commit**

```bash
git add infra/k8s/training/job-template.yaml
git commit -m "refactor(k8s): training job fetches binary from MinIO

Replace PVC-based training binary distribution with MinIO rclone
initContainer. Training jobs no longer need training-binaries PVC,
eliminating L40S GPU node autoscale just for binary distribution."
```

---

### Task 12: Delete PVC manifests

**Files:**
- Delete: `infra/k8s/storage/foxhunt-binaries-pvc.yaml`
- Delete: `infra/k8s/training/training-binaries-pvc.yaml`

**Step 1: Delete the files**

```bash
rm infra/k8s/storage/foxhunt-binaries-pvc.yaml
rm infra/k8s/training/training-binaries-pvc.yaml
```

**Step 2: Verify no remaining references to deleted PVCs**

Run: `grep -r "foxhunt-binaries" infra/ .gitlab-ci.yml --include='*.yaml' --include='*.yml'`
Expected: Only MinIO bucket references (`s3://foxhunt-binaries/`), no PVC references.

Run: `grep -r "training-binaries" infra/ .gitlab-ci.yml --include='*.yaml' --include='*.yml'`
Expected: No results (the PVC name is gone).

**Step 3: Commit**

```bash
git add -A infra/k8s/storage/foxhunt-binaries-pvc.yaml infra/k8s/training/training-binaries-pvc.yaml
git commit -m "chore(k8s): delete foxhunt-binaries and training-binaries PVCs

These PVCs are replaced by MinIO-based binary distribution.
Service pods and training jobs now fetch binaries from MinIO
via rclone initContainers.

NOTE: Run 'kubectl delete pvc foxhunt-binaries training-binaries -n foxhunt'
on the cluster to reclaim the block storage volumes."
```

---

### Task 13: Final verification

**Step 1: Grep for any remaining PVC binary references**

Run: `grep -rn "claimName: foxhunt-binaries\|claimName: training-binaries" infra/ .gitlab-ci.yml`
Expected: No results.

**Step 2: Grep for binary-writer references**

Run: `grep -rn "binary-writer\|binary_writer" .gitlab-ci.yml`
Expected: No results.

**Step 3: Verify all service yamls have initContainers**

Run: `grep -l "fetch-binary" infra/k8s/services/*.yaml | wc -l`
Expected: `8`

**Step 4: Verify training job has MinIO fetch**

Run: `grep "foxhunt-binaries/training" infra/k8s/training/job-template.yaml`
Expected: Match on the rclone line.

---

## Summary of Changes

| File | Action | Lines ~Δ |
|------|--------|----------|
| `.gitlab-ci.yml` | Replace binary-writer pods with `aws s3 cp` | -100/+20 |
| `infra/k8s/services/trading-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
| `infra/k8s/services/api-gateway.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
| `infra/k8s/services/broker-gateway.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
| `infra/k8s/services/ml-training-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
| `infra/k8s/services/backtesting-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
| `infra/k8s/services/trading-agent-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
| `infra/k8s/services/data-acquisition-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
| `infra/k8s/services/web-gateway.yaml` | Add initContainer (binary+static), PVC→emptyDir | +35/-4 |
| `infra/k8s/training/job-template.yaml` | Replace PVC initContainer with MinIO | +30/-20 |
| `infra/k8s/storage/foxhunt-binaries-pvc.yaml` | Delete | -19 |
| `infra/k8s/training/training-binaries-pvc.yaml` | Delete | -19 |

**Total: 12 files, ~+275/-210 lines, 0 image rebuilds**

## Post-Deploy Cluster Cleanup

After deploying this change, manually delete the orphaned PVCs:
```bash
kubectl delete pvc foxhunt-binaries training-binaries -n foxhunt
```

Add `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` as CI/CD variables in GitLab
(Settings → CI/CD → Variables), or source them from the cluster secret during the deploy job.