cc209aec7e
Eliminate foxhunt-binaries and training-binaries PVCs — services and training jobs now fetch binaries from MinIO via rclone initContainers. This removes L40S GPU autoscale-for-PVC-writes, removes RWO node affinity constraints, and requires zero image rebuilds. Changes: - .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with aws s3 cp to MinIO (~25 lines) - 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir - Training job template: MinIO rclone fetch replaces PVC copy - Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml - Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block - Replace pod-writer-deploy skill with MinIO-based deploy skill Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
535 lines
17 KiB
Markdown
535 lines
17 KiB
Markdown
# MinIO Binary Distribution Implementation Plan
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Replace PVC-based binary distribution with MinIO S3 fetch, eliminating the L40S autoscale-for-PVC-writes problem.
|
|
|
|
**Architecture:** Deploy job uploads compiled binaries to MinIO via `aws s3 cp`. Service pods and training jobs fetch their binary at startup via rclone initContainer into an emptyDir. Two binary PVCs are deleted.
|
|
|
|
**Tech Stack:** MinIO (in-cluster S3), rclone (in runtime images), aws-cli (in infra-runner)
|
|
|
|
---
|
|
|
|
### Task 1: Replace binary-writer pods with MinIO upload in deploy job
|
|
|
|
**Files:**
|
|
- Modify: `.gitlab-ci.yml:1564-1681` (deploy job binary section)
|
|
|
|
**Step 1: Delete PVC apply lines and entire binary-writer block**
|
|
|
|
In `.gitlab-ci.yml`, find and delete these two PVC apply lines (around line 1565-1566):
|
|
```yaml
|
|
- kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml
|
|
- kubectl apply -f infra/k8s/training/training-binaries-pvc.yaml
|
|
```
|
|
|
|
Then delete the entire block from `# ── Step 1: Copy binaries to PVCs` through `echo "=== Binary deployment complete ==="` (lines ~1567-1681). This removes all binary-writer/training-binary-writer pod logic.
|
|
|
|
**Step 2: Add MinIO upload block**
|
|
|
|
Insert in place of the deleted block (same indentation level as other `- |` script blocks in the deploy job):
|
|
|
|
```yaml
|
|
# ── Step 1: Upload binaries to MinIO (replaces PVC binary-writer pods) ──
|
|
- |
|
|
if [ -d "build-out/services" ] || [ -d "build-out/training" ] || [ -d "web-dashboard/dist" ]; then
|
|
set -e
|
|
echo "=== Uploading binaries to MinIO ==="
|
|
export AWS_ACCESS_KEY_ID="${MINIO_ACCESS_KEY}"
|
|
export AWS_SECRET_ACCESS_KEY="${MINIO_SECRET_KEY}"
|
|
S3="aws s3 --endpoint-url https://minio.foxhunt.svc.cluster.local:9000 --no-verify-ssl"
|
|
|
|
if [ -d "build-out/services" ]; then
|
|
$S3 cp build-out/services/ s3://foxhunt-binaries/services/ --recursive
|
|
echo "Service binaries uploaded to MinIO"
|
|
fi
|
|
if [ -d "build-out/training" ]; then
|
|
$S3 cp build-out/training/ s3://foxhunt-binaries/training/ --recursive
|
|
echo "Training binaries uploaded to MinIO"
|
|
fi
|
|
if [ -d "web-dashboard/dist" ]; then
|
|
$S3 cp web-dashboard/dist/ s3://foxhunt-binaries/static/ --recursive
|
|
echo "Web dashboard assets uploaded to MinIO"
|
|
fi
|
|
echo "=== MinIO upload complete ==="
|
|
else
|
|
echo "No new binaries — compile jobs did not run"
|
|
fi
|
|
```
|
|
|
|
**Step 3: Verify YAML syntax**
|
|
|
|
Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo OK`
|
|
Expected: `OK`
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add .gitlab-ci.yml
|
|
git commit -m "refactor(ci): replace PVC binary-writers with MinIO upload
|
|
|
|
Deploy job now uploads binaries to MinIO via aws-cli (already in
|
|
infra-runner image). Eliminates binary-writer and training-binary-writer
|
|
pods, removing the L40S autoscale trigger for PVC writes."
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: Update trading-service.yaml — PVC to initContainer+emptyDir
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/trading-service.yaml`
|
|
|
|
**Step 1: Add initContainer block**
|
|
|
|
Insert after `nodeSelector:` / before `containers:` (between lines 39 and 40):
|
|
|
|
```yaml
|
|
initContainers:
|
|
- name: fetch-binary
|
|
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
|
|
command: ["/bin/sh", "-c"]
|
|
args:
|
|
- |
|
|
set -e
|
|
rclone copyto \
|
|
":s3:foxhunt-binaries/services/trading_service" \
|
|
"/binaries/trading_service" \
|
|
--s3-provider=Minio \
|
|
--s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
|
|
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
|
|
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
|
|
--s3-no-check-bucket \
|
|
--no-check-certificate
|
|
chmod +x /binaries/trading_service
|
|
echo "Fetched trading_service ($(stat -c%s /binaries/trading_service) bytes)"
|
|
env:
|
|
- name: MINIO_ACCESS_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: minio-credentials
|
|
key: access-key
|
|
- name: MINIO_SECRET_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: minio-credentials
|
|
key: secret-key
|
|
volumeMounts:
|
|
- name: binaries
|
|
mountPath: /binaries
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 64Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 128Mi
|
|
```
|
|
|
|
**Step 2: Replace PVC volume with emptyDir**
|
|
|
|
Replace:
|
|
```yaml
|
|
- name: binaries
|
|
persistentVolumeClaim:
|
|
claimName: foxhunt-binaries
|
|
readOnly: true
|
|
```
|
|
|
|
With:
|
|
```yaml
|
|
- name: binaries
|
|
emptyDir:
|
|
sizeLimit: 200Mi
|
|
```
|
|
|
|
**Step 3: Remove readOnly from main container's binaries volumeMount**
|
|
|
|
The main container `volumeMounts` has `readOnly: true` on the binaries mount. The initContainer needs write access. Change:
|
|
```yaml
|
|
- name: binaries
|
|
mountPath: /binaries
|
|
readOnly: true
|
|
```
|
|
To:
|
|
```yaml
|
|
- name: binaries
|
|
mountPath: /binaries
|
|
readOnly: true
|
|
```
|
|
|
|
Actually — the initContainer writes, the main container reads. emptyDir supports this: the initContainer writes, then terminates, then the main container mounts the same emptyDir read-only. Keep `readOnly: true` on the main container's mount. The initContainer's mount does NOT have `readOnly: true`.
|
|
|
|
**Step 4: Verify YAML**
|
|
|
|
Run: `python3 -c "import yaml; list(yaml.safe_load_all(open('infra/k8s/services/trading-service.yaml')))" && echo OK`
|
|
Expected: `OK`
|
|
|
|
---
|
|
|
|
### Task 3: Update api-gateway.yaml
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/api-gateway.yaml`
|
|
|
|
Same pattern as Task 2. Binary name: `api_gateway`
|
|
|
|
**Step 1: Add initContainer** (between `nodeSelector:` and `containers:`, after line 40)
|
|
|
|
Same initContainer as Task 2 but with binary name `api_gateway`:
|
|
- `rclone copyto ":s3:foxhunt-binaries/services/api_gateway" "/binaries/api_gateway"`
|
|
- `chmod +x /binaries/api_gateway`
|
|
- `echo "Fetched api_gateway ..."`
|
|
|
|
**Step 2: Replace PVC volume with emptyDir** (lines 122-126)
|
|
|
|
Replace:
|
|
```yaml
|
|
- name: binaries
|
|
persistentVolumeClaim:
|
|
claimName: foxhunt-binaries
|
|
readOnly: true
|
|
```
|
|
With:
|
|
```yaml
|
|
- name: binaries
|
|
emptyDir:
|
|
sizeLimit: 200Mi
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: Update broker-gateway.yaml
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/broker-gateway.yaml`
|
|
|
|
Same pattern. Binary name: `broker_gateway_service`
|
|
|
|
**Step 1: Add initContainer** (between `nodeSelector:` line 39 and `containers:` line 40)
|
|
|
|
Binary: `broker_gateway_service`
|
|
|
|
**Step 2: Replace PVC volume with emptyDir** (lines 116-120)
|
|
|
|
---
|
|
|
|
### Task 5: Update ml-training-service.yaml
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/ml-training-service.yaml`
|
|
|
|
Binary name: `ml_training_service`. This service already has `minio-credentials` env vars and `minio-ca` volume.
|
|
|
|
**Step 1: Add initContainer** (between `nodeSelector:` line 82 and `containers:` line 83)
|
|
|
|
Same pattern. Binary: `ml_training_service`. Command includes `"serve"` arg — that's on the main container, not the initContainer.
|
|
|
|
**Step 2: Replace PVC volume with emptyDir** (lines 166-170)
|
|
|
|
Replace:
|
|
```yaml
|
|
- name: binaries
|
|
persistentVolumeClaim:
|
|
claimName: foxhunt-binaries
|
|
readOnly: true
|
|
```
|
|
With:
|
|
```yaml
|
|
- name: binaries
|
|
emptyDir:
|
|
sizeLimit: 200Mi
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: Update backtesting-service.yaml
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/backtesting-service.yaml`
|
|
|
|
Binary name: `backtesting_service`
|
|
|
|
**Step 1: Add initContainer** (between `nodeSelector:` line 39 and `containers:` line 40)
|
|
**Step 2: Replace PVC volume with emptyDir** (lines 105-109)
|
|
|
|
---
|
|
|
|
### Task 7: Update trading-agent-service.yaml
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/trading-agent-service.yaml`
|
|
|
|
Binary name: `trading_agent_service`
|
|
|
|
**Step 1: Add initContainer** (between `nodeSelector:` line 39 and `containers:` line 40)
|
|
**Step 2: Replace PVC volume with emptyDir** (lines 103-107)
|
|
|
|
---
|
|
|
|
### Task 8: Update data-acquisition-service.yaml
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/data-acquisition-service.yaml`
|
|
|
|
Binary name: `data_acquisition_service`
|
|
|
|
**Step 1: Add initContainer** (between `nodeSelector:` line 39 and `containers:` line 40)
|
|
**Step 2: Replace PVC volume with emptyDir** (lines 96-100)
|
|
|
|
---
|
|
|
|
### Task 9: Update web-gateway.yaml (special — needs static assets too)
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/services/web-gateway.yaml`
|
|
|
|
Binary name: `web-gateway`. Additionally fetches static dashboard assets.
|
|
|
|
**Step 1: Add initContainer with dual fetch**
|
|
|
|
Insert between `nodeSelector:` line 39 and `containers:` line 40:
|
|
|
|
```yaml
|
|
initContainers:
|
|
- name: fetch-binary
|
|
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
|
|
command: ["/bin/sh", "-c"]
|
|
args:
|
|
- |
|
|
set -e
|
|
RCLONE_FLAGS="--s3-provider=Minio --s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 --s3-access-key-id=${MINIO_ACCESS_KEY} --s3-secret-access-key=${MINIO_SECRET_KEY} --s3-no-check-bucket --no-check-certificate"
|
|
rclone copyto ":s3:foxhunt-binaries/services/web-gateway" "/binaries/web-gateway" $RCLONE_FLAGS
|
|
chmod +x /binaries/web-gateway
|
|
echo "Fetched web-gateway ($(stat -c%s /binaries/web-gateway) bytes)"
|
|
rclone copy ":s3:foxhunt-binaries/static/" "/binaries/static/" $RCLONE_FLAGS
|
|
echo "Fetched static assets"
|
|
env:
|
|
- name: MINIO_ACCESS_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: minio-credentials
|
|
key: access-key
|
|
- name: MINIO_SECRET_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: minio-credentials
|
|
key: secret-key
|
|
volumeMounts:
|
|
- name: binaries
|
|
mountPath: /binaries
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 64Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 256Mi
|
|
```
|
|
|
|
Note: `sizeLimit` on the emptyDir should be `500Mi` for web-gateway (binary + static assets).
|
|
|
|
**Step 2: Replace PVC volume with emptyDir** (lines 102-106)
|
|
|
|
Replace:
|
|
```yaml
|
|
- name: binaries
|
|
persistentVolumeClaim:
|
|
claimName: foxhunt-binaries
|
|
readOnly: true
|
|
```
|
|
With:
|
|
```yaml
|
|
- name: binaries
|
|
emptyDir:
|
|
sizeLimit: 500Mi
|
|
```
|
|
|
|
---
|
|
|
|
### Task 10: Commit all service yaml changes
|
|
|
|
```bash
|
|
git add infra/k8s/services/
|
|
git commit -m "refactor(k8s): replace PVC mounts with MinIO rclone initContainers
|
|
|
|
All 8 service deployments now fetch their binary from MinIO at pod
|
|
startup via rclone initContainer into emptyDir. Eliminates dependency
|
|
on foxhunt-binaries PVC (RWO node affinity constraint).
|
|
|
|
web-gateway additionally fetches static dashboard assets."
|
|
```
|
|
|
|
---
|
|
|
|
### Task 11: Update training job-template.yaml
|
|
|
|
**Files:**
|
|
- Modify: `infra/k8s/training/job-template.yaml`
|
|
|
|
**Step 1: Replace existing fetch-binary initContainer**
|
|
|
|
The current initContainer (lines 41-73) copies from `training-binaries` PVC. Replace entire initContainer with:
|
|
|
|
```yaml
|
|
initContainers:
|
|
- name: fetch-binary
|
|
image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
|
|
command: ["/bin/sh", "-c"]
|
|
args:
|
|
- |
|
|
set -e
|
|
rclone copyto \
|
|
":s3:foxhunt-binaries/training/$(TRAINING_BINARY)" \
|
|
"/binaries/$(TRAINING_BINARY)" \
|
|
--s3-provider=Minio \
|
|
--s3-endpoint=https://minio.foxhunt.svc.cluster.local:9000 \
|
|
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
|
|
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
|
|
--s3-no-check-bucket \
|
|
--no-check-certificate
|
|
chmod +x "/binaries/$(TRAINING_BINARY)"
|
|
echo "Fetched $(TRAINING_BINARY) ($(stat -c%s /binaries/$(TRAINING_BINARY)) bytes)"
|
|
env:
|
|
- name: TRAINING_BINARY
|
|
value: train_baseline_supervised
|
|
- name: MINIO_ACCESS_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: minio-credentials
|
|
key: access-key
|
|
- name: MINIO_SECRET_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: minio-credentials
|
|
key: secret-key
|
|
volumeMounts:
|
|
- name: binaries
|
|
mountPath: /binaries
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 64Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 128Mi
|
|
```
|
|
|
|
**Step 2: Remove training-binaries-src volume**
|
|
|
|
Delete from volumes section (lines 122-127):
|
|
```yaml
|
|
- name: training-binaries-src
|
|
persistentVolumeClaim:
|
|
claimName: training-binaries
|
|
readOnly: true
|
|
```
|
|
|
|
**Step 3: Verify YAML**
|
|
|
|
Run: `python3 -c "import yaml; list(yaml.safe_load_all(open('infra/k8s/training/job-template.yaml')))" && echo OK`
|
|
Expected: `OK`
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add infra/k8s/training/job-template.yaml
|
|
git commit -m "refactor(k8s): training job fetches binary from MinIO
|
|
|
|
Replace PVC-based training binary distribution with MinIO rclone
|
|
initContainer. Training jobs no longer need training-binaries PVC,
|
|
eliminating L40S GPU node autoscale just for binary distribution."
|
|
```
|
|
|
|
---
|
|
|
|
### Task 12: Delete PVC manifests
|
|
|
|
**Files:**
|
|
- Delete: `infra/k8s/storage/foxhunt-binaries-pvc.yaml`
|
|
- Delete: `infra/k8s/training/training-binaries-pvc.yaml`
|
|
|
|
**Step 1: Delete the files**
|
|
|
|
```bash
|
|
rm infra/k8s/storage/foxhunt-binaries-pvc.yaml
|
|
rm infra/k8s/training/training-binaries-pvc.yaml
|
|
```
|
|
|
|
**Step 2: Verify no remaining references to deleted PVCs**
|
|
|
|
Run: `grep -r "foxhunt-binaries" infra/ .gitlab-ci.yml --include='*.yaml' --include='*.yml'`
|
|
Expected: Only MinIO bucket references (`s3://foxhunt-binaries/`), no PVC references.
|
|
|
|
Run: `grep -r "training-binaries" infra/ .gitlab-ci.yml --include='*.yaml' --include='*.yml'`
|
|
Expected: No results (the PVC name is gone).
|
|
|
|
**Step 3: Commit**
|
|
|
|
```bash
|
|
git add -A infra/k8s/storage/foxhunt-binaries-pvc.yaml infra/k8s/training/training-binaries-pvc.yaml
|
|
git commit -m "chore(k8s): delete foxhunt-binaries and training-binaries PVCs
|
|
|
|
These PVCs are replaced by MinIO-based binary distribution.
|
|
Service pods and training jobs now fetch binaries from MinIO
|
|
via rclone initContainers.
|
|
|
|
NOTE: Run 'kubectl delete pvc foxhunt-binaries training-binaries -n foxhunt'
|
|
on the cluster to reclaim the block storage volumes."
|
|
```
|
|
|
|
---
|
|
|
|
### Task 13: Final verification
|
|
|
|
**Step 1: Grep for any remaining PVC binary references**
|
|
|
|
Run: `grep -rn "claimName: foxhunt-binaries\|claimName: training-binaries" infra/ .gitlab-ci.yml`
|
|
Expected: No results.
|
|
|
|
**Step 2: Grep for binary-writer references**
|
|
|
|
Run: `grep -rn "binary-writer\|binary_writer" .gitlab-ci.yml`
|
|
Expected: No results.
|
|
|
|
**Step 3: Verify all service yamls have initContainers**
|
|
|
|
Run: `grep -l "fetch-binary" infra/k8s/services/*.yaml | wc -l`
|
|
Expected: `8`
|
|
|
|
**Step 4: Verify training job has MinIO fetch**
|
|
|
|
Run: `grep "foxhunt-binaries/training" infra/k8s/training/job-template.yaml`
|
|
Expected: Match on the rclone line.
|
|
|
|
---
|
|
|
|
## Summary of Changes
|
|
|
|
| File | Action | Lines ~Δ |
|
|
|------|--------|----------|
|
|
| `.gitlab-ci.yml` | Replace binary-writer pods with `aws s3 cp` | -100/+20 |
|
|
| `infra/k8s/services/trading-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
|
|
| `infra/k8s/services/api-gateway.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
|
|
| `infra/k8s/services/broker-gateway.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
|
|
| `infra/k8s/services/ml-training-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
|
|
| `infra/k8s/services/backtesting-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
|
|
| `infra/k8s/services/trading-agent-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
|
|
| `infra/k8s/services/data-acquisition-service.yaml` | Add initContainer, PVC→emptyDir | +30/-4 |
|
|
| `infra/k8s/services/web-gateway.yaml` | Add initContainer (binary+static), PVC→emptyDir | +35/-4 |
|
|
| `infra/k8s/training/job-template.yaml` | Replace PVC initContainer with MinIO | +30/-20 |
|
|
| `infra/k8s/storage/foxhunt-binaries-pvc.yaml` | Delete | -19 |
|
|
| `infra/k8s/training/training-binaries-pvc.yaml` | Delete | -19 |
|
|
|
|
**Total: 12 files, ~+275/-210 lines, 0 image rebuilds**
|
|
|
|
## Post-Deploy Cluster Cleanup
|
|
|
|
After deploying this change, manually delete the orphaned PVCs:
|
|
```bash
|
|
kubectl delete pvc foxhunt-binaries training-binaries -n foxhunt
|
|
```
|
|
|
|
Add `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` as CI/CD variables in GitLab
|
|
(Settings → CI/CD → Variables), or source them from the cluster secret during the deploy job.
|