chore(infra): delete 8 per-service binary-cache PVCs (replaced by foxhunt-binaries)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,715 @@
|
||||
# Binary Cache PVC Consolidation Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Replace 8 per-service binary-cache PVCs + S3 round-trip with a single shared `foxhunt-binaries` PVC, eliminating initContainers, S3 uploads, and 7 unnecessary PVCs.
|
||||
|
||||
**Architecture:** CI compile jobs produce artifacts. Deploy job creates a temporary writer pod on the foxhunt node, `kubectl cp`s binaries + static assets onto the shared PVC, deletes the writer pod, then rollout-restarts all services. Services mount the shared PVC read-only at `/binaries/` — no initContainer, no S3.
|
||||
|
||||
**Tech Stack:** Kubernetes PVC (scw-bssd RWO), GitLab CI artifacts, kubectl cp, busybox writer pod
|
||||
|
||||
**Design doc:** `docs/plans/2026-03-02-binary-cache-consolidation-design.md`
|
||||
|
||||
---
|
||||
|
||||
## Reference: Services and Binary Names
|
||||
|
||||
| Service YAML | Binary Name | PVC to Delete |
|
||||
|---|---|---|
|
||||
| `api-gateway.yaml` | `api_gateway` | `binary-cache-api-gateway` |
|
||||
| `trading-service.yaml` | `trading_service` | `binary-cache-trading-service` |
|
||||
| `broker-gateway.yaml` | `broker_gateway_service` | `binary-cache-broker-gateway` |
|
||||
| `ml-training-service.yaml` | `ml_training_service` | `binary-cache-ml-training-service` |
|
||||
| `backtesting-service.yaml` | `backtesting_service` | `binary-cache-backtesting-service` |
|
||||
| `trading-agent-service.yaml` | `trading_agent_service` | `binary-cache-trading-agent-service` |
|
||||
| `data-acquisition-service.yaml` | `data_acquisition_service` | `binary-cache-data-acquisition-service` |
|
||||
| `web-gateway.yaml` | `web-gateway` | `binary-cache-web-gateway` |
|
||||
|
||||
**Not modified:** `ib-gateway.yaml` (third-party Docker image), `*-gpu.yaml` overlays (different node pools, keep S3)
|
||||
|
||||
**Special cases:**
|
||||
- `ml-training-service.yaml`: keeps `minio-ca` volume in the container (S3 for model storage, not binary fetch)
|
||||
- `web-gateway.yaml`: initContainer also fetched static assets from S3 — these go into shared PVC at `/binaries/static/`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Create shared PVC YAML
|
||||
|
||||
**Files:**
|
||||
- Create: `infra/k8s/storage/foxhunt-binaries-pvc.yaml`
|
||||
|
||||
**Step 1: Create the PVC manifest**
|
||||
|
||||
```yaml
|
||||
# Shared binary PVC — all foxhunt services mount read-only
|
||||
# CI deploy job writes via temporary binary-writer pod (kubectl cp)
|
||||
# RWO on scw-bssd: all pods on the same foxhunt node can mount
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: foxhunt-binaries
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: foxhunt-binaries
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
```
|
||||
|
||||
**Step 2: Verify YAML is valid**
|
||||
|
||||
Run: `kubectl apply --dry-run=client -f infra/k8s/storage/foxhunt-binaries-pvc.yaml`
|
||||
Expected: `persistentvolumeclaim/foxhunt-binaries created (dry run)`
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/storage/foxhunt-binaries-pvc.yaml
|
||||
git commit -m "feat(infra): add shared foxhunt-binaries PVC (2Gi)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Modify compile-services CI job
|
||||
|
||||
**Files:**
|
||||
- Modify: `.gitlab-ci.yml` (lines 409-458, `compile-services` job)
|
||||
|
||||
**Step 1: Add CI artifacts, remove S3 upload**
|
||||
|
||||
In the `compile-services` job, replace the S3 upload section (lines 445-458) with CI artifacts:
|
||||
|
||||
Remove this block (lines 445-458):
|
||||
```yaml
|
||||
# Upload stripped binaries to MinIO (in-cluster S3)
|
||||
# Trust MinIO CA — rclone (Go) uses system CAs, not RCLONE_CA_CERT for S3.
|
||||
# Container runs non-root, so merge system + MinIO CA into writable /tmp bundle.
|
||||
- cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-bundle.crt 2>/dev/null || true
|
||||
- cat /etc/ssl/minio-ca/ca.crt >> /tmp/ca-bundle.crt 2>/dev/null || true
|
||||
- export SSL_CERT_FILE=/tmp/ca-bundle.crt
|
||||
- export RCLONE_S3_PROVIDER=Minio RCLONE_S3_ENDPOINT=https://minio.foxhunt.svc.cluster.local:9000 RCLONE_S3_REGION=us-east-1
|
||||
- export RCLONE_S3_ACCESS_KEY_ID=$MINIO_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY
|
||||
- |
|
||||
for bin in trading_service api_gateway broker_gateway_service ml_training_service backtesting_service trading_agent_service data_acquisition_service web-gateway; do
|
||||
rclone copyto build-out/services/$bin :s3:foxhunt-binaries/latest/services/$bin
|
||||
done
|
||||
- rclone copy build-out/services/ :s3:foxhunt-binaries/archive/${CI_COMMIT_SHA}/services/
|
||||
- echo "Service binaries uploaded to MinIO"
|
||||
```
|
||||
|
||||
Replace with:
|
||||
```yaml
|
||||
- echo "Service binaries ready as CI artifacts"
|
||||
```
|
||||
|
||||
Add artifacts section at the end of the job:
|
||||
```yaml
|
||||
artifacts:
|
||||
paths:
|
||||
- build-out/services/
|
||||
expire_in: 1 day
|
||||
```
|
||||
|
||||
**Step 2: Verify CI syntax**
|
||||
|
||||
Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"`
|
||||
Expected: `Valid YAML`
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add .gitlab-ci.yml
|
||||
git commit -m "feat(ci): replace S3 upload with CI artifacts in compile-services"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Modify build-web-dashboard CI job
|
||||
|
||||
**Files:**
|
||||
- Modify: `.gitlab-ci.yml` (lines 529-563, `build-web-dashboard` job)
|
||||
|
||||
**Step 1: Add CI artifacts, remove S3 upload**
|
||||
|
||||
In the `build-web-dashboard` job, remove the rclone/S3 section (lines 547-563) and add CI artifacts.
|
||||
|
||||
Remove the `before_script` block (lines 548-550, rclone install):
|
||||
```yaml
|
||||
before_script:
|
||||
- apt-get update && apt-get install -y --no-install-recommends curl unzip ca-certificates
|
||||
- curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip
|
||||
- unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ && rm /tmp/rclone.zip
|
||||
```
|
||||
|
||||
In the `script` section, remove the rclone upload lines (555-563):
|
||||
```yaml
|
||||
# Trust MinIO CA — rclone (Go) uses system CAs, not RCLONE_CA_CERT for S3.
|
||||
# Container runs non-root, so merge system + MinIO CA into writable /tmp bundle.
|
||||
- cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-bundle.crt 2>/dev/null || true
|
||||
- cat /etc/ssl/minio-ca/ca.crt >> /tmp/ca-bundle.crt 2>/dev/null || true
|
||||
- export SSL_CERT_FILE=/tmp/ca-bundle.crt
|
||||
- export RCLONE_S3_PROVIDER=Minio RCLONE_S3_ENDPOINT=https://minio.foxhunt.svc.cluster.local:9000 RCLONE_S3_REGION=us-east-1
|
||||
- export RCLONE_S3_ACCESS_KEY_ID=$MINIO_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$MINIO_SECRET_KEY
|
||||
- rclone sync dist/ :s3:foxhunt-binaries/latest/web-dashboard/dist/
|
||||
- echo "Web dashboard assets uploaded to MinIO"
|
||||
```
|
||||
|
||||
Replace with:
|
||||
```yaml
|
||||
- echo "Web dashboard assets ready as CI artifacts"
|
||||
```
|
||||
|
||||
Add artifacts section:
|
||||
```yaml
|
||||
artifacts:
|
||||
paths:
|
||||
- web-dashboard/dist/
|
||||
expire_in: 1 day
|
||||
```
|
||||
|
||||
**Step 2: Verify CI syntax**
|
||||
|
||||
Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"`
|
||||
Expected: `Valid YAML`
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add .gitlab-ci.yml
|
||||
git commit -m "feat(ci): replace S3 upload with CI artifacts in build-web-dashboard"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Remove write-manifest CI job
|
||||
|
||||
**Files:**
|
||||
- Modify: `.gitlab-ci.yml` (lines 565-597, `write-manifest` job)
|
||||
|
||||
**Step 1: Delete the entire write-manifest job**
|
||||
|
||||
Remove lines 565-597 (the entire `write-manifest` job block including the section header comment). MANIFEST.json was S3-specific tracking — no longer needed.
|
||||
|
||||
**Step 2: Verify CI syntax**
|
||||
|
||||
Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"`
|
||||
Expected: `Valid YAML`
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add .gitlab-ci.yml
|
||||
git commit -m "chore(ci): remove write-manifest job (S3 tracking no longer needed)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Rewrite deploy CI job
|
||||
|
||||
**Files:**
|
||||
- Modify: `.gitlab-ci.yml` (deploy job, currently starting around line 1460)
|
||||
|
||||
**Step 1: Update job dependencies**
|
||||
|
||||
Change the `needs` section from:
|
||||
```yaml
|
||||
needs:
|
||||
- job: write-manifest
|
||||
optional: true
|
||||
```
|
||||
|
||||
To:
|
||||
```yaml
|
||||
needs:
|
||||
- job: compile-services
|
||||
optional: true
|
||||
artifacts: true
|
||||
- job: build-web-dashboard
|
||||
optional: true
|
||||
artifacts: true
|
||||
```
|
||||
|
||||
**Step 2: Replace service deploy section**
|
||||
|
||||
Find the section that applies binary-cache-pvcs.yaml and does rollout restarts (around line 1595-1613). Replace:
|
||||
|
||||
```yaml
|
||||
# Apply service deployments and binary cache PVCs (idempotent — picks up any YAML changes)
|
||||
# Skip training/job-template.yaml (placeholder names, not directly applicable)
|
||||
# Skip storage/training-data-pv.yaml (PVC spec immutable after creation)
|
||||
- kubectl apply -f infra/k8s/services/ -f infra/k8s/storage/binary-cache-pvcs.yaml
|
||||
# Rolling restart triggers initContainers to fetch latest binaries from MinIO
|
||||
- |
|
||||
for svc in trading-service api-gateway broker-gateway \
|
||||
ml-training-service backtesting-service \
|
||||
trading-agent-service data-acquisition-service web-gateway; do
|
||||
kubectl -n foxhunt rollout restart deployment/$svc
|
||||
echo "Restarted $svc"
|
||||
done
|
||||
- |
|
||||
for svc in trading-service api-gateway broker-gateway \
|
||||
ml-training-service backtesting-service \
|
||||
trading-agent-service data-acquisition-service web-gateway; do
|
||||
kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
|
||||
done
|
||||
- echo "All services deployed with MinIO binaries"
|
||||
```
|
||||
|
||||
With:
|
||||
|
||||
```yaml
|
||||
# Apply shared binaries PVC (idempotent)
|
||||
- kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml
|
||||
# Apply service deployments (picks up any YAML changes)
|
||||
# Skip training/job-template.yaml (placeholder names, not directly applicable)
|
||||
# Skip storage/training-data-pv.yaml (PVC spec immutable after creation)
|
||||
- kubectl apply -f infra/k8s/services/
|
||||
# Copy new binaries to shared PVC (if compile ran)
|
||||
- |
|
||||
if [ -d "build-out/services" ]; then
|
||||
echo "Deploying new binaries via kubectl cp..."
|
||||
# Create temporary writer pod on foxhunt node
|
||||
kubectl apply -f - <<'WRITER_EOF'
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: binary-writer
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-writer
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: foxhunt
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
containers:
|
||||
- name: writer
|
||||
image: busybox:1.36
|
||||
command: ["sleep", "600"]
|
||||
volumeMounts:
|
||||
- name: binaries
|
||||
mountPath: /binaries
|
||||
volumes:
|
||||
- name: binaries
|
||||
persistentVolumeClaim:
|
||||
claimName: foxhunt-binaries
|
||||
restartPolicy: Never
|
||||
WRITER_EOF
|
||||
kubectl wait --for=condition=Ready pod/binary-writer -n foxhunt --timeout=120s
|
||||
# Copy service binaries
|
||||
for bin in trading_service api_gateway broker_gateway_service \
|
||||
ml_training_service backtesting_service \
|
||||
trading_agent_service data_acquisition_service web-gateway; do
|
||||
kubectl cp build-out/services/$bin foxhunt/binary-writer:/binaries/$bin
|
||||
echo " Copied $bin ($(stat -c%s build-out/services/$bin) bytes)"
|
||||
done
|
||||
# Copy web dashboard static assets (if build-web-dashboard ran)
|
||||
if [ -d "web-dashboard/dist" ]; then
|
||||
kubectl exec -n foxhunt binary-writer -- mkdir -p /binaries/static
|
||||
tar -C web-dashboard/dist -cf - . | kubectl exec -i -n foxhunt binary-writer -- tar -C /binaries/static -xf -
|
||||
echo " Copied web-dashboard static assets"
|
||||
fi
|
||||
# Make binaries executable
|
||||
for bin in trading_service api_gateway broker_gateway_service \
|
||||
ml_training_service backtesting_service \
|
||||
trading_agent_service data_acquisition_service web-gateway; do
|
||||
kubectl exec -n foxhunt binary-writer -- chmod +x /binaries/$bin
|
||||
done
|
||||
# Cleanup writer pod
|
||||
kubectl delete pod binary-writer -n foxhunt --wait=false
|
||||
echo "Binaries deployed to shared PVC"
|
||||
else
|
||||
echo "No new binaries — compile-services did not run, PVC has existing binaries"
|
||||
fi
|
||||
# Rolling restart all services (they read from shared PVC, no initContainer needed)
|
||||
- |
|
||||
for svc in trading-service api-gateway broker-gateway \
|
||||
ml-training-service backtesting-service \
|
||||
trading-agent-service data-acquisition-service web-gateway; do
|
||||
kubectl -n foxhunt rollout restart deployment/$svc
|
||||
echo "Restarted $svc"
|
||||
done
|
||||
- |
|
||||
for svc in trading-service api-gateway broker-gateway \
|
||||
ml-training-service backtesting-service \
|
||||
trading-agent-service data-acquisition-service web-gateway; do
|
||||
kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
|
||||
done
|
||||
- echo "All services deployed"
|
||||
```
|
||||
|
||||
Also update the comment at line 1507:
|
||||
```yaml
|
||||
# S3 binary share: services use generic runtime image + initContainer to fetch binaries from S3.
|
||||
# Deploy = apply K8s manifests (picks up YAML changes) + rollout restart (fetch new binaries).
|
||||
```
|
||||
Change to:
|
||||
```yaml
|
||||
# Shared PVC binary deploy: CI copies binaries to foxhunt-binaries PVC via writer pod.
|
||||
# Services mount the PVC read-only — no initContainer, no S3.
|
||||
```
|
||||
|
||||
**Step 3: Verify CI syntax**
|
||||
|
||||
Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" && echo "Valid YAML"`
|
||||
Expected: `Valid YAML`
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add .gitlab-ci.yml
|
||||
git commit -m "feat(ci): deploy binaries via kubectl cp to shared PVC instead of S3"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Modify all 8 service YAMLs
|
||||
|
||||
**Files:**
|
||||
- Modify: `infra/k8s/services/api-gateway.yaml`
|
||||
- Modify: `infra/k8s/services/trading-service.yaml`
|
||||
- Modify: `infra/k8s/services/broker-gateway.yaml`
|
||||
- Modify: `infra/k8s/services/ml-training-service.yaml`
|
||||
- Modify: `infra/k8s/services/backtesting-service.yaml`
|
||||
- Modify: `infra/k8s/services/trading-agent-service.yaml`
|
||||
- Modify: `infra/k8s/services/data-acquisition-service.yaml`
|
||||
- Modify: `infra/k8s/services/web-gateway.yaml`
|
||||
|
||||
**Step 1: For each of the 8 services, apply these changes:**
|
||||
|
||||
**A) Remove the entire `initContainers` section** (the `fetch-binary` initContainer with rclone S3 download, ~57 lines per service)
|
||||
|
||||
**B) Replace the `volumes` section.** Remove `binary-cache` (per-service PVC) and `minio-ca` (ConfigMap). Change `binaries` from `emptyDir` to shared PVC.
|
||||
|
||||
Before (standard service — api-gateway, trading-service, broker-gateway, backtesting-service, trading-agent-service, data-acquisition-service):
|
||||
```yaml
|
||||
volumes:
|
||||
- name: binaries
|
||||
emptyDir:
|
||||
sizeLimit: 200Mi
|
||||
- name: binary-cache
|
||||
persistentVolumeClaim:
|
||||
claimName: binary-cache-<service-name>
|
||||
- name: minio-ca
|
||||
configMap:
|
||||
name: minio-ca-cert
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
sizeLimit: 50Mi
|
||||
```
|
||||
|
||||
After:
|
||||
```yaml
|
||||
volumes:
|
||||
- name: binaries
|
||||
persistentVolumeClaim:
|
||||
claimName: foxhunt-binaries
|
||||
readOnly: true
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
sizeLimit: 50Mi
|
||||
```
|
||||
|
||||
**Special case — ml-training-service:** Keep `minio-ca` (used by container for S3 model storage) and keep `tls-certs`:
|
||||
```yaml
|
||||
volumes:
|
||||
- name: binaries
|
||||
persistentVolumeClaim:
|
||||
claimName: foxhunt-binaries
|
||||
readOnly: true
|
||||
- name: tls-certs
|
||||
secret:
|
||||
secretName: ml-training-tls
|
||||
- name: minio-ca
|
||||
configMap:
|
||||
name: minio-ca-cert
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
sizeLimit: 50Mi
|
||||
```
|
||||
|
||||
**Special case — web-gateway:** Same as standard (minio-ca removed — static assets are on the PVC now).
|
||||
|
||||
**C) No changes to container section** — containers already mount `/binaries` readOnly and `/tmp`. The `command` field still points to `/binaries/<binary_name>`.
|
||||
|
||||
**Step 2: Verify all YAMLs**
|
||||
|
||||
Run: `for f in infra/k8s/services/*.yaml; do kubectl apply --dry-run=client -f "$f" 2>&1 | head -5; done`
|
||||
Expected: Each file shows `deployment.apps/<name> created (dry run)` and `service/<name> created (dry run)`
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add infra/k8s/services/api-gateway.yaml \
|
||||
infra/k8s/services/trading-service.yaml \
|
||||
infra/k8s/services/broker-gateway.yaml \
|
||||
infra/k8s/services/ml-training-service.yaml \
|
||||
infra/k8s/services/backtesting-service.yaml \
|
||||
infra/k8s/services/trading-agent-service.yaml \
|
||||
infra/k8s/services/data-acquisition-service.yaml \
|
||||
infra/k8s/services/web-gateway.yaml
|
||||
git commit -m "feat(infra): remove initContainers, mount shared foxhunt-binaries PVC"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Apply shared PVC and seed initial binaries
|
||||
|
||||
**Prerequisite:** Tasks 1-6 completed (code changes done locally). This task operates on the live cluster.
|
||||
|
||||
**Step 1: Create the shared PVC**
|
||||
|
||||
Run: `kubectl apply -f infra/k8s/storage/foxhunt-binaries-pvc.yaml`
|
||||
Expected: `persistentvolumeclaim/foxhunt-binaries created`
|
||||
|
||||
Verify: `kubectl get pvc foxhunt-binaries -n foxhunt`
|
||||
Expected: STATUS=Bound (or Pending briefly, then Bound)
|
||||
|
||||
**Step 2: Seed binaries from current running pods**
|
||||
|
||||
The current pods have working binaries in their `/binaries/` emptyDir (fetched from S3). Copy them to the new shared PVC via a writer pod.
|
||||
|
||||
```bash
|
||||
# Create writer pod
|
||||
kubectl apply -f - <<'EOF'
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: binary-writer
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-writer
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: foxhunt
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
containers:
|
||||
- name: writer
|
||||
image: busybox:1.36
|
||||
command: ["sleep", "600"]
|
||||
volumeMounts:
|
||||
- name: binaries
|
||||
mountPath: /binaries
|
||||
volumes:
|
||||
- name: binaries
|
||||
persistentVolumeClaim:
|
||||
claimName: foxhunt-binaries
|
||||
restartPolicy: Never
|
||||
EOF
|
||||
|
||||
# Wait for writer pod
|
||||
kubectl wait --for=condition=Ready pod/binary-writer -n foxhunt --timeout=120s
|
||||
```
|
||||
|
||||
**Step 3: Copy binaries from running service pods to writer pod**
|
||||
|
||||
```bash
|
||||
# For each service, copy the binary from the running pod to the writer pod
|
||||
for svc_bin in "api-gateway:api_gateway" \
|
||||
"trading-service:trading_service" \
|
||||
"broker-gateway:broker_gateway_service" \
|
||||
"ml-training-service:ml_training_service" \
|
||||
"backtesting-service:backtesting_service" \
|
||||
"trading-agent-service:trading_agent_service" \
|
||||
"data-acquisition-service:data_acquisition_service" \
|
||||
"web-gateway:web-gateway"; do
|
||||
svc="${svc_bin%%:*}"
|
||||
bin="${svc_bin##*:}"
|
||||
POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=$svc -o jsonpath='{.items[0].metadata.name}')
|
||||
if [ -n "$POD" ]; then
|
||||
kubectl cp foxhunt/$POD:/binaries/$bin /tmp/seed-$bin
|
||||
kubectl cp /tmp/seed-$bin foxhunt/binary-writer:/binaries/$bin
|
||||
rm /tmp/seed-$bin
|
||||
echo "Seeded $bin from $POD"
|
||||
else
|
||||
echo "WARNING: no pod for $svc"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
**Step 4: Copy web-gateway static assets**
|
||||
|
||||
```bash
|
||||
WG_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=web-gateway -o jsonpath='{.items[0].metadata.name}')
|
||||
kubectl exec -n foxhunt binary-writer -- mkdir -p /binaries/static
|
||||
kubectl exec -n foxhunt $WG_POD -- tar -C /binaries/static -cf - . 2>/dev/null | kubectl exec -i -n foxhunt binary-writer -- tar -C /binaries/static -xf -
|
||||
echo "Seeded web-gateway static assets"
|
||||
```
|
||||
|
||||
**Step 5: Make binaries executable and verify**
|
||||
|
||||
```bash
|
||||
for bin in trading_service api_gateway broker_gateway_service \
|
||||
ml_training_service backtesting_service \
|
||||
trading_agent_service data_acquisition_service web-gateway; do
|
||||
kubectl exec -n foxhunt binary-writer -- chmod +x /binaries/$bin
|
||||
done
|
||||
|
||||
# Verify all binaries present
|
||||
kubectl exec -n foxhunt binary-writer -- ls -lh /binaries/
|
||||
kubectl exec -n foxhunt binary-writer -- ls /binaries/static/ | head -5
|
||||
```
|
||||
|
||||
Expected: 8 binaries (~150-250MB each) + `static/` directory
|
||||
|
||||
**Step 6: Delete writer pod**
|
||||
|
||||
Run: `kubectl delete pod binary-writer -n foxhunt`
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Apply modified service YAMLs and verify
|
||||
|
||||
**Step 1: Apply all service YAMLs**
|
||||
|
||||
Run: `kubectl apply -f infra/k8s/services/`
|
||||
Expected: 8 deployments configured, services unchanged
|
||||
|
||||
**Step 2: Rolling restart all services**
|
||||
|
||||
```bash
|
||||
for svc in trading-service api-gateway broker-gateway \
|
||||
ml-training-service backtesting-service \
|
||||
trading-agent-service data-acquisition-service web-gateway; do
|
||||
kubectl -n foxhunt rollout restart deployment/$svc
|
||||
done
|
||||
```
|
||||
|
||||
**Step 3: Wait for rollout**
|
||||
|
||||
```bash
|
||||
for svc in trading-service api-gateway broker-gateway \
|
||||
ml-training-service backtesting-service \
|
||||
trading-agent-service data-acquisition-service web-gateway; do
|
||||
kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
|
||||
done
|
||||
```
|
||||
|
||||
Expected: All 8 deployments successfully rolled out
|
||||
|
||||
**Step 4: Verify pods have no initContainers**
|
||||
|
||||
```bash
|
||||
kubectl get pods -n foxhunt -l app.kubernetes.io/part-of=foxhunt -o custom-columns='NAME:.metadata.name,INIT:.spec.initContainers[*].name,STATUS:.status.phase' | grep -v ib-gateway
|
||||
```
|
||||
|
||||
Expected: INIT column is empty for all service pods (no initContainers)
|
||||
|
||||
**Step 5: Verify binaries are accessible**
|
||||
|
||||
```bash
|
||||
# Spot-check: api-gateway can read its binary
|
||||
API_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=api-gateway -o jsonpath='{.items[0].metadata.name}')
|
||||
kubectl exec -n foxhunt $API_POD -- ls -l /binaries/api_gateway
|
||||
```
|
||||
|
||||
Expected: Binary exists and is executable
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Delete old PVCs and cleanup files
|
||||
|
||||
**Step 1: Delete old per-service PVCs from cluster**
|
||||
|
||||
```bash
|
||||
for pvc in binary-cache-trading-service binary-cache-api-gateway \
|
||||
binary-cache-broker-gateway binary-cache-ml-training-service \
|
||||
binary-cache-backtesting-service binary-cache-trading-agent-service \
|
||||
binary-cache-data-acquisition-service binary-cache-web-gateway; do
|
||||
kubectl delete pvc $pvc -n foxhunt --ignore-not-found
|
||||
done
|
||||
```
|
||||
|
||||
Expected: 8 PVCs deleted
|
||||
|
||||
**Step 2: Verify only foxhunt-binaries PVC remains for services**
|
||||
|
||||
Run: `kubectl get pvc -n foxhunt -l app.kubernetes.io/name=binary-cache`
|
||||
Expected: No resources found
|
||||
|
||||
Run: `kubectl get pvc foxhunt-binaries -n foxhunt`
|
||||
Expected: STATUS=Bound
|
||||
|
||||
**Step 3: Delete old PVC manifest file**
|
||||
|
||||
Run: `rm infra/k8s/storage/binary-cache-pvcs.yaml`
|
||||
|
||||
**Step 4: Commit cleanup**
|
||||
|
||||
```bash
|
||||
git add -A infra/k8s/storage/binary-cache-pvcs.yaml
|
||||
git commit -m "chore(infra): delete 8 per-service binary-cache PVCs (replaced by foxhunt-binaries)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 10: End-to-end verification
|
||||
|
||||
**Step 1: Verify all pods running**
|
||||
|
||||
Run: `kubectl get pods -n foxhunt -l app.kubernetes.io/part-of=foxhunt -o wide`
|
||||
Expected: All pods Running, no restarts, no initContainers
|
||||
|
||||
**Step 2: Verify gRPC health for key services**
|
||||
|
||||
```bash
|
||||
API_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=api-gateway -o jsonpath='{.items[0].metadata.name}')
|
||||
kubectl exec -n foxhunt $API_POD -- grpc_health_probe -addr=localhost:50051
|
||||
```
|
||||
|
||||
Expected: `status: SERVING`
|
||||
|
||||
**Step 3: Verify web-gateway serves static assets**
|
||||
|
||||
```bash
|
||||
WG_POD=$(kubectl get pod -n foxhunt -l app.kubernetes.io/name=web-gateway -o jsonpath='{.items[0].metadata.name}')
|
||||
kubectl exec -n foxhunt $WG_POD -- ls /binaries/static/index.html
|
||||
```
|
||||
|
||||
Expected: File exists
|
||||
|
||||
**Step 4: Verify PVC usage**
|
||||
|
||||
Run: `kubectl get pvc -n foxhunt | grep -E 'foxhunt-binaries|binary-cache'`
|
||||
Expected: Only `foxhunt-binaries` (Bound, 2Gi)
|
||||
|
||||
**Step 5: Final commit with all changes**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "feat(infra): consolidate 8 binary-cache PVCs into single shared foxhunt-binaries PVC
|
||||
|
||||
Eliminates S3 round-trip for service binaries. CI deploys via kubectl cp
|
||||
to a temporary writer pod. Services mount shared PVC read-only.
|
||||
Training pipeline (GPU nodes) unchanged — still uses S3."
|
||||
```
|
||||
|
||||
**Step 6: Push to GitLab**
|
||||
|
||||
Run: `git push origin main`
|
||||
|
||||
---
|
||||
|
||||
## Summary of Changes
|
||||
|
||||
| What | Before | After |
|
||||
|---|---|---|
|
||||
| Binary source | S3 (MinIO) via initContainer | Shared PVC via kubectl cp |
|
||||
| PVCs | 8 × 1Gi per-service | 1 × 2Gi shared |
|
||||
| initContainers | 8 (fetch-binary) | 0 |
|
||||
| S3 uploads (services) | compile-services + build-web-dashboard | None |
|
||||
| CI artifact transfer | None | compile-services + build-web-dashboard |
|
||||
| Deploy mechanism | rollout restart → initContainer fetches from S3 | kubectl cp to PVC → rollout restart |
|
||||
| Training pipeline | S3 (unchanged) | S3 (unchanged) |
|
||||
| GPU overlays | S3 (unchanged) | S3 (unchanged) |
|
||||
| Cost | 8Gi PVCs (~€0.16/mo) | 2Gi PVC (~€0.04/mo) |
|
||||
@@ -1,124 +0,0 @@
|
||||
# Per-service binary cache PVCs — fallback when S3 is unavailable
|
||||
# Each service gets its own 1Gi PVC (ReadWriteOnce, scw-bssd)
|
||||
# Scaleway minimum block volume is 1Gi — cannot request less
|
||||
# Total: 8 × 1Gi = 8Gi, ~€0.16/month
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: binary-cache-trading-service
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: binary-cache-api-gateway
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: binary-cache-broker-gateway
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: binary-cache-ml-training-service
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: binary-cache-backtesting-service
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: binary-cache-trading-agent-service
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: binary-cache-data-acquisition-service
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: binary-cache-web-gateway
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: binary-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
Reference in New Issue
Block a user