From df20ed74d4ded81b1de6fbfb6b12aa8c4f7be4cd Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 28 Feb 2026 17:46:13 +0100 Subject: [PATCH] =?UTF-8?q?docs:=20add=20S3=20binary=20share=20design=20?= =?UTF-8?q?=E2=80=94=20eliminate=20Docker=20builds,=20faster=20pod=20start?= =?UTF-8?q?up?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace per-service Docker images with generic base images + S3-based binary distribution. CI compiles and uploads stripped binaries to S3. Pods fetch binaries via initContainer with PVC cache fallback for trading resilience. Co-Authored-By: Claude Opus 4.6 --- .../2026-02-28-s3-binary-share-design.md | 454 ++++++++++++++++++ 1 file changed, 454 insertions(+) create mode 100644 docs/plans/2026-02-28-s3-binary-share-design.md diff --git a/docs/plans/2026-02-28-s3-binary-share-design.md b/docs/plans/2026-02-28-s3-binary-share-design.md new file mode 100644 index 000000000..91daf8566 --- /dev/null +++ b/docs/plans/2026-02-28-s3-binary-share-design.md @@ -0,0 +1,454 @@ +# S3 Binary Share — Eliminate Docker Builds, Faster Pod Startup + +**Date**: 2026-02-28 +**Status**: Approved + +## Problem + +Every code change triggers a long pipeline: compile → Kaniko packages 9 images → push to registry → pods pull images. The training image alone is ~3GB (CUDA runtime). Even with an image-prepuller DaemonSet, rebuilding and pushing images adds 3-7 minutes per pipeline. During active trading, a pod crash + image pull could cause unacceptable downtime. + +## Solution + +Replace per-service Docker images with generic base images + S3-based binary distribution. CI compiles and uploads stripped binaries to S3. Pods run generic base images (pre-pulled, rarely change) and fetch binaries from S3 via initContainer on startup. A PVC cache per service provides fallback if S3 is unavailable during trading. + +## Architecture + +### S3 Bucket Structure + +``` +s3://foxhunt-binaries/ +├── latest/ +│ ├── services/ +│ │ ├── trading_service +│ │ ├── api_gateway +│ │ ├── broker_gateway_service +│ │ ├── ml_training_service +│ │ ├── backtesting_service +│ │ ├── trading_agent_service +│ │ ├── data_acquisition_service +│ │ └── web-gateway +│ ├── training/ +│ │ ├── train_baseline_rl +│ │ ├── train_baseline_supervised +│ │ ├── evaluate_baseline +│ │ ├── evaluate_supervised +│ │ ├── hyperopt_baseline_rl +│ │ ├── hyperopt_baseline_supervised +│ │ └── training_uploader +│ ├── web-dashboard/ +│ │ └── dist/ # Vite build output +│ └── MANIFEST.json # {sha, timestamp} +└── archive// # optional audit trail +``` + +- Compile always overwrites `latest/` +- Optional: copy to `archive//` for debugging +- Pods always fetch from `latest/` +- Lifecycle rule: delete archive entries older than 30 days +- Bucket region: fr-par (same as cluster) + +### Generic Base Images (3 total) + +All pushed to `rg.fr-par.scw.cloud/foxhunt-ci/` (Scaleway Container Registry). + +#### 1. foxhunt-runtime (services) + +```dockerfile +FROM debian:bookworm-slim + +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates libssl3 curl unzip \ + && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \ + && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \ + && rm /tmp/rclone.zip \ + && apt-get purge -y unzip && rm -rf /var/lib/apt/lists/* + +RUN curl -fsSL https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.25/grpc_health_probe-linux-amd64 \ + -o /usr/local/bin/grpc_health_probe && chmod +x /usr/local/bin/grpc_health_probe + +RUN groupadd -g 1000 foxhunt && useradd -u 1000 -g foxhunt -m -s /bin/false foxhunt +USER foxhunt +``` + +Size: ~100MB. Rebuild: only when libssl/rclone/grpc_health_probe updates. + +#### 2. foxhunt-training-runtime (GPU training) + +```dockerfile +FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 + +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates libssl3 curl unzip cuda-nvrtc-12-4 \ + && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \ + && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \ + && rm /tmp/rclone.zip \ + && apt-get purge -y unzip && rm -rf /var/lib/apt/lists/* + +RUN groupadd -g 1000 foxhunt && useradd -u 1000 -g foxhunt -m -s /bin/false foxhunt + +ENV NVIDIA_VISIBLE_DEVICES=all +ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility +USER foxhunt +``` + +Size: ~3GB (CUDA dominates). Rebuild: only on CUDA version bump. +Image-prepuller DaemonSet still caches this on GPU nodes. + +#### 3. foxhunt-node-builder (Vite build CI job) + +```dockerfile +FROM node:22-slim + +RUN apt-get update && apt-get install -y --no-install-recommends \ + curl unzip \ + && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \ + && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \ + && rm /tmp/rclone.zip \ + && apt-get purge -y unzip && rm -rf /var/lib/apt/lists/* +``` + +Size: ~200MB. Rebuild: only on Node major version bump. + +### CI Pipeline Changes + +#### New pipeline flow + +``` +prepare → test → compile → deploy +``` + +Build stage is entirely eliminated. + +#### compile-services (modified) + +Same cargo build, but uploads to S3 instead of creating GitLab artifacts: + +```yaml +compile-services: + extends: .rust-base-cpu + stage: compile + needs: [] + script: + - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true + - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release} + - TARGET_DIR="target/$PROFILE" + - cargo build --profile $PROFILE + -p trading_service -p api_gateway -p broker_gateway_service + -p ml_training_service -p backtesting_service + -p trading_agent_service -p data_acquisition_service -p web-gateway + - sccache --show-stats || true + - mkdir -p build-out/services + - | + for bin in trading_service api_gateway broker_gateway_service \ + ml_training_service backtesting_service \ + trading_agent_service data_acquisition_service web-gateway; do + cp $TARGET_DIR/$bin build-out/services/ + strip build-out/services/$bin + done + - ls -lh build-out/services/ + # Upload to S3 + - | + for bin in trading_service api_gateway broker_gateway_service \ + ml_training_service backtesting_service \ + trading_agent_service data_acquisition_service web-gateway; do + rclone copyto build-out/services/$bin s3:foxhunt-binaries/latest/services/$bin + done + # Archive by SHA + - rclone sync s3:foxhunt-binaries/latest/services/ s3:foxhunt-binaries/archive/${CI_COMMIT_SHA}/services/ +``` + +#### compile-training (modified) + +Same pattern — uploads to `s3:foxhunt-binaries/latest/training/`. + +#### build-web-dashboard (new) + +```yaml +build-web-dashboard: + stage: compile + image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-node-builder:latest + needs: [] + tags: [kapsule, docker] + variables: + KUBERNETES_CPU_REQUEST: "2000m" + KUBERNETES_MEMORY_REQUEST: "2Gi" + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: [web-dashboard/**] + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + script: + - cd web-dashboard + - npm ci + - npm run build + - rclone sync dist/ s3:foxhunt-binaries/latest/web-dashboard/dist/ +``` + +#### write-manifest (new, runs after all compiles) + +```yaml +write-manifest: + stage: compile + image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest + needs: [compile-services, compile-training, build-web-dashboard] + tags: [kapsule, docker] + script: + - echo '{"sha":"'${CI_COMMIT_SHA}'","ts":"'$(date -Iseconds)'"}' \ + | rclone rcat s3:foxhunt-binaries/latest/MANIFEST.json +``` + +#### deploy (new) + +```yaml +deploy: + stage: deploy + image: bitnami/kubectl:latest + needs: [write-manifest] + tags: [kapsule, docker] + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + script: + - rclone cat s3:foxhunt-binaries/latest/MANIFEST.json + - | + for svc in trading-service api-gateway broker-gateway \ + ml-training-service backtesting-service \ + trading-agent-service data-acquisition-service web-gateway; do + kubectl -n foxhunt rollout restart deployment/$svc + done + - | + for svc in trading-service api-gateway broker-gateway \ + ml-training-service backtesting-service \ + trading-agent-service data-acquisition-service web-gateway; do + kubectl -n foxhunt rollout status deployment/$svc --timeout=120s + done +``` + +#### Deleted CI jobs + +- `.kaniko-service-base` template +- `.kaniko-training-base` template +- `build-trading-service`, `build-api-gateway`, `build-broker-gateway` +- `build-ml-training`, `build-backtesting`, `build-trading-agent` +- `build-data-acquisition`, `build-web-gateway`, `build-training` +- `artifacts:` sections in compile jobs + +### Deployment Pattern + +Every service deployment uses this initContainer pattern: + +```yaml +initContainers: + - name: fetch-binary + image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest + command: ["/bin/sh", "-c"] + args: + - | + set -e + SERVICE=trading_service + if rclone copyto s3:foxhunt-binaries/latest/services/$SERVICE /binaries/$SERVICE 2>/dev/null; then + chmod +x /binaries/$SERVICE + cp /binaries/$SERVICE /cache/$SERVICE + echo "Fetched from S3 ($(stat -c%s /binaries/$SERVICE) bytes)" + elif [ -f /cache/$SERVICE ]; then + cp /cache/$SERVICE /binaries/$SERVICE + chmod +x /binaries/$SERVICE + echo "WARNING: S3 unavailable, using cached binary" + else + echo "FATAL: No binary available (S3 down + empty cache)" + exit 1 + fi + env: + - name: RCLONE_S3_PROVIDER + value: Scaleway + - name: RCLONE_S3_ENDPOINT + value: s3.fr-par.scw.cloud + - name: RCLONE_S3_REGION + value: fr-par + - name: RCLONE_S3_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: s3-credentials + key: access-key + - name: RCLONE_S3_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: s3-credentials + key: secret-key + volumeMounts: + - name: binaries + mountPath: /binaries + - name: binary-cache + mountPath: /cache + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 500m + memory: 128Mi +containers: + - name: trading-service + image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest + command: ["/binaries/trading_service"] + volumeMounts: + - name: binaries + mountPath: /binaries + readOnly: true +volumes: + - name: binaries + emptyDir: + sizeLimit: 200Mi + - name: binary-cache + persistentVolumeClaim: + claimName: binary-cache-trading-service +``` + +#### web-gateway special case + +initContainer fetches both binary and dashboard assets: + +```yaml +args: + - | + set -e + SERVICE=web-gateway + if rclone copyto s3:foxhunt-binaries/latest/services/$SERVICE /binaries/$SERVICE 2>/dev/null; then + chmod +x /binaries/$SERVICE + cp /binaries/$SERVICE /cache/$SERVICE + rclone sync s3:foxhunt-binaries/latest/web-dashboard/dist/ /binaries/static/ + cp -r /binaries/static /cache/static + echo "Fetched from S3" + elif [ -f /cache/$SERVICE ]; then + cp /cache/$SERVICE /binaries/$SERVICE + chmod +x /binaries/$SERVICE + cp -r /cache/static /binaries/static 2>/dev/null || true + echo "WARNING: S3 unavailable, using cached binary" + else + echo "FATAL: No binary available" + exit 1 + fi +``` + +#### Training job template + +Uses `foxhunt-training-runtime`, no cache PVC (batch jobs can retry): + +```yaml +initContainers: + - name: fetch-binaries + image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest + command: ["/bin/sh", "-c"] + args: + - | + set -e + rclone sync s3:foxhunt-binaries/latest/training/ /binaries/ + chmod +x /binaries/* + echo "Fetched training binaries" + volumeMounts: + - name: binaries + mountPath: /binaries +containers: + - name: training + image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest + command: ["/binaries/$(TRAINING_BINARY)"] + volumeMounts: + - name: binaries + mountPath: /binaries + readOnly: true +``` + +### Cache PVCs + +8 PVCs, one per service deployment (500Mi each, scw-bssd): + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: binary-cache-trading-service + namespace: foxhunt +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 500Mi + storageClassName: scw-bssd +``` + +Total: 8 × 500Mi = 4Gi. Cost: ~€0.08/month. + +### Resilience During Trading + +1. **Running pods unaffected**: Binaries live in emptyDir for the pod's lifetime. S3 outage has zero impact on running services. +2. **Pod crash + S3 down**: initContainer falls back to PVC cache (last known good binary). +3. **Pod crash + S3 down + empty cache**: Only on first-ever deploy. Pod stays in Init:Error; deployment's `maxUnavailable: 0` prevents the old pod from terminating. +4. **Deployment strategy**: `maxSurge: 1, maxUnavailable: 0` ensures old pod keeps running until new pod is ready. + +### What Gets Deleted + +| Item | Notes | +|------|-------| +| `Dockerfile.runtime` | Replaced by `foxhunt-runtime` | +| `Dockerfile.training` | Replaced by `foxhunt-training-runtime` | +| `Dockerfile.web-gateway-runtime` | Replaced by `foxhunt-runtime` + initContainer | +| `Dockerfile.web-gateway` | Full multi-stage builder, no longer needed | +| `Dockerfile.service` | Full multi-stage builder, no longer needed | +| 9 Kaniko CI jobs | All `build-*` jobs in `.gitlab-ci.yml` | +| 10 per-service Docker images | In Scaleway CR and GitLab registry | + +### What's Kept + +| Item | Notes | +|------|-------| +| `Dockerfile.ci-builder` | Compile infrastructure, unchanged | +| `Dockerfile.ci-builder-cpu` | Compile infrastructure, unchanged | +| `Dockerfile.infra-runner` | IaC pipeline, unchanged | +| Image-prepuller DaemonSet | Updated to pull `foxhunt-training-runtime` | + +### Timing Comparison + +| Step | Before | After | +|------|--------|-------| +| Compile | 8-22 min | 8-22 min (unchanged) | +| Image build | 2-5 min (9 Kaniko) | 0 (eliminated) | +| Registry push | 1-2 min | ~10s (S3 upload) | +| Pod startup | 30s-3min (image pull) | 2-5s (initContainer S3 fetch) | +| Total pipeline | 15-30 min | 8-23 min | +| Pod restart | 30s-3min | 2-5s | + +### New Infrastructure + +| Resource | Spec | Cost | +|----------|------|------| +| S3 bucket `foxhunt-binaries` | fr-par, 30-day lifecycle | ~€0.01/month | +| 8 cache PVCs (500Mi each) | scw-bssd | ~€0.08/month | +| 3 base image Dockerfiles | In `infra/docker/` | 0 | + +### rclone Configuration + +rclone in CI and pods is configured via environment variables (no config file needed): + +``` +RCLONE_S3_PROVIDER=Scaleway +RCLONE_S3_ENDPOINT=s3.fr-par.scw.cloud +RCLONE_S3_REGION=fr-par +RCLONE_S3_ACCESS_KEY_ID= +RCLONE_S3_SECRET_ACCESS_KEY= +``` + +The `s3-credentials` K8s secret already exists in the foxhunt namespace (used by training_uploader). CI uses `SCW_ACCESS_KEY`/`SCW_SECRET_KEY` variables mapped to `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`. + +For rclone in CI, the S3 remote is referenced as `s3:foxhunt-binaries/...` using `RCLONE_S3_*` env vars. + +### rclone in ci-builder images + +Add rclone to both `Dockerfile.ci-builder` and `Dockerfile.ci-builder-cpu` for the upload step: + +```dockerfile +# Add to both ci-builder Dockerfiles +RUN curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \ + && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \ + && rm /tmp/rclone.zip +```