foxhunt/docs/plans/2026-02-28-s3-binary-share-design.md

# S3 Binary Share — Eliminate Docker Builds, Faster Pod Startup

**Date**: 2026-02-28
**Status**: Approved

## Problem

Every code change triggers a long pipeline: compile → Kaniko packages 9 images → push to registry → pods pull images. The training image alone is ~3GB (CUDA runtime). Even with an image-prepuller DaemonSet, rebuilding and pushing images adds 3-7 minutes per pipeline. During active trading, a pod crash + image pull could cause unacceptable downtime.

## Solution

Replace per-service Docker images with generic base images + S3-based binary distribution. CI compiles and uploads stripped binaries to S3. Pods run generic base images (pre-pulled, rarely change) and fetch binaries from S3 via initContainer on startup. A PVC cache per service provides fallback if S3 is unavailable during trading.

## Architecture

### S3 Bucket Structure

```
s3://foxhunt-binaries/
├── latest/
│   ├── services/
│   │   ├── trading_service
│   │   ├── api_gateway
│   │   ├── broker_gateway_service
│   │   ├── ml_training_service
│   │   ├── backtesting_service
│   │   ├── trading_agent_service
│   │   ├── data_acquisition_service
│   │   └── web-gateway
│   ├── training/
│   │   ├── train_baseline_rl
│   │   ├── train_baseline_supervised
│   │   ├── evaluate_baseline
│   │   ├── evaluate_supervised
│   │   ├── hyperopt_baseline_rl
│   │   ├── hyperopt_baseline_supervised
│   │   └── training_uploader
│   ├── web-dashboard/
│   │   └── dist/                    # Vite build output
│   └── MANIFEST.json                # {sha, timestamp}
└── archive/<commit-sha>/            # optional audit trail
```

- Compile always overwrites `latest/`
- Optional: copy to `archive/<sha>/` for debugging
- Pods always fetch from `latest/`
- Lifecycle rule: delete archive entries older than 30 days
- Bucket region: fr-par (same as cluster)

### Generic Base Images (3 total)

All pushed to `rg.fr-par.scw.cloud/foxhunt-ci/` (Scaleway Container Registry).

#### 1. foxhunt-runtime (services)

```dockerfile
FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates libssl3 curl unzip \
    && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
    && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
    && rm /tmp/rclone.zip \
    && apt-get purge -y unzip && rm -rf /var/lib/apt/lists/*

RUN curl -fsSL https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.25/grpc_health_probe-linux-amd64 \
    -o /usr/local/bin/grpc_health_probe && chmod +x /usr/local/bin/grpc_health_probe

RUN groupadd -g 1000 foxhunt && useradd -u 1000 -g foxhunt -m -s /bin/false foxhunt
USER foxhunt
```

Size: ~100MB. Rebuild: only when libssl/rclone/grpc_health_probe updates.

#### 2. foxhunt-training-runtime (GPU training)

```dockerfile
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates libssl3 curl unzip cuda-nvrtc-12-4 \
    && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
    && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
    && rm /tmp/rclone.zip \
    && apt-get purge -y unzip && rm -rf /var/lib/apt/lists/*

RUN groupadd -g 1000 foxhunt && useradd -u 1000 -g foxhunt -m -s /bin/false foxhunt

ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
USER foxhunt
```

Size: ~3GB (CUDA dominates). Rebuild: only on CUDA version bump.
Image-prepuller DaemonSet still caches this on GPU nodes.

#### 3. foxhunt-node-builder (Vite build CI job)

```dockerfile
FROM node:22-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl unzip \
    && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
    && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
    && rm /tmp/rclone.zip \
    && apt-get purge -y unzip && rm -rf /var/lib/apt/lists/*
```

Size: ~200MB. Rebuild: only on Node major version bump.

### CI Pipeline Changes

#### New pipeline flow

```
prepare → test → compile → deploy
```

Build stage is entirely eliminated.

#### compile-services (modified)

Same cargo build, but uploads to S3 instead of creating GitLab artifacts:

```yaml
compile-services:
  extends: .rust-base-cpu
  stage: compile
  needs: []
  script:
    - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true
    - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release}
    - TARGET_DIR="target/$PROFILE"
    - cargo build --profile $PROFILE
        -p trading_service -p api_gateway -p broker_gateway_service
        -p ml_training_service -p backtesting_service
        -p trading_agent_service -p data_acquisition_service -p web-gateway
    - sccache --show-stats || true
    - mkdir -p build-out/services
    - |
      for bin in trading_service api_gateway broker_gateway_service \
                 ml_training_service backtesting_service \
                 trading_agent_service data_acquisition_service web-gateway; do
        cp $TARGET_DIR/$bin build-out/services/
        strip build-out/services/$bin
      done
    - ls -lh build-out/services/
    # Upload to S3
    - |
      for bin in trading_service api_gateway broker_gateway_service \
                 ml_training_service backtesting_service \
                 trading_agent_service data_acquisition_service web-gateway; do
        rclone copyto build-out/services/$bin s3:foxhunt-binaries/latest/services/$bin
      done
    # Archive by SHA
    - rclone sync s3:foxhunt-binaries/latest/services/ s3:foxhunt-binaries/archive/${CI_COMMIT_SHA}/services/
```

#### compile-training (modified)

Same pattern — uploads to `s3:foxhunt-binaries/latest/training/`.

#### build-web-dashboard (new)

```yaml
build-web-dashboard:
  stage: compile
  image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-node-builder:latest
  needs: []
  tags: [kapsule, docker]
  variables:
    KUBERNETES_CPU_REQUEST: "2000m"
    KUBERNETES_MEMORY_REQUEST: "2Gi"
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes: [web-dashboard/**]
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
  script:
    - cd web-dashboard
    - npm ci
    - npm run build
    - rclone sync dist/ s3:foxhunt-binaries/latest/web-dashboard/dist/
```

#### write-manifest (new, runs after all compiles)

```yaml
write-manifest:
  stage: compile
  image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
  needs: [compile-services, compile-training, build-web-dashboard]
  tags: [kapsule, docker]
  script:
    - echo '{"sha":"'${CI_COMMIT_SHA}'","ts":"'$(date -Iseconds)'"}' \
      | rclone rcat s3:foxhunt-binaries/latest/MANIFEST.json
```

#### deploy (new)

```yaml
deploy:
  stage: deploy
  image: bitnami/kubectl:latest
  needs: [write-manifest]
  tags: [kapsule, docker]
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
  script:
    - rclone cat s3:foxhunt-binaries/latest/MANIFEST.json
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout restart deployment/$svc
      done
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout status deployment/$svc --timeout=120s
      done
```

#### Deleted CI jobs

- `.kaniko-service-base` template
- `.kaniko-training-base` template
- `build-trading-service`, `build-api-gateway`, `build-broker-gateway`
- `build-ml-training`, `build-backtesting`, `build-trading-agent`
- `build-data-acquisition`, `build-web-gateway`, `build-training`
- `artifacts:` sections in compile jobs

### Deployment Pattern

Every service deployment uses this initContainer pattern:

```yaml
initContainers:
  - name: fetch-binary
    image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
    command: ["/bin/sh", "-c"]
    args:
      - |
        set -e
        SERVICE=trading_service
        if rclone copyto s3:foxhunt-binaries/latest/services/$SERVICE /binaries/$SERVICE 2>/dev/null; then
          chmod +x /binaries/$SERVICE
          cp /binaries/$SERVICE /cache/$SERVICE
          echo "Fetched from S3 ($(stat -c%s /binaries/$SERVICE) bytes)"
        elif [ -f /cache/$SERVICE ]; then
          cp /cache/$SERVICE /binaries/$SERVICE
          chmod +x /binaries/$SERVICE
          echo "WARNING: S3 unavailable, using cached binary"
        else
          echo "FATAL: No binary available (S3 down + empty cache)"
          exit 1
        fi
    env:
      - name: RCLONE_S3_PROVIDER
        value: Scaleway
      - name: RCLONE_S3_ENDPOINT
        value: s3.fr-par.scw.cloud
      - name: RCLONE_S3_REGION
        value: fr-par
      - name: RCLONE_S3_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: s3-credentials
            key: access-key
      - name: RCLONE_S3_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: s3-credentials
            key: secret-key
    volumeMounts:
      - name: binaries
        mountPath: /binaries
      - name: binary-cache
        mountPath: /cache
    resources:
      requests:
        cpu: 100m
        memory: 64Mi
      limits:
        cpu: 500m
        memory: 128Mi
containers:
  - name: trading-service
    image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
    command: ["/binaries/trading_service"]
    volumeMounts:
      - name: binaries
        mountPath: /binaries
        readOnly: true
volumes:
  - name: binaries
    emptyDir:
      sizeLimit: 200Mi
  - name: binary-cache
    persistentVolumeClaim:
      claimName: binary-cache-trading-service
```

#### web-gateway special case

initContainer fetches both binary and dashboard assets:

```yaml
args:
  - |
    set -e
    SERVICE=web-gateway
    if rclone copyto s3:foxhunt-binaries/latest/services/$SERVICE /binaries/$SERVICE 2>/dev/null; then
      chmod +x /binaries/$SERVICE
      cp /binaries/$SERVICE /cache/$SERVICE
      rclone sync s3:foxhunt-binaries/latest/web-dashboard/dist/ /binaries/static/
      cp -r /binaries/static /cache/static
      echo "Fetched from S3"
    elif [ -f /cache/$SERVICE ]; then
      cp /cache/$SERVICE /binaries/$SERVICE
      chmod +x /binaries/$SERVICE
      cp -r /cache/static /binaries/static 2>/dev/null || true
      echo "WARNING: S3 unavailable, using cached binary"
    else
      echo "FATAL: No binary available"
      exit 1
    fi
```

#### Training job template

Uses `foxhunt-training-runtime`, no cache PVC (batch jobs can retry):

```yaml
initContainers:
  - name: fetch-binaries
    image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
    command: ["/bin/sh", "-c"]
    args:
      - |
        set -e
        rclone sync s3:foxhunt-binaries/latest/training/ /binaries/
        chmod +x /binaries/*
        echo "Fetched training binaries"
    volumeMounts:
      - name: binaries
        mountPath: /binaries
containers:
  - name: training
    image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
    command: ["/binaries/$(TRAINING_BINARY)"]
    volumeMounts:
      - name: binaries
        mountPath: /binaries
        readOnly: true
```

### Cache PVCs

8 PVCs, one per service deployment (500Mi each, scw-bssd):

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-trading-service
  namespace: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 500Mi
  storageClassName: scw-bssd
```

Total: 8 × 500Mi = 4Gi. Cost: ~€0.08/month.

### Resilience During Trading

1. **Running pods unaffected**: Binaries live in emptyDir for the pod's lifetime. S3 outage has zero impact on running services.
2. **Pod crash + S3 down**: initContainer falls back to PVC cache (last known good binary).
3. **Pod crash + S3 down + empty cache**: Only on first-ever deploy. Pod stays in Init:Error; deployment's `maxUnavailable: 0` prevents the old pod from terminating.
4. **Deployment strategy**: `maxSurge: 1, maxUnavailable: 0` ensures old pod keeps running until new pod is ready.

### What Gets Deleted

| Item | Notes |
|------|-------|
| `Dockerfile.runtime` | Replaced by `foxhunt-runtime` |
| `Dockerfile.training` | Replaced by `foxhunt-training-runtime` |
| `Dockerfile.web-gateway-runtime` | Replaced by `foxhunt-runtime` + initContainer |
| `Dockerfile.web-gateway` | Full multi-stage builder, no longer needed |
| `Dockerfile.service` | Full multi-stage builder, no longer needed |
| 9 Kaniko CI jobs | All `build-*` jobs in `.gitlab-ci.yml` |
| 10 per-service Docker images | In Scaleway CR and GitLab registry |

### What's Kept

| Item | Notes |
|------|-------|
| `Dockerfile.ci-builder` | Compile infrastructure, unchanged |
| `Dockerfile.ci-builder-cpu` | Compile infrastructure, unchanged |
| `Dockerfile.infra-runner` | IaC pipeline, unchanged |
| Image-prepuller DaemonSet | Updated to pull `foxhunt-training-runtime` |

### Timing Comparison

| Step | Before | After |
|------|--------|-------|
| Compile | 8-22 min | 8-22 min (unchanged) |
| Image build | 2-5 min (9 Kaniko) | 0 (eliminated) |
| Registry push | 1-2 min | ~10s (S3 upload) |
| Pod startup | 30s-3min (image pull) | 2-5s (initContainer S3 fetch) |
| Total pipeline | 15-30 min | 8-23 min |
| Pod restart | 30s-3min | 2-5s |

### New Infrastructure

| Resource | Spec | Cost |
|----------|------|------|
| S3 bucket `foxhunt-binaries` | fr-par, 30-day lifecycle | ~€0.01/month |
| 8 cache PVCs (500Mi each) | scw-bssd | ~€0.08/month |
| 3 base image Dockerfiles | In `infra/docker/` | 0 |

### rclone Configuration

rclone in CI and pods is configured via environment variables (no config file needed):

```
RCLONE_S3_PROVIDER=Scaleway
RCLONE_S3_ENDPOINT=s3.fr-par.scw.cloud
RCLONE_S3_REGION=fr-par
RCLONE_S3_ACCESS_KEY_ID=<from secret>
RCLONE_S3_SECRET_ACCESS_KEY=<from secret>
```

The `s3-credentials` K8s secret already exists in the foxhunt namespace (used by training_uploader). CI uses `SCW_ACCESS_KEY`/`SCW_SECRET_KEY` variables mapped to `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`.

For rclone in CI, the S3 remote is referenced as `s3:foxhunt-binaries/...` using `RCLONE_S3_*` env vars.

### rclone in ci-builder images

Add rclone to both `Dockerfile.ci-builder` and `Dockerfile.ci-builder-cpu` for the upload step:

```dockerfile
# Add to both ci-builder Dockerfiles
RUN curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
    && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
    && rm /tmp/rclone.zip
```