foxhunt/docs/plans/2026-02-28-s3-binary-share-implementation.md

# S3 Binary Share Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Replace per-service Docker images with generic base images + S3-based binary distribution to eliminate the Kaniko build stage and speed up pod startup.

**Architecture:** CI compiles binaries and uploads them to `s3://foxhunt-binaries/latest/`. Pods run pre-pulled generic base images with an initContainer that fetches the binary from S3 into an emptyDir. A per-service PVC cache provides fallback if S3 is unavailable. The build stage (9 Kaniko jobs) is entirely eliminated.

**Tech Stack:** Scaleway Object Storage (S3), rclone, Kubernetes initContainers, Terragrunt/Terraform, GitLab CI

**Design doc:** `docs/plans/2026-02-28-s3-binary-share-design.md`

---

### Task 1: Create S3 bucket via Terraform

**Files:**
- Modify: `infra/modules/object-storage/main.tf` (append new resource)
- Modify: `infra/modules/object-storage/outputs.tf` (append new output)

**Step 1: Add the foxhunt-binaries bucket resource**

In `infra/modules/object-storage/main.tf`, append after the `sccache` resource (line ~62):

```hcl
resource "scaleway_object_bucket" "binaries" {
  name   = "${var.bucket_name_prefix}-binaries"
  region = var.region

  lifecycle_rule {
    enabled = true
    prefix  = "archive/"

    expiration {
      days = 30
    }
  }

  versioning {
    enabled = false
  }
}
```

**Step 2: Add the output**

In `infra/modules/object-storage/outputs.tf`, append:

```hcl
output "binaries_bucket_name" {
  description = "Name of the binaries distribution bucket"
  value       = scaleway_object_bucket.binaries.name
}
```

**Step 3: Validate locally**

Run: `cd infra/live/production/object-storage && terragrunt validate`
Expected: Success (no errors)

**Step 4: Commit**

```bash
git add infra/modules/object-storage/main.tf infra/modules/object-storage/outputs.tf
git commit -m "infra: add foxhunt-binaries S3 bucket for binary distribution"
```

> **Note:** The actual `terragrunt apply` runs in CI (infra-apply job) or manually. The bucket name will be `foxhunt-binaries` (prefix `foxhunt` + `-binaries`).

---

### Task 2: Add rclone to CI builder images

**Files:**
- Modify: `infra/docker/Dockerfile.ci-builder` (add rclone install after sccache)
- Modify: `infra/docker/Dockerfile.ci-builder-cpu` (add rclone install after sccache)

**Step 1: Add rclone to Dockerfile.ci-builder**

In `infra/docker/Dockerfile.ci-builder`, after the sccache install block (line ~44, after `chmod +x /usr/local/bin/sccache`), add:

```dockerfile
# rclone for S3 binary uploads (compile → S3)
RUN curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
    && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
    && rm /tmp/rclone.zip \
    && chmod +x /usr/local/bin/rclone
```

Also add `rclone --version` to the verify line (line ~60):

```dockerfile
RUN rustc --version && cargo --version && git --version && protoc --version && sccache --version && make --version && nvcc --version && rclone --version
```

**Step 2: Add rclone to Dockerfile.ci-builder-cpu**

Same change in `infra/docker/Dockerfile.ci-builder-cpu`, after the sccache install (line ~43), add the same rclone block. Update verify line (line ~47):

```dockerfile
RUN rustc --version && cargo --version && git --version && protoc --version && sccache --version && make --version && mold --version && rclone --version
```

**Step 3: Commit**

```bash
git add infra/docker/Dockerfile.ci-builder infra/docker/Dockerfile.ci-builder-cpu
git commit -m "infra: add rclone to CI builder images for S3 binary uploads"
```

> **Note:** CI builder images are rebuilt via the `build-ci-builder` / `build-ci-builder-cpu` Kaniko jobs in the prepare stage. Trigger manually from the pipeline UI after merging.

---

### Task 3: Create 3 generic base image Dockerfiles

**Files:**
- Create: `infra/docker/Dockerfile.foxhunt-runtime`
- Create: `infra/docker/Dockerfile.foxhunt-training-runtime`
- Create: `infra/docker/Dockerfile.foxhunt-node-builder`

**Step 1: Create foxhunt-runtime**

Write `infra/docker/Dockerfile.foxhunt-runtime`:

```dockerfile
# Generic runtime base image for all Foxhunt service pods
# Contains: rclone (S3 binary fetch), grpc_health_probe (K8s health checks)
# Binaries are fetched by initContainer at pod startup, not baked into this image.
# Rebuild: only when system deps change (libssl, rclone, grpc_health_probe)

FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    libssl3 \
    curl \
    unzip \
    && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
    && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
    && rm /tmp/rclone.zip \
    && apt-get purge -y unzip \
    && rm -rf /var/lib/apt/lists/*

# grpc_health_probe for Kubernetes health checks
RUN curl -fsSL https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.25/grpc_health_probe-linux-amd64 \
    -o /usr/local/bin/grpc_health_probe \
    && chmod +x /usr/local/bin/grpc_health_probe

RUN groupadd -g 1000 foxhunt \
    && useradd -u 1000 -g foxhunt -m -s /bin/false foxhunt

USER foxhunt
```

**Step 2: Create foxhunt-training-runtime**

Write `infra/docker/Dockerfile.foxhunt-training-runtime`:

```dockerfile
# Generic GPU runtime base image for Foxhunt training pods
# Contains: CUDA 12.4 + cuDNN + NVRTC + rclone
# Binaries are fetched by initContainer at pod startup, not baked into this image.
# Rebuild: only on CUDA version bump

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    libssl3 \
    curl \
    unzip \
    cuda-nvrtc-12-4 \
    && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
    && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
    && rm /tmp/rclone.zip \
    && apt-get purge -y unzip \
    && rm -rf /var/lib/apt/lists/*

RUN groupadd -g 1000 foxhunt \
    && useradd -u 1000 -g foxhunt -m -s /bin/false foxhunt

ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility

USER foxhunt
```

**Step 3: Create foxhunt-node-builder**

Write `infra/docker/Dockerfile.foxhunt-node-builder`:

```dockerfile
# Node.js CI image for web-dashboard Vite builds + S3 upload
# Contains: Node 22 + rclone
# Rebuild: only on Node major version bump

FROM node:22-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    unzip \
    && curl -fsSL https://downloads.rclone.org/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o /tmp/rclone.zip \
    && unzip -j /tmp/rclone.zip '*/rclone' -d /usr/local/bin/ \
    && rm /tmp/rclone.zip \
    && apt-get purge -y unzip \
    && rm -rf /var/lib/apt/lists/*
```

**Step 4: Commit**

```bash
git add infra/docker/Dockerfile.foxhunt-runtime \
        infra/docker/Dockerfile.foxhunt-training-runtime \
        infra/docker/Dockerfile.foxhunt-node-builder
git commit -m "infra: add 3 generic base images for S3 binary share"
```

---

### Task 4: Add base image build jobs to CI prepare stage

**Files:**
- Modify: `.gitlab-ci.yml` (add 3 Kaniko jobs in the `prepare` stage, after existing builder jobs)

**Step 1: Add build-foxhunt-runtime job**

In `.gitlab-ci.yml`, after the `build-infra-runner` job (around line 188), add:

```yaml
# --------------------------------------------------------------------------
# Stage 0d: Build generic runtime base images → push to Scaleway CR
# --------------------------------------------------------------------------
build-foxhunt-runtime:
  stage: prepare
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  tags:
    - kapsule
    - docker
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"
      changes:
        - infra/docker/Dockerfile.foxhunt-runtime
      when: on_success
    - when: manual
      allow_failure: true
  before_script:
    - mkdir -p /kaniko/.docker
    - >-
      echo "{\"auths\":{
      \"rg.fr-par.scw.cloud\":{\"username\":\"nologin\",\"password\":\"${SCW_SECRET_KEY}\"},
      \"https://index.docker.io/v1/\":{\"username\":\"${DOCKERHUB_USERNAME}\",\"password\":\"${DOCKERHUB_TOKEN}\"}
      }}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/infra/docker/Dockerfile.foxhunt-runtime"
      --cache=true --cache-repo="${REGISTRY}/cache"
      --destination "${REGISTRY}/foxhunt-runtime:latest"

build-foxhunt-training-runtime:
  stage: prepare
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  tags:
    - kapsule
    - docker
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"
      changes:
        - infra/docker/Dockerfile.foxhunt-training-runtime
      when: on_success
    - when: manual
      allow_failure: true
  before_script:
    - mkdir -p /kaniko/.docker
    - >-
      echo "{\"auths\":{
      \"rg.fr-par.scw.cloud\":{\"username\":\"nologin\",\"password\":\"${SCW_SECRET_KEY}\"},
      \"https://index.docker.io/v1/\":{\"username\":\"${DOCKERHUB_USERNAME}\",\"password\":\"${DOCKERHUB_TOKEN}\"}
      }}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/infra/docker/Dockerfile.foxhunt-training-runtime"
      --cache=true --cache-repo="${REGISTRY}/cache"
      --destination "${REGISTRY}/foxhunt-training-runtime:latest"

build-foxhunt-node-builder:
  stage: prepare
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  tags:
    - kapsule
    - docker
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"
      changes:
        - infra/docker/Dockerfile.foxhunt-node-builder
      when: on_success
    - when: manual
      allow_failure: true
  before_script:
    - mkdir -p /kaniko/.docker
    - >-
      echo "{\"auths\":{
      \"rg.fr-par.scw.cloud\":{\"username\":\"nologin\",\"password\":\"${SCW_SECRET_KEY}\"},
      \"https://index.docker.io/v1/\":{\"username\":\"${DOCKERHUB_USERNAME}\",\"password\":\"${DOCKERHUB_TOKEN}\"}
      }}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/infra/docker/Dockerfile.foxhunt-node-builder"
      --cache=true --cache-repo="${REGISTRY}/cache"
      --destination "${REGISTRY}/foxhunt-node-builder:latest"
```

**Step 2: Also add these image names as variables at the top of `.gitlab-ci.yml`**

After the `INFRA_RUNNER_IMAGE` variable (around line 59), add:

```yaml
  # Generic runtime images for S3 binary share (service pods + training pods + node builder)
  RUNTIME_IMAGE: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
  TRAINING_RUNTIME_IMAGE: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
  NODE_BUILDER_IMAGE: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-node-builder:latest
```

**Step 3: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "ci: add build jobs for 3 generic base images in prepare stage"
```

---

### Task 5: Modify compile-services to upload to S3

**Files:**
- Modify: `.gitlab-ci.yml` — `compile-services` job

**Step 1: Replace the `artifacts:` block with S3 upload**

In the `compile-services` job, replace the `artifacts:` block (the final 3 lines, around line 326-328) and append upload steps to the `script:` section.

The entire `compile-services` job should become:

```yaml
compile-services:
  extends: .rust-base-cpu
  stage: compile
  needs: []
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - Cargo.toml
        - Cargo.lock
        - crates/**
        - bin/**
        - services/**
        - infra/docker/Dockerfile.*
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
  script:
    - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true
    - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release}
    - echo "Building services with --profile $PROFILE (CPU-only, no CUDA)"
    - TARGET_DIR="target/$PROFILE"
    - cargo build --profile $PROFILE
        -p trading_service
        -p api_gateway
        -p broker_gateway_service
        -p ml_training_service
        -p backtesting_service
        -p trading_agent_service
        -p data_acquisition_service
        -p web-gateway
    - sccache --show-stats || true
    - mkdir -p build-out/services
    - |
      for bin in trading_service api_gateway broker_gateway_service ml_training_service backtesting_service trading_agent_service data_acquisition_service web-gateway; do
        cp $TARGET_DIR/$bin build-out/services/
        strip build-out/services/$bin
      done
    - ls -lh build-out/services/
    # Upload stripped binaries to S3
    - export RCLONE_S3_PROVIDER=Scaleway RCLONE_S3_ENDPOINT=s3.fr-par.scw.cloud RCLONE_S3_REGION=fr-par
    - export RCLONE_S3_ACCESS_KEY_ID=$SCW_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$SCW_SECRET_KEY
    - |
      for bin in trading_service api_gateway broker_gateway_service ml_training_service backtesting_service trading_agent_service data_acquisition_service web-gateway; do
        rclone copyto build-out/services/$bin :s3:foxhunt-binaries/latest/services/$bin
      done
    - rclone copy build-out/services/ :s3:foxhunt-binaries/archive/${CI_COMMIT_SHA}/services/
    - echo "Service binaries uploaded to S3"
```

Note: No more `artifacts:` block. The `:s3:` prefix is rclone's backend syntax when using env-var config (no named remote).

**Step 2: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "ci: compile-services uploads to S3 instead of GitLab artifacts"
```

---

### Task 6: Modify compile-training to upload to S3

**Files:**
- Modify: `.gitlab-ci.yml` — `compile-training` job

**Step 1: Replace artifacts with S3 upload**

The `compile-training` job currently ends with `artifacts:` (around line 383-384). Replace the entire job to match this:

```yaml
compile-training:
  extends: .rust-base-cpu
  stage: compile
  image: ${CI_BUILDER_IMAGE}
  needs: [compile-services]
  variables:
    CUDA_COMPUTE_CAP: "89"
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - Cargo.toml
        - Cargo.lock
        - crates/ml/**
        - crates/trading_engine/**
        - crates/common/**
        - crates/risk/**
        - crates/data/**
        - crates/config/**
        - crates/storage/**
        - services/training_uploader/**
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
  script:
    - echo "CUDA_COMPUTE_CAP=$CUDA_COMPUTE_CAP"
    - export SCCACHE_DIR="/mnt/sccache/sm_${CUDA_COMPUTE_CAP}"
    - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true
    - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release}
    - echo "Building training binaries with --profile $PROFILE (CUDA stubs)"
    - TARGET_DIR="target/$PROFILE"
    - cargo build --profile $PROFILE -p ml --features ml/cuda
        --example train_baseline_rl
        --example train_baseline_supervised
        --example evaluate_baseline
        --example evaluate_supervised
        --example hyperopt_baseline_rl
        --example hyperopt_baseline_supervised
    - cargo build --profile $PROFILE -p training_uploader
    - sccache --show-stats || true
    - mkdir -p build-out/training
    - |
      for bin in train_baseline_rl train_baseline_supervised evaluate_baseline evaluate_supervised hyperopt_baseline_rl hyperopt_baseline_supervised; do
        cp $TARGET_DIR/examples/$bin build-out/training/
        strip build-out/training/$bin
      done
      cp $TARGET_DIR/training_uploader build-out/training/
      strip build-out/training/training_uploader
    - ls -lh build-out/training/
    # Upload stripped binaries to S3
    - export RCLONE_S3_PROVIDER=Scaleway RCLONE_S3_ENDPOINT=s3.fr-par.scw.cloud RCLONE_S3_REGION=fr-par
    - export RCLONE_S3_ACCESS_KEY_ID=$SCW_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$SCW_SECRET_KEY
    - |
      for bin in train_baseline_rl train_baseline_supervised evaluate_baseline evaluate_supervised hyperopt_baseline_rl hyperopt_baseline_supervised training_uploader; do
        rclone copyto build-out/training/$bin :s3:foxhunt-binaries/latest/training/$bin
      done
    - rclone copy build-out/training/ :s3:foxhunt-binaries/archive/${CI_COMMIT_SHA}/training/
    - echo "Training binaries uploaded to S3"
```

**Step 2: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "ci: compile-training uploads to S3 instead of GitLab artifacts"
```

---

### Task 7: Add build-web-dashboard, write-manifest, and deploy CI jobs

**Files:**
- Modify: `.gitlab-ci.yml` — add 3 new jobs

**Step 1: Add build-web-dashboard job**

After the `compile-training` job, add:

```yaml
# --------------------------------------------------------------------------
# Stage 3c: Build web-dashboard static assets (Vite)
# --------------------------------------------------------------------------
build-web-dashboard:
  stage: compile
  image: ${NODE_BUILDER_IMAGE}
  needs: []
  tags:
    - kapsule
    - docker
  variables:
    KUBERNETES_CPU_REQUEST: "2000m"
    KUBERNETES_CPU_LIMIT: "4000m"
    KUBERNETES_MEMORY_REQUEST: "2Gi"
    KUBERNETES_MEMORY_LIMIT: "4Gi"
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - web-dashboard/**
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
  script:
    - cd web-dashboard
    - npm ci
    - npm run build
    - export RCLONE_S3_PROVIDER=Scaleway RCLONE_S3_ENDPOINT=s3.fr-par.scw.cloud RCLONE_S3_REGION=fr-par
    - export RCLONE_S3_ACCESS_KEY_ID=$SCW_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$SCW_SECRET_KEY
    - rclone sync dist/ :s3:foxhunt-binaries/latest/web-dashboard/dist/
    - echo "Web dashboard assets uploaded to S3"

# --------------------------------------------------------------------------
# Stage 3d: Write manifest after all compiles complete
# --------------------------------------------------------------------------
write-manifest:
  stage: compile
  image: ${RUNTIME_IMAGE}
  needs:
    - job: compile-services
    - job: compile-training
    - job: build-web-dashboard
      optional: true
  tags:
    - kapsule
    - docker
  variables:
    KUBERNETES_CPU_REQUEST: "100m"
    KUBERNETES_MEMORY_REQUEST: "64Mi"
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
  script:
    - export RCLONE_S3_PROVIDER=Scaleway RCLONE_S3_ENDPOINT=s3.fr-par.scw.cloud RCLONE_S3_REGION=fr-par
    - export RCLONE_S3_ACCESS_KEY_ID=$SCW_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$SCW_SECRET_KEY
    - echo "{\"sha\":\"${CI_COMMIT_SHA}\",\"ts\":\"$(date -Iseconds)\"}" | rclone rcat :s3:foxhunt-binaries/latest/MANIFEST.json
    - echo "Manifest written for ${CI_COMMIT_SHA}"
```

**Step 2: Add deploy job (at the end of the file, in the deploy stage)**

After the training jobs section, add:

```yaml
# --------------------------------------------------------------------------
# Stage 5: Deploy — rolling restart of all service deployments
# --------------------------------------------------------------------------
deploy-services:
  stage: deploy
  image:
    name: bitnami/kubectl:latest
    entrypoint: [""]
  tags:
    - kapsule
    - docker
  needs:
    - job: write-manifest
  variables:
    KUBERNETES_CPU_REQUEST: "100m"
    KUBERNETES_MEMORY_REQUEST: "64Mi"
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      when: manual
      allow_failure: true
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
      when: manual
      allow_failure: true
  script:
    - echo "Deploying ${CI_COMMIT_SHA}..."
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout restart deployment/$svc
        echo "Restarted $svc"
      done
    - |
      for svc in trading-service api-gateway broker-gateway \
                 ml-training-service backtesting-service \
                 trading-agent-service data-acquisition-service web-gateway; do
        kubectl -n foxhunt rollout status deployment/$svc --timeout=120s
      done
    - echo "All services deployed"
```

**Step 3: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "ci: add build-web-dashboard, write-manifest, and deploy-services jobs"
```

---

### Task 8: Delete all Kaniko build jobs from CI

**Files:**
- Modify: `.gitlab-ci.yml` — remove the entire build stage section

**Step 1: Delete the following blocks**

Remove these sections from `.gitlab-ci.yml`:
1. `.kaniko-service-base` template (the `stage: build` template starting with "Kaniko base for service images")
2. `.kaniko-training-base` template
3. `build-trading-service` job
4. `build-api-gateway` job
5. `build-broker-gateway` job
6. `build-ml-training` job
7. `build-backtesting` job
8. `build-trading-agent` job
9. `build-data-acquisition` job
10. `build-web-gateway` job
11. `build-training` job

These are roughly lines 390-549 in the current `.gitlab-ci.yml`.

Also remove `build` from the `stages:` list (line 31). The stages should become:

```yaml
stages:
  - prepare
  - test
  - compile
  - train
  - deploy
```

**Step 2: Update training job `needs:`**

All training jobs currently have `needs: [build-training]` (with `optional: true`). Change them to depend on `compile-training` instead:

Find all instances of:
```yaml
  needs:
    - job: build-training
      optional: true
```

Replace with:
```yaml
  needs:
    - job: compile-training
      optional: true
```

This applies to: `train-validate-rl`, `train-validate-tft`, `train-validate-mamba2`, `train-validate-liquid`, `train-validate-tggn`, `train-validate-tlob`, `train-validate-kan`, `train-validate-xlstm`, `train-validate-diffusion`, `hyperopt-ppo`, `hyperopt-dqn`, `hyperopt-tft`, `hyperopt-mamba2`, `hyperopt-liquid`, `hyperopt-tggn`.

**Step 3: Update training job `image:`**

All training jobs currently use `image: ${REGISTRY}/training:${CI_COMMIT_SHA}`. Change to the generic runtime:

Find: `image: ${REGISTRY}/training:${CI_COMMIT_SHA}`
Replace with: `image: ${TRAINING_RUNTIME_IMAGE}`

This applies to `.train-rl-base` and `.train-validate-base` templates.

**Step 4: Update training job scripts — fetch binaries first**

In `.train-rl-base` and `.train-validate-base`, add binary fetch to `before_script:`:

Replace the existing `before_script:` in `.train-rl-base`:
```yaml
  before_script:
    - export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
    - nvidia-smi
    - mkdir -p ${CI_PROJECT_DIR}/output
```

With:
```yaml
  before_script:
    - export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
    - nvidia-smi
    - mkdir -p ${CI_PROJECT_DIR}/output
    # Fetch training binaries from S3
    - export RCLONE_S3_PROVIDER=Scaleway RCLONE_S3_ENDPOINT=s3.fr-par.scw.cloud RCLONE_S3_REGION=fr-par
    - export RCLONE_S3_ACCESS_KEY_ID=$SCW_ACCESS_KEY RCLONE_S3_SECRET_ACCESS_KEY=$SCW_SECRET_KEY
    - rclone sync :s3:foxhunt-binaries/latest/training/ /usr/local/bin/ --include "*.{rl,supervised,baseline,uploader}*" 2>/dev/null || rclone sync :s3:foxhunt-binaries/latest/training/ /usr/local/bin/
    - chmod +x /usr/local/bin/train_baseline_rl /usr/local/bin/train_baseline_supervised /usr/local/bin/evaluate_baseline /usr/local/bin/evaluate_supervised /usr/local/bin/hyperopt_baseline_rl /usr/local/bin/hyperopt_baseline_supervised /usr/local/bin/training_uploader
```

Same change for `.train-validate-base`.

**Step 5: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "ci: delete all Kaniko build jobs, switch training to S3 binary fetch"
```

---

### Task 9: Create binary cache PVCs

**Files:**
- Create: `infra/k8s/storage/binary-cache-pvcs.yaml`

**Step 1: Write the PVC definitions**

Write `infra/k8s/storage/binary-cache-pvcs.yaml`:

```yaml
# Per-service binary cache PVCs — fallback when S3 is unavailable
# Each service gets its own 500Mi PVC (ReadWriteOnce, scw-bssd)
# Total: 8 × 500Mi = 4Gi, ~€0.08/month
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-trading-service
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-api-gateway
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-broker-gateway
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-ml-training-service
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-backtesting-service
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-trading-agent-service
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-data-acquisition-service
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: binary-cache-web-gateway
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: binary-cache
    app.kubernetes.io/part-of: foxhunt
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 1Gi
```

Note: web-gateway gets 1Gi because it also caches the dashboard static assets (~5MB).

**Step 2: Commit**

```bash
git add infra/k8s/storage/binary-cache-pvcs.yaml
git commit -m "infra: add 8 binary cache PVCs for S3 fallback during trading"
```

> **Apply:** `kubectl apply -f infra/k8s/storage/binary-cache-pvcs.yaml`

---

### Task 10: Update all 8 service deployment YAMLs

This is the largest task. Each deployment gets:
1. Image changed from per-service GitLab registry → generic `foxhunt-runtime` from SCW registry
2. `imagePullSecrets` changed from `gitlab-registry` → `scw-registry`
3. `initContainers` added for S3 binary fetch with PVC cache fallback
4. `command` added to main container to run from `/binaries/`
5. `volumes` updated with `binaries` emptyDir + `binary-cache` PVC
6. Deployment strategy set to `maxSurge: 1, maxUnavailable: 0`

**Files:**
- Modify: `infra/k8s/services/trading-service.yaml`
- Modify: `infra/k8s/services/api-gateway.yaml`
- Modify: `infra/k8s/services/broker-gateway.yaml`
- Modify: `infra/k8s/services/ml-training-service.yaml`
- Modify: `infra/k8s/services/backtesting-service.yaml`
- Modify: `infra/k8s/services/trading-agent-service.yaml`
- Modify: `infra/k8s/services/data-acquisition-service.yaml`
- Modify: `infra/k8s/services/web-gateway.yaml`

**The pattern for each service (example: trading-service):**

For each deployment YAML, make these changes:

1. **Add deployment strategy** after `spec.replicas`:
```yaml
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
```

2. **Change `imagePullSecrets`** from `gitlab-registry` to `scw-registry`:
```yaml
      imagePullSecrets:
        - name: scw-registry
```

3. **Add `initContainers`** before `containers`:
```yaml
      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              SERVICE=trading_service
              if rclone copyto :s3:foxhunt-binaries/latest/services/$SERVICE /binaries/$SERVICE 2>/dev/null; then
                chmod +x /binaries/$SERVICE
                cp /binaries/$SERVICE /cache/$SERVICE
                echo "Fetched $SERVICE from S3 ($(stat -c%s /binaries/$SERVICE) bytes)"
              elif [ -f /cache/$SERVICE ]; then
                cp /cache/$SERVICE /binaries/$SERVICE
                chmod +x /binaries/$SERVICE
                echo "WARNING: S3 unavailable, using cached $SERVICE"
              else
                echo "FATAL: No binary available for $SERVICE (S3 down + empty cache)"
                exit 1
              fi
          env:
            - name: RCLONE_S3_PROVIDER
              value: Scaleway
            - name: RCLONE_S3_ENDPOINT
              value: s3.fr-par.scw.cloud
            - name: RCLONE_S3_REGION
              value: fr-par
            - name: RCLONE_S3_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: s3-credentials
                  key: access-key
            - name: RCLONE_S3_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: s3-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
            - name: binary-cache
              mountPath: /cache
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
```

4. **Change container image** from `git.fxhnt.ai:5050/foxhunt/foxhunt/<service>:latest` to:
```yaml
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
```

5. **Add `command`** to the main container:
```yaml
          command: ["/binaries/trading_service"]
```

6. **Add volume mounts** to the main container:
```yaml
          volumeMounts:
            - name: binaries
              mountPath: /binaries
              readOnly: true
```
(Keep existing volumeMounts like `logs`, `tls-certs`)

7. **Add volumes** to the pod spec:
```yaml
      volumes:
        - name: binaries
          emptyDir:
            sizeLimit: 200Mi
        - name: binary-cache
          persistentVolumeClaim:
            claimName: binary-cache-trading-service
```
(Keep existing volumes like `logs`, `tls-certs`)

**Per-service binary names and PVC names:**

| Deployment | Binary name | PVC name | Special |
|-----------|-------------|----------|---------|
| trading-service | `trading_service` | `binary-cache-trading-service` | — |
| api-gateway | `api_gateway` | `binary-cache-api-gateway` | — |
| broker-gateway | `broker_gateway_service` | `binary-cache-broker-gateway` | — |
| ml-training-service | `ml_training_service` | `binary-cache-ml-training-service` | Has `command: ["./ml_training_service", "serve"]` → change to `command: ["/binaries/ml_training_service", "serve"]` |
| backtesting-service | `backtesting_service` | `binary-cache-backtesting-service` | — |
| trading-agent-service | `trading_agent_service` | `binary-cache-trading-agent-service` | — |
| data-acquisition-service | `data_acquisition_service` | `binary-cache-data-acquisition-service` | Uses `scw-registry` already |
| web-gateway | `web-gateway` | `binary-cache-web-gateway` | See below |

**web-gateway special case:**

The initContainer fetches both the binary AND dashboard static assets:

```yaml
      initContainers:
        - name: fetch-binary
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              SERVICE=web-gateway
              if rclone copyto :s3:foxhunt-binaries/latest/services/$SERVICE /binaries/$SERVICE 2>/dev/null; then
                chmod +x /binaries/$SERVICE
                cp /binaries/$SERVICE /cache/$SERVICE
                rclone sync :s3:foxhunt-binaries/latest/web-dashboard/dist/ /binaries/static/
                cp -r /binaries/static /cache/static 2>/dev/null || true
                echo "Fetched $SERVICE + static assets from S3"
              elif [ -f /cache/$SERVICE ]; then
                cp /cache/$SERVICE /binaries/$SERVICE
                chmod +x /binaries/$SERVICE
                cp -r /cache/static /binaries/static 2>/dev/null || true
                echo "WARNING: S3 unavailable, using cached $SERVICE"
              else
                echo "FATAL: No binary available for $SERVICE"
                exit 1
              fi
          env:
            # ... same RCLONE_S3_* env vars ...
          volumeMounts:
            - name: binaries
              mountPath: /binaries
            - name: binary-cache
              mountPath: /cache
```

The web-gateway container gets:
```yaml
          command: ["/binaries/web-gateway"]
```

And the `STATIC_DIR` environment variable (if the binary supports it, otherwise it defaults to `./static/` which would be `/binaries/static/` since the workdir would need adjustment — check the binary).

> **Important:** The web-gateway binary currently serves static files from `./static/` (relative to WORKDIR `/app`). With the new layout, static files are at `/binaries/static/`. Either:
> (a) Set env `STATIC_DIR=/binaries/static/` if the binary supports it, or
> (b) Symlink: add `ln -sf /binaries/static /app/static` to the initContainer.
> Check the Rust source (`services/web-gateway/src/main.rs` or similar) to determine the static path config.

**Step 1-8: Update each YAML file** following the pattern above.

**Step 9: Commit**

```bash
git add infra/k8s/services/trading-service.yaml \
        infra/k8s/services/api-gateway.yaml \
        infra/k8s/services/broker-gateway.yaml \
        infra/k8s/services/ml-training-service.yaml \
        infra/k8s/services/backtesting-service.yaml \
        infra/k8s/services/trading-agent-service.yaml \
        infra/k8s/services/data-acquisition-service.yaml \
        infra/k8s/services/web-gateway.yaml
git commit -m "infra: update all service deployments with S3 binary fetch initContainer"
```

---

### Task 11: Update GPU overlay deployments

**Files:**
- Modify: `infra/k8s/services/trading-service-gpu.yaml`
- Modify: `infra/k8s/services/ml-training-service-gpu.yaml`

Apply the same changes as Task 10 to both GPU overlay files. The only differences from the base manifests are `nodeSelector` (gpu-inference), `tolerations`, and resource requests (GPU).

**Step 1: Commit**

```bash
git add infra/k8s/services/trading-service-gpu.yaml \
        infra/k8s/services/ml-training-service-gpu.yaml
git commit -m "infra: update GPU overlay deployments with S3 binary fetch"
```

---

### Task 12: Update training job template

**Files:**
- Modify: `infra/k8s/training/job-template.yaml`

**Step 1: Update the job template**

Change the image from `rg.fr-par.scw.cloud/foxhunt-ci/training:latest` to `rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest`.

Add an initContainer to fetch training binaries. The training job doesn't need a cache PVC (batch jobs can retry).

Replace the `initContainers:` section (currently the uploader sidecar) to also include binary fetch, and update the main container to use fetched binaries:

```yaml
      initContainers:
        # Fetch training binaries from S3
        - name: fetch-binaries
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              export RCLONE_S3_PROVIDER=Scaleway RCLONE_S3_ENDPOINT=s3.fr-par.scw.cloud RCLONE_S3_REGION=fr-par
              rclone sync :s3:foxhunt-binaries/latest/training/ /binaries/
              chmod +x /binaries/*
              ls -lh /binaries/
              echo "Fetched training binaries from S3"
          env:
            - name: RCLONE_S3_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: s3-credentials
                  key: access-key
            - name: RCLONE_S3_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: s3-credentials
                  key: secret-key
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 256Mi
        # Uploader sidecar (native sidecar, restartPolicy: Always)
        - name: uploader
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
          restartPolicy: Always
          command: ["/binaries/training_uploader"]
          # ... rest of uploader config unchanged, but add binaries mount ...
          volumeMounts:
            - name: output
              mountPath: /output
              readOnly: true
            - name: binaries
              mountPath: /binaries
              readOnly: true
```

Update main container:
```yaml
      containers:
        - name: training
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
          command: ["/binaries/$(TRAINING_BINARY)"]
          volumeMounts:
            - name: training-data
              mountPath: /data
              readOnly: true
            - name: output
              mountPath: /output
            - name: binaries
              mountPath: /binaries
              readOnly: true
```

Add the binaries volume:
```yaml
      volumes:
        - name: training-data
          persistentVolumeClaim:
            claimName: training-data-pvc
        - name: output
          emptyDir:
            sizeLimit: 2Gi
        - name: binaries
          emptyDir:
            sizeLimit: 500Mi
```

**Step 2: Commit**

```bash
git add infra/k8s/training/job-template.yaml
git commit -m "infra: update training job template with S3 binary fetch"
```

---

### Task 13: Update image-prepuller DaemonSet

**Files:**
- Modify: `infra/k8s/training/image-prepuller.yaml`

**Step 1: Change the init container image**

Replace:
```yaml
        - name: pull-training
          image: rg.fr-par.scw.cloud/foxhunt-ci/training:latest
          command: ["echo", "training image pulled"]
```

With:
```yaml
        - name: pull-training-runtime
          image: rg.fr-par.scw.cloud/foxhunt-ci/foxhunt-training-runtime:latest
          command: ["echo", "foxhunt-training-runtime image pulled"]
```

**Step 2: Commit**

```bash
git add infra/k8s/training/image-prepuller.yaml
git commit -m "infra: update image-prepuller to cache foxhunt-training-runtime"
```

---

### Task 14: Delete old Dockerfiles

**Files:**
- Delete: `infra/docker/Dockerfile.runtime`
- Delete: `infra/docker/Dockerfile.training`
- Delete: `infra/docker/Dockerfile.web-gateway-runtime`
- Delete: `infra/docker/Dockerfile.web-gateway`
- Delete: `infra/docker/Dockerfile.service`

**Step 1: Remove the files**

```bash
git rm infra/docker/Dockerfile.runtime \
       infra/docker/Dockerfile.training \
       infra/docker/Dockerfile.web-gateway-runtime \
       infra/docker/Dockerfile.web-gateway \
       infra/docker/Dockerfile.service
```

**Step 2: Commit**

```bash
git commit -m "infra: delete 5 obsolete Dockerfiles replaced by S3 binary share"
```

---

### Task 15: Verify and seed — first manual pipeline run

This is a manual verification task, not automated.

**Step 1: Build base images first**

Trigger the 3 new base image builds from GitLab pipeline UI:
- `build-foxhunt-runtime` (manual trigger)
- `build-foxhunt-training-runtime` (manual trigger)
- `build-foxhunt-node-builder` (manual trigger)

Also trigger CI builder rebuilds (they now include rclone):
- `build-ci-builder` (manual trigger)
- `build-ci-builder-cpu` (manual trigger)

**Step 2: Apply the S3 bucket via Terragrunt**

```bash
cd infra/live/production/object-storage
terragrunt apply
```

**Step 3: Apply PVCs**

```bash
kubectl apply -f infra/k8s/storage/binary-cache-pvcs.yaml
```

**Step 4: Run a full pipeline**

Push to main — the compile jobs should upload to S3, write-manifest should succeed, and deploy-services should trigger rolling restarts.

**Step 5: Verify pods**

```bash
kubectl -n foxhunt get pods
kubectl -n foxhunt logs deployment/trading-service -c fetch-binary
```

**Step 6: Apply service manifests**

```bash
kubectl apply -f infra/k8s/services/
kubectl apply -f infra/k8s/training/image-prepuller.yaml
```

---

### Execution Order Summary

| Task | Dependencies | Can Parallelize |
|------|-------------|-----------------|
| 1. S3 bucket TF | None | Yes |
| 2. rclone in CI builders | None | Yes (with 1) |
| 3. Base image Dockerfiles | None | Yes (with 1, 2) |
| 4. Base image CI build jobs | 3 | No |
| 5. compile-services S3 upload | 2 | Yes (with 6, 7) |
| 6. compile-training S3 upload | 2 | Yes (with 5, 7) |
| 7. New CI jobs (dashboard, manifest, deploy) | 3 | Yes (with 5, 6) |
| 8. Delete Kaniko build jobs | 5, 6, 7 | No |
| 9. Binary cache PVCs | None | Yes (with anything) |
| 10. Service deployment YAMLs | 3, 9 | Yes (with 11, 12) |
| 11. GPU overlay YAMLs | 10 | Yes (with 12) |
| 12. Training job template | 3 | Yes (with 10, 11) |
| 13. Image-prepuller update | 3 | Yes |
| 14. Delete old Dockerfiles | 8, 10 | No |
| 15. Manual verify + seed | All above | No |