foxhunt/docs/plans/2026-03-07-gitlab-releases-plan.md

GitLab Releases with CalVer Auto-Versioning — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Replace MinIO binary distribution with GitLab Generic Package Registry + Releases, with CalVer auto-tagging on every code push to main.

Architecture: Argo CI pipeline creates a CalVer git tag, compiles binaries, uploads them to GitLab's Generic Package Registry, creates a GitLab Release with auto-generated notes, then patches service deployments to fetch from GitLab. MinIO retains sccache, training data, and model checkpoints.

Tech Stack: Argo Workflows, GitLab CE API v4, Generic Package Registry, K8s strategic merge patches, curl

---

Context

GitLab internal URL: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181 GitLab project ID: 1 (root/foxhunt) Existing secrets: gitlab-pat (auto-rotated PAT, monthly), minio-credentials Existing file: crates/common/src/build_info.rs — already captures FOXHUNT_BUILD_VERSION at compile time Web dashboard: stays on MinIO (not code, not part of release)

Services affected (7): api, trading-service, ml-training-service, backtesting-service, trading-agent-service, broker-gateway, data-acquisition-service

Binary names from cargo build:

• Services: api, trading-service, ml-training-service, backtesting-service, trading-agent-service, broker-gateway, data-acquisition-service
• Training: train_baseline_rl, train_baseline_supervised, evaluate_baseline, evaluate_supervised, hyperopt_baseline_rl, hyperopt_baseline_supervised, training_uploader

---

Task 1: Create GitLab deploy token and K8s secret

This is a one-time manual setup. The deploy token lets service pods download packages from GitLab's Generic Package Registry (read-only).

Step 1: Create deploy token via GitLab API

Run from a machine with access to the cluster:

# Use existing PAT to create a project deploy token
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
PAT=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)

curl -sf -X POST "${GITLAB}/api/v4/projects/1/deploy_tokens" \
  -H "PRIVATE-TOKEN: ${PAT}" \
  -d "name=foxhunt-package-reader" \
  -d "scopes[]=read_package_registry" | jq .

Save the returned token value.

Step 2: Create K8s secret

kubectl -n foxhunt create secret generic gitlab-deploy-token \
  --from-literal=token=<token-from-step-1>

Step 3: Verify token can read packages

# Should return 200 (empty list is fine — no packages yet)
curl -sf -o /dev/null -w "%{http_code}" \
  -H "DEPLOY-TOKEN: <token>" \
  "${GITLAB}/api/v4/projects/1/packages"

Expected: 200

Step 4: Verify existing gitlab-pat has api scope

curl -sf "${GITLAB}/api/v4/personal_access_tokens/self" \
  -H "PRIVATE-TOKEN: ${PAT}" | jq '.scopes'

Expected: scopes must include api (for creating tags, releases, uploading packages). If not, recreate the PAT with api scope.

---

Task 2: Add needs-code output to detect-changes

Files:

• Modify: infra/k8s/argo/ci-pipeline-template.yaml:191-220 (detect-changes script outputs)

Step 1: Add needs-code computation to detect-changes script

In the detect-changes shell script, after the NEEDS_TRAINING computation (line ~207) and before the echo "=== Build decisions ===" line, add:

if [ "$NEEDS_SERVICES" = "true" ] || [ "$NEEDS_TRAINING" = "true" ]; then
  NEEDS_CODE="true"
else
  NEEDS_CODE="false"
fi

And after echo -n "$DOCKER_IMAGES" > /tmp/outputs/docker-images, add:

echo -n "$NEEDS_CODE" > /tmp/outputs/needs-code

Also add the build decisions echo:

echo "needs-code: $NEEDS_CODE"

Step 2: Add needs-code output parameter

After the docker-images output parameter (line ~235), add:

          - name: needs-code
            valueFrom:
              path: /tmp/outputs/needs-code

Step 3: Verify YAML is valid

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"

Expected: no error

---

Task 3: New create-tag template

Files:

• Modify: infra/k8s/argo/ci-pipeline-template.yaml (add template after detect-changes, before compile-services)

Step 1: Add create-tag template

Insert after the detect-changes template outputs: block (after line ~237) and before the build-web-dashboard template:

    # ── create-tag: CalVer auto-tag on code changes ──
    - name: create-tag
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      container:
        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        env:
          - name: GITLAB_PAT
            valueFrom:
              secretKeyRef:
                name: gitlab-pat
                key: token
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi
        args:
          - |
            set -e
            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
            PROJECT_ID=1
            SHA="{{workflow.parameters.commit-sha}}"

            # Compute CalVer prefix: vYYYY.MM
            PREFIX="v$(date +%Y.%m)"

            # Query existing tags for this month
            TAGS=$(curl -sf \
              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?search=${PREFIX}" \
              || echo "[]")

            # Parse highest N from vYYYY.MM.N tags
            LAST_N=$(echo "$TAGS" | grep -oP "\"name\":\"${PREFIX}\.\K[0-9]+" | sort -n | tail -1)
            if [ -z "$LAST_N" ]; then
              NEXT_N=1
            else
              NEXT_N=$((LAST_N + 1))
            fi

            TAG="${PREFIX}.${NEXT_N}"
            echo "Creating tag: ${TAG} at ${SHA}"

            # Create the tag
            RESULT=$(curl -sf -X POST \
              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
              -d "tag_name=${TAG}" \
              -d "ref=${SHA}" \
              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags")

            echo "$RESULT" | grep -q "$TAG" || { echo "Tag creation failed: $RESULT"; exit 1; }
            echo "Tag ${TAG} created successfully"

            mkdir -p /tmp/outputs
            echo -n "$TAG" > /tmp/outputs/tag
      outputs:
        parameters:
          - name: tag
            valueFrom:
              path: /tmp/outputs/tag

Step 2: Add gitlab-pat volume to workflow spec

In the spec.volumes list (around line 22-32), add:

    - name: gitlab-pat
      secret:
        secretName: gitlab-pat

Step 3: Verify YAML

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"

Expected: no error

---

Task 4: Modify compile-services — version from tag + upload to GitLab packages

Files:

• Modify: infra/k8s/argo/ci-pipeline-template.yaml (compile-services template, lines ~309-426)

Step 1: Replace version computation

Find and replace the 3 lines (around line 393-395):

YEAR=$(date +%Y)
MONTH=$(date +%m)
export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.argo"

With:

export FOXHUNT_BUILD_VERSION="{{tasks.create-tag.outputs.parameters.tag}}"

Step 2: Replace MinIO upload with GitLab package upload

Find and replace the rclone upload block (around lines 418-424):

echo "=== Uploading service binaries ==="
rclone copy "$WORKSPACE/bin/services/" :s3:foxhunt-binaries/services/ \
  --s3-provider=Minio \
  --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
  --s3-access-key-id="${MINIO_ACCESS_KEY}" \
  --s3-secret-access-key="${MINIO_SECRET_KEY}" \
  --s3-no-check-bucket

With:

echo "=== Uploading service binaries to GitLab packages ==="
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
TAG="${FOXHUNT_BUILD_VERSION}"
for bin in "$WORKSPACE/bin/services/"*; do
  BIN_NAME=$(basename "$bin")
  echo "Uploading ${BIN_NAME}..."
  curl -f --upload-file "$bin" \
    -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
    "${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-services/${TAG}/${BIN_NAME}"
done

Step 3: Add GITLAB_PAT env var, remove MINIO upload env vars

In the compile-services container env: list, add:

          - name: GITLAB_PAT
            valueFrom:
              secretKeyRef:
                name: gitlab-pat
                key: token

Remove the MINIO_ACCESS_KEY and MINIO_SECRET_KEY env entries (keep AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY — they're needed by sccache).

Step 4: Verify YAML

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"

---

Task 5: Modify compile-training — version from tag + upload to GitLab packages

Files:

• Modify: infra/k8s/argo/ci-pipeline-template.yaml (compile-training template, lines ~430-547)

Step 1: Replace version computation

Find and replace (around line 513-514):

YEAR=$(date +%Y)
MONTH=$(date +%m)
export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.argo"

With:

export FOXHUNT_BUILD_VERSION="{{tasks.create-tag.outputs.parameters.tag}}"

Step 2: Replace MinIO upload with GitLab package upload

Find and replace the rclone upload block (around lines 539-545):

echo "=== Uploading training binaries ==="
rclone copy "$WORKSPACE/bin/training/" :s3:foxhunt-binaries/training/ \
  --s3-provider=Minio \
  --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
  --s3-access-key-id="${MINIO_ACCESS_KEY}" \
  --s3-secret-access-key="${MINIO_SECRET_KEY}" \
  --s3-no-check-bucket

With:

echo "=== Uploading training binaries to GitLab packages ==="
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
TAG="${FOXHUNT_BUILD_VERSION}"
for bin in "$WORKSPACE/bin/training/"*; do
  BIN_NAME=$(basename "$bin")
  echo "Uploading ${BIN_NAME}..."
  curl -f --upload-file "$bin" \
    -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
    "${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-training/${TAG}/${BIN_NAME}"
done

Step 3: Add GITLAB_PAT env var, remove MINIO upload env vars

Same as Task 4 Step 3. Add GITLAB_PAT from gitlab-pat secret. Remove MINIO_ACCESS_KEY and MINIO_SECRET_KEY (keep AWS_* for sccache).

Step 4: Verify YAML

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"

---

Task 6: New upload-release template

Files:

• Modify: infra/k8s/argo/ci-pipeline-template.yaml (add template after compile-training, before gpu-warmup)

Step 1: Add upload-release template

    # ── upload-release: create GitLab Release with package links ──
    - name: upload-release
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      container:
        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        env:
          - name: GITLAB_PAT
            valueFrom:
              secretKeyRef:
                name: gitlab-pat
                key: token
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi
        args:
          - |
            set -e
            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
            PROJECT_ID=1
            TAG="{{tasks.create-tag.outputs.parameters.tag}}"

            echo "=== Creating GitLab Release ${TAG} ==="

            # Get commits since previous tag for release notes
            PREV_TAG=$(curl -sf \
              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?per_page=2" \
              | grep -oP '"name":"\K[^"]+' | sed -n '2p')

            if [ -n "$PREV_TAG" ]; then
              COMMITS=$(curl -sf \
                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
                "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/compare?from=${PREV_TAG}&to=${TAG}" \
                | grep -oP '"title":"\K[^"]+' | head -20 \
                | sed 's/^/- /' || echo "- Release ${TAG}")
              DESCRIPTION="## Changes since ${PREV_TAG}\n\n${COMMITS}"
            else
              DESCRIPTION="## Initial release\n\nFirst CalVer release."
            fi

            # Create the release
            RESPONSE=$(curl -sf -X POST \
              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
              -H "Content-Type: application/json" \
              -d "{
                \"tag_name\": \"${TAG}\",
                \"name\": \"${TAG}\",
                \"description\": \"$(printf '%s' "$DESCRIPTION" | sed 's/"/\\"/g')\"
              }" \
              "${GITLAB}/api/v4/projects/${PROJECT_ID}/releases") || {
              echo "WARN: Release creation failed (may already exist)"
            }

            echo "Release ${TAG} created"
            echo "$RESPONSE" | head -5

            # Add package links as release assets
            PKG_URL="${GITLAB}/api/v4/projects/${PROJECT_ID}/packages/generic"
            for pkg_name in foxhunt-services foxhunt-training; do
              # Check if package exists for this tag
              PKG_CHECK=$(curl -sf \
                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
                "${GITLAB}/api/v4/projects/${PROJECT_ID}/packages?package_name=${pkg_name}&package_version=${TAG}" \
                || echo "[]")

              if echo "$PKG_CHECK" | grep -q "$TAG"; then
                curl -sf -X POST \
                  -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
                  -H "Content-Type: application/json" \
                  -d "{
                    \"name\": \"${pkg_name}\",
                    \"url\": \"${GITLAB}/-/packages?type=generic&search=${pkg_name}&version=${TAG}\",
                    \"link_type\": \"package\"
                  }" \
                  "${GITLAB}/api/v4/projects/${PROJECT_ID}/releases/${TAG}/assets/links" || true
                echo "Linked ${pkg_name} package to release"
              fi
            done

            echo "=== Release ${TAG} complete ==="

Step 2: Verify YAML

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"

---

Task 7: Modify deploy-services — patch FOXHUNT_RELEASE on deployments

Files:

• Modify: infra/k8s/argo/ci-pipeline-template.yaml (deploy-services template, lines ~582-620)

Step 1: Replace deploy-services script

Replace the entire args block of deploy-services with:

set -e
if ! command -v kubectl >/dev/null 2>&1; then
  echo "=== Installing kubectl ==="
  curl -sLo /tmp/kubectl "https://dl.k8s.io/release/v1.31.4/bin/linux/amd64/kubectl"
  chmod +x /tmp/kubectl
  export PATH="/tmp:$PATH"
fi

TAG="{{tasks.create-tag.outputs.parameters.tag}}"
echo "=== Deploying release ${TAG} ==="

SERVICES="api trading-service ml-training-service backtesting-service trading-agent-service broker-gateway data-acquisition-service"
for svc in $SERVICES; do
  echo "Patching $svc with FOXHUNT_RELEASE=${TAG}..."
  kubectl -n foxhunt patch deployment "$svc" -p "{
    \"spec\":{\"template\":{
      \"metadata\":{\"annotations\":{\"foxhunt.io/release\":\"${TAG}\"}},
      \"spec\":{\"initContainers\":[{
        \"name\":\"fetch-binary\",
        \"env\":[{\"name\":\"FOXHUNT_RELEASE\",\"value\":\"${TAG}\"}]
      }]}
    }}
  }" || echo "WARN: $svc patch failed (may not exist)"
done

echo "=== Waiting for rollouts ==="
for svc in $SERVICES; do
  kubectl -n foxhunt rollout status deployment "$svc" --timeout=120s || echo "WARN: $svc rollout timeout"
done

echo "=== Deploy ${TAG} complete ==="

Step 2: Verify YAML

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"

---

Task 8: Update DAG dependencies

Files:

• Modify: infra/k8s/argo/ci-pipeline-template.yaml (DAG tasks, lines ~44-136)

Step 1: Replace the DAG tasks section

Replace the entire dag.tasks list with:

        tasks:
          - name: detect-changes
            template: detect-changes

          # Create CalVer tag when code changed
          - name: create-tag
            dependencies: [detect-changes]
            template: create-tag
            when: "{{tasks.detect-changes.outputs.parameters.needs-code}} == true"

          # Parallel: each compile step is self-contained (own git clone)
          # sccache backed by MinIO — no PVC needed, shared across all nodes
          - name: compile-services
            dependencies: [create-tag]
            template: compile-services
            when: "{{tasks.detect-changes.outputs.parameters.needs-services}} == true"

          - name: compile-training
            dependencies: [create-tag]
            template: compile-training
            when: "{{tasks.detect-changes.outputs.parameters.needs-training}} == true"

          # GPU warmup: trigger GPU node autoscale during compile so the node
          # is ready when we submit a training workflow after CI completes.
          - name: gpu-warmup
            dependencies: [detect-changes]
            template: gpu-warmup
            when: "{{tasks.detect-changes.outputs.parameters.needs-training}} == true"

          - name: build-web-dashboard
            dependencies: [detect-changes]
            template: build-web-dashboard
            when: "{{tasks.detect-changes.outputs.parameters.needs-dashboard}} == true"

          # Create GitLab Release after all binaries are uploaded
          - name: upload-release
            dependencies: [compile-services, compile-training]
            template: upload-release
            when: "{{tasks.detect-changes.outputs.parameters.needs-code}} == true"

          # Deploy after release is created
          - name: deploy-services
            dependencies: [upload-release]
            template: deploy-services
            when: "{{tasks.detect-changes.outputs.parameters.needs-services}} == true"

          - name: rebuild-ci-builder
            dependencies: [detect-changes]
            templateRef:
              name: build-ci-image
              template: build
            arguments:
              parameters:
                - name: commit-sha
                  value: "{{workflow.parameters.commit-sha}}"
                - name: dockerfile
                  value: Dockerfile.ci-builder
                - name: image-name
                  value: ci-builder
            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"

          - name: rebuild-ci-builder-cpu
            dependencies: [detect-changes]
            templateRef:
              name: build-ci-image
              template: build
            arguments:
              parameters:
                - name: commit-sha
                  value: "{{workflow.parameters.commit-sha}}"
                - name: dockerfile
                  value: Dockerfile.ci-builder-cpu
                - name: image-name
                  value: ci-builder-cpu
            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"

          - name: rebuild-runtime
            dependencies: [detect-changes]
            templateRef:
              name: build-ci-image
              template: build
            arguments:
              parameters:
                - name: commit-sha
                  value: "{{workflow.parameters.commit-sha}}"
                - name: dockerfile
                  value: Dockerfile.foxhunt-runtime
                - name: image-name
                  value: foxhunt-runtime
            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"

          - name: rebuild-training-runtime
            dependencies: [detect-changes]
            templateRef:
              name: build-ci-image
              template: build
            arguments:
              parameters:
                - name: commit-sha
                  value: "{{workflow.parameters.commit-sha}}"
                - name: dockerfile
                  value: Dockerfile.foxhunt-training-runtime
                - name: image-name
                  value: foxhunt-training-runtime
            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"

Step 2: Verify the full pipeline YAML

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"

---

Task 9: Update service initContainers — fetch from GitLab

Files (7 service deployments):

• Modify: infra/k8s/services/api.yaml:41-78
• Modify: infra/k8s/services/trading-service.yaml:41-77
• Modify: infra/k8s/services/ml-training-service.yaml:83-120 (offset by SA/RBAC)
• Modify: infra/k8s/services/backtesting-service.yaml (same initContainer pattern)
• Modify: infra/k8s/services/broker-gateway.yaml (same initContainer pattern)
• Modify: infra/k8s/services/data-acquisition-service.yaml (same initContainer pattern)
• Modify: infra/k8s/services/trading-agent-service.yaml (same initContainer pattern)

Each file has the same initContainer pattern. Replace the fetch-binary initContainer in all 7 files.

Step 1: Replace initContainer in each service

For each service, replace the fetch-binary initContainer block. The template (substitute SERVICE_NAME for each):

        - name: fetch-binary
          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              BINARY="SERVICE_NAME"
              curl -fSL -o "/binaries/${BINARY}" \
                --header "DEPLOY-TOKEN: ${GITLAB_DEPLOY_TOKEN}" \
                "${GITLAB_API}/projects/1/packages/generic/foxhunt-services/${FOXHUNT_RELEASE}/${BINARY}"
              chmod +x "/binaries/${BINARY}"
              echo "Fetched ${BINARY} ${FOXHUNT_RELEASE} ($(stat -c%s /binaries/${BINARY}) bytes)"
          env:
            - name: GITLAB_DEPLOY_TOKEN
              valueFrom:
                secretKeyRef:
                  name: gitlab-deploy-token
                  key: token
            - name: GITLAB_API
              value: "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181/api/v4"
            - name: FOXHUNT_RELEASE
              value: "latest"
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi

Substitutions per file:

• api.yaml: SERVICE_NAME = api
• trading-service.yaml: SERVICE_NAME = trading-service
• ml-training-service.yaml: SERVICE_NAME = ml-training-service
• backtesting-service.yaml: SERVICE_NAME = backtesting-service
• trading-agent-service.yaml: SERVICE_NAME = trading-agent-service
• broker-gateway.yaml: SERVICE_NAME = broker-gateway
• data-acquisition-service.yaml: SERVICE_NAME = data-acquisition-service

Note: The FOXHUNT_RELEASE value "latest" is a placeholder — the deploy-services CI step patches it to the actual tag via kubectl patch.

Step 2: Verify all 7 service YAMLs

for f in infra/k8s/services/{api,trading-service,ml-training-service,backtesting-service,trading-agent-service,broker-gateway,data-acquisition-service}.yaml; do
  python3 -c "import yaml; list(yaml.safe_load_all(open('$f')))" && echo "$f OK" || echo "$f FAILED"
done

Expected: all OK

---

Task 10: Update training job-template — binary fetch from GitLab

Files:

• Modify: infra/k8s/training/job-template.yaml:38-78 (fetch-binary initContainer only; sync-training-data stays MinIO)

Step 1: Replace fetch-binary initContainer

Replace the first initContainer (fetch-binary, lines 39-78) with:

        - name: fetch-binary
          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              BINARY="$(TRAINING_BINARY)"
              curl -fSL -o "/binaries/${BINARY}" \
                --header "DEPLOY-TOKEN: ${GITLAB_DEPLOY_TOKEN}" \
                "${GITLAB_API}/projects/1/packages/generic/foxhunt-training/${FOXHUNT_RELEASE}/${BINARY}"
              chmod +x "/binaries/${BINARY}"
              echo "Fetched ${BINARY} ${FOXHUNT_RELEASE} ($(stat -c%s /binaries/${BINARY}) bytes)"
          env:
            - name: TRAINING_BINARY
              value: train_baseline_supervised
            - name: GITLAB_DEPLOY_TOKEN
              valueFrom:
                secretKeyRef:
                  name: gitlab-deploy-token
                  key: token
            - name: GITLAB_API
              value: "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181/api/v4"
            - name: FOXHUNT_RELEASE
              value: "latest"
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi

Note: sync-training-data initContainer stays unchanged (MinIO). Only binary fetch moves to GitLab.

Step 2: Verify YAML

python3 -c "import yaml; yaml.safe_load(open('infra/k8s/training/job-template.yaml'))"

---

Task 11: Commit and apply to cluster

Step 1: Commit all changes

git add infra/k8s/argo/ci-pipeline-template.yaml \
  infra/k8s/services/api.yaml \
  infra/k8s/services/trading-service.yaml \
  infra/k8s/services/ml-training-service.yaml \
  infra/k8s/services/backtesting-service.yaml \
  infra/k8s/services/trading-agent-service.yaml \
  infra/k8s/services/broker-gateway.yaml \
  infra/k8s/services/data-acquisition-service.yaml \
  infra/k8s/training/job-template.yaml

git commit -m "feat(infra): GitLab releases with CalVer auto-versioning

Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.

- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints"

Step 2: Apply CI pipeline template

kubectl apply -f infra/k8s/argo/ci-pipeline-template.yaml

Step 3: Apply service deployments (will NOT restart — just updates spec)

kubectl apply -f infra/k8s/services/api.yaml
kubectl apply -f infra/k8s/services/trading-service.yaml
kubectl apply -f infra/k8s/services/ml-training-service.yaml
kubectl apply -f infra/k8s/services/backtesting-service.yaml
kubectl apply -f infra/k8s/services/trading-agent-service.yaml
kubectl apply -f infra/k8s/services/broker-gateway.yaml
kubectl apply -f infra/k8s/services/data-acquisition-service.yaml

Note: Don't apply job-template.yaml — it's a template that is applied per-job, not a persistent resource.

Step 4: Push to trigger first release

git push origin main

Step 5: Monitor the Argo workflow

# Watch the workflow
kubectl -n foxhunt get workflows -w

# Check tag was created
kubectl -n foxhunt logs -l workflows.argoproj.io/workflow -c main --tail=50 | grep "Creating tag"

Expected: workflow creates tag v2026.03.1, compiles, uploads to GitLab packages, creates release, patches deployments.

Step 6: Verify services restarted with new binary

kubectl -n foxhunt get pods -o wide
kubectl -n foxhunt logs deployment/api -c fetch-binary

Expected: initContainer logs show Fetched api v2026.03.1 (... bytes)

---

Task 12: Cleanup — remove MinIO binary bucket

Only do this after Task 11 is verified working.

Step 1: Verify GitLab packages exist

GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
PAT=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)

curl -sf -H "PRIVATE-TOKEN: ${PAT}" \
  "${GITLAB}/api/v4/projects/1/packages?package_type=generic" | python3 -m json.tool

Expected: lists foxhunt-services and foxhunt-training packages with the tag version.

Step 2: Remove binary bucket contents from MinIO

kubectl -n foxhunt exec deployment/minio -- mc rm --recursive --force /data/foxhunt-binaries/services/
kubectl -n foxhunt exec deployment/minio -- mc rm --recursive --force /data/foxhunt-binaries/training/

Keep the bucket itself (other paths may reference it).

Step 3: Commit cleanup note

No code changes needed — the MinIO upload code was already removed in Task 4/5.