# GitLab Releases with CalVer Auto-Versioning — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Replace MinIO binary distribution with GitLab Generic Package Registry + Releases, with CalVer auto-tagging on every code push to main.

**Architecture:** Argo CI pipeline creates a CalVer git tag, compiles binaries, uploads them to GitLab's Generic Package Registry, creates a GitLab Release with auto-generated notes, then patches service deployments to fetch from GitLab. MinIO retains sccache, training data, and model checkpoints.

**Tech Stack:** Argo Workflows, GitLab CE API v4, Generic Package Registry, K8s strategic merge patches, curl

---

## Context

**GitLab internal URL:** `http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181`
**GitLab project ID:** `1` (root/foxhunt)
**Existing secrets:** `gitlab-pat` (auto-rotated PAT, monthly), `minio-credentials`
**Existing file:** `crates/common/src/build_info.rs` — already captures `FOXHUNT_BUILD_VERSION` at compile time
**Web dashboard:** stays on MinIO (not code, not part of release)

**Services affected (7):** api, trading-service, ml-training-service, backtesting-service, trading-agent-service, broker-gateway, data-acquisition-service

**Binary names from cargo build:**
- Services: `api`, `trading-service`, `ml-training-service`, `backtesting-service`, `trading-agent-service`, `broker-gateway`, `data-acquisition-service`
- Training: `train_baseline_rl`, `train_baseline_supervised`, `evaluate_baseline`, `evaluate_supervised`, `hyperopt_baseline_rl`, `hyperopt_baseline_supervised`, `training_uploader`

---

### Task 1: Create GitLab deploy token and K8s secret

This is a one-time manual setup. The deploy token lets service pods download packages from GitLab's Generic Package Registry (read-only).

**Step 1: Create deploy token via GitLab API**

Run from a machine with access to the cluster:

```bash
# Use existing PAT to create a project deploy token
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
PAT=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)

curl -sf -X POST "${GITLAB}/api/v4/projects/1/deploy_tokens" \
  -H "PRIVATE-TOKEN: ${PAT}" \
  -d "name=foxhunt-package-reader" \
  -d "scopes[]=read_package_registry" | jq .
```

Save the returned `token` value.

**Step 2: Create K8s secret**

```bash
kubectl -n foxhunt create secret generic gitlab-deploy-token \
  --from-literal=token=<token-from-step-1>
```

**Step 3: Verify token can read packages**

```bash
# Should return 200 (empty list is fine — no packages yet)
curl -sf -o /dev/null -w "%{http_code}" \
  -H "DEPLOY-TOKEN: <token>" \
  "${GITLAB}/api/v4/projects/1/packages"
```

Expected: `200`

**Step 4: Verify existing gitlab-pat has api scope**

```bash
curl -sf "${GITLAB}/api/v4/personal_access_tokens/self" \
  -H "PRIVATE-TOKEN: ${PAT}" | jq '.scopes'
```

Expected: scopes must include `api` (for creating tags, releases, uploading packages). If not, recreate the PAT with `api` scope.

---

### Task 2: Add `needs-code` output to detect-changes

**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml:191-220` (detect-changes script outputs)

**Step 1: Add needs-code computation to detect-changes script**

In the detect-changes shell script, after the `NEEDS_TRAINING` computation (line ~207) and before the `echo "=== Build decisions ==="` line, add:

```sh
if [ "$NEEDS_SERVICES" = "true" ] || [ "$NEEDS_TRAINING" = "true" ]; then
  NEEDS_CODE="true"
else
  NEEDS_CODE="false"
fi
```

And after `echo -n "$DOCKER_IMAGES" > /tmp/outputs/docker-images`, add:

```sh
echo -n "$NEEDS_CODE" > /tmp/outputs/needs-code
```

Also add the build decisions echo:

```sh
echo "needs-code: $NEEDS_CODE"
```

**Step 2: Add needs-code output parameter**

After the `docker-images` output parameter (line ~235), add:

```yaml
          - name: needs-code
            valueFrom:
              path: /tmp/outputs/needs-code
```

**Step 3: Verify YAML is valid**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```

Expected: no error

---

### Task 3: New `create-tag` template

**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (add template after detect-changes, before compile-services)

**Step 1: Add create-tag template**

Insert after the detect-changes template `outputs:` block (after line ~237) and before the `build-web-dashboard` template:

```yaml
    # ── create-tag: CalVer auto-tag on code changes ──
    - name: create-tag
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      container:
        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        env:
          - name: GITLAB_PAT
            valueFrom:
              secretKeyRef:
                name: gitlab-pat
                key: token
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi
        args:
          - |
            set -e
            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
            PROJECT_ID=1
            SHA="{{workflow.parameters.commit-sha}}"

            # Compute CalVer prefix: vYYYY.MM
            PREFIX="v$(date +%Y.%m)"

            # Query existing tags for this month
            TAGS=$(curl -sf \
              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?search=${PREFIX}" \
              || echo "[]")

            # Parse highest N from vYYYY.MM.N tags
            LAST_N=$(echo "$TAGS" | grep -oP "\"name\":\"${PREFIX}\.\K[0-9]+" | sort -n | tail -1)
            if [ -z "$LAST_N" ]; then
              NEXT_N=1
            else
              NEXT_N=$((LAST_N + 1))
            fi

            TAG="${PREFIX}.${NEXT_N}"
            echo "Creating tag: ${TAG} at ${SHA}"

            # Create the tag
            RESULT=$(curl -sf -X POST \
              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
              -d "tag_name=${TAG}" \
              -d "ref=${SHA}" \
              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags")

            echo "$RESULT" | grep -q "$TAG" || { echo "Tag creation failed: $RESULT"; exit 1; }
            echo "Tag ${TAG} created successfully"

            mkdir -p /tmp/outputs
            echo -n "$TAG" > /tmp/outputs/tag
      outputs:
        parameters:
          - name: tag
            valueFrom:
              path: /tmp/outputs/tag
```

**Step 2: Add gitlab-pat volume to workflow spec**

In the `spec.volumes` list (around line 22-32), add:

```yaml
    - name: gitlab-pat
      secret:
        secretName: gitlab-pat
```

**Step 3: Verify YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```

Expected: no error

---

### Task 4: Modify compile-services — version from tag + upload to GitLab packages

**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (compile-services template, lines ~309-426)

**Step 1: Replace version computation**

Find and replace the 3 lines (around line 393-395):

```sh
YEAR=$(date +%Y)
MONTH=$(date +%m)
export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.argo"
```

With:

```sh
export FOXHUNT_BUILD_VERSION="{{tasks.create-tag.outputs.parameters.tag}}"
```

**Step 2: Replace MinIO upload with GitLab package upload**

Find and replace the rclone upload block (around lines 418-424):

```sh
echo "=== Uploading service binaries ==="
rclone copy "$WORKSPACE/bin/services/" :s3:foxhunt-binaries/services/ \
  --s3-provider=Minio \
  --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
  --s3-access-key-id="${MINIO_ACCESS_KEY}" \
  --s3-secret-access-key="${MINIO_SECRET_KEY}" \
  --s3-no-check-bucket
```

With:

```sh
echo "=== Uploading service binaries to GitLab packages ==="
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
TAG="${FOXHUNT_BUILD_VERSION}"
for bin in "$WORKSPACE/bin/services/"*; do
  BIN_NAME=$(basename "$bin")
  echo "Uploading ${BIN_NAME}..."
  curl -f --upload-file "$bin" \
    -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
    "${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-services/${TAG}/${BIN_NAME}"
done
```

**Step 3: Add GITLAB_PAT env var, remove MINIO upload env vars**

In the compile-services container `env:` list, add:

```yaml
          - name: GITLAB_PAT
            valueFrom:
              secretKeyRef:
                name: gitlab-pat
                key: token
```

Remove the `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` env entries (keep `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — they're needed by sccache).

**Step 4: Verify YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```

---

### Task 5: Modify compile-training — version from tag + upload to GitLab packages

**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (compile-training template, lines ~430-547)

**Step 1: Replace version computation**

Find and replace (around line 513-514):

```sh
YEAR=$(date +%Y)
MONTH=$(date +%m)
export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.argo"
```

With:

```sh
export FOXHUNT_BUILD_VERSION="{{tasks.create-tag.outputs.parameters.tag}}"
```

**Step 2: Replace MinIO upload with GitLab package upload**

Find and replace the rclone upload block (around lines 539-545):

```sh
echo "=== Uploading training binaries ==="
rclone copy "$WORKSPACE/bin/training/" :s3:foxhunt-binaries/training/ \
  --s3-provider=Minio \
  --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
  --s3-access-key-id="${MINIO_ACCESS_KEY}" \
  --s3-secret-access-key="${MINIO_SECRET_KEY}" \
  --s3-no-check-bucket
```

With:

```sh
echo "=== Uploading training binaries to GitLab packages ==="
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
TAG="${FOXHUNT_BUILD_VERSION}"
for bin in "$WORKSPACE/bin/training/"*; do
  BIN_NAME=$(basename "$bin")
  echo "Uploading ${BIN_NAME}..."
  curl -f --upload-file "$bin" \
    -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
    "${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-training/${TAG}/${BIN_NAME}"
done
```

**Step 3: Add GITLAB_PAT env var, remove MINIO upload env vars**

Same as Task 4 Step 3. Add `GITLAB_PAT` from `gitlab-pat` secret. Remove `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` (keep `AWS_*` for sccache).

**Step 4: Verify YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```

---

### Task 6: New `upload-release` template

**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (add template after compile-training, before gpu-warmup)

**Step 1: Add upload-release template**

```yaml
    # ── upload-release: create GitLab Release with package links ──
    - name: upload-release
      nodeSelector:
        k8s.scaleway.com/pool-name: platform
      container:
        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
        command: ["/bin/sh", "-c"]
        env:
          - name: GITLAB_PAT
            valueFrom:
              secretKeyRef:
                name: gitlab-pat
                key: token
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi
        args:
          - |
            set -e
            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
            PROJECT_ID=1
            TAG="{{tasks.create-tag.outputs.parameters.tag}}"

            echo "=== Creating GitLab Release ${TAG} ==="

            # Get commits since previous tag for release notes
            PREV_TAG=$(curl -sf \
              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?per_page=2" \
              | grep -oP '"name":"\K[^"]+' | sed -n '2p')

            if [ -n "$PREV_TAG" ]; then
              COMMITS=$(curl -sf \
                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
                "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/compare?from=${PREV_TAG}&to=${TAG}" \
                | grep -oP '"title":"\K[^"]+' | head -20 \
                | sed 's/^/- /' || echo "- Release ${TAG}")
              DESCRIPTION="## Changes since ${PREV_TAG}\n\n${COMMITS}"
            else
              DESCRIPTION="## Initial release\n\nFirst CalVer release."
            fi

            # Create the release
            RESPONSE=$(curl -sf -X POST \
              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
              -H "Content-Type: application/json" \
              -d "{
                \"tag_name\": \"${TAG}\",
                \"name\": \"${TAG}\",
                \"description\": \"$(printf '%s' "$DESCRIPTION" | sed 's/"/\\"/g')\"
              }" \
              "${GITLAB}/api/v4/projects/${PROJECT_ID}/releases") || {
              echo "WARN: Release creation failed (may already exist)"
            }

            echo "Release ${TAG} created"
            echo "$RESPONSE" | head -5

            # Add package links as release assets
            PKG_URL="${GITLAB}/api/v4/projects/${PROJECT_ID}/packages/generic"
            for pkg_name in foxhunt-services foxhunt-training; do
              # Check if package exists for this tag
              PKG_CHECK=$(curl -sf \
                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
                "${GITLAB}/api/v4/projects/${PROJECT_ID}/packages?package_name=${pkg_name}&package_version=${TAG}" \
                || echo "[]")

              if echo "$PKG_CHECK" | grep -q "$TAG"; then
                curl -sf -X POST \
                  -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
                  -H "Content-Type: application/json" \
                  -d "{
                    \"name\": \"${pkg_name}\",
                    \"url\": \"${GITLAB}/-/packages?type=generic&search=${pkg_name}&version=${TAG}\",
                    \"link_type\": \"package\"
                  }" \
                  "${GITLAB}/api/v4/projects/${PROJECT_ID}/releases/${TAG}/assets/links" || true
                echo "Linked ${pkg_name} package to release"
              fi
            done

            echo "=== Release ${TAG} complete ==="
```

**Step 2: Verify YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```

---

### Task 7: Modify deploy-services — patch FOXHUNT_RELEASE on deployments

**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (deploy-services template, lines ~582-620)

**Step 1: Replace deploy-services script**

Replace the entire `args` block of deploy-services with:

```sh
set -e
if ! command -v kubectl >/dev/null 2>&1; then
  echo "=== Installing kubectl ==="
  curl -sLo /tmp/kubectl "https://dl.k8s.io/release/v1.31.4/bin/linux/amd64/kubectl"
  chmod +x /tmp/kubectl
  export PATH="/tmp:$PATH"
fi

TAG="{{tasks.create-tag.outputs.parameters.tag}}"
echo "=== Deploying release ${TAG} ==="

SERVICES="api trading-service ml-training-service backtesting-service trading-agent-service broker-gateway data-acquisition-service"
for svc in $SERVICES; do
  echo "Patching $svc with FOXHUNT_RELEASE=${TAG}..."
  kubectl -n foxhunt patch deployment "$svc" -p "{
    \"spec\":{\"template\":{
      \"metadata\":{\"annotations\":{\"foxhunt.io/release\":\"${TAG}\"}},
      \"spec\":{\"initContainers\":[{
        \"name\":\"fetch-binary\",
        \"env\":[{\"name\":\"FOXHUNT_RELEASE\",\"value\":\"${TAG}\"}]
      }]}
    }}
  }" || echo "WARN: $svc patch failed (may not exist)"
done

echo "=== Waiting for rollouts ==="
for svc in $SERVICES; do
  kubectl -n foxhunt rollout status deployment "$svc" --timeout=120s || echo "WARN: $svc rollout timeout"
done

echo "=== Deploy ${TAG} complete ==="
```

**Step 2: Verify YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```

---

### Task 8: Update DAG dependencies

**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (DAG tasks, lines ~44-136)

**Step 1: Replace the DAG tasks section**

Replace the entire `dag.tasks` list with:

```yaml
        tasks:
          - name: detect-changes
            template: detect-changes

          # Create CalVer tag when code changed
          - name: create-tag
            dependencies: [detect-changes]
            template: create-tag
            when: "{{tasks.detect-changes.outputs.parameters.needs-code}} == true"

          # Parallel: each compile step is self-contained (own git clone)
          # sccache backed by MinIO — no PVC needed, shared across all nodes
          - name: compile-services
            dependencies: [create-tag]
            template: compile-services
            when: "{{tasks.detect-changes.outputs.parameters.needs-services}} == true"

          - name: compile-training
            dependencies: [create-tag]
            template: compile-training
            when: "{{tasks.detect-changes.outputs.parameters.needs-training}} == true"

          # GPU warmup: trigger GPU node autoscale during compile so the node
          # is ready when we submit a training workflow after CI completes.
          - name: gpu-warmup
            dependencies: [detect-changes]
            template: gpu-warmup
            when: "{{tasks.detect-changes.outputs.parameters.needs-training}} == true"

          - name: build-web-dashboard
            dependencies: [detect-changes]
            template: build-web-dashboard
            when: "{{tasks.detect-changes.outputs.parameters.needs-dashboard}} == true"

          # Create GitLab Release after all binaries are uploaded
          - name: upload-release
            dependencies: [compile-services, compile-training]
            template: upload-release
            when: "{{tasks.detect-changes.outputs.parameters.needs-code}} == true"

          # Deploy after release is created
          - name: deploy-services
            dependencies: [upload-release]
            template: deploy-services
            when: "{{tasks.detect-changes.outputs.parameters.needs-services}} == true"

          - name: rebuild-ci-builder
            dependencies: [detect-changes]
            templateRef:
              name: build-ci-image
              template: build
            arguments:
              parameters:
                - name: commit-sha
                  value: "{{workflow.parameters.commit-sha}}"
                - name: dockerfile
                  value: Dockerfile.ci-builder
                - name: image-name
                  value: ci-builder
            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"

          - name: rebuild-ci-builder-cpu
            dependencies: [detect-changes]
            templateRef:
              name: build-ci-image
              template: build
            arguments:
              parameters:
                - name: commit-sha
                  value: "{{workflow.parameters.commit-sha}}"
                - name: dockerfile
                  value: Dockerfile.ci-builder-cpu
                - name: image-name
                  value: ci-builder-cpu
            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"

          - name: rebuild-runtime
            dependencies: [detect-changes]
            templateRef:
              name: build-ci-image
              template: build
            arguments:
              parameters:
                - name: commit-sha
                  value: "{{workflow.parameters.commit-sha}}"
                - name: dockerfile
                  value: Dockerfile.foxhunt-runtime
                - name: image-name
                  value: foxhunt-runtime
            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"

          - name: rebuild-training-runtime
            dependencies: [detect-changes]
            templateRef:
              name: build-ci-image
              template: build
            arguments:
              parameters:
                - name: commit-sha
                  value: "{{workflow.parameters.commit-sha}}"
                - name: dockerfile
                  value: Dockerfile.foxhunt-training-runtime
                - name: image-name
                  value: foxhunt-training-runtime
            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
```

**Step 2: Verify the full pipeline YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```

---

### Task 9: Update service initContainers — fetch from GitLab

**Files (7 service deployments):**
- Modify: `infra/k8s/services/api.yaml:41-78`
- Modify: `infra/k8s/services/trading-service.yaml:41-77`
- Modify: `infra/k8s/services/ml-training-service.yaml:83-120` (offset by SA/RBAC)
- Modify: `infra/k8s/services/backtesting-service.yaml` (same initContainer pattern)
- Modify: `infra/k8s/services/broker-gateway.yaml` (same initContainer pattern)
- Modify: `infra/k8s/services/data-acquisition-service.yaml` (same initContainer pattern)
- Modify: `infra/k8s/services/trading-agent-service.yaml` (same initContainer pattern)

Each file has the same initContainer pattern. Replace the `fetch-binary` initContainer in all 7 files.

**Step 1: Replace initContainer in each service**

For each service, replace the `fetch-binary` initContainer block. The template (substitute `SERVICE_NAME` for each):

```yaml
        - name: fetch-binary
          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              BINARY="SERVICE_NAME"
              curl -fSL -o "/binaries/${BINARY}" \
                --header "DEPLOY-TOKEN: ${GITLAB_DEPLOY_TOKEN}" \
                "${GITLAB_API}/projects/1/packages/generic/foxhunt-services/${FOXHUNT_RELEASE}/${BINARY}"
              chmod +x "/binaries/${BINARY}"
              echo "Fetched ${BINARY} ${FOXHUNT_RELEASE} ($(stat -c%s /binaries/${BINARY}) bytes)"
          env:
            - name: GITLAB_DEPLOY_TOKEN
              valueFrom:
                secretKeyRef:
                  name: gitlab-deploy-token
                  key: token
            - name: GITLAB_API
              value: "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181/api/v4"
            - name: FOXHUNT_RELEASE
              value: "latest"
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
```

Substitutions per file:
- `api.yaml`: `SERVICE_NAME` = `api`
- `trading-service.yaml`: `SERVICE_NAME` = `trading-service`
- `ml-training-service.yaml`: `SERVICE_NAME` = `ml-training-service`
- `backtesting-service.yaml`: `SERVICE_NAME` = `backtesting-service`
- `trading-agent-service.yaml`: `SERVICE_NAME` = `trading-agent-service`
- `broker-gateway.yaml`: `SERVICE_NAME` = `broker-gateway`
- `data-acquisition-service.yaml`: `SERVICE_NAME` = `data-acquisition-service`

Note: The `FOXHUNT_RELEASE` value `"latest"` is a placeholder — the deploy-services CI step patches it to the actual tag via `kubectl patch`.

**Step 2: Verify all 7 service YAMLs**

```bash
for f in infra/k8s/services/{api,trading-service,ml-training-service,backtesting-service,trading-agent-service,broker-gateway,data-acquisition-service}.yaml; do
  python3 -c "import yaml; list(yaml.safe_load_all(open('$f')))" && echo "$f OK" || echo "$f FAILED"
done
```

Expected: all OK

---

### Task 10: Update training job-template — binary fetch from GitLab

**Files:**
- Modify: `infra/k8s/training/job-template.yaml:38-78` (fetch-binary initContainer only; sync-training-data stays MinIO)

**Step 1: Replace fetch-binary initContainer**

Replace the first initContainer (`fetch-binary`, lines 39-78) with:

```yaml
        - name: fetch-binary
          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              BINARY="$(TRAINING_BINARY)"
              curl -fSL -o "/binaries/${BINARY}" \
                --header "DEPLOY-TOKEN: ${GITLAB_DEPLOY_TOKEN}" \
                "${GITLAB_API}/projects/1/packages/generic/foxhunt-training/${FOXHUNT_RELEASE}/${BINARY}"
              chmod +x "/binaries/${BINARY}"
              echo "Fetched ${BINARY} ${FOXHUNT_RELEASE} ($(stat -c%s /binaries/${BINARY}) bytes)"
          env:
            - name: TRAINING_BINARY
              value: train_baseline_supervised
            - name: GITLAB_DEPLOY_TOKEN
              valueFrom:
                secretKeyRef:
                  name: gitlab-deploy-token
                  key: token
            - name: GITLAB_API
              value: "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181/api/v4"
            - name: FOXHUNT_RELEASE
              value: "latest"
          volumeMounts:
            - name: binaries
              mountPath: /binaries
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
```

Note: `sync-training-data` initContainer stays unchanged (MinIO). Only binary fetch moves to GitLab.

**Step 2: Verify YAML**

```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/training/job-template.yaml'))"
```

---

### Task 11: Commit and apply to cluster

**Step 1: Commit all changes**

```bash
git add infra/k8s/argo/ci-pipeline-template.yaml \
  infra/k8s/services/api.yaml \
  infra/k8s/services/trading-service.yaml \
  infra/k8s/services/ml-training-service.yaml \
  infra/k8s/services/backtesting-service.yaml \
  infra/k8s/services/trading-agent-service.yaml \
  infra/k8s/services/broker-gateway.yaml \
  infra/k8s/services/data-acquisition-service.yaml \
  infra/k8s/training/job-template.yaml

git commit -m "feat(infra): GitLab releases with CalVer auto-versioning

Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.

- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints"
```

**Step 2: Apply CI pipeline template**

```bash
kubectl apply -f infra/k8s/argo/ci-pipeline-template.yaml
```

**Step 3: Apply service deployments (will NOT restart — just updates spec)**

```bash
kubectl apply -f infra/k8s/services/api.yaml
kubectl apply -f infra/k8s/services/trading-service.yaml
kubectl apply -f infra/k8s/services/ml-training-service.yaml
kubectl apply -f infra/k8s/services/backtesting-service.yaml
kubectl apply -f infra/k8s/services/trading-agent-service.yaml
kubectl apply -f infra/k8s/services/broker-gateway.yaml
kubectl apply -f infra/k8s/services/data-acquisition-service.yaml
```

Note: Don't apply job-template.yaml — it's a template that is applied per-job, not a persistent resource.

**Step 4: Push to trigger first release**

```bash
git push origin main
```

**Step 5: Monitor the Argo workflow**

```bash
# Watch the workflow
kubectl -n foxhunt get workflows -w

# Check tag was created
kubectl -n foxhunt logs -l workflows.argoproj.io/workflow -c main --tail=50 | grep "Creating tag"
```

Expected: workflow creates tag `v2026.03.1`, compiles, uploads to GitLab packages, creates release, patches deployments.

**Step 6: Verify services restarted with new binary**

```bash
kubectl -n foxhunt get pods -o wide
kubectl -n foxhunt logs deployment/api -c fetch-binary
```

Expected: initContainer logs show `Fetched api v2026.03.1 (... bytes)`

---

### Task 12: Cleanup — remove MinIO binary bucket

Only do this after Task 11 is verified working.

**Step 1: Verify GitLab packages exist**

```bash
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
PAT=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)

curl -sf -H "PRIVATE-TOKEN: ${PAT}" \
  "${GITLAB}/api/v4/projects/1/packages?package_type=generic" | python3 -m json.tool
```

Expected: lists foxhunt-services and foxhunt-training packages with the tag version.

**Step 2: Remove binary bucket contents from MinIO**

```bash
kubectl -n foxhunt exec deployment/minio -- mc rm --recursive --force /data/foxhunt-binaries/services/
kubectl -n foxhunt exec deployment/minio -- mc rm --recursive --force /data/foxhunt-binaries/training/
```

Keep the bucket itself (other paths may reference it).

**Step 3: Commit cleanup note**

No code changes needed — the MinIO upload code was already removed in Task 4/5.