gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
ae704a7b7346da17039e4aeef8265f65c6df40ee
foxhunt/docs/plans/2026-03-07-gitlab-releases-plan.md
jgrusewski 7a9683d5fb feat(infra): GitLab releases with CalVer auto-versioning 
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.

- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 20:38:41 +01:00

876 lines
29 KiB
Markdown
Raw Blame History

# GitLab Releases with CalVer Auto-Versioning — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Replace MinIO binary distribution with GitLab Generic Package Registry + Releases, with CalVer auto-tagging on every code push to main.
**Architecture:** Argo CI pipeline creates a CalVer git tag, compiles binaries, uploads them to GitLab's Generic Package Registry, creates a GitLab Release with auto-generated notes, then patches service deployments to fetch from GitLab. MinIO retains sccache, training data, and model checkpoints.
**Tech Stack:** Argo Workflows, GitLab CE API v4, Generic Package Registry, K8s strategic merge patches, curl
---
## Context
**GitLab internal URL:** `http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181`
**GitLab project ID:** `1` (root/foxhunt)
**Existing secrets:** `gitlab-pat` (auto-rotated PAT, monthly), `minio-credentials`
**Existing file:** `crates/common/src/build_info.rs` — already captures `FOXHUNT_BUILD_VERSION` at compile time
**Web dashboard:** stays on MinIO (not code, not part of release)
**Services affected (7):** api, trading-service, ml-training-service, backtesting-service, trading-agent-service, broker-gateway, data-acquisition-service
**Binary names from cargo build:**
- Services: `api`, `trading-service`, `ml-training-service`, `backtesting-service`, `trading-agent-service`, `broker-gateway`, `data-acquisition-service`
- Training: `train_baseline_rl`, `train_baseline_supervised`, `evaluate_baseline`, `evaluate_supervised`, `hyperopt_baseline_rl`, `hyperopt_baseline_supervised`, `training_uploader`
---
### Task 1: Create GitLab deploy token and K8s secret
This is a one-time manual setup. The deploy token lets service pods download packages from GitLab's Generic Package Registry (read-only).
**Step 1: Create deploy token via GitLab API**
Run from a machine with access to the cluster:
```bash
# Use existing PAT to create a project deploy token
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
PAT=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)
curl -sf -X POST "${GITLAB}/api/v4/projects/1/deploy_tokens" \
-H "PRIVATE-TOKEN: ${PAT}" \
-d "name=foxhunt-package-reader" \
-d "scopes[]=read_package_registry" | jq .
```
Save the returned `token` value.
**Step 2: Create K8s secret**
```bash
kubectl -n foxhunt create secret generic gitlab-deploy-token \
--from-literal=token=<token-from-step-1>
```
**Step 3: Verify token can read packages**
```bash
# Should return 200 (empty list is fine — no packages yet)
curl -sf -o /dev/null -w "%{http_code}" \
-H "DEPLOY-TOKEN: <token>" \
"${GITLAB}/api/v4/projects/1/packages"
```
Expected: `200`
**Step 4: Verify existing gitlab-pat has api scope**
```bash
curl -sf "${GITLAB}/api/v4/personal_access_tokens/self" \
-H "PRIVATE-TOKEN: ${PAT}" | jq '.scopes'
```
Expected: scopes must include `api` (for creating tags, releases, uploading packages). If not, recreate the PAT with `api` scope.
---
### Task 2: Add `needs-code` output to detect-changes
**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml:191-220` (detect-changes script outputs)
**Step 1: Add needs-code computation to detect-changes script**
In the detect-changes shell script, after the `NEEDS_TRAINING` computation (line ~207) and before the `echo "=== Build decisions ==="` line, add:
```sh
if [ "$NEEDS_SERVICES" = "true" ] || [ "$NEEDS_TRAINING" = "true" ]; then
NEEDS_CODE="true"
else
NEEDS_CODE="false"
fi
```
And after `echo -n "$DOCKER_IMAGES" > /tmp/outputs/docker-images`, add:
```sh
echo -n "$NEEDS_CODE" > /tmp/outputs/needs-code
```
Also add the build decisions echo:
```sh
echo "needs-code: $NEEDS_CODE"
```
**Step 2: Add needs-code output parameter**
After the `docker-images` output parameter (line ~235), add:
```yaml
- name: needs-code
valueFrom:
path: /tmp/outputs/needs-code
```
**Step 3: Verify YAML is valid**
```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```
Expected: no error
---
### Task 3: New `create-tag` template
**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (add template after detect-changes, before compile-services)
**Step 1: Add create-tag template**
Insert after the detect-changes template `outputs:` block (after line ~237) and before the `build-web-dashboard` template:
```yaml
# ── create-tag: CalVer auto-tag on code changes ──
- name: create-tag
nodeSelector:
k8s.scaleway.com/pool-name: platform
container:
image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
command: ["/bin/sh", "-c"]
env:
- name: GITLAB_PAT
valueFrom:
secretKeyRef:
name: gitlab-pat
key: token
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
args:
- |
set -e
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
PROJECT_ID=1
SHA="{{workflow.parameters.commit-sha}}"
# Compute CalVer prefix: vYYYY.MM
PREFIX="v$(date +%Y.%m)"
# Query existing tags for this month
TAGS=$(curl -sf \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
"${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?search=${PREFIX}" \
|| echo "[]")
# Parse highest N from vYYYY.MM.N tags
LAST_N=$(echo "$TAGS" | grep -oP "\"name\":\"${PREFIX}\.\K[0-9]+" | sort -n | tail -1)
if [ -z "$LAST_N" ]; then
NEXT_N=1
else
NEXT_N=$((LAST_N + 1))
fi
TAG="${PREFIX}.${NEXT_N}"
echo "Creating tag: ${TAG} at ${SHA}"
# Create the tag
RESULT=$(curl -sf -X POST \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-d "tag_name=${TAG}" \
-d "ref=${SHA}" \
"${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags")
echo "$RESULT" | grep -q "$TAG" || { echo "Tag creation failed: $RESULT"; exit 1; }
echo "Tag ${TAG} created successfully"
mkdir -p /tmp/outputs
echo -n "$TAG" > /tmp/outputs/tag
outputs:
parameters:
- name: tag
valueFrom:
path: /tmp/outputs/tag
```
**Step 2: Add gitlab-pat volume to workflow spec**
In the `spec.volumes` list (around line 22-32), add:
```yaml
- name: gitlab-pat
secret:
secretName: gitlab-pat
```
**Step 3: Verify YAML**
```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```
Expected: no error
---
### Task 4: Modify compile-services — version from tag + upload to GitLab packages
**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (compile-services template, lines ~309-426)
**Step 1: Replace version computation**
Find and replace the 3 lines (around line 393-395):
```sh
YEAR=$(date +%Y)
MONTH=$(date +%m)
export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.argo"
```
With:
```sh
export FOXHUNT_BUILD_VERSION="{{tasks.create-tag.outputs.parameters.tag}}"
```
**Step 2: Replace MinIO upload with GitLab package upload**
Find and replace the rclone upload block (around lines 418-424):
```sh
echo "=== Uploading service binaries ==="
rclone copy "$WORKSPACE/bin/services/" :s3:foxhunt-binaries/services/ \
--s3-provider=Minio \
--s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
--s3-no-check-bucket
```
With:
```sh
echo "=== Uploading service binaries to GitLab packages ==="
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
TAG="${FOXHUNT_BUILD_VERSION}"
for bin in "$WORKSPACE/bin/services/"*; do
BIN_NAME=$(basename "$bin")
echo "Uploading ${BIN_NAME}..."
curl -f --upload-file "$bin" \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
"${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-services/${TAG}/${BIN_NAME}"
done
```
**Step 3: Add GITLAB_PAT env var, remove MINIO upload env vars**
In the compile-services container `env:` list, add:
```yaml
- name: GITLAB_PAT
valueFrom:
secretKeyRef:
name: gitlab-pat
key: token
```
Remove the `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` env entries (keep `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — they're needed by sccache).
**Step 4: Verify YAML**
```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```
---
### Task 5: Modify compile-training — version from tag + upload to GitLab packages
**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (compile-training template, lines ~430-547)
**Step 1: Replace version computation**
Find and replace (around line 513-514):
```sh
YEAR=$(date +%Y)
MONTH=$(date +%m)
export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.argo"
```
With:
```sh
export FOXHUNT_BUILD_VERSION="{{tasks.create-tag.outputs.parameters.tag}}"
```
**Step 2: Replace MinIO upload with GitLab package upload**
Find and replace the rclone upload block (around lines 539-545):
```sh
echo "=== Uploading training binaries ==="
rclone copy "$WORKSPACE/bin/training/" :s3:foxhunt-binaries/training/ \
--s3-provider=Minio \
--s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
--s3-access-key-id="${MINIO_ACCESS_KEY}" \
--s3-secret-access-key="${MINIO_SECRET_KEY}" \
--s3-no-check-bucket
```
With:
```sh
echo "=== Uploading training binaries to GitLab packages ==="
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
TAG="${FOXHUNT_BUILD_VERSION}"
for bin in "$WORKSPACE/bin/training/"*; do
BIN_NAME=$(basename "$bin")
echo "Uploading ${BIN_NAME}..."
curl -f --upload-file "$bin" \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
"${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-training/${TAG}/${BIN_NAME}"
done
```
**Step 3: Add GITLAB_PAT env var, remove MINIO upload env vars**
Same as Task 4 Step 3. Add `GITLAB_PAT` from `gitlab-pat` secret. Remove `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` (keep `AWS_*` for sccache).
**Step 4: Verify YAML**
```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```
---
### Task 6: New `upload-release` template
**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (add template after compile-training, before gpu-warmup)
**Step 1: Add upload-release template**
```yaml
# ── upload-release: create GitLab Release with package links ──
- name: upload-release
nodeSelector:
k8s.scaleway.com/pool-name: platform
container:
image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
command: ["/bin/sh", "-c"]
env:
- name: GITLAB_PAT
valueFrom:
secretKeyRef:
name: gitlab-pat
key: token
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
args:
- |
set -e
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
PROJECT_ID=1
TAG="{{tasks.create-tag.outputs.parameters.tag}}"
echo "=== Creating GitLab Release ${TAG} ==="
# Get commits since previous tag for release notes
PREV_TAG=$(curl -sf \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
"${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?per_page=2" \
| grep -oP '"name":"\K[^"]+' | sed -n '2p')
if [ -n "$PREV_TAG" ]; then
COMMITS=$(curl -sf \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
"${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/compare?from=${PREV_TAG}&to=${TAG}" \
| grep -oP '"title":"\K[^"]+' | head -20 \
| sed 's/^/- /' || echo "- Release ${TAG}")
DESCRIPTION="## Changes since ${PREV_TAG}\n\n${COMMITS}"
else
DESCRIPTION="## Initial release\n\nFirst CalVer release."
fi
# Create the release
RESPONSE=$(curl -sf -X POST \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-H "Content-Type: application/json" \
-d "{
\"tag_name\": \"${TAG}\",
\"name\": \"${TAG}\",
\"description\": \"$(printf '%s' "$DESCRIPTION" | sed 's/"/\\"/g')\"
}" \
"${GITLAB}/api/v4/projects/${PROJECT_ID}/releases") || {
echo "WARN: Release creation failed (may already exist)"
}
echo "Release ${TAG} created"
echo "$RESPONSE" | head -5
# Add package links as release assets
PKG_URL="${GITLAB}/api/v4/projects/${PROJECT_ID}/packages/generic"
for pkg_name in foxhunt-services foxhunt-training; do
# Check if package exists for this tag
PKG_CHECK=$(curl -sf \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
"${GITLAB}/api/v4/projects/${PROJECT_ID}/packages?package_name=${pkg_name}&package_version=${TAG}" \
|| echo "[]")
if echo "$PKG_CHECK" | grep -q "$TAG"; then
curl -sf -X POST \
-H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-H "Content-Type: application/json" \
-d "{
\"name\": \"${pkg_name}\",
\"url\": \"${GITLAB}/-/packages?type=generic&search=${pkg_name}&version=${TAG}\",
\"link_type\": \"package\"
}" \
"${GITLAB}/api/v4/projects/${PROJECT_ID}/releases/${TAG}/assets/links" || true
echo "Linked ${pkg_name} package to release"
fi
done
echo "=== Release ${TAG} complete ==="
```
**Step 2: Verify YAML**
```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```
---
### Task 7: Modify deploy-services — patch FOXHUNT_RELEASE on deployments
**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (deploy-services template, lines ~582-620)
**Step 1: Replace deploy-services script**
Replace the entire `args` block of deploy-services with:
```sh
set -e
if ! command -v kubectl >/dev/null 2>&1; then
echo "=== Installing kubectl ==="
curl -sLo /tmp/kubectl "https://dl.k8s.io/release/v1.31.4/bin/linux/amd64/kubectl"
chmod +x /tmp/kubectl
export PATH="/tmp:$PATH"
fi
TAG="{{tasks.create-tag.outputs.parameters.tag}}"
echo "=== Deploying release ${TAG} ==="
SERVICES="api trading-service ml-training-service backtesting-service trading-agent-service broker-gateway data-acquisition-service"
for svc in $SERVICES; do
echo "Patching $svc with FOXHUNT_RELEASE=${TAG}..."
kubectl -n foxhunt patch deployment "$svc" -p "{
\"spec\":{\"template\":{
\"metadata\":{\"annotations\":{\"foxhunt.io/release\":\"${TAG}\"}},
\"spec\":{\"initContainers\":[{
\"name\":\"fetch-binary\",
\"env\":[{\"name\":\"FOXHUNT_RELEASE\",\"value\":\"${TAG}\"}]
}]}
}}
}" || echo "WARN: $svc patch failed (may not exist)"
done
echo "=== Waiting for rollouts ==="
for svc in $SERVICES; do
kubectl -n foxhunt rollout status deployment "$svc" --timeout=120s || echo "WARN: $svc rollout timeout"
done
echo "=== Deploy ${TAG} complete ==="
```
**Step 2: Verify YAML**
```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```
---
### Task 8: Update DAG dependencies
**Files:**
- Modify: `infra/k8s/argo/ci-pipeline-template.yaml` (DAG tasks, lines ~44-136)
**Step 1: Replace the DAG tasks section**
Replace the entire `dag.tasks` list with:
```yaml
tasks:
- name: detect-changes
template: detect-changes
# Create CalVer tag when code changed
- name: create-tag
dependencies: [detect-changes]
template: create-tag
when: "{{tasks.detect-changes.outputs.parameters.needs-code}} == true"
# Parallel: each compile step is self-contained (own git clone)
# sccache backed by MinIO — no PVC needed, shared across all nodes
- name: compile-services
dependencies: [create-tag]
template: compile-services
when: "{{tasks.detect-changes.outputs.parameters.needs-services}} == true"
- name: compile-training
dependencies: [create-tag]
template: compile-training
when: "{{tasks.detect-changes.outputs.parameters.needs-training}} == true"
# GPU warmup: trigger GPU node autoscale during compile so the node
# is ready when we submit a training workflow after CI completes.
- name: gpu-warmup
dependencies: [detect-changes]
template: gpu-warmup
when: "{{tasks.detect-changes.outputs.parameters.needs-training}} == true"
- name: build-web-dashboard
dependencies: [detect-changes]
template: build-web-dashboard
when: "{{tasks.detect-changes.outputs.parameters.needs-dashboard}} == true"
# Create GitLab Release after all binaries are uploaded
- name: upload-release
dependencies: [compile-services, compile-training]
template: upload-release
when: "{{tasks.detect-changes.outputs.parameters.needs-code}} == true"
# Deploy after release is created
- name: deploy-services
dependencies: [upload-release]
template: deploy-services
when: "{{tasks.detect-changes.outputs.parameters.needs-services}} == true"
- name: rebuild-ci-builder
dependencies: [detect-changes]
templateRef:
name: build-ci-image
template: build
arguments:
parameters:
- name: commit-sha
value: "{{workflow.parameters.commit-sha}}"
- name: dockerfile
value: Dockerfile.ci-builder
- name: image-name
value: ci-builder
when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
- name: rebuild-ci-builder-cpu
dependencies: [detect-changes]
templateRef:
name: build-ci-image
template: build
arguments:
parameters:
- name: commit-sha
value: "{{workflow.parameters.commit-sha}}"
- name: dockerfile
value: Dockerfile.ci-builder-cpu
- name: image-name
value: ci-builder-cpu
when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
- name: rebuild-runtime
dependencies: [detect-changes]
templateRef:
name: build-ci-image
template: build
arguments:
parameters:
- name: commit-sha
value: "{{workflow.parameters.commit-sha}}"
- name: dockerfile
value: Dockerfile.foxhunt-runtime
- name: image-name
value: foxhunt-runtime
when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
- name: rebuild-training-runtime
dependencies: [detect-changes]
templateRef:
name: build-ci-image
template: build
arguments:
parameters:
- name: commit-sha
value: "{{workflow.parameters.commit-sha}}"
- name: dockerfile
value: Dockerfile.foxhunt-training-runtime
- name: image-name
value: foxhunt-training-runtime
when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
```
**Step 2: Verify the full pipeline YAML**
```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/argo/ci-pipeline-template.yaml'))"
```
---
### Task 9: Update service initContainers — fetch from GitLab
**Files (7 service deployments):**
- Modify: `infra/k8s/services/api.yaml:41-78`
- Modify: `infra/k8s/services/trading-service.yaml:41-77`
- Modify: `infra/k8s/services/ml-training-service.yaml:83-120` (offset by SA/RBAC)
- Modify: `infra/k8s/services/backtesting-service.yaml` (same initContainer pattern)
- Modify: `infra/k8s/services/broker-gateway.yaml` (same initContainer pattern)
- Modify: `infra/k8s/services/data-acquisition-service.yaml` (same initContainer pattern)
- Modify: `infra/k8s/services/trading-agent-service.yaml` (same initContainer pattern)
Each file has the same initContainer pattern. Replace the `fetch-binary` initContainer in all 7 files.
**Step 1: Replace initContainer in each service**
For each service, replace the `fetch-binary` initContainer block. The template (substitute `SERVICE_NAME` for each):
```yaml
- name: fetch-binary
image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
command: ["/bin/sh", "-c"]
args:
- |
set -e
BINARY="SERVICE_NAME"
curl -fSL -o "/binaries/${BINARY}" \
--header "DEPLOY-TOKEN: ${GITLAB_DEPLOY_TOKEN}" \
"${GITLAB_API}/projects/1/packages/generic/foxhunt-services/${FOXHUNT_RELEASE}/${BINARY}"
chmod +x "/binaries/${BINARY}"
echo "Fetched ${BINARY} ${FOXHUNT_RELEASE} ($(stat -c%s /binaries/${BINARY}) bytes)"
env:
- name: GITLAB_DEPLOY_TOKEN
valueFrom:
secretKeyRef:
name: gitlab-deploy-token
key: token
- name: GITLAB_API
value: "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181/api/v4"
- name: FOXHUNT_RELEASE
value: "latest"
volumeMounts:
- name: binaries
mountPath: /binaries
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
```
Substitutions per file:
- `api.yaml`: `SERVICE_NAME` = `api`
- `trading-service.yaml`: `SERVICE_NAME` = `trading-service`
- `ml-training-service.yaml`: `SERVICE_NAME` = `ml-training-service`
- `backtesting-service.yaml`: `SERVICE_NAME` = `backtesting-service`
- `trading-agent-service.yaml`: `SERVICE_NAME` = `trading-agent-service`
- `broker-gateway.yaml`: `SERVICE_NAME` = `broker-gateway`
- `data-acquisition-service.yaml`: `SERVICE_NAME` = `data-acquisition-service`
Note: The `FOXHUNT_RELEASE` value `"latest"` is a placeholder — the deploy-services CI step patches it to the actual tag via `kubectl patch`.
**Step 2: Verify all 7 service YAMLs**
```bash
for f in infra/k8s/services/{api,trading-service,ml-training-service,backtesting-service,trading-agent-service,broker-gateway,data-acquisition-service}.yaml; do
python3 -c "import yaml; list(yaml.safe_load_all(open('$f')))" && echo "$f OK" || echo "$f FAILED"
done
```
Expected: all OK
---
### Task 10: Update training job-template — binary fetch from GitLab
**Files:**
- Modify: `infra/k8s/training/job-template.yaml:38-78` (fetch-binary initContainer only; sync-training-data stays MinIO)
**Step 1: Replace fetch-binary initContainer**
Replace the first initContainer (`fetch-binary`, lines 39-78) with:
```yaml
- name: fetch-binary
image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
command: ["/bin/sh", "-c"]
args:
- |
set -e
BINARY="$(TRAINING_BINARY)"
curl -fSL -o "/binaries/${BINARY}" \
--header "DEPLOY-TOKEN: ${GITLAB_DEPLOY_TOKEN}" \
"${GITLAB_API}/projects/1/packages/generic/foxhunt-training/${FOXHUNT_RELEASE}/${BINARY}"
chmod +x "/binaries/${BINARY}"
echo "Fetched ${BINARY} ${FOXHUNT_RELEASE} ($(stat -c%s /binaries/${BINARY}) bytes)"
env:
- name: TRAINING_BINARY
value: train_baseline_supervised
- name: GITLAB_DEPLOY_TOKEN
valueFrom:
secretKeyRef:
name: gitlab-deploy-token
key: token
- name: GITLAB_API
value: "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181/api/v4"
- name: FOXHUNT_RELEASE
value: "latest"
volumeMounts:
- name: binaries
mountPath: /binaries
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
```
Note: `sync-training-data` initContainer stays unchanged (MinIO). Only binary fetch moves to GitLab.
**Step 2: Verify YAML**
```bash
python3 -c "import yaml; yaml.safe_load(open('infra/k8s/training/job-template.yaml'))"
```
---
### Task 11: Commit and apply to cluster
**Step 1: Commit all changes**
```bash
git add infra/k8s/argo/ci-pipeline-template.yaml \
infra/k8s/services/api.yaml \
infra/k8s/services/trading-service.yaml \
infra/k8s/services/ml-training-service.yaml \
infra/k8s/services/backtesting-service.yaml \
infra/k8s/services/trading-agent-service.yaml \
infra/k8s/services/broker-gateway.yaml \
infra/k8s/services/data-acquisition-service.yaml \
infra/k8s/training/job-template.yaml
git commit -m "feat(infra): GitLab releases with CalVer auto-versioning
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.
- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints"
```
**Step 2: Apply CI pipeline template**
```bash
kubectl apply -f infra/k8s/argo/ci-pipeline-template.yaml
```
**Step 3: Apply service deployments (will NOT restart — just updates spec)**
```bash
kubectl apply -f infra/k8s/services/api.yaml
kubectl apply -f infra/k8s/services/trading-service.yaml
kubectl apply -f infra/k8s/services/ml-training-service.yaml
kubectl apply -f infra/k8s/services/backtesting-service.yaml
kubectl apply -f infra/k8s/services/trading-agent-service.yaml
kubectl apply -f infra/k8s/services/broker-gateway.yaml
kubectl apply -f infra/k8s/services/data-acquisition-service.yaml
```
Note: Don't apply job-template.yaml — it's a template that is applied per-job, not a persistent resource.
**Step 4: Push to trigger first release**
```bash
git push origin main
```
**Step 5: Monitor the Argo workflow**
```bash
# Watch the workflow
kubectl -n foxhunt get workflows -w
# Check tag was created
kubectl -n foxhunt logs -l workflows.argoproj.io/workflow -c main --tail=50 | grep "Creating tag"
```
Expected: workflow creates tag `v2026.03.1`, compiles, uploads to GitLab packages, creates release, patches deployments.
**Step 6: Verify services restarted with new binary**
```bash
kubectl -n foxhunt get pods -o wide
kubectl -n foxhunt logs deployment/api -c fetch-binary
```
Expected: initContainer logs show `Fetched api v2026.03.1 (... bytes)`
---
### Task 12: Cleanup — remove MinIO binary bucket
Only do this after Task 11 is verified working.
**Step 1: Verify GitLab packages exist**
```bash
GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
PAT=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)
curl -sf -H "PRIVATE-TOKEN: ${PAT}" \
"${GITLAB}/api/v4/projects/1/packages?package_type=generic" | python3 -m json.tool
```
Expected: lists foxhunt-services and foxhunt-training packages with the tag version.
**Step 2: Remove binary bucket contents from MinIO**
```bash
kubectl -n foxhunt exec deployment/minio -- mc rm --recursive --force /data/foxhunt-binaries/services/
kubectl -n foxhunt exec deployment/minio -- mc rm --recursive --force /data/foxhunt-binaries/training/
```
Keep the bucket itself (other paths may reference it).
**Step 3: Commit cleanup note**
No code changes needed — the MinIO upload code was already removed in Task 4/5.
Reference in New Issue View Git Blame Copy Permalink