From 3c4af61cb0a1585b0ff75984a660f1d513fa09d4 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 24 Feb 2026 23:13:00 +0100 Subject: [PATCH] =?UTF-8?q?docs:=20GitLab=20CE=20migration=20design=20(Git?= =?UTF-8?q?ea=20=E2=86=92=20GitLab=20on=20Kapsule)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers: in-cluster GitLab CE with external Postgres/Redis, dedicated ci-build node pool (scale-to-zero), Kaniko image builds, GitLab built-in registry, and 4-phase migration plan with rollback. Co-Authored-By: Claude Opus 4.6 --- .../2026-02-24-gitlab-migration-design.md | 275 ++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 docs/plans/2026-02-24-gitlab-migration-design.md diff --git a/docs/plans/2026-02-24-gitlab-migration-design.md b/docs/plans/2026-02-24-gitlab-migration-design.md new file mode 100644 index 000000000..ac39e01e6 --- /dev/null +++ b/docs/plans/2026-02-24-gitlab-migration-design.md @@ -0,0 +1,275 @@ +# GitLab CE Migration Design + +## Problem + +Gitea + act_runner on a dedicated GP1-S VM cannot spawn ephemeral build pods in Kapsule. The CI runner is a fixed-capacity VM that idles most of the time, requires a custom auto-start watcher, and can't leverage K8s autoscaling. We need CI that scales to zero and integrates natively with our Kubernetes cluster. + +## Solution + +Migrate from Gitea to GitLab CE running as a pod in Kapsule with the GitLab Runner Kubernetes executor. Build pods schedule on a dedicated autoscaling node pool (0-N), giving true scale-to-zero CI. + +## Infrastructure Topology + +``` + ┌─────────────────────────────────────────────┐ + │ Scaleway Kapsule Cluster │ + │ │ + │ ┌───────────────┐ ┌─────────────────────┐ │ + │ │ always-on │ │ gitlab (NEW) │ │ + │ │ DEV1-M │ │ DEV1-L, fixed 1 │ │ + │ │ ──────────── │ │ ────────────────── │ │ + │ │ postgres │ │ GitLab CE (Helm) │ │ + │ │ redis │ │ Puma, Sidekiq, │ │ + │ │ questdb │ │ Gitaly, Workhorse, │ │ + │ └───────┬───────┘ │ Shell, Registry, │ │ + │ │ │ Runner Manager │ │ + │ │ ext DB └──────────┬──────────┘ │ + │ └─────────────────────┘ │ + │ │ + │ ┌───────────────┐ ┌─────────────────────┐ │ + │ │ ci │ │ ci-build (NEW) │ │ + │ │ GP1-XS, 0-1 │ │ GP1-XS, 0-2 │ │ + │ │ ──────────── │ │ ────────────────── │ │ + │ │ api-gateway │ │ GitLab Runner │ │ + │ │ web-gateway │ │ build pods │ │ + │ │ trading-svc │ │ (ephemeral) │ │ + │ │ ...services │ │ scale-to-zero │ │ + │ └───────────────┘ └─────────────────────┘ │ + │ │ + │ ┌───────────────┐ ┌─────────────────────┐ │ + │ │ gpu-training │ │ gpu-inference │ │ + │ │ H100, 0-1 │ │ L4, 0-1 │ │ + │ └───────────────┘ └─────────────────────┘ │ + └─────────────────────────────────────────────┘ + +DNS: gitlab.fxhnt.ai → Tailscale IP (subnet router) +Access: Tailscale-only (no public ingress) +``` + +## New Node Pools + +### `gitlab` pool +- **Type**: DEV1-L (8GB RAM, 4 vCPU) +- **Autoscale**: Fixed 1 node (always on) +- **Taint**: `gitlab=true:NoSchedule` +- **Purpose**: GitLab CE core components + Runner manager pod +- **Cost**: ~€0.03/hr (€22/mo) + +### `ci-build` pool +- **Type**: GP1-XS (16GB RAM, 8 vCPU) +- **Autoscale**: 0-2 nodes +- **Taint**: `ci-build=true:NoSchedule` +- **Purpose**: Ephemeral GitLab Runner build pods +- **Cost**: €0.00 idle, ~€0.08/hr per node during builds +- **Scale-down delay**: 10 minutes (matches existing autoscaler config) + +## GitLab CE Deployment + +### Helm Chart: `gitlab/gitlab` + +**Enabled subcharts:** +- Puma (webserver): 2 workers +- Sidekiq: 1 replica, concurrency 10 +- Gitaly: local storage, 50Gi PVC +- Workhorse: bundled +- Shell: git-over-SSH on port 2222 +- Registry: built-in, S3-backed + +**Disabled subcharts:** +- PostgreSQL — external (existing always-on pod) +- Redis — external (existing always-on pod) +- MinIO — replaced by Scaleway Object Storage +- Prometheus/Grafana — not needed +- cert-manager — Tailscale-only access +- nginx-ingress — Tailscale-only access + +### External Database Configuration + +**PostgreSQL** (existing TimescaleDB on always-on pool): +```sql +CREATE DATABASE gitlab; +CREATE USER gitlab WITH PASSWORD ''; +GRANT ALL ON DATABASE gitlab TO gitlab; +CREATE EXTENSION IF NOT EXISTS pg_trgm; +CREATE EXTENSION IF NOT EXISTS btree_gist; +``` + +**Redis** (existing Redis on always-on pool): +- GitLab configured to use DB indexes 8-15 (avoids collision with application data on default index 0) + +### Storage + +| Data | Backend | Details | +|------|---------|---------| +| Git repos (Gitaly) | PVC | 50Gi on gitlab node | +| Container registry blobs | Scaleway Object Storage | Bucket: `foxhunt-gitlab-registry` | +| CI artifacts & uploads | Scaleway Object Storage | Bucket: `foxhunt-gitlab-artifacts` | +| CI cache (sccache) | Scaleway Object Storage | Existing bucket, 30-day lifecycle | + +### Access + +- **Web UI**: `https://gitlab.fxhnt.ai` via Tailscale +- **SSH**: `ssh://git@gitlab.fxhnt.ai:2222/foxhunt/foxhunt.git` +- **Registry**: `gitlab.fxhnt.ai:5050/foxhunt/foxhunt/` +- **DNS**: A record `gitlab.fxhnt.ai` → Tailscale IP (Scaleway DNS zone `fxhnt.ai`) +- **TLS**: Self-signed or Tailscale HTTPS certs (no public CA needed) + +## GitLab Runner Configuration + +### Runner Manager +- Deployed via `gitlab-runner` Helm chart on `gitlab` node pool +- **Executor**: Kubernetes +- **Concurrent**: 4 (can run 4 build pods in parallel) +- **Pod scheduling**: nodeSelector `ci-build=true`, toleration for taint + +### Build Pod Spec +``` +┌─ Build Pod (ephemeral, ci-build pool) ─────────────────┐ +│ build container: rust:1.89-slim + protoc + sccache │ +│ helper container: gitlab-runner-helper (clone, upload) │ +│ sidecar: kaniko (docker image builds) │ +└────────────────────────────────────────────────────────┘ +``` + +### Docker Image Builds: Kaniko +- No privileged containers needed (unlike Docker-in-Docker) +- Kaniko builds images in userspace, pushes directly to GitLab registry +- Replaces `docker build && docker push` in current pipeline + +### Scale-to-Zero Flow +1. Push to main → GitLab creates pipeline +2. Runner manager requests build pod → K8s autoscaler sees pending pod +3. Autoscaler provisions ci-build node (GP1-XS) — ~2min startup +4. Build pods execute (check, test, build images, deploy) +5. Pods terminate → after 10min idle → autoscaler removes ci-build node +6. Cost returns to €0.00 for CI + +## CI Pipeline Translation + +### Current: `.gitea/workflows/ci.yaml` +``` +check → test → build-images (matrix: 6 services) + build-web-gateway + build-training → deploy +``` + +### New: `.gitlab-ci.yml` + +| Stage | Jobs | Notes | +|-------|------|-------| +| `check` | `cargo check --workspace`, `cargo clippy --workspace -- -D warnings` | sccache enabled | +| `test` | `cargo test --workspace --lib` | sccache enabled | +| `build` | 8 parallel Kaniko jobs (6 services + web-gateway + training) | Push to GitLab registry | +| `deploy` | `kubectl set image` for each deployment | kubeconfig from CI variable | + +**Environment variables (CI/CD settings):** +- `SCCACHE_BUCKET`, `SCCACHE_ENDPOINT` — existing S3 cache +- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` — Scaleway S3 creds +- `KUBECONFIG` — base64-encoded cluster config (file variable) +- `SQLX_OFFLINE=true` + +**Registry change impact:** +- K8s manifests: `image: rg.fr-par.scw.cloud/foxhunt/` → `image: gitlab.fxhnt.ai:5050/foxhunt/foxhunt/` +- imagePullSecret: Scaleway registry creds → GitLab deploy token +- CI pipeline: Kaniko pushes natively to GitLab registry (no separate `docker login`) + +## Terragrunt Changes + +### New/Modified Modules + +**`infra/modules/kapsule/main.tf`** — add two node pools: +```hcl +resource "scaleway_k8s_pool" "gitlab" { + cluster_id = scaleway_k8s_cluster.foxhunt.id + name = "gitlab" + node_type = "DEV1-L" + size = 1 + autoscaling = false + tags = ["gitlab"] +} + +resource "scaleway_k8s_pool" "ci_build" { + cluster_id = scaleway_k8s_cluster.foxhunt.id + name = "ci-build" + node_type = "GP1-XS" + size = 0 + min_size = 0 + max_size = 2 + autoscaling = true + autohealing = true + tags = ["ci-build"] +} +``` + +**`infra/modules/object-storage/main.tf`** — add GitLab buckets: +- `foxhunt-gitlab-registry` +- `foxhunt-gitlab-artifacts` + +### Deleted Modules +- `infra/modules/ci-runner/` — entire module (VM, cloud-init, auto-start watcher) +- `infra/live/production/ci-runner/` — terragrunt config + +### Deleted Infrastructure +- GP1-S CI runner VM (`vm-fxhnt-ci`) +- Auto-start watcher systemd units on Gitea server +- DEV1-S Gitea VM (`vm-fxhnt-git`) — after successful cutover + +## Migration Plan + +### Phase 1: Stand Up GitLab (parallel with Gitea) +1. Terragrunt: add `gitlab` + `ci-build` node pools to kapsule module +2. Terragrunt: add object storage buckets for registry + artifacts +3. Create `gitlab` database + user in existing Postgres +4. Helm install GitLab CE with external Postgres/Redis, disabled subcharts +5. DNS: add `gitlab` A record in `fxhnt.ai` zone → Tailscale IP +6. Verify GitLab web UI accessible, create admin account + project + +### Phase 2: Migrate Repo & Configure CI +1. `git push --mirror` from Gitea to GitLab +2. Deploy GitLab Runner (Helm) with Kubernetes executor targeting ci-build pool +3. Create `.gitlab-ci.yml` (translate from `.gitea/workflows/ci.yaml`) +4. Configure CI/CD variables (sccache, kubeconfig, S3 creds) +5. Trigger test pipeline, verify all stages pass +6. Verify images in GitLab registry, K8s can pull them + +### Phase 3: Cutover +1. Update local git remote: `origin` → `ssh://git@gitlab.fxhnt.ai:2222/foxhunt/foxhunt.git` +2. Update K8s manifests: image refs → GitLab registry +3. Update imagePullSecret → GitLab deploy token +4. Verify deploy from GitLab pipeline works end-to-end +5. Run full CI cycle: push → check → test → build → deploy → verify services healthy + +### Phase 4: Decommission +1. `terragrunt destroy` ci-runner module (GP1-S VM gone) +2. Decommission Gitea VM (DEV1-S) +3. Delete from repo: `infra/modules/ci-runner/`, `infra/live/production/ci-runner/`, `.gitea/` +4. Optionally delete Scaleway container registry namespace +5. Update `CLAUDE.md`, memory files, infrastructure docs + +### Rollback +Gitea remains running until Phase 4. If anything fails in Phase 1-3, revert to Gitea with zero data loss. The mirror is one-way push. + +## Cost Analysis + +| Resource | Current | After Migration | +|----------|---------|-----------------| +| Gitea VM (DEV1-S) | €7/mo | €0 (deleted) | +| CI runner VM (GP1-S) | €44/mo | €0 (deleted) | +| GitLab node (DEV1-L) | — | €22/mo | +| ci-build pool (GP1-XS, 0-2) | — | ~€5-15/mo (usage-based) | +| Object storage (registry + artifacts) | ~€3/mo | ~€5/mo | +| **Total CI/Git infra** | **~€54/mo** | **~€32-42/mo** | + +Net savings: ~€12-22/mo, plus the operational benefit of K8s-native CI with scale-to-zero. + +## Decisions Log + +| Decision | Choice | Rationale | +|----------|--------|-----------| +| GitLab location | K8s pod (dedicated node pool) | Native K8s integration, no extra VM | +| GitLab node type | DEV1-L (8GB) | Comfortable for CE, external DB/Redis saves RAM | +| External Postgres/Redis | Yes | Reuse existing always-on pods, less resource waste | +| Build pod pool | Dedicated `ci-build` (GP1-XS, 0-2) | Clean separation from services, true scale-to-zero | +| Docker builds | Kaniko | No privileged containers, pushes directly to registry | +| Container registry | GitLab built-in (S3-backed) | Consolidate, one less external dependency | +| Migration strategy | Push mirror, parallel run | Zero-risk cutover with rollback window | +| Access | Tailscale-only | Matches existing security posture | +| DNS | `gitlab.fxhnt.ai` (new subdomain) | Separate during migration, optionally alias later |