gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
2e3f41de4e9f15e09786d2bb7775d8a9df324613
foxhunt/docs/plans/2026-02-24-gitlab-migration-design.md
jgrusewski 3c4af61cb0 docs: GitLab CE migration design (Gitea → GitLab on Kapsule) 
Covers: in-cluster GitLab CE with external Postgres/Redis, dedicated
ci-build node pool (scale-to-zero), Kaniko image builds, GitLab
built-in registry, and 4-phase migration plan with rollback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:13:00 +01:00

12 KiB
Raw Blame History

GitLab CE Migration Design

Problem

Gitea + act_runner on a dedicated GP1-S VM cannot spawn ephemeral build pods in Kapsule. The CI runner is a fixed-capacity VM that idles most of the time, requires a custom auto-start watcher, and can't leverage K8s autoscaling. We need CI that scales to zero and integrates natively with our Kubernetes cluster.

Solution

Migrate from Gitea to GitLab CE running as a pod in Kapsule with the GitLab Runner Kubernetes executor. Build pods schedule on a dedicated autoscaling node pool (0-N), giving true scale-to-zero CI.

Infrastructure Topology

                 ┌─────────────────────────────────────────────┐
                 │           Scaleway Kapsule Cluster           │
                 │                                              │
                 │  ┌───────────────┐  ┌─────────────────────┐ │
                 │  │ always-on     │  │ gitlab (NEW)        │ │
                 │  │ DEV1-M        │  │ DEV1-L, fixed 1     │ │
                 │  │ ────────────  │  │ ──────────────────  │ │
                 │  │ postgres      │  │ GitLab CE (Helm)    │ │
                 │  │ redis         │  │ Puma, Sidekiq,      │ │
                 │  │ questdb       │  │ Gitaly, Workhorse,  │ │
                 │  └───────┬───────┘  │ Shell, Registry,    │ │
                 │          │          │ Runner Manager       │ │
                 │          │ ext DB   └──────────┬──────────┘ │
                 │          └─────────────────────┘            │
                 │                                              │
                 │  ┌───────────────┐  ┌─────────────────────┐ │
                 │  │ ci            │  │ ci-build (NEW)      │ │
                 │  │ GP1-XS, 0-1  │  │ GP1-XS, 0-2        │ │
                 │  │ ────────────  │  │ ──────────────────  │ │
                 │  │ api-gateway   │  │ GitLab Runner       │ │
                 │  │ web-gateway   │  │ build pods          │ │
                 │  │ trading-svc   │  │ (ephemeral)         │ │
                 │  │ ...services   │  │ scale-to-zero       │ │
                 │  └───────────────┘  └─────────────────────┘ │
                 │                                              │
                 │  ┌───────────────┐  ┌─────────────────────┐ │
                 │  │ gpu-training  │  │ gpu-inference        │ │
                 │  │ H100, 0-1    │  │ L4, 0-1             │ │
                 │  └───────────────┘  └─────────────────────┘ │
                 └─────────────────────────────────────────────┘

DNS: gitlab.fxhnt.ai → Tailscale IP (subnet router)
Access: Tailscale-only (no public ingress)

New Node Pools

gitlab pool

  • Type: DEV1-L (8GB RAM, 4 vCPU)
  • Autoscale: Fixed 1 node (always on)
  • Taint: gitlab=true:NoSchedule
  • Purpose: GitLab CE core components + Runner manager pod
  • Cost: ~€0.03/hr (€22/mo)

ci-build pool

  • Type: GP1-XS (16GB RAM, 8 vCPU)
  • Autoscale: 0-2 nodes
  • Taint: ci-build=true:NoSchedule
  • Purpose: Ephemeral GitLab Runner build pods
  • Cost: €0.00 idle, ~€0.08/hr per node during builds
  • Scale-down delay: 10 minutes (matches existing autoscaler config)

GitLab CE Deployment

Helm Chart: gitlab/gitlab

Enabled subcharts:

  • Puma (webserver): 2 workers
  • Sidekiq: 1 replica, concurrency 10
  • Gitaly: local storage, 50Gi PVC
  • Workhorse: bundled
  • Shell: git-over-SSH on port 2222
  • Registry: built-in, S3-backed

Disabled subcharts:

  • PostgreSQL — external (existing always-on pod)
  • Redis — external (existing always-on pod)
  • MinIO — replaced by Scaleway Object Storage
  • Prometheus/Grafana — not needed
  • cert-manager — Tailscale-only access
  • nginx-ingress — Tailscale-only access

External Database Configuration

PostgreSQL (existing TimescaleDB on always-on pool):

CREATE DATABASE gitlab;
CREATE USER gitlab WITH PASSWORD '<generated-by-terraform>';
GRANT ALL ON DATABASE gitlab TO gitlab;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS btree_gist;

Redis (existing Redis on always-on pool):

  • GitLab configured to use DB indexes 8-15 (avoids collision with application data on default index 0)

Storage

Data Backend Details
Git repos (Gitaly) PVC 50Gi on gitlab node
Container registry blobs Scaleway Object Storage Bucket: foxhunt-gitlab-registry
CI artifacts & uploads Scaleway Object Storage Bucket: foxhunt-gitlab-artifacts
CI cache (sccache) Scaleway Object Storage Existing bucket, 30-day lifecycle

Access

  • Web UI: https://gitlab.fxhnt.ai via Tailscale
  • SSH: ssh://git@gitlab.fxhnt.ai:2222/foxhunt/foxhunt.git
  • Registry: gitlab.fxhnt.ai:5050/foxhunt/foxhunt/<service>
  • DNS: A record gitlab.fxhnt.ai → Tailscale IP (Scaleway DNS zone fxhnt.ai)
  • TLS: Self-signed or Tailscale HTTPS certs (no public CA needed)

GitLab Runner Configuration

Runner Manager

  • Deployed via gitlab-runner Helm chart on gitlab node pool
  • Executor: Kubernetes
  • Concurrent: 4 (can run 4 build pods in parallel)
  • Pod scheduling: nodeSelector ci-build=true, toleration for taint

Build Pod Spec

┌─ Build Pod (ephemeral, ci-build pool) ─────────────────┐
│  build container:  rust:1.89-slim + protoc + sccache   │
│  helper container: gitlab-runner-helper (clone, upload) │
│  sidecar:          kaniko (docker image builds)         │
└────────────────────────────────────────────────────────┘

Docker Image Builds: Kaniko

  • No privileged containers needed (unlike Docker-in-Docker)
  • Kaniko builds images in userspace, pushes directly to GitLab registry
  • Replaces docker build && docker push in current pipeline

Scale-to-Zero Flow

  1. Push to main → GitLab creates pipeline
  2. Runner manager requests build pod → K8s autoscaler sees pending pod
  3. Autoscaler provisions ci-build node (GP1-XS) — ~2min startup
  4. Build pods execute (check, test, build images, deploy)
  5. Pods terminate → after 10min idle → autoscaler removes ci-build node
  6. Cost returns to €0.00 for CI

CI Pipeline Translation

Current: .gitea/workflows/ci.yaml

check → test → build-images (matrix: 6 services) + build-web-gateway + build-training → deploy

New: .gitlab-ci.yml

Stage Jobs Notes
check cargo check --workspace, cargo clippy --workspace -- -D warnings sccache enabled
test cargo test --workspace --lib sccache enabled
build 8 parallel Kaniko jobs (6 services + web-gateway + training) Push to GitLab registry
deploy kubectl set image for each deployment kubeconfig from CI variable

Environment variables (CI/CD settings):

  • SCCACHE_BUCKET, SCCACHE_ENDPOINT — existing S3 cache
  • AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY — Scaleway S3 creds
  • KUBECONFIG — base64-encoded cluster config (file variable)
  • SQLX_OFFLINE=true

Registry change impact:

  • K8s manifests: image: rg.fr-par.scw.cloud/foxhunt/<svc> → image: gitlab.fxhnt.ai:5050/foxhunt/foxhunt/<svc>
  • imagePullSecret: Scaleway registry creds → GitLab deploy token
  • CI pipeline: Kaniko pushes natively to GitLab registry (no separate docker login)

Terragrunt Changes

New/Modified Modules

infra/modules/kapsule/main.tf — add two node pools:

resource "scaleway_k8s_pool" "gitlab" {
  cluster_id  = scaleway_k8s_cluster.foxhunt.id
  name        = "gitlab"
  node_type   = "DEV1-L"
  size        = 1
  autoscaling = false
  tags        = ["gitlab"]
}

resource "scaleway_k8s_pool" "ci_build" {
  cluster_id    = scaleway_k8s_cluster.foxhunt.id
  name          = "ci-build"
  node_type     = "GP1-XS"
  size          = 0
  min_size      = 0
  max_size      = 2
  autoscaling   = true
  autohealing   = true
  tags          = ["ci-build"]
}

infra/modules/object-storage/main.tf — add GitLab buckets:

  • foxhunt-gitlab-registry
  • foxhunt-gitlab-artifacts

Deleted Modules

  • infra/modules/ci-runner/ — entire module (VM, cloud-init, auto-start watcher)
  • infra/live/production/ci-runner/ — terragrunt config

Deleted Infrastructure

  • GP1-S CI runner VM (vm-fxhnt-ci)
  • Auto-start watcher systemd units on Gitea server
  • DEV1-S Gitea VM (vm-fxhnt-git) — after successful cutover

Migration Plan

Phase 1: Stand Up GitLab (parallel with Gitea)

  1. Terragrunt: add gitlab + ci-build node pools to kapsule module
  2. Terragrunt: add object storage buckets for registry + artifacts
  3. Create gitlab database + user in existing Postgres
  4. Helm install GitLab CE with external Postgres/Redis, disabled subcharts
  5. DNS: add gitlab A record in fxhnt.ai zone → Tailscale IP
  6. Verify GitLab web UI accessible, create admin account + project

Phase 2: Migrate Repo & Configure CI

  1. git push --mirror from Gitea to GitLab
  2. Deploy GitLab Runner (Helm) with Kubernetes executor targeting ci-build pool
  3. Create .gitlab-ci.yml (translate from .gitea/workflows/ci.yaml)
  4. Configure CI/CD variables (sccache, kubeconfig, S3 creds)
  5. Trigger test pipeline, verify all stages pass
  6. Verify images in GitLab registry, K8s can pull them

Phase 3: Cutover

  1. Update local git remote: origin → ssh://git@gitlab.fxhnt.ai:2222/foxhunt/foxhunt.git
  2. Update K8s manifests: image refs → GitLab registry
  3. Update imagePullSecret → GitLab deploy token
  4. Verify deploy from GitLab pipeline works end-to-end
  5. Run full CI cycle: push → check → test → build → deploy → verify services healthy

Phase 4: Decommission

  1. terragrunt destroy ci-runner module (GP1-S VM gone)
  2. Decommission Gitea VM (DEV1-S)
  3. Delete from repo: infra/modules/ci-runner/, infra/live/production/ci-runner/, .gitea/
  4. Optionally delete Scaleway container registry namespace
  5. Update CLAUDE.md, memory files, infrastructure docs

Rollback

Gitea remains running until Phase 4. If anything fails in Phase 1-3, revert to Gitea with zero data loss. The mirror is one-way push.

Cost Analysis

Resource Current After Migration
Gitea VM (DEV1-S) €7/mo €0 (deleted)
CI runner VM (GP1-S) €44/mo €0 (deleted)
GitLab node (DEV1-L) — €22/mo
ci-build pool (GP1-XS, 0-2) — ~€5-15/mo (usage-based)
Object storage (registry + artifacts) ~€3/mo ~€5/mo
Total CI/Git infra ~€54/mo ~€32-42/mo

Net savings: ~€12-22/mo, plus the operational benefit of K8s-native CI with scale-to-zero.

Decisions Log

Decision Choice Rationale
GitLab location K8s pod (dedicated node pool) Native K8s integration, no extra VM
GitLab node type DEV1-L (8GB) Comfortable for CE, external DB/Redis saves RAM
External Postgres/Redis Yes Reuse existing always-on pods, less resource waste
Build pod pool Dedicated ci-build (GP1-XS, 0-2) Clean separation from services, true scale-to-zero
Docker builds Kaniko No privileged containers, pushes directly to registry
Container registry GitLab built-in (S3-backed) Consolidate, one less external dependency
Migration strategy Push mirror, parallel run Zero-risk cutover with rollback window
Access Tailscale-only Matches existing security posture
DNS gitlab.fxhnt.ai (new subdomain) Separate during migration, optionally alias later
Reference in New Issue View Git Blame Copy Permalink