docs: GitLab CE migration design (Gitea → GitLab on Kapsule)
Covers: in-cluster GitLab CE with external Postgres/Redis, dedicated ci-build node pool (scale-to-zero), Kaniko image builds, GitLab built-in registry, and 4-phase migration plan with rollback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
275
docs/plans/2026-02-24-gitlab-migration-design.md
Normal file
275
docs/plans/2026-02-24-gitlab-migration-design.md
Normal file
@@ -0,0 +1,275 @@
|
||||
# GitLab CE Migration Design
|
||||
|
||||
## Problem
|
||||
|
||||
Gitea + act_runner on a dedicated GP1-S VM cannot spawn ephemeral build pods in Kapsule. The CI runner is a fixed-capacity VM that idles most of the time, requires a custom auto-start watcher, and can't leverage K8s autoscaling. We need CI that scales to zero and integrates natively with our Kubernetes cluster.
|
||||
|
||||
## Solution
|
||||
|
||||
Migrate from Gitea to GitLab CE running as a pod in Kapsule with the GitLab Runner Kubernetes executor. Build pods schedule on a dedicated autoscaling node pool (0-N), giving true scale-to-zero CI.
|
||||
|
||||
## Infrastructure Topology
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Scaleway Kapsule Cluster │
|
||||
│ │
|
||||
│ ┌───────────────┐ ┌─────────────────────┐ │
|
||||
│ │ always-on │ │ gitlab (NEW) │ │
|
||||
│ │ DEV1-M │ │ DEV1-L, fixed 1 │ │
|
||||
│ │ ──────────── │ │ ────────────────── │ │
|
||||
│ │ postgres │ │ GitLab CE (Helm) │ │
|
||||
│ │ redis │ │ Puma, Sidekiq, │ │
|
||||
│ │ questdb │ │ Gitaly, Workhorse, │ │
|
||||
│ └───────┬───────┘ │ Shell, Registry, │ │
|
||||
│ │ │ Runner Manager │ │
|
||||
│ │ ext DB └──────────┬──────────┘ │
|
||||
│ └─────────────────────┘ │
|
||||
│ │
|
||||
│ ┌───────────────┐ ┌─────────────────────┐ │
|
||||
│ │ ci │ │ ci-build (NEW) │ │
|
||||
│ │ GP1-XS, 0-1 │ │ GP1-XS, 0-2 │ │
|
||||
│ │ ──────────── │ │ ────────────────── │ │
|
||||
│ │ api-gateway │ │ GitLab Runner │ │
|
||||
│ │ web-gateway │ │ build pods │ │
|
||||
│ │ trading-svc │ │ (ephemeral) │ │
|
||||
│ │ ...services │ │ scale-to-zero │ │
|
||||
│ └───────────────┘ └─────────────────────┘ │
|
||||
│ │
|
||||
│ ┌───────────────┐ ┌─────────────────────┐ │
|
||||
│ │ gpu-training │ │ gpu-inference │ │
|
||||
│ │ H100, 0-1 │ │ L4, 0-1 │ │
|
||||
│ └───────────────┘ └─────────────────────┘ │
|
||||
└─────────────────────────────────────────────┘
|
||||
|
||||
DNS: gitlab.fxhnt.ai → Tailscale IP (subnet router)
|
||||
Access: Tailscale-only (no public ingress)
|
||||
```
|
||||
|
||||
## New Node Pools
|
||||
|
||||
### `gitlab` pool
|
||||
- **Type**: DEV1-L (8GB RAM, 4 vCPU)
|
||||
- **Autoscale**: Fixed 1 node (always on)
|
||||
- **Taint**: `gitlab=true:NoSchedule`
|
||||
- **Purpose**: GitLab CE core components + Runner manager pod
|
||||
- **Cost**: ~€0.03/hr (€22/mo)
|
||||
|
||||
### `ci-build` pool
|
||||
- **Type**: GP1-XS (16GB RAM, 8 vCPU)
|
||||
- **Autoscale**: 0-2 nodes
|
||||
- **Taint**: `ci-build=true:NoSchedule`
|
||||
- **Purpose**: Ephemeral GitLab Runner build pods
|
||||
- **Cost**: €0.00 idle, ~€0.08/hr per node during builds
|
||||
- **Scale-down delay**: 10 minutes (matches existing autoscaler config)
|
||||
|
||||
## GitLab CE Deployment
|
||||
|
||||
### Helm Chart: `gitlab/gitlab`
|
||||
|
||||
**Enabled subcharts:**
|
||||
- Puma (webserver): 2 workers
|
||||
- Sidekiq: 1 replica, concurrency 10
|
||||
- Gitaly: local storage, 50Gi PVC
|
||||
- Workhorse: bundled
|
||||
- Shell: git-over-SSH on port 2222
|
||||
- Registry: built-in, S3-backed
|
||||
|
||||
**Disabled subcharts:**
|
||||
- PostgreSQL — external (existing always-on pod)
|
||||
- Redis — external (existing always-on pod)
|
||||
- MinIO — replaced by Scaleway Object Storage
|
||||
- Prometheus/Grafana — not needed
|
||||
- cert-manager — Tailscale-only access
|
||||
- nginx-ingress — Tailscale-only access
|
||||
|
||||
### External Database Configuration
|
||||
|
||||
**PostgreSQL** (existing TimescaleDB on always-on pool):
|
||||
```sql
|
||||
CREATE DATABASE gitlab;
|
||||
CREATE USER gitlab WITH PASSWORD '<generated-by-terraform>';
|
||||
GRANT ALL ON DATABASE gitlab TO gitlab;
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
CREATE EXTENSION IF NOT EXISTS btree_gist;
|
||||
```
|
||||
|
||||
**Redis** (existing Redis on always-on pool):
|
||||
- GitLab configured to use DB indexes 8-15 (avoids collision with application data on default index 0)
|
||||
|
||||
### Storage
|
||||
|
||||
| Data | Backend | Details |
|
||||
|------|---------|---------|
|
||||
| Git repos (Gitaly) | PVC | 50Gi on gitlab node |
|
||||
| Container registry blobs | Scaleway Object Storage | Bucket: `foxhunt-gitlab-registry` |
|
||||
| CI artifacts & uploads | Scaleway Object Storage | Bucket: `foxhunt-gitlab-artifacts` |
|
||||
| CI cache (sccache) | Scaleway Object Storage | Existing bucket, 30-day lifecycle |
|
||||
|
||||
### Access
|
||||
|
||||
- **Web UI**: `https://gitlab.fxhnt.ai` via Tailscale
|
||||
- **SSH**: `ssh://git@gitlab.fxhnt.ai:2222/foxhunt/foxhunt.git`
|
||||
- **Registry**: `gitlab.fxhnt.ai:5050/foxhunt/foxhunt/<service>`
|
||||
- **DNS**: A record `gitlab.fxhnt.ai` → Tailscale IP (Scaleway DNS zone `fxhnt.ai`)
|
||||
- **TLS**: Self-signed or Tailscale HTTPS certs (no public CA needed)
|
||||
|
||||
## GitLab Runner Configuration
|
||||
|
||||
### Runner Manager
|
||||
- Deployed via `gitlab-runner` Helm chart on `gitlab` node pool
|
||||
- **Executor**: Kubernetes
|
||||
- **Concurrent**: 4 (can run 4 build pods in parallel)
|
||||
- **Pod scheduling**: nodeSelector `ci-build=true`, toleration for taint
|
||||
|
||||
### Build Pod Spec
|
||||
```
|
||||
┌─ Build Pod (ephemeral, ci-build pool) ─────────────────┐
|
||||
│ build container: rust:1.89-slim + protoc + sccache │
|
||||
│ helper container: gitlab-runner-helper (clone, upload) │
|
||||
│ sidecar: kaniko (docker image builds) │
|
||||
└────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Docker Image Builds: Kaniko
|
||||
- No privileged containers needed (unlike Docker-in-Docker)
|
||||
- Kaniko builds images in userspace, pushes directly to GitLab registry
|
||||
- Replaces `docker build && docker push` in current pipeline
|
||||
|
||||
### Scale-to-Zero Flow
|
||||
1. Push to main → GitLab creates pipeline
|
||||
2. Runner manager requests build pod → K8s autoscaler sees pending pod
|
||||
3. Autoscaler provisions ci-build node (GP1-XS) — ~2min startup
|
||||
4. Build pods execute (check, test, build images, deploy)
|
||||
5. Pods terminate → after 10min idle → autoscaler removes ci-build node
|
||||
6. Cost returns to €0.00 for CI
|
||||
|
||||
## CI Pipeline Translation
|
||||
|
||||
### Current: `.gitea/workflows/ci.yaml`
|
||||
```
|
||||
check → test → build-images (matrix: 6 services) + build-web-gateway + build-training → deploy
|
||||
```
|
||||
|
||||
### New: `.gitlab-ci.yml`
|
||||
|
||||
| Stage | Jobs | Notes |
|
||||
|-------|------|-------|
|
||||
| `check` | `cargo check --workspace`, `cargo clippy --workspace -- -D warnings` | sccache enabled |
|
||||
| `test` | `cargo test --workspace --lib` | sccache enabled |
|
||||
| `build` | 8 parallel Kaniko jobs (6 services + web-gateway + training) | Push to GitLab registry |
|
||||
| `deploy` | `kubectl set image` for each deployment | kubeconfig from CI variable |
|
||||
|
||||
**Environment variables (CI/CD settings):**
|
||||
- `SCCACHE_BUCKET`, `SCCACHE_ENDPOINT` — existing S3 cache
|
||||
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` — Scaleway S3 creds
|
||||
- `KUBECONFIG` — base64-encoded cluster config (file variable)
|
||||
- `SQLX_OFFLINE=true`
|
||||
|
||||
**Registry change impact:**
|
||||
- K8s manifests: `image: rg.fr-par.scw.cloud/foxhunt/<svc>` → `image: gitlab.fxhnt.ai:5050/foxhunt/foxhunt/<svc>`
|
||||
- imagePullSecret: Scaleway registry creds → GitLab deploy token
|
||||
- CI pipeline: Kaniko pushes natively to GitLab registry (no separate `docker login`)
|
||||
|
||||
## Terragrunt Changes
|
||||
|
||||
### New/Modified Modules
|
||||
|
||||
**`infra/modules/kapsule/main.tf`** — add two node pools:
|
||||
```hcl
|
||||
resource "scaleway_k8s_pool" "gitlab" {
|
||||
cluster_id = scaleway_k8s_cluster.foxhunt.id
|
||||
name = "gitlab"
|
||||
node_type = "DEV1-L"
|
||||
size = 1
|
||||
autoscaling = false
|
||||
tags = ["gitlab"]
|
||||
}
|
||||
|
||||
resource "scaleway_k8s_pool" "ci_build" {
|
||||
cluster_id = scaleway_k8s_cluster.foxhunt.id
|
||||
name = "ci-build"
|
||||
node_type = "GP1-XS"
|
||||
size = 0
|
||||
min_size = 0
|
||||
max_size = 2
|
||||
autoscaling = true
|
||||
autohealing = true
|
||||
tags = ["ci-build"]
|
||||
}
|
||||
```
|
||||
|
||||
**`infra/modules/object-storage/main.tf`** — add GitLab buckets:
|
||||
- `foxhunt-gitlab-registry`
|
||||
- `foxhunt-gitlab-artifacts`
|
||||
|
||||
### Deleted Modules
|
||||
- `infra/modules/ci-runner/` — entire module (VM, cloud-init, auto-start watcher)
|
||||
- `infra/live/production/ci-runner/` — terragrunt config
|
||||
|
||||
### Deleted Infrastructure
|
||||
- GP1-S CI runner VM (`vm-fxhnt-ci`)
|
||||
- Auto-start watcher systemd units on Gitea server
|
||||
- DEV1-S Gitea VM (`vm-fxhnt-git`) — after successful cutover
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 1: Stand Up GitLab (parallel with Gitea)
|
||||
1. Terragrunt: add `gitlab` + `ci-build` node pools to kapsule module
|
||||
2. Terragrunt: add object storage buckets for registry + artifacts
|
||||
3. Create `gitlab` database + user in existing Postgres
|
||||
4. Helm install GitLab CE with external Postgres/Redis, disabled subcharts
|
||||
5. DNS: add `gitlab` A record in `fxhnt.ai` zone → Tailscale IP
|
||||
6. Verify GitLab web UI accessible, create admin account + project
|
||||
|
||||
### Phase 2: Migrate Repo & Configure CI
|
||||
1. `git push --mirror` from Gitea to GitLab
|
||||
2. Deploy GitLab Runner (Helm) with Kubernetes executor targeting ci-build pool
|
||||
3. Create `.gitlab-ci.yml` (translate from `.gitea/workflows/ci.yaml`)
|
||||
4. Configure CI/CD variables (sccache, kubeconfig, S3 creds)
|
||||
5. Trigger test pipeline, verify all stages pass
|
||||
6. Verify images in GitLab registry, K8s can pull them
|
||||
|
||||
### Phase 3: Cutover
|
||||
1. Update local git remote: `origin` → `ssh://git@gitlab.fxhnt.ai:2222/foxhunt/foxhunt.git`
|
||||
2. Update K8s manifests: image refs → GitLab registry
|
||||
3. Update imagePullSecret → GitLab deploy token
|
||||
4. Verify deploy from GitLab pipeline works end-to-end
|
||||
5. Run full CI cycle: push → check → test → build → deploy → verify services healthy
|
||||
|
||||
### Phase 4: Decommission
|
||||
1. `terragrunt destroy` ci-runner module (GP1-S VM gone)
|
||||
2. Decommission Gitea VM (DEV1-S)
|
||||
3. Delete from repo: `infra/modules/ci-runner/`, `infra/live/production/ci-runner/`, `.gitea/`
|
||||
4. Optionally delete Scaleway container registry namespace
|
||||
5. Update `CLAUDE.md`, memory files, infrastructure docs
|
||||
|
||||
### Rollback
|
||||
Gitea remains running until Phase 4. If anything fails in Phase 1-3, revert to Gitea with zero data loss. The mirror is one-way push.
|
||||
|
||||
## Cost Analysis
|
||||
|
||||
| Resource | Current | After Migration |
|
||||
|----------|---------|-----------------|
|
||||
| Gitea VM (DEV1-S) | €7/mo | €0 (deleted) |
|
||||
| CI runner VM (GP1-S) | €44/mo | €0 (deleted) |
|
||||
| GitLab node (DEV1-L) | — | €22/mo |
|
||||
| ci-build pool (GP1-XS, 0-2) | — | ~€5-15/mo (usage-based) |
|
||||
| Object storage (registry + artifacts) | ~€3/mo | ~€5/mo |
|
||||
| **Total CI/Git infra** | **~€54/mo** | **~€32-42/mo** |
|
||||
|
||||
Net savings: ~€12-22/mo, plus the operational benefit of K8s-native CI with scale-to-zero.
|
||||
|
||||
## Decisions Log
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| GitLab location | K8s pod (dedicated node pool) | Native K8s integration, no extra VM |
|
||||
| GitLab node type | DEV1-L (8GB) | Comfortable for CE, external DB/Redis saves RAM |
|
||||
| External Postgres/Redis | Yes | Reuse existing always-on pods, less resource waste |
|
||||
| Build pod pool | Dedicated `ci-build` (GP1-XS, 0-2) | Clean separation from services, true scale-to-zero |
|
||||
| Docker builds | Kaniko | No privileged containers, pushes directly to registry |
|
||||
| Container registry | GitLab built-in (S3-backed) | Consolidate, one less external dependency |
|
||||
| Migration strategy | Push mirror, parallel run | Zero-risk cutover with rollback window |
|
||||
| Access | Tailscale-only | Matches existing security posture |
|
||||
| DNS | `gitlab.fxhnt.ai` (new subdomain) | Separate during migration, optionally alias later |
|
||||
Reference in New Issue
Block a user