Files

jgrusewski b21d9b671d docs(plan): SSH on :22 only, retire :2222, update remotes (Phase 2B refinement)

2026-06-21 15:49:54 +02:00

19 KiB

Raw Permalink Blame History

Gitea Replaces GitLab + Cockpit Registry to Scaleway — Implementation Plan (Phase 2B)

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints). Do NOT run subagent-driven — this touches live git access, the container registry, and the cockpit deploy; each cutover step needs a human-confirmed gate. Steps use checkbox (- [ ]) syntax.

Goal: Stand up Gitea (chart + existing Postgres) as the git host behind the unchanged git.fxhnt.ai, migrate both repos, and move the cockpit image to Scaleway Container Registry — so GitLab can be removed in 2C.

Architecture: Gitea comes up internal-only; repos migrate + validate over a port-forward; the cockpit image moves to Scaleway registry and is build+rollout tested; THEN git.fxhnt.ai (HTTPS + SSH :2222) cuts over to Gitea in one switch; finally a Gitea webhook → Argo Events auto-deploys the cockpit. GitLab stays running (unaddressed) as the rollback net until 2C.

Tech Stack: Gitea Helm chart (gitea-charts/gitea), existing in-cluster PostgreSQL, nginx+socat Tailscale proxy, Scaleway Container Registry, Argo Events, kaniko.

Spec: docs/superpowers/specs/2026-06-21-gitea-replace-gitlab-design.md

TWO REPOS: infra changes → foxhunt (~/Work/foxhunt); cockpit build/deploy → fxhnt (~/Work/fxhnt). Each gets its own branch + commits.

Shared env

export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2

Task 0: Branches

Step 1

cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/gitea-replace-gitlab
cd /home/jgrusewski/Work/fxhnt   && git checkout -b chore/gitea-cockpit-registry

Task 1: Postgres `gitea` DB + secrets

Files: none committed (cluster secrets). Record secret names only.

Step 1: Read existing postgres superuser creds

PGU=$(kubectl get deploy postgres -n foxhunt -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="POSTGRES_USER")].value}')
PGPOD=$(kubectl get pod -n foxhunt -l app=postgres -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || kubectl get pod -n foxhunt | grep '^postgres-' | grep -v backup | awk '{print $1}' | head -1)
echo "postgres user=$PGU pod=$PGPOD"

Expected: a username + a running pod name. (If POSTGRES_USER is via secretRef not value, read it from the referenced secret instead.)

Step 2: Generate a Gitea DB password + create role/db

GITEA_DB_PW=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
kubectl exec -n foxhunt "$PGPOD" -- psql -U "$PGU" -v ON_ERROR_STOP=1 -c \
  "CREATE ROLE gitea LOGIN PASSWORD '$GITEA_DB_PW';" -c \
  "CREATE DATABASE gitea OWNER gitea ENCODING 'UTF8';" 2>&1 | grep -iE "CREATE|already exists|Error"
echo "$GITEA_DB_PW" # capture for the next step

Expected: CREATE ROLE + CREATE DATABASE (or already exists if re-run).

Step 3: Create the k8s secrets Gitea needs (DB creds + admin)

GITEA_ADMIN_PW=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
kubectl create secret generic gitea-db -n foxhunt \
  --from-literal=password="$GITEA_DB_PW" --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic gitea-admin -n foxhunt \
  --from-literal=username=gitadmin --from-literal=password="$GITEA_ADMIN_PW" \
  --from-literal=email=jeroen@bizworx.nl --dry-run=client -o yaml | kubectl apply -f -
echo "ADMIN PW (save in your password manager): $GITEA_ADMIN_PW"

Expected: both secrets created/configured. Save the admin password — needed for Gitea login.

Task 2: Gitea Helm values + install (internal-only)

Files: Create infra/k8s/gitea/values.yaml (foxhunt repo).

Step 1: Add the Gitea Helm repo

helm repo add gitea-charts https://dl.gitea.com/charts/ && helm repo update gitea-charts
helm search repo gitea-charts/gitea --versions | head -3

Expected: lists chart versions (use the latest stable in Step 3).

Step 2: Write infra/k8s/gitea/values.yaml

# Gitea — lightweight git host replacing GitLab (Phase 2B). External Postgres (existing in-cluster
# `postgres`), no bundled DB/redis/memcached, Actions off (Argo does CI). Internal-only until cutover.
replicaCount: 1
image:
  rootless: true

# Disable all bundled subcharts — reuse the existing in-cluster postgres
postgresql:
  enabled: false
postgresql-ha:
  enabled: false
redis-cluster:
  enabled: false
redis:
  enabled: false

persistence:
  enabled: true
  size: 5Gi
  storageClass: sbs-default-retain

resources:
  requests: { cpu: 100m, memory: 128Mi }
  limits:   { cpu: "1",  memory: 512Mi }

service:
  http: { type: ClusterIP, port: 3000 }
  ssh:  { type: ClusterIP, port: 22 }

actions:
  enabled: false

gitea:
  admin:
    existingSecret: gitea-admin
  config:
    server:
      ROOT_URL: https://git.fxhnt.ai/
      DOMAIN: git.fxhnt.ai
      SSH_DOMAIN: git.fxhnt.ai
      SSH_PORT: "22"            # clean git@git.fxhnt.ai clone URLs. Port 22 is free (nothing host-SSHes
                               # the git node); :2222 retired, local remotes updated in Task 5 Step 2b.
      DISABLE_SSH: "false"
    database:
      DB_TYPE: postgres
      HOST: postgres.foxhunt.svc.cluster.local:5432
      NAME: gitea
      USER: gitea
    service:
      DISABLE_REGISTRATION: "true"
    cache:
      ADAPTER: memory
  additionalConfigFromEnvs:
    - name: GITEA__database__PASSWD
      valueFrom:
        secretKeyRef: { name: gitea-db, key: password }

Step 3: Install (internal-only — no public hostname yet)

cd /home/jgrusewski/Work/foxhunt
helm install gitea gitea-charts/gitea -n foxhunt -f infra/k8s/gitea/values.yaml --wait --timeout 5m 2>&1 | tail -5

Expected: STATUS: deployed. (If --wait times out, continue; verify pod in Step 4.)

Step 4: Verify Gitea is healthy

kubectl get pods -n foxhunt | grep gitea
kubectl run gitea-probe --rm -i --restart=Never -n foxhunt --image=curlimages/curl -- \
  curl -s http://gitea-http.foxhunt.svc.cluster.local:3000/api/healthz

Expected: a JSON health blob with "status": "pass". Pod Running.

Step 5: Commit values (foxhunt repo)

cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/gitea/values.yaml
git commit -m "feat(infra): Gitea Helm values (external postgres, internal-only) — Phase 2B"

Task 3: Migrate both repos via port-forward + validate

Files: none committed (git data operations).

Step 1: Port-forward Gitea + create the repos via API

kubectl port-forward -n foxhunt svc/gitea-http 3000:3000 >/tmp/gitea-pf.log 2>&1 & echo $! >/tmp/gitea-pf.pid
sleep 3
GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
for r in fxhnt foxhunt; do
  curl -s -u "$GA:$GP" -X POST http://localhost:3000/api/v1/user/repos \
    -H 'Content-Type: application/json' -d "{\"name\":\"$r\",\"private\":true}" -o /dev/null -w "$r: %{http_code}\n"
done

Expected: fxhnt: 201 and foxhunt: 201.

Step 2: Mirror-push both repos from local clones

GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
( cd /home/jgrusewski/Work/fxhnt   && git push --mirror "http://$GA:$GP@localhost:3000/$GA/fxhnt.git" 2>&1 | tail -2 )
( cd /home/jgrusewski/Work/foxhunt && git push --mirror "http://$GA:$GP@localhost:3000/$GA/foxhunt.git" 2>&1 | tail -2 )

Expected: both report refs pushed (no error). foxhunt is 876 MB — may take a minute.

Step 3: VALIDATE — clone back + diff top SHAs vs GitLab

GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
for r in fxhnt foxhunt; do
  gitea_sha=$(git ls-remote "http://$GA:$GP@localhost:3000/$GA/$r.git" HEAD | awk '{print $1}')
  gitlab_sha=$(cd /home/jgrusewski/Work/$r && git ls-remote origin HEAD | awk '{print $1}')
  [ "$gitea_sha" = "$gitlab_sha" ] && echo "$r: MATCH ($gitea_sha)" || echo "$r: MISMATCH gitea=$gitea_sha gitlab=$gitlab_sha"
done

Expected: fxhnt: MATCH and foxhunt: MATCH. STOP if any MISMATCH.

Step 4: Mark foxhunt archived (read-only)

GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
curl -s -u "$GA:$GP" -X PATCH "http://localhost:3000/api/v1/repos/$GA/foxhunt" \
  -H 'Content-Type: application/json' -d '{"archived":true}' -o /dev/null -w "archive foxhunt: %{http_code}\n"
kill $(cat /tmp/gitea-pf.pid) 2>/dev/null

Expected: archive foxhunt: 200.

Task 4: Move cockpit image to Scaleway Container Registry

Files: Modify ~/Work/fxhnt/infra/argo/cockpit-build-deploy.yaml, ~/Work/fxhnt/infra/k8s/orchestration/dagster.yaml (+ dashboard manifest if separate).

Step 1: Create the Scaleway registry pull/push secret

kubectl create secret docker-registry scw-registry -n foxhunt \
  --docker-server=rg.fr-par.scw.cloud \
  --docker-username=nologin --docker-password="$SCW_SECRET_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

Expected: secret created/configured. (Scaleway registry auth: username nologin, password = a Scaleway secret key.)

Step 2: Repoint kaniko build → Scaleway registry

In ~/Work/fxhnt/infra/argo/cockpit-build-deploy.yaml, change the kaniko args:

              --destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest \
              --cache-repo=rg.fr-par.scw.cloud/bizworx/cache \

Remove the --insecure-registry=…gitlab-registry…:5000 and --skip-tls-verify-registry=… lines (Scaleway is TLS). Mount the scw-registry dockerconfig for kaniko (replace the gitlab-registry dockerconfig volume): set the kaniko DOCKER_CONFIG/volume to the scw-registry secret at /kaniko/.docker/config.json.

Step 3: Repoint the deploy image refs

In ~/Work/fxhnt/infra/k8s/orchestration/dagster.yaml (and the dashboard manifest), change every image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/fxhnt-cockpit:latest → image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest, and add under each pod spec:

      imagePullSecrets:
        - name: scw-registry

Step 4: Run one cockpit build → Scaleway + verify the image exists

cd /home/jgrusewski/Work/fxhnt && ./scripts/argo-deploy-cockpit.sh 2>&1 | tail -8
# verify the pushed image
scw registry image list namespace-id=6561a90a-f4ba-4f44-b43f-be6e27cbdfca 2>/dev/null | grep -i fxhnt-cockpit || echo "(verify via Scaleway console)"

Expected: workflow Succeeded; fxhnt-cockpit image listed in the bizworx namespace.

Step 5: Verify a rollout pulls from Scaleway

kubectl rollout restart deploy/dagster deploy/fxhnt-dashboard -n foxhunt 2>/dev/null
kubectl rollout status deploy/dagster -n foxhunt --timeout=180s
kubectl get pods -n foxhunt -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | grep -i "rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit" | head

Expected: rollout succeeds; pods now reference the rg.fr-par.scw.cloud/bizworx/... image (no ImagePullBackOff).

Step 6: Commit (fxhnt repo)

cd /home/jgrusewski/Work/fxhnt
git add infra/argo/cockpit-build-deploy.yaml infra/k8s/orchestration/dagster.yaml
git commit -m "feat(infra): cockpit image -> Scaleway Container Registry (Phase 2B)"

Task 5: Cut `git.fxhnt.ai` + SSH `:2222` over to Gitea

Files: Modify infra/k8s/gitlab/tailscale-proxy.yaml (foxhunt repo). CHECKPOINT before applying.

Step 1: Repoint nginx git.fxhnt.ai:443 → Gitea

In infra/k8s/gitlab/tailscale-proxy.yaml, in the server_name git.fxhnt.ai; (listen 443) block, change:

            proxy_pass http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181;

to:

            proxy_pass http://gitea-http.foxhunt.svc.cluster.local:3000;

Delete the entire listen 5050 ssl; server_name git.fxhnt.ai; registry block (Scaleway registry is external — not proxied).

Step 2: Repoint the socat SSH proxy to Gitea SSH on :22 (drop :2222)

Nothing host-SSHes the git node, so use the standard port 22 and retire :2222 entirely (clean git@git.fxhnt.ai URLs). In the ssh-proxy container args, change the listen port to 22 and the target to Gitea:

            - "TCP-LISTEN:22,fork,reuseaddr,sndbuf=1048576,rcvbuf=1048576"
            - "TCP:gitea-ssh.foxhunt.svc.cluster.local:22,sndbuf=1048576,rcvbuf=1048576"

Update the container ports: entry from containerPort: 2222 → containerPort: 22. Ensure the Tailscale node/proxy advertises port 22 (update the tailnet sidecar's exposed ports / serve config: 22, replacing 2222).

Step 2b: Update local git remotes from :2222 → default (port 22)

cd /home/jgrusewski/Work/foxhunt && git remote set-url origin ssh://git@git.fxhnt.ai/gitadmin/foxhunt.git
cd /home/jgrusewski/Work/fxhnt   && git remote set-url origin ssh://git@git.fxhnt.ai/gitadmin/fxhnt.git
cd /home/jgrusewski/Work/foxhunt && git remote -v | grep fetch

Expected: remotes now show ssh://git@git.fxhnt.ai/... (no :2222).

Step 3: Apply + restart the proxy

cd /home/jgrusewski/Work/foxhunt
kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml
kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s

Expected: rollout complete.

Step 4: GATE — verify git.fxhnt.ai serves Gitea

curl -sk https://git.fxhnt.ai/api/healthz | head -c 200; echo
echo "--- SSH on :22 (git@git.fxhnt.ai) ---"; git ls-remote ssh://git@git.fxhnt.ai/gitadmin/fxhnt.git HEAD 2>&1 | head -1

Expected: Gitea healthz JSON; the :22 ssh ls-remote returns the HEAD sha (proves Gitea web + SSH on the canonical name). STOP + rollback (revert this file, re-apply) if either fails.

Step 5: Commit (foxhunt repo)

cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/gitlab/tailscale-proxy.yaml
git commit -m "feat(infra): cut git.fxhnt.ai + SSH :2222 over to Gitea; drop :5050 registry (Phase 2B)"

Task 6: Wire Gitea push webhook → Argo Events auto-deploy

Files: Create infra/k8s/argo/events/gitea-push-eventsource.yaml + infra/k8s/argo/events/gitea-deploy-sensor.yaml (foxhunt repo).

Step 1: Create the webhook eventsource

Create infra/k8s/argo/events/gitea-push-eventsource.yaml:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: gitea-push
  namespace: foxhunt
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  webhook:
    fxhnt-push:
      port: "12000"
      endpoint: /push
      method: POST

Step 2: Create the sensor that submits the cockpit workflow

Create infra/k8s/argo/events/gitea-deploy-sensor.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: gitea-deploy
  namespace: foxhunt
spec:
  dependencies:
    - name: push
      eventSourceName: gitea-push
      eventName: fxhnt-push
      filters:
        data:
          - path: body.ref
            type: string
            value: ["refs/heads/main", "refs/heads/master"]
  triggers:
    - template:
        name: submit-cockpit
        argoWorkflow:
          operation: submit
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: fxhnt-cockpit-
                namespace: foxhunt
              spec:
                workflowTemplateRef:
                  name: fxhnt-cockpit

Step 3: Apply + register the webhook in Gitea

cd /home/jgrusewski/Work/foxhunt
kubectl apply -f infra/k8s/argo/events/gitea-push-eventsource.yaml -f infra/k8s/argo/events/gitea-deploy-sensor.yaml
kubectl get pods -n foxhunt | grep -iE "gitea-push|gitea-deploy"
# register webhook (in-cluster URL) on the fxhnt repo
GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
kubectl port-forward -n foxhunt svc/gitea-http 3000:3000 >/tmp/gitea-pf.log 2>&1 & echo $! >/tmp/gitea-pf.pid; sleep 3
curl -s -u "$GA:$GP" -X POST "http://localhost:3000/api/v1/repos/$GA/fxhnt/hooks" -H 'Content-Type: application/json' -d '{
  "type":"gitea","active":true,"events":["push"],
  "config":{"url":"http://gitea-push-eventsource-svc.foxhunt.svc.cluster.local:12000/push","content_type":"json"}
}' -o /dev/null -w "hook: %{http_code}\n"
kill $(cat /tmp/gitea-pf.pid) 2>/dev/null

Expected: eventsource + sensor pods Running; hook: 201.

Step 4: GATE — test push triggers a deploy

cd /home/jgrusewski/Work/fxhnt
git commit --allow-empty -m "test: trigger Gitea->Argo cockpit deploy"
git push origin main   # origin already = git.fxhnt.ai (now Gitea)
sleep 10
kubectl get wf -n foxhunt --sort-by=.metadata.creationTimestamp 2>/dev/null | grep fxhnt-cockpit | tail -2

Expected: a new fxhnt-cockpit-* workflow appears (triggered by the push). If none fires, check the eventsource pod logs; fallback is the manual ./scripts/argo-deploy-cockpit.sh (still works).

Step 5: Commit (foxhunt repo)

cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/argo/events/gitea-push-eventsource.yaml infra/k8s/argo/events/gitea-deploy-sensor.yaml
git commit -m "feat(infra): Gitea push webhook -> Argo Events cockpit auto-deploy (Phase 2B)"

Task 7: Final verification

Step 1: Footprint + health summary

echo "=== Gitea (new) ==="; kubectl top pods -n foxhunt 2>/dev/null | grep -i gitea
echo "=== cockpit healthy on Scaleway image ==="; curl -sk -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
echo "=== git.fxhnt.ai serves Gitea ==="; curl -sk https://git.fxhnt.ai/api/healthz | head -c 80; echo
echo "=== GitLab still running (fallback for 2C) ==="; kubectl get pods -n foxhunt | grep -c gitlab | xargs echo "gitlab pods:"

Expected: Gitea ≤~300 MiB; dashboard 200; git.fxhnt.ai healthz pass; GitLab pods still present.

Rollback

Cutover (Task 5): git checkout infra/k8s/gitlab/tailscale-proxy.yaml + kubectl apply + rollout restart → git.fxhnt.ai/SSH back to GitLab.
Registry (Task 4): revert the fxhnt manifests + rollout restart → pods pull the GitLab image again (still present until 2C).
Gitea: helm uninstall gitea -n foxhunt; drop the gitea DB. GitLab was never touched.

Acceptance criteria (from spec)

Gitea running on existing postgres, healthz pass, ≤~300 MiB. ✅ Task 2, Task 7
Both repos in Gitea, SHAs match GitLab, foxhunt archived. ✅ Task 3
Cockpit builds to + deploys from Scaleway registry; dashboard 200. ✅ Task 4, Task 7
git.fxhnt.ai (HTTPS + SSH :2222) serves Gitea; remotes unchanged. ✅ Task 5
Push to fxhnt main auto-triggers a cockpit deploy. ✅ Task 6
GitLab still running as fallback. ✅ Task 7

19 KiB Raw Permalink Blame History