docs: spec — migrate Terraform state GitLab -> Scaleway Object Storage (Phase 2A)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/superpowers/specs/2026-06-21-tfstate-to-scaleway-design.md

@@ -0,0 +1,94 @@
+# Migrate Terraform State to Scaleway Object Storage — Design (Phase 2A)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** foxhunt (IaC current home; relocation to fxhnt is Phase 2D)
+**Part of:** Phase 2 (GitLab → Gitea consolidation). Sub-projects: **2A state migration (this)**, 2B Gitea
+stand-up + repo migration + Argo wiring, 2C GitLab decommission, 2D IaC relocation into fxhnt.
+
+## Motivation
+
+Terraform state for the whole production cluster currently lives in the **self-hosted GitLab** http
+backend (project ID=1), whose object data sits in **in-cluster MinIO**. That is a chicken-and-egg
+fragility: the state describing the cluster lives *inside* the cluster it manages. GitLab is also slated
+for decommission (Phase 2C, replaced by lightweight Gitea). Moving state to **external Scaleway Object
+Storage** decouples it from both GitLab and the cluster — a prerequisite for removing GitLab and a
+durability win on its own (state survives cluster loss).
+
+## Decision (keystone)
+
+**Target backend: Scaleway Object Storage (native S3), bucket `foxhunt-tfstate`** (already exists,
+fr-par, created ~3 months ago but never wired). Reuse it (no rename — avoids a needless create; the
+`foxhunt-` prefix is cosmetic and harmless). Locking via **OpenTofu native S3 lockfile** (`use_lockfile`,
+OpenTofu v1.11.5 present) — no DynamoDB.
+
+## Scope — what changes
+
+1. **`infra/live/root.hcl`** — replace the single `remote_state` block: `backend = "http"` (GitLab) →
+   `backend = "s3"` (Scaleway). New config:
+   ```hcl
+   remote_state {
+     backend = "s3"
+     generate = { path = "backend.tf", if_exists = "overwrite" }
+     config = {
+       bucket = "foxhunt-tfstate"
+       key    = "${path_relative_to_include()}/terraform.tfstate"
+       region = "fr-par"
+       endpoints = { s3 = "https://s3.fr-par.scw.cloud" }
+       skip_credentials_validation = true
+       skip_region_validation      = true
+       skip_requesting_account_id  = true
+       skip_metadata_api_check     = true
+       use_lockfile                = true
+     }
+   }
+   ```
+   Credentials: Scaleway access/secret keys exported as `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`
+   (the s3 backend reads AWS-style env vars) from the existing `scaleway-credentials` k8s secret.
+2. **State migration, per module, sequential** (`kapsule`, `public-gateway`, `dns`, `block-storage`):
+   - Back up first: `terragrunt state pull > /tmp/tfstate-backup-<module>.tfstate` (from GitLab, BEFORE
+     changing the backend for that module).
+   - `terragrunt init -migrate-state -force-copy` → copies GitLab state → Scaleway S3.
+   - `terragrunt plan` → **must report `No changes`** before proceeding to the next module.
+3. **Lock smoke test** (once, on one module): confirm `use_lockfile` works against Scaleway — a lock
+   object is created during apply/plan-with-lock and released after (verify no orphan `.tflock` left).
+
+## Migration order & gates
+
+`kapsule` → `public-gateway` → `dns` → `block-storage`. After EACH: backup taken, `-migrate-state`
+succeeded, `plan = No changes` confirmed. **STOP and revert if any module's post-migration plan shows
+drift** (revert = restore `http` backend in root.hcl + `terragrunt init` to re-attach GitLab state; the
+GitLab copy is never deleted in 2A).
+
+## Safety / reversibility
+
+- **Old GitLab state is NOT deleted in 2A** — it stays as a live fallback until GitLab is removed in 2C.
+  So the entire migration is reversible by flipping `root.hcl` back to the http backend.
+- `/tmp` state backups taken before each module (belt-and-suspenders; never committed — contains
+  resource IDs, not secrets, but treat as sensitive).
+- One module at a time with a `plan = No changes` gate prevents a bad backend config from cascading.
+- `endpoints.s3` (not deprecated top-level `endpoint`) + the four `skip_*` flags are required for the
+  AWS s3 backend to talk to Scaleway's S3-compatible API without AWS-specific preflight calls.
+
+## Risks
+
+- **Scaleway S3 quirks:** some provider/backend versions need `skip_s3_checksum = true` for Scaleway. If
+  `-migrate-state` errors on checksum, add it. (Documented as a known fallback in the plan.)
+- **Lock semantics:** Scaleway Object Storage must honor conditional-write for `use_lockfile`. The lock
+  smoke test catches this; fallback is to run with locking disabled (single maintainer) if unsupported.
+- **Credential confusion:** the s3 backend reads `AWS_*` env vars, NOT `SCW_*`. Plan must export both
+  (SCW_* for the provider, AWS_* for the backend) — a common foot-gun.
+
+## Out of scope (later sub-projects)
+
+Gitea stand-up + repo migration + Argo wiring (2B); GitLab helm removal + GitLab state/bucket cleanup
+(2C); `git mv` of IaC from foxhunt → fxhnt + path updates (2D). Files stay in foxhunt for 2A; only the
+backend changes.
+
+## Acceptance criteria
+
+- `infra/live/root.hcl` uses the Scaleway s3 backend; committed.
+- All 4 modules migrated: each shows `terragrunt plan = No changes` against the new backend.
+- State objects present in `foxhunt-tfstate` at `<module>/terraform.tfstate` (verified via `scw`/`mc`).
+- Lock smoke test passes (lock acquired + released, no orphan lockfile).
+- GitLab http state still intact (untouched fallback) — not yet deleted.