diff --git a/docs/plans/2026-02-25-iac-pipeline-design.md b/docs/plans/2026-02-25-iac-pipeline-design.md new file mode 100644 index 000000000..3b53e39c4 --- /dev/null +++ b/docs/plans/2026-02-25-iac-pipeline-design.md @@ -0,0 +1,131 @@ +# IaC Pipeline & Tfstate Migration Design + +**Goal:** Migrate Terraform state from nl-ams to fr-par, close IaC drift gaps, and add GitLab CI/CD for infrastructure management. + +**Architecture:** Terragrunt run-all with S3 backend in fr-par. MR-based plan/apply workflow. Weekly drift detection via scheduled pipeline. + +**Tech Stack:** Terragrunt 0.72.6, OpenTofu/Terraform 1.11.2, Scaleway S3, GitLab CI/CD, glab CLI + +--- + +## 1. Tfstate Migration (nl-ams → fr-par) + +1. Create `foxhunt-tfstate` bucket in fr-par via `scw` CLI (bootstrap, not TF-managed) +2. Copy all state files: `aws s3 sync` cross-region using Scaleway S3 endpoints +3. Update `infra/live/production/root.hcl`: + - `region = "fr-par"` + - `endpoints.s3 = "https://s3.fr-par.scw.cloud"` +4. Run `terragrunt init -migrate-state` in each module directory (kapsule, dns, object-storage, registry, block-storage, secrets) +5. Verify with `terragrunt plan` — should show no changes +6. Delete old nl-ams bucket + +**Risk:** Low. State is JSON files being copied. `-migrate-state` handles backend reconfiguration. Old bucket remains as fallback until explicitly deleted. + +## 2. Resource Import — Close IaC Gap + +### Audit Results + +| Resource | Actual State | In Terraform? | Action | +|----------|-------------|---------------|--------| +| `foxhunt-sccache` bucket (fr-par) | exists | No | Add to `object-storage` module + import | +| `foxhunt-ci` registry namespace (fr-par) | exists | No | Add to `registry` module + import | +| `grafana` A record (fxhnt.ai → 100.90.76.85) | exists | No | Add to `dns` module + import | +| `prometheus` A record (fxhnt.ai → 100.90.76.85) | exists | No | Add to `dns` module + import | +| `ci-runner/` live dir | orphan `.terragrunt-cache` | No terragrunt.hcl | Delete | +| `foxhunt-ibkr-*` secrets | exist | Yes (applied) | No action | +| K8s PVCs (dev-home, postgres, etc.) | exist | N/A | Leave as kubectl/Helm managed | +| `foxhunt-tfstate` bucket (fr-par, new) | bootstrap | N/A | Not TF-managed (circular dep) | + +### Module Changes + +**`infra/modules/object-storage/main.tf`** — Add sccache bucket: +```hcl +resource "scaleway_object_bucket" "sccache" { + name = "foxhunt-sccache" + region = var.region + + lifecycle_rule { + enabled = true + prefix = "" + expiration { days = 14 } + } + versioning { enabled = false } +} +``` + +**`infra/modules/registry/main.tf`** — Add foxhunt-ci namespace: +```hcl +resource "scaleway_registry_namespace" "foxhunt_ci" { + name = "foxhunt-ci" + region = var.region + is_public = false +} +``` + +**`infra/modules/dns/main.tf`** — Add grafana + prometheus records: +```hcl +resource "scaleway_domain_record" "grafana" { + dns_zone = var.dns_zone + name = "grafana" + type = "A" + data = var.git_ip + ttl = 300 +} + +resource "scaleway_domain_record" "prometheus" { + dns_zone = var.dns_zone + name = "prometheus" + type = "A" + data = var.git_ip + ttl = 300 +} +``` + +After all imports, `terragrunt run-all plan` should show zero changes. + +## 3. GitLab CI/CD Pipeline for IaC + +### CI Image + +`infra/docker/Dockerfile.infra-runner` — lightweight image with: +- OpenTofu/Terraform 1.11.2 +- Terragrunt 0.72.6 +- `scw` CLI +- `glab` CLI + +Built via Kaniko, pushed to SCW registry (`rg.fr-par.scw.cloud/foxhunt-ci/infra-runner`). + +### Pipeline Jobs + +**`infra-plan`** (MR gate): +- Stage: `check` +- Image: `${REGISTRY}/infra-runner:latest` +- Tags: `kapsule` (gitlab pool) +- Trigger: MR pipelines when `infra/**` changes +- Script: `terragrunt run-all plan` across all modules +- Output: plan summary as job artifact + +**`infra-apply`** (merge to main): +- Stage: `deploy` +- Same image + tags +- Trigger: push to `main` when `infra/**` changed +- Script: `terragrunt run-all apply -auto-approve` +- Environment: `production/infrastructure` + +**`infra-drift-check`** (weekly scheduled): +- Stage: `check` +- Same image + tags +- Trigger: scheduled pipeline (weekly, Sunday night) +- Script: `terragrunt run-all plan -detailed-exitcode` + - Exit 0 = no drift + - Exit 2 = drift → create GitLab issue via `glab issue create` +- No apply, detection + alerting only + +### Credentials + +Jobs use existing CI/CD variables (protected + masked): +- `SCW_ACCESS_KEY`, `SCW_SECRET_KEY`, `SCW_DEFAULT_PROJECT_ID` + +### Separation + +Infra jobs are independent from Rust build pipeline — different triggers (`infra/**`), different image, different pool (gitlab vs gpu-training). No interference.