docs: IaC pipeline & tfstate migration design

Covers tfstate migration nl-ams→fr-par, resource import for
drift gaps (sccache bucket, foxhunt-ci registry, DNS records),
and GitLab CI/CD pipeline (plan on MR, apply on merge, weekly
drift detection).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-02-25 21:41:45 +01:00
parent f3619ebcec
commit 7d80a0db48

View File

@@ -0,0 +1,131 @@
# IaC Pipeline & Tfstate Migration Design
**Goal:** Migrate Terraform state from nl-ams to fr-par, close IaC drift gaps, and add GitLab CI/CD for infrastructure management.
**Architecture:** Terragrunt run-all with S3 backend in fr-par. MR-based plan/apply workflow. Weekly drift detection via scheduled pipeline.
**Tech Stack:** Terragrunt 0.72.6, OpenTofu/Terraform 1.11.2, Scaleway S3, GitLab CI/CD, glab CLI
---
## 1. Tfstate Migration (nl-ams → fr-par)
1. Create `foxhunt-tfstate` bucket in fr-par via `scw` CLI (bootstrap, not TF-managed)
2. Copy all state files: `aws s3 sync` cross-region using Scaleway S3 endpoints
3. Update `infra/live/production/root.hcl`:
- `region = "fr-par"`
- `endpoints.s3 = "https://s3.fr-par.scw.cloud"`
4. Run `terragrunt init -migrate-state` in each module directory (kapsule, dns, object-storage, registry, block-storage, secrets)
5. Verify with `terragrunt plan` — should show no changes
6. Delete old nl-ams bucket
**Risk:** Low. State is JSON files being copied. `-migrate-state` handles backend reconfiguration. Old bucket remains as fallback until explicitly deleted.
## 2. Resource Import — Close IaC Gap
### Audit Results
| Resource | Actual State | In Terraform? | Action |
|----------|-------------|---------------|--------|
| `foxhunt-sccache` bucket (fr-par) | exists | No | Add to `object-storage` module + import |
| `foxhunt-ci` registry namespace (fr-par) | exists | No | Add to `registry` module + import |
| `grafana` A record (fxhnt.ai → 100.90.76.85) | exists | No | Add to `dns` module + import |
| `prometheus` A record (fxhnt.ai → 100.90.76.85) | exists | No | Add to `dns` module + import |
| `ci-runner/` live dir | orphan `.terragrunt-cache` | No terragrunt.hcl | Delete |
| `foxhunt-ibkr-*` secrets | exist | Yes (applied) | No action |
| K8s PVCs (dev-home, postgres, etc.) | exist | N/A | Leave as kubectl/Helm managed |
| `foxhunt-tfstate` bucket (fr-par, new) | bootstrap | N/A | Not TF-managed (circular dep) |
### Module Changes
**`infra/modules/object-storage/main.tf`** — Add sccache bucket:
```hcl
resource "scaleway_object_bucket" "sccache" {
name = "foxhunt-sccache"
region = var.region
lifecycle_rule {
enabled = true
prefix = ""
expiration { days = 14 }
}
versioning { enabled = false }
}
```
**`infra/modules/registry/main.tf`** — Add foxhunt-ci namespace:
```hcl
resource "scaleway_registry_namespace" "foxhunt_ci" {
name = "foxhunt-ci"
region = var.region
is_public = false
}
```
**`infra/modules/dns/main.tf`** — Add grafana + prometheus records:
```hcl
resource "scaleway_domain_record" "grafana" {
dns_zone = var.dns_zone
name = "grafana"
type = "A"
data = var.git_ip
ttl = 300
}
resource "scaleway_domain_record" "prometheus" {
dns_zone = var.dns_zone
name = "prometheus"
type = "A"
data = var.git_ip
ttl = 300
}
```
After all imports, `terragrunt run-all plan` should show zero changes.
## 3. GitLab CI/CD Pipeline for IaC
### CI Image
`infra/docker/Dockerfile.infra-runner` — lightweight image with:
- OpenTofu/Terraform 1.11.2
- Terragrunt 0.72.6
- `scw` CLI
- `glab` CLI
Built via Kaniko, pushed to SCW registry (`rg.fr-par.scw.cloud/foxhunt-ci/infra-runner`).
### Pipeline Jobs
**`infra-plan`** (MR gate):
- Stage: `check`
- Image: `${REGISTRY}/infra-runner:latest`
- Tags: `kapsule` (gitlab pool)
- Trigger: MR pipelines when `infra/**` changes
- Script: `terragrunt run-all plan` across all modules
- Output: plan summary as job artifact
**`infra-apply`** (merge to main):
- Stage: `deploy`
- Same image + tags
- Trigger: push to `main` when `infra/**` changed
- Script: `terragrunt run-all apply -auto-approve`
- Environment: `production/infrastructure`
**`infra-drift-check`** (weekly scheduled):
- Stage: `check`
- Same image + tags
- Trigger: scheduled pipeline (weekly, Sunday night)
- Script: `terragrunt run-all plan -detailed-exitcode`
- Exit 0 = no drift
- Exit 2 = drift → create GitLab issue via `glab issue create`
- No apply, detection + alerting only
### Credentials
Jobs use existing CI/CD variables (protected + masked):
- `SCW_ACCESS_KEY`, `SCW_SECRET_KEY`, `SCW_DEFAULT_PROJECT_ID`
### Separation
Infra jobs are independent from Rust build pipeline — different triggers (`infra/**`), different image, different pool (gitlab vs gpu-training). No interference.