docs: IaC pipeline & tfstate migration design
Covers tfstate migration nl-ams→fr-par, resource import for drift gaps (sccache bucket, foxhunt-ci registry, DNS records), and GitLab CI/CD pipeline (plan on MR, apply on merge, weekly drift detection). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
131
docs/plans/2026-02-25-iac-pipeline-design.md
Normal file
131
docs/plans/2026-02-25-iac-pipeline-design.md
Normal file
@@ -0,0 +1,131 @@
|
||||
# IaC Pipeline & Tfstate Migration Design
|
||||
|
||||
**Goal:** Migrate Terraform state from nl-ams to fr-par, close IaC drift gaps, and add GitLab CI/CD for infrastructure management.
|
||||
|
||||
**Architecture:** Terragrunt run-all with S3 backend in fr-par. MR-based plan/apply workflow. Weekly drift detection via scheduled pipeline.
|
||||
|
||||
**Tech Stack:** Terragrunt 0.72.6, OpenTofu/Terraform 1.11.2, Scaleway S3, GitLab CI/CD, glab CLI
|
||||
|
||||
---
|
||||
|
||||
## 1. Tfstate Migration (nl-ams → fr-par)
|
||||
|
||||
1. Create `foxhunt-tfstate` bucket in fr-par via `scw` CLI (bootstrap, not TF-managed)
|
||||
2. Copy all state files: `aws s3 sync` cross-region using Scaleway S3 endpoints
|
||||
3. Update `infra/live/production/root.hcl`:
|
||||
- `region = "fr-par"`
|
||||
- `endpoints.s3 = "https://s3.fr-par.scw.cloud"`
|
||||
4. Run `terragrunt init -migrate-state` in each module directory (kapsule, dns, object-storage, registry, block-storage, secrets)
|
||||
5. Verify with `terragrunt plan` — should show no changes
|
||||
6. Delete old nl-ams bucket
|
||||
|
||||
**Risk:** Low. State is JSON files being copied. `-migrate-state` handles backend reconfiguration. Old bucket remains as fallback until explicitly deleted.
|
||||
|
||||
## 2. Resource Import — Close IaC Gap
|
||||
|
||||
### Audit Results
|
||||
|
||||
| Resource | Actual State | In Terraform? | Action |
|
||||
|----------|-------------|---------------|--------|
|
||||
| `foxhunt-sccache` bucket (fr-par) | exists | No | Add to `object-storage` module + import |
|
||||
| `foxhunt-ci` registry namespace (fr-par) | exists | No | Add to `registry` module + import |
|
||||
| `grafana` A record (fxhnt.ai → 100.90.76.85) | exists | No | Add to `dns` module + import |
|
||||
| `prometheus` A record (fxhnt.ai → 100.90.76.85) | exists | No | Add to `dns` module + import |
|
||||
| `ci-runner/` live dir | orphan `.terragrunt-cache` | No terragrunt.hcl | Delete |
|
||||
| `foxhunt-ibkr-*` secrets | exist | Yes (applied) | No action |
|
||||
| K8s PVCs (dev-home, postgres, etc.) | exist | N/A | Leave as kubectl/Helm managed |
|
||||
| `foxhunt-tfstate` bucket (fr-par, new) | bootstrap | N/A | Not TF-managed (circular dep) |
|
||||
|
||||
### Module Changes
|
||||
|
||||
**`infra/modules/object-storage/main.tf`** — Add sccache bucket:
|
||||
```hcl
|
||||
resource "scaleway_object_bucket" "sccache" {
|
||||
name = "foxhunt-sccache"
|
||||
region = var.region
|
||||
|
||||
lifecycle_rule {
|
||||
enabled = true
|
||||
prefix = ""
|
||||
expiration { days = 14 }
|
||||
}
|
||||
versioning { enabled = false }
|
||||
}
|
||||
```
|
||||
|
||||
**`infra/modules/registry/main.tf`** — Add foxhunt-ci namespace:
|
||||
```hcl
|
||||
resource "scaleway_registry_namespace" "foxhunt_ci" {
|
||||
name = "foxhunt-ci"
|
||||
region = var.region
|
||||
is_public = false
|
||||
}
|
||||
```
|
||||
|
||||
**`infra/modules/dns/main.tf`** — Add grafana + prometheus records:
|
||||
```hcl
|
||||
resource "scaleway_domain_record" "grafana" {
|
||||
dns_zone = var.dns_zone
|
||||
name = "grafana"
|
||||
type = "A"
|
||||
data = var.git_ip
|
||||
ttl = 300
|
||||
}
|
||||
|
||||
resource "scaleway_domain_record" "prometheus" {
|
||||
dns_zone = var.dns_zone
|
||||
name = "prometheus"
|
||||
type = "A"
|
||||
data = var.git_ip
|
||||
ttl = 300
|
||||
}
|
||||
```
|
||||
|
||||
After all imports, `terragrunt run-all plan` should show zero changes.
|
||||
|
||||
## 3. GitLab CI/CD Pipeline for IaC
|
||||
|
||||
### CI Image
|
||||
|
||||
`infra/docker/Dockerfile.infra-runner` — lightweight image with:
|
||||
- OpenTofu/Terraform 1.11.2
|
||||
- Terragrunt 0.72.6
|
||||
- `scw` CLI
|
||||
- `glab` CLI
|
||||
|
||||
Built via Kaniko, pushed to SCW registry (`rg.fr-par.scw.cloud/foxhunt-ci/infra-runner`).
|
||||
|
||||
### Pipeline Jobs
|
||||
|
||||
**`infra-plan`** (MR gate):
|
||||
- Stage: `check`
|
||||
- Image: `${REGISTRY}/infra-runner:latest`
|
||||
- Tags: `kapsule` (gitlab pool)
|
||||
- Trigger: MR pipelines when `infra/**` changes
|
||||
- Script: `terragrunt run-all plan` across all modules
|
||||
- Output: plan summary as job artifact
|
||||
|
||||
**`infra-apply`** (merge to main):
|
||||
- Stage: `deploy`
|
||||
- Same image + tags
|
||||
- Trigger: push to `main` when `infra/**` changed
|
||||
- Script: `terragrunt run-all apply -auto-approve`
|
||||
- Environment: `production/infrastructure`
|
||||
|
||||
**`infra-drift-check`** (weekly scheduled):
|
||||
- Stage: `check`
|
||||
- Same image + tags
|
||||
- Trigger: scheduled pipeline (weekly, Sunday night)
|
||||
- Script: `terragrunt run-all plan -detailed-exitcode`
|
||||
- Exit 0 = no drift
|
||||
- Exit 2 = drift → create GitLab issue via `glab issue create`
|
||||
- No apply, detection + alerting only
|
||||
|
||||
### Credentials
|
||||
|
||||
Jobs use existing CI/CD variables (protected + masked):
|
||||
- `SCW_ACCESS_KEY`, `SCW_SECRET_KEY`, `SCW_DEFAULT_PROJECT_ID`
|
||||
|
||||
### Separation
|
||||
|
||||
Infra jobs are independent from Rust build pipeline — different triggers (`infra/**`), different image, different pool (gitlab vs gpu-training). No interference.
|
||||
Reference in New Issue
Block a user