From a0181431b3a9c8bb3e0771f0ae98ae2011d7a6fe Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 21 Jun 2026 14:24:41 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20plan=20=E2=80=94=20decommission=20dead?= =?UTF-8?q?=20Rust=20infra=20(Phase=201),=20inline+gated=20execution?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 --- .../2026-06-21-decommission-rust-infra.md | 263 ++++++++++++++++++ 1 file changed, 263 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-21-decommission-rust-infra.md diff --git a/docs/superpowers/plans/2026-06-21-decommission-rust-infra.md b/docs/superpowers/plans/2026-06-21-decommission-rust-infra.md new file mode 100644 index 000000000..a05f09afc --- /dev/null +++ b/docs/superpowers/plans/2026-06-21-decommission-rust-infra.md @@ -0,0 +1,263 @@ +# Decommission Dead Rust Infra — Implementation Plan (Phase 1) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints). +> **Do NOT run this subagent-driven** — every task performs irreversible production deletes that need +> human confirmation at each gate. Steps use checkbox (`- [ ]`) syntax. + +**Goal:** Remove the now-unused Rust-specific infra (GPU pools, GPU CI runners, build-cache PVCs, Rust +artifact buckets, training/GPU manifests) — preserving ALL `.dbn` market data and every fxhnt/platform resource. + +**Architecture:** Verify-then-delete, gated. Terraform destroys the GPU/precompute pools (plan reviewed +for exactly-3); helm uninstalls the GPU runners; kubectl deletes the unmounted build-cache PVCs; mc deletes +the no-`.dbn` Rust buckets; repo + cluster Argo templates cleaned up. Cockpit/dagster/fund verified healthy. + +**Tech Stack:** terragrunt + OpenTofu (Scaleway provider), helm, kubectl, mc (MinIO), Scaleway Kapsule. + +**Spec:** `docs/superpowers/specs/2026-06-21-decommission-rust-infra-design.md` + +--- + +## Shared env (export at the start of each shell session that runs terragrunt/scw) +```bash +cd /home/jgrusewski/Work/foxhunt +export TG_TF_PATH=tofu TERRAGRUNT_TFPATH=tofu +export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d) +export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d) +export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d) +export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2 +export TF_HTTP_USERNAME=root TF_HTTP_PASSWORD=$(kubectl get secret gitlab-pat -n foxhunt -o jsonpath='{.data.token}'|base64 -d) +``` + +--- + +## Task 0: Branch +- [ ] **Step 1** +```bash +cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/decommission-rust-infra +``` + +--- + +## Task 1: Destroy the GPU + precompute pools (Terraform) + +**Files:** Modify `infra/live/production/kapsule/terragrunt.hcl` + +- [ ] **Step 1: Set the three enable flags to false** + +In `infra/live/production/kapsule/terragrunt.hcl`, change: +``` + enable_ci_compile_cpu_hm_pool = true +``` +to `false`; and +``` + enable_ci_training_l40s_pool = true +``` +to `false`; and +``` + enable_ci_training_h100_pool = true +``` +to `false`. (Leave `enable_ci_compile_cpu_pool = true` — Python cockpit builds use it.) + +- [ ] **Step 2: Plan and GATE on exactly-3-destroys** + +Run (with shared env, in `infra/live/production/kapsule`): +```bash +cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule +terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|will be destroyed|# scaleway" +``` +Expected: `Plan: 0 to add, 0 to change, 3 to destroy.` and the 3 destroyed are EXACTLY +`scaleway_k8s_pool.ci_compile_cpu_hm[0]`, `scaleway_k8s_pool.ci_training_l40s[0]`, +`scaleway_k8s_pool.ci_training_h100[0]`. +**STOP if the count ≠ 3 or any other resource is destroyed.** + +- [ ] **Step 3: Apply** +```bash +terragrunt apply -input=false -no-color -auto-approve 2>&1 | sed 's/.*tofu: //' | grep -iE "Apply complete|Destroy complete|Error" +``` +Expected: `Apply complete! Resources: 0 added, 0 changed, 3 destroyed.` + +- [ ] **Step 4: Verify pools gone** +```bash +scw k8s pool list cluster-id=34a1e3c4-ac35-48c8-ab49-5f6ec4df32c1 -o json 2>/dev/null | python3 -c "import sys,json;print([p['name'] for p in json.load(sys.stdin)])" +``` +Expected: `['ci-compile-cpu', 'platform']` (no l40s/h100/hm). + +- [ ] **Step 5: Commit** +```bash +cd /home/jgrusewski/Work/foxhunt +git add infra/live/production/kapsule/terragrunt.hcl +git commit -m "chore(infra): destroy unused Rust GPU + precompute pools (L40S/H100/hm) + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +## Task 2: Uninstall the GPU CI runners (helm) + +- [ ] **Step 1: Uninstall the 3 GPU runners** +```bash +for r in gitlab-runner-h100 gitlab-runner-h100-sxm gitlab-runner-h100x2; do + helm uninstall "$r" -n foxhunt 2>&1 | tail -1 +done +``` +Expected: `release "" uninstalled` for each. + +- [ ] **Step 2: Decide the main `gitlab-runner`** + +Check whether anything still uses it (fxhnt has no `.gitlab-ci.yml`; cockpit deploys via Argo): +```bash +kubectl get pods -n foxhunt | grep gitlab-runner # expect: no running runner pods +# Inspect what the main runner is registered for (manual judgement): +helm get values gitlab-runner -n foxhunt 2>/dev/null | grep -iE "tags|runUntagged|description" | head +``` +- [ ] **Step 3: If confirmed dead, uninstall it; else keep (document the decision):** +```bash +helm uninstall gitlab-runner -n foxhunt 2>&1 | tail -1 # ONLY if Step 2 confirms no consumer +``` + +- [ ] **Step 4: Verify** +```bash +helm list -n foxhunt | grep -i runner # expect: none (or only a deliberately-kept one) +``` + +--- + +## Task 3: Delete the build-cache PVCs (re-verify unmounted first) + +- [ ] **Step 1: Re-confirm each PVC is unmounted (safety)** +```bash +for p in cargo-target-cpu cargo-target-cuda cargo-target-cuda-test sccache-cpu sccache-cuda; do + echo -n "$p: "; kubectl describe pvc $p -n foxhunt 2>/dev/null | grep -A1 "Used By" | tr '\n' ' '; echo +done +``` +Expected: every line shows `Used By: `. **STOP on any PVC that lists a pod.** + +- [ ] **Step 2: Delete them** +```bash +kubectl delete pvc cargo-target-cpu cargo-target-cuda cargo-target-cuda-test sccache-cpu sccache-cuda -n foxhunt 2>&1 | tail +``` +Expected: `persistentvolumeclaim "" deleted` ×5. + +- [ ] **Step 3: Verify reclaim** +```bash +kubectl get pvc -n foxhunt | grep -E "cargo-target|sccache" || echo "all build-cache PVCs gone (~175Gi reclaimed)" +``` + +--- + +## Task 4: Delete the Rust artifact buckets (re-verify 0 `.dbn` first) + +- [ ] **Step 1: Start a MinIO port-forward + mc alias** +```bash +pkill -f "port-forward svc/minio" 2>/dev/null; sleep 1 +kubectl port-forward svc/minio -n foxhunt 19000:9000 >/tmp/minio-pf.log 2>&1 & +sleep 4 +ak=$(kubectl get secret minio-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d) +sk=$(kubectl get secret minio-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d) +mc alias set fxl http://localhost:19000 "$ak" "$sk" +``` + +- [ ] **Step 2: Re-verify the 3 target buckets contain ZERO `.dbn`/market data, and the KEPT one does** +```bash +for b in foxhunt-binaries foxhunt-training-results foxhunt-models; do + echo -n "$b dbn-count="; mc ls --recursive fxl/$b 2>/dev/null | grep -icE "\.dbn|\.zst|databento|mbp" +done +echo -n "KEEP foxhunt-training-data dbn-count="; mc ls --recursive fxl/foxhunt-training-data 2>/dev/null | grep -icE "\.dbn|\.zst|databento|mbp" +``` +Expected: the 3 targets show `0`; `foxhunt-training-data` shows `>0`. **STOP if any target shows >0.** + +- [ ] **Step 3: Delete the 3 Rust buckets** +```bash +for b in foxhunt-binaries foxhunt-training-results foxhunt-models; do + mc rb --force fxl/$b 2>&1 | tail -1 +done +``` +Expected: `Removed '' successfully.` ×3. + +- [ ] **Step 4: Verify market data intact** +```bash +mc du fxl/foxhunt-training-data # expect: still ~38GiB, 54+ objects +kubectl get pvc training-data-pvc test-data-pvc feature-cache-pvc -n foxhunt # expect: all 3 still Bound +pkill -f "port-forward svc/minio" 2>/dev/null +``` + +--- + +## Task 5: Remove dead manifests + Argo templates (repo + cluster) + +**Files:** Remove `infra/k8s/training/`, `infra/k8s/gpu-overlays/`, Rust Argo templates, the databento job, GPU-runner helm values. + +- [ ] **Step 1: Identify the Rust Argo WorkflowTemplates in the cluster (keep fxhnt-cockpit!)** +```bash +kubectl get wftmpl -n foxhunt -o name 2>/dev/null +``` +Note the Rust ones (e.g. train-multi-seed, lob-backtest-sweep, ci-pipeline, alpha-rl-*). **Do NOT touch `fxhnt-cockpit`.** + +- [ ] **Step 2: Delete the dead WorkflowTemplates from the cluster** (substitute the exact names from Step 1): +```bash +# example — replace with the actual Rust template names from Step 1: +kubectl delete wftmpl -n foxhunt ... 2>&1 | tail +``` + +- [ ] **Step 3: Remove dead manifests + the now-unused Terraform pool blocks from the repo** +```bash +cd /home/jgrusewski/Work/foxhunt +git rm -r infra/k8s/training infra/k8s/gpu-overlays 2>/dev/null +git rm infra/k8s/jobs/download-trades-job.yaml infra/k8s/argo/train-multi-seed-template.yaml infra/k8s/argo/lob-backtest-sweep-template.yaml infra/k8s/argo/ci-pipeline-template.yaml 2>/dev/null +# (also git rm any GPU-runner helm value files + alpha-rl argo templates found under infra/) +ls infra/k8s/argo infra/k8s/jobs # eyeball what remains; keep cockpit/fxhnt/fund + cert-manager etc. +``` +- [ ] **Step 4: Remove the dead pool resource blocks + variables from the kapsule module** (optional tidy): +in `infra/modules/kapsule/main.tf` remove the `scaleway_k8s_pool.ci_training_l40s`, +`...ci_training_h100`, `...ci_compile_cpu_hm` resource blocks (now count=0); in `variables.tf` remove +their `enable_*`/`*_type`/`*_max_size` vars; in `terragrunt.hcl` remove the 3 dead `enable_*`/`*_type` +lines. Then re-plan to confirm still clean: +```bash +cd infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|No changes" +``` +Expected: `No changes` (removing count=0 blocks is a no-op against the cluster). + +- [ ] **Step 5: Commit** +```bash +cd /home/jgrusewski/Work/foxhunt +git add -A infra +git commit -m "chore(infra): remove dead Rust training/GPU manifests, Argo templates, pool config + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +## Task 6: Final verification (cluster + fund still healthy) + +- [ ] **Step 1: Platform/fund health** +```bash +curl -sS -m 12 -o /dev/null -w "dashboard HTTP %{http_code}\n" https://dashboard.fxhnt.ai/ +kubectl get pods -n foxhunt | grep -E "dagster|fxhnt-dashboard|gitlab-webservice" | grep -vE "Completed" +kubectl get pvc training-data-pvc test-data-pvc feature-cache-pvc fxhnt-surfer-data fxhnt-backtest-data -n foxhunt +``` +Expected: dashboard 200; dagster/dashboard/gitlab Running; all kept PVCs Bound. + +- [ ] **Step 2: Terraform clean** +```bash +cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|No changes" +``` +Expected: `No changes`. + +- [ ] **Step 3: Merge to main** +```bash +cd /home/jgrusewski/Work/foxhunt +git checkout main && git merge --ff-only chore/decommission-rust-infra && git branch -d chore/decommission-rust-infra +git push origin main +``` + +--- + +## Notes for the executor +- **Data safety is paramount:** never delete `training-data-pvc`, `test-data-pvc`, `feature-cache-pvc`, the + `foxhunt-training-data` bucket, or any fxhnt/platform resource. Re-verify (`Used By`, `.dbn` count) at each + delete gate; STOP on any surprise. +- **Terraform gate:** Task 1 Step 2 must show exactly the 3 GPU/precompute pools destroyed — stop otherwise. +- `pkill -f "port-forward svc/minio"` after Task 4 to clean up the forward. +- If the main `gitlab-runner` is ambiguous, KEEP it (a kept idle runner is harmless; a wrongly-removed one breaks CI).