docs: plan — decommission dead Rust infra (Phase 1), inline+gated execution
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
263
docs/superpowers/plans/2026-06-21-decommission-rust-infra.md
Normal file
263
docs/superpowers/plans/2026-06-21-decommission-rust-infra.md
Normal file
@@ -0,0 +1,263 @@
|
||||
# Decommission Dead Rust Infra — Implementation Plan (Phase 1)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
|
||||
> **Do NOT run this subagent-driven** — every task performs irreversible production deletes that need
|
||||
> human confirmation at each gate. Steps use checkbox (`- [ ]`) syntax.
|
||||
|
||||
**Goal:** Remove the now-unused Rust-specific infra (GPU pools, GPU CI runners, build-cache PVCs, Rust
|
||||
artifact buckets, training/GPU manifests) — preserving ALL `.dbn` market data and every fxhnt/platform resource.
|
||||
|
||||
**Architecture:** Verify-then-delete, gated. Terraform destroys the GPU/precompute pools (plan reviewed
|
||||
for exactly-3); helm uninstalls the GPU runners; kubectl deletes the unmounted build-cache PVCs; mc deletes
|
||||
the no-`.dbn` Rust buckets; repo + cluster Argo templates cleaned up. Cockpit/dagster/fund verified healthy.
|
||||
|
||||
**Tech Stack:** terragrunt + OpenTofu (Scaleway provider), helm, kubectl, mc (MinIO), Scaleway Kapsule.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-21-decommission-rust-infra-design.md`
|
||||
|
||||
---
|
||||
|
||||
## Shared env (export at the start of each shell session that runs terragrunt/scw)
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
export TG_TF_PATH=tofu TERRAGRUNT_TFPATH=tofu
|
||||
export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
|
||||
export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
|
||||
export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
|
||||
export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
|
||||
export TF_HTTP_USERNAME=root TF_HTTP_PASSWORD=$(kubectl get secret gitlab-pat -n foxhunt -o jsonpath='{.data.token}'|base64 -d)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 0: Branch
|
||||
- [ ] **Step 1**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/decommission-rust-infra
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Destroy the GPU + precompute pools (Terraform)
|
||||
|
||||
**Files:** Modify `infra/live/production/kapsule/terragrunt.hcl`
|
||||
|
||||
- [ ] **Step 1: Set the three enable flags to false**
|
||||
|
||||
In `infra/live/production/kapsule/terragrunt.hcl`, change:
|
||||
```
|
||||
enable_ci_compile_cpu_hm_pool = true
|
||||
```
|
||||
to `false`; and
|
||||
```
|
||||
enable_ci_training_l40s_pool = true
|
||||
```
|
||||
to `false`; and
|
||||
```
|
||||
enable_ci_training_h100_pool = true
|
||||
```
|
||||
to `false`. (Leave `enable_ci_compile_cpu_pool = true` — Python cockpit builds use it.)
|
||||
|
||||
- [ ] **Step 2: Plan and GATE on exactly-3-destroys**
|
||||
|
||||
Run (with shared env, in `infra/live/production/kapsule`):
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule
|
||||
terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|will be destroyed|# scaleway"
|
||||
```
|
||||
Expected: `Plan: 0 to add, 0 to change, 3 to destroy.` and the 3 destroyed are EXACTLY
|
||||
`scaleway_k8s_pool.ci_compile_cpu_hm[0]`, `scaleway_k8s_pool.ci_training_l40s[0]`,
|
||||
`scaleway_k8s_pool.ci_training_h100[0]`.
|
||||
**STOP if the count ≠ 3 or any other resource is destroyed.**
|
||||
|
||||
- [ ] **Step 3: Apply**
|
||||
```bash
|
||||
terragrunt apply -input=false -no-color -auto-approve 2>&1 | sed 's/.*tofu: //' | grep -iE "Apply complete|Destroy complete|Error"
|
||||
```
|
||||
Expected: `Apply complete! Resources: 0 added, 0 changed, 3 destroyed.`
|
||||
|
||||
- [ ] **Step 4: Verify pools gone**
|
||||
```bash
|
||||
scw k8s pool list cluster-id=34a1e3c4-ac35-48c8-ab49-5f6ec4df32c1 -o json 2>/dev/null | python3 -c "import sys,json;print([p['name'] for p in json.load(sys.stdin)])"
|
||||
```
|
||||
Expected: `['ci-compile-cpu', 'platform']` (no l40s/h100/hm).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
git add infra/live/production/kapsule/terragrunt.hcl
|
||||
git commit -m "chore(infra): destroy unused Rust GPU + precompute pools (L40S/H100/hm)
|
||||
|
||||
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Uninstall the GPU CI runners (helm)
|
||||
|
||||
- [ ] **Step 1: Uninstall the 3 GPU runners**
|
||||
```bash
|
||||
for r in gitlab-runner-h100 gitlab-runner-h100-sxm gitlab-runner-h100x2; do
|
||||
helm uninstall "$r" -n foxhunt 2>&1 | tail -1
|
||||
done
|
||||
```
|
||||
Expected: `release "<r>" uninstalled` for each.
|
||||
|
||||
- [ ] **Step 2: Decide the main `gitlab-runner`**
|
||||
|
||||
Check whether anything still uses it (fxhnt has no `.gitlab-ci.yml`; cockpit deploys via Argo):
|
||||
```bash
|
||||
kubectl get pods -n foxhunt | grep gitlab-runner # expect: no running runner pods
|
||||
# Inspect what the main runner is registered for (manual judgement):
|
||||
helm get values gitlab-runner -n foxhunt 2>/dev/null | grep -iE "tags|runUntagged|description" | head
|
||||
```
|
||||
- [ ] **Step 3: If confirmed dead, uninstall it; else keep (document the decision):**
|
||||
```bash
|
||||
helm uninstall gitlab-runner -n foxhunt 2>&1 | tail -1 # ONLY if Step 2 confirms no consumer
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify**
|
||||
```bash
|
||||
helm list -n foxhunt | grep -i runner # expect: none (or only a deliberately-kept one)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Delete the build-cache PVCs (re-verify unmounted first)
|
||||
|
||||
- [ ] **Step 1: Re-confirm each PVC is unmounted (safety)**
|
||||
```bash
|
||||
for p in cargo-target-cpu cargo-target-cuda cargo-target-cuda-test sccache-cpu sccache-cuda; do
|
||||
echo -n "$p: "; kubectl describe pvc $p -n foxhunt 2>/dev/null | grep -A1 "Used By" | tr '\n' ' '; echo
|
||||
done
|
||||
```
|
||||
Expected: every line shows `Used By: <none>`. **STOP on any PVC that lists a pod.**
|
||||
|
||||
- [ ] **Step 2: Delete them**
|
||||
```bash
|
||||
kubectl delete pvc cargo-target-cpu cargo-target-cuda cargo-target-cuda-test sccache-cpu sccache-cuda -n foxhunt 2>&1 | tail
|
||||
```
|
||||
Expected: `persistentvolumeclaim "<name>" deleted` ×5.
|
||||
|
||||
- [ ] **Step 3: Verify reclaim**
|
||||
```bash
|
||||
kubectl get pvc -n foxhunt | grep -E "cargo-target|sccache" || echo "all build-cache PVCs gone (~175Gi reclaimed)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Delete the Rust artifact buckets (re-verify 0 `.dbn` first)
|
||||
|
||||
- [ ] **Step 1: Start a MinIO port-forward + mc alias**
|
||||
```bash
|
||||
pkill -f "port-forward svc/minio" 2>/dev/null; sleep 1
|
||||
kubectl port-forward svc/minio -n foxhunt 19000:9000 >/tmp/minio-pf.log 2>&1 &
|
||||
sleep 4
|
||||
ak=$(kubectl get secret minio-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
|
||||
sk=$(kubectl get secret minio-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
|
||||
mc alias set fxl http://localhost:19000 "$ak" "$sk"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Re-verify the 3 target buckets contain ZERO `.dbn`/market data, and the KEPT one does**
|
||||
```bash
|
||||
for b in foxhunt-binaries foxhunt-training-results foxhunt-models; do
|
||||
echo -n "$b dbn-count="; mc ls --recursive fxl/$b 2>/dev/null | grep -icE "\.dbn|\.zst|databento|mbp"
|
||||
done
|
||||
echo -n "KEEP foxhunt-training-data dbn-count="; mc ls --recursive fxl/foxhunt-training-data 2>/dev/null | grep -icE "\.dbn|\.zst|databento|mbp"
|
||||
```
|
||||
Expected: the 3 targets show `0`; `foxhunt-training-data` shows `>0`. **STOP if any target shows >0.**
|
||||
|
||||
- [ ] **Step 3: Delete the 3 Rust buckets**
|
||||
```bash
|
||||
for b in foxhunt-binaries foxhunt-training-results foxhunt-models; do
|
||||
mc rb --force fxl/$b 2>&1 | tail -1
|
||||
done
|
||||
```
|
||||
Expected: `Removed '<bucket>' successfully.` ×3.
|
||||
|
||||
- [ ] **Step 4: Verify market data intact**
|
||||
```bash
|
||||
mc du fxl/foxhunt-training-data # expect: still ~38GiB, 54+ objects
|
||||
kubectl get pvc training-data-pvc test-data-pvc feature-cache-pvc -n foxhunt # expect: all 3 still Bound
|
||||
pkill -f "port-forward svc/minio" 2>/dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Remove dead manifests + Argo templates (repo + cluster)
|
||||
|
||||
**Files:** Remove `infra/k8s/training/`, `infra/k8s/gpu-overlays/`, Rust Argo templates, the databento job, GPU-runner helm values.
|
||||
|
||||
- [ ] **Step 1: Identify the Rust Argo WorkflowTemplates in the cluster (keep fxhnt-cockpit!)**
|
||||
```bash
|
||||
kubectl get wftmpl -n foxhunt -o name 2>/dev/null
|
||||
```
|
||||
Note the Rust ones (e.g. train-multi-seed, lob-backtest-sweep, ci-pipeline, alpha-rl-*). **Do NOT touch `fxhnt-cockpit`.**
|
||||
|
||||
- [ ] **Step 2: Delete the dead WorkflowTemplates from the cluster** (substitute the exact names from Step 1):
|
||||
```bash
|
||||
# example — replace with the actual Rust template names from Step 1:
|
||||
kubectl delete wftmpl -n foxhunt <rust-template-1> <rust-template-2> ... 2>&1 | tail
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Remove dead manifests + the now-unused Terraform pool blocks from the repo**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
git rm -r infra/k8s/training infra/k8s/gpu-overlays 2>/dev/null
|
||||
git rm infra/k8s/jobs/download-trades-job.yaml infra/k8s/argo/train-multi-seed-template.yaml infra/k8s/argo/lob-backtest-sweep-template.yaml infra/k8s/argo/ci-pipeline-template.yaml 2>/dev/null
|
||||
# (also git rm any GPU-runner helm value files + alpha-rl argo templates found under infra/)
|
||||
ls infra/k8s/argo infra/k8s/jobs # eyeball what remains; keep cockpit/fxhnt/fund + cert-manager etc.
|
||||
```
|
||||
- [ ] **Step 4: Remove the dead pool resource blocks + variables from the kapsule module** (optional tidy):
|
||||
in `infra/modules/kapsule/main.tf` remove the `scaleway_k8s_pool.ci_training_l40s`,
|
||||
`...ci_training_h100`, `...ci_compile_cpu_hm` resource blocks (now count=0); in `variables.tf` remove
|
||||
their `enable_*`/`*_type`/`*_max_size` vars; in `terragrunt.hcl` remove the 3 dead `enable_*`/`*_type`
|
||||
lines. Then re-plan to confirm still clean:
|
||||
```bash
|
||||
cd infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|No changes"
|
||||
```
|
||||
Expected: `No changes` (removing count=0 blocks is a no-op against the cluster).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
git add -A infra
|
||||
git commit -m "chore(infra): remove dead Rust training/GPU manifests, Argo templates, pool config
|
||||
|
||||
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Final verification (cluster + fund still healthy)
|
||||
|
||||
- [ ] **Step 1: Platform/fund health**
|
||||
```bash
|
||||
curl -sS -m 12 -o /dev/null -w "dashboard HTTP %{http_code}\n" https://dashboard.fxhnt.ai/
|
||||
kubectl get pods -n foxhunt | grep -E "dagster|fxhnt-dashboard|gitlab-webservice" | grep -vE "Completed"
|
||||
kubectl get pvc training-data-pvc test-data-pvc feature-cache-pvc fxhnt-surfer-data fxhnt-backtest-data -n foxhunt
|
||||
```
|
||||
Expected: dashboard 200; dagster/dashboard/gitlab Running; all kept PVCs Bound.
|
||||
|
||||
- [ ] **Step 2: Terraform clean**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|No changes"
|
||||
```
|
||||
Expected: `No changes`.
|
||||
|
||||
- [ ] **Step 3: Merge to main**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
git checkout main && git merge --ff-only chore/decommission-rust-infra && git branch -d chore/decommission-rust-infra
|
||||
git push origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes for the executor
|
||||
- **Data safety is paramount:** never delete `training-data-pvc`, `test-data-pvc`, `feature-cache-pvc`, the
|
||||
`foxhunt-training-data` bucket, or any fxhnt/platform resource. Re-verify (`Used By`, `.dbn` count) at each
|
||||
delete gate; STOP on any surprise.
|
||||
- **Terraform gate:** Task 1 Step 2 must show exactly the 3 GPU/precompute pools destroyed — stop otherwise.
|
||||
- `pkill -f "port-forward svc/minio"` after Task 4 to clean up the forward.
|
||||
- If the main `gitlab-runner` is ambiguous, KEEP it (a kept idle runner is harmless; a wrongly-removed one breaks CI).
|
||||
Reference in New Issue
Block a user