diff --git a/docs/superpowers/specs/2026-06-21-decommission-rust-infra-design.md b/docs/superpowers/specs/2026-06-21-decommission-rust-infra-design.md new file mode 100644 index 000000000..78e394f2e --- /dev/null +++ b/docs/superpowers/specs/2026-06-21-decommission-rust-infra-design.md @@ -0,0 +1,80 @@ +# Decommission Dead Rust Infra — Design (Phase 1 of infra consolidation) + +**Date:** 2026-06-21 +**Status:** Design (pending approval) +**Repo:** foxhunt (where the platform IaC currently lives) + +## Motivation + +The active trading work is now the Python **fxhnt** fund; the Rust **foxhunt** ML/HFT system is dormant. +Its dedicated infra (GPU training pools, Rust build/training caches, GPU CI runners, training artifact +buckets, training manifests) is unused but still declared/provisioned. Phase 1 decommissions it — for +cost/clutter savings and to shrink the surface before Phase 2 (relocating the remaining *platform* IaC +into `fxhnt/infra`, a separate spec). + +**Verified unused (2026-06-21):** L40S + H100 pools have 0 nodes; `cargo-target-{cpu,cuda,cuda-test}` + +`feature-cache-pvc` are `Used By: `; no gitlab-runner pods running. + +## Scope — what gets removed (all verified unused) + +1. **Terraform (kapsule module + terragrunt.hcl):** set to `false` → + - `enable_ci_training_l40s_pool` (L40S-1-48G GPU pool) + - `enable_ci_training_h100_pool` (H100-1-80G GPU pool) + - `enable_ci_compile_cpu_hm_pool` (POP2-HM-32C-256G precompute pool — Rust `precompute_features` only) + `terragrunt apply` destroys exactly these 3 pools. Also delete their now-dead var blocks/resources. +2. **Helm uninstall** (foxhunt ns): `gitlab-runner-h100`, `gitlab-runner-h100-sxm`, `gitlab-runner-h100x2` + (Rust GPU CI runners). **Main `gitlab-runner`:** verify nothing non-Rust uses it (fxhnt has no + `.gitlab-ci.yml`; cockpit deploys via Argo) → uninstall if confirmed dead, else keep. (confirm step) +3. **PVCs delete** (foxhunt ns — PURE BUILD CACHES only, unmounted-verified, ~175 GB reclaimed): + `cargo-target-cpu` (60Gi), `cargo-target-cuda` (45Gi), `cargo-target-cuda-test` (30Gi), + `sccache-cpu` (20Gi), `sccache-cuda` (20Gi). These hold zero data — regenerated on any build. +4. **MinIO buckets delete** (NO market data — verified 0 `.dbn` objects): `foxhunt-binaries` (2.3 GiB + compiled binaries), `foxhunt-training-results` (4.4 GiB run logs/outputs), `foxhunt-models` (empty). +5. **Repo manifests/templates remove** (foxhunt repo): `infra/k8s/training/`, `infra/k8s/gpu-overlays/`, + the Rust Argo workflow templates (`train-multi-seed-template.yaml`, `lob-backtest-sweep-template.yaml`, + `ci-pipeline-template.yaml`, alpha-rl train templates), `infra/k8s/jobs/download-trades-job.yaml` + (databento), the 3 GPU-runner helm value files. Also remove the dead Argo `WorkflowTemplate`s from the + cluster (`kubectl delete wftmpl`) for the Rust train/backtest pipelines. + +## 🛑 EXPLICITLY PRESERVE (do NOT delete — market data + reusable assets) +- **`training-data-pvc` (500 GiB)** — the raw Databento MBP-10 `.dbn` market data. KEEP. +- **`foxhunt-training-data` bucket (38 GiB, 54 `.dbn`/`.zst`)** — Databento market data (expensive to + re-acquire; the fund's `.dbn` backtests read it). KEEP. +- **`test-data-pvc` (50 GiB)** — `.dbn` test subsets (tier-1.5 smoke uses test_data). KEEP (verify, don't delete). +- **`feature-cache-pvc` (100 GiB)** — derived ML features; regenerable but compute-costly. KEEP (conservative). +- All fxhnt/platform data: `fxhnt-backtest-data`, `fxhnt-surfer-data`, `multistrat-state`, `questdb-pvc`, + `tempo-data`, `netbird-data`, `foxhunt-gitlab-*` + `foxhunt-backups` buckets. +General rule: delete only **pure compute/build artifacts** (caches, compiled binaries, run logs, GPU pools); +**never** anything holding `.dbn`/market data or any potentially-reusable dataset. + +## Out of scope (KEEP — shared platform / Python fund) +`ci-compile-cpu` pool (Python cockpit builds), platform pool, GitLab, MinIO, monitoring, Mattermost, +Stalwart, Kanidm, NetBird, DNS, databases, cert-manager, tailscale proxy, public-gateway, dagster/cockpit, +the fxhnt forward-track jobs. Phase 2 (IaC relocation to `fxhnt/infra` + TF-state move) is a separate spec. + +## Execution order (each destructive step verified + confirmed) +1. **Terraform first**: edit terragrunt.hcl (3 `enable_*=false`), `terragrunt plan` → **STOP unless the + plan shows ONLY the 3 pools destroyed + 0 other destroys**; then `apply`. +2. **Helm uninstalls** (GPU runners; main runner only after the use-check confirms dead). +3. **PVC deletes** (re-confirm `Used By: ` immediately before each delete — irreversible). +4. **Bucket deletes** (list contents first; irreversible — explicit confirm; `mc rb --force` via the + port-forward + minio creds). +5. **Repo cleanup**: remove the dead manifests/templates + module var blocks, `kubectl delete wftmpl` the + dead templates, commit. + +## Risks / safety +- **Irreversible**: PVC + bucket deletes, GPU-pool destroys. Mitigation: verify-unused immediately before + each; Terraform plan reviewed for 0-unexpected-destroys; explicit confirm on PVC/bucket deletes. +- **Mis-scope risk**: a kept resource accidentally listed. Mitigation: the Terraform plan gate (only the 3 + pools) + the `Used By` re-check + bucket content listing before delete. +- **Main gitlab-runner**: don't uninstall without confirming no active consumer (could break a CI path). +- Cluster health unaffected: nothing here touches the cockpit/dagster/fund/platform services. + +## Acceptance criteria +- `terragrunt plan` (kapsule) shows the 3 GPU/precompute pools gone, then clean (no changes). +- GPU-runner helm releases uninstalled; main runner resolved (kept or uninstalled per check). +- The 5 build-cache PVCs deleted (cargo×3 + sccache×2, ~175 GB reclaimed); 3 Rust buckets deleted + (binaries, training-results, models). **`.dbn` market data untouched** (training-data-pvc + the + foxhunt-training-data bucket + test-data-pvc still present, byte-for-byte). +- Dead training/gpu manifests + Argo templates removed from repo + cluster; committed. +- Cockpit (`dashboard.fxhnt.ai` 200), dagster, and the fund tracks still healthy post-cleanup.