docs: spec — decommission dead Rust infra (Phase 1), data-preserving (keep all .dbn)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/superpowers/specs/2026-06-21-decommission-rust-infra-design.md

@@ -0,0 +1,80 @@
+# Decommission Dead Rust Infra — Design (Phase 1 of infra consolidation)
+
+**Date:** 2026-06-21
+**Status:** Design (pending approval)
+**Repo:** foxhunt (where the platform IaC currently lives)
+
+## Motivation
+
+The active trading work is now the Python **fxhnt** fund; the Rust **foxhunt** ML/HFT system is dormant.
+Its dedicated infra (GPU training pools, Rust build/training caches, GPU CI runners, training artifact
+buckets, training manifests) is unused but still declared/provisioned. Phase 1 decommissions it — for
+cost/clutter savings and to shrink the surface before Phase 2 (relocating the remaining *platform* IaC
+into `fxhnt/infra`, a separate spec).
+
+**Verified unused (2026-06-21):** L40S + H100 pools have 0 nodes; `cargo-target-{cpu,cuda,cuda-test}` +
+`feature-cache-pvc` are `Used By: <none>`; no gitlab-runner pods running.
+
+## Scope — what gets removed (all verified unused)
+
+1. **Terraform (kapsule module + terragrunt.hcl):** set to `false` →
+   - `enable_ci_training_l40s_pool` (L40S-1-48G GPU pool)
+   - `enable_ci_training_h100_pool` (H100-1-80G GPU pool)
+   - `enable_ci_compile_cpu_hm_pool` (POP2-HM-32C-256G precompute pool — Rust `precompute_features` only)
+   `terragrunt apply` destroys exactly these 3 pools. Also delete their now-dead var blocks/resources.
+2. **Helm uninstall** (foxhunt ns): `gitlab-runner-h100`, `gitlab-runner-h100-sxm`, `gitlab-runner-h100x2`
+   (Rust GPU CI runners). **Main `gitlab-runner`:** verify nothing non-Rust uses it (fxhnt has no
+   `.gitlab-ci.yml`; cockpit deploys via Argo) → uninstall if confirmed dead, else keep. (confirm step)
+3. **PVCs delete** (foxhunt ns — PURE BUILD CACHES only, unmounted-verified, ~175 GB reclaimed):
+   `cargo-target-cpu` (60Gi), `cargo-target-cuda` (45Gi), `cargo-target-cuda-test` (30Gi),
+   `sccache-cpu` (20Gi), `sccache-cuda` (20Gi). These hold zero data — regenerated on any build.
+4. **MinIO buckets delete** (NO market data — verified 0 `.dbn` objects): `foxhunt-binaries` (2.3 GiB
+   compiled binaries), `foxhunt-training-results` (4.4 GiB run logs/outputs), `foxhunt-models` (empty).
+5. **Repo manifests/templates remove** (foxhunt repo): `infra/k8s/training/`, `infra/k8s/gpu-overlays/`,
+   the Rust Argo workflow templates (`train-multi-seed-template.yaml`, `lob-backtest-sweep-template.yaml`,
+   `ci-pipeline-template.yaml`, alpha-rl train templates), `infra/k8s/jobs/download-trades-job.yaml`
+   (databento), the 3 GPU-runner helm value files. Also remove the dead Argo `WorkflowTemplate`s from the
+   cluster (`kubectl delete wftmpl`) for the Rust train/backtest pipelines.
+
+## 🛑 EXPLICITLY PRESERVE (do NOT delete — market data + reusable assets)
+- **`training-data-pvc` (500 GiB)** — the raw Databento MBP-10 `.dbn` market data. KEEP.
+- **`foxhunt-training-data` bucket (38 GiB, 54 `.dbn`/`.zst`)** — Databento market data (expensive to
+  re-acquire; the fund's `.dbn` backtests read it). KEEP.
+- **`test-data-pvc` (50 GiB)** — `.dbn` test subsets (tier-1.5 smoke uses test_data). KEEP (verify, don't delete).
+- **`feature-cache-pvc` (100 GiB)** — derived ML features; regenerable but compute-costly. KEEP (conservative).
+- All fxhnt/platform data: `fxhnt-backtest-data`, `fxhnt-surfer-data`, `multistrat-state`, `questdb-pvc`,
+  `tempo-data`, `netbird-data`, `foxhunt-gitlab-*` + `foxhunt-backups` buckets.
+General rule: delete only **pure compute/build artifacts** (caches, compiled binaries, run logs, GPU pools);
+**never** anything holding `.dbn`/market data or any potentially-reusable dataset.
+
+## Out of scope (KEEP — shared platform / Python fund)
+`ci-compile-cpu` pool (Python cockpit builds), platform pool, GitLab, MinIO, monitoring, Mattermost,
+Stalwart, Kanidm, NetBird, DNS, databases, cert-manager, tailscale proxy, public-gateway, dagster/cockpit,
+the fxhnt forward-track jobs. Phase 2 (IaC relocation to `fxhnt/infra` + TF-state move) is a separate spec.
+
+## Execution order (each destructive step verified + confirmed)
+1. **Terraform first**: edit terragrunt.hcl (3 `enable_*=false`), `terragrunt plan` → **STOP unless the
+   plan shows ONLY the 3 pools destroyed + 0 other destroys**; then `apply`.
+2. **Helm uninstalls** (GPU runners; main runner only after the use-check confirms dead).
+3. **PVC deletes** (re-confirm `Used By: <none>` immediately before each delete — irreversible).
+4. **Bucket deletes** (list contents first; irreversible — explicit confirm; `mc rb --force` via the
+   port-forward + minio creds).
+5. **Repo cleanup**: remove the dead manifests/templates + module var blocks, `kubectl delete wftmpl` the
+   dead templates, commit.
+
+## Risks / safety
+- **Irreversible**: PVC + bucket deletes, GPU-pool destroys. Mitigation: verify-unused immediately before
+  each; Terraform plan reviewed for 0-unexpected-destroys; explicit confirm on PVC/bucket deletes.
+- **Mis-scope risk**: a kept resource accidentally listed. Mitigation: the Terraform plan gate (only the 3
+  pools) + the `Used By` re-check + bucket content listing before delete.
+- **Main gitlab-runner**: don't uninstall without confirming no active consumer (could break a CI path).
+- Cluster health unaffected: nothing here touches the cockpit/dagster/fund/platform services.
+
+## Acceptance criteria
+- `terragrunt plan` (kapsule) shows the 3 GPU/precompute pools gone, then clean (no changes).
+- GPU-runner helm releases uninstalled; main runner resolved (kept or uninstalled per check).
+- The 5 build-cache PVCs deleted (cargo×3 + sccache×2, ~175 GB reclaimed); 3 Rust buckets deleted
+  (binaries, training-results, models). **`.dbn` market data untouched** (training-data-pvc + the
+  foxhunt-training-data bucket + test-data-pvc still present, byte-for-byte).
+- Dead training/gpu manifests + Argo templates removed from repo + cluster; committed.
+- Cockpit (`dashboard.fxhnt.ai` 200), dagster, and the fund tracks still healthy post-cleanup.