Files

jgrusewski f6cf857db2 fix(infra): remove unused L4 (ci-rl) pool — consolidate all training on ci-training

The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.

Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-01 14:02:54 +01:00

3.0 KiB

Raw Blame History

Cockpit + NAT Gateway — Apply Order

Prerequisites

scw CLI authenticated
terragrunt installed
kubectl access to foxhunt cluster
helm installed

Phase 1: Cockpit (metrics backend)

Apply Cockpit Terraform module:

cd infra/live/production/cockpit
terragrunt apply

Note the outputs:

terragrunt output metrics_push_url
terragrunt output -raw push_token
terragrunt output grafana_url
terragrunt output -raw grafana_password

Create the K8s secret for the push token:

kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
kubectl -n monitoring create secret generic cockpit-push-token \
  --from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)

Update placeholders in infra/k8s/monitoring/alloy-values.yaml:
- Replace COCKPIT_METRICS_PUSH_URL with the metrics_push_url output

Deploy DCGM Exporter:

kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml

Deploy Grafana Alloy (k8s-monitoring):

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install k8s-monitoring grafana/k8s-monitoring \
  --namespace monitoring \
  -f infra/k8s/monitoring/alloy-values.yaml

Verify metrics flow:
- Open Cockpit Grafana (URL from step 2)
- Login with foxhunt-admin / (password from step 2)
- Check "Kubernetes / Compute Resources / Cluster" dashboard

Phase 2: GitLab Prometheus remote_write

Update placeholders in infra/k8s/gitlab/values.yaml:
- Replace COCKPIT_METRICS_PUSH_URL and COCKPIT_PUSH_TOKEN

Upgrade GitLab Helm release:

helm upgrade gitlab gitlab/gitlab \
  --namespace foxhunt \
  -f infra/k8s/gitlab/values.yaml

Phase 3: NAT Gateway

Apply Public Gateway module:
```
cd infra/live/production/public-gateway
terragrunt apply
```
This creates the gateway and attaches it to the private network. Existing nodes still have public IPs — nothing breaks yet.
Apply Kapsule pool changes (disables public IPs):
```
cd infra/live/production/kapsule
terragrunt apply
```
WARNING: This may recreate nodes. For zero-downtime:

Zero-downtime node migration for services pool:

# Temporarily allow 2 nodes
scw k8s pool update <services-pool-id> max-size=2 size=2
# Wait for new node (private IP) to be Ready
kubectl get nodes -w
# Drain old node
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
# Scale back
scw k8s pool update <services-pool-id> max-size=1 size=1

Repeat for gitlab pool (if not autoscaling, same process).
Autoscaling pools (ci-training, gpu-dev):
- No action needed — next scale-up automatically uses private IP.

Verify Tailscale still works:

tailscale status  # should show foxhunt-kapsule and foxhunt-gitlab nodes
curl -k https://git.fxhnt.ai  # should still resolve via Tailscale

3.0 KiB Raw Blame History