f6cf857db2
The L4 pool was unused: all training routes to ci-training (L40S) and all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt and applied to destroy the pool. Removed all ci-rl references from K8s manifests and CI comments. Final pool layout: - ci-compile-cpu (POP2-32C-128G) — Rust compilation - ci-training (L40S-1-48G) — all GPU training + hyperopt - services (DEV1-L) — production services - gitlab (GP1-XS) — GitLab CE - gpu-dev (GP1-L) — DevPod development Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
105 lines
3.0 KiB
Markdown
105 lines
3.0 KiB
Markdown
# Cockpit + NAT Gateway — Apply Order
|
|
|
|
## Prerequisites
|
|
- `scw` CLI authenticated
|
|
- `terragrunt` installed
|
|
- `kubectl` access to foxhunt cluster
|
|
- `helm` installed
|
|
|
|
## Phase 1: Cockpit (metrics backend)
|
|
|
|
1. Apply Cockpit Terraform module:
|
|
```bash
|
|
cd infra/live/production/cockpit
|
|
terragrunt apply
|
|
```
|
|
|
|
2. Note the outputs:
|
|
```bash
|
|
terragrunt output metrics_push_url
|
|
terragrunt output -raw push_token
|
|
terragrunt output grafana_url
|
|
terragrunt output -raw grafana_password
|
|
```
|
|
|
|
3. Create the K8s secret for the push token:
|
|
```bash
|
|
kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
|
|
kubectl -n monitoring create secret generic cockpit-push-token \
|
|
--from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
|
|
```
|
|
|
|
4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`:
|
|
- Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output
|
|
|
|
5. Deploy DCGM Exporter:
|
|
```bash
|
|
kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
|
|
```
|
|
|
|
6. Deploy Grafana Alloy (k8s-monitoring):
|
|
```bash
|
|
helm repo add grafana https://grafana.github.io/helm-charts
|
|
helm repo update
|
|
helm install k8s-monitoring grafana/k8s-monitoring \
|
|
--namespace monitoring \
|
|
-f infra/k8s/monitoring/alloy-values.yaml
|
|
```
|
|
|
|
7. Verify metrics flow:
|
|
- Open Cockpit Grafana (URL from step 2)
|
|
- Login with foxhunt-admin / (password from step 2)
|
|
- Check "Kubernetes / Compute Resources / Cluster" dashboard
|
|
|
|
## Phase 2: GitLab Prometheus remote_write
|
|
|
|
1. Update placeholders in `infra/k8s/gitlab/values.yaml`:
|
|
- Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN`
|
|
|
|
2. Upgrade GitLab Helm release:
|
|
```bash
|
|
helm upgrade gitlab gitlab/gitlab \
|
|
--namespace foxhunt \
|
|
-f infra/k8s/gitlab/values.yaml
|
|
```
|
|
|
|
## Phase 3: NAT Gateway
|
|
|
|
1. Apply Public Gateway module:
|
|
```bash
|
|
cd infra/live/production/public-gateway
|
|
terragrunt apply
|
|
```
|
|
This creates the gateway and attaches it to the private network.
|
|
Existing nodes still have public IPs — nothing breaks yet.
|
|
|
|
2. Apply Kapsule pool changes (disables public IPs):
|
|
```bash
|
|
cd infra/live/production/kapsule
|
|
terragrunt apply
|
|
```
|
|
**WARNING**: This may recreate nodes. For zero-downtime:
|
|
|
|
3. Zero-downtime node migration for services pool:
|
|
```bash
|
|
# Temporarily allow 2 nodes
|
|
scw k8s pool update <services-pool-id> max-size=2 size=2
|
|
# Wait for new node (private IP) to be Ready
|
|
kubectl get nodes -w
|
|
# Drain old node
|
|
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
|
|
# Scale back
|
|
scw k8s pool update <services-pool-id> max-size=1 size=1
|
|
```
|
|
|
|
4. Repeat for gitlab pool (if not autoscaling, same process).
|
|
|
|
5. Autoscaling pools (ci-training, gpu-dev):
|
|
- No action needed — next scale-up automatically uses private IP.
|
|
|
|
6. Verify Tailscale still works:
|
|
```bash
|
|
tailscale status # should show foxhunt-kapsule and foxhunt-gitlab nodes
|
|
curl -k https://git.fxhnt.ai # should still resolve via Tailscale
|
|
```
|