# Cockpit + NAT Gateway — Apply Order

## Prerequisites
- `scw` CLI authenticated
- `terragrunt` installed
- `kubectl` access to foxhunt cluster
- `helm` installed

## Phase 1: Cockpit (metrics backend)

1. Apply Cockpit Terraform module:
   ```bash
   cd infra/live/production/cockpit
   terragrunt apply
   ```

2. Note the outputs:
   ```bash
   terragrunt output metrics_push_url
   terragrunt output -raw push_token
   terragrunt output grafana_url
   terragrunt output -raw grafana_password
   ```

3. Create the K8s secret for the push token:
   ```bash
   kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
   kubectl -n monitoring create secret generic cockpit-push-token \
     --from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
   ```

4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`:
   - Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output

5. Deploy DCGM Exporter:
   ```bash
   kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
   ```

6. Deploy Grafana Alloy (k8s-monitoring):
   ```bash
   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   helm install k8s-monitoring grafana/k8s-monitoring \
     --namespace monitoring \
     -f infra/k8s/monitoring/alloy-values.yaml
   ```

7. Verify metrics flow:
   - Open Cockpit Grafana (URL from step 2)
   - Login with foxhunt-admin / (password from step 2)
   - Check "Kubernetes / Compute Resources / Cluster" dashboard

## Phase 2: GitLab Prometheus remote_write

1. Update placeholders in `infra/k8s/gitlab/values.yaml`:
   - Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN`

2. Upgrade GitLab Helm release:
   ```bash
   helm upgrade gitlab gitlab/gitlab \
     --namespace foxhunt \
     -f infra/k8s/gitlab/values.yaml
   ```

## Phase 3: NAT Gateway

1. Apply Public Gateway module:
   ```bash
   cd infra/live/production/public-gateway
   terragrunt apply
   ```
   This creates the gateway and attaches it to the private network.
   Existing nodes still have public IPs — nothing breaks yet.

2. Apply Kapsule pool changes (disables public IPs):
   ```bash
   cd infra/live/production/kapsule
   terragrunt apply
   ```
   **WARNING**: This may recreate nodes. For zero-downtime:

3. Zero-downtime node migration for services pool:
   ```bash
   # Temporarily allow 2 nodes
   scw k8s pool update <services-pool-id> max-size=2 size=2
   # Wait for new node (private IP) to be Ready
   kubectl get nodes -w
   # Drain old node
   kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
   # Scale back
   scw k8s pool update <services-pool-id> max-size=1 size=1
   ```

4. Repeat for gitlab pool (if not autoscaling, same process).

5. Autoscaling pools (ci-training, gpu-dev):
   - No action needed — next scale-up automatically uses private IP.

6. Verify Tailscale still works:
   ```bash
   tailscale status  # should show foxhunt-kapsule and foxhunt-gitlab nodes
   curl -k https://git.fxhnt.ai  # should still resolve via Tailscale
   ```