From 74615823ce31582307c0a2945a176086c3bea529 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Wed, 25 Feb 2026 21:09:31 +0100 Subject: [PATCH] docs: add DevPod implementation plan (10 tasks) Terraform pool, PVC, Dockerfile, devcontainer.json, pod template, CI job, setup script, infra apply, GitLab badge. TDD-style steps with exact file paths and commands. Co-Authored-By: Claude Opus 4.6 --- docs/plans/2026-02-25-devcontainer-plan.md | 659 +++++++++++++++++++++ 1 file changed, 659 insertions(+) create mode 100644 docs/plans/2026-02-25-devcontainer-plan.md diff --git a/docs/plans/2026-02-25-devcontainer-plan.md b/docs/plans/2026-02-25-devcontainer-plan.md new file mode 100644 index 000000000..4b0514d53 --- /dev/null +++ b/docs/plans/2026-02-25-devcontainer-plan.md @@ -0,0 +1,659 @@ +# DevPod Remote Development Environment — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** One-click remote dev environment on Kapsule with Claude Code, persistent storage, and autoscale-to-zero. + +**Architecture:** DevPod (client-side) provisions a pod on a dedicated GP1-L CPU pool via the kubernetes provider. A devcontainer image (extending the CI builder) is built by CI and pushed to the GitLab registry. PVCs persist home directory and cargo cache across sessions. "Open in DevPod" badge on GitLab project page. + +**Tech Stack:** DevPod, devcontainer spec, Terraform/Terragrunt (Kapsule), Kaniko (CI image build), Scaleway block storage (PVCs) + +**Design doc:** `docs/plans/2026-02-25-devcontainer-design.md` + +--- + +### Task 1: Add gpu-dev pool to Terraform + +**Files:** +- Modify: `infra/modules/kapsule/variables.tf` (append after line 71) +- Modify: `infra/modules/kapsule/main.tf` (append after line 90) +- Modify: `infra/live/production/kapsule/terragrunt.hcl` (add inputs) + +**Step 1: Add variables for the dev pool** + +Append to `infra/modules/kapsule/variables.tf` after the `gitlab_type` variable: + +```hcl +variable "enable_dev_pool" { + description = "Create CPU node pool for remote development (DevPod)" + type = bool + default = false +} + +variable "dev_pool_type" { + description = "Instance type for the dev node pool" + type = string + default = "GP1-L" +} + +variable "dev_pool_max_size" { + description = "Maximum number of nodes in the dev pool" + type = number + default = 1 +} +``` + +**Step 2: Add the pool resource** + +Append to `infra/modules/kapsule/main.tf` after the `gitlab` pool resource (after line 90): + +```hcl +# CPU pool for remote development (DevPod — autoscales to zero) +resource "scaleway_k8s_pool" "dev" { + count = var.enable_dev_pool ? 1 : 0 + cluster_id = scaleway_k8s_cluster.foxhunt.id + name = "gpu-dev" + node_type = var.dev_pool_type + size = 1 + min_size = 0 + max_size = var.dev_pool_max_size + autoscaling = true + autohealing = true + region = var.region + + lifecycle { + ignore_changes = [size] + } +} +``` + +**Step 3: Enable the dev pool in production inputs** + +Add to `infra/live/production/kapsule/terragrunt.hcl` inputs block: + +```hcl + # Dev pool (GP1-L for DevPod remote development, autoscales to zero) + enable_dev_pool = true + dev_pool_type = "GP1-L" + dev_pool_max_size = 1 +``` + +**Step 4: Validate terraform syntax** + +Run: `cd infra/modules/kapsule && terraform validate` +Expected: Success (or "no backend configured" which is fine for syntax check) + +**Step 5: Commit** + +```bash +git add infra/modules/kapsule/main.tf infra/modules/kapsule/variables.tf infra/live/production/kapsule/terragrunt.hcl +git commit -m "infra: add gpu-dev pool for DevPod remote development + +GP1-L (16 vCPU, 64GB), autoscaling 0→1, dedicated for dev pods. +Separate from CI (H100) and inference (L4) pools." +``` + +--- + +### Task 2: Create dev-home PVC manifest + +**Files:** +- Create: `infra/k8s/storage/dev-home-pvc.yaml` + +**Step 1: Create the PVC manifest** + +Follow the pattern from `infra/k8s/storage/training-data-pv.yaml` but use dynamic provisioning (no pre-existing volume — Scaleway CSI creates it on first claim): + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: dev-home-pvc + namespace: foxhunt + labels: + app.kubernetes.io/name: dev-home + app.kubernetes.io/part-of: foxhunt + app.kubernetes.io/component: devpod +spec: + storageClassName: scw-bssd + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 50Gi +``` + +**Step 2: Commit** + +```bash +git add infra/k8s/storage/dev-home-pvc.yaml +git commit -m "infra: add 50Gi PVC for DevPod home directory + +Persistent block storage for Claude Code, cargo registry, sccache +cache, and shell config. Survives pod restarts and node scale-down." +``` + +--- + +### Task 3: Create the devcontainer Dockerfile + +**Files:** +- Create: `.devcontainer/Dockerfile` + +**Step 1: Write the Dockerfile** + +Extends the CI builder image. Adds dev tooling (Node.js, Claude Code, Python, kubectl, helm, scw, glab, terraform, terragrunt, shell tools). Creates non-root `dev` user with sudo. + +```dockerfile +# Foxhunt devcontainer — extends CI builder with interactive dev tooling +# Built by CI, pushed to git.fxhnt.ai:5050/root/foxhunt/devcontainer:latest +# See: docs/plans/2026-02-25-devcontainer-design.md + +ARG CI_BUILDER_IMAGE=rg.fr-par.scw.cloud/foxhunt-ci/ci-builder:latest +FROM ${CI_BUILDER_IMAGE} + +ENV DEBIAN_FRONTEND=noninteractive + +# Dev tooling: Node.js 22 (Claude Code), Python 3, shell tools +RUN apt-get update && apt-get install -y --no-install-recommends \ + python3 \ + python3-pip \ + python3-venv \ + zsh \ + sudo \ + openssh-server \ + git-lfs \ + jq \ + ripgrep \ + fd-find \ + htop \ + less \ + vim \ + && rm -rf /var/lib/apt/lists/* + +# Node.js 22 LTS (Claude Code runtime) +RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \ + && apt-get install -y nodejs \ + && rm -rf /var/lib/apt/lists/* + +# Claude Code CLI +RUN npm install -g @anthropic-ai/claude-code + +# kubectl +RUN curl -fsSL "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" \ + -o /usr/local/bin/kubectl \ + && chmod +x /usr/local/bin/kubectl + +# helm +RUN curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash + +# scw CLI (Scaleway) +RUN curl -fsSL https://github.com/scaleway/scaleway-cli/releases/latest/download/scaleway-cli_linux_amd64 \ + -o /usr/local/bin/scw \ + && chmod +x /usr/local/bin/scw + +# glab (GitLab CLI) +RUN curl -fsSL "https://gitlab.com/gitlab-org/cli/-/releases/permalink/latest/downloads/glab_linux_amd64.deb" \ + -o /tmp/glab.deb \ + && dpkg -i /tmp/glab.deb \ + && rm /tmp/glab.deb + +# terraform +RUN curl -fsSL https://releases.hashicorp.com/terraform/1.11.2/terraform_1.11.2_linux_amd64.zip \ + -o /tmp/terraform.zip \ + && unzip -o /tmp/terraform.zip -d /usr/local/bin \ + && rm /tmp/terraform.zip + +# terragrunt +RUN curl -fsSL https://github.com/gruntwork-io/terragrunt/releases/download/v0.72.6/terragrunt_linux_amd64 \ + -o /usr/local/bin/terragrunt \ + && chmod +x /usr/local/bin/terragrunt + +# yq (YAML processor) +RUN curl -fsSL https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 \ + -o /usr/local/bin/yq \ + && chmod +x /usr/local/bin/yq + +# Non-root dev user with sudo +RUN groupadd -g 1000 dev \ + && useradd -m -u 1000 -g dev -s /bin/zsh dev \ + && echo "dev ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/dev + +# Install oh-my-zsh for dev user +RUN su - dev -c 'sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" "" --unattended' + +# Move Rust toolchain to shared location so dev user can access it +# (CI builder installs to /root/.cargo and /root/.rustup) +RUN cp -a /root/.cargo /opt/cargo \ + && cp -a /root/.rustup /opt/rustup \ + && chown -R dev:dev /opt/cargo /opt/rustup + +# Environment for dev user +ENV CARGO_HOME=/home/dev/.cargo +ENV RUSTUP_HOME=/opt/rustup +ENV PATH="/home/dev/.cargo/bin:/opt/rustup/bin:/opt/cargo/bin:${PATH}" +ENV SQLX_OFFLINE=true +ENV CUDA_COMPUTE_CAP=90 + +# sccache local disk cache (PVC-backed home dir) +ENV SCCACHE_DIR=/home/dev/.cache/sccache +ENV RUSTC_WRAPPER=/usr/local/bin/sccache + +# First-run init script: copies cargo bins to home if not already present +COPY .devcontainer/init-home.sh /usr/local/bin/init-home.sh +RUN chmod +x /usr/local/bin/init-home.sh + +USER dev +WORKDIR /workspace + +# Verify key tools +RUN rustc --version && cargo --version && node --version && claude --version \ + && kubectl version --client 2>/dev/null && helm version --short \ + && terraform --version && terragrunt --version && scw version && glab --version +``` + +**Step 2: Commit** + +```bash +git add .devcontainer/Dockerfile +git commit -m "feat(devcontainer): add Dockerfile extending CI builder + +Adds Node.js 22, Claude Code, Python 3, kubectl, helm, scw, glab, +terraform, terragrunt, zsh, ripgrep, fd-find. Non-root dev user." +``` + +--- + +### Task 4: Create the home directory init script + +**Files:** +- Create: `.devcontainer/init-home.sh` + +**Step 1: Write the init script** + +On first boot with a fresh PVC, the home directory is empty. This script +bootstraps cargo toolchain links and shell config. On subsequent boots it's +a no-op. + +```bash +#!/bin/bash +set -euo pipefail + +# Bootstrap cargo into PVC-backed home if not already present +if [ ! -d "$HOME/.cargo/bin" ]; then + echo "[init-home] First run: linking cargo toolchain..." + mkdir -p "$HOME/.cargo/bin" + # Copy cargo config and link binaries from shared rustup + for bin in /opt/cargo/bin/*; do + ln -sf "$bin" "$HOME/.cargo/bin/$(basename "$bin")" + done + echo "[init-home] Cargo toolchain linked." +fi + +# Ensure sccache cache dir exists +mkdir -p "${SCCACHE_DIR:-$HOME/.cache/sccache}" + +# Default .zshrc if none exists (oh-my-zsh is in the image, config in PVC) +if [ ! -f "$HOME/.zshrc" ]; then + cp /home/dev/.oh-my-zsh/templates/zshrc.zsh-template "$HOME/.zshrc" 2>/dev/null || true + echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$HOME/.zshrc" + echo 'export SQLX_OFFLINE=true' >> "$HOME/.zshrc" +fi + +echo "[init-home] Ready." +``` + +**Step 2: Commit** + +```bash +git add .devcontainer/init-home.sh +git commit -m "feat(devcontainer): add init-home.sh for PVC bootstrap + +Links cargo binaries, creates sccache dir, and writes default .zshrc +on first boot. No-op on subsequent sessions." +``` + +--- + +### Task 5: Create the pod template for DevPod + +**Files:** +- Create: `.devcontainer/pod-template.yaml` + +**Step 1: Write the pod manifest template** + +DevPod's `POD_MANIFEST_TEMPLATE` lets us add PVC mounts and node selection +that devcontainer.json cannot express. DevPod merges this with its own pod spec. + +```yaml +# DevPod pod template — merged with DevPod's generated pod spec +# Ref: https://github.com/loft-sh/devpod-provider-kubernetes +apiVersion: v1 +kind: Pod +metadata: + labels: + app.kubernetes.io/name: devpod + app.kubernetes.io/part-of: foxhunt + app.kubernetes.io/component: dev-environment +spec: + nodeSelector: + k8s.scaleway.com/pool-name: gpu-dev + containers: + - name: devpod + resources: + requests: + cpu: "4000m" + memory: "16Gi" + limits: + cpu: "14000m" + memory: "56Gi" + volumeMounts: + - name: dev-home + mountPath: /home/dev + - name: training-data + mountPath: /data/training + readOnly: true + initContainers: + - name: init-home + image: busybox:latest + command: ["sh", "-c", "chown -R 1000:1000 /home/dev"] + volumeMounts: + - name: dev-home + mountPath: /home/dev + volumes: + - name: dev-home + persistentVolumeClaim: + claimName: dev-home-pvc + - name: training-data + persistentVolumeClaim: + claimName: training-data-pvc +``` + +**Step 2: Commit** + +```bash +git add .devcontainer/pod-template.yaml +git commit -m "feat(devcontainer): add k8s pod template for DevPod + +Mounts dev-home PVC (50Gi) and training-data PVC (read-only). +Targets gpu-dev node pool. Resource requests: 4 CPU / 16Gi, +limits: 14 CPU / 56Gi." +``` + +--- + +### Task 6: Create devcontainer.json + +**Files:** +- Create: `.devcontainer/devcontainer.json` + +**Step 1: Write devcontainer.json** + +```json +{ + "name": "foxhunt", + "build": { + "dockerfile": "Dockerfile", + "context": "..", + "args": { + "CI_BUILDER_IMAGE": "rg.fr-par.scw.cloud/foxhunt-ci/ci-builder:latest" + } + }, + "remoteUser": "dev", + "containerEnv": { + "SQLX_OFFLINE": "true", + "CUDA_COMPUTE_CAP": "90", + "EDITOR": "vim" + }, + "postCreateCommand": "/usr/local/bin/init-home.sh", + "postStartCommand": "claude --version && cargo --version && kubectl cluster-info 2>/dev/null || true" +} +``` + +**Step 2: Commit** + +```bash +git add .devcontainer/devcontainer.json +git commit -m "feat(devcontainer): add devcontainer.json + +Builds from CI builder base, runs as non-root dev user. +postCreateCommand bootstraps PVC home dir on first session." +``` + +--- + +### Task 7: Add build-devcontainer CI job + +**Files:** +- Modify: `.gitlab-ci.yml` (insert after `build-ci-builder` job, around line 60) + +**Step 1: Add the CI job** + +Insert after the `build-ci-builder` job block (after line 59) and before the `.rust-base` template: + +```yaml +# -------------------------------------------------------------------------- +# Stage 0b: Build devcontainer image → push to internal GitLab registry +# -------------------------------------------------------------------------- +build-devcontainer: + stage: prepare + image: + name: gcr.io/kaniko-project/executor:debug + entrypoint: [""] + tags: + - kapsule + - docker + rules: + - if: $CI_PIPELINE_SOURCE == "push" + changes: + - .devcontainer/** + when: on_success + - when: manual + allow_failure: true + before_script: + - mkdir -p /kaniko/.docker + - | + echo "{\"auths\":{\"${INTERNAL_REGISTRY}\":{\"username\":\"${CI_REGISTRY_USER}\",\"password\":\"${CI_REGISTRY_PASSWORD}\"},\"rg.fr-par.scw.cloud\":{\"username\":\"nologin\",\"password\":\"${SCW_SECRET_KEY}\"}}}" > /kaniko/.docker/config.json + script: + - /kaniko/executor + --context "${CI_PROJECT_DIR}" + --dockerfile "${CI_PROJECT_DIR}/.devcontainer/Dockerfile" + --insecure-registry "${INTERNAL_REGISTRY}" + --destination "${REGISTRY}/devcontainer:${CI_COMMIT_SHA}" + --destination "${REGISTRY}/devcontainer:latest" +``` + +**Step 2: Verify YAML syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))"` +Expected: No errors + +**Step 3: Commit** + +```bash +git add .gitlab-ci.yml +git commit -m "ci: add build-devcontainer job + +Builds .devcontainer/Dockerfile with Kaniko, pushes to internal +GitLab registry. Triggers on .devcontainer/ changes or manual run." +``` + +--- + +### Task 8: Create DevPod setup script + +**Files:** +- Create: `scripts/devpod-setup.sh` + +**Step 1: Write the setup script** + +One-time setup to run on the developer's laptop. Configures DevPod with the +kubernetes provider pointing at the foxhunt cluster. + +```bash +#!/bin/bash +set -euo pipefail + +# DevPod one-time setup for Foxhunt development +# Prerequisites: devpod CLI installed, kubectl configured for foxhunt cluster +# +# Install DevPod: +# curl -fsSL https://devpod.sh/install.sh | bash +# +# Usage: +# ./scripts/devpod-setup.sh +# devpod up . # from the foxhunt repo root +# devpod up git@100.90.76.85:2222/root/foxhunt.git # from anywhere + +echo "=== Foxhunt DevPod Setup ===" + +# Check prerequisites +command -v devpod >/dev/null 2>&1 || { echo "Error: devpod not installed. Run: curl -fsSL https://devpod.sh/install.sh | bash"; exit 1; } +command -v kubectl >/dev/null 2>&1 || { echo "Error: kubectl not found"; exit 1; } + +# Verify cluster access +if ! kubectl -n foxhunt get ns foxhunt >/dev/null 2>&1; then + echo "Error: Cannot reach foxhunt namespace. Check kubectl config." + exit 1 +fi + +# Ensure PVC exists +if ! kubectl -n foxhunt get pvc dev-home-pvc >/dev/null 2>&1; then + echo "Creating dev-home-pvc (50Gi block storage)..." + kubectl apply -f infra/k8s/storage/dev-home-pvc.yaml +fi + +# Add kubernetes provider (idempotent) +devpod provider add kubernetes 2>/dev/null || true + +# Configure provider +devpod provider set kubernetes \ + KUBERNETES_NAMESPACE=foxhunt \ + NODE_SELECTOR="k8s.scaleway.com/pool-name=gpu-dev" \ + DISK_SIZE=10Gi \ + STORAGE_CLASS=scw-bssd \ + INACTIVITY_TIMEOUT=30m \ + POD_MANIFEST_TEMPLATE=.devcontainer/pod-template.yaml + +# Configure image pull secrets if not already present +if ! kubectl -n foxhunt get secret gitlab-registry >/dev/null 2>&1; then + echo "" + echo "NOTE: You need to create the gitlab-registry pull secret:" + echo " kubectl -n foxhunt create secret docker-registry gitlab-registry \\" + echo " --docker-server=gitlab-registry.foxhunt.svc.cluster.local:5000 \\" + echo " --docker-username= --docker-password=" + echo "" +fi + +echo "" +echo "=== Setup complete ===" +echo "" +echo "Usage:" +echo " devpod up . # from repo root" +echo " devpod up git@100.90.76.85:2222/root/foxhunt.git # from anywhere" +echo "" +echo "Open in DevPod link (bookmark this):" +echo " https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git" +echo "" +echo "Session lifecycle:" +echo " devpod up foxhunt # start/reconnect" +echo " devpod stop foxhunt # stop (PVC preserved, node scales down)" +echo " devpod delete foxhunt # delete workspace (PVC preserved)" +echo "" +``` + +**Step 2: Make executable** + +Run: `chmod +x scripts/devpod-setup.sh` + +**Step 3: Commit** + +```bash +git add scripts/devpod-setup.sh +git commit -m "feat: add devpod-setup.sh for one-time provider config + +Configures DevPod kubernetes provider, creates dev-home PVC, +verifies cluster access. Run once on developer laptop." +``` + +--- + +### Task 9: Apply infrastructure (manual — requires cluster access) + +This task runs `terragrunt apply` and `kubectl apply`. It is manual because +it modifies live infrastructure. + +**Step 1: Apply Terraform changes** + +Run from a machine with Scaleway credentials: + +```bash +cd infra/live/production/kapsule +terragrunt plan # review the new gpu-dev pool +terragrunt apply # create the pool (starts at 0 nodes) +``` + +Expected: New `gpu-dev` pool created with 0 nodes. + +**Step 2: Apply PVC** + +```bash +kubectl apply -f infra/k8s/storage/dev-home-pvc.yaml +``` + +Expected: PVC `dev-home-pvc` created in `foxhunt` namespace, status `Pending` +(binds on first pod mount). + +**Step 3: Build devcontainer image (first time)** + +Trigger the CI job manually from GitLab UI, or push a commit touching +`.devcontainer/`. + +**Step 4: Run DevPod setup** + +```bash +./scripts/devpod-setup.sh +``` + +**Step 5: Test the full cycle** + +```bash +devpod up . +# Should: provision GP1-L node (~2-3 min), start pod, SSH in +# Verify inside the pod: +claude --version +cargo --version +kubectl get nodes +``` + +**Step 6: Test idle scale-down** + +Disconnect from the pod. Wait 30 min for DevPod inactivity timeout. +Then wait 10 min for Kapsule autoscaler to remove the node. +Verify: `kubectl get nodes` shows no gpu-dev node. + +**Step 7: Test reconnect** + +```bash +devpod up foxhunt +# Should: provision new node, mount same PVCs, all state preserved +``` + +--- + +### Task 10: Add GitLab project badge + +This is done via the GitLab UI (Settings → General → Badges). + +**Step 1: Add badge** + +``` +Badge name: Open in DevPod +Badge link: https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git +Badge image: https://devpod.sh/assets/open-in-devpod.svg +``` + +**Step 2: Verify** + +Navigate to the project page. The "Open in DevPod" badge should appear. +Click it — DevPod should open (if installed) and begin provisioning.