2026-02-25 21:09:31 +01:00
parent f3a3b61049
commit 74615823ce
1 changed files with 659 additions and 0 deletions
							
							
							
						
@@ -0,0 +1,659 @@
# DevPod Remote Development Environment — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** One-click remote dev environment on Kapsule with Claude Code, persistent storage, and autoscale-to-zero.
**Architecture:** DevPod (client-side) provisions a pod on a dedicated GP1-L CPU pool via the kubernetes provider. A devcontainer image (extending the CI builder) is built by CI and pushed to the GitLab registry. PVCs persist home directory and cargo cache across sessions. "Open in DevPod" badge on GitLab project page.
**Tech Stack:** DevPod, devcontainer spec, Terraform/Terragrunt (Kapsule), Kaniko (CI image build), Scaleway block storage (PVCs)
**Design doc:** `docs/plans/2026-02-25-devcontainer-design.md`
---
### Task 1: Add gpu-dev pool to Terraform
**Files:**
- Modify: `infra/modules/kapsule/variables.tf` (append after line 71)
- Modify: `infra/modules/kapsule/main.tf` (append after line 90)
- Modify: `infra/live/production/kapsule/terragrunt.hcl` (add inputs)
**Step 1: Add variables for the dev pool**
Append to `infra/modules/kapsule/variables.tf` after the `gitlab_type` variable:
```hcl
variable "enable_dev_pool" {
  description = "Create CPU node pool for remote development (DevPod)"
  type        = bool
  default     = false
}
variable "dev_pool_type" {
  description = "Instance type for the dev node pool"
  type        = string
  default     = "GP1-L"
}
variable "dev_pool_max_size" {
  description = "Maximum number of nodes in the dev pool"
  type        = number
  default     = 1
}
```
**Step 2: Add the pool resource**
Append to `infra/modules/kapsule/main.tf` after the `gitlab` pool resource (after line 90):
```hcl
# CPU pool for remote development (DevPod — autoscales to zero)
resource "scaleway_k8s_pool" "dev" {
  count       = var.enable_dev_pool ? 1 : 0
  cluster_id  = scaleway_k8s_cluster.foxhunt.id
  name        = "gpu-dev"
  node_type   = var.dev_pool_type
  size        = 1
  min_size    = 0
  max_size    = var.dev_pool_max_size
  autoscaling = true
  autohealing = true
  region      = var.region
  lifecycle {
    ignore_changes = [size]
  }
}
```
**Step 3: Enable the dev pool in production inputs**
Add to `infra/live/production/kapsule/terragrunt.hcl` inputs block:
```hcl
  # Dev pool (GP1-L for DevPod remote development, autoscales to zero)
  enable_dev_pool  = true
  dev_pool_type    = "GP1-L"
  dev_pool_max_size = 1
```
**Step 4: Validate terraform syntax**
Run: `cd infra/modules/kapsule && terraform validate`
Expected: Success (or "no backend configured" which is fine for syntax check)
**Step 5: Commit**
```bash
git add infra/modules/kapsule/main.tf infra/modules/kapsule/variables.tf infra/live/production/kapsule/terragrunt.hcl
git commit -m "infra: add gpu-dev pool for DevPod remote development
GP1-L (16 vCPU, 64GB), autoscaling 0→1, dedicated for dev pods.
Separate from CI (H100) and inference (L4) pools."
```
---
### Task 2: Create dev-home PVC manifest
**Files:**
- Create: `infra/k8s/storage/dev-home-pvc.yaml`
**Step 1: Create the PVC manifest**
Follow the pattern from `infra/k8s/storage/training-data-pv.yaml` but use dynamic provisioning (no pre-existing volume — Scaleway CSI creates it on first claim):
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dev-home-pvc
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: dev-home
    app.kubernetes.io/part-of: foxhunt
    app.kubernetes.io/component: devpod
spec:
  storageClassName: scw-bssd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
```
**Step 2: Commit**
```bash
git add infra/k8s/storage/dev-home-pvc.yaml
git commit -m "infra: add 50Gi PVC for DevPod home directory
Persistent block storage for Claude Code, cargo registry, sccache
cache, and shell config. Survives pod restarts and node scale-down."
```
---
### Task 3: Create the devcontainer Dockerfile
**Files:**
- Create: `.devcontainer/Dockerfile`
**Step 1: Write the Dockerfile**
Extends the CI builder image. Adds dev tooling (Node.js, Claude Code, Python, kubectl, helm, scw, glab, terraform, terragrunt, shell tools). Creates non-root `dev` user with sudo.
```dockerfile
# Foxhunt devcontainer — extends CI builder with interactive dev tooling
# Built by CI, pushed to git.fxhnt.ai:5050/root/foxhunt/devcontainer:latest
# See: docs/plans/2026-02-25-devcontainer-design.md
ARG CI_BUILDER_IMAGE=rg.fr-par.scw.cloud/foxhunt-ci/ci-builder:latest
FROM ${CI_BUILDER_IMAGE}
ENV DEBIAN_FRONTEND=noninteractive
# Dev tooling: Node.js 22 (Claude Code), Python 3, shell tools
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 \
    python3-pip \
    python3-venv \
    zsh \
    sudo \
    openssh-server \
    git-lfs \
    jq \
    ripgrep \
    fd-find \
    htop \
    less \
    vim \
    && rm -rf /var/lib/apt/lists/*
# Node.js 22 LTS (Claude Code runtime)
RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
    && apt-get install -y nodejs \
    && rm -rf /var/lib/apt/lists/*
# Claude Code CLI
RUN npm install -g @anthropic-ai/claude-code
# kubectl
RUN curl -fsSL "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" \
    -o /usr/local/bin/kubectl \
    && chmod +x /usr/local/bin/kubectl
# helm
RUN curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
# scw CLI (Scaleway)
RUN curl -fsSL https://github.com/scaleway/scaleway-cli/releases/latest/download/scaleway-cli_linux_amd64 \
    -o /usr/local/bin/scw \
    && chmod +x /usr/local/bin/scw
# glab (GitLab CLI)
RUN curl -fsSL "https://gitlab.com/gitlab-org/cli/-/releases/permalink/latest/downloads/glab_linux_amd64.deb" \
    -o /tmp/glab.deb \
    && dpkg -i /tmp/glab.deb \
    && rm /tmp/glab.deb
# terraform
RUN curl -fsSL https://releases.hashicorp.com/terraform/1.11.2/terraform_1.11.2_linux_amd64.zip \
    -o /tmp/terraform.zip \
    && unzip -o /tmp/terraform.zip -d /usr/local/bin \
    && rm /tmp/terraform.zip
# terragrunt
RUN curl -fsSL https://github.com/gruntwork-io/terragrunt/releases/download/v0.72.6/terragrunt_linux_amd64 \
    -o /usr/local/bin/terragrunt \
    && chmod +x /usr/local/bin/terragrunt
# yq (YAML processor)
RUN curl -fsSL https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 \
    -o /usr/local/bin/yq \
    && chmod +x /usr/local/bin/yq
# Non-root dev user with sudo
RUN groupadd -g 1000 dev \
    && useradd -m -u 1000 -g dev -s /bin/zsh dev \
    && echo "dev ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/dev
# Install oh-my-zsh for dev user
RUN su - dev -c 'sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" "" --unattended'
# Move Rust toolchain to shared location so dev user can access it
# (CI builder installs to /root/.cargo and /root/.rustup)
RUN cp -a /root/.cargo /opt/cargo \
    && cp -a /root/.rustup /opt/rustup \
    && chown -R dev:dev /opt/cargo /opt/rustup
# Environment for dev user
ENV CARGO_HOME=/home/dev/.cargo
ENV RUSTUP_HOME=/opt/rustup
ENV PATH="/home/dev/.cargo/bin:/opt/rustup/bin:/opt/cargo/bin:${PATH}"
ENV SQLX_OFFLINE=true
ENV CUDA_COMPUTE_CAP=90
# sccache local disk cache (PVC-backed home dir)
ENV SCCACHE_DIR=/home/dev/.cache/sccache
ENV RUSTC_WRAPPER=/usr/local/bin/sccache
# First-run init script: copies cargo bins to home if not already present
COPY .devcontainer/init-home.sh /usr/local/bin/init-home.sh
RUN chmod +x /usr/local/bin/init-home.sh
USER dev
WORKDIR /workspace
# Verify key tools
RUN rustc --version && cargo --version && node --version && claude --version \
    && kubectl version --client 2>/dev/null && helm version --short \
    && terraform --version && terragrunt --version && scw version && glab --version
```
**Step 2: Commit**
```bash
git add .devcontainer/Dockerfile
git commit -m "feat(devcontainer): add Dockerfile extending CI builder
Adds Node.js 22, Claude Code, Python 3, kubectl, helm, scw, glab,
terraform, terragrunt, zsh, ripgrep, fd-find. Non-root dev user."
```
---
### Task 4: Create the home directory init script
**Files:**
- Create: `.devcontainer/init-home.sh`
**Step 1: Write the init script**
On first boot with a fresh PVC, the home directory is empty. This script
bootstraps cargo toolchain links and shell config. On subsequent boots it's
a no-op.
```bash
#!/bin/bash
set -euo pipefail
# Bootstrap cargo into PVC-backed home if not already present
if [ ! -d "$HOME/.cargo/bin" ]; then
    echo "[init-home] First run: linking cargo toolchain..."
    mkdir -p "$HOME/.cargo/bin"
    # Copy cargo config and link binaries from shared rustup
    for bin in /opt/cargo/bin/*; do
        ln -sf "$bin" "$HOME/.cargo/bin/$(basename "$bin")"
    done
    echo "[init-home] Cargo toolchain linked."
fi
# Ensure sccache cache dir exists
mkdir -p "${SCCACHE_DIR:-$HOME/.cache/sccache}"
# Default .zshrc if none exists (oh-my-zsh is in the image, config in PVC)
if [ ! -f "$HOME/.zshrc" ]; then
    cp /home/dev/.oh-my-zsh/templates/zshrc.zsh-template "$HOME/.zshrc" 2>/dev/null || true
    echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$HOME/.zshrc"
    echo 'export SQLX_OFFLINE=true' >> "$HOME/.zshrc"
fi
echo "[init-home] Ready."
```
**Step 2: Commit**
```bash
git add .devcontainer/init-home.sh
git commit -m "feat(devcontainer): add init-home.sh for PVC bootstrap
Links cargo binaries, creates sccache dir, and writes default .zshrc
on first boot. No-op on subsequent sessions."
```
---
### Task 5: Create the pod template for DevPod
**Files:**
- Create: `.devcontainer/pod-template.yaml`
**Step 1: Write the pod manifest template**
DevPod's `POD_MANIFEST_TEMPLATE` lets us add PVC mounts and node selection
that devcontainer.json cannot express. DevPod merges this with its own pod spec.
```yaml
# DevPod pod template — merged with DevPod's generated pod spec
# Ref: https://github.com/loft-sh/devpod-provider-kubernetes
apiVersion: v1
kind: Pod
metadata:
  labels:
    app.kubernetes.io/name: devpod
    app.kubernetes.io/part-of: foxhunt
    app.kubernetes.io/component: dev-environment
spec:
  nodeSelector:
    k8s.scaleway.com/pool-name: gpu-dev
  containers:
    - name: devpod
      resources:
        requests:
          cpu: "4000m"
          memory: "16Gi"
        limits:
          cpu: "14000m"
          memory: "56Gi"
      volumeMounts:
        - name: dev-home
          mountPath: /home/dev
        - name: training-data
          mountPath: /data/training
          readOnly: true
  initContainers:
    - name: init-home
      image: busybox:latest
      command: ["sh", "-c", "chown -R 1000:1000 /home/dev"]
      volumeMounts:
        - name: dev-home
          mountPath: /home/dev
  volumes:
    - name: dev-home
      persistentVolumeClaim:
        claimName: dev-home-pvc
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc
```
**Step 2: Commit**
```bash
git add .devcontainer/pod-template.yaml
git commit -m "feat(devcontainer): add k8s pod template for DevPod
Mounts dev-home PVC (50Gi) and training-data PVC (read-only).
Targets gpu-dev node pool. Resource requests: 4 CPU / 16Gi,
limits: 14 CPU / 56Gi."
```
---
### Task 6: Create devcontainer.json
**Files:**
- Create: `.devcontainer/devcontainer.json`
**Step 1: Write devcontainer.json**
```json
{
  "name": "foxhunt",
  "build": {
    "dockerfile": "Dockerfile",
    "context": "..",
    "args": {
      "CI_BUILDER_IMAGE": "rg.fr-par.scw.cloud/foxhunt-ci/ci-builder:latest"
    }
  },
  "remoteUser": "dev",
  "containerEnv": {
    "SQLX_OFFLINE": "true",
    "CUDA_COMPUTE_CAP": "90",
    "EDITOR": "vim"
  },
  "postCreateCommand": "/usr/local/bin/init-home.sh",
  "postStartCommand": "claude --version && cargo --version && kubectl cluster-info 2>/dev/null || true"
}
```
**Step 2: Commit**
```bash
git add .devcontainer/devcontainer.json
git commit -m "feat(devcontainer): add devcontainer.json
Builds from CI builder base, runs as non-root dev user.
postCreateCommand bootstraps PVC home dir on first session."
```
---
### Task 7: Add build-devcontainer CI job
**Files:**
- Modify: `.gitlab-ci.yml` (insert after `build-ci-builder` job, around line 60)
**Step 1: Add the CI job**
Insert after the `build-ci-builder` job block (after line 59) and before the `.rust-base` template:
```yaml
# --------------------------------------------------------------------------
# Stage 0b: Build devcontainer image → push to internal GitLab registry
# --------------------------------------------------------------------------
build-devcontainer:
  stage: prepare
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  tags:
    - kapsule
    - docker
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"
      changes:
        - .devcontainer/**
      when: on_success
    - when: manual
      allow_failure: true
  before_script:
    - mkdir -p /kaniko/.docker
    - |
      echo "{\"auths\":{\"${INTERNAL_REGISTRY}\":{\"username\":\"${CI_REGISTRY_USER}\",\"password\":\"${CI_REGISTRY_PASSWORD}\"},\"rg.fr-par.scw.cloud\":{\"username\":\"nologin\",\"password\":\"${SCW_SECRET_KEY}\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/.devcontainer/Dockerfile"
      --insecure-registry "${INTERNAL_REGISTRY}"
      --destination "${REGISTRY}/devcontainer:${CI_COMMIT_SHA}"
      --destination "${REGISTRY}/devcontainer:latest"
```
**Step 2: Verify YAML syntax**
Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))"`
Expected: No errors
**Step 3: Commit**
```bash
git add .gitlab-ci.yml
git commit -m "ci: add build-devcontainer job
Builds .devcontainer/Dockerfile with Kaniko, pushes to internal
GitLab registry. Triggers on .devcontainer/ changes or manual run."
```
---
### Task 8: Create DevPod setup script
**Files:**
- Create: `scripts/devpod-setup.sh`
**Step 1: Write the setup script**
One-time setup to run on the developer's laptop. Configures DevPod with the
kubernetes provider pointing at the foxhunt cluster.
```bash
#!/bin/bash
set -euo pipefail
# DevPod one-time setup for Foxhunt development
# Prerequisites: devpod CLI installed, kubectl configured for foxhunt cluster
#
# Install DevPod:
#   curl -fsSL https://devpod.sh/install.sh | bash
#
# Usage:
#   ./scripts/devpod-setup.sh
#   devpod up .                  # from the foxhunt repo root
#   devpod up git@100.90.76.85:2222/root/foxhunt.git  # from anywhere
echo "=== Foxhunt DevPod Setup ==="
# Check prerequisites
command -v devpod >/dev/null 2>&1 || { echo "Error: devpod not installed. Run: curl -fsSL https://devpod.sh/install.sh | bash"; exit 1; }
command -v kubectl >/dev/null 2>&1 || { echo "Error: kubectl not found"; exit 1; }
# Verify cluster access
if ! kubectl -n foxhunt get ns foxhunt >/dev/null 2>&1; then
    echo "Error: Cannot reach foxhunt namespace. Check kubectl config."
    exit 1
fi
# Ensure PVC exists
if ! kubectl -n foxhunt get pvc dev-home-pvc >/dev/null 2>&1; then
    echo "Creating dev-home-pvc (50Gi block storage)..."
    kubectl apply -f infra/k8s/storage/dev-home-pvc.yaml
fi
# Add kubernetes provider (idempotent)
devpod provider add kubernetes 2>/dev/null || true
# Configure provider
devpod provider set kubernetes \
    KUBERNETES_NAMESPACE=foxhunt \
    NODE_SELECTOR="k8s.scaleway.com/pool-name=gpu-dev" \
    DISK_SIZE=10Gi \
    STORAGE_CLASS=scw-bssd \
    INACTIVITY_TIMEOUT=30m \
    POD_MANIFEST_TEMPLATE=.devcontainer/pod-template.yaml
# Configure image pull secrets if not already present
if ! kubectl -n foxhunt get secret gitlab-registry >/dev/null 2>&1; then
    echo ""
    echo "NOTE: You need to create the gitlab-registry pull secret:"
    echo "  kubectl -n foxhunt create secret docker-registry gitlab-registry \\"
    echo "    --docker-server=gitlab-registry.foxhunt.svc.cluster.local:5000 \\"
    echo "    --docker-username=<user> --docker-password=<token>"
    echo ""
fi
echo ""
echo "=== Setup complete ==="
echo ""
echo "Usage:"
echo "  devpod up .                                         # from repo root"
echo "  devpod up git@100.90.76.85:2222/root/foxhunt.git   # from anywhere"
echo ""
echo "Open in DevPod link (bookmark this):"
echo "  https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git"
echo ""
echo "Session lifecycle:"
echo "  devpod up foxhunt     # start/reconnect"
echo "  devpod stop foxhunt   # stop (PVC preserved, node scales down)"
echo "  devpod delete foxhunt # delete workspace (PVC preserved)"
echo ""
```
**Step 2: Make executable**
Run: `chmod +x scripts/devpod-setup.sh`
**Step 3: Commit**
```bash
git add scripts/devpod-setup.sh
git commit -m "feat: add devpod-setup.sh for one-time provider config
Configures DevPod kubernetes provider, creates dev-home PVC,
verifies cluster access. Run once on developer laptop."
```
---
### Task 9: Apply infrastructure (manual — requires cluster access)
This task runs `terragrunt apply` and `kubectl apply`. It is manual because
it modifies live infrastructure.
**Step 1: Apply Terraform changes**
Run from a machine with Scaleway credentials:
```bash
cd infra/live/production/kapsule
terragrunt plan    # review the new gpu-dev pool
terragrunt apply   # create the pool (starts at 0 nodes)
```
Expected: New `gpu-dev` pool created with 0 nodes.
**Step 2: Apply PVC**
```bash
kubectl apply -f infra/k8s/storage/dev-home-pvc.yaml
```
Expected: PVC `dev-home-pvc` created in `foxhunt` namespace, status `Pending`
(binds on first pod mount).
**Step 3: Build devcontainer image (first time)**
Trigger the CI job manually from GitLab UI, or push a commit touching
`.devcontainer/`.
**Step 4: Run DevPod setup**
```bash
./scripts/devpod-setup.sh
```
**Step 5: Test the full cycle**
```bash
devpod up .
# Should: provision GP1-L node (~2-3 min), start pod, SSH in
# Verify inside the pod:
claude --version
cargo --version
kubectl get nodes
```
**Step 6: Test idle scale-down**
Disconnect from the pod. Wait 30 min for DevPod inactivity timeout.
Then wait 10 min for Kapsule autoscaler to remove the node.
Verify: `kubectl get nodes` shows no gpu-dev node.
**Step 7: Test reconnect**
```bash
devpod up foxhunt
# Should: provision new node, mount same PVCs, all state preserved
```
---
### Task 10: Add GitLab project badge
This is done via the GitLab UI (Settings → General → Badges).
**Step 1: Add badge**
```
Badge name: Open in DevPod
Badge link: https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git
Badge image: https://devpod.sh/assets/open-in-devpod.svg
```
**Step 2: Verify**
Navigate to the project page. The "Open in DevPod" badge should appear.
Click it — DevPod should open (if installed) and begin provisioning.