foxhunt/docs/plans/2026-02-25-devcontainer-plan.md

DevPod Remote Development Environment — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: One-click remote dev environment on Kapsule with Claude Code, persistent storage, and autoscale-to-zero.

Architecture: DevPod (client-side) provisions a pod on a dedicated GP1-L CPU pool via the kubernetes provider. A devcontainer image (extending the CI builder) is built by CI and pushed to the GitLab registry. PVCs persist home directory and cargo cache across sessions. "Open in DevPod" badge on GitLab project page.

Tech Stack: DevPod, devcontainer spec, Terraform/Terragrunt (Kapsule), Kaniko (CI image build), Scaleway block storage (PVCs)

Design doc: docs/plans/2026-02-25-devcontainer-design.md

---

Task 1: Add gpu-dev pool to Terraform

Files:

• Modify: infra/modules/kapsule/variables.tf (append after line 71)
• Modify: infra/modules/kapsule/main.tf (append after line 90)
• Modify: infra/live/production/kapsule/terragrunt.hcl (add inputs)

Step 1: Add variables for the dev pool

Append to infra/modules/kapsule/variables.tf after the gitlab_type variable:

variable "enable_dev_pool" {
  description = "Create CPU node pool for remote development (DevPod)"
  type        = bool
  default     = false
}

variable "dev_pool_type" {
  description = "Instance type for the dev node pool"
  type        = string
  default     = "GP1-L"
}

variable "dev_pool_max_size" {
  description = "Maximum number of nodes in the dev pool"
  type        = number
  default     = 1
}

Step 2: Add the pool resource

Append to infra/modules/kapsule/main.tf after the gitlab pool resource (after line 90):

# CPU pool for remote development (DevPod — autoscales to zero)
resource "scaleway_k8s_pool" "dev" {
  count       = var.enable_dev_pool ? 1 : 0
  cluster_id  = scaleway_k8s_cluster.foxhunt.id
  name        = "gpu-dev"
  node_type   = var.dev_pool_type
  size        = 1
  min_size    = 0
  max_size    = var.dev_pool_max_size
  autoscaling = true
  autohealing = true
  region      = var.region

  lifecycle {
    ignore_changes = [size]
  }
}

Step 3: Enable the dev pool in production inputs

Add to infra/live/production/kapsule/terragrunt.hcl inputs block:

  # Dev pool (GP1-L for DevPod remote development, autoscales to zero)
  enable_dev_pool  = true
  dev_pool_type    = "GP1-L"
  dev_pool_max_size = 1

Step 4: Validate terraform syntax

Run: cd infra/modules/kapsule && terraform validate Expected: Success (or "no backend configured" which is fine for syntax check)

Step 5: Commit

git add infra/modules/kapsule/main.tf infra/modules/kapsule/variables.tf infra/live/production/kapsule/terragrunt.hcl
git commit -m "infra: add gpu-dev pool for DevPod remote development

GP1-L (16 vCPU, 64GB), autoscaling 0→1, dedicated for dev pods.
Separate from CI (H100) and inference (L4) pools."

---

Task 2: Create dev-home PVC manifest

Files:

• Create: infra/k8s/storage/dev-home-pvc.yaml

Step 1: Create the PVC manifest

Follow the pattern from infra/k8s/storage/training-data-pv.yaml but use dynamic provisioning (no pre-existing volume — Scaleway CSI creates it on first claim):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dev-home-pvc
  namespace: foxhunt
  labels:
    app.kubernetes.io/name: dev-home
    app.kubernetes.io/part-of: foxhunt
    app.kubernetes.io/component: devpod
spec:
  storageClassName: scw-bssd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Step 2: Commit

git add infra/k8s/storage/dev-home-pvc.yaml
git commit -m "infra: add 50Gi PVC for DevPod home directory

Persistent block storage for Claude Code, cargo registry, sccache
cache, and shell config. Survives pod restarts and node scale-down."

---

Task 3: Create the devcontainer Dockerfile

Files:

• Create: .devcontainer/Dockerfile

Step 1: Write the Dockerfile

Extends the CI builder image. Adds dev tooling (Node.js, Claude Code, Python, kubectl, helm, scw, glab, terraform, terragrunt, shell tools). Creates non-root dev user with sudo.

# Foxhunt devcontainer — extends CI builder with interactive dev tooling
# Built by CI, pushed to git.fxhnt.ai:5050/root/foxhunt/devcontainer:latest
# See: docs/plans/2026-02-25-devcontainer-design.md

ARG CI_BUILDER_IMAGE=rg.fr-par.scw.cloud/foxhunt-ci/ci-builder:latest
FROM ${CI_BUILDER_IMAGE}

ENV DEBIAN_FRONTEND=noninteractive

# Dev tooling: Node.js 22 (Claude Code), Python 3, shell tools
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 \
    python3-pip \
    python3-venv \
    zsh \
    sudo \
    openssh-server \
    git-lfs \
    jq \
    ripgrep \
    fd-find \
    htop \
    less \
    vim \
    && rm -rf /var/lib/apt/lists/*

# Node.js 22 LTS (Claude Code runtime)
RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
    && apt-get install -y nodejs \
    && rm -rf /var/lib/apt/lists/*

# Claude Code CLI
RUN npm install -g @anthropic-ai/claude-code

# kubectl
RUN curl -fsSL "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" \
    -o /usr/local/bin/kubectl \
    && chmod +x /usr/local/bin/kubectl

# helm
RUN curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# scw CLI (Scaleway)
RUN curl -fsSL https://github.com/scaleway/scaleway-cli/releases/latest/download/scaleway-cli_linux_amd64 \
    -o /usr/local/bin/scw \
    && chmod +x /usr/local/bin/scw

# glab (GitLab CLI)
RUN curl -fsSL "https://gitlab.com/gitlab-org/cli/-/releases/permalink/latest/downloads/glab_linux_amd64.deb" \
    -o /tmp/glab.deb \
    && dpkg -i /tmp/glab.deb \
    && rm /tmp/glab.deb

# terraform
RUN curl -fsSL https://releases.hashicorp.com/terraform/1.11.2/terraform_1.11.2_linux_amd64.zip \
    -o /tmp/terraform.zip \
    && unzip -o /tmp/terraform.zip -d /usr/local/bin \
    && rm /tmp/terraform.zip

# terragrunt
RUN curl -fsSL https://github.com/gruntwork-io/terragrunt/releases/download/v0.72.6/terragrunt_linux_amd64 \
    -o /usr/local/bin/terragrunt \
    && chmod +x /usr/local/bin/terragrunt

# yq (YAML processor)
RUN curl -fsSL https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 \
    -o /usr/local/bin/yq \
    && chmod +x /usr/local/bin/yq

# Non-root dev user with sudo
RUN groupadd -g 1000 dev \
    && useradd -m -u 1000 -g dev -s /bin/zsh dev \
    && echo "dev ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/dev

# Install oh-my-zsh for dev user
RUN su - dev -c 'sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" "" --unattended'

# Move Rust toolchain to shared location so dev user can access it
# (CI builder installs to /root/.cargo and /root/.rustup)
RUN cp -a /root/.cargo /opt/cargo \
    && cp -a /root/.rustup /opt/rustup \
    && chown -R dev:dev /opt/cargo /opt/rustup

# Environment for dev user
ENV CARGO_HOME=/home/dev/.cargo
ENV RUSTUP_HOME=/opt/rustup
ENV PATH="/home/dev/.cargo/bin:/opt/rustup/bin:/opt/cargo/bin:${PATH}"
ENV SQLX_OFFLINE=true
ENV CUDA_COMPUTE_CAP=90

# sccache local disk cache (PVC-backed home dir)
ENV SCCACHE_DIR=/home/dev/.cache/sccache
ENV RUSTC_WRAPPER=/usr/local/bin/sccache

# First-run init script: copies cargo bins to home if not already present
COPY .devcontainer/init-home.sh /usr/local/bin/init-home.sh
RUN chmod +x /usr/local/bin/init-home.sh

USER dev
WORKDIR /workspace

# Verify key tools
RUN rustc --version && cargo --version && node --version && claude --version \
    && kubectl version --client 2>/dev/null && helm version --short \
    && terraform --version && terragrunt --version && scw version && glab --version

Step 2: Commit

git add .devcontainer/Dockerfile
git commit -m "feat(devcontainer): add Dockerfile extending CI builder

Adds Node.js 22, Claude Code, Python 3, kubectl, helm, scw, glab,
terraform, terragrunt, zsh, ripgrep, fd-find. Non-root dev user."

---

Task 4: Create the home directory init script

Files:

• Create: .devcontainer/init-home.sh

Step 1: Write the init script

On first boot with a fresh PVC, the home directory is empty. This script bootstraps cargo toolchain links and shell config. On subsequent boots it's a no-op.

#!/bin/bash
set -euo pipefail

# Bootstrap cargo into PVC-backed home if not already present
if [ ! -d "$HOME/.cargo/bin" ]; then
    echo "[init-home] First run: linking cargo toolchain..."
    mkdir -p "$HOME/.cargo/bin"
    # Copy cargo config and link binaries from shared rustup
    for bin in /opt/cargo/bin/*; do
        ln -sf "$bin" "$HOME/.cargo/bin/$(basename "$bin")"
    done
    echo "[init-home] Cargo toolchain linked."
fi

# Ensure sccache cache dir exists
mkdir -p "${SCCACHE_DIR:-$HOME/.cache/sccache}"

# Default .zshrc if none exists (oh-my-zsh is in the image, config in PVC)
if [ ! -f "$HOME/.zshrc" ]; then
    cp /home/dev/.oh-my-zsh/templates/zshrc.zsh-template "$HOME/.zshrc" 2>/dev/null || true
    echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> "$HOME/.zshrc"
    echo 'export SQLX_OFFLINE=true' >> "$HOME/.zshrc"
fi

echo "[init-home] Ready."

Step 2: Commit

git add .devcontainer/init-home.sh
git commit -m "feat(devcontainer): add init-home.sh for PVC bootstrap

Links cargo binaries, creates sccache dir, and writes default .zshrc
on first boot. No-op on subsequent sessions."

---

Task 5: Create the pod template for DevPod

Files:

• Create: .devcontainer/pod-template.yaml

Step 1: Write the pod manifest template

DevPod's POD_MANIFEST_TEMPLATE lets us add PVC mounts and node selection that devcontainer.json cannot express. DevPod merges this with its own pod spec.

# DevPod pod template — merged with DevPod's generated pod spec
# Ref: https://github.com/loft-sh/devpod-provider-kubernetes
apiVersion: v1
kind: Pod
metadata:
  labels:
    app.kubernetes.io/name: devpod
    app.kubernetes.io/part-of: foxhunt
    app.kubernetes.io/component: dev-environment
spec:
  nodeSelector:
    k8s.scaleway.com/pool-name: gpu-dev
  containers:
    - name: devpod
      resources:
        requests:
          cpu: "4000m"
          memory: "16Gi"
        limits:
          cpu: "14000m"
          memory: "56Gi"
      volumeMounts:
        - name: dev-home
          mountPath: /home/dev
        - name: training-data
          mountPath: /data/training
          readOnly: true
  initContainers:
    - name: init-home
      image: busybox:latest
      command: ["sh", "-c", "chown -R 1000:1000 /home/dev"]
      volumeMounts:
        - name: dev-home
          mountPath: /home/dev
  volumes:
    - name: dev-home
      persistentVolumeClaim:
        claimName: dev-home-pvc
    - name: training-data
      persistentVolumeClaim:
        claimName: training-data-pvc

Step 2: Commit

git add .devcontainer/pod-template.yaml
git commit -m "feat(devcontainer): add k8s pod template for DevPod

Mounts dev-home PVC (50Gi) and training-data PVC (read-only).
Targets gpu-dev node pool. Resource requests: 4 CPU / 16Gi,
limits: 14 CPU / 56Gi."

---

Task 6: Create devcontainer.json

Files:

• Create: .devcontainer/devcontainer.json

Step 1: Write devcontainer.json

{
  "name": "foxhunt",
  "build": {
    "dockerfile": "Dockerfile",
    "context": "..",
    "args": {
      "CI_BUILDER_IMAGE": "rg.fr-par.scw.cloud/foxhunt-ci/ci-builder:latest"
    }
  },
  "remoteUser": "dev",
  "containerEnv": {
    "SQLX_OFFLINE": "true",
    "CUDA_COMPUTE_CAP": "90",
    "EDITOR": "vim"
  },
  "postCreateCommand": "/usr/local/bin/init-home.sh",
  "postStartCommand": "claude --version && cargo --version && kubectl cluster-info 2>/dev/null || true"
}

Step 2: Commit

git add .devcontainer/devcontainer.json
git commit -m "feat(devcontainer): add devcontainer.json

Builds from CI builder base, runs as non-root dev user.
postCreateCommand bootstraps PVC home dir on first session."

---

Task 7: Add build-devcontainer CI job

Files:

• Modify: .gitlab-ci.yml (insert after build-ci-builder job, around line 60)

Step 1: Add the CI job

Insert after the build-ci-builder job block (after line 59) and before the .rust-base template:

# --------------------------------------------------------------------------
# Stage 0b: Build devcontainer image → push to internal GitLab registry
# --------------------------------------------------------------------------
build-devcontainer:
  stage: prepare
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  tags:
    - kapsule
    - docker
  rules:
    - if: $CI_PIPELINE_SOURCE == "push"
      changes:
        - .devcontainer/**
      when: on_success
    - when: manual
      allow_failure: true
  before_script:
    - mkdir -p /kaniko/.docker
    - |
      echo "{\"auths\":{\"${INTERNAL_REGISTRY}\":{\"username\":\"${CI_REGISTRY_USER}\",\"password\":\"${CI_REGISTRY_PASSWORD}\"},\"rg.fr-par.scw.cloud\":{\"username\":\"nologin\",\"password\":\"${SCW_SECRET_KEY}\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/.devcontainer/Dockerfile"
      --insecure-registry "${INTERNAL_REGISTRY}"
      --destination "${REGISTRY}/devcontainer:${CI_COMMIT_SHA}"
      --destination "${REGISTRY}/devcontainer:latest"

Step 2: Verify YAML syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))" Expected: No errors

Step 3: Commit

git add .gitlab-ci.yml
git commit -m "ci: add build-devcontainer job

Builds .devcontainer/Dockerfile with Kaniko, pushes to internal
GitLab registry. Triggers on .devcontainer/ changes or manual run."

---

Task 8: Create DevPod setup script

Files:

• Create: scripts/devpod-setup.sh

Step 1: Write the setup script

One-time setup to run on the developer's laptop. Configures DevPod with the kubernetes provider pointing at the foxhunt cluster.

#!/bin/bash
set -euo pipefail

# DevPod one-time setup for Foxhunt development
# Prerequisites: devpod CLI installed, kubectl configured for foxhunt cluster
#
# Install DevPod:
#   curl -fsSL https://devpod.sh/install.sh | bash
#
# Usage:
#   ./scripts/devpod-setup.sh
#   devpod up .                  # from the foxhunt repo root
#   devpod up git@100.90.76.85:2222/root/foxhunt.git  # from anywhere

echo "=== Foxhunt DevPod Setup ==="

# Check prerequisites
command -v devpod >/dev/null 2>&1 || { echo "Error: devpod not installed. Run: curl -fsSL https://devpod.sh/install.sh | bash"; exit 1; }
command -v kubectl >/dev/null 2>&1 || { echo "Error: kubectl not found"; exit 1; }

# Verify cluster access
if ! kubectl -n foxhunt get ns foxhunt >/dev/null 2>&1; then
    echo "Error: Cannot reach foxhunt namespace. Check kubectl config."
    exit 1
fi

# Ensure PVC exists
if ! kubectl -n foxhunt get pvc dev-home-pvc >/dev/null 2>&1; then
    echo "Creating dev-home-pvc (50Gi block storage)..."
    kubectl apply -f infra/k8s/storage/dev-home-pvc.yaml
fi

# Add kubernetes provider (idempotent)
devpod provider add kubernetes 2>/dev/null || true

# Configure provider
devpod provider set kubernetes \
    KUBERNETES_NAMESPACE=foxhunt \
    NODE_SELECTOR="k8s.scaleway.com/pool-name=gpu-dev" \
    DISK_SIZE=10Gi \
    STORAGE_CLASS=scw-bssd \
    INACTIVITY_TIMEOUT=30m \
    POD_MANIFEST_TEMPLATE=.devcontainer/pod-template.yaml

# Configure image pull secrets if not already present
if ! kubectl -n foxhunt get secret gitlab-registry >/dev/null 2>&1; then
    echo ""
    echo "NOTE: You need to create the gitlab-registry pull secret:"
    echo "  kubectl -n foxhunt create secret docker-registry gitlab-registry \\"
    echo "    --docker-server=gitlab-registry.foxhunt.svc.cluster.local:5000 \\"
    echo "    --docker-username=<user> --docker-password=<token>"
    echo ""
fi

echo ""
echo "=== Setup complete ==="
echo ""
echo "Usage:"
echo "  devpod up .                                         # from repo root"
echo "  devpod up git@100.90.76.85:2222/root/foxhunt.git   # from anywhere"
echo ""
echo "Open in DevPod link (bookmark this):"
echo "  https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git"
echo ""
echo "Session lifecycle:"
echo "  devpod up foxhunt     # start/reconnect"
echo "  devpod stop foxhunt   # stop (PVC preserved, node scales down)"
echo "  devpod delete foxhunt # delete workspace (PVC preserved)"
echo ""

Step 2: Make executable

Run: chmod +x scripts/devpod-setup.sh

Step 3: Commit

git add scripts/devpod-setup.sh
git commit -m "feat: add devpod-setup.sh for one-time provider config

Configures DevPod kubernetes provider, creates dev-home PVC,
verifies cluster access. Run once on developer laptop."

---

Task 9: Apply infrastructure (manual — requires cluster access)

This task runs terragrunt apply and kubectl apply. It is manual because it modifies live infrastructure.

Step 1: Apply Terraform changes

Run from a machine with Scaleway credentials:

cd infra/live/production/kapsule
terragrunt plan    # review the new gpu-dev pool
terragrunt apply   # create the pool (starts at 0 nodes)

Expected: New gpu-dev pool created with 0 nodes.

Step 2: Apply PVC

kubectl apply -f infra/k8s/storage/dev-home-pvc.yaml

Expected: PVC dev-home-pvc created in foxhunt namespace, status Pending (binds on first pod mount).

Step 3: Build devcontainer image (first time)

Trigger the CI job manually from GitLab UI, or push a commit touching .devcontainer/.

Step 4: Run DevPod setup

./scripts/devpod-setup.sh

Step 5: Test the full cycle

devpod up .
# Should: provision GP1-L node (~2-3 min), start pod, SSH in
# Verify inside the pod:
claude --version
cargo --version
kubectl get nodes

Step 6: Test idle scale-down

Disconnect from the pod. Wait 30 min for DevPod inactivity timeout. Then wait 10 min for Kapsule autoscaler to remove the node. Verify: kubectl get nodes shows no gpu-dev node.

Step 7: Test reconnect

devpod up foxhunt
# Should: provision new node, mount same PVCs, all state preserved

---

Task 10: Add GitLab project badge

This is done via the GitLab UI (Settings → General → Badges).

Step 1: Add badge

Badge name: Open in DevPod
Badge link: https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git
Badge image: https://devpod.sh/assets/open-in-devpod.svg

Step 2: Verify

Navigate to the project page. The "Open in DevPod" badge should appear. Click it — DevPod should open (if installed) and begin provisioning.