foxhunt/docs/plans/2026-02-25-devcontainer-design.md

DevPod Remote Development Environment — Design

Date: 2026-02-25 Status: Approved

Goal

A one-click remote development environment on Kapsule with full Claude Code toolchain, persistent storage, and autoscale-to-zero cost control. Accessible from anywhere via an "Open in DevPod" link on the GitLab project page.

Decisions

Decision	Choice	Rationale
Tool	DevPod (Loft Labs)	Zero server infra, native devcontainer.json, built-in idle timeout, provider-agnostic
Dev node	GP1-L (16 vCPU, 64GB)	Fast cargo builds on 37-crate workspace, autoscales to zero
GPU for dev	None (CPU-only)	cargo check/clippy/test are CPU-bound; GPU training uses H100 via CI/jobs
GPU for training	H100 (existing pool)	Stays autoscaled to zero, wakes for CI and ad-hoc training jobs
Connection	Terminal + Claude Code	No VS Code/JetBrains server overhead
Persistence	Block storage PVCs	Survive pod restarts, node scale-down, Claude Code auto-updates
GitLab integration	Project badge + CI image build	"Open in DevPod" button, image rebuilt on .devcontainer/ changes
Eliminated	GitLab Workspaces (no GPU, Devfile only), Coder (overkill), raw SSH (manual lifecycle)

Infrastructure

New Kapsule Pool: gpu-dev

resource "scaleway_k8s_pool" "gpu_dev" {
  name        = "gpu-dev"
  node_type   = "GP1-L"     # 16 vCPU, 64GB RAM
  min_size    = 0
  max_size    = 1
  autoscaling = true
  autohealing = true

  lifecycle {
    ignore_changes = [size]
  }
}

No GPU taint, no special tolerations. The node selector k8s.scaleway.com/pool-name=gpu-dev directs the dev pod here.

Persistent Volumes

PVC	Size	StorageClass	Mount	Purpose
dev-home-pvc	50Gi	scw-bssd	/home/dev	Claude Code + ~/.claude/ + cargo registry + sccache cache + shell/git config
training-data-pvc	10Gi (existing)	scw-bssd	/data/training	Databento .dbn.zst futures data (read-only)

DevPod also creates a workspace volume for /workspace (the repo checkout).

The 50Gi home volume is intentionally large: cargo registry + sccache for 37 crates, Claude Code plugins/MCP caches, and future datasets or model checkpoints. The PVC can be resized, mounted in other pods, or snapshotted as needed.

Devcontainer Image

Built by CI, pushed to git.fxhnt.ai:5050/root/foxhunt/devcontainer:latest.

Base

Extends Dockerfile.ci-builder (CUDA 12.4, Rust 1.89, protoc, sccache, clippy).

Additional Layers

Dev tooling:

• Node.js 22 LTS (Claude Code runtime)
• Claude Code CLI (npm install -g @anthropic-ai/claude-code)
• Python 3.12 + pip (scripts, data tooling)
• kubectl, helm (cluster management)
• scw CLI (Scaleway management)
• glab (GitLab CLI)
• terraform, terragrunt (infra management)
• jq, yq, ripgrep, fd-find (CLI essentials)
• zsh + oh-my-zsh (shell)
• git-lfs

Environment:

• Non-root user dev with passwordless sudo
• Cargo env in /home/dev/.cargo
• sccache configured (local disk cache on PVC)
• SQLX_OFFLINE=true
• CUDA_COMPUTE_CAP=90

.devcontainer/ Structure

.devcontainer/
├── devcontainer.json      # DevPod + devcontainer spec
├── Dockerfile             # Extends ci-builder with dev tooling
└── pod-template.yaml      # K8s pod overrides (PVC mounts, node selector)

devcontainer.json

{
  "name": "foxhunt",
  "build": {
    "dockerfile": "Dockerfile",
    "context": ".."
  },
  "remoteUser": "dev",
  "containerEnv": {
    "SQLX_OFFLINE": "true",
    "CUDA_COMPUTE_CAP": "90"
  },
  "postStartCommand": "claude --version && cargo --version && kubectl cluster-info 2>/dev/null || true",
  "customizations": {
    "devpod": {
      "provider": "kubernetes"
    }
  }
}

PVC mounts and node selector are handled by pod-template.yaml (DevPod's POD_MANIFEST_TEMPLATE option), since devcontainer.json cannot express k8s- specific volume claims.

DevPod Provider Configuration

One-time setup on the developer's machine (scripts/devpod-setup.sh):

devpod provider add kubernetes
devpod provider set kubernetes \
  KUBERNETES_NAMESPACE=foxhunt \
  NODE_SELECTOR="k8s.scaleway.com/pool-name=gpu-dev" \
  DISK_SIZE=10Gi \
  STORAGE_CLASS=scw-bssd \
  INACTIVITY_TIMEOUT=30m \
  IMAGE_PULL_SECRETS=gitlab-registry \
  POD_MANIFEST_TEMPLATE=.devcontainer/pod-template.yaml

GitLab Integration

"Open in DevPod" Badge

Added as a GitLab project badge (Settings → General → Badges):

Name:  Open in DevPod
Link:  https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git
Image: https://devpod.sh/assets/open-in-devpod.svg

Also usable as:

• CLI: devpod up git@100.90.76.85:2222/root/foxhunt.git
• Bookmark in any browser
• Shared link in docs/chat

CI Image Build

New job in .gitlab-ci.yml:

build-devcontainer:
  stage: build
  rules:
    - changes: [".devcontainer/**"]
  script:
    # Kaniko build + push to GitLab registry

Only triggers when .devcontainer/ files change. Uses existing Kaniko pattern from the service image builds.

Session Lifecycle

Click "Open in DevPod" (or `devpod up`)
  → DevPod creates pod targeting gpu-dev pool
  → Kapsule autoscaler provisions GP1-L node (~2-3 min)
  → Pod starts, mounts PVCs (home + training data)
  → SSH tunnel established
  → Terminal opens

Active development
  → cargo check, claude, kubectl, glab, terraform
  → All writes go to persistent /home/dev PVC

Idle 30 minutes (no SSH activity)
  → DevPod stops pod
  → Node idle 10 min
  → Kapsule autoscaler removes GP1-L node
  → Cost: €0

Return (click link or `devpod up` again)
  → Node provisions (~2-3 min)
  → Pod starts, mounts SAME PVCs
  → Claude Code config, cargo cache, shell history intact
  → Incremental cargo builds (sccache hits)

GPU Access (On-Demand)

The dev pod is CPU-only. GPU access for training/inference:

• CI training jobs → H100 pool (existing, autoscaled to zero)
• Ad-hoc training → kubectl apply -f infra/k8s/training/job-template.yaml from inside the dev pod
• Future: second DevPod provider profile targeting L4 or H100 pool with nvidia.com/gpu=1 for interactive GPU sessions

Cost Estimate

Scenario	Node	Rate	Daily (8hr)	Monthly
Active dev	GP1-L	~€0.24/hr	~€2	~€44
Idle	—	€0	€0	€0
CI build (H100)	H100-1-80G	~€3-4/hr	per-job	per-job

No conflict between dev and CI — separate node pools.

Files to Create/Modify

File	Action
.devcontainer/devcontainer.json	Create
.devcontainer/Dockerfile	Create
.devcontainer/pod-template.yaml	Create
infra/modules/kapsule/main.tf	Add gpu-dev pool resource
infra/modules/kapsule/variables.tf	Add dev pool variables
infra/live/production/kapsule/terragrunt.hcl	Enable dev pool, set type
.gitlab-ci.yml	Add build-devcontainer job
scripts/devpod-setup.sh	One-time provider config script