diff --git a/docs/plans/2026-02-25-devcontainer-design.md b/docs/plans/2026-02-25-devcontainer-design.md new file mode 100644 index 000000000..2c2a5e076 --- /dev/null +++ b/docs/plans/2026-02-25-devcontainer-design.md @@ -0,0 +1,232 @@ +# DevPod Remote Development Environment — Design + +**Date**: 2026-02-25 +**Status**: Approved + +## Goal + +A one-click remote development environment on Kapsule with full Claude Code +toolchain, persistent storage, and autoscale-to-zero cost control. Accessible +from anywhere via an "Open in DevPod" link on the GitLab project page. + +## Decisions + +| Decision | Choice | Rationale | +|----------|--------|-----------| +| Tool | DevPod (Loft Labs) | Zero server infra, native devcontainer.json, built-in idle timeout, provider-agnostic | +| Dev node | GP1-L (16 vCPU, 64GB) | Fast cargo builds on 37-crate workspace, autoscales to zero | +| GPU for dev | None (CPU-only) | cargo check/clippy/test are CPU-bound; GPU training uses H100 via CI/jobs | +| GPU for training | H100 (existing pool) | Stays autoscaled to zero, wakes for CI and ad-hoc training jobs | +| Connection | Terminal + Claude Code | No VS Code/JetBrains server overhead | +| Persistence | Block storage PVCs | Survive pod restarts, node scale-down, Claude Code auto-updates | +| GitLab integration | Project badge + CI image build | "Open in DevPod" button, image rebuilt on .devcontainer/ changes | +| Eliminated | GitLab Workspaces (no GPU, Devfile only), Coder (overkill), raw SSH (manual lifecycle) | + +## Infrastructure + +### New Kapsule Pool: `gpu-dev` + +```hcl +resource "scaleway_k8s_pool" "gpu_dev" { + name = "gpu-dev" + node_type = "GP1-L" # 16 vCPU, 64GB RAM + min_size = 0 + max_size = 1 + autoscaling = true + autohealing = true + + lifecycle { + ignore_changes = [size] + } +} +``` + +No GPU taint, no special tolerations. The node selector +`k8s.scaleway.com/pool-name=gpu-dev` directs the dev pod here. + +### Persistent Volumes + +| PVC | Size | StorageClass | Mount | Purpose | +|-----|------|-------------|-------|---------| +| `dev-home-pvc` | 50Gi | scw-bssd | `/home/dev` | Claude Code + ~/.claude/ + cargo registry + sccache cache + shell/git config | +| `training-data-pvc` | 10Gi (existing) | scw-bssd | `/data/training` | Databento .dbn.zst futures data (read-only) | + +DevPod also creates a workspace volume for `/workspace` (the repo checkout). + +The 50Gi home volume is intentionally large: cargo registry + sccache for 37 +crates, Claude Code plugins/MCP caches, and future datasets or model checkpoints. +The PVC can be resized, mounted in other pods, or snapshotted as needed. + +## Devcontainer Image + +Built by CI, pushed to `git.fxhnt.ai:5050/root/foxhunt/devcontainer:latest`. + +### Base + +Extends `Dockerfile.ci-builder` (CUDA 12.4, Rust 1.89, protoc, sccache, clippy). + +### Additional Layers + +**Dev tooling**: +- Node.js 22 LTS (Claude Code runtime) +- Claude Code CLI (`npm install -g @anthropic-ai/claude-code`) +- Python 3.12 + pip (scripts, data tooling) +- kubectl, helm (cluster management) +- scw CLI (Scaleway management) +- glab (GitLab CLI) +- terraform, terragrunt (infra management) +- jq, yq, ripgrep, fd-find (CLI essentials) +- zsh + oh-my-zsh (shell) +- git-lfs + +**Environment**: +- Non-root user `dev` with passwordless sudo +- Cargo env in `/home/dev/.cargo` +- sccache configured (local disk cache on PVC) +- `SQLX_OFFLINE=true` +- `CUDA_COMPUTE_CAP=90` + +## `.devcontainer/` Structure + +``` +.devcontainer/ +├── devcontainer.json # DevPod + devcontainer spec +├── Dockerfile # Extends ci-builder with dev tooling +└── pod-template.yaml # K8s pod overrides (PVC mounts, node selector) +``` + +### devcontainer.json + +```jsonc +{ + "name": "foxhunt", + "build": { + "dockerfile": "Dockerfile", + "context": ".." + }, + "remoteUser": "dev", + "containerEnv": { + "SQLX_OFFLINE": "true", + "CUDA_COMPUTE_CAP": "90" + }, + "postStartCommand": "claude --version && cargo --version && kubectl cluster-info 2>/dev/null || true", + "customizations": { + "devpod": { + "provider": "kubernetes" + } + } +} +``` + +PVC mounts and node selector are handled by `pod-template.yaml` (DevPod's +`POD_MANIFEST_TEMPLATE` option), since devcontainer.json cannot express k8s- +specific volume claims. + +## DevPod Provider Configuration + +One-time setup on the developer's machine (`scripts/devpod-setup.sh`): + +```bash +devpod provider add kubernetes +devpod provider set kubernetes \ + KUBERNETES_NAMESPACE=foxhunt \ + NODE_SELECTOR="k8s.scaleway.com/pool-name=gpu-dev" \ + DISK_SIZE=10Gi \ + STORAGE_CLASS=scw-bssd \ + INACTIVITY_TIMEOUT=30m \ + IMAGE_PULL_SECRETS=gitlab-registry \ + POD_MANIFEST_TEMPLATE=.devcontainer/pod-template.yaml +``` + +## GitLab Integration + +### "Open in DevPod" Badge + +Added as a GitLab project badge (Settings → General → Badges): + +``` +Name: Open in DevPod +Link: https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git +Image: https://devpod.sh/assets/open-in-devpod.svg +``` + +Also usable as: +- CLI: `devpod up git@100.90.76.85:2222/root/foxhunt.git` +- Bookmark in any browser +- Shared link in docs/chat + +### CI Image Build + +New job in `.gitlab-ci.yml`: + +```yaml +build-devcontainer: + stage: build + rules: + - changes: [".devcontainer/**"] + script: + # Kaniko build + push to GitLab registry +``` + +Only triggers when `.devcontainer/` files change. Uses existing Kaniko +pattern from the service image builds. + +## Session Lifecycle + +``` +Click "Open in DevPod" (or `devpod up`) + → DevPod creates pod targeting gpu-dev pool + → Kapsule autoscaler provisions GP1-L node (~2-3 min) + → Pod starts, mounts PVCs (home + training data) + → SSH tunnel established + → Terminal opens + +Active development + → cargo check, claude, kubectl, glab, terraform + → All writes go to persistent /home/dev PVC + +Idle 30 minutes (no SSH activity) + → DevPod stops pod + → Node idle 10 min + → Kapsule autoscaler removes GP1-L node + → Cost: €0 + +Return (click link or `devpod up` again) + → Node provisions (~2-3 min) + → Pod starts, mounts SAME PVCs + → Claude Code config, cargo cache, shell history intact + → Incremental cargo builds (sccache hits) +``` + +## GPU Access (On-Demand) + +The dev pod is CPU-only. GPU access for training/inference: + +- **CI training jobs** → H100 pool (existing, autoscaled to zero) +- **Ad-hoc training** → `kubectl apply -f infra/k8s/training/job-template.yaml` + from inside the dev pod +- **Future**: second DevPod provider profile targeting L4 or H100 pool with + `nvidia.com/gpu=1` for interactive GPU sessions + +## Cost Estimate + +| Scenario | Node | Rate | Daily (8hr) | Monthly | +|----------|------|------|-------------|---------| +| Active dev | GP1-L | ~€0.24/hr | ~€2 | ~€44 | +| Idle | — | €0 | €0 | €0 | +| CI build (H100) | H100-1-80G | ~€3-4/hr | per-job | per-job | + +No conflict between dev and CI — separate node pools. + +## Files to Create/Modify + +| File | Action | +|------|--------| +| `.devcontainer/devcontainer.json` | Create | +| `.devcontainer/Dockerfile` | Create | +| `.devcontainer/pod-template.yaml` | Create | +| `infra/modules/kapsule/main.tf` | Add gpu-dev pool resource | +| `infra/modules/kapsule/variables.tf` | Add dev pool variables | +| `infra/live/production/kapsule/terragrunt.hcl` | Enable dev pool, set type | +| `.gitlab-ci.yml` | Add build-devcontainer job | +| `scripts/devpod-setup.sh` | One-time provider config script |