docs: add DevPod remote development environment design
DevPod on Kapsule with GP1-L (16 vCPU, 64GB) autoscaling to zero, persistent PVCs for Claude Code + cargo cache, "Open in DevPod" GitLab badge, CI-built devcontainer image. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
232
docs/plans/2026-02-25-devcontainer-design.md
Normal file
232
docs/plans/2026-02-25-devcontainer-design.md
Normal file
@@ -0,0 +1,232 @@
|
||||
# DevPod Remote Development Environment — Design
|
||||
|
||||
**Date**: 2026-02-25
|
||||
**Status**: Approved
|
||||
|
||||
## Goal
|
||||
|
||||
A one-click remote development environment on Kapsule with full Claude Code
|
||||
toolchain, persistent storage, and autoscale-to-zero cost control. Accessible
|
||||
from anywhere via an "Open in DevPod" link on the GitLab project page.
|
||||
|
||||
## Decisions
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| Tool | DevPod (Loft Labs) | Zero server infra, native devcontainer.json, built-in idle timeout, provider-agnostic |
|
||||
| Dev node | GP1-L (16 vCPU, 64GB) | Fast cargo builds on 37-crate workspace, autoscales to zero |
|
||||
| GPU for dev | None (CPU-only) | cargo check/clippy/test are CPU-bound; GPU training uses H100 via CI/jobs |
|
||||
| GPU for training | H100 (existing pool) | Stays autoscaled to zero, wakes for CI and ad-hoc training jobs |
|
||||
| Connection | Terminal + Claude Code | No VS Code/JetBrains server overhead |
|
||||
| Persistence | Block storage PVCs | Survive pod restarts, node scale-down, Claude Code auto-updates |
|
||||
| GitLab integration | Project badge + CI image build | "Open in DevPod" button, image rebuilt on .devcontainer/ changes |
|
||||
| Eliminated | GitLab Workspaces (no GPU, Devfile only), Coder (overkill), raw SSH (manual lifecycle) |
|
||||
|
||||
## Infrastructure
|
||||
|
||||
### New Kapsule Pool: `gpu-dev`
|
||||
|
||||
```hcl
|
||||
resource "scaleway_k8s_pool" "gpu_dev" {
|
||||
name = "gpu-dev"
|
||||
node_type = "GP1-L" # 16 vCPU, 64GB RAM
|
||||
min_size = 0
|
||||
max_size = 1
|
||||
autoscaling = true
|
||||
autohealing = true
|
||||
|
||||
lifecycle {
|
||||
ignore_changes = [size]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
No GPU taint, no special tolerations. The node selector
|
||||
`k8s.scaleway.com/pool-name=gpu-dev` directs the dev pod here.
|
||||
|
||||
### Persistent Volumes
|
||||
|
||||
| PVC | Size | StorageClass | Mount | Purpose |
|
||||
|-----|------|-------------|-------|---------|
|
||||
| `dev-home-pvc` | 50Gi | scw-bssd | `/home/dev` | Claude Code + ~/.claude/ + cargo registry + sccache cache + shell/git config |
|
||||
| `training-data-pvc` | 10Gi (existing) | scw-bssd | `/data/training` | Databento .dbn.zst futures data (read-only) |
|
||||
|
||||
DevPod also creates a workspace volume for `/workspace` (the repo checkout).
|
||||
|
||||
The 50Gi home volume is intentionally large: cargo registry + sccache for 37
|
||||
crates, Claude Code plugins/MCP caches, and future datasets or model checkpoints.
|
||||
The PVC can be resized, mounted in other pods, or snapshotted as needed.
|
||||
|
||||
## Devcontainer Image
|
||||
|
||||
Built by CI, pushed to `git.fxhnt.ai:5050/root/foxhunt/devcontainer:latest`.
|
||||
|
||||
### Base
|
||||
|
||||
Extends `Dockerfile.ci-builder` (CUDA 12.4, Rust 1.89, protoc, sccache, clippy).
|
||||
|
||||
### Additional Layers
|
||||
|
||||
**Dev tooling**:
|
||||
- Node.js 22 LTS (Claude Code runtime)
|
||||
- Claude Code CLI (`npm install -g @anthropic-ai/claude-code`)
|
||||
- Python 3.12 + pip (scripts, data tooling)
|
||||
- kubectl, helm (cluster management)
|
||||
- scw CLI (Scaleway management)
|
||||
- glab (GitLab CLI)
|
||||
- terraform, terragrunt (infra management)
|
||||
- jq, yq, ripgrep, fd-find (CLI essentials)
|
||||
- zsh + oh-my-zsh (shell)
|
||||
- git-lfs
|
||||
|
||||
**Environment**:
|
||||
- Non-root user `dev` with passwordless sudo
|
||||
- Cargo env in `/home/dev/.cargo`
|
||||
- sccache configured (local disk cache on PVC)
|
||||
- `SQLX_OFFLINE=true`
|
||||
- `CUDA_COMPUTE_CAP=90`
|
||||
|
||||
## `.devcontainer/` Structure
|
||||
|
||||
```
|
||||
.devcontainer/
|
||||
├── devcontainer.json # DevPod + devcontainer spec
|
||||
├── Dockerfile # Extends ci-builder with dev tooling
|
||||
└── pod-template.yaml # K8s pod overrides (PVC mounts, node selector)
|
||||
```
|
||||
|
||||
### devcontainer.json
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"name": "foxhunt",
|
||||
"build": {
|
||||
"dockerfile": "Dockerfile",
|
||||
"context": ".."
|
||||
},
|
||||
"remoteUser": "dev",
|
||||
"containerEnv": {
|
||||
"SQLX_OFFLINE": "true",
|
||||
"CUDA_COMPUTE_CAP": "90"
|
||||
},
|
||||
"postStartCommand": "claude --version && cargo --version && kubectl cluster-info 2>/dev/null || true",
|
||||
"customizations": {
|
||||
"devpod": {
|
||||
"provider": "kubernetes"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
PVC mounts and node selector are handled by `pod-template.yaml` (DevPod's
|
||||
`POD_MANIFEST_TEMPLATE` option), since devcontainer.json cannot express k8s-
|
||||
specific volume claims.
|
||||
|
||||
## DevPod Provider Configuration
|
||||
|
||||
One-time setup on the developer's machine (`scripts/devpod-setup.sh`):
|
||||
|
||||
```bash
|
||||
devpod provider add kubernetes
|
||||
devpod provider set kubernetes \
|
||||
KUBERNETES_NAMESPACE=foxhunt \
|
||||
NODE_SELECTOR="k8s.scaleway.com/pool-name=gpu-dev" \
|
||||
DISK_SIZE=10Gi \
|
||||
STORAGE_CLASS=scw-bssd \
|
||||
INACTIVITY_TIMEOUT=30m \
|
||||
IMAGE_PULL_SECRETS=gitlab-registry \
|
||||
POD_MANIFEST_TEMPLATE=.devcontainer/pod-template.yaml
|
||||
```
|
||||
|
||||
## GitLab Integration
|
||||
|
||||
### "Open in DevPod" Badge
|
||||
|
||||
Added as a GitLab project badge (Settings → General → Badges):
|
||||
|
||||
```
|
||||
Name: Open in DevPod
|
||||
Link: https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git
|
||||
Image: https://devpod.sh/assets/open-in-devpod.svg
|
||||
```
|
||||
|
||||
Also usable as:
|
||||
- CLI: `devpod up git@100.90.76.85:2222/root/foxhunt.git`
|
||||
- Bookmark in any browser
|
||||
- Shared link in docs/chat
|
||||
|
||||
### CI Image Build
|
||||
|
||||
New job in `.gitlab-ci.yml`:
|
||||
|
||||
```yaml
|
||||
build-devcontainer:
|
||||
stage: build
|
||||
rules:
|
||||
- changes: [".devcontainer/**"]
|
||||
script:
|
||||
# Kaniko build + push to GitLab registry
|
||||
```
|
||||
|
||||
Only triggers when `.devcontainer/` files change. Uses existing Kaniko
|
||||
pattern from the service image builds.
|
||||
|
||||
## Session Lifecycle
|
||||
|
||||
```
|
||||
Click "Open in DevPod" (or `devpod up`)
|
||||
→ DevPod creates pod targeting gpu-dev pool
|
||||
→ Kapsule autoscaler provisions GP1-L node (~2-3 min)
|
||||
→ Pod starts, mounts PVCs (home + training data)
|
||||
→ SSH tunnel established
|
||||
→ Terminal opens
|
||||
|
||||
Active development
|
||||
→ cargo check, claude, kubectl, glab, terraform
|
||||
→ All writes go to persistent /home/dev PVC
|
||||
|
||||
Idle 30 minutes (no SSH activity)
|
||||
→ DevPod stops pod
|
||||
→ Node idle 10 min
|
||||
→ Kapsule autoscaler removes GP1-L node
|
||||
→ Cost: €0
|
||||
|
||||
Return (click link or `devpod up` again)
|
||||
→ Node provisions (~2-3 min)
|
||||
→ Pod starts, mounts SAME PVCs
|
||||
→ Claude Code config, cargo cache, shell history intact
|
||||
→ Incremental cargo builds (sccache hits)
|
||||
```
|
||||
|
||||
## GPU Access (On-Demand)
|
||||
|
||||
The dev pod is CPU-only. GPU access for training/inference:
|
||||
|
||||
- **CI training jobs** → H100 pool (existing, autoscaled to zero)
|
||||
- **Ad-hoc training** → `kubectl apply -f infra/k8s/training/job-template.yaml`
|
||||
from inside the dev pod
|
||||
- **Future**: second DevPod provider profile targeting L4 or H100 pool with
|
||||
`nvidia.com/gpu=1` for interactive GPU sessions
|
||||
|
||||
## Cost Estimate
|
||||
|
||||
| Scenario | Node | Rate | Daily (8hr) | Monthly |
|
||||
|----------|------|------|-------------|---------|
|
||||
| Active dev | GP1-L | ~€0.24/hr | ~€2 | ~€44 |
|
||||
| Idle | — | €0 | €0 | €0 |
|
||||
| CI build (H100) | H100-1-80G | ~€3-4/hr | per-job | per-job |
|
||||
|
||||
No conflict between dev and CI — separate node pools.
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `.devcontainer/devcontainer.json` | Create |
|
||||
| `.devcontainer/Dockerfile` | Create |
|
||||
| `.devcontainer/pod-template.yaml` | Create |
|
||||
| `infra/modules/kapsule/main.tf` | Add gpu-dev pool resource |
|
||||
| `infra/modules/kapsule/variables.tf` | Add dev pool variables |
|
||||
| `infra/live/production/kapsule/terragrunt.hcl` | Enable dev pool, set type |
|
||||
| `.gitlab-ci.yml` | Add build-devcontainer job |
|
||||
| `scripts/devpod-setup.sh` | One-time provider config script |
|
||||
Reference in New Issue
Block a user