From 73cb7e500f8e06b27e37a196aede70c3b355759d Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 24 Feb 2026 14:07:56 +0100 Subject: [PATCH] docs: add Kapsule infrastructure design Scaleway Kapsule with 3 node pools (always-on, CI, GPU), Tailscale-only networking, H100-preferred GPU training, and Gitea Actions CI/CD pipeline. Co-Authored-By: Claude Opus 4.6 --- docs/plans/2026-02-24-kapsule-infra-design.md | 154 ++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 docs/plans/2026-02-24-kapsule-infra-design.md diff --git a/docs/plans/2026-02-24-kapsule-infra-design.md b/docs/plans/2026-02-24-kapsule-infra-design.md new file mode 100644 index 000000000..734093b77 --- /dev/null +++ b/docs/plans/2026-02-24-kapsule-infra-design.md @@ -0,0 +1,154 @@ +# Kapsule Infrastructure Design + +**Date**: 2026-02-24 +**Status**: Approved + +## Problem + +Foxhunt needs CI/CD and a deployment target for paper trading. Current state: Gitea on Scaleway DEV1-S (Tailscale-only), local development only. No automated builds, no staging environment, no GPU training infrastructure. + +## Decisions + +- **Provider**: Scaleway only (single provider) +- **Orchestration**: Kapsule (free control plane, scale-to-zero node pools) +- **Network**: Zero public access, Tailscale subnet router as sole gateway +- **GPU**: H100-80GB preferred, fallback to L4/3070 if unavailable, single pool +- **Storage**: Scaleway native (Object Storage, Block Storage, Secret Manager) +- **CI**: Gitea Actions with Kapsule runner on GP1-XS + +## Architecture + +### Kapsule Cluster: 3 Node Pools + +| Pool | Instance | Scale | Purpose | +|------|----------|-------|---------| +| `always-on` | DEV1-M (3 vCPU, 4GB) | 1 | Services, DBs, training orchestrator, Tailscale router | +| `ci` | GP1-XS (4 vCPU, 16GB) | 0 → 1 → 0 | CI builds (Gitea Actions runner) | +| `gpu` | H100-80GB | 0 → N → 0 | All ML training (fallback: L4-24GB, GPU-3070-S-8GB) | + +Control plane is free. Always-on node: ~EUR 15/mo. CI and GPU pools cost only when running. + +### Networking: Tailscale Only + +``` +Developer laptop (Tailscale) ──┐ + ├── Tailscale mesh ── subnet-router pod ── Kapsule pod CIDR +Gitea server (Tailscale) ─────┘ +``` + +- No public IPs on any node +- No ingress controllers, no LoadBalancers +- One Tailscale subnet router pod on always-on node advertises Kapsule pod CIDR to tailnet +- All access (kubectl, services, dashboards, web-dashboard) through Tailscale +- Same security model as existing Gitea server + +### CI/CD Pipeline + +``` +git push + → Gitea webhook + → Act runner job (ci pool scales 0 → 1) + → cargo check --workspace (sccache from Object Storage) + → cargo test --workspace + → cargo clippy --workspace + → docker build + push → Scaleway Container Registry + → kubectl apply (deploy to Kapsule) + → ci pool scales back to 0 +``` + +- **Build cache**: sccache backed by Scaleway Object Storage (S3-compatible) +- **Cost per run**: ~EUR 0.035 on GP1-XS (minutes, not hours) +- **Runner**: Gitea Act runner as K8s Job, triggered by webhook +- **Registry**: Scaleway Container Registry (private, same region) + +### GPU Training + +#### Single Pool, H100-Preferred + +One GPU node pool. H100-80GB is the default request. If Scaleway inventory is unavailable, training orchestrator falls back to L4-24GB, then GPU-3070-S-8GB. Pool scales to zero when no jobs are running. + +#### Training Orchestrator + +Runs on always-on node as a long-lived deployment. Receives training requests, decomposes them into Kubernetes Jobs, manages GPU pool scaling. + +``` +CLI / API request (preset + params) + → training-orchestrator (always-on node) + → scale gpu pool 0 → N + → create K8s Jobs (one per model) + → monitor completion + → upload checkpoints to Object Storage + → scale gpu pool N → 0 +``` + +#### Presets + +| Preset | Models | Data | Est. H100 Time | Use Case | +|--------|--------|------|----------------|----------| +| `quick-test` | 1 model | 3mo, 1 asset | seconds | Dev iteration | +| `single-model` | 1 model | full dataset | minutes | Single model tuning | +| `full-ensemble` | all 10 | full dataset | minutes | Full retrain | +| `hyperopt` | 1 model, N trials | configurable | varies | Hyperparameter search | + +#### Budget Guardrails + +- `max_concurrent_gpus`: cap simultaneous GPU nodes (default: 2) +- `max_cost_per_run`: hard EUR limit per training job +- `idle_timeout`: 10 minutes no-activity → scale GPU pool to 0 +- Orphaned node reaper: kills GPU nodes with no running K8s Jobs after timeout +- All GPU jobs have hard wall-clock timeouts + +### Storage + +| What | Service | Access | +|------|---------|--------| +| Training data (OHLCV + MBP10) | Block Storage (shared read-only) | Mounted on GPU nodes | +| Model checkpoints + artifacts | Object Storage (S3) | S3 API from training jobs | +| Build cache (sccache) | Object Storage (S3) | S3 API from CI jobs | +| Docker images | Container Registry | K8s image pull | +| Secrets (JWT, DB creds, API keys) | Scaleway Secret Manager | K8s SecretProviderClass | + +Block Storage is shared read-only across all GPU nodes — training data is downloaded once, mounted everywhere. No re-downloading per job. + +### Paper Trading Deployment (always-on node) + +Stateful services with PersistentVolumeClaims: +- PostgreSQL +- Redis +- QuestDB + +Stateless services (K8s Deployments): +- trading_service +- ml_training_service +- api_gateway +- broker_gateway_service +- web-gateway (serves web-dashboard static assets) + +All services communicate over internal cluster networking. External access only via Tailscale. + +## Cost Estimate + +| Component | Monthly Cost | +|-----------|-------------| +| Kapsule control plane | Free | +| always-on DEV1-M | ~EUR 15 | +| CI GP1-XS (on-demand) | ~EUR 1-5 (per usage) | +| GPU H100 (on-demand) | Pay per minute | +| Object Storage | ~EUR 2-5 | +| Block Storage 100GB | ~EUR 8 | +| Container Registry | ~EUR 1-2 | +| Secret Manager | Free tier | +| **Total baseline** | **~EUR 25-35/mo + GPU** | + +GPU cost scales with usage. Quick tests cost cents. Full ensemble retrains cost more but complete in minutes on H100. + +## Implementation Order + +1. Kapsule cluster + always-on node pool + Tailscale subnet router +2. Scaleway Object Storage bucket + Secret Manager setup +3. Container Registry + base Docker images +4. Gitea Actions CI pipeline (webhook → build → push → deploy) +5. Paper trading stack deployment (DBs + services) +6. Block Storage mount + training orchestrator +7. GPU node pool + training presets +8. Budget guardrails + monitoring