docs: add Kapsule infrastructure design

Scaleway Kapsule with 3 node pools (always-on, CI, GPU),
Tailscale-only networking, H100-preferred GPU training,
and Gitea Actions CI/CD pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs/plans/2026-02-24-kapsule-infra-design.md

@@ -0,0 +1,154 @@
+# Kapsule Infrastructure Design
+
+**Date**: 2026-02-24
+**Status**: Approved
+
+## Problem
+
+Foxhunt needs CI/CD and a deployment target for paper trading. Current state: Gitea on Scaleway DEV1-S (Tailscale-only), local development only. No automated builds, no staging environment, no GPU training infrastructure.
+
+## Decisions
+
+- **Provider**: Scaleway only (single provider)
+- **Orchestration**: Kapsule (free control plane, scale-to-zero node pools)
+- **Network**: Zero public access, Tailscale subnet router as sole gateway
+- **GPU**: H100-80GB preferred, fallback to L4/3070 if unavailable, single pool
+- **Storage**: Scaleway native (Object Storage, Block Storage, Secret Manager)
+- **CI**: Gitea Actions with Kapsule runner on GP1-XS
+
+## Architecture
+
+### Kapsule Cluster: 3 Node Pools
+
+| Pool | Instance | Scale | Purpose |
+|------|----------|-------|---------|
+| `always-on` | DEV1-M (3 vCPU, 4GB) | 1 | Services, DBs, training orchestrator, Tailscale router |
+| `ci` | GP1-XS (4 vCPU, 16GB) | 0 → 1 → 0 | CI builds (Gitea Actions runner) |
+| `gpu` | H100-80GB | 0 → N → 0 | All ML training (fallback: L4-24GB, GPU-3070-S-8GB) |
+
+Control plane is free. Always-on node: ~EUR 15/mo. CI and GPU pools cost only when running.
+
+### Networking: Tailscale Only
+
+```
+Developer laptop (Tailscale) ──┐
+                                ├── Tailscale mesh ── subnet-router pod ── Kapsule pod CIDR
+Gitea server (Tailscale)  ─────┘
+```
+
+- No public IPs on any node
+- No ingress controllers, no LoadBalancers
+- One Tailscale subnet router pod on always-on node advertises Kapsule pod CIDR to tailnet
+- All access (kubectl, services, dashboards, web-dashboard) through Tailscale
+- Same security model as existing Gitea server
+
+### CI/CD Pipeline
+
+```
+git push
+  → Gitea webhook
+  → Act runner job (ci pool scales 0 → 1)
+    → cargo check --workspace (sccache from Object Storage)
+    → cargo test --workspace
+    → cargo clippy --workspace
+    → docker build + push → Scaleway Container Registry
+    → kubectl apply (deploy to Kapsule)
+  → ci pool scales back to 0
+```
+
+- **Build cache**: sccache backed by Scaleway Object Storage (S3-compatible)
+- **Cost per run**: ~EUR 0.035 on GP1-XS (minutes, not hours)
+- **Runner**: Gitea Act runner as K8s Job, triggered by webhook
+- **Registry**: Scaleway Container Registry (private, same region)
+
+### GPU Training
+
+#### Single Pool, H100-Preferred
+
+One GPU node pool. H100-80GB is the default request. If Scaleway inventory is unavailable, training orchestrator falls back to L4-24GB, then GPU-3070-S-8GB. Pool scales to zero when no jobs are running.
+
+#### Training Orchestrator
+
+Runs on always-on node as a long-lived deployment. Receives training requests, decomposes them into Kubernetes Jobs, manages GPU pool scaling.
+
+```
+CLI / API request (preset + params)
+  → training-orchestrator (always-on node)
+    → scale gpu pool 0 → N
+    → create K8s Jobs (one per model)
+    → monitor completion
+    → upload checkpoints to Object Storage
+    → scale gpu pool N → 0
+```
+
+#### Presets
+
+| Preset | Models | Data | Est. H100 Time | Use Case |
+|--------|--------|------|----------------|----------|
+| `quick-test` | 1 model | 3mo, 1 asset | seconds | Dev iteration |
+| `single-model` | 1 model | full dataset | minutes | Single model tuning |
+| `full-ensemble` | all 10 | full dataset | minutes | Full retrain |
+| `hyperopt` | 1 model, N trials | configurable | varies | Hyperparameter search |
+
+#### Budget Guardrails
+
+- `max_concurrent_gpus`: cap simultaneous GPU nodes (default: 2)
+- `max_cost_per_run`: hard EUR limit per training job
+- `idle_timeout`: 10 minutes no-activity → scale GPU pool to 0
+- Orphaned node reaper: kills GPU nodes with no running K8s Jobs after timeout
+- All GPU jobs have hard wall-clock timeouts
+
+### Storage
+
+| What | Service | Access |
+|------|---------|--------|
+| Training data (OHLCV + MBP10) | Block Storage (shared read-only) | Mounted on GPU nodes |
+| Model checkpoints + artifacts | Object Storage (S3) | S3 API from training jobs |
+| Build cache (sccache) | Object Storage (S3) | S3 API from CI jobs |
+| Docker images | Container Registry | K8s image pull |
+| Secrets (JWT, DB creds, API keys) | Scaleway Secret Manager | K8s SecretProviderClass |
+
+Block Storage is shared read-only across all GPU nodes — training data is downloaded once, mounted everywhere. No re-downloading per job.
+
+### Paper Trading Deployment (always-on node)
+
+Stateful services with PersistentVolumeClaims:
+- PostgreSQL
+- Redis
+- QuestDB
+
+Stateless services (K8s Deployments):
+- trading_service
+- ml_training_service
+- api_gateway
+- broker_gateway_service
+- web-gateway (serves web-dashboard static assets)
+
+All services communicate over internal cluster networking. External access only via Tailscale.
+
+## Cost Estimate
+
+| Component | Monthly Cost |
+|-----------|-------------|
+| Kapsule control plane | Free |
+| always-on DEV1-M | ~EUR 15 |
+| CI GP1-XS (on-demand) | ~EUR 1-5 (per usage) |
+| GPU H100 (on-demand) | Pay per minute |
+| Object Storage | ~EUR 2-5 |
+| Block Storage 100GB | ~EUR 8 |
+| Container Registry | ~EUR 1-2 |
+| Secret Manager | Free tier |
+| **Total baseline** | **~EUR 25-35/mo + GPU** |
+
+GPU cost scales with usage. Quick tests cost cents. Full ensemble retrains cost more but complete in minutes on H100.
+
+## Implementation Order
+
+1. Kapsule cluster + always-on node pool + Tailscale subnet router
+2. Scaleway Object Storage bucket + Secret Manager setup
+3. Container Registry + base Docker images
+4. Gitea Actions CI pipeline (webhook → build → push → deploy)
+5. Paper trading stack deployment (DBs + services)
+6. Block Storage mount + training orchestrator
+7. GPU node pool + training presets
+8. Budget guardrails + monitoring