From 67f01ac4a1162a042d82f6d6a78e6363c559e809 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 24 Feb 2026 14:12:45 +0100 Subject: [PATCH] docs: add Kapsule infrastructure implementation plan 14 tasks across 5 batches: cluster foundation, CI/CD, paper trading deployment, GPU training, and verification. Co-Authored-By: Claude Opus 4.6 --- ...2026-02-24-kapsule-infra-implementation.md | 1951 +++++++++++++++++ 1 file changed, 1951 insertions(+) create mode 100644 docs/plans/2026-02-24-kapsule-infra-implementation.md diff --git a/docs/plans/2026-02-24-kapsule-infra-implementation.md b/docs/plans/2026-02-24-kapsule-infra-implementation.md new file mode 100644 index 000000000..196f56e30 --- /dev/null +++ b/docs/plans/2026-02-24-kapsule-infra-implementation.md @@ -0,0 +1,1951 @@ +# Kapsule Infrastructure Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Deploy foxhunt on Scaleway Kapsule with CI/CD, paper trading, and GPU training — all behind Tailscale. + +**Architecture:** 3-pool Kapsule cluster (always-on DEV1-M, CI GP1-XS scale-to-zero, GPU H100 scale-to-zero). Tailscale subnet router for zero-public-access networking. Gitea Actions for CI. Scaleway Object Storage / Secret Manager / Block Storage for data. + +**Tech Stack:** Scaleway CLI (`scw`), Kubernetes (`kubectl`), Helm, Tailscale, Gitea Actions, sccache, Scaleway Container Registry + +--- + +## Batch 1: Cluster Foundation + +### Task 1: Scaleway CLI setup and Kapsule cluster creation + +**Files:** +- Create: `infra/scripts/create-cluster.sh` + +**Step 1: Write the cluster creation script** + +```bash +#!/usr/bin/env bash +set -euo pipefail + +# Foxhunt Kapsule Cluster Creation +# Prerequisites: scw CLI configured, kubectl installed + +CLUSTER_NAME="foxhunt" +REGION="nl-ams" # Amsterdam (same as Gitea server) +K8S_VERSION="1.30" + +echo "Creating Kapsule cluster: ${CLUSTER_NAME}" +scw k8s cluster create \ + name="${CLUSTER_NAME}" \ + version="${K8S_VERSION}" \ + cni=cilium \ + region="${REGION}" \ + --wait + +CLUSTER_ID=$(scw k8s cluster list name="${CLUSTER_NAME}" region="${REGION}" -o json | jq -r '.[0].id') +echo "Cluster ID: ${CLUSTER_ID}" + +# Create always-on node pool (DEV1-M: 3 vCPU, 4GB) +echo "Creating always-on node pool" +scw k8s pool create \ + cluster-id="${CLUSTER_ID}" \ + name="always-on" \ + node-type="DEV1-M" \ + size=1 \ + min-size=1 \ + max-size=1 \ + autoscaling=false \ + autohealing=true \ + region="${REGION}" \ + --wait + +# Create CI node pool (GP1-XS: 4 vCPU, 16GB, scale-to-zero) +echo "Creating CI node pool (scale-to-zero)" +scw k8s pool create \ + cluster-id="${CLUSTER_ID}" \ + name="ci" \ + node-type="GP1-XS" \ + size=0 \ + min-size=0 \ + max-size=1 \ + autoscaling=true \ + autohealing=true \ + region="${REGION}" \ + --wait + +# Create GPU node pool (H100-80GB, scale-to-zero) +echo "Creating GPU node pool (scale-to-zero, H100)" +scw k8s pool create \ + cluster-id="${CLUSTER_ID}" \ + name="gpu" \ + node-type="H100-1-80G" \ + size=0 \ + min-size=0 \ + max-size=2 \ + autoscaling=true \ + autohealing=true \ + region="${REGION}" \ + --wait + +# Download kubeconfig +echo "Downloading kubeconfig" +scw k8s kubeconfig install "${CLUSTER_ID}" region="${REGION}" + +echo "Cluster ready. Verify with: kubectl get nodes" +kubectl get nodes +``` + +**Step 2: Run the script** + +Run: `chmod +x infra/scripts/create-cluster.sh && bash infra/scripts/create-cluster.sh` +Expected: Cluster created, kubeconfig installed, 1 node in `Ready` state (always-on pool). + +**Step 3: Verify cluster** + +Run: `kubectl get nodes -o wide && kubectl get namespaces` +Expected: 1 node from always-on pool, default namespaces present. + +**Step 4: Create foxhunt namespace** + +Run: `kubectl create namespace foxhunt && kubectl config set-context --current --namespace=foxhunt` +Expected: Namespace created, context switched. + +**Step 5: Commit** + +```bash +git add infra/scripts/create-cluster.sh +git commit -m "infra: add Kapsule cluster creation script" +``` + +--- + +### Task 2: Tailscale subnet router deployment + +**Files:** +- Create: `infra/k8s/tailscale/namespace.yaml` +- Create: `infra/k8s/tailscale/rbac.yaml` +- Create: `infra/k8s/tailscale/deployment.yaml` +- Create: `infra/k8s/tailscale/secret.yaml.example` + +**Step 1: Write the Tailscale namespace and RBAC** + +`infra/k8s/tailscale/namespace.yaml`: +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: tailscale +``` + +`infra/k8s/tailscale/rbac.yaml`: +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: tailscale + namespace: tailscale +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: tailscale + namespace: tailscale +rules: + - apiGroups: [""] + resources: ["secrets"] + verbs: ["create", "get", "update", "patch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: tailscale + namespace: tailscale +subjects: + - kind: ServiceAccount + name: tailscale +roleRef: + kind: Role + name: tailscale + apiGroup: rbac.authorization.k8s.io +``` + +**Step 2: Write the Tailscale deployment** + +`infra/k8s/tailscale/deployment.yaml`: +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: tailscale-subnet-router + namespace: tailscale +spec: + replicas: 1 + selector: + matchLabels: + app: tailscale-subnet-router + template: + metadata: + labels: + app: tailscale-subnet-router + spec: + serviceAccountName: tailscale + nodeSelector: + # Pin to always-on pool + k8s.scaleway.com/pool-name: always-on + containers: + - name: tailscale + image: ghcr.io/tailscale/tailscale:latest + env: + - name: TS_AUTHKEY + valueFrom: + secretKeyRef: + name: tailscale-auth + key: TS_AUTHKEY + - name: TS_KUBE_SECRET + value: tailscale-state + - name: TS_USERSPACE + value: "false" + - name: TS_ROUTES + # Advertise Kapsule pod CIDR — get from: + # kubectl get nodes -o jsonpath='{.items[0].spec.podCIDR}' + value: "10.42.0.0/16" + - name: TS_HOSTNAME + value: "foxhunt-kapsule" + - name: TS_ACCEPT_DNS + value: "false" + securityContext: + capabilities: + add: ["NET_ADMIN", "NET_RAW"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 200m + memory: 128Mi +``` + +**Step 3: Write the secret example file** + +`infra/k8s/tailscale/secret.yaml.example`: +```yaml +# Copy to secret.yaml and fill in your Tailscale auth key +# Generate at: https://login.tailscale.com/admin/settings/keys +# Use a reusable, ephemeral key tagged with tag:k8s +apiVersion: v1 +kind: Secret +metadata: + name: tailscale-auth + namespace: tailscale +type: Opaque +stringData: + TS_AUTHKEY: "tskey-auth-REPLACE_ME" +``` + +**Step 4: Apply manifests** + +Run: +```bash +kubectl apply -f infra/k8s/tailscale/namespace.yaml +kubectl apply -f infra/k8s/tailscale/rbac.yaml +# Create real secret (not the example) +kubectl create secret generic tailscale-auth \ + --namespace=tailscale \ + --from-literal=TS_AUTHKEY="$(read -sp 'Tailscale auth key: ' key && echo $key)" +kubectl apply -f infra/k8s/tailscale/deployment.yaml +``` + +Expected: Pod running, Tailscale connected. + +**Step 5: Verify Tailscale connectivity** + +Run: `kubectl -n tailscale logs deployment/tailscale-subnet-router | grep -i "connected\|ready"` +Then from your laptop: `tailscale status | grep foxhunt-kapsule` +Expected: `foxhunt-kapsule` visible in tailnet, subnet routes advertised. + +**Step 6: Approve routes in Tailscale admin** + +Go to https://login.tailscale.com/admin/machines → find `foxhunt-kapsule` → approve subnet routes `10.42.0.0/16`. + +**Step 7: Commit** + +```bash +git add infra/k8s/tailscale/ +git commit -m "infra: add Tailscale subnet router for Kapsule" +``` + +--- + +### Task 3: Scaleway Object Storage + Container Registry + +**Files:** +- Create: `infra/scripts/setup-storage.sh` + +**Step 1: Write the storage setup script** + +```bash +#!/usr/bin/env bash +set -euo pipefail + +REGION="nl-ams" + +# Create Object Storage bucket for build cache and model artifacts +echo "Creating Object Storage bucket" +scw object bucket create name=foxhunt-artifacts region="${REGION}" + +# Create sub-prefixes (virtual directories) +# sccache/, models/, training-data/ — created automatically on first write + +# Create Container Registry namespace +echo "Creating Container Registry namespace" +scw registry namespace create \ + name=foxhunt \ + region="${REGION}" \ + is-public=false + +REGISTRY_ENDPOINT=$(scw registry namespace list name=foxhunt region="${REGION}" -o json | jq -r '.[0].endpoint') +echo "Registry endpoint: ${REGISTRY_ENDPOINT}" +echo "Login with: docker login ${REGISTRY_ENDPOINT}" + +# Create Kubernetes secret for registry pull +echo "Creating K8s registry pull secret" +kubectl create secret docker-registry scw-registry \ + --namespace=foxhunt \ + --docker-server="${REGISTRY_ENDPOINT}" \ + --docker-username=foxhunt \ + --docker-password="$(scw iam api-key list -o json | jq -r '.[0].secret_key')" + +echo "Done. Registry: ${REGISTRY_ENDPOINT}" +``` + +**Step 2: Run the script** + +Run: `chmod +x infra/scripts/setup-storage.sh && bash infra/scripts/setup-storage.sh` +Expected: Bucket created, registry namespace created, K8s pull secret created. + +**Step 3: Commit** + +```bash +git add infra/scripts/setup-storage.sh +git commit -m "infra: add Object Storage and Container Registry setup" +``` + +--- + +### Task 4: Scaleway Secret Manager integration + +**Files:** +- Create: `infra/scripts/setup-secrets.sh` +- Create: `infra/k8s/secrets/external-secrets.yaml` + +**Step 1: Write secret provisioning script** + +```bash +#!/usr/bin/env bash +set -euo pipefail + +REGION="nl-ams" +PROJECT_ID=$(scw account project list -o json | jq -r '.[0].id') + +# Create secrets in Scaleway Secret Manager +echo "Creating secrets in Scaleway Secret Manager" + +# JWT secret (generate random 64-char) +JWT_SECRET=$(openssl rand -base64 48) +scw secret secret create \ + name=foxhunt-jwt-secret \ + project-id="${PROJECT_ID}" \ + region="${REGION}" + +JWT_SECRET_ID=$(scw secret secret list name=foxhunt-jwt-secret region="${REGION}" -o json | jq -r '.[0].id') +echo -n "${JWT_SECRET}" | scw secret version create \ + secret-id="${JWT_SECRET_ID}" \ + data=- \ + region="${REGION}" + +# Database password +DB_PASSWORD=$(openssl rand -base64 32) +scw secret secret create \ + name=foxhunt-db-password \ + project-id="${PROJECT_ID}" \ + region="${REGION}" + +DB_SECRET_ID=$(scw secret secret list name=foxhunt-db-password region="${REGION}" -o json | jq -r '.[0].id') +echo -n "${DB_PASSWORD}" | scw secret version create \ + secret-id="${DB_SECRET_ID}" \ + data=- \ + region="${REGION}" + +echo "Secrets created. IDs:" +echo " JWT: ${JWT_SECRET_ID}" +echo " DB: ${DB_SECRET_ID}" +echo "" +echo "For K8s, create opaque secrets manually or use external-secrets-operator." +``` + +**Step 2: Write K8s secrets (pulled from Secret Manager at deploy time)** + +`infra/k8s/secrets/external-secrets.yaml`: +```yaml +# For now, secrets are created manually via setup-secrets.sh +# and injected as K8s secrets. When external-secrets-operator +# is needed, add Scaleway provider config here. +# +# Manual K8s secret creation (run after setup-secrets.sh): +# +# kubectl create secret generic foxhunt-secrets \ +# --namespace=foxhunt \ +# --from-literal=JWT_SECRET="" \ +# --from-literal=DATABASE_PASSWORD="" \ +# --from-literal=REDIS_PASSWORD="" +``` + +**Step 3: Run the script** + +Run: `chmod +x infra/scripts/setup-secrets.sh && bash infra/scripts/setup-secrets.sh` +Expected: Secrets created in Scaleway Secret Manager. + +**Step 4: Create K8s secret** + +Run: +```bash +kubectl create secret generic foxhunt-secrets \ + --namespace=foxhunt \ + --from-literal=JWT_SECRET="$(scw secret version access-payload /1 region=nl-ams -o json | jq -r '.data' | base64 -d)" \ + --from-literal=DATABASE_PASSWORD="$(scw secret version access-payload /1 region=nl-ams -o json | jq -r '.data' | base64 -d)" +``` + +**Step 5: Commit** + +```bash +git add infra/scripts/setup-secrets.sh infra/k8s/secrets/ +git commit -m "infra: add Scaleway Secret Manager setup" +``` + +--- + +## Batch 2: CI/CD Pipeline + +### Task 5: Unified service Dockerfile with sccache + +**Files:** +- Create: `infra/docker/Dockerfile.service` + +The existing per-service Dockerfiles copy the entire workspace. This unified Dockerfile does the same but adds sccache for S3-backed build caching. + +**Step 1: Write the unified Dockerfile** + +```dockerfile +# Foxhunt Service Builder — Unified multi-stage build +# Usage: docker build --build-arg SERVICE=trading_service -f infra/docker/Dockerfile.service . +# +# Supports sccache via S3 (set SCCACHE_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) + +ARG RUST_VERSION=1.89 + +# ============================================================================= +# Builder +# ============================================================================= +FROM rust:${RUST_VERSION}-slim-bookworm AS builder + +ARG SERVICE +ARG SCCACHE_BUCKET="" +ARG AWS_ACCESS_KEY_ID="" +ARG AWS_SECRET_ACCESS_KEY="" +ARG AWS_DEFAULT_REGION="nl-ams" +ARG SCCACHE_ENDPOINT="" + +RUN apt-get update && apt-get install -y \ + pkg-config libssl-dev protobuf-compiler perl make curl \ + && rm -rf /var/lib/apt/lists/* + +# Install sccache if bucket is configured +RUN if [ -n "${SCCACHE_BUCKET}" ]; then \ + curl -sSL https://github.com/mozilla/sccache/releases/download/v0.8.1/sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz \ + | tar xz -C /usr/local/bin --strip-components=1 sccache-v0.8.1-x86_64-unknown-linux-musl/sccache; \ + fi + +ENV RUSTC_WRAPPER=${SCCACHE_BUCKET:+/usr/local/bin/sccache} +ENV SCCACHE_BUCKET=${SCCACHE_BUCKET} +ENV SCCACHE_S3_USE_SSL=true +ENV SCCACHE_ENDPOINT=${SCCACHE_ENDPOINT} + +WORKDIR /build + +COPY Cargo.toml Cargo.lock ./ +COPY .sqlx ./.sqlx + +# Copy all workspace members +COPY trading_engine ./trading_engine +COPY risk ./risk +COPY risk-data ./risk-data +COPY trading-data ./trading-data +COPY fxt ./fxt +COPY ml ./ml +COPY ml-data ./ml-data +COPY data ./data +COPY backtesting ./backtesting +COPY adaptive-strategy ./adaptive-strategy +COPY common ./common +COPY storage ./storage +COPY model_loader ./model_loader +COPY market-data ./market-data +COPY database ./database +COPY config ./config +COPY web-gateway ./web-gateway +COPY ctrader-openapi ./ctrader-openapi +COPY foxhunt-deploy ./foxhunt-deploy +COPY services ./services +COPY tests ./tests + +ENV SQLX_OFFLINE=true + +RUN cargo build --release -p ${SERVICE} \ + && mkdir -p /build/out \ + && cp /build/target/release/${SERVICE} /build/out/ + +RUN strip /build/out/${SERVICE} + +# Print sccache stats if active +RUN if [ -n "${SCCACHE_BUCKET}" ]; then sccache --show-stats; fi + +# ============================================================================= +# Runtime +# ============================================================================= +FROM debian:bookworm-slim + +ARG SERVICE + +RUN apt-get update && apt-get install -y \ + ca-certificates libssl3 curl \ + && rm -rf /var/lib/apt/lists/* + +RUN curl -sSL https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.25/grpc_health_probe-linux-amd64 \ + -o /usr/local/bin/grpc_health_probe && chmod +x /usr/local/bin/grpc_health_probe + +RUN groupadd --system --gid 1000 foxhunt && \ + useradd --system --uid 1000 --gid foxhunt --shell /bin/bash foxhunt + +RUN mkdir -p /app/config /app/logs /app/data && chown -R foxhunt:foxhunt /app + +WORKDIR /app + +COPY --from=builder /build/out/${SERVICE} ./${SERVICE} +RUN chmod +x ./${SERVICE} + +USER foxhunt + +ENTRYPOINT ["./${SERVICE}"] +``` + +**Step 2: Test build locally for one service** + +Run: `DOCKER_BUILDKIT=1 docker build --build-arg SERVICE=trading_service -f infra/docker/Dockerfile.service -t foxhunt-trading-service:test .` +Expected: Build succeeds, image created. + +**Step 3: Commit** + +```bash +git add infra/docker/Dockerfile.service +git commit -m "infra: add unified service Dockerfile with sccache support" +``` + +--- + +### Task 6: Web-gateway Dockerfile (serves dashboard static assets) + +**Files:** +- Create: `infra/docker/Dockerfile.web-gateway` + +**Step 1: Write the web-gateway Dockerfile** + +```dockerfile +# Foxhunt Web Gateway — Axum server + React dashboard static files +# Two-phase: build dashboard (Node), build web-gateway (Rust), combine in runtime + +# ============================================================================= +# Stage 1: Build React dashboard +# ============================================================================= +FROM node:22-slim AS dashboard-builder + +WORKDIR /dashboard +COPY web-dashboard/package.json web-dashboard/package-lock.json ./ +RUN npm ci +COPY web-dashboard/ ./ +RUN npm run build +# Output: /dashboard/dist/ + +# ============================================================================= +# Stage 2: Build web-gateway Rust binary +# ============================================================================= +FROM rust:1.89-slim-bookworm AS rust-builder + +RUN apt-get update && apt-get install -y \ + pkg-config libssl-dev protobuf-compiler perl make \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /build + +COPY Cargo.toml Cargo.lock ./ +COPY .sqlx ./.sqlx +COPY trading_engine ./trading_engine +COPY risk ./risk +COPY risk-data ./risk-data +COPY trading-data ./trading-data +COPY fxt ./fxt +COPY ml ./ml +COPY ml-data ./ml-data +COPY data ./data +COPY backtesting ./backtesting +COPY adaptive-strategy ./adaptive-strategy +COPY common ./common +COPY storage ./storage +COPY model_loader ./model_loader +COPY market-data ./market-data +COPY database ./database +COPY config ./config +COPY web-gateway ./web-gateway +COPY ctrader-openapi ./ctrader-openapi +COPY foxhunt-deploy ./foxhunt-deploy +COPY services ./services +COPY tests ./tests + +ENV SQLX_OFFLINE=true +RUN cargo build --release -p web-gateway \ + && mkdir -p /build/out \ + && cp /build/target/release/web-gateway /build/out/ \ + && strip /build/out/web-gateway + +# ============================================================================= +# Stage 3: Runtime +# ============================================================================= +FROM debian:bookworm-slim + +RUN apt-get update && apt-get install -y \ + ca-certificates libssl3 curl \ + && rm -rf /var/lib/apt/lists/* + +RUN groupadd --system --gid 1000 foxhunt && \ + useradd --system --uid 1000 --gid foxhunt --shell /bin/bash foxhunt + +RUN mkdir -p /app/static && chown -R foxhunt:foxhunt /app + +WORKDIR /app + +COPY --from=rust-builder /build/out/web-gateway ./web-gateway +COPY --from=dashboard-builder /dashboard/dist/ ./static/ +RUN chmod +x ./web-gateway + +USER foxhunt + +EXPOSE 3000 + +HEALTHCHECK --interval=10s --timeout=5s --start-period=10s --retries=3 \ + CMD curl -f http://localhost:3000/api/health || exit 1 + +ENTRYPOINT ["./web-gateway"] +``` + +**Step 2: Test build locally** + +Run: `DOCKER_BUILDKIT=1 docker build -f infra/docker/Dockerfile.web-gateway -t foxhunt-web-gateway:test .` +Expected: Build succeeds (dashboard + Rust binary combined). + +**Step 3: Commit** + +```bash +git add infra/docker/Dockerfile.web-gateway +git commit -m "infra: add web-gateway Dockerfile with embedded dashboard" +``` + +--- + +### Task 7: Gitea Actions CI workflow + +**Files:** +- Create: `.gitea/workflows/ci.yaml` + +**Step 1: Write the CI workflow** + +```yaml +name: CI + +on: + push: + branches: [main] + pull_request: + branches: [main] + +env: + SQLX_OFFLINE: "true" + CARGO_TERM_COLOR: always + SCCACHE_BUCKET: foxhunt-artifacts + SCCACHE_ENDPOINT: https://s3.nl-ams.scw.cloud + SCCACHE_S3_USE_SSL: "true" + RUSTC_WRAPPER: /usr/local/bin/sccache + +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install Rust + uses: dtolnay/rust-toolchain@stable + with: + components: clippy + + - name: Install sccache + run: | + curl -sSL https://github.com/mozilla/sccache/releases/download/v0.8.1/sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz \ + | tar xz -C /usr/local/bin --strip-components=1 sccache-v0.8.1-x86_64-unknown-linux-musl/sccache + + - name: Install protobuf compiler + run: sudo apt-get update && sudo apt-get install -y protobuf-compiler + + - name: Cargo check + run: cargo check --workspace + + - name: Cargo clippy + run: cargo clippy --workspace -- -D warnings + + - name: sccache stats + run: sccache --show-stats + + test: + runs-on: ubuntu-latest + needs: check + steps: + - uses: actions/checkout@v4 + + - name: Install Rust + uses: dtolnay/rust-toolchain@stable + + - name: Install sccache + run: | + curl -sSL https://github.com/mozilla/sccache/releases/download/v0.8.1/sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz \ + | tar xz -C /usr/local/bin --strip-components=1 sccache-v0.8.1-x86_64-unknown-linux-musl/sccache + + - name: Install protobuf compiler + run: sudo apt-get update && sudo apt-get install -y protobuf-compiler + + - name: Run tests + run: cargo test --workspace --lib + + - name: sccache stats + run: sccache --show-stats + + build-images: + runs-on: ubuntu-latest + needs: test + if: github.ref == 'refs/heads/main' + strategy: + matrix: + service: + - trading_service + - api_gateway + - broker_gateway_service + - ml_training_service + - backtesting_service + - trading_agent_service + steps: + - uses: actions/checkout@v4 + + - name: Login to Scaleway Registry + run: echo "${{ secrets.SCW_SECRET_KEY }}" | docker login rg.nl-ams.scw.cloud -u nologin --password-stdin + + - name: Build and push ${{ matrix.service }} + run: | + IMAGE=rg.nl-ams.scw.cloud/foxhunt/${{ matrix.service }}:${{ github.sha }} + IMAGE_LATEST=rg.nl-ams.scw.cloud/foxhunt/${{ matrix.service }}:latest + DOCKER_BUILDKIT=1 docker build \ + --build-arg SERVICE=${{ matrix.service }} \ + --build-arg SCCACHE_BUCKET=foxhunt-artifacts \ + --build-arg SCCACHE_ENDPOINT=https://s3.nl-ams.scw.cloud \ + --build-arg AWS_ACCESS_KEY_ID=${{ secrets.SCW_ACCESS_KEY }} \ + --build-arg AWS_SECRET_ACCESS_KEY=${{ secrets.SCW_SECRET_KEY }} \ + -f infra/docker/Dockerfile.service \ + -t "${IMAGE}" \ + -t "${IMAGE_LATEST}" \ + . + docker push "${IMAGE}" + docker push "${IMAGE_LATEST}" + + build-web-gateway: + runs-on: ubuntu-latest + needs: test + if: github.ref == 'refs/heads/main' + steps: + - uses: actions/checkout@v4 + + - name: Login to Scaleway Registry + run: echo "${{ secrets.SCW_SECRET_KEY }}" | docker login rg.nl-ams.scw.cloud -u nologin --password-stdin + + - name: Build and push web-gateway + run: | + IMAGE=rg.nl-ams.scw.cloud/foxhunt/web-gateway:${{ github.sha }} + IMAGE_LATEST=rg.nl-ams.scw.cloud/foxhunt/web-gateway:latest + DOCKER_BUILDKIT=1 docker build \ + -f infra/docker/Dockerfile.web-gateway \ + -t "${IMAGE}" \ + -t "${IMAGE_LATEST}" \ + . + docker push "${IMAGE}" + docker push "${IMAGE_LATEST}" + + deploy: + runs-on: ubuntu-latest + needs: [build-images, build-web-gateway] + if: github.ref == 'refs/heads/main' + steps: + - uses: actions/checkout@v4 + + - name: Install kubectl + run: | + curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" + chmod +x kubectl && mv kubectl /usr/local/bin/ + + - name: Configure kubeconfig + run: echo "${{ secrets.KUBECONFIG }}" | base64 -d > /tmp/kubeconfig + env: + KUBECONFIG: /tmp/kubeconfig + + - name: Deploy to Kapsule + run: | + export KUBECONFIG=/tmp/kubeconfig + # Update image tags in all deployments + for svc in trading-service api-gateway broker-gateway ml-training-service backtesting-service trading-agent-service web-gateway; do + kubectl -n foxhunt set image deployment/${svc} \ + ${svc}=rg.nl-ams.scw.cloud/foxhunt/${svc}:${{ github.sha }} \ + 2>/dev/null || true + done + kubectl -n foxhunt rollout status deployment --timeout=120s +``` + +**Step 2: Verify workflow syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/ci.yaml'))" && echo "YAML valid"` +Expected: "YAML valid" + +**Step 3: Commit** + +```bash +git add .gitea/workflows/ci.yaml +git commit -m "ci: add Gitea Actions workflow (check, test, build, deploy)" +``` + +--- + +## Batch 3: Paper Trading Deployment + +### Task 8: Database manifests (PostgreSQL, Redis, QuestDB) + +**Files:** +- Create: `infra/k8s/databases/postgres.yaml` +- Create: `infra/k8s/databases/redis.yaml` +- Create: `infra/k8s/databases/questdb.yaml` + +**Step 1: Write PostgreSQL manifest** + +`infra/k8s/databases/postgres.yaml`: +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: postgres-pvc + namespace: foxhunt +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 10Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: postgres + namespace: foxhunt +spec: + replicas: 1 + selector: + matchLabels: + app: postgres + template: + metadata: + labels: + app: postgres + spec: + nodeSelector: + k8s.scaleway.com/pool-name: always-on + containers: + - name: postgres + image: timescale/timescaledb:latest-pg16 + ports: + - containerPort: 5432 + env: + - name: POSTGRES_DB + value: foxhunt + - name: POSTGRES_USER + value: foxhunt + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: foxhunt-secrets + key: DATABASE_PASSWORD + volumeMounts: + - name: postgres-data + mountPath: /var/lib/postgresql/data + readinessProbe: + exec: + command: ["pg_isready", "-U", "foxhunt"] + initialDelaySeconds: 5 + periodSeconds: 10 + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + cpu: 500m + memory: 1Gi + volumes: + - name: postgres-data + persistentVolumeClaim: + claimName: postgres-pvc +--- +apiVersion: v1 +kind: Service +metadata: + name: postgres + namespace: foxhunt +spec: + selector: + app: postgres + ports: + - port: 5432 + targetPort: 5432 +``` + +**Step 2: Write Redis manifest** + +`infra/k8s/databases/redis.yaml`: +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: redis + namespace: foxhunt +spec: + replicas: 1 + selector: + matchLabels: + app: redis + template: + metadata: + labels: + app: redis + spec: + nodeSelector: + k8s.scaleway.com/pool-name: always-on + containers: + - name: redis + image: redis:7-alpine + command: ["redis-server", "--maxmemory", "512mb", "--maxmemory-policy", "allkeys-lru"] + ports: + - containerPort: 6379 + readinessProbe: + exec: + command: ["redis-cli", "ping"] + periodSeconds: 10 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 250m + memory: 640Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: redis + namespace: foxhunt +spec: + selector: + app: redis + ports: + - port: 6379 + targetPort: 6379 +``` + +**Step 3: Write QuestDB manifest** + +`infra/k8s/databases/questdb.yaml`: +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: questdb-pvc + namespace: foxhunt +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 10Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: questdb + namespace: foxhunt +spec: + replicas: 1 + selector: + matchLabels: + app: questdb + template: + metadata: + labels: + app: questdb + spec: + nodeSelector: + k8s.scaleway.com/pool-name: always-on + containers: + - name: questdb + image: questdb/questdb:8.2.3 + ports: + - containerPort: 9009 # ILP + - containerPort: 8812 # PostgreSQL wire + - containerPort: 9003 # HTTP + env: + - name: QDB_PG_ENABLED + value: "true" + volumeMounts: + - name: questdb-data + mountPath: /var/lib/questdb + readinessProbe: + httpGet: + path: /exec?query=SELECT%201 + port: 9003 + periodSeconds: 10 + resources: + requests: + cpu: 200m + memory: 512Mi + limits: + cpu: 500m + memory: 1Gi + volumes: + - name: questdb-data + persistentVolumeClaim: + claimName: questdb-pvc +--- +apiVersion: v1 +kind: Service +metadata: + name: questdb + namespace: foxhunt +spec: + selector: + app: questdb + ports: + - name: ilp + port: 9009 + targetPort: 9009 + - name: pg + port: 8812 + targetPort: 8812 + - name: http + port: 9003 + targetPort: 9003 +``` + +**Step 4: Apply database manifests** + +Run: +```bash +kubectl apply -f infra/k8s/databases/postgres.yaml +kubectl apply -f infra/k8s/databases/redis.yaml +kubectl apply -f infra/k8s/databases/questdb.yaml +kubectl -n foxhunt get pods -w +``` + +Expected: All 3 pods running and ready. + +**Step 5: Commit** + +```bash +git add infra/k8s/databases/ +git commit -m "infra: add K8s manifests for PostgreSQL, Redis, QuestDB" +``` + +--- + +### Task 9: Service deployments + +**Files:** +- Create: `infra/k8s/services/trading-service.yaml` +- Create: `infra/k8s/services/api-gateway.yaml` +- Create: `infra/k8s/services/broker-gateway.yaml` +- Create: `infra/k8s/services/ml-training-service.yaml` +- Create: `infra/k8s/services/backtesting-service.yaml` +- Create: `infra/k8s/services/trading-agent-service.yaml` +- Create: `infra/k8s/services/web-gateway.yaml` + +**Step 1: Write trading-service deployment** + +`infra/k8s/services/trading-service.yaml`: +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: trading-service + namespace: foxhunt +spec: + replicas: 1 + selector: + matchLabels: + app: trading-service + template: + metadata: + labels: + app: trading-service + spec: + nodeSelector: + k8s.scaleway.com/pool-name: always-on + imagePullSecrets: + - name: scw-registry + containers: + - name: trading-service + image: rg.nl-ams.scw.cloud/foxhunt/trading_service:latest + ports: + - containerPort: 50051 + name: grpc + - containerPort: 9092 + name: metrics + env: + - name: DATABASE_URL + value: postgresql://foxhunt:$(DATABASE_PASSWORD)@postgres:5432/foxhunt + - name: DATABASE_PASSWORD + valueFrom: + secretKeyRef: + name: foxhunt-secrets + key: DATABASE_PASSWORD + - name: REDIS_URL + value: redis://redis:6379 + - name: JWT_SECRET + valueFrom: + secretKeyRef: + name: foxhunt-secrets + key: JWT_SECRET + - name: JWT_ISSUER + value: foxhunt-api-gateway + - name: JWT_AUDIENCE + value: foxhunt-services + - name: QUESTDB_ILP_HOST + value: questdb:9009 + - name: GRPC_PORT + value: "50051" + - name: RUST_LOG + value: info + readinessProbe: + exec: + command: ["/usr/local/bin/grpc_health_probe", "-addr=localhost:50051"] + initialDelaySeconds: 10 + periodSeconds: 10 + resources: + requests: + cpu: 200m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: trading-service + namespace: foxhunt +spec: + selector: + app: trading-service + ports: + - name: grpc + port: 50051 + targetPort: 50051 + - name: metrics + port: 9092 + targetPort: 9092 +``` + +**Step 2: Write remaining service deployments** + +Follow the same pattern as trading-service for each service. Key differences per service: + +| Service | Image | gRPC Port | Metrics Port | Extra Env | +|---------|-------|-----------|-------------|-----------| +| `api-gateway` | `api_gateway` | 50050 | 9091 | `TRADING_SERVICE_URL=http://trading-service:50051` etc. | +| `broker-gateway` | `broker_gateway_service` | 50056 | 9096 | — | +| `ml-training-service` | `ml_training_service` | 50053 | 9094 | `S3_ENDPOINT`, `S3_BUCKET` | +| `backtesting-service` | `backtesting_service` | 50053 | 9093 | — | +| `trading-agent-service` | `trading_agent_service` | 50055 | 9095 | — | +| `web-gateway` | `web-gateway` | 3000 (HTTP) | — | `STATIC_DIR=/app/static` | + +Write each as a separate YAML file following the trading-service pattern. For `web-gateway`, use HTTP health check instead of gRPC probe. + +`infra/k8s/services/web-gateway.yaml`: +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: web-gateway + namespace: foxhunt +spec: + replicas: 1 + selector: + matchLabels: + app: web-gateway + template: + metadata: + labels: + app: web-gateway + spec: + nodeSelector: + k8s.scaleway.com/pool-name: always-on + imagePullSecrets: + - name: scw-registry + containers: + - name: web-gateway + image: rg.nl-ams.scw.cloud/foxhunt/web-gateway:latest + ports: + - containerPort: 3000 + name: http + env: + - name: JWT_SECRET + valueFrom: + secretKeyRef: + name: foxhunt-secrets + key: JWT_SECRET + - name: TRADING_SERVICE_URL + value: http://trading-service:50051 + - name: BACKTESTING_SERVICE_URL + value: http://backtesting-service:50053 + - name: ML_TRAINING_SERVICE_URL + value: http://ml-training-service:50053 + - name: TRADING_AGENT_SERVICE_URL + value: http://trading-agent-service:50055 + - name: RUST_LOG + value: info + readinessProbe: + httpGet: + path: /api/health + port: 3000 + initialDelaySeconds: 5 + periodSeconds: 10 + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 250m + memory: 256Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: web-gateway + namespace: foxhunt +spec: + selector: + app: web-gateway + ports: + - name: http + port: 3000 + targetPort: 3000 +``` + +**Step 3: Apply all service manifests** + +Run: +```bash +kubectl apply -f infra/k8s/services/ +kubectl -n foxhunt get pods -w +``` + +Expected: All 7 service pods running. + +**Step 4: Verify end-to-end through Tailscale** + +From your laptop (on Tailscale): +```bash +# Get web-gateway pod IP +WG_IP=$(kubectl -n foxhunt get svc web-gateway -o jsonpath='{.spec.clusterIP}') +curl http://${WG_IP}:3000/api/health +``` + +Expected: Health check returns OK. + +**Step 5: Commit** + +```bash +git add infra/k8s/services/ +git commit -m "infra: add K8s service deployments for paper trading stack" +``` + +--- + +## Batch 4: GPU Training Infrastructure + +### Task 10: Block Storage for training data + +**Files:** +- Create: `infra/scripts/setup-block-storage.sh` +- Create: `infra/k8s/storage/training-data-pv.yaml` + +**Step 1: Write Block Storage provisioning script** + +```bash +#!/usr/bin/env bash +set -euo pipefail + +REGION="nl-ams" +ZONE="nl-ams-1" + +# Create 100GB Block Storage volume for training data +echo "Creating Block Storage volume for training data" +scw instance volume create \ + name=foxhunt-training-data \ + volume-type=b_ssd \ + size=100GB \ + zone="${ZONE}" + +VOL_ID=$(scw instance volume list name=foxhunt-training-data zone="${ZONE}" -o json | jq -r '.[0].id') +echo "Volume ID: ${VOL_ID}" +echo "Use this ID in the PersistentVolume manifest" +``` + +**Step 2: Write PV/PVC manifests** + +`infra/k8s/storage/training-data-pv.yaml`: +```yaml +# PersistentVolume backed by Scaleway Block Storage +# Replace VOLUME_ID with actual volume ID from setup-block-storage.sh +apiVersion: v1 +kind: PersistentVolume +metadata: + name: training-data-pv +spec: + capacity: + storage: 100Gi + accessModes: + - ReadOnlyMany + csi: + driver: csi.scaleway.com + volumeHandle: "VOLUME_ID" + fsType: ext4 +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: training-data-pvc + namespace: foxhunt +spec: + accessModes: + - ReadOnlyMany + resources: + requests: + storage: 100Gi + volumeName: training-data-pv +``` + +**Step 3: Upload training data to block storage** + +This is a manual step — attach the volume to the always-on node, mount it, upload data: +```bash +# From a node with the volume attached: +mount /dev/sda /mnt/training-data +rsync -avz data/cache/futures-baseline/ /mnt/training-data/futures-baseline/ +``` + +**Step 4: Commit** + +```bash +git add infra/scripts/setup-block-storage.sh infra/k8s/storage/ +git commit -m "infra: add Block Storage for shared training data" +``` + +--- + +### Task 11: GPU training Job template + +**Files:** +- Create: `infra/k8s/training/job-template.yaml` +- Create: `infra/docker/Dockerfile.training` + +**Step 1: Write GPU training Dockerfile** + +`infra/docker/Dockerfile.training`: +```dockerfile +# Foxhunt ML Training — CUDA-enabled build for GPU training jobs +# Based on existing Dockerfile.foxhunt-build pattern + +ARG RUST_VERSION=1.89 +ARG CUDA_VERSION=12.4.1 + +# ============================================================================= +# Builder +# ============================================================================= +FROM nvidia/cuda:${CUDA_VERSION}-cudnn-devel-ubuntu22.04 AS builder + +RUN apt-get update && apt-get install -y \ + curl git build-essential pkg-config libssl-dev ca-certificates protobuf-compiler \ + && rm -rf /var/lib/apt/lists/* + +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable +ENV PATH="/root/.cargo/bin:${PATH}" + +WORKDIR /build + +COPY Cargo.toml Cargo.lock ./ +COPY .sqlx ./.sqlx +COPY trading_engine ./trading_engine +COPY risk ./risk +COPY risk-data ./risk-data +COPY trading-data ./trading-data +COPY fxt ./fxt +COPY ml ./ml +COPY ml-data ./ml-data +COPY data ./data +COPY backtesting ./backtesting +COPY adaptive-strategy ./adaptive-strategy +COPY common ./common +COPY storage ./storage +COPY model_loader ./model_loader +COPY market-data ./market-data +COPY database ./database +COPY config ./config +COPY web-gateway ./web-gateway +COPY ctrader-openapi ./ctrader-openapi +COPY foxhunt-deploy ./foxhunt-deploy +COPY services ./services +COPY tests ./tests + +ENV SQLX_OFFLINE=true +ENV CUDA_COMPUTE_CAP=90 + +# Build all training examples +RUN --mount=type=cache,target=/usr/local/cargo/registry \ + --mount=type=cache,target=/usr/local/cargo/git \ + --mount=type=cache,target=/build/target \ + cargo build -p ml --release --features ml/cuda \ + --example train_dqn \ + --example train_ppo_parquet \ + --example hyperopt_dqn_demo \ + --example hyperopt_ppo_demo \ + --example hyperopt_tft_demo \ + --example hyperopt_mamba2_demo && \ + mkdir -p /build/out && \ + cp /build/target/release/examples/train_dqn /build/out/ && \ + cp /build/target/release/examples/train_ppo_parquet /build/out/ && \ + cp /build/target/release/examples/hyperopt_dqn_demo /build/out/ && \ + cp /build/target/release/examples/hyperopt_ppo_demo /build/out/ && \ + cp /build/target/release/examples/hyperopt_tft_demo /build/out/ && \ + cp /build/target/release/examples/hyperopt_mamba2_demo /build/out/ + +RUN strip /build/out/* + +# ============================================================================= +# Runtime +# ============================================================================= +FROM nvidia/cuda:${CUDA_VERSION}-cudnn-runtime-ubuntu22.04 + +RUN apt-get update && apt-get install -y ca-certificates libssl3 curl && rm -rf /var/lib/apt/lists/* + +RUN useradd -m -u 1000 foxhunt + +WORKDIR /workspace + +COPY --from=builder /build/out/ /usr/local/bin/ + +RUN chmod +x /usr/local/bin/train_* /usr/local/bin/hyperopt_* +RUN chown -R foxhunt:foxhunt /workspace + +USER foxhunt + +ENV CUDA_HOME=/usr/local/cuda +ENV PATH=${CUDA_HOME}/bin:${PATH} +ENV LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH} + +CMD ["echo", "Specify training command via K8s Job args"] +``` + +**Step 2: Write K8s Job template** + +`infra/k8s/training/job-template.yaml`: +```yaml +# Template for GPU training jobs +# Usage: copy and customize command/args per training run +# +# The training orchestrator will generate Job manifests from this template. +apiVersion: batch/v1 +kind: Job +metadata: + # name: train-dqn-TIMESTAMP (generated) + namespace: foxhunt + labels: + foxhunt/job-type: training +spec: + backoffLimit: 1 + activeDeadlineSeconds: 3600 # 1 hour hard timeout + ttlSecondsAfterFinished: 600 # Cleanup after 10 min + template: + metadata: + labels: + foxhunt/job-type: training + spec: + restartPolicy: Never + nodeSelector: + k8s.scaleway.com/pool-name: gpu + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + imagePullSecrets: + - name: scw-registry + containers: + - name: training + image: rg.nl-ams.scw.cloud/foxhunt/training:latest + # Override per job: + command: ["train_dqn"] + args: + - "--symbol" + - "ES.FUT" + - "--data-dir" + - "/data/futures-baseline" + - "--max-steps-per-epoch" + - "2000" + env: + - name: RUST_LOG + value: info + - name: SQLX_OFFLINE + value: "true" + volumeMounts: + - name: training-data + mountPath: /data + readOnly: true + - name: output + mountPath: /workspace/output + resources: + requests: + nvidia.com/gpu: 1 + cpu: "4" + memory: 16Gi + limits: + nvidia.com/gpu: 1 + cpu: "8" + memory: 32Gi + volumes: + - name: training-data + persistentVolumeClaim: + claimName: training-data-pvc + - name: output + emptyDir: {} +``` + +**Step 3: Commit** + +```bash +git add infra/docker/Dockerfile.training infra/k8s/training/ +git commit -m "infra: add GPU training Dockerfile and K8s Job template" +``` + +--- + +### Task 12: Training orchestrator script + +**Files:** +- Create: `infra/scripts/train.sh` + +**Step 1: Write the training orchestrator** + +```bash +#!/usr/bin/env bash +set -euo pipefail + +# Foxhunt Training Orchestrator +# Submits K8s training Jobs to GPU pool +# +# Usage: +# ./infra/scripts/train.sh quick-test --model dqn --symbol ES.FUT +# ./infra/scripts/train.sh single-model --model ppo --symbol ES.FUT +# ./infra/scripts/train.sh full-ensemble --symbol ES.FUT +# ./infra/scripts/train.sh hyperopt --model dqn --trials 20 --symbol ES.FUT + +PRESET="${1:?Usage: train.sh [--model MODEL] [--symbol SYMBOL] [--trials N]}" +shift + +# Defaults +MODEL="" +SYMBOL="ES.FUT" +TRIALS=20 +MAX_STEPS=2000 +TIMEOUT=3600 +DATA_DIR="/data/futures-baseline" + +while [[ $# -gt 0 ]]; do + case "$1" in + --model) MODEL="$2"; shift 2 ;; + --symbol) SYMBOL="$2"; shift 2 ;; + --trials) TRIALS="$2"; shift 2 ;; + --max-steps) MAX_STEPS="$2"; shift 2 ;; + --timeout) TIMEOUT="$2"; shift 2 ;; + *) echo "Unknown arg: $1"; exit 1 ;; + esac +done + +NAMESPACE="foxhunt" +IMAGE="rg.nl-ams.scw.cloud/foxhunt/training:latest" +TIMESTAMP=$(date +%s) + +# Map model names to training binaries +declare -A MODEL_BINARY=( + [dqn]="train_dqn" + [ppo]="train_ppo_parquet" + [tft]="hyperopt_tft_demo" + [mamba2]="hyperopt_mamba2_demo" +) + +ALL_MODELS=(dqn ppo tft mamba2) + +submit_job() { + local model="$1" + local binary="${MODEL_BINARY[$model]}" + local job_name="train-${model}-${TIMESTAMP}" + local extra_args="$2" + + echo "Submitting job: ${job_name} (${binary})" + + kubectl create -n "${NAMESPACE}" -f - </dev/null | wc -l) + echo "Active training jobs: ${ACTIVE}" + if [ "${ACTIVE}" -eq 0 ]; then + echo "No active training jobs. GPU pool will scale to 0 via autoscaler." + fi +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: gpu-reaper + namespace: foxhunt +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: gpu-reaper + namespace: foxhunt +rules: + - apiGroups: ["batch"] + resources: ["jobs"] + verbs: ["get", "list"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: gpu-reaper + namespace: foxhunt +subjects: + - kind: ServiceAccount + name: gpu-reaper +roleRef: + kind: Role + name: gpu-reaper + apiGroup: rbac.authorization.k8s.io +``` + +**Step 2: Apply** + +Run: `kubectl apply -f infra/k8s/training/idle-reaper.yaml` +Expected: CronJob created. + +**Step 3: Commit** + +```bash +git add infra/k8s/training/idle-reaper.yaml +git commit -m "infra: add GPU idle reaper CronJob for budget control" +``` + +--- + +## Batch 5: Verification + +### Task 14: End-to-end smoke test + +**Files:** +- Create: `infra/scripts/smoke-test.sh` + +**Step 1: Write the smoke test** + +```bash +#!/usr/bin/env bash +set -euo pipefail + +echo "=== Foxhunt Kapsule Smoke Test ===" + +NAMESPACE="foxhunt" +FAILURES=0 + +check() { + local name="$1" + local cmd="$2" + echo -n " ${name}... " + if eval "${cmd}" > /dev/null 2>&1; then + echo "OK" + else + echo "FAIL" + FAILURES=$((FAILURES + 1)) + fi +} + +echo "1. Cluster connectivity" +check "kubectl" "kubectl get nodes" +check "namespace" "kubectl get namespace ${NAMESPACE}" + +echo "2. Tailscale subnet router" +check "ts-pod" "kubectl -n tailscale get pods -l app=tailscale-subnet-router --field-selector=status.phase=Running --no-headers | grep -q ." +check "ts-mesh" "tailscale status | grep -q foxhunt-kapsule" + +echo "3. Databases" +check "postgres" "kubectl -n ${NAMESPACE} exec deployment/postgres -- pg_isready -U foxhunt" +check "redis" "kubectl -n ${NAMESPACE} exec deployment/redis -- redis-cli ping" +check "questdb" "kubectl -n ${NAMESPACE} exec deployment/questdb -- wget -qO- http://localhost:9003/exec?query=SELECT%201" + +echo "4. Services" +for svc in trading-service api-gateway broker-gateway ml-training-service backtesting-service trading-agent-service; do + check "${svc}" "kubectl -n ${NAMESPACE} get pods -l app=${svc} --field-selector=status.phase=Running --no-headers | grep -q ." +done +check "web-gateway" "kubectl -n ${NAMESPACE} get pods -l app=web-gateway --field-selector=status.phase=Running --no-headers | grep -q ." + +echo "5. Web gateway health (via Tailscale)" +WG_IP=$(kubectl -n ${NAMESPACE} get svc web-gateway -o jsonpath='{.spec.clusterIP}') +check "health-api" "curl -sf http://${WG_IP}:3000/api/health" + +echo "6. Node pools" +check "always-on" "kubectl get nodes -l k8s.scaleway.com/pool-name=always-on --no-headers | grep -q Ready" +check "ci-pool-exists" "scw k8s pool list cluster-id=\$(scw k8s cluster list name=foxhunt -o json | jq -r '.[0].id') -o json | jq -e '.[] | select(.name==\"ci\")'" +check "gpu-pool-exists" "scw k8s pool list cluster-id=\$(scw k8s cluster list name=foxhunt -o json | jq -r '.[0].id') -o json | jq -e '.[] | select(.name==\"gpu\")'" + +echo "" +if [ "${FAILURES}" -eq 0 ]; then + echo "ALL CHECKS PASSED" +else + echo "${FAILURES} CHECK(S) FAILED" + exit 1 +fi +``` + +**Step 2: Run the smoke test** + +Run: `chmod +x infra/scripts/smoke-test.sh && bash infra/scripts/smoke-test.sh` +Expected: All checks pass. + +**Step 3: Commit** + +```bash +git add infra/scripts/smoke-test.sh +git commit -m "infra: add end-to-end smoke test for Kapsule deployment" +``` + +--- + +## File Summary + +``` +infra/ +├── docker/ +│ ├── Dockerfile.service # Unified service builder (Task 5) +│ ├── Dockerfile.web-gateway # Web gateway + dashboard (Task 6) +│ └── Dockerfile.training # GPU training image (Task 11) +├── k8s/ +│ ├── tailscale/ +│ │ ├── namespace.yaml # (Task 2) +│ │ ├── rbac.yaml # (Task 2) +│ │ ├── deployment.yaml # (Task 2) +│ │ └── secret.yaml.example # (Task 2) +│ ├── databases/ +│ │ ├── postgres.yaml # (Task 8) +│ │ ├── redis.yaml # (Task 8) +│ │ └── questdb.yaml # (Task 8) +│ ├── services/ +│ │ ├── trading-service.yaml # (Task 9) +│ │ ├── api-gateway.yaml # (Task 9) +│ │ ├── broker-gateway.yaml # (Task 9) +│ │ ├── ml-training-service.yaml # (Task 9) +│ │ ├── backtesting-service.yaml # (Task 9) +│ │ ├── trading-agent-service.yaml # (Task 9) +│ │ └── web-gateway.yaml # (Task 9) +│ ├── secrets/ +│ │ └── external-secrets.yaml # (Task 4) +│ ├── storage/ +│ │ └── training-data-pv.yaml # (Task 10) +│ └── training/ +│ ├── job-template.yaml # (Task 11) +│ └── idle-reaper.yaml # (Task 13) +└── scripts/ + ├── create-cluster.sh # (Task 1) + ├── setup-storage.sh # (Task 3) + ├── setup-secrets.sh # (Task 4) + ├── setup-block-storage.sh # (Task 10) + ├── train.sh # (Task 12) + └── smoke-test.sh # (Task 14) + +.gitea/workflows/ +└── ci.yaml # (Task 7) +``` + +## Batch Execution Order + +| Batch | Tasks | Dependency | +|-------|-------|-----------| +| 1: Cluster Foundation | 1, 2, 3, 4 | Sequential (cluster first, then networking, then storage) | +| 2: CI/CD Pipeline | 5, 6, 7 | Parallel (Dockerfiles independent, CI workflow depends on both) | +| 3: Paper Trading | 8, 9 | Sequential (databases before services) | +| 4: GPU Training | 10, 11, 12, 13 | Sequential (storage → image → orchestrator → reaper) | +| 5: Verification | 14 | After all batches |