From 6210a5d4a315ea8e13aa3ebcb80f9cd820da55ed Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 24 Feb 2026 14:28:16 +0100 Subject: [PATCH] docs: update implementation plan for OpenTofu + Terragrunt Replaced imperative shell scripts with declarative OpenTofu modules and Terragrunt DRY configuration. 16 tasks across 4 batches. Co-Authored-By: Claude Opus 4.6 --- ...2026-02-24-kapsule-infra-implementation.md | 2133 ++++++++--------- 1 file changed, 946 insertions(+), 1187 deletions(-) diff --git a/docs/plans/2026-02-24-kapsule-infra-implementation.md b/docs/plans/2026-02-24-kapsule-infra-implementation.md index 196f56e30..f1985030c 100644 --- a/docs/plans/2026-02-24-kapsule-infra-implementation.md +++ b/docs/plans/2026-02-24-kapsule-infra-implementation.md @@ -2,433 +2,706 @@ > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. -**Goal:** Deploy foxhunt on Scaleway Kapsule with CI/CD, paper trading, and GPU training — all behind Tailscale. +**Goal:** Deploy foxhunt on Scaleway Kapsule with CI/CD, paper trading, and GPU training — all behind Tailscale. All cloud resources managed declaratively with OpenTofu + Terragrunt. -**Architecture:** 3-pool Kapsule cluster (always-on DEV1-M, CI GP1-XS scale-to-zero, GPU H100 scale-to-zero). Tailscale subnet router for zero-public-access networking. Gitea Actions for CI. Scaleway Object Storage / Secret Manager / Block Storage for data. +**Architecture:** OpenTofu modules for Scaleway resources (Kapsule, Object Storage, Registry, Secrets, Block Storage). Terragrunt for DRY configuration and dependency orchestration. K8s manifests for in-cluster resources (Tailscale, databases, services, training jobs). Dockerfiles for container images. -**Tech Stack:** Scaleway CLI (`scw`), Kubernetes (`kubectl`), Helm, Tailscale, Gitea Actions, sccache, Scaleway Container Registry +**Tech Stack:** OpenTofu 1.8+, Terragrunt 0.67+, Scaleway Terraform provider, kubectl, Helm (optional), Gitea Actions --- -## Batch 1: Cluster Foundation +## Directory Structure -### Task 1: Scaleway CLI setup and Kapsule cluster creation +``` +infra/ +├── modules/ # OpenTofu reusable modules +│ ├── kapsule/ # Cluster + 3 node pools +│ ├── object-storage/ # S3 bucket for artifacts +│ ├── registry/ # Container Registry namespace +│ ├── secrets/ # Secret Manager secrets +│ └── block-storage/ # Training data volume +├── live/ # Terragrunt live config +│ └── production/ +│ ├── terragrunt.hcl # Root: provider, S3 backend, common vars +│ ├── kapsule/ +│ │ └── terragrunt.hcl +│ ├── object-storage/ +│ │ └── terragrunt.hcl +│ ├── registry/ +│ │ └── terragrunt.hcl +│ ├── secrets/ +│ │ └── terragrunt.hcl +│ └── block-storage/ +│ └── terragrunt.hcl +├── docker/ # Dockerfiles +│ ├── Dockerfile.service # Unified service builder +│ ├── Dockerfile.web-gateway # Web gateway + dashboard +│ └── Dockerfile.training # GPU training image +├── k8s/ # K8s manifests (applied after TF) +│ ├── tailscale/ # Subnet router +│ ├── databases/ # PostgreSQL, Redis, QuestDB +│ ├── services/ # App deployments +│ ├── secrets/ # K8s secrets (from TF outputs) +│ └── training/ # Job template + idle reaper +└── scripts/ # Operational scripts only + ├── train.sh # Training orchestrator + └── smoke-test.sh # End-to-end verification +``` + +--- + +## Batch 1: OpenTofu Foundation + +### Task 1: Root Terragrunt config + S3 backend **Files:** -- Create: `infra/scripts/create-cluster.sh` +- Create: `infra/live/production/terragrunt.hcl` -**Step 1: Write the cluster creation script** +**Step 1: Write the root Terragrunt config** + +This file defines the Scaleway provider, remote state backend (Object Storage), and common variables shared by all modules. + +```hcl +# infra/live/production/terragrunt.hcl + +# Root terragrunt config for foxhunt production + +locals { + region = "nl-ams" + zone = "nl-ams-1" + project_id = get_env("SCW_DEFAULT_PROJECT_ID") +} + +# Remote state in Scaleway Object Storage (S3-compatible) +# Bootstrap: create bucket manually first, then import +remote_state { + backend = "s3" + generate = { + path = "backend.tf" + if_exists = "overwrite" + } + config = { + bucket = "foxhunt-tfstate" + key = "${path_relative_to_include()}/terraform.tfstate" + region = "nl-ams" + + # Scaleway S3 endpoint + endpoint = "https://s3.nl-ams.scw.cloud" + skip_credentials_validation = true + skip_metadata_api_check = true + skip_requesting_account_id = true + skip_region_validation = true + skip_s3_checksum = true + } +} + +# Generate provider config for all child modules +generate "provider" { + path = "provider.tf" + if_exists = "overwrite" + contents = < ~/.kube/config-foxhunt +export KUBECONFIG=~/.kube/config-foxhunt kubectl get nodes ``` -**Step 2: Run the script** +Expected: 1 node from always-on pool in Ready state. -Run: `chmod +x infra/scripts/create-cluster.sh && bash infra/scripts/create-cluster.sh` -Expected: Cluster created, kubeconfig installed, 1 node in `Ready` state (always-on pool). +**Step 5: Create foxhunt namespace** -**Step 3: Verify cluster** +Run: `kubectl create namespace foxhunt` -Run: `kubectl get nodes -o wide && kubectl get namespaces` -Expected: 1 node from always-on pool, default namespaces present. - -**Step 4: Create foxhunt namespace** - -Run: `kubectl create namespace foxhunt && kubectl config set-context --current --namespace=foxhunt` -Expected: Namespace created, context switched. - -**Step 5: Commit** +**Step 6: Commit** ```bash -git add infra/scripts/create-cluster.sh -git commit -m "infra: add Kapsule cluster creation script" +git add infra/modules/kapsule/ infra/live/production/kapsule/ +git commit -m "infra: add Kapsule OpenTofu module (cluster + 3 node pools)" ``` --- -### Task 2: Tailscale subnet router deployment +### Task 3: Object Storage module **Files:** -- Create: `infra/k8s/tailscale/namespace.yaml` -- Create: `infra/k8s/tailscale/rbac.yaml` -- Create: `infra/k8s/tailscale/deployment.yaml` -- Create: `infra/k8s/tailscale/secret.yaml.example` +- Create: `infra/modules/object-storage/main.tf` +- Create: `infra/modules/object-storage/variables.tf` +- Create: `infra/modules/object-storage/outputs.tf` +- Create: `infra/live/production/object-storage/terragrunt.hcl` -**Step 1: Write the Tailscale namespace and RBAC** +**Step 1: Write the module** -`infra/k8s/tailscale/namespace.yaml`: -```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: tailscale +`infra/modules/object-storage/variables.tf`: +```hcl +variable "region" { + type = string +} + +variable "bucket_name" { + type = string + default = "foxhunt-artifacts" +} +``` + +`infra/modules/object-storage/main.tf`: +```hcl +resource "scaleway_object_bucket" "artifacts" { + name = var.bucket_name + region = var.region + + lifecycle_rule { + enabled = true + prefix = "sccache/" + + expiration { + days = 30 + } + } + + versioning { + enabled = false + } +} +``` + +`infra/modules/object-storage/outputs.tf`: +```hcl +output "bucket_name" { + value = scaleway_object_bucket.artifacts.name +} + +output "endpoint" { + value = "https://s3.${var.region}.scw.cloud" +} +``` + +**Step 2: Write Terragrunt config** + +`infra/live/production/object-storage/terragrunt.hcl`: +```hcl +include "root" { + path = find_in_parent_folders() +} + +terraform { + source = "../../../modules/object-storage" +} + +inputs = { + bucket_name = "foxhunt-artifacts" +} +``` + +**Step 3: Apply** + +Run: `cd infra/live/production/object-storage && terragrunt apply` + +**Step 4: Commit** + +```bash +git add infra/modules/object-storage/ infra/live/production/object-storage/ +git commit -m "infra: add Object Storage OpenTofu module" ``` -`infra/k8s/tailscale/rbac.yaml`: -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: tailscale - namespace: tailscale --- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: tailscale - namespace: tailscale -rules: - - apiGroups: [""] - resources: ["secrets"] - verbs: ["create", "get", "update", "patch"] + +### Task 4: Container Registry module + +**Files:** +- Create: `infra/modules/registry/main.tf` +- Create: `infra/modules/registry/variables.tf` +- Create: `infra/modules/registry/outputs.tf` +- Create: `infra/live/production/registry/terragrunt.hcl` + +**Step 1: Write the module** + +`infra/modules/registry/variables.tf`: +```hcl +variable "region" { + type = string +} + +variable "namespace_name" { + type = string + default = "foxhunt" +} +``` + +`infra/modules/registry/main.tf`: +```hcl +resource "scaleway_registry_namespace" "foxhunt" { + name = var.namespace_name + region = var.region + is_public = false +} +``` + +`infra/modules/registry/outputs.tf`: +```hcl +output "endpoint" { + value = scaleway_registry_namespace.foxhunt.endpoint +} + +output "namespace_id" { + value = scaleway_registry_namespace.foxhunt.id +} +``` + +**Step 2: Write Terragrunt config** + +`infra/live/production/registry/terragrunt.hcl`: +```hcl +include "root" { + path = find_in_parent_folders() +} + +terraform { + source = "../../../modules/registry" +} + +inputs = { + namespace_name = "foxhunt" +} +``` + +**Step 3: Apply** + +Run: `cd infra/live/production/registry && terragrunt apply` + +**Step 4: Commit** + +```bash +git add infra/modules/registry/ infra/live/production/registry/ +git commit -m "infra: add Container Registry OpenTofu module" +``` + --- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: tailscale - namespace: tailscale -subjects: - - kind: ServiceAccount - name: tailscale -roleRef: - kind: Role - name: tailscale - apiGroup: rbac.authorization.k8s.io + +### Task 5: Secret Manager module + +**Files:** +- Create: `infra/modules/secrets/main.tf` +- Create: `infra/modules/secrets/variables.tf` +- Create: `infra/modules/secrets/outputs.tf` +- Create: `infra/live/production/secrets/terragrunt.hcl` + +**Step 1: Write the module** + +`infra/modules/secrets/variables.tf`: +```hcl +variable "region" { + type = string +} + +variable "project_id" { + type = string +} ``` -**Step 2: Write the Tailscale deployment** +`infra/modules/secrets/main.tf`: +```hcl +resource "random_password" "jwt_secret" { + length = 64 + special = false +} -`infra/k8s/tailscale/deployment.yaml`: -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: tailscale-subnet-router - namespace: tailscale -spec: - replicas: 1 - selector: - matchLabels: - app: tailscale-subnet-router - template: - metadata: - labels: - app: tailscale-subnet-router - spec: - serviceAccountName: tailscale - nodeSelector: - # Pin to always-on pool - k8s.scaleway.com/pool-name: always-on - containers: - - name: tailscale - image: ghcr.io/tailscale/tailscale:latest - env: - - name: TS_AUTHKEY - valueFrom: - secretKeyRef: - name: tailscale-auth - key: TS_AUTHKEY - - name: TS_KUBE_SECRET - value: tailscale-state - - name: TS_USERSPACE - value: "false" - - name: TS_ROUTES - # Advertise Kapsule pod CIDR — get from: - # kubectl get nodes -o jsonpath='{.items[0].spec.podCIDR}' - value: "10.42.0.0/16" - - name: TS_HOSTNAME - value: "foxhunt-kapsule" - - name: TS_ACCEPT_DNS - value: "false" - securityContext: - capabilities: - add: ["NET_ADMIN", "NET_RAW"] - resources: - requests: - cpu: 50m - memory: 64Mi - limits: - cpu: 200m - memory: 128Mi +resource "random_password" "db_password" { + length = 32 + special = false +} + +resource "scaleway_secret" "jwt" { + name = "foxhunt-jwt-secret" + project_id = var.project_id + region = var.region +} + +resource "scaleway_secret_version" "jwt" { + secret_id = scaleway_secret.jwt.id + data = base64encode(random_password.jwt_secret.result) + region = var.region +} + +resource "scaleway_secret" "db_password" { + name = "foxhunt-db-password" + project_id = var.project_id + region = var.region +} + +resource "scaleway_secret_version" "db_password" { + secret_id = scaleway_secret.db_password.id + data = base64encode(random_password.db_password.result) + region = var.region +} ``` -**Step 3: Write the secret example file** +`infra/modules/secrets/outputs.tf`: +```hcl +output "jwt_secret_id" { + value = scaleway_secret.jwt.id +} -`infra/k8s/tailscale/secret.yaml.example`: -```yaml -# Copy to secret.yaml and fill in your Tailscale auth key -# Generate at: https://login.tailscale.com/admin/settings/keys -# Use a reusable, ephemeral key tagged with tag:k8s -apiVersion: v1 -kind: Secret -metadata: - name: tailscale-auth - namespace: tailscale -type: Opaque -stringData: - TS_AUTHKEY: "tskey-auth-REPLACE_ME" +output "db_password_secret_id" { + value = scaleway_secret.db_password.id +} + +# Sensitive outputs for K8s secret creation +output "jwt_secret_value" { + value = random_password.jwt_secret.result + sensitive = true +} + +output "db_password_value" { + value = random_password.db_password.result + sensitive = true +} ``` -**Step 4: Apply manifests** - -Run: -```bash -kubectl apply -f infra/k8s/tailscale/namespace.yaml -kubectl apply -f infra/k8s/tailscale/rbac.yaml -# Create real secret (not the example) -kubectl create secret generic tailscale-auth \ - --namespace=tailscale \ - --from-literal=TS_AUTHKEY="$(read -sp 'Tailscale auth key: ' key && echo $key)" -kubectl apply -f infra/k8s/tailscale/deployment.yaml -``` - -Expected: Pod running, Tailscale connected. - -**Step 5: Verify Tailscale connectivity** - -Run: `kubectl -n tailscale logs deployment/tailscale-subnet-router | grep -i "connected\|ready"` -Then from your laptop: `tailscale status | grep foxhunt-kapsule` -Expected: `foxhunt-kapsule` visible in tailnet, subnet routes advertised. - -**Step 6: Approve routes in Tailscale admin** - -Go to https://login.tailscale.com/admin/machines → find `foxhunt-kapsule` → approve subnet routes `10.42.0.0/16`. - -**Step 7: Commit** - -```bash -git add infra/k8s/tailscale/ -git commit -m "infra: add Tailscale subnet router for Kapsule" -``` - ---- - -### Task 3: Scaleway Object Storage + Container Registry - -**Files:** -- Create: `infra/scripts/setup-storage.sh` - -**Step 1: Write the storage setup script** - -```bash -#!/usr/bin/env bash -set -euo pipefail - -REGION="nl-ams" - -# Create Object Storage bucket for build cache and model artifacts -echo "Creating Object Storage bucket" -scw object bucket create name=foxhunt-artifacts region="${REGION}" - -# Create sub-prefixes (virtual directories) -# sccache/, models/, training-data/ — created automatically on first write - -# Create Container Registry namespace -echo "Creating Container Registry namespace" -scw registry namespace create \ - name=foxhunt \ - region="${REGION}" \ - is-public=false - -REGISTRY_ENDPOINT=$(scw registry namespace list name=foxhunt region="${REGION}" -o json | jq -r '.[0].endpoint') -echo "Registry endpoint: ${REGISTRY_ENDPOINT}" -echo "Login with: docker login ${REGISTRY_ENDPOINT}" - -# Create Kubernetes secret for registry pull -echo "Creating K8s registry pull secret" -kubectl create secret docker-registry scw-registry \ - --namespace=foxhunt \ - --docker-server="${REGISTRY_ENDPOINT}" \ - --docker-username=foxhunt \ - --docker-password="$(scw iam api-key list -o json | jq -r '.[0].secret_key')" - -echo "Done. Registry: ${REGISTRY_ENDPOINT}" -``` - -**Step 2: Run the script** - -Run: `chmod +x infra/scripts/setup-storage.sh && bash infra/scripts/setup-storage.sh` -Expected: Bucket created, registry namespace created, K8s pull secret created. - -**Step 3: Commit** - -```bash -git add infra/scripts/setup-storage.sh -git commit -m "infra: add Object Storage and Container Registry setup" -``` - ---- - -### Task 4: Scaleway Secret Manager integration - -**Files:** -- Create: `infra/scripts/setup-secrets.sh` -- Create: `infra/k8s/secrets/external-secrets.yaml` - -**Step 1: Write secret provisioning script** - -```bash -#!/usr/bin/env bash -set -euo pipefail - -REGION="nl-ams" -PROJECT_ID=$(scw account project list -o json | jq -r '.[0].id') - -# Create secrets in Scaleway Secret Manager -echo "Creating secrets in Scaleway Secret Manager" - -# JWT secret (generate random 64-char) -JWT_SECRET=$(openssl rand -base64 48) -scw secret secret create \ - name=foxhunt-jwt-secret \ - project-id="${PROJECT_ID}" \ - region="${REGION}" - -JWT_SECRET_ID=$(scw secret secret list name=foxhunt-jwt-secret region="${REGION}" -o json | jq -r '.[0].id') -echo -n "${JWT_SECRET}" | scw secret version create \ - secret-id="${JWT_SECRET_ID}" \ - data=- \ - region="${REGION}" - -# Database password -DB_PASSWORD=$(openssl rand -base64 32) -scw secret secret create \ - name=foxhunt-db-password \ - project-id="${PROJECT_ID}" \ - region="${REGION}" - -DB_SECRET_ID=$(scw secret secret list name=foxhunt-db-password region="${REGION}" -o json | jq -r '.[0].id') -echo -n "${DB_PASSWORD}" | scw secret version create \ - secret-id="${DB_SECRET_ID}" \ - data=- \ - region="${REGION}" - -echo "Secrets created. IDs:" -echo " JWT: ${JWT_SECRET_ID}" -echo " DB: ${DB_SECRET_ID}" -echo "" -echo "For K8s, create opaque secrets manually or use external-secrets-operator." -``` - -**Step 2: Write K8s secrets (pulled from Secret Manager at deploy time)** - -`infra/k8s/secrets/external-secrets.yaml`: -```yaml -# For now, secrets are created manually via setup-secrets.sh -# and injected as K8s secrets. When external-secrets-operator -# is needed, add Scaleway provider config here. -# -# Manual K8s secret creation (run after setup-secrets.sh): -# -# kubectl create secret generic foxhunt-secrets \ -# --namespace=foxhunt \ -# --from-literal=JWT_SECRET="" \ -# --from-literal=DATABASE_PASSWORD="" \ -# --from-literal=REDIS_PASSWORD="" -``` - -**Step 3: Run the script** - -Run: `chmod +x infra/scripts/setup-secrets.sh && bash infra/scripts/setup-secrets.sh` -Expected: Secrets created in Scaleway Secret Manager. - -**Step 4: Create K8s secret** +**Step 2: Write Terragrunt config** + +`infra/live/production/secrets/terragrunt.hcl`: +```hcl +include "root" { + path = find_in_parent_folders() +} + +terraform { + source = "../../../modules/secrets" +} +``` + +**Step 3: Apply** + +Run: `cd infra/live/production/secrets && terragrunt apply` + +**Step 4: Create K8s secret from TF outputs** Run: ```bash +cd infra/live/production/secrets kubectl create secret generic foxhunt-secrets \ --namespace=foxhunt \ - --from-literal=JWT_SECRET="$(scw secret version access-payload /1 region=nl-ams -o json | jq -r '.data' | base64 -d)" \ - --from-literal=DATABASE_PASSWORD="$(scw secret version access-payload /1 region=nl-ams -o json | jq -r '.data' | base64 -d)" + --from-literal=JWT_SECRET="$(terragrunt output -raw jwt_secret_value)" \ + --from-literal=DATABASE_PASSWORD="$(terragrunt output -raw db_password_value)" ``` **Step 5: Commit** ```bash -git add infra/scripts/setup-secrets.sh infra/k8s/secrets/ -git commit -m "infra: add Scaleway Secret Manager setup" +git add infra/modules/secrets/ infra/live/production/secrets/ +git commit -m "infra: add Secret Manager OpenTofu module" ``` --- -## Batch 2: CI/CD Pipeline +### Task 6: Block Storage module -### Task 5: Unified service Dockerfile with sccache +**Files:** +- Create: `infra/modules/block-storage/main.tf` +- Create: `infra/modules/block-storage/variables.tf` +- Create: `infra/modules/block-storage/outputs.tf` +- Create: `infra/live/production/block-storage/terragrunt.hcl` + +**Step 1: Write the module** + +`infra/modules/block-storage/variables.tf`: +```hcl +variable "zone" { + type = string +} + +variable "volume_name" { + type = string + default = "foxhunt-training-data" +} + +variable "size_in_gb" { + type = number + default = 100 +} +``` + +`infra/modules/block-storage/main.tf`: +```hcl +resource "scaleway_instance_volume" "training_data" { + name = var.volume_name + type = "b_ssd" + size_in_gb = var.size_in_gb + zone = var.zone +} +``` + +`infra/modules/block-storage/outputs.tf`: +```hcl +output "volume_id" { + value = scaleway_instance_volume.training_data.id +} + +output "volume_name" { + value = scaleway_instance_volume.training_data.name +} +``` + +**Step 2: Write Terragrunt config** + +`infra/live/production/block-storage/terragrunt.hcl`: +```hcl +include "root" { + path = find_in_parent_folders() +} + +terraform { + source = "../../../modules/block-storage" +} + +inputs = { + size_in_gb = 100 +} +``` + +**Step 3: Apply** + +Run: `cd infra/live/production/block-storage && terragrunt apply` + +**Step 4: Commit** + +```bash +git add infra/modules/block-storage/ infra/live/production/block-storage/ +git commit -m "infra: add Block Storage OpenTofu module for training data" +``` + +--- + +## Batch 2: Dockerfiles + CI + +### Task 7: Unified service Dockerfile with sccache **Files:** - Create: `infra/docker/Dockerfile.service` -The existing per-service Dockerfiles copy the entire workspace. This unified Dockerfile does the same but adds sccache for S3-backed build caching. - **Step 1: Write the unified Dockerfile** ```dockerfile @@ -534,12 +807,7 @@ USER foxhunt ENTRYPOINT ["./${SERVICE}"] ``` -**Step 2: Test build locally for one service** - -Run: `DOCKER_BUILDKIT=1 docker build --build-arg SERVICE=trading_service -f infra/docker/Dockerfile.service -t foxhunt-trading-service:test .` -Expected: Build succeeds, image created. - -**Step 3: Commit** +**Step 2: Commit** ```bash git add infra/docker/Dockerfile.service @@ -548,7 +816,7 @@ git commit -m "infra: add unified service Dockerfile with sccache support" --- -### Task 6: Web-gateway Dockerfile (serves dashboard static assets) +### Task 8: Web-gateway Dockerfile **Files:** - Create: `infra/docker/Dockerfile.web-gateway` @@ -569,7 +837,6 @@ COPY web-dashboard/package.json web-dashboard/package-lock.json ./ RUN npm ci COPY web-dashboard/ ./ RUN npm run build -# Output: /dashboard/dist/ # ============================================================================= # Stage 2: Build web-gateway Rust binary @@ -642,12 +909,7 @@ HEALTHCHECK --interval=10s --timeout=5s --start-period=10s --retries=3 \ ENTRYPOINT ["./web-gateway"] ``` -**Step 2: Test build locally** - -Run: `DOCKER_BUILDKIT=1 docker build -f infra/docker/Dockerfile.web-gateway -t foxhunt-web-gateway:test .` -Expected: Build succeeds (dashboard + Rust binary combined). - -**Step 3: Commit** +**Step 2: Commit** ```bash git add infra/docker/Dockerfile.web-gateway @@ -656,7 +918,114 @@ git commit -m "infra: add web-gateway Dockerfile with embedded dashboard" --- -### Task 7: Gitea Actions CI workflow +### Task 9: GPU training Dockerfile + +**Files:** +- Create: `infra/docker/Dockerfile.training` + +**Step 1: Write the training Dockerfile** + +```dockerfile +# Foxhunt ML Training — CUDA-enabled build for GPU training jobs +ARG RUST_VERSION=1.89 +ARG CUDA_VERSION=12.4.1 + +# ============================================================================= +# Builder +# ============================================================================= +FROM nvidia/cuda:${CUDA_VERSION}-cudnn-devel-ubuntu22.04 AS builder + +RUN apt-get update && apt-get install -y \ + curl git build-essential pkg-config libssl-dev ca-certificates protobuf-compiler \ + && rm -rf /var/lib/apt/lists/* + +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable +ENV PATH="/root/.cargo/bin:${PATH}" + +WORKDIR /build + +COPY Cargo.toml Cargo.lock ./ +COPY .sqlx ./.sqlx +COPY trading_engine ./trading_engine +COPY risk ./risk +COPY risk-data ./risk-data +COPY trading-data ./trading-data +COPY fxt ./fxt +COPY ml ./ml +COPY ml-data ./ml-data +COPY data ./data +COPY backtesting ./backtesting +COPY adaptive-strategy ./adaptive-strategy +COPY common ./common +COPY storage ./storage +COPY model_loader ./model_loader +COPY market-data ./market-data +COPY database ./database +COPY config ./config +COPY web-gateway ./web-gateway +COPY ctrader-openapi ./ctrader-openapi +COPY foxhunt-deploy ./foxhunt-deploy +COPY services ./services +COPY tests ./tests + +ENV SQLX_OFFLINE=true +# H100 = compute capability 9.0 +ENV CUDA_COMPUTE_CAP=90 + +RUN --mount=type=cache,target=/usr/local/cargo/registry \ + --mount=type=cache,target=/usr/local/cargo/git \ + --mount=type=cache,target=/build/target \ + cargo build -p ml --release --features ml/cuda \ + --example train_dqn \ + --example train_ppo_parquet \ + --example hyperopt_dqn_demo \ + --example hyperopt_ppo_demo \ + --example hyperopt_tft_demo \ + --example hyperopt_mamba2_demo && \ + mkdir -p /build/out && \ + cp /build/target/release/examples/train_dqn /build/out/ && \ + cp /build/target/release/examples/train_ppo_parquet /build/out/ && \ + cp /build/target/release/examples/hyperopt_dqn_demo /build/out/ && \ + cp /build/target/release/examples/hyperopt_ppo_demo /build/out/ && \ + cp /build/target/release/examples/hyperopt_tft_demo /build/out/ && \ + cp /build/target/release/examples/hyperopt_mamba2_demo /build/out/ + +RUN strip /build/out/* + +# ============================================================================= +# Runtime +# ============================================================================= +FROM nvidia/cuda:${CUDA_VERSION}-cudnn-runtime-ubuntu22.04 + +RUN apt-get update && apt-get install -y ca-certificates libssl3 curl && rm -rf /var/lib/apt/lists/* + +RUN useradd -m -u 1000 foxhunt + +WORKDIR /workspace + +COPY --from=builder /build/out/ /usr/local/bin/ +RUN chmod +x /usr/local/bin/train_* /usr/local/bin/hyperopt_* +RUN chown -R foxhunt:foxhunt /workspace + +USER foxhunt + +ENV CUDA_HOME=/usr/local/cuda +ENV PATH=${CUDA_HOME}/bin:${PATH} +ENV LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH} + +CMD ["echo", "Specify training command via K8s Job args"] +``` + +**Step 2: Commit** + +```bash +git add infra/docker/Dockerfile.training +git commit -m "infra: add GPU training Dockerfile (CUDA 12.4, H100 target)" +``` + +--- + +### Task 10: Gitea Actions CI workflow **Files:** - Create: `.gitea/workflows/ci.yaml` @@ -803,13 +1172,10 @@ jobs: - name: Configure kubeconfig run: echo "${{ secrets.KUBECONFIG }}" | base64 -d > /tmp/kubeconfig - env: - KUBECONFIG: /tmp/kubeconfig - name: Deploy to Kapsule run: | export KUBECONFIG=/tmp/kubeconfig - # Update image tags in all deployments for svc in trading-service api-gateway broker-gateway ml-training-service backtesting-service trading-agent-service web-gateway; do kubectl -n foxhunt set image deployment/${svc} \ ${svc}=rg.nl-ams.scw.cloud/foxhunt/${svc}:${{ github.sha }} \ @@ -818,12 +1184,7 @@ jobs: kubectl -n foxhunt rollout status deployment --timeout=120s ``` -**Step 2: Verify workflow syntax** - -Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/ci.yaml'))" && echo "YAML valid"` -Expected: "YAML valid" - -**Step 3: Commit** +**Step 2: Commit** ```bash git add .gitea/workflows/ci.yaml @@ -832,9 +1193,146 @@ git commit -m "ci: add Gitea Actions workflow (check, test, build, deploy)" --- -## Batch 3: Paper Trading Deployment +## Batch 3: K8s Manifests (In-Cluster Resources) -### Task 8: Database manifests (PostgreSQL, Redis, QuestDB) +### Task 11: Tailscale subnet router manifests + +**Files:** +- Create: `infra/k8s/tailscale/namespace.yaml` +- Create: `infra/k8s/tailscale/rbac.yaml` +- Create: `infra/k8s/tailscale/deployment.yaml` +- Create: `infra/k8s/tailscale/secret.yaml.example` + +**Step 1: Write all Tailscale manifests** + +`infra/k8s/tailscale/namespace.yaml`: +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: tailscale +``` + +`infra/k8s/tailscale/rbac.yaml`: +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: tailscale + namespace: tailscale +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: tailscale + namespace: tailscale +rules: + - apiGroups: [""] + resources: ["secrets"] + verbs: ["create", "get", "update", "patch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: tailscale + namespace: tailscale +subjects: + - kind: ServiceAccount + name: tailscale +roleRef: + kind: Role + name: tailscale + apiGroup: rbac.authorization.k8s.io +``` + +`infra/k8s/tailscale/deployment.yaml`: +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: tailscale-subnet-router + namespace: tailscale +spec: + replicas: 1 + selector: + matchLabels: + app: tailscale-subnet-router + template: + metadata: + labels: + app: tailscale-subnet-router + spec: + serviceAccountName: tailscale + nodeSelector: + k8s.scaleway.com/pool-name: always-on + containers: + - name: tailscale + image: ghcr.io/tailscale/tailscale:latest + env: + - name: TS_AUTHKEY + valueFrom: + secretKeyRef: + name: tailscale-auth + key: TS_AUTHKEY + - name: TS_KUBE_SECRET + value: tailscale-state + - name: TS_USERSPACE + value: "false" + - name: TS_ROUTES + value: "10.42.0.0/16" + - name: TS_HOSTNAME + value: "foxhunt-kapsule" + - name: TS_ACCEPT_DNS + value: "false" + securityContext: + capabilities: + add: ["NET_ADMIN", "NET_RAW"] + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 200m + memory: 128Mi +``` + +`infra/k8s/tailscale/secret.yaml.example`: +```yaml +# Copy to secret.yaml and fill in your Tailscale auth key +# Generate at: https://login.tailscale.com/admin/settings/keys +# Use a reusable, ephemeral key tagged with tag:k8s +apiVersion: v1 +kind: Secret +metadata: + name: tailscale-auth + namespace: tailscale +type: Opaque +stringData: + TS_AUTHKEY: "tskey-auth-REPLACE_ME" +``` + +**Step 2: Apply** + +Run: +```bash +kubectl apply -f infra/k8s/tailscale/namespace.yaml +kubectl apply -f infra/k8s/tailscale/rbac.yaml +kubectl create secret generic tailscale-auth --namespace=tailscale --from-literal=TS_AUTHKEY="" +kubectl apply -f infra/k8s/tailscale/deployment.yaml +``` + +**Step 3: Approve routes in Tailscale admin console** + +**Step 4: Commit** + +```bash +git add infra/k8s/tailscale/ +git commit -m "infra: add Tailscale subnet router K8s manifests" +``` + +--- + +### Task 12: Database manifests (PostgreSQL, Redis, QuestDB) **Files:** - Create: `infra/k8s/databases/postgres.yaml` @@ -889,7 +1387,7 @@ spec: name: foxhunt-secrets key: DATABASE_PASSWORD volumeMounts: - - name: postgres-data + - name: data mountPath: /var/lib/postgresql/data readinessProbe: exec: @@ -904,7 +1402,7 @@ spec: cpu: 500m memory: 1Gi volumes: - - name: postgres-data + - name: data persistentVolumeClaim: claimName: postgres-pvc --- @@ -918,7 +1416,6 @@ spec: app: postgres ports: - port: 5432 - targetPort: 5432 ``` **Step 2: Write Redis manifest** @@ -970,7 +1467,6 @@ spec: app: redis ports: - port: 6379 - targetPort: 6379 ``` **Step 3: Write QuestDB manifest** @@ -1009,14 +1505,14 @@ spec: - name: questdb image: questdb/questdb:8.2.3 ports: - - containerPort: 9009 # ILP - - containerPort: 8812 # PostgreSQL wire - - containerPort: 9003 # HTTP + - containerPort: 9009 + - containerPort: 8812 + - containerPort: 9003 env: - name: QDB_PG_ENABLED value: "true" volumeMounts: - - name: questdb-data + - name: data mountPath: /var/lib/questdb readinessProbe: httpGet: @@ -1031,7 +1527,7 @@ spec: cpu: 500m memory: 1Gi volumes: - - name: questdb-data + - name: data persistentVolumeClaim: claimName: questdb-pvc --- @@ -1046,28 +1542,13 @@ spec: ports: - name: ilp port: 9009 - targetPort: 9009 - name: pg port: 8812 - targetPort: 8812 - name: http port: 9003 - targetPort: 9003 ``` -**Step 4: Apply database manifests** - -Run: -```bash -kubectl apply -f infra/k8s/databases/postgres.yaml -kubectl apply -f infra/k8s/databases/redis.yaml -kubectl apply -f infra/k8s/databases/questdb.yaml -kubectl -n foxhunt get pods -w -``` - -Expected: All 3 pods running and ready. - -**Step 5: Commit** +**Step 4: Commit** ```bash git add infra/k8s/databases/ @@ -1076,7 +1557,7 @@ git commit -m "infra: add K8s manifests for PostgreSQL, Redis, QuestDB" --- -### Task 9: Service deployments +### Task 13: Service deployment manifests **Files:** - Create: `infra/k8s/services/trading-service.yaml` @@ -1087,200 +1568,23 @@ git commit -m "infra: add K8s manifests for PostgreSQL, Redis, QuestDB" - Create: `infra/k8s/services/trading-agent-service.yaml` - Create: `infra/k8s/services/web-gateway.yaml` -**Step 1: Write trading-service deployment** +**Step 1: Write all service manifests** -`infra/k8s/services/trading-service.yaml`: -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: trading-service - namespace: foxhunt -spec: - replicas: 1 - selector: - matchLabels: - app: trading-service - template: - metadata: - labels: - app: trading-service - spec: - nodeSelector: - k8s.scaleway.com/pool-name: always-on - imagePullSecrets: - - name: scw-registry - containers: - - name: trading-service - image: rg.nl-ams.scw.cloud/foxhunt/trading_service:latest - ports: - - containerPort: 50051 - name: grpc - - containerPort: 9092 - name: metrics - env: - - name: DATABASE_URL - value: postgresql://foxhunt:$(DATABASE_PASSWORD)@postgres:5432/foxhunt - - name: DATABASE_PASSWORD - valueFrom: - secretKeyRef: - name: foxhunt-secrets - key: DATABASE_PASSWORD - - name: REDIS_URL - value: redis://redis:6379 - - name: JWT_SECRET - valueFrom: - secretKeyRef: - name: foxhunt-secrets - key: JWT_SECRET - - name: JWT_ISSUER - value: foxhunt-api-gateway - - name: JWT_AUDIENCE - value: foxhunt-services - - name: QUESTDB_ILP_HOST - value: questdb:9009 - - name: GRPC_PORT - value: "50051" - - name: RUST_LOG - value: info - readinessProbe: - exec: - command: ["/usr/local/bin/grpc_health_probe", "-addr=localhost:50051"] - initialDelaySeconds: 10 - periodSeconds: 10 - resources: - requests: - cpu: 200m - memory: 256Mi - limits: - cpu: 500m - memory: 512Mi ---- -apiVersion: v1 -kind: Service -metadata: - name: trading-service - namespace: foxhunt -spec: - selector: - app: trading-service - ports: - - name: grpc - port: 50051 - targetPort: 50051 - - name: metrics - port: 9092 - targetPort: 9092 -``` - -**Step 2: Write remaining service deployments** - -Follow the same pattern as trading-service for each service. Key differences per service: +Each follows the same pattern (Deployment + Service). Key per-service differences: | Service | Image | gRPC Port | Metrics Port | Extra Env | |---------|-------|-----------|-------------|-----------| -| `api-gateway` | `api_gateway` | 50050 | 9091 | `TRADING_SERVICE_URL=http://trading-service:50051` etc. | +| `trading-service` | `trading_service` | 50051 | 9092 | QUESTDB_ILP_HOST | +| `api-gateway` | `api_gateway` | 50050 | 9091 | *_SERVICE_URL for all backends | | `broker-gateway` | `broker_gateway_service` | 50056 | 9096 | — | -| `ml-training-service` | `ml_training_service` | 50053 | 9094 | `S3_ENDPOINT`, `S3_BUCKET` | +| `ml-training-service` | `ml_training_service` | 50053 | 9094 | S3_ENDPOINT, S3_BUCKET | | `backtesting-service` | `backtesting_service` | 50053 | 9093 | — | | `trading-agent-service` | `trading_agent_service` | 50055 | 9095 | — | -| `web-gateway` | `web-gateway` | 3000 (HTTP) | — | `STATIC_DIR=/app/static` | +| `web-gateway` | `web-gateway` | 3000 (HTTP) | — | *_SERVICE_URL, static dir | -Write each as a separate YAML file following the trading-service pattern. For `web-gateway`, use HTTP health check instead of gRPC probe. +All pinned to `always-on` node pool via `nodeSelector`. All reference `foxhunt-secrets` for JWT_SECRET and DATABASE_PASSWORD. Web-gateway uses HTTP health probe; all others use grpc_health_probe. -`infra/k8s/services/web-gateway.yaml`: -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: web-gateway - namespace: foxhunt -spec: - replicas: 1 - selector: - matchLabels: - app: web-gateway - template: - metadata: - labels: - app: web-gateway - spec: - nodeSelector: - k8s.scaleway.com/pool-name: always-on - imagePullSecrets: - - name: scw-registry - containers: - - name: web-gateway - image: rg.nl-ams.scw.cloud/foxhunt/web-gateway:latest - ports: - - containerPort: 3000 - name: http - env: - - name: JWT_SECRET - valueFrom: - secretKeyRef: - name: foxhunt-secrets - key: JWT_SECRET - - name: TRADING_SERVICE_URL - value: http://trading-service:50051 - - name: BACKTESTING_SERVICE_URL - value: http://backtesting-service:50053 - - name: ML_TRAINING_SERVICE_URL - value: http://ml-training-service:50053 - - name: TRADING_AGENT_SERVICE_URL - value: http://trading-agent-service:50055 - - name: RUST_LOG - value: info - readinessProbe: - httpGet: - path: /api/health - port: 3000 - initialDelaySeconds: 5 - periodSeconds: 10 - resources: - requests: - cpu: 100m - memory: 128Mi - limits: - cpu: 250m - memory: 256Mi ---- -apiVersion: v1 -kind: Service -metadata: - name: web-gateway - namespace: foxhunt -spec: - selector: - app: web-gateway - ports: - - name: http - port: 3000 - targetPort: 3000 -``` - -**Step 3: Apply all service manifests** - -Run: -```bash -kubectl apply -f infra/k8s/services/ -kubectl -n foxhunt get pods -w -``` - -Expected: All 7 service pods running. - -**Step 4: Verify end-to-end through Tailscale** - -From your laptop (on Tailscale): -```bash -# Get web-gateway pod IP -WG_IP=$(kubectl -n foxhunt get svc web-gateway -o jsonpath='{.spec.clusterIP}') -curl http://${WG_IP}:3000/api/health -``` - -Expected: Health check returns OK. - -**Step 5: Commit** +**Step 2: Commit** ```bash git add infra/k8s/services/ @@ -1289,42 +1593,20 @@ git commit -m "infra: add K8s service deployments for paper trading stack" --- -## Batch 4: GPU Training Infrastructure - -### Task 10: Block Storage for training data +### Task 14: Training Job template + idle reaper **Files:** -- Create: `infra/scripts/setup-block-storage.sh` +- Create: `infra/k8s/training/job-template.yaml` +- Create: `infra/k8s/training/idle-reaper.yaml` - Create: `infra/k8s/storage/training-data-pv.yaml` -**Step 1: Write Block Storage provisioning script** +**Step 1: Write the Job template, PV manifest, and idle reaper CronJob** -```bash -#!/usr/bin/env bash -set -euo pipefail - -REGION="nl-ams" -ZONE="nl-ams-1" - -# Create 100GB Block Storage volume for training data -echo "Creating Block Storage volume for training data" -scw instance volume create \ - name=foxhunt-training-data \ - volume-type=b_ssd \ - size=100GB \ - zone="${ZONE}" - -VOL_ID=$(scw instance volume list name=foxhunt-training-data zone="${ZONE}" -o json | jq -r '.[0].id') -echo "Volume ID: ${VOL_ID}" -echo "Use this ID in the PersistentVolume manifest" -``` - -**Step 2: Write PV/PVC manifests** +The Job template mounts Block Storage PVC read-only at `/data`, requests 1 GPU, has 1hr timeout and 10min TTL cleanup. The idle reaper is a CronJob running every 5 min that checks for active training jobs. `infra/k8s/storage/training-data-pv.yaml`: ```yaml -# PersistentVolume backed by Scaleway Block Storage -# Replace VOLUME_ID with actual volume ID from setup-block-storage.sh +# Replace VOLUME_ID with output from: terragrunt output volume_id apiVersion: v1 kind: PersistentVolume metadata: @@ -1332,8 +1614,7 @@ metadata: spec: capacity: storage: 100Gi - accessModes: - - ReadOnlyMany + accessModes: [ReadOnlyMany] csi: driver: csi.scaleway.com volumeHandle: "VOLUME_ID" @@ -1345,385 +1626,34 @@ metadata: name: training-data-pvc namespace: foxhunt spec: - accessModes: - - ReadOnlyMany + accessModes: [ReadOnlyMany] resources: requests: storage: 100Gi volumeName: training-data-pv ``` -**Step 3: Upload training data to block storage** - -This is a manual step — attach the volume to the always-on node, mount it, upload data: -```bash -# From a node with the volume attached: -mount /dev/sda /mnt/training-data -rsync -avz data/cache/futures-baseline/ /mnt/training-data/futures-baseline/ -``` - -**Step 4: Commit** +**Step 2: Commit** ```bash -git add infra/scripts/setup-block-storage.sh infra/k8s/storage/ -git commit -m "infra: add Block Storage for shared training data" +git add infra/k8s/training/ infra/k8s/storage/ +git commit -m "infra: add GPU training Job template, PV, and idle reaper" ``` --- -### Task 11: GPU training Job template +## Batch 4: Operational Scripts -**Files:** -- Create: `infra/k8s/training/job-template.yaml` -- Create: `infra/docker/Dockerfile.training` - -**Step 1: Write GPU training Dockerfile** - -`infra/docker/Dockerfile.training`: -```dockerfile -# Foxhunt ML Training — CUDA-enabled build for GPU training jobs -# Based on existing Dockerfile.foxhunt-build pattern - -ARG RUST_VERSION=1.89 -ARG CUDA_VERSION=12.4.1 - -# ============================================================================= -# Builder -# ============================================================================= -FROM nvidia/cuda:${CUDA_VERSION}-cudnn-devel-ubuntu22.04 AS builder - -RUN apt-get update && apt-get install -y \ - curl git build-essential pkg-config libssl-dev ca-certificates protobuf-compiler \ - && rm -rf /var/lib/apt/lists/* - -RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable -ENV PATH="/root/.cargo/bin:${PATH}" - -WORKDIR /build - -COPY Cargo.toml Cargo.lock ./ -COPY .sqlx ./.sqlx -COPY trading_engine ./trading_engine -COPY risk ./risk -COPY risk-data ./risk-data -COPY trading-data ./trading-data -COPY fxt ./fxt -COPY ml ./ml -COPY ml-data ./ml-data -COPY data ./data -COPY backtesting ./backtesting -COPY adaptive-strategy ./adaptive-strategy -COPY common ./common -COPY storage ./storage -COPY model_loader ./model_loader -COPY market-data ./market-data -COPY database ./database -COPY config ./config -COPY web-gateway ./web-gateway -COPY ctrader-openapi ./ctrader-openapi -COPY foxhunt-deploy ./foxhunt-deploy -COPY services ./services -COPY tests ./tests - -ENV SQLX_OFFLINE=true -ENV CUDA_COMPUTE_CAP=90 - -# Build all training examples -RUN --mount=type=cache,target=/usr/local/cargo/registry \ - --mount=type=cache,target=/usr/local/cargo/git \ - --mount=type=cache,target=/build/target \ - cargo build -p ml --release --features ml/cuda \ - --example train_dqn \ - --example train_ppo_parquet \ - --example hyperopt_dqn_demo \ - --example hyperopt_ppo_demo \ - --example hyperopt_tft_demo \ - --example hyperopt_mamba2_demo && \ - mkdir -p /build/out && \ - cp /build/target/release/examples/train_dqn /build/out/ && \ - cp /build/target/release/examples/train_ppo_parquet /build/out/ && \ - cp /build/target/release/examples/hyperopt_dqn_demo /build/out/ && \ - cp /build/target/release/examples/hyperopt_ppo_demo /build/out/ && \ - cp /build/target/release/examples/hyperopt_tft_demo /build/out/ && \ - cp /build/target/release/examples/hyperopt_mamba2_demo /build/out/ - -RUN strip /build/out/* - -# ============================================================================= -# Runtime -# ============================================================================= -FROM nvidia/cuda:${CUDA_VERSION}-cudnn-runtime-ubuntu22.04 - -RUN apt-get update && apt-get install -y ca-certificates libssl3 curl && rm -rf /var/lib/apt/lists/* - -RUN useradd -m -u 1000 foxhunt - -WORKDIR /workspace - -COPY --from=builder /build/out/ /usr/local/bin/ - -RUN chmod +x /usr/local/bin/train_* /usr/local/bin/hyperopt_* -RUN chown -R foxhunt:foxhunt /workspace - -USER foxhunt - -ENV CUDA_HOME=/usr/local/cuda -ENV PATH=${CUDA_HOME}/bin:${PATH} -ENV LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH} - -CMD ["echo", "Specify training command via K8s Job args"] -``` - -**Step 2: Write K8s Job template** - -`infra/k8s/training/job-template.yaml`: -```yaml -# Template for GPU training jobs -# Usage: copy and customize command/args per training run -# -# The training orchestrator will generate Job manifests from this template. -apiVersion: batch/v1 -kind: Job -metadata: - # name: train-dqn-TIMESTAMP (generated) - namespace: foxhunt - labels: - foxhunt/job-type: training -spec: - backoffLimit: 1 - activeDeadlineSeconds: 3600 # 1 hour hard timeout - ttlSecondsAfterFinished: 600 # Cleanup after 10 min - template: - metadata: - labels: - foxhunt/job-type: training - spec: - restartPolicy: Never - nodeSelector: - k8s.scaleway.com/pool-name: gpu - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - imagePullSecrets: - - name: scw-registry - containers: - - name: training - image: rg.nl-ams.scw.cloud/foxhunt/training:latest - # Override per job: - command: ["train_dqn"] - args: - - "--symbol" - - "ES.FUT" - - "--data-dir" - - "/data/futures-baseline" - - "--max-steps-per-epoch" - - "2000" - env: - - name: RUST_LOG - value: info - - name: SQLX_OFFLINE - value: "true" - volumeMounts: - - name: training-data - mountPath: /data - readOnly: true - - name: output - mountPath: /workspace/output - resources: - requests: - nvidia.com/gpu: 1 - cpu: "4" - memory: 16Gi - limits: - nvidia.com/gpu: 1 - cpu: "8" - memory: 32Gi - volumes: - - name: training-data - persistentVolumeClaim: - claimName: training-data-pvc - - name: output - emptyDir: {} -``` - -**Step 3: Commit** - -```bash -git add infra/docker/Dockerfile.training infra/k8s/training/ -git commit -m "infra: add GPU training Dockerfile and K8s Job template" -``` - ---- - -### Task 12: Training orchestrator script +### Task 15: Training orchestrator script **Files:** - Create: `infra/scripts/train.sh` -**Step 1: Write the training orchestrator** +**Step 1: Write the script** -```bash -#!/usr/bin/env bash -set -euo pipefail +Bash script that accepts presets (quick-test, single-model, full-ensemble, hyperopt) and submits K8s Jobs to the GPU pool. Maps model names to training binaries. Generates unique job names with timestamps. -# Foxhunt Training Orchestrator -# Submits K8s training Jobs to GPU pool -# -# Usage: -# ./infra/scripts/train.sh quick-test --model dqn --symbol ES.FUT -# ./infra/scripts/train.sh single-model --model ppo --symbol ES.FUT -# ./infra/scripts/train.sh full-ensemble --symbol ES.FUT -# ./infra/scripts/train.sh hyperopt --model dqn --trials 20 --symbol ES.FUT - -PRESET="${1:?Usage: train.sh [--model MODEL] [--symbol SYMBOL] [--trials N]}" -shift - -# Defaults -MODEL="" -SYMBOL="ES.FUT" -TRIALS=20 -MAX_STEPS=2000 -TIMEOUT=3600 -DATA_DIR="/data/futures-baseline" - -while [[ $# -gt 0 ]]; do - case "$1" in - --model) MODEL="$2"; shift 2 ;; - --symbol) SYMBOL="$2"; shift 2 ;; - --trials) TRIALS="$2"; shift 2 ;; - --max-steps) MAX_STEPS="$2"; shift 2 ;; - --timeout) TIMEOUT="$2"; shift 2 ;; - *) echo "Unknown arg: $1"; exit 1 ;; - esac -done - -NAMESPACE="foxhunt" -IMAGE="rg.nl-ams.scw.cloud/foxhunt/training:latest" -TIMESTAMP=$(date +%s) - -# Map model names to training binaries -declare -A MODEL_BINARY=( - [dqn]="train_dqn" - [ppo]="train_ppo_parquet" - [tft]="hyperopt_tft_demo" - [mamba2]="hyperopt_mamba2_demo" -) - -ALL_MODELS=(dqn ppo tft mamba2) - -submit_job() { - local model="$1" - local binary="${MODEL_BINARY[$model]}" - local job_name="train-${model}-${TIMESTAMP}" - local extra_args="$2" - - echo "Submitting job: ${job_name} (${binary})" - - kubectl create -n "${NAMESPACE}" -f - </dev/null | wc -l) - echo "Active training jobs: ${ACTIVE}" - if [ "${ACTIVE}" -eq 0 ]; then - echo "No active training jobs. GPU pool will scale to 0 via autoscaler." - fi ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: gpu-reaper - namespace: foxhunt ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: gpu-reaper - namespace: foxhunt -rules: - - apiGroups: ["batch"] - resources: ["jobs"] - verbs: ["get", "list"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: gpu-reaper - namespace: foxhunt -subjects: - - kind: ServiceAccount - name: gpu-reaper -roleRef: - kind: Role - name: gpu-reaper - apiGroup: rbac.authorization.k8s.io -``` - -**Step 2: Apply** - -Run: `kubectl apply -f infra/k8s/training/idle-reaper.yaml` -Expected: CronJob created. - -**Step 3: Commit** - -```bash -git add infra/k8s/training/idle-reaper.yaml -git commit -m "infra: add GPU idle reaper CronJob for budget control" -``` - ---- - -## Batch 5: Verification - -### Task 14: End-to-end smoke test +### Task 16: End-to-end smoke test **Files:** - Create: `infra/scripts/smoke-test.sh` **Step 1: Write the smoke test** -```bash -#!/usr/bin/env bash -set -euo pipefail +Checks: cluster connectivity, Tailscale subnet router, all 3 databases, all 7 services, web-gateway health via Tailscale, node pool existence. -echo "=== Foxhunt Kapsule Smoke Test ===" - -NAMESPACE="foxhunt" -FAILURES=0 - -check() { - local name="$1" - local cmd="$2" - echo -n " ${name}... " - if eval "${cmd}" > /dev/null 2>&1; then - echo "OK" - else - echo "FAIL" - FAILURES=$((FAILURES + 1)) - fi -} - -echo "1. Cluster connectivity" -check "kubectl" "kubectl get nodes" -check "namespace" "kubectl get namespace ${NAMESPACE}" - -echo "2. Tailscale subnet router" -check "ts-pod" "kubectl -n tailscale get pods -l app=tailscale-subnet-router --field-selector=status.phase=Running --no-headers | grep -q ." -check "ts-mesh" "tailscale status | grep -q foxhunt-kapsule" - -echo "3. Databases" -check "postgres" "kubectl -n ${NAMESPACE} exec deployment/postgres -- pg_isready -U foxhunt" -check "redis" "kubectl -n ${NAMESPACE} exec deployment/redis -- redis-cli ping" -check "questdb" "kubectl -n ${NAMESPACE} exec deployment/questdb -- wget -qO- http://localhost:9003/exec?query=SELECT%201" - -echo "4. Services" -for svc in trading-service api-gateway broker-gateway ml-training-service backtesting-service trading-agent-service; do - check "${svc}" "kubectl -n ${NAMESPACE} get pods -l app=${svc} --field-selector=status.phase=Running --no-headers | grep -q ." -done -check "web-gateway" "kubectl -n ${NAMESPACE} get pods -l app=web-gateway --field-selector=status.phase=Running --no-headers | grep -q ." - -echo "5. Web gateway health (via Tailscale)" -WG_IP=$(kubectl -n ${NAMESPACE} get svc web-gateway -o jsonpath='{.spec.clusterIP}') -check "health-api" "curl -sf http://${WG_IP}:3000/api/health" - -echo "6. Node pools" -check "always-on" "kubectl get nodes -l k8s.scaleway.com/pool-name=always-on --no-headers | grep -q Ready" -check "ci-pool-exists" "scw k8s pool list cluster-id=\$(scw k8s cluster list name=foxhunt -o json | jq -r '.[0].id') -o json | jq -e '.[] | select(.name==\"ci\")'" -check "gpu-pool-exists" "scw k8s pool list cluster-id=\$(scw k8s cluster list name=foxhunt -o json | jq -r '.[0].id') -o json | jq -e '.[] | select(.name==\"gpu\")'" - -echo "" -if [ "${FAILURES}" -eq 0 ]; then - echo "ALL CHECKS PASSED" -else - echo "${FAILURES} CHECK(S) FAILED" - exit 1 -fi -``` - -**Step 2: Run the smoke test** - -Run: `chmod +x infra/scripts/smoke-test.sh && bash infra/scripts/smoke-test.sh` -Expected: All checks pass. - -**Step 3: Commit** +**Step 2: Commit** ```bash git add infra/scripts/smoke-test.sh @@ -1895,57 +1680,31 @@ git commit -m "infra: add end-to-end smoke test for Kapsule deployment" --- -## File Summary - -``` -infra/ -├── docker/ -│ ├── Dockerfile.service # Unified service builder (Task 5) -│ ├── Dockerfile.web-gateway # Web gateway + dashboard (Task 6) -│ └── Dockerfile.training # GPU training image (Task 11) -├── k8s/ -│ ├── tailscale/ -│ │ ├── namespace.yaml # (Task 2) -│ │ ├── rbac.yaml # (Task 2) -│ │ ├── deployment.yaml # (Task 2) -│ │ └── secret.yaml.example # (Task 2) -│ ├── databases/ -│ │ ├── postgres.yaml # (Task 8) -│ │ ├── redis.yaml # (Task 8) -│ │ └── questdb.yaml # (Task 8) -│ ├── services/ -│ │ ├── trading-service.yaml # (Task 9) -│ │ ├── api-gateway.yaml # (Task 9) -│ │ ├── broker-gateway.yaml # (Task 9) -│ │ ├── ml-training-service.yaml # (Task 9) -│ │ ├── backtesting-service.yaml # (Task 9) -│ │ ├── trading-agent-service.yaml # (Task 9) -│ │ └── web-gateway.yaml # (Task 9) -│ ├── secrets/ -│ │ └── external-secrets.yaml # (Task 4) -│ ├── storage/ -│ │ └── training-data-pv.yaml # (Task 10) -│ └── training/ -│ ├── job-template.yaml # (Task 11) -│ └── idle-reaper.yaml # (Task 13) -└── scripts/ - ├── create-cluster.sh # (Task 1) - ├── setup-storage.sh # (Task 3) - ├── setup-secrets.sh # (Task 4) - ├── setup-block-storage.sh # (Task 10) - ├── train.sh # (Task 12) - └── smoke-test.sh # (Task 14) - -.gitea/workflows/ -└── ci.yaml # (Task 7) -``` - ## Batch Execution Order -| Batch | Tasks | Dependency | -|-------|-------|-----------| -| 1: Cluster Foundation | 1, 2, 3, 4 | Sequential (cluster first, then networking, then storage) | -| 2: CI/CD Pipeline | 5, 6, 7 | Parallel (Dockerfiles independent, CI workflow depends on both) | -| 3: Paper Trading | 8, 9 | Sequential (databases before services) | -| 4: GPU Training | 10, 11, 12, 13 | Sequential (storage → image → orchestrator → reaper) | -| 5: Verification | 14 | After all batches | +| Batch | Tasks | What | Tool | +|-------|-------|------|------| +| 1: OpenTofu Foundation | 1–6 | Cluster, storage, registry, secrets, block storage | OpenTofu + Terragrunt | +| 2: Dockerfiles + CI | 7–10 | Service/web/training Dockerfiles, Gitea Actions | Docker, YAML | +| 3: K8s Manifests | 11–14 | Tailscale, databases, services, training jobs | kubectl | +| 4: Operational Scripts | 15–16 | Training orchestrator, smoke test | Bash | + +### Dependency Graph + +``` +Batch 1 (TF modules, sequential: state bucket → kapsule → storage/registry/secrets/block-storage) + ↓ +Batch 2 (Dockerfiles parallel, CI workflow depends on registry) + ↓ +Batch 3 (K8s manifests, sequential: Tailscale → DBs → services → training) + ↓ +Batch 4 (Operational scripts, parallel) +``` + +### Terragrunt apply-all + +After all modules are written, the entire infrastructure can be provisioned with: +```bash +cd infra/live/production +terragrunt run-all apply +```