docs: add per-service CI compilation & selective deploy design

Replaces monolithic compile-services with 8 per-service compile jobs, each with dependency-aware changes: filters. Deploy job only restarts services whose binary actually changed. Also renames service crates from snake_case to kebab-case to match k8s deployment names. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 21:47:44 +01:00
parent 77fe520e08
commit 4e30f99bd5
1 changed files with 303 additions and 0 deletions
--- a/docs/plans/2026-03-04-per-service-ci-design.md
+++ b/docs/plans/2026-03-04-per-service-ci-design.md
@@ -0,0 +1,303 @@
+# Per-Service CI Compilation & Selective Deploy
+
+**Date**: 2026-03-04
+**Status**: Approved
+
+## Problem
+
+The current `.gitlab-ci.yml` has a single `compile-services` job that builds all 8 service binaries, and a `deploy` job that restarts all 8 deployments regardless of what changed. A one-line change to `services/api_gateway/` triggers recompilation and redeployment of all services including trading-service and ml-training-service.
+
+## Design
+
+### 1. Rename Service Crates to Kebab-Case
+
+All 7 service crates with underscored names get renamed to kebab-case. Binary output names then match k8s deployment names and S3 keys directly — no mapping needed.
+
+| Current package name | New package name | K8s deployment |
+|---|---|---|
+| `api_gateway` | `api-gateway` | `api-gateway` |
+| `trading_service` | `trading-service` | `trading-service` |
+| `broker_gateway_service` | `broker-gateway` | `broker-gateway` |
+| `ml_training_service` | `ml-training-service` | `ml-training-service` |
+| `backtesting_service` | `backtesting-service` | `backtesting-service` |
+| `trading_agent_service` | `trading-agent-service` | `trading-agent-service` |
+| `data_acquisition_service` | `data-acquisition-service` | `data-acquisition-service` |
+| `web-gateway` | (already kebab) | `web-gateway` |
+
+Changes required per crate:
+- `Cargo.toml`: `name = "api-gateway"` (and `[[bin]] name = "api-gateway"` if present)
+- `infra/k8s/services/<name>.yaml`: update initContainer rclone S3 path
+- Root `Cargo.toml` workspace members: update if directory names change
+- Internal `extern crate` or `use` references (rare — Rust auto-maps hyphens to underscores)
+- Cross-service Cargo.toml `[dependencies]` entries that reference by package name
+
+Note: Directory names stay as-is (`services/api_gateway/`) to avoid mass file renames. Only the Cargo package name and binary output change.
+
+### 2. Per-Service Compile Jobs
+
+Replace `compile-services` with 8 individual jobs using a shared YAML anchor.
+
+#### Base template
+
+```yaml
+.compile-service-base:
+  extends: .rust-base-cpu
+  stage: compile
+  needs: []
+  interruptible: true
+  variables:
+    CARGO_BUILD_JOBS: "4"
+  script:
+    - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true
+    - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release}
+    - |
+      YEAR=$(date -d "$CI_COMMIT_TIMESTAMP" +%Y 2>/dev/null || date +%Y)
+      MONTH=$(date -d "$CI_COMMIT_TIMESTAMP" +%m 2>/dev/null || date +%m)
+      export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.${CI_PIPELINE_IID:-0}"
+    - cargo build --profile $PROFILE -p ${SERVICE_PACKAGE}
+    - sccache --show-stats || true
+    - mkdir -p build-out/services
+    - cp target/$PROFILE/${SERVICE_PACKAGE} build-out/services/
+    - strip build-out/services/${SERVICE_PACKAGE}
+  artifacts:
+    paths:
+      - build-out/services/
+    expire_in: 1 day
+```
+
+#### Per-service jobs with dependency-aware `changes:` filters
+
+```yaml
+compile-api-gateway:
+  extends: .compile-service-base
+  variables:
+    SERVICE_PACKAGE: api-gateway
+  rules:
+    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
+      changes:
+        - services/api_gateway/**
+        - crates/common/**
+        - crates/config/**
+        - crates/trading_engine/**
+        - bin/fxt/**
+        - Cargo.toml
+        - Cargo.lock
+    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
+
+compile-trading-service:
+  extends: .compile-service-base
+  variables:
+    SERVICE_PACKAGE: trading-service
+  rules:
+    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
+      changes:
+        - services/trading_service/**
+        - services/api_gateway/**
+        - crates/common/**
+        - crates/config/**
+        - crates/data/**
+        - crates/database/**
+        - crates/ml/**
+        - crates/risk/**
+        - crates/storage/**
+        - crates/trading_engine/**
+        - Cargo.toml
+        - Cargo.lock
+    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
+
+compile-ml-training-service:
+  extends: .compile-service-base
+  variables:
+    SERVICE_PACKAGE: ml-training-service
+  rules:
+    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
+      changes:
+        - services/ml_training_service/**
+        - crates/common/**
+        - crates/config/**
+        - crates/data/**
+        - crates/ml/**
+        - crates/risk/**
+        - crates/storage/**
+        - crates/trading_engine/**
+        - Cargo.toml
+        - Cargo.lock
+    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
+
+compile-backtesting-service:
+  extends: .compile-service-base
+  variables:
+    SERVICE_PACKAGE: backtesting-service
+  rules:
+    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
+      changes:
+        - services/backtesting_service/**
+        - crates/common/**
+        - crates/config/**
+        - crates/data/**
+        - crates/ml/**
+        - crates/model_loader/**
+        - crates/risk/**
+        - crates/storage/**
+        - crates/trading_engine/**
+        - bin/fxt/**
+        - Cargo.toml
+        - Cargo.lock
+    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
+
+compile-trading-agent-service:
+  extends: .compile-service-base
+  variables:
+    SERVICE_PACKAGE: trading-agent-service
+  rules:
+    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
+      changes:
+        - services/trading_agent_service/**
+        - crates/common/**
+        - crates/config/**
+        - crates/ml/**
+        - crates/risk/**
+        - Cargo.toml
+        - Cargo.lock
+    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
+
+compile-broker-gateway:
+  extends: .compile-service-base
+  variables:
+    SERVICE_PACKAGE: broker-gateway
+  rules:
+    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
+      changes:
+        - services/broker_gateway_service/**
+        - crates/common/**
+        - crates/config/**
+        - crates/trading_engine/**
+        - vendor/ctrader-openapi/**
+        - Cargo.toml
+        - Cargo.lock
+    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
+
+compile-data-acquisition-service:
+  extends: .compile-service-base
+  variables:
+    SERVICE_PACKAGE: data-acquisition-service
+  rules:
+    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
+      changes:
+        - services/data_acquisition_service/**
+        - crates/common/**
+        - crates/config/**
+        - crates/storage/**
+        - Cargo.toml
+        - Cargo.lock
+    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
+
+compile-web-gateway:
+  extends: .compile-service-base
+  variables:
+    SERVICE_PACKAGE: web-gateway
+  rules:
+    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
+      changes:
+        - services/web-gateway/**
+        - crates/common/**
+        - Cargo.toml
+        - Cargo.lock
+    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
+```
+
+### 3. Smart Deploy Job
+
+The deploy job collects artifacts from whichever compile jobs ran (all `optional: true`), uploads only the binaries that exist, and restarts only those deployments.
+
+```yaml
+deploy:
+  needs:
+    - job: compile-api-gateway
+      optional: true
+      artifacts: true
+    - job: compile-trading-service
+      optional: true
+      artifacts: true
+    - job: compile-ml-training-service
+      optional: true
+      artifacts: true
+    - job: compile-backtesting-service
+      optional: true
+      artifacts: true
+    - job: compile-trading-agent-service
+      optional: true
+      artifacts: true
+    - job: compile-broker-gateway
+      optional: true
+      artifacts: true
+    - job: compile-data-acquisition-service
+      optional: true
+      artifacts: true
+    - job: compile-web-gateway
+      optional: true
+      artifacts: true
+    - job: compile-training
+      optional: true
+      artifacts: true
+    - job: build-web-dashboard
+      optional: true
+      artifacts: true
+  script:
+    # Upload only binaries that were compiled
+    - |
+      DEPLOYED=""
+      for bin in build-out/services/*; do
+        [ -f "$bin" ] || continue
+        name=$(basename "$bin")
+        $S3 cp "$bin" s3://foxhunt-binaries/services/$name
+        echo "Uploaded $name"
+        DEPLOYED="$DEPLOYED $name"
+      done
+    # Apply manifests (idempotent)
+    - kubectl apply -f infra/k8s/services/
+    # Restart only services whose binary was uploaded
+    # Binary name = deployment name (kebab-case)
+    - |
+      for svc in $DEPLOYED; do
+        kubectl -n foxhunt rollout restart deployment/$svc
+        echo "Restarted $svc"
+      done
+    - |
+      for svc in $DEPLOYED; do
+        kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
+      done
+    - echo "Deployed: ${DEPLOYED:-nothing (manifests-only)}"
+```
+
+### 4. S3 Path Updates
+
+Each `infra/k8s/services/<name>.yaml` initContainer updates its rclone source to use the kebab-case binary name:
+
+```
+rclone copyto :s3:foxhunt-binaries/services/api-gateway /binaries/api-gateway
+```
+
+### 5. Training Compile (Unchanged)
+
+`compile-training` keeps its existing `changes:` filter and CUDA builder image. No changes needed.
+
+## Trigger Matrix
+
+| Changed path | Jobs triggered |
+|---|---|
+| `services/api_gateway/**` | compile-api-gateway, compile-trading-service (dep), deploy |
+| `services/trading_service/**` | compile-trading-service, deploy |
+| `crates/ml/**` | compile-trading-service, compile-ml-training-service, compile-backtesting-service, compile-trading-agent-service, compile-training, deploy |
+| `crates/common/**` | ALL 8 compile jobs + compile-training, deploy |
+| `docs/**` | nothing |
+| `crates/ml/tests/**` | Same as `crates/ml/**` (GitLab glob can't exclude tests/) |
+| `infra/k8s/**` | deploy only (manifests, no recompile) |
+
+## CPU Budget
+
+- POP2-32C-128G node: 32 vCPUs
+- Per-service compile job: 4 CPUs requested (burstable)
+- Max concurrent: 8 service jobs + 1 training job = 36 CPUs requested
+- In practice: typically 2-4 service jobs fire. With sccache, each takes <60s for incremental builds
+- Worst case (Cargo.toml change): all 8 fire, may queue on CPU. Acceptable — full rebuilds are rare