diff --git a/docs/plans/2026-03-04-per-service-ci-design.md b/docs/plans/2026-03-04-per-service-ci-design.md new file mode 100644 index 000000000..d72536662 --- /dev/null +++ b/docs/plans/2026-03-04-per-service-ci-design.md @@ -0,0 +1,303 @@ +# Per-Service CI Compilation & Selective Deploy + +**Date**: 2026-03-04 +**Status**: Approved + +## Problem + +The current `.gitlab-ci.yml` has a single `compile-services` job that builds all 8 service binaries, and a `deploy` job that restarts all 8 deployments regardless of what changed. A one-line change to `services/api_gateway/` triggers recompilation and redeployment of all services including trading-service and ml-training-service. + +## Design + +### 1. Rename Service Crates to Kebab-Case + +All 7 service crates with underscored names get renamed to kebab-case. Binary output names then match k8s deployment names and S3 keys directly — no mapping needed. + +| Current package name | New package name | K8s deployment | +|---|---|---| +| `api_gateway` | `api-gateway` | `api-gateway` | +| `trading_service` | `trading-service` | `trading-service` | +| `broker_gateway_service` | `broker-gateway` | `broker-gateway` | +| `ml_training_service` | `ml-training-service` | `ml-training-service` | +| `backtesting_service` | `backtesting-service` | `backtesting-service` | +| `trading_agent_service` | `trading-agent-service` | `trading-agent-service` | +| `data_acquisition_service` | `data-acquisition-service` | `data-acquisition-service` | +| `web-gateway` | (already kebab) | `web-gateway` | + +Changes required per crate: +- `Cargo.toml`: `name = "api-gateway"` (and `[[bin]] name = "api-gateway"` if present) +- `infra/k8s/services/.yaml`: update initContainer rclone S3 path +- Root `Cargo.toml` workspace members: update if directory names change +- Internal `extern crate` or `use` references (rare — Rust auto-maps hyphens to underscores) +- Cross-service Cargo.toml `[dependencies]` entries that reference by package name + +Note: Directory names stay as-is (`services/api_gateway/`) to avoid mass file renames. Only the Cargo package name and binary output change. + +### 2. Per-Service Compile Jobs + +Replace `compile-services` with 8 individual jobs using a shared YAML anchor. + +#### Base template + +```yaml +.compile-service-base: + extends: .rust-base-cpu + stage: compile + needs: [] + interruptible: true + variables: + CARGO_BUILD_JOBS: "4" + script: + - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true + - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release} + - | + YEAR=$(date -d "$CI_COMMIT_TIMESTAMP" +%Y 2>/dev/null || date +%Y) + MONTH=$(date -d "$CI_COMMIT_TIMESTAMP" +%m 2>/dev/null || date +%m) + export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.${CI_PIPELINE_IID:-0}" + - cargo build --profile $PROFILE -p ${SERVICE_PACKAGE} + - sccache --show-stats || true + - mkdir -p build-out/services + - cp target/$PROFILE/${SERVICE_PACKAGE} build-out/services/ + - strip build-out/services/${SERVICE_PACKAGE} + artifacts: + paths: + - build-out/services/ + expire_in: 1 day +``` + +#### Per-service jobs with dependency-aware `changes:` filters + +```yaml +compile-api-gateway: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: api-gateway + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/api_gateway/** + - crates/common/** + - crates/config/** + - crates/trading_engine/** + - bin/fxt/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-trading-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: trading-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/trading_service/** + - services/api_gateway/** + - crates/common/** + - crates/config/** + - crates/data/** + - crates/database/** + - crates/ml/** + - crates/risk/** + - crates/storage/** + - crates/trading_engine/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-ml-training-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: ml-training-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/ml_training_service/** + - crates/common/** + - crates/config/** + - crates/data/** + - crates/ml/** + - crates/risk/** + - crates/storage/** + - crates/trading_engine/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-backtesting-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: backtesting-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/backtesting_service/** + - crates/common/** + - crates/config/** + - crates/data/** + - crates/ml/** + - crates/model_loader/** + - crates/risk/** + - crates/storage/** + - crates/trading_engine/** + - bin/fxt/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-trading-agent-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: trading-agent-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/trading_agent_service/** + - crates/common/** + - crates/config/** + - crates/ml/** + - crates/risk/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-broker-gateway: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: broker-gateway + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/broker_gateway_service/** + - crates/common/** + - crates/config/** + - crates/trading_engine/** + - vendor/ctrader-openapi/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-data-acquisition-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: data-acquisition-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/data_acquisition_service/** + - crates/common/** + - crates/config/** + - crates/storage/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-web-gateway: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: web-gateway + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/web-gateway/** + - crates/common/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" +``` + +### 3. Smart Deploy Job + +The deploy job collects artifacts from whichever compile jobs ran (all `optional: true`), uploads only the binaries that exist, and restarts only those deployments. + +```yaml +deploy: + needs: + - job: compile-api-gateway + optional: true + artifacts: true + - job: compile-trading-service + optional: true + artifacts: true + - job: compile-ml-training-service + optional: true + artifacts: true + - job: compile-backtesting-service + optional: true + artifacts: true + - job: compile-trading-agent-service + optional: true + artifacts: true + - job: compile-broker-gateway + optional: true + artifacts: true + - job: compile-data-acquisition-service + optional: true + artifacts: true + - job: compile-web-gateway + optional: true + artifacts: true + - job: compile-training + optional: true + artifacts: true + - job: build-web-dashboard + optional: true + artifacts: true + script: + # Upload only binaries that were compiled + - | + DEPLOYED="" + for bin in build-out/services/*; do + [ -f "$bin" ] || continue + name=$(basename "$bin") + $S3 cp "$bin" s3://foxhunt-binaries/services/$name + echo "Uploaded $name" + DEPLOYED="$DEPLOYED $name" + done + # Apply manifests (idempotent) + - kubectl apply -f infra/k8s/services/ + # Restart only services whose binary was uploaded + # Binary name = deployment name (kebab-case) + - | + for svc in $DEPLOYED; do + kubectl -n foxhunt rollout restart deployment/$svc + echo "Restarted $svc" + done + - | + for svc in $DEPLOYED; do + kubectl -n foxhunt rollout status deployment/$svc --timeout=300s + done + - echo "Deployed: ${DEPLOYED:-nothing (manifests-only)}" +``` + +### 4. S3 Path Updates + +Each `infra/k8s/services/.yaml` initContainer updates its rclone source to use the kebab-case binary name: + +``` +rclone copyto :s3:foxhunt-binaries/services/api-gateway /binaries/api-gateway +``` + +### 5. Training Compile (Unchanged) + +`compile-training` keeps its existing `changes:` filter and CUDA builder image. No changes needed. + +## Trigger Matrix + +| Changed path | Jobs triggered | +|---|---| +| `services/api_gateway/**` | compile-api-gateway, compile-trading-service (dep), deploy | +| `services/trading_service/**` | compile-trading-service, deploy | +| `crates/ml/**` | compile-trading-service, compile-ml-training-service, compile-backtesting-service, compile-trading-agent-service, compile-training, deploy | +| `crates/common/**` | ALL 8 compile jobs + compile-training, deploy | +| `docs/**` | nothing | +| `crates/ml/tests/**` | Same as `crates/ml/**` (GitLab glob can't exclude tests/) | +| `infra/k8s/**` | deploy only (manifests, no recompile) | + +## CPU Budget + +- POP2-32C-128G node: 32 vCPUs +- Per-service compile job: 4 CPUs requested (burstable) +- Max concurrent: 8 service jobs + 1 training job = 36 CPUs requested +- In practice: typically 2-4 service jobs fire. With sccache, each takes <60s for incremental builds +- Worst case (Cargo.toml change): all 8 fire, may queue on CPU. Acceptable — full rebuilds are rare