foxhunt/docs/plans/2026-03-04-per-service-ci-design.md

Per-Service CI Compilation & Selective Deploy

Date: 2026-03-04 Status: Approved

Problem

The current .gitlab-ci.yml has a single compile-services job that builds all 8 service binaries, and a deploy job that restarts all 8 deployments regardless of what changed. A one-line change to services/api_gateway/ triggers recompilation and redeployment of all services including trading-service and ml-training-service.

Design

1. Rename Service Crates to Kebab-Case

All 7 service crates with underscored names get renamed to kebab-case. Binary output names then match k8s deployment names and S3 keys directly — no mapping needed.

Current package name	New package name	K8s deployment
api_gateway	api-gateway	api-gateway
trading_service	trading-service	trading-service
broker_gateway_service	broker-gateway	broker-gateway
ml_training_service	ml-training-service	ml-training-service
backtesting_service	backtesting-service	backtesting-service
trading_agent_service	trading-agent-service	trading-agent-service
data_acquisition_service	data-acquisition-service	data-acquisition-service
web-gateway	(already kebab)	web-gateway

Changes required per crate:

• Cargo.toml: name = "api-gateway" (and [[bin]] name = "api-gateway" if present)
• infra/k8s/services/<name>.yaml: update initContainer rclone S3 path
• Root Cargo.toml workspace members: update if directory names change
• Internal extern crate or use references (rare — Rust auto-maps hyphens to underscores)
• Cross-service Cargo.toml [dependencies] entries that reference by package name

Note: Directory names stay as-is (services/api_gateway/) to avoid mass file renames. Only the Cargo package name and binary output change.

2. Per-Service Compile Jobs

Replace compile-services with 8 individual jobs using a shared YAML anchor.

Base template

.compile-service-base:
  extends: .rust-base-cpu
  stage: compile
  needs: []
  interruptible: true
  variables:
    CARGO_BUILD_JOBS: "4"
  script:
    - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true
    - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release}
    - |
      YEAR=$(date -d "$CI_COMMIT_TIMESTAMP" +%Y 2>/dev/null || date +%Y)
      MONTH=$(date -d "$CI_COMMIT_TIMESTAMP" +%m 2>/dev/null || date +%m)
      export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.${CI_PIPELINE_IID:-0}"
    - cargo build --profile $PROFILE -p ${SERVICE_PACKAGE}
    - sccache --show-stats || true
    - mkdir -p build-out/services
    - cp target/$PROFILE/${SERVICE_PACKAGE} build-out/services/
    - strip build-out/services/${SERVICE_PACKAGE}
  artifacts:
    paths:
      - build-out/services/
    expire_in: 1 day

Per-service jobs with dependency-aware changes: filters

compile-api-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: api-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/api_gateway/**
        - crates/common/**
        - crates/config/**
        - crates/trading_engine/**
        - bin/fxt/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-trading-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: trading-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/trading_service/**
        - services/api_gateway/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/database/**
        - crates/ml/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-ml-training-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: ml-training-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/ml_training_service/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/ml/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-backtesting-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: backtesting-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/backtesting_service/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/ml/**
        - crates/model_loader/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - bin/fxt/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-trading-agent-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: trading-agent-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/trading_agent_service/**
        - crates/common/**
        - crates/config/**
        - crates/ml/**
        - crates/risk/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-broker-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: broker-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/broker_gateway_service/**
        - crates/common/**
        - crates/config/**
        - crates/trading_engine/**
        - vendor/ctrader-openapi/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-data-acquisition-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: data-acquisition-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/data_acquisition_service/**
        - crates/common/**
        - crates/config/**
        - crates/storage/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-web-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: web-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/web-gateway/**
        - crates/common/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

3. Smart Deploy Job

The deploy job collects artifacts from whichever compile jobs ran (all optional: true), uploads only the binaries that exist, and restarts only those deployments.

deploy:
  needs:
    - job: compile-api-gateway
      optional: true
      artifacts: true
    - job: compile-trading-service
      optional: true
      artifacts: true
    - job: compile-ml-training-service
      optional: true
      artifacts: true
    - job: compile-backtesting-service
      optional: true
      artifacts: true
    - job: compile-trading-agent-service
      optional: true
      artifacts: true
    - job: compile-broker-gateway
      optional: true
      artifacts: true
    - job: compile-data-acquisition-service
      optional: true
      artifacts: true
    - job: compile-web-gateway
      optional: true
      artifacts: true
    - job: compile-training
      optional: true
      artifacts: true
    - job: build-web-dashboard
      optional: true
      artifacts: true
  script:
    # Upload only binaries that were compiled
    - |
      DEPLOYED=""
      for bin in build-out/services/*; do
        [ -f "$bin" ] || continue
        name=$(basename "$bin")
        $S3 cp "$bin" s3://foxhunt-binaries/services/$name
        echo "Uploaded $name"
        DEPLOYED="$DEPLOYED $name"
      done
    # Apply manifests (idempotent)
    - kubectl apply -f infra/k8s/services/
    # Restart only services whose binary was uploaded
    # Binary name = deployment name (kebab-case)
    - |
      for svc in $DEPLOYED; do
        kubectl -n foxhunt rollout restart deployment/$svc
        echo "Restarted $svc"
      done
    - |
      for svc in $DEPLOYED; do
        kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
      done
    - echo "Deployed: ${DEPLOYED:-nothing (manifests-only)}"

4. S3 Path Updates

Each infra/k8s/services/<name>.yaml initContainer updates its rclone source to use the kebab-case binary name:

rclone copyto :s3:foxhunt-binaries/services/api-gateway /binaries/api-gateway

5. Training Compile (Unchanged)

compile-training keeps its existing changes: filter and CUDA builder image. No changes needed.

Trigger Matrix

Changed path	Jobs triggered
services/api_gateway/**	compile-api-gateway, compile-trading-service (dep), deploy
services/trading_service/**	compile-trading-service, deploy
crates/ml/**	compile-trading-service, compile-ml-training-service, compile-backtesting-service, compile-trading-agent-service, compile-training, deploy
crates/common/**	ALL 8 compile jobs + compile-training, deploy
docs/**	nothing
crates/ml/tests/**	Same as crates/ml/** (GitLab glob can't exclude tests/)
infra/k8s/**	deploy only (manifests, no recompile)

CPU Budget

• POP2-32C-128G node: 32 vCPUs
• Per-service compile job: 4 CPUs requested (burstable)
• Max concurrent: 8 service jobs + 1 training job = 36 CPUs requested
• In practice: typically 2-4 service jobs fire. With sccache, each takes <60s for incremental builds
• Worst case (Cargo.toml change): all 8 fire, may queue on CPU. Acceptable — full rebuilds are rare