# Per-Service CI Compilation & Selective Deploy

**Date**: 2026-03-04
**Status**: Approved

## Problem

The current `.gitlab-ci.yml` has a single `compile-services` job that builds all 8 service binaries, and a `deploy` job that restarts all 8 deployments regardless of what changed. A one-line change to `services/api_gateway/` triggers recompilation and redeployment of all services including trading-service and ml-training-service.

## Design

### 1. Rename Service Crates to Kebab-Case

All 7 service crates with underscored names get renamed to kebab-case. Binary output names then match k8s deployment names and S3 keys directly — no mapping needed.

| Current package name | New package name | K8s deployment |
|---|---|---|
| `api_gateway` | `api-gateway` | `api-gateway` |
| `trading_service` | `trading-service` | `trading-service` |
| `broker_gateway_service` | `broker-gateway` | `broker-gateway` |
| `ml_training_service` | `ml-training-service` | `ml-training-service` |
| `backtesting_service` | `backtesting-service` | `backtesting-service` |
| `trading_agent_service` | `trading-agent-service` | `trading-agent-service` |
| `data_acquisition_service` | `data-acquisition-service` | `data-acquisition-service` |
| `web-gateway` | (already kebab) | `web-gateway` |

Changes required per crate:
- `Cargo.toml`: `name = "api-gateway"` (and `[[bin]] name = "api-gateway"` if present)
- `infra/k8s/services/<name>.yaml`: update initContainer rclone S3 path
- Root `Cargo.toml` workspace members: update if directory names change
- Internal `extern crate` or `use` references (rare — Rust auto-maps hyphens to underscores)
- Cross-service Cargo.toml `[dependencies]` entries that reference by package name

Note: Directory names stay as-is (`services/api_gateway/`) to avoid mass file renames. Only the Cargo package name and binary output change.

### 2. Per-Service Compile Jobs

Replace `compile-services` with 8 individual jobs using a shared YAML anchor.

#### Base template

```yaml
.compile-service-base:
  extends: .rust-base-cpu
  stage: compile
  needs: []
  interruptible: true
  variables:
    CARGO_BUILD_JOBS: "4"
  script:
    - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true
    - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release}
    - |
      YEAR=$(date -d "$CI_COMMIT_TIMESTAMP" +%Y 2>/dev/null || date +%Y)
      MONTH=$(date -d "$CI_COMMIT_TIMESTAMP" +%m 2>/dev/null || date +%m)
      export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.${CI_PIPELINE_IID:-0}"
    - cargo build --profile $PROFILE -p ${SERVICE_PACKAGE}
    - sccache --show-stats || true
    - mkdir -p build-out/services
    - cp target/$PROFILE/${SERVICE_PACKAGE} build-out/services/
    - strip build-out/services/${SERVICE_PACKAGE}
  artifacts:
    paths:
      - build-out/services/
    expire_in: 1 day
```

#### Per-service jobs with dependency-aware `changes:` filters

```yaml
compile-api-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: api-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/api_gateway/**
        - crates/common/**
        - crates/config/**
        - crates/trading_engine/**
        - bin/fxt/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-trading-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: trading-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/trading_service/**
        - services/api_gateway/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/database/**
        - crates/ml/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-ml-training-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: ml-training-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/ml_training_service/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/ml/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-backtesting-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: backtesting-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/backtesting_service/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/ml/**
        - crates/model_loader/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - bin/fxt/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-trading-agent-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: trading-agent-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/trading_agent_service/**
        - crates/common/**
        - crates/config/**
        - crates/ml/**
        - crates/risk/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-broker-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: broker-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/broker_gateway_service/**
        - crates/common/**
        - crates/config/**
        - crates/trading_engine/**
        - vendor/ctrader-openapi/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-data-acquisition-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: data-acquisition-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/data_acquisition_service/**
        - crates/common/**
        - crates/config/**
        - crates/storage/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-web-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: web-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/web-gateway/**
        - crates/common/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
```

### 3. Smart Deploy Job

The deploy job collects artifacts from whichever compile jobs ran (all `optional: true`), uploads only the binaries that exist, and restarts only those deployments.

```yaml
deploy:
  needs:
    - job: compile-api-gateway
      optional: true
      artifacts: true
    - job: compile-trading-service
      optional: true
      artifacts: true
    - job: compile-ml-training-service
      optional: true
      artifacts: true
    - job: compile-backtesting-service
      optional: true
      artifacts: true
    - job: compile-trading-agent-service
      optional: true
      artifacts: true
    - job: compile-broker-gateway
      optional: true
      artifacts: true
    - job: compile-data-acquisition-service
      optional: true
      artifacts: true
    - job: compile-web-gateway
      optional: true
      artifacts: true
    - job: compile-training
      optional: true
      artifacts: true
    - job: build-web-dashboard
      optional: true
      artifacts: true
  script:
    # Upload only binaries that were compiled
    - |
      DEPLOYED=""
      for bin in build-out/services/*; do
        [ -f "$bin" ] || continue
        name=$(basename "$bin")
        $S3 cp "$bin" s3://foxhunt-binaries/services/$name
        echo "Uploaded $name"
        DEPLOYED="$DEPLOYED $name"
      done
    # Apply manifests (idempotent)
    - kubectl apply -f infra/k8s/services/
    # Restart only services whose binary was uploaded
    # Binary name = deployment name (kebab-case)
    - |
      for svc in $DEPLOYED; do
        kubectl -n foxhunt rollout restart deployment/$svc
        echo "Restarted $svc"
      done
    - |
      for svc in $DEPLOYED; do
        kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
      done
    - echo "Deployed: ${DEPLOYED:-nothing (manifests-only)}"
```

### 4. S3 Path Updates

Each `infra/k8s/services/<name>.yaml` initContainer updates its rclone source to use the kebab-case binary name:

```
rclone copyto :s3:foxhunt-binaries/services/api-gateway /binaries/api-gateway
```

### 5. Training Compile (Unchanged)

`compile-training` keeps its existing `changes:` filter and CUDA builder image. No changes needed.

## Trigger Matrix

| Changed path | Jobs triggered |
|---|---|
| `services/api_gateway/**` | compile-api-gateway, compile-trading-service (dep), deploy |
| `services/trading_service/**` | compile-trading-service, deploy |
| `crates/ml/**` | compile-trading-service, compile-ml-training-service, compile-backtesting-service, compile-trading-agent-service, compile-training, deploy |
| `crates/common/**` | ALL 8 compile jobs + compile-training, deploy |
| `docs/**` | nothing |
| `crates/ml/tests/**` | Same as `crates/ml/**` (GitLab glob can't exclude tests/) |
| `infra/k8s/**` | deploy only (manifests, no recompile) |

## CPU Budget

- POP2-32C-128G node: 32 vCPUs
- Per-service compile job: 4 CPUs requested (burstable)
- Max concurrent: 8 service jobs + 1 training job = 36 CPUs requested
- In practice: typically 2-4 service jobs fire. With sccache, each takes <60s for incremental builds
- Worst case (Cargo.toml change): all 8 fire, may queue on CPU. Acceptable — full rebuilds are rare
