foxhunt/docs/plans/2026-03-04-per-service-ci-implementation.md

Per-Service CI Compilation & Selective Deploy — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Replace monolithic compile-services with 8 per-service compile jobs and selective deploy, plus rename all service binaries to kebab-case.

Architecture: Each service gets its own GitLab CI compile job with dependency-aware changes: filters. The deploy job only uploads and restarts services whose binaries were actually built. Service crate names change from snake_case to kebab-case so binary names match k8s deployment names directly.

Tech Stack: GitLab CI YAML, Cargo workspace, Kubernetes manifests, shell scripts

Key insight: Rust auto-maps hyphens to underscores for crate identifiers, so use trading_service::... still compiles after renaming the package to trading-service. No Rust source use statements need changing.

---

Task 1: Rename api_gateway crate to api-gateway

Files:

• Modify: services/api_gateway/Cargo.toml (lines 2, 11)
• Modify: infra/k8s/services/api-gateway.yaml (lines 49, 50, 57, 88)

Step 1: Update Cargo.toml package and bin names

In services/api_gateway/Cargo.toml, change:

name = "api_gateway"

to:

name = "api-gateway"

And the [[bin]] section:

name = "api_gateway"

to:

name = "api-gateway"

Step 2: Update K8s manifest S3 path and command

In infra/k8s/services/api-gateway.yaml, update the initContainer rclone line:

":s3:foxhunt-binaries/services/api_gateway"  →  ":s3:foxhunt-binaries/services/api-gateway"
"/binaries/api_gateway"  →  "/binaries/api-gateway"

And the container command:

command: ["/binaries/api_gateway"]  →  command: ["/binaries/api-gateway"]

And the chmod+echo lines in the initContainer script.

Step 3: Update cross-crate dependency in trading_service

In services/trading_service/Cargo.toml, the dev-dependency:

api_gateway = { path = "../api_gateway" }

becomes:

api-gateway = { path = "../api_gateway" }

(Path stays the same — only the package key changes. Rust still uses use api_gateway:: because of hyphen-to-underscore mapping.)

Step 4: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p api-gateway Expected: compiles successfully

Step 5: Commit

git add services/api_gateway/Cargo.toml infra/k8s/services/api-gateway.yaml services/trading_service/Cargo.toml
git commit -m "refactor: rename api_gateway crate to api-gateway (kebab-case)"

---

Task 2: Rename trading_service crate to trading-service

Files:

• Modify: services/trading_service/Cargo.toml (lines 2, 11)
• Modify: infra/k8s/services/trading-service.yaml (initContainer + command)
• Modify: testing/integration/Cargo.toml (line 22: trading_service dep)

Step 1: Update Cargo.toml package and bin names

name = "trading_service"  →  name = "trading-service"
[[bin]] name = "trading_service"  →  [[bin]] name = "trading-service"

Note: keep the second [[bin]] name = "latency_validator" unchanged (it's a separate binary).

Step 2: Update K8s manifest

In infra/k8s/services/trading-service.yaml:

":s3:foxhunt-binaries/services/trading_service"  →  ":s3:foxhunt-binaries/services/trading-service"
"/binaries/trading_service"  →  "/binaries/trading-service"
command: ["/binaries/trading_service"]  →  command: ["/binaries/trading-service"]

Step 3: Update testing crate dependency

In testing/integration/Cargo.toml:

trading_service = { path = "../../services/trading_service" }

→

trading-service = { path = "../../services/trading_service" }

Step 4: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p trading-service

Step 5: Commit

git commit -m "refactor: rename trading_service crate to trading-service (kebab-case)"

---

Task 3: Rename broker_gateway_service crate to broker-gateway

Files:

• Modify: services/broker_gateway_service/Cargo.toml (lines 2, 11)
• Modify: infra/k8s/services/broker-gateway.yaml (initContainer + command)

Step 1: Update Cargo.toml

name = "broker_gateway_service"  →  name = "broker-gateway"
[[bin]] name = "broker_gateway_service"  →  [[bin]] name = "broker-gateway"

Step 2: Update K8s manifest

":s3:foxhunt-binaries/services/broker_gateway_service"  →  ":s3:foxhunt-binaries/services/broker-gateway"
"/binaries/broker_gateway_service"  →  "/binaries/broker-gateway"
command: ["/binaries/broker_gateway_service"]  →  command: ["/binaries/broker-gateway"]

No cross-crate dependencies reference this crate.

Step 3: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p broker-gateway

Step 4: Commit

git commit -m "refactor: rename broker_gateway_service crate to broker-gateway (kebab-case)"

---

Task 4: Rename ml_training_service crate to ml-training-service

Files:

• Modify: services/ml_training_service/Cargo.toml (lines 2, 107)
• Modify: infra/k8s/services/ml-training-service.yaml (initContainer + command)
• Modify: testing/e2e/Cargo.toml (line 68: ml_training_service dep)

Step 1: Update Cargo.toml

name = "ml_training_service"  →  name = "ml-training-service"
[[bin]] name = "ml_training_service"  →  [[bin]] name = "ml-training-service"

Step 2: Update K8s manifest

":s3:foxhunt-binaries/services/ml_training_service"  →  ":s3:foxhunt-binaries/services/ml-training-service"
"/binaries/ml_training_service"  →  "/binaries/ml-training-service"
command: ["/binaries/ml_training_service", "serve"]  →  command: ["/binaries/ml-training-service", "serve"]

Step 3: Update testing crate dependency

In testing/e2e/Cargo.toml:

ml_training_service = { path = "../../services/ml_training_service" }

→

ml-training-service = { path = "../../services/ml_training_service" }

Step 4: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p ml-training-service

Step 5: Commit

git commit -m "refactor: rename ml_training_service crate to ml-training-service (kebab-case)"

---

Task 5: Rename backtesting_service crate to backtesting-service

Files:

• Modify: services/backtesting_service/Cargo.toml (lines 2, 11)
• Modify: infra/k8s/services/backtesting-service.yaml (initContainer + command)
• Modify: testing/service-integration/Cargo.toml (line 52: backtesting_service dep)

Step 1: Update Cargo.toml

name = "backtesting_service"  →  name = "backtesting-service"
[[bin]] name = "backtesting_service"  →  [[bin]] name = "backtesting-service"

Step 2: Update K8s manifest

":s3:foxhunt-binaries/services/backtesting_service"  →  ":s3:foxhunt-binaries/services/backtesting-service"
"/binaries/backtesting_service"  →  "/binaries/backtesting-service"
command: ["/binaries/backtesting_service"]  →  command: ["/binaries/backtesting-service"]

Step 3: Update testing crate dependency

In testing/service-integration/Cargo.toml:

backtesting_service = { path = "../../services/backtesting_service" }

→

backtesting-service = { path = "../../services/backtesting_service" }

Step 4: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p backtesting-service

Step 5: Commit

git commit -m "refactor: rename backtesting_service crate to backtesting-service (kebab-case)"

---

Task 6: Rename trading_agent_service crate to trading-agent-service

Files:

• Modify: services/trading_agent_service/Cargo.toml (lines 2, 11)
• Modify: infra/k8s/services/trading-agent-service.yaml (initContainer + command)

No cross-crate dependencies reference this crate.

Step 1: Update Cargo.toml

name = "trading_agent_service"  →  name = "trading-agent-service"
[[bin]] name = "trading_agent_service"  →  [[bin]] name = "trading-agent-service"

Step 2: Update K8s manifest

":s3:foxhunt-binaries/services/trading_agent_service"  →  ":s3:foxhunt-binaries/services/trading-agent-service"
"/binaries/trading_agent_service"  →  "/binaries/trading-agent-service"
command: ["/binaries/trading_agent_service"]  →  command: ["/binaries/trading-agent-service"]

Step 3: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p trading-agent-service

Step 4: Commit

git commit -m "refactor: rename trading_agent_service crate to trading-agent-service (kebab-case)"

---

Task 7: Rename data_acquisition_service crate to data-acquisition-service

Files:

• Modify: services/data_acquisition_service/Cargo.toml (lines 2, 71)
• Modify: infra/k8s/services/data-acquisition-service.yaml (initContainer + command)

No cross-crate dependencies reference this crate.

Step 1: Update Cargo.toml

name = "data_acquisition_service"  →  name = "data-acquisition-service"
[[bin]] name = "data_acquisition_service"  →  [[bin]] name = "data-acquisition-service"

Step 2: Update K8s manifest

":s3:foxhunt-binaries/services/data_acquisition_service"  →  ":s3:foxhunt-binaries/services/data-acquisition-service"
"/binaries/data_acquisition_service"  →  "/binaries/data-acquisition-service"
command: ["/binaries/data_acquisition_service"]  →  command: ["/binaries/data-acquisition-service"]

Step 3: Verify compilation

Run: SQLX_OFFLINE=true cargo check -p data-acquisition-service

Step 4: Commit

git commit -m "refactor: rename data_acquisition_service crate to data-acquisition-service (kebab-case)"

---

Task 8: Update scripts for kebab-case binary names

Files:

• Modify: scripts/start.sh (lines ~85-118: pkill and cargo run references)
• Modify: scripts/stop.sh (lines ~18-43: pid files and pkill references)
• Modify: scripts/validate_data_quality.sh (lines 109, 113: -p backtesting_service)
• Modify: scripts/setup-database.sh (line 112: --bin trading_service)
• Modify: infra/scripts/build-and-push.sh (lines 26-31: STANDARD_SERVICES array)

Step 1: Update start.sh

Replace all trading_service → trading-service, backtesting_service → backtesting-service, ml_training_service → ml-training-service in cargo run/build/pkill commands.

Step 2: Update stop.sh

Same pattern: update pid file names and pkill patterns.

Step 3: Update validate_data_quality.sh

cargo build --release -p backtesting_service  →  cargo build --release -p backtesting-service

Step 4: Update setup-database.sh

cargo run --bin trading_service  →  cargo run --bin trading-service

Step 5: Update build-and-push.sh

STANDARD_SERVICES=(
  api-gateway
  trading-service
  trading-agent-service
  ml-training-service
  backtesting-service
  broker-gateway
)

Step 6: Commit

git commit -m "refactor: update scripts for kebab-case service binary names"

---

Task 9: Replace compile-services with per-service compile jobs in .gitlab-ci.yml

Files:

• Modify: .gitlab-ci.yml (replace compile-services job ~lines 411-459 with base template + 8 per-service jobs)

Step 1: Add .compile-service-base YAML anchor

Insert after the .rust-base-cpu template (after line 360):

# Base template for per-service compile jobs
.compile-service-base:
  extends: .rust-base-cpu
  stage: compile
  needs: []
  interruptible: true
  variables:
    CARGO_BUILD_JOBS: "4"
  script:
    - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true
    - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release}
    - |
      YEAR=$(date -d "$CI_COMMIT_TIMESTAMP" +%Y 2>/dev/null || date +%Y)
      MONTH=$(date -d "$CI_COMMIT_TIMESTAMP" +%m 2>/dev/null || date +%m)
      export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.${CI_PIPELINE_IID:-0}"
    - echo "Building ${SERVICE_PACKAGE} with --profile $PROFILE"
    - TARGET_DIR="target/$PROFILE"
    - cargo build --profile $PROFILE -p ${SERVICE_PACKAGE}
    - sccache --show-stats || true
    - mkdir -p build-out/services
    - cp $TARGET_DIR/${SERVICE_PACKAGE} build-out/services/
    - strip build-out/services/${SERVICE_PACKAGE}
    - ls -lh build-out/services/
  artifacts:
    paths:
      - build-out/services/
    expire_in: 1 day

Step 2: Delete the old compile-services job (lines 411-459)

Step 3: Add 8 per-service compile jobs

compile-api-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: api-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/api_gateway/**
        - crates/common/**
        - crates/config/**
        - crates/trading_engine/**
        - bin/fxt/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-trading-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: trading-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/trading_service/**
        - services/api_gateway/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/database/**
        - crates/ml/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-ml-training-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: ml-training-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/ml_training_service/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/ml/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-backtesting-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: backtesting-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/backtesting_service/**
        - crates/common/**
        - crates/config/**
        - crates/data/**
        - crates/ml/**
        - crates/model_loader/**
        - crates/risk/**
        - crates/storage/**
        - crates/trading_engine/**
        - bin/fxt/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-trading-agent-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: trading-agent-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/trading_agent_service/**
        - crates/common/**
        - crates/config/**
        - crates/ml/**
        - crates/risk/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-broker-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: broker-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/broker_gateway_service/**
        - crates/common/**
        - crates/config/**
        - crates/trading_engine/**
        - vendor/ctrader-openapi/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-data-acquisition-service:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: data-acquisition-service
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - services/data_acquisition_service/**
        - crates/common/**
        - crates/config/**
        - crates/storage/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

compile-web-gateway:
  extends: .compile-service-base
  variables:
    SERVICE_PACKAGE: web-gateway
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      changes:
        - crates/web-gateway/**
        - crates/common/**
        - Cargo.toml
        - Cargo.lock
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"

Step 4: Verify YAML syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))"

Step 5: Commit

git commit -m "ci: replace compile-services with 8 per-service compile jobs"

---

Task 10: Update deploy job for selective restart

Files:

• Modify: .gitlab-ci.yml (deploy job ~lines 1462-1639)

Step 1: Update needs: to reference per-service jobs

Replace the single compile-services need with 8 optional per-service needs:

needs:
  - job: compile-api-gateway
    optional: true
    artifacts: true
  - job: compile-trading-service
    optional: true
    artifacts: true
  - job: compile-ml-training-service
    optional: true
    artifacts: true
  - job: compile-backtesting-service
    optional: true
    artifacts: true
  - job: compile-trading-agent-service
    optional: true
    artifacts: true
  - job: compile-broker-gateway
    optional: true
    artifacts: true
  - job: compile-data-acquisition-service
    optional: true
    artifacts: true
  - job: compile-web-gateway
    optional: true
    artifacts: true
  - job: compile-training
    optional: true
    artifacts: true
  - job: build-web-dashboard
    optional: true
    artifacts: true

Step 2: Replace hardcoded rollout restart loop with selective restart

Replace the for svc in trading-service api-gateway ... loops with:

# Restart only services whose binary was uploaded
DEPLOYED=""
for bin in build-out/services/*; do
  [ -f "$bin" ] || continue
  name=$(basename "$bin")
  DEPLOYED="$DEPLOYED $name"
done

if [ -n "$DEPLOYED" ]; then
  for svc in $DEPLOYED; do
    kubectl -n foxhunt rollout restart deployment/$svc
    echo "Restarted $svc"
  done
  for svc in $DEPLOYED; do
    kubectl -n foxhunt rollout status deployment/$svc --timeout=300s
  done
  echo "Deployed:$DEPLOYED"
else
  echo "No service binaries changed — manifests-only deploy"
fi

Step 3: Verify YAML syntax

Run: python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))"

Step 4: Commit

git commit -m "ci: make deploy job selective — only restart services with new binaries"

---

Task 11: Full workspace compilation check

Step 1: Run full workspace check

Run: SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -20 Expected: all 37+ crates compile successfully

Step 2: Run full workspace tests (lib only, fast)

Run: SQLX_OFFLINE=true cargo test --workspace --lib 2>&1 | tail -5 Expected: all tests pass

Step 3: Verify no warnings from the renames

Run: SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep -i "warning.*rename\|error" | head -10 Expected: no rename-related warnings or errors

Step 4: Commit any fixups if needed

---

Task 12: Update GPU overlay manifests if they reference old binary names

Files:

• Check: infra/k8s/gpu-overlays/trading-service-gpu.yaml
• Check: infra/k8s/gpu-overlays/ml-training-service-gpu.yaml

Step 1: Check for old binary name references

Search for trading_service, ml_training_service in GPU overlay YAMLs. These are strategic merge patches — they may reference container names or commands.

Step 2: Update any found references to kebab-case

Step 3: Commit if changes were needed

git commit -m "infra: update GPU overlay manifests for kebab-case binary names"