diff --git a/docs/plans/2026-03-04-per-service-ci-implementation.md b/docs/plans/2026-03-04-per-service-ci-implementation.md new file mode 100644 index 000000000..9fa5b95b8 --- /dev/null +++ b/docs/plans/2026-03-04-per-service-ci-implementation.md @@ -0,0 +1,686 @@ +# Per-Service CI Compilation & Selective Deploy — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Replace monolithic compile-services with 8 per-service compile jobs and selective deploy, plus rename all service binaries to kebab-case. + +**Architecture:** Each service gets its own GitLab CI compile job with dependency-aware `changes:` filters. The deploy job only uploads and restarts services whose binaries were actually built. Service crate names change from snake_case to kebab-case so binary names match k8s deployment names directly. + +**Tech Stack:** GitLab CI YAML, Cargo workspace, Kubernetes manifests, shell scripts + +**Key insight:** Rust auto-maps hyphens to underscores for crate identifiers, so `use trading_service::...` still compiles after renaming the package to `trading-service`. No Rust source `use` statements need changing. + +--- + +### Task 1: Rename api_gateway crate to api-gateway + +**Files:** +- Modify: `services/api_gateway/Cargo.toml` (lines 2, 11) +- Modify: `infra/k8s/services/api-gateway.yaml` (lines 49, 50, 57, 88) + +**Step 1: Update Cargo.toml package and bin names** + +In `services/api_gateway/Cargo.toml`, change: +```toml +name = "api_gateway" +``` +to: +```toml +name = "api-gateway" +``` + +And the `[[bin]]` section: +```toml +name = "api_gateway" +``` +to: +```toml +name = "api-gateway" +``` + +**Step 2: Update K8s manifest S3 path and command** + +In `infra/k8s/services/api-gateway.yaml`, update the initContainer rclone line: +``` +":s3:foxhunt-binaries/services/api_gateway" → ":s3:foxhunt-binaries/services/api-gateway" +"/binaries/api_gateway" → "/binaries/api-gateway" +``` +And the container command: +```yaml +command: ["/binaries/api_gateway"] → command: ["/binaries/api-gateway"] +``` +And the chmod+echo lines in the initContainer script. + +**Step 3: Update cross-crate dependency in trading_service** + +In `services/trading_service/Cargo.toml`, the dev-dependency: +```toml +api_gateway = { path = "../api_gateway" } +``` +becomes: +```toml +api-gateway = { path = "../api_gateway" } +``` +(Path stays the same — only the package key changes. Rust still uses `use api_gateway::` because of hyphen-to-underscore mapping.) + +**Step 4: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p api-gateway` +Expected: compiles successfully + +**Step 5: Commit** + +```bash +git add services/api_gateway/Cargo.toml infra/k8s/services/api-gateway.yaml services/trading_service/Cargo.toml +git commit -m "refactor: rename api_gateway crate to api-gateway (kebab-case)" +``` + +--- + +### Task 2: Rename trading_service crate to trading-service + +**Files:** +- Modify: `services/trading_service/Cargo.toml` (lines 2, 11) +- Modify: `infra/k8s/services/trading-service.yaml` (initContainer + command) +- Modify: `testing/integration/Cargo.toml` (line 22: `trading_service` dep) + +**Step 1: Update Cargo.toml package and bin names** + +```toml +name = "trading_service" → name = "trading-service" +[[bin]] name = "trading_service" → [[bin]] name = "trading-service" +``` +Note: keep the second `[[bin]] name = "latency_validator"` unchanged (it's a separate binary). + +**Step 2: Update K8s manifest** + +In `infra/k8s/services/trading-service.yaml`: +``` +":s3:foxhunt-binaries/services/trading_service" → ":s3:foxhunt-binaries/services/trading-service" +"/binaries/trading_service" → "/binaries/trading-service" +command: ["/binaries/trading_service"] → command: ["/binaries/trading-service"] +``` + +**Step 3: Update testing crate dependency** + +In `testing/integration/Cargo.toml`: +```toml +trading_service = { path = "../../services/trading_service" } +``` +→ +```toml +trading-service = { path = "../../services/trading_service" } +``` + +**Step 4: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p trading-service` + +**Step 5: Commit** + +```bash +git commit -m "refactor: rename trading_service crate to trading-service (kebab-case)" +``` + +--- + +### Task 3: Rename broker_gateway_service crate to broker-gateway + +**Files:** +- Modify: `services/broker_gateway_service/Cargo.toml` (lines 2, 11) +- Modify: `infra/k8s/services/broker-gateway.yaml` (initContainer + command) + +**Step 1: Update Cargo.toml** + +```toml +name = "broker_gateway_service" → name = "broker-gateway" +[[bin]] name = "broker_gateway_service" → [[bin]] name = "broker-gateway" +``` + +**Step 2: Update K8s manifest** + +``` +":s3:foxhunt-binaries/services/broker_gateway_service" → ":s3:foxhunt-binaries/services/broker-gateway" +"/binaries/broker_gateway_service" → "/binaries/broker-gateway" +command: ["/binaries/broker_gateway_service"] → command: ["/binaries/broker-gateway"] +``` + +No cross-crate dependencies reference this crate. + +**Step 3: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p broker-gateway` + +**Step 4: Commit** + +```bash +git commit -m "refactor: rename broker_gateway_service crate to broker-gateway (kebab-case)" +``` + +--- + +### Task 4: Rename ml_training_service crate to ml-training-service + +**Files:** +- Modify: `services/ml_training_service/Cargo.toml` (lines 2, 107) +- Modify: `infra/k8s/services/ml-training-service.yaml` (initContainer + command) +- Modify: `testing/e2e/Cargo.toml` (line 68: `ml_training_service` dep) + +**Step 1: Update Cargo.toml** + +```toml +name = "ml_training_service" → name = "ml-training-service" +[[bin]] name = "ml_training_service" → [[bin]] name = "ml-training-service" +``` + +**Step 2: Update K8s manifest** + +``` +":s3:foxhunt-binaries/services/ml_training_service" → ":s3:foxhunt-binaries/services/ml-training-service" +"/binaries/ml_training_service" → "/binaries/ml-training-service" +command: ["/binaries/ml_training_service", "serve"] → command: ["/binaries/ml-training-service", "serve"] +``` + +**Step 3: Update testing crate dependency** + +In `testing/e2e/Cargo.toml`: +```toml +ml_training_service = { path = "../../services/ml_training_service" } +``` +→ +```toml +ml-training-service = { path = "../../services/ml_training_service" } +``` + +**Step 4: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p ml-training-service` + +**Step 5: Commit** + +```bash +git commit -m "refactor: rename ml_training_service crate to ml-training-service (kebab-case)" +``` + +--- + +### Task 5: Rename backtesting_service crate to backtesting-service + +**Files:** +- Modify: `services/backtesting_service/Cargo.toml` (lines 2, 11) +- Modify: `infra/k8s/services/backtesting-service.yaml` (initContainer + command) +- Modify: `testing/service-integration/Cargo.toml` (line 52: `backtesting_service` dep) + +**Step 1: Update Cargo.toml** + +```toml +name = "backtesting_service" → name = "backtesting-service" +[[bin]] name = "backtesting_service" → [[bin]] name = "backtesting-service" +``` + +**Step 2: Update K8s manifest** + +``` +":s3:foxhunt-binaries/services/backtesting_service" → ":s3:foxhunt-binaries/services/backtesting-service" +"/binaries/backtesting_service" → "/binaries/backtesting-service" +command: ["/binaries/backtesting_service"] → command: ["/binaries/backtesting-service"] +``` + +**Step 3: Update testing crate dependency** + +In `testing/service-integration/Cargo.toml`: +```toml +backtesting_service = { path = "../../services/backtesting_service" } +``` +→ +```toml +backtesting-service = { path = "../../services/backtesting_service" } +``` + +**Step 4: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p backtesting-service` + +**Step 5: Commit** + +```bash +git commit -m "refactor: rename backtesting_service crate to backtesting-service (kebab-case)" +``` + +--- + +### Task 6: Rename trading_agent_service crate to trading-agent-service + +**Files:** +- Modify: `services/trading_agent_service/Cargo.toml` (lines 2, 11) +- Modify: `infra/k8s/services/trading-agent-service.yaml` (initContainer + command) + +No cross-crate dependencies reference this crate. + +**Step 1: Update Cargo.toml** + +```toml +name = "trading_agent_service" → name = "trading-agent-service" +[[bin]] name = "trading_agent_service" → [[bin]] name = "trading-agent-service" +``` + +**Step 2: Update K8s manifest** + +``` +":s3:foxhunt-binaries/services/trading_agent_service" → ":s3:foxhunt-binaries/services/trading-agent-service" +"/binaries/trading_agent_service" → "/binaries/trading-agent-service" +command: ["/binaries/trading_agent_service"] → command: ["/binaries/trading-agent-service"] +``` + +**Step 3: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p trading-agent-service` + +**Step 4: Commit** + +```bash +git commit -m "refactor: rename trading_agent_service crate to trading-agent-service (kebab-case)" +``` + +--- + +### Task 7: Rename data_acquisition_service crate to data-acquisition-service + +**Files:** +- Modify: `services/data_acquisition_service/Cargo.toml` (lines 2, 71) +- Modify: `infra/k8s/services/data-acquisition-service.yaml` (initContainer + command) + +No cross-crate dependencies reference this crate. + +**Step 1: Update Cargo.toml** + +```toml +name = "data_acquisition_service" → name = "data-acquisition-service" +[[bin]] name = "data_acquisition_service" → [[bin]] name = "data-acquisition-service" +``` + +**Step 2: Update K8s manifest** + +``` +":s3:foxhunt-binaries/services/data_acquisition_service" → ":s3:foxhunt-binaries/services/data-acquisition-service" +"/binaries/data_acquisition_service" → "/binaries/data-acquisition-service" +command: ["/binaries/data_acquisition_service"] → command: ["/binaries/data-acquisition-service"] +``` + +**Step 3: Verify compilation** + +Run: `SQLX_OFFLINE=true cargo check -p data-acquisition-service` + +**Step 4: Commit** + +```bash +git commit -m "refactor: rename data_acquisition_service crate to data-acquisition-service (kebab-case)" +``` + +--- + +### Task 8: Update scripts for kebab-case binary names + +**Files:** +- Modify: `scripts/start.sh` (lines ~85-118: pkill and cargo run references) +- Modify: `scripts/stop.sh` (lines ~18-43: pid files and pkill references) +- Modify: `scripts/validate_data_quality.sh` (lines 109, 113: `-p backtesting_service`) +- Modify: `scripts/setup-database.sh` (line 112: `--bin trading_service`) +- Modify: `infra/scripts/build-and-push.sh` (lines 26-31: STANDARD_SERVICES array) + +**Step 1: Update start.sh** + +Replace all `trading_service` → `trading-service`, `backtesting_service` → `backtesting-service`, `ml_training_service` → `ml-training-service` in cargo run/build/pkill commands. + +**Step 2: Update stop.sh** + +Same pattern: update pid file names and pkill patterns. + +**Step 3: Update validate_data_quality.sh** + +```bash +cargo build --release -p backtesting_service → cargo build --release -p backtesting-service +``` + +**Step 4: Update setup-database.sh** + +```bash +cargo run --bin trading_service → cargo run --bin trading-service +``` + +**Step 5: Update build-and-push.sh** + +```bash +STANDARD_SERVICES=( + api-gateway + trading-service + trading-agent-service + ml-training-service + backtesting-service + broker-gateway +) +``` + +**Step 6: Commit** + +```bash +git commit -m "refactor: update scripts for kebab-case service binary names" +``` + +--- + +### Task 9: Replace compile-services with per-service compile jobs in .gitlab-ci.yml + +**Files:** +- Modify: `.gitlab-ci.yml` (replace compile-services job ~lines 411-459 with base template + 8 per-service jobs) + +**Step 1: Add .compile-service-base YAML anchor** + +Insert after the `.rust-base-cpu` template (after line 360): + +```yaml +# Base template for per-service compile jobs +.compile-service-base: + extends: .rust-base-cpu + stage: compile + needs: [] + interruptible: true + variables: + CARGO_BUILD_JOBS: "4" + script: + - mkdir -p "$SCCACHE_DIR" && sccache --zero-stats || true + - PROFILE=${DEV_RELEASE:+dev-release}; PROFILE=${PROFILE:-release} + - | + YEAR=$(date -d "$CI_COMMIT_TIMESTAMP" +%Y 2>/dev/null || date +%Y) + MONTH=$(date -d "$CI_COMMIT_TIMESTAMP" +%m 2>/dev/null || date +%m) + export FOXHUNT_BUILD_VERSION="${YEAR}.${MONTH}.${CI_PIPELINE_IID:-0}" + - echo "Building ${SERVICE_PACKAGE} with --profile $PROFILE" + - TARGET_DIR="target/$PROFILE" + - cargo build --profile $PROFILE -p ${SERVICE_PACKAGE} + - sccache --show-stats || true + - mkdir -p build-out/services + - cp $TARGET_DIR/${SERVICE_PACKAGE} build-out/services/ + - strip build-out/services/${SERVICE_PACKAGE} + - ls -lh build-out/services/ + artifacts: + paths: + - build-out/services/ + expire_in: 1 day +``` + +**Step 2: Delete the old compile-services job (lines 411-459)** + +**Step 3: Add 8 per-service compile jobs** + +```yaml +compile-api-gateway: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: api-gateway + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/api_gateway/** + - crates/common/** + - crates/config/** + - crates/trading_engine/** + - bin/fxt/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-trading-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: trading-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/trading_service/** + - services/api_gateway/** + - crates/common/** + - crates/config/** + - crates/data/** + - crates/database/** + - crates/ml/** + - crates/risk/** + - crates/storage/** + - crates/trading_engine/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-ml-training-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: ml-training-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/ml_training_service/** + - crates/common/** + - crates/config/** + - crates/data/** + - crates/ml/** + - crates/risk/** + - crates/storage/** + - crates/trading_engine/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-backtesting-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: backtesting-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/backtesting_service/** + - crates/common/** + - crates/config/** + - crates/data/** + - crates/ml/** + - crates/model_loader/** + - crates/risk/** + - crates/storage/** + - crates/trading_engine/** + - bin/fxt/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-trading-agent-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: trading-agent-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/trading_agent_service/** + - crates/common/** + - crates/config/** + - crates/ml/** + - crates/risk/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-broker-gateway: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: broker-gateway + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/broker_gateway_service/** + - crates/common/** + - crates/config/** + - crates/trading_engine/** + - vendor/ctrader-openapi/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-data-acquisition-service: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: data-acquisition-service + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - services/data_acquisition_service/** + - crates/common/** + - crates/config/** + - crates/storage/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" + +compile-web-gateway: + extends: .compile-service-base + variables: + SERVICE_PACKAGE: web-gateway + rules: + - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" + changes: + - crates/web-gateway/** + - crates/common/** + - Cargo.toml + - Cargo.lock + - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api" +``` + +**Step 4: Verify YAML syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))"` + +**Step 5: Commit** + +```bash +git commit -m "ci: replace compile-services with 8 per-service compile jobs" +``` + +--- + +### Task 10: Update deploy job for selective restart + +**Files:** +- Modify: `.gitlab-ci.yml` (deploy job ~lines 1462-1639) + +**Step 1: Update `needs:` to reference per-service jobs** + +Replace the single `compile-services` need with 8 optional per-service needs: + +```yaml +needs: + - job: compile-api-gateway + optional: true + artifacts: true + - job: compile-trading-service + optional: true + artifacts: true + - job: compile-ml-training-service + optional: true + artifacts: true + - job: compile-backtesting-service + optional: true + artifacts: true + - job: compile-trading-agent-service + optional: true + artifacts: true + - job: compile-broker-gateway + optional: true + artifacts: true + - job: compile-data-acquisition-service + optional: true + artifacts: true + - job: compile-web-gateway + optional: true + artifacts: true + - job: compile-training + optional: true + artifacts: true + - job: build-web-dashboard + optional: true + artifacts: true +``` + +**Step 2: Replace hardcoded rollout restart loop with selective restart** + +Replace the `for svc in trading-service api-gateway ...` loops with: + +```bash +# Restart only services whose binary was uploaded +DEPLOYED="" +for bin in build-out/services/*; do + [ -f "$bin" ] || continue + name=$(basename "$bin") + DEPLOYED="$DEPLOYED $name" +done + +if [ -n "$DEPLOYED" ]; then + for svc in $DEPLOYED; do + kubectl -n foxhunt rollout restart deployment/$svc + echo "Restarted $svc" + done + for svc in $DEPLOYED; do + kubectl -n foxhunt rollout status deployment/$svc --timeout=300s + done + echo "Deployed:$DEPLOYED" +else + echo "No service binaries changed — manifests-only deploy" +fi +``` + +**Step 3: Verify YAML syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml'))"` + +**Step 4: Commit** + +```bash +git commit -m "ci: make deploy job selective — only restart services with new binaries" +``` + +--- + +### Task 11: Full workspace compilation check + +**Step 1: Run full workspace check** + +Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | tail -20` +Expected: all 37+ crates compile successfully + +**Step 2: Run full workspace tests (lib only, fast)** + +Run: `SQLX_OFFLINE=true cargo test --workspace --lib 2>&1 | tail -5` +Expected: all tests pass + +**Step 3: Verify no warnings from the renames** + +Run: `SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep -i "warning.*rename\|error" | head -10` +Expected: no rename-related warnings or errors + +**Step 4: Commit any fixups if needed** + +--- + +### Task 12: Update GPU overlay manifests if they reference old binary names + +**Files:** +- Check: `infra/k8s/gpu-overlays/trading-service-gpu.yaml` +- Check: `infra/k8s/gpu-overlays/ml-training-service-gpu.yaml` + +**Step 1: Check for old binary name references** + +Search for `trading_service`, `ml_training_service` in GPU overlay YAMLs. +These are strategic merge patches — they may reference container names or commands. + +**Step 2: Update any found references to kebab-case** + +**Step 3: Commit if changes were needed** + +```bash +git commit -m "infra: update GPU overlay manifests for kebab-case binary names" +```