foxhunt/docs/plans/2026-03-04-argo-migration-design.md

Full GitLab CI → Argo Workflows Migration

Date: 2026-03-04 Status: Approved

Problem

The entire CI/CD pipeline (test, compile, deploy, training, IaC) runs in GitLab CI via a 1600+ line .gitlab-ci.yml. GitLab CE lacks advanced pipeline features (child pipelines, pipeline subscriptions). The training stage already runs in Argo Workflows. Moving everything to Argo gives a single orchestration layer with DAG workflows, parameterized templates, a proper UI (ci.fxhnt.ai), and event-driven triggers.

Design

1. Trigger Layer — Argo Events

Install Argo Events (Helm chart) in the foxhunt namespace. Configure a webhook EventSource that receives GitLab push events.

EventSource (gitlab-push):

• Type: webhook
• Port: 12000, endpoint: /push, method: POST
• Exposed as ClusterIP service gitlab-push-eventsource-svc in foxhunt namespace

Sensor (ci-pipeline-trigger):

• Dependency: gitlab-push EventSource
• Filter: body.ref == "refs/heads/main" (only main branch pushes)
• Trigger: Submit ci-pipeline Workflow with parameters:
  • commits: JSON array of commit objects (contains added, modified, removed arrays)
  • checkout-sha: the HEAD commit SHA
  • pipeline-iid: extracted from webhook or generated

GitLab webhook config: Settings → Webhooks → URL: http://gitlab-push-eventsource-svc.foxhunt:12000/push, trigger: Push events, branch filter: main, secret token for HMAC validation.

2. CI Pipeline WorkflowTemplate (Orchestrator)

A single ci-pipeline WorkflowTemplate orchestrates the full pipeline as a DAG:

detect-changes
  ├── test-workspace (if code changed)
  ├── compile-api-gateway (if api_gateway deps changed)
  ├── compile-trading-service (if trading_service deps changed)
  ├── compile-ml-training-service (if ml deps changed)
  ├── compile-backtesting-service (if backtesting deps changed)
  ├── compile-trading-agent-service (if trading_agent deps changed)
  ├── compile-broker-gateway (if broker deps changed)
  ├── compile-data-acquisition-service (if data_acquisition deps changed)
  ├── compile-web-gateway (if web-gateway deps changed)
  ├── compile-training (if ML/training crates changed)
  ├── build-web-dashboard (if web-dashboard/ changed)
  └── deploy (after all compile steps complete)

The detect-changes step:

• Receives the commits JSON from the webhook payload
• Extracts all changed file paths from added + modified + removed
• Checks each service's dependency paths (same map as GitLab changes: filters)
• Outputs a JSON object: {"services": ["api-gateway", "trading-service"], "training": true, "web_dashboard": false, "test": true, "infra": false}
• Downstream DAG tasks use when expressions to conditionally execute

Service dependency map (hardcoded in detect-changes script)

Service	Trigger paths
api-gateway	services/api_gateway/, crates/common/, crates/config/, crates/trading_engine/, bin/fxt/, Cargo.toml, Cargo.lock
trading-service	services/trading_service/, services/api_gateway/, crates/common/, crates/config/, crates/data/, crates/database/, crates/ml/, crates/risk/, crates/storage/, crates/trading_engine/, Cargo.toml, Cargo.lock
ml-training-service	services/ml_training_service/, crates/common/, crates/config/, crates/data/, crates/ml/, crates/risk/, crates/storage/, crates/trading_engine/, Cargo.toml, Cargo.lock
backtesting-service	services/backtesting_service/, crates/common/, crates/config/, crates/data/, crates/ml/, crates/model_loader/, crates/risk/, crates/storage/, crates/trading_engine/, bin/fxt/, Cargo.toml, Cargo.lock
trading-agent-service	services/trading_agent_service/, crates/common/, crates/config/, crates/ml/, crates/risk/, Cargo.toml, Cargo.lock
broker-gateway	services/broker_gateway_service/, crates/common/, crates/config/, crates/trading_engine/, vendor/ctrader-openapi/, Cargo.toml, Cargo.lock
data-acquisition-service	services/data_acquisition_service/, crates/common/, crates/config/, crates/storage/, Cargo.toml, Cargo.lock
web-gateway	crates/web-gateway/, crates/common/, Cargo.toml, Cargo.lock
training	crates/ml/, crates/trading_engine/, crates/common/, crates/risk/, crates/data/, crates/config/, crates/storage/, services/training_uploader/, Cargo.toml, Cargo.lock

3. Compile Service WorkflowTemplate

Single parameterized template, invoked once per service that needs recompilation:

name: compile-service
arguments:
  parameters:
    - name: service-package    # e.g. "api-gateway"
    - name: commit-sha

• Image: ci-builder-cpu:latest (Rust 1.89 + mold + sccache, no CUDA)
• Node: ci-compile-cpu pool (POP2-32C-128G)
• Resources: 4 CPU request / 6 CPU limit, 8Gi / 16Gi memory (same as current per-service jobs)
• Volumes: sccache PVC (/mnt/sccache), workspace volumeClaimTemplate for build output
• Script: cargo build --profile dev-release -p <service-package>, strip, copy to workspace
• Artifact output: Binary at /workspace/services/<service-package> → Argo artifact (S3/MinIO)

Git checkout: The container needs the source code. Options:

• Git init container: Clone repo at checkout-sha into workspace volume
• Use Argo's built-in git artifact source

A shared git-checkout step at the start of the DAG clones once into a workspace PVC. All compile steps mount the same PVC (ReadWriteOnce — all on same node, which is the case with 1 POP2-32C node).

4. Compile Training WorkflowTemplate

Same as current compile-training job but as an Argo template:

• Image: ci-builder:latest (CUDA 12.4 + Rust 1.89 — needs CUDA stubs)
• Node: ci-compile-cpu pool (CPU compile with CUDA stubs, no GPU needed)
• Variables: CUDA_COMPUTE_CAP=89
• Script: cargo build --profile dev-release -p ml --features ml/cuda --example train_baseline_rl ...
• Output: 7 training binaries as Argo artifacts

5. Build Web Dashboard WorkflowTemplate

• Image: node:22-slim
• Node: ci-compile-cpu pool
• Resources: 2 CPU / 2Gi memory
• Script: cd web-dashboard && npm ci && npm run build
• Output: web-dashboard/dist/ as Argo artifact

6. Test Workspace WorkflowTemplate

• Image: ci-builder:latest (needs CUDA stubs for full workspace test)
• Node: ci-compile-cpu pool
• Resources: 14 CPU / 24Gi memory (same as current test job)
• Services: Redis sidecar container
• Script: cargo test --workspace --lib

7. Deploy WorkflowTemplate

Collects artifacts from all compile steps, uploads to MinIO, applies manifests, does selective rollout restart.

Steps:

1. Upload binaries: For each compiled service binary artifact, aws s3 cp to s3://foxhunt-binaries/services/
2. Upload training binaries: aws s3 cp to s3://foxhunt-binaries/training/
3. Upload web assets: aws s3 cp to s3://foxhunt-binaries/static/
4. Apply migrations: Same kubectl exec logic as current deploy
5. Apply monitoring: kubectl apply -f infra/k8s/monitoring/
6. Apply dashboards: Same ConfigMap generation script
7. Apply service manifests: kubectl apply -f infra/k8s/services/
8. Selective rollout restart: Only restart deployments whose binary was uploaded
9. Wait for rollout: kubectl rollout status with 300s timeout

RBAC: Dedicated ServiceAccount argo-deploy with:

• deployments, statefulsets, configmaps, services — get, list, create, update, patch
• pods — get, list, exec (for migrations)
• Namespace: foxhunt

8. Image Build WorkflowTemplate

Handles CI builder image rebuilds when Dockerfiles change:

• Trigger: detect-changes identifies infra/docker/Dockerfile.* changes
• Image: gcr.io/kaniko-project/executor:debug
• Script: Kaniko build + push to Scaleway Container Registry
• Trivy scan: Follow-up step scans pushed image for CRITICAL/HIGH vulns

9. IaC WorkflowTemplate

Terragrunt plan/apply:

• Trigger: detect-changes identifies infra/ changes
• Image: infra-runner:latest
• Plan step: terragrunt run-all plan (always runs)
• Apply step: terragrunt run-all apply -auto-approve (only on main)

10. Volumes & Storage

Volume	Type	Mount	Used by
sccache	PVC (existing)	/mnt/sccache	compile-service, compile-training, test
training-data	PVC (existing)	/mnt/training-data	training workflows
workspace	volumeClaimTemplate (5Gi)	/workspace	per-workflow scratch space
source	PVC or emptyDir	/src	git checkout, shared by all compile steps

11. Migration Strategy

Deploy new first, test, then disable old:

1. Install Argo Events: helm install argo-events argo/argo-events -n foxhunt -f infra/k8s/argo/events-values.yaml
2. Deploy WorkflowTemplates: kubectl apply -f infra/k8s/argo/
3. Deploy EventSource + Sensor: kubectl apply -f infra/k8s/argo/events/
4. Configure GitLab webhook: Point to EventSource endpoint
5. Test: Push a small change to main, verify Argo workflow runs and deploys correctly
6. Disable GitLab CI: Set all jobs to when: never in .gitlab-ci.yml
7. Validation period: Run for 1-2 weeks with GitLab CI disabled but .gitlab-ci.yml still present
8. Delete: Remove .gitlab-ci.yml, uninstall GitLab Runner

Rollback: Revert .gitlab-ci.yml to re-enable GitLab CI. The webhook can coexist — Argo Events ignores pushes if sensor is deleted.

File Layout

infra/k8s/argo/
├── values.yaml                          # Argo Workflows Helm values (existing)
├── events-values.yaml                   # Argo Events Helm values (new)
├── kustomization.yaml                   # Updated to include new templates
├── argo-workflow-netpol.yaml            # Network policy (existing)
├── training-workflow-template.yaml      # Training DAG (existing, unchanged)
├── ci-pipeline-template.yaml            # Main CI orchestrator (new)
├── compile-service-template.yaml        # Per-service compile (new)
├── compile-training-template.yaml       # Training binary compile (new)
├── build-web-dashboard-template.yaml    # Vite build (new)
├── test-workspace-template.yaml         # cargo test (new)
├── deploy-template.yaml                 # Deploy to k8s (new)
├── build-ci-image-template.yaml         # Kaniko image builds (new)
├── infra-template.yaml                  # Terragrunt plan/apply (new)
├── deploy-rbac.yaml                     # ServiceAccount + RBAC for deploy (new)
└── events/
    ├── gitlab-push-eventsource.yaml     # Webhook EventSource (new)
    └── ci-pipeline-sensor.yaml          # Sensor + trigger (new)

What Gets Deleted

After validation:

• .gitlab-ci.yml (1600+ lines)
• GitLab Runner pods and configuration
• GitLab Runner Helm release
• CI-related GitLab project settings (variables stay as k8s secrets)

What Stays

• GitLab CE as git server (push, pull, webhooks)
• All existing Argo Workflows infrastructure
• MinIO for artifacts and binary storage
• Scaleway Container Registry for images
• All k8s service manifests, monitoring, IaC