foxhunt/docs/plans/2026-03-04-argo-migration-design.md

# Full GitLab CI → Argo Workflows Migration

**Date**: 2026-03-04
**Status**: Approved

## Problem

The entire CI/CD pipeline (test, compile, deploy, training, IaC) runs in GitLab CI via a 1600+ line `.gitlab-ci.yml`. GitLab CE lacks advanced pipeline features (child pipelines, pipeline subscriptions). The training stage already runs in Argo Workflows. Moving everything to Argo gives a single orchestration layer with DAG workflows, parameterized templates, a proper UI (ci.fxhnt.ai), and event-driven triggers.

## Design

### 1. Trigger Layer — Argo Events

Install Argo Events (Helm chart) in the `foxhunt` namespace. Configure a webhook EventSource that receives GitLab push events.

**EventSource** (`gitlab-push`):
- Type: `webhook`
- Port: 12000, endpoint: `/push`, method: POST
- Exposed as ClusterIP service `gitlab-push-eventsource-svc` in `foxhunt` namespace

**Sensor** (`ci-pipeline-trigger`):
- Dependency: `gitlab-push` EventSource
- Filter: `body.ref == "refs/heads/main"` (only main branch pushes)
- Trigger: Submit `ci-pipeline` Workflow with parameters:
  - `commits`: JSON array of commit objects (contains `added`, `modified`, `removed` arrays)
  - `checkout-sha`: the HEAD commit SHA
  - `pipeline-iid`: extracted from webhook or generated

**GitLab webhook config**: Settings → Webhooks → URL: `http://gitlab-push-eventsource-svc.foxhunt:12000/push`, trigger: Push events, branch filter: `main`, secret token for HMAC validation.

### 2. CI Pipeline WorkflowTemplate (Orchestrator)

A single `ci-pipeline` WorkflowTemplate orchestrates the full pipeline as a DAG:

```
detect-changes
  ├── test-workspace (if code changed)
  ├── compile-api-gateway (if api_gateway deps changed)
  ├── compile-trading-service (if trading_service deps changed)
  ├── compile-ml-training-service (if ml deps changed)
  ├── compile-backtesting-service (if backtesting deps changed)
  ├── compile-trading-agent-service (if trading_agent deps changed)
  ├── compile-broker-gateway (if broker deps changed)
  ├── compile-data-acquisition-service (if data_acquisition deps changed)
  ├── compile-web-gateway (if web-gateway deps changed)
  ├── compile-training (if ML/training crates changed)
  ├── build-web-dashboard (if web-dashboard/ changed)
  └── deploy (after all compile steps complete)
```

The `detect-changes` step:
- Receives the commits JSON from the webhook payload
- Extracts all changed file paths from `added + modified + removed`
- Checks each service's dependency paths (same map as GitLab `changes:` filters)
- Outputs a JSON object: `{"services": ["api-gateway", "trading-service"], "training": true, "web_dashboard": false, "test": true, "infra": false}`
- Downstream DAG tasks use `when` expressions to conditionally execute

#### Service dependency map (hardcoded in detect-changes script)

| Service | Trigger paths |
|---|---|
| api-gateway | `services/api_gateway/`, `crates/common/`, `crates/config/`, `crates/trading_engine/`, `bin/fxt/`, `Cargo.toml`, `Cargo.lock` |
| trading-service | `services/trading_service/`, `services/api_gateway/`, `crates/common/`, `crates/config/`, `crates/data/`, `crates/database/`, `crates/ml/`, `crates/risk/`, `crates/storage/`, `crates/trading_engine/`, `Cargo.toml`, `Cargo.lock` |
| ml-training-service | `services/ml_training_service/`, `crates/common/`, `crates/config/`, `crates/data/`, `crates/ml/`, `crates/risk/`, `crates/storage/`, `crates/trading_engine/`, `Cargo.toml`, `Cargo.lock` |
| backtesting-service | `services/backtesting_service/`, `crates/common/`, `crates/config/`, `crates/data/`, `crates/ml/`, `crates/model_loader/`, `crates/risk/`, `crates/storage/`, `crates/trading_engine/`, `bin/fxt/`, `Cargo.toml`, `Cargo.lock` |
| trading-agent-service | `services/trading_agent_service/`, `crates/common/`, `crates/config/`, `crates/ml/`, `crates/risk/`, `Cargo.toml`, `Cargo.lock` |
| broker-gateway | `services/broker_gateway_service/`, `crates/common/`, `crates/config/`, `crates/trading_engine/`, `vendor/ctrader-openapi/`, `Cargo.toml`, `Cargo.lock` |
| data-acquisition-service | `services/data_acquisition_service/`, `crates/common/`, `crates/config/`, `crates/storage/`, `Cargo.toml`, `Cargo.lock` |
| web-gateway | `crates/web-gateway/`, `crates/common/`, `Cargo.toml`, `Cargo.lock` |
| training | `crates/ml/`, `crates/trading_engine/`, `crates/common/`, `crates/risk/`, `crates/data/`, `crates/config/`, `crates/storage/`, `services/training_uploader/`, `Cargo.toml`, `Cargo.lock` |

### 3. Compile Service WorkflowTemplate

Single parameterized template, invoked once per service that needs recompilation:

```yaml
name: compile-service
arguments:
  parameters:
    - name: service-package    # e.g. "api-gateway"
    - name: commit-sha
```

- **Image**: `ci-builder-cpu:latest` (Rust 1.89 + mold + sccache, no CUDA)
- **Node**: `ci-compile-cpu` pool (POP2-32C-128G)
- **Resources**: 4 CPU request / 6 CPU limit, 8Gi / 16Gi memory (same as current per-service jobs)
- **Volumes**: sccache PVC (`/mnt/sccache`), workspace volumeClaimTemplate for build output
- **Script**: `cargo build --profile dev-release -p <service-package>`, strip, copy to workspace
- **Artifact output**: Binary at `/workspace/services/<service-package>` → Argo artifact (S3/MinIO)

Git checkout: The container needs the source code. Options:
- **Git init container**: Clone repo at `checkout-sha` into workspace volume
- Use Argo's built-in `git` artifact source

A shared `git-checkout` step at the start of the DAG clones once into a workspace PVC. All compile steps mount the same PVC (ReadWriteOnce — all on same node, which is the case with 1 POP2-32C node).

### 4. Compile Training WorkflowTemplate

Same as current `compile-training` job but as an Argo template:

- **Image**: `ci-builder:latest` (CUDA 12.4 + Rust 1.89 — needs CUDA stubs)
- **Node**: `ci-compile-cpu` pool (CPU compile with CUDA stubs, no GPU needed)
- **Variables**: `CUDA_COMPUTE_CAP=89`
- **Script**: `cargo build --profile dev-release -p ml --features ml/cuda --example train_baseline_rl ...`
- **Output**: 7 training binaries as Argo artifacts

### 5. Build Web Dashboard WorkflowTemplate

- **Image**: `node:22-slim`
- **Node**: `ci-compile-cpu` pool
- **Resources**: 2 CPU / 2Gi memory
- **Script**: `cd web-dashboard && npm ci && npm run build`
- **Output**: `web-dashboard/dist/` as Argo artifact

### 6. Test Workspace WorkflowTemplate

- **Image**: `ci-builder:latest` (needs CUDA stubs for full workspace test)
- **Node**: `ci-compile-cpu` pool
- **Resources**: 14 CPU / 24Gi memory (same as current test job)
- **Services**: Redis sidecar container
- **Script**: `cargo test --workspace --lib`

### 7. Deploy WorkflowTemplate

Collects artifacts from all compile steps, uploads to MinIO, applies manifests, does selective rollout restart.

Steps:
1. **Upload binaries**: For each compiled service binary artifact, `aws s3 cp` to `s3://foxhunt-binaries/services/`
2. **Upload training binaries**: `aws s3 cp` to `s3://foxhunt-binaries/training/`
3. **Upload web assets**: `aws s3 cp` to `s3://foxhunt-binaries/static/`
4. **Apply migrations**: Same kubectl exec logic as current deploy
5. **Apply monitoring**: `kubectl apply -f infra/k8s/monitoring/`
6. **Apply dashboards**: Same ConfigMap generation script
7. **Apply service manifests**: `kubectl apply -f infra/k8s/services/`
8. **Selective rollout restart**: Only restart deployments whose binary was uploaded
9. **Wait for rollout**: `kubectl rollout status` with 300s timeout

**RBAC**: Dedicated ServiceAccount `argo-deploy` with:
- `deployments`, `statefulsets`, `configmaps`, `services` — get, list, create, update, patch
- `pods` — get, list, exec (for migrations)
- Namespace: `foxhunt`

### 8. Image Build WorkflowTemplate

Handles CI builder image rebuilds when Dockerfiles change:

- **Trigger**: `detect-changes` identifies `infra/docker/Dockerfile.*` changes
- **Image**: `gcr.io/kaniko-project/executor:debug`
- **Script**: Kaniko build + push to Scaleway Container Registry
- **Trivy scan**: Follow-up step scans pushed image for CRITICAL/HIGH vulns

### 9. IaC WorkflowTemplate

Terragrunt plan/apply:

- **Trigger**: `detect-changes` identifies `infra/` changes
- **Image**: `infra-runner:latest`
- **Plan step**: `terragrunt run-all plan` (always runs)
- **Apply step**: `terragrunt run-all apply -auto-approve` (only on main)

### 10. Volumes & Storage

| Volume | Type | Mount | Used by |
|---|---|---|---|
| sccache | PVC (existing) | `/mnt/sccache` | compile-service, compile-training, test |
| training-data | PVC (existing) | `/mnt/training-data` | training workflows |
| workspace | volumeClaimTemplate (5Gi) | `/workspace` | per-workflow scratch space |
| source | PVC or emptyDir | `/src` | git checkout, shared by all compile steps |

### 11. Migration Strategy

Deploy new first, test, then disable old:

1. **Install Argo Events**: `helm install argo-events argo/argo-events -n foxhunt -f infra/k8s/argo/events-values.yaml`
2. **Deploy WorkflowTemplates**: `kubectl apply -f infra/k8s/argo/`
3. **Deploy EventSource + Sensor**: `kubectl apply -f infra/k8s/argo/events/`
4. **Configure GitLab webhook**: Point to EventSource endpoint
5. **Test**: Push a small change to main, verify Argo workflow runs and deploys correctly
6. **Disable GitLab CI**: Set all jobs to `when: never` in `.gitlab-ci.yml`
7. **Validation period**: Run for 1-2 weeks with GitLab CI disabled but `.gitlab-ci.yml` still present
8. **Delete**: Remove `.gitlab-ci.yml`, uninstall GitLab Runner

**Rollback**: Revert `.gitlab-ci.yml` to re-enable GitLab CI. The webhook can coexist — Argo Events ignores pushes if sensor is deleted.

## File Layout

```
infra/k8s/argo/
├── values.yaml                          # Argo Workflows Helm values (existing)
├── events-values.yaml                   # Argo Events Helm values (new)
├── kustomization.yaml                   # Updated to include new templates
├── argo-workflow-netpol.yaml            # Network policy (existing)
├── training-workflow-template.yaml      # Training DAG (existing, unchanged)
├── ci-pipeline-template.yaml            # Main CI orchestrator (new)
├── compile-service-template.yaml        # Per-service compile (new)
├── compile-training-template.yaml       # Training binary compile (new)
├── build-web-dashboard-template.yaml    # Vite build (new)
├── test-workspace-template.yaml         # cargo test (new)
├── deploy-template.yaml                 # Deploy to k8s (new)
├── build-ci-image-template.yaml         # Kaniko image builds (new)
├── infra-template.yaml                  # Terragrunt plan/apply (new)
├── deploy-rbac.yaml                     # ServiceAccount + RBAC for deploy (new)
└── events/
    ├── gitlab-push-eventsource.yaml     # Webhook EventSource (new)
    └── ci-pipeline-sensor.yaml          # Sensor + trigger (new)
```

## What Gets Deleted

After validation:
- `.gitlab-ci.yml` (1600+ lines)
- GitLab Runner pods and configuration
- GitLab Runner Helm release
- CI-related GitLab project settings (variables stay as k8s secrets)

## What Stays

- GitLab CE as git server (push, pull, webhooks)
- All existing Argo Workflows infrastructure
- MinIO for artifacts and binary storage
- Scaleway Container Registry for images
- All k8s service manifests, monitoring, IaC