# Autonomous CI Agents — Design Document **Date**: 2026-02-27 **Status**: Approved ## Problem CI pipeline failures and feature enhancements currently require manual human intervention. The foxhunt workspace has 4,500+ tests across 30 crate targets, and a failed pipeline means a developer must read logs, diagnose the issue, write a fix, push, and verify. This cycle can be automated. ## Solution Deploy **OpenHands** (open-source autonomous coding agent, 72% SWE-bench) on Kapsule with a **Rust dispatcher** service and **LiteLLM** proxy. GitLab issues serve as the durable communication bus between 5 specialist agents that execute a fix loop. ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ GitLab CE (git.fxhnt.ai) │ │ ┌──────────┐ ┌──────────────┐ ┌───────────────────────────┐ │ │ │ Webhooks │ │ Issues/MRs │ │ CI Pipelines │ │ │ │ (pipeline,│ │ (agent bus, │ │ (check→test→build→deploy) │ │ │ │ note) │ │ labels, │ │ │ │ │ │ │ │ comments) │ │ │ │ │ └─────┬─────┘ └──────┬───────┘ └─────────────┬─────────────┘ │ └────────┼───────────────┼─────────────────────────┼───────────────┘ │ │ │ ▼ │ │ ┌────────────────┐ │ │ │ Dispatcher │◄──────┘ │ │ (Rust/Axum) │ reads issues, labels │ │ on ci pool │ posts comments │ │ │─────────────────────┐ │ └───────┬────────┘ │ │ │ spawns headless │ glab CLI │ ▼ ▼ │ ┌────────────────────────────────────────────┐ │ │ OpenHands Runtime (Kubernetes) │ │ │ ┌────────┐ ┌────────┐ ┌──────────────┐ │ │ │ │Diagnose│ │ Plan │ │ Implement │ │ │ │ │ Agent │ │ Agent │ │ Agent │ │ │ │ └───┬────┘ └───┬────┘ └──────┬───────┘ │ │ │ │ │ │ │ │ │ ┌───┴────┐ ┌───┴────┐ │ │ │ │Verify │ │Deliver │ │ │ │ │ Agent │ │ Agent │ │ │ │ └────────┘ └────────┘ │ │ └────────────────────────────────────────────┘ │ │ │ ▼ │ ┌────────────────┐ │ │ LiteLLM Proxy │ routes to: │ │ on ci pool │ • Claude Opus 4.6 (diagnose, │ │ │ plan) │ │ │ • Claude Sonnet 4.6 (implement) │ │ │ • SCW devstral-2-123b (verify, │ │ │ deliver) │ └────────────────┘ │ ``` ## Trigger Mechanisms ### Automatic: CI Pipeline Failure 1. Pipeline fails → GitLab webhook fires `pipeline` event with `status: "failed"` 2. Dispatcher receives webhook, fetches job log via `GET /api/v4/projects/:id/jobs/:job_id/trace` 3. Dispatcher creates GitLab issue with structured error context 4. Issue gets labels: `trigger/ci-failure` + `phase/diagnose` + `agent/active` 5. Dispatcher invokes OpenHands diagnose agent ### Manual: @agent Mention in Issue 1. Human creates issue describing feature/bug, mentions `@agent` in description or comment 2. GitLab webhook fires `note` event 3. Dispatcher detects `@agent` mention, adds labels: `trigger/feature` + `phase/diagnose` + `agent/active` 4. Dispatcher invokes OpenHands diagnose agent ## 5-Phase Agent Loop ``` [TRIGGER] → [1. DIAGNOSE] → [2. PLAN] → [3. IMPLEMENT] → [4. VERIFY] → [5. DELIVER] ▲ │ └──────────── retry (max 3) ───────────────────┘ ``` ### Phase 1: DIAGNOSE - **LLM**: Claude Opus 4.6 - **Input**: CI log excerpt or issue description - **Output**: Root cause analysis as structured GitLab comment - **Handoff**: Remove `phase/diagnose`, add `phase/plan` ### Phase 2: PLAN - **LLM**: Claude Opus 4.6 - **Input**: Diagnosis comment from Phase 1 - **Output**: Fix strategy with affected files, approach, risks - **Handoff**: Remove `phase/plan`, add `phase/implement` ### Phase 3: IMPLEMENT - **LLM**: Claude Sonnet 4.6 - **Input**: Plan comment from Phase 2 - **Output**: Code changes on `agent/fix-{issue_id}` branch, pushed to GitLab - **Handoff**: Remove `phase/implement`, add `phase/verify` ### Phase 4: VERIFY - **LLM**: Devstral-2-123b (Scaleway Generative APIs) - **Input**: Branch name from Phase 3 - **Output**: CI pipeline triggered, results assessed - **On GREEN**: Remove `phase/verify`, add `phase/deliver` - **On RED**: Add `retry/N` label, loop to `phase/diagnose` - **On retry/3**: Add `agent/blocked`, remove `agent/active` (escalate to human) ### Phase 5: DELIVER - **LLM**: Devstral-2-123b (Scaleway Generative APIs) - **Input**: Green pipeline confirmation - **Output**: MR opened via `glab mr create` with `--auto-merge` flag (uses GitLab `merge_when_pipeline_succeeds` API) - **CI-failure triggers**: MR auto-merges when MR pipeline passes (no human gate) - **Feature triggers**: MR created but requires human approval before merge - **Handoff**: Remove `agent/active`, add `agent/done`, close issue ## GitLab Issue Label Schema | Label | Color | Purpose | |-------|-------|---------| | `agent/active` | `#28a745` | Agent currently working | | `agent/blocked` | `#dc3545` | Needs human intervention | | `agent/done` | `#6c757d` | Complete | | `phase/diagnose` | `#007bff` | Phase 1 active | | `phase/plan` | `#6f42c1` | Phase 2 active | | `phase/implement` | `#fd7e14` | Phase 3 active | | `phase/verify` | `#ffc107` | Phase 4 active | | `phase/deliver` | `#28a745` | Phase 5 active | | `trigger/ci-failure` | `#dc3545` | Auto-created from CI | | `trigger/feature` | `#17a2b8` | Human-requested | | `retry/1` .. `retry/3` | `#6c757d` | Retry counter | ## Agent Comment Format (Handoff Protocol) ```markdown ## Phase N: {Phase Name} Complete **Agent**: {agent_name} **LLM**: {model_used} **Duration**: {elapsed}s **Tokens**: {input_tokens} in / {output_tokens} out ### Findings {structured findings} ### Handoff **Next phase**: {next_phase} **Context for next agent**: {key information the next agent needs} ``` ## Human Intervention Points - **Comment on issue**: Human comment pauses agent loop - **Remove `agent/active`**: Stops all agent work immediately - **Add `agent/blocked`**: Escalates to human (e.g., Slack notification) - **Close issue**: Terminates agent work - **Edit plan comment**: Human can modify Phase 2 plan before implementation proceeds ## Component Details ### 1. Dispatcher Service (`services/agent_dispatcher/`) **Language**: Rust (Axum) **Pool**: ci (GP1-XS) **Port**: 8080 Responsibilities: - Receive GitLab webhooks (pipeline, issue label, note events) - Create issues with structured context on pipeline failure - Invoke OpenHands headless on label transitions - Track retry count, enforce max 3 - Concurrency control: max 2 simultaneous agent invocations - Parse agent comments for handoff metadata Dependencies: - `axum` (HTTP server) - `reqwest` (GitLab API client) - `tokio` (async runtime, process spawning) - `serde`/`serde_json` (webhook payload parsing) - `tracing` (structured logging) ### 2. LiteLLM Proxy **Image**: `ghcr.io/berriai/litellm:main-latest` **Pool**: ci (GP1-XS) **Port**: 4000 ```yaml model_list: - model_name: "opus" litellm_params: model: "claude-opus-4-6" api_key: "os.environ/ANTHROPIC_API_KEY" - model_name: "sonnet" litellm_params: model: "claude-sonnet-4-6" api_key: "os.environ/ANTHROPIC_API_KEY" - model_name: "devstral" litellm_params: model: "openai/devstral-2-123b-instruct-2512" api_base: "https://api.scaleway.ai/v1" api_key: "os.environ/SCW_GENERATIVE_API_KEY" litellm_settings: max_budget: 50.0 # $50/day global cap budget_duration: "1d" ``` ### 3. OpenHands Deployment **Runtime**: Kubernetes (native pod scheduling) **Namespace**: `foxhunt-agents` **Image**: `ghcr.io/openhands/openhands:latest` ```toml # config.toml [core] workspace_base = "/tmp/openhands-workspace" max_iterations = 50 [llm] model = "opus" base_url = "http://litellm.foxhunt-agents.svc.cluster.local:4000/v1" api_key = "via-litellm" [sandbox] timeout = 300 enable_gpu = false [kubernetes] namespace = "foxhunt-agents" ``` ### 4. Microagent Definitions (`.openhands/microagents/`) 5 markdown files in the repo root, each defining phase-specific behavior: - `diagnose.md` — Parse CI logs, identify root cause, structured output - `plan.md` — Read diagnosis, propose fix, estimate scope - `implement.md` — Create branch, write code, run `cargo check` - `verify.md` — Push, trigger CI, poll results - `deliver.md` — Open MR, summarize changes, close issue ### 5. LLM Routing by Phase | Phase | Model | Provider | Cost (per 1M tokens) | Rationale | |-------|-------|----------|---------------------|-----------| | Diagnose | Opus 4.6 | Anthropic API | $15 in / $75 out | Hardest: deep Rust error understanding | | Plan | Opus 4.6 | Anthropic API | $15 in / $75 out | Architectural judgment, multi-file reasoning | | Implement | Sonnet 4.6 | Anthropic API | $3 in / $15 out | Good code gen, 5x cheaper than Opus | | Verify | Devstral-2-123b | Scaleway | ~€0.40 in / €2.00 out | Log parsing, pass/fail, mechanical | | Deliver | Devstral-2-123b | Scaleway | ~€0.40 in / €2.00 out | MR description, mechanical | ## Infrastructure (New K8s Resources) | Resource | Pool | Purpose | |----------|------|---------| | `Namespace foxhunt-agents` | — | Isolates agent workloads | | `Deployment dispatcher` | ci | Webhook receiver + agent invoker | | `Deployment litellm` | ci | LLM proxy with cost tracking | | `Deployment openhands` | ci | OpenHands server (K8s runtime) | | `ServiceAccount openhands-agent` | — | RBAC for pod creation | | `Role + RoleBinding` | — | Create/delete pods in foxhunt-agents | | `Secret agent-credentials` | — | ANTHROPIC_API_KEY, SCW_GENERATIVE_API_KEY, GITLAB_TOKEN | | `ConfigMap openhands-config` | — | config.toml for OpenHands | | `ConfigMap litellm-config` | — | LiteLLM model routing config | | `PVC agent-workspace` (10Gi) | ci | Workspace for agent repo checkouts | **Incremental cost**: All on ci pool (GP1-XS, ~€0.08/hr). No new GPU nodes. LLM cost is pay-per-token. ## Security & Guardrails - **Branch protection**: Agents push to `agent/fix-*` branches only, never `main` - **Auto-merge**: Agent MRs auto-merge when CI passes (GitLab `merge_when_pipeline_succeeds` API). Configurable per trigger type — CI-failure fixes auto-merge by default, feature MRs require human approval - **MR approval bypass**: Only for `trigger/ci-failure` issues. Feature-triggered MRs still require human approval - **Max 3 retries**: After 3 failed attempts, `agent/blocked` label + human notification - **Concurrency limit**: Max 2 simultaneous agent invocations (configurable in dispatcher) - **Token scoping**: GitLab PAT with `api` + `write_repository`, project-scoped - **Cost cap**: LiteLLM enforces $50/day global budget - **Audit trail**: Every action is a GitLab issue comment — fully observable - **Human override**: Remove `agent/active` label to stop any agent immediately ## Incremental Rollout Plan 1. **Phase 0**: Deploy LiteLLM proxy, verify model routing works 2. **Phase 1**: Deploy dispatcher, wire up pipeline failure webhook → issue creation 3. **Phase 2**: Deploy OpenHands, implement diagnose agent only — verify it produces useful comments 4. **Phase 3**: Add plan + implement agents — verify they produce reasonable code 5. **Phase 4**: Add verify + deliver agents — close the loop 6. **Phase 5**: Enable @agent mention trigger for feature work ## References - [OpenHands Headless Mode](https://docs.openhands.dev/openhands/usage/run-openhands/headless-mode) - [OpenHands GitHub](https://github.com/OpenHands/OpenHands) - [OpenHands GitLab Integration](https://deepwiki.com/OpenHands/OpenHands/11.1-docker-deployment) - [LiteLLM Proxy](https://docs.litellm.ai/docs/tutorials/claude_responses_api) - [Scaleway Generative APIs](https://www.scaleway.com/en/generative-apis/) - [Claude Code GitLab CI/CD (Official Beta)](https://code.claude.com/docs/en/gitlab-ci-cd) - [GitLab Webhook Events](https://docs.gitlab.com/user/project/integrations/webhook_events/)