From 9937c67b4a4febdc05051a95be7ac32f60622be2 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Fri, 27 Feb 2026 08:14:17 +0100 Subject: [PATCH] =?UTF-8?q?docs:=20autonomous=20CI=20agents=20design=20?= =?UTF-8?q?=E2=80=94=20OpenHands=20+=20GitLab=20issue=20bus?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 5-phase agent loop (diagnose → plan → implement → verify → deliver) with GitLab issues as durable communication/handoff layer. OpenHands on Kapsule K8s runtime, LiteLLM proxy routing to Claude Opus/Sonnet and Scaleway devstral. Rust dispatcher service. Auto-merge for CI failure fixes. Co-Authored-By: Claude Opus 4.6 --- .../2026-02-27-autonomous-agents-design.md | 299 ++++++++++++++++++ 1 file changed, 299 insertions(+) create mode 100644 docs/plans/2026-02-27-autonomous-agents-design.md diff --git a/docs/plans/2026-02-27-autonomous-agents-design.md b/docs/plans/2026-02-27-autonomous-agents-design.md new file mode 100644 index 000000000..ef05e8a3e --- /dev/null +++ b/docs/plans/2026-02-27-autonomous-agents-design.md @@ -0,0 +1,299 @@ +# Autonomous CI Agents — Design Document + +**Date**: 2026-02-27 +**Status**: Approved + +## Problem + +CI pipeline failures and feature enhancements currently require manual human intervention. The foxhunt workspace has 4,500+ tests across 30 crate targets, and a failed pipeline means a developer must read logs, diagnose the issue, write a fix, push, and verify. This cycle can be automated. + +## Solution + +Deploy **OpenHands** (open-source autonomous coding agent, 72% SWE-bench) on Kapsule with a **Rust dispatcher** service and **LiteLLM** proxy. GitLab issues serve as the durable communication bus between 5 specialist agents that execute a fix loop. + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ GitLab CE (git.fxhnt.ai) │ +│ ┌──────────┐ ┌──────────────┐ ┌───────────────────────────┐ │ +│ │ Webhooks │ │ Issues/MRs │ │ CI Pipelines │ │ +│ │ (pipeline,│ │ (agent bus, │ │ (check→test→build→deploy) │ │ +│ │ note) │ │ labels, │ │ │ │ +│ │ │ │ comments) │ │ │ │ +│ └─────┬─────┘ └──────┬───────┘ └─────────────┬─────────────┘ │ +└────────┼───────────────┼─────────────────────────┼───────────────┘ + │ │ │ + ▼ │ │ +┌────────────────┐ │ │ +│ Dispatcher │◄──────┘ │ +│ (Rust/Axum) │ reads issues, labels │ +│ on ci pool │ posts comments │ +│ │─────────────────────┐ │ +└───────┬────────┘ │ │ + │ spawns headless │ glab CLI │ + ▼ ▼ │ +┌────────────────────────────────────────────┐ │ +│ OpenHands Runtime (Kubernetes) │ │ +│ ┌────────┐ ┌────────┐ ┌──────────────┐ │ │ +│ │Diagnose│ │ Plan │ │ Implement │ │ │ +│ │ Agent │ │ Agent │ │ Agent │ │ │ +│ └───┬────┘ └───┬────┘ └──────┬───────┘ │ │ +│ │ │ │ │ │ +│ ┌───┴────┐ ┌───┴────┐ │ │ +│ │Verify │ │Deliver │ │ │ +│ │ Agent │ │ Agent │ │ │ +│ └────────┘ └────────┘ │ │ +└────────────────────────────────────────────┘ │ + │ │ + ▼ │ +┌────────────────┐ │ +│ LiteLLM Proxy │ routes to: │ +│ on ci pool │ • Claude Opus 4.6 (diagnose, │ +│ │ plan) │ +│ │ • Claude Sonnet 4.6 (implement) │ +│ │ • SCW devstral-2-123b (verify, │ +│ │ deliver) │ +└────────────────┘ │ +``` + +## Trigger Mechanisms + +### Automatic: CI Pipeline Failure +1. Pipeline fails → GitLab webhook fires `pipeline` event with `status: "failed"` +2. Dispatcher receives webhook, fetches job log via `GET /api/v4/projects/:id/jobs/:job_id/trace` +3. Dispatcher creates GitLab issue with structured error context +4. Issue gets labels: `trigger/ci-failure` + `phase/diagnose` + `agent/active` +5. Dispatcher invokes OpenHands diagnose agent + +### Manual: @agent Mention in Issue +1. Human creates issue describing feature/bug, mentions `@agent` in description or comment +2. GitLab webhook fires `note` event +3. Dispatcher detects `@agent` mention, adds labels: `trigger/feature` + `phase/diagnose` + `agent/active` +4. Dispatcher invokes OpenHands diagnose agent + +## 5-Phase Agent Loop + +``` +[TRIGGER] → [1. DIAGNOSE] → [2. PLAN] → [3. IMPLEMENT] → [4. VERIFY] → [5. DELIVER] + ▲ │ + └──────────── retry (max 3) ───────────────────┘ +``` + +### Phase 1: DIAGNOSE +- **LLM**: Claude Opus 4.6 +- **Input**: CI log excerpt or issue description +- **Output**: Root cause analysis as structured GitLab comment +- **Handoff**: Remove `phase/diagnose`, add `phase/plan` + +### Phase 2: PLAN +- **LLM**: Claude Opus 4.6 +- **Input**: Diagnosis comment from Phase 1 +- **Output**: Fix strategy with affected files, approach, risks +- **Handoff**: Remove `phase/plan`, add `phase/implement` + +### Phase 3: IMPLEMENT +- **LLM**: Claude Sonnet 4.6 +- **Input**: Plan comment from Phase 2 +- **Output**: Code changes on `agent/fix-{issue_id}` branch, pushed to GitLab +- **Handoff**: Remove `phase/implement`, add `phase/verify` + +### Phase 4: VERIFY +- **LLM**: Devstral-2-123b (Scaleway Generative APIs) +- **Input**: Branch name from Phase 3 +- **Output**: CI pipeline triggered, results assessed +- **On GREEN**: Remove `phase/verify`, add `phase/deliver` +- **On RED**: Add `retry/N` label, loop to `phase/diagnose` +- **On retry/3**: Add `agent/blocked`, remove `agent/active` (escalate to human) + +### Phase 5: DELIVER +- **LLM**: Devstral-2-123b (Scaleway Generative APIs) +- **Input**: Green pipeline confirmation +- **Output**: MR opened via `glab mr create` with `--auto-merge` flag (uses GitLab `merge_when_pipeline_succeeds` API) +- **CI-failure triggers**: MR auto-merges when MR pipeline passes (no human gate) +- **Feature triggers**: MR created but requires human approval before merge +- **Handoff**: Remove `agent/active`, add `agent/done`, close issue + +## GitLab Issue Label Schema + +| Label | Color | Purpose | +|-------|-------|---------| +| `agent/active` | `#28a745` | Agent currently working | +| `agent/blocked` | `#dc3545` | Needs human intervention | +| `agent/done` | `#6c757d` | Complete | +| `phase/diagnose` | `#007bff` | Phase 1 active | +| `phase/plan` | `#6f42c1` | Phase 2 active | +| `phase/implement` | `#fd7e14` | Phase 3 active | +| `phase/verify` | `#ffc107` | Phase 4 active | +| `phase/deliver` | `#28a745` | Phase 5 active | +| `trigger/ci-failure` | `#dc3545` | Auto-created from CI | +| `trigger/feature` | `#17a2b8` | Human-requested | +| `retry/1` .. `retry/3` | `#6c757d` | Retry counter | + +## Agent Comment Format (Handoff Protocol) + +```markdown +## Phase N: {Phase Name} Complete + +**Agent**: {agent_name} +**LLM**: {model_used} +**Duration**: {elapsed}s +**Tokens**: {input_tokens} in / {output_tokens} out + +### Findings +{structured findings} + +### Handoff +**Next phase**: {next_phase} +**Context for next agent**: {key information the next agent needs} +``` + +## Human Intervention Points + +- **Comment on issue**: Human comment pauses agent loop +- **Remove `agent/active`**: Stops all agent work immediately +- **Add `agent/blocked`**: Escalates to human (e.g., Slack notification) +- **Close issue**: Terminates agent work +- **Edit plan comment**: Human can modify Phase 2 plan before implementation proceeds + +## Component Details + +### 1. Dispatcher Service (`services/agent_dispatcher/`) + +**Language**: Rust (Axum) +**Pool**: ci (GP1-XS) +**Port**: 8080 + +Responsibilities: +- Receive GitLab webhooks (pipeline, issue label, note events) +- Create issues with structured context on pipeline failure +- Invoke OpenHands headless on label transitions +- Track retry count, enforce max 3 +- Concurrency control: max 2 simultaneous agent invocations +- Parse agent comments for handoff metadata + +Dependencies: +- `axum` (HTTP server) +- `reqwest` (GitLab API client) +- `tokio` (async runtime, process spawning) +- `serde`/`serde_json` (webhook payload parsing) +- `tracing` (structured logging) + +### 2. LiteLLM Proxy + +**Image**: `ghcr.io/berriai/litellm:main-latest` +**Pool**: ci (GP1-XS) +**Port**: 4000 + +```yaml +model_list: + - model_name: "opus" + litellm_params: + model: "claude-opus-4-6" + api_key: "os.environ/ANTHROPIC_API_KEY" + - model_name: "sonnet" + litellm_params: + model: "claude-sonnet-4-6" + api_key: "os.environ/ANTHROPIC_API_KEY" + - model_name: "devstral" + litellm_params: + model: "openai/devstral-2-123b-instruct-2512" + api_base: "https://api.scaleway.ai/v1" + api_key: "os.environ/SCW_GENERATIVE_API_KEY" + +litellm_settings: + max_budget: 50.0 # $50/day global cap + budget_duration: "1d" +``` + +### 3. OpenHands Deployment + +**Runtime**: Kubernetes (native pod scheduling) +**Namespace**: `foxhunt-agents` +**Image**: `ghcr.io/openhands/openhands:latest` + +```toml +# config.toml +[core] +workspace_base = "/tmp/openhands-workspace" +max_iterations = 50 + +[llm] +model = "opus" +base_url = "http://litellm.foxhunt-agents.svc.cluster.local:4000/v1" +api_key = "via-litellm" + +[sandbox] +timeout = 300 +enable_gpu = false + +[kubernetes] +namespace = "foxhunt-agents" +``` + +### 4. Microagent Definitions (`.openhands/microagents/`) + +5 markdown files in the repo root, each defining phase-specific behavior: +- `diagnose.md` — Parse CI logs, identify root cause, structured output +- `plan.md` — Read diagnosis, propose fix, estimate scope +- `implement.md` — Create branch, write code, run `cargo check` +- `verify.md` — Push, trigger CI, poll results +- `deliver.md` — Open MR, summarize changes, close issue + +### 5. LLM Routing by Phase + +| Phase | Model | Provider | Cost (per 1M tokens) | Rationale | +|-------|-------|----------|---------------------|-----------| +| Diagnose | Opus 4.6 | Anthropic API | $15 in / $75 out | Hardest: deep Rust error understanding | +| Plan | Opus 4.6 | Anthropic API | $15 in / $75 out | Architectural judgment, multi-file reasoning | +| Implement | Sonnet 4.6 | Anthropic API | $3 in / $15 out | Good code gen, 5x cheaper than Opus | +| Verify | Devstral-2-123b | Scaleway | ~€0.40 in / €2.00 out | Log parsing, pass/fail, mechanical | +| Deliver | Devstral-2-123b | Scaleway | ~€0.40 in / €2.00 out | MR description, mechanical | + +## Infrastructure (New K8s Resources) + +| Resource | Pool | Purpose | +|----------|------|---------| +| `Namespace foxhunt-agents` | — | Isolates agent workloads | +| `Deployment dispatcher` | ci | Webhook receiver + agent invoker | +| `Deployment litellm` | ci | LLM proxy with cost tracking | +| `Deployment openhands` | ci | OpenHands server (K8s runtime) | +| `ServiceAccount openhands-agent` | — | RBAC for pod creation | +| `Role + RoleBinding` | — | Create/delete pods in foxhunt-agents | +| `Secret agent-credentials` | — | ANTHROPIC_API_KEY, SCW_GENERATIVE_API_KEY, GITLAB_TOKEN | +| `ConfigMap openhands-config` | — | config.toml for OpenHands | +| `ConfigMap litellm-config` | — | LiteLLM model routing config | +| `PVC agent-workspace` (10Gi) | ci | Workspace for agent repo checkouts | + +**Incremental cost**: All on ci pool (GP1-XS, ~€0.08/hr). No new GPU nodes. LLM cost is pay-per-token. + +## Security & Guardrails + +- **Branch protection**: Agents push to `agent/fix-*` branches only, never `main` +- **Auto-merge**: Agent MRs auto-merge when CI passes (GitLab `merge_when_pipeline_succeeds` API). Configurable per trigger type — CI-failure fixes auto-merge by default, feature MRs require human approval +- **MR approval bypass**: Only for `trigger/ci-failure` issues. Feature-triggered MRs still require human approval +- **Max 3 retries**: After 3 failed attempts, `agent/blocked` label + human notification +- **Concurrency limit**: Max 2 simultaneous agent invocations (configurable in dispatcher) +- **Token scoping**: GitLab PAT with `api` + `write_repository`, project-scoped +- **Cost cap**: LiteLLM enforces $50/day global budget +- **Audit trail**: Every action is a GitLab issue comment — fully observable +- **Human override**: Remove `agent/active` label to stop any agent immediately + +## Incremental Rollout Plan + +1. **Phase 0**: Deploy LiteLLM proxy, verify model routing works +2. **Phase 1**: Deploy dispatcher, wire up pipeline failure webhook → issue creation +3. **Phase 2**: Deploy OpenHands, implement diagnose agent only — verify it produces useful comments +4. **Phase 3**: Add plan + implement agents — verify they produce reasonable code +5. **Phase 4**: Add verify + deliver agents — close the loop +6. **Phase 5**: Enable @agent mention trigger for feature work + +## References + +- [OpenHands Headless Mode](https://docs.openhands.dev/openhands/usage/run-openhands/headless-mode) +- [OpenHands GitHub](https://github.com/OpenHands/OpenHands) +- [OpenHands GitLab Integration](https://deepwiki.com/OpenHands/OpenHands/11.1-docker-deployment) +- [LiteLLM Proxy](https://docs.litellm.ai/docs/tutorials/claude_responses_api) +- [Scaleway Generative APIs](https://www.scaleway.com/en/generative-apis/) +- [Claude Code GitLab CI/CD (Official Beta)](https://code.claude.com/docs/en/gitlab-ci-cd) +- [GitLab Webhook Events](https://docs.gitlab.com/user/project/integrations/webhook_events/)