docs: autonomous CI agents design — OpenHands + GitLab issue bus
5-phase agent loop (diagnose → plan → implement → verify → deliver) with GitLab issues as durable communication/handoff layer. OpenHands on Kapsule K8s runtime, LiteLLM proxy routing to Claude Opus/Sonnet and Scaleway devstral. Rust dispatcher service. Auto-merge for CI failure fixes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
299
docs/plans/2026-02-27-autonomous-agents-design.md
Normal file
299
docs/plans/2026-02-27-autonomous-agents-design.md
Normal file
@@ -0,0 +1,299 @@
|
||||
# Autonomous CI Agents — Design Document
|
||||
|
||||
**Date**: 2026-02-27
|
||||
**Status**: Approved
|
||||
|
||||
## Problem
|
||||
|
||||
CI pipeline failures and feature enhancements currently require manual human intervention. The foxhunt workspace has 4,500+ tests across 30 crate targets, and a failed pipeline means a developer must read logs, diagnose the issue, write a fix, push, and verify. This cycle can be automated.
|
||||
|
||||
## Solution
|
||||
|
||||
Deploy **OpenHands** (open-source autonomous coding agent, 72% SWE-bench) on Kapsule with a **Rust dispatcher** service and **LiteLLM** proxy. GitLab issues serve as the durable communication bus between 5 specialist agents that execute a fix loop.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ GitLab CE (git.fxhnt.ai) │
|
||||
│ ┌──────────┐ ┌──────────────┐ ┌───────────────────────────┐ │
|
||||
│ │ Webhooks │ │ Issues/MRs │ │ CI Pipelines │ │
|
||||
│ │ (pipeline,│ │ (agent bus, │ │ (check→test→build→deploy) │ │
|
||||
│ │ note) │ │ labels, │ │ │ │
|
||||
│ │ │ │ comments) │ │ │ │
|
||||
│ └─────┬─────┘ └──────┬───────┘ └─────────────┬─────────────┘ │
|
||||
└────────┼───────────────┼─────────────────────────┼───────────────┘
|
||||
│ │ │
|
||||
▼ │ │
|
||||
┌────────────────┐ │ │
|
||||
│ Dispatcher │◄──────┘ │
|
||||
│ (Rust/Axum) │ reads issues, labels │
|
||||
│ on ci pool │ posts comments │
|
||||
│ │─────────────────────┐ │
|
||||
└───────┬────────┘ │ │
|
||||
│ spawns headless │ glab CLI │
|
||||
▼ ▼ │
|
||||
┌────────────────────────────────────────────┐ │
|
||||
│ OpenHands Runtime (Kubernetes) │ │
|
||||
│ ┌────────┐ ┌────────┐ ┌──────────────┐ │ │
|
||||
│ │Diagnose│ │ Plan │ │ Implement │ │ │
|
||||
│ │ Agent │ │ Agent │ │ Agent │ │ │
|
||||
│ └───┬────┘ └───┬────┘ └──────┬───────┘ │ │
|
||||
│ │ │ │ │ │
|
||||
│ ┌───┴────┐ ┌───┴────┐ │ │
|
||||
│ │Verify │ │Deliver │ │ │
|
||||
│ │ Agent │ │ Agent │ │ │
|
||||
│ └────────┘ └────────┘ │ │
|
||||
└────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
▼ │
|
||||
┌────────────────┐ │
|
||||
│ LiteLLM Proxy │ routes to: │
|
||||
│ on ci pool │ • Claude Opus 4.6 (diagnose, │
|
||||
│ │ plan) │
|
||||
│ │ • Claude Sonnet 4.6 (implement) │
|
||||
│ │ • SCW devstral-2-123b (verify, │
|
||||
│ │ deliver) │
|
||||
└────────────────┘ │
|
||||
```
|
||||
|
||||
## Trigger Mechanisms
|
||||
|
||||
### Automatic: CI Pipeline Failure
|
||||
1. Pipeline fails → GitLab webhook fires `pipeline` event with `status: "failed"`
|
||||
2. Dispatcher receives webhook, fetches job log via `GET /api/v4/projects/:id/jobs/:job_id/trace`
|
||||
3. Dispatcher creates GitLab issue with structured error context
|
||||
4. Issue gets labels: `trigger/ci-failure` + `phase/diagnose` + `agent/active`
|
||||
5. Dispatcher invokes OpenHands diagnose agent
|
||||
|
||||
### Manual: @agent Mention in Issue
|
||||
1. Human creates issue describing feature/bug, mentions `@agent` in description or comment
|
||||
2. GitLab webhook fires `note` event
|
||||
3. Dispatcher detects `@agent` mention, adds labels: `trigger/feature` + `phase/diagnose` + `agent/active`
|
||||
4. Dispatcher invokes OpenHands diagnose agent
|
||||
|
||||
## 5-Phase Agent Loop
|
||||
|
||||
```
|
||||
[TRIGGER] → [1. DIAGNOSE] → [2. PLAN] → [3. IMPLEMENT] → [4. VERIFY] → [5. DELIVER]
|
||||
▲ │
|
||||
└──────────── retry (max 3) ───────────────────┘
|
||||
```
|
||||
|
||||
### Phase 1: DIAGNOSE
|
||||
- **LLM**: Claude Opus 4.6
|
||||
- **Input**: CI log excerpt or issue description
|
||||
- **Output**: Root cause analysis as structured GitLab comment
|
||||
- **Handoff**: Remove `phase/diagnose`, add `phase/plan`
|
||||
|
||||
### Phase 2: PLAN
|
||||
- **LLM**: Claude Opus 4.6
|
||||
- **Input**: Diagnosis comment from Phase 1
|
||||
- **Output**: Fix strategy with affected files, approach, risks
|
||||
- **Handoff**: Remove `phase/plan`, add `phase/implement`
|
||||
|
||||
### Phase 3: IMPLEMENT
|
||||
- **LLM**: Claude Sonnet 4.6
|
||||
- **Input**: Plan comment from Phase 2
|
||||
- **Output**: Code changes on `agent/fix-{issue_id}` branch, pushed to GitLab
|
||||
- **Handoff**: Remove `phase/implement`, add `phase/verify`
|
||||
|
||||
### Phase 4: VERIFY
|
||||
- **LLM**: Devstral-2-123b (Scaleway Generative APIs)
|
||||
- **Input**: Branch name from Phase 3
|
||||
- **Output**: CI pipeline triggered, results assessed
|
||||
- **On GREEN**: Remove `phase/verify`, add `phase/deliver`
|
||||
- **On RED**: Add `retry/N` label, loop to `phase/diagnose`
|
||||
- **On retry/3**: Add `agent/blocked`, remove `agent/active` (escalate to human)
|
||||
|
||||
### Phase 5: DELIVER
|
||||
- **LLM**: Devstral-2-123b (Scaleway Generative APIs)
|
||||
- **Input**: Green pipeline confirmation
|
||||
- **Output**: MR opened via `glab mr create` with `--auto-merge` flag (uses GitLab `merge_when_pipeline_succeeds` API)
|
||||
- **CI-failure triggers**: MR auto-merges when MR pipeline passes (no human gate)
|
||||
- **Feature triggers**: MR created but requires human approval before merge
|
||||
- **Handoff**: Remove `agent/active`, add `agent/done`, close issue
|
||||
|
||||
## GitLab Issue Label Schema
|
||||
|
||||
| Label | Color | Purpose |
|
||||
|-------|-------|---------|
|
||||
| `agent/active` | `#28a745` | Agent currently working |
|
||||
| `agent/blocked` | `#dc3545` | Needs human intervention |
|
||||
| `agent/done` | `#6c757d` | Complete |
|
||||
| `phase/diagnose` | `#007bff` | Phase 1 active |
|
||||
| `phase/plan` | `#6f42c1` | Phase 2 active |
|
||||
| `phase/implement` | `#fd7e14` | Phase 3 active |
|
||||
| `phase/verify` | `#ffc107` | Phase 4 active |
|
||||
| `phase/deliver` | `#28a745` | Phase 5 active |
|
||||
| `trigger/ci-failure` | `#dc3545` | Auto-created from CI |
|
||||
| `trigger/feature` | `#17a2b8` | Human-requested |
|
||||
| `retry/1` .. `retry/3` | `#6c757d` | Retry counter |
|
||||
|
||||
## Agent Comment Format (Handoff Protocol)
|
||||
|
||||
```markdown
|
||||
## Phase N: {Phase Name} Complete
|
||||
|
||||
**Agent**: {agent_name}
|
||||
**LLM**: {model_used}
|
||||
**Duration**: {elapsed}s
|
||||
**Tokens**: {input_tokens} in / {output_tokens} out
|
||||
|
||||
### Findings
|
||||
{structured findings}
|
||||
|
||||
### Handoff
|
||||
**Next phase**: {next_phase}
|
||||
**Context for next agent**: {key information the next agent needs}
|
||||
```
|
||||
|
||||
## Human Intervention Points
|
||||
|
||||
- **Comment on issue**: Human comment pauses agent loop
|
||||
- **Remove `agent/active`**: Stops all agent work immediately
|
||||
- **Add `agent/blocked`**: Escalates to human (e.g., Slack notification)
|
||||
- **Close issue**: Terminates agent work
|
||||
- **Edit plan comment**: Human can modify Phase 2 plan before implementation proceeds
|
||||
|
||||
## Component Details
|
||||
|
||||
### 1. Dispatcher Service (`services/agent_dispatcher/`)
|
||||
|
||||
**Language**: Rust (Axum)
|
||||
**Pool**: ci (GP1-XS)
|
||||
**Port**: 8080
|
||||
|
||||
Responsibilities:
|
||||
- Receive GitLab webhooks (pipeline, issue label, note events)
|
||||
- Create issues with structured context on pipeline failure
|
||||
- Invoke OpenHands headless on label transitions
|
||||
- Track retry count, enforce max 3
|
||||
- Concurrency control: max 2 simultaneous agent invocations
|
||||
- Parse agent comments for handoff metadata
|
||||
|
||||
Dependencies:
|
||||
- `axum` (HTTP server)
|
||||
- `reqwest` (GitLab API client)
|
||||
- `tokio` (async runtime, process spawning)
|
||||
- `serde`/`serde_json` (webhook payload parsing)
|
||||
- `tracing` (structured logging)
|
||||
|
||||
### 2. LiteLLM Proxy
|
||||
|
||||
**Image**: `ghcr.io/berriai/litellm:main-latest`
|
||||
**Pool**: ci (GP1-XS)
|
||||
**Port**: 4000
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: "opus"
|
||||
litellm_params:
|
||||
model: "claude-opus-4-6"
|
||||
api_key: "os.environ/ANTHROPIC_API_KEY"
|
||||
- model_name: "sonnet"
|
||||
litellm_params:
|
||||
model: "claude-sonnet-4-6"
|
||||
api_key: "os.environ/ANTHROPIC_API_KEY"
|
||||
- model_name: "devstral"
|
||||
litellm_params:
|
||||
model: "openai/devstral-2-123b-instruct-2512"
|
||||
api_base: "https://api.scaleway.ai/v1"
|
||||
api_key: "os.environ/SCW_GENERATIVE_API_KEY"
|
||||
|
||||
litellm_settings:
|
||||
max_budget: 50.0 # $50/day global cap
|
||||
budget_duration: "1d"
|
||||
```
|
||||
|
||||
### 3. OpenHands Deployment
|
||||
|
||||
**Runtime**: Kubernetes (native pod scheduling)
|
||||
**Namespace**: `foxhunt-agents`
|
||||
**Image**: `ghcr.io/openhands/openhands:latest`
|
||||
|
||||
```toml
|
||||
# config.toml
|
||||
[core]
|
||||
workspace_base = "/tmp/openhands-workspace"
|
||||
max_iterations = 50
|
||||
|
||||
[llm]
|
||||
model = "opus"
|
||||
base_url = "http://litellm.foxhunt-agents.svc.cluster.local:4000/v1"
|
||||
api_key = "via-litellm"
|
||||
|
||||
[sandbox]
|
||||
timeout = 300
|
||||
enable_gpu = false
|
||||
|
||||
[kubernetes]
|
||||
namespace = "foxhunt-agents"
|
||||
```
|
||||
|
||||
### 4. Microagent Definitions (`.openhands/microagents/`)
|
||||
|
||||
5 markdown files in the repo root, each defining phase-specific behavior:
|
||||
- `diagnose.md` — Parse CI logs, identify root cause, structured output
|
||||
- `plan.md` — Read diagnosis, propose fix, estimate scope
|
||||
- `implement.md` — Create branch, write code, run `cargo check`
|
||||
- `verify.md` — Push, trigger CI, poll results
|
||||
- `deliver.md` — Open MR, summarize changes, close issue
|
||||
|
||||
### 5. LLM Routing by Phase
|
||||
|
||||
| Phase | Model | Provider | Cost (per 1M tokens) | Rationale |
|
||||
|-------|-------|----------|---------------------|-----------|
|
||||
| Diagnose | Opus 4.6 | Anthropic API | $15 in / $75 out | Hardest: deep Rust error understanding |
|
||||
| Plan | Opus 4.6 | Anthropic API | $15 in / $75 out | Architectural judgment, multi-file reasoning |
|
||||
| Implement | Sonnet 4.6 | Anthropic API | $3 in / $15 out | Good code gen, 5x cheaper than Opus |
|
||||
| Verify | Devstral-2-123b | Scaleway | ~€0.40 in / €2.00 out | Log parsing, pass/fail, mechanical |
|
||||
| Deliver | Devstral-2-123b | Scaleway | ~€0.40 in / €2.00 out | MR description, mechanical |
|
||||
|
||||
## Infrastructure (New K8s Resources)
|
||||
|
||||
| Resource | Pool | Purpose |
|
||||
|----------|------|---------|
|
||||
| `Namespace foxhunt-agents` | — | Isolates agent workloads |
|
||||
| `Deployment dispatcher` | ci | Webhook receiver + agent invoker |
|
||||
| `Deployment litellm` | ci | LLM proxy with cost tracking |
|
||||
| `Deployment openhands` | ci | OpenHands server (K8s runtime) |
|
||||
| `ServiceAccount openhands-agent` | — | RBAC for pod creation |
|
||||
| `Role + RoleBinding` | — | Create/delete pods in foxhunt-agents |
|
||||
| `Secret agent-credentials` | — | ANTHROPIC_API_KEY, SCW_GENERATIVE_API_KEY, GITLAB_TOKEN |
|
||||
| `ConfigMap openhands-config` | — | config.toml for OpenHands |
|
||||
| `ConfigMap litellm-config` | — | LiteLLM model routing config |
|
||||
| `PVC agent-workspace` (10Gi) | ci | Workspace for agent repo checkouts |
|
||||
|
||||
**Incremental cost**: All on ci pool (GP1-XS, ~€0.08/hr). No new GPU nodes. LLM cost is pay-per-token.
|
||||
|
||||
## Security & Guardrails
|
||||
|
||||
- **Branch protection**: Agents push to `agent/fix-*` branches only, never `main`
|
||||
- **Auto-merge**: Agent MRs auto-merge when CI passes (GitLab `merge_when_pipeline_succeeds` API). Configurable per trigger type — CI-failure fixes auto-merge by default, feature MRs require human approval
|
||||
- **MR approval bypass**: Only for `trigger/ci-failure` issues. Feature-triggered MRs still require human approval
|
||||
- **Max 3 retries**: After 3 failed attempts, `agent/blocked` label + human notification
|
||||
- **Concurrency limit**: Max 2 simultaneous agent invocations (configurable in dispatcher)
|
||||
- **Token scoping**: GitLab PAT with `api` + `write_repository`, project-scoped
|
||||
- **Cost cap**: LiteLLM enforces $50/day global budget
|
||||
- **Audit trail**: Every action is a GitLab issue comment — fully observable
|
||||
- **Human override**: Remove `agent/active` label to stop any agent immediately
|
||||
|
||||
## Incremental Rollout Plan
|
||||
|
||||
1. **Phase 0**: Deploy LiteLLM proxy, verify model routing works
|
||||
2. **Phase 1**: Deploy dispatcher, wire up pipeline failure webhook → issue creation
|
||||
3. **Phase 2**: Deploy OpenHands, implement diagnose agent only — verify it produces useful comments
|
||||
4. **Phase 3**: Add plan + implement agents — verify they produce reasonable code
|
||||
5. **Phase 4**: Add verify + deliver agents — close the loop
|
||||
6. **Phase 5**: Enable @agent mention trigger for feature work
|
||||
|
||||
## References
|
||||
|
||||
- [OpenHands Headless Mode](https://docs.openhands.dev/openhands/usage/run-openhands/headless-mode)
|
||||
- [OpenHands GitHub](https://github.com/OpenHands/OpenHands)
|
||||
- [OpenHands GitLab Integration](https://deepwiki.com/OpenHands/OpenHands/11.1-docker-deployment)
|
||||
- [LiteLLM Proxy](https://docs.litellm.ai/docs/tutorials/claude_responses_api)
|
||||
- [Scaleway Generative APIs](https://www.scaleway.com/en/generative-apis/)
|
||||
- [Claude Code GitLab CI/CD (Official Beta)](https://code.claude.com/docs/en/gitlab-ci-cd)
|
||||
- [GitLab Webhook Events](https://docs.gitlab.com/user/project/integrations/webhook_events/)
|
||||
Reference in New Issue
Block a user