foxhunt/docs/plans/2026-02-27-autonomous-agents-design.md

# Autonomous CI Agents — Design Document

**Date**: 2026-02-27
**Status**: Approved

## Problem

CI pipeline failures and feature enhancements currently require manual human intervention. The foxhunt workspace has 4,500+ tests across 30 crate targets, and a failed pipeline means a developer must read logs, diagnose the issue, write a fix, push, and verify. This cycle can be automated.

## Solution

Deploy **OpenHands** (open-source autonomous coding agent, 72% SWE-bench) on Kapsule with a **Rust dispatcher** service and **LiteLLM** proxy. GitLab issues serve as the durable communication bus between 5 specialist agents that execute a fix loop.

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                     GitLab CE (git.fxhnt.ai)                    │
│  ┌──────────┐  ┌──────────────┐  ┌───────────────────────────┐ │
│  │ Webhooks  │  │ Issues/MRs   │  │ CI Pipelines              │ │
│  │ (pipeline,│  │ (agent bus,  │  │ (check→test→build→deploy) │ │
│  │  note)    │  │  labels,     │  │                           │ │
│  │           │  │  comments)   │  │                           │ │
│  └─────┬─────┘  └──────┬───────┘  └─────────────┬─────────────┘ │
└────────┼───────────────┼─────────────────────────┼───────────────┘
         │               │                         │
         ▼               │                         │
┌────────────────┐       │                         │
│  Dispatcher    │◄──────┘                         │
│  (Rust/Axum)   │  reads issues, labels           │
│  on ci pool    │  posts comments                 │
│                │─────────────────────┐            │
└───────┬────────┘                     │            │
        │ spawns headless              │ glab CLI   │
        ▼                              ▼            │
┌────────────────────────────────────────────┐     │
│  OpenHands Runtime (Kubernetes)             │     │
│  ┌────────┐ ┌────────┐ ┌──────────────┐   │     │
│  │Diagnose│ │ Plan   │ │ Implement    │   │     │
│  │ Agent  │ │ Agent  │ │ Agent        │   │     │
│  └───┬────┘ └───┬────┘ └──────┬───────┘   │     │
│      │          │              │            │     │
│  ┌───┴────┐ ┌───┴────┐                     │     │
│  │Verify  │ │Deliver │                     │     │
│  │ Agent  │ │ Agent  │                     │     │
│  └────────┘ └────────┘                     │     │
└────────────────────────────────────────────┘     │
        │                                          │
        ▼                                          │
┌────────────────┐                                 │
│  LiteLLM Proxy │  routes to:                     │
│  on ci pool    │  • Claude Opus 4.6 (diagnose,   │
│                │    plan)                         │
│                │  • Claude Sonnet 4.6 (implement) │
│                │  • SCW devstral-2-123b (verify,  │
│                │    deliver)                      │
└────────────────┘                                 │
```

## Trigger Mechanisms

### Automatic: CI Pipeline Failure
1. Pipeline fails → GitLab webhook fires `pipeline` event with `status: "failed"`
2. Dispatcher receives webhook, fetches job log via `GET /api/v4/projects/:id/jobs/:job_id/trace`
3. Dispatcher creates GitLab issue with structured error context
4. Issue gets labels: `trigger/ci-failure` + `phase/diagnose` + `agent/active`
5. Dispatcher invokes OpenHands diagnose agent

### Manual: @agent Mention in Issue
1. Human creates issue describing feature/bug, mentions `@agent` in description or comment
2. GitLab webhook fires `note` event
3. Dispatcher detects `@agent` mention, adds labels: `trigger/feature` + `phase/diagnose` + `agent/active`
4. Dispatcher invokes OpenHands diagnose agent

## 5-Phase Agent Loop

```
[TRIGGER] → [1. DIAGNOSE] → [2. PLAN] → [3. IMPLEMENT] → [4. VERIFY] → [5. DELIVER]
                  ▲                                              │
                  └──────────── retry (max 3) ───────────────────┘
```

### Phase 1: DIAGNOSE
- **LLM**: Claude Opus 4.6
- **Input**: CI log excerpt or issue description
- **Output**: Root cause analysis as structured GitLab comment
- **Handoff**: Remove `phase/diagnose`, add `phase/plan`

### Phase 2: PLAN
- **LLM**: Claude Opus 4.6
- **Input**: Diagnosis comment from Phase 1
- **Output**: Fix strategy with affected files, approach, risks
- **Handoff**: Remove `phase/plan`, add `phase/implement`

### Phase 3: IMPLEMENT
- **LLM**: Claude Sonnet 4.6
- **Input**: Plan comment from Phase 2
- **Output**: Code changes on `agent/fix-{issue_id}` branch, pushed to GitLab
- **Handoff**: Remove `phase/implement`, add `phase/verify`

### Phase 4: VERIFY
- **LLM**: Devstral-2-123b (Scaleway Generative APIs)
- **Input**: Branch name from Phase 3
- **Output**: CI pipeline triggered, results assessed
- **On GREEN**: Remove `phase/verify`, add `phase/deliver`
- **On RED**: Add `retry/N` label, loop to `phase/diagnose`
- **On retry/3**: Add `agent/blocked`, remove `agent/active` (escalate to human)

### Phase 5: DELIVER
- **LLM**: Devstral-2-123b (Scaleway Generative APIs)
- **Input**: Green pipeline confirmation
- **Output**: MR opened via `glab mr create` with `--auto-merge` flag (uses GitLab `merge_when_pipeline_succeeds` API)
- **CI-failure triggers**: MR auto-merges when MR pipeline passes (no human gate)
- **Feature triggers**: MR created but requires human approval before merge
- **Handoff**: Remove `agent/active`, add `agent/done`, close issue

## GitLab Issue Label Schema

| Label | Color | Purpose |
|-------|-------|---------|
| `agent/active` | `#28a745` | Agent currently working |
| `agent/blocked` | `#dc3545` | Needs human intervention |
| `agent/done` | `#6c757d` | Complete |
| `phase/diagnose` | `#007bff` | Phase 1 active |
| `phase/plan` | `#6f42c1` | Phase 2 active |
| `phase/implement` | `#fd7e14` | Phase 3 active |
| `phase/verify` | `#ffc107` | Phase 4 active |
| `phase/deliver` | `#28a745` | Phase 5 active |
| `trigger/ci-failure` | `#dc3545` | Auto-created from CI |
| `trigger/feature` | `#17a2b8` | Human-requested |
| `retry/1` .. `retry/3` | `#6c757d` | Retry counter |

## Agent Comment Format (Handoff Protocol)

```markdown
## Phase N: {Phase Name} Complete

**Agent**: {agent_name}
**LLM**: {model_used}
**Duration**: {elapsed}s
**Tokens**: {input_tokens} in / {output_tokens} out

### Findings
{structured findings}

### Handoff
**Next phase**: {next_phase}
**Context for next agent**: {key information the next agent needs}
```

## Human Intervention Points

- **Comment on issue**: Human comment pauses agent loop
- **Remove `agent/active`**: Stops all agent work immediately
- **Add `agent/blocked`**: Escalates to human (e.g., Slack notification)
- **Close issue**: Terminates agent work
- **Edit plan comment**: Human can modify Phase 2 plan before implementation proceeds

## Component Details

### 1. Dispatcher Service (`services/agent_dispatcher/`)

**Language**: Rust (Axum)
**Pool**: ci (GP1-XS)
**Port**: 8080

Responsibilities:
- Receive GitLab webhooks (pipeline, issue label, note events)
- Create issues with structured context on pipeline failure
- Invoke OpenHands headless on label transitions
- Track retry count, enforce max 3
- Concurrency control: max 2 simultaneous agent invocations
- Parse agent comments for handoff metadata

Dependencies:
- `axum` (HTTP server)
- `reqwest` (GitLab API client)
- `tokio` (async runtime, process spawning)
- `serde`/`serde_json` (webhook payload parsing)
- `tracing` (structured logging)

### 2. LiteLLM Proxy

**Image**: `ghcr.io/berriai/litellm:main-latest`
**Pool**: ci (GP1-XS)
**Port**: 4000

```yaml
model_list:
  - model_name: "opus"
    litellm_params:
      model: "claude-opus-4-6"
      api_key: "os.environ/ANTHROPIC_API_KEY"
  - model_name: "sonnet"
    litellm_params:
      model: "claude-sonnet-4-6"
      api_key: "os.environ/ANTHROPIC_API_KEY"
  - model_name: "devstral"
    litellm_params:
      model: "openai/devstral-2-123b-instruct-2512"
      api_base: "https://api.scaleway.ai/v1"
      api_key: "os.environ/SCW_GENERATIVE_API_KEY"

litellm_settings:
  max_budget: 50.0        # $50/day global cap
  budget_duration: "1d"
```

### 3. OpenHands Deployment

**Runtime**: Kubernetes (native pod scheduling)
**Namespace**: `foxhunt-agents`
**Image**: `ghcr.io/openhands/openhands:latest`

```toml
# config.toml
[core]
workspace_base = "/tmp/openhands-workspace"
max_iterations = 50

[llm]
model = "opus"
base_url = "http://litellm.foxhunt-agents.svc.cluster.local:4000/v1"
api_key = "via-litellm"

[sandbox]
timeout = 300
enable_gpu = false

[kubernetes]
namespace = "foxhunt-agents"
```

### 4. Microagent Definitions (`.openhands/microagents/`)

5 markdown files in the repo root, each defining phase-specific behavior:
- `diagnose.md` — Parse CI logs, identify root cause, structured output
- `plan.md` — Read diagnosis, propose fix, estimate scope
- `implement.md` — Create branch, write code, run `cargo check`
- `verify.md` — Push, trigger CI, poll results
- `deliver.md` — Open MR, summarize changes, close issue

### 5. LLM Routing by Phase

| Phase | Model | Provider | Cost (per 1M tokens) | Rationale |
|-------|-------|----------|---------------------|-----------|
| Diagnose | Opus 4.6 | Anthropic API | $15 in / $75 out | Hardest: deep Rust error understanding |
| Plan | Opus 4.6 | Anthropic API | $15 in / $75 out | Architectural judgment, multi-file reasoning |
| Implement | Sonnet 4.6 | Anthropic API | $3 in / $15 out | Good code gen, 5x cheaper than Opus |
| Verify | Devstral-2-123b | Scaleway | ~€0.40 in / €2.00 out | Log parsing, pass/fail, mechanical |
| Deliver | Devstral-2-123b | Scaleway | ~€0.40 in / €2.00 out | MR description, mechanical |

## Infrastructure (New K8s Resources)

| Resource | Pool | Purpose |
|----------|------|---------|
| `Namespace foxhunt-agents` | — | Isolates agent workloads |
| `Deployment dispatcher` | ci | Webhook receiver + agent invoker |
| `Deployment litellm` | ci | LLM proxy with cost tracking |
| `Deployment openhands` | ci | OpenHands server (K8s runtime) |
| `ServiceAccount openhands-agent` | — | RBAC for pod creation |
| `Role + RoleBinding` | — | Create/delete pods in foxhunt-agents |
| `Secret agent-credentials` | — | ANTHROPIC_API_KEY, SCW_GENERATIVE_API_KEY, GITLAB_TOKEN |
| `ConfigMap openhands-config` | — | config.toml for OpenHands |
| `ConfigMap litellm-config` | — | LiteLLM model routing config |
| `PVC agent-workspace` (10Gi) | ci | Workspace for agent repo checkouts |

**Incremental cost**: All on ci pool (GP1-XS, ~€0.08/hr). No new GPU nodes. LLM cost is pay-per-token.

## Security & Guardrails

- **Branch protection**: Agents push to `agent/fix-*` branches only, never `main`
- **Auto-merge**: Agent MRs auto-merge when CI passes (GitLab `merge_when_pipeline_succeeds` API). Configurable per trigger type — CI-failure fixes auto-merge by default, feature MRs require human approval
- **MR approval bypass**: Only for `trigger/ci-failure` issues. Feature-triggered MRs still require human approval
- **Max 3 retries**: After 3 failed attempts, `agent/blocked` label + human notification
- **Concurrency limit**: Max 2 simultaneous agent invocations (configurable in dispatcher)
- **Token scoping**: GitLab PAT with `api` + `write_repository`, project-scoped
- **Cost cap**: LiteLLM enforces $50/day global budget
- **Audit trail**: Every action is a GitLab issue comment — fully observable
- **Human override**: Remove `agent/active` label to stop any agent immediately

## Incremental Rollout Plan

1. **Phase 0**: Deploy LiteLLM proxy, verify model routing works
2. **Phase 1**: Deploy dispatcher, wire up pipeline failure webhook → issue creation
3. **Phase 2**: Deploy OpenHands, implement diagnose agent only — verify it produces useful comments
4. **Phase 3**: Add plan + implement agents — verify they produce reasonable code
5. **Phase 4**: Add verify + deliver agents — close the loop
6. **Phase 5**: Enable @agent mention trigger for feature work

## References

- [OpenHands Headless Mode](https://docs.openhands.dev/openhands/usage/run-openhands/headless-mode)
- [OpenHands GitHub](https://github.com/OpenHands/OpenHands)
- [OpenHands GitLab Integration](https://deepwiki.com/OpenHands/OpenHands/11.1-docker-deployment)
- [LiteLLM Proxy](https://docs.litellm.ai/docs/tutorials/claude_responses_api)
- [Scaleway Generative APIs](https://www.scaleway.com/en/generative-apis/)
- [Claude Code GitLab CI/CD (Official Beta)](https://code.claude.com/docs/en/gitlab-ci-cd)
- [GitLab Webhook Events](https://docs.gitlab.com/user/project/integrations/webhook_events/)