foxhunt/docs/plans/2026-02-27-autonomous-agents-design.md

Autonomous CI Agents — Design Document

Date: 2026-02-27 Status: Approved

Problem

CI pipeline failures and feature enhancements currently require manual human intervention. The foxhunt workspace has 4,500+ tests across 30 crate targets, and a failed pipeline means a developer must read logs, diagnose the issue, write a fix, push, and verify. This cycle can be automated.

Solution

Deploy OpenHands (open-source autonomous coding agent, 72% SWE-bench) on Kapsule with a Rust dispatcher service and LiteLLM proxy. GitLab issues serve as the durable communication bus between 5 specialist agents that execute a fix loop.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     GitLab CE (git.fxhnt.ai)                    │
│  ┌──────────┐  ┌──────────────┐  ┌───────────────────────────┐ │
│  │ Webhooks  │  │ Issues/MRs   │  │ CI Pipelines              │ │
│  │ (pipeline,│  │ (agent bus,  │  │ (check→test→build→deploy) │ │
│  │  note)    │  │  labels,     │  │                           │ │
│  │           │  │  comments)   │  │                           │ │
│  └─────┬─────┘  └──────┬───────┘  └─────────────┬─────────────┘ │
└────────┼───────────────┼─────────────────────────┼───────────────┘
         │               │                         │
         ▼               │                         │
┌────────────────┐       │                         │
│  Dispatcher    │◄──────┘                         │
│  (Rust/Axum)   │  reads issues, labels           │
│  on ci pool    │  posts comments                 │
│                │─────────────────────┐            │
└───────┬────────┘                     │            │
        │ spawns headless              │ glab CLI   │
        ▼                              ▼            │
┌────────────────────────────────────────────┐     │
│  OpenHands Runtime (Kubernetes)             │     │
│  ┌────────┐ ┌────────┐ ┌──────────────┐   │     │
│  │Diagnose│ │ Plan   │ │ Implement    │   │     │
│  │ Agent  │ │ Agent  │ │ Agent        │   │     │
│  └───┬────┘ └───┬────┘ └──────┬───────┘   │     │
│      │          │              │            │     │
│  ┌───┴────┐ ┌───┴────┐                     │     │
│  │Verify  │ │Deliver │                     │     │
│  │ Agent  │ │ Agent  │                     │     │
│  └────────┘ └────────┘                     │     │
└────────────────────────────────────────────┘     │
        │                                          │
        ▼                                          │
┌────────────────┐                                 │
│  LiteLLM Proxy │  routes to:                     │
│  on ci pool    │  • Claude Opus 4.6 (diagnose,   │
│                │    plan)                         │
│                │  • Claude Sonnet 4.6 (implement) │
│                │  • SCW devstral-2-123b (verify,  │
│                │    deliver)                      │
└────────────────┘                                 │

Trigger Mechanisms

Automatic: CI Pipeline Failure

1. Pipeline fails → GitLab webhook fires pipeline event with status: "failed"
2. Dispatcher receives webhook, fetches job log via GET /api/v4/projects/:id/jobs/:job_id/trace
3. Dispatcher creates GitLab issue with structured error context
4. Issue gets labels: trigger/ci-failure + phase/diagnose + agent/active
5. Dispatcher invokes OpenHands diagnose agent

Manual: @agent Mention in Issue

1. Human creates issue describing feature/bug, mentions @agent in description or comment
2. GitLab webhook fires note event
3. Dispatcher detects @agent mention, adds labels: trigger/feature + phase/diagnose + agent/active
4. Dispatcher invokes OpenHands diagnose agent

5-Phase Agent Loop

[TRIGGER] → [1. DIAGNOSE] → [2. PLAN] → [3. IMPLEMENT] → [4. VERIFY] → [5. DELIVER]
                  ▲                                              │
                  └──────────── retry (max 3) ───────────────────┘

Phase 1: DIAGNOSE

• LLM: Claude Opus 4.6
• Input: CI log excerpt or issue description
• Output: Root cause analysis as structured GitLab comment
• Handoff: Remove phase/diagnose, add phase/plan

Phase 2: PLAN

• LLM: Claude Opus 4.6
• Input: Diagnosis comment from Phase 1
• Output: Fix strategy with affected files, approach, risks
• Handoff: Remove phase/plan, add phase/implement

Phase 3: IMPLEMENT

• LLM: Claude Sonnet 4.6
• Input: Plan comment from Phase 2
• Output: Code changes on agent/fix-{issue_id} branch, pushed to GitLab
• Handoff: Remove phase/implement, add phase/verify

Phase 4: VERIFY

• LLM: Devstral-2-123b (Scaleway Generative APIs)
• Input: Branch name from Phase 3
• Output: CI pipeline triggered, results assessed
• On GREEN: Remove phase/verify, add phase/deliver
• On RED: Add retry/N label, loop to phase/diagnose
• On retry/3: Add agent/blocked, remove agent/active (escalate to human)

Phase 5: DELIVER

• LLM: Devstral-2-123b (Scaleway Generative APIs)
• Input: Green pipeline confirmation
• Output: MR opened via glab mr create with --auto-merge flag (uses GitLab merge_when_pipeline_succeeds API)
• CI-failure triggers: MR auto-merges when MR pipeline passes (no human gate)
• Feature triggers: MR created but requires human approval before merge
• Handoff: Remove agent/active, add agent/done, close issue

GitLab Issue Label Schema

Label	Color	Purpose
agent/active	#28a745	Agent currently working
agent/blocked	#dc3545	Needs human intervention
agent/done	#6c757d	Complete
phase/diagnose	#007bff	Phase 1 active
phase/plan	#6f42c1	Phase 2 active
phase/implement	#fd7e14	Phase 3 active
phase/verify	#ffc107	Phase 4 active
phase/deliver	#28a745	Phase 5 active
trigger/ci-failure	#dc3545	Auto-created from CI
trigger/feature	#17a2b8	Human-requested
retry/1 .. retry/3	#6c757d	Retry counter

Agent Comment Format (Handoff Protocol)

## Phase N: {Phase Name} Complete

**Agent**: {agent_name}
**LLM**: {model_used}
**Duration**: {elapsed}s
**Tokens**: {input_tokens} in / {output_tokens} out

### Findings
{structured findings}

### Handoff
**Next phase**: {next_phase}
**Context for next agent**: {key information the next agent needs}

Human Intervention Points

• Comment on issue: Human comment pauses agent loop
• Remove agent/active: Stops all agent work immediately
• Add agent/blocked: Escalates to human (e.g., Slack notification)
• Close issue: Terminates agent work
• Edit plan comment: Human can modify Phase 2 plan before implementation proceeds

Component Details

1. Dispatcher Service (services/agent_dispatcher/)

Language: Rust (Axum) Pool: ci (GP1-XS) Port: 8080

Responsibilities:

• Receive GitLab webhooks (pipeline, issue label, note events)
• Create issues with structured context on pipeline failure
• Invoke OpenHands headless on label transitions
• Track retry count, enforce max 3
• Concurrency control: max 2 simultaneous agent invocations
• Parse agent comments for handoff metadata

Dependencies:

• axum (HTTP server)
• reqwest (GitLab API client)
• tokio (async runtime, process spawning)
• serde/serde_json (webhook payload parsing)
• tracing (structured logging)

2. LiteLLM Proxy

Image: ghcr.io/berriai/litellm:main-latest Pool: ci (GP1-XS) Port: 4000

model_list:
  - model_name: "opus"
    litellm_params:
      model: "claude-opus-4-6"
      api_key: "os.environ/ANTHROPIC_API_KEY"
  - model_name: "sonnet"
    litellm_params:
      model: "claude-sonnet-4-6"
      api_key: "os.environ/ANTHROPIC_API_KEY"
  - model_name: "devstral"
    litellm_params:
      model: "openai/devstral-2-123b-instruct-2512"
      api_base: "https://api.scaleway.ai/v1"
      api_key: "os.environ/SCW_GENERATIVE_API_KEY"

litellm_settings:
  max_budget: 50.0        # $50/day global cap
  budget_duration: "1d"

3. OpenHands Deployment

Runtime: Kubernetes (native pod scheduling) Namespace: foxhunt-agents Image: ghcr.io/openhands/openhands:latest

# config.toml
[core]
workspace_base = "/tmp/openhands-workspace"
max_iterations = 50

[llm]
model = "opus"
base_url = "http://litellm.foxhunt-agents.svc.cluster.local:4000/v1"
api_key = "via-litellm"

[sandbox]
timeout = 300
enable_gpu = false

[kubernetes]
namespace = "foxhunt-agents"

4. Microagent Definitions (.openhands/microagents/)

5 markdown files in the repo root, each defining phase-specific behavior:

• diagnose.md — Parse CI logs, identify root cause, structured output
• plan.md — Read diagnosis, propose fix, estimate scope
• implement.md — Create branch, write code, run cargo check
• verify.md — Push, trigger CI, poll results
• deliver.md — Open MR, summarize changes, close issue

5. LLM Routing by Phase

Phase	Model	Provider	Cost (per 1M tokens)	Rationale
Diagnose	Opus 4.6	Anthropic API	$15 in / $75 out	Hardest: deep Rust error understanding
Plan	Opus 4.6	Anthropic API	$15 in / $75 out	Architectural judgment, multi-file reasoning
Implement	Sonnet 4.6	Anthropic API	$3 in / $15 out	Good code gen, 5x cheaper than Opus
Verify	Devstral-2-123b	Scaleway	~€0.40 in / €2.00 out	Log parsing, pass/fail, mechanical
Deliver	Devstral-2-123b	Scaleway	~€0.40 in / €2.00 out	MR description, mechanical

Infrastructure (New K8s Resources)

Resource	Pool	Purpose
Namespace foxhunt-agents	—	Isolates agent workloads
Deployment dispatcher	ci	Webhook receiver + agent invoker
Deployment litellm	ci	LLM proxy with cost tracking
Deployment openhands	ci	OpenHands server (K8s runtime)
ServiceAccount openhands-agent	—	RBAC for pod creation
Role + RoleBinding	—	Create/delete pods in foxhunt-agents
Secret agent-credentials	—	ANTHROPIC_API_KEY, SCW_GENERATIVE_API_KEY, GITLAB_TOKEN
ConfigMap openhands-config	—	config.toml for OpenHands
ConfigMap litellm-config	—	LiteLLM model routing config
PVC agent-workspace (10Gi)	ci	Workspace for agent repo checkouts

Incremental cost: All on ci pool (GP1-XS, ~€0.08/hr). No new GPU nodes. LLM cost is pay-per-token.

Security & Guardrails

• Branch protection: Agents push to agent/fix-* branches only, never main
• Auto-merge: Agent MRs auto-merge when CI passes (GitLab merge_when_pipeline_succeeds API). Configurable per trigger type — CI-failure fixes auto-merge by default, feature MRs require human approval
• MR approval bypass: Only for trigger/ci-failure issues. Feature-triggered MRs still require human approval
• Max 3 retries: After 3 failed attempts, agent/blocked label + human notification
• Concurrency limit: Max 2 simultaneous agent invocations (configurable in dispatcher)
• Token scoping: GitLab PAT with api + write_repository, project-scoped
• Cost cap: LiteLLM enforces $50/day global budget
• Audit trail: Every action is a GitLab issue comment — fully observable
• Human override: Remove agent/active label to stop any agent immediately

Incremental Rollout Plan

1. Phase 0: Deploy LiteLLM proxy, verify model routing works
2. Phase 1: Deploy dispatcher, wire up pipeline failure webhook → issue creation
3. Phase 2: Deploy OpenHands, implement diagnose agent only — verify it produces useful comments
4. Phase 3: Add plan + implement agents — verify they produce reasonable code
5. Phase 4: Add verify + deliver agents — close the loop
6. Phase 5: Enable @agent mention trigger for feature work

References

• OpenHands Headless Mode
• OpenHands GitHub
• OpenHands GitLab Integration
• LiteLLM Proxy
• Scaleway Generative APIs
• Claude Code GitLab CI/CD (Official Beta)
• GitLab Webhook Events