# Autonomous CI Agents — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Deploy an autonomous agent system that detects CI failures and @agent mentions, then runs a 5-phase fix loop (Diagnose → Plan → Implement → Verify → Deliver) using OpenHands with LLM routing through LiteLLM.

**Architecture:** Rust Axum dispatcher receives GitLab webhooks, creates/labels issues, and invokes OpenHands headless with phase-specific microagents. LiteLLM proxy routes to Opus 4.6 (diagnose/plan), Sonnet 4.6 (implement), and Scaleway devstral-2-123b (verify/deliver). GitLab issues are the durable bus — labels track state, comments carry handoff context.

**Tech Stack:** Rust (Axum, reqwest, tokio), LiteLLM, OpenHands, GitLab CE API, Kubernetes (Kapsule)

**Design Doc:** `docs/plans/2026-02-27-autonomous-agents-design.md`

---

## Phase 0: LiteLLM Proxy

Deploy LiteLLM on Kapsule ci pool, verify all 3 model routes work.

---

### Task 1: Create K8s namespace and secrets

**Files:**
- Create: `infra/k8s/agents/namespace.yaml`
- Create: `infra/k8s/agents/secrets.yaml`

**Step 1: Write namespace manifest**

```yaml
# infra/k8s/agents/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: foxhunt-agents
  labels:
    app.kubernetes.io/part-of: foxhunt
    app.kubernetes.io/component: agents
```

**Step 2: Write secrets manifest (template — real values via `kubectl create secret`)**

```yaml
# infra/k8s/agents/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: agent-credentials
  namespace: foxhunt-agents
type: Opaque
stringData:
  ANTHROPIC_API_KEY: "REPLACE_VIA_KUBECTL"
  SCW_GENERATIVE_API_KEY: "REPLACE_VIA_KUBECTL"
  GITLAB_TOKEN: "REPLACE_VIA_KUBECTL"
```

**Step 3: Apply namespace**

Run: `kubectl apply -f infra/k8s/agents/namespace.yaml`
Expected: `namespace/foxhunt-agents created`

**Step 4: Create real secret (not from template)**

Run: `kubectl create secret generic agent-credentials -n foxhunt-agents --from-literal=ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" --from-literal=SCW_GENERATIVE_API_KEY="$SCW_GENERATIVE_API_KEY" --from-literal=GITLAB_TOKEN="$GITLAB_TOKEN" --dry-run=client -o yaml | kubectl apply -f -`
Expected: `secret/agent-credentials created`

**Step 5: Commit**

```bash
git add -f infra/k8s/agents/namespace.yaml infra/k8s/agents/secrets.yaml
git commit -m "infra(agents): add foxhunt-agents namespace and secrets template"
```

---

### Task 2: Deploy LiteLLM proxy

**Files:**
- Create: `infra/k8s/agents/litellm-config.yaml`
- Create: `infra/k8s/agents/litellm.yaml`

**Step 1: Write LiteLLM config**

```yaml
# infra/k8s/agents/litellm-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-config
  namespace: foxhunt-agents
data:
  config.yaml: |
    model_list:
      - model_name: "opus"
        litellm_params:
          model: "claude-opus-4-6"
          api_key: "os.environ/ANTHROPIC_API_KEY"
      - model_name: "sonnet"
        litellm_params:
          model: "claude-sonnet-4-6"
          api_key: "os.environ/ANTHROPIC_API_KEY"
      - model_name: "devstral"
        litellm_params:
          model: "openai/devstral-2-123b-instruct-2512"
          api_base: "https://api.scaleway.ai/v1"
          api_key: "os.environ/SCW_GENERATIVE_API_KEY"

    litellm_settings:
      max_budget: 50.0
      budget_duration: "1d"
```

**Step 2: Write LiteLLM deployment + service**

```yaml
# infra/k8s/agents/litellm.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
  namespace: foxhunt-agents
  labels:
    app: litellm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      nodeSelector:
        pool: ci
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:main-latest
          args: ["--config", "/etc/litellm/config.yaml"]
          ports:
            - containerPort: 4000
          envFrom:
            - secretRef:
                name: agent-credentials
          volumeMounts:
            - name: config
              mountPath: /etc/litellm
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 15
            periodSeconds: 20
      volumes:
        - name: config
          configMap:
            name: litellm-config
---
apiVersion: v1
kind: Service
metadata:
  name: litellm
  namespace: foxhunt-agents
spec:
  selector:
    app: litellm
  ports:
    - port: 4000
      targetPort: 4000
```

**Step 3: Apply**

Run: `kubectl apply -f infra/k8s/agents/litellm-config.yaml -f infra/k8s/agents/litellm.yaml`
Expected: `configmap/litellm-config created`, `deployment.apps/litellm created`, `service/litellm created`

**Step 4: Wait for ready**

Run: `kubectl rollout status deployment/litellm -n foxhunt-agents --timeout=120s`
Expected: `deployment "litellm" successfully rolled out`

**Step 5: Commit**

```bash
git add -f infra/k8s/agents/litellm-config.yaml infra/k8s/agents/litellm.yaml
git commit -m "infra(agents): deploy LiteLLM proxy with 3-model routing"
```

---

### Task 3: Smoke-test LiteLLM model routing

**Step 1: Port-forward LiteLLM**

Run: `kubectl port-forward -n foxhunt-agents svc/litellm 4000:4000 &`

**Step 2: Test Opus route**

Run:
```bash
curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"opus","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5}' \
  | jq '.choices[0].message.content'
```
Expected: A short response (confirms Anthropic API key works)

**Step 3: Test Sonnet route**

Run:
```bash
curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"sonnet","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5}' \
  | jq '.choices[0].message.content'
```
Expected: A short response

**Step 4: Test Devstral route**

Run:
```bash
curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"devstral","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5}' \
  | jq '.choices[0].message.content'
```
Expected: A short response (confirms SCW Generative API key works)

**Step 5: Kill port-forward**

Run: `kill %1`

No commit needed — this is a manual verification step.

---

## Phase 1: Rust Dispatcher — Webhook Receiver + Issue Creator

Scaffold the Axum service that receives GitLab webhooks and creates issues.

---

### Task 4: Scaffold dispatcher crate

**Files:**
- Create: `services/agent_dispatcher/Cargo.toml`
- Create: `services/agent_dispatcher/src/main.rs`
- Modify: `Cargo.toml` (root — add workspace member)

**Step 1: Create Cargo.toml**

```toml
# services/agent_dispatcher/Cargo.toml
[package]
name = "agent_dispatcher"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
authors.workspace = true
license.workspace = true

[[bin]]
name = "agent-dispatcher"
path = "src/main.rs"

[dependencies]
axum.workspace = true
tokio.workspace = true
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
reqwest = { workspace = true, features = ["json"] }
tracing.workspace = true
tracing-subscriber.workspace = true
tower-http = { version = "0.6", features = ["cors", "trace"] }
hmac = "0.12"
sha2 = "0.10"
hex = "0.4"
```

**Step 2: Add to workspace members in root Cargo.toml**

Add `"services/agent_dispatcher"` to the `members` list.

**Step 3: Write minimal main.rs with health endpoint**

```rust
// services/agent_dispatcher/src/main.rs
use axum::{Router, routing::get};
use tracing_subscriber::EnvFilter;

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_env_filter(EnvFilter::from_default_env())
        .init();

    let app = Router::new()
        .route("/health", get(|| async { "ok" }));

    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080")
        .await
        .expect("failed to bind port 8080");

    tracing::info!("dispatcher listening on :8080");
    axum::serve(listener, app).await.expect("server error");
}
```

**Step 4: Verify it compiles**

Run: `SQLX_OFFLINE=true cargo check -p agent_dispatcher`
Expected: zero errors

**Step 5: Commit**

```bash
git add services/agent_dispatcher/Cargo.toml services/agent_dispatcher/src/main.rs Cargo.toml
git commit -m "feat(agent-dispatcher): scaffold Axum service with health endpoint"
```

---

### Task 5: GitLab webhook types and signature verification

**Files:**
- Create: `services/agent_dispatcher/src/gitlab.rs`
- Create: `services/agent_dispatcher/src/webhook.rs`

**Step 1: Write the failing test (webhook signature verification)**

```rust
// services/agent_dispatcher/src/webhook.rs
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_verify_valid_signature() {
        let secret = "test-secret";
        let body = b"hello world";
        // Pre-computed HMAC-SHA256 of "hello world" with key "test-secret"
        let sig = compute_hmac(secret.as_bytes(), body);
        assert!(verify_signature(secret, &sig, body));
    }

    #[test]
    fn test_reject_invalid_signature() {
        assert!(!verify_signature("secret", "bad-sig", b"body"));
    }
}
```

**Step 2: Run test to verify it fails**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- webhook`
Expected: FAIL (functions not defined)

**Step 3: Implement webhook signature verification and GitLab types**

`webhook.rs` — HMAC-SHA256 verification for `X-Gitlab-Token` header:

```rust
// services/agent_dispatcher/src/webhook.rs
use hmac::{Hmac, Mac};
use sha2::Sha256;

type HmacSha256 = Hmac<Sha256>;

pub fn compute_hmac(key: &[u8], body: &[u8]) -> String {
    let mut mac = HmacSha256::new_from_slice(key).expect("HMAC key");
    mac.update(body);
    hex::encode(mac.finalize().into_bytes())
}

pub fn verify_signature(secret: &str, signature: &str, body: &[u8]) -> bool {
    let expected = compute_hmac(secret.as_bytes(), body);
    // Constant-time comparison
    expected.len() == signature.len()
        && expected
            .bytes()
            .zip(signature.bytes())
            .fold(0u8, |acc, (a, b)| acc | (a ^ b))
            == 0
}
```

`gitlab.rs` — webhook payload types:

```rust
// services/agent_dispatcher/src/gitlab.rs
use serde::Deserialize;

#[derive(Debug, Deserialize)]
pub struct PipelineWebhook {
    pub object_kind: String,
    pub object_attributes: PipelineAttributes,
    pub project: Project,
    pub builds: Option<Vec<Build>>,
}

#[derive(Debug, Deserialize)]
pub struct PipelineAttributes {
    pub id: u64,
    pub status: String,
    #[serde(rename = "ref")]
    pub ref_name: String,
    pub sha: String,
}

#[derive(Debug, Deserialize)]
pub struct NoteWebhook {
    pub object_kind: String,
    pub object_attributes: NoteAttributes,
    pub project: Project,
    pub issue: Option<IssueRef>,
}

#[derive(Debug, Deserialize)]
pub struct NoteAttributes {
    pub id: u64,
    pub note: String,
    pub noteable_type: String,
}

#[derive(Debug, Deserialize)]
pub struct Project {
    pub id: u64,
    pub name: String,
    pub web_url: String,
}

#[derive(Debug, Deserialize)]
pub struct Build {
    pub id: u64,
    pub name: String,
    pub status: String,
    pub stage: String,
}

#[derive(Debug, Deserialize)]
pub struct IssueRef {
    pub iid: u64,
    pub title: String,
}
```

**Step 4: Run tests to verify they pass**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- webhook`
Expected: 2 tests PASS

**Step 5: Commit**

```bash
git add services/agent_dispatcher/src/webhook.rs services/agent_dispatcher/src/gitlab.rs
git commit -m "feat(agent-dispatcher): webhook signature verification + GitLab types"
```

---

### Task 6: GitLab API client

**Files:**
- Create: `services/agent_dispatcher/src/gitlab_client.rs`

**Step 1: Write the failing test**

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_client_builds_issue_body() {
        let body = format_ci_failure_issue(
            "check",
            "cargo check --workspace",
            "error[E0433]: failed to resolve",
            "abc1234",
            "main",
        );
        assert!(body.contains("## CI Failure"));
        assert!(body.contains("error[E0433]"));
        assert!(body.contains("abc1234"));
    }
}
```

**Step 2: Run test to verify it fails**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- gitlab_client`
Expected: FAIL

**Step 3: Implement GitLab API client**

```rust
// services/agent_dispatcher/src/gitlab_client.rs
use reqwest::Client;
use serde::{Deserialize, Serialize};

#[derive(Clone)]
pub struct GitLabClient {
    client: Client,
    base_url: String,
    token: String,
    project_id: u64,
}

#[derive(Serialize)]
struct CreateIssueRequest {
    title: String,
    description: String,
    labels: String,
}

#[derive(Serialize)]
struct CreateCommentRequest {
    body: String,
}

#[derive(Serialize)]
struct UpdateLabelsRequest {
    labels: String,
}

#[derive(Deserialize)]
pub struct Issue {
    pub iid: u64,
    pub title: String,
    pub labels: Vec<String>,
}

impl GitLabClient {
    pub fn new(base_url: String, token: String, project_id: u64) -> Self {
        Self {
            client: Client::new(),
            base_url,
            token,
            project_id,
        }
    }

    pub async fn fetch_job_log(&self, job_id: u64) -> Result<String, reqwest::Error> {
        let url = format!(
            "{}/api/v4/projects/{}/jobs/{}/trace",
            self.base_url, self.project_id, job_id
        );
        self.client
            .get(&url)
            .header("PRIVATE-TOKEN", &self.token)
            .send()
            .await?
            .text()
            .await
    }

    pub async fn create_issue(
        &self,
        title: &str,
        description: &str,
        labels: &[&str],
    ) -> Result<Issue, reqwest::Error> {
        let url = format!(
            "{}/api/v4/projects/{}/issues",
            self.base_url, self.project_id
        );
        self.client
            .post(&url)
            .header("PRIVATE-TOKEN", &self.token)
            .json(&CreateIssueRequest {
                title: title.to_string(),
                description: description.to_string(),
                labels: labels.join(","),
            })
            .send()
            .await?
            .json()
            .await
    }

    pub async fn add_comment(
        &self,
        issue_iid: u64,
        body: &str,
    ) -> Result<(), reqwest::Error> {
        let url = format!(
            "{}/api/v4/projects/{}/issues/{}/notes",
            self.base_url, self.project_id, issue_iid
        );
        self.client
            .post(&url)
            .header("PRIVATE-TOKEN", &self.token)
            .json(&CreateCommentRequest {
                body: body.to_string(),
            })
            .send()
            .await?
            .error_for_status()?;
        Ok(())
    }

    pub async fn update_labels(
        &self,
        issue_iid: u64,
        labels: &[&str],
    ) -> Result<(), reqwest::Error> {
        let url = format!(
            "{}/api/v4/projects/{}/issues/{}",
            self.base_url, self.project_id, issue_iid
        );
        self.client
            .put(&url)
            .header("PRIVATE-TOKEN", &self.token)
            .json(&UpdateLabelsRequest {
                labels: labels.join(","),
            })
            .send()
            .await?
            .error_for_status()?;
        Ok(())
    }

    pub async fn get_issue_labels(&self, issue_iid: u64) -> Result<Vec<String>, reqwest::Error> {
        let url = format!(
            "{}/api/v4/projects/{}/issues/{}",
            self.base_url, self.project_id, issue_iid
        );
        let issue: Issue = self.client
            .get(&url)
            .header("PRIVATE-TOKEN", &self.token)
            .send()
            .await?
            .json()
            .await?;
        Ok(issue.labels)
    }
}

pub fn format_ci_failure_issue(
    stage: &str,
    job_name: &str,
    log_excerpt: &str,
    sha: &str,
    ref_name: &str,
) -> String {
    // Truncate log to last 200 lines if needed
    let lines: Vec<&str> = log_excerpt.lines().collect();
    let truncated = if lines.len() > 200 {
        lines[lines.len() - 200..].join("\n")
    } else {
        log_excerpt.to_string()
    };

    format!(
        "## CI Failure: `{job_name}` in stage `{stage}`\n\n\
         **Commit**: `{sha}`\n\
         **Branch**: `{ref_name}`\n\n\
         ### Log Excerpt\n\n\
         ```\n{truncated}\n```\n\n\
         ---\n\
         *Auto-created by agent dispatcher. Agent will diagnose shortly.*"
    )
}
```

**Step 4: Run test to verify it passes**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- gitlab_client`
Expected: PASS

**Step 5: Commit**

```bash
git add services/agent_dispatcher/src/gitlab_client.rs
git commit -m "feat(agent-dispatcher): GitLab API client (issues, comments, labels, job logs)"
```

---

### Task 7: Webhook handler + pipeline failure flow

**Files:**
- Modify: `services/agent_dispatcher/src/main.rs`
- Create: `services/agent_dispatcher/src/config.rs`
- Create: `services/agent_dispatcher/src/handler.rs`

**Step 1: Write config module**

```rust
// services/agent_dispatcher/src/config.rs
#[derive(Clone)]
pub struct AppConfig {
    pub gitlab_base_url: String,
    pub gitlab_token: String,
    pub gitlab_project_id: u64,
    pub webhook_secret: String,
    pub max_concurrent_agents: usize,
    pub max_retries: u8,
}

impl AppConfig {
    pub fn from_env() -> Self {
        Self {
            gitlab_base_url: std::env::var("GITLAB_BASE_URL")
                .unwrap_or_else(|_| "https://git.fxhnt.ai".to_string()),
            gitlab_token: std::env::var("GITLAB_TOKEN")
                .expect("GITLAB_TOKEN required"),
            gitlab_project_id: std::env::var("GITLAB_PROJECT_ID")
                .expect("GITLAB_PROJECT_ID required")
                .parse()
                .expect("GITLAB_PROJECT_ID must be u64"),
            webhook_secret: std::env::var("WEBHOOK_SECRET")
                .unwrap_or_default(),
            max_concurrent_agents: std::env::var("MAX_CONCURRENT_AGENTS")
                .unwrap_or_else(|_| "2".to_string())
                .parse()
                .unwrap_or(2),
            max_retries: 3,
        }
    }
}
```

**Step 2: Write the failing test (handler routes webhook to correct flow)**

```rust
// services/agent_dispatcher/src/handler.rs
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_extract_failed_jobs() {
        let builds = vec![
            Build { id: 1, name: "check".to_string(), status: "success".to_string(), stage: "check".to_string() },
            Build { id: 2, name: "test-ml".to_string(), status: "failed".to_string(), stage: "test".to_string() },
            Build { id: 3, name: "test-risk".to_string(), status: "failed".to_string(), stage: "test".to_string() },
        ];
        let failed = extract_failed_jobs(&builds);
        assert_eq!(failed.len(), 2);
        assert_eq!(failed[0].name, "test-ml");
    }

    #[test]
    fn test_detect_agent_mention() {
        assert!(contains_agent_mention("Please fix this @agent"));
        assert!(contains_agent_mention("@agent can you look at this?"));
        assert!(!contains_agent_mention("No mention here"));
    }
}
```

**Step 3: Run tests to verify they fail**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- handler`
Expected: FAIL

**Step 4: Implement handler**

```rust
// services/agent_dispatcher/src/handler.rs
use axum::{
    body::Bytes,
    extract::State,
    http::{HeaderMap, StatusCode},
    response::IntoResponse,
};
use std::sync::Arc;
use tokio::sync::Semaphore;

use crate::config::AppConfig;
use crate::gitlab::{Build, NoteWebhook, PipelineWebhook};
use crate::gitlab_client::{format_ci_failure_issue, GitLabClient};
use crate::webhook::verify_signature;

pub struct AppState {
    pub config: AppConfig,
    pub gitlab: GitLabClient,
    pub concurrency: Arc<Semaphore>,
}

pub fn extract_failed_jobs(builds: &[Build]) -> Vec<&Build> {
    builds.iter().filter(|b| b.status == "failed").collect()
}

pub fn contains_agent_mention(text: &str) -> bool {
    text.contains("@agent")
}

pub async fn handle_webhook(
    State(state): State<Arc<AppState>>,
    headers: HeaderMap,
    body: Bytes,
) -> impl IntoResponse {
    // Verify signature if secret is configured
    if !state.config.webhook_secret.is_empty() {
        let sig = headers
            .get("X-Gitlab-Token")
            .and_then(|v| v.to_str().ok())
            .unwrap_or("");
        if sig != state.config.webhook_secret {
            return StatusCode::UNAUTHORIZED;
        }
    }

    let event = headers
        .get("X-Gitlab-Event")
        .and_then(|v| v.to_str().ok())
        .unwrap_or("");

    match event {
        "Pipeline Hook" => {
            if let Ok(payload) = serde_json::from_slice::<PipelineWebhook>(&body) {
                if payload.object_attributes.status == "failed" {
                    let state = Arc::clone(&state);
                    tokio::spawn(async move {
                        if let Err(e) = handle_pipeline_failure(state, payload).await {
                            tracing::error!("pipeline failure handler error: {e}");
                        }
                    });
                }
            }
            StatusCode::OK
        }
        "Note Hook" => {
            if let Ok(payload) = serde_json::from_slice::<NoteWebhook>(&body) {
                if contains_agent_mention(&payload.object_attributes.note) {
                    let state = Arc::clone(&state);
                    tokio::spawn(async move {
                        if let Err(e) = handle_agent_mention(state, payload).await {
                            tracing::error!("agent mention handler error: {e}");
                        }
                    });
                }
            }
            StatusCode::OK
        }
        _ => StatusCode::OK,
    }
}

async fn handle_pipeline_failure(
    state: Arc<AppState>,
    payload: PipelineWebhook,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let failed = payload.builds.as_deref().map(extract_failed_jobs).unwrap_or_default();
    if failed.is_empty() {
        return Ok(());
    }

    // Fetch log from first failed job
    let job = failed[0];
    let log = state.gitlab.fetch_job_log(job.id).await.unwrap_or_default();

    let title = format!(
        "[CI] {} failed on {}",
        job.name, payload.object_attributes.ref_name
    );
    let body = format_ci_failure_issue(
        &job.stage,
        &job.name,
        &log,
        &payload.object_attributes.sha,
        &payload.object_attributes.ref_name,
    );

    let labels = &["trigger/ci-failure", "phase/diagnose", "agent/active"];
    let issue = state.gitlab.create_issue(&title, &body, labels).await?;

    tracing::info!(
        issue_iid = issue.iid,
        job_name = job.name,
        "created agent issue for CI failure"
    );

    // TODO Phase 2: invoke OpenHands diagnose agent here
    Ok(())
}

async fn handle_agent_mention(
    state: Arc<AppState>,
    payload: NoteWebhook,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let issue = match payload.issue {
        Some(issue) => issue,
        None => return Ok(()), // Only handle issue comments
    };

    let labels = &["trigger/feature", "phase/diagnose", "agent/active"];
    state.gitlab.update_labels(issue.iid, labels).await?;

    tracing::info!(
        issue_iid = issue.iid,
        "activated agent for @agent mention"
    );

    // TODO Phase 2: invoke OpenHands diagnose agent here
    Ok(())
}
```

**Step 5: Wire up main.rs**

```rust
// services/agent_dispatcher/src/main.rs
mod config;
mod gitlab;
mod gitlab_client;
mod handler;
mod webhook;

use std::sync::Arc;

use axum::{Router, routing::{get, post}};
use tokio::sync::Semaphore;
use tracing_subscriber::EnvFilter;

use config::AppConfig;
use gitlab_client::GitLabClient;
use handler::{AppState, handle_webhook};

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_env_filter(EnvFilter::from_default_env())
        .init();

    let config = AppConfig::from_env();
    let gitlab = GitLabClient::new(
        config.gitlab_base_url.clone(),
        config.gitlab_token.clone(),
        config.gitlab_project_id,
    );
    let concurrency = Arc::new(Semaphore::new(config.max_concurrent_agents));

    let state = Arc::new(AppState {
        config,
        gitlab,
        concurrency,
    });

    let app = Router::new()
        .route("/health", get(|| async { "ok" }))
        .route("/webhook", post(handle_webhook))
        .with_state(state);

    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080")
        .await
        .expect("failed to bind port 8080");

    tracing::info!("dispatcher listening on :8080");
    axum::serve(listener, app).await.expect("server error");
}
```

**Step 6: Run tests**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib`
Expected: all tests PASS

**Step 7: Run check**

Run: `SQLX_OFFLINE=true cargo check -p agent_dispatcher`
Expected: zero errors

**Step 8: Commit**

```bash
git add services/agent_dispatcher/src/
git commit -m "feat(agent-dispatcher): webhook handler with pipeline failure → issue creation"
```

---

### Task 8: Deploy dispatcher to Kapsule

**Files:**
- Create: `infra/k8s/agents/dispatcher.yaml`

**Step 1: Write deployment manifest**

```yaml
# infra/k8s/agents/dispatcher.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dispatcher
  namespace: foxhunt-agents
  labels:
    app: dispatcher
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dispatcher
  template:
    metadata:
      labels:
        app: dispatcher
    spec:
      nodeSelector:
        pool: ci
      containers:
        - name: dispatcher
          image: rg.fr-par.scw.cloud/foxhunt-ci/agent-dispatcher:latest
          ports:
            - containerPort: 8080
          env:
            - name: GITLAB_BASE_URL
              value: "https://git.fxhnt.ai"
            - name: GITLAB_PROJECT_ID
              value: "1"  # Adjust to actual project ID
            - name: WEBHOOK_SECRET
              valueFrom:
                secretKeyRef:
                  name: agent-credentials
                  key: GITLAB_TOKEN
            - name: MAX_CONCURRENT_AGENTS
              value: "2"
            - name: RUST_LOG
              value: "info"
          envFrom:
            - secretRef:
                name: agent-credentials
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
            limits:
              cpu: "200m"
              memory: "128Mi"
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: dispatcher
  namespace: foxhunt-agents
spec:
  selector:
    app: dispatcher
  ports:
    - port: 8080
      targetPort: 8080
```

**Step 2: Create Dockerfile**

```dockerfile
# infra/docker/Dockerfile.agent-dispatcher
FROM rust:1.89-bookworm AS builder
RUN apt-get update && apt-get install -y mold clang
WORKDIR /build
COPY . .
RUN SQLX_OFFLINE=true cargo build --release -p agent_dispatcher
RUN strip target/release/agent-dispatcher

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
COPY --from=builder /build/target/release/agent-dispatcher /usr/local/bin/
ENTRYPOINT ["agent-dispatcher"]
```

**Step 3: Commit**

```bash
git add -f infra/k8s/agents/dispatcher.yaml infra/docker/Dockerfile.agent-dispatcher
git commit -m "infra(agents): dispatcher deployment manifest + Dockerfile"
```

**Step 4: Build and push (manual)**

Run: `docker build -f infra/docker/Dockerfile.agent-dispatcher -t rg.fr-par.scw.cloud/foxhunt-ci/agent-dispatcher:latest . && docker push rg.fr-par.scw.cloud/foxhunt-ci/agent-dispatcher:latest`

**Step 5: Deploy**

Run: `kubectl apply -f infra/k8s/agents/dispatcher.yaml`

---

### Task 9: Configure GitLab webhook

**Step 1: Create labels via GitLab API**

Run:
```bash
for label in "agent/active:#28a745" "agent/blocked:#dc3545" "agent/done:#6c757d" \
  "phase/diagnose:#007bff" "phase/plan:#6f42c1" "phase/implement:#fd7e14" \
  "phase/verify:#ffc107" "phase/deliver:#28a745" \
  "trigger/ci-failure:#dc3545" "trigger/feature:#17a2b8" \
  "retry/1:#6c757d" "retry/2:#6c757d" "retry/3:#6c757d"; do
  name="${label%%:*}"
  color="${label#*:}"
  curl -s --request POST "https://git.fxhnt.ai/api/v4/projects/1/labels" \
    --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
    --data "name=$name&color=$color"
done
```

**Step 2: Register webhook in GitLab**

Run:
```bash
curl -s --request POST "https://git.fxhnt.ai/api/v4/projects/1/hooks" \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --data "url=http://dispatcher.foxhunt-agents.svc.cluster.local:8080/webhook" \
  --data "pipeline_events=true" \
  --data "note_events=true" \
  --data "token=$WEBHOOK_SECRET"
```

Expected: 201 Created with hook ID

No commit — this is GitLab configuration.

---

## Phase 2: OpenHands + Diagnose Agent

Deploy OpenHands with K8s runtime and implement the first microagent.

---

### Task 10: OpenHands RBAC and config

**Files:**
- Create: `infra/k8s/agents/openhands-rbac.yaml`
- Create: `infra/k8s/agents/openhands-config.yaml`

**Step 1: Write RBAC for OpenHands pod scheduling**

```yaml
# infra/k8s/agents/openhands-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: openhands-agent
  namespace: foxhunt-agents
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: openhands-pod-manager
  namespace: foxhunt-agents
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/exec"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: openhands-pod-manager
  namespace: foxhunt-agents
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: openhands-pod-manager
subjects:
  - kind: ServiceAccount
    name: openhands-agent
    namespace: foxhunt-agents
```

**Step 2: Write OpenHands config**

```yaml
# infra/k8s/agents/openhands-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: openhands-config
  namespace: foxhunt-agents
data:
  config.toml: |
    [core]
    workspace_base = "/tmp/openhands-workspace"
    max_iterations = 50

    [llm]
    model = "opus"
    base_url = "http://litellm.foxhunt-agents.svc.cluster.local:4000/v1"
    api_key = "via-litellm"

    [sandbox]
    timeout = 300
    enable_gpu = false

    [kubernetes]
    namespace = "foxhunt-agents"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: agent-workspace
  namespace: foxhunt-agents
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: scw-bssd
  resources:
    requests:
      storage: 10Gi
```

**Step 3: Apply**

Run: `kubectl apply -f infra/k8s/agents/openhands-rbac.yaml -f infra/k8s/agents/openhands-config.yaml`

**Step 4: Commit**

```bash
git add -f infra/k8s/agents/openhands-rbac.yaml infra/k8s/agents/openhands-config.yaml
git commit -m "infra(agents): OpenHands RBAC, config, and workspace PVC"
```

---

### Task 11: Deploy OpenHands server

**Files:**
- Create: `infra/k8s/agents/openhands.yaml`

**Step 1: Write OpenHands deployment**

```yaml
# infra/k8s/agents/openhands.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openhands
  namespace: foxhunt-agents
  labels:
    app: openhands
spec:
  replicas: 1
  selector:
    matchLabels:
      app: openhands
  template:
    metadata:
      labels:
        app: openhands
    spec:
      serviceAccountName: openhands-agent
      nodeSelector:
        pool: ci
      containers:
        - name: openhands
          image: ghcr.io/openhands/openhands:latest
          ports:
            - containerPort: 3000
          env:
            - name: LLM_BASE_URL
              value: "http://litellm.foxhunt-agents.svc.cluster.local:4000/v1"
            - name: LLM_API_KEY
              value: "via-litellm"
            - name: LLM_MODEL
              value: "opus"
            - name: SANDBOX_RUNTIME_CONTAINER_IMAGE
              value: "ghcr.io/openhands/runtime:latest"
          envFrom:
            - secretRef:
                name: agent-credentials
          volumeMounts:
            - name: config
              mountPath: /app/config.toml
              subPath: config.toml
            - name: workspace
              mountPath: /tmp/openhands-workspace
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "4Gi"
      volumes:
        - name: config
          configMap:
            name: openhands-config
        - name: workspace
          persistentVolumeClaim:
            claimName: agent-workspace
---
apiVersion: v1
kind: Service
metadata:
  name: openhands
  namespace: foxhunt-agents
spec:
  selector:
    app: openhands
  ports:
    - port: 3000
      targetPort: 3000
```

**Step 2: Apply**

Run: `kubectl apply -f infra/k8s/agents/openhands.yaml`

**Step 3: Wait for ready**

Run: `kubectl rollout status deployment/openhands -n foxhunt-agents --timeout=180s`

**Step 4: Commit**

```bash
git add -f infra/k8s/agents/openhands.yaml
git commit -m "infra(agents): deploy OpenHands server with K8s runtime"
```

---

### Task 12: Write diagnose microagent

**Files:**
- Create: `.openhands/microagents/diagnose.md`

**Step 1: Write microagent definition**

```markdown
<!-- .openhands/microagents/diagnose.md -->
---
name: diagnose
agent: CodeActAgent
---

# Diagnose Agent

You are a Rust CI diagnostician for the foxhunt HFT trading system.

## Your Task

Analyze the CI failure described in this GitLab issue and produce a root cause analysis.

## Context

- This is a Rust workspace with 30+ crates, 4500+ tests
- Build uses `SQLX_OFFLINE=true cargo check --workspace`
- Tests use `SQLX_OFFLINE=true cargo test -p <crate> --lib`
- Clippy: `SQLX_OFFLINE=true cargo clippy --workspace`
- Crate deny rules: `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)]`
- Linker: clang+mold

## Process

1. Read the issue description to understand the failure
2. Clone the repo if not already available: `git clone ssh://git@100.90.76.85:2222/root/foxhunt.git`
3. Identify the failing stage and job from the CI log excerpt
4. Reproduce the error locally: run the exact command that failed
5. Trace the error to its root cause (file, line, reason)
6. Check recent commits with `git log --oneline -10` for likely culprits

## Output Format

Post a GitLab comment with this exact structure:

```
## Phase 1: Diagnose Complete

**Agent**: diagnose
**LLM**: opus
**Duration**: {elapsed}s
**Tokens**: {input_tokens} in / {output_tokens} out

### Root Cause
{1-2 sentence summary}

### Details
- **Failing stage**: {stage name}
- **Failing job**: {job name}
- **Error type**: {compilation|test|clippy|link}
- **File(s)**: {file paths with line numbers}
- **Cause**: {detailed explanation}
- **Likely culprit commit**: {short sha + message}

### Handoff
**Next phase**: plan
**Context for next agent**: {key facts the plan agent needs — affected files, error type, whether it's a simple typo or architectural issue}
```

## Constraints

- Do NOT attempt to fix the code — only diagnose
- Focus on precision: identify the exact file and line
- If multiple errors, identify the root one (others may cascade)
- If the issue is a feature request (not CI failure), summarize the requirements instead
```

**Step 2: Commit**

```bash
git add -f .openhands/microagents/diagnose.md
git commit -m "feat(agents): add diagnose microagent definition"
```

---

### Task 13: Add OpenHands invocation to dispatcher

**Files:**
- Create: `services/agent_dispatcher/src/openhands.rs`
- Modify: `services/agent_dispatcher/src/handler.rs`

**Step 1: Write the failing test**

```rust
// services/agent_dispatcher/src/openhands.rs
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_build_headless_task() {
        let task = build_task(
            Phase::Diagnose,
            42,
            "CI failure in test-ml",
        );
        assert!(task.contains("issue #42"));
        assert!(task.contains("diagnose"));
    }

    #[test]
    fn test_phase_to_model() {
        assert_eq!(phase_model(&Phase::Diagnose), "opus");
        assert_eq!(phase_model(&Phase::Plan), "opus");
        assert_eq!(phase_model(&Phase::Implement), "sonnet");
        assert_eq!(phase_model(&Phase::Verify), "devstral");
        assert_eq!(phase_model(&Phase::Deliver), "devstral");
    }
}
```

**Step 2: Run test to verify it fails**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- openhands`
Expected: FAIL

**Step 3: Implement OpenHands invocation module**

```rust
// services/agent_dispatcher/src/openhands.rs
use reqwest::Client;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Phase {
    Diagnose,
    Plan,
    Implement,
    Verify,
    Deliver,
}

impl Phase {
    pub fn label(&self) -> &'static str {
        match self {
            Phase::Diagnose => "phase/diagnose",
            Phase::Plan => "phase/plan",
            Phase::Implement => "phase/implement",
            Phase::Verify => "phase/verify",
            Phase::Deliver => "phase/deliver",
        }
    }

    pub fn next(&self) -> Option<Phase> {
        match self {
            Phase::Diagnose => Some(Phase::Plan),
            Phase::Plan => Some(Phase::Implement),
            Phase::Implement => Some(Phase::Verify),
            Phase::Verify => Some(Phase::Deliver),
            Phase::Deliver => None,
        }
    }
}

pub fn phase_model(phase: &Phase) -> &'static str {
    match phase {
        Phase::Diagnose | Phase::Plan => "opus",
        Phase::Implement => "sonnet",
        Phase::Verify | Phase::Deliver => "devstral",
    }
}

pub fn build_task(phase: Phase, issue_iid: u64, context: &str) -> String {
    let phase_name = match phase {
        Phase::Diagnose => "diagnose",
        Phase::Plan => "plan",
        Phase::Implement => "implement",
        Phase::Verify => "verify",
        Phase::Deliver => "deliver",
    };
    format!(
        "You are the {phase_name} agent for foxhunt issue #{issue_iid}.\n\n\
         Follow the instructions in .openhands/microagents/{phase_name}.md\n\n\
         ## Issue Context\n\n{context}"
    )
}

#[derive(Serialize)]
struct HeadlessRequest {
    task: String,
    model: String,
    #[serde(skip_serializing_if = "Option::is_none")]
    max_iterations: Option<u32>,
}

#[derive(Deserialize)]
pub struct HeadlessResponse {
    pub status: String,
    pub output: Option<String>,
}

pub struct OpenHandsClient {
    client: Client,
    base_url: String,
}

impl OpenHandsClient {
    pub fn new(base_url: String) -> Self {
        Self {
            client: Client::new(),
            base_url,
        }
    }

    pub async fn invoke(
        &self,
        phase: Phase,
        issue_iid: u64,
        context: &str,
    ) -> Result<HeadlessResponse, reqwest::Error> {
        let task = build_task(phase, issue_iid, context);
        let model = phase_model(&phase);

        self.client
            .post(format!("{}/api/submit-task", self.base_url))
            .json(&HeadlessRequest {
                task,
                model: model.to_string(),
                max_iterations: Some(50),
            })
            .send()
            .await?
            .json()
            .await
    }
}
```

**Step 4: Run tests**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- openhands`
Expected: PASS

**Step 5: Wire invocation into handler (replace TODO comments)**

In `handler.rs`, add the OpenHands invocation call at the `// TODO Phase 2` comment sites.

**Step 6: Run full crate tests**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib`
Expected: all PASS

**Step 7: Commit**

```bash
git add services/agent_dispatcher/src/openhands.rs services/agent_dispatcher/src/handler.rs
git commit -m "feat(agent-dispatcher): OpenHands invocation with phase-based model routing"
```

---

## Phase 3: Plan + Implement Agents

Add the plan and implement microagents so the loop can produce code.

---

### Task 14: Write plan microagent

**Files:**
- Create: `.openhands/microagents/plan.md`

**Step 1: Write microagent**

```markdown
<!-- .openhands/microagents/plan.md -->
---
name: plan
agent: CodeActAgent
---

# Plan Agent

You are a Rust architecture planner for the foxhunt HFT trading system.

## Your Task

Read the diagnosis from Phase 1 and create a fix strategy.

## Process

1. Read the most recent "Phase 1: Diagnose Complete" comment on this issue
2. Examine the identified file(s) in the codebase
3. Assess scope: is this a 1-file fix or multi-file change?
4. Check for related tests that need updating
5. Identify risks (breaking changes, cascading effects)

## Output Format

Post a GitLab comment:

```
## Phase 2: Plan Complete

**Agent**: plan
**LLM**: opus
**Duration**: {elapsed}s
**Tokens**: {input_tokens} in / {output_tokens} out

### Fix Strategy
{1-2 sentence summary}

### Changes
| File | Action | Description |
|------|--------|-------------|
| `path/to/file.rs:42` | Modify | {what to change} |
| `path/to/test.rs` | Modify | {update test expectations} |

### Approach
{step-by-step implementation guide, 3-7 steps}

### Risks
- {risk 1 and mitigation}

### Handoff
**Next phase**: implement
**Context for next agent**: {branch name to create, exact changes to make, test commands to run}
```

## Constraints

- Do NOT write code — only plan
- Be specific: exact file paths and line numbers
- Consider the `#![deny(clippy::unwrap_used, ...)]` rules
- Verify your proposed changes won't break other crates
```

**Step 2: Commit**

```bash
git add -f .openhands/microagents/plan.md
git commit -m "feat(agents): add plan microagent definition"
```

---

### Task 15: Write implement microagent

**Files:**
- Create: `.openhands/microagents/implement.md`

**Step 1: Write microagent**

```markdown
<!-- .openhands/microagents/implement.md -->
---
name: implement
agent: CodeActAgent
---

# Implement Agent

You are a Rust implementation agent for the foxhunt HFT trading system.

## Your Task

Read the plan from Phase 2 and execute the code changes.

## Process

1. Read the most recent "Phase 2: Plan Complete" comment on this issue
2. Create branch: `git checkout -b agent/fix-{issue_iid}`
3. Make the planned changes, following the exact file/line references
4. Run `SQLX_OFFLINE=true cargo check --workspace` — fix any errors
5. Run `SQLX_OFFLINE=true cargo clippy --workspace` — fix any warnings
6. Run relevant tests: `SQLX_OFFLINE=true cargo test -p {crate} --lib`
7. Commit with message: `fix: {description} (agent #{issue_iid})`
8. Push: `git push origin agent/fix-{issue_iid}`

## Rules

- NEVER use `.unwrap()`, `.expect()`, `.panic!()`, or `[index]` — use `.get()`, `?`, `.ok_or()`
- NEVER push to `main` — only `agent/fix-*` branches
- Keep changes minimal — fix the issue, nothing more
- If `cargo check` fails after your changes, fix it before pushing
- If the plan is unclear, post a comment asking for clarification and stop

## Output Format

Post a GitLab comment:

```
## Phase 3: Implement Complete

**Agent**: implement
**LLM**: sonnet
**Duration**: {elapsed}s
**Tokens**: {input_tokens} in / {output_tokens} out

### Changes Made
| File | Lines Changed | Description |
|------|--------------|-------------|
| `path/to/file.rs` | +5/-2 | {what changed} |

### Verification
- `cargo check`: ✅ PASS
- `cargo clippy`: ✅ PASS
- `cargo test -p {crate}`: ✅ PASS ({N} tests)

### Branch
`agent/fix-{issue_iid}` pushed to origin

### Handoff
**Next phase**: verify
**Context for next agent**: Branch `agent/fix-{issue_iid}` is ready. Run full CI pipeline to verify.
```
```

**Step 2: Commit**

```bash
git add -f .openhands/microagents/implement.md
git commit -m "feat(agents): add implement microagent definition"
```

---

### Task 16: Add phase transition logic to dispatcher

**Files:**
- Create: `services/agent_dispatcher/src/phase_machine.rs`
- Modify: `services/agent_dispatcher/src/handler.rs`

**Step 1: Write the failing test**

```rust
// services/agent_dispatcher/src/phase_machine.rs
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_next_labels_diagnose_to_plan() {
        let current = vec!["trigger/ci-failure", "phase/diagnose", "agent/active"];
        let next = transition_labels(&current, Phase::Diagnose, Phase::Plan);
        assert!(next.contains(&"phase/plan".to_string()));
        assert!(!next.contains(&"phase/diagnose".to_string()));
        assert!(next.contains(&"agent/active".to_string()));
    }

    #[test]
    fn test_retry_labels() {
        let current = vec!["phase/verify", "agent/active", "retry/1"];
        let next = add_retry_label(&current, 2);
        assert!(next.contains(&"retry/2".to_string()));
        assert!(!next.contains(&"retry/1".to_string()));
    }

    #[test]
    fn test_blocked_on_max_retry() {
        assert!(should_block(3, 3));
        assert!(!should_block(2, 3));
    }

    #[test]
    fn test_parse_retry_count() {
        let labels = vec!["phase/verify".to_string(), "retry/2".to_string(), "agent/active".to_string()];
        assert_eq!(current_retry_count(&labels), 2);
    }

    #[test]
    fn test_parse_retry_count_none() {
        let labels = vec!["phase/verify".to_string(), "agent/active".to_string()];
        assert_eq!(current_retry_count(&labels), 0);
    }
}
```

**Step 2: Run test to verify it fails**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- phase_machine`
Expected: FAIL

**Step 3: Implement phase machine**

```rust
// services/agent_dispatcher/src/phase_machine.rs
use crate::openhands::Phase;

pub fn transition_labels(
    current: &[impl AsRef<str>],
    from: Phase,
    to: Phase,
) -> Vec<String> {
    let from_label = from.label();
    let to_label = to.label();
    let mut result: Vec<String> = current
        .iter()
        .filter(|l| l.as_ref() != from_label)
        .map(|l| l.as_ref().to_string())
        .collect();
    if !result.iter().any(|l| l == to_label) {
        result.push(to_label.to_string());
    }
    result
}

pub fn current_retry_count(labels: &[String]) -> u8 {
    labels
        .iter()
        .filter_map(|l| l.strip_prefix("retry/"))
        .filter_map(|n| n.parse::<u8>().ok())
        .max()
        .unwrap_or(0)
}

pub fn add_retry_label(current: &[impl AsRef<str>], retry_num: u8) -> Vec<String> {
    let mut result: Vec<String> = current
        .iter()
        .filter(|l| !l.as_ref().starts_with("retry/"))
        .map(|l| l.as_ref().to_string())
        .collect();
    result.push(format!("retry/{retry_num}"));
    result
}

pub fn should_block(retry_count: u8, max_retries: u8) -> bool {
    retry_count >= max_retries
}

pub fn blocked_labels(current: &[impl AsRef<str>]) -> Vec<String> {
    let mut result: Vec<String> = current
        .iter()
        .filter(|l| l.as_ref() != "agent/active")
        .map(|l| l.as_ref().to_string())
        .collect();
    if !result.iter().any(|l| l == "agent/blocked") {
        result.push("agent/blocked".to_string());
    }
    result
}

pub fn done_labels(current: &[impl AsRef<str>]) -> Vec<String> {
    let mut result: Vec<String> = current
        .iter()
        .filter(|l| {
            let s = l.as_ref();
            s != "agent/active" && !s.starts_with("phase/")
        })
        .map(|l| l.as_ref().to_string())
        .collect();
    if !result.iter().any(|l| l == "agent/done") {
        result.push("agent/done".to_string());
    }
    result
}
```

**Step 4: Run tests**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- phase_machine`
Expected: all PASS

**Step 5: Commit**

```bash
git add services/agent_dispatcher/src/phase_machine.rs services/agent_dispatcher/src/handler.rs
git commit -m "feat(agent-dispatcher): phase transition state machine with retry logic"
```

---

## Phase 4: Verify + Deliver Agents

Close the loop with CI verification and MR creation.

---

### Task 17: Write verify microagent

**Files:**
- Create: `.openhands/microagents/verify.md`

**Step 1: Write microagent**

```markdown
<!-- .openhands/microagents/verify.md -->
---
name: verify
agent: CodeActAgent
---

# Verify Agent

You verify that the implementation from Phase 3 passes CI.

## Your Task

1. Read the "Phase 3: Implement Complete" comment to get the branch name
2. Trigger CI pipeline on the branch (push is sufficient — GitLab auto-triggers)
3. Poll pipeline status: `glab ci status --branch agent/fix-{issue_iid}`
4. Wait up to 15 minutes for completion

## Output Format

### On GREEN pipeline:

```
## Phase 4: Verify Complete

**Agent**: verify
**LLM**: devstral
**Duration**: {elapsed}s

### Result: ✅ PASS
Pipeline #{pipeline_id} passed on `agent/fix-{issue_iid}`

### Handoff
**Next phase**: deliver
**Context for next agent**: Pipeline green. Create MR from `agent/fix-{issue_iid}` to `main`.
```

### On RED pipeline:

```
## Phase 4: Verify Failed

**Agent**: verify
**LLM**: devstral

### Result: ❌ FAIL
Pipeline #{pipeline_id} failed on `agent/fix-{issue_iid}`

### Failed Jobs
| Job | Stage | Error |
|-----|-------|-------|
| {name} | {stage} | {first error line} |

### Handoff
**Next phase**: diagnose (retry)
**Context for next agent**: {error details for re-diagnosis}
```

## Constraints

- Do NOT modify code — only check pipeline status
- If pipeline takes >15 minutes, report timeout and let dispatcher handle retry
```

**Step 2: Commit**

```bash
git add -f .openhands/microagents/verify.md
git commit -m "feat(agents): add verify microagent definition"
```

---

### Task 18: Write deliver microagent

**Files:**
- Create: `.openhands/microagents/deliver.md`

**Step 1: Write microagent**

```markdown
<!-- .openhands/microagents/deliver.md -->
---
name: deliver
agent: CodeActAgent
---

# Deliver Agent

You create merge requests for verified fixes.

## Your Task

1. Read the "Phase 4: Verify Complete" comment for confirmation
2. Create MR using `glab`:
   ```bash
   glab mr create \
     --source-branch agent/fix-{issue_iid} \
     --target-branch main \
     --title "fix: {short description} (agent #{issue_iid})" \
     --description "$(cat <<'EOF'
   ## Summary
   {1-2 bullet points from the plan}

   ## Agent Trail
   - Issue: #{issue_iid}
   - Diagnose: Phase 1 comment
   - Plan: Phase 2 comment
   - Implement: Phase 3 comment
   - Verify: Phase 4 comment (pipeline #{pipeline_id})

   ---
   *Auto-generated by foxhunt agent system*
   EOF
   )"
   ```

## Auto-Merge Rules

- If issue has `trigger/ci-failure` label: add `--auto-merge` flag (merge when MR pipeline passes)
- If issue has `trigger/feature` label: do NOT auto-merge (requires human review)

## Output Format

```
## Phase 5: Deliver Complete

**Agent**: deliver
**LLM**: devstral
**Duration**: {elapsed}s

### Merge Request
MR !{mr_iid}: {title}
URL: {mr_url}
Auto-merge: {yes/no}

### Handoff
Agent work complete. Removing `agent/active`, adding `agent/done`.
```

## Constraints

- NEVER force-merge or skip CI on the MR
- NEVER target any branch other than `main`
```

**Step 2: Commit**

```bash
git add -f .openhands/microagents/deliver.md
git commit -m "feat(agents): add deliver microagent definition"
```

---

### Task 19: Add auto-merge support to dispatcher

**Files:**
- Modify: `services/agent_dispatcher/src/gitlab_client.rs`

**Step 1: Write the failing test**

```rust
#[test]
fn test_is_ci_failure_trigger() {
    let labels = vec!["trigger/ci-failure".to_string(), "agent/active".to_string()];
    assert!(is_ci_failure(&labels));

    let labels = vec!["trigger/feature".to_string(), "agent/active".to_string()];
    assert!(!is_ci_failure(&labels));
}
```

**Step 2: Run test to verify it fails**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- gitlab_client::tests::test_is_ci_failure`
Expected: FAIL

**Step 3: Add auto-merge API call to GitLabClient**

```rust
// Add to gitlab_client.rs

pub fn is_ci_failure(labels: &[String]) -> bool {
    labels.iter().any(|l| l == "trigger/ci-failure")
}

impl GitLabClient {
    pub async fn set_auto_merge(&self, mr_iid: u64) -> Result<(), reqwest::Error> {
        let url = format!(
            "{}/api/v4/projects/{}/merge_requests/{}/merge",
            self.base_url, self.project_id, mr_iid
        );
        self.client
            .put(&url)
            .header("PRIVATE-TOKEN", &self.token)
            .json(&serde_json::json!({
                "merge_when_pipeline_succeeds": true
            }))
            .send()
            .await?
            .error_for_status()?;
        Ok(())
    }

    pub async fn close_issue(&self, issue_iid: u64) -> Result<(), reqwest::Error> {
        let url = format!(
            "{}/api/v4/projects/{}/issues/{}",
            self.base_url, self.project_id, issue_iid
        );
        self.client
            .put(&url)
            .header("PRIVATE-TOKEN", &self.token)
            .json(&serde_json::json!({ "state_event": "close" }))
            .send()
            .await?
            .error_for_status()?;
        Ok(())
    }
}
```

**Step 4: Run test**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib -- gitlab_client`
Expected: PASS

**Step 5: Commit**

```bash
git add services/agent_dispatcher/src/gitlab_client.rs
git commit -m "feat(agent-dispatcher): auto-merge for CI-failure MRs + issue close"
```

---

### Task 20: Wire full phase loop in dispatcher handler

**Files:**
- Modify: `services/agent_dispatcher/src/handler.rs`

**Step 1: Implement the full agent loop**

Add a `run_agent_loop` function that:
1. Acquires semaphore permit (concurrency control)
2. Invokes OpenHands for the current phase
3. Parses agent comment for handoff signals
4. Transitions labels
5. If verify fails → check retry count → loop or block
6. If deliver completes → set auto-merge (if ci-failure) → close issue → add `agent/done`

**Step 2: Run all tests**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib`
Expected: all PASS

**Step 3: Run cargo check**

Run: `SQLX_OFFLINE=true cargo check -p agent_dispatcher`
Expected: zero errors

**Step 4: Run clippy**

Run: `SQLX_OFFLINE=true cargo clippy -p agent_dispatcher`
Expected: zero warnings

**Step 5: Commit**

```bash
git add services/agent_dispatcher/src/
git commit -m "feat(agent-dispatcher): full 5-phase agent loop with retry and auto-merge"
```

---

## Phase 5: @agent Mention Trigger

This was already wired in Task 7 (`handle_agent_mention`). Remaining work is end-to-end testing.

---

### Task 21: End-to-end smoke test

**Step 1: Create a test issue with @agent mention**

Run:
```bash
curl -s --request POST "https://git.fxhnt.ai/api/v4/projects/1/issues" \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --data-urlencode "title=[Test] Agent smoke test" \
  --data-urlencode "description=@agent Please add a comment to this file: README.md" \
  | jq '.iid'
```
Expected: returns issue IID

**Step 2: Verify dispatcher picks up the webhook**

Run: `kubectl logs -n foxhunt-agents deployment/dispatcher --tail=50 | grep "activated agent"`
Expected: log line showing agent activation for the test issue

**Step 3: Verify labels applied**

Run: `curl -s "https://git.fxhnt.ai/api/v4/projects/1/issues/{IID}" --header "PRIVATE-TOKEN: $GITLAB_TOKEN" | jq '.labels'`
Expected: `["trigger/feature", "phase/diagnose", "agent/active"]`

**Step 4: Close test issue**

Run: `curl -s --request PUT "https://git.fxhnt.ai/api/v4/projects/1/issues/{IID}" --header "PRIVATE-TOKEN: $GITLAB_TOKEN" --data "state_event=close"`

---

### Task 22: Add CI job to build dispatcher image

**Files:**
- Modify: `.gitlab-ci.yml`

**Step 1: Add build stage for dispatcher**

Add a `build-dispatcher` job in the `build` stage:

```yaml
build-dispatcher:
  stage: build
  image: rg.fr-par.scw.cloud/foxhunt-ci/ci-builder:latest
  script:
    - docker build -f infra/docker/Dockerfile.agent-dispatcher
        -t rg.fr-par.scw.cloud/foxhunt-ci/agent-dispatcher:$CI_COMMIT_SHORT_SHA
        -t rg.fr-par.scw.cloud/foxhunt-ci/agent-dispatcher:latest .
    - docker push rg.fr-par.scw.cloud/foxhunt-ci/agent-dispatcher:$CI_COMMIT_SHORT_SHA
    - docker push rg.fr-par.scw.cloud/foxhunt-ci/agent-dispatcher:latest
  rules:
    - changes:
        - services/agent_dispatcher/**/*
        - infra/docker/Dockerfile.agent-dispatcher
```

**Step 2: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "ci: add build job for agent-dispatcher image"
```

---

### Task 23: Final workspace verification

**Step 1: Cargo check entire workspace**

Run: `SQLX_OFFLINE=true cargo check --workspace -j12`
Expected: zero errors

**Step 2: Cargo clippy entire workspace**

Run: `SQLX_OFFLINE=true cargo clippy --workspace --examples -j12`
Expected: zero warnings

**Step 3: Run dispatcher tests**

Run: `SQLX_OFFLINE=true cargo test -p agent_dispatcher --lib`
Expected: all PASS

**Step 4: Run ML tests (regression check)**

Run: `SQLX_OFFLINE=true cargo test -p ml --lib -- confidence`
Expected: 13 tests PASS (ported confidence aggregator still works)

---

## Summary

| Phase | Tasks | What Ships |
|-------|-------|-----------|
| 0 | 1-3 | K8s namespace, secrets, LiteLLM proxy with 3 model routes |
| 1 | 4-9 | Rust dispatcher (webhook → issue), GitLab labels + webhook |
| 2 | 10-13 | OpenHands deployment, RBAC, diagnose microagent, invocation wiring |
| 3 | 14-16 | Plan + implement microagents, phase transition state machine |
| 4 | 17-20 | Verify + deliver microagents, auto-merge, full loop closure |
| 5 | 21-23 | @agent trigger e2e test, CI build job, workspace verification |

**Total**: 23 tasks across 6 rollout phases.

**New files**: ~15 (K8s manifests, Dockerfile, Rust service modules, microagent definitions)

**Estimated LLM cost per agent run**: ~$2-5 for CI fix (mostly Sonnet), ~$10-20 for feature work (Opus heavy)