# Zuul CI Slack Bot - Design Spec

**Date:** 2026-03-25
**Status:** Draft
**Author:** Itay Matza

## Problem

Engineers need to ask questions about their failing Zuul CI jobs ("why did my build fail?", "is this job flaky?", "what's blocking the queue?") but currently must navigate web UIs, parse logs manually, or rely on team members with tribal knowledge.

mcp-zuul provides 36 generic Zuul API tools published on PyPI. The challenge is making these tools accessible to any engineer in the org - not just Claude Code power users - with project-specific context (failure patterns, naming conventions, debugging guides).

Additionally, the CI operations team (CIOps) has accumulated deep knowledge about failure patterns that is currently tribal - locked in people's heads or scattered across docs. This knowledge needs to be captured, curated, and made available to everyone through the bot.

## Goals

- Any engineer can ask Zuul questions via Slack - zero setup on their end
- Bot provides project-aware answers (not just raw API data)
- CIOps team can manage the knowledge base - add patterns, approve suggestions, curate content
- Bot learns from usage - detects unclassified patterns and suggests them for CIOps review
- mcp-zuul stays public and generic (PyPI package, any Zuul instance)
- Project-specific knowledge lives in internal repos
- Start with Zuul, add integrations (Jira, GitLab) incrementally

## Non-Goals

- Replacing mcp-zuul's MCP interface (it continues serving Claude Code/Desktop users)
- Building a web UI (Slack is the delivery channel)
- Write operations (enqueue/dequeue) - read-only Zuul access
- Fully autonomous learning (CIOps approval is always required)

## Two-Tier Access Model

```
EVERYONE (any engineer)              CIOPS TEAM (operators)
+----------------------------+      +-----------------------------------+
| Ask questions:             |      | Everything engineers can do       |
| "@zuulbot why did X fail?" |      | PLUS:                            |
| "@zuulbot is Y flaky?"    |      |                                   |
| "@zuulbot status of Z"    |      | Add patterns:                    |
|                            |      | "/zuulbot add-pattern ..."       |
| Read-only Zuul access      |      |                                   |
| Uses existing knowledge    |      | Approve suggestions:             |
|                            |      | React with checkmark on          |
|                            |      | #ciops-patterns suggestions      |
|                            |      |                                   |
|                            |      | Review pending patterns:         |
|                            |      | "/zuulbot patterns pending"      |
|                            |      |                                   |
|                            |      | Graduate patterns to baseline:   |
|                            |      | "/zuulbot patterns graduate"     |
|                            |      |                                   |
|                            |      | Knowledge stats:                 |
|                            |      | "/zuulbot knowledge stats"       |
+----------------------------+      +-----------------------------------+

Access control: channel-based (not user-based).
Management commands only work in CIOPS_CHANNELS.
```

## Architecture

```
                    INTERNAL REPO: zuul-ci-bot

 +-----------+     +--------------------+     +--------------+
 |  Slack    |---->|  Slack Bolt        |---->| MCP Client   |
 |  @zuulbot |     |  Socket Mode       |     | (python-sdk) |
 +-----------+     |  lazy listeners    |     | stdio_client |
                   +--------+-----------+     +------+-------+
                            |                        |
                   +--------v-----------+     +------v-------+
                   |  Claude API        |     | mcp-zuul     |
                   |  (tool_use loop)   |     | (persistent  |
                   |  system prompt =   |     |  subprocess) |
                   |  baseline+learned) |     | PUBLIC PKG   |
                   +--------+-----------+     +--------------+
                            |
                   +--------v-----------+
                   | Knowledge Store    |
                   | +-- YAML baseline  | <-- git, curated
                   | +-- SQLite overlay | <-- learned, pending
                   +---------+----------+
                             |
                   +---------v----------+
                   | Pattern Tracker    |
                   | Detects unknown    |
                   | errors, suggests   |---> #ciops-patterns
                   | to CIOps          |     (approval channel)
                   +--------------------+

                    PUBLIC REPO: mcp-zuul (PyPI)
                    36 tools, 3 prompts, 3 resources
                    Zero changes required
```

### Repo Split

| Component | Repo | Visibility | Maintainer |
|-----------|------|------------|------------|
| mcp-zuul (36 Zuul tools) | `imatza-rh/mcp-zuul` | Public (PyPI) | Upstream author |
| Slack bot service | `ci-framework/zuul-ci-bot` | Internal (GitLab) | Team |
| Baseline knowledge (YAML) | `ci-framework/zuul-ci-bot` | Internal (GitLab) | Team (PRs) |
| Learned patterns (SQLite) | Runtime state (container volume) | N/A | Bot + CIOps |

### Key Design Decisions

**1. mcp-zuul as persistent stdio subprocess (not HTTP)**

The bot spawns `mcp-zuul` as a long-lived child process and communicates via MCP stdio protocol. The subprocess and MCP session persist for the bot's lifetime - NOT spawned per request. Reconnect on failure.

Alternative considered: `MCP_TRANSPORT=streamable-http` with Anthropic's MCP Connector. Rejected because it requires deploying mcp-zuul as a persistent HTTP service. Can migrate to this later if multiple services need to share the mcp-zuul instance.

**2. Hybrid knowledge store (YAML baseline + SQLite overlay)**

Curated patterns live in YAML files in git (versioned, reviewed, deployed). Bot-suggested patterns go to SQLite (real-time, pending approval). Approved SQLite patterns can be graduated to YAML periodically. Both sources are merged into Claude's system prompt.

Alternative considered: YAML-only with git PRs for new patterns. Rejected because the feedback loop is too slow (MR -> review -> merge -> redeploy) for a learning system.

Alternative considered: database-only. Rejected because core patterns should be version-controlled and reviewed like code.

**3. Channel-based access control**

Management commands (`add-pattern`, `approve`, `graduate`) only work in channels listed in `CIOPS_CHANNELS`. No user group management or RBAC needed - Slack channel membership IS the access control.

Alternative considered: Slack user group checks. Rejected as over-engineering - channel membership is simpler and sufficient.

**4. Slack Bolt with lazy listeners**

Slack requires `ack()` within 3 seconds. Lazy listeners split handling into immediate acknowledgment and background processing. The Claude API + MCP tool loop runs in the `lazy` function with no timeout constraint. Note: `process_before_response=True` is required in the App constructor for lazy listeners to function.

**5. Manual agentic loop (not tool runner)**

The bot implements the Claude API tool_use loop explicitly: send message -> receive tool_use -> forward to MCP -> send result -> repeat. This gives full control over intermediate Slack status updates ("Analyzing build...", "Checking logs..."). Loop is capped at 15 iterations to prevent runaway tool calls.

## The Learning Loop

The bot gets smarter over time through a detect-suggest-approve cycle:

```
1. Engineer asks: "why did build abc123 fail?"

2. Bot diagnoses via mcp-zuul tools
   -> diagnose_build returns classification: UNKNOWN, confidence: low
   -> No matching pattern in baseline or learned knowledge

3. Bot answers the engineer (best effort from Claude)
   AND logs the unclassified signature to Pattern Tracker

4. Pattern Tracker detects: this signature seen 3+ times this week
   -> Posts to #ciops-patterns:
   +-------------------------------------------------------+
   | :bulb: Unclassified pattern detected                   |
   |                                                        |
   | Signature: "galera_recovery: bootstrap failed"         |
   | Seen in: deploy-osp (3 times this week)                |
   | Category: ? (Claude suggested: REAL_FAILURE)           |
   | Sample builds: abc123, def456, ghi789                  |
   |                                                        |
   | React: :white_check_mark: = approve as REAL_FAILURE    |
   |        :eyes: = needs investigation                    |
   |        :x: = noise, ignore                             |
   |                                                        |
   | Or: /zuulbot approve-pattern --id 42                   |
   |     --category REAL_FAILURE                             |
   |     --diagnosis "Galera cluster failed to bootstrap"   |
   |     --action "Check galera on controller-0"            |
   +-------------------------------------------------------+

5. CIOps engineer reacts with :white_check_mark:
   -> Pattern saved to SQLite as "approved"
   -> System prompt rebuilt with new pattern
   -> Next time anyone hits this error, bot knows the answer

6. Monthly: CIOps runs /zuulbot patterns graduate
   -> Stable approved patterns written to YAML baseline
   -> Git commit + MR for review
```

## Components

### 1. Slack Event Handler (`bot/app.py`)

Slack Bolt app using Socket Mode (no public URL needed). Handles:

- **`@zuulbot` mentions** - main interaction (all channels). Passes the message + thread context to the agent.
- **Slash command `/zuulbot`** - alternative entry point. Uses `/zuulbot` instead of `/zuul` to avoid conflicts.
- **Thread replies** - fetches thread history via `conversations.replies` for multi-turn context.
- **Reactions on #ciops-patterns** - `reaction_added` events for pattern approval workflow.
- **Management commands** - `/zuulbot add-pattern`, `/zuulbot patterns pending|approve|graduate|stats` (CIOps channels only).

Pattern:

```python
from slack_bolt import App
from slack_bolt.adapter.socket_mode import SocketModeHandler

# process_before_response=True is REQUIRED for lazy listeners to work.
# Without it, lazy functions will not execute in Socket Mode.
app = App(token=SLACK_BOT_TOKEN, process_before_response=True)

CIOPS_CHANNELS = os.environ.get("CIOPS_CHANNELS", "").split(",")

def ack_mention(event, client, ack):
    ack()
    result = client.chat_postMessage(
        channel=event["channel"],
        thread_ts=event.get("thread_ts", event["ts"]),
        text=":hourglass_flowing_sand: Analyzing...",
    )
    event["_placeholder_ts"] = result["ts"]

def process_mention(event, client):
    thread_ts = event.get("thread_ts", event["ts"])
    placeholder_ts = event.get("_placeholder_ts")

    # Fetch thread history for multi-turn context
    thread_messages = []
    if event.get("thread_ts"):
        replies = client.conversations_replies(
            channel=event["channel"], ts=event["thread_ts"], limit=20)
        thread_messages = _convert_thread_to_claude_messages(
            replies["messages"])

    # Run Claude + MCP loop
    answer, classification = await agent.ask(
        question=event["text"],
        thread_messages=thread_messages,
        on_tool_call=lambda name, args: client.chat_update(
            channel=event["channel"], ts=placeholder_ts,
            text=f":mag: Calling {name}..."),
    )

    # Post final answer
    client.chat_update(
        channel=event["channel"], ts=placeholder_ts, text=answer)

    # Track unclassified patterns
    if classification and classification.get("confidence") in ("low", None):
        pattern_tracker.record(classification, event)

app.event("app_mention")(ack=ack_mention, lazy=[process_mention])

# --- CIOps Management Commands ---

@app.command("/zuulbot")
def handle_command(ack, command, client, respond):
    ack()
    channel = command["channel_id"]
    args = command["text"].split()
    subcommand = args[0] if args else "help"

    # Management commands restricted to CIOps channels
    if subcommand in ("add-pattern", "approve-pattern", "patterns",
                      "knowledge"):
        if channel not in CIOPS_CHANNELS:
            respond("Management commands are only available in "
                    "CIOps channels.")
            return

    if subcommand == "patterns":
        _handle_patterns_command(args[1:], respond)
    elif subcommand == "add-pattern":
        _handle_add_pattern(args[1:], respond)
    elif subcommand == "approve-pattern":
        _handle_approve_pattern(args[1:], respond)
    elif subcommand == "knowledge":
        _handle_knowledge_command(args[1:], respond)
    else:
        respond(_help_text())

# --- Reaction-based approval ---

@app.event("reaction_added")
def handle_reaction(event, client):
    if event["item"]["channel"] not in CIOPS_CHANNELS:
        return
    if event["reaction"] == "white_check_mark":
        pattern_id = _extract_pattern_id(event["item"]["ts"])
        if pattern_id:
            knowledge_store.approve_pattern(pattern_id)
            client.chat_postMessage(
                channel=event["item"]["channel"],
                thread_ts=event["item"]["ts"],
                text=f":white_check_mark: Pattern #{pattern_id} approved.")
```

### 2. MCP Bridge (`bot/agent.py`)

Maintains a persistent MCP connection to mcp-zuul. Discovers tools once at startup. Handles the agentic loop with iteration limits and safe content parsing. Returns both the answer and classification metadata for the pattern tracker.

```python
import os
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
import anthropic

MAX_ITERATIONS = 15

class ZuulAgent:
    def __init__(self, zuul_url: str, tenant: str,
                 knowledge_store, extra_env: dict | None = None):
        # Pass full environment so Kerberos, proxy, and SSL vars
        # are inherited. StdioServerParameters.env with only explicit
        # vars drops KRB5CCNAME, HTTPS_PROXY, etc.
        env = {**os.environ, "ZUUL_URL": zuul_url,
               "ZUUL_DEFAULT_TENANT": tenant}
        if extra_env:
            env.update(extra_env)
        self.server_params = StdioServerParameters(
            command="mcp-zuul", env=env)
        self.knowledge_store = knowledge_store
        self.client = anthropic.Anthropic()
        self._session = None
        self._claude_tools = None

    async def connect(self):
        """Start persistent MCP session. Call once at bot startup."""
        self._ctx = stdio_client(self.server_params)
        r, w = await self._ctx.__aenter__()
        self._session = ClientSession(r, w)
        await self._session.__aenter__()
        await self._session.initialize()
        tools = await self._session.list_tools()
        self._claude_tools = [
            {"name": t.name, "description": t.description,
             "input_schema": t.inputSchema}
            for t in tools.tools
        ]

    async def reconnect(self):
        """Reconnect on subprocess crash."""
        try:
            if self._session:
                await self._session.__aexit__(None, None, None)
            if self._ctx:
                await self._ctx.__aexit__(None, None, None)
        except Exception:
            pass
        await self.connect()

    async def ask(self, question: str, thread_messages=None,
                  on_tool_call=None) -> tuple[str, dict | None]:
        """Returns (answer_text, classification_metadata)."""
        if not self._session:
            await self.connect()

        # Build system prompt from both baseline and learned patterns
        system_prompt = self.knowledge_store.build_system_prompt()

        messages = thread_messages or []
        messages.append({"role": "user", "content": question})
        classification = None

        for _ in range(MAX_ITERATIONS):
            try:
                resp = self.client.messages.create(
                    model=os.environ.get(
                        "CLAUDE_MODEL", "claude-sonnet-4-20250514"),
                    max_tokens=4096,
                    system=system_prompt,
                    tools=self._claude_tools,
                    messages=messages,
                )
            except anthropic.APIError:
                return ("Sorry, Claude API is unavailable. "
                        "Try again later.", None)

            if resp.stop_reason != "tool_use":
                text_parts = [
                    block.text for block in resp.content
                    if hasattr(block, "text")]
                return ("\n".join(text_parts) or
                        "No response generated.", classification)

            results = []
            for block in resp.content:
                if block.type == "tool_use":
                    if on_tool_call:
                        on_tool_call(block.name, block.input)
                    try:
                        result = await self._session.call_tool(
                            block.name, arguments=block.input)
                        content = ""
                        if result.content:
                            for part in result.content:
                                if hasattr(part, "text"):
                                    content += part.text
                        if result.isError:
                            content = f"Error: {content}"
                        # Capture classification from diagnose_build
                        if block.name == "diagnose_build":
                            classification = _extract_classification(
                                content)
                    except Exception as e:
                        await self.reconnect()
                        content = f"Tool error: {e}"
                    results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": content,
                    })
            messages.append(
                {"role": "assistant", "content": resp.content})
            messages.append({"role": "user", "content": results})

        return ("Reached maximum tool call limit. "
                "Please try a more specific question.", classification)
```

### 3. Knowledge Store (`bot/knowledge_store.py`)

Hybrid storage - YAML baseline files + SQLite for learned patterns.

```python
import json
import sqlite3
from pathlib import Path

class KnowledgeStore:
    def __init__(self, knowledge_dir: str, db_path: str = "patterns.db"):
        self.knowledge_dir = knowledge_dir
        self.db = sqlite3.connect(db_path)
        self._init_db()
        self._baseline = self._load_baseline()

    def _init_db(self):
        self.db.execute("""
            CREATE TABLE IF NOT EXISTS patterns (
                id INTEGER PRIMARY KEY,
                name TEXT UNIQUE,
                category TEXT,
                signatures TEXT,  -- JSON array
                diagnosis TEXT,
                action TEXT,
                retryable BOOLEAN,
                status TEXT DEFAULT 'pending',
                  -- pending | approved | rejected | graduated
                occurrences INTEGER DEFAULT 1,
                first_seen TEXT,
                last_seen TEXT,
                sample_builds TEXT,  -- JSON array of UUIDs
                suggested_by TEXT,   -- Claude's suggestion
                approved_by TEXT     -- Slack user ID
            )
        """)
        self.db.commit()

    def _load_baseline(self) -> str:
        """Load YAML/markdown knowledge files."""
        parts = []
        for path in sorted(Path(self.knowledge_dir).glob("*")):
            if path.suffix in (".yaml", ".yml", ".md"):
                parts.append(f"## {path.stem}\n\n{path.read_text()}")
        return "\n\n---\n\n".join(parts)

    def build_system_prompt(self) -> str:
        """Merge baseline knowledge with approved learned patterns."""
        approved = self.get_patterns(status="approved")
        if not approved:
            return self._baseline

        learned_section = "## Learned Patterns (approved by CIOps)\n\n"
        for p in approved:
            learned_section += (
                f"- **{p['name']}** ({p['category']}): "
                f"{', '.join(json.loads(p['signatures']))}\n"
                f"  Diagnosis: {p['diagnosis']}\n"
                f"  Action: {p['action']}\n\n"
            )
        return self._baseline + "\n\n---\n\n" + learned_section

    def record_unclassified(self, signature: str, build_uuid: str,
                            job_name: str, claude_suggestion: str):
        """Track an unclassified error. Increments count if seen before."""
        existing = self.db.execute(
            "SELECT id, occurrences, sample_builds FROM patterns "
            "WHERE name = ? AND status = 'pending'",
            (signature,)).fetchone()
        if existing:
            samples = json.loads(existing[2] or "[]")
            samples.append(build_uuid)
            self.db.execute(
                "UPDATE patterns SET occurrences = ?, last_seen = "
                "datetime('now'), sample_builds = ? WHERE id = ?",
                (existing[1] + 1, json.dumps(samples[-10:]), existing[0]))
        else:
            self.db.execute(
                "INSERT INTO patterns (name, signatures, diagnosis, "
                "status, first_seen, last_seen, sample_builds, "
                "suggested_by, category, action, retryable) "
                "VALUES (?, ?, ?, 'pending', datetime('now'), "
                "datetime('now'), ?, ?, ?, '', 0)",
                (signature,
                 json.dumps([signature]),
                 claude_suggestion,
                 json.dumps([build_uuid]),
                 claude_suggestion,
                 "UNKNOWN"))
        self.db.commit()

    def approve_pattern(self, pattern_id: int, approved_by: str = "",
                        category: str = "", diagnosis: str = "",
                        action: str = ""):
        """Approve a pending pattern. Optional field overrides."""
        updates = ["status = 'approved'", "approved_by = ?"]
        params = [approved_by]
        if category:
            updates.append("category = ?")
            params.append(category)
        if diagnosis:
            updates.append("diagnosis = ?")
            params.append(diagnosis)
        if action:
            updates.append("action = ?")
            params.append(action)
        params.append(pattern_id)
        self.db.execute(
            f"UPDATE patterns SET {', '.join(updates)} WHERE id = ?",
            params)
        self.db.commit()

    def reject_pattern(self, pattern_id: int):
        self.db.execute(
            "UPDATE patterns SET status = 'rejected' WHERE id = ?",
            (pattern_id,))
        self.db.commit()

    def get_patterns(self, status: str = "pending") -> list[dict]:
        rows = self.db.execute(
            "SELECT * FROM patterns WHERE status = ? "
            "ORDER BY occurrences DESC", (status,)).fetchall()
        cols = [d[0] for d in self.db.execute(
            "SELECT * FROM patterns LIMIT 0").description]
        return [dict(zip(cols, row)) for row in rows]

    def get_suggestion_candidates(self, min_occurrences: int = 3):
        """Patterns seen enough times to suggest to CIOps."""
        return self.db.execute(
            "SELECT * FROM patterns WHERE status = 'pending' "
            "AND occurrences >= ?", (min_occurrences,)).fetchall()

    def graduate_patterns(self) -> list[dict]:
        """Move approved patterns to YAML baseline. Returns graduated."""
        approved = self.get_patterns(status="approved")
        if not approved:
            return []
        # Write to YAML file
        import yaml
        grad_file = Path(self.knowledge_dir) / "99-graduated.yaml"
        existing = {}
        if grad_file.exists():
            existing = yaml.safe_load(grad_file.read_text()) or {}
        patterns = existing.get("patterns", [])
        for p in approved:
            patterns.append({
                "name": p["name"],
                "category": p["category"],
                "signatures": json.loads(p["signatures"]),
                "diagnosis": p["diagnosis"],
                "action": p["action"],
                "retryable": bool(p["retryable"]),
            })
        grad_file.write_text(yaml.dump(
            {"patterns": patterns}, default_flow_style=False))
        # Mark as graduated in DB
        ids = [p["id"] for p in approved]
        self.db.executemany(
            "UPDATE patterns SET status = 'graduated' WHERE id = ?",
            [(i,) for i in ids])
        self.db.commit()
        # Reload baseline
        self._baseline = self._load_baseline()
        return approved
```

### 4. Pattern Tracker (`bot/pattern_tracker.py`)

Detects unclassified errors and posts suggestions to #ciops-patterns when threshold is reached.

```python
SUGGESTION_THRESHOLD = 3  # Post to CIOps after N occurrences

class PatternTracker:
    def __init__(self, knowledge_store, slack_client,
                 ciops_channel: str):
        self.store = knowledge_store
        self.client = slack_client
        self.channel = ciops_channel
        self._suggested = set()  # Avoid duplicate suggestions

    def record(self, classification: dict, event: dict):
        """Record an unclassified pattern from a diagnosis result."""
        reason = classification.get("classification_reason", "")
        category = classification.get("classification", "UNKNOWN")
        build_uuid = classification.get("build_uuid", "")
        job_name = classification.get("job", "")

        if not reason or category not in ("UNKNOWN", "REAL_FAILURE"):
            return
        # Normalize signature (first 100 chars of reason)
        signature = reason[:100].strip()
        self.store.record_unclassified(
            signature, build_uuid, job_name,
            claude_suggestion=category)

        # Check if threshold reached and not yet suggested
        if signature in self._suggested:
            return
        candidates = self.store.get_suggestion_candidates(
            SUGGESTION_THRESHOLD)
        for c in candidates:
            sig = c[1]  # name column
            if sig not in self._suggested:
                self._suggested.add(sig)
                self._post_suggestion(c)

    def _post_suggestion(self, pattern_row):
        pattern_id = pattern_row[0]
        name = pattern_row[1]
        category = pattern_row[2]
        occurrences = pattern_row[7]
        samples = pattern_row[11]

        self.client.chat_postMessage(
            channel=self.channel,
            text=(
                f":bulb: *Unclassified pattern detected*\n\n"
                f"*Signature:* `{name}`\n"
                f"*Seen:* {occurrences} times\n"
                f"*Claude suggested:* {category}\n"
                f"*Sample builds:* {samples}\n\n"
                f"React :white_check_mark: to approve | "
                f":x: to reject\n"
                f"Or: `/zuulbot approve-pattern --id {pattern_id} "
                f"--category REAL_FAILURE "
                f"--diagnosis \"...\" --action \"...\"`"
            ),
            metadata={
                "event_type": "pattern_suggestion",
                "event_payload": {"pattern_id": str(pattern_id)},
            },
        )
```

### 5. Baseline Knowledge Files (`knowledge/`)

**`01-conventions.md`** - job naming patterns, pipeline structure, scenario numbering.

**`02-failure-patterns.yaml`** - curated failure signatures:

```yaml
patterns:
  - name: ssh_unreachable
    category: INFRA_FLAKE
    signatures:
      - "UNREACHABLE"
      - "kex_exchange_identification"
      - "Connection reset by peer"
    diagnosis: "Transient SSH/network issue between Zuul executor and node."
    action: "Safe to retry. If 3+ consecutive on same hypervisor, check libvirt utility network."
    retryable: true

  - name: image_pull_failure
    category: INFRA_FLAKE
    signatures:
      - "ImagePullBackOff"
      - "ErrImagePull"
      - "manifest unknown"
    diagnosis: "Container registry unavailable or image tag pruned."
    action: "Check if tag exists. For toomanyrequests, retry with backoff."
    retryable: true

  - name: ansible_undefined_variable
    category: REAL_FAILURE
    signatures:
      - "AnsibleUndefinedVariable"
    diagnosis: "Variable not set in the job's variable cascade."
    action: "Trace the variable through scenario files and job vars."
    retryable: false
```

**`03-debugging-guides.md`** - common workflows (DLRN promotions, adoption chains, etc.).

**`99-graduated.yaml`** - auto-generated by `/zuulbot patterns graduate`. Contains patterns that were learned, approved, and promoted to baseline.

### 6. Configuration (`bot/config.py`)

| Variable | Required | Description |
|----------|----------|-------------|
| `SLACK_BOT_TOKEN` | Yes | Slack bot OAuth token (`xoxb-...`) |
| `SLACK_APP_TOKEN` | Yes | Slack app-level token for Socket Mode (`xapp-...`) |
| `ANTHROPIC_API_KEY` | Yes | Claude API key |
| `ZUUL_URL` | Yes | Zuul base URL |
| `ZUUL_DEFAULT_TENANT` | Yes | Default Zuul tenant |
| `ZUUL_USE_KERBEROS` | No | Enable Kerberos auth for Zuul |
| `KNOWLEDGE_DIR` | No | Path to baseline knowledge files (default: `./knowledge/`) |
| `PATTERNS_DB` | No | Path to SQLite patterns database (default: `./patterns.db`) |
| `CIOPS_CHANNELS` | No | Comma-separated Slack channel IDs for management commands |
| `CIOPS_PATTERNS_CHANNEL` | No | Channel ID for pattern suggestions |
| `CLAUDE_MODEL` | No | Claude model (default: `claude-sonnet-4-20250514`) |
| `MAX_CONCURRENT_REQUESTS` | No | Concurrent question limit (default: 5) |
| `USER_COOLDOWN_SECONDS` | No | Per-user rate limit (default: 10) |
| `SUGGESTION_THRESHOLD` | No | Occurrences before suggesting pattern (default: 3) |

## Data Flow

### Engineer Asking a Question

```
1. Engineer: "@zuulbot why did build abc123 fail?"

2. Slack Bolt receives app_mention event
   -> ack() immediately (< 3 seconds)
   -> posts ":hourglass: Analyzing build abc123..."
   -> lazy function starts

3. Rate limit check (per-user cooldown + concurrent limit)

4. ZuulAgent.ask() called with question + thread context
   a. Builds system prompt from baseline YAML + approved SQLite patterns
   b. Claude API call with 32 read-only Zuul tools
   c. Max 15 tool call iterations

5. Claude calls diagnose_build -> interprets with knowledge -> responds

6. Bot updates Slack with diagnosis

7. If classification was UNKNOWN or low confidence:
   -> Pattern Tracker records the signature
   -> If threshold reached, posts suggestion to #ciops-patterns
```

### CIOps Approving a Pattern

```
1. Bot posts suggestion to #ciops-patterns:
   ":bulb: Unclassified pattern: galera_recovery: bootstrap failed
    Seen 3 times. React :white_check_mark: to approve."

2. CIOps engineer reacts with :white_check_mark:
   OR runs: /zuulbot approve-pattern --id 42
            --category REAL_FAILURE
            --diagnosis "Galera cluster failed to bootstrap"
            --action "SSH to controller-0, check galera status"

3. Pattern saved to SQLite as "approved"

4. System prompt rebuilt on next question
   -> Pattern now available to all future diagnoses

5. (Monthly) CIOps runs: /zuulbot patterns graduate
   -> Approved patterns written to 99-graduated.yaml
   -> Git commit created for team review
```

## Deployment

### Container

```dockerfile
FROM python:3.12-slim

# Kerberos system dependencies (required for mcp-zuul[kerberos])
RUN apt-get update && apt-get install -y --no-install-recommends \
    libkrb5-dev gcc && rm -rf /var/lib/apt/lists/*

RUN pip install slack-bolt anthropic mcp "mcp-zuul[kerberos]" pyyaml
COPY bot/ /app/bot/
COPY knowledge/ /app/knowledge/
WORKDIR /app
CMD ["python", "-m", "bot.app"]
```

For non-Kerberos deployments, replace `mcp-zuul[kerberos]` with `mcp-zuul` and remove the `apt-get` line.

### Slack App Setup

Required OAuth scopes:

- `app_mentions:read` - receive @zuulbot mentions
- `chat:write` - post and update messages
- `channels:history` - read thread history for multi-turn context
- `groups:history` - read thread history in private channels
- `commands` - register slash commands
- `reactions:read` - detect approval reactions on pattern suggestions

Socket Mode must be enabled (Settings > Socket Mode > Enable).

### Infrastructure

- Single container (bot + mcp-zuul persistent subprocess)
- Socket Mode - no ingress/public URL required
- **Persistent volume** for `patterns.db` (SQLite) - must survive container restarts
- Kerberos: mount `krb5.conf` and use a **keytab** for automatic renewal via sidecar/cron
- Health check: write timestamp to `/tmp/healthz` on each Slack event; liveness probe checks staleness
- Secrets: `SLACK_BOT_TOKEN`, `SLACK_APP_TOKEN`, `ANTHROPIC_API_KEY` via k8s secrets

## Phased Rollout

| Phase | What | Components | Effort |
|-------|------|------------|--------|
| **Phase 1: MVP** | Everyone asks questions, static YAML knowledge | app.py, agent.py, baseline knowledge | Small |
| **Phase 1.5: Pattern detection** | Bot detects unclassified errors, posts to #ciops-patterns | pattern_tracker.py, SQLite schema | Small (~50 lines) |
| **Phase 2: Approval workflow** | CIOps approves via reactions, patterns stored in SQLite, merged into prompts | knowledge_store.py, reaction handler | Medium |
| **Phase 2.5: Manual entry** | `/zuulbot add-pattern` for direct CIOps input | Command handler | Small |
| **Phase 3: Graduation** | Promote approved patterns to YAML baseline via git | graduate command, YAML writer | Medium |
| **Phase 4: Integrations** | Jira ticket creation, GitLab MR cross-reference | Additional MCP servers or API calls | Medium-Large |
| **Phase 5: Proactive** | Alert channels on watched job failures, trend detection | Scheduled polling, notification logic | Medium |

Phase 1 is the original spec. Each subsequent phase is additive - no changes to earlier phases needed.

## Testing

### Unit Tests

- Knowledge store: YAML loading, SQLite CRUD, system prompt merging, graduation
- Pattern tracker: occurrence counting, threshold detection, suggestion dedup
- MCP bridge: tool conversion, loop logic, iteration limit, content type handling, classification extraction
- Slack handlers: ack/lazy pattern, thread history, reaction handling, channel-based access control

### Integration Tests

- Persistent MCP session lifecycle (connect, use, reconnect on crash)
- End-to-end: Slack event -> Claude API + mcp-zuul -> knowledge store -> response
- Pattern lifecycle: record -> threshold -> suggest -> approve -> graduate

### Manual Validation

- Deploy to staging Slack workspace
- Test each Phase 1 capability against real Zuul builds
- Test CIOps workflow: trigger unknown pattern -> verify suggestion in channel -> approve -> verify next diagnosis uses it
- Verify Kerberos auth with keytab renewal

## Error Handling

| Error | Handling |
|-------|----------|
| mcp-zuul subprocess crash | Auto-reconnect via `reconnect()`. Post error if reconnect fails. |
| Claude API timeout/error | Post "Sorry, couldn't analyze. Try again." |
| Zuul API unreachable | Claude receives error from tool, explains to user. |
| Kerberos ticket expired | Auto-renewed via keytab. Alert if renewal fails. |
| Slack rate limit | Throttle `chat_update` to 1/second. |
| Tool call loop exceeded | Return limit message after 15 iterations. |
| Concurrent request limit | Post "Too many requests. Try again in a moment." |
| SQLite corruption | Recreate DB from schema. Approved patterns in graduated YAML survive. |
| Management command in wrong channel | Respond with "Only available in CIOps channels." |

## Security

- No secrets in code - all via environment variables
- mcp-zuul runs read-only (`ZUUL_READ_ONLY=true`)
- Slack bot scoped to minimum permissions (see Slack App Setup)
- Claude API key in k8s secrets
- Knowledge files and patterns contain no credentials
- Per-user rate limiting prevents API credit abuse
- Channel-based access control for management commands
- MCP subprocess env inherits from parent - review for sensitive vars
- SQLite is local-only, no network exposure

## Future: Pluggable Knowledge in mcp-zuul (Optional)

If the knowledge layer proves valuable, it could be upstreamed to mcp-zuul as a generic feature:

- `ZUUL_KNOWLEDGE_DIR` env var
- Server loads pattern files at startup
- `diagnose_build` enriches response with matched patterns
- New `search_knowledge` tool for explicit queries
- Any Zuul project registers their own patterns

This would benefit ALL mcp-zuul users (Claude Code, Claude Desktop, Cursor) not just the Slack bot. But it's not a prerequisite - the bot works without it.
