# Structural Improvements Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Eliminate code duplication across tools/resources/prompts, harden server.py against FastMCP API changes, add write-operation retry, and split the 2,400-line tools.py monolith into focused modules.

**Architecture:** Five interconnected improvements executed in dependency order. First extract shared formatters (#2) and a shared job-output fetcher (#3/#4) to eliminate diverged copies. Then fix server.py fragility (#5) and minor suggestions. Finally split tools.py (#1) into a package with sub-modules, using `__init__.py` re-exports for zero test breakage.

**Tech Stack:** Python 3.11+, FastMCP, httpx, pytest-asyncio, respx

---

## File Structure

| File | Action | Responsibility |
|------|--------|----------------|
| `src/mcp_zuul/formatters.py` | Modify | Add `fmt_project()`, `fmt_job_variants()`; move `_TERMINAL_RESULTS` to module level |
| `src/mcp_zuul/helpers.py` | Modify | Add 500/503 retry to `api_post`/`api_delete` |
| `src/mcp_zuul/server.py` | Modify | Wrap private `_tool_manager` access with error handling |
| `src/mcp_zuul/tools/__init__.py` | Create | Package init, re-exports all tool functions for backward compat |
| `src/mcp_zuul/tools/_common.py` | Create | Shared constants, `_resolve()`, annotations, `_fetch_job_output()` |
| `src/mcp_zuul/tools/_builds.py` | Create | Build tools: list_builds, get_build, get_buildset, list_buildsets, get_build_failures, diagnose_build + parse helpers |
| `src/mcp_zuul/tools/_logs.py` | Create | Log tools: get_build_log, browse_build_logs, tail_build_log |
| `src/mcp_zuul/tools/_status.py` | Create | Status/analytics: list_tenants, get_status, get_change_status, find_flaky_jobs, get_build_times, get_job_durations |
| `src/mcp_zuul/tools/_config.py` | Create | Config/infra: list_jobs, get_job, get_project, list_pipelines, get_config_errors, list_projects, list_nodes, list_labels, list_semaphores, list_autoholds, get_freeze_jobs, get_freeze_job, get_tenant_info, get_connections, get_components |
| `src/mcp_zuul/tools/_write.py` | Create | Write ops: enqueue, dequeue, autohold_create, autohold_delete |
| `src/mcp_zuul/tools/_tests.py` | Create | JUnit XML: get_build_test_results + parsing helpers |
| `src/mcp_zuul/tools/_logjuicer.py` | Create | LogJuicer: get_build_anomalies |
| `src/mcp_zuul/prompts.py` | Modify | Use shared `_fetch_job_output()` from tools._common |
| `src/mcp_zuul/resources.py` | Modify | Use `fmt_project()`, `fmt_job_variants()` from formatters |
| `src/mcp_zuul/__init__.py` | Modify | Update import from `tools` (package) |
| `tests/conftest.py` | No change | Already fixed in prior commit |

**Key constraint:** All test files import `from mcp_zuul.tools import <function>`. The package `__init__.py` re-exports everything, so **zero test changes** needed.

---

### Task 1: Extract shared formatters (#2)

**Files:**
- Modify: `src/mcp_zuul/formatters.py` (add `fmt_project`, `fmt_job_variants`)
- Modify: `src/mcp_zuul/tools.py:1226-1293` (use new formatters)
- Modify: `src/mcp_zuul/resources.py:23-77` (use new formatters)

- [ ] **Step 1: Add `fmt_job_variants()` and `fmt_project()` to formatters.py**

Add after `fmt_buildset()`:

```python
def fmt_job_variants(data: list) -> list[dict]:
    """Format job API response into compact variant list."""
    variants = []
    for v in data:
        sc = v.get("source_context") or {}
        variants.append(
            clean(
                {
                    "parent": v.get("parent"),
                    "branches": v.get("branches", []) or None,
                    "nodeset": v.get("nodeset"),
                    "timeout": v.get("timeout"),
                    "voting": v.get("voting", True),
                    "abstract": v.get("abstract", False) or None,
                    "description": (v.get("description") or "")[:200] or None,
                    "source_project": sc.get("project"),
                }
            )
        )
    return variants


def fmt_project(data: dict) -> dict:
    """Format project API response into compact representation."""
    configs: dict[str, list[str]] = {}
    for cfg in data.get("configs", []):
        for pip in cfg.get("pipelines", []):
            pname = pip.get("name", "")
            jobs = []
            for j in pip.get("jobs", []):
                if isinstance(j, list):
                    jobs.append(j[0]["name"] if j else "")
                elif isinstance(j, dict):
                    jobs.append(j.get("name", ""))
            if jobs:
                configs[pname] = jobs
    return clean(
        {
            "project": data.get("canonical_name") or "",
            "canonical_name": data.get("canonical_name"),
            "connection": data.get("connection_name"),
            "type": data.get("type"),
            "pipelines": configs,
        }
    )
```

- [ ] **Step 2: Update `get_job` and `get_project` in tools.py to use formatters**

Replace the inline formatting with calls to `fmt_job_variants()` and `fmt_project()`.

- [ ] **Step 3: Update `job_resource` and `project_resource` in resources.py**

Replace inline formatting with calls to `fmt_job_variants()` and `fmt_project()`. The `project_resource` needs to override the `project` key with `f"{org}/{repo}"` after calling `fmt_project()`.

- [ ] **Step 4: Run tests**

Run: `uv run pytest tests/ -v -k "test_returns_variants or test_returns_pipeline_jobs or TestResources" --tb=short`

- [ ] **Step 5: Commit**

```
[REFACTOR] Extract fmt_project and fmt_job_variants to formatters
```

---

### Task 2: Extract shared job-output fetcher (#3 + #4)

**Files:**
- Modify: `src/mcp_zuul/tools.py` (extract `_fetch_job_output()`, refactor `get_build_failures` and `diagnose_build`)
- Modify: `src/mcp_zuul/prompts.py` (use `_fetch_job_output()`)

- [ ] **Step 1: Add `_fetch_job_output()` to tools.py**

Add after `_no_log_url_error()`. This replaces the duplicated suffix loop:

```python
async def _fetch_job_output(
    ctx: Context, build: dict
) -> tuple[list[dict], list[dict], list[list[dict]], bool]:
    """Fetch and parse job-output.json with gz/json fallback + text log grep.

    Returns (playbooks, failed_tasks, log_context, log_truncated).
    Shared by get_build_failures, diagnose_build, and prompts.
    """
    log_url = build.get("log_url", "")
    a = app(ctx)

    playbooks: list[dict] = []
    failed_tasks: list[dict] = []
    for suffix in ("job-output.json.gz", "job-output.json"):
        try:
            resp = await fetch_log_url(a, log_url.rstrip("/") + "/" + suffix)
            if resp.status_code != 200:
                continue
            data = json.loads(resp.content[:_MAX_JSON_LOG_BYTES])
            if isinstance(data, list):
                playbooks, failed_tasks = _parse_playbooks(data)
                break
        except (httpx.DecodingError, json.JSONDecodeError, KeyError):
            continue

    log_context: list[list[dict]] = []
    log_truncated = False
    txt_url = log_url.rstrip("/") + "/job-output.txt"
    try:
        log_bytes, log_truncated = await stream_log(a, txt_url)
        raw = strip_ansi(log_bytes.decode("utf-8", errors="replace"))
        log_context = _grep_log_context(raw)
    except Exception:
        pass

    return playbooks, failed_tasks, log_context, log_truncated
```

- [ ] **Step 2: Refactor `get_build_failures` to use `_fetch_job_output()`**

Replace the inline suffix loop and text grep fallback with a single call.

- [ ] **Step 3: Refactor `diagnose_build` to use `_fetch_job_output()`**

Replace the inline suffix loop and text grep with a single call.

- [ ] **Step 4: Update `_fetch_failed_tasks` in prompts.py**

Replace the old gz→json fallback with `_fetch_job_output()`.

- [ ] **Step 5: Run tests**

Run: `uv run pytest tests/ -v -k "test_build_failures or test_diagnose or TestDebugBuild or TestCompareBuilds" --tb=short`

- [ ] **Step 6: Commit**

```
[REFACTOR] Extract _fetch_job_output shared helper
```

---

### Task 3: Fix server.py private API access (#5) + suggestions

**Files:**
- Modify: `src/mcp_zuul/server.py:58-79`
- Modify: `src/mcp_zuul/formatters.py:168` (move `_TERMINAL_RESULTS` to module level)
- Modify: `src/mcp_zuul/helpers.py:93-130` (add retry to write operations)

- [ ] **Step 1: Wrap `_tool_manager` access with try/except in server.py**

```python
def _remove_tool(server: FastMCP, name: str) -> bool:
    """Remove a tool by name, handling FastMCP internal API changes."""
    try:
        server._tool_manager.remove_tool(name)
        return True
    except (AttributeError, KeyError):
        return False


def _list_tool_names(server: FastMCP) -> list[str]:
    """List registered tool names, handling FastMCP internal API changes."""
    try:
        return list(server._tool_manager._tools.keys())
    except AttributeError:
        log.warning("Cannot list tools — FastMCP internal API changed")
        return []
```

Then update the lifespan to use these helpers.

- [ ] **Step 2: Move `_TERMINAL_RESULTS` to module level in formatters.py**

Move the frozenset from inside `_compute_chain_summary()` to module level.

- [ ] **Step 3: Add 500/503 retry to `api_post` and `api_delete` in helpers.py**

Add the same retry pattern used by `api()`.

- [ ] **Step 4: Run tests**

Run: `uv run pytest tests/ -v --tb=short`

- [ ] **Step 5: Commit**

```
[REFACTOR] Harden server.py, add write retry, module-level constant
```

---

### Task 4: Split tools.py into package (#1)

**Files:**
- Delete: `src/mcp_zuul/tools.py` (2,403 lines)
- Create: `src/mcp_zuul/tools/__init__.py` (~50 lines, re-exports)
- Create: `src/mcp_zuul/tools/_common.py` (~120 lines)
- Create: `src/mcp_zuul/tools/_builds.py` (~500 lines)
- Create: `src/mcp_zuul/tools/_logs.py` (~350 lines)
- Create: `src/mcp_zuul/tools/_status.py` (~450 lines)
- Create: `src/mcp_zuul/tools/_config.py` (~550 lines)
- Create: `src/mcp_zuul/tools/_write.py` (~140 lines)
- Create: `src/mcp_zuul/tools/_tests.py` (~200 lines)
- Create: `src/mcp_zuul/tools/_logjuicer.py` (~80 lines)
- Modify: `src/mcp_zuul/prompts.py` (import path)

Module contents:

**`_common.py`**: `_READ_ONLY`, `_WRITE`, `_DESTRUCTIVE` annotations, `_resolve()`, `_no_log_url_error()`, `_fetch_job_output()`, `_MAX_LOG_LINES`, `_MAX_JSON_LOG_BYTES`, `_MAX_FILE_BYTES`, `_ERROR_PATTERNS`, `_ERROR_NOISE`, `_parse_playbooks`, `_smart_truncate`, `_extract_inner_recap`, `_truncate_invocation`, `_grep_log_context`, `_FATAL_PATTERN`, `_PLAY_RECAP_RE`, `_RUN_END_MARKER`

**`_builds.py`**: `list_builds`, `get_build`, `get_build_failures`, `diagnose_build`, `get_buildset`, `list_buildsets`

**`_logs.py`**: `get_build_log`, `browse_build_logs`, `tail_build_log`

**`_status.py`**: `list_tenants`, `get_status`, `get_change_status`, `find_flaky_jobs`, `get_build_times`, `get_job_durations`

**`_config.py`**: `list_jobs`, `get_job`, `get_project`, `list_pipelines`, `get_config_errors`, `list_projects`, `list_nodes`, `list_labels`, `list_semaphores`, `list_autoholds`, `get_freeze_jobs`, `get_freeze_job`, `get_tenant_info`, `get_connections`, `get_components`

**`_write.py`**: `enqueue`, `dequeue`, `autohold_create`, `autohold_delete`

**`_tests.py`**: `get_build_test_results`, `_find_test_xmls`, `_safe_float`, `_parse_junit_xml`, `_MAX_XML_BYTES`

**`_logjuicer.py`**: `get_build_anomalies`

**`__init__.py`**: Imports all sub-modules (triggers `@mcp.tool()` registration), re-exports `_parse_playbooks`, `_fetch_job_output`, `_MAX_JSON_LOG_BYTES` for prompts.py.

- [ ] **Step 1: Create `src/mcp_zuul/tools/` directory**

- [ ] **Step 2: Create `_common.py` with shared constants, helpers, annotations**

- [ ] **Step 3: Create `_builds.py` with build tools**

- [ ] **Step 4: Create `_logs.py` with log tools**

- [ ] **Step 5: Create `_status.py` with status/analytics tools**

- [ ] **Step 6: Create `_config.py` with config/infra tools**

- [ ] **Step 7: Create `_write.py` with write operations**

- [ ] **Step 8: Create `_tests.py` with JUnit XML parsing**

- [ ] **Step 9: Create `_logjuicer.py` with LogJuicer tool**

- [ ] **Step 10: Create `__init__.py` with re-exports**

- [ ] **Step 11: Update `prompts.py` import path**

Change `from .tools import _MAX_JSON_LOG_BYTES, _parse_playbooks` to `from .tools._common import ...` or keep as-is if `__init__.py` re-exports.

- [ ] **Step 12: Delete old `src/mcp_zuul/tools.py`**

- [ ] **Step 13: Run full test suite + lint + mypy**

Run: `uv run pytest tests/ -v && uv run ruff check src/ tests/ && uv run ruff format --check src/ tests/ && uv run mypy src/mcp_zuul/`
Expected: 373 tests pass, all checks clean.

- [ ] **Step 14: Commit**

```
[REFACTOR] Split tools.py monolith into package with sub-modules
```

---

### Task 5: Final verification

- [ ] **Step 1: Run full test suite with coverage**

Run: `uv run pytest tests/ --cov=mcp_zuul --cov-report=term-missing -v`
Expected: 373 pass, >= 85% coverage

- [ ] **Step 2: Run all linters**

Run: `uv run ruff check src/ tests/ && uv run ruff format --check src/ tests/ && uv run mypy src/mcp_zuul/`

- [ ] **Step 3: Verify line counts show improvement**

Run: `wc -l src/mcp_zuul/tools/*.py` — no single file over 600 lines.

- [ ] **Step 4: Update CLAUDE.md module dependency flow**

Update the architecture section to reflect the new `tools/` package structure.
