# Session Findings: mcp-zuul tool interactions

**Session**: 2026-03-27 00:22–01:10 IST
**Task**: Babysit tp!2134 — diagnose install-shiftstack-before failure,
extract root cause from corrupted logs, push fix, trigger retry.
**Tools used**: `get_change_status`, `diagnose_build`, `get_build_failures`,
`tail_build_log`, `get_build_log`, `browse_build_logs`, `list_builds`
**Overall friction**: **HIGH** — root cause extraction took ~30 min due to
tool limitations; a custom Python script using mcp-zuul's auth module was
required. Without the script, the investigation would have stalled.

---

## 1. Session Summary

### What was done
1. Checked build status for tp!2134 (change 2134)
2. Diagnosed install-shiftstack-before failure (build 09e8b09b)
3. Attempted to read full logs — blocked by corrupted gzip
4. Navigated log directory tree looking for alternative log files
5. Checked earlier build (3892c0a7) — same corruption
6. Found historical SUCCESS builds for comparison
7. Wrote custom Python script to download and parse job-output.json.gz
8. Extracted root cause: OCP bootstrap timeout due to 16GB master flavor

### Tool call inventory

| # | Tool | Parameters (key ones) | Result | Useful? |
|---|------|-----------------------|--------|---------|
| 1 | `get_change_status` | `change=2134` | Buildset + 4 builds with results | YES |
| 2 | `diagnose_build` | `uuid=09e8b09b` | Failed task, truncated stdout (1.7M→8K) | PARTIAL |
| 3 | `tail_build_log` | `uuid=09e8b09b, lines=200` | ERROR: corrupted gzip | NO |
| 4 | `get_build_log` | `uuid=09e8b09b, grep=FAILED\|fatal` | ERROR: corrupted gzip | NO |
| 5-14 | `browse_build_logs` | various paths | Directory listings, file contents | YES |
| 15 | `get_build_failures` | `uuid=09e8b09b` | Same as diagnose_build minus classification | REDUNDANT |
| 16 | `browse_build_logs` | `zuul-info/inventory.yaml` | 38K inventory YAML | YES (verbose) |
| 17 | `list_builds` | `job_name=..., change=2134, result=FAILURE` | 2 builds | YES |
| 18 | `tail_build_log` | `uuid=3892c0a7, lines=200` | ERROR: corrupted gzip | NO |
| 19 | `get_build_failures` | `uuid=3892c0a7` | Same truncation pattern | PARTIAL |
| 20 | `get_build_log` | `log_name=job-output.json.gz, grep=...` | ERROR: corrupted gzip | NO |
| 21 | `list_builds` | `job_name=..., result=SUCCESS` | 3 SUCCESS builds | YES |

**Score**: 10 useful, 5 failed/unhelpful, 6 partially useful = 21 total calls.
With fixes, calls 2-4, 18-20 would have been sufficient (6 calls instead of 21).

---

## 2. Findings

### F1: `_smart_truncate` discards the most important part of large output

**Severity**: P0 (broken — makes the tool unable to do its job)

**Tool**: `diagnose_build`, `get_build_failures`

**What was attempted**: Diagnose why install-shiftstack-before failed.

**Parameters used**:
```
diagnose_build(uuid="09e8b09b55eb40a58045a6bfcb51eb5a", tenant="components-integration")
```

**Response received** (relevant excerpt):
```json
{
  "failed_tasks": [{
    "task": "Run shiftstack playbook with host override",
    "msg": "non-zero return code",
    "rc": 2,
    "stdout": "[WARNING]: Invalid characters...\n\n[... 1734480 chars omitted ...]\n\ncontainers.podman.podman_container_exec --- 4132.62s\n...",
    "inner_recap": "osp-undercloud-0: ok=26 failed=1"
  }],
  "classification": "REAL_FAILURE",
  "classification_confidence": "medium"
}
```

**What went wrong**: The actual error — `fatal: [localhost]: FAILED! =>
{"cmd": "openshift-install create cluster ...", "rc": 4, "stderr":
"failed to provision control-plane machines: machines are not ready:
client rate limiter Wait returned an error"}` — was at position ~850K
in the 1.7M char stdout. `_smart_truncate` kept the first ~4K (ansible
version banner) and last ~4K (profile_tasks summary). The error was
invisible.

**The tool knew there WAS an inner failure** — `inner_recap` showed
`failed=1`. But it couldn't show WHAT failed.

**Suggested fix**: Extract error patterns from the full text before
truncating. Add a new field `inner_errors` or `extracted_errors` that
contains lines matching `fatal:`, `FAILED!`, `level=error`, etc., with
surrounding context. This is a small change to the truncation logic
that would have immediately solved the investigation.

Rough implementation:
```python
# In _smart_truncate or in the caller before truncation
error_lines = []
for pattern in [r'fatal:.*FAILED', r'level=error msg=']:
    for m in re.finditer(pattern, text, re.IGNORECASE):
        start = text.rfind('\n', max(0, m.start() - 200), m.start()) + 1
        end = text.find('\n', m.end() + 200)
        error_lines.append(text[start:end if end > 0 else len(text)])
```

---

### F2: Corrupted gzip fallback chain is broken

**Severity**: P0 (broken — error message suggests a fix that also fails)

**Tool**: `tail_build_log`, `get_build_log`

**What was attempted**: Read the build log to find the failure.

**Parameters used**:
```
tail_build_log(uuid="09e8b09b", tenant="components-integration", lines=200, skip_postrun=true)
get_build_log(uuid="09e8b09b", tenant="components-integration", grep="FAILED|fatal|failed=1", context=5)
```

**Response received** (both):
```json
{"error": "Log file decompression failed (corrupted gzip). Use
get_build_log with grep='FAILED|fatal' for text-based diagnosis,
or diagnose_build which falls back automatically."}
```

**What went wrong**: Three issues:

1. **Misleading error message**: The error says "Use get_build_log with
   grep=" but that tool ALSO fails with the same error on the same file.
   The caller follows the suggestion, wastes a tool call, gets the same
   error. The message should say "Use `diagnose_build` (reads
   `job-output.json.gz` instead)" since that's the only fallback that
   actually works.

2. **No partial decompression**: gzip streams can often be partially read
   even when truncated/corrupted. Python's `zlib.decompressobj` can extract
   whatever data is valid before the corruption point. The tool gives up
   entirely instead of trying.

3. **No cross-file fallback**: When `job-output.txt.gz` is corrupted, the
   tool could automatically try `job-output.json.gz` (structured version)
   and extract text content from the JSON task results. The json.gz file
   was NOT corrupted in this case — `diagnose_build` read it successfully.

**Additionally**: Tried `get_build_log` with `log_name="job-output.json.gz"`
explicitly — same error. The tool applies the same gzip handling to all
files without checking if the file is actually gzip (json.gz files are
served decompressed by some Zuul log servers when Accept-Encoding doesn't
include gzip).

**Suggested fix**:
- Fix error message to point to `diagnose_build` as the real fallback
- Implement partial gzip decompression via `zlib.decompressobj`
- When `.txt.gz` fails AND the user passed `grep=`, automatically try
  searching the `.json.gz` structured data as a fallback
- Handle server-side decompression (check Content-Encoding before
  attempting local decompression)

---

### F3: `diagnose_build` can't extract errors from nested playbook output

**Severity**: P1 (significant gap)

**Tool**: `diagnose_build`

**What was attempted**: Understand why the inner playbook failed.

**Response**: The tool correctly identified the outer task failure and
provided `inner_recap` showing `failed=1`. But it couldn't tell us
WHICH inner task failed or what the error was, because that information
was in the truncated `stdout`.

**Suggested fix**: When `inner_recap` contains `failed > 0`, parse the
full (pre-truncation) `stdout` for Ansible `fatal:` blocks. Extract
them into a new response field:

```json
{
  "inner_failures": [
    {
      "task": "Use an openshift-install flag to wait until the cluster is ready",
      "host": "localhost",
      "msg": "non-zero return code",
      "rc": 6,
      "cmd": "openshift-install wait-for install-complete ...",
      "stderr_excerpt": "failed to provision control-plane machines..."
    }
  ]
}
```

This requires parsing ANSI-colored Ansible output, which is messy but
well-defined. The `fatal:` blocks contain JSON that can be extracted
with regex.

---

### F4: `diagnose_build` classification is shallow for nested failures

**Severity**: P2 (improvement)

**Tool**: `diagnose_build`

**Response**:
```json
{
  "classification": "REAL_FAILURE",
  "classification_reason": "Task 'Run shiftstack playbook with host override' failed: non-zero return code",
  "classification_confidence": "medium"
}
```

**What went wrong**: The classification describes the OUTER failure
("ansible-playbook returned non-zero") not the ROOT CAUSE ("OCP
bootstrap timeout due to compute memory pressure"). With inner failure
extraction (F3), the classifier could inspect the inner error and
produce:

```json
{
  "classification": "REAL_FAILURE",
  "classification_reason": "Inner playbook: openshift-install bootstrap timeout (rc=4, 20m deadline). Control-plane machines not provisioned in time.",
  "classification_confidence": "high"
}
```

**Depends on**: F3 (inner failure extraction)

---

### F5: `get_build_failures` is redundant after `diagnose_build`

**Severity**: P3 (polish)

**Tool**: `get_build_failures`

**What happened**: Called `diagnose_build` first (which includes failures
+ classification + log context). Then called `get_build_failures` hoping
for different/more data. Got the exact same failure data without the
classification. Wasted a tool call.

**Observation**: `diagnose_build` is strictly a superset of
`get_build_failures` — it includes everything `get_build_failures`
returns plus classification, log context, and ref metadata. The only
reason to call `get_build_failures` separately is if `diagnose_build`
failed or if you explicitly don't want classification.

**Suggested fix**: Document clearly that `diagnose_build` includes all
`get_build_failures` data. Consider adding a note in the
`get_build_failures` description: "For most use cases, prefer
`diagnose_build` which includes this data plus classification and log
context."

---

### F6: `browse_build_logs` returns full file content for large files

**Severity**: P3 (polish)

**Tool**: `browse_build_logs`

**Parameters used**:
```
browse_build_logs(uuid="09e8b09b", path="zuul-info/inventory.yaml")
```

**Response**: 38,109 bytes of YAML — the full Zuul inventory including
all project refs, playbook contexts, and build metadata. Most of this
wasn't needed (only wanted `hypervisor` hostname and `ansible_host` IP).

**What went well**: The tool correctly fetched and returned the file.
The content WAS useful for checking the build's configuration.

**Suggested improvement**: For files > 10KB, consider returning a
summary or first N lines with a note about total size, letting the
caller request specific sections. Or add optional `grep` and
`max_lines` parameters to `browse_build_logs` for file content mode.

---

### F7: No tool to download raw log files with authentication

**Severity**: P2 (improvement)

**What was attempted**: When all standard tools failed on corrupted
gzip, tried to download `job-output.json.gz` directly via `curl` and
Python `httpx`. Both failed because the log server requires OIDC/SPNEGO
authentication that can't be done with simple `--negotiate`.

**Workaround**: Wrote a custom Python script importing
`mcp_zuul.auth.kerberos_auth` to authenticate, then downloaded and
parsed the file. This worked but required knowledge of mcp-zuul
internals.

**Suggested fix**: Add a `download_build_artifact` tool or add a
`raw=true` parameter to `browse_build_logs` that returns base64-encoded
file content for binary/compressed files. This would enable the caller
to decompress and parse locally when the standard tools fail.

Alternatively, expose a `search_build_json` tool that searches inside
`job-output.json.gz` specifically:
```
search_build_json(uuid="09e8b09b", pattern="fatal:|FAILED", context=5)
```
This would be the nuclear option for when `get_build_log` fails on
corrupted txt.gz.

---

### F8: `get_build_log` applies gzip decompression uniformly

**Severity**: P2 (improvement)

**Tool**: `get_build_log`

**Parameters used**:
```
get_build_log(uuid="09e8b09b", log_name="job-output.json.gz",
              grep="FAILED|fatal|failed=1", context=3)
```

**Response**: Same "corrupted gzip" error as for `job-output.txt.gz`.

**What went wrong**: The tool name specifies `.json.gz` but the server
may serve it with `Content-Encoding: gzip` (already decompressed by the
HTTP layer) or without (raw gzip bytes). The tool seems to always
attempt gzip decompression without checking:
1. Whether the response has `Content-Encoding: gzip` (already handled)
2. Whether the raw bytes actually start with gzip magic bytes (`\x1f\x8b`)
3. Whether the file is the `.txt.gz` variant vs `.json.gz` variant

The custom script that worked checked for gzip magic bytes before
attempting decompression:
```python
data = gzip.decompress(raw) if raw[:2] == b'\x1f\x8b' else raw
```

**Suggested fix**: In `fetch_log_url` or the caller:
1. Check `Content-Encoding` header — if `gzip`, the HTTP client already
   decompressed
2. Check magic bytes before attempting `gzip.decompress`
3. If decompression fails, try using the raw bytes (might already be
   decompressed)

---

## 3. Missing Tools

### M1: Search inside structured job-output.json

**Use case**: When `job-output.txt.gz` is corrupted but `.json.gz` is
fine, search the structured JSON for error patterns across all tasks'
stdout/stderr/msg fields.

**Proposed tool**:
```
search_job_output(uuid, pattern, fields=["stdout", "stderr", "msg"],
                  context_chars=500, max_matches=10)
```

Returns matches with task name, host, and surrounding context. This is
what the custom Python script did — it should be a first-class tool.

### M2: Compare two builds' configurations

**Use case**: After finding that build 09e8b09b failed and build
594ab02f succeeded, wanted to compare what was different (vars,
inventory, job definition). Currently requires manually browsing
`zuul-info/inventory.yaml` for both builds and diffing.

**Proposed tool**:
```
compare_builds(uuid_a, uuid_b, sections=["vars", "projects", "nodeset"])
```

Returns a diff of the significant configuration differences between two
builds of the same job.

### M3: Get build's Zuul job vars only (not full inventory)

**Use case**: Needed to check `shiftstack_qa_gerrit_ref` and
`hypervisor` vars. Had to download the full 38K inventory to find 2
values.

**Proposed**: Add a `vars_only=true` parameter to an existing tool, or
return vars as a top-level field in `get_build` / `diagnose_build`.

---

## 4. What Worked Well

### `get_change_status`
**Call**: `get_change_status(change="2134", tenant="components-integration")`

Returned the complete buildset with all 4 builds, their results,
durations, and UUIDs in a single call. No follow-up needed. The
`not_in_pipeline` detection with automatic latest-buildset fallback
is excellent — no need to separately check pipeline status vs build
history.

### `browse_build_logs`
Called 14 times across the session for directory navigation and file
reading. Every call returned correct results quickly. The directory
listing mode is clean and the file content mode handles text files
well. The only issue was large files (F6) which is minor.

### `list_builds`
**Call**: `list_builds(job_name="...", change="2134", result="FAILURE")`

Clean, fast, correct. Filtering by job_name + change + result is
exactly the right API. The response format (UUID, duration, project,
change) has everything needed for follow-up calls.

### `diagnose_build` (partial)
Despite the truncation issue (F1), the tool correctly:
- Identified the failed playbook and phase (run, not pre/post)
- Provided playbook-level pass/fail for all 19 playbooks
- Extracted the failed task name, host, cmd, rc, invocation
- Provided `inner_recap` detecting the nested playbook failure
- Classified as REAL_FAILURE (correct, even if shallow)
- Included `ref_url`, `project`, `change` for cross-referencing

The structure is right — it just needs deeper extraction for nested
failures.

### Kerberos auth module
When all tools failed, the `mcp_zuul.auth.kerberos_auth()` function
worked flawlessly from a custom script. The auth flow (OIDC redirect →
401 Negotiate → SPNEGO token → callback → session cookie) completed
in <2 seconds. This module is well-designed and reusable.

---

## 5. Recommendations

### Quick wins (can ship in one PR each)
1. **Fix error message** in corrupted gzip handler — point to
   `diagnose_build`, not `get_build_log with grep=` (30 min fix)
2. **Add gzip magic byte check** before attempting decompression (15 min)
3. **Add `get_build_failures` docstring note** about `diagnose_build`
   being a superset (5 min)

### Medium effort (1-2 PRs)
4. **Extract error patterns before truncating** in `_smart_truncate` —
   add `extracted_errors` field to response (2-3 hours)
5. **Partial gzip decompression** via `zlib.decompressobj` (1 hour)
6. **json.gz fallback** when txt.gz is corrupted and `grep=` is used
   (2 hours)

### Larger features
7. **Inner playbook error extraction** — parse Ansible FATAL blocks from
   nested `ansible-playbook` stdout (4-6 hours)
8. **`search_job_output` tool** — search structured json.gz fields
   (3-4 hours)
9. **Build comparison tool** (4-6 hours)
