# Session Review: mcp-zuul gaps during babysit-tp investigation

**Date**: 2026-03-27
**Context**: Babysitting tp!2134 (shiftstack adoption pipeline on titan35).
install-shiftstack-before failed twice (~70m, ~100m). Both `job-output.txt.gz`
files were corrupted gzip. SSH to lab blocked by key rotation. Had to extract
the root cause entirely from Zuul build artifacts via mcp-zuul tools.

**Outcome**: Root-caused after ~30 min of tool wrestling. A custom Python
script using mcp-zuul's auth module was needed to parse the full
`job-output.json.gz` and grep for FATAL errors in 1.7M chars of truncated
stdout. The fix (12GB masters) was pushed without this being necessary if the
tools had better error extraction.

---

## Critical Issues

### 1. `_smart_truncate` hides errors in the middle of large output

**Impact**: HIGH — this was the primary blocker in the investigation.

**What happened**: `get_build_failures` and `diagnose_build` both returned
the failed task with a 1.7M char `stdout` field truncated to head+tail
(~4000 chars each). The actual FATAL error from the inner
`openshift-install` was at position ~850K — invisible in both head and tail.

The truncated output showed:
- **Head**: `ansible-playbook [core 2.15.13]` version banner
- **Tail**: `profile_tasks` timing summary + PLAY RECAP
- **Middle (omitted)**: The actual `fatal: [localhost]: FAILED! => {...}`

**Proposed fix**: Before truncating, scan the full text for high-signal
patterns and include them in the output. Something like:

```python
def smart_truncate(text, max_size=4000, *, _pre_stripped=False):
    if len(text) <= max_size * 2:
        return text

    # NEW: extract error lines from the middle
    error_patterns = [
        r'fatal:.*FAILED',
        r'level=error msg=',
        r'NoValidHost',
        r'non-zero return code',
    ]
    error_lines = []
    for pattern in error_patterns:
        for m in re.finditer(pattern, text, re.IGNORECASE):
            start = text.rfind('\n', 0, m.start()) + 1
            end = text.find('\n', m.end())
            if end == -1:
                end = len(text)
            line = text[start:end].strip()
            if line and line not in error_lines:
                error_lines.append(line)

    head = text[:max_size]
    tail = text[-max_size:]
    middle = ""
    if error_lines:
        # Deduplicate and limit
        unique = list(dict.fromkeys(error_lines))[:10]
        middle = "\n\n[... errors extracted from omitted section ...]\n"
        middle += "\n".join(unique[:10])
        middle += "\n"

    omitted = len(text) - len(head) - len(tail)
    return f"{head}\n\n[... {omitted} chars omitted ...]{middle}\n\n{tail}"
```

This would have immediately surfaced:
```
fatal: [localhost]: FAILED! => {"cmd": "openshift-install create cluster ...",
  "msg": "non-zero return code", "rc": 4, ...}
failed to provision control-plane machines: machines are not ready:
  client rate limiter Wait returned an error
```

### 2. `get_build_log` / `tail_build_log` give up entirely on corrupted gzip

**Impact**: HIGH — both tools returned an error with no data.

**What happened**: `job-output.txt.gz` was corrupted. Both tools returned:
```json
{"error": "Log file decompression failed (corrupted gzip). Use
get_build_log with grep='FAILED|fatal' for text-based diagnosis,
or diagnose_build which falls back automatically."}
```

But `get_build_log` with `grep=` ALSO failed with the same error. The
suggested fallback didn't work. `diagnose_build` DID work — but only
because it reads `job-output.json.gz` (a different file), not
`job-output.txt.gz`.

**Proposed fixes**:

a. **Fix the error message**: Don't suggest `get_build_log with grep=` as a
   fallback when that tool also fails on corrupted gzip. Instead suggest:
   "Use `diagnose_build` (reads job-output.json.gz instead) or
   `browse_build_logs` to find alternative log files."

b. **Try partial gzip decompression**: Python's `gzip` module can sometimes
   read partial data from a truncated/corrupted gzip stream using
   `decompressobj` with error tolerance. Try reading as much as possible
   before giving up:

   ```python
   import zlib
   def partial_decompress(data):
       d = zlib.decompressobj(16 + zlib.MAX_WBITS)
       chunks = []
       try:
           for i in range(0, len(data), 65536):
               chunks.append(d.decompress(data[i:i+65536]))
       except zlib.error:
           pass  # got what we could
       return b''.join(chunks)
   ```

c. **Fall back to json.gz automatically**: When `job-output.txt.gz` is
   corrupted and the caller used `grep=`, try extracting from
   `job-output.json.gz` instead (parse the structured JSON, search all
   `stdout`/`stderr`/`msg` fields for the grep pattern).

### 3. No way to search inside nested playbook output

**Impact**: MEDIUM — required manual Python script.

**What happened**: The failed task was `ansible.builtin.command` running
`ansible-playbook` (an inner playbook). The inner playbook's entire output
is captured in the outer task's `stdout` field. `diagnose_build` correctly
identified the outer task failure but couldn't see inside the inner
playbook to find which specific task failed.

The `inner_recap` field correctly showed `failed=1` on `osp-undercloud-0`,
confirming an inner failure exists. But the actual error was in the
truncated `stdout`.

**Proposed fix**: When `inner_recap` shows `failed > 0`, the tool should
search the full (pre-truncation) `stdout` for `fatal:` patterns and
extract them into a new field like `inner_failures`:

```json
{
  "inner_recap": "osp-undercloud-0: ok=26 failed=1",
  "inner_failures": [
    {
      "task": "Use an openshift-install flag to wait until the cluster is ready",
      "host": "localhost",
      "msg": "non-zero return code",
      "rc": 6,
      "stderr_excerpt": "failed to provision control-plane machines: machines are not ready: client rate limiter Wait returned an error"
    }
  ]
}
```

This would require parsing the inner playbook's ANSI-colored output for
`fatal:` blocks and extracting the JSON error structure.

---

## Moderate Issues

### 4. `diagnose_build` classification is shallow for nested failures

**Impact**: MEDIUM — classification said "medium confidence" and missed
the real cause.

**What happened**: `diagnose_build` returned:
```json
{
  "classification": "REAL_FAILURE",
  "classification_reason": "Task 'Run shiftstack playbook with host
    override' failed: non-zero return code",
  "classification_confidence": "medium",
  "retryable": false
}
```

This is the OUTER task classification. It knows `ansible-playbook` failed
but can't say WHY. If the tool had extracted the inner failure (Issue #3),
it could classify more precisely:

```json
{
  "classification": "REAL_FAILURE",
  "classification_reason": "Inner playbook failed: openshift-install
    bootstrap timeout (rc=4) — control-plane machines not provisioned
    within deadline",
  "classification_confidence": "high",
  "retryable": false,
  "suggested_cause": "Resource exhaustion — placement failure on
    undersized compute nodes"
}
```

### 5. `files_in_failure` empty for nested playbook failures

**Impact**: LOW — this field is useful for cross-referencing with PR
changes but not critical.

**What happened**: `diagnose_build` didn't return `files_in_failure`
because the error was in the truncated stdout, not in the visible
task output. If the inner failure were extracted (Issue #3), file
paths could be extracted from the inner playbook's task names.

### 6. Log collection verification gap

**Impact**: LOW — nice-to-have.

**What happened**: The post-run `collect_shiftstack_logs.yaml` ran
"successfully" (no failure in playbook list) but produced no output —
the `logs/undercloud/` directory was missing from the build artifacts.
The log collection failed silently because it uses `failed_when: false`.

The tools correctly showed the post-run succeeded, but there's no way
to detect that a log collection produced empty results. A potential
enhancement: `diagnose_build` could check whether expected log
directories exist in the build artifacts and flag gaps.

---

## What Worked Well

### Tools that performed correctly

| Tool | Usage | Verdict |
|------|-------|---------|
| `get_change_status` | Detect completed buildset, get per-job results | Perfect — one call got all build results |
| `diagnose_build` | Initial failure classification | Good — correctly identified failed task, provided inner_recap |
| `get_build_failures` | Detailed task-level failure data | Good — returned cmd, rc, invocation, inner_recap |
| `list_builds` | Find historical SUCCESS/FAILURE builds | Perfect |
| `browse_build_logs` | Navigate log directory tree | Perfect — fast, useful for finding alternative logs |
| Kerberos auth | SPNEGO handshake for SF Zuul | Flawless — worked on first try from custom Python script |

### Auth module reusability

The `mcp_zuul.auth.kerberos_auth()` function worked perfectly when
called directly from an ad-hoc Python script. This enabled the
workaround that ultimately solved the investigation. The auth module
is well-designed and composable.

---

## Workaround Used (for reference)

When all standard tools failed to show the error, this script extracted
the root cause in ~5 seconds:

```python
import asyncio, json, gzip, re, ssl
import httpx
from mcp_zuul.auth import kerberos_auth

async def main():
    ctx = ssl.create_default_context()
    client = httpx.AsyncClient(verify=ctx, timeout=120, follow_redirects=True)
    await kerberos_auth(client, "https://sf.../zuul")

    url = "https://sf.../logs/.../job-output.json.gz"
    resp = await client.get(url)
    data = gzip.decompress(resp.content)
    parsed = json.loads(data)

    for pb in parsed:
        for play in pb.get("plays", []):
            for task in play.get("tasks", []):
                for host, result in task.get("hosts", {}).items():
                    if result.get("failed"):
                        stdout = result.get("stdout", "")
                        if len(stdout) > 10000:
                            for pattern in ['fatal:', 'FAILED!']:
                                for m in re.finditer(pattern, stdout, re.I):
                                    start = max(0, m.start() - 200)
                                    end = min(len(stdout), m.end() + 500)
                                    print(stdout[start:end])

asyncio.run(main())
```

This should NOT be necessary. Issues #1 and #3 together would have
made `diagnose_build` surface the root cause directly.

---

## Priority Summary

| # | Issue | Impact | Effort | Priority |
|---|-------|--------|--------|----------|
| 1 | `_smart_truncate` hides mid-text errors | HIGH | Small | **P1** |
| 2 | Corrupted gzip gives up entirely | HIGH | Medium | **P1** |
| 3 | No inner playbook error extraction | MEDIUM | Medium | **P2** |
| 4 | Shallow classification for nested failures | MEDIUM | Small (depends on #3) | **P2** |
| 5 | `files_in_failure` empty for nested | LOW | Small (depends on #3) | P3 |
| 6 | Silent log collection gap detection | LOW | Medium | P3 |

**Recommended order**: #1 → #2 → #3 → #4. Issues #1 and #2 are
independent quick wins. Issue #3 enables #4 and #5.

---

## Timeline of Tool Interactions

```
00:22  get_change_status(2134)     → buildset 9ae96ca0 FAILURE
00:23  diagnose_build(09e8b09b)    → "non-zero return code", stdout truncated (1.7M → 8K)
00:24  tail_build_log(09e8b09b)    → "corrupted gzip" ERROR
00:24  get_build_log(grep=FAILED)  → "corrupted gzip" ERROR  (suggested fallback also fails!)
00:25  browse_build_logs(logs/)    → navigated directory tree, found no undercloud/ logs
00:30  get_build_failures(3892c0a7)→ earlier build also corrupted, same truncated pattern
00:35  browse_build_logs(various)  → checked controller-0, hypervisor — no inner playbook logs
00:38  curl download attempt       → Kerberos redirect, 401 (can't auth via curl)
00:41  curl with --negotiate       → 401 (OIDC flow needs browser-like cookie handling)
00:46  Custom Python script #1     → SUCCESS — found "fatal:" at pos 856878, 951710
00:47  Custom Python script #2     → Extracted full error: bootstrap timeout, rc=4/6
00:55  Custom Python script #3     → Extracted "machines are not ready: client rate limiter"
       ────────────────────────────
       ~33 min total, ~25 min fighting tools, ~8 min with workaround
```

If `diagnose_build` had extracted inner errors (Issue #1 + #3), the
root cause would have been visible at 00:23 — saving 30 minutes.
