# Eager Body Fetch Fix Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Fix CDP response body capture so auth-critical response bodies are reliably captured in HAR output, and eliminate post-Ctrl+C ConnectionRefused log spam.

**Architecture:** Three targeted changes to `NodriverCaptureBackend` in `capture.py`: (1) increase Chrome's CDP response body buffer and enable durable messages via `Network.enable()` params, (2) bypass nodriver's `_register_handlers()` overhead in the eager fetch handler by passing `_is_update=True` to `tab.send()`, (3) detect dead browser connections in `stop_capture_async` and bail early instead of per-request warnings. Also improve error logging in the eager fetch handler to include the actual exception.

**Tech Stack:** Python, nodriver (CDP), asyncio

---

### Task 1: Increase CDP buffer size in `start_capture_async`

**Files:**
- Modify: `src/graftpunk/observe/capture.py:563`
- Test: `tests/unit/test_observe.py`

**Context:** Currently `network.enable()` is called with no args, using Chrome's small default buffer. This causes Chrome to evict response bodies before the eager fetch handler can retrieve them. The `Network.enable` CDP command accepts `max_total_buffer_size`, `max_resource_buffer_size`, and `enable_durable_messages` parameters.

**Step 1: Write the failing test**

Add to `TestNodriverCaptureBackend` in `tests/unit/test_observe.py`:

```python
@pytest.mark.asyncio
async def test_start_capture_enables_network_with_buffer_params(self) -> None:
    """start_capture_async passes buffer size params to Network.enable."""
    import nodriver.cdp.network as cdp_net

    browser = MagicMock()
    tab = MagicMock()
    tab.send = AsyncMock()
    tab.add_handler = MagicMock()
    backend = NodriverCaptureBackend(browser, get_tab=lambda: tab)

    await backend.start_capture_async()

    # First tab.send call should be network.enable with buffer params
    first_call_args = tab.send.call_args_list[0]
    enable_gen = first_call_args[0][0]
    # Consume the generator to get the command dict
    cmd = next(enable_gen)
    assert cmd["method"] == "Network.enable"
    params = cmd["params"]
    assert "maxTotalBufferSize" in params
    assert "maxResourceBufferSize" in params
    assert params.get("enableDurableMessages") is True
```

**Step 2: Run test to verify it fails**

Run: `uv run pytest tests/unit/test_observe.py::TestNodriverCaptureBackend::test_start_capture_enables_network_with_buffer_params -v`
Expected: FAIL — current `network.enable()` sends no params.

**Step 3: Implement the change**

In `capture.py:563`, change:
```python
await tab.send(network.enable())  # type: ignore[attr-defined]
```
to:
```python
await tab.send(network.enable(  # type: ignore[attr-defined]
    max_total_buffer_size=100 * 1024 * 1024,      # 100 MB total
    max_resource_buffer_size=10 * 1024 * 1024,     # 10 MB per resource
    enable_durable_messages=True,                   # persist across renderer
))
```

**Step 4: Run test to verify it passes**

Run: `uv run pytest tests/unit/test_observe.py::TestNodriverCaptureBackend::test_start_capture_enables_network_with_buffer_params -v`
Expected: PASS

**Step 5: Run full test suite**

Run: `uv run pytest tests/ -v`
Expected: All tests pass.

**Step 6: Commit**

```bash
git add src/graftpunk/observe/capture.py tests/unit/test_observe.py
git commit -m "feat: increase CDP response body buffer to prevent eviction

Chrome's default CDP buffer is small and evicts response bodies before
the eager fetch handler can retrieve them. Pass max_total_buffer_size
(100MB), max_resource_buffer_size (10MB), and enable_durable_messages
to Network.enable to retain bodies longer.

Co-authored-by: stavxyz <hi@stav.xyz>"
```

---

### Task 2: Bypass `_register_handlers()` overhead in eager fetch

**Files:**
- Modify: `src/graftpunk/observe/capture.py:692-694`
- Test: `tests/unit/test_observe.py`

**Context:** nodriver's `tab.send()` calls `_register_handlers()` before sending the actual CDP command. This iterates all registered handlers and potentially sends `domain.enable()` commands, adding latency. In `_on_loading_finished`, the network domain is already enabled, so this overhead is unnecessary and delays `getResponseBody` enough for Chrome to evict the body. Passing `_is_update=True` skips this overhead. This is a nodriver internal API but we already depend heavily on nodriver internals.

**Step 1: Write the failing test**

Add to the existing `TestNodriverEagerBodyFetch` section in `tests/unit/test_observe.py`:

```python
@pytest.mark.asyncio
async def test_on_loading_finished_uses_is_update_flag(self) -> None:
    """Eager fetch sends CDP command with _is_update=True to skip handler registration."""
    browser = MagicMock()
    tab = MagicMock()
    tab.send = AsyncMock(return_value=('{"ok": true}', False))
    backend = NodriverCaptureBackend(browser, get_tab=lambda: tab)

    backend._request_map["req-fast"] = {
        "url": "https://example.com/api",
        "method": "GET",
        "headers": {},
        "post_data": None,
        "has_post_data": False,
        "timestamp": None,
        "response": {
            "status": 200,
            "statusText": "OK",
            "headers": {},
            "mimeType": "application/json",
        },
    }

    event = MagicMock()
    event.request_id = "req-fast"
    await backend._on_loading_finished(event)

    # Verify _is_update=True was passed
    tab.send.assert_called_once()
    _, kwargs = tab.send.call_args
    assert kwargs.get("_is_update") is True
```

**Step 2: Run test to verify it fails**

Run: `uv run pytest tests/unit/test_observe.py -k "test_on_loading_finished_uses_is_update_flag" -v`
Expected: FAIL — current code doesn't pass `_is_update=True`.

**Step 3: Implement the change**

In `capture.py`, change the `_on_loading_finished` handler (around line 692):
```python
            body, base64_encoded = await tab.send(
                cdp_net.get_response_body(cdp_net.RequestId(rid))  # type: ignore[attr-defined]
            )
```
to:
```python
            body, base64_encoded = await tab.send(
                cdp_net.get_response_body(cdp_net.RequestId(rid)),  # type: ignore[attr-defined]
                _is_update=True,  # skip _register_handlers — network domain already enabled
            )
```

**Step 4: Run test to verify it passes**

Run: `uv run pytest tests/unit/test_observe.py -k "test_on_loading_finished_uses_is_update_flag" -v`
Expected: PASS

**Step 5: Run full test suite**

Run: `uv run pytest tests/ -v`
Expected: All tests pass.

**Step 6: Commit**

```bash
git add src/graftpunk/observe/capture.py tests/unit/test_observe.py
git commit -m "perf: bypass _register_handlers overhead in eager body fetch

nodriver's tab.send() calls _register_handlers() before every CDP
command, iterating all handlers and potentially sending domain.enable()
calls. This adds latency to getResponseBody in the LoadingFinished
handler, giving Chrome time to evict the body. Pass _is_update=True
to skip this overhead since the network domain is already enabled.

Co-authored-by: stavxyz <hi@stav.xyz>"
```

---

### Task 3: Log actual exception in eager fetch handler

**Files:**
- Modify: `src/graftpunk/observe/capture.py:705-711`
- Test: `tests/unit/test_observe.py`

**Context:** The eager fetch handler catches `Exception` but only logs the URL, not the actual exception type or message. This makes it impossible to diagnose failures without adding temporary debug code. Adding `error=str(exc)` to the log makes the failure reason visible in verbose output.

**Step 1: Write the failing test**

Add to the eager fetch test section:

```python
@pytest.mark.asyncio
async def test_on_loading_finished_logs_actual_error(self) -> None:
    """Eager fetch failure log includes the actual exception message."""
    browser = MagicMock()
    tab = MagicMock()
    tab.send = AsyncMock(side_effect=RuntimeError("No resource with given identifier found"))
    backend = NodriverCaptureBackend(browser, get_tab=lambda: tab)

    backend._request_map["req-err"] = {
        "url": "https://example.com/api",
        "method": "GET",
        "headers": {},
        "post_data": None,
        "has_post_data": False,
        "timestamp": None,
        "response": {
            "status": 200,
            "statusText": "OK",
            "headers": {},
            "mimeType": "application/json",
        },
    }

    event = MagicMock()
    event.request_id = "req-err"

    with patch("graftpunk.observe.capture.LOG") as mock_log:
        await backend._on_loading_finished(event)

    mock_log.warning.assert_called_once()
    call_kwargs = mock_log.warning.call_args
    # Should include error= kwarg with actual exception message
    assert "No resource with given identifier found" in str(call_kwargs)
```

**Step 2: Run test to verify it fails**

Run: `uv run pytest tests/unit/test_observe.py -k "test_on_loading_finished_logs_actual_error" -v`
Expected: FAIL — current code doesn't include `error=` in the log.

**Step 3: Implement the change**

In `capture.py`, change the except block in `_on_loading_finished` (around line 705):
```python
        except Exception:  # noqa: BLE001 — best-effort eager fetch; stop_capture_async is fallback
            url = data.get("url", "unknown") if data is not None else "unknown"
            LOG.warning(
                "nodriver_eager_body_fetch_failed",
                request_id=str(getattr(event, "request_id", "unknown")),
                url=url,
            )
```
to:
```python
        except Exception as exc:  # noqa: BLE001 — best-effort eager fetch; stop_capture_async is fallback
            url = data.get("url", "unknown") if data is not None else "unknown"
            LOG.warning(
                "nodriver_eager_body_fetch_failed",
                request_id=str(getattr(event, "request_id", "unknown")),
                url=url,
                error=str(exc),
            )
```

**Step 4: Run test to verify it passes**

Run: `uv run pytest tests/unit/test_observe.py -k "test_on_loading_finished_logs_actual_error" -v`
Expected: PASS

**Step 5: Run full test suite**

Run: `uv run pytest tests/ -v`
Expected: All tests pass.

**Step 6: Commit**

```bash
git add src/graftpunk/observe/capture.py tests/unit/test_observe.py
git commit -m "fix: log actual exception in eager body fetch handler

The eager fetch handler caught Exception but only logged the URL,
making it impossible to diagnose why fetches fail. Add error=str(exc)
to the warning log so the actual CDP error message is visible.

Co-authored-by: stavxyz <hi@stav.xyz>"
```

---

### Task 4: Detect dead browser in `stop_capture_async` and bail early

**Files:**
- Modify: `src/graftpunk/observe/capture.py:574-623`
- Test: `tests/unit/test_observe.py`

**Context:** After Ctrl+C, `_save_observe_results` calls `take_screenshot`, `get_page_source`, and `stop_capture_async`. If the browser has already exited (e.g., user closed Chrome window, or Chrome crashed), every CDP call fails with `ConnectionRefusedError`. Currently, `stop_capture_async` tries each request individually and logs a warning per failure, producing dozens of identical connection-refused lines. The fix: catch connection errors and break out of the loop immediately.

**Step 1: Write the failing test**

Add to `TestNodriverCaptureBackend`:

```python
@pytest.mark.asyncio
async def test_stop_capture_bails_on_connection_error(self) -> None:
    """stop_capture_async breaks early when browser connection is dead."""
    browser = MagicMock()
    tab = MagicMock()
    tab.send = AsyncMock(side_effect=ConnectionRefusedError("[Errno 61] Connection refused"))
    backend = NodriverCaptureBackend(browser, get_tab=lambda: tab)

    # Seed multiple requests that need body fetching
    for i in range(5):
        backend._request_map[f"req-{i}"] = {
            "url": f"https://example.com/page{i}",
            "method": "GET",
            "headers": {},
            "post_data": None,
            "has_post_data": True,  # triggers post_data fetch attempt
            "timestamp": None,
            "response": {
                "status": 200,
                "statusText": "OK",
                "headers": {},
                "mimeType": "application/json",
            },
        }

    await backend.stop_capture_async()

    # Should have bailed after first connection error, not tried all 5
    assert tab.send.call_count == 1
```

**Step 2: Run test to verify it fails**

Run: `uv run pytest tests/unit/test_observe.py -k "test_stop_capture_bails_on_connection_error" -v`
Expected: FAIL — current code logs per-request and continues; `tab.send.call_count` will be 5 (or more).

**Step 3: Implement the change**

Replace the body of `stop_capture_async` (lines 574-623) with logic that catches connection errors and breaks:

```python
    async def stop_capture_async(self) -> None:
        """Stop capturing and fetch request/response bodies asynchronously."""
        tab = self._tab
        if tab is None:
            return

        import nodriver.cdp.network as cdp_net

        for request_id, data in list(self._request_map.items()):
            # Fetch POST body if has_post_data but no inline post_data
            if data.get("has_post_data") and not data.get("post_data"):
                try:
                    result = await tab.send(
                        cdp_net.get_request_post_data(cdp_net.RequestId(request_id))  # type: ignore[attr-defined]
                    )
                    data["post_data"] = result
                except (ConnectionRefusedError, ConnectionError, OSError):
                    LOG.warning("nodriver_browser_disconnected_during_stop", phase="post_data")
                    return
                except Exception as exc:  # noqa: BLE001 — CDP body fetch is best-effort
                    LOG.warning(
                        "nodriver_post_data_fetch_failed",
                        request_id=request_id,
                        error=str(exc),
                    )

            # Skip response body if already eagerly fetched by _on_loading_finished
            if request_id in self._bodies_fetched:
                continue

            # Fetch response body (fallback for any missed by eager fetch)
            response = data.get("response", {})
            mime = response.get("mimeType", "")
            if _is_text_mime(mime) or _is_binary_mime(mime):
                try:
                    body, base64_encoded = await tab.send(
                        cdp_net.get_response_body(cdp_net.RequestId(request_id))  # type: ignore[attr-defined]
                    )
                    _process_response_body(
                        response=response,
                        body=body,
                        base64_encoded=base64_encoded,
                        request_id=request_id,
                        mime=mime,
                        max_body_size=self._max_body_size,
                        bodies_dir=self._bodies_dir,
                    )
                except (ConnectionRefusedError, ConnectionError, OSError):
                    LOG.warning("nodriver_browser_disconnected_during_stop", phase="response_body")
                    return
                except Exception as exc:  # noqa: BLE001 — CDP body fetch is best-effort
                    LOG.warning(
                        "nodriver_response_body_fetch_failed",
                        request_id=request_id,
                        error=str(exc),
                    )
```

Key changes:
- Added `except (ConnectionRefusedError, ConnectionError, OSError)` before the broad `except Exception` in both the post_data and response_body fetch blocks.
- On connection error: log once with the phase, then `return` (exit the method entirely).
- Non-connection errors still log per-request and continue as before.

**Step 4: Run test to verify it passes**

Run: `uv run pytest tests/unit/test_observe.py -k "test_stop_capture_bails_on_connection_error" -v`
Expected: PASS

**Step 5: Run full test suite**

Run: `uv run pytest tests/ -v`
Expected: All tests pass.

**Step 6: Commit**

```bash
git add src/graftpunk/observe/capture.py tests/unit/test_observe.py
git commit -m "fix: bail early in stop_capture_async when browser is dead

After Ctrl+C or browser crash, every CDP call in stop_capture_async
fails with ConnectionRefusedError. Previously each failure was logged
individually, producing dozens of identical warning lines. Now detect
connection errors and return immediately with a single log message.

Co-authored-by: stavxyz <hi@stav.xyz>"
```

---

### Task 5: Lint, format, type check, and create PR

**Files:**
- All modified files from tasks 1-4

**Step 1: Run quality checks**

```bash
uvx ruff check --fix . && uvx ruff format . && uvx ty check src/
```

Fix any issues.

**Step 2: Run full test suite one final time**

```bash
uv run pytest tests/ -v
```

Expected: All tests pass.

**Step 3: Push and create PR**

```bash
git push -u origin fix/eager-body-fetch
gh pr create --title "fix: improve CDP response body capture reliability" --body "$(cat <<'EOF'
## Summary

- Increase Chrome's CDP response body buffer (100MB total, 10MB per resource) and enable durable messages to prevent body eviction before eager fetch can retrieve them
- Bypass nodriver's `_register_handlers()` overhead in the eager fetch handler via `_is_update=True` to reduce latency between `LoadingFinished` event and `getResponseBody` command
- Detect dead browser connections in `stop_capture_async` and bail immediately instead of logging per-request `ConnectionRefusedError` warnings
- Include actual exception message in eager fetch warning logs for diagnosability

Fixes response body capture for auth-critical requests (login endpoints, token exchanges) that were being lost due to Chrome buffer eviction.

## Test plan

- [ ] New test: `start_capture_async` passes buffer params to `Network.enable`
- [ ] New test: `_on_loading_finished` passes `_is_update=True` to `tab.send`
- [ ] New test: eager fetch failure log includes actual error message
- [ ] New test: `stop_capture_async` bails after first `ConnectionRefusedError`
- [ ] All existing capture tests still pass
- [ ] Manual: `gp observe --no-session interactive <url>` — verify fewer eager fetch warnings and no ConnectionRefused spam after Ctrl+C
EOF
)"
```
