# Plan: `fetch_url` Tool

## Goal

Add a `fetch_url` tool that retrieves web page content and returns it as markdown (default), plain text, or raw HTML. Inspired by opencode's `webfetch` tool.

## Tool Schema

```json
{
  "type": "function",
  "function": {
    "name": "fetch_url",
    "description": "Fetch the content of a URL and return it as markdown, plain text, or raw HTML. Use this to read documentation, APIs, web pages, or any HTTP-accessible resource.",
    "parameters": {
      "type": "object",
      "properties": {
        "url": {
          "type": "string",
          "description": "The URL to fetch (must start with http:// or https://)"
        },
        "format": {
          "type": "string",
          "enum": ["markdown", "text", "html"],
          "description": "Output format. 'markdown' converts HTML to readable markdown (default). 'text' extracts plain text. 'html' returns raw HTML."
        },
        "timeout": {
          "type": "integer",
          "description": "Request timeout in seconds (1-120, default 30)"
        }
      },
      "required": ["url"]
    }
  }
}
```

## Implementation: `fetch.py`

New file, keeps `tools.py` focused on routing. Roughly 120-170 lines.

### Core function

```python
def fetch_url(url: str, format: str = "markdown", timeout: int = 30) -> str
```

Returns content string on success, `"error: ..."` on failure (matching existing convention).

### URL validation and network restrictions

- Must start with `http://` or `https://`. Reject everything else.
- **Block private/internal network addresses.** After parsing the hostname, resolve it to an IP and reject:
  - Loopback: `127.0.0.0/8`, `::1`
  - Private: `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`
  - Link-local: `169.254.0.0/16`, `fe80::/10`
  - Metadata endpoints: `169.254.169.254` (cloud metadata)

  Use `ipaddress.ip_address()` with its `.is_private`, `.is_loopback`, `.is_link_local` properties (stdlib). This prevents prompt-injection chains from using the agent to probe internal services or exfiltrate data to localhost listeners.

```python
import ipaddress
import socket

def _check_url_safety(url: str) -> str | None:
    """Return an error string if the URL has a bad scheme or targets a private address, else None."""
    parsed = urllib.parse.urlparse(url)
    if parsed.scheme not in ("http", "https"):
        return f"error: url scheme {parsed.scheme!r} is not allowed, must be http or https"
    hostname = parsed.hostname
    if not hostname:
        return "error: could not parse hostname from url"
    try:
        infos = socket.getaddrinfo(hostname, None, socket.AF_UNSPEC, socket.SOCK_STREAM)
    except socket.gaierror as e:
        return f"error: could not resolve hostname {hostname!r}: {e}"
    for family, _, _, _, sockaddr in infos:
        addr = ipaddress.ip_address(sockaddr[0])
        if addr.is_private or addr.is_loopback or addr.is_link_local or addr.is_reserved:
            return f"error: url resolves to private/internal address ({addr}), blocked for security"
    return None
```

### Timeout validation

Clamp timeout to the valid range before use (matching `run_command`'s pattern at `tools.py:785`):

```python
timeout = max(1, min(timeout, 120))
```

### HTTP fetching

Use `urllib.request` with the custom redirect-safe opener (see Redirect handling below). Headers:

```python
headers = {
    "User-Agent": "Mozilla/5.0 ...",  # Browser-like UA to avoid bot blocks
    "Accept": "text/html,text/plain,text/markdown,application/json,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
```

The fetch loop (described in Redirect handling) calls `opener.open(req, timeout=timeout)` and reads up to `MAX_RESPONSE_SIZE + 1` bytes from the response.

### Size limits

- `MAX_RESPONSE_SIZE = 5 * 1024 * 1024` (5 MB raw download cap)
- After conversion, truncate output to `MAX_OUTPUT_BYTES` (50 KB, same as `read_file`). Truncation is measured in **bytes** (UTF-8 encoded length), consistent with existing tools. Truncate at a character boundary to avoid splitting multi-byte sequences the hint: `[content truncated at N bytes, total was M bytes]`

### Format handling

**`html`** -- Return the decoded body as-is (after size check).

**`text`** -- Strip HTML tags and extract readable text. Use `html.parser` from stdlib:

- Subclass `html.parser.HTMLParser` to walk the document
- Skip content inside `<script>`, `<style>`, `<noscript>`, `<svg>` tags
- Insert newlines after block elements (`<p>`, `<div>`, `<br>`, `<h1>`-`<h6>`, `<li>`, etc.)
- Decode HTML entities via `html.unescape()`
- Collapse excessive whitespace/blank lines

**`markdown`** -- Use the `html-to-markdown` package:

```python
from html_to_markdown import convert

markdown = convert(html_body)
```

The library handles script/style removal, headings, links, images, code blocks, lists, tables, etc. out of the box. No custom converter needed.

### Error handling

All errors return `"error: ..."` strings (never raise), matching existing tool convention:

- Invalid URL scheme: `"error: url must start with http:// or https://"`
- Private address: `"error: url resolves to private/internal address (127.0.0.1), blocked for security"`
- Timeout: `"error: request timed out after N seconds"`
- HTTP errors: `"error: HTTP 404 — Not Found"` (include status + reason)
- Too large: `"error: response too large (N bytes, limit is 5MB)"`
- Connection errors: `"error: could not connect to <host>: <reason>"`
- Encoding errors: `"error: could not decode response as text"`

### Character encoding

- Check `Content-Type` header for `charset=`
- Fall back to UTF-8
- If decoding fails, try `latin-1` as last resort (it never fails but may be wrong)
- Binary content (images, PDFs, etc.): detect via content-type or null-byte check, return `"error: binary content (content-type: image/png), cannot display as text"`

### Redirect handling

Disable automatic redirects and handle them manually to prevent redirect-based SSRF. A public URL can return a 302 pointing to `http://127.0.0.1/...` or a cloud metadata endpoint, bypassing the initial `_check_url_safety` check.

Use a custom `urllib.request` handler that intercepts redirects instead of following them:

```python
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        raise _RedirectError(newurl, code)

class _RedirectError(Exception):
    def __init__(self, url: str, code: int):
        self.url = url
        self.code = code
```

The fetch loop then:
1. Builds an opener with `_NoRedirectHandler`
2. On `_RedirectError`, runs `_check_url_safety(redirect_url)` on the new target
3. If safe, retries with the new URL
4. Caps redirect depth at 10 (return error on the 11th)

```python
MAX_REDIRECTS = 10
opener = urllib.request.build_opener(_NoRedirectHandler)
for _ in range(MAX_REDIRECTS + 1):
    err = _check_url_safety(current_url)
    if err:
        return err
    req = urllib.request.Request(current_url, headers=headers)
    try:
        resp = opener.open(req, timeout=timeout)
        break  # success, no redirect
    except _RedirectError as r:
        current_url = urllib.parse.urljoin(current_url, r.url)
    except ...
else:
    return "error: too many redirects (limit is 10)"
```

Redirect targets from the `Location` header can be relative (e.g. `/login`), so `urljoin` resolves them against the current URL before safety-checking. This is necessary because most real servers use relative redirects.

This ensures every URL in the redirect chain is validated against the private-address blocklist before any connection is made to it.

**Residual risk: DNS rebinding.** A hostname could resolve to a public IP during `_check_url_safety` and then to a private IP when `opener.open` actually connects. Mitigating this would require socket-level connection pinning, which adds significant complexity. Accepted as residual risk since the output only flows back to the model as text, not to an attacker-controlled channel.

## Integration into tools.py

### TOOLS list

Add `FETCH_URL_TOOL` schema as a separate constant (like `RUN_COMMAND_TOOL`). Always included in the tools list -- unlike `run_command` (which executes arbitrary commands), `fetch_url` is read-only with network safety checks. A `--no-fetch` flag can be added later if needed using the established conditional pattern.

### Dispatch

Add to `dispatch()`:

```python
if name == "fetch_url":
    from fetch import fetch_url as _fetch_url
    return _fetch_url(
        args.get("url", ""),
        args.get("format", "markdown"),
        args.get("timeout", 30),
    )
```

No extra kwargs needed (no base_dir dependency, no state). Lazy import to keep startup fast.

## Wheel packaging

Add `fetch.py` to the `only-include` list in `pyproject.toml` so it ships in the wheel:

```toml
[tool.hatch.build.targets.wheel]
packages = ["."]
only-include = ["agent.py", "tools.py", "edit.py", "thinking.py", "fetch.py", "system_prompt.txt"]
```

Also adding `thinking.py` and `system_prompt.txt`, both missing from the current list. `system_prompt.txt` is loaded at startup by `agent.py:523` when `--no-system-prompt` is not set, so omitting it from the wheel breaks packaged installs.

## System prompt update

Add a short section to `system_prompt.txt` documenting the tool:

```
## fetch_url
Fetch web page content. Returns markdown by default.
- Use for reading documentation, API references, web pages
- Set format="text" for cleaner extraction from complex pages
- Set format="html" if you need the raw markup
```

## Tests: `tests/test_fetch.py`

Test the conversion functions and error handling. Mock `opener.open` (the custom opener built with `_NoRedirectHandler`) and `socket.getaddrinfo` to avoid real HTTP calls. The opener is the actual call site -- mocking `urlopen` would miss the redirect logic entirely.

### Test cases

1. **HTML to text**: Feed sample HTML, verify script/style stripped, text preserved
2. **HTML to markdown**: Feed sample HTML, verify conversion via `html-to-markdown`
3. **Raw HTML**: Verify passthrough
4. **URL validation**: Reject `ftp://`, `file://`, empty string, no-scheme
5. **Private address blocking**: `127.0.0.1`, `192.168.1.1`, `10.0.0.1`, `169.254.169.254`, `::1` all rejected
6. **Redirect safety**: Mock `opener.open` to raise `_RedirectError`. Cases: absolute redirect to `http://127.0.0.1/secret` (blocked), absolute redirect to `http://169.254.169.254/latest/meta-data/` (blocked), redirect to non-HTTP scheme like `ftp://evil.com/file` (blocked by per-hop scheme check), relative redirect `/login` resolved via `urljoin` to the original host (allowed), redirect chain to a safe public URL (succeeds), and the 10-redirect depth limit (error).
7. **Timeout clamping**: Values below 1 clamped to 1, above 120 clamped to 120
8. **Size limit**: Response over 5MB returns error
9. **Binary detection**: Content-type `image/png` returns error
10. **HTTP errors**: 404, 500 return appropriate error strings
11. **Encoding**: UTF-8, latin-1 fallback, charset in content-type header
12. **Output truncation**: Large converted output truncated at byte boundary with hint
13. **Dispatch wiring**: `dispatch("fetch_url", {...}, base_dir)` routes correctly

## Files changed

| File | Change |
|---|---|
| `fetch.py` | New file. `fetch_url()`, `_html_to_text()`, `_check_url_safety()`, URL validation, encoding detection, timeout clamping. |
| `pyproject.toml` | Add `html-to-markdown` dependency. Add `fetch.py`, `thinking.py`, and `system_prompt.txt` to `only-include` wheel list. |
| `tools.py` | Add `FETCH_URL_TOOL` schema to `TOOLS`. Add dispatch branch for `"fetch_url"`. |
| `system_prompt.txt` | Add `fetch_url` tool documentation section. |
| `tests/test_fetch.py` | New file. Unit tests for all conversion, security, and error paths. |

## Non-goals

- **No image/binary download support.** This is a text tool. Binary URLs get an error message.
- **No JavaScript rendering.** We fetch the raw HTML. SPAs that need JS won't work. That's fine -- the model can be told to try a different approach.
- **No caching.** Each call fetches fresh. The agent loop is short-lived.
- **No authentication.** If a URL needs auth headers, the user can use `run_command` with `curl` instead.
- **No robots.txt checking.** This is a user-initiated tool, not a crawler.

## New dependency

**`html-to-markdown`** -- Rust-backed HTML-to-markdown converter. Handles headings, links, images, code blocks, lists, tables, and automatically strips script/style tags. Added to `pyproject.toml`. Pre-built wheels are available for Linux/macOS/Windows on Python 3.10+, so no Rust toolchain needed at install time.
