# Web Forager

![Illustration of a determined scribe wielding a giant quill fighting a tangle of papers and monsters, with a duck in a cap at his side and stacks of documents and crates behind](assets/header.png)

[![PyPI](https://img.shields.io/pypi/v/web-forager?style=flat-square)](https://pypi.org/project/web-forager/)
[![Python Version](https://img.shields.io/pypi/pyversions/web-forager?style=flat-square)](https://pypi.org/project/web-forager/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=flat-square)](https://opensource.org/licenses/MIT)
[![Downloads](https://static.pepy.tech/badge/web-forager/month)](https://pepy.tech/project/web-forager)

*The thing about information on the web is that it doesn't want to be found. It wants to hide behind cookie banners, keep itself to itself, and generally behave like a cat that knows it's time for the vet. Web Forager is the sort of dogged, slightly grubby assistant who goes out there anyway — accompanied by a duck of questionable temperament — rummages through DuckDuckGo, tries Exa when DuckDuckGo pretends not to be home, fetches pages via Jina Reader, and when Jina is having one of its days, simply grabs them by hand. The results come back neatly converted for LLM consumption, which is to say, in a format that would make a librarian weep with either joy or despair, depending on the librarian.*

A search-and-fetch toolkit for AI agents, available as an MCP server and as standalone Agent Skills:
1. **Search** the web via DuckDuckGo
2. **Search news** via DuckDuckGo News
3. **Fetch** and convert web pages via Jina Reader

Also ships five **[Agent Skills](#agent-skills)** that work independently — no MCP required — for research, fact-checking, news monitoring, competitive analysis, and technology evaluation.

## Features

- DuckDuckGo web search with safe search controls
- DuckDuckGo news search with date-sorted results and source attribution
- Fetch and convert URLs to markdown or JSON using Jina Reader
- LLM-friendly output format option for search results
- CLI for search, news, fetch, serve, and version commands
- MCP tools for LLM integration
- Five standalone Agent Skills for specialized research workflows
- Docker support for containerized deployment

## Installation

### Prerequisites

- Python 3.10 or higher
- [uv](https://github.com/astral-sh/uv) (recommended) or pip

### Install from PyPI (recommended)

```bash
# Using uv (recommended)
uv pip install web-forager

# Or using pip
pip install web-forager
```

### Install with UVX (for Claude Desktop)

```bash
# Install UVX if you haven't already
pip install uvx

# Install the Web Forager package
uvx install web-forager
```

### Install from source

For development or to get the latest changes:

```bash
# Clone the repository
git clone https://github.com/CyranoB/web-forager.git
cd web-forager

# Install with uv (recommended)
uv pip install -e .

# Or with pip
pip install -e .
```

### Docker

Build and run with Docker:

```bash
# Build the image (uses version from latest git tag)
docker build --build-arg VERSION=$(git describe --tags --abbrev=0 | sed 's/^v//') -t web-forager .

# Or specify a version manually
docker build --build-arg VERSION=2.0.2 -t web-forager .

# Run the server (MCP servers use STDIO, so typically run within an MCP client)
docker run -i web-forager
```

## Usage

### Starting the Server (STDIO Mode)

```bash
# Start the server in STDIO mode (for use with MCP clients like Claude)
web-forager serve

# Enable debug logging
web-forager serve --debug
```

### Testing the Search Tool

```bash
# Search DuckDuckGo (JSON output, default)
web-forager search "your search query" --max-results 5 --safesearch moderate

# Search with LLM-friendly text output
web-forager search "your search query" --output-format text
```

### Testing the News Search Tool

```bash
# Search DuckDuckGo news (JSON output, default)
web-forager news "your search query" --max-results 10 --safesearch moderate

# Search with LLM-friendly text output
web-forager news "your search query" --output-format text
```

### Testing the Fetch Tool

```bash
# Fetch a URL and return markdown
web-forager fetch "https://example.com" --format markdown

# Fetch a URL and return JSON
web-forager fetch "https://example.com" --format json

# Limit output length
web-forager fetch "https://example.com" --max-length 2000

# Include generated image alt text
web-forager fetch "https://example.com" --with-images
```

### Version Information

```bash
# Show version
web-forager version

# Show detailed version info
web-forager version --debug
```

## MCP Client Setup

This MCP server works with any MCP-compatible client. Use one of the setups below.

Python 3.10-3.13 is supported (3.14 not yet). Use `--python ">=3.10,<3.14"` with `uvx` to enforce. Verified with Python 3.12 and 3.13.

### Claude Desktop

1. Open Claude Desktop > Settings > Developer > Edit Config.
2. Edit the config file:
   - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
   - Windows: `%APPDATA%\Claude\claude_desktop_config.json`
3. Add the server config under `mcpServers`:
   ```json
    {
      "mcpServers": {
        "web-forager": {
          "command": "uvx",
          "args": ["--python", ">=3.10,<3.14", "web-forager", "serve"]
        }
      }
    }

   ```
4. Restart Claude Desktop.

### Claude Code

Add a local stdio server:

```bash
claude mcp add --transport stdio web-forager -- uvx --python ">=3.10,<3.14" web-forager serve
```

Optional: `claude mcp list` to verify, or `claude mcp add-from-claude-desktop` to import.

### Codex (CLI + IDE)

Add via CLI:

```bash
codex mcp add web-forager -- uvx --python ">=3.10,<3.14" web-forager serve
```

Or configure `~/.codex/config.toml`:

```toml
[mcp_servers.web-forager]
command = "uvx"
args = ["--python", ">=3.10,<3.14", "web-forager", "serve"]
```

### OpenCode

Add to your OpenCode config (`~/.config/opencode/opencode.json` or project `opencode.json`):

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "web-forager": {
      "type": "local",
      "command": ["uvx", "--python", ">=3.10,<3.14", "web-forager", "serve"],
      "enabled": true
    }
  }
}
```

Or run `opencode mcp add` and follow the prompts.

### Cursor

Add to `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project):

```json
{
  "mcpServers": {
    "web-forager": {
      "command": "uvx",
      "args": ["--python", ">=3.10,<3.14", "web-forager", "serve"]
    }
  }
}
```

Verify with:

```bash
cursor-agent mcp list
```

## MCP Tools

The server exposes these tools to MCP clients:

```python
@mcp.tool()
def duckduckgo_search(
    query: str,
    max_results: int = 5,
    safesearch: str = "moderate",
    output_format: str = "json"
) -> list | str:
    """Search DuckDuckGo for the given query."""
```

```python
@mcp.tool()
def duckduckgo_news_search(
    query: str,
    max_results: int = 10,
    safesearch: str = "moderate",
    output_format: str = "json"
) -> list | str:
    """Search DuckDuckGo for recent news articles."""
```

```python
@mcp.tool()
def jina_fetch(url: str, format: str = "markdown", max_length: int | None = None, with_images: bool = False) -> str | dict:
    """Fetch a URL and convert it using Jina Reader."""
```

Example usage in an MCP client:

```python
# This is handled automatically by the MCP client
results = duckduckgo_search("Python programming", max_results=3)
news = duckduckgo_news_search("AI regulation 2026", max_results=5)
content = jina_fetch("https://example.com", format="markdown")

# Get LLM-friendly text output
text_results = duckduckgo_search("Python programming", output_format="text")
```

## API

### Tool 1: Search

- **Tool Name**: `duckduckgo_search`
- **Description**: Search the web using DuckDuckGo (powered by the `ddgs` library)

#### Parameters

- `query` (string, required): The search query
- `max_results` (integer, optional, default: 5): Maximum number of search results to return
- `safesearch` (string, optional, default: "moderate"): Safe search setting ("on", "moderate", or "off")
- `output_format` (string, optional, default: "json"): Output format - "json" for structured data, "text" for LLM-friendly formatted string

#### Response

**JSON format** (default): A list of dictionaries:

```json
[
  {
    "title": "Result title",
    "url": "https://example.com",
    "snippet": "Text snippet from the search result"
  }
]
```

**Text format**: An LLM-friendly formatted string:

```
Found 3 search results:

1. Result title
   URL: https://example.com
   Summary: Text snippet from the search result

2. Another result
   URL: https://example2.com
   Summary: Another snippet
```

### Tool 2: News Search

- **Tool Name**: `duckduckgo_news_search`
- **Description**: Search for recent news articles using DuckDuckGo (powered by the `ddgs` library)

#### Parameters

- `query` (string, required): The news search query
- `max_results` (integer, optional, default: 10): Maximum number of news results to return
- `safesearch` (string, optional, default: "moderate"): Safe search setting ("on", "moderate", or "off")
- `output_format` (string, optional, default: "json"): Output format - "json" for structured data, "text" for LLM-friendly formatted string

#### Response

**JSON format** (default): A list of dictionaries:

```json
[
  {
    "title": "News headline",
    "url": "https://example.com/article",
    "snippet": "Article summary text",
    "date": "2026-03-01T12:00:00+00:00",
    "source": "News Outlet"
  }
]
```

**Text format**: An LLM-friendly formatted string:

```
Found 3 news results:

1. News headline
   URL: https://example.com/article
   Date: 2026-03-01T12:00:00+00:00
   Source: News Outlet
   Summary: Article summary text
```

### Tool 3: Fetch

- **Tool Name**: `jina_fetch`
- **Description**: Fetch a URL and convert it to markdown or JSON using Jina Reader

#### Parameters

- `url` (string, required): The URL to fetch and convert
- `format` (string, optional, default: "markdown"): Output format ("markdown" or "json")
- `max_length` (integer, optional): Maximum content length to return (None for no limit)
- `with_images` (boolean, optional, default: false): Whether to include image alt text generation

#### Response

For markdown format: a string containing markdown content

For JSON format: a dictionary with the structure:

```json
{
  "url": "https://example.com",
  "title": "Page title",
  "content": "Markdown content"
}
```

## Agent Skills

This repo includes five **Agent Skills** that orchestrate the MCP's search and fetch tools into specialized workflows. Each skill follows the open [Agent Skills](https://agentskills.io/) specification and works with Claude Code, Codex CLI, and other compatible agents.

All skills work **without** the MCP configured — they use the `ddgs` Python library and the Jina Reader HTTP API directly. If MCP tools are available in the session, they prefer those automatically.

### Install via Plugin Marketplace (recommended)

Register this repo as a plugin marketplace, then install all five skills at once:

```bash
# Add the marketplace
/plugin marketplace add CyranoB/web-forager

# Install all 5 skills
/plugin install forager-skills@web-forager
```

### Install individual skills

**Claude Code:**
```bash
# Install a specific skill
claude install-skill ./skills/web-research

# Or from GitHub
claude install-skill github:CyranoB/web-forager/skills/web-research
```

**Manual (any agent):**
Copy a skill folder from `skills/` into your agent's skills directory (e.g., `~/.claude/skills/` or `.claude/skills/`).

### Available skills

| Skill | Triggers on | Output |
|-------|-------------|--------|
| **[web-research](skills/web-research/)** | "research X", "look up X", "deep dive into X" | Adaptive report (quick answer / standard / deep dive) with citations |
| **[fact-check](skills/fact-check/)** | "is it true that X", "verify this claim", "fact check this" | Verdict (Confirmed -> False) with evidence for and against |
| **[news-monitor](skills/news-monitor/)** | "what's new with X", "recent news about X", "catch me up on X" | Chronological news briefing with headlines and details |
| **[competitive-intel](skills/competitive-intel/)** | "compare X vs Y", "which is better", "help me choose between" | Comparison matrix with pricing, pros/cons, and recommendation |
| **[tech-advisor](skills/tech-advisor/)** | "should we adopt X", "is X production ready", "X vs Y for my needs" | Maturity scorecard (Adopt/Trial/Assess/Hold) or product comparison with evidence |

## Notes

- Search and news search use the `ddgs` package (renamed from `duckduckgo-search`).
- Fetch uses the Jina Reader API at `https://r.jina.ai/`.

## Contributing

Contributions are welcome! Here's how you can contribute:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## Support

If you encounter any issues or have questions, please [open an issue](https://github.com/CyranoB/web-forager/issues).

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
