[**🇨🇳中文**](https://github.com/shibing624/webresearcher/blob/main/README.md) | [**🌐English**](https://github.com/shibing624/webresearcher/blob/main/README_EN.md) 

<p align="center">
  <img src="./docs/webresearcher.jpg" alt="WebResearcher Logo" height="150" alt="Logo"/>
</p>

---

# WebResearcher: An Iterative Deep-Research Agent
<p align="center">
  <a href="https://pypi.org/project/webresearcher/"><img src="https://img.shields.io/pypi/v/webresearcher.svg" alt="PyPI version"></a>
  <a href="https://pepy.tech/project/webresearcher"><img src="https://static.pepy.tech/badge/webresearcher" alt="Downloads"></a>
  <a href="https://github.com/shibing624/WebResearcher/blob/main/LICENSE"><img src="https://img.shields.io/github/license/shibing624/WebResearcher.svg" alt="License"></a>
  <a href="https://pypi.org/project/webresearcher/"><img src="https://img.shields.io/badge/Python-3.10%2B-green.svg" alt="Python versions"></a>
  <a href="https://arxiv.org/abs/2509.13309"><img src="https://img.shields.io/badge/arXiv-2509.13309-b31b1b.svg" alt="arXiv"></a>
  <a href="https://github.com/shibing624/WebResearcher"><img src="https://img.shields.io/badge/wechat-group-green.svg?logo=wechat" alt="Wechat Group"></a>
</p>

- 🧠 **Iterative Deep-Research**: Novel paradigm that prevents context overflow through periodic synthesis
- 🔄 **Unbounded Reasoning**: Practically unlimited research depth via evolving summary reports
- 🛠️ **Rich Tool Ecosystem**: Web search, academic papers, code execution, file parsing
- 🎯 **Production Ready**: Zero external agent framework dependencies, fully self-contained
- ⚡ **High Performance**: Async-first design, smart token management, robust error handling
- 🎨 **Easy to Use**: Simple CLI, clean Python API, modern WebUI, extensive examples
- 📱 **Cross-Platform**: Perfect desktop and mobile support with responsive design

## 📖 Introduction

**WebResearcher** is an autonomous research agent built on the **IterResearch paradigm**, designed to emulate expert-level research workflows. Unlike traditional agents that suffer from context overflow and noise accumulation, WebResearcher breaks research into discrete rounds with iterative synthesis.

This project provides two research agents:
- **WebResearcher Agent**: Single-agent iterative research, ideal for quick answers
- **WebWeaver Agent**: Dual-agent collaborative research, ideal for structured long-form reports

### The Problem with Traditional Agents

Current open-source research agents rely on **mono-contextual, linear accumulation**:

1. **🚫 Cognitive Workspace Suffocation**: Ever-expanding context constrains deep reasoning
2. **🚫 Irreversible Noise Contamination**: Errors and irrelevant info accumulate
3. **🚫 Lack of Synthesis**: No pausing to distill, re-evaluate, and plan strategically

### The WebResearcher Solution

WebResearcher implements the **IterResearch paradigm**, where each round involves a **single LLM call** that simultaneously generates:

- **Plan**: Internal reasoning and analysis
- **Report**: Updated research summary synthesizing all findings so far
- **Action**: Tool call or final answer

This **one-step approach** (vs. traditional two-step "plan→act→synthesize") delivers:
- ⚡ **50% faster** - One LLM call per round instead of two
- 💰 **40% cheaper** - Reduced token usage
- 🧠 **Better reasoning** - Plan, Report, and Action generated in unified context

This enables **unbounded research depth** while maintaining a lean, focused cognitive workspace.

<p align="center">
  <img src="https://github.com/shibing624/WebResearcher/blob/main/docs/iterresearch.png" alt="Paradigm Comparison" width="100%"/>
  <br>
  <em>Figure: Mono-contextual Paradigm (Top) vs. Iterative Deep-Research Paradigm (Bottom)</em>
</p>

## 🏗️ Architecture

### Core Components

**IterResearch Paradigm - Single LLM Call Per Round:**

```python
Round i:
┌─────────────────────────────────────────────────────────┐
│  Workspace State: (Question, Report_{i-1}, Result_{i-1}) │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  Single LLM Call → Generates All Three:                  │
│  ├─ <plan>: Analyze current state                       │
│  ├─ <report>: Updated synthesis of all findings          │
│  └─ <tool_call> or <answer>: Next action                 │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│  If <tool_call>: Execute Tool                            │
│  If <answer>: Return Final Answer                        │
└─────────────────────────────────────────────────────────┘
                          ↓
        Next Round with Updated Report and Tool Result
```

**Key Advantage**: Report is synthesized *before* deciding the next action, ensuring coherent reasoning in a unified context.

### Available Tools

| Tool | Description | Use Case |
|------|-------------|----------|
| `search` | Google Search via Serper API | General web information |
| `google_scholar` | Academic paper search | Scientific research |
| `visit` | Webpage content extraction | Deep content analysis |
| `python` | Sandboxed code execution | Data analysis, calculations |
| `parse_file` | Multi-format file parser | Document processing |

## 🚀 Quick Start

### Installation

```bash
pip install webresearcher
```

### WebUI (Recommended)

WebResearcher provides a modern web interface with desktop and mobile support:

```bash
# Start WebUI service
cd webui
python app.py

# Access at http://localhost:8000
```

<img src="https://github.com/shibing624/WebResearcher/blob/main/docs/webui-snap.png" alt="webui" width="100%"/>

**WebUI Features:**
- 🎨 **Modern Interface**: Clean conversational UI
- 📱 **Mobile Optimized**: Perfect support for phones and tablets
- 🔄 **Real-time Streaming**: Live visualization of research process
- 📚 **History Management**: Save, edit, and delete sessions
- 🎯 **Process Visualization**: Step-by-step display of plans, reports, and tool calls
- ⚙️ **Flexible Configuration**: Custom instructions and tool selection

See [WebUI Documentation](./webui/README.md) for details.

### CLI Basic Usage

```bash
# Set your API keys
export LLM_API_KEY="your_key"
export SERPER_API_KEY="your_key"

# Run a research query
webresearcher "Who won the Nobel Prize in Physics in 2024?"
```

### Python API

```python
import asyncio
from webresearcher import WebResearcherAgent

# Configure
llm_config = {
    "model": "gpt-4o",
    "api_key": "your-api-key",      # Optional, defaults to LLM_API_KEY env var
    "base_url": "https://api.openai.com/v1",  # Optional, defaults to LLM_BASE_URL env var
    "generate_cfg": {"temperature": 0.6}
}

# Create agent (can also pass api_key and base_url directly as parameters)
agent = WebResearcherAgent(
    llm_config=llm_config,
    function_list=["search", "google_scholar", "python"],
    api_key="your-api-key",    # Optional, overrides llm_config setting
    base_url="https://api.openai.com/v1"  # Optional
)

# Research
async def main():
    result = await agent.run("Your research question")
    print(result['prediction'])

asyncio.run(main())
```

### Multi-Turn ReAct: ReactAgent

If you prefer a multi-turn dialog implementation closer to the ReAct paper, this project provides a `ReactAgent`.

Usage example:

```python
import asyncio
from webresearcher.react_agent import ReactAgent

llm_config = {
    "model": "gpt-4o",
    "api_key": "your-api-key",      # Optional, defaults to LLM_API_KEY env var
    "base_url": "https://api.openai.com/v1",  # Optional, defaults to LLM_BASE_URL env var
    "generate_cfg": {"temperature": 0.6}
}

agent = ReactAgent(
    llm_config=llm_config,
    function_list=["search", "google_scholar", "visit", "python"],
    api_key="your-api-key",    # Optional, overrides llm_config setting
    base_url="https://api.openai.com/v1"  # Optional
)

async def main():
    result = await agent.run("What is the population of Paris in 2024? Also compute its square root.")
    # Result dict includes: question / prediction / termination / trajectory
    print(result["prediction"])  # Always non-empty

asyncio.run(main())
```

Message trajectory illustration (for logging/inspection):
- system: system prompt
- user: original question
- user: merged tool interaction (`<tool_call>...</tool_call>` + `OBS_START/OBS_END` tool output)
- assistant: model proceeds or emits final `<answer>`

## 📚 Advanced Usage

### Test-Time Scaling (TTS)

For critical questions requiring maximum accuracy, use TTS mode (3-5x cost):

```bash
webresearcher "Complex question" --use-tts --num-agents 3
```

```python
from webresearcher import TestTimeScalingAgent

agent = TestTimeScalingAgent(llm_config, function_list)
result = await agent.run("Complex question", num_parallel_agents=3)
```

### Custom Tools

Create your own tools by extending `BaseTool`:

```python
from webresearcher import BaseTool, WebResearcherAgent, TOOL_MAP

class MyCustomTool(BaseTool):
    name = "my_tool"
    description = "What my tool does"
    parameters = {"type": "object", "properties": {...}}
    
    def call(self, params, **kwargs):
        # Your tool logic
        return "result"

# Register and use
TOOL_MAP['my_tool'] = MyCustomTool()
agent = WebResearcherAgent(llm_config, function_list=["my_tool", "search"])
```

See [examples/custom_agent.py](./examples/custom_agent.py) for full examples.

### Batch Processing

Process multiple questions efficiently:

```python
from webresearcher import WebResearcherAgent

questions = ["Question 1", "Question 2", "Question 3"]
agent = WebResearcherAgent(llm_config)

for question in questions:
    result = await agent.run(question)
    print(f"Q: {question}\nA: {result['prediction']}\n")
```

See [examples/batch_research.py](./examples/batch_research.py) for advanced batch processing.

### Python Interpreter Configuration

The `python` tool supports two execution modes:

**1. Sandbox Mode (Recommended for Production):**
```bash
# Configure sandbox endpoints
export SANDBOX_FUSION_ENDPOINTS="http://your-sandbox-endpoint.com"
```

**2. Local Mode (Automatic Fallback):**
- When `SANDBOX_FUSION_ENDPOINTS` is not configured, code executes locally
- Useful for development and testing
- ⚠️ **Warning**: Local execution runs code in the current Python environment

```python
from webresearcher import PythonInterpreter

# Will use sandbox if configured, otherwise falls back to local execution
interpreter = PythonInterpreter()
result = interpreter.call({'code': 'print("Hello, World!")'})
```

See [examples/python_interpreter_example.py](./examples/python_interpreter_example.py) for more examples.

### Logging Management

WebResearcher provides unified logging control for the entire package. You can control log levels via environment variables or programmatically:

**Via Environment Variable:**

```bash
# Set log level before running
export WEBRESEARCHER_LOG_LEVEL=DEBUG  # Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
webresearcher "Your question"
```

**Programmatically:**

```python
from webresearcher import set_log_level, add_file_logger

# Set console log level
set_log_level("WARNING")  # Only show warnings and errors

# Add file logging with rotation
add_file_logger("research.log", level="DEBUG")

# Now run your research
agent = WebResearcherAgent(llm_config)
result = await agent.run("Your question")
```

**File Logging Features:**
- Automatic rotation when file size exceeds 10MB
- Keeps logs for 7 days
- Compresses old logs to .zip format

See  [logger.py](https://github.com/shibing624/WebResearcher/blob/main/webresearcher/logger.py) for detailed usage.

## 🎯 Features

### Core Features

- ✅ **Iterative Synthesis**: Prevents context overflow through periodic report updates
- ✅ **Unbounded Depth**: Practically unlimited research rounds
- ✅ **Smart Token Management**: Automatic context pruning and compression
- ✅ **Robust Error Handling**: Retry logic, fallback strategies, forced answer generation
- ✅ **Async/Await**: Non-blocking I/O for performance
- ✅ **Type Safe**: Full type hints throughout
- ✅ **Forced Final Answer (ReactAgent)**: Guarantees non-empty `prediction` on quota exhaustion/timeout

### Tool Features

- ✅ **Web Search**: Google Search integration via Serper
- ✅ **Academic Search**: Google Scholar for research papers
- ✅ **Web Scraping**: Intelligent content extraction from URLs
- ✅ **Code Execution**: Sandboxed Python interpreter
- ✅ **File Processing**: PDF, DOCX, CSV, Excel, and more
- ✅ **Extensible**: Easy custom tool creation

### Production Features

- ✅ **Zero Framework Lock-in**: No qwen-agent or similar dependencies
- ✅ **Lightweight**: Only 59KB wheel package
- ✅ **Well Documented**: Comprehensive docstrings and examples
- ✅ **CLI + API**: Use from command line or Python
- ✅ **Configurable**: Extensive configuration options
- ✅ **Logging**: Rich logging with loguru

## 📊 Performance

Based on the paper's evaluation:

- **HotpotQA**: Superior performance on multi-hop reasoning
- **Bamboogle**: Excellent on complex factual questions
- **Context Management**: Maintains lean workspace even after 50+ rounds
- **Accuracy**: Competitive with or exceeds baseline agents

<p align="center">
  <img src="https://github.com/shibing624/WebResearcher/blob/main/docs/performance.png" alt="Performance" width="100%"/>
</p>

## 🔧 Configuration

### Environment Variables

```bash
# Required
LLM_API_KEY=...              # LLM API key (OpenAI/DeepSeek etc.)
SERPER_API_KEY=...                 # Serper API for Google Search

# Optional
LLM_BASE_URL=https://...        # Custom LLM endpoint, or deepseek base url
LLM_MODEL_NAME=gpt-4o          # Default model name
JINA_API_KEY=...                   # Jina AI for web scraping
SANDBOX_FUSION_ENDPOINTS=...       # Code execution sandbox
MAX_LLM_CALL_PER_RUN=50           # Max iterations per research
FILE_DIR=./files                   # File storage directory
```

### LLM Configuration

```python
llm_config = {
    "model": "deepseek-v3.1",              # Or: o3-mini, gpt-4-turbo, etc.
    "api_key": "your-api-key",             # Optional, defaults to LLM_API_KEY env var
    "base_url": "https://api.openai.com/v1",  # Optional, defaults to LLM_BASE_URL env var
    "generate_cfg": {
        "temperature": 0.6,          # Sampling temperature (0.0-2.0)
        "top_p": 0.95,              # Nucleus sampling
        "presence_penalty": 1.1,     # Repetition penalty
        "model_thinking_type": "enabled"  # enabled|disabled|auto, if not supported, do not set
    },
    "max_input_tokens": 32000,      # Context window limit
    "llm_timeout": 300.0,           # LLM API timeout (seconds)
    "agent_timeout": 600.0,         # Total agent timeout (seconds)
}
```

## 🎭 WebWeaver Agent

**WebWeaver** is a dual-agent research framework implementing the dynamic outline paradigm, providing a more structured approach to research compared to the single-agent WebResearcher.

### Architecture Components

#### 1. Memory Bank
A shared evidence storage that bridges the Planner and Writer agents:
- **Add Evidence**: Planner stores findings with citation IDs
- **Retrieve Evidence**: Writer fetches specific evidence by ID
- **Decoupled Storage**: Keeps agents focused on their specific tasks

#### 2. Planner Agent
Explores research questions and builds a citation-grounded outline:
- **Actions**: 
  - `search`: Gather information from web
  - `write_outline`: Create/update research outline with citations
  - `terminate`: Finish planning phase
- **Output**: A structured outline with citation IDs linking to evidence

#### 3. Writer Agent
Writes comprehensive reports section-by-section:
- **Actions**:
  - `retrieve`: Fetch evidence from Memory Bank
  - `write`: Write report sections with inline citations
  - `terminate`: Finish writing phase
- **Output**: A complete research report with proper citations

<p align="center">
  <img src="https://github.com/shibing624/WebResearcher/blob/main/docs/webweaver.png" alt="WebWeaver架构" width="100%"/>
</p>

### Key Features

#### Dynamic Outline
Unlike traditional static outlines, WebWeaver's outline evolves as new evidence is discovered:
1. Planner searches and discovers evidence
2. Each finding gets a unique citation ID
3. Outline is updated to incorporate new evidence
4. Process repeats until outline is comprehensive

#### Citation-Grounded Reports
All claims in the final report are backed by specific evidence:
- Evidence is stored with full context in Memory Bank
- Writer retrieves only relevant evidence for each section
- Citations are embedded inline (e.g., `[cite:id_1]`)

### WebWeaver Usage

#### Basic Usage

```python
import asyncio
from webresearcher import WebWeaverAgent

async def main():
    # Configure LLM
    llm_config = {
        "model": "gpt-4o",
        "api_key": "your-api-key",      # Optional, defaults to LLM_API_KEY env var
        "base_url": "https://api.openai.com/v1",  # Optional, defaults to LLM_BASE_URL env var
        "generate_cfg": {
            "temperature": 0.1,  # Low temperature for factual research
            "top_p": 0.95,
            "max_tokens": 10000,
        },
        "llm_timeout": 300.0,
    }
    
    # Initialize agent (can also pass api_key and base_url directly as parameters)
    agent = WebWeaverAgent(
        llm_config=llm_config,
        api_key="your-api-key",    # Optional, overrides llm_config setting
        base_url="https://api.openai.com/v1"  # Optional
    )
    
    # Run research
    question = "What are the main causes of climate change?"
    result = await agent.run(question)
    
    # Access results
    print("Final Outline:", result['final_outline'])
    print("Final Report:", result['final_report'])
    print("Memory Bank Size:", result['memory_bank_size'])

if __name__ == "__main__":
    asyncio.run(main())
```

#### Command Line Interface

```bash
# Use WebWeaver mode
webresearcher "What are the causes of climate change?" --use-webweaver

# Save results to file
webresearcher "Research question" --use-webweaver --output report.json

# Verbose logging
webresearcher "Question" --use-webweaver --verbose
```

### WebResearcher vs WebWeaver Comparison

| Feature | WebResearcher | WebWeaver |
|---------|---------------|-----------|
| Architecture | Single-agent | Dual-agent |
| Paradigm | IterResearch | Dynamic Outline |
| Memory | Stateless workspace | Memory Bank |
| Output | Direct answer | Outline + Report |
| Citations | Implicit | Explicit with IDs |
| Structure | Iterative synthesis | Hierarchical |
| Best for | Quick answers | Comprehensive reports |

### When to Use WebWeaver

Choose **WebWeaver** when you need:
- ✅ Long-form, comprehensive research reports
- ✅ Explicit citation tracking
- ✅ Structured outline with evidence mapping
- ✅ Reproducible research process
- ✅ Multi-section documents

Choose **WebResearcher** when you need:
- ✅ Quick, focused answers
- ✅ Simpler architecture
- ✅ Direct question-answer format
- ✅ Lower token usage
- ✅ Faster results

## 📝 Examples

See the [examples/](./examples/) directory for complete examples:

- **[basic_usage.py](examples/webresearcher_usage.py)** - WebResearcher Agent usage example
- **[batch_research.py](./examples/batch_research.py)** - Processing multiple questions
- **[custom_agent.py](./examples/custom_agent.py)** - Creating custom tools
- **[webweaver_usage.py](examples/webweaver_usage.py)** - WebWeaver Agent usage example

## 🧪 Testing

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run with coverage
pytest --cov=webresearcher
```

## 🤝 Contributing

We welcome contributions! Ways to contribute:

- 🐛 Report bugs
- 💡 Suggest features
- 📝 Improve documentation
- 🔧 Submit pull requests

Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for detailed guidelines.

## 📧 Contact

- **GitHub Issues**: [Report bugs or request features](https://github.com/shibing624/WebResearcher/issues)
- **Email**: xuming624@qq.com
- **WeChat**: xuming624 (Note: Name-Company-NLP)

<p align="center">
  <img src="https://github.com/shibing624/WebResearcher/blob/main/docs/wechat.jpeg" width="200" />
</p>

## 🌟 Star History

[![Star History Chart](https://api.star-history.com/svg?repos=shibing624/WebResearcher&type=Date)](https://star-history.com/#shibing624/WebResearcher&Date)

## 📑 Citation

If you use WebResearcher in your research, please cite:

```bibtex
@misc{qiao2025webresearcher,
    title={WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents}, 
    author={Zile Qiao and Guoxin Chen and Xuanzhong Chen and Donglei Yu and Wenbiao Yin and Xinyu Wang and Zhen Zhang and Baixuan Li and Huifeng Yin and Kuan Li and Rui Min and Minpeng Liao and Yong Jiang and Pengjun Xie and Fei Huang and Jingren Zhou},
    year={2025},
    eprint={2509.13309},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2509.13309}, 
}
```

```bibtex
@misc{li2025webweaverstructuringwebscaleevidence,
      title={WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research}, 
      author={Zijian Li and Xin Guan and Bo Zhang and Shen Huang and Houquan Zhou and Shaopeng Lai and Ming Yan and Yong Jiang and Pengjun Xie and Fei Huang and Jun Zhang and Jingren Zhou},
      year={2025},
      eprint={2509.13312},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.13312}, 
}
```

## 📄 License

This project is licensed under the [Apache License 2.0](./LICENSE) - free for commercial use.

## 🙏 Acknowledgements

This project is inspired by and built upon the research from:

- **[WebResearcher Paper](https://arxiv.org/abs/2509.13309)** by Qiao et al.
- **[WebWeaver Paper](https://arxiv.org/abs/2509.13312)** by Li et al.
- **[Alibaba-NLP/DeepResearch](https://github.com/Alibaba-NLP/DeepResearch)** - Original research implementation

Special thanks to the authors for their groundbreaking work on iterative research paradigms!
