# Local Deep Research

A powerful AI-powered research assistant that performs deep, iterative analysis using multiple LLMs and web searches. The system can be run locally for privacy or configured to use cloud-based LLMs for enhanced capabilities.

<div align="center">
  <a href="https://www.youtube.com/watch?v=0ISreg9q0p0">
    <img src="https://img.youtube.com/vi/0ISreg9q0p0/0.jpg" alt="Local Deep Research">
    <br>
    <span>▶️ Watch Video</span>
  </a>
</div>

## Quick Start

```bash
# Install the package
pip install local-deep-research

# Install required browser automation tools
playwright install

# For local models, install Ollama
# Download from https://ollama.ai and then pull a model
ollama pull gemma3:12b
```

Then run:

```bash
# Start the web interface (recommended)
ldr-web # (OR python -m local_deep_research.web.app)

# OR run the command line version
ldr # (OR python -m local_deep_research.main)
```

Access the web interface at `http://127.0.0.1:5000` in your browser.

## Features
<div align="center">
  <a href="https://www.youtube.com/watch?v=4tDqV__jzKY">
    <img src="https://img.youtube.com/vi/4tDqV__jzKY/0.jpg" alt="Local Deep Research">
    <br>
    <span>▶️ Watch Video</span>
  </a>
</div>

- 🔍 **Advanced Research Capabilities**
  - Automated deep research with intelligent follow-up questions
  - Proper inline citation and source verification
  - Multi-iteration analysis for comprehensive coverage
  - Full webpage content analysis (not just snippets)

- 🤖 **Flexible LLM Support**
  - Local AI processing with Ollama models
  - Cloud LLM support (Claude, GPT)
  - Supports all Langchain models
  - Configurable model selection based on needs

- 📊 **Rich Output Options**
  - Detailed research findings with proper citations
  - Well-structured comprehensive research reports
  - Quick summaries for rapid insights
  - Source tracking and verification

- 🔒 **Privacy-Focused**
  - Runs entirely on your machine when using local models
  - Configurable search settings
  - Transparent data handling

- 🌐 **Enhanced Search Integration**
  - **Auto-selection of search sources**: The "auto" search engine intelligently analyzes your query and selects the most appropriate search engine
  - Multiple search engines including Wikipedia, arXiv, PubMed, Semantic Scholar, and more
  - **Local RAG search for private documents** - search your own documents with vector embeddings
  - Full webpage content retrieval and intelligent filtering

- 🎓 **Academic & Scientific Integration**
  - Direct integration with PubMed, arXiv, Wikipedia, Semantic Scholar
  - Properly formatted citations from academic sources
  - Report structure suitable for literature reviews
  - Cross-disciplinary synthesis of information

## Configuration System

The package automatically creates and manages configuration files in your user directory:

- **Windows**: `Documents\LearningCircuit\local-deep-research\config\`
- **Linux/Mac**: `~/.config/local_deep_research/config/`

### Default Configuration Files

When you first run the tool, it creates these configuration files:

| File | Purpose |
|------|---------|
| `settings.toml` | General settings for research, web interface, and search |
| `llm_config.py` | Advanced LLM configuration (rarely needs modification) |
| `search_engines.toml` | Define and configure search engines |
| `local_collections.toml` | Configure local document collections for RAG |
| `.env` | Environment variables for configuration (recommended for API keys) |

> **Note:** For comprehensive environment variable configuration, see our [Environment Variables Guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/env_configuration.md).

## Setting Up AI Models

The system supports multiple LLM providers:

### Local Models (via Ollama)

1. [Install Ollama](https://ollama.ai) 
2. Pull a model: `ollama pull gemma3:12b` (recommended model)
3. Ollama runs on port 11434 by default

### Cloud Models

Add API keys to your environment variables (recommended) by creating a `.env` file in your config directory:

```bash
# Set API keys for cloud providers in .env
ANTHROPIC_API_KEY=your-api-key-here      # For Claude models
OPENAI_API_KEY=your-openai-key-here      # For GPT models
OPENAI_ENDPOINT_API_KEY=your-key-here    # For OpenRouter or similar services

# Set your preferred LLM provider and model (no need to edit llm_config.py)
LDR_LLM__PROVIDER=ollama                 # Options: ollama, openai, anthropic, etc.
LDR_LLM__MODEL=gemma3:12b                # Model name to use
```

> **Important:** In most cases, you don't need to modify the `llm_config.py` file. Simply set the `LDR_LLM__PROVIDER` and `LDR_LLM__MODEL` environment variables to use your preferred model.

### Supported LLM Providers

The system supports multiple LLM providers:

| Provider | Type | API Key | Setup Details | Models |
|----------|------|---------|---------------|--------|
| `OLLAMA` | Local | No | Install from [ollama.ai](https://ollama.ai) | Mistral, Llama, Gemma, etc. |
| `OPENAI` | Cloud | `OPENAI_API_KEY` | Set in environment | GPT-3.5, GPT-4, GPT-4o |
| `ANTHROPIC` | Cloud | `ANTHROPIC_API_KEY` | Set in environment | Claude 3 Opus, Sonnet, Haiku |
| `OPENAI_ENDPOINT` | Cloud | `OPENAI_ENDPOINT_API_KEY` | Set in environment | Any OpenAI-compatible model |
| `VLLM` | Local | No | Requires GPU setup | Any supported by vLLM |
| `LMSTUDIO` | Local | No | Use LM Studio server | Models from LM Studio |
| `LLAMACPP` | Local | No | Configure model path | GGUF model formats |

The `OPENAI_ENDPOINT` provider can access any service with an OpenAI-compatible API, including:
- OpenRouter (access to hundreds of models)
- Azure OpenAI
- Together.ai
- Groq
- Anyscale
- Self-hosted LLM servers with OpenAI compatibility

## Setting Up Search Engines

Some search engines require API keys. Add them to your environment variables by creating a `.env` file in your config directory:

```bash
# Search engine API keys (add to .env file)
SERP_API_KEY=your-serpapi-key-here        # For Google results via SerpAPI
GOOGLE_PSE_API_KEY=your-google-key-here   # For Google Programmable Search
GOOGLE_PSE_ENGINE_ID=your-pse-id-here     # For Google Programmable Search
BRAVE_API_KEY=your-brave-search-key-here  # For Brave Search
GUARDIAN_API_KEY=your-guardian-key-here   # For The Guardian

# Set your preferred search tool
LDR_SEARCH__TOOL=auto                     # Default: intelligently selects best engine
```

> **Tip:** To override other settings via environment variables (e.g., to change the web port), use: **LDR_WEB__PORT=8080**

### Available Search Engines

| Engine | Purpose | API Key Required? | Rate Limit |
|--------|---------|-------------------|------------|
| `auto` | Intelligently selects the best engine | No | Based on selected engine |
| `wikipedia` | General knowledge and facts | No | No strict limit |
| `arxiv` | Scientific papers and research | No | No strict limit |
| `pubmed` | Medical and biomedical research | No | No strict limit |
| `semantic_scholar` | Academic literature across all fields | No | 100/5min |
| `github` | Code repositories and documentation | No | 60/hour (unauthenticated) |
| `brave` | Web search (privacy-focused) | Yes | Based on plan |
| `serpapi` | Google search results | Yes | Based on plan |
| `google_pse` | Custom Google search | Yes | 100/day free tier |
| `wayback` | Historical web content | No | No strict limit |
| `searxng` | Local web search engine | No (requires local server) | No limit |
| Any collection name | Search your local documents | No | No limit |

> **Note:** For detailed SearXNG setup, see our [SearXNG Setup Guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/SearXNG-Setup.md).

## Local Document Search (RAG)

The system can search through your local documents using vector embeddings.

### Setting Up Document Collections

1. Define collections in `local_collections.toml`. Default collections include:

```toml
[project_docs]
name = "Project Documents"
description = "Project documentation and specifications"
paths = ["@format ${DOCS_DIR}/project_documents"]
enabled = true
embedding_model = "all-MiniLM-L6-v2"
embedding_device = "cpu"
embedding_model_type = "sentence_transformers"
max_results = 20
max_filtered_results = 5
chunk_size = 1000
chunk_overlap = 200
cache_dir = "__CACHE_DIR__/local_search/project_docs"
```

2. Create your document directories:
   - The `${DOCS_DIR}` variable points to a default location in your Documents folder
   - Documents are automatically indexed when the search is first used

### Using Local Search

You can use local document search in several ways:

1. **Auto-selection**: Set `tool = "auto"` in `settings.toml` [search] section
2. **Explicit collection**: Set `tool = "project_docs"` to search only that collection
3. **All collections**: Set `tool = "local_all"` to search across all collections
4. **Query syntax**: Type `collection:project_docs your query` to target a specific collection

## Docker Support

Local Deep Research can run in Docker containers for easy deployment across environments.

### Quick Docker Run

```bash
# Run with default settings (connects to Ollama running on the host)
docker run --network=host \
  -e LDR_LLM__PROVIDER="ollama" \
  -e LDR_LLM__MODEL="mistral" \
  local-deep-research
```

For comprehensive Docker setup information, see:
- [Docker Usage Guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/docker-usage-readme.md)
- [Docker Compose Guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/docker-compose-guide.md)

## Advanced Configuration

### Research Parameters

Edit `settings.toml` to customize research parameters or use environment variables:

```toml
[search]
# Search tool to use (auto, wikipedia, arxiv, etc.)
tool = "auto"

# Number of research cycles
iterations = 2

# Questions generated per cycle
questions_per_iteration = 2

# Results per search query
max_results = 50

# Results after relevance filtering
max_filtered_results = 5
```

Using environment variables:
```bash
LDR_SEARCH__TOOL=auto
LDR_SEARCH__ITERATIONS=3
LDR_SEARCH__QUESTIONS_PER_ITERATION=2
```

## Web Interface

The web interface offers several features:

- **Dashboard**: Start and manage research queries
- **Real-time Updates**: Track research progress
- **Research History**: Access past queries
- **PDF Export**: Download reports
- **Research Management**: Terminate processes or delete records

## Command Line Interface

The CLI version allows you to:

1. Choose between a quick summary or detailed report
2. Enter your research query
3. View results directly in the terminal
4. Save reports automatically to the configured output directory

## Development Setup

If you want to develop or modify the package, you can install it in development mode:

```bash
# Clone the repository
git clone https://github.com/LearningCircuit/local-deep-research.git
cd local-deep-research

# Install in development mode
pip install -e .
```

You can run the application directly using Python module syntax:

```bash
# Run the web interface
python -m local_deep_research.web.app

# Run the CLI version
python -m local_deep_research.main
```

## Community & Support

Join our [Discord server](https://discord.gg/2E6gYU2Z) to exchange ideas, discuss usage patterns, and share research approaches.

## License

This project is licensed under the MIT License.

## Acknowledgments

- Built with [Ollama](https://ollama.ai) for local AI processing
- Search powered by multiple sources:
  - [Wikipedia](https://www.wikipedia.org/) for factual knowledge
  - [arXiv](https://arxiv.org/) for scientific papers
  - [PubMed](https://pubmed.ncbi.nlm.nih.gov/) for biomedical literature
  - [Semantic Scholar](https://www.semanticscholar.org/) for academic literature
  - [DuckDuckGo](https://duckduckgo.com) for web search
  - [The Guardian](https://www.theguardian.com/) for journalism
  - [SerpAPI](https://serpapi.com) for Google search results
  - [SearXNG](https://searxng.org/) for local web-search engine
  - [Brave Search](https://search.brave.com/) for privacy-focused web search
- Built on [LangChain](https://github.com/hwchase17/langchain) framework
- Uses [justext](https://github.com/miso-belica/justext), [Playwright](https://playwright.dev), [FAISS](https://github.com/facebookresearch/faiss), and more

> **Support Free Knowledge:** If you frequently use the search engines in this tool, please consider making a donation to these organizations:
> - [Donate to Wikipedia](https://donate.wikimedia.org)
> - [Support arXiv](https://arxiv.org/about/give)
> - [Donate to DuckDuckGo](https://duckduckgo.com/donations)
> - [Support PubMed/NCBI](https://www.nlm.nih.gov/pubs/donations/donations.html)

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Make your changes
4. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
5. Push to the branch (`git push origin feature/AmazingFeature`)
6. **Important:** Open a Pull Request against the `dev` branch, not the `main` branch

We prefer all pull requests to be submitted against the `dev` branch for easier testing and integration before releasing to the main branch.
