# Local Deep Research

A powerful AI-powered research assistant that performs deep, iterative analysis using multiple LLMs and web searches. The system can be run locally for privacy or configured to use cloud-based LLMs for enhanced capabilities.

## Features

- 🔍 **Advanced Research Capabilities**
  - Automated deep research with intelligent follow-up questions
  - Citation tracking and source verification
  - Multi-iteration analysis for comprehensive coverage
  - Full webpage content analysis (not just snippets)

- 🤖 **Flexible LLM Support**
  - Local AI processing with Ollama models
  - Cloud LLM support (Claude, GPT)
  - Supports all Langchain models
  - Configurable model selection based on needs

- 📊 **Rich Output Options**
  - Detailed research findings with citations
  - Comprehensive research reports
  - Quick summaries for rapid insights
  - Source tracking and verification

- 🔒 **Privacy-Focused**
  - Runs entirely on your machine when using local models
  - Configurable search settings
  - Transparent data handling

- 🌐 **Enhanced Search Integration**
  - **Auto-selection of search sources**: The "auto" search engine intelligently analyzes your query and selects the most appropriate search engine based on the query content
  - **SearXNG** integration for local web-search engine, great for privacy, no API key required (requires a searxng server)
  - Wikipedia integration for factual knowledge
  - arXiv integration for scientific papers and academic research
  - PubMed integration for biomedical literature and medical research
  - DuckDuckGo integration for web searches (may experience rate limiting)
  - SerpAPI integration for Google search results (requires API key)
  - Google Programmable Search Engine integration for custom search experiences (requires API key)
  - The Guardian integration for news articles and journalism (requires API key)
  - **Local RAG search for private documents** - search your own documents with vector embeddings
  - Full webpage content retrieval
  - Source filtering and validation
  - Configurable search parameters

- 📑 **Local Document Search (RAG)**
  - Vector embedding-based search of your local documents
  - Create custom document collections for different topics
  - Privacy-preserving - your documents stay on your machine
  - Intelligent chunking and retrieval 
  - Compatible with various document formats (PDF, text, markdown, etc.)
  - Automatic integration with meta-search for unified queries

## Example Research: Fusion Energy Developments

The repository includes complete research examples demonstrating the tool's capabilities. For instance, our [fusion energy research analysis](https://github.com/LearningCircuit/local-deep-research/blob/main/examples/fusion-energy-research-developments.md) provides a comprehensive overview of:

- Latest scientific breakthroughs in fusion research (2022-2025)
- Private sector funding developments exceeding $6 billion
- Expert projections for commercial fusion energy timelines
- Regulatory frameworks being developed for fusion deployment
- Technical challenges that must be overcome for commercial viability

This example showcases the system's ability to perform multiple research iterations, follow evidence trails across scientific and commercial domains, and synthesize information from diverse sources while maintaining proper citation.

## Installation

1. Clone the repository:
```bash
git clone https://github.com/LearningCircuit/local-deep-research.git
cd local-deep-research
```
(experimental pip install with new features (but not so well tested yet): **pip install local-deep-research** )
2. Install dependencies:
```bash
pip install -r requirements.txt
playwright install
```

3. Install Ollama (for local models):
```bash
# Install Ollama from https://ollama.ai
ollama pull mistral  # Default model - many work really well choose best for your hardware (fits in GPU)
```

4. Configure environment variables:
```bash
# Copy the template
cp .env.template .env
```

## Experimental install
```bash
#experimental pip install with new features (but not so well tested yet):
pip install local-deep-research
playwright install
ollama pull mistral 
```

# Edit .env with your API keys (if using cloud LLMs)
ANTHROPIC_API_KEY=your-api-key-here  # For Claude
OPENAI_API_KEY=your-openai-key-here  # For GPT models
GUARDIAN_API_KEY=your-guardian-api-key-here  # For The Guardian search
```

## Usage
Terminal usage (not recommended):
```bash
python main.py
```

### Web Interface

The project includes a web interface for a more user-friendly experience:

```bash
python app.py
```

This will start a local web server, accessible at `http://127.0.0.1:5000` in your browser.

#### Web Interface Features:

- **Dashboard**: Intuitive interface for starting and managing research queries
- **Real-time Updates**: Track research progress with live updates
- **Research History**: Access and manage past research queries
- **PDF Export**: Download completed research reports as PDF documents
- **Research Management**: Terminate ongoing research processes or delete past records

![Web Interface](./web1.png)
![Web Interface](./web2.png)
### Configuration
**Please report your best settings in issues so we can improve the default settings.**

Key settings in `config.py`:
```python
# LLM Configuration
DEFAULT_MODEL = "mistral"  # Change based on your needs
DEFAULT_TEMPERATURE = 0.7
MAX_TOKENS = 8000

# Search Configuration
MAX_SEARCH_RESULTS = 40
SEARCH_REGION = "us-en"
TIME_PERIOD = "y"
SAFE_SEARCH = True
SEARCH_SNIPPETS_ONLY = False

# Choose search tool: "wiki", "arxiv", "duckduckgo", "guardian", "serp", "local_all", or "auto"
search_tool = "auto"  # "auto" will intelligently select the best search engine for your query
```

## Local Document Search (RAG)

The system includes powerful local document search capabilities using Retrieval-Augmented Generation (RAG). This allows you to search and retrieve content from your own document collections.

### Setting Up Local Collections

Create a file named `local_collections.py` in the project root directory:

```python
# local_collections.py
import os
from typing import Dict, Any

# Registry of local document collections
LOCAL_COLLECTIONS = {
    # Research Papers Collection
    "research_papers": {
        "name": "Research Papers",
        "description": "Academic research papers and articles",
        "paths": [os.path.abspath("local_search_files/research_papers")],  # Use absolute paths
        "enabled": True,
        "embedding_model": "all-MiniLM-L6-v2",
        "embedding_device": "cpu",
        "embedding_model_type": "sentence_transformers",
        "max_results": 20,
        "max_filtered_results": 5,
        "chunk_size": 800,  # Smaller chunks for academic content
        "chunk_overlap": 150,
        "cache_dir": ".cache/local_search/research_papers"
    },
    
    # Personal Notes Collection
    "personal_notes": {
        "name": "Personal Notes",
        "description": "Personal notes and documents",
        "paths": [os.path.abspath("local_search_files/personal_notes")],  # Use absolute paths
        "enabled": True,
        "embedding_model": "all-MiniLM-L6-v2",
        "embedding_device": "cpu",
        "embedding_model_type": "sentence_transformers",
        "max_results": 30,
        "max_filtered_results": 10,
        "chunk_size": 500,  # Smaller chunks for notes
        "chunk_overlap": 100,
        "cache_dir": ".cache/local_search/personal_notes"
    }
}
```

Create directories for your collections:

```bash
mkdir -p local_search_files/research_papers
mkdir -p local_search_files/personal_notes
```

Add your documents to these folders, and the system will automatically index them and make them available for searching.

### Using Local Search

You can use local search in several ways:

1. **Auto-selection**: Set `search_tool = "auto"` in `config.py` and the system will automatically use your local collections when appropriate for the query.

2. **Explicit Selection**: Set `search_tool = "research_papers"` to search only that specific collection.

3. **Search All Local Collections**: Set `search_tool = "local_all"` to search across all your local document collections.

4. **Query Syntax**: Use `collection:collection_name your query` to target a specific collection within a query.

### Search Engine Options

The system supports multiple search engines that can be selected by changing the `search_tool` variable in `config.py`:

- **Auto** (`auto`): Intelligent search engine selector that analyzes your query and chooses the most appropriate source (Wikipedia, arXiv, local collections, etc.)
- **SearXNG** (`searxng`): Local web-search engine, great for privacy, no API key required (requires a searxng server)
- **Wikipedia** (`wiki`): Best for general knowledge, facts, and overview information
- **arXiv** (`arxiv`): Great for scientific and academic research, accessing preprints and papers
- **PubMed** (`pubmed`): Excellent for biomedical literature, medical research, and health information
- **DuckDuckGo** (`duckduckgo`): General web search that doesn't require an API key
- **The Guardian** (`guardian`): Quality journalism and news articles (requires an API key)
- **SerpAPI** (`serp`): Google search results (requires an API key)
- **Google Programmable Search Engine** (`google_pse`): Custom search experiences with control over search scope and domains (requires API key and search engine ID)
- **Local Collections**: Any collections defined in your `local_collections.py` file

> **Note:** The "auto" option will intelligently select the best search engine based on your query. For example, if you ask about physics research papers, it might select arXiv or your research_papers collection, while if you ask about current events, it might select The Guardian or DuckDuckGo.

> **Support Free Knowledge:** If you frequently use the search engines in this tool, please consider making a donation to these organizations. They provide valuable services and rely on user support to maintain their operations:
> - [Donate to Wikipedia](https://donate.wikimedia.org)
> - [Support The Guardian](https://support.theguardian.com)
> - [Support arXiv](https://arxiv.org/about/give)
> - [Donate to DuckDuckGo](https://duckduckgo.com/donations)
> - [Support PubMed/NCBI](https://www.nlm.nih.gov/pubs/donations/donations.html)

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Acknowledgments
- Built with [Ollama](https://ollama.ai) for local AI processing
- Search powered by multiple sources:
  - [Wikipedia](https://www.wikipedia.org/) for factual knowledge (default search engine)
  - [arXiv](https://arxiv.org/) for scientific papers
  - [PubMed](https://pubmed.ncbi.nlm.nih.gov/) for biomedical literature
  - [DuckDuckGo](https://duckduckgo.com) for web search
  - [The Guardian](https://www.theguardian.com/) for quality journalism
  - [SerpAPI](https://serpapi.com) for Google search results (requires API key)
  - [SearXNG](https://searxng.org/) for local web-search engine 
- Built on [LangChain](https://github.com/hwchase17/langchain) framework
- Uses [justext](https://github.com/miso-belica/justext) for content extraction
- [Playwright](https://playwright.dev) for web content retrieval
- Uses [FAISS](https://github.com/facebookresearch/faiss) for vector similarity search
- Uses [sentence-transformers](https://github.com/UKPLab/sentence-transformers) for embeddings

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

## Star History
 
 [![Star History Chart](https://api.star-history.com/svg?repos=LearningCircuit/local-deep-research&type=Date)](https://www.star-history.com/#LearningCircuit/local-deep-research&Date)
