# LDR Benchmarking Framework

The Local Deep Research (LDR) Benchmarking Framework allows you to evaluate and compare LDR's performance on standardized benchmarks.

## Features

- Run benchmarks on SimpleQA and BrowseComp datasets
- Configure search parameters (iterations, questions per iteration, search tool)
- Use automated evaluation with Claude 3.7 Sonnet (default) or other models
- Generate detailed reports with metrics and examples
- Compare multiple configurations to find optimal settings
- API, CLI, and web interface integration

## Benchmark Datasets

### SimpleQA

A straightforward question-answering benchmark with factual questions. This benchmark tests LDR's ability to find and synthesize factual information.

### BrowseComp

A web browsing comprehension benchmark with more complex questions requiring synthesis across multiple sources. This benchmark tests LDR's ability to understand and navigate complex information needs.

## Usage

### Programmatic API

```python
from local_deep_research.api.benchmark_functions import evaluate_simpleqa

# Run SimpleQA benchmark with 20 examples
result = evaluate_simpleqa(
    num_examples=20,
    search_iterations=3,
    questions_per_iteration=3,
    search_tool="searxng"
)

# Print accuracy
print(f"Accuracy: {result['metrics']['accuracy']:.3f}")
```

### Command Line Interface

```bash
# Run SimpleQA benchmark
python -m local_deep_research.cli.benchmark_commands simpleqa --examples 20 --iterations 3

# Run BrowseComp benchmark
python -m local_deep_research.cli.benchmark_commands browsecomp --examples 10 --search-tool wikipedia

# Compare configurations
python -m local_deep_research.cli.benchmark_commands compare --dataset simpleqa --examples 5
```

### Web Interface

The benchmark dashboard is available at `/benchmark` in the LDR web interface. You can:

1. Select a benchmark to run
2. Configure parameters
3. Run the benchmark
4. View results and reports

## Evaluation

By default, benchmarks are evaluated using Claude 3.7 Sonnet via OpenRouter. You can customize the evaluation model:

```python
# Use a different model for evaluation
result = evaluate_simpleqa(
    num_examples=10,
    evaluation_model="gpt-4o",
    evaluation_provider="openai"
)
```

You can also use human evaluation:

```python
# Use human evaluation
result = evaluate_simpleqa(
    num_examples=5,
    human_evaluation=True
)
```

## Configuration Comparison

Compare multiple configurations to find optimal settings:

```python
from local_deep_research.api.benchmark_functions import compare_configurations

# Define configurations to compare
configurations = [
    {
        "name": "Base Config",
        "search_tool": "searxng",
        "iterations": 1,
        "questions_per_iteration": 3
    },
    {
        "name": "More Iterations",
        "search_tool": "searxng",
        "iterations": 3,
        "questions_per_iteration": 3
    },
    {
        "name": "Different Search Engine",
        "search_tool": "wikipedia",
        "iterations": 1,
        "questions_per_iteration": 3
    }
]

# Run comparison
result = compare_configurations(
    dataset_type="simpleqa",
    num_examples=10,
    configurations=configurations
)
```

## Output Format

Benchmark results include:

- **metrics**: Accuracy, processing time, confidence scores
- **report_path**: Path to generated report
- **results_path**: Path to raw results file
- **total_examples**: Number of examples processed
- **status**: Completion status

## Example Reports

Reports include:

- Overall accuracy and metrics
- Configuration details
- Example correct and incorrect answers
- Time and date information

## Integration with LDR Web App

The benchmarking framework is fully integrated with the LDR web interface. You can run benchmarks and view results directly in the web app.

## Adding Custom Benchmarks

To add a custom benchmark:

1. Create a dataset loader in `datasets.py`
2. Add evaluation templates in `templates.py`
3. Create benchmark runners in `runners.py`
4. Expose the benchmark through the API in `api/benchmark_functions.py`

## Performance Considerations

- Running benchmarks can be resource-intensive
- Start with a small number of examples for testing
- Full benchmarks with 100+ examples may take several hours to complete
- Consider using a more powerful machine for large benchmarks
