# MCP-DBLP Evaluation

## Objective

Evaluate the accuracy of BibTeX generation when using MCP-DBLP versus baseline LLM with web search.

## Methodology

1. **Ground Truth**: Start with verified DBLP BibTeX entries
2. **Test Input**: Create obfuscated/informal citation text from ground truth
3. **Test Groups**:
   - **Control**: Claude + web search (no MCP-DBLP)
   - **Treatment**: Claude + MCP-DBLP + web search
4. **Evaluation**: LLM-as-judge compares generated BibTeX against ground truth
5. **Metrics**:
   - Author accuracy (exact match)
   - Title accuracy (exact match)
   - Venue accuracy
   - Year accuracy
   - DOI accuracy
   - Fabrication detection (hallucinated entries)

## Error Categories

- Correct: Perfect match with ground truth
- Minor error: Small formatting differences, venue abbreviation
- Major error: Wrong paper, wrong authors, wrong year
- Fabrication: Nonexistent paper or completely wrong metadata

## Reproducibility

All steps use explicit prompt files to ensure reproducibility:

- `prompts/create_ground_truth.md`: Fetch verified DBLP entries
- `prompts/create_test_input.md`: Generate obfuscated citations from ground truth
- `prompts/control_group.md`: Generate BibTeX without MCP-DBLP (web search only)
- `prompts/treatment_group.md`: Generate BibTeX with MCP-DBLP
- `prompts/evaluate.md`: LLM judge evaluation against ground truth

Each prompt file is executed by spawning a subagent with the prompt content.

## Files

- `ground_truth.bib`: Verified DBLP entries (source of truth)
- `test_input.txt`: Obfuscated informal citations derived from ground truth
- `control_output.bib`: BibTeX generated by Claude + web search
- `treatment_output.bib`: BibTeX generated by Claude + MCP-DBLP
- `evaluation_results.md`: LLM judge evaluation comparing outputs to ground truth

## Preliminary Test

Start small: 5-10 citations with varying difficulty levels.
