# Benchmark: paper2md vs MarkItDown vs PyMuPDF4LLM

**Date:** 2026-03-10  
**Papers:** 10 recent ML/AI papers from arXiv  
**Dimensions:** 11 quality metrics  

## Tool Versions

- **paper2md**: 0.1.0
- **MarkItDown**: 0.1.5
- **PyMuPDF4LLM**: 0.3.4
- **PyMuPDF**: 1.27.1

## Test Corpus

| Paper | Year | Pages |
|-------|------|-------|
| clip | 2020-2025 | 48 |
| ddpm | 2020-2025 | 25 |
| deepseek_r1 | 2020-2025 | 86 |
| dit | 2020-2025 | 25 |
| flow_matching | 2020-2025 | 28 |
| gpt4 | 2020-2025 | 100 |
| llama2 | 2020-2025 | 77 |
| lora | 2020-2025 | 26 |
| mamba | 2020-2025 | 36 |
| yolov4 | 2020-2025 | 17 |

## Overall Results

| Dimension | paper2md | MarkItDown | PyMuPDF4LLM |
|---|---:|---:|---:|
| title | **100.0** | **100.0** | **100.0** |
| abstract | **100.0** | 95.0 | 30.0 |
| authors | **90.0** | 0.0 | 0.0 |
| structure | **100.0** | 10.0 | 64.0 |
| references | **100.0** | 50.0 | 53.0 |
| clean_text | **100.0** | 56.4 | 78.3 |
| math | **100.0** | 20.0 | 15.0 |
| tables | **100.0** | 40.0 | 80.0 |
| figures | **100.0** | 85.0 | **100.0** |
| completeness | **100.0** | **100.0** | **100.0** |
| dehyphenation | **100.0** | 68.0 | 88.0 |
| **OVERALL** | **99.1** | 56.8 | 64.4 |

## Speed Comparison

| Paper | paper2md | MarkItDown | PyMuPDF4LLM |
|---|---:|---:|---:|
| clip | 4.7s | 5.5s | 24.9s |
| ddpm | 2.1s | 2.1s | 3.5s |
| deepseek_r1 | 5.0s | 6.2s | 20.1s |
| dit | 5.2s | 4.9s | 8.0s |
| flow_matching | 2.5s | 2.0s | 5.5s |
| gpt4 | 2.6s | 5.8s | 8.2s |
| llama2 | 7.7s | 18.9s | 23.3s |
| lora | 9.8s | 5.8s | 17.4s |
| mamba | 0.9s | 2.9s | 4.2s |
| yolov4 | 0.7s | 0.9s | 2.2s |
| **Average** | **4.1s** | 5.5s | 11.7s |

## Methodology

Each paper is converted with all three tools. The output Markdown is
scored across 11 quality dimensions on a 0-100 scale:

| Dimension | What it measures |
|-----------|-----------------|
| title | Correct paper title extraction |
| abstract | Abstract text detection and extraction |
| authors | Author name extraction |
| structure | Section headings (## and ###) detected |
| references | Bibliography entries parsed |
| clean_text | Absence of garbled/control characters |
| math | LaTeX math expressions ($...$, $$...$$) |
| tables | Markdown table rows detected |
| figures | Figure references and image embeds |
| completeness | Word count as proxy for content preservation |
| dehyphenation | Absence of broken line-break hyphens |
