# Technical landscape for PDF-to-Markdown conversion of research papers

**No existing tool reliably converts research papers to clean Markdown.** The four leading open-source tools — MarkItDown, Docling, Marker, and MinerU — each fail on at least one critical dimension (equations, tables, citations, or multi-column layout), and none produce structured Markdown with proper heading hierarchy, LaTeX math, and parsed references out of the box. This gap creates a clear opportunity for a purpose-built library that chains the best available PDF parsing primitives into a research-paper-aware pipeline. Below is a detailed technical landscape to inform that build.

---

## Existing tools solve parts of the problem but none solve all of it

### MarkItDown (Microsoft) — fast but structurally blind

MarkItDown (MIT license, ~50k GitHub stars) calls `pdfminer.high_level.extract_text()` and returns raw text with zero layout analysis. **It achieves roughly 25% accuracy on PDFs** and 47.3% across all formats. It strips all formatting — headings, emphasis, lists vanish. Multi-column papers produce garbled interleaved text. Equations emit `(cid:xxx)` artifacts from embedded fonts. Tables and figures are undetected. An optional Azure Document Intelligence integration exists for higher-quality conversion, but the open-source path is a dead end for research papers. Its value is as a fast first-pass filter (roughly **100× faster than Docling**) in a tiered pipeline with fallback to better tools.

### Docling (IBM) — strongest tables, maturing equations

Docling (MIT license, v2.38+) runs a sequential pipeline: a custom `docling-parse` backend (built on `qpdf`) extracts text tokens with coordinates, then an **RT-DETR layout model trained on DocLayNet** (~81k labeled pages) classifies page regions, and **TableFormer** handles complex table structure recognition (partial borders, spanning cells, empty cells). Its VLM option, **Granite-Docling (258M parameters)**, uses a SigLIP2 visual encoder and Granite 3 language backbone, outputting IBM's custom DocTags format. Docling runs at **0.49 sec/page on an Nvidia L4 GPU** — slower than MinerU but faster than Marker. Equation support arrived via Granite-Docling for inline and floating math but is less battle-tested than MinerU's dedicated UniMERNet. Citation parsing is not a core feature. Exports to Markdown, HTML, JSON, and JATS XML.

### Marker (Datalab) — best with LLM boost, restrictive license

Marker (~31.5k stars) is built on **Surya**, Datalab's OCR toolkit using a modified Donut architecture for text recognition and a modified EfficientViT/SegFormer for detection. Its layout model detects 12 label types: Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Figure, Section-header, Table, Text, Title. The `--use_llm` flag (Google Gemini API) dramatically improves inline math formatting, cross-page table merging, and form extraction. Throughput reaches **~122 pages/second with 22 parallel processes on H100**. However, the **GPL-3.0 code license and cc-by-nc-sa-4.0 model weights** severely limit commercial use. Without the LLM boost, complex tables and inline equations degrade noticeably.

### MinerU (OpenDataLab) — fastest, best all-rounder for science

MinerU (AGPL license due to YOLO dependency) offers three backends. The **pipeline backend** chains DocLayout-YOLO (10× faster than LayoutLMv3) for layout detection, YOLOv8 for formula detection, **UniMERNet** for formula-to-LaTeX recognition, PaddleOCR for text, and RapidTable for table structure. The **VLM backend (MinerU 2.5)** is a 1.2B-parameter model with NaViT encoder and Qwen2-Instruct decoder that achieves state-of-the-art on OmniDocBench. At **0.21 sec/page on L4 GPU**, MinerU is the fastest of the four tools. Reading order uses LayoutReader or XY-Cut algorithms. It was born from the InternLM pre-training data pipeline, making it inherently science-paper-oriented. Key gaps: AGPL license complicates embedding, code blocks unsupported, complex table edge cases persist, and rare-language OCR has diacritical-mark issues.

### The universal gaps across all four tools

Every tool shares critical weaknesses for research papers. **Heading hierarchy detection** remains unreliable — multi-level headings often need manual correction. **Citation/reference structured parsing** (linking in-text `[1]` markers to bibliography entries with author, title, DOI) is absent from all four; this requires a dedicated NLP post-processor like GROBID. **Cross-reference resolution**, author/affiliation extraction, and supplementary material handling are not primary features of any tool. The table below summarizes capabilities:

| Capability | MarkItDown | Docling | Marker | MinerU |
|---|---|---|---|---|
| Multi-column layout | ❌ Garbled | ✅ AI reading order | ✅ Surya layout | ✅ Best (LayoutReader) |
| Equation → LaTeX | ❌ Lost | ⚠️ Granite-Docling | ✅ Surya LaTeX OCR | ✅ UniMERNet (best) |
| Table structure | ❌ None | ✅ TableFormer (best) | ✅ Good with LLM | ✅ RapidTable |
| Figure/caption | ❌ None | ✅ Picture detection | ✅ Extracts + saves | ✅ Extracts + captions |
| Citation parsing | ❌ None | ⚠️ JATS only | ⚠️ Format only | ⚠️ Layout label only |
| License | MIT | MIT | GPL/NC | AGPL |
| Speed (L4 GPU) | ~100× fastest | 0.49s/page | 0.86s/page | **0.21s/page** |

---

## PyMuPDF is the foundation; pdfplumber and pdfminer fill gaps

### PyMuPDF delivers speed and structured data

PyMuPDF (`pip install pymupdf`, AGPL/commercial) wraps the C library MuPDF and is **50–60× faster than pdfminer.six**. Its `page.get_text("dict")` API returns a hierarchical structure — blocks → lines → spans — where each span carries `font`, `size`, `flags` (bold=16, italic=2, superscript=1, monospaced=8), `color`, `bbox`, and `origin`. This font metadata is the essential signal for heading detection: cluster font sizes across the document, identify the title as the largest, section headings as the next tier, and body text as the mode.

The `"blocks"` mode returns tuples of `(x0, y0, x1, y1, text, block_no, type)` where `type=0` is text and `type=1` is image — sufficient for column detection via x-coordinate gap analysis. Image extraction is first-class: `page.get_images()` returns references, `doc.extract_image(xref)` returns bytes and format. Table detection landed in v1.23+ via `page.find_tables()` with pandas export. The companion library **pymupdf4llm** (`pip install pymupdf4llm`) produces Markdown at ~0.12s/page and includes `column_boxes()` for multi-column detection.

```python
# Key API pattern for structured extraction
data = page.get_text("dict", sort=True)
for block in data["blocks"]:
    if block["type"] == 0:  # text
        for line in block["lines"]:
            for span in line["spans"]:
                # span["font"], span["size"], span["flags"], span["bbox"]
```

### pdfplumber excels at table extraction

pdfplumber (MIT license, `pip install pdfplumber`) builds on pdfminer.six and provides the **best rule-based table detection** in the Python ecosystem. Its algorithm finds explicit and implied lines, merges overlapping edges, identifies intersections, and groups contiguous cells. The `table_settings` parameter offers fine-grained control: `vertical_strategy` and `horizontal_strategy` accept `"lines"`, `"text"`, or `"explicit"`. Its visual debugging via `page.to_image().debug_tablefinder()` is invaluable during development. Character-level access includes `x0`, `top`, `fontname`, `size` per character. The tradeoff is speed: **~95× slower than PyMuPDF** for text extraction.

### pdfminer.six provides the most sophisticated layout analysis

pdfminer.six (MIT license) offers `LAParams` with tunable parameters: `char_margin` (horizontal grouping threshold), `line_margin` (paragraph grouping), `word_margin` (space insertion), and `boxes_flow` (-1.0 to +1.0, controlling horizontal vs. vertical reading order). Its layout object hierarchy — `LTPage → LTTextBox → LTTextLine → LTChar` — provides character-level font and position data. This is why MarkItDown chose it: MIT license plus the most control over text grouping. But it is **50–60× slower than PyMuPDF** and lacks table detection, image extraction, and OCR.

### pypdf and other options

pypdf (BSD-3, `pip install pypdf`) handles PDF manipulation (merge, split, encrypt) and basic text extraction but provides **no structured layout data** — no block/line/span hierarchy, no table detection, no image content extraction. Its `visitor_text` callback gives transformation matrices but is low-level and awkward. Use pypdf only for metadata extraction and PDF manipulation, not for content parsing.

For tables specifically, **Camelot** (MIT, requires Ghostscript) provides two modes — Lattice (border-based, image processing) and Stream (position-based, borderless tables) — with accuracy/whitespace quality metrics. **Tabula-py** (MIT, requires Java) wraps Tabula and handles multi-page tables but over-detects non-table text. For scanned PDFs, **pdf2image + pytesseract** is the essential fallback. **pypdfium2** (Apache-2.0) is as fast as PyMuPDF but provides only raw text without structural metadata.

---

## Research paper elements each demand specialized approaches

### Multi-column detection via gap analysis and reading order

The canonical open-source approach is PyMuPDF4LLM's `column_boxes()`: extract all text blocks with bounding boxes, project x-coordinates onto the horizontal axis, identify significant vertical whitespace bands spanning most of the page height, and cluster blocks into columns. Reading order is then left-to-right within detected columns, top-to-bottom within each. This handles standard two-column layouts (IEEE, ACM, RevTeX) well. Single-column sections within multi-column pages (title, abstract) require special handling — typically detected by blocks spanning the full page width.

For higher accuracy, ML-based approaches use object detection models trained on **DocLayNet** (80k+ annotated pages, 11 label types) or **PubLayNet**. **LayoutReader** (Microsoft) is a seq2seq model pretrained specifically on reading order detection. Docling and MinerU both use AI-based reading order as a post-processing step.

### Equations require visual recognition, not text extraction

Equations in PDFs exist as individual character glyphs at arbitrary positions, embedded bitmaps, or vector drawing commands — **PDF stores visual appearance, not semantic meaning**. A fraction is just characters at different y-coordinates. Three viable extraction approaches exist:

**End-to-end page-level models** like **Nougat** (Meta, Swin Transformer encoder + mBART decoder, trained on arXiv source/PDF pairs) convert entire page images to Markdown with LaTeX. Best for arXiv papers but suffers from repetition degeneration on ~1.5% of pages. **Region-level OCR** with **Texify** (Donut-based, handles inline + display math + surrounding text, benchmarks higher than pix2tex and Nougat on mixed content) or **pix2tex** (ViT encoder + Transformer decoder, single formula images only). **MinerU's pipeline** combines YOLOv8 for formula detection with **UniMERNet** for recognition — currently the strongest open-source equation pipeline.

Display equations are detected by centered alignment, vertical whitespace, and numbering patterns like `(1)`. Inline equations are detected by font changes (math italic), special characters (∑, ∫, Greek letters), and baseline shifts.

### Tables need ML detection plus rule-based structure recognition

The problem decomposes into detection (where are tables?) and structure recognition (what are the rows, columns, cells, and spans?). **Table Transformer (TATR, Microsoft)** uses DETR (ResNet + encoder-decoder Transformer) trained on **PubTables-1M** (947k annotated tables from PubMed Central), achieving >0.9 recall on scientific documents. Two models: `microsoft/table-transformer-detection` and `microsoft/table-transformer-structure-recognition-v1.1-all`, both on HuggingFace. IBM's **TableFormer** (used in Docling) handles the hardest cases: partial borders, empty cells, spanning cells, hierarchical headers.

For a lightweight approach, detect table regions via TATR, then extract cell content with pdfplumber's coordinate-based algorithm. Markdown pipe tables cannot represent merged cells natively; complex tables may need HTML `<table>` fallback within the Markdown output.

### Figure extraction combines detection with caption association

**PDFFigures 2.0** (Allen AI, Scala) is the gold standard for scholarly papers: it detects caption keywords ("Figure", "Fig.", "Table"), classifies text blocks as body/image/caption by font and position, generates rectangular region proposals adjacent to each caption, and scores assignment. **DeepFigures** (ResNet-101, trained on 5.5M auto-labeled arXiv/PubMed examples) provides neural figure detection at scale. For a Python-native approach, Docling's layout model detects "Picture" elements, and images can be extracted via PyMuPDF's `page.get_images()` or by rasterizing detected bounding box regions at high DPI.

Caption-to-figure association follows proximity rules: figure captions appear below figures, table captions appear above tables. Multi-figure panels (subfigures) remain a hard unsolved problem.

### Citation parsing is a separate NLP problem best served by GROBID

**GROBID** (Java, open-source) is the industry standard, used by ResearchGate, Internet Archive, CORE, and others at scale. It runs a cascade of CRF/deep learning sequence labeling models with 68 final labels, achieving **~0.87–0.95 F1** for reference parsing depending on corpus. It processes ~3 PDFs/second on commodity hardware and consolidates against CrossRef for DOI resolution. Python access is via `grobid-client-python`. GROBID handles in-text citation detection (numeric `[1]`, author-year `(Smith, 2020)`, superscript) and outputs structured TEI XML with separate fields for author, title, year, journal, volume, DOI.

For a lighter-weight approach, **AnyStyle** (Ruby, CRF-based) parses reference strings without the full GROBID infrastructure. **Science Parse** (Allen AI) extracts title, authors, abstract, body text, and bibliography.

### arXiv papers offer a unique advantage: LaTeX source availability

Most arXiv submissions include LaTeX source (downloadable via `https://arxiv.org/e-print/{arxiv_id}` as .tar.gz). The gold-standard approach for arXiv is to **parse LaTeX directly** using `TexSoup` or `pylatexenc`, converting sections, equations, and tables to Markdown without touching the PDF at all. Only fall back to PDF parsing when source is unavailable. Common templates include `article` class (single-column preprints), `IEEEtran.cls` (two-column), `revtex4-2` (APS, two-column), `acmart.cls` (ACM), and NeurIPS/ICML conference styles. arXiv PDFs use standard TeX fonts (Computer Modern, Latin Modern), have no DRM, and produce clean text extraction — making them the easiest category of research PDFs to parse.

---

## MCP server architecture follows a clear, documented pattern

### The protocol uses JSON-RPC 2.0 with three primitives

MCP (Model Context Protocol, by Anthropic) defines **tools** (model-controlled functions), **resources** (application-controlled data), and **prompts** (user-controlled templates). Communication uses JSON-RPC 2.0 over **stdio** (local, for Claude Desktop) or **Streamable HTTP** (remote/production; SSE transport is deprecated). The official Python SDK (`pip install "mcp[cli]"`, 21.6k GitHub stars) includes **FastMCP** at `mcp.server.fastmcp`, which provides decorator-based tool/resource/prompt registration.

A standalone **FastMCP** package (PrefectHQ, `pip install fastmcp`) adds server composition, proxying, OpenAPI spec generation, and authorization. Version 3.0 (released January 2026) powers ~70% of MCP servers across all languages.

### markitdown-mcp demonstrates the minimal viable pattern

Microsoft's `markitdown-mcp` exposes a **single tool** — `convert_to_markdown(uri)` — that accepts `file:`, `http:`, `https:`, or `data:` URIs and returns Markdown. This "outcomes over operations" design (one unified tool rather than per-format tools) is the recommended MCP pattern. Key design decisions worth adopting: URI-based input for flexibility, file size guards before processing, truncation with notes for large outputs, and error messages as actionable agent instructions.

For a research-paper converter, the recommended tool surface is **3–5 tools**: `convert_paper(uri)` for full conversion, `extract_metadata(uri)` for title/authors/abstract, `extract_references(uri)` for structured bibliography, and optionally `list_supported_formats()` for discovery. Tool annotations (`readOnlyHint: true`, `openWorldHint` as appropriate) help clients understand behavior. The lifespan pattern initializes heavy models once at server startup.

```python
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("PaperConverter")

@mcp.tool(annotations={"readOnlyHint": True})
def convert_paper(uri: str, max_pages: int | None = None) -> str:
    """Convert a research paper PDF to structured Markdown with LaTeX equations."""
    ...
```

---

## Modern Python library architecture follows established patterns

### Package structure: src layout with pyproject.toml

The `src/` layout is the modern standard (PyPA, pyOpenSci). It prevents accidental imports during testing and ensures tests run against the installed package. **`pyproject.toml`** (PEP 621) replaces `setup.py` as the single source of truth. **Hatchling** is the recommended build backend for its good defaults and simplicity; **uv** (`uv build`, `uv publish`) offers the fastest build times.

```
paper2md/
├── src/
│   └── paper2md/
│       ├── __init__.py        # Public API: convert, ConversionResult
│       ├── __main__.py        # python -m paper2md
│       ├── core.py            # Pipeline orchestration
│       ├── cli.py             # Typer CLI
│       ├── server.py          # MCP server
│       ├── _registry.py       # Plugin discovery
│       ├── converters/
│       │   ├── base.py        # ABC: BaseConverter
│       │   ├── pdf.py         # PDF → structured blocks
│       │   ├── latex.py       # LaTeX source → Markdown
│       │   └── html.py        # HTML → Markdown
│       ├── extractors/
│       │   ├── layout.py      # Column detection, reading order
│       │   ├── equations.py   # Formula detection + LaTeX OCR
│       │   ├── tables.py      # Table detection + structure
│       │   ├── figures.py     # Figure/caption extraction
│       │   └── references.py  # Citation/bibliography parsing
│       └── utils/
├── tests/
├── docs/
└── pyproject.toml
```

### Dual interface: same core, thin wrappers

The core `convert()` function lives in `core.py`. Both the Python API (exported from `__init__.py`) and CLI (`cli.py` using **Typer**) are thin wrappers. Typer is the recommended CLI framework — it derives parameters from Python type hints, includes shell completion, and integrates with Rich for formatted output. Register the CLI via `[project.scripts]` in pyproject.toml: `paper2md = "paper2md.cli:app"`.

### Plugin system via entry points

Use `importlib.metadata.entry_points(group="paper2md.converters")` for plugin discovery — the standard PyPA mechanism. Define an `AbstractBaseClass` for converters with `convert()` and `supported_extensions`. Built-in converters register via `[project.entry-points."paper2md.converters"]` in pyproject.toml. Third-party plugins do the same in their own pyproject.toml and are automatically discovered when installed. This mirrors pytest's plugin system (`pytest11` entry point group).

### Optional dependencies with graceful degradation

Declare extras in `[project.optional-dependencies]`: a minimal core with `typer` and `rich`, then `pdf = ["pymupdf"]`, `ml = ["torch", "transformers"]`, `ocr = ["pytesseract"]`, `all = ["paper2md[pdf,ml,ocr]"]`. Use lazy imports inside functions with actionable error messages:

```python
try:
    import pymupdf
except ImportError:
    raise ImportError(
        "PDF support requires: pip install paper2md[pdf]"
    ) from None
```

### Publishing and CI/CD

Use **Trusted Publishing** (OIDC) via `pypa/gh-action-pypi-publish@release/v1` — no long-lived API tokens, PyPI issues temporary credentials to GitHub Actions. Version management via **setuptools-scm** (derives version from git tags). Testing with **pytest + nox** across Python 3.10–3.13. Linting with **Ruff** (replaces flake8, black, isort — 10–100× faster). Type checking with **mypy** in strict mode plus a `py.typed` marker file. Documentation with **MkDocs Material + mkdocstrings**.

---

## Recommended architecture for the new library

The optimal design chains specialized extractors in a configurable pipeline rather than building a monolithic converter. Each stage is independently testable and swappable.

**Stage 1 — Source detection:** If the input is an arXiv ID, download LaTeX source and parse directly (bypassing PDF entirely). If it's a PDF, proceed to Stage 2. This single optimization handles a large fraction of research paper use cases with near-perfect accuracy.

**Stage 2 — Page-level extraction:** Use PyMuPDF `get_text("dict")` for fast structured text with font/position data. Render page images at 200+ DPI for ML model input. Detect whether OCR is needed (check if text layer exists).

**Stage 3 — Layout analysis:** Use pymupdf4llm's `column_boxes()` for column detection as the fast default. Optionally use a DocLayout-YOLO or RT-DETR model for ML-based layout classification. Determine reading order via x-coordinate clustering for rule-based mode or LayoutReader for ML mode.

**Stage 4 — Element extraction:** Route detected regions to specialized extractors. Tables go to pdfplumber or TATR + TableFormer. Equations go to Texify or UniMERNet. Figures get rasterized and captions extracted via regex patterns. References go to GROBID (via HTTP client) or a lightweight CRF parser.

**Stage 5 — Markdown assembly:** Map sections to heading levels using font-size clustering and numbering patterns (`^\d+\.\s+`). Embed equations as `$...$` (inline) and `$$...$$` (display). Format tables as pipe tables or HTML for complex spans. Reference figures as `![Figure N caption](figures/fig_N.png)`. Format bibliography as a numbered list with structured fields.

The plugin architecture allows swapping any extractor (e.g., replacing the rule-based table extractor with an ML-based one) without changing the pipeline. The MCP server wraps the same `convert()` function, adding URI handling and output truncation. This design keeps the core library dependency-light (PyMuPDF + pdfplumber + Typer) while offering ML-powered extraction as optional extras.