# paper2md Hardening — Final State

## Completed (129 tests, all passing)

### Output Quality Fixes (latest)
- [x] **Footnote detection**: Position+size+marker filter (`∗†‡§¶*`) separates footnotes from body text
- [x] **Pre-abstract duplication**: BODY blocks between title and Abstract reclassified as METADATA
- [x] **Table row splitting**: `_split_into_rows()` uses y0-to-y0 distance (gap=6pt) instead of y1-to-y0 gap
- [x] **Math subscript merging**: Consecutive same-type sub/superscripts merged (`_{t-1}` not `_{t}_{-}_{1}`)
- [x] **Inline math spacing**: Regex post-processing adds spaces around `$...$` and trims internal whitespace
- [x] **\sqrt bracing**: `\sqrtd_{k}` → `\sqrt{d_{k}}`, `\sqrt_{dk}` → `\sqrt{dk}`
- [x] **arXiv cover page**: Raw page text check for arXiv stamps + duplicate title detection

### Author Extraction (13/13 papers)
- [x] `_parse_authors()` with multi-strategy extraction
- [x] `_split_capitalized_names()` for space-separated names (ResNet, LDM)
- [x] HEADING blocks as names (BERT-style)
- [x] Title-embedded authors via `∗,` boundary (ViT-style)
- [x] Title text cleaned of embedded author names
- [x] Long text allowed when footnote markers present (GPT-3)
- [x] Tiered affiliation detection: `_STRONG_AFFIL` (always filter) + `_WEAK_AFFIL` (2+ required)
- [x] `_NAME_PARTICLES` support (de, von, van, der, etc.)
- [x] ACM all-caps format: `_is_allcaps_name_line()` + `_extract_allcaps_name_from_line()`
- [x] Tests: 14 author tests (13 papers + unit tests)

### Vector Graphics
- [x] Detection via `page.get_drawings()` + spatial clustering
- [x] **PNG rendering via `page.get_pixmap(clip=rect, dpi=150)`**
- [x] `Figure.image_data: bytes` for rendered PNG data
- [x] `Figure.xref: int` for raster image extraction
- [x] Tests: 3 tests (detection, rendering, no-table-rules)

### Clustered Figure Rendering
- [x] Multi-tile figures (diffusion grids) render as composite PNG
- [x] `_cluster_nearby_figures()` sets `xref=None` for multi-tile clusters
- [x] Pipeline renders merged bbox via `page.get_pixmap(clip=bbox, dpi=150)`
- [x] CLI saves composite images via `fig.image_data` path

### OCR Fallback
- [x] Mock scanned PDF at `test_papers/mock_scanned.pdf`
- [x] End-to-end OCR pipeline tested
- [x] Tests: 5 tests in TestOCR

### Math
- [x] Complexity scorer wired into pipeline (score ≥3 → image)
- [x] Fraction detection confirmed dead code (PDFs use vector lines)
- [x] Expanded CM_TO_LATEX: ~120 entries (up from ~60)
- [x] Greek variants (ϵ/ε, ϕ/φ, µ/μ), operators (∼, ∥, ∝, ◦), accents (¯, ˜, ˆ)
- [x] CMEX control char filtering (large delimiter fragments → empty)
- [x] PUA char filtering (U+F8xx bracket pieces → empty)
- [x] Garbled fragments reduced: DDPM 82→74, KAN 350→85 unmapped

### Content Quality Tests (19 tests)
- [x] Abstract content verification (4 papers)
- [x] Table cell value assertions (attention BLEU scores, ResNet error rates)
- [x] Reference content quality (Bahdanau ref, author/year population)
- [x] Math LaTeX quality (commands, subscripts, garble ceiling)
- [x] Section ordering (3 papers)

### MCP Server (3 tools + progress)
- [x] `convert_pdf` — full markdown with error handling + progress via ctx.info()
- [x] `convert_pdf_structured` — markdown + rich metadata (authors, abstract, captions)
- [x] `extract_metadata` — lightweight: title, authors, abstract, table of contents, refs
- [x] Progress callbacks via `Context.info()` for all tools

### CLI (progress reporting)
- [x] `--output-dir` / `-d` saves markdown + all figures to disk
- [x] Raster images extracted via `doc.extract_image(xref)`
- [x] Vector figures saved as pre-rendered PNG
- [x] Composite clustered figures saved via `image_data`
- [x] Markdown image paths rewritten to point to saved files
- [x] Progress reporting: `\r{stage}: {current}/{total}`

### Progress Callbacks
- [x] `PipelineConfig.progress_callback: Optional[Callable[[str, int, int], None]]`
- [x] 4 stages: analyzing → extracting (per-page) → references → assembling
- [x] CLI: inline progress display
- [x] MCP: ctx.info() logging
- [x] Errors in callbacks never crash pipeline

## Results (13 papers tested)
| Paper | Authors | Tables | Figs | Refs | Format |
|-------|---------|--------|------|------|--------|
| attention | 8 | 4 | 8 | 40 | arXiv single-col |
| resnet | 4 | 7 | 19 | 50 | arXiv two-col |
| bert | 4 | 10 | 7 | 61 | arXiv two-col |
| vit | 12 | 14 | 20 | 55 | ICLR all-caps |
| llama | 14 | 20 | 8 | 68 | Meta AI two-col |
| ddpm | 3 | 8 | 19 | 72 | arXiv two-col |
| ldm | 5 | 16 | 41 | 109 | CVPR image-heavy |
| gpt3 | 31 | 39 | 53 | 144 | arXiv single-col |
| webgis | 3 | 6 | 21 | 48 | thesis-style |
| chain_of_thought | 22 | 22 | 17 | 136 | survey format |
| kan | 8 | 12 | 50 | 118 | arXiv |
| mamba | 2 | 16 | 14 | 116 | arXiv |
| kan_survey | 5 | 13 | 7 | 96 | ACM format |
| mamba | 2 | 16 | 14 | 116 | arXiv |

### Latest Fixes (136 tests)
- [x] **Mamba abstract**: `_KNOWN_BOLD_HEADINGS` detects "Abstract" even when smaller than body size
- [x] **d_model rendering**: 3 regex patterns merge `$d$ model`, `R^{d}$ model`, `d}$ model` → `$d_{\text{model}}$`
- [x] **Display equation merging**: `_merge_same_y_lines()` combines equation fragments at same y-position (fractions, continuations)

## Known Remaining Gaps
1. **3-column layout**: Never tested (no IEEE 3-col paper available)
2. **Nature/Springer/Elsevier**: Never tested (would likely need journal-specific adaptations)
3. **Math quality**: ~74 garbled single-char math fragments in DDPM (irreducible — PDF math font encoding limitations)
