# Extractor — Implementation Plan

**Schema Version:** 1.1.0
**Authors:** ZySec AI
**Date:** 2026-03-24

---

## Technology Decisions

| Concern | Chosen Technology | Rationale |
|---|---|---|
| Build backend | hatchling | PEP 517/518, no setup.py required, clean extras |
| CLI framework | typer | Type-annotated, auto-generates help, integrates with rich |
| Progress / output | rich | Progress bars, pretty error panels, no extra deps |
| PDF digital | pymupdf4llm | Fast, high-quality text + layout extraction from selectable PDFs |
| PDF scientific | GROBID + MinerU | Best-in-class reference/equation extraction for academic PDFs |
| PDF OCR | MinerU OCR branch | Same pipeline, OCR mode activated for scanned pages |
| DOCX/XLSX/PPTX | python-docx, openpyxl (read_only), python-pptx | Official OOXML libraries, stable APIs |
| HTML | trafilatura + lxml | trafilatura handles boilerplate removal; lxml handles structural parsing |
| Markdown | mistletoe | Pure Python, full AST, no subprocess |
| EPUB | ebooklib + BeautifulSoup4 | ebooklib for container; BS4 for HTML content extraction |
| LaTeX | TexSoup + latexmlpy | TexSoup for document structure; latexmlpy for MathML conversion |
| CSV | stdlib csv | No external dep needed |
| Audio | whisperx (faster-whisper + pyannote.audio) | Accurate transcription + speaker diarization |
| Magic bytes | puremagic | Pure Python, no libmagic system dep |
| PDF encryption | pikepdf | Detects encrypted PDFs without decrypting |
| Encoding detection | charset-normalizer | Better than chardet for multilingual docs |
| Token counting | tiktoken cl100k_base | Fast, consistent, GPT-4 compatible |
| Schema validation | pydantic v2 | Fast, Pythonic, JSON Schema export |
| ID generation | hashlib.sha256 (stdlib) | No extra dep |
| Testing | pytest + pytest-cov | Standard, integrates with CI |
| Linting | ruff | Replaces flake8 + isort + pyupgrade in one tool |
| Type checking | mypy (strict) | Catches interface mismatches early |
| CI | GitHub Actions | Matrix test across Python 3.10/3.11/3.12 |

---

## Phase 0: Scaffolding

**Goal:** A working repo skeleton that passes CI, with schema types defined and no actual parsing logic yet.

**Estimated duration:** 1–2 days

### Files to create

```
extractor/
├── .github/
│   └── workflows/
│       └── ci.yml
├── src/
│   └── extractor/
│       ├── __init__.py
│       ├── cli.py
│       ├── schema.py
│       ├── errors.py
│       ├── registry.py
│       ├── classifier.py
│       ├── quarantine.py
│       ├── heading_stack.py
│       ├── chunker.py
│       ├── writer.py
│       └── parsers/
│           ├── __init__.py
│           └── base.py
├── tests/
│   ├── conftest.py
│   ├── unit/
│   │   ├── test_schema.py
│   │   ├── test_heading_stack.py
│   │   ├── test_chunker.py
│   │   └── test_id_generation.py
│   ├── integration/   # empty, populated in Phase 1+
│   └── golden/
│       ├── fixtures/  # empty, populated in Phase 1+
│       └── expected/
├── pyproject.toml
├── .ruff.toml
└── mypy.ini
```

### pyproject.toml (complete)

```toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "extractor"
version = "1.2.0"
description = "Production-grade document extraction to versioned RAG-ready JSON/JSONL"
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
dependencies = [
    "typer>=0.12",
    "rich>=13",
    "pydantic>=2.0",
    "puremagic>=1.28",
    "pikepdf>=8",
    "charset-normalizer>=3",
    "tiktoken>=0.7",
    "pymupdf4llm>=0.0.17",
    "python-docx>=1.1",
    "openpyxl>=3.1",
    "python-pptx>=0.6",
    "trafilatura>=1.12",
    "lxml>=5",
    "mistletoe>=1.3",
    "ebooklib>=0.18",
    "beautifulsoup4>=4.12",
    "requests>=2.32",   # GROBID HTTP client
]

[project.optional-dependencies]
scientific = [
    "mineru>=1.0",
    "texsoup>=0.3",
    "latexmlpy>=0.1",
]
ocr = [
    "mineru[ocr]>=1.0",
]
audio = [
    "whisperx>=3.1",
    "faster-whisper>=1.0",
    "pyannote.audio>=3.1",
]
full = [
    "extractor[scientific]",
    "extractor[ocr]",
    "extractor[audio]",
]
dev = [
    "pytest>=8",
    "pytest-cov>=5",
    "ruff>=0.4",
    "mypy>=1.10",
    "types-requests",
    "types-beautifulsoup4",
]

[project.scripts]
extractor = "extractor.cli:app"

[tool.hatch.build.targets.wheel]
packages = ["src/extractor"]
```

### schema.py skeleton (required fields, element_type registry)

Define in `src/extractor/schema.py`:
- `SCHEMA_VERSION = "1.1.0"`
- `ELEMENT_TYPE_FAMILIES` dict mapping family prefix → list of valid subtypes
- `BaseElement` Pydantic model with all required fields
- Subclass models: `TableElement`, `EquationElement`, `CitationElement`, `ChunkRecord`, `ManifestRecord`

All required fields on BaseElement:
```
id, element_type, text, section_path, section_path_tier,
sequence_index, extraction_method, confidence, schema_version
```

### CI workflow (.github/workflows/ci.yml)

```yaml
name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pip install -e ".[dev]"
      - run: ruff check src/ tests/
      - run: mypy src/extractor
      - run: pytest tests/unit/ --cov=extractor --cov-report=term-missing
```

### Deliverables checklist

- [ ] `pyproject.toml` with all deps and extras defined
- [ ] `src/extractor/schema.py` with all Pydantic models and schema_version = "1.1.0"
- [ ] `src/extractor/errors.py` with full exception hierarchy
- [ ] `src/extractor/parsers/base.py` with `BaseParser` abstract class and `RawElement` dataclass
- [ ] `src/extractor/heading_stack.py` full implementation
- [ ] `src/extractor/chunker.py` full implementation with tiktoken
- [ ] `tests/unit/test_schema.py` — required field presence, schema_version, table dual-format, equation latex
- [ ] `tests/unit/test_heading_stack.py` — all unit cases listed in architecture.md §6.2
- [ ] `tests/unit/test_chunker.py` — all unit cases listed in architecture.md §6.2
- [ ] CI green on Python 3.10/3.11/3.12
- [ ] `ruff` and `mypy --strict` pass with zero errors

---

## Phase 1: Core Formats

**Goal:** PDF (digital), DOCX, HTML, CSV, plain text fully working end-to-end with schema, golden tests.

**Estimated duration:** 5–7 days

### Files to create

```
src/extractor/parsers/
├── pdf_digital.py     # pymupdf4llm
├── docx.py            # python-docx
├── html.py            # trafilatura + lxml
├── csv.py             # stdlib csv
└── plaintext.py       # line-based, headings by heuristic

tests/integration/
├── test_pdf_digital.py
├── test_docx.py
├── test_html.py
├── test_csv.py
└── test_cli_e2e.py

tests/golden/fixtures/
├── simple.pdf         # 3-section digital PDF with a table and a code block
├── document.docx      # h1/h2/h3 headings, 1 table, 1 image
├── page.html          # article with nav, aside (to test boilerplate removal)
└── data.csv           # 5 columns, 20 rows

tests/golden/expected/
├── simple.pdf.elements.jsonl
├── simple.pdf.chunks.jsonl
├── document.docx.elements.jsonl
├── page.html.elements.jsonl
└── data.csv.elements.jsonl
```

### Parser implementation notes

**pdf_digital.py**
- Call `pymupdf4llm.to_markdown(path)` for initial extraction
- Parse the returned markdown to identify headings (# prefix), tables, code fences
- Map to RawElement stream
- Capture `page_num` from pymupdf metadata as `metadata.page_number`
- Image blocks → `media:image` with `metadata.bbox` and `metadata.page_number`

**docx.py**
- Iterate `document.paragraphs` and `document.tables`
- Paragraph style names starting with "Heading" → heading element, extract level from style name
- Tables → dual-format table element (iterate `.rows`, `.cells`)
- Inline images via `paragraph.runs` → `media:image`
- Preserve `metadata.style_name` on all paragraph elements

**html.py**
- Run trafilatura.extract() with `include_tables=True`, `include_images=True`
- Fall back to lxml etree parse for structural headings if trafilatura returns None
- Map h1–h6 → heading levels 1–4 (h5/h6 → tier 4)
- `<pre>` / `<code>` blocks → `code:block`

**csv.py**
- Entire file is a single `table:simple` element
- First row → headers if `--csv-has-header` (default True)
- `table.markdown` generated by joining with ` | ` pipes
- `table.structured.headers` and `table.structured.rows` populated directly
- Confidence: 1.0 (no ambiguity)

### CLI commands to implement in Phase 1

```bash
# Single file extraction
extractor run <file> [--mode elements|chunks] [--out FILE] [--strategy fast|accurate|ocr]

# Validate a JSONL output against the schema
extractor validate <jsonl-file>

# Print detected format info without extracting
extractor info <file>
```

### Integration test structure (example for PDF)

```python
# tests/integration/test_pdf_digital.py
def test_pdf_digital_elements_schema_valid(pdf_fixture):
    elements = list(run_extract(pdf_fixture, mode="elements"))
    for el in elements:
        assert el["schema_version"] == "1.1.0"
        assert el["element_type"].startswith(("structural:", "text:", "table:", "media:"))
        assert isinstance(el["section_path"], list)
        assert 1 <= el["section_path_tier"] <= 4

def test_pdf_digital_table_dual_format(pdf_fixture):
    tables = [e for e in run_extract(pdf_fixture) if e["element_type"] == "table:simple"]
    assert len(tables) >= 1
    for t in tables:
        assert "markdown" in t["table"]
        assert "structured" in t["table"]
        assert "headers" in t["table"]["structured"]
        assert "rows" in t["table"]["structured"]

def test_pdf_digital_sequence_index_monotonic(pdf_fixture):
    elements = list(run_extract(pdf_fixture, mode="elements"))
    indices = [e["sequence_index"] for e in elements]
    assert indices == list(range(len(indices)))
```

### Deliverables checklist

- [ ] `src/extractor/parsers/pdf_digital.py`
- [ ] `src/extractor/parsers/docx.py`
- [ ] `src/extractor/parsers/html.py`
- [ ] `src/extractor/parsers/csv.py`
- [ ] `src/extractor/parsers/plaintext.py`
- [ ] `src/extractor/quarantine.py` (full implementation with all checks)
- [ ] `src/extractor/classifier.py` (full MIME + extension routing)
- [ ] `src/extractor/registry.py` (all Phase 1 parsers registered)
- [ ] `src/extractor/cli.py` (`run`, `validate`, `info` commands)
- [ ] `src/extractor/writer.py` (JSONL serialization + manifest generation)
- [ ] All golden fixtures created and expected outputs committed
- [ ] Integration tests passing for all 5 formats
- [ ] `extractor run tests/golden/fixtures/simple.pdf` produces valid JSONL

---

## Phase 2: Extended Formats

**Goal:** XLSX, PPTX, EPUB, Markdown, JSON passthrough, XML.

**Estimated duration:** 4–5 days

### Files to create

```
src/extractor/parsers/
├── xlsx.py        # openpyxl read_only
├── pptx.py        # python-pptx
├── epub.py        # ebooklib + BeautifulSoup4
├── markdown.py    # mistletoe AST
├── json_pass.py   # JSON → meta:document wrapper + content elements
└── xml.py         # lxml etree, generic XML

tests/integration/
├── test_xlsx.py
├── test_pptx.py
├── test_epub.py
└── test_markdown.py

tests/golden/fixtures/
├── spreadsheet.xlsx    # 2 sheets, formulas, merged cells
├── presentation.pptx   # 5 slides, speaker notes, embedded image
├── book.epub           # 3-chapter epub
└── notes.md            # nested headings, table, code fence, blockquote
```

### Parser implementation notes

**xlsx.py**
- Open with `openpyxl.load_workbook(path, read_only=True, data_only=True)`
- Each worksheet → section (heading element with sheet name)
- Each worksheet → one `table:simple` element (use dual-format)
- `data_only=True` reads computed cell values, not formulas
- Merged cells: expand to fill all covered cells with first cell's value, add `metadata.merged_cell = true`
- Empty sheets → `meta:extraction_warning` element

**pptx.py**
- Each slide → section boundary (push slide title as heading)
- Slide title shape → `structural:heading` level 1
- Text shapes → `text:paragraph`
- Table shapes → `table:simple` dual-format
- Image shapes → `media:image`
- Speaker notes → `text:note` with `metadata.is_speaker_note = true`

**markdown.py**
- Use mistletoe to generate AST: `mistletoe.Document(text)`
- Walk AST nodes: `Heading`, `Paragraph`, `Table`, `CodeFence`, `BlockCode`, `Image`, `List`
- Map ATX heading level directly to heading_level 1–4
- Fenced code blocks → `code:block` with `metadata.language` from fence info string
- GFM tables → dual-format table element

**epub.py**
- `book = ebooklib.epub.read_epub(path)`
- Iterate `book.get_items_of_type(ebooklib.ITEM_DOCUMENT)` in spine order
- For each HTML document item, use BeautifulSoup to walk headings and paragraphs
- Chapter boundaries set by spine item changes (push chapter title as h1)

### Deliverables checklist

- [ ] All 6 parser files implemented
- [ ] Golden fixtures for all formats
- [ ] Integration tests for XLSX, PPTX, EPUB, Markdown
- [ ] `extractor info` correctly identifies all new formats
- [ ] Merged cell handling tested explicitly in `test_xlsx.py`
- [ ] Speaker notes appear as `text:note` elements in `test_pptx.py`

---

## Phase 3: Scientific Formats

**Goal:** GROBID + MinerU for scientific PDFs, LaTeX parser, equation extraction, citation linking.

**Estimated duration:** 6–8 days

### Files to create

```
src/extractor/parsers/
├── pdf_scientific.py   # GROBID REST + MinerU fallback
├── pdf_ocr.py          # MinerU OCR branch
└── latex.py            # TexSoup + latexmlpy

src/extractor/
└── grobid_client.py    # thin HTTP client for GROBID REST API

tests/integration/
├── test_pdf_scientific.py   # skipped if GROBID not running
├── test_pdf_ocr.py          # skipped if [ocr] not installed
└── test_latex.py

tests/golden/fixtures/
├── scientific.pdf    # arXiv-style paper with abstract, equations, references
├── scanned.pdf       # scanned single-page document
└── paper.tex         # LaTeX source with equations and bibliography
```

### Scientific schema elements

**Equations:**
```json
{
  "element_type": "scientific:equation_display",
  "text": "E = mc^2",
  "equation": {
    "latex": "E = mc^{2}",
    "mathml": "<math>...</math>"
  },
  "metadata": { "equation_number": "1" }
}
```

**Citations:**
```json
{
  "element_type": "scientific:citation",
  "text": "[1]",
  "citation": {
    "reference_entry_id": "<sha256-id-of-reference-element>",
    "citation_text": "[1]"
  }
}
```

**Reference entries:**
```json
{
  "element_type": "scientific:reference_entry",
  "text": "LeCun et al. (1989) ...",
  "reference": {
    "authors": ["LeCun, Y.", "Boser, B."],
    "year": "1989",
    "title": "...",
    "venue": "...",
    "doi": "..."
  }
}
```

### GROBID client (grobid_client.py)

```python
class GROBIDClient:
    def __init__(self, base_url: str = "http://localhost:8070"):
        self.base_url = base_url

    def is_available(self) -> bool:
        # GET /api/isalive → 200
        ...

    def process_fulltext(self, pdf_path: Path) -> dict:
        # POST /api/processFulltextDocument → TEI XML
        # Parse TEI XML → structured dict
        ...
```

### LaTeX parser notes

- TexSoup: walk `\section`, `\subsection`, `\subsubsection` → headings
- `\begin{equation}` / `\begin{align}` blocks → `scientific:equation_display` with raw LaTeX in `equation.latex`
- latexmlpy: convert each equation LaTeX → MathML for `equation.mathml`
- `\cite{key}` → `scientific:citation` element (cross-reference to bibliography)
- `\bibitem` entries → `scientific:reference_entry`
- `\begin{figure}` with `\caption` → `media:figure`

### Deliverables checklist

- [ ] `src/extractor/grobid_client.py`
- [ ] `src/extractor/parsers/pdf_scientific.py` with GROBID fallback logic
- [ ] `src/extractor/parsers/pdf_ocr.py`
- [ ] `src/extractor/parsers/latex.py`
- [ ] `pytest.mark.skipif` on GROBID-dependent tests
- [ ] Equation elements have both `equation.latex` and `equation.mathml` (mathml may be null if latexmlpy fails)
- [ ] Citation → reference_entry cross-references validated in `test_pdf_scientific.py`
- [ ] Manifest warning emitted when GROBID falls back to pdf_digital

---

## Phase 4: OCR + Audio Extras

**Goal:** Full OCR pipeline for scanned documents, audio transcription with speaker diarization.

**Estimated duration:** 4–5 days

### Files to create

```
src/extractor/parsers/
└── audio.py    # whisperx pipeline

tests/integration/
├── test_audio.py         # skipped if [audio] not installed
└── test_pdf_ocr.py       # may overlap with Phase 3
```

### Audio schema elements

```json
{
  "element_type": "text:transcript_segment",
  "text": "The quick brown fox jumps over the lazy dog.",
  "metadata": {
    "speaker": "SPEAKER_00",
    "start_time_s": 0.0,
    "end_time_s": 3.4,
    "language": "en",
    "word_timestamps": [
      {"word": "The", "start": 0.0, "end": 0.1},
      ...
    ]
  },
  "confidence": 0.97
}
```

### Audio parser notes (audio.py)

- Load audio via whisperx: `whisperx.load_audio(path)`
- Transcribe: `model.transcribe(audio, batch_size=16)`
- Align: `whisperx.align(result["segments"], align_model, metadata, audio, device)`
- Diarize: `diarize_model(audio)` → speaker segments
- Assign speakers: `whisperx.assign_word_speakers(diarize_segments, aligned_result)`
- Each segment → `text:transcript_segment` element
- `section_path` = `[filename_stem]` (no heading structure in audio)
- `sequence_index` increments per segment

### OCR pipeline notes (pdf_ocr.py)

- MinerU `magic-pdf` CLI or Python API, OCR mode
- Post-process MinerU output markdown into RawElement stream same as pdf_digital
- Add `extraction_method: "mineru_ocr"` on all elements
- `confidence` from MinerU's per-block confidence score if available, else 0.85 default

### Deliverables checklist

- [ ] `src/extractor/parsers/audio.py` with whisperx pipeline
- [ ] `tests/golden/fixtures/interview.mp3` (short synthetic audio)
- [ ] `tests/integration/test_audio.py` with skip guard
- [ ] Audio elements have `metadata.speaker`, `metadata.start_time_s`, `metadata.end_time_s`
- [ ] `extractor run interview.mp3` works end-to-end
- [ ] OCR confidence scores propagated to element `confidence` field

---

## Phase 4.5: GLiNER2 NER (v1.1.0)

**Goal:** Optional named entity recognition as a post-reassembly pipeline stage; entity metadata on elements and chunks.

### Files created / modified

- `src/extractor/schema.py`: `EntityAnnotation` model, `entities` field on `BaseElement`/`ChunkRecord`, `entity_types` on `ChunkMetadata`
- `src/extractor/ner.py` (new): `NERStep`, `NERStats`, `preload_model()`, local `_load_model()`, `NER_SKIP_TYPES` element gating, length-bucket batching
- `src/extractor/chunker.py`: `_build_chunk_entities()`, `_build_chunk_entity_types()` rollup functions
- `src/extractor/pipeline.py`: NER wired post-reassembly, graceful `ImportError` fallback
- `src/extractor/cli.py`: `--entities/--no-entities`, `--entities-model`, `--entities-types`, `--entities-threshold`
- `src/extractor/integrations/`: LangChain + LlamaIndex adapters updated with entity metadata

### Local model loading

```python
# Local inference only — requires extractor[ner]
from gliner import GLiNER
model = GLiNER.from_pretrained("fastino/gliner2-base-v1")
```

### Schema additions

```json
// On BaseElement (entities: null when NER not run, [] when ran and found nothing)
"entities": [{"text": "Paris", "label": "LOC", "score": 0.97, "start": 0, "end": 5}]

// On ChunkMetadata (fast-filter field for vector store payload filtering)
"entity_types": ["LOC", "ORG"]
```

### Deliverables checklist

- [ ] `src/extractor/ner.py` with `NERStep`, singleton model cache, element gating
- [ ] `src/extractor/schema.py` updated with `EntityAnnotation`, `entities`, `entity_types`
- [ ] `src/extractor/chunker.py` rollup functions implemented
- [ ] `src/extractor/pipeline.py` wires NER after table reassembly
- [ ] CLI flags `--entities/--no-entities`, `--entities-model`, `--entities-types`, `--entities-threshold`
- [ ] Integration adapters updated (`extractor.integrations.langchain`, `extractor.integrations.llamaindex`)
- [ ] Unit tests: element gating, rollup deduplication, `entities: None` vs `[]` semantics
- [ ] Graceful fallback when neither API key nor `[ner]` extras are present

---

## Phase 5: Polish

**Goal:** Production-grade CLI UX, documentation, PyPI publish readiness.

**Estimated duration:** 3–4 days

### CLI UX improvements

**Progress bar (rich):**
```
Extracting document.docx...
 ████████████████████ 100%  42 elements  3 chunks  0 warnings
```

**Error panels (rich):**
```
╭─ Quarantine Error ───────────────────────────────────────────────────────────╮
│ File: encrypted.pdf                                                           │
│ Reason: PDF is password-protected                                             │
│ Exit code: 2                                                                  │
╰───────────────────────────────────────────────────────────────────────────────╯
```

**New CLI commands:**
```bash
# Batch extract a directory
extractor run <dir> [--glob "**/*.pdf"] [--out DIR] [--workers N]

# Print schema for an element type
extractor schema [element-type]

# Check which parsers are available (optional deps)
extractor info
```

**extractor info output example:**
```
extractor v1.1.0

Core parsers:     pdf_digital, docx, xlsx, pptx, html, markdown, csv, epub, plaintext
Scientific:       [installed] pdf_scientific, pdf_ocr, latex
OCR:              [installed] pdf_ocr
Audio:            [NOT INSTALLED] run: pip install coresdk-extractor[audio]

GROBID:           [reachable] http://localhost:8070
```

### Files to create/update

```
src/extractor/
└── cli.py              # add run (directory support), schema, info commands + progress bars

docs/
├── architecture.md     # this document's sibling
├── implementation.md   # this document
└── schema_reference.md # generated from Pydantic models

CHANGELOG.md
```

### Deliverables checklist

- [ ] Progress bar on `run` using rich.progress
- [ ] Rich error panels for all ExtractorError subclasses
- [ ] `run` supports directory targets with `--workers` (multiprocessing.Pool)
- [ ] `extractor info` command showing parser availability + GROBID status
- [ ] `extractor schema` prints JSON Schema for any element type
- [ ] `pyproject.toml` version bumped to `1.1.0`
- [ ] `CHANGELOG.md` written
- [ ] All golden tests passing
- [ ] `pytest tests/ --cov=extractor --cov-report=term-missing` → ≥ 85% coverage
- [ ] `ruff check` passes
- [ ] `mypy --strict src/extractor` passes
- [ ] `python -m build` produces valid wheel and sdist
- [ ] Test install from wheel in clean venv: `pip install dist/extractor-1.1.0-py3-none-any.whl`

---

## Success Metrics for v1.0

| Metric | Target |
|---|---|
| Supported formats | PDF (digital/scientific/scanned), DOCX, XLSX, PPTX, HTML, Markdown, EPUB, LaTeX, CSV, plain text, JSON, XML, MP3/WAV/M4A |
| Schema compliance | 100% of emitted elements pass Pydantic validation |
| Test coverage | ≥ 85% line coverage across all non-optional code |
| Golden test pass rate | 100% on committed fixtures |
| CI green | Python 3.10, 3.11, 3.12 on ubuntu-latest |
| CLI exit codes | Correct exit code on all error paths (tested in test_cli_e2e.py) |
| Chunk invariants | No atomic element split across chunks (property-based test with hypothesis) |
| Install size (core) | `pip install extractor` (no extras) < 50 MB |
| Extraction speed | ≥ 10 pages/second for digital PDFs on a 4-core machine |
| PyPI publish | `pip install extractor` works from PyPI |
| Documentation | architecture.md, implementation.md, schema_reference.md all present and accurate |

---

## Full Phase Summary

| Phase | Focus | Key Deliverables | Est. Days |
|---|---|---|---|
| 0 | Scaffolding | pyproject.toml, schema models, CI, base classes | 1–2 |
| 1 | Core formats | PDF digital, DOCX, HTML, CSV, plaintext | 5–7 |
| 2 | Extended formats | XLSX, PPTX, EPUB, Markdown, JSON, XML | 4–5 |
| 3 | Scientific | GROBID, MinerU, LaTeX, equations, citations | 6–8 |
| 4 | OCR + Audio | MinerU OCR, whisperx diarization | 4–5 |
| 4.5 | GLiNER2 NER (v1.1.0) | NER stage, entity schema, chunk rollup, CLI flags, integration adapters | 3–4 |
| 5 | Polish | CLI UX, progress, docs, PyPI | 3–4 |
| **Total** | | | **26–35 days** |

---

## Appendix: Key Commands Reference

```bash
# Development setup
git clone <repo>
cd extractor
pip install -e ".[full,dev]"

# Run all tests
pytest tests/ -v --cov=extractor

# Run only unit tests (no fixtures needed)
pytest tests/unit/ -v

# Run golden tests and update expected outputs
pytest tests/golden/ --update-golden

# Lint + typecheck
ruff check src/ tests/
mypy --strict src/extractor

# Build distribution
python -m build

# Extract a file (elements mode)
extractor run paper.pdf --mode elements --out paper.elements.jsonl

# Extract a file (chunks mode), opt-in to 50-token overlap
extractor run paper.pdf --mode chunks --out paper.chunks.jsonl --overlap 50
# (default: 0, disabled). Opt in with --overlap 50

# Batch extract a directory
extractor run ./papers/ --mode elements --out ./output/

# Validate output
extractor validate paper.chunks.jsonl

# Check parser availability and detected format info
extractor info paper.pdf
```
