# extractor — Product Definition

## Problem Statement
Document extraction is the root cause of most RAG quality failures. Current tools (Docling, Unstructured, LlamaParse, pymupdf4llm) are either format silos or monolithic platforms. None provides a versioned, schema-stable interchange protocol that works as a composable pipeline primitive across all document types.

Key developer pains:
- Every tool requires a custom normalization wrapper to use downstream
- No tool provides section-based chunks (page-based = 13% RAG accuracy vs 87% for section-aware)
- No versioned schema contract: every library upgrade risks breaking vector store metadata
- Tables are either Markdown blobs OR JSON — never both; splitting tables destroys retrieval
- Scientific content (equations, citations) is silently dropped or mangled by most parsers

## Solution
**extractor** — a document extraction CLI + Python library that emits a formal, versioned JSONL stream where every record is schema-stable, section-aware, and RAG-ready.

The key insight: **the schema layer IS the product**. Parser quality fluctuates as the ecosystem evolves. The versioned schema contract is the durable value that no current tool provides.

## Feature Set

### P0 (v0.1)
- Streaming JSONL output (one element or chunk per line)
- manifest.json companion with TOC, stats, parser chain, warnings, errors
- Two modes: `elements` (raw semantic units) and `chunks` (pre-committed, section-aware)
- `section_path` breadcrumb on every record (4-tier fallback: native → font-heuristic → keyword → positional)
- Dual-format tables (Markdown + structured JSON), always atomic
- content-addressed IDs (SHA256-based, stable across re-runs)
- `extraction_method` + `confidence` on every record
- `schema_version` (semver) on every record
- Formats: PDF (digital), DOCX, HTML
- Python library: `extract(path) -> Iterator[Element]`
- CLI: `extractor run <file|dir|url> --mode --output --chunk-size --overlap`

### P1 (v0.3–v0.5)
- All 15+ formats: XLSX, PPTX, EPUB, Markdown, LaTeX, CSV, JSON, XML, plain text, RTF
- Scientific elements: equations (LaTeX), citations, reference entries, theorems
- Optional extras: `[ocr]`, `[audio]`, `[scientific]`, `[full]`
- JSON Schema 2020-12 document published at `schemas/v1/`
- Batch directory processing with progress bars
- `extractor validate` command

### P2 (v1.0)
- Audio transcription + speaker diarization (`[audio]` extra)
- Scientific PDF via GROBID + MinerU (`[scientific]` extra)
- Image OCR (`[ocr]` extra)
- Golden-file regression tests for all formats
- Incremental re-extraction (skip unchanged files)
- ReadTheDocs site

## Ideal Customer Profile

### Primary: ML/Data Engineer building RAG pipelines
- Needs: reliable JSONL for vector indexing, stable schema for downstream tools, section-aware chunks
- Pain: writes custom normalization wrappers around every parser used
- Value: drop-in pipeline primitive that replaces 200 lines of custom normalization code

### Secondary: Research Engineer / Data Scientist
- Needs: section-path metadata for hybrid retrieval, table structure for precise Q&A
- Pain: current tools produce flat text blobs; retrieval quality is unpredictable
- Value: structured extraction with provenance metadata enables systematic RAG evaluation

### Tertiary: Scientific Computing / Publishing
- Needs: equations in LaTeX, citation graphs, theorem/proof structure
- Pain: every scientific paper parser destroys mathematical content
- Value: `[scientific]` extra preserves equation + citation structure for RAG over academic literature

## Competitive Landscape

| Tool | Market Position | Our Advantage |
|---|---|---|
| Docling (IBM/LF AI) | 42k stars; best table accuracy; MIT | extractor is a routing layer; Docling can be a backend |
| Unstructured | $68M raised; 71+ connectors | No versioned schema; extractor's protocol is the differentiator |
| pymupdf4llm | Fastest PDF parser | PDF-only; no schema; extractor uses it as a backend |
| LlamaParse | Cloud-only, fast | Cloud-only; extractor is local/offline |
| MinerU | CVPR 2025 winner | GPU-heavy; PDF-only; extractor uses it as backend |

## Business Model
Open source (MIT). Growth via:
1. PyPI adoption → GitHub stars → community contributions
2. Potential: hosted API for teams that want cloud extraction with schema guarantees
3. Potential: enterprise support contracts for custom format parsers

## Roles & Stakeholders
| Role | Responsibility |
|---|---|
| Protocol Architect | Owns SPEC.md versioning contract and schema stability |
| Parser Engineer | Implements format-specific parsers + quarantine stage |
| RAG Quality Engineer | Owns chunker algorithm, section_path construction, confidence scoring |
| CLI/DX Engineer | Owns CLI UX, stdout/stderr contract, exit codes |
| Documentation | Maintains SPEC.md, README, format guides |
