# Extractor — Architecture Design

**Schema Version:** 1.1.0
**Authors:** ZySec AI
**Date:** 2026-03-24

---

## 1. System Diagram

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                            extractor CLI / Python API                        │
│                                                                               │
│   extractor convert <file> [--output-mode elements|chunks] [--out FILE]      │
└────────────────────────────────┬────────────────────────────────────────────┘
                                 │ file path + options
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                            Quarantine Stage                                   │
│                                                                               │
│  puremagic (magic bytes)   pikepdf (PDF encryption)   charset-normalizer     │
│  zip bomb check (ratio)    file size ceiling          extension mismatch      │
│  OOXML zip bomb check      (.docx/.xlsx/.pptx/.odt/.ods/.odp containers)     │
│                                                                               │
│  PASS ──────────────────────────────────────────────────► FileRecord         │
│  FAIL ──► QuarantineError (quarantine log entry, exit 2)                     │
└────────────────────────────────┬────────────────────────────────────────────┘
                                 │ FileRecord { path, mime, encoding, size }
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           Format Classifier                                   │
│                                                                               │
│  mime_type + file extension + subtype heuristics                              │
│  Returns: FormatSpec { family, subtype, parser_key }                         │
│                                                                               │
│  Examples:                                                                    │
│    application/pdf + digital      → parser_key: "pdf_digital"                │
│    application/pdf + scientific   → parser_key: "pdf_scientific"             │
│    application/pdf + scanned      → parser_key: "pdf_ocr"                    │
│    application/vnd.openxmlformats → parser_key: "docx" / "xlsx" / "pptx"    │
│    text/html                      → parser_key: "html"                       │
│    text/plain + .md               → parser_key: "markdown"                   │
│    audio/*                        → parser_key: "audio"                      │
└────────────────────────────────┬────────────────────────────────────────────┘
                                 │ FormatSpec
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          Parser Registry                                      │
│                                                                               │
│  Registry maps parser_key → ParserClass                                      │
│  Each parser implements BaseParser.parse() → Iterator[RawElement]            │
│                                                                               │
│  ┌──────────────┐  ┌───────────────┐  ┌──────────────┐  ┌───────────────┐  │
│  │ PDFDigital   │  │ PDFScientific │  │  PDFScanned  │  │     DOCX      │  │
│  │ pymupdf4llm  │  │ GROBID+MinerU │  │  MinerU OCR  │  │  python-docx  │  │
│  └──────────────┘  └───────────────┘  └──────────────┘  └───────────────┘  │
│  ┌──────────────┐  ┌───────────────┐  ┌──────────────┐  ┌───────────────┐  │
│  │     XLSX     │  │     PPTX      │  │     HTML     │  │   Markdown    │  │
│  │  openpyxl    │  │  python-pptx  │  │ trafilatura  │  │   mistletoe   │  │
│  │  read_only   │  │               │  │   + lxml     │  │               │  │
│  └──────────────┘  └───────────────┘  └──────────────┘  └───────────────┘  │
│  ┌──────────────┐  ┌───────────────┐  ┌──────────────┐  ┌───────────────┐  │
│  │     EPUB     │  │     LaTeX     │  │     CSV      │  │     Audio     │  │
│  │   ebooklib   │  │ TexSoup+      │  │   csv stdlib │  │   whisperx    │  │
│  │              │  │ latexmlpy     │  │              │  │ faster-whisper│  │
│  └──────────────┘  └───────────────┘  └──────────────┘  └───────────────┘  │
└────────────────────────────────┬────────────────────────────────────────────┘
                                 │ Iterator[RawElement]
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          HeadingStack                                         │
│                                                                               │
│  Python dataclass stack — consumes RawElement stream                         │
│                                                                               │
│  @dataclass                                                                   │
│  class HeadingFrame:                                                          │
│      level: int          # 1-6, relative heading depth                        │
│      text: str           # heading text                                       │
│      detection_tier: int # quality tier at which this heading was detected    │
│                          # (1-4)                                              │
│                                                                               │
│  @dataclass                                                                   │
│  class HeadingStack:                                                          │
│      detection_tier: int = 1  # document-level baseline, set by classifier   │
│      frames: list[HeadingFrame] = field(default_factory=list)                │
│                                                                               │
│      def current_section_path_tier(self) -> int:                             │
│          # Use the most recently pushed frame's tier, or document baseline   │
│          return self.frames[-1].detection_tier if self.frames else \          │
│              self.detection_tier                                              │
│                                                                               │
│  Emits: section_path (ordered array), section_path_tier (quality tier 1–4)  │
│  Assigns: sequence_index (monotonic counter per document)                    │
│  Assigns: id (SHA256 of doc_id + sequence_index + text[:64])                 │
│                                                                               │
│  RawElement + section_path + id → NormalizedElement                          │
└────────────────────────────────┬────────────────────────────────────────────┘
                                 │ Iterator[NormalizedElement]
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                       Table Continuation Reassembly                           │
│                                                                               │
│  Merges table:continuation elements into their parent table:complex          │
│  Orphan continuation elements (no preceding table) emit meta:extraction_warning │
└────────────────────────────┬────────────────────────────────────────────────┘
                                 │ Iterator[NormalizedElement]
                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                     NER Stage — GLiNER2 (optional, v1.1.0)                   │
│                                                                               │
│  Local inference:  [ner] extras → fastino/gliner2-base-v1 (205M BERT)      │
│  Fallback:         [ner] not installed → W_NER_MODEL_UNAVAILABLE, skip NER  │
│                                                                               │
│  Skipped element types: code:block, scientific:equation_*, media:*           │
│  Text truncated to 1024 chars before BERT inference (512-token limit)        │
│  entities: None = NER not run  /  entities: [] = NER ran, nothing found     │
└────────────────────────────┬────────────────────────────────────────────────┘
                                 │ Iterator[NormalizedElement]  (entities populated)
                                 ▼
                    ┌────────────┴─────────────┐
                    │  --output-mode elements  │  --output-mode chunks
                    ▼                          ▼
          ┌──────────────────┐      ┌──────────────────────────┐
          │   Element Emitter │      │    Chunker               │
          │                  │      │                          │
          │  NormalizedElement│      │  section-aware window    │
          │  → JSON line      │      │  512 tok default         │
          │  (1:1 passthrough)│      │  0 tok overlap (user opt-in) │
          │                  │      │  atomic elements unsplit  │
          └────────┬─────────┘      │  2500 tok hard ceiling   │
                   │                │  → ChunkRecord            │
                   │                └──────────┬───────────────┘
                   │                           │
                   └──────────┬────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             Output Writer                                     │
│                                                                               │
│  Writes JSONL to stdout or --out FILE                                        │
│  On completion, writes manifest sidecar: <file>.manifest.json                │
│                                                                               │
│  Manifest: { source_file, sha256, schema_version, element_count,             │
│              chunk_count, parser_key, extraction_ts, warnings[] }            │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## 2. Component Responsibilities

| Component | Module | Responsibility |
|---|---|---|
| CLI entrypoint | `extractor.cli` | Argument parsing via Typer, exit codes, progress bar (rich) |
| Quarantine | `extractor.quarantine` | Magic-byte verification, encryption detection, zip bomb guard (generic + OOXML), encoding detection |
| Format Classifier | `extractor.classifier` | MIME + extension → FormatSpec, subtype detection (digital/scanned/scientific) |
| Parser Registry | `extractor.registry` | Maps parser_key → ParserClass, lazy import to avoid heavy optional deps |
| BaseParser | `extractor.parsers.base` | Abstract interface: `parse(file_record) → Iterator[RawElement]` |
| PDF Digital | `extractor.parsers.pdf_digital` | pymupdf4llm, extracts text blocks, headings, tables, images with bounding boxes |
| PDF Scientific | `extractor.parsers.pdf_scientific` | GROBID REST + MinerU, structured extraction of abstract/body/references/equations |
| PDF OCR | `extractor.parsers.pdf_ocr` | MinerU OCR branch for scanned PDFs |
| DOCX | `extractor.parsers.docx` | python-docx, preserves paragraph styles, tables, embedded images |
| XLSX | `extractor.parsers.xlsx` | openpyxl read_only, sheet-per-table, formula cells → computed value |
| PPTX | `extractor.parsers.pptx` | python-pptx, slide-per-section, speaker notes, shapes |
| HTML | `extractor.parsers.html` | trafilatura for main content extraction + lxml for structural details |
| Markdown | `extractor.parsers.markdown` | mistletoe AST walk, preserves heading hierarchy |
| EPUB | `extractor.parsers.epub` | ebooklib + BeautifulSoup, chapter-per-section |
| LaTeX | `extractor.parsers.latex` | TexSoup for structure + latexmlpy for equation conversion |
| CSV | `extractor.parsers.csv` | stdlib csv, entire file → single table element |
| Audio | `extractor.parsers.audio` | whisperx (faster-whisper + pyannote diarization), transcript segments |
| HeadingStack | `extractor.heading_stack` | Streaming section_path maintenance, sequence_index, id assignment |
| Table Reassembly | `extractor.pipeline` | Merges table:continuation elements into parent table:complex before chunking |
| NER | `extractor.ner` | GLiNER2 local NER (fastino/gliner2-base-v1), element-level inference, chunk-level rollup |
| Chunker | `extractor.chunker` | Generator-based section-aware chunker with tiktoken; rolls up entities per chunk |
| Schema | `extractor.schema` | Pydantic v2 models for all element types, schema_version enforcement |
| Output Writer | `extractor.writer` | JSONL serialization, manifest generation |
| Errors | `extractor.errors` | Typed exception hierarchy |

---

## 3. Architecture Decision Records

### ADR-001: src/ layout with pyproject.toml

**Status:** Accepted
**Context:** The package must be installable as a library and as a CLI, with optional heavy dependencies.
**Decision:** Use `src/extractor/` layout with `pyproject.toml` (hatchling build backend). Extras: `[ocr]`, `[audio]`, `[scientific]`, `[full]`.
**Consequences:** Prevents accidental imports from the repo root during development. Requires `pip install -e .[full]` for local dev. Extras gate the installation of whisperx, MinerU, and GROBID client.

---

### ADR-002: Parser routing by format+subtype key, not a universal parser

**Status:** Accepted
**Context:** Treating all PDFs identically leads to poor extraction quality for scientific PDFs and silently fails on scanned pages.
**Decision:** The classifier assigns a specific `parser_key` (e.g., `pdf_digital` vs `pdf_scientific` vs `pdf_ocr`). The registry routes to the correct parser class.
**Consequences:** More parser modules to maintain, but each parser is optimized for its format. Users can override the auto-detected parser_key via `--parser-key` flag.

---

### ADR-003: Content-addressed element IDs

**Status:** Accepted
**Context:** Downstream vector databases need stable, deduplicated IDs across re-extraction runs.
**Decision:** `id = SHA256(doc_id + ":" + str(sequence_index) + ":" + text[:64])`, hex-encoded, truncated to 16 hex chars (8 bytes, 64-bit ID space).
**Consequences:** IDs are stable if the document content and sequence do not change. Purely positional — if a section is reordered, IDs shift. Acceptable trade-off for simplicity.

---

### ADR-004: Dual-format tables (markdown + structured)

**Status:** Accepted
**Context:** RAG pipelines want markdown for LLM context windows; data pipelines want structured rows/cols.
**Decision:** Every `table:simple` element carries both `table.markdown` (GitHub Flavored Markdown) and `table.structured` (`{headers: [...], rows: [[...]]}` ).
**Consequences:** ~2x storage for table data. Both formats are derived from the same source parse; no secondary extraction step needed.

---

### ADR-005: Tables and equations are always atomic

**Status:** Accepted
**Context:** Splitting a table mid-row or an equation across chunk boundaries breaks meaning and downstream rendering.
**Decision:** The chunker treats `table:simple`, `table:complex`, `table:continuation`, `media:figure`, `code:block`, and `scientific:equation_display` element types as atomic — they are never split, and they are never shared between two chunks. Only display equations (`scientific:equation_display`) are atomic; inline equations (`scientific:equation_inline`) flow with surrounding text. If an atomic element exceeds the 512-token default window, it is placed in its own chunk (up to the 2500-token hard ceiling).
**Consequences:** Chunks containing large tables may exceed the nominal window size. The manifest records the maximum observed chunk size.

---

### ADR-006: Two output modes — elements and chunks

**Status:** Accepted
**Context:** Some consumers want raw elements for custom chunking; others want pre-committed chunks for direct vector insertion.
**Decision:** `--output-mode elements` emits a flat NormalizedElement stream. `--output-mode chunks` runs the chunker and emits ChunkRecord lines. Default is `chunks`.
**Consequences:** The chunker is only instantiated in chunks mode, keeping elements mode fast and dependency-free at runtime.

---

### ADR-007: Quarantine as hard gate, not advisory

**Status:** Accepted
**Context:** Encrypted PDFs and zip bombs must not silently produce empty or corrupt output.
**Decision:** Any quarantine failure raises `QuarantineError`, writes a quarantine log entry, and exits with code 2. No partial output is written.
**Consequences:** Stricter than advisory warnings, but prevents silent data loss. Users can bypass with `--skip-quarantine` (expert flag, undocumented in main help).

#### Quarantine Checklist

| Check | Applies To | Error Code |
|---|---|---|
| Magic-byte / MIME mismatch | All files | `E_MAGIC_MISMATCH` |
| PDF encryption | `.pdf` | `E_ENCRYPTED` |
| Generic zip bomb (ratio > 100x) | Any ZIP container | `E_ZIP_BOMB` |
| File size ceiling (> 500 MB compressed) | All files | `E_TOO_LARGE` |
| OOXML zip bomb — compression ratio | `.docx`, `.xlsx`, `.pptx`, `.odt`, `.ods`, `.odp` | `E_ZIP_BOMB` |
| OOXML uncompressed ceiling (> 500 MB) | `.docx`, `.xlsx`, `.pptx`, `.odt`, `.ods`, `.odp` | `E_TOO_LARGE` |
| Encoding detection failure | Text files | `E_ENCODING_UNKNOWN` |

**OOXML Zip Bomb Check** (runs before routing to any OOXML parser):

```python
OOXML_EXTENSIONS = {".docx", ".xlsx", ".pptx", ".odt", ".ods", ".odp"}
MAX_COMPRESSION_RATIO = 100  # reject if uncompressed/compressed > 100x
MAX_UNCOMPRESSED_BYTES = 500_000_000  # 500 MB uncompressed ceiling

def check_ooxml_zip_bomb(path: Path) -> None:
    """Uses stdlib zipfile only — no new dependency."""
    if path.suffix.lower() not in OOXML_EXTENSIONS:
        return
    with zipfile.ZipFile(path) as zf:
        total_compressed = sum(i.compress_size for i in zf.infolist())
        total_uncompressed = sum(i.file_size for i in zf.infolist())
        if total_compressed > 0 and (total_uncompressed / total_compressed) > MAX_COMPRESSION_RATIO:
            raise QuarantineError("E_ZIP_BOMB", f"OOXML compression ratio {total_uncompressed/total_compressed:.0f}x exceeds limit")
        if total_uncompressed > MAX_UNCOMPRESSED_BYTES:
            raise QuarantineError("E_TOO_LARGE", f"OOXML uncompressed size {total_uncompressed} exceeds 500MB limit")
```

This check is critical because python-docx and openpyxl unzip OOXML containers during instantiation. Without this check, a zip bomb in a `.docx` bypasses the generic zip bomb guard entirely.

---

### ADR-008: tiktoken cl100k_base for token counting

**Status:** Accepted
**Context:** Token counts must be consistent and fast across all chunk window calculations.
**Decision:** Use tiktoken with the `cl100k_base` encoding (GPT-4 / text-embedding-3 compatible). Token count is cached per element.
**Consequences:** Token counts are approximate for non-OpenAI models but sufficient for chunking window decisions. The encoding can be overridden via `--tokenizer` if needed.

---

### ADR-009: GROBID as optional external service for scientific PDFs

**Status:** Accepted
**Context:** GROBID requires a running Java service; bundling it is impractical.
**Decision:** The `pdf_scientific` parser calls GROBID via HTTP (default: `http://localhost:8070`). If GROBID is unavailable, the parser falls back to `pdf_digital` with a manifest warning.
**Consequences:** Scientific extraction requires the caller to stand up GROBID (Docker image available). The fallback ensures the pipeline never hard-fails on scientific PDFs.

---

### ADR-010: Section path as ordered string array, not a dotted string

**Status:** Accepted
**Context:** Heading text may contain dots, slashes, or other delimiters, making a flattened string ambiguous.
**Decision:** `section_path` is a JSON array of strings: `["Document Title", "Chapter 2", "Section 2.3"]`. `section_path_tier` is a detection quality indicator (1=native semantic markup, 2=font heuristic, 3=keyword heuristic, 4=positional fallback) — NOT a depth indicator. Use `len(section_path)` for depth.
**Consequences:** Downstream consumers must handle arrays, not strings. Joining with " > " for display is straightforward.

---

## Entity Extraction (v1.1.0)

GLiNER2-powered named entity recognition runs as an optional pipeline stage between table reassembly and chunking.

### Pipeline Position

```
File → Quarantine → Classifier → Parser → HeadingStack → TableReassembly → [NER] → Chunker → Writer
```

### Local Inference

- **Local model:** `[ner]` extras installed → `fastino/gliner2-base-v1` (205M BERT-based)
- **Fallback:** `[ner]` not installed → `W_NER_MODEL_UNAVAILABLE` warning, extraction continues

### Element-Level → Chunk-Level Rollup

NER runs per-element (with type gating — skips code, equations, media). `ChunkRecord.entities` is a deduplicated rollup. `ChunkMetadata.entity_types` is the fast-filter field for vector store payload filtering.

### Key Design Decisions

- NER runs on elements, not chunks — stable element IDs, type gating, per-element failure isolation
- `entities: None` (absent in JSON) = NER not run; `entities: []` = NER ran, nothing found
- Module-level singleton: model loaded once per process, keyed `"__api__"` for remote or `model_id` for local
- BERT 512-token limit: text truncated to 1024 chars (~256 cl100k tokens) before inference

---

## v1.1.0 Correctness Fixes

The following self-heal fixes landed alongside the GLiNER2 NER feature:

- `aextract()` async fire-and-forget fixed — executor now properly awaited, backpressure via `queue.put`
- `table:continuation` orphan elements no longer silently dropped — emit `meta:extraction_warning` instead
- `extract()` now forwards `tokenizer` and `inject_context_prefix` params to the pipeline
- `_collect_files` excludes the output directory to prevent re-parsing own JSONL output
- `OutputWriter` context manager (`__enter__`/`__exit__`) added for safe file cleanup on exception
- Chunk size guard now includes `overlap_tail` token count in the window calculation

---

## 4. Data Flow

```
FILE IN
  │
  ▼
[Quarantine Stage]
  puremagic → actual MIME
  size check → reject > 500 MB default
  pikepdf → detect encryption
  charset-normalizer → detect encoding for text files
  zip bomb → compressed/uncompressed ratio > 100x → reject
  OOXML zip bomb → zipfile inspect .docx/.xlsx/.pptx/.odt/.ods/.odp before parser open
    E_ZIP_BOMB: uncompressed/compressed > 100x
    E_TOO_LARGE: total uncompressed > 500 MB
  │
  ├─ FAIL → QuarantineError → quarantine.log entry → exit 2
  │
  ▼ PASS
FileRecord { path, actual_mime, declared_ext, encoding, size_bytes }
  │
  ▼
[Format Classifier]
  MIME + extension → family
  PDF subtype heuristics:
    - has selectable text layer? → digital
    - GROBID available + journal-like metadata? → scientific
    - no text layer → ocr
  │
  ▼
FormatSpec { family, subtype, parser_key }
  │
  ▼
[Parser Registry] → ParserClass(config)
  │
  ▼
parser.parse(file_record)
  │
  ▼ streaming Iterator[RawElement]
  Each RawElement:
  {
    raw_type: str,          # parser-internal type string
    text: str,              # extracted text content
    metadata: dict,         # parser-specific (bbox, style, page_num, etc.)
    heading_level: int|None # 1-4 if this element is a heading, else None
  }
  │
  ▼
[HeadingStack]
  Consumes RawElement stream
  Maintains stack: List[Tuple[level, heading_text]]
  On heading element: push/pop stack to correct depth
  On any element:
    section_path = [h.text for h in stack]
    section_path_tier = detection_quality_tier  # inherited from HeadingStack.detection_tier
    # 1 = native semantic markup (PDF tags, DOCX heading styles, HTML h1-h6)
    # 2 = font heuristic (font size + bold detection)
    # 3 = keyword heuristic (ALL CAPS, numbered patterns like "1.2.3 Title")
    # 4 = positional fallback (no heading signals found)
    sequence_index = monotonic counter
    id = sha256_id(doc_id, sequence_index, element.text)
    element_type = map raw_type → schema element_type family
  │
  ▼ Iterator[NormalizedElement]
  Each NormalizedElement is a fully-populated schema object.
  │
  ▼
[Table Continuation Reassembly]
  Merge table:continuation into preceding table:complex
  Orphan continuations → meta:extraction_warning
  │
  ▼
[NER Stage] (optional — v1.1.0)
  Local: [ner] extras → fastino/gliner2-base-v1 (205M, local)
  Fallback: [ner] not installed → W_NER_MODEL_UNAVAILABLE, skip
  Skips: code:block, scientific:equation_*, media:*
  Text truncated to 1024 chars before BERT inference
  entities: None = NER not run; entities: [] = ran, nothing found
  │
  ├─── --output-mode elements ──────────────────────────────────────────────┐
  │                                                                          │
  ▼                                                                          │
[Chunker] (only in chunks mode)                                             │
  section-aware sliding window                                              │
  window = 512 tokens, overlap = 0 tokens (user opt-in via --overlap N)     │
  hard ceiling = 2500 tokens                                                │
  atomic types (table:simple, table:complex, table:continuation, media:figure, code:block, scientific:equation_display) → never split                         │
  section boundary → always flush current chunk                            │
  each chunk gets: chunk_id, element_ids[], token_count, section_path      │
  │                                                                          │
  ▼                                                                          │
[Output Writer] ◄──────────────────────────────────────────────────────────┘
  serialize each record as JSON line
  write to stdout or --out FILE
  on StopIteration: write manifest sidecar
  manifest: {
    source_file, source_sha256, schema_version,
    extraction_ts, parser_key, output_mode,
    element_count, chunk_count (if chunks mode),
    max_chunk_tokens, warnings[]
  }
  │
  ▼
OUTPUT: <name>.jsonl + <name>.manifest.json
```

---

## 5. Error Handling Strategy

### 5.1 Error Hierarchy

```
ExtractorError (base)
├── QuarantineError       # exit code 2, file unsafe
├── ClassifierError       # exit code 3, cannot determine format
├── ParserError           # exit code 4, parser failed
│   ├── ParserUnavailableError   # optional dependency not installed
│   └── ParserTimeoutError       # parser exceeded time limit
├── OutputError           # exit code 5, cannot write output
└── ConfigurationError    # exit code 6, invalid CLI flags/options
```

### 5.2 Partial Output Policy

- **Quarantine failure:** No output written. Hard stop.
- **Classifier failure:** No output written. Hard stop. User must supply `--parser-key` to bypass.
- **Parser failure (hard):** If the parser raises an unrecoverable exception, the run fails. A partial `.jsonl` is NOT left on disk (written to a temp file, promoted only on success).
- **Parser failure (soft / element-level):** Individual elements that fail validation are replaced with a `meta:extraction_warning` element carrying the error text. Extraction continues. The manifest records `warnings[]`.
- **GROBID unavailable:** Falls back to `pdf_digital`. Manifest records warning.
- **Atomic element exceeds 2500-token ceiling:** Element is emitted as its own chunk with a manifest warning. Never truncated.

### 5.3 Exit Codes

| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Partial success — some elements written, some failed |
| 2 | Quarantine failure |
| 3 | Format classification failure |
| 4 | Parser failure |
| 5 | Output error — cannot write to output path |
| 6 | Configuration error — invalid CLI flags/options |

> **Source of truth for exit codes is SPEC.md §15.** A single `ExitCode` IntEnum in `src/extractor/exit_codes.py` must be imported by both CLI and error classes.

---

## 6. Testing Strategy

### 6.1 Test Layout

```
tests/
├── unit/
│   ├── test_quarantine.py
│   ├── test_classifier.py
│   ├── test_heading_stack.py
│   ├── test_chunker.py
│   ├── test_schema.py
│   └── test_id_generation.py
├── integration/
│   ├── test_pdf_digital.py
│   ├── test_pdf_scientific.py   # skipped if GROBID not available
│   ├── test_docx.py
│   ├── test_xlsx.py
│   ├── test_html.py
│   ├── test_markdown.py
│   ├── test_csv.py
│   ├── test_audio.py            # skipped if [audio] not installed
│   └── test_cli_e2e.py
├── golden/
│   ├── fixtures/                # input files for golden tests
│   │   ├── simple.pdf
│   │   ├── scientific.pdf
│   │   ├── scanned.pdf
│   │   ├── document.docx
│   │   ├── spreadsheet.xlsx
│   │   ├── presentation.pptx
│   │   ├── page.html
│   │   ├── notes.md
│   │   ├── paper.tex
│   │   ├── data.csv
│   │   └── interview.mp3
│   └── expected/                # committed expected JSONL outputs
│       ├── simple.pdf.elements.jsonl
│       ├── simple.pdf.chunks.jsonl
│       └── ...
└── conftest.py
```

### 6.2 Unit Test Cases (key examples)

**test_heading_stack.py**
- `test_push_single_heading`: single h1 → section_path = [h1_text], tier = 1
- `test_push_nested_headings`: h1 > h2 > h3 → correct 3-element path
- `test_pop_on_same_level`: second h2 after first h2 → stack correctly replaces
- `test_skip_level`: h1 then h3 (no h2) → section_path has 2 entries, tier unchanged (tier reflects detection quality, not depth)
- `test_sequence_index_monotonic`: 100 elements → indices 0..99 with no gaps

**test_chunker.py**
- `test_atomic_table_not_split`: table exceeding window stays in one chunk
- `test_section_boundary_flush`: elements across section boundary go to separate chunks
- `test_overlap_carries_context`: overlap tokens from previous chunk appear at chunk start
- `test_hard_ceiling_single_element`: 3000-token table → own chunk, manifest warning
- `test_empty_document`: zero elements → zero chunks, no error

**test_schema.py**
- `test_required_fields_present`: every element type must have all required fields
- `test_schema_version_pinned`: schema_version must equal "1.1.0"
- `test_table_dual_format`: table element must have both markdown and structured keys
- `test_equation_latex_required`: equation element must have latex key

### 6.3 Golden File Regression Tests

Golden tests compare the full JSONL output of a known fixture file against a committed expected output. The comparison:
1. Parse both sides as lists of JSON objects
2. Compare field-by-field, excluding `extraction_ts` and `id` (which include timestamps or content-dependent hashes)
3. On mismatch, diff is printed element-by-element for easy debugging

To update golden files: `pytest tests/golden/ --update-golden`

### 6.4 Format-Specific Fixture Requirements

Each fixture must exercise:
- At least 2 heading levels
- At least 1 table
- At least 1 code block (where format supports it)
- Consistent encoding (UTF-8 unless testing encoding detection)
- A "tricky" case: embedded image, merged table cells, math formula, etc.

---

## 7. Extension Points

### 7.1 Adding a New Format Parser

1. Create `src/extractor/parsers/<format>.py` implementing `BaseParser`:

```python
from extractor.parsers.base import BaseParser, RawElement
from extractor.schema import FileRecord
from typing import Iterator

class MyFormatParser(BaseParser):
    parser_key = "my_format"
    supported_mimes = {"application/x-myformat"}

    def parse(self, file_record: FileRecord) -> Iterator[RawElement]:
        # yield RawElement instances
        ...
```

2. Register in `src/extractor/registry.py`:

```python
PARSER_REGISTRY["my_format"] = "extractor.parsers.my_format:MyFormatParser"
```

3. Add MIME/extension mapping in `src/extractor/classifier.py`:

```python
MIME_TO_PARSER["application/x-myformat"] = "my_format"
EXT_TO_PARSER[".myext"] = "my_format"
```

4. If the parser requires optional dependencies, add an extras group in `pyproject.toml`:

```toml
[project.optional-dependencies]
myformat = ["some-library>=1.0"]
```

5. Guard the import in the parser module:

```python
try:
    import some_library
except ImportError as e:
    raise ParserUnavailableError("my_format requires: pip install coresdk-extractor[myformat]") from e
```

6. Add integration test in `tests/integration/test_myformat.py` and a fixture file in `tests/golden/fixtures/`.

### 7.2 Adding a New Element Type

1. Add the type string to `ELEMENT_TYPE_REGISTRY` in `src/extractor/schema.py`.
2. If the type has custom required fields, add a Pydantic model inheriting from `BaseElement`.
3. Update the element_type family documentation in this file.
4. Add schema validation tests in `tests/unit/test_schema.py`.

### 7.3 Overriding the Chunker

The chunker is injected via the `OutputPipeline`:

```python
from extractor import OutputPipeline
from extractor.chunker import BaseChunker

class MyChunker(BaseChunker):
    def chunk(self, elements):
        ...

pipeline = OutputPipeline(chunker=MyChunker())
```

This allows downstream users to plug in custom chunking logic without forking the library.
