> **Implementation Status:** This document describes the target protocol. For current implementation coverage and known gaps, see [`docs/gap-analysis.md`](gap-analysis.md) and [`docs/implementation-gaps.md`](implementation-gaps.md). The `extractor` library currently implements schema v1.1.0.

# extractor Protocol Specification
## Version 1.1.0

**Status:** Draft (Expert Panel Reviewed)
**Schema ID:** `https://coresdk.io/schemas/v1/`
**JSON Schema dialect:** 2020-12
**License:** MIT

---

## Table of Contents

1. [Overview](#1-overview)
2. [Versioning Contract](#2-versioning-contract)
3. [Output Files](#3-output-files)
4. [Root Envelope](#4-root-envelope)
5. [Base Element Schema](#5-base-element-schema)
6. [Element Type Reference](#6-element-type-reference)
7. [Section Path Construction](#7-section-path-construction)
8. [Chunk Schema](#8-chunk-schema)
9. [Table Representation Rules](#9-table-representation-rules)
10. [Scientific Element Representation](#10-scientific-element-representation)
11. [Manifest Schema](#11-manifest-schema)
12. [Stats Schema](#12-stats-schema)
13. [Stability Contract](#13-stability-contract)
14. [CLI Contract](#14-cli-contract)
15. [Exit Codes](#15-exit-codes)
16. [Error Records](#16-error-records)
17. [Parser Routing Table](#17-parser-routing-table)
18. [Chunker Algorithm](#18-chunker-algorithm)
19. [ID Derivation](#19-id-derivation)
20. [Confidence Scoring](#20-confidence-scoring)

---

## 1. Overview

The extractor protocol defines a **format-independent, versioned document interchange schema** for RAG (Retrieval-Augmented Generation) pipelines. Any compliant extractor implementation MUST:

1. Accept any supported file format (see §17)
2. Emit a streaming JSONL file — one self-describing JSON object per line
3. Emit a companion `manifest.json` with document metadata, stats, and processing log
4. Assign a `section_path` breadcrumb to every emitted record
5. Include `extraction_method` and `confidence` on every emitted record
6. Use content-addressed IDs (see §19)
7. Honor the stability contract (see §13)

### Design Principles

- **Section-based, not page-based** — chunk boundaries follow semantic heading structure, not physical page breaks
- **Atomic structured elements** — tables, code blocks, equations, and figures are never split across chunks
- **Dual-format tables** — every table carries both Markdown and structured JSON representations
- **Streaming by default** — output is a JSONL stream; no consumer needs to load the full document in memory
- **Provenance on every record** — consumers can always determine how a piece of content was extracted and with what confidence
- **Schema stability as a first-class contract** — downstream tools can pin to a schema version without fear of breakage

---

## 2. Versioning Contract

### Semantic Versioning

```
schema_version: "MAJOR.MINOR.PATCH"
```

| Change Type | Version Bump | Example |
|---|---|---|
| Breaking change to any stable field or type | MAJOR | Renaming `section_path` → `breadcrumb` |
| New optional field added | MINOR | Adding `metadata.word_count` |
| Experimental field promoted to stable | MINOR | Promoting `equation.mathml` |
| Bug fix, no schema change | PATCH | Correcting a field description |
| Experimental field removed | MINOR | Removing `x:slide_notes` |

### Negotiation Rules

```
Consumers MUST:   check schema_version before processing
Consumers MUST:   reject payloads where MAJOR version differs from expected
Consumers SHOULD: accept any 1.x.x payload when expecting 1.y.y (y ≤ x)
Consumers MUST NOT: assume presence of optional fields without checking
```

### Experimental Fields

Fields and element types prefixed with `x:` or listed in §13 as `experimental` may change in any minor version. Consumers MUST treat them as advisory only.

---

## 3. Output Files

Every extraction run produces exactly two files:

```
{basename}_elements.jsonl     # when --mode elements (default)
{basename}_chunks.jsonl       # when --mode chunks
{basename}_manifest.json      # always
```

Or to stdout (JSONL) + file (manifest):

```bash
extractor run document.pdf                         # stdout JSONL + document_manifest.json
extractor run document.pdf --out ./output/         # writes both files to ./output/
extractor run document.pdf --no-manifest           # suppresses manifest (disables reproducibility guarantees)
```

### JSONL Integrity

- Every line MUST be a valid, self-contained JSON object parseable in isolation
- A corrupt or truncated line does not invalidate preceding lines
- The file is written atomically: `{name}.jsonl.tmp` during extraction, renamed to `{name}.jsonl` on clean completion
- Partial files (`.tmp` suffix) indicate incomplete extraction; `manifest.status = "partial"`

### Stream Termination Sentinel

When output is written to stdout (no `--out` flag), the JSONL stream MUST be terminated with a sentinel record as the final line:

```json
{"type": "stream_end", "status": "complete", "total_elements": 847, "total_chunks": 94, "schema_version": "1.1.0"}
```

On partial completion or crash recovery:
```json
{"type": "stream_end", "status": "partial", "last_sequence_index": 213, "error_code": "E_PARSER_CRASH", "schema_version": "1.1.0"}
```

Rules:
- The sentinel is ONLY emitted when writing to stdout (pipe mode). It is NOT added when writing to `--out FILE` (the `.tmp` → rename pattern provides atomicity instead).
- Consumers of stdout streams MUST check for `stream_end` before treating the stream as complete.
- A stdout stream with no `stream_end` record MUST be treated as truncated/partial.
- The `type` field distinguishes the sentinel from element records (which use `element_type`, not `type`).

---

## 4. Root Envelope

The root envelope wraps all JSONL records. In streaming mode, the envelope fields are emitted as the **first line** of the JSONL stream (a special `type: "envelope"` record). In single-file JSON mode (`--format json`), the envelope is the root object.

### Envelope Record (first line of JSONL stream)

```json
{
  "type": "envelope",
  "schema_version": "1.1.0",
  "extractor_version": "1.2.0",
  "output_mode": "elements",
  "source": {
    "filename": "annual_report_2025.pdf",
    "sha256": "e3b0c44298fc1c149afb4c8996fb92427ae41e4649b934ca495991b7852b855",
    "media_type": "application/pdf",
    "size_bytes": 4821933,
    "page_count": 48,
    "parser": "pdf:pymupdf4llm",
    "ingested_at": "2026-03-24T09:15:00Z"
  },
  "manifest_ref": "./annual_report_2025_manifest.json"
}
```

### Envelope Fields

| Field | Required | Type | Stability | Description |
|---|---|---|---|---|
| `type` | yes | `"envelope"` | stable | Discriminator for first-line special record |
| `schema_version` | yes | semver string | stable | Protocol version (this document) |
| `extractor_version` | yes | string | stable | Library release tag |
| `output_mode` | yes | `"elements"` \| `"chunks"` | stable | Determines record types in stream |
| `source.filename` | yes | string | stable | Original filename |
| `source.sha256` | yes | hex string | stable | SHA-256 of raw input bytes |
| `source.media_type` | yes | MIME string | stable | Detected MIME type (from magic bytes) |
| `source.size_bytes` | yes | integer | stable | Raw file size |
| `source.page_count` | no | integer \| null | stable | Null for non-paginated sources |
| `source.parser` | yes | `"family:impl"` | stable | Parser used (e.g. `"pdf:pymupdf4llm"`) |
| `source.ingested_at` | yes | ISO 8601 UTC | stable | Extraction timestamp |
| `manifest_ref` | yes | relative path | stable | Path to companion manifest.json |

---

## 5. Base Element Schema

Every element record in the JSONL stream (excluding the envelope and error records) MUST include all required base fields.

### Required Fields (all element types)

| Field | Type | Stability | Description |
|---|---|---|---|
| `id` | string | stable | Content-addressed ID (see §19). Format: `"el_" + 16-char hex` |
| `element_type` | enum string | stable | One value from §6. Colon-namespaced: `"family:subtype"` |
| `text` | string | stable | Plain-text rendering. Always present. For tables: pipe-delimited. For equations: `equation.plain_text` |
| `section_path` | array\<string\> | stable | Ordered heading breadcrumb from root. `[]` when no headings found (never null) |
| `section_path_tier` | integer 1–4 | stable | Quality indicator (1=native heading, 4=fallback). See §7 |
| `sequence_index` | integer | stable | 0-based absolute position in document reading order |
| `extraction_method` | string | stable | `"family:impl"` token. e.g. `"layout:rule-based"`, `"ocr:tesseract-5"` |
| `confidence` | float 0.0–1.0 \| null | stable | Parser confidence. Null if parser cannot produce a meaningful score |
| `schema_version` | string | stable | Repeated from envelope for standalone portability |

### Optional Fields (base)

| Field | Type | Stability | Description |
|---|---|---|---|
| `page` | integer \| null | stable | 1-based page number. Null for non-paginated sources |
| `source_filename` | string | stable | Basename of the source file. Strongly recommended; required in chunks mode. Propagated from envelope. |
| `source_sha256` | hex string | stable | SHA-256 of the source file binary. Strongly recommended; required in chunks mode. Same value as `envelope.source.sha256`. |
| `source_media_type` | MIME string | stable | MIME type of the source file. Strongly recommended; required in chunks mode. |
| `metadata.language` | BCP-47 string | stable | Detected language of this element |
| `metadata.char_count` | integer | stable | `len(text)` |
| `metadata.bbox` | array\<float\>[4] | experimental | `[x0, y0, x1, y1]` in points from top-left origin |
| `metadata.font_size` | float | experimental | Dominant font size in points |
| `metadata.is_bold` | boolean | experimental | |
| `metadata.is_italic` | boolean | experimental | |
| `metadata.parent_id` | string | experimental | ID of logical parent element (e.g., caption's parent figure) |
| `metadata.continuation_of` | string | experimental | ID of element this continues |

> **Note:** `source_filename`, `source_sha256`, and `source_media_type` become **required** on chunk records (see §8). These provenance fields are propagated from the envelope so elements can be used standalone without reading the envelope.

### Minimal Valid Element Example

```json
{
  "id": "el_7f3a9c2b1d4e5f6a",
  "element_type": "text:narrative",
  "text": "The extraction layer is the root cause of most RAG quality failures.",
  "section_path": ["Introduction", "Problem Statement"],
  "section_path_tier": 1,
  "sequence_index": 42,
  "extraction_method": "layout:rule-based",
  "confidence": 0.94,
  "schema_version": "1.1.0",
  "page": 2
}
```

---

## 6. Element Type Reference

The `element_type` enum is the single canonical source of truth. All parsers MUST map to these exact strings. A Python constant `ELEMENT_TYPES` in `src/extractor/schema.py` exports the full set.

### 6.1 Structural Family

Elements that define document hierarchy. Setting heading_level on these is what drives section_path construction.

| Type | Description | Additional Required Fields |
|---|---|---|
| `structural:title` | Document-level title (H1 equivalent) | `heading_level: 1` |
| `structural:subtitle` | Document subtitle — does NOT affect section_path | `heading_level: null` |
| `structural:section_header` | Section heading (H2 equivalent) | `heading_level: integer 1-6` |
| `structural:subsection_header` | Subsection heading (H3–H6) | `heading_level: integer 1-6` |
| `structural:page_header` | Running header (repeated across pages) | — |
| `structural:page_footer` | Running footer | — |
| `structural:page_break` | Explicit page break marker | — |

> **Note:** `structural:subtitle` MUST have `heading_level: null`. It does not push or pop the heading stack and does not affect `section_path`. See §7 for the subtitle stack rule.

Additional fields:
- `heading_level` (required on title/section_header/subsection_header): integer 1–6
- `heading_index` (optional, experimental): explicit numbering string e.g. `"3.2.1"`

```json
{
  "id": "el_a1b2c3d4e5f6a7b8",
  "element_type": "structural:section_header",
  "text": "3.2 Methodology",
  "heading_level": 2,
  "heading_index": "3.2",
  "section_path": ["3 Background"],
  "section_path_tier": 1,
  "sequence_index": 89,
  "extraction_method": "layout:rule-based",
  "confidence": 0.99,
  "schema_version": "1.1.0",
  "page": 7
}
```

### 6.2 Text Family

| Type | Description |
|---|---|
| `text:narrative` | Flowing paragraph body text |
| `text:abstract` | Document abstract (typically at start) |
| `text:caption` | Caption for a figure, table, or equation |
| `text:footnote` | Footnote (anchor in text + note at bottom) |
| `text:admonition` | NOTE / WARNING / TIP / CAUTION / DANGER callout boxes | `admonition.kind`, `admonition.title` |
| `text:sidebar` | Aside / sidebar content distinct from main narrative | — |
| `text:endnote` | Endnote |
| `text:marginalia` | Margin annotations |
| `text:pull_quote` | Highlighted quote pulled from body |
| `text:transcript_segment` | Audio transcript segment with timing and speaker | `transcript.speaker`, `transcript.start_time_s`, `transcript.end_time_s` |
| `text:uncategorized` | Parser could not classify; always `confidence ≤ 0.5` |

`text:admonition` JSON example:

```json
{
  "id": "el_9a2b3c4d5e6f7a8b",
  "element_type": "text:admonition",
  "text": "Warning: Do not run this command in a production environment without first backing up your data.",
  "section_path": ["Deployment", "Database Migration"],
  "section_path_tier": 1,
  "sequence_index": 89,
  "admonition": {
    "kind": "warning",
    "title": "Warning"
  },
  "extraction_method": "html:semantic",
  "confidence": 0.99,
  "schema_version": "1.1.0"
}
```

`admonition.kind` enum: `"note"` | `"warning"` | `"tip"` | `"caution"` | `"important"` | `"danger"` | `"custom"`

Parser mappings:
- HTML: `<div class="note/warning/tip">`, `<aside>`, `<blockquote class="warning">`
- Markdown: `> [!NOTE]`, `> [!WARNING]`, `> [!TIP]` (GitHub Flavored Markdown alerts)
- RST: `.. note::`, `.. warning::`, `.. tip::`
- DOCX: styled paragraphs with "Note" or "Warning" character styles

`text:transcript_segment` JSON example:

```json
{
  "id": "el_1a2b3c4d5e6f7a8b",
  "element_type": "text:transcript_segment",
  "text": "The extraction layer is the root cause of most RAG quality failures.",
  "section_path": ["interview_2026_03_24"],
  "section_path_tier": 4,
  "sequence_index": 12,
  "transcript": {
    "speaker": "Speaker 1",
    "start_time_s": 142.3,
    "end_time_s": 147.8,
    "word_timestamps": [
      {"word": "The", "start": 142.3, "end": 142.5},
      {"word": "extraction", "start": 142.5, "end": 143.1}
    ]
  },
  "extraction_method": "whisperx:large-v3",
  "confidence": 0.94,
  "schema_version": "1.1.0"
}
```

`word_timestamps` is optional (experimental). `transcript` is stable.

Audio elements always use `section_path = ["<source_filename_stem>"]` and `section_path_tier = 4`.

### 6.3 List Family

| Type | Description | Additional Required Fields |
|---|---|---|
| `list:item` | Unordered list item | `list_index`, `list_depth`, `list_id` |
| `list:item_ordered` | Ordered/numbered list item | `list_index`, `list_depth`, `list_id` |
| `list:continuation` | List item continuation after break | `list_index`, `list_depth`, `list_id` |

`list_id` field:

| Field | Type | Required | Description |
|---|---|---|---|
| `list_id` | string | Required | Content-addressed ID grouping items from the same list. Format: `"li_" + sha256(source_sha256 + ":list:" + str(first_item_sequence_index))[:16]` |

```json
{
  "id": "el_c3d4e5f6a7b8c9d0",
  "element_type": "list:item",
  "text": "Pre-process document with layout detector",
  "list_index": 0,
  "list_depth": 0,
  "list_id": "li_4f5a6b7c8d9e0a1b",
  "section_path": ["Pipeline", "Preprocessing Steps"],
  "section_path_tier": 1,
  "sequence_index": 61,
  "extraction_method": "layout:rule-based",
  "confidence": 0.88,
  "schema_version": "1.1.0",
  "page": 5
}
```

All items belonging to the same list share the same `list_id`. Nested lists have their own `list_id`. Use `list_id` + `list_depth` + `list_index` to reconstruct list hierarchy.

### 6.4 Table Family

See §9 for full table representation rules.

| Type | Description |
|---|---|
| `table:simple` | Table without merged cells |
| `table:complex` | Table with merged cells or multi-row headers |
| `table:continuation` | Second+ segment of a table split across pages |

### 6.5 Code Family

| Type | Description | Additional Fields |
|---|---|---|
| `code:block` | Block-level code (fenced or indented) | `code.language`, `code.language_confidence`, `code.line_count` |
| `code:inline` | Inline code span | — |

```json
{
  "id": "el_c0d1e2f3a4b5c6d7",
  "element_type": "code:block",
  "text": "def extract(path: str) -> list[Element]:\n    return parse(load(path))",
  "section_path": ["Implementation", "Core API"],
  "section_path_tier": 1,
  "sequence_index": 250,
  "extraction_method": "layout:rule-based",
  "confidence": 0.95,
  "schema_version": "1.1.0",
  "page": 15,
  "code": {
    "language": "python",
    "language_confidence": 0.97,
    "line_count": 2,
    "is_executable": false
  }
}
```

`code:block` elements are **atomic** — never split across chunks.

### 6.6 Media Family

| Type | Description |
|---|---|
| `media:image` | Standalone image without caption |
| `media:figure` | Image + caption as an atomic compound unit |
| `media:diagram` | Vector diagram (SVG, Visio) |
| `media:chart` | Data chart (bar, line, pie) |

`media:image` JSON example:

```json
{
  "id": "el_2b3c4d5e6f7a8b9c",
  "element_type": "media:image",
  "text": "",
  "section_path": ["Appendix", "Screenshots"],
  "section_path_tier": 1,
  "sequence_index": 204,
  "image": {
    "image_ref": "images/screenshot_001.png",
    "image_sha256": "a3f9b2c1d4e5...",
    "format": "png",
    "width_px": 1920,
    "height_px": 1080,
    "alt_text": null,
    "ocr_text": null,
    "storage": "ref"
  },
  "extraction_method": "layout:rule-based",
  "confidence": 0.88,
  "schema_version": "1.1.0"
}
```

Required `image` fields: `image_ref` (string | null), `image_sha256` (string | null), `format` (string | null), `width_px` (integer | null), `height_px` (integer | null), `alt_text` (string | null), `ocr_text` (string | null), `storage` ("ref" | "inline" | "skip").

`media:figure` carries a `figure` object:
```json
"figure": {
  "figure_number": 3,
  "caption": "System architecture overview.",
  "caption_id": "el_e2f3a4b5c6d7e8f9",
  "image_ref": "./figures/fig3.png",
  "image_sha256": "4a5b6c7d8e9f...",
  "width_px": 1200,
  "height_px": 800,
  "ocr_text": null,
  "subfigures": [
    {"label": "a", "caption": "Training loss"},
    {"label": "b", "caption": "Validation loss"}
  ]
}
```

`subfigures` (experimental) — array of `{label, caption}` for multi-panel figures.

### 6.7 Scientific Family

See §10 for full scientific element representation.

| Type | Description |
|---|---|
| `scientific:equation_display` | Block/display equation |
| `scientific:equation_inline` | Inline equation within prose |
| `scientific:citation` | In-text citation marker |
| `scientific:reference_entry` | Bibliography/reference list entry |
| `scientific:theorem` | Theorem, lemma, corollary, proposition, definition, axiom |
| `scientific:proof` | Proof block |
| `scientific:algorithm` | Algorithm pseudocode block |

### 6.8 Form Family

| Type | Description | Additional Fields |
|---|---|---|
| `form:key_value` | Key-value pair (e.g., "Invoice #: 12345") | `kv.key`, `kv.value`, `kv.value_type` |
| `form:checkbox` | Checkbox field | `form_field.checked`, `form_field.label` |
| `form:field` | Form input field | `form_field.label`, `form_field.value` |

### 6.9 Meta Family

Document-level metadata elements (typically extracted from headers/footers or PDF metadata).

| Type | Description |
|---|---|
| `meta:document_title` | Title from document metadata |
| `meta:author` | Author name |
| `meta:date` | Publication/creation date |
| `meta:address` | Mailing or institutional address |
| `meta:email` | Email address |
| `meta:url` | URL |
| `meta:page_number` | Page number marker in text |

### 6.10 Composite Family

`composite:chunk` — used exclusively in `chunks` output mode. See §8.

**Invariant:** `composite:chunk` MUST NOT appear in `elements` mode output. Any validator MUST reject it.

---

## 7. Section Path Construction

`section_path` is an ordered array of strings representing the heading hierarchy from document root to the heading immediately containing the element.

```json
"section_path": ["Introduction", "Related Work", "Neural Approaches"]
```

This means the element lives under: Introduction → Related Work → Neural Approaches.

### Tier 1 — Explicit Structural Headings (Gold)
**Condition:** Parser produces `structural:*` heading elements with `heading_level`.

**Algorithm:**
```python
stack: list[HeadingFrame] = []

def push_heading(frame: HeadingFrame) -> None:
    # Pop all frames at same or deeper level
    while stack and stack[-1].level >= frame.level:
        stack.pop()
    stack.append(frame)

def current_path() -> list[str]:
    return [f.text for f in stack]
```

**Reliability:** ~95%+ on DOCX, HTML, EPUB, well-formed PDFs.

### Tier 2 — Font/Style Heuristic (Silver)
**Condition:** No semantic heading elements but font-size/bold signals available.

**Algorithm:** Cluster text elements by font size. Top 2–3 clusters (appearing sparsely, < 5% of elements) treated as heading surrogates. Apply heading stack.

**Reliability:** ~70–80% on single-column PDFs.

### Tier 3 — Keyword/Numbering Heuristic (Bronze)
**Condition:** No font signals available (e.g., OCR output).

**Algorithm:** Regex detection of:
- Explicit numbering: `^(\d+\.){1,4}\s+\S`
- ALL CAPS short strings: `^[A-Z ]{3,50}$` with char_count < 60
- Common academic keywords: `Abstract|Introduction|Conclusion|References|Appendix`

**Reliability:** ~40–60%.

### Tier 4 — Positional Fallback (Last Resort)
**Condition:** No headings detectable by any method.

**Algorithm:** Inherit last detected heading from anywhere in document. If no heading ever found: `section_path = []`.

### Tier Propagation Rules
1. Heading elements carry the tier at which they were detected
2. Non-heading elements inherit the tier of their most recent anchor heading
3. If two adjacent headings have different tiers, the **lower tier number** (higher quality) wins for child elements
4. `section_path_tier` is NEVER null; always 1–4

> **Important:** `section_path_tier` is a detection quality indicator, NOT a depth indicator. Use `len(section_path)` for depth.

### subtitle Elements and the Heading Stack
`structural:subtitle` elements MUST NOT participate in the heading stack (they do not push/pop frames). Set `heading_level: null` on subtitle elements. Subtitles do not affect `section_path`.

### section_path on Headings Themselves
A heading element's `section_path` reflects the path BEFORE itself is pushed. It represents the context the heading opens into, not a self-referential loop.

```
Given: [Chapter 1 H1] → [Section 1.1 H2] → [paragraph]

Chapter 1:    section_path = []              (nothing above it)
Section 1.1:  section_path = ["Chapter 1"]  (parent context)
paragraph:    section_path = ["Chapter 1", "Section 1.1"]
```

---

## 8. Chunk Schema

Chunks are only emitted in `--mode chunks`. Each chunk is a `composite:chunk` record.

```json
{
  "id": "ch_8f3a2b1c9d4e5f6a",
  "element_type": "composite:chunk",
  "text": "3.2 Methodology\n\nOur approach combines transformer-based layout analysis...",
  "section_path": ["Methodology", "System Architecture"],
  "section_path_tier": 1,
  "sequence_index": 89,
  "extraction_method": "chunker:section-aware-v1",
  "confidence": 0.91,
  "schema_version": "1.1.0",
  "page": 8,
  "token_count": 487,
  "char_count": 2341,
  "page_start": 8,
  "page_end": 8,
  "sequence_index_start": 89,
  "sequence_index_end": 105,
  "source_filename": "system_design_v2.pdf",
  "source_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "source_media_type": "application/pdf",
  "chunk_metadata": {
    "element_ids": ["el_a1b2", "el_c3d4", "el_e5f6"],
    "element_types": ["structural:section_header", "text:narrative", "text:narrative"],
    "has_table": false,
    "has_equation": false,
    "has_code": false,
    "has_figure": false,
    "window_context": {
      "prev_chunk_id": "ch_7e2a1b0c8d3f4e5a",
      "next_chunk_id": "ch_9a4b3c2d1e6f7a8b",
      "prev_section_path": ["Methodology", "Related Work"],
      "next_section_path": ["Methodology", "Evaluation"]
    },
    "chunking_strategy": "section-aware-v1",
    "max_tokens": 512,
    "overlap_tokens": 0,
    "overlap_source": "none"
  }
}
```

### Chunk Required Fields

| Field | Type | Description |
|---|---|---|
| `token_count` | integer | Token count using configured tokenizer |
| `page_start` | integer \| null | First page of contributing elements |
| `page_end` | integer \| null | Last page of contributing elements |
| `sequence_index_start` | integer | sequence_index of first contributing element |
| `sequence_index_end` | integer | sequence_index of last contributing element |
| `chunk_metadata.element_ids` | array\<string\> | IDs of contributing elements |
| `chunk_metadata.element_types` | array\<string\> | Types of contributing elements (parallel to element_ids) |
| `chunk_metadata.chunking_strategy` | string | Chunking algorithm used |
| `chunk_metadata.max_tokens` | integer | Configured max tokens for this chunk |
| `source_filename` | string | Basename of the source file (required on chunks) |
| `source_sha256` | hex string | SHA-256 of the source file binary (required on chunks; same value as `envelope.source.sha256`) |
| `source_media_type` | MIME string | MIME type of the source file (required on chunks) |

> These provenance fields enable direct vector store upsert without envelope state, document-level deletion by `source_sha256`, and deduplication across re-extraction runs.

### Entity Fields on Chunks (schema v1.1.0)

When NER is enabled, chunks carry two additional fields:

| Field | Type | Description |
|---|---|---|
| `entities` | array\<EntityAnnotation\> \| absent | Deduplicated rollup of named entity annotations from all contributing elements. Absent when NER did not run. See §21 for the full `EntityAnnotation` schema. |
| `chunk_metadata.entity_types` | array\<string\> | Sorted deduplicated list of `entity_type` values across all chunk entities. Use for fast payload filtering without deserializing `entities`. |

### Chunk Invariants

1. `token_count` MUST NOT exceed `chunk_metadata.max_tokens` (hard, testable invariant)
2. Atomic element types (`table:simple`, `table:complex`, `table:continuation`, `media:figure`, `code:block`, `scientific:equation_display`) MUST NOT share a chunk with any other element
3. Chunks MUST be in ascending `sequence_index_start` order in the stream
4. `chunk_metadata.window_context.prev/next_chunk_id` MAY be null for first/last chunks

---

## 9. Table Representation Rules

### Rule 1: Atomicity
Every table is a single element. Tables are NEVER split into row-per-element.
`table:continuation` exists only when a table physically spans page boundaries.

### Rule 2: Dual Format Mandatory
Both `table.markdown` (GFM) and `table.structured` MUST be present. Parsers MUST NOT emit a table with only one format.

### Rule 3: The text Field
`text` for tables: pipe-delimited plain text for embedding purposes.
`"col1 | col2 | col3\nval1 | val2 | val3"`
NOT authoritative — use `table.markdown` or `table.structured` for structural queries.

### Rule 4: Merged Cells
`table:simple` — no `merged_cells` field.
`table:complex` — `merged_cells` required; `headers` becomes `array<array<string>>` for multi-row headers.

### Rule 5: Split Tables
First segment: carries headers + rows that fit. No `continuation_of`.
Continuation: `metadata.continuation_of` = first segment's ID; `has_header_row: false`; `headers: null`.
Consumers: merge rows from all segments in `sequence_index` order.

### Rule 6: Caption Linking
Caption emitted as both: `text:caption` element (with `metadata.parent_id`) AND inside `table.caption`. Both required; redundancy is intentional.

### Rule 7: Confidence Threshold
`confidence < 0.70` → W_LOW_CONFIDENCE_TABLE warning in manifest
`confidence < 0.50` → additionally sets `metadata.requires_review: true`

### Full Table:Simple Example

```json
{
  "id": "el_f4a3b2c1d0e9f8a7",
  "element_type": "table:simple",
  "text": "Model | Precision | Recall | F1\nLayoutLM-v3 | 0.923 | 0.911 | 0.917\nDocFormer | 0.908 | 0.897 | 0.902",
  "section_path": ["Results", "Benchmark Comparison"],
  "section_path_tier": 1,
  "sequence_index": 200,
  "extraction_method": "ml:table-transformer",
  "confidence": 0.87,
  "schema_version": "1.1.0",
  "page": 12,
  "table": {
    "format": "dual",
    "row_count": 2,
    "col_count": 4,
    "has_header_row": true,
    "has_header_col": false,
    "markdown": "| Model | Precision | Recall | F1 |\n|---|---|---|---|\n| LayoutLM-v3 | 0.923 | 0.911 | 0.917 |\n| DocFormer | 0.908 | 0.897 | 0.902 |",
    "structured": {
      "headers": ["Model", "Precision", "Recall", "F1"],
      "rows": [
        ["LayoutLM-v3", "0.923", "0.911", "0.917"],
        ["DocFormer", "0.908", "0.897", "0.902"]
      ],
      "data_types": ["string", "float", "float", "float"]
    },
    "caption": "Table 2: Benchmark comparison on DocBank test set.",
    "caption_id": "el_e5f6a7b8c9d0e1f2",
    "footnotes": []
  }
}
```

### Full Table:Complex Example (merged cells)

```json
{
  "id": "el_a8b7c6d5e4f3a2b1",
  "element_type": "table:complex",
  "text": "Revenue | APAC | 12.4M | EMEA | 9.1M",
  "section_path": ["Financials", "Regional Breakdown"],
  "section_path_tier": 1,
  "sequence_index": 415,
  "extraction_method": "ml:table-transformer",
  "confidence": 0.73,
  "schema_version": "1.1.0",
  "page": 22,
  "table": {
    "format": "dual",
    "row_count": 4,
    "col_count": 5,
    "has_header_row": true,
    "has_header_col": true,
    "merged_cells": [
      {"row_start": 0, "row_end": 0, "col_start": 1, "col_end": 2, "text": "Revenue"},
      {"row_start": 0, "row_end": 0, "col_start": 3, "col_end": 4, "text": "Costs"}
    ],
    "markdown": "| | Revenue | | Costs | |\n|---|---|---|---|---|\n| Region | APAC | EMEA | APAC | EMEA |\n| Q1 | 12.4M | 9.1M | 3.2M | 2.8M |",
    "structured": {
      "headers": [["", "Revenue", "Revenue", "Costs", "Costs"], ["Region", "APAC", "EMEA", "APAC", "EMEA"]],
      "rows": [["Q1", "12.4M", "9.1M", "3.2M", "2.8M"]],
      "data_types": ["string", "currency", "currency", "currency", "currency"]
    },
    "caption": null,
    "caption_id": null,
    "footnotes": ["All figures in USD millions."]
  },
  "metadata": {
    "requires_review": true
  }
}
```

---

## 10. Scientific Element Representation

### Equations

`scientific:equation_display` (block) and `scientific:equation_inline` (inline).

```json
{
  "id": "el_e3f4a5b6c7d8e9f0",
  "element_type": "scientific:equation_display",
  "text": "E = mc^2",
  "section_path": ["Theoretical Framework"],
  "section_path_tier": 1,
  "sequence_index": 55,
  "extraction_method": "ml:nougat-0.1.0",
  "confidence": 0.91,
  "schema_version": "1.1.0",
  "page": 4,
  "equation": {
    "latex": "E = mc^{2}",
    "plain_text": "E = m * c^2",
    "mathml": "<math>...</math>",
    "label": "eq:mass-energy",
    "equation_number": "1",
    "is_numbered": true,
    "inline": false
  }
}
```

| Equation Field | Stability | Description |
|---|---|---|
| `equation.latex` | stable | LaTeX source. Primary representation |
| `equation.plain_text` | stable | Human-readable fallback for embedding |
| `equation.inline` | stable | `true` for inline, `false` for display |
| `equation.is_numbered` | stable | Whether equation carries a number |
| `equation.equation_number` | stable | Number string e.g. `"(3.1)"` |
| `equation.label` | stable | LaTeX label e.g. `"eq:loss"` |
| `equation.mathml` | experimental | MathML representation |

For `scientific:equation_inline`, additionally:
- `equation.host_element_id` (experimental): ID of containing `text:narrative` element

### Citations

Two-element system: `scientific:citation` (in-text marker) ↔ `scientific:reference_entry` (bibliography).

```json
{
  "id": "el_a5b6c7d8e9f0a1b2",
  "element_type": "scientific:citation",
  "text": "[12]",
  "section_path": ["Introduction"],
  "section_path_tier": 1,
  "sequence_index": 22,
  "extraction_method": "regex:citation-pattern",
  "confidence": 0.99,
  "schema_version": "1.1.0",
  "page": 2,
  "citation": {
    "style": "numeric",
    "key": "12",
    "reference_entry_id": "el_b6c7d8e9f0a1b2c3",
    "host_element_id": "el_d5e6f7a8b9c0d1e2"
  }
}
```

```json
{
  "id": "el_b6c7d8e9f0a1b2c3",
  "element_type": "scientific:reference_entry",
  "text": "Smith, J. & Doe, A. (2024). Document extraction at scale. EMNLP 2024.",
  "section_path": ["References"],
  "section_path_tier": 1,
  "sequence_index": 620,
  "extraction_method": "layout:rule-based",
  "confidence": 0.94,
  "schema_version": "1.1.0",
  "page": 18,
  "reference": {
    "key": "12",
    "style": "numeric",
    "raw": "Smith, J. & Doe, A. (2024). Document extraction at scale. EMNLP 2024.",
    "parsed": {
      "authors": ["Smith, J.", "Doe, A."],
      "title": "Document extraction at scale",
      "venue": "EMNLP 2024",
      "year": 2024,
      "doi": null,
      "url": null
    }
  }
}
```

`reference.parsed` is **experimental**. `reference.raw` is **stable**.

Citation styles: `"numeric"` | `"author-year"` | `"footnote"` | `"unknown"`

### Theorems

```json
{
  "id": "el_c7d8e9f0a1b2c3d4",
  "element_type": "scientific:theorem",
  "text": "For any compact operator T on a Hilbert space H, the spectrum is at most countable.",
  "section_path": ["Mathematical Foundations", "Spectral Theory"],
  "section_path_tier": 1,
  "sequence_index": 80,
  "extraction_method": "layout:rule-based",
  "confidence": 0.86,
  "schema_version": "1.1.0",
  "page": 6,
  "theorem": {
    "theorem_type": "theorem",
    "label": "thm:compact-spectrum",
    "number": "2.1",
    "proof_id": "el_d8e9f0a1b2c3d4e5"
  }
}
```

`theorem_type` values: `"theorem"` | `"lemma"` | `"corollary"` | `"proposition"` | `"definition"` | `"axiom"` | `"conjecture"`

---

## 11. Manifest Schema

One `{basename}_manifest.json` per extraction run.

```json
{
  "$schema": "https://coresdk.io/schemas/v1/manifest.json",
  "schema_version": "1.1.0",
  "manifest_id": "mf_3c2b1a0f9e8d7c6b",
  "status": "complete",
  "created_at": "2026-03-24T09:15:42Z",
  "extractor_version": "1.2.0",
  "source": {
    "filename": "annual_report_2025.pdf",
    "sha256": "e3b0c44298fc1c149afb...",
    "media_type": "application/pdf",
    "size_bytes": 4821933,
    "page_count": 48
  },
  "run_config": {
    "output_mode": "elements",
    "parser": "pdf:pymupdf4llm",
    "chunking_strategy": null,
    "max_chunk_tokens": null,
    "overlap_tokens": null,
    "extract_tables": true,
    "extract_images": true,
    "extract_equations": true,
    "ocr_enabled": false,
    "language_hint": "en",
    "include_types": null,
    "exclude_types": null
  },
  "parser_chain": [
    {"step": 1, "name": "quarantine", "impl": "extractor:internal", "duration_ms": 32},
    {"step": 2, "name": "classifier", "impl": "extractor:internal", "duration_ms": 8},
    {"step": 3, "name": "pdf_parser", "impl": "pymupdf4llm:1.27.2", "duration_ms": 1240},
    {"step": 4, "name": "heading_stack", "impl": "extractor:internal", "duration_ms": 45},
    {"step": 5, "name": "output_writer", "impl": "extractor:internal", "duration_ms": 210}
  ],
  "toc": [
    {"level": 1, "title": "Executive Summary", "page": 1, "sequence_index": 0},
    {"level": 2, "title": "Financial Highlights", "page": 3, "sequence_index": 42},
    {"level": 1, "title": "Risk Factors", "page": 12, "sequence_index": 189}
  ],
  "stats": {
    "total_elements": 312,
    "total_chunks": 47,
    "total_tokens": 18420,
    "elements_by_type": {
      "text:narrative": 198,
      "structural:section_header": 24,
      "table:simple": 8,
      "table:complex": 2,
      "code:block": 12,
      "list:item": 45,
      "media:figure": 7,
      "scientific:equation_display": 4,
      "text:footnote": 12
    },
    "confidence": {
      "mean": 0.91,
      "p10": 0.72,
      "p90": 0.98,
      "below_0_5": 2,
      "below_0_7": 8
    },
    "section_path_tiers": {
      "1": 298,
      "2": 10,
      "3": 4,
      "4": 0
    },
    "tables": {
      "total": 10,
      "with_header": 9,
      "continuation_segments": 1
    },
    "chunks": {
      "total": 47,
      "min_tokens": 38,
      "max_tokens": 512,
      "mean_tokens": 391.9,
      "atomic_solo": 12
    }
  },
  "warnings": [
    {
      "code": "W_LOW_CONFIDENCE_TABLE",
      "element_id": "el_a8b7c6d5e4f3a2b1",
      "page": 22,
      "message": "Table confidence 0.73 below threshold 0.80"
    }
  ],
  "errors": [],
  "output_files": {
    "elements": "./annual_report_2025_elements.jsonl",
    "manifest": "./annual_report_2025_manifest.json"
  }
}
```

> **Note:** `stats` is a required field. It MUST be populated post-processing after the full element stream is written. A `stats: {}` empty object is a protocol violation.

### Manifest Status Values
- `"complete"` — all elements written, no fatal errors
- `"partial"` — some elements written; one or more parser failures
- `"failed"` — quarantine or fatal error; no elements written

### Manifest TOC
`toc` is an array of detected top-level structural headings. Extracted from heading elements in the stream. Provides a navigable document map without parsing the full JSONL.

### Warning Codes
| Code | Description |
|---|---|
| `W_LOW_CONFIDENCE_TABLE` | Table confidence below 0.80 |
| `W_SECTION_PATH_TIER_FALLBACK` | Section path derived from Tier 3 or 4 for a page range |
| `W_GROBID_UNAVAILABLE` | GROBID service not reachable; fell back to MinerU-only |
| `W_EXT_MISMATCH` | File extension doesn't match detected MIME type |
| `W_HEADING_DETECTION_HEURISTIC` | Headings inferred from font size, not native markup |
| `W_TRUNCATED_ELEMENT` | An element's text was truncated due to max_element_tokens |

### Error Codes
| Code | Description |
|---|---|
| `E_ENCRYPTED` | File is password-protected or encrypted |
| `E_MAGIC_MISMATCH` | Magic bytes don't match declared/detected MIME |
| `E_CORRUPT_ZIP` | OOXML container ZIP is malformed |
| `E_TOO_LARGE` | File exceeds configured size limit |
| `E_ZIP_BOMB` | Compression ratio suggests zip bomb — applies to generic ZIP containers and OOXML files (.docx, .xlsx, .pptx, .odt, .ods, .odp); checked via stdlib zipfile before any OOXML parser is instantiated |
| `E_ENCODING_UNKNOWN` | Text encoding could not be determined |
| `E_EMPTY` | Zero-byte file |
| `E_PARSER_CRASH` | Parser raised an unhandled exception |
| `E_PARSER_TIMEOUT` | Parser exceeded time limit |
| `E_UNSUPPORTED_FORMAT` | No parser available for this format |

---

## 12. Stats Schema

Lives at `manifest.stats`. Also available from Python SDK as `result.stats`.

```json
{
  "total_elements": 847,
  "total_tokens_est": 38420,
  "total_chars": 189330,
  "page_count": 48,
  "processing_time_ms": 5105,
  "elements_by_type": {
    "structural:title": 1,
    "structural:section_header": 28,
    "structural:subsection_header": 47,
    "text:narrative": 312,
    "text:caption": 18,
    "text:footnote": 22,
    "list:item": 89,
    "table:simple": 7,
    "table:complex": 3,
    "code:block": 5,
    "media:figure": 18,
    "scientific:equation_display": 14,
    "scientific:equation_inline": 31,
    "scientific:citation": 124,
    "scientific:reference_entry": 42,
    "text:uncategorized": 7
  },
  "confidence": {
    "mean": 0.891,
    "median": 0.924,
    "min": 0.43,
    "p10": 0.71,
    "p90": 0.98,
    "below_0_5": 3,
    "below_0_7": 18
  },
  "section_path_tiers": {
    "tier_1": 789,
    "tier_2": 31,
    "tier_3": 22,
    "tier_4": 5
  },
  "tables": {
    "total": 12,
    "simple": 7,
    "complex": 3,
    "continuation_pairs": 2,
    "avg_rows": 6.2,
    "avg_cols": 4.8
  },
  "suppressed_by_filter": 0,
  "chunks": null
}
```

`suppressed_by_filter` — count of elements that matched an active `--exclude-types` or were absent from an active `--include-types` filter. These elements are parsed and counted but not written to the output stream. `sequence_index` gaps caused by suppression are intentional (see §14).

When `output_mode = "chunks"`, `chunks` replaces raw element counts:
```json
"chunks": {
  "total": 94,
  "avg_tokens": 408,
  "max_tokens": 487,
  "min_tokens": 41,
  "with_table": 8,
  "with_equation": 12,
  "with_code": 4
}
```

---

## 13. Stability Contract

### Stable Fields (v1.0.0+)
Breaking changes require MAJOR version bump:

**On every element (base):**
`id`, `element_type`, `text`, `section_path`, `section_path_tier`, `sequence_index`, `extraction_method`, `confidence`, `schema_version`, `page`, `metadata.language`, `metadata.char_count`

**Element-type specific:**
- All `table.*` fields on `table:simple` (markdown, structured.headers, structured.rows, row_count, col_count)
- `equation.latex`, `equation.plain_text`, `equation.inline`, `equation.is_numbered`
- `citation.key`, `citation.style`, `citation.reference_entry_id`
- `reference.key`, `reference.raw`
- `chunk_metadata.element_ids`, `chunk_metadata.element_types`, `chunk_metadata.max_tokens`, `token_count`
- Full element type enum (all non-`x:` types)

**Manifest:** `source`, `run_config`, `parser_chain`, `stats`, `warnings`, `errors`, `toc`, `output_files`, `status`

**Stdout stream sentinel:** `stream_end.type`

### Experimental Fields
May change in minor versions with deprecation notice:
- `metadata.bbox`, `metadata.font_size`, `metadata.is_bold`, `metadata.is_italic`
- `table.structured.data_types`, `table.merged_cells`
- `equation.mathml`
- `reference.parsed.*`
- `figure.ocr_text`, `figure.subfigures`
- `chunk_metadata.window_context`
- `code.is_executable`
- `theorem.proof_id`
- All `x:` prefixed element types
- `heading_index`
- `entities`, `entity_type`

### Deprecated
None at v1.0.0.

---

## 14. CLI Contract

```
USAGE
  extractor run <target> [OPTIONS]
  extractor validate <jsonl-file>
  extractor info <target>

TARGETS
  <file>        Local file path
  <directory>   Process all supported files recursively
  <url>         HTTP/HTTPS URL (downloads + extracts)
  -             Read from stdin

OPTIONS
  --mode        elements | chunks         (default: elements)
  --out         Output directory or file  (default: stdout for JSONL)
  --format      jsonl | json              (default: jsonl)
  --chunk-size  Integer token count       (default: 512)
  --overlap     Integer token count       (default: 0, disabled). Number of overlap tokens to prepend from previous chunk. Overlap does not cross section boundaries.
  --strategy    fast | accurate | ocr     (default: fast)
  --include-types TEXT  Comma-separated element types to emit. Supports family wildcards
                        (e.g. "text:*,table:*"). Default: all types emitted.
  --exclude-types TEXT  Comma-separated element types to suppress from output.
                        Supports family wildcards (e.g. "structural:page_header,structural:page_footer").
                        Default: none suppressed.
  --no-manifest Suppress manifest.json    (disables reproducibility guarantees)
  --quiet       Suppress stderr progress
  --json-errors Emit errors as JSONL records (for programmatic consumers)
  --version     Show extractor and schema versions

TYPE FILTER RULES
  --include-types and --exclude-types are mutually exclusive.
  If both are provided → exit code 6 (configuration error).
  Suppressed elements are counted in manifest.stats.suppressed_by_filter.
  sequence_index is NOT renumbered when elements are suppressed — gaps are
  intentional to preserve ID stability across re-extraction runs.
  The active filter is recorded in run_config.include_types / run_config.exclude_types
  in the envelope and manifest.

STDOUT / STDERR CONTRACT
  stdout:  JSONL stream ONLY (elements or chunks + envelope record)
  stderr:  progress bars, warnings, human-readable diagnostics
  NEVER:  progress text or log messages on stdout
  TTY detection: if stdout is a terminal → pretty-print with color
  Pipe detection: if stdout is piped → raw JSONL, no color

EXAMPLES
  extractor run report.pdf | jq '.section_path'
  extractor run report.pdf --mode chunks --chunk-size 1024 --out ./chunks/
  extractor run ./docs/ --mode elements --format jsonl
  extractor validate output_elements.jsonl
  extractor info report.pdf    # shows detected format, page count, subtype

  # Skip running headers/footers and page numbers
  extractor run report.pdf --exclude-types "structural:page_header,structural:page_footer,structural:page_number"

  # Only extract tables and code blocks
  extractor run docs/ --include-types "table:*,code:block" --mode elements

  # Skip all scientific citation noise for non-scientific RAG
  extractor run paper.pdf --exclude-types "scientific:citation,scientific:reference_entry"
```

---

## 15. Exit Codes

| Code | Meaning |
|---|---|
| 0 | Clean success — all output written, `manifest.status = "complete"` |
| 1 | Partial success — some elements written; one or more documents failed |
| 2 | Quarantine failure — file rejected before parsing; no output |
| 3 | Unsupported format — no parser available |
| 4 | Parser crash — mid-parse exception; partial `.jsonl.tmp` preserved |
| 5 | Output error — cannot write to output path |
| 6 | Configuration error — invalid CLI flags or missing required extra |

---

## 16. Error Records

When `--json-errors` is set, errors are emitted as JSONL records to stdout alongside normal output:

```json
{
  "type": "error",
  "code": "E_PARSER_CRASH",
  "element_index": 142,
  "page": 22,
  "message": "Table extraction failed: unexpected row span",
  "recoverable": true,
  "schema_version": "1.1.0"
}
```

Non-recoverable errors (`recoverable: false`) halt the stream after this record.

---

## 17. Parser Routing Table

| Format | MIME / Extension | Primary Parser | Fallback Chain |
|---|---|---|---|
| PDF (digital) | `application/pdf` + digital | pymupdf4llm | → marker-pdf |
| PDF (scientific) | `application/pdf` + scientific | GROBID + MinerU | → MinerU-only |
| PDF (scanned) | `application/pdf` + scanned | MinerU (OCR) | → marker-pdf (OCR) |
| DOCX | OOXML wordprocessingml | python-docx | → pypandoc |
| XLSX | OOXML spreadsheetml | openpyxl read_only | → calamine |
| PPTX | OOXML presentationml | python-pptx | → pypandoc |
| HTML / MHTML | `text/html` | trafilatura + lxml | → html2text |
| EPUB | `application/epub+zip` | ebooklib + bs4 | — |
| Markdown | `text/markdown` / `.md` | mistletoe | → raw text |
| RST | `text/x-rst` | docutils | → pypandoc |
| LaTeX | `text/x-tex` | TexSoup + latexmlpy | → pypandoc |
| arXiv bundle | `.tar.gz` + TeX | TexSoup AST | → pypandoc |
| Plain text | `text/plain` | readline iterator | — |
| RTF | `application/rtf` | striprtf | → pypandoc |
| CSV / TSV | `text/csv` | csv stdlib | — |
| JSON / JSONL | `application/json` | ijson streaming | — |
| XML | `text/xml` | lxml iterparse | — |
| Image | `image/*` | pytesseract / easyocr [ocr] | — |
| Audio | `audio/*` | whisperx [audio] | — |

PDF subtype classification (runs during quarantine):
- `digital`: has text layer, no reference section heuristics
- `scientific`: has text layer + detected reference section
- `scanned`: image-to-text area ratio > 0.85

---

## 18. Chunker Algorithm

### Section-Aware Chunker v1

Default parameters: `max_tokens=512`, `overlap=0`, `hard_ceiling=2500`

Overlap is **disabled by default** (0 tokens). Enable via `--overlap N`.

```
ATOMIC_TYPES = {table:simple, table:complex, table:continuation, media:image, media:figure, code:block, scientific:equation_display}

For each element in reading order:
  elem_tokens = count_tokens(element.text)

  CASE: element is ATOMIC and elem_tokens > max_tokens:
    flush current buffer as chunk (if non-empty)
    emit atomic element as solo chunk (regardless of size)
    reset overlap tail to []
    continue

  CASE: element is heading AND buffer is non-empty:
    flush current buffer as chunk
    reset overlap_tail = []          # section-boundary flush: NO overlap carried
    start new buffer with heading element

  CASE: adding element would exceed max_tokens:
    if element is ATOMIC:
      flush buffer; start new buffer with atomic element
    else:
      flush buffer; prepend overlap tail; add element to new buffer
      # token-overflow flush: overlap IS carried forward

  DEFAULT:
    append element to buffer

  On stream end: flush remaining buffer as final chunk
```

### Overlap Tail Computation
Walk buffer from end; collect whole elements until `overlap_tokens` budget accumulated. These are prepended to the next chunk. Only token-overflow flushes propagate the overlap tail.

**RULE: Overlap tail MUST NOT carry elements across a section boundary flush.**
- If the flush was triggered by a new heading element: reset overlap_tail = []
- If the flush was triggered by token budget exceeded: carry overlap_tail normally

Only token-overflow flushes propagate overlap. Section-boundary flushes always start clean.

**SENTENCE-LEVEL FALLBACK:** If no whole element fits within overlap_tokens budget, take the last N sentences of the last text element (split on sentence boundaries: `. ` / `? ` / `! ` followed by uppercase) until the token budget is met. Record in chunk_metadata: `"overlap_source": "sentence_split"`.

### Atomic Element Rule
Atomic elements are **always solo chunks**. They never share a chunk with text, other atomic elements, or anything else. The solo chunk's `chunk_metadata.element_types[0]` records the original type.

Only display equations are atomic. Inline equations (`scientific:equation_inline`) are NOT atomic and flow with surrounding text.

### Context Cliff Protection
`hard_ceiling=2500` — no chunk may exceed 2500 tokens regardless of configuration. This is not a soft warning; the chunker enforces it by splitting at natural boundaries (sentence, list item) before reaching the ceiling.

---

## 19. ID Derivation

### Element ID
```python
import hashlib

def element_id(text: str, page: int | None, sequence_index: int, source_sha256: str) -> str:
    raw = f"{source_sha256}:{page}:{sequence_index}:{text}"
    digest = hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16]
    return f"el_{digest}"
```

### Chunk ID
```python
def chunk_id(sequence_index_start: int, sequence_index_end: int, text: str, source_sha256: str) -> str:
    raw = f"{source_sha256}:{sequence_index_start}:{sequence_index_end}:{text}"
    digest = hashlib.sha256(raw.encode("utf-8")).hexdigest()[:10]
    return f"ch_{digest}"
```

### Manifest ID
```python
def manifest_id(source_sha256: str, created_at: str) -> str:
    raw = f"{source_sha256}:{created_at}"
    digest = hashlib.sha256(raw.encode("utf-8")).hexdigest()[:10]
    return f"mf_{digest}"
```

### Properties
- **Deterministic**: same input → same ID always
- **Source-namespaced**: `source_sha256` prevents cross-document collisions
- **Compact**: 19-character string (`el_` + 16 hex chars)
- **Stable**: re-running extraction on unchanged file → unchanged IDs

---

## 20. Confidence Scoring

Confidence is a `float` in `[0.0, 1.0]` or `null`.

### Per-Format Baseline Floors
| Format / Method | Confidence Floor |
|---|---|
| DOCX native (python-docx) | 1.00 |
| HTML native (trafilatura) | 0.95 |
| Digital PDF (pymupdf4llm, text layer) | 0.90 |
| Scientific PDF (GROBID+MinerU) | 0.85 |
| LaTeX AST (TexSoup) | 0.95 |
| Audio (whisperx) | 0.80 |
| Scientific PDF (MinerU-only, no GROBID) | 0.75 |
| Scanned PDF (OCR: tesseract) | 0.60 |
| Scanned PDF (OCR: easyocr) | 0.65 |
| Heuristic heading detection | 0.70 (ceiling) |
| VLM fallback | 0.70 |

### Per-Element Signals
Applied on top of format floor:
- **OCR word confidence**: `mean(word_confidences)` from tesseract/easyocr
- **Table completeness**: `1.0 - (empty_cells / total_cells)`
- **Heading detection**: 1.0 native / 0.7 font-heuristic / 0.5 keyword-heuristic
- **Audio segment**: `sigmoid(mean_log_prob)` from whisperx

### Chunk Confidence
`mean([e.confidence for e in chunk.elements if e.confidence is not None])`
If all elements have `null` confidence, chunk confidence is `null`.

### Null Confidence
Permitted when the parser genuinely cannot produce a meaningful score (e.g., a purely structural element like `structural:page_break`). Do not emit `confidence: 0.0` when you mean "unknown" — use `null`.

---

## 21. Entity Extraction (schema v1.1.0)

### Overview

When NER is enabled (`--entities` flag, default on when GLiNER2 available), each element and chunk record carries an `entities` field containing named entity annotations produced by GLiNER2.

### EntityAnnotation Schema

```json
{
  "entity_type": "person",
  "text": "Tim Cook",
  "start": 0,
  "end": 8,
  "score": 0.94,
  "extraction_method": "gliner2",
  "source_element_id": "el_7f3a2b1c"
}
```

| Field | Required | Type | Description |
|---|---|---|---|
| `entity_type` | yes | string | Label from the NER label set |
| `text` | yes | string | Matched surface form |
| `start` | no | integer | Char offset in `element.text` (element-level only) |
| `end` | no | integer | Char offset end, exclusive (element-level only) |
| `score` | yes | float | Model confidence 0.0–1.0 |
| `extraction_method` | yes | string | Always `"gliner2"` |
| `source_element_id` | no | string | Source element ID (chunk rollup only) |

### Absent vs Empty

| Value | Meaning |
|---|---|
| Field absent (serialized with `exclude_none=True`) | NER did not run, or element type was skipped |
| `"entities": []` | NER ran but found no entities above threshold |
| `"entities": [...]` | NER ran and found entities |

**Never** emit `"entities": null` — use field absence.

### Element-Level Annotations

Applied after table continuation reassembly, before chunking. Element types in `NER_SKIP_TYPES` (code blocks, equations, media, page headers/footers, form fields, citations) are skipped.

Default label set: `organization`, `person`, `date`, `location`, `product`, `technology`, `event`, `quantity`.

### Chunk-Level Rollup

`ChunkRecord.entities` is a deduplicated rollup from all contributing elements:
- If **all** elements have `entities=None` → chunk `entities` is absent (NER not run)
- If any element has `entities` set → rollup runs; deduplication by `(entity_type, text.lower())`, highest score wins
- `source_element_id` on each chunk annotation points to the element that contributed it

`ChunkMetadata.entity_types` is the sorted deduplicated list of `entity_type` values — use this for fast payload filtering in vector stores without deserializing the full `entities` array.

### Inference

Local inference only: `fastino/gliner2-base-v1` (205M BERT-based) via `[ner]` extras. No external API required.

Graceful fallback: if `[ner]` extras are not installed, extraction continues without NER and a `W_NER_MODEL_UNAVAILABLE` warning is added to the manifest.

### CLI Flags

| Flag | Default | Description |
|---|---|---|
| `--entities` / `--no-entities` | `--entities` | Enable/disable NER |
| `--entities-model` | `fastino/gliner2-base-v1` | Local model name or path |
| `--entities-types` | 8 default labels | Comma-separated label list |
| `--entities-threshold` | `0.50` | Minimum score to emit an annotation |

### Warning Codes

| Code | Meaning |
|---|---|
| `W_NER_MODEL_UNAVAILABLE` | `extractor[ner]` not installed; NER skipped |
| `W_NER_INFERENCE_FAILED` | Per-batch inference error; affected elements get `entities: []` |
| `W_NER_INPUT_TRUNCATED` | Element text truncated to 1024 chars before NER inference |

### Full Example (chunk with entities)

```json
{
  "id": "ch_8f3a2b1c",
  "element_type": "composite:chunk",
  "text": "Tim Cook joined Apple in 1998 as VP of Operations.",
  "schema_version": "1.1.0",
  "section_path": ["Executive Profiles"],
  "token_count": 12,
  "chunk_metadata": {
    "entity_types": ["date", "organization", "person"]
  },
  "entities": [
    {"entity_type": "person", "text": "Tim Cook", "score": 0.94, "extraction_method": "gliner2", "source_element_id": "el_7f3a"},
    {"entity_type": "organization", "text": "Apple", "score": 0.97, "extraction_method": "gliner2", "source_element_id": "el_7f3a"},
    {"entity_type": "date", "text": "1998", "score": 0.89, "extraction_method": "gliner2", "source_element_id": "el_7f3a"}
  ],
  "source_sha256": "e3b0c44298fc1c149afb",
  "source_filename": "annual_report.pdf"
}
```

---

*End of extractor Protocol Specification v1.1.0*

*Reviewed by expert panel: ZySec AI

*Panel consensus: GO — 82% confidence. Schema layer is the primary differentiator.*
