# GLiNER2 Capabilities Design — extractor v1.2.0

Synthesized from 3-expert, 2-cycle panel: Dr. Priya Anand (architecture), Marcus Webb (implementation),
Dr. Aisha Okonkwo (vector DB consumer perspective).

## Summary

All four GLiNER2 capabilities (entity extraction, classification, relation extraction, structured
extraction) are implemented as optional pipeline steps using the same `fastino/gliner2-base-v1`
model checkpoint — no additional downloads required.

## Architecture: GLiNER2Manager

A shared model singleton in `src/extractor/gliner2_manager.py` ensures the model loads once
regardless of how many steps are active. All steps call `get_model(model_id)`.

```python
from extractor.gliner2_manager import get_model
model = get_model("fastino/gliner2-base-v1")
```

## Pipeline Execution Order

```
parse → normalize → _reassemble_table_continuations
  → [NERStep]                ← pre-chunking: entities on BaseElement
  → [StructuredExtractStep]  ← pre-chunking: structured_fields on targeted element types
  → chunker
  → [ClassifyStep]           ← post-chunking: doc_category on ChunkRecord
  → [RelationStep]           ← post-chunking: relations on ChunkRecord
  → writer / sink
```

## Schema Additions (v1.2.0)

### New models

```python
class CategoryScore(BaseModel):
    label: str
    score: float

class ClassificationResult(BaseModel):
    top_label: str
    top_score: float
    scores: list[CategoryScore] | None = None
    extraction_method: str = "gliner2"

class RelationTriple(BaseModel):
    subject: str
    predicate: str
    object: str
    score: float | None = None
    subject_type: str | None = None
    object_type: str | None = None
    source_text: str | None = None
    extraction_method: str = "gliner2"
```

### BaseElement additions

```python
structured_fields: dict[str, str | int | float | bool | list[str]] | None = None
classification: ClassificationResult | None = None  # element-level (optional)
relations: list[RelationTriple] | None = None        # element-level (optional)
```

### ChunkRecord additions

```python
doc_category: str | None = None            # = classification.top_label (shortcut for filtering)
doc_category_score: float | None = None    # = classification.top_score
classification: ClassificationResult | None = None
relations: list[RelationTriple] | None = None
relation_subjects: list[str] | None = None  # rollup for Pinecone-compatible filtering
relation_objects: list[str] | None = None   # rollup for Pinecone-compatible filtering
```

### Why list[CategoryScore] not dict[str, float]

`dict[str, float]` for category scores is unindexable in every major vector store:
- Qdrant: cannot build field conditions on dynamic keys
- Weaviate: opaque blob, no `where` filter
- Pinecone: silently coerced to string, completely unqueryable

`list[CategoryScore]` with promoted `doc_category: str` and `doc_category_score: float`
unlocks all meaningful retrieval patterns with a single-field equality/range filter.

### Why relation_subjects/objects rollup fields

Pinecone has no array-of-nested-objects filter. Storing `list[str]` rollups alongside
`list[RelationTriple]` gives Pinecone consumers a filterable surface without losing the
full triple data for Qdrant/Weaviate.

## Step 1: NERStep (existing, unchanged)

Already implemented. Uses `batch_extract_entities` with 9 SLIMER-style descriptive labels.
See `src/extractor/ner.py`.

## Step 2: ClassifyStep

**File:** `src/extractor/classify.py`

Classifies each chunk by document category. Uses `batch_extract_entities` with category
label strings — GLiNER2 treats them as entity types and scores the whole text. The label
with highest max-confidence across all spans becomes `top_label`.

**Default labels:**
```python
DEFAULT_CLASSIFY_LABELS = [
    "scientific paper or research article",
    "technical documentation or manual",
    "news article or press release",
    "legal document or contract",
    "financial report or earnings",
    "educational content or textbook",
    "product description or marketing",
    "code documentation or API reference",
    "meeting notes or transcript",
    "general prose or narrative",
]
```

**CLI:** `--classify-as "scientific paper,legal document"`

**Python API:**
```python
from extractor import extract
chunks = list(extract("paper.pdf", mode="chunks", classify_labels=["scientific paper", "legal"]))
print(chunks[0].doc_category)  # "scientific paper"
```

## Step 3: RelationStep

**File:** `src/extractor/relations.py`

Extracts (subject, predicate, object) triples from chunk text. Operates post-chunking
so the full chunk context is available for co-occurrence pairing.

**Strategy:** Run `batch_extract_entities` with merged entity + relation predicate labels.
Pair entity spans with nearest predicate span by character distance. Apply min joint confidence.

**Default relation types (16):**
```
caused by, part of, references, defined by, approved by, supersedes,
employed by, located in, cites, regulates, developed by, published at,
uses, outperforms, founded by, acquired by
```

**CLI:** `--relations --relation-types "cites,developed by"`

**Python API:**
```python
chunks = list(extract("paper.pdf", mode="chunks", extract_relations=True))
for triple in (chunks[0].relations or []):
    print(f"{triple.subject} --[{triple.predicate}]--> {triple.object}")
```

## Step 4: StructuredExtractStep

**File:** `src/extractor/structured_extract.py`

Extracts structured JSON fields from targeted element types using a user-provided schema.
Runs pre-chunking at the element level. Uses `batch_extract_json` if available, falls back
to `batch_extract_entities` with schema keys as label strings.

**Target types (default):** `text:abstract`, `text:narrative`, `text:paragraph`,
`scientific:reference_entry`

**CLI:** `--extract-schema schema.json --extract-on text:abstract,scientific:reference_entry`

**Python API:**
```python
schema = {"title": "string", "authors": ["string"], "year": "integer"}
elements = list(extract("paper.pdf", mode="elements", extract_schema=schema,
                        extract_on=["text:abstract"]))
print(elements[0].structured_fields)
# {"title": "Attention Is All You Need", "authors": ["Vaswani", ...], "year": 2017}
```

**Note on types:** `structured_fields` values are typed as `str | int | float | bool | list[str]`
(no nested dicts). Nested objects from the model are flattened with dot-notation keys at the
enrichment step before writing to the schema field.

## Database Sinks

**Files:** `src/extractor/sinks/`

For millions-of-documents scale, records can be streamed directly to ClickHouse or MongoDB
without buffering to disk. The sink runs alongside JSONL output — both `--out` and `--sink`
can be used simultaneously.

### ClickHouse

```bash
extractor run /data/corpus/ \
  --mode chunks --workers 8 \
  --sink clickhouse --sink-uri localhost:8123 \
  --sink-database extractor --sink-batch 5000
```

Table schema uses `MergeTree`, `ORDER BY (source_sha256, sequence_index)`,
`PARTITION BY source_media_type`. Nested fields (entities, relations, classification)
stored as JSON strings in String columns.

Install: `pip install 'coresdk-extractor[clickhouse]'`

### MongoDB

```bash
extractor run /data/corpus/ \
  --mode chunks --workers 4 \
  --entities --relations \
  --sink mongodb --sink-uri mongodb://localhost:27017 \
  --sink-batch 500
```

Uses `insert_many(ordered=False)` for throughput. Indexes on `source_sha256`,
`element_type`, `(source_filename, sequence_index)` created on connect. `--upsert`
mode available for idempotent re-runs.

Install: `pip install 'coresdk-extractor[mongodb]'`

## GLiNER2 API Compatibility Notes

The `gliner2` package (`pip install gliner2`, model `fastino/gliner2-base-v1`) exposes:

| Method | Status |
|--------|--------|
| `batch_extract_entities(texts, labels, include_confidence, include_spans)` | Confirmed working |
| `batch_classify_text(texts, categories)` | May not exist — use `batch_extract_entities` instead |
| `batch_extract_json(texts, schemas)` | May not exist — use `batch_extract_entities` with keys as labels |
| `batch_extract_relations(texts, relations)` | May not exist — use span-pairing approach |

All steps use `batch_extract_entities` as the universal fallback. This is robust across
gliner2 package versions.

## Full CLI Example (all features)

```bash
extractor run research_papers/ \
  --mode chunks --chunk-size 512 \
  --entities \
  --classify-as "scientific paper,technical doc,legal" \
  --relations \
  --relation-types "cites,developed by,outperforms" \
  --out output/ \
  --sink clickhouse --sink-uri localhost:8123 \
  --workers 4
```
