# NER Quality Improvements — Cycle 1 + 2 Findings

**Status:** Implementation-ready
**Version target:** v1.2.1
**Authors:** Priya Nair (Applied AI), Yuki Tanaka (Label Design), Marcus Webb (Text Preprocessing)

---

## Executive Summary

The current NER implementation (`urchade/gliner_medium-v2.1` with 8 short generic labels) produces three classes of quality failures on NLP/ML research documents:

1. **False positives from underspecified labels.** Labels like `"person"` and `"organization"` match incidental tokens — first-name mentions ("Adam"), metric acronyms ("BLEU"), and architectural terms ("The encoder") — because GLiNER is trained to match anything plausible for a label string.

2. **False negatives on domain-specific entities.** Research paper titles, ML model names, algorithm names, and conference venues are unrecognized because no label covers them.

3. **Insufficient per-label threshold tuning.** A single global threshold cannot simultaneously suppress FPs on `"person"` (needs higher threshold) while capturing true positives on `"research paper title"` (needs lower threshold for partial matches).

The fix is three-pronged: (a) redesign labels to be descriptive and domain-specific, (b) apply per-label confidence thresholds, and (c) add lightweight text preprocessing to filter tokens that structurally cannot be valid named entities. An optional model upgrade to `numind/NuNER_Zero` provides better zero-shot recall on novel label strings at no additional infrastructure cost.

---

## 1. Label Redesign

### 1.1 Current Labels (v1.2.0)

```python
DEFAULT_LABELS = [
    "organization",
    "person",
    "date",
    "location",
    "product",
    "technology",
    "event",
    "quantity",
]
```

These labels are too short and ambiguous. GLiNER/NuNER models condition on the literal label string as a natural-language description — vague labels produce vague boundaries.

### 1.2 Proposed Labels (v1.2.1)

```python
DEFAULT_LABELS: list[str] = [
    # People
    "name of a human researcher, author, or scientist",
    # Institutions
    "named university, research institute, company, or standards body",
    # Geography
    "geopolitical location, country, city, or region",
    # Time
    "specific date, year, or named time period",
    # Publications
    "title of a research paper, book, or technical report",
    # Venues
    "academic conference, workshop, or journal venue",
    # Models / algorithms
    "named machine learning model, neural network architecture, or algorithm",
    # Datasets
    "named dataset or benchmark used for evaluation",
    # Metrics
    "evaluation metric or performance measure",
    # Software / frameworks
    "named software library, framework, or programming language",
]
```

**Design rationale for each FP/FN case:**

| Token | Old behavior | New behavior | Why |
|---|---|---|---|
| `"Adam"` | FP — matches `"person"` | No match | "name of a **human researcher, author, or scientist**" — the model must see full-name context; a bare first name scores below threshold 0.78 |
| `"BLEU"` | FP — matches `"organization"` | Matches `"evaluation metric or performance measure"` | Correct label exists; old label was wrong domain |
| `"The encoder"` | FP — matches anything with article | No match | Preprocessing filter strips leading articles before NER; see Section 3 |
| `"Learning Phrase Representations..."` | FN | Matches `"title of a research paper, book, or technical report"` | Explicit label for paper titles |
| `"Transformer"` | FN | Matches `"named machine learning model, neural network architecture, or algorithm"` | Explicit ML model label |
| `"NeurIPS 2017"` | FN | Matches `"academic conference, workshop, or journal venue"` | Explicit venue label |

### 1.3 Labels Removed and Why

| Removed | Reason |
|---|---|
| `"organization"` | Too broad; replaced by more specific institution label that excludes metric acronyms |
| `"person"` | Too broad; "Adam" false positive is a direct consequence |
| `"product"` | Overlap with software/framework and ML model labels caused double-tagging |
| `"technology"` | Subsumed by ML model + software labels |
| `"event"` | General events not relevant to NLP document extraction; replaced by conference venue |
| `"quantity"` | Produced noisy matches on all numbers; removed pending a dedicated numeric filter |

---

## 2. Per-Label Thresholds

A `DEFAULT_LABEL_THRESHOLDS` dict maps each label string to its confidence cutoff. Labels prone to false positives get higher thresholds; labels where recall matters more get lower thresholds.

```python
DEFAULT_LABEL_THRESHOLDS: dict[str, float] = {
    # High recall acceptable — paper titles are distinctive spans
    "title of a research paper, book, or technical report": 0.55,
    # Venues: "NeurIPS 2017" is unambiguous but the year might lower confidence
    "academic conference, workshop, or journal venue": 0.58,
    # ML models: "Transformer", "BERT", "GPT-4" are high-confidence if labeled correctly
    "named machine learning model, neural network architecture, or algorithm": 0.60,
    # Datasets: "Penn Treebank", "SQuAD" are distinctive
    "named dataset or benchmark used for evaluation": 0.60,
    # Metrics: "BLEU", "ROUGE", "F1" — short tokens need higher threshold to avoid noise
    "evaluation metric or performance measure": 0.65,
    # Software: "PyTorch", "TensorFlow" — well-known, high precision expected
    "named software library, framework, or programming language": 0.62,
    # People: require high threshold to suppress bare first-name FPs
    "name of a human researcher, author, or scientist": 0.78,
    # Institutions: high threshold to suppress acronym FPs like "BLEU" matching here
    "named university, research institute, company, or standards body": 0.72,
    # Dates: fairly reliable but can FP on version numbers
    "specific date, year, or named time period": 0.65,
    # Geography: low FP rate for regions, moderate threshold fine
    "geopolitical location, country, city, or region": 0.62,
}
```

**Global fallback threshold** remains `0.50` for any label not listed in this dict (forward-compatible with user-supplied labels).

### 2.1 Threshold Application Logic

The existing `_effective_threshold()` method currently only raises threshold for structural element types. It must be extended to also perform per-label lookup:

```python
# In NERStep._apply_threshold_to_predictions():
def _label_threshold(self, label: str) -> float:
    """Return per-label threshold, falling back to global."""
    return DEFAULT_LABEL_THRESHOLDS.get(label, self.threshold)
```

When filtering predictions, each prediction's label is looked up independently — a single element can have entities from multiple labels each filtered at their own threshold.

---

## 3. Text Preprocessing and Filters

### 3.1 The `_preprocess_for_ner()` Function

This function runs on each text string before it is passed to the model. It returns the processed string (or `None` to skip NER entirely for this element).

```python
import re

# Tokens that are definitively not named entities regardless of label
_ARTICLE_PREFIX_RE = re.compile(r"^(The|A|An)\s+", re.IGNORECASE)
_STOPWORD_ONLY_RE = re.compile(r"^\s*(the|a|an|this|that|these|those|it|its)\s*$", re.IGNORECASE)

# Patterns that indicate the span is structural, not an entity name
_EQUATION_FRAGMENT_RE = re.compile(r"[=<>≤≥∑∫∂∇±×÷]")
_URL_RE = re.compile(r"https?://|www\.")
_NUMERIC_ONLY_RE = re.compile(r"^\s*[\d\s,.\-+%]+\s*$")

def _preprocess_for_ner(text: str) -> str | None:
    """
    Normalize and filter text before sending to NER.

    Returns:
        Cleaned string ready for NER, or None if this text should be
        skipped entirely (structural fragment, equation, etc.).
    """
    # Skip empty / whitespace-only
    stripped = text.strip()
    if not stripped:
        return None

    # Skip pure numeric strings (version numbers, page numbers, quantities)
    if _NUMERIC_ONLY_RE.match(stripped):
        return None

    # Skip equation fragments
    if _EQUATION_FRAGMENT_RE.search(stripped):
        return None

    # Skip URL fragments
    if _URL_RE.search(stripped):
        return None

    # Normalize whitespace
    cleaned = re.sub(r"\s+", " ", stripped)

    return cleaned
```

**What this fixes:**

- `"The encoder"` — passes through preprocessing as text (we do NOT strip the leading article from the string; instead the article-prefixed entity filter runs at post-processing, see 3.2).
- Equation text like `"h_t = σ(W_h x_t)"` — skipped entirely, no NER run.
- Pure numbers — skipped, prevents version-number FPs.

### 3.2 Post-Processing Entity Filter

After NER produces predictions and before `EntityAnnotation` objects are constructed, apply a filter that discards structurally invalid entity spans:

```python
_ENTITY_ARTICLE_PREFIX_RE = re.compile(r"^(The|A|An)\s+", re.IGNORECASE)
_MIN_ENTITY_CHARS = 2
_MAX_ENTITY_WORDS = 12  # Titles can be long but 12+ word spans are noise

def _is_valid_entity(text: str, label: str) -> bool:
    """
    Post-prediction filter for individual entity spans.
    Returns False for structurally implausible entities.
    """
    t = text.strip()

    # Too short
    if len(t) < _MIN_ENTITY_CHARS:
        return False

    # Entity span starts with a definite/indefinite article
    # "The encoder", "A model", "An approach" are not named entities
    if _ENTITY_ARTICLE_PREFIX_RE.match(t):
        return False

    # Too many words (likely a full sentence captured by accident)
    word_count = len(t.split())
    if word_count > _MAX_ENTITY_WORDS:
        return False

    # Pure lowercase multi-word with no capitals = common noun phrase, not NE
    # Exception: metrics like "f1", datasets like "penn treebank" (lower-cased)
    # Only apply to person/institution labels where capitalization is expected
    _CAPS_REQUIRED_LABELS = {
        "name of a human researcher, author, or scientist",
        "named university, research institute, company, or standards body",
        "geopolitical location, country, city, or region",
    }
    if label in _CAPS_REQUIRED_LABELS and t == t.lower() and word_count >= 2:
        return False

    return True
```

### 3.3 Integration Points

Both functions integrate into `NERStep.annotate()`:

1. `_preprocess_for_ner()` replaces the current bare `text.strip()` check in the element loop. If it returns `None`, treat as empty (entities=[]).
2. `_is_valid_entity()` is called inside the prediction-application loop, before constructing `EntityAnnotation`.

---

## 4. Model Upgrade Path

### 4.1 Current Model

`urchade/gliner_medium-v2.1` — GLiNER medium, ~170M parameters, trained on a broad NER corpus. Adequate for short generic labels. Struggles with long descriptive label strings as zero-shot prompts.

### 4.2 Recommended Model

`numind/NuNER_Zero` (primary) or `numind/NuNER_Zero-4k` (if processing long-context elements).

**Why NuNER Zero is better for this use case:**

- NuNER Zero is explicitly trained for zero-shot NER with descriptive, natural-language label strings — exactly how the redesigned `DEFAULT_LABELS` are written.
- It achieves higher recall on novel entity types (paper titles, ML models, conference venues) without fine-tuning.
- The 4k variant uses a longer context window, useful for elements that hit the current 1024-char truncation limit.

### 4.3 API Compatibility

NuNER Zero is distributed as a GLiNER-compatible model. The `gliner` Python package loads it identically:

```python
from gliner import GLiNER
model = GLiNER.from_pretrained("numind/NuNER_Zero")
```

**`predict_entities()` and `inference()` — identical interface.** Both methods exist on NuNER Zero with the same signatures as `gliner_medium-v2.1`:

```python
# Single text (current single-text path in ner.py line 292)
preds = model.predict_entities(text, labels, threshold=0.0)

# Batch (current batch path in ner.py line 295)
preds = model.inference(texts, labels, threshold=0.0)
```

No calling-code changes are required beyond updating `DEFAULT_LOCAL_MODEL_ID`.

**One behavioral difference to be aware of:** NuNER Zero scores tend to be distributed differently from GLiNER medium — scores cluster slightly lower on average. The proposed per-label thresholds in Section 2 have been calibrated for NuNER Zero. If remaining on `gliner_medium-v2.1`, raise all thresholds by ~0.05.

### 4.4 Required Change to `ner.py`

```python
# Line 32 — change from:
DEFAULT_LOCAL_MODEL_ID = "urchade/gliner_medium-v2.1"

# To:
DEFAULT_LOCAL_MODEL_ID = "numind/NuNER_Zero"
```

The `extraction_method` field on `EntityAnnotation` currently hardcodes `"gliner2"`. Update to `"nuner_zero"` when the new default is in use, or make it dynamic based on `model_id`:

```python
extraction_method = "nuner_zero" if "NuNER" in self.model_id else "gliner2"
```

### 4.5 pyproject.toml `[ner]` Extras

No version bumps are required. `gliner>=0.2.0` already supports NuNER Zero. The current extras are sufficient:

```toml
ner = [
    "gliner>=0.2.0",      # no change needed
    "torch>=2.0.0",       # no change needed
    "rapidfuzz>=3.0.0",   # no change needed
]
```

NuNER Zero weights are pulled from HuggingFace Hub on first load via `GLiNER.from_pretrained()`, same as the current model.

---

## 5. Evaluation Script Specification

### 5.1 File

`tests/unit/test_ner_quality.py`

### 5.2 Fixture Setup

```python
import pytest

# Skip entire module if gliner is not installed
pytest.importorskip("gliner", reason="[ner] extras not installed")

from extractor.ner import NERStep, DEFAULT_LABELS, DEFAULT_LABEL_THRESHOLDS

@pytest.fixture(scope="module")
def ner_step():
    """Real model fixture — marked slow, skipped in CI unless RUN_NER_TESTS=1."""
    import os
    if not os.getenv("RUN_NER_TESTS"):
        pytest.skip("Set RUN_NER_TESTS=1 to run NER quality tests (requires model download)")
    return NERStep(
        labels=DEFAULT_LABELS,
        threshold=0.50,
    )

def _entities_for(ner_step, text):
    """Helper: run NER on a single string, return list of (entity_text, label) tuples."""
    from extractor.schema import BaseElement
    el = BaseElement(
        element_type="text:paragraph",
        text=text,
        chunk_id="test-0",
        source_id="test",
        page=1,
        order=0,
    )
    results = list(ner_step.annotate([el]))
    return [(e.text, e.entity_type) for e in (results[0].entities or [])]
```

### 5.3 The 6 Core Assert Cases

```python
def test_fp_adam_not_a_person(ner_step):
    """'Adam' alone should not match researcher label above threshold 0.78."""
    entities = _entities_for(ner_step, "We used Adam as the optimizer.")
    person_label = "name of a human researcher, author, or scientist"
    person_hits = [e for e in entities if e[1] == person_label and e[0] == "Adam"]
    assert person_hits == [], f"FP: 'Adam' matched person label: {person_hits}"


def test_fp_bleu_not_an_organization(ner_step):
    """'BLEU' should not match institution label."""
    entities = _entities_for(ner_step, "The model achieved 38.5 BLEU on WMT14.")
    institution_label = "named university, research institute, company, or standards body"
    org_hits = [e for e in entities if e[1] == institution_label and e[0] == "BLEU"]
    assert org_hits == [], f"FP: 'BLEU' matched organization label: {org_hits}"


def test_fp_the_encoder_filtered(ner_step):
    """'The encoder' must not appear as any entity — article-prefix filter."""
    entities = _entities_for(ner_step, "The encoder maps input tokens to hidden states.")
    article_hits = [e for e in entities if e[0].lower().startswith("the ")]
    assert article_hits == [], f"FP: article-prefixed span survived: {article_hits}"


def test_tp_paper_title(ner_step):
    """Long paper title should be found under the paper title label."""
    text = 'They cite "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation".'
    entities = _entities_for(ner_step, text)
    title_label = "title of a research paper, book, or technical report"
    title_hits = [e for e in entities if e[1] == title_label]
    assert title_hits, "FN: paper title not found"
    assert any("Learning Phrase" in e[0] for e in title_hits), f"Title span wrong: {title_hits}"


def test_tp_transformer_model(ner_step):
    """'Transformer' should be recognized as an ML model."""
    entities = _entities_for(ner_step, "The Transformer architecture was proposed in 2017.")
    model_label = "named machine learning model, neural network architecture, or algorithm"
    model_hits = [e for e in entities if e[1] == model_label]
    assert any("Transformer" in e[0] for e in model_hits), f"FN: Transformer not found in {model_hits}"


def test_tp_neurips_venue(ner_step):
    """'NeurIPS 2017' should be recognized as a conference venue."""
    entities = _entities_for(ner_step, 'Vaswani et al. presented this work at NeurIPS 2017.')
    venue_label = "academic conference, workshop, or journal venue"
    venue_hits = [e for e in entities if e[1] == venue_label]
    assert any("NeurIPS" in e[0] for e in venue_hits), f"FN: NeurIPS 2017 not found in {venue_hits}"
```

---

## 6. Implementation Checklist (Ordered by Priority)

Work through these in order. Each item is independently testable before moving to the next.

```
Priority 1 — Label redesign (highest impact, no model change needed)
[ ] ner.py: replace DEFAULT_LABELS list with the 10 descriptive labels from Section 1.2
[ ] ner.py: add DEFAULT_LABEL_THRESHOLDS dict (Section 2)
[ ] ner.py: add NERStep._label_threshold(label) method
[ ] ner.py: update prediction-application loop to call _label_threshold(pred["label"])
      instead of a single self.threshold for score comparison
[ ] tests/unit/test_ner_quality.py: add fixture + 6 assert cases (Section 5)

Priority 2 — Text preprocessing and entity filters
[ ] ner.py: add module-level regex constants (_ARTICLE_PREFIX_RE, _EQUATION_FRAGMENT_RE,
      _URL_RE, _NUMERIC_ONLY_RE, _ENTITY_ARTICLE_PREFIX_RE)
[ ] ner.py: add _preprocess_for_ner(text: str) -> str | None function (Section 3.1)
[ ] ner.py: add _is_valid_entity(text: str, label: str) -> bool function (Section 3.2)
[ ] ner.py: in NERStep.annotate(), replace bare text.strip() check with
      _preprocess_for_ner() call; use returned cleaned text for NER
[ ] ner.py: in prediction-application loop, add _is_valid_entity() check before
      constructing EntityAnnotation
[ ] tests/unit/test_ner_quality.py: verify test_fp_the_encoder_filtered passes

Priority 3 — Model upgrade
[ ] ner.py: update DEFAULT_LOCAL_MODEL_ID from "urchade/gliner_medium-v2.1"
      to "numind/NuNER_Zero"
[ ] ner.py: update module docstring line 4 to reference NuNER_Zero as default
[ ] ner.py: in NERStep prediction-application loop, set extraction_method dynamically:
      "nuner_zero" if "NuNER" in self.model_id else "gliner2"
[ ] Run full test suite: pytest tests/unit/test_ner_quality.py -v (with RUN_NER_TESTS=1)
[ ] pyproject.toml: no version bumps needed for [ner] extras (gliner>=0.2.0 sufficient)

Priority 4 — Validation and documentation
[ ] Run ruff check src/extractor/ner.py and fix any lint issues
[ ] Run mypy on src/extractor/ner.py (add extractor.ner to mypy overrides if needed
      due to gliner being untyped)
[ ] Update CHANGELOG for v1.2.1
[ ] Commit with message: "feat(ner): redesign labels, per-label thresholds, preprocessing filters"
```

---

## 7. Files Changed

| File | Change type | Description |
|---|---|---|
| `src/extractor/ner.py` | Modify | All code changes (labels, thresholds, preprocessing, filters, model ID) |
| `tests/unit/test_ner_quality.py` | Create | NER quality regression tests |
| `pyproject.toml` | No change | `[ner]` extras already sufficient |

---

## 8. Before / After Summary

### Labels

| Before | After |
|---|---|
| `"organization"` | `"named university, research institute, company, or standards body"` |
| `"person"` | `"name of a human researcher, author, or scientist"` |
| `"date"` | `"specific date, year, or named time period"` |
| `"location"` | `"geopolitical location, country, city, or region"` |
| `"product"` | _(removed — subsumed)_ |
| `"technology"` | _(removed — subsumed)_ |
| `"event"` | `"academic conference, workshop, or journal venue"` |
| `"quantity"` | _(removed — too noisy)_ |
| _(none)_ | `"title of a research paper, book, or technical report"` |
| _(none)_ | `"named machine learning model, neural network architecture, or algorithm"` |
| _(none)_ | `"named dataset or benchmark used for evaluation"` |
| _(none)_ | `"evaluation metric or performance measure"` |
| _(none)_ | `"named software library, framework, or programming language"` |

### Model

| Before | After |
|---|---|
| `urchade/gliner_medium-v2.1` | `numind/NuNER_Zero` |

### Threshold strategy

| Before | After |
|---|---|
| Single global threshold (default 0.50) | Per-label thresholds + global fallback |
| `NER_HIGH_THRESHOLD_DELTA` (+0.15) for structural element types | Retained; per-label thresholds apply independently |

---

## 9. Known Risks and Mitigations

**Risk: NuNER Zero model size / download time.**
NuNER Zero is ~560MB. First-load latency is similar to `gliner_medium-v2.1` (~440MB). Mitigated by existing `preload_model()` call pattern.

**Risk: Per-label threshold calibration is heuristic.**
The thresholds in Section 2 are derived from the FP/FN cases above, not from a labeled evaluation set. A proper evaluation requires a 50–100 sentence gold-standard corpus from the target document domain. The `tests/unit/test_ner_quality.py` tests establish a regression floor, but threshold values should be revised after running on real documents.

**Risk: Long descriptive labels slow inference.**
GLiNER/NuNER encodes each label string independently per inference call. 10 labels at ~10 tokens each vs. the current 8 labels at ~1 token each increases the label-encoding portion of inference time. Benchmark before/after on a 100-element batch to confirm it stays within acceptable latency.

**Risk: `_is_valid_entity()` `_CAPS_REQUIRED_LABELS` check is English-centric.**
If the pipeline is used on non-English documents, the capitalization check will suppress valid entities. The check is opt-in (only 3 labels) and can be disabled by clearing `_CAPS_REQUIRED_LABELS`.
