# Label Generation Library

## 1. Objective

Build a **Python library** that generates **discrete multi-label annotations** from a corpus of text paragraphs.

This library is **not** coupled to RAG, retrieval, ranking, or answer generation. Treat it as a standalone offline pipeline.

The immediate goal is to deliver a **baseline v0** implementation with a clean architecture, reproducible behavior, and testable modules.

The library should support the following end-to-end workflow:

1. Accept a collection of input paragraphs.
2. Extract concepts from each paragraph.
3. Build a global concept co-occurrence graph.
4. Run community detection over the graph.
5. Assign one or more discrete labels to each paragraph based on community membership.
6. Produce a structured result object that can be serialized and inspected.

Do **not** implement advanced research improvements yet unless they are necessary for basic robustness.

---

## 2. Scope Constraints

### In scope for v0

- Paragraph-level processing
- Deterministic paragraph IDs
- Concept extraction using a practical NLP baseline
- Basic concept normalization and filtering
- Concept co-occurrence graph construction
- Community detection with Leiden
- Paragraph-to-label assignment
- Simple human-readable label verbalization
- Unit tests for major modules
- Clean public API

### Explicitly out of scope for v0

Do not spend time on the following unless the implementation naturally leaves extension points:

- Query-side labeling
- Retrieval integration
- RAG-specific evaluation
- LLM-based label naming
- Coreference resolution
- Sophisticated synonym merging
- Advanced graph reweighting schemes
- Embedding-based concept consolidation
- Web app / UI / notebook-first code structure

The deliverable is a **library**, not an experiment notebook.

---

## 3. Design Principles

### 3.1 Reproducibility

The pipeline must be deterministic as far as practical.

Requirements:
- Stable paragraph IDs
- Explicit random seed in config
- Stable iteration / sorting when order matters
- Config-driven behavior rather than hidden constants

### 3.2 Modularity

Separate the pipeline into replaceable components. The baseline implementation can be simple, but the architecture must make future replacement easy.

Target replaceable components:
- concept extractor
- graph builder
- community detector
- label assigner
- label verbalizer

### 3.3 Interpretability

The output must make it possible to inspect:
- which concepts were extracted from each paragraph
- which communities were found
- why a paragraph received a label
- what each label corresponds to in terms of representative concepts

### 3.4 Library-first engineering

Use typed Python, clean modules, explicit data models, and tests. Avoid a script pile.

---

## 4. Proposed Public API

The public API should be compact.

### Primary usage

```python
from labelgen import LabelGenerator, LabelGeneratorConfig

config = LabelGeneratorConfig()
generator = LabelGenerator(config)
result = generator.fit_transform(paragraphs)
```

### Minimum public methods

```python
fit(paragraphs)
transform(paragraphs)
fit_transform(paragraphs)
```

### Optional if straightforward

```python
save(path)
load(path)
```

If persistence is implemented, keep it simple and predictable.

---

## 5. Input / Output Expectations

### Input

For v0, assume input is already paragraphized or close to paragraphized.

Support at least:
- `list[str]`
- optionally `list[Paragraph]`

If raw strings are passed, wrap them into internal `Paragraph` objects.

### Output

Return a structured result object rather than plain dictionaries.

The result should include at least:
- normalized paragraph objects
- extracted concepts
- paragraph-level concept mentions
- concept graph summary
- communities
- paragraph label assignments
- readable label names or summaries

---

## 6. Core Data Models

Use `dataclasses` or `pydantic` models. Prefer simple typed structures over nested dicts.

At minimum define the following models.

### Paragraph

Fields:
- `id: str`
- `text: str`
- `metadata: dict[str, Any] | None`

### Concept

Fields:
- `id: str`
- `surface: str`
- `normalized: str`
- `kind: str`  
  Expected values for v0: `"entity"`, `"noun_phrase"`
- `document_frequency: int | None`

### ConceptMention

Fields:
- `paragraph_id: str`
- `concept_id: str`
- `surface: str`
- `normalized: str`
- `kind: str`
- optional span offsets if convenient

### Community

Fields:
- `id: str`
- `concept_ids: list[str]`
- `display_name: str`
- `representative_concepts: list[str]`
- `size: int`

### ParagraphLabels

Fields:
- `paragraph_id: str`
- `label_ids: list[str]`
- `evidence_concept_ids: list[str] | None`

### LabelGenerationResult

Fields:
- `paragraphs: list[Paragraph]`
- `concepts: list[Concept]`
- `mentions: list[ConceptMention]`
- `communities: list[Community]`
- `paragraph_labels: list[ParagraphLabels]`
- optional graph statistics / debug metadata

---

## 7. Project Structure

Use a maintainable package structure.

```text
labelgen/
  __init__.py
  config.py
  types.py

  preprocessing/
    __init__.py
    paragraphs.py

  extraction/
    __init__.py
    concept_extractor.py
    spacy_extractor.py
    normalization.py
    filtering.py

  graph/
    __init__.py
    builder.py
    concept_graph.py

  community/
    __init__.py
    detector.py
    leiden_detector.py

  labeling/
    __init__.py
    assigner.py
    verbalizer.py

  pipeline/
    __init__.py
    label_generator.py

  io/
    __init__.py
    serialize.py

tests/
  ...
```

This structure can be simplified if necessary, but keep module boundaries clear.

---

## 8. Implementation Requirements by Module

## 8.1 Configuration

Create a central config object, for example `LabelGeneratorConfig`.

Include at least:
- `random_seed`
- `min_concept_chars`
- `min_concept_df`
- `max_concept_df_ratio` or equivalent
- `min_edge_weight`
- `min_label_support`
- `top_k_label_terms`
- `spacy_model_name`
- Leiden-related parameters such as resolution if supported

Do not scatter magic numbers across modules.

---

## 8.2 Preprocessing

Implement paragraph normalization utilities.

Requirements:
- Strip leading/trailing whitespace
- Drop empty paragraphs
- Generate deterministic IDs
- Preserve original ordering

Deterministic ID can be based on stable hashing of paragraph text plus index, or similar. Simpler is acceptable as long as it is deterministic.

Do not over-engineer document parsing in v0.

---

## 8.3 Concept Extraction

### Baseline approach

Use spaCy as the baseline NLP backend.

Extract from each paragraph:
- named entities
- noun chunks

For v0, combine both and allow overlap. Deduplication can happen after normalization.

### Normalization rules

Implement simple normalization, for example:
- strip whitespace
- collapse internal repeated whitespace
- lowercase normalized form
- remove trivial punctuation-only items
- reject too-short concepts
- optionally reject all-stopword concepts

Keep both:
- original surface form
- normalized form

Do not implement aggressive lemmatization if it risks damaging multi-word expressions.

### Filtering

After extraction across the corpus, compute document frequency for normalized concepts.

Support basic filtering:
- remove concepts below `min_concept_df`
- optionally remove overly frequent concepts via `max_concept_df_ratio`

Important: document frequency should be counted per paragraph, not per mention.

---

## 8.4 Concept Mentions

Track concept mentions at paragraph level.

Requirements:
- a paragraph can mention the same concept multiple times, but graph construction should usually use paragraph-level unique concepts
- preserve enough mention data for debugging

---

## 8.5 Graph Construction

Build a global undirected weighted co-occurrence graph.

### Nodes
- normalized concepts

### Edges
- connect two concepts if they appear in the same paragraph

### Edge weight
- number of distinct paragraphs in which the pair co-occurs

### Important implementation detail
For each paragraph:
- deduplicate concepts first
- then add pairwise co-occurrence counts

This avoids inflating weights from repeated mentions inside a single paragraph.

### Basic pruning
Support at least:
- concept-level filtering before graph construction
- edge-level filtering by `min_edge_weight`

Use a standard graph library such as `networkx` unless a stronger reason exists not to.

---

## 8.6 Community Detection

Implement a community detector abstraction and provide a Leiden-based implementation.

Acceptable approach:
- convert the graph into a format suitable for Leiden
- run community detection with explicit random seed

Output must map each concept to a community ID.

Community IDs should be stable enough for a single run, but they do not need to be semantically stable across different corpora.

Also compute:
- community size
- list of concepts per community

If Leiden requires additional library dependencies, set them up cleanly and document them in the project config.

---

## 8.7 Label Verbalization

v0 should include a minimal readable labeling strategy.

For each community, derive representative concepts using a simple scoring rule. Examples of acceptable approaches:
- highest document frequency within the community
- degree-central nodes in the community subgraph
- combined heuristic

Then set:
- `display_name` to top concept or joined top concepts
- `representative_concepts` to top-k normalized concepts

Example display names:
- `"machine learning / neural networks / training"`
- `"roman empire / charlemagne / christianity"`

Do not use LLM calls.

---

## 8.8 Paragraph Label Assignment

Assign paragraph labels based on the communities of the concepts it contains.

### Baseline rule

For each paragraph:
1. collect all retained concepts in the paragraph
2. map them to community IDs
3. count support per community
4. assign all communities meeting `min_label_support`

Example:
- if a paragraph contains 3 concepts from community A and 1 from community B
- and `min_label_support = 1`, assign both A and B
- and if `min_label_support = 2`, assign only A

Return evidence concept IDs if practical.

This logic should be explicit and testable.

---

## 8.9 Pipeline Orchestration

Implement a `LabelGenerator` class that coordinates the full process.

Suggested internal flow:

1. normalize input paragraphs
2. extract raw mentions
3. build concept inventory
4. compute document frequencies and filter concepts
5. build graph
6. detect communities
7. verbalize communities
8. assign paragraph labels
9. return `LabelGenerationResult`

The code should be readable and stage-oriented rather than overly abstract.

---

## 9. Suggested Dependencies

Use only what is necessary.

Likely acceptable dependencies:
- `spacy`
- `networkx`
- `numpy`
- `python-igraph` and/or `leidenalg` if needed for Leiden
- `pytest`

Optional:
- `pydantic` if desired for data validation

Avoid dependency bloat.

---

## 10. Testing Requirements

Write tests early. This library is structured enough that tests matter.

### Minimum required tests

#### Preprocessing
- empty paragraph removal
- deterministic ID generation
- stable ordering

#### Extraction
- entity extraction returns expected concepts on simple text
- noun phrase extraction returns expected concepts on simple text
- normalization behaves as expected
- DF filtering behaves as expected

#### Graph construction
- graph nodes reflect retained concepts
- co-occurrence edges are created correctly
- repeated mentions in one paragraph do not inflate edge counts
- edge pruning works

#### Community detection
- returns a concept-to-community mapping on a toy graph
- deterministic behavior under fixed seed, as far as the backend allows

#### Label assignment
- paragraph gets expected labels based on concept-community mapping
- `min_label_support` is respected

#### Pipeline integration
- a small end-to-end test on a toy corpus
- output object contains expected structures

Do not skip tests just because the first implementation is a baseline.

---

## 11. Quality Bar

The implementation should aim for:
- clean Python packaging
- type hints throughout
- small focused modules
- docstrings on public classes/functions
- no giant monolithic file
- no notebook-only workflow
- no hidden global state

If tradeoffs arise, prefer clarity and reliability over cleverness.

---

## 12. Non-Goals and Cautions

### Avoid premature optimization
Do not optimize for very large-scale corpora yet.

### Avoid over-abstracting
Use abstractions where future replacement is likely, but do not build a framework for its own sake.

### Avoid task drift
Do not turn this into a retrieval or RAG library.

### Avoid unreadable output
Community IDs alone are not enough. Provide minimal readable verbalization.

---

## 13. Expected Deliverables

The code work should produce:

1. A Python package implementing the described v0 pipeline.
2. A clear README with:
   - installation instructions
   - required NLP model download steps
   - a minimal example
3. Unit tests for the major modules.
4. Reasonable defaults in config.
5. Example output on a toy corpus.

---

## 14. Nice-to-Have If Time Permits

These are optional and should not block the core library:

- JSON serialization for result objects
- export helpers for communities and paragraph labels
- debug report helpers
- simple CLI entry point for running the pipeline on a text file

Again: optional.

---

## 15. Final Implementation Priority Order

Build in this order:

1. data models and config
2. paragraph preprocessing
3. spaCy-based concept extraction
4. concept filtering
5. graph construction
6. Leiden community detection
7. label verbalization
8. paragraph label assignment
9. pipeline class
10. tests and README

Do not invert this order unless there is a strong reason.

---

## 16. Summary

Build a **baseline, modular, reproducible Python label generation library** that:
- works at paragraph level
- extracts concepts using a practical NLP baseline
- constructs a concept co-occurrence graph
- detects concept communities
- assigns discrete multi-label outputs to paragraphs
- exposes a clean and testable API

Keep the implementation narrow, disciplined, and extensible. The point of v0 is to close the pipeline cleanly, not to solve every downstream research problem in the first pass.

