# Workspace Notes

## Purpose

This document captures the current acceptance-oriented change recommendations for the
`labelgen` project. It is intended to align implementation work with
`docs/implementation.md`.

## Current Assessment

The project already has a reasonable library structure, typed models, basic
serialization support, and a runnable end-to-end baseline. However, it does not yet
fully satisfy the documented baseline v0 requirements.

The two highest-priority issues are:

1. Paragraph ID generation is currently too weak and can collapse distinct paragraphs
   when their text is identical.
2. Community label naming is currently stable but not sufficiently representative or
   interpretable.

## Required Changes

### 1. Replace the current paragraph ID strategy

The current implementation derives paragraph IDs from `hash(text)` only. This is not
acceptable because repeated paragraph text produces the same ID and breaks
paragraph-level semantics.

Required direction:

- Stop using text-only hashing as the default ID strategy.
- Make paragraph IDs deterministic and position-aware.
- Preserve caller-provided IDs when they already exist.

Recommended precedence:

1. If an input `Paragraph` already has an `id`, keep it.
2. If metadata contains `doc_id`, generate `"{doc_id}#p{index}"`.
3. If metadata contains `title`, generate a normalized title-based ID such as
   `"{normalized_title}#p{index}"`.
4. Otherwise, fall back to a deterministic ID based on normalized text plus index,
   such as `p{index}-{short_hash}` or an equivalent stable scheme.

### 2. Strengthen preprocessing behavior

Preprocessing should match the implementation document more closely.

Required behavior:

- Strip leading and trailing whitespace.
- Drop empty paragraphs after normalization.
- Preserve original ordering.
- Generate deterministic IDs using the improved strategy above.

### 3. Fix the public API semantics

The public API exposes `fit`, `transform`, and `fit_transform`, but the current
implementation does not distinguish their behavior meaningfully.

Required direction:

- Define whether the library is truly stateful across `fit` and `transform`.
- If it is stateful, persist learned state and make `transform` use it.
- If it is intentionally stateless, simplify the public API contract or document the
  behavior explicitly.

### 4. Improve community label naming

Community naming should be derived from representative concepts, not from a stable but
arbitrary ordering of concept IDs.

Required direction:

- Rank concepts within each community using a simple, explicit heuristic.
- Build `representative_concepts` from the top-ranked concepts.
- Build `display_name` from the top one to three ranked concepts.
- Keep the strategy deterministic and easy to inspect.

Acceptable ranking signals for v0:

- document frequency within the corpus
- degree or weighted degree inside the community subgraph
- a simple combined heuristic

### 5. Expand filtering and configuration to meet the documented baseline

The current configuration and filtering logic are thinner than the implementation
document expects.

Required additions:

- support a concept frequency ceiling such as `max_concept_df_ratio`
- support a label threshold such as `min_label_support`
- support a naming parameter such as `top_k_label_terms`
- reject trivial punctuation-only concepts
- optionally reject all-stopword concepts

## Recommended Changes

### 1. Extract paragraph ID construction into a dedicated helper

Create a single helper responsible for paragraph ID generation, for example
`build_paragraph_id(paragraph, index)`. This keeps the policy explicit and easy to
change later.

### 2. Normalize title-based IDs before use

If `title` is used for ID construction, normalize it first:

- lowercase
- collapse whitespace
- replace non-alphanumeric characters with `-`
- collapse repeated `-`

This keeps IDs readable and stable.

### 3. Make ordering guarantees explicit

The implementation should preserve input order wherever order is semantically relevant,
especially for:

- normalized paragraphs
- paragraph label outputs
- any user-facing serialized result

### 4. Make community naming explainable in the result

If practical, include enough metadata or derivation clarity so a reviewer can answer:

- why this paragraph received this label
- why this community is named this way

## Suggested Implementation Shape

### Paragraph ID strategy

Recommended policy:

- `Paragraph.id` provided by caller: keep as-is
- `metadata.doc_id` available: `doc_id#p{index}`
- `metadata.title` available: `normalized_title#p{index}`
- fallback: `p{index}-{short_hash(normalized_text)}`

This preserves deterministic behavior while avoiding collisions from repeated text.

### Community naming strategy

Recommended baseline policy:

1. Score each concept in a community.
2. Sort by score descending, then by normalized text ascending for deterministic
   tie-breaking.
3. Set `representative_concepts` to the top `k` normalized concepts.
4. Set `display_name` to the top one to three concepts joined by `" / "`.

Recommended score inputs:

- primary: document frequency
- tie-breaker: degree or weighted degree in the graph
- final tie-breaker: normalized text

## Testing Priorities

The current test suite is useful but still too light for strict acceptance. The next
round of tests should prioritize:

- repeated paragraph text does not produce duplicate paragraph IDs
- preprocessing strips whitespace and removes empty paragraphs
- paragraph label outputs preserve input order
- community naming uses representative ranking instead of raw concept ID order
- `min_label_support` is respected explicitly
- graph edge counts are not inflated by repeated mentions inside one paragraph
- extraction normalization removes trivial concepts correctly

## Acceptance Standard

This project should be considered ready for baseline v0 acceptance only after:

- paragraph-level identity is reliable
- preprocessing behavior matches the documented contract
- community naming is interpretable
- public API semantics are coherent
- tests cover the required baseline behaviors

## Implementation Response

The requested acceptance-oriented changes have now been implemented.

Completed changes:

- Paragraph preprocessing now strips leading and trailing whitespace, collapses internal
  whitespace, drops empty paragraphs, and preserves input order.
- Paragraph ID generation no longer relies on text-only hashing. The implemented
  precedence is:
  - keep caller-provided `Paragraph.id`
  - use `metadata["doc_id"]` as `"{doc_id}#p{index}"`
  - use normalized `metadata["title"]` as `"{normalized_title}#p{index}"`
  - otherwise fall back to `p{index}-{short_hash(normalized_text)}`
- Paragraph label outputs now preserve input paragraph order explicitly.
- The public API semantics are now stateful and explicit:
  - `fit()` learns concept communities from a corpus
  - `transform()` applies previously learned communities to new paragraphs
  - `fit_transform()` learns and labels the same input in one pass
  - `transform()` raises an error if called before `fit()`
- Community naming is now derived from representative concepts rather than raw concept
  ID ordering. Ranking is deterministic and uses:
  - document frequency
  - weighted degree inside the concept graph
  - normalized text as the final tie-breaker
- Filtering and configuration were expanded to better match the baseline v0 contract:
  - `max_concept_df_ratio`
  - `min_label_support`
  - `top_k_label_terms`
  - rejection of punctuation-only concepts
  - optional rejection of all-stopword concepts
- Graph construction behavior is now explicitly tested to confirm repeated mentions in a
  single paragraph do not inflate edge weights.

Test coverage added or strengthened for:

- repeated paragraph text producing distinct paragraph IDs
- preprocessing whitespace stripping and empty paragraph removal
- paragraph label order preservation
- community naming based on representative ranking
- `min_label_support`
- `max_concept_df_ratio`
- graph edge deduplication within a paragraph
- stateful `fit` / `transform` behavior

Current verification status:

- `ruff check .` passes
- `pyright` passes
- `pytest` passes

At the time of this response, the test suite contains 22 passing tests.
