# `0.1.1` Iteration Plan

## Summary

This plan is based on an exploratory experiment with `paralabelgen 0.1.0`
against the `PrimeQA/TechQA` dataset. The experiment goal was to evaluate
concept extraction quality, compare the default pipeline against the explicit
fallback pipeline, and identify the highest-value improvements for the next
release.

The main conclusion is clear: the default spaCy + Leiden pipeline is already
better than the heuristic fallback on technical paragraphs, but both modes are
still heavily affected by document boilerplate, URLs, fix-list formatting, and
generic status language. The next version should focus on technical-document
cleanup and concept quality before expanding the algorithm surface area.

## Experiment Setup

- Environment:
  - isolated Python `3.11` virtual environment in `~/repo/temp/.venv`
  - published package from PyPI: `paralabelgen==0.1.0`
  - spaCy model: `en_core_web_sm`
- Dataset:
  - source: `PrimeQA/TechQA`
  - corpus member: `TechQA/technote_corpus/technotes.id_title_text`
- Sampling:
  - fixed random seed: `42`
  - sampled paragraphs: `100`
  - paragraph filter: at least `40` characters and `6` words
  - practical scope: the first `2000` technote documents streamed from the
    corpus archive
  - candidate paragraphs seen inside that scope: `24,866`
- Compared configurations:
  - default: `use_nlp_extractor=True`, `use_graph_community_detection=True`
  - fallback: `use_nlp_extractor=False`, `use_graph_community_detection=False`

## Key Findings

### 1. The default pipeline is materially better than fallback

- Default stats:
  - `1,279` mentions
  - `1,032` concepts
  - `43` communities
  - `12.79` mentions per paragraph on average
- Fallback stats:
  - `3,668` mentions
  - `2,733` concepts
  - `31` communities
  - `36.68` mentions per paragraph on average
- The fallback extractor greatly over-generates phrase fragments on list-heavy
  technical text and creates very large noisy communities.
- The most obvious failure is one fallback community with `2,345` concepts and
  the label `www ibm com support / com support docview wss / ibm com support docview`.

### 2. Boilerplate and markup dominate many extracted concepts

- The default path still surfaces highly generic or boilerplate concepts such as:
  - `you`
  - `problem summary`
  - `problem description`
  - `fix information`
  - `all active apars`
- The sampled paragraph-level concept sets still contained:
  - `37` default concepts with `http`
  - `27` default concepts starting with `*`
  - `277` fallback concepts with `http`
- Community names are often driven by document boilerplate instead of topic
  semantics. Examples:
  - `you / ibm / all active apars`
  - `problem summary / problem description / error`
  - `a user / errors / " (a blank space`

### 3. The default extractor is acceptable on prose-like bug descriptions

- On short narrative paragraphs the default path often produces usable concepts.
- Example:
  - doc id: `swg1IJ04413`
  - text: `PROBLEM DESCRIPTION: An internal compiler optimization was incorrectly folding an internal variable leading to incorrect inlining results.`
  - extracted concepts: `an internal compiler optimization`, `an internal variable`, `incorrect inlining results`
- This suggests the next iteration should not replace the default extractor.
  It should clean and constrain its inputs and outputs.

### 4. Structured fix-list paragraphs are still noisy even with spaCy

- The default path performs better than fallback on long fix lists, but still
  keeps many low-quality concepts derived from:
  - URLs
  - repeated APAR templates
  - bullet separators
  - version strings and mixed punctuation blocks
- Example:
  - doc id: `swg27050982`
  - default unique concepts: `150`
  - fallback unique concepts: `646`
- The default output is smaller, but still too noisy for stable community
  naming and paragraph labeling.

### 5. Label quality is still bottlenecked by generic communities

- Even when paragraph-level concepts are partly reasonable, assigned labels are
  often too generic because community names are dominated by repeated support
  language.
- Example:
  - doc id: `swg1IT23704`
  - paragraph concepts include `cisco port-channel status` and `the switch probe`
  - one assigned default label is still `problem summary / problem description / error`
- This points to a ranking problem in community verbalization and label
  assignment, not only an extraction problem.

## `0.1.1` Priorities

### Priority 1: Add technical-document cleanup before extraction

- Introduce a preprocessing layer for support-note style text that can:
  - strip or downweight raw URLs
  - normalize repeated separator banners such as long `***` blocks
  - suppress known section headers when they appear as standalone boilerplate
    concepts, for example:
    - `PROBLEM SUMMARY`
    - `PROBLEM DESCRIPTION`
    - `FIX INFORMATION`
    - `SUBSCRIBE TO THIS APAR`
  - collapse list formatting artifacts before concept extraction
- This should be configurable, but enabled by default for the library pipeline.

### Priority 2: Improve concept filtering and canonicalization

- Add post-extraction filters for concepts that are:
  - URL-like
  - markup-heavy
  - mostly punctuation
  - obvious pronouns or generic shells such as `you`, `one`, `they`
- Merge or deduplicate concepts that only differ by extraction kind when their
  normalized text is identical and the distinction is not useful.
- Add optional heuristics for technical tokens:
  - preserve meaningful product/version identifiers
  - avoid splitting useful identifiers into noisy fragments

### Priority 3: Make community naming more resistant to generic concepts

- Update community verbalization ranking so repeated boilerplate terms do not
  dominate `display_name` and `representative_concepts`.
- Downweight concepts that are globally common, structurally generic, or matched
  by the new cleanup filters.
- Ensure community names prefer technical content terms over support-site noise.

### Priority 4: Tighten paragraph label assignment

- Reduce the influence of generic communities during label assignment.
- Prefer communities with more content-bearing concepts as evidence.
- Add a guard so one oversized noisy community does not become the default label
  for many unrelated paragraphs.

### Priority 5: Keep a reproducible evaluation harness in the repository

- Add a small experiment script or documented workflow that can rerun this
  dataset-based evaluation on a fixed sample.
- Persist the sample seed, sampling scope, and summary metrics so concept
  quality regressions can be checked before release.

## Acceptance Criteria For `0.1.1`

- Default extraction on the same TechQA-style sample should noticeably reduce:
  - URL-heavy concepts
  - boilerplate section-header concepts
  - pronoun/generic filler concepts
- Community names should stop being dominated by phrases like
  `problem summary`, `you`, or `www ibm com support`.
- The fallback pipeline may remain simpler, but the default pipeline should
  become clearly more reliable on support-note and fix-list paragraphs.
- The repository should contain a documented, reproducible way to rerun this
  experiment and compare future output quality.

## Chosen Defaults

- The next release target is `0.1.1`.
- The default pipeline remains spaCy + Leiden.
- The immediate focus is quality improvement for technical support documents,
  not new external integrations or larger model dependencies.
