---
title: "feat: Filter chapter list to remove noise from TOC detection"
type: feat
status: completed
date: 2026-03-08
origin: docs/brainstorms/2026-03-08-chapter-filtering-brainstorm.md
---

# feat: Filter chapter list to remove noise from TOC detection

## Overview

Improve `get_chapters()` to return only meaningful content chapters by filtering out two types of noise from TOC-based detection:

1. **Sub-sections** — Level-2 TOC entries (e.g., "What is your reason for being?") detected as chapters
2. **Meta entries** — Non-content front/back matter (Copyright, Dedication, Contents, Notes, About the Authors, etc.)

Changes are scoped to `_detect_from_toc()` in `condensr/parser.py`. Heading-based fallback (Tier 2) is unaffected (see brainstorm: `docs/brainstorms/2026-03-08-chapter-filtering-brainstorm.md`).

## Proposed Solution

### Change 1: Restrict to level-1 TOC entries

In `parser.py:161`, change:

```python
entries = [(title, page - 1) for level, title, page in toc if level <= 2]
```

to:

```python
entries = [(title, page - 1) for level, title, page in toc if level == 1]
```

### Change 2: Add multi-language blocklist

Add a module-level `frozenset` of common non-content chapter titles in EN, FR, ES, DE. Filter entries **before** page boundary calculation so that filtered pages are absorbed by adjacent chapters rather than orphaned.

```python
# parser.py — module level
_BLOCKLIST_TITLES: frozenset[str] = frozenset({
    # English
    "title page", "half title", "copyright", "copyright page",
    "dedication", "epigraph", "contents", "table of contents",
    "list of figures", "list of tables", "list of illustrations",
    "notes", "endnotes", "footnotes", "bibliography", "references",
    "glossary", "index", "appendix", "appendices",
    "acknowledgements", "acknowledgments", "about the author",
    "about the authors", "colophon",
    "suggestions for further reading", "further reading",
    "also by", "other books by",
    # French
    "page de titre", "droits d'auteur", "copyright",
    "dedicace", "dédicace", "epigraphe", "épigraphe",
    "table des matieres", "table des matières", "sommaire",
    "notes", "bibliographie", "references", "références",
    "glossaire", "index", "annexe", "annexes",
    "remerciements", "a propos de l'auteur", "à propos de l'auteur",
    # Spanish
    "pagina de titulo", "página de título",
    "derechos de autor", "dedicatoria", "epigrafe", "epígrafe",
    "contenido", "tabla de contenido", "indice", "índice",
    "notas", "bibliografia", "bibliografía", "referencias",
    "glosario", "apendice", "apéndice",
    "agradecimientos", "sobre el autor", "sobre la autora",
    # German
    "titelseite", "impressum", "urheberrecht",
    "widmung", "inhaltsverzeichnis",
    "anmerkungen", "literaturverzeichnis", "quellenverzeichnis",
    "glossar", "stichwortverzeichnis", "anhang",
    "danksagung", "uber den autor", "über den autor",
    "uber die autorin", "über die autorin",
})
```

Filtering logic in `_detect_from_toc()`:

```python
# Keep all level-1 entries for boundary calculation
all_entries = [(title, page - 1) for level, title, page in toc if level == 1]

# In the boundary loop, skip blocklisted titles
for i, (title, start_page) in enumerate(all_entries):
    if title.strip().lower() in _BLOCKLIST_TITLES:
        continue
    # compute end_page from next entry in all_entries (not just kept ones)
```

## Technical Considerations

### Filtering order: before boundary calculation, skip filtered pages

Filtering happens **before** page boundaries are computed. Filtered entries' pages are **excluded** from all chapter ranges — they are not absorbed by adjacent chapters. This keeps the LLM input clean (no copyright boilerplate, no "Notes" section, etc.).

Each chapter runs from its own start page to the next chapter's start page minus 1 (computed from the filtered list). The first chapter starts at its own start page (not page 0), so front-matter pages before the first real chapter are excluded.

### Whitespace stripping

Apply `.strip()` before `.lower()` comparison. PDF TOC entries sometimes contain leading/trailing whitespace.

### Unicode normalization — not needed for v1

Full NFC/NFD normalization adds complexity. The blocklist includes both accented and unaccented variants of common terms (e.g., "dédicace" and "dedicace"), which handles the most common cases. Revisit if real-world mismatches are reported.

### Known limitation: Part/Chapter academic PDFs

Some academic PDFs use level 1 for "Part" and level 2 for actual chapters. Restricting to level 1 loses chapter granularity for these. This is acceptable — condensr targets book summarization, not academic papers.

## Acceptance Criteria

- [x] Level-2 TOC entries are excluded from chapter detection
- [x] Meta entries (Copyright, Dedication, Contents, Notes, About the Authors, etc.) are excluded
- [x] Multi-language meta entries (FR, ES, DE) are excluded
- [x] Real content chapters (Prologue, Chapter I, Epilogue, Foreword, Preface) are preserved
- [x] Chapters with titles that are superstrings of blocklist entries are preserved (e.g., "Notes on the Revolution")
- [x] Page boundaries are recalculated after filtering (no orphaned pages)
- [x] Empty result after filtering falls through to Tier 2 heading detection as normal
- [x] Update docstring on `_detect_from_toc` to reflect new behavior

## MVP

### condensr/parser.py

```python
_BLOCKLIST_TITLES: frozenset[str] = frozenset({
    # English
    "title page", "half title", "copyright", "copyright page",
    "dedication", "epigraph", "contents", "table of contents",
    "list of figures", "list of tables", "list of illustrations",
    "notes", "endnotes", "footnotes", "bibliography", "references",
    "glossary", "index", "appendix", "appendices",
    "acknowledgements", "acknowledgments", "about the author",
    "about the authors", "colophon",
    "suggestions for further reading", "further reading",
    "also by", "other books by",
    # French
    "page de titre", "droits d'auteur",
    "dedicace", "dédicace", "epigraphe", "épigraphe",
    "table des matieres", "table des matières", "sommaire",
    "bibliographie", "references", "références",
    "glossaire", "annexe", "annexes",
    "remerciements", "a propos de l'auteur", "à propos de l'auteur",
    # Spanish
    "pagina de titulo", "página de título",
    "derechos de autor", "dedicatoria", "epigrafe", "epígrafe",
    "contenido", "tabla de contenido", "indice", "índice",
    "notas", "bibliografia", "bibliografía", "referencias",
    "glosario", "apendice", "apéndice",
    "agradecimientos", "sobre el autor", "sobre la autora",
    # German
    "titelseite", "impressum", "urheberrecht",
    "widmung", "inhaltsverzeichnis",
    "anmerkungen", "literaturverzeichnis", "quellenverzeichnis",
    "glossar", "stichwortverzeichnis", "anhang",
    "danksagung", "uber den autor", "über den autor",
    "uber die autorin", "über die autorin",
})


def _detect_from_toc(doc: pymupdf.Document) -> list[Chapter]:
    """Use doc.get_toc() to build chapter list with page boundaries.

    Filters to level 1 entries only and excludes common non-content
    entries (copyright, dedication, index, etc.) using a multi-language
    blocklist. Page boundaries are computed from all level-1 entries
    before filtering, so each chapter gets exactly its own page range
    and filtered entries' pages are excluded entirely.
    """
    toc = doc.get_toc(simple=True)
    if not toc:
        return []

    # Keep all level-1 entries for boundary calculation
    all_entries = [
        (title, page - 1)
        for level, title, page in toc
        if level == 1
    ]
    if not all_entries:
        return []

    last_page = len(doc) - 1
    chapters = []
    for i, (title, start_page) in enumerate(all_entries):
        # Skip blocklisted titles
        if title.strip().lower() in _BLOCKLIST_TITLES:
            continue
        # End page is bounded by the next level-1 entry (filtered or not)
        if i + 1 < len(all_entries):
            end_page = all_entries[i + 1][1] - 1
        else:
            end_page = last_page
        start_page = max(0, min(start_page, last_page))
        end_page = max(start_page, min(end_page, last_page))
        chapters.append(Chapter(title=title, start_page=start_page, end_page=end_page))

    return chapters
```

### tests/conftest.py — new fixtures

```python
@pytest.fixture
def pdf_with_toc_and_subchapters(tmp_path):
    """PDF with level-1 chapters and level-2 sub-sections."""
    doc = pymupdf.open()
    for i in range(5):
        page = doc.new_page()
        page.insert_text((72, 72), f"Page {i + 1} content")
    toc = [
        [1, "Chapter One", 1],
        [2, "Section 1.1", 1],
        [2, "Section 1.2", 2],
        [1, "Chapter Two", 3],
        [2, "Section 2.1", 4],
        [1, "Chapter Three", 5],
    ]
    doc.set_toc(toc)
    path = tmp_path / "toc_subchapters.pdf"
    doc.save(str(path))
    doc.close()
    return str(path)


@pytest.fixture
def pdf_with_meta_chapters(tmp_path):
    """PDF with meta entries (Copyright, etc.) mixed with real chapters."""
    doc = pymupdf.open()
    for i in range(7):
        page = doc.new_page()
        page.insert_text((72, 72), f"Page {i + 1} content")
    toc = [
        [1, "TITLE PAGE", 1],
        [1, "COPYRIGHT", 2],
        [1, "DEDICATION", 3],
        [1, "CONTENTS", 4],
        [1, "Chapter One", 5],
        [1, "Chapter Two", 6],
        [1, "ABOUT THE AUTHORS", 7],
    ]
    doc.set_toc(toc)
    path = tmp_path / "meta_chapters.pdf"
    doc.save(str(path))
    doc.close()
    return str(path)


@pytest.fixture
def pdf_with_meta_chapters_fr(tmp_path):
    """PDF with French meta entries."""
    doc = pymupdf.open()
    for i in range(5):
        page = doc.new_page()
        page.insert_text((72, 72), f"Page {i + 1} contenu")
    toc = [
        [1, "Dédicace", 1],
        [1, "Remerciements", 2],
        [1, "Chapitre Premier", 3],
        [1, "Chapitre Deux", 4],
        [1, "Bibliographie", 5],
    ]
    doc.set_toc(toc)
    path = tmp_path / "meta_chapters_fr.pdf"
    doc.save(str(path))
    doc.close()
    return str(path)


@pytest.fixture
def pdf_with_superstring_titles(tmp_path):
    """PDF with titles that contain blocklist words but are real chapters."""
    doc = pymupdf.open()
    for i in range(3):
        page = doc.new_page()
        page.insert_text((72, 72), f"Page {i + 1} content")
    toc = [
        [1, "Notes on the Revolution", 1],
        [1, "A Dedication to Craft", 2],
        [1, "The Copyright Wars", 3],
    ]
    doc.set_toc(toc)
    path = tmp_path / "superstring_titles.pdf"
    doc.save(str(path))
    doc.close()
    return str(path)
```

### tests/test_parser.py — new tests

```python
class TestChapterFiltering:
    """Tests for chapter list filtering (level restriction + blocklist)."""

    def test_excludes_level2_subchapters(self, pdf_with_toc_and_subchapters):
        chapters = detect_chapters(pdf_with_toc_and_subchapters)
        titles = [c.title for c in chapters]
        assert titles == ["Chapter One", "Chapter Two", "Chapter Three"]

    def test_excludes_meta_chapters(self, pdf_with_meta_chapters):
        chapters = detect_chapters(pdf_with_meta_chapters)
        titles = [c.title for c in chapters]
        assert titles == ["Chapter One", "Chapter Two"]

    def test_excludes_french_meta_chapters(self, pdf_with_meta_chapters_fr):
        chapters = detect_chapters(pdf_with_meta_chapters_fr)
        titles = [c.title for c in chapters]
        assert titles == ["Chapitre Premier", "Chapitre Deux"]

    def test_preserves_superstring_titles(self, pdf_with_superstring_titles):
        chapters = detect_chapters(pdf_with_superstring_titles)
        titles = [c.title for c in chapters]
        assert titles == [
            "Notes on the Revolution",
            "A Dedication to Craft",
            "The Copyright Wars",
        ]

    def test_page_boundaries_after_filtering(self, pdf_with_meta_chapters):
        chapters = detect_chapters(pdf_with_meta_chapters)
        # TOC: TITLE PAGE(p0), COPYRIGHT(p1), DEDICATION(p2), CONTENTS(p3),
        #       Chapter One(p4), Chapter Two(p5), ABOUT THE AUTHORS(p6)
        # Chapter One: starts at p4, next entry is Chapter Two at p5 → end = p4
        assert chapters[0].start_page == 4
        assert chapters[0].end_page == 4
        # Chapter Two: starts at p5, next entry is ABOUT THE AUTHORS at p6 → end = p5
        # (ABOUT THE AUTHORS pages are excluded, not absorbed)
        assert chapters[1].start_page == 5
        assert chapters[1].end_page == 5
```

## Sources & References

- **Origin brainstorm:** [docs/brainstorms/2026-03-08-chapter-filtering-brainstorm.md](docs/brainstorms/2026-03-08-chapter-filtering-brainstorm.md) — Key decisions: level-1 only, multi-language blocklist, case-insensitive exact match, TOC scope only
- Implementation target: [parser.py:149-177](condensr/parser.py#L149-L177) (`_detect_from_toc`)
- Test fixtures: [conftest.py](tests/conftest.py)
- Test cases: [test_parser.py](tests/test_parser.py)
