# `sayou-chunking`

[![Build Status](https://img.shields.io/github/actions/workflow/status/sayouzone/sayou-fabric/ci.yml?branch=main)](https://github.com/sayouzone/sayou-fabric/actions)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Docs](https://img.shields.io/badge/docs-mkdocs-blue.svg)](https://sayouzone.github.io/sayou-fabric/library-guides/chunking/)

`sayou-chunking` is a context-aware text splitting library for Python. It transforms raw text documents into **Knowledge Graph-ready nodes**, focusing on preserving semantic structure, hierarchy, and context.

This library is the "Assembler Preparation" component of the **Sayou Data Platform**. It sits between data cleaning (`sayou-refinery`) and knowledge graph construction (`sayou-assembler`), ensuring that RAG pipelines operate on logical, structured units of information rather than fragmented text.

## Philosophy

`sayou-chunking` believes that **"How you split determines how you retrieve."**
Naive chunking destroys context (e.g., splitting a table in half). We prioritize **Structure-First Splitting**:

1.  **Atomic Protection:** Never split atomic blocks like Tables or Code Snippets.
2.  **Hierarchical Binding:** Headers are parents; contents are children. We maintain `parent_id` linkages for KG construction.
3.  **Composite Strategies:** Combine multiple splitting strategies (e.g., Structure for context, Recursive for retrieval units).

## 🚀 Key Features

* **4-Tier Architecture:** Highly extensible design (Engine -> Interface -> Template -> Plugin -> Composite).
* **Atomic Protection:** Built-in `TextSegmenter` engine prevents breaking Markdown tables and code blocks.
* **KG-Ready Output:** Automatically generates `parent_id`, `doc_level`, and `semantic_type` metadata.
* **Smart Plugins:**
    * **MarkdownPlugin:** Anchors chunks to Headers (`#`) and classifies content (Table, List, H1...).
    * **ParentDocument:** Implements "Small-to-Big" retrieval strategy using composite splitters.
* **Semantic Awareness:** Detects topic shifts to create logically grouped chunks (Tier 2 Template).

## 📦 Installation

```bash
pip install sayou-chunking
```

## ⚡ Quickstart

The `ChunkingPipeline` orchestrates the splitting process. You can register any combination of Tier 2 Templates or Tier 3 Plugins.

Here is a complete example demonstrating how to process a Markdown file generated by `sayou-refinery`.

```python
import os
import json
from typing import List

# 1. Import Core Pipeline & Interface
from sayou.chunking.pipeline import ChunkingPipeline
from sayou.chunking.interfaces.base_splitter import BaseSplitter

# 2. Import Splitters (Templates & Plugins)
from sayou.chunking.splitter.fixed_length import FixedLengthSplitter
from sayou.chunking.splitter.recursive import RecursiveSplitter
from sayou.chunking.splitter.structure import StructureSplitter
from sayou.chunking.splitter.semantic import SemanticSplitter
from sayou.chunking.splitter.parent_document import ParentDocumentSplitter
from sayou.chunking.plugins.markdown_plugin import MarkdownPlugin

def run_chunking_demo():
    # Setup File Paths
    refinery_output_md = os.path.join(".", "test.md")

    with open(REFINERY_OUTPUT_MD, "r", encoding="utf-8") as f:
        markdown_content = f.read()

    source_metadata = {
        "source_file": refinery_output_md, 
        "id": "doc_refinery_output"
    }

    # 3. Register Splitters
    default_splitters: List[BaseSplitter] = [
        FixedLengthSplitter(),
        ParentDocumentSplitter(),
        RecursiveSplitter(),
        SemanticSplitter(),
        StructureSplitter(),
        MarkdownPlugin(),
    ]

    # 4. Initialize Pipeline
    pipeline = ChunkingPipeline(splitters=default_splitters)
    pipeline.initialize()

    # 5. Create Split Request
    # We use 'markdown' type to leverage structure-aware splitting
    split_request = {
        "type": "markdown",
        "content": markdown_content,
        "metadata": source_metadata,
        "chunk_size": 1000,
        "chunk_overlap": 50
    }

    print(f"--- [Sayou Chunking Demo] ---")
    print(f"Splitting using '{split_request['type']}'...")

    try:
        # 6. Run Splitting
        chunks = pipeline.split(split_request)
        print(f"✅ Successfully split content into {len(chunks)} chunks.\n")

        # 7. Save Output
        output_dir = os.path.join(os.path.dirname(__file__), "output")
        os.makedirs(output_dir, exist_ok=True)
        
        output_path = os.path.join(output_dir, f"chunks_output.json")
        with open(output_path, "w", encoding="utf-8") as f:
            json.dump(chunks, f, indent=2, ensure_ascii=False)
        
        print(f"Full output saved to {output_path}")

    except Exception as e:
        print(f"❌ Chunking failed: {e}")

if __name__ == "__main__":
    run_chunking_demo()
```

### Example JSON Output (KG-Ready)

Notice how `semantic_type` is identified and `parent_id` links the content to its header.

```json
[
    {
    "chunk_content": "# 1. Introduction",
    "metadata": {
        "chunk_id": "doc_123_h_0",
        "semantic_type": "h1",
        "is_header": true,
        "part_index": 0
    }
    },
    {
    "chunk_content": "| Feature | Status |\n|---|---|\n| Protect | Done |...",
    "metadata": {
        "chunk_id": "doc_123_part_1",
        "semantic_type": "table",
        "parent_id": "doc_123_h_0", 
        "section_title": "1. Introduction",
        "part_index": 1
    }
    }
]
```

## 🗺️ Roadmap (v0.1.0+)

`sayou-chunking` v0.0.1 establishes the structural foundation.

* **HTML Plugin:** Applying the "Parent-Child" strategy to HTML DOM trees.
* **Real Semantic Engine:** Integrating OpenAI/HuggingFace embeddings into `SemanticSplitter`.
* **Tokenizer Support:** Switching `chunk_size` calculation from characters to tokens (e.g., tiktoken).

## 🤝 Contributing

We welcome contributions! Whether it's a new Tier 3 Plugin for a specific format or optimization of the Tier 1 Engine. Please check our contributing guidelines.

## 📜 License

Apache 2.0 License © 2025 Sayouzone