# Requirements Document

## Introduction

Данный документ описывает требования к доработке библиотеки **Chunkana** для достижения полной совместимости (parity) с плагином **dify-markdown-chunker** и создания самодостаточного руководства по миграции (MIGRATION_GUIDE.md), позволяющего механически перенести плагин на библиотеку с сохранением функциональности.

## Source of Truth

**Baseline Anchor:** Parity определяется относительно конкретного коммита плагина, зафиксированного в `BASELINE.md`.

**Golden Outputs:** Эталонные выходные данные генерируются из плагина на pinned commit и хранятся в:
- `tests/baseline/golden_canonical/` — канонический вывод (list[Chunk])
- `tests/baseline/golden_dify_style/` — view-level для include_metadata=True
- `tests/baseline/golden_no_metadata/` — view-level для include_metadata=False

**Generation Script:** `scripts/generate_baseline.py` — воспроизводимая генерация goldens из плагина.

## Glossary

- **Chunkana**: Standalone Python-библиотека для интеллектуального чанкинга Markdown-документов
- **Plugin**: Плагин dify-markdown-chunker для платформы Dify
- **Chunk**: Объект, представляющий сегмент Markdown-документа с контентом и метаданными
- **Renderer**: Функция форматирования чанков для различных выходных форматов
- **Canonical_Output**: Базовый вывод чанкера (list[Chunk]) без view-level трансформаций
- **View_Level_Output**: Отформатированный вывод после применения renderer'а
- **Parity**: Полное соответствие поведения и выходных данных между библиотекой и плагином
- **Golden_Output**: Эталонные выходные данные для baseline-тестирования, сгенерированные из плагина
- **ChunkerConfig**: Класс конфигурации чанкера в Chunkana
- **Overlap**: Контекстное перекрытие между соседними чанками (хранится в metadata)
- **Atomic_Block**: Неделимый блок (code fence, table, LaTeX) который не разрывается при чанкинге
- **JSONL**: JSON Lines формат — один JSON-объект на строку

## Requirements

### Requirement 1: Config Parity

**User Story:** As a plugin developer, I want ChunkerConfig in Chunkana to have all parameters from the plugin's ChunkConfig, so that I can migrate without losing configuration options.

#### Acceptance Criteria

**1.1 Plugin Parity Fields:**

THE ChunkerConfig SHALL contain all fields listed in `tests/baseline/plugin_config_keys.json`, generated from plugin's ChunkConfig.to_dict() at baseline commit.

**Source of truth:** `tests/baseline/plugin_config_keys.json` (auto-generated by `scripts/generate_baseline.py`)

Example fields (for reference only, actual list is in plugin_config_keys.json):
- Size parameters: `max_chunk_size`, `min_chunk_size`, `overlap_size`
- Behavior: `preserve_atomic_blocks`, `extract_preamble`
- Strategy thresholds: `code_threshold`, `structure_threshold`, etc.
- Code-context binding: `enable_code_context_binding`, etc.
- Computed: `enable_overlap` (computed property)

**1.2 Chunkana Extensions:**

THE ChunkerConfig MAY contain additional fields not present in plugin (i.e., fields NOT in `tests/baseline/plugin_config_keys.json`).

Example extensions (for reference):
- Overlap: `overlap_cap_ratio`
- Adaptive sizing: `use_adaptive_sizing`, `adaptive_config`
- Hierarchical: `include_document_summary`
- Preprocessing: `strip_obsidian_block_ids`
- LaTeX: `preserve_latex_blocks`, `latex_display_only`, `latex_max_context_chars`
- Table grouping: `group_related_tables`, `table_grouping_config`

**Note:** Extensions are defined as: `to_dict().keys() - plugin_config_keys.json`

**1.3 Serialization:**

THE ChunkerConfig.to_dict() method SHALL serialize:
- All plugin parity fields (1.1)
- All chunkana extension fields (1.2)
- Nested configs (adaptive_config, table_grouping_config) when present

Note: `streaming_config` is NOT part of ChunkerConfig — it's passed separately to streaming functions.

**1.4 Deserialization:**

THE ChunkerConfig.from_dict() method SHALL:
- Deserialize all fields including nested configs
- Use default values for missing fields
- Ignore unknown fields (forward compatibility)
- Handle `enable_overlap` as computed (remove from input, derive from overlap_size)

**1.5 Validation:**

THE ChunkerConfig SHALL validate all parameters in __post_init__ with clear error messages

**1.6 Round-trip:**

THE round-trip from_dict(to_dict(config)) SHALL produce equivalent config object (excluding computed `enable_overlap` which is derived)

### Requirement 2: API Wrappers

**User Story:** As a plugin developer, I want convenient API functions for common operations, so that I can simplify migration code.

#### Acceptance Criteria

1. THE Chunkana SHALL export `chunk_text(text, config=None)` function as an alias for `chunk_markdown`

2. THE Chunkana SHALL export `chunk_file(path, config=None, encoding="utf-8")` function that:
   - WHEN given a valid file path, SHALL read the file and return list[Chunk]
   - IF file does not exist, THEN SHALL raise FileNotFoundError with descriptive message
   - IF encoding error occurs, THEN SHALL raise UnicodeDecodeError

3. THE Chunkana SHALL export `chunk_file_streaming(path, config=None, streaming_config=None)` function that:
   - SHALL yield Chunk objects incrementally
   - SHALL respect StreamingConfig parameters
   - **Best-effort contract:** Streaming output MAY differ from batch output in:
     - Chunk boundaries near buffer splits
     - Overlap metadata at buffer boundaries
   - **Guaranteed invariants:**
     - All source lines SHALL be covered by at least one chunk
     - Atomic blocks SHALL NOT be split (when preserve_atomic_blocks=True)
     - start_line values SHALL be monotonically non-decreasing

4. THE Chunkana SHALL export `chunk_hierarchical(text, config=None)` function that:
   - SHALL return HierarchicalChunkingResult with tree structure
   - SHALL provide access to leaf chunks via `get_flat_chunks()`
   - SHALL provide parent/children navigation via `get_parent()`, `get_children()`

### Requirement 3: Renderer Parity

**User Story:** As a plugin developer, I want renderers that produce output identical to plugin's include_metadata modes, so that downstream systems receive the same format.

#### Acceptance Criteria

1. THE render_dify_style(chunks) function SHALL produce byte-for-byte identical output to plugin baseline golden set for include_metadata=True, as defined by `tests/baseline/golden_dify_style/*`
   - **Source of truth:** Golden outputs generated from plugin at pinned commit
   - JSON formatting SHALL match plugin exactly (indent, key order, ensure_ascii)

2. THE render_with_embedded_overlap(chunks) function SHALL produce byte-for-byte identical output to plugin baseline golden set for include_metadata=False, as defined by `tests/baseline/golden_no_metadata/*`
   - **Source of truth:** Golden outputs generated from plugin at pinned commit

3. THE render_with_prev_overlap(chunks) function SHALL produce sliding window format:
   - Format: `{previous_content}\n{content}` (if previous exists)
   - WHEN previous_content is empty, SHALL return only content

4. THE render_json(chunks) function SHALL return list[dict] with full chunk data via chunk.to_dict()

5. THE render_inline_metadata(chunks) function SHALL produce format with sorted keys for deterministic output
   - Note: This renderer is Chunkana-specific, not plugin parity

6. Renderer output newline behavior SHALL match view-level goldens; goldens are the source of truth for all formatting details including trailing newlines.

### Requirement 4: Public Exports

**User Story:** As a plugin developer, I want all necessary types and functions exported from chunkana package, so that I can import them directly.

#### Acceptance Criteria

**4.1 Minimal Stable API (for migration, stable contract):**

1. THE chunkana.__init__.py SHALL export core types:
   - `Chunk`, `ChunkConfig`, `ChunkerConfig` (alias)

2. THE chunkana.__init__.py SHALL export primary API functions:
   - `chunk_markdown`, `chunk_text`, `chunk_file`

3. THE chunkana.__init__.py SHALL export primary renderers:
   - `render_dify_style`, `render_with_embedded_overlap`

**4.2 Extended API (may change in minor versions):**

4. THE chunkana.__init__.py SHALL export extended types:
   - `ContentAnalysis`, `ChunkingResult`, `ChunkingMetrics`
   - `HierarchicalChunkingResult`

5. THE chunkana.__init__.py SHALL export extended API functions:
   - `chunk_file_streaming`, `chunk_hierarchical`
   - `analyze_markdown`, `chunk_with_analysis`, `chunk_with_metrics`, `iter_chunks`

6. THE chunkana.__init__.py SHALL export config classes:
   - `StreamingConfig`, `AdaptiveSizeConfig`, `TableGroupingConfig`

7. THE chunkana.__init__.py SHALL export extended renderers:
   - `render_with_prev_overlap`, `render_json`, `render_inline_metadata`

**4.3 Constraints:**

8. THE chunkana SHALL NOT import dify_plugin or any Dify SDK modules — it SHALL remain a pure Python library

### Requirement 5: MIGRATION_GUIDE.md v2

**User Story:** As a plugin developer, I want a comprehensive migration guide, so that I can mechanically migrate the plugin to Chunkana with full functionality preservation.

#### Acceptance Criteria

1. THE MIGRATION_GUIDE.md SHALL contain introduction section explaining:
   - What is being migrated (plugin → library)
   - Key changes (return type always list[Chunk], include_metadata → renderer selection)
   - Definition of "full compatibility" (canonical + view-level output match)

2. THE MIGRATION_GUIDE.md SHALL contain step-by-step migration procedure:
   - Changes to requirements.txt/pyproject.toml
   - Import replacements with before/after examples
   - Parameter mapping from Dify tool inputs to ChunkerConfig
   - Renderer selection based on include_metadata value

3. THE MIGRATION_GUIDE.md SHALL contain complete parameter mapping table with columns:
   - Plugin parameter name and type
   - Default value in plugin
   - Chunkana equivalent
   - Status (supported/partially/not supported)
   - Usage example

4. THE MIGRATION_GUIDE.md SHALL include mapping for every input parameter defined in plugin tool schema/manifest

5. THE MIGRATION_GUIDE.md SHALL contain advanced features recipes:
   - Hierarchy: tree access, leaf chunks, parent/children navigation
   - Streaming: large file processing, buffer configuration
   - Adaptive sizing: enabling, metrics in metadata
   - Table grouping: configuration, verification
   - Code-context binding: configuration, metadata fields
   - LaTeX: modes, enabling

6. THE MIGRATION_GUIDE.md SHALL contain migration verification checklist:
   - Baseline canonical tests
   - View-level golden tests
   - Property tests
   - Key fixture comparison (nested fences, lists, tables, latex)

7. THE MIGRATION_GUIDE.md SHALL contain "Not Guaranteed" section documenting any differences

### Requirement 6: Parity Matrix Documentation

**User Story:** As a plugin developer, I want a detailed parity matrix, so that I can verify all plugin features are supported.

#### Acceptance Criteria

1. THE docs/migration/parity_matrix.md SHALL list all ChunkConfig fields from plugin

2. FOR EACH field, THE parity_matrix.md SHALL specify:
   - Field name and type
   - Default value
   - Chunkana equivalent path
   - Support status: supported | partially | not_supported
   - Notes and examples

3. THE parity_matrix.md SHALL be organized by category:
   - Size parameters
   - Behavior parameters
   - Strategy thresholds
   - Code-context binding
   - Adaptive sizing
   - LaTeX handling
   - Table grouping
   - Streaming

### Requirement 7: Baseline Tests - Canonical

**User Story:** As a library maintainer, I want baseline tests for canonical output, so that refactoring doesn't break chunk boundaries.

#### Acceptance Criteria

1. THE baseline canonical tests SHALL verify for each chunk:
   - Full chunk.content (with CRLF→LF normalization, without strip)
   - start_line and end_line values
   - Full metadata dict (all keys and values)

2. THE baseline tests SHALL use golden output files in `tests/baseline/golden_canonical/`

3. THE golden files SHALL use JSONL format with schema:
   - `{"chunk_index": N, "content": "...", "start_line": N, "end_line": N, "metadata": {...}}`

4. WHEN baseline test fails, THE test output SHALL show exact diff between expected and actual

5. THE baseline tests SHALL cover fixtures:
   - Simple document
   - Nested code fences
   - Complex lists
   - Tables
   - LaTeX formulas
   - Mixed content
   - Code-context binding scenarios
   - Adaptive sizing scenarios
   - Table grouping scenarios

### Requirement 8: Baseline Tests - View Level

**User Story:** As a library maintainer, I want baseline tests for renderer output, so that view-level format is preserved.

#### Acceptance Criteria

1. THE view-level tests SHALL compare render_dify_style output with tests/baseline/golden_dify_style/*.jsonl

2. THE view-level tests SHALL compare render_with_embedded_overlap output with tests/baseline/golden_no_metadata/*.jsonl

3. THE golden files SHALL use JSONL format with schema:
   - `{"chunk_index": N, "text": "..."}`
   - where "text" is the full rendered string for that chunk

4. WHEN view-level test fails, THE test output SHALL show exact string diff

### Requirement 9: Property Tests

**User Story:** As a library maintainer, I want property-based tests for critical invariants, so that edge cases are caught.

#### Acceptance Criteria

1. THE property tests SHALL verify atomic block integrity for code fences:
   - FOR ALL documents with fenced code blocks, WHEN preserve_atomic_blocks=True, code blocks SHALL NOT be split across chunks

2. THE property tests SHALL verify atomic block integrity for tables:
   - FOR ALL documents with markdown tables, WHEN preserve_atomic_blocks=True, tables SHALL NOT be split across chunks

3. THE property tests SHALL verify atomic block integrity for LaTeX:
   - FOR ALL documents with LaTeX blocks, WHEN preserve_latex_blocks=True, LaTeX blocks SHALL NOT be split across chunks

4. THE property tests SHALL verify overlap cap ratio:
   - FOR ALL chunks with overlap, overlap size SHALL NOT exceed overlap_cap_ratio * adjacent_chunk_size

5. THE property tests SHALL verify monotonic start_line:
   - FOR ALL chunk sequences, start_line values SHALL be monotonically non-decreasing

6. THE property tests SHALL verify no empty chunks:
   - FOR ALL chunks, content.strip() SHALL NOT be empty

7. THE property tests SHALL verify line coverage (content preservation):
   - FOR ALL documents, every source line (1..N) SHALL be covered by at least one chunk's [start_line, end_line] range
   - Verification: `set(range(1, total_lines+1)) ⊆ union(range(c.start_line, c.end_line+1) for c in chunks)`

### Requirement 10: Config Parity Test

**User Story:** As a library maintainer, I want a test that verifies config field parity, so that new plugin fields are not missed.

#### Acceptance Criteria

1. THE test_config_parity.py SHALL load PLUGIN_CONFIG_KEYS from `tests/baseline/plugin_config_keys.json` (generated by generate_baseline.py from plugin's ChunkConfig.to_dict() at pinned commit)

2. THE test_config_parity.py SHALL compute CHUNKANA_EXTENSION_KEYS as: `ChunkerConfig.to_dict().keys() - PLUGIN_CONFIG_KEYS`

3. THE test SHALL verify ChunkerConfig.to_dict() contains all PLUGIN_CONFIG_KEYS

4. WHEN a plugin key is missing, THE test SHALL fail with clear message indicating missing field

5. THE test SHALL verify round-trip: from_dict(to_dict(config)) produces equivalent config (enable_overlap is computed, verified separately)

### Requirement 11: Documentation Updates

**User Story:** As a library user, I want updated documentation, so that I can understand all configuration options and renderers.

#### Acceptance Criteria

1. THE docs/config.md SHALL document all ChunkerConfig fields with:
   - Field name, type, default value
   - Description
   - Example usage

2. THE docs/renderers.md SHALL document renderer selection:
   - When to use each renderer
   - Output format examples
   - Dify integration guidance

3. THE docs/integrations/dify.md SHALL provide:
   - Minimal example of plugin migration
   - Parameter mapping quick reference
   - Common pitfalls

### Requirement 12: Invariants Preservation

**User Story:** As a library user, I want guaranteed invariants, so that I can rely on consistent behavior.

#### Acceptance Criteria

1. THE chunk_markdown() function SHALL always return list[Chunk], never a union type

2. THE Chunk.content SHALL contain canonical text without embedded overlap duplicates

3. THE overlap SHALL be stored only in metadata.previous_content and metadata.next_content

4. THE header_path SHALL be a string in format "/H1/H2/H3", not a list

5. WHEN preserve_atomic_blocks=True, THE Chunker SHALL NOT split fenced code blocks or tables

6. WHEN preserve_latex_blocks=True, THE Chunker SHALL NOT split LaTeX blocks

7. THE baseline tests SHALL protect these invariants; refactoring SHALL require green baseline first

### Requirement 13: Baseline Generation Infrastructure

**User Story:** As a library maintainer, I want reproducible baseline generation, so that goldens can be regenerated from plugin.

#### Acceptance Criteria

1. THE BASELINE.md SHALL pin plugin commit SHA used to generate goldens

2. THE repository SHALL include scripts/generate_baseline.py script that:
   - Generates canonical goldens from plugin at pinned commit
   - Generates view-level goldens (dify_style, no_metadata)
   - **Generates `tests/baseline/plugin_config_keys.json`** from plugin's ChunkConfig.to_dict().keys()
   - **Generates `tests/baseline/plugin_tool_params.json`** from plugin's tool schema/manifest
   - Outputs to agreed directory structure

3. THE generate_baseline.py SHALL accept --plugin-path argument to specify plugin location

4. THE generate_baseline.py SHALL produce deterministic output via:
   - Pinned plugin commit (environment determinism)
   - Pinned Python version and dependencies
   - **View-level goldens (golden_dify_style, golden_no_metadata) SHALL be saved as-is from plugin output WITHOUT any reformatting or key sorting**
   - Canonical goldens MAY use sorted keys for dict comparison in tests, but this is test implementation detail

5. THE generate_baseline.py SHALL NOT modify plugin output for view-level goldens — byte-for-byte preservation is required for renderer parity tests
