---
phase: 15-core-ast-chunker
plan: 02
type: execute
wave: 2
depends_on:
  - 15-01
files_modified:
  - pyproject.toml
  - src/corpus_analyzer/ingest/ts_chunker.py
  - src/corpus_analyzer/ingest/chunker.py
autonomous: true
requirements:
  - DEP-01
  - IDX-01
  - IDX-02
  - IDX-03
  - IDX-04
  - IDX-05
  - IDX-06
  - IDX-07

must_haves:
  truths:
    - "uv sync succeeds after adding tree-sitter and tree-sitter-language-pack with no C toolchain"
    - "chunk_typescript() extracts all 8 target node types as separate chunks with correct 1-indexed line boundaries"
    - "export function foo() and export class Bar produce chunks with full exported text"
    - ".tsx and .jsx use tsx grammar; .ts uses typescript grammar; .js uses javascript grammar"
    - "has_error partial tree does NOT fall back; uncaught exception DOES fall back"
    - "chunk_name field present in every returned chunk dict"
    - "_get_cached_parser uses @lru_cache to load grammar binaries once per dialect"
    - "chunk_file() routes .ts/.tsx/.js/.jsx to chunk_typescript()"
    - "All TestChunkTypeScript and TestChunkFile tests pass GREEN"
  artifacts:
    - path: "src/corpus_analyzer/ingest/ts_chunker.py"
      provides: "chunk_typescript function + _get_cached_parser + _DIALECT + _TARGET_TYPES + _extract_name"
      exports:
        - "chunk_typescript"
    - path: "src/corpus_analyzer/ingest/chunker.py"
      provides: "Updated chunk_file dispatch for .ts/.tsx/.js/.jsx"
      contains: "chunk_typescript"
    - path: "pyproject.toml"
      provides: "tree-sitter and tree-sitter-language-pack dependencies + mypy overrides"
      contains: "tree-sitter>=0.25.0"
  key_links:
    - from: "src/corpus_analyzer/ingest/chunker.py"
      to: "src/corpus_analyzer/ingest/ts_chunker.py"
      via: "lazy import in chunk_file elif branch"
      pattern: "from corpus_analyzer.ingest.ts_chunker import chunk_typescript"
    - from: "src/corpus_analyzer/ingest/ts_chunker.py"
      to: "tree_sitter_language_pack"
      via: "_get_cached_parser(dialect)"
      pattern: "get_parser\\(dialect\\)"
---

<objective>
Implement chunk_typescript() to turn the RED tests from Plan 01 GREEN.

Purpose: AST-aware chunking for .ts/.tsx/.js/.jsx files that extracts named top-level constructs with correct line boundaries and chunk_name fields.
Output: ingest/ts_chunker.py (~110 lines), updated chunker.py dispatch, pyproject.toml with deps, all 21 TestChunkTypeScript tests passing.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/ROADMAP.md
@.planning/phases/15-core-ast-chunker/15-CONTEXT.md
@.planning/phases/15-core-ast-chunker/15-RESEARCH.md
@.planning/phases/15-core-ast-chunker/15-01-SUMMARY.md
@src/corpus_analyzer/ingest/chunker.py
@tests/ingest/test_chunker.py
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add deps to pyproject.toml and run uv sync</name>
  <files>pyproject.toml</files>
  <action>
    1. In the [project] dependencies list, add after "pandas>=2.0.0,":
         "tree-sitter>=0.25.0",
         "tree-sitter-language-pack>=0.13.0",

    2. Add a new [[tool.mypy.overrides]] block after the existing frontmatter override:
         [[tool.mypy.overrides]]
         module = ["tree_sitter", "tree_sitter_language_pack"]
         ignore_missing_imports = true

    3. Run: cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv sync

    4. Verify import works: uv run python -c "from tree_sitter_language_pack import get_parser; p = get_parser('typescript'); print('OK')"
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv sync && uv run python -c "from tree_sitter_language_pack import get_parser; p = get_parser('typescript'); print('OK')"</automated>
    <sampling_rate>Run before Task 2 — implementation depends on packages being installed</sampling_rate>
  </verify>
  <done>
    - pyproject.toml has tree-sitter>=0.25.0 and tree-sitter-language-pack>=0.13.0 in dependencies
    - pyproject.toml has [[tool.mypy.overrides]] for tree_sitter and tree_sitter_language_pack
    - uv sync exits 0 with no C toolchain error
    - python -c import succeeds and prints "OK"
  </done>
</task>

<task type="auto">
  <name>Task 2: Implement ts_chunker.py and wire dispatch in chunker.py</name>
  <files>
    src/corpus_analyzer/ingest/ts_chunker.py
    src/corpus_analyzer/ingest/chunker.py
  </files>
  <action>
    CREATE src/corpus_analyzer/ingest/ts_chunker.py:

    ```python
    """TypeScript/JavaScript AST-aware chunker using tree-sitter.

    Provides chunk_typescript() which splits .ts, .tsx, .js, and .jsx files
    into chunks by top-level named constructs, mirroring chunk_python() in style
    and return format.
    """

    from __future__ import annotations

    from functools import lru_cache
    from pathlib import Path
    from typing import Any

    from tree_sitter import Parser  # type: ignore[import-untyped]
    from tree_sitter_language_pack import get_parser  # type: ignore[import-untyped]

    _DIALECT: dict[str, str] = {
        ".ts": "typescript",
        ".tsx": "tsx",
        ".js": "javascript",
        ".jsx": "tsx",
    }

    _TARGET_TYPES: frozenset[str] = frozenset({
        "function_declaration",
        "generator_function_declaration",
        "class_declaration",
        "abstract_class_declaration",
        "interface_declaration",
        "type_alias_declaration",
        "lexical_declaration",
        "enum_declaration",
    })


    @lru_cache(maxsize=8)
    def _get_cached_parser(dialect: str) -> Parser:
        """Return a cached tree-sitter Parser for the given grammar dialect.

        Args:
            dialect: One of "typescript", "tsx", or "javascript".

        Returns:
            Configured tree_sitter.Parser instance (loaded once per dialect per process).
        """
        return get_parser(dialect)


    def _extract_name(node: Any, export_node: Any | None) -> str:
        """Extract the human-readable name from a declaration AST node.

        Args:
            node: The declaration node (after export unwrapping).
            export_node: The export_statement node if this was an export, else None.

        Returns:
            Name string for the chunk_name field. Falls back to "<anonymous>".
        """
        # Check for export default — name is always "default"
        if export_node is not None:
            for child in export_node.children:
                if child.type == "default":
                    return "default"

        if node.type == "lexical_declaration":
            # No direct name field; first variable_declarator holds the identifier
            for child in node.children:
                if child.type == "variable_declarator":
                    name_node = child.child_by_field_name("name")
                    if name_node is not None and name_node.text is not None:
                        return name_node.text.decode("utf-8")
            return "<anonymous>"

        # All other target types (function, class, interface, type alias, enum, generator)
        # expose a "name" field directly
        name_node = node.child_by_field_name("name")
        if name_node is not None and name_node.text is not None:
            return name_node.text.decode("utf-8")
        return "<anonymous>"


    def chunk_typescript(path: Path) -> list[dict[str, Any]]:
        """Split a TypeScript/JavaScript file into chunks by top-level constructs.

        Handles .ts, .tsx, .js, and .jsx files using the appropriate tree-sitter
        grammar. Falls back to chunk_lines() if parsing fails or no named constructs
        are found. Does NOT fall back on root_node.has_error alone — tree-sitter
        produces useful partial trees.

        JSDoc blocks (/** ... */) immediately preceding a construct (no intervening
        blank lines) are included in the chunk text. Single-line // comments are not.

        Args:
            path: Path to the TypeScript/JavaScript source file.

        Returns:
            List of chunk dicts with "text", "start_line", "end_line", and
            "chunk_name" keys. start_line and end_line are 1-indexed, inclusive.
        """
        from corpus_analyzer.ingest.chunker import chunk_lines  # lazy: avoid circular

        ext = path.suffix.lower()
        dialect = _DIALECT.get(ext)
        if dialect is None:
            return chunk_lines(path)

        try:
            with open(path, encoding="utf-8") as f:
                source = f.read()
        except (UnicodeDecodeError, OSError):
            return chunk_lines(path)

        if not source.strip():
            return []

        try:
            parser = _get_cached_parser(dialect)
            tree = parser.parse(source.encode("utf-8"))
        except Exception:
            return chunk_lines(path)

        root_node = tree.root_node
        lines = source.splitlines()
        chunks: list[dict[str, Any]] = []

        for child in root_node.children:
            export_node: Any | None = None
            node = child

            if child.type == "export_statement":
                inner = child.child_by_field_name("declaration")
                if inner is None:
                    # Re-export or export default expression — no declaration to extract
                    continue
                export_node = child
                node = inner

            if node.type not in _TARGET_TYPES:
                continue

            # outer is the export wrapper (if present) or the declaration itself.
            # Line range for the emitted chunk is always outer's span.
            outer = export_node if export_node is not None else node
            chunk_start_row = outer.start_point[0]  # 0-indexed row

            # JSDoc lookback: include immediately preceding /** ... */ block comment.
            # "Immediately preceding" means the comment ends on the row just before
            # outer starts (at most one row gap, to allow for tree-sitter whitespace nodes).
            prev = outer.prev_sibling
            # prev_sibling may return a whitespace/newline token; skip those
            if prev is not None and prev.type not in ("comment",):
                prev = prev.prev_sibling  # walk one more step past whitespace
            if prev is not None and prev.type == "comment":
                comment_bytes = prev.text
                if comment_bytes is not None:
                    comment_text = comment_bytes.decode("utf-8")
                    if comment_text.startswith("/**"):
                        # Adjacency check: comment ends on the row immediately before outer
                        if outer.start_point[0] - prev.end_point[0] <= 1:
                            chunk_start_row = prev.start_point[0]

            start_line = chunk_start_row + 1  # convert 0-indexed row → 1-indexed line
            end_line = outer.end_point[0] + 1  # inclusive, 1-indexed

            chunk_text = "\n".join(lines[chunk_start_row : outer.end_point[0] + 1]).rstrip()
            chunk_name = _extract_name(node, export_node)

            if chunk_text:
                chunks.append({
                    "text": chunk_text,
                    "start_line": start_line,
                    "end_line": end_line,
                    "chunk_name": chunk_name,
                })

        if not chunks:
            return chunk_lines(path)

        return chunks
    ```

    UPDATE src/corpus_analyzer/ingest/chunker.py — in chunk_file(), replace the elif branch:

    Old line:
        elif ext in (".ts", ".js", ".json", ".yaml", ".yml", ".txt", ".toml"):
            chunks = chunk_lines(path)

    New lines:
        elif ext in (".ts", ".tsx", ".js", ".jsx"):
            from corpus_analyzer.ingest.ts_chunker import chunk_typescript  # noqa: PLC0415
            chunks = chunk_typescript(path)
        elif ext in (".json", ".yaml", ".yml", ".txt", ".toml"):
            chunks = chunk_lines(path)

    IMPORTANT notes for implementation:
    - The noqa comment is not needed if ruff doesn't flag PLC0415 (it's not in the project's selected rules). Omit it.
    - The lazy import inside chunk_file avoids circular imports (ts_chunker imports from chunker).
    - Do NOT add chunk_typescript to the chunker.py module-level imports.
    - Leave all other chunker.py code unchanged.
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/ingest/test_chunker.py -v</automated>
    <sampling_rate>Run after this task; all TestChunkTypeScript and TestChunkFile tests must be GREEN</sampling_rate>
  </verify>
  <done>
    - src/corpus_analyzer/ingest/ts_chunker.py exists with chunk_typescript, _get_cached_parser, _extract_name, _DIALECT, _TARGET_TYPES
    - chunk_file() in chunker.py routes .ts/.tsx/.js/.jsx to chunk_typescript via lazy import
    - uv run pytest tests/ingest/test_chunker.py -v shows all TestChunkTypeScript and TestChunkFile tests GREEN
    - uv run pytest -v (full suite) shows 293+ tests passing
    - Commit: feat(15-02): implement chunk_typescript with tree-sitter AST chunking
  </done>
</task>

</tasks>

<verification>
Full suite green:
  uv run pytest -v → all 293+ tests pass

Specific new tests:
  uv run pytest tests/ingest/test_chunker.py::TestChunkTypeScript -v → all 21 pass
  uv run pytest tests/ingest/test_chunker.py::TestChunkFile -v → all dispatch tests pass

Dep install verified:
  uv run python -c "from tree_sitter_language_pack import get_parser; get_parser('typescript')" → no error

Requirements coverage:
  DEP-01: uv sync succeeded, wheels pre-compiled (no C toolchain)
  IDX-01: chunk_file routes .ts/.tsx/.js/.jsx → chunk_typescript (TestChunkFile)
  IDX-02: 8 node types extracted (TestChunkTypeScript test_*_extraction methods)
  IDX-03: export unwrapping produces full text (test_export_function_unwrapping, test_export_class_unwrapping)
  IDX-04: grammar dispatch by extension (_DIALECT map, test_jsx_in_tsx_parses, test_jsx_in_jsx_parses)
  IDX-05: has_error no fallback (test_has_error_does_not_fall_back); exception falls back (test_catastrophic_failure_falls_back)
  IDX-06: chunk_name in every chunk (test_chunk_name_field_present)
  IDX-07: @lru_cache on _get_cached_parser (test_parser_cache)
</verification>

<success_criteria>
- uv run pytest -v passes all 293+ existing tests plus 21 new TestChunkTypeScript tests
- uv run pytest tests/ingest/test_chunker.py::TestChunkFile -v passes all 4 new dispatch tests
- ts_chunker.py is ~110 lines, well-typed, with PEP 257 docstrings
- chunk_file() dispatch updated for all 4 extensions with lazy import
- pyproject.toml has both tree-sitter deps and mypy overrides
- Phase 15 requirements DEP-01, IDX-01 through IDX-07 all verified GREEN
</success_criteria>

<output>
After completion, create `.planning/phases/15-core-ast-chunker/15-02-SUMMARY.md`
</output>
