# ChunkHive

**Hierarchical, semantic code chunking with embeddings for AI systems.**

ChunkHive is a production-grade code chunking engine designed for modern AI workflows such as
code embeddings, retrieval-augmented generation (RAG), agentic systems, and dataset synthesis.

It converts raw repositories into **clean, structured, semantically accurate chunks**
with byte-level precision and preserved hierarchy.

---

## 🚀 Why ChunkHive?

Modern AI systems need **more than naive text splitting**.

chunkhive provides:
- AST-first semantic correctness
- Hierarchical structure awareness
- Byte-accurate spans
- Robust parsing across real-world repositories

---

## 🧠 Core Principle

> **AST is the Authority, Tree-sitter is Enrichment**

- **Primary source of truth**: Language AST (semantic accuracy)
- **Fallback & enrichment**: Tree-sitter (structural robustness)
- **Result**: Maximum parsing success across diverse codebases

---

## ✨ Features

- Semantic AST-first chunking (no filename-based chunks)
- Preserves hierarchy: Module → Class → Method / Function
- Accurate parent–child relationships
- Byte-level precision (`start_byte`, `end_byte`)
- Clean symbol naming (`ast.name`)
- Import & decorator capture
- Robust handling of edge cases (empty files, `__init__.py`)
- Supports documentation + code chunking flows

## ✨ New Features

- 🌳 **Hierarchical Chunking** - Preserves code structure
- 🧠 **Semantic Embeddings** - Turn code into vectors (NEW in v0.3.1!)
- 🔍 **Vector Search** - Find code by meaning, not just names
- 🤖 **4 Pre-trained Models** - CodeBERT, UniXcoder, and more
- ⚡ **Fast & Scalable** - Process large codebases efficiently

### Semantic Search (NEW!)
```bash
# Generate embeddings
chunkhive embed generate chunks.jsonl -o chunks_embedded.jsonl

# Generate with model choice
chunkhive embed generate chunks.jsonl --model unixcoder -o embedded_unixcoder.jsonl

# Create index
chunkhive embed index data\embeddings\chunks_embedded -o chunks_embedded.jsonl

# search query :
chunkhive embed search-index "your query" --index data\embeddings\crewai_index -k 2

# List available models
chunkhive embed models

# Search semantically ( for debugging,testing etc)
chunkhive embed search "authentication function" --chunks chunks_embedded.jsonl

```

## What's New in v0.3.1

🎉 **Semantic Embeddings!**
- Generate vector embeddings for code chunks
- 4 pre-trained models (CodeBERT, UniXcoder, CodeSage, MiniLM)
- Fast similarity search with FAISS
- Natural language code search
---

## 🔄 Multi-Language Support

### Currently Supported:
Python: Full AST parsing with decorators, imports, docstrings

Markdown/RST: Documentation chunking with code block detection

Configuration Files: JSON, YAML, TOML, INI, Dockerfiles

Text Files: README, LICENSE, requirements.txt, scripts

### 🔄 Coming Soon:
JavaScript/TypeScript 

C++/Java/Go

## 🗂 Supported Chunk Types

module

class

method

function

documentation

configuration (JSON, YAML, TOML)

text

imports

## 🏢 Production Features
Deterministic IDs: Same code → same chunk ID across runs

Progress Indicators: Real-time processing feedback

Error Resilience: Graceful handling of malformed code

Statistics Generation: Detailed analytics and metrics

Batch Processing: Process multiple repositories from config file

Permission Handling: Intelligent output path resolution



## 📦 Installation

pip install chunkhive

## Quick Start

### Basic usage (creates in current directory)
chunkhive chunk local ./my_project

### With output directory
chunkhive chunk local ./my_project -o ./output

### With custom name and statistics
chunkhive chunk local ./my_project --name my_dataset --stats 

chunkhive chunk repo https://github.com/user/repo --name my_dataset --stats  

### Clone and chunk any GitHub repository
chunkhive chunk repo https://github.com/user/repo

### With filtering and limits
chunkhive chunk repo https://github.com/langchain-ai/langchain \
  --extensions .py,.md \
  --max-files 100 \
  --name langchain_chunks 

### Single File Processing
chunkhive chunk file example.py

chunkhive chunk file example.py -o ./chunks.jsonl --stats

## Repository Analysis

### Analyze repository metadata
chunkhive analyze https://github.com/crewAIInc/crewAI

chunkhive analyze ./local/repo --output analysis.json

### Show Examples

chunkhive examples

### Check Version & Info

chunkhive version    # Show current version

chunkhive info       # Show system information


## 📦 Output Schema (Simplified)

```json
{
  "chunk_id": "primary_a1b2c3d4",
  "file_path": "src/example.py",
  "chunk_type": "function",
  "language": "python",
  "code": "...",
  "ast": {
    "name": "my_function",
    "parent": "MyClass",
    "symbol_type": "function",
    "docstring": "Function documentation",
    "decorators": ["@decorator"],
    "imports": ["import module"]
  },
  "span": {
    "start_byte": 123,
    "end_byte": 456,
    "start_line": 10,
    "end_line": 25
  },
  "hierarchy": {
    "parent_id": "parent_chunk_id",
    "children_ids": ["child1", "child2"],
    "depth": 2,
    "is_primary": true
  },
  "metadata": {
    "byte_accuracy": "exact_bytes",
    "repo_info": {
      "agentic_detection": {"langchain": "usage"},
      "dependencies": {"python_packages": ["pandas", "numpy"]},
      "git": {"remote_url": "https://github.com/user/repo"},
      "structure": {"file_types": {".py": 50, ".md": 10}}
    },
    "repository_context": {
      "similar_files": ["src/other.py"],
      "total_similar_files": 5
    }
  }
}

```

## 🛠 Use Cases

Code embedding model training

RAG pipelines

Agentic AI systems

Code search & navigation

QA dataset generation

Static analysis & tooling

Enterprise codebase intelligence

AI training data generation


## 📜 License

Apache License 2.0 — free to use, modify, and distribute, including commercial use.


