# Krira Augment ⚡🦀

**The High-Performance Rust Chunking Engine for RAG Pipelines**

[![PyPI version](https://badge.fury.io/py/krira-augment.svg)](https://badge.fury.io/py/krira-augment)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Rust](https://img.shields.io/badge/Built_with-Rust-orange)](https://www.rust-lang.org/)

**Krira Augment** is a production-grade Python library backed by a highly optimized Rust core. It is designed to replace slow, memory-intensive preprocessing steps in large-scale Retrieval Augmented Generation (RAG) systems.

It processes gigabytes of raw unstructured data (CSV, JSONL, TXT) into high-quality, clean chunks in seconds—utilizing **zero-copy memory mapping** and **parallel CPU execution**.

---

## 🚀 Performance Benchmarks

Benchmarks run on a standard 8-core machine (M2 Air equivalent).

| Dataset Size | Legacy (LangChain/Pandas) | Krira V2 (Rust Core) | Speedup |
| :--- | :--- | :--- | :--- |
| **100 MB** | ~45 sec | **~0.8 sec** | **56x** 🚀 |
| **1 GB** | ~8.0 min | **~12.0 sec** | **40x** 🚀 |
| **10 GB** | *Timeout / OOM* | **~2.1 min** | **Stable** ✅ |

> **Note:** Krira uses O(1) memory. Processing a 100GB file uses the same amount of RAM as a 10MB file.

---

## 📦 Installation

```bash
pip install krira-augment
```

*Requirements: Python 3.8+*

---

## 🛠️ Usage

### 1. Quick Start
For standard use cases, use the default high-throughput pipeline.

```python
from krira_augment import Pipeline

# Initialize the pipeline
pipeline = Pipeline()

# Process a 1GB file in seconds
stats = pipeline.process(
    input_path="data/raw_knowledge_base.csv",
    output_path="data/processed_chunks.jsonl"
)

print(f"✅ Processing complete chunking job.")
```

### 2. Advanced Configuration (Professional)
For production RAG, you need fine-grained control over chunking strategies, overlap, and data cleaning.

```python
from krira_augment import Pipeline, PipelineConfig, SplitStrategy

# Define a robust configuration
config = PipelineConfig(
    # Chunking Strategy
    chunk_size=512,             # Target characters per chunk
    chunk_overlap=50,           # Context overlap for better retrieval
    strategy=SplitStrategy.SMART, # Respects sentence/paragraph boundaries
    
    # Data Cleaning Rules (Rust-native regex)
    clean_html=True,            # Remove <div>, <br>, etc.
    clean_unicode=True,         # Normalize whitespace and emojis
    min_chunk_len=20,           # Discard garbage/empty chunks
    
    # System Performance
    threads=8,                  # Force usage of 8 CPU cores
    batch_size=1000             # Write to disk every 1k chunks (Low RAM usage)
)

# Initialize with config
pipeline = Pipeline(config=config)

# Execute
result = pipeline.process(
    input_path="large_corpus.csv", 
    output_path="corpus_vectors.jsonl"
)

print(f"Job ID: {result.job_id}")
print(f"Throughput: {result.mb_per_second:.2f} MB/s")
```

---

## 📄 Output Format

The library outputs standard **JSONL** (JSON Lines), ready for direct ingestion into vector databases (Pinecone, Weaviate, Qdrant).

**`processed_chunks.jsonl`**:
```json
{"text": "The mitochondria is the powerhouse...", "metadata": {"source": "doc1.csv", "row": 1, "chunk_index": 0}}
{"text": "It generates most of the chemical energy...", "metadata": {"source": "doc1.csv", "row": 1, "chunk_index": 1}}
```

---

## 🏗️ Architecture

Krira differs from standard Python loaders by offloading the entire ETL process to a compiled Rust binary.

1.  **Memory Mapping (mmap):** The file is mapped directly from disk to virtual memory. No loading 1GB CSVs into Python RAM.
2.  **Rayon Parallelism:** The file is sliced into segments and processed across all available CPU cores simultaneously.
3.  **Serde Serialization:** Chunks are serialized to JSONL directly on the Rust thread, minimizing Python GIL interaction.

---

## 🤝 Integration Example

Seamlessly integrate with generic Python generators to feed embeddings.

```python
import json
import openai

def stream_chunks(jsonl_path):
    """Yields chunks efficiently for embedding API calls."""
    with open(jsonl_path, 'r') as f:
        for line in f:
            yield json.loads(line)

# Use in your downstream application
for chunk in stream_chunks("processed_chunks.jsonl"):
    # Mock embedding call
    # embedding = openai.Embedding.create(input=chunk['text'])
    pass
    
    # Upsert to Vector DB (e.g., Pinecone)
    # index.upsert(vectors=[(chunk['id'], embedding, chunk['metadata'])])
```

---

## 🧑‍💻 Development

If you want to modify the Rust core:

1.  **Clone the repo**
2.  **Install Maturin** (Rust-Python bridge builder)
    ```bash
    pip install maturin
    ```
3.  **Build and Install locally**
    ```bash
    maturin develop --release
    ```

---

## License

MIT License.
