# RAG module (`ketju.rag`)

Ketju’s RAG implementations live under `ketju.rag`. To keep the default install lightweight,
the actual backends are shipped as **optional extras**.

## Installing

```bash
pip install -e ".[rag]"
```

Docling-based variants additionally need:

```bash
pip install -e ".[docling]"
```

## Public API

Importing `ketju.rag` is safe even without extras, but accessing the concrete variants
requires installing the right extra.

### Variants

- **`SimplePdfRag`** (`ketju.rag.llama_index.variants.simple.SimplePdfRag`): LlamaIndex-based PDF ingestion.
- **`FullDocInCtxRag`** (`ketju.rag.llama_index.variants.full_doc_in_ctx.FullDocInCtxRag`): loads full PDF text into context.
- **`SimpleDoclingRag`** / **`SimpleDoclingFullCtxRag`**: Docling-backed variants (experimental).

## Storage layout

Each variant stores its state under `data/<variant>/` (configurable via `DATA_DIR`):

```
data/
└── <variant>/
    ├── cache/
    └── index/
```

### LlamaIndex Integration
```python
# Pre-configured settings
Settings.llm          # OpenAI LLM
Settings.embed_model  # OpenAI embeddings
```

## Ingestion

### Batch Ingestion
```python
# Ingest all PDFs in docs directory
rag.ingest_files()

# Limit number of files
rag.ingest_files(max_files=10)

# Force reprocessing
rag.ingest_files(force_reprocess=True)

# Specific files
from pathlib import Path
rag.ingest_files(files=[
    Path("docs/doc1.pdf"),
    Path("docs/doc2.pdf"),
])
```

### Ingest for Multiple Variants
```python
# Create variants
variants = [
    SimpleVectorRAG(variant_name="simple"),
    ChapterRAG(variant_name="chapter"),
]

# Ingest for all
for rag in variants:
    rag.ingest_files(max_files=5)
```

## Querying

### Basic Query
```python
response = rag.query("What are the main topics?")
print(response)
```

### With Parameters
```python
# SimpleVectorRAG
response = rag.query(
    "What are the main topics?",
    similarity_top_k=5,        # Number of chunks
    response_mode="compact",    # LlamaIndex mode
)

# ChapterRAG
response = rag.query(
    "What are the main topics?",
    similarity_top_k=3,         # Number of chapters
    return_full_text=True,      # Return full chapter text
)
```

### Comparison
```python
# Create variants
simple = SimpleVectorRAG(variant_name="simple")
chapter = ChapterRAG(variant_name="chapter")

# Query both
question = "What are the main topics?"

results = {
    "simple": simple.query(question, similarity_top_k=3),
    "chapter": chapter.query(question, similarity_top_k=3),
}

for name, response in results.items():
    print(f"\n{name}:")
    print(response)
```

## Statistics

```python
stats = rag.get_stats()
print(stats)
# {
#     "variant": "simple_vector",
#     "data_dir": "/path/to/data/simple_vector",
#     "docs_dir": "/path/to/docs",
#     "num_documents": 150,
# }
```

## Best Practices

### 1. Start Simple
Begin with `SimpleVectorRAG` to establish a baseline, then try more complex approaches.

### 2. Use Isolated Variants
Don't modify existing variants in place. Create new ones for experiments:
```python
manager.register("simple_v2", SimpleVectorRAG, "Experiment with X")
```

### 3. Cache Strategically
- Let caching work automatically for unchanged files
- Use `force_reprocess=True` when you change extraction logic
- Delete the variant's data directory to start fresh

### 4. Compare Systematically
```python
# Always use the same question for fair comparison
test_questions = [
    "What are the main topics?",
    "Explain concept X",
    "How does Y relate to Z?",
]

for q in test_questions:
    results = manager.compare_variants(["v1", "v2"], q)
    # Evaluate results
```

### 5. Monitor Performance
```python
import time

start = time.time()
response = rag.query(question)
elapsed = time.time() - start

print(f"Response time: {elapsed:.2f}s")
print(f"Stats: {rag.get_stats()}")
```

## Examples

See the `examples/` directory:
- `rag_comparison.py` - Complete comparison workflow
- `basic_usage.py` - Simple single-variant usage

## Troubleshooting

### "No relevant information found"
- Check that documents were ingested: `rag.get_stats()`
- Try increasing `similarity_top_k`
- Verify your query matches document content

### Files keep reprocessing
- Check if PDFs are being modified (timestamp changes)
- Verify cache directory permissions
- Check manifest files in cache directories

### Out of memory
- Reduce `max_files` during ingestion
- Use smaller embedding models
- Increase chunk size to reduce number of embeddings

### Slow queries
- Reduce `similarity_top_k`
- Use faster embedding models
- Consider caching query results

## Architecture Decisions

### Why isolated directories?
- Easy cleanup (delete one variant)
- No cross-contamination
- Clear disk usage per variant

### Why hash-based caching?
- Detect file changes automatically
- Avoid duplicate processing
- Fast existence checks

### Why metadata for ChapterRAG?
- Smaller embeddings (faster retrieval)
- Better semantic matching (summaries vs raw text)
- Richer context (key concepts, questions)

### Why LlamaIndex?
- Production-ready
- Rich ecosystem
- Flexible architecture
- Good documentation

## Future Variants

Potential RAG variants to implement:

1. **HybridRAG** - Combine vector + keyword search
2. **HierarchicalRAG** - Multi-level (section → chapter → document)
3. **GraphRAG** - Knowledge graph relationships
4. **TemporalRAG** - Track document versions over time
5. **MultiModalRAG** - Handle images, tables, etc.

To implement, inherit from `BaseRAG` and register with the manager!
