# evalkit Examples

Each example builds a minimal RAG pipeline using a different framework, then
evaluates it with evalkit.

## Prerequisites

1. Set your OpenAI key:
   ```bash
   export OPENAI_API_KEY=sk-...
   ```

2. Install the base package with OpenAI support:
   ```bash
   pip install evalkit-llm[openai]
   ```

## Examples

### `openai_rag.py` — Raw OpenAI (no framework)

Minimal in-memory retriever + OpenAI chat completion. Good starting point.

```bash
python examples/openai_rag.py
```

### `langchain_rag.py` — LangChain + FAISS

Uses `RetrievalQA` with a FAISS in-memory vector store and OpenAI embeddings.

```bash
pip install langchain langchain-openai langchain-community faiss-cpu
python examples/langchain_rag.py
```

### `llamaindex_rag.py` — LlamaIndex

Uses `VectorStoreIndex` with OpenAI embeddings and a query engine.

```bash
pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai
python examples/llamaindex_rag.py
```

### `pydantic_ai_rag.py` — PydanticAI

Uses a typed PydanticAI agent with a `retrieve` tool for document lookup.

```bash
pip install pydantic-ai
python examples/pydantic_ai_rag.py
```

## What each example does

1. Defines a small corpus of documents (5–7 sentences)
2. Builds a retriever that selects the top-2 relevant documents per question
3. Generates answers using an LLM + retrieved context
4. Wraps results into `evalkit.TestCase` objects
5. Runs `EvaluationEngine` with `RAGQASuite` (faithfulness, answer relevancy, contextual relevancy, hallucination, completeness)
6. Prints the evaluation report with per-metric scores and fix suggestions
