# RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

## Installation

```bash
pip install rag-sentinel
```

## Quick Start

### 1. Initialize Project

```bash
mkdir my-rag-eval
cd my-rag-eval
rag-sentinel init
```

This creates:
- `.env` - LLM/Embeddings API keys
- `config.ini` - App settings and authentication
- `rag_eval_config.yaml` - Master configuration

### 2. Configure

Edit `.env`:
```bash
LLM_PROVIDER=azure
AZURE_LLM_API_KEY=your-api-key
AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com/
```

Edit `config.ini`:
```ini
[app]
app_url = https://your-rag-app.com/backend

[auth]
type = cookie
cookie_name = session
cookie_value = your-session-cookie
```

### 3. Create Test Dataset

Create `test_dataset.csv`:
```csv
query,ground_truth,chat_id
Hello,Hello! How can I help you?,1
```

### 4. Run Evaluation

```bash
rag-sentinel run
```

View results at: http://127.0.0.1:5001

## CLI Commands

```bash
rag-sentinel init              # Initialize project
rag-sentinel run               # Run evaluation (auto-starts MLflow)
rag-sentinel run --no-server   # Run without starting MLflow
rag-sentinel validate          # Validate configuration
```

## Metrics

- **Faithfulness** - Factual consistency of answer with context
- **Answer Relevancy** - How relevant the answer is to the question
- **Context Precision** - Quality of retrieved context
- **Answer Correctness** - Comparison against ground truth

## License

MIT

