# RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

## Installation

### 1. Create Virtual Environment

```bash
# Create project directory
mkdir my-rag-eval
cd my-rag-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

### 2. Install Package

```bash
pip install rag-sentinel
```

## Quick Start

### 1. Initialize Project

```bash
rag-sentinel init
```

This creates:
- `.env` - LLM/Embeddings API keys
- `config.ini` - Backend API endpoint and authentication settings
- `test_dataset.csv` - Sample test dataset

**Note:** The master configuration (`rag_eval_config.yaml`) is managed internally by the package. You only need to configure `.env` and `config.ini`.

### 2. Configure

#### A. Edit `.env` file
Configure your LLM and Embeddings providers:

```bash
# LLM Provider (azure, openai, or ollama)
LLM_PROVIDER=azure

# Azure OpenAI LLM Configuration
AZURE_LLM_API_KEY=your-api-key
AZURE_LLM_ENDPOINT=https://your-resource.openai.azure.com
AZURE_LLM_DEPLOYMENT_NAME=gpt-4
AZURE_LLM_MODEL=gpt-4
AZURE_LLM_TEMPERATURE=0.0
AZURE_LLM_API_VERSION=2024-02-15-preview

# Embeddings Provider (azure, openai, or ollama)
EMBEDDINGS_PROVIDER=azure

# Azure OpenAI Embeddings Configuration
AZURE_EMBEDDINGS_API_KEY=your-api-key
AZURE_EMBEDDINGS_ENDPOINT=https://your-resource.openai.azure.com
AZURE_EMBEDDINGS_DEPLOYMENT_NAME=text-embedding-ada-002
AZURE_EMBEDDINGS_API_VERSION=2024-02-15-preview
```

#### B. Edit `config.ini` file
Configure your RAG backend API and evaluation settings:

**MLflow Settings:**
```ini
[mlflow]
tracking_uri = http://127.0.0.1:5000
experiment_name = RAG Evaluation
run_name = RAG Evaluation Run
```

**Backend API Endpoint:**
```ini
[endpoints]
# Your complete RAG API endpoint
response_api_path = https://your-app.com/api/respond

# HTTP method (POST or GET)
method = POST

# Request headers (comma-separated key:value pairs)
headers = Content-Type:application/json

# Request body fields (use {query} and {chat_id} as placeholders)
body_fields = prompt:{query},chat_id:{chat_id}

# JSON response keys
answer_key = response
contexts_key = context
```

**Authentication:**
```ini
[auth]
# Options: cookie, bearer, header, or none
type = cookie
cookie_name = session
cookie_value = your-session-cookie-value
```

**Dataset:**
```ini
[dataset]
path = test_dataset.csv
category = simple  # Options: simple (RAGAS) or guardrail (security)
```

#### C. Edit `test_dataset.csv` file
Add your test queries in the format: `query,ground_truth,chat_id`

Example:
```csv
query,ground_truth,chat_id
What is RAG?,RAG stands for Retrieval-Augmented Generation,1
How does vector search work?,Vector search finds similar items using embeddings,2
```

For detailed configuration help, see the comments in each config file.

### 3. Validate & Run

```bash
# Validate configuration
rag-sentinel validate

# Run evaluation
rag-sentinel run
```

Results will be available in the MLflow UI at the configured tracking URI.

## CLI Commands

```bash
# Initialize new project
rag-sentinel init

# Validate configuration
rag-sentinel validate

# Run evaluation (auto-starts MLflow)
rag-sentinel run

# Run without starting MLflow server
rag-sentinel run --no-server

# Overwrite existing config files
rag-sentinel init --force

# Check package version
pip show rag-sentinel

# Upgrade to latest version
pip install --upgrade rag-sentinel
```

## Evaluation Categories

Set `category` in `config.ini` to choose evaluation type:

### Simple (RAGAS Quality Metrics)
```ini
category = simple
```
- **Faithfulness** - Factual consistency of answer with context
- **Answer Relevancy** - How relevant the answer is to the question
- **Context Precision** - Quality of retrieved context
- **Answer Correctness** - Comparison against ground truth

### Guardrail (Security Metrics)
```ini
category = guardrail
```
- **Toxicity Score** - Detects toxic content in responses
- **Bias Score** - Detects biased content in responses

## Performance Metrics

Logged for all evaluation categories:
- **Avg Response Time** - Average API response time (ms)
- **P90 Latency** - 90th percentile latency (ms)
- **Queries Per Second** - Throughput (QPS)

## License

MIT

