# RAGSentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking.

## Installation

### 1. Create Virtual Environment

```bash
# Create project directory
mkdir my-rag-eval
cd my-rag-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

### 2. Install Package

```bash
pip install rag-sentinel
```

## Quick Start

### 1. Initialize Project

```bash
rag-sentinel init
```

This creates:
- `.env` - LLM/Embeddings API keys
- `config.ini` - App settings and authentication
- `rag_eval_config.yaml` - Master configuration
- `test_dataset.csv` - Sample test dataset

### 2. Configure

Edit `.env`:
```env
AZURE_OPENAI_API_KEY=your-api-key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-02-15-preview
AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-4
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-ada-002
```

Edit `config.ini`:
```ini
[app]
app_url = https://your-rag-app.com/backend

[endpoints]
context_api_path = /api/retrieve_context
answer_api_path = /api/respond

[auth]
type = cookie
cookie_name = session
cookie_value = your-session-cookie

[mlflow]
tracking_uri = http://127.0.0.1:5001
```

### 3. Edit Test Dataset

Edit `test_dataset.csv`:
```csv
chat_id,query,ground_truth
1,What is RAG?,RAG stands for Retrieval-Augmented Generation
2,How does it work?,RAG combines retrieval and generation
```

### 4. Validate & Run

```bash
# Validate configuration
rag-sentinel validate

# Run evaluation
rag-sentinel run
```

Results will be available in the MLflow UI at the configured tracking URI.

## CLI Commands

```bash
# Initialize new project
rag-sentinel init

# Validate configuration
rag-sentinel validate

# Run evaluation (auto-starts MLflow)
rag-sentinel run

# Run without starting MLflow server
rag-sentinel run --no-server

# Overwrite existing config files
rag-sentinel init --force

# Check package version
pip show rag-sentinel

# Upgrade to latest version
pip install --upgrade rag-sentinel
```

## Metrics

- **Faithfulness** - Factual consistency of answer with context
- **Answer Relevancy** - How relevant the answer is to the question
- **Context Precision** - Quality of retrieved context
- **Answer Correctness** - Comparison against ground truth

## License

MIT

