# {{name}} - Evaluation Project

{{description}}

## Overview

This project provides a complete evaluation workflow for RAG assistants, including:
- Dataset creation with test questions
- Evaluation plan setup
- Result analysis and metrics

## Prerequisites

1. **PyGEAI installed and configured:**
   ```bash
   pip install pygeai
   geai configure --key YOUR_API_KEY --url YOUR_API_URL
   ```

2. **RAG assistant must exist:**
   Ensure the RAG assistant `{{assistant_name}}` is created and has documents uploaded.

## Workflow

### Step 1: Create Evaluation Dataset

Run the dataset creation script:

```bash
python create_dataset.py
```

This will:
- Create a new evaluation dataset
- Add sample test rows with questions and expected answers
- Output a Dataset ID

**Save the Dataset ID** - you'll need it in the next step.

### Step 2: Create Evaluation Plan

1. Open `create_evaluation_plan.py`
2. Replace `YOUR_DATASET_ID_HERE` with the Dataset ID from Step 1
3. Run the script:

```bash
python create_evaluation_plan.py
```

This creates an evaluation plan linking your dataset to the RAG assistant `{{assistant_name}}`.

**Save the Plan ID** - you'll need it in the next step.

### Step 3: Run Evaluation and View Results

1. Open `run_evaluation.py`
2. Replace `YOUR_PLAN_ID_HERE` with the Plan ID from Step 2
3. Run the script:

```bash
python run_evaluation.py
```

This will:
- List all evaluation results for the plan
- Show detailed metrics (cost, duration, status)
- Display row-level analysis

## Understanding Evaluation Metrics

Evaluation plans use system metrics to score responses:

- **Accuracy** (weight: 0.7): How correct the answer is
- **Fluency** (weight: 0.3): How natural the answer reads

You can customize metrics and weights in `create_evaluation_plan.py`.

## Customization

### Add More Test Rows

Edit `create_dataset.py` to add more test cases:

```python
row3 = client.create_dataset_row(
    dataset_id=dataset_id,
    row={
        "dataSetRowInput": "Your question here?",
        "dataSetRowExpectedAnswer": "Expected answer here",
        "dataSetRowContextDocument": "Context information"
    }
)
```

### Adjust Evaluation Metrics

Modify `create_evaluation_plan.py` to change metric weights:

```python
system_metrics=[
    {
        "systemMetricId": "accuracy",
        "systemMetricWeight": 0.5  # Changed from 0.7
    },
    {
        "systemMetricId": "fluency",
        "systemMetricWeight": 0.5  # Changed from 0.3
    }
]
```

### Add Expected Sources

For advanced evaluation, add expected source documents:

```python
from pygeai.evaluation.dataset.clients import EvaluationDatasetClient

client = EvaluationDatasetClient()

source = client.create_expected_source(
    dataset_id=dataset_id,
    dataset_row_id=row_id,
    expected_source_name="Source Document Title",
    expected_source_value="Expected source content...",
    expected_source_extension="pdf"
)
```

### Add Filter Variables

To test metadata filtering:

```python
filter_var = client.create_filter_variable(
    dataset_id=dataset_id,
    dataset_row_id=row_id,
    metadata_type="V",
    filter_variable_key="category",
    filter_variable_value="technical",
    filter_variable_operator="="
)
```

## Troubleshooting

**Dataset creation fails:**
Check your API credentials: `geai config show`

**Evaluation plan requires dataset ID:**
Make sure you updated `create_evaluation_plan.py` with the correct Dataset ID.

**No evaluation results found:**
Evaluation plans must be executed (typically through the UI or separate API call).

**Assistant not found:**
Verify the assistant name matches exactly: `geai assistants list`

## Next Steps

- Expand your dataset with more diverse test cases
- Compare evaluations across different assistant configurations
- Use evaluation-comparative template for A/B testing
- Export results for reporting and analysis

## Documentation

- [PyGEAI Evaluation Documentation](https://pygeai.readthedocs.io/)
- [Evaluation Dataset API](https://pygeai.readthedocs.io/en/latest/evaluation/dataset/)
- [Evaluation Plan API](https://pygeai.readthedocs.io/en/latest/evaluation/plan/)
- [Evaluation Result API](https://pygeai.readthedocs.io/en/latest/evaluation/result/)
