# Welcome to GAICo

![quickstart](https://raw.githubusercontent.com/ai4society/GenAIResultsComparator/refs/heads/main/quickstart.gif)

The library provides a set of metrics for evaluating **2 text strings as inputs**. **Outputs are on a scale of 0 to 1** (normalized), where 1 indicates a perfect match between the two texts.

**_Class Structure:_** All metrics are implemented as classes, and they can be easily extended to add new metrics. The metrics start with the `BaseMetric` class under the `gaico/base.py` file.

Each metric class inherits from this base class and is implemented with **just one required method**: `calculate()`.

This `calculate()` method takes two parameters:

- `generated_texts`: Either a string or a Iterables of strings representing the texts generated by an LLM.
- `reference_texts`: Either a string or a Iterables of strings representing the expected or reference texts.

If the inputs are Iterables (lists, Numpy arrays, etc.), then the method assumes that there exists a one-to-one mapping between the generated texts and reference texts, meaning that the first generated text corresponds to the first reference text, and so on.

**_Note:_** While the library can be used to compare strings, it's main purpose is to aid with comparing results from various LLMs.

**_Inspiration_** for the library and evaluation metrics was taken from [Microsoft's
article on evaluating LLM-generated content](https://learn.microsoft.com/en-us/ai/playbook/technology-guidance/generative-ai/working-with-llms/evaluation/list-of-eval-metrics). In the article, Microsoft describes 3 categories of evaluation metrics: **(1)** Reference-based metrics, **(2)** Reference-free metrics, and **(3)** LLM-based metrics. _The library currently supports reference-based metrics._

<figure markdown="span">
  ![GAICo Overview](https://raw.githubusercontent.com/ai4society/GenAIResultsComparator/refs/heads/main/gaico.drawio.png){ width="500" }
  <figcaption><em>Overview of the workflow supported by the <i>GAICo</i> library</em></figcaption>
</figure>

## Code Style

We use `pre-commit` hooks to maintain code quality and consistency. The configuration for these hooks is in the `.pre-commit-config.yaml` file. These hooks run automatically on `git commit`, but you can also run them manually:

```
pre-commit run --all-files
```

## Running Tests

Navigate to the project root in your terminal and run:

```bash
uv run pytest
```

Or, for more verbose output:

```bash
uv run pytest -v
```

To skip the slow BERTScore tests:

```bash
uv run pytest -m "not bertscore"
```

To run only the slow BERTScore tests:

```bash
uv run pytest -m bertscore
```

## Citation

If you find this project useful, please consider citing it in your work:

```bibtex
@software{AI4Society_GAICo_GenAI_Results,
  author = {{AI4Society}},
  license = {MIT},
  title = {{GAICo: GenAI Results Comparator}},
  url = {https://github.com/ai4society/GenAIResultsComparator}
}
```

## Acknowledgments

- The library is developed by [Nitin Gupta](https://github.com/g-nitin), [Pallav Koppisetti](https://github.com/pallavkoppisetti), and [Biplav Srivastava](https://github.com/biplav-s). Members of [AI4Society](https://ai4society.github.io) contributed to this tool as part of ongoing discussions. Major contributors are credited.
- This library uses several open-source packages including NLTK, scikit-learn, and others. Special thanks to the creators and maintainers of the implemented metrics.

## Contact

If you have any questions, feel free to reach out to us at [ai4societyteam@gmail.com](mailto:ai4societyteam@gmail.com).
