# Welcome to GAICo

<figure markdown="span">
  <img src="https://raw.githubusercontent.com/ai4society/GenAIResultsComparator/refs/heads/main/quickstart.gif" alt="GIF Showing GAICo's Quickstart" style="display: block; margin: auto; border-radius: 10px; box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);">
  <figcaption><em>GAICo Quickstart Demonstration</em></figcaption>
</figure>

_GenAI Results Comparator (GAICo) is a Python library_ to help compare, analyze and visualize outputs from Large Language Models (LLMs), often against a reference text.

At the core, the library provides a set of metrics for evaluating text strings given as inputs and produce outputs on a scale of 0 to 1 (normalized), where 1 indicates a perfect match between the two texts. These scores are use to analyze LLM outputs as well as visualize.

**_Class Structure:_** All metrics are implemented as classes, and they can be easily extended to add new metrics. The metrics start with the `BaseMetric` class under the `gaico/base.py` file.

Each metric class inherits from this base class and is implemented with **just one required method**: `calculate()`.

This `calculate()` method takes two parameters:

- `generated_texts`: Either a string or a Iterables of strings representing the texts generated by an LLM.
- `reference_texts`: Either a string or a Iterables of strings representing the expected or reference texts.

If the inputs are Iterables (lists, Numpy arrays, etc.), then the method assumes that there exists a one-to-one mapping between the generated texts and reference texts, meaning that the first generated text corresponds to the first reference text, and so on.

**_Note:_** While the library can be used to compare strings, it's main purpose is to aid with comparing results from various LLMs.

**_Inspiration_** for the library and evaluation metrics was taken from [Microsoft's
article on evaluating LLM-generated content](https://learn.microsoft.com/en-us/ai/playbook/technology-guidance/generative-ai/working-with-llms/evaluation/list-of-eval-metrics). In the article, Microsoft describes 3 categories of evaluation metrics: **(1)** Reference-based metrics, **(2)** Reference-free metrics, and **(3)** LLM-based metrics. _The library currently supports reference-based metrics._

<figure markdown="span">
  ![GAICo Overview](https://raw.githubusercontent.com/ai4society/GenAIResultsComparator/refs/heads/main/gaico.drawio.png){ width="500" }
  <figcaption><em>Overview of the workflow supported by the <i>GAICo</i> library</em></figcaption>
</figure>

## Features

- Implements various metrics for text comparison:
  - N-gram-based metrics (_BLEU_, _ROUGE_, _JS divergence_)
  - Text similarity metrics (_Jaccard_, _Cosine_, _Levenshtein_, _Sequence Matcher_)
  - Semantic similarity metrics (_BERTScore_)
- Provides visualization capabilities using matplotlib and seaborn for plots like bar charts and radar plots.
- Allows exportation of results to CSV files, including scores and threshold pass/fail status.
- Provides streamlined `Experiment` class for easy comparison of multiple models, applying thresholds, plotting, and reporting.
- Supports batch processing for efficient computation.
- Optimized for different input types (lists, numpy arrays, pandas Series).
- Has extendable architecture for easy addition of new metrics.
- Supports automated testing of metrics using [Pytest](https://docs.pytest.org/en/stable/).

## Installation

You can install GAICo directly from PyPI using pip:

```shell
pip install gaico
```

The default installation includes core metrics and some visualization modules which are lightweight. For optional features:

- To include the **BERTScore** metric (which has larger dependencies like PyTorch):

  ```shell
  pip install gaico[bertscore]
  ```

- To include the **CosineSimilarity** metric (requires scikit-learn):

  ```shell
  pip install gaico[cosine]
  ```

- To include the **JSDivergence** metric (requires SciPy):

  ```shell
  pip install gaico[jsd]
  ```

- To install with **all optional features**:
  ```shell
  pip install gaico[bertscore,cosine,jsd]
  ```

### Installation Size Comparison

The following table provides an _estimated_ overview of the relative disk space impact of different installation options. Actual sizes may vary depending on your operating system, Python version, and existing packages. These are primarily to illustrate the relative impact of optional dependencies.

_Note:_ Core dependencies include: `levenshtein`, `matplotlib`, `numpy`, `pandas`, `rouge-score`, and `seaborn`.

| Installation Command                      | Dependencies                                                 | Estimated Total Size Impact |
| ----------------------------------------- | ------------------------------------------------------------ | --------------------------- |
| `pip install gaico`                       | Core                                                         | 210 MB                      |
| `pip install gaico[jsd]`                  | Core + `scipy`                                               | 310 MB                      |
| `pip install gaico[cosine]`               | Core + `scikit-learn`                                        | 360 MB                      |
| `pip install gaico[bertscore]`            | Core + `bert-score` (includes `torch`, `transformers`, etc.) | 800 MB                      |
| `pip install gaico[bertscore,cosine,jsd]` | Core + all dependencies from above                           | 950 MB                      |

### For Developers (Installing from source)

If you want to contribute to GAICo or install it from source for development:

1. Clone the repository:

   ```shell
   git clone https://github.com/ai4society/GenAIResultsComparator.git
   cd GenAIResultsComparator
   ```

2. Set up a virtual environment and install dependencies:

   We recommend using [UV](https://docs.astral.sh/uv/#installation) for managing environments and dependencies.

   ```shell
   # Create a virtual environment (e.g., Python 3.12 recommended)
   uv venv
   # Activate the environment
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   # Install the package in editable mode with all development dependencies (includes all optional features)
   uv pip install -e ".[dev]"
   ```

   _If you don't want to use `uv`,_ you can install the dependencies with the following commands:

   ```shell
   # Create a virtual environment (e.g., Python 3.12 recommended)
   python3 -m venv .venv
   # Activate the environment
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate
   # Install the package in editable mode with development and visualization extras
   pip install -e ".[dev]"
   ```

   _(Note: The `dev` extra installs GAICo with all its optional features like `visualization` and `bertscore`, plus dependencies for testing, linting, building, and documentation.)_

3. Set up pre-commit hooks (optional but recommended for contributors):

   ```shell
   pre-commit install
   ```

## Citation

If you find this project useful, please consider citing it in your work:

```bibtex
@software{AI4Society_GAICo_GenAI_Results,
  author = {{Nitin Gupta, Pallav Koppisetti, Biplav Srivastava}},
  license = {MIT},
  title = {{GAICo: GenAI Results Comparator}},
  year = {2025},
  url = {https://github.com/ai4society/GenAIResultsComparator}
}
```

## Acknowledgments

- The library is developed by [Nitin Gupta](https://github.com/g-nitin), [Pallav Koppisetti](https://github.com/pallavkoppisetti), and [Biplav Srivastava](https://github.com/biplav-s). Members of [AI4Society](https://ai4society.github.io) contributed to this tool as part of ongoing discussions. Major contributors are credited.
- This library uses several open-source packages including NLTK, scikit-learn, and others. Special thanks to the creators and maintainers of the implemented metrics.

## Contact

If you have any questions, feel free to reach out to us at [ai4societyteam@gmail.com](mailto:ai4societyteam@gmail.com).
